History

Merge tag 'amd-drm-next-6.16-2025-05-09' of https://gitlab.freedesktop.org/agd5f/linux into drm-next

amd-drm-next-6.16-2025-05-09: amdgpu: - IPS fixes - DSC cleanup - DC Scaling updates - DC FP fixes - Fused I2C-over-AUX updates - SubVP fixes - Freesync fix - DMUB AUX fixes - VCN fix - Hibernation fixes - HDP fixes - DCN 2.1 fixes - DPIA fixes - DMUB updates - Use drm_file_err in amdgpu - Enforce isolation updates - Use new dma_fence helpers - USERQ fixes - Documentation updates - Misc code cleanups - SR-IOV updates - RAS updates - PSP 12 cleanups amdkfd: - Update error messages for SDMA - Userptr updates drm: - Add drm_file_err function dma-buf: - Add a helper to sort and deduplicate dma_fence arrays

Dave Airlie committed 1 year ago

Commit 1faeeb32 parents f9fa012 afc6053338 files · +15952 −9761Browse files

Unified · Split

338 files
  1. Documentation/gpu/amdgpu/amd-hardware-list-info.rst +23 −0
  2. Documentation/gpu/amdgpu/amdgpu-glossary.rst +75 −0
  3. Documentation/gpu/amdgpu/apu-asic-info-table.csv +2 −0
  4. Documentation/gpu/amdgpu/debugfs.rst +210 −0
  5. Documentation/gpu/amdgpu/debugging.rst +7 −0
  6. Documentation/gpu/amdgpu/display/dc-debug.rst +1 −1
  7. Documentation/gpu/amdgpu/driver-core.rst +56 −25
  8. Documentation/gpu/amdgpu/driver-misc.rst +0 −17
  9. Documentation/gpu/amdgpu/gc/index.rst +52 −0
  10. Documentation/gpu/amdgpu/gc/mes.rst +38 −0
  11. Documentation/gpu/amdgpu/index.rst +4 −1
  12. Documentation/gpu/amdgpu/pipe_and_queue_abstraction.svg +1279 −0
  13. drivers/dma-buf/dma-fence-unwrap.c +37 −14
  14. drivers/gpu/drm/amd/amdgpu/Makefile +7 −2
  15. drivers/gpu/drm/amd/amdgpu/amdgpu.h +98 −6
  16. drivers/gpu/drm/amd/amdgpu/amdgpu_aca.c +3 −0
  17. drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c +0 −18
  18. drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +12 −0
  19. drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c +24 −85
  20. drivers/gpu/drm/amd/amdgpu/amdgpu_cper.c +2 −2
  21. drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +23 −1
  22. drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +351 −178
  23. drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +94 −16
  24. drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.c +233 −0
  25. drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.h +69 −0
  26. drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +166 −13
  27. drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c +150 −36
  28. drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +55 −7
  29. drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +4 −0
  30. drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.c +20 −0
  31. drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.h +2 −0
  32. drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c +2 −2
  33. drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c +11 −3
  34. drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c +13 −2
  35. drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h +1 −0
  36. drivers/gpu/drm/amd/amdgpu/amdgpu_job.h +1 −0
  37. drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +71 −10
  38. drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +55 −1051
  39. drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +4 −64
  40. drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5 −1
  41. drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +2 −1
  42. drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +10 −0
  43. drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +10 −5
  44. drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +27 −8
  45. drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +38 −74
  46. drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +75 −15
  47. drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h +14 −0
  48. drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c +43 −65
  49. drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h +9 −9
  50. drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +19 −6
  51. drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h +2 −1
  52. drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c +2 −3
  53. drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h +2 −0
  54. drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +919 −0
  55. drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h +135 −0
  56. drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c +966 −0
  57. drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.h +76 −0
  58. drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c +2 −2
  59. drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h +0 −1
  60. drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +25 −4
  61. drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +14 −3
  62. drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +3 −2
  63. drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +18 −6
  64. drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h +9 −4
  65. drivers/gpu/drm/amd/amdgpu/atom.c +1 −0
  66. drivers/gpu/drm/amd/amdgpu/cik_sdma.c +4 −14
  67. drivers/gpu/drm/amd/amdgpu/cikd.h +0 −3
  68. drivers/gpu/drm/amd/amdgpu/dce_v10_0.c +2 −2
  69. drivers/gpu/drm/amd/amdgpu/dce_v11_0.c +3 −4
  70. drivers/gpu/drm/amd/amdgpu/dce_v6_0.c +100 −106
  71. drivers/gpu/drm/amd/amdgpu/dce_v8_0.c +8 −8
  72. drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +91 −48
  73. drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +539 −193
  74. drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c +425 −191
  75. drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c +15 −31
  76. drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c +9 −51
  77. drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +5 −31
  78. drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +73 −98
  79. drivers/gpu/drm/amd/amdgpu/gfx_v9_4_2.c +2 −2
  80. drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c +37 −15
  81. drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +0 −4
  82. drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c +1 −5
  83. drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c +1 −5
  84. drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c +29 −17
  85. drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +1 −8
  86. drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +2 −12
  87. drivers/gpu/drm/amd/amdgpu/hdp_v4_0.c +1 −12
  88. drivers/gpu/drm/amd/amdgpu/hdp_v5_0.c +1 −12
  89. drivers/gpu/drm/amd/amdgpu/hdp_v5_2.c +11 −1
  90. drivers/gpu/drm/amd/amdgpu/hdp_v6_0.c +1 −12
  91. drivers/gpu/drm/amd/amdgpu/hdp_v7_0.c +1 −12
  92. drivers/gpu/drm/amd/amdgpu/ih_v6_0.c +5 −1
  93. drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c +2 −2
  94. drivers/gpu/drm/amd/amdgpu/mes_userqueue.c +355 −0
  95. drivers/gpu/drm/amd/amdgpu/mes_userqueue.h +30 −0
  96. drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +27 −47
  97. drivers/gpu/drm/amd/amdgpu/mes_v12_0.c +25 −46
  98. drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c +52 −14
  99. drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +46 −17
  100. drivers/gpu/drm/amd/amdgpu/mxgpu_ai.h +4 −0
  101. drivers/gpu/drm/amd/amdgpu/mxgpu_nv.c +59 −3
  102. drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h +4 −0
  103. drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c +6 −2
  104. drivers/gpu/drm/amd/amdgpu/psp_v12_0.c +17 −85
  105. drivers/gpu/drm/amd/amdgpu/psp_v13_0.c +51 −0
  106. drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c +17 −19
  107. drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +114 −157
  108. drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +87 −117
  109. drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c +80 −60
  110. drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c +107 −114
  111. drivers/gpu/drm/amd/amdgpu/si.c +214 −199
  112. drivers/gpu/drm/amd/amdgpu/si_dma.c +66 −50
  113. drivers/gpu/drm/amd/amdgpu/si_enums.h +1 −233
  114. drivers/gpu/drm/amd/amdgpu/si_ih.c +9 −8
  115. drivers/gpu/drm/amd/amdgpu/sid.h +107 −1448
  116. drivers/gpu/drm/amd/amdgpu/soc15.c +5 −0
  117. drivers/gpu/drm/amd/amdgpu/soc15d.h +5 −0
  118. drivers/gpu/drm/amd/amdgpu/umc_v12_0.c +10 −0
  119. drivers/gpu/drm/amd/amdgpu/umc_v12_0.h +2 −0
  120. drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c +1 −0
  121. drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c +1 −0
  122. drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c +1 −0
  123. drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c +19 −2
  124. drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c +62 −22
  125. drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c +27 −0
  126. drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c +18 −2
  127. drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c +2 −2
  128. drivers/gpu/drm/amd/amdgpu/vega20_ih.c +5 −1
  129. drivers/gpu/drm/amd/amdkfd/kfd_debugfs.c +76 −0
  130. drivers/gpu/drm/amd/amdkfd/kfd_device.c +0 −5
  131. drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +8 −6
  132. drivers/gpu/drm/amd/amdkfd/kfd_events.c +19 −0
  133. drivers/gpu/drm/amd/amdkfd/kfd_int_process_v10.c +94 −48
  134. drivers/gpu/drm/amd/amdkfd/kfd_int_process_v11.c +59 −33
  135. drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c +110 −32
  136. drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c +4 −0
  137. drivers/gpu/drm/amd/amdkfd/kfd_packet_manager_v9.c +6 −5
  138. drivers/gpu/drm/amd/amdkfd/kfd_priv.h +7 −0
  139. drivers/gpu/drm/amd/amdkfd/kfd_process.c +7 −0
  140. drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +9 −2
  141. drivers/gpu/drm/amd/amdkfd/kfd_queue.c +2 −2
  142. drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c +22 −2
  143. drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h +1 −0
  144. drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1 −2
  145. drivers/gpu/drm/amd/amdkfd/kfd_topology.c +24 −17
  146. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +375 −214
  147. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h +24 −1
  148. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c +49 −7
  149. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c +13 −0
  150. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c +33 −12
  151. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +2 −2
  152. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.c +0 −8
  153. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.c +2 −2
  154. drivers/gpu/drm/amd/display/dc/Makefile +20 −21
  155. drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c +0 −5
  156. drivers/gpu/drm/amd/display/dc/bios/bios_parser.c +2 −2
  157. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.c +0 −14
  158. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.h +0 −1
  159. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.c +0 −14
  160. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.h +0 −1
  161. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c +14 −15
  162. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn351_clk_mgr.c +1 −0
  163. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_clk_mgr.c +7 −1
  164. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_smu.c +3 −3
  165. drivers/gpu/drm/amd/display/dc/core/dc.c +131 −11
  166. drivers/gpu/drm/amd/display/dc/core/dc_debug.c +2 −0
  167. drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c +2 −2
  168. drivers/gpu/drm/amd/display/dc/core/dc_resource.c +6 −42
  169. drivers/gpu/drm/amd/display/dc/core/dc_state.c +107 −4
  170. drivers/gpu/drm/amd/display/dc/core/dc_stream.c +63 −15
  171. drivers/gpu/drm/amd/display/dc/dc.h +182 −165
  172. drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c +109 −93
  173. drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h +1 −1
  174. drivers/gpu/drm/amd/display/dc/dc_fused_io.c +148 −0
  175. drivers/gpu/drm/amd/display/dc/dc_fused_io.h +31 −0
  176. drivers/gpu/drm/amd/display/dc/dc_helper.c +7 −1
  177. drivers/gpu/drm/amd/display/dc/dc_state_priv.h +20 −0
  178. drivers/gpu/drm/amd/display/dc/dc_stream.h +11 −0
  179. drivers/gpu/drm/amd/display/dc/dc_types.h +15 −2
  180. drivers/gpu/drm/amd/display/dc/dccg/dcn35/dcn35_dccg.c +37 −25
  181. drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c +5 −4
  182. drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.h +1 −1
  183. drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c +2 −0
  184. drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c +2 −1
  185. drivers/gpu/drm/amd/display/dc/dce/dmub_replay.c +13 −6
  186. drivers/gpu/drm/amd/display/dc/dce60/Makefile +1 −2
  187. drivers/gpu/drm/amd/display/dc/dce80/dce80_timing_generator.c +2 −2
  188. drivers/gpu/drm/amd/display/dc/dm_helpers.h +8 −0
  189. drivers/gpu/drm/amd/display/dc/dml/dcn30/display_mode_vba_30.c +1 −0
  190. drivers/gpu/drm/amd/display/dc/dml/dcn31/display_mode_vba_31.c +1 −0
  191. drivers/gpu/drm/amd/display/dc/dml/dcn314/display_mode_vba_314.c +1 −0
  192. drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c +1 −0
  193. drivers/gpu/drm/amd/display/dc/dml2/Makefile +0 −3
  194. drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_translation_helper.c +9 −3
  195. drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_wrapper.c +4 −4
  196. drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top.h +1 −0
  197. drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_dchub_registers.h +2 −0
  198. drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_display_cfg_types.h +2 −1
  199. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4.c +5 −5
  200. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c +2138 −2208
  201. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_shared_types.h +2 −0
  202. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_utils.c +71 −71
  203. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_dpmm/dml2_dpmm_dcn4.c +4 −0
  204. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_mcg/dml2_mcg_dcn4.c +4 −0
  205. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_pmo/dml2_pmo_dcn4_fams2.c +4 −4
  206. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_top/dml2_top_soc15.c +1 −1
  207. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.c +0 −31
  208. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.h +42 −35
  209. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_internal_shared_types.h +6 −0
  210. drivers/gpu/drm/amd/display/dc/dml2/dml2_dc_resource_mgmt.c +12 −12
  211. drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c +6 −10
  212. drivers/gpu/drm/amd/display/dc/dml2/dml2_wrapper.c +4 −1
  213. drivers/gpu/drm/amd/display/dc/dpp/dcn30/dcn30_dpp.c +3 −6
  214. drivers/gpu/drm/amd/display/dc/dpp/dcn35/dcn35_dpp.c +1 −1
  215. drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.c +0 −7
  216. drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.h +6 −0
  217. drivers/gpu/drm/amd/display/dc/dsc/dcn401/dcn401_dsc.c +0 −6
  218. drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c +14 −0
  219. drivers/gpu/drm/amd/display/dc/hubbub/dcn21/dcn21_hubbub.c +3 −3
  220. drivers/gpu/drm/amd/display/dc/hwss/Makefile +18 −8
  221. drivers/gpu/drm/amd/display/dc/hwss/dce110/dce110_hwseq.c +4 −4
  222. drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.c +1 −1
  223. drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.h +0 −0
  224. drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.c +122 −1
  225. drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.h +7 −0
  226. drivers/gpu/drm/amd/display/dc/hwss/dcn20/dcn20_hwseq.c +1 −1
  227. drivers/gpu/drm/amd/display/dc/hwss/dcn314/dcn314_hwseq.c +14 −0
  228. drivers/gpu/drm/amd/display/dc/hwss/dcn32/dcn32_hwseq.c +3 −1
  229. drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c +14 −7
  230. drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c +2 −0
  231. drivers/gpu/drm/amd/display/dc/hwss/dcn351/dcn351_init.c +3 −1
  232. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c +317 −37
  233. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h +8 −0
  234. drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h +6 −2
  235. drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer_private.h +2 −0
  236. drivers/gpu/drm/amd/display/dc/inc/core_status.h +3 −0
  237. drivers/gpu/drm/amd/display/dc/inc/core_types.h +10 −4
  238. drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h +2 −1
  239. drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h +29 −17
  240. drivers/gpu/drm/amd/display/dc/inc/hw/optc.h +1 −0
  241. drivers/gpu/drm/amd/display/dc/inc/reg_helper.h +4 −0
  242. drivers/gpu/drm/amd/display/dc/inc/resource.h +0 −2
  243. drivers/gpu/drm/amd/display/dc/irq/dcn32/irq_service_dcn32.c +46 −15
  244. drivers/gpu/drm/amd/display/dc/irq/dcn401/irq_service_dcn401.c +46 −14
  245. drivers/gpu/drm/amd/display/dc/irq_types.h +16 −0
  246. drivers/gpu/drm/amd/display/dc/link/link_factory.c +5 −1
  247. drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_dpia.c +1 −0
  248. drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c +4 −3
  249. drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c +7 −4
  250. drivers/gpu/drm/amd/display/dc/mpc/dcn32/dcn32_mpc.c +192 −188
  251. drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.c +0 −11
  252. drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.h +0 −14
  253. drivers/gpu/drm/amd/display/dc/optc/dcn35/dcn35_optc.c +1 −0
  254. drivers/gpu/drm/amd/display/dc/resource/Makefile +18 −8
  255. drivers/gpu/drm/amd/display/dc/resource/dce100/dce100_resource.c +3 −3
  256. drivers/gpu/drm/amd/display/dc/resource/dce110/dce110_resource.c +2 −2
  257. drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.c +2 −2
  258. drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.h +1 −1
  259. drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.c +3 −3
  260. drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.h +0 −0
  261. drivers/gpu/drm/amd/display/dc/resource/dce80/dce80_resource.c +2 −2
  262. drivers/gpu/drm/amd/display/dc/resource/dcn10/dcn10_resource.c +7 −2
  263. drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.c +3 −3
  264. drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.h +1 −1
  265. drivers/gpu/drm/amd/display/dc/resource/dcn21/dcn21_resource.c +3 −3
  266. drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.c +2 −4
  267. drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.h +1 −1
  268. drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.c +2 −2
  269. drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.h +1 −1
  270. drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.c +2 −2
  271. drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.h +1 −1
  272. drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c +59 −13
  273. drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.h +5 −1
  274. drivers/gpu/drm/amd/display/dc/resource/dcn321/dcn321_resource.c +3 −1
  275. drivers/gpu/drm/amd/display/dc/resource/dcn35/dcn35_resource.c +4 −4
  276. drivers/gpu/drm/amd/display/dc/resource/dcn351/dcn351_resource.c +3 −3
  277. drivers/gpu/drm/amd/display/dc/resource/dcn36/dcn36_resource.c +4 −4
  278. drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.c +45 −7
  279. drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.h +1 −1
  280. drivers/gpu/drm/amd/display/dc/sspl/dc_spl.c +14 −11
  281. drivers/gpu/drm/amd/display/dc/sspl/spl_fixpt31_32.c +0 −4
  282. drivers/gpu/drm/amd/display/dmub/dmub_srv.h +110 −47
  283. drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h +73 −2
  284. drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.c +83 −38
  285. drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.h +3 −1
  286. drivers/gpu/drm/amd/display/dmub/src/dmub_srv.c +247 −98
  287. drivers/gpu/drm/amd/display/dmub/src/dmub_srv_stat.c +4 −17
  288. drivers/gpu/drm/amd/display/include/gpio_service_interface.h +3 −0
  289. drivers/gpu/drm/amd/display/modules/hdcp/hdcp.h +1 −0
  290. drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c +43 −10
  291. drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_transition.c +36 −12
  292. drivers/gpu/drm/amd/display/modules/hdcp/hdcp_ddc.c +73 −0
  293. drivers/gpu/drm/amd/display/modules/inc/mod_hdcp.h +36 −2
  294. drivers/gpu/drm/amd/include/amd_shared.h +1 −1
  295. drivers/gpu/drm/amd/include/asic_reg/dce/dce_6_0_sh_mask.h +2 −0
  296. drivers/gpu/drm/amd/include/asic_reg/gca/gfx_6_0_d.h +2 −0
  297. drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_d.h +20 −3
  298. drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_sh_mask.h +41 −0
  299. drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_d.h +44 −0
  300. drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_sh_mask.h +185 −3
  301. drivers/gpu/drm/amd/include/atombios.h +1 −1
  302. drivers/gpu/drm/amd/include/kgd_pp_interface.h +1 −0
  303. drivers/gpu/drm/amd/include/v11_structs.h +4 −4
  304. drivers/gpu/drm/amd/include/v12_structs.h +4 −4
  305. drivers/gpu/drm/amd/pm/amdgpu_dpm.c +43 −0
  306. drivers/gpu/drm/amd/pm/amdgpu_pm.c +34 −0
  307. drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h +3 −0
  308. drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c +190 −168
  309. drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c +24 −18
  310. drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c +8 −0
  311. drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.c +0 −36
  312. drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.h +0 −2
  313. drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c +34 −0
  314. drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h +26 −3
  315. drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h +7 −0
  316. drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h +3 −1
  317. drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h +1 −0
  318. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v11_0.h +0 −5
  319. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v13_0.h +2 −12
  320. drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c +1 −1
  321. drivers/gpu/drm/amd/pm/swsmu/smu11/smu_v11_0.c +0 −39
  322. drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c +11 −2
  323. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c +10 −101
  324. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_12_ppt.c +14 −24
  325. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c +126 −29
  326. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.h +21 −0
  327. drivers/gpu/drm/drm_file.c +34 −0
  328. drivers/gpu/drm/radeon/atombios.h +1 −2
  329. drivers/gpu/drm/radeon/r600_hdmi.c +0 −22
  330. drivers/gpu/drm/radeon/radeon.h +0 −3
  331. drivers/gpu/drm/radeon/radeon_asic.h +0 −1
  332. drivers/gpu/drm/radeon/radeon_cs.c +0 −1
  333. drivers/gpu/drm/radeon/radeon_fence.c +0 −42
  334. drivers/gpu/drm/radeon/sid.h +1 −1
  335. include/drm/drm_file.h +3 −0
  336. include/linux/dma-fence-unwrap.h +2 −0
  337. include/uapi/drm/amdgpu_drm.h +316 −0
  338. include/uapi/linux/kfd_ioctl.h +5 −0
Documentation/gpu/amdgpu/amd-hardware-list-info.rst+23−0
@@ -0,0 +1,23 @@
1+=================================================
2+ AMD Hardware Components Information per Product
3+=================================================
4+
5+On this page, you can find the AMD product name and which component version is
6+part of it.
7+
8+Accelerated Processing Units (APU) Info
9+---------------------------------------
10+
11+.. csv-table::
12+ :header-rows: 1
13+ :widths: 3, 2, 2, 1, 1, 1, 1
14+ :file: ./apu-asic-info-table.csv
15+
16+Discrete GPU Info
17+-----------------
18+
19+.. csv-table::
20+ :header-rows: 1
21+ :widths: 3, 2, 2, 1, 1, 1
22+ :file: ./dgpu-asic-info-table.csv
23+
Documentation/gpu/amdgpu/amdgpu-glossary.rst+75−0
@@ -12,18 +12,39 @@ we have a dedicated glossary for Display Core at · 11 unmodified lines
1212 The number of CUs that are active on the system. The number of active
1313 CUs may be less than SE * SH * CU depending on the board configuration.
1414
15+ BACO
16+ Bus Alive, Chip Off
17+
18+ BOCO
19+ Bus Off, Chip Off
20+
1521 CE
1622 Constant Engine
1723
24+ CIK
25+ Sea Islands
26+
27+ CB
28+ Color Buffer
29+
1830 CP
1931 Command Processor
2032
2133 CPLIB
2234 Content Protection Library
2335
36+ CS
37+ Command Submission
38+
39+ CSB
40+ Clear State Indirect Buffer
41+
2442 CU
2543 Compute Unit
2644
45+ DB
46+ Depth Buffer
47+
2748 DFS
2849 Digital Frequency Synthesizer
2950
@@ -33,6 +54,9 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
3354 EOP
3455 End Of Pipe/Pipeline
3556
57+ FLR
58+ Function Level Reset
59+
3660 GART
3761 Graphics Address Remapping Table. This is the name we use for the GPUVM
3862 page table used by the GPU kernel driver. It remaps system resources
@@ -45,6 +69,12 @@ we have a dedicated glossary for Display Core at · 6 unmodified lines
4569 GC
4670 Graphics and Compute
4771
72+ GDS
73+ Global Data Share
74+
75+ GE
76+ Geometry Engine
77+
4878 GMC
4979 Graphic Memory Controller
5080
@@ -80,6 +110,9 @@ we have a dedicated glossary for Display Core at · 29 unmodified lines
80110 KCQ
81111 Kernel Compute Queue
82112
113+ KFD
114+ Kernel Fusion Driver
115+
83116 KGQ
84117 Kernel Graphics Queue
85118
@@ -89,6 +122,9 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
89122 MC
90123 Memory Controller
91124
125+ MCBP
126+ Mid Command Buffer Preemption
127+
92128 ME
93129 MicroEngine (Graphics)
94130
@@ -104,6 +140,9 @@ we have a dedicated glossary for Display Core at · 9 unmodified lines
104140 MQD
105141 Memory Queue Descriptor
106142
143+ PA
144+ Primitive Assembler / Physical Address
145+
107146 PFP
108147 Pre-Fetch Parser (Graphics)
109148
@@ -113,24 +152,39 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
113152 PSP
114153 Platform Security Processor
115154
155+ RB
156+ Render Backends. Some people called it ROPs.
157+
116158 RLC
117159 RunList Controller. This name is a remnant of past ages and doesn't have
118160 much meaning today. It's a group of general-purpose helper engines for
119161 the GFX block. It's involved in GFX power management and SR-IOV, among
120162 other things.
121163
164+ SC
165+ Scan Converter
166+
122167 SDMA
123168 System DMA
124169
125170 SE
126171 Shader Engine
127172
173+ SGPR
174+ Scalar General-Purpose Registers
175+
128176 SH
129177 SHader array
130178
179+ SI
180+ Southern Islands
181+
131182 SMU/SMC
132183 System Management Unit / System Management Controller
133184
185+ SPI
186+ Shader Processor Input
187+
134188 SRLC
135189 Save/Restore List Control
136190
@@ -143,12 +197,21 @@ we have a dedicated glossary for Display Core at · 6 unmodified lines
143197 SS
144198 Spread Spectrum
145199
200+ SX
201+ Shader Export
202+
146203 TA
147204 Trusted Application
148205
206+ TC
207+ Texture Cache
208+
149209 TOC
150210 Table of Contents
151211
212+ UMSCH
213+ User Mode Scheduler
214+
152215 UVD
153216 Unified Video Decoder
154217
@@ -158,5 +221,17 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
158221 VCN
159222 Video Codec Next
160223
224+ VGPR
225+ Vector General-Purpose Registers
226+
227+ VMID
228+ Virtual Memory ID
229+
161230 VPE
162231 Video Processing Engine
232+
233+ XCC
234+ Accelerator Core Complex
235+
236+ XCP
237+ Accelerator Core Partition
Documentation/gpu/amdgpu/apu-asic-info-table.csv+2−0
@@ -13,3 +13,5 @@ Ryzen 7x20 series, Mendocino, 3.1.6, 10.3.7, 3.1.1, 5.2.7, 13.0.8 · 12 unmodified lines
1313 Ryzen 7x40 series, Phoenix, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11
1414 Ryzen 8x40 series, Hawk Point, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11
1515 Ryzen AI 300 series, Strix Point, 3.5.0, 11.5.0, 4.0.5, 6.1.0, 14.0.0
16+Ryzen AI 350 series, Krackan Point, 3.5.0, 11.5.2, 4.0.5, 6.1.2, 14.0.4
17+Ryzen AI Max 300 series, Strix Halo, 3.5.1, 11.5.1, 4.0.6, 6.1.1, 14.0.1
Documentation/gpu/amdgpu/debugfs.rst+210−0
@@ -0,0 +1,210 @@
1+==============
2+AMDGPU DebugFS
3+==============
4+
5+The amdgpu driver provides a number of debugfs files to aid in debugging
6+issues in the driver. These are usually found in
7+/sys/kernel/debug/dri/<num>.
8+
9+DebugFS Files
10+=============
11+
12+amdgpu_benchmark
13+----------------
14+
15+Run benchmarks using the DMA engine the driver uses for GPU memory paging.
16+Write a number to the file to run the test. The results are written to the
17+kernel log. VRAM is on device memory (dGPUs) or carve out (APUs) and GTT
18+(Graphics Translation Tables) is system memory that is accessible by the GPU.
19+The following tests are available:
20+
21+- 1: simple test, VRAM to GTT and GTT to VRAM
22+- 2: simple test, VRAM to VRAM
23+- 3: GTT to VRAM, buffer size sweep, powers of 2
24+- 4: VRAM to GTT, buffer size sweep, powers of 2
25+- 5: VRAM to VRAM, buffer size sweep, powers of 2
26+- 6: GTT to VRAM, buffer size sweep, common display sizes
27+- 7: VRAM to GTT, buffer size sweep, common display sizes
28+- 8: VRAM to VRAM, buffer size sweep, common display sizes
29+
30+amdgpu_test_ib
31+--------------
32+
33+Read this file to run simple IB (Indirect Buffer) tests on all kernel managed
34+rings. IBs are command buffers usually generated by userspace applications
35+which are submitted to the kernel for execution on an particular GPU engine.
36+This just runs the simple IB tests included in the kernel. These tests
37+are engine specific and verify that IB submission works.
38+
39+amdgpu_discovery
40+----------------
41+
42+Provides raw access to the IP discovery binary provided by the GPU. Read this
43+file to access the raw binary. This is useful for verifying the contents of
44+the IP discovery table. It is chip specific.
45+
46+amdgpu_vbios
47+------------
48+
49+Provides raw access to the ROM binary image from the GPU. Read this file to
50+access the raw binary. This is useful for verifying the contents of the
51+video BIOS ROM. It is board specific.
52+
53+amdgpu_evict_gtt
54+----------------
55+
56+Evict all buffers from the GTT memory pool. Read this file to evict all
57+buffers from this pool.
58+
59+amdgpu_evict_vram
60+-----------------
61+
62+Evict all buffers from the VRAM memory pool. Read this file to evict all
63+buffers from this pool.
64+
65+amdgpu_gpu_recover
66+------------------
67+
68+Trigger a GPU reset. Read this file to trigger reset the entire GPU.
69+All work currently running on the GPU will be lost.
70+
71+amdgpu_ring_<name>
72+------------------
73+
74+Provides read access to the kernel managed ring buffers for each ring <name>.
75+These are useful for debugging problems on a particular ring. The ring buffer
76+is how the CPU sends commands to the GPU. The CPU writes commands into the
77+buffer and then asks the GPU engine to process it. This is the raw binary
78+contents of the ring buffer. Use a tool like UMR to decode the rings into human
79+readable form.
80+
81+amdgpu_mqd_<name>
82+-----------------
83+
84+Provides read access to the kernel managed MQD (Memory Queue Descriptor) for
85+ring <name> managed by the kernel driver. MQDs define the features of the ring
86+and are used to store the ring's state when it is not connected to hardware.
87+The driver writes the requested ring features and metadata (GPU addresses of
88+the ring itself and associated buffers) to the MQD and the firmware uses the MQD
89+to populate the hardware when the ring is mapped to a hardware slot. Only
90+available on engines which use MQDs. This provides access to the raw MQD
91+binary.
92+
93+amdgpu_error_<name>
94+-------------------
95+
96+Provides an interface to set an error code on the dma fences associated with
97+ring <name>. The error code specified is propogated to all fences associated
98+with the ring. Use this to inject a fence error into a ring.
99+
100+amdgpu_pm_info
101+--------------
102+
103+Provides human readable information about the power management features
104+and state of the GPU. This includes current GFX clock, Memory clock,
105+voltages, average SoC power, temperature, GFX load, Memory load, SMU
106+feature mask, VCN power state, clock and power gating features.
107+
108+amdgpu_firmware_info
109+--------------------
110+
111+Lists the firmware versions for all firmwares used by the GPU. Only
112+entries with a non-0 version are valid. If the version is 0, the firmware
113+is not valid for the GPU.
114+
115+amdgpu_fence_info
116+-----------------
117+
118+Shows the last signalled and emitted fence sequence numbers for each
119+kernel driver managed ring. Fences are associated with submissions
120+to the engine. Emitted fences have been submitted to the ring
121+and signalled fences have been signalled by the GPU. Rings with a
122+larger emitted fence value have outstanding work that is still being
123+processed by the engine that owns that ring. When the emitted and
124+signalled fence values are equal, the ring is idle.
125+
126+amdgpu_gem_info
127+---------------
128+
129+Lists all of the PIDs using the GPU and the GPU buffers that they have
130+allocated. This lists the buffer size, pool (VRAM, GTT, etc.), and buffer
131+attributes (CPU access required, CPU cache attributes, etc.).
132+
133+amdgpu_vm_info
134+--------------
135+
136+Lists all of the PIDs using the GPU and the GPU buffers that they have
137+allocated as well as the status of those buffers relative to that process'
138+GPU virtual address space (e.g., evicted, idle, invalidated, etc.).
139+
140+amdgpu_sa_info
141+--------------
142+
143+Prints out all of the suballocations (sa) by the suballocation manager in the
144+kernel driver. Prints the GPU address, size, and fence info associated
145+with each suballocation. The suballocations are used internally within
146+the kernel driver for various things.
147+
148+amdgpu_<pool>_mm
149+----------------
150+
151+Prints TTM information about the memory pool <pool>.
152+
153+amdgpu_vram
154+-----------
155+
156+Provides direct access to VRAM. Used by tools like UMR to inspect
157+objects in VRAM.
158+
159+amdgpu_iomem
160+------------
161+
162+Provides direct access to GTT memory. Used by tools like UMR to inspect
163+GTT memory.
164+
165+amdgpu_regs_*
166+-------------
167+
168+Provides direct access to various register aperatures on the GPU. Used
169+by tools like UMR to access GPU registers.
170+
171+amdgpu_regs2
172+------------
173+
174+Provides an IOCTL interface used by UMR for interacting with GPU registers.
175+
176+
177+amdgpu_sensors
178+--------------
179+
180+Provides an interface to query GPU power metrics (temperature, average
181+power, etc.). Used by tools like UMR to query GPU power metrics.
182+
183+
184+amdgpu_gca_config
185+-----------------
186+
187+Provides an interface to query GPU details (Graphics/Compute Array config,
188+PCI config, GPU family, etc.). Used by tools like UMR to query GPU details.
189+
190+amdgpu_wave
191+-----------
192+
193+Used to query GFX/compute wave information from the hardware. Used by tools
194+like UMR to query GFX/compute wave information.
195+
196+amdgpu_gpr
197+----------
198+
199+Used to query GFX/compute GPR (General Purpose Register) information from the
200+hardware. Used by tools like UMR to query GPRs when debugging shaders.
201+
202+amdgpu_gprwave
203+--------------
204+
205+Provides an IOCTL interface used by UMR for interacting with shader waves.
206+
207+amdgpu_fw_attestation
208+---------------------
209+
210+Provides an interface for reading back firmware attestation records.
Documentation/gpu/amdgpu/debugging.rst+7−0
@@ -2,6 +2,13 @@ · 1 unmodified line
22 GPU Debugging
33 ===============
44
5+General Debugging Options
6+=========================
7+
8+The DebugFS section provides documentation on a number files to aid in debugging
9+issues on the GPU.
10+
11+
512 GPUVM Debugging
613 ===============
714
Documentation/gpu/amdgpu/display/dc-debug.rst+1−1
@@ -154,7 +154,7 @@ of the display parameters, but the userspace might also cause this issue. One · 153 unmodified lines
154154 way to identify the source of the problem is to take a screenshot or make a
155155 desktop video capture when the problem happens; after checking the
156156 screenshot/video recording, if you don't see any of the artifacts, it means
157−that the issue is likely on the the driver side. If you can still see the
157+that the issue is likely on the driver side. If you can still see the
158158 problem in the data collected, it is an issue that probably happened during
159159 rendering, and the display code just got the framebuffer already corrupted.
160160
Documentation/gpu/amdgpu/driver-core.rst+56−25
@@ -67,36 +67,66 @@ GC (Graphics and Compute) · 66 unmodified lines
6767 This is the graphics and compute engine, i.e., the block that
6868 encompasses the 3D pipeline and and shader blocks. This is by far the
6969 largest block on the GPU. The 3D pipeline has tons of sub-blocks. In
70− addition to that, it also contains the CP microcontrollers (ME, PFP,
71− CE, MEC) and the RLC microcontroller. It's exposed to userspace for
72− user mode drivers (OpenGL, Vulkan, OpenCL, etc.)
70+ addition to that, it also contains the CP microcontrollers (ME, PFP, CE,
71+ MEC) and the RLC microcontroller. It's exposed to userspace for user mode
72+ drivers (OpenGL, Vulkan, OpenCL, etc.). More details in :ref:`Graphics (GFX)
73+ and Compute <amdgpu-gc>`.
7374
7475 VCN (Video Core Next)
7576 This is the multi-media engine. It handles video and image encode and
7677 decode. It's exposed to userspace for user mode drivers (VA-API,
7778 OpenMAX, etc.)
7879
79−Graphics and Compute Microcontrollers
80−-------------------------------------
81−
82−CP (Command Processor)
83− The name for the hardware block that encompasses the front end of the
84− GFX/Compute pipeline. Consists mainly of a bunch of microcontrollers
85− (PFP, ME, CE, MEC). The firmware that runs on these microcontrollers
86− provides the driver interface to interact with the GFX/Compute engine.
87−
88− MEC (MicroEngine Compute)
89− This is the microcontroller that controls the compute queues on the
90− GFX/compute engine.
91−
92− MES (MicroEngine Scheduler)
93− This is a new engine for managing queues. This is currently unused.
94−
95−RLC (RunList Controller)
96− This is another microcontroller in the GFX/Compute engine. It handles
97− power management related functionality within the GFX/Compute engine.
98− The name is a vestige of old hardware where it was originally added
99− and doesn't really have much relation to what the engine does now.
80+.. _pipes-and-queues-description:
81+
82+GFX, Compute, and SDMA Overall Behavior
83+=======================================
84+
85+.. note:: For simplicity, whenever the term block is used in this section, it
86+ means GFX, Compute, and SDMA.
87+
88+GFX, Compute and SDMA share a similar form of operation that can be abstracted
89+to facilitate understanding of the behavior of these blocks. See the figure
90+below illustrating the common components of these blocks:
91+
92+.. kernel-figure:: pipe_and_queue_abstraction.svg
93+
94+In the central part of this figure, you can see two hardware elements, one called
95+**Pipes** and another called **Queues**; it is important to highlight that Queues
96+must be associated with a Pipe and vice-versa. Every specific hardware IP may have
97+a different number of Pipes and, in turn, a different number of Queues; for
98+example, GFX 11 has two Pipes and two Queues per Pipe for the GFX front end.
99+
100+Pipe is the hardware that processes the instructions available in the Queues;
101+in other words, it is a thread executing the operations inserted in the Queue.
102+One crucial characteristic of Pipes is that they can only execute one Queue at
103+a time; no matter if the hardware has multiple Queues in the Pipe, it only runs
104+one Queue per Pipe.
105+
106+Pipes have the mechanics of swapping between queues at the hardware level.
107+Nonetheless, they only make use of Queues that are considered mapped. Pipes can
108+switch between queues based on any of the following inputs:
109+
110+1. Command Stream;
111+2. Packet by Packet;
112+3. Other hardware requests the change (e.g., MES).
113+
114+Queues within Pipes are defined by the Hardware Queue Descriptors (HQD).
115+Associated with the HQD concept, we have the Memory Queue Descriptor (MQD),
116+which is responsible for storing information about the state of each of the
117+available Queues in the memory. The state of a Queue contains information such
118+as the GPU virtual address of the queue itself, save areas, doorbell, etc. The
119+MQD also stores the HQD registers, which are vital for activating or
120+deactivating a given Queue. The scheduling firmware (e.g., MES) is responsible
121+for loading HQDs from MQDs and vice versa.
122+
123+The Queue-switching process can also happen with the firmware requesting the
124+preemption or unmapping of a Queue. The firmware waits for the HQD_ACTIVE bit
125+to change to low before saving the state into the MQD. To make a different
126+Queue become active, the firmware copies the MQD state into the HQD registers
127+and loads any additional state. Finally, it sets the HQD_ACTIVE bit to high to
128+indicate that the queue is active. The Pipe will then execute work from active
129+Queues.
100130
101131 Driver Structure
102132 ================
@@ -110,7 +140,8 @@ Some useful constructs: · 7 unmodified lines
110140 KIQ (Kernel Interface Queue)
111141 This is a control queue used by the kernel driver to manage other gfx
112142 and compute queues on the GFX/compute engine. You can use it to
113− map/unmap additional queues, etc.
143+ map/unmap additional queues, etc. This is replaced by MES on
144+ GFX 11 and newer hardware.
114145
115146 IB (Indirect Buffer)
116147 A command buffer for a particular engine. Rather than writing
Documentation/gpu/amdgpu/driver-misc.rst+0−17
@@ -50,23 +50,6 @@ board_info · 49 unmodified lines
5050 .. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
5151 :doc: board_info
5252
53−Accelerated Processing Units (APU) Info
54−---------------------------------------
55−
56−.. csv-table::
57− :header-rows: 1
58− :widths: 3, 2, 2, 1, 1, 1, 1
59− :file: ./apu-asic-info-table.csv
60−
61−Discrete GPU Info
62−-----------------
63−
64−.. csv-table::
65− :header-rows: 1
66− :widths: 3, 2, 2, 1, 1, 1
67− :file: ./dgpu-asic-info-table.csv
68−
69−
7053 GPU Memory Usage Information
7154 ============================
7255
Documentation/gpu/amdgpu/gc/index.rst+52−0
@@ -0,0 +1,52 @@
1+.. _amdgpu-gc:
2+
3+========================================
4+ drm/amdgpu - Graphics and Compute (GC)
5+========================================
6+
7+The relationship between the CPU and GPU can be described as the
8+producer-consumer problem, where the CPU fills out a buffer with operations
9+(producer) to be executed by the GPU (consumer). The requested operations in
10+the buffer are called Command Packets, which can be summarized as a compressed
11+way of transmitting command information to the graphics controller.
12+
13+The component that acts as the front end between the CPU and the GPU is called
14+the Command Processor (CP). This component is responsible for providing greater
15+flexibility to the GC since CP makes it possible to program various aspects of
16+the GPU pipeline. CP also coordinates the communication between the CPU and GPU
17+via a mechanism named **Ring Buffers**, where the CPU appends information to
18+the buffer while the GPU removes operations. It is relevant to highlight that a
19+CPU can add a pointer to the Ring Buffer that points to another region of
20+memory outside the Ring Buffer, and CP can handle it; this mechanism is called
21+**Indirect Buffer (IB)**. CP receives and parses the Command Streams (CS), and
22+writes the operations to the correct hardware blocks.
23+
24+Graphics (GFX) and Compute Microcontrollers
25+-------------------------------------------
26+
27+GC is a large block, and as a result, it has multiple firmware associated with
28+it. Some of them are:
29+
30+CP (Command Processor)
31+ The name for the hardware block that encompasses the front end of the
32+ GFX/Compute pipeline. Consists mainly of a bunch of microcontrollers
33+ (PFP, ME, CE, MEC). The firmware that runs on these microcontrollers
34+ provides the driver interface to interact with the GFX/Compute engine.
35+
36+ MEC (MicroEngine Compute)
37+ This is the microcontroller that controls the compute queues on the
38+ GFX/compute engine.
39+
40+ MES (MicroEngine Scheduler)
41+ This is the engine for managing queues. For more details check
42+ :ref:`MicroEngine Scheduler (MES) <amdgpu-mes>`.
43+
44+RLC (RunList Controller)
45+ This is another microcontroller in the GFX/Compute engine. It handles
46+ power management related functionality within the GFX/Compute engine.
47+ The name is a vestige of old hardware where it was originally added
48+ and doesn't really have much relation to what the engine does now.
49+
50+.. toctree::
51+
52+ mes.rst
Documentation/gpu/amdgpu/gc/mes.rst+38−0
@@ -0,0 +1,38 @@
1+.. _amdgpu-mes:
2+
3+=============================
4+ MicroEngine Scheduler (MES)
5+=============================
6+
7+.. note::
8+ Queue and ring buffer are used as a synonymous.
9+
10+.. note::
11+ This section assumes that you are familiar with the concept of Pipes, Queues, and GC.
12+ If not, check :ref:`GFX, Compute, and SDMA Overall Behavior<pipes-and-queues-description>`
13+ and :ref:`drm/amdgpu - Graphics and Compute (GC) <amdgpu-gc>`.
14+
15+Every GFX has a pipe component with one or more hardware queues. Pipes can
16+switch between queues depending on certain conditions, and one of the
17+components that can request a queue switch to a pipe is the MicroEngine
18+Scheduler (MES). Whenever the driver is initialized, it creates one MQD per
19+hardware queue, and then the MQDs are handed to the MES firmware for mapping
20+to:
21+
22+1. Kernel Queues (legacy): This queue is statically mapped to HQDs and never
23+ preempted. Even though this is a legacy feature, it is the current default, and
24+ most existing hardware supports it. When an application submits work to the
25+ kernel driver, it submits all of the application command buffers to the kernel
26+ queues. The CS IOCTL takes the command buffer from the applications and
27+ schedules them on the kernel queue.
28+
29+2. User Queues: These queues are dynamically mapped to the HQDs. Regarding the
30+ utilization of User Queues, the userspace application will create its user
31+ queues and submit work directly to its user queues with no need to IOCTL for
32+ each submission and no need to share a single kernel queue.
33+
34+In terms of User Queues, MES can dynamically map them to the HQD. If there are
35+more MQDs than HQDs, the MES firmware will preempt other user queues to make
36+sure each queues get a time slice; in other words, MES is a microcontroller
37+that handles the mapping and unmapping of MQDs into HQDs, as well as the
38+priorities and oversubscription of MQDs.
Documentation/gpu/amdgpu/index.rst+4−1
@@ -7,8 +7,10 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures. · 6 unmodified lines
77
88 .. toctree::
99
10− module-parameters
1110 driver-core
11+ amd-hardware-list-info
12+ module-parameters
13+ gc/index
1214 display/index
1315 flashing
1416 xgmi
@@ -16,5 +18,6 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures. · 1 unmodified line
1618 thermal
1719 driver-misc
1820 debugging
21+ debugfs
1922 process-isolation
2023 amdgpu-glossary
Documentation/gpu/amdgpu/pipe_and_queue_abstraction.svg+1279−0
@@ -0,0 +1,1279 @@
1+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
2+<!-- Created with Inkscape (http://www.inkscape.org/) -->
3+
4+<svg
5+ width="395.47891mm"
6+ height="234.73715mm"
7+ viewBox="0 0 395.47891 234.73714"
8+ version="1.1"
9+ id="svg1"
10+ inkscape:version="1.4 (e7c3feb100, 2024-10-09)"
11+ sodipodi:docname="pipe_and_queue_abstraction.svg"
12+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
13+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
14+ xmlns="http://www.w3.org/2000/svg"
15+ xmlns:svg="http://www.w3.org/2000/svg">
16+ <sodipodi:namedview
17+ id="namedview1"
18+ pagecolor="#ffffff"
19+ bordercolor="#000000"
20+ borderopacity="0.25"
21+ inkscape:showpageshadow="2"
22+ inkscape:pageopacity="0.0"
23+ inkscape:pagecheckerboard="0"
24+ inkscape:deskcolor="#d1d1d1"
25+ inkscape:document-units="mm"
26+ inkscape:zoom="1.6489689"
27+ inkscape:cx="713.17296"
28+ inkscape:cy="466.65527"
29+ inkscape:window-width="3840"
30+ inkscape:window-height="2083"
31+ inkscape:window-x="0"
32+ inkscape:window-y="0"
33+ inkscape:window-maximized="1"
34+ inkscape:current-layer="layer1" />
35+ <defs
36+ id="defs1">
37+ <marker
38+ style="overflow:visible"
39+ id="Dot"
40+ refX="0"
41+ refY="0"
42+ orient="auto"
43+ inkscape:stockid="Dot"
44+ markerWidth="1"
45+ markerHeight="1"
46+ viewBox="0 0 1 1"
47+ inkscape:isstock="true"
48+ inkscape:collect="always"
49+ preserveAspectRatio="xMidYMid">
50+ <path
51+ transform="scale(0.5)"
52+ style="fill:context-stroke;fill-rule:evenodd;stroke:none"
53+ d="M 5,0 C 5,2.76 2.76,5 0,5 -2.76,5 -5,2.76 -5,0 c 0,-2.76 2.3,-5 5,-5 2.76,0 5,2.24 5,5 z"
54+ sodipodi:nodetypes="sssss"
55+ id="path110" />
56+ </marker>
57+ <marker
58+ style="overflow:visible"
59+ id="marker109"
60+ refX="0"
61+ refY="0"
62+ orient="auto-start-reverse"
63+ inkscape:stockid="Stylized triangle arrow"
64+ markerWidth="1"
65+ markerHeight="1"
66+ viewBox="0 0 1 1"
67+ inkscape:isstock="true"
68+ inkscape:collect="always"
69+ preserveAspectRatio="xMidYMid">
70+ <path
71+ transform="scale(0.5)"
72+ style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
73+ d="m 6,0 c -3,1 -7,3 -9,5 0,0 0,-4 2,-5 -2,-1 -2,-5 -2,-5 2,2 6,4 9,5 z"
74+ id="path109" />
75+ </marker>
76+ <marker
77+ style="overflow:visible"
78+ id="ArrowTriangleStylized"
79+ refX="0"
80+ refY="0"
81+ orient="auto-start-reverse"
82+ inkscape:stockid="Stylized triangle arrow"
83+ markerWidth="1"
84+ markerHeight="1"
85+ viewBox="0 0 1 1"
86+ inkscape:isstock="true"
87+ inkscape:collect="always"
88+ preserveAspectRatio="xMidYMid">
89+ <path
90+ transform="scale(0.5)"
91+ style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
92+ d="m 6,0 c -3,1 -7,3 -9,5 0,0 0,-4 2,-5 -2,-1 -2,-5 -2,-5 2,2 6,4 9,5 z"
93+ id="path108" />
94+ </marker>
95+ <marker
96+ style="overflow:visible"
97+ id="ArrowWide"
98+ refX="0"
99+ refY="0"
100+ orient="auto-start-reverse"
101+ inkscape:stockid="Wide arrow"
102+ markerWidth="1"
103+ markerHeight="1"
104+ viewBox="0 0 1 1"
105+ inkscape:isstock="true"
106+ inkscape:collect="always"
107+ preserveAspectRatio="xMidYMid">
108+ <path
109+ style="fill:none;stroke:context-stroke;stroke-width:1;stroke-linecap:butt"
110+ d="M 3,-3 0,0 3,3"
111+ transform="rotate(180,0.125,0)"
112+ sodipodi:nodetypes="ccc"
113+ id="path1" />
114+ </marker>
115+ <marker
116+ style="overflow:visible"
117+ id="Triangle"
118+ refX="0"
119+ refY="0"
120+ orient="auto-start-reverse"
121+ inkscape:stockid="Triangle arrow"
122+ markerWidth="1"
123+ markerHeight="1"
124+ viewBox="0 0 1 1"
125+ inkscape:isstock="true"
126+ inkscape:collect="always"
127+ preserveAspectRatio="xMidYMid">
128+ <path
129+ transform="scale(0.5)"
130+ style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
131+ d="M 5.77,0 -2.88,5 V -5 Z"
132+ id="path135" />
133+ </marker>
134+ <marker
135+ style="overflow:visible"
136+ id="ArrowWideHeavy"
137+ refX="0"
138+ refY="0"
139+ orient="auto-start-reverse"
140+ inkscape:stockid="Wide, heavy arrow"
141+ markerWidth="1"
142+ markerHeight="1"
143+ viewBox="0 0 1 1"
144+ inkscape:isstock="true"
145+ inkscape:collect="always"
146+ preserveAspectRatio="xMidYMid">
147+ <path
148+ style="fill:context-stroke;fill-rule:evenodd;stroke:none"
149+ d="m 1,0 -3,3 h -2 l 3,-3 -3,-3 h 2 z"
150+ id="path71" />
151+ </marker>
152+ </defs>
153+ <g
154+ inkscape:label="Layer 1"
155+ inkscape:groupmode="layer"
156+ id="layer1"
157+ transform="translate(149.03517,55.110629)">
158+ <circle
159+ style="fill:#ffeeaa;fill-opacity:1;stroke:#1a1a1a;stroke-width:0.733436;stroke-dasharray:none;stroke-dashoffset:0"
160+ id="path98"
161+ cx="-35.757576"
162+ cy="-10.495151"
163+ r="44.24876" />
164+ <rect
165+ style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none"
166+ id="rect1"
167+ width="167.79619"
168+ height="24.831829"
169+ x="14.21942"
170+ y="57.862854" />
171+ <g
172+ id="g11"
173+ transform="translate(24.021362,-46.545299)">
174+ <rect
175+ style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
176+ id="rect2"
177+ width="131.09708"
178+ height="5.8163381"
179+ x="23.245802"
180+ y="107.16314" />
181+ <path
182+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
183+ d="m 37.884707,107.26337 v 5.53565"
184+ id="path2"
185+ sodipodi:nodetypes="cc" />
186+ <path
187+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
188+ d="m 52.466233,107.26337 v 5.53565"
189+ id="path3"
190+ sodipodi:nodetypes="cc" />
191+ <path
192+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
193+ d="m 67.047758,107.26337 v 5.53565"
194+ id="path4"
195+ sodipodi:nodetypes="cc" />
196+ <path
197+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
198+ d="m 81.629283,107.26337 v 5.53565"
199+ id="path5"
200+ sodipodi:nodetypes="cc" />
201+ <path
202+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
203+ d="m 96.210809,107.26337 v 5.53565"
204+ id="path6"
205+ sodipodi:nodetypes="cc" />
206+ <path
207+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
208+ d="m 110.79233,107.26337 v 5.53565"
209+ id="path7"
210+ sodipodi:nodetypes="cc" />
211+ <path
212+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
213+ d="m 125.37386,107.26337 v 5.53565"
214+ id="path8"
215+ sodipodi:nodetypes="cc" />
216+ <path
217+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
218+ d="m 139.95538,107.26337 v 5.53565"
219+ id="path9"
220+ sodipodi:nodetypes="cc" />
221+ <text
222+ xml:space="preserve"
223+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
224+ x="86.713425"
225+ y="110.1963"
226+ id="text11"><tspan
227+ sodipodi:role="line"
228+ id="tspan11"
229+ style="stroke:#006680;stroke-width:0.5"
230+ x="86.713425"
231+ y="110.1963">. . .</tspan></text>
232+ </g>
233+ <g
234+ id="g18"
235+ transform="translate(24.021362,-32.25779)">
236+ <rect
237+ style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
238+ id="rect11"
239+ width="131.09708"
240+ height="5.8163381"
241+ x="23.245802"
242+ y="107.16314" />
243+ <path
244+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
245+ d="m 37.884707,107.26337 v 5.53565"
246+ id="path11"
247+ sodipodi:nodetypes="cc" />
248+ <path
249+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
250+ d="m 52.466233,107.26337 v 5.53565"
251+ id="path12"
252+ sodipodi:nodetypes="cc" />
253+ <path
254+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
255+ d="m 67.047758,107.26337 v 5.53565"
256+ id="path13"
257+ sodipodi:nodetypes="cc" />
258+ <path
259+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
260+ d="m 81.629283,107.26337 v 5.53565"
261+ id="path14"
262+ sodipodi:nodetypes="cc" />
263+ <path
264+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
265+ d="m 96.210809,107.26337 v 5.53565"
266+ id="path15"
267+ sodipodi:nodetypes="cc" />
268+ <path
269+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
270+ d="m 110.79233,107.26337 v 5.53565"
271+ id="path16"
272+ sodipodi:nodetypes="cc" />
273+ <path
274+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
275+ d="m 125.37386,107.26337 v 5.53565"
276+ id="path17"
277+ sodipodi:nodetypes="cc" />
278+ <path
279+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
280+ d="m 139.95538,107.26337 v 5.53565"
281+ id="path18"
282+ sodipodi:nodetypes="cc" />
283+ <text
284+ xml:space="preserve"
285+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
286+ x="86.713425"
287+ y="110.1963"
288+ id="text18"><tspan
289+ sodipodi:role="line"
290+ id="tspan18"
291+ style="stroke:#006680;stroke-width:0.5"
292+ x="86.713425"
293+ y="110.1963">. . .</tspan></text>
294+ </g>
295+ <text
296+ xml:space="preserve"
297+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
298+ x="113.37768"
299+ y="68.463142"
300+ id="text19"><tspan
301+ sodipodi:role="line"
302+ id="tspan19"
303+ style="writing-mode:tb-rl;stroke-width:0.5"
304+ x="113.37768"
305+ y="68.463142">. . .</tspan></text>
306+ <text
307+ xml:space="preserve"
308+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
309+ x="25.005701"
310+ y="55.308445"
311+ id="text844-2-9"><tspan
312+ sodipodi:role="line"
313+ x="25.005701"
314+ y="55.308445"
315+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
316+ id="tspan2868">Pipe[0]</tspan></text>
317+ <text
318+ xml:space="preserve"
319+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
320+ x="-74.441521"
321+ y="63.075123"
322+ id="text844-2-9-4"><tspan
323+ sodipodi:role="line"
324+ x="-74.441521"
325+ y="63.075123"
326+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
327+ id="tspan2868-7">MQD</tspan></text>
328+ <text
329+ xml:space="preserve"
330+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
331+ x="30.264952"
332+ y="65.490654"
333+ id="text20"><tspan
334+ sodipodi:role="line"
335+ x="30.264952"
336+ y="65.490654"
337+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
338+ id="tspan20">Queue[0]</tspan></text>
339+ <text
340+ xml:space="preserve"
341+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
342+ x="30.264952"
343+ y="79.249001"
344+ id="text21"><tspan
345+ sodipodi:role="line"
346+ x="30.264952"
347+ y="79.249001"
348+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
349+ id="tspan21">Queue[n]</tspan></text>
350+ <text
351+ xml:space="preserve"
352+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
353+ x="30.264952"
354+ y="71.84066"
355+ id="text22"><tspan
356+ sodipodi:role="line"
357+ x="30.264952"
358+ y="71.84066"
359+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
360+ id="tspan22">...</tspan></text>
361+ <g
362+ id="g71"
363+ transform="translate(-23.283342)">
364+ <rect
365+ style="fill:#ffffff;fill-opacity:0;stroke:#000000;stroke-width:0.5;stroke-dasharray:0.5, 0.5;stroke-dashoffset:0"
366+ id="rect67"
367+ width="18.533583"
368+ height="114.96632"
369+ x="250.9435"
370+ y="54.754276"
371+ ry="6.0427966" />
372+ </g>
373+ <rect
374+ style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none;stroke-dashoffset:0"
375+ id="rect68"
376+ width="188.21231"
377+ height="139.5948"
378+ x="4.0113592"
379+ y="37.597778"
380+ ry="0" />
381+ <g
382+ id="g43"
383+ transform="translate(0,40.745853)">
384+ <rect
385+ style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none"
386+ id="rect22"
387+ width="167.79619"
388+ height="24.831829"
389+ x="14.21942"
390+ y="57.862854" />
391+ <g
392+ id="g30"
393+ transform="translate(24.021362,-46.545299)">
394+ <rect
395+ style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
396+ id="rect23"
397+ width="131.09708"
398+ height="5.8163381"
399+ x="23.245802"
400+ y="107.16314" />
401+ <path
402+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
403+ d="m 37.884707,107.26337 v 5.53565"
404+ id="path23"
405+ sodipodi:nodetypes="cc" />
406+ <path
407+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
408+ d="m 52.466233,107.26337 v 5.53565"
409+ id="path24"
410+ sodipodi:nodetypes="cc" />
411+ <path
412+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
413+ d="m 67.047758,107.26337 v 5.53565"
414+ id="path25"
415+ sodipodi:nodetypes="cc" />
416+ <path
417+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
418+ d="m 81.629283,107.26337 v 5.53565"
419+ id="path26"
420+ sodipodi:nodetypes="cc" />
421+ <path
422+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
423+ d="m 96.210809,107.26337 v 5.53565"
424+ id="path27"
425+ sodipodi:nodetypes="cc" />
426+ <path
427+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
428+ d="m 110.79233,107.26337 v 5.53565"
429+ id="path28"
430+ sodipodi:nodetypes="cc" />
431+ <path
432+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
433+ d="m 125.37386,107.26337 v 5.53565"
434+ id="path29"
435+ sodipodi:nodetypes="cc" />
436+ <path
437+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
438+ d="m 139.95538,107.26337 v 5.53565"
439+ id="path30"
440+ sodipodi:nodetypes="cc" />
441+ <text
442+ xml:space="preserve"
443+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
444+ x="86.713425"
445+ y="110.1963"
446+ id="text30"><tspan
447+ sodipodi:role="line"
448+ id="tspan30"
449+ style="stroke:#006680;stroke-width:0.5"
450+ x="86.713425"
451+ y="110.1963">. . .</tspan></text>
452+ </g>
453+ <g
454+ id="g38"
455+ transform="translate(24.021362,-32.25779)">
456+ <rect
457+ style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
458+ id="rect30"
459+ width="131.09708"
460+ height="5.8163381"
461+ x="23.245802"
462+ y="107.16314" />
463+ <path
464+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
465+ d="m 37.884707,107.26337 v 5.53565"
466+ id="path31"
467+ sodipodi:nodetypes="cc" />
468+ <path
469+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
470+ d="m 52.466233,107.26337 v 5.53565"
471+ id="path32"
472+ sodipodi:nodetypes="cc" />
473+ <path
474+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
475+ d="m 67.047758,107.26337 v 5.53565"
476+ id="path33"
477+ sodipodi:nodetypes="cc" />
478+ <path
479+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
480+ d="m 81.629283,107.26337 v 5.53565"
481+ id="path34"
482+ sodipodi:nodetypes="cc" />
483+ <path
484+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
485+ d="m 96.210809,107.26337 v 5.53565"
486+ id="path35"
487+ sodipodi:nodetypes="cc" />
488+ <path
489+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
490+ d="m 110.79233,107.26337 v 5.53565"
491+ id="path36"
492+ sodipodi:nodetypes="cc" />
493+ <path
494+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
495+ d="m 125.37386,107.26337 v 5.53565"
496+ id="path37"
497+ sodipodi:nodetypes="cc" />
498+ <path
499+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
500+ d="m 139.95538,107.26337 v 5.53565"
501+ id="path38"
502+ sodipodi:nodetypes="cc" />
503+ <text
504+ xml:space="preserve"
505+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
506+ x="86.713425"
507+ y="110.1963"
508+ id="text38"><tspan
509+ sodipodi:role="line"
510+ id="tspan38"
511+ style="stroke:#006680;stroke-width:0.5"
512+ x="86.713425"
513+ y="110.1963">. . .</tspan></text>
514+ </g>
515+ <text
516+ xml:space="preserve"
517+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
518+ x="113.37768"
519+ y="68.463142"
520+ id="text39"><tspan
521+ sodipodi:role="line"
522+ id="tspan39"
523+ style="writing-mode:tb-rl;stroke-width:0.5"
524+ x="113.37768"
525+ y="68.463142">. . .</tspan></text>
526+ <text
527+ xml:space="preserve"
528+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
529+ x="25.005701"
530+ y="55.308445"
531+ id="text40"><tspan
532+ sodipodi:role="line"
533+ x="25.005701"
534+ y="55.308445"
535+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
536+ id="tspan40">Pipe[1]</tspan></text>
537+ <text
538+ xml:space="preserve"
539+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
540+ x="30.264952"
541+ y="65.490654"
542+ id="text41"><tspan
543+ sodipodi:role="line"
544+ x="30.264952"
545+ y="65.490654"
546+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
547+ id="tspan41">Queue[0]</tspan></text>
548+ <text
549+ xml:space="preserve"
550+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
551+ x="30.264952"
552+ y="79.249001"
553+ id="text42"><tspan
554+ sodipodi:role="line"
555+ x="30.264952"
556+ y="79.249001"
557+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
558+ id="tspan42">Queue[n]</tspan></text>
559+ <text
560+ xml:space="preserve"
561+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
562+ x="30.264952"
563+ y="71.84066"
564+ id="text43"><tspan
565+ sodipodi:role="line"
566+ x="30.264952"
567+ y="71.84066"
568+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
569+ id="tspan43">...</tspan></text>
570+ </g>
571+ <g
572+ id="g64"
573+ transform="translate(0,85.195881)">
574+ <rect
575+ style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none"
576+ id="rect43"
577+ width="167.79619"
578+ height="24.831829"
579+ x="14.21942"
580+ y="57.862854" />
581+ <g
582+ id="g51"
583+ transform="translate(24.021362,-46.545299)">
584+ <rect
585+ style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
586+ id="rect44"
587+ width="131.09708"
588+ height="5.8163381"
589+ x="23.245802"
590+ y="107.16314" />
591+ <path
592+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
593+ d="m 37.884707,107.26337 v 5.53565"
594+ id="path44"
595+ sodipodi:nodetypes="cc" />
596+ <path
597+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
598+ d="m 52.466233,107.26337 v 5.53565"
599+ id="path45"
600+ sodipodi:nodetypes="cc" />
601+ <path
602+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
603+ d="m 67.047758,107.26337 v 5.53565"
604+ id="path46"
605+ sodipodi:nodetypes="cc" />
606+ <path
607+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
608+ d="m 81.629283,107.26337 v 5.53565"
609+ id="path47"
610+ sodipodi:nodetypes="cc" />
611+ <path
612+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
613+ d="m 96.210809,107.26337 v 5.53565"
614+ id="path48"
615+ sodipodi:nodetypes="cc" />
616+ <path
617+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
618+ d="m 110.79233,107.26337 v 5.53565"
619+ id="path49"
620+ sodipodi:nodetypes="cc" />
621+ <path
622+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
623+ d="m 125.37386,107.26337 v 5.53565"
624+ id="path50"
625+ sodipodi:nodetypes="cc" />
626+ <path
627+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
628+ d="m 139.95538,107.26337 v 5.53565"
629+ id="path51"
630+ sodipodi:nodetypes="cc" />
631+ <text
632+ xml:space="preserve"
633+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
634+ x="86.713425"
635+ y="110.1963"
636+ id="text51"><tspan
637+ sodipodi:role="line"
638+ id="tspan51"
639+ style="stroke:#006680;stroke-width:0.5"
640+ x="86.713425"
641+ y="110.1963">. . .</tspan></text>
642+ </g>
643+ <g
644+ id="g59"
645+ transform="translate(24.021362,-32.25779)">
646+ <rect
647+ style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
648+ id="rect51"
649+ width="131.09708"
650+ height="5.8163381"
651+ x="23.245802"
652+ y="107.16314" />
653+ <path
654+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
655+ d="m 37.884707,107.26337 v 5.53565"
656+ id="path52"
657+ sodipodi:nodetypes="cc" />
658+ <path
659+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
660+ d="m 52.466233,107.26337 v 5.53565"
661+ id="path53"
662+ sodipodi:nodetypes="cc" />
663+ <path
664+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
665+ d="m 67.047758,107.26337 v 5.53565"
666+ id="path54"
667+ sodipodi:nodetypes="cc" />
668+ <path
669+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
670+ d="m 81.629283,107.26337 v 5.53565"
671+ id="path55"
672+ sodipodi:nodetypes="cc" />
673+ <path
674+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
675+ d="m 96.210809,107.26337 v 5.53565"
676+ id="path56"
677+ sodipodi:nodetypes="cc" />
678+ <path
679+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
680+ d="m 110.79233,107.26337 v 5.53565"
681+ id="path57"
682+ sodipodi:nodetypes="cc" />
683+ <path
684+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
685+ d="m 125.37386,107.26337 v 5.53565"
686+ id="path58"
687+ sodipodi:nodetypes="cc" />
688+ <path
689+ style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
690+ d="m 139.95538,107.26337 v 5.53565"
691+ id="path59"
692+ sodipodi:nodetypes="cc" />
693+ <text
694+ xml:space="preserve"
695+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
696+ x="86.713425"
697+ y="110.1963"
698+ id="text59"><tspan
699+ sodipodi:role="line"
700+ id="tspan59"
701+ style="stroke:#006680;stroke-width:0.5"
702+ x="86.713425"
703+ y="110.1963">. . .</tspan></text>
704+ </g>
705+ <text
706+ xml:space="preserve"
707+ style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
708+ x="113.37768"
709+ y="68.463142"
710+ id="text60"><tspan
711+ sodipodi:role="line"
712+ id="tspan60"
713+ style="writing-mode:tb-rl;stroke-width:0.5"
714+ x="113.37768"
715+ y="68.463142">. . .</tspan></text>
716+ <text
717+ xml:space="preserve"
718+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
719+ x="25.005701"
720+ y="55.308445"
721+ id="text61"><tspan
722+ sodipodi:role="line"
723+ x="25.005701"
724+ y="55.308445"
725+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
726+ id="tspan61">Pipe[n]</tspan></text>
727+ <text
728+ xml:space="preserve"
729+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
730+ x="30.264952"
731+ y="65.490654"
732+ id="text62"><tspan
733+ sodipodi:role="line"
734+ x="30.264952"
735+ y="65.490654"
736+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
737+ id="tspan62">Queue[0]</tspan></text>
738+ <text
739+ xml:space="preserve"
740+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
741+ x="30.264952"
742+ y="79.249001"
743+ id="text63"><tspan
744+ sodipodi:role="line"
745+ x="30.264952"
746+ y="79.249001"
747+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
748+ id="tspan63">Queue[n]</tspan></text>
749+ <text
750+ xml:space="preserve"
751+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
752+ x="30.264952"
753+ y="71.84066"
754+ id="text64"><tspan
755+ sodipodi:role="line"
756+ x="30.264952"
757+ y="71.84066"
758+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
759+ id="tspan64">...</tspan></text>
760+ </g>
761+ <text
762+ xml:space="preserve"
763+ style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.76111px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:#000000;fill-opacity:0;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
764+ x="92.18071"
765+ y="128.21965"
766+ id="text65"><tspan
767+ sodipodi:role="line"
768+ id="tspan65"
769+ style="font-size:7.76111px;writing-mode:tb-rl;fill:#1a1a1a;stroke:#000000;stroke-width:0.5"
770+ x="92.18071"
771+ y="128.21965">...</tspan></text>
772+ <text
773+ xml:space="preserve"
774+ style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.7611px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none"
775+ x="198.72205"
776+ y="80.708267"
777+ id="text66"><tspan
778+ sodipodi:role="line"
779+ id="tspan66"
780+ style="stroke-width:0.5"
781+ x="198.72205"
782+ y="80.708267" /></text>
783+ <text
784+ xml:space="preserve"
785+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
786+ x="97.905846"
787+ y="44.725101"
788+ id="text68"><tspan
789+ sodipodi:role="line"
790+ x="97.905846"
791+ y="44.725101"
792+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
793+ id="tspan68">Hardware Block</tspan></text>
794+ <text
795+ xml:space="preserve"
796+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;writing-mode:vertical-lr;text-orientation:upright;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
797+ x="236.36934"
798+ y="112.10503"
799+ id="text68-7"><tspan
800+ sodipodi:role="line"
801+ x="236.36934"
802+ y="112.10503"
803+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;writing-mode:vertical-lr;text-orientation:upright;text-anchor:middle;stroke-width:0.0690111"
804+ id="tspan68-4">EXECUTION</tspan></text>
805+ <rect
806+ style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.940575;stroke-dasharray:7.5246, 0.940575;stroke-dashoffset:0"
807+ id="rect68-1"
808+ width="68.749969"
809+ height="141.2751"
810+ x="-129.49162"
811+ y="37.881134"
812+ ry="0" />
813+ <text
814+ xml:space="preserve"
815+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
816+ x="-114.4223"
817+ y="44.966106"
818+ id="text68-2"><tspan
819+ sodipodi:role="line"
820+ x="-114.4223"
821+ y="44.966106"
822+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
823+ id="tspan68-9">Memory</tspan></text>
824+ <text
825+ xml:space="preserve"
826+ style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12.7px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-orientation:upright;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none;stroke-dashoffset:0"
827+ x="212.6013"
828+ y="64.823341"
829+ id="text69"><tspan
830+ sodipodi:role="line"
831+ id="tspan69"
832+ style="font-size:12.7px;stroke-width:0.5"
833+ x="212.6013"
834+ y="64.823341" /></text>
835+ <g
836+ id="g72">
837+ <path
838+ style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
839+ d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
840+ id="text70"
841+ aria-label="}"
842+ sodipodi:nodetypes="ccccsccccccccscscscscc" />
843+ <path
844+ style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)"
845+ d="M 190.40199,70.278769 H 224.6663"
846+ id="path70" />
847+ </g>
848+ <path
849+ style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
850+ d="m -88.44114,74.337891 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 V 62.57347 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 v -9.376826 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
851+ id="text70-1"
852+ aria-label="}"
853+ sodipodi:nodetypes="ccccsccccccccscscscscc" />
854+ <text
855+ xml:space="preserve"
856+ style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.9333px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-orientation:upright;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0"
857+ x="200.88817"
858+ y="38.990276"
859+ id="text72"><tspan
860+ sodipodi:role="line"
861+ id="tspan72"
862+ style="stroke-width:1.25"
863+ x="200.88817"
864+ y="38.990276" /></text>
865+ <text
866+ xml:space="preserve"
867+ style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
868+ x="205.12828"
869+ y="60.090775"
870+ id="text73"><tspan
871+ sodipodi:role="line"
872+ x="205.12828"
873+ y="60.090775"
874+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
875+ id="tspan73">e.g.,:</tspan><tspan
876+ sodipodi:role="line"
877+ x="205.12828"
878+ y="68.028275"
879+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
880+ id="tspan74">queue[0]</tspan></text>
881+ <g
882+ id="g75"
883+ transform="translate(0,40.745853)">
884+ <path
885+ style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
886+ d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
887+ id="path74"
888+ aria-label="}"
889+ sodipodi:nodetypes="ccccsccccccccscscscscc" />
890+ <path
891+ style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)"
892+ d="M 190.40199,70.278769 H 224.6663"
893+ id="path75" />
894+ </g>
895+ <text
896+ xml:space="preserve"
897+ style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
898+ x="205.12828"
899+ y="100.83664"
900+ id="text76"><tspan
901+ sodipodi:role="line"
902+ x="205.12828"
903+ y="100.83664"
904+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
905+ id="tspan75">e.g.,:</tspan><tspan
906+ sodipodi:role="line"
907+ x="205.12828"
908+ y="108.77414"
909+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
910+ id="tspan76">queue[4]</tspan></text>
911+ <g
912+ id="g77"
913+ transform="translate(0,85.725048)">
914+ <path
915+ style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
916+ d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
917+ id="path76"
918+ aria-label="}"
919+ sodipodi:nodetypes="ccccsccccccccscscscscc" />
920+ <path
921+ style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)"
922+ d="M 190.40199,70.278769 H 224.6663"
923+ id="path77" />
924+ </g>
925+ <text
926+ xml:space="preserve"
927+ style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
928+ x="205.12828"
929+ y="145.81558"
930+ id="text78"><tspan
931+ sodipodi:role="line"
932+ x="205.12828"
933+ y="145.81558"
934+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
935+ id="tspan77">e.g.,:</tspan><tspan
936+ sodipodi:role="line"
937+ x="205.12828"
938+ y="153.75308"
939+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
940+ id="tspan78">queue[n]</tspan></text>
941+ <g
942+ id="g81">
943+ <text
944+ xml:space="preserve"
945+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
946+ x="-18.502264"
947+ y="65.642387"
948+ id="text79"><tspan
949+ sodipodi:role="line"
950+ x="-18.502264"
951+ y="65.642387"
952+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
953+ id="tspan79">HQD</tspan></text>
954+ <ellipse
955+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
956+ id="path79"
957+ cx="-18.407015"
958+ cy="63.2188"
959+ rx="13.317666"
960+ ry="4.4124799" />
961+ <path
962+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
963+ d="M -4.6361224,63.2188 H 10.606988"
964+ id="path80" />
965+ </g>
966+ <g
967+ id="g82"
968+ transform="translate(0,14.287503)">
969+ <text
970+ xml:space="preserve"
971+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
972+ x="-18.502264"
973+ y="65.642387"
974+ id="text81"><tspan
975+ sodipodi:role="line"
976+ x="-18.502264"
977+ y="65.642387"
978+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
979+ id="tspan81">HQD</tspan></text>
980+ <ellipse
981+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
982+ id="ellipse81"
983+ cx="-18.407015"
984+ cy="63.2188"
985+ rx="13.317666"
986+ ry="4.4124799" />
987+ <path
988+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
989+ d="M -4.6361224,63.2188 H 10.606988"
990+ id="path81" />
991+ </g>
992+ <g
993+ id="g83"
994+ transform="translate(0,40.745853)">
995+ <text
996+ xml:space="preserve"
997+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
998+ x="-18.502264"
999+ y="65.642387"
1000+ id="text82"><tspan
1001+ sodipodi:role="line"
1002+ x="-18.502264"
1003+ y="65.642387"
1004+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1005+ id="tspan82">HQD</tspan></text>
1006+ <ellipse
1007+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1008+ id="ellipse82"
1009+ cx="-18.407015"
1010+ cy="63.2188"
1011+ rx="13.317666"
1012+ ry="4.4124799" />
1013+ <path
1014+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1015+ d="M -4.6361224,63.2188 H 10.606988"
1016+ id="path82" />
1017+ </g>
1018+ <g
1019+ id="g84"
1020+ transform="translate(0,55.033362)">
1021+ <text
1022+ xml:space="preserve"
1023+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1024+ x="-18.502264"
1025+ y="65.642387"
1026+ id="text83"><tspan
1027+ sodipodi:role="line"
1028+ x="-18.502264"
1029+ y="65.642387"
1030+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1031+ id="tspan83">HQD</tspan></text>
1032+ <ellipse
1033+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1034+ id="ellipse83"
1035+ cx="-18.407015"
1036+ cy="63.2188"
1037+ rx="13.317666"
1038+ ry="4.4124799" />
1039+ <path
1040+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1041+ d="M -4.6361224,63.2188 H 10.606988"
1042+ id="path83" />
1043+ </g>
1044+ <g
1045+ id="g85"
1046+ transform="translate(0,85.195881)">
1047+ <text
1048+ xml:space="preserve"
1049+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1050+ x="-18.502264"
1051+ y="65.642387"
1052+ id="text84"><tspan
1053+ sodipodi:role="line"
1054+ x="-18.502264"
1055+ y="65.642387"
1056+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1057+ id="tspan84">HQD</tspan></text>
1058+ <ellipse
1059+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1060+ id="ellipse84"
1061+ cx="-18.407015"
1062+ cy="63.2188"
1063+ rx="13.317666"
1064+ ry="4.4124799" />
1065+ <path
1066+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1067+ d="M -4.6361224,63.2188 H 10.606988"
1068+ id="path84" />
1069+ </g>
1070+ <g
1071+ id="g86"
1072+ transform="translate(0,99.48339)">
1073+ <text
1074+ xml:space="preserve"
1075+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1076+ x="-18.502264"
1077+ y="65.642387"
1078+ id="text85"><tspan
1079+ sodipodi:role="line"
1080+ x="-18.502264"
1081+ y="65.642387"
1082+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1083+ id="tspan85">HQD</tspan></text>
1084+ <ellipse
1085+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1086+ id="ellipse85"
1087+ cx="-18.407015"
1088+ cy="63.2188"
1089+ rx="13.317666"
1090+ ry="4.4124799" />
1091+ <path
1092+ style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1093+ d="M -4.6361224,63.2188 H 10.606988"
1094+ id="path85" />
1095+ </g>
1096+ <text
1097+ xml:space="preserve"
1098+ style="font-style:normal;font-weight:normal;font-size:5.64444px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1099+ x="-35.2131"
1100+ y="54.673237"
1101+ id="text86"><tspan
1102+ sodipodi:role="line"
1103+ x="-35.2131"
1104+ y="54.673237"
1105+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1106+ id="tspan86">Registers</tspan></text>
1107+ <path
1108+ style="fill:#800000;fill-opacity:0;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:1, 1;stroke-dashoffset:0;marker-end:url(#ArrowWide)"
1109+ d="m -45.247972,57.442462 v 5.888987 h 11.344412"
1110+ id="path86"
1111+ sodipodi:nodetypes="ccc" />
1112+ <text
1113+ xml:space="preserve"
1114+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1115+ x="-74.441521"
1116+ y="82.441582"
1117+ id="text87"><tspan
1118+ sodipodi:role="line"
1119+ x="-74.441521"
1120+ y="82.441582"
1121+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1122+ id="tspan87">MQD</tspan></text>
1123+ <text
1124+ xml:space="preserve"
1125+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1126+ x="-74.441521"
1127+ y="99.240776"
1128+ id="text88"><tspan
1129+ sodipodi:role="line"
1130+ x="-74.441521"
1131+ y="99.240776"
1132+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1133+ id="tspan88">MQD</tspan></text>
1134+ <text
1135+ xml:space="preserve"
1136+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1137+ x="-74.441521"
1138+ y="116.03998"
1139+ id="text89"><tspan
1140+ sodipodi:role="line"
1141+ x="-74.441521"
1142+ y="116.03998"
1143+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1144+ id="tspan89">MQD</tspan></text>
1145+ <text
1146+ xml:space="preserve"
1147+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1148+ x="-74.441521"
1149+ y="132.83917"
1150+ id="text90"><tspan
1151+ sodipodi:role="line"
1152+ x="-74.441521"
1153+ y="132.83917"
1154+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1155+ id="tspan90">MQD</tspan></text>
1156+ <text
1157+ xml:space="preserve"
1158+ style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1159+ x="-74.441521"
1160+ y="168.3002"
1161+ id="text91"><tspan
1162+ sodipodi:role="line"
1163+ x="-74.441521"
1164+ y="168.3002"
1165+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1166+ id="tspan91">MQD</tspan></text>
1167+ <text
1168+ xml:space="preserve"
1169+ style="font-style:normal;font-weight:normal;font-size:10.5833px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1170+ x="-74.441521"
1171+ y="148.1461"
1172+ id="text92"><tspan
1173+ sodipodi:role="line"
1174+ x="-74.441521"
1175+ y="148.1461"
1176+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:10.5833px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;writing-mode:tb-rl;text-anchor:middle;stroke-width:0.0690111"
1177+ id="tspan92">...</tspan></text>
1178+ <g
1179+ id="g97"
1180+ transform="translate(-5.8208336)">
1181+ <rect
1182+ style="fill:#aaffcc;fill-opacity:1;stroke:#1a1a1a;stroke-width:0.5;stroke-dasharray:none;stroke-dashoffset:0"
1183+ id="rect96"
1184+ width="58.726093"
1185+ height="27.598055"
1186+ x="-142.96434"
1187+ y="46.852512" />
1188+ <text
1189+ xml:space="preserve"
1190+ style="font-style:normal;font-weight:normal;font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:start;letter-spacing:0px;word-spacing:0px;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1191+ x="-139.74622"
1192+ y="52.62756"
1193+ id="text93"><tspan
1194+ sodipodi:role="line"
1195+ x="-139.74622"
1196+ y="52.62756"
1197+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1198+ id="tspan93">HQD Registers</tspan><tspan
1199+ sodipodi:role="line"
1200+ x="-139.74622"
1201+ y="57.91922"
1202+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1203+ id="tspan95">Queue Address in the GPU</tspan><tspan
1204+ sodipodi:role="line"
1205+ x="-139.74622"
1206+ y="63.210884"
1207+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1208+ id="tspan96">Doorbell</tspan><tspan
1209+ sodipodi:role="line"
1210+ x="-139.74622"
1211+ y="68.502548"
1212+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1213+ id="tspan97">...</tspan><tspan
1214+ sodipodi:role="line"
1215+ x="-139.74622"
1216+ y="73.794212"
1217+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1218+ id="tspan94" /></text>
1219+ </g>
1220+ <text
1221+ xml:space="preserve"
1222+ style="font-style:normal;font-weight:normal;font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:start;letter-spacing:0px;word-spacing:0px;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1223+ x="-61.293022"
1224+ y="-19.380915"
1225+ id="text93-1"><tspan
1226+ sodipodi:role="line"
1227+ x="-61.293022"
1228+ y="-19.380915"
1229+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1230+ id="tspan104">SWITCH QUEUE:</tspan><tspan
1231+ sodipodi:role="line"
1232+ x="-61.293022"
1233+ y="-14.089252"
1234+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1235+ id="tspan101">WAIT FOR HQD_ACTIVE = 0</tspan><tspan
1236+ sodipodi:role="line"
1237+ x="-61.293022"
1238+ y="-8.7975903"
1239+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1240+ id="tspan105">SAVE QUEUE STATE TO THE MQD</tspan><tspan
1241+ sodipodi:role="line"
1242+ x="-61.293022"
1243+ y="-3.505928"
1244+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1245+ id="tspan102">COPY NEW MQD STATE</tspan><tspan
1246+ sodipodi:role="line"
1247+ x="-61.293022"
1248+ y="1.7857342"
1249+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1250+ id="tspan103">SET HQD_ACTIVE = 1</tspan></text>
1251+ <circle
1252+ style="fill:#ffeeaa;fill-opacity:1;stroke:#1a1a1a;stroke-width:1.88976;stroke-dasharray:none;stroke-dashoffset:0"
1253+ id="path97"
1254+ cx="0"
1255+ cy="0"
1256+ r="0"
1257+ transform="matrix(0.26458333,0,0,0.26458333,-149.03517,37.347779)" />
1258+ <text
1259+ xml:space="preserve"
1260+ style="font-style:normal;font-weight:normal;font-size:5.64444px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1261+ x="-36.659206"
1262+ y="-44.828983"
1263+ id="text106"><tspan
1264+ sodipodi:role="line"
1265+ x="-36.659206"
1266+ y="-44.828983"
1267+ style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1268+ id="tspan106">Firmware</tspan></text>
1269+ <path
1270+ style="fill:none;fill-opacity:1;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-start:url(#marker109);marker-end:url(#ArrowTriangleStylized)"
1271+ d="M -84.242601,-9.1838245 H -98.041629 V 33.17598"
1272+ id="path106" />
1273+ <path
1274+ style="fill:none;fill-opacity:1;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Dot)"
1275+ d="M 9.9745536,-9.3442784 H 29.549918 V 37.170287"
1276+ id="path107"
1277+ sodipodi:nodetypes="ccc" />
1278+ </g>
1279+</svg>
drivers/dma-buf/dma-fence-unwrap.c+37−14
@@ -79,6 +79,41 @@ static int fence_cmp(const void *_a, const void *_b) · 78 unmodified lines
7979 return 0;
8080 }
8181
82+/**
83+ * dma_fence_dedup_array - Sort and deduplicate an array of dma_fence pointers
84+ * @fences: Array of dma_fence pointers to be deduplicated
85+ * @num_fences: Number of entries in the @fences array
86+ *
87+ * Sorts the input array by context, then removes duplicate
88+ * fences with the same context, keeping only the most recent one.
89+ *
90+ * The array is modified in-place and unreferenced duplicate fences are released
91+ * via dma_fence_put(). The function returns the new number of fences after
92+ * deduplication.
93+ *
94+ * Return: Number of unique fences remaining in the array.
95+ */
96+int dma_fence_dedup_array(struct dma_fence **fences, int num_fences)
97+{
98+ int i, j;
99+
100+ sort(fences, num_fences, sizeof(*fences), fence_cmp, NULL);
101+
102+ /*
103+ * Only keep the most recent fence for each context.
104+ */
105+ j = 0;
106+ for (i = 1; i < num_fences; i++) {
107+ if (fences[i]->context == fences[j]->context)
108+ dma_fence_put(fences[i]);
109+ else
110+ fences[++j] = fences[i];
111+ }
112+
113+ return ++j;
114+}
115+EXPORT_SYMBOL_GPL(dma_fence_dedup_array);
116+
82117 /* Implementation for the dma_fence_merge() marco, don't use directly */
83118 struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences,
84119 struct dma_fence **fences,
@@ -87,7 +122,7 @@ struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, · 2 unmodified lines
87122 struct dma_fence *tmp, *unsignaled = NULL, **array;
88123 struct dma_fence_array *result;
89124 ktime_t timestamp;
90− int i, j, count;
125+ int i, count;
91126
92127 count = 0;
93128 timestamp = ns_to_ktime(0);
@@ -141,19 +176,7 @@ struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, · 47 unmodified lines
141176 if (count == 0 || count == 1)
142177 goto return_fastpath;
143178
144− sort(array, count, sizeof(*array), fence_cmp, NULL);
145−
146− /*
147− * Only keep the most recent fence for each context.
148− */
149− j = 0;
150− for (i = 1; i < count; i++) {
151− if (array[i]->context == array[j]->context)
152− dma_fence_put(array[i]);
153− else
154− array[++j] = array[i];
155− }
156− count = ++j;
179+ count = dma_fence_dedup_array(array, count);
157180
158181 if (count > 1) {
159182 result = dma_fence_array_create(count, array,
drivers/gpu/drm/amd/amdgpu/Makefile+7−2
@@ -66,7 +66,7 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o amdgpu_kms.o \ · 65 unmodified lines
6666 amdgpu_fw_attestation.o amdgpu_securedisplay.o \
6767 amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o \
6868 amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_aca.o amdgpu_dev_coredump.o \
69− amdgpu_cper.o
69+ amdgpu_cper.o amdgpu_userq_fence.o amdgpu_eviction_fence.o
7070
7171 amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o
7272
@@ -174,7 +174,10 @@ amdgpu-y += \ · 101 unmodified lines
174174 amdgpu-y += \
175175 amdgpu_mes.o \
176176 mes_v11_0.o \
177− mes_v12_0.o
177+ mes_v12_0.o \
178+
179+# add GFX userqueue support
180+amdgpu-y += mes_userqueue.o
178181
179182 # add UVD block
180183 amdgpu-y += \
@@ -253,6 +256,8 @@ amdgpu-y += \ · 72 unmodified lines
253256 # add amdkfd interfaces
254257 amdgpu-y += amdgpu_amdkfd.o
255258
259+# add gfx usermode queue
260+amdgpu-y += amdgpu_userq.o
256261
257262 ifneq ($(CONFIG_HSA_AMD),)
258263 AMDKFD_PATH := ../amdkfd
drivers/gpu/drm/amd/amdgpu/amdgpu.h+98−6
@@ -113,6 +113,8 @@ · 112 unmodified lines
113113 #include "amdgpu_xcp.h"
114114 #include "amdgpu_seq64.h"
115115 #include "amdgpu_reg_state.h"
116+#include "amdgpu_userq.h"
117+#include "amdgpu_eviction_fence.h"
116118 #if defined(CONFIG_DRM_AMD_ISP)
117119 #include "amdgpu_isp.h"
118120 #endif
@@ -228,7 +230,7 @@ extern int amdgpu_force_asic_type; · 109 unmodified lines
228230 extern int amdgpu_smartshift_bias;
229231 extern int amdgpu_use_xgmi_p2p;
230232 extern int amdgpu_mtype_local;
231−extern bool enforce_isolation;
233+extern int amdgpu_enforce_isolation;
232234 #ifdef CONFIG_HSA_AMD
233235 extern int sched_policy;
234236 extern bool debug_evictions;
@@ -266,8 +268,10 @@ extern int amdgpu_umsch_mm_fwlog; · 31 unmodified lines
266268
267269 extern int amdgpu_user_partt_mode;
268270 extern int amdgpu_agp;
271+extern int amdgpu_rebar;
269272
270273 extern int amdgpu_wbrf;
274+extern int amdgpu_user_queue;
271275
272276 #define AMDGPU_VM_MAX_NUM_CTX 4096
273277 #define AMDGPU_SG_THRESHOLD (256*1024*1024)
@@ -488,7 +492,6 @@ struct amdgpu_flip_work { · 214 unmodified lines
488492 bool async;
489493 };
490494
491−
492495 /*
493496 * file private structure
494497 */
@@ -501,6 +504,11 @@ struct amdgpu_fpriv { · 6 unmodified lines
501504 struct mutex bo_list_lock;
502505 struct idr bo_list_handles;
503506 struct amdgpu_ctx_mgr ctx_mgr;
507+ struct amdgpu_userq_mgr userq_mgr;
508+
509+ /* Eviction fence infra */
510+ struct amdgpu_eviction_fence_mgr evf_mgr;
511+
504512 /** GPU partition selection */
505513 uint32_t xcp_id;
506514 };
@@ -512,12 +520,62 @@ int amdgpu_file_to_fpriv(struct file *filp, struct amdgpu_fpriv **fpriv); · 5 unmodified lines
512520 */
513521 #define AMDGPU_MAX_WB 1024 /* Reserve at most 1024 WB slots for amdgpu-owned rings. */
514522
523+/**
524+ * amdgpu_wb - This struct is used for small GPU memory allocation.
525+ *
526+ * This struct is used to allocate a small amount of GPU memory that can be
527+ * used to shadow certain states into the memory. This is especially useful for
528+ * providing easy CPU access to some states without requiring register access
529+ * (e.g., if some block is power gated, reading register may be problematic).
530+ *
531+ * Note: the term writeback was initially used because many of the amdgpu
532+ * components had some level of writeback memory, and this struct initially
533+ * described those components.
534+ */
515535 struct amdgpu_wb {
536+
537+ /**
538+ * @wb_obj:
539+ *
540+ * Buffer Object used for the writeback memory.
541+ */
516542 struct amdgpu_bo *wb_obj;
543+
544+ /**
545+ * @wb:
546+ *
547+ * Pointer to the first writeback slot. In terms of CPU address
548+ * this value can be accessed directly by using the offset as an index.
549+ * For the GPU address, it is necessary to use gpu_addr and the offset.
550+ */
517551 volatile uint32_t *wb;
552+
553+ /**
554+ * @gpu_addr:
555+ *
556+ * Writeback base address in the GPU.
557+ */
518558 uint64_t gpu_addr;
519− u32 num_wb; /* Number of wb slots actually reserved for amdgpu. */
559+
560+ /**
561+ * @num_wb:
562+ *
563+ * Number of writeback slots reserved for amdgpu.
564+ */
565+ u32 num_wb;
566+
567+ /**
568+ * @used:
569+ *
570+ * Track the writeback slot already used.
571+ */
520572 unsigned long used[DIV_ROUND_UP(AMDGPU_MAX_WB, BITS_PER_LONG)];
573+
574+ /**
575+ * @lock:
576+ *
577+ * Protects read and write of the used field array.
578+ */
521579 spinlock_t lock;
522580 };
523581
@@ -551,6 +609,7 @@ struct amdgpu_allowed_register_entry { · 27 unmodified lines
551609 * are reset depends on the ASIC. Notably doesn't reset IPs
552610 * shared with the CPU on APUs or the memory controllers (so
553611 * VRAM is not lost). Not available on all ASICs.
612+ * @AMD_RESET_LINK: Triggers SW-UP link reset on other GPUs
554613 * @AMD_RESET_BACO: BACO (Bus Alive, Chip Off) method powers off and on the card
555614 * but without powering off the PCI bus. Suitable only for
556615 * discrete GPUs.
@@ -568,6 +627,7 @@ enum amd_reset_method { · 11 unmodified lines
568627 AMD_RESET_METHOD_MODE0,
569628 AMD_RESET_METHOD_MODE1,
570629 AMD_RESET_METHOD_MODE2,
630+ AMD_RESET_METHOD_LINK,
571631 AMD_RESET_METHOD_BACO,
572632 AMD_RESET_METHOD_PCI,
573633 AMD_RESET_METHOD_ON_INIT,
@@ -821,6 +881,11 @@ struct amdgpu_mqd_prop { · 247 unmodified lines
821881 uint32_t hqd_queue_priority;
822882 bool allow_tunneling;
823883 bool hqd_active;
884+ uint64_t shadow_addr;
885+ uint64_t gds_bkup_addr;
886+ uint64_t csa_addr;
887+ uint64_t fence_address;
888+ bool tmz_queue;
824889 };
825890
826891 struct amdgpu_mqd {
@@ -829,6 +894,12 @@ struct amdgpu_mqd { · 2 unmodified lines
829894 struct amdgpu_mqd_prop *p);
830895 };
831896
897+struct amdgpu_pcie_reset_ctx {
898+ bool in_link_reset;
899+ bool occurs_dpc;
900+ bool audio_suspended;
901+};
902+
832903 /*
833904 * Custom Init levels could be defined for different situations where a full
834905 * initialization of all hardware blocks are not expected. Sample cases are
@@ -853,6 +924,14 @@ struct amdgpu_init_level { · 18 unmodified lines
853924 struct amdgpu_reset_domain;
854925 struct amdgpu_fru_info;
855926
927+enum amdgpu_enforce_isolation_mode {
928+ AMDGPU_ENFORCE_ISOLATION_DISABLE = 0,
929+ AMDGPU_ENFORCE_ISOLATION_ENABLE = 1,
930+ AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY = 2,
931+ AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER = 3,
932+};
933+
934+
856935 /*
857936 * Non-zero (true) if the GPU has VRAM. Zero (false) otherwise.
858937 */
@@ -1081,6 +1160,13 @@ struct amdgpu_device { · 222 unmodified lines
10811160 bool enable_uni_mes;
10821161 struct amdgpu_mes mes;
10831162 struct amdgpu_mqd mqds[AMDGPU_HW_IP_NUM];
1163+ const struct amdgpu_userq_funcs *userq_funcs[AMDGPU_HW_IP_NUM];
1164+
1165+ /* xarray used to retrieve the user queue fence driver reference
1166+ * in the EOP interrupt handler to signal the particular user
1167+ * queue fence.
1168+ */
1169+ struct xarray userq_xa;
10841170
10851171 /* df */
10861172 struct amdgpu_df df;
@@ -1160,6 +1246,8 @@ struct amdgpu_device { · 73 unmodified lines
11601246 struct pci_saved_state *pci_state;
11611247 pci_channel_state_t pci_channel_state;
11621248
1249+ struct amdgpu_pcie_reset_ctx pcie_reset_ctx;
1250+
11631251 /* Track auto wait count on s_barrier settings */
11641252 bool barrier_has_auto_waitcnt;
11651253
@@ -1193,10 +1281,11 @@ struct amdgpu_device { · 27 unmodified lines
11931281 bool debug_enable_ras_aca;
11941282 bool debug_exp_resets;
11951283 bool debug_disable_gpu_ring_reset;
1284+ bool debug_vm_userptr;
11961285
11971286 /* Protection for the following isolation structure */
11981287 struct mutex enforce_isolation_mutex;
1199− bool enforce_isolation[MAX_XCP];
1288+ enum amdgpu_enforce_isolation_mode enforce_isolation[MAX_XCP];
12001289 struct amdgpu_isolation {
12011290 void *owner;
12021291 struct dma_fence *spearhead;
@@ -1210,6 +1299,10 @@ struct amdgpu_device { · 7 unmodified lines
12101299 * in KFD: VRAM or GTT.
12111300 */
12121301 bool apu_prefer_gtt;
1302+
1303+ struct list_head userq_mgr_list;
1304+ struct mutex userq_mutex;
1305+ bool userq_halt_for_enforce_isolation;
12131306 };
12141307
12151308 static inline uint32_t amdgpu_ip_version(const struct amdgpu_device *adev,
@@ -1464,6 +1557,7 @@ void amdgpu_device_program_register_sequence(struct amdgpu_device *adev, · 248 unmodified lines
14641557 const u32 array_size);
14651558
14661559 int amdgpu_device_mode1_reset(struct amdgpu_device *adev);
1560+int amdgpu_device_link_reset(struct amdgpu_device *adev);
14671561 bool amdgpu_device_supports_atpx(struct drm_device *dev);
14681562 bool amdgpu_device_supports_px(struct drm_device *dev);
14691563 bool amdgpu_device_supports_boco(struct drm_device *dev);
@@ -1614,11 +1708,9 @@ static inline void amdgpu_acpi_get_backlight_caps(struct amdgpu_dm_backlight_cap · 144 unmodified lines
16141708 #if defined(CONFIG_ACPI) && defined(CONFIG_SUSPEND)
16151709 bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev);
16161710 bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev);
1617−void amdgpu_choose_low_power_state(struct amdgpu_device *adev);
16181711 #else
16191712 static inline bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) { return false; }
16201713 static inline bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev) { return false; }
1621−static inline void amdgpu_choose_low_power_state(struct amdgpu_device *adev) { }
16221714 #endif
16231715
16241716 void amdgpu_register_gpu_instance(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_aca.c+3−0
@@ -120,6 +120,9 @@ static void aca_smu_bank_dump(struct amdgpu_device *adev, int idx, int total, st · 119 unmodified lines
120120 for (i = 0; i < ARRAY_SIZE(aca_regs); i++)
121121 RAS_EVENT_LOG(adev, event_id, HW_ERR "ACA[%02d/%02d].%s=0x%016llx\n",
122122 idx + 1, total, aca_regs[i].name, bank->regs[aca_regs[i].reg_idx]);
123+
124+ if (ACA_REG__STATUS__SCRUB(bank->regs[ACA_REG_IDX_STATUS]))
125+ RAS_EVENT_LOG(adev, event_id, HW_ERR "hardware error logged by the scrubber\n");
123126 }
124127
125128 static int aca_smu_get_valid_aca_banks(struct amdgpu_device *adev, enum aca_smu_type type,
drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c+0−18
@@ -1533,22 +1533,4 @@ bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) · 1532 unmodified lines
15331533 #endif /* CONFIG_AMD_PMC */
15341534 }
15351535
1536−/**
1537− * amdgpu_choose_low_power_state
1538− *
1539− * @adev: amdgpu_device_pointer
1540− *
1541− * Choose the target low power state for the GPU
1542− */
1543−void amdgpu_choose_low_power_state(struct amdgpu_device *adev)
1544−{
1545− if (adev->in_runpm)
1546− return;
1547−
1548− if (amdgpu_acpi_is_s0ix_active(adev))
1549− adev->in_s0ix = true;
1550− else if (amdgpu_acpi_is_s3_active(adev))
1551− adev->in_s3 = true;
1552−}
1553−
15541536 #endif /* CONFIG_SUSPEND */
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c+12−0
@@ -2559,6 +2559,18 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info, · 2558 unmodified lines
25592559 if (ret != -EFAULT)
25602560 return ret;
25612561
2562+ /* If applications unmap memory before destroying the userptr
2563+ * from the KFD, trigger a segmentation fault in VM debug mode.
2564+ */
2565+ if (amdgpu_ttm_adev(bo->tbo.bdev)->debug_vm_userptr) {
2566+ pr_err("Pid %d unmapped memory before destroying userptr at GPU addr 0x%llx\n",
2567+ pid_nr(process_info->pid), mem->va);
2568+
2569+ // Send GPU VM fault to user space
2570+ kfd_signal_vm_fault_event_with_userptr(kfd_lookup_process_by_pid(process_info->pid),
2571+ mem->va);
2572+ }
2573+
25622574 ret = 0;
25632575 }
25642576
drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c+24−85
@@ -252,83 +252,22 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 251 unmodified lines
252252
253253 if (!adev->pm.fw) {
254254 switch (adev->asic_type) {
255− case CHIP_TAHITI:
256− strcpy(fw_name, "radeon/tahiti_smc.bin");
257− break;
258− case CHIP_PITCAIRN:
259− if ((adev->pdev->revision == 0x81) &&
260− ((adev->pdev->device == 0x6810) ||
261− (adev->pdev->device == 0x6811))) {
262− info->is_kicker = true;
263− strcpy(fw_name, "radeon/pitcairn_k_smc.bin");
264− } else {
265− strcpy(fw_name, "radeon/pitcairn_smc.bin");
266− }
267− break;
268− case CHIP_VERDE:
269− if (((adev->pdev->device == 0x6820) &&
270− ((adev->pdev->revision == 0x81) ||
271− (adev->pdev->revision == 0x83))) ||
272− ((adev->pdev->device == 0x6821) &&
273− ((adev->pdev->revision == 0x83) ||
274− (adev->pdev->revision == 0x87))) ||
275− ((adev->pdev->revision == 0x87) &&
276− ((adev->pdev->device == 0x6823) ||
277− (adev->pdev->device == 0x682b)))) {
278− info->is_kicker = true;
279− strcpy(fw_name, "radeon/verde_k_smc.bin");
280− } else {
281− strcpy(fw_name, "radeon/verde_smc.bin");
282− }
283− break;
284− case CHIP_OLAND:
285− if (((adev->pdev->revision == 0x81) &&
286− ((adev->pdev->device == 0x6600) ||
287− (adev->pdev->device == 0x6604) ||
288− (adev->pdev->device == 0x6605) ||
289− (adev->pdev->device == 0x6610))) ||
290− ((adev->pdev->revision == 0x83) &&
291− (adev->pdev->device == 0x6610))) {
292− info->is_kicker = true;
293− strcpy(fw_name, "radeon/oland_k_smc.bin");
294− } else {
295− strcpy(fw_name, "radeon/oland_smc.bin");
296− }
297− break;
298− case CHIP_HAINAN:
299− if (((adev->pdev->revision == 0x81) &&
300− (adev->pdev->device == 0x6660)) ||
301− ((adev->pdev->revision == 0x83) &&
302− ((adev->pdev->device == 0x6660) ||
303− (adev->pdev->device == 0x6663) ||
304− (adev->pdev->device == 0x6665) ||
305− (adev->pdev->device == 0x6667)))) {
306− info->is_kicker = true;
307− strcpy(fw_name, "radeon/hainan_k_smc.bin");
308− } else if ((adev->pdev->revision == 0xc3) &&
309− (adev->pdev->device == 0x6665)) {
310− info->is_kicker = true;
311− strcpy(fw_name, "radeon/banks_k_2_smc.bin");
312− } else {
313− strcpy(fw_name, "radeon/hainan_smc.bin");
314− }
315− break;
316255 case CHIP_BONAIRE:
317256 if ((adev->pdev->revision == 0x80) ||
318257 (adev->pdev->revision == 0x81) ||
319258 (adev->pdev->device == 0x665f)) {
320259 info->is_kicker = true;
321− strcpy(fw_name, "amdgpu/bonaire_k_smc.bin");
260+ strscpy(fw_name, "amdgpu/bonaire_k_smc.bin");
322261 } else {
323− strcpy(fw_name, "amdgpu/bonaire_smc.bin");
262+ strscpy(fw_name, "amdgpu/bonaire_smc.bin");
324263 }
325264 break;
326265 case CHIP_HAWAII:
327266 if (adev->pdev->revision == 0x80) {
328267 info->is_kicker = true;
329− strcpy(fw_name, "amdgpu/hawaii_k_smc.bin");
268+ strscpy(fw_name, "amdgpu/hawaii_k_smc.bin");
330269 } else {
331− strcpy(fw_name, "amdgpu/hawaii_smc.bin");
270+ strscpy(fw_name, "amdgpu/hawaii_smc.bin");
332271 }
333272 break;
334273 case CHIP_TOPAZ:
@@ -338,76 +277,76 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 3 unmodified lines
338277 ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD1)) ||
339278 ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD3))) {
340279 info->is_kicker = true;
341− strcpy(fw_name, "amdgpu/topaz_k_smc.bin");
280+ strscpy(fw_name, "amdgpu/topaz_k_smc.bin");
342281 } else
343− strcpy(fw_name, "amdgpu/topaz_smc.bin");
282+ strscpy(fw_name, "amdgpu/topaz_smc.bin");
344283 break;
345284 case CHIP_TONGA:
346285 if (((adev->pdev->device == 0x6939) && (adev->pdev->revision == 0xf1)) ||
347286 ((adev->pdev->device == 0x6938) && (adev->pdev->revision == 0xf1))) {
348287 info->is_kicker = true;
349− strcpy(fw_name, "amdgpu/tonga_k_smc.bin");
288+ strscpy(fw_name, "amdgpu/tonga_k_smc.bin");
350289 } else
351− strcpy(fw_name, "amdgpu/tonga_smc.bin");
290+ strscpy(fw_name, "amdgpu/tonga_smc.bin");
352291 break;
353292 case CHIP_FIJI:
354− strcpy(fw_name, "amdgpu/fiji_smc.bin");
293+ strscpy(fw_name, "amdgpu/fiji_smc.bin");
355294 break;
356295 case CHIP_POLARIS11:
357296 if (type == CGS_UCODE_ID_SMU) {
358297 if (ASICID_IS_P21(adev->pdev->device, adev->pdev->revision)) {
359298 info->is_kicker = true;
360− strcpy(fw_name, "amdgpu/polaris11_k_smc.bin");
299+ strscpy(fw_name, "amdgpu/polaris11_k_smc.bin");
361300 } else if (ASICID_IS_P31(adev->pdev->device, adev->pdev->revision)) {
362301 info->is_kicker = true;
363− strcpy(fw_name, "amdgpu/polaris11_k2_smc.bin");
302+ strscpy(fw_name, "amdgpu/polaris11_k2_smc.bin");
364303 } else {
365− strcpy(fw_name, "amdgpu/polaris11_smc.bin");
304+ strscpy(fw_name, "amdgpu/polaris11_smc.bin");
366305 }
367306 } else if (type == CGS_UCODE_ID_SMU_SK) {
368− strcpy(fw_name, "amdgpu/polaris11_smc_sk.bin");
307+ strscpy(fw_name, "amdgpu/polaris11_smc_sk.bin");
369308 }
370309 break;
371310 case CHIP_POLARIS10:
372311 if (type == CGS_UCODE_ID_SMU) {
373312 if (ASICID_IS_P20(adev->pdev->device, adev->pdev->revision)) {
374313 info->is_kicker = true;
375− strcpy(fw_name, "amdgpu/polaris10_k_smc.bin");
314+ strscpy(fw_name, "amdgpu/polaris10_k_smc.bin");
376315 } else if (ASICID_IS_P30(adev->pdev->device, adev->pdev->revision)) {
377316 info->is_kicker = true;
378− strcpy(fw_name, "amdgpu/polaris10_k2_smc.bin");
317+ strscpy(fw_name, "amdgpu/polaris10_k2_smc.bin");
379318 } else {
380− strcpy(fw_name, "amdgpu/polaris10_smc.bin");
319+ strscpy(fw_name, "amdgpu/polaris10_smc.bin");
381320 }
382321 } else if (type == CGS_UCODE_ID_SMU_SK) {
383− strcpy(fw_name, "amdgpu/polaris10_smc_sk.bin");
322+ strscpy(fw_name, "amdgpu/polaris10_smc_sk.bin");
384323 }
385324 break;
386325 case CHIP_POLARIS12:
387326 if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision)) {
388327 info->is_kicker = true;
389− strcpy(fw_name, "amdgpu/polaris12_k_smc.bin");
328+ strscpy(fw_name, "amdgpu/polaris12_k_smc.bin");
390329 } else {
391− strcpy(fw_name, "amdgpu/polaris12_smc.bin");
330+ strscpy(fw_name, "amdgpu/polaris12_smc.bin");
392331 }
393332 break;
394333 case CHIP_VEGAM:
395− strcpy(fw_name, "amdgpu/vegam_smc.bin");
334+ strscpy(fw_name, "amdgpu/vegam_smc.bin");
396335 break;
397336 case CHIP_VEGA10:
398337 if ((adev->pdev->device == 0x687f) &&
399338 ((adev->pdev->revision == 0xc0) ||
400339 (adev->pdev->revision == 0xc1) ||
401340 (adev->pdev->revision == 0xc3)))
402− strcpy(fw_name, "amdgpu/vega10_acg_smc.bin");
341+ strscpy(fw_name, "amdgpu/vega10_acg_smc.bin");
403342 else
404− strcpy(fw_name, "amdgpu/vega10_smc.bin");
343+ strscpy(fw_name, "amdgpu/vega10_smc.bin");
405344 break;
406345 case CHIP_VEGA12:
407− strcpy(fw_name, "amdgpu/vega12_smc.bin");
346+ strscpy(fw_name, "amdgpu/vega12_smc.bin");
408347 break;
409348 case CHIP_VEGA20:
410− strcpy(fw_name, "amdgpu/vega20_smc.bin");
349+ strscpy(fw_name, "amdgpu/vega20_smc.bin");
411350 break;
412351 default:
413352 DRM_ERROR("SMC firmware not supported\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_cper.c+2−2
@@ -549,7 +549,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 548 unmodified lines
549549 {
550550 int r;
551551
552− if (!amdgpu_aca_is_enabled(adev))
552+ if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev))
553553 return 0;
554554
555555 r = amdgpu_cper_ring_init(adev);
@@ -568,7 +568,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 12 unmodified lines
568568
569569 int amdgpu_cper_fini(struct amdgpu_device *adev)
570570 {
571− if (!amdgpu_aca_is_enabled(adev))
571+ if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev))
572572 return 0;
573573
574574 adev->cper.enabled = false;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c+23−1
@@ -296,7 +296,25 @@ static int amdgpu_cs_pass1(struct amdgpu_cs_parser *p, · 295 unmodified lines
296296 num_ibs[i], &p->jobs[i]);
297297 if (ret)
298298 goto free_all_kdata;
299− p->jobs[i]->enforce_isolation = p->adev->enforce_isolation[fpriv->xcp_id];
299+ switch (p->adev->enforce_isolation[fpriv->xcp_id]) {
300+ case AMDGPU_ENFORCE_ISOLATION_DISABLE:
301+ default:
302+ p->jobs[i]->enforce_isolation = false;
303+ p->jobs[i]->run_cleaner_shader = false;
304+ break;
305+ case AMDGPU_ENFORCE_ISOLATION_ENABLE:
306+ p->jobs[i]->enforce_isolation = true;
307+ p->jobs[i]->run_cleaner_shader = true;
308+ break;
309+ case AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY:
310+ p->jobs[i]->enforce_isolation = true;
311+ p->jobs[i]->run_cleaner_shader = false;
312+ break;
313+ case AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER:
314+ p->jobs[i]->enforce_isolation = true;
315+ p->jobs[i]->run_cleaner_shader = false;
316+ break;
317+ }
300318 }
301319 p->gang_leader = p->jobs[p->gang_leader_idx];
302320
@@ -349,6 +367,10 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p, · 46 unmodified lines
349367 ring = amdgpu_job_ring(job);
350368 ib = &job->ibs[job->num_ibs++];
351369
370+ /* submissions to kernel queues are disabled */
371+ if (ring->no_user_submission)
372+ return -EINVAL;
373+
352374 /* MM engine doesn't support user fences */
353375 if (p->uf_bo && ring->funcs->no_user_fence)
354376 return -EINVAL;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c+351−178
@@ -85,6 +85,7 @@ · 84 unmodified lines
8585
8686 #if IS_ENABLED(CONFIG_X86)
8787 #include <asm/intel-family.h>
88+#include <asm/cpu_device_id.h>
8889 #endif
8990
9091 MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin");
@@ -1680,6 +1681,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev) · 1589 unmodified lines
16801681 if (amdgpu_sriov_vf(adev))
16811682 return 0;
16821683
1684+ if (!amdgpu_rebar)
1685+ return 0;
1686+
16831687 /* resizing on Dell G5 SE platforms causes problems with runtime pm */
16841688 if ((amdgpu_runtime_pm != 0) &&
16851689 adev->pdev->vendor == PCI_VENDOR_ID_ATI &&
@@ -1870,6 +1874,35 @@ static bool amdgpu_device_pcie_dynamic_switching_supported(struct amdgpu_device · 184 unmodified lines
18701874 return true;
18711875 }
18721876
1877+static bool amdgpu_device_aspm_support_quirk(struct amdgpu_device *adev)
1878+{
1879+#if IS_ENABLED(CONFIG_X86)
1880+ struct cpuinfo_x86 *c = &cpu_data(0);
1881+
1882+ if (!(amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 0, 0) ||
1883+ amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 0, 1)))
1884+ return false;
1885+
1886+ if (c->x86 == 6 &&
1887+ adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN5) {
1888+ switch (c->x86_model) {
1889+ case VFM_MODEL(INTEL_ALDERLAKE):
1890+ case VFM_MODEL(INTEL_ALDERLAKE_L):
1891+ case VFM_MODEL(INTEL_RAPTORLAKE):
1892+ case VFM_MODEL(INTEL_RAPTORLAKE_P):
1893+ case VFM_MODEL(INTEL_RAPTORLAKE_S):
1894+ return true;
1895+ default:
1896+ return false;
1897+ }
1898+ } else {
1899+ return false;
1900+ }
1901+#else
1902+ return false;
1903+#endif
1904+}
1905+
18731906 /**
18741907 * amdgpu_device_should_use_aspm - check if the device should program ASPM
18751908 *
@@ -1894,7 +1927,7 @@ bool amdgpu_device_should_use_aspm(struct amdgpu_device *adev) · 18 unmodified lines
18941927 }
18951928 if (adev->flags & AMD_IS_APU)
18961929 return false;
1897− if (!(adev->pm.pp_feature & PP_PCIE_DPM_MASK))
1930+ if (amdgpu_device_aspm_support_quirk(adev))
18981931 return false;
18991932 return pcie_aspm_enabled(adev->pdev);
19001933 }
@@ -2112,8 +2145,31 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev) · 211 unmodified lines
21122145
21132146 adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, amdgpu_fw_load_type);
21142147
2115− for (i = 0; i < MAX_XCP; i++)
2116− adev->enforce_isolation[i] = !!enforce_isolation;
2148+ for (i = 0; i < MAX_XCP; i++) {
2149+ switch (amdgpu_enforce_isolation) {
2150+ case -1:
2151+ case 0:
2152+ default:
2153+ /* disable */
2154+ adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE;
2155+ break;
2156+ case 1:
2157+ /* enable */
2158+ adev->enforce_isolation[i] =
2159+ AMDGPU_ENFORCE_ISOLATION_ENABLE;
2160+ break;
2161+ case 2:
2162+ /* enable legacy mode */
2163+ adev->enforce_isolation[i] =
2164+ AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY;
2165+ break;
2166+ case 3:
2167+ /* enable only process isolation without submitting cleaner shader */
2168+ adev->enforce_isolation[i] =
2169+ AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER;
2170+ break;
2171+ }
2172+ }
21172173
21182174 return 0;
21192175 }
@@ -2689,6 +2745,13 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 569 unmodified lines
26892745 break;
26902746 }
26912747
2748+ /* Check for IP version 9.4.3 with A0 hardware */
2749+ if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) &&
2750+ !amdgpu_device_get_rev_id(adev)) {
2751+ dev_err(adev->dev, "Unsupported A0 hardware\n");
2752+ return -ENODEV; /* device unsupported - no device error */
2753+ }
2754+
26922755 if (amdgpu_has_atpx() &&
26932756 (amdgpu_is_atpx_hybrid() ||
26942757 amdgpu_has_atpx_dgpu_power_cntl()) &&
@@ -2701,7 +2764,6 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 6 unmodified lines
27012764 adev->has_pr3 = parent ? pci_pr3_present(parent) : false;
27022765 }
27032766
2704−
27052767 adev->pm.pp_feature = amdgpu_pp_feature_mask;
27062768 if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS)
27072769 adev->pm.pp_feature &= ~PP_GFXOFF_MASK;
@@ -3172,6 +3234,7 @@ static bool amdgpu_device_check_vram_lost(struct amdgpu_device *adev) · 464 unmodified lines
31723234 * always assumed to be lost.
31733235 */
31743236 switch (amdgpu_asic_reset_method(adev)) {
3237+ case AMD_RESET_METHOD_LINK:
31753238 case AMD_RESET_METHOD_BACO:
31763239 case AMD_RESET_METHOD_MODE1:
31773240 return true;
@@ -3455,6 +3518,7 @@ static int amdgpu_device_ip_fini_early(struct amdgpu_device *adev) · 277 unmodified lines
34553518 amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE);
34563519
34573520 amdgpu_amdkfd_suspend(adev, false);
3521+ amdgpu_userq_suspend(adev);
34583522
34593523 /* Workaround for ASICs need to disable SMC first */
34603524 amdgpu_device_smu_fini_early(adev);
@@ -4307,9 +4371,10 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 846 unmodified lines
43074371 amdgpu_sync_create(&adev->isolation[i].active);
43084372 amdgpu_sync_create(&adev->isolation[i].prev);
43094373 }
4310− mutex_init(&adev->gfx.kfd_sch_mutex);
4374+ mutex_init(&adev->gfx.userq_sch_mutex);
43114375 mutex_init(&adev->gfx.workload_profile_mutex);
43124376 mutex_init(&adev->vcn.workload_profile_mutex);
4377+ mutex_init(&adev->userq_mutex);
43134378
43144379 amdgpu_device_init_apu_flags(adev);
43154380
@@ -4329,12 +4394,16 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 13 unmodified lines
43294394 spin_lock_init(&adev->virt.rlcg_reg_lock);
43304395 spin_lock_init(&adev->wb.lock);
43314396
4397+ xa_init_flags(&adev->userq_xa, XA_FLAGS_LOCK_IRQ);
4398+
43324399 INIT_LIST_HEAD(&adev->reset_list);
43334400
43344401 INIT_LIST_HEAD(&adev->ras_list);
43354402
43364403 INIT_LIST_HEAD(&adev->pm.od_kobj_list);
43374404
4405+ INIT_LIST_HEAD(&adev->userq_mgr_list);
4406+
43384407 INIT_DELAYED_WORK(&adev->delayed_init_work,
43394408 amdgpu_device_delayed_init_work_handler);
43404409 INIT_DELAYED_WORK(&adev->gfx.gfx_off_delay_work,
@@ -4907,28 +4976,20 @@ static int amdgpu_device_evict_resources(struct amdgpu_device *adev) · 566 unmodified lines
49074976 * @data: data
49084977 *
49094978 * This function is called when the system is about to suspend or hibernate.
4910− * It is used to evict resources from the device before the system goes to
4911− * sleep while there is still access to swap.
4979+ * It is used to set the appropriate flags so that eviction can be optimized
4980+ * in the pm prepare callback.
49124981 */
49134982 static int amdgpu_device_pm_notifier(struct notifier_block *nb, unsigned long mode,
49144983 void *data)
49154984 {
49164985 struct amdgpu_device *adev = container_of(nb, struct amdgpu_device, pm_nb);
4917− int r;
49184986
49194987 switch (mode) {
49204988 case PM_HIBERNATION_PREPARE:
49214989 adev->in_s4 = true;
4922− fallthrough;
4923− case PM_SUSPEND_PREPARE:
4924− r = amdgpu_device_evict_resources(adev);
4925− /*
4926− * This is considered non-fatal at this time because
4927− * amdgpu_device_prepare() will also fatally evict resources.
4928− * See https://gitlab.freedesktop.org/drm/amd/-/issues/3781
4929− */
4930− if (r)
4931− drm_warn(adev_to_drm(adev), "Failed to evict resources, freeze active processes if problems occur: %d\n", r);
4990+ break;
4991+ case PM_POST_HIBERNATION:
4992+ adev->in_s4 = false;
49324993 break;
49334994 }
49344995
@@ -4949,15 +5010,13 @@ int amdgpu_device_prepare(struct drm_device *dev) · 14 unmodified lines
49495010 struct amdgpu_device *adev = drm_to_adev(dev);
49505011 int i, r;
49515012
4952− amdgpu_choose_low_power_state(adev);
4953−
49545013 if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
49555014 return 0;
49565015
49575016 /* Evict the majority of BOs before starting suspend sequence */
49585017 r = amdgpu_device_evict_resources(adev);
49595018 if (r)
4960− goto unprepare;
5019+ return r;
49615020
49625021 flush_delayed_work(&adev->gfx.gfx_off_delay_work);
49635022
@@ -4968,15 +5027,10 @@ int amdgpu_device_prepare(struct drm_device *dev) · 4 unmodified lines
49685027 continue;
49695028 r = adev->ip_blocks[i].version->funcs->prepare_suspend(&adev->ip_blocks[i]);
49705029 if (r)
4971− goto unprepare;
5030+ return r;
49725031 }
49735032
49745033 return 0;
4975−
4976−unprepare:
4977− adev->in_s0ix = adev->in_s3 = adev->in_s4 = false;
4978−
4979− return r;
49805034 }
49815035
49825036 /**
@@ -5018,8 +5072,10 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients) · 35 unmodified lines
50185072
50195073 amdgpu_device_ip_suspend_phase1(adev);
50205074
5021− if (!adev->in_s0ix)
5075+ if (!adev->in_s0ix) {
50225076 amdgpu_amdkfd_suspend(adev, adev->in_runpm);
5077+ amdgpu_userq_suspend(adev);
5078+ }
50235079
50245080 r = amdgpu_device_evict_resources(adev);
50255081 if (r)
@@ -5086,6 +5142,10 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) · 60 unmodified lines
50865142 r = amdgpu_amdkfd_resume(adev, adev->in_runpm);
50875143 if (r)
50885144 goto exit;
5145+
5146+ r = amdgpu_userq_resume(adev);
5147+ if (r)
5148+ goto exit;
50895149 }
50905150
50915151 r = amdgpu_device_ip_late_init(adev);
@@ -5134,9 +5194,6 @@ exit: · 42 unmodified lines
51345194 }
51355195 adev->in_suspend = false;
51365196
5137− if (adev->enable_mes)
5138− amdgpu_mes_self_test(adev);
5139−
51405197 if (amdgpu_acpi_smart_shift_update(dev, AMDGPU_SS_DEV_D0))
51415198 DRM_WARN("smart shift update failed\n");
51425199
@@ -5517,6 +5574,29 @@ mode1_reset_failed: · 374 unmodified lines
55175574 return ret;
55185575 }
55195576
5577+int amdgpu_device_link_reset(struct amdgpu_device *adev)
5578+{
5579+ int ret = 0;
5580+
5581+ dev_info(adev->dev, "GPU link reset\n");
5582+
5583+ if (!adev->pcie_reset_ctx.occurs_dpc)
5584+ ret = amdgpu_dpm_link_reset(adev);
5585+
5586+ if (ret)
5587+ goto link_reset_failed;
5588+
5589+ ret = amdgpu_psp_wait_for_bootloader(adev);
5590+ if (ret)
5591+ goto link_reset_failed;
5592+
5593+ return 0;
5594+
5595+link_reset_failed:
5596+ dev_err(adev->dev, "GPU link reset failed\n");
5597+ return ret;
5598+}
5599+
55205600 int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
55215601 struct amdgpu_reset_context *reset_context)
55225602 {
@@ -5821,6 +5901,7 @@ static void amdgpu_device_set_mp1_state(struct amdgpu_device *adev) · 298 unmodified lines
58215901
58225902 switch (amdgpu_asic_reset_method(adev)) {
58235903 case AMD_RESET_METHOD_MODE1:
5904+ case AMD_RESET_METHOD_LINK:
58245905 adev->mp1_state = PP_MP1_STATE_SHUTDOWN;
58255906 break;
58265907 case AMD_RESET_METHOD_MODE2:
@@ -5937,94 +6018,42 @@ static int amdgpu_device_health_check(struct list_head *device_list_handle) · 110 unmodified lines
59376018 return ret;
59386019 }
59396020
5940−/**
5941− * amdgpu_device_gpu_recover - reset the asic and recover scheduler
5942− *
5943− * @adev: amdgpu_device pointer
5944− * @job: which job trigger hang
5945− * @reset_context: amdgpu reset context pointer
5946− *
5947− * Attempt to reset the GPU if it has hung (all asics).
5948− * Attempt to do soft-reset or full-reset and reinitialize Asic
5949− * Returns 0 for success or an error on failure.
5950− */
5951−
5952−int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
6021+static int amdgpu_device_halt_activities(struct amdgpu_device *adev,
59536022 struct amdgpu_job *job,
5954− struct amdgpu_reset_context *reset_context)
6023+ struct amdgpu_reset_context *reset_context,
6024+ struct list_head *device_list,
6025+ struct amdgpu_hive_info *hive,
6026+ bool need_emergency_restart)
59556027 {
5956− struct list_head device_list, *device_list_handle = NULL;
5957− bool job_signaled = false;
5958− struct amdgpu_hive_info *hive = NULL;
6028+ struct list_head *device_list_handle = NULL;
59596029 struct amdgpu_device *tmp_adev = NULL;
59606030 int i, r = 0;
5961− bool need_emergency_restart = false;
5962− bool audio_suspended = false;
5963− int retry_limit = AMDGPU_MAX_RETRY_LIMIT;
59646031
5965− /*
5966− * If it reaches here because of hang/timeout and a RAS error is
5967− * detected at the same time, let RAS recovery take care of it.
5968− */
5969− if (amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY) &&
5970− !amdgpu_sriov_vf(adev) &&
5971− reset_context->src != AMDGPU_RESET_SRC_RAS) {
5972− dev_dbg(adev->dev,
5973− "Gpu recovery from source: %d yielding to RAS error recovery handling",
5974− reset_context->src);
5975− return 0;
5976− }
5977− /*
5978− * Special case: RAS triggered and full reset isn't supported
5979− */
5980− need_emergency_restart = amdgpu_ras_need_emergency_restart(adev);
5981−
5982− /*
5983− * Flush RAM to disk so that after reboot
5984− * the user can read log and see why the system rebooted.
5985− */
5986− if (need_emergency_restart && amdgpu_ras_get_context(adev) &&
5987− amdgpu_ras_get_context(adev)->reboot) {
5988− DRM_WARN("Emergency reboot.");
5989−
5990− ksys_sync_helper();
5991− emergency_restart();
5992− }
5993−
5994− dev_info(adev->dev, "GPU %s begin!\n",
5995− need_emergency_restart ? "jobs stop":"reset");
5996−
5997− if (!amdgpu_sriov_vf(adev))
5998− hive = amdgpu_get_xgmi_hive(adev);
5999− if (hive)
6000− mutex_lock(&hive->hive_lock);
6001−
6002− reset_context->job = job;
6003− reset_context->hive = hive;
60046032 /*
60056033 * Build list of devices to reset.
60066034 * In case we are in XGMI hive mode, resort the device list
60076035 * to put adev in the 1st position.
60086036 */
6009− INIT_LIST_HEAD(&device_list);
60106037 if (!amdgpu_sriov_vf(adev) && (adev->gmc.xgmi.num_physical_nodes > 1) && hive) {
60116038 list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
6012− list_add_tail(&tmp_adev->reset_list, &device_list);
6039+ list_add_tail(&tmp_adev->reset_list, device_list);
60136040 if (adev->shutdown)
60146041 tmp_adev->shutdown = true;
6042+ if (adev->pcie_reset_ctx.occurs_dpc)
6043+ tmp_adev->pcie_reset_ctx.in_link_reset = true;
60156044 }
6016− if (!list_is_first(&adev->reset_list, &device_list))
6017− list_rotate_to_front(&adev->reset_list, &device_list);
6018− device_list_handle = &device_list;
6045+ if (!list_is_first(&adev->reset_list, device_list))
6046+ list_rotate_to_front(&adev->reset_list, device_list);
6047+ device_list_handle = device_list;
60196048 } else {
6020− list_add_tail(&adev->reset_list, &device_list);
6021− device_list_handle = &device_list;
6049+ list_add_tail(&adev->reset_list, device_list);
6050+ device_list_handle = device_list;
60226051 }
60236052
6024− if (!amdgpu_sriov_vf(adev)) {
6053+ if (!amdgpu_sriov_vf(adev) && (!adev->pcie_reset_ctx.occurs_dpc)) {
60256054 r = amdgpu_device_health_check(device_list_handle);
60266055 if (r)
6027− goto end_reset;
6056+ return r;
60286057 }
60296058
60306059 /* We need to lock reset domain only once both for XGMI and single device */
@@ -6048,7 +6077,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 17 unmodified lines
60486077 * some audio codec errors.
60496078 */
60506079 if (!amdgpu_device_suspend_display_audio(tmp_adev))
6051− audio_suspended = true;
6080+ tmp_adev->pcie_reset_ctx.audio_suspended = true;
60526081
60536082 amdgpu_ras_set_error_query_ready(tmp_adev, false);
60546083
@@ -6066,6 +6095,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 11 unmodified lines
60666095
60676096 /* disable ras on ALL IPs */
60686097 if (!need_emergency_restart &&
6098+ (!adev->pcie_reset_ctx.occurs_dpc) &&
60696099 amdgpu_device_ip_need_full_reset(tmp_adev))
60706100 amdgpu_ras_suspend(tmp_adev);
60716101
@@ -6083,24 +6113,24 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 11 unmodified lines
60836113 atomic_inc(&tmp_adev->gpu_reset_counter);
60846114 }
60856115
6086− if (need_emergency_restart)
6087− goto skip_sched_resume;
6116+ return r;
6117+}
60886118
6089− /*
6090− * Must check guilty signal here since after this point all old
6091− * HW fences are force signaled.
6092− *
6093− * job->base holds a reference to parent fence
6094− */
6095− if (job && dma_fence_is_signaled(&job->hw_fence)) {
6096− job_signaled = true;
6097− dev_info(adev->dev, "Guilty job already signaled, skipping HW reset");
6098− goto skip_hw_reset;
6099− }
6119+static int amdgpu_device_asic_reset(struct amdgpu_device *adev,
6120+ struct list_head *device_list,
6121+ struct amdgpu_reset_context *reset_context)
6122+{
6123+ struct amdgpu_device *tmp_adev = NULL;
6124+ int retry_limit = AMDGPU_MAX_RETRY_LIMIT;
6125+ int r = 0;
61006126
61016127 retry: /* Rest of adevs pre asic reset from XGMI hive. */
6102− list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
6128+ list_for_each_entry(tmp_adev, device_list, reset_list) {
6129+ if (adev->pcie_reset_ctx.occurs_dpc)
6130+ tmp_adev->no_hw_access = true;
61036131 r = amdgpu_device_pre_asic_reset(tmp_adev, reset_context);
6132+ if (adev->pcie_reset_ctx.occurs_dpc)
6133+ tmp_adev->no_hw_access = false;
61046134 /*TODO Should we stop ?*/
61056135 if (r) {
61066136 dev_err(tmp_adev->dev, "GPU pre asic reset failed with err, %d for drm dev, %s ",
@@ -6112,6 +6142,11 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 5 unmodified lines
61126142 /* Actual ASIC resets if needed.*/
61136143 /* Host driver will handle XGMI hive reset for SRIOV */
61146144 if (amdgpu_sriov_vf(adev)) {
6145+
6146+ /* Bail out of reset early */
6147+ if (amdgpu_ras_is_rma(adev))
6148+ return -ENODEV;
6149+
61156150 if (amdgpu_ras_get_fed_status(adev) || amdgpu_virt_rcvd_ras_interrupt(adev)) {
61166151 dev_dbg(adev->dev, "Detected RAS error, wait for FLR completion\n");
61176152 amdgpu_ras_set_fed(adev, true);
@@ -6126,12 +6161,12 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 8 unmodified lines
61266161 if (r)
61276162 adev->asic_reset_res = r;
61286163 } else {
6129− r = amdgpu_do_asic_reset(device_list_handle, reset_context);
6164+ r = amdgpu_do_asic_reset(device_list, reset_context);
61306165 if (r && r == -EAGAIN)
61316166 goto retry;
61326167 }
61336168
6134− list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
6169+ list_for_each_entry(tmp_adev, device_list, reset_list) {
61356170 /*
61366171 * Drop any pending non scheduler resets queued before reset is done.
61376172 * Any reset scheduled after this point would be valid. Scheduler resets
@@ -6141,10 +6176,18 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 3 unmodified lines
61416176 amdgpu_device_stop_pending_resets(tmp_adev);
61426177 }
61436178
6144−skip_hw_reset:
6179+ return r;
6180+}
6181+
6182+static int amdgpu_device_sched_resume(struct list_head *device_list,
6183+ struct amdgpu_reset_context *reset_context,
6184+ bool job_signaled)
6185+{
6186+ struct amdgpu_device *tmp_adev = NULL;
6187+ int i, r = 0;
61456188
61466189 /* Post ASIC reset for all devs .*/
6147− list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
6190+ list_for_each_entry(tmp_adev, device_list, reset_list) {
61486191
61496192 for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
61506193 struct amdgpu_ring *ring = tmp_adev->rings[i];
@@ -6180,8 +6223,16 @@ skip_hw_reset: · 29 unmodified lines
61806223 }
61816224 }
61826225
6183−skip_sched_resume:
6184− list_for_each_entry(tmp_adev, device_list_handle, reset_list) {
6226+ return r;
6227+}
6228+
6229+static void amdgpu_device_gpu_resume(struct amdgpu_device *adev,
6230+ struct list_head *device_list,
6231+ bool need_emergency_restart)
6232+{
6233+ struct amdgpu_device *tmp_adev = NULL;
6234+
6235+ list_for_each_entry(tmp_adev, device_list, reset_list) {
61856236 /* unlock kfd: SRIOV would do it separately */
61866237 if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev))
61876238 amdgpu_amdkfd_post_reset(tmp_adev);
@@ -6192,18 +6243,114 @@ skip_sched_resume: · 4 unmodified lines
61926243 if (!adev->kfd.init_complete)
61936244 amdgpu_amdkfd_device_init(adev);
61946245
6195− if (audio_suspended)
6246+ if (tmp_adev->pcie_reset_ctx.audio_suspended)
61966247 amdgpu_device_resume_display_audio(tmp_adev);
61976248
61986249 amdgpu_device_unset_mp1_state(tmp_adev);
61996250
62006251 amdgpu_ras_set_error_query_ready(tmp_adev, true);
6252+
62016253 }
62026254
6203− tmp_adev = list_first_entry(device_list_handle, struct amdgpu_device,
6255+ tmp_adev = list_first_entry(device_list, struct amdgpu_device,
62046256 reset_list);
62056257 amdgpu_device_unlock_reset_domain(tmp_adev->reset_domain);
62066258
6259+}
6260+
6261+
6262+/**
6263+ * amdgpu_device_gpu_recover - reset the asic and recover scheduler
6264+ *
6265+ * @adev: amdgpu_device pointer
6266+ * @job: which job trigger hang
6267+ * @reset_context: amdgpu reset context pointer
6268+ *
6269+ * Attempt to reset the GPU if it has hung (all asics).
6270+ * Attempt to do soft-reset or full-reset and reinitialize Asic
6271+ * Returns 0 for success or an error on failure.
6272+ */
6273+
6274+int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
6275+ struct amdgpu_job *job,
6276+ struct amdgpu_reset_context *reset_context)
6277+{
6278+ struct list_head device_list;
6279+ bool job_signaled = false;
6280+ struct amdgpu_hive_info *hive = NULL;
6281+ int r = 0;
6282+ bool need_emergency_restart = false;
6283+
6284+ /*
6285+ * If it reaches here because of hang/timeout and a RAS error is
6286+ * detected at the same time, let RAS recovery take care of it.
6287+ */
6288+ if (amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY) &&
6289+ !amdgpu_sriov_vf(adev) &&
6290+ reset_context->src != AMDGPU_RESET_SRC_RAS) {
6291+ dev_dbg(adev->dev,
6292+ "Gpu recovery from source: %d yielding to RAS error recovery handling",
6293+ reset_context->src);
6294+ return 0;
6295+ }
6296+
6297+ /*
6298+ * Special case: RAS triggered and full reset isn't supported
6299+ */
6300+ need_emergency_restart = amdgpu_ras_need_emergency_restart(adev);
6301+
6302+ /*
6303+ * Flush RAM to disk so that after reboot
6304+ * the user can read log and see why the system rebooted.
6305+ */
6306+ if (need_emergency_restart && amdgpu_ras_get_context(adev) &&
6307+ amdgpu_ras_get_context(adev)->reboot) {
6308+ DRM_WARN("Emergency reboot.");
6309+
6310+ ksys_sync_helper();
6311+ emergency_restart();
6312+ }
6313+
6314+ dev_info(adev->dev, "GPU %s begin!\n",
6315+ need_emergency_restart ? "jobs stop":"reset");
6316+
6317+ if (!amdgpu_sriov_vf(adev))
6318+ hive = amdgpu_get_xgmi_hive(adev);
6319+ if (hive)
6320+ mutex_lock(&hive->hive_lock);
6321+
6322+ reset_context->job = job;
6323+ reset_context->hive = hive;
6324+ INIT_LIST_HEAD(&device_list);
6325+
6326+ r = amdgpu_device_halt_activities(adev, job, reset_context, &device_list,
6327+ hive, need_emergency_restart);
6328+ if (r)
6329+ goto end_reset;
6330+
6331+ if (need_emergency_restart)
6332+ goto skip_sched_resume;
6333+ /*
6334+ * Must check guilty signal here since after this point all old
6335+ * HW fences are force signaled.
6336+ *
6337+ * job->base holds a reference to parent fence
6338+ */
6339+ if (job && dma_fence_is_signaled(&job->hw_fence)) {
6340+ job_signaled = true;
6341+ dev_info(adev->dev, "Guilty job already signaled, skipping HW reset");
6342+ goto skip_hw_reset;
6343+ }
6344+
6345+ r = amdgpu_device_asic_reset(adev, &device_list, reset_context);
6346+ if (r)
6347+ goto end_reset;
6348+skip_hw_reset:
6349+ r = amdgpu_device_sched_resume(&device_list, reset_context, job_signaled);
6350+ if (r)
6351+ goto end_reset;
6352+skip_sched_resume:
6353+ amdgpu_device_gpu_resume(adev, &device_list, need_emergency_restart);
62076354 end_reset:
62086355 if (hive) {
62096356 mutex_unlock(&hive->hive_lock);
@@ -6587,12 +6734,15 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 377 unmodified lines
65876734 {
65886735 struct drm_device *dev = pci_get_drvdata(pdev);
65896736 struct amdgpu_device *adev = drm_to_adev(dev);
6590− int i;
6737+ struct amdgpu_hive_info *hive = amdgpu_get_xgmi_hive(adev);
6738+ struct amdgpu_reset_context reset_context;
6739+ struct list_head device_list;
6740+ int r = 0;
65916741
6592− DRM_INFO("PCI error: detected callback, state(%d)!!\n", state);
6742+ dev_info(adev->dev, "PCI error: detected callback!!\n");
65936743
6594− if (adev->gmc.xgmi.num_physical_nodes > 1) {
6595− DRM_WARN("No support for XGMI hive yet...");
6744+ if (!amdgpu_dpm_is_link_reset_supported(adev)) {
6745+ dev_warn(adev->dev, "No support for XGMI hive yet...\n");
65966746 return PCI_ERS_RESULT_DISCONNECT;
65976747 }
65986748
@@ -6600,32 +6750,30 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 1 unmodified line
66006750
66016751 switch (state) {
66026752 case pci_channel_io_normal:
6753+ dev_info(adev->dev, "pci_channel_io_normal: state(%d)!!\n", state);
66036754 return PCI_ERS_RESULT_CAN_RECOVER;
6604− /* Fatal error, prepare for slot reset */
66056755 case pci_channel_io_frozen:
6606− /*
6607− * Locking adev->reset_domain->sem will prevent any external access
6608− * to GPU during PCI error recovery
6609− */
6610− amdgpu_device_lock_reset_domain(adev->reset_domain);
6611− amdgpu_device_set_mp1_state(adev);
6612−
6613− /*
6614− * Block any work scheduling as we do for regular GPU reset
6615− * for the duration of the recovery
6616− */
6617− for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
6618− struct amdgpu_ring *ring = adev->rings[i];
6619−
6620− if (!amdgpu_ring_sched_ready(ring))
6621− continue;
6622−
6623− drm_sched_stop(&ring->sched, NULL);
6756+ /* Fatal error, prepare for slot reset */
6757+ dev_info(adev->dev, "pci_channel_io_frozen: state(%d)!!\n", state);
6758+
6759+ if (hive)
6760+ mutex_lock(&hive->hive_lock);
6761+ adev->pcie_reset_ctx.occurs_dpc = true;
6762+ memset(&reset_context, 0, sizeof(reset_context));
6763+ INIT_LIST_HEAD(&device_list);
6764+
6765+ r = amdgpu_device_halt_activities(adev, NULL, &reset_context, &device_list,
6766+ hive, false);
6767+ if (hive) {
6768+ mutex_unlock(&hive->hive_lock);
6769+ amdgpu_put_xgmi_hive(hive);
66246770 }
6625− atomic_inc(&adev->gpu_reset_counter);
6771+ if (r)
6772+ return PCI_ERS_RESULT_DISCONNECT;
66266773 return PCI_ERS_RESULT_NEED_RESET;
66276774 case pci_channel_io_perm_failure:
66286775 /* Permanent error, prepare for device removal */
6776+ dev_info(adev->dev, "pci_channel_io_perm_failure: state(%d)!!\n", state);
66296777 return PCI_ERS_RESULT_DISCONNECT;
66306778 }
66316779
@@ -6638,8 +6786,10 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 6 unmodified lines
66386786 */
66396787 pci_ers_result_t amdgpu_pci_mmio_enabled(struct pci_dev *pdev)
66406788 {
6789+ struct drm_device *dev = pci_get_drvdata(pdev);
6790+ struct amdgpu_device *adev = drm_to_adev(dev);
66416791
6642− DRM_INFO("PCI error: mmio enabled callback!!\n");
6792+ dev_info(adev->dev, "PCI error: mmio enabled callback!!\n");
66436793
66446794 /* TODO - dump whatever for debugging purposes */
66456795
@@ -6663,10 +6813,12 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 17 unmodified lines
66636813 {
66646814 struct drm_device *dev = pci_get_drvdata(pdev);
66656815 struct amdgpu_device *adev = drm_to_adev(dev);
6666− int r, i;
66676816 struct amdgpu_reset_context reset_context;
6668− u32 memsize;
6817+ struct amdgpu_device *tmp_adev;
6818+ struct amdgpu_hive_info *hive;
66696819 struct list_head device_list;
6820+ int r = 0, i;
6821+ u32 memsize;
66706822
66716823 /* PCI error slot reset should be skipped During RAS recovery */
66726824 if ((amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) ||
@@ -6674,15 +6826,12 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 1 unmodified line
66746826 amdgpu_ras_in_recovery(adev))
66756827 return PCI_ERS_RESULT_RECOVERED;
66766828
6677− DRM_INFO("PCI error: slot reset callback!!\n");
6829+ dev_info(adev->dev, "PCI error: slot reset callback!!\n");
66786830
66796831 memset(&reset_context, 0, sizeof(reset_context));
66806832
6681− INIT_LIST_HEAD(&device_list);
6682− list_add_tail(&adev->reset_list, &device_list);
6683−
66846833 /* wait for asic to come out of reset */
6685− msleep(500);
6834+ msleep(700);
66866835
66876836 /* Restore PCI confspace */
66886837 amdgpu_device_load_pci_state(pdev);
@@ -6703,26 +6852,40 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 14 unmodified lines
67036852 reset_context.method = AMD_RESET_METHOD_NONE;
67046853 reset_context.reset_req_dev = adev;
67056854 set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);
6706− set_bit(AMDGPU_SKIP_HW_RESET, &reset_context.flags);
6707−
6708− adev->no_hw_access = true;
6709− r = amdgpu_device_pre_asic_reset(adev, &reset_context);
6710− adev->no_hw_access = false;
6711− if (r)
6712− goto out;
6855+ set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags);
6856+ INIT_LIST_HEAD(&device_list);
67136857
6714− r = amdgpu_do_asic_reset(&device_list, &reset_context);
6858+ hive = amdgpu_get_xgmi_hive(adev);
6859+ if (hive) {
6860+ mutex_lock(&hive->hive_lock);
6861+ reset_context.hive = hive;
6862+ list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
6863+ tmp_adev->pcie_reset_ctx.in_link_reset = true;
6864+ list_add_tail(&tmp_adev->reset_list, &device_list);
6865+ }
6866+ } else {
6867+ set_bit(AMDGPU_SKIP_HW_RESET, &reset_context.flags);
6868+ list_add_tail(&adev->reset_list, &device_list);
6869+ }
67156870
6871+ r = amdgpu_device_asic_reset(adev, &device_list, &reset_context);
67166872 out:
67176873 if (!r) {
67186874 if (amdgpu_device_cache_pci_state(adev->pdev))
67196875 pci_restore_state(adev->pdev);
6720−
6721− DRM_INFO("PCIe error recovery succeeded\n");
6876+ dev_info(adev->dev, "PCIe error recovery succeeded\n");
67226877 } else {
6723− DRM_ERROR("PCIe error recovery failed, err:%d", r);
6724− amdgpu_device_unset_mp1_state(adev);
6725− amdgpu_device_unlock_reset_domain(adev->reset_domain);
6878+ dev_err(adev->dev, "PCIe error recovery failed, err:%d\n", r);
6879+ if (hive) {
6880+ list_for_each_entry(tmp_adev, &device_list, reset_list)
6881+ amdgpu_device_unset_mp1_state(tmp_adev);
6882+ amdgpu_device_unlock_reset_domain(adev->reset_domain);
6883+ }
6884+ }
6885+
6886+ if (hive) {
6887+ mutex_unlock(&hive->hive_lock);
6888+ amdgpu_put_xgmi_hive(hive);
67266889 }
67276890
67286891 return r ? PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_RECOVERED;
@@ -6739,26 +6902,36 @@ void amdgpu_pci_resume(struct pci_dev *pdev) · 10 unmodified lines
67396902 {
67406903 struct drm_device *dev = pci_get_drvdata(pdev);
67416904 struct amdgpu_device *adev = drm_to_adev(dev);
6742− int i;
6743−
6905+ struct list_head device_list;
6906+ struct amdgpu_hive_info *hive = NULL;
6907+ struct amdgpu_device *tmp_adev = NULL;
67446908
6745− DRM_INFO("PCI error: resume callback!!\n");
6909+ dev_info(adev->dev, "PCI error: resume callback!!\n");
67466910
67476911 /* Only continue execution for the case of pci_channel_io_frozen */
67486912 if (adev->pci_channel_state != pci_channel_io_frozen)
67496913 return;
67506914
6751− for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
6752− struct amdgpu_ring *ring = adev->rings[i];
6915+ INIT_LIST_HEAD(&device_list);
67536916
6754− if (!amdgpu_ring_sched_ready(ring))
6755− continue;
6917+ hive = amdgpu_get_xgmi_hive(adev);
6918+ if (hive) {
6919+ mutex_lock(&hive->hive_lock);
6920+ list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
6921+ tmp_adev->pcie_reset_ctx.in_link_reset = false;
6922+ list_add_tail(&tmp_adev->reset_list, &device_list);
6923+ }
6924+ } else
6925+ list_add_tail(&adev->reset_list, &device_list);
67566926
6757− drm_sched_start(&ring->sched, 0);
6758− }
6927+ amdgpu_device_sched_resume(&device_list, NULL, NULL);
6928+ amdgpu_device_gpu_resume(adev, &device_list, false);
6929+ adev->pcie_reset_ctx.occurs_dpc = false;
67596930
6760− amdgpu_device_unset_mp1_state(adev);
6761− amdgpu_device_unlock_reset_domain(adev->reset_domain);
6931+ if (hive) {
6932+ mutex_unlock(&hive->hive_lock);
6933+ amdgpu_put_xgmi_hive(hive);
6934+ }
67626935 }
67636936
67646937 bool amdgpu_device_cache_pci_state(struct pci_dev *pdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c+94−16
@@ -51,6 +51,8 @@ · 50 unmodified lines
5151 #include "amdgpu_reset.h"
5252 #include "amdgpu_sched.h"
5353 #include "amdgpu_xgmi.h"
54+#include "amdgpu_userq.h"
55+#include "amdgpu_userq_fence.h"
5456 #include "../amdxcp/amdgpu_xcp_drv.h"
5557
5658 /*
@@ -123,9 +125,10 @@ · 66 unmodified lines
123125 * - 3.61.0 - Contains fix for RV/PCO compute queues
124126 * - 3.62.0 - Add AMDGPU_IDS_FLAGS_MODE_PF, AMDGPU_IDS_FLAGS_MODE_VF & AMDGPU_IDS_FLAGS_MODE_PT
125127 * - 3.63.0 - GFX12 display DCC supports 256B max compressed block size
128+ * - 3.64.0 - Userq IP support query
126129 */
127130 #define KMS_DRIVER_MAJOR 3
128−#define KMS_DRIVER_MINOR 63
131+#define KMS_DRIVER_MINOR 64
129132 #define KMS_DRIVER_PATCHLEVEL 0
130133
131134 /*
@@ -140,6 +143,7 @@ enum AMDGPU_DEBUG_MASK { · 8 unmodified lines
140143 AMDGPU_DEBUG_ENABLE_EXP_RESETS = BIT(5),
141144 AMDGPU_DEBUG_DISABLE_GPU_RING_RESET = BIT(6),
142145 AMDGPU_DEBUG_SMU_POOL = BIT(7),
146+ AMDGPU_DEBUG_VM_USERPTR = BIT(8),
143147 };
144148
145149 unsigned int amdgpu_vram_limit = UINT_MAX;
@@ -176,7 +180,7 @@ uint amdgpu_pg_mask = 0xffffffff; · 30 unmodified lines
176180 uint amdgpu_sdma_phase_quantum = 32;
177181 char *amdgpu_disable_cu;
178182 char *amdgpu_virtual_display;
179−bool enforce_isolation;
183+int amdgpu_enforce_isolation = -1;
180184 int amdgpu_modeset = -1;
181185
182186 /* Specifies the default granularity for SVM, used in buffer
@@ -238,6 +242,8 @@ int amdgpu_agp = -1; /* auto */ · 55 unmodified lines
238242 int amdgpu_wbrf = -1;
239243 int amdgpu_damage_clips = -1; /* auto */
240244 int amdgpu_umsch_mm_fwlog;
245+int amdgpu_rebar = -1; /* auto */
246+int amdgpu_user_queue = -1;
241247
242248 DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0,
243249 "DRM_UT_CORE",
@@ -1033,11 +1039,13 @@ module_param_named(user_partt_mode, amdgpu_user_partt_mode, uint, 0444); · 789 unmodified lines
10331039
10341040
10351041 /**
1036− * DOC: enforce_isolation (bool)
1037− * enforce process isolation between graphics and compute via using the same reserved vmid.
1042+ * DOC: enforce_isolation (int)
1043+ * enforce process isolation between graphics and compute.
1044+ * (-1 = auto, 0 = disable, 1 = enable, 2 = enable legacy mode, 3 = enable without cleaner shader)
10381045 */
1039−module_param(enforce_isolation, bool, 0444);
1040−MODULE_PARM_DESC(enforce_isolation, "enforce process isolation between graphics and compute . enforce_isolation = on");
1046+module_param_named(enforce_isolation, amdgpu_enforce_isolation, int, 0444);
1047+MODULE_PARM_DESC(enforce_isolation,
1048+"enforce process isolation between graphics and compute. (-1 = auto, 0 = disable, 1 = enable, 2 = enable legacy mode, 3 = enable without cleaner shader)");
10411049
10421050 /**
10431051 * DOC: modeset (int)
@@ -1096,6 +1104,28 @@ MODULE_PARM_DESC(wbrf, · 52 unmodified lines
10961104 "Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)");
10971105 module_param_named(wbrf, amdgpu_wbrf, int, 0444);
10981106
1107+/**
1108+ * DOC: rebar (int)
1109+ * Allow BAR resizing. Disable this to prevent the driver from attempting
1110+ * to resize the BAR if the GPU supports it and there is available MMIO space.
1111+ * Note that this just prevents the driver from resizing the BAR. The BIOS
1112+ * may have already resized the BAR at boot time.
1113+ */
1114+MODULE_PARM_DESC(rebar, "Resizable BAR (-1 = auto (default), 0 = disable, 1 = enable)");
1115+module_param_named(rebar, amdgpu_rebar, int, 0444);
1116+
1117+/**
1118+ * DOC: user_queue (int)
1119+ * Enable user queues on systems that support user queues. Possible values:
1120+ *
1121+ * - -1 = auto (ASIC specific default)
1122+ * - 0 = user queues disabled
1123+ * - 1 = user queues enabled and kernel queues enabled (if supported)
1124+ * - 2 = user queues enabled and kernel queues disabled
1125+ */
1126+MODULE_PARM_DESC(user_queue, "Enable user queues (-1 = auto (default), 0 = disable, 1 = enable, 2 = enable UQs and disable KQs)");
1127+module_param_named(user_queue, amdgpu_user_queue, int, 0444);
1128+
10991129 /* These devices are not supported by amdgpu.
11001130 * They are supported by the mach64, r128, radeon drivers
11011131 */
@@ -2244,6 +2274,10 @@ static void amdgpu_init_debug_options(struct amdgpu_device *adev) · 1142 unmodified lines
22442274 pr_info("debug: use vram for smu pool\n");
22452275 adev->pm.smu_debug_mask |= SMU_DEBUG_POOL_USE_VRAM;
22462276 }
2277+ if (amdgpu_debug_mask & AMDGPU_DEBUG_VM_USERPTR) {
2278+ pr_info("debug: VM mode debug for userptr is enabled\n");
2279+ adev->debug_vm_userptr = true;
2280+ }
22472281 }
22482282
22492283 static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags)
@@ -2615,13 +2649,8 @@ static int amdgpu_pmops_freeze(struct device *dev) · 365 unmodified lines
26152649 static int amdgpu_pmops_thaw(struct device *dev)
26162650 {
26172651 struct drm_device *drm_dev = dev_get_drvdata(dev);
2618− struct amdgpu_device *adev = drm_to_adev(drm_dev);
2619− int r;
26202652
2621− r = amdgpu_device_resume(drm_dev, true);
2622− adev->in_s4 = false;
2623−
2624− return r;
2653+ return amdgpu_device_resume(drm_dev, true);
26252654 }
26262655
26272656 static int amdgpu_pmops_poweroff(struct device *dev)
@@ -2634,9 +2663,6 @@ static int amdgpu_pmops_poweroff(struct device *dev) · 6 unmodified lines
26342663 static int amdgpu_pmops_restore(struct device *dev)
26352664 {
26362665 struct drm_device *drm_dev = dev_get_drvdata(dev);
2637− struct amdgpu_device *adev = drm_to_adev(drm_dev);
2638−
2639− adev->in_s4 = false;
26402666
26412667 return amdgpu_device_resume(drm_dev, true);
26422668 }
@@ -2708,6 +2734,29 @@ static int amdgpu_runtime_idle_check_display(struct device *dev) · 65 unmodified lines
27082734 return 0;
27092735 }
27102736
2737+static int amdgpu_runtime_idle_check_userq(struct device *dev)
2738+{
2739+ struct pci_dev *pdev = to_pci_dev(dev);
2740+ struct drm_device *drm_dev = pci_get_drvdata(pdev);
2741+ struct amdgpu_device *adev = drm_to_adev(drm_dev);
2742+ struct amdgpu_usermode_queue *queue;
2743+ struct amdgpu_userq_mgr *uqm, *tmp;
2744+ int queue_id;
2745+ int ret = 0;
2746+
2747+ mutex_lock(&adev->userq_mutex);
2748+ list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
2749+ idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
2750+ ret = -EBUSY;
2751+ goto done;
2752+ }
2753+ }
2754+done:
2755+ mutex_unlock(&adev->userq_mutex);
2756+
2757+ return ret;
2758+}
2759+
27112760 static int amdgpu_pmops_runtime_suspend(struct device *dev)
27122761 {
27132762 struct pci_dev *pdev = to_pci_dev(dev);
@@ -2721,6 +2770,9 @@ static int amdgpu_pmops_runtime_suspend(struct device *dev) · 7 unmodified lines
27212770 }
27222771
27232772 ret = amdgpu_runtime_idle_check_display(dev);
2773+ if (ret)
2774+ return ret;
2775+ ret = amdgpu_runtime_idle_check_userq(dev);
27242776 if (ret)
27252777 return ret;
27262778
@@ -2844,12 +2896,30 @@ static int amdgpu_pmops_runtime_idle(struct device *dev) · 117 unmodified lines
28442896 }
28452897
28462898 ret = amdgpu_runtime_idle_check_display(dev);
2899+ if (ret)
2900+ goto done;
28472901
2902+ ret = amdgpu_runtime_idle_check_userq(dev);
2903+done:
28482904 pm_runtime_mark_last_busy(dev);
28492905 pm_runtime_autosuspend(dev);
28502906 return ret;
28512907 }
28522908
2909+static int amdgpu_drm_release(struct inode *inode, struct file *filp)
2910+{
2911+ struct drm_file *file_priv = filp->private_data;
2912+ struct amdgpu_fpriv *fpriv = file_priv->driver_priv;
2913+
2914+ if (fpriv) {
2915+ fpriv->evf_mgr.fd_closing = true;
2916+ amdgpu_userq_mgr_fini(&fpriv->userq_mgr);
2917+ amdgpu_eviction_fence_destroy(&fpriv->evf_mgr);
2918+ }
2919+
2920+ return drm_release(inode, filp);
2921+}
2922+
28532923 long amdgpu_drm_ioctl(struct file *filp,
28542924 unsigned int cmd, unsigned long arg)
28552925 {
@@ -2901,7 +2971,7 @@ static const struct file_operations amdgpu_driver_kms_fops = { · 45 unmodified lines
29012971 .owner = THIS_MODULE,
29022972 .open = drm_open,
29032973 .flush = amdgpu_flush,
2904− .release = drm_release,
2974+ .release = amdgpu_drm_release,
29052975 .unlocked_ioctl = amdgpu_drm_ioctl,
29062976 .mmap = drm_gem_mmap,
29072977 .poll = drm_poll,
@@ -2948,6 +3018,9 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = { · 40 unmodified lines
29483018 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
29493019 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
29503020 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3021+ DRM_IOCTL_DEF_DRV(AMDGPU_USERQ, amdgpu_userq_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3022+ DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_SIGNAL, amdgpu_userq_signal_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3023+ DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
29513024 };
29523025
29533026 static const struct drm_driver amdgpu_kms_driver = {
@@ -3038,6 +3111,10 @@ static int __init amdgpu_init(void) · 84 unmodified lines
30383111 if (r)
30393112 goto error_fence;
30403113
3114+ r = amdgpu_userq_fence_slab_init();
3115+ if (r)
3116+ goto error_fence;
3117+
30413118 DRM_INFO("amdgpu kernel modesetting enabled.\n");
30423119 amdgpu_register_atpx_handler();
30433120 amdgpu_acpi_detect();
@@ -3069,6 +3146,7 @@ static void __exit amdgpu_exit(void) · 25 unmodified lines
30693146 amdgpu_acpi_release();
30703147 amdgpu_sync_fini();
30713148 amdgpu_fence_slab_fini();
3149+ amdgpu_userq_fence_slab_fini();
30723150 mmu_notifier_synchronize();
30733151 amdgpu_xcp_drv_release();
30743152 }
drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.c+233−0
@@ -0,0 +1,233 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright 2024 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+#include <linux/sched.h>
25+#include <drm/drm_exec.h>
26+#include "amdgpu.h"
27+
28+#define work_to_evf_mgr(w, name) container_of(w, struct amdgpu_eviction_fence_mgr, name)
29+#define evf_mgr_to_fpriv(e) container_of(e, struct amdgpu_fpriv, evf_mgr)
30+
31+static const char *
32+amdgpu_eviction_fence_get_driver_name(struct dma_fence *fence)
33+{
34+ return "amdgpu_eviction_fence";
35+}
36+
37+static const char *
38+amdgpu_eviction_fence_get_timeline_name(struct dma_fence *f)
39+{
40+ struct amdgpu_eviction_fence *ef;
41+
42+ ef = container_of(f, struct amdgpu_eviction_fence, base);
43+ return ef->timeline_name;
44+}
45+
46+int
47+amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr,
48+ struct drm_exec *exec)
49+{
50+ struct amdgpu_eviction_fence *old_ef, *new_ef;
51+ struct drm_gem_object *obj;
52+ unsigned long index;
53+ int ret;
54+
55+ if (evf_mgr->ev_fence &&
56+ !dma_fence_is_signaled(&evf_mgr->ev_fence->base))
57+ return 0;
58+ /*
59+ * Steps to replace eviction fence:
60+ * * lock all objects in exec (caller)
61+ * * create a new eviction fence
62+ * * update new eviction fence in evf_mgr
63+ * * attach the new eviction fence to BOs
64+ * * release the old fence
65+ * * unlock the objects (caller)
66+ */
67+ new_ef = amdgpu_eviction_fence_create(evf_mgr);
68+ if (!new_ef) {
69+ DRM_ERROR("Failed to create new eviction fence\n");
70+ return -ENOMEM;
71+ }
72+
73+ /* Update the eviction fence now */
74+ spin_lock(&evf_mgr->ev_fence_lock);
75+ old_ef = evf_mgr->ev_fence;
76+ evf_mgr->ev_fence = new_ef;
77+ spin_unlock(&evf_mgr->ev_fence_lock);
78+
79+ /* Attach the new fence */
80+ drm_exec_for_each_locked_object(exec, index, obj) {
81+ struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
82+
83+ if (!bo)
84+ continue;
85+ ret = amdgpu_eviction_fence_attach(evf_mgr, bo);
86+ if (ret) {
87+ DRM_ERROR("Failed to attch new eviction fence\n");
88+ goto free_err;
89+ }
90+ }
91+
92+ /* Free old fence */
93+ if (old_ef)
94+ dma_fence_put(&old_ef->base);
95+ return 0;
96+
97+free_err:
98+ kfree(new_ef);
99+ return ret;
100+}
101+
102+static void
103+amdgpu_eviction_fence_suspend_worker(struct work_struct *work)
104+{
105+ struct amdgpu_eviction_fence_mgr *evf_mgr = work_to_evf_mgr(work, suspend_work.work);
106+ struct amdgpu_fpriv *fpriv = evf_mgr_to_fpriv(evf_mgr);
107+ struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
108+ struct amdgpu_eviction_fence *ev_fence;
109+
110+ mutex_lock(&uq_mgr->userq_mutex);
111+ ev_fence = evf_mgr->ev_fence;
112+ if (!ev_fence)
113+ goto unlock;
114+
115+ amdgpu_userq_evict(uq_mgr, ev_fence);
116+
117+unlock:
118+ mutex_unlock(&uq_mgr->userq_mutex);
119+}
120+
121+static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f)
122+{
123+ struct amdgpu_eviction_fence_mgr *evf_mgr;
124+ struct amdgpu_eviction_fence *ev_fence;
125+
126+ if (!f)
127+ return true;
128+
129+ ev_fence = to_ev_fence(f);
130+ evf_mgr = ev_fence->evf_mgr;
131+
132+ schedule_delayed_work(&evf_mgr->suspend_work, 0);
133+ return true;
134+}
135+
136+static const struct dma_fence_ops amdgpu_eviction_fence_ops = {
137+ .use_64bit_seqno = true,
138+ .get_driver_name = amdgpu_eviction_fence_get_driver_name,
139+ .get_timeline_name = amdgpu_eviction_fence_get_timeline_name,
140+ .enable_signaling = amdgpu_eviction_fence_enable_signaling,
141+};
142+
143+void amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr,
144+ struct amdgpu_eviction_fence *ev_fence)
145+{
146+ spin_lock(&evf_mgr->ev_fence_lock);
147+ dma_fence_signal(&ev_fence->base);
148+ spin_unlock(&evf_mgr->ev_fence_lock);
149+}
150+
151+struct amdgpu_eviction_fence *
152+amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr)
153+{
154+ struct amdgpu_eviction_fence *ev_fence;
155+
156+ ev_fence = kzalloc(sizeof(*ev_fence), GFP_KERNEL);
157+ if (!ev_fence)
158+ return NULL;
159+
160+ ev_fence->evf_mgr = evf_mgr;
161+ get_task_comm(ev_fence->timeline_name, current);
162+ spin_lock_init(&ev_fence->lock);
163+ dma_fence_init(&ev_fence->base, &amdgpu_eviction_fence_ops,
164+ &ev_fence->lock, evf_mgr->ev_fence_ctx,
165+ atomic_inc_return(&evf_mgr->ev_fence_seq));
166+ return ev_fence;
167+}
168+
169+void amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr)
170+{
171+ struct amdgpu_eviction_fence *ev_fence;
172+
173+ /* Wait for any pending work to execute */
174+ flush_delayed_work(&evf_mgr->suspend_work);
175+
176+ spin_lock(&evf_mgr->ev_fence_lock);
177+ ev_fence = evf_mgr->ev_fence;
178+ spin_unlock(&evf_mgr->ev_fence_lock);
179+
180+ if (!ev_fence)
181+ return;
182+
183+ dma_fence_wait(&ev_fence->base, false);
184+
185+ /* Last unref of ev_fence */
186+ dma_fence_put(&evf_mgr->ev_fence->base);
187+}
188+
189+int amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr,
190+ struct amdgpu_bo *bo)
191+{
192+ struct amdgpu_eviction_fence *ev_fence;
193+ struct dma_resv *resv = bo->tbo.base.resv;
194+ int ret;
195+
196+ if (!resv)
197+ return 0;
198+
199+ ret = dma_resv_reserve_fences(resv, 1);
200+ if (ret) {
201+ DRM_DEBUG_DRIVER("Failed to resv fence space\n");
202+ return ret;
203+ }
204+
205+ spin_lock(&evf_mgr->ev_fence_lock);
206+ ev_fence = evf_mgr->ev_fence;
207+ if (ev_fence)
208+ dma_resv_add_fence(resv, &ev_fence->base, DMA_RESV_USAGE_BOOKKEEP);
209+ spin_unlock(&evf_mgr->ev_fence_lock);
210+
211+ return 0;
212+}
213+
214+void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr,
215+ struct amdgpu_bo *bo)
216+{
217+ struct dma_fence *stub = dma_fence_get_stub();
218+
219+ dma_resv_replace_fences(bo->tbo.base.resv, evf_mgr->ev_fence_ctx,
220+ stub, DMA_RESV_USAGE_BOOKKEEP);
221+ dma_fence_put(stub);
222+}
223+
224+int amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr)
225+{
226+ /* This needs to be done one time per open */
227+ atomic_set(&evf_mgr->ev_fence_seq, 0);
228+ evf_mgr->ev_fence_ctx = dma_fence_context_alloc(1);
229+ spin_lock_init(&evf_mgr->ev_fence_lock);
230+
231+ INIT_DELAYED_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker);
232+ return 0;
233+}
drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.h+69−0
@@ -0,0 +1,69 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright 2023 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+
25+#ifndef AMDGPU_EV_FENCE_H_
26+#define AMDGPU_EV_FENCE_H_
27+
28+struct amdgpu_eviction_fence {
29+ struct dma_fence base;
30+ spinlock_t lock;
31+ char timeline_name[TASK_COMM_LEN];
32+ struct amdgpu_eviction_fence_mgr *evf_mgr;
33+};
34+
35+struct amdgpu_eviction_fence_mgr {
36+ u64 ev_fence_ctx;
37+ atomic_t ev_fence_seq;
38+ spinlock_t ev_fence_lock;
39+ struct amdgpu_eviction_fence *ev_fence;
40+ struct delayed_work suspend_work;
41+ uint8_t fd_closing;
42+};
43+
44+/* Eviction fence helper functions */
45+struct amdgpu_eviction_fence *
46+amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr);
47+
48+void
49+amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr);
50+
51+int
52+amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr,
53+ struct amdgpu_bo *bo);
54+
55+void
56+amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr,
57+ struct amdgpu_bo *bo);
58+
59+int
60+amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr);
61+
62+void
63+amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr,
64+ struct amdgpu_eviction_fence *ev_fence);
65+
66+int
67+amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr,
68+ struct drm_exec *exec);
69+#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c+166−13
@@ -36,6 +36,7 @@ · 35 unmodified lines
3636 #include <drm/drm_exec.h>
3737 #include <drm/drm_gem_ttm_helper.h>
3838 #include <drm/ttm/ttm_tt.h>
39+#include <drm/drm_syncobj.h>
3940
4041 #include "amdgpu.h"
4142 #include "amdgpu_display.h"
@@ -44,6 +45,114 @@ · 2 unmodified lines
4445 #include "amdgpu_xgmi.h"
4546 #include "amdgpu_vm.h"
4647
48+static int
49+amdgpu_gem_add_input_fence(struct drm_file *filp,
50+ uint64_t syncobj_handles_array,
51+ uint32_t num_syncobj_handles)
52+{
53+ struct dma_fence *fence;
54+ uint32_t *syncobj_handles;
55+ int ret, i;
56+
57+ if (!num_syncobj_handles)
58+ return 0;
59+
60+ syncobj_handles = memdup_user(u64_to_user_ptr(syncobj_handles_array),
61+ sizeof(uint32_t) * num_syncobj_handles);
62+ if (IS_ERR(syncobj_handles))
63+ return PTR_ERR(syncobj_handles);
64+
65+ for (i = 0; i < num_syncobj_handles; i++) {
66+
67+ if (!syncobj_handles[i]) {
68+ ret = -EINVAL;
69+ goto free_memdup;
70+ }
71+
72+ ret = drm_syncobj_find_fence(filp, syncobj_handles[i], 0, 0, &fence);
73+ if (ret)
74+ goto free_memdup;
75+
76+ dma_fence_wait(fence, false);
77+
78+ /* TODO: optimize async handling */
79+ dma_fence_put(fence);
80+ }
81+
82+free_memdup:
83+ kfree(syncobj_handles);
84+ return ret;
85+}
86+
87+static int
88+amdgpu_gem_update_timeline_node(struct drm_file *filp,
89+ uint32_t syncobj_handle,
90+ uint64_t point,
91+ struct drm_syncobj **syncobj,
92+ struct dma_fence_chain **chain)
93+{
94+ if (!syncobj_handle)
95+ return 0;
96+
97+ /* Find the sync object */
98+ *syncobj = drm_syncobj_find(filp, syncobj_handle);
99+ if (!*syncobj)
100+ return -ENOENT;
101+
102+ if (!point)
103+ return 0;
104+
105+ /* Allocate the chain node */
106+ *chain = dma_fence_chain_alloc();
107+ if (!*chain) {
108+ drm_syncobj_put(*syncobj);
109+ return -ENOMEM;
110+ }
111+
112+ return 0;
113+}
114+
115+static void
116+amdgpu_gem_update_bo_mapping(struct drm_file *filp,
117+ struct amdgpu_bo_va *bo_va,
118+ uint32_t operation,
119+ uint64_t point,
120+ struct dma_fence *fence,
121+ struct drm_syncobj *syncobj,
122+ struct dma_fence_chain *chain)
123+{
124+ struct amdgpu_bo *bo = bo_va ? bo_va->base.bo : NULL;
125+ struct amdgpu_fpriv *fpriv = filp->driver_priv;
126+ struct amdgpu_vm *vm = &fpriv->vm;
127+ struct dma_fence *last_update;
128+
129+ if (!syncobj)
130+ return;
131+
132+ /* Find the last update fence */
133+ switch (operation) {
134+ case AMDGPU_VA_OP_MAP:
135+ case AMDGPU_VA_OP_REPLACE:
136+ if (bo && (bo->tbo.base.resv == vm->root.bo->tbo.base.resv))
137+ last_update = vm->last_update;
138+ else
139+ last_update = bo_va->last_pt_update;
140+ break;
141+ case AMDGPU_VA_OP_UNMAP:
142+ case AMDGPU_VA_OP_CLEAR:
143+ last_update = fence;
144+ break;
145+ default:
146+ return;
147+ }
148+
149+ /* Add fence to timeline */
150+ if (!point)
151+ drm_syncobj_replace_fence(syncobj, last_update);
152+ else
153+ drm_syncobj_add_point(syncobj, chain, last_update, point);
154+}
155+
47156 static vm_fault_t amdgpu_gem_fault(struct vm_fault *vmf)
48157 {
49158 struct ttm_buffer_object *bo = vmf->vma->vm_private_data;
@@ -184,6 +293,15 @@ static int amdgpu_gem_object_open(struct drm_gem_object *obj, · 134 unmodified lines
184293 bo_va = amdgpu_vm_bo_add(adev, vm, abo);
185294 else
186295 ++bo_va->ref_count;
296+
297+ /* attach gfx eviction fence */
298+ r = amdgpu_eviction_fence_attach(&fpriv->evf_mgr, abo);
299+ if (r) {
300+ DRM_DEBUG_DRIVER("Failed to attach eviction fence to BO\n");
301+ amdgpu_bo_unreserve(abo);
302+ return r;
303+ }
304+
187305 amdgpu_bo_unreserve(abo);
188306
189307 /* Validate and add eviction fence to DMABuf imports with dynamic
@@ -247,6 +365,9 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj, · 57 unmodified lines
247365 goto out_unlock;
248366 }
249367
368+ if (!amdgpu_vm_is_bo_always_valid(vm, bo))
369+ amdgpu_eviction_fence_detach(&fpriv->evf_mgr, bo);
370+
250371 bo_va = amdgpu_vm_bo_find(vm, bo);
251372 if (!bo_va || --bo_va->ref_count)
252373 goto out_unlock;
@@ -321,10 +442,6 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data, · 68 unmodified lines
321442 uint32_t handle, initial_domain;
322443 int r;
323444
324− /* reject DOORBELLs until userspace code to use it is available */
325− if (args->in.domains & AMDGPU_GEM_DOMAIN_DOORBELL)
326− return -EINVAL;
327−
328445 /* reject invalid gem flags */
329446 if (flags & ~(AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED |
330447 AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
@@ -638,18 +755,23 @@ out: · 307 unmodified lines
638755 *
639756 * Update the bo_va directly after setting its address. Errors are not
640757 * vital here, so they are not reported back to userspace.
758+ *
759+ * Returns resulting fence if freed BO(s) got cleared from the PT.
760+ * otherwise stub fence in case of error.
641761 */
642−static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
643− struct amdgpu_vm *vm,
644− struct amdgpu_bo_va *bo_va,
645− uint32_t operation)
762+static struct dma_fence *
763+amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
764+ struct amdgpu_vm *vm,
765+ struct amdgpu_bo_va *bo_va,
766+ uint32_t operation)
646767 {
768+ struct dma_fence *fence = dma_fence_get_stub();
647769 int r;
648770
649771 if (!amdgpu_vm_ready(vm))
650− return;
772+ return fence;
651773
652− r = amdgpu_vm_clear_freed(adev, vm, NULL);
774+ r = amdgpu_vm_clear_freed(adev, vm, &fence);
653775 if (r)
654776 goto error;
655777
@@ -665,6 +787,8 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 9 unmodified lines
665787 error:
666788 if (r && r != -ERESTARTSYS)
667789 DRM_ERROR("Couldn't update BO_VA (%d)\n", r);
790+
791+ return fence;
668792 }
669793
670794 /**
@@ -713,6 +837,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 42 unmodified lines
713837 struct amdgpu_fpriv *fpriv = filp->driver_priv;
714838 struct amdgpu_bo *abo;
715839 struct amdgpu_bo_va *bo_va;
840+ struct drm_syncobj *timeline_syncobj = NULL;
841+ struct dma_fence_chain *timeline_chain = NULL;
842+ struct dma_fence *fence;
716843 struct drm_exec exec;
717844 uint64_t va_flags;
718845 uint64_t vm_size;
@@ -774,6 +901,12 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 55 unmodified lines
774901 abo = NULL;
775902 }
776903
904+ r = amdgpu_gem_add_input_fence(filp,
905+ args->input_fence_syncobj_handles,
906+ args->num_syncobj_handles);
907+ if (r)
908+ goto error_put_gobj;
909+
777910 drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT |
778911 DRM_EXEC_IGNORE_DUPLICATES, 0);
779912 drm_exec_until_all_locked(&exec) {
@@ -802,6 +935,14 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 22 unmodified lines
802935 bo_va = NULL;
803936 }
804937
938+ r = amdgpu_gem_update_timeline_node(filp,
939+ args->vm_timeline_syncobj_out,
940+ args->vm_timeline_point,
941+ &timeline_syncobj,
942+ &timeline_chain);
943+ if (r)
944+ goto error;
945+
805946 switch (args->operation) {
806947 case AMDGPU_VA_OP_MAP:
807948 va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
@@ -827,12 +968,24 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 19 unmodified lines
827968 default:
828969 break;
829970 }
830− if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm)
831− amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
832− args->operation);
971+ if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm) {
972+ fence = amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
973+ args->operation);
974+
975+ if (timeline_syncobj)
976+ amdgpu_gem_update_bo_mapping(filp, bo_va,
977+ args->operation,
978+ args->vm_timeline_point,
979+ fence, timeline_syncobj,
980+ timeline_chain);
981+ else
982+ dma_fence_put(fence);
983+
984+ }
833985
834986 error:
835987 drm_exec_fini(&exec);
988+error_put_gobj:
836989 drm_gem_object_put(gobj);
837990 return r;
838991 }
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c+150−36
@@ -33,6 +33,7 @@ · 32 unmodified lines
3333 #include "amdgpu_reset.h"
3434 #include "amdgpu_xcp.h"
3535 #include "amdgpu_xgmi.h"
36+#include "nvd.h"
3637
3738 /* delay 0.1 second to enable gfx off feature */
3839 #define GFX_OFF_DELAY_ENABLE msecs_to_jiffies(100)
@@ -74,14 +75,15 @@ bool amdgpu_gfx_is_mec_queue_enabled(struct amdgpu_device *adev, · 35 unmodified lines
7475 adev->gfx.mec_bitmap[xcc_id].queue_bitmap);
7576 }
7677
77−int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev,
78− int me, int pipe, int queue)
78+static int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev,
79+ int me, int pipe, int queue)
7980 {
81+ int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
8082 int bit = 0;
8183
8284 bit += me * adev->gfx.me.num_pipe_per_me
83− * adev->gfx.me.num_queue_per_pipe;
84− bit += pipe * adev->gfx.me.num_queue_per_pipe;
85+ * num_queue_per_pipe;
86+ bit += pipe * num_queue_per_pipe;
8587 bit += queue;
8688
8789 return bit;
@@ -238,8 +240,8 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 150 unmodified lines
238240 {
239241 int i, queue, pipe;
240242 bool multipipe_policy = amdgpu_gfx_is_graphics_multipipe_capable(adev);
241− int max_queues_per_me = adev->gfx.me.num_pipe_per_me *
242− adev->gfx.me.num_queue_per_pipe;
243+ int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
244+ int max_queues_per_me = adev->gfx.me.num_pipe_per_me * num_queue_per_pipe;
243245
244246 if (multipipe_policy) {
245247 /* policy: amdgpu owns the first queue per pipe at this stage
@@ -247,9 +249,9 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 1 unmodified line
247249 for (i = 0; i < max_queues_per_me; i++) {
248250 pipe = i % adev->gfx.me.num_pipe_per_me;
249251 queue = (i / adev->gfx.me.num_pipe_per_me) %
250− adev->gfx.me.num_queue_per_pipe;
252+ num_queue_per_pipe;
251253
252− set_bit(pipe * adev->gfx.me.num_queue_per_pipe + queue,
254+ set_bit(pipe * num_queue_per_pipe + queue,
253255 adev->gfx.me.queue_bitmap);
254256 }
255257 } else {
@@ -258,8 +260,9 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 2 unmodified lines
258260 }
259261
260262 /* update the number of active graphics rings */
261− adev->gfx.num_gfx_rings =
262− bitmap_weight(adev->gfx.me.queue_bitmap, AMDGPU_MAX_GFX_QUEUES);
263+ if (adev->gfx.num_gfx_rings)
264+ adev->gfx.num_gfx_rings =
265+ bitmap_weight(adev->gfx.me.queue_bitmap, AMDGPU_MAX_GFX_QUEUES);
263266 }
264267
265268 static int amdgpu_gfx_kiq_acquire(struct amdgpu_device *adev,
@@ -1351,6 +1354,10 @@ static ssize_t amdgpu_gfx_get_current_compute_partition(struct device *dev, · 1085 unmodified lines
13511354 struct amdgpu_device *adev = drm_to_adev(ddev);
13521355 int mode;
13531356
1357+ /* Only minimal precaution taken to reject requests while in reset.*/
1358+ if (amdgpu_in_reset(adev))
1359+ return -EPERM;
1360+
13541361 mode = amdgpu_xcp_query_partition_mode(adev->xcp_mgr,
13551362 AMDGPU_XCP_FL_NONE);
13561363
@@ -1394,8 +1401,14 @@ static ssize_t amdgpu_gfx_set_compute_partition(struct device *dev, · 37 unmodified lines
13941401 return -EINVAL;
13951402 }
13961403
1404+ /* Don't allow a switch while under reset */
1405+ if (!down_read_trylock(&adev->reset_domain->sem))
1406+ return -EPERM;
1407+
13971408 ret = amdgpu_xcp_switch_partition_mode(adev->xcp_mgr, mode);
13981409
1410+ up_read(&adev->reset_domain->sem);
1411+
13991412 if (ret)
14001413 return ret;
14011414
@@ -1466,6 +1479,8 @@ static int amdgpu_gfx_run_cleaner_shader_job(struct amdgpu_ring *ring) · 64 unmodified lines
14661479 goto err;
14671480
14681481 job->enforce_isolation = true;
1482+ /* always run the cleaner shader */
1483+ job->run_cleaner_shader = true;
14691484
14701485 ib = &job->ibs[0];
14711486 for (i = 0; i <= ring->funcs->align_mask; ++i)
@@ -1552,6 +1567,9 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 80 unmodified lines
15521567 if (adev->in_suspend && !adev->in_runpm)
15531568 return -EPERM;
15541569
1570+ if (adev->gfx.disable_kq)
1571+ return -EPERM;
1572+
15551573 ret = kstrtol(buf, 0, &value);
15561574
15571575 if (ret)
@@ -1594,7 +1612,8 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 36 unmodified lines
15941612 * Provides the sysfs read interface to get the current settings of the 'enforce_isolation'
15951613 * feature for each GPU partition. Reading from the 'enforce_isolation'
15961614 * sysfs file returns the isolation settings for all partitions, where '0'
1597− * indicates disabled and '1' indicates enabled.
1615+ * indicates disabled, '1' indicates enabled, and '2' indicates enabled in legacy mode,
1616+ * and '3' indicates enabled without cleaner shader.
15981617 *
15991618 * Return: The number of bytes read from the sysfs file.
16001619 */
@@ -1629,9 +1648,12 @@ static ssize_t amdgpu_gfx_get_enforce_isolation(struct device *dev, · 28 unmodified lines
16291648 * @count: The size of the input data
16301649 *
16311650 * This function allows control over the 'enforce_isolation' feature, which
1632− * serializes access to the graphics engine. Writing '1' or '0' to the
1633− * 'enforce_isolation' sysfs file enables or disables process isolation for
1634− * each partition. The input should specify the setting for all partitions.
1651+ * serializes access to the graphics engine. Writing '0' to disable, '1' to
1652+ * enable isolation with cleaner shader, '2' to enable legacy isolation without
1653+ * cleaner shader, or '3' to enable process isolation without submitting the
1654+ * cleaner shader to the 'enforce_isolation' sysfs file sets the isolation mode
1655+ * for each partition. The input should specify the setting for all
1656+ * partitions.
16351657 *
16361658 * Return: The number of bytes written to the sysfs file.
16371659 */
@@ -1668,13 +1690,34 @@ static ssize_t amdgpu_gfx_set_enforce_isolation(struct device *dev, · 30 unmodified lines
16681690 return -EINVAL;
16691691
16701692 for (i = 0; i < num_partitions; i++) {
1671− if (partition_values[i] != 0 && partition_values[i] != 1)
1693+ if (partition_values[i] != 0 &&
1694+ partition_values[i] != 1 &&
1695+ partition_values[i] != 2 &&
1696+ partition_values[i] != 3)
16721697 return -EINVAL;
16731698 }
16741699
16751700 mutex_lock(&adev->enforce_isolation_mutex);
1676− for (i = 0; i < num_partitions; i++)
1677− adev->enforce_isolation[i] = partition_values[i];
1701+ for (i = 0; i < num_partitions; i++) {
1702+ switch (partition_values[i]) {
1703+ case 0:
1704+ default:
1705+ adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE;
1706+ break;
1707+ case 1:
1708+ adev->enforce_isolation[i] =
1709+ AMDGPU_ENFORCE_ISOLATION_ENABLE;
1710+ break;
1711+ case 2:
1712+ adev->enforce_isolation[i] =
1713+ AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY;
1714+ break;
1715+ case 3:
1716+ adev->enforce_isolation[i] =
1717+ AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER;
1718+ break;
1719+ }
1720+ }
16781721 mutex_unlock(&adev->enforce_isolation_mutex);
16791722
16801723 amdgpu_mes_update_enforce_isolation(adev);
@@ -1923,39 +1966,41 @@ void amdgpu_gfx_cleaner_shader_init(struct amdgpu_device *adev, · 242 unmodified lines
19231966 static void amdgpu_gfx_kfd_sch_ctrl(struct amdgpu_device *adev, u32 idx,
19241967 bool enable)
19251968 {
1926− mutex_lock(&adev->gfx.kfd_sch_mutex);
1969+ mutex_lock(&adev->gfx.userq_sch_mutex);
19271970
19281971 if (enable) {
19291972 /* If the count is already 0, it means there's an imbalance bug somewhere.
19301973 * Note that the bug may be in a different caller than the one which triggers the
19311974 * WARN_ON_ONCE.
19321975 */
1933− if (WARN_ON_ONCE(adev->gfx.kfd_sch_req_count[idx] == 0)) {
1976+ if (WARN_ON_ONCE(adev->gfx.userq_sch_req_count[idx] == 0)) {
19341977 dev_err(adev->dev, "Attempted to enable KFD scheduler when reference count is already zero\n");
19351978 goto unlock;
19361979 }
19371980
1938− adev->gfx.kfd_sch_req_count[idx]--;
1981+ adev->gfx.userq_sch_req_count[idx]--;
19391982
1940− if (adev->gfx.kfd_sch_req_count[idx] == 0 &&
1941− adev->gfx.kfd_sch_inactive[idx]) {
1983+ if (adev->gfx.userq_sch_req_count[idx] == 0 &&
1984+ adev->gfx.userq_sch_inactive[idx]) {
19421985 schedule_delayed_work(&adev->gfx.enforce_isolation[idx].work,
19431986 msecs_to_jiffies(adev->gfx.enforce_isolation_time[idx]));
19441987 }
19451988 } else {
1946− if (adev->gfx.kfd_sch_req_count[idx] == 0) {
1989+ if (adev->gfx.userq_sch_req_count[idx] == 0) {
19471990 cancel_delayed_work_sync(&adev->gfx.enforce_isolation[idx].work);
1948− if (!adev->gfx.kfd_sch_inactive[idx]) {
1949− amdgpu_amdkfd_stop_sched(adev, idx);
1950− adev->gfx.kfd_sch_inactive[idx] = true;
1991+ if (!adev->gfx.userq_sch_inactive[idx]) {
1992+ amdgpu_userq_stop_sched_for_enforce_isolation(adev, idx);
1993+ if (adev->kfd.init_complete)
1994+ amdgpu_amdkfd_stop_sched(adev, idx);
1995+ adev->gfx.userq_sch_inactive[idx] = true;
19511996 }
19521997 }
19531998
1954− adev->gfx.kfd_sch_req_count[idx]++;
1999+ adev->gfx.userq_sch_req_count[idx]++;
19552000 }
19562001
19572002 unlock:
1958− mutex_unlock(&adev->gfx.kfd_sch_mutex);
2003+ mutex_unlock(&adev->gfx.userq_sch_mutex);
19592004 }
19602005
19612006 /**
@@ -2000,12 +2045,13 @@ void amdgpu_gfx_enforce_isolation_handler(struct work_struct *work) · 38 unmodified lines
20002045 msecs_to_jiffies(1));
20012046 } else {
20022047 /* Tell KFD to resume the runqueue */
2003− if (adev->kfd.init_complete) {
2004− WARN_ON_ONCE(!adev->gfx.kfd_sch_inactive[idx]);
2005− WARN_ON_ONCE(adev->gfx.kfd_sch_req_count[idx]);
2048+ WARN_ON_ONCE(!adev->gfx.userq_sch_inactive[idx]);
2049+ WARN_ON_ONCE(adev->gfx.userq_sch_req_count[idx]);
2050+
2051+ amdgpu_userq_start_sched_for_enforce_isolation(adev, idx);
2052+ if (adev->kfd.init_complete)
20062053 amdgpu_amdkfd_start_sched(adev, idx);
2007− adev->gfx.kfd_sch_inactive[idx] = false;
2008− }
2054+ adev->gfx.userq_sch_inactive[idx] = false;
20092055 }
20102056 mutex_unlock(&adev->enforce_isolation_mutex);
20112057 }
@@ -2029,7 +2075,7 @@ amdgpu_gfx_enforce_isolation_wait_for_kfd(struct amdgpu_device *adev, · 17 unmodified lines
20292075 bool wait = false;
20302076
20312077 mutex_lock(&adev->enforce_isolation_mutex);
2032− if (adev->enforce_isolation[idx]) {
2078+ if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) {
20332079 /* set the initial values if nothing is set */
20342080 if (!adev->gfx.enforce_isolation_jiffies[idx]) {
20352081 adev->gfx.enforce_isolation_jiffies[idx] = jiffies;
@@ -2096,7 +2142,7 @@ void amdgpu_gfx_enforce_isolation_ring_begin_use(struct amdgpu_ring *ring) · 60 unmodified lines
20962142 amdgpu_gfx_enforce_isolation_wait_for_kfd(adev, idx);
20972143
20982144 mutex_lock(&adev->enforce_isolation_mutex);
2099− if (adev->enforce_isolation[idx]) {
2145+ if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) {
21002146 if (adev->kfd.init_complete)
21012147 sched_work = true;
21022148 }
@@ -2133,7 +2179,7 @@ void amdgpu_gfx_enforce_isolation_ring_end_use(struct amdgpu_ring *ring) · 30 unmodified lines
21332179 return;
21342180
21352181 mutex_lock(&adev->enforce_isolation_mutex);
2136− if (adev->enforce_isolation[idx]) {
2182+ if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) {
21372183 if (adev->kfd.init_complete)
21382184 sched_work = true;
21392185 }
@@ -2217,6 +2263,74 @@ void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring) · 77 unmodified lines
22172263 schedule_delayed_work(&ring->adev->gfx.idle_work, GFX_PROFILE_IDLE_TIMEOUT);
22182264 }
22192265
2266+/**
2267+ * amdgpu_gfx_csb_preamble_start - Set CSB preamble start
2268+ *
2269+ * @buffer: This is an output variable that gets the PACKET3 preamble setup.
2270+ *
2271+ * Return:
2272+ * return the latest index.
2273+ */
2274+u32 amdgpu_gfx_csb_preamble_start(volatile u32 *buffer)
2275+{
2276+ u32 count = 0;
2277+
2278+ buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
2279+ buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
2280+
2281+ buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
2282+ buffer[count++] = cpu_to_le32(0x80000000);
2283+ buffer[count++] = cpu_to_le32(0x80000000);
2284+
2285+ return count;
2286+}
2287+
2288+/**
2289+ * amdgpu_gfx_csb_data_parser - Parser CS data
2290+ *
2291+ * @adev: amdgpu_device pointer used to get the CS data and other gfx info.
2292+ * @buffer: This is an output variable that gets the PACKET3 preamble end.
2293+ * @count: Index to start set the preemble end.
2294+ *
2295+ * Return:
2296+ * return the latest index.
2297+ */
2298+u32 amdgpu_gfx_csb_data_parser(struct amdgpu_device *adev, volatile u32 *buffer, u32 count)
2299+{
2300+ const struct cs_section_def *sect = NULL;
2301+ const struct cs_extent_def *ext = NULL;
2302+ u32 i;
2303+
2304+ for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
2305+ for (ext = sect->section; ext->extent != NULL; ++ext) {
2306+ if (sect->id == SECT_CONTEXT) {
2307+ buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
2308+ buffer[count++] = cpu_to_le32(ext->reg_index - PACKET3_SET_CONTEXT_REG_START);
2309+
2310+ for (i = 0; i < ext->reg_count; i++)
2311+ buffer[count++] = cpu_to_le32(ext->extent[i]);
2312+ }
2313+ }
2314+ }
2315+
2316+ return count;
2317+}
2318+
2319+/**
2320+ * amdgpu_gfx_csb_preamble_end - Set CSB preamble end
2321+ *
2322+ * @buffer: This is an output variable that gets the PACKET3 preamble end.
2323+ * @count: Index to start set the preemble end.
2324+ */
2325+void amdgpu_gfx_csb_preamble_end(volatile u32 *buffer, u32 count)
2326+{
2327+ buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
2328+ buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
2329+
2330+ buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
2331+ buffer[count++] = cpu_to_le32(0);
2332+}
2333+
22202334 /*
22212335 * debugfs for to enable/disable gfx job submission to specific core.
22222336 */
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h+55−7
@@ -170,10 +170,46 @@ struct amdgpu_kiq { · 169 unmodified lines
170170 #define AMDGPU_GFX_MAX_SE 4
171171 #define AMDGPU_GFX_MAX_SH_PER_SE 2
172172
173+/**
174+ * amdgpu_rb_config - Configure a single Render Backend (RB)
175+ *
176+ * Bad RBs are fused off and there is a harvest register the driver reads to
177+ * determine which RB(s) are fused off so that the driver can configure the
178+ * hardware state so that nothing gets sent to them. There are also user
179+ * harvest registers that the driver can program to disable additional RBs,
180+ * etc., for testing purposes.
181+ */
173182 struct amdgpu_rb_config {
183+ /**
184+ * @rb_backend_disable:
185+ *
186+ * The value captured from register RB_BACKEND_DISABLE indicates if the
187+ * RB backend is disabled or not.
188+ */
174189 uint32_t rb_backend_disable;
190+
191+ /**
192+ * @user_rb_backend_disable:
193+ *
194+ * The value captured from register USER_RB_BACKEND_DISABLE indicates
195+ * if the User RB backend is disabled or not.
196+ */
175197 uint32_t user_rb_backend_disable;
198+
199+ /**
200+ * @raster_config:
201+ *
202+ * To set up all of the states, it is necessary to have two registers
203+ * to keep all of the states. This field holds the first register.
204+ */
176205 uint32_t raster_config;
206+
207+ /**
208+ * @raster_config_1:
209+ *
210+ * To set up all of the states, it is necessary to have two registers
211+ * to keep all of the states. This field holds the second register.
212+ */
177213 uint32_t raster_config_1;
178214 };
179215
@@ -221,6 +257,13 @@ struct amdgpu_gfx_config { · 41 unmodified lines
221257 uint32_t macrotile_mode_array[16];
222258
223259 struct gb_addr_config gb_addr_config_fields;
260+
261+ /**
262+ * @rb_config:
263+ *
264+ * Matrix that keeps all the Render Backend (color and depth buffer
265+ * handling) configuration on the 3D engine.
266+ */
224267 struct amdgpu_rb_config rb_config[AMDGPU_GFX_MAX_SE][AMDGPU_GFX_MAX_SH_PER_SE];
225268
226269 /* gfx configure feature */
@@ -305,7 +348,8 @@ struct amdgpu_gfx_funcs { · 78 unmodified lines
305348 void (*init_spm_golden)(struct amdgpu_device *adev);
306349 void (*update_perfmon_mgcg)(struct amdgpu_device *adev, bool enable);
307350 int (*get_gfx_shadow_info)(struct amdgpu_device *adev,
308− struct amdgpu_gfx_shadow_info *shadow_info);
351+ struct amdgpu_gfx_shadow_info *shadow_info,
352+ bool skip_check);
309353 enum amdgpu_gfx_partition
310354 (*query_partition_mode)(struct amdgpu_device *adev);
311355 int (*switch_partition_mode)(struct amdgpu_device *adev,
@@ -474,9 +518,9 @@ struct amdgpu_gfx { · 162 unmodified lines
474518 bool enable_cleaner_shader;
475519 struct amdgpu_isolation_work enforce_isolation[MAX_XCP];
476520 /* Mutex for synchronizing KFD scheduler operations */
477− struct mutex kfd_sch_mutex;
478− u64 kfd_sch_req_count[MAX_XCP];
479− bool kfd_sch_inactive[MAX_XCP];
521+ struct mutex userq_sch_mutex;
522+ u64 userq_sch_req_count[MAX_XCP];
523+ bool userq_sch_inactive[MAX_XCP];
480524 unsigned long enforce_isolation_jiffies[MAX_XCP];
481525 unsigned long enforce_isolation_time[MAX_XCP];
482526
@@ -484,6 +528,9 @@ struct amdgpu_gfx { · 1 unmodified line
484528 struct delayed_work idle_work;
485529 bool workload_profile_active;
486530 struct mutex workload_profile_mutex;
531+
532+ bool disable_kq;
533+ bool disable_uq;
487534 };
488535
489536 struct amdgpu_gfx_ras_reg_entry {
@@ -503,7 +550,7 @@ struct amdgpu_gfx_ras_mem_id_entry { · 13 unmodified lines
503550 #define amdgpu_gfx_select_se_sh(adev, se, sh, instance, xcc_id) ((adev)->gfx.funcs->select_se_sh((adev), (se), (sh), (instance), (xcc_id)))
504551 #define amdgpu_gfx_select_me_pipe_q(adev, me, pipe, q, vmid, xcc_id) ((adev)->gfx.funcs->select_me_pipe_q((adev), (me), (pipe), (q), (vmid), (xcc_id)))
505552 #define amdgpu_gfx_init_spm_golden(adev) (adev)->gfx.funcs->init_spm_golden((adev))
506−#define amdgpu_gfx_get_gfx_shadow_info(adev, si) ((adev)->gfx.funcs->get_gfx_shadow_info((adev), (si)))
553+#define amdgpu_gfx_get_gfx_shadow_info(adev, si) ((adev)->gfx.funcs->get_gfx_shadow_info((adev), (si), false))
507554
508555 /**
509556 * amdgpu_gfx_create_bitmask - create a bitmask
@@ -550,8 +597,6 @@ bool amdgpu_gfx_is_high_priority_compute_queue(struct amdgpu_device *adev, · 40 unmodified lines
550597 struct amdgpu_ring *ring);
551598 bool amdgpu_gfx_is_high_priority_graphics_queue(struct amdgpu_device *adev,
552599 struct amdgpu_ring *ring);
553−int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev, int me,
554− int pipe, int queue);
555600 bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me,
556601 int pipe, int queue);
557602 void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable);
@@ -597,6 +642,9 @@ void amdgpu_gfx_enforce_isolation_ring_end_use(struct amdgpu_ring *ring); · 39 unmodified lines
597642 void amdgpu_gfx_profile_idle_work_handler(struct work_struct *work);
598643 void amdgpu_gfx_profile_ring_begin_use(struct amdgpu_ring *ring);
599644 void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring);
645+u32 amdgpu_gfx_csb_preamble_start(volatile u32 *buffer);
646+u32 amdgpu_gfx_csb_data_parser(struct amdgpu_device *adev, volatile u32 *buffer, u32 count);
647+void amdgpu_gfx_csb_preamble_end(volatile u32 *buffer, u32 count);
600648
601649 void amdgpu_debugfs_gfx_sched_mask_init(struct amdgpu_device *adev);
602650 void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c+4−0
@@ -1230,6 +1230,10 @@ static ssize_t current_memory_partition_show( · 1229 unmodified lines
12301230 struct amdgpu_device *adev = drm_to_adev(ddev);
12311231 enum amdgpu_memory_partition mode;
12321232
1233+ /* Only minimal precaution taken to reject requests while in reset */
1234+ if (amdgpu_in_reset(adev))
1235+ return -EPERM;
1236+
12331237 mode = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);
12341238 if ((mode >= ARRAY_SIZE(nps_desc)) ||
12351239 (BIT(mode) & AMDGPU_ALL_NPS_MASK) != BIT(mode))
drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.c+20−0
@@ -22,6 +22,7 @@ · 21 unmodified lines
2222 */
2323 #include "amdgpu.h"
2424 #include "amdgpu_ras.h"
25+#include <uapi/linux/kfd_ioctl.h>
2526
2627 int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev)
2728 {
@@ -46,3 +47,22 @@ int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev) · 18 unmodified lines
4647 /* hdp ras follows amdgpu_ras_block_late_init_default for late init */
4748 return 0;
4849 }
50+
51+void amdgpu_hdp_generic_flush(struct amdgpu_device *adev,
52+ struct amdgpu_ring *ring)
53+{
54+ if (!ring || !ring->funcs->emit_wreg) {
55+ WREG32((adev->rmmio_remap.reg_offset +
56+ KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >>
57+ 2,
58+ 0);
59+ if (adev->nbio.funcs->get_memsize)
60+ adev->nbio.funcs->get_memsize(adev);
61+ } else {
62+ amdgpu_ring_emit_wreg(ring,
63+ (adev->rmmio_remap.reg_offset +
64+ KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >>
65+ 2,
66+ 0);
67+ }
68+}
drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.h+2−0
@@ -44,4 +44,6 @@ struct amdgpu_hdp { · 43 unmodified lines
4444 };
4545
4646 int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev);
47+void amdgpu_hdp_generic_flush(struct amdgpu_device *adev,
48+ struct amdgpu_ring *ring);
4749 #endif /* __AMDGPU_HDP_H__ */
drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c+2−2
@@ -163,12 +163,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 162 unmodified lines
163163 init_shadow = false;
164164 }
165165
166− if (!ring->sched.ready && !ring->is_mes_queue) {
166+ if (!ring->sched.ready) {
167167 dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name);
168168 return -EINVAL;
169169 }
170170
171− if (vm && !job->vmid && !ring->is_mes_queue) {
171+ if (vm && !job->vmid) {
172172 dev_err(adev->dev, "VM IB without ID\n");
173173 return -EINVAL;
174174 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c+11−3
@@ -576,8 +576,16 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev) · 575 unmodified lines
576576 INIT_LIST_HEAD(&id_mgr->ids_lru);
577577 id_mgr->reserved_use_count = 0;
578578
579− /* manage only VMIDs not used by KFD */
580− id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
579+ /* for GC <10, SDMA uses MMHUB so use first_kfd_vmid for both GC and MM */
580+ if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(10, 0, 0))
581+ /* manage only VMIDs not used by KFD */
582+ id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
583+ else if (AMDGPU_IS_MMHUB0(i) ||
584+ AMDGPU_IS_MMHUB1(i))
585+ id_mgr->num_ids = 16;
586+ else
587+ /* manage only VMIDs not used by KFD */
588+ id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
581589
582590 /* skip over VMID 0, since it is the system VM */
583591 for (j = 1; j < id_mgr->num_ids; ++j) {
@@ -588,7 +596,7 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev) · 4 unmodified lines
588596 }
589597 /* alloc a default reserved vmid to enforce isolation */
590598 for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) {
591− if (adev->enforce_isolation[i])
599+ if (adev->enforce_isolation[i] != AMDGPU_ENFORCE_ISOLATION_DISABLE)
592600 amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(i));
593601 }
594602 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c+13−2
@@ -25,6 +25,7 @@ · 24 unmodified lines
2525
2626 #include "amdgpu.h"
2727 #include "amdgpu_ih.h"
28+#include "amdgpu_reset.h"
2829
2930 /**
3031 * amdgpu_ih_ring_init - initialize the IH state
@@ -227,13 +228,23 @@ restart_ih: · 196 unmodified lines
227228 ih->rptr &= ih->ptr_mask;
228229 }
229230
230− amdgpu_ih_set_rptr(adev, ih);
231+ if (!ih->overflow)
232+ amdgpu_ih_set_rptr(adev, ih);
233+
231234 wake_up_all(&ih->wait_process);
232235
233236 /* make sure wptr hasn't changed while processing */
234237 wptr = amdgpu_ih_get_wptr(adev, ih);
235238 if (wptr != ih->rptr)
236− goto restart_ih;
239+ if (!ih->overflow)
240+ goto restart_ih;
241+
242+ if (ih->overflow)
243+ if (amdgpu_sriov_runtime(adev))
244+ WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
245+ &adev->virt.flr_work),
246+ "Failed to queue work! at %s",
247+ __func__);
237248
238249 return IRQ_HANDLED;
239250 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h+1−0
@@ -72,6 +72,7 @@ struct amdgpu_ih_ring { · 71 unmodified lines
7272 /* For waiting on IH processing at checkpoint. */
7373 wait_queue_head_t wait_process;
7474 uint64_t processed_timestamp;
75+ bool overflow;
7576 };
7677
7778 /* return true if time stamp t2 is after t1 with 48bit wrap around */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.h+1−0
@@ -78,6 +78,7 @@ struct amdgpu_job { · 77 unmodified lines
7878
7979 /* enforce isolation */
8080 bool enforce_isolation;
81+ bool run_cleaner_shader;
8182
8283 uint32_t num_ibs;
8384 struct amdgpu_ib ibs[];
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c+71−10
@@ -45,6 +45,7 @@ · 44 unmodified lines
4545 #include "amdgpu_ras.h"
4646 #include "amdgpu_reset.h"
4747 #include "amd_pcie.h"
48+#include "amdgpu_userq.h"
4849
4950 void amdgpu_unregister_gpu_instance(struct amdgpu_device *adev)
5051 {
@@ -370,6 +371,26 @@ static int amdgpu_firmware_info(struct drm_amdgpu_info_firmware *fw_info, · 319 unmodified lines
370371 return 0;
371372 }
372373
374+static int amdgpu_userq_metadata_info_gfx(struct amdgpu_device *adev,
375+ struct drm_amdgpu_info *info,
376+ struct drm_amdgpu_info_uq_metadata_gfx *meta)
377+{
378+ int ret = -EOPNOTSUPP;
379+
380+ if (adev->gfx.funcs->get_gfx_shadow_info) {
381+ struct amdgpu_gfx_shadow_info shadow = {};
382+
383+ adev->gfx.funcs->get_gfx_shadow_info(adev, &shadow, true);
384+ meta->shadow_size = shadow.shadow_size;
385+ meta->shadow_alignment = shadow.shadow_alignment;
386+ meta->csa_size = shadow.csa_size;
387+ meta->csa_alignment = shadow.csa_alignment;
388+ ret = 0;
389+ }
390+
391+ return ret;
392+}
393+
373394 static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
374395 struct drm_amdgpu_info *info,
375396 struct drm_amdgpu_info_hw_ip *result)
@@ -387,7 +408,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 11 unmodified lines
387408 case AMDGPU_HW_IP_GFX:
388409 type = AMD_IP_BLOCK_TYPE_GFX;
389410 for (i = 0; i < adev->gfx.num_gfx_rings; i++)
390− if (adev->gfx.gfx_ring[i].sched.ready)
411+ if (adev->gfx.gfx_ring[i].sched.ready &&
412+ !adev->gfx.gfx_ring[i].no_user_submission)
391413 ++num_rings;
392414 ib_start_alignment = 32;
393415 ib_size_alignment = 32;
@@ -395,7 +417,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line
395417 case AMDGPU_HW_IP_COMPUTE:
396418 type = AMD_IP_BLOCK_TYPE_GFX;
397419 for (i = 0; i < adev->gfx.num_compute_rings; i++)
398− if (adev->gfx.compute_ring[i].sched.ready)
420+ if (adev->gfx.compute_ring[i].sched.ready &&
421+ !adev->gfx.compute_ring[i].no_user_submission)
399422 ++num_rings;
400423 ib_start_alignment = 32;
401424 ib_size_alignment = 32;
@@ -403,7 +426,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line
403426 case AMDGPU_HW_IP_DMA:
404427 type = AMD_IP_BLOCK_TYPE_SDMA;
405428 for (i = 0; i < adev->sdma.num_instances; i++)
406− if (adev->sdma.instance[i].ring.sched.ready)
429+ if (adev->sdma.instance[i].ring.sched.ready &&
430+ !adev->sdma.instance[i].ring.no_user_submission)
407431 ++num_rings;
408432 ib_start_alignment = 256;
409433 ib_size_alignment = 4;
@@ -414,7 +438,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 4 unmodified lines
414438 if (adev->uvd.harvest_config & (1 << i))
415439 continue;
416440
417− if (adev->uvd.inst[i].ring.sched.ready)
441+ if (adev->uvd.inst[i].ring.sched.ready &&
442+ !adev->uvd.inst[i].ring.no_user_submission)
418443 ++num_rings;
419444 }
420445 ib_start_alignment = 256;
@@ -423,7 +448,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 2 unmodified lines
423448 case AMDGPU_HW_IP_VCE:
424449 type = AMD_IP_BLOCK_TYPE_VCE;
425450 for (i = 0; i < adev->vce.num_rings; i++)
426− if (adev->vce.ring[i].sched.ready)
451+ if (adev->vce.ring[i].sched.ready &&
452+ !adev->vce.ring[i].no_user_submission)
427453 ++num_rings;
428454 ib_start_alignment = 256;
429455 ib_size_alignment = 4;
@@ -435,7 +461,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 5 unmodified lines
435461 continue;
436462
437463 for (j = 0; j < adev->uvd.num_enc_rings; j++)
438− if (adev->uvd.inst[i].ring_enc[j].sched.ready)
464+ if (adev->uvd.inst[i].ring_enc[j].sched.ready &&
465+ !adev->uvd.inst[i].ring_enc[j].no_user_submission)
439466 ++num_rings;
440467 }
441468 ib_start_alignment = 256;
@@ -447,7 +474,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 5 unmodified lines
447474 if (adev->vcn.harvest_config & (1 << i))
448475 continue;
449476
450− if (adev->vcn.inst[i].ring_dec.sched.ready)
477+ if (adev->vcn.inst[i].ring_dec.sched.ready &&
478+ !adev->vcn.inst[i].ring_dec.no_user_submission)
451479 ++num_rings;
452480 }
453481 ib_start_alignment = 256;
@@ -460,7 +488,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 6 unmodified lines
460488 continue;
461489
462490 for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++)
463− if (adev->vcn.inst[i].ring_enc[j].sched.ready)
491+ if (adev->vcn.inst[i].ring_enc[j].sched.ready &&
492+ !adev->vcn.inst[i].ring_enc[j].no_user_submission)
464493 ++num_rings;
465494 }
466495 ib_start_alignment = 256;
@@ -475,7 +504,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 8 unmodified lines
475504 continue;
476505
477506 for (j = 0; j < adev->jpeg.num_jpeg_rings; j++)
478− if (adev->jpeg.inst[i].ring_dec[j].sched.ready)
507+ if (adev->jpeg.inst[i].ring_dec[j].sched.ready &&
508+ !adev->jpeg.inst[i].ring_dec[j].no_user_submission)
479509 ++num_rings;
480510 }
481511 ib_start_alignment = 256;
@@ -483,7 +513,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line
483513 break;
484514 case AMDGPU_HW_IP_VPE:
485515 type = AMD_IP_BLOCK_TYPE_VPE;
486− if (adev->vpe.ring.sched.ready)
516+ if (adev->vpe.ring.sched.ready &&
517+ !adev->vpe.ring.no_user_submission)
487518 ++num_rings;
488519 ib_start_alignment = 256;
489520 ib_size_alignment = 4;
@@ -978,6 +1009,8 @@ out: · 488 unmodified lines
9781009 }
9791010 }
9801011
1012+ dev_info->userq_ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
1013+
9811014 ret = copy_to_user(out, dev_info,
9821015 min((size_t)size, sizeof(*dev_info))) ? -EFAULT : 0;
9831016 kfree(dev_info);
@@ -1293,6 +1326,22 @@ out: · 309 unmodified lines
12931326 return copy_to_user(out, &gpuvm_fault,
12941327 min((size_t)size, sizeof(gpuvm_fault))) ? -EFAULT : 0;
12951328 }
1329+ case AMDGPU_INFO_UQ_FW_AREAS: {
1330+ struct drm_amdgpu_info_uq_metadata meta_info = {};
1331+
1332+ switch (info->query_hw_ip.type) {
1333+ case AMDGPU_HW_IP_GFX:
1334+ ret = amdgpu_userq_metadata_info_gfx(adev, info, &meta_info.gfx);
1335+ if (ret)
1336+ return ret;
1337+
1338+ ret = copy_to_user(out, &meta_info,
1339+ min((size_t)size, sizeof(meta_info))) ? -EFAULT : 0;
1340+ return 0;
1341+ default:
1342+ return -EINVAL;
1343+ }
1344+ }
12961345 default:
12971346 DRM_DEBUG_KMS("Invalid request %d\n", info->query);
12981347 return -EINVAL;
@@ -1376,8 +1425,16 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · 77 unmodified lines
13761425 mutex_init(&fpriv->bo_list_lock);
13771426 idr_init_base(&fpriv->bo_list_handles, 1);
13781427
1428+ r = amdgpu_eviction_fence_init(&fpriv->evf_mgr);
1429+ if (r)
1430+ goto error_vm;
1431+
13791432 amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev);
13801433
1434+ r = amdgpu_userq_mgr_init(&fpriv->userq_mgr, file_priv, adev);
1435+ if (r)
1436+ DRM_WARN("Can't setup usermode queues, use legacy workload submission only\n");
1437+
13811438 file_priv->driver_priv = fpriv;
13821439 goto out_suspend;
13831440
@@ -1445,6 +1502,10 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 61 unmodified lines
14451502 amdgpu_bo_unreserve(pd);
14461503 }
14471504
1505+ fpriv->evf_mgr.fd_closing = true;
1506+ amdgpu_userq_mgr_fini(&fpriv->userq_mgr);
1507+ amdgpu_eviction_fence_destroy(&fpriv->evf_mgr);
1508+
14481509 amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr);
14491510 amdgpu_vm_fini(adev, &fpriv->vm);
14501511
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c+55−1051
@@ -39,42 +39,6 @@ int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev) · 38 unmodified lines
3939 PAGE_SIZE);
4040 }
4141
42−static int amdgpu_mes_kernel_doorbell_get(struct amdgpu_device *adev,
43− int ip_type, uint64_t *doorbell_index)
44−{
45− unsigned int offset, found;
46− struct amdgpu_mes *mes = &adev->mes;
47−
48− if (ip_type == AMDGPU_RING_TYPE_SDMA)
49− offset = adev->doorbell_index.sdma_engine[0];
50− else
51− offset = 0;
52−
53− found = find_next_zero_bit(mes->doorbell_bitmap, mes->num_mes_dbs, offset);
54− if (found >= mes->num_mes_dbs) {
55− DRM_WARN("No doorbell available\n");
56− return -ENOSPC;
57− }
58−
59− set_bit(found, mes->doorbell_bitmap);
60−
61− /* Get the absolute doorbell index on BAR */
62− *doorbell_index = mes->db_start_dw_offset + found * 2;
63− return 0;
64−}
65−
66−static void amdgpu_mes_kernel_doorbell_free(struct amdgpu_device *adev,
67− uint32_t doorbell_index)
68−{
69− unsigned int old, rel_index;
70− struct amdgpu_mes *mes = &adev->mes;
71−
72− /* Find the relative index of the doorbell in this object */
73− rel_index = (doorbell_index - mes->db_start_dw_offset) / 2;
74− old = test_and_clear_bit(rel_index, mes->doorbell_bitmap);
75− WARN_ON(!old);
76−}
77−
7842 static int amdgpu_mes_doorbell_init(struct amdgpu_device *adev)
7943 {
8044 int i;
@@ -126,7 +90,7 @@ static void amdgpu_mes_doorbell_free(struct amdgpu_device *adev) · 45 unmodified lines
12690
12791 int amdgpu_mes_init(struct amdgpu_device *adev)
12892 {
129− int i, r;
93+ int i, r, num_pipes;
13094
13195 adev->mes.adev = adev;
13296
@@ -142,19 +106,52 @@ int amdgpu_mes_init(struct amdgpu_device *adev) · 9 unmodified lines
142106
143107 adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK;
144108 adev->mes.vmid_mask_mmhub = 0xffffff00;
145− adev->mes.vmid_mask_gfxhub = 0xffffff00;
109+ adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xfffffffe : 0xffffff00;
110+
111+ num_pipes = adev->gfx.me.num_pipe_per_me * adev->gfx.me.num_me;
112+ if (num_pipes > AMDGPU_MES_MAX_GFX_PIPES)
113+ dev_warn(adev->dev, "more gfx pipes than supported by MES! (%d vs %d)\n",
114+ num_pipes, AMDGPU_MES_MAX_GFX_PIPES);
115+
116+ for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) {
117+ if (i >= num_pipes)
118+ break;
119+ if (amdgpu_ip_version(adev, GC_HWIP, 0) >=
120+ IP_VERSION(12, 0, 0))
121+ /*
122+ * GFX V12 has only one GFX pipe, but 8 queues in it.
123+ * GFX pipe 0 queue 0 is being used by Kernel queue.
124+ * Set GFX pipe 0 queue 1-7 for MES scheduling
125+ * mask = 1111 1110b
126+ */
127+ adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0xFF : 0xFE;
128+ else
129+ /*
130+ * GFX pipe 0 queue 0 is being used by Kernel queue.
131+ * Set GFX pipe 0 queue 1 for MES scheduling
132+ * mask = 10b
133+ */
134+ adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0x3 : 0x2;
135+ }
136+
137+ num_pipes = adev->gfx.mec.num_pipe_per_mec * adev->gfx.mec.num_mec;
138+ if (num_pipes > AMDGPU_MES_MAX_COMPUTE_PIPES)
139+ dev_warn(adev->dev, "more compute pipes than supported by MES! (%d vs %d)\n",
140+ num_pipes, AMDGPU_MES_MAX_COMPUTE_PIPES);
146141
147142 for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) {
148− if (i >= (adev->gfx.mec.num_pipe_per_mec * adev->gfx.mec.num_mec))
143+ if (i >= num_pipes)
149144 break;
150− adev->mes.compute_hqd_mask[i] = 0xc;
145+ adev->mes.compute_hqd_mask[i] = adev->gfx.disable_kq ? 0xF : 0xC;
151146 }
152147
153− for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++)
154− adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffffffe;
148+ num_pipes = adev->sdma.num_instances;
149+ if (num_pipes > AMDGPU_MES_MAX_SDMA_PIPES)
150+ dev_warn(adev->dev, "more SDMA pipes than supported by MES! (%d vs %d)\n",
151+ num_pipes, AMDGPU_MES_MAX_SDMA_PIPES);
155152
156153 for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
157− if (i >= adev->sdma.num_instances)
154+ if (i >= num_pipes)
158155 break;
159156 adev->mes.sdma_hqd_mask[i] = 0xfc;
160157 }
@@ -240,244 +237,6 @@ void amdgpu_mes_fini(struct amdgpu_device *adev) · 79 unmodified lines
240237 mutex_destroy(&adev->mes.mutex_hidden);
241238 }
242239
243−static void amdgpu_mes_queue_free_mqd(struct amdgpu_mes_queue *q)
244−{
245− amdgpu_bo_free_kernel(&q->mqd_obj,
246− &q->mqd_gpu_addr,
247− &q->mqd_cpu_ptr);
248−}
249−
250−int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
251− struct amdgpu_vm *vm)
252−{
253− struct amdgpu_mes_process *process;
254− int r;
255−
256− /* allocate the mes process buffer */
257− process = kzalloc(sizeof(struct amdgpu_mes_process), GFP_KERNEL);
258− if (!process) {
259− DRM_ERROR("no more memory to create mes process\n");
260− return -ENOMEM;
261− }
262−
263− /* allocate the process context bo and map it */
264− r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_PROC_CTX_SIZE, PAGE_SIZE,
265− AMDGPU_GEM_DOMAIN_GTT,
266− &process->proc_ctx_bo,
267− &process->proc_ctx_gpu_addr,
268− &process->proc_ctx_cpu_ptr);
269− if (r) {
270− DRM_ERROR("failed to allocate process context bo\n");
271− goto clean_up_memory;
272− }
273− memset(process->proc_ctx_cpu_ptr, 0, AMDGPU_MES_PROC_CTX_SIZE);
274−
275− /*
276− * Avoid taking any other locks under MES lock to avoid circular
277− * lock dependencies.
278− */
279− amdgpu_mes_lock(&adev->mes);
280−
281− /* add the mes process to idr list */
282− r = idr_alloc(&adev->mes.pasid_idr, process, pasid, pasid + 1,
283− GFP_KERNEL);
284− if (r < 0) {
285− DRM_ERROR("failed to lock pasid=%d\n", pasid);
286− goto clean_up_ctx;
287− }
288−
289− INIT_LIST_HEAD(&process->gang_list);
290− process->vm = vm;
291− process->pasid = pasid;
292− process->process_quantum = adev->mes.default_process_quantum;
293− process->pd_gpu_addr = amdgpu_bo_gpu_offset(vm->root.bo);
294−
295− amdgpu_mes_unlock(&adev->mes);
296− return 0;
297−
298−clean_up_ctx:
299− amdgpu_mes_unlock(&adev->mes);
300− amdgpu_bo_free_kernel(&process->proc_ctx_bo,
301− &process->proc_ctx_gpu_addr,
302− &process->proc_ctx_cpu_ptr);
303−clean_up_memory:
304− kfree(process);
305− return r;
306−}
307−
308−void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid)
309−{
310− struct amdgpu_mes_process *process;
311− struct amdgpu_mes_gang *gang, *tmp1;
312− struct amdgpu_mes_queue *queue, *tmp2;
313− struct mes_remove_queue_input queue_input;
314− unsigned long flags;
315− int r;
316−
317− /*
318− * Avoid taking any other locks under MES lock to avoid circular
319− * lock dependencies.
320− */
321− amdgpu_mes_lock(&adev->mes);
322−
323− process = idr_find(&adev->mes.pasid_idr, pasid);
324− if (!process) {
325− DRM_WARN("pasid %d doesn't exist\n", pasid);
326− amdgpu_mes_unlock(&adev->mes);
327− return;
328− }
329−
330− /* Remove all queues from hardware */
331− list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) {
332− list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) {
333− spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
334− idr_remove(&adev->mes.queue_id_idr, queue->queue_id);
335− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
336−
337− queue_input.doorbell_offset = queue->doorbell_off;
338− queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
339−
340− r = adev->mes.funcs->remove_hw_queue(&adev->mes,
341− &queue_input);
342− if (r)
343− DRM_WARN("failed to remove hardware queue\n");
344− }
345−
346− idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
347− }
348−
349− idr_remove(&adev->mes.pasid_idr, pasid);
350− amdgpu_mes_unlock(&adev->mes);
351−
352− /* free all memory allocated by the process */
353− list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) {
354− /* free all queues in the gang */
355− list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) {
356− amdgpu_mes_queue_free_mqd(queue);
357− list_del(&queue->list);
358− kfree(queue);
359− }
360− amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
361− &gang->gang_ctx_gpu_addr,
362− &gang->gang_ctx_cpu_ptr);
363− list_del(&gang->list);
364− kfree(gang);
365−
366− }
367− amdgpu_bo_free_kernel(&process->proc_ctx_bo,
368− &process->proc_ctx_gpu_addr,
369− &process->proc_ctx_cpu_ptr);
370− kfree(process);
371−}
372−
373−int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
374− struct amdgpu_mes_gang_properties *gprops,
375− int *gang_id)
376−{
377− struct amdgpu_mes_process *process;
378− struct amdgpu_mes_gang *gang;
379− int r;
380−
381− /* allocate the mes gang buffer */
382− gang = kzalloc(sizeof(struct amdgpu_mes_gang), GFP_KERNEL);
383− if (!gang) {
384− return -ENOMEM;
385− }
386−
387− /* allocate the gang context bo and map it to cpu space */
388− r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_GANG_CTX_SIZE, PAGE_SIZE,
389− AMDGPU_GEM_DOMAIN_GTT,
390− &gang->gang_ctx_bo,
391− &gang->gang_ctx_gpu_addr,
392− &gang->gang_ctx_cpu_ptr);
393− if (r) {
394− DRM_ERROR("failed to allocate process context bo\n");
395− goto clean_up_mem;
396− }
397− memset(gang->gang_ctx_cpu_ptr, 0, AMDGPU_MES_GANG_CTX_SIZE);
398−
399− /*
400− * Avoid taking any other locks under MES lock to avoid circular
401− * lock dependencies.
402− */
403− amdgpu_mes_lock(&adev->mes);
404−
405− process = idr_find(&adev->mes.pasid_idr, pasid);
406− if (!process) {
407− DRM_ERROR("pasid %d doesn't exist\n", pasid);
408− r = -EINVAL;
409− goto clean_up_ctx;
410− }
411−
412− /* add the mes gang to idr list */
413− r = idr_alloc(&adev->mes.gang_id_idr, gang, 1, 0,
414− GFP_KERNEL);
415− if (r < 0) {
416− DRM_ERROR("failed to allocate idr for gang\n");
417− goto clean_up_ctx;
418− }
419−
420− gang->gang_id = r;
421− *gang_id = r;
422−
423− INIT_LIST_HEAD(&gang->queue_list);
424− gang->process = process;
425− gang->priority = gprops->priority;
426− gang->gang_quantum = gprops->gang_quantum ?
427− gprops->gang_quantum : adev->mes.default_gang_quantum;
428− gang->global_priority_level = gprops->global_priority_level;
429− gang->inprocess_gang_priority = gprops->inprocess_gang_priority;
430− list_add_tail(&gang->list, &process->gang_list);
431−
432− amdgpu_mes_unlock(&adev->mes);
433− return 0;
434−
435−clean_up_ctx:
436− amdgpu_mes_unlock(&adev->mes);
437− amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
438− &gang->gang_ctx_gpu_addr,
439− &gang->gang_ctx_cpu_ptr);
440−clean_up_mem:
441− kfree(gang);
442− return r;
443−}
444−
445−int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id)
446−{
447− struct amdgpu_mes_gang *gang;
448−
449− /*
450− * Avoid taking any other locks under MES lock to avoid circular
451− * lock dependencies.
452− */
453− amdgpu_mes_lock(&adev->mes);
454−
455− gang = idr_find(&adev->mes.gang_id_idr, gang_id);
456− if (!gang) {
457− DRM_ERROR("gang id %d doesn't exist\n", gang_id);
458− amdgpu_mes_unlock(&adev->mes);
459− return -EINVAL;
460− }
461−
462− if (!list_empty(&gang->queue_list)) {
463− DRM_ERROR("queue list is not empty\n");
464− amdgpu_mes_unlock(&adev->mes);
465− return -EBUSY;
466− }
467−
468− idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
469− list_del(&gang->list);
470− amdgpu_mes_unlock(&adev->mes);
471−
472− amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
473− &gang->gang_ctx_gpu_addr,
474− &gang->gang_ctx_cpu_ptr);
475−
476− kfree(gang);
477−
478− return 0;
479−}
480−
481240 int amdgpu_mes_suspend(struct amdgpu_device *adev)
482241 {
483242 struct mes_suspend_gang_input input;
@@ -526,304 +285,6 @@ int amdgpu_mes_resume(struct amdgpu_device *adev) · 42 unmodified lines
526285 return r;
527286 }
528287
529−static int amdgpu_mes_queue_alloc_mqd(struct amdgpu_device *adev,
530− struct amdgpu_mes_queue *q,
531− struct amdgpu_mes_queue_properties *p)
532−{
533− struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type];
534− u32 mqd_size = mqd_mgr->mqd_size;
535− int r;
536−
537− r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
538− AMDGPU_GEM_DOMAIN_GTT,
539− &q->mqd_obj,
540− &q->mqd_gpu_addr, &q->mqd_cpu_ptr);
541− if (r) {
542− dev_warn(adev->dev, "failed to create queue mqd bo (%d)", r);
543− return r;
544− }
545− memset(q->mqd_cpu_ptr, 0, mqd_size);
546−
547− r = amdgpu_bo_reserve(q->mqd_obj, false);
548− if (unlikely(r != 0))
549− goto clean_up;
550−
551− return 0;
552−
553−clean_up:
554− amdgpu_bo_free_kernel(&q->mqd_obj,
555− &q->mqd_gpu_addr,
556− &q->mqd_cpu_ptr);
557− return r;
558−}
559−
560−static void amdgpu_mes_queue_init_mqd(struct amdgpu_device *adev,
561− struct amdgpu_mes_queue *q,
562− struct amdgpu_mes_queue_properties *p)
563−{
564− struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type];
565− struct amdgpu_mqd_prop mqd_prop = {0};
566−
567− mqd_prop.mqd_gpu_addr = q->mqd_gpu_addr;
568− mqd_prop.hqd_base_gpu_addr = p->hqd_base_gpu_addr;
569− mqd_prop.rptr_gpu_addr = p->rptr_gpu_addr;
570− mqd_prop.wptr_gpu_addr = p->wptr_gpu_addr;
571− mqd_prop.queue_size = p->queue_size;
572− mqd_prop.use_doorbell = true;
573− mqd_prop.doorbell_index = p->doorbell_off;
574− mqd_prop.eop_gpu_addr = p->eop_gpu_addr;
575− mqd_prop.hqd_pipe_priority = p->hqd_pipe_priority;
576− mqd_prop.hqd_queue_priority = p->hqd_queue_priority;
577− mqd_prop.hqd_active = false;
578−
579− if (p->queue_type == AMDGPU_RING_TYPE_GFX ||
580− p->queue_type == AMDGPU_RING_TYPE_COMPUTE) {
581− mutex_lock(&adev->srbm_mutex);
582− amdgpu_gfx_select_me_pipe_q(adev, p->ring->me, p->ring->pipe, 0, 0, 0);
583− }
584−
585− mqd_mgr->init_mqd(adev, q->mqd_cpu_ptr, &mqd_prop);
586−
587− if (p->queue_type == AMDGPU_RING_TYPE_GFX ||
588− p->queue_type == AMDGPU_RING_TYPE_COMPUTE) {
589− amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
590− mutex_unlock(&adev->srbm_mutex);
591− }
592−
593− amdgpu_bo_unreserve(q->mqd_obj);
594−}
595−
596−int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
597− struct amdgpu_mes_queue_properties *qprops,
598− int *queue_id)
599−{
600− struct amdgpu_mes_queue *queue;
601− struct amdgpu_mes_gang *gang;
602− struct mes_add_queue_input queue_input;
603− unsigned long flags;
604− int r;
605−
606− memset(&queue_input, 0, sizeof(struct mes_add_queue_input));
607−
608− /* allocate the mes queue buffer */
609− queue = kzalloc(sizeof(struct amdgpu_mes_queue), GFP_KERNEL);
610− if (!queue) {
611− DRM_ERROR("Failed to allocate memory for queue\n");
612− return -ENOMEM;
613− }
614−
615− /* Allocate the queue mqd */
616− r = amdgpu_mes_queue_alloc_mqd(adev, queue, qprops);
617− if (r)
618− goto clean_up_memory;
619−
620− /*
621− * Avoid taking any other locks under MES lock to avoid circular
622− * lock dependencies.
623− */
624− amdgpu_mes_lock(&adev->mes);
625−
626− gang = idr_find(&adev->mes.gang_id_idr, gang_id);
627− if (!gang) {
628− DRM_ERROR("gang id %d doesn't exist\n", gang_id);
629− r = -EINVAL;
630− goto clean_up_mqd;
631− }
632−
633− /* add the mes gang to idr list */
634− spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
635− r = idr_alloc(&adev->mes.queue_id_idr, queue, 1, 0,
636− GFP_ATOMIC);
637− if (r < 0) {
638− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
639− goto clean_up_mqd;
640− }
641− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
642− *queue_id = queue->queue_id = r;
643−
644− /* allocate a doorbell index for the queue */
645− r = amdgpu_mes_kernel_doorbell_get(adev,
646− qprops->queue_type,
647− &qprops->doorbell_off);
648− if (r)
649− goto clean_up_queue_id;
650−
651− /* initialize the queue mqd */
652− amdgpu_mes_queue_init_mqd(adev, queue, qprops);
653−
654− /* add hw queue to mes */
655− queue_input.process_id = gang->process->pasid;
656−
657− queue_input.page_table_base_addr =
658− adev->vm_manager.vram_base_offset + gang->process->pd_gpu_addr -
659− adev->gmc.vram_start;
660−
661− queue_input.process_va_start = 0;
662− queue_input.process_va_end =
663− (adev->vm_manager.max_pfn - 1) << AMDGPU_GPU_PAGE_SHIFT;
664− queue_input.process_quantum = gang->process->process_quantum;
665− queue_input.process_context_addr = gang->process->proc_ctx_gpu_addr;
666− queue_input.gang_quantum = gang->gang_quantum;
667− queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
668− queue_input.inprocess_gang_priority = gang->inprocess_gang_priority;
669− queue_input.gang_global_priority_level = gang->global_priority_level;
670− queue_input.doorbell_offset = qprops->doorbell_off;
671− queue_input.mqd_addr = queue->mqd_gpu_addr;
672− queue_input.wptr_addr = qprops->wptr_gpu_addr;
673− queue_input.wptr_mc_addr = qprops->wptr_mc_addr;
674− queue_input.queue_type = qprops->queue_type;
675− queue_input.paging = qprops->paging;
676− queue_input.is_kfd_process = 0;
677−
678− r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input);
679− if (r) {
680− DRM_ERROR("failed to add hardware queue to MES, doorbell=0x%llx\n",
681− qprops->doorbell_off);
682− goto clean_up_doorbell;
683− }
684−
685− DRM_DEBUG("MES hw queue was added, pasid=%d, gang id=%d, "
686− "queue type=%d, doorbell=0x%llx\n",
687− gang->process->pasid, gang_id, qprops->queue_type,
688− qprops->doorbell_off);
689−
690− queue->ring = qprops->ring;
691− queue->doorbell_off = qprops->doorbell_off;
692− queue->wptr_gpu_addr = qprops->wptr_gpu_addr;
693− queue->queue_type = qprops->queue_type;
694− queue->paging = qprops->paging;
695− queue->gang = gang;
696− queue->ring->mqd_ptr = queue->mqd_cpu_ptr;
697− list_add_tail(&queue->list, &gang->queue_list);
698−
699− amdgpu_mes_unlock(&adev->mes);
700− return 0;
701−
702−clean_up_doorbell:
703− amdgpu_mes_kernel_doorbell_free(adev, qprops->doorbell_off);
704−clean_up_queue_id:
705− spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
706− idr_remove(&adev->mes.queue_id_idr, queue->queue_id);
707− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
708−clean_up_mqd:
709− amdgpu_mes_unlock(&adev->mes);
710− amdgpu_mes_queue_free_mqd(queue);
711−clean_up_memory:
712− kfree(queue);
713− return r;
714−}
715−
716−int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id)
717−{
718− unsigned long flags;
719− struct amdgpu_mes_queue *queue;
720− struct amdgpu_mes_gang *gang;
721− struct mes_remove_queue_input queue_input;
722− int r;
723−
724− /*
725− * Avoid taking any other locks under MES lock to avoid circular
726− * lock dependencies.
727− */
728− amdgpu_mes_lock(&adev->mes);
729−
730− /* remove the mes gang from idr list */
731− spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
732−
733− queue = idr_find(&adev->mes.queue_id_idr, queue_id);
734− if (!queue) {
735− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
736− amdgpu_mes_unlock(&adev->mes);
737− DRM_ERROR("queue id %d doesn't exist\n", queue_id);
738− return -EINVAL;
739− }
740−
741− idr_remove(&adev->mes.queue_id_idr, queue_id);
742− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
743−
744− DRM_DEBUG("try to remove queue, doorbell off = 0x%llx\n",
745− queue->doorbell_off);
746−
747− gang = queue->gang;
748− queue_input.doorbell_offset = queue->doorbell_off;
749− queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
750−
751− r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);
752− if (r)
753− DRM_ERROR("failed to remove hardware queue, queue id = %d\n",
754− queue_id);
755−
756− list_del(&queue->list);
757− amdgpu_mes_kernel_doorbell_free(adev, queue->doorbell_off);
758− amdgpu_mes_unlock(&adev->mes);
759−
760− amdgpu_mes_queue_free_mqd(queue);
761− kfree(queue);
762− return 0;
763−}
764−
765−int amdgpu_mes_reset_hw_queue(struct amdgpu_device *adev, int queue_id)
766−{
767− unsigned long flags;
768− struct amdgpu_mes_queue *queue;
769− struct amdgpu_mes_gang *gang;
770− struct mes_reset_queue_input queue_input;
771− int r;
772−
773− /*
774− * Avoid taking any other locks under MES lock to avoid circular
775− * lock dependencies.
776− */
777− amdgpu_mes_lock(&adev->mes);
778−
779− /* remove the mes gang from idr list */
780− spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
781−
782− queue = idr_find(&adev->mes.queue_id_idr, queue_id);
783− if (!queue) {
784− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
785− amdgpu_mes_unlock(&adev->mes);
786− DRM_ERROR("queue id %d doesn't exist\n", queue_id);
787− return -EINVAL;
788− }
789− spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
790−
791− DRM_DEBUG("try to reset queue, doorbell off = 0x%llx\n",
792− queue->doorbell_off);
793−
794− gang = queue->gang;
795− queue_input.doorbell_offset = queue->doorbell_off;
796− queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
797−
798− r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
799− if (r)
800− DRM_ERROR("failed to reset hardware queue, queue id = %d\n",
801− queue_id);
802−
803− amdgpu_mes_unlock(&adev->mes);
804−
805− return 0;
806−}
807−
808−int amdgpu_mes_reset_hw_queue_mmio(struct amdgpu_device *adev, int queue_type,
809− int me_id, int pipe_id, int queue_id, int vmid)
810−{
811− struct mes_reset_queue_input queue_input;
812− int r;
813−
814− queue_input.queue_type = queue_type;
815− queue_input.use_mmio = true;
816− queue_input.me_id = me_id;
817− queue_input.pipe_id = pipe_id;
818− queue_input.queue_id = queue_id;
819− queue_input.vmid = vmid;
820− r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
821− if (r)
822− DRM_ERROR("failed to reset hardware queue by mmio, queue id = %d\n",
823− queue_id);
824− return r;
825−}
826−
827288 int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev,
828289 struct amdgpu_ring *ring)
829290 {
@@ -874,7 +335,7 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 44 unmodified lines
874335 unsigned int vmid,
875336 bool use_mmio)
876337 {
877− struct mes_reset_legacy_queue_input queue_input;
338+ struct mes_reset_queue_input queue_input;
878339 int r;
879340
880341 memset(&queue_input, 0, sizeof(queue_input));
@@ -888,8 +349,11 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 7 unmodified lines
888349 queue_input.wptr_addr = ring->wptr_gpu_addr;
889350 queue_input.vmid = vmid;
890351 queue_input.use_mmio = use_mmio;
352+ queue_input.is_kq = true;
353+ if (ring->funcs->type == AMDGPU_RING_TYPE_GFX)
354+ queue_input.legacy_gfx = true;
891355
892− r = adev->mes.funcs->reset_legacy_queue(&adev->mes, &queue_input);
356+ r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
893357 if (r)
894358 DRM_ERROR("failed to reset legacy queue\n");
895359
@@ -905,7 +369,7 @@ uint32_t amdgpu_mes_rreg(struct amdgpu_device *adev, uint32_t reg) · 9 unmodified lines
905369 uint32_t *read_val_ptr;
906370
907371 if (amdgpu_device_wb_get(adev, &addr_offset)) {
908− DRM_ERROR("critical bug! too many mes readers\n");
372+ dev_err(adev->dev, "critical bug! too many mes readers\n");
909373 goto error;
910374 }
911375 read_val_gpu_addr = adev->wb.gpu_addr + (addr_offset * 4);
@@ -915,13 +379,13 @@ uint32_t amdgpu_mes_rreg(struct amdgpu_device *adev, uint32_t reg) · 3 unmodified lines
915379 op_input.read_reg.buffer_addr = read_val_gpu_addr;
916380
917381 if (!adev->mes.funcs->misc_op) {
918− DRM_ERROR("mes rreg is not supported!\n");
382+ dev_err(adev->dev, "mes rreg is not supported!\n");
919383 goto error;
920384 }
921385
922386 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
923387 if (r)
924− DRM_ERROR("failed to read reg (0x%x)\n", reg);
388+ dev_err(adev->dev, "failed to read reg (0x%x)\n", reg);
925389 else
926390 val = *(read_val_ptr);
927391
@@ -942,14 +406,14 @@ int amdgpu_mes_wreg(struct amdgpu_device *adev, · 14 unmodified lines
942406 op_input.write_reg.reg_value = val;
943407
944408 if (!adev->mes.funcs->misc_op) {
945− DRM_ERROR("mes wreg is not supported!\n");
409+ dev_err(adev->dev, "mes wreg is not supported!\n");
946410 r = -EINVAL;
947411 goto error;
948412 }
949413
950414 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
951415 if (r)
952− DRM_ERROR("failed to write reg (0x%x)\n", reg);
416+ dev_err(adev->dev, "failed to write reg (0x%x)\n", reg);
953417
954418 error:
955419 return r;
@@ -969,14 +433,14 @@ int amdgpu_mes_reg_write_reg_wait(struct amdgpu_device *adev, · 13 unmodified lines
969433 op_input.wrm_reg.mask = mask;
970434
971435 if (!adev->mes.funcs->misc_op) {
972− DRM_ERROR("mes reg_write_reg_wait is not supported!\n");
436+ dev_err(adev->dev, "mes reg_write_reg_wait is not supported!\n");
973437 r = -EINVAL;
974438 goto error;
975439 }
976440
977441 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
978442 if (r)
979− DRM_ERROR("failed to reg_write_reg_wait\n");
443+ dev_err(adev->dev, "failed to reg_write_reg_wait\n");
980444
981445 error:
982446 return r;
@@ -994,14 +458,14 @@ int amdgpu_mes_reg_wait(struct amdgpu_device *adev, uint32_t reg, · 11 unmodified lines
994458 op_input.wrm_reg.mask = mask;
995459
996460 if (!adev->mes.funcs->misc_op) {
997− DRM_ERROR("mes reg wait is not supported!\n");
461+ dev_err(adev->dev, "mes reg wait is not supported!\n");
998462 r = -EINVAL;
999463 goto error;
1000464 }
1001465
1002466 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
1003467 if (r)
1004− DRM_ERROR("failed to reg_write_reg_wait\n");
468+ dev_err(adev->dev, "failed to reg_write_reg_wait\n");
1005469
1006470 error:
1007471 return r;
@@ -1075,25 +539,6 @@ int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev, · 67 unmodified lines
1075539 return r;
1076540 }
1077541
1078−static void
1079−amdgpu_mes_ring_to_queue_props(struct amdgpu_device *adev,
1080− struct amdgpu_ring *ring,
1081− struct amdgpu_mes_queue_properties *props)
1082−{
1083− props->queue_type = ring->funcs->type;
1084− props->hqd_base_gpu_addr = ring->gpu_addr;
1085− props->rptr_gpu_addr = ring->rptr_gpu_addr;
1086− props->wptr_gpu_addr = ring->wptr_gpu_addr;
1087− props->wptr_mc_addr =
1088− ring->mes_ctx->meta_data_mc_addr + ring->wptr_offs;
1089− props->queue_size = ring->ring_size;
1090− props->eop_gpu_addr = ring->eop_gpu_addr;
1091− props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
1092− props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
1093− props->paging = false;
1094− props->ring = ring;
1095−}
1096−
1097542 #define DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(_eng) \
1098543 do { \
1099544 if (id_offs < AMDGPU_MES_CTX_MAX_OFFS) \
@@ -1130,453 +575,12 @@ int amdgpu_mes_ctx_get_offs(struct amdgpu_ring *ring, unsigned int id_offs) · 30 unmodified lines
1130575 return -EINVAL;
1131576 }
1132577
1133−int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
1134− int queue_type, int idx,
1135− struct amdgpu_mes_ctx_data *ctx_data,
1136− struct amdgpu_ring **out)
1137−{
1138− struct amdgpu_ring *ring;
1139− struct amdgpu_mes_gang *gang;
1140− struct amdgpu_mes_queue_properties qprops = {0};
1141− int r, queue_id, pasid;
1142−
1143− /*
1144− * Avoid taking any other locks under MES lock to avoid circular
1145− * lock dependencies.
1146− */
1147− amdgpu_mes_lock(&adev->mes);
1148− gang = idr_find(&adev->mes.gang_id_idr, gang_id);
1149− if (!gang) {
1150− DRM_ERROR("gang id %d doesn't exist\n", gang_id);
1151− amdgpu_mes_unlock(&adev->mes);
1152− return -EINVAL;
1153− }
1154− pasid = gang->process->pasid;
1155−
1156− ring = kzalloc(sizeof(struct amdgpu_ring), GFP_KERNEL);
1157− if (!ring) {
1158− amdgpu_mes_unlock(&adev->mes);
1159− return -ENOMEM;
1160− }
1161−
1162− ring->ring_obj = NULL;
1163− ring->use_doorbell = true;
1164− ring->is_mes_queue = true;
1165− ring->mes_ctx = ctx_data;
1166− ring->idx = idx;
1167− ring->no_scheduler = true;
1168−
1169− if (queue_type == AMDGPU_RING_TYPE_COMPUTE) {
1170− int offset = offsetof(struct amdgpu_mes_ctx_meta_data,
1171− compute[ring->idx].mec_hpd);
1172− ring->eop_gpu_addr =
1173− amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1174− }
1175−
1176− switch (queue_type) {
1177− case AMDGPU_RING_TYPE_GFX:
1178− ring->funcs = adev->gfx.gfx_ring[0].funcs;
1179− ring->me = adev->gfx.gfx_ring[0].me;
1180− ring->pipe = adev->gfx.gfx_ring[0].pipe;
1181− break;
1182− case AMDGPU_RING_TYPE_COMPUTE:
1183− ring->funcs = adev->gfx.compute_ring[0].funcs;
1184− ring->me = adev->gfx.compute_ring[0].me;
1185− ring->pipe = adev->gfx.compute_ring[0].pipe;
1186− break;
1187− case AMDGPU_RING_TYPE_SDMA:
1188− ring->funcs = adev->sdma.instance[0].ring.funcs;
1189− break;
1190− default:
1191− BUG();
1192− }
1193−
1194− r = amdgpu_ring_init(adev, ring, 1024, NULL, 0,
1195− AMDGPU_RING_PRIO_DEFAULT, NULL);
1196− if (r) {
1197− amdgpu_mes_unlock(&adev->mes);
1198− goto clean_up_memory;
1199− }
1200−
1201− amdgpu_mes_ring_to_queue_props(adev, ring, &qprops);
1202−
1203− dma_fence_wait(gang->process->vm->last_update, false);
1204− dma_fence_wait(ctx_data->meta_data_va->last_pt_update, false);
1205− amdgpu_mes_unlock(&adev->mes);
1206−
1207− r = amdgpu_mes_add_hw_queue(adev, gang_id, &qprops, &queue_id);
1208− if (r)
1209− goto clean_up_ring;
1210−
1211− ring->hw_queue_id = queue_id;
1212− ring->doorbell_index = qprops.doorbell_off;
1213−
1214− if (queue_type == AMDGPU_RING_TYPE_GFX)
1215− sprintf(ring->name, "gfx_%d.%d.%d", pasid, gang_id, queue_id);
1216− else if (queue_type == AMDGPU_RING_TYPE_COMPUTE)
1217− sprintf(ring->name, "compute_%d.%d.%d", pasid, gang_id,
1218− queue_id);
1219− else if (queue_type == AMDGPU_RING_TYPE_SDMA)
1220− sprintf(ring->name, "sdma_%d.%d.%d", pasid, gang_id,
1221− queue_id);
1222− else
1223− BUG();
1224−
1225− *out = ring;
1226− return 0;
1227−
1228−clean_up_ring:
1229− amdgpu_ring_fini(ring);
1230−clean_up_memory:
1231− kfree(ring);
1232− return r;
1233−}
1234−
1235−void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
1236− struct amdgpu_ring *ring)
1237−{
1238− if (!ring)
1239− return;
1240−
1241− amdgpu_mes_remove_hw_queue(adev, ring->hw_queue_id);
1242− timer_delete_sync(&ring->fence_drv.fallback_timer);
1243− amdgpu_ring_fini(ring);
1244− kfree(ring);
1245−}
1246−
1247578 uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev,
1248579 enum amdgpu_mes_priority_level prio)
1249580 {
1250581 return adev->mes.aggregated_doorbells[prio];
1251582 }
1252583
1253−int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
1254− struct amdgpu_mes_ctx_data *ctx_data)
1255−{
1256− int r;
1257−
1258− r = amdgpu_bo_create_kernel(adev,
1259− sizeof(struct amdgpu_mes_ctx_meta_data),
1260− PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
1261− &ctx_data->meta_data_obj,
1262− &ctx_data->meta_data_mc_addr,
1263− &ctx_data->meta_data_ptr);
1264− if (r) {
1265− dev_warn(adev->dev, "(%d) create CTX bo failed\n", r);
1266− return r;
1267− }
1268−
1269− if (!ctx_data->meta_data_obj)
1270− return -ENOMEM;
1271−
1272− memset(ctx_data->meta_data_ptr, 0,
1273− sizeof(struct amdgpu_mes_ctx_meta_data));
1274−
1275− return 0;
1276−}
1277−
1278−void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data)
1279−{
1280− if (ctx_data->meta_data_obj)
1281− amdgpu_bo_free_kernel(&ctx_data->meta_data_obj,
1282− &ctx_data->meta_data_mc_addr,
1283− &ctx_data->meta_data_ptr);
1284−}
1285−
1286−int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
1287− struct amdgpu_vm *vm,
1288− struct amdgpu_mes_ctx_data *ctx_data)
1289−{
1290− struct amdgpu_bo_va *bo_va;
1291− struct amdgpu_sync sync;
1292− struct drm_exec exec;
1293− int r;
1294−
1295− amdgpu_sync_create(&sync);
1296−
1297− drm_exec_init(&exec, 0, 0);
1298− drm_exec_until_all_locked(&exec) {
1299− r = drm_exec_lock_obj(&exec,
1300− &ctx_data->meta_data_obj->tbo.base);
1301− drm_exec_retry_on_contention(&exec);
1302− if (unlikely(r))
1303− goto error_fini_exec;
1304−
1305− r = amdgpu_vm_lock_pd(vm, &exec, 0);
1306− drm_exec_retry_on_contention(&exec);
1307− if (unlikely(r))
1308− goto error_fini_exec;
1309− }
1310−
1311− bo_va = amdgpu_vm_bo_add(adev, vm, ctx_data->meta_data_obj);
1312− if (!bo_va) {
1313− DRM_ERROR("failed to create bo_va for meta data BO\n");
1314− r = -ENOMEM;
1315− goto error_fini_exec;
1316− }
1317−
1318− r = amdgpu_vm_bo_map(adev, bo_va, ctx_data->meta_data_gpu_addr, 0,
1319− sizeof(struct amdgpu_mes_ctx_meta_data),
1320− AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
1321− AMDGPU_PTE_EXECUTABLE);
1322−
1323− if (r) {
1324− DRM_ERROR("failed to do bo_map on meta data, err=%d\n", r);
1325− goto error_del_bo_va;
1326− }
1327−
1328− r = amdgpu_vm_bo_update(adev, bo_va, false);
1329− if (r) {
1330− DRM_ERROR("failed to do vm_bo_update on meta data\n");
1331− goto error_del_bo_va;
1332− }
1333− amdgpu_sync_fence(&sync, bo_va->last_pt_update, GFP_KERNEL);
1334−
1335− r = amdgpu_vm_update_pdes(adev, vm, false);
1336− if (r) {
1337− DRM_ERROR("failed to update pdes on meta data\n");
1338− goto error_del_bo_va;
1339− }
1340− amdgpu_sync_fence(&sync, vm->last_update, GFP_KERNEL);
1341−
1342− amdgpu_sync_wait(&sync, false);
1343− drm_exec_fini(&exec);
1344−
1345− amdgpu_sync_free(&sync);
1346− ctx_data->meta_data_va = bo_va;
1347− return 0;
1348−
1349−error_del_bo_va:
1350− amdgpu_vm_bo_del(adev, bo_va);
1351−
1352−error_fini_exec:
1353− drm_exec_fini(&exec);
1354− amdgpu_sync_free(&sync);
1355− return r;
1356−}
1357−
1358−int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev,
1359− struct amdgpu_mes_ctx_data *ctx_data)
1360−{
1361− struct amdgpu_bo_va *bo_va = ctx_data->meta_data_va;
1362− struct amdgpu_bo *bo = ctx_data->meta_data_obj;
1363− struct amdgpu_vm *vm = bo_va->base.vm;
1364− struct dma_fence *fence;
1365− struct drm_exec exec;
1366− long r;
1367−
1368− drm_exec_init(&exec, 0, 0);
1369− drm_exec_until_all_locked(&exec) {
1370− r = drm_exec_lock_obj(&exec,
1371− &ctx_data->meta_data_obj->tbo.base);
1372− drm_exec_retry_on_contention(&exec);
1373− if (unlikely(r))
1374− goto out_unlock;
1375−
1376− r = amdgpu_vm_lock_pd(vm, &exec, 0);
1377− drm_exec_retry_on_contention(&exec);
1378− if (unlikely(r))
1379− goto out_unlock;
1380− }
1381−
1382− amdgpu_vm_bo_del(adev, bo_va);
1383− if (!amdgpu_vm_ready(vm))
1384− goto out_unlock;
1385−
1386− r = dma_resv_get_singleton(bo->tbo.base.resv, DMA_RESV_USAGE_BOOKKEEP,
1387− &fence);
1388− if (r)
1389− goto out_unlock;
1390− if (fence) {
1391− amdgpu_bo_fence(bo, fence, true);
1392− fence = NULL;
1393− }
1394−
1395− r = amdgpu_vm_clear_freed(adev, vm, &fence);
1396− if (r || !fence)
1397− goto out_unlock;
1398−
1399− dma_fence_wait(fence, false);
1400− amdgpu_bo_fence(bo, fence, true);
1401− dma_fence_put(fence);
1402−
1403−out_unlock:
1404− if (unlikely(r < 0))
1405− dev_err(adev->dev, "failed to clear page tables (%ld)\n", r);
1406− drm_exec_fini(&exec);
1407−
1408− return r;
1409−}
1410−
1411−static int amdgpu_mes_test_create_gang_and_queues(struct amdgpu_device *adev,
1412− int pasid, int *gang_id,
1413− int queue_type, int num_queue,
1414− struct amdgpu_ring **added_rings,
1415− struct amdgpu_mes_ctx_data *ctx_data)
1416−{
1417− struct amdgpu_ring *ring;
1418− struct amdgpu_mes_gang_properties gprops = {0};
1419− int r, j;
1420−
1421− /* create a gang for the process */
1422− gprops.priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1423− gprops.gang_quantum = adev->mes.default_gang_quantum;
1424− gprops.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1425− gprops.priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1426− gprops.global_priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1427−
1428− r = amdgpu_mes_add_gang(adev, pasid, &gprops, gang_id);
1429− if (r) {
1430− DRM_ERROR("failed to add gang\n");
1431− return r;
1432− }
1433−
1434− /* create queues for the gang */
1435− for (j = 0; j < num_queue; j++) {
1436− r = amdgpu_mes_add_ring(adev, *gang_id, queue_type, j,
1437− ctx_data, &ring);
1438− if (r) {
1439− DRM_ERROR("failed to add ring\n");
1440− break;
1441− }
1442−
1443− DRM_INFO("ring %s was added\n", ring->name);
1444− added_rings[j] = ring;
1445− }
1446−
1447− return 0;
1448−}
1449−
1450−static int amdgpu_mes_test_queues(struct amdgpu_ring **added_rings)
1451−{
1452− struct amdgpu_ring *ring;
1453− int i, r;
1454−
1455− for (i = 0; i < AMDGPU_MES_CTX_MAX_RINGS; i++) {
1456− ring = added_rings[i];
1457− if (!ring)
1458− continue;
1459−
1460− r = amdgpu_ring_test_helper(ring);
1461− if (r)
1462− return r;
1463−
1464− r = amdgpu_ring_test_ib(ring, 1000 * 10);
1465− if (r) {
1466− DRM_DEV_ERROR(ring->adev->dev,
1467− "ring %s ib test failed (%d)\n",
1468− ring->name, r);
1469− return r;
1470− } else
1471− DRM_INFO("ring %s ib test pass\n", ring->name);
1472− }
1473−
1474− return 0;
1475−}
1476−
1477−int amdgpu_mes_self_test(struct amdgpu_device *adev)
1478−{
1479− struct amdgpu_vm *vm = NULL;
1480− struct amdgpu_mes_ctx_data ctx_data = {0};
1481− struct amdgpu_ring *added_rings[AMDGPU_MES_CTX_MAX_RINGS] = { NULL };
1482− int gang_ids[3] = {0};
1483− int queue_types[][2] = { { AMDGPU_RING_TYPE_GFX, 1 },
1484− { AMDGPU_RING_TYPE_COMPUTE, 1 },
1485− { AMDGPU_RING_TYPE_SDMA, 1} };
1486− int i, r, pasid, k = 0;
1487−
1488− pasid = amdgpu_pasid_alloc(16);
1489− if (pasid < 0) {
1490− dev_warn(adev->dev, "No more PASIDs available!");
1491− pasid = 0;
1492− }
1493−
1494− vm = kzalloc(sizeof(*vm), GFP_KERNEL);
1495− if (!vm) {
1496− r = -ENOMEM;
1497− goto error_pasid;
1498− }
1499−
1500− r = amdgpu_vm_init(adev, vm, -1);
1501− if (r) {
1502− DRM_ERROR("failed to initialize vm\n");
1503− goto error_pasid;
1504− }
1505−
1506− r = amdgpu_mes_ctx_alloc_meta_data(adev, &ctx_data);
1507− if (r) {
1508− DRM_ERROR("failed to alloc ctx meta data\n");
1509− goto error_fini;
1510− }
1511−
1512− ctx_data.meta_data_gpu_addr = AMDGPU_VA_RESERVED_BOTTOM;
1513− r = amdgpu_mes_ctx_map_meta_data(adev, vm, &ctx_data);
1514− if (r) {
1515− DRM_ERROR("failed to map ctx meta data\n");
1516− goto error_vm;
1517− }
1518−
1519− r = amdgpu_mes_create_process(adev, pasid, vm);
1520− if (r) {
1521− DRM_ERROR("failed to create MES process\n");
1522− goto error_vm;
1523− }
1524−
1525− for (i = 0; i < ARRAY_SIZE(queue_types); i++) {
1526− /* On GFX v10.3, fw hasn't supported to map sdma queue. */
1527− if (amdgpu_ip_version(adev, GC_HWIP, 0) >=
1528− IP_VERSION(10, 3, 0) &&
1529− amdgpu_ip_version(adev, GC_HWIP, 0) <
1530− IP_VERSION(11, 0, 0) &&
1531− queue_types[i][0] == AMDGPU_RING_TYPE_SDMA)
1532− continue;
1533−
1534− r = amdgpu_mes_test_create_gang_and_queues(adev, pasid,
1535− &gang_ids[i],
1536− queue_types[i][0],
1537− queue_types[i][1],
1538− &added_rings[k],
1539− &ctx_data);
1540− if (r)
1541− goto error_queues;
1542−
1543− k += queue_types[i][1];
1544− }
1545−
1546− /* start ring test and ib test for MES queues */
1547− amdgpu_mes_test_queues(added_rings);
1548−
1549−error_queues:
1550− /* remove all queues */
1551− for (i = 0; i < ARRAY_SIZE(added_rings); i++) {
1552− if (!added_rings[i])
1553− continue;
1554− amdgpu_mes_remove_ring(adev, added_rings[i]);
1555− }
1556−
1557− for (i = 0; i < ARRAY_SIZE(gang_ids); i++) {
1558− if (!gang_ids[i])
1559− continue;
1560− amdgpu_mes_remove_gang(adev, gang_ids[i]);
1561− }
1562−
1563− amdgpu_mes_destroy_process(adev, pasid);
1564−
1565−error_vm:
1566− amdgpu_mes_ctx_unmap_meta_data(adev, &ctx_data);
1567−
1568−error_fini:
1569− amdgpu_vm_fini(adev, vm);
1570−
1571−error_pasid:
1572− if (pasid)
1573− amdgpu_pasid_free(pasid);
1574−
1575− amdgpu_mes_ctx_free_meta_data(&ctx_data);
1576− kfree(vm);
1577− return 0;
1578−}
1579−
1580584 int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe)
1581585 {
1582586 const struct mes_firmware_header_v1_0 *mes_hdr;
@@ -1705,7 +709,7 @@ int amdgpu_mes_update_enforce_isolation(struct amdgpu_device *adev) · 122 unmodified lines
1705709 if (adev->enable_mes && adev->gfx.enable_cleaner_shader) {
1706710 mutex_lock(&adev->enforce_isolation_mutex);
1707711 for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) {
1708− if (adev->enforce_isolation[i])
712+ if (adev->enforce_isolation[i] == AMDGPU_ENFORCE_ISOLATION_ENABLE)
1709713 r |= amdgpu_mes_set_enforce_isolation(adev, i, true);
1710714 else
1711715 r |= amdgpu_mes_set_enforce_isolation(adev, i, false);
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h+4−64
@@ -111,8 +111,8 @@ struct amdgpu_mes { · 110 unmodified lines
111111
112112 uint32_t vmid_mask_gfxhub;
113113 uint32_t vmid_mask_mmhub;
114− uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES];
115114 uint32_t gfx_hqd_mask[AMDGPU_MES_MAX_GFX_PIPES];
115+ uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES];
116116 uint32_t sdma_hqd_mask[AMDGPU_MES_MAX_SDMA_PIPES];
117117 uint32_t aggregated_doorbells[AMDGPU_MES_PRIORITY_NUM_LEVELS];
118118 uint32_t sch_ctx_offs[AMDGPU_MAX_MES_PIPES];
@@ -149,19 +149,6 @@ struct amdgpu_mes { · 30 unmodified lines
149149
150150 };
151151
152−struct amdgpu_mes_process {
153− int pasid;
154− struct amdgpu_vm *vm;
155− uint64_t pd_gpu_addr;
156− struct amdgpu_bo *proc_ctx_bo;
157− uint64_t proc_ctx_gpu_addr;
158− void *proc_ctx_cpu_ptr;
159− uint64_t process_quantum;
160− struct list_head gang_list;
161− uint32_t doorbell_index;
162− struct mutex doorbell_lock;
163−};
164−
165152 struct amdgpu_mes_gang {
166153 int gang_id;
167154 int priority;
@@ -248,18 +235,6 @@ struct mes_remove_queue_input { · 80 unmodified lines
248235 uint64_t gang_context_addr;
249236 };
250237
251−struct mes_reset_queue_input {
252− uint32_t doorbell_offset;
253− uint64_t gang_context_addr;
254− bool use_mmio;
255− uint32_t queue_type;
256− uint32_t me_id;
257− uint32_t pipe_id;
258− uint32_t queue_id;
259− uint32_t xcc_id;
260− uint32_t vmid;
261−};
262−
263238 struct mes_map_legacy_queue_input {
264239 uint32_t queue_type;
265240 uint32_t doorbell_offset;
@@ -291,7 +266,7 @@ struct mes_resume_gang_input { · 25 unmodified lines
291266 uint64_t gang_context_addr;
292267 };
293268
294−struct mes_reset_legacy_queue_input {
269+struct mes_reset_queue_input {
295270 uint32_t queue_type;
296271 uint32_t doorbell_offset;
297272 bool use_mmio;
@@ -301,6 +276,8 @@ struct mes_reset_legacy_queue_input { · 3 unmodified lines
301276 uint64_t mqd_addr;
302277 uint64_t wptr_addr;
303278 uint32_t vmid;
279+ bool legacy_gfx;
280+ bool is_kq;
304281 };
305282
306283 enum mes_misc_opcode {
@@ -388,9 +365,6 @@ struct amdgpu_mes_funcs { · 81 unmodified lines
388365 int (*misc_op)(struct amdgpu_mes *mes,
389366 struct mes_misc_op_input *input);
390367
391− int (*reset_legacy_queue)(struct amdgpu_mes *mes,
392− struct mes_reset_legacy_queue_input *input);
393−
394368 int (*reset_hw_queue)(struct amdgpu_mes *mes,
395369 struct mes_reset_queue_input *input);
396370 };
@@ -404,26 +378,9 @@ int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe); · 7 unmodified lines
404378 int amdgpu_mes_init(struct amdgpu_device *adev);
405379 void amdgpu_mes_fini(struct amdgpu_device *adev);
406380
407−int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
408− struct amdgpu_vm *vm);
409−void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid);
410−
411−int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
412− struct amdgpu_mes_gang_properties *gprops,
413− int *gang_id);
414−int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id);
415−
416381 int amdgpu_mes_suspend(struct amdgpu_device *adev);
417382 int amdgpu_mes_resume(struct amdgpu_device *adev);
418383
419−int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
420− struct amdgpu_mes_queue_properties *qprops,
421− int *queue_id);
422−int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id);
423−int amdgpu_mes_reset_hw_queue(struct amdgpu_device *adev, int queue_id);
424−int amdgpu_mes_reset_hw_queue_mmio(struct amdgpu_device *adev, int queue_type,
425− int me_id, int pipe_id, int queue_id, int vmid);
426−
427384 int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev,
428385 struct amdgpu_ring *ring);
429386 int amdgpu_mes_unmap_legacy_queue(struct amdgpu_device *adev,
@@ -451,27 +408,10 @@ int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, · 21 unmodified lines
451408 bool trap_en);
452409 int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev,
453410 uint64_t process_context_addr);
454−int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
455− int queue_type, int idx,
456− struct amdgpu_mes_ctx_data *ctx_data,
457− struct amdgpu_ring **out);
458−void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
459− struct amdgpu_ring *ring);
460411
461412 uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev,
462413 enum amdgpu_mes_priority_level prio);
463414
464−int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
465− struct amdgpu_mes_ctx_data *ctx_data);
466−void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data);
467−int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
468− struct amdgpu_vm *vm,
469− struct amdgpu_mes_ctx_data *ctx_data);
470−int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev,
471− struct amdgpu_mes_ctx_data *ctx_data);
472−
473−int amdgpu_mes_self_test(struct amdgpu_device *adev);
474−
475415 int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev);
476416
477417 /*
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c+5−1
@@ -1644,7 +1644,11 @@ u64 amdgpu_bo_print_info(int id, struct amdgpu_bo *bo, struct seq_file *m) · 1643 unmodified lines
16441644 amdgpu_bo_print_flag(m, bo, VRAM_CONTIGUOUS);
16451645 amdgpu_bo_print_flag(m, bo, VM_ALWAYS_VALID);
16461646 amdgpu_bo_print_flag(m, bo, EXPLICIT_SYNC);
1647−
1647+ /* Add the gem obj resv fence dump*/
1648+ if (dma_resv_trylock(bo->tbo.base.resv)) {
1649+ dma_resv_describe(bo->tbo.base.resv, m);
1650+ dma_resv_unlock(bo->tbo.base.resv);
1651+ }
16481652 seq_puts(m, "\n");
16491653
16501654 return size;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c+2−1
@@ -2214,7 +2214,8 @@ static int psp_securedisplay_initialize(struct psp_context *psp) · 2213 unmodified lines
22142214
22152215 if (!psp->securedisplay_context.context.bin_desc.size_bytes ||
22162216 !psp->securedisplay_context.context.bin_desc.start_addr) {
2217− dev_info(psp->adev->dev, "SECUREDISPLAY: securedisplay ta ucode is not available\n");
2217+ dev_info(psp->adev->dev,
2218+ "SECUREDISPLAY: optional securedisplay ta ucode is not available\n");
22182219 return 0;
22192220 }
22202221
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h+10−0
@@ -107,6 +107,7 @@ enum psp_reg_prog_id { · 106 unmodified lines
107107 PSP_REG_IH_RB_CNTL = 0, /* register IH_RB_CNTL */
108108 PSP_REG_IH_RB_CNTL_RING1 = 1, /* register IH_RB_CNTL_RING1 */
109109 PSP_REG_IH_RB_CNTL_RING2 = 2, /* register IH_RB_CNTL_RING2 */
110+ PSP_REG_MMHUB_L1_TLB_CNTL = 25,
110111 PSP_REG_LAST
111112 };
112113
@@ -142,6 +143,8 @@ struct psp_funcs { · 29 unmodified lines
142143 bool (*get_ras_capability)(struct psp_context *psp);
143144 bool (*is_aux_sos_load_required)(struct psp_context *psp);
144145 bool (*is_reload_needed)(struct psp_context *psp);
146+ int (*reg_program_no_ring)(struct psp_context *psp, uint32_t val,
147+ enum psp_reg_prog_id id);
145148 };
146149
147150 struct ta_funcs {
@@ -475,6 +478,10 @@ struct amdgpu_psp_funcs { · 327 unmodified lines
475478 #define psp_is_aux_sos_load_required(psp) \
476479 ((psp)->funcs->is_aux_sos_load_required ? (psp)->funcs->is_aux_sos_load_required((psp)) : 0)
477480
481+#define psp_reg_program_no_ring(psp, val, id) \
482+ ((psp)->funcs->reg_program_no_ring ? \
483+ (psp)->funcs->reg_program_no_ring((psp), val, id) : -EINVAL)
484+
478485 extern const struct amd_ip_funcs psp_ip_funcs;
479486
480487 extern const struct amdgpu_ip_block_version psp_v3_1_ip_block;
@@ -569,5 +576,8 @@ bool amdgpu_psp_get_ras_capability(struct psp_context *psp); · 88 unmodified lines
569576 int psp_config_sq_perfmon(struct psp_context *psp, uint32_t xcp_id,
570577 bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable);
571578 bool amdgpu_psp_tos_reload_needed(struct amdgpu_device *adev);
579+int amdgpu_psp_reg_program_no_ring(struct psp_context *psp, uint32_t val,
580+ enum psp_reg_prog_id id);
581+
572582
573583 #endif
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c+10−5
@@ -1498,6 +1498,9 @@ int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, · 1497 unmodified lines
14981498 !amdgpu_ras_get_aca_debug_mode(adev))
14991499 return -EOPNOTSUPP;
15001500
1501+ if (amdgpu_sriov_vf(adev))
1502+ return -EOPNOTSUPP;
1503+
15011504 /* skip ras error reset in gpu reset */
15021505 if ((amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) &&
15031506 ((smu_funcs && smu_funcs->set_debug_mode) ||
@@ -2161,7 +2164,7 @@ void amdgpu_ras_interrupt_fatal_error_handler(struct amdgpu_device *adev) · 657 unmodified lines
21612164 /* Fatal error events are handled on host side */
21622165 if (amdgpu_sriov_vf(adev))
21632166 return;
2164− /**
2167+ /*
21652168 * If the current interrupt is caused by a non-fatal RAS error, skip
21662169 * check for fatal error. For fatal errors, FED status of all devices
21672170 * in XGMI hive gets set when the first device gets fatal error
@@ -3793,10 +3796,12 @@ init_ras_enabled_flag: · 1625 unmodified lines
37933796 adev->ras_hw_enabled & amdgpu_ras_mask;
37943797
37953798 /* aca is disabled by default except for psp v13_0_6/v13_0_12/v13_0_14 */
3796− adev->aca.is_enabled =
3797− (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||
3798− amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||
3799− amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14));
3799+ if (!amdgpu_sriov_vf(adev)) {
3800+ adev->aca.is_enabled =
3801+ (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||
3802+ amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||
3803+ amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14));
3804+ }
38003805
38013806 /* bad page feature is not applicable to specific app platform */
38023807 if (adev->gmc.is_app_apu &&
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c+27−8
@@ -418,6 +418,7 @@ static void amdgpu_ras_set_eeprom_table_version(struct amdgpu_ras_eeprom_control · 417 unmodified lines
418418 hdr->version = RAS_TABLE_VER_V2_1;
419419 return;
420420 case IP_VERSION(12, 0, 0):
421+ case IP_VERSION(12, 5, 0):
421422 hdr->version = RAS_TABLE_VER_V3;
422423 return;
423424 default:
@@ -1392,17 +1393,39 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 968 unmodified lines
13921393
13931394 __decode_table_header_from_buf(hdr, buf);
13941395
1395− if (hdr->version >= RAS_TABLE_VER_V2_1) {
1396+ if (hdr->header != RAS_TABLE_HDR_VAL &&
1397+ hdr->header != RAS_TABLE_HDR_BAD) {
1398+ dev_info(adev->dev, "Creating a new EEPROM table");
1399+ return amdgpu_ras_eeprom_reset_table(control);
1400+ }
1401+
1402+ switch (hdr->version) {
1403+ case RAS_TABLE_VER_V2_1:
1404+ case RAS_TABLE_VER_V3:
13961405 control->ras_num_recs = RAS_NUM_RECS_V2_1(hdr);
13971406 control->ras_record_offset = RAS_RECORD_START_V2_1;
13981407 control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1;
1399− } else {
1408+ break;
1409+ case RAS_TABLE_VER_V1:
14001410 control->ras_num_recs = RAS_NUM_RECS(hdr);
14011411 control->ras_record_offset = RAS_RECORD_START;
14021412 control->ras_max_record_count = RAS_MAX_RECORD_COUNT;
1413+ break;
1414+ default:
1415+ dev_err(adev->dev,
1416+ "RAS header invalid, unsupported version: %u",
1417+ hdr->version);
1418+ return -EINVAL;
14031419 }
1404− control->ras_fri = RAS_OFFSET_TO_INDEX(control, hdr->first_rec_offset);
14051420
1421+ if (control->ras_num_recs > control->ras_max_record_count) {
1422+ dev_err(adev->dev,
1423+ "RAS header invalid, records in header: %u max allowed :%u",
1424+ control->ras_num_recs, control->ras_max_record_count);
1425+ return -EINVAL;
1426+ }
1427+
1428+ control->ras_fri = RAS_OFFSET_TO_INDEX(control, hdr->first_rec_offset);
14061429 control->ras_num_mca_recs = 0;
14071430 control->ras_num_pa_recs = 0;
14081431 return 0;
@@ -1413,7 +1436,7 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 4 unmodified lines
14131436 struct amdgpu_device *adev = to_amdgpu_device(control);
14141437 struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr;
14151438 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
1416− int res;
1439+ int res = 0;
14171440
14181441 if (!__is_ras_eeprom_supported(adev))
14191442 return 0;
@@ -1494,10 +1517,6 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 74 unmodified lines
14941517 "User defined threshold is set, runtime service will be halt when threshold is reached\n");
14951518 }
14961519 }
1497− } else {
1498− DRM_INFO("Creating a new EEPROM table");
1499−
1500− res = amdgpu_ras_eeprom_reset_table(control);
15011520 }
15021521
15031522 return res < 0 ? res : 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c+38−74
@@ -187,14 +187,10 @@ void amdgpu_ring_undo(struct amdgpu_ring *ring) · 186 unmodified lines
187187 }
188188
189189 #define amdgpu_ring_get_gpu_addr(ring, offset) \
190− (ring->is_mes_queue ? \
191− (ring->mes_ctx->meta_data_gpu_addr + offset) : \
192− (ring->adev->wb.gpu_addr + offset * 4))
190+ (ring->adev->wb.gpu_addr + offset * 4)
193191
194192 #define amdgpu_ring_get_cpu_addr(ring, offset) \
195− (ring->is_mes_queue ? \
196− (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \
197− (&ring->adev->wb.wb[offset]))
193+ (&ring->adev->wb.wb[offset])
198194
199195 /**
200196 * amdgpu_ring_init - init driver ring struct.
@@ -243,57 +239,42 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · 42 unmodified lines
243239 ring->sched_score = sched_score;
244240 ring->vmid_wait = dma_fence_get_stub();
245241
246− if (!ring->is_mes_queue) {
247− ring->idx = adev->num_rings++;
248− adev->rings[ring->idx] = ring;
249− }
242+ ring->idx = adev->num_rings++;
243+ adev->rings[ring->idx] = ring;
250244
251245 r = amdgpu_fence_driver_init_ring(ring);
252246 if (r)
253247 return r;
254248 }
255249
256− if (ring->is_mes_queue) {
257− ring->rptr_offs = amdgpu_mes_ctx_get_offs(ring,
258− AMDGPU_MES_CTX_RPTR_OFFS);
259− ring->wptr_offs = amdgpu_mes_ctx_get_offs(ring,
260− AMDGPU_MES_CTX_WPTR_OFFS);
261− ring->fence_offs = amdgpu_mes_ctx_get_offs(ring,
262− AMDGPU_MES_CTX_FENCE_OFFS);
263− ring->trail_fence_offs = amdgpu_mes_ctx_get_offs(ring,
264− AMDGPU_MES_CTX_TRAIL_FENCE_OFFS);
265− ring->cond_exe_offs = amdgpu_mes_ctx_get_offs(ring,
266− AMDGPU_MES_CTX_COND_EXE_OFFS);
267− } else {
268− r = amdgpu_device_wb_get(adev, &ring->rptr_offs);
269− if (r) {
270− dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r);
271− return r;
272− }
250+ r = amdgpu_device_wb_get(adev, &ring->rptr_offs);
251+ if (r) {
252+ dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r);
253+ return r;
254+ }
273255
274− r = amdgpu_device_wb_get(adev, &ring->wptr_offs);
275− if (r) {
276− dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r);
277− return r;
278− }
256+ r = amdgpu_device_wb_get(adev, &ring->wptr_offs);
257+ if (r) {
258+ dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r);
259+ return r;
260+ }
279261
280− r = amdgpu_device_wb_get(adev, &ring->fence_offs);
281− if (r) {
282− dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r);
283− return r;
284− }
262+ r = amdgpu_device_wb_get(adev, &ring->fence_offs);
263+ if (r) {
264+ dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r);
265+ return r;
266+ }
285267
286− r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs);
287− if (r) {
288− dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r);
289− return r;
290− }
268+ r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs);
269+ if (r) {
270+ dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r);
271+ return r;
272+ }
291273
292− r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs);
293− if (r) {
294− dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r);
295− return r;
296− }
274+ r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs);
275+ if (r) {
276+ dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r);
277+ return r;
297278 }
298279
299280 ring->fence_gpu_addr =
@@ -353,18 +334,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · 53 unmodified lines
353334 ring->cached_rptr = 0;
354335
355336 /* Allocate ring buffer */
356− if (ring->is_mes_queue) {
357− int offset = 0;
358−
359− BUG_ON(ring->ring_size > PAGE_SIZE*4);
360−
361− offset = amdgpu_mes_ctx_get_offs(ring,
362− AMDGPU_MES_CTX_RING_OFFS);
363− ring->gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
364− ring->ring = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
365− amdgpu_ring_clear_ring(ring);
366−
367− } else if (ring->ring_obj == NULL) {
337+ if (ring->ring_obj == NULL) {
368338 r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
369339 AMDGPU_GEM_DOMAIN_GTT,
370340 &ring->ring_obj,
@@ -401,32 +371,26 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring) · 30 unmodified lines
401371 {
402372
403373 /* Not to finish a ring which is not initialized */
404− if (!(ring->adev) ||
405− (!ring->is_mes_queue && !(ring->adev->rings[ring->idx])))
374+ if (!(ring->adev) || !(ring->adev->rings[ring->idx]))
406375 return;
407376
408377 ring->sched.ready = false;
409378
410− if (!ring->is_mes_queue) {
411− amdgpu_device_wb_free(ring->adev, ring->rptr_offs);
412− amdgpu_device_wb_free(ring->adev, ring->wptr_offs);
379+ amdgpu_device_wb_free(ring->adev, ring->rptr_offs);
380+ amdgpu_device_wb_free(ring->adev, ring->wptr_offs);
413381
414− amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs);
415− amdgpu_device_wb_free(ring->adev, ring->fence_offs);
382+ amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs);
383+ amdgpu_device_wb_free(ring->adev, ring->fence_offs);
416384
417− amdgpu_bo_free_kernel(&ring->ring_obj,
418− &ring->gpu_addr,
419− (void **)&ring->ring);
420− } else {
421− kfree(ring->fence_drv.fences);
422− }
385+ amdgpu_bo_free_kernel(&ring->ring_obj,
386+ &ring->gpu_addr,
387+ (void **)&ring->ring);
423388
424389 dma_fence_put(ring->vmid_wait);
425390 ring->vmid_wait = NULL;
426391 ring->me = 0;
427392
428− if (!ring->is_mes_queue)
429− ring->adev->rings[ring->idx] = NULL;
393+ ring->adev->rings[ring->idx] = NULL;
430394 }
431395
432396 /**
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h+75−15
@@ -164,8 +164,24 @@ void amdgpu_fence_update_start_timestamp(struct amdgpu_ring *ring, uint32_t seq, · 163 unmodified lines
164164
165165 /* provided by hw blocks that expose a ring buffer for commands */
166166 struct amdgpu_ring_funcs {
167+ /**
168+ * @type:
169+ *
170+ * GFX, Compute, SDMA, UVD, VCE, VCN, VPE, KIQ, MES, UMSCH, and CPER
171+ * use ring buffers. The type field just identifies which component the
172+ * ring buffer is associated with.
173+ */
167174 enum amdgpu_ring_type type;
168175 uint32_t align_mask;
176+
177+ /**
178+ * @nop:
179+ *
180+ * Every block in the amdgpu has no-op instructions (e.g., GFX 10
181+ * uses PACKET3(PACKET3_NOP, 0x3FFF), VCN 5 uses VCN_ENC_CMD_NO_OP,
182+ * etc). This field receives the specific no-op for the component
183+ * that initializes the ring.
184+ */
169185 u32 nop;
170186 bool support_64bit_ptrs;
171187 bool no_user_fence;
@@ -241,6 +257,9 @@ struct amdgpu_ring_funcs { · 69 unmodified lines
241257 bool (*is_guilty)(struct amdgpu_ring *ring);
242258 };
243259
260+/**
261+ * amdgpu_ring - Holds ring information
262+ */
244263 struct amdgpu_ring {
245264 struct amdgpu_device *adev;
246265 const struct amdgpu_ring_funcs *funcs;
@@ -252,13 +271,61 @@ struct amdgpu_ring { · 5 unmodified lines
252271 unsigned rptr_offs;
253272 u64 rptr_gpu_addr;
254273 volatile u32 *rptr_cpu_addr;
274+
275+ /**
276+ * @wptr:
277+ *
278+ * This is part of the Ring buffer implementation and represents the
279+ * write pointer. The wptr determines where the host has written.
280+ */
255281 u64 wptr;
282+
283+ /**
284+ * @wptr_old:
285+ *
286+ * Before update wptr with the new value, usually the old value is
287+ * stored in the wptr_old.
288+ */
256289 u64 wptr_old;
257290 unsigned ring_size;
291+
292+ /**
293+ * @max_dw:
294+ *
295+ * Maximum number of DWords for ring allocation. This information is
296+ * provided at the ring initialization time, and each IP block can
297+ * specify a specific value. Check places that invoke
298+ * amdgpu_ring_init() to see the maximum size per block.
299+ */
258300 unsigned max_dw;
301+
302+ /**
303+ * @count_dw:
304+ *
305+ * This value starts with the maximum amount of DWords supported by the
306+ * ring. This value is updated based on the ring manipulation.
307+ */
259308 int count_dw;
260309 uint64_t gpu_addr;
310+
311+ /**
312+ * @ptr_mask:
313+ *
314+ * Some IPs provide support for 64-bit pointers and others for 32-bit
315+ * only; this behavior is component-specific and defined by the field
316+ * support_64bit_ptr. If the IP block supports 64-bits, the mask
317+ * 0xffffffffffffffff is set; otherwise, this value assumes buf_mask.
318+ * Notice that this field is used to keep wptr under a valid range.
319+ */
261320 uint64_t ptr_mask;
321+
322+ /**
323+ * @buf_mask:
324+ *
325+ * Buffer mask is a value used to keep wptr count under its
326+ * thresholding. Buffer mask initialized during the ring buffer
327+ * initialization time, and it is defined as (ring_size / 4) -1.
328+ */
262329 uint32_t buf_mask;
263330 u32 idx;
264331 u32 xcc_id;
@@ -276,6 +343,13 @@ struct amdgpu_ring { · 11 unmodified lines
276343 bool use_pollmem;
277344 unsigned wptr_offs;
278345 u64 wptr_gpu_addr;
346+
347+ /**
348+ * @wptr_cpu_addr:
349+ *
350+ * This is the CPU address pointer in the writeback slot. This is used
351+ * to commit changes to the GPU.
352+ */
279353 volatile u32 *wptr_cpu_addr;
280354 unsigned fence_offs;
281355 u64 fence_gpu_addr;
@@ -297,20 +371,15 @@ struct amdgpu_ring { · 15 unmodified lines
297371 struct dma_fence *vmid_wait;
298372 bool has_compute_vm_bug;
299373 bool no_scheduler;
374+ bool no_user_submission;
300375 int hw_prio;
301376 unsigned num_hw_submission;
302377 atomic_t *sched_score;
303378
304− /* used for mes */
305− bool is_mes_queue;
306− uint32_t hw_queue_id;
307− struct amdgpu_mes_ctx_data *mes_ctx;
308−
309379 bool is_sw_ring;
310380 unsigned int entry_index;
311381 /* store the cached rptr to restore after reset */
312382 uint64_t cached_rptr;
313−
314383 };
315384
316385 #define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))
@@ -435,15 +504,6 @@ static inline void amdgpu_ring_patch_cond_exec(struct amdgpu_ring *ring, · 118 unmodified lines
435504 ring->ring[offset] = cur - offset;
436505 }
437506
438−#define amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset) \
439− (ring->is_mes_queue && ring->mes_ctx ? \
440− (ring->mes_ctx->meta_data_gpu_addr + offset) : 0)
441−
442−#define amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset) \
443− (ring->is_mes_queue && ring->mes_ctx ? \
444− (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \
445− NULL)
446−
447507 int amdgpu_ring_test_helper(struct amdgpu_ring *ring);
448508
449509 void amdgpu_debugfs_ring_init(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h+14−0
@@ -237,6 +237,20 @@ struct amdgpu_rlc_funcs { · 236 unmodified lines
237237 void (*unset_safe_mode)(struct amdgpu_device *adev, int xcc_id);
238238 int (*init)(struct amdgpu_device *adev);
239239 u32 (*get_csb_size)(struct amdgpu_device *adev);
240+
241+ /**
242+ * @get_csb_buffer: Get the clear state to be put into the hardware.
243+ *
244+ * The parameter adev is used to get the CS data and other gfx info,
245+ * and buffer is the RLC CS pointer
246+ *
247+ * Sometimes, the user space puts a request to clear the state in the
248+ * command buffer; this function provides the clear state that gets put
249+ * into the hardware. Note that the driver programs Clear State
250+ * Indirect Buffer (CSB) explicitly when it sets up the kernel rings,
251+ * and it also provides a pointer to it which is used by the firmware
252+ * to load the clear state in some cases.
253+ */
240254 void (*get_csb_buffer)(struct amdgpu_device *adev, volatile u32 *buffer);
241255 int (*get_cp_table_num)(struct amdgpu_device *adev);
242256 int (*resume)(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c+43−65
@@ -26,6 +26,8 @@ · 25 unmodified lines
2626 #include "amdgpu_sdma.h"
2727 #include "amdgpu_ras.h"
2828 #include "amdgpu_reset.h"
29+#include "gc/gc_10_1_0_offset.h"
30+#include "gc/gc_10_3_0_sh_mask.h"
2931
3032 #define AMDGPU_CSA_SDMA_SIZE 64
3133 /* SDMA CSA reside in the 3rd page of CSA */
@@ -76,22 +78,14 @@ uint64_t amdgpu_sdma_get_csa_mc_addr(struct amdgpu_ring *ring, · 44 unmodified lines
7678 if (amdgpu_sriov_vf(adev) || vmid == 0 || !adev->gfx.mcbp)
7779 return 0;
7880
79− if (ring->is_mes_queue) {
80− uint32_t offset = 0;
81+ r = amdgpu_sdma_get_index_from_ring(ring, &index);
8182
82− offset = offsetof(struct amdgpu_mes_ctx_meta_data,
83− sdma[ring->idx].sdma_meta_data);
84− csa_mc_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
85− } else {
86− r = amdgpu_sdma_get_index_from_ring(ring, &index);
87−
88− if (r || index > 31)
89− csa_mc_addr = 0;
90− else
91− csa_mc_addr = amdgpu_csa_vaddr(adev) +
92− AMDGPU_CSA_SDMA_OFFSET +
93− index * AMDGPU_CSA_SDMA_SIZE;
94− }
83+ if (r || index > 31)
84+ csa_mc_addr = 0;
85+ else
86+ csa_mc_addr = amdgpu_csa_vaddr(adev) +
87+ AMDGPU_CSA_SDMA_OFFSET +
88+ index * AMDGPU_CSA_SDMA_SIZE;
9589
9690 return csa_mc_addr;
9791 }
@@ -537,28 +531,38 @@ bool amdgpu_sdma_is_shared_inv_eng(struct amdgpu_device *adev, struct amdgpu_rin · 439 unmodified lines
537531 return false;
538532 }
539533
540−/**
541− * amdgpu_sdma_register_on_reset_callbacks - Register SDMA reset callbacks
542− * @funcs: Pointer to the callback structure containing pre_reset and post_reset functions
543− *
544− * This function allows KFD and AMDGPU to register their own callbacks for handling
545− * pre-reset and post-reset operations for engine reset. These are needed because engine
546− * reset will stop all queues on that engine.
547− */
548−void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct sdma_on_reset_funcs *funcs)
534+static int amdgpu_sdma_soft_reset(struct amdgpu_device *adev, u32 instance_id)
549535 {
550− if (!funcs)
551− return;
552−
553− /* Ensure the reset_callback_list is initialized */
554− if (!adev->sdma.reset_callback_list.next) {
555− INIT_LIST_HEAD(&adev->sdma.reset_callback_list);
536+ struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];
537+ int r = -EOPNOTSUPP;
538+
539+ switch (amdgpu_ip_version(adev, SDMA0_HWIP, 0)) {
540+ case IP_VERSION(4, 4, 2):
541+ case IP_VERSION(4, 4, 4):
542+ case IP_VERSION(4, 4, 5):
543+ /* For SDMA 4.x, use the existing DPM interface for backward compatibility */
544+ r = amdgpu_dpm_reset_sdma(adev, 1 << instance_id);
545+ break;
546+ case IP_VERSION(5, 0, 0):
547+ case IP_VERSION(5, 0, 1):
548+ case IP_VERSION(5, 0, 2):
549+ case IP_VERSION(5, 0, 5):
550+ case IP_VERSION(5, 2, 0):
551+ case IP_VERSION(5, 2, 2):
552+ case IP_VERSION(5, 2, 4):
553+ case IP_VERSION(5, 2, 5):
554+ case IP_VERSION(5, 2, 6):
555+ case IP_VERSION(5, 2, 3):
556+ case IP_VERSION(5, 2, 1):
557+ case IP_VERSION(5, 2, 7):
558+ if (sdma_instance->funcs->soft_reset_kernel_queue)
559+ r = sdma_instance->funcs->soft_reset_kernel_queue(adev, instance_id);
560+ break;
561+ default:
562+ break;
556563 }
557− /* Initialize the list node in the callback structure */
558− INIT_LIST_HEAD(&funcs->list);
559564
560− /* Add the callback structure to the global list */
561− list_add_tail(&funcs->list, &adev->sdma.reset_callback_list);
565+ return r;
562566 }
563567
564568 /**
@@ -566,16 +570,10 @@ void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct · 1 unmodified line
566570 * @adev: Pointer to the AMDGPU device
567571 * @instance_id: ID of the SDMA engine instance to reset
568572 *
569− * This function performs the following steps:
570− * 1. Calls all registered pre_reset callbacks to allow KFD and AMDGPU to save their state.
571− * 2. Resets the specified SDMA engine instance.
572− * 3. Calls all registered post_reset callbacks to allow KFD and AMDGPU to restore their state.
573− *
574573 * Returns: 0 on success, or a negative error code on failure.
575574 */
576575 int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id)
577576 {
578− struct sdma_on_reset_funcs *funcs;
579577 int ret = 0;
580578 struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];
581579 struct amdgpu_ring *gfx_ring = &sdma_instance->ring;
@@ -597,38 +595,18 @@ int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id) · 15 unmodified lines
597595 page_sched_stopped = true;
598596 }
599597
600− /* Invoke all registered pre_reset callbacks */
601− list_for_each_entry(funcs, &adev->sdma.reset_callback_list, list) {
602− if (funcs->pre_reset) {
603− ret = funcs->pre_reset(adev, instance_id);
604− if (ret) {
605− dev_err(adev->dev,
606− "beforeReset callback failed for instance %u: %d\n",
607− instance_id, ret);
608− goto exit;
609− }
610− }
611− }
598+ if (sdma_instance->funcs->stop_kernel_queue)
599+ sdma_instance->funcs->stop_kernel_queue(gfx_ring);
612600
613601 /* Perform the SDMA reset for the specified instance */
614− ret = amdgpu_dpm_reset_sdma(adev, 1 << instance_id);
602+ ret = amdgpu_sdma_soft_reset(adev, instance_id);
615603 if (ret) {
616604 dev_err(adev->dev, "Failed to reset SDMA instance %u\n", instance_id);
617605 goto exit;
618606 }
619607
620− /* Invoke all registered post_reset callbacks */
621− list_for_each_entry(funcs, &adev->sdma.reset_callback_list, list) {
622− if (funcs->post_reset) {
623− ret = funcs->post_reset(adev, instance_id);
624− if (ret) {
625− dev_err(adev->dev,
626− "afterReset callback failed for instance %u: %d\n",
627− instance_id, ret);
628− goto exit;
629− }
630− }
631− }
608+ if (sdma_instance->funcs->start_kernel_queue)
609+ sdma_instance->funcs->start_kernel_queue(gfx_ring);
632610
633611 exit:
634612 /* Restart the scheduler's work queue for the GFX and page rings
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h+9−9
@@ -50,6 +50,12 @@ enum amdgpu_sdma_irq { · 49 unmodified lines
5050
5151 #define NUM_SDMA(x) hweight32(x)
5252
53+struct amdgpu_sdma_funcs {
54+ int (*stop_kernel_queue)(struct amdgpu_ring *ring);
55+ int (*start_kernel_queue)(struct amdgpu_ring *ring);
56+ int (*soft_reset_kernel_queue)(struct amdgpu_device *adev, u32 instance_id);
57+};
58+
5359 struct amdgpu_sdma_instance {
5460 /* SDMA firmware */
5561 const struct firmware *fw;
@@ -68,7 +74,7 @@ struct amdgpu_sdma_instance { · 12 unmodified lines
6874 /* track guilty state of GFX and PAGE queues */
6975 bool gfx_guilty;
7076 bool page_guilty;
71−
77+ const struct amdgpu_sdma_funcs *funcs;
7278 };
7379
7480 enum amdgpu_sdma_ras_memory_id {
@@ -103,13 +109,6 @@ struct amdgpu_sdma_ras { · 28 unmodified lines
103109 struct amdgpu_ras_block_object ras_block;
104110 };
105111
106−struct sdma_on_reset_funcs {
107− int (*pre_reset)(struct amdgpu_device *adev, uint32_t instance_id);
108− int (*post_reset)(struct amdgpu_device *adev, uint32_t instance_id);
109− /* Linked list node to store this structure in a list; */
110− struct list_head list;
111−};
112−
113112 struct amdgpu_sdma {
114113 struct amdgpu_sdma_instance instance[AMDGPU_MAX_SDMA_INSTANCES];
115114 struct amdgpu_irq_src trap_irq;
@@ -131,6 +130,8 @@ struct amdgpu_sdma { · 15 unmodified lines
131130 uint32_t *ip_dump;
132131 uint32_t supported_reset;
133132 struct list_head reset_callback_list;
133+ bool no_user_submission;
134+ bool disable_uq;
134135 };
135136
136137 /*
@@ -170,7 +171,6 @@ struct amdgpu_buffer_funcs { · 33 unmodified lines
170171 uint32_t byte_count);
171172 };
172173
173−void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct sdma_on_reset_funcs *funcs);
174174 int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id);
175175
176176 #define amdgpu_emit_copy_buffer(adev, ib, s, d, b, t) (adev)->mman.buffer_funcs->emit_copy_buffer((ib), (s), (d), (b), (t))
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c+19−6
@@ -45,7 +45,11 @@ · 44 unmodified lines
4545 */
4646 static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev)
4747 {
48− return AMDGPU_VA_RESERVED_SEQ64_START(adev);
48+ u64 addr = AMDGPU_VA_RESERVED_SEQ64_START(adev);
49+
50+ addr = amdgpu_gmc_sign_extend(addr);
51+
52+ return addr;
4953 }
5054
5155 /**
@@ -63,9 +67,9 @@ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) · 11 unmodified lines
6367 int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,
6468 struct amdgpu_bo_va **bo_va)
6569 {
70+ u64 seq64_addr, va_flags;
6671 struct amdgpu_bo *bo;
6772 struct drm_exec exec;
68− u64 seq64_addr;
6973 int r;
7074
7175 bo = adev->seq64.sbo;
@@ -88,9 +92,11 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 16 unmodified lines
8892 goto error;
8993 }
9094
91− seq64_addr = amdgpu_seq64_get_va_base(adev);
95+ seq64_addr = amdgpu_seq64_get_va_base(adev) & AMDGPU_GMC_HOLE_MASK;
96+
97+ va_flags = amdgpu_gem_va_map_flags(adev, AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_MTYPE_UC);
9298 r = amdgpu_vm_bo_map(adev, *bo_va, seq64_addr, 0, AMDGPU_VA_RESERVED_SEQ64_SIZE,
93− AMDGPU_PTE_READABLE);
99+ va_flags);
94100 if (r) {
95101 DRM_ERROR("failed to do bo_map on userq sem, err=%d\n", r);
96102 amdgpu_vm_bo_del(adev, *bo_va);
@@ -156,6 +162,7 @@ error: · 59 unmodified lines
156162 *
157163 * @adev: amdgpu_device pointer
158164 * @va: VA to access the seq in process address space
165+ * @gpu_addr: GPU address to access the seq
159166 * @cpu_addr: CPU address to access the seq
160167 *
161168 * Alloc a 64 bit memory from seq64 pool.
@@ -163,7 +170,8 @@ error: · 1 unmodified line
163170 * Returns:
164171 * 0 on success or a negative error code on failure
165172 */
166−int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 **cpu_addr)
173+int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va,
174+ u64 *gpu_addr, u64 **cpu_addr)
167175 {
168176 unsigned long bit_pos;
169177
@@ -172,7 +180,12 @@ int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 **cpu_addr) · 2 unmodified lines
172180 return -ENOSPC;
173181
174182 __set_bit(bit_pos, adev->seq64.used);
183+
175184 *va = bit_pos * sizeof(u64) + amdgpu_seq64_get_va_base(adev);
185+
186+ if (gpu_addr)
187+ *gpu_addr = bit_pos * sizeof(u64) + adev->seq64.gpu_addr;
188+
176189 *cpu_addr = bit_pos + adev->seq64.cpu_base_addr;
177190
178191 return 0;
@@ -233,7 +246,7 @@ int amdgpu_seq64_init(struct amdgpu_device *adev) · 54 unmodified lines
233246 */
234247 r = amdgpu_bo_create_kernel(adev, AMDGPU_VA_RESERVED_SEQ64_SIZE,
235248 PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
236− &adev->seq64.sbo, NULL,
249+ &adev->seq64.sbo, &adev->seq64.gpu_addr,
237250 (void **)&adev->seq64.cpu_base_addr);
238251 if (r) {
239252 dev_warn(adev->dev, "(%d) create seq64 failed\n", r);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h+2−1
@@ -32,13 +32,14 @@ · 31 unmodified lines
3232 struct amdgpu_seq64 {
3333 struct amdgpu_bo *sbo;
3434 u32 num_sem;
35+ u64 gpu_addr;
3536 u64 *cpu_base_addr;
3637 DECLARE_BITMAP(used, AMDGPU_MAX_SEQ64_SLOTS);
3738 };
3839
3940 void amdgpu_seq64_fini(struct amdgpu_device *adev);
4041 int amdgpu_seq64_init(struct amdgpu_device *adev);
41−int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *gpu_addr, u64 **cpu_addr);
42+int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 *gpu_addr, u64 **cpu_addr);
4243 void amdgpu_seq64_free(struct amdgpu_device *adev, u64 gpu_addr);
4344 int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,
4445 struct amdgpu_bo_va **bo_va);
drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c+2−3
@@ -249,9 +249,8 @@ int amdgpu_sync_resv(struct amdgpu_device *adev, struct amdgpu_sync *sync, · 248 unmodified lines
249249
250250 if (resv == NULL)
251251 return -EINVAL;
252−
253− /* TODO: Use DMA_RESV_USAGE_READ here */
254− dma_resv_for_each_fence(&cursor, resv, DMA_RESV_USAGE_BOOKKEEP, f) {
252+ /* Implicitly sync only to KERNEL, WRITE and READ */
253+ dma_resv_for_each_fence(&cursor, resv, DMA_RESV_USAGE_READ, f) {
255254 dma_fence_chain_for_each(f, f) {
256255 struct dma_fence *tmp = dma_fence_chain_contained(f);
257256
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h+2−0
@@ -25,6 +25,8 @@ · 24 unmodified lines
2525
2626 #include "amdgpu_socbb.h"
2727
28+#define RS64_FW_UC_START_ADDR_LO 0x3000
29+
2830 struct common_firmware_header {
2931 uint32_t size_bytes; /* size of the entire header+image(s) in bytes */
3032 uint32_t header_size_bytes; /* size of just the header in bytes */
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c+919−0
@@ -0,0 +1,919 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright 2023 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+
25+#include <drm/drm_auth.h>
26+#include <drm/drm_exec.h>
27+#include <linux/pm_runtime.h>
28+
29+#include "amdgpu.h"
30+#include "amdgpu_vm.h"
31+#include "amdgpu_userq.h"
32+#include "amdgpu_userq_fence.h"
33+
34+u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev)
35+{
36+ int i;
37+ u32 userq_ip_mask = 0;
38+
39+ for (i = 0; i < AMDGPU_HW_IP_NUM; i++) {
40+ if (adev->userq_funcs[i])
41+ userq_ip_mask |= (1 << i);
42+ }
43+
44+ return userq_ip_mask;
45+}
46+
47+static int
48+amdgpu_userq_unmap_helper(struct amdgpu_userq_mgr *uq_mgr,
49+ struct amdgpu_usermode_queue *queue)
50+{
51+ struct amdgpu_device *adev = uq_mgr->adev;
52+ const struct amdgpu_userq_funcs *userq_funcs =
53+ adev->userq_funcs[queue->queue_type];
54+ int r = 0;
55+
56+ if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
57+ r = userq_funcs->unmap(uq_mgr, queue);
58+ if (r)
59+ queue->state = AMDGPU_USERQ_STATE_HUNG;
60+ else
61+ queue->state = AMDGPU_USERQ_STATE_UNMAPPED;
62+ }
63+ return r;
64+}
65+
66+static int
67+amdgpu_userq_map_helper(struct amdgpu_userq_mgr *uq_mgr,
68+ struct amdgpu_usermode_queue *queue)
69+{
70+ struct amdgpu_device *adev = uq_mgr->adev;
71+ const struct amdgpu_userq_funcs *userq_funcs =
72+ adev->userq_funcs[queue->queue_type];
73+ int r = 0;
74+
75+ if (queue->state == AMDGPU_USERQ_STATE_UNMAPPED) {
76+ r = userq_funcs->map(uq_mgr, queue);
77+ if (r) {
78+ queue->state = AMDGPU_USERQ_STATE_HUNG;
79+ } else {
80+ queue->state = AMDGPU_USERQ_STATE_MAPPED;
81+ }
82+ }
83+ return r;
84+}
85+
86+static void
87+amdgpu_userq_wait_for_last_fence(struct amdgpu_userq_mgr *uq_mgr,
88+ struct amdgpu_usermode_queue *queue)
89+{
90+ struct dma_fence *f = queue->last_fence;
91+ int ret;
92+
93+ if (f && !dma_fence_is_signaled(f)) {
94+ ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100));
95+ if (ret <= 0)
96+ drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n",
97+ f->context, f->seqno);
98+ }
99+}
100+
101+static void
102+amdgpu_userq_cleanup(struct amdgpu_userq_mgr *uq_mgr,
103+ struct amdgpu_usermode_queue *queue,
104+ int queue_id)
105+{
106+ struct amdgpu_device *adev = uq_mgr->adev;
107+ const struct amdgpu_userq_funcs *uq_funcs = adev->userq_funcs[queue->queue_type];
108+
109+ uq_funcs->mqd_destroy(uq_mgr, queue);
110+ amdgpu_userq_fence_driver_free(queue);
111+ idr_remove(&uq_mgr->userq_idr, queue_id);
112+ kfree(queue);
113+}
114+
115+int
116+amdgpu_userq_active(struct amdgpu_userq_mgr *uq_mgr)
117+{
118+ struct amdgpu_usermode_queue *queue;
119+ int queue_id;
120+ int ret = 0;
121+
122+ mutex_lock(&uq_mgr->userq_mutex);
123+ /* Resume all the queues for this process */
124+ idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id)
125+ ret += queue->state == AMDGPU_USERQ_STATE_MAPPED;
126+
127+ mutex_unlock(&uq_mgr->userq_mutex);
128+ return ret;
129+}
130+
131+static struct amdgpu_usermode_queue *
132+amdgpu_userq_find(struct amdgpu_userq_mgr *uq_mgr, int qid)
133+{
134+ return idr_find(&uq_mgr->userq_idr, qid);
135+}
136+
137+void
138+amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *uq_mgr,
139+ struct amdgpu_eviction_fence_mgr *evf_mgr)
140+{
141+ struct amdgpu_eviction_fence *ev_fence;
142+
143+retry:
144+ /* Flush any pending resume work to create ev_fence */
145+ flush_delayed_work(&uq_mgr->resume_work);
146+
147+ mutex_lock(&uq_mgr->userq_mutex);
148+ spin_lock(&evf_mgr->ev_fence_lock);
149+ ev_fence = evf_mgr->ev_fence;
150+ spin_unlock(&evf_mgr->ev_fence_lock);
151+ if (!ev_fence || dma_fence_is_signaled(&ev_fence->base)) {
152+ mutex_unlock(&uq_mgr->userq_mutex);
153+ /*
154+ * Looks like there was no pending resume work,
155+ * add one now to create a valid eviction fence
156+ */
157+ schedule_delayed_work(&uq_mgr->resume_work, 0);
158+ goto retry;
159+ }
160+}
161+
162+int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr,
163+ struct amdgpu_userq_obj *userq_obj,
164+ int size)
165+{
166+ struct amdgpu_device *adev = uq_mgr->adev;
167+ struct amdgpu_bo_param bp;
168+ int r;
169+
170+ memset(&bp, 0, sizeof(bp));
171+ bp.byte_align = PAGE_SIZE;
172+ bp.domain = AMDGPU_GEM_DOMAIN_GTT;
173+ bp.flags = AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
174+ AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
175+ bp.type = ttm_bo_type_kernel;
176+ bp.size = size;
177+ bp.resv = NULL;
178+ bp.bo_ptr_size = sizeof(struct amdgpu_bo);
179+
180+ r = amdgpu_bo_create(adev, &bp, &userq_obj->obj);
181+ if (r) {
182+ drm_file_err(uq_mgr->file, "Failed to allocate BO for userqueue (%d)", r);
183+ return r;
184+ }
185+
186+ r = amdgpu_bo_reserve(userq_obj->obj, true);
187+ if (r) {
188+ drm_file_err(uq_mgr->file, "Failed to reserve BO to map (%d)", r);
189+ goto free_obj;
190+ }
191+
192+ r = amdgpu_ttm_alloc_gart(&(userq_obj->obj)->tbo);
193+ if (r) {
194+ drm_file_err(uq_mgr->file, "Failed to alloc GART for userqueue object (%d)", r);
195+ goto unresv;
196+ }
197+
198+ r = amdgpu_bo_kmap(userq_obj->obj, &userq_obj->cpu_ptr);
199+ if (r) {
200+ drm_file_err(uq_mgr->file, "Failed to map BO for userqueue (%d)", r);
201+ goto unresv;
202+ }
203+
204+ userq_obj->gpu_addr = amdgpu_bo_gpu_offset(userq_obj->obj);
205+ amdgpu_bo_unreserve(userq_obj->obj);
206+ memset(userq_obj->cpu_ptr, 0, size);
207+ return 0;
208+
209+unresv:
210+ amdgpu_bo_unreserve(userq_obj->obj);
211+
212+free_obj:
213+ amdgpu_bo_unref(&userq_obj->obj);
214+ return r;
215+}
216+
217+void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr,
218+ struct amdgpu_userq_obj *userq_obj)
219+{
220+ amdgpu_bo_kunmap(userq_obj->obj);
221+ amdgpu_bo_unref(&userq_obj->obj);
222+}
223+
224+uint64_t
225+amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr,
226+ struct amdgpu_db_info *db_info,
227+ struct drm_file *filp)
228+{
229+ uint64_t index;
230+ struct drm_gem_object *gobj;
231+ struct amdgpu_userq_obj *db_obj = db_info->db_obj;
232+ int r, db_size;
233+
234+ gobj = drm_gem_object_lookup(filp, db_info->doorbell_handle);
235+ if (gobj == NULL) {
236+ drm_file_err(uq_mgr->file, "Can't find GEM object for doorbell\n");
237+ return -EINVAL;
238+ }
239+
240+ db_obj->obj = amdgpu_bo_ref(gem_to_amdgpu_bo(gobj));
241+ drm_gem_object_put(gobj);
242+
243+ /* Pin the BO before generating the index, unpin in queue destroy */
244+ r = amdgpu_bo_pin(db_obj->obj, AMDGPU_GEM_DOMAIN_DOORBELL);
245+ if (r) {
246+ drm_file_err(uq_mgr->file, "[Usermode queues] Failed to pin doorbell object\n");
247+ goto unref_bo;
248+ }
249+
250+ r = amdgpu_bo_reserve(db_obj->obj, true);
251+ if (r) {
252+ drm_file_err(uq_mgr->file, "[Usermode queues] Failed to pin doorbell object\n");
253+ goto unpin_bo;
254+ }
255+
256+ switch (db_info->queue_type) {
257+ case AMDGPU_HW_IP_GFX:
258+ case AMDGPU_HW_IP_COMPUTE:
259+ case AMDGPU_HW_IP_DMA:
260+ db_size = sizeof(u64);
261+ break;
262+
263+ case AMDGPU_HW_IP_VCN_ENC:
264+ db_size = sizeof(u32);
265+ db_info->doorbell_offset += AMDGPU_NAVI10_DOORBELL64_VCN0_1 << 1;
266+ break;
267+
268+ case AMDGPU_HW_IP_VPE:
269+ db_size = sizeof(u32);
270+ db_info->doorbell_offset += AMDGPU_NAVI10_DOORBELL64_VPE << 1;
271+ break;
272+
273+ default:
274+ drm_file_err(uq_mgr->file, "[Usermode queues] IP %d not support\n",
275+ db_info->queue_type);
276+ r = -EINVAL;
277+ goto unpin_bo;
278+ }
279+
280+ index = amdgpu_doorbell_index_on_bar(uq_mgr->adev, db_obj->obj,
281+ db_info->doorbell_offset, db_size);
282+ drm_dbg_driver(adev_to_drm(uq_mgr->adev),
283+ "[Usermode queues] doorbell index=%lld\n", index);
284+ amdgpu_bo_unreserve(db_obj->obj);
285+ return index;
286+
287+unpin_bo:
288+ amdgpu_bo_unpin(db_obj->obj);
289+
290+unref_bo:
291+ amdgpu_bo_unref(&db_obj->obj);
292+ return r;
293+}
294+
295+static int
296+amdgpu_userq_destroy(struct drm_file *filp, int queue_id)
297+{
298+ struct amdgpu_fpriv *fpriv = filp->driver_priv;
299+ struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
300+ struct amdgpu_device *adev = uq_mgr->adev;
301+ struct amdgpu_usermode_queue *queue;
302+ int r = 0;
303+
304+ cancel_delayed_work(&uq_mgr->resume_work);
305+ mutex_lock(&uq_mgr->userq_mutex);
306+
307+ queue = amdgpu_userq_find(uq_mgr, queue_id);
308+ if (!queue) {
309+ drm_dbg_driver(adev_to_drm(uq_mgr->adev), "Invalid queue id to destroy\n");
310+ mutex_unlock(&uq_mgr->userq_mutex);
311+ return -EINVAL;
312+ }
313+ amdgpu_userq_wait_for_last_fence(uq_mgr, queue);
314+ r = amdgpu_userq_unmap_helper(uq_mgr, queue);
315+ amdgpu_bo_unpin(queue->db_obj.obj);
316+ amdgpu_bo_unref(&queue->db_obj.obj);
317+ amdgpu_userq_cleanup(uq_mgr, queue, queue_id);
318+ mutex_unlock(&uq_mgr->userq_mutex);
319+
320+ pm_runtime_mark_last_busy(adev_to_drm(adev)->dev);
321+ pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
322+
323+ return r;
324+}
325+
326+static int amdgpu_userq_priority_permit(struct drm_file *filp,
327+ int priority)
328+{
329+ if (priority < AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_HIGH)
330+ return 0;
331+
332+ if (capable(CAP_SYS_NICE))
333+ return 0;
334+
335+ if (drm_is_current_master(filp))
336+ return 0;
337+
338+ return -EACCES;
339+}
340+
341+static int
342+amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
343+{
344+ struct amdgpu_fpriv *fpriv = filp->driver_priv;
345+ struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
346+ struct amdgpu_device *adev = uq_mgr->adev;
347+ const struct amdgpu_userq_funcs *uq_funcs;
348+ struct amdgpu_usermode_queue *queue;
349+ struct amdgpu_db_info db_info;
350+ bool skip_map_queue;
351+ uint64_t index;
352+ int qid, r = 0;
353+ int priority =
354+ (args->in.flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_MASK) >>
355+ AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_SHIFT;
356+
357+ /* Usermode queues are only supported for GFX IP as of now */
358+ if (args->in.ip_type != AMDGPU_HW_IP_GFX &&
359+ args->in.ip_type != AMDGPU_HW_IP_DMA &&
360+ args->in.ip_type != AMDGPU_HW_IP_COMPUTE) {
361+ drm_file_err(uq_mgr->file, "Usermode queue doesn't support IP type %u\n",
362+ args->in.ip_type);
363+ return -EINVAL;
364+ }
365+
366+ r = amdgpu_userq_priority_permit(filp, priority);
367+ if (r)
368+ return r;
369+
370+ if ((args->in.flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE) &&
371+ (args->in.ip_type != AMDGPU_HW_IP_GFX) &&
372+ (args->in.ip_type != AMDGPU_HW_IP_COMPUTE) &&
373+ !amdgpu_is_tmz(adev)) {
374+ drm_file_err(uq_mgr->file, "Secure only supported on GFX/Compute queues\n");
375+ return -EINVAL;
376+ }
377+
378+ r = pm_runtime_get_sync(adev_to_drm(adev)->dev);
379+ if (r < 0) {
380+ drm_file_err(uq_mgr->file, "pm_runtime_get_sync() failed for userqueue create\n");
381+ pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
382+ return r;
383+ }
384+
385+ /*
386+ * There could be a situation that we are creating a new queue while
387+ * the other queues under this UQ_mgr are suspended. So if there is any
388+ * resume work pending, wait for it to get done.
389+ *
390+ * This will also make sure we have a valid eviction fence ready to be used.
391+ */
392+ amdgpu_userq_ensure_ev_fence(&fpriv->userq_mgr, &fpriv->evf_mgr);
393+
394+ uq_funcs = adev->userq_funcs[args->in.ip_type];
395+ if (!uq_funcs) {
396+ drm_file_err(uq_mgr->file, "Usermode queue is not supported for this IP (%u)\n",
397+ args->in.ip_type);
398+ r = -EINVAL;
399+ goto unlock;
400+ }
401+
402+ queue = kzalloc(sizeof(struct amdgpu_usermode_queue), GFP_KERNEL);
403+ if (!queue) {
404+ drm_file_err(uq_mgr->file, "Failed to allocate memory for queue\n");
405+ r = -ENOMEM;
406+ goto unlock;
407+ }
408+ queue->doorbell_handle = args->in.doorbell_handle;
409+ queue->queue_type = args->in.ip_type;
410+ queue->vm = &fpriv->vm;
411+ queue->priority = priority;
412+
413+ db_info.queue_type = queue->queue_type;
414+ db_info.doorbell_handle = queue->doorbell_handle;
415+ db_info.db_obj = &queue->db_obj;
416+ db_info.doorbell_offset = args->in.doorbell_offset;
417+
418+ /* Convert relative doorbell offset into absolute doorbell index */
419+ index = amdgpu_userq_get_doorbell_index(uq_mgr, &db_info, filp);
420+ if (index == (uint64_t)-EINVAL) {
421+ drm_file_err(uq_mgr->file, "Failed to get doorbell for queue\n");
422+ kfree(queue);
423+ goto unlock;
424+ }
425+
426+ queue->doorbell_index = index;
427+ xa_init_flags(&queue->fence_drv_xa, XA_FLAGS_ALLOC);
428+ r = amdgpu_userq_fence_driver_alloc(adev, queue);
429+ if (r) {
430+ drm_file_err(uq_mgr->file, "Failed to alloc fence driver\n");
431+ goto unlock;
432+ }
433+
434+ r = uq_funcs->mqd_create(uq_mgr, &args->in, queue);
435+ if (r) {
436+ drm_file_err(uq_mgr->file, "Failed to create Queue\n");
437+ amdgpu_userq_fence_driver_free(queue);
438+ kfree(queue);
439+ goto unlock;
440+ }
441+
442+
443+ qid = idr_alloc(&uq_mgr->userq_idr, queue, 1, AMDGPU_MAX_USERQ_COUNT, GFP_KERNEL);
444+ if (qid < 0) {
445+ drm_file_err(uq_mgr->file, "Failed to allocate a queue id\n");
446+ amdgpu_userq_fence_driver_free(queue);
447+ uq_funcs->mqd_destroy(uq_mgr, queue);
448+ kfree(queue);
449+ r = -ENOMEM;
450+ goto unlock;
451+ }
452+
453+ /* don't map the queue if scheduling is halted */
454+ mutex_lock(&adev->userq_mutex);
455+ if (adev->userq_halt_for_enforce_isolation &&
456+ ((queue->queue_type == AMDGPU_HW_IP_GFX) ||
457+ (queue->queue_type == AMDGPU_HW_IP_COMPUTE)))
458+ skip_map_queue = true;
459+ else
460+ skip_map_queue = false;
461+ if (!skip_map_queue) {
462+ r = amdgpu_userq_map_helper(uq_mgr, queue);
463+ if (r) {
464+ mutex_unlock(&adev->userq_mutex);
465+ drm_file_err(uq_mgr->file, "Failed to map Queue\n");
466+ idr_remove(&uq_mgr->userq_idr, qid);
467+ amdgpu_userq_fence_driver_free(queue);
468+ uq_funcs->mqd_destroy(uq_mgr, queue);
469+ kfree(queue);
470+ goto unlock;
471+ }
472+ }
473+ mutex_unlock(&adev->userq_mutex);
474+
475+
476+ args->out.queue_id = qid;
477+
478+unlock:
479+ mutex_unlock(&uq_mgr->userq_mutex);
480+
481+ return r;
482+}
483+
484+int amdgpu_userq_ioctl(struct drm_device *dev, void *data,
485+ struct drm_file *filp)
486+{
487+ union drm_amdgpu_userq *args = data;
488+ int r;
489+
490+ switch (args->in.op) {
491+ case AMDGPU_USERQ_OP_CREATE:
492+ if (args->in.flags & ~(AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_MASK |
493+ AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE))
494+ return -EINVAL;
495+ r = amdgpu_userq_create(filp, args);
496+ if (r)
497+ drm_file_err(filp, "Failed to create usermode queue\n");
498+ break;
499+
500+ case AMDGPU_USERQ_OP_FREE:
501+ if (args->in.ip_type ||
502+ args->in.doorbell_handle ||
503+ args->in.doorbell_offset ||
504+ args->in.flags ||
505+ args->in.queue_va ||
506+ args->in.queue_size ||
507+ args->in.rptr_va ||
508+ args->in.wptr_va ||
509+ args->in.wptr_va ||
510+ args->in.mqd ||
511+ args->in.mqd_size)
512+ return -EINVAL;
513+ r = amdgpu_userq_destroy(filp, args->in.queue_id);
514+ if (r)
515+ drm_file_err(filp, "Failed to destroy usermode queue\n");
516+ break;
517+
518+ default:
519+ drm_dbg_driver(dev, "Invalid user queue op specified: %d\n", args->in.op);
520+ return -EINVAL;
521+ }
522+
523+ return r;
524+}
525+
526+static int
527+amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
528+{
529+ struct amdgpu_usermode_queue *queue;
530+ int queue_id;
531+ int ret = 0, r;
532+
533+ /* Resume all the queues for this process */
534+ idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) {
535+ r = amdgpu_userq_map_helper(uq_mgr, queue);
536+ if (r)
537+ ret = r;
538+ }
539+
540+ if (ret)
541+ drm_file_err(uq_mgr->file, "Failed to map all the queues\n");
542+ return ret;
543+}
544+
545+static int
546+amdgpu_userq_validate_vm_bo(void *_unused, struct amdgpu_bo *bo)
547+{
548+ struct ttm_operation_ctx ctx = { false, false };
549+ int ret;
550+
551+ amdgpu_bo_placement_from_domain(bo, bo->allowed_domains);
552+
553+ ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
554+ if (ret)
555+ DRM_ERROR("Fail to validate\n");
556+
557+ return ret;
558+}
559+
560+static int
561+amdgpu_userq_validate_bos(struct amdgpu_userq_mgr *uq_mgr)
562+{
563+ struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
564+ struct amdgpu_vm *vm = &fpriv->vm;
565+ struct amdgpu_device *adev = uq_mgr->adev;
566+ struct amdgpu_bo_va *bo_va;
567+ struct ww_acquire_ctx *ticket;
568+ struct drm_exec exec;
569+ struct amdgpu_bo *bo;
570+ struct dma_resv *resv;
571+ bool clear, unlock;
572+ int ret = 0;
573+
574+ drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);
575+ drm_exec_until_all_locked(&exec) {
576+ ret = amdgpu_vm_lock_pd(vm, &exec, 2);
577+ drm_exec_retry_on_contention(&exec);
578+ if (unlikely(ret)) {
579+ drm_file_err(uq_mgr->file, "Failed to lock PD\n");
580+ goto unlock_all;
581+ }
582+
583+ /* Lock the done list */
584+ list_for_each_entry(bo_va, &vm->done, base.vm_status) {
585+ bo = bo_va->base.bo;
586+ if (!bo)
587+ continue;
588+
589+ ret = drm_exec_lock_obj(&exec, &bo->tbo.base);
590+ drm_exec_retry_on_contention(&exec);
591+ if (unlikely(ret))
592+ goto unlock_all;
593+ }
594+ }
595+
596+ spin_lock(&vm->status_lock);
597+ while (!list_empty(&vm->moved)) {
598+ bo_va = list_first_entry(&vm->moved, struct amdgpu_bo_va,
599+ base.vm_status);
600+ spin_unlock(&vm->status_lock);
601+
602+ /* Per VM BOs never need to bo cleared in the page tables */
603+ ret = amdgpu_vm_bo_update(adev, bo_va, false);
604+ if (ret)
605+ goto unlock_all;
606+ spin_lock(&vm->status_lock);
607+ }
608+
609+ ticket = &exec.ticket;
610+ while (!list_empty(&vm->invalidated)) {
611+ bo_va = list_first_entry(&vm->invalidated, struct amdgpu_bo_va,
612+ base.vm_status);
613+ resv = bo_va->base.bo->tbo.base.resv;
614+ spin_unlock(&vm->status_lock);
615+
616+ bo = bo_va->base.bo;
617+ ret = amdgpu_userq_validate_vm_bo(NULL, bo);
618+ if (ret) {
619+ drm_file_err(uq_mgr->file, "Failed to validate BO\n");
620+ goto unlock_all;
621+ }
622+
623+ /* Try to reserve the BO to avoid clearing its ptes */
624+ if (!adev->debug_vm && dma_resv_trylock(resv)) {
625+ clear = false;
626+ unlock = true;
627+ /* The caller is already holding the reservation lock */
628+ } else if (dma_resv_locking_ctx(resv) == ticket) {
629+ clear = false;
630+ unlock = false;
631+ /* Somebody else is using the BO right now */
632+ } else {
633+ clear = true;
634+ unlock = false;
635+ }
636+
637+ ret = amdgpu_vm_bo_update(adev, bo_va, clear);
638+
639+ if (unlock)
640+ dma_resv_unlock(resv);
641+ if (ret)
642+ goto unlock_all;
643+
644+ spin_lock(&vm->status_lock);
645+ }
646+ spin_unlock(&vm->status_lock);
647+
648+ ret = amdgpu_eviction_fence_replace_fence(&fpriv->evf_mgr, &exec);
649+ if (ret)
650+ drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n");
651+
652+unlock_all:
653+ drm_exec_fini(&exec);
654+ return ret;
655+}
656+
657+static void amdgpu_userq_restore_worker(struct work_struct *work)
658+{
659+ struct amdgpu_userq_mgr *uq_mgr = work_to_uq_mgr(work, resume_work.work);
660+ struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
661+ int ret;
662+
663+ flush_work(&fpriv->evf_mgr.suspend_work.work);
664+
665+ mutex_lock(&uq_mgr->userq_mutex);
666+
667+ ret = amdgpu_userq_validate_bos(uq_mgr);
668+ if (ret) {
669+ drm_file_err(uq_mgr->file, "Failed to validate BOs to restore\n");
670+ goto unlock;
671+ }
672+
673+ ret = amdgpu_userq_restore_all(uq_mgr);
674+ if (ret) {
675+ drm_file_err(uq_mgr->file, "Failed to restore all queues\n");
676+ goto unlock;
677+ }
678+
679+unlock:
680+ mutex_unlock(&uq_mgr->userq_mutex);
681+}
682+
683+static int
684+amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr)
685+{
686+ struct amdgpu_usermode_queue *queue;
687+ int queue_id;
688+ int ret = 0, r;
689+
690+ /* Try to unmap all the queues in this process ctx */
691+ idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) {
692+ r = amdgpu_userq_unmap_helper(uq_mgr, queue);
693+ if (r)
694+ ret = r;
695+ }
696+
697+ if (ret)
698+ drm_file_err(uq_mgr->file, "Couldn't unmap all the queues\n");
699+ return ret;
700+}
701+
702+static int
703+amdgpu_userq_wait_for_signal(struct amdgpu_userq_mgr *uq_mgr)
704+{
705+ struct amdgpu_usermode_queue *queue;
706+ int queue_id, ret;
707+
708+ idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) {
709+ struct dma_fence *f = queue->last_fence;
710+
711+ if (!f || dma_fence_is_signaled(f))
712+ continue;
713+ ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100));
714+ if (ret <= 0) {
715+ drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n",
716+ f->context, f->seqno);
717+ return -ETIMEDOUT;
718+ }
719+ }
720+
721+ return 0;
722+}
723+
724+void
725+amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr,
726+ struct amdgpu_eviction_fence *ev_fence)
727+{
728+ int ret;
729+ struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
730+ struct amdgpu_eviction_fence_mgr *evf_mgr = &fpriv->evf_mgr;
731+
732+ /* Wait for any pending userqueue fence work to finish */
733+ ret = amdgpu_userq_wait_for_signal(uq_mgr);
734+ if (ret) {
735+ drm_file_err(uq_mgr->file, "Not evicting userqueue, timeout waiting for work\n");
736+ return;
737+ }
738+
739+ ret = amdgpu_userq_evict_all(uq_mgr);
740+ if (ret) {
741+ drm_file_err(uq_mgr->file, "Failed to evict userqueue\n");
742+ return;
743+ }
744+
745+ /* Signal current eviction fence */
746+ amdgpu_eviction_fence_signal(evf_mgr, ev_fence);
747+
748+ if (evf_mgr->fd_closing) {
749+ cancel_delayed_work(&uq_mgr->resume_work);
750+ return;
751+ }
752+
753+ /* Schedule a resume work */
754+ schedule_delayed_work(&uq_mgr->resume_work, 0);
755+}
756+
757+int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv,
758+ struct amdgpu_device *adev)
759+{
760+ mutex_init(&userq_mgr->userq_mutex);
761+ idr_init_base(&userq_mgr->userq_idr, 1);
762+ userq_mgr->adev = adev;
763+ userq_mgr->file = file_priv;
764+
765+ mutex_lock(&adev->userq_mutex);
766+ list_add(&userq_mgr->list, &adev->userq_mgr_list);
767+ mutex_unlock(&adev->userq_mutex);
768+
769+ INIT_DELAYED_WORK(&userq_mgr->resume_work, amdgpu_userq_restore_worker);
770+ return 0;
771+}
772+
773+void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr)
774+{
775+ struct amdgpu_device *adev = userq_mgr->adev;
776+ struct amdgpu_usermode_queue *queue;
777+ struct amdgpu_userq_mgr *uqm, *tmp;
778+ uint32_t queue_id;
779+
780+ cancel_delayed_work(&userq_mgr->resume_work);
781+
782+ mutex_lock(&userq_mgr->userq_mutex);
783+ idr_for_each_entry(&userq_mgr->userq_idr, queue, queue_id) {
784+ amdgpu_userq_wait_for_last_fence(userq_mgr, queue);
785+ amdgpu_userq_unmap_helper(userq_mgr, queue);
786+ amdgpu_userq_cleanup(userq_mgr, queue, queue_id);
787+ }
788+ mutex_lock(&adev->userq_mutex);
789+ list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
790+ if (uqm == userq_mgr) {
791+ list_del(&uqm->list);
792+ break;
793+ }
794+ }
795+ mutex_unlock(&adev->userq_mutex);
796+ idr_destroy(&userq_mgr->userq_idr);
797+ mutex_unlock(&userq_mgr->userq_mutex);
798+ mutex_destroy(&userq_mgr->userq_mutex);
799+}
800+
801+int amdgpu_userq_suspend(struct amdgpu_device *adev)
802+{
803+ u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
804+ struct amdgpu_usermode_queue *queue;
805+ struct amdgpu_userq_mgr *uqm, *tmp;
806+ int queue_id;
807+ int ret = 0, r;
808+
809+ if (!ip_mask)
810+ return 0;
811+
812+ mutex_lock(&adev->userq_mutex);
813+ list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
814+ cancel_delayed_work_sync(&uqm->resume_work);
815+ mutex_lock(&uqm->userq_mutex);
816+ idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
817+ r = amdgpu_userq_unmap_helper(uqm, queue);
818+ if (r)
819+ ret = r;
820+ }
821+ mutex_unlock(&uqm->userq_mutex);
822+ }
823+ mutex_unlock(&adev->userq_mutex);
824+ return ret;
825+}
826+
827+int amdgpu_userq_resume(struct amdgpu_device *adev)
828+{
829+ u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
830+ struct amdgpu_usermode_queue *queue;
831+ struct amdgpu_userq_mgr *uqm, *tmp;
832+ int queue_id;
833+ int ret = 0, r;
834+
835+ if (!ip_mask)
836+ return 0;
837+
838+ mutex_lock(&adev->userq_mutex);
839+ list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
840+ mutex_lock(&uqm->userq_mutex);
841+ idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
842+ r = amdgpu_userq_map_helper(uqm, queue);
843+ if (r)
844+ ret = r;
845+ }
846+ mutex_unlock(&uqm->userq_mutex);
847+ }
848+ mutex_unlock(&adev->userq_mutex);
849+ return ret;
850+}
851+
852+int amdgpu_userq_stop_sched_for_enforce_isolation(struct amdgpu_device *adev,
853+ u32 idx)
854+{
855+ u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
856+ struct amdgpu_usermode_queue *queue;
857+ struct amdgpu_userq_mgr *uqm, *tmp;
858+ int queue_id;
859+ int ret = 0, r;
860+
861+ /* only need to stop gfx/compute */
862+ if (!(ip_mask & ((1 << AMDGPU_HW_IP_GFX) | (1 << AMDGPU_HW_IP_COMPUTE))))
863+ return 0;
864+
865+ mutex_lock(&adev->userq_mutex);
866+ if (adev->userq_halt_for_enforce_isolation)
867+ dev_warn(adev->dev, "userq scheduling already stopped!\n");
868+ adev->userq_halt_for_enforce_isolation = true;
869+ list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
870+ cancel_delayed_work_sync(&uqm->resume_work);
871+ mutex_lock(&uqm->userq_mutex);
872+ idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
873+ if (((queue->queue_type == AMDGPU_HW_IP_GFX) ||
874+ (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) &&
875+ (queue->xcp_id == idx)) {
876+ r = amdgpu_userq_unmap_helper(uqm, queue);
877+ if (r)
878+ ret = r;
879+ }
880+ }
881+ mutex_unlock(&uqm->userq_mutex);
882+ }
883+ mutex_unlock(&adev->userq_mutex);
884+ return ret;
885+}
886+
887+int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev,
888+ u32 idx)
889+{
890+ u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
891+ struct amdgpu_usermode_queue *queue;
892+ struct amdgpu_userq_mgr *uqm, *tmp;
893+ int queue_id;
894+ int ret = 0, r;
895+
896+ /* only need to stop gfx/compute */
897+ if (!(ip_mask & ((1 << AMDGPU_HW_IP_GFX) | (1 << AMDGPU_HW_IP_COMPUTE))))
898+ return 0;
899+
900+ mutex_lock(&adev->userq_mutex);
901+ if (!adev->userq_halt_for_enforce_isolation)
902+ dev_warn(adev->dev, "userq scheduling already started!\n");
903+ adev->userq_halt_for_enforce_isolation = false;
904+ list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
905+ mutex_lock(&uqm->userq_mutex);
906+ idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
907+ if (((queue->queue_type == AMDGPU_HW_IP_GFX) ||
908+ (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) &&
909+ (queue->xcp_id == idx)) {
910+ r = amdgpu_userq_map_helper(uqm, queue);
911+ if (r)
912+ ret = r;
913+ }
914+ }
915+ mutex_unlock(&uqm->userq_mutex);
916+ }
917+ mutex_unlock(&adev->userq_mutex);
918+ return ret;
919+}
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h+135−0
@@ -0,0 +1,135 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright 2023 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+
25+#ifndef AMDGPU_USERQ_H_
26+#define AMDGPU_USERQ_H_
27+#include "amdgpu_eviction_fence.h"
28+
29+#define AMDGPU_MAX_USERQ_COUNT 512
30+
31+#define to_ev_fence(f) container_of(f, struct amdgpu_eviction_fence, base)
32+#define uq_mgr_to_fpriv(u) container_of(u, struct amdgpu_fpriv, userq_mgr)
33+#define work_to_uq_mgr(w, name) container_of(w, struct amdgpu_userq_mgr, name)
34+
35+enum amdgpu_userq_state {
36+ AMDGPU_USERQ_STATE_UNMAPPED = 0,
37+ AMDGPU_USERQ_STATE_MAPPED,
38+ AMDGPU_USERQ_STATE_PREEMPTED,
39+ AMDGPU_USERQ_STATE_HUNG,
40+};
41+
42+struct amdgpu_mqd_prop;
43+
44+struct amdgpu_userq_obj {
45+ void *cpu_ptr;
46+ uint64_t gpu_addr;
47+ struct amdgpu_bo *obj;
48+};
49+
50+struct amdgpu_usermode_queue {
51+ int queue_type;
52+ enum amdgpu_userq_state state;
53+ uint64_t doorbell_handle;
54+ uint64_t doorbell_index;
55+ uint64_t flags;
56+ struct amdgpu_mqd_prop *userq_prop;
57+ struct amdgpu_userq_mgr *userq_mgr;
58+ struct amdgpu_vm *vm;
59+ struct amdgpu_userq_obj mqd;
60+ struct amdgpu_userq_obj db_obj;
61+ struct amdgpu_userq_obj fw_obj;
62+ struct amdgpu_userq_obj wptr_obj;
63+ struct xarray fence_drv_xa;
64+ struct amdgpu_userq_fence_driver *fence_drv;
65+ struct dma_fence *last_fence;
66+ u32 xcp_id;
67+ int priority;
68+};
69+
70+struct amdgpu_userq_funcs {
71+ int (*mqd_create)(struct amdgpu_userq_mgr *uq_mgr,
72+ struct drm_amdgpu_userq_in *args,
73+ struct amdgpu_usermode_queue *queue);
74+ void (*mqd_destroy)(struct amdgpu_userq_mgr *uq_mgr,
75+ struct amdgpu_usermode_queue *uq);
76+ int (*unmap)(struct amdgpu_userq_mgr *uq_mgr,
77+ struct amdgpu_usermode_queue *queue);
78+ int (*map)(struct amdgpu_userq_mgr *uq_mgr,
79+ struct amdgpu_usermode_queue *queue);
80+};
81+
82+/* Usermode queues for gfx */
83+struct amdgpu_userq_mgr {
84+ struct idr userq_idr;
85+ struct mutex userq_mutex;
86+ struct amdgpu_device *adev;
87+ struct delayed_work resume_work;
88+ struct list_head list;
89+ struct drm_file *file;
90+};
91+
92+struct amdgpu_db_info {
93+ uint64_t doorbell_handle;
94+ uint32_t queue_type;
95+ uint32_t doorbell_offset;
96+ struct amdgpu_userq_obj *db_obj;
97+};
98+
99+int amdgpu_userq_ioctl(struct drm_device *dev, void *data, struct drm_file *filp);
100+
101+int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv,
102+ struct amdgpu_device *adev);
103+
104+void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr);
105+
106+int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr,
107+ struct amdgpu_userq_obj *userq_obj,
108+ int size);
109+
110+void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr,
111+ struct amdgpu_userq_obj *userq_obj);
112+
113+void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr,
114+ struct amdgpu_eviction_fence *ev_fence);
115+
116+int amdgpu_userq_active(struct amdgpu_userq_mgr *uq_mgr);
117+
118+void amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *userq_mgr,
119+ struct amdgpu_eviction_fence_mgr *evf_mgr);
120+
121+uint64_t amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr,
122+ struct amdgpu_db_info *db_info,
123+ struct drm_file *filp);
124+
125+u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev);
126+
127+int amdgpu_userq_suspend(struct amdgpu_device *adev);
128+int amdgpu_userq_resume(struct amdgpu_device *adev);
129+
130+int amdgpu_userq_stop_sched_for_enforce_isolation(struct amdgpu_device *adev,
131+ u32 idx);
132+int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev,
133+ u32 idx);
134+
135+#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c+966−0
@@ -0,0 +1,966 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright 2023 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+
25+#include <linux/kref.h>
26+#include <linux/slab.h>
27+#include <linux/dma-fence-unwrap.h>
28+
29+#include <drm/drm_exec.h>
30+#include <drm/drm_syncobj.h>
31+
32+#include "amdgpu.h"
33+#include "amdgpu_userq_fence.h"
34+
35+static const struct dma_fence_ops amdgpu_userq_fence_ops;
36+static struct kmem_cache *amdgpu_userq_fence_slab;
37+
38+int amdgpu_userq_fence_slab_init(void)
39+{
40+ amdgpu_userq_fence_slab = kmem_cache_create("amdgpu_userq_fence",
41+ sizeof(struct amdgpu_userq_fence),
42+ 0,
43+ SLAB_HWCACHE_ALIGN,
44+ NULL);
45+ if (!amdgpu_userq_fence_slab)
46+ return -ENOMEM;
47+
48+ return 0;
49+}
50+
51+void amdgpu_userq_fence_slab_fini(void)
52+{
53+ rcu_barrier();
54+ kmem_cache_destroy(amdgpu_userq_fence_slab);
55+}
56+
57+static inline struct amdgpu_userq_fence *to_amdgpu_userq_fence(struct dma_fence *f)
58+{
59+ if (!f || f->ops != &amdgpu_userq_fence_ops)
60+ return NULL;
61+
62+ return container_of(f, struct amdgpu_userq_fence, base);
63+}
64+
65+static u64 amdgpu_userq_fence_read(struct amdgpu_userq_fence_driver *fence_drv)
66+{
67+ return le64_to_cpu(*fence_drv->cpu_addr);
68+}
69+
70+int amdgpu_userq_fence_driver_alloc(struct amdgpu_device *adev,
71+ struct amdgpu_usermode_queue *userq)
72+{
73+ struct amdgpu_userq_fence_driver *fence_drv;
74+ unsigned long flags;
75+ int r;
76+
77+ fence_drv = kzalloc(sizeof(*fence_drv), GFP_KERNEL);
78+ if (!fence_drv)
79+ return -ENOMEM;
80+
81+ /* Acquire seq64 memory */
82+ r = amdgpu_seq64_alloc(adev, &fence_drv->va, &fence_drv->gpu_addr,
83+ &fence_drv->cpu_addr);
84+ if (r)
85+ goto free_fence_drv;
86+
87+ memset(fence_drv->cpu_addr, 0, sizeof(u64));
88+
89+ kref_init(&fence_drv->refcount);
90+ INIT_LIST_HEAD(&fence_drv->fences);
91+ spin_lock_init(&fence_drv->fence_list_lock);
92+
93+ fence_drv->adev = adev;
94+ fence_drv->context = dma_fence_context_alloc(1);
95+ get_task_comm(fence_drv->timeline_name, current);
96+
97+ xa_lock_irqsave(&adev->userq_xa, flags);
98+ r = xa_err(__xa_store(&adev->userq_xa, userq->doorbell_index,
99+ fence_drv, GFP_KERNEL));
100+ xa_unlock_irqrestore(&adev->userq_xa, flags);
101+ if (r)
102+ goto free_seq64;
103+
104+ userq->fence_drv = fence_drv;
105+
106+ return 0;
107+
108+free_seq64:
109+ amdgpu_seq64_free(adev, fence_drv->va);
110+free_fence_drv:
111+ kfree(fence_drv);
112+
113+ return r;
114+}
115+
116+static void amdgpu_userq_walk_and_drop_fence_drv(struct xarray *xa)
117+{
118+ struct amdgpu_userq_fence_driver *fence_drv;
119+ unsigned long index;
120+
121+ if (xa_empty(xa))
122+ return;
123+
124+ xa_lock(xa);
125+ xa_for_each(xa, index, fence_drv) {
126+ __xa_erase(xa, index);
127+ amdgpu_userq_fence_driver_put(fence_drv);
128+ }
129+
130+ xa_unlock(xa);
131+}
132+
133+void
134+amdgpu_userq_fence_driver_free(struct amdgpu_usermode_queue *userq)
135+{
136+ amdgpu_userq_walk_and_drop_fence_drv(&userq->fence_drv_xa);
137+ xa_destroy(&userq->fence_drv_xa);
138+ /* Drop the fence_drv reference held by user queue */
139+ amdgpu_userq_fence_driver_put(userq->fence_drv);
140+}
141+
142+void amdgpu_userq_fence_driver_process(struct amdgpu_userq_fence_driver *fence_drv)
143+{
144+ struct amdgpu_userq_fence *userq_fence, *tmp;
145+ struct dma_fence *fence;
146+ u64 rptr;
147+ int i;
148+
149+ if (!fence_drv)
150+ return;
151+
152+ rptr = amdgpu_userq_fence_read(fence_drv);
153+
154+ spin_lock(&fence_drv->fence_list_lock);
155+ list_for_each_entry_safe(userq_fence, tmp, &fence_drv->fences, link) {
156+ fence = &userq_fence->base;
157+
158+ if (rptr < fence->seqno)
159+ break;
160+
161+ dma_fence_signal(fence);
162+
163+ for (i = 0; i < userq_fence->fence_drv_array_count; i++)
164+ amdgpu_userq_fence_driver_put(userq_fence->fence_drv_array[i]);
165+
166+ list_del(&userq_fence->link);
167+ dma_fence_put(fence);
168+ }
169+ spin_unlock(&fence_drv->fence_list_lock);
170+}
171+
172+void amdgpu_userq_fence_driver_destroy(struct kref *ref)
173+{
174+ struct amdgpu_userq_fence_driver *fence_drv = container_of(ref,
175+ struct amdgpu_userq_fence_driver,
176+ refcount);
177+ struct amdgpu_userq_fence_driver *xa_fence_drv;
178+ struct amdgpu_device *adev = fence_drv->adev;
179+ struct amdgpu_userq_fence *fence, *tmp;
180+ struct xarray *xa = &adev->userq_xa;
181+ unsigned long index, flags;
182+ struct dma_fence *f;
183+
184+ spin_lock(&fence_drv->fence_list_lock);
185+ list_for_each_entry_safe(fence, tmp, &fence_drv->fences, link) {
186+ f = &fence->base;
187+
188+ if (!dma_fence_is_signaled(f)) {
189+ dma_fence_set_error(f, -ECANCELED);
190+ dma_fence_signal(f);
191+ }
192+
193+ list_del(&fence->link);
194+ dma_fence_put(f);
195+ }
196+ spin_unlock(&fence_drv->fence_list_lock);
197+
198+ xa_lock_irqsave(xa, flags);
199+ xa_for_each(xa, index, xa_fence_drv)
200+ if (xa_fence_drv == fence_drv)
201+ __xa_erase(xa, index);
202+ xa_unlock_irqrestore(xa, flags);
203+
204+ /* Free seq64 memory */
205+ amdgpu_seq64_free(adev, fence_drv->va);
206+ kfree(fence_drv);
207+}
208+
209+void amdgpu_userq_fence_driver_get(struct amdgpu_userq_fence_driver *fence_drv)
210+{
211+ kref_get(&fence_drv->refcount);
212+}
213+
214+void amdgpu_userq_fence_driver_put(struct amdgpu_userq_fence_driver *fence_drv)
215+{
216+ kref_put(&fence_drv->refcount, amdgpu_userq_fence_driver_destroy);
217+}
218+
219+static int amdgpu_userq_fence_alloc(struct amdgpu_userq_fence **userq_fence)
220+{
221+ *userq_fence = kmem_cache_alloc(amdgpu_userq_fence_slab, GFP_ATOMIC);
222+ return *userq_fence ? 0 : -ENOMEM;
223+}
224+
225+static int amdgpu_userq_fence_create(struct amdgpu_usermode_queue *userq,
226+ struct amdgpu_userq_fence *userq_fence,
227+ u64 seq, struct dma_fence **f)
228+{
229+ struct amdgpu_userq_fence_driver *fence_drv;
230+ struct dma_fence *fence;
231+ unsigned long flags;
232+
233+ fence_drv = userq->fence_drv;
234+ if (!fence_drv)
235+ return -EINVAL;
236+
237+ spin_lock_init(&userq_fence->lock);
238+ INIT_LIST_HEAD(&userq_fence->link);
239+ fence = &userq_fence->base;
240+ userq_fence->fence_drv = fence_drv;
241+
242+ dma_fence_init(fence, &amdgpu_userq_fence_ops, &userq_fence->lock,
243+ fence_drv->context, seq);
244+
245+ amdgpu_userq_fence_driver_get(fence_drv);
246+ dma_fence_get(fence);
247+
248+ if (!xa_empty(&userq->fence_drv_xa)) {
249+ struct amdgpu_userq_fence_driver *stored_fence_drv;
250+ unsigned long index, count = 0;
251+ int i = 0;
252+
253+ xa_lock(&userq->fence_drv_xa);
254+ xa_for_each(&userq->fence_drv_xa, index, stored_fence_drv)
255+ count++;
256+
257+ userq_fence->fence_drv_array =
258+ kvmalloc_array(count,
259+ sizeof(struct amdgpu_userq_fence_driver *),
260+ GFP_ATOMIC);
261+
262+ if (userq_fence->fence_drv_array) {
263+ xa_for_each(&userq->fence_drv_xa, index, stored_fence_drv) {
264+ userq_fence->fence_drv_array[i] = stored_fence_drv;
265+ __xa_erase(&userq->fence_drv_xa, index);
266+ i++;
267+ }
268+ }
269+
270+ userq_fence->fence_drv_array_count = i;
271+ xa_unlock(&userq->fence_drv_xa);
272+ } else {
273+ userq_fence->fence_drv_array = NULL;
274+ userq_fence->fence_drv_array_count = 0;
275+ }
276+
277+ /* Check if hardware has already processed the job */
278+ spin_lock_irqsave(&fence_drv->fence_list_lock, flags);
279+ if (!dma_fence_is_signaled_locked(fence))
280+ list_add_tail(&userq_fence->link, &fence_drv->fences);
281+ else
282+ dma_fence_put(fence);
283+
284+ spin_unlock_irqrestore(&fence_drv->fence_list_lock, flags);
285+
286+ *f = fence;
287+
288+ return 0;
289+}
290+
291+static const char *amdgpu_userq_fence_get_driver_name(struct dma_fence *f)
292+{
293+ return "amdgpu_userq_fence";
294+}
295+
296+static const char *amdgpu_userq_fence_get_timeline_name(struct dma_fence *f)
297+{
298+ struct amdgpu_userq_fence *fence = to_amdgpu_userq_fence(f);
299+
300+ return fence->fence_drv->timeline_name;
301+}
302+
303+static bool amdgpu_userq_fence_signaled(struct dma_fence *f)
304+{
305+ struct amdgpu_userq_fence *fence = to_amdgpu_userq_fence(f);
306+ struct amdgpu_userq_fence_driver *fence_drv = fence->fence_drv;
307+ u64 rptr, wptr;
308+
309+ rptr = amdgpu_userq_fence_read(fence_drv);
310+ wptr = fence->base.seqno;
311+
312+ if (rptr >= wptr)
313+ return true;
314+
315+ return false;
316+}
317+
318+static void amdgpu_userq_fence_free(struct rcu_head *rcu)
319+{
320+ struct dma_fence *fence = container_of(rcu, struct dma_fence, rcu);
321+ struct amdgpu_userq_fence *userq_fence = to_amdgpu_userq_fence(fence);
322+ struct amdgpu_userq_fence_driver *fence_drv = userq_fence->fence_drv;
323+
324+ /* Release the fence driver reference */
325+ amdgpu_userq_fence_driver_put(fence_drv);
326+
327+ kvfree(userq_fence->fence_drv_array);
328+ kmem_cache_free(amdgpu_userq_fence_slab, userq_fence);
329+}
330+
331+static void amdgpu_userq_fence_release(struct dma_fence *f)
332+{
333+ call_rcu(&f->rcu, amdgpu_userq_fence_free);
334+}
335+
336+static const struct dma_fence_ops amdgpu_userq_fence_ops = {
337+ .use_64bit_seqno = true,
338+ .get_driver_name = amdgpu_userq_fence_get_driver_name,
339+ .get_timeline_name = amdgpu_userq_fence_get_timeline_name,
340+ .signaled = amdgpu_userq_fence_signaled,
341+ .release = amdgpu_userq_fence_release,
342+};
343+
344+/**
345+ * amdgpu_userq_fence_read_wptr - Read the userq wptr value
346+ *
347+ * @queue: user mode queue structure pointer
348+ * @wptr: write pointer value
349+ *
350+ * Read the wptr value from userq's MQD. The userq signal IOCTL
351+ * creates a dma_fence for the shared buffers that expects the
352+ * RPTR value written to seq64 memory >= WPTR.
353+ *
354+ * Returns wptr value on success, error on failure.
355+ */
356+static int amdgpu_userq_fence_read_wptr(struct amdgpu_usermode_queue *queue,
357+ u64 *wptr)
358+{
359+ struct amdgpu_bo_va_mapping *mapping;
360+ struct amdgpu_bo *bo;
361+ u64 addr, *ptr;
362+ int r;
363+
364+ r = amdgpu_bo_reserve(queue->vm->root.bo, false);
365+ if (r)
366+ return r;
367+
368+ addr = queue->userq_prop->wptr_gpu_addr;
369+ addr &= AMDGPU_GMC_HOLE_MASK;
370+
371+ mapping = amdgpu_vm_bo_lookup_mapping(queue->vm, addr >> PAGE_SHIFT);
372+ if (!mapping) {
373+ amdgpu_bo_unreserve(queue->vm->root.bo);
374+ DRM_ERROR("Failed to lookup amdgpu_bo_va_mapping\n");
375+ return -EINVAL;
376+ }
377+
378+ bo = amdgpu_bo_ref(mapping->bo_va->base.bo);
379+ amdgpu_bo_unreserve(queue->vm->root.bo);
380+ r = amdgpu_bo_reserve(bo, true);
381+ if (r) {
382+ DRM_ERROR("Failed to reserve userqueue wptr bo");
383+ return r;
384+ }
385+
386+ r = amdgpu_bo_kmap(bo, (void **)&ptr);
387+ if (r) {
388+ DRM_ERROR("Failed mapping the userqueue wptr bo");
389+ goto map_error;
390+ }
391+
392+ *wptr = le64_to_cpu(*ptr);
393+
394+ amdgpu_bo_kunmap(bo);
395+ amdgpu_bo_unreserve(bo);
396+ amdgpu_bo_unref(&bo);
397+
398+ return 0;
399+
400+map_error:
401+ amdgpu_bo_unreserve(bo);
402+ amdgpu_bo_unref(&bo);
403+
404+ return r;
405+}
406+
407+static void amdgpu_userq_fence_cleanup(struct dma_fence *fence)
408+{
409+ dma_fence_put(fence);
410+}
411+
412+int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data,
413+ struct drm_file *filp)
414+{
415+ struct amdgpu_fpriv *fpriv = filp->driver_priv;
416+ struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr;
417+ struct drm_amdgpu_userq_signal *args = data;
418+ struct drm_gem_object **gobj_write = NULL;
419+ struct drm_gem_object **gobj_read = NULL;
420+ struct amdgpu_usermode_queue *queue;
421+ struct amdgpu_userq_fence *userq_fence;
422+ struct drm_syncobj **syncobj = NULL;
423+ u32 *bo_handles_write, num_write_bo_handles;
424+ u32 *syncobj_handles, num_syncobj_handles;
425+ u32 *bo_handles_read, num_read_bo_handles;
426+ int r, i, entry, rentry, wentry;
427+ struct dma_fence *fence;
428+ struct drm_exec exec;
429+ u64 wptr;
430+
431+ num_syncobj_handles = args->num_syncobj_handles;
432+ syncobj_handles = memdup_user(u64_to_user_ptr(args->syncobj_handles),
433+ sizeof(u32) * num_syncobj_handles);
434+ if (IS_ERR(syncobj_handles))
435+ return PTR_ERR(syncobj_handles);
436+
437+ /* Array of pointers to the looked up syncobjs */
438+ syncobj = kmalloc_array(num_syncobj_handles, sizeof(*syncobj), GFP_KERNEL);
439+ if (!syncobj) {
440+ r = -ENOMEM;
441+ goto free_syncobj_handles;
442+ }
443+
444+ for (entry = 0; entry < num_syncobj_handles; entry++) {
445+ syncobj[entry] = drm_syncobj_find(filp, syncobj_handles[entry]);
446+ if (!syncobj[entry]) {
447+ r = -ENOENT;
448+ goto free_syncobj;
449+ }
450+ }
451+
452+ num_read_bo_handles = args->num_bo_read_handles;
453+ bo_handles_read = memdup_user(u64_to_user_ptr(args->bo_read_handles),
454+ sizeof(u32) * num_read_bo_handles);
455+ if (IS_ERR(bo_handles_read)) {
456+ r = PTR_ERR(bo_handles_read);
457+ goto free_syncobj;
458+ }
459+
460+ /* Array of pointers to the GEM read objects */
461+ gobj_read = kmalloc_array(num_read_bo_handles, sizeof(*gobj_read), GFP_KERNEL);
462+ if (!gobj_read) {
463+ r = -ENOMEM;
464+ goto free_bo_handles_read;
465+ }
466+
467+ for (rentry = 0; rentry < num_read_bo_handles; rentry++) {
468+ gobj_read[rentry] = drm_gem_object_lookup(filp, bo_handles_read[rentry]);
469+ if (!gobj_read[rentry]) {
470+ r = -ENOENT;
471+ goto put_gobj_read;
472+ }
473+ }
474+
475+ num_write_bo_handles = args->num_bo_write_handles;
476+ bo_handles_write = memdup_user(u64_to_user_ptr(args->bo_write_handles),
477+ sizeof(u32) * num_write_bo_handles);
478+ if (IS_ERR(bo_handles_write)) {
479+ r = PTR_ERR(bo_handles_write);
480+ goto put_gobj_read;
481+ }
482+
483+ /* Array of pointers to the GEM write objects */
484+ gobj_write = kmalloc_array(num_write_bo_handles, sizeof(*gobj_write), GFP_KERNEL);
485+ if (!gobj_write) {
486+ r = -ENOMEM;
487+ goto free_bo_handles_write;
488+ }
489+
490+ for (wentry = 0; wentry < num_write_bo_handles; wentry++) {
491+ gobj_write[wentry] = drm_gem_object_lookup(filp, bo_handles_write[wentry]);
492+ if (!gobj_write[wentry]) {
493+ r = -ENOENT;
494+ goto put_gobj_write;
495+ }
496+ }
497+
498+ /* Retrieve the user queue */
499+ queue = idr_find(&userq_mgr->userq_idr, args->queue_id);
500+ if (!queue) {
501+ r = -ENOENT;
502+ goto put_gobj_write;
503+ }
504+
505+ r = amdgpu_userq_fence_read_wptr(queue, &wptr);
506+ if (r)
507+ goto put_gobj_write;
508+
509+ r = amdgpu_userq_fence_alloc(&userq_fence);
510+ if (r)
511+ goto put_gobj_write;
512+
513+ /* We are here means UQ is active, make sure the eviction fence is valid */
514+ amdgpu_userq_ensure_ev_fence(&fpriv->userq_mgr, &fpriv->evf_mgr);
515+
516+ /* Create a new fence */
517+ r = amdgpu_userq_fence_create(queue, userq_fence, wptr, &fence);
518+ if (r) {
519+ mutex_unlock(&userq_mgr->userq_mutex);
520+ kmem_cache_free(amdgpu_userq_fence_slab, userq_fence);
521+ goto put_gobj_write;
522+ }
523+
524+ dma_fence_put(queue->last_fence);
525+ queue->last_fence = dma_fence_get(fence);
526+ mutex_unlock(&userq_mgr->userq_mutex);
527+
528+ drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,
529+ (num_read_bo_handles + num_write_bo_handles));
530+
531+ /* Lock all BOs with retry handling */
532+ drm_exec_until_all_locked(&exec) {
533+ r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1);
534+ drm_exec_retry_on_contention(&exec);
535+ if (r) {
536+ amdgpu_userq_fence_cleanup(fence);
537+ goto exec_fini;
538+ }
539+
540+ r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1);
541+ drm_exec_retry_on_contention(&exec);
542+ if (r) {
543+ amdgpu_userq_fence_cleanup(fence);
544+ goto exec_fini;
545+ }
546+ }
547+
548+ for (i = 0; i < num_read_bo_handles; i++) {
549+ if (!gobj_read || !gobj_read[i]->resv)
550+ continue;
551+
552+ dma_resv_add_fence(gobj_read[i]->resv, fence,
553+ DMA_RESV_USAGE_READ);
554+ }
555+
556+ for (i = 0; i < num_write_bo_handles; i++) {
557+ if (!gobj_write || !gobj_write[i]->resv)
558+ continue;
559+
560+ dma_resv_add_fence(gobj_write[i]->resv, fence,
561+ DMA_RESV_USAGE_WRITE);
562+ }
563+
564+ /* Add the created fence to syncobj/BO's */
565+ for (i = 0; i < num_syncobj_handles; i++)
566+ drm_syncobj_replace_fence(syncobj[i], fence);
567+
568+ /* drop the reference acquired in fence creation function */
569+ dma_fence_put(fence);
570+
571+exec_fini:
572+ drm_exec_fini(&exec);
573+put_gobj_write:
574+ while (wentry-- > 0)
575+ drm_gem_object_put(gobj_write[wentry]);
576+ kfree(gobj_write);
577+free_bo_handles_write:
578+ kfree(bo_handles_write);
579+put_gobj_read:
580+ while (rentry-- > 0)
581+ drm_gem_object_put(gobj_read[rentry]);
582+ kfree(gobj_read);
583+free_bo_handles_read:
584+ kfree(bo_handles_read);
585+free_syncobj:
586+ while (entry-- > 0)
587+ if (syncobj[entry])
588+ drm_syncobj_put(syncobj[entry]);
589+ kfree(syncobj);
590+free_syncobj_handles:
591+ kfree(syncobj_handles);
592+
593+ return r;
594+}
595+
596+int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data,
597+ struct drm_file *filp)
598+{
599+ u32 *syncobj_handles, *timeline_points, *timeline_handles, *bo_handles_read, *bo_handles_write;
600+ u32 num_syncobj, num_read_bo_handles, num_write_bo_handles;
601+ struct drm_amdgpu_userq_fence_info *fence_info = NULL;
602+ struct drm_amdgpu_userq_wait *wait_info = data;
603+ struct amdgpu_fpriv *fpriv = filp->driver_priv;
604+ struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr;
605+ struct amdgpu_usermode_queue *waitq;
606+ struct drm_gem_object **gobj_write;
607+ struct drm_gem_object **gobj_read;
608+ struct dma_fence **fences = NULL;
609+ u16 num_points, num_fences = 0;
610+ int r, i, rentry, wentry, cnt;
611+ struct drm_exec exec;
612+
613+ num_read_bo_handles = wait_info->num_bo_read_handles;
614+ bo_handles_read = memdup_user(u64_to_user_ptr(wait_info->bo_read_handles),
615+ sizeof(u32) * num_read_bo_handles);
616+ if (IS_ERR(bo_handles_read))
617+ return PTR_ERR(bo_handles_read);
618+
619+ num_write_bo_handles = wait_info->num_bo_write_handles;
620+ bo_handles_write = memdup_user(u64_to_user_ptr(wait_info->bo_write_handles),
621+ sizeof(u32) * num_write_bo_handles);
622+ if (IS_ERR(bo_handles_write)) {
623+ r = PTR_ERR(bo_handles_write);
624+ goto free_bo_handles_read;
625+ }
626+
627+ num_syncobj = wait_info->num_syncobj_handles;
628+ syncobj_handles = memdup_user(u64_to_user_ptr(wait_info->syncobj_handles),
629+ sizeof(u32) * num_syncobj);
630+ if (IS_ERR(syncobj_handles)) {
631+ r = PTR_ERR(syncobj_handles);
632+ goto free_bo_handles_write;
633+ }
634+
635+ num_points = wait_info->num_syncobj_timeline_handles;
636+ timeline_handles = memdup_user(u64_to_user_ptr(wait_info->syncobj_timeline_handles),
637+ sizeof(u32) * num_points);
638+ if (IS_ERR(timeline_handles)) {
639+ r = PTR_ERR(timeline_handles);
640+ goto free_syncobj_handles;
641+ }
642+
643+ timeline_points = memdup_user(u64_to_user_ptr(wait_info->syncobj_timeline_points),
644+ sizeof(u32) * num_points);
645+ if (IS_ERR(timeline_points)) {
646+ r = PTR_ERR(timeline_points);
647+ goto free_timeline_handles;
648+ }
649+
650+ gobj_read = kmalloc_array(num_read_bo_handles, sizeof(*gobj_read), GFP_KERNEL);
651+ if (!gobj_read) {
652+ r = -ENOMEM;
653+ goto free_timeline_points;
654+ }
655+
656+ for (rentry = 0; rentry < num_read_bo_handles; rentry++) {
657+ gobj_read[rentry] = drm_gem_object_lookup(filp, bo_handles_read[rentry]);
658+ if (!gobj_read[rentry]) {
659+ r = -ENOENT;
660+ goto put_gobj_read;
661+ }
662+ }
663+
664+ gobj_write = kmalloc_array(num_write_bo_handles, sizeof(*gobj_write), GFP_KERNEL);
665+ if (!gobj_write) {
666+ r = -ENOMEM;
667+ goto put_gobj_read;
668+ }
669+
670+ for (wentry = 0; wentry < num_write_bo_handles; wentry++) {
671+ gobj_write[wentry] = drm_gem_object_lookup(filp, bo_handles_write[wentry]);
672+ if (!gobj_write[wentry]) {
673+ r = -ENOENT;
674+ goto put_gobj_write;
675+ }
676+ }
677+
678+ drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,
679+ (num_read_bo_handles + num_write_bo_handles));
680+
681+ /* Lock all BOs with retry handling */
682+ drm_exec_until_all_locked(&exec) {
683+ r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1);
684+ drm_exec_retry_on_contention(&exec);
685+ if (r) {
686+ drm_exec_fini(&exec);
687+ goto put_gobj_write;
688+ }
689+
690+ r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1);
691+ drm_exec_retry_on_contention(&exec);
692+ if (r) {
693+ drm_exec_fini(&exec);
694+ goto put_gobj_write;
695+ }
696+ }
697+
698+ if (!wait_info->num_fences) {
699+ if (num_points) {
700+ struct dma_fence_unwrap iter;
701+ struct dma_fence *fence;
702+ struct dma_fence *f;
703+
704+ for (i = 0; i < num_points; i++) {
705+ r = drm_syncobj_find_fence(filp, timeline_handles[i],
706+ timeline_points[i],
707+ DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
708+ &fence);
709+ if (r)
710+ goto exec_fini;
711+
712+ dma_fence_unwrap_for_each(f, &iter, fence)
713+ num_fences++;
714+
715+ dma_fence_put(fence);
716+ }
717+ }
718+
719+ /* Count syncobj's fence */
720+ for (i = 0; i < num_syncobj; i++) {
721+ struct dma_fence *fence;
722+
723+ r = drm_syncobj_find_fence(filp, syncobj_handles[i],
724+ 0,
725+ DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
726+ &fence);
727+ if (r)
728+ goto exec_fini;
729+
730+ num_fences++;
731+ dma_fence_put(fence);
732+ }
733+
734+ /* Count GEM objects fence */
735+ for (i = 0; i < num_read_bo_handles; i++) {
736+ struct dma_resv_iter resv_cursor;
737+ struct dma_fence *fence;
738+
739+ dma_resv_for_each_fence(&resv_cursor, gobj_read[i]->resv,
740+ DMA_RESV_USAGE_READ, fence)
741+ num_fences++;
742+ }
743+
744+ for (i = 0; i < num_write_bo_handles; i++) {
745+ struct dma_resv_iter resv_cursor;
746+ struct dma_fence *fence;
747+
748+ dma_resv_for_each_fence(&resv_cursor, gobj_write[i]->resv,
749+ DMA_RESV_USAGE_WRITE, fence)
750+ num_fences++;
751+ }
752+
753+ /*
754+ * Passing num_fences = 0 means that userspace doesn't want to
755+ * retrieve userq_fence_info. If num_fences = 0 we skip filling
756+ * userq_fence_info and return the actual number of fences on
757+ * args->num_fences.
758+ */
759+ wait_info->num_fences = num_fences;
760+ } else {
761+ /* Array of fence info */
762+ fence_info = kmalloc_array(wait_info->num_fences, sizeof(*fence_info), GFP_KERNEL);
763+ if (!fence_info) {
764+ r = -ENOMEM;
765+ goto exec_fini;
766+ }
767+
768+ /* Array of fences */
769+ fences = kmalloc_array(wait_info->num_fences, sizeof(*fences), GFP_KERNEL);
770+ if (!fences) {
771+ r = -ENOMEM;
772+ goto free_fence_info;
773+ }
774+
775+ /* Retrieve GEM read objects fence */
776+ for (i = 0; i < num_read_bo_handles; i++) {
777+ struct dma_resv_iter resv_cursor;
778+ struct dma_fence *fence;
779+
780+ dma_resv_for_each_fence(&resv_cursor, gobj_read[i]->resv,
781+ DMA_RESV_USAGE_READ, fence) {
782+ if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
783+ r = -EINVAL;
784+ goto free_fences;
785+ }
786+
787+ fences[num_fences++] = fence;
788+ dma_fence_get(fence);
789+ }
790+ }
791+
792+ /* Retrieve GEM write objects fence */
793+ for (i = 0; i < num_write_bo_handles; i++) {
794+ struct dma_resv_iter resv_cursor;
795+ struct dma_fence *fence;
796+
797+ dma_resv_for_each_fence(&resv_cursor, gobj_write[i]->resv,
798+ DMA_RESV_USAGE_WRITE, fence) {
799+ if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
800+ r = -EINVAL;
801+ goto free_fences;
802+ }
803+
804+ fences[num_fences++] = fence;
805+ dma_fence_get(fence);
806+ }
807+ }
808+
809+ if (num_points) {
810+ struct dma_fence_unwrap iter;
811+ struct dma_fence *fence;
812+ struct dma_fence *f;
813+
814+ for (i = 0; i < num_points; i++) {
815+ r = drm_syncobj_find_fence(filp, timeline_handles[i],
816+ timeline_points[i],
817+ DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
818+ &fence);
819+ if (r)
820+ goto free_fences;
821+
822+ dma_fence_unwrap_for_each(f, &iter, fence) {
823+ if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
824+ r = -EINVAL;
825+ goto free_fences;
826+ }
827+
828+ dma_fence_get(f);
829+ fences[num_fences++] = f;
830+ }
831+
832+ dma_fence_put(fence);
833+ }
834+ }
835+
836+ /* Retrieve syncobj's fence */
837+ for (i = 0; i < num_syncobj; i++) {
838+ struct dma_fence *fence;
839+
840+ r = drm_syncobj_find_fence(filp, syncobj_handles[i],
841+ 0,
842+ DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
843+ &fence);
844+ if (r)
845+ goto free_fences;
846+
847+ if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
848+ r = -EINVAL;
849+ goto free_fences;
850+ }
851+
852+ fences[num_fences++] = fence;
853+ }
854+
855+ /*
856+ * Keep only the latest fences to reduce the number of values
857+ * given back to userspace.
858+ */
859+ num_fences = dma_fence_dedup_array(fences, num_fences);
860+
861+ waitq = idr_find(&userq_mgr->userq_idr, wait_info->waitq_id);
862+ if (!waitq)
863+ goto free_fences;
864+
865+ for (i = 0, cnt = 0; i < num_fences; i++) {
866+ struct amdgpu_userq_fence_driver *fence_drv;
867+ struct amdgpu_userq_fence *userq_fence;
868+ u32 index;
869+
870+ userq_fence = to_amdgpu_userq_fence(fences[i]);
871+ if (!userq_fence) {
872+ /*
873+ * Just waiting on other driver fences should
874+ * be good for now
875+ */
876+ r = dma_fence_wait(fences[i], true);
877+ if (r) {
878+ dma_fence_put(fences[i]);
879+ goto free_fences;
880+ }
881+
882+ dma_fence_put(fences[i]);
883+ continue;
884+ }
885+
886+ fence_drv = userq_fence->fence_drv;
887+ /*
888+ * We need to make sure the user queue release their reference
889+ * to the fence drivers at some point before queue destruction.
890+ * Otherwise, we would gather those references until we don't
891+ * have any more space left and crash.
892+ */
893+ r = xa_alloc(&waitq->fence_drv_xa, &index, fence_drv,
894+ xa_limit_32b, GFP_KERNEL);
895+ if (r)
896+ goto free_fences;
897+
898+ amdgpu_userq_fence_driver_get(fence_drv);
899+
900+ /* Store drm syncobj's gpu va address and value */
901+ fence_info[cnt].va = fence_drv->va;
902+ fence_info[cnt].value = fences[i]->seqno;
903+
904+ dma_fence_put(fences[i]);
905+ /* Increment the actual userq fence count */
906+ cnt++;
907+ }
908+
909+ wait_info->num_fences = cnt;
910+ /* Copy userq fence info to user space */
911+ if (copy_to_user(u64_to_user_ptr(wait_info->out_fences),
912+ fence_info, wait_info->num_fences * sizeof(*fence_info))) {
913+ r = -EFAULT;
914+ goto free_fences;
915+ }
916+
917+ kfree(fences);
918+ kfree(fence_info);
919+ }
920+
921+ drm_exec_fini(&exec);
922+ for (i = 0; i < num_read_bo_handles; i++)
923+ drm_gem_object_put(gobj_read[i]);
924+ kfree(gobj_read);
925+
926+ for (i = 0; i < num_write_bo_handles; i++)
927+ drm_gem_object_put(gobj_write[i]);
928+ kfree(gobj_write);
929+
930+ kfree(timeline_points);
931+ kfree(timeline_handles);
932+ kfree(syncobj_handles);
933+ kfree(bo_handles_write);
934+ kfree(bo_handles_read);
935+
936+ return 0;
937+
938+free_fences:
939+ while (num_fences-- > 0)
940+ dma_fence_put(fences[num_fences]);
941+ kfree(fences);
942+free_fence_info:
943+ kfree(fence_info);
944+exec_fini:
945+ drm_exec_fini(&exec);
946+put_gobj_write:
947+ while (wentry-- > 0)
948+ drm_gem_object_put(gobj_write[wentry]);
949+ kfree(gobj_write);
950+put_gobj_read:
951+ while (rentry-- > 0)
952+ drm_gem_object_put(gobj_read[rentry]);
953+ kfree(gobj_read);
954+free_timeline_points:
955+ kfree(timeline_points);
956+free_timeline_handles:
957+ kfree(timeline_handles);
958+free_syncobj_handles:
959+ kfree(syncobj_handles);
960+free_bo_handles_write:
961+ kfree(bo_handles_write);
962+free_bo_handles_read:
963+ kfree(bo_handles_read);
964+
965+ return r;
966+}
drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.h+76−0
@@ -0,0 +1,76 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright 2023 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+
25+#ifndef __AMDGPU_USERQ_FENCE_H__
26+#define __AMDGPU_USERQ_FENCE_H__
27+
28+#include <linux/types.h>
29+
30+#include "amdgpu_userq.h"
31+
32+struct amdgpu_userq_fence {
33+ struct dma_fence base;
34+ /*
35+ * This lock is necessary to synchronize the
36+ * userqueue dma fence operations.
37+ */
38+ spinlock_t lock;
39+ struct list_head link;
40+ unsigned long fence_drv_array_count;
41+ struct amdgpu_userq_fence_driver *fence_drv;
42+ struct amdgpu_userq_fence_driver **fence_drv_array;
43+};
44+
45+struct amdgpu_userq_fence_driver {
46+ struct kref refcount;
47+ u64 va;
48+ u64 gpu_addr;
49+ u64 *cpu_addr;
50+ u64 context;
51+ /*
52+ * This lock is necesaary to synchronize the access
53+ * to the fences list by the fence driver.
54+ */
55+ spinlock_t fence_list_lock;
56+ struct list_head fences;
57+ struct amdgpu_device *adev;
58+ char timeline_name[TASK_COMM_LEN];
59+};
60+
61+int amdgpu_userq_fence_slab_init(void);
62+void amdgpu_userq_fence_slab_fini(void);
63+
64+void amdgpu_userq_fence_driver_get(struct amdgpu_userq_fence_driver *fence_drv);
65+void amdgpu_userq_fence_driver_put(struct amdgpu_userq_fence_driver *fence_drv);
66+int amdgpu_userq_fence_driver_alloc(struct amdgpu_device *adev,
67+ struct amdgpu_usermode_queue *userq);
68+void amdgpu_userq_fence_driver_free(struct amdgpu_usermode_queue *userq);
69+void amdgpu_userq_fence_driver_process(struct amdgpu_userq_fence_driver *fence_drv);
70+void amdgpu_userq_fence_driver_destroy(struct kref *ref);
71+int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data,
72+ struct drm_file *filp);
73+int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data,
74+ struct drm_file *filp);
75+
76+#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c+2−2
@@ -353,9 +353,9 @@ int amdgpu_vcn_suspend(struct amdgpu_device *adev, int i) · 352 unmodified lines
353353
354354 cancel_delayed_work_sync(&adev->vcn.inst[i].idle_work);
355355
356− /* err_event_athub will corrupt VCPU buffer, so we need to
356+ /* err_event_athub and dpc recovery will corrupt VCPU buffer, so we need to
357357 * restore fw data and clear buffer in amdgpu_vcn_resume() */
358− if (in_ras_intr)
358+ if (in_ras_intr || adev->pcie_reset_ctx.in_link_reset)
359359 return 0;
360360
361361 return amdgpu_vcn_save_vcpu_bo_inst(adev, i);
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h+0−1
@@ -66,7 +66,6 @@ · 65 unmodified lines
6666 #define VCN_ENC_CMD_REG_WAIT 0x0000000c
6767
6868 #define VCN_AON_SOC_ADDRESS_2_0 0x1f800
69−#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
7069 #define VCN_VID_IP_ADDRESS_2_0 0x0
7170 #define VCN_AON_IP_ADDRESS_2_0 0x30000
7271
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c+25−4
@@ -1323,6 +1323,9 @@ static int amdgpu_virt_req_ras_err_count_internal(struct amdgpu_device *adev, bo · 1322 unmodified lines
13231323 {
13241324 struct amdgpu_virt *virt = &adev->virt;
13251325
1326+ if (!virt->ops || !virt->ops->req_ras_err_count)
1327+ return -EOPNOTSUPP;
1328+
13261329 /* Host allows 15 ras telemetry requests per 60 seconds. Afterwhich, the Host
13271330 * will ignore incoming guest messages. Ratelimit the guest messages to
13281331 * prevent guest self DOS.
@@ -1378,14 +1381,16 @@ amdgpu_virt_write_cpers_to_ring(struct amdgpu_device *adev, · 49 unmodified lines
13781381 used_size = host_telemetry->header.used_size;
13791382
13801383 if (used_size > (AMD_SRIOV_RAS_TELEMETRY_SIZE_KB << 10))
1381− return 0;
1384+ return -EINVAL;
13821385
13831386 cper_dump = kmemdup(&host_telemetry->body.cper_dump, used_size, GFP_KERNEL);
13841387 if (!cper_dump)
13851388 return -ENOMEM;
13861389
1387− if (checksum != amd_sriov_msg_checksum(cper_dump, used_size, 0, 0))
1390+ if (checksum != amd_sriov_msg_checksum(cper_dump, used_size, 0, 0)) {
1391+ ret = -EINVAL;
13881392 goto out;
1393+ }
13891394
13901395 *more = cper_dump->more;
13911396
@@ -1425,7 +1430,7 @@ static int amdgpu_virt_req_ras_cper_dump_internal(struct amdgpu_device *adev) · 33 unmodified lines
14251430 int ret = 0;
14261431 uint32_t more = 0;
14271432
1428− if (!amdgpu_sriov_ras_cper_en(adev))
1433+ if (!virt->ops || !virt->ops->req_ras_cper_dump)
14291434 return -EOPNOTSUPP;
14301435
14311436 do {
@@ -1434,7 +1439,7 @@ static int amdgpu_virt_req_ras_cper_dump_internal(struct amdgpu_device *adev) · 2 unmodified lines
14341439 adev, virt->fw_reserve.ras_telemetry, &more);
14351440 else
14361441 ret = 0;
1437− } while (more);
1442+ } while (more && !ret);
14381443
14391444 return ret;
14401445 }
@@ -1444,6 +1449,9 @@ int amdgpu_virt_req_ras_cper_dump(struct amdgpu_device *adev, bool force_update) · 3 unmodified lines
14441449 struct amdgpu_virt *virt = &adev->virt;
14451450 int ret = 0;
14461451
1452+ if (!amdgpu_sriov_ras_cper_en(adev))
1453+ return -EOPNOTSUPP;
1454+
14471455 if ((__ratelimit(&virt->ras.ras_cper_dump_rs) || force_update) &&
14481456 down_read_trylock(&adev->reset_domain->sem)) {
14491457 mutex_lock(&virt->ras.ras_telemetry_mutex);
@@ -1480,3 +1488,16 @@ bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev, · 30 unmodified lines
14801488
14811489 return true;
14821490 }
1491+
1492+/*
1493+ * amdgpu_virt_request_bad_pages() - request bad pages
1494+ * @adev: amdgpu device.
1495+ * Send command to GPU hypervisor to write new bad pages into the shared PF2VF region
1496+ */
1497+void amdgpu_virt_request_bad_pages(struct amdgpu_device *adev)
1498+{
1499+ struct amdgpu_virt *virt = &adev->virt;
1500+
1501+ if (virt->ops && virt->ops->req_bad_pages)
1502+ virt->ops->req_bad_pages(adev);
1503+}
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h+14−3
@@ -97,6 +97,7 @@ struct amdgpu_virt_ops { · 96 unmodified lines
9797 bool (*rcvd_ras_intr)(struct amdgpu_device *adev);
9898 int (*req_ras_err_count)(struct amdgpu_device *adev);
9999 int (*req_ras_cper_dump)(struct amdgpu_device *adev, u64 vf_rptr);
100+ int (*req_bad_pages)(struct amdgpu_device *adev);
100101 };
101102
102103 /*
@@ -146,11 +147,13 @@ enum AMDGIM_FEATURE_FLAG { · 43 unmodified lines
146147
147148 enum AMDGIM_REG_ACCESS_FLAG {
148149 /* Use PSP to program IH_RB_CNTL */
149− AMDGIM_FEATURE_IH_REG_PSP_EN = (1 << 0),
150+ AMDGIM_FEATURE_IH_REG_PSP_EN = (1 << 0),
150151 /* Use RLC to program MMHUB regs */
151− AMDGIM_FEATURE_MMHUB_REG_RLC_EN = (1 << 1),
152+ AMDGIM_FEATURE_MMHUB_REG_RLC_EN = (1 << 1),
152153 /* Use RLC to program GC regs */
153− AMDGIM_FEATURE_GC_REG_RLC_EN = (1 << 2),
154+ AMDGIM_FEATURE_GC_REG_RLC_EN = (1 << 2),
155+ /* Use PSP to program L1_TLB_CNTL*/
156+ AMDGIM_FEATURE_L1_TLB_CNTL_PSP_EN = (1 << 3),
154157 };
155158
156159 struct amdgim_pf2vf_info_v1 {
@@ -260,7 +263,10 @@ struct amdgpu_virt { · 103 unmodified lines
260263 uint32_t reg_val_offs;
261264 struct amdgpu_irq_src ack_irq;
262265 struct amdgpu_irq_src rcv_irq;
266+
263267 struct work_struct flr_work;
268+ struct work_struct bad_pages_work;
269+
264270 struct amdgpu_mm_table mm_table;
265271 const struct amdgpu_virt_ops *ops;
266272 struct amdgpu_vf_error_buffer vf_errors;
@@ -330,6 +336,10 @@ struct amdgpu_video_codec_info; · 63 unmodified lines
330336 (amdgpu_sriov_vf((adev)) && \
331337 ((adev)->virt.reg_access & (AMDGIM_FEATURE_GC_REG_RLC_EN)))
332338
339+#define amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev) \
340+(amdgpu_sriov_vf((adev)) && \
341+ ((adev)->virt.reg_access & (AMDGIM_FEATURE_L1_TLB_CNTL_PSP_EN)))
342+
333343 #define amdgpu_sriov_rlcg_error_report_enabled(adev) \
334344 (amdgpu_sriov_reg_indirect_mmhub(adev) || amdgpu_sriov_reg_indirect_gc(adev))
335345
@@ -423,4 +433,5 @@ int amdgpu_virt_req_ras_cper_dump(struct amdgpu_device *adev, bool force_update) · 87 unmodified lines
423433 int amdgpu_virt_ras_telemetry_post_reset(struct amdgpu_device *adev);
424434 bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev,
425435 enum amdgpu_ras_block block);
436+void amdgpu_virt_request_bad_pages(struct amdgpu_device *adev);
426437 #endif
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c+3−2
@@ -787,7 +787,8 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · 786 unmodified lines
787787 pasid_mapping_needed &= adev->gmc.gmc_funcs->emit_pasid_mapping &&
788788 ring->funcs->emit_wreg;
789789
790− cleaner_shader_needed = adev->gfx.enable_cleaner_shader &&
790+ cleaner_shader_needed = job->run_cleaner_shader &&
791+ adev->gfx.enable_cleaner_shader &&
791792 ring->funcs->emit_cleaner_shader && job->base.s_fence &&
792793 &job->base.s_fence->scheduled == isolation->spearhead;
793794
@@ -817,7 +818,7 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · 23 unmodified lines
817818 if (spm_update_needed && adev->gfx.rlc.funcs->update_spm_vmid)
818819 adev->gfx.rlc.funcs->update_spm_vmid(adev, ring, job->vmid);
819820
820− if (!ring->is_mes_queue && ring->funcs->emit_gds_switch &&
821+ if (ring->funcs->emit_gds_switch &&
821822 gds_switch_needed) {
822823 amdgpu_ring_emit_gds_switch(ring, job->vmid, job->gds_base,
823824 job->gds_size, job->gws_base,
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c+18−6
@@ -296,15 +296,27 @@ static const struct amdgpu_pcs_ras_field xgmi3x16_pcs_ras_fields[] = { · 295 unmodified lines
296296
297297 static u32 xgmi_v6_4_get_link_status(struct amdgpu_device *adev, int global_link_num)
298298 {
299− const u32 smnpcs_xgmi3x16_pcs_state_hist1 = 0x11a00070;
300− const int xgmi_inst = 2;
301− u32 link_inst;
299+ const u32 smn_xgmi_6_4_pcs_state_hist1[2] = { 0x11a00070, 0x11b00070 };
300+ const u32 smn_xgmi_6_4_1_pcs_state_hist1[2] = { 0x12100070,
301+ 0x11b00070 };
302+ u32 i, n;
302303 u64 addr;
303304
304− link_inst = global_link_num % xgmi_inst;
305+ switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
306+ case IP_VERSION(6, 4, 0):
307+ n = ARRAY_SIZE(smn_xgmi_6_4_pcs_state_hist1);
308+ addr = smn_xgmi_6_4_pcs_state_hist1[global_link_num % n];
309+ break;
310+ case IP_VERSION(6, 4, 1):
311+ n = ARRAY_SIZE(smn_xgmi_6_4_1_pcs_state_hist1);
312+ addr = smn_xgmi_6_4_1_pcs_state_hist1[global_link_num % n];
313+ break;
314+ default:
315+ return U32_MAX;
316+ }
305317
306− addr = (smnpcs_xgmi3x16_pcs_state_hist1 | (link_inst << 20)) +
307− adev->asic_funcs->encode_ext_smn_addressing(global_link_num / xgmi_inst);
318+ i = global_link_num / n;
319+ addr += adev->asic_funcs->encode_ext_smn_addressing(i);
308320
309321 return RREG32_PCIE_EXT(addr);
310322 }
drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h+9−4
@@ -109,10 +109,11 @@ union amd_sriov_msg_feature_flags { · 108 unmodified lines
109109
110110 union amd_sriov_reg_access_flags {
111111 struct {
112− uint32_t vf_reg_access_ih : 1;
113− uint32_t vf_reg_access_mmhub : 1;
114− uint32_t vf_reg_access_gc : 1;
115− uint32_t reserved : 29;
112+ uint32_t vf_reg_access_ih : 1;
113+ uint32_t vf_reg_access_mmhub : 1;
114+ uint32_t vf_reg_access_gc : 1;
115+ uint32_t vf_reg_access_l1_tlb_cntl : 1;
116+ uint32_t reserved : 28;
116117 } flags;
117118 uint32_t all;
118119 };
@@ -330,6 +331,7 @@ enum amd_sriov_mailbox_request_message { · 211 unmodified lines
330331 MB_REQ_MSG_RAS_POISON = 202,
331332 MB_REQ_RAS_ERROR_COUNT = 203,
332333 MB_REQ_RAS_CPER_DUMP = 204,
334+ MB_REQ_RAS_BAD_PAGES = 205,
333335 };
334336
335337 /* mailbox message send from host to guest */
@@ -347,6 +349,9 @@ enum amd_sriov_mailbox_response_message { · 11 unmodified lines
347349 MB_RES_MSG_GPU_RMA = 10,
348350 MB_RES_MSG_RAS_ERROR_COUNT_READY = 11,
349351 MB_REQ_RAS_CPER_DUMP_READY = 14,
352+ MB_RES_MSG_RAS_BAD_PAGES_READY = 15,
353+ MB_RES_MSG_RAS_BAD_PAGES_NOTIFICATION = 16,
354+ MB_RES_MSG_UNRECOV_ERR_NOTIFICATION = 17,
350355 MB_RES_MSG_TEXT_MESSAGE = 255
351356 };
352357
drivers/gpu/drm/amd/amdgpu/atom.c+1−0
@@ -1444,6 +1444,7 @@ static void atom_get_vbios_pn(struct atom_context *ctx) · 1443 unmodified lines
14441444 if (vbios_str == NULL)
14451445 vbios_str += sizeof(BIOS_ATOM_PREFIX) - 1;
14461446 }
1447+ OPTIMIZER_HIDE_VAR(vbios_str);
14471448 if (vbios_str != NULL && *vbios_str == 0)
14481449 vbios_str++;
14491450
drivers/gpu/drm/amd/amdgpu/cik_sdma.c+4−14
@@ -56,6 +56,8 @@ static void cik_sdma_set_buffer_funcs(struct amdgpu_device *adev); · 55 unmodified lines
5656 static void cik_sdma_set_vm_pte_funcs(struct amdgpu_device *adev);
5757 static int cik_sdma_soft_reset(struct amdgpu_ip_block *ip_block);
5858
59+u32 amdgpu_cik_gpu_check_soft_reset(struct amdgpu_device *adev);
60+
5961 MODULE_FIRMWARE("amdgpu/bonaire_sdma.bin");
6062 MODULE_FIRMWARE("amdgpu/bonaire_sdma1.bin");
6163 MODULE_FIRMWARE("amdgpu/hawaii_sdma.bin");
@@ -67,9 +69,6 @@ MODULE_FIRMWARE("amdgpu/kabini_sdma1.bin"); · 5 unmodified lines
6769 MODULE_FIRMWARE("amdgpu/mullins_sdma.bin");
6870 MODULE_FIRMWARE("amdgpu/mullins_sdma1.bin");
6971
70−u32 amdgpu_cik_gpu_check_soft_reset(struct amdgpu_device *adev);
71−
72−
7372 static void cik_sdma_free_microcode(struct amdgpu_device *adev)
7473 {
7574 int i;
@@ -993,14 +992,9 @@ static int cik_sdma_sw_fini(struct amdgpu_ip_block *ip_block) · 917 unmodified lines
993992
994993 static int cik_sdma_hw_init(struct amdgpu_ip_block *ip_block)
995994 {
996− int r;
997995 struct amdgpu_device *adev = ip_block->adev;
998996
999− r = cik_sdma_start(adev);
1000− if (r)
1001− return r;
1002−
1003− return r;
997+ return cik_sdma_start(adev);
1004998 }
1005999
10061000 static int cik_sdma_hw_fini(struct amdgpu_ip_block *ip_block)
@@ -1040,14 +1034,10 @@ static bool cik_sdma_is_idle(struct amdgpu_ip_block *ip_block) · 33 unmodified lines
10401034 static int cik_sdma_wait_for_idle(struct amdgpu_ip_block *ip_block)
10411035 {
10421036 unsigned i;
1043− u32 tmp;
10441037 struct amdgpu_device *adev = ip_block->adev;
10451038
10461039 for (i = 0; i < adev->usec_timeout; i++) {
1047− tmp = RREG32(mmSRBM_STATUS2) & (SRBM_STATUS2__SDMA_BUSY_MASK |
1048− SRBM_STATUS2__SDMA1_BUSY_MASK);
1049−
1050− if (!tmp)
1040+ if (cik_sdma_is_idle(ip_block))
10511041 return 0;
10521042 udelay(1);
10531043 }
drivers/gpu/drm/amd/amdgpu/cikd.h+0−3
@@ -60,9 +60,6 @@ · 59 unmodified lines
6060 #define AUD5_REGISTER_OFFSET (0x179d - 0x1780)
6161 #define AUD6_REGISTER_OFFSET (0x17a4 - 0x1780)
6262
63−#define BONAIRE_GB_ADDR_CONFIG_GOLDEN 0x12010001
64−#define HAWAII_GB_ADDR_CONFIG_GOLDEN 0x12011003
65−
6663 #define PIPEID(x) ((x) << 0)
6764 #define MEID(x) ((x) << 2)
6865 #define VMID(x) ((x) << 4)
drivers/gpu/drm/amd/amdgpu/dce_v10_0.c+2−2
@@ -3075,7 +3075,7 @@ static int dce_v10_0_set_hpd_irq_state(struct amdgpu_device *adev, · 3074 unmodified lines
30753075 u32 tmp;
30763076
30773077 if (hpd >= adev->mode_info.num_hpd) {
3078− DRM_DEBUG("invalid hdp %d\n", hpd);
3078+ DRM_DEBUG("invalid hpd %d\n", hpd);
30793079 return 0;
30803080 }
30813081
@@ -3227,7 +3227,7 @@ static void dce_v10_0_hpd_int_ack(struct amdgpu_device *adev, · 145 unmodified lines
32273227 u32 tmp;
32283228
32293229 if (hpd >= adev->mode_info.num_hpd) {
3230− DRM_DEBUG("invalid hdp %d\n", hpd);
3230+ DRM_DEBUG("invalid hpd %d\n", hpd);
32313231 return;
32323232 }
32333233
drivers/gpu/drm/amd/amdgpu/dce_v11_0.c+3−4
@@ -3206,7 +3206,7 @@ static int dce_v11_0_set_hpd_irq_state(struct amdgpu_device *adev, · 3205 unmodified lines
32063206 u32 tmp;
32073207
32083208 if (hpd >= adev->mode_info.num_hpd) {
3209− DRM_DEBUG("invalid hdp %d\n", hpd);
3209+ DRM_DEBUG("invalid hpd %d\n", hpd);
32103210 return 0;
32113211 }
32123212
@@ -3358,7 +3358,7 @@ static void dce_v11_0_hpd_int_ack(struct amdgpu_device *adev, · 145 unmodified lines
33583358 u32 tmp;
33593359
33603360 if (hpd >= adev->mode_info.num_hpd) {
3361− DRM_DEBUG("invalid hdp %d\n", hpd);
3361+ DRM_DEBUG("invalid hpd %d\n", hpd);
33623362 return;
33633363 }
33643364
@@ -3488,8 +3488,7 @@ static const struct amd_ip_funcs dce_v11_0_ip_funcs = { · 123 unmodified lines
34883488 .set_powergating_state = dce_v11_0_set_powergating_state,
34893489 };
34903490
3491−static void
3492−dce_v11_0_encoder_mode_set(struct drm_encoder *encoder,
3491+static void dce_v11_0_encoder_mode_set(struct drm_encoder *encoder,
34933492 struct drm_display_mode *mode,
34943493 struct drm_display_mode *adjusted_mode)
34953494 {
drivers/gpu/drm/amd/amdgpu/dce_v6_0.c+100−106
@@ -287,7 +287,7 @@ static void dce_v6_0_hpd_int_ack(struct amdgpu_device *adev, · 286 unmodified lines
287287 u32 tmp;
288288
289289 if (hpd >= adev->mode_info.num_hpd) {
290− DRM_DEBUG("invalid hdp %d\n", hpd);
290+ DRM_DEBUG("invalid hpd %d\n", hpd);
291291 return;
292292 }
293293
@@ -412,7 +412,7 @@ static void dce_v6_0_set_vga_render_state(struct amdgpu_device *adev, · 118 unmodified lines
412412 {
413413 if (!render)
414414 WREG32(mmVGA_RENDER_CONTROL,
415− RREG32(mmVGA_RENDER_CONTROL) & VGA_VSTATUS_CNTL);
415+ RREG32(mmVGA_RENDER_CONTROL) & ~VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK);
416416 }
417417
418418 static int dce_v6_0_get_num_crtc(struct amdgpu_device *adev)
@@ -1011,16 +1011,16 @@ static void dce_v6_0_program_watermarks(struct amdgpu_device *adev, · 592 unmodified lines
10111011 /* select wm A */
10121012 arb_control3 = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset);
10131013 tmp = arb_control3;
1014− tmp &= ~LATENCY_WATERMARK_MASK(3);
1015− tmp |= LATENCY_WATERMARK_MASK(1);
1014+ tmp &= ~(3 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
1015+ tmp |= (1 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
10161016 WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp);
10171017 WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset,
10181018 ((latency_watermark_a << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) |
10191019 (line_time << DPG_PIPE_URGENCY_CONTROL__URGENCY_HIGH_WATERMARK__SHIFT)));
10201020 /* select wm B */
10211021 tmp = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset);
1022− tmp &= ~LATENCY_WATERMARK_MASK(3);
1023− tmp |= LATENCY_WATERMARK_MASK(2);
1022+ tmp &= ~(3 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
1023+ tmp |= (2 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
10241024 WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp);
10251025 WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset,
10261026 ((latency_watermark_b << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) |
@@ -1089,7 +1089,7 @@ static u32 dce_v6_0_line_buffer_adjust(struct amdgpu_device *adev, · 62 unmodified lines
10891089 }
10901090
10911091 WREG32(mmDC_LB_MEMORY_SPLIT + amdgpu_crtc->crtc_offset,
1092− DC_LB_MEMORY_CONFIG(tmp));
1092+ (tmp << DC_LB_MEMORY_SPLIT__DC_LB_MEMORY_CONFIG__SHIFT));
10931093
10941094 WREG32(mmPIPE0_DMIF_BUFFER_CONTROL + pipe_offset,
10951095 (buffer_alloc << PIPE0_DMIF_BUFFER_CONTROL__DMIF_BUFFERS_ALLOCATED__SHIFT));
@@ -1306,6 +1306,7 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 210 unmodified lines
13061306 struct amdgpu_device *adev = drm_to_adev(dev);
13071307 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
13081308 struct amdgpu_encoder_atom_dig *dig = amdgpu_encoder->enc_priv;
1309+ u32 offset;
13091310 struct drm_connector *connector;
13101311 struct drm_connector_list_iter iter;
13111312 struct amdgpu_connector *amdgpu_connector = NULL;
@@ -1327,6 +1328,11 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 15 unmodified lines
13271328 { ixAZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13, HDMI_AUDIO_CODING_TYPE_WMA_PRO },
13281329 };
13291330
1331+ if (!dig || !dig->afmt || !dig->afmt->pin)
1332+ return;
1333+
1334+ offset = dig->afmt->pin->offset;
1335+
13301336 drm_connector_list_iter_begin(dev, &iter);
13311337 drm_for_each_connector_iter(connector, &iter) {
13321338 if (connector->encoder == encoder) {
@@ -1348,7 +1354,7 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 15 unmodified lines
13481354 return;
13491355
13501356 for (i = 0; i < ARRAY_SIZE(eld_reg_to_type); i++) {
1351− u32 tmp = 0;
1357+ u32 value = 0;
13521358 u8 stereo_freqs = 0;
13531359 int max_channels = -1;
13541360 int j;
@@ -1358,12 +1364,12 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 3 unmodified lines
13581364
13591365 if (sad->format == eld_reg_to_type[i][1]) {
13601366 if (sad->channels > max_channels) {
1361− tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,
1362− MAX_CHANNELS, sad->channels);
1363− tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,
1364− DESCRIPTOR_BYTE_2, sad->byte2);
1365− tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,
1366− SUPPORTED_FREQUENCIES, sad->freq);
1367+ value = (sad->channels <<
1368+ AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__MAX_CHANNELS__SHIFT) |
1369+ (sad->byte2 <<
1370+ AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__DESCRIPTOR_BYTE_2__SHIFT) |
1371+ (sad->freq <<
1372+ AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__SUPPORTED_FREQUENCIES__SHIFT);
13671373 max_channels = sad->channels;
13681374 }
13691375
@@ -1374,13 +1380,13 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 4 unmodified lines
13741380 }
13751381 }
13761382
1377− tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,
1378− SUPPORTED_FREQUENCIES_STEREO, stereo_freqs);
1379− WREG32_AUDIO_ENDPT(dig->afmt->pin->offset, eld_reg_to_type[i][0], tmp);
1383+ value |= (stereo_freqs <<
1384+ AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__SUPPORTED_FREQUENCIES_STEREO__SHIFT);
1385+
1386+ WREG32_AUDIO_ENDPT(offset, eld_reg_to_type[i][0], value);
13801387 }
13811388
13821389 kfree(sads);
1383−
13841390 }
13851391
13861392 static void dce_v6_0_audio_enable(struct amdgpu_device *adev,
@@ -1886,7 +1892,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 499 unmodified lines
18861892 struct amdgpu_bo *abo;
18871893 uint64_t fb_location, tiling_flags;
18881894 uint32_t fb_format, fb_pitch_pixels, pipe_config;
1889− u32 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_NONE);
1895+ u32 fb_swap = (GRPH_ENDIAN_NONE << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
18901896 u32 viewport_w, viewport_h;
18911897 int r;
18921898 bool bypass_lut = false;
@@ -1926,76 +1932,76 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 33 unmodified lines
19261932
19271933 switch (target_fb->format->format) {
19281934 case DRM_FORMAT_C8:
1929− fb_format = (GRPH_DEPTH(GRPH_DEPTH_8BPP) |
1930− GRPH_FORMAT(GRPH_FORMAT_INDEXED));
1935+ fb_format = ((GRPH_DEPTH_8BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1936+ (GRPH_FORMAT_INDEXED << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19311937 break;
19321938 case DRM_FORMAT_XRGB4444:
19331939 case DRM_FORMAT_ARGB4444:
1934− fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |
1935− GRPH_FORMAT(GRPH_FORMAT_ARGB4444));
1940+ fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1941+ (GRPH_FORMAT_ARGB4444 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19361942 #ifdef __BIG_ENDIAN
1937− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);
1943+ fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19381944 #endif
19391945 break;
19401946 case DRM_FORMAT_XRGB1555:
19411947 case DRM_FORMAT_ARGB1555:
1942− fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |
1943− GRPH_FORMAT(GRPH_FORMAT_ARGB1555));
1948+ fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1949+ (GRPH_FORMAT_ARGB1555 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19441950 #ifdef __BIG_ENDIAN
1945− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);
1951+ fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19461952 #endif
19471953 break;
19481954 case DRM_FORMAT_BGRX5551:
19491955 case DRM_FORMAT_BGRA5551:
1950− fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |
1951− GRPH_FORMAT(GRPH_FORMAT_BGRA5551));
1956+ fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1957+ (GRPH_FORMAT_BGRA5551 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19521958 #ifdef __BIG_ENDIAN
1953− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);
1959+ fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19541960 #endif
19551961 break;
19561962 case DRM_FORMAT_RGB565:
1957− fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |
1958− GRPH_FORMAT(GRPH_FORMAT_ARGB565));
1963+ fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1964+ (GRPH_FORMAT_ARGB565 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19591965 #ifdef __BIG_ENDIAN
1960− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);
1966+ fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19611967 #endif
19621968 break;
19631969 case DRM_FORMAT_XRGB8888:
19641970 case DRM_FORMAT_ARGB8888:
1965− fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |
1966− GRPH_FORMAT(GRPH_FORMAT_ARGB8888));
1971+ fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1972+ (GRPH_FORMAT_ARGB8888 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19671973 #ifdef __BIG_ENDIAN
1968− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);
1974+ fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19691975 #endif
19701976 break;
19711977 case DRM_FORMAT_XRGB2101010:
19721978 case DRM_FORMAT_ARGB2101010:
1973− fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |
1974− GRPH_FORMAT(GRPH_FORMAT_ARGB2101010));
1979+ fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1980+ (GRPH_FORMAT_ARGB2101010 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19751981 #ifdef __BIG_ENDIAN
1976− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);
1982+ fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19771983 #endif
19781984 /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */
19791985 bypass_lut = true;
19801986 break;
19811987 case DRM_FORMAT_BGRX1010102:
19821988 case DRM_FORMAT_BGRA1010102:
1983− fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |
1984− GRPH_FORMAT(GRPH_FORMAT_BGRA1010102));
1989+ fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1990+ (GRPH_FORMAT_BGRA1010102 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
19851991 #ifdef __BIG_ENDIAN
1986− fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);
1992+ fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19871993 #endif
19881994 /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */
19891995 bypass_lut = true;
19901996 break;
19911997 case DRM_FORMAT_XBGR8888:
19921998 case DRM_FORMAT_ABGR8888:
1993− fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |
1994− GRPH_FORMAT(GRPH_FORMAT_ARGB8888));
1995− fb_swap = (GRPH_RED_CROSSBAR(GRPH_RED_SEL_B) |
1996− GRPH_BLUE_CROSSBAR(GRPH_BLUE_SEL_R));
1999+ fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
2000+ (GRPH_FORMAT_ARGB8888 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
2001+ fb_swap = ((GRPH_RED_SEL_B << GRPH_SWAP_CNTL__GRPH_RED_CROSSBAR__SHIFT) |
2002+ (GRPH_BLUE_SEL_R << GRPH_SWAP_CNTL__GRPH_BLUE_CROSSBAR__SHIFT));
19972003 #ifdef __BIG_ENDIAN
1998− fb_swap |= GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);
2004+ fb_swap |= (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
19992005 #endif
20002006 break;
20012007 default:
@@ -2013,18 +2019,18 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 11 unmodified lines
20132019 tile_split = AMDGPU_TILING_GET(tiling_flags, TILE_SPLIT);
20142020 num_banks = AMDGPU_TILING_GET(tiling_flags, NUM_BANKS);
20152021
2016− fb_format |= GRPH_NUM_BANKS(num_banks);
2017− fb_format |= GRPH_ARRAY_MODE(GRPH_ARRAY_2D_TILED_THIN1);
2018− fb_format |= GRPH_TILE_SPLIT(tile_split);
2019− fb_format |= GRPH_BANK_WIDTH(bankw);
2020− fb_format |= GRPH_BANK_HEIGHT(bankh);
2021− fb_format |= GRPH_MACRO_TILE_ASPECT(mtaspect);
2022+ fb_format |= (num_banks << GRPH_CONTROL__GRPH_NUM_BANKS__SHIFT);
2023+ fb_format |= (GRPH_ARRAY_2D_TILED_THIN1 << GRPH_CONTROL__GRPH_ARRAY_MODE__SHIFT);
2024+ fb_format |= (tile_split << GRPH_CONTROL__GRPH_TILE_SPLIT__SHIFT);
2025+ fb_format |= (bankw << GRPH_CONTROL__GRPH_BANK_WIDTH__SHIFT);
2026+ fb_format |= (bankh << GRPH_CONTROL__GRPH_BANK_HEIGHT__SHIFT);
2027+ fb_format |= (mtaspect << GRPH_CONTROL__GRPH_MACRO_TILE_ASPECT__SHIFT);
20222028 } else if (AMDGPU_TILING_GET(tiling_flags, ARRAY_MODE) == ARRAY_1D_TILED_THIN1) {
2023− fb_format |= GRPH_ARRAY_MODE(GRPH_ARRAY_1D_TILED_THIN1);
2029+ fb_format |= (GRPH_ARRAY_1D_TILED_THIN1 << GRPH_CONTROL__GRPH_ARRAY_MODE__SHIFT);
20242030 }
20252031
20262032 pipe_config = AMDGPU_TILING_GET(tiling_flags, PIPE_CONFIG);
2027− fb_format |= GRPH_PIPE_CONFIG(pipe_config);
2033+ fb_format |= (pipe_config << GRPH_CONTROL__GRPH_PIPE_CONFIG__SHIFT);
20282034
20292035 dce_v6_0_vga_enable(crtc, false);
20302036
@@ -2040,7 +2046,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 9 unmodified lines
20402046 WREG32(mmGRPH_PRIMARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset,
20412047 (u32)fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK);
20422048 WREG32(mmGRPH_SECONDARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset,
2043− (u32) fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK);
2049+ (u32) fb_location & GRPH_SECONDARY_SURFACE_ADDRESS__GRPH_SECONDARY_SURFACE_ADDRESS_MASK);
20442050 WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format);
20452051 WREG32(mmGRPH_SWAP_CNTL + amdgpu_crtc->crtc_offset, fb_swap);
20462052
@@ -2108,14 +2114,13 @@ static void dce_v6_0_set_interleave(struct drm_crtc *crtc, · 61 unmodified lines
21082114
21092115 if (mode->flags & DRM_MODE_FLAG_INTERLACE)
21102116 WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset,
2111− INTERLEAVE_EN);
2117+ DATA_FORMAT__INTERLEAVE_EN_MASK);
21122118 else
21132119 WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, 0);
21142120 }
21152121
21162122 static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc)
21172123 {
2118−
21192124 struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc);
21202125 struct drm_device *dev = crtc->dev;
21212126 struct amdgpu_device *adev = drm_to_adev(dev);
@@ -2125,15 +2130,15 @@ static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) · 3 unmodified lines
21252130 DRM_DEBUG_KMS("%d\n", amdgpu_crtc->crtc_id);
21262131
21272132 WREG32(mmINPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset,
2128− ((0 << INPUT_CSC_CONTROL__INPUT_CSC_GRPH_MODE__SHIFT) |
2129− (0 << INPUT_CSC_CONTROL__INPUT_CSC_OVL_MODE__SHIFT)));
2133+ ((INPUT_CSC_BYPASS << INPUT_CSC_CONTROL__INPUT_CSC_GRPH_MODE__SHIFT) |
2134+ (INPUT_CSC_BYPASS << INPUT_CSC_CONTROL__INPUT_CSC_OVL_MODE__SHIFT)));
21302135 WREG32(mmPRESCALE_GRPH_CONTROL + amdgpu_crtc->crtc_offset,
21312136 PRESCALE_GRPH_CONTROL__GRPH_PRESCALE_BYPASS_MASK);
21322137 WREG32(mmPRESCALE_OVL_CONTROL + amdgpu_crtc->crtc_offset,
21332138 PRESCALE_OVL_CONTROL__OVL_PRESCALE_BYPASS_MASK);
21342139 WREG32(mmINPUT_GAMMA_CONTROL + amdgpu_crtc->crtc_offset,
2135− ((0 << INPUT_GAMMA_CONTROL__GRPH_INPUT_GAMMA_MODE__SHIFT) |
2136− (0 << INPUT_GAMMA_CONTROL__OVL_INPUT_GAMMA_MODE__SHIFT)));
2140+ ((INPUT_GAMMA_USE_LUT << INPUT_GAMMA_CONTROL__GRPH_INPUT_GAMMA_MODE__SHIFT) |
2141+ (INPUT_GAMMA_USE_LUT << INPUT_GAMMA_CONTROL__OVL_INPUT_GAMMA_MODE__SHIFT)));
21372142
21382143 WREG32(mmDC_LUT_CONTROL + amdgpu_crtc->crtc_offset, 0);
21392144
@@ -2160,19 +2165,19 @@ static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) · 20 unmodified lines
21602165 }
21612166
21622167 WREG32(mmDEGAMMA_CONTROL + amdgpu_crtc->crtc_offset,
2163− ((0 << DEGAMMA_CONTROL__GRPH_DEGAMMA_MODE__SHIFT) |
2164− (0 << DEGAMMA_CONTROL__OVL_DEGAMMA_MODE__SHIFT) |
2165− ICON_DEGAMMA_MODE(0) |
2166− (0 << DEGAMMA_CONTROL__CURSOR_DEGAMMA_MODE__SHIFT)));
2168+ ((DEGAMMA_BYPASS << DEGAMMA_CONTROL__GRPH_DEGAMMA_MODE__SHIFT) |
2169+ (DEGAMMA_BYPASS << DEGAMMA_CONTROL__OVL_DEGAMMA_MODE__SHIFT) |
2170+ (DEGAMMA_BYPASS << DEGAMMA_CONTROL__ICON_DEGAMMA_MODE__SHIFT) |
2171+ (DEGAMMA_BYPASS << DEGAMMA_CONTROL__CURSOR_DEGAMMA_MODE__SHIFT)));
21672172 WREG32(mmGAMUT_REMAP_CONTROL + amdgpu_crtc->crtc_offset,
2168− ((0 << GAMUT_REMAP_CONTROL__GRPH_GAMUT_REMAP_MODE__SHIFT) |
2169− (0 << GAMUT_REMAP_CONTROL__OVL_GAMUT_REMAP_MODE__SHIFT)));
2173+ ((GAMUT_REMAP_BYPASS << GAMUT_REMAP_CONTROL__GRPH_GAMUT_REMAP_MODE__SHIFT) |
2174+ (GAMUT_REMAP_BYPASS << GAMUT_REMAP_CONTROL__OVL_GAMUT_REMAP_MODE__SHIFT)));
21702175 WREG32(mmREGAMMA_CONTROL + amdgpu_crtc->crtc_offset,
2171− ((0 << REGAMMA_CONTROL__GRPH_REGAMMA_MODE__SHIFT) |
2172− (0 << REGAMMA_CONTROL__OVL_REGAMMA_MODE__SHIFT)));
2176+ ((REGAMMA_BYPASS << REGAMMA_CONTROL__GRPH_REGAMMA_MODE__SHIFT) |
2177+ (REGAMMA_BYPASS << REGAMMA_CONTROL__OVL_REGAMMA_MODE__SHIFT)));
21732178 WREG32(mmOUTPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset,
2174− ((0 << OUTPUT_CSC_CONTROL__OUTPUT_CSC_GRPH_MODE__SHIFT) |
2175− (0 << OUTPUT_CSC_CONTROL__OUTPUT_CSC_OVL_MODE__SHIFT)));
2179+ ((OUTPUT_CSC_BYPASS << OUTPUT_CSC_CONTROL__OUTPUT_CSC_GRPH_MODE__SHIFT) |
2180+ (OUTPUT_CSC_BYPASS << OUTPUT_CSC_CONTROL__OUTPUT_CSC_OVL_MODE__SHIFT)));
21762181 /* XXX match this to the depth of the crtc fmt block, move to modeset? */
21772182 WREG32(0x1a50 + amdgpu_crtc->crtc_offset, 0);
21782183
@@ -2267,8 +2272,6 @@ static void dce_v6_0_hide_cursor(struct drm_crtc *crtc) · 88 unmodified lines
22672272 WREG32(mmCUR_CONTROL + amdgpu_crtc->crtc_offset,
22682273 (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) |
22692274 (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT));
2270−
2271−
22722275 }
22732276
22742277 static void dce_v6_0_show_cursor(struct drm_crtc *crtc)
@@ -2285,7 +2288,6 @@ static void dce_v6_0_show_cursor(struct drm_crtc *crtc) · 10 unmodified lines
22852288 CUR_CONTROL__CURSOR_EN_MASK |
22862289 (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) |
22872290 (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT));
2288−
22892291 }
22902292
22912293 static int dce_v6_0_cursor_move_locked(struct drm_crtc *crtc,
@@ -2596,7 +2598,6 @@ static bool dce_v6_0_crtc_mode_fixup(struct drm_crtc *crtc, · 304 unmodified lines
25962598 const struct drm_display_mode *mode,
25972599 struct drm_display_mode *adjusted_mode)
25982600 {
2599−
26002601 struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc);
26012602 struct drm_device *dev = crtc->dev;
26022603 struct drm_encoder *encoder;
@@ -2669,7 +2670,7 @@ static void dce_v6_0_panic_flush(struct drm_plane *plane) · 66 unmodified lines
26692670
26702671 /* Disable DC tiling */
26712672 fb_format = RREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset);
2672− fb_format &= ~GRPH_ARRAY_MODE(0x7);
2673+ fb_format &= ~GRPH_CONTROL__GRPH_ARRAY_MODE_MASK;
26732674 WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format);
26742675
26752676 }
@@ -2745,7 +2746,6 @@ static int dce_v6_0_early_init(struct amdgpu_ip_block *ip_block) · 69 unmodified lines
27452746 static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block)
27462747 {
27472748 int r, i;
2748− bool ret;
27492749 struct amdgpu_device *adev = ip_block->adev;
27502750
27512751 for (i = 0; i < adev->mode_info.num_crtc; i++) {
@@ -2789,8 +2789,7 @@ static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 37 unmodified lines
27892789 return r;
27902790 }
27912791
2792− ret = amdgpu_atombios_get_connector_info_from_object_table(adev);
2793− if (ret)
2792+ if (amdgpu_atombios_get_connector_info_from_object_table(adev))
27942793 amdgpu_display_print_display_setup(adev_to_drm(adev));
27952794 else
27962795 return -EINVAL;
@@ -2986,12 +2985,12 @@ static void dce_v6_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev, · 189 unmodified lines
29862985 switch (state) {
29872986 case AMDGPU_IRQ_STATE_DISABLE:
29882987 interrupt_mask = RREG32(mmINT_MASK + reg_block);
2989− interrupt_mask &= ~VBLANK_INT_MASK;
2988+ interrupt_mask &= ~INT_MASK__VBLANK_INT_MASK;
29902989 WREG32(mmINT_MASK + reg_block, interrupt_mask);
29912990 break;
29922991 case AMDGPU_IRQ_STATE_ENABLE:
29932992 interrupt_mask = RREG32(mmINT_MASK + reg_block);
2994− interrupt_mask |= VBLANK_INT_MASK;
2993+ interrupt_mask |= INT_MASK__VBLANK_INT_MASK;
29952994 WREG32(mmINT_MASK + reg_block, interrupt_mask);
29962995 break;
29972996 default:
@@ -3006,28 +3005,28 @@ static void dce_v6_0_set_crtc_vline_interrupt_state(struct amdgpu_device *adev, · 8 unmodified lines
30063005
30073006 }
30083007
3009−static int dce_v6_0_set_hpd_interrupt_state(struct amdgpu_device *adev,
3008+static int dce_v6_0_set_hpd_irq_state(struct amdgpu_device *adev,
30103009 struct amdgpu_irq_src *src,
3011− unsigned type,
3010+ unsigned hpd,
30123011 enum amdgpu_interrupt_state state)
30133012 {
30143013 u32 dc_hpd_int_cntl;
30153014
3016− if (type >= adev->mode_info.num_hpd) {
3017− DRM_DEBUG("invalid hdp %d\n", type);
3015+ if (hpd >= adev->mode_info.num_hpd) {
3016+ DRM_DEBUG("invalid hpd %d\n", hpd);
30183017 return 0;
30193018 }
30203019
30213020 switch (state) {
30223021 case AMDGPU_IRQ_STATE_DISABLE:
3023− dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type]);
3024− dc_hpd_int_cntl &= ~DC_HPDx_INT_EN;
3025− WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type], dc_hpd_int_cntl);
3022+ dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd]);
3023+ dc_hpd_int_cntl &= ~DC_HPD1_INT_CONTROL__DC_HPD1_INT_EN_MASK;
3024+ WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], dc_hpd_int_cntl);
30263025 break;
30273026 case AMDGPU_IRQ_STATE_ENABLE:
3028− dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type]);
3029− dc_hpd_int_cntl |= DC_HPDx_INT_EN;
3030− WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type], dc_hpd_int_cntl);
3027+ dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd]);
3028+ dc_hpd_int_cntl |= DC_HPD1_INT_CONTROL__DC_HPD1_INT_EN_MASK;
3029+ WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], dc_hpd_int_cntl);
30313030 break;
30323031 default:
30333032 break;
@@ -3036,7 +3035,7 @@ static int dce_v6_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 2 unmodified lines
30363035 return 0;
30373036 }
30383037
3039−static int dce_v6_0_set_crtc_interrupt_state(struct amdgpu_device *adev,
3038+static int dce_v6_0_set_crtc_irq_state(struct amdgpu_device *adev,
30403039 struct amdgpu_irq_src *src,
30413040 unsigned type,
30423041 enum amdgpu_interrupt_state state)
@@ -3096,7 +3095,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 53 unmodified lines
30963095 switch (entry->src_data[0]) {
30973096 case 0: /* vblank */
30983097 if (disp_int & interrupt_status_offsets[crtc].vblank)
3099− WREG32(mmVBLANK_STATUS + crtc_offsets[crtc], VBLANK_ACK);
3098+ WREG32(mmVBLANK_STATUS + crtc_offsets[crtc], VBLANK_STATUS__VBLANK_ACK_MASK);
31003099 else
31013100 DRM_DEBUG("IH: IH event w/o asserted irq bit?\n");
31023101
@@ -3107,7 +3106,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 4 unmodified lines
31073106 break;
31083107 case 1: /* vline */
31093108 if (disp_int & interrupt_status_offsets[crtc].vline)
3110− WREG32(mmVLINE_STATUS + crtc_offsets[crtc], VLINE_ACK);
3109+ WREG32(mmVLINE_STATUS + crtc_offsets[crtc], VLINE_STATUS__VLINE_ACK_MASK);
31113110 else
31123111 DRM_DEBUG("IH: IH event w/o asserted irq bit?\n");
31133112
@@ -3121,7 +3120,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 7 unmodified lines
31213120 return 0;
31223121 }
31233122
3124−static int dce_v6_0_set_pageflip_interrupt_state(struct amdgpu_device *adev,
3123+static int dce_v6_0_set_pageflip_irq_state(struct amdgpu_device *adev,
31253124 struct amdgpu_irq_src *src,
31263125 unsigned type,
31273126 enum amdgpu_interrupt_state state)
@@ -3172,7 +3171,7 @@ static int dce_v6_0_pageflip_irq(struct amdgpu_device *adev, · 44 unmodified lines
31723171
31733172 spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);
31743173 works = amdgpu_crtc->pflip_works;
3175− if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED){
3174+ if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED) {
31763175 DRM_DEBUG_DRIVER("amdgpu_crtc->pflip_status = %d != "
31773176 "AMDGPU_FLIP_SUBMITTED(%d)\n",
31783177 amdgpu_crtc->pflip_status,
@@ -3249,12 +3248,10 @@ static const struct amd_ip_funcs dce_v6_0_ip_funcs = { · 70 unmodified lines
32493248 .set_powergating_state = dce_v6_0_set_powergating_state,
32503249 };
32513250
3252−static void
3253−dce_v6_0_encoder_mode_set(struct drm_encoder *encoder,
3251+static void dce_v6_0_encoder_mode_set(struct drm_encoder *encoder,
32543252 struct drm_display_mode *mode,
32553253 struct drm_display_mode *adjusted_mode)
32563254 {
3257−
32583255 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
32593256 int em = amdgpu_atombios_encoder_get_encoder_mode(encoder);
32603257
@@ -3274,7 +3271,6 @@ dce_v6_0_encoder_mode_set(struct drm_encoder *encoder, · 13 unmodified lines
32743271
32753272 static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder)
32763273 {
3277−
32783274 struct amdgpu_device *adev = drm_to_adev(encoder->dev);
32793275 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
32803276 struct drm_connector *connector = amdgpu_get_connector_for_encoder(encoder);
@@ -3314,7 +3310,6 @@ static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder) · 33 unmodified lines
33143310
33153311 static void dce_v6_0_encoder_commit(struct drm_encoder *encoder)
33163312 {
3317−
33183313 struct drm_device *dev = encoder->dev;
33193314 struct amdgpu_device *adev = drm_to_adev(dev);
33203315
@@ -3325,7 +3320,6 @@ static void dce_v6_0_encoder_commit(struct drm_encoder *encoder) · 4 unmodified lines
33253320
33263321 static void dce_v6_0_encoder_disable(struct drm_encoder *encoder)
33273322 {
3328−
33293323 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
33303324 struct amdgpu_encoder_atom_dig *dig;
33313325 int em = amdgpu_atombios_encoder_get_encoder_mode(encoder);
@@ -3541,17 +3535,17 @@ static void dce_v6_0_set_display_funcs(struct amdgpu_device *adev) · 209 unmodified lines
35413535 }
35423536
35433537 static const struct amdgpu_irq_src_funcs dce_v6_0_crtc_irq_funcs = {
3544− .set = dce_v6_0_set_crtc_interrupt_state,
3538+ .set = dce_v6_0_set_crtc_irq_state,
35453539 .process = dce_v6_0_crtc_irq,
35463540 };
35473541
35483542 static const struct amdgpu_irq_src_funcs dce_v6_0_pageflip_irq_funcs = {
3549− .set = dce_v6_0_set_pageflip_interrupt_state,
3543+ .set = dce_v6_0_set_pageflip_irq_state,
35503544 .process = dce_v6_0_pageflip_irq,
35513545 };
35523546
35533547 static const struct amdgpu_irq_src_funcs dce_v6_0_hpd_irq_funcs = {
3554− .set = dce_v6_0_set_hpd_interrupt_state,
3548+ .set = dce_v6_0_set_hpd_irq_state,
35553549 .process = dce_v6_0_hpd_irq,
35563550 };
35573551
drivers/gpu/drm/amd/amdgpu/dce_v8_0.c+8−8
@@ -271,7 +271,7 @@ static void dce_v8_0_hpd_int_ack(struct amdgpu_device *adev, · 270 unmodified lines
271271 u32 tmp;
272272
273273 if (hpd >= adev->mode_info.num_hpd) {
274− DRM_DEBUG("invalid hdp %d\n", hpd);
274+ DRM_DEBUG("invalid hpd %d\n", hpd);
275275 return;
276276 }
277277
@@ -3021,7 +3021,7 @@ static void dce_v8_0_set_crtc_vline_interrupt_state(struct amdgpu_device *adev, · 2743 unmodified lines
30213021 }
30223022 }
30233023
3024−static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev,
3024+static int dce_v8_0_set_hpd_irq_state(struct amdgpu_device *adev,
30253025 struct amdgpu_irq_src *src,
30263026 unsigned type,
30273027 enum amdgpu_interrupt_state state)
@@ -3029,7 +3029,7 @@ static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 1 unmodified line
30293029 u32 dc_hpd_int_cntl;
30303030
30313031 if (type >= adev->mode_info.num_hpd) {
3032− DRM_DEBUG("invalid hdp %d\n", type);
3032+ DRM_DEBUG("invalid hpd %d\n", type);
30333033 return 0;
30343034 }
30353035
@@ -3051,7 +3051,7 @@ static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 15 unmodified lines
30513051 return 0;
30523052 }
30533053
3054−static int dce_v8_0_set_crtc_interrupt_state(struct amdgpu_device *adev,
3054+static int dce_v8_0_set_crtc_irq_state(struct amdgpu_device *adev,
30553055 struct amdgpu_irq_src *src,
30563056 unsigned type,
30573057 enum amdgpu_interrupt_state state)
@@ -3136,7 +3136,7 @@ static int dce_v8_0_crtc_irq(struct amdgpu_device *adev, · 78 unmodified lines
31363136 return 0;
31373137 }
31383138
3139−static int dce_v8_0_set_pageflip_interrupt_state(struct amdgpu_device *adev,
3139+static int dce_v8_0_set_pageflip_irq_state(struct amdgpu_device *adev,
31403140 struct amdgpu_irq_src *src,
31413141 unsigned type,
31423142 enum amdgpu_interrupt_state state)
@@ -3547,17 +3547,17 @@ static void dce_v8_0_set_display_funcs(struct amdgpu_device *adev) · 404 unmodified lines
35473547 }
35483548
35493549 static const struct amdgpu_irq_src_funcs dce_v8_0_crtc_irq_funcs = {
3550− .set = dce_v8_0_set_crtc_interrupt_state,
3550+ .set = dce_v8_0_set_crtc_irq_state,
35513551 .process = dce_v8_0_crtc_irq,
35523552 };
35533553
35543554 static const struct amdgpu_irq_src_funcs dce_v8_0_pageflip_irq_funcs = {
3555− .set = dce_v8_0_set_pageflip_interrupt_state,
3555+ .set = dce_v8_0_set_pageflip_irq_state,
35563556 .process = dce_v8_0_pageflip_irq,
35573557 };
35583558
35593559 static const struct amdgpu_irq_src_funcs dce_v8_0_hpd_irq_funcs = {
3560− .set = dce_v8_0_set_hpd_interrupt_state,
3560+ .set = dce_v8_0_set_hpd_irq_state,
35613561 .process = dce_v8_0_hpd_irq,
35623562 };
35633563
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c+91−48
@@ -368,11 +368,6 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_10_1[] = { · 367 unmodified lines
368368 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_DEBUG_INST_ADDR),
369369 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_LX6_CORE_PDEBUG_INST),
370370 /* cp header registers */
371− SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
372− SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
373− SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME2_HEADER_DUMP),
374− SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
375− SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
376371 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MES_HEADER_DUMP),
377372 /* SE status registers */
378373 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0),
@@ -421,7 +416,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_10[] = { · 42 unmodified lines
421416 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_OFFSET),
422417 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),
423418 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_WG_STATE_OFFSET),
424− SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_DEQUEUE_STATUS)
419+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_DEQUEUE_STATUS),
420+ /* cp header registers */
421+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
422+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
423+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
424+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
425+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
426+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
427+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
428+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
425429 };
426430
427431 static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = {
@@ -448,7 +452,32 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = { · 20 unmodified lines
448452 SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR),
449453 SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR_HI),
450454 SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO),
451− SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI)
455+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI),
456+ /* gfx header registers */
457+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
458+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
459+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
460+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
461+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
462+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
463+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
464+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
465+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
466+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
467+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
468+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
469+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
470+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
471+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
472+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
473+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
474+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
475+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
476+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
477+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
478+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
479+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
480+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
452481 };
453482
454483 static const struct soc15_reg_golden golden_settings_gc_10_1[] = {
@@ -4296,9 +4325,7 @@ static u32 gfx_v10_0_get_csb_size(struct amdgpu_device *adev) · 3841 unmodified lines
42964325 static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev,
42974326 volatile u32 *buffer)
42984327 {
4299− u32 count = 0, i;
4300− const struct cs_section_def *sect = NULL;
4301− const struct cs_extent_def *ext = NULL;
4328+ u32 count = 0;
43024329 int ctx_reg_offset;
43034330
43044331 if (adev->gfx.rlc.cs_data == NULL)
@@ -4306,39 +4333,15 @@ static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev, · 1 unmodified line
43064333 if (buffer == NULL)
43074334 return;
43084335
4309− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
4310− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
4336+ count = amdgpu_gfx_csb_preamble_start(buffer);
4337+ count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
43114338
4312− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
4313− buffer[count++] = cpu_to_le32(0x80000000);
4314− buffer[count++] = cpu_to_le32(0x80000000);
4315−
4316− for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
4317− for (ext = sect->section; ext->extent != NULL; ++ext) {
4318− if (sect->id == SECT_CONTEXT) {
4319− buffer[count++] =
4320− cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
4321− buffer[count++] = cpu_to_le32(ext->reg_index -
4322− PACKET3_SET_CONTEXT_REG_START);
4323− for (i = 0; i < ext->reg_count; i++)
4324− buffer[count++] = cpu_to_le32(ext->extent[i]);
4325− } else {
4326− return;
4327− }
4328− }
4329− }
4330−
4331− ctx_reg_offset =
4332− SOC15_REG_OFFSET(GC, 0, mmPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
4339+ ctx_reg_offset = SOC15_REG_OFFSET(GC, 0, mmPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
43334340 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));
43344341 buffer[count++] = cpu_to_le32(ctx_reg_offset);
43354342 buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override);
43364343
4337− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
4338− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
4339−
4340− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
4341− buffer[count++] = cpu_to_le32(0);
4344+ amdgpu_gfx_csb_preamble_end(buffer, count);
43424345 }
43434346
43444347 static void gfx_v10_0_rlc_fini(struct amdgpu_device *adev)
@@ -4752,6 +4755,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 407 unmodified lines
47524755 int i, j, k, r, ring_id = 0;
47534756 int xcc_id = 0;
47544757 struct amdgpu_device *adev = ip_block->adev;
4758+ int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
47554759
47564760 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);
47574761
@@ -4763,7 +4767,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
47634767 case IP_VERSION(10, 1, 4):
47644768 adev->gfx.me.num_me = 1;
47654769 adev->gfx.me.num_pipe_per_me = 1;
4766− adev->gfx.me.num_queue_per_pipe = 1;
4770+ adev->gfx.me.num_queue_per_pipe = 8;
47674771 adev->gfx.mec.num_mec = 2;
47684772 adev->gfx.mec.num_pipe_per_mec = 4;
47694773 adev->gfx.mec.num_queue_per_pipe = 8;
@@ -4778,7 +4782,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
47784782 case IP_VERSION(10, 3, 7):
47794783 adev->gfx.me.num_me = 1;
47804784 adev->gfx.me.num_pipe_per_me = 2;
4781− adev->gfx.me.num_queue_per_pipe = 1;
4785+ adev->gfx.me.num_queue_per_pipe = 2;
47824786 adev->gfx.mec.num_mec = 2;
47834787 adev->gfx.mec.num_pipe_per_mec = 4;
47844788 adev->gfx.mec.num_queue_per_pipe = 4;
@@ -4800,7 +4804,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 15 unmodified lines
48004804 adev->gfx.cleaner_shader_size = sizeof(gfx_10_1_10_cleaner_shader_hex);
48014805 if (adev->gfx.me_fw_version >= 101 &&
48024806 adev->gfx.pfp_fw_version >= 158 &&
4803− adev->gfx.mec_fw_version >= 152) {
4807+ adev->gfx.mec_fw_version >= 151) {
48044808 adev->gfx.enable_cleaner_shader = true;
48054809 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
48064810 if (r) {
@@ -4810,7 +4814,9 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
48104814 }
48114815 break;
48124816 case IP_VERSION(10, 3, 0):
4817+ case IP_VERSION(10, 3, 1):
48134818 case IP_VERSION(10, 3, 2):
4819+ case IP_VERSION(10, 3, 3):
48144820 case IP_VERSION(10, 3, 4):
48154821 case IP_VERSION(10, 3, 5):
48164822 adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;
@@ -4826,6 +4832,34 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 9 unmodified lines
48264832 }
48274833 }
48284834 break;
4835+ case IP_VERSION(10, 3, 6):
4836+ adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;
4837+ adev->gfx.cleaner_shader_size = sizeof(gfx_10_3_0_cleaner_shader_hex);
4838+ if (adev->gfx.me_fw_version >= 14 &&
4839+ adev->gfx.pfp_fw_version >= 17 &&
4840+ adev->gfx.mec_fw_version >= 24) {
4841+ adev->gfx.enable_cleaner_shader = true;
4842+ r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
4843+ if (r) {
4844+ adev->gfx.enable_cleaner_shader = false;
4845+ dev_err(adev->dev, "Failed to initialize cleaner shader\n");
4846+ }
4847+ }
4848+ break;
4849+ case IP_VERSION(10, 3, 7):
4850+ adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;
4851+ adev->gfx.cleaner_shader_size = sizeof(gfx_10_3_0_cleaner_shader_hex);
4852+ if (adev->gfx.me_fw_version >= 4 &&
4853+ adev->gfx.pfp_fw_version >= 9 &&
4854+ adev->gfx.mec_fw_version >= 12) {
4855+ adev->gfx.enable_cleaner_shader = true;
4856+ r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
4857+ if (r) {
4858+ adev->gfx.enable_cleaner_shader = false;
4859+ dev_err(adev->dev, "Failed to initialize cleaner shader\n");
4860+ }
4861+ }
4862+ break;
48294863 default:
48304864 adev->gfx.enable_cleaner_shader = false;
48314865 break;
@@ -4886,7 +4920,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 54 unmodified lines
48864920
48874921 /* set up the gfx ring */
48884922 for (i = 0; i < adev->gfx.me.num_me; i++) {
4889− for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
4923+ for (j = 0; j < num_queue_per_pipe; j++) {
48904924 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
48914925 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
48924926 continue;
@@ -9645,9 +9679,14 @@ static void gfx_v10_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printe · 4752 unmodified lines
96459679 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {
96469680 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);
96479681 for (reg = 0; reg < reg_count; reg++) {
9648− drm_printf(p, "%-50s \t 0x%08x\n",
9649− gc_cp_reg_list_10[reg].reg_name,
9650− adev->gfx.ip_dump_compute_queues[index + reg]);
9682+ if (i && gc_cp_reg_list_10[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
9683+ drm_printf(p, "%-50s \t 0x%08x\n",
9684+ "mmCP_MEC_ME2_HEADER_DUMP",
9685+ adev->gfx.ip_dump_compute_queues[index + reg]);
9686+ else
9687+ drm_printf(p, "%-50s \t 0x%08x\n",
9688+ gc_cp_reg_list_10[reg].reg_name,
9689+ adev->gfx.ip_dump_compute_queues[index + reg]);
96519690 }
96529691 index += reg_count;
96539692 }
@@ -9708,9 +9747,13 @@ static void gfx_v10_ip_dump(struct amdgpu_ip_block *ip_block) · 54 unmodified lines
97089747 nv_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0);
97099748
97109749 for (reg = 0; reg < reg_count; reg++) {
9711− adev->gfx.ip_dump_compute_queues[index + reg] =
9712− RREG32(SOC15_REG_ENTRY_OFFSET(
9713− gc_cp_reg_list_10[reg]));
9750+ if (i && gc_cp_reg_list_10[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
9751+ adev->gfx.ip_dump_compute_queues[index + reg] =
9752+ RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_MEC_ME2_HEADER_DUMP));
9753+ else
9754+ adev->gfx.ip_dump_compute_queues[index + reg] =
9755+ RREG32(SOC15_REG_ENTRY_OFFSET(
9756+ gc_cp_reg_list_10[reg]));
97149757 }
97159758 index += reg_count;
97169759 }
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c+539−193
@@ -48,6 +48,8 @@ · 47 unmodified lines
4848 #include "gfx_v11_0_3.h"
4949 #include "nbio_v4_3.h"
5050 #include "mes_v11_0.h"
51+#include "mes_userqueue.h"
52+#include "amdgpu_userq_fence.h"
5153
5254 #define GFX11_NUM_GFX_RINGS 1
5355 #define GFX11_MEC_HPD_SIZE 2048
@@ -177,9 +179,13 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_11_0[] = { · 123 unmodified lines
177179 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_INSTR_PNTR),
178180 SOC15_REG_ENTRY_STR(GC, 0, regCP_CPC_STATUS),
179181 /* cp header registers */
180− SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
181− SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
182− SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
182+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
183+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
184+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
185+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
186+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
187+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
188+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
183189 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
184190 /* SE status registers */
185191 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),
@@ -230,7 +236,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_11[] = { · 44 unmodified lines
230236 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET),
231237 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),
232238 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET),
233− SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS)
239+ SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS),
240+ /* cp header registers */
241+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
242+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
243+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
244+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
245+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
246+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
247+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
248+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
234249 };
235250
236251 static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = {
@@ -259,7 +274,24 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = { · 22 unmodified lines
259274 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO),
260275 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI),
261276 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ),
262− SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ)
277+ SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ),
278+ /* cp header registers */
279+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
280+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
281+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
282+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
283+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
284+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
285+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
286+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
287+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
288+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
289+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
290+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
291+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
292+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
293+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
294+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
263295 };
264296
265297 static const struct soc15_reg_golden golden_settings_gc_11_0[] = {
@@ -580,33 +612,18 @@ static int gfx_v11_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 314 unmodified lines
580612
581613 memset(&ib, 0, sizeof(ib));
582614
583− if (ring->is_mes_queue) {
584− uint32_t padding, offset;
585−
586− offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
587− padding = amdgpu_mes_ctx_get_offs(ring,
588− AMDGPU_MES_CTX_PADDING_OFFS);
589−
590− ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
591− ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
592−
593− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding);
594− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding);
595− *cpu_ptr = cpu_to_le32(0xCAFEDEAD);
596− } else {
597− r = amdgpu_device_wb_get(adev, &index);
598− if (r)
599− return r;
615+ r = amdgpu_device_wb_get(adev, &index);
616+ if (r)
617+ return r;
600618
601− gpu_addr = adev->wb.gpu_addr + (index * 4);
602− adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
603− cpu_ptr = &adev->wb.wb[index];
619+ gpu_addr = adev->wb.gpu_addr + (index * 4);
620+ adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
621+ cpu_ptr = &adev->wb.wb[index];
604622
605− r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib);
606− if (r) {
607− DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
608− goto err1;
609− }
623+ r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib);
624+ if (r) {
625+ DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
626+ goto err1;
610627 }
611628
612629 ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);
@@ -633,12 +650,10 @@ static int gfx_v11_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 20 unmodified lines
633650 else
634651 r = -EINVAL;
635652 err2:
636− if (!ring->is_mes_queue)
637− amdgpu_ib_free(&ib, NULL);
653+ amdgpu_ib_free(&ib, NULL);
638654 dma_fence_put(f);
639655 err1:
640− if (!ring->is_mes_queue)
641− amdgpu_device_wb_free(adev, index);
656+ amdgpu_device_wb_free(adev, index);
642657 return r;
643658 }
644659
@@ -833,9 +848,7 @@ static u32 gfx_v11_0_get_csb_size(struct amdgpu_device *adev) · 188 unmodified lines
833848 static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev,
834849 volatile u32 *buffer)
835850 {
836− u32 count = 0, i;
837− const struct cs_section_def *sect = NULL;
838− const struct cs_extent_def *ext = NULL;
851+ u32 count = 0;
839852 int ctx_reg_offset;
840853
841854 if (adev->gfx.rlc.cs_data == NULL)
@@ -843,39 +856,15 @@ static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev, · 1 unmodified line
843856 if (buffer == NULL)
844857 return;
845858
846− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
847− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
848−
849− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
850− buffer[count++] = cpu_to_le32(0x80000000);
851− buffer[count++] = cpu_to_le32(0x80000000);
852−
853− for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
854− for (ext = sect->section; ext->extent != NULL; ++ext) {
855− if (sect->id == SECT_CONTEXT) {
856− buffer[count++] =
857− cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
858− buffer[count++] = cpu_to_le32(ext->reg_index -
859− PACKET3_SET_CONTEXT_REG_START);
860− for (i = 0; i < ext->reg_count; i++)
861− buffer[count++] = cpu_to_le32(ext->extent[i]);
862− } else {
863− return;
864− }
865− }
866− }
859+ count = amdgpu_gfx_csb_preamble_start(buffer);
860+ count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
867861
868− ctx_reg_offset =
869− SOC15_REG_OFFSET(GC, 0, regPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
862+ ctx_reg_offset = SOC15_REG_OFFSET(GC, 0, regPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
870863 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));
871864 buffer[count++] = cpu_to_le32(ctx_reg_offset);
872865 buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override);
873866
874− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
875− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
876−
877− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
878− buffer[count++] = cpu_to_le32(0);
867+ amdgpu_gfx_csb_preamble_end(buffer, count);
879868 }
880869
881870 static void gfx_v11_0_rlc_fini(struct amdgpu_device *adev)
@@ -1056,14 +1045,21 @@ static void gfx_v11_0_select_me_pipe_q(struct amdgpu_device *adev, · 174 unmodified lines
10561045 #define MQD_FWWORKAREA_SIZE 484
10571046 #define MQD_FWWORKAREA_ALIGNMENT 256
10581047
1059−static int gfx_v11_0_get_gfx_shadow_info(struct amdgpu_device *adev,
1048+static void gfx_v11_0_get_gfx_shadow_info_nocheck(struct amdgpu_device *adev,
10601049 struct amdgpu_gfx_shadow_info *shadow_info)
10611050 {
1062− if (adev->gfx.cp_gfx_shadow) {
1063− shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE;
1064− shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT;
1065− shadow_info->csa_size = MQD_FWWORKAREA_SIZE;
1066− shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT;
1051+ shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE;
1052+ shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT;
1053+ shadow_info->csa_size = MQD_FWWORKAREA_SIZE;
1054+ shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT;
1055+}
1056+
1057+static int gfx_v11_0_get_gfx_shadow_info(struct amdgpu_device *adev,
1058+ struct amdgpu_gfx_shadow_info *shadow_info,
1059+ bool skip_check)
1060+{
1061+ if (adev->gfx.cp_gfx_shadow || skip_check) {
1062+ gfx_v11_0_get_gfx_shadow_info_nocheck(adev, shadow_info);
10671063 return 0;
10681064 } else {
10691065 memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info));
@@ -1136,6 +1132,10 @@ static int gfx_v11_0_gfx_ring_init(struct amdgpu_device *adev, int ring_id, · 66 unmodified lines
11361132
11371133 ring->ring_obj = NULL;
11381134 ring->use_doorbell = true;
1135+ if (adev->gfx.disable_kq) {
1136+ ring->no_scheduler = true;
1137+ ring->no_user_submission = true;
1138+ }
11391139
11401140 if (!ring_id)
11411141 ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1;
@@ -1568,24 +1568,18 @@ static void gfx_v11_0_alloc_ip_dump(struct amdgpu_device *adev) · 426 unmodified lines
15681568
15691569 static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block)
15701570 {
1571− int i, j, k, r, ring_id = 0;
1571+ int i, j, k, r, ring_id;
15721572 int xcc_id = 0;
15731573 struct amdgpu_device *adev = ip_block->adev;
1574+ int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
15741575
15751576 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);
15761577
15771578 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
15781579 case IP_VERSION(11, 0, 0):
1580+ case IP_VERSION(11, 0, 1):
15791581 case IP_VERSION(11, 0, 2):
15801582 case IP_VERSION(11, 0, 3):
1581− adev->gfx.me.num_me = 1;
1582− adev->gfx.me.num_pipe_per_me = 1;
1583− adev->gfx.me.num_queue_per_pipe = 1;
1584− adev->gfx.mec.num_mec = 1;
1585− adev->gfx.mec.num_pipe_per_mec = 4;
1586− adev->gfx.mec.num_queue_per_pipe = 4;
1587− break;
1588− case IP_VERSION(11, 0, 1):
15891583 case IP_VERSION(11, 0, 4):
15901584 case IP_VERSION(11, 5, 0):
15911585 case IP_VERSION(11, 5, 1):
@@ -1593,7 +1587,7 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 1 unmodified line
15931587 case IP_VERSION(11, 5, 3):
15941588 adev->gfx.me.num_me = 1;
15951589 adev->gfx.me.num_pipe_per_me = 1;
1596− adev->gfx.me.num_queue_per_pipe = 1;
1590+ adev->gfx.me.num_queue_per_pipe = 2;
15971591 adev->gfx.mec.num_mec = 1;
15981592 adev->gfx.mec.num_pipe_per_mec = 4;
15991593 adev->gfx.mec.num_queue_per_pipe = 4;
@@ -1608,6 +1602,35 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
16081602 break;
16091603 }
16101604
1605+ switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1606+ case IP_VERSION(11, 0, 0):
1607+ case IP_VERSION(11, 0, 2):
1608+ case IP_VERSION(11, 0, 3):
1609+ if (!adev->gfx.disable_uq &&
1610+ adev->gfx.me_fw_version >= 2390 &&
1611+ adev->gfx.pfp_fw_version >= 2530 &&
1612+ adev->gfx.mec_fw_version >= 2600 &&
1613+ adev->mes.fw_version[0] >= 120) {
1614+ adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs;
1615+ adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs;
1616+ }
1617+ break;
1618+ case IP_VERSION(11, 0, 1):
1619+ case IP_VERSION(11, 0, 4):
1620+ case IP_VERSION(11, 5, 0):
1621+ case IP_VERSION(11, 5, 1):
1622+ case IP_VERSION(11, 5, 2):
1623+ case IP_VERSION(11, 5, 3):
1624+ /* add firmware version checks here */
1625+ if (0 && !adev->gfx.disable_uq) {
1626+ adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs;
1627+ adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs;
1628+ }
1629+ break;
1630+ default:
1631+ break;
1632+ }
1633+
16111634 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
16121635 case IP_VERSION(11, 0, 0):
16131636 case IP_VERSION(11, 0, 2):
@@ -1640,6 +1663,34 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 26 unmodified lines
16401663 }
16411664 }
16421665 break;
1666+ case IP_VERSION(11, 5, 2):
1667+ adev->gfx.cleaner_shader_ptr = gfx_11_0_3_cleaner_shader_hex;
1668+ adev->gfx.cleaner_shader_size = sizeof(gfx_11_0_3_cleaner_shader_hex);
1669+ if (adev->gfx.me_fw_version >= 12 &&
1670+ adev->gfx.pfp_fw_version >= 15 &&
1671+ adev->gfx.mec_fw_version >= 15) {
1672+ adev->gfx.enable_cleaner_shader = true;
1673+ r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
1674+ if (r) {
1675+ adev->gfx.enable_cleaner_shader = false;
1676+ dev_err(adev->dev, "Failed to initialize cleaner shader\n");
1677+ }
1678+ }
1679+ break;
1680+ case IP_VERSION(11, 5, 3):
1681+ adev->gfx.cleaner_shader_ptr = gfx_11_0_3_cleaner_shader_hex;
1682+ adev->gfx.cleaner_shader_size = sizeof(gfx_11_0_3_cleaner_shader_hex);
1683+ if (adev->gfx.me_fw_version >= 7 &&
1684+ adev->gfx.pfp_fw_version >= 8 &&
1685+ adev->gfx.mec_fw_version >= 8) {
1686+ adev->gfx.enable_cleaner_shader = true;
1687+ r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
1688+ if (r) {
1689+ adev->gfx.enable_cleaner_shader = false;
1690+ dev_err(adev->dev, "Failed to initialize cleaner shader\n");
1691+ }
1692+ }
1693+ break;
16431694 default:
16441695 adev->gfx.enable_cleaner_shader = false;
16451696 break;
@@ -1701,37 +1752,42 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 55 unmodified lines
17011752 return r;
17021753 }
17031754
1704− /* set up the gfx ring */
1705− for (i = 0; i < adev->gfx.me.num_me; i++) {
1706− for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
1707− for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
1708− if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
1709− continue;
1710−
1711− r = gfx_v11_0_gfx_ring_init(adev, ring_id,
1712− i, k, j);
1713− if (r)
1714− return r;
1715− ring_id++;
1755+ if (adev->gfx.num_gfx_rings) {
1756+ ring_id = 0;
1757+ /* set up the gfx ring */
1758+ for (i = 0; i < adev->gfx.me.num_me; i++) {
1759+ for (j = 0; j < num_queue_per_pipe; j++) {
1760+ for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
1761+ if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
1762+ continue;
1763+
1764+ r = gfx_v11_0_gfx_ring_init(adev, ring_id,
1765+ i, k, j);
1766+ if (r)
1767+ return r;
1768+ ring_id++;
1769+ }
17161770 }
17171771 }
17181772 }
17191773
1720− ring_id = 0;
1721− /* set up the compute queues - allocate horizontally across pipes */
1722− for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1723− for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1724− for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
1725− if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
1726− k, j))
1727− continue;
1774+ if (adev->gfx.num_compute_rings) {
1775+ ring_id = 0;
1776+ /* set up the compute queues - allocate horizontally across pipes */
1777+ for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1778+ for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1779+ for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
1780+ if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
1781+ k, j))
1782+ continue;
17281783
1729− r = gfx_v11_0_compute_ring_init(adev, ring_id,
1730− i, k, j);
1731− if (r)
1732− return r;
1784+ r = gfx_v11_0_compute_ring_init(adev, ring_id,
1785+ i, k, j);
1786+ if (r)
1787+ return r;
17331788
1734− ring_id++;
1789+ ring_id++;
1790+ }
17351791 }
17361792 }
17371793 }
@@ -4061,6 +4117,8 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 2323 unmodified lines
40614117 #ifdef __BIG_ENDIAN
40624118 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1);
40634119 #endif
4120+ if (prop->tmz_queue)
4121+ tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, TMZ_MATCH, 1);
40644122 mqd->cp_gfx_hqd_cntl = tmp;
40654123
40664124 /* set up cp_doorbell_control */
@@ -4081,6 +4139,16 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 14 unmodified lines
40814139 /* active the queue */
40824140 mqd->cp_gfx_hqd_active = 1;
40834141
4142+ /* set gfx UQ items */
4143+ mqd->shadow_base_lo = lower_32_bits(prop->shadow_addr);
4144+ mqd->shadow_base_hi = upper_32_bits(prop->shadow_addr);
4145+ mqd->gds_bkup_base_lo = lower_32_bits(prop->gds_bkup_addr);
4146+ mqd->gds_bkup_base_hi = upper_32_bits(prop->gds_bkup_addr);
4147+ mqd->fw_work_area_base_lo = lower_32_bits(prop->csa_addr);
4148+ mqd->fw_work_area_base_hi = upper_32_bits(prop->csa_addr);
4149+ mqd->fence_address_lo = lower_32_bits(prop->fence_address);
4150+ mqd->fence_address_hi = upper_32_bits(prop->fence_address);
4151+
40844152 return 0;
40854153 }
40864154
@@ -4205,6 +4273,8 @@ static int gfx_v11_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 118 unmodified lines
42054273 prop->allow_tunneling);
42064274 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
42074275 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
4276+ if (prop->tmz_queue)
4277+ tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TMZ, 1);
42084278 mqd->cp_hqd_pq_control = tmp;
42094279
42104280 /* set the wb address whether it's enabled or not */
@@ -4256,6 +4326,10 @@ static int gfx_v11_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 45 unmodified lines
42564326
42574327 mqd->cp_hqd_active = prop->hqd_active;
42584328
4329+ /* set UQ fenceaddress */
4330+ mqd->fence_address_lo = lower_32_bits(prop->fence_address);
4331+ mqd->fence_address_hi = upper_32_bits(prop->fence_address);
4332+
42594333 return 0;
42604334 }
42614335
@@ -4509,11 +4583,23 @@ static int gfx_v11_0_cp_resume(struct amdgpu_device *adev) · 247 unmodified lines
45094583 return r;
45104584 }
45114585
4512− for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4513− ring = &adev->gfx.gfx_ring[i];
4514− r = amdgpu_ring_test_helper(ring);
4515− if (r)
4516− return r;
4586+ if (adev->gfx.disable_kq) {
4587+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4588+ ring = &adev->gfx.gfx_ring[i];
4589+ /* we don't want to set ring->ready */
4590+ r = amdgpu_ring_test_ring(ring);
4591+ if (r)
4592+ return r;
4593+ }
4594+ if (amdgpu_async_gfx_ring)
4595+ amdgpu_gfx_disable_kgq(adev, 0);
4596+ } else {
4597+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4598+ ring = &adev->gfx.gfx_ring[i];
4599+ r = amdgpu_ring_test_helper(ring);
4600+ if (r)
4601+ return r;
4602+ }
45174603 }
45184604
45194605 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
@@ -4722,6 +4808,49 @@ static int gfx_v11_0_hw_init(struct amdgpu_ip_block *ip_block) · 202 unmodified lines
47224808 return r;
47234809 }
47244810
4811+static int gfx_v11_0_set_userq_eop_interrupts(struct amdgpu_device *adev,
4812+ bool enable)
4813+{
4814+ unsigned int irq_type;
4815+ int m, p, r;
4816+
4817+ if (adev->userq_funcs[AMDGPU_HW_IP_GFX]) {
4818+ for (m = 0; m < adev->gfx.me.num_me; m++) {
4819+ for (p = 0; p < adev->gfx.me.num_pipe_per_me; p++) {
4820+ irq_type = AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP + p;
4821+ if (enable)
4822+ r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
4823+ irq_type);
4824+ else
4825+ r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
4826+ irq_type);
4827+ if (r)
4828+ return r;
4829+ }
4830+ }
4831+ }
4832+
4833+ if (adev->userq_funcs[AMDGPU_HW_IP_COMPUTE]) {
4834+ for (m = 0; m < adev->gfx.mec.num_mec; ++m) {
4835+ for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) {
4836+ irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
4837+ + (m * adev->gfx.mec.num_pipe_per_mec)
4838+ + p;
4839+ if (enable)
4840+ r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
4841+ irq_type);
4842+ else
4843+ r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
4844+ irq_type);
4845+ if (r)
4846+ return r;
4847+ }
4848+ }
4849+ }
4850+
4851+ return 0;
4852+}
4853+
47254854 static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block)
47264855 {
47274856 struct amdgpu_device *adev = ip_block->adev;
@@ -4731,9 +4860,11 @@ static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
47314860 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
47324861 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
47334862 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
4863+ gfx_v11_0_set_userq_eop_interrupts(adev, false);
47344864
47354865 if (!adev->no_hw_access) {
4736− if (amdgpu_async_gfx_ring) {
4866+ if (amdgpu_async_gfx_ring &&
4867+ !adev->gfx.disable_kq) {
47374868 if (amdgpu_gfx_disable_kgq(adev, 0))
47384869 DRM_ERROR("KGQ disable failed\n");
47394870 }
@@ -5059,11 +5190,36 @@ static int gfx_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 319 unmodified lines
50595190 {
50605191 struct amdgpu_device *adev = ip_block->adev;
50615192
5193+ switch (amdgpu_user_queue) {
5194+ case -1:
5195+ case 0:
5196+ default:
5197+ adev->gfx.disable_kq = false;
5198+ adev->gfx.disable_uq = true;
5199+ break;
5200+ case 1:
5201+ adev->gfx.disable_kq = false;
5202+ adev->gfx.disable_uq = false;
5203+ break;
5204+ case 2:
5205+ adev->gfx.disable_kq = true;
5206+ adev->gfx.disable_uq = false;
5207+ break;
5208+ }
5209+
50625210 adev->gfx.funcs = &gfx_v11_0_gfx_funcs;
50635211
5064− adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS;
5065− adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
5066− AMDGPU_MAX_COMPUTE_RINGS);
5212+ if (adev->gfx.disable_kq) {
5213+ /* We need one GFX ring temporarily to set up
5214+ * the clear state.
5215+ */
5216+ adev->gfx.num_gfx_rings = 1;
5217+ adev->gfx.num_compute_rings = 0;
5218+ } else {
5219+ adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS;
5220+ adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
5221+ AMDGPU_MAX_COMPUTE_RINGS);
5222+ }
50675223
50685224 gfx_v11_0_set_kiq_pm4_funcs(adev);
50695225 gfx_v11_0_set_ring_funcs(adev);
@@ -5094,6 +5250,11 @@ static int gfx_v11_0_late_init(struct amdgpu_ip_block *ip_block) · 24 unmodified lines
50945250 r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0);
50955251 if (r)
50965252 return r;
5253+
5254+ r = gfx_v11_0_set_userq_eop_interrupts(adev, true);
5255+ if (r)
5256+ return r;
5257+
50975258 return 0;
50985259 }
50995260
@@ -5691,10 +5852,6 @@ static void gfx_v11_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, · 591 unmodified lines
56915852 (!amdgpu_sriov_vf(ring->adev) && flags & AMDGPU_IB_PREEMPTED) ? true : false);
56925853 }
56935854
5694− if (ring->is_mes_queue)
5695− /* inherit vmid from mqd */
5696− control |= 0x400000;
5697−
56985855 amdgpu_ring_write(ring, header);
56995856 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
57005857 amdgpu_ring_write(ring,
@@ -5714,10 +5871,6 @@ static void gfx_v11_0_ring_emit_ib_compute(struct amdgpu_ring *ring, · 13 unmodified lines
57145871 unsigned vmid = AMDGPU_JOB_GET_VMID(job);
57155872 u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);
57165873
5717− if (ring->is_mes_queue)
5718− /* inherit vmid from mqd */
5719− control |= 0x40000000;
5720−
57215874 /* Currently, there is a high possibility to get wave ID mismatch
57225875 * between ME and GDS, leading to a hw deadlock, because ME generates
57235876 * different wave IDs than the GDS expects. This situation happens
@@ -5775,8 +5928,7 @@ static void gfx_v11_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, · 51 unmodified lines
57755928 amdgpu_ring_write(ring, upper_32_bits(addr));
57765929 amdgpu_ring_write(ring, lower_32_bits(seq));
57775930 amdgpu_ring_write(ring, upper_32_bits(seq));
5778− amdgpu_ring_write(ring, ring->is_mes_queue ?
5779− (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0);
5931+ amdgpu_ring_write(ring, 0);
57805932 }
57815933
57825934 static void gfx_v11_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
@@ -5804,10 +5956,7 @@ static void gfx_v11_0_ring_invalidate_tlbs(struct amdgpu_ring *ring, · 21 unmodified lines
58045956 static void gfx_v11_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
58055957 unsigned vmid, uint64_t pd_addr)
58065958 {
5807− if (ring->is_mes_queue)
5808− gfx_v11_0_ring_invalidate_tlbs(ring, 0, 0, false, 0);
5809− else
5810− amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
5959+ amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
58115960
58125961 /* compute doesn't have PFP */
58135962 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {
@@ -6036,28 +6185,13 @@ static void gfx_v11_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume) · 222 unmodified lines
60366185 void *de_payload_cpu_addr;
60376186 int cnt;
60386187
6039− if (ring->is_mes_queue) {
6040− offset = offsetof(struct amdgpu_mes_ctx_meta_data,
6041− gfx[0].gfx_meta_data) +
6042− offsetof(struct v10_gfx_meta_data, de_payload);
6043− de_payload_gpu_addr =
6044− amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
6045− de_payload_cpu_addr =
6046− amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
6047−
6048− offset = offsetof(struct amdgpu_mes_ctx_meta_data,
6049− gfx[0].gds_backup) +
6050− offsetof(struct v10_gfx_meta_data, de_payload);
6051− gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
6052− } else {
6053− offset = offsetof(struct v10_gfx_meta_data, de_payload);
6054− de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
6055− de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
6188+ offset = offsetof(struct v10_gfx_meta_data, de_payload);
6189+ de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
6190+ de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
60566191
6057− gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
6058− AMDGPU_CSA_SIZE - adev->gds.gds_size,
6059− PAGE_SIZE);
6060− }
6192+ gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
6193+ AMDGPU_CSA_SIZE - adev->gds.gds_size,
6194+ PAGE_SIZE);
60616195
60626196 de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);
60636197 de_payload.gds_backup_addrhi = upper_32_bits(gds_addr);
@@ -6296,25 +6430,23 @@ static int gfx_v11_0_eop_irq(struct amdgpu_device *adev, · 232 unmodified lines
62966430 struct amdgpu_irq_src *source,
62976431 struct amdgpu_iv_entry *entry)
62986432 {
6299− int i;
6433+ u32 doorbell_offset = entry->src_data[0];
63006434 u8 me_id, pipe_id, queue_id;
63016435 struct amdgpu_ring *ring;
6302− uint32_t mes_queue_id = entry->src_data[0];
6436+ int i;
63036437
63046438 DRM_DEBUG("IH: CP EOP\n");
63056439
6306− if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {
6307− struct amdgpu_mes_queue *queue;
6308−
6309− mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;
6440+ if (adev->enable_mes && doorbell_offset) {
6441+ struct amdgpu_userq_fence_driver *fence_drv = NULL;
6442+ struct xarray *xa = &adev->userq_xa;
6443+ unsigned long flags;
63106444
6311− spin_lock(&adev->mes.queue_id_lock);
6312− queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);
6313− if (queue) {
6314− DRM_DEBUG("process mes queue id = %d\n", mes_queue_id);
6315− amdgpu_fence_process(queue->ring);
6316− }
6317− spin_unlock(&adev->mes.queue_id_lock);
6445+ xa_lock_irqsave(xa, flags);
6446+ fence_drv = xa_load(xa, doorbell_offset);
6447+ if (fence_drv)
6448+ amdgpu_userq_fence_driver_process(fence_drv);
6449+ xa_unlock_irqrestore(xa, flags);
63186450 } else {
63196451 me_id = (entry->ring_id & 0x0c) >> 2;
63206452 pipe_id = (entry->ring_id & 0x03) >> 0;
@@ -6481,27 +6613,29 @@ static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev, · 160 unmodified lines
64816613 pipe_id = (entry->ring_id & 0x03) >> 0;
64826614 queue_id = (entry->ring_id & 0x70) >> 4;
64836615
6484− switch (me_id) {
6485− case 0:
6486− for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
6487− ring = &adev->gfx.gfx_ring[i];
6488− if (ring->me == me_id && ring->pipe == pipe_id &&
6489− ring->queue == queue_id)
6490− drm_sched_fault(&ring->sched);
6491− }
6492− break;
6493− case 1:
6494− case 2:
6495− for (i = 0; i < adev->gfx.num_compute_rings; i++) {
6496− ring = &adev->gfx.compute_ring[i];
6497− if (ring->me == me_id && ring->pipe == pipe_id &&
6498− ring->queue == queue_id)
6499− drm_sched_fault(&ring->sched);
6616+ if (!adev->gfx.disable_kq) {
6617+ switch (me_id) {
6618+ case 0:
6619+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
6620+ ring = &adev->gfx.gfx_ring[i];
6621+ if (ring->me == me_id && ring->pipe == pipe_id &&
6622+ ring->queue == queue_id)
6623+ drm_sched_fault(&ring->sched);
6624+ }
6625+ break;
6626+ case 1:
6627+ case 2:
6628+ for (i = 0; i < adev->gfx.num_compute_rings; i++) {
6629+ ring = &adev->gfx.compute_ring[i];
6630+ if (ring->me == me_id && ring->pipe == pipe_id &&
6631+ ring->queue == queue_id)
6632+ drm_sched_fault(&ring->sched);
6633+ }
6634+ break;
6635+ default:
6636+ BUG();
6637+ break;
65006638 }
6501− break;
6502− default:
6503− BUG();
6504− break;
65056639 }
65066640 }
65076641
@@ -6609,6 +6743,69 @@ static void gfx_v11_0_emit_mem_sync(struct amdgpu_ring *ring) · 101 unmodified lines
66096743 amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */
66106744 }
66116745
6746+static bool gfx_v11_pipe_reset_support(struct amdgpu_device *adev)
6747+{
6748+ /* Disable the pipe reset until the CPFW fully support it.*/
6749+ dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
6750+ return false;
6751+}
6752+
6753+
6754+static int gfx_v11_reset_gfx_pipe(struct amdgpu_ring *ring)
6755+{
6756+ struct amdgpu_device *adev = ring->adev;
6757+ uint32_t reset_pipe = 0, clean_pipe = 0;
6758+ int r;
6759+
6760+ if (!gfx_v11_pipe_reset_support(adev))
6761+ return -EOPNOTSUPP;
6762+
6763+ gfx_v11_0_set_safe_mode(adev, 0);
6764+ mutex_lock(&adev->srbm_mutex);
6765+ soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
6766+
6767+ switch (ring->pipe) {
6768+ case 0:
6769+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6770+ PFP_PIPE0_RESET, 1);
6771+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6772+ ME_PIPE0_RESET, 1);
6773+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6774+ PFP_PIPE0_RESET, 0);
6775+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6776+ ME_PIPE0_RESET, 0);
6777+ break;
6778+ case 1:
6779+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6780+ PFP_PIPE1_RESET, 1);
6781+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6782+ ME_PIPE1_RESET, 1);
6783+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6784+ PFP_PIPE1_RESET, 0);
6785+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6786+ ME_PIPE1_RESET, 0);
6787+ break;
6788+ default:
6789+ break;
6790+ }
6791+
6792+ WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
6793+ WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
6794+
6795+ r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
6796+ RS64_FW_UC_START_ADDR_LO;
6797+ soc21_grbm_select(adev, 0, 0, 0, 0);
6798+ mutex_unlock(&adev->srbm_mutex);
6799+ gfx_v11_0_unset_safe_mode(adev, 0);
6800+
6801+ dev_info(adev->dev, "The ring %s pipe reset to the ME firmware start PC: %s\n", ring->name,
6802+ r == 0 ? "successfully" : "failed");
6803+ /* FIXME: Sometimes driver can't cache the ME firmware start PC correctly,
6804+ * so the pipe reset status relies on the later gfx ring test result.
6805+ */
6806+ return 0;
6807+}
6808+
66126809 static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
66136810 {
66146811 struct amdgpu_device *adev = ring->adev;
@@ -6618,8 +6815,13 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 3 unmodified lines
66186815 return -EINVAL;
66196816
66206817 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false);
6621− if (r)
6622− return r;
6818+ if (r) {
6819+
6820+ dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r);
6821+ r = gfx_v11_reset_gfx_pipe(ring);
6822+ if (r)
6823+ return r;
6824+ }
66236825
66246826 r = gfx_v11_0_kgq_init_queue(ring, true);
66256827 if (r) {
@@ -6636,6 +6838,136 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 10 unmodified lines
66366838 return amdgpu_ring_test_ring(ring);
66376839 }
66386840
6841+static int gfx_v11_0_reset_compute_pipe(struct amdgpu_ring *ring)
6842+{
6843+
6844+ struct amdgpu_device *adev = ring->adev;
6845+ uint32_t reset_pipe = 0, clean_pipe = 0;
6846+ int r;
6847+
6848+ if (!gfx_v11_pipe_reset_support(adev))
6849+ return -EOPNOTSUPP;
6850+
6851+ gfx_v11_0_set_safe_mode(adev, 0);
6852+ mutex_lock(&adev->srbm_mutex);
6853+ soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
6854+
6855+ reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
6856+ clean_pipe = reset_pipe;
6857+
6858+ if (adev->gfx.rs64_enable) {
6859+
6860+ switch (ring->pipe) {
6861+ case 0:
6862+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6863+ MEC_PIPE0_RESET, 1);
6864+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6865+ MEC_PIPE0_RESET, 0);
6866+ break;
6867+ case 1:
6868+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6869+ MEC_PIPE1_RESET, 1);
6870+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6871+ MEC_PIPE1_RESET, 0);
6872+ break;
6873+ case 2:
6874+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6875+ MEC_PIPE2_RESET, 1);
6876+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6877+ MEC_PIPE2_RESET, 0);
6878+ break;
6879+ case 3:
6880+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6881+ MEC_PIPE3_RESET, 1);
6882+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6883+ MEC_PIPE3_RESET, 0);
6884+ break;
6885+ default:
6886+ break;
6887+ }
6888+ WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe);
6889+ WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe);
6890+ r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
6891+ RS64_FW_UC_START_ADDR_LO;
6892+ } else {
6893+ if (ring->me == 1) {
6894+ switch (ring->pipe) {
6895+ case 0:
6896+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6897+ MEC_ME1_PIPE0_RESET, 1);
6898+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6899+ MEC_ME1_PIPE0_RESET, 0);
6900+ break;
6901+ case 1:
6902+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6903+ MEC_ME1_PIPE1_RESET, 1);
6904+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6905+ MEC_ME1_PIPE1_RESET, 0);
6906+ break;
6907+ case 2:
6908+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6909+ MEC_ME1_PIPE2_RESET, 1);
6910+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6911+ MEC_ME1_PIPE2_RESET, 0);
6912+ break;
6913+ case 3:
6914+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6915+ MEC_ME1_PIPE3_RESET, 1);
6916+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6917+ MEC_ME1_PIPE3_RESET, 0);
6918+ break;
6919+ default:
6920+ break;
6921+ }
6922+ /* mec1 fw pc: CP_MEC1_INSTR_PNTR */
6923+ } else {
6924+ switch (ring->pipe) {
6925+ case 0:
6926+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6927+ MEC_ME2_PIPE0_RESET, 1);
6928+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6929+ MEC_ME2_PIPE0_RESET, 0);
6930+ break;
6931+ case 1:
6932+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6933+ MEC_ME2_PIPE1_RESET, 1);
6934+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6935+ MEC_ME2_PIPE1_RESET, 0);
6936+ break;
6937+ case 2:
6938+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6939+ MEC_ME2_PIPE2_RESET, 1);
6940+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6941+ MEC_ME2_PIPE2_RESET, 0);
6942+ break;
6943+ case 3:
6944+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6945+ MEC_ME2_PIPE3_RESET, 1);
6946+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6947+ MEC_ME2_PIPE3_RESET, 0);
6948+ break;
6949+ default:
6950+ break;
6951+ }
6952+ /* mec2 fw pc: CP:CP_MEC2_INSTR_PNTR */
6953+ }
6954+ WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe);
6955+ WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe);
6956+ r = RREG32(SOC15_REG_OFFSET(GC, 0, regCP_MEC1_INSTR_PNTR));
6957+ }
6958+
6959+ soc21_grbm_select(adev, 0, 0, 0, 0);
6960+ mutex_unlock(&adev->srbm_mutex);
6961+ gfx_v11_0_unset_safe_mode(adev, 0);
6962+
6963+ dev_info(adev->dev, "The ring %s pipe resets to MEC FW start PC: %s\n", ring->name,
6964+ r == 0 ? "successfully" : "failed");
6965+ /*FIXME:Sometimes driver can't cache the MEC firmware start PC correctly, so the pipe
6966+ * reset status relies on the compute ring test result.
6967+ */
6968+ return 0;
6969+}
6970+
66396971 static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)
66406972 {
66416973 struct amdgpu_device *adev = ring->adev;
@@ -6646,8 +6978,10 @@ static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines
66466978
66476979 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true);
66486980 if (r) {
6649− dev_err(adev->dev, "reset via MMIO failed %d\n", r);
6650− return r;
6981+ dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r);
6982+ r = gfx_v11_0_reset_compute_pipe(ring);
6983+ if (r)
6984+ return r;
66516985 }
66526986
66536987 r = gfx_v11_0_kcq_init_queue(ring, true);
@@ -6693,9 +7027,14 @@ static void gfx_v11_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printe · 39 unmodified lines
66937027 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {
66947028 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);
66957029 for (reg = 0; reg < reg_count; reg++) {
6696− drm_printf(p, "%-50s \t 0x%08x\n",
6697− gc_cp_reg_list_11[reg].reg_name,
6698− adev->gfx.ip_dump_compute_queues[index + reg]);
7030+ if (i && gc_cp_reg_list_11[reg].reg_offset == regCP_MEC_ME1_HEADER_DUMP)
7031+ drm_printf(p, "%-50s \t 0x%08x\n",
7032+ "regCP_MEC_ME2_HEADER_DUMP",
7033+ adev->gfx.ip_dump_compute_queues[index + reg]);
7034+ else
7035+ drm_printf(p, "%-50s \t 0x%08x\n",
7036+ gc_cp_reg_list_11[reg].reg_name,
7037+ adev->gfx.ip_dump_compute_queues[index + reg]);
66997038 }
67007039 index += reg_count;
67017040 }
@@ -6755,9 +7094,16 @@ static void gfx_v11_ip_dump(struct amdgpu_ip_block *ip_block) · 53 unmodified lines
67557094 /* ME0 is for GFX so start from 1 for CP */
67567095 soc21_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0);
67577096 for (reg = 0; reg < reg_count; reg++) {
6758− adev->gfx.ip_dump_compute_queues[index + reg] =
6759− RREG32(SOC15_REG_ENTRY_OFFSET(
6760− gc_cp_reg_list_11[reg]));
7097+ if (i &&
7098+ gc_cp_reg_list_11[reg].reg_offset ==
7099+ regCP_MEC_ME1_HEADER_DUMP)
7100+ adev->gfx.ip_dump_compute_queues[index + reg] =
7101+ RREG32(SOC15_REG_OFFSET(GC, 0,
7102+ regCP_MEC_ME2_HEADER_DUMP));
7103+ else
7104+ adev->gfx.ip_dump_compute_queues[index + reg] =
7105+ RREG32(SOC15_REG_ENTRY_OFFSET(
7106+ gc_cp_reg_list_11[reg]));
67617107 }
67627108 index += reg_count;
67637109 }
drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c+425−191
@@ -44,6 +44,8 @@ · 43 unmodified lines
4444 #include "gfx_v12_0.h"
4545 #include "nbif_v6_3_1.h"
4646 #include "mes_v12_0.h"
47+#include "mes_userqueue.h"
48+#include "amdgpu_userq_fence.h"
4749
4850 #define GFX12_NUM_GFX_RINGS 1
4951 #define GFX12_MEC_HPD_SIZE 2048
@@ -133,11 +135,14 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_12_0[] = { · 83 unmodified lines
133135 SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR0),
134136 SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR1),
135137 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_RS64_INSTR_PNTR),
136−
137138 /* cp header registers */
138− SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
139− SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
140− SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
139+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
140+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
141+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
142+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
143+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
144+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
145+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
141146 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
142147 /* SE status registers */
143148 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),
@@ -186,7 +191,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_12[] = { · 42 unmodified lines
186191 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET),
187192 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),
188193 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET),
189− SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS)
194+ SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS),
195+ /* cp header registers */
196+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
197+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
198+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
199+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
200+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
201+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
202+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
203+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
190204 };
191205
192206 static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = {
@@ -215,7 +229,24 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = { · 22 unmodified lines
215229 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO),
216230 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI),
217231 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ),
218− SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ)
232+ SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ),
233+ /* cp header registers */
234+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
235+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
236+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
237+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
238+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
239+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
240+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
241+ SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
242+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
243+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
244+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
245+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
246+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
247+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
248+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
249+ SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
219250 };
220251
221252 static const struct soc15_reg_golden golden_settings_gc_12_0_rev0[] = {
@@ -475,33 +506,18 @@ static int gfx_v12_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 253 unmodified lines
475506
476507 memset(&ib, 0, sizeof(ib));
477508
478− if (ring->is_mes_queue) {
479− uint32_t padding, offset;
480−
481− offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
482− padding = amdgpu_mes_ctx_get_offs(ring,
483− AMDGPU_MES_CTX_PADDING_OFFS);
484−
485− ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
486− ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
487−
488− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding);
489− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding);
490− *cpu_ptr = cpu_to_le32(0xCAFEDEAD);
491− } else {
492− r = amdgpu_device_wb_get(adev, &index);
493− if (r)
494− return r;
509+ r = amdgpu_device_wb_get(adev, &index);
510+ if (r)
511+ return r;
495512
496− gpu_addr = adev->wb.gpu_addr + (index * 4);
497− adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
498− cpu_ptr = &adev->wb.wb[index];
513+ gpu_addr = adev->wb.gpu_addr + (index * 4);
514+ adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
515+ cpu_ptr = &adev->wb.wb[index];
499516
500− r = amdgpu_ib_get(adev, NULL, 16, AMDGPU_IB_POOL_DIRECT, &ib);
501− if (r) {
502− dev_err(adev->dev, "amdgpu: failed to get ib (%ld).\n", r);
503− goto err1;
504− }
517+ r = amdgpu_ib_get(adev, NULL, 16, AMDGPU_IB_POOL_DIRECT, &ib);
518+ if (r) {
519+ dev_err(adev->dev, "amdgpu: failed to get ib (%ld).\n", r);
520+ goto err1;
505521 }
506522
507523 ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);
@@ -528,12 +544,10 @@ static int gfx_v12_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 20 unmodified lines
528544 else
529545 r = -EINVAL;
530546 err2:
531− if (!ring->is_mes_queue)
532− amdgpu_ib_free(&ib, NULL);
547+ amdgpu_ib_free(&ib, NULL);
533548 dma_fence_put(f);
534549 err1:
535− if (!ring->is_mes_queue)
536− amdgpu_device_wb_free(adev, index);
550+ amdgpu_device_wb_free(adev, index);
537551 return r;
538552 }
539553
@@ -881,6 +895,34 @@ static void gfx_v12_0_select_me_pipe_q(struct amdgpu_device *adev, · 341 unmodified lines
881895 soc24_grbm_select(adev, me, pipe, q, vm);
882896 }
883897
898+/* all sizes are in bytes */
899+#define MQD_SHADOW_BASE_SIZE 73728
900+#define MQD_SHADOW_BASE_ALIGNMENT 256
901+#define MQD_FWWORKAREA_SIZE 484
902+#define MQD_FWWORKAREA_ALIGNMENT 256
903+
904+static void gfx_v12_0_get_gfx_shadow_info_nocheck(struct amdgpu_device *adev,
905+ struct amdgpu_gfx_shadow_info *shadow_info)
906+{
907+ shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE;
908+ shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT;
909+ shadow_info->csa_size = MQD_FWWORKAREA_SIZE;
910+ shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT;
911+}
912+
913+static int gfx_v12_0_get_gfx_shadow_info(struct amdgpu_device *adev,
914+ struct amdgpu_gfx_shadow_info *shadow_info,
915+ bool skip_check)
916+{
917+ if (adev->gfx.cp_gfx_shadow || skip_check) {
918+ gfx_v12_0_get_gfx_shadow_info_nocheck(adev, shadow_info);
919+ return 0;
920+ }
921+
922+ memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info));
923+ return -EINVAL;
924+}
925+
884926 static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = {
885927 .get_gpu_clock_counter = &gfx_v12_0_get_gpu_clock_counter,
886928 .select_se_sh = &gfx_v12_0_select_se_sh,
@@ -889,6 +931,7 @@ static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = { · 2 unmodified lines
889931 .read_wave_vgprs = &gfx_v12_0_read_wave_vgprs,
890932 .select_me_pipe_q = &gfx_v12_0_select_me_pipe_q,
891933 .update_perfmon_mgcg = &gfx_v12_0_update_perf_clk,
934+ .get_gfx_shadow_info = &gfx_v12_0_get_gfx_shadow_info,
892935 };
893936
894937 static int gfx_v12_0_gpu_early_init(struct amdgpu_device *adev)
@@ -1346,6 +1389,7 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 451 unmodified lines
13461389 unsigned num_compute_rings;
13471390 int xcc_id = 0;
13481391 struct amdgpu_device *adev = ip_block->adev;
1392+ int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
13491393
13501394 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);
13511395
@@ -1354,7 +1398,7 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 2 unmodified lines
13541398 case IP_VERSION(12, 0, 1):
13551399 adev->gfx.me.num_me = 1;
13561400 adev->gfx.me.num_pipe_per_me = 1;
1357− adev->gfx.me.num_queue_per_pipe = 1;
1401+ adev->gfx.me.num_queue_per_pipe = 8;
13581402 adev->gfx.mec.num_mec = 1;
13591403 adev->gfx.mec.num_pipe_per_mec = 2;
13601404 adev->gfx.mec.num_queue_per_pipe = 4;
@@ -1369,6 +1413,22 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
13691413 break;
13701414 }
13711415
1416+ switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1417+ case IP_VERSION(12, 0, 0):
1418+ case IP_VERSION(12, 0, 1):
1419+ if (!adev->gfx.disable_uq &&
1420+ adev->gfx.me_fw_version >= 2780 &&
1421+ adev->gfx.pfp_fw_version >= 2840 &&
1422+ adev->gfx.mec_fw_version >= 3050 &&
1423+ adev->mes.fw_version[0] >= 123) {
1424+ adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs;
1425+ adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs;
1426+ }
1427+ break;
1428+ default:
1429+ break;
1430+ }
1431+
13721432 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
13731433 case IP_VERSION(12, 0, 0):
13741434 case IP_VERSION(12, 0, 1):
@@ -1383,11 +1443,13 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
13831443 break;
13841444 }
13851445
1386− /* recalculate compute rings to use based on hardware configuration */
1387− num_compute_rings = (adev->gfx.mec.num_pipe_per_mec *
1388− adev->gfx.mec.num_queue_per_pipe) / 2;
1389− adev->gfx.num_compute_rings = min(adev->gfx.num_compute_rings,
1390− num_compute_rings);
1446+ if (adev->gfx.num_compute_rings) {
1447+ /* recalculate compute rings to use based on hardware configuration */
1448+ num_compute_rings = (adev->gfx.mec.num_pipe_per_mec *
1449+ adev->gfx.mec.num_queue_per_pipe) / 2;
1450+ adev->gfx.num_compute_rings = min(adev->gfx.num_compute_rings,
1451+ num_compute_rings);
1452+ }
13911453
13921454 /* EOP Event */
13931455 r = amdgpu_irq_add_id(adev, SOC21_IH_CLIENTID_GRBM_CP,
@@ -1433,37 +1495,41 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 39 unmodified lines
14331495 return r;
14341496 }
14351497
1436− /* set up the gfx ring */
1437− for (i = 0; i < adev->gfx.me.num_me; i++) {
1438− for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {
1439− for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
1440− if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
1441− continue;
1442−
1443− r = gfx_v12_0_gfx_ring_init(adev, ring_id,
1444− i, k, j);
1445− if (r)
1446− return r;
1447− ring_id++;
1498+ if (adev->gfx.num_gfx_rings) {
1499+ /* set up the gfx ring */
1500+ for (i = 0; i < adev->gfx.me.num_me; i++) {
1501+ for (j = 0; j < num_queue_per_pipe; j++) {
1502+ for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
1503+ if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
1504+ continue;
1505+
1506+ r = gfx_v12_0_gfx_ring_init(adev, ring_id,
1507+ i, k, j);
1508+ if (r)
1509+ return r;
1510+ ring_id++;
1511+ }
14481512 }
14491513 }
14501514 }
14511515
1452− ring_id = 0;
1453− /* set up the compute queues - allocate horizontally across pipes */
1454− for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1455− for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1456− for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
1457− if (!amdgpu_gfx_is_mec_queue_enabled(adev,
1458− 0, i, k, j))
1459− continue;
1516+ if (adev->gfx.num_compute_rings) {
1517+ ring_id = 0;
1518+ /* set up the compute queues - allocate horizontally across pipes */
1519+ for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1520+ for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1521+ for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
1522+ if (!amdgpu_gfx_is_mec_queue_enabled(adev,
1523+ 0, i, k, j))
1524+ continue;
14601525
1461− r = gfx_v12_0_compute_ring_init(adev, ring_id,
1462− i, k, j);
1463− if (r)
1464− return r;
1526+ r = gfx_v12_0_compute_ring_init(adev, ring_id,
1527+ i, k, j);
1528+ if (r)
1529+ return r;
14651530
1466− ring_id++;
1531+ ring_id++;
1532+ }
14671533 }
14681534 }
14691535 }
@@ -2948,6 +3014,8 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 1478 unmodified lines
29483014 #ifdef __BIG_ENDIAN
29493015 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1);
29503016 #endif
3017+ if (prop->tmz_queue)
3018+ tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, TMZ_MATCH, 1);
29513019 mqd->cp_gfx_hqd_cntl = tmp;
29523020
29533021 /* set up cp_doorbell_control */
@@ -2968,6 +3036,14 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 14 unmodified lines
29683036 /* active the queue */
29693037 mqd->cp_gfx_hqd_active = 1;
29703038
3039+ /* set gfx UQ items */
3040+ mqd->shadow_base_lo = lower_32_bits(prop->shadow_addr);
3041+ mqd->shadow_base_hi = upper_32_bits(prop->shadow_addr);
3042+ mqd->fw_work_area_base_lo = lower_32_bits(prop->csa_addr);
3043+ mqd->fw_work_area_base_hi = upper_32_bits(prop->csa_addr);
3044+ mqd->fence_address_lo = lower_32_bits(prop->fence_address);
3045+ mqd->fence_address_hi = upper_32_bits(prop->fence_address);
3046+
29713047 return 0;
29723048 }
29733049
@@ -3091,6 +3167,8 @@ static int gfx_v12_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 117 unmodified lines
30913167 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TUNNEL_DISPATCH, 0);
30923168 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
30933169 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
3170+ if (prop->tmz_queue)
3171+ tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TMZ, 1);
30943172 mqd->cp_hqd_pq_control = tmp;
30953173
30963174 /* set the wb address whether it's enabled or not */
@@ -3142,6 +3220,10 @@ static int gfx_v12_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 45 unmodified lines
31423220
31433221 mqd->cp_hqd_active = prop->hqd_active;
31443222
3223+ /* set UQ fenceaddress */
3224+ mqd->fence_address_lo = lower_32_bits(prop->fence_address);
3225+ mqd->fence_address_hi = upper_32_bits(prop->fence_address);
3226+
31453227 return 0;
31463228 }
31473229
@@ -3600,6 +3682,49 @@ static int gfx_v12_0_hw_init(struct amdgpu_ip_block *ip_block) · 452 unmodified lines
36003682 return r;
36013683 }
36023684
3685+static int gfx_v12_0_set_userq_eop_interrupts(struct amdgpu_device *adev,
3686+ bool enable)
3687+{
3688+ unsigned int irq_type;
3689+ int m, p, r;
3690+
3691+ if (adev->userq_funcs[AMDGPU_HW_IP_GFX]) {
3692+ for (m = 0; m < adev->gfx.me.num_me; m++) {
3693+ for (p = 0; p < adev->gfx.me.num_pipe_per_me; p++) {
3694+ irq_type = AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP + p;
3695+ if (enable)
3696+ r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
3697+ irq_type);
3698+ else
3699+ r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
3700+ irq_type);
3701+ if (r)
3702+ return r;
3703+ }
3704+ }
3705+ }
3706+
3707+ if (adev->userq_funcs[AMDGPU_HW_IP_COMPUTE]) {
3708+ for (m = 0; m < adev->gfx.mec.num_mec; ++m) {
3709+ for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) {
3710+ irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
3711+ + (m * adev->gfx.mec.num_pipe_per_mec)
3712+ + p;
3713+ if (enable)
3714+ r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
3715+ irq_type);
3716+ else
3717+ r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
3718+ irq_type);
3719+ if (r)
3720+ return r;
3721+ }
3722+ }
3723+ }
3724+
3725+ return 0;
3726+}
3727+
36033728 static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block)
36043729 {
36053730 struct amdgpu_device *adev = ip_block->adev;
@@ -3610,6 +3735,7 @@ static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block) · 4 unmodified lines
36103735 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
36113736 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
36123737 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
3738+ gfx_v12_0_set_userq_eop_interrupts(adev, false);
36133739
36143740 if (!adev->no_hw_access) {
36153741 if (amdgpu_async_gfx_ring) {
@@ -3698,11 +3824,33 @@ static int gfx_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 82 unmodified lines
36983824 {
36993825 struct amdgpu_device *adev = ip_block->adev;
37003826
3827+ switch (amdgpu_user_queue) {
3828+ case -1:
3829+ case 0:
3830+ default:
3831+ adev->gfx.disable_kq = false;
3832+ adev->gfx.disable_uq = true;
3833+ break;
3834+ case 1:
3835+ adev->gfx.disable_kq = false;
3836+ adev->gfx.disable_uq = false;
3837+ break;
3838+ case 2:
3839+ adev->gfx.disable_kq = true;
3840+ adev->gfx.disable_uq = false;
3841+ break;
3842+ }
3843+
37013844 adev->gfx.funcs = &gfx_v12_0_gfx_funcs;
37023845
3703− adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS;
3704− adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
3705− AMDGPU_MAX_COMPUTE_RINGS);
3846+ if (adev->gfx.disable_kq) {
3847+ adev->gfx.num_gfx_rings = 0;
3848+ adev->gfx.num_compute_rings = 0;
3849+ } else {
3850+ adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS;
3851+ adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
3852+ AMDGPU_MAX_COMPUTE_RINGS);
3853+ }
37063854
37073855 gfx_v12_0_set_kiq_pm4_funcs(adev);
37083856 gfx_v12_0_set_ring_funcs(adev);
@@ -3733,6 +3881,10 @@ static int gfx_v12_0_late_init(struct amdgpu_ip_block *ip_block) · 24 unmodified lines
37333881 if (r)
37343882 return r;
37353883
3884+ r = gfx_v12_0_set_userq_eop_interrupts(adev, true);
3885+ if (r)
3886+ return r;
3887+
37363888 return 0;
37373889 }
37383890
@@ -4172,45 +4324,17 @@ static u64 gfx_v12_0_ring_get_wptr_gfx(struct amdgpu_ring *ring) · 433 unmodified lines
41724324 static void gfx_v12_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
41734325 {
41744326 struct amdgpu_device *adev = ring->adev;
4175− uint32_t *wptr_saved;
4176− uint32_t *is_queue_unmap;
4177− uint64_t aggregated_db_index;
4178− uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_GFX].mqd_size;
4179− uint64_t wptr_tmp;
4180−
4181− if (ring->is_mes_queue) {
4182− wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);
4183− is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +
4184− sizeof(uint32_t));
4185− aggregated_db_index =
4186− amdgpu_mes_get_aggregated_doorbell_index(adev,
4187− ring->hw_prio);
4188−
4189− wptr_tmp = ring->wptr & ring->buf_mask;
4190− atomic64_set((atomic64_t *)ring->wptr_cpu_addr, wptr_tmp);
4191− *wptr_saved = wptr_tmp;
4192− /* assume doorbell always being used by mes mapped queue */
4193− if (*is_queue_unmap) {
4194− WDOORBELL64(aggregated_db_index, wptr_tmp);
4195− WDOORBELL64(ring->doorbell_index, wptr_tmp);
4196− } else {
4197− WDOORBELL64(ring->doorbell_index, wptr_tmp);
41984327
4199− if (*is_queue_unmap)
4200− WDOORBELL64(aggregated_db_index, wptr_tmp);
4201− }
4328+ if (ring->use_doorbell) {
4329+ /* XXX check if swapping is necessary on BE */
4330+ atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4331+ ring->wptr);
4332+ WDOORBELL64(ring->doorbell_index, ring->wptr);
42024333 } else {
4203− if (ring->use_doorbell) {
4204− /* XXX check if swapping is necessary on BE */
4205− atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4206− ring->wptr);
4207− WDOORBELL64(ring->doorbell_index, ring->wptr);
4208− } else {
4209− WREG32_SOC15(GC, 0, regCP_RB0_WPTR,
4210− lower_32_bits(ring->wptr));
4211− WREG32_SOC15(GC, 0, regCP_RB0_WPTR_HI,
4212− upper_32_bits(ring->wptr));
4213− }
4334+ WREG32_SOC15(GC, 0, regCP_RB0_WPTR,
4335+ lower_32_bits(ring->wptr));
4336+ WREG32_SOC15(GC, 0, regCP_RB0_WPTR_HI,
4337+ upper_32_bits(ring->wptr));
42144338 }
42154339 }
42164340
@@ -4235,42 +4359,14 @@ static u64 gfx_v12_0_ring_get_wptr_compute(struct amdgpu_ring *ring) · 18 unmodified lines
42354359 static void gfx_v12_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
42364360 {
42374361 struct amdgpu_device *adev = ring->adev;
4238− uint32_t *wptr_saved;
4239− uint32_t *is_queue_unmap;
4240− uint64_t aggregated_db_index;
4241− uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_COMPUTE].mqd_size;
4242− uint64_t wptr_tmp;
4243−
4244− if (ring->is_mes_queue) {
4245− wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);
4246− is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +
4247− sizeof(uint32_t));
4248− aggregated_db_index =
4249− amdgpu_mes_get_aggregated_doorbell_index(adev,
4250− ring->hw_prio);
4251−
4252− wptr_tmp = ring->wptr & ring->buf_mask;
4253− atomic64_set((atomic64_t *)ring->wptr_cpu_addr, wptr_tmp);
4254− *wptr_saved = wptr_tmp;
4255− /* assume doorbell always used by mes mapped queue */
4256− if (*is_queue_unmap) {
4257− WDOORBELL64(aggregated_db_index, wptr_tmp);
4258− WDOORBELL64(ring->doorbell_index, wptr_tmp);
4259− } else {
4260− WDOORBELL64(ring->doorbell_index, wptr_tmp);
42614362
4262− if (*is_queue_unmap)
4263− WDOORBELL64(aggregated_db_index, wptr_tmp);
4264− }
4363+ /* XXX check if swapping is necessary on BE */
4364+ if (ring->use_doorbell) {
4365+ atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4366+ ring->wptr);
4367+ WDOORBELL64(ring->doorbell_index, ring->wptr);
42654368 } else {
4266− /* XXX check if swapping is necessary on BE */
4267− if (ring->use_doorbell) {
4268− atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4269− ring->wptr);
4270− WDOORBELL64(ring->doorbell_index, ring->wptr);
4271− } else {
4272− BUG(); /* only DOORBELL method supported on gfx12 now */
4273− }
4369+ BUG(); /* only DOORBELL method supported on gfx12 now */
42744370 }
42754371 }
42764372
@@ -4317,10 +4413,6 @@ static void gfx_v12_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, · 40 unmodified lines
43174413
43184414 control |= ib->length_dw | (vmid << 24);
43194415
4320− if (ring->is_mes_queue)
4321− /* inherit vmid from mqd */
4322− control |= 0x400000;
4323−
43244416 amdgpu_ring_write(ring, header);
43254417 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
43264418 amdgpu_ring_write(ring,
@@ -4340,10 +4432,6 @@ static void gfx_v12_0_ring_emit_ib_compute(struct amdgpu_ring *ring, · 13 unmodified lines
43404432 unsigned vmid = AMDGPU_JOB_GET_VMID(job);
43414433 u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);
43424434
4343− if (ring->is_mes_queue)
4344− /* inherit vmid from mqd */
4345− control |= 0x40000000;
4346−
43474435 amdgpu_ring_write(ring, PACKET3(PACKET3_INDIRECT_BUFFER, 2));
43484436 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
43494437 amdgpu_ring_write(ring,
@@ -4383,8 +4471,7 @@ static void gfx_v12_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, · 33 unmodified lines
43834471 amdgpu_ring_write(ring, upper_32_bits(addr));
43844472 amdgpu_ring_write(ring, lower_32_bits(seq));
43854473 amdgpu_ring_write(ring, upper_32_bits(seq));
4386− amdgpu_ring_write(ring, ring->is_mes_queue ?
4387− (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0);
4474+ amdgpu_ring_write(ring, 0);
43884475 }
43894476
43904477 static void gfx_v12_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
@@ -4412,10 +4499,7 @@ static void gfx_v12_0_ring_invalidate_tlbs(struct amdgpu_ring *ring, · 21 unmodified lines
44124499 static void gfx_v12_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
44134500 unsigned vmid, uint64_t pd_addr)
44144501 {
4415− if (ring->is_mes_queue)
4416− gfx_v12_0_ring_invalidate_tlbs(ring, 0, 0, false, 0);
4417− else
4418− amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
4502+ amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
44194503
44204504 /* compute doesn't have PFP */
44214505 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {
@@ -4749,25 +4833,23 @@ static int gfx_v12_0_eop_irq(struct amdgpu_device *adev, · 327 unmodified lines
47494833 struct amdgpu_irq_src *source,
47504834 struct amdgpu_iv_entry *entry)
47514835 {
4752− int i;
4836+ u32 doorbell_offset = entry->src_data[0];
47534837 u8 me_id, pipe_id, queue_id;
47544838 struct amdgpu_ring *ring;
4755− uint32_t mes_queue_id = entry->src_data[0];
4839+ int i;
47564840
47574841 DRM_DEBUG("IH: CP EOP\n");
47584842
4759− if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {
4760− struct amdgpu_mes_queue *queue;
4843+ if (adev->enable_mes && doorbell_offset) {
4844+ struct amdgpu_userq_fence_driver *fence_drv = NULL;
4845+ struct xarray *xa = &adev->userq_xa;
4846+ unsigned long flags;
47614847
4762− mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;
4763−
4764− spin_lock(&adev->mes.queue_id_lock);
4765− queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);
4766− if (queue) {
4767− DRM_DEBUG("process mes queue id = %d\n", mes_queue_id);
4768− amdgpu_fence_process(queue->ring);
4769− }
4770− spin_unlock(&adev->mes.queue_id_lock);
4848+ xa_lock_irqsave(xa, flags);
4849+ fence_drv = xa_load(xa, doorbell_offset);
4850+ if (fence_drv)
4851+ amdgpu_userq_fence_driver_process(fence_drv);
4852+ xa_unlock_irqrestore(xa, flags);
47714853 } else {
47724854 me_id = (entry->ring_id & 0x0c) >> 2;
47734855 pipe_id = (entry->ring_id & 0x03) >> 0;
@@ -4934,27 +5016,29 @@ static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev, · 160 unmodified lines
49345016 pipe_id = (entry->ring_id & 0x03) >> 0;
49355017 queue_id = (entry->ring_id & 0x70) >> 4;
49365018
4937− switch (me_id) {
4938− case 0:
4939− for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4940− ring = &adev->gfx.gfx_ring[i];
4941− if (ring->me == me_id && ring->pipe == pipe_id &&
4942− ring->queue == queue_id)
4943− drm_sched_fault(&ring->sched);
4944− }
4945− break;
4946− case 1:
4947− case 2:
4948− for (i = 0; i < adev->gfx.num_compute_rings; i++) {
4949− ring = &adev->gfx.compute_ring[i];
4950− if (ring->me == me_id && ring->pipe == pipe_id &&
4951− ring->queue == queue_id)
4952− drm_sched_fault(&ring->sched);
5019+ if (!adev->gfx.disable_kq) {
5020+ switch (me_id) {
5021+ case 0:
5022+ for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
5023+ ring = &adev->gfx.gfx_ring[i];
5024+ if (ring->me == me_id && ring->pipe == pipe_id &&
5025+ ring->queue == queue_id)
5026+ drm_sched_fault(&ring->sched);
5027+ }
5028+ break;
5029+ case 1:
5030+ case 2:
5031+ for (i = 0; i < adev->gfx.num_compute_rings; i++) {
5032+ ring = &adev->gfx.compute_ring[i];
5033+ if (ring->me == me_id && ring->pipe == pipe_id &&
5034+ ring->queue == queue_id)
5035+ drm_sched_fault(&ring->sched);
5036+ }
5037+ break;
5038+ default:
5039+ BUG();
5040+ break;
49535041 }
4954− break;
4955− default:
4956− BUG();
4957− break;
49585042 }
49595043 }
49605044
@@ -5160,6 +5244,69 @@ static void gfx_v12_ip_dump(struct amdgpu_ip_block *ip_block) · 199 unmodified lines
51605244 amdgpu_gfx_off_ctrl(adev, true);
51615245 }
51625246
5247+static bool gfx_v12_pipe_reset_support(struct amdgpu_device *adev)
5248+{
5249+ /* Disable the pipe reset until the CPFW fully support it.*/
5250+ dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
5251+ return false;
5252+}
5253+
5254+static int gfx_v12_reset_gfx_pipe(struct amdgpu_ring *ring)
5255+{
5256+ struct amdgpu_device *adev = ring->adev;
5257+ uint32_t reset_pipe = 0, clean_pipe = 0;
5258+ int r;
5259+
5260+ if (!gfx_v12_pipe_reset_support(adev))
5261+ return -EOPNOTSUPP;
5262+
5263+ gfx_v12_0_set_safe_mode(adev, 0);
5264+ mutex_lock(&adev->srbm_mutex);
5265+ soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5266+
5267+ switch (ring->pipe) {
5268+ case 0:
5269+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5270+ PFP_PIPE0_RESET, 1);
5271+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5272+ ME_PIPE0_RESET, 1);
5273+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5274+ PFP_PIPE0_RESET, 0);
5275+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5276+ ME_PIPE0_RESET, 0);
5277+ break;
5278+ case 1:
5279+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5280+ PFP_PIPE1_RESET, 1);
5281+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5282+ ME_PIPE1_RESET, 1);
5283+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5284+ PFP_PIPE1_RESET, 0);
5285+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5286+ ME_PIPE1_RESET, 0);
5287+ break;
5288+ default:
5289+ break;
5290+ }
5291+
5292+ WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
5293+ WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
5294+
5295+ r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
5296+ RS64_FW_UC_START_ADDR_LO;
5297+ soc24_grbm_select(adev, 0, 0, 0, 0);
5298+ mutex_unlock(&adev->srbm_mutex);
5299+ gfx_v12_0_unset_safe_mode(adev, 0);
5300+
5301+ dev_info(adev->dev, "The ring %s pipe reset: %s\n", ring->name,
5302+ r == 0 ? "successfully" : "failed");
5303+ /* Sometimes the ME start pc counter can't cache correctly, so the
5304+ * PC check only as a reference and pipe reset result rely on the
5305+ * later ring test.
5306+ */
5307+ return 0;
5308+}
5309+
51635310 static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
51645311 {
51655312 struct amdgpu_device *adev = ring->adev;
@@ -5170,8 +5317,10 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines
51705317
51715318 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false);
51725319 if (r) {
5173− dev_err(adev->dev, "reset via MES failed %d\n", r);
5174− return r;
5320+ dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r);
5321+ r = gfx_v12_reset_gfx_pipe(ring);
5322+ if (r)
5323+ return r;
51755324 }
51765325
51775326 r = gfx_v12_0_kgq_init_queue(ring, true);
@@ -5189,6 +5338,89 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 11 unmodified lines
51895338 return amdgpu_ring_test_ring(ring);
51905339 }
51915340
5341+static int gfx_v12_0_reset_compute_pipe(struct amdgpu_ring *ring)
5342+{
5343+ struct amdgpu_device *adev = ring->adev;
5344+ uint32_t reset_pipe = 0, clean_pipe = 0;
5345+ int r = 0;
5346+
5347+ if (!gfx_v12_pipe_reset_support(adev))
5348+ return -EOPNOTSUPP;
5349+
5350+ gfx_v12_0_set_safe_mode(adev, 0);
5351+ mutex_lock(&adev->srbm_mutex);
5352+ soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5353+
5354+ reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
5355+ clean_pipe = reset_pipe;
5356+
5357+ if (adev->gfx.rs64_enable) {
5358+ switch (ring->pipe) {
5359+ case 0:
5360+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5361+ MEC_PIPE0_RESET, 1);
5362+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5363+ MEC_PIPE0_RESET, 0);
5364+ break;
5365+ case 1:
5366+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5367+ MEC_PIPE1_RESET, 1);
5368+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5369+ MEC_PIPE1_RESET, 0);
5370+ break;
5371+ case 2:
5372+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5373+ MEC_PIPE2_RESET, 1);
5374+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5375+ MEC_PIPE2_RESET, 0);
5376+ break;
5377+ case 3:
5378+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5379+ MEC_PIPE3_RESET, 1);
5380+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5381+ MEC_PIPE3_RESET, 0);
5382+ break;
5383+ default:
5384+ break;
5385+ }
5386+ WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe);
5387+ WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe);
5388+ r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
5389+ RS64_FW_UC_START_ADDR_LO;
5390+ } else {
5391+ switch (ring->pipe) {
5392+ case 0:
5393+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
5394+ MEC_ME1_PIPE0_RESET, 1);
5395+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
5396+ MEC_ME1_PIPE0_RESET, 0);
5397+ break;
5398+ case 1:
5399+ reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
5400+ MEC_ME1_PIPE1_RESET, 1);
5401+ clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
5402+ MEC_ME1_PIPE1_RESET, 0);
5403+ break;
5404+ default:
5405+ break;
5406+ }
5407+ WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe);
5408+ WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe);
5409+ /* Doesn't find the F32 MEC instruction pointer register, and suppose
5410+ * the driver won't run into the F32 mode.
5411+ */
5412+ }
5413+
5414+ soc24_grbm_select(adev, 0, 0, 0, 0);
5415+ mutex_unlock(&adev->srbm_mutex);
5416+ gfx_v12_0_unset_safe_mode(adev, 0);
5417+
5418+ dev_info(adev->dev, "The ring %s pipe resets: %s\n", ring->name,
5419+ r == 0 ? "successfully" : "failed");
5420+ /* Need the ring test to verify the pipe reset result.*/
5421+ return 0;
5422+}
5423+
51925424 static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)
51935425 {
51945426 struct amdgpu_device *adev = ring->adev;
@@ -5199,8 +5431,10 @@ static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines
51995431
52005432 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true);
52015433 if (r) {
5202− dev_err(adev->dev, "reset via MMIO failed %d\n", r);
5203− return r;
5434+ dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r);
5435+ r = gfx_v12_0_reset_compute_pipe(ring);
5436+ if (r)
5437+ return r;
52045438 }
52055439
52065440 r = gfx_v12_0_kcq_init_queue(ring, true);
drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c+15−31
@@ -53,6 +53,9 @@ · 52 unmodified lines
5353 #define VERDE_GB_ADDR_CONFIG_GOLDEN 0x12010002
5454 #define HAINAN_GB_ADDR_CONFIG_GOLDEN 0x02010001
5555
56+#define GFX6_NUM_GFX_RINGS 1
57+#define GFX6_NUM_COMPUTE_RINGS 2
58+
5659 static void gfx_v6_0_set_ring_funcs(struct amdgpu_device *adev);
5760 static void gfx_v6_0_set_irq_funcs(struct amdgpu_device *adev);
5861 static void gfx_v6_0_get_cu_info(struct amdgpu_device *adev);
@@ -1732,10 +1735,14 @@ static void gfx_v6_0_constants_init(struct amdgpu_device *adev) · 1673 unmodified lines
17321735 gfx_v6_0_get_cu_info(adev);
17331736 gfx_v6_0_config_init(adev);
17341737
1735− WREG32(mmCP_QUEUE_THRESHOLDS, ((0x16 << CP_QUEUE_THRESHOLDS__ROQ_IB1_START__SHIFT) |
1736− (0x2b << CP_QUEUE_THRESHOLDS__ROQ_IB2_START__SHIFT)));
1737− WREG32(mmCP_MEQ_THRESHOLDS, (0x30 << CP_MEQ_THRESHOLDS__MEQ1_START__SHIFT) |
1738− (0x60 << CP_MEQ_THRESHOLDS__MEQ2_START__SHIFT));
1738+ WREG32(mmCP_QUEUE_THRESHOLDS,
1739+ ((0x16 << CP_QUEUE_THRESHOLDS__ROQ_IB1_START__SHIFT) |
1740+ (0x2b << CP_QUEUE_THRESHOLDS__ROQ_IB2_START__SHIFT)));
1741+
1742+ /* set HW defaults for 3D engine */
1743+ WREG32(mmCP_MEQ_THRESHOLDS,
1744+ (0x30 << CP_MEQ_THRESHOLDS__MEQ1_START__SHIFT) |
1745+ (0x60 << CP_MEQ_THRESHOLDS__MEQ2_START__SHIFT));
17391746
17401747 sx_debug_1 = RREG32(mmSX_DEBUG_1);
17411748 WREG32(mmSX_DEBUG_1, sx_debug_1);
@@ -2851,44 +2858,21 @@ static u32 gfx_v6_0_get_csb_size(struct amdgpu_device *adev) · 1109 unmodified lines
28512858 static void gfx_v6_0_get_csb_buffer(struct amdgpu_device *adev,
28522859 volatile u32 *buffer)
28532860 {
2854− u32 count = 0, i;
2855− const struct cs_section_def *sect = NULL;
2856− const struct cs_extent_def *ext = NULL;
2861+ u32 count = 0;
28572862
28582863 if (adev->gfx.rlc.cs_data == NULL)
28592864 return;
28602865 if (buffer == NULL)
28612866 return;
28622867
2863− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
2864− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
2865− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
2866− buffer[count++] = cpu_to_le32(0x80000000);
2867− buffer[count++] = cpu_to_le32(0x80000000);
2868−
2869− for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
2870− for (ext = sect->section; ext->extent != NULL; ++ext) {
2871− if (sect->id == SECT_CONTEXT) {
2872− buffer[count++] =
2873− cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
2874− buffer[count++] = cpu_to_le32(ext->reg_index - 0xa000);
2875− for (i = 0; i < ext->reg_count; i++)
2876− buffer[count++] = cpu_to_le32(ext->extent[i]);
2877− } else {
2878− return;
2879− }
2880− }
2881− }
2868+ count = amdgpu_gfx_csb_preamble_start(buffer);
2869+ count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
28822870
28832871 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));
28842872 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);
28852873 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);
28862874
2887− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
2888− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
2889−
2890− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
2891− buffer[count++] = cpu_to_le32(0);
2875+ amdgpu_gfx_csb_preamble_end(buffer, count);
28922876 }
28932877
28942878 static void gfx_v6_0_init_pg(struct amdgpu_device *adev)
drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c+9−51
@@ -55,6 +55,9 @@ · 54 unmodified lines
5555 #define GFX7_NUM_GFX_RINGS 1
5656 #define GFX7_MEC_HPD_SIZE 2048
5757
58+#define BONAIRE_GB_ADDR_CONFIG_GOLDEN 0x12010001
59+#define HAWAII_GB_ADDR_CONFIG_GOLDEN 0x12011003
60+
5861 static void gfx_v7_0_set_ring_funcs(struct amdgpu_device *adev);
5962 static void gfx_v7_0_set_irq_funcs(struct amdgpu_device *adev);
6063 static void gfx_v7_0_set_gds_init(struct amdgpu_device *adev);
@@ -3882,67 +3885,22 @@ static u32 gfx_v7_0_get_csb_size(struct amdgpu_device *adev) · 3821 unmodified lines
38823885 static void gfx_v7_0_get_csb_buffer(struct amdgpu_device *adev,
38833886 volatile u32 *buffer)
38843887 {
3885− u32 count = 0, i;
3886− const struct cs_section_def *sect = NULL;
3887− const struct cs_extent_def *ext = NULL;
3888+ u32 count = 0;
38883889
38893890 if (adev->gfx.rlc.cs_data == NULL)
38903891 return;
38913892 if (buffer == NULL)
38923893 return;
38933894
3894− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
3895− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
3896−
3897− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
3898− buffer[count++] = cpu_to_le32(0x80000000);
3899− buffer[count++] = cpu_to_le32(0x80000000);
3900−
3901− for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
3902− for (ext = sect->section; ext->extent != NULL; ++ext) {
3903− if (sect->id == SECT_CONTEXT) {
3904− buffer[count++] =
3905− cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
3906− buffer[count++] = cpu_to_le32(ext->reg_index - PACKET3_SET_CONTEXT_REG_START);
3907− for (i = 0; i < ext->reg_count; i++)
3908− buffer[count++] = cpu_to_le32(ext->extent[i]);
3909− } else {
3910− return;
3911− }
3912− }
3913− }
3895+ count = amdgpu_gfx_csb_preamble_start(buffer);
3896+ count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
39143897
39153898 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2));
39163899 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);
3917− switch (adev->asic_type) {
3918− case CHIP_BONAIRE:
3919− buffer[count++] = cpu_to_le32(0x16000012);
3920− buffer[count++] = cpu_to_le32(0x00000000);
3921− break;
3922− case CHIP_KAVERI:
3923− buffer[count++] = cpu_to_le32(0x00000000); /* XXX */
3924− buffer[count++] = cpu_to_le32(0x00000000);
3925− break;
3926− case CHIP_KABINI:
3927− case CHIP_MULLINS:
3928− buffer[count++] = cpu_to_le32(0x00000000); /* XXX */
3929− buffer[count++] = cpu_to_le32(0x00000000);
3930− break;
3931− case CHIP_HAWAII:
3932− buffer[count++] = cpu_to_le32(0x3a00161a);
3933− buffer[count++] = cpu_to_le32(0x0000002e);
3934− break;
3935− default:
3936− buffer[count++] = cpu_to_le32(0x00000000);
3937− buffer[count++] = cpu_to_le32(0x00000000);
3938− break;
3939− }
3940−
3941− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
3942− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
3900+ buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);
3901+ buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1);
39433902
3944− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
3945− buffer[count++] = cpu_to_le32(0);
3903+ amdgpu_gfx_csb_preamble_end(buffer, count);
39463904 }
39473905
39483906 static void gfx_v7_0_init_pg(struct amdgpu_device *adev)
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c+5−31
@@ -1223,48 +1223,22 @@ out: · 1222 unmodified lines
12231223 static void gfx_v8_0_get_csb_buffer(struct amdgpu_device *adev,
12241224 volatile u32 *buffer)
12251225 {
1226− u32 count = 0, i;
1227− const struct cs_section_def *sect = NULL;
1228− const struct cs_extent_def *ext = NULL;
1226+ u32 count = 0;
12291227
12301228 if (adev->gfx.rlc.cs_data == NULL)
12311229 return;
12321230 if (buffer == NULL)
12331231 return;
12341232
1235− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
1236− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
1237−
1238− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
1239− buffer[count++] = cpu_to_le32(0x80000000);
1240− buffer[count++] = cpu_to_le32(0x80000000);
1241−
1242− for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
1243− for (ext = sect->section; ext->extent != NULL; ++ext) {
1244− if (sect->id == SECT_CONTEXT) {
1245− buffer[count++] =
1246− cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
1247− buffer[count++] = cpu_to_le32(ext->reg_index -
1248− PACKET3_SET_CONTEXT_REG_START);
1249− for (i = 0; i < ext->reg_count; i++)
1250− buffer[count++] = cpu_to_le32(ext->extent[i]);
1251− } else {
1252− return;
1253− }
1254− }
1255− }
1233+ count = amdgpu_gfx_csb_preamble_start(buffer);
1234+ count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
12561235
12571236 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2));
1258− buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG -
1259− PACKET3_SET_CONTEXT_REG_START);
1237+ buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);
12601238 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);
12611239 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1);
12621240
1263− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
1264− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
1265−
1266− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
1267− buffer[count++] = cpu_to_le32(0);
1241+ amdgpu_gfx_csb_preamble_end(buffer, count);
12681242 }
12691243
12701244 static int gfx_v8_0_cp_jump_table_num(struct amdgpu_device *adev)
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c+73−98
@@ -225,17 +225,36 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_9[] = { · 224 unmodified lines
225225 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_SMU_SAFE_MODE),
226226 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_INT_STAT),
227227 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_GENERAL_6),
228− /* cp header registers */
229− SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
230− SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
231− SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME2_HEADER_DUMP),
232− SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
233− SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
234228 /* SE status registers */
235229 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0),
236230 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE1),
237231 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE2),
238− SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE3)
232+ SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE3),
233+ /* packet headers */
234+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
235+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
236+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
237+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
238+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
239+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
240+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
241+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
242+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
243+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
244+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
245+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
246+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
247+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
248+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
249+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
250+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
251+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
252+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
253+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
254+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
255+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
256+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
257+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP)
239258 };
240259
241260 static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = {
@@ -277,6 +296,14 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = { · 35 unmodified lines
277296 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_LO),
278297 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_HI),
279298 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_GFX_STATUS),
299+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
300+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
301+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
302+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
303+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
304+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
305+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
306+ SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP)
280307 };
281308
282309 enum ta_ras_gfx_subblock {
@@ -1624,42 +1651,16 @@ static u32 gfx_v9_0_get_csb_size(struct amdgpu_device *adev) · 1341 unmodified lines
16241651 static void gfx_v9_0_get_csb_buffer(struct amdgpu_device *adev,
16251652 volatile u32 *buffer)
16261653 {
1627− u32 count = 0, i;
1628− const struct cs_section_def *sect = NULL;
1629− const struct cs_extent_def *ext = NULL;
1654+ u32 count = 0;
16301655
16311656 if (adev->gfx.rlc.cs_data == NULL)
16321657 return;
16331658 if (buffer == NULL)
16341659 return;
16351660
1636− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
1637− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
1638−
1639− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
1640− buffer[count++] = cpu_to_le32(0x80000000);
1641− buffer[count++] = cpu_to_le32(0x80000000);
1642−
1643− for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
1644− for (ext = sect->section; ext->extent != NULL; ++ext) {
1645− if (sect->id == SECT_CONTEXT) {
1646− buffer[count++] =
1647− cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
1648− buffer[count++] = cpu_to_le32(ext->reg_index -
1649− PACKET3_SET_CONTEXT_REG_START);
1650− for (i = 0; i < ext->reg_count; i++)
1651− buffer[count++] = cpu_to_le32(ext->extent[i]);
1652− } else {
1653− return;
1654− }
1655− }
1656− }
1657−
1658− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
1659− buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
1660−
1661− buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
1662− buffer[count++] = cpu_to_le32(0);
1661+ count = amdgpu_gfx_csb_preamble_start(buffer);
1662+ count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
1663+ amdgpu_gfx_csb_preamble_end(buffer, count);
16631664 }
16641665
16651666 static void gfx_v9_0_init_always_on_cu_mask(struct amdgpu_device *adev)
@@ -5441,16 +5442,8 @@ static void gfx_v9_0_ring_patch_ce_meta(struct amdgpu_ring *ring, · 3775 unmodified lines
54415442
54425443 payload_size = sizeof(struct v9_ce_ib_state);
54435444
5444− if (ring->is_mes_queue) {
5445− payload_offset = offsetof(struct amdgpu_mes_ctx_meta_data,
5446− gfx[0].gfx_meta_data) +
5447− offsetof(struct v9_gfx_meta_data, ce_payload);
5448− ce_payload_cpu_addr =
5449− amdgpu_mes_ctx_get_offs_cpu_addr(ring, payload_offset);
5450− } else {
5451− payload_offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5452− ce_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
5453− }
5445+ payload_offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5446+ ce_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
54545447
54555448 if (offset + (payload_size >> 2) <= ring->buf_mask + 1) {
54565449 memcpy((void *)&ring->ring[offset], ce_payload_cpu_addr, payload_size);
@@ -5473,16 +5466,8 @@ static void gfx_v9_0_ring_patch_de_meta(struct amdgpu_ring *ring, · 16 unmodified lines
54735466
54745467 payload_size = sizeof(struct v9_de_ib_state);
54755468
5476− if (ring->is_mes_queue) {
5477− payload_offset = offsetof(struct amdgpu_mes_ctx_meta_data,
5478− gfx[0].gfx_meta_data) +
5479− offsetof(struct v9_gfx_meta_data, de_payload);
5480− de_payload_cpu_addr =
5481− amdgpu_mes_ctx_get_offs_cpu_addr(ring, payload_offset);
5482− } else {
5483− payload_offset = offsetof(struct v9_gfx_meta_data, de_payload);
5484− de_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
5485− }
5469+ payload_offset = offsetof(struct v9_gfx_meta_data, de_payload);
5470+ de_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
54865471
54875472 ((struct v9_de_ib_state *)de_payload_cpu_addr)->ib_completion_status =
54885473 IB_COMPLETION_STATUS_PREEMPTED;
@@ -5672,19 +5657,9 @@ static void gfx_v9_0_ring_emit_ce_meta(struct amdgpu_ring *ring, bool resume) · 183 unmodified lines
56725657
56735658 cnt = (sizeof(ce_payload) >> 2) + 4 - 2;
56745659
5675− if (ring->is_mes_queue) {
5676− offset = offsetof(struct amdgpu_mes_ctx_meta_data,
5677− gfx[0].gfx_meta_data) +
5678− offsetof(struct v9_gfx_meta_data, ce_payload);
5679− ce_payload_gpu_addr =
5680− amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
5681− ce_payload_cpu_addr =
5682− amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
5683− } else {
5684− offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5685− ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5686− ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
5687− }
5660+ offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5661+ ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5662+ ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
56885663
56895664 amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
56905665 amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(2) |
@@ -5770,28 +5745,13 @@ static void gfx_v9_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume, bo · 79 unmodified lines
57705745 void *de_payload_cpu_addr;
57715746 int cnt;
57725747
5773− if (ring->is_mes_queue) {
5774− offset = offsetof(struct amdgpu_mes_ctx_meta_data,
5775− gfx[0].gfx_meta_data) +
5776− offsetof(struct v9_gfx_meta_data, de_payload);
5777− de_payload_gpu_addr =
5778− amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
5779− de_payload_cpu_addr =
5780− amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
5781−
5782− offset = offsetof(struct amdgpu_mes_ctx_meta_data,
5783− gfx[0].gds_backup) +
5784− offsetof(struct v9_gfx_meta_data, de_payload);
5785− gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
5786− } else {
5787− offset = offsetof(struct v9_gfx_meta_data, de_payload);
5788− de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5789− de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
5748+ offset = offsetof(struct v9_gfx_meta_data, de_payload);
5749+ de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5750+ de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
57905751
5791− gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
5792− AMDGPU_CSA_SIZE - adev->gds.gds_size,
5793− PAGE_SIZE);
5794− }
5752+ gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
5753+ AMDGPU_CSA_SIZE - adev->gds.gds_size,
5754+ PAGE_SIZE);
57955755
57965756 if (usegds) {
57975757 de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);
@@ -7339,9 +7299,14 @@ static void gfx_v9_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer · 1541 unmodified lines
73397299 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {
73407300 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);
73417301 for (reg = 0; reg < reg_count; reg++) {
7342− drm_printf(p, "%-50s \t 0x%08x\n",
7343− gc_cp_reg_list_9[reg].reg_name,
7344− adev->gfx.ip_dump_compute_queues[index + reg]);
7302+ if (i && gc_cp_reg_list_9[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
7303+ drm_printf(p, "%-50s \t 0x%08x\n",
7304+ "mmCP_MEC_ME2_HEADER_DUMP",
7305+ adev->gfx.ip_dump_compute_queues[index + reg]);
7306+ else
7307+ drm_printf(p, "%-50s \t 0x%08x\n",
7308+ gc_cp_reg_list_9[reg].reg_name,
7309+ adev->gfx.ip_dump_compute_queues[index + reg]);
73457310 }
73467311 index += reg_count;
73477312 }
@@ -7378,9 +7343,13 @@ static void gfx_v9_ip_dump(struct amdgpu_ip_block *ip_block) · 30 unmodified lines
73787343 soc15_grbm_select(adev, 1 + i, j, k, 0, 0);
73797344
73807345 for (reg = 0; reg < reg_count; reg++) {
7381− adev->gfx.ip_dump_compute_queues[index + reg] =
7382− RREG32(SOC15_REG_ENTRY_OFFSET(
7383− gc_cp_reg_list_9[reg]));
7346+ if (i && gc_cp_reg_list_9[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
7347+ adev->gfx.ip_dump_compute_queues[index + reg] =
7348+ RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_MEC_ME2_HEADER_DUMP));
7349+ else
7350+ adev->gfx.ip_dump_compute_queues[index + reg] =
7351+ RREG32(SOC15_REG_ENTRY_OFFSET(
7352+ gc_cp_reg_list_9[reg]));
73847353 }
73857354 index += reg_count;
73867355 }
@@ -7394,8 +7363,14 @@ static void gfx_v9_ip_dump(struct amdgpu_ip_block *ip_block) · 7 unmodified lines
73947363
73957364 static void gfx_v9_0_ring_emit_cleaner_shader(struct amdgpu_ring *ring)
73967365 {
7366+ struct amdgpu_device *adev = ring->adev;
7367+
73977368 /* Emit the cleaner shader */
7398− amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER, 0));
7369+ if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 2))
7370+ amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER, 0));
7371+ else
7372+ amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER_9_0, 0));
7373+
73997374 amdgpu_ring_write(ring, 0); /* RESERVED field, programmed to zero */
74007375 }
74017376
drivers/gpu/drm/amd/amdgpu/gfx_v9_4_2.c+2−2
@@ -1547,7 +1547,7 @@ static void gfx_v9_4_2_log_utc_edc_count(struct amdgpu_device *adev, · 1546 unmodified lines
15471547 {
15481548 uint32_t bank, way, mem;
15491549 static const char * const vml2_way_str[] = { "BIGK", "4K" };
1550− static const char * const utcl2_rounter_str[] = { "VMC", "APT" };
1550+ static const char * const utcl2_router_str[] = { "VMC", "APT" };
15511551
15521552 mem = instance % blk->num_mem_blocks;
15531553 way = (instance / blk->num_mem_blocks) % blk->num_ways;
@@ -1568,7 +1568,7 @@ static void gfx_v9_4_2_log_utc_edc_count(struct amdgpu_device *adev, · 14 unmodified lines
15681568 dev_info(
15691569 adev->dev,
15701570 "GFX SubBlock UTCL2_ROUTER_IFIF%d_GROUP0_%s, SED %d, DED %d\n",
1571− bank, utcl2_rounter_str[mem], sec_cnt, ded_cnt);
1571+ bank, utcl2_router_str[mem], sec_cnt, ded_cnt);
15721572 break;
15731573 case ATC_L2_CACHE_2M:
15741574 dev_info(
drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c+37−15
@@ -105,9 +105,6 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_9_4_3[] = { · 104 unmodified lines
105105 SOC15_REG_ENTRY_STR(GC, 0, regRLC_SMU_SAFE_MODE),
106106 SOC15_REG_ENTRY_STR(GC, 0, regRLC_INT_STAT),
107107 SOC15_REG_ENTRY_STR(GC, 0, regRLC_GPM_GENERAL_6),
108− /* cp header registers */
109− SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
110− SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME2_HEADER_DUMP),
111108 /* SE status registers */
112109 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),
113110 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE1),
@@ -154,6 +151,14 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9_4_3[] = { · 40 unmodified lines
154151 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_LO),
155152 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_HI),
156153 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_GFX_STATUS),
154+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
155+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
156+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
157+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
158+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
159+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
160+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
161+ SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
157162 };
158163
159164 struct amdgpu_gfx_ras gfx_v9_4_3_ras;
@@ -4558,12 +4563,21 @@ static void gfx_v9_4_3_ip_print(struct amdgpu_ip_block *ip_block, struct drm_pri · 4398 unmodified lines
45584563 "\nxcc:%d mec:%d, pipe:%d, queue:%d\n",
45594564 xcc_id, i, j, k);
45604565 for (reg = 0; reg < reg_count; reg++) {
4561− drm_printf(p,
4562− "%-50s \t 0x%08x\n",
4563− gc_cp_reg_list_9_4_3[reg].reg_name,
4564− adev->gfx.ip_dump_compute_queues
4565− [xcc_offset + inst_offset +
4566− reg]);
4566+ if (i && gc_cp_reg_list_9_4_3[reg].reg_offset ==
4567+ regCP_MEC_ME1_HEADER_DUMP)
4568+ drm_printf(p,
4569+ "%-50s \t 0x%08x\n",
4570+ "regCP_MEC_ME2_HEADER_DUMP",
4571+ adev->gfx.ip_dump_compute_queues
4572+ [xcc_offset + inst_offset +
4573+ reg]);
4574+ else
4575+ drm_printf(p,
4576+ "%-50s \t 0x%08x\n",
4577+ gc_cp_reg_list_9_4_3[reg].reg_name,
4578+ adev->gfx.ip_dump_compute_queues
4579+ [xcc_offset + inst_offset +
4580+ reg]);
45674581 }
45684582 inst_offset += reg_count;
45694583 }
@@ -4612,12 +4626,20 @@ static void gfx_v9_4_3_ip_dump(struct amdgpu_ip_block *ip_block) · 42 unmodified lines
46124626 GET_INST(GC, xcc_id));
46134627
46144628 for (reg = 0; reg < reg_count; reg++) {
4615− adev->gfx.ip_dump_compute_queues
4616− [xcc_offset +
4617− inst_offset + reg] =
4618− RREG32(SOC15_REG_ENTRY_OFFSET_INST(
4619− gc_cp_reg_list_9_4_3[reg],
4620− GET_INST(GC, xcc_id)));
4629+ if (i && gc_cp_reg_list_9_4_3[reg].reg_offset ==
4630+ regCP_MEC_ME1_HEADER_DUMP)
4631+ adev->gfx.ip_dump_compute_queues
4632+ [xcc_offset +
4633+ inst_offset + reg] =
4634+ RREG32(SOC15_REG_OFFSET(GC, GET_INST(GC, xcc_id),
4635+ regCP_MEC_ME2_HEADER_DUMP));
4636+ else
4637+ adev->gfx.ip_dump_compute_queues
4638+ [xcc_offset +
4639+ inst_offset + reg] =
4640+ RREG32(SOC15_REG_ENTRY_OFFSET_INST(
4641+ gc_cp_reg_list_9_4_3[reg],
4642+ GET_INST(GC, xcc_id)));
46214643 }
46224644 inst_offset += reg_count;
46234645 }
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c+0−4
@@ -428,10 +428,6 @@ static void gmc_v10_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned int · 427 unmodified lines
428428 struct amdgpu_device *adev = ring->adev;
429429 uint32_t reg;
430430
431− /* MES fw manages IH_VMID_x_LUT updating */
432− if (ring->is_mes_queue)
433− return;
434−
435431 if (ring->vm_hub == AMDGPU_GFXHUB(0))
436432 reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid;
437433 else
drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c+1−5
@@ -393,10 +393,6 @@ static void gmc_v11_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned int · 392 unmodified lines
393393 struct amdgpu_device *adev = ring->adev;
394394 uint32_t reg;
395395
396− /* MES fw manages IH_VMID_x_LUT updating */
397− if (ring->is_mes_queue)
398− return;
399−
400396 if (ring->vm_hub == AMDGPU_GFXHUB(0))
401397 reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid;
402398 else
@@ -832,7 +828,7 @@ static int gmc_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 429 unmodified lines
832828 * amdgpu graphics/compute will use VMIDs 1-7
833829 * amdkfd will use VMIDs 8-15
834830 */
835− adev->vm_manager.first_kfd_vmid = 8;
831+ adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
836832
837833 amdgpu_vm_manager_init(adev);
838834
drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c+1−5
@@ -413,10 +413,6 @@ static void gmc_v12_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned vmid · 412 unmodified lines
413413 struct amdgpu_device *adev = ring->adev;
414414 uint32_t reg;
415415
416− /* MES fw manages IH_VMID_x_LUT updating */
417− if (ring->is_mes_queue)
418− return;
419−
420416 if (ring->vm_hub == AMDGPU_GFXHUB(0))
421417 reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid;
422418 else
@@ -820,7 +816,7 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 397 unmodified lines
820816 * amdgpu graphics/compute will use VMIDs 1-7
821817 * amdkfd will use VMIDs 8-15
822818 */
823− adev->vm_manager.first_kfd_vmid = 8;
819+ adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
824820
825821 amdgpu_vm_manager_init(adev);
826822
drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c+29−17
@@ -249,7 +249,7 @@ static void gmc_v6_0_mc_program(struct amdgpu_device *adev) · 248 unmodified lines
249249
250250 /* disable VGA render */
251251 tmp = RREG32(mmVGA_RENDER_CONTROL);
252− tmp &= ~VGA_VSTATUS_CNTL;
252+ tmp &= VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK;
253253 WREG32(mmVGA_RENDER_CONTROL, tmp);
254254 }
255255 /* Update configuration */
@@ -627,17 +627,16 @@ static void gmc_v6_0_vm_decode_fault(struct amdgpu_device *adev, · 371 unmodified lines
627627 "write" : "read", block, mc_client, mc_id);
628628 }
629629
630−/*
631630 static const u32 mc_cg_registers[] = {
632− MC_HUB_MISC_HUB_CG,
633− MC_HUB_MISC_SIP_CG,
634− MC_HUB_MISC_VM_CG,
635− MC_XPB_CLK_GAT,
636− ATC_MISC_CG,
637− MC_CITF_MISC_WR_CG,
638− MC_CITF_MISC_RD_CG,
639− MC_CITF_MISC_VM_CG,
640− VM_L2_CG,
631+ mmMC_HUB_MISC_HUB_CG,
632+ mmMC_HUB_MISC_SIP_CG,
633+ mmMC_HUB_MISC_VM_CG,
634+ mmMC_XPB_CLK_GAT,
635+ mmATC_MISC_CG,
636+ mmMC_CITF_MISC_WR_CG,
637+ mmMC_CITF_MISC_RD_CG,
638+ mmMC_CITF_MISC_VM_CG,
639+ mmVM_L2_CG,
641640 };
642641
643642 static const u32 mc_cg_ls_en[] = {
@@ -672,7 +671,7 @@ static void gmc_v6_0_enable_mc_ls(struct amdgpu_device *adev, · 28 unmodified lines
672671
673672 for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {
674673 orig = data = RREG32(mc_cg_registers[i]);
675− if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_LS))
674+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_LS))
676675 data |= mc_cg_ls_en[i];
677676 else
678677 data &= ~mc_cg_ls_en[i];
@@ -689,7 +688,7 @@ static void gmc_v6_0_enable_mc_mgcg(struct amdgpu_device *adev, · 10 unmodified lines
689688
690689 for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {
691690 orig = data = RREG32(mc_cg_registers[i]);
692− if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_MGCG))
691+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_MGCG))
693692 data |= mc_cg_en[i];
694693 else
695694 data &= ~mc_cg_en[i];
@@ -705,7 +704,7 @@ static void gmc_v6_0_enable_bif_mgls(struct amdgpu_device *adev, · 9 unmodified lines
705704
706705 orig = data = RREG32_PCIE(ixPCIE_CNTL2);
707706
708− if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_BIF_LS)) {
707+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_BIF_LS)) {
709708 data = REG_SET_FIELD(data, PCIE_CNTL2, SLV_MEM_LS_EN, 1);
710709 data = REG_SET_FIELD(data, PCIE_CNTL2, MST_MEM_LS_EN, 1);
711710 data = REG_SET_FIELD(data, PCIE_CNTL2, REPLAY_MEM_LS_EN, 1);
@@ -728,7 +727,7 @@ static void gmc_v6_0_enable_hdp_mgcg(struct amdgpu_device *adev, · 16 unmodified lines
728727
729728 orig = data = RREG32(mmHDP_HOST_PATH_CNTL);
730729
731− if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_MGCG))
730+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_MGCG))
732731 data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 0);
733732 else
734733 data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 1);
@@ -744,7 +743,7 @@ static void gmc_v6_0_enable_hdp_ls(struct amdgpu_device *adev, · 9 unmodified lines
744743
745744 orig = data = RREG32(mmHDP_MEM_POWER_LS);
746745
747− if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_LS))
746+ if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_LS))
748747 data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 1);
749748 else
750749 data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 0);
@@ -752,7 +751,6 @@ static void gmc_v6_0_enable_hdp_ls(struct amdgpu_device *adev, · 1 unmodified line
752751 if (orig != data)
753752 WREG32(mmHDP_MEM_POWER_LS, data);
754753 }
755−*/
756754
757755 static int gmc_v6_0_convert_vram_type(int mc_seq_vram_type)
758756 {
@@ -1098,6 +1096,20 @@ static int gmc_v6_0_process_interrupt(struct amdgpu_device *adev, · 339 unmodified lines
10981096 static int gmc_v6_0_set_clockgating_state(struct amdgpu_ip_block *ip_block,
10991097 enum amd_clockgating_state state)
11001098 {
1099+ struct amdgpu_device *adev = ip_block->adev;
1100+ bool gate = false;
1101+
1102+ if (state == AMD_CG_STATE_GATE)
1103+ gate = true;
1104+
1105+ if (!(adev->flags & AMD_IS_APU)) {
1106+ gmc_v6_0_enable_mc_mgcg(adev, gate);
1107+ gmc_v6_0_enable_mc_ls(adev, gate);
1108+ }
1109+ gmc_v6_0_enable_bif_mgls(adev, gate);
1110+ gmc_v6_0_enable_hdp_mgcg(adev, gate);
1111+ gmc_v6_0_enable_hdp_ls(adev, gate);
1112+
11011113 return 0;
11021114 }
11031115
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c+1−8
@@ -1157,17 +1157,10 @@ static bool gmc_v7_0_is_idle(struct amdgpu_ip_block *ip_block) · 1156 unmodified lines
11571157 static int gmc_v7_0_wait_for_idle(struct amdgpu_ip_block *ip_block)
11581158 {
11591159 unsigned int i;
1160− u32 tmp;
11611160 struct amdgpu_device *adev = ip_block->adev;
11621161
11631162 for (i = 0; i < adev->usec_timeout; i++) {
1164− /* read MC_STATUS */
1165− tmp = RREG32(mmSRBM_STATUS) & (SRBM_STATUS__MCB_BUSY_MASK |
1166− SRBM_STATUS__MCB_NON_DISPLAY_BUSY_MASK |
1167− SRBM_STATUS__MCC_BUSY_MASK |
1168− SRBM_STATUS__MCD_BUSY_MASK |
1169− SRBM_STATUS__VMC_BUSY_MASK);
1170− if (!tmp)
1163+ if (gmc_v7_0_is_idle(ip_block))
11711164 return 0;
11721165 udelay(1);
11731166 }
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c+2−12
@@ -1213,10 +1213,7 @@ static void gmc_v9_0_get_coherence_flags(struct amdgpu_device *adev, · 1212 unmodified lines
12131213 if (uncached) {
12141214 mtype = MTYPE_UC;
12151215 } else if (ext_coherent) {
1216− if (gc_ip_version == IP_VERSION(9, 5, 0) || adev->rev_id)
1217− mtype = is_local ? MTYPE_CC : MTYPE_UC;
1218− else
1219− mtype = MTYPE_UC;
1216+ mtype = is_local ? MTYPE_CC : MTYPE_UC;
12201217 } else if (adev->flags & AMD_IS_APU) {
12211218 mtype = is_local ? mtype_local : MTYPE_NC;
12221219 } else {
@@ -1336,7 +1333,7 @@ static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev, · 113 unmodified lines
13361333 mtype_local = MTYPE_CC;
13371334
13381335 *flags = AMDGPU_PTE_MTYPE_VG10(*flags, mtype_local);
1339− } else if (adev->rev_id) {
1336+ } else {
13401337 /* MTYPE_UC case */
13411338 *flags = AMDGPU_PTE_MTYPE_VG10(*flags, MTYPE_CC);
13421339 }
@@ -2411,13 +2408,6 @@ static int gmc_v9_0_hw_init(struct amdgpu_ip_block *ip_block) · 1068 unmodified lines
24112408 adev->gmc.flush_tlb_needs_extra_type_2 =
24122409 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 0) &&
24132410 adev->gmc.xgmi.num_physical_nodes;
2414− /*
2415− * TODO: This workaround is badly documented and had a buggy
2416− * implementation. We should probably verify what we do here.
2417− */
2418− adev->gmc.flush_tlb_needs_extra_type_0 =
2419− amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) &&
2420− adev->rev_id == 0;
24212411
24222412 /* The sequence of these two function calls matters.*/
24232413 gmc_v9_0_init_golden_registers(adev);
drivers/gpu/drm/amd/amdgpu/hdp_v4_0.c+1−12
@@ -36,17 +36,6 @@ · 35 unmodified lines
3636 #define HDP_MEM_POWER_CTRL__RC_MEM_POWER_LS_EN_MASK 0x00020000L
3737 #define mmHDP_MEM_POWER_CTRL_BASE_IDX 0
3838
39−static void hdp_v4_0_flush_hdp(struct amdgpu_device *adev,
40− struct amdgpu_ring *ring)
41−{
42− if (!ring || !ring->funcs->emit_wreg) {
43− WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
44− RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
45− } else {
46− amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
47− }
48−}
49−
5039 static void hdp_v4_0_invalidate_hdp(struct amdgpu_device *adev,
5140 struct amdgpu_ring *ring)
5241 {
@@ -180,7 +169,7 @@ struct amdgpu_hdp_ras hdp_v4_0_ras = { · 127 unmodified lines
180169 };
181170
182171 const struct amdgpu_hdp_funcs hdp_v4_0_funcs = {
183− .flush_hdp = hdp_v4_0_flush_hdp,
172+ .flush_hdp = amdgpu_hdp_generic_flush,
184173 .invalidate_hdp = hdp_v4_0_invalidate_hdp,
185174 .update_clock_gating = hdp_v4_0_update_clock_gating,
186175 .get_clock_gating_state = hdp_v4_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/hdp_v5_0.c+1−12
@@ -27,17 +27,6 @@ · 26 unmodified lines
2727 #include "hdp/hdp_5_0_0_sh_mask.h"
2828 #include <uapi/linux/kfd_ioctl.h>
2929
30−static void hdp_v5_0_flush_hdp(struct amdgpu_device *adev,
31− struct amdgpu_ring *ring)
32−{
33− if (!ring || !ring->funcs->emit_wreg) {
34− WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
35− RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
36− } else {
37− amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
38− }
39−}
40−
4130 static void hdp_v5_0_invalidate_hdp(struct amdgpu_device *adev,
4231 struct amdgpu_ring *ring)
4332 {
@@ -217,7 +206,7 @@ static void hdp_v5_0_init_registers(struct amdgpu_device *adev) · 173 unmodified lines
217206 }
218207
219208 const struct amdgpu_hdp_funcs hdp_v5_0_funcs = {
220− .flush_hdp = hdp_v5_0_flush_hdp,
209+ .flush_hdp = amdgpu_hdp_generic_flush,
221210 .invalidate_hdp = hdp_v5_0_invalidate_hdp,
222211 .update_clock_gating = hdp_v5_0_update_clock_gating,
223212 .get_clock_gating_state = hdp_v5_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/hdp_v5_2.c+11−1
@@ -33,7 +33,17 @@ static void hdp_v5_2_flush_hdp(struct amdgpu_device *adev, · 32 unmodified lines
3333 if (!ring || !ring->funcs->emit_wreg) {
3434 WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2,
3535 0);
36− RREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
36+ if (amdgpu_sriov_vf(adev)) {
37+ /* this is fine because SR_IOV doesn't remap the register */
38+ RREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
39+ } else {
40+ /* We just need to read back a register to post the write.
41+ * Reading back the remapped register causes problems on
42+ * some platforms so just read back the memory size register.
43+ */
44+ if (adev->nbio.funcs->get_memsize)
45+ adev->nbio.funcs->get_memsize(adev);
46+ }
3747 } else {
3848 amdgpu_ring_emit_wreg(ring,
3949 (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2,
drivers/gpu/drm/amd/amdgpu/hdp_v6_0.c+1−12
@@ -30,17 +30,6 @@ · 29 unmodified lines
3030 #define regHDP_CLK_CNTL_V6_1 0xd5
3131 #define regHDP_CLK_CNTL_V6_1_BASE_IDX 0
3232
33−static void hdp_v6_0_flush_hdp(struct amdgpu_device *adev,
34− struct amdgpu_ring *ring)
35−{
36− if (!ring || !ring->funcs->emit_wreg) {
37− WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
38− RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
39− } else {
40− amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
41− }
42−}
43−
4433 static void hdp_v6_0_update_clock_gating(struct amdgpu_device *adev,
4534 bool enable)
4635 {
@@ -149,7 +138,7 @@ static void hdp_v6_0_get_clockgating_state(struct amdgpu_device *adev, · 102 unmodified lines
149138 }
150139
151140 const struct amdgpu_hdp_funcs hdp_v6_0_funcs = {
152− .flush_hdp = hdp_v6_0_flush_hdp,
141+ .flush_hdp = amdgpu_hdp_generic_flush,
153142 .update_clock_gating = hdp_v6_0_update_clock_gating,
154143 .get_clock_gating_state = hdp_v6_0_get_clockgating_state,
155144 };
drivers/gpu/drm/amd/amdgpu/hdp_v7_0.c+1−12
@@ -27,17 +27,6 @@ · 26 unmodified lines
2727 #include "hdp/hdp_7_0_0_sh_mask.h"
2828 #include <uapi/linux/kfd_ioctl.h>
2929
30−static void hdp_v7_0_flush_hdp(struct amdgpu_device *adev,
31− struct amdgpu_ring *ring)
32−{
33− if (!ring || !ring->funcs->emit_wreg) {
34− WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
35− RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
36− } else {
37− amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
38− }
39−}
40−
4130 static void hdp_v7_0_update_clock_gating(struct amdgpu_device *adev,
4231 bool enable)
4332 {
@@ -137,7 +126,7 @@ static void hdp_v7_0_get_clockgating_state(struct amdgpu_device *adev, · 93 unmodified lines
137126 }
138127
139128 const struct amdgpu_hdp_funcs hdp_v7_0_funcs = {
140− .flush_hdp = hdp_v7_0_flush_hdp,
129+ .flush_hdp = amdgpu_hdp_generic_flush,
141130 .update_clock_gating = hdp_v7_0_update_clock_gating,
142131 .get_clock_gating_state = hdp_v7_0_get_clockgating_state,
143132 };
drivers/gpu/drm/amd/amdgpu/ih_v6_0.c+5−1
@@ -349,6 +349,7 @@ static int ih_v6_0_irq_init(struct amdgpu_device *adev) · 348 unmodified lines
349349 if (ret)
350350 return ret;
351351 }
352+ ih[i]->overflow = false;
352353 }
353354
354355 /* update doorbell range for ih ring 0 */
@@ -446,7 +447,10 @@ static u32 ih_v6_0_get_wptr(struct amdgpu_device *adev, · 91 unmodified lines
446447 wptr = RREG32_NO_KIQ(ih_regs->ih_rb_wptr);
447448 if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW))
448449 goto out;
449− wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);
450+ if (!amdgpu_sriov_vf(adev))
451+ wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);
452+ else
453+ ih->overflow = true;
450454
451455 /* When a ring buffer overflow happen start parsing interrupt
452456 * from the last not overwritten vector (wptr + 32). Hopefully
drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c+2−2
@@ -156,7 +156,7 @@ static int jpeg_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 155 unmodified lines
156156
157157 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {
158158 ring = &adev->jpeg.inst[i].ring_dec[j];
159− ring->use_doorbell = false;
159+ ring->use_doorbell = true;
160160 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);
161161 if (!amdgpu_sriov_vf(adev)) {
162162 ring->doorbell_index =
@@ -264,7 +264,7 @@ static int jpeg_v5_0_1_hw_init(struct amdgpu_ip_block *ip_block) · 101 unmodified lines
264264 ring = &adev->jpeg.inst[i].ring_dec[j];
265265 if (ring->use_doorbell)
266266 WREG32_SOC15_OFFSET(VCN, GET_INST(VCN, i), regVCN_JPEG_DB_CTRL,
267− (ring->pipe ? (ring->pipe - 0x15) : 0),
267+ ring->pipe,
268268 ring->doorbell_index <<
269269 VCN_JPEG_DB_CTRL__OFFSET__SHIFT |
270270 VCN_JPEG_DB_CTRL__EN_MASK);
drivers/gpu/drm/amd/amdgpu/mes_userqueue.c+355−0
@@ -0,0 +1,355 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright 2024 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+#include "amdgpu.h"
25+#include "amdgpu_gfx.h"
26+#include "mes_userqueue.h"
27+#include "amdgpu_userq_fence.h"
28+
29+#define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
30+#define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE
31+
32+static int
33+mes_userq_map_gtt_bo_to_gart(struct amdgpu_bo *bo)
34+{
35+ int ret;
36+
37+ ret = amdgpu_bo_reserve(bo, true);
38+ if (ret) {
39+ DRM_ERROR("Failed to reserve bo. ret %d\n", ret);
40+ goto err_reserve_bo_failed;
41+ }
42+
43+ ret = amdgpu_ttm_alloc_gart(&bo->tbo);
44+ if (ret) {
45+ DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
46+ goto err_map_bo_gart_failed;
47+ }
48+
49+ amdgpu_bo_unreserve(bo);
50+ bo = amdgpu_bo_ref(bo);
51+
52+ return 0;
53+
54+err_map_bo_gart_failed:
55+ amdgpu_bo_unreserve(bo);
56+err_reserve_bo_failed:
57+ return ret;
58+}
59+
60+static int
61+mes_userq_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr,
62+ struct amdgpu_usermode_queue *queue,
63+ uint64_t wptr)
64+{
65+ struct amdgpu_bo_va_mapping *wptr_mapping;
66+ struct amdgpu_vm *wptr_vm;
67+ struct amdgpu_userq_obj *wptr_obj = &queue->wptr_obj;
68+ int ret;
69+
70+ wptr_vm = queue->vm;
71+ ret = amdgpu_bo_reserve(wptr_vm->root.bo, false);
72+ if (ret)
73+ return ret;
74+
75+ wptr &= AMDGPU_GMC_HOLE_MASK;
76+ wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> PAGE_SHIFT);
77+ amdgpu_bo_unreserve(wptr_vm->root.bo);
78+ if (!wptr_mapping) {
79+ DRM_ERROR("Failed to lookup wptr bo\n");
80+ return -EINVAL;
81+ }
82+
83+ wptr_obj->obj = wptr_mapping->bo_va->base.bo;
84+ if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) {
85+ DRM_ERROR("Requested GART mapping for wptr bo larger than one page\n");
86+ return -EINVAL;
87+ }
88+
89+ ret = mes_userq_map_gtt_bo_to_gart(wptr_obj->obj);
90+ if (ret) {
91+ DRM_ERROR("Failed to map wptr bo to GART\n");
92+ return ret;
93+ }
94+
95+ queue->wptr_obj.gpu_addr = amdgpu_bo_gpu_offset_no_check(wptr_obj->obj);
96+ return 0;
97+}
98+
99+static int convert_to_mes_priority(int priority)
100+{
101+ switch (priority) {
102+ case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_NORMAL_LOW:
103+ default:
104+ return AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
105+ case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_LOW:
106+ return AMDGPU_MES_PRIORITY_LEVEL_LOW;
107+ case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_NORMAL_HIGH:
108+ return AMDGPU_MES_PRIORITY_LEVEL_MEDIUM;
109+ case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_HIGH:
110+ return AMDGPU_MES_PRIORITY_LEVEL_HIGH;
111+ }
112+}
113+
114+static int mes_userq_map(struct amdgpu_userq_mgr *uq_mgr,
115+ struct amdgpu_usermode_queue *queue)
116+{
117+ struct amdgpu_device *adev = uq_mgr->adev;
118+ struct amdgpu_userq_obj *ctx = &queue->fw_obj;
119+ struct amdgpu_mqd_prop *userq_props = queue->userq_prop;
120+ struct mes_add_queue_input queue_input;
121+ int r;
122+
123+ memset(&queue_input, 0x0, sizeof(struct mes_add_queue_input));
124+
125+ queue_input.process_va_start = 0;
126+ queue_input.process_va_end = adev->vm_manager.max_pfn - 1;
127+
128+ /* set process quantum to 10 ms and gang quantum to 1 ms as default */
129+ queue_input.process_quantum = 100000;
130+ queue_input.gang_quantum = 10000;
131+ queue_input.paging = false;
132+
133+ queue_input.process_context_addr = ctx->gpu_addr;
134+ queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;
135+ queue_input.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
136+ queue_input.gang_global_priority_level = convert_to_mes_priority(queue->priority);
137+
138+ queue_input.process_id = queue->vm->pasid;
139+ queue_input.queue_type = queue->queue_type;
140+ queue_input.mqd_addr = queue->mqd.gpu_addr;
141+ queue_input.wptr_addr = userq_props->wptr_gpu_addr;
142+ queue_input.queue_size = userq_props->queue_size >> 2;
143+ queue_input.doorbell_offset = userq_props->doorbell_index;
144+ queue_input.page_table_base_addr = amdgpu_gmc_pd_addr(queue->vm->root.bo);
145+ queue_input.wptr_mc_addr = queue->wptr_obj.gpu_addr;
146+
147+ amdgpu_mes_lock(&adev->mes);
148+ r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input);
149+ amdgpu_mes_unlock(&adev->mes);
150+ if (r) {
151+ DRM_ERROR("Failed to map queue in HW, err (%d)\n", r);
152+ return r;
153+ }
154+
155+ DRM_DEBUG_DRIVER("Queue (doorbell:%d) mapped successfully\n", userq_props->doorbell_index);
156+ return 0;
157+}
158+
159+static int mes_userq_unmap(struct amdgpu_userq_mgr *uq_mgr,
160+ struct amdgpu_usermode_queue *queue)
161+{
162+ struct amdgpu_device *adev = uq_mgr->adev;
163+ struct mes_remove_queue_input queue_input;
164+ struct amdgpu_userq_obj *ctx = &queue->fw_obj;
165+ int r;
166+
167+ memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));
168+ queue_input.doorbell_offset = queue->doorbell_index;
169+ queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;
170+
171+ amdgpu_mes_lock(&adev->mes);
172+ r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);
173+ amdgpu_mes_unlock(&adev->mes);
174+ if (r)
175+ DRM_ERROR("Failed to unmap queue in HW, err (%d)\n", r);
176+ return r;
177+}
178+
179+static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr,
180+ struct amdgpu_usermode_queue *queue,
181+ struct drm_amdgpu_userq_in *mqd_user)
182+{
183+ struct amdgpu_userq_obj *ctx = &queue->fw_obj;
184+ int r, size;
185+
186+ /*
187+ * The FW expects at least one page space allocated for
188+ * process ctx and gang ctx each. Create an object
189+ * for the same.
190+ */
191+ size = AMDGPU_USERQ_PROC_CTX_SZ + AMDGPU_USERQ_GANG_CTX_SZ;
192+ r = amdgpu_userq_create_object(uq_mgr, ctx, size);
193+ if (r) {
194+ DRM_ERROR("Failed to allocate ctx space bo for userqueue, err:%d\n", r);
195+ return r;
196+ }
197+
198+ return 0;
199+}
200+
201+static int mes_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
202+ struct drm_amdgpu_userq_in *args_in,
203+ struct amdgpu_usermode_queue *queue)
204+{
205+ struct amdgpu_device *adev = uq_mgr->adev;
206+ struct amdgpu_mqd *mqd_hw_default = &adev->mqds[queue->queue_type];
207+ struct drm_amdgpu_userq_in *mqd_user = args_in;
208+ struct amdgpu_mqd_prop *userq_props;
209+ int r;
210+
211+ /* Structure to initialize MQD for userqueue using generic MQD init function */
212+ userq_props = kzalloc(sizeof(struct amdgpu_mqd_prop), GFP_KERNEL);
213+ if (!userq_props) {
214+ DRM_ERROR("Failed to allocate memory for userq_props\n");
215+ return -ENOMEM;
216+ }
217+
218+ if (!mqd_user->wptr_va || !mqd_user->rptr_va ||
219+ !mqd_user->queue_va || mqd_user->queue_size == 0) {
220+ DRM_ERROR("Invalid MQD parameters for userqueue\n");
221+ r = -EINVAL;
222+ goto free_props;
223+ }
224+
225+ r = amdgpu_userq_create_object(uq_mgr, &queue->mqd, mqd_hw_default->mqd_size);
226+ if (r) {
227+ DRM_ERROR("Failed to create MQD object for userqueue\n");
228+ goto free_props;
229+ }
230+
231+ /* Initialize the MQD BO with user given values */
232+ userq_props->wptr_gpu_addr = mqd_user->wptr_va;
233+ userq_props->rptr_gpu_addr = mqd_user->rptr_va;
234+ userq_props->queue_size = mqd_user->queue_size;
235+ userq_props->hqd_base_gpu_addr = mqd_user->queue_va;
236+ userq_props->mqd_gpu_addr = queue->mqd.gpu_addr;
237+ userq_props->use_doorbell = true;
238+ userq_props->doorbell_index = queue->doorbell_index;
239+ userq_props->fence_address = queue->fence_drv->gpu_addr;
240+
241+ if (queue->queue_type == AMDGPU_HW_IP_COMPUTE) {
242+ struct drm_amdgpu_userq_mqd_compute_gfx11 *compute_mqd;
243+
244+ if (mqd_user->mqd_size != sizeof(*compute_mqd)) {
245+ DRM_ERROR("Invalid compute IP MQD size\n");
246+ r = -EINVAL;
247+ goto free_mqd;
248+ }
249+
250+ compute_mqd = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
251+ if (IS_ERR(compute_mqd)) {
252+ DRM_ERROR("Failed to read user MQD\n");
253+ r = -ENOMEM;
254+ goto free_mqd;
255+ }
256+
257+ userq_props->eop_gpu_addr = compute_mqd->eop_va;
258+ userq_props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
259+ userq_props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
260+ userq_props->hqd_active = false;
261+ userq_props->tmz_queue =
262+ mqd_user->flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE;
263+ kfree(compute_mqd);
264+ } else if (queue->queue_type == AMDGPU_HW_IP_GFX) {
265+ struct drm_amdgpu_userq_mqd_gfx11 *mqd_gfx_v11;
266+
267+ if (mqd_user->mqd_size != sizeof(*mqd_gfx_v11) || !mqd_user->mqd) {
268+ DRM_ERROR("Invalid GFX MQD\n");
269+ r = -EINVAL;
270+ goto free_mqd;
271+ }
272+
273+ mqd_gfx_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
274+ if (IS_ERR(mqd_gfx_v11)) {
275+ DRM_ERROR("Failed to read user MQD\n");
276+ r = -ENOMEM;
277+ goto free_mqd;
278+ }
279+
280+ userq_props->shadow_addr = mqd_gfx_v11->shadow_va;
281+ userq_props->csa_addr = mqd_gfx_v11->csa_va;
282+ userq_props->tmz_queue =
283+ mqd_user->flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE;
284+ kfree(mqd_gfx_v11);
285+ } else if (queue->queue_type == AMDGPU_HW_IP_DMA) {
286+ struct drm_amdgpu_userq_mqd_sdma_gfx11 *mqd_sdma_v11;
287+
288+ if (mqd_user->mqd_size != sizeof(*mqd_sdma_v11) || !mqd_user->mqd) {
289+ DRM_ERROR("Invalid SDMA MQD\n");
290+ r = -EINVAL;
291+ goto free_mqd;
292+ }
293+
294+ mqd_sdma_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
295+ if (IS_ERR(mqd_sdma_v11)) {
296+ DRM_ERROR("Failed to read sdma user MQD\n");
297+ r = -ENOMEM;
298+ goto free_mqd;
299+ }
300+
301+ userq_props->csa_addr = mqd_sdma_v11->csa_va;
302+ kfree(mqd_sdma_v11);
303+ }
304+
305+ queue->userq_prop = userq_props;
306+
307+ r = mqd_hw_default->init_mqd(adev, (void *)queue->mqd.cpu_ptr, userq_props);
308+ if (r) {
309+ DRM_ERROR("Failed to initialize MQD for userqueue\n");
310+ goto free_mqd;
311+ }
312+
313+ /* Create BO for FW operations */
314+ r = mes_userq_create_ctx_space(uq_mgr, queue, mqd_user);
315+ if (r) {
316+ DRM_ERROR("Failed to allocate BO for userqueue (%d)", r);
317+ goto free_mqd;
318+ }
319+
320+ /* FW expects WPTR BOs to be mapped into GART */
321+ r = mes_userq_create_wptr_mapping(uq_mgr, queue, userq_props->wptr_gpu_addr);
322+ if (r) {
323+ DRM_ERROR("Failed to create WPTR mapping\n");
324+ goto free_ctx;
325+ }
326+
327+ return 0;
328+
329+free_ctx:
330+ amdgpu_userq_destroy_object(uq_mgr, &queue->fw_obj);
331+
332+free_mqd:
333+ amdgpu_userq_destroy_object(uq_mgr, &queue->mqd);
334+
335+free_props:
336+ kfree(userq_props);
337+
338+ return r;
339+}
340+
341+static void
342+mes_userq_mqd_destroy(struct amdgpu_userq_mgr *uq_mgr,
343+ struct amdgpu_usermode_queue *queue)
344+{
345+ amdgpu_userq_destroy_object(uq_mgr, &queue->fw_obj);
346+ kfree(queue->userq_prop);
347+ amdgpu_userq_destroy_object(uq_mgr, &queue->mqd);
348+}
349+
350+const struct amdgpu_userq_funcs userq_mes_funcs = {
351+ .mqd_create = mes_userq_mqd_create,
352+ .mqd_destroy = mes_userq_mqd_destroy,
353+ .unmap = mes_userq_unmap,
354+ .map = mes_userq_map,
355+};
drivers/gpu/drm/amd/amdgpu/mes_userqueue.h+30−0
@@ -0,0 +1,30 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright 2024 Advanced Micro Devices, Inc.
4+ *
5+ * Permission is hereby granted, free of charge, to any person obtaining a
6+ * copy of this software and associated documentation files (the "Software"),
7+ * to deal in the Software without restriction, including without limitation
8+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9+ * and/or sell copies of the Software, and to permit persons to whom the
10+ * Software is furnished to do so, subject to the following conditions:
11+ *
12+ * The above copyright notice and this permission notice shall be included in
13+ * all copies or substantial portions of the Software.
14+ *
15+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21+ * OTHER DEALINGS IN THE SOFTWARE.
22+ *
23+ */
24+
25+#ifndef MES_USERQ_H
26+#define MES_USERQ_H
27+#include "amdgpu_userq.h"
28+
29+extern const struct amdgpu_userq_funcs userq_mes_funcs;
30+#endif
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c+27−47
@@ -287,6 +287,23 @@ static int convert_to_mes_queue_type(int queue_type) · 286 unmodified lines
287287 return -1;
288288 }
289289
290+static int convert_to_mes_priority_level(int priority_level)
291+{
292+ switch (priority_level) {
293+ case AMDGPU_MES_PRIORITY_LEVEL_LOW:
294+ return AMD_PRIORITY_LEVEL_LOW;
295+ case AMDGPU_MES_PRIORITY_LEVEL_NORMAL:
296+ default:
297+ return AMD_PRIORITY_LEVEL_NORMAL;
298+ case AMDGPU_MES_PRIORITY_LEVEL_MEDIUM:
299+ return AMD_PRIORITY_LEVEL_MEDIUM;
300+ case AMDGPU_MES_PRIORITY_LEVEL_HIGH:
301+ return AMD_PRIORITY_LEVEL_HIGH;
302+ case AMDGPU_MES_PRIORITY_LEVEL_REALTIME:
303+ return AMD_PRIORITY_LEVEL_REALTIME;
304+ }
305+}
306+
290307 static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes,
291308 struct mes_add_queue_input *input)
292309 {
@@ -310,9 +327,9 @@ static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines
310327 mes_add_queue_pkt.gang_quantum = input->gang_quantum;
311328 mes_add_queue_pkt.gang_context_addr = input->gang_context_addr;
312329 mes_add_queue_pkt.inprocess_gang_priority =
313− input->inprocess_gang_priority;
330+ convert_to_mes_priority_level(input->inprocess_gang_priority);
314331 mes_add_queue_pkt.gang_global_priority_level =
315− input->gang_global_priority_level;
332+ convert_to_mes_priority_level(input->gang_global_priority_level);
316333 mes_add_queue_pkt.doorbell_offset = input->doorbell_offset;
317334 mes_add_queue_pkt.mqd_addr = input->mqd_addr;
318335
@@ -458,31 +475,6 @@ static int mes_v11_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 139 unmodified lines
458475 return r;
459476 }
460477
461−static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes,
462− struct mes_reset_queue_input *input)
463−{
464− if (input->use_mmio)
465− return mes_v11_0_reset_queue_mmio(mes, input->queue_type,
466− input->me_id, input->pipe_id,
467− input->queue_id, input->vmid);
468−
469− union MESAPI__RESET mes_reset_queue_pkt;
470−
471− memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));
472−
473− mes_reset_queue_pkt.header.type = MES_API_TYPE_SCHEDULER;
474− mes_reset_queue_pkt.header.opcode = MES_SCH_API_RESET;
475− mes_reset_queue_pkt.header.dwsize = API_FRAME_SIZE_IN_DWORDS;
476−
477− mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
478− mes_reset_queue_pkt.gang_context_addr = input->gang_context_addr;
479− /*mes_reset_queue_pkt.reset_queue_only = 1;*/
480−
481− return mes_v11_0_submit_pkt_and_poll_completion(mes,
482− &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt),
483− offsetof(union MESAPI__REMOVE_QUEUE, api_status));
484−}
485−
486478 static int mes_v11_0_map_legacy_queue(struct amdgpu_mes *mes,
487479 struct mes_map_legacy_queue_input *input)
488480 {
@@ -649,7 +641,7 @@ static int mes_v11_0_misc_op(struct amdgpu_mes *mes, · 160 unmodified lines
649641 break;
650642 case MES_MISC_OP_CHANGE_CONFIG:
651643 if ((mes->adev->mes.sched_version & AMDGPU_MES_VERSION_MASK) < 0x63) {
652− dev_err(mes->adev->dev, "MES FW versoin must be larger than 0x63 to support limit single process feature.\n");
644+ dev_err(mes->adev->dev, "MES FW version must be larger than 0x63 to support limit single process feature.\n");
653645 return -EINVAL;
654646 }
655647 misc_pkt.opcode = MESAPI_MISC__CHANGE_CONFIG;
@@ -694,7 +686,8 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes) · 38 unmodified lines
694686 mes->compute_hqd_mask[i];
695687
696688 for (i = 0; i < MAX_GFX_PIPES; i++)
697− mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];
689+ mes_set_hw_res_pkt.gfx_hqd_mask[i] =
690+ mes->gfx_hqd_mask[i];
698691
699692 for (i = 0; i < MAX_SDMA_PIPES; i++)
700693 mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
@@ -723,7 +716,7 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes) · 22 unmodified lines
723716 mes->event_log_gpu_addr;
724717 }
725718
726− if (enforce_isolation)
719+ if (adev->enforce_isolation[0] == AMDGPU_ENFORCE_ISOLATION_ENABLE)
727720 mes_set_hw_res_pkt.limit_single_process = 1;
728721
729722 return mes_v11_0_submit_pkt_and_poll_completion(mes,
@@ -753,8 +746,8 @@ static int mes_v11_0_set_hw_resources_1(struct amdgpu_mes *mes) · 23 unmodified lines
753746 offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status));
754747 }
755748
756−static int mes_v11_0_reset_legacy_queue(struct amdgpu_mes *mes,
757− struct mes_reset_legacy_queue_input *input)
749+static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes,
750+ struct mes_reset_queue_input *input)
758751 {
759752 union MESAPI__RESET mes_reset_queue_pkt;
760753
@@ -772,7 +765,7 @@ static int mes_v11_0_reset_legacy_queue(struct amdgpu_mes *mes, · 11 unmodified lines
772765 mes_reset_queue_pkt.queue_type =
773766 convert_to_mes_queue_type(input->queue_type);
774767
775− if (mes_reset_queue_pkt.queue_type == MES_QUEUE_TYPE_GFX) {
768+ if (input->legacy_gfx) {
776769 mes_reset_queue_pkt.reset_legacy_gfx = 1;
777770 mes_reset_queue_pkt.pipe_id_lp = input->pipe_id;
778771 mes_reset_queue_pkt.queue_id_lp = input->queue_id;
@@ -798,7 +791,6 @@ static const struct amdgpu_mes_funcs mes_v11_0_funcs = { · 19 unmodified lines
798791 .suspend_gang = mes_v11_0_suspend_gang,
799792 .resume_gang = mes_v11_0_resume_gang,
800793 .misc_op = mes_v11_0_misc_op,
801− .reset_legacy_queue = mes_v11_0_reset_legacy_queue,
802794 .reset_hw_queue = mes_v11_0_reset_hw_queue,
803795 };
804796
@@ -1701,22 +1693,10 @@ static int mes_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 896 unmodified lines
17011693 return 0;
17021694 }
17031695
1704−static int mes_v11_0_late_init(struct amdgpu_ip_block *ip_block)
1705−{
1706− struct amdgpu_device *adev = ip_block->adev;
1707−
1708− /* it's only intended for use in mes_self_test case, not for s0ix and reset */
1709− if (!amdgpu_in_reset(adev) && !adev->in_s0ix && !adev->in_suspend &&
1710− (amdgpu_ip_version(adev, GC_HWIP, 0) != IP_VERSION(11, 0, 3)))
1711− amdgpu_mes_self_test(adev);
1712−
1713− return 0;
1714−}
1715−
17161696 static const struct amd_ip_funcs mes_v11_0_ip_funcs = {
17171697 .name = "mes_v11_0",
17181698 .early_init = mes_v11_0_early_init,
1719− .late_init = mes_v11_0_late_init,
1699+ .late_init = NULL,
17201700 .sw_init = mes_v11_0_sw_init,
17211701 .sw_fini = mes_v11_0_sw_fini,
17221702 .hw_init = mes_v11_0_hw_init,
drivers/gpu/drm/amd/amdgpu/mes_v12_0.c+25−46
@@ -274,6 +274,23 @@ static int convert_to_mes_queue_type(int queue_type) · 273 unmodified lines
274274 return -1;
275275 }
276276
277+static int convert_to_mes_priority_level(int priority_level)
278+{
279+ switch (priority_level) {
280+ case AMDGPU_MES_PRIORITY_LEVEL_LOW:
281+ return AMD_PRIORITY_LEVEL_LOW;
282+ case AMDGPU_MES_PRIORITY_LEVEL_NORMAL:
283+ default:
284+ return AMD_PRIORITY_LEVEL_NORMAL;
285+ case AMDGPU_MES_PRIORITY_LEVEL_MEDIUM:
286+ return AMD_PRIORITY_LEVEL_MEDIUM;
287+ case AMDGPU_MES_PRIORITY_LEVEL_HIGH:
288+ return AMD_PRIORITY_LEVEL_HIGH;
289+ case AMDGPU_MES_PRIORITY_LEVEL_REALTIME:
290+ return AMD_PRIORITY_LEVEL_REALTIME;
291+ }
292+}
293+
277294 static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes,
278295 struct mes_add_queue_input *input)
279296 {
@@ -297,9 +314,9 @@ static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines
297314 mes_add_queue_pkt.gang_quantum = input->gang_quantum;
298315 mes_add_queue_pkt.gang_context_addr = input->gang_context_addr;
299316 mes_add_queue_pkt.inprocess_gang_priority =
300− input->inprocess_gang_priority;
317+ convert_to_mes_priority_level(input->inprocess_gang_priority);
301318 mes_add_queue_pkt.gang_global_priority_level =
302− input->gang_global_priority_level;
319+ convert_to_mes_priority_level(input->gang_global_priority_level);
303320 mes_add_queue_pkt.doorbell_offset = input->doorbell_offset;
304321 mes_add_queue_pkt.mqd_addr = input->mqd_addr;
305322
@@ -477,32 +494,6 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 171 unmodified lines
477494 return r;
478495 }
479496
480−static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes,
481− struct mes_reset_queue_input *input)
482−{
483− union MESAPI__RESET mes_reset_queue_pkt;
484− int pipe;
485−
486− memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));
487−
488− mes_reset_queue_pkt.header.type = MES_API_TYPE_SCHEDULER;
489− mes_reset_queue_pkt.header.opcode = MES_SCH_API_RESET;
490− mes_reset_queue_pkt.header.dwsize = API_FRAME_SIZE_IN_DWORDS;
491−
492− mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
493− mes_reset_queue_pkt.gang_context_addr = input->gang_context_addr;
494− /*mes_reset_queue_pkt.reset_queue_only = 1;*/
495−
496− if (mes->adev->enable_uni_mes)
497− pipe = AMDGPU_MES_KIQ_PIPE;
498− else
499− pipe = AMDGPU_MES_SCHED_PIPE;
500−
501− return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
502− &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt),
503− offsetof(union MESAPI__REMOVE_QUEUE, api_status));
504−}
505−
506497 static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes,
507498 struct mes_map_legacy_queue_input *input)
508499 {
@@ -762,7 +753,7 @@ static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe) · 253 unmodified lines
762753 pipe * (AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE);
763754 }
764755
765− if (enforce_isolation)
756+ if (adev->enforce_isolation[0] == AMDGPU_ENFORCE_ISOLATION_ENABLE)
766757 mes_set_hw_res_pkt.limit_single_process = 1;
767758
768759 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
@@ -845,8 +836,8 @@ static void mes_v12_0_enable_unmapped_doorbell_handling( · 76 unmodified lines
845836 WREG32_SOC15(GC, 0, regCP_UNMAPPED_DOORBELL, data);
846837 }
847838
848−static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes,
849− struct mes_reset_legacy_queue_input *input)
839+static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes,
840+ struct mes_reset_queue_input *input)
850841 {
851842 union MESAPI__RESET mes_reset_queue_pkt;
852843 int pipe;
@@ -865,7 +856,7 @@ static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, · 12 unmodified lines
865856 mes_reset_queue_pkt.queue_type =
866857 convert_to_mes_queue_type(input->queue_type);
867858
868− if (mes_reset_queue_pkt.queue_type == MES_QUEUE_TYPE_GFX) {
859+ if (input->legacy_gfx) {
869860 mes_reset_queue_pkt.reset_legacy_gfx = 1;
870861 mes_reset_queue_pkt.pipe_id_lp = input->pipe_id;
871862 mes_reset_queue_pkt.queue_id_lp = input->queue_id;
@@ -878,7 +869,7 @@ static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, · 6 unmodified lines
878869 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
879870 }
880871
881− if (mes->adev->enable_uni_mes)
872+ if (input->is_kq)
882873 pipe = AMDGPU_MES_KIQ_PIPE;
883874 else
884875 pipe = AMDGPU_MES_SCHED_PIPE;
@@ -896,7 +887,6 @@ static const struct amdgpu_mes_funcs mes_v12_0_funcs = { · 11 unmodified lines
896887 .suspend_gang = mes_v12_0_suspend_gang,
897888 .resume_gang = mes_v12_0_resume_gang,
898889 .misc_op = mes_v12_0_misc_op,
899− .reset_legacy_queue = mes_v12_0_reset_legacy_queue,
900890 .reset_hw_queue = mes_v12_0_reset_hw_queue,
901891 };
902892
@@ -1811,21 +1801,10 @@ static int mes_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 908 unmodified lines
18111801 return 0;
18121802 }
18131803
1814−static int mes_v12_0_late_init(struct amdgpu_ip_block *ip_block)
1815−{
1816− struct amdgpu_device *adev = ip_block->adev;
1817−
1818− /* it's only intended for use in mes_self_test case, not for s0ix and reset */
1819− if (!amdgpu_in_reset(adev) && !adev->in_s0ix && !adev->in_suspend)
1820− amdgpu_mes_self_test(adev);
1821−
1822− return 0;
1823−}
1824−
18251804 static const struct amd_ip_funcs mes_v12_0_ip_funcs = {
18261805 .name = "mes_v12_0",
18271806 .early_init = mes_v12_0_early_init,
1828− .late_init = mes_v12_0_late_init,
1807+ .late_init = NULL,
18291808 .sw_init = mes_v12_0_sw_init,
18301809 .sw_fini = mes_v12_0_sw_fini,
18311810 .hw_init = mes_v12_0_hw_init,
drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c+52−14
@@ -30,6 +30,7 @@ · 29 unmodified lines
3030 #include "soc15_common.h"
3131 #include "soc15.h"
3232 #include "amdgpu_ras.h"
33+#include "amdgpu_psp.h"
3334
3435 #define regVM_L2_CNTL3_DEFAULT 0x80100007
3536 #define regVM_L2_CNTL4_DEFAULT 0x000000c1
@@ -192,10 +193,8 @@ static void mmhub_v1_8_init_tlb_regs(struct amdgpu_device *adev) · 156 unmodified lines
192193 uint32_t tmp, inst_mask;
193194 int i;
194195
195− /* Setup TLB control */
196− inst_mask = adev->aid_mask;
197− for_each_inst(i, inst_mask) {
198− tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL);
196+ if (amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev)) {
197+ tmp = RREG32_SOC15(MMHUB, 0, regMC_VM_MX_L1_TLB_CNTL);
199198
200199 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
201200 1);
@@ -209,7 +208,26 @@ static void mmhub_v1_8_init_tlb_regs(struct amdgpu_device *adev) · 7 unmodified lines
209208 MTYPE, MTYPE_UC);/* XXX for emulation. */
210209 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1);
211210
212− WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp);
211+ psp_reg_program_no_ring(&adev->psp, tmp, PSP_REG_MMHUB_L1_TLB_CNTL);
212+ } else {
213+ inst_mask = adev->aid_mask;
214+ for_each_inst(i, inst_mask) {
215+ tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL);
216+
217+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
218+ 1);
219+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
220+ SYSTEM_ACCESS_MODE, 3);
221+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
222+ ENABLE_ADVANCED_DRIVER_MODEL, 1);
223+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
224+ SYSTEM_APERTURE_UNMAPPED_ACCESS, 0);
225+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
226+ MTYPE, MTYPE_UC);/* XXX for emulation. */
227+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1);
228+
229+ WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp);
230+ }
213231 }
214232 }
215233
@@ -221,6 +239,9 @@ static void mmhub_v1_8_init_snoop_override_regs(struct amdgpu_device *adev) · 5 unmodified lines
221239 uint32_t distance = regDAGB1_WRCLI_GPU_SNOOP_OVERRIDE -
222240 regDAGB0_WRCLI_GPU_SNOOP_OVERRIDE;
223241
242+ if (amdgpu_sriov_vf(adev))
243+ return;
244+
224245 inst_mask = adev->aid_mask;
225246 for_each_inst(i, inst_mask) {
226247 for (j = 0; j < 5; j++) { /* DAGB instances */
@@ -454,6 +475,30 @@ static int mmhub_v1_8_gart_enable(struct amdgpu_device *adev) · 227 unmodified lines
454475 return 0;
455476 }
456477
478+static void mmhub_v1_8_disable_l1_tlb(struct amdgpu_device *adev)
479+{
480+ u32 tmp;
481+ u32 i, inst_mask;
482+
483+ if (amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev)) {
484+ tmp = RREG32_SOC15(MMHUB, 0, regMC_VM_MX_L1_TLB_CNTL);
485+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, 0);
486+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
487+ ENABLE_ADVANCED_DRIVER_MODEL, 0);
488+ psp_reg_program_no_ring(&adev->psp, tmp, PSP_REG_MMHUB_L1_TLB_CNTL);
489+ } else {
490+ inst_mask = adev->aid_mask;
491+ for_each_inst(i, inst_mask) {
492+ tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL);
493+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
494+ 0);
495+ tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
496+ ENABLE_ADVANCED_DRIVER_MODEL, 0);
497+ WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp);
498+ }
499+ }
500+}
501+
457502 static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev)
458503 {
459504 struct amdgpu_vmhub *hub;
@@ -467,15 +512,6 @@ static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) · 7 unmodified lines
467512 for (i = 0; i < 16; i++)
468513 WREG32_SOC15_OFFSET(MMHUB, j, regVM_CONTEXT0_CNTL,
469514 i * hub->ctx_distance, 0);
470−
471− /* Setup TLB control */
472− tmp = RREG32_SOC15(MMHUB, j, regMC_VM_MX_L1_TLB_CNTL);
473− tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
474− 0);
475− tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
476− ENABLE_ADVANCED_DRIVER_MODEL, 0);
477− WREG32_SOC15(MMHUB, j, regMC_VM_MX_L1_TLB_CNTL, tmp);
478−
479515 if (!amdgpu_sriov_vf(adev)) {
480516 /* Setup L2 cache */
481517 tmp = RREG32_SOC15(MMHUB, j, regVM_L2_CNTL);
@@ -485,6 +521,8 @@ static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) · 3 unmodified lines
485521 WREG32_SOC15(MMHUB, j, regVM_L2_CNTL3, 0);
486522 }
487523 }
524+
525+ mmhub_v1_8_disable_l1_tlb(adev);
488526 }
489527
490528 /**
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c+46−17
@@ -274,6 +274,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 273 unmodified lines
274274 {
275275 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work);
276276 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
277+ struct amdgpu_reset_context reset_context = { 0 };
277278
278279 amdgpu_virt_fini_data_exchange(adev);
279280
@@ -281,8 +282,6 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 1 unmodified line
281282 if (amdgpu_device_should_recover_gpu(adev)
282283 && (!amdgpu_device_has_job_running(adev) ||
283284 adev->sdma_timeout == MAX_SCHEDULE_TIMEOUT)) {
284− struct amdgpu_reset_context reset_context;
285− memset(&reset_context, 0, sizeof(reset_context));
286285
287286 reset_context.method = AMD_RESET_METHOD_NONE;
288287 reset_context.reset_req_dev = adev;
@@ -293,6 +292,19 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines
293292 }
294293 }
295294
295+static void xgpu_ai_mailbox_bad_pages_work(struct work_struct *work)
296+{
297+ struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, bad_pages_work);
298+ struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
299+
300+ if (down_read_trylock(&adev->reset_domain->sem)) {
301+ amdgpu_virt_fini_data_exchange(adev);
302+ amdgpu_virt_request_bad_pages(adev);
303+ amdgpu_virt_init_data_exchange(adev);
304+ up_read(&adev->reset_domain->sem);
305+ }
306+}
307+
296308 static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
297309 struct amdgpu_irq_src *src,
298310 unsigned type,
@@ -312,26 +324,42 @@ static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev, · 13 unmodified lines
312324 struct amdgpu_iv_entry *entry)
313325 {
314326 enum idh_event event = xgpu_ai_mailbox_peek_msg(adev);
327+ struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
315328
316329 switch (event) {
317− case IDH_FLR_NOTIFICATION:
330+ case IDH_RAS_BAD_PAGES_NOTIFICATION:
331+ xgpu_ai_mailbox_send_ack(adev);
332+ if (amdgpu_sriov_runtime(adev))
333+ schedule_work(&adev->virt.bad_pages_work);
334+ break;
335+ case IDH_UNRECOV_ERR_NOTIFICATION:
336+ xgpu_ai_mailbox_send_ack(adev);
337+ ras->is_rma = true;
338+ dev_err(adev->dev, "VF is in an unrecoverable state. Runtime Services are halted.\n");
318339 if (amdgpu_sriov_runtime(adev))
319340 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
320− &adev->virt.flr_work),
321− "Failed to queue work! at %s",
322− __func__);
341+ &adev->virt.flr_work),
342+ "Failed to queue work! at %s",
343+ __func__);
323344 break;
324− case IDH_QUERY_ALIVE:
325− xgpu_ai_mailbox_send_ack(adev);
326− break;
327− /* READY_TO_ACCESS_GPU is fetched by kernel polling, IRQ can ignore
328− * it byfar since that polling thread will handle it,
329− * other msg like flr complete is not handled here.
330− */
331− case IDH_CLR_MSG_BUF:
332− case IDH_FLR_NOTIFICATION_CMPL:
333− case IDH_READY_TO_ACCESS_GPU:
334− default:
345+ case IDH_FLR_NOTIFICATION:
346+ if (amdgpu_sriov_runtime(adev))
347+ WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
348+ &adev->virt.flr_work),
349+ "Failed to queue work! at %s",
350+ __func__);
351+ break;
352+ case IDH_QUERY_ALIVE:
353+ xgpu_ai_mailbox_send_ack(adev);
354+ break;
355+ /* READY_TO_ACCESS_GPU is fetched by kernel polling, IRQ can ignore
356+ * it byfar since that polling thread will handle it,
357+ * other msg like flr complete is not handled here.
358+ */
359+ case IDH_CLR_MSG_BUF:
360+ case IDH_FLR_NOTIFICATION_CMPL:
361+ case IDH_READY_TO_ACCESS_GPU:
362+ default:
335363 break;
336364 }
337365
@@ -387,6 +415,7 @@ int xgpu_ai_mailbox_get_irq(struct amdgpu_device *adev) · 49 unmodified lines
387415 }
388416
389417 INIT_WORK(&adev->virt.flr_work, xgpu_ai_mailbox_flr_work);
418+ INIT_WORK(&adev->virt.bad_pages_work, xgpu_ai_mailbox_bad_pages_work);
390419
391420 return 0;
392421 }
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.h+4−0
@@ -40,6 +40,7 @@ enum idh_request { · 39 unmodified lines
4040 IDH_LOG_VF_ERROR = 200,
4141 IDH_READY_TO_RESET = 201,
4242 IDH_RAS_POISON = 202,
43+ IDH_REQ_RAS_BAD_PAGES = 205,
4344 };
4445
4546 enum idh_event {
@@ -54,6 +55,9 @@ enum idh_event { · 8 unmodified lines
5455 IDH_RAS_POISON_READY,
5556 IDH_PF_SOFT_FLR_NOTIFICATION,
5657 IDH_RAS_ERROR_DETECTED,
58+ IDH_RAS_BAD_PAGES_READY = 15,
59+ IDH_RAS_BAD_PAGES_NOTIFICATION = 16,
60+ IDH_UNRECOV_ERR_NOTIFICATION = 17,
5761 IDH_TEXT_MESSAGE = 255,
5862 };
5963
drivers/gpu/drm/amd/amdgpu/mxgpu_nv.c+59−3
@@ -67,6 +67,8 @@ static int xgpu_nv_mailbox_rcv_msg(struct amdgpu_device *adev, · 66 unmodified lines
6767 reg = RREG32_NO_KIQ(mmMAILBOX_MSGBUF_RCV_DW0);
6868 if (reg == IDH_FAIL)
6969 r = -EINVAL;
70+ if (reg == IDH_UNRECOV_ERR_NOTIFICATION)
71+ r = -ENODEV;
7072 else if (reg != event)
7173 return -ENOENT;
7274
@@ -103,6 +105,7 @@ static int xgpu_nv_poll_msg(struct amdgpu_device *adev, enum idh_event event) · 30 unmodified lines
103105 {
104106 int r;
105107 uint64_t timeout, now;
108+ struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
106109
107110 now = (uint64_t)ktime_to_ms(ktime_get());
108111 timeout = now + NV_MAILBOX_POLL_MSG_TIMEDOUT;
@@ -110,8 +113,16 @@ static int xgpu_nv_poll_msg(struct amdgpu_device *adev, enum idh_event event) · 1 unmodified line
110113 do {
111114 r = xgpu_nv_mailbox_rcv_msg(adev, event);
112115 if (!r) {
113− dev_dbg(adev->dev, "rcv_msg 0x%x after %llu ms\n", event, NV_MAILBOX_POLL_MSG_TIMEDOUT - timeout + now);
116+ dev_dbg(adev->dev, "rcv_msg 0x%x after %llu ms\n",
117+ event, NV_MAILBOX_POLL_MSG_TIMEDOUT - timeout + now);
114118 return 0;
119+ } else if (r == -ENODEV) {
120+ if (!amdgpu_ras_is_rma(adev)) {
121+ ras->is_rma = true;
122+ dev_err(adev->dev, "VF is in an unrecoverable state. "
123+ "Runtime Services are halted.\n");
124+ }
125+ return r;
115126 }
116127
117128 msleep(10);
@@ -166,6 +177,10 @@ static int xgpu_nv_send_access_requests_with_param(struct amdgpu_device *adev, · 48 unmodified lines
166177 enum idh_event event = -1;
167178
168179 send_request:
180+
181+ if (amdgpu_ras_is_rma(adev))
182+ return -ENODEV;
183+
169184 xgpu_nv_mailbox_trans_msg(adev, req, data1, data2, data3);
170185
171186 switch (req) {
@@ -187,6 +202,9 @@ send_request: · 15 unmodified lines
187202 case IDH_REQ_RAS_CPER_DUMP:
188203 event = IDH_RAS_CPER_DUMP_READY;
189204 break;
205+ case IDH_REQ_RAS_BAD_PAGES:
206+ event = IDH_RAS_BAD_PAGES_READY;
207+ break;
190208 default:
191209 break;
192210 }
@@ -320,6 +338,7 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 127 unmodified lines
320338 {
321339 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work);
322340 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
341+ struct amdgpu_reset_context reset_context = { 0 };
323342
324343 amdgpu_virt_fini_data_exchange(adev);
325344
@@ -330,8 +349,6 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines
330349 adev->gfx_timeout == MAX_SCHEDULE_TIMEOUT ||
331350 adev->compute_timeout == MAX_SCHEDULE_TIMEOUT ||
332351 adev->video_timeout == MAX_SCHEDULE_TIMEOUT)) {
333− struct amdgpu_reset_context reset_context;
334− memset(&reset_context, 0, sizeof(reset_context));
335352
336353 reset_context.method = AMD_RESET_METHOD_NONE;
337354 reset_context.reset_req_dev = adev;
@@ -342,6 +359,19 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines
342359 }
343360 }
344361
362+static void xgpu_nv_mailbox_bad_pages_work(struct work_struct *work)
363+{
364+ struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, bad_pages_work);
365+ struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
366+
367+ if (down_read_trylock(&adev->reset_domain->sem)) {
368+ amdgpu_virt_fini_data_exchange(adev);
369+ amdgpu_virt_request_bad_pages(adev);
370+ amdgpu_virt_init_data_exchange(adev);
371+ up_read(&adev->reset_domain->sem);
372+ }
373+}
374+
345375 static int xgpu_nv_set_mailbox_rcv_irq(struct amdgpu_device *adev,
346376 struct amdgpu_irq_src *src,
347377 unsigned type,
@@ -364,8 +394,27 @@ static int xgpu_nv_mailbox_rcv_irq(struct amdgpu_device *adev, · 16 unmodified lines
364394 struct amdgpu_iv_entry *entry)
365395 {
366396 enum idh_event event = xgpu_nv_mailbox_peek_msg(adev);
397+ struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
367398
368399 switch (event) {
400+ case IDH_RAS_BAD_PAGES_NOTIFICATION:
401+ xgpu_nv_mailbox_send_ack(adev);
402+ if (amdgpu_sriov_runtime(adev))
403+ schedule_work(&adev->virt.bad_pages_work);
404+ break;
405+ case IDH_UNRECOV_ERR_NOTIFICATION:
406+ xgpu_nv_mailbox_send_ack(adev);
407+ if (!amdgpu_ras_is_rma(adev)) {
408+ ras->is_rma = true;
409+ dev_err(adev->dev, "VF is in an unrecoverable state. Runtime Services are halted.\n");
410+ }
411+
412+ if (amdgpu_sriov_runtime(adev))
413+ WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
414+ &adev->virt.flr_work),
415+ "Failed to queue work! at %s",
416+ __func__);
417+ break;
369418 case IDH_FLR_NOTIFICATION:
370419 if (amdgpu_sriov_runtime(adev))
371420 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
@@ -436,6 +485,7 @@ int xgpu_nv_mailbox_get_irq(struct amdgpu_device *adev) · 64 unmodified lines
436485 }
437486
438487 INIT_WORK(&adev->virt.flr_work, xgpu_nv_mailbox_flr_work);
488+ INIT_WORK(&adev->virt.bad_pages_work, xgpu_nv_mailbox_bad_pages_work);
439489
440490 return 0;
441491 }
@@ -480,6 +530,11 @@ static int xgpu_nv_req_ras_cper_dump(struct amdgpu_device *adev, u64 vf_rptr) · 38 unmodified lines
480530 adev, IDH_REQ_RAS_CPER_DUMP, vf_rptr_hi, vf_rptr_lo, 0);
481531 }
482532
533+static int xgpu_nv_req_ras_bad_pages(struct amdgpu_device *adev)
534+{
535+ return xgpu_nv_send_access_requests(adev, IDH_REQ_RAS_BAD_PAGES);
536+}
537+
483538 const struct amdgpu_virt_ops xgpu_nv_virt_ops = {
484539 .req_full_gpu = xgpu_nv_request_full_gpu_access,
485540 .rel_full_gpu = xgpu_nv_release_full_gpu_access,
@@ -492,4 +547,5 @@ const struct amdgpu_virt_ops xgpu_nv_virt_ops = { · 6 unmodified lines
492547 .rcvd_ras_intr = xgpu_nv_rcvd_ras_intr,
493548 .req_ras_err_count = xgpu_nv_req_ras_err_count,
494549 .req_ras_cper_dump = xgpu_nv_req_ras_cper_dump,
550+ .req_bad_pages = xgpu_nv_req_ras_bad_pages,
495551 };
drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h+4−0
@@ -42,6 +42,7 @@ enum idh_request { · 41 unmodified lines
4242 IDH_RAS_POISON = 202,
4343 IDH_REQ_RAS_ERROR_COUNT = 203,
4444 IDH_REQ_RAS_CPER_DUMP = 204,
45+ IDH_REQ_RAS_BAD_PAGES = 205,
4546 };
4647
4748 enum idh_event {
@@ -58,6 +59,9 @@ enum idh_event { · 10 unmodified lines
5859 IDH_RAS_ERROR_DETECTED,
5960 IDH_RAS_ERROR_COUNT_READY = 11,
6061 IDH_RAS_CPER_DUMP_READY = 14,
62+ IDH_RAS_BAD_PAGES_READY = 15,
63+ IDH_RAS_BAD_PAGES_NOTIFICATION = 16,
64+ IDH_UNRECOV_ERR_NOTIFICATION = 17,
6165
6266 IDH_TEXT_MESSAGE = 255,
6367 };
drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c+6−2
@@ -177,8 +177,12 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 176 unmodified lines
177177 {
178178 u32 doorbell_range = 0, doorbell_ctrl = 0;
179179 u32 aid_id = instance;
180+ u32 range_size;
180181
181182 if (use_doorbell) {
183+ range_size = (amdgpu_ip_version(adev, GC_HWIP, 0) ==
184+ IP_VERSION(9, 5, 0)) ?
185+ 0xb : 0x9;
182186 doorbell_range = REG_SET_FIELD(doorbell_range,
183187 DOORBELL0_CTRL_ENTRY_0,
184188 BIF_DOORBELL0_RANGE_OFFSET_ENTRY,
@@ -186,7 +190,7 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 1 unmodified line
186190 doorbell_range = REG_SET_FIELD(doorbell_range,
187191 DOORBELL0_CTRL_ENTRY_0,
188192 BIF_DOORBELL0_RANGE_SIZE_ENTRY,
189− 0x9);
193+ range_size);
190194 if (aid_id)
191195 doorbell_range = REG_SET_FIELD(doorbell_range,
192196 DOORBELL0_CTRL_ENTRY_0,
@@ -204,7 +208,7 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 11 unmodified lines
204208 S2A_DOORBELL_PORT1_RANGE_OFFSET, 0x4);
205209 doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl,
206210 S2A_DOORBELL_ENTRY_1_CTRL,
207− S2A_DOORBELL_PORT1_RANGE_SIZE, 0x9);
211+ S2A_DOORBELL_PORT1_RANGE_SIZE, range_size);
208212 doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl,
209213 S2A_DOORBELL_ENTRY_1_CTRL,
210214 S2A_DOORBELL_PORT1_AWADDR_31_28_VALUE, 0x4);
drivers/gpu/drm/amd/amdgpu/psp_v12_0.c+17−85
@@ -34,9 +34,6 @@ · 33 unmodified lines
3434 #include "sdma0/sdma0_4_0_offset.h"
3535 #include "nbio/nbio_7_4_offset.h"
3636
37−#include "oss/osssys_4_0_offset.h"
38−#include "oss/osssys_4_0_sh_mask.h"
39−
4037 MODULE_FIRMWARE("amdgpu/renoir_asd.bin");
4138 MODULE_FIRMWARE("amdgpu/renoir_ta.bin");
4239 MODULE_FIRMWARE("amdgpu/green_sardine_asd.bin");
@@ -99,9 +96,6 @@ static int psp_v12_0_bootloader_load_sysdrv(struct psp_context *psp) · 56 unmodified lines
9996 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35,
10097 psp_gfxdrv_command_reg);
10198
102− /* there might be handshake issue with hardware which needs delay */
103− mdelay(20);
104−
10599 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_35),
106100 0x80000000, 0x80000000, false);
107101
@@ -138,8 +132,6 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) · 30 unmodified lines
138132 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35,
139133 psp_gfxdrv_command_reg);
140134
141− /* there might be handshake issue with hardware which needs delay */
142− mdelay(20);
143135 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_81),
144136 RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_81),
145137 0, true);
@@ -147,37 +139,6 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) · 1 unmodified line
147139 return ret;
148140 }
149141
150−static void psp_v12_0_reroute_ih(struct psp_context *psp)
151−{
152− struct amdgpu_device *adev = psp->adev;
153− uint32_t tmp;
154−
155− /* Change IH ring for VMC */
156− tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1244b);
157− tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, CLIENT_TYPE, 1);
158− tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
159−
160− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 3);
161− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
162− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
163−
164− mdelay(20);
165− psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
166− 0x80000000, 0x8000FFFF, false);
167−
168− /* Change IH ring for UMC */
169− tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1216b);
170− tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
171−
172− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 4);
173− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
174− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
175−
176− mdelay(20);
177− psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
178− 0x80000000, 0x8000FFFF, false);
179−}
180−
181142 static int psp_v12_0_ring_create(struct psp_context *psp,
182143 enum psp_ring_type ring_type)
183144 {
@@ -186,49 +147,23 @@ static int psp_v12_0_ring_create(struct psp_context *psp, · 2 unmodified lines
186147 struct psp_ring *ring = &psp->km_ring;
187148 struct amdgpu_device *adev = psp->adev;
188149
189− psp_v12_0_reroute_ih(psp);
190−
191− if (amdgpu_sriov_vf(psp->adev)) {
192− /* Write low address of the ring to C2PMSG_102 */
193− psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);
194− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_102, psp_ring_reg);
195− /* Write high address of the ring to C2PMSG_103 */
196− psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr);
197− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_103, psp_ring_reg);
198−
199− /* Write the ring initialization command to C2PMSG_101 */
200− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_101,
201− GFX_CTRL_CMD_ID_INIT_GPCOM_RING);
202−
203− /* there might be handshake issue with hardware which needs delay */
204− mdelay(20);
205−
206− /* Wait for response flag (bit 31) in C2PMSG_101 */
207− ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101),
208− 0x80000000, 0x8000FFFF, false);
209−
210− } else {
211− /* Write low address of the ring to C2PMSG_69 */
212− psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);
213− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg);
214− /* Write high address of the ring to C2PMSG_70 */
215− psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr);
216− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, psp_ring_reg);
217− /* Write size of ring to C2PMSG_71 */
218− psp_ring_reg = ring->ring_size;
219− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_71, psp_ring_reg);
220− /* Write the ring initialization command to C2PMSG_64 */
221− psp_ring_reg = ring_type;
222− psp_ring_reg = psp_ring_reg << 16;
223− WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, psp_ring_reg);
224−
225− /* there might be handshake issue with hardware which needs delay */
226− mdelay(20);
227−
228− /* Wait for response flag (bit 31) in C2PMSG_64 */
229− ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
230− 0x80000000, 0x8000FFFF, false);
231− }
150+ /* Write low address of the ring to C2PMSG_69 */
151+ psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);
152+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg);
153+ /* Write high address of the ring to C2PMSG_70 */
154+ psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr);
155+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, psp_ring_reg);
156+ /* Write size of ring to C2PMSG_71 */
157+ psp_ring_reg = ring->ring_size;
158+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_71, psp_ring_reg);
159+ /* Write the ring initialization command to C2PMSG_64 */
160+ psp_ring_reg = ring_type;
161+ psp_ring_reg = psp_ring_reg << 16;
162+ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, psp_ring_reg);
163+
164+ /* Wait for response flag (bit 31) in C2PMSG_64 */
165+ ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
166+ 0x80000000, 0x8000FFFF, false);
232167
233168 return ret;
234169 }
@@ -247,9 +182,6 @@ static int psp_v12_0_ring_stop(struct psp_context *psp, · 12 unmodified lines
247182 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64,
248183 GFX_CTRL_CMD_ID_DESTROY_RINGS);
249184
250− /* there might be handshake issue with hardware which needs delay */
251− mdelay(20);
252−
253185 /* Wait for response flag (bit 31) */
254186 if (amdgpu_sriov_vf(adev))
255187 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101),
drivers/gpu/drm/amd/amdgpu/psp_v13_0.c+51−0
@@ -85,6 +85,8 @@ MODULE_FIRMWARE("amdgpu/psp_14_0_4_ta.bin"); · 84 unmodified lines
8585
8686 #define regMP1_PUB_SCRATCH0 0x3b10090
8787
88+#define PSP13_BL_STATUS_SIZE 100
89+
8890 static int psp_v13_0_init_microcode(struct psp_context *psp)
8991 {
9092 struct amdgpu_device *adev = psp->adev;
@@ -151,6 +153,32 @@ static bool psp_v13_0_is_sos_alive(struct psp_context *psp) · 60 unmodified lines
151153 return sol_reg != 0x0;
152154 }
153155
156+static void psp_v13_0_bootloader_print_status(struct psp_context *psp,
157+ const char *msg)
158+{
159+ struct amdgpu_device *adev = psp->adev;
160+ u32 bl_status_reg;
161+ char bl_status_msg[PSP13_BL_STATUS_SIZE];
162+ int i, at;
163+
164+ if (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||
165+ amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||
166+ amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)) {
167+ at = 0;
168+ for_each_inst(i, adev->aid_mask) {
169+ bl_status_reg =
170+ (SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_92)
171+ << 2) +
172+ adev->asic_funcs->encode_ext_smn_addressing(i);
173+ at += snprintf(bl_status_msg + at,
174+ PSP13_BL_STATUS_SIZE - at,
175+ " status(%02i): 0x%08x", i,
176+ RREG32_PCIE_EXT(bl_status_reg));
177+ }
178+ dev_info(adev->dev, "%s - %s", msg, bl_status_msg);
179+ }
180+}
181+
154182 static int psp_v13_0_wait_for_vmbx_ready(struct psp_context *psp)
155183 {
156184 struct amdgpu_device *adev = psp->adev;
@@ -196,6 +224,9 @@ static int psp_v13_0_wait_for_bootloader(struct psp_context *psp) · 39 unmodified lines
196224
197225 if (ret == 0)
198226 return 0;
227+ if (retry_loop && !(retry_loop % 10))
228+ psp_v13_0_bootloader_print_status(
229+ psp, "Waiting for bootloader completion");
199230 }
200231
201232 return ret;
@@ -858,6 +889,25 @@ static bool psp_v13_0_is_reload_needed(struct psp_context *psp) · 656 unmodified lines
858889 return false;
859890 }
860891
892+static int psp_v13_0_reg_program_no_ring(struct psp_context *psp, uint32_t val,
893+ enum psp_reg_prog_id id)
894+{
895+ struct amdgpu_device *adev = psp->adev;
896+ int ret = -EOPNOTSUPP;
897+
898+ /* PSP will broadcast the value to all instances */
899+ if (amdgpu_sriov_vf(adev)) {
900+ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_101, GFX_CTRL_CMD_ID_GBR_IH_SET);
901+ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_102, id);
902+ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_103, val);
903+
904+ ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_101),
905+ 0x80000000, 0x80000000, false);
906+ }
907+
908+ return ret;
909+}
910+
861911 static const struct psp_funcs psp_v13_0_funcs = {
862912 .init_microcode = psp_v13_0_init_microcode,
863913 .wait_for_bootloader = psp_v13_0_wait_for_bootloader_steady_state,
@@ -884,6 +934,7 @@ static const struct psp_funcs psp_v13_0_funcs = { · 20 unmodified lines
884934 .get_ras_capability = psp_v13_0_get_ras_capability,
885935 .is_aux_sos_load_required = psp_v13_0_is_aux_sos_load_required,
886936 .is_reload_needed = psp_v13_0_is_reload_needed,
937+ .reg_program_no_ring = psp_v13_0_reg_program_no_ring,
887938 };
888939
889940 void psp_v13_0_set_psp_funcs(struct psp_context *psp)
drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c+17−19
@@ -106,8 +106,9 @@ static void sdma_v4_4_2_set_buffer_funcs(struct amdgpu_device *adev); · 105 unmodified lines
106106 static void sdma_v4_4_2_set_vm_pte_funcs(struct amdgpu_device *adev);
107107 static void sdma_v4_4_2_set_irq_funcs(struct amdgpu_device *adev);
108108 static void sdma_v4_4_2_set_ras_funcs(struct amdgpu_device *adev);
109−static void sdma_v4_4_2_set_engine_reset_funcs(struct amdgpu_device *adev);
110109 static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev);
110+static int sdma_v4_4_2_stop_queue(struct amdgpu_ring *ring);
111+static int sdma_v4_4_2_restore_queue(struct amdgpu_ring *ring);
111112
112113 static u32 sdma_v4_4_2_get_reg_offset(struct amdgpu_device *adev,
113114 u32 instance, u32 offset)
@@ -1333,6 +1334,11 @@ static bool sdma_v4_4_2_fw_support_paging_queue(struct amdgpu_device *adev) · 1219 unmodified lines
13331334 }
13341335 }
13351336
1337+static const struct amdgpu_sdma_funcs sdma_v4_4_2_sdma_funcs = {
1338+ .stop_kernel_queue = &sdma_v4_4_2_stop_queue,
1339+ .start_kernel_queue = &sdma_v4_4_2_restore_queue,
1340+};
1341+
13361342 static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block)
13371343 {
13381344 struct amdgpu_device *adev = ip_block->adev;
@@ -1351,8 +1357,6 @@ static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block) · 12 unmodified lines
13511357 sdma_v4_4_2_set_vm_pte_funcs(adev);
13521358 sdma_v4_4_2_set_irq_funcs(adev);
13531359 sdma_v4_4_2_set_ras_funcs(adev);
1354− sdma_v4_4_2_set_engine_reset_funcs(adev);
1355−
13561360 return 0;
13571361 }
13581362
@@ -1447,6 +1451,7 @@ static int sdma_v4_4_2_sw_init(struct amdgpu_ip_block *ip_block) · 88 unmodified lines
14471451 /* Initialize guilty flags for GFX and PAGE queues */
14481452 adev->sdma.instance[i].gfx_guilty = false;
14491453 adev->sdma.instance[i].page_guilty = false;
1454+ adev->sdma.instance[i].funcs = &sdma_v4_4_2_sdma_funcs;
14501455
14511456 ring = &adev->sdma.instance[i].ring;
14521457 ring->ring_obj = NULL;
@@ -1678,11 +1683,12 @@ static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 225 unmodified lines
16781683 return r;
16791684 }
16801685
1681−static int sdma_v4_4_2_stop_queue(struct amdgpu_device *adev, uint32_t instance_id)
1686+static int sdma_v4_4_2_stop_queue(struct amdgpu_ring *ring)
16821687 {
1688+ struct amdgpu_device *adev = ring->adev;
1689+ u32 instance_id = GET_INST(SDMA0, ring->me);
16831690 u32 inst_mask;
16841691 uint64_t rptr;
1685− struct amdgpu_ring *ring = &adev->sdma.instance[instance_id].ring;
16861692
16871693 if (amdgpu_sriov_vf(adev))
16881694 return -EINVAL;
@@ -1715,11 +1721,11 @@ static int sdma_v4_4_2_stop_queue(struct amdgpu_device *adev, uint32_t instance_ · 26 unmodified lines
17151721 return 0;
17161722 }
17171723
1718−static int sdma_v4_4_2_restore_queue(struct amdgpu_device *adev, uint32_t instance_id)
1724+static int sdma_v4_4_2_restore_queue(struct amdgpu_ring *ring)
17191725 {
1720− int i;
1726+ struct amdgpu_device *adev = ring->adev;
17211727 u32 inst_mask;
1722− struct amdgpu_ring *ring = &adev->sdma.instance[instance_id].ring;
1728+ int i;
17231729
17241730 inst_mask = 1 << ring->me;
17251731 udelay(50);
@@ -1739,16 +1745,6 @@ static int sdma_v4_4_2_restore_queue(struct amdgpu_device *adev, uint32_t instan · 13 unmodified lines
17391745 return sdma_v4_4_2_inst_start(adev, inst_mask, true);
17401746 }
17411747
1742−static struct sdma_on_reset_funcs sdma_v4_4_2_engine_reset_funcs = {
1743− .pre_reset = sdma_v4_4_2_stop_queue,
1744− .post_reset = sdma_v4_4_2_restore_queue,
1745−};
1746−
1747−static void sdma_v4_4_2_set_engine_reset_funcs(struct amdgpu_device *adev)
1748−{
1749− amdgpu_sdma_register_on_reset_callbacks(adev, &sdma_v4_4_2_engine_reset_funcs);
1750−}
1751−
17521748 static int sdma_v4_4_2_set_trap_irq_state(struct amdgpu_device *adev,
17531749 struct amdgpu_irq_src *source,
17541750 unsigned type,
@@ -2373,7 +2369,9 @@ static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev) · 618 unmodified lines
23732369 adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
23742370 break;
23752371 case IP_VERSION(9, 5, 0):
2376− /*TODO: enable the queue reset flag until fw supported */
2372+ if ((adev->gfx.mec_fw_version >= 0xf) && amdgpu_dpm_reset_sdma_is_supported(adev))
2373+ adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
2374+ break;
23772375 default:
23782376 break;
23792377 }
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c+114−157
@@ -112,6 +112,8 @@ static void sdma_v5_0_set_ring_funcs(struct amdgpu_device *adev); · 111 unmodified lines
112112 static void sdma_v5_0_set_buffer_funcs(struct amdgpu_device *adev);
113113 static void sdma_v5_0_set_vm_pte_funcs(struct amdgpu_device *adev);
114114 static void sdma_v5_0_set_irq_funcs(struct amdgpu_device *adev);
115+static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring);
116+static int sdma_v5_0_restore_queue(struct amdgpu_ring *ring);
115117
116118 static const struct soc15_reg_golden golden_settings_sdma_5[] = {
117119 SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_CHICKEN_BITS, 0xffbf1f0f, 0x03ab0107),
@@ -369,67 +371,36 @@ static uint64_t sdma_v5_0_ring_get_wptr(struct amdgpu_ring *ring) · 251 unmodified lines
369371 static void sdma_v5_0_ring_set_wptr(struct amdgpu_ring *ring)
370372 {
371373 struct amdgpu_device *adev = ring->adev;
372− uint32_t *wptr_saved;
373− uint32_t *is_queue_unmap;
374− uint64_t aggregated_db_index;
375− uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_DMA].mqd_size;
376374
377375 DRM_DEBUG("Setting write pointer\n");
378− if (ring->is_mes_queue) {
379− wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);
380− is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +
381− sizeof(uint32_t));
382− aggregated_db_index =
383− amdgpu_mes_get_aggregated_doorbell_index(adev,
384− AMDGPU_MES_PRIORITY_LEVEL_NORMAL);
385−
376+ if (ring->use_doorbell) {
377+ DRM_DEBUG("Using doorbell -- "
378+ "wptr_offs == 0x%08x "
379+ "lower_32_bits(ring->wptr) << 2 == 0x%08x "
380+ "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",
381+ ring->wptr_offs,
382+ lower_32_bits(ring->wptr << 2),
383+ upper_32_bits(ring->wptr << 2));
384+ /* XXX check if swapping is necessary on BE */
386385 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
387386 ring->wptr << 2);
388− *wptr_saved = ring->wptr << 2;
389− if (*is_queue_unmap) {
390− WDOORBELL64(aggregated_db_index, ring->wptr << 2);
391− DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
392− ring->doorbell_index, ring->wptr << 2);
393− WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
394− } else {
395− DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
396− ring->doorbell_index, ring->wptr << 2);
397− WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
398−
399− if (*is_queue_unmap)
400− WDOORBELL64(aggregated_db_index,
401− ring->wptr << 2);
402− }
387+ DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
388+ ring->doorbell_index, ring->wptr << 2);
389+ WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
403390 } else {
404− if (ring->use_doorbell) {
405− DRM_DEBUG("Using doorbell -- "
406− "wptr_offs == 0x%08x "
407− "lower_32_bits(ring->wptr) << 2 == 0x%08x "
408− "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",
409− ring->wptr_offs,
410− lower_32_bits(ring->wptr << 2),
411− upper_32_bits(ring->wptr << 2));
412− /* XXX check if swapping is necessary on BE */
413− atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
414− ring->wptr << 2);
415− DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
416− ring->doorbell_index, ring->wptr << 2);
417− WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
418− } else {
419− DRM_DEBUG("Not using doorbell -- "
420− "mmSDMA%i_GFX_RB_WPTR == 0x%08x "
421− "mmSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
422− ring->me,
423− lower_32_bits(ring->wptr << 2),
424− ring->me,
425− upper_32_bits(ring->wptr << 2));
426− WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
427− ring->me, mmSDMA0_GFX_RB_WPTR),
428− lower_32_bits(ring->wptr << 2));
429− WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
430− ring->me, mmSDMA0_GFX_RB_WPTR_HI),
431− upper_32_bits(ring->wptr << 2));
432− }
391+ DRM_DEBUG("Not using doorbell -- "
392+ "mmSDMA%i_GFX_RB_WPTR == 0x%08x "
393+ "mmSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
394+ ring->me,
395+ lower_32_bits(ring->wptr << 2),
396+ ring->me,
397+ upper_32_bits(ring->wptr << 2));
398+ WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
399+ ring->me, mmSDMA0_GFX_RB_WPTR),
400+ lower_32_bits(ring->wptr << 2));
401+ WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
402+ ring->me, mmSDMA0_GFX_RB_WPTR_HI),
403+ upper_32_bits(ring->wptr << 2));
433404 }
434405 }
435406
@@ -575,11 +546,9 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 139 unmodified lines
575546 }
576547
577548 if (flags & AMDGPU_FENCE_FLAG_INT) {
578− uint32_t ctx = ring->is_mes_queue ?
579− (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
580549 /* generate an interrupt */
581550 amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));
582− amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));
551+ amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
583552 }
584553 }
585554
@@ -588,15 +557,15 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 2 unmodified lines
588557 * sdma_v5_0_gfx_stop - stop the gfx async dma engines
589558 *
590559 * @adev: amdgpu_device pointer
591− *
560+ * @inst_mask: mask of dma engine instances to be disabled
592561 * Stop the gfx async dma ring buffers (NAVI10).
593562 */
594−static void sdma_v5_0_gfx_stop(struct amdgpu_device *adev)
563+static void sdma_v5_0_gfx_stop(struct amdgpu_device *adev, uint32_t inst_mask)
595564 {
596565 u32 rb_cntl, ib_cntl;
597566 int i;
598567
599− for (i = 0; i < adev->sdma.num_instances; i++) {
568+ for_each_inst(i, inst_mask) {
600569 rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
601570 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
602571 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
@@ -688,9 +657,11 @@ static void sdma_v5_0_enable(struct amdgpu_device *adev, bool enable) · 85 unmodified lines
688657 {
689658 u32 f32_cntl;
690659 int i;
660+ uint32_t inst_mask;
691661
662+ inst_mask = GENMASK(adev->sdma.num_instances - 1, 0);
692663 if (!enable) {
693− sdma_v5_0_gfx_stop(adev);
664+ sdma_v5_0_gfx_stop(adev, 1 << inst_mask);
694665 sdma_v5_0_rlc_stop(adev);
695666 }
696667
@@ -1046,33 +1017,22 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 349 unmodified lines
10461017 int r;
10471018 u32 tmp;
10481019 u64 gpu_addr;
1049− volatile uint32_t *cpu_ptr = NULL;
10501020
10511021 tmp = 0xCAFEDEAD;
10521022
1053− if (ring->is_mes_queue) {
1054− uint32_t offset = 0;
1055− offset = amdgpu_mes_ctx_get_offs(ring,
1056− AMDGPU_MES_CTX_PADDING_OFFS);
1057− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1058− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1059− *cpu_ptr = tmp;
1060− } else {
1061− r = amdgpu_device_wb_get(adev, &index);
1062− if (r) {
1063− dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
1064− return r;
1065− }
1066−
1067− gpu_addr = adev->wb.gpu_addr + (index * 4);
1068− adev->wb.wb[index] = cpu_to_le32(tmp);
1023+ r = amdgpu_device_wb_get(adev, &index);
1024+ if (r) {
1025+ dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
1026+ return r;
10691027 }
10701028
1029+ gpu_addr = adev->wb.gpu_addr + (index * 4);
1030+ adev->wb.wb[index] = cpu_to_le32(tmp);
1031+
10711032 r = amdgpu_ring_alloc(ring, 20);
10721033 if (r) {
10731034 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
1074− if (!ring->is_mes_queue)
1075− amdgpu_device_wb_free(adev, index);
1035+ amdgpu_device_wb_free(adev, index);
10761036 return r;
10771037 }
10781038
@@ -1085,10 +1045,7 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
10851045 amdgpu_ring_commit(ring);
10861046
10871047 for (i = 0; i < adev->usec_timeout; i++) {
1088− if (ring->is_mes_queue)
1089− tmp = le32_to_cpu(*cpu_ptr);
1090− else
1091− tmp = le32_to_cpu(adev->wb.wb[index]);
1048+ tmp = le32_to_cpu(adev->wb.wb[index]);
10921049 if (tmp == 0xDEADBEEF)
10931050 break;
10941051 if (amdgpu_emu_mode == 1)
@@ -1100,8 +1057,7 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
11001057 if (i >= adev->usec_timeout)
11011058 r = -ETIMEDOUT;
11021059
1103− if (!ring->is_mes_queue)
1104− amdgpu_device_wb_free(adev, index);
1060+ amdgpu_device_wb_free(adev, index);
11051061
11061062 return r;
11071063 }
@@ -1124,38 +1080,24 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
11241080 long r;
11251081 u32 tmp = 0;
11261082 u64 gpu_addr;
1127− volatile uint32_t *cpu_ptr = NULL;
11281083
11291084 tmp = 0xCAFEDEAD;
11301085 memset(&ib, 0, sizeof(ib));
11311086
1132− if (ring->is_mes_queue) {
1133− uint32_t offset = 0;
1134− offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
1135− ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1136− ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1137−
1138− offset = amdgpu_mes_ctx_get_offs(ring,
1139− AMDGPU_MES_CTX_PADDING_OFFS);
1140− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1141− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1142− *cpu_ptr = tmp;
1143− } else {
1144− r = amdgpu_device_wb_get(adev, &index);
1145− if (r) {
1146− dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1147− return r;
1148− }
1087+ r = amdgpu_device_wb_get(adev, &index);
1088+ if (r) {
1089+ dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1090+ return r;
1091+ }
11491092
1150− gpu_addr = adev->wb.gpu_addr + (index * 4);
1151− adev->wb.wb[index] = cpu_to_le32(tmp);
1093+ gpu_addr = adev->wb.gpu_addr + (index * 4);
1094+ adev->wb.wb[index] = cpu_to_le32(tmp);
11521095
1153− r = amdgpu_ib_get(adev, NULL, 256,
1154− AMDGPU_IB_POOL_DIRECT, &ib);
1155− if (r) {
1156− DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1157− goto err0;
1158− }
1096+ r = amdgpu_ib_get(adev, NULL, 256,
1097+ AMDGPU_IB_POOL_DIRECT, &ib);
1098+ if (r) {
1099+ DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1100+ goto err0;
11591101 }
11601102
11611103 ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |
@@ -1183,10 +1125,7 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
11831125 goto err1;
11841126 }
11851127
1186− if (ring->is_mes_queue)
1187− tmp = le32_to_cpu(*cpu_ptr);
1188− else
1189− tmp = le32_to_cpu(adev->wb.wb[index]);
1128+ tmp = le32_to_cpu(adev->wb.wb[index]);
11901129
11911130 if (tmp == 0xDEADBEEF)
11921131 r = 0;
@@ -1197,8 +1136,7 @@ err1: · 4 unmodified lines
11971136 amdgpu_ib_free(&ib, NULL);
11981137 dma_fence_put(f);
11991138 err0:
1200− if (!ring->is_mes_queue)
1201− amdgpu_device_wb_free(adev, index);
1139+ amdgpu_device_wb_free(adev, index);
12021140 return r;
12031141 }
12041142
@@ -1389,6 +1327,36 @@ static void sdma_v5_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring, · 184 unmodified lines
13891327 amdgpu_ring_emit_reg_wait(ring, reg1, mask, mask);
13901328 }
13911329
1330+static int sdma_v5_0_soft_reset_engine(struct amdgpu_device *adev, u32 instance_id)
1331+{
1332+ u32 grbm_soft_reset;
1333+ u32 tmp;
1334+
1335+ grbm_soft_reset = REG_SET_FIELD(0,
1336+ GRBM_SOFT_RESET, SOFT_RESET_SDMA0,
1337+ 1);
1338+ grbm_soft_reset <<= instance_id;
1339+
1340+ tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1341+ tmp |= grbm_soft_reset;
1342+ DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp);
1343+ WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
1344+ tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1345+
1346+ udelay(50);
1347+
1348+ tmp &= ~grbm_soft_reset;
1349+ WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
1350+ tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1351+ return 0;
1352+}
1353+
1354+static const struct amdgpu_sdma_funcs sdma_v5_0_sdma_funcs = {
1355+ .stop_kernel_queue = &sdma_v5_0_stop_queue,
1356+ .start_kernel_queue = &sdma_v5_0_restore_queue,
1357+ .soft_reset_kernel_queue = &sdma_v5_0_soft_reset_engine,
1358+};
1359+
13921360 static int sdma_v5_0_early_init(struct amdgpu_ip_block *ip_block)
13931361 {
13941362 struct amdgpu_device *adev = ip_block->adev;
@@ -1431,6 +1399,7 @@ static int sdma_v5_0_sw_init(struct amdgpu_ip_block *ip_block) · 36 unmodified lines
14311399 return r;
14321400
14331401 for (i = 0; i < adev->sdma.num_instances; i++) {
1402+ adev->sdma.instance[i].funcs = &sdma_v5_0_sdma_funcs;
14341403 ring = &adev->sdma.instance[i].ring;
14351404 ring->ring_obj = NULL;
14361405 ring->use_doorbell = true;
@@ -1572,32 +1541,25 @@ static int sdma_v5_0_soft_reset(struct amdgpu_ip_block *ip_block) · 135 unmodified lines
15721541 static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
15731542 {
15741543 struct amdgpu_device *adev = ring->adev;
1575− int i, j, r;
1576− u32 rb_cntl, ib_cntl, f32_cntl, freeze, cntl, preempt, soft_reset, stat1_reg;
1544+ u32 inst_id = ring->me;
15771545
1578− if (amdgpu_sriov_vf(adev))
1579− return -EINVAL;
1546+ return amdgpu_sdma_reset_engine(adev, inst_id);
1547+}
15801548
1581− for (i = 0; i < adev->sdma.num_instances; i++) {
1582− if (ring == &adev->sdma.instance[i].ring)
1583− break;
1584− }
1549+static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring)
1550+{
1551+ u32 f32_cntl, freeze, cntl, stat1_reg;
1552+ struct amdgpu_device *adev = ring->adev;
1553+ int i, j, r = 0;
15851554
1586− if (i == adev->sdma.num_instances) {
1587− DRM_ERROR("sdma instance not found\n");
1555+ if (amdgpu_sriov_vf(adev))
15881556 return -EINVAL;
1589− }
15901557
1558+ i = ring->me;
15911559 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
15921560
15931561 /* stop queue */
1594− ib_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL));
1595− ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0);
1596− WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl);
1597−
1598− rb_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
1599− rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
1600− WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
1562+ sdma_v5_0_gfx_stop(adev, 1 << i);
16011563
16021564 /* engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */
16031565 freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE));
@@ -1628,30 +1590,25 @@ static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 24 unmodified lines
16281590 cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL));
16291591 cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0);
16301592 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl);
1593+err0:
1594+ amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
1595+ return r;
1596+}
16311597
1632− /* soft reset SDMA_GFX_PREEMPT.IB_PREEMPT = 0 mmGRBM_SOFT_RESET.SOFT_RESET_SDMA0/1 = 1 */
1633− preempt = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT));
1634− preempt = REG_SET_FIELD(preempt, SDMA0_GFX_PREEMPT, IB_PREEMPT, 0);
1635− WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT), preempt);
1636−
1637− soft_reset = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1638− soft_reset |= 1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i;
1639−
1640− WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1641−
1642− udelay(50);
1643−
1644− soft_reset &= ~(1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i);
1645− WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1598+static int sdma_v5_0_restore_queue(struct amdgpu_ring *ring)
1599+{
1600+ struct amdgpu_device *adev = ring->adev;
1601+ u32 inst_id = ring->me;
1602+ u32 freeze;
1603+ int r;
16461604
1605+ amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
16471606 /* unfreeze*/
1648− freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE));
1607+ freeze = RREG32(sdma_v5_0_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE));
16491608 freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0);
1650− WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE), freeze);
1609+ WREG32(sdma_v5_0_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE), freeze);
16511610
1652− r = sdma_v5_0_gfx_resume_instance(adev, i, true);
1653−
1654−err0:
1611+ r = sdma_v5_0_gfx_resume_instance(adev, inst_id, true);
16551612 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
16561613 return r;
16571614 }
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c+87−117
@@ -113,6 +113,8 @@ static void sdma_v5_2_set_ring_funcs(struct amdgpu_device *adev); · 112 unmodified lines
113113 static void sdma_v5_2_set_buffer_funcs(struct amdgpu_device *adev);
114114 static void sdma_v5_2_set_vm_pte_funcs(struct amdgpu_device *adev);
115115 static void sdma_v5_2_set_irq_funcs(struct amdgpu_device *adev);
116+static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring);
117+static int sdma_v5_2_restore_queue(struct amdgpu_ring *ring);
116118
117119 static u32 sdma_v5_2_get_reg_offset(struct amdgpu_device *adev, u32 instance, u32 internal_offset)
118120 {
@@ -394,11 +396,9 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 275 unmodified lines
394396 }
395397
396398 if ((flags & AMDGPU_FENCE_FLAG_INT)) {
397− uint32_t ctx = ring->is_mes_queue ?
398− (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
399399 /* generate an interrupt */
400400 amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));
401− amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));
401+ amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
402402 }
403403 }
404404
@@ -407,15 +407,15 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 2 unmodified lines
407407 * sdma_v5_2_gfx_stop - stop the gfx async dma engines
408408 *
409409 * @adev: amdgpu_device pointer
410− *
410+ * @inst_mask: mask of dma engine instances to be disabled
411411 * Stop the gfx async dma ring buffers.
412412 */
413−static void sdma_v5_2_gfx_stop(struct amdgpu_device *adev)
413+static void sdma_v5_2_gfx_stop(struct amdgpu_device *adev, uint32_t inst_mask)
414414 {
415415 u32 rb_cntl, ib_cntl;
416416 int i;
417417
418− for (i = 0; i < adev->sdma.num_instances; i++) {
418+ for_each_inst(i, inst_mask) {
419419 rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
420420 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
421421 WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
@@ -506,9 +506,11 @@ static void sdma_v5_2_enable(struct amdgpu_device *adev, bool enable) · 84 unmodified lines
506506 {
507507 u32 f32_cntl;
508508 int i;
509+ uint32_t inst_mask;
509510
511+ inst_mask = GENMASK(adev->sdma.num_instances - 1, 0);
510512 if (!enable) {
511− sdma_v5_2_gfx_stop(adev);
513+ sdma_v5_2_gfx_stop(adev, inst_mask);
512514 sdma_v5_2_rlc_stop(adev);
513515 }
514516
@@ -761,37 +763,49 @@ static int sdma_v5_2_load_microcode(struct amdgpu_device *adev) · 246 unmodified lines
761763 return 0;
762764 }
763765
764−static int sdma_v5_2_soft_reset(struct amdgpu_ip_block *ip_block)
766+static int sdma_v5_2_soft_reset_engine(struct amdgpu_device *adev, u32 instance_id)
765767 {
766− struct amdgpu_device *adev = ip_block->adev;
767768 u32 grbm_soft_reset;
768769 u32 tmp;
769− int i;
770770
771− for (i = 0; i < adev->sdma.num_instances; i++) {
772− grbm_soft_reset = REG_SET_FIELD(0,
773− GRBM_SOFT_RESET, SOFT_RESET_SDMA0,
774− 1);
775− grbm_soft_reset <<= i;
771+ grbm_soft_reset = REG_SET_FIELD(0,
772+ GRBM_SOFT_RESET, SOFT_RESET_SDMA0,
773+ 1);
774+ grbm_soft_reset <<= instance_id;
776775
777− tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
778− tmp |= grbm_soft_reset;
779− DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp);
780− WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
781− tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
776+ tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
777+ tmp |= grbm_soft_reset;
778+ DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp);
779+ WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
780+ tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
782781
783− udelay(50);
782+ udelay(50);
783+
784+ tmp &= ~grbm_soft_reset;
785+ WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
786+ tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
787+ return 0;
788+}
784789
785− tmp &= ~grbm_soft_reset;
786− WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
787− tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
790+static int sdma_v5_2_soft_reset(struct amdgpu_ip_block *ip_block)
791+{
792+ struct amdgpu_device *adev = ip_block->adev;
793+ int i;
788794
795+ for (i = 0; i < adev->sdma.num_instances; i++) {
796+ sdma_v5_2_soft_reset_engine(adev, i);
789797 udelay(50);
790798 }
791799
792800 return 0;
793801 }
794802
803+static const struct amdgpu_sdma_funcs sdma_v5_2_sdma_funcs = {
804+ .stop_kernel_queue = &sdma_v5_2_stop_queue,
805+ .start_kernel_queue = &sdma_v5_2_restore_queue,
806+ .soft_reset_kernel_queue = &sdma_v5_2_soft_reset_engine,
807+};
808+
795809 /**
796810 * sdma_v5_2_start - setup and start the async dma engines
797811 *
@@ -903,33 +917,22 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 105 unmodified lines
903917 int r;
904918 u32 tmp;
905919 u64 gpu_addr;
906− volatile uint32_t *cpu_ptr = NULL;
907920
908921 tmp = 0xCAFEDEAD;
909922
910− if (ring->is_mes_queue) {
911− uint32_t offset = 0;
912− offset = amdgpu_mes_ctx_get_offs(ring,
913− AMDGPU_MES_CTX_PADDING_OFFS);
914− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
915− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
916− *cpu_ptr = tmp;
917− } else {
918− r = amdgpu_device_wb_get(adev, &index);
919− if (r) {
920− dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
921− return r;
922− }
923−
924− gpu_addr = adev->wb.gpu_addr + (index * 4);
925− adev->wb.wb[index] = cpu_to_le32(tmp);
923+ r = amdgpu_device_wb_get(adev, &index);
924+ if (r) {
925+ dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
926+ return r;
926927 }
927928
929+ gpu_addr = adev->wb.gpu_addr + (index * 4);
930+ adev->wb.wb[index] = cpu_to_le32(tmp);
931+
928932 r = amdgpu_ring_alloc(ring, 20);
929933 if (r) {
930934 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
931− if (!ring->is_mes_queue)
932− amdgpu_device_wb_free(adev, index);
935+ amdgpu_device_wb_free(adev, index);
933936 return r;
934937 }
935938
@@ -942,10 +945,7 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
942945 amdgpu_ring_commit(ring);
943946
944947 for (i = 0; i < adev->usec_timeout; i++) {
945− if (ring->is_mes_queue)
946− tmp = le32_to_cpu(*cpu_ptr);
947− else
948− tmp = le32_to_cpu(adev->wb.wb[index]);
948+ tmp = le32_to_cpu(adev->wb.wb[index]);
949949 if (tmp == 0xDEADBEEF)
950950 break;
951951 if (amdgpu_emu_mode == 1)
@@ -957,8 +957,7 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
957957 if (i >= adev->usec_timeout)
958958 r = -ETIMEDOUT;
959959
960− if (!ring->is_mes_queue)
961− amdgpu_device_wb_free(adev, index);
960+ amdgpu_device_wb_free(adev, index);
962961
963962 return r;
964963 }
@@ -981,37 +980,23 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
981980 long r;
982981 u32 tmp = 0;
983982 u64 gpu_addr;
984− volatile uint32_t *cpu_ptr = NULL;
985983
986984 tmp = 0xCAFEDEAD;
987985 memset(&ib, 0, sizeof(ib));
988986
989− if (ring->is_mes_queue) {
990− uint32_t offset = 0;
991− offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
992− ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
993− ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
994−
995− offset = amdgpu_mes_ctx_get_offs(ring,
996− AMDGPU_MES_CTX_PADDING_OFFS);
997− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
998− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
999− *cpu_ptr = tmp;
1000− } else {
1001− r = amdgpu_device_wb_get(adev, &index);
1002− if (r) {
1003− dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1004− return r;
1005− }
987+ r = amdgpu_device_wb_get(adev, &index);
988+ if (r) {
989+ dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
990+ return r;
991+ }
1006992
1007− gpu_addr = adev->wb.gpu_addr + (index * 4);
1008− adev->wb.wb[index] = cpu_to_le32(tmp);
993+ gpu_addr = adev->wb.gpu_addr + (index * 4);
994+ adev->wb.wb[index] = cpu_to_le32(tmp);
1009995
1010− r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1011− if (r) {
1012− DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1013− goto err0;
1014− }
996+ r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
997+ if (r) {
998+ DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
999+ goto err0;
10151000 }
10161001
10171002 ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |
@@ -1039,10 +1024,7 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
10391024 goto err1;
10401025 }
10411026
1042− if (ring->is_mes_queue)
1043− tmp = le32_to_cpu(*cpu_ptr);
1044− else
1045− tmp = le32_to_cpu(adev->wb.wb[index]);
1027+ tmp = le32_to_cpu(adev->wb.wb[index]);
10461028
10471029 if (tmp == 0xDEADBEEF)
10481030 r = 0;
@@ -1053,8 +1035,7 @@ err1: · 4 unmodified lines
10531035 amdgpu_ib_free(&ib, NULL);
10541036 dma_fence_put(f);
10551037 err0:
1056− if (!ring->is_mes_queue)
1057− amdgpu_device_wb_free(adev, index);
1038+ amdgpu_device_wb_free(adev, index);
10581039 return r;
10591040 }
10601041
@@ -1337,6 +1318,7 @@ static int sdma_v5_2_sw_init(struct amdgpu_ip_block *ip_block) · 276 unmodified lines
13371318 }
13381319
13391320 for (i = 0; i < adev->sdma.num_instances; i++) {
1321+ adev->sdma.instance[i].funcs = &sdma_v5_2_sdma_funcs;
13401322 ring = &adev->sdma.instance[i].ring;
13411323 ring->ring_obj = NULL;
13421324 ring->use_doorbell = true;
@@ -1472,32 +1454,25 @@ static int sdma_v5_2_wait_for_idle(struct amdgpu_ip_block *ip_block) · 129 unmodified lines
14721454 static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
14731455 {
14741456 struct amdgpu_device *adev = ring->adev;
1475− int i, j, r;
1476− u32 rb_cntl, ib_cntl, f32_cntl, freeze, cntl, preempt, soft_reset, stat1_reg;
1457+ u32 inst_id = ring->me;
14771458
1478− if (amdgpu_sriov_vf(adev))
1479− return -EINVAL;
1459+ return amdgpu_sdma_reset_engine(adev, inst_id);
1460+}
14801461
1481− for (i = 0; i < adev->sdma.num_instances; i++) {
1482− if (ring == &adev->sdma.instance[i].ring)
1483− break;
1484− }
1462+static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring)
1463+{
1464+ u32 f32_cntl, freeze, cntl, stat1_reg;
1465+ struct amdgpu_device *adev = ring->adev;
1466+ int i, j, r = 0;
14851467
1486− if (i == adev->sdma.num_instances) {
1487− DRM_ERROR("sdma instance not found\n");
1468+ if (amdgpu_sriov_vf(adev))
14881469 return -EINVAL;
1489− }
14901470
1471+ i = ring->me;
14911472 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
14921473
14931474 /* stop queue */
1494− ib_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL));
1495− ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0);
1496− WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl);
1497−
1498− rb_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
1499− rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
1500− WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
1475+ sdma_v5_2_gfx_stop(adev, 1 << i);
15011476
15021477 /*engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */
15031478 freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE));
@@ -1530,31 +1505,26 @@ static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 26 unmodified lines
15301505 cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0);
15311506 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl);
15321507
1533− /* soft reset SDMA_GFX_PREEMPT.IB_PREEMPT = 0 mmGRBM_SOFT_RESET.SOFT_RESET_SDMA0/1 = 1 */
1534− preempt = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT));
1535− preempt = REG_SET_FIELD(preempt, SDMA0_GFX_PREEMPT, IB_PREEMPT, 0);
1536− WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT), preempt);
1537−
1538− soft_reset = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1539− soft_reset |= 1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i;
1540−
1541−
1542− WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1543−
1544− udelay(50);
1545−
1546− soft_reset &= ~(1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i);
1508+err0:
1509+ amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
1510+ return r;
1511+}
15471512
1548− WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1513+static int sdma_v5_2_restore_queue(struct amdgpu_ring *ring)
1514+{
1515+ struct amdgpu_device *adev = ring->adev;
1516+ u32 inst_id = ring->me;
1517+ u32 freeze;
1518+ int r;
15491519
1520+ amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
15501521 /* unfreeze and unhalt */
1551− freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE));
1522+ freeze = RREG32(sdma_v5_2_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE));
15521523 freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0);
1553− WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE), freeze);
1524+ WREG32(sdma_v5_2_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE), freeze);
15541525
1555− r = sdma_v5_2_gfx_resume_instance(adev, i, true);
1526+ r = sdma_v5_2_gfx_resume_instance(adev, inst_id, true);
15561527
1557−err0:
15581528 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
15591529 return r;
15601530 }
drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c+80−60
@@ -43,6 +43,7 @@ · 42 unmodified lines
4343 #include "sdma_common.h"
4444 #include "sdma_v6_0.h"
4545 #include "v11_structs.h"
46+#include "mes_userqueue.h"
4647
4748 MODULE_FIRMWARE("amdgpu/sdma_6_0_0.bin");
4849 MODULE_FIRMWARE("amdgpu/sdma_6_0_1.bin");
@@ -376,11 +377,9 @@ static void sdma_v6_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 327 unmodified lines
376377 }
377378
378379 if (flags & AMDGPU_FENCE_FLAG_INT) {
379− uint32_t ctx = ring->is_mes_queue ?
380− (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
381380 /* generate an interrupt */
382381 amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP));
383− amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));
382+ amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
384383 }
385384 }
386385
@@ -891,6 +890,9 @@ static int sdma_v6_0_mqd_init(struct amdgpu_device *adev, void *mqd, · 504 unmodified lines
891890 m->sdmax_rlcx_rb_aql_cntl = regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT;
892891 m->sdmax_rlcx_dummy_reg = regSDMA0_QUEUE0_DUMMY_REG_DEFAULT;
893892
893+ m->sdmax_rlcx_csa_addr_lo = lower_32_bits(prop->csa_addr);
894+ m->sdmax_rlcx_csa_addr_hi = upper_32_bits(prop->csa_addr);
895+
894896 return 0;
895897 }
896898
@@ -917,33 +919,22 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 20 unmodified lines
917919 int r;
918920 u32 tmp;
919921 u64 gpu_addr;
920− volatile uint32_t *cpu_ptr = NULL;
921922
922923 tmp = 0xCAFEDEAD;
923924
924− if (ring->is_mes_queue) {
925− uint32_t offset = 0;
926− offset = amdgpu_mes_ctx_get_offs(ring,
927− AMDGPU_MES_CTX_PADDING_OFFS);
928− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
929− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
930− *cpu_ptr = tmp;
931− } else {
932− r = amdgpu_device_wb_get(adev, &index);
933− if (r) {
934− dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
935− return r;
936− }
937−
938− gpu_addr = adev->wb.gpu_addr + (index * 4);
939− adev->wb.wb[index] = cpu_to_le32(tmp);
925+ r = amdgpu_device_wb_get(adev, &index);
926+ if (r) {
927+ dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
928+ return r;
940929 }
941930
931+ gpu_addr = adev->wb.gpu_addr + (index * 4);
932+ adev->wb.wb[index] = cpu_to_le32(tmp);
933+
942934 r = amdgpu_ring_alloc(ring, 5);
943935 if (r) {
944936 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
945− if (!ring->is_mes_queue)
946− amdgpu_device_wb_free(adev, index);
937+ amdgpu_device_wb_free(adev, index);
947938 return r;
948939 }
949940
@@ -956,10 +947,7 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
956947 amdgpu_ring_commit(ring);
957948
958949 for (i = 0; i < adev->usec_timeout; i++) {
959− if (ring->is_mes_queue)
960− tmp = le32_to_cpu(*cpu_ptr);
961− else
962− tmp = le32_to_cpu(adev->wb.wb[index]);
950+ tmp = le32_to_cpu(adev->wb.wb[index]);
963951 if (tmp == 0xDEADBEEF)
964952 break;
965953 if (amdgpu_emu_mode == 1)
@@ -971,8 +959,7 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
971959 if (i >= adev->usec_timeout)
972960 r = -ETIMEDOUT;
973961
974− if (!ring->is_mes_queue)
975− amdgpu_device_wb_free(adev, index);
962+ amdgpu_device_wb_free(adev, index);
976963
977964 return r;
978965 }
@@ -995,37 +982,23 @@ static int sdma_v6_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
995982 long r;
996983 u32 tmp = 0;
997984 u64 gpu_addr;
998− volatile uint32_t *cpu_ptr = NULL;
999985
1000986 tmp = 0xCAFEDEAD;
1001987 memset(&ib, 0, sizeof(ib));
1002988
1003− if (ring->is_mes_queue) {
1004− uint32_t offset = 0;
1005− offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
1006− ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1007− ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1008−
1009− offset = amdgpu_mes_ctx_get_offs(ring,
1010− AMDGPU_MES_CTX_PADDING_OFFS);
1011− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1012− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1013− *cpu_ptr = tmp;
1014− } else {
1015− r = amdgpu_device_wb_get(adev, &index);
1016− if (r) {
1017− dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1018− return r;
1019− }
989+ r = amdgpu_device_wb_get(adev, &index);
990+ if (r) {
991+ dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
992+ return r;
993+ }
1020994
1021− gpu_addr = adev->wb.gpu_addr + (index * 4);
1022− adev->wb.wb[index] = cpu_to_le32(tmp);
995+ gpu_addr = adev->wb.gpu_addr + (index * 4);
996+ adev->wb.wb[index] = cpu_to_le32(tmp);
1023997
1024− r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1025− if (r) {
1026− DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1027− goto err0;
1028− }
998+ r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
999+ if (r) {
1000+ DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1001+ goto err0;
10291002 }
10301003
10311004 ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) |
@@ -1053,10 +1026,7 @@ static int sdma_v6_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
10531026 goto err1;
10541027 }
10551028
1056− if (ring->is_mes_queue)
1057− tmp = le32_to_cpu(*cpu_ptr);
1058− else
1059− tmp = le32_to_cpu(adev->wb.wb[index]);
1029+ tmp = le32_to_cpu(adev->wb.wb[index]);
10601030
10611031 if (tmp == 0xDEADBEEF)
10621032 r = 0;
@@ -1067,8 +1037,7 @@ err1: · 4 unmodified lines
10671037 amdgpu_ib_free(&ib, NULL);
10681038 dma_fence_put(f);
10691039 err0:
1070− if (!ring->is_mes_queue)
1071− amdgpu_device_wb_free(adev, index);
1040+ amdgpu_device_wb_free(adev, index);
10721041 return r;
10731042 }
10741043
@@ -1300,6 +1269,23 @@ static int sdma_v6_0_early_init(struct amdgpu_ip_block *ip_block) · 225 unmodified lines
13001269 struct amdgpu_device *adev = ip_block->adev;
13011270 int r;
13021271
1272+ switch (amdgpu_user_queue) {
1273+ case -1:
1274+ case 0:
1275+ default:
1276+ adev->sdma.no_user_submission = false;
1277+ adev->sdma.disable_uq = true;
1278+ break;
1279+ case 1:
1280+ adev->sdma.no_user_submission = false;
1281+ adev->sdma.disable_uq = false;
1282+ break;
1283+ case 2:
1284+ adev->sdma.no_user_submission = true;
1285+ adev->sdma.disable_uq = false;
1286+ break;
1287+ }
1288+
13031289 r = amdgpu_sdma_init_microcode(adev, 0, true);
13041290 if (r)
13051291 return r;
@@ -1334,6 +1320,7 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 28 unmodified lines
13341320 ring->ring_obj = NULL;
13351321 ring->use_doorbell = true;
13361322 ring->me = i;
1323+ ring->no_user_submission = adev->sdma.no_user_submission;
13371324
13381325 DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
13391326 ring->use_doorbell?"true":"false");
@@ -1376,6 +1363,10 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 36 unmodified lines
13761363 else
13771364 DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n");
13781365
1366+ /* add firmware version checks here */
1367+ if (0 && !adev->sdma.disable_uq)
1368+ adev->userq_funcs[AMDGPU_HW_IP_DMA] = &userq_mes_funcs;
1369+
13791370 r = amdgpu_sdma_sysfs_reset_mask_init(adev);
13801371 if (r)
13811372 return r;
@@ -1399,11 +1390,39 @@ static int sdma_v6_0_sw_fini(struct amdgpu_ip_block *ip_block) · 17 unmodified lines
13991390 return 0;
14001391 }
14011392
1393+static int sdma_v6_0_set_userq_trap_interrupts(struct amdgpu_device *adev,
1394+ bool enable)
1395+{
1396+ unsigned int irq_type;
1397+ int i, r;
1398+
1399+ if (adev->userq_funcs[AMDGPU_HW_IP_DMA]) {
1400+ for (i = 0; i < adev->sdma.num_instances; i++) {
1401+ irq_type = AMDGPU_SDMA_IRQ_INSTANCE0 + i;
1402+ if (enable)
1403+ r = amdgpu_irq_get(adev, &adev->sdma.trap_irq,
1404+ irq_type);
1405+ else
1406+ r = amdgpu_irq_put(adev, &adev->sdma.trap_irq,
1407+ irq_type);
1408+ if (r)
1409+ return r;
1410+ }
1411+ }
1412+
1413+ return 0;
1414+}
1415+
14021416 static int sdma_v6_0_hw_init(struct amdgpu_ip_block *ip_block)
14031417 {
14041418 struct amdgpu_device *adev = ip_block->adev;
1419+ int r;
14051420
1406− return sdma_v6_0_start(adev);
1421+ r = sdma_v6_0_start(adev);
1422+ if (r)
1423+ return r;
1424+
1425+ return sdma_v6_0_set_userq_trap_interrupts(adev, true);
14071426 }
14081427
14091428 static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block)
@@ -1415,6 +1434,7 @@ static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
14151434
14161435 sdma_v6_0_ctxempty_int_enable(adev, false);
14171436 sdma_v6_0_enable(adev, false);
1437+ sdma_v6_0_set_userq_trap_interrupts(adev, false);
14181438
14191439 return 0;
14201440 }
drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c+107−114
@@ -42,6 +42,7 @@ · 41 unmodified lines
4242 #include "sdma_common.h"
4343 #include "sdma_v7_0.h"
4444 #include "v12_structs.h"
45+#include "mes_userqueue.h"
4546
4647 MODULE_FIRMWARE("amdgpu/sdma_7_0_0.bin");
4748 MODULE_FIRMWARE("amdgpu/sdma_7_0_1.bin");
@@ -204,66 +205,39 @@ static uint64_t sdma_v7_0_ring_get_wptr(struct amdgpu_ring *ring) · 156 unmodified lines
204205 static void sdma_v7_0_ring_set_wptr(struct amdgpu_ring *ring)
205206 {
206207 struct amdgpu_device *adev = ring->adev;
207− uint32_t *wptr_saved;
208− uint32_t *is_queue_unmap;
209− uint64_t aggregated_db_index;
210− uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_DMA].mqd_size;
211208
212209 DRM_DEBUG("Setting write pointer\n");
213210
214− if (ring->is_mes_queue) {
215− wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);
216− is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +
217− sizeof(uint32_t));
218− aggregated_db_index =
219− amdgpu_mes_get_aggregated_doorbell_index(adev,
220− ring->hw_prio);
221−
211+ if (ring->use_doorbell) {
212+ DRM_DEBUG("Using doorbell -- "
213+ "wptr_offs == 0x%08x "
214+ "lower_32_bits(ring->wptr) << 2 == 0x%08x "
215+ "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",
216+ ring->wptr_offs,
217+ lower_32_bits(ring->wptr << 2),
218+ upper_32_bits(ring->wptr << 2));
219+ /* XXX check if swapping is necessary on BE */
222220 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
223221 ring->wptr << 2);
224− *wptr_saved = ring->wptr << 2;
225− if (*is_queue_unmap) {
226− WDOORBELL64(aggregated_db_index, ring->wptr << 2);
227− DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
228− ring->doorbell_index, ring->wptr << 2);
229− WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
230− } else {
231− DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
232− ring->doorbell_index, ring->wptr << 2);
233− WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
234− }
222+ DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
223+ ring->doorbell_index, ring->wptr << 2);
224+ WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
235225 } else {
236− if (ring->use_doorbell) {
237− DRM_DEBUG("Using doorbell -- "
238− "wptr_offs == 0x%08x "
239− "lower_32_bits(ring->wptr) << 2 == 0x%08x "
240− "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",
241− ring->wptr_offs,
242− lower_32_bits(ring->wptr << 2),
243− upper_32_bits(ring->wptr << 2));
244− /* XXX check if swapping is necessary on BE */
245− atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
246− ring->wptr << 2);
247− DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
248− ring->doorbell_index, ring->wptr << 2);
249− WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
250− } else {
251− DRM_DEBUG("Not using doorbell -- "
252− "regSDMA%i_GFX_RB_WPTR == 0x%08x "
253− "regSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
254− ring->me,
255− lower_32_bits(ring->wptr << 2),
256− ring->me,
257− upper_32_bits(ring->wptr << 2));
258− WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
259− ring->me,
260− regSDMA0_QUEUE0_RB_WPTR),
261− lower_32_bits(ring->wptr << 2));
262− WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
263− ring->me,
264− regSDMA0_QUEUE0_RB_WPTR_HI),
265− upper_32_bits(ring->wptr << 2));
266− }
226+ DRM_DEBUG("Not using doorbell -- "
227+ "regSDMA%i_GFX_RB_WPTR == 0x%08x "
228+ "regSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
229+ ring->me,
230+ lower_32_bits(ring->wptr << 2),
231+ ring->me,
232+ upper_32_bits(ring->wptr << 2));
233+ WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
234+ ring->me,
235+ regSDMA0_QUEUE0_RB_WPTR),
236+ lower_32_bits(ring->wptr << 2));
237+ WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
238+ ring->me,
239+ regSDMA0_QUEUE0_RB_WPTR_HI),
240+ upper_32_bits(ring->wptr << 2));
267241 }
268242 }
269243
@@ -407,11 +381,9 @@ static void sdma_v7_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 137 unmodified lines
407381 }
408382
409383 if (flags & AMDGPU_FENCE_FLAG_INT) {
410− uint32_t ctx = ring->is_mes_queue ?
411− (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
412384 /* generate an interrupt */
413385 amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP));
414− amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));
386+ amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
415387 }
416388 }
417389
@@ -935,6 +907,9 @@ static int sdma_v7_0_mqd_init(struct amdgpu_device *adev, void *mqd, · 517 unmodified lines
935907 m->sdmax_rlcx_rb_aql_cntl = 0x4000; //regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT;
936908 m->sdmax_rlcx_dummy_reg = 0xf; //regSDMA0_QUEUE0_DUMMY_REG_DEFAULT;
937909
910+ m->sdmax_rlcx_csa_addr_lo = lower_32_bits(prop->csa_addr);
911+ m->sdmax_rlcx_csa_addr_hi = upper_32_bits(prop->csa_addr);
912+
938913 return 0;
939914 }
940915
@@ -961,33 +936,22 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 20 unmodified lines
961936 int r;
962937 u32 tmp;
963938 u64 gpu_addr;
964− volatile uint32_t *cpu_ptr = NULL;
965939
966940 tmp = 0xCAFEDEAD;
967941
968− if (ring->is_mes_queue) {
969− uint32_t offset = 0;
970− offset = amdgpu_mes_ctx_get_offs(ring,
971− AMDGPU_MES_CTX_PADDING_OFFS);
972− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
973− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
974− *cpu_ptr = tmp;
975− } else {
976− r = amdgpu_device_wb_get(adev, &index);
977− if (r) {
978− dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
979− return r;
980− }
981−
982− gpu_addr = adev->wb.gpu_addr + (index * 4);
983− adev->wb.wb[index] = cpu_to_le32(tmp);
942+ r = amdgpu_device_wb_get(adev, &index);
943+ if (r) {
944+ dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
945+ return r;
984946 }
985947
948+ gpu_addr = adev->wb.gpu_addr + (index * 4);
949+ adev->wb.wb[index] = cpu_to_le32(tmp);
950+
986951 r = amdgpu_ring_alloc(ring, 5);
987952 if (r) {
988953 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
989− if (!ring->is_mes_queue)
990− amdgpu_device_wb_free(adev, index);
954+ amdgpu_device_wb_free(adev, index);
991955 return r;
992956 }
993957
@@ -1000,10 +964,7 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
1000964 amdgpu_ring_commit(ring);
1001965
1002966 for (i = 0; i < adev->usec_timeout; i++) {
1003− if (ring->is_mes_queue)
1004− tmp = le32_to_cpu(*cpu_ptr);
1005− else
1006− tmp = le32_to_cpu(adev->wb.wb[index]);
967+ tmp = le32_to_cpu(adev->wb.wb[index]);
1007968 if (tmp == 0xDEADBEEF)
1008969 break;
1009970 if (amdgpu_emu_mode == 1)
@@ -1015,8 +976,7 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
1015976 if (i >= adev->usec_timeout)
1016977 r = -ETIMEDOUT;
1017978
1018− if (!ring->is_mes_queue)
1019− amdgpu_device_wb_free(adev, index);
979+ amdgpu_device_wb_free(adev, index);
1020980
1021981 return r;
1022982 }
@@ -1039,37 +999,23 @@ static int sdma_v7_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
1039999 long r;
10401000 u32 tmp = 0;
10411001 u64 gpu_addr;
1042− volatile uint32_t *cpu_ptr = NULL;
10431002
10441003 tmp = 0xCAFEDEAD;
10451004 memset(&ib, 0, sizeof(ib));
10461005
1047− if (ring->is_mes_queue) {
1048− uint32_t offset = 0;
1049− offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
1050− ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1051− ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1052−
1053− offset = amdgpu_mes_ctx_get_offs(ring,
1054− AMDGPU_MES_CTX_PADDING_OFFS);
1055− gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1056− cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1057− *cpu_ptr = tmp;
1058− } else {
1059− r = amdgpu_device_wb_get(adev, &index);
1060− if (r) {
1061− dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1062− return r;
1063− }
1006+ r = amdgpu_device_wb_get(adev, &index);
1007+ if (r) {
1008+ dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1009+ return r;
1010+ }
10641011
1065− gpu_addr = adev->wb.gpu_addr + (index * 4);
1066− adev->wb.wb[index] = cpu_to_le32(tmp);
1012+ gpu_addr = adev->wb.gpu_addr + (index * 4);
1013+ adev->wb.wb[index] = cpu_to_le32(tmp);
10671014
1068− r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1069− if (r) {
1070− DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1071− goto err0;
1072− }
1015+ r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1016+ if (r) {
1017+ DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1018+ goto err0;
10731019 }
10741020
10751021 ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) |
@@ -1097,10 +1043,7 @@ static int sdma_v7_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
10971043 goto err1;
10981044 }
10991045
1100− if (ring->is_mes_queue)
1101− tmp = le32_to_cpu(*cpu_ptr);
1102− else
1103− tmp = le32_to_cpu(adev->wb.wb[index]);
1046+ tmp = le32_to_cpu(adev->wb.wb[index]);
11041047
11051048 if (tmp == 0xDEADBEEF)
11061049 r = 0;
@@ -1111,8 +1054,7 @@ err1: · 4 unmodified lines
11111054 amdgpu_ib_free(&ib, NULL);
11121055 dma_fence_put(f);
11131056 err0:
1114− if (!ring->is_mes_queue)
1115− amdgpu_device_wb_free(adev, index);
1057+ amdgpu_device_wb_free(adev, index);
11161058 return r;
11171059 }
11181060
@@ -1312,6 +1254,23 @@ static int sdma_v7_0_early_init(struct amdgpu_ip_block *ip_block) · 193 unmodified lines
13121254 struct amdgpu_device *adev = ip_block->adev;
13131255 int r;
13141256
1257+ switch (amdgpu_user_queue) {
1258+ case -1:
1259+ case 0:
1260+ default:
1261+ adev->sdma.no_user_submission = false;
1262+ adev->sdma.disable_uq = true;
1263+ break;
1264+ case 1:
1265+ adev->sdma.no_user_submission = false;
1266+ adev->sdma.disable_uq = false;
1267+ break;
1268+ case 2:
1269+ adev->sdma.no_user_submission = true;
1270+ adev->sdma.disable_uq = false;
1271+ break;
1272+ }
1273+
13151274 r = amdgpu_sdma_init_microcode(adev, 0, true);
13161275 if (r) {
13171276 DRM_ERROR("Failed to init sdma firmware!\n");
@@ -1347,6 +1306,7 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block) · 29 unmodified lines
13471306 ring->ring_obj = NULL;
13481307 ring->use_doorbell = true;
13491308 ring->me = i;
1309+ ring->no_user_submission = adev->sdma.no_user_submission;
13501310
13511311 DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
13521312 ring->use_doorbell?"true":"false");
@@ -1378,6 +1338,10 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block) · 25 unmodified lines
13781338 else
13791339 DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n");
13801340
1341+ /* add firmware version checks here */
1342+ if (0 && !adev->sdma.disable_uq)
1343+ adev->userq_funcs[AMDGPU_HW_IP_DMA] = &userq_mes_funcs;
1344+
13811345 return r;
13821346 }
13831347
@@ -1400,11 +1364,39 @@ static int sdma_v7_0_sw_fini(struct amdgpu_ip_block *ip_block) · 16 unmodified lines
14001364 return 0;
14011365 }
14021366
1367+static int sdma_v7_0_set_userq_trap_interrupts(struct amdgpu_device *adev,
1368+ bool enable)
1369+{
1370+ unsigned int irq_type;
1371+ int i, r;
1372+
1373+ if (adev->userq_funcs[AMDGPU_HW_IP_DMA]) {
1374+ for (i = 0; i < adev->sdma.num_instances; i++) {
1375+ irq_type = AMDGPU_SDMA_IRQ_INSTANCE0 + i;
1376+ if (enable)
1377+ r = amdgpu_irq_get(adev, &adev->sdma.trap_irq,
1378+ irq_type);
1379+ else
1380+ r = amdgpu_irq_put(adev, &adev->sdma.trap_irq,
1381+ irq_type);
1382+ if (r)
1383+ return r;
1384+ }
1385+ }
1386+
1387+ return 0;
1388+}
1389+
14031390 static int sdma_v7_0_hw_init(struct amdgpu_ip_block *ip_block)
14041391 {
14051392 struct amdgpu_device *adev = ip_block->adev;
1393+ int r;
14061394
1407− return sdma_v7_0_start(adev);
1395+ r = sdma_v7_0_start(adev);
1396+ if (r)
1397+ return r;
1398+
1399+ return sdma_v7_0_set_userq_trap_interrupts(adev, true);
14081400 }
14091401
14101402 static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block)
@@ -1416,6 +1408,7 @@ static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
14161408
14171409 sdma_v7_0_ctx_switch_enable(adev, false);
14181410 sdma_v7_0_enable(adev, false);
1411+ sdma_v7_0_set_userq_trap_interrupts(adev, false);
14191412
14201413 return 0;
14211414 }
drivers/gpu/drm/amd/amdgpu/si.c+214−199
@@ -35,6 +35,7 @@ · 34 unmodified lines
3535 #include "amdgpu_vce.h"
3636 #include "atom.h"
3737 #include "amd_pcie.h"
38+
3839 #include "si_dpm.h"
3940 #include "sid.h"
4041 #include "si_ih.h"
@@ -44,17 +45,30 @@ · 3 unmodified lines
4445 #include "dce_v6_0.h"
4546 #include "si.h"
4647 #include "uvd_v3_1.h"
47−#include "amdgpu_vkms.h"
48+
49+#include "uvd/uvd_4_0_d.h"
50+
51+#include "smu/smu_6_0_d.h"
52+#include "smu/smu_6_0_sh_mask.h"
53+
4854 #include "gca/gfx_6_0_d.h"
55+#include "gca/gfx_6_0_sh_mask.h"
56+
4957 #include "oss/oss_1_0_d.h"
5058 #include "oss/oss_1_0_sh_mask.h"
59+
5160 #include "gmc/gmc_6_0_d.h"
61+#include"gmc/gmc_6_0_sh_mask.h"
62+
5263 #include "dce/dce_6_0_d.h"
53−#include "uvd/uvd_4_0_d.h"
64+#include "dce/dce_6_0_sh_mask.h"
65+
5466 #include "bif/bif_3_0_d.h"
5567 #include "bif/bif_3_0_sh_mask.h"
68+#include "si_enums.h"
5669
5770 #include "amdgpu_dm.h"
71+#include "amdgpu_vkms.h"
5872
5973 static const u32 tahiti_golden_registers[] =
6074 {
@@ -1071,8 +1085,8 @@ static u32 si_smc_rreg(struct amdgpu_device *adev, u32 reg) · 1010 unmodified lines
10711085 u32 r;
10721086
10731087 spin_lock_irqsave(&adev->smc_idx_lock, flags);
1074− WREG32(SMC_IND_INDEX_0, (reg));
1075− r = RREG32(SMC_IND_DATA_0);
1088+ WREG32(mmSMC_IND_INDEX_0, (reg));
1089+ r = RREG32(mmSMC_IND_DATA_0);
10761090 spin_unlock_irqrestore(&adev->smc_idx_lock, flags);
10771091 return r;
10781092 }
@@ -1082,8 +1096,8 @@ static void si_smc_wreg(struct amdgpu_device *adev, u32 reg, u32 v) · 3 unmodified lines
10821096 unsigned long flags;
10831097
10841098 spin_lock_irqsave(&adev->smc_idx_lock, flags);
1085− WREG32(SMC_IND_INDEX_0, (reg));
1086− WREG32(SMC_IND_DATA_0, (v));
1099+ WREG32(mmSMC_IND_INDEX_0, (reg));
1100+ WREG32(mmSMC_IND_DATA_0, (v));
10871101 spin_unlock_irqrestore(&adev->smc_idx_lock, flags);
10881102 }
10891103
@@ -1110,20 +1124,20 @@ static void si_uvd_ctx_wreg(struct amdgpu_device *adev, u32 reg, u32 v) · 20 unmodified lines
11101124 }
11111125
11121126 static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = {
1113− {GRBM_STATUS},
1127+ {mmGRBM_STATUS},
11141128 {mmGRBM_STATUS2},
11151129 {mmGRBM_STATUS_SE0},
11161130 {mmGRBM_STATUS_SE1},
11171131 {mmSRBM_STATUS},
11181132 {mmSRBM_STATUS2},
1119− {DMA_STATUS_REG + DMA0_REGISTER_OFFSET},
1120− {DMA_STATUS_REG + DMA1_REGISTER_OFFSET},
1133+ {mmDMA_STATUS_REG + DMA0_REGISTER_OFFSET},
1134+ {mmDMA_STATUS_REG + DMA1_REGISTER_OFFSET},
11211135 {mmCP_STAT},
11221136 {mmCP_STALLED_STAT1},
11231137 {mmCP_STALLED_STAT2},
11241138 {mmCP_STALLED_STAT3},
1125− {GB_ADDR_CONFIG},
1126− {MC_ARB_RAMCFG},
1139+ {mmGB_ADDR_CONFIG},
1140+ {mmMC_ARB_RAMCFG},
11271141 {mmGB_TILE_MODE0},
11281142 {mmGB_TILE_MODE1},
11291143 {mmGB_TILE_MODE2},
@@ -1156,7 +1170,7 @@ static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = { · 26 unmodified lines
11561170 {mmGB_TILE_MODE29},
11571171 {mmGB_TILE_MODE30},
11581172 {mmGB_TILE_MODE31},
1159− {CC_RB_BACKEND_DISABLE, true},
1173+ {mmCC_RB_BACKEND_DISABLE, true},
11601174 {mmGC_USER_RB_BACKEND_DISABLE, true},
11611175 {mmPA_SC_RASTER_CONFIG, true},
11621176 };
@@ -1264,37 +1278,37 @@ static bool si_read_disabled_bios(struct amdgpu_device *adev) · 101 unmodified lines
12641278 u32 rom_cntl;
12651279 bool r;
12661280
1267− bus_cntl = RREG32(R600_BUS_CNTL);
1281+ bus_cntl = RREG32(mmBUS_CNTL);
12681282 if (adev->mode_info.num_crtc) {
1269− d1vga_control = RREG32(AVIVO_D1VGA_CONTROL);
1270− d2vga_control = RREG32(AVIVO_D2VGA_CONTROL);
1271− vga_render_control = RREG32(VGA_RENDER_CONTROL);
1283+ d1vga_control = RREG32(mmD1VGA_CONTROL);
1284+ d2vga_control = RREG32(mmD2VGA_CONTROL);
1285+ vga_render_control = RREG32(mmVGA_RENDER_CONTROL);
12721286 }
12731287 rom_cntl = RREG32(R600_ROM_CNTL);
12741288
12751289 /* enable the rom */
1276− WREG32(R600_BUS_CNTL, (bus_cntl & ~R600_BIOS_ROM_DIS));
1290+ WREG32(mmBUS_CNTL, (bus_cntl & ~BUS_CNTL__BIOS_ROM_DIS_MASK));
12771291 if (adev->mode_info.num_crtc) {
12781292 /* Disable VGA mode */
1279− WREG32(AVIVO_D1VGA_CONTROL,
1280− (d1vga_control & ~(AVIVO_DVGA_CONTROL_MODE_ENABLE |
1281− AVIVO_DVGA_CONTROL_TIMING_SELECT)));
1282− WREG32(AVIVO_D2VGA_CONTROL,
1283− (d2vga_control & ~(AVIVO_DVGA_CONTROL_MODE_ENABLE |
1284− AVIVO_DVGA_CONTROL_TIMING_SELECT)));
1285− WREG32(VGA_RENDER_CONTROL,
1286− (vga_render_control & C_000300_VGA_VSTATUS_CNTL));
1293+ WREG32(mmD1VGA_CONTROL,
1294+ (d1vga_control & ~(D1VGA_CONTROL__D1VGA_MODE_ENABLE_MASK |
1295+ D1VGA_CONTROL__D1VGA_TIMING_SELECT_MASK)));
1296+ WREG32(mmD2VGA_CONTROL,
1297+ (d2vga_control & ~(D1VGA_CONTROL__D1VGA_MODE_ENABLE_MASK |
1298+ D1VGA_CONTROL__D1VGA_TIMING_SELECT_MASK)));
1299+ WREG32(mmVGA_RENDER_CONTROL,
1300+ (vga_render_control & ~VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK));
12871301 }
12881302 WREG32(R600_ROM_CNTL, rom_cntl | R600_SCK_OVERWRITE);
12891303
12901304 r = amdgpu_read_bios(adev);
12911305
12921306 /* restore regs */
1293− WREG32(R600_BUS_CNTL, bus_cntl);
1307+ WREG32(mmBUS_CNTL, bus_cntl);
12941308 if (adev->mode_info.num_crtc) {
1295− WREG32(AVIVO_D1VGA_CONTROL, d1vga_control);
1296− WREG32(AVIVO_D2VGA_CONTROL, d2vga_control);
1297− WREG32(VGA_RENDER_CONTROL, vga_render_control);
1309+ WREG32(mmD1VGA_CONTROL, d1vga_control);
1310+ WREG32(mmD2VGA_CONTROL, d2vga_control);
1311+ WREG32(mmVGA_RENDER_CONTROL, vga_render_control);
12981312 }
12991313 WREG32(R600_ROM_CNTL, rom_cntl);
13001314 return r;
@@ -1331,23 +1345,24 @@ static void si_set_clk_bypass_mode(struct amdgpu_device *adev) · 30 unmodified lines
13311345 {
13321346 u32 tmp, i;
13331347
1334− tmp = RREG32(CG_SPLL_FUNC_CNTL);
1335− tmp |= SPLL_BYPASS_EN;
1336− WREG32(CG_SPLL_FUNC_CNTL, tmp);
1348+ tmp = RREG32(mmCG_SPLL_FUNC_CNTL);
1349+ tmp |= CG_SPLL_FUNC_CNTL__SPLL_BYPASS_EN_MASK;
1350+ WREG32(mmCG_SPLL_FUNC_CNTL, tmp);
13371351
1338− tmp = RREG32(CG_SPLL_FUNC_CNTL_2);
1339− tmp |= SPLL_CTLREQ_CHG;
1340− WREG32(CG_SPLL_FUNC_CNTL_2, tmp);
1352+ tmp = RREG32(mmCG_SPLL_FUNC_CNTL_2);
1353+ tmp |= CG_SPLL_FUNC_CNTL_2__SPLL_CTLREQ_CHG_MASK;
1354+ WREG32(mmCG_SPLL_FUNC_CNTL_2, tmp);
13411355
13421356 for (i = 0; i < adev->usec_timeout; i++) {
1343− if (RREG32(SPLL_STATUS) & SPLL_CHG_STATUS)
1357+ if (RREG32(mmCG_SPLL_STATUS) & CG_SPLL_STATUS__SPLL_CHG_STATUS_MASK)
13441358 break;
13451359 udelay(1);
13461360 }
13471361
1348− tmp = RREG32(CG_SPLL_FUNC_CNTL_2);
1349− tmp &= ~(SPLL_CTLREQ_CHG | SCLK_MUX_UPDATE);
1350− WREG32(CG_SPLL_FUNC_CNTL_2, tmp);
1362+ tmp = RREG32(mmCG_SPLL_FUNC_CNTL_2);
1363+ tmp &= ~(CG_SPLL_FUNC_CNTL_2__SPLL_CTLREQ_CHG_MASK |
1364+ CG_SPLL_FUNC_CNTL_2__SCLK_MUX_UPDATE_MASK);
1365+ WREG32(mmCG_SPLL_FUNC_CNTL_2, tmp);
13511366
13521367 tmp = RREG32(MPLL_CNTL_MODE);
13531368 tmp &= ~MPLL_MCLK_SEL;
@@ -1358,21 +1373,21 @@ static void si_spll_powerdown(struct amdgpu_device *adev) · 4 unmodified lines
13581373 {
13591374 u32 tmp;
13601375
1361− tmp = RREG32(SPLL_CNTL_MODE);
1362− tmp |= SPLL_SW_DIR_CONTROL;
1363− WREG32(SPLL_CNTL_MODE, tmp);
1376+ tmp = RREG32(mmSPLL_CNTL_MODE);
1377+ tmp |= SPLL_CNTL_MODE__SPLL_SW_DIR_CONTROL_MASK;
1378+ WREG32(mmSPLL_CNTL_MODE, tmp);
13641379
1365− tmp = RREG32(CG_SPLL_FUNC_CNTL);
1366− tmp |= SPLL_RESET;
1367− WREG32(CG_SPLL_FUNC_CNTL, tmp);
1380+ tmp = RREG32(mmCG_SPLL_FUNC_CNTL);
1381+ tmp |= CG_SPLL_FUNC_CNTL__SPLL_RESET_MASK;
1382+ WREG32(mmCG_SPLL_FUNC_CNTL, tmp);
13681383
1369− tmp = RREG32(CG_SPLL_FUNC_CNTL);
1370− tmp |= SPLL_SLEEP;
1371− WREG32(CG_SPLL_FUNC_CNTL, tmp);
1384+ tmp = RREG32(mmCG_SPLL_FUNC_CNTL);
1385+ tmp |= CG_SPLL_FUNC_CNTL__SPLL_SLEEP_MASK;
1386+ WREG32(mmCG_SPLL_FUNC_CNTL, tmp);
13721387
1373− tmp = RREG32(SPLL_CNTL_MODE);
1374− tmp &= ~SPLL_SW_DIR_CONTROL;
1375− WREG32(SPLL_CNTL_MODE, tmp);
1388+ tmp = RREG32(mmSPLL_CNTL_MODE);
1389+ tmp &= ~SPLL_CNTL_MODE__SPLL_SW_DIR_CONTROL_MASK;
1390+ WREG32(mmSPLL_CNTL_MODE, tmp);
13761391 }
13771392
13781393 static int si_gpu_pci_config_reset(struct amdgpu_device *adev)
@@ -1454,14 +1469,14 @@ static void si_vga_set_state(struct amdgpu_device *adev, bool state) · 75 unmodified lines
14541469 {
14551470 uint32_t temp;
14561471
1457− temp = RREG32(CONFIG_CNTL);
1472+ temp = RREG32(mmCONFIG_CNTL);
14581473 if (!state) {
14591474 temp &= ~(1<<0);
14601475 temp |= (1<<1);
14611476 } else {
14621477 temp &= ~(1<<1);
14631478 }
1464− WREG32(CONFIG_CNTL, temp);
1479+ WREG32(mmCONFIG_CNTL, temp);
14651480 }
14661481
14671482 static u32 si_get_xclk(struct amdgpu_device *adev)
@@ -1469,12 +1484,12 @@ static u32 si_get_xclk(struct amdgpu_device *adev) · 1 unmodified line
14691484 u32 reference_clock = adev->clock.spll.reference_freq;
14701485 u32 tmp;
14711486
1472− tmp = RREG32(CG_CLKPIN_CNTL_2);
1473− if (tmp & MUX_TCLK_TO_XCLK)
1487+ tmp = RREG32(mmCG_CLKPIN_CNTL_2);
1488+ if (tmp & CG_CLKPIN_CNTL_2__MUX_TCLK_TO_XCLK_MASK)
14741489 return TCLK;
14751490
1476− tmp = RREG32(CG_CLKPIN_CNTL);
1477− if (tmp & XTALIN_DIVIDE)
1491+ tmp = RREG32(mmCG_CLKPIN_CNTL);
1492+ if (tmp & CG_CLKPIN_CNTL__XTALIN_DIVIDE_MASK)
14781493 return reference_clock / 4;
14791494
14801495 return reference_clock;
@@ -1519,9 +1534,9 @@ static int si_get_pcie_lanes(struct amdgpu_device *adev) · 38 unmodified lines
15191534 if (adev->flags & AMD_IS_APU)
15201535 return 0;
15211536
1522− link_width_cntl = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);
1537+ link_width_cntl = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
15231538
1524− switch ((link_width_cntl & LC_LINK_WIDTH_RD_MASK) >> LC_LINK_WIDTH_RD_SHIFT) {
1539+ switch ((link_width_cntl & PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD_MASK) >> PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD__SHIFT) {
15251540 case LC_LINK_WIDTH_X1:
15261541 return 1;
15271542 case LC_LINK_WIDTH_X2:
@@ -1568,13 +1583,13 @@ static void si_set_pcie_lanes(struct amdgpu_device *adev, int lanes) · 40 unmodified lines
15681583 return;
15691584 }
15701585
1571− link_width_cntl = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);
1572− link_width_cntl &= ~LC_LINK_WIDTH_MASK;
1573− link_width_cntl |= mask << LC_LINK_WIDTH_SHIFT;
1574− link_width_cntl |= (LC_RECONFIG_NOW |
1575− LC_RECONFIG_ARC_MISSING_ESCAPE);
1586+ link_width_cntl = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
1587+ link_width_cntl &= ~PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_MASK;
1588+ link_width_cntl |= mask << PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH__SHIFT;
1589+ link_width_cntl |= (PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_NOW_MASK |
1590+ PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_ARC_MISSING_ESCAPE_MASK);
15761591
1577− WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, link_width_cntl);
1592+ WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, link_width_cntl);
15781593 }
15791594
15801595 static void si_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,
@@ -2018,7 +2033,7 @@ static const struct amdgpu_asic_funcs si_asic_funcs = · 437 unmodified lines
20182033
20192034 static uint32_t si_get_rev_id(struct amdgpu_device *adev)
20202035 {
2021− return (RREG32(CC_DRM_ID_STRAPS) & CC_DRM_ID_STRAPS__ATI_REV_ID_MASK)
2036+ return (RREG32(mmCC_DRM_ID_STRAPS) & CC_DRM_ID_STRAPS__ATI_REV_ID_MASK)
20222037 >> CC_DRM_ID_STRAPS__ATI_REV_ID__SHIFT;
20232038 }
20242039
@@ -2239,9 +2254,9 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 214 unmodified lines
22392254 CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)))
22402255 return;
22412256
2242− speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL);
2243− current_data_rate = (speed_cntl & LC_CURRENT_DATA_RATE_MASK) >>
2244− LC_CURRENT_DATA_RATE_SHIFT;
2257+ speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL);
2258+ current_data_rate = (speed_cntl & PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE_MASK) >>
2259+ PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE__SHIFT;
22452260 if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) {
22462261 if (current_data_rate == 2) {
22472262 DRM_INFO("PCIE gen 3 link speeds already enabled\n");
@@ -2268,17 +2283,17 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 20 unmodified lines
22682283 pcie_capability_set_word(root, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD);
22692284 pcie_capability_set_word(adev->pdev, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD);
22702285
2271− tmp = RREG32_PCIE(PCIE_LC_STATUS1);
2272− max_lw = (tmp & LC_DETECTED_LINK_WIDTH_MASK) >> LC_DETECTED_LINK_WIDTH_SHIFT;
2273− current_lw = (tmp & LC_OPERATING_LINK_WIDTH_MASK) >> LC_OPERATING_LINK_WIDTH_SHIFT;
2286+ tmp = RREG32_PCIE(ixPCIE_LC_STATUS1);
2287+ max_lw = (tmp & PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH_MASK) >> PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH__SHIFT;
2288+ current_lw = (tmp & PCIE_LC_STATUS1__LC_OPERATING_LINK_WIDTH_MASK) >> PCIE_LC_STATUS1__LC_OPERATING_LINK_WIDTH__SHIFT;
22742289
22752290 if (current_lw < max_lw) {
2276− tmp = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);
2277− if (tmp & LC_RENEGOTIATION_SUPPORT) {
2278− tmp &= ~(LC_LINK_WIDTH_MASK | LC_UPCONFIGURE_DIS);
2279− tmp |= (max_lw << LC_LINK_WIDTH_SHIFT);
2280− tmp |= LC_UPCONFIGURE_SUPPORT | LC_RENEGOTIATE_EN | LC_RECONFIG_NOW;
2281− WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, tmp);
2291+ tmp = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
2292+ if (tmp & PCIE_LC_LINK_WIDTH_CNTL__LC_RENEGOTIATION_SUPPORT_MASK) {
2293+ tmp &= ~(PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_UPCONFIGURE_DIS_MASK);
2294+ tmp |= (max_lw << PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH__SHIFT);
2295+ tmp |= PCIE_LC_LINK_WIDTH_CNTL__LC_UPCONFIGURE_SUPPORT_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_RENEGOTIATE_EN_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_NOW_MASK;
2296+ WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, tmp);
22822297 }
22832298 }
22842299
@@ -2301,13 +2316,13 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 16 unmodified lines
23012316 PCI_EXP_LNKCTL2,
23022317 &gpu_cfg2);
23032318
2304− tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4);
2305− tmp |= LC_SET_QUIESCE;
2306− WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp);
2319+ tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4);
2320+ tmp |= PCIE_LC_CNTL4__LC_SET_QUIESCE_MASK;
2321+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp);
23072322
2308− tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4);
2309− tmp |= LC_REDO_EQ;
2310− WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp);
2323+ tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4);
2324+ tmp |= PCIE_LC_CNTL4__LC_REDO_EQ_MASK;
2325+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp);
23112326
23122327 mdelay(100);
23132328
@@ -2333,16 +2348,16 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 19 unmodified lines
23332348 (PCI_EXP_LNKCTL2_ENTER_COMP |
23342349 PCI_EXP_LNKCTL2_TX_MARGIN));
23352350
2336− tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4);
2337− tmp &= ~LC_SET_QUIESCE;
2338− WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp);
2351+ tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4);
2352+ tmp &= ~PCIE_LC_CNTL4__LC_SET_QUIESCE_MASK;
2353+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp);
23392354 }
23402355 }
23412356 }
23422357
2343− speed_cntl |= LC_FORCE_EN_SW_SPEED_CHANGE | LC_FORCE_DIS_HW_SPEED_CHANGE;
2344− speed_cntl &= ~LC_FORCE_DIS_SW_SPEED_CHANGE;
2345− WREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL, speed_cntl);
2358+ speed_cntl |= PCIE_LC_SPEED_CNTL__LC_FORCE_EN_SW_SPEED_CHANGE_MASK | PCIE_LC_SPEED_CNTL__LC_FORCE_DIS_HW_SPEED_CHANGE_MASK;
2359+ speed_cntl &= ~PCIE_LC_SPEED_CNTL__LC_FORCE_DIS_SW_SPEED_CHANGE_MASK;
2360+ WREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL, speed_cntl);
23462361
23472362 tmp16 = 0;
23482363 if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)
@@ -2354,13 +2369,13 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 5 unmodified lines
23542369 pcie_capability_clear_and_set_word(adev->pdev, PCI_EXP_LNKCTL2,
23552370 PCI_EXP_LNKCTL2_TLS, tmp16);
23562371
2357− speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL);
2358− speed_cntl |= LC_INITIATE_LINK_SPEED_CHANGE;
2359− WREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL, speed_cntl);
2372+ speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL);
2373+ speed_cntl |= PCIE_LC_SPEED_CNTL__LC_INITIATE_LINK_SPEED_CHANGE_MASK;
2374+ WREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL, speed_cntl);
23602375
23612376 for (i = 0; i < adev->usec_timeout; i++) {
2362− speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL);
2363− if ((speed_cntl & LC_INITIATE_LINK_SPEED_CHANGE) == 0)
2377+ speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL);
2378+ if ((speed_cntl & PCIE_LC_SPEED_CNTL__LC_INITIATE_LINK_SPEED_CHANGE_MASK) == 0)
23642379 break;
23652380 udelay(1);
23662381 }
@@ -2418,121 +2433,121 @@ static void si_program_aspm(struct amdgpu_device *adev) · 51 unmodified lines
24182433 if (!amdgpu_device_should_use_aspm(adev))
24192434 return;
24202435
2421− orig = data = RREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL);
2422− data &= ~LC_XMIT_N_FTS_MASK;
2423− data |= LC_XMIT_N_FTS(0x24) | LC_XMIT_N_FTS_OVERRIDE_EN;
2436+ orig = data = RREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL);
2437+ data &= ~PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS_MASK;
2438+ data |= (0x24 << PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS__SHIFT) | PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS_OVERRIDE_EN_MASK;
24242439 if (orig != data)
2425− WREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL, data);
2440+ WREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL, data);
24262441
2427− orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL3);
2428− data |= LC_GO_TO_RECOVERY;
2442+ orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL3);
2443+ data |= PCIE_LC_CNTL3__LC_GO_TO_RECOVERY_MASK;
24292444 if (orig != data)
2430− WREG32_PCIE_PORT(PCIE_LC_CNTL3, data);
2445+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL3, data);
24312446
2432− orig = data = RREG32_PCIE(PCIE_P_CNTL);
2433− data |= P_IGNORE_EDB_ERR;
2447+ orig = data = RREG32_PCIE(ixPCIE_P_CNTL);
2448+ data |= PCIE_P_CNTL__P_IGNORE_EDB_ERR_MASK;
24342449 if (orig != data)
2435− WREG32_PCIE(PCIE_P_CNTL, data);
2450+ WREG32_PCIE(ixPCIE_P_CNTL, data);
24362451
2437− orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL);
2438− data &= ~(LC_L0S_INACTIVITY_MASK | LC_L1_INACTIVITY_MASK);
2439− data |= LC_PMI_TO_L1_DIS;
2452+ orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL);
2453+ data &= ~(PCIE_LC_CNTL__LC_L0S_INACTIVITY_MASK | PCIE_LC_CNTL__LC_L1_INACTIVITY_MASK);
2454+ data |= PCIE_LC_CNTL__LC_PMI_TO_L1_DIS_MASK;
24402455 if (!disable_l0s)
2441− data |= LC_L0S_INACTIVITY(7);
2456+ data |= (7 << PCIE_LC_CNTL__LC_L0S_INACTIVITY__SHIFT);
24422457
24432458 if (!disable_l1) {
2444− data |= LC_L1_INACTIVITY(7);
2445− data &= ~LC_PMI_TO_L1_DIS;
2459+ data |= (7 << PCIE_LC_CNTL__LC_L1_INACTIVITY__SHIFT);
2460+ data &= ~PCIE_LC_CNTL__LC_PMI_TO_L1_DIS_MASK;
24462461 if (orig != data)
2447− WREG32_PCIE_PORT(PCIE_LC_CNTL, data);
2462+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data);
24482463
24492464 if (!disable_plloff_in_l1) {
24502465 bool clk_req_support;
24512466
2452− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_0);
2453− data &= ~(PLL_POWER_STATE_IN_OFF_0_MASK | PLL_POWER_STATE_IN_TXS2_0_MASK);
2454− data |= PLL_POWER_STATE_IN_OFF_0(7) | PLL_POWER_STATE_IN_TXS2_0(7);
2467+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_0);
2468+ data &= ~(PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0_MASK | PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0_MASK);
2469+ data |= (7 << PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0__SHIFT) | (7 << PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0__SHIFT);
24552470 if (orig != data)
2456− si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_0, data);
2471+ si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_0, data);
24572472
2458− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_1);
2459− data &= ~(PLL_POWER_STATE_IN_OFF_1_MASK | PLL_POWER_STATE_IN_TXS2_1_MASK);
2460− data |= PLL_POWER_STATE_IN_OFF_1(7) | PLL_POWER_STATE_IN_TXS2_1(7);
2473+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_1);
2474+ data &= ~(PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1_MASK | PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1_MASK);
2475+ data |= (7 << PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1__SHIFT) | (7 << PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1__SHIFT);
24612476 if (orig != data)
2462− si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_1, data);
2477+ si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_1, data);
24632478
2464− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_0);
2465− data &= ~(PLL_POWER_STATE_IN_OFF_0_MASK | PLL_POWER_STATE_IN_TXS2_0_MASK);
2466− data |= PLL_POWER_STATE_IN_OFF_0(7) | PLL_POWER_STATE_IN_TXS2_0(7);
2479+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_0);
2480+ data &= ~(PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0_MASK | PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0_MASK);
2481+ data |= (7 << PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0__SHIFT) | (7 << PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0__SHIFT);
24672482 if (orig != data)
2468− si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_0, data);
2483+ si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_0, data);
24692484
2470− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_1);
2471− data &= ~(PLL_POWER_STATE_IN_OFF_1_MASK | PLL_POWER_STATE_IN_TXS2_1_MASK);
2472− data |= PLL_POWER_STATE_IN_OFF_1(7) | PLL_POWER_STATE_IN_TXS2_1(7);
2485+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_1);
2486+ data &= ~(PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1_MASK | PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1_MASK);
2487+ data |= (7 << PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1__SHIFT) | (7 << PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1__SHIFT);
24732488 if (orig != data)
2474− si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_1, data);
2489+ si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_1, data);
24752490
24762491 if ((adev->asic_type != CHIP_OLAND) && (adev->asic_type != CHIP_HAINAN)) {
2477− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_0);
2478− data &= ~PLL_RAMP_UP_TIME_0_MASK;
2492+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_0);
2493+ data &= ~PB0_PIF_PWRDOWN_0__PLL_RAMP_UP_TIME_0_MASK;
24792494 if (orig != data)
2480− si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_0, data);
2495+ si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_0, data);
24812496
2482− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_1);
2483− data &= ~PLL_RAMP_UP_TIME_1_MASK;
2497+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_1);
2498+ data &= ~PB0_PIF_PWRDOWN_1__PLL_RAMP_UP_TIME_1_MASK;
24842499 if (orig != data)
2485− si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_1, data);
2500+ si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_1, data);
24862501
2487− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_2);
2488− data &= ~PLL_RAMP_UP_TIME_2_MASK;
2502+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_2);
2503+ data &= ~PB0_PIF_PWRDOWN_2__PLL_RAMP_UP_TIME_2_MASK;
24892504 if (orig != data)
2490− si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_2, data);
2505+ si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_2, data);
24912506
2492− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_3);
2493− data &= ~PLL_RAMP_UP_TIME_3_MASK;
2507+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_3);
2508+ data &= ~PB0_PIF_PWRDOWN_3__PLL_RAMP_UP_TIME_3_MASK;
24942509 if (orig != data)
2495− si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_3, data);
2510+ si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_3, data);
24962511
2497− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_0);
2498− data &= ~PLL_RAMP_UP_TIME_0_MASK;
2512+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_0);
2513+ data &= ~PB1_PIF_PWRDOWN_0__PLL_RAMP_UP_TIME_0_MASK;
24992514 if (orig != data)
2500− si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_0, data);
2515+ si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_0, data);
25012516
2502− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_1);
2503− data &= ~PLL_RAMP_UP_TIME_1_MASK;
2517+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_1);
2518+ data &= ~PB1_PIF_PWRDOWN_1__PLL_RAMP_UP_TIME_1_MASK;
25042519 if (orig != data)
2505− si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_1, data);
2520+ si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_1, data);
25062521
2507− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_2);
2508− data &= ~PLL_RAMP_UP_TIME_2_MASK;
2522+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_2);
2523+ data &= ~PB1_PIF_PWRDOWN_2__PLL_RAMP_UP_TIME_2_MASK;
25092524 if (orig != data)
2510− si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_2, data);
2525+ si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_2, data);
25112526
2512− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_3);
2513− data &= ~PLL_RAMP_UP_TIME_3_MASK;
2527+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_3);
2528+ data &= ~PB1_PIF_PWRDOWN_3__PLL_RAMP_UP_TIME_3_MASK;
25142529 if (orig != data)
2515− si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_3, data);
2530+ si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_3, data);
25162531 }
2517− orig = data = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);
2518− data &= ~LC_DYN_LANES_PWR_STATE_MASK;
2519− data |= LC_DYN_LANES_PWR_STATE(3);
2532+ orig = data = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
2533+ data &= ~PCIE_LC_LINK_WIDTH_CNTL__LC_DYN_LANES_PWR_STATE_MASK;
2534+ data |= (3 << PCIE_LC_LINK_WIDTH_CNTL__LC_DYN_LANES_PWR_STATE__SHIFT);
25202535 if (orig != data)
2521− WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, data);
2536+ WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, data);
25222537
2523− orig = data = si_pif_phy0_rreg(adev,PB0_PIF_CNTL);
2524− data &= ~LS2_EXIT_TIME_MASK;
2538+ orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_CNTL);
2539+ data &= ~PB0_PIF_CNTL__LS2_EXIT_TIME_MASK;
25252540 if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN))
2526− data |= LS2_EXIT_TIME(5);
2541+ data |= (5 << PB0_PIF_CNTL__LS2_EXIT_TIME__SHIFT);
25272542 if (orig != data)
2528− si_pif_phy0_wreg(adev,PB0_PIF_CNTL, data);
2543+ si_pif_phy0_wreg(adev,ixPB0_PIF_CNTL, data);
25292544
2530− orig = data = si_pif_phy1_rreg(adev,PB1_PIF_CNTL);
2531− data &= ~LS2_EXIT_TIME_MASK;
2545+ orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_CNTL);
2546+ data &= ~PB1_PIF_CNTL__LS2_EXIT_TIME_MASK;
25322547 if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN))
2533− data |= LS2_EXIT_TIME(5);
2548+ data |= (5 << PB1_PIF_CNTL__LS2_EXIT_TIME__SHIFT);
25342549 if (orig != data)
2535− si_pif_phy1_wreg(adev,PB1_PIF_CNTL, data);
2550+ si_pif_phy1_wreg(adev,ixPB1_PIF_CNTL, data);
25362551
25372552 if (!disable_clkreq &&
25382553 !pci_is_root_bus(adev->pdev->bus)) {
@@ -2548,64 +2563,64 @@ static void si_program_aspm(struct amdgpu_device *adev) · 9 unmodified lines
25482563 }
25492564
25502565 if (clk_req_support) {
2551− orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL2);
2552− data |= LC_ALLOW_PDWN_IN_L1 | LC_ALLOW_PDWN_IN_L23;
2566+ orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL2);
2567+ data |= PCIE_LC_CNTL2__LC_ALLOW_PDWN_IN_L1_MASK | PCIE_LC_CNTL2__LC_ALLOW_PDWN_IN_L23_MASK;
25532568 if (orig != data)
2554− WREG32_PCIE_PORT(PCIE_LC_CNTL2, data);
2569+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL2, data);
25552570
2556− orig = data = RREG32(THM_CLK_CNTL);
2557− data &= ~(CMON_CLK_SEL_MASK | TMON_CLK_SEL_MASK);
2558− data |= CMON_CLK_SEL(1) | TMON_CLK_SEL(1);
2571+ orig = data = RREG32(mmTHM_CLK_CNTL);
2572+ data &= ~(THM_CLK_CNTL__CMON_CLK_SEL_MASK | THM_CLK_CNTL__TMON_CLK_SEL_MASK);
2573+ data |= (1 << THM_CLK_CNTL__CMON_CLK_SEL__SHIFT) | (1 << THM_CLK_CNTL__TMON_CLK_SEL__SHIFT);
25592574 if (orig != data)
2560− WREG32(THM_CLK_CNTL, data);
2575+ WREG32(mmTHM_CLK_CNTL, data);
25612576
2562− orig = data = RREG32(MISC_CLK_CNTL);
2563− data &= ~(DEEP_SLEEP_CLK_SEL_MASK | ZCLK_SEL_MASK);
2564− data |= DEEP_SLEEP_CLK_SEL(1) | ZCLK_SEL(1);
2577+ orig = data = RREG32(mmMISC_CLK_CNTL);
2578+ data &= ~(MISC_CLK_CNTL__DEEP_SLEEP_CLK_SEL_MASK | MISC_CLK_CNTL__ZCLK_SEL_MASK);
2579+ data |= (1 << MISC_CLK_CNTL__DEEP_SLEEP_CLK_SEL__SHIFT) | (1 << MISC_CLK_CNTL__ZCLK_SEL__SHIFT);
25652580 if (orig != data)
2566− WREG32(MISC_CLK_CNTL, data);
2581+ WREG32(mmMISC_CLK_CNTL, data);
25672582
2568− orig = data = RREG32(CG_CLKPIN_CNTL);
2569− data &= ~BCLK_AS_XCLK;
2583+ orig = data = RREG32(mmCG_CLKPIN_CNTL);
2584+ data &= ~CG_CLKPIN_CNTL__BCLK_AS_XCLK_MASK;
25702585 if (orig != data)
2571− WREG32(CG_CLKPIN_CNTL, data);
2586+ WREG32(mmCG_CLKPIN_CNTL, data);
25722587
2573− orig = data = RREG32(CG_CLKPIN_CNTL_2);
2574− data &= ~FORCE_BIF_REFCLK_EN;
2588+ orig = data = RREG32(mmCG_CLKPIN_CNTL_2);
2589+ data &= ~CG_CLKPIN_CNTL_2__FORCE_BIF_REFCLK_EN_MASK;
25752590 if (orig != data)
2576− WREG32(CG_CLKPIN_CNTL_2, data);
2591+ WREG32(mmCG_CLKPIN_CNTL_2, data);
25772592
2578− orig = data = RREG32(MPLL_BYPASSCLK_SEL);
2579− data &= ~MPLL_CLKOUT_SEL_MASK;
2580− data |= MPLL_CLKOUT_SEL(4);
2593+ orig = data = RREG32(mmMPLL_BYPASSCLK_SEL);
2594+ data &= ~MPLL_BYPASSCLK_SEL__MPLL_CLKOUT_SEL_MASK;
2595+ data |= 4 << MPLL_BYPASSCLK_SEL__MPLL_CLKOUT_SEL__SHIFT;
25812596 if (orig != data)
2582− WREG32(MPLL_BYPASSCLK_SEL, data);
2597+ WREG32(mmMPLL_BYPASSCLK_SEL, data);
25832598
2584− orig = data = RREG32(SPLL_CNTL_MODE);
2585− data &= ~SPLL_REFCLK_SEL_MASK;
2599+ orig = data = RREG32(mmSPLL_CNTL_MODE);
2600+ data &= ~SPLL_CNTL_MODE__SPLL_REFCLK_SEL_MASK;
25862601 if (orig != data)
2587− WREG32(SPLL_CNTL_MODE, data);
2602+ WREG32(mmSPLL_CNTL_MODE, data);
25882603 }
25892604 }
25902605 } else {
25912606 if (orig != data)
2592− WREG32_PCIE_PORT(PCIE_LC_CNTL, data);
2607+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data);
25932608 }
25942609
2595− orig = data = RREG32_PCIE(PCIE_CNTL2);
2596− data |= SLV_MEM_LS_EN | MST_MEM_LS_EN | REPLAY_MEM_LS_EN;
2610+ orig = data = RREG32_PCIE(ixPCIE_CNTL2);
2611+ data |= PCIE_CNTL2__SLV_MEM_LS_EN_MASK | PCIE_CNTL2__MST_MEM_LS_EN_MASK | PCIE_CNTL2__REPLAY_MEM_LS_EN_MASK;
25972612 if (orig != data)
2598− WREG32_PCIE(PCIE_CNTL2, data);
2613+ WREG32_PCIE(ixPCIE_CNTL2, data);
25992614
26002615 if (!disable_l0s) {
2601− data = RREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL);
2602− if((data & LC_N_FTS_MASK) == LC_N_FTS_MASK) {
2603− data = RREG32_PCIE(PCIE_LC_STATUS1);
2604− if ((data & LC_REVERSE_XMIT) && (data & LC_REVERSE_RCVR)) {
2605− orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL);
2606− data &= ~LC_L0S_INACTIVITY_MASK;
2616+ data = RREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL);
2617+ if((data & PCIE_LC_N_FTS_CNTL__LC_N_FTS_MASK) == PCIE_LC_N_FTS_CNTL__LC_N_FTS_MASK) {
2618+ data = RREG32_PCIE(ixPCIE_LC_STATUS1);
2619+ if ((data & PCIE_LC_STATUS1__LC_REVERSE_XMIT_MASK) && (data & PCIE_LC_STATUS1__LC_REVERSE_RCVR_MASK)) {
2620+ orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL);
2621+ data &= ~PCIE_LC_CNTL__LC_L0S_INACTIVITY_MASK;
26072622 if (orig != data)
2608− WREG32_PCIE_PORT(PCIE_LC_CNTL, data);
2623+ WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data);
26092624 }
26102625 }
26112626 }
drivers/gpu/drm/amd/amdgpu/si_dma.c+66−50
@@ -27,6 +27,8 @@ · 26 unmodified lines
2727 #include "si.h"
2828 #include "sid.h"
2929
30+#include "oss/oss_1_0_d.h"
31+#include "oss/oss_1_0_sh_mask.h"
3032 const u32 sdma_offsets[SDMA_MAX_INSTANCE] =
3133 {
3234 DMA0_REGISTER_OFFSET,
@@ -38,17 +40,31 @@ static void si_dma_set_buffer_funcs(struct amdgpu_device *adev); · 5 unmodified lines
3840 static void si_dma_set_vm_pte_funcs(struct amdgpu_device *adev);
3941 static void si_dma_set_irq_funcs(struct amdgpu_device *adev);
4042
43+/**
44+ * si_dma_ring_get_rptr - get the current read pointer
45+ *
46+ * @ring: amdgpu ring pointer
47+ *
48+ * Get the current rptr from the hardware (SI).
49+ */
4150 static uint64_t si_dma_ring_get_rptr(struct amdgpu_ring *ring)
4251 {
4352 return *ring->rptr_cpu_addr;
4453 }
4554
55+/**
56+ * si_dma_ring_get_wptr - get the current write pointer
57+ *
58+ * @ring: amdgpu ring pointer
59+ *
60+ * Get the current wptr from the hardware (SI).
61+ */
4662 static uint64_t si_dma_ring_get_wptr(struct amdgpu_ring *ring)
4763 {
4864 struct amdgpu_device *adev = ring->adev;
4965 u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1;
5066
51− return (RREG32(DMA_RB_WPTR + sdma_offsets[me]) & 0x3fffc) >> 2;
67+ return (RREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[me]) & 0x3fffc) >> 2;
5268 }
5369
5470 static void si_dma_ring_set_wptr(struct amdgpu_ring *ring)
@@ -56,7 +72,7 @@ static void si_dma_ring_set_wptr(struct amdgpu_ring *ring) · 1 unmodified line
5672 struct amdgpu_device *adev = ring->adev;
5773 u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1;
5874
59− WREG32(DMA_RB_WPTR + sdma_offsets[me], (ring->wptr << 2) & 0x3fffc);
75+ WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[me], (ring->wptr << 2) & 0x3fffc);
6076 }
6177
6278 static void si_dma_ring_emit_ib(struct amdgpu_ring *ring,
@@ -117,9 +133,9 @@ static void si_dma_stop(struct amdgpu_device *adev) · 54 unmodified lines
117133
118134 for (i = 0; i < adev->sdma.num_instances; i++) {
119135 /* dma0 */
120− rb_cntl = RREG32(DMA_RB_CNTL + sdma_offsets[i]);
121− rb_cntl &= ~DMA_RB_ENABLE;
122− WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl);
136+ rb_cntl = RREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i]);
137+ rb_cntl &= ~DMA_GFX_RB_CNTL__RB_ENABLE_MASK;
138+ WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl);
123139 }
124140 }
125141
@@ -133,44 +149,44 @@ static int si_dma_start(struct amdgpu_device *adev) · 7 unmodified lines
133149 for (i = 0; i < adev->sdma.num_instances; i++) {
134150 ring = &adev->sdma.instance[i].ring;
135151
136− WREG32(DMA_SEM_INCOMPLETE_TIMER_CNTL + sdma_offsets[i], 0);
137− WREG32(DMA_SEM_WAIT_FAIL_TIMER_CNTL + sdma_offsets[i], 0);
152+ WREG32(mmDMA_SEM_INCOMPLETE_TIMER_CNTL + sdma_offsets[i], 0);
153+ WREG32(mmDMA_SEM_WAIT_FAIL_TIMER_CNTL + sdma_offsets[i], 0);
138154
139155 /* Set ring buffer size in dwords */
140156 rb_bufsz = order_base_2(ring->ring_size / 4);
141157 rb_cntl = rb_bufsz << 1;
142158 #ifdef __BIG_ENDIAN
143− rb_cntl |= DMA_RB_SWAP_ENABLE | DMA_RPTR_WRITEBACK_SWAP_ENABLE;
159+ rb_cntl |= DMA_GFX_RB_CNTL__RB_SWAP_ENABLE_MASK | DMA_GFX_RB_CNTL__RPTR_WRITEBACK_SWAP_ENABLE_MASK;
144160 #endif
145− WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl);
161+ WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl);
146162
147163 /* Initialize the ring buffer's read and write pointers */
148− WREG32(DMA_RB_RPTR + sdma_offsets[i], 0);
149− WREG32(DMA_RB_WPTR + sdma_offsets[i], 0);
164+ WREG32(mmDMA_GFX_RB_RPTR + sdma_offsets[i], 0);
165+ WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[i], 0);
150166
151167 rptr_addr = ring->rptr_gpu_addr;
152168
153− WREG32(DMA_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr));
154− WREG32(DMA_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF);
169+ WREG32(mmDMA_GFX_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr));
170+ WREG32(mmDMA_GFX_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF);
155171
156− rb_cntl |= DMA_RPTR_WRITEBACK_ENABLE;
172+ rb_cntl |= DMA_GFX_RB_CNTL__RPTR_WRITEBACK_ENABLE_MASK;
157173
158− WREG32(DMA_RB_BASE + sdma_offsets[i], ring->gpu_addr >> 8);
174+ WREG32(mmDMA_GFX_RB_BASE + sdma_offsets[i], ring->gpu_addr >> 8);
159175
160176 /* enable DMA IBs */
161− ib_cntl = DMA_IB_ENABLE | CMD_VMID_FORCE;
177+ ib_cntl = DMA_GFX_IB_CNTL__IB_ENABLE_MASK | DMA_GFX_IB_CNTL__CMD_VMID_FORCE_MASK;
162178 #ifdef __BIG_ENDIAN
163− ib_cntl |= DMA_IB_SWAP_ENABLE;
179+ ib_cntl |= DMA_GFX_IB_CNTL__IB_SWAP_ENABLE_MASK;
164180 #endif
165− WREG32(DMA_IB_CNTL + sdma_offsets[i], ib_cntl);
181+ WREG32(mmDMA_GFX_IB_CNTL + sdma_offsets[i], ib_cntl);
166182
167− dma_cntl = RREG32(DMA_CNTL + sdma_offsets[i]);
168− dma_cntl &= ~CTXEMPTY_INT_ENABLE;
169− WREG32(DMA_CNTL + sdma_offsets[i], dma_cntl);
183+ dma_cntl = RREG32(mmDMA_CNTL + sdma_offsets[i]);
184+ dma_cntl &= ~DMA_CNTL__CTXEMPTY_INT_ENABLE_MASK;
185+ WREG32(mmDMA_CNTL + sdma_offsets[i], dma_cntl);
170186
171187 ring->wptr = 0;
172− WREG32(DMA_RB_WPTR + sdma_offsets[i], ring->wptr << 2);
173− WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl | DMA_RB_ENABLE);
188+ WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[i], ring->wptr << 2);
189+ WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl | DMA_GFX_RB_CNTL__RB_ENABLE_MASK);
174190
175191 r = amdgpu_ring_test_helper(ring);
176192 if (r)
@@ -461,7 +477,7 @@ static int si_dma_early_init(struct amdgpu_ip_block *ip_block) · 284 unmodified lines
461477 {
462478 struct amdgpu_device *adev = ip_block->adev;
463479
464− adev->sdma.num_instances = 2;
480+ adev->sdma.num_instances = SDMA_MAX_INSTANCE;
465481
466482 si_dma_set_ring_funcs(adev);
467483 si_dma_set_buffer_funcs(adev);
@@ -545,9 +561,9 @@ static bool si_dma_is_idle(struct amdgpu_ip_block *ip_block) · 77 unmodified lines
545561 {
546562 struct amdgpu_device *adev = ip_block->adev;
547563
548− u32 tmp = RREG32(SRBM_STATUS2);
564+ u32 tmp = RREG32(mmSRBM_STATUS2);
549565
550− if (tmp & (DMA_BUSY_MASK | DMA1_BUSY_MASK))
566+ if (tmp & (SRBM_STATUS2__DMA_BUSY_MASK | SRBM_STATUS2__DMA1_BUSY_MASK))
551567 return false;
552568
553569 return true;
@@ -583,14 +599,14 @@ static int si_dma_set_trap_irq_state(struct amdgpu_device *adev, · 29 unmodified lines
583599 case AMDGPU_SDMA_IRQ_INSTANCE0:
584600 switch (state) {
585601 case AMDGPU_IRQ_STATE_DISABLE:
586− sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET);
587− sdma_cntl &= ~TRAP_ENABLE;
588− WREG32(DMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);
602+ sdma_cntl = RREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET);
603+ sdma_cntl &= ~DMA_CNTL__TRAP_ENABLE_MASK;
604+ WREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);
589605 break;
590606 case AMDGPU_IRQ_STATE_ENABLE:
591− sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET);
592− sdma_cntl |= TRAP_ENABLE;
593− WREG32(DMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);
607+ sdma_cntl = RREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET);
608+ sdma_cntl |= DMA_CNTL__TRAP_ENABLE_MASK;
609+ WREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);
594610 break;
595611 default:
596612 break;
@@ -599,14 +615,14 @@ static int si_dma_set_trap_irq_state(struct amdgpu_device *adev, · 2 unmodified lines
599615 case AMDGPU_SDMA_IRQ_INSTANCE1:
600616 switch (state) {
601617 case AMDGPU_IRQ_STATE_DISABLE:
602− sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET);
603− sdma_cntl &= ~TRAP_ENABLE;
604− WREG32(DMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);
618+ sdma_cntl = RREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET);
619+ sdma_cntl &= ~DMA_CNTL__TRAP_ENABLE_MASK;
620+ WREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);
605621 break;
606622 case AMDGPU_IRQ_STATE_ENABLE:
607− sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET);
608− sdma_cntl |= TRAP_ENABLE;
609− WREG32(DMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);
623+ sdma_cntl = RREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET);
624+ sdma_cntl |= DMA_CNTL__TRAP_ENABLE_MASK;
625+ WREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);
610626 break;
611627 default:
612628 break;
@@ -645,11 +661,11 @@ static int si_dma_set_clockgating_state(struct amdgpu_ip_block *ip_block, · 32 unmodified lines
645661 offset = DMA0_REGISTER_OFFSET;
646662 else
647663 offset = DMA1_REGISTER_OFFSET;
648− orig = data = RREG32(DMA_POWER_CNTL + offset);
649− data &= ~MEM_POWER_OVERRIDE;
664+ orig = data = RREG32(mmDMA_POWER_CNTL + offset);
665+ data &= ~DMA_POWER_CNTL__MEM_POWER_OVERRIDE_MASK;
650666 if (data != orig)
651− WREG32(DMA_POWER_CNTL + offset, data);
652− WREG32(DMA_CLK_CTRL + offset, 0x00000100);
667+ WREG32(mmDMA_POWER_CNTL + offset, data);
668+ WREG32(mmDMA_CLK_CTRL + offset, 0x00000100);
653669 }
654670 } else {
655671 for (i = 0; i < adev->sdma.num_instances; i++) {
@@ -657,15 +673,15 @@ static int si_dma_set_clockgating_state(struct amdgpu_ip_block *ip_block, · 1 unmodified line
657673 offset = DMA0_REGISTER_OFFSET;
658674 else
659675 offset = DMA1_REGISTER_OFFSET;
660− orig = data = RREG32(DMA_POWER_CNTL + offset);
661− data |= MEM_POWER_OVERRIDE;
676+ orig = data = RREG32(mmDMA_POWER_CNTL + offset);
677+ data |= DMA_POWER_CNTL__MEM_POWER_OVERRIDE_MASK;
662678 if (data != orig)
663− WREG32(DMA_POWER_CNTL + offset, data);
679+ WREG32(mmDMA_POWER_CNTL + offset, data);
664680
665− orig = data = RREG32(DMA_CLK_CTRL + offset);
681+ orig = data = RREG32(mmDMA_CLK_CTRL + offset);
666682 data = 0xff000000;
667683 if (data != orig)
668− WREG32(DMA_CLK_CTRL + offset, data);
684+ WREG32(mmDMA_CLK_CTRL + offset, data);
669685 }
670686 }
671687
@@ -679,11 +695,11 @@ static int si_dma_set_powergating_state(struct amdgpu_ip_block *ip_block, · 7 unmodified lines
679695
680696 struct amdgpu_device *adev = ip_block->adev;
681697
682− WREG32(DMA_PGFSM_WRITE, 0x00002000);
683− WREG32(DMA_PGFSM_CONFIG, 0x100010ff);
698+ WREG32(mmDMA_PGFSM_WRITE, 0x00002000);
699+ WREG32(mmDMA_PGFSM_CONFIG, 0x100010ff);
684700
685701 for (tmp = 0; tmp < 5; tmp++)
686− WREG32(DMA_PGFSM_WRITE, 0);
702+ WREG32(mmDMA_PGFSM_WRITE, 0);
687703
688704 return 0;
689705 }
drivers/gpu/drm/amd/amdgpu/si_enums.h+1−233
@@ -23,115 +23,15 @@ · 22 unmodified lines
2323 #ifndef SI_ENUMS_H
2424 #define SI_ENUMS_H
2525
26−#define VBLANK_INT_MASK (1 << 0)
27−#define DC_HPDx_INT_EN (1 << 16)
28−#define VBLANK_ACK (1 << 4)
29−#define VLINE_ACK (1 << 4)
30−
31−#define CURSOR_WIDTH 64
32−#define CURSOR_HEIGHT 64
33−
34−#define VGA_VSTATUS_CNTL 0xFFFCFFFF
3526 #define PRIORITY_MARK_MASK 0x7fff
3627 #define PRIORITY_OFF (1 << 16)
3728 #define PRIORITY_ALWAYS_ON (1 << 20)
38−#define INTERLEAVE_EN (1 << 0)
39−
40−#define LATENCY_WATERMARK_MASK(x) ((x) << 16)
41−#define DC_LB_MEMORY_CONFIG(x) ((x) << 20)
42−#define ICON_DEGAMMA_MODE(x) (((x) & 0x3) << 8)
43−
44−#define GRPH_ENDIAN_SWAP(x) (((x) & 0x3) << 0)
45−#define GRPH_ENDIAN_NONE 0
46−#define GRPH_ENDIAN_8IN16 1
47−#define GRPH_ENDIAN_8IN32 2
48−#define GRPH_ENDIAN_8IN64 3
49−#define GRPH_RED_CROSSBAR(x) (((x) & 0x3) << 4)
50−#define GRPH_RED_SEL_R 0
51−#define GRPH_RED_SEL_G 1
52−#define GRPH_RED_SEL_B 2
53−#define GRPH_RED_SEL_A 3
54−#define GRPH_GREEN_CROSSBAR(x) (((x) & 0x3) << 6)
55−#define GRPH_GREEN_SEL_G 0
56−#define GRPH_GREEN_SEL_B 1
57−#define GRPH_GREEN_SEL_A 2
58−#define GRPH_GREEN_SEL_R 3
59−#define GRPH_BLUE_CROSSBAR(x) (((x) & 0x3) << 8)
60−#define GRPH_BLUE_SEL_B 0
61−#define GRPH_BLUE_SEL_A 1
62−#define GRPH_BLUE_SEL_R 2
63−#define GRPH_BLUE_SEL_G 3
64−#define GRPH_ALPHA_CROSSBAR(x) (((x) & 0x3) << 10)
65−#define GRPH_ALPHA_SEL_A 0
66−#define GRPH_ALPHA_SEL_R 1
67−#define GRPH_ALPHA_SEL_G 2
68−#define GRPH_ALPHA_SEL_B 3
69−
70−#define GRPH_DEPTH(x) (((x) & 0x3) << 0)
71−#define GRPH_DEPTH_8BPP 0
72−#define GRPH_DEPTH_16BPP 1
73−#define GRPH_DEPTH_32BPP 2
74−
75−#define GRPH_FORMAT(x) (((x) & 0x7) << 8)
76−#define GRPH_FORMAT_INDEXED 0
77−#define GRPH_FORMAT_ARGB1555 0
78−#define GRPH_FORMAT_ARGB565 1
79−#define GRPH_FORMAT_ARGB4444 2
80−#define GRPH_FORMAT_AI88 3
81−#define GRPH_FORMAT_MONO16 4
82−#define GRPH_FORMAT_BGRA5551 5
83−#define GRPH_FORMAT_ARGB8888 0
84−#define GRPH_FORMAT_ARGB2101010 1
85−#define GRPH_FORMAT_32BPP_DIG 2
86−#define GRPH_FORMAT_8B_ARGB2101010 3
87−#define GRPH_FORMAT_BGRA1010102 4
88−#define GRPH_FORMAT_8B_BGRA1010102 5
89−#define GRPH_FORMAT_RGB111110 6
90−#define GRPH_FORMAT_BGR101111 7
91−
92−#define GRPH_NUM_BANKS(x) (((x) & 0x3) << 2)
93−#define GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20)
94−#define GRPH_ARRAY_LINEAR_GENERAL 0
95−#define GRPH_ARRAY_LINEAR_ALIGNED 1
96−#define GRPH_ARRAY_1D_TILED_THIN1 2
97−#define GRPH_ARRAY_2D_TILED_THIN1 4
98−#define GRPH_TILE_SPLIT(x) (((x) & 0x7) << 13)
99−#define GRPH_BANK_WIDTH(x) (((x) & 0x3) << 6)
100−#define GRPH_BANK_HEIGHT(x) (((x) & 0x3) << 11)
101−#define GRPH_MACRO_TILE_ASPECT(x) (((x) & 0x3) << 18)
102−#define GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20)
103−#define GRPH_PIPE_CONFIG(x) (((x) & 0x1f) << 24)
104−
105−#define CURSOR_EN (1 << 0)
106−#define CURSOR_MODE(x) (((x) & 0x3) << 8)
107−#define CURSOR_MONO 0
108−#define CURSOR_24_1 1
109−#define CURSOR_24_8_PRE_MULT 2
110−#define CURSOR_24_8_UNPRE_MULT 3
111−#define CURSOR_2X_MAGNIFY (1 << 16)
112−#define CURSOR_FORCE_MC_ON (1 << 20)
113−#define CURSOR_URGENT_CONTROL(x) (((x) & 0x7) << 24)
114−#define CURSOR_URGENT_ALWAYS 0
115−#define CURSOR_URGENT_1_8 1
116−#define CURSOR_URGENT_1_4 2
117−#define CURSOR_URGENT_3_8 3
118−#define CURSOR_URGENT_1_2 4
119−#define CURSOR_UPDATE_PENDING (1 << 0)
120−#define CURSOR_UPDATE_TAKEN (1 << 1)
121−#define CURSOR_UPDATE_LOCK (1 << 16)
122−#define CURSOR_DISABLE_MULTIPLE_UPDATE (1 << 24)
123−
124−
125−#define ES_AND_GS_AUTO 3
126−#define RADEON_PACKET_TYPE3 3
127−#define CE_PARTITION_BASE 3
128−#define BUF_SWAP_32BIT (2 << 16)
12929
13030 #define GFX_POWER_STATUS (1 << 1)
13131 #define GFX_CLOCK_STATUS (1 << 2)
13232 #define GFX_LS_STATUS (1 << 3)
133−#define RLC_BUSY_STATUS (1 << 0)
13433
34+#define RLC_BUSY_STATUS (1 << 0)
13535 #define RLC_PUD(x) ((x) << 0)
13636 #define RLC_PUD_MASK (0xff << 0)
13737 #define RLC_PDD(x) ((x) << 8)
@@ -140,140 +40,8 @@ · 2 unmodified lines
14040 #define RLC_TTPD_MASK (0xff << 16)
14141 #define RLC_MSD(x) ((x) << 24)
14242 #define RLC_MSD_MASK (0xff << 24)
143−#define WRITE_DATA_ENGINE_SEL(x) ((x) << 30)
144−#define WRITE_DATA_DST_SEL(x) ((x) << 8)
145−#define EVENT_TYPE(x) ((x) << 0)
146−#define EVENT_INDEX(x) ((x) << 8)
147−#define WAIT_REG_MEM_MEM_SPACE(x) ((x) << 4)
148−#define WAIT_REG_MEM_FUNCTION(x) ((x) << 0)
149−#define WAIT_REG_MEM_ENGINE(x) ((x) << 8)
15043
151−#define GFX6_NUM_GFX_RINGS 1
152−#define GFX6_NUM_COMPUTE_RINGS 2
15344 #define RLC_SAVE_AND_RESTORE_STARTING_OFFSET 0x90
15445 #define RLC_CLEAR_STATE_DESCRIPTOR_OFFSET 0x3D
15546
156−#define PACKET3(op, n) ((RADEON_PACKET_TYPE3 << 30) | \
157− (((op) & 0xFF) << 8) | \
158− ((n) & 0x3FFF) << 16)
159−#define PACKET3_COMPUTE(op, n) (PACKET3(op, n) | 1 << 1)
160−#define PACKET3_NOP 0x10
161−#define PACKET3_SET_BASE 0x11
162−#define PACKET3_BASE_INDEX(x) ((x) << 0)
163−#define PACKET3_CLEAR_STATE 0x12
164−#define PACKET3_INDEX_BUFFER_SIZE 0x13
165−#define PACKET3_DISPATCH_DIRECT 0x15
166−#define PACKET3_DISPATCH_INDIRECT 0x16
167−#define PACKET3_ALLOC_GDS 0x1B
168−#define PACKET3_WRITE_GDS_RAM 0x1C
169−#define PACKET3_ATOMIC_GDS 0x1D
170−#define PACKET3_ATOMIC 0x1E
171−#define PACKET3_OCCLUSION_QUERY 0x1F
172−#define PACKET3_SET_PREDICATION 0x20
173−#define PACKET3_REG_RMW 0x21
174−#define PACKET3_COND_EXEC 0x22
175−#define PACKET3_PRED_EXEC 0x23
176−#define PACKET3_DRAW_INDIRECT 0x24
177−#define PACKET3_DRAW_INDEX_INDIRECT 0x25
178−#define PACKET3_INDEX_BASE 0x26
179−#define PACKET3_DRAW_INDEX_2 0x27
180−#define PACKET3_CONTEXT_CONTROL 0x28
181−#define PACKET3_INDEX_TYPE 0x2A
182−#define PACKET3_DRAW_INDIRECT_MULTI 0x2C
183−#define PACKET3_DRAW_INDEX_AUTO 0x2D
184−#define PACKET3_DRAW_INDEX_IMMD 0x2E
185−#define PACKET3_NUM_INSTANCES 0x2F
186−#define PACKET3_DRAW_INDEX_MULTI_AUTO 0x30
187−#define PACKET3_INDIRECT_BUFFER_CONST 0x31
188−#define PACKET3_INDIRECT_BUFFER 0x3F
189−#define PACKET3_STRMOUT_BUFFER_UPDATE 0x34
190−#define PACKET3_DRAW_INDEX_OFFSET_2 0x35
191−#define PACKET3_DRAW_INDEX_MULTI_ELEMENT 0x36
192−#define PACKET3_WRITE_DATA 0x37
193−#define PACKET3_DRAW_INDEX_INDIRECT_MULTI 0x38
194−#define PACKET3_MEM_SEMAPHORE 0x39
195−#define PACKET3_MPEG_INDEX 0x3A
196−#define PACKET3_COPY_DW 0x3B
197−#define PACKET3_WAIT_REG_MEM 0x3C
198−#define PACKET3_MEM_WRITE 0x3D
199−#define PACKET3_COPY_DATA 0x40
200−#define PACKET3_CP_DMA 0x41
201−# define PACKET3_CP_DMA_DST_SEL(x) ((x) << 20)
202−# define PACKET3_CP_DMA_ENGINE(x) ((x) << 27)
203−# define PACKET3_CP_DMA_SRC_SEL(x) ((x) << 29)
204−# define PACKET3_CP_DMA_CP_SYNC (1 << 31)
205−# define PACKET3_CP_DMA_DIS_WC (1 << 21)
206−# define PACKET3_CP_DMA_CMD_SRC_SWAP(x) ((x) << 22)
207−# define PACKET3_CP_DMA_CMD_DST_SWAP(x) ((x) << 24)
208−# define PACKET3_CP_DMA_CMD_SAS (1 << 26)
209−# define PACKET3_CP_DMA_CMD_DAS (1 << 27)
210−# define PACKET3_CP_DMA_CMD_SAIC (1 << 28)
211−# define PACKET3_CP_DMA_CMD_DAIC (1 << 29)
212−# define PACKET3_CP_DMA_CMD_RAW_WAIT (1 << 30)
213−#define PACKET3_PFP_SYNC_ME 0x42
214−#define PACKET3_SURFACE_SYNC 0x43
215−# define PACKET3_DEST_BASE_0_ENA (1 << 0)
216−# define PACKET3_DEST_BASE_1_ENA (1 << 1)
217−# define PACKET3_CB0_DEST_BASE_ENA (1 << 6)
218−# define PACKET3_CB1_DEST_BASE_ENA (1 << 7)
219−# define PACKET3_CB2_DEST_BASE_ENA (1 << 8)
220−# define PACKET3_CB3_DEST_BASE_ENA (1 << 9)
221−# define PACKET3_CB4_DEST_BASE_ENA (1 << 10)
222−# define PACKET3_CB5_DEST_BASE_ENA (1 << 11)
223−# define PACKET3_CB6_DEST_BASE_ENA (1 << 12)
224−# define PACKET3_CB7_DEST_BASE_ENA (1 << 13)
225−# define PACKET3_DB_DEST_BASE_ENA (1 << 14)
226−# define PACKET3_DEST_BASE_2_ENA (1 << 19)
227−# define PACKET3_DEST_BASE_3_ENA (1 << 21)
228−# define PACKET3_TCL1_ACTION_ENA (1 << 22)
229−# define PACKET3_TC_ACTION_ENA (1 << 23)
230−# define PACKET3_CB_ACTION_ENA (1 << 25)
231−# define PACKET3_DB_ACTION_ENA (1 << 26)
232−# define PACKET3_SH_KCACHE_ACTION_ENA (1 << 27)
233−# define PACKET3_SH_ICACHE_ACTION_ENA (1 << 29)
234−#define PACKET3_ME_INITIALIZE 0x44
235−#define PACKET3_ME_INITIALIZE_DEVICE_ID(x) ((x) << 16)
236−#define PACKET3_COND_WRITE 0x45
237−#define PACKET3_EVENT_WRITE 0x46
238−#define PACKET3_EVENT_WRITE_EOP 0x47
239−#define PACKET3_EVENT_WRITE_EOS 0x48
240−#define PACKET3_PREAMBLE_CNTL 0x4A
241−# define PACKET3_PREAMBLE_BEGIN_CLEAR_STATE (2 << 28)
242−# define PACKET3_PREAMBLE_END_CLEAR_STATE (3 << 28)
243−#define PACKET3_ONE_REG_WRITE 0x57
244−#define PACKET3_LOAD_CONFIG_REG 0x5F
245−#define PACKET3_LOAD_CONTEXT_REG 0x60
246−#define PACKET3_LOAD_SH_REG 0x61
247−#define PACKET3_SET_CONFIG_REG 0x68
248−#define PACKET3_SET_CONFIG_REG_START 0x00002000
249−#define PACKET3_SET_CONFIG_REG_END 0x00002c00
250−#define PACKET3_SET_CONTEXT_REG 0x69
251−#define PACKET3_SET_CONTEXT_REG_START 0x000a000
252−#define PACKET3_SET_CONTEXT_REG_END 0x000a400
253−#define PACKET3_SET_CONTEXT_REG_INDIRECT 0x73
254−#define PACKET3_SET_RESOURCE_INDIRECT 0x74
255−#define PACKET3_SET_SH_REG 0x76
256−#define PACKET3_SET_SH_REG_START 0x00002c00
257−#define PACKET3_SET_SH_REG_END 0x00003000
258−#define PACKET3_SET_SH_REG_OFFSET 0x77
259−#define PACKET3_ME_WRITE 0x7A
260−#define PACKET3_SCRATCH_RAM_WRITE 0x7D
261−#define PACKET3_SCRATCH_RAM_READ 0x7E
262−#define PACKET3_CE_WRITE 0x7F
263−#define PACKET3_LOAD_CONST_RAM 0x80
264−#define PACKET3_WRITE_CONST_RAM 0x81
265−#define PACKET3_WRITE_CONST_RAM_OFFSET 0x82
266−#define PACKET3_DUMP_CONST_RAM 0x83
267−#define PACKET3_INCREMENT_CE_COUNTER 0x84
268−#define PACKET3_INCREMENT_DE_COUNTER 0x85
269−#define PACKET3_WAIT_ON_CE_COUNTER 0x86
270−#define PACKET3_WAIT_ON_DE_COUNTER 0x87
271−#define PACKET3_WAIT_ON_DE_COUNTER_DIFF 0x88
272−#define PACKET3_SET_CE_DE_COUNTERS 0x89
273−#define PACKET3_WAIT_ON_AVAIL_BUFFER 0x8A
274−#define PACKET3_SWITCH_BUFFER 0x8B
275−#define PACKET3_SEM_WAIT_ON_SIGNAL (0x1 << 12)
276−#define PACKET3_SEM_SEL_SIGNAL (0x6 << 29)
277−#define PACKET3_SEM_SEL_WAIT (0x7 << 29)
278−
27947 #endif
drivers/gpu/drm/amd/amdgpu/si_ih.c+9−8
@@ -27,6 +27,7 @@ · 26 unmodified lines
2727 #include "amdgpu_ih.h"
2828 #include "sid.h"
2929 #include "si_ih.h"
30+
3031 #include "oss/oss_1_0_d.h"
3132 #include "oss/oss_1_0_sh_mask.h"
3233
@@ -213,7 +214,7 @@ static int si_ih_resume(struct amdgpu_ip_block *ip_block) · 180 unmodified lines
213214 static bool si_ih_is_idle(struct amdgpu_ip_block *ip_block)
214215 {
215216 struct amdgpu_device *adev = ip_block->adev;
216− u32 tmp = RREG32(SRBM_STATUS);
217+ u32 tmp = RREG32(mmSRBM_STATUS);
217218
218219 if (tmp & SRBM_STATUS__IH_BUSY_MASK)
219220 return false;
@@ -239,23 +240,23 @@ static int si_ih_soft_reset(struct amdgpu_ip_block *ip_block) · 19 unmodified lines
239240 struct amdgpu_device *adev = ip_block->adev;
240241
241242 u32 srbm_soft_reset = 0;
242− u32 tmp = RREG32(SRBM_STATUS);
243+ u32 tmp = RREG32(mmSRBM_STATUS);
243244
244245 if (tmp & SRBM_STATUS__IH_BUSY_MASK)
245246 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_IH_MASK;
246247
247248 if (srbm_soft_reset) {
248− tmp = RREG32(SRBM_SOFT_RESET);
249+ tmp = RREG32(mmSRBM_SOFT_RESET);
249250 tmp |= srbm_soft_reset;
250− dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp);
251− WREG32(SRBM_SOFT_RESET, tmp);
252− tmp = RREG32(SRBM_SOFT_RESET);
251+ dev_info(adev->dev, "mmSRBM_SOFT_RESET=0x%08X\n", tmp);
252+ WREG32(mmSRBM_SOFT_RESET, tmp);
253+ tmp = RREG32(mmSRBM_SOFT_RESET);
253254
254255 udelay(50);
255256
256257 tmp &= ~srbm_soft_reset;
257− WREG32(SRBM_SOFT_RESET, tmp);
258− tmp = RREG32(SRBM_SOFT_RESET);
258+ WREG32(mmSRBM_SOFT_RESET, tmp);
259+ tmp = RREG32(mmSRBM_SOFT_RESET);
259260
260261 udelay(50);
261262 }
drivers/gpu/drm/amd/amdgpu/sid.h+107−1448
@@ -24,43 +24,12 @@ · 23 unmodified lines
2424 #ifndef SI_H
2525 #define SI_H
2626
27−#define TAHITI_RB_BITMAP_WIDTH_PER_SH 2
28−
29−#define SI_MAX_SH_GPRS 256
30−#define SI_MAX_TEMP_GPRS 16
31−#define SI_MAX_SH_THREADS 256
32−#define SI_MAX_SH_STACK_ENTRIES 4096
33−#define SI_MAX_FRC_EOV_CNT 16384
34−#define SI_MAX_BACKENDS 8
35−#define SI_MAX_BACKENDS_MASK 0xFF
36−#define SI_MAX_BACKENDS_PER_SE_MASK 0x0F
37−#define SI_MAX_SIMDS 12
38−#define SI_MAX_SIMDS_MASK 0x0FFF
39−#define SI_MAX_SIMDS_PER_SE_MASK 0x00FF
40−#define SI_MAX_PIPES 8
41−#define SI_MAX_PIPES_MASK 0xFF
42−#define SI_MAX_PIPES_PER_SIMD_MASK 0x3F
43−#define SI_MAX_LDS_NUM 0xFFFF
44−#define SI_MAX_TCC 16
45−#define SI_MAX_TCC_MASK 0xFFFF
4627 #define SI_MAX_CTLACKS_ASSERTION_WAIT 100
4728
48−/* SMC IND accessor regs */
49−#define SMC_IND_INDEX_0 0x80
50−#define SMC_IND_DATA_0 0x81
51−
52−#define SMC_IND_ACCESS_CNTL 0x8A
53−# define AUTO_INCREMENT_IND_0 (1 << 0)
54−#define SMC_MESSAGE_0 0x8B
55−#define SMC_RESP_0 0x8C
56−
5729 /* CG IND registers are accessed via SMC indirect space + SMC_CG_IND_START */
5830 #define SMC_CG_IND_START 0xc0030000
5931 #define SMC_CG_IND_END 0xc0040000
6032
61−#define CG_CGTT_LOCAL_0 0x400
62−#define CG_CGTT_LOCAL_1 0x401
63−
6433 /* SMC IND registers */
6534 #define SMC_SYSCON_RESET_CNTL 0x80000000
6635 # define RST_REG (1 << 0)
@@ -68,9 +37,6 @@ · 1 unmodified line
6837 # define CK_DISABLE (1 << 0)
6938 # define CKEN (1 << 24)
7039
71−#define VGA_HDP_CONTROL 0xCA
72−#define VGA_MEMORY_DISABLE (1 << 4)
73−
7440 #define DCCG_DISP_SLOW_SELECT_REG 0x13F
7541 #define DCCG_DISP1_SLOW_SELECT(x) ((x) << 0)
7642 #define DCCG_DISP1_SLOW_SELECT_MASK (7 << 0)
@@ -79,47 +45,6 @@ · 2 unmodified lines
7945 #define DCCG_DISP2_SLOW_SELECT_MASK (7 << 4)
8046 #define DCCG_DISP2_SLOW_SELECT_SHIFT 4
8147
82−#define CG_SPLL_FUNC_CNTL 0x180
83−#define SPLL_RESET (1 << 0)
84−#define SPLL_SLEEP (1 << 1)
85−#define SPLL_BYPASS_EN (1 << 3)
86−#define SPLL_REF_DIV(x) ((x) << 4)
87−#define SPLL_REF_DIV_MASK (0x3f << 4)
88−#define SPLL_PDIV_A(x) ((x) << 20)
89−#define SPLL_PDIV_A_MASK (0x7f << 20)
90−#define SPLL_PDIV_A_SHIFT 20
91−#define CG_SPLL_FUNC_CNTL_2 0x181
92−#define SCLK_MUX_SEL(x) ((x) << 0)
93−#define SCLK_MUX_SEL_MASK (0x1ff << 0)
94−#define SPLL_CTLREQ_CHG (1 << 23)
95−#define SCLK_MUX_UPDATE (1 << 26)
96−#define CG_SPLL_FUNC_CNTL_3 0x182
97−#define SPLL_FB_DIV(x) ((x) << 0)
98−#define SPLL_FB_DIV_MASK (0x3ffffff << 0)
99−#define SPLL_FB_DIV_SHIFT 0
100−#define SPLL_DITHEN (1 << 28)
101−#define CG_SPLL_FUNC_CNTL_4 0x183
102−
103−#define SPLL_STATUS 0x185
104−#define SPLL_CHG_STATUS (1 << 1)
105−#define SPLL_CNTL_MODE 0x186
106−#define SPLL_SW_DIR_CONTROL (1 << 0)
107−# define SPLL_REFCLK_SEL(x) ((x) << 26)
108−# define SPLL_REFCLK_SEL_MASK (3 << 26)
109−
110−#define CG_SPLL_SPREAD_SPECTRUM 0x188
111−#define SSEN (1 << 0)
112−#define CLK_S(x) ((x) << 4)
113−#define CLK_S_MASK (0xfff << 4)
114−#define CLK_S_SHIFT 4
115−#define CG_SPLL_SPREAD_SPECTRUM_2 0x189
116−#define CLK_V(x) ((x) << 0)
117−#define CLK_V_MASK (0x3ffffff << 0)
118−#define CLK_V_SHIFT 0
119−
120−#define CG_SPLL_AUTOSCALE_CNTL 0x18b
121−# define AUTOSCALE_ON_SS_CLEAR (1 << 9)
122−
12348 /* discrete uvd clocks */
12449 #define CG_UPLL_FUNC_CNTL 0x18d
12550 # define UPLL_RESET_MASK 0x00000001
@@ -149,317 +74,13 @@ · 23 unmodified lines
14974 #define CG_UPLL_SPREAD_SPECTRUM 0x194
15075 # define SSEN_MASK 0x00000001
15176
152−#define MPLL_BYPASSCLK_SEL 0x197
153−# define MPLL_CLKOUT_SEL(x) ((x) << 8)
154−# define MPLL_CLKOUT_SEL_MASK 0xFF00
155−
156−#define CG_CLKPIN_CNTL 0x198
157−# define XTALIN_DIVIDE (1 << 1)
158−# define BCLK_AS_XCLK (1 << 2)
159−#define CG_CLKPIN_CNTL_2 0x199
160−# define FORCE_BIF_REFCLK_EN (1 << 3)
161−# define MUX_TCLK_TO_XCLK (1 << 8)
162−
163−#define THM_CLK_CNTL 0x19b
164−# define CMON_CLK_SEL(x) ((x) << 0)
165−# define CMON_CLK_SEL_MASK 0xFF
166−# define TMON_CLK_SEL(x) ((x) << 8)
167−# define TMON_CLK_SEL_MASK 0xFF00
168−#define MISC_CLK_CNTL 0x19c
169−# define DEEP_SLEEP_CLK_SEL(x) ((x) << 0)
170−# define DEEP_SLEEP_CLK_SEL_MASK 0xFF
171−# define ZCLK_SEL(x) ((x) << 8)
172−# define ZCLK_SEL_MASK 0xFF00
173−
174−#define CG_THERMAL_CTRL 0x1c0
175−#define DPM_EVENT_SRC(x) ((x) << 0)
176−#define DPM_EVENT_SRC_MASK (7 << 0)
177−#define DIG_THERM_DPM(x) ((x) << 14)
178−#define DIG_THERM_DPM_MASK 0x003FC000
179−#define DIG_THERM_DPM_SHIFT 14
180−#define CG_THERMAL_STATUS 0x1c1
181−#define FDO_PWM_DUTY(x) ((x) << 9)
182−#define FDO_PWM_DUTY_MASK (0xff << 9)
183−#define FDO_PWM_DUTY_SHIFT 9
184−#define CG_THERMAL_INT 0x1c2
185−#define DIG_THERM_INTH(x) ((x) << 8)
186−#define DIG_THERM_INTH_MASK 0x0000FF00
187−#define DIG_THERM_INTH_SHIFT 8
188−#define DIG_THERM_INTL(x) ((x) << 16)
189−#define DIG_THERM_INTL_MASK 0x00FF0000
190−#define DIG_THERM_INTL_SHIFT 16
191−#define THERM_INT_MASK_HIGH (1 << 24)
192−#define THERM_INT_MASK_LOW (1 << 25)
193−
194−#define CG_MULT_THERMAL_CTRL 0x1c4
195−#define TEMP_SEL(x) ((x) << 20)
196−#define TEMP_SEL_MASK (0xff << 20)
197−#define TEMP_SEL_SHIFT 20
198−#define CG_MULT_THERMAL_STATUS 0x1c5
199−#define ASIC_MAX_TEMP(x) ((x) << 0)
200−#define ASIC_MAX_TEMP_MASK 0x000001ff
201−#define ASIC_MAX_TEMP_SHIFT 0
202−#define CTF_TEMP(x) ((x) << 9)
203−#define CTF_TEMP_MASK 0x0003fe00
204−#define CTF_TEMP_SHIFT 9
205−
206−#define CG_FDO_CTRL0 0x1d5
207−#define FDO_STATIC_DUTY(x) ((x) << 0)
208−#define FDO_STATIC_DUTY_MASK 0x000000FF
209−#define FDO_STATIC_DUTY_SHIFT 0
210−#define CG_FDO_CTRL1 0x1d6
211−#define FMAX_DUTY100(x) ((x) << 0)
212−#define FMAX_DUTY100_MASK 0x000000FF
213−#define FMAX_DUTY100_SHIFT 0
214−#define CG_FDO_CTRL2 0x1d7
215−#define TMIN(x) ((x) << 0)
216−#define TMIN_MASK 0x000000FF
217−#define TMIN_SHIFT 0
218−#define FDO_PWM_MODE(x) ((x) << 11)
219−#define FDO_PWM_MODE_MASK (7 << 11)
220−#define FDO_PWM_MODE_SHIFT 11
221−#define TACH_PWM_RESP_RATE(x) ((x) << 25)
222−#define TACH_PWM_RESP_RATE_MASK (0x7f << 25)
223−#define TACH_PWM_RESP_RATE_SHIFT 25
224−
225−#define CG_TACH_CTRL 0x1dc
226−# define EDGE_PER_REV(x) ((x) << 0)
227−# define EDGE_PER_REV_MASK (0x7 << 0)
228−# define EDGE_PER_REV_SHIFT 0
229−# define TARGET_PERIOD(x) ((x) << 3)
230−# define TARGET_PERIOD_MASK 0xfffffff8
231−# define TARGET_PERIOD_SHIFT 3
232−#define CG_TACH_STATUS 0x1dd
233−# define TACH_PERIOD(x) ((x) << 0)
234−# define TACH_PERIOD_MASK 0xffffffff
235−# define TACH_PERIOD_SHIFT 0
236−
237−#define GENERAL_PWRMGT 0x1e0
238−# define GLOBAL_PWRMGT_EN (1 << 0)
239−# define STATIC_PM_EN (1 << 1)
240−# define THERMAL_PROTECTION_DIS (1 << 2)
241−# define THERMAL_PROTECTION_TYPE (1 << 3)
242−# define SW_SMIO_INDEX(x) ((x) << 6)
243−# define SW_SMIO_INDEX_MASK (1 << 6)
244−# define SW_SMIO_INDEX_SHIFT 6
245−# define VOLT_PWRMGT_EN (1 << 10)
246−# define DYN_SPREAD_SPECTRUM_EN (1 << 23)
247−#define CG_TPC 0x1e1
248−#define SCLK_PWRMGT_CNTL 0x1e2
249−# define SCLK_PWRMGT_OFF (1 << 0)
250−# define SCLK_LOW_D1 (1 << 1)
251−# define FIR_RESET (1 << 4)
252−# define FIR_FORCE_TREND_SEL (1 << 5)
253−# define FIR_TREND_MODE (1 << 6)
254−# define DYN_GFX_CLK_OFF_EN (1 << 7)
255−# define GFX_CLK_FORCE_ON (1 << 8)
256−# define GFX_CLK_REQUEST_OFF (1 << 9)
257−# define GFX_CLK_FORCE_OFF (1 << 10)
258−# define GFX_CLK_OFF_ACPI_D1 (1 << 11)
259−# define GFX_CLK_OFF_ACPI_D2 (1 << 12)
260−# define GFX_CLK_OFF_ACPI_D3 (1 << 13)
261−# define DYN_LIGHT_SLEEP_EN (1 << 14)
262−
263−#define TARGET_AND_CURRENT_PROFILE_INDEX 0x1e6
264−# define CURRENT_STATE_INDEX_MASK (0xf << 4)
265−# define CURRENT_STATE_INDEX_SHIFT 4
266−
267−#define CG_FTV 0x1ef
268−
269−#define CG_FFCT_0 0x1f0
270−# define UTC_0(x) ((x) << 0)
271−# define UTC_0_MASK (0x3ff << 0)
272−# define DTC_0(x) ((x) << 10)
273−# define DTC_0_MASK (0x3ff << 10)
274−
275−#define CG_BSP 0x1ff
276−# define BSP(x) ((x) << 0)
277−# define BSP_MASK (0xffff << 0)
278−# define BSU(x) ((x) << 16)
279−# define BSU_MASK (0xf << 16)
280−#define CG_AT 0x200
281−# define CG_R(x) ((x) << 0)
282−# define CG_R_MASK (0xffff << 0)
283−# define CG_L(x) ((x) << 16)
284−# define CG_L_MASK (0xffff << 16)
285−
286−#define CG_GIT 0x201
287−# define CG_GICST(x) ((x) << 0)
288−# define CG_GICST_MASK (0xffff << 0)
289−# define CG_GIPOT(x) ((x) << 16)
290−# define CG_GIPOT_MASK (0xffff << 16)
291−
292−#define CG_SSP 0x203
293−# define SST(x) ((x) << 0)
294−# define SST_MASK (0xffff << 0)
295−# define SSTU(x) ((x) << 16)
296−# define SSTU_MASK (0xf << 16)
297−
298−#define CG_DISPLAY_GAP_CNTL 0x20a
299−# define DISP1_GAP(x) ((x) << 0)
300−# define DISP1_GAP_MASK (3 << 0)
301−# define DISP2_GAP(x) ((x) << 2)
302−# define DISP2_GAP_MASK (3 << 2)
303−# define VBI_TIMER_COUNT(x) ((x) << 4)
304−# define VBI_TIMER_COUNT_MASK (0x3fff << 4)
305−# define VBI_TIMER_UNIT(x) ((x) << 20)
306−# define VBI_TIMER_UNIT_MASK (7 << 20)
307−# define DISP1_GAP_MCHG(x) ((x) << 24)
308−# define DISP1_GAP_MCHG_MASK (3 << 24)
309−# define DISP2_GAP_MCHG(x) ((x) << 26)
310−# define DISP2_GAP_MCHG_MASK (3 << 26)
311−
312−#define CG_ULV_CONTROL 0x21e
313−#define CG_ULV_PARAMETER 0x21f
314−
315−#define SMC_SCRATCH0 0x221
316−
317−#define CG_CAC_CTRL 0x22e
318−# define CAC_WINDOW(x) ((x) << 0)
319−# define CAC_WINDOW_MASK 0x00ffffff
320−
321−#define DMIF_ADDR_CONFIG 0x2F5
322−
323−#define DMIF_ADDR_CALC 0x300
324−
325−#define PIPE0_DMIF_BUFFER_CONTROL 0x0328
326−# define DMIF_BUFFERS_ALLOCATED(x) ((x) << 0)
327−# define DMIF_BUFFERS_ALLOCATED_COMPLETED (1 << 4)
328−
329−#define SRBM_STATUS 0x394
330−#define GRBM_RQ_PENDING (1 << 5)
331−#define VMC_BUSY (1 << 8)
332−#define MCB_BUSY (1 << 9)
333−#define MCB_NON_DISPLAY_BUSY (1 << 10)
334−#define MCC_BUSY (1 << 11)
335−#define MCD_BUSY (1 << 12)
336−#define SEM_BUSY (1 << 14)
337−#define IH_BUSY (1 << 17)
338−
339−#define SRBM_SOFT_RESET 0x398
340−#define SOFT_RESET_BIF (1 << 1)
341−#define SOFT_RESET_DC (1 << 5)
342−#define SOFT_RESET_DMA1 (1 << 6)
343−#define SOFT_RESET_GRBM (1 << 8)
344−#define SOFT_RESET_HDP (1 << 9)
345−#define SOFT_RESET_IH (1 << 10)
346−#define SOFT_RESET_MC (1 << 11)
347−#define SOFT_RESET_ROM (1 << 14)
348−#define SOFT_RESET_SEM (1 << 15)
349−#define SOFT_RESET_VMC (1 << 17)
350−#define SOFT_RESET_DMA (1 << 20)
351−#define SOFT_RESET_TST (1 << 21)
352−#define SOFT_RESET_REGBB (1 << 22)
353−#define SOFT_RESET_ORB (1 << 23)
354−
355−#define CC_SYS_RB_BACKEND_DISABLE 0x3A0
356−#define GC_USER_SYS_RB_BACKEND_DISABLE 0x3A1
357−
358−#define SRBM_READ_ERROR 0x3A6
359−#define SRBM_INT_CNTL 0x3A8
360−#define SRBM_INT_ACK 0x3AA
361−
362−#define SRBM_STATUS2 0x3B1
363−#define DMA_BUSY (1 << 5)
364−#define DMA1_BUSY (1 << 6)
365−
366−#define VM_L2_CNTL 0x500
367−#define ENABLE_L2_CACHE (1 << 0)
368−#define ENABLE_L2_FRAGMENT_PROCESSING (1 << 1)
369−#define L2_CACHE_PTE_ENDIAN_SWAP_MODE(x) ((x) << 2)
370−#define L2_CACHE_PDE_ENDIAN_SWAP_MODE(x) ((x) << 4)
371−#define ENABLE_L2_PTE_CACHE_LRU_UPDATE_BY_WRITE (1 << 9)
372−#define ENABLE_L2_PDE0_CACHE_LRU_UPDATE_BY_WRITE (1 << 10)
373−#define EFFECTIVE_L2_QUEUE_SIZE(x) (((x) & 7) << 15)
374−#define CONTEXT1_IDENTITY_ACCESS_MODE(x) (((x) & 3) << 19)
375−#define VM_L2_CNTL2 0x501
376−#define INVALIDATE_ALL_L1_TLBS (1 << 0)
377−#define INVALIDATE_L2_CACHE (1 << 1)
378−#define INVALIDATE_CACHE_MODE(x) ((x) << 26)
379−#define INVALIDATE_PTE_AND_PDE_CACHES 0
380−#define INVALIDATE_ONLY_PTE_CACHES 1
381−#define INVALIDATE_ONLY_PDE_CACHES 2
382−#define VM_L2_CNTL3 0x502
383−#define BANK_SELECT(x) ((x) << 0)
384−#define L2_CACHE_UPDATE_MODE(x) ((x) << 6)
385−#define L2_CACHE_BIGK_FRAGMENT_SIZE(x) ((x) << 15)
386−#define L2_CACHE_BIGK_ASSOCIATIVITY (1 << 20)
387−#define VM_L2_STATUS 0x503
388−#define L2_BUSY (1 << 0)
389−#define VM_CONTEXT0_CNTL 0x504
390−#define ENABLE_CONTEXT (1 << 0)
391−#define PAGE_TABLE_DEPTH(x) (((x) & 3) << 1)
392−#define RANGE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 3)
393−#define RANGE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 4)
394−#define DUMMY_PAGE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 6)
395−#define DUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 7)
396−#define PDE0_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 9)
397−#define PDE0_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 10)
398−#define VALID_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 12)
399−#define VALID_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 13)
400−#define READ_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 15)
401−#define READ_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 16)
402−#define WRITE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 18)
403−#define WRITE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 19)
404−#define PAGE_TABLE_BLOCK_SIZE(x) (((x) & 0xF) << 24)
405−#define VM_CONTEXT1_CNTL 0x505
406−#define VM_CONTEXT0_CNTL2 0x50C
407−#define VM_CONTEXT1_CNTL2 0x50D
408−#define VM_CONTEXT8_PAGE_TABLE_BASE_ADDR 0x50E
409−#define VM_CONTEXT9_PAGE_TABLE_BASE_ADDR 0x50F
410−#define VM_CONTEXT10_PAGE_TABLE_BASE_ADDR 0x510
411−#define VM_CONTEXT11_PAGE_TABLE_BASE_ADDR 0x511
412−#define VM_CONTEXT12_PAGE_TABLE_BASE_ADDR 0x512
413−#define VM_CONTEXT13_PAGE_TABLE_BASE_ADDR 0x513
414−#define VM_CONTEXT14_PAGE_TABLE_BASE_ADDR 0x514
415−#define VM_CONTEXT15_PAGE_TABLE_BASE_ADDR 0x515
416−
417−#define VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x53f
418−#define VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x537
419−#define PROTECTIONS_MASK (0xf << 0)
420−#define PROTECTIONS_SHIFT 0
421− /* bit 0: range
422− * bit 1: pde0
423− * bit 2: valid
424− * bit 3: read
425− * bit 4: write
426− */
427−#define MEMORY_CLIENT_ID_MASK (0xff << 12)
428−#define MEMORY_CLIENT_ID_SHIFT 12
429−#define MEMORY_CLIENT_RW_MASK (1 << 24)
430−#define MEMORY_CLIENT_RW_SHIFT 24
431−#define FAULT_VMID_MASK (0xf << 25)
432−#define FAULT_VMID_SHIFT 25
433−
43477 #define VM_INVALIDATE_REQUEST 0x51E
43578 #define VM_INVALIDATE_RESPONSE 0x51F
43679
437−#define VM_CONTEXT0_PROTECTION_FAULT_DEFAULT_ADDR 0x546
438−#define VM_CONTEXT1_PROTECTION_FAULT_DEFAULT_ADDR 0x547
439−
440−#define VM_CONTEXT0_PAGE_TABLE_BASE_ADDR 0x54F
441−#define VM_CONTEXT1_PAGE_TABLE_BASE_ADDR 0x550
442−#define VM_CONTEXT2_PAGE_TABLE_BASE_ADDR 0x551
443−#define VM_CONTEXT3_PAGE_TABLE_BASE_ADDR 0x552
444−#define VM_CONTEXT4_PAGE_TABLE_BASE_ADDR 0x553
445−#define VM_CONTEXT5_PAGE_TABLE_BASE_ADDR 0x554
446−#define VM_CONTEXT6_PAGE_TABLE_BASE_ADDR 0x555
447−#define VM_CONTEXT7_PAGE_TABLE_BASE_ADDR 0x556
448−#define VM_CONTEXT0_PAGE_TABLE_START_ADDR 0x557
449−#define VM_CONTEXT1_PAGE_TABLE_START_ADDR 0x558
450−
451−#define VM_CONTEXT0_PAGE_TABLE_END_ADDR 0x55F
452−#define VM_CONTEXT1_PAGE_TABLE_END_ADDR 0x560
453−
45480 #define VM_L2_CG 0x570
45581 #define MC_CG_ENABLE (1 << 18)
45682 #define MC_LS_ENABLE (1 << 19)
45783
458−#define MC_SHARED_CHMAP 0x801
459−#define NOOFCHAN_SHIFT 12
460−#define NOOFCHAN_MASK 0x0000f000
461−#define MC_SHARED_CHREMAP 0x802
462−
46384 #define MC_VM_FB_LOCATION 0x809
46485 #define MC_VM_AGP_TOP 0x80A
46586 #define MC_VM_AGP_BOT 0x80B
@@ -491,21 +112,6 @@ · 25 unmodified lines
491112 #define MC_CITF_MISC_WR_CG 0x993
492113 #define MC_CITF_MISC_VM_CG 0x994
493114
494−#define MC_ARB_RAMCFG 0x9D8
495−#define NOOFBANK_SHIFT 0
496−#define NOOFBANK_MASK 0x00000003
497−#define NOOFRANK_SHIFT 2
498−#define NOOFRANK_MASK 0x00000004
499−#define NOOFROWS_SHIFT 3
500−#define NOOFROWS_MASK 0x00000038
501−#define NOOFCOLS_SHIFT 6
502−#define NOOFCOLS_MASK 0x000000C0
503−#define CHANSIZE_SHIFT 8
504−#define CHANSIZE_MASK 0x00000100
505−#define CHANSIZE_OVERRIDE (1 << 11)
506−#define NOOFGROUPS_SHIFT 12
507−#define NOOFGROUPS_MASK 0x00001000
508−
509115 #define MC_ARB_DRAM_TIMING 0x9DD
510116 #define MC_ARB_DRAM_TIMING2 0x9DE
511117
@@ -631,20 +237,6 @@ · 119 unmodified lines
631237 #define CLKS(x) ((x) << 0)
632238 #define CLKS_MASK (0xfff << 0)
633239
634−#define HDP_HOST_PATH_CNTL 0xB00
635−#define CLOCK_GATING_DIS (1 << 23)
636−#define HDP_NONSURFACE_BASE 0xB01
637−#define HDP_NONSURFACE_INFO 0xB02
638−#define HDP_NONSURFACE_SIZE 0xB03
639−
640−#define HDP_DEBUG0 0xBCC
641−
642−#define HDP_ADDR_CONFIG 0xBD2
643−#define HDP_MISC_CNTL 0xBD3
644−#define HDP_FLUSH_INVALIDATE_CACHE (1 << 0)
645−#define HDP_MEM_POWER_LS 0xBD4
646−#define HDP_LS_ENABLE (1 << 0)
647−
648240 #define ATC_MISC_CG 0xCD4
649241
650242 #define IH_RB_CNTL 0xF80
@@ -674,8 +266,6 @@ · 23 unmodified lines
674266 # define MC_WR_CLEAN_CNT(x) ((x) << 20)
675267 # define MC_VMID(x) ((x) << 25)
676268
677−#define CONFIG_MEMSIZE 0x150A
678−
679269 #define INTERRUPT_CNTL 0x151A
680270 # define IH_DUMMY_RD_OVERRIDE (1 << 0)
681271 # define IH_DUMMY_RD_EN (1 << 1)
@@ -683,486 +273,22 @@ · 1 unmodified line
683273 # define GEN_IH_INT_EN (1 << 8)
684274 #define INTERRUPT_CNTL2 0x151B
685275
686−#define HDP_MEM_COHERENCY_FLUSH_CNTL 0x1520
687−
688−#define BIF_FB_EN 0x1524
689−#define FB_READ_EN (1 << 0)
690−#define FB_WRITE_EN (1 << 1)
691−
692−#define HDP_REG_COHERENCY_FLUSH_CNTL 0x1528
693−
694−/* DCE6 ELD audio interface */
695−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0 0x28 /* LPCM */
696−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR1 0x29 /* AC3 */
697−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR2 0x2A /* MPEG1 */
698−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR3 0x2B /* MP3 */
699−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR4 0x2C /* MPEG2 */
700−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR5 0x2D /* AAC */
701−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR6 0x2E /* DTS */
702−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR7 0x2F /* ATRAC */
703−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR8 0x30 /* one bit audio - leave at 0 (default) */
704−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR9 0x31 /* Dolby Digital */
705−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR10 0x32 /* DTS-HD */
706−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR11 0x33 /* MAT-MLP */
707−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR12 0x34 /* DTS */
708−#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13 0x35 /* WMA Pro */
709−# define MAX_CHANNELS(x) (((x) & 0x7) << 0)
710−/* max channels minus one. 7 = 8 channels */
711−# define SUPPORTED_FREQUENCIES(x) (((x) & 0xff) << 8)
712−# define DESCRIPTOR_BYTE_2(x) (((x) & 0xff) << 16)
713−# define SUPPORTED_FREQUENCIES_STEREO(x) (((x) & 0xff) << 24) /* LPCM only */
714−/* SUPPORTED_FREQUENCIES, SUPPORTED_FREQUENCIES_STEREO
715− * bit0 = 32 kHz
716− * bit1 = 44.1 kHz
717− * bit2 = 48 kHz
718− * bit3 = 88.2 kHz
719− * bit4 = 96 kHz
720− * bit5 = 176.4 kHz
721− * bit6 = 192 kHz
722− */
723−
724−#define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_LIPSYNC 0x37
725−# define VIDEO_LIPSYNC(x) (((x) & 0xff) << 0)
726−# define AUDIO_LIPSYNC(x) (((x) & 0xff) << 8)
727−/* VIDEO_LIPSYNC, AUDIO_LIPSYNC
728− * 0 = invalid
729− * x = legal delay value
730− * 255 = sync not supported
731− */
732−#define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_HBR 0x38
733−# define HBR_CAPABLE (1 << 0) /* enabled by default */
734−
735−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO0 0x3a
736−# define MANUFACTURER_ID(x) (((x) & 0xffff) << 0)
737−# define PRODUCT_ID(x) (((x) & 0xffff) << 16)
738−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO1 0x3b
739−# define SINK_DESCRIPTION_LEN(x) (((x) & 0xff) << 0)
740−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO2 0x3c
741−# define PORT_ID0(x) (((x) & 0xffffffff) << 0)
742−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO3 0x3d
743−# define PORT_ID1(x) (((x) & 0xffffffff) << 0)
744−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO4 0x3e
745−# define DESCRIPTION0(x) (((x) & 0xff) << 0)
746−# define DESCRIPTION1(x) (((x) & 0xff) << 8)
747−# define DESCRIPTION2(x) (((x) & 0xff) << 16)
748−# define DESCRIPTION3(x) (((x) & 0xff) << 24)
749−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO5 0x3f
750−# define DESCRIPTION4(x) (((x) & 0xff) << 0)
751−# define DESCRIPTION5(x) (((x) & 0xff) << 8)
752−# define DESCRIPTION6(x) (((x) & 0xff) << 16)
753−# define DESCRIPTION7(x) (((x) & 0xff) << 24)
754−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO6 0x40
755−# define DESCRIPTION8(x) (((x) & 0xff) << 0)
756−# define DESCRIPTION9(x) (((x) & 0xff) << 8)
757−# define DESCRIPTION10(x) (((x) & 0xff) << 16)
758−# define DESCRIPTION11(x) (((x) & 0xff) << 24)
759−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO7 0x41
760−# define DESCRIPTION12(x) (((x) & 0xff) << 0)
761−# define DESCRIPTION13(x) (((x) & 0xff) << 8)
762−# define DESCRIPTION14(x) (((x) & 0xff) << 16)
763−# define DESCRIPTION15(x) (((x) & 0xff) << 24)
764−#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO8 0x42
765−# define DESCRIPTION16(x) (((x) & 0xff) << 0)
766−# define DESCRIPTION17(x) (((x) & 0xff) << 8)
767−
768−#define AZ_F0_CODEC_PIN_CONTROL_HOT_PLUG_CONTROL 0x54
769−# define AUDIO_ENABLED (1 << 31)
770−
771−#define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_CONFIGURATION_DEFAULT 0x56
772−#define PORT_CONNECTIVITY_MASK (3 << 30)
773−#define PORT_CONNECTIVITY_SHIFT 30
774−
775−#define DC_LB_MEMORY_SPLIT 0x1AC3
776−#define DC_LB_MEMORY_CONFIG(x) ((x) << 20)
777−
778−#define PRIORITY_A_CNT 0x1AC6
779−#define PRIORITY_MARK_MASK 0x7fff
780−#define PRIORITY_OFF (1 << 16)
781−#define PRIORITY_ALWAYS_ON (1 << 20)
782−#define PRIORITY_B_CNT 0x1AC7
783−
784−#define DPG_PIPE_ARBITRATION_CONTROL3 0x1B32
785−# define LATENCY_WATERMARK_MASK(x) ((x) << 16)
786−#define DPG_PIPE_LATENCY_CONTROL 0x1B33
787−# define LATENCY_LOW_WATERMARK(x) ((x) << 0)
788−# define LATENCY_HIGH_WATERMARK(x) ((x) << 16)
789−
790−/* 0x6bb8, 0x77b8, 0x103b8, 0x10fb8, 0x11bb8, 0x127b8 */
791−#define VLINE_STATUS 0x1AEE
792−# define VLINE_OCCURRED (1 << 0)
793−# define VLINE_ACK (1 << 4)
794−# define VLINE_STAT (1 << 12)
795−# define VLINE_INTERRUPT (1 << 16)
796−# define VLINE_INTERRUPT_TYPE (1 << 17)
797−/* 0x6bbc, 0x77bc, 0x103bc, 0x10fbc, 0x11bbc, 0x127bc */
798−#define VBLANK_STATUS 0x1AEF
799−# define VBLANK_OCCURRED (1 << 0)
800−# define VBLANK_ACK (1 << 4)
801−# define VBLANK_STAT (1 << 12)
802−# define VBLANK_INTERRUPT (1 << 16)
803−# define VBLANK_INTERRUPT_TYPE (1 << 17)
804−
805−/* 0x6b40, 0x7740, 0x10340, 0x10f40, 0x11b40, 0x12740 */
806−#define INT_MASK 0x1AD0
807−# define VBLANK_INT_MASK (1 << 0)
808−# define VLINE_INT_MASK (1 << 4)
809−
810−#define DISP_INTERRUPT_STATUS 0x183D
811−# define LB_D1_VLINE_INTERRUPT (1 << 2)
812−# define LB_D1_VBLANK_INTERRUPT (1 << 3)
813−# define DC_HPD1_INTERRUPT (1 << 17)
814−# define DC_HPD1_RX_INTERRUPT (1 << 18)
815−# define DACA_AUTODETECT_INTERRUPT (1 << 22)
816−# define DACB_AUTODETECT_INTERRUPT (1 << 23)
817−# define DC_I2C_SW_DONE_INTERRUPT (1 << 24)
818−# define DC_I2C_HW_DONE_INTERRUPT (1 << 25)
819−#define DISP_INTERRUPT_STATUS_CONTINUE 0x183E
820−# define LB_D2_VLINE_INTERRUPT (1 << 2)
821−# define LB_D2_VBLANK_INTERRUPT (1 << 3)
822−# define DC_HPD2_INTERRUPT (1 << 17)
823−# define DC_HPD2_RX_INTERRUPT (1 << 18)
824−# define DISP_TIMER_INTERRUPT (1 << 24)
825−#define DISP_INTERRUPT_STATUS_CONTINUE2 0x183F
826−# define LB_D3_VLINE_INTERRUPT (1 << 2)
827−# define LB_D3_VBLANK_INTERRUPT (1 << 3)
828−# define DC_HPD3_INTERRUPT (1 << 17)
829−# define DC_HPD3_RX_INTERRUPT (1 << 18)
830−#define DISP_INTERRUPT_STATUS_CONTINUE3 0x1840
831−# define LB_D4_VLINE_INTERRUPT (1 << 2)
832−# define LB_D4_VBLANK_INTERRUPT (1 << 3)
833−# define DC_HPD4_INTERRUPT (1 << 17)
834−# define DC_HPD4_RX_INTERRUPT (1 << 18)
835−#define DISP_INTERRUPT_STATUS_CONTINUE4 0x1853
836−# define LB_D5_VLINE_INTERRUPT (1 << 2)
837−# define LB_D5_VBLANK_INTERRUPT (1 << 3)
838−# define DC_HPD5_INTERRUPT (1 << 17)
839−# define DC_HPD5_RX_INTERRUPT (1 << 18)
840−#define DISP_INTERRUPT_STATUS_CONTINUE5 0x1854
841−# define LB_D6_VLINE_INTERRUPT (1 << 2)
842−# define LB_D6_VBLANK_INTERRUPT (1 << 3)
843−# define DC_HPD6_INTERRUPT (1 << 17)
844−# define DC_HPD6_RX_INTERRUPT (1 << 18)
845−
846−/* 0x6858, 0x7458, 0x10058, 0x10c58, 0x11858, 0x12458 */
847−#define GRPH_INT_STATUS 0x1A16
848−# define GRPH_PFLIP_INT_OCCURRED (1 << 0)
849−# define GRPH_PFLIP_INT_CLEAR (1 << 8)
850−/* 0x685c, 0x745c, 0x1005c, 0x10c5c, 0x1185c, 0x1245c */
851−#define GRPH_INT_CONTROL 0x1A17
852−# define GRPH_PFLIP_INT_MASK (1 << 0)
853−# define GRPH_PFLIP_INT_TYPE (1 << 8)
854−
855−#define DAC_AUTODETECT_INT_CONTROL 0x19F2
856−
857−#define DC_HPD1_INT_STATUS 0x1807
858−#define DC_HPD2_INT_STATUS 0x180A
859−#define DC_HPD3_INT_STATUS 0x180D
860−#define DC_HPD4_INT_STATUS 0x1810
861−#define DC_HPD5_INT_STATUS 0x1813
862−#define DC_HPD6_INT_STATUS 0x1816
863−# define DC_HPDx_INT_STATUS (1 << 0)
864−# define DC_HPDx_SENSE (1 << 1)
865−# define DC_HPDx_RX_INT_STATUS (1 << 8)
866−
867−#define DC_HPD1_INT_CONTROL 0x1808
868−#define DC_HPD2_INT_CONTROL 0x180B
869−#define DC_HPD3_INT_CONTROL 0x180E
870−#define DC_HPD4_INT_CONTROL 0x1811
871−#define DC_HPD5_INT_CONTROL 0x1814
872−#define DC_HPD6_INT_CONTROL 0x1817
873−# define DC_HPDx_INT_ACK (1 << 0)
874−# define DC_HPDx_INT_POLARITY (1 << 8)
875−# define DC_HPDx_INT_EN (1 << 16)
876−# define DC_HPDx_RX_INT_ACK (1 << 20)
877−# define DC_HPDx_RX_INT_EN (1 << 24)
878−
879−#define DC_HPD1_CONTROL 0x1809
880−#define DC_HPD2_CONTROL 0x180C
881−#define DC_HPD3_CONTROL 0x180F
882−#define DC_HPD4_CONTROL 0x1812
883−#define DC_HPD5_CONTROL 0x1815
884−#define DC_HPD6_CONTROL 0x1818
885−# define DC_HPDx_CONNECTION_TIMER(x) ((x) << 0)
886−# define DC_HPDx_RX_INT_TIMER(x) ((x) << 16)
887−# define DC_HPDx_EN (1 << 28)
888−
889−#define DPG_PIPE_STUTTER_CONTROL 0x1B35
890−# define STUTTER_ENABLE (1 << 0)
891−
892−/* 0x6e98, 0x7a98, 0x10698, 0x11298, 0x11e98, 0x12a98 */
893−#define CRTC_STATUS_FRAME_COUNT 0x1BA6
894−
895−/* Audio clocks */
896−#define DCCG_AUDIO_DTO0_PHASE 0x05b0
897−#define DCCG_AUDIO_DTO0_MODULE 0x05b4
898−#define DCCG_AUDIO_DTO1_PHASE 0x05c0
899−#define DCCG_AUDIO_DTO1_MODULE 0x05c4
900−
901−#define GRBM_CNTL 0x2000
902−#define GRBM_READ_TIMEOUT(x) ((x) << 0)
903−
904−#define GRBM_STATUS2 0x2002
905−#define RLC_RQ_PENDING (1 << 0)
906−#define RLC_BUSY (1 << 8)
907−#define TC_BUSY (1 << 9)
908−
909−#define GRBM_STATUS 0x2004
910−#define CMDFIFO_AVAIL_MASK 0x0000000F
911−#define RING2_RQ_PENDING (1 << 4)
912−#define SRBM_RQ_PENDING (1 << 5)
913−#define RING1_RQ_PENDING (1 << 6)
914−#define CF_RQ_PENDING (1 << 7)
915−#define PF_RQ_PENDING (1 << 8)
916−#define GDS_DMA_RQ_PENDING (1 << 9)
917−#define GRBM_EE_BUSY (1 << 10)
918−#define DB_CLEAN (1 << 12)
919−#define CB_CLEAN (1 << 13)
920−#define TA_BUSY (1 << 14)
921−#define GDS_BUSY (1 << 15)
922−#define VGT_BUSY (1 << 17)
923−#define IA_BUSY_NO_DMA (1 << 18)
924−#define IA_BUSY (1 << 19)
925−#define SX_BUSY (1 << 20)
926−#define SPI_BUSY (1 << 22)
927−#define BCI_BUSY (1 << 23)
928−#define SC_BUSY (1 << 24)
929−#define PA_BUSY (1 << 25)
930−#define DB_BUSY (1 << 26)
931−#define CP_COHERENCY_BUSY (1 << 28)
932−#define CP_BUSY (1 << 29)
933−#define CB_BUSY (1 << 30)
934−#define GUI_ACTIVE (1 << 31)
935−#define GRBM_STATUS_SE0 0x2005
936−#define GRBM_STATUS_SE1 0x2006
937−#define SE_DB_CLEAN (1 << 1)
938−#define SE_CB_CLEAN (1 << 2)
939−#define SE_BCI_BUSY (1 << 22)
940−#define SE_VGT_BUSY (1 << 23)
941−#define SE_PA_BUSY (1 << 24)
942−#define SE_TA_BUSY (1 << 25)
943−#define SE_SX_BUSY (1 << 26)
944−#define SE_SPI_BUSY (1 << 27)
945−#define SE_SC_BUSY (1 << 29)
946−#define SE_DB_BUSY (1 << 30)
947−#define SE_CB_BUSY (1 << 31)
948−
949−#define GRBM_INT_CNTL 0x2018
950−# define RDERR_INT_ENABLE (1 << 0)
951−# define GUI_IDLE_INT_ENABLE (1 << 19)
952−
953−#define CP_STRMOUT_CNTL 0x213F
954−#define SCRATCH_REG0 0x2140
955−#define SCRATCH_REG1 0x2141
956−#define SCRATCH_REG2 0x2142
957−#define SCRATCH_REG3 0x2143
958−#define SCRATCH_REG4 0x2144
959−#define SCRATCH_REG5 0x2145
960−#define SCRATCH_REG6 0x2146
961−#define SCRATCH_REG7 0x2147
962−
963−#define SCRATCH_UMSK 0x2150
964−#define SCRATCH_ADDR 0x2151
965−
966−#define CP_SEM_WAIT_TIMER 0x216F
967−
968−#define CP_SEM_INCOMPLETE_TIMER_CNTL 0x2172
969−
970−#define CP_ME_CNTL 0x21B6
971−#define CP_CE_HALT (1 << 24)
972−#define CP_PFP_HALT (1 << 26)
973−#define CP_ME_HALT (1 << 28)
974−
975−#define CP_COHER_CNTL2 0x217A
976−
977−#define CP_RB2_RPTR 0x21BE
978−#define CP_RB1_RPTR 0x21BF
979−#define CP_RB0_RPTR 0x21C0
980−#define CP_RB_WPTR_DELAY 0x21C1
981−
982−#define CP_QUEUE_THRESHOLDS 0x21D8
983−#define ROQ_IB1_START(x) ((x) << 0)
984−#define ROQ_IB2_START(x) ((x) << 8)
985−#define CP_MEQ_THRESHOLDS 0x21D9
986−#define MEQ1_START(x) ((x) << 0)
987−#define MEQ2_START(x) ((x) << 8)
988−
989−#define CP_PERFMON_CNTL 0x21FF
990−
991276 #define VGT_VTX_VECT_EJECT_REG 0x222C
992−
993277 #define VGT_ESGS_RING_SIZE 0x2232
994278 #define VGT_GSVS_RING_SIZE 0x2233
995−
996279 #define VGT_GS_VERTEX_REUSE 0x2235
997−
998280 #define VGT_PRIMITIVE_TYPE 0x2256
999281 #define VGT_INDEX_TYPE 0x2257
1000−
1001282 #define VGT_NUM_INDICES 0x225C
1002283 #define VGT_NUM_INSTANCES 0x225D
1003−
1004284 #define VGT_TF_RING_SIZE 0x2262
1005−
1006285 #define VGT_HS_OFFCHIP_PARAM 0x226C
1007−
1008286 #define VGT_TF_MEMORY_BASE 0x226E
1009287
1010−#define PA_CL_ENHANCE 0x2285
1011−#define CLIP_VTX_REORDER_ENA (1 << 0)
1012−#define NUM_CLIP_SEQ(x) ((x) << 1)
1013−
1014−#define PA_SU_LINE_STIPPLE_VALUE 0x2298
1015−
1016−#define PA_SC_LINE_STIPPLE_STATE 0x22C4
1017−
1018−#define PA_SC_FORCE_EOV_MAX_CNTS 0x22C9
1019−#define FORCE_EOV_MAX_CLK_CNT(x) ((x) << 0)
1020−#define FORCE_EOV_MAX_REZ_CNT(x) ((x) << 16)
1021−
1022−#define PA_SC_FIFO_SIZE 0x22F3
1023−#define SC_FRONTEND_PRIM_FIFO_SIZE(x) ((x) << 0)
1024−#define SC_BACKEND_PRIM_FIFO_SIZE(x) ((x) << 6)
1025−#define SC_HIZ_TILE_FIFO_SIZE(x) ((x) << 15)
1026−#define SC_EARLYZ_TILE_FIFO_SIZE(x) ((x) << 23)
1027−
1028288 #define PA_SC_ENHANCE 0x22FC
1029289
1030−#define SQ_CONFIG 0x2300
1031−
1032−#define SQC_CACHES 0x2302
1033−
1034−#define SQ_POWER_THROTTLE 0x2396
1035−#define MIN_POWER(x) ((x) << 0)
1036−#define MIN_POWER_MASK (0x3fff << 0)
1037−#define MIN_POWER_SHIFT 0
1038−#define MAX_POWER(x) ((x) << 16)
1039−#define MAX_POWER_MASK (0x3fff << 16)
1040−#define MAX_POWER_SHIFT 0
1041−#define SQ_POWER_THROTTLE2 0x2397
1042−#define MAX_POWER_DELTA(x) ((x) << 0)
1043−#define MAX_POWER_DELTA_MASK (0x3fff << 0)
1044−#define MAX_POWER_DELTA_SHIFT 0
1045−#define STI_SIZE(x) ((x) << 16)
1046−#define STI_SIZE_MASK (0x3ff << 16)
1047−#define STI_SIZE_SHIFT 16
1048−#define LTI_RATIO(x) ((x) << 27)
1049−#define LTI_RATIO_MASK (0xf << 27)
1050−#define LTI_RATIO_SHIFT 27
1051−
1052−#define SX_DEBUG_1 0x2418
1053−
1054−#define SPI_STATIC_THREAD_MGMT_1 0x2438
1055−#define SPI_STATIC_THREAD_MGMT_2 0x2439
1056−#define SPI_STATIC_THREAD_MGMT_3 0x243A
1057−#define SPI_PS_MAX_WAVE_ID 0x243B
1058−
1059−#define SPI_CONFIG_CNTL 0x2440
1060−
1061−#define SPI_CONFIG_CNTL_1 0x244F
1062−#define VTX_DONE_DELAY(x) ((x) << 0)
1063−#define INTERP_ONE_PRIM_PER_ROW (1 << 4)
1064−
1065−#define CGTS_TCC_DISABLE 0x2452
1066−#define CGTS_USER_TCC_DISABLE 0x2453
1067−#define TCC_DISABLE_MASK 0xFFFF0000
1068−#define TCC_DISABLE_SHIFT 16
1069−#define CGTS_SM_CTRL_REG 0x2454
1070−#define OVERRIDE (1 << 21)
1071−#define LS_OVERRIDE (1 << 22)
1072−
1073−#define SPI_LB_CU_MASK 0x24D5
1074−
1075290 #define TA_CNTL_AUX 0x2542
1076291
1077−#define CC_RB_BACKEND_DISABLE 0x263D
1078−#define BACKEND_DISABLE(x) ((x) << 16)
1079−#define GB_ADDR_CONFIG 0x263E
1080−#define NUM_PIPES(x) ((x) << 0)
1081−#define NUM_PIPES_MASK 0x00000007
1082−#define NUM_PIPES_SHIFT 0
1083−#define PIPE_INTERLEAVE_SIZE(x) ((x) << 4)
1084−#define PIPE_INTERLEAVE_SIZE_MASK 0x00000070
1085−#define PIPE_INTERLEAVE_SIZE_SHIFT 4
1086−#define NUM_SHADER_ENGINES(x) ((x) << 12)
1087−#define NUM_SHADER_ENGINES_MASK 0x00003000
1088−#define NUM_SHADER_ENGINES_SHIFT 12
1089−#define SHADER_ENGINE_TILE_SIZE(x) ((x) << 16)
1090−#define SHADER_ENGINE_TILE_SIZE_MASK 0x00070000
1091−#define SHADER_ENGINE_TILE_SIZE_SHIFT 16
1092−#define NUM_GPUS(x) ((x) << 20)
1093−#define NUM_GPUS_MASK 0x00700000
1094−#define NUM_GPUS_SHIFT 20
1095−#define MULTI_GPU_TILE_SIZE(x) ((x) << 24)
1096−#define MULTI_GPU_TILE_SIZE_MASK 0x03000000
1097−#define MULTI_GPU_TILE_SIZE_SHIFT 24
1098−#define ROW_SIZE(x) ((x) << 28)
1099−#define ROW_SIZE_MASK 0x30000000
1100−#define ROW_SIZE_SHIFT 28
1101−
1102−#define CB_PERFCOUNTER0_SELECT0 0x2688
1103−#define CB_PERFCOUNTER0_SELECT1 0x2689
1104−#define CB_PERFCOUNTER1_SELECT0 0x268A
1105−#define CB_PERFCOUNTER1_SELECT1 0x268B
1106−#define CB_PERFCOUNTER2_SELECT0 0x268C
1107−#define CB_PERFCOUNTER2_SELECT1 0x268D
1108−#define CB_PERFCOUNTER3_SELECT0 0x268E
1109−#define CB_PERFCOUNTER3_SELECT1 0x268F
1110−
1111−#define CB_CGTT_SCLK_CTRL 0x2698
1112−
1113−#define TCP_CHAN_STEER_LO 0x2B03
1114−#define TCP_CHAN_STEER_HI 0x2B94
1115−
1116−#define CP_RB0_BASE 0x3040
1117−#define CP_RB0_CNTL 0x3041
1118−#define RB_BUFSZ(x) ((x) << 0)
1119−#define RB_BLKSZ(x) ((x) << 8)
1120−#define BUF_SWAP_32BIT (2 << 16)
1121−#define RB_NO_UPDATE (1 << 27)
1122−#define RB_RPTR_WR_ENA (1 << 31)
1123−
1124−#define CP_RB0_RPTR_ADDR 0x3043
1125−#define CP_RB0_RPTR_ADDR_HI 0x3044
1126−#define CP_RB0_WPTR 0x3045
1127−
1128−#define CP_PFP_UCODE_ADDR 0x3054
1129−#define CP_PFP_UCODE_DATA 0x3055
1130−#define CP_ME_RAM_RADDR 0x3056
1131−#define CP_ME_RAM_WADDR 0x3057
1132−#define CP_ME_RAM_DATA 0x3058
1133−
1134−#define CP_CE_UCODE_ADDR 0x305A
1135−#define CP_CE_UCODE_DATA 0x305B
1136−
1137−#define CP_RB1_BASE 0x3060
1138−#define CP_RB1_CNTL 0x3061
1139−#define CP_RB1_RPTR_ADDR 0x3062
1140−#define CP_RB1_RPTR_ADDR_HI 0x3063
1141−#define CP_RB1_WPTR 0x3064
1142−#define CP_RB2_BASE 0x3065
1143−#define CP_RB2_CNTL 0x3066
1144−#define CP_RB2_RPTR_ADDR 0x3067
1145−#define CP_RB2_RPTR_ADDR_HI 0x3068
1146−#define CP_RB2_WPTR 0x3069
1147−#define CP_INT_CNTL_RING0 0x306A
1148−#define CP_INT_CNTL_RING1 0x306B
1149−#define CP_INT_CNTL_RING2 0x306C
1150−# define CNTX_BUSY_INT_ENABLE (1 << 19)
1151−# define CNTX_EMPTY_INT_ENABLE (1 << 20)
1152−# define WAIT_MEM_SEM_INT_ENABLE (1 << 21)
1153−# define TIME_STAMP_INT_ENABLE (1 << 26)
1154−# define CP_RINGID2_INT_ENABLE (1 << 29)
1155−# define CP_RINGID1_INT_ENABLE (1 << 30)
1156−# define CP_RINGID0_INT_ENABLE (1 << 31)
1157−#define CP_INT_STATUS_RING0 0x306D
1158−#define CP_INT_STATUS_RING1 0x306E
1159−#define CP_INT_STATUS_RING2 0x306F
1160−# define WAIT_MEM_SEM_INT_STAT (1 << 21)
1161−# define TIME_STAMP_INT_STAT (1 << 26)
1162−# define CP_RINGID2_INT_STAT (1 << 29)
1163−# define CP_RINGID1_INT_STAT (1 << 30)
1164−# define CP_RINGID0_INT_STAT (1 << 31)
1165−
1166292 // #define PA_SC_RASTER_CONFIG 0xA0D4
1167293 # define RB_XSEL2(x) ((x) << 4)
1168294 # define RB_XSEL2_MASK (0x3 << 4)
@@ -1185,171 +311,14 @@ · 16 unmodified lines
1185311 # define SE_YSEL(x) ((x) << 28)
1186312 # define SE_YSEL_MASK (0x3 << 28)
1187313
1188−/* PIF PHY0 registers idx/data 0x8/0xc */
1189−#define PB0_PIF_CNTL 0x10
1190−# define LS2_EXIT_TIME(x) ((x) << 17)
1191−# define LS2_EXIT_TIME_MASK (0x7 << 17)
1192−# define LS2_EXIT_TIME_SHIFT 17
1193−#define PB0_PIF_PAIRING 0x11
1194−# define MULTI_PIF (1 << 25)
1195−#define PB0_PIF_PWRDOWN_0 0x12
1196−# define PLL_POWER_STATE_IN_TXS2_0(x) ((x) << 7)
1197−# define PLL_POWER_STATE_IN_TXS2_0_MASK (0x7 << 7)
1198−# define PLL_POWER_STATE_IN_TXS2_0_SHIFT 7
1199−# define PLL_POWER_STATE_IN_OFF_0(x) ((x) << 10)
1200−# define PLL_POWER_STATE_IN_OFF_0_MASK (0x7 << 10)
1201−# define PLL_POWER_STATE_IN_OFF_0_SHIFT 10
1202−# define PLL_RAMP_UP_TIME_0(x) ((x) << 24)
1203−# define PLL_RAMP_UP_TIME_0_MASK (0x7 << 24)
1204−# define PLL_RAMP_UP_TIME_0_SHIFT 24
1205−#define PB0_PIF_PWRDOWN_1 0x13
1206−# define PLL_POWER_STATE_IN_TXS2_1(x) ((x) << 7)
1207−# define PLL_POWER_STATE_IN_TXS2_1_MASK (0x7 << 7)
1208−# define PLL_POWER_STATE_IN_TXS2_1_SHIFT 7
1209−# define PLL_POWER_STATE_IN_OFF_1(x) ((x) << 10)
1210−# define PLL_POWER_STATE_IN_OFF_1_MASK (0x7 << 10)
1211−# define PLL_POWER_STATE_IN_OFF_1_SHIFT 10
1212−# define PLL_RAMP_UP_TIME_1(x) ((x) << 24)
1213−# define PLL_RAMP_UP_TIME_1_MASK (0x7 << 24)
1214−# define PLL_RAMP_UP_TIME_1_SHIFT 24
1215−
1216−#define PB0_PIF_PWRDOWN_2 0x17
1217−# define PLL_POWER_STATE_IN_TXS2_2(x) ((x) << 7)
1218−# define PLL_POWER_STATE_IN_TXS2_2_MASK (0x7 << 7)
1219−# define PLL_POWER_STATE_IN_TXS2_2_SHIFT 7
1220−# define PLL_POWER_STATE_IN_OFF_2(x) ((x) << 10)
1221−# define PLL_POWER_STATE_IN_OFF_2_MASK (0x7 << 10)
1222−# define PLL_POWER_STATE_IN_OFF_2_SHIFT 10
1223−# define PLL_RAMP_UP_TIME_2(x) ((x) << 24)
1224−# define PLL_RAMP_UP_TIME_2_MASK (0x7 << 24)
1225−# define PLL_RAMP_UP_TIME_2_SHIFT 24
1226−#define PB0_PIF_PWRDOWN_3 0x18
1227−# define PLL_POWER_STATE_IN_TXS2_3(x) ((x) << 7)
1228−# define PLL_POWER_STATE_IN_TXS2_3_MASK (0x7 << 7)
1229−# define PLL_POWER_STATE_IN_TXS2_3_SHIFT 7
1230−# define PLL_POWER_STATE_IN_OFF_3(x) ((x) << 10)
1231−# define PLL_POWER_STATE_IN_OFF_3_MASK (0x7 << 10)
1232−# define PLL_POWER_STATE_IN_OFF_3_SHIFT 10
1233−# define PLL_RAMP_UP_TIME_3(x) ((x) << 24)
1234−# define PLL_RAMP_UP_TIME_3_MASK (0x7 << 24)
1235−# define PLL_RAMP_UP_TIME_3_SHIFT 24
1236−/* PIF PHY1 registers idx/data 0x10/0x14 */
1237−#define PB1_PIF_CNTL 0x10
1238−#define PB1_PIF_PAIRING 0x11
1239−#define PB1_PIF_PWRDOWN_0 0x12
1240−#define PB1_PIF_PWRDOWN_1 0x13
1241−
1242−#define PB1_PIF_PWRDOWN_2 0x17
1243−#define PB1_PIF_PWRDOWN_3 0x18
1244−/* PCIE registers idx/data 0x30/0x34 */
1245−#define PCIE_CNTL2 0x1c /* PCIE */
1246−# define SLV_MEM_LS_EN (1 << 16)
1247−# define SLV_MEM_AGGRESSIVE_LS_EN (1 << 17)
1248−# define MST_MEM_LS_EN (1 << 18)
1249−# define REPLAY_MEM_LS_EN (1 << 19)
1250−#define PCIE_LC_STATUS1 0x28 /* PCIE */
1251−# define LC_REVERSE_RCVR (1 << 0)
1252−# define LC_REVERSE_XMIT (1 << 1)
1253−# define LC_OPERATING_LINK_WIDTH_MASK (0x7 << 2)
1254−# define LC_OPERATING_LINK_WIDTH_SHIFT 2
1255−# define LC_DETECTED_LINK_WIDTH_MASK (0x7 << 5)
1256−# define LC_DETECTED_LINK_WIDTH_SHIFT 5
1257−
1258−#define PCIE_P_CNTL 0x40 /* PCIE */
1259−# define P_IGNORE_EDB_ERR (1 << 6)
1260−
1261314 /* PCIE PORT registers idx/data 0x38/0x3c */
1262−#define PCIE_LC_CNTL 0xa0
1263−# define LC_L0S_INACTIVITY(x) ((x) << 8)
1264−# define LC_L0S_INACTIVITY_MASK (0xf << 8)
1265−# define LC_L0S_INACTIVITY_SHIFT 8
1266−# define LC_L1_INACTIVITY(x) ((x) << 12)
1267−# define LC_L1_INACTIVITY_MASK (0xf << 12)
1268−# define LC_L1_INACTIVITY_SHIFT 12
1269−# define LC_PMI_TO_L1_DIS (1 << 16)
1270−# define LC_ASPM_TO_L1_DIS (1 << 24)
1271−#define PCIE_LC_LINK_WIDTH_CNTL 0xa2 /* PCIE_P */
1272−# define LC_LINK_WIDTH_SHIFT 0
1273−# define LC_LINK_WIDTH_MASK 0x7
315+// #define PCIE_LC_LINK_WIDTH_CNTL 0xa2 /* PCIE_P */
1274316 # define LC_LINK_WIDTH_X0 0
1275317 # define LC_LINK_WIDTH_X1 1
1276318 # define LC_LINK_WIDTH_X2 2
1277319 # define LC_LINK_WIDTH_X4 3
1278320 # define LC_LINK_WIDTH_X8 4
1279321 # define LC_LINK_WIDTH_X16 6
1280−# define LC_LINK_WIDTH_RD_SHIFT 4
1281−# define LC_LINK_WIDTH_RD_MASK 0x70
1282−# define LC_RECONFIG_ARC_MISSING_ESCAPE (1 << 7)
1283−# define LC_RECONFIG_NOW (1 << 8)
1284−# define LC_RENEGOTIATION_SUPPORT (1 << 9)
1285−# define LC_RENEGOTIATE_EN (1 << 10)
1286−# define LC_SHORT_RECONFIG_EN (1 << 11)
1287−# define LC_UPCONFIGURE_SUPPORT (1 << 12)
1288−# define LC_UPCONFIGURE_DIS (1 << 13)
1289−# define LC_DYN_LANES_PWR_STATE(x) ((x) << 21)
1290−# define LC_DYN_LANES_PWR_STATE_MASK (0x3 << 21)
1291−# define LC_DYN_LANES_PWR_STATE_SHIFT 21
1292−#define PCIE_LC_N_FTS_CNTL 0xa3 /* PCIE_P */
1293−# define LC_XMIT_N_FTS(x) ((x) << 0)
1294−# define LC_XMIT_N_FTS_MASK (0xff << 0)
1295−# define LC_XMIT_N_FTS_SHIFT 0
1296−# define LC_XMIT_N_FTS_OVERRIDE_EN (1 << 8)
1297−# define LC_N_FTS_MASK (0xff << 24)
1298−#define PCIE_LC_SPEED_CNTL 0xa4 /* PCIE_P */
1299−# define LC_GEN2_EN_STRAP (1 << 0)
1300−# define LC_GEN3_EN_STRAP (1 << 1)
1301−# define LC_TARGET_LINK_SPEED_OVERRIDE_EN (1 << 2)
1302−# define LC_TARGET_LINK_SPEED_OVERRIDE_MASK (0x3 << 3)
1303−# define LC_TARGET_LINK_SPEED_OVERRIDE_SHIFT 3
1304−# define LC_FORCE_EN_SW_SPEED_CHANGE (1 << 5)
1305−# define LC_FORCE_DIS_SW_SPEED_CHANGE (1 << 6)
1306−# define LC_FORCE_EN_HW_SPEED_CHANGE (1 << 7)
1307−# define LC_FORCE_DIS_HW_SPEED_CHANGE (1 << 8)
1308−# define LC_INITIATE_LINK_SPEED_CHANGE (1 << 9)
1309−# define LC_SPEED_CHANGE_ATTEMPTS_ALLOWED_MASK (0x3 << 10)
1310−# define LC_SPEED_CHANGE_ATTEMPTS_ALLOWED_SHIFT 10
1311−# define LC_CURRENT_DATA_RATE_MASK (0x3 << 13) /* 0/1/2 = gen1/2/3 */
1312−# define LC_CURRENT_DATA_RATE_SHIFT 13
1313−# define LC_CLR_FAILED_SPD_CHANGE_CNT (1 << 16)
1314−# define LC_OTHER_SIDE_EVER_SENT_GEN2 (1 << 18)
1315−# define LC_OTHER_SIDE_SUPPORTS_GEN2 (1 << 19)
1316−# define LC_OTHER_SIDE_EVER_SENT_GEN3 (1 << 20)
1317−# define LC_OTHER_SIDE_SUPPORTS_GEN3 (1 << 21)
1318−
1319−#define PCIE_LC_CNTL2 0xb1
1320−# define LC_ALLOW_PDWN_IN_L1 (1 << 17)
1321−# define LC_ALLOW_PDWN_IN_L23 (1 << 18)
1322−
1323−#define PCIE_LC_CNTL3 0xb5 /* PCIE_P */
1324−# define LC_GO_TO_RECOVERY (1 << 30)
1325−#define PCIE_LC_CNTL4 0xb6 /* PCIE_P */
1326−# define LC_REDO_EQ (1 << 5)
1327−# define LC_SET_QUIESCE (1 << 13)
1328−
1329−/*
1330− * UVD
1331− */
1332−#define UVD_UDEC_ADDR_CONFIG 0x3bd3
1333−#define UVD_UDEC_DB_ADDR_CONFIG 0x3bd4
1334−#define UVD_UDEC_DBW_ADDR_CONFIG 0x3bd5
1335−#define UVD_RBC_RB_RPTR 0x3da4
1336−#define UVD_RBC_RB_WPTR 0x3da5
1337−#define UVD_STATUS 0x3daf
1338−
1339−#define UVD_CGC_CTRL 0x3dc2
1340−# define DCM (1 << 0)
1341−# define CG_DT(x) ((x) << 2)
1342−# define CG_DT_MASK (0xf << 2)
1343−# define CLK_OD(x) ((x) << 6)
1344−# define CLK_OD_MASK (0x1f << 6)
1345−
1346− /* UVD CTX indirect */
1347−#define UVD_CGC_MEM_CTRL 0xC0
1348−#define UVD_CGC_CTRL2 0xC1
1349−# define DYN_OR_EN (1 << 0)
1350−# define DYN_RR_EN (1 << 1)
1351−# define G_DIV_ID(x) ((x) << 2)
1352−# define G_DIV_ID_MASK (0x7 << 2)
1353322
1354323 /*
1355324 * PM4
@@ -1583,45 +552,7 @@ · 227 unmodified lines
1583552 /* ASYNC DMA - first instance at 0xd000, second at 0xd800 */
1584553 #define DMA0_REGISTER_OFFSET 0x0 /* not a register */
1585554 #define DMA1_REGISTER_OFFSET 0x200 /* not a register */
1586−
1587−#define DMA_RB_CNTL 0x3400
1588−# define DMA_RB_ENABLE (1 << 0)
1589−# define DMA_RB_SIZE(x) ((x) << 1) /* log2 */
1590−# define DMA_RB_SWAP_ENABLE (1 << 9) /* 8IN32 */
1591−# define DMA_RPTR_WRITEBACK_ENABLE (1 << 12)
1592−# define DMA_RPTR_WRITEBACK_SWAP_ENABLE (1 << 13) /* 8IN32 */
1593−# define DMA_RPTR_WRITEBACK_TIMER(x) ((x) << 16) /* log2 */
1594−#define DMA_RB_BASE 0x3401
1595−#define DMA_RB_RPTR 0x3402
1596−#define DMA_RB_WPTR 0x3403
1597−
1598−#define DMA_RB_RPTR_ADDR_HI 0x3407
1599−#define DMA_RB_RPTR_ADDR_LO 0x3408
1600−
1601−#define DMA_IB_CNTL 0x3409
1602−# define DMA_IB_ENABLE (1 << 0)
1603−# define DMA_IB_SWAP_ENABLE (1 << 4)
1604−# define CMD_VMID_FORCE (1 << 31)
1605−#define DMA_IB_RPTR 0x340a
1606−#define DMA_CNTL 0x340b
1607−# define TRAP_ENABLE (1 << 0)
1608−# define SEM_INCOMPLETE_INT_ENABLE (1 << 1)
1609−# define SEM_WAIT_INT_ENABLE (1 << 2)
1610−# define DATA_SWAP_ENABLE (1 << 3)
1611−# define FENCE_SWAP_ENABLE (1 << 4)
1612−# define CTXEMPTY_INT_ENABLE (1 << 28)
1613−#define DMA_STATUS_REG 0x340d
1614−# define DMA_IDLE (1 << 0)
1615−#define DMA_TILING_CONFIG 0x342e
1616−
1617−#define DMA_POWER_CNTL 0x342f
1618−# define MEM_POWER_OVERRIDE (1 << 8)
1619−#define DMA_CLK_CTRL 0x3430
1620−
1621−#define DMA_PG 0x3435
1622−# define PG_CNTL_ENABLE (1 << 0)
1623−#define DMA_PGFSM_CONFIG 0x3436
1624−#define DMA_PGFSM_WRITE 0x3437
555+#define SDMA_MAX_INSTANCE 2
1625556
1626557 #define DMA_PACKET(cmd, b, t, s, n) ((((cmd) & 0xF) << 28) | \
1627558 (((b) & 0x1) << 26) | \
@@ -1650,6 +581,7 @@ · 22 unmodified lines
1650581 #define DMA_PACKET_POLL_REG_MEM 0xe
1651582 #define DMA_PACKET_NOP 0xf
1652583
584+/* VCE */
1653585 #define VCE_STATUS 0x20004
1654586 #define VCE_VCPU_CNTL 0x20014
1655587 #define VCE_CLK_EN (1 << 0)
@@ -1726,378 +658,118 @@ · 70 unmodified lines
1726658
1727659 #define CURSOR_WIDTH 64
1728660 #define CURSOR_HEIGHT 64
1729−#define AMDGPU_MM_INDEX 0x0000
1730−#define AMDGPU_MM_DATA 0x0001
1731−
1732−#define VERDE_NUM_CRTC 6
1733−#define BLACKOUT_MODE_MASK 0x00000007
1734−#define VGA_RENDER_CONTROL 0xC0
1735−#define R_000300_VGA_RENDER_CONTROL 0xC0
1736−#define C_000300_VGA_VSTATUS_CNTL 0xFFFCFFFF
1737−#define EVERGREEN_CRTC_STATUS 0x1BA3
1738−#define EVERGREEN_CRTC_V_BLANK (1 << 0)
1739−#define EVERGREEN_CRTC_STATUS_POSITION 0x1BA4
1740−/* CRTC blocks at 0x6df0, 0x79f0, 0x105f0, 0x111f0, 0x11df0, 0x129f0 */
1741−#define EVERGREEN_CRTC_V_BLANK_START_END 0x1b8d
1742−#define EVERGREEN_CRTC_CONTROL 0x1b9c
1743−#define EVERGREEN_CRTC_MASTER_EN (1 << 0)
1744−#define EVERGREEN_CRTC_DISP_READ_REQUEST_DISABLE (1 << 24)
1745−#define EVERGREEN_CRTC_BLANK_CONTROL 0x1b9d
1746−#define EVERGREEN_CRTC_BLANK_DATA_EN (1 << 8)
1747−#define EVERGREEN_CRTC_V_BLANK (1 << 0)
1748−#define EVERGREEN_CRTC_STATUS_HV_COUNT 0x1ba8
1749−#define EVERGREEN_CRTC_UPDATE_LOCK 0x1bb5
1750−#define EVERGREEN_MASTER_UPDATE_LOCK 0x1bbd
1751−#define EVERGREEN_MASTER_UPDATE_MODE 0x1bbe
1752−#define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16)
1753−#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07
1754−#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08
1755−#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS 0x1a04
1756−#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS 0x1a05
1757−#define EVERGREEN_GRPH_UPDATE 0x1a11
1758−#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS 0xc4
1759−#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS_HIGH 0xc9
1760−#define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2)
1761−
1762−#define EVERGREEN_DATA_FORMAT 0x1ac0
1763−# define EVERGREEN_INTERLEAVE_EN (1 << 0)
1764−
1765−#define R600_D1GRPH_ARRAY_MODE_LINEAR_GENERAL (0 << 20)
1766−#define R600_D1GRPH_ARRAY_MODE_LINEAR_ALIGNED (1 << 20)
1767−#define R600_D1GRPH_ARRAY_MODE_1D_TILED_THIN1 (2 << 20)
1768−#define R600_D1GRPH_ARRAY_MODE_2D_TILED_THIN1 (4 << 20)
1769−
1770−#define R700_D1GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a45
1771−#define R700_D2GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1845
1772−
1773−#define R700_D2GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1847
1774−#define R700_D1GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a47
1775−
1776−#define R600_D1GRPH_SWAP_CONTROL 0x1843
1777−#define R600_D1GRPH_SWAP_ENDIAN_NONE (0 << 0)
1778−#define R600_D1GRPH_SWAP_ENDIAN_16BIT (1 << 0)
1779−#define R600_D1GRPH_SWAP_ENDIAN_32BIT (2 << 0)
1780−#define R600_D1GRPH_SWAP_ENDIAN_64BIT (3 << 0)
1781−
1782−#define AVIVO_D1VGA_CONTROL 0x00cc
1783−# define AVIVO_DVGA_CONTROL_MODE_ENABLE (1 << 0)
1784−# define AVIVO_DVGA_CONTROL_TIMING_SELECT (1 << 8)
1785−# define AVIVO_DVGA_CONTROL_SYNC_POLARITY_SELECT (1 << 9)
1786−# define AVIVO_DVGA_CONTROL_OVERSCAN_TIMING_SELECT (1 << 10)
1787−# define AVIVO_DVGA_CONTROL_OVERSCAN_COLOR_EN (1 << 16)
1788−# define AVIVO_DVGA_CONTROL_ROTATE (1 << 24)
1789−#define AVIVO_D2VGA_CONTROL 0x00ce
1790−
1791−#define R600_BUS_CNTL 0x1508
1792−# define R600_BIOS_ROM_DIS (1 << 1)
661+
1793662
1794663 #define R600_ROM_CNTL 0x580
1795664 # define R600_SCK_OVERWRITE (1 << 1)
1796665 # define R600_SCK_PRESCALE_CRYSTAL_CLK_SHIFT 28
1797666 # define R600_SCK_PRESCALE_CRYSTAL_CLK_MASK (0xf << 28)
1798667
1799−#define FMT_BIT_DEPTH_CONTROL 0x1bf2
1800−#define FMT_TRUNCATE_EN (1 << 0)
1801−#define FMT_TRUNCATE_DEPTH (1 << 4)
1802−#define FMT_SPATIAL_DITHER_EN (1 << 8)
1803−#define FMT_SPATIAL_DITHER_MODE(x) ((x) << 9)
1804−#define FMT_SPATIAL_DITHER_DEPTH (1 << 12)
1805−#define FMT_FRAME_RANDOM_ENABLE (1 << 13)
1806−#define FMT_RGB_RANDOM_ENABLE (1 << 14)
1807−#define FMT_HIGHPASS_RANDOM_ENABLE (1 << 15)
1808−#define FMT_TEMPORAL_DITHER_EN (1 << 16)
1809−#define FMT_TEMPORAL_DITHER_DEPTH (1 << 20)
1810−#define FMT_TEMPORAL_DITHER_OFFSET(x) ((x) << 21)
1811−#define FMT_TEMPORAL_LEVEL (1 << 24)
1812−#define FMT_TEMPORAL_DITHER_RESET (1 << 25)
1813−#define FMT_25FRC_SEL(x) ((x) << 26)
1814−#define FMT_50FRC_SEL(x) ((x) << 28)
1815−#define FMT_75FRC_SEL(x) ((x) << 30)
1816−
1817−#define EVERGREEN_DC_LUT_CONTROL 0x1a80
1818−#define EVERGREEN_DC_LUT_BLACK_OFFSET_BLUE 0x1a81
1819−#define EVERGREEN_DC_LUT_BLACK_OFFSET_GREEN 0x1a82
1820−#define EVERGREEN_DC_LUT_BLACK_OFFSET_RED 0x1a83
1821−#define EVERGREEN_DC_LUT_WHITE_OFFSET_BLUE 0x1a84
1822−#define EVERGREEN_DC_LUT_WHITE_OFFSET_GREEN 0x1a85
1823−#define EVERGREEN_DC_LUT_WHITE_OFFSET_RED 0x1a86
1824−#define EVERGREEN_DC_LUT_30_COLOR 0x1a7c
1825−#define EVERGREEN_DC_LUT_RW_INDEX 0x1a79
1826−#define EVERGREEN_DC_LUT_WRITE_EN_MASK 0x1a7e
1827−#define EVERGREEN_DC_LUT_RW_MODE 0x1a78
1828−
1829−#define EVERGREEN_GRPH_ENABLE 0x1a00
1830−#define EVERGREEN_GRPH_CONTROL 0x1a01
1831−#define EVERGREEN_GRPH_DEPTH(x) (((x) & 0x3) << 0)
1832−#define EVERGREEN_GRPH_DEPTH_8BPP 0
1833−#define EVERGREEN_GRPH_DEPTH_16BPP 1
1834−#define EVERGREEN_GRPH_DEPTH_32BPP 2
1835−#define EVERGREEN_GRPH_NUM_BANKS(x) (((x) & 0x3) << 2)
1836−#define EVERGREEN_ADDR_SURF_2_BANK 0
1837−#define EVERGREEN_ADDR_SURF_4_BANK 1
1838−#define EVERGREEN_ADDR_SURF_8_BANK 2
1839−#define EVERGREEN_ADDR_SURF_16_BANK 3
1840−#define EVERGREEN_GRPH_Z(x) (((x) & 0x3) << 4)
1841−#define EVERGREEN_GRPH_BANK_WIDTH(x) (((x) & 0x3) << 6)
1842−#define EVERGREEN_ADDR_SURF_BANK_WIDTH_1 0
1843−#define EVERGREEN_ADDR_SURF_BANK_WIDTH_2 1
1844−#define EVERGREEN_ADDR_SURF_BANK_WIDTH_4 2
1845−#define EVERGREEN_ADDR_SURF_BANK_WIDTH_8 3
1846−#define EVERGREEN_GRPH_FORMAT(x) (((x) & 0x7) << 8)
1847−
1848−#define EVERGREEN_GRPH_FORMAT_INDEXED 0
1849−#define EVERGREEN_GRPH_FORMAT_ARGB1555 0
1850−#define EVERGREEN_GRPH_FORMAT_ARGB565 1
1851−#define EVERGREEN_GRPH_FORMAT_ARGB4444 2
1852−#define EVERGREEN_GRPH_FORMAT_AI88 3
1853−#define EVERGREEN_GRPH_FORMAT_MONO16 4
1854−#define EVERGREEN_GRPH_FORMAT_BGRA5551 5
668+#define GRPH_ARRAY_LINEAR_GENERAL 0
669+#define GRPH_ARRAY_LINEAR_ALIGNED 1
670+#define GRPH_ARRAY_1D_TILED_THIN1 2
671+#define GRPH_ARRAY_2D_TILED_THIN1 4
672+
673+#define ES_AND_GS_AUTO 3
674+#define BUF_SWAP_32BIT (2 << 16)
675+
676+#define GRPH_DEPTH_8BPP 0
677+#define GRPH_DEPTH_16BPP 1
678+#define GRPH_DEPTH_32BPP 2
679+
680+/* 8 BPP */
681+#define GRPH_FORMAT_INDEXED 0
682+
683+/* 16 BPP */
684+#define GRPH_FORMAT_ARGB1555 0
685+#define GRPH_FORMAT_ARGB565 1
686+#define GRPH_FORMAT_ARGB4444 2
687+#define GRPH_FORMAT_AI88 3
688+#define GRPH_FORMAT_MONO16 4
689+#define GRPH_FORMAT_BGRA5551 5
1855690
1856691 /* 32 BPP */
1857−#define EVERGREEN_GRPH_FORMAT_ARGB8888 0
1858−#define EVERGREEN_GRPH_FORMAT_ARGB2101010 1
1859−#define EVERGREEN_GRPH_FORMAT_32BPP_DIG 2
1860−#define EVERGREEN_GRPH_FORMAT_8B_ARGB2101010 3
1861−#define EVERGREEN_GRPH_FORMAT_BGRA1010102 4
1862−#define EVERGREEN_GRPH_FORMAT_8B_BGRA1010102 5
1863−#define EVERGREEN_GRPH_FORMAT_RGB111110 6
1864−#define EVERGREEN_GRPH_FORMAT_BGR101111 7
1865−#define EVERGREEN_GRPH_BANK_HEIGHT(x) (((x) & 0x3) << 11)
1866−#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_1 0
1867−#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_2 1
1868−#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_4 2
1869−#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_8 3
1870−#define EVERGREEN_GRPH_TILE_SPLIT(x) (((x) & 0x7) << 13)
1871−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_64B 0
1872−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_128B 1
1873−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_256B 2
1874−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_512B 3
1875−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_1KB 4
1876−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_2KB 5
1877−#define EVERGREEN_ADDR_SURF_TILE_SPLIT_4KB 6
1878−#define EVERGREEN_GRPH_MACRO_TILE_ASPECT(x) (((x) & 0x3) << 18)
1879−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_1 0
1880−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_2 1
1881−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_4 2
1882−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_8 3
1883−#define EVERGREEN_GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20)
1884−#define EVERGREEN_GRPH_ARRAY_LINEAR_GENERAL 0
1885−#define EVERGREEN_GRPH_ARRAY_LINEAR_ALIGNED 1
1886−#define EVERGREEN_GRPH_ARRAY_1D_TILED_THIN1 2
1887−#define EVERGREEN_GRPH_ARRAY_2D_TILED_THIN1 4
1888−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_1 0
1889−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_2 1
1890−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_4 2
1891−#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_8 3
1892−
1893−#define EVERGREEN_GRPH_SWAP_CONTROL 0x1a03
1894−#define EVERGREEN_GRPH_ENDIAN_SWAP(x) (((x) & 0x3) << 0)
1895−# define EVERGREEN_GRPH_ENDIAN_NONE 0
1896−# define EVERGREEN_GRPH_ENDIAN_8IN16 1
1897−# define EVERGREEN_GRPH_ENDIAN_8IN32 2
1898−# define EVERGREEN_GRPH_ENDIAN_8IN64 3
1899−#define EVERGREEN_GRPH_RED_CROSSBAR(x) (((x) & 0x3) << 4)
1900−# define EVERGREEN_GRPH_RED_SEL_R 0
1901−# define EVERGREEN_GRPH_RED_SEL_G 1
1902−# define EVERGREEN_GRPH_RED_SEL_B 2
1903−# define EVERGREEN_GRPH_RED_SEL_A 3
1904−#define EVERGREEN_GRPH_GREEN_CROSSBAR(x) (((x) & 0x3) << 6)
1905−# define EVERGREEN_GRPH_GREEN_SEL_G 0
1906−# define EVERGREEN_GRPH_GREEN_SEL_B 1
1907−# define EVERGREEN_GRPH_GREEN_SEL_A 2
1908−# define EVERGREEN_GRPH_GREEN_SEL_R 3
1909−#define EVERGREEN_GRPH_BLUE_CROSSBAR(x) (((x) & 0x3) << 8)
1910−# define EVERGREEN_GRPH_BLUE_SEL_B 0
1911−# define EVERGREEN_GRPH_BLUE_SEL_A 1
1912−# define EVERGREEN_GRPH_BLUE_SEL_R 2
1913−# define EVERGREEN_GRPH_BLUE_SEL_G 3
1914−#define EVERGREEN_GRPH_ALPHA_CROSSBAR(x) (((x) & 0x3) << 10)
1915−# define EVERGREEN_GRPH_ALPHA_SEL_A 0
1916−# define EVERGREEN_GRPH_ALPHA_SEL_R 1
1917−# define EVERGREEN_GRPH_ALPHA_SEL_G 2
1918−# define EVERGREEN_GRPH_ALPHA_SEL_B 3
1919−
1920−#define EVERGREEN_D3VGA_CONTROL 0xf8
1921−#define EVERGREEN_D4VGA_CONTROL 0xf9
1922−#define EVERGREEN_D5VGA_CONTROL 0xfa
1923−#define EVERGREEN_D6VGA_CONTROL 0xfb
1924−
1925−#define EVERGREEN_GRPH_SURFACE_ADDRESS_MASK 0xffffff00
1926−
1927−#define EVERGREEN_GRPH_LUT_10BIT_BYPASS_CONTROL 0x1a02
1928−#define EVERGREEN_LUT_10BIT_BYPASS_EN (1 << 8)
1929−
1930−#define EVERGREEN_GRPH_PITCH 0x1a06
1931−#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07
1932−#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08
1933−#define EVERGREEN_GRPH_SURFACE_OFFSET_X 0x1a09
1934−#define EVERGREEN_GRPH_SURFACE_OFFSET_Y 0x1a0a
1935−#define EVERGREEN_GRPH_X_START 0x1a0b
1936−#define EVERGREEN_GRPH_Y_START 0x1a0c
1937−#define EVERGREEN_GRPH_X_END 0x1a0d
1938−#define EVERGREEN_GRPH_Y_END 0x1a0e
1939−#define EVERGREEN_GRPH_UPDATE 0x1a11
1940−#define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2)
1941−#define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16)
1942−#define EVERGREEN_GRPH_FLIP_CONTROL 0x1a12
1943−#define EVERGREEN_GRPH_SURFACE_UPDATE_H_RETRACE_EN (1 << 0)
1944−
1945−#define EVERGREEN_VIEWPORT_START 0x1b5c
1946−#define EVERGREEN_VIEWPORT_SIZE 0x1b5d
1947−#define EVERGREEN_DESKTOP_HEIGHT 0x1ac1
1948−
1949−/* CUR blocks at 0x6998, 0x7598, 0x10198, 0x10d98, 0x11998, 0x12598 */
1950−#define EVERGREEN_CUR_CONTROL 0x1a66
1951−# define EVERGREEN_CURSOR_EN (1 << 0)
1952−# define EVERGREEN_CURSOR_MODE(x) (((x) & 0x3) << 8)
1953−# define EVERGREEN_CURSOR_MONO 0
1954−# define EVERGREEN_CURSOR_24_1 1
1955−# define EVERGREEN_CURSOR_24_8_PRE_MULT 2
1956−# define EVERGREEN_CURSOR_24_8_UNPRE_MULT 3
1957−# define EVERGREEN_CURSOR_2X_MAGNIFY (1 << 16)
1958−# define EVERGREEN_CURSOR_FORCE_MC_ON (1 << 20)
1959−# define EVERGREEN_CURSOR_URGENT_CONTROL(x) (((x) & 0x7) << 24)
1960−# define EVERGREEN_CURSOR_URGENT_ALWAYS 0
1961−# define EVERGREEN_CURSOR_URGENT_1_8 1
1962−# define EVERGREEN_CURSOR_URGENT_1_4 2
1963−# define EVERGREEN_CURSOR_URGENT_3_8 3
1964−# define EVERGREEN_CURSOR_URGENT_1_2 4
1965−#define EVERGREEN_CUR_SURFACE_ADDRESS 0x1a67
1966−# define EVERGREEN_CUR_SURFACE_ADDRESS_MASK 0xfffff000
1967−#define EVERGREEN_CUR_SIZE 0x1a68
1968−#define EVERGREEN_CUR_SURFACE_ADDRESS_HIGH 0x1a69
1969−#define EVERGREEN_CUR_POSITION 0x1a6a
1970−#define EVERGREEN_CUR_HOT_SPOT 0x1a6b
1971−#define EVERGREEN_CUR_COLOR1 0x1a6c
1972−#define EVERGREEN_CUR_COLOR2 0x1a6d
1973−#define EVERGREEN_CUR_UPDATE 0x1a6e
1974−# define EVERGREEN_CURSOR_UPDATE_PENDING (1 << 0)
1975−# define EVERGREEN_CURSOR_UPDATE_TAKEN (1 << 1)
1976−# define EVERGREEN_CURSOR_UPDATE_LOCK (1 << 16)
1977−# define EVERGREEN_CURSOR_DISABLE_MULTIPLE_UPDATE (1 << 24)
1978−
1979−
1980−#define NI_INPUT_CSC_CONTROL 0x1a35
1981−# define NI_INPUT_CSC_GRPH_MODE(x) (((x) & 0x3) << 0)
1982−# define NI_INPUT_CSC_BYPASS 0
1983−# define NI_INPUT_CSC_PROG_COEFF 1
1984−# define NI_INPUT_CSC_PROG_SHARED_MATRIXA 2
1985−# define NI_INPUT_CSC_OVL_MODE(x) (((x) & 0x3) << 4)
1986−
1987−#define NI_OUTPUT_CSC_CONTROL 0x1a3c
1988−# define NI_OUTPUT_CSC_GRPH_MODE(x) (((x) & 0x7) << 0)
1989−# define NI_OUTPUT_CSC_BYPASS 0
1990−# define NI_OUTPUT_CSC_TV_RGB 1
1991−# define NI_OUTPUT_CSC_YCBCR_601 2
1992−# define NI_OUTPUT_CSC_YCBCR_709 3
1993−# define NI_OUTPUT_CSC_PROG_COEFF 4
1994−# define NI_OUTPUT_CSC_PROG_SHARED_MATRIXB 5
1995−# define NI_OUTPUT_CSC_OVL_MODE(x) (((x) & 0x7) << 4)
1996−
1997−#define NI_DEGAMMA_CONTROL 0x1a58
1998−# define NI_GRPH_DEGAMMA_MODE(x) (((x) & 0x3) << 0)
1999−# define NI_DEGAMMA_BYPASS 0
2000−# define NI_DEGAMMA_SRGB_24 1
2001−# define NI_DEGAMMA_XVYCC_222 2
2002−# define NI_OVL_DEGAMMA_MODE(x) (((x) & 0x3) << 4)
2003−# define NI_ICON_DEGAMMA_MODE(x) (((x) & 0x3) << 8)
2004−# define NI_CURSOR_DEGAMMA_MODE(x) (((x) & 0x3) << 12)
2005−
2006−#define NI_GAMUT_REMAP_CONTROL 0x1a59
2007−# define NI_GRPH_GAMUT_REMAP_MODE(x) (((x) & 0x3) << 0)
2008−# define NI_GAMUT_REMAP_BYPASS 0
2009−# define NI_GAMUT_REMAP_PROG_COEFF 1
2010−# define NI_GAMUT_REMAP_PROG_SHARED_MATRIXA 2
2011−# define NI_GAMUT_REMAP_PROG_SHARED_MATRIXB 3
2012−# define NI_OVL_GAMUT_REMAP_MODE(x) (((x) & 0x3) << 4)
2013−
2014−#define NI_REGAMMA_CONTROL 0x1aa0
2015−# define NI_GRPH_REGAMMA_MODE(x) (((x) & 0x7) << 0)
2016−# define NI_REGAMMA_BYPASS 0
2017−# define NI_REGAMMA_SRGB_24 1
2018−# define NI_REGAMMA_XVYCC_222 2
2019−# define NI_REGAMMA_PROG_A 3
2020−# define NI_REGAMMA_PROG_B 4
2021−# define NI_OVL_REGAMMA_MODE(x) (((x) & 0x7) << 4)
2022−
2023−
2024−#define NI_PRESCALE_GRPH_CONTROL 0x1a2d
2025−# define NI_GRPH_PRESCALE_BYPASS (1 << 4)
2026−
2027−#define NI_PRESCALE_OVL_CONTROL 0x1a31
2028−# define NI_OVL_PRESCALE_BYPASS (1 << 4)
2029−
2030−#define NI_INPUT_GAMMA_CONTROL 0x1a10
2031−# define NI_GRPH_INPUT_GAMMA_MODE(x) (((x) & 0x3) << 0)
2032−# define NI_INPUT_GAMMA_USE_LUT 0
2033−# define NI_INPUT_GAMMA_BYPASS 1
2034−# define NI_INPUT_GAMMA_SRGB_24 2
2035−# define NI_INPUT_GAMMA_XVYCC_222 3
2036−# define NI_OVL_INPUT_GAMMA_MODE(x) (((x) & 0x3) << 4)
2037−
2038−#define BLACKOUT_MODE_MASK 0x00000007
2039−#define VGA_RENDER_CONTROL 0xC0
2040−#define R_000300_VGA_RENDER_CONTROL 0xC0
2041−#define C_000300_VGA_VSTATUS_CNTL 0xFFFCFFFF
2042−#define EVERGREEN_CRTC_STATUS 0x1BA3
2043−#define EVERGREEN_CRTC_V_BLANK (1 << 0)
2044−#define EVERGREEN_CRTC_STATUS_POSITION 0x1BA4
2045−/* CRTC blocks at 0x6df0, 0x79f0, 0x105f0, 0x111f0, 0x11df0, 0x129f0 */
2046−#define EVERGREEN_CRTC_V_BLANK_START_END 0x1b8d
2047−#define EVERGREEN_CRTC_CONTROL 0x1b9c
2048−# define EVERGREEN_CRTC_MASTER_EN (1 << 0)
2049−# define EVERGREEN_CRTC_DISP_READ_REQUEST_DISABLE (1 << 24)
2050−#define EVERGREEN_CRTC_BLANK_CONTROL 0x1b9d
2051−# define EVERGREEN_CRTC_BLANK_DATA_EN (1 << 8)
2052−# define EVERGREEN_CRTC_V_BLANK (1 << 0)
2053−#define EVERGREEN_CRTC_STATUS_HV_COUNT 0x1ba8
2054−#define EVERGREEN_CRTC_UPDATE_LOCK 0x1bb5
2055−#define EVERGREEN_MASTER_UPDATE_LOCK 0x1bbd
2056−#define EVERGREEN_MASTER_UPDATE_MODE 0x1bbe
2057−#define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16)
2058−#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07
2059−#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08
2060−#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS 0x1a04
2061−#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS 0x1a05
2062−#define EVERGREEN_GRPH_UPDATE 0x1a11
2063−#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS 0xc4
2064−#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS_HIGH 0xc9
2065−#define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2)
2066−
2067−#define mmVM_CONTEXT1_CNTL__xxRANGE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x10
2068−#define mmVM_CONTEXT1_CNTL__xxRANGE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x4
2069−#define mmVM_CONTEXT1_CNTL__xxDUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x80
2070−#define mmVM_CONTEXT1_CNTL__xxDUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x7
2071−#define mmVM_CONTEXT1_CNTL__xxPDE0_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x400
2072−#define mmVM_CONTEXT1_CNTL__xxPDE0_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0xa
2073−#define mmVM_CONTEXT1_CNTL__xxVALID_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x2000
2074−#define mmVM_CONTEXT1_CNTL__xxVALID_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0xd
2075−#define mmVM_CONTEXT1_CNTL__xxREAD_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x10000
2076−#define mmVM_CONTEXT1_CNTL__xxREAD_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x10
2077−#define mmVM_CONTEXT1_CNTL__xxWRITE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x80000
2078−#define mmVM_CONTEXT1_CNTL__xxWRITE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x13
2079−
2080−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxVMID_MASK 0x1e000000
2081−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxVMID__SHIFT 0x19
2082−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxPROTECTIONS_MASK 0xff
2083−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxPROTECTIONS__SHIFT 0x0
2084−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_ID_MASK 0xff000
2085−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_ID__SHIFT 0xc
2086−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_RW_MASK 0x1000000
2087−#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_RW__SHIFT 0x18
2088−
2089−#define mmMC_SHARED_BLACKOUT_CNTL__xxBLACKOUT_MODE_MASK 0x7
2090−#define mmMC_SHARED_BLACKOUT_CNTL__xxBLACKOUT_MODE__SHIFT 0x0
2091−
2092−#define mmBIF_FB_EN__xxFB_READ_EN_MASK 0x1
2093−#define mmBIF_FB_EN__xxFB_READ_EN__SHIFT 0x0
2094−#define mmBIF_FB_EN__xxFB_WRITE_EN_MASK 0x2
2095−#define mmBIF_FB_EN__xxFB_WRITE_EN__SHIFT 0x1
2096−
2097−#define mmSRBM_SOFT_RESET__xxSOFT_RESET_VMC_MASK 0x20000
2098−#define mmSRBM_SOFT_RESET__xxSOFT_RESET_VMC__SHIFT 0x11
2099−#define mmSRBM_SOFT_RESET__xxSOFT_RESET_MC_MASK 0x800
2100−#define mmSRBM_SOFT_RESET__xxSOFT_RESET_MC__SHIFT 0xb
692+#define GRPH_FORMAT_ARGB8888 0
693+#define GRPH_FORMAT_ARGB2101010 1
694+#define GRPH_FORMAT_32BPP_DIG 2
695+#define GRPH_FORMAT_8B_ARGB2101010 3
696+#define GRPH_FORMAT_BGRA1010102 4
697+#define GRPH_FORMAT_8B_BGRA1010102 5
698+#define GRPH_FORMAT_RGB111110 6
699+#define GRPH_FORMAT_BGR101111 7
700+
701+#define GRPH_ENDIAN_NONE 0
702+#define GRPH_ENDIAN_8IN16 1
703+#define GRPH_ENDIAN_8IN32 2
704+#define GRPH_ENDIAN_8IN64 3
705+#define GRPH_RED_SEL_R 0
706+#define GRPH_RED_SEL_G 1
707+#define GRPH_RED_SEL_B 2
708+#define GRPH_RED_SEL_A 3
709+
710+#define GRPH_GREEN_SEL_G 0
711+#define GRPH_GREEN_SEL_B 1
712+#define GRPH_GREEN_SEL_A 2
713+#define GRPH_GREEN_SEL_R 3
714+
715+#define GRPH_BLUE_SEL_B 0
716+#define GRPH_BLUE_SEL_A 1
717+#define GRPH_BLUE_SEL_R 2
718+#define GRPH_BLUE_SEL_G 3
719+
720+#define GRPH_ALPHA_SEL_A 0
721+#define GRPH_ALPHA_SEL_R 1
722+#define GRPH_ALPHA_SEL_G 2
723+#define GRPH_ALPHA_SEL_B 3
724+
725+/* CUR_CONTROL */
726+ #define CURSOR_MONO 0
727+ #define CURSOR_24_1 1
728+ #define CURSOR_24_8_PRE_MULT 2
729+ #define CURSOR_24_8_UNPRE_MULT 3
730+ #define CURSOR_URGENT_ALWAYS 0
731+ #define CURSOR_URGENT_1_8 1
732+ #define CURSOR_URGENT_1_4 2
733+ #define CURSOR_URGENT_3_8 3
734+ #define CURSOR_URGENT_1_2 4
735+
736+/* INPUT_CSC_CONTROL */
737+# define INPUT_CSC_BYPASS 0
738+# define INPUT_CSC_PROG_COEFF 1
739+# define INPUT_CSC_PROG_SHARED_MATRIXA 2
740+
741+/* OUTPUT_CSC_CONTROL */
742+# define OUTPUT_CSC_BYPASS 0
743+# define OUTPUT_CSC_TV_RGB 1
744+# define OUTPUT_CSC_YCBCR_601 2
745+# define OUTPUT_CSC_YCBCR_709 3
746+# define OUTPUT_CSC_PROG_COEFF 4
747+# define OUTPUT_CSC_PROG_SHARED_MATRIXB 5
748+
749+/* DEGAMMA_CONTROL */
750+# define DEGAMMA_BYPASS 0
751+# define DEGAMMA_SRGB_24 1
752+# define DEGAMMA_XVYCC_222 2
753+
754+/* GAMUT_REMAP_CONTROL */
755+# define GAMUT_REMAP_BYPASS 0
756+# define GAMUT_REMAP_PROG_COEFF 1
757+# define GAMUT_REMAP_PROG_SHARED_MATRIXA 2
758+# define GAMUT_REMAP_PROG_SHARED_MATRIXB 3
759+
760+/* REGAMMA_CONTROL */
761+# define REGAMMA_BYPASS 0
762+# define REGAMMA_SRGB_24 1
763+# define REGAMMA_XVYCC_222 2
764+# define REGAMMA_PROG_A 3
765+# define REGAMMA_PROG_B 4
766+
767+
768+/* INPUT_GAMMA_CONTROL */
769+# define INPUT_GAMMA_USE_LUT 0
770+# define INPUT_GAMMA_BYPASS 1
771+# define INPUT_GAMMA_SRGB_24 2
772+# define INPUT_GAMMA_XVYCC_222 3
2101773
2102774 #define MC_SEQ_MISC0__MT__MASK 0xf0000000
2103775 #define MC_SEQ_MISC0__MT__GDDR1 0x10000000
@@ -2113,20 +785,9 @@ · 9 unmodified lines
2113785 #define PACKET3_SEM_SEL_SIGNAL (0x6 << 29)
2114786 #define PACKET3_SEM_SEL_WAIT (0x7 << 29)
2115787
2116−#define CONFIG_CNTL 0x1509
2117−#define CC_DRM_ID_STRAPS 0X1559
2118788 #define AMDGPU_PCIE_INDEX 0xc
2119789 #define AMDGPU_PCIE_DATA 0xd
2120790
2121−#define DMA_SEM_INCOMPLETE_TIMER_CNTL 0x3411
2122−#define DMA_SEM_WAIT_FAIL_TIMER_CNTL 0x3412
2123−#define DMA_MODE 0x342f
2124−#define DMA_RB_RPTR_ADDR_HI 0x3407
2125−#define DMA_RB_RPTR_ADDR_LO 0x3408
2126−#define DMA_BUSY_MASK 0x20
2127−#define DMA1_BUSY_MASK 0X40
2128−#define SDMA_MAX_INSTANCE 2
2129−
2130791 #define PCIE_BUS_CLK 10000
2131792 #define TCLK (PCIE_BUS_CLK / 10)
2132793 #define PCIE_PORT_INDEX 0xe
@@ -2136,8 +797,6 @@ · 3 unmodified lines
2136797 #define EVERGREEN_PIF_PHY1_INDEX 0x10
2137798 #define EVERGREEN_PIF_PHY1_DATA 0x14
2138799
2139−#define MC_VM_FB_OFFSET 0x81a
2140−
2141800 /* Discrete VCE clocks */
2142801 #define CG_VCEPLL_FUNC_CNTL 0xc0030600
2143802 #define VCEPLL_RESET_MASK 0x00000001
drivers/gpu/drm/amd/amdgpu/soc15.c+5−0
@@ -584,6 +584,8 @@ soc15_asic_reset_method(struct amdgpu_device *adev) · 583 unmodified lines
584584 * Enable triggering of GPU reset only if specified
585585 * by module parameter.
586586 */
587+ if (adev->pcie_reset_ctx.in_link_reset)
588+ return AMD_RESET_METHOD_LINK;
587589 if (amdgpu_gpu_recovery == 4 || amdgpu_gpu_recovery == 5)
588590 return AMD_RESET_METHOD_MODE2;
589591 else if (!(adev->flags & AMD_IS_APU))
@@ -640,6 +642,9 @@ asic_reset: · 50 unmodified lines
640642 case AMD_RESET_METHOD_MODE2:
641643 dev_info(adev->dev, "MODE2 reset\n");
642644 return amdgpu_dpm_mode2_reset(adev);
645+ case AMD_RESET_METHOD_LINK:
646+ dev_info(adev->dev, "Link reset\n");
647+ return amdgpu_device_link_reset(adev);
643648 default:
644649 dev_info(adev->dev, "MODE1 reset\n");
645650 return amdgpu_device_mode1_reset(adev);
drivers/gpu/drm/amd/amdgpu/soc15d.h+5−0
@@ -552,6 +552,11 @@ · 551 unmodified lines
552552 # define PACKET3_QUERY_STATUS_DOORBELL_OFFSET(x) ((x) << 2)
553553 # define PACKET3_QUERY_STATUS_ENG_SEL(x) ((x) << 25)
554554
555+#define PACKET3_RUN_CLEANER_SHADER_9_0 0xD7
556+/* 1. header
557+ * 2. RESERVED [31:0]
558+ */
559+
555560 #define PACKET3_RUN_CLEANER_SHADER 0xD2
556561 /* 1. header
557562 * 2. RESERVED [31:0]
drivers/gpu/drm/amd/amdgpu/umc_v12_0.c+10−0
@@ -220,6 +220,13 @@ static int umc_v12_0_convert_error_address(struct amdgpu_device *adev, · 219 unmodified lines
220220 nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);
221221
222222 /* other nps modes are taken as nps1 */
223+ if (nps == AMDGPU_NPS2_PARTITION_MODE) {
224+ loop_bits[0] = UMC_V12_0_PA_CH5_BIT;
225+ loop_bits[1] = UMC_V12_0_PA_C2_BIT;
226+ loop_bits[2] = UMC_V12_0_PA_B1_BIT;
227+ loop_bits[3] = UMC_V12_0_PA_R12_BIT;
228+ }
229+
223230 if (nps == AMDGPU_NPS4_PARTITION_MODE) {
224231 loop_bits[0] = UMC_V12_0_PA_CH4_BIT;
225232 loop_bits[1] = UMC_V12_0_PA_CH5_BIT;
@@ -517,6 +524,9 @@ static int umc_v12_0_update_ecc_status(struct amdgpu_device *adev, · 291 unmodified lines
517524
518525 if (adev->gmc.gmc_funcs->query_mem_partition_mode)
519526 nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);
527+
528+ if (nps == AMDGPU_NPS2_PARTITION_MODE)
529+ shift_bit = UMC_V12_0_PA_B1_BIT;
520530 if (nps == AMDGPU_NPS4_PARTITION_MODE)
521531 shift_bit = UMC_V12_0_PA_B0_BIT;
522532
drivers/gpu/drm/amd/amdgpu/umc_v12_0.h+2−0
@@ -65,12 +65,14 @@ · 64 unmodified lines
6565 /* row bits in SOC physical address */
6666 #define UMC_V12_0_PA_R0_BIT 22
6767 #define UMC_V12_0_PA_R11_BIT 33
68+#define UMC_V12_0_PA_R12_BIT 34
6869 #define UMC_V12_0_PA_R13_BIT 35
6970 /* channel bit in SOC physical address */
7071 #define UMC_V12_0_PA_CH4_BIT 12
7172 #define UMC_V12_0_PA_CH5_BIT 13
7273 /* bank bit in SOC physical address */
7374 #define UMC_V12_0_PA_B0_BIT 19
75+#define UMC_V12_0_PA_B1_BIT 20
7476 /* row bits in MCA address */
7577 #define UMC_V12_0_MA_R0_BIT 10
7678
drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c+1−0
@@ -39,6 +39,7 @@ · 38 unmodified lines
3939
4040 #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00
4141 #define VCN1_VID_SOC_ADDRESS_3_0 0x48200
42+#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
4243
4344 #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x1fd
4445 #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x503
drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c+1−0
@@ -39,6 +39,7 @@ · 38 unmodified lines
3939
4040 #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00
4141 #define VCN1_VID_SOC_ADDRESS_3_0 0x48200
42+#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
4243
4344 #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27
4445 #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f
drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c+1−0
@@ -40,6 +40,7 @@ · 39 unmodified lines
4040
4141 #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00
4242 #define VCN1_VID_SOC_ADDRESS_3_0 0x48200
43+#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
4344
4445 #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27
4546 #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f
drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c+19−2
@@ -46,6 +46,7 @@ · 45 unmodified lines
4646
4747 #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00
4848 #define VCN1_VID_SOC_ADDRESS_3_0 0x48300
49+#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
4950
5051 #define VCN_HARVEST_MMSCH 0
5152
@@ -238,9 +239,9 @@ static int vcn_v4_0_sw_init(struct amdgpu_ip_block *ip_block) · 186 unmodified lines
238239 adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_pause_dpg_mode;
239240 }
240241
241− /* TODO: Add queue reset mask when FW fully supports it */
242242 adev->vcn.supported_reset =
243243 amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);
244+ adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
244245
245246 if (amdgpu_sriov_vf(adev)) {
246247 r = amdgpu_virt_alloc_mm_table(adev);
@@ -614,7 +615,8 @@ static void vcn_v4_0_mc_resume_dpg_mode(struct amdgpu_vcn_inst *vinst, · 367 unmodified lines
614615
615616 /* VCN global tiling registers */
616617 WREG32_SOC15_DPG_MODE(inst_idx, SOC15_DPG_MODE_OFFSET(
617− VCN, 0, regUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect);
618+ VCN, inst_idx, regUVD_GFX10_ADDR_CONFIG),
619+ adev->gfx.config.gb_addr_config, 0, indirect);
618620 }
619621
620622 /**
@@ -1945,6 +1947,20 @@ static int vcn_v4_0_ring_patch_cs_in_place(struct amdgpu_cs_parser *p, · 1324 unmodified lines
19451947 return 0;
19461948 }
19471949
1950+static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1951+{
1952+ struct amdgpu_device *adev = ring->adev;
1953+ struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1954+
1955+ if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1956+ return -EOPNOTSUPP;
1957+
1958+ vcn_v4_0_stop(vinst);
1959+ vcn_v4_0_start(vinst);
1960+
1961+ return amdgpu_ring_test_helper(ring);
1962+}
1963+
19481964 static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
19491965 .type = AMDGPU_RING_TYPE_VCN_ENC,
19501966 .align_mask = 0x3f,
@@ -1974,6 +1990,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { · 23 unmodified lines
19741990 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
19751991 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
19761992 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1993+ .reset = vcn_v4_0_ring_reset,
19771994 };
19781995
19791996 /**
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c+62−22
@@ -45,6 +45,7 @@ · 44 unmodified lines
4545
4646 #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00
4747 #define VCN1_VID_SOC_ADDRESS_3_0 0x48300
48+#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
4849
4950 static const struct amdgpu_hwip_reg_entry vcn_reg_list_4_0_3[] = {
5051 SOC15_REG_ENTRY_STR(VCN, 0, regUVD_POWER_STATUS),
@@ -287,6 +288,31 @@ static int vcn_v4_0_3_sw_fini(struct amdgpu_ip_block *ip_block) · 236 unmodified lines
287288 return 0;
288289 }
289290
291+static int vcn_v4_0_3_hw_init_inst(struct amdgpu_vcn_inst *vinst)
292+{
293+ int vcn_inst;
294+ struct amdgpu_device *adev = vinst->adev;
295+ struct amdgpu_ring *ring;
296+ int inst_idx = vinst->inst;
297+
298+ vcn_inst = GET_INST(VCN, inst_idx);
299+ ring = &adev->vcn.inst[inst_idx].ring_enc[0];
300+ if (ring->use_doorbell) {
301+ adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell,
302+ (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 9 * vcn_inst,
303+ adev->vcn.inst[inst_idx].aid_id);
304+
305+ WREG32_SOC15(VCN, vcn_inst, regVCN_RB1_DB_CTRL,
306+ ring->doorbell_index << VCN_RB1_DB_CTRL__OFFSET__SHIFT |
307+ VCN_RB1_DB_CTRL__EN_MASK);
308+
309+ /* Read DB_CTRL to flush the write DB_CTRL command. */
310+ RREG32_SOC15(VCN, vcn_inst, regVCN_RB1_DB_CTRL);
311+ }
312+
313+ return 0;
314+}
315+
290316 /**
291317 * vcn_v4_0_3_hw_init - start and test VCN block
292318 *
@@ -298,7 +324,8 @@ static int vcn_v4_0_3_hw_init(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
298324 {
299325 struct amdgpu_device *adev = ip_block->adev;
300326 struct amdgpu_ring *ring;
301− int i, r, vcn_inst;
327+ struct amdgpu_vcn_inst *vinst;
328+ int i, r;
302329
303330 if (amdgpu_sriov_vf(adev)) {
304331 r = vcn_v4_0_3_start_sriov(adev);
@@ -321,28 +348,9 @@ static int vcn_v4_0_3_hw_init(struct amdgpu_ip_block *ip_block) · 16 unmodified lines
321348 for (i = 0; i < adev->vcn.num_vcn_inst; ++i) {
322349 struct amdgpu_vcn4_fw_shared *fw_shared;
323350
324− vcn_inst = GET_INST(VCN, i);
325351 ring = &adev->vcn.inst[i].ring_enc[0];
326−
327− if (ring->use_doorbell) {
328− adev->nbio.funcs->vcn_doorbell_range(
329− adev, ring->use_doorbell,
330− (adev->doorbell_index.vcn.vcn_ring0_1 << 1) +
331− 9 * vcn_inst,
332− adev->vcn.inst[i].aid_id);
333−
334− WREG32_SOC15(
335− VCN, GET_INST(VCN, ring->me),
336− regVCN_RB1_DB_CTRL,
337− ring->doorbell_index
338− << VCN_RB1_DB_CTRL__OFFSET__SHIFT |
339− VCN_RB1_DB_CTRL__EN_MASK);
340−
341− /* Read DB_CTRL to flush the write DB_CTRL command. */
342− RREG32_SOC15(
343− VCN, GET_INST(VCN, ring->me),
344− regVCN_RB1_DB_CTRL);
345− }
352+ vinst = &adev->vcn.inst[i];
353+ vcn_v4_0_3_hw_init_inst(vinst);
346354
347355 /* Re-init fw_shared when RAS fatal error occurred */
348356 fw_shared = adev->vcn.inst[i].fw_shared.cpu_addr;
@@ -1563,6 +1571,37 @@ static void vcn_v4_0_3_unified_ring_set_wptr(struct amdgpu_ring *ring) · 1214 unmodified lines
15631571 }
15641572 }
15651573
1574+static int vcn_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1575+{
1576+ int r = 0;
1577+ int vcn_inst;
1578+ struct amdgpu_device *adev = ring->adev;
1579+ struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1580+
1581+ if (amdgpu_sriov_vf(ring->adev))
1582+ return -EOPNOTSUPP;
1583+
1584+ if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1585+ return -EOPNOTSUPP;
1586+
1587+ vcn_inst = GET_INST(VCN, ring->me);
1588+ r = amdgpu_dpm_reset_vcn(adev, 1 << vcn_inst);
1589+
1590+ if (r) {
1591+ DRM_DEV_ERROR(adev->dev, "VCN reset fail : %d\n", r);
1592+ return r;
1593+ }
1594+
1595+ /* This flag is not set for VF, assumed to be disabled always */
1596+ if (RREG32_SOC15(VCN, GET_INST(VCN, 0), regVCN_RRMT_CNTL) & 0x100)
1597+ adev->vcn.caps |= AMDGPU_VCN_CAPS(RRMT_ENABLED);
1598+ vcn_v4_0_3_hw_init_inst(vinst);
1599+ vcn_v4_0_3_start_dpg_mode(vinst, adev->vcn.inst[ring->me].indirect_sram);
1600+ r = amdgpu_ring_test_helper(ring);
1601+
1602+ return r;
1603+}
1604+
15661605 static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = {
15671606 .type = AMDGPU_RING_TYPE_VCN_ENC,
15681607 .align_mask = 0x3f,
@@ -1591,6 +1630,7 @@ static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = { · 22 unmodified lines
15911630 .emit_wreg = vcn_v4_0_3_enc_ring_emit_wreg,
15921631 .emit_reg_wait = vcn_v4_0_3_enc_ring_emit_reg_wait,
15931632 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1633+ .reset = vcn_v4_0_3_ring_reset,
15941634 };
15951635
15961636 /**
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c+27−0
@@ -46,6 +46,7 @@ · 45 unmodified lines
4646
4747 #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00
4848 #define VCN1_VID_SOC_ADDRESS_3_0 (0x48300 + 0x38000)
49+#define VCN1_AON_SOC_ADDRESS_3_0 (0x48000 + 0x38000)
4950
5051 #define VCN_HARVEST_MMSCH 0
5152
@@ -207,6 +208,10 @@ static int vcn_v4_0_5_sw_init(struct amdgpu_ip_block *ip_block) · 155 unmodified lines
207208 if (amdgpu_sriov_vf(adev))
208209 fw_shared->present_flag_0 |= cpu_to_le32(AMDGPU_VCN_VF_RB_SETUP_FLAG);
209210
211+ fw_shared->present_flag_0 |= AMDGPU_FW_SHARED_FLAG_0_DRM_KEY_INJECT;
212+ fw_shared->drm_key_wa.method =
213+ AMDGPU_DRM_KEY_INJECT_WORKAROUND_VCNFW_ASD_HANDSHAKING;
214+
210215 if (amdgpu_vcnfw_log)
211216 amdgpu_vcn_fwlog_init(&adev->vcn.inst[i]);
212217
@@ -214,6 +219,13 @@ static int vcn_v4_0_5_sw_init(struct amdgpu_ip_block *ip_block) · 1 unmodified line
214219 adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_5_pause_dpg_mode;
215220 }
216221
222+ adev->vcn.supported_reset = amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);
223+ adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
224+
225+ r = amdgpu_vcn_sysfs_reset_mask_init(adev);
226+ if (r)
227+ return r;
228+
217229 if (amdgpu_sriov_vf(adev)) {
218230 r = amdgpu_virt_alloc_mm_table(adev);
219231 if (r)
@@ -1435,6 +1447,20 @@ static void vcn_v4_0_5_unified_ring_set_wptr(struct amdgpu_ring *ring) · 1215 unmodified lines
14351447 }
14361448 }
14371449
1450+static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1451+{
1452+ struct amdgpu_device *adev = ring->adev;
1453+ struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1454+
1455+ if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1456+ return -EOPNOTSUPP;
1457+
1458+ vcn_v4_0_5_stop(vinst);
1459+ vcn_v4_0_5_start(vinst);
1460+
1461+ return amdgpu_ring_test_helper(ring);
1462+}
1463+
14381464 static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = {
14391465 .type = AMDGPU_RING_TYPE_VCN_ENC,
14401466 .align_mask = 0x3f,
@@ -1462,6 +1488,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = { · 21 unmodified lines
14621488 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
14631489 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
14641490 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1491+ .reset = vcn_v4_0_5_ring_reset,
14651492 };
14661493
14671494 /**
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c+18−2
@@ -196,9 +196,9 @@ static int vcn_v5_0_0_sw_init(struct amdgpu_ip_block *ip_block) · 195 unmodified lines
196196 adev->vcn.inst[i].pause_dpg_mode = vcn_v5_0_0_pause_dpg_mode;
197197 }
198198
199− /* TODO: Add queue reset mask when FW fully supports it */
200199 adev->vcn.supported_reset =
201200 amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);
201+ adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
202202
203203 vcn_v5_0_0_alloc_ip_dump(adev);
204204
@@ -533,7 +533,8 @@ static void vcn_v5_0_0_mc_resume_dpg_mode(struct amdgpu_vcn_inst *vinst, · 328 unmodified lines
533533
534534 /* VCN global tiling registers */
535535 WREG32_SOC24_DPG_MODE(inst_idx, SOC24_DPG_MODE_OFFSET(
536− VCN, 0, regUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect);
536+ VCN, inst_idx, regUVD_GFX10_ADDR_CONFIG),
537+ adev->gfx.config.gb_addr_config, 0, indirect);
537538
538539 return;
539540 }
@@ -1171,6 +1172,20 @@ static void vcn_v5_0_0_unified_ring_set_wptr(struct amdgpu_ring *ring) · 631 unmodified lines
11711172 }
11721173 }
11731174
1175+static int vcn_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1176+{
1177+ struct amdgpu_device *adev = ring->adev;
1178+ struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1179+
1180+ if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1181+ return -EOPNOTSUPP;
1182+
1183+ vcn_v5_0_0_stop(vinst);
1184+ vcn_v5_0_0_start(vinst);
1185+
1186+ return amdgpu_ring_test_helper(ring);
1187+}
1188+
11741189 static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = {
11751190 .type = AMDGPU_RING_TYPE_VCN_ENC,
11761191 .align_mask = 0x3f,
@@ -1198,6 +1213,7 @@ static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = { · 21 unmodified lines
11981213 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
11991214 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
12001215 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1216+ .reset = vcn_v5_0_0_ring_reset,
12011217 };
12021218
12031219 /**
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c+2−2
@@ -126,7 +126,7 @@ static int vcn_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 125 unmodified lines
126126
127127 ring = &adev->vcn.inst[i].ring_enc[0];
128128 ring->use_doorbell = true;
129− ring->doorbell_index = (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 9 * vcn_inst;
129+ ring->doorbell_index = (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 11 * vcn_inst;
130130
131131 ring->vm_hub = AMDGPU_MMHUB0(adev->vcn.inst[i].aid_id);
132132 sprintf(ring->name, "vcn_unified_%d", adev->vcn.inst[i].aid_id);
@@ -213,7 +213,7 @@ static int vcn_v5_0_1_hw_init(struct amdgpu_ip_block *ip_block) · 80 unmodified lines
213213 if (ring->use_doorbell)
214214 adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell,
215215 ((adev->doorbell_index.vcn.vcn_ring0_1 << 1) +
216− 9 * vcn_inst),
216+ 11 * vcn_inst),
217217 adev->vcn.inst[i].aid_id);
218218
219219 /* Re-init fw_shared, if required */
drivers/gpu/drm/amd/amdgpu/vega20_ih.c+5−1
@@ -350,6 +350,7 @@ static int vega20_ih_irq_init(struct amdgpu_device *adev) · 349 unmodified lines
350350 if (ret)
351351 return ret;
352352 }
353+ ih[i]->overflow = false;
353354 }
354355
355356 if (!amdgpu_sriov_vf(adev))
@@ -437,7 +438,10 @@ static u32 vega20_ih_get_wptr(struct amdgpu_device *adev, · 81 unmodified lines
437438 if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW))
438439 goto out;
439440
440− wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);
441+ if (!amdgpu_sriov_vf(adev))
442+ wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);
443+ else
444+ ih->overflow = true;
441445
442446 /* When a ring buffer overflow happen start parsing interrupt
443447 * from the last not overwritten vector (wptr + 32). Hopefully
drivers/gpu/drm/amd/amdkfd/kfd_debugfs.c+76−0
@@ -27,6 +27,16 @@ · 26 unmodified lines
2727 #include "kfd_priv.h"
2828
2929 static struct dentry *debugfs_root;
30+static struct dentry *debugfs_proc;
31+static struct list_head procs;
32+
33+struct debugfs_proc_entry {
34+ struct list_head list;
35+ struct dentry *proc_dentry;
36+ pid_t pid;
37+};
38+
39+#define MAX_DEBUGFS_FILENAME_LEN 32
3040
3141 static int kfd_debugfs_open(struct inode *inode, struct file *file)
3242 {
@@ -92,6 +102,8 @@ static const struct file_operations kfd_debugfs_hang_hws_fops = { · 59 unmodified lines
92102 void kfd_debugfs_init(void)
93103 {
94104 debugfs_root = debugfs_create_dir("kfd", NULL);
105+ debugfs_proc = debugfs_create_dir("proc", debugfs_root);
106+ INIT_LIST_HEAD(&procs);
95107
96108 debugfs_create_file("mqds", S_IFREG | 0444, debugfs_root,
97109 kfd_debugfs_mqds_by_process, &kfd_debugfs_fops);
@@ -107,5 +119,69 @@ void kfd_debugfs_init(void) · 9 unmodified lines
107119
108120 void kfd_debugfs_fini(void)
109121 {
122+ debugfs_remove_recursive(debugfs_proc);
110123 debugfs_remove_recursive(debugfs_root);
111124 }
125+
126+static ssize_t kfd_debugfs_pasid_read(struct file *file, char __user *buf,
127+ size_t count, loff_t *ppos)
128+{
129+ struct kfd_process_device *pdd = file_inode(file)->i_private;
130+ char tmp[32];
131+ int len;
132+
133+ len = snprintf(tmp, sizeof(tmp), "%u\n", pdd->pasid);
134+
135+ return simple_read_from_buffer(buf, count, ppos, tmp, len);
136+}
137+
138+static const struct file_operations kfd_debugfs_pasid_fops = {
139+ .owner = THIS_MODULE,
140+ .read = kfd_debugfs_pasid_read,
141+};
142+
143+void kfd_debugfs_add_process(struct kfd_process *p)
144+{
145+ int i;
146+ char name[MAX_DEBUGFS_FILENAME_LEN];
147+ struct debugfs_proc_entry *entry;
148+
149+ entry = kzalloc(sizeof(*entry), GFP_KERNEL);
150+ if (!entry)
151+ return;
152+
153+ list_add(&entry->list, &procs);
154+ entry->pid = p->lead_thread->pid;
155+ snprintf(name, MAX_DEBUGFS_FILENAME_LEN, "%d",
156+ (int)entry->pid);
157+ entry->proc_dentry = debugfs_create_dir(name, debugfs_proc);
158+
159+ /* Create debugfs files for each GPU:
160+ * - proc/<pid>/pasid_<gpuid>
161+ */
162+ for (i = 0; i < p->n_pdds; i++) {
163+ struct kfd_process_device *pdd = p->pdds[i];
164+
165+ snprintf(name, MAX_DEBUGFS_FILENAME_LEN, "pasid_%u",
166+ pdd->dev->id);
167+ debugfs_create_file((const char *)name, S_IFREG | 0444,
168+ entry->proc_dentry, pdd,
169+ &kfd_debugfs_pasid_fops);
170+ }
171+}
172+
173+void kfd_debugfs_remove_process(struct kfd_process *p)
174+{
175+ struct debugfs_proc_entry *entry, *next;
176+
177+ mutex_lock(&kfd_processes_mutex);
178+ list_for_each_entry_safe(entry, next, &procs, list) {
179+ if (entry->pid != p->lead_thread->pid)
180+ continue;
181+
182+ debugfs_remove_recursive(entry->proc_dentry);
183+ list_del(&entry->list);
184+ kfree(entry);
185+ }
186+ mutex_unlock(&kfd_processes_mutex);
187+}
drivers/gpu/drm/amd/amdkfd/kfd_device.c+0−5
@@ -352,11 +352,6 @@ struct kfd_dev *kgd2kfd_probe(struct amdgpu_device *adev, bool vf) · 351 unmodified lines
352352 f2g = &aldebaran_kfd2kgd;
353353 break;
354354 case IP_VERSION(9, 4, 3):
355− gfx_target_version = adev->rev_id >= 1 ? 90402
356− : adev->flags & AMD_IS_APU ? 90400
357− : 90401;
358− f2g = &gc_9_4_3_kfd2kgd;
359− break;
360355 case IP_VERSION(9, 4, 4):
361356 gfx_target_version = 90402;
362357 f2g = &gc_9_4_3_kfd2kgd;
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c+8−6
@@ -1576,8 +1576,9 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 1575 unmodified lines
15761576 int bit;
15771577
15781578 if (q->properties.type == KFD_QUEUE_TYPE_SDMA) {
1579− if (bitmap_empty(dqm->sdma_bitmap, KFD_MAX_SDMA_QUEUES)) {
1580− dev_err(dev, "No more SDMA queue to allocate\n");
1579+ if (bitmap_empty(dqm->sdma_bitmap, get_num_sdma_queues(dqm))) {
1580+ dev_warn(dev, "No more SDMA queue to allocate (%d total queues)\n",
1581+ get_num_sdma_queues(dqm));
15811582 return -ENOMEM;
15821583 }
15831584
@@ -1602,8 +1603,9 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 18 unmodified lines
16021603 q->properties.sdma_queue_id = q->sdma_id /
16031604 kfd_get_num_sdma_engines(dqm->dev);
16041605 } else if (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI) {
1605− if (bitmap_empty(dqm->xgmi_sdma_bitmap, KFD_MAX_SDMA_QUEUES)) {
1606− dev_err(dev, "No more XGMI SDMA queue to allocate\n");
1606+ if (bitmap_empty(dqm->xgmi_sdma_bitmap, get_num_xgmi_sdma_queues(dqm))) {
1607+ dev_warn(dev, "No more XGMI SDMA queue to allocate (%d total queues)\n",
1608+ get_num_xgmi_sdma_queues(dqm));
16071609 return -ENOMEM;
16081610 }
16091611 if (restore_sdma_id) {
@@ -1662,8 +1664,8 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 52 unmodified lines
16621664 }
16631665
16641666 if (!free_bit_found) {
1665− dev_err(dev, "No more SDMA queue to allocate for target ID %i\n",
1666− q->properties.sdma_engine_id);
1667+ dev_warn(dev, "No more SDMA queue to allocate for target ID %i (%d total queues)\n",
1668+ q->properties.sdma_engine_id, num_queues);
16671669 return -ENOMEM;
16681670 }
16691671 }
drivers/gpu/drm/amd/amdkfd/kfd_events.c+19−0
@@ -1177,6 +1177,25 @@ void kfd_signal_hw_exception_event(u32 pasid) · 1176 unmodified lines
11771177 kfd_unref_process(p);
11781178 }
11791179
1180+void kfd_signal_vm_fault_event_with_userptr(struct kfd_process *p, uint64_t gpu_va)
1181+{
1182+ struct kfd_process_device *pdd;
1183+ struct kfd_hsa_memory_exception_data exception_data;
1184+ int i;
1185+
1186+ memset(&exception_data, 0, sizeof(exception_data));
1187+ exception_data.va = gpu_va;
1188+ exception_data.failure.NotPresent = 1;
1189+
1190+ // Send VM seg fault to all kfd process device
1191+ for (i = 0; i < p->n_pdds; i++) {
1192+ pdd = p->pdds[i];
1193+ exception_data.gpu_id = pdd->user_gpu_id;
1194+ kfd_evict_process_device(pdd);
1195+ kfd_signal_vm_fault_event(pdd, NULL, &exception_data);
1196+ }
1197+}
1198+
11801199 void kfd_signal_vm_fault_event(struct kfd_process_device *pdd,
11811200 struct kfd_vm_fault_info *info,
11821201 struct kfd_hsa_memory_exception_data *data)
drivers/gpu/drm/amd/amdkfd/kfd_int_process_v10.c+94−48
@@ -168,11 +168,12 @@ static bool event_interrupt_isr_v10(struct kfd_node *dev, · 167 unmodified lines
168168 client_id != SOC15_IH_CLIENTID_SE3SH)
169169 return false;
170170
171− pr_debug("client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw data:\n",
172− client_id, source_id, vmid, pasid);
173− pr_debug("%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n",
174− data[0], data[1], data[2], data[3],
175− data[4], data[5], data[6], data[7]);
171+ dev_dbg(dev->adev->dev,
172+ "client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw data:\n",
173+ client_id, source_id, vmid, pasid);
174+ dev_dbg(dev->adev->dev, "%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n",
175+ data[0], data[1], data[2], data[3], data[4], data[5], data[6],
176+ data[7]);
176177
177178 /* If there is no valid PASID, it's likely a bug */
178179 if (WARN_ONCE(pasid == 0, "Bug: No PASID in KFD interrupt"))
@@ -217,37 +218,66 @@ static void event_interrupt_wq_v10(struct kfd_node *dev, · 38 unmodified lines
217218 SQ_INTERRUPT_WORD_WAVE_CTXID1, ENCODING);
218219 switch (encoding) {
219220 case SQ_INTERRUPT_WORD_ENCODING_AUTO:
220− pr_debug_ratelimited(
221+ dev_dbg_ratelimited(
222+ dev->adev->dev,
221223 "sq_intr: auto, se %d, ttrace %d, wlt %d, ttrac_buf0_full %d, ttrac_buf1_full %d, ttrace_utc_err %d\n",
222− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_AUTO_CTXID1,
223− SE_ID),
224− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
225− THREAD_TRACE),
226− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
227− WLT),
228− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
229− THREAD_TRACE_BUF0_FULL),
230− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
231− THREAD_TRACE_BUF1_FULL),
232− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
233− THREAD_TRACE_UTC_ERROR));
224+ REG_GET_FIELD(
225+ context_id1,
226+ SQ_INTERRUPT_WORD_AUTO_CTXID1,
227+ SE_ID),
228+ REG_GET_FIELD(
229+ context_id0,
230+ SQ_INTERRUPT_WORD_AUTO_CTXID0,
231+ THREAD_TRACE),
232+ REG_GET_FIELD(
233+ context_id0,
234+ SQ_INTERRUPT_WORD_AUTO_CTXID0,
235+ WLT),
236+ REG_GET_FIELD(
237+ context_id0,
238+ SQ_INTERRUPT_WORD_AUTO_CTXID0,
239+ THREAD_TRACE_BUF0_FULL),
240+ REG_GET_FIELD(
241+ context_id0,
242+ SQ_INTERRUPT_WORD_AUTO_CTXID0,
243+ THREAD_TRACE_BUF1_FULL),
244+ REG_GET_FIELD(
245+ context_id0,
246+ SQ_INTERRUPT_WORD_AUTO_CTXID0,
247+ THREAD_TRACE_UTC_ERROR));
234248 break;
235249 case SQ_INTERRUPT_WORD_ENCODING_INST:
236− pr_debug_ratelimited("sq_intr: inst, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",
237− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
238− SE_ID),
239− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
240− DATA),
241− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
242− SA_ID),
243− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
244− PRIV),
245− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
246− WAVE_ID),
247− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
248− SIMD_ID),
249− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
250− WGP_ID));
250+ dev_dbg_ratelimited(
251+ dev->adev->dev,
252+ "sq_intr: inst, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",
253+ REG_GET_FIELD(
254+ context_id1,
255+ SQ_INTERRUPT_WORD_WAVE_CTXID1,
256+ SE_ID),
257+ REG_GET_FIELD(
258+ context_id0,
259+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
260+ DATA),
261+ REG_GET_FIELD(
262+ context_id0,
263+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
264+ SA_ID),
265+ REG_GET_FIELD(
266+ context_id0,
267+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
268+ PRIV),
269+ REG_GET_FIELD(
270+ context_id0,
271+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
272+ WAVE_ID),
273+ REG_GET_FIELD(
274+ context_id0,
275+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
276+ SIMD_ID),
277+ REG_GET_FIELD(
278+ context_id1,
279+ SQ_INTERRUPT_WORD_WAVE_CTXID1,
280+ WGP_ID));
251281 if (context_id0 & SQ_INTERRUPT_WORD_WAVE_CTXID0__PRIV_MASK) {
252282 if (kfd_set_dbg_ev_from_interrupt(dev, pasid,
253283 KFD_DEBUG_DOORBELL_ID(context_id0),
@@ -259,21 +289,37 @@ static void event_interrupt_wq_v10(struct kfd_node *dev, · 5 unmodified lines
259289 case SQ_INTERRUPT_WORD_ENCODING_ERROR:
260290 sq_intr_err_type = REG_GET_FIELD(context_id0, KFD_CTXID0,
261291 ERR_TYPE);
262− pr_warn_ratelimited("sq_intr: error, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d, err_type %d\n",
263− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
264− SE_ID),
265− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
266− DATA),
267− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
268− SA_ID),
269− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
270− PRIV),
271− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
272− WAVE_ID),
273− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
274− SIMD_ID),
275− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
276− WGP_ID),
292+ dev_warn_ratelimited(
293+ dev->adev->dev,
294+ "sq_intr: error, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d, err_type %d\n",
295+ REG_GET_FIELD(
296+ context_id1,
297+ SQ_INTERRUPT_WORD_WAVE_CTXID1,
298+ SE_ID),
299+ REG_GET_FIELD(
300+ context_id0,
301+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
302+ DATA),
303+ REG_GET_FIELD(
304+ context_id0,
305+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
306+ SA_ID),
307+ REG_GET_FIELD(
308+ context_id0,
309+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
310+ PRIV),
311+ REG_GET_FIELD(
312+ context_id0,
313+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
314+ WAVE_ID),
315+ REG_GET_FIELD(
316+ context_id0,
317+ SQ_INTERRUPT_WORD_WAVE_CTXID0,
318+ SIMD_ID),
319+ REG_GET_FIELD(
320+ context_id1,
321+ SQ_INTERRUPT_WORD_WAVE_CTXID1,
322+ WGP_ID),
277323 sq_intr_err_type);
278324 break;
279325 default:
drivers/gpu/drm/amd/amdkfd/kfd_int_process_v11.c+59−33
@@ -148,44 +148,69 @@ enum SQ_INTERRUPT_ERROR_TYPE { · 147 unmodified lines
148148 #define KFD_CTXID0_DOORBELL_ID(ctxid0) ((ctxid0) & \
149149 KFD_CTXID0_DOORBELL_ID_MASK)
150150
151−static void print_sq_intr_info_auto(uint32_t context_id0, uint32_t context_id1)
151+static void print_sq_intr_info_auto(struct kfd_node *dev, uint32_t context_id0,
152+ uint32_t context_id1)
152153 {
153− pr_debug_ratelimited(
154+ dev_dbg_ratelimited(
155+ dev->adev->dev,
154156 "sq_intr: auto, ttrace %d, wlt %d, ttrace_buf_full %d, reg_tms %d, cmd_tms %d, host_cmd_ovf %d, host_reg_ovf %d, immed_ovf %d, ttrace_utc_err %d\n",
155− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE),
157+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
158+ THREAD_TRACE),
156159 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, WLT),
157− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE_BUF_FULL),
158− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, REG_TIMESTAMP),
159− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, CMD_TIMESTAMP),
160− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, HOST_CMD_OVERFLOW),
161− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, HOST_REG_OVERFLOW),
162− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, IMMED_OVERFLOW),
163− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE_UTC_ERROR));
160+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
161+ THREAD_TRACE_BUF_FULL),
162+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
163+ REG_TIMESTAMP),
164+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
165+ CMD_TIMESTAMP),
166+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
167+ HOST_CMD_OVERFLOW),
168+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
169+ HOST_REG_OVERFLOW),
170+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
171+ IMMED_OVERFLOW),
172+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
173+ THREAD_TRACE_UTC_ERROR));
164174 }
165175
166−static void print_sq_intr_info_inst(uint32_t context_id0, uint32_t context_id1)
176+static void print_sq_intr_info_inst(struct kfd_node *dev, uint32_t context_id0,
177+ uint32_t context_id1)
167178 {
168− pr_debug_ratelimited(
179+ dev_dbg_ratelimited(
180+ dev->adev->dev,
169181 "sq_intr: inst, data 0x%08x, sh %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",
170182 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, DATA),
171− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, SH_ID),
183+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
184+ SH_ID),
172185 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, PRIV),
173− REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, WAVE_ID),
174− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, SIMD_ID),
175− REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, WGP_ID));
186+ REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
187+ WAVE_ID),
188+ REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
189+ SIMD_ID),
190+ REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
191+ WGP_ID));
176192 }
177193

72 more lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c+110−32

This diff is large: its 166 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_packet_manager_v9.c+6−5

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_priv.h+7−0

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_process.c+7−0

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c+9−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_queue.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c+22−2

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h+1−0

This diff is large: its 6 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_svm.c+1−2

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_topology.c+24−17

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c+375−214

This diff is large: its 1707 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h+24−1

This diff is large: its 60 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c+49−7

This diff is large: its 84 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c+13−0

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c+33−12

This diff is large: its 80 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c+2−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.c+0−8

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.c+2−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/Makefile+20−21

This diff is large: its 52 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c+0−5

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/bios/bios_parser.c+2−2

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.c+0−14

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.h+0−1

This diff is large: its 7 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.c+0−14

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.h+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c+14−15

This diff is large: its 57 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn351_clk_mgr.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_clk_mgr.c+7−1

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_smu.c+3−3

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc.c+131−11

This diff is large: its 254 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_debug.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_resource.c+6−42

This diff is large: its 76 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_state.c+107−4

This diff is large: its 141 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_stream.c+63−15

This diff is large: its 113 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc.h+182−165

This diff is large: its 421 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c+109−93

This diff is large: its 333 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_fused_io.c+148−0

This diff is large: its 149 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_fused_io.h+31−0

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_helper.c+7−1

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_state_priv.h+20−0

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_stream.h+11−0

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_types.h+15−2

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dccg/dcn35/dcn35_dccg.c+37−25

This diff is large: its 208 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c+5−4

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c+2−1

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dmub_replay.c+13−6

This diff is large: its 61 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce60/Makefile+1−2

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce80/dce80_timing_generator.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dm_helpers.h+8−0

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn30/display_mode_vba_30.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn31/display_mode_vba_31.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn314/display_mode_vba_314.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/Makefile+0−3

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_translation_helper.c+9−3

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_wrapper.c+4−4

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top.h+1−0

This diff is large: its 6 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_dchub_registers.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_display_cfg_types.h+2−1

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4.c+5−5

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c+2138−2208

This diff is large: its 6570 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_shared_types.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_utils.c+71−71

This diff is large: its 285 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_dpmm/dml2_dpmm_dcn4.c+4−0

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_mcg/dml2_mcg_dcn4.c+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_pmo/dml2_pmo_dcn4_fams2.c+4−4

This diff is large: its 36 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_top/dml2_top_soc15.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.c+0−31

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.h+42−35

This diff is large: its 98 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_internal_shared_types.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml2_dc_resource_mgmt.c+12−12

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c+6−10

This diff is large: its 51 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml2_wrapper.c+4−1

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dpp/dcn30/dcn30_dpp.c+3−6

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dpp/dcn35/dcn35_dpp.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.c+0−7

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dsc/dcn401/dcn401_dsc.c+0−6

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c+14−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hubbub/dcn21/dcn21_hubbub.c+3−3

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/Makefile+18−8

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce110/dce110_hwseq.c+4−4

This diff is large: its 33 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.crenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_hw_sequencer.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.hrenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_hw_sequencer.h+0−0

File renamed without changes.

drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.c+122−1

This diff is large: its 137 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.h+7−0

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn20/dcn20_hwseq.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn314/dcn314_hwseq.c+14−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn32/dcn32_hwseq.c+3−1

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c+14−7

This diff is large: its 68 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn351/dcn351_init.c+3−1

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c+317−37

This diff is large: its 444 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h+8−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h+6−2

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer_private.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/core_status.h+3−0

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/core_types.h+10−4

This diff is large: its 49 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h+2−1

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h+29−17

This diff is large: its 60 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/optc.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/reg_helper.h+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/resource.h+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/irq/dcn32/irq_service_dcn32.c+46−15

This diff is large: its 104 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/irq/dcn401/irq_service_dcn401.c+46−14

This diff is large: its 103 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/irq_types.h+16−0

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/link_factory.c+5−1

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_dpia.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c+4−3

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c+7−4

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn32/dcn32_mpc.c+192−188

This diff is large: its 468 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.c+0−11

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.h+0−14

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn35/dcn35_optc.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/Makefile+18−8

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce100/dce100_resource.c+3−3

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce110/dce110_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.crenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c+3−3

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.hrenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.h+0−0

File renamed without changes.

drivers/gpu/drm/amd/display/dc/resource/dce80/dce80_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn10/dcn10_resource.c+7−2

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.c+3−3

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn21/dcn21_resource.c+3−3

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.c+2−4

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c+59−13

This diff is large: its 132 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.h+5−1

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn321/dcn321_resource.c+3−1

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn35/dcn35_resource.c+4−4

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn351/dcn351_resource.c+3−3

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn36/dcn36_resource.c+4−4

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.c+45−7

This diff is large: its 79 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/sspl/dc_spl.c+14−11

This diff is large: its 80 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/sspl/spl_fixpt31_32.c+0−4

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/dmub_srv.h+110−47

This diff is large: its 283 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h+73−2

This diff is large: its 138 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.c+83−38

This diff is large: its 155 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.h+3−1

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_srv.c+247−98

This diff is large: its 484 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_srv_stat.c+4−17

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/display/include/gpio_service_interface.h+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c+43−10

This diff is large: its 72 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_transition.c+36−12

This diff is large: its 81 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp_ddc.c+73−0

This diff is large: its 77 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/inc/mod_hdcp.h+36−2

This diff is large: its 68 lines aren’t shown here.

drivers/gpu/drm/amd/include/amd_shared.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/dce/dce_6_0_sh_mask.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/gca/gfx_6_0_d.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_d.h+20−3

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_sh_mask.h+41−0

This diff is large: its 55 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_d.h+44−0

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_sh_mask.h+185−3

This diff is large: its 235 lines aren’t shown here.

drivers/gpu/drm/amd/include/atombios.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/kgd_pp_interface.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/include/v11_structs.h+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/include/v12_structs.h+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/amdgpu_dpm.c+43−0

This diff is large: its 57 lines aren’t shown here.

drivers/gpu/drm/amd/pm/amdgpu_pm.c+34−0

This diff is large: its 69 lines aren’t shown here.

drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h+3−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c+190−168

This diff is large: its 806 lines aren’t shown here.

drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c+24−18

This diff is large: its 127 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c+8−0

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.c+0−36

This diff is large: its 43 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.h+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c+34−0

This diff is large: its 52 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h+26−3

This diff is large: its 77 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h+7−0

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h+3−1

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v11_0.h+0−5

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v13_0.h+2−12

This diff is large: its 42 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/smu_v11_0.c+0−39

This diff is large: its 46 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c+11−2

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c+10−101

This diff is large: its 136 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_12_ppt.c+14−24

This diff is large: its 73 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c+126−29

This diff is large: its 319 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.h+21−0

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/drm_file.c+34−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/radeon/atombios.h+1−2

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/radeon/r600_hdmi.c+0−22

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon.h+0−3

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon_asic.h+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon_cs.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon_fence.c+0−42

This diff is large: its 49 lines aren’t shown here.

drivers/gpu/drm/radeon/sid.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/drm/drm_file.h+3−0

This diff is large: its 10 lines aren’t shown here.

include/linux/dma-fence-unwrap.h+2−0

This diff is large: its 9 lines aren’t shown here.

include/uapi/drm/amdgpu_drm.h+316−0

This diff is large: its 372 lines aren’t shown here.

include/uapi/linux/kfd_ioctl.h+5−0

This diff is large: its 19 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t