History

Merge tag 'amd-drm-next-6.16-2025-05-09' of https://gitlab.freedesktop.org/agd5f/linux into drm-next

amd-drm-next-6.16-2025-05-09: amdgpu: - IPS fixes - DSC cleanup - DC Scaling updates - DC FP fixes - Fused I2C-over-AUX updates - SubVP fixes - Freesync fix - DMUB AUX fixes - VCN fix - Hibernation fixes - HDP fixes - DCN 2.1 fixes - DPIA fixes - DMUB updates - Use drm_file_err in amdgpu - Enforce isolation updates - Use new dma_fence helpers - USERQ fixes - Documentation updates - Misc code cleanups - SR-IOV updates - RAS updates - PSP 12 cleanups amdkfd: - Update error messages for SDMA - Userptr updates drm: - Add drm_file_err function dma-buf: - Add a helper to sort and deduplicate dma_fence arrays

Dave Airlie committed 1 year ago

Commit 1faeeb32 parents f9fa012 afc6053338 files · +15952 −9761Browse files

Unified · Split

338 files
  1. Documentation/gpu/amdgpu/amd-hardware-list-info.rst +23 −0
  2. Documentation/gpu/amdgpu/amdgpu-glossary.rst +75 −0
  3. Documentation/gpu/amdgpu/apu-asic-info-table.csv +2 −0
  4. Documentation/gpu/amdgpu/debugfs.rst +210 −0
  5. Documentation/gpu/amdgpu/debugging.rst +7 −0
  6. Documentation/gpu/amdgpu/display/dc-debug.rst +1 −1
  7. Documentation/gpu/amdgpu/driver-core.rst +56 −25
  8. Documentation/gpu/amdgpu/driver-misc.rst +0 −17
  9. Documentation/gpu/amdgpu/gc/index.rst +52 −0
  10. Documentation/gpu/amdgpu/gc/mes.rst +38 −0
  11. Documentation/gpu/amdgpu/index.rst +4 −1
  12. Documentation/gpu/amdgpu/pipe_and_queue_abstraction.svg +1279 −0
  13. drivers/dma-buf/dma-fence-unwrap.c +37 −14
  14. drivers/gpu/drm/amd/amdgpu/Makefile +7 −2
  15. drivers/gpu/drm/amd/amdgpu/amdgpu.h +98 −6
  16. drivers/gpu/drm/amd/amdgpu/amdgpu_aca.c +3 −0
  17. drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c +0 −18
  18. drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +12 −0
  19. drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c +24 −85
  20. drivers/gpu/drm/amd/amdgpu/amdgpu_cper.c +2 −2
  21. drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +23 −1
  22. drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +351 −178
  23. drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +94 −16
  24. drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.c +233 −0
  25. drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.h +69 −0
  26. drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +166 −13
  27. drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c +150 −36
  28. drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +55 −7
  29. drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +4 −0
  30. drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.c +20 −0
  31. drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.h +2 −0
  32. drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c +2 −2
  33. drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c +11 −3
  34. drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c +13 −2
  35. drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h +1 −0
  36. drivers/gpu/drm/amd/amdgpu/amdgpu_job.h +1 −0
  37. drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +71 −10
  38. drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +55 −1051
  39. drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +4 −64
  40. drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5 −1
  41. drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +2 −1
  42. drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +10 −0
  43. drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +10 −5
  44. drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +27 −8
  45. drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +38 −74
  46. drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +75 −15
  47. drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h +14 −0
  48. drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c +43 −65
  49. drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h +9 −9
  50. drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c +19 −6
  51. drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h +2 −1
  52. drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c +2 −3
  53. drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h +2 −0
  54. drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +919 −0
  55. drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h +135 −0
  56. drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c +966 −0
  57. drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.h +76 −0
  58. drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c +2 −2
  59. drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h +0 −1
  60. drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +25 −4
  61. drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +14 −3
  62. drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +3 −2
  63. drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +18 −6
  64. drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h +9 −4
  65. drivers/gpu/drm/amd/amdgpu/atom.c +1 −0
  66. drivers/gpu/drm/amd/amdgpu/cik_sdma.c +4 −14
  67. drivers/gpu/drm/amd/amdgpu/cikd.h +0 −3
  68. drivers/gpu/drm/amd/amdgpu/dce_v10_0.c +2 −2
  69. drivers/gpu/drm/amd/amdgpu/dce_v11_0.c +3 −4
  70. drivers/gpu/drm/amd/amdgpu/dce_v6_0.c +100 −106
  71. drivers/gpu/drm/amd/amdgpu/dce_v8_0.c +8 −8
  72. drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +91 −48
  73. drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +539 −193
  74. drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c +425 −191
  75. drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c +15 −31
  76. drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c +9 −51
  77. drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +5 −31
  78. drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +73 −98
  79. drivers/gpu/drm/amd/amdgpu/gfx_v9_4_2.c +2 −2
  80. drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c +37 −15
  81. drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +0 −4
  82. drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c +1 −5
  83. drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c +1 −5
  84. drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c +29 −17
  85. drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +1 −8
  86. drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +2 −12
  87. drivers/gpu/drm/amd/amdgpu/hdp_v4_0.c +1 −12
  88. drivers/gpu/drm/amd/amdgpu/hdp_v5_0.c +1 −12
  89. drivers/gpu/drm/amd/amdgpu/hdp_v5_2.c +11 −1
  90. drivers/gpu/drm/amd/amdgpu/hdp_v6_0.c +1 −12
  91. drivers/gpu/drm/amd/amdgpu/hdp_v7_0.c +1 −12
  92. drivers/gpu/drm/amd/amdgpu/ih_v6_0.c +5 −1
  93. drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c +2 −2
  94. drivers/gpu/drm/amd/amdgpu/mes_userqueue.c +355 −0
  95. drivers/gpu/drm/amd/amdgpu/mes_userqueue.h +30 −0
  96. drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +27 −47
  97. drivers/gpu/drm/amd/amdgpu/mes_v12_0.c +25 −46
  98. drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c +52 −14
  99. drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +46 −17
  100. drivers/gpu/drm/amd/amdgpu/mxgpu_ai.h +4 −0
  101. drivers/gpu/drm/amd/amdgpu/mxgpu_nv.c +59 −3
  102. drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h +4 −0
  103. drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c +6 −2
  104. drivers/gpu/drm/amd/amdgpu/psp_v12_0.c +17 −85
  105. drivers/gpu/drm/amd/amdgpu/psp_v13_0.c +51 −0
  106. drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c +17 −19
  107. drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c +114 −157
  108. drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c +87 −117
  109. drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c +80 −60
  110. drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c +107 −114
  111. drivers/gpu/drm/amd/amdgpu/si.c +214 −199
  112. drivers/gpu/drm/amd/amdgpu/si_dma.c +66 −50
  113. drivers/gpu/drm/amd/amdgpu/si_enums.h +1 −233
  114. drivers/gpu/drm/amd/amdgpu/si_ih.c +9 −8
  115. drivers/gpu/drm/amd/amdgpu/sid.h +107 −1448
  116. drivers/gpu/drm/amd/amdgpu/soc15.c +5 −0
  117. drivers/gpu/drm/amd/amdgpu/soc15d.h +5 −0
  118. drivers/gpu/drm/amd/amdgpu/umc_v12_0.c +10 −0
  119. drivers/gpu/drm/amd/amdgpu/umc_v12_0.h +2 −0
  120. drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c +1 −0
  121. drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c +1 −0
  122. drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c +1 −0
  123. drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c +19 −2
  124. drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c +62 −22
  125. drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c +27 −0
  126. drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c +18 −2
  127. drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c +2 −2
  128. drivers/gpu/drm/amd/amdgpu/vega20_ih.c +5 −1
  129. drivers/gpu/drm/amd/amdkfd/kfd_debugfs.c +76 −0
  130. drivers/gpu/drm/amd/amdkfd/kfd_device.c +0 −5
  131. drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +8 −6
  132. drivers/gpu/drm/amd/amdkfd/kfd_events.c +19 −0
  133. drivers/gpu/drm/amd/amdkfd/kfd_int_process_v10.c +94 −48
  134. drivers/gpu/drm/amd/amdkfd/kfd_int_process_v11.c +59 −33
  135. drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c +110 −32
  136. drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c +4 −0
  137. drivers/gpu/drm/amd/amdkfd/kfd_packet_manager_v9.c +6 −5
  138. drivers/gpu/drm/amd/amdkfd/kfd_priv.h +7 −0
  139. drivers/gpu/drm/amd/amdkfd/kfd_process.c +7 −0
  140. drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +9 −2
  141. drivers/gpu/drm/amd/amdkfd/kfd_queue.c +2 −2
  142. drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c +22 −2
  143. drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h +1 −0
  144. drivers/gpu/drm/amd/amdkfd/kfd_svm.c +1 −2
  145. drivers/gpu/drm/amd/amdkfd/kfd_topology.c +24 −17
  146. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +375 −214
  147. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h +24 −1
  148. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c +49 −7
  149. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c +13 −0
  150. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c +33 −12
  151. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +2 −2
  152. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.c +0 −8
  153. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.c +2 −2
  154. drivers/gpu/drm/amd/display/dc/Makefile +20 −21
  155. drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c +0 −5
  156. drivers/gpu/drm/amd/display/dc/bios/bios_parser.c +2 −2
  157. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.c +0 −14
  158. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.h +0 −1
  159. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.c +0 −14
  160. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.h +0 −1
  161. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c +14 −15
  162. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn351_clk_mgr.c +1 −0
  163. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_clk_mgr.c +7 −1
  164. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_smu.c +3 −3
  165. drivers/gpu/drm/amd/display/dc/core/dc.c +131 −11
  166. drivers/gpu/drm/amd/display/dc/core/dc_debug.c +2 −0
  167. drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c +2 −2
  168. drivers/gpu/drm/amd/display/dc/core/dc_resource.c +6 −42
  169. drivers/gpu/drm/amd/display/dc/core/dc_state.c +107 −4
  170. drivers/gpu/drm/amd/display/dc/core/dc_stream.c +63 −15
  171. drivers/gpu/drm/amd/display/dc/dc.h +182 −165
  172. drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c +109 −93
  173. drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h +1 −1
  174. drivers/gpu/drm/amd/display/dc/dc_fused_io.c +148 −0
  175. drivers/gpu/drm/amd/display/dc/dc_fused_io.h +31 −0
  176. drivers/gpu/drm/amd/display/dc/dc_helper.c +7 −1
  177. drivers/gpu/drm/amd/display/dc/dc_state_priv.h +20 −0
  178. drivers/gpu/drm/amd/display/dc/dc_stream.h +11 −0
  179. drivers/gpu/drm/amd/display/dc/dc_types.h +15 −2
  180. drivers/gpu/drm/amd/display/dc/dccg/dcn35/dcn35_dccg.c +37 −25
  181. drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c +5 −4
  182. drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.h +1 −1
  183. drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c +2 −0
  184. drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c +2 −1
  185. drivers/gpu/drm/amd/display/dc/dce/dmub_replay.c +13 −6
  186. drivers/gpu/drm/amd/display/dc/dce60/Makefile +1 −2
  187. drivers/gpu/drm/amd/display/dc/dce80/dce80_timing_generator.c +2 −2
  188. drivers/gpu/drm/amd/display/dc/dm_helpers.h +8 −0
  189. drivers/gpu/drm/amd/display/dc/dml/dcn30/display_mode_vba_30.c +1 −0
  190. drivers/gpu/drm/amd/display/dc/dml/dcn31/display_mode_vba_31.c +1 −0
  191. drivers/gpu/drm/amd/display/dc/dml/dcn314/display_mode_vba_314.c +1 −0
  192. drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c +1 −0
  193. drivers/gpu/drm/amd/display/dc/dml2/Makefile +0 −3
  194. drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_translation_helper.c +9 −3
  195. drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_wrapper.c +4 −4
  196. drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top.h +1 −0
  197. drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_dchub_registers.h +2 −0
  198. drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_display_cfg_types.h +2 −1
  199. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4.c +5 −5
  200. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c +2138 −2208
  201. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_shared_types.h +2 −0
  202. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_utils.c +71 −71
  203. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_dpmm/dml2_dpmm_dcn4.c +4 −0
  204. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_mcg/dml2_mcg_dcn4.c +4 −0
  205. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_pmo/dml2_pmo_dcn4_fams2.c +4 −4
  206. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_top/dml2_top_soc15.c +1 −1
  207. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.c +0 −31
  208. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.h +42 −35
  209. drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_internal_shared_types.h +6 −0
  210. drivers/gpu/drm/amd/display/dc/dml2/dml2_dc_resource_mgmt.c +12 −12
  211. drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c +6 −10
  212. drivers/gpu/drm/amd/display/dc/dml2/dml2_wrapper.c +4 −1
  213. drivers/gpu/drm/amd/display/dc/dpp/dcn30/dcn30_dpp.c +3 −6
  214. drivers/gpu/drm/amd/display/dc/dpp/dcn35/dcn35_dpp.c +1 −1
  215. drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.c +0 −7
  216. drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.h +6 −0
  217. drivers/gpu/drm/amd/display/dc/dsc/dcn401/dcn401_dsc.c +0 −6
  218. drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c +14 −0
  219. drivers/gpu/drm/amd/display/dc/hubbub/dcn21/dcn21_hubbub.c +3 −3
  220. drivers/gpu/drm/amd/display/dc/hwss/Makefile +18 −8
  221. drivers/gpu/drm/amd/display/dc/hwss/dce110/dce110_hwseq.c +4 −4
  222. drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.c +1 −1
  223. drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.h +0 −0
  224. drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.c +122 −1
  225. drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.h +7 −0
  226. drivers/gpu/drm/amd/display/dc/hwss/dcn20/dcn20_hwseq.c +1 −1
  227. drivers/gpu/drm/amd/display/dc/hwss/dcn314/dcn314_hwseq.c +14 −0
  228. drivers/gpu/drm/amd/display/dc/hwss/dcn32/dcn32_hwseq.c +3 −1
  229. drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c +14 −7
  230. drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c +2 −0
  231. drivers/gpu/drm/amd/display/dc/hwss/dcn351/dcn351_init.c +3 −1
  232. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c +317 −37
  233. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h +8 −0
  234. drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h +6 −2
  235. drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer_private.h +2 −0
  236. drivers/gpu/drm/amd/display/dc/inc/core_status.h +3 −0
  237. drivers/gpu/drm/amd/display/dc/inc/core_types.h +10 −4
  238. drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h +2 −1
  239. drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h +29 −17
  240. drivers/gpu/drm/amd/display/dc/inc/hw/optc.h +1 −0
  241. drivers/gpu/drm/amd/display/dc/inc/reg_helper.h +4 −0
  242. drivers/gpu/drm/amd/display/dc/inc/resource.h +0 −2
  243. drivers/gpu/drm/amd/display/dc/irq/dcn32/irq_service_dcn32.c +46 −15
  244. drivers/gpu/drm/amd/display/dc/irq/dcn401/irq_service_dcn401.c +46 −14
  245. drivers/gpu/drm/amd/display/dc/irq_types.h +16 −0
  246. drivers/gpu/drm/amd/display/dc/link/link_factory.c +5 −1
  247. drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_dpia.c +1 −0
  248. drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c +4 −3
  249. drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c +7 −4
  250. drivers/gpu/drm/amd/display/dc/mpc/dcn32/dcn32_mpc.c +192 −188
  251. drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.c +0 −11
  252. drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.h +0 −14
  253. drivers/gpu/drm/amd/display/dc/optc/dcn35/dcn35_optc.c +1 −0
  254. drivers/gpu/drm/amd/display/dc/resource/Makefile +18 −8
  255. drivers/gpu/drm/amd/display/dc/resource/dce100/dce100_resource.c +3 −3
  256. drivers/gpu/drm/amd/display/dc/resource/dce110/dce110_resource.c +2 −2
  257. drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.c +2 −2
  258. drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.h +1 −1
  259. drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.c +3 −3
  260. drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.h +0 −0
  261. drivers/gpu/drm/amd/display/dc/resource/dce80/dce80_resource.c +2 −2
  262. drivers/gpu/drm/amd/display/dc/resource/dcn10/dcn10_resource.c +7 −2
  263. drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.c +3 −3
  264. drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.h +1 −1
  265. drivers/gpu/drm/amd/display/dc/resource/dcn21/dcn21_resource.c +3 −3
  266. drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.c +2 −4
  267. drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.h +1 −1
  268. drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.c +2 −2
  269. drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.h +1 −1
  270. drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.c +2 −2
  271. drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.h +1 −1
  272. drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c +59 −13
  273. drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.h +5 −1
  274. drivers/gpu/drm/amd/display/dc/resource/dcn321/dcn321_resource.c +3 −1
  275. drivers/gpu/drm/amd/display/dc/resource/dcn35/dcn35_resource.c +4 −4
  276. drivers/gpu/drm/amd/display/dc/resource/dcn351/dcn351_resource.c +3 −3
  277. drivers/gpu/drm/amd/display/dc/resource/dcn36/dcn36_resource.c +4 −4
  278. drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.c +45 −7
  279. drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.h +1 −1
  280. drivers/gpu/drm/amd/display/dc/sspl/dc_spl.c +14 −11
  281. drivers/gpu/drm/amd/display/dc/sspl/spl_fixpt31_32.c +0 −4
  282. drivers/gpu/drm/amd/display/dmub/dmub_srv.h +110 −47
  283. drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h +73 −2
  284. drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.c +83 −38
  285. drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.h +3 −1
  286. drivers/gpu/drm/amd/display/dmub/src/dmub_srv.c +247 −98
  287. drivers/gpu/drm/amd/display/dmub/src/dmub_srv_stat.c +4 −17
  288. drivers/gpu/drm/amd/display/include/gpio_service_interface.h +3 −0
  289. drivers/gpu/drm/amd/display/modules/hdcp/hdcp.h +1 −0
  290. drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c +43 −10
  291. drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_transition.c +36 −12
  292. drivers/gpu/drm/amd/display/modules/hdcp/hdcp_ddc.c +73 −0
  293. drivers/gpu/drm/amd/display/modules/inc/mod_hdcp.h +36 −2
  294. drivers/gpu/drm/amd/include/amd_shared.h +1 −1
  295. drivers/gpu/drm/amd/include/asic_reg/dce/dce_6_0_sh_mask.h +2 −0
  296. drivers/gpu/drm/amd/include/asic_reg/gca/gfx_6_0_d.h +2 −0
  297. drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_d.h +20 −3
  298. drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_sh_mask.h +41 −0
  299. drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_d.h +44 −0
  300. drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_sh_mask.h +185 −3
  301. drivers/gpu/drm/amd/include/atombios.h +1 −1
  302. drivers/gpu/drm/amd/include/kgd_pp_interface.h +1 −0
  303. drivers/gpu/drm/amd/include/v11_structs.h +4 −4
  304. drivers/gpu/drm/amd/include/v12_structs.h +4 −4
  305. drivers/gpu/drm/amd/pm/amdgpu_dpm.c +43 −0
  306. drivers/gpu/drm/amd/pm/amdgpu_pm.c +34 −0
  307. drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h +3 −0
  308. drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c +190 −168
  309. drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c +24 −18
  310. drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c +8 −0
  311. drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.c +0 −36
  312. drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.h +0 −2
  313. drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c +34 −0
  314. drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h +26 −3
  315. drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h +7 −0
  316. drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h +3 −1
  317. drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h +1 −0
  318. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v11_0.h +0 −5
  319. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v13_0.h +2 −12
  320. drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c +1 −1
  321. drivers/gpu/drm/amd/pm/swsmu/smu11/smu_v11_0.c +0 −39
  322. drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c +11 −2
  323. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c +10 −101
  324. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_12_ppt.c +14 −24
  325. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c +126 −29
  326. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.h +21 −0
  327. drivers/gpu/drm/drm_file.c +34 −0
  328. drivers/gpu/drm/radeon/atombios.h +1 −2
  329. drivers/gpu/drm/radeon/r600_hdmi.c +0 −22
  330. drivers/gpu/drm/radeon/radeon.h +0 −3
  331. drivers/gpu/drm/radeon/radeon_asic.h +0 −1
  332. drivers/gpu/drm/radeon/radeon_cs.c +0 −1
  333. drivers/gpu/drm/radeon/radeon_fence.c +0 −42
  334. drivers/gpu/drm/radeon/sid.h +1 −1
  335. include/drm/drm_file.h +3 −0
  336. include/linux/dma-fence-unwrap.h +2 −0
  337. include/uapi/drm/amdgpu_drm.h +316 −0
  338. include/uapi/linux/kfd_ioctl.h +5 −0
Documentation/gpu/amdgpu/amd-hardware-list-info.rst+23−0
@@ -0,0 +1,23 @@
1=================================================
2 AMD Hardware Components Information per Product
3=================================================
4
5On this page, you can find the AMD product name and which component version is
6part of it.
7
8Accelerated Processing Units (APU) Info
9---------------------------------------
10
11.. csv-table::
12 :header-rows: 1
13 :widths: 3, 2, 2, 1, 1, 1, 1
14 :file: ./apu-asic-info-table.csv
15
16Discrete GPU Info
17-----------------
18
19.. csv-table::
20 :header-rows: 1
21 :widths: 3, 2, 2, 1, 1, 1
22 :file: ./dgpu-asic-info-table.csv
23
Documentation/gpu/amdgpu/amdgpu-glossary.rst+75−0
@@ -12,18 +12,39 @@ we have a dedicated glossary for Display Core at · 11 unmodified lines
12 The number of CUs that are active on the system. The number of active12 The number of CUs that are active on the system. The number of active
13 CUs may be less than SE * SH * CU depending on the board configuration.13 CUs may be less than SE * SH * CU depending on the board configuration.
14 14
15 BACO
16 Bus Alive, Chip Off
17
18 BOCO
19 Bus Off, Chip Off
20
15 CE21 CE
16 Constant Engine22 Constant Engine
17 23
24 CIK
25 Sea Islands
26
27 CB
28 Color Buffer
29
18 CP30 CP
19 Command Processor31 Command Processor
20 32
21 CPLIB33 CPLIB
22 Content Protection Library34 Content Protection Library
23 35
36 CS
37 Command Submission
38
39 CSB
40 Clear State Indirect Buffer
41
24 CU42 CU
25 Compute Unit43 Compute Unit
26 44
45 DB
46 Depth Buffer
47
27 DFS48 DFS
28 Digital Frequency Synthesizer49 Digital Frequency Synthesizer
29 50
@@ -33,6 +54,9 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
33 EOP54 EOP
34 End Of Pipe/Pipeline55 End Of Pipe/Pipeline
35 56
57 FLR
58 Function Level Reset
59
36 GART60 GART
37 Graphics Address Remapping Table. This is the name we use for the GPUVM61 Graphics Address Remapping Table. This is the name we use for the GPUVM
38 page table used by the GPU kernel driver. It remaps system resources62 page table used by the GPU kernel driver. It remaps system resources
@@ -45,6 +69,12 @@ we have a dedicated glossary for Display Core at · 6 unmodified lines
45 GC69 GC
46 Graphics and Compute70 Graphics and Compute
47 71
72 GDS
73 Global Data Share
74
75 GE
76 Geometry Engine
77
48 GMC78 GMC
49 Graphic Memory Controller79 Graphic Memory Controller
50 80
@@ -80,6 +110,9 @@ we have a dedicated glossary for Display Core at · 29 unmodified lines
80 KCQ110 KCQ
81 Kernel Compute Queue111 Kernel Compute Queue
82 112
113 KFD
114 Kernel Fusion Driver
115
83 KGQ116 KGQ
84 Kernel Graphics Queue117 Kernel Graphics Queue
85 118
@@ -89,6 +122,9 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
89 MC122 MC
90 Memory Controller123 Memory Controller
91 124
125 MCBP
126 Mid Command Buffer Preemption
127
92 ME128 ME
93 MicroEngine (Graphics)129 MicroEngine (Graphics)
94 130
@@ -104,6 +140,9 @@ we have a dedicated glossary for Display Core at · 9 unmodified lines
104 MQD140 MQD
105 Memory Queue Descriptor141 Memory Queue Descriptor
106 142
143 PA
144 Primitive Assembler / Physical Address
145
107 PFP146 PFP
108 Pre-Fetch Parser (Graphics)147 Pre-Fetch Parser (Graphics)
109 148
@@ -113,24 +152,39 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
113 PSP152 PSP
114 Platform Security Processor153 Platform Security Processor
115 154
155 RB
156 Render Backends. Some people called it ROPs.
157
116 RLC158 RLC
117 RunList Controller. This name is a remnant of past ages and doesn't have159 RunList Controller. This name is a remnant of past ages and doesn't have
118 much meaning today. It's a group of general-purpose helper engines for160 much meaning today. It's a group of general-purpose helper engines for
119 the GFX block. It's involved in GFX power management and SR-IOV, among161 the GFX block. It's involved in GFX power management and SR-IOV, among
120 other things.162 other things.
121 163
164 SC
165 Scan Converter
166
122 SDMA167 SDMA
123 System DMA168 System DMA
124 169
125 SE170 SE
126 Shader Engine171 Shader Engine
127 172
173 SGPR
174 Scalar General-Purpose Registers
175
128 SH176 SH
129 SHader array177 SHader array
130 178
179 SI
180 Southern Islands
181
131 SMU/SMC182 SMU/SMC
132 System Management Unit / System Management Controller183 System Management Unit / System Management Controller
133 184
185 SPI
186 Shader Processor Input
187
134 SRLC188 SRLC
135 Save/Restore List Control189 Save/Restore List Control
136 190
@@ -143,12 +197,21 @@ we have a dedicated glossary for Display Core at · 6 unmodified lines
143 SS197 SS
144 Spread Spectrum198 Spread Spectrum
145 199
200 SX
201 Shader Export
202
146 TA203 TA
147 Trusted Application204 Trusted Application
148 205
206 TC
207 Texture Cache
208
149 TOC209 TOC
150 Table of Contents210 Table of Contents
151 211
212 UMSCH
213 User Mode Scheduler
214
152 UVD215 UVD
153 Unified Video Decoder216 Unified Video Decoder
154 217
@@ -158,5 +221,17 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines
158 VCN221 VCN
159 Video Codec Next222 Video Codec Next
160 223
224 VGPR
225 Vector General-Purpose Registers
226
227 VMID
228 Virtual Memory ID
229
161 VPE230 VPE
162 Video Processing Engine231 Video Processing Engine
232
233 XCC
234 Accelerator Core Complex
235
236 XCP
237 Accelerator Core Partition
Documentation/gpu/amdgpu/apu-asic-info-table.csv+2−0
@@ -13,3 +13,5 @@ Ryzen 7x20 series, Mendocino, 3.1.6, 10.3.7, 3.1.1, 5.2.7, 13.0.8 · 12 unmodified lines
13Ryzen 7x40 series, Phoenix, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.1113Ryzen 7x40 series, Phoenix, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11
14Ryzen 8x40 series, Hawk Point, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.1114Ryzen 8x40 series, Hawk Point, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11
15Ryzen AI 300 series, Strix Point, 3.5.0, 11.5.0, 4.0.5, 6.1.0, 14.0.015Ryzen AI 300 series, Strix Point, 3.5.0, 11.5.0, 4.0.5, 6.1.0, 14.0.0
16Ryzen AI 350 series, Krackan Point, 3.5.0, 11.5.2, 4.0.5, 6.1.2, 14.0.4
17Ryzen AI Max 300 series, Strix Halo, 3.5.1, 11.5.1, 4.0.6, 6.1.1, 14.0.1
Documentation/gpu/amdgpu/debugfs.rst+210−0
@@ -0,0 +1,210 @@
1==============
2AMDGPU DebugFS
3==============
4
5The amdgpu driver provides a number of debugfs files to aid in debugging
6issues in the driver. These are usually found in
7/sys/kernel/debug/dri/<num>.
8
9DebugFS Files
10=============
11
12amdgpu_benchmark
13----------------
14
15Run benchmarks using the DMA engine the driver uses for GPU memory paging.
16Write a number to the file to run the test. The results are written to the
17kernel log. VRAM is on device memory (dGPUs) or carve out (APUs) and GTT
18(Graphics Translation Tables) is system memory that is accessible by the GPU.
19The following tests are available:
20
21- 1: simple test, VRAM to GTT and GTT to VRAM
22- 2: simple test, VRAM to VRAM
23- 3: GTT to VRAM, buffer size sweep, powers of 2
24- 4: VRAM to GTT, buffer size sweep, powers of 2
25- 5: VRAM to VRAM, buffer size sweep, powers of 2
26- 6: GTT to VRAM, buffer size sweep, common display sizes
27- 7: VRAM to GTT, buffer size sweep, common display sizes
28- 8: VRAM to VRAM, buffer size sweep, common display sizes
29
30amdgpu_test_ib
31--------------
32
33Read this file to run simple IB (Indirect Buffer) tests on all kernel managed
34rings. IBs are command buffers usually generated by userspace applications
35which are submitted to the kernel for execution on an particular GPU engine.
36This just runs the simple IB tests included in the kernel. These tests
37are engine specific and verify that IB submission works.
38
39amdgpu_discovery
40----------------
41
42Provides raw access to the IP discovery binary provided by the GPU. Read this
43file to access the raw binary. This is useful for verifying the contents of
44the IP discovery table. It is chip specific.
45
46amdgpu_vbios
47------------
48
49Provides raw access to the ROM binary image from the GPU. Read this file to
50access the raw binary. This is useful for verifying the contents of the
51video BIOS ROM. It is board specific.
52
53amdgpu_evict_gtt
54----------------
55
56Evict all buffers from the GTT memory pool. Read this file to evict all
57buffers from this pool.
58
59amdgpu_evict_vram
60-----------------
61
62Evict all buffers from the VRAM memory pool. Read this file to evict all
63buffers from this pool.
64
65amdgpu_gpu_recover
66------------------
67
68Trigger a GPU reset. Read this file to trigger reset the entire GPU.
69All work currently running on the GPU will be lost.
70
71amdgpu_ring_<name>
72------------------
73
74Provides read access to the kernel managed ring buffers for each ring <name>.
75These are useful for debugging problems on a particular ring. The ring buffer
76is how the CPU sends commands to the GPU. The CPU writes commands into the
77buffer and then asks the GPU engine to process it. This is the raw binary
78contents of the ring buffer. Use a tool like UMR to decode the rings into human
79readable form.
80
81amdgpu_mqd_<name>
82-----------------
83
84Provides read access to the kernel managed MQD (Memory Queue Descriptor) for
85ring <name> managed by the kernel driver. MQDs define the features of the ring
86and are used to store the ring's state when it is not connected to hardware.
87The driver writes the requested ring features and metadata (GPU addresses of
88the ring itself and associated buffers) to the MQD and the firmware uses the MQD
89to populate the hardware when the ring is mapped to a hardware slot. Only
90available on engines which use MQDs. This provides access to the raw MQD
91binary.
92
93amdgpu_error_<name>
94-------------------
95
96Provides an interface to set an error code on the dma fences associated with
97ring <name>. The error code specified is propogated to all fences associated
98with the ring. Use this to inject a fence error into a ring.
99
100amdgpu_pm_info
101--------------
102
103Provides human readable information about the power management features
104and state of the GPU. This includes current GFX clock, Memory clock,
105voltages, average SoC power, temperature, GFX load, Memory load, SMU
106feature mask, VCN power state, clock and power gating features.
107
108amdgpu_firmware_info
109--------------------
110
111Lists the firmware versions for all firmwares used by the GPU. Only
112entries with a non-0 version are valid. If the version is 0, the firmware
113is not valid for the GPU.
114
115amdgpu_fence_info
116-----------------
117
118Shows the last signalled and emitted fence sequence numbers for each
119kernel driver managed ring. Fences are associated with submissions
120to the engine. Emitted fences have been submitted to the ring
121and signalled fences have been signalled by the GPU. Rings with a
122larger emitted fence value have outstanding work that is still being
123processed by the engine that owns that ring. When the emitted and
124signalled fence values are equal, the ring is idle.
125
126amdgpu_gem_info
127---------------
128
129Lists all of the PIDs using the GPU and the GPU buffers that they have
130allocated. This lists the buffer size, pool (VRAM, GTT, etc.), and buffer
131attributes (CPU access required, CPU cache attributes, etc.).
132
133amdgpu_vm_info
134--------------
135
136Lists all of the PIDs using the GPU and the GPU buffers that they have
137allocated as well as the status of those buffers relative to that process'
138GPU virtual address space (e.g., evicted, idle, invalidated, etc.).
139
140amdgpu_sa_info
141--------------
142
143Prints out all of the suballocations (sa) by the suballocation manager in the
144kernel driver. Prints the GPU address, size, and fence info associated
145with each suballocation. The suballocations are used internally within
146the kernel driver for various things.
147
148amdgpu_<pool>_mm
149----------------
150
151Prints TTM information about the memory pool <pool>.
152
153amdgpu_vram
154-----------
155
156Provides direct access to VRAM. Used by tools like UMR to inspect
157objects in VRAM.
158
159amdgpu_iomem
160------------
161
162Provides direct access to GTT memory. Used by tools like UMR to inspect
163GTT memory.
164
165amdgpu_regs_*
166-------------
167
168Provides direct access to various register aperatures on the GPU. Used
169by tools like UMR to access GPU registers.
170
171amdgpu_regs2
172------------
173
174Provides an IOCTL interface used by UMR for interacting with GPU registers.
175
176
177amdgpu_sensors
178--------------
179
180Provides an interface to query GPU power metrics (temperature, average
181power, etc.). Used by tools like UMR to query GPU power metrics.
182
183
184amdgpu_gca_config
185-----------------
186
187Provides an interface to query GPU details (Graphics/Compute Array config,
188PCI config, GPU family, etc.). Used by tools like UMR to query GPU details.
189
190amdgpu_wave
191-----------
192
193Used to query GFX/compute wave information from the hardware. Used by tools
194like UMR to query GFX/compute wave information.
195
196amdgpu_gpr
197----------
198
199Used to query GFX/compute GPR (General Purpose Register) information from the
200hardware. Used by tools like UMR to query GPRs when debugging shaders.
201
202amdgpu_gprwave
203--------------
204
205Provides an IOCTL interface used by UMR for interacting with shader waves.
206
207amdgpu_fw_attestation
208---------------------
209
210Provides an interface for reading back firmware attestation records.
Documentation/gpu/amdgpu/debugging.rst+7−0
@@ -2,6 +2,13 @@ · 1 unmodified line
2 GPU Debugging2 GPU Debugging
3===============3===============
4 4
5General Debugging Options
6=========================
7
8The DebugFS section provides documentation on a number files to aid in debugging
9issues on the GPU.
10
11
5GPUVM Debugging12GPUVM Debugging
6===============13===============
7 14
Documentation/gpu/amdgpu/display/dc-debug.rst+1−1
@@ -154,7 +154,7 @@ of the display parameters, but the userspace might also cause this issue. One · 153 unmodified lines
154way to identify the source of the problem is to take a screenshot or make a154way to identify the source of the problem is to take a screenshot or make a
155desktop video capture when the problem happens; after checking the155desktop video capture when the problem happens; after checking the
156screenshot/video recording, if you don't see any of the artifacts, it means156screenshot/video recording, if you don't see any of the artifacts, it means
157that the issue is likely on the the driver side. If you can still see the157that the issue is likely on the driver side. If you can still see the
158problem in the data collected, it is an issue that probably happened during158problem in the data collected, it is an issue that probably happened during
159rendering, and the display code just got the framebuffer already corrupted.159rendering, and the display code just got the framebuffer already corrupted.
160 160
Documentation/gpu/amdgpu/driver-core.rst+56−25
@@ -67,36 +67,66 @@ GC (Graphics and Compute) · 66 unmodified lines
67 This is the graphics and compute engine, i.e., the block that67 This is the graphics and compute engine, i.e., the block that
68 encompasses the 3D pipeline and and shader blocks. This is by far the68 encompasses the 3D pipeline and and shader blocks. This is by far the
69 largest block on the GPU. The 3D pipeline has tons of sub-blocks. In69 largest block on the GPU. The 3D pipeline has tons of sub-blocks. In
70 addition to that, it also contains the CP microcontrollers (ME, PFP,70 addition to that, it also contains the CP microcontrollers (ME, PFP, CE,
71 CE, MEC) and the RLC microcontroller. It's exposed to userspace for71 MEC) and the RLC microcontroller. It's exposed to userspace for user mode
72 user mode drivers (OpenGL, Vulkan, OpenCL, etc.)72 drivers (OpenGL, Vulkan, OpenCL, etc.). More details in :ref:`Graphics (GFX)
73 and Compute <amdgpu-gc>`.
73 74
74VCN (Video Core Next)75VCN (Video Core Next)
75 This is the multi-media engine. It handles video and image encode and76 This is the multi-media engine. It handles video and image encode and
76 decode. It's exposed to userspace for user mode drivers (VA-API,77 decode. It's exposed to userspace for user mode drivers (VA-API,
77 OpenMAX, etc.)78 OpenMAX, etc.)
78 79
79Graphics and Compute Microcontrollers80.. _pipes-and-queues-description:
80-------------------------------------81
81 82GFX, Compute, and SDMA Overall Behavior
82CP (Command Processor)83=======================================
83 The name for the hardware block that encompasses the front end of the84
84 GFX/Compute pipeline. Consists mainly of a bunch of microcontrollers85.. note:: For simplicity, whenever the term block is used in this section, it
85 (PFP, ME, CE, MEC). The firmware that runs on these microcontrollers86 means GFX, Compute, and SDMA.
86 provides the driver interface to interact with the GFX/Compute engine.87
87 88GFX, Compute and SDMA share a similar form of operation that can be abstracted
88 MEC (MicroEngine Compute)89to facilitate understanding of the behavior of these blocks. See the figure
89 This is the microcontroller that controls the compute queues on the90below illustrating the common components of these blocks:
90 GFX/compute engine.91
91 92.. kernel-figure:: pipe_and_queue_abstraction.svg
92 MES (MicroEngine Scheduler)93
93 This is a new engine for managing queues. This is currently unused.94In the central part of this figure, you can see two hardware elements, one called
94 95**Pipes** and another called **Queues**; it is important to highlight that Queues
95RLC (RunList Controller)96must be associated with a Pipe and vice-versa. Every specific hardware IP may have
96 This is another microcontroller in the GFX/Compute engine. It handles97a different number of Pipes and, in turn, a different number of Queues; for
97 power management related functionality within the GFX/Compute engine.98example, GFX 11 has two Pipes and two Queues per Pipe for the GFX front end.
98 The name is a vestige of old hardware where it was originally added99
99 and doesn't really have much relation to what the engine does now.100Pipe is the hardware that processes the instructions available in the Queues;
101in other words, it is a thread executing the operations inserted in the Queue.
102One crucial characteristic of Pipes is that they can only execute one Queue at
103a time; no matter if the hardware has multiple Queues in the Pipe, it only runs
104one Queue per Pipe.
105
106Pipes have the mechanics of swapping between queues at the hardware level.
107Nonetheless, they only make use of Queues that are considered mapped. Pipes can
108switch between queues based on any of the following inputs:
109
1101. Command Stream;
1112. Packet by Packet;
1123. Other hardware requests the change (e.g., MES).
113
114Queues within Pipes are defined by the Hardware Queue Descriptors (HQD).
115Associated with the HQD concept, we have the Memory Queue Descriptor (MQD),
116which is responsible for storing information about the state of each of the
117available Queues in the memory. The state of a Queue contains information such
118as the GPU virtual address of the queue itself, save areas, doorbell, etc. The
119MQD also stores the HQD registers, which are vital for activating or
120deactivating a given Queue. The scheduling firmware (e.g., MES) is responsible
121for loading HQDs from MQDs and vice versa.
122
123The Queue-switching process can also happen with the firmware requesting the
124preemption or unmapping of a Queue. The firmware waits for the HQD_ACTIVE bit
125to change to low before saving the state into the MQD. To make a different
126Queue become active, the firmware copies the MQD state into the HQD registers
127and loads any additional state. Finally, it sets the HQD_ACTIVE bit to high to
128indicate that the queue is active. The Pipe will then execute work from active
129Queues.
100 130
101Driver Structure131Driver Structure
102================132================
@@ -110,7 +140,8 @@ Some useful constructs: · 7 unmodified lines
110KIQ (Kernel Interface Queue)140KIQ (Kernel Interface Queue)
111 This is a control queue used by the kernel driver to manage other gfx141 This is a control queue used by the kernel driver to manage other gfx
112 and compute queues on the GFX/compute engine. You can use it to142 and compute queues on the GFX/compute engine. You can use it to
113 map/unmap additional queues, etc.143 map/unmap additional queues, etc. This is replaced by MES on
144 GFX 11 and newer hardware.
114 145
115IB (Indirect Buffer)146IB (Indirect Buffer)
116 A command buffer for a particular engine. Rather than writing147 A command buffer for a particular engine. Rather than writing
Documentation/gpu/amdgpu/driver-misc.rst+0−17
@@ -50,23 +50,6 @@ board_info · 49 unmodified lines
50.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c50.. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
51 :doc: board_info51 :doc: board_info
52 52
53Accelerated Processing Units (APU) Info
54---------------------------------------
55
56.. csv-table::
57 :header-rows: 1
58 :widths: 3, 2, 2, 1, 1, 1, 1
59 :file: ./apu-asic-info-table.csv
60
61Discrete GPU Info
62-----------------
63
64.. csv-table::
65 :header-rows: 1
66 :widths: 3, 2, 2, 1, 1, 1
67 :file: ./dgpu-asic-info-table.csv
68
69
70GPU Memory Usage Information53GPU Memory Usage Information
71============================54============================
72 55
Documentation/gpu/amdgpu/gc/index.rst+52−0
@@ -0,0 +1,52 @@
1.. _amdgpu-gc:
2
3========================================
4 drm/amdgpu - Graphics and Compute (GC)
5========================================
6
7The relationship between the CPU and GPU can be described as the
8producer-consumer problem, where the CPU fills out a buffer with operations
9(producer) to be executed by the GPU (consumer). The requested operations in
10the buffer are called Command Packets, which can be summarized as a compressed
11way of transmitting command information to the graphics controller.
12
13The component that acts as the front end between the CPU and the GPU is called
14the Command Processor (CP). This component is responsible for providing greater
15flexibility to the GC since CP makes it possible to program various aspects of
16the GPU pipeline. CP also coordinates the communication between the CPU and GPU
17via a mechanism named **Ring Buffers**, where the CPU appends information to
18the buffer while the GPU removes operations. It is relevant to highlight that a
19CPU can add a pointer to the Ring Buffer that points to another region of
20memory outside the Ring Buffer, and CP can handle it; this mechanism is called
21**Indirect Buffer (IB)**. CP receives and parses the Command Streams (CS), and
22writes the operations to the correct hardware blocks.
23
24Graphics (GFX) and Compute Microcontrollers
25-------------------------------------------
26
27GC is a large block, and as a result, it has multiple firmware associated with
28it. Some of them are:
29
30CP (Command Processor)
31 The name for the hardware block that encompasses the front end of the
32 GFX/Compute pipeline. Consists mainly of a bunch of microcontrollers
33 (PFP, ME, CE, MEC). The firmware that runs on these microcontrollers
34 provides the driver interface to interact with the GFX/Compute engine.
35
36 MEC (MicroEngine Compute)
37 This is the microcontroller that controls the compute queues on the
38 GFX/compute engine.
39
40 MES (MicroEngine Scheduler)
41 This is the engine for managing queues. For more details check
42 :ref:`MicroEngine Scheduler (MES) <amdgpu-mes>`.
43
44RLC (RunList Controller)
45 This is another microcontroller in the GFX/Compute engine. It handles
46 power management related functionality within the GFX/Compute engine.
47 The name is a vestige of old hardware where it was originally added
48 and doesn't really have much relation to what the engine does now.
49
50.. toctree::
51
52 mes.rst
Documentation/gpu/amdgpu/gc/mes.rst+38−0
@@ -0,0 +1,38 @@
1.. _amdgpu-mes:
2
3=============================
4 MicroEngine Scheduler (MES)
5=============================
6
7.. note::
8 Queue and ring buffer are used as a synonymous.
9
10.. note::
11 This section assumes that you are familiar with the concept of Pipes, Queues, and GC.
12 If not, check :ref:`GFX, Compute, and SDMA Overall Behavior<pipes-and-queues-description>`
13 and :ref:`drm/amdgpu - Graphics and Compute (GC) <amdgpu-gc>`.
14
15Every GFX has a pipe component with one or more hardware queues. Pipes can
16switch between queues depending on certain conditions, and one of the
17components that can request a queue switch to a pipe is the MicroEngine
18Scheduler (MES). Whenever the driver is initialized, it creates one MQD per
19hardware queue, and then the MQDs are handed to the MES firmware for mapping
20to:
21
221. Kernel Queues (legacy): This queue is statically mapped to HQDs and never
23 preempted. Even though this is a legacy feature, it is the current default, and
24 most existing hardware supports it. When an application submits work to the
25 kernel driver, it submits all of the application command buffers to the kernel
26 queues. The CS IOCTL takes the command buffer from the applications and
27 schedules them on the kernel queue.
28
292. User Queues: These queues are dynamically mapped to the HQDs. Regarding the
30 utilization of User Queues, the userspace application will create its user
31 queues and submit work directly to its user queues with no need to IOCTL for
32 each submission and no need to share a single kernel queue.
33
34In terms of User Queues, MES can dynamically map them to the HQD. If there are
35more MQDs than HQDs, the MES firmware will preempt other user queues to make
36sure each queues get a time slice; in other words, MES is a microcontroller
37that handles the mapping and unmapping of MQDs into HQDs, as well as the
38priorities and oversubscription of MQDs.
Documentation/gpu/amdgpu/index.rst+4−1
@@ -7,8 +7,10 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures. · 6 unmodified lines
7 7
8.. toctree::8.. toctree::
9 9
10 module-parameters
11 driver-core10 driver-core
11 amd-hardware-list-info
12 module-parameters
13 gc/index
12 display/index14 display/index
13 flashing15 flashing
14 xgmi16 xgmi
@@ -16,5 +18,6 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures. · 1 unmodified line
16 thermal18 thermal
17 driver-misc19 driver-misc
18 debugging20 debugging
21 debugfs
19 process-isolation22 process-isolation
20 amdgpu-glossary23 amdgpu-glossary
Documentation/gpu/amdgpu/pipe_and_queue_abstraction.svg+1279−0
@@ -0,0 +1,1279 @@
1<?xml version="1.0" encoding="UTF-8" standalone="no"?>
2<!-- Created with Inkscape (http://www.inkscape.org/) -->
3
4<svg
5 width="395.47891mm"
6 height="234.73715mm"
7 viewBox="0 0 395.47891 234.73714"
8 version="1.1"
9 id="svg1"
10 inkscape:version="1.4 (e7c3feb100, 2024-10-09)"
11 sodipodi:docname="pipe_and_queue_abstraction.svg"
12 xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
13 xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
14 xmlns="http://www.w3.org/2000/svg"
15 xmlns:svg="http://www.w3.org/2000/svg">
16 <sodipodi:namedview
17 id="namedview1"
18 pagecolor="#ffffff"
19 bordercolor="#000000"
20 borderopacity="0.25"
21 inkscape:showpageshadow="2"
22 inkscape:pageopacity="0.0"
23 inkscape:pagecheckerboard="0"
24 inkscape:deskcolor="#d1d1d1"
25 inkscape:document-units="mm"
26 inkscape:zoom="1.6489689"
27 inkscape:cx="713.17296"
28 inkscape:cy="466.65527"
29 inkscape:window-width="3840"
30 inkscape:window-height="2083"
31 inkscape:window-x="0"
32 inkscape:window-y="0"
33 inkscape:window-maximized="1"
34 inkscape:current-layer="layer1" />
35 <defs
36 id="defs1">
37 <marker
38 style="overflow:visible"
39 id="Dot"
40 refX="0"
41 refY="0"
42 orient="auto"
43 inkscape:stockid="Dot"
44 markerWidth="1"
45 markerHeight="1"
46 viewBox="0 0 1 1"
47 inkscape:isstock="true"
48 inkscape:collect="always"
49 preserveAspectRatio="xMidYMid">
50 <path
51 transform="scale(0.5)"
52 style="fill:context-stroke;fill-rule:evenodd;stroke:none"
53 d="M 5,0 C 5,2.76 2.76,5 0,5 -2.76,5 -5,2.76 -5,0 c 0,-2.76 2.3,-5 5,-5 2.76,0 5,2.24 5,5 z"
54 sodipodi:nodetypes="sssss"
55 id="path110" />
56 </marker>
57 <marker
58 style="overflow:visible"
59 id="marker109"
60 refX="0"
61 refY="0"
62 orient="auto-start-reverse"
63 inkscape:stockid="Stylized triangle arrow"
64 markerWidth="1"
65 markerHeight="1"
66 viewBox="0 0 1 1"
67 inkscape:isstock="true"
68 inkscape:collect="always"
69 preserveAspectRatio="xMidYMid">
70 <path
71 transform="scale(0.5)"
72 style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
73 d="m 6,0 c -3,1 -7,3 -9,5 0,0 0,-4 2,-5 -2,-1 -2,-5 -2,-5 2,2 6,4 9,5 z"
74 id="path109" />
75 </marker>
76 <marker
77 style="overflow:visible"
78 id="ArrowTriangleStylized"
79 refX="0"
80 refY="0"
81 orient="auto-start-reverse"
82 inkscape:stockid="Stylized triangle arrow"
83 markerWidth="1"
84 markerHeight="1"
85 viewBox="0 0 1 1"
86 inkscape:isstock="true"
87 inkscape:collect="always"
88 preserveAspectRatio="xMidYMid">
89 <path
90 transform="scale(0.5)"
91 style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
92 d="m 6,0 c -3,1 -7,3 -9,5 0,0 0,-4 2,-5 -2,-1 -2,-5 -2,-5 2,2 6,4 9,5 z"
93 id="path108" />
94 </marker>
95 <marker
96 style="overflow:visible"
97 id="ArrowWide"
98 refX="0"
99 refY="0"
100 orient="auto-start-reverse"
101 inkscape:stockid="Wide arrow"
102 markerWidth="1"
103 markerHeight="1"
104 viewBox="0 0 1 1"
105 inkscape:isstock="true"
106 inkscape:collect="always"
107 preserveAspectRatio="xMidYMid">
108 <path
109 style="fill:none;stroke:context-stroke;stroke-width:1;stroke-linecap:butt"
110 d="M 3,-3 0,0 3,3"
111 transform="rotate(180,0.125,0)"
112 sodipodi:nodetypes="ccc"
113 id="path1" />
114 </marker>
115 <marker
116 style="overflow:visible"
117 id="Triangle"
118 refX="0"
119 refY="0"
120 orient="auto-start-reverse"
121 inkscape:stockid="Triangle arrow"
122 markerWidth="1"
123 markerHeight="1"
124 viewBox="0 0 1 1"
125 inkscape:isstock="true"
126 inkscape:collect="always"
127 preserveAspectRatio="xMidYMid">
128 <path
129 transform="scale(0.5)"
130 style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt"
131 d="M 5.77,0 -2.88,5 V -5 Z"
132 id="path135" />
133 </marker>
134 <marker
135 style="overflow:visible"
136 id="ArrowWideHeavy"
137 refX="0"
138 refY="0"
139 orient="auto-start-reverse"
140 inkscape:stockid="Wide, heavy arrow"
141 markerWidth="1"
142 markerHeight="1"
143 viewBox="0 0 1 1"
144 inkscape:isstock="true"
145 inkscape:collect="always"
146 preserveAspectRatio="xMidYMid">
147 <path
148 style="fill:context-stroke;fill-rule:evenodd;stroke:none"
149 d="m 1,0 -3,3 h -2 l 3,-3 -3,-3 h 2 z"
150 id="path71" />
151 </marker>
152 </defs>
153 <g
154 inkscape:label="Layer 1"
155 inkscape:groupmode="layer"
156 id="layer1"
157 transform="translate(149.03517,55.110629)">
158 <circle
159 style="fill:#ffeeaa;fill-opacity:1;stroke:#1a1a1a;stroke-width:0.733436;stroke-dasharray:none;stroke-dashoffset:0"
160 id="path98"
161 cx="-35.757576"
162 cy="-10.495151"
163 r="44.24876" />
164 <rect
165 style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none"
166 id="rect1"
167 width="167.79619"
168 height="24.831829"
169 x="14.21942"
170 y="57.862854" />
171 <g
172 id="g11"
173 transform="translate(24.021362,-46.545299)">
174 <rect
175 style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
176 id="rect2"
177 width="131.09708"
178 height="5.8163381"
179 x="23.245802"
180 y="107.16314" />
181 <path
182 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
183 d="m 37.884707,107.26337 v 5.53565"
184 id="path2"
185 sodipodi:nodetypes="cc" />
186 <path
187 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
188 d="m 52.466233,107.26337 v 5.53565"
189 id="path3"
190 sodipodi:nodetypes="cc" />
191 <path
192 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
193 d="m 67.047758,107.26337 v 5.53565"
194 id="path4"
195 sodipodi:nodetypes="cc" />
196 <path
197 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
198 d="m 81.629283,107.26337 v 5.53565"
199 id="path5"
200 sodipodi:nodetypes="cc" />
201 <path
202 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
203 d="m 96.210809,107.26337 v 5.53565"
204 id="path6"
205 sodipodi:nodetypes="cc" />
206 <path
207 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
208 d="m 110.79233,107.26337 v 5.53565"
209 id="path7"
210 sodipodi:nodetypes="cc" />
211 <path
212 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
213 d="m 125.37386,107.26337 v 5.53565"
214 id="path8"
215 sodipodi:nodetypes="cc" />
216 <path
217 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
218 d="m 139.95538,107.26337 v 5.53565"
219 id="path9"
220 sodipodi:nodetypes="cc" />
221 <text
222 xml:space="preserve"
223 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
224 x="86.713425"
225 y="110.1963"
226 id="text11"><tspan
227 sodipodi:role="line"
228 id="tspan11"
229 style="stroke:#006680;stroke-width:0.5"
230 x="86.713425"
231 y="110.1963">. . .</tspan></text>
232 </g>
233 <g
234 id="g18"
235 transform="translate(24.021362,-32.25779)">
236 <rect
237 style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
238 id="rect11"
239 width="131.09708"
240 height="5.8163381"
241 x="23.245802"
242 y="107.16314" />
243 <path
244 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
245 d="m 37.884707,107.26337 v 5.53565"
246 id="path11"
247 sodipodi:nodetypes="cc" />
248 <path
249 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
250 d="m 52.466233,107.26337 v 5.53565"
251 id="path12"
252 sodipodi:nodetypes="cc" />
253 <path
254 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
255 d="m 67.047758,107.26337 v 5.53565"
256 id="path13"
257 sodipodi:nodetypes="cc" />
258 <path
259 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
260 d="m 81.629283,107.26337 v 5.53565"
261 id="path14"
262 sodipodi:nodetypes="cc" />
263 <path
264 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
265 d="m 96.210809,107.26337 v 5.53565"
266 id="path15"
267 sodipodi:nodetypes="cc" />
268 <path
269 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
270 d="m 110.79233,107.26337 v 5.53565"
271 id="path16"
272 sodipodi:nodetypes="cc" />
273 <path
274 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
275 d="m 125.37386,107.26337 v 5.53565"
276 id="path17"
277 sodipodi:nodetypes="cc" />
278 <path
279 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
280 d="m 139.95538,107.26337 v 5.53565"
281 id="path18"
282 sodipodi:nodetypes="cc" />
283 <text
284 xml:space="preserve"
285 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
286 x="86.713425"
287 y="110.1963"
288 id="text18"><tspan
289 sodipodi:role="line"
290 id="tspan18"
291 style="stroke:#006680;stroke-width:0.5"
292 x="86.713425"
293 y="110.1963">. . .</tspan></text>
294 </g>
295 <text
296 xml:space="preserve"
297 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
298 x="113.37768"
299 y="68.463142"
300 id="text19"><tspan
301 sodipodi:role="line"
302 id="tspan19"
303 style="writing-mode:tb-rl;stroke-width:0.5"
304 x="113.37768"
305 y="68.463142">. . .</tspan></text>
306 <text
307 xml:space="preserve"
308 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
309 x="25.005701"
310 y="55.308445"
311 id="text844-2-9"><tspan
312 sodipodi:role="line"
313 x="25.005701"
314 y="55.308445"
315 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
316 id="tspan2868">Pipe[0]</tspan></text>
317 <text
318 xml:space="preserve"
319 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
320 x="-74.441521"
321 y="63.075123"
322 id="text844-2-9-4"><tspan
323 sodipodi:role="line"
324 x="-74.441521"
325 y="63.075123"
326 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
327 id="tspan2868-7">MQD</tspan></text>
328 <text
329 xml:space="preserve"
330 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
331 x="30.264952"
332 y="65.490654"
333 id="text20"><tspan
334 sodipodi:role="line"
335 x="30.264952"
336 y="65.490654"
337 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
338 id="tspan20">Queue[0]</tspan></text>
339 <text
340 xml:space="preserve"
341 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
342 x="30.264952"
343 y="79.249001"
344 id="text21"><tspan
345 sodipodi:role="line"
346 x="30.264952"
347 y="79.249001"
348 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
349 id="tspan21">Queue[n]</tspan></text>
350 <text
351 xml:space="preserve"
352 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
353 x="30.264952"
354 y="71.84066"
355 id="text22"><tspan
356 sodipodi:role="line"
357 x="30.264952"
358 y="71.84066"
359 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
360 id="tspan22">...</tspan></text>
361 <g
362 id="g71"
363 transform="translate(-23.283342)">
364 <rect
365 style="fill:#ffffff;fill-opacity:0;stroke:#000000;stroke-width:0.5;stroke-dasharray:0.5, 0.5;stroke-dashoffset:0"
366 id="rect67"
367 width="18.533583"
368 height="114.96632"
369 x="250.9435"
370 y="54.754276"
371 ry="6.0427966" />
372 </g>
373 <rect
374 style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none;stroke-dashoffset:0"
375 id="rect68"
376 width="188.21231"
377 height="139.5948"
378 x="4.0113592"
379 y="37.597778"
380 ry="0" />
381 <g
382 id="g43"
383 transform="translate(0,40.745853)">
384 <rect
385 style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none"
386 id="rect22"
387 width="167.79619"
388 height="24.831829"
389 x="14.21942"
390 y="57.862854" />
391 <g
392 id="g30"
393 transform="translate(24.021362,-46.545299)">
394 <rect
395 style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
396 id="rect23"
397 width="131.09708"
398 height="5.8163381"
399 x="23.245802"
400 y="107.16314" />
401 <path
402 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
403 d="m 37.884707,107.26337 v 5.53565"
404 id="path23"
405 sodipodi:nodetypes="cc" />
406 <path
407 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
408 d="m 52.466233,107.26337 v 5.53565"
409 id="path24"
410 sodipodi:nodetypes="cc" />
411 <path
412 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
413 d="m 67.047758,107.26337 v 5.53565"
414 id="path25"
415 sodipodi:nodetypes="cc" />
416 <path
417 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
418 d="m 81.629283,107.26337 v 5.53565"
419 id="path26"
420 sodipodi:nodetypes="cc" />
421 <path
422 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
423 d="m 96.210809,107.26337 v 5.53565"
424 id="path27"
425 sodipodi:nodetypes="cc" />
426 <path
427 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
428 d="m 110.79233,107.26337 v 5.53565"
429 id="path28"
430 sodipodi:nodetypes="cc" />
431 <path
432 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
433 d="m 125.37386,107.26337 v 5.53565"
434 id="path29"
435 sodipodi:nodetypes="cc" />
436 <path
437 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
438 d="m 139.95538,107.26337 v 5.53565"
439 id="path30"
440 sodipodi:nodetypes="cc" />
441 <text
442 xml:space="preserve"
443 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
444 x="86.713425"
445 y="110.1963"
446 id="text30"><tspan
447 sodipodi:role="line"
448 id="tspan30"
449 style="stroke:#006680;stroke-width:0.5"
450 x="86.713425"
451 y="110.1963">. . .</tspan></text>
452 </g>
453 <g
454 id="g38"
455 transform="translate(24.021362,-32.25779)">
456 <rect
457 style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
458 id="rect30"
459 width="131.09708"
460 height="5.8163381"
461 x="23.245802"
462 y="107.16314" />
463 <path
464 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
465 d="m 37.884707,107.26337 v 5.53565"
466 id="path31"
467 sodipodi:nodetypes="cc" />
468 <path
469 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
470 d="m 52.466233,107.26337 v 5.53565"
471 id="path32"
472 sodipodi:nodetypes="cc" />
473 <path
474 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
475 d="m 67.047758,107.26337 v 5.53565"
476 id="path33"
477 sodipodi:nodetypes="cc" />
478 <path
479 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
480 d="m 81.629283,107.26337 v 5.53565"
481 id="path34"
482 sodipodi:nodetypes="cc" />
483 <path
484 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
485 d="m 96.210809,107.26337 v 5.53565"
486 id="path35"
487 sodipodi:nodetypes="cc" />
488 <path
489 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
490 d="m 110.79233,107.26337 v 5.53565"
491 id="path36"
492 sodipodi:nodetypes="cc" />
493 <path
494 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
495 d="m 125.37386,107.26337 v 5.53565"
496 id="path37"
497 sodipodi:nodetypes="cc" />
498 <path
499 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
500 d="m 139.95538,107.26337 v 5.53565"
501 id="path38"
502 sodipodi:nodetypes="cc" />
503 <text
504 xml:space="preserve"
505 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
506 x="86.713425"
507 y="110.1963"
508 id="text38"><tspan
509 sodipodi:role="line"
510 id="tspan38"
511 style="stroke:#006680;stroke-width:0.5"
512 x="86.713425"
513 y="110.1963">. . .</tspan></text>
514 </g>
515 <text
516 xml:space="preserve"
517 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
518 x="113.37768"
519 y="68.463142"
520 id="text39"><tspan
521 sodipodi:role="line"
522 id="tspan39"
523 style="writing-mode:tb-rl;stroke-width:0.5"
524 x="113.37768"
525 y="68.463142">. . .</tspan></text>
526 <text
527 xml:space="preserve"
528 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
529 x="25.005701"
530 y="55.308445"
531 id="text40"><tspan
532 sodipodi:role="line"
533 x="25.005701"
534 y="55.308445"
535 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
536 id="tspan40">Pipe[1]</tspan></text>
537 <text
538 xml:space="preserve"
539 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
540 x="30.264952"
541 y="65.490654"
542 id="text41"><tspan
543 sodipodi:role="line"
544 x="30.264952"
545 y="65.490654"
546 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
547 id="tspan41">Queue[0]</tspan></text>
548 <text
549 xml:space="preserve"
550 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
551 x="30.264952"
552 y="79.249001"
553 id="text42"><tspan
554 sodipodi:role="line"
555 x="30.264952"
556 y="79.249001"
557 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
558 id="tspan42">Queue[n]</tspan></text>
559 <text
560 xml:space="preserve"
561 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
562 x="30.264952"
563 y="71.84066"
564 id="text43"><tspan
565 sodipodi:role="line"
566 x="30.264952"
567 y="71.84066"
568 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
569 id="tspan43">...</tspan></text>
570 </g>
571 <g
572 id="g64"
573 transform="translate(0,85.195881)">
574 <rect
575 style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none"
576 id="rect43"
577 width="167.79619"
578 height="24.831829"
579 x="14.21942"
580 y="57.862854" />
581 <g
582 id="g51"
583 transform="translate(24.021362,-46.545299)">
584 <rect
585 style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
586 id="rect44"
587 width="131.09708"
588 height="5.8163381"
589 x="23.245802"
590 y="107.16314" />
591 <path
592 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
593 d="m 37.884707,107.26337 v 5.53565"
594 id="path44"
595 sodipodi:nodetypes="cc" />
596 <path
597 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
598 d="m 52.466233,107.26337 v 5.53565"
599 id="path45"
600 sodipodi:nodetypes="cc" />
601 <path
602 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
603 d="m 67.047758,107.26337 v 5.53565"
604 id="path46"
605 sodipodi:nodetypes="cc" />
606 <path
607 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
608 d="m 81.629283,107.26337 v 5.53565"
609 id="path47"
610 sodipodi:nodetypes="cc" />
611 <path
612 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
613 d="m 96.210809,107.26337 v 5.53565"
614 id="path48"
615 sodipodi:nodetypes="cc" />
616 <path
617 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
618 d="m 110.79233,107.26337 v 5.53565"
619 id="path49"
620 sodipodi:nodetypes="cc" />
621 <path
622 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
623 d="m 125.37386,107.26337 v 5.53565"
624 id="path50"
625 sodipodi:nodetypes="cc" />
626 <path
627 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
628 d="m 139.95538,107.26337 v 5.53565"
629 id="path51"
630 sodipodi:nodetypes="cc" />
631 <text
632 xml:space="preserve"
633 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
634 x="86.713425"
635 y="110.1963"
636 id="text51"><tspan
637 sodipodi:role="line"
638 id="tspan51"
639 style="stroke:#006680;stroke-width:0.5"
640 x="86.713425"
641 y="110.1963">. . .</tspan></text>
642 </g>
643 <g
644 id="g59"
645 transform="translate(24.021362,-32.25779)">
646 <rect
647 style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none"
648 id="rect51"
649 width="131.09708"
650 height="5.8163381"
651 x="23.245802"
652 y="107.16314" />
653 <path
654 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
655 d="m 37.884707,107.26337 v 5.53565"
656 id="path52"
657 sodipodi:nodetypes="cc" />
658 <path
659 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
660 d="m 52.466233,107.26337 v 5.53565"
661 id="path53"
662 sodipodi:nodetypes="cc" />
663 <path
664 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
665 d="m 67.047758,107.26337 v 5.53565"
666 id="path54"
667 sodipodi:nodetypes="cc" />
668 <path
669 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
670 d="m 81.629283,107.26337 v 5.53565"
671 id="path55"
672 sodipodi:nodetypes="cc" />
673 <path
674 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
675 d="m 96.210809,107.26337 v 5.53565"
676 id="path56"
677 sodipodi:nodetypes="cc" />
678 <path
679 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
680 d="m 110.79233,107.26337 v 5.53565"
681 id="path57"
682 sodipodi:nodetypes="cc" />
683 <path
684 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
685 d="m 125.37386,107.26337 v 5.53565"
686 id="path58"
687 sodipodi:nodetypes="cc" />
688 <path
689 style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
690 d="m 139.95538,107.26337 v 5.53565"
691 id="path59"
692 sodipodi:nodetypes="cc" />
693 <text
694 xml:space="preserve"
695 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
696 x="86.713425"
697 y="110.1963"
698 id="text59"><tspan
699 sodipodi:role="line"
700 id="tspan59"
701 style="stroke:#006680;stroke-width:0.5"
702 x="86.713425"
703 y="110.1963">. . .</tspan></text>
704 </g>
705 <text
706 xml:space="preserve"
707 style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
708 x="113.37768"
709 y="68.463142"
710 id="text60"><tspan
711 sodipodi:role="line"
712 id="tspan60"
713 style="writing-mode:tb-rl;stroke-width:0.5"
714 x="113.37768"
715 y="68.463142">. . .</tspan></text>
716 <text
717 xml:space="preserve"
718 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
719 x="25.005701"
720 y="55.308445"
721 id="text61"><tspan
722 sodipodi:role="line"
723 x="25.005701"
724 y="55.308445"
725 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
726 id="tspan61">Pipe[n]</tspan></text>
727 <text
728 xml:space="preserve"
729 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
730 x="30.264952"
731 y="65.490654"
732 id="text62"><tspan
733 sodipodi:role="line"
734 x="30.264952"
735 y="65.490654"
736 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
737 id="tspan62">Queue[0]</tspan></text>
738 <text
739 xml:space="preserve"
740 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
741 x="30.264952"
742 y="79.249001"
743 id="text63"><tspan
744 sodipodi:role="line"
745 x="30.264952"
746 y="79.249001"
747 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
748 id="tspan63">Queue[n]</tspan></text>
749 <text
750 xml:space="preserve"
751 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
752 x="30.264952"
753 y="71.84066"
754 id="text64"><tspan
755 sodipodi:role="line"
756 x="30.264952"
757 y="71.84066"
758 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
759 id="tspan64">...</tspan></text>
760 </g>
761 <text
762 xml:space="preserve"
763 style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.76111px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:#000000;fill-opacity:0;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none"
764 x="92.18071"
765 y="128.21965"
766 id="text65"><tspan
767 sodipodi:role="line"
768 id="tspan65"
769 style="font-size:7.76111px;writing-mode:tb-rl;fill:#1a1a1a;stroke:#000000;stroke-width:0.5"
770 x="92.18071"
771 y="128.21965">...</tspan></text>
772 <text
773 xml:space="preserve"
774 style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.7611px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none"
775 x="198.72205"
776 y="80.708267"
777 id="text66"><tspan
778 sodipodi:role="line"
779 id="tspan66"
780 style="stroke-width:0.5"
781 x="198.72205"
782 y="80.708267" /></text>
783 <text
784 xml:space="preserve"
785 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
786 x="97.905846"
787 y="44.725101"
788 id="text68"><tspan
789 sodipodi:role="line"
790 x="97.905846"
791 y="44.725101"
792 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
793 id="tspan68">Hardware Block</tspan></text>
794 <text
795 xml:space="preserve"
796 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;writing-mode:vertical-lr;text-orientation:upright;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
797 x="236.36934"
798 y="112.10503"
799 id="text68-7"><tspan
800 sodipodi:role="line"
801 x="236.36934"
802 y="112.10503"
803 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;writing-mode:vertical-lr;text-orientation:upright;text-anchor:middle;stroke-width:0.0690111"
804 id="tspan68-4">EXECUTION</tspan></text>
805 <rect
806 style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.940575;stroke-dasharray:7.5246, 0.940575;stroke-dashoffset:0"
807 id="rect68-1"
808 width="68.749969"
809 height="141.2751"
810 x="-129.49162"
811 y="37.881134"
812 ry="0" />
813 <text
814 xml:space="preserve"
815 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
816 x="-114.4223"
817 y="44.966106"
818 id="text68-2"><tspan
819 sodipodi:role="line"
820 x="-114.4223"
821 y="44.966106"
822 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
823 id="tspan68-9">Memory</tspan></text>
824 <text
825 xml:space="preserve"
826 style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12.7px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-orientation:upright;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none;stroke-dashoffset:0"
827 x="212.6013"
828 y="64.823341"
829 id="text69"><tspan
830 sodipodi:role="line"
831 id="tspan69"
832 style="font-size:12.7px;stroke-width:0.5"
833 x="212.6013"
834 y="64.823341" /></text>
835 <g
836 id="g72">
837 <path
838 style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
839 d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
840 id="text70"
841 aria-label="}"
842 sodipodi:nodetypes="ccccsccccccccscscscscc" />
843 <path
844 style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)"
845 d="M 190.40199,70.278769 H 224.6663"
846 id="path70" />
847 </g>
848 <path
849 style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
850 d="m -88.44114,74.337891 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 V 62.57347 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 v -9.376826 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
851 id="text70-1"
852 aria-label="}"
853 sodipodi:nodetypes="ccccsccccccccscscscscc" />
854 <text
855 xml:space="preserve"
856 style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.9333px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-orientation:upright;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0"
857 x="200.88817"
858 y="38.990276"
859 id="text72"><tspan
860 sodipodi:role="line"
861 id="tspan72"
862 style="stroke-width:1.25"
863 x="200.88817"
864 y="38.990276" /></text>
865 <text
866 xml:space="preserve"
867 style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
868 x="205.12828"
869 y="60.090775"
870 id="text73"><tspan
871 sodipodi:role="line"
872 x="205.12828"
873 y="60.090775"
874 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
875 id="tspan73">e.g.,:</tspan><tspan
876 sodipodi:role="line"
877 x="205.12828"
878 y="68.028275"
879 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
880 id="tspan74">queue[0]</tspan></text>
881 <g
882 id="g75"
883 transform="translate(0,40.745853)">
884 <path
885 style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
886 d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
887 id="path74"
888 aria-label="}"
889 sodipodi:nodetypes="ccccsccccccccscscscscc" />
890 <path
891 style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)"
892 d="M 190.40199,70.278769 H 224.6663"
893 id="path75" />
894 </g>
895 <text
896 xml:space="preserve"
897 style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
898 x="205.12828"
899 y="100.83664"
900 id="text76"><tspan
901 sodipodi:role="line"
902 x="205.12828"
903 y="100.83664"
904 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
905 id="tspan75">e.g.,:</tspan><tspan
906 sodipodi:role="line"
907 x="205.12828"
908 y="108.77414"
909 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
910 id="tspan76">queue[4]</tspan></text>
911 <g
912 id="g77"
913 transform="translate(0,85.725048)">
914 <path
915 style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111"
916 d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z"
917 id="path76"
918 aria-label="}"
919 sodipodi:nodetypes="ccccsccccccccscscscscc" />
920 <path
921 style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)"
922 d="M 190.40199,70.278769 H 224.6663"
923 id="path77" />
924 </g>
925 <text
926 xml:space="preserve"
927 style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
928 x="205.12828"
929 y="145.81558"
930 id="text78"><tspan
931 sodipodi:role="line"
932 x="205.12828"
933 y="145.81558"
934 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
935 id="tspan77">e.g.,:</tspan><tspan
936 sodipodi:role="line"
937 x="205.12828"
938 y="153.75308"
939 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
940 id="tspan78">queue[n]</tspan></text>
941 <g
942 id="g81">
943 <text
944 xml:space="preserve"
945 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
946 x="-18.502264"
947 y="65.642387"
948 id="text79"><tspan
949 sodipodi:role="line"
950 x="-18.502264"
951 y="65.642387"
952 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
953 id="tspan79">HQD</tspan></text>
954 <ellipse
955 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
956 id="path79"
957 cx="-18.407015"
958 cy="63.2188"
959 rx="13.317666"
960 ry="4.4124799" />
961 <path
962 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
963 d="M -4.6361224,63.2188 H 10.606988"
964 id="path80" />
965 </g>
966 <g
967 id="g82"
968 transform="translate(0,14.287503)">
969 <text
970 xml:space="preserve"
971 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
972 x="-18.502264"
973 y="65.642387"
974 id="text81"><tspan
975 sodipodi:role="line"
976 x="-18.502264"
977 y="65.642387"
978 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
979 id="tspan81">HQD</tspan></text>
980 <ellipse
981 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
982 id="ellipse81"
983 cx="-18.407015"
984 cy="63.2188"
985 rx="13.317666"
986 ry="4.4124799" />
987 <path
988 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
989 d="M -4.6361224,63.2188 H 10.606988"
990 id="path81" />
991 </g>
992 <g
993 id="g83"
994 transform="translate(0,40.745853)">
995 <text
996 xml:space="preserve"
997 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
998 x="-18.502264"
999 y="65.642387"
1000 id="text82"><tspan
1001 sodipodi:role="line"
1002 x="-18.502264"
1003 y="65.642387"
1004 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1005 id="tspan82">HQD</tspan></text>
1006 <ellipse
1007 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1008 id="ellipse82"
1009 cx="-18.407015"
1010 cy="63.2188"
1011 rx="13.317666"
1012 ry="4.4124799" />
1013 <path
1014 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1015 d="M -4.6361224,63.2188 H 10.606988"
1016 id="path82" />
1017 </g>
1018 <g
1019 id="g84"
1020 transform="translate(0,55.033362)">
1021 <text
1022 xml:space="preserve"
1023 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1024 x="-18.502264"
1025 y="65.642387"
1026 id="text83"><tspan
1027 sodipodi:role="line"
1028 x="-18.502264"
1029 y="65.642387"
1030 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1031 id="tspan83">HQD</tspan></text>
1032 <ellipse
1033 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1034 id="ellipse83"
1035 cx="-18.407015"
1036 cy="63.2188"
1037 rx="13.317666"
1038 ry="4.4124799" />
1039 <path
1040 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1041 d="M -4.6361224,63.2188 H 10.606988"
1042 id="path83" />
1043 </g>
1044 <g
1045 id="g85"
1046 transform="translate(0,85.195881)">
1047 <text
1048 xml:space="preserve"
1049 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1050 x="-18.502264"
1051 y="65.642387"
1052 id="text84"><tspan
1053 sodipodi:role="line"
1054 x="-18.502264"
1055 y="65.642387"
1056 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1057 id="tspan84">HQD</tspan></text>
1058 <ellipse
1059 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1060 id="ellipse84"
1061 cx="-18.407015"
1062 cy="63.2188"
1063 rx="13.317666"
1064 ry="4.4124799" />
1065 <path
1066 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1067 d="M -4.6361224,63.2188 H 10.606988"
1068 id="path84" />
1069 </g>
1070 <g
1071 id="g86"
1072 transform="translate(0,99.48339)">
1073 <text
1074 xml:space="preserve"
1075 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1076 x="-18.502264"
1077 y="65.642387"
1078 id="text85"><tspan
1079 sodipodi:role="line"
1080 x="-18.502264"
1081 y="65.642387"
1082 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111"
1083 id="tspan85">HQD</tspan></text>
1084 <ellipse
1085 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0"
1086 id="ellipse85"
1087 cx="-18.407015"
1088 cy="63.2188"
1089 rx="13.317666"
1090 ry="4.4124799" />
1091 <path
1092 style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)"
1093 d="M -4.6361224,63.2188 H 10.606988"
1094 id="path85" />
1095 </g>
1096 <text
1097 xml:space="preserve"
1098 style="font-style:normal;font-weight:normal;font-size:5.64444px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1099 x="-35.2131"
1100 y="54.673237"
1101 id="text86"><tspan
1102 sodipodi:role="line"
1103 x="-35.2131"
1104 y="54.673237"
1105 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1106 id="tspan86">Registers</tspan></text>
1107 <path
1108 style="fill:#800000;fill-opacity:0;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:1, 1;stroke-dashoffset:0;marker-end:url(#ArrowWide)"
1109 d="m -45.247972,57.442462 v 5.888987 h 11.344412"
1110 id="path86"
1111 sodipodi:nodetypes="ccc" />
1112 <text
1113 xml:space="preserve"
1114 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1115 x="-74.441521"
1116 y="82.441582"
1117 id="text87"><tspan
1118 sodipodi:role="line"
1119 x="-74.441521"
1120 y="82.441582"
1121 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1122 id="tspan87">MQD</tspan></text>
1123 <text
1124 xml:space="preserve"
1125 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1126 x="-74.441521"
1127 y="99.240776"
1128 id="text88"><tspan
1129 sodipodi:role="line"
1130 x="-74.441521"
1131 y="99.240776"
1132 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1133 id="tspan88">MQD</tspan></text>
1134 <text
1135 xml:space="preserve"
1136 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1137 x="-74.441521"
1138 y="116.03998"
1139 id="text89"><tspan
1140 sodipodi:role="line"
1141 x="-74.441521"
1142 y="116.03998"
1143 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1144 id="tspan89">MQD</tspan></text>
1145 <text
1146 xml:space="preserve"
1147 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1148 x="-74.441521"
1149 y="132.83917"
1150 id="text90"><tspan
1151 sodipodi:role="line"
1152 x="-74.441521"
1153 y="132.83917"
1154 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1155 id="tspan90">MQD</tspan></text>
1156 <text
1157 xml:space="preserve"
1158 style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1159 x="-74.441521"
1160 y="168.3002"
1161 id="text91"><tspan
1162 sodipodi:role="line"
1163 x="-74.441521"
1164 y="168.3002"
1165 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1166 id="tspan91">MQD</tspan></text>
1167 <text
1168 xml:space="preserve"
1169 style="font-style:normal;font-weight:normal;font-size:10.5833px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1170 x="-74.441521"
1171 y="148.1461"
1172 id="text92"><tspan
1173 sodipodi:role="line"
1174 x="-74.441521"
1175 y="148.1461"
1176 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:10.5833px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;writing-mode:tb-rl;text-anchor:middle;stroke-width:0.0690111"
1177 id="tspan92">...</tspan></text>
1178 <g
1179 id="g97"
1180 transform="translate(-5.8208336)">
1181 <rect
1182 style="fill:#aaffcc;fill-opacity:1;stroke:#1a1a1a;stroke-width:0.5;stroke-dasharray:none;stroke-dashoffset:0"
1183 id="rect96"
1184 width="58.726093"
1185 height="27.598055"
1186 x="-142.96434"
1187 y="46.852512" />
1188 <text
1189 xml:space="preserve"
1190 style="font-style:normal;font-weight:normal;font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:start;letter-spacing:0px;word-spacing:0px;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1191 x="-139.74622"
1192 y="52.62756"
1193 id="text93"><tspan
1194 sodipodi:role="line"
1195 x="-139.74622"
1196 y="52.62756"
1197 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1198 id="tspan93">HQD Registers</tspan><tspan
1199 sodipodi:role="line"
1200 x="-139.74622"
1201 y="57.91922"
1202 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1203 id="tspan95">Queue Address in the GPU</tspan><tspan
1204 sodipodi:role="line"
1205 x="-139.74622"
1206 y="63.210884"
1207 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1208 id="tspan96">Doorbell</tspan><tspan
1209 sodipodi:role="line"
1210 x="-139.74622"
1211 y="68.502548"
1212 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1213 id="tspan97">...</tspan><tspan
1214 sodipodi:role="line"
1215 x="-139.74622"
1216 y="73.794212"
1217 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1218 id="tspan94" /></text>
1219 </g>
1220 <text
1221 xml:space="preserve"
1222 style="font-style:normal;font-weight:normal;font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:start;letter-spacing:0px;word-spacing:0px;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1223 x="-61.293022"
1224 y="-19.380915"
1225 id="text93-1"><tspan
1226 sodipodi:role="line"
1227 x="-61.293022"
1228 y="-19.380915"
1229 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1230 id="tspan104">SWITCH QUEUE:</tspan><tspan
1231 sodipodi:role="line"
1232 x="-61.293022"
1233 y="-14.089252"
1234 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1235 id="tspan101">WAIT FOR HQD_ACTIVE = 0</tspan><tspan
1236 sodipodi:role="line"
1237 x="-61.293022"
1238 y="-8.7975903"
1239 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1240 id="tspan105">SAVE QUEUE STATE TO THE MQD</tspan><tspan
1241 sodipodi:role="line"
1242 x="-61.293022"
1243 y="-3.505928"
1244 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1245 id="tspan102">COPY NEW MQD STATE</tspan><tspan
1246 sodipodi:role="line"
1247 x="-61.293022"
1248 y="1.7857342"
1249 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111"
1250 id="tspan103">SET HQD_ACTIVE = 1</tspan></text>
1251 <circle
1252 style="fill:#ffeeaa;fill-opacity:1;stroke:#1a1a1a;stroke-width:1.88976;stroke-dasharray:none;stroke-dashoffset:0"
1253 id="path97"
1254 cx="0"
1255 cy="0"
1256 r="0"
1257 transform="matrix(0.26458333,0,0,0.26458333,-149.03517,37.347779)" />
1258 <text
1259 xml:space="preserve"
1260 style="font-style:normal;font-weight:normal;font-size:5.64444px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111"
1261 x="-36.659206"
1262 y="-44.828983"
1263 id="text106"><tspan
1264 sodipodi:role="line"
1265 x="-36.659206"
1266 y="-44.828983"
1267 style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111"
1268 id="tspan106">Firmware</tspan></text>
1269 <path
1270 style="fill:none;fill-opacity:1;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-start:url(#marker109);marker-end:url(#ArrowTriangleStylized)"
1271 d="M -84.242601,-9.1838245 H -98.041629 V 33.17598"
1272 id="path106" />
1273 <path
1274 style="fill:none;fill-opacity:1;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Dot)"
1275 d="M 9.9745536,-9.3442784 H 29.549918 V 37.170287"
1276 id="path107"
1277 sodipodi:nodetypes="ccc" />
1278 </g>
1279</svg>
drivers/dma-buf/dma-fence-unwrap.c+37−14
@@ -79,6 +79,41 @@ static int fence_cmp(const void *_a, const void *_b) · 78 unmodified lines
79 return 0;79 return 0;
80}80}
81 81
82/**
83 * dma_fence_dedup_array - Sort and deduplicate an array of dma_fence pointers
84 * @fences: Array of dma_fence pointers to be deduplicated
85 * @num_fences: Number of entries in the @fences array
86 *
87 * Sorts the input array by context, then removes duplicate
88 * fences with the same context, keeping only the most recent one.
89 *
90 * The array is modified in-place and unreferenced duplicate fences are released
91 * via dma_fence_put(). The function returns the new number of fences after
92 * deduplication.
93 *
94 * Return: Number of unique fences remaining in the array.
95 */
96int dma_fence_dedup_array(struct dma_fence **fences, int num_fences)
97{
98 int i, j;
99
100 sort(fences, num_fences, sizeof(*fences), fence_cmp, NULL);
101
102 /*
103 * Only keep the most recent fence for each context.
104 */
105 j = 0;
106 for (i = 1; i < num_fences; i++) {
107 if (fences[i]->context == fences[j]->context)
108 dma_fence_put(fences[i]);
109 else
110 fences[++j] = fences[i];
111 }
112
113 return ++j;
114}
115EXPORT_SYMBOL_GPL(dma_fence_dedup_array);
116
82/* Implementation for the dma_fence_merge() marco, don't use directly */117/* Implementation for the dma_fence_merge() marco, don't use directly */
83struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences,118struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences,
84 struct dma_fence **fences,119 struct dma_fence **fences,
@@ -87,7 +122,7 @@ struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, · 2 unmodified lines
87 struct dma_fence *tmp, *unsignaled = NULL, **array;122 struct dma_fence *tmp, *unsignaled = NULL, **array;
88 struct dma_fence_array *result;123 struct dma_fence_array *result;
89 ktime_t timestamp;124 ktime_t timestamp;
90 int i, j, count;125 int i, count;
91 126
92 count = 0;127 count = 0;
93 timestamp = ns_to_ktime(0);128 timestamp = ns_to_ktime(0);
@@ -141,19 +176,7 @@ struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, · 47 unmodified lines
141 if (count == 0 || count == 1)176 if (count == 0 || count == 1)
142 goto return_fastpath;177 goto return_fastpath;
143 178
144 sort(array, count, sizeof(*array), fence_cmp, NULL);179 count = dma_fence_dedup_array(array, count);
145
146 /*
147 * Only keep the most recent fence for each context.
148 */
149 j = 0;
150 for (i = 1; i < count; i++) {
151 if (array[i]->context == array[j]->context)
152 dma_fence_put(array[i]);
153 else
154 array[++j] = array[i];
155 }
156 count = ++j;
157 180
158 if (count > 1) {181 if (count > 1) {
159 result = dma_fence_array_create(count, array,182 result = dma_fence_array_create(count, array,
drivers/gpu/drm/amd/amdgpu/Makefile+7−2
@@ -66,7 +66,7 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o amdgpu_kms.o \ · 65 unmodified lines
66 amdgpu_fw_attestation.o amdgpu_securedisplay.o \66 amdgpu_fw_attestation.o amdgpu_securedisplay.o \
67 amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o \67 amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o \
68 amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_aca.o amdgpu_dev_coredump.o \68 amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_aca.o amdgpu_dev_coredump.o \
69 amdgpu_cper.o69 amdgpu_cper.o amdgpu_userq_fence.o amdgpu_eviction_fence.o
70 70
71amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o71amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o
72 72
@@ -174,7 +174,10 @@ amdgpu-y += \ · 101 unmodified lines
174amdgpu-y += \174amdgpu-y += \
175 amdgpu_mes.o \175 amdgpu_mes.o \
176 mes_v11_0.o \176 mes_v11_0.o \
177 mes_v12_0.o177 mes_v12_0.o \
178
179# add GFX userqueue support
180amdgpu-y += mes_userqueue.o
178 181
179# add UVD block182# add UVD block
180amdgpu-y += \183amdgpu-y += \
@@ -253,6 +256,8 @@ amdgpu-y += \ · 72 unmodified lines
253# add amdkfd interfaces256# add amdkfd interfaces
254amdgpu-y += amdgpu_amdkfd.o257amdgpu-y += amdgpu_amdkfd.o
255 258
259# add gfx usermode queue
260amdgpu-y += amdgpu_userq.o
256 261
257ifneq ($(CONFIG_HSA_AMD),)262ifneq ($(CONFIG_HSA_AMD),)
258AMDKFD_PATH := ../amdkfd263AMDKFD_PATH := ../amdkfd
drivers/gpu/drm/amd/amdgpu/amdgpu.h+98−6
@@ -113,6 +113,8 @@ · 112 unmodified lines
113#include "amdgpu_xcp.h"113#include "amdgpu_xcp.h"
114#include "amdgpu_seq64.h"114#include "amdgpu_seq64.h"
115#include "amdgpu_reg_state.h"115#include "amdgpu_reg_state.h"
116#include "amdgpu_userq.h"
117#include "amdgpu_eviction_fence.h"
116#if defined(CONFIG_DRM_AMD_ISP)118#if defined(CONFIG_DRM_AMD_ISP)
117#include "amdgpu_isp.h"119#include "amdgpu_isp.h"
118#endif120#endif
@@ -228,7 +230,7 @@ extern int amdgpu_force_asic_type; · 109 unmodified lines
228extern int amdgpu_smartshift_bias;230extern int amdgpu_smartshift_bias;
229extern int amdgpu_use_xgmi_p2p;231extern int amdgpu_use_xgmi_p2p;
230extern int amdgpu_mtype_local;232extern int amdgpu_mtype_local;
231extern bool enforce_isolation;233extern int amdgpu_enforce_isolation;
232#ifdef CONFIG_HSA_AMD234#ifdef CONFIG_HSA_AMD
233extern int sched_policy;235extern int sched_policy;
234extern bool debug_evictions;236extern bool debug_evictions;
@@ -266,8 +268,10 @@ extern int amdgpu_umsch_mm_fwlog; · 31 unmodified lines
266 268
267extern int amdgpu_user_partt_mode;269extern int amdgpu_user_partt_mode;
268extern int amdgpu_agp;270extern int amdgpu_agp;
271extern int amdgpu_rebar;
269 272
270extern int amdgpu_wbrf;273extern int amdgpu_wbrf;
274extern int amdgpu_user_queue;
271 275
272#define AMDGPU_VM_MAX_NUM_CTX 4096276#define AMDGPU_VM_MAX_NUM_CTX 4096
273#define AMDGPU_SG_THRESHOLD (256*1024*1024)277#define AMDGPU_SG_THRESHOLD (256*1024*1024)
@@ -488,7 +492,6 @@ struct amdgpu_flip_work { · 214 unmodified lines
488 bool async;492 bool async;
489};493};
490 494
491
492/*495/*
493 * file private structure496 * file private structure
494 */497 */
@@ -501,6 +504,11 @@ struct amdgpu_fpriv { · 6 unmodified lines
501 struct mutex bo_list_lock;504 struct mutex bo_list_lock;
502 struct idr bo_list_handles;505 struct idr bo_list_handles;
503 struct amdgpu_ctx_mgr ctx_mgr;506 struct amdgpu_ctx_mgr ctx_mgr;
507 struct amdgpu_userq_mgr userq_mgr;
508
509 /* Eviction fence infra */
510 struct amdgpu_eviction_fence_mgr evf_mgr;
511
504 /** GPU partition selection */512 /** GPU partition selection */
505 uint32_t xcp_id;513 uint32_t xcp_id;
506};514};
@@ -512,12 +520,62 @@ int amdgpu_file_to_fpriv(struct file *filp, struct amdgpu_fpriv **fpriv); · 5 unmodified lines
512 */520 */
513#define AMDGPU_MAX_WB 1024 /* Reserve at most 1024 WB slots for amdgpu-owned rings. */521#define AMDGPU_MAX_WB 1024 /* Reserve at most 1024 WB slots for amdgpu-owned rings. */
514 522
523/**
524 * amdgpu_wb - This struct is used for small GPU memory allocation.
525 *
526 * This struct is used to allocate a small amount of GPU memory that can be
527 * used to shadow certain states into the memory. This is especially useful for
528 * providing easy CPU access to some states without requiring register access
529 * (e.g., if some block is power gated, reading register may be problematic).
530 *
531 * Note: the term writeback was initially used because many of the amdgpu
532 * components had some level of writeback memory, and this struct initially
533 * described those components.
534 */
515struct amdgpu_wb {535struct amdgpu_wb {
536
537 /**
538 * @wb_obj:
539 *
540 * Buffer Object used for the writeback memory.
541 */
516 struct amdgpu_bo *wb_obj;542 struct amdgpu_bo *wb_obj;
543
544 /**
545 * @wb:
546 *
547 * Pointer to the first writeback slot. In terms of CPU address
548 * this value can be accessed directly by using the offset as an index.
549 * For the GPU address, it is necessary to use gpu_addr and the offset.
550 */
517 volatile uint32_t *wb;551 volatile uint32_t *wb;
552
553 /**
554 * @gpu_addr:
555 *
556 * Writeback base address in the GPU.
557 */
518 uint64_t gpu_addr;558 uint64_t gpu_addr;
519 u32 num_wb; /* Number of wb slots actually reserved for amdgpu. */559
560 /**
561 * @num_wb:
562 *
563 * Number of writeback slots reserved for amdgpu.
564 */
565 u32 num_wb;
566
567 /**
568 * @used:
569 *
570 * Track the writeback slot already used.
571 */
520 unsigned long used[DIV_ROUND_UP(AMDGPU_MAX_WB, BITS_PER_LONG)];572 unsigned long used[DIV_ROUND_UP(AMDGPU_MAX_WB, BITS_PER_LONG)];
573
574 /**
575 * @lock:
576 *
577 * Protects read and write of the used field array.
578 */
521 spinlock_t lock;579 spinlock_t lock;
522};580};
523 581
@@ -551,6 +609,7 @@ struct amdgpu_allowed_register_entry { · 27 unmodified lines
551 * are reset depends on the ASIC. Notably doesn't reset IPs609 * are reset depends on the ASIC. Notably doesn't reset IPs
552 * shared with the CPU on APUs or the memory controllers (so610 * shared with the CPU on APUs or the memory controllers (so
553 * VRAM is not lost). Not available on all ASICs.611 * VRAM is not lost). Not available on all ASICs.
612 * @AMD_RESET_LINK: Triggers SW-UP link reset on other GPUs
554 * @AMD_RESET_BACO: BACO (Bus Alive, Chip Off) method powers off and on the card613 * @AMD_RESET_BACO: BACO (Bus Alive, Chip Off) method powers off and on the card
555 * but without powering off the PCI bus. Suitable only for614 * but without powering off the PCI bus. Suitable only for
556 * discrete GPUs.615 * discrete GPUs.
@@ -568,6 +627,7 @@ enum amd_reset_method { · 11 unmodified lines
568 AMD_RESET_METHOD_MODE0,627 AMD_RESET_METHOD_MODE0,
569 AMD_RESET_METHOD_MODE1,628 AMD_RESET_METHOD_MODE1,
570 AMD_RESET_METHOD_MODE2,629 AMD_RESET_METHOD_MODE2,
630 AMD_RESET_METHOD_LINK,
571 AMD_RESET_METHOD_BACO,631 AMD_RESET_METHOD_BACO,
572 AMD_RESET_METHOD_PCI,632 AMD_RESET_METHOD_PCI,
573 AMD_RESET_METHOD_ON_INIT,633 AMD_RESET_METHOD_ON_INIT,
@@ -821,6 +881,11 @@ struct amdgpu_mqd_prop { · 247 unmodified lines
821 uint32_t hqd_queue_priority;881 uint32_t hqd_queue_priority;
822 bool allow_tunneling;882 bool allow_tunneling;
823 bool hqd_active;883 bool hqd_active;
884 uint64_t shadow_addr;
885 uint64_t gds_bkup_addr;
886 uint64_t csa_addr;
887 uint64_t fence_address;
888 bool tmz_queue;
824};889};
825 890
826struct amdgpu_mqd {891struct amdgpu_mqd {
@@ -829,6 +894,12 @@ struct amdgpu_mqd { · 2 unmodified lines
829 struct amdgpu_mqd_prop *p);894 struct amdgpu_mqd_prop *p);
830};895};
831 896
897struct amdgpu_pcie_reset_ctx {
898 bool in_link_reset;
899 bool occurs_dpc;
900 bool audio_suspended;
901};
902
832/*903/*
833 * Custom Init levels could be defined for different situations where a full904 * Custom Init levels could be defined for different situations where a full
834 * initialization of all hardware blocks are not expected. Sample cases are905 * initialization of all hardware blocks are not expected. Sample cases are
@@ -853,6 +924,14 @@ struct amdgpu_init_level { · 18 unmodified lines
853struct amdgpu_reset_domain;924struct amdgpu_reset_domain;
854struct amdgpu_fru_info;925struct amdgpu_fru_info;
855 926
927enum amdgpu_enforce_isolation_mode {
928 AMDGPU_ENFORCE_ISOLATION_DISABLE = 0,
929 AMDGPU_ENFORCE_ISOLATION_ENABLE = 1,
930 AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY = 2,
931 AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER = 3,
932};
933
934
856/*935/*
857 * Non-zero (true) if the GPU has VRAM. Zero (false) otherwise.936 * Non-zero (true) if the GPU has VRAM. Zero (false) otherwise.
858 */937 */
@@ -1081,6 +1160,13 @@ struct amdgpu_device { · 222 unmodified lines
1081 bool enable_uni_mes;1160 bool enable_uni_mes;
1082 struct amdgpu_mes mes;1161 struct amdgpu_mes mes;
1083 struct amdgpu_mqd mqds[AMDGPU_HW_IP_NUM];1162 struct amdgpu_mqd mqds[AMDGPU_HW_IP_NUM];
1163 const struct amdgpu_userq_funcs *userq_funcs[AMDGPU_HW_IP_NUM];
1164
1165 /* xarray used to retrieve the user queue fence driver reference
1166 * in the EOP interrupt handler to signal the particular user
1167 * queue fence.
1168 */
1169 struct xarray userq_xa;
1084 1170
1085 /* df */1171 /* df */
1086 struct amdgpu_df df;1172 struct amdgpu_df df;
@@ -1160,6 +1246,8 @@ struct amdgpu_device { · 73 unmodified lines
1160 struct pci_saved_state *pci_state;1246 struct pci_saved_state *pci_state;
1161 pci_channel_state_t pci_channel_state;1247 pci_channel_state_t pci_channel_state;
1162 1248
1249 struct amdgpu_pcie_reset_ctx pcie_reset_ctx;
1250
1163 /* Track auto wait count on s_barrier settings */1251 /* Track auto wait count on s_barrier settings */
1164 bool barrier_has_auto_waitcnt;1252 bool barrier_has_auto_waitcnt;
1165 1253
@@ -1193,10 +1281,11 @@ struct amdgpu_device { · 27 unmodified lines
1193 bool debug_enable_ras_aca;1281 bool debug_enable_ras_aca;
1194 bool debug_exp_resets;1282 bool debug_exp_resets;
1195 bool debug_disable_gpu_ring_reset;1283 bool debug_disable_gpu_ring_reset;
1284 bool debug_vm_userptr;
1196 1285
1197 /* Protection for the following isolation structure */1286 /* Protection for the following isolation structure */
1198 struct mutex enforce_isolation_mutex;1287 struct mutex enforce_isolation_mutex;
1199 bool enforce_isolation[MAX_XCP];1288 enum amdgpu_enforce_isolation_mode enforce_isolation[MAX_XCP];
1200 struct amdgpu_isolation {1289 struct amdgpu_isolation {
1201 void *owner;1290 void *owner;
1202 struct dma_fence *spearhead;1291 struct dma_fence *spearhead;
@@ -1210,6 +1299,10 @@ struct amdgpu_device { · 7 unmodified lines
1210 * in KFD: VRAM or GTT.1299 * in KFD: VRAM or GTT.
1211 */1300 */
1212 bool apu_prefer_gtt;1301 bool apu_prefer_gtt;
1302
1303 struct list_head userq_mgr_list;
1304 struct mutex userq_mutex;
1305 bool userq_halt_for_enforce_isolation;
1213};1306};
1214 1307
1215static inline uint32_t amdgpu_ip_version(const struct amdgpu_device *adev,1308static inline uint32_t amdgpu_ip_version(const struct amdgpu_device *adev,
@@ -1464,6 +1557,7 @@ void amdgpu_device_program_register_sequence(struct amdgpu_device *adev, · 248 unmodified lines
1464 const u32 array_size);1557 const u32 array_size);
1465 1558
1466int amdgpu_device_mode1_reset(struct amdgpu_device *adev);1559int amdgpu_device_mode1_reset(struct amdgpu_device *adev);
1560int amdgpu_device_link_reset(struct amdgpu_device *adev);
1467bool amdgpu_device_supports_atpx(struct drm_device *dev);1561bool amdgpu_device_supports_atpx(struct drm_device *dev);
1468bool amdgpu_device_supports_px(struct drm_device *dev);1562bool amdgpu_device_supports_px(struct drm_device *dev);
1469bool amdgpu_device_supports_boco(struct drm_device *dev);1563bool amdgpu_device_supports_boco(struct drm_device *dev);
@@ -1614,11 +1708,9 @@ static inline void amdgpu_acpi_get_backlight_caps(struct amdgpu_dm_backlight_cap · 144 unmodified lines
1614#if defined(CONFIG_ACPI) && defined(CONFIG_SUSPEND)1708#if defined(CONFIG_ACPI) && defined(CONFIG_SUSPEND)
1615bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev);1709bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev);
1616bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev);1710bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev);
1617void amdgpu_choose_low_power_state(struct amdgpu_device *adev);
1618#else1711#else
1619static inline bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) { return false; }1712static inline bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) { return false; }
1620static inline bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev) { return false; }1713static inline bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev) { return false; }
1621static inline void amdgpu_choose_low_power_state(struct amdgpu_device *adev) { }
1622#endif1714#endif
1623 1715
1624void amdgpu_register_gpu_instance(struct amdgpu_device *adev);1716void amdgpu_register_gpu_instance(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_aca.c+3−0
@@ -120,6 +120,9 @@ static void aca_smu_bank_dump(struct amdgpu_device *adev, int idx, int total, st · 119 unmodified lines
120 for (i = 0; i < ARRAY_SIZE(aca_regs); i++)120 for (i = 0; i < ARRAY_SIZE(aca_regs); i++)
121 RAS_EVENT_LOG(adev, event_id, HW_ERR "ACA[%02d/%02d].%s=0x%016llx\n",121 RAS_EVENT_LOG(adev, event_id, HW_ERR "ACA[%02d/%02d].%s=0x%016llx\n",
122 idx + 1, total, aca_regs[i].name, bank->regs[aca_regs[i].reg_idx]);122 idx + 1, total, aca_regs[i].name, bank->regs[aca_regs[i].reg_idx]);
123
124 if (ACA_REG__STATUS__SCRUB(bank->regs[ACA_REG_IDX_STATUS]))
125 RAS_EVENT_LOG(adev, event_id, HW_ERR "hardware error logged by the scrubber\n");
123}126}
124 127
125static int aca_smu_get_valid_aca_banks(struct amdgpu_device *adev, enum aca_smu_type type,128static int aca_smu_get_valid_aca_banks(struct amdgpu_device *adev, enum aca_smu_type type,
drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c+0−18
@@ -1533,22 +1533,4 @@ bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) · 1532 unmodified lines
1533#endif /* CONFIG_AMD_PMC */1533#endif /* CONFIG_AMD_PMC */
1534}1534}
1535 1535
1536/**
1537 * amdgpu_choose_low_power_state
1538 *
1539 * @adev: amdgpu_device_pointer
1540 *
1541 * Choose the target low power state for the GPU
1542 */
1543void amdgpu_choose_low_power_state(struct amdgpu_device *adev)
1544{
1545 if (adev->in_runpm)
1546 return;
1547
1548 if (amdgpu_acpi_is_s0ix_active(adev))
1549 adev->in_s0ix = true;
1550 else if (amdgpu_acpi_is_s3_active(adev))
1551 adev->in_s3 = true;
1552}
1553
1554#endif /* CONFIG_SUSPEND */1536#endif /* CONFIG_SUSPEND */
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c+12−0
@@ -2559,6 +2559,18 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info, · 2558 unmodified lines
2559 if (ret != -EFAULT)2559 if (ret != -EFAULT)
2560 return ret;2560 return ret;
2561 2561
2562 /* If applications unmap memory before destroying the userptr
2563 * from the KFD, trigger a segmentation fault in VM debug mode.
2564 */
2565 if (amdgpu_ttm_adev(bo->tbo.bdev)->debug_vm_userptr) {
2566 pr_err("Pid %d unmapped memory before destroying userptr at GPU addr 0x%llx\n",
2567 pid_nr(process_info->pid), mem->va);
2568
2569 // Send GPU VM fault to user space
2570 kfd_signal_vm_fault_event_with_userptr(kfd_lookup_process_by_pid(process_info->pid),
2571 mem->va);
2572 }
2573
2562 ret = 0;2574 ret = 0;
2563 }2575 }
2564 2576
drivers/gpu/drm/amd/amdgpu/amdgpu_cgs.c+24−85
@@ -252,83 +252,22 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 251 unmodified lines
252 252
253 if (!adev->pm.fw) {253 if (!adev->pm.fw) {
254 switch (adev->asic_type) {254 switch (adev->asic_type) {
255 case CHIP_TAHITI:
256 strcpy(fw_name, "radeon/tahiti_smc.bin");
257 break;
258 case CHIP_PITCAIRN:
259 if ((adev->pdev->revision == 0x81) &&
260 ((adev->pdev->device == 0x6810) ||
261 (adev->pdev->device == 0x6811))) {
262 info->is_kicker = true;
263 strcpy(fw_name, "radeon/pitcairn_k_smc.bin");
264 } else {
265 strcpy(fw_name, "radeon/pitcairn_smc.bin");
266 }
267 break;
268 case CHIP_VERDE:
269 if (((adev->pdev->device == 0x6820) &&
270 ((adev->pdev->revision == 0x81) ||
271 (adev->pdev->revision == 0x83))) ||
272 ((adev->pdev->device == 0x6821) &&
273 ((adev->pdev->revision == 0x83) ||
274 (adev->pdev->revision == 0x87))) ||
275 ((adev->pdev->revision == 0x87) &&
276 ((adev->pdev->device == 0x6823) ||
277 (adev->pdev->device == 0x682b)))) {
278 info->is_kicker = true;
279 strcpy(fw_name, "radeon/verde_k_smc.bin");
280 } else {
281 strcpy(fw_name, "radeon/verde_smc.bin");
282 }
283 break;
284 case CHIP_OLAND:
285 if (((adev->pdev->revision == 0x81) &&
286 ((adev->pdev->device == 0x6600) ||
287 (adev->pdev->device == 0x6604) ||
288 (adev->pdev->device == 0x6605) ||
289 (adev->pdev->device == 0x6610))) ||
290 ((adev->pdev->revision == 0x83) &&
291 (adev->pdev->device == 0x6610))) {
292 info->is_kicker = true;
293 strcpy(fw_name, "radeon/oland_k_smc.bin");
294 } else {
295 strcpy(fw_name, "radeon/oland_smc.bin");
296 }
297 break;
298 case CHIP_HAINAN:
299 if (((adev->pdev->revision == 0x81) &&
300 (adev->pdev->device == 0x6660)) ||
301 ((adev->pdev->revision == 0x83) &&
302 ((adev->pdev->device == 0x6660) ||
303 (adev->pdev->device == 0x6663) ||
304 (adev->pdev->device == 0x6665) ||
305 (adev->pdev->device == 0x6667)))) {
306 info->is_kicker = true;
307 strcpy(fw_name, "radeon/hainan_k_smc.bin");
308 } else if ((adev->pdev->revision == 0xc3) &&
309 (adev->pdev->device == 0x6665)) {
310 info->is_kicker = true;
311 strcpy(fw_name, "radeon/banks_k_2_smc.bin");
312 } else {
313 strcpy(fw_name, "radeon/hainan_smc.bin");
314 }
315 break;
316 case CHIP_BONAIRE:255 case CHIP_BONAIRE:
317 if ((adev->pdev->revision == 0x80) ||256 if ((adev->pdev->revision == 0x80) ||
318 (adev->pdev->revision == 0x81) ||257 (adev->pdev->revision == 0x81) ||
319 (adev->pdev->device == 0x665f)) {258 (adev->pdev->device == 0x665f)) {
320 info->is_kicker = true;259 info->is_kicker = true;
321 strcpy(fw_name, "amdgpu/bonaire_k_smc.bin");260 strscpy(fw_name, "amdgpu/bonaire_k_smc.bin");
322 } else {261 } else {
323 strcpy(fw_name, "amdgpu/bonaire_smc.bin");262 strscpy(fw_name, "amdgpu/bonaire_smc.bin");
324 }263 }
325 break;264 break;
326 case CHIP_HAWAII:265 case CHIP_HAWAII:
327 if (adev->pdev->revision == 0x80) {266 if (adev->pdev->revision == 0x80) {
328 info->is_kicker = true;267 info->is_kicker = true;
329 strcpy(fw_name, "amdgpu/hawaii_k_smc.bin");268 strscpy(fw_name, "amdgpu/hawaii_k_smc.bin");
330 } else {269 } else {
331 strcpy(fw_name, "amdgpu/hawaii_smc.bin");270 strscpy(fw_name, "amdgpu/hawaii_smc.bin");
332 }271 }
333 break;272 break;
334 case CHIP_TOPAZ:273 case CHIP_TOPAZ:
@@ -338,76 +277,76 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 3 unmodified lines
338 ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD1)) ||277 ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD1)) ||
339 ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD3))) {278 ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD3))) {
340 info->is_kicker = true;279 info->is_kicker = true;
341 strcpy(fw_name, "amdgpu/topaz_k_smc.bin");280 strscpy(fw_name, "amdgpu/topaz_k_smc.bin");
342 } else281 } else
343 strcpy(fw_name, "amdgpu/topaz_smc.bin");282 strscpy(fw_name, "amdgpu/topaz_smc.bin");
344 break;283 break;
345 case CHIP_TONGA:284 case CHIP_TONGA:
346 if (((adev->pdev->device == 0x6939) && (adev->pdev->revision == 0xf1)) ||285 if (((adev->pdev->device == 0x6939) && (adev->pdev->revision == 0xf1)) ||
347 ((adev->pdev->device == 0x6938) && (adev->pdev->revision == 0xf1))) {286 ((adev->pdev->device == 0x6938) && (adev->pdev->revision == 0xf1))) {
348 info->is_kicker = true;287 info->is_kicker = true;
349 strcpy(fw_name, "amdgpu/tonga_k_smc.bin");288 strscpy(fw_name, "amdgpu/tonga_k_smc.bin");
350 } else289 } else
351 strcpy(fw_name, "amdgpu/tonga_smc.bin");290 strscpy(fw_name, "amdgpu/tonga_smc.bin");
352 break;291 break;
353 case CHIP_FIJI:292 case CHIP_FIJI:
354 strcpy(fw_name, "amdgpu/fiji_smc.bin");293 strscpy(fw_name, "amdgpu/fiji_smc.bin");
355 break;294 break;
356 case CHIP_POLARIS11:295 case CHIP_POLARIS11:
357 if (type == CGS_UCODE_ID_SMU) {296 if (type == CGS_UCODE_ID_SMU) {
358 if (ASICID_IS_P21(adev->pdev->device, adev->pdev->revision)) {297 if (ASICID_IS_P21(adev->pdev->device, adev->pdev->revision)) {
359 info->is_kicker = true;298 info->is_kicker = true;
360 strcpy(fw_name, "amdgpu/polaris11_k_smc.bin");299 strscpy(fw_name, "amdgpu/polaris11_k_smc.bin");
361 } else if (ASICID_IS_P31(adev->pdev->device, adev->pdev->revision)) {300 } else if (ASICID_IS_P31(adev->pdev->device, adev->pdev->revision)) {
362 info->is_kicker = true;301 info->is_kicker = true;
363 strcpy(fw_name, "amdgpu/polaris11_k2_smc.bin");302 strscpy(fw_name, "amdgpu/polaris11_k2_smc.bin");
364 } else {303 } else {
365 strcpy(fw_name, "amdgpu/polaris11_smc.bin");304 strscpy(fw_name, "amdgpu/polaris11_smc.bin");
366 }305 }
367 } else if (type == CGS_UCODE_ID_SMU_SK) {306 } else if (type == CGS_UCODE_ID_SMU_SK) {
368 strcpy(fw_name, "amdgpu/polaris11_smc_sk.bin");307 strscpy(fw_name, "amdgpu/polaris11_smc_sk.bin");
369 }308 }
370 break;309 break;
371 case CHIP_POLARIS10:310 case CHIP_POLARIS10:
372 if (type == CGS_UCODE_ID_SMU) {311 if (type == CGS_UCODE_ID_SMU) {
373 if (ASICID_IS_P20(adev->pdev->device, adev->pdev->revision)) {312 if (ASICID_IS_P20(adev->pdev->device, adev->pdev->revision)) {
374 info->is_kicker = true;313 info->is_kicker = true;
375 strcpy(fw_name, "amdgpu/polaris10_k_smc.bin");314 strscpy(fw_name, "amdgpu/polaris10_k_smc.bin");
376 } else if (ASICID_IS_P30(adev->pdev->device, adev->pdev->revision)) {315 } else if (ASICID_IS_P30(adev->pdev->device, adev->pdev->revision)) {
377 info->is_kicker = true;316 info->is_kicker = true;
378 strcpy(fw_name, "amdgpu/polaris10_k2_smc.bin");317 strscpy(fw_name, "amdgpu/polaris10_k2_smc.bin");
379 } else {318 } else {
380 strcpy(fw_name, "amdgpu/polaris10_smc.bin");319 strscpy(fw_name, "amdgpu/polaris10_smc.bin");
381 }320 }
382 } else if (type == CGS_UCODE_ID_SMU_SK) {321 } else if (type == CGS_UCODE_ID_SMU_SK) {
383 strcpy(fw_name, "amdgpu/polaris10_smc_sk.bin");322 strscpy(fw_name, "amdgpu/polaris10_smc_sk.bin");
384 }323 }
385 break;324 break;
386 case CHIP_POLARIS12:325 case CHIP_POLARIS12:
387 if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision)) {326 if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision)) {
388 info->is_kicker = true;327 info->is_kicker = true;
389 strcpy(fw_name, "amdgpu/polaris12_k_smc.bin");328 strscpy(fw_name, "amdgpu/polaris12_k_smc.bin");
390 } else {329 } else {
391 strcpy(fw_name, "amdgpu/polaris12_smc.bin");330 strscpy(fw_name, "amdgpu/polaris12_smc.bin");
392 }331 }
393 break;332 break;
394 case CHIP_VEGAM:333 case CHIP_VEGAM:
395 strcpy(fw_name, "amdgpu/vegam_smc.bin");334 strscpy(fw_name, "amdgpu/vegam_smc.bin");
396 break;335 break;
397 case CHIP_VEGA10:336 case CHIP_VEGA10:
398 if ((adev->pdev->device == 0x687f) &&337 if ((adev->pdev->device == 0x687f) &&
399 ((adev->pdev->revision == 0xc0) ||338 ((adev->pdev->revision == 0xc0) ||
400 (adev->pdev->revision == 0xc1) ||339 (adev->pdev->revision == 0xc1) ||
401 (adev->pdev->revision == 0xc3)))340 (adev->pdev->revision == 0xc3)))
402 strcpy(fw_name, "amdgpu/vega10_acg_smc.bin");341 strscpy(fw_name, "amdgpu/vega10_acg_smc.bin");
403 else342 else
404 strcpy(fw_name, "amdgpu/vega10_smc.bin");343 strscpy(fw_name, "amdgpu/vega10_smc.bin");
405 break;344 break;
406 case CHIP_VEGA12:345 case CHIP_VEGA12:
407 strcpy(fw_name, "amdgpu/vega12_smc.bin");346 strscpy(fw_name, "amdgpu/vega12_smc.bin");
408 break;347 break;
409 case CHIP_VEGA20:348 case CHIP_VEGA20:
410 strcpy(fw_name, "amdgpu/vega20_smc.bin");349 strscpy(fw_name, "amdgpu/vega20_smc.bin");
411 break;350 break;
412 default:351 default:
413 DRM_ERROR("SMC firmware not supported\n");352 DRM_ERROR("SMC firmware not supported\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_cper.c+2−2
@@ -549,7 +549,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 548 unmodified lines
549{549{
550 int r;550 int r;
551 551
552 if (!amdgpu_aca_is_enabled(adev))552 if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev))
553 return 0;553 return 0;
554 554
555 r = amdgpu_cper_ring_init(adev);555 r = amdgpu_cper_ring_init(adev);
@@ -568,7 +568,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 12 unmodified lines
568 568
569int amdgpu_cper_fini(struct amdgpu_device *adev)569int amdgpu_cper_fini(struct amdgpu_device *adev)
570{570{
571 if (!amdgpu_aca_is_enabled(adev))571 if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev))
572 return 0;572 return 0;
573 573
574 adev->cper.enabled = false;574 adev->cper.enabled = false;
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c+23−1
@@ -296,7 +296,25 @@ static int amdgpu_cs_pass1(struct amdgpu_cs_parser *p, · 295 unmodified lines
296 num_ibs[i], &p->jobs[i]);296 num_ibs[i], &p->jobs[i]);
297 if (ret)297 if (ret)
298 goto free_all_kdata;298 goto free_all_kdata;
299 p->jobs[i]->enforce_isolation = p->adev->enforce_isolation[fpriv->xcp_id];299 switch (p->adev->enforce_isolation[fpriv->xcp_id]) {
300 case AMDGPU_ENFORCE_ISOLATION_DISABLE:
301 default:
302 p->jobs[i]->enforce_isolation = false;
303 p->jobs[i]->run_cleaner_shader = false;
304 break;
305 case AMDGPU_ENFORCE_ISOLATION_ENABLE:
306 p->jobs[i]->enforce_isolation = true;
307 p->jobs[i]->run_cleaner_shader = true;
308 break;
309 case AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY:
310 p->jobs[i]->enforce_isolation = true;
311 p->jobs[i]->run_cleaner_shader = false;
312 break;
313 case AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER:
314 p->jobs[i]->enforce_isolation = true;
315 p->jobs[i]->run_cleaner_shader = false;
316 break;
317 }
300 }318 }
301 p->gang_leader = p->jobs[p->gang_leader_idx];319 p->gang_leader = p->jobs[p->gang_leader_idx];
302 320
@@ -349,6 +367,10 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p, · 46 unmodified lines
349 ring = amdgpu_job_ring(job);367 ring = amdgpu_job_ring(job);
350 ib = &job->ibs[job->num_ibs++];368 ib = &job->ibs[job->num_ibs++];
351 369
370 /* submissions to kernel queues are disabled */
371 if (ring->no_user_submission)
372 return -EINVAL;
373
352 /* MM engine doesn't support user fences */374 /* MM engine doesn't support user fences */
353 if (p->uf_bo && ring->funcs->no_user_fence)375 if (p->uf_bo && ring->funcs->no_user_fence)
354 return -EINVAL;376 return -EINVAL;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c+351−178
@@ -85,6 +85,7 @@ · 84 unmodified lines
85 85
86#if IS_ENABLED(CONFIG_X86)86#if IS_ENABLED(CONFIG_X86)
87#include <asm/intel-family.h>87#include <asm/intel-family.h>
88#include <asm/cpu_device_id.h>
88#endif89#endif
89 90
90MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin");91MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin");
@@ -1680,6 +1681,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev) · 1589 unmodified lines
1680 if (amdgpu_sriov_vf(adev))1681 if (amdgpu_sriov_vf(adev))
1681 return 0;1682 return 0;
1682 1683
1684 if (!amdgpu_rebar)
1685 return 0;
1686
1683 /* resizing on Dell G5 SE platforms causes problems with runtime pm */1687 /* resizing on Dell G5 SE platforms causes problems with runtime pm */
1684 if ((amdgpu_runtime_pm != 0) &&1688 if ((amdgpu_runtime_pm != 0) &&
1685 adev->pdev->vendor == PCI_VENDOR_ID_ATI &&1689 adev->pdev->vendor == PCI_VENDOR_ID_ATI &&
@@ -1870,6 +1874,35 @@ static bool amdgpu_device_pcie_dynamic_switching_supported(struct amdgpu_device · 184 unmodified lines
1870 return true;1874 return true;
1871}1875}
1872 1876
1877static bool amdgpu_device_aspm_support_quirk(struct amdgpu_device *adev)
1878{
1879#if IS_ENABLED(CONFIG_X86)
1880 struct cpuinfo_x86 *c = &cpu_data(0);
1881
1882 if (!(amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 0, 0) ||
1883 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 0, 1)))
1884 return false;
1885
1886 if (c->x86 == 6 &&
1887 adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN5) {
1888 switch (c->x86_model) {
1889 case VFM_MODEL(INTEL_ALDERLAKE):
1890 case VFM_MODEL(INTEL_ALDERLAKE_L):
1891 case VFM_MODEL(INTEL_RAPTORLAKE):
1892 case VFM_MODEL(INTEL_RAPTORLAKE_P):
1893 case VFM_MODEL(INTEL_RAPTORLAKE_S):
1894 return true;
1895 default:
1896 return false;
1897 }
1898 } else {
1899 return false;
1900 }
1901#else
1902 return false;
1903#endif
1904}
1905
1873/**1906/**
1874 * amdgpu_device_should_use_aspm - check if the device should program ASPM1907 * amdgpu_device_should_use_aspm - check if the device should program ASPM
1875 *1908 *
@@ -1894,7 +1927,7 @@ bool amdgpu_device_should_use_aspm(struct amdgpu_device *adev) · 18 unmodified lines
1894 }1927 }
1895 if (adev->flags & AMD_IS_APU)1928 if (adev->flags & AMD_IS_APU)
1896 return false;1929 return false;
1897 if (!(adev->pm.pp_feature & PP_PCIE_DPM_MASK))1930 if (amdgpu_device_aspm_support_quirk(adev))
1898 return false;1931 return false;
1899 return pcie_aspm_enabled(adev->pdev);1932 return pcie_aspm_enabled(adev->pdev);
1900}1933}
@@ -2112,8 +2145,31 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev) · 211 unmodified lines
2112 2145
2113 adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, amdgpu_fw_load_type);2146 adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, amdgpu_fw_load_type);
2114 2147
2115 for (i = 0; i < MAX_XCP; i++)2148 for (i = 0; i < MAX_XCP; i++) {
2116 adev->enforce_isolation[i] = !!enforce_isolation;2149 switch (amdgpu_enforce_isolation) {
2150 case -1:
2151 case 0:
2152 default:
2153 /* disable */
2154 adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE;
2155 break;
2156 case 1:
2157 /* enable */
2158 adev->enforce_isolation[i] =
2159 AMDGPU_ENFORCE_ISOLATION_ENABLE;
2160 break;
2161 case 2:
2162 /* enable legacy mode */
2163 adev->enforce_isolation[i] =
2164 AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY;
2165 break;
2166 case 3:
2167 /* enable only process isolation without submitting cleaner shader */
2168 adev->enforce_isolation[i] =
2169 AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER;
2170 break;
2171 }
2172 }
2117 2173
2118 return 0;2174 return 0;
2119}2175}
@@ -2689,6 +2745,13 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 569 unmodified lines
2689 break;2745 break;
2690 }2746 }
2691 2747
2748 /* Check for IP version 9.4.3 with A0 hardware */
2749 if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) &&
2750 !amdgpu_device_get_rev_id(adev)) {
2751 dev_err(adev->dev, "Unsupported A0 hardware\n");
2752 return -ENODEV; /* device unsupported - no device error */
2753 }
2754
2692 if (amdgpu_has_atpx() &&2755 if (amdgpu_has_atpx() &&
2693 (amdgpu_is_atpx_hybrid() ||2756 (amdgpu_is_atpx_hybrid() ||
2694 amdgpu_has_atpx_dgpu_power_cntl()) &&2757 amdgpu_has_atpx_dgpu_power_cntl()) &&
@@ -2701,7 +2764,6 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 6 unmodified lines
2701 adev->has_pr3 = parent ? pci_pr3_present(parent) : false;2764 adev->has_pr3 = parent ? pci_pr3_present(parent) : false;
2702 }2765 }
2703 2766
2704
2705 adev->pm.pp_feature = amdgpu_pp_feature_mask;2767 adev->pm.pp_feature = amdgpu_pp_feature_mask;
2706 if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS)2768 if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS)
2707 adev->pm.pp_feature &= ~PP_GFXOFF_MASK;2769 adev->pm.pp_feature &= ~PP_GFXOFF_MASK;
@@ -3172,6 +3234,7 @@ static bool amdgpu_device_check_vram_lost(struct amdgpu_device *adev) · 464 unmodified lines
3172 * always assumed to be lost.3234 * always assumed to be lost.
3173 */3235 */
3174 switch (amdgpu_asic_reset_method(adev)) {3236 switch (amdgpu_asic_reset_method(adev)) {
3237 case AMD_RESET_METHOD_LINK:
3175 case AMD_RESET_METHOD_BACO:3238 case AMD_RESET_METHOD_BACO:
3176 case AMD_RESET_METHOD_MODE1:3239 case AMD_RESET_METHOD_MODE1:
3177 return true;3240 return true;
@@ -3455,6 +3518,7 @@ static int amdgpu_device_ip_fini_early(struct amdgpu_device *adev) · 277 unmodified lines
3455 amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE);3518 amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE);
3456 3519
3457 amdgpu_amdkfd_suspend(adev, false);3520 amdgpu_amdkfd_suspend(adev, false);
3521 amdgpu_userq_suspend(adev);
3458 3522
3459 /* Workaround for ASICs need to disable SMC first */3523 /* Workaround for ASICs need to disable SMC first */
3460 amdgpu_device_smu_fini_early(adev);3524 amdgpu_device_smu_fini_early(adev);
@@ -4307,9 +4371,10 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 846 unmodified lines
4307 amdgpu_sync_create(&adev->isolation[i].active);4371 amdgpu_sync_create(&adev->isolation[i].active);
4308 amdgpu_sync_create(&adev->isolation[i].prev);4372 amdgpu_sync_create(&adev->isolation[i].prev);
4309 }4373 }
4310 mutex_init(&adev->gfx.kfd_sch_mutex);4374 mutex_init(&adev->gfx.userq_sch_mutex);
4311 mutex_init(&adev->gfx.workload_profile_mutex);4375 mutex_init(&adev->gfx.workload_profile_mutex);
4312 mutex_init(&adev->vcn.workload_profile_mutex);4376 mutex_init(&adev->vcn.workload_profile_mutex);
4377 mutex_init(&adev->userq_mutex);
4313 4378
4314 amdgpu_device_init_apu_flags(adev);4379 amdgpu_device_init_apu_flags(adev);
4315 4380
@@ -4329,12 +4394,16 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 13 unmodified lines
4329 spin_lock_init(&adev->virt.rlcg_reg_lock);4394 spin_lock_init(&adev->virt.rlcg_reg_lock);
4330 spin_lock_init(&adev->wb.lock);4395 spin_lock_init(&adev->wb.lock);
4331 4396
4397 xa_init_flags(&adev->userq_xa, XA_FLAGS_LOCK_IRQ);
4398
4332 INIT_LIST_HEAD(&adev->reset_list);4399 INIT_LIST_HEAD(&adev->reset_list);
4333 4400
4334 INIT_LIST_HEAD(&adev->ras_list);4401 INIT_LIST_HEAD(&adev->ras_list);
4335 4402
4336 INIT_LIST_HEAD(&adev->pm.od_kobj_list);4403 INIT_LIST_HEAD(&adev->pm.od_kobj_list);
4337 4404
4405 INIT_LIST_HEAD(&adev->userq_mgr_list);
4406
4338 INIT_DELAYED_WORK(&adev->delayed_init_work,4407 INIT_DELAYED_WORK(&adev->delayed_init_work,
4339 amdgpu_device_delayed_init_work_handler);4408 amdgpu_device_delayed_init_work_handler);
4340 INIT_DELAYED_WORK(&adev->gfx.gfx_off_delay_work,4409 INIT_DELAYED_WORK(&adev->gfx.gfx_off_delay_work,
@@ -4907,28 +4976,20 @@ static int amdgpu_device_evict_resources(struct amdgpu_device *adev) · 566 unmodified lines
4907 * @data: data4976 * @data: data
4908 *4977 *
4909 * This function is called when the system is about to suspend or hibernate.4978 * This function is called when the system is about to suspend or hibernate.
4910 * It is used to evict resources from the device before the system goes to4979 * It is used to set the appropriate flags so that eviction can be optimized
4911 * sleep while there is still access to swap.4980 * in the pm prepare callback.
4912 */4981 */
4913static int amdgpu_device_pm_notifier(struct notifier_block *nb, unsigned long mode,4982static int amdgpu_device_pm_notifier(struct notifier_block *nb, unsigned long mode,
4914 void *data)4983 void *data)
4915{4984{
4916 struct amdgpu_device *adev = container_of(nb, struct amdgpu_device, pm_nb);4985 struct amdgpu_device *adev = container_of(nb, struct amdgpu_device, pm_nb);
4917 int r;
4918 4986
4919 switch (mode) {4987 switch (mode) {
4920 case PM_HIBERNATION_PREPARE:4988 case PM_HIBERNATION_PREPARE:
4921 adev->in_s4 = true;4989 adev->in_s4 = true;
4922 fallthrough;4990 break;
4923 case PM_SUSPEND_PREPARE:4991 case PM_POST_HIBERNATION:
4924 r = amdgpu_device_evict_resources(adev);4992 adev->in_s4 = false;
4925 /*
4926 * This is considered non-fatal at this time because
4927 * amdgpu_device_prepare() will also fatally evict resources.
4928 * See https://gitlab.freedesktop.org/drm/amd/-/issues/3781
4929 */
4930 if (r)
4931 drm_warn(adev_to_drm(adev), "Failed to evict resources, freeze active processes if problems occur: %d\n", r);
4932 break;4993 break;
4933 }4994 }
4934 4995
@@ -4949,15 +5010,13 @@ int amdgpu_device_prepare(struct drm_device *dev) · 14 unmodified lines
4949 struct amdgpu_device *adev = drm_to_adev(dev);5010 struct amdgpu_device *adev = drm_to_adev(dev);
4950 int i, r;5011 int i, r;
4951 5012
4952 amdgpu_choose_low_power_state(adev);
4953
4954 if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)5013 if (dev->switch_power_state == DRM_SWITCH_POWER_OFF)
4955 return 0;5014 return 0;
4956 5015
4957 /* Evict the majority of BOs before starting suspend sequence */5016 /* Evict the majority of BOs before starting suspend sequence */
4958 r = amdgpu_device_evict_resources(adev);5017 r = amdgpu_device_evict_resources(adev);
4959 if (r)5018 if (r)
4960 goto unprepare;5019 return r;
4961 5020
4962 flush_delayed_work(&adev->gfx.gfx_off_delay_work);5021 flush_delayed_work(&adev->gfx.gfx_off_delay_work);
4963 5022
@@ -4968,15 +5027,10 @@ int amdgpu_device_prepare(struct drm_device *dev) · 4 unmodified lines
4968 continue;5027 continue;
4969 r = adev->ip_blocks[i].version->funcs->prepare_suspend(&adev->ip_blocks[i]);5028 r = adev->ip_blocks[i].version->funcs->prepare_suspend(&adev->ip_blocks[i]);
4970 if (r)5029 if (r)
4971 goto unprepare;5030 return r;
4972 }5031 }
4973 5032
4974 return 0;5033 return 0;
4975
4976unprepare:
4977 adev->in_s0ix = adev->in_s3 = adev->in_s4 = false;
4978
4979 return r;
4980}5034}
4981 5035
4982/**5036/**
@@ -5018,8 +5072,10 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients) · 35 unmodified lines
5018 5072
5019 amdgpu_device_ip_suspend_phase1(adev);5073 amdgpu_device_ip_suspend_phase1(adev);
5020 5074
5021 if (!adev->in_s0ix)5075 if (!adev->in_s0ix) {
5022 amdgpu_amdkfd_suspend(adev, adev->in_runpm);5076 amdgpu_amdkfd_suspend(adev, adev->in_runpm);
5077 amdgpu_userq_suspend(adev);
5078 }
5023 5079
5024 r = amdgpu_device_evict_resources(adev);5080 r = amdgpu_device_evict_resources(adev);
5025 if (r)5081 if (r)
@@ -5086,6 +5142,10 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) · 60 unmodified lines
5086 r = amdgpu_amdkfd_resume(adev, adev->in_runpm);5142 r = amdgpu_amdkfd_resume(adev, adev->in_runpm);
5087 if (r)5143 if (r)
5088 goto exit;5144 goto exit;
5145
5146 r = amdgpu_userq_resume(adev);
5147 if (r)
5148 goto exit;
5089 }5149 }
5090 5150
5091 r = amdgpu_device_ip_late_init(adev);5151 r = amdgpu_device_ip_late_init(adev);
@@ -5134,9 +5194,6 @@ exit: · 42 unmodified lines
5134 }5194 }
5135 adev->in_suspend = false;5195 adev->in_suspend = false;
5136 5196
5137 if (adev->enable_mes)
5138 amdgpu_mes_self_test(adev);
5139
5140 if (amdgpu_acpi_smart_shift_update(dev, AMDGPU_SS_DEV_D0))5197 if (amdgpu_acpi_smart_shift_update(dev, AMDGPU_SS_DEV_D0))
5141 DRM_WARN("smart shift update failed\n");5198 DRM_WARN("smart shift update failed\n");
5142 5199
@@ -5517,6 +5574,29 @@ mode1_reset_failed: · 374 unmodified lines
5517 return ret;5574 return ret;
5518}5575}
5519 5576
5577int amdgpu_device_link_reset(struct amdgpu_device *adev)
5578{
5579 int ret = 0;
5580
5581 dev_info(adev->dev, "GPU link reset\n");
5582
5583 if (!adev->pcie_reset_ctx.occurs_dpc)
5584 ret = amdgpu_dpm_link_reset(adev);
5585
5586 if (ret)
5587 goto link_reset_failed;
5588
5589 ret = amdgpu_psp_wait_for_bootloader(adev);
5590 if (ret)
5591 goto link_reset_failed;
5592
5593 return 0;
5594
5595link_reset_failed:
5596 dev_err(adev->dev, "GPU link reset failed\n");
5597 return ret;
5598}
5599
5520int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,5600int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
5521 struct amdgpu_reset_context *reset_context)5601 struct amdgpu_reset_context *reset_context)
5522{5602{
@@ -5821,6 +5901,7 @@ static void amdgpu_device_set_mp1_state(struct amdgpu_device *adev) · 298 unmodified lines
5821 5901
5822 switch (amdgpu_asic_reset_method(adev)) {5902 switch (amdgpu_asic_reset_method(adev)) {
5823 case AMD_RESET_METHOD_MODE1:5903 case AMD_RESET_METHOD_MODE1:
5904 case AMD_RESET_METHOD_LINK:
5824 adev->mp1_state = PP_MP1_STATE_SHUTDOWN;5905 adev->mp1_state = PP_MP1_STATE_SHUTDOWN;
5825 break;5906 break;
5826 case AMD_RESET_METHOD_MODE2:5907 case AMD_RESET_METHOD_MODE2:
@@ -5937,94 +6018,42 @@ static int amdgpu_device_health_check(struct list_head *device_list_handle) · 110 unmodified lines
5937 return ret;6018 return ret;
5938}6019}
5939 6020
5940/**6021static int amdgpu_device_halt_activities(struct amdgpu_device *adev,
5941 * amdgpu_device_gpu_recover - reset the asic and recover scheduler
5942 *
5943 * @adev: amdgpu_device pointer
5944 * @job: which job trigger hang
5945 * @reset_context: amdgpu reset context pointer
5946 *
5947 * Attempt to reset the GPU if it has hung (all asics).
5948 * Attempt to do soft-reset or full-reset and reinitialize Asic
5949 * Returns 0 for success or an error on failure.
5950 */
5951
5952int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
5953 struct amdgpu_job *job,6022 struct amdgpu_job *job,
5954 struct amdgpu_reset_context *reset_context)6023 struct amdgpu_reset_context *reset_context,
6024 struct list_head *device_list,
6025 struct amdgpu_hive_info *hive,
6026 bool need_emergency_restart)
5955{6027{
5956 struct list_head device_list, *device_list_handle = NULL;6028 struct list_head *device_list_handle = NULL;
5957 bool job_signaled = false;
5958 struct amdgpu_hive_info *hive = NULL;
5959 struct amdgpu_device *tmp_adev = NULL;6029 struct amdgpu_device *tmp_adev = NULL;
5960 int i, r = 0;6030 int i, r = 0;
5961 bool need_emergency_restart = false;
5962 bool audio_suspended = false;
5963 int retry_limit = AMDGPU_MAX_RETRY_LIMIT;
5964 6031
5965 /*
5966 * If it reaches here because of hang/timeout and a RAS error is
5967 * detected at the same time, let RAS recovery take care of it.
5968 */
5969 if (amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY) &&
5970 !amdgpu_sriov_vf(adev) &&
5971 reset_context->src != AMDGPU_RESET_SRC_RAS) {
5972 dev_dbg(adev->dev,
5973 "Gpu recovery from source: %d yielding to RAS error recovery handling",
5974 reset_context->src);
5975 return 0;
5976 }
5977 /*
5978 * Special case: RAS triggered and full reset isn't supported
5979 */
5980 need_emergency_restart = amdgpu_ras_need_emergency_restart(adev);
5981
5982 /*
5983 * Flush RAM to disk so that after reboot
5984 * the user can read log and see why the system rebooted.
5985 */
5986 if (need_emergency_restart && amdgpu_ras_get_context(adev) &&
5987 amdgpu_ras_get_context(adev)->reboot) {
5988 DRM_WARN("Emergency reboot.");
5989
5990 ksys_sync_helper();
5991 emergency_restart();
5992 }
5993
5994 dev_info(adev->dev, "GPU %s begin!\n",
5995 need_emergency_restart ? "jobs stop":"reset");
5996
5997 if (!amdgpu_sriov_vf(adev))
5998 hive = amdgpu_get_xgmi_hive(adev);
5999 if (hive)
6000 mutex_lock(&hive->hive_lock);
6001
6002 reset_context->job = job;
6003 reset_context->hive = hive;
6004 /*6032 /*
6005 * Build list of devices to reset.6033 * Build list of devices to reset.
6006 * In case we are in XGMI hive mode, resort the device list6034 * In case we are in XGMI hive mode, resort the device list
6007 * to put adev in the 1st position.6035 * to put adev in the 1st position.
6008 */6036 */
6009 INIT_LIST_HEAD(&device_list);
6010 if (!amdgpu_sriov_vf(adev) && (adev->gmc.xgmi.num_physical_nodes > 1) && hive) {6037 if (!amdgpu_sriov_vf(adev) && (adev->gmc.xgmi.num_physical_nodes > 1) && hive) {
6011 list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {6038 list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
6012 list_add_tail(&tmp_adev->reset_list, &device_list);6039 list_add_tail(&tmp_adev->reset_list, device_list);
6013 if (adev->shutdown)6040 if (adev->shutdown)
6014 tmp_adev->shutdown = true;6041 tmp_adev->shutdown = true;
6042 if (adev->pcie_reset_ctx.occurs_dpc)
6043 tmp_adev->pcie_reset_ctx.in_link_reset = true;
6015 }6044 }
6016 if (!list_is_first(&adev->reset_list, &device_list))6045 if (!list_is_first(&adev->reset_list, device_list))
6017 list_rotate_to_front(&adev->reset_list, &device_list);6046 list_rotate_to_front(&adev->reset_list, device_list);
6018 device_list_handle = &device_list;6047 device_list_handle = device_list;
6019 } else {6048 } else {
6020 list_add_tail(&adev->reset_list, &device_list);6049 list_add_tail(&adev->reset_list, device_list);
6021 device_list_handle = &device_list;6050 device_list_handle = device_list;
6022 }6051 }
6023 6052
6024 if (!amdgpu_sriov_vf(adev)) {6053 if (!amdgpu_sriov_vf(adev) && (!adev->pcie_reset_ctx.occurs_dpc)) {
6025 r = amdgpu_device_health_check(device_list_handle);6054 r = amdgpu_device_health_check(device_list_handle);
6026 if (r)6055 if (r)
6027 goto end_reset;6056 return r;
6028 }6057 }
6029 6058
6030 /* We need to lock reset domain only once both for XGMI and single device */6059 /* We need to lock reset domain only once both for XGMI and single device */
@@ -6048,7 +6077,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 17 unmodified lines
6048 * some audio codec errors.6077 * some audio codec errors.
6049 */6078 */
6050 if (!amdgpu_device_suspend_display_audio(tmp_adev))6079 if (!amdgpu_device_suspend_display_audio(tmp_adev))
6051 audio_suspended = true;6080 tmp_adev->pcie_reset_ctx.audio_suspended = true;
6052 6081
6053 amdgpu_ras_set_error_query_ready(tmp_adev, false);6082 amdgpu_ras_set_error_query_ready(tmp_adev, false);
6054 6083
@@ -6066,6 +6095,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 11 unmodified lines
6066 6095
6067 /* disable ras on ALL IPs */6096 /* disable ras on ALL IPs */
6068 if (!need_emergency_restart &&6097 if (!need_emergency_restart &&
6098 (!adev->pcie_reset_ctx.occurs_dpc) &&
6069 amdgpu_device_ip_need_full_reset(tmp_adev))6099 amdgpu_device_ip_need_full_reset(tmp_adev))
6070 amdgpu_ras_suspend(tmp_adev);6100 amdgpu_ras_suspend(tmp_adev);
6071 6101
@@ -6083,24 +6113,24 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 11 unmodified lines
6083 atomic_inc(&tmp_adev->gpu_reset_counter);6113 atomic_inc(&tmp_adev->gpu_reset_counter);
6084 }6114 }
6085 6115
6086 if (need_emergency_restart)6116 return r;
6087 goto skip_sched_resume;6117}
6088 6118
6089 /*6119static int amdgpu_device_asic_reset(struct amdgpu_device *adev,
6090 * Must check guilty signal here since after this point all old6120 struct list_head *device_list,
6091 * HW fences are force signaled.6121 struct amdgpu_reset_context *reset_context)
6092 *6122{
6093 * job->base holds a reference to parent fence6123 struct amdgpu_device *tmp_adev = NULL;
6094 */6124 int retry_limit = AMDGPU_MAX_RETRY_LIMIT;
6095 if (job && dma_fence_is_signaled(&job->hw_fence)) {6125 int r = 0;
6096 job_signaled = true;
6097 dev_info(adev->dev, "Guilty job already signaled, skipping HW reset");
6098 goto skip_hw_reset;
6099 }
6100 6126
6101retry: /* Rest of adevs pre asic reset from XGMI hive. */6127retry: /* Rest of adevs pre asic reset from XGMI hive. */
6102 list_for_each_entry(tmp_adev, device_list_handle, reset_list) {6128 list_for_each_entry(tmp_adev, device_list, reset_list) {
6129 if (adev->pcie_reset_ctx.occurs_dpc)
6130 tmp_adev->no_hw_access = true;
6103 r = amdgpu_device_pre_asic_reset(tmp_adev, reset_context);6131 r = amdgpu_device_pre_asic_reset(tmp_adev, reset_context);
6132 if (adev->pcie_reset_ctx.occurs_dpc)
6133 tmp_adev->no_hw_access = false;
6104 /*TODO Should we stop ?*/6134 /*TODO Should we stop ?*/
6105 if (r) {6135 if (r) {
6106 dev_err(tmp_adev->dev, "GPU pre asic reset failed with err, %d for drm dev, %s ",6136 dev_err(tmp_adev->dev, "GPU pre asic reset failed with err, %d for drm dev, %s ",
@@ -6112,6 +6142,11 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 5 unmodified lines
6112 /* Actual ASIC resets if needed.*/6142 /* Actual ASIC resets if needed.*/
6113 /* Host driver will handle XGMI hive reset for SRIOV */6143 /* Host driver will handle XGMI hive reset for SRIOV */
6114 if (amdgpu_sriov_vf(adev)) {6144 if (amdgpu_sriov_vf(adev)) {
6145
6146 /* Bail out of reset early */
6147 if (amdgpu_ras_is_rma(adev))
6148 return -ENODEV;
6149
6115 if (amdgpu_ras_get_fed_status(adev) || amdgpu_virt_rcvd_ras_interrupt(adev)) {6150 if (amdgpu_ras_get_fed_status(adev) || amdgpu_virt_rcvd_ras_interrupt(adev)) {
6116 dev_dbg(adev->dev, "Detected RAS error, wait for FLR completion\n");6151 dev_dbg(adev->dev, "Detected RAS error, wait for FLR completion\n");
6117 amdgpu_ras_set_fed(adev, true);6152 amdgpu_ras_set_fed(adev, true);
@@ -6126,12 +6161,12 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 8 unmodified lines
6126 if (r)6161 if (r)
6127 adev->asic_reset_res = r;6162 adev->asic_reset_res = r;
6128 } else {6163 } else {
6129 r = amdgpu_do_asic_reset(device_list_handle, reset_context);6164 r = amdgpu_do_asic_reset(device_list, reset_context);
6130 if (r && r == -EAGAIN)6165 if (r && r == -EAGAIN)
6131 goto retry;6166 goto retry;
6132 }6167 }
6133 6168
6134 list_for_each_entry(tmp_adev, device_list_handle, reset_list) {6169 list_for_each_entry(tmp_adev, device_list, reset_list) {
6135 /*6170 /*
6136 * Drop any pending non scheduler resets queued before reset is done.6171 * Drop any pending non scheduler resets queued before reset is done.
6137 * Any reset scheduled after this point would be valid. Scheduler resets6172 * Any reset scheduled after this point would be valid. Scheduler resets
@@ -6141,10 +6176,18 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 3 unmodified lines
6141 amdgpu_device_stop_pending_resets(tmp_adev);6176 amdgpu_device_stop_pending_resets(tmp_adev);
6142 }6177 }
6143 6178
6144skip_hw_reset:6179 return r;
6180}
6181
6182static int amdgpu_device_sched_resume(struct list_head *device_list,
6183 struct amdgpu_reset_context *reset_context,
6184 bool job_signaled)
6185{
6186 struct amdgpu_device *tmp_adev = NULL;
6187 int i, r = 0;
6145 6188
6146 /* Post ASIC reset for all devs .*/6189 /* Post ASIC reset for all devs .*/
6147 list_for_each_entry(tmp_adev, device_list_handle, reset_list) {6190 list_for_each_entry(tmp_adev, device_list, reset_list) {
6148 6191
6149 for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {6192 for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
6150 struct amdgpu_ring *ring = tmp_adev->rings[i];6193 struct amdgpu_ring *ring = tmp_adev->rings[i];
@@ -6180,8 +6223,16 @@ skip_hw_reset: · 29 unmodified lines
6180 }6223 }
6181 }6224 }
6182 6225
6183skip_sched_resume:6226 return r;
6184 list_for_each_entry(tmp_adev, device_list_handle, reset_list) {6227}
6228
6229static void amdgpu_device_gpu_resume(struct amdgpu_device *adev,
6230 struct list_head *device_list,
6231 bool need_emergency_restart)
6232{
6233 struct amdgpu_device *tmp_adev = NULL;
6234
6235 list_for_each_entry(tmp_adev, device_list, reset_list) {
6185 /* unlock kfd: SRIOV would do it separately */6236 /* unlock kfd: SRIOV would do it separately */
6186 if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev))6237 if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev))
6187 amdgpu_amdkfd_post_reset(tmp_adev);6238 amdgpu_amdkfd_post_reset(tmp_adev);
@@ -6192,18 +6243,114 @@ skip_sched_resume: · 4 unmodified lines
6192 if (!adev->kfd.init_complete)6243 if (!adev->kfd.init_complete)
6193 amdgpu_amdkfd_device_init(adev);6244 amdgpu_amdkfd_device_init(adev);
6194 6245
6195 if (audio_suspended)6246 if (tmp_adev->pcie_reset_ctx.audio_suspended)
6196 amdgpu_device_resume_display_audio(tmp_adev);6247 amdgpu_device_resume_display_audio(tmp_adev);
6197 6248
6198 amdgpu_device_unset_mp1_state(tmp_adev);6249 amdgpu_device_unset_mp1_state(tmp_adev);
6199 6250
6200 amdgpu_ras_set_error_query_ready(tmp_adev, true);6251 amdgpu_ras_set_error_query_ready(tmp_adev, true);
6252
6201 }6253 }
6202 6254
6203 tmp_adev = list_first_entry(device_list_handle, struct amdgpu_device,6255 tmp_adev = list_first_entry(device_list, struct amdgpu_device,
6204 reset_list);6256 reset_list);
6205 amdgpu_device_unlock_reset_domain(tmp_adev->reset_domain);6257 amdgpu_device_unlock_reset_domain(tmp_adev->reset_domain);
6206 6258
6259}
6260
6261
6262/**
6263 * amdgpu_device_gpu_recover - reset the asic and recover scheduler
6264 *
6265 * @adev: amdgpu_device pointer
6266 * @job: which job trigger hang
6267 * @reset_context: amdgpu reset context pointer
6268 *
6269 * Attempt to reset the GPU if it has hung (all asics).
6270 * Attempt to do soft-reset or full-reset and reinitialize Asic
6271 * Returns 0 for success or an error on failure.
6272 */
6273
6274int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
6275 struct amdgpu_job *job,
6276 struct amdgpu_reset_context *reset_context)
6277{
6278 struct list_head device_list;
6279 bool job_signaled = false;
6280 struct amdgpu_hive_info *hive = NULL;
6281 int r = 0;
6282 bool need_emergency_restart = false;
6283
6284 /*
6285 * If it reaches here because of hang/timeout and a RAS error is
6286 * detected at the same time, let RAS recovery take care of it.
6287 */
6288 if (amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY) &&
6289 !amdgpu_sriov_vf(adev) &&
6290 reset_context->src != AMDGPU_RESET_SRC_RAS) {
6291 dev_dbg(adev->dev,
6292 "Gpu recovery from source: %d yielding to RAS error recovery handling",
6293 reset_context->src);
6294 return 0;
6295 }
6296
6297 /*
6298 * Special case: RAS triggered and full reset isn't supported
6299 */
6300 need_emergency_restart = amdgpu_ras_need_emergency_restart(adev);
6301
6302 /*
6303 * Flush RAM to disk so that after reboot
6304 * the user can read log and see why the system rebooted.
6305 */
6306 if (need_emergency_restart && amdgpu_ras_get_context(adev) &&
6307 amdgpu_ras_get_context(adev)->reboot) {
6308 DRM_WARN("Emergency reboot.");
6309
6310 ksys_sync_helper();
6311 emergency_restart();
6312 }
6313
6314 dev_info(adev->dev, "GPU %s begin!\n",
6315 need_emergency_restart ? "jobs stop":"reset");
6316
6317 if (!amdgpu_sriov_vf(adev))
6318 hive = amdgpu_get_xgmi_hive(adev);
6319 if (hive)
6320 mutex_lock(&hive->hive_lock);
6321
6322 reset_context->job = job;
6323 reset_context->hive = hive;
6324 INIT_LIST_HEAD(&device_list);
6325
6326 r = amdgpu_device_halt_activities(adev, job, reset_context, &device_list,
6327 hive, need_emergency_restart);
6328 if (r)
6329 goto end_reset;
6330
6331 if (need_emergency_restart)
6332 goto skip_sched_resume;
6333 /*
6334 * Must check guilty signal here since after this point all old
6335 * HW fences are force signaled.
6336 *
6337 * job->base holds a reference to parent fence
6338 */
6339 if (job && dma_fence_is_signaled(&job->hw_fence)) {
6340 job_signaled = true;
6341 dev_info(adev->dev, "Guilty job already signaled, skipping HW reset");
6342 goto skip_hw_reset;
6343 }
6344
6345 r = amdgpu_device_asic_reset(adev, &device_list, reset_context);
6346 if (r)
6347 goto end_reset;
6348skip_hw_reset:
6349 r = amdgpu_device_sched_resume(&device_list, reset_context, job_signaled);
6350 if (r)
6351 goto end_reset;
6352skip_sched_resume:
6353 amdgpu_device_gpu_resume(adev, &device_list, need_emergency_restart);
6207end_reset:6354end_reset:
6208 if (hive) {6355 if (hive) {
6209 mutex_unlock(&hive->hive_lock);6356 mutex_unlock(&hive->hive_lock);
@@ -6587,12 +6734,15 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 377 unmodified lines
6587{6734{
6588 struct drm_device *dev = pci_get_drvdata(pdev);6735 struct drm_device *dev = pci_get_drvdata(pdev);
6589 struct amdgpu_device *adev = drm_to_adev(dev);6736 struct amdgpu_device *adev = drm_to_adev(dev);
6590 int i;6737 struct amdgpu_hive_info *hive = amdgpu_get_xgmi_hive(adev);
6738 struct amdgpu_reset_context reset_context;
6739 struct list_head device_list;
6740 int r = 0;
6591 6741
6592 DRM_INFO("PCI error: detected callback, state(%d)!!\n", state);6742 dev_info(adev->dev, "PCI error: detected callback!!\n");
6593 6743
6594 if (adev->gmc.xgmi.num_physical_nodes > 1) {6744 if (!amdgpu_dpm_is_link_reset_supported(adev)) {
6595 DRM_WARN("No support for XGMI hive yet...");6745 dev_warn(adev->dev, "No support for XGMI hive yet...\n");
6596 return PCI_ERS_RESULT_DISCONNECT;6746 return PCI_ERS_RESULT_DISCONNECT;
6597 }6747 }
6598 6748
@@ -6600,32 +6750,30 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 1 unmodified line
6600 6750
6601 switch (state) {6751 switch (state) {
6602 case pci_channel_io_normal:6752 case pci_channel_io_normal:
6753 dev_info(adev->dev, "pci_channel_io_normal: state(%d)!!\n", state);
6603 return PCI_ERS_RESULT_CAN_RECOVER;6754 return PCI_ERS_RESULT_CAN_RECOVER;
6604 /* Fatal error, prepare for slot reset */
6605 case pci_channel_io_frozen:6755 case pci_channel_io_frozen:
6606 /*6756 /* Fatal error, prepare for slot reset */
6607 * Locking adev->reset_domain->sem will prevent any external access6757 dev_info(adev->dev, "pci_channel_io_frozen: state(%d)!!\n", state);
6608 * to GPU during PCI error recovery6758
6609 */6759 if (hive)
6610 amdgpu_device_lock_reset_domain(adev->reset_domain);6760 mutex_lock(&hive->hive_lock);
6611 amdgpu_device_set_mp1_state(adev);6761 adev->pcie_reset_ctx.occurs_dpc = true;
6612 6762 memset(&reset_context, 0, sizeof(reset_context));
6613 /*6763 INIT_LIST_HEAD(&device_list);
6614 * Block any work scheduling as we do for regular GPU reset6764
6615 * for the duration of the recovery6765 r = amdgpu_device_halt_activities(adev, NULL, &reset_context, &device_list,
6616 */6766 hive, false);
6617 for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {6767 if (hive) {
6618 struct amdgpu_ring *ring = adev->rings[i];6768 mutex_unlock(&hive->hive_lock);
6619 6769 amdgpu_put_xgmi_hive(hive);
6620 if (!amdgpu_ring_sched_ready(ring))
6621 continue;
6622
6623 drm_sched_stop(&ring->sched, NULL);
6624 }6770 }
6625 atomic_inc(&adev->gpu_reset_counter);6771 if (r)
6772 return PCI_ERS_RESULT_DISCONNECT;
6626 return PCI_ERS_RESULT_NEED_RESET;6773 return PCI_ERS_RESULT_NEED_RESET;
6627 case pci_channel_io_perm_failure:6774 case pci_channel_io_perm_failure:
6628 /* Permanent error, prepare for device removal */6775 /* Permanent error, prepare for device removal */
6776 dev_info(adev->dev, "pci_channel_io_perm_failure: state(%d)!!\n", state);
6629 return PCI_ERS_RESULT_DISCONNECT;6777 return PCI_ERS_RESULT_DISCONNECT;
6630 }6778 }
6631 6779
@@ -6638,8 +6786,10 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 6 unmodified lines
6638 */6786 */
6639pci_ers_result_t amdgpu_pci_mmio_enabled(struct pci_dev *pdev)6787pci_ers_result_t amdgpu_pci_mmio_enabled(struct pci_dev *pdev)
6640{6788{
6789 struct drm_device *dev = pci_get_drvdata(pdev);
6790 struct amdgpu_device *adev = drm_to_adev(dev);
6641 6791
6642 DRM_INFO("PCI error: mmio enabled callback!!\n");6792 dev_info(adev->dev, "PCI error: mmio enabled callback!!\n");
6643 6793
6644 /* TODO - dump whatever for debugging purposes */6794 /* TODO - dump whatever for debugging purposes */
6645 6795
@@ -6663,10 +6813,12 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 17 unmodified lines
6663{6813{
6664 struct drm_device *dev = pci_get_drvdata(pdev);6814 struct drm_device *dev = pci_get_drvdata(pdev);
6665 struct amdgpu_device *adev = drm_to_adev(dev);6815 struct amdgpu_device *adev = drm_to_adev(dev);
6666 int r, i;
6667 struct amdgpu_reset_context reset_context;6816 struct amdgpu_reset_context reset_context;
6668 u32 memsize;6817 struct amdgpu_device *tmp_adev;
6818 struct amdgpu_hive_info *hive;
6669 struct list_head device_list;6819 struct list_head device_list;
6820 int r = 0, i;
6821 u32 memsize;
6670 6822
6671 /* PCI error slot reset should be skipped During RAS recovery */6823 /* PCI error slot reset should be skipped During RAS recovery */
6672 if ((amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) ||6824 if ((amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) ||
@@ -6674,15 +6826,12 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 1 unmodified line
6674 amdgpu_ras_in_recovery(adev))6826 amdgpu_ras_in_recovery(adev))
6675 return PCI_ERS_RESULT_RECOVERED;6827 return PCI_ERS_RESULT_RECOVERED;
6676 6828
6677 DRM_INFO("PCI error: slot reset callback!!\n");6829 dev_info(adev->dev, "PCI error: slot reset callback!!\n");
6678 6830
6679 memset(&reset_context, 0, sizeof(reset_context));6831 memset(&reset_context, 0, sizeof(reset_context));
6680 6832
6681 INIT_LIST_HEAD(&device_list);
6682 list_add_tail(&adev->reset_list, &device_list);
6683
6684 /* wait for asic to come out of reset */6833 /* wait for asic to come out of reset */
6685 msleep(500);6834 msleep(700);
6686 6835
6687 /* Restore PCI confspace */6836 /* Restore PCI confspace */
6688 amdgpu_device_load_pci_state(pdev);6837 amdgpu_device_load_pci_state(pdev);
@@ -6703,26 +6852,40 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 14 unmodified lines
6703 reset_context.method = AMD_RESET_METHOD_NONE;6852 reset_context.method = AMD_RESET_METHOD_NONE;
6704 reset_context.reset_req_dev = adev;6853 reset_context.reset_req_dev = adev;
6705 set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);6854 set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);
6706 set_bit(AMDGPU_SKIP_HW_RESET, &reset_context.flags);6855 set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags);
6707 6856 INIT_LIST_HEAD(&device_list);
6708 adev->no_hw_access = true;
6709 r = amdgpu_device_pre_asic_reset(adev, &reset_context);
6710 adev->no_hw_access = false;
6711 if (r)
6712 goto out;
6713 6857
6714 r = amdgpu_do_asic_reset(&device_list, &reset_context);6858 hive = amdgpu_get_xgmi_hive(adev);
6859 if (hive) {
6860 mutex_lock(&hive->hive_lock);
6861 reset_context.hive = hive;
6862 list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
6863 tmp_adev->pcie_reset_ctx.in_link_reset = true;
6864 list_add_tail(&tmp_adev->reset_list, &device_list);
6865 }
6866 } else {
6867 set_bit(AMDGPU_SKIP_HW_RESET, &reset_context.flags);
6868 list_add_tail(&adev->reset_list, &device_list);
6869 }
6715 6870
6871 r = amdgpu_device_asic_reset(adev, &device_list, &reset_context);
6716out:6872out:
6717 if (!r) {6873 if (!r) {
6718 if (amdgpu_device_cache_pci_state(adev->pdev))6874 if (amdgpu_device_cache_pci_state(adev->pdev))
6719 pci_restore_state(adev->pdev);6875 pci_restore_state(adev->pdev);
6720 6876 dev_info(adev->dev, "PCIe error recovery succeeded\n");
6721 DRM_INFO("PCIe error recovery succeeded\n");
6722 } else {6877 } else {
6723 DRM_ERROR("PCIe error recovery failed, err:%d", r);6878 dev_err(adev->dev, "PCIe error recovery failed, err:%d\n", r);
6724 amdgpu_device_unset_mp1_state(adev);6879 if (hive) {
6725 amdgpu_device_unlock_reset_domain(adev->reset_domain);6880 list_for_each_entry(tmp_adev, &device_list, reset_list)
6881 amdgpu_device_unset_mp1_state(tmp_adev);
6882 amdgpu_device_unlock_reset_domain(adev->reset_domain);
6883 }
6884 }
6885
6886 if (hive) {
6887 mutex_unlock(&hive->hive_lock);
6888 amdgpu_put_xgmi_hive(hive);
6726 }6889 }
6727 6890
6728 return r ? PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_RECOVERED;6891 return r ? PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_RECOVERED;
@@ -6739,26 +6902,36 @@ void amdgpu_pci_resume(struct pci_dev *pdev) · 10 unmodified lines
6739{6902{
6740 struct drm_device *dev = pci_get_drvdata(pdev);6903 struct drm_device *dev = pci_get_drvdata(pdev);
6741 struct amdgpu_device *adev = drm_to_adev(dev);6904 struct amdgpu_device *adev = drm_to_adev(dev);
6742 int i;6905 struct list_head device_list;
6743 6906 struct amdgpu_hive_info *hive = NULL;
6907 struct amdgpu_device *tmp_adev = NULL;
6744 6908
6745 DRM_INFO("PCI error: resume callback!!\n");6909 dev_info(adev->dev, "PCI error: resume callback!!\n");
6746 6910
6747 /* Only continue execution for the case of pci_channel_io_frozen */6911 /* Only continue execution for the case of pci_channel_io_frozen */
6748 if (adev->pci_channel_state != pci_channel_io_frozen)6912 if (adev->pci_channel_state != pci_channel_io_frozen)
6749 return;6913 return;
6750 6914
6751 for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {6915 INIT_LIST_HEAD(&device_list);
6752 struct amdgpu_ring *ring = adev->rings[i];
6753 6916
6754 if (!amdgpu_ring_sched_ready(ring))6917 hive = amdgpu_get_xgmi_hive(adev);
6755 continue;6918 if (hive) {
6919 mutex_lock(&hive->hive_lock);
6920 list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
6921 tmp_adev->pcie_reset_ctx.in_link_reset = false;
6922 list_add_tail(&tmp_adev->reset_list, &device_list);
6923 }
6924 } else
6925 list_add_tail(&adev->reset_list, &device_list);
6756 6926
6757 drm_sched_start(&ring->sched, 0);6927 amdgpu_device_sched_resume(&device_list, NULL, NULL);
6758 }6928 amdgpu_device_gpu_resume(adev, &device_list, false);
6929 adev->pcie_reset_ctx.occurs_dpc = false;
6759 6930
6760 amdgpu_device_unset_mp1_state(adev);6931 if (hive) {
6761 amdgpu_device_unlock_reset_domain(adev->reset_domain);6932 mutex_unlock(&hive->hive_lock);
6933 amdgpu_put_xgmi_hive(hive);
6934 }
6762}6935}
6763 6936
6764bool amdgpu_device_cache_pci_state(struct pci_dev *pdev)6937bool amdgpu_device_cache_pci_state(struct pci_dev *pdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c+94−16
@@ -51,6 +51,8 @@ · 50 unmodified lines
51#include "amdgpu_reset.h"51#include "amdgpu_reset.h"
52#include "amdgpu_sched.h"52#include "amdgpu_sched.h"
53#include "amdgpu_xgmi.h"53#include "amdgpu_xgmi.h"
54#include "amdgpu_userq.h"
55#include "amdgpu_userq_fence.h"
54#include "../amdxcp/amdgpu_xcp_drv.h"56#include "../amdxcp/amdgpu_xcp_drv.h"
55 57
56/*58/*
@@ -123,9 +125,10 @@ · 66 unmodified lines
123 * - 3.61.0 - Contains fix for RV/PCO compute queues125 * - 3.61.0 - Contains fix for RV/PCO compute queues
124 * - 3.62.0 - Add AMDGPU_IDS_FLAGS_MODE_PF, AMDGPU_IDS_FLAGS_MODE_VF & AMDGPU_IDS_FLAGS_MODE_PT126 * - 3.62.0 - Add AMDGPU_IDS_FLAGS_MODE_PF, AMDGPU_IDS_FLAGS_MODE_VF & AMDGPU_IDS_FLAGS_MODE_PT
125 * - 3.63.0 - GFX12 display DCC supports 256B max compressed block size127 * - 3.63.0 - GFX12 display DCC supports 256B max compressed block size
128 * - 3.64.0 - Userq IP support query
126 */129 */
127#define KMS_DRIVER_MAJOR 3130#define KMS_DRIVER_MAJOR 3
128#define KMS_DRIVER_MINOR 63131#define KMS_DRIVER_MINOR 64
129#define KMS_DRIVER_PATCHLEVEL 0132#define KMS_DRIVER_PATCHLEVEL 0
130 133
131/*134/*
@@ -140,6 +143,7 @@ enum AMDGPU_DEBUG_MASK { · 8 unmodified lines
140 AMDGPU_DEBUG_ENABLE_EXP_RESETS = BIT(5),143 AMDGPU_DEBUG_ENABLE_EXP_RESETS = BIT(5),
141 AMDGPU_DEBUG_DISABLE_GPU_RING_RESET = BIT(6),144 AMDGPU_DEBUG_DISABLE_GPU_RING_RESET = BIT(6),
142 AMDGPU_DEBUG_SMU_POOL = BIT(7),145 AMDGPU_DEBUG_SMU_POOL = BIT(7),
146 AMDGPU_DEBUG_VM_USERPTR = BIT(8),
143};147};
144 148
145unsigned int amdgpu_vram_limit = UINT_MAX;149unsigned int amdgpu_vram_limit = UINT_MAX;
@@ -176,7 +180,7 @@ uint amdgpu_pg_mask = 0xffffffff; · 30 unmodified lines
176uint amdgpu_sdma_phase_quantum = 32;180uint amdgpu_sdma_phase_quantum = 32;
177char *amdgpu_disable_cu;181char *amdgpu_disable_cu;
178char *amdgpu_virtual_display;182char *amdgpu_virtual_display;
179bool enforce_isolation;183int amdgpu_enforce_isolation = -1;
180int amdgpu_modeset = -1;184int amdgpu_modeset = -1;
181 185
182/* Specifies the default granularity for SVM, used in buffer186/* Specifies the default granularity for SVM, used in buffer
@@ -238,6 +242,8 @@ int amdgpu_agp = -1; /* auto */ · 55 unmodified lines
238int amdgpu_wbrf = -1;242int amdgpu_wbrf = -1;
239int amdgpu_damage_clips = -1; /* auto */243int amdgpu_damage_clips = -1; /* auto */
240int amdgpu_umsch_mm_fwlog;244int amdgpu_umsch_mm_fwlog;
245int amdgpu_rebar = -1; /* auto */
246int amdgpu_user_queue = -1;
241 247
242DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0,248DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0,
243 "DRM_UT_CORE",249 "DRM_UT_CORE",
@@ -1033,11 +1039,13 @@ module_param_named(user_partt_mode, amdgpu_user_partt_mode, uint, 0444); · 789 unmodified lines
1033 1039
1034 1040
1035/**1041/**
1036 * DOC: enforce_isolation (bool)1042 * DOC: enforce_isolation (int)
1037 * enforce process isolation between graphics and compute via using the same reserved vmid.1043 * enforce process isolation between graphics and compute.
1044 * (-1 = auto, 0 = disable, 1 = enable, 2 = enable legacy mode, 3 = enable without cleaner shader)
1038 */1045 */
1039module_param(enforce_isolation, bool, 0444);1046module_param_named(enforce_isolation, amdgpu_enforce_isolation, int, 0444);
1040MODULE_PARM_DESC(enforce_isolation, "enforce process isolation between graphics and compute . enforce_isolation = on");1047MODULE_PARM_DESC(enforce_isolation,
1048"enforce process isolation between graphics and compute. (-1 = auto, 0 = disable, 1 = enable, 2 = enable legacy mode, 3 = enable without cleaner shader)");
1041 1049
1042/**1050/**
1043 * DOC: modeset (int)1051 * DOC: modeset (int)
@@ -1096,6 +1104,28 @@ MODULE_PARM_DESC(wbrf, · 52 unmodified lines
1096 "Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)");1104 "Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)");
1097module_param_named(wbrf, amdgpu_wbrf, int, 0444);1105module_param_named(wbrf, amdgpu_wbrf, int, 0444);
1098 1106
1107/**
1108 * DOC: rebar (int)
1109 * Allow BAR resizing. Disable this to prevent the driver from attempting
1110 * to resize the BAR if the GPU supports it and there is available MMIO space.
1111 * Note that this just prevents the driver from resizing the BAR. The BIOS
1112 * may have already resized the BAR at boot time.
1113 */
1114MODULE_PARM_DESC(rebar, "Resizable BAR (-1 = auto (default), 0 = disable, 1 = enable)");
1115module_param_named(rebar, amdgpu_rebar, int, 0444);
1116
1117/**
1118 * DOC: user_queue (int)
1119 * Enable user queues on systems that support user queues. Possible values:
1120 *
1121 * - -1 = auto (ASIC specific default)
1122 * - 0 = user queues disabled
1123 * - 1 = user queues enabled and kernel queues enabled (if supported)
1124 * - 2 = user queues enabled and kernel queues disabled
1125 */
1126MODULE_PARM_DESC(user_queue, "Enable user queues (-1 = auto (default), 0 = disable, 1 = enable, 2 = enable UQs and disable KQs)");
1127module_param_named(user_queue, amdgpu_user_queue, int, 0444);
1128
1099/* These devices are not supported by amdgpu.1129/* These devices are not supported by amdgpu.
1100 * They are supported by the mach64, r128, radeon drivers1130 * They are supported by the mach64, r128, radeon drivers
1101 */1131 */
@@ -2244,6 +2274,10 @@ static void amdgpu_init_debug_options(struct amdgpu_device *adev) · 1142 unmodified lines
2244 pr_info("debug: use vram for smu pool\n");2274 pr_info("debug: use vram for smu pool\n");
2245 adev->pm.smu_debug_mask |= SMU_DEBUG_POOL_USE_VRAM;2275 adev->pm.smu_debug_mask |= SMU_DEBUG_POOL_USE_VRAM;
2246 }2276 }
2277 if (amdgpu_debug_mask & AMDGPU_DEBUG_VM_USERPTR) {
2278 pr_info("debug: VM mode debug for userptr is enabled\n");
2279 adev->debug_vm_userptr = true;
2280 }
2247}2281}
2248 2282
2249static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags)2283static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags)
@@ -2615,13 +2649,8 @@ static int amdgpu_pmops_freeze(struct device *dev) · 365 unmodified lines
2615static int amdgpu_pmops_thaw(struct device *dev)2649static int amdgpu_pmops_thaw(struct device *dev)
2616{2650{
2617 struct drm_device *drm_dev = dev_get_drvdata(dev);2651 struct drm_device *drm_dev = dev_get_drvdata(dev);
2618 struct amdgpu_device *adev = drm_to_adev(drm_dev);
2619 int r;
2620 2652
2621 r = amdgpu_device_resume(drm_dev, true);2653 return amdgpu_device_resume(drm_dev, true);
2622 adev->in_s4 = false;
2623
2624 return r;
2625}2654}
2626 2655
2627static int amdgpu_pmops_poweroff(struct device *dev)2656static int amdgpu_pmops_poweroff(struct device *dev)
@@ -2634,9 +2663,6 @@ static int amdgpu_pmops_poweroff(struct device *dev) · 6 unmodified lines
2634static int amdgpu_pmops_restore(struct device *dev)2663static int amdgpu_pmops_restore(struct device *dev)
2635{2664{
2636 struct drm_device *drm_dev = dev_get_drvdata(dev);2665 struct drm_device *drm_dev = dev_get_drvdata(dev);
2637 struct amdgpu_device *adev = drm_to_adev(drm_dev);
2638
2639 adev->in_s4 = false;
2640 2666
2641 return amdgpu_device_resume(drm_dev, true);2667 return amdgpu_device_resume(drm_dev, true);
2642}2668}
@@ -2708,6 +2734,29 @@ static int amdgpu_runtime_idle_check_display(struct device *dev) · 65 unmodified lines
2708 return 0;2734 return 0;
2709}2735}
2710 2736
2737static int amdgpu_runtime_idle_check_userq(struct device *dev)
2738{
2739 struct pci_dev *pdev = to_pci_dev(dev);
2740 struct drm_device *drm_dev = pci_get_drvdata(pdev);
2741 struct amdgpu_device *adev = drm_to_adev(drm_dev);
2742 struct amdgpu_usermode_queue *queue;
2743 struct amdgpu_userq_mgr *uqm, *tmp;
2744 int queue_id;
2745 int ret = 0;
2746
2747 mutex_lock(&adev->userq_mutex);
2748 list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
2749 idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
2750 ret = -EBUSY;
2751 goto done;
2752 }
2753 }
2754done:
2755 mutex_unlock(&adev->userq_mutex);
2756
2757 return ret;
2758}
2759
2711static int amdgpu_pmops_runtime_suspend(struct device *dev)2760static int amdgpu_pmops_runtime_suspend(struct device *dev)
2712{2761{
2713 struct pci_dev *pdev = to_pci_dev(dev);2762 struct pci_dev *pdev = to_pci_dev(dev);
@@ -2721,6 +2770,9 @@ static int amdgpu_pmops_runtime_suspend(struct device *dev) · 7 unmodified lines
2721 }2770 }
2722 2771
2723 ret = amdgpu_runtime_idle_check_display(dev);2772 ret = amdgpu_runtime_idle_check_display(dev);
2773 if (ret)
2774 return ret;
2775 ret = amdgpu_runtime_idle_check_userq(dev);
2724 if (ret)2776 if (ret)
2725 return ret;2777 return ret;
2726 2778
@@ -2844,12 +2896,30 @@ static int amdgpu_pmops_runtime_idle(struct device *dev) · 117 unmodified lines
2844 }2896 }
2845 2897
2846 ret = amdgpu_runtime_idle_check_display(dev);2898 ret = amdgpu_runtime_idle_check_display(dev);
2899 if (ret)
2900 goto done;
2847 2901
2902 ret = amdgpu_runtime_idle_check_userq(dev);
2903done:
2848 pm_runtime_mark_last_busy(dev);2904 pm_runtime_mark_last_busy(dev);
2849 pm_runtime_autosuspend(dev);2905 pm_runtime_autosuspend(dev);
2850 return ret;2906 return ret;
2851}2907}
2852 2908
2909static int amdgpu_drm_release(struct inode *inode, struct file *filp)
2910{
2911 struct drm_file *file_priv = filp->private_data;
2912 struct amdgpu_fpriv *fpriv = file_priv->driver_priv;
2913
2914 if (fpriv) {
2915 fpriv->evf_mgr.fd_closing = true;
2916 amdgpu_userq_mgr_fini(&fpriv->userq_mgr);
2917 amdgpu_eviction_fence_destroy(&fpriv->evf_mgr);
2918 }
2919
2920 return drm_release(inode, filp);
2921}
2922
2853long amdgpu_drm_ioctl(struct file *filp,2923long amdgpu_drm_ioctl(struct file *filp,
2854 unsigned int cmd, unsigned long arg)2924 unsigned int cmd, unsigned long arg)
2855{2925{
@@ -2901,7 +2971,7 @@ static const struct file_operations amdgpu_driver_kms_fops = { · 45 unmodified lines
2901 .owner = THIS_MODULE,2971 .owner = THIS_MODULE,
2902 .open = drm_open,2972 .open = drm_open,
2903 .flush = amdgpu_flush,2973 .flush = amdgpu_flush,
2904 .release = drm_release,2974 .release = amdgpu_drm_release,
2905 .unlocked_ioctl = amdgpu_drm_ioctl,2975 .unlocked_ioctl = amdgpu_drm_ioctl,
2906 .mmap = drm_gem_mmap,2976 .mmap = drm_gem_mmap,
2907 .poll = drm_poll,2977 .poll = drm_poll,
@@ -2948,6 +3018,9 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = { · 40 unmodified lines
2948 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),3018 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
2949 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),3019 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
2950 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),3020 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3021 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ, amdgpu_userq_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3022 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_SIGNAL, amdgpu_userq_signal_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3023 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
2951};3024};
2952 3025
2953static const struct drm_driver amdgpu_kms_driver = {3026static const struct drm_driver amdgpu_kms_driver = {
@@ -3038,6 +3111,10 @@ static int __init amdgpu_init(void) · 84 unmodified lines
3038 if (r)3111 if (r)
3039 goto error_fence;3112 goto error_fence;
3040 3113
3114 r = amdgpu_userq_fence_slab_init();
3115 if (r)
3116 goto error_fence;
3117
3041 DRM_INFO("amdgpu kernel modesetting enabled.\n");3118 DRM_INFO("amdgpu kernel modesetting enabled.\n");
3042 amdgpu_register_atpx_handler();3119 amdgpu_register_atpx_handler();
3043 amdgpu_acpi_detect();3120 amdgpu_acpi_detect();
@@ -3069,6 +3146,7 @@ static void __exit amdgpu_exit(void) · 25 unmodified lines
3069 amdgpu_acpi_release();3146 amdgpu_acpi_release();
3070 amdgpu_sync_fini();3147 amdgpu_sync_fini();
3071 amdgpu_fence_slab_fini();3148 amdgpu_fence_slab_fini();
3149 amdgpu_userq_fence_slab_fini();
3072 mmu_notifier_synchronize();3150 mmu_notifier_synchronize();
3073 amdgpu_xcp_drv_release();3151 amdgpu_xcp_drv_release();
3074}3152}
drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.c+233−0
@@ -0,0 +1,233 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright 2024 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24#include <linux/sched.h>
25#include <drm/drm_exec.h>
26#include "amdgpu.h"
27
28#define work_to_evf_mgr(w, name) container_of(w, struct amdgpu_eviction_fence_mgr, name)
29#define evf_mgr_to_fpriv(e) container_of(e, struct amdgpu_fpriv, evf_mgr)
30
31static const char *
32amdgpu_eviction_fence_get_driver_name(struct dma_fence *fence)
33{
34 return "amdgpu_eviction_fence";
35}
36
37static const char *
38amdgpu_eviction_fence_get_timeline_name(struct dma_fence *f)
39{
40 struct amdgpu_eviction_fence *ef;
41
42 ef = container_of(f, struct amdgpu_eviction_fence, base);
43 return ef->timeline_name;
44}
45
46int
47amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr,
48 struct drm_exec *exec)
49{
50 struct amdgpu_eviction_fence *old_ef, *new_ef;
51 struct drm_gem_object *obj;
52 unsigned long index;
53 int ret;
54
55 if (evf_mgr->ev_fence &&
56 !dma_fence_is_signaled(&evf_mgr->ev_fence->base))
57 return 0;
58 /*
59 * Steps to replace eviction fence:
60 * * lock all objects in exec (caller)
61 * * create a new eviction fence
62 * * update new eviction fence in evf_mgr
63 * * attach the new eviction fence to BOs
64 * * release the old fence
65 * * unlock the objects (caller)
66 */
67 new_ef = amdgpu_eviction_fence_create(evf_mgr);
68 if (!new_ef) {
69 DRM_ERROR("Failed to create new eviction fence\n");
70 return -ENOMEM;
71 }
72
73 /* Update the eviction fence now */
74 spin_lock(&evf_mgr->ev_fence_lock);
75 old_ef = evf_mgr->ev_fence;
76 evf_mgr->ev_fence = new_ef;
77 spin_unlock(&evf_mgr->ev_fence_lock);
78
79 /* Attach the new fence */
80 drm_exec_for_each_locked_object(exec, index, obj) {
81 struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
82
83 if (!bo)
84 continue;
85 ret = amdgpu_eviction_fence_attach(evf_mgr, bo);
86 if (ret) {
87 DRM_ERROR("Failed to attch new eviction fence\n");
88 goto free_err;
89 }
90 }
91
92 /* Free old fence */
93 if (old_ef)
94 dma_fence_put(&old_ef->base);
95 return 0;
96
97free_err:
98 kfree(new_ef);
99 return ret;
100}
101
102static void
103amdgpu_eviction_fence_suspend_worker(struct work_struct *work)
104{
105 struct amdgpu_eviction_fence_mgr *evf_mgr = work_to_evf_mgr(work, suspend_work.work);
106 struct amdgpu_fpriv *fpriv = evf_mgr_to_fpriv(evf_mgr);
107 struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
108 struct amdgpu_eviction_fence *ev_fence;
109
110 mutex_lock(&uq_mgr->userq_mutex);
111 ev_fence = evf_mgr->ev_fence;
112 if (!ev_fence)
113 goto unlock;
114
115 amdgpu_userq_evict(uq_mgr, ev_fence);
116
117unlock:
118 mutex_unlock(&uq_mgr->userq_mutex);
119}
120
121static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f)
122{
123 struct amdgpu_eviction_fence_mgr *evf_mgr;
124 struct amdgpu_eviction_fence *ev_fence;
125
126 if (!f)
127 return true;
128
129 ev_fence = to_ev_fence(f);
130 evf_mgr = ev_fence->evf_mgr;
131
132 schedule_delayed_work(&evf_mgr->suspend_work, 0);
133 return true;
134}
135
136static const struct dma_fence_ops amdgpu_eviction_fence_ops = {
137 .use_64bit_seqno = true,
138 .get_driver_name = amdgpu_eviction_fence_get_driver_name,
139 .get_timeline_name = amdgpu_eviction_fence_get_timeline_name,
140 .enable_signaling = amdgpu_eviction_fence_enable_signaling,
141};
142
143void amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr,
144 struct amdgpu_eviction_fence *ev_fence)
145{
146 spin_lock(&evf_mgr->ev_fence_lock);
147 dma_fence_signal(&ev_fence->base);
148 spin_unlock(&evf_mgr->ev_fence_lock);
149}
150
151struct amdgpu_eviction_fence *
152amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr)
153{
154 struct amdgpu_eviction_fence *ev_fence;
155
156 ev_fence = kzalloc(sizeof(*ev_fence), GFP_KERNEL);
157 if (!ev_fence)
158 return NULL;
159
160 ev_fence->evf_mgr = evf_mgr;
161 get_task_comm(ev_fence->timeline_name, current);
162 spin_lock_init(&ev_fence->lock);
163 dma_fence_init(&ev_fence->base, &amdgpu_eviction_fence_ops,
164 &ev_fence->lock, evf_mgr->ev_fence_ctx,
165 atomic_inc_return(&evf_mgr->ev_fence_seq));
166 return ev_fence;
167}
168
169void amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr)
170{
171 struct amdgpu_eviction_fence *ev_fence;
172
173 /* Wait for any pending work to execute */
174 flush_delayed_work(&evf_mgr->suspend_work);
175
176 spin_lock(&evf_mgr->ev_fence_lock);
177 ev_fence = evf_mgr->ev_fence;
178 spin_unlock(&evf_mgr->ev_fence_lock);
179
180 if (!ev_fence)
181 return;
182
183 dma_fence_wait(&ev_fence->base, false);
184
185 /* Last unref of ev_fence */
186 dma_fence_put(&evf_mgr->ev_fence->base);
187}
188
189int amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr,
190 struct amdgpu_bo *bo)
191{
192 struct amdgpu_eviction_fence *ev_fence;
193 struct dma_resv *resv = bo->tbo.base.resv;
194 int ret;
195
196 if (!resv)
197 return 0;
198
199 ret = dma_resv_reserve_fences(resv, 1);
200 if (ret) {
201 DRM_DEBUG_DRIVER("Failed to resv fence space\n");
202 return ret;
203 }
204
205 spin_lock(&evf_mgr->ev_fence_lock);
206 ev_fence = evf_mgr->ev_fence;
207 if (ev_fence)
208 dma_resv_add_fence(resv, &ev_fence->base, DMA_RESV_USAGE_BOOKKEEP);
209 spin_unlock(&evf_mgr->ev_fence_lock);
210
211 return 0;
212}
213
214void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr,
215 struct amdgpu_bo *bo)
216{
217 struct dma_fence *stub = dma_fence_get_stub();
218
219 dma_resv_replace_fences(bo->tbo.base.resv, evf_mgr->ev_fence_ctx,
220 stub, DMA_RESV_USAGE_BOOKKEEP);
221 dma_fence_put(stub);
222}
223
224int amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr)
225{
226 /* This needs to be done one time per open */
227 atomic_set(&evf_mgr->ev_fence_seq, 0);
228 evf_mgr->ev_fence_ctx = dma_fence_context_alloc(1);
229 spin_lock_init(&evf_mgr->ev_fence_lock);
230
231 INIT_DELAYED_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker);
232 return 0;
233}
drivers/gpu/drm/amd/amdgpu/amdgpu_eviction_fence.h+69−0
@@ -0,0 +1,69 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright 2023 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#ifndef AMDGPU_EV_FENCE_H_
26#define AMDGPU_EV_FENCE_H_
27
28struct amdgpu_eviction_fence {
29 struct dma_fence base;
30 spinlock_t lock;
31 char timeline_name[TASK_COMM_LEN];
32 struct amdgpu_eviction_fence_mgr *evf_mgr;
33};
34
35struct amdgpu_eviction_fence_mgr {
36 u64 ev_fence_ctx;
37 atomic_t ev_fence_seq;
38 spinlock_t ev_fence_lock;
39 struct amdgpu_eviction_fence *ev_fence;
40 struct delayed_work suspend_work;
41 uint8_t fd_closing;
42};
43
44/* Eviction fence helper functions */
45struct amdgpu_eviction_fence *
46amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr);
47
48void
49amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr);
50
51int
52amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr,
53 struct amdgpu_bo *bo);
54
55void
56amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr,
57 struct amdgpu_bo *bo);
58
59int
60amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr);
61
62void
63amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr,
64 struct amdgpu_eviction_fence *ev_fence);
65
66int
67amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr,
68 struct drm_exec *exec);
69#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c+166−13
@@ -36,6 +36,7 @@ · 35 unmodified lines
36#include <drm/drm_exec.h>36#include <drm/drm_exec.h>
37#include <drm/drm_gem_ttm_helper.h>37#include <drm/drm_gem_ttm_helper.h>
38#include <drm/ttm/ttm_tt.h>38#include <drm/ttm/ttm_tt.h>
39#include <drm/drm_syncobj.h>
39 40
40#include "amdgpu.h"41#include "amdgpu.h"
41#include "amdgpu_display.h"42#include "amdgpu_display.h"
@@ -44,6 +45,114 @@ · 2 unmodified lines
44#include "amdgpu_xgmi.h"45#include "amdgpu_xgmi.h"
45#include "amdgpu_vm.h"46#include "amdgpu_vm.h"
46 47
48static int
49amdgpu_gem_add_input_fence(struct drm_file *filp,
50 uint64_t syncobj_handles_array,
51 uint32_t num_syncobj_handles)
52{
53 struct dma_fence *fence;
54 uint32_t *syncobj_handles;
55 int ret, i;
56
57 if (!num_syncobj_handles)
58 return 0;
59
60 syncobj_handles = memdup_user(u64_to_user_ptr(syncobj_handles_array),
61 sizeof(uint32_t) * num_syncobj_handles);
62 if (IS_ERR(syncobj_handles))
63 return PTR_ERR(syncobj_handles);
64
65 for (i = 0; i < num_syncobj_handles; i++) {
66
67 if (!syncobj_handles[i]) {
68 ret = -EINVAL;
69 goto free_memdup;
70 }
71
72 ret = drm_syncobj_find_fence(filp, syncobj_handles[i], 0, 0, &fence);
73 if (ret)
74 goto free_memdup;
75
76 dma_fence_wait(fence, false);
77
78 /* TODO: optimize async handling */
79 dma_fence_put(fence);
80 }
81
82free_memdup:
83 kfree(syncobj_handles);
84 return ret;
85}
86
87static int
88amdgpu_gem_update_timeline_node(struct drm_file *filp,
89 uint32_t syncobj_handle,
90 uint64_t point,
91 struct drm_syncobj **syncobj,
92 struct dma_fence_chain **chain)
93{
94 if (!syncobj_handle)
95 return 0;
96
97 /* Find the sync object */
98 *syncobj = drm_syncobj_find(filp, syncobj_handle);
99 if (!*syncobj)
100 return -ENOENT;
101
102 if (!point)
103 return 0;
104
105 /* Allocate the chain node */
106 *chain = dma_fence_chain_alloc();
107 if (!*chain) {
108 drm_syncobj_put(*syncobj);
109 return -ENOMEM;
110 }
111
112 return 0;
113}
114
115static void
116amdgpu_gem_update_bo_mapping(struct drm_file *filp,
117 struct amdgpu_bo_va *bo_va,
118 uint32_t operation,
119 uint64_t point,
120 struct dma_fence *fence,
121 struct drm_syncobj *syncobj,
122 struct dma_fence_chain *chain)
123{
124 struct amdgpu_bo *bo = bo_va ? bo_va->base.bo : NULL;
125 struct amdgpu_fpriv *fpriv = filp->driver_priv;
126 struct amdgpu_vm *vm = &fpriv->vm;
127 struct dma_fence *last_update;
128
129 if (!syncobj)
130 return;
131
132 /* Find the last update fence */
133 switch (operation) {
134 case AMDGPU_VA_OP_MAP:
135 case AMDGPU_VA_OP_REPLACE:
136 if (bo && (bo->tbo.base.resv == vm->root.bo->tbo.base.resv))
137 last_update = vm->last_update;
138 else
139 last_update = bo_va->last_pt_update;
140 break;
141 case AMDGPU_VA_OP_UNMAP:
142 case AMDGPU_VA_OP_CLEAR:
143 last_update = fence;
144 break;
145 default:
146 return;
147 }
148
149 /* Add fence to timeline */
150 if (!point)
151 drm_syncobj_replace_fence(syncobj, last_update);
152 else
153 drm_syncobj_add_point(syncobj, chain, last_update, point);
154}
155
47static vm_fault_t amdgpu_gem_fault(struct vm_fault *vmf)156static vm_fault_t amdgpu_gem_fault(struct vm_fault *vmf)
48{157{
49 struct ttm_buffer_object *bo = vmf->vma->vm_private_data;158 struct ttm_buffer_object *bo = vmf->vma->vm_private_data;
@@ -184,6 +293,15 @@ static int amdgpu_gem_object_open(struct drm_gem_object *obj, · 134 unmodified lines
184 bo_va = amdgpu_vm_bo_add(adev, vm, abo);293 bo_va = amdgpu_vm_bo_add(adev, vm, abo);
185 else294 else
186 ++bo_va->ref_count;295 ++bo_va->ref_count;
296
297 /* attach gfx eviction fence */
298 r = amdgpu_eviction_fence_attach(&fpriv->evf_mgr, abo);
299 if (r) {
300 DRM_DEBUG_DRIVER("Failed to attach eviction fence to BO\n");
301 amdgpu_bo_unreserve(abo);
302 return r;
303 }
304
187 amdgpu_bo_unreserve(abo);305 amdgpu_bo_unreserve(abo);
188 306
189 /* Validate and add eviction fence to DMABuf imports with dynamic307 /* Validate and add eviction fence to DMABuf imports with dynamic
@@ -247,6 +365,9 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj, · 57 unmodified lines
247 goto out_unlock;365 goto out_unlock;
248 }366 }
249 367
368 if (!amdgpu_vm_is_bo_always_valid(vm, bo))
369 amdgpu_eviction_fence_detach(&fpriv->evf_mgr, bo);
370
250 bo_va = amdgpu_vm_bo_find(vm, bo);371 bo_va = amdgpu_vm_bo_find(vm, bo);
251 if (!bo_va || --bo_va->ref_count)372 if (!bo_va || --bo_va->ref_count)
252 goto out_unlock;373 goto out_unlock;
@@ -321,10 +442,6 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data, · 68 unmodified lines
321 uint32_t handle, initial_domain;442 uint32_t handle, initial_domain;
322 int r;443 int r;
323 444
324 /* reject DOORBELLs until userspace code to use it is available */
325 if (args->in.domains & AMDGPU_GEM_DOMAIN_DOORBELL)
326 return -EINVAL;
327
328 /* reject invalid gem flags */445 /* reject invalid gem flags */
329 if (flags & ~(AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED |446 if (flags & ~(AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED |
330 AMDGPU_GEM_CREATE_NO_CPU_ACCESS |447 AMDGPU_GEM_CREATE_NO_CPU_ACCESS |
@@ -638,18 +755,23 @@ out: · 307 unmodified lines
638 *755 *
639 * Update the bo_va directly after setting its address. Errors are not756 * Update the bo_va directly after setting its address. Errors are not
640 * vital here, so they are not reported back to userspace.757 * vital here, so they are not reported back to userspace.
758 *
759 * Returns resulting fence if freed BO(s) got cleared from the PT.
760 * otherwise stub fence in case of error.
641 */761 */
642static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev,762static struct dma_fence *
643 struct amdgpu_vm *vm,763amdgpu_gem_va_update_vm(struct amdgpu_device *adev,
644 struct amdgpu_bo_va *bo_va,764 struct amdgpu_vm *vm,
645 uint32_t operation)765 struct amdgpu_bo_va *bo_va,
766 uint32_t operation)
646{767{
768 struct dma_fence *fence = dma_fence_get_stub();
647 int r;769 int r;
648 770
649 if (!amdgpu_vm_ready(vm))771 if (!amdgpu_vm_ready(vm))
650 return;772 return fence;
651 773
652 r = amdgpu_vm_clear_freed(adev, vm, NULL);774 r = amdgpu_vm_clear_freed(adev, vm, &fence);
653 if (r)775 if (r)
654 goto error;776 goto error;
655 777
@@ -665,6 +787,8 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 9 unmodified lines
665error:787error:
666 if (r && r != -ERESTARTSYS)788 if (r && r != -ERESTARTSYS)
667 DRM_ERROR("Couldn't update BO_VA (%d)\n", r);789 DRM_ERROR("Couldn't update BO_VA (%d)\n", r);
790
791 return fence;
668}792}
669 793
670/**794/**
@@ -713,6 +837,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 42 unmodified lines
713 struct amdgpu_fpriv *fpriv = filp->driver_priv;837 struct amdgpu_fpriv *fpriv = filp->driver_priv;
714 struct amdgpu_bo *abo;838 struct amdgpu_bo *abo;
715 struct amdgpu_bo_va *bo_va;839 struct amdgpu_bo_va *bo_va;
840 struct drm_syncobj *timeline_syncobj = NULL;
841 struct dma_fence_chain *timeline_chain = NULL;
842 struct dma_fence *fence;
716 struct drm_exec exec;843 struct drm_exec exec;
717 uint64_t va_flags;844 uint64_t va_flags;
718 uint64_t vm_size;845 uint64_t vm_size;
@@ -774,6 +901,12 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 55 unmodified lines
774 abo = NULL;901 abo = NULL;
775 }902 }
776 903
904 r = amdgpu_gem_add_input_fence(filp,
905 args->input_fence_syncobj_handles,
906 args->num_syncobj_handles);
907 if (r)
908 goto error_put_gobj;
909
777 drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT |910 drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT |
778 DRM_EXEC_IGNORE_DUPLICATES, 0);911 DRM_EXEC_IGNORE_DUPLICATES, 0);
779 drm_exec_until_all_locked(&exec) {912 drm_exec_until_all_locked(&exec) {
@@ -802,6 +935,14 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 22 unmodified lines
802 bo_va = NULL;935 bo_va = NULL;
803 }936 }
804 937
938 r = amdgpu_gem_update_timeline_node(filp,
939 args->vm_timeline_syncobj_out,
940 args->vm_timeline_point,
941 &timeline_syncobj,
942 &timeline_chain);
943 if (r)
944 goto error;
945
805 switch (args->operation) {946 switch (args->operation) {
806 case AMDGPU_VA_OP_MAP:947 case AMDGPU_VA_OP_MAP:
807 va_flags = amdgpu_gem_va_map_flags(adev, args->flags);948 va_flags = amdgpu_gem_va_map_flags(adev, args->flags);
@@ -827,12 +968,24 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 19 unmodified lines
827 default:968 default:
828 break;969 break;
829 }970 }
830 if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm)971 if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm) {
831 amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,972 fence = amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va,
832 args->operation);973 args->operation);
974
975 if (timeline_syncobj)
976 amdgpu_gem_update_bo_mapping(filp, bo_va,
977 args->operation,
978 args->vm_timeline_point,
979 fence, timeline_syncobj,
980 timeline_chain);
981 else
982 dma_fence_put(fence);
983
984 }
833 985
834error:986error:
835 drm_exec_fini(&exec);987 drm_exec_fini(&exec);
988error_put_gobj:
836 drm_gem_object_put(gobj);989 drm_gem_object_put(gobj);
837 return r;990 return r;
838}991}
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c+150−36
@@ -33,6 +33,7 @@ · 32 unmodified lines
33#include "amdgpu_reset.h"33#include "amdgpu_reset.h"
34#include "amdgpu_xcp.h"34#include "amdgpu_xcp.h"
35#include "amdgpu_xgmi.h"35#include "amdgpu_xgmi.h"
36#include "nvd.h"
36 37
37/* delay 0.1 second to enable gfx off feature */38/* delay 0.1 second to enable gfx off feature */
38#define GFX_OFF_DELAY_ENABLE msecs_to_jiffies(100)39#define GFX_OFF_DELAY_ENABLE msecs_to_jiffies(100)
@@ -74,14 +75,15 @@ bool amdgpu_gfx_is_mec_queue_enabled(struct amdgpu_device *adev, · 35 unmodified lines
74 adev->gfx.mec_bitmap[xcc_id].queue_bitmap);75 adev->gfx.mec_bitmap[xcc_id].queue_bitmap);
75}76}
76 77
77int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev,78static int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev,
78 int me, int pipe, int queue)79 int me, int pipe, int queue)
79{80{
81 int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
80 int bit = 0;82 int bit = 0;
81 83
82 bit += me * adev->gfx.me.num_pipe_per_me84 bit += me * adev->gfx.me.num_pipe_per_me
83 * adev->gfx.me.num_queue_per_pipe;85 * num_queue_per_pipe;
84 bit += pipe * adev->gfx.me.num_queue_per_pipe;86 bit += pipe * num_queue_per_pipe;
85 bit += queue;87 bit += queue;
86 88
87 return bit;89 return bit;
@@ -238,8 +240,8 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 150 unmodified lines
238{240{
239 int i, queue, pipe;241 int i, queue, pipe;
240 bool multipipe_policy = amdgpu_gfx_is_graphics_multipipe_capable(adev);242 bool multipipe_policy = amdgpu_gfx_is_graphics_multipipe_capable(adev);
241 int max_queues_per_me = adev->gfx.me.num_pipe_per_me *243 int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
242 adev->gfx.me.num_queue_per_pipe;244 int max_queues_per_me = adev->gfx.me.num_pipe_per_me * num_queue_per_pipe;
243 245
244 if (multipipe_policy) {246 if (multipipe_policy) {
245 /* policy: amdgpu owns the first queue per pipe at this stage247 /* policy: amdgpu owns the first queue per pipe at this stage
@@ -247,9 +249,9 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 1 unmodified line
247 for (i = 0; i < max_queues_per_me; i++) {249 for (i = 0; i < max_queues_per_me; i++) {
248 pipe = i % adev->gfx.me.num_pipe_per_me;250 pipe = i % adev->gfx.me.num_pipe_per_me;
249 queue = (i / adev->gfx.me.num_pipe_per_me) %251 queue = (i / adev->gfx.me.num_pipe_per_me) %
250 adev->gfx.me.num_queue_per_pipe;252 num_queue_per_pipe;
251 253
252 set_bit(pipe * adev->gfx.me.num_queue_per_pipe + queue,254 set_bit(pipe * num_queue_per_pipe + queue,
253 adev->gfx.me.queue_bitmap);255 adev->gfx.me.queue_bitmap);
254 }256 }
255 } else {257 } else {
@@ -258,8 +260,9 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 2 unmodified lines
258 }260 }
259 261
260 /* update the number of active graphics rings */262 /* update the number of active graphics rings */
261 adev->gfx.num_gfx_rings =263 if (adev->gfx.num_gfx_rings)
262 bitmap_weight(adev->gfx.me.queue_bitmap, AMDGPU_MAX_GFX_QUEUES);264 adev->gfx.num_gfx_rings =
265 bitmap_weight(adev->gfx.me.queue_bitmap, AMDGPU_MAX_GFX_QUEUES);
263}266}
264 267
265static int amdgpu_gfx_kiq_acquire(struct amdgpu_device *adev,268static int amdgpu_gfx_kiq_acquire(struct amdgpu_device *adev,
@@ -1351,6 +1354,10 @@ static ssize_t amdgpu_gfx_get_current_compute_partition(struct device *dev, · 1085 unmodified lines
1351 struct amdgpu_device *adev = drm_to_adev(ddev);1354 struct amdgpu_device *adev = drm_to_adev(ddev);
1352 int mode;1355 int mode;
1353 1356
1357 /* Only minimal precaution taken to reject requests while in reset.*/
1358 if (amdgpu_in_reset(adev))
1359 return -EPERM;
1360
1354 mode = amdgpu_xcp_query_partition_mode(adev->xcp_mgr,1361 mode = amdgpu_xcp_query_partition_mode(adev->xcp_mgr,
1355 AMDGPU_XCP_FL_NONE);1362 AMDGPU_XCP_FL_NONE);
1356 1363
@@ -1394,8 +1401,14 @@ static ssize_t amdgpu_gfx_set_compute_partition(struct device *dev, · 37 unmodified lines
1394 return -EINVAL;1401 return -EINVAL;
1395 }1402 }
1396 1403
1404 /* Don't allow a switch while under reset */
1405 if (!down_read_trylock(&adev->reset_domain->sem))
1406 return -EPERM;
1407
1397 ret = amdgpu_xcp_switch_partition_mode(adev->xcp_mgr, mode);1408 ret = amdgpu_xcp_switch_partition_mode(adev->xcp_mgr, mode);
1398 1409
1410 up_read(&adev->reset_domain->sem);
1411
1399 if (ret)1412 if (ret)
1400 return ret;1413 return ret;
1401 1414
@@ -1466,6 +1479,8 @@ static int amdgpu_gfx_run_cleaner_shader_job(struct amdgpu_ring *ring) · 64 unmodified lines
1466 goto err;1479 goto err;
1467 1480
1468 job->enforce_isolation = true;1481 job->enforce_isolation = true;
1482 /* always run the cleaner shader */
1483 job->run_cleaner_shader = true;
1469 1484
1470 ib = &job->ibs[0];1485 ib = &job->ibs[0];
1471 for (i = 0; i <= ring->funcs->align_mask; ++i)1486 for (i = 0; i <= ring->funcs->align_mask; ++i)
@@ -1552,6 +1567,9 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 80 unmodified lines
1552 if (adev->in_suspend && !adev->in_runpm)1567 if (adev->in_suspend && !adev->in_runpm)
1553 return -EPERM;1568 return -EPERM;
1554 1569
1570 if (adev->gfx.disable_kq)
1571 return -EPERM;
1572
1555 ret = kstrtol(buf, 0, &value);1573 ret = kstrtol(buf, 0, &value);
1556 1574
1557 if (ret)1575 if (ret)
@@ -1594,7 +1612,8 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 36 unmodified lines
1594 * Provides the sysfs read interface to get the current settings of the 'enforce_isolation'1612 * Provides the sysfs read interface to get the current settings of the 'enforce_isolation'
1595 * feature for each GPU partition. Reading from the 'enforce_isolation'1613 * feature for each GPU partition. Reading from the 'enforce_isolation'
1596 * sysfs file returns the isolation settings for all partitions, where '0'1614 * sysfs file returns the isolation settings for all partitions, where '0'
1597 * indicates disabled and '1' indicates enabled.1615 * indicates disabled, '1' indicates enabled, and '2' indicates enabled in legacy mode,
1616 * and '3' indicates enabled without cleaner shader.
1598 *1617 *
1599 * Return: The number of bytes read from the sysfs file.1618 * Return: The number of bytes read from the sysfs file.
1600 */1619 */
@@ -1629,9 +1648,12 @@ static ssize_t amdgpu_gfx_get_enforce_isolation(struct device *dev, · 28 unmodified lines
1629 * @count: The size of the input data1648 * @count: The size of the input data
1630 *1649 *
1631 * This function allows control over the 'enforce_isolation' feature, which1650 * This function allows control over the 'enforce_isolation' feature, which
1632 * serializes access to the graphics engine. Writing '1' or '0' to the1651 * serializes access to the graphics engine. Writing '0' to disable, '1' to
1633 * 'enforce_isolation' sysfs file enables or disables process isolation for1652 * enable isolation with cleaner shader, '2' to enable legacy isolation without
1634 * each partition. The input should specify the setting for all partitions.1653 * cleaner shader, or '3' to enable process isolation without submitting the
1654 * cleaner shader to the 'enforce_isolation' sysfs file sets the isolation mode
1655 * for each partition. The input should specify the setting for all
1656 * partitions.
1635 *1657 *
1636 * Return: The number of bytes written to the sysfs file.1658 * Return: The number of bytes written to the sysfs file.
1637 */1659 */
@@ -1668,13 +1690,34 @@ static ssize_t amdgpu_gfx_set_enforce_isolation(struct device *dev, · 30 unmodified lines
1668 return -EINVAL;1690 return -EINVAL;
1669 1691
1670 for (i = 0; i < num_partitions; i++) {1692 for (i = 0; i < num_partitions; i++) {
1671 if (partition_values[i] != 0 && partition_values[i] != 1)1693 if (partition_values[i] != 0 &&
1694 partition_values[i] != 1 &&
1695 partition_values[i] != 2 &&
1696 partition_values[i] != 3)
1672 return -EINVAL;1697 return -EINVAL;
1673 }1698 }
1674 1699
1675 mutex_lock(&adev->enforce_isolation_mutex);1700 mutex_lock(&adev->enforce_isolation_mutex);
1676 for (i = 0; i < num_partitions; i++)1701 for (i = 0; i < num_partitions; i++) {
1677 adev->enforce_isolation[i] = partition_values[i];1702 switch (partition_values[i]) {
1703 case 0:
1704 default:
1705 adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE;
1706 break;
1707 case 1:
1708 adev->enforce_isolation[i] =
1709 AMDGPU_ENFORCE_ISOLATION_ENABLE;
1710 break;
1711 case 2:
1712 adev->enforce_isolation[i] =
1713 AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY;
1714 break;
1715 case 3:
1716 adev->enforce_isolation[i] =
1717 AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER;
1718 break;
1719 }
1720 }
1678 mutex_unlock(&adev->enforce_isolation_mutex);1721 mutex_unlock(&adev->enforce_isolation_mutex);
1679 1722
1680 amdgpu_mes_update_enforce_isolation(adev);1723 amdgpu_mes_update_enforce_isolation(adev);
@@ -1923,39 +1966,41 @@ void amdgpu_gfx_cleaner_shader_init(struct amdgpu_device *adev, · 242 unmodified lines
1923static void amdgpu_gfx_kfd_sch_ctrl(struct amdgpu_device *adev, u32 idx,1966static void amdgpu_gfx_kfd_sch_ctrl(struct amdgpu_device *adev, u32 idx,
1924 bool enable)1967 bool enable)
1925{1968{
1926 mutex_lock(&adev->gfx.kfd_sch_mutex);1969 mutex_lock(&adev->gfx.userq_sch_mutex);
1927 1970
1928 if (enable) {1971 if (enable) {
1929 /* If the count is already 0, it means there's an imbalance bug somewhere.1972 /* If the count is already 0, it means there's an imbalance bug somewhere.
1930 * Note that the bug may be in a different caller than the one which triggers the1973 * Note that the bug may be in a different caller than the one which triggers the
1931 * WARN_ON_ONCE.1974 * WARN_ON_ONCE.
1932 */1975 */
1933 if (WARN_ON_ONCE(adev->gfx.kfd_sch_req_count[idx] == 0)) {1976 if (WARN_ON_ONCE(adev->gfx.userq_sch_req_count[idx] == 0)) {
1934 dev_err(adev->dev, "Attempted to enable KFD scheduler when reference count is already zero\n");1977 dev_err(adev->dev, "Attempted to enable KFD scheduler when reference count is already zero\n");
1935 goto unlock;1978 goto unlock;
1936 }1979 }
1937 1980
1938 adev->gfx.kfd_sch_req_count[idx]--;1981 adev->gfx.userq_sch_req_count[idx]--;
1939 1982
1940 if (adev->gfx.kfd_sch_req_count[idx] == 0 &&1983 if (adev->gfx.userq_sch_req_count[idx] == 0 &&
1941 adev->gfx.kfd_sch_inactive[idx]) {1984 adev->gfx.userq_sch_inactive[idx]) {
1942 schedule_delayed_work(&adev->gfx.enforce_isolation[idx].work,1985 schedule_delayed_work(&adev->gfx.enforce_isolation[idx].work,
1943 msecs_to_jiffies(adev->gfx.enforce_isolation_time[idx]));1986 msecs_to_jiffies(adev->gfx.enforce_isolation_time[idx]));
1944 }1987 }
1945 } else {1988 } else {
1946 if (adev->gfx.kfd_sch_req_count[idx] == 0) {1989 if (adev->gfx.userq_sch_req_count[idx] == 0) {
1947 cancel_delayed_work_sync(&adev->gfx.enforce_isolation[idx].work);1990 cancel_delayed_work_sync(&adev->gfx.enforce_isolation[idx].work);
1948 if (!adev->gfx.kfd_sch_inactive[idx]) {1991 if (!adev->gfx.userq_sch_inactive[idx]) {
1949 amdgpu_amdkfd_stop_sched(adev, idx);1992 amdgpu_userq_stop_sched_for_enforce_isolation(adev, idx);
1950 adev->gfx.kfd_sch_inactive[idx] = true;1993 if (adev->kfd.init_complete)
1994 amdgpu_amdkfd_stop_sched(adev, idx);
1995 adev->gfx.userq_sch_inactive[idx] = true;
1951 }1996 }
1952 }1997 }
1953 1998
1954 adev->gfx.kfd_sch_req_count[idx]++;1999 adev->gfx.userq_sch_req_count[idx]++;
1955 }2000 }
1956 2001
1957unlock:2002unlock:
1958 mutex_unlock(&adev->gfx.kfd_sch_mutex);2003 mutex_unlock(&adev->gfx.userq_sch_mutex);
1959}2004}
1960 2005
1961/**2006/**
@@ -2000,12 +2045,13 @@ void amdgpu_gfx_enforce_isolation_handler(struct work_struct *work) · 38 unmodified lines
2000 msecs_to_jiffies(1));2045 msecs_to_jiffies(1));
2001 } else {2046 } else {
2002 /* Tell KFD to resume the runqueue */2047 /* Tell KFD to resume the runqueue */
2003 if (adev->kfd.init_complete) {2048 WARN_ON_ONCE(!adev->gfx.userq_sch_inactive[idx]);
2004 WARN_ON_ONCE(!adev->gfx.kfd_sch_inactive[idx]);2049 WARN_ON_ONCE(adev->gfx.userq_sch_req_count[idx]);
2005 WARN_ON_ONCE(adev->gfx.kfd_sch_req_count[idx]);2050
2051 amdgpu_userq_start_sched_for_enforce_isolation(adev, idx);
2052 if (adev->kfd.init_complete)
2006 amdgpu_amdkfd_start_sched(adev, idx);2053 amdgpu_amdkfd_start_sched(adev, idx);
2007 adev->gfx.kfd_sch_inactive[idx] = false;2054 adev->gfx.userq_sch_inactive[idx] = false;
2008 }
2009 }2055 }
2010 mutex_unlock(&adev->enforce_isolation_mutex);2056 mutex_unlock(&adev->enforce_isolation_mutex);
2011}2057}
@@ -2029,7 +2075,7 @@ amdgpu_gfx_enforce_isolation_wait_for_kfd(struct amdgpu_device *adev, · 17 unmodified lines
2029 bool wait = false;2075 bool wait = false;
2030 2076
2031 mutex_lock(&adev->enforce_isolation_mutex);2077 mutex_lock(&adev->enforce_isolation_mutex);
2032 if (adev->enforce_isolation[idx]) {2078 if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) {
2033 /* set the initial values if nothing is set */2079 /* set the initial values if nothing is set */
2034 if (!adev->gfx.enforce_isolation_jiffies[idx]) {2080 if (!adev->gfx.enforce_isolation_jiffies[idx]) {
2035 adev->gfx.enforce_isolation_jiffies[idx] = jiffies;2081 adev->gfx.enforce_isolation_jiffies[idx] = jiffies;
@@ -2096,7 +2142,7 @@ void amdgpu_gfx_enforce_isolation_ring_begin_use(struct amdgpu_ring *ring) · 60 unmodified lines
2096 amdgpu_gfx_enforce_isolation_wait_for_kfd(adev, idx);2142 amdgpu_gfx_enforce_isolation_wait_for_kfd(adev, idx);
2097 2143
2098 mutex_lock(&adev->enforce_isolation_mutex);2144 mutex_lock(&adev->enforce_isolation_mutex);
2099 if (adev->enforce_isolation[idx]) {2145 if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) {
2100 if (adev->kfd.init_complete)2146 if (adev->kfd.init_complete)
2101 sched_work = true;2147 sched_work = true;
2102 }2148 }
@@ -2133,7 +2179,7 @@ void amdgpu_gfx_enforce_isolation_ring_end_use(struct amdgpu_ring *ring) · 30 unmodified lines
2133 return;2179 return;
2134 2180
2135 mutex_lock(&adev->enforce_isolation_mutex);2181 mutex_lock(&adev->enforce_isolation_mutex);
2136 if (adev->enforce_isolation[idx]) {2182 if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) {
2137 if (adev->kfd.init_complete)2183 if (adev->kfd.init_complete)
2138 sched_work = true;2184 sched_work = true;
2139 }2185 }
@@ -2217,6 +2263,74 @@ void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring) · 77 unmodified lines
2217 schedule_delayed_work(&ring->adev->gfx.idle_work, GFX_PROFILE_IDLE_TIMEOUT);2263 schedule_delayed_work(&ring->adev->gfx.idle_work, GFX_PROFILE_IDLE_TIMEOUT);
2218}2264}
2219 2265
2266/**
2267 * amdgpu_gfx_csb_preamble_start - Set CSB preamble start
2268 *
2269 * @buffer: This is an output variable that gets the PACKET3 preamble setup.
2270 *
2271 * Return:
2272 * return the latest index.
2273 */
2274u32 amdgpu_gfx_csb_preamble_start(volatile u32 *buffer)
2275{
2276 u32 count = 0;
2277
2278 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
2279 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);
2280
2281 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
2282 buffer[count++] = cpu_to_le32(0x80000000);
2283 buffer[count++] = cpu_to_le32(0x80000000);
2284
2285 return count;
2286}
2287
2288/**
2289 * amdgpu_gfx_csb_data_parser - Parser CS data
2290 *
2291 * @adev: amdgpu_device pointer used to get the CS data and other gfx info.
2292 * @buffer: This is an output variable that gets the PACKET3 preamble end.
2293 * @count: Index to start set the preemble end.
2294 *
2295 * Return:
2296 * return the latest index.
2297 */
2298u32 amdgpu_gfx_csb_data_parser(struct amdgpu_device *adev, volatile u32 *buffer, u32 count)
2299{
2300 const struct cs_section_def *sect = NULL;
2301 const struct cs_extent_def *ext = NULL;
2302 u32 i;
2303
2304 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
2305 for (ext = sect->section; ext->extent != NULL; ++ext) {
2306 if (sect->id == SECT_CONTEXT) {
2307 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
2308 buffer[count++] = cpu_to_le32(ext->reg_index - PACKET3_SET_CONTEXT_REG_START);
2309
2310 for (i = 0; i < ext->reg_count; i++)
2311 buffer[count++] = cpu_to_le32(ext->extent[i]);
2312 }
2313 }
2314 }
2315
2316 return count;
2317}
2318
2319/**
2320 * amdgpu_gfx_csb_preamble_end - Set CSB preamble end
2321 *
2322 * @buffer: This is an output variable that gets the PACKET3 preamble end.
2323 * @count: Index to start set the preemble end.
2324 */
2325void amdgpu_gfx_csb_preamble_end(volatile u32 *buffer, u32 count)
2326{
2327 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
2328 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
2329
2330 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
2331 buffer[count++] = cpu_to_le32(0);
2332}
2333
2220/*2334/*
2221 * debugfs for to enable/disable gfx job submission to specific core.2335 * debugfs for to enable/disable gfx job submission to specific core.
2222 */2336 */
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h+55−7
@@ -170,10 +170,46 @@ struct amdgpu_kiq { · 169 unmodified lines
170#define AMDGPU_GFX_MAX_SE 4170#define AMDGPU_GFX_MAX_SE 4
171#define AMDGPU_GFX_MAX_SH_PER_SE 2171#define AMDGPU_GFX_MAX_SH_PER_SE 2
172 172
173/**
174 * amdgpu_rb_config - Configure a single Render Backend (RB)
175 *
176 * Bad RBs are fused off and there is a harvest register the driver reads to
177 * determine which RB(s) are fused off so that the driver can configure the
178 * hardware state so that nothing gets sent to them. There are also user
179 * harvest registers that the driver can program to disable additional RBs,
180 * etc., for testing purposes.
181 */
173struct amdgpu_rb_config {182struct amdgpu_rb_config {
183 /**
184 * @rb_backend_disable:
185 *
186 * The value captured from register RB_BACKEND_DISABLE indicates if the
187 * RB backend is disabled or not.
188 */
174 uint32_t rb_backend_disable;189 uint32_t rb_backend_disable;
190
191 /**
192 * @user_rb_backend_disable:
193 *
194 * The value captured from register USER_RB_BACKEND_DISABLE indicates
195 * if the User RB backend is disabled or not.
196 */
175 uint32_t user_rb_backend_disable;197 uint32_t user_rb_backend_disable;
198
199 /**
200 * @raster_config:
201 *
202 * To set up all of the states, it is necessary to have two registers
203 * to keep all of the states. This field holds the first register.
204 */
176 uint32_t raster_config;205 uint32_t raster_config;
206
207 /**
208 * @raster_config_1:
209 *
210 * To set up all of the states, it is necessary to have two registers
211 * to keep all of the states. This field holds the second register.
212 */
177 uint32_t raster_config_1;213 uint32_t raster_config_1;
178};214};
179 215
@@ -221,6 +257,13 @@ struct amdgpu_gfx_config { · 41 unmodified lines
221 uint32_t macrotile_mode_array[16];257 uint32_t macrotile_mode_array[16];
222 258
223 struct gb_addr_config gb_addr_config_fields;259 struct gb_addr_config gb_addr_config_fields;
260
261 /**
262 * @rb_config:
263 *
264 * Matrix that keeps all the Render Backend (color and depth buffer
265 * handling) configuration on the 3D engine.
266 */
224 struct amdgpu_rb_config rb_config[AMDGPU_GFX_MAX_SE][AMDGPU_GFX_MAX_SH_PER_SE];267 struct amdgpu_rb_config rb_config[AMDGPU_GFX_MAX_SE][AMDGPU_GFX_MAX_SH_PER_SE];
225 268
226 /* gfx configure feature */269 /* gfx configure feature */
@@ -305,7 +348,8 @@ struct amdgpu_gfx_funcs { · 78 unmodified lines
305 void (*init_spm_golden)(struct amdgpu_device *adev);348 void (*init_spm_golden)(struct amdgpu_device *adev);
306 void (*update_perfmon_mgcg)(struct amdgpu_device *adev, bool enable);349 void (*update_perfmon_mgcg)(struct amdgpu_device *adev, bool enable);
307 int (*get_gfx_shadow_info)(struct amdgpu_device *adev,350 int (*get_gfx_shadow_info)(struct amdgpu_device *adev,
308 struct amdgpu_gfx_shadow_info *shadow_info);351 struct amdgpu_gfx_shadow_info *shadow_info,
352 bool skip_check);
309 enum amdgpu_gfx_partition353 enum amdgpu_gfx_partition
310 (*query_partition_mode)(struct amdgpu_device *adev);354 (*query_partition_mode)(struct amdgpu_device *adev);
311 int (*switch_partition_mode)(struct amdgpu_device *adev,355 int (*switch_partition_mode)(struct amdgpu_device *adev,
@@ -474,9 +518,9 @@ struct amdgpu_gfx { · 162 unmodified lines
474 bool enable_cleaner_shader;518 bool enable_cleaner_shader;
475 struct amdgpu_isolation_work enforce_isolation[MAX_XCP];519 struct amdgpu_isolation_work enforce_isolation[MAX_XCP];
476 /* Mutex for synchronizing KFD scheduler operations */520 /* Mutex for synchronizing KFD scheduler operations */
477 struct mutex kfd_sch_mutex;521 struct mutex userq_sch_mutex;
478 u64 kfd_sch_req_count[MAX_XCP];522 u64 userq_sch_req_count[MAX_XCP];
479 bool kfd_sch_inactive[MAX_XCP];523 bool userq_sch_inactive[MAX_XCP];
480 unsigned long enforce_isolation_jiffies[MAX_XCP];524 unsigned long enforce_isolation_jiffies[MAX_XCP];
481 unsigned long enforce_isolation_time[MAX_XCP];525 unsigned long enforce_isolation_time[MAX_XCP];
482 526
@@ -484,6 +528,9 @@ struct amdgpu_gfx { · 1 unmodified line
484 struct delayed_work idle_work;528 struct delayed_work idle_work;
485 bool workload_profile_active;529 bool workload_profile_active;
486 struct mutex workload_profile_mutex;530 struct mutex workload_profile_mutex;
531
532 bool disable_kq;
533 bool disable_uq;
487};534};
488 535
489struct amdgpu_gfx_ras_reg_entry {536struct amdgpu_gfx_ras_reg_entry {
@@ -503,7 +550,7 @@ struct amdgpu_gfx_ras_mem_id_entry { · 13 unmodified lines
503#define amdgpu_gfx_select_se_sh(adev, se, sh, instance, xcc_id) ((adev)->gfx.funcs->select_se_sh((adev), (se), (sh), (instance), (xcc_id)))550#define amdgpu_gfx_select_se_sh(adev, se, sh, instance, xcc_id) ((adev)->gfx.funcs->select_se_sh((adev), (se), (sh), (instance), (xcc_id)))
504#define amdgpu_gfx_select_me_pipe_q(adev, me, pipe, q, vmid, xcc_id) ((adev)->gfx.funcs->select_me_pipe_q((adev), (me), (pipe), (q), (vmid), (xcc_id)))551#define amdgpu_gfx_select_me_pipe_q(adev, me, pipe, q, vmid, xcc_id) ((adev)->gfx.funcs->select_me_pipe_q((adev), (me), (pipe), (q), (vmid), (xcc_id)))
505#define amdgpu_gfx_init_spm_golden(adev) (adev)->gfx.funcs->init_spm_golden((adev))552#define amdgpu_gfx_init_spm_golden(adev) (adev)->gfx.funcs->init_spm_golden((adev))
506#define amdgpu_gfx_get_gfx_shadow_info(adev, si) ((adev)->gfx.funcs->get_gfx_shadow_info((adev), (si)))553#define amdgpu_gfx_get_gfx_shadow_info(adev, si) ((adev)->gfx.funcs->get_gfx_shadow_info((adev), (si), false))
507 554
508/**555/**
509 * amdgpu_gfx_create_bitmask - create a bitmask556 * amdgpu_gfx_create_bitmask - create a bitmask
@@ -550,8 +597,6 @@ bool amdgpu_gfx_is_high_priority_compute_queue(struct amdgpu_device *adev, · 40 unmodified lines
550 struct amdgpu_ring *ring);597 struct amdgpu_ring *ring);
551bool amdgpu_gfx_is_high_priority_graphics_queue(struct amdgpu_device *adev,598bool amdgpu_gfx_is_high_priority_graphics_queue(struct amdgpu_device *adev,
552 struct amdgpu_ring *ring);599 struct amdgpu_ring *ring);
553int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev, int me,
554 int pipe, int queue);
555bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me,600bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me,
556 int pipe, int queue);601 int pipe, int queue);
557void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable);602void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable);
@@ -597,6 +642,9 @@ void amdgpu_gfx_enforce_isolation_ring_end_use(struct amdgpu_ring *ring); · 39 unmodified lines
597void amdgpu_gfx_profile_idle_work_handler(struct work_struct *work);642void amdgpu_gfx_profile_idle_work_handler(struct work_struct *work);
598void amdgpu_gfx_profile_ring_begin_use(struct amdgpu_ring *ring);643void amdgpu_gfx_profile_ring_begin_use(struct amdgpu_ring *ring);
599void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring);644void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring);
645u32 amdgpu_gfx_csb_preamble_start(volatile u32 *buffer);
646u32 amdgpu_gfx_csb_data_parser(struct amdgpu_device *adev, volatile u32 *buffer, u32 count);
647void amdgpu_gfx_csb_preamble_end(volatile u32 *buffer, u32 count);
600 648
601void amdgpu_debugfs_gfx_sched_mask_init(struct amdgpu_device *adev);649void amdgpu_debugfs_gfx_sched_mask_init(struct amdgpu_device *adev);
602void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev);650void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c+4−0
@@ -1230,6 +1230,10 @@ static ssize_t current_memory_partition_show( · 1229 unmodified lines
1230 struct amdgpu_device *adev = drm_to_adev(ddev);1230 struct amdgpu_device *adev = drm_to_adev(ddev);
1231 enum amdgpu_memory_partition mode;1231 enum amdgpu_memory_partition mode;
1232 1232
1233 /* Only minimal precaution taken to reject requests while in reset */
1234 if (amdgpu_in_reset(adev))
1235 return -EPERM;
1236
1233 mode = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);1237 mode = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);
1234 if ((mode >= ARRAY_SIZE(nps_desc)) ||1238 if ((mode >= ARRAY_SIZE(nps_desc)) ||
1235 (BIT(mode) & AMDGPU_ALL_NPS_MASK) != BIT(mode))1239 (BIT(mode) & AMDGPU_ALL_NPS_MASK) != BIT(mode))
drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.c+20−0
@@ -22,6 +22,7 @@ · 21 unmodified lines
22 */22 */
23#include "amdgpu.h"23#include "amdgpu.h"
24#include "amdgpu_ras.h"24#include "amdgpu_ras.h"
25#include <uapi/linux/kfd_ioctl.h>
25 26
26int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev)27int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev)
27{28{
@@ -46,3 +47,22 @@ int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev) · 18 unmodified lines
46 /* hdp ras follows amdgpu_ras_block_late_init_default for late init */47 /* hdp ras follows amdgpu_ras_block_late_init_default for late init */
47 return 0;48 return 0;
48}49}
50
51void amdgpu_hdp_generic_flush(struct amdgpu_device *adev,
52 struct amdgpu_ring *ring)
53{
54 if (!ring || !ring->funcs->emit_wreg) {
55 WREG32((adev->rmmio_remap.reg_offset +
56 KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >>
57 2,
58 0);
59 if (adev->nbio.funcs->get_memsize)
60 adev->nbio.funcs->get_memsize(adev);
61 } else {
62 amdgpu_ring_emit_wreg(ring,
63 (adev->rmmio_remap.reg_offset +
64 KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >>
65 2,
66 0);
67 }
68}
drivers/gpu/drm/amd/amdgpu/amdgpu_hdp.h+2−0
@@ -44,4 +44,6 @@ struct amdgpu_hdp { · 43 unmodified lines
44};44};
45 45
46int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev);46int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev);
47void amdgpu_hdp_generic_flush(struct amdgpu_device *adev,
48 struct amdgpu_ring *ring);
47#endif /* __AMDGPU_HDP_H__ */49#endif /* __AMDGPU_HDP_H__ */
drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c+2−2
@@ -163,12 +163,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 162 unmodified lines
163 init_shadow = false;163 init_shadow = false;
164 }164 }
165 165
166 if (!ring->sched.ready && !ring->is_mes_queue) {166 if (!ring->sched.ready) {
167 dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name);167 dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name);
168 return -EINVAL;168 return -EINVAL;
169 }169 }
170 170
171 if (vm && !job->vmid && !ring->is_mes_queue) {171 if (vm && !job->vmid) {
172 dev_err(adev->dev, "VM IB without ID\n");172 dev_err(adev->dev, "VM IB without ID\n");
173 return -EINVAL;173 return -EINVAL;
174 }174 }
drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c+11−3
@@ -576,8 +576,16 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev) · 575 unmodified lines
576 INIT_LIST_HEAD(&id_mgr->ids_lru);576 INIT_LIST_HEAD(&id_mgr->ids_lru);
577 id_mgr->reserved_use_count = 0;577 id_mgr->reserved_use_count = 0;
578 578
579 /* manage only VMIDs not used by KFD */579 /* for GC <10, SDMA uses MMHUB so use first_kfd_vmid for both GC and MM */
580 id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;580 if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(10, 0, 0))
581 /* manage only VMIDs not used by KFD */
582 id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
583 else if (AMDGPU_IS_MMHUB0(i) ||
584 AMDGPU_IS_MMHUB1(i))
585 id_mgr->num_ids = 16;
586 else
587 /* manage only VMIDs not used by KFD */
588 id_mgr->num_ids = adev->vm_manager.first_kfd_vmid;
581 589
582 /* skip over VMID 0, since it is the system VM */590 /* skip over VMID 0, since it is the system VM */
583 for (j = 1; j < id_mgr->num_ids; ++j) {591 for (j = 1; j < id_mgr->num_ids; ++j) {
@@ -588,7 +596,7 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev) · 4 unmodified lines
588 }596 }
589 /* alloc a default reserved vmid to enforce isolation */597 /* alloc a default reserved vmid to enforce isolation */
590 for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) {598 for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) {
591 if (adev->enforce_isolation[i])599 if (adev->enforce_isolation[i] != AMDGPU_ENFORCE_ISOLATION_DISABLE)
592 amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(i));600 amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(i));
593 }601 }
594}602}
drivers/gpu/drm/amd/amdgpu/amdgpu_ih.c+13−2
@@ -25,6 +25,7 @@ · 24 unmodified lines
25 25
26#include "amdgpu.h"26#include "amdgpu.h"
27#include "amdgpu_ih.h"27#include "amdgpu_ih.h"
28#include "amdgpu_reset.h"
28 29
29/**30/**
30 * amdgpu_ih_ring_init - initialize the IH state31 * amdgpu_ih_ring_init - initialize the IH state
@@ -227,13 +228,23 @@ restart_ih: · 196 unmodified lines
227 ih->rptr &= ih->ptr_mask;228 ih->rptr &= ih->ptr_mask;
228 }229 }
229 230
230 amdgpu_ih_set_rptr(adev, ih);231 if (!ih->overflow)
232 amdgpu_ih_set_rptr(adev, ih);
233
231 wake_up_all(&ih->wait_process);234 wake_up_all(&ih->wait_process);
232 235
233 /* make sure wptr hasn't changed while processing */236 /* make sure wptr hasn't changed while processing */
234 wptr = amdgpu_ih_get_wptr(adev, ih);237 wptr = amdgpu_ih_get_wptr(adev, ih);
235 if (wptr != ih->rptr)238 if (wptr != ih->rptr)
236 goto restart_ih;239 if (!ih->overflow)
240 goto restart_ih;
241
242 if (ih->overflow)
243 if (amdgpu_sriov_runtime(adev))
244 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
245 &adev->virt.flr_work),
246 "Failed to queue work! at %s",
247 __func__);
237 248
238 return IRQ_HANDLED;249 return IRQ_HANDLED;
239}250}
drivers/gpu/drm/amd/amdgpu/amdgpu_ih.h+1−0
@@ -72,6 +72,7 @@ struct amdgpu_ih_ring { · 71 unmodified lines
72 /* For waiting on IH processing at checkpoint. */72 /* For waiting on IH processing at checkpoint. */
73 wait_queue_head_t wait_process;73 wait_queue_head_t wait_process;
74 uint64_t processed_timestamp;74 uint64_t processed_timestamp;
75 bool overflow;
75};76};
76 77
77/* return true if time stamp t2 is after t1 with 48bit wrap around */78/* return true if time stamp t2 is after t1 with 48bit wrap around */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.h+1−0
@@ -78,6 +78,7 @@ struct amdgpu_job { · 77 unmodified lines
78 78
79 /* enforce isolation */79 /* enforce isolation */
80 bool enforce_isolation;80 bool enforce_isolation;
81 bool run_cleaner_shader;
81 82
82 uint32_t num_ibs;83 uint32_t num_ibs;
83 struct amdgpu_ib ibs[];84 struct amdgpu_ib ibs[];
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c+71−10
@@ -45,6 +45,7 @@ · 44 unmodified lines
45#include "amdgpu_ras.h"45#include "amdgpu_ras.h"
46#include "amdgpu_reset.h"46#include "amdgpu_reset.h"
47#include "amd_pcie.h"47#include "amd_pcie.h"
48#include "amdgpu_userq.h"
48 49
49void amdgpu_unregister_gpu_instance(struct amdgpu_device *adev)50void amdgpu_unregister_gpu_instance(struct amdgpu_device *adev)
50{51{
@@ -370,6 +371,26 @@ static int amdgpu_firmware_info(struct drm_amdgpu_info_firmware *fw_info, · 319 unmodified lines
370 return 0;371 return 0;
371}372}
372 373
374static int amdgpu_userq_metadata_info_gfx(struct amdgpu_device *adev,
375 struct drm_amdgpu_info *info,
376 struct drm_amdgpu_info_uq_metadata_gfx *meta)
377{
378 int ret = -EOPNOTSUPP;
379
380 if (adev->gfx.funcs->get_gfx_shadow_info) {
381 struct amdgpu_gfx_shadow_info shadow = {};
382
383 adev->gfx.funcs->get_gfx_shadow_info(adev, &shadow, true);
384 meta->shadow_size = shadow.shadow_size;
385 meta->shadow_alignment = shadow.shadow_alignment;
386 meta->csa_size = shadow.csa_size;
387 meta->csa_alignment = shadow.csa_alignment;
388 ret = 0;
389 }
390
391 return ret;
392}
393
373static int amdgpu_hw_ip_info(struct amdgpu_device *adev,394static int amdgpu_hw_ip_info(struct amdgpu_device *adev,
374 struct drm_amdgpu_info *info,395 struct drm_amdgpu_info *info,
375 struct drm_amdgpu_info_hw_ip *result)396 struct drm_amdgpu_info_hw_ip *result)
@@ -387,7 +408,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 11 unmodified lines
387 case AMDGPU_HW_IP_GFX:408 case AMDGPU_HW_IP_GFX:
388 type = AMD_IP_BLOCK_TYPE_GFX;409 type = AMD_IP_BLOCK_TYPE_GFX;
389 for (i = 0; i < adev->gfx.num_gfx_rings; i++)410 for (i = 0; i < adev->gfx.num_gfx_rings; i++)
390 if (adev->gfx.gfx_ring[i].sched.ready)411 if (adev->gfx.gfx_ring[i].sched.ready &&
412 !adev->gfx.gfx_ring[i].no_user_submission)
391 ++num_rings;413 ++num_rings;
392 ib_start_alignment = 32;414 ib_start_alignment = 32;
393 ib_size_alignment = 32;415 ib_size_alignment = 32;
@@ -395,7 +417,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line
395 case AMDGPU_HW_IP_COMPUTE:417 case AMDGPU_HW_IP_COMPUTE:
396 type = AMD_IP_BLOCK_TYPE_GFX;418 type = AMD_IP_BLOCK_TYPE_GFX;
397 for (i = 0; i < adev->gfx.num_compute_rings; i++)419 for (i = 0; i < adev->gfx.num_compute_rings; i++)
398 if (adev->gfx.compute_ring[i].sched.ready)420 if (adev->gfx.compute_ring[i].sched.ready &&
421 !adev->gfx.compute_ring[i].no_user_submission)
399 ++num_rings;422 ++num_rings;
400 ib_start_alignment = 32;423 ib_start_alignment = 32;
401 ib_size_alignment = 32;424 ib_size_alignment = 32;
@@ -403,7 +426,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line
403 case AMDGPU_HW_IP_DMA:426 case AMDGPU_HW_IP_DMA:
404 type = AMD_IP_BLOCK_TYPE_SDMA;427 type = AMD_IP_BLOCK_TYPE_SDMA;
405 for (i = 0; i < adev->sdma.num_instances; i++)428 for (i = 0; i < adev->sdma.num_instances; i++)
406 if (adev->sdma.instance[i].ring.sched.ready)429 if (adev->sdma.instance[i].ring.sched.ready &&
430 !adev->sdma.instance[i].ring.no_user_submission)
407 ++num_rings;431 ++num_rings;
408 ib_start_alignment = 256;432 ib_start_alignment = 256;
409 ib_size_alignment = 4;433 ib_size_alignment = 4;
@@ -414,7 +438,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 4 unmodified lines
414 if (adev->uvd.harvest_config & (1 << i))438 if (adev->uvd.harvest_config & (1 << i))
415 continue;439 continue;
416 440
417 if (adev->uvd.inst[i].ring.sched.ready)441 if (adev->uvd.inst[i].ring.sched.ready &&
442 !adev->uvd.inst[i].ring.no_user_submission)
418 ++num_rings;443 ++num_rings;
419 }444 }
420 ib_start_alignment = 256;445 ib_start_alignment = 256;
@@ -423,7 +448,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 2 unmodified lines
423 case AMDGPU_HW_IP_VCE:448 case AMDGPU_HW_IP_VCE:
424 type = AMD_IP_BLOCK_TYPE_VCE;449 type = AMD_IP_BLOCK_TYPE_VCE;
425 for (i = 0; i < adev->vce.num_rings; i++)450 for (i = 0; i < adev->vce.num_rings; i++)
426 if (adev->vce.ring[i].sched.ready)451 if (adev->vce.ring[i].sched.ready &&
452 !adev->vce.ring[i].no_user_submission)
427 ++num_rings;453 ++num_rings;
428 ib_start_alignment = 256;454 ib_start_alignment = 256;
429 ib_size_alignment = 4;455 ib_size_alignment = 4;
@@ -435,7 +461,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 5 unmodified lines
435 continue;461 continue;
436 462
437 for (j = 0; j < adev->uvd.num_enc_rings; j++)463 for (j = 0; j < adev->uvd.num_enc_rings; j++)
438 if (adev->uvd.inst[i].ring_enc[j].sched.ready)464 if (adev->uvd.inst[i].ring_enc[j].sched.ready &&
465 !adev->uvd.inst[i].ring_enc[j].no_user_submission)
439 ++num_rings;466 ++num_rings;
440 }467 }
441 ib_start_alignment = 256;468 ib_start_alignment = 256;
@@ -447,7 +474,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 5 unmodified lines
447 if (adev->vcn.harvest_config & (1 << i))474 if (adev->vcn.harvest_config & (1 << i))
448 continue;475 continue;
449 476
450 if (adev->vcn.inst[i].ring_dec.sched.ready)477 if (adev->vcn.inst[i].ring_dec.sched.ready &&
478 !adev->vcn.inst[i].ring_dec.no_user_submission)
451 ++num_rings;479 ++num_rings;
452 }480 }
453 ib_start_alignment = 256;481 ib_start_alignment = 256;
@@ -460,7 +488,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 6 unmodified lines
460 continue;488 continue;
461 489
462 for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++)490 for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++)
463 if (adev->vcn.inst[i].ring_enc[j].sched.ready)491 if (adev->vcn.inst[i].ring_enc[j].sched.ready &&
492 !adev->vcn.inst[i].ring_enc[j].no_user_submission)
464 ++num_rings;493 ++num_rings;
465 }494 }
466 ib_start_alignment = 256;495 ib_start_alignment = 256;
@@ -475,7 +504,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 8 unmodified lines
475 continue;504 continue;
476 505
477 for (j = 0; j < adev->jpeg.num_jpeg_rings; j++)506 for (j = 0; j < adev->jpeg.num_jpeg_rings; j++)
478 if (adev->jpeg.inst[i].ring_dec[j].sched.ready)507 if (adev->jpeg.inst[i].ring_dec[j].sched.ready &&
508 !adev->jpeg.inst[i].ring_dec[j].no_user_submission)
479 ++num_rings;509 ++num_rings;
480 }510 }
481 ib_start_alignment = 256;511 ib_start_alignment = 256;
@@ -483,7 +513,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line
483 break;513 break;
484 case AMDGPU_HW_IP_VPE:514 case AMDGPU_HW_IP_VPE:
485 type = AMD_IP_BLOCK_TYPE_VPE;515 type = AMD_IP_BLOCK_TYPE_VPE;
486 if (adev->vpe.ring.sched.ready)516 if (adev->vpe.ring.sched.ready &&
517 !adev->vpe.ring.no_user_submission)
487 ++num_rings;518 ++num_rings;
488 ib_start_alignment = 256;519 ib_start_alignment = 256;
489 ib_size_alignment = 4;520 ib_size_alignment = 4;
@@ -978,6 +1009,8 @@ out: · 488 unmodified lines
978 }1009 }
979 }1010 }
980 1011
1012 dev_info->userq_ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
1013
981 ret = copy_to_user(out, dev_info,1014 ret = copy_to_user(out, dev_info,
982 min((size_t)size, sizeof(*dev_info))) ? -EFAULT : 0;1015 min((size_t)size, sizeof(*dev_info))) ? -EFAULT : 0;
983 kfree(dev_info);1016 kfree(dev_info);
@@ -1293,6 +1326,22 @@ out: · 309 unmodified lines
1293 return copy_to_user(out, &gpuvm_fault,1326 return copy_to_user(out, &gpuvm_fault,
1294 min((size_t)size, sizeof(gpuvm_fault))) ? -EFAULT : 0;1327 min((size_t)size, sizeof(gpuvm_fault))) ? -EFAULT : 0;
1295 }1328 }
1329 case AMDGPU_INFO_UQ_FW_AREAS: {
1330 struct drm_amdgpu_info_uq_metadata meta_info = {};
1331
1332 switch (info->query_hw_ip.type) {
1333 case AMDGPU_HW_IP_GFX:
1334 ret = amdgpu_userq_metadata_info_gfx(adev, info, &meta_info.gfx);
1335 if (ret)
1336 return ret;
1337
1338 ret = copy_to_user(out, &meta_info,
1339 min((size_t)size, sizeof(meta_info))) ? -EFAULT : 0;
1340 return 0;
1341 default:
1342 return -EINVAL;
1343 }
1344 }
1296 default:1345 default:
1297 DRM_DEBUG_KMS("Invalid request %d\n", info->query);1346 DRM_DEBUG_KMS("Invalid request %d\n", info->query);
1298 return -EINVAL;1347 return -EINVAL;
@@ -1376,8 +1425,16 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · 77 unmodified lines
1376 mutex_init(&fpriv->bo_list_lock);1425 mutex_init(&fpriv->bo_list_lock);
1377 idr_init_base(&fpriv->bo_list_handles, 1);1426 idr_init_base(&fpriv->bo_list_handles, 1);
1378 1427
1428 r = amdgpu_eviction_fence_init(&fpriv->evf_mgr);
1429 if (r)
1430 goto error_vm;
1431
1379 amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev);1432 amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev);
1380 1433
1434 r = amdgpu_userq_mgr_init(&fpriv->userq_mgr, file_priv, adev);
1435 if (r)
1436 DRM_WARN("Can't setup usermode queues, use legacy workload submission only\n");
1437
1381 file_priv->driver_priv = fpriv;1438 file_priv->driver_priv = fpriv;
1382 goto out_suspend;1439 goto out_suspend;
1383 1440
@@ -1445,6 +1502,10 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 61 unmodified lines
1445 amdgpu_bo_unreserve(pd);1502 amdgpu_bo_unreserve(pd);
1446 }1503 }
1447 1504
1505 fpriv->evf_mgr.fd_closing = true;
1506 amdgpu_userq_mgr_fini(&fpriv->userq_mgr);
1507 amdgpu_eviction_fence_destroy(&fpriv->evf_mgr);
1508
1448 amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr);1509 amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr);
1449 amdgpu_vm_fini(adev, &fpriv->vm);1510 amdgpu_vm_fini(adev, &fpriv->vm);
1450 1511
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c+55−1051
@@ -39,42 +39,6 @@ int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev) · 38 unmodified lines
39 PAGE_SIZE);39 PAGE_SIZE);
40}40}
41 41
42static int amdgpu_mes_kernel_doorbell_get(struct amdgpu_device *adev,
43 int ip_type, uint64_t *doorbell_index)
44{
45 unsigned int offset, found;
46 struct amdgpu_mes *mes = &adev->mes;
47
48 if (ip_type == AMDGPU_RING_TYPE_SDMA)
49 offset = adev->doorbell_index.sdma_engine[0];
50 else
51 offset = 0;
52
53 found = find_next_zero_bit(mes->doorbell_bitmap, mes->num_mes_dbs, offset);
54 if (found >= mes->num_mes_dbs) {
55 DRM_WARN("No doorbell available\n");
56 return -ENOSPC;
57 }
58
59 set_bit(found, mes->doorbell_bitmap);
60
61 /* Get the absolute doorbell index on BAR */
62 *doorbell_index = mes->db_start_dw_offset + found * 2;
63 return 0;
64}
65
66static void amdgpu_mes_kernel_doorbell_free(struct amdgpu_device *adev,
67 uint32_t doorbell_index)
68{
69 unsigned int old, rel_index;
70 struct amdgpu_mes *mes = &adev->mes;
71
72 /* Find the relative index of the doorbell in this object */
73 rel_index = (doorbell_index - mes->db_start_dw_offset) / 2;
74 old = test_and_clear_bit(rel_index, mes->doorbell_bitmap);
75 WARN_ON(!old);
76}
77
78static int amdgpu_mes_doorbell_init(struct amdgpu_device *adev)42static int amdgpu_mes_doorbell_init(struct amdgpu_device *adev)
79{43{
80 int i;44 int i;
@@ -126,7 +90,7 @@ static void amdgpu_mes_doorbell_free(struct amdgpu_device *adev) · 45 unmodified lines
126 90
127int amdgpu_mes_init(struct amdgpu_device *adev)91int amdgpu_mes_init(struct amdgpu_device *adev)
128{92{
129 int i, r;93 int i, r, num_pipes;
130 94
131 adev->mes.adev = adev;95 adev->mes.adev = adev;
132 96
@@ -142,19 +106,52 @@ int amdgpu_mes_init(struct amdgpu_device *adev) · 9 unmodified lines
142 106
143 adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK;107 adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK;
144 adev->mes.vmid_mask_mmhub = 0xffffff00;108 adev->mes.vmid_mask_mmhub = 0xffffff00;
145 adev->mes.vmid_mask_gfxhub = 0xffffff00;109 adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xfffffffe : 0xffffff00;
110
111 num_pipes = adev->gfx.me.num_pipe_per_me * adev->gfx.me.num_me;
112 if (num_pipes > AMDGPU_MES_MAX_GFX_PIPES)
113 dev_warn(adev->dev, "more gfx pipes than supported by MES! (%d vs %d)\n",
114 num_pipes, AMDGPU_MES_MAX_GFX_PIPES);
115
116 for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) {
117 if (i >= num_pipes)
118 break;
119 if (amdgpu_ip_version(adev, GC_HWIP, 0) >=
120 IP_VERSION(12, 0, 0))
121 /*
122 * GFX V12 has only one GFX pipe, but 8 queues in it.
123 * GFX pipe 0 queue 0 is being used by Kernel queue.
124 * Set GFX pipe 0 queue 1-7 for MES scheduling
125 * mask = 1111 1110b
126 */
127 adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0xFF : 0xFE;
128 else
129 /*
130 * GFX pipe 0 queue 0 is being used by Kernel queue.
131 * Set GFX pipe 0 queue 1 for MES scheduling
132 * mask = 10b
133 */
134 adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0x3 : 0x2;
135 }
136
137 num_pipes = adev->gfx.mec.num_pipe_per_mec * adev->gfx.mec.num_mec;
138 if (num_pipes > AMDGPU_MES_MAX_COMPUTE_PIPES)
139 dev_warn(adev->dev, "more compute pipes than supported by MES! (%d vs %d)\n",
140 num_pipes, AMDGPU_MES_MAX_COMPUTE_PIPES);
146 141
147 for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) {142 for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) {
148 if (i >= (adev->gfx.mec.num_pipe_per_mec * adev->gfx.mec.num_mec))143 if (i >= num_pipes)
149 break;144 break;
150 adev->mes.compute_hqd_mask[i] = 0xc;145 adev->mes.compute_hqd_mask[i] = adev->gfx.disable_kq ? 0xF : 0xC;
151 }146 }
152 147
153 for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++)148 num_pipes = adev->sdma.num_instances;
154 adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffffffe;149 if (num_pipes > AMDGPU_MES_MAX_SDMA_PIPES)
150 dev_warn(adev->dev, "more SDMA pipes than supported by MES! (%d vs %d)\n",
151 num_pipes, AMDGPU_MES_MAX_SDMA_PIPES);
155 152
156 for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {153 for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) {
157 if (i >= adev->sdma.num_instances)154 if (i >= num_pipes)
158 break;155 break;
159 adev->mes.sdma_hqd_mask[i] = 0xfc;156 adev->mes.sdma_hqd_mask[i] = 0xfc;
160 }157 }
@@ -240,244 +237,6 @@ void amdgpu_mes_fini(struct amdgpu_device *adev) · 79 unmodified lines
240 mutex_destroy(&adev->mes.mutex_hidden);237 mutex_destroy(&adev->mes.mutex_hidden);
241}238}
242 239
243static void amdgpu_mes_queue_free_mqd(struct amdgpu_mes_queue *q)
244{
245 amdgpu_bo_free_kernel(&q->mqd_obj,
246 &q->mqd_gpu_addr,
247 &q->mqd_cpu_ptr);
248}
249
250int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
251 struct amdgpu_vm *vm)
252{
253 struct amdgpu_mes_process *process;
254 int r;
255
256 /* allocate the mes process buffer */
257 process = kzalloc(sizeof(struct amdgpu_mes_process), GFP_KERNEL);
258 if (!process) {
259 DRM_ERROR("no more memory to create mes process\n");
260 return -ENOMEM;
261 }
262
263 /* allocate the process context bo and map it */
264 r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_PROC_CTX_SIZE, PAGE_SIZE,
265 AMDGPU_GEM_DOMAIN_GTT,
266 &process->proc_ctx_bo,
267 &process->proc_ctx_gpu_addr,
268 &process->proc_ctx_cpu_ptr);
269 if (r) {
270 DRM_ERROR("failed to allocate process context bo\n");
271 goto clean_up_memory;
272 }
273 memset(process->proc_ctx_cpu_ptr, 0, AMDGPU_MES_PROC_CTX_SIZE);
274
275 /*
276 * Avoid taking any other locks under MES lock to avoid circular
277 * lock dependencies.
278 */
279 amdgpu_mes_lock(&adev->mes);
280
281 /* add the mes process to idr list */
282 r = idr_alloc(&adev->mes.pasid_idr, process, pasid, pasid + 1,
283 GFP_KERNEL);
284 if (r < 0) {
285 DRM_ERROR("failed to lock pasid=%d\n", pasid);
286 goto clean_up_ctx;
287 }
288
289 INIT_LIST_HEAD(&process->gang_list);
290 process->vm = vm;
291 process->pasid = pasid;
292 process->process_quantum = adev->mes.default_process_quantum;
293 process->pd_gpu_addr = amdgpu_bo_gpu_offset(vm->root.bo);
294
295 amdgpu_mes_unlock(&adev->mes);
296 return 0;
297
298clean_up_ctx:
299 amdgpu_mes_unlock(&adev->mes);
300 amdgpu_bo_free_kernel(&process->proc_ctx_bo,
301 &process->proc_ctx_gpu_addr,
302 &process->proc_ctx_cpu_ptr);
303clean_up_memory:
304 kfree(process);
305 return r;
306}
307
308void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid)
309{
310 struct amdgpu_mes_process *process;
311 struct amdgpu_mes_gang *gang, *tmp1;
312 struct amdgpu_mes_queue *queue, *tmp2;
313 struct mes_remove_queue_input queue_input;
314 unsigned long flags;
315 int r;
316
317 /*
318 * Avoid taking any other locks under MES lock to avoid circular
319 * lock dependencies.
320 */
321 amdgpu_mes_lock(&adev->mes);
322
323 process = idr_find(&adev->mes.pasid_idr, pasid);
324 if (!process) {
325 DRM_WARN("pasid %d doesn't exist\n", pasid);
326 amdgpu_mes_unlock(&adev->mes);
327 return;
328 }
329
330 /* Remove all queues from hardware */
331 list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) {
332 list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) {
333 spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
334 idr_remove(&adev->mes.queue_id_idr, queue->queue_id);
335 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
336
337 queue_input.doorbell_offset = queue->doorbell_off;
338 queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
339
340 r = adev->mes.funcs->remove_hw_queue(&adev->mes,
341 &queue_input);
342 if (r)
343 DRM_WARN("failed to remove hardware queue\n");
344 }
345
346 idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
347 }
348
349 idr_remove(&adev->mes.pasid_idr, pasid);
350 amdgpu_mes_unlock(&adev->mes);
351
352 /* free all memory allocated by the process */
353 list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) {
354 /* free all queues in the gang */
355 list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) {
356 amdgpu_mes_queue_free_mqd(queue);
357 list_del(&queue->list);
358 kfree(queue);
359 }
360 amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
361 &gang->gang_ctx_gpu_addr,
362 &gang->gang_ctx_cpu_ptr);
363 list_del(&gang->list);
364 kfree(gang);
365
366 }
367 amdgpu_bo_free_kernel(&process->proc_ctx_bo,
368 &process->proc_ctx_gpu_addr,
369 &process->proc_ctx_cpu_ptr);
370 kfree(process);
371}
372
373int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
374 struct amdgpu_mes_gang_properties *gprops,
375 int *gang_id)
376{
377 struct amdgpu_mes_process *process;
378 struct amdgpu_mes_gang *gang;
379 int r;
380
381 /* allocate the mes gang buffer */
382 gang = kzalloc(sizeof(struct amdgpu_mes_gang), GFP_KERNEL);
383 if (!gang) {
384 return -ENOMEM;
385 }
386
387 /* allocate the gang context bo and map it to cpu space */
388 r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_GANG_CTX_SIZE, PAGE_SIZE,
389 AMDGPU_GEM_DOMAIN_GTT,
390 &gang->gang_ctx_bo,
391 &gang->gang_ctx_gpu_addr,
392 &gang->gang_ctx_cpu_ptr);
393 if (r) {
394 DRM_ERROR("failed to allocate process context bo\n");
395 goto clean_up_mem;
396 }
397 memset(gang->gang_ctx_cpu_ptr, 0, AMDGPU_MES_GANG_CTX_SIZE);
398
399 /*
400 * Avoid taking any other locks under MES lock to avoid circular
401 * lock dependencies.
402 */
403 amdgpu_mes_lock(&adev->mes);
404
405 process = idr_find(&adev->mes.pasid_idr, pasid);
406 if (!process) {
407 DRM_ERROR("pasid %d doesn't exist\n", pasid);
408 r = -EINVAL;
409 goto clean_up_ctx;
410 }
411
412 /* add the mes gang to idr list */
413 r = idr_alloc(&adev->mes.gang_id_idr, gang, 1, 0,
414 GFP_KERNEL);
415 if (r < 0) {
416 DRM_ERROR("failed to allocate idr for gang\n");
417 goto clean_up_ctx;
418 }
419
420 gang->gang_id = r;
421 *gang_id = r;
422
423 INIT_LIST_HEAD(&gang->queue_list);
424 gang->process = process;
425 gang->priority = gprops->priority;
426 gang->gang_quantum = gprops->gang_quantum ?
427 gprops->gang_quantum : adev->mes.default_gang_quantum;
428 gang->global_priority_level = gprops->global_priority_level;
429 gang->inprocess_gang_priority = gprops->inprocess_gang_priority;
430 list_add_tail(&gang->list, &process->gang_list);
431
432 amdgpu_mes_unlock(&adev->mes);
433 return 0;
434
435clean_up_ctx:
436 amdgpu_mes_unlock(&adev->mes);
437 amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
438 &gang->gang_ctx_gpu_addr,
439 &gang->gang_ctx_cpu_ptr);
440clean_up_mem:
441 kfree(gang);
442 return r;
443}
444
445int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id)
446{
447 struct amdgpu_mes_gang *gang;
448
449 /*
450 * Avoid taking any other locks under MES lock to avoid circular
451 * lock dependencies.
452 */
453 amdgpu_mes_lock(&adev->mes);
454
455 gang = idr_find(&adev->mes.gang_id_idr, gang_id);
456 if (!gang) {
457 DRM_ERROR("gang id %d doesn't exist\n", gang_id);
458 amdgpu_mes_unlock(&adev->mes);
459 return -EINVAL;
460 }
461
462 if (!list_empty(&gang->queue_list)) {
463 DRM_ERROR("queue list is not empty\n");
464 amdgpu_mes_unlock(&adev->mes);
465 return -EBUSY;
466 }
467
468 idr_remove(&adev->mes.gang_id_idr, gang->gang_id);
469 list_del(&gang->list);
470 amdgpu_mes_unlock(&adev->mes);
471
472 amdgpu_bo_free_kernel(&gang->gang_ctx_bo,
473 &gang->gang_ctx_gpu_addr,
474 &gang->gang_ctx_cpu_ptr);
475
476 kfree(gang);
477
478 return 0;
479}
480
481int amdgpu_mes_suspend(struct amdgpu_device *adev)240int amdgpu_mes_suspend(struct amdgpu_device *adev)
482{241{
483 struct mes_suspend_gang_input input;242 struct mes_suspend_gang_input input;
@@ -526,304 +285,6 @@ int amdgpu_mes_resume(struct amdgpu_device *adev) · 42 unmodified lines
526 return r;285 return r;
527}286}
528 287
529static int amdgpu_mes_queue_alloc_mqd(struct amdgpu_device *adev,
530 struct amdgpu_mes_queue *q,
531 struct amdgpu_mes_queue_properties *p)
532{
533 struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type];
534 u32 mqd_size = mqd_mgr->mqd_size;
535 int r;
536
537 r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE,
538 AMDGPU_GEM_DOMAIN_GTT,
539 &q->mqd_obj,
540 &q->mqd_gpu_addr, &q->mqd_cpu_ptr);
541 if (r) {
542 dev_warn(adev->dev, "failed to create queue mqd bo (%d)", r);
543 return r;
544 }
545 memset(q->mqd_cpu_ptr, 0, mqd_size);
546
547 r = amdgpu_bo_reserve(q->mqd_obj, false);
548 if (unlikely(r != 0))
549 goto clean_up;
550
551 return 0;
552
553clean_up:
554 amdgpu_bo_free_kernel(&q->mqd_obj,
555 &q->mqd_gpu_addr,
556 &q->mqd_cpu_ptr);
557 return r;
558}
559
560static void amdgpu_mes_queue_init_mqd(struct amdgpu_device *adev,
561 struct amdgpu_mes_queue *q,
562 struct amdgpu_mes_queue_properties *p)
563{
564 struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type];
565 struct amdgpu_mqd_prop mqd_prop = {0};
566
567 mqd_prop.mqd_gpu_addr = q->mqd_gpu_addr;
568 mqd_prop.hqd_base_gpu_addr = p->hqd_base_gpu_addr;
569 mqd_prop.rptr_gpu_addr = p->rptr_gpu_addr;
570 mqd_prop.wptr_gpu_addr = p->wptr_gpu_addr;
571 mqd_prop.queue_size = p->queue_size;
572 mqd_prop.use_doorbell = true;
573 mqd_prop.doorbell_index = p->doorbell_off;
574 mqd_prop.eop_gpu_addr = p->eop_gpu_addr;
575 mqd_prop.hqd_pipe_priority = p->hqd_pipe_priority;
576 mqd_prop.hqd_queue_priority = p->hqd_queue_priority;
577 mqd_prop.hqd_active = false;
578
579 if (p->queue_type == AMDGPU_RING_TYPE_GFX ||
580 p->queue_type == AMDGPU_RING_TYPE_COMPUTE) {
581 mutex_lock(&adev->srbm_mutex);
582 amdgpu_gfx_select_me_pipe_q(adev, p->ring->me, p->ring->pipe, 0, 0, 0);
583 }
584
585 mqd_mgr->init_mqd(adev, q->mqd_cpu_ptr, &mqd_prop);
586
587 if (p->queue_type == AMDGPU_RING_TYPE_GFX ||
588 p->queue_type == AMDGPU_RING_TYPE_COMPUTE) {
589 amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0);
590 mutex_unlock(&adev->srbm_mutex);
591 }
592
593 amdgpu_bo_unreserve(q->mqd_obj);
594}
595
596int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
597 struct amdgpu_mes_queue_properties *qprops,
598 int *queue_id)
599{
600 struct amdgpu_mes_queue *queue;
601 struct amdgpu_mes_gang *gang;
602 struct mes_add_queue_input queue_input;
603 unsigned long flags;
604 int r;
605
606 memset(&queue_input, 0, sizeof(struct mes_add_queue_input));
607
608 /* allocate the mes queue buffer */
609 queue = kzalloc(sizeof(struct amdgpu_mes_queue), GFP_KERNEL);
610 if (!queue) {
611 DRM_ERROR("Failed to allocate memory for queue\n");
612 return -ENOMEM;
613 }
614
615 /* Allocate the queue mqd */
616 r = amdgpu_mes_queue_alloc_mqd(adev, queue, qprops);
617 if (r)
618 goto clean_up_memory;
619
620 /*
621 * Avoid taking any other locks under MES lock to avoid circular
622 * lock dependencies.
623 */
624 amdgpu_mes_lock(&adev->mes);
625
626 gang = idr_find(&adev->mes.gang_id_idr, gang_id);
627 if (!gang) {
628 DRM_ERROR("gang id %d doesn't exist\n", gang_id);
629 r = -EINVAL;
630 goto clean_up_mqd;
631 }
632
633 /* add the mes gang to idr list */
634 spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
635 r = idr_alloc(&adev->mes.queue_id_idr, queue, 1, 0,
636 GFP_ATOMIC);
637 if (r < 0) {
638 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
639 goto clean_up_mqd;
640 }
641 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
642 *queue_id = queue->queue_id = r;
643
644 /* allocate a doorbell index for the queue */
645 r = amdgpu_mes_kernel_doorbell_get(adev,
646 qprops->queue_type,
647 &qprops->doorbell_off);
648 if (r)
649 goto clean_up_queue_id;
650
651 /* initialize the queue mqd */
652 amdgpu_mes_queue_init_mqd(adev, queue, qprops);
653
654 /* add hw queue to mes */
655 queue_input.process_id = gang->process->pasid;
656
657 queue_input.page_table_base_addr =
658 adev->vm_manager.vram_base_offset + gang->process->pd_gpu_addr -
659 adev->gmc.vram_start;
660
661 queue_input.process_va_start = 0;
662 queue_input.process_va_end =
663 (adev->vm_manager.max_pfn - 1) << AMDGPU_GPU_PAGE_SHIFT;
664 queue_input.process_quantum = gang->process->process_quantum;
665 queue_input.process_context_addr = gang->process->proc_ctx_gpu_addr;
666 queue_input.gang_quantum = gang->gang_quantum;
667 queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
668 queue_input.inprocess_gang_priority = gang->inprocess_gang_priority;
669 queue_input.gang_global_priority_level = gang->global_priority_level;
670 queue_input.doorbell_offset = qprops->doorbell_off;
671 queue_input.mqd_addr = queue->mqd_gpu_addr;
672 queue_input.wptr_addr = qprops->wptr_gpu_addr;
673 queue_input.wptr_mc_addr = qprops->wptr_mc_addr;
674 queue_input.queue_type = qprops->queue_type;
675 queue_input.paging = qprops->paging;
676 queue_input.is_kfd_process = 0;
677
678 r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input);
679 if (r) {
680 DRM_ERROR("failed to add hardware queue to MES, doorbell=0x%llx\n",
681 qprops->doorbell_off);
682 goto clean_up_doorbell;
683 }
684
685 DRM_DEBUG("MES hw queue was added, pasid=%d, gang id=%d, "
686 "queue type=%d, doorbell=0x%llx\n",
687 gang->process->pasid, gang_id, qprops->queue_type,
688 qprops->doorbell_off);
689
690 queue->ring = qprops->ring;
691 queue->doorbell_off = qprops->doorbell_off;
692 queue->wptr_gpu_addr = qprops->wptr_gpu_addr;
693 queue->queue_type = qprops->queue_type;
694 queue->paging = qprops->paging;
695 queue->gang = gang;
696 queue->ring->mqd_ptr = queue->mqd_cpu_ptr;
697 list_add_tail(&queue->list, &gang->queue_list);
698
699 amdgpu_mes_unlock(&adev->mes);
700 return 0;
701
702clean_up_doorbell:
703 amdgpu_mes_kernel_doorbell_free(adev, qprops->doorbell_off);
704clean_up_queue_id:
705 spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
706 idr_remove(&adev->mes.queue_id_idr, queue->queue_id);
707 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
708clean_up_mqd:
709 amdgpu_mes_unlock(&adev->mes);
710 amdgpu_mes_queue_free_mqd(queue);
711clean_up_memory:
712 kfree(queue);
713 return r;
714}
715
716int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id)
717{
718 unsigned long flags;
719 struct amdgpu_mes_queue *queue;
720 struct amdgpu_mes_gang *gang;
721 struct mes_remove_queue_input queue_input;
722 int r;
723
724 /*
725 * Avoid taking any other locks under MES lock to avoid circular
726 * lock dependencies.
727 */
728 amdgpu_mes_lock(&adev->mes);
729
730 /* remove the mes gang from idr list */
731 spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
732
733 queue = idr_find(&adev->mes.queue_id_idr, queue_id);
734 if (!queue) {
735 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
736 amdgpu_mes_unlock(&adev->mes);
737 DRM_ERROR("queue id %d doesn't exist\n", queue_id);
738 return -EINVAL;
739 }
740
741 idr_remove(&adev->mes.queue_id_idr, queue_id);
742 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
743
744 DRM_DEBUG("try to remove queue, doorbell off = 0x%llx\n",
745 queue->doorbell_off);
746
747 gang = queue->gang;
748 queue_input.doorbell_offset = queue->doorbell_off;
749 queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
750
751 r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);
752 if (r)
753 DRM_ERROR("failed to remove hardware queue, queue id = %d\n",
754 queue_id);
755
756 list_del(&queue->list);
757 amdgpu_mes_kernel_doorbell_free(adev, queue->doorbell_off);
758 amdgpu_mes_unlock(&adev->mes);
759
760 amdgpu_mes_queue_free_mqd(queue);
761 kfree(queue);
762 return 0;
763}
764
765int amdgpu_mes_reset_hw_queue(struct amdgpu_device *adev, int queue_id)
766{
767 unsigned long flags;
768 struct amdgpu_mes_queue *queue;
769 struct amdgpu_mes_gang *gang;
770 struct mes_reset_queue_input queue_input;
771 int r;
772
773 /*
774 * Avoid taking any other locks under MES lock to avoid circular
775 * lock dependencies.
776 */
777 amdgpu_mes_lock(&adev->mes);
778
779 /* remove the mes gang from idr list */
780 spin_lock_irqsave(&adev->mes.queue_id_lock, flags);
781
782 queue = idr_find(&adev->mes.queue_id_idr, queue_id);
783 if (!queue) {
784 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
785 amdgpu_mes_unlock(&adev->mes);
786 DRM_ERROR("queue id %d doesn't exist\n", queue_id);
787 return -EINVAL;
788 }
789 spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags);
790
791 DRM_DEBUG("try to reset queue, doorbell off = 0x%llx\n",
792 queue->doorbell_off);
793
794 gang = queue->gang;
795 queue_input.doorbell_offset = queue->doorbell_off;
796 queue_input.gang_context_addr = gang->gang_ctx_gpu_addr;
797
798 r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
799 if (r)
800 DRM_ERROR("failed to reset hardware queue, queue id = %d\n",
801 queue_id);
802
803 amdgpu_mes_unlock(&adev->mes);
804
805 return 0;
806}
807
808int amdgpu_mes_reset_hw_queue_mmio(struct amdgpu_device *adev, int queue_type,
809 int me_id, int pipe_id, int queue_id, int vmid)
810{
811 struct mes_reset_queue_input queue_input;
812 int r;
813
814 queue_input.queue_type = queue_type;
815 queue_input.use_mmio = true;
816 queue_input.me_id = me_id;
817 queue_input.pipe_id = pipe_id;
818 queue_input.queue_id = queue_id;
819 queue_input.vmid = vmid;
820 r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
821 if (r)
822 DRM_ERROR("failed to reset hardware queue by mmio, queue id = %d\n",
823 queue_id);
824 return r;
825}
826
827int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev,288int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev,
828 struct amdgpu_ring *ring)289 struct amdgpu_ring *ring)
829{290{
@@ -874,7 +335,7 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 44 unmodified lines
874 unsigned int vmid,335 unsigned int vmid,
875 bool use_mmio)336 bool use_mmio)
876{337{
877 struct mes_reset_legacy_queue_input queue_input;338 struct mes_reset_queue_input queue_input;
878 int r;339 int r;
879 340
880 memset(&queue_input, 0, sizeof(queue_input));341 memset(&queue_input, 0, sizeof(queue_input));
@@ -888,8 +349,11 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 7 unmodified lines
888 queue_input.wptr_addr = ring->wptr_gpu_addr;349 queue_input.wptr_addr = ring->wptr_gpu_addr;
889 queue_input.vmid = vmid;350 queue_input.vmid = vmid;
890 queue_input.use_mmio = use_mmio;351 queue_input.use_mmio = use_mmio;
352 queue_input.is_kq = true;
353 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX)
354 queue_input.legacy_gfx = true;
891 355
892 r = adev->mes.funcs->reset_legacy_queue(&adev->mes, &queue_input);356 r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
893 if (r)357 if (r)
894 DRM_ERROR("failed to reset legacy queue\n");358 DRM_ERROR("failed to reset legacy queue\n");
895 359
@@ -905,7 +369,7 @@ uint32_t amdgpu_mes_rreg(struct amdgpu_device *adev, uint32_t reg) · 9 unmodified lines
905 uint32_t *read_val_ptr;369 uint32_t *read_val_ptr;
906 370
907 if (amdgpu_device_wb_get(adev, &addr_offset)) {371 if (amdgpu_device_wb_get(adev, &addr_offset)) {
908 DRM_ERROR("critical bug! too many mes readers\n");372 dev_err(adev->dev, "critical bug! too many mes readers\n");
909 goto error;373 goto error;
910 }374 }
911 read_val_gpu_addr = adev->wb.gpu_addr + (addr_offset * 4);375 read_val_gpu_addr = adev->wb.gpu_addr + (addr_offset * 4);
@@ -915,13 +379,13 @@ uint32_t amdgpu_mes_rreg(struct amdgpu_device *adev, uint32_t reg) · 3 unmodified lines
915 op_input.read_reg.buffer_addr = read_val_gpu_addr;379 op_input.read_reg.buffer_addr = read_val_gpu_addr;
916 380
917 if (!adev->mes.funcs->misc_op) {381 if (!adev->mes.funcs->misc_op) {
918 DRM_ERROR("mes rreg is not supported!\n");382 dev_err(adev->dev, "mes rreg is not supported!\n");
919 goto error;383 goto error;
920 }384 }
921 385
922 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);386 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
923 if (r)387 if (r)
924 DRM_ERROR("failed to read reg (0x%x)\n", reg);388 dev_err(adev->dev, "failed to read reg (0x%x)\n", reg);
925 else389 else
926 val = *(read_val_ptr);390 val = *(read_val_ptr);
927 391
@@ -942,14 +406,14 @@ int amdgpu_mes_wreg(struct amdgpu_device *adev, · 14 unmodified lines
942 op_input.write_reg.reg_value = val;406 op_input.write_reg.reg_value = val;
943 407
944 if (!adev->mes.funcs->misc_op) {408 if (!adev->mes.funcs->misc_op) {
945 DRM_ERROR("mes wreg is not supported!\n");409 dev_err(adev->dev, "mes wreg is not supported!\n");
946 r = -EINVAL;410 r = -EINVAL;
947 goto error;411 goto error;
948 }412 }
949 413
950 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);414 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
951 if (r)415 if (r)
952 DRM_ERROR("failed to write reg (0x%x)\n", reg);416 dev_err(adev->dev, "failed to write reg (0x%x)\n", reg);
953 417
954error:418error:
955 return r;419 return r;
@@ -969,14 +433,14 @@ int amdgpu_mes_reg_write_reg_wait(struct amdgpu_device *adev, · 13 unmodified lines
969 op_input.wrm_reg.mask = mask;433 op_input.wrm_reg.mask = mask;
970 434
971 if (!adev->mes.funcs->misc_op) {435 if (!adev->mes.funcs->misc_op) {
972 DRM_ERROR("mes reg_write_reg_wait is not supported!\n");436 dev_err(adev->dev, "mes reg_write_reg_wait is not supported!\n");
973 r = -EINVAL;437 r = -EINVAL;
974 goto error;438 goto error;
975 }439 }
976 440
977 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);441 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
978 if (r)442 if (r)
979 DRM_ERROR("failed to reg_write_reg_wait\n");443 dev_err(adev->dev, "failed to reg_write_reg_wait\n");
980 444
981error:445error:
982 return r;446 return r;
@@ -994,14 +458,14 @@ int amdgpu_mes_reg_wait(struct amdgpu_device *adev, uint32_t reg, · 11 unmodified lines
994 op_input.wrm_reg.mask = mask;458 op_input.wrm_reg.mask = mask;
995 459
996 if (!adev->mes.funcs->misc_op) {460 if (!adev->mes.funcs->misc_op) {
997 DRM_ERROR("mes reg wait is not supported!\n");461 dev_err(adev->dev, "mes reg wait is not supported!\n");
998 r = -EINVAL;462 r = -EINVAL;
999 goto error;463 goto error;
1000 }464 }
1001 465
1002 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);466 r = adev->mes.funcs->misc_op(&adev->mes, &op_input);
1003 if (r)467 if (r)
1004 DRM_ERROR("failed to reg_write_reg_wait\n");468 dev_err(adev->dev, "failed to reg_write_reg_wait\n");
1005 469
1006error:470error:
1007 return r;471 return r;
@@ -1075,25 +539,6 @@ int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev, · 67 unmodified lines
1075 return r;539 return r;
1076}540}
1077 541
1078static void
1079amdgpu_mes_ring_to_queue_props(struct amdgpu_device *adev,
1080 struct amdgpu_ring *ring,
1081 struct amdgpu_mes_queue_properties *props)
1082{
1083 props->queue_type = ring->funcs->type;
1084 props->hqd_base_gpu_addr = ring->gpu_addr;
1085 props->rptr_gpu_addr = ring->rptr_gpu_addr;
1086 props->wptr_gpu_addr = ring->wptr_gpu_addr;
1087 props->wptr_mc_addr =
1088 ring->mes_ctx->meta_data_mc_addr + ring->wptr_offs;
1089 props->queue_size = ring->ring_size;
1090 props->eop_gpu_addr = ring->eop_gpu_addr;
1091 props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
1092 props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
1093 props->paging = false;
1094 props->ring = ring;
1095}
1096
1097#define DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(_eng) \542#define DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(_eng) \
1098do { \543do { \
1099 if (id_offs < AMDGPU_MES_CTX_MAX_OFFS) \544 if (id_offs < AMDGPU_MES_CTX_MAX_OFFS) \
@@ -1130,453 +575,12 @@ int amdgpu_mes_ctx_get_offs(struct amdgpu_ring *ring, unsigned int id_offs) · 30 unmodified lines
1130 return -EINVAL;575 return -EINVAL;
1131}576}
1132 577
1133int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
1134 int queue_type, int idx,
1135 struct amdgpu_mes_ctx_data *ctx_data,
1136 struct amdgpu_ring **out)
1137{
1138 struct amdgpu_ring *ring;
1139 struct amdgpu_mes_gang *gang;
1140 struct amdgpu_mes_queue_properties qprops = {0};
1141 int r, queue_id, pasid;
1142
1143 /*
1144 * Avoid taking any other locks under MES lock to avoid circular
1145 * lock dependencies.
1146 */
1147 amdgpu_mes_lock(&adev->mes);
1148 gang = idr_find(&adev->mes.gang_id_idr, gang_id);
1149 if (!gang) {
1150 DRM_ERROR("gang id %d doesn't exist\n", gang_id);
1151 amdgpu_mes_unlock(&adev->mes);
1152 return -EINVAL;
1153 }
1154 pasid = gang->process->pasid;
1155
1156 ring = kzalloc(sizeof(struct amdgpu_ring), GFP_KERNEL);
1157 if (!ring) {
1158 amdgpu_mes_unlock(&adev->mes);
1159 return -ENOMEM;
1160 }
1161
1162 ring->ring_obj = NULL;
1163 ring->use_doorbell = true;
1164 ring->is_mes_queue = true;
1165 ring->mes_ctx = ctx_data;
1166 ring->idx = idx;
1167 ring->no_scheduler = true;
1168
1169 if (queue_type == AMDGPU_RING_TYPE_COMPUTE) {
1170 int offset = offsetof(struct amdgpu_mes_ctx_meta_data,
1171 compute[ring->idx].mec_hpd);
1172 ring->eop_gpu_addr =
1173 amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1174 }
1175
1176 switch (queue_type) {
1177 case AMDGPU_RING_TYPE_GFX:
1178 ring->funcs = adev->gfx.gfx_ring[0].funcs;
1179 ring->me = adev->gfx.gfx_ring[0].me;
1180 ring->pipe = adev->gfx.gfx_ring[0].pipe;
1181 break;
1182 case AMDGPU_RING_TYPE_COMPUTE:
1183 ring->funcs = adev->gfx.compute_ring[0].funcs;
1184 ring->me = adev->gfx.compute_ring[0].me;
1185 ring->pipe = adev->gfx.compute_ring[0].pipe;
1186 break;
1187 case AMDGPU_RING_TYPE_SDMA:
1188 ring->funcs = adev->sdma.instance[0].ring.funcs;
1189 break;
1190 default:
1191 BUG();
1192 }
1193
1194 r = amdgpu_ring_init(adev, ring, 1024, NULL, 0,
1195 AMDGPU_RING_PRIO_DEFAULT, NULL);
1196 if (r) {
1197 amdgpu_mes_unlock(&adev->mes);
1198 goto clean_up_memory;
1199 }
1200
1201 amdgpu_mes_ring_to_queue_props(adev, ring, &qprops);
1202
1203 dma_fence_wait(gang->process->vm->last_update, false);
1204 dma_fence_wait(ctx_data->meta_data_va->last_pt_update, false);
1205 amdgpu_mes_unlock(&adev->mes);
1206
1207 r = amdgpu_mes_add_hw_queue(adev, gang_id, &qprops, &queue_id);
1208 if (r)
1209 goto clean_up_ring;
1210
1211 ring->hw_queue_id = queue_id;
1212 ring->doorbell_index = qprops.doorbell_off;
1213
1214 if (queue_type == AMDGPU_RING_TYPE_GFX)
1215 sprintf(ring->name, "gfx_%d.%d.%d", pasid, gang_id, queue_id);
1216 else if (queue_type == AMDGPU_RING_TYPE_COMPUTE)
1217 sprintf(ring->name, "compute_%d.%d.%d", pasid, gang_id,
1218 queue_id);
1219 else if (queue_type == AMDGPU_RING_TYPE_SDMA)
1220 sprintf(ring->name, "sdma_%d.%d.%d", pasid, gang_id,
1221 queue_id);
1222 else
1223 BUG();
1224
1225 *out = ring;
1226 return 0;
1227
1228clean_up_ring:
1229 amdgpu_ring_fini(ring);
1230clean_up_memory:
1231 kfree(ring);
1232 return r;
1233}
1234
1235void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
1236 struct amdgpu_ring *ring)
1237{
1238 if (!ring)
1239 return;
1240
1241 amdgpu_mes_remove_hw_queue(adev, ring->hw_queue_id);
1242 timer_delete_sync(&ring->fence_drv.fallback_timer);
1243 amdgpu_ring_fini(ring);
1244 kfree(ring);
1245}
1246
1247uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev,578uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev,
1248 enum amdgpu_mes_priority_level prio)579 enum amdgpu_mes_priority_level prio)
1249{580{
1250 return adev->mes.aggregated_doorbells[prio];581 return adev->mes.aggregated_doorbells[prio];
1251}582}
1252 583
1253int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
1254 struct amdgpu_mes_ctx_data *ctx_data)
1255{
1256 int r;
1257
1258 r = amdgpu_bo_create_kernel(adev,
1259 sizeof(struct amdgpu_mes_ctx_meta_data),
1260 PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
1261 &ctx_data->meta_data_obj,
1262 &ctx_data->meta_data_mc_addr,
1263 &ctx_data->meta_data_ptr);
1264 if (r) {
1265 dev_warn(adev->dev, "(%d) create CTX bo failed\n", r);
1266 return r;
1267 }
1268
1269 if (!ctx_data->meta_data_obj)
1270 return -ENOMEM;
1271
1272 memset(ctx_data->meta_data_ptr, 0,
1273 sizeof(struct amdgpu_mes_ctx_meta_data));
1274
1275 return 0;
1276}
1277
1278void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data)
1279{
1280 if (ctx_data->meta_data_obj)
1281 amdgpu_bo_free_kernel(&ctx_data->meta_data_obj,
1282 &ctx_data->meta_data_mc_addr,
1283 &ctx_data->meta_data_ptr);
1284}
1285
1286int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
1287 struct amdgpu_vm *vm,
1288 struct amdgpu_mes_ctx_data *ctx_data)
1289{
1290 struct amdgpu_bo_va *bo_va;
1291 struct amdgpu_sync sync;
1292 struct drm_exec exec;
1293 int r;
1294
1295 amdgpu_sync_create(&sync);
1296
1297 drm_exec_init(&exec, 0, 0);
1298 drm_exec_until_all_locked(&exec) {
1299 r = drm_exec_lock_obj(&exec,
1300 &ctx_data->meta_data_obj->tbo.base);
1301 drm_exec_retry_on_contention(&exec);
1302 if (unlikely(r))
1303 goto error_fini_exec;
1304
1305 r = amdgpu_vm_lock_pd(vm, &exec, 0);
1306 drm_exec_retry_on_contention(&exec);
1307 if (unlikely(r))
1308 goto error_fini_exec;
1309 }
1310
1311 bo_va = amdgpu_vm_bo_add(adev, vm, ctx_data->meta_data_obj);
1312 if (!bo_va) {
1313 DRM_ERROR("failed to create bo_va for meta data BO\n");
1314 r = -ENOMEM;
1315 goto error_fini_exec;
1316 }
1317
1318 r = amdgpu_vm_bo_map(adev, bo_va, ctx_data->meta_data_gpu_addr, 0,
1319 sizeof(struct amdgpu_mes_ctx_meta_data),
1320 AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE |
1321 AMDGPU_PTE_EXECUTABLE);
1322
1323 if (r) {
1324 DRM_ERROR("failed to do bo_map on meta data, err=%d\n", r);
1325 goto error_del_bo_va;
1326 }
1327
1328 r = amdgpu_vm_bo_update(adev, bo_va, false);
1329 if (r) {
1330 DRM_ERROR("failed to do vm_bo_update on meta data\n");
1331 goto error_del_bo_va;
1332 }
1333 amdgpu_sync_fence(&sync, bo_va->last_pt_update, GFP_KERNEL);
1334
1335 r = amdgpu_vm_update_pdes(adev, vm, false);
1336 if (r) {
1337 DRM_ERROR("failed to update pdes on meta data\n");
1338 goto error_del_bo_va;
1339 }
1340 amdgpu_sync_fence(&sync, vm->last_update, GFP_KERNEL);
1341
1342 amdgpu_sync_wait(&sync, false);
1343 drm_exec_fini(&exec);
1344
1345 amdgpu_sync_free(&sync);
1346 ctx_data->meta_data_va = bo_va;
1347 return 0;
1348
1349error_del_bo_va:
1350 amdgpu_vm_bo_del(adev, bo_va);
1351
1352error_fini_exec:
1353 drm_exec_fini(&exec);
1354 amdgpu_sync_free(&sync);
1355 return r;
1356}
1357
1358int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev,
1359 struct amdgpu_mes_ctx_data *ctx_data)
1360{
1361 struct amdgpu_bo_va *bo_va = ctx_data->meta_data_va;
1362 struct amdgpu_bo *bo = ctx_data->meta_data_obj;
1363 struct amdgpu_vm *vm = bo_va->base.vm;
1364 struct dma_fence *fence;
1365 struct drm_exec exec;
1366 long r;
1367
1368 drm_exec_init(&exec, 0, 0);
1369 drm_exec_until_all_locked(&exec) {
1370 r = drm_exec_lock_obj(&exec,
1371 &ctx_data->meta_data_obj->tbo.base);
1372 drm_exec_retry_on_contention(&exec);
1373 if (unlikely(r))
1374 goto out_unlock;
1375
1376 r = amdgpu_vm_lock_pd(vm, &exec, 0);
1377 drm_exec_retry_on_contention(&exec);
1378 if (unlikely(r))
1379 goto out_unlock;
1380 }
1381
1382 amdgpu_vm_bo_del(adev, bo_va);
1383 if (!amdgpu_vm_ready(vm))
1384 goto out_unlock;
1385
1386 r = dma_resv_get_singleton(bo->tbo.base.resv, DMA_RESV_USAGE_BOOKKEEP,
1387 &fence);
1388 if (r)
1389 goto out_unlock;
1390 if (fence) {
1391 amdgpu_bo_fence(bo, fence, true);
1392 fence = NULL;
1393 }
1394
1395 r = amdgpu_vm_clear_freed(adev, vm, &fence);
1396 if (r || !fence)
1397 goto out_unlock;
1398
1399 dma_fence_wait(fence, false);
1400 amdgpu_bo_fence(bo, fence, true);
1401 dma_fence_put(fence);
1402
1403out_unlock:
1404 if (unlikely(r < 0))
1405 dev_err(adev->dev, "failed to clear page tables (%ld)\n", r);
1406 drm_exec_fini(&exec);
1407
1408 return r;
1409}
1410
1411static int amdgpu_mes_test_create_gang_and_queues(struct amdgpu_device *adev,
1412 int pasid, int *gang_id,
1413 int queue_type, int num_queue,
1414 struct amdgpu_ring **added_rings,
1415 struct amdgpu_mes_ctx_data *ctx_data)
1416{
1417 struct amdgpu_ring *ring;
1418 struct amdgpu_mes_gang_properties gprops = {0};
1419 int r, j;
1420
1421 /* create a gang for the process */
1422 gprops.priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1423 gprops.gang_quantum = adev->mes.default_gang_quantum;
1424 gprops.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1425 gprops.priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1426 gprops.global_priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
1427
1428 r = amdgpu_mes_add_gang(adev, pasid, &gprops, gang_id);
1429 if (r) {
1430 DRM_ERROR("failed to add gang\n");
1431 return r;
1432 }
1433
1434 /* create queues for the gang */
1435 for (j = 0; j < num_queue; j++) {
1436 r = amdgpu_mes_add_ring(adev, *gang_id, queue_type, j,
1437 ctx_data, &ring);
1438 if (r) {
1439 DRM_ERROR("failed to add ring\n");
1440 break;
1441 }
1442
1443 DRM_INFO("ring %s was added\n", ring->name);
1444 added_rings[j] = ring;
1445 }
1446
1447 return 0;
1448}
1449
1450static int amdgpu_mes_test_queues(struct amdgpu_ring **added_rings)
1451{
1452 struct amdgpu_ring *ring;
1453 int i, r;
1454
1455 for (i = 0; i < AMDGPU_MES_CTX_MAX_RINGS; i++) {
1456 ring = added_rings[i];
1457 if (!ring)
1458 continue;
1459
1460 r = amdgpu_ring_test_helper(ring);
1461 if (r)
1462 return r;
1463
1464 r = amdgpu_ring_test_ib(ring, 1000 * 10);
1465 if (r) {
1466 DRM_DEV_ERROR(ring->adev->dev,
1467 "ring %s ib test failed (%d)\n",
1468 ring->name, r);
1469 return r;
1470 } else
1471 DRM_INFO("ring %s ib test pass\n", ring->name);
1472 }
1473
1474 return 0;
1475}
1476
1477int amdgpu_mes_self_test(struct amdgpu_device *adev)
1478{
1479 struct amdgpu_vm *vm = NULL;
1480 struct amdgpu_mes_ctx_data ctx_data = {0};
1481 struct amdgpu_ring *added_rings[AMDGPU_MES_CTX_MAX_RINGS] = { NULL };
1482 int gang_ids[3] = {0};
1483 int queue_types[][2] = { { AMDGPU_RING_TYPE_GFX, 1 },
1484 { AMDGPU_RING_TYPE_COMPUTE, 1 },
1485 { AMDGPU_RING_TYPE_SDMA, 1} };
1486 int i, r, pasid, k = 0;
1487
1488 pasid = amdgpu_pasid_alloc(16);
1489 if (pasid < 0) {
1490 dev_warn(adev->dev, "No more PASIDs available!");
1491 pasid = 0;
1492 }
1493
1494 vm = kzalloc(sizeof(*vm), GFP_KERNEL);
1495 if (!vm) {
1496 r = -ENOMEM;
1497 goto error_pasid;
1498 }
1499
1500 r = amdgpu_vm_init(adev, vm, -1);
1501 if (r) {
1502 DRM_ERROR("failed to initialize vm\n");
1503 goto error_pasid;
1504 }
1505
1506 r = amdgpu_mes_ctx_alloc_meta_data(adev, &ctx_data);
1507 if (r) {
1508 DRM_ERROR("failed to alloc ctx meta data\n");
1509 goto error_fini;
1510 }
1511
1512 ctx_data.meta_data_gpu_addr = AMDGPU_VA_RESERVED_BOTTOM;
1513 r = amdgpu_mes_ctx_map_meta_data(adev, vm, &ctx_data);
1514 if (r) {
1515 DRM_ERROR("failed to map ctx meta data\n");
1516 goto error_vm;
1517 }
1518
1519 r = amdgpu_mes_create_process(adev, pasid, vm);
1520 if (r) {
1521 DRM_ERROR("failed to create MES process\n");
1522 goto error_vm;
1523 }
1524
1525 for (i = 0; i < ARRAY_SIZE(queue_types); i++) {
1526 /* On GFX v10.3, fw hasn't supported to map sdma queue. */
1527 if (amdgpu_ip_version(adev, GC_HWIP, 0) >=
1528 IP_VERSION(10, 3, 0) &&
1529 amdgpu_ip_version(adev, GC_HWIP, 0) <
1530 IP_VERSION(11, 0, 0) &&
1531 queue_types[i][0] == AMDGPU_RING_TYPE_SDMA)
1532 continue;
1533
1534 r = amdgpu_mes_test_create_gang_and_queues(adev, pasid,
1535 &gang_ids[i],
1536 queue_types[i][0],
1537 queue_types[i][1],
1538 &added_rings[k],
1539 &ctx_data);
1540 if (r)
1541 goto error_queues;
1542
1543 k += queue_types[i][1];
1544 }
1545
1546 /* start ring test and ib test for MES queues */
1547 amdgpu_mes_test_queues(added_rings);
1548
1549error_queues:
1550 /* remove all queues */
1551 for (i = 0; i < ARRAY_SIZE(added_rings); i++) {
1552 if (!added_rings[i])
1553 continue;
1554 amdgpu_mes_remove_ring(adev, added_rings[i]);
1555 }
1556
1557 for (i = 0; i < ARRAY_SIZE(gang_ids); i++) {
1558 if (!gang_ids[i])
1559 continue;
1560 amdgpu_mes_remove_gang(adev, gang_ids[i]);
1561 }
1562
1563 amdgpu_mes_destroy_process(adev, pasid);
1564
1565error_vm:
1566 amdgpu_mes_ctx_unmap_meta_data(adev, &ctx_data);
1567
1568error_fini:
1569 amdgpu_vm_fini(adev, vm);
1570
1571error_pasid:
1572 if (pasid)
1573 amdgpu_pasid_free(pasid);
1574
1575 amdgpu_mes_ctx_free_meta_data(&ctx_data);
1576 kfree(vm);
1577 return 0;
1578}
1579
1580int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe)584int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe)
1581{585{
1582 const struct mes_firmware_header_v1_0 *mes_hdr;586 const struct mes_firmware_header_v1_0 *mes_hdr;
@@ -1705,7 +709,7 @@ int amdgpu_mes_update_enforce_isolation(struct amdgpu_device *adev) · 122 unmodified lines
1705 if (adev->enable_mes && adev->gfx.enable_cleaner_shader) {709 if (adev->enable_mes && adev->gfx.enable_cleaner_shader) {
1706 mutex_lock(&adev->enforce_isolation_mutex);710 mutex_lock(&adev->enforce_isolation_mutex);
1707 for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) {711 for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) {
1708 if (adev->enforce_isolation[i])712 if (adev->enforce_isolation[i] == AMDGPU_ENFORCE_ISOLATION_ENABLE)
1709 r |= amdgpu_mes_set_enforce_isolation(adev, i, true);713 r |= amdgpu_mes_set_enforce_isolation(adev, i, true);
1710 else714 else
1711 r |= amdgpu_mes_set_enforce_isolation(adev, i, false);715 r |= amdgpu_mes_set_enforce_isolation(adev, i, false);
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h+4−64
@@ -111,8 +111,8 @@ struct amdgpu_mes { · 110 unmodified lines
111 111
112 uint32_t vmid_mask_gfxhub;112 uint32_t vmid_mask_gfxhub;
113 uint32_t vmid_mask_mmhub;113 uint32_t vmid_mask_mmhub;
114 uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES];
115 uint32_t gfx_hqd_mask[AMDGPU_MES_MAX_GFX_PIPES];114 uint32_t gfx_hqd_mask[AMDGPU_MES_MAX_GFX_PIPES];
115 uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES];
116 uint32_t sdma_hqd_mask[AMDGPU_MES_MAX_SDMA_PIPES];116 uint32_t sdma_hqd_mask[AMDGPU_MES_MAX_SDMA_PIPES];
117 uint32_t aggregated_doorbells[AMDGPU_MES_PRIORITY_NUM_LEVELS];117 uint32_t aggregated_doorbells[AMDGPU_MES_PRIORITY_NUM_LEVELS];
118 uint32_t sch_ctx_offs[AMDGPU_MAX_MES_PIPES];118 uint32_t sch_ctx_offs[AMDGPU_MAX_MES_PIPES];
@@ -149,19 +149,6 @@ struct amdgpu_mes { · 30 unmodified lines
149 149
150};150};
151 151
152struct amdgpu_mes_process {
153 int pasid;
154 struct amdgpu_vm *vm;
155 uint64_t pd_gpu_addr;
156 struct amdgpu_bo *proc_ctx_bo;
157 uint64_t proc_ctx_gpu_addr;
158 void *proc_ctx_cpu_ptr;
159 uint64_t process_quantum;
160 struct list_head gang_list;
161 uint32_t doorbell_index;
162 struct mutex doorbell_lock;
163};
164
165struct amdgpu_mes_gang {152struct amdgpu_mes_gang {
166 int gang_id;153 int gang_id;
167 int priority;154 int priority;
@@ -248,18 +235,6 @@ struct mes_remove_queue_input { · 80 unmodified lines
248 uint64_t gang_context_addr;235 uint64_t gang_context_addr;
249};236};
250 237
251struct mes_reset_queue_input {
252 uint32_t doorbell_offset;
253 uint64_t gang_context_addr;
254 bool use_mmio;
255 uint32_t queue_type;
256 uint32_t me_id;
257 uint32_t pipe_id;
258 uint32_t queue_id;
259 uint32_t xcc_id;
260 uint32_t vmid;
261};
262
263struct mes_map_legacy_queue_input {238struct mes_map_legacy_queue_input {
264 uint32_t queue_type;239 uint32_t queue_type;
265 uint32_t doorbell_offset;240 uint32_t doorbell_offset;
@@ -291,7 +266,7 @@ struct mes_resume_gang_input { · 25 unmodified lines
291 uint64_t gang_context_addr;266 uint64_t gang_context_addr;
292};267};
293 268
294struct mes_reset_legacy_queue_input {269struct mes_reset_queue_input {
295 uint32_t queue_type;270 uint32_t queue_type;
296 uint32_t doorbell_offset;271 uint32_t doorbell_offset;
297 bool use_mmio;272 bool use_mmio;
@@ -301,6 +276,8 @@ struct mes_reset_legacy_queue_input { · 3 unmodified lines
301 uint64_t mqd_addr;276 uint64_t mqd_addr;
302 uint64_t wptr_addr;277 uint64_t wptr_addr;
303 uint32_t vmid;278 uint32_t vmid;
279 bool legacy_gfx;
280 bool is_kq;
304};281};
305 282
306enum mes_misc_opcode {283enum mes_misc_opcode {
@@ -388,9 +365,6 @@ struct amdgpu_mes_funcs { · 81 unmodified lines
388 int (*misc_op)(struct amdgpu_mes *mes,365 int (*misc_op)(struct amdgpu_mes *mes,
389 struct mes_misc_op_input *input);366 struct mes_misc_op_input *input);
390 367
391 int (*reset_legacy_queue)(struct amdgpu_mes *mes,
392 struct mes_reset_legacy_queue_input *input);
393
394 int (*reset_hw_queue)(struct amdgpu_mes *mes,368 int (*reset_hw_queue)(struct amdgpu_mes *mes,
395 struct mes_reset_queue_input *input);369 struct mes_reset_queue_input *input);
396};370};
@@ -404,26 +378,9 @@ int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe); · 7 unmodified lines
404int amdgpu_mes_init(struct amdgpu_device *adev);378int amdgpu_mes_init(struct amdgpu_device *adev);
405void amdgpu_mes_fini(struct amdgpu_device *adev);379void amdgpu_mes_fini(struct amdgpu_device *adev);
406 380
407int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid,
408 struct amdgpu_vm *vm);
409void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid);
410
411int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid,
412 struct amdgpu_mes_gang_properties *gprops,
413 int *gang_id);
414int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id);
415
416int amdgpu_mes_suspend(struct amdgpu_device *adev);381int amdgpu_mes_suspend(struct amdgpu_device *adev);
417int amdgpu_mes_resume(struct amdgpu_device *adev);382int amdgpu_mes_resume(struct amdgpu_device *adev);
418 383
419int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id,
420 struct amdgpu_mes_queue_properties *qprops,
421 int *queue_id);
422int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id);
423int amdgpu_mes_reset_hw_queue(struct amdgpu_device *adev, int queue_id);
424int amdgpu_mes_reset_hw_queue_mmio(struct amdgpu_device *adev, int queue_type,
425 int me_id, int pipe_id, int queue_id, int vmid);
426
427int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev,384int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev,
428 struct amdgpu_ring *ring);385 struct amdgpu_ring *ring);
429int amdgpu_mes_unmap_legacy_queue(struct amdgpu_device *adev,386int amdgpu_mes_unmap_legacy_queue(struct amdgpu_device *adev,
@@ -451,27 +408,10 @@ int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, · 21 unmodified lines
451 bool trap_en);408 bool trap_en);
452int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev,409int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev,
453 uint64_t process_context_addr);410 uint64_t process_context_addr);
454int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id,
455 int queue_type, int idx,
456 struct amdgpu_mes_ctx_data *ctx_data,
457 struct amdgpu_ring **out);
458void amdgpu_mes_remove_ring(struct amdgpu_device *adev,
459 struct amdgpu_ring *ring);
460 411
461uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev,412uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev,
462 enum amdgpu_mes_priority_level prio);413 enum amdgpu_mes_priority_level prio);
463 414
464int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev,
465 struct amdgpu_mes_ctx_data *ctx_data);
466void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data);
467int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev,
468 struct amdgpu_vm *vm,
469 struct amdgpu_mes_ctx_data *ctx_data);
470int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev,
471 struct amdgpu_mes_ctx_data *ctx_data);
472
473int amdgpu_mes_self_test(struct amdgpu_device *adev);
474
475int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev);415int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev);
476 416
477/*417/*
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c+5−1
@@ -1644,7 +1644,11 @@ u64 amdgpu_bo_print_info(int id, struct amdgpu_bo *bo, struct seq_file *m) · 1643 unmodified lines
1644 amdgpu_bo_print_flag(m, bo, VRAM_CONTIGUOUS);1644 amdgpu_bo_print_flag(m, bo, VRAM_CONTIGUOUS);
1645 amdgpu_bo_print_flag(m, bo, VM_ALWAYS_VALID);1645 amdgpu_bo_print_flag(m, bo, VM_ALWAYS_VALID);
1646 amdgpu_bo_print_flag(m, bo, EXPLICIT_SYNC);1646 amdgpu_bo_print_flag(m, bo, EXPLICIT_SYNC);
1647 1647 /* Add the gem obj resv fence dump*/
1648 if (dma_resv_trylock(bo->tbo.base.resv)) {
1649 dma_resv_describe(bo->tbo.base.resv, m);
1650 dma_resv_unlock(bo->tbo.base.resv);
1651 }
1648 seq_puts(m, "\n");1652 seq_puts(m, "\n");
1649 1653
1650 return size;1654 return size;
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c+2−1
@@ -2214,7 +2214,8 @@ static int psp_securedisplay_initialize(struct psp_context *psp) · 2213 unmodified lines
2214 2214
2215 if (!psp->securedisplay_context.context.bin_desc.size_bytes ||2215 if (!psp->securedisplay_context.context.bin_desc.size_bytes ||
2216 !psp->securedisplay_context.context.bin_desc.start_addr) {2216 !psp->securedisplay_context.context.bin_desc.start_addr) {
2217 dev_info(psp->adev->dev, "SECUREDISPLAY: securedisplay ta ucode is not available\n");2217 dev_info(psp->adev->dev,
2218 "SECUREDISPLAY: optional securedisplay ta ucode is not available\n");
2218 return 0;2219 return 0;
2219 }2220 }
2220 2221
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h+10−0
@@ -107,6 +107,7 @@ enum psp_reg_prog_id { · 106 unmodified lines
107 PSP_REG_IH_RB_CNTL = 0, /* register IH_RB_CNTL */107 PSP_REG_IH_RB_CNTL = 0, /* register IH_RB_CNTL */
108 PSP_REG_IH_RB_CNTL_RING1 = 1, /* register IH_RB_CNTL_RING1 */108 PSP_REG_IH_RB_CNTL_RING1 = 1, /* register IH_RB_CNTL_RING1 */
109 PSP_REG_IH_RB_CNTL_RING2 = 2, /* register IH_RB_CNTL_RING2 */109 PSP_REG_IH_RB_CNTL_RING2 = 2, /* register IH_RB_CNTL_RING2 */
110 PSP_REG_MMHUB_L1_TLB_CNTL = 25,
110 PSP_REG_LAST111 PSP_REG_LAST
111};112};
112 113
@@ -142,6 +143,8 @@ struct psp_funcs { · 29 unmodified lines
142 bool (*get_ras_capability)(struct psp_context *psp);143 bool (*get_ras_capability)(struct psp_context *psp);
143 bool (*is_aux_sos_load_required)(struct psp_context *psp);144 bool (*is_aux_sos_load_required)(struct psp_context *psp);
144 bool (*is_reload_needed)(struct psp_context *psp);145 bool (*is_reload_needed)(struct psp_context *psp);
146 int (*reg_program_no_ring)(struct psp_context *psp, uint32_t val,
147 enum psp_reg_prog_id id);
145};148};
146 149
147struct ta_funcs {150struct ta_funcs {
@@ -475,6 +478,10 @@ struct amdgpu_psp_funcs { · 327 unmodified lines
475#define psp_is_aux_sos_load_required(psp) \478#define psp_is_aux_sos_load_required(psp) \
476 ((psp)->funcs->is_aux_sos_load_required ? (psp)->funcs->is_aux_sos_load_required((psp)) : 0)479 ((psp)->funcs->is_aux_sos_load_required ? (psp)->funcs->is_aux_sos_load_required((psp)) : 0)
477 480
481#define psp_reg_program_no_ring(psp, val, id) \
482 ((psp)->funcs->reg_program_no_ring ? \
483 (psp)->funcs->reg_program_no_ring((psp), val, id) : -EINVAL)
484
478extern const struct amd_ip_funcs psp_ip_funcs;485extern const struct amd_ip_funcs psp_ip_funcs;
479 486
480extern const struct amdgpu_ip_block_version psp_v3_1_ip_block;487extern const struct amdgpu_ip_block_version psp_v3_1_ip_block;
@@ -569,5 +576,8 @@ bool amdgpu_psp_get_ras_capability(struct psp_context *psp); · 88 unmodified lines
569int psp_config_sq_perfmon(struct psp_context *psp, uint32_t xcp_id,576int psp_config_sq_perfmon(struct psp_context *psp, uint32_t xcp_id,
570 bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable);577 bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable);
571bool amdgpu_psp_tos_reload_needed(struct amdgpu_device *adev);578bool amdgpu_psp_tos_reload_needed(struct amdgpu_device *adev);
579int amdgpu_psp_reg_program_no_ring(struct psp_context *psp, uint32_t val,
580 enum psp_reg_prog_id id);
581
572 582
573#endif583#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c+10−5
@@ -1498,6 +1498,9 @@ int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, · 1497 unmodified lines
1498 !amdgpu_ras_get_aca_debug_mode(adev))1498 !amdgpu_ras_get_aca_debug_mode(adev))
1499 return -EOPNOTSUPP;1499 return -EOPNOTSUPP;
1500 1500
1501 if (amdgpu_sriov_vf(adev))
1502 return -EOPNOTSUPP;
1503
1501 /* skip ras error reset in gpu reset */1504 /* skip ras error reset in gpu reset */
1502 if ((amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) &&1505 if ((amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) &&
1503 ((smu_funcs && smu_funcs->set_debug_mode) ||1506 ((smu_funcs && smu_funcs->set_debug_mode) ||
@@ -2161,7 +2164,7 @@ void amdgpu_ras_interrupt_fatal_error_handler(struct amdgpu_device *adev) · 657 unmodified lines
2161 /* Fatal error events are handled on host side */2164 /* Fatal error events are handled on host side */
2162 if (amdgpu_sriov_vf(adev))2165 if (amdgpu_sriov_vf(adev))
2163 return;2166 return;
2164 /**2167 /*
2165 * If the current interrupt is caused by a non-fatal RAS error, skip2168 * If the current interrupt is caused by a non-fatal RAS error, skip
2166 * check for fatal error. For fatal errors, FED status of all devices2169 * check for fatal error. For fatal errors, FED status of all devices
2167 * in XGMI hive gets set when the first device gets fatal error2170 * in XGMI hive gets set when the first device gets fatal error
@@ -3793,10 +3796,12 @@ init_ras_enabled_flag: · 1625 unmodified lines
3793 adev->ras_hw_enabled & amdgpu_ras_mask;3796 adev->ras_hw_enabled & amdgpu_ras_mask;
3794 3797
3795 /* aca is disabled by default except for psp v13_0_6/v13_0_12/v13_0_14 */3798 /* aca is disabled by default except for psp v13_0_6/v13_0_12/v13_0_14 */
3796 adev->aca.is_enabled =3799 if (!amdgpu_sriov_vf(adev)) {
3797 (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||3800 adev->aca.is_enabled =
3798 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||3801 (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||
3799 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14));3802 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||
3803 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14));
3804 }
3800 3805
3801 /* bad page feature is not applicable to specific app platform */3806 /* bad page feature is not applicable to specific app platform */
3802 if (adev->gmc.is_app_apu &&3807 if (adev->gmc.is_app_apu &&
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c+27−8
@@ -418,6 +418,7 @@ static void amdgpu_ras_set_eeprom_table_version(struct amdgpu_ras_eeprom_control · 417 unmodified lines
418 hdr->version = RAS_TABLE_VER_V2_1;418 hdr->version = RAS_TABLE_VER_V2_1;
419 return;419 return;
420 case IP_VERSION(12, 0, 0):420 case IP_VERSION(12, 0, 0):
421 case IP_VERSION(12, 5, 0):
421 hdr->version = RAS_TABLE_VER_V3;422 hdr->version = RAS_TABLE_VER_V3;
422 return;423 return;
423 default:424 default:
@@ -1392,17 +1393,39 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 968 unmodified lines
1392 1393
1393 __decode_table_header_from_buf(hdr, buf);1394 __decode_table_header_from_buf(hdr, buf);
1394 1395
1395 if (hdr->version >= RAS_TABLE_VER_V2_1) {1396 if (hdr->header != RAS_TABLE_HDR_VAL &&
1397 hdr->header != RAS_TABLE_HDR_BAD) {
1398 dev_info(adev->dev, "Creating a new EEPROM table");
1399 return amdgpu_ras_eeprom_reset_table(control);
1400 }
1401
1402 switch (hdr->version) {
1403 case RAS_TABLE_VER_V2_1:
1404 case RAS_TABLE_VER_V3:
1396 control->ras_num_recs = RAS_NUM_RECS_V2_1(hdr);1405 control->ras_num_recs = RAS_NUM_RECS_V2_1(hdr);
1397 control->ras_record_offset = RAS_RECORD_START_V2_1;1406 control->ras_record_offset = RAS_RECORD_START_V2_1;
1398 control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1;1407 control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1;
1399 } else {1408 break;
1409 case RAS_TABLE_VER_V1:
1400 control->ras_num_recs = RAS_NUM_RECS(hdr);1410 control->ras_num_recs = RAS_NUM_RECS(hdr);
1401 control->ras_record_offset = RAS_RECORD_START;1411 control->ras_record_offset = RAS_RECORD_START;
1402 control->ras_max_record_count = RAS_MAX_RECORD_COUNT;1412 control->ras_max_record_count = RAS_MAX_RECORD_COUNT;
1413 break;
1414 default:
1415 dev_err(adev->dev,
1416 "RAS header invalid, unsupported version: %u",
1417 hdr->version);
1418 return -EINVAL;
1403 }1419 }
1404 control->ras_fri = RAS_OFFSET_TO_INDEX(control, hdr->first_rec_offset);
1405 1420
1421 if (control->ras_num_recs > control->ras_max_record_count) {
1422 dev_err(adev->dev,
1423 "RAS header invalid, records in header: %u max allowed :%u",
1424 control->ras_num_recs, control->ras_max_record_count);
1425 return -EINVAL;
1426 }
1427
1428 control->ras_fri = RAS_OFFSET_TO_INDEX(control, hdr->first_rec_offset);
1406 control->ras_num_mca_recs = 0;1429 control->ras_num_mca_recs = 0;
1407 control->ras_num_pa_recs = 0;1430 control->ras_num_pa_recs = 0;
1408 return 0;1431 return 0;
@@ -1413,7 +1436,7 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 4 unmodified lines
1413 struct amdgpu_device *adev = to_amdgpu_device(control);1436 struct amdgpu_device *adev = to_amdgpu_device(control);
1414 struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr;1437 struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr;
1415 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);1438 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
1416 int res;1439 int res = 0;
1417 1440
1418 if (!__is_ras_eeprom_supported(adev))1441 if (!__is_ras_eeprom_supported(adev))
1419 return 0;1442 return 0;
@@ -1494,10 +1517,6 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 74 unmodified lines
1494 "User defined threshold is set, runtime service will be halt when threshold is reached\n");1517 "User defined threshold is set, runtime service will be halt when threshold is reached\n");
1495 }1518 }
1496 }1519 }
1497 } else {
1498 DRM_INFO("Creating a new EEPROM table");
1499
1500 res = amdgpu_ras_eeprom_reset_table(control);
1501 }1520 }
1502 1521
1503 return res < 0 ? res : 0;1522 return res < 0 ? res : 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c+38−74
@@ -187,14 +187,10 @@ void amdgpu_ring_undo(struct amdgpu_ring *ring) · 186 unmodified lines
187}187}
188 188
189#define amdgpu_ring_get_gpu_addr(ring, offset) \189#define amdgpu_ring_get_gpu_addr(ring, offset) \
190 (ring->is_mes_queue ? \190 (ring->adev->wb.gpu_addr + offset * 4)
191 (ring->mes_ctx->meta_data_gpu_addr + offset) : \
192 (ring->adev->wb.gpu_addr + offset * 4))
193 191
194#define amdgpu_ring_get_cpu_addr(ring, offset) \192#define amdgpu_ring_get_cpu_addr(ring, offset) \
195 (ring->is_mes_queue ? \193 (&ring->adev->wb.wb[offset])
196 (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \
197 (&ring->adev->wb.wb[offset]))
198 194
199/**195/**
200 * amdgpu_ring_init - init driver ring struct.196 * amdgpu_ring_init - init driver ring struct.
@@ -243,57 +239,42 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · 42 unmodified lines
243 ring->sched_score = sched_score;239 ring->sched_score = sched_score;
244 ring->vmid_wait = dma_fence_get_stub();240 ring->vmid_wait = dma_fence_get_stub();
245 241
246 if (!ring->is_mes_queue) {242 ring->idx = adev->num_rings++;
247 ring->idx = adev->num_rings++;243 adev->rings[ring->idx] = ring;
248 adev->rings[ring->idx] = ring;
249 }
250 244
251 r = amdgpu_fence_driver_init_ring(ring);245 r = amdgpu_fence_driver_init_ring(ring);
252 if (r)246 if (r)
253 return r;247 return r;
254 }248 }
255 249
256 if (ring->is_mes_queue) {250 r = amdgpu_device_wb_get(adev, &ring->rptr_offs);
257 ring->rptr_offs = amdgpu_mes_ctx_get_offs(ring,251 if (r) {
258 AMDGPU_MES_CTX_RPTR_OFFS);252 dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r);
259 ring->wptr_offs = amdgpu_mes_ctx_get_offs(ring,253 return r;
260 AMDGPU_MES_CTX_WPTR_OFFS);254 }
261 ring->fence_offs = amdgpu_mes_ctx_get_offs(ring,
262 AMDGPU_MES_CTX_FENCE_OFFS);
263 ring->trail_fence_offs = amdgpu_mes_ctx_get_offs(ring,
264 AMDGPU_MES_CTX_TRAIL_FENCE_OFFS);
265 ring->cond_exe_offs = amdgpu_mes_ctx_get_offs(ring,
266 AMDGPU_MES_CTX_COND_EXE_OFFS);
267 } else {
268 r = amdgpu_device_wb_get(adev, &ring->rptr_offs);
269 if (r) {
270 dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r);
271 return r;
272 }
273 255
274 r = amdgpu_device_wb_get(adev, &ring->wptr_offs);256 r = amdgpu_device_wb_get(adev, &ring->wptr_offs);
275 if (r) {257 if (r) {
276 dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r);258 dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r);
277 return r;259 return r;
278 }260 }
279 261
280 r = amdgpu_device_wb_get(adev, &ring->fence_offs);262 r = amdgpu_device_wb_get(adev, &ring->fence_offs);
281 if (r) {263 if (r) {
282 dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r);264 dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r);
283 return r;265 return r;
284 }266 }
285 267
286 r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs);268 r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs);
287 if (r) {269 if (r) {
288 dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r);270 dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r);
289 return r;271 return r;
290 }272 }
291 273
292 r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs);274 r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs);
293 if (r) {275 if (r) {
294 dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r);276 dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r);
295 return r;277 return r;
296 }
297 }278 }
298 279
299 ring->fence_gpu_addr =280 ring->fence_gpu_addr =
@@ -353,18 +334,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · 53 unmodified lines
353 ring->cached_rptr = 0;334 ring->cached_rptr = 0;
354 335
355 /* Allocate ring buffer */336 /* Allocate ring buffer */
356 if (ring->is_mes_queue) {337 if (ring->ring_obj == NULL) {
357 int offset = 0;
358
359 BUG_ON(ring->ring_size > PAGE_SIZE*4);
360
361 offset = amdgpu_mes_ctx_get_offs(ring,
362 AMDGPU_MES_CTX_RING_OFFS);
363 ring->gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
364 ring->ring = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
365 amdgpu_ring_clear_ring(ring);
366
367 } else if (ring->ring_obj == NULL) {
368 r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,338 r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE,
369 AMDGPU_GEM_DOMAIN_GTT,339 AMDGPU_GEM_DOMAIN_GTT,
370 &ring->ring_obj,340 &ring->ring_obj,
@@ -401,32 +371,26 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring) · 30 unmodified lines
401{371{
402 372
403 /* Not to finish a ring which is not initialized */373 /* Not to finish a ring which is not initialized */
404 if (!(ring->adev) ||374 if (!(ring->adev) || !(ring->adev->rings[ring->idx]))
405 (!ring->is_mes_queue && !(ring->adev->rings[ring->idx])))
406 return;375 return;
407 376
408 ring->sched.ready = false;377 ring->sched.ready = false;
409 378
410 if (!ring->is_mes_queue) {379 amdgpu_device_wb_free(ring->adev, ring->rptr_offs);
411 amdgpu_device_wb_free(ring->adev, ring->rptr_offs);380 amdgpu_device_wb_free(ring->adev, ring->wptr_offs);
412 amdgpu_device_wb_free(ring->adev, ring->wptr_offs);
413 381
414 amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs);382 amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs);
415 amdgpu_device_wb_free(ring->adev, ring->fence_offs);383 amdgpu_device_wb_free(ring->adev, ring->fence_offs);
416 384
417 amdgpu_bo_free_kernel(&ring->ring_obj,385 amdgpu_bo_free_kernel(&ring->ring_obj,
418 &ring->gpu_addr,386 &ring->gpu_addr,
419 (void **)&ring->ring);387 (void **)&ring->ring);
420 } else {
421 kfree(ring->fence_drv.fences);
422 }
423 388
424 dma_fence_put(ring->vmid_wait);389 dma_fence_put(ring->vmid_wait);
425 ring->vmid_wait = NULL;390 ring->vmid_wait = NULL;
426 ring->me = 0;391 ring->me = 0;
427 392
428 if (!ring->is_mes_queue)393 ring->adev->rings[ring->idx] = NULL;
429 ring->adev->rings[ring->idx] = NULL;
430}394}
431 395
432/**396/**
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h+75−15
@@ -164,8 +164,24 @@ void amdgpu_fence_update_start_timestamp(struct amdgpu_ring *ring, uint32_t seq, · 163 unmodified lines
164 164
165/* provided by hw blocks that expose a ring buffer for commands */165/* provided by hw blocks that expose a ring buffer for commands */
166struct amdgpu_ring_funcs {166struct amdgpu_ring_funcs {
167 /**
168 * @type:
169 *
170 * GFX, Compute, SDMA, UVD, VCE, VCN, VPE, KIQ, MES, UMSCH, and CPER
171 * use ring buffers. The type field just identifies which component the
172 * ring buffer is associated with.
173 */
167 enum amdgpu_ring_type type;174 enum amdgpu_ring_type type;
168 uint32_t align_mask;175 uint32_t align_mask;
176
177 /**
178 * @nop:
179 *
180 * Every block in the amdgpu has no-op instructions (e.g., GFX 10
181 * uses PACKET3(PACKET3_NOP, 0x3FFF), VCN 5 uses VCN_ENC_CMD_NO_OP,
182 * etc). This field receives the specific no-op for the component
183 * that initializes the ring.
184 */
169 u32 nop;185 u32 nop;
170 bool support_64bit_ptrs;186 bool support_64bit_ptrs;
171 bool no_user_fence;187 bool no_user_fence;
@@ -241,6 +257,9 @@ struct amdgpu_ring_funcs { · 69 unmodified lines
241 bool (*is_guilty)(struct amdgpu_ring *ring);257 bool (*is_guilty)(struct amdgpu_ring *ring);
242};258};
243 259
260/**
261 * amdgpu_ring - Holds ring information
262 */
244struct amdgpu_ring {263struct amdgpu_ring {
245 struct amdgpu_device *adev;264 struct amdgpu_device *adev;
246 const struct amdgpu_ring_funcs *funcs;265 const struct amdgpu_ring_funcs *funcs;
@@ -252,13 +271,61 @@ struct amdgpu_ring { · 5 unmodified lines
252 unsigned rptr_offs;271 unsigned rptr_offs;
253 u64 rptr_gpu_addr;272 u64 rptr_gpu_addr;
254 volatile u32 *rptr_cpu_addr;273 volatile u32 *rptr_cpu_addr;
274
275 /**
276 * @wptr:
277 *
278 * This is part of the Ring buffer implementation and represents the
279 * write pointer. The wptr determines where the host has written.
280 */
255 u64 wptr;281 u64 wptr;
282
283 /**
284 * @wptr_old:
285 *
286 * Before update wptr with the new value, usually the old value is
287 * stored in the wptr_old.
288 */
256 u64 wptr_old;289 u64 wptr_old;
257 unsigned ring_size;290 unsigned ring_size;
291
292 /**
293 * @max_dw:
294 *
295 * Maximum number of DWords for ring allocation. This information is
296 * provided at the ring initialization time, and each IP block can
297 * specify a specific value. Check places that invoke
298 * amdgpu_ring_init() to see the maximum size per block.
299 */
258 unsigned max_dw;300 unsigned max_dw;
301
302 /**
303 * @count_dw:
304 *
305 * This value starts with the maximum amount of DWords supported by the
306 * ring. This value is updated based on the ring manipulation.
307 */
259 int count_dw;308 int count_dw;
260 uint64_t gpu_addr;309 uint64_t gpu_addr;
310
311 /**
312 * @ptr_mask:
313 *
314 * Some IPs provide support for 64-bit pointers and others for 32-bit
315 * only; this behavior is component-specific and defined by the field
316 * support_64bit_ptr. If the IP block supports 64-bits, the mask
317 * 0xffffffffffffffff is set; otherwise, this value assumes buf_mask.
318 * Notice that this field is used to keep wptr under a valid range.
319 */
261 uint64_t ptr_mask;320 uint64_t ptr_mask;
321
322 /**
323 * @buf_mask:
324 *
325 * Buffer mask is a value used to keep wptr count under its
326 * thresholding. Buffer mask initialized during the ring buffer
327 * initialization time, and it is defined as (ring_size / 4) -1.
328 */
262 uint32_t buf_mask;329 uint32_t buf_mask;
263 u32 idx;330 u32 idx;
264 u32 xcc_id;331 u32 xcc_id;
@@ -276,6 +343,13 @@ struct amdgpu_ring { · 11 unmodified lines
276 bool use_pollmem;343 bool use_pollmem;
277 unsigned wptr_offs;344 unsigned wptr_offs;
278 u64 wptr_gpu_addr;345 u64 wptr_gpu_addr;
346
347 /**
348 * @wptr_cpu_addr:
349 *
350 * This is the CPU address pointer in the writeback slot. This is used
351 * to commit changes to the GPU.
352 */
279 volatile u32 *wptr_cpu_addr;353 volatile u32 *wptr_cpu_addr;
280 unsigned fence_offs;354 unsigned fence_offs;
281 u64 fence_gpu_addr;355 u64 fence_gpu_addr;
@@ -297,20 +371,15 @@ struct amdgpu_ring { · 15 unmodified lines
297 struct dma_fence *vmid_wait;371 struct dma_fence *vmid_wait;
298 bool has_compute_vm_bug;372 bool has_compute_vm_bug;
299 bool no_scheduler;373 bool no_scheduler;
374 bool no_user_submission;
300 int hw_prio;375 int hw_prio;
301 unsigned num_hw_submission;376 unsigned num_hw_submission;
302 atomic_t *sched_score;377 atomic_t *sched_score;
303 378
304 /* used for mes */
305 bool is_mes_queue;
306 uint32_t hw_queue_id;
307 struct amdgpu_mes_ctx_data *mes_ctx;
308
309 bool is_sw_ring;379 bool is_sw_ring;
310 unsigned int entry_index;380 unsigned int entry_index;
311 /* store the cached rptr to restore after reset */381 /* store the cached rptr to restore after reset */
312 uint64_t cached_rptr;382 uint64_t cached_rptr;
313
314};383};
315 384
316#define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))385#define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib)))
@@ -435,15 +504,6 @@ static inline void amdgpu_ring_patch_cond_exec(struct amdgpu_ring *ring, · 118 unmodified lines
435 ring->ring[offset] = cur - offset;504 ring->ring[offset] = cur - offset;
436}505}
437 506
438#define amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset) \
439 (ring->is_mes_queue && ring->mes_ctx ? \
440 (ring->mes_ctx->meta_data_gpu_addr + offset) : 0)
441
442#define amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset) \
443 (ring->is_mes_queue && ring->mes_ctx ? \
444 (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \
445 NULL)
446
447int amdgpu_ring_test_helper(struct amdgpu_ring *ring);507int amdgpu_ring_test_helper(struct amdgpu_ring *ring);
448 508
449void amdgpu_debugfs_ring_init(struct amdgpu_device *adev,509void amdgpu_debugfs_ring_init(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h+14−0
@@ -237,6 +237,20 @@ struct amdgpu_rlc_funcs { · 236 unmodified lines
237 void (*unset_safe_mode)(struct amdgpu_device *adev, int xcc_id);237 void (*unset_safe_mode)(struct amdgpu_device *adev, int xcc_id);
238 int (*init)(struct amdgpu_device *adev);238 int (*init)(struct amdgpu_device *adev);
239 u32 (*get_csb_size)(struct amdgpu_device *adev);239 u32 (*get_csb_size)(struct amdgpu_device *adev);
240
241 /**
242 * @get_csb_buffer: Get the clear state to be put into the hardware.
243 *
244 * The parameter adev is used to get the CS data and other gfx info,
245 * and buffer is the RLC CS pointer
246 *
247 * Sometimes, the user space puts a request to clear the state in the
248 * command buffer; this function provides the clear state that gets put
249 * into the hardware. Note that the driver programs Clear State
250 * Indirect Buffer (CSB) explicitly when it sets up the kernel rings,
251 * and it also provides a pointer to it which is used by the firmware
252 * to load the clear state in some cases.
253 */
240 void (*get_csb_buffer)(struct amdgpu_device *adev, volatile u32 *buffer);254 void (*get_csb_buffer)(struct amdgpu_device *adev, volatile u32 *buffer);
241 int (*get_cp_table_num)(struct amdgpu_device *adev);255 int (*get_cp_table_num)(struct amdgpu_device *adev);
242 int (*resume)(struct amdgpu_device *adev);256 int (*resume)(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c+43−65
@@ -26,6 +26,8 @@ · 25 unmodified lines
26#include "amdgpu_sdma.h"26#include "amdgpu_sdma.h"
27#include "amdgpu_ras.h"27#include "amdgpu_ras.h"
28#include "amdgpu_reset.h"28#include "amdgpu_reset.h"
29#include "gc/gc_10_1_0_offset.h"
30#include "gc/gc_10_3_0_sh_mask.h"
29 31
30#define AMDGPU_CSA_SDMA_SIZE 6432#define AMDGPU_CSA_SDMA_SIZE 64
31/* SDMA CSA reside in the 3rd page of CSA */33/* SDMA CSA reside in the 3rd page of CSA */
@@ -76,22 +78,14 @@ uint64_t amdgpu_sdma_get_csa_mc_addr(struct amdgpu_ring *ring, · 44 unmodified lines
76 if (amdgpu_sriov_vf(adev) || vmid == 0 || !adev->gfx.mcbp)78 if (amdgpu_sriov_vf(adev) || vmid == 0 || !adev->gfx.mcbp)
77 return 0;79 return 0;
78 80
79 if (ring->is_mes_queue) {81 r = amdgpu_sdma_get_index_from_ring(ring, &index);
80 uint32_t offset = 0;
81 82
82 offset = offsetof(struct amdgpu_mes_ctx_meta_data,83 if (r || index > 31)
83 sdma[ring->idx].sdma_meta_data);84 csa_mc_addr = 0;
84 csa_mc_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);85 else
85 } else {86 csa_mc_addr = amdgpu_csa_vaddr(adev) +
86 r = amdgpu_sdma_get_index_from_ring(ring, &index);87 AMDGPU_CSA_SDMA_OFFSET +
87 88 index * AMDGPU_CSA_SDMA_SIZE;
88 if (r || index > 31)
89 csa_mc_addr = 0;
90 else
91 csa_mc_addr = amdgpu_csa_vaddr(adev) +
92 AMDGPU_CSA_SDMA_OFFSET +
93 index * AMDGPU_CSA_SDMA_SIZE;
94 }
95 89
96 return csa_mc_addr;90 return csa_mc_addr;
97}91}
@@ -537,28 +531,38 @@ bool amdgpu_sdma_is_shared_inv_eng(struct amdgpu_device *adev, struct amdgpu_rin · 439 unmodified lines
537 return false;531 return false;
538}532}
539 533
540/**534static int amdgpu_sdma_soft_reset(struct amdgpu_device *adev, u32 instance_id)
541 * amdgpu_sdma_register_on_reset_callbacks - Register SDMA reset callbacks
542 * @funcs: Pointer to the callback structure containing pre_reset and post_reset functions
543 *
544 * This function allows KFD and AMDGPU to register their own callbacks for handling
545 * pre-reset and post-reset operations for engine reset. These are needed because engine
546 * reset will stop all queues on that engine.
547 */
548void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct sdma_on_reset_funcs *funcs)
549{535{
550 if (!funcs)536 struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];
551 return;537 int r = -EOPNOTSUPP;
552 538
553 /* Ensure the reset_callback_list is initialized */539 switch (amdgpu_ip_version(adev, SDMA0_HWIP, 0)) {
554 if (!adev->sdma.reset_callback_list.next) {540 case IP_VERSION(4, 4, 2):
555 INIT_LIST_HEAD(&adev->sdma.reset_callback_list);541 case IP_VERSION(4, 4, 4):
542 case IP_VERSION(4, 4, 5):
543 /* For SDMA 4.x, use the existing DPM interface for backward compatibility */
544 r = amdgpu_dpm_reset_sdma(adev, 1 << instance_id);
545 break;
546 case IP_VERSION(5, 0, 0):
547 case IP_VERSION(5, 0, 1):
548 case IP_VERSION(5, 0, 2):
549 case IP_VERSION(5, 0, 5):
550 case IP_VERSION(5, 2, 0):
551 case IP_VERSION(5, 2, 2):
552 case IP_VERSION(5, 2, 4):
553 case IP_VERSION(5, 2, 5):
554 case IP_VERSION(5, 2, 6):
555 case IP_VERSION(5, 2, 3):
556 case IP_VERSION(5, 2, 1):
557 case IP_VERSION(5, 2, 7):
558 if (sdma_instance->funcs->soft_reset_kernel_queue)
559 r = sdma_instance->funcs->soft_reset_kernel_queue(adev, instance_id);
560 break;
561 default:
562 break;
556 }563 }
557 /* Initialize the list node in the callback structure */
558 INIT_LIST_HEAD(&funcs->list);
559 564
560 /* Add the callback structure to the global list */565 return r;
561 list_add_tail(&funcs->list, &adev->sdma.reset_callback_list);
562}566}
563 567
564/**568/**
@@ -566,16 +570,10 @@ void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct · 1 unmodified line
566 * @adev: Pointer to the AMDGPU device570 * @adev: Pointer to the AMDGPU device
567 * @instance_id: ID of the SDMA engine instance to reset571 * @instance_id: ID of the SDMA engine instance to reset
568 *572 *
569 * This function performs the following steps:
570 * 1. Calls all registered pre_reset callbacks to allow KFD and AMDGPU to save their state.
571 * 2. Resets the specified SDMA engine instance.
572 * 3. Calls all registered post_reset callbacks to allow KFD and AMDGPU to restore their state.
573 *
574 * Returns: 0 on success, or a negative error code on failure.573 * Returns: 0 on success, or a negative error code on failure.
575 */574 */
576int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id)575int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id)
577{576{
578 struct sdma_on_reset_funcs *funcs;
579 int ret = 0;577 int ret = 0;
580 struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];578 struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];
581 struct amdgpu_ring *gfx_ring = &sdma_instance->ring;579 struct amdgpu_ring *gfx_ring = &sdma_instance->ring;
@@ -597,38 +595,18 @@ int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id) · 15 unmodified lines
597 page_sched_stopped = true;595 page_sched_stopped = true;
598 }596 }
599 597
600 /* Invoke all registered pre_reset callbacks */598 if (sdma_instance->funcs->stop_kernel_queue)
601 list_for_each_entry(funcs, &adev->sdma.reset_callback_list, list) {599 sdma_instance->funcs->stop_kernel_queue(gfx_ring);
602 if (funcs->pre_reset) {
603 ret = funcs->pre_reset(adev, instance_id);
604 if (ret) {
605 dev_err(adev->dev,
606 "beforeReset callback failed for instance %u: %d\n",
607 instance_id, ret);
608 goto exit;
609 }
610 }
611 }
612 600
613 /* Perform the SDMA reset for the specified instance */601 /* Perform the SDMA reset for the specified instance */
614 ret = amdgpu_dpm_reset_sdma(adev, 1 << instance_id);602 ret = amdgpu_sdma_soft_reset(adev, instance_id);
615 if (ret) {603 if (ret) {
616 dev_err(adev->dev, "Failed to reset SDMA instance %u\n", instance_id);604 dev_err(adev->dev, "Failed to reset SDMA instance %u\n", instance_id);
617 goto exit;605 goto exit;
618 }606 }
619 607
620 /* Invoke all registered post_reset callbacks */608 if (sdma_instance->funcs->start_kernel_queue)
621 list_for_each_entry(funcs, &adev->sdma.reset_callback_list, list) {609 sdma_instance->funcs->start_kernel_queue(gfx_ring);
622 if (funcs->post_reset) {
623 ret = funcs->post_reset(adev, instance_id);
624 if (ret) {
625 dev_err(adev->dev,
626 "afterReset callback failed for instance %u: %d\n",
627 instance_id, ret);
628 goto exit;
629 }
630 }
631 }
632 610
633exit:611exit:
634 /* Restart the scheduler's work queue for the GFX and page rings612 /* Restart the scheduler's work queue for the GFX and page rings
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h+9−9
@@ -50,6 +50,12 @@ enum amdgpu_sdma_irq { · 49 unmodified lines
50 50
51#define NUM_SDMA(x) hweight32(x)51#define NUM_SDMA(x) hweight32(x)
52 52
53struct amdgpu_sdma_funcs {
54 int (*stop_kernel_queue)(struct amdgpu_ring *ring);
55 int (*start_kernel_queue)(struct amdgpu_ring *ring);
56 int (*soft_reset_kernel_queue)(struct amdgpu_device *adev, u32 instance_id);
57};
58
53struct amdgpu_sdma_instance {59struct amdgpu_sdma_instance {
54 /* SDMA firmware */60 /* SDMA firmware */
55 const struct firmware *fw;61 const struct firmware *fw;
@@ -68,7 +74,7 @@ struct amdgpu_sdma_instance { · 12 unmodified lines
68 /* track guilty state of GFX and PAGE queues */74 /* track guilty state of GFX and PAGE queues */
69 bool gfx_guilty;75 bool gfx_guilty;
70 bool page_guilty;76 bool page_guilty;
71 77 const struct amdgpu_sdma_funcs *funcs;
72};78};
73 79
74enum amdgpu_sdma_ras_memory_id {80enum amdgpu_sdma_ras_memory_id {
@@ -103,13 +109,6 @@ struct amdgpu_sdma_ras { · 28 unmodified lines
103 struct amdgpu_ras_block_object ras_block;109 struct amdgpu_ras_block_object ras_block;
104};110};
105 111
106struct sdma_on_reset_funcs {
107 int (*pre_reset)(struct amdgpu_device *adev, uint32_t instance_id);
108 int (*post_reset)(struct amdgpu_device *adev, uint32_t instance_id);
109 /* Linked list node to store this structure in a list; */
110 struct list_head list;
111};
112
113struct amdgpu_sdma {112struct amdgpu_sdma {
114 struct amdgpu_sdma_instance instance[AMDGPU_MAX_SDMA_INSTANCES];113 struct amdgpu_sdma_instance instance[AMDGPU_MAX_SDMA_INSTANCES];
115 struct amdgpu_irq_src trap_irq;114 struct amdgpu_irq_src trap_irq;
@@ -131,6 +130,8 @@ struct amdgpu_sdma { · 15 unmodified lines
131 uint32_t *ip_dump;130 uint32_t *ip_dump;
132 uint32_t supported_reset;131 uint32_t supported_reset;
133 struct list_head reset_callback_list;132 struct list_head reset_callback_list;
133 bool no_user_submission;
134 bool disable_uq;
134};135};
135 136
136/*137/*
@@ -170,7 +171,6 @@ struct amdgpu_buffer_funcs { · 33 unmodified lines
170 uint32_t byte_count);171 uint32_t byte_count);
171};172};
172 173
173void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct sdma_on_reset_funcs *funcs);
174int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id);174int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id);
175 175
176#define amdgpu_emit_copy_buffer(adev, ib, s, d, b, t) (adev)->mman.buffer_funcs->emit_copy_buffer((ib), (s), (d), (b), (t))176#define amdgpu_emit_copy_buffer(adev, ib, s, d, b, t) (adev)->mman.buffer_funcs->emit_copy_buffer((ib), (s), (d), (b), (t))
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.c+19−6
@@ -45,7 +45,11 @@ · 44 unmodified lines
45 */45 */
46static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev)46static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev)
47{47{
48 return AMDGPU_VA_RESERVED_SEQ64_START(adev);48 u64 addr = AMDGPU_VA_RESERVED_SEQ64_START(adev);
49
50 addr = amdgpu_gmc_sign_extend(addr);
51
52 return addr;
49}53}
50 54
51/**55/**
@@ -63,9 +67,9 @@ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) · 11 unmodified lines
63int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,67int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,
64 struct amdgpu_bo_va **bo_va)68 struct amdgpu_bo_va **bo_va)
65{69{
70 u64 seq64_addr, va_flags;
66 struct amdgpu_bo *bo;71 struct amdgpu_bo *bo;
67 struct drm_exec exec;72 struct drm_exec exec;
68 u64 seq64_addr;
69 int r;73 int r;
70 74
71 bo = adev->seq64.sbo;75 bo = adev->seq64.sbo;
@@ -88,9 +92,11 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 16 unmodified lines
88 goto error;92 goto error;
89 }93 }
90 94
91 seq64_addr = amdgpu_seq64_get_va_base(adev);95 seq64_addr = amdgpu_seq64_get_va_base(adev) & AMDGPU_GMC_HOLE_MASK;
96
97 va_flags = amdgpu_gem_va_map_flags(adev, AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_MTYPE_UC);
92 r = amdgpu_vm_bo_map(adev, *bo_va, seq64_addr, 0, AMDGPU_VA_RESERVED_SEQ64_SIZE,98 r = amdgpu_vm_bo_map(adev, *bo_va, seq64_addr, 0, AMDGPU_VA_RESERVED_SEQ64_SIZE,
93 AMDGPU_PTE_READABLE);99 va_flags);
94 if (r) {100 if (r) {
95 DRM_ERROR("failed to do bo_map on userq sem, err=%d\n", r);101 DRM_ERROR("failed to do bo_map on userq sem, err=%d\n", r);
96 amdgpu_vm_bo_del(adev, *bo_va);102 amdgpu_vm_bo_del(adev, *bo_va);
@@ -156,6 +162,7 @@ error: · 59 unmodified lines
156 *162 *
157 * @adev: amdgpu_device pointer163 * @adev: amdgpu_device pointer
158 * @va: VA to access the seq in process address space164 * @va: VA to access the seq in process address space
165 * @gpu_addr: GPU address to access the seq
159 * @cpu_addr: CPU address to access the seq166 * @cpu_addr: CPU address to access the seq
160 *167 *
161 * Alloc a 64 bit memory from seq64 pool.168 * Alloc a 64 bit memory from seq64 pool.
@@ -163,7 +170,8 @@ error: · 1 unmodified line
163 * Returns:170 * Returns:
164 * 0 on success or a negative error code on failure171 * 0 on success or a negative error code on failure
165 */172 */
166int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 **cpu_addr)173int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va,
174 u64 *gpu_addr, u64 **cpu_addr)
167{175{
168 unsigned long bit_pos;176 unsigned long bit_pos;
169 177
@@ -172,7 +180,12 @@ int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 **cpu_addr) · 2 unmodified lines
172 return -ENOSPC;180 return -ENOSPC;
173 181
174 __set_bit(bit_pos, adev->seq64.used);182 __set_bit(bit_pos, adev->seq64.used);
183
175 *va = bit_pos * sizeof(u64) + amdgpu_seq64_get_va_base(adev);184 *va = bit_pos * sizeof(u64) + amdgpu_seq64_get_va_base(adev);
185
186 if (gpu_addr)
187 *gpu_addr = bit_pos * sizeof(u64) + adev->seq64.gpu_addr;
188
176 *cpu_addr = bit_pos + adev->seq64.cpu_base_addr;189 *cpu_addr = bit_pos + adev->seq64.cpu_base_addr;
177 190
178 return 0;191 return 0;
@@ -233,7 +246,7 @@ int amdgpu_seq64_init(struct amdgpu_device *adev) · 54 unmodified lines
233 */246 */
234 r = amdgpu_bo_create_kernel(adev, AMDGPU_VA_RESERVED_SEQ64_SIZE,247 r = amdgpu_bo_create_kernel(adev, AMDGPU_VA_RESERVED_SEQ64_SIZE,
235 PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,248 PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT,
236 &adev->seq64.sbo, NULL,249 &adev->seq64.sbo, &adev->seq64.gpu_addr,
237 (void **)&adev->seq64.cpu_base_addr);250 (void **)&adev->seq64.cpu_base_addr);
238 if (r) {251 if (r) {
239 dev_warn(adev->dev, "(%d) create seq64 failed\n", r);252 dev_warn(adev->dev, "(%d) create seq64 failed\n", r);
drivers/gpu/drm/amd/amdgpu/amdgpu_seq64.h+2−1
@@ -32,13 +32,14 @@ · 31 unmodified lines
32struct amdgpu_seq64 {32struct amdgpu_seq64 {
33 struct amdgpu_bo *sbo;33 struct amdgpu_bo *sbo;
34 u32 num_sem;34 u32 num_sem;
35 u64 gpu_addr;
35 u64 *cpu_base_addr;36 u64 *cpu_base_addr;
36 DECLARE_BITMAP(used, AMDGPU_MAX_SEQ64_SLOTS);37 DECLARE_BITMAP(used, AMDGPU_MAX_SEQ64_SLOTS);
37};38};
38 39
39void amdgpu_seq64_fini(struct amdgpu_device *adev);40void amdgpu_seq64_fini(struct amdgpu_device *adev);
40int amdgpu_seq64_init(struct amdgpu_device *adev);41int amdgpu_seq64_init(struct amdgpu_device *adev);
41int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *gpu_addr, u64 **cpu_addr);42int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 *gpu_addr, u64 **cpu_addr);
42void amdgpu_seq64_free(struct amdgpu_device *adev, u64 gpu_addr);43void amdgpu_seq64_free(struct amdgpu_device *adev, u64 gpu_addr);
43int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,44int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm,
44 struct amdgpu_bo_va **bo_va);45 struct amdgpu_bo_va **bo_va);
drivers/gpu/drm/amd/amdgpu/amdgpu_sync.c+2−3
@@ -249,9 +249,8 @@ int amdgpu_sync_resv(struct amdgpu_device *adev, struct amdgpu_sync *sync, · 248 unmodified lines
249 249
250 if (resv == NULL)250 if (resv == NULL)
251 return -EINVAL;251 return -EINVAL;
252 252 /* Implicitly sync only to KERNEL, WRITE and READ */
253 /* TODO: Use DMA_RESV_USAGE_READ here */253 dma_resv_for_each_fence(&cursor, resv, DMA_RESV_USAGE_READ, f) {
254 dma_resv_for_each_fence(&cursor, resv, DMA_RESV_USAGE_BOOKKEEP, f) {
255 dma_fence_chain_for_each(f, f) {254 dma_fence_chain_for_each(f, f) {
256 struct dma_fence *tmp = dma_fence_chain_contained(f);255 struct dma_fence *tmp = dma_fence_chain_contained(f);
257 256
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.h+2−0
@@ -25,6 +25,8 @@ · 24 unmodified lines
25 25
26#include "amdgpu_socbb.h"26#include "amdgpu_socbb.h"
27 27
28#define RS64_FW_UC_START_ADDR_LO 0x3000
29
28struct common_firmware_header {30struct common_firmware_header {
29 uint32_t size_bytes; /* size of the entire header+image(s) in bytes */31 uint32_t size_bytes; /* size of the entire header+image(s) in bytes */
30 uint32_t header_size_bytes; /* size of just the header in bytes */32 uint32_t header_size_bytes; /* size of just the header in bytes */
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c+919−0
@@ -0,0 +1,919 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright 2023 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#include <drm/drm_auth.h>
26#include <drm/drm_exec.h>
27#include <linux/pm_runtime.h>
28
29#include "amdgpu.h"
30#include "amdgpu_vm.h"
31#include "amdgpu_userq.h"
32#include "amdgpu_userq_fence.h"
33
34u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev)
35{
36 int i;
37 u32 userq_ip_mask = 0;
38
39 for (i = 0; i < AMDGPU_HW_IP_NUM; i++) {
40 if (adev->userq_funcs[i])
41 userq_ip_mask |= (1 << i);
42 }
43
44 return userq_ip_mask;
45}
46
47static int
48amdgpu_userq_unmap_helper(struct amdgpu_userq_mgr *uq_mgr,
49 struct amdgpu_usermode_queue *queue)
50{
51 struct amdgpu_device *adev = uq_mgr->adev;
52 const struct amdgpu_userq_funcs *userq_funcs =
53 adev->userq_funcs[queue->queue_type];
54 int r = 0;
55
56 if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
57 r = userq_funcs->unmap(uq_mgr, queue);
58 if (r)
59 queue->state = AMDGPU_USERQ_STATE_HUNG;
60 else
61 queue->state = AMDGPU_USERQ_STATE_UNMAPPED;
62 }
63 return r;
64}
65
66static int
67amdgpu_userq_map_helper(struct amdgpu_userq_mgr *uq_mgr,
68 struct amdgpu_usermode_queue *queue)
69{
70 struct amdgpu_device *adev = uq_mgr->adev;
71 const struct amdgpu_userq_funcs *userq_funcs =
72 adev->userq_funcs[queue->queue_type];
73 int r = 0;
74
75 if (queue->state == AMDGPU_USERQ_STATE_UNMAPPED) {
76 r = userq_funcs->map(uq_mgr, queue);
77 if (r) {
78 queue->state = AMDGPU_USERQ_STATE_HUNG;
79 } else {
80 queue->state = AMDGPU_USERQ_STATE_MAPPED;
81 }
82 }
83 return r;
84}
85
86static void
87amdgpu_userq_wait_for_last_fence(struct amdgpu_userq_mgr *uq_mgr,
88 struct amdgpu_usermode_queue *queue)
89{
90 struct dma_fence *f = queue->last_fence;
91 int ret;
92
93 if (f && !dma_fence_is_signaled(f)) {
94 ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100));
95 if (ret <= 0)
96 drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n",
97 f->context, f->seqno);
98 }
99}
100
101static void
102amdgpu_userq_cleanup(struct amdgpu_userq_mgr *uq_mgr,
103 struct amdgpu_usermode_queue *queue,
104 int queue_id)
105{
106 struct amdgpu_device *adev = uq_mgr->adev;
107 const struct amdgpu_userq_funcs *uq_funcs = adev->userq_funcs[queue->queue_type];
108
109 uq_funcs->mqd_destroy(uq_mgr, queue);
110 amdgpu_userq_fence_driver_free(queue);
111 idr_remove(&uq_mgr->userq_idr, queue_id);
112 kfree(queue);
113}
114
115int
116amdgpu_userq_active(struct amdgpu_userq_mgr *uq_mgr)
117{
118 struct amdgpu_usermode_queue *queue;
119 int queue_id;
120 int ret = 0;
121
122 mutex_lock(&uq_mgr->userq_mutex);
123 /* Resume all the queues for this process */
124 idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id)
125 ret += queue->state == AMDGPU_USERQ_STATE_MAPPED;
126
127 mutex_unlock(&uq_mgr->userq_mutex);
128 return ret;
129}
130
131static struct amdgpu_usermode_queue *
132amdgpu_userq_find(struct amdgpu_userq_mgr *uq_mgr, int qid)
133{
134 return idr_find(&uq_mgr->userq_idr, qid);
135}
136
137void
138amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *uq_mgr,
139 struct amdgpu_eviction_fence_mgr *evf_mgr)
140{
141 struct amdgpu_eviction_fence *ev_fence;
142
143retry:
144 /* Flush any pending resume work to create ev_fence */
145 flush_delayed_work(&uq_mgr->resume_work);
146
147 mutex_lock(&uq_mgr->userq_mutex);
148 spin_lock(&evf_mgr->ev_fence_lock);
149 ev_fence = evf_mgr->ev_fence;
150 spin_unlock(&evf_mgr->ev_fence_lock);
151 if (!ev_fence || dma_fence_is_signaled(&ev_fence->base)) {
152 mutex_unlock(&uq_mgr->userq_mutex);
153 /*
154 * Looks like there was no pending resume work,
155 * add one now to create a valid eviction fence
156 */
157 schedule_delayed_work(&uq_mgr->resume_work, 0);
158 goto retry;
159 }
160}
161
162int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr,
163 struct amdgpu_userq_obj *userq_obj,
164 int size)
165{
166 struct amdgpu_device *adev = uq_mgr->adev;
167 struct amdgpu_bo_param bp;
168 int r;
169
170 memset(&bp, 0, sizeof(bp));
171 bp.byte_align = PAGE_SIZE;
172 bp.domain = AMDGPU_GEM_DOMAIN_GTT;
173 bp.flags = AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS |
174 AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
175 bp.type = ttm_bo_type_kernel;
176 bp.size = size;
177 bp.resv = NULL;
178 bp.bo_ptr_size = sizeof(struct amdgpu_bo);
179
180 r = amdgpu_bo_create(adev, &bp, &userq_obj->obj);
181 if (r) {
182 drm_file_err(uq_mgr->file, "Failed to allocate BO for userqueue (%d)", r);
183 return r;
184 }
185
186 r = amdgpu_bo_reserve(userq_obj->obj, true);
187 if (r) {
188 drm_file_err(uq_mgr->file, "Failed to reserve BO to map (%d)", r);
189 goto free_obj;
190 }
191
192 r = amdgpu_ttm_alloc_gart(&(userq_obj->obj)->tbo);
193 if (r) {
194 drm_file_err(uq_mgr->file, "Failed to alloc GART for userqueue object (%d)", r);
195 goto unresv;
196 }
197
198 r = amdgpu_bo_kmap(userq_obj->obj, &userq_obj->cpu_ptr);
199 if (r) {
200 drm_file_err(uq_mgr->file, "Failed to map BO for userqueue (%d)", r);
201 goto unresv;
202 }
203
204 userq_obj->gpu_addr = amdgpu_bo_gpu_offset(userq_obj->obj);
205 amdgpu_bo_unreserve(userq_obj->obj);
206 memset(userq_obj->cpu_ptr, 0, size);
207 return 0;
208
209unresv:
210 amdgpu_bo_unreserve(userq_obj->obj);
211
212free_obj:
213 amdgpu_bo_unref(&userq_obj->obj);
214 return r;
215}
216
217void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr,
218 struct amdgpu_userq_obj *userq_obj)
219{
220 amdgpu_bo_kunmap(userq_obj->obj);
221 amdgpu_bo_unref(&userq_obj->obj);
222}
223
224uint64_t
225amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr,
226 struct amdgpu_db_info *db_info,
227 struct drm_file *filp)
228{
229 uint64_t index;
230 struct drm_gem_object *gobj;
231 struct amdgpu_userq_obj *db_obj = db_info->db_obj;
232 int r, db_size;
233
234 gobj = drm_gem_object_lookup(filp, db_info->doorbell_handle);
235 if (gobj == NULL) {
236 drm_file_err(uq_mgr->file, "Can't find GEM object for doorbell\n");
237 return -EINVAL;
238 }
239
240 db_obj->obj = amdgpu_bo_ref(gem_to_amdgpu_bo(gobj));
241 drm_gem_object_put(gobj);
242
243 /* Pin the BO before generating the index, unpin in queue destroy */
244 r = amdgpu_bo_pin(db_obj->obj, AMDGPU_GEM_DOMAIN_DOORBELL);
245 if (r) {
246 drm_file_err(uq_mgr->file, "[Usermode queues] Failed to pin doorbell object\n");
247 goto unref_bo;
248 }
249
250 r = amdgpu_bo_reserve(db_obj->obj, true);
251 if (r) {
252 drm_file_err(uq_mgr->file, "[Usermode queues] Failed to pin doorbell object\n");
253 goto unpin_bo;
254 }
255
256 switch (db_info->queue_type) {
257 case AMDGPU_HW_IP_GFX:
258 case AMDGPU_HW_IP_COMPUTE:
259 case AMDGPU_HW_IP_DMA:
260 db_size = sizeof(u64);
261 break;
262
263 case AMDGPU_HW_IP_VCN_ENC:
264 db_size = sizeof(u32);
265 db_info->doorbell_offset += AMDGPU_NAVI10_DOORBELL64_VCN0_1 << 1;
266 break;
267
268 case AMDGPU_HW_IP_VPE:
269 db_size = sizeof(u32);
270 db_info->doorbell_offset += AMDGPU_NAVI10_DOORBELL64_VPE << 1;
271 break;
272
273 default:
274 drm_file_err(uq_mgr->file, "[Usermode queues] IP %d not support\n",
275 db_info->queue_type);
276 r = -EINVAL;
277 goto unpin_bo;
278 }
279
280 index = amdgpu_doorbell_index_on_bar(uq_mgr->adev, db_obj->obj,
281 db_info->doorbell_offset, db_size);
282 drm_dbg_driver(adev_to_drm(uq_mgr->adev),
283 "[Usermode queues] doorbell index=%lld\n", index);
284 amdgpu_bo_unreserve(db_obj->obj);
285 return index;
286
287unpin_bo:
288 amdgpu_bo_unpin(db_obj->obj);
289
290unref_bo:
291 amdgpu_bo_unref(&db_obj->obj);
292 return r;
293}
294
295static int
296amdgpu_userq_destroy(struct drm_file *filp, int queue_id)
297{
298 struct amdgpu_fpriv *fpriv = filp->driver_priv;
299 struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
300 struct amdgpu_device *adev = uq_mgr->adev;
301 struct amdgpu_usermode_queue *queue;
302 int r = 0;
303
304 cancel_delayed_work(&uq_mgr->resume_work);
305 mutex_lock(&uq_mgr->userq_mutex);
306
307 queue = amdgpu_userq_find(uq_mgr, queue_id);
308 if (!queue) {
309 drm_dbg_driver(adev_to_drm(uq_mgr->adev), "Invalid queue id to destroy\n");
310 mutex_unlock(&uq_mgr->userq_mutex);
311 return -EINVAL;
312 }
313 amdgpu_userq_wait_for_last_fence(uq_mgr, queue);
314 r = amdgpu_userq_unmap_helper(uq_mgr, queue);
315 amdgpu_bo_unpin(queue->db_obj.obj);
316 amdgpu_bo_unref(&queue->db_obj.obj);
317 amdgpu_userq_cleanup(uq_mgr, queue, queue_id);
318 mutex_unlock(&uq_mgr->userq_mutex);
319
320 pm_runtime_mark_last_busy(adev_to_drm(adev)->dev);
321 pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
322
323 return r;
324}
325
326static int amdgpu_userq_priority_permit(struct drm_file *filp,
327 int priority)
328{
329 if (priority < AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_HIGH)
330 return 0;
331
332 if (capable(CAP_SYS_NICE))
333 return 0;
334
335 if (drm_is_current_master(filp))
336 return 0;
337
338 return -EACCES;
339}
340
341static int
342amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
343{
344 struct amdgpu_fpriv *fpriv = filp->driver_priv;
345 struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr;
346 struct amdgpu_device *adev = uq_mgr->adev;
347 const struct amdgpu_userq_funcs *uq_funcs;
348 struct amdgpu_usermode_queue *queue;
349 struct amdgpu_db_info db_info;
350 bool skip_map_queue;
351 uint64_t index;
352 int qid, r = 0;
353 int priority =
354 (args->in.flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_MASK) >>
355 AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_SHIFT;
356
357 /* Usermode queues are only supported for GFX IP as of now */
358 if (args->in.ip_type != AMDGPU_HW_IP_GFX &&
359 args->in.ip_type != AMDGPU_HW_IP_DMA &&
360 args->in.ip_type != AMDGPU_HW_IP_COMPUTE) {
361 drm_file_err(uq_mgr->file, "Usermode queue doesn't support IP type %u\n",
362 args->in.ip_type);
363 return -EINVAL;
364 }
365
366 r = amdgpu_userq_priority_permit(filp, priority);
367 if (r)
368 return r;
369
370 if ((args->in.flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE) &&
371 (args->in.ip_type != AMDGPU_HW_IP_GFX) &&
372 (args->in.ip_type != AMDGPU_HW_IP_COMPUTE) &&
373 !amdgpu_is_tmz(adev)) {
374 drm_file_err(uq_mgr->file, "Secure only supported on GFX/Compute queues\n");
375 return -EINVAL;
376 }
377
378 r = pm_runtime_get_sync(adev_to_drm(adev)->dev);
379 if (r < 0) {
380 drm_file_err(uq_mgr->file, "pm_runtime_get_sync() failed for userqueue create\n");
381 pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
382 return r;
383 }
384
385 /*
386 * There could be a situation that we are creating a new queue while
387 * the other queues under this UQ_mgr are suspended. So if there is any
388 * resume work pending, wait for it to get done.
389 *
390 * This will also make sure we have a valid eviction fence ready to be used.
391 */
392 amdgpu_userq_ensure_ev_fence(&fpriv->userq_mgr, &fpriv->evf_mgr);
393
394 uq_funcs = adev->userq_funcs[args->in.ip_type];
395 if (!uq_funcs) {
396 drm_file_err(uq_mgr->file, "Usermode queue is not supported for this IP (%u)\n",
397 args->in.ip_type);
398 r = -EINVAL;
399 goto unlock;
400 }
401
402 queue = kzalloc(sizeof(struct amdgpu_usermode_queue), GFP_KERNEL);
403 if (!queue) {
404 drm_file_err(uq_mgr->file, "Failed to allocate memory for queue\n");
405 r = -ENOMEM;
406 goto unlock;
407 }
408 queue->doorbell_handle = args->in.doorbell_handle;
409 queue->queue_type = args->in.ip_type;
410 queue->vm = &fpriv->vm;
411 queue->priority = priority;
412
413 db_info.queue_type = queue->queue_type;
414 db_info.doorbell_handle = queue->doorbell_handle;
415 db_info.db_obj = &queue->db_obj;
416 db_info.doorbell_offset = args->in.doorbell_offset;
417
418 /* Convert relative doorbell offset into absolute doorbell index */
419 index = amdgpu_userq_get_doorbell_index(uq_mgr, &db_info, filp);
420 if (index == (uint64_t)-EINVAL) {
421 drm_file_err(uq_mgr->file, "Failed to get doorbell for queue\n");
422 kfree(queue);
423 goto unlock;
424 }
425
426 queue->doorbell_index = index;
427 xa_init_flags(&queue->fence_drv_xa, XA_FLAGS_ALLOC);
428 r = amdgpu_userq_fence_driver_alloc(adev, queue);
429 if (r) {
430 drm_file_err(uq_mgr->file, "Failed to alloc fence driver\n");
431 goto unlock;
432 }
433
434 r = uq_funcs->mqd_create(uq_mgr, &args->in, queue);
435 if (r) {
436 drm_file_err(uq_mgr->file, "Failed to create Queue\n");
437 amdgpu_userq_fence_driver_free(queue);
438 kfree(queue);
439 goto unlock;
440 }
441
442
443 qid = idr_alloc(&uq_mgr->userq_idr, queue, 1, AMDGPU_MAX_USERQ_COUNT, GFP_KERNEL);
444 if (qid < 0) {
445 drm_file_err(uq_mgr->file, "Failed to allocate a queue id\n");
446 amdgpu_userq_fence_driver_free(queue);
447 uq_funcs->mqd_destroy(uq_mgr, queue);
448 kfree(queue);
449 r = -ENOMEM;
450 goto unlock;
451 }
452
453 /* don't map the queue if scheduling is halted */
454 mutex_lock(&adev->userq_mutex);
455 if (adev->userq_halt_for_enforce_isolation &&
456 ((queue->queue_type == AMDGPU_HW_IP_GFX) ||
457 (queue->queue_type == AMDGPU_HW_IP_COMPUTE)))
458 skip_map_queue = true;
459 else
460 skip_map_queue = false;
461 if (!skip_map_queue) {
462 r = amdgpu_userq_map_helper(uq_mgr, queue);
463 if (r) {
464 mutex_unlock(&adev->userq_mutex);
465 drm_file_err(uq_mgr->file, "Failed to map Queue\n");
466 idr_remove(&uq_mgr->userq_idr, qid);
467 amdgpu_userq_fence_driver_free(queue);
468 uq_funcs->mqd_destroy(uq_mgr, queue);
469 kfree(queue);
470 goto unlock;
471 }
472 }
473 mutex_unlock(&adev->userq_mutex);
474
475
476 args->out.queue_id = qid;
477
478unlock:
479 mutex_unlock(&uq_mgr->userq_mutex);
480
481 return r;
482}
483
484int amdgpu_userq_ioctl(struct drm_device *dev, void *data,
485 struct drm_file *filp)
486{
487 union drm_amdgpu_userq *args = data;
488 int r;
489
490 switch (args->in.op) {
491 case AMDGPU_USERQ_OP_CREATE:
492 if (args->in.flags & ~(AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_MASK |
493 AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE))
494 return -EINVAL;
495 r = amdgpu_userq_create(filp, args);
496 if (r)
497 drm_file_err(filp, "Failed to create usermode queue\n");
498 break;
499
500 case AMDGPU_USERQ_OP_FREE:
501 if (args->in.ip_type ||
502 args->in.doorbell_handle ||
503 args->in.doorbell_offset ||
504 args->in.flags ||
505 args->in.queue_va ||
506 args->in.queue_size ||
507 args->in.rptr_va ||
508 args->in.wptr_va ||
509 args->in.wptr_va ||
510 args->in.mqd ||
511 args->in.mqd_size)
512 return -EINVAL;
513 r = amdgpu_userq_destroy(filp, args->in.queue_id);
514 if (r)
515 drm_file_err(filp, "Failed to destroy usermode queue\n");
516 break;
517
518 default:
519 drm_dbg_driver(dev, "Invalid user queue op specified: %d\n", args->in.op);
520 return -EINVAL;
521 }
522
523 return r;
524}
525
526static int
527amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
528{
529 struct amdgpu_usermode_queue *queue;
530 int queue_id;
531 int ret = 0, r;
532
533 /* Resume all the queues for this process */
534 idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) {
535 r = amdgpu_userq_map_helper(uq_mgr, queue);
536 if (r)
537 ret = r;
538 }
539
540 if (ret)
541 drm_file_err(uq_mgr->file, "Failed to map all the queues\n");
542 return ret;
543}
544
545static int
546amdgpu_userq_validate_vm_bo(void *_unused, struct amdgpu_bo *bo)
547{
548 struct ttm_operation_ctx ctx = { false, false };
549 int ret;
550
551 amdgpu_bo_placement_from_domain(bo, bo->allowed_domains);
552
553 ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
554 if (ret)
555 DRM_ERROR("Fail to validate\n");
556
557 return ret;
558}
559
560static int
561amdgpu_userq_validate_bos(struct amdgpu_userq_mgr *uq_mgr)
562{
563 struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
564 struct amdgpu_vm *vm = &fpriv->vm;
565 struct amdgpu_device *adev = uq_mgr->adev;
566 struct amdgpu_bo_va *bo_va;
567 struct ww_acquire_ctx *ticket;
568 struct drm_exec exec;
569 struct amdgpu_bo *bo;
570 struct dma_resv *resv;
571 bool clear, unlock;
572 int ret = 0;
573
574 drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0);
575 drm_exec_until_all_locked(&exec) {
576 ret = amdgpu_vm_lock_pd(vm, &exec, 2);
577 drm_exec_retry_on_contention(&exec);
578 if (unlikely(ret)) {
579 drm_file_err(uq_mgr->file, "Failed to lock PD\n");
580 goto unlock_all;
581 }
582
583 /* Lock the done list */
584 list_for_each_entry(bo_va, &vm->done, base.vm_status) {
585 bo = bo_va->base.bo;
586 if (!bo)
587 continue;
588
589 ret = drm_exec_lock_obj(&exec, &bo->tbo.base);
590 drm_exec_retry_on_contention(&exec);
591 if (unlikely(ret))
592 goto unlock_all;
593 }
594 }
595
596 spin_lock(&vm->status_lock);
597 while (!list_empty(&vm->moved)) {
598 bo_va = list_first_entry(&vm->moved, struct amdgpu_bo_va,
599 base.vm_status);
600 spin_unlock(&vm->status_lock);
601
602 /* Per VM BOs never need to bo cleared in the page tables */
603 ret = amdgpu_vm_bo_update(adev, bo_va, false);
604 if (ret)
605 goto unlock_all;
606 spin_lock(&vm->status_lock);
607 }
608
609 ticket = &exec.ticket;
610 while (!list_empty(&vm->invalidated)) {
611 bo_va = list_first_entry(&vm->invalidated, struct amdgpu_bo_va,
612 base.vm_status);
613 resv = bo_va->base.bo->tbo.base.resv;
614 spin_unlock(&vm->status_lock);
615
616 bo = bo_va->base.bo;
617 ret = amdgpu_userq_validate_vm_bo(NULL, bo);
618 if (ret) {
619 drm_file_err(uq_mgr->file, "Failed to validate BO\n");
620 goto unlock_all;
621 }
622
623 /* Try to reserve the BO to avoid clearing its ptes */
624 if (!adev->debug_vm && dma_resv_trylock(resv)) {
625 clear = false;
626 unlock = true;
627 /* The caller is already holding the reservation lock */
628 } else if (dma_resv_locking_ctx(resv) == ticket) {
629 clear = false;
630 unlock = false;
631 /* Somebody else is using the BO right now */
632 } else {
633 clear = true;
634 unlock = false;
635 }
636
637 ret = amdgpu_vm_bo_update(adev, bo_va, clear);
638
639 if (unlock)
640 dma_resv_unlock(resv);
641 if (ret)
642 goto unlock_all;
643
644 spin_lock(&vm->status_lock);
645 }
646 spin_unlock(&vm->status_lock);
647
648 ret = amdgpu_eviction_fence_replace_fence(&fpriv->evf_mgr, &exec);
649 if (ret)
650 drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n");
651
652unlock_all:
653 drm_exec_fini(&exec);
654 return ret;
655}
656
657static void amdgpu_userq_restore_worker(struct work_struct *work)
658{
659 struct amdgpu_userq_mgr *uq_mgr = work_to_uq_mgr(work, resume_work.work);
660 struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
661 int ret;
662
663 flush_work(&fpriv->evf_mgr.suspend_work.work);
664
665 mutex_lock(&uq_mgr->userq_mutex);
666
667 ret = amdgpu_userq_validate_bos(uq_mgr);
668 if (ret) {
669 drm_file_err(uq_mgr->file, "Failed to validate BOs to restore\n");
670 goto unlock;
671 }
672
673 ret = amdgpu_userq_restore_all(uq_mgr);
674 if (ret) {
675 drm_file_err(uq_mgr->file, "Failed to restore all queues\n");
676 goto unlock;
677 }
678
679unlock:
680 mutex_unlock(&uq_mgr->userq_mutex);
681}
682
683static int
684amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr)
685{
686 struct amdgpu_usermode_queue *queue;
687 int queue_id;
688 int ret = 0, r;
689
690 /* Try to unmap all the queues in this process ctx */
691 idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) {
692 r = amdgpu_userq_unmap_helper(uq_mgr, queue);
693 if (r)
694 ret = r;
695 }
696
697 if (ret)
698 drm_file_err(uq_mgr->file, "Couldn't unmap all the queues\n");
699 return ret;
700}
701
702static int
703amdgpu_userq_wait_for_signal(struct amdgpu_userq_mgr *uq_mgr)
704{
705 struct amdgpu_usermode_queue *queue;
706 int queue_id, ret;
707
708 idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) {
709 struct dma_fence *f = queue->last_fence;
710
711 if (!f || dma_fence_is_signaled(f))
712 continue;
713 ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100));
714 if (ret <= 0) {
715 drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n",
716 f->context, f->seqno);
717 return -ETIMEDOUT;
718 }
719 }
720
721 return 0;
722}
723
724void
725amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr,
726 struct amdgpu_eviction_fence *ev_fence)
727{
728 int ret;
729 struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
730 struct amdgpu_eviction_fence_mgr *evf_mgr = &fpriv->evf_mgr;
731
732 /* Wait for any pending userqueue fence work to finish */
733 ret = amdgpu_userq_wait_for_signal(uq_mgr);
734 if (ret) {
735 drm_file_err(uq_mgr->file, "Not evicting userqueue, timeout waiting for work\n");
736 return;
737 }
738
739 ret = amdgpu_userq_evict_all(uq_mgr);
740 if (ret) {
741 drm_file_err(uq_mgr->file, "Failed to evict userqueue\n");
742 return;
743 }
744
745 /* Signal current eviction fence */
746 amdgpu_eviction_fence_signal(evf_mgr, ev_fence);
747
748 if (evf_mgr->fd_closing) {
749 cancel_delayed_work(&uq_mgr->resume_work);
750 return;
751 }
752
753 /* Schedule a resume work */
754 schedule_delayed_work(&uq_mgr->resume_work, 0);
755}
756
757int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv,
758 struct amdgpu_device *adev)
759{
760 mutex_init(&userq_mgr->userq_mutex);
761 idr_init_base(&userq_mgr->userq_idr, 1);
762 userq_mgr->adev = adev;
763 userq_mgr->file = file_priv;
764
765 mutex_lock(&adev->userq_mutex);
766 list_add(&userq_mgr->list, &adev->userq_mgr_list);
767 mutex_unlock(&adev->userq_mutex);
768
769 INIT_DELAYED_WORK(&userq_mgr->resume_work, amdgpu_userq_restore_worker);
770 return 0;
771}
772
773void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr)
774{
775 struct amdgpu_device *adev = userq_mgr->adev;
776 struct amdgpu_usermode_queue *queue;
777 struct amdgpu_userq_mgr *uqm, *tmp;
778 uint32_t queue_id;
779
780 cancel_delayed_work(&userq_mgr->resume_work);
781
782 mutex_lock(&userq_mgr->userq_mutex);
783 idr_for_each_entry(&userq_mgr->userq_idr, queue, queue_id) {
784 amdgpu_userq_wait_for_last_fence(userq_mgr, queue);
785 amdgpu_userq_unmap_helper(userq_mgr, queue);
786 amdgpu_userq_cleanup(userq_mgr, queue, queue_id);
787 }
788 mutex_lock(&adev->userq_mutex);
789 list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
790 if (uqm == userq_mgr) {
791 list_del(&uqm->list);
792 break;
793 }
794 }
795 mutex_unlock(&adev->userq_mutex);
796 idr_destroy(&userq_mgr->userq_idr);
797 mutex_unlock(&userq_mgr->userq_mutex);
798 mutex_destroy(&userq_mgr->userq_mutex);
799}
800
801int amdgpu_userq_suspend(struct amdgpu_device *adev)
802{
803 u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
804 struct amdgpu_usermode_queue *queue;
805 struct amdgpu_userq_mgr *uqm, *tmp;
806 int queue_id;
807 int ret = 0, r;
808
809 if (!ip_mask)
810 return 0;
811
812 mutex_lock(&adev->userq_mutex);
813 list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
814 cancel_delayed_work_sync(&uqm->resume_work);
815 mutex_lock(&uqm->userq_mutex);
816 idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
817 r = amdgpu_userq_unmap_helper(uqm, queue);
818 if (r)
819 ret = r;
820 }
821 mutex_unlock(&uqm->userq_mutex);
822 }
823 mutex_unlock(&adev->userq_mutex);
824 return ret;
825}
826
827int amdgpu_userq_resume(struct amdgpu_device *adev)
828{
829 u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
830 struct amdgpu_usermode_queue *queue;
831 struct amdgpu_userq_mgr *uqm, *tmp;
832 int queue_id;
833 int ret = 0, r;
834
835 if (!ip_mask)
836 return 0;
837
838 mutex_lock(&adev->userq_mutex);
839 list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
840 mutex_lock(&uqm->userq_mutex);
841 idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
842 r = amdgpu_userq_map_helper(uqm, queue);
843 if (r)
844 ret = r;
845 }
846 mutex_unlock(&uqm->userq_mutex);
847 }
848 mutex_unlock(&adev->userq_mutex);
849 return ret;
850}
851
852int amdgpu_userq_stop_sched_for_enforce_isolation(struct amdgpu_device *adev,
853 u32 idx)
854{
855 u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
856 struct amdgpu_usermode_queue *queue;
857 struct amdgpu_userq_mgr *uqm, *tmp;
858 int queue_id;
859 int ret = 0, r;
860
861 /* only need to stop gfx/compute */
862 if (!(ip_mask & ((1 << AMDGPU_HW_IP_GFX) | (1 << AMDGPU_HW_IP_COMPUTE))))
863 return 0;
864
865 mutex_lock(&adev->userq_mutex);
866 if (adev->userq_halt_for_enforce_isolation)
867 dev_warn(adev->dev, "userq scheduling already stopped!\n");
868 adev->userq_halt_for_enforce_isolation = true;
869 list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
870 cancel_delayed_work_sync(&uqm->resume_work);
871 mutex_lock(&uqm->userq_mutex);
872 idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
873 if (((queue->queue_type == AMDGPU_HW_IP_GFX) ||
874 (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) &&
875 (queue->xcp_id == idx)) {
876 r = amdgpu_userq_unmap_helper(uqm, queue);
877 if (r)
878 ret = r;
879 }
880 }
881 mutex_unlock(&uqm->userq_mutex);
882 }
883 mutex_unlock(&adev->userq_mutex);
884 return ret;
885}
886
887int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev,
888 u32 idx)
889{
890 u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev);
891 struct amdgpu_usermode_queue *queue;
892 struct amdgpu_userq_mgr *uqm, *tmp;
893 int queue_id;
894 int ret = 0, r;
895
896 /* only need to stop gfx/compute */
897 if (!(ip_mask & ((1 << AMDGPU_HW_IP_GFX) | (1 << AMDGPU_HW_IP_COMPUTE))))
898 return 0;
899
900 mutex_lock(&adev->userq_mutex);
901 if (!adev->userq_halt_for_enforce_isolation)
902 dev_warn(adev->dev, "userq scheduling already started!\n");
903 adev->userq_halt_for_enforce_isolation = false;
904 list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) {
905 mutex_lock(&uqm->userq_mutex);
906 idr_for_each_entry(&uqm->userq_idr, queue, queue_id) {
907 if (((queue->queue_type == AMDGPU_HW_IP_GFX) ||
908 (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) &&
909 (queue->xcp_id == idx)) {
910 r = amdgpu_userq_map_helper(uqm, queue);
911 if (r)
912 ret = r;
913 }
914 }
915 mutex_unlock(&uqm->userq_mutex);
916 }
917 mutex_unlock(&adev->userq_mutex);
918 return ret;
919}
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h+135−0
@@ -0,0 +1,135 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright 2023 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#ifndef AMDGPU_USERQ_H_
26#define AMDGPU_USERQ_H_
27#include "amdgpu_eviction_fence.h"
28
29#define AMDGPU_MAX_USERQ_COUNT 512
30
31#define to_ev_fence(f) container_of(f, struct amdgpu_eviction_fence, base)
32#define uq_mgr_to_fpriv(u) container_of(u, struct amdgpu_fpriv, userq_mgr)
33#define work_to_uq_mgr(w, name) container_of(w, struct amdgpu_userq_mgr, name)
34
35enum amdgpu_userq_state {
36 AMDGPU_USERQ_STATE_UNMAPPED = 0,
37 AMDGPU_USERQ_STATE_MAPPED,
38 AMDGPU_USERQ_STATE_PREEMPTED,
39 AMDGPU_USERQ_STATE_HUNG,
40};
41
42struct amdgpu_mqd_prop;
43
44struct amdgpu_userq_obj {
45 void *cpu_ptr;
46 uint64_t gpu_addr;
47 struct amdgpu_bo *obj;
48};
49
50struct amdgpu_usermode_queue {
51 int queue_type;
52 enum amdgpu_userq_state state;
53 uint64_t doorbell_handle;
54 uint64_t doorbell_index;
55 uint64_t flags;
56 struct amdgpu_mqd_prop *userq_prop;
57 struct amdgpu_userq_mgr *userq_mgr;
58 struct amdgpu_vm *vm;
59 struct amdgpu_userq_obj mqd;
60 struct amdgpu_userq_obj db_obj;
61 struct amdgpu_userq_obj fw_obj;
62 struct amdgpu_userq_obj wptr_obj;
63 struct xarray fence_drv_xa;
64 struct amdgpu_userq_fence_driver *fence_drv;
65 struct dma_fence *last_fence;
66 u32 xcp_id;
67 int priority;
68};
69
70struct amdgpu_userq_funcs {
71 int (*mqd_create)(struct amdgpu_userq_mgr *uq_mgr,
72 struct drm_amdgpu_userq_in *args,
73 struct amdgpu_usermode_queue *queue);
74 void (*mqd_destroy)(struct amdgpu_userq_mgr *uq_mgr,
75 struct amdgpu_usermode_queue *uq);
76 int (*unmap)(struct amdgpu_userq_mgr *uq_mgr,
77 struct amdgpu_usermode_queue *queue);
78 int (*map)(struct amdgpu_userq_mgr *uq_mgr,
79 struct amdgpu_usermode_queue *queue);
80};
81
82/* Usermode queues for gfx */
83struct amdgpu_userq_mgr {
84 struct idr userq_idr;
85 struct mutex userq_mutex;
86 struct amdgpu_device *adev;
87 struct delayed_work resume_work;
88 struct list_head list;
89 struct drm_file *file;
90};
91
92struct amdgpu_db_info {
93 uint64_t doorbell_handle;
94 uint32_t queue_type;
95 uint32_t doorbell_offset;
96 struct amdgpu_userq_obj *db_obj;
97};
98
99int amdgpu_userq_ioctl(struct drm_device *dev, void *data, struct drm_file *filp);
100
101int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv,
102 struct amdgpu_device *adev);
103
104void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr);
105
106int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr,
107 struct amdgpu_userq_obj *userq_obj,
108 int size);
109
110void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr,
111 struct amdgpu_userq_obj *userq_obj);
112
113void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr,
114 struct amdgpu_eviction_fence *ev_fence);
115
116int amdgpu_userq_active(struct amdgpu_userq_mgr *uq_mgr);
117
118void amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *userq_mgr,
119 struct amdgpu_eviction_fence_mgr *evf_mgr);
120
121uint64_t amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr,
122 struct amdgpu_db_info *db_info,
123 struct drm_file *filp);
124
125u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev);
126
127int amdgpu_userq_suspend(struct amdgpu_device *adev);
128int amdgpu_userq_resume(struct amdgpu_device *adev);
129
130int amdgpu_userq_stop_sched_for_enforce_isolation(struct amdgpu_device *adev,
131 u32 idx);
132int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev,
133 u32 idx);
134
135#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c+966−0
@@ -0,0 +1,966 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright 2023 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#include <linux/kref.h>
26#include <linux/slab.h>
27#include <linux/dma-fence-unwrap.h>
28
29#include <drm/drm_exec.h>
30#include <drm/drm_syncobj.h>
31
32#include "amdgpu.h"
33#include "amdgpu_userq_fence.h"
34
35static const struct dma_fence_ops amdgpu_userq_fence_ops;
36static struct kmem_cache *amdgpu_userq_fence_slab;
37
38int amdgpu_userq_fence_slab_init(void)
39{
40 amdgpu_userq_fence_slab = kmem_cache_create("amdgpu_userq_fence",
41 sizeof(struct amdgpu_userq_fence),
42 0,
43 SLAB_HWCACHE_ALIGN,
44 NULL);
45 if (!amdgpu_userq_fence_slab)
46 return -ENOMEM;
47
48 return 0;
49}
50
51void amdgpu_userq_fence_slab_fini(void)
52{
53 rcu_barrier();
54 kmem_cache_destroy(amdgpu_userq_fence_slab);
55}
56
57static inline struct amdgpu_userq_fence *to_amdgpu_userq_fence(struct dma_fence *f)
58{
59 if (!f || f->ops != &amdgpu_userq_fence_ops)
60 return NULL;
61
62 return container_of(f, struct amdgpu_userq_fence, base);
63}
64
65static u64 amdgpu_userq_fence_read(struct amdgpu_userq_fence_driver *fence_drv)
66{
67 return le64_to_cpu(*fence_drv->cpu_addr);
68}
69
70int amdgpu_userq_fence_driver_alloc(struct amdgpu_device *adev,
71 struct amdgpu_usermode_queue *userq)
72{
73 struct amdgpu_userq_fence_driver *fence_drv;
74 unsigned long flags;
75 int r;
76
77 fence_drv = kzalloc(sizeof(*fence_drv), GFP_KERNEL);
78 if (!fence_drv)
79 return -ENOMEM;
80
81 /* Acquire seq64 memory */
82 r = amdgpu_seq64_alloc(adev, &fence_drv->va, &fence_drv->gpu_addr,
83 &fence_drv->cpu_addr);
84 if (r)
85 goto free_fence_drv;
86
87 memset(fence_drv->cpu_addr, 0, sizeof(u64));
88
89 kref_init(&fence_drv->refcount);
90 INIT_LIST_HEAD(&fence_drv->fences);
91 spin_lock_init(&fence_drv->fence_list_lock);
92
93 fence_drv->adev = adev;
94 fence_drv->context = dma_fence_context_alloc(1);
95 get_task_comm(fence_drv->timeline_name, current);
96
97 xa_lock_irqsave(&adev->userq_xa, flags);
98 r = xa_err(__xa_store(&adev->userq_xa, userq->doorbell_index,
99 fence_drv, GFP_KERNEL));
100 xa_unlock_irqrestore(&adev->userq_xa, flags);
101 if (r)
102 goto free_seq64;
103
104 userq->fence_drv = fence_drv;
105
106 return 0;
107
108free_seq64:
109 amdgpu_seq64_free(adev, fence_drv->va);
110free_fence_drv:
111 kfree(fence_drv);
112
113 return r;
114}
115
116static void amdgpu_userq_walk_and_drop_fence_drv(struct xarray *xa)
117{
118 struct amdgpu_userq_fence_driver *fence_drv;
119 unsigned long index;
120
121 if (xa_empty(xa))
122 return;
123
124 xa_lock(xa);
125 xa_for_each(xa, index, fence_drv) {
126 __xa_erase(xa, index);
127 amdgpu_userq_fence_driver_put(fence_drv);
128 }
129
130 xa_unlock(xa);
131}
132
133void
134amdgpu_userq_fence_driver_free(struct amdgpu_usermode_queue *userq)
135{
136 amdgpu_userq_walk_and_drop_fence_drv(&userq->fence_drv_xa);
137 xa_destroy(&userq->fence_drv_xa);
138 /* Drop the fence_drv reference held by user queue */
139 amdgpu_userq_fence_driver_put(userq->fence_drv);
140}
141
142void amdgpu_userq_fence_driver_process(struct amdgpu_userq_fence_driver *fence_drv)
143{
144 struct amdgpu_userq_fence *userq_fence, *tmp;
145 struct dma_fence *fence;
146 u64 rptr;
147 int i;
148
149 if (!fence_drv)
150 return;
151
152 rptr = amdgpu_userq_fence_read(fence_drv);
153
154 spin_lock(&fence_drv->fence_list_lock);
155 list_for_each_entry_safe(userq_fence, tmp, &fence_drv->fences, link) {
156 fence = &userq_fence->base;
157
158 if (rptr < fence->seqno)
159 break;
160
161 dma_fence_signal(fence);
162
163 for (i = 0; i < userq_fence->fence_drv_array_count; i++)
164 amdgpu_userq_fence_driver_put(userq_fence->fence_drv_array[i]);
165
166 list_del(&userq_fence->link);
167 dma_fence_put(fence);
168 }
169 spin_unlock(&fence_drv->fence_list_lock);
170}
171
172void amdgpu_userq_fence_driver_destroy(struct kref *ref)
173{
174 struct amdgpu_userq_fence_driver *fence_drv = container_of(ref,
175 struct amdgpu_userq_fence_driver,
176 refcount);
177 struct amdgpu_userq_fence_driver *xa_fence_drv;
178 struct amdgpu_device *adev = fence_drv->adev;
179 struct amdgpu_userq_fence *fence, *tmp;
180 struct xarray *xa = &adev->userq_xa;
181 unsigned long index, flags;
182 struct dma_fence *f;
183
184 spin_lock(&fence_drv->fence_list_lock);
185 list_for_each_entry_safe(fence, tmp, &fence_drv->fences, link) {
186 f = &fence->base;
187
188 if (!dma_fence_is_signaled(f)) {
189 dma_fence_set_error(f, -ECANCELED);
190 dma_fence_signal(f);
191 }
192
193 list_del(&fence->link);
194 dma_fence_put(f);
195 }
196 spin_unlock(&fence_drv->fence_list_lock);
197
198 xa_lock_irqsave(xa, flags);
199 xa_for_each(xa, index, xa_fence_drv)
200 if (xa_fence_drv == fence_drv)
201 __xa_erase(xa, index);
202 xa_unlock_irqrestore(xa, flags);
203
204 /* Free seq64 memory */
205 amdgpu_seq64_free(adev, fence_drv->va);
206 kfree(fence_drv);
207}
208
209void amdgpu_userq_fence_driver_get(struct amdgpu_userq_fence_driver *fence_drv)
210{
211 kref_get(&fence_drv->refcount);
212}
213
214void amdgpu_userq_fence_driver_put(struct amdgpu_userq_fence_driver *fence_drv)
215{
216 kref_put(&fence_drv->refcount, amdgpu_userq_fence_driver_destroy);
217}
218
219static int amdgpu_userq_fence_alloc(struct amdgpu_userq_fence **userq_fence)
220{
221 *userq_fence = kmem_cache_alloc(amdgpu_userq_fence_slab, GFP_ATOMIC);
222 return *userq_fence ? 0 : -ENOMEM;
223}
224
225static int amdgpu_userq_fence_create(struct amdgpu_usermode_queue *userq,
226 struct amdgpu_userq_fence *userq_fence,
227 u64 seq, struct dma_fence **f)
228{
229 struct amdgpu_userq_fence_driver *fence_drv;
230 struct dma_fence *fence;
231 unsigned long flags;
232
233 fence_drv = userq->fence_drv;
234 if (!fence_drv)
235 return -EINVAL;
236
237 spin_lock_init(&userq_fence->lock);
238 INIT_LIST_HEAD(&userq_fence->link);
239 fence = &userq_fence->base;
240 userq_fence->fence_drv = fence_drv;
241
242 dma_fence_init(fence, &amdgpu_userq_fence_ops, &userq_fence->lock,
243 fence_drv->context, seq);
244
245 amdgpu_userq_fence_driver_get(fence_drv);
246 dma_fence_get(fence);
247
248 if (!xa_empty(&userq->fence_drv_xa)) {
249 struct amdgpu_userq_fence_driver *stored_fence_drv;
250 unsigned long index, count = 0;
251 int i = 0;
252
253 xa_lock(&userq->fence_drv_xa);
254 xa_for_each(&userq->fence_drv_xa, index, stored_fence_drv)
255 count++;
256
257 userq_fence->fence_drv_array =
258 kvmalloc_array(count,
259 sizeof(struct amdgpu_userq_fence_driver *),
260 GFP_ATOMIC);
261
262 if (userq_fence->fence_drv_array) {
263 xa_for_each(&userq->fence_drv_xa, index, stored_fence_drv) {
264 userq_fence->fence_drv_array[i] = stored_fence_drv;
265 __xa_erase(&userq->fence_drv_xa, index);
266 i++;
267 }
268 }
269
270 userq_fence->fence_drv_array_count = i;
271 xa_unlock(&userq->fence_drv_xa);
272 } else {
273 userq_fence->fence_drv_array = NULL;
274 userq_fence->fence_drv_array_count = 0;
275 }
276
277 /* Check if hardware has already processed the job */
278 spin_lock_irqsave(&fence_drv->fence_list_lock, flags);
279 if (!dma_fence_is_signaled_locked(fence))
280 list_add_tail(&userq_fence->link, &fence_drv->fences);
281 else
282 dma_fence_put(fence);
283
284 spin_unlock_irqrestore(&fence_drv->fence_list_lock, flags);
285
286 *f = fence;
287
288 return 0;
289}
290
291static const char *amdgpu_userq_fence_get_driver_name(struct dma_fence *f)
292{
293 return "amdgpu_userq_fence";
294}
295
296static const char *amdgpu_userq_fence_get_timeline_name(struct dma_fence *f)
297{
298 struct amdgpu_userq_fence *fence = to_amdgpu_userq_fence(f);
299
300 return fence->fence_drv->timeline_name;
301}
302
303static bool amdgpu_userq_fence_signaled(struct dma_fence *f)
304{
305 struct amdgpu_userq_fence *fence = to_amdgpu_userq_fence(f);
306 struct amdgpu_userq_fence_driver *fence_drv = fence->fence_drv;
307 u64 rptr, wptr;
308
309 rptr = amdgpu_userq_fence_read(fence_drv);
310 wptr = fence->base.seqno;
311
312 if (rptr >= wptr)
313 return true;
314
315 return false;
316}
317
318static void amdgpu_userq_fence_free(struct rcu_head *rcu)
319{
320 struct dma_fence *fence = container_of(rcu, struct dma_fence, rcu);
321 struct amdgpu_userq_fence *userq_fence = to_amdgpu_userq_fence(fence);
322 struct amdgpu_userq_fence_driver *fence_drv = userq_fence->fence_drv;
323
324 /* Release the fence driver reference */
325 amdgpu_userq_fence_driver_put(fence_drv);
326
327 kvfree(userq_fence->fence_drv_array);
328 kmem_cache_free(amdgpu_userq_fence_slab, userq_fence);
329}
330
331static void amdgpu_userq_fence_release(struct dma_fence *f)
332{
333 call_rcu(&f->rcu, amdgpu_userq_fence_free);
334}
335
336static const struct dma_fence_ops amdgpu_userq_fence_ops = {
337 .use_64bit_seqno = true,
338 .get_driver_name = amdgpu_userq_fence_get_driver_name,
339 .get_timeline_name = amdgpu_userq_fence_get_timeline_name,
340 .signaled = amdgpu_userq_fence_signaled,
341 .release = amdgpu_userq_fence_release,
342};
343
344/**
345 * amdgpu_userq_fence_read_wptr - Read the userq wptr value
346 *
347 * @queue: user mode queue structure pointer
348 * @wptr: write pointer value
349 *
350 * Read the wptr value from userq's MQD. The userq signal IOCTL
351 * creates a dma_fence for the shared buffers that expects the
352 * RPTR value written to seq64 memory >= WPTR.
353 *
354 * Returns wptr value on success, error on failure.
355 */
356static int amdgpu_userq_fence_read_wptr(struct amdgpu_usermode_queue *queue,
357 u64 *wptr)
358{
359 struct amdgpu_bo_va_mapping *mapping;
360 struct amdgpu_bo *bo;
361 u64 addr, *ptr;
362 int r;
363
364 r = amdgpu_bo_reserve(queue->vm->root.bo, false);
365 if (r)
366 return r;
367
368 addr = queue->userq_prop->wptr_gpu_addr;
369 addr &= AMDGPU_GMC_HOLE_MASK;
370
371 mapping = amdgpu_vm_bo_lookup_mapping(queue->vm, addr >> PAGE_SHIFT);
372 if (!mapping) {
373 amdgpu_bo_unreserve(queue->vm->root.bo);
374 DRM_ERROR("Failed to lookup amdgpu_bo_va_mapping\n");
375 return -EINVAL;
376 }
377
378 bo = amdgpu_bo_ref(mapping->bo_va->base.bo);
379 amdgpu_bo_unreserve(queue->vm->root.bo);
380 r = amdgpu_bo_reserve(bo, true);
381 if (r) {
382 DRM_ERROR("Failed to reserve userqueue wptr bo");
383 return r;
384 }
385
386 r = amdgpu_bo_kmap(bo, (void **)&ptr);
387 if (r) {
388 DRM_ERROR("Failed mapping the userqueue wptr bo");
389 goto map_error;
390 }
391
392 *wptr = le64_to_cpu(*ptr);
393
394 amdgpu_bo_kunmap(bo);
395 amdgpu_bo_unreserve(bo);
396 amdgpu_bo_unref(&bo);
397
398 return 0;
399
400map_error:
401 amdgpu_bo_unreserve(bo);
402 amdgpu_bo_unref(&bo);
403
404 return r;
405}
406
407static void amdgpu_userq_fence_cleanup(struct dma_fence *fence)
408{
409 dma_fence_put(fence);
410}
411
412int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data,
413 struct drm_file *filp)
414{
415 struct amdgpu_fpriv *fpriv = filp->driver_priv;
416 struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr;
417 struct drm_amdgpu_userq_signal *args = data;
418 struct drm_gem_object **gobj_write = NULL;
419 struct drm_gem_object **gobj_read = NULL;
420 struct amdgpu_usermode_queue *queue;
421 struct amdgpu_userq_fence *userq_fence;
422 struct drm_syncobj **syncobj = NULL;
423 u32 *bo_handles_write, num_write_bo_handles;
424 u32 *syncobj_handles, num_syncobj_handles;
425 u32 *bo_handles_read, num_read_bo_handles;
426 int r, i, entry, rentry, wentry;
427 struct dma_fence *fence;
428 struct drm_exec exec;
429 u64 wptr;
430
431 num_syncobj_handles = args->num_syncobj_handles;
432 syncobj_handles = memdup_user(u64_to_user_ptr(args->syncobj_handles),
433 sizeof(u32) * num_syncobj_handles);
434 if (IS_ERR(syncobj_handles))
435 return PTR_ERR(syncobj_handles);
436
437 /* Array of pointers to the looked up syncobjs */
438 syncobj = kmalloc_array(num_syncobj_handles, sizeof(*syncobj), GFP_KERNEL);
439 if (!syncobj) {
440 r = -ENOMEM;
441 goto free_syncobj_handles;
442 }
443
444 for (entry = 0; entry < num_syncobj_handles; entry++) {
445 syncobj[entry] = drm_syncobj_find(filp, syncobj_handles[entry]);
446 if (!syncobj[entry]) {
447 r = -ENOENT;
448 goto free_syncobj;
449 }
450 }
451
452 num_read_bo_handles = args->num_bo_read_handles;
453 bo_handles_read = memdup_user(u64_to_user_ptr(args->bo_read_handles),
454 sizeof(u32) * num_read_bo_handles);
455 if (IS_ERR(bo_handles_read)) {
456 r = PTR_ERR(bo_handles_read);
457 goto free_syncobj;
458 }
459
460 /* Array of pointers to the GEM read objects */
461 gobj_read = kmalloc_array(num_read_bo_handles, sizeof(*gobj_read), GFP_KERNEL);
462 if (!gobj_read) {
463 r = -ENOMEM;
464 goto free_bo_handles_read;
465 }
466
467 for (rentry = 0; rentry < num_read_bo_handles; rentry++) {
468 gobj_read[rentry] = drm_gem_object_lookup(filp, bo_handles_read[rentry]);
469 if (!gobj_read[rentry]) {
470 r = -ENOENT;
471 goto put_gobj_read;
472 }
473 }
474
475 num_write_bo_handles = args->num_bo_write_handles;
476 bo_handles_write = memdup_user(u64_to_user_ptr(args->bo_write_handles),
477 sizeof(u32) * num_write_bo_handles);
478 if (IS_ERR(bo_handles_write)) {
479 r = PTR_ERR(bo_handles_write);
480 goto put_gobj_read;
481 }
482
483 /* Array of pointers to the GEM write objects */
484 gobj_write = kmalloc_array(num_write_bo_handles, sizeof(*gobj_write), GFP_KERNEL);
485 if (!gobj_write) {
486 r = -ENOMEM;
487 goto free_bo_handles_write;
488 }
489
490 for (wentry = 0; wentry < num_write_bo_handles; wentry++) {
491 gobj_write[wentry] = drm_gem_object_lookup(filp, bo_handles_write[wentry]);
492 if (!gobj_write[wentry]) {
493 r = -ENOENT;
494 goto put_gobj_write;
495 }
496 }
497
498 /* Retrieve the user queue */
499 queue = idr_find(&userq_mgr->userq_idr, args->queue_id);
500 if (!queue) {
501 r = -ENOENT;
502 goto put_gobj_write;
503 }
504
505 r = amdgpu_userq_fence_read_wptr(queue, &wptr);
506 if (r)
507 goto put_gobj_write;
508
509 r = amdgpu_userq_fence_alloc(&userq_fence);
510 if (r)
511 goto put_gobj_write;
512
513 /* We are here means UQ is active, make sure the eviction fence is valid */
514 amdgpu_userq_ensure_ev_fence(&fpriv->userq_mgr, &fpriv->evf_mgr);
515
516 /* Create a new fence */
517 r = amdgpu_userq_fence_create(queue, userq_fence, wptr, &fence);
518 if (r) {
519 mutex_unlock(&userq_mgr->userq_mutex);
520 kmem_cache_free(amdgpu_userq_fence_slab, userq_fence);
521 goto put_gobj_write;
522 }
523
524 dma_fence_put(queue->last_fence);
525 queue->last_fence = dma_fence_get(fence);
526 mutex_unlock(&userq_mgr->userq_mutex);
527
528 drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,
529 (num_read_bo_handles + num_write_bo_handles));
530
531 /* Lock all BOs with retry handling */
532 drm_exec_until_all_locked(&exec) {
533 r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1);
534 drm_exec_retry_on_contention(&exec);
535 if (r) {
536 amdgpu_userq_fence_cleanup(fence);
537 goto exec_fini;
538 }
539
540 r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1);
541 drm_exec_retry_on_contention(&exec);
542 if (r) {
543 amdgpu_userq_fence_cleanup(fence);
544 goto exec_fini;
545 }
546 }
547
548 for (i = 0; i < num_read_bo_handles; i++) {
549 if (!gobj_read || !gobj_read[i]->resv)
550 continue;
551
552 dma_resv_add_fence(gobj_read[i]->resv, fence,
553 DMA_RESV_USAGE_READ);
554 }
555
556 for (i = 0; i < num_write_bo_handles; i++) {
557 if (!gobj_write || !gobj_write[i]->resv)
558 continue;
559
560 dma_resv_add_fence(gobj_write[i]->resv, fence,
561 DMA_RESV_USAGE_WRITE);
562 }
563
564 /* Add the created fence to syncobj/BO's */
565 for (i = 0; i < num_syncobj_handles; i++)
566 drm_syncobj_replace_fence(syncobj[i], fence);
567
568 /* drop the reference acquired in fence creation function */
569 dma_fence_put(fence);
570
571exec_fini:
572 drm_exec_fini(&exec);
573put_gobj_write:
574 while (wentry-- > 0)
575 drm_gem_object_put(gobj_write[wentry]);
576 kfree(gobj_write);
577free_bo_handles_write:
578 kfree(bo_handles_write);
579put_gobj_read:
580 while (rentry-- > 0)
581 drm_gem_object_put(gobj_read[rentry]);
582 kfree(gobj_read);
583free_bo_handles_read:
584 kfree(bo_handles_read);
585free_syncobj:
586 while (entry-- > 0)
587 if (syncobj[entry])
588 drm_syncobj_put(syncobj[entry]);
589 kfree(syncobj);
590free_syncobj_handles:
591 kfree(syncobj_handles);
592
593 return r;
594}
595
596int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data,
597 struct drm_file *filp)
598{
599 u32 *syncobj_handles, *timeline_points, *timeline_handles, *bo_handles_read, *bo_handles_write;
600 u32 num_syncobj, num_read_bo_handles, num_write_bo_handles;
601 struct drm_amdgpu_userq_fence_info *fence_info = NULL;
602 struct drm_amdgpu_userq_wait *wait_info = data;
603 struct amdgpu_fpriv *fpriv = filp->driver_priv;
604 struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr;
605 struct amdgpu_usermode_queue *waitq;
606 struct drm_gem_object **gobj_write;
607 struct drm_gem_object **gobj_read;
608 struct dma_fence **fences = NULL;
609 u16 num_points, num_fences = 0;
610 int r, i, rentry, wentry, cnt;
611 struct drm_exec exec;
612
613 num_read_bo_handles = wait_info->num_bo_read_handles;
614 bo_handles_read = memdup_user(u64_to_user_ptr(wait_info->bo_read_handles),
615 sizeof(u32) * num_read_bo_handles);
616 if (IS_ERR(bo_handles_read))
617 return PTR_ERR(bo_handles_read);
618
619 num_write_bo_handles = wait_info->num_bo_write_handles;
620 bo_handles_write = memdup_user(u64_to_user_ptr(wait_info->bo_write_handles),
621 sizeof(u32) * num_write_bo_handles);
622 if (IS_ERR(bo_handles_write)) {
623 r = PTR_ERR(bo_handles_write);
624 goto free_bo_handles_read;
625 }
626
627 num_syncobj = wait_info->num_syncobj_handles;
628 syncobj_handles = memdup_user(u64_to_user_ptr(wait_info->syncobj_handles),
629 sizeof(u32) * num_syncobj);
630 if (IS_ERR(syncobj_handles)) {
631 r = PTR_ERR(syncobj_handles);
632 goto free_bo_handles_write;
633 }
634
635 num_points = wait_info->num_syncobj_timeline_handles;
636 timeline_handles = memdup_user(u64_to_user_ptr(wait_info->syncobj_timeline_handles),
637 sizeof(u32) * num_points);
638 if (IS_ERR(timeline_handles)) {
639 r = PTR_ERR(timeline_handles);
640 goto free_syncobj_handles;
641 }
642
643 timeline_points = memdup_user(u64_to_user_ptr(wait_info->syncobj_timeline_points),
644 sizeof(u32) * num_points);
645 if (IS_ERR(timeline_points)) {
646 r = PTR_ERR(timeline_points);
647 goto free_timeline_handles;
648 }
649
650 gobj_read = kmalloc_array(num_read_bo_handles, sizeof(*gobj_read), GFP_KERNEL);
651 if (!gobj_read) {
652 r = -ENOMEM;
653 goto free_timeline_points;
654 }
655
656 for (rentry = 0; rentry < num_read_bo_handles; rentry++) {
657 gobj_read[rentry] = drm_gem_object_lookup(filp, bo_handles_read[rentry]);
658 if (!gobj_read[rentry]) {
659 r = -ENOENT;
660 goto put_gobj_read;
661 }
662 }
663
664 gobj_write = kmalloc_array(num_write_bo_handles, sizeof(*gobj_write), GFP_KERNEL);
665 if (!gobj_write) {
666 r = -ENOMEM;
667 goto put_gobj_read;
668 }
669
670 for (wentry = 0; wentry < num_write_bo_handles; wentry++) {
671 gobj_write[wentry] = drm_gem_object_lookup(filp, bo_handles_write[wentry]);
672 if (!gobj_write[wentry]) {
673 r = -ENOENT;
674 goto put_gobj_write;
675 }
676 }
677
678 drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT,
679 (num_read_bo_handles + num_write_bo_handles));
680
681 /* Lock all BOs with retry handling */
682 drm_exec_until_all_locked(&exec) {
683 r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1);
684 drm_exec_retry_on_contention(&exec);
685 if (r) {
686 drm_exec_fini(&exec);
687 goto put_gobj_write;
688 }
689
690 r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1);
691 drm_exec_retry_on_contention(&exec);
692 if (r) {
693 drm_exec_fini(&exec);
694 goto put_gobj_write;
695 }
696 }
697
698 if (!wait_info->num_fences) {
699 if (num_points) {
700 struct dma_fence_unwrap iter;
701 struct dma_fence *fence;
702 struct dma_fence *f;
703
704 for (i = 0; i < num_points; i++) {
705 r = drm_syncobj_find_fence(filp, timeline_handles[i],
706 timeline_points[i],
707 DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
708 &fence);
709 if (r)
710 goto exec_fini;
711
712 dma_fence_unwrap_for_each(f, &iter, fence)
713 num_fences++;
714
715 dma_fence_put(fence);
716 }
717 }
718
719 /* Count syncobj's fence */
720 for (i = 0; i < num_syncobj; i++) {
721 struct dma_fence *fence;
722
723 r = drm_syncobj_find_fence(filp, syncobj_handles[i],
724 0,
725 DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
726 &fence);
727 if (r)
728 goto exec_fini;
729
730 num_fences++;
731 dma_fence_put(fence);
732 }
733
734 /* Count GEM objects fence */
735 for (i = 0; i < num_read_bo_handles; i++) {
736 struct dma_resv_iter resv_cursor;
737 struct dma_fence *fence;
738
739 dma_resv_for_each_fence(&resv_cursor, gobj_read[i]->resv,
740 DMA_RESV_USAGE_READ, fence)
741 num_fences++;
742 }
743
744 for (i = 0; i < num_write_bo_handles; i++) {
745 struct dma_resv_iter resv_cursor;
746 struct dma_fence *fence;
747
748 dma_resv_for_each_fence(&resv_cursor, gobj_write[i]->resv,
749 DMA_RESV_USAGE_WRITE, fence)
750 num_fences++;
751 }
752
753 /*
754 * Passing num_fences = 0 means that userspace doesn't want to
755 * retrieve userq_fence_info. If num_fences = 0 we skip filling
756 * userq_fence_info and return the actual number of fences on
757 * args->num_fences.
758 */
759 wait_info->num_fences = num_fences;
760 } else {
761 /* Array of fence info */
762 fence_info = kmalloc_array(wait_info->num_fences, sizeof(*fence_info), GFP_KERNEL);
763 if (!fence_info) {
764 r = -ENOMEM;
765 goto exec_fini;
766 }
767
768 /* Array of fences */
769 fences = kmalloc_array(wait_info->num_fences, sizeof(*fences), GFP_KERNEL);
770 if (!fences) {
771 r = -ENOMEM;
772 goto free_fence_info;
773 }
774
775 /* Retrieve GEM read objects fence */
776 for (i = 0; i < num_read_bo_handles; i++) {
777 struct dma_resv_iter resv_cursor;
778 struct dma_fence *fence;
779
780 dma_resv_for_each_fence(&resv_cursor, gobj_read[i]->resv,
781 DMA_RESV_USAGE_READ, fence) {
782 if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
783 r = -EINVAL;
784 goto free_fences;
785 }
786
787 fences[num_fences++] = fence;
788 dma_fence_get(fence);
789 }
790 }
791
792 /* Retrieve GEM write objects fence */
793 for (i = 0; i < num_write_bo_handles; i++) {
794 struct dma_resv_iter resv_cursor;
795 struct dma_fence *fence;
796
797 dma_resv_for_each_fence(&resv_cursor, gobj_write[i]->resv,
798 DMA_RESV_USAGE_WRITE, fence) {
799 if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
800 r = -EINVAL;
801 goto free_fences;
802 }
803
804 fences[num_fences++] = fence;
805 dma_fence_get(fence);
806 }
807 }
808
809 if (num_points) {
810 struct dma_fence_unwrap iter;
811 struct dma_fence *fence;
812 struct dma_fence *f;
813
814 for (i = 0; i < num_points; i++) {
815 r = drm_syncobj_find_fence(filp, timeline_handles[i],
816 timeline_points[i],
817 DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
818 &fence);
819 if (r)
820 goto free_fences;
821
822 dma_fence_unwrap_for_each(f, &iter, fence) {
823 if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
824 r = -EINVAL;
825 goto free_fences;
826 }
827
828 dma_fence_get(f);
829 fences[num_fences++] = f;
830 }
831
832 dma_fence_put(fence);
833 }
834 }
835
836 /* Retrieve syncobj's fence */
837 for (i = 0; i < num_syncobj; i++) {
838 struct dma_fence *fence;
839
840 r = drm_syncobj_find_fence(filp, syncobj_handles[i],
841 0,
842 DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT,
843 &fence);
844 if (r)
845 goto free_fences;
846
847 if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) {
848 r = -EINVAL;
849 goto free_fences;
850 }
851
852 fences[num_fences++] = fence;
853 }
854
855 /*
856 * Keep only the latest fences to reduce the number of values
857 * given back to userspace.
858 */
859 num_fences = dma_fence_dedup_array(fences, num_fences);
860
861 waitq = idr_find(&userq_mgr->userq_idr, wait_info->waitq_id);
862 if (!waitq)
863 goto free_fences;
864
865 for (i = 0, cnt = 0; i < num_fences; i++) {
866 struct amdgpu_userq_fence_driver *fence_drv;
867 struct amdgpu_userq_fence *userq_fence;
868 u32 index;
869
870 userq_fence = to_amdgpu_userq_fence(fences[i]);
871 if (!userq_fence) {
872 /*
873 * Just waiting on other driver fences should
874 * be good for now
875 */
876 r = dma_fence_wait(fences[i], true);
877 if (r) {
878 dma_fence_put(fences[i]);
879 goto free_fences;
880 }
881
882 dma_fence_put(fences[i]);
883 continue;
884 }
885
886 fence_drv = userq_fence->fence_drv;
887 /*
888 * We need to make sure the user queue release their reference
889 * to the fence drivers at some point before queue destruction.
890 * Otherwise, we would gather those references until we don't
891 * have any more space left and crash.
892 */
893 r = xa_alloc(&waitq->fence_drv_xa, &index, fence_drv,
894 xa_limit_32b, GFP_KERNEL);
895 if (r)
896 goto free_fences;
897
898 amdgpu_userq_fence_driver_get(fence_drv);
899
900 /* Store drm syncobj's gpu va address and value */
901 fence_info[cnt].va = fence_drv->va;
902 fence_info[cnt].value = fences[i]->seqno;
903
904 dma_fence_put(fences[i]);
905 /* Increment the actual userq fence count */
906 cnt++;
907 }
908
909 wait_info->num_fences = cnt;
910 /* Copy userq fence info to user space */
911 if (copy_to_user(u64_to_user_ptr(wait_info->out_fences),
912 fence_info, wait_info->num_fences * sizeof(*fence_info))) {
913 r = -EFAULT;
914 goto free_fences;
915 }
916
917 kfree(fences);
918 kfree(fence_info);
919 }
920
921 drm_exec_fini(&exec);
922 for (i = 0; i < num_read_bo_handles; i++)
923 drm_gem_object_put(gobj_read[i]);
924 kfree(gobj_read);
925
926 for (i = 0; i < num_write_bo_handles; i++)
927 drm_gem_object_put(gobj_write[i]);
928 kfree(gobj_write);
929
930 kfree(timeline_points);
931 kfree(timeline_handles);
932 kfree(syncobj_handles);
933 kfree(bo_handles_write);
934 kfree(bo_handles_read);
935
936 return 0;
937
938free_fences:
939 while (num_fences-- > 0)
940 dma_fence_put(fences[num_fences]);
941 kfree(fences);
942free_fence_info:
943 kfree(fence_info);
944exec_fini:
945 drm_exec_fini(&exec);
946put_gobj_write:
947 while (wentry-- > 0)
948 drm_gem_object_put(gobj_write[wentry]);
949 kfree(gobj_write);
950put_gobj_read:
951 while (rentry-- > 0)
952 drm_gem_object_put(gobj_read[rentry]);
953 kfree(gobj_read);
954free_timeline_points:
955 kfree(timeline_points);
956free_timeline_handles:
957 kfree(timeline_handles);
958free_syncobj_handles:
959 kfree(syncobj_handles);
960free_bo_handles_write:
961 kfree(bo_handles_write);
962free_bo_handles_read:
963 kfree(bo_handles_read);
964
965 return r;
966}
drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.h+76−0
@@ -0,0 +1,76 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright 2023 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#ifndef __AMDGPU_USERQ_FENCE_H__
26#define __AMDGPU_USERQ_FENCE_H__
27
28#include <linux/types.h>
29
30#include "amdgpu_userq.h"
31
32struct amdgpu_userq_fence {
33 struct dma_fence base;
34 /*
35 * This lock is necessary to synchronize the
36 * userqueue dma fence operations.
37 */
38 spinlock_t lock;
39 struct list_head link;
40 unsigned long fence_drv_array_count;
41 struct amdgpu_userq_fence_driver *fence_drv;
42 struct amdgpu_userq_fence_driver **fence_drv_array;
43};
44
45struct amdgpu_userq_fence_driver {
46 struct kref refcount;
47 u64 va;
48 u64 gpu_addr;
49 u64 *cpu_addr;
50 u64 context;
51 /*
52 * This lock is necesaary to synchronize the access
53 * to the fences list by the fence driver.
54 */
55 spinlock_t fence_list_lock;
56 struct list_head fences;
57 struct amdgpu_device *adev;
58 char timeline_name[TASK_COMM_LEN];
59};
60
61int amdgpu_userq_fence_slab_init(void);
62void amdgpu_userq_fence_slab_fini(void);
63
64void amdgpu_userq_fence_driver_get(struct amdgpu_userq_fence_driver *fence_drv);
65void amdgpu_userq_fence_driver_put(struct amdgpu_userq_fence_driver *fence_drv);
66int amdgpu_userq_fence_driver_alloc(struct amdgpu_device *adev,
67 struct amdgpu_usermode_queue *userq);
68void amdgpu_userq_fence_driver_free(struct amdgpu_usermode_queue *userq);
69void amdgpu_userq_fence_driver_process(struct amdgpu_userq_fence_driver *fence_drv);
70void amdgpu_userq_fence_driver_destroy(struct kref *ref);
71int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data,
72 struct drm_file *filp);
73int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data,
74 struct drm_file *filp);
75
76#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c+2−2
@@ -353,9 +353,9 @@ int amdgpu_vcn_suspend(struct amdgpu_device *adev, int i) · 352 unmodified lines
353 353
354 cancel_delayed_work_sync(&adev->vcn.inst[i].idle_work);354 cancel_delayed_work_sync(&adev->vcn.inst[i].idle_work);
355 355
356 /* err_event_athub will corrupt VCPU buffer, so we need to356 /* err_event_athub and dpc recovery will corrupt VCPU buffer, so we need to
357 * restore fw data and clear buffer in amdgpu_vcn_resume() */357 * restore fw data and clear buffer in amdgpu_vcn_resume() */
358 if (in_ras_intr)358 if (in_ras_intr || adev->pcie_reset_ctx.in_link_reset)
359 return 0;359 return 0;
360 360
361 return amdgpu_vcn_save_vcpu_bo_inst(adev, i);361 return amdgpu_vcn_save_vcpu_bo_inst(adev, i);
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h+0−1
@@ -66,7 +66,6 @@ · 65 unmodified lines
66#define VCN_ENC_CMD_REG_WAIT 0x0000000c66#define VCN_ENC_CMD_REG_WAIT 0x0000000c
67 67
68#define VCN_AON_SOC_ADDRESS_2_0 0x1f80068#define VCN_AON_SOC_ADDRESS_2_0 0x1f800
69#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
70#define VCN_VID_IP_ADDRESS_2_0 0x069#define VCN_VID_IP_ADDRESS_2_0 0x0
71#define VCN_AON_IP_ADDRESS_2_0 0x3000070#define VCN_AON_IP_ADDRESS_2_0 0x30000
72 71
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c+25−4
@@ -1323,6 +1323,9 @@ static int amdgpu_virt_req_ras_err_count_internal(struct amdgpu_device *adev, bo · 1322 unmodified lines
1323{1323{
1324 struct amdgpu_virt *virt = &adev->virt;1324 struct amdgpu_virt *virt = &adev->virt;
1325 1325
1326 if (!virt->ops || !virt->ops->req_ras_err_count)
1327 return -EOPNOTSUPP;
1328
1326 /* Host allows 15 ras telemetry requests per 60 seconds. Afterwhich, the Host1329 /* Host allows 15 ras telemetry requests per 60 seconds. Afterwhich, the Host
1327 * will ignore incoming guest messages. Ratelimit the guest messages to1330 * will ignore incoming guest messages. Ratelimit the guest messages to
1328 * prevent guest self DOS.1331 * prevent guest self DOS.
@@ -1378,14 +1381,16 @@ amdgpu_virt_write_cpers_to_ring(struct amdgpu_device *adev, · 49 unmodified lines
1378 used_size = host_telemetry->header.used_size;1381 used_size = host_telemetry->header.used_size;
1379 1382
1380 if (used_size > (AMD_SRIOV_RAS_TELEMETRY_SIZE_KB << 10))1383 if (used_size > (AMD_SRIOV_RAS_TELEMETRY_SIZE_KB << 10))
1381 return 0;1384 return -EINVAL;
1382 1385
1383 cper_dump = kmemdup(&host_telemetry->body.cper_dump, used_size, GFP_KERNEL);1386 cper_dump = kmemdup(&host_telemetry->body.cper_dump, used_size, GFP_KERNEL);
1384 if (!cper_dump)1387 if (!cper_dump)
1385 return -ENOMEM;1388 return -ENOMEM;
1386 1389
1387 if (checksum != amd_sriov_msg_checksum(cper_dump, used_size, 0, 0))1390 if (checksum != amd_sriov_msg_checksum(cper_dump, used_size, 0, 0)) {
1391 ret = -EINVAL;
1388 goto out;1392 goto out;
1393 }
1389 1394
1390 *more = cper_dump->more;1395 *more = cper_dump->more;
1391 1396
@@ -1425,7 +1430,7 @@ static int amdgpu_virt_req_ras_cper_dump_internal(struct amdgpu_device *adev) · 33 unmodified lines
1425 int ret = 0;1430 int ret = 0;
1426 uint32_t more = 0;1431 uint32_t more = 0;
1427 1432
1428 if (!amdgpu_sriov_ras_cper_en(adev))1433 if (!virt->ops || !virt->ops->req_ras_cper_dump)
1429 return -EOPNOTSUPP;1434 return -EOPNOTSUPP;
1430 1435
1431 do {1436 do {
@@ -1434,7 +1439,7 @@ static int amdgpu_virt_req_ras_cper_dump_internal(struct amdgpu_device *adev) · 2 unmodified lines
1434 adev, virt->fw_reserve.ras_telemetry, &more);1439 adev, virt->fw_reserve.ras_telemetry, &more);
1435 else1440 else
1436 ret = 0;1441 ret = 0;
1437 } while (more);1442 } while (more && !ret);
1438 1443
1439 return ret;1444 return ret;
1440}1445}
@@ -1444,6 +1449,9 @@ int amdgpu_virt_req_ras_cper_dump(struct amdgpu_device *adev, bool force_update) · 3 unmodified lines
1444 struct amdgpu_virt *virt = &adev->virt;1449 struct amdgpu_virt *virt = &adev->virt;
1445 int ret = 0;1450 int ret = 0;
1446 1451
1452 if (!amdgpu_sriov_ras_cper_en(adev))
1453 return -EOPNOTSUPP;
1454
1447 if ((__ratelimit(&virt->ras.ras_cper_dump_rs) || force_update) &&1455 if ((__ratelimit(&virt->ras.ras_cper_dump_rs) || force_update) &&
1448 down_read_trylock(&adev->reset_domain->sem)) {1456 down_read_trylock(&adev->reset_domain->sem)) {
1449 mutex_lock(&virt->ras.ras_telemetry_mutex);1457 mutex_lock(&virt->ras.ras_telemetry_mutex);
@@ -1480,3 +1488,16 @@ bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev, · 30 unmodified lines
1480 1488
1481 return true;1489 return true;
1482}1490}
1491
1492/*
1493 * amdgpu_virt_request_bad_pages() - request bad pages
1494 * @adev: amdgpu device.
1495 * Send command to GPU hypervisor to write new bad pages into the shared PF2VF region
1496 */
1497void amdgpu_virt_request_bad_pages(struct amdgpu_device *adev)
1498{
1499 struct amdgpu_virt *virt = &adev->virt;
1500
1501 if (virt->ops && virt->ops->req_bad_pages)
1502 virt->ops->req_bad_pages(adev);
1503}
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h+14−3
@@ -97,6 +97,7 @@ struct amdgpu_virt_ops { · 96 unmodified lines
97 bool (*rcvd_ras_intr)(struct amdgpu_device *adev);97 bool (*rcvd_ras_intr)(struct amdgpu_device *adev);
98 int (*req_ras_err_count)(struct amdgpu_device *adev);98 int (*req_ras_err_count)(struct amdgpu_device *adev);
99 int (*req_ras_cper_dump)(struct amdgpu_device *adev, u64 vf_rptr);99 int (*req_ras_cper_dump)(struct amdgpu_device *adev, u64 vf_rptr);
100 int (*req_bad_pages)(struct amdgpu_device *adev);
100};101};
101 102
102/*103/*
@@ -146,11 +147,13 @@ enum AMDGIM_FEATURE_FLAG { · 43 unmodified lines
146 147
147enum AMDGIM_REG_ACCESS_FLAG {148enum AMDGIM_REG_ACCESS_FLAG {
148 /* Use PSP to program IH_RB_CNTL */149 /* Use PSP to program IH_RB_CNTL */
149 AMDGIM_FEATURE_IH_REG_PSP_EN = (1 << 0),150 AMDGIM_FEATURE_IH_REG_PSP_EN = (1 << 0),
150 /* Use RLC to program MMHUB regs */151 /* Use RLC to program MMHUB regs */
151 AMDGIM_FEATURE_MMHUB_REG_RLC_EN = (1 << 1),152 AMDGIM_FEATURE_MMHUB_REG_RLC_EN = (1 << 1),
152 /* Use RLC to program GC regs */153 /* Use RLC to program GC regs */
153 AMDGIM_FEATURE_GC_REG_RLC_EN = (1 << 2),154 AMDGIM_FEATURE_GC_REG_RLC_EN = (1 << 2),
155 /* Use PSP to program L1_TLB_CNTL*/
156 AMDGIM_FEATURE_L1_TLB_CNTL_PSP_EN = (1 << 3),
154};157};
155 158
156struct amdgim_pf2vf_info_v1 {159struct amdgim_pf2vf_info_v1 {
@@ -260,7 +263,10 @@ struct amdgpu_virt { · 103 unmodified lines
260 uint32_t reg_val_offs;263 uint32_t reg_val_offs;
261 struct amdgpu_irq_src ack_irq;264 struct amdgpu_irq_src ack_irq;
262 struct amdgpu_irq_src rcv_irq;265 struct amdgpu_irq_src rcv_irq;
266
263 struct work_struct flr_work;267 struct work_struct flr_work;
268 struct work_struct bad_pages_work;
269
264 struct amdgpu_mm_table mm_table;270 struct amdgpu_mm_table mm_table;
265 const struct amdgpu_virt_ops *ops;271 const struct amdgpu_virt_ops *ops;
266 struct amdgpu_vf_error_buffer vf_errors;272 struct amdgpu_vf_error_buffer vf_errors;
@@ -330,6 +336,10 @@ struct amdgpu_video_codec_info; · 63 unmodified lines
330(amdgpu_sriov_vf((adev)) && \336(amdgpu_sriov_vf((adev)) && \
331 ((adev)->virt.reg_access & (AMDGIM_FEATURE_GC_REG_RLC_EN)))337 ((adev)->virt.reg_access & (AMDGIM_FEATURE_GC_REG_RLC_EN)))
332 338
339#define amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev) \
340(amdgpu_sriov_vf((adev)) && \
341 ((adev)->virt.reg_access & (AMDGIM_FEATURE_L1_TLB_CNTL_PSP_EN)))
342
333#define amdgpu_sriov_rlcg_error_report_enabled(adev) \343#define amdgpu_sriov_rlcg_error_report_enabled(adev) \
334 (amdgpu_sriov_reg_indirect_mmhub(adev) || amdgpu_sriov_reg_indirect_gc(adev))344 (amdgpu_sriov_reg_indirect_mmhub(adev) || amdgpu_sriov_reg_indirect_gc(adev))
335 345
@@ -423,4 +433,5 @@ int amdgpu_virt_req_ras_cper_dump(struct amdgpu_device *adev, bool force_update) · 87 unmodified lines
423int amdgpu_virt_ras_telemetry_post_reset(struct amdgpu_device *adev);433int amdgpu_virt_ras_telemetry_post_reset(struct amdgpu_device *adev);
424bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev,434bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev,
425 enum amdgpu_ras_block block);435 enum amdgpu_ras_block block);
436void amdgpu_virt_request_bad_pages(struct amdgpu_device *adev);
426#endif437#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c+3−2
@@ -787,7 +787,8 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · 786 unmodified lines
787 pasid_mapping_needed &= adev->gmc.gmc_funcs->emit_pasid_mapping &&787 pasid_mapping_needed &= adev->gmc.gmc_funcs->emit_pasid_mapping &&
788 ring->funcs->emit_wreg;788 ring->funcs->emit_wreg;
789 789
790 cleaner_shader_needed = adev->gfx.enable_cleaner_shader &&790 cleaner_shader_needed = job->run_cleaner_shader &&
791 adev->gfx.enable_cleaner_shader &&
791 ring->funcs->emit_cleaner_shader && job->base.s_fence &&792 ring->funcs->emit_cleaner_shader && job->base.s_fence &&
792 &job->base.s_fence->scheduled == isolation->spearhead;793 &job->base.s_fence->scheduled == isolation->spearhead;
793 794
@@ -817,7 +818,7 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · 23 unmodified lines
817 if (spm_update_needed && adev->gfx.rlc.funcs->update_spm_vmid)818 if (spm_update_needed && adev->gfx.rlc.funcs->update_spm_vmid)
818 adev->gfx.rlc.funcs->update_spm_vmid(adev, ring, job->vmid);819 adev->gfx.rlc.funcs->update_spm_vmid(adev, ring, job->vmid);
819 820
820 if (!ring->is_mes_queue && ring->funcs->emit_gds_switch &&821 if (ring->funcs->emit_gds_switch &&
821 gds_switch_needed) {822 gds_switch_needed) {
822 amdgpu_ring_emit_gds_switch(ring, job->vmid, job->gds_base,823 amdgpu_ring_emit_gds_switch(ring, job->vmid, job->gds_base,
823 job->gds_size, job->gws_base,824 job->gds_size, job->gws_base,
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c+18−6
@@ -296,15 +296,27 @@ static const struct amdgpu_pcs_ras_field xgmi3x16_pcs_ras_fields[] = { · 295 unmodified lines
296 296
297static u32 xgmi_v6_4_get_link_status(struct amdgpu_device *adev, int global_link_num)297static u32 xgmi_v6_4_get_link_status(struct amdgpu_device *adev, int global_link_num)
298{298{
299 const u32 smnpcs_xgmi3x16_pcs_state_hist1 = 0x11a00070;299 const u32 smn_xgmi_6_4_pcs_state_hist1[2] = { 0x11a00070, 0x11b00070 };
300 const int xgmi_inst = 2;300 const u32 smn_xgmi_6_4_1_pcs_state_hist1[2] = { 0x12100070,
301 u32 link_inst;301 0x11b00070 };
302 u32 i, n;
302 u64 addr;303 u64 addr;
303 304
304 link_inst = global_link_num % xgmi_inst;305 switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
306 case IP_VERSION(6, 4, 0):
307 n = ARRAY_SIZE(smn_xgmi_6_4_pcs_state_hist1);
308 addr = smn_xgmi_6_4_pcs_state_hist1[global_link_num % n];
309 break;
310 case IP_VERSION(6, 4, 1):
311 n = ARRAY_SIZE(smn_xgmi_6_4_1_pcs_state_hist1);
312 addr = smn_xgmi_6_4_1_pcs_state_hist1[global_link_num % n];
313 break;
314 default:
315 return U32_MAX;
316 }
305 317
306 addr = (smnpcs_xgmi3x16_pcs_state_hist1 | (link_inst << 20)) +318 i = global_link_num / n;
307 adev->asic_funcs->encode_ext_smn_addressing(global_link_num / xgmi_inst);319 addr += adev->asic_funcs->encode_ext_smn_addressing(i);
308 320
309 return RREG32_PCIE_EXT(addr);321 return RREG32_PCIE_EXT(addr);
310}322}
drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h+9−4
@@ -109,10 +109,11 @@ union amd_sriov_msg_feature_flags { · 108 unmodified lines
109 109
110union amd_sriov_reg_access_flags {110union amd_sriov_reg_access_flags {
111 struct {111 struct {
112 uint32_t vf_reg_access_ih : 1;112 uint32_t vf_reg_access_ih : 1;
113 uint32_t vf_reg_access_mmhub : 1;113 uint32_t vf_reg_access_mmhub : 1;
114 uint32_t vf_reg_access_gc : 1;114 uint32_t vf_reg_access_gc : 1;
115 uint32_t reserved : 29;115 uint32_t vf_reg_access_l1_tlb_cntl : 1;
116 uint32_t reserved : 28;
116 } flags;117 } flags;
117 uint32_t all;118 uint32_t all;
118};119};
@@ -330,6 +331,7 @@ enum amd_sriov_mailbox_request_message { · 211 unmodified lines
330 MB_REQ_MSG_RAS_POISON = 202,331 MB_REQ_MSG_RAS_POISON = 202,
331 MB_REQ_RAS_ERROR_COUNT = 203,332 MB_REQ_RAS_ERROR_COUNT = 203,
332 MB_REQ_RAS_CPER_DUMP = 204,333 MB_REQ_RAS_CPER_DUMP = 204,
334 MB_REQ_RAS_BAD_PAGES = 205,
333};335};
334 336
335/* mailbox message send from host to guest */337/* mailbox message send from host to guest */
@@ -347,6 +349,9 @@ enum amd_sriov_mailbox_response_message { · 11 unmodified lines
347 MB_RES_MSG_GPU_RMA = 10,349 MB_RES_MSG_GPU_RMA = 10,
348 MB_RES_MSG_RAS_ERROR_COUNT_READY = 11,350 MB_RES_MSG_RAS_ERROR_COUNT_READY = 11,
349 MB_REQ_RAS_CPER_DUMP_READY = 14,351 MB_REQ_RAS_CPER_DUMP_READY = 14,
352 MB_RES_MSG_RAS_BAD_PAGES_READY = 15,
353 MB_RES_MSG_RAS_BAD_PAGES_NOTIFICATION = 16,
354 MB_RES_MSG_UNRECOV_ERR_NOTIFICATION = 17,
350 MB_RES_MSG_TEXT_MESSAGE = 255355 MB_RES_MSG_TEXT_MESSAGE = 255
351};356};
352 357
drivers/gpu/drm/amd/amdgpu/atom.c+1−0
@@ -1444,6 +1444,7 @@ static void atom_get_vbios_pn(struct atom_context *ctx) · 1443 unmodified lines
1444 if (vbios_str == NULL)1444 if (vbios_str == NULL)
1445 vbios_str += sizeof(BIOS_ATOM_PREFIX) - 1;1445 vbios_str += sizeof(BIOS_ATOM_PREFIX) - 1;
1446 }1446 }
1447 OPTIMIZER_HIDE_VAR(vbios_str);
1447 if (vbios_str != NULL && *vbios_str == 0)1448 if (vbios_str != NULL && *vbios_str == 0)
1448 vbios_str++;1449 vbios_str++;
1449 1450
drivers/gpu/drm/amd/amdgpu/cik_sdma.c+4−14
@@ -56,6 +56,8 @@ static void cik_sdma_set_buffer_funcs(struct amdgpu_device *adev); · 55 unmodified lines
56static void cik_sdma_set_vm_pte_funcs(struct amdgpu_device *adev);56static void cik_sdma_set_vm_pte_funcs(struct amdgpu_device *adev);
57static int cik_sdma_soft_reset(struct amdgpu_ip_block *ip_block);57static int cik_sdma_soft_reset(struct amdgpu_ip_block *ip_block);
58 58
59u32 amdgpu_cik_gpu_check_soft_reset(struct amdgpu_device *adev);
60
59MODULE_FIRMWARE("amdgpu/bonaire_sdma.bin");61MODULE_FIRMWARE("amdgpu/bonaire_sdma.bin");
60MODULE_FIRMWARE("amdgpu/bonaire_sdma1.bin");62MODULE_FIRMWARE("amdgpu/bonaire_sdma1.bin");
61MODULE_FIRMWARE("amdgpu/hawaii_sdma.bin");63MODULE_FIRMWARE("amdgpu/hawaii_sdma.bin");
@@ -67,9 +69,6 @@ MODULE_FIRMWARE("amdgpu/kabini_sdma1.bin"); · 5 unmodified lines
67MODULE_FIRMWARE("amdgpu/mullins_sdma.bin");69MODULE_FIRMWARE("amdgpu/mullins_sdma.bin");
68MODULE_FIRMWARE("amdgpu/mullins_sdma1.bin");70MODULE_FIRMWARE("amdgpu/mullins_sdma1.bin");
69 71
70u32 amdgpu_cik_gpu_check_soft_reset(struct amdgpu_device *adev);
71
72
73static void cik_sdma_free_microcode(struct amdgpu_device *adev)72static void cik_sdma_free_microcode(struct amdgpu_device *adev)
74{73{
75 int i;74 int i;
@@ -993,14 +992,9 @@ static int cik_sdma_sw_fini(struct amdgpu_ip_block *ip_block) · 917 unmodified lines
993 992
994static int cik_sdma_hw_init(struct amdgpu_ip_block *ip_block)993static int cik_sdma_hw_init(struct amdgpu_ip_block *ip_block)
995{994{
996 int r;
997 struct amdgpu_device *adev = ip_block->adev;995 struct amdgpu_device *adev = ip_block->adev;
998 996
999 r = cik_sdma_start(adev);997 return cik_sdma_start(adev);
1000 if (r)
1001 return r;
1002
1003 return r;
1004}998}
1005 999
1006static int cik_sdma_hw_fini(struct amdgpu_ip_block *ip_block)1000static int cik_sdma_hw_fini(struct amdgpu_ip_block *ip_block)
@@ -1040,14 +1034,10 @@ static bool cik_sdma_is_idle(struct amdgpu_ip_block *ip_block) · 33 unmodified lines
1040static int cik_sdma_wait_for_idle(struct amdgpu_ip_block *ip_block)1034static int cik_sdma_wait_for_idle(struct amdgpu_ip_block *ip_block)
1041{1035{
1042 unsigned i;1036 unsigned i;
1043 u32 tmp;
1044 struct amdgpu_device *adev = ip_block->adev;1037 struct amdgpu_device *adev = ip_block->adev;
1045 1038
1046 for (i = 0; i < adev->usec_timeout; i++) {1039 for (i = 0; i < adev->usec_timeout; i++) {
1047 tmp = RREG32(mmSRBM_STATUS2) & (SRBM_STATUS2__SDMA_BUSY_MASK |1040 if (cik_sdma_is_idle(ip_block))
1048 SRBM_STATUS2__SDMA1_BUSY_MASK);
1049
1050 if (!tmp)
1051 return 0;1041 return 0;
1052 udelay(1);1042 udelay(1);
1053 }1043 }
drivers/gpu/drm/amd/amdgpu/cikd.h+0−3
@@ -60,9 +60,6 @@ · 59 unmodified lines
60#define AUD5_REGISTER_OFFSET (0x179d - 0x1780)60#define AUD5_REGISTER_OFFSET (0x179d - 0x1780)
61#define AUD6_REGISTER_OFFSET (0x17a4 - 0x1780)61#define AUD6_REGISTER_OFFSET (0x17a4 - 0x1780)
62 62
63#define BONAIRE_GB_ADDR_CONFIG_GOLDEN 0x12010001
64#define HAWAII_GB_ADDR_CONFIG_GOLDEN 0x12011003
65
66#define PIPEID(x) ((x) << 0)63#define PIPEID(x) ((x) << 0)
67#define MEID(x) ((x) << 2)64#define MEID(x) ((x) << 2)
68#define VMID(x) ((x) << 4)65#define VMID(x) ((x) << 4)
drivers/gpu/drm/amd/amdgpu/dce_v10_0.c+2−2
@@ -3075,7 +3075,7 @@ static int dce_v10_0_set_hpd_irq_state(struct amdgpu_device *adev, · 3074 unmodified lines
3075 u32 tmp;3075 u32 tmp;
3076 3076
3077 if (hpd >= adev->mode_info.num_hpd) {3077 if (hpd >= adev->mode_info.num_hpd) {
3078 DRM_DEBUG("invalid hdp %d\n", hpd);3078 DRM_DEBUG("invalid hpd %d\n", hpd);
3079 return 0;3079 return 0;
3080 }3080 }
3081 3081
@@ -3227,7 +3227,7 @@ static void dce_v10_0_hpd_int_ack(struct amdgpu_device *adev, · 145 unmodified lines
3227 u32 tmp;3227 u32 tmp;
3228 3228
3229 if (hpd >= adev->mode_info.num_hpd) {3229 if (hpd >= adev->mode_info.num_hpd) {
3230 DRM_DEBUG("invalid hdp %d\n", hpd);3230 DRM_DEBUG("invalid hpd %d\n", hpd);
3231 return;3231 return;
3232 }3232 }
3233 3233
drivers/gpu/drm/amd/amdgpu/dce_v11_0.c+3−4
@@ -3206,7 +3206,7 @@ static int dce_v11_0_set_hpd_irq_state(struct amdgpu_device *adev, · 3205 unmodified lines
3206 u32 tmp;3206 u32 tmp;
3207 3207
3208 if (hpd >= adev->mode_info.num_hpd) {3208 if (hpd >= adev->mode_info.num_hpd) {
3209 DRM_DEBUG("invalid hdp %d\n", hpd);3209 DRM_DEBUG("invalid hpd %d\n", hpd);
3210 return 0;3210 return 0;
3211 }3211 }
3212 3212
@@ -3358,7 +3358,7 @@ static void dce_v11_0_hpd_int_ack(struct amdgpu_device *adev, · 145 unmodified lines
3358 u32 tmp;3358 u32 tmp;
3359 3359
3360 if (hpd >= adev->mode_info.num_hpd) {3360 if (hpd >= adev->mode_info.num_hpd) {
3361 DRM_DEBUG("invalid hdp %d\n", hpd);3361 DRM_DEBUG("invalid hpd %d\n", hpd);
3362 return;3362 return;
3363 }3363 }
3364 3364
@@ -3488,8 +3488,7 @@ static const struct amd_ip_funcs dce_v11_0_ip_funcs = { · 123 unmodified lines
3488 .set_powergating_state = dce_v11_0_set_powergating_state,3488 .set_powergating_state = dce_v11_0_set_powergating_state,
3489};3489};
3490 3490
3491static void3491static void dce_v11_0_encoder_mode_set(struct drm_encoder *encoder,
3492dce_v11_0_encoder_mode_set(struct drm_encoder *encoder,
3493 struct drm_display_mode *mode,3492 struct drm_display_mode *mode,
3494 struct drm_display_mode *adjusted_mode)3493 struct drm_display_mode *adjusted_mode)
3495{3494{
drivers/gpu/drm/amd/amdgpu/dce_v6_0.c+100−106
@@ -287,7 +287,7 @@ static void dce_v6_0_hpd_int_ack(struct amdgpu_device *adev, · 286 unmodified lines
287 u32 tmp;287 u32 tmp;
288 288
289 if (hpd >= adev->mode_info.num_hpd) {289 if (hpd >= adev->mode_info.num_hpd) {
290 DRM_DEBUG("invalid hdp %d\n", hpd);290 DRM_DEBUG("invalid hpd %d\n", hpd);
291 return;291 return;
292 }292 }
293 293
@@ -412,7 +412,7 @@ static void dce_v6_0_set_vga_render_state(struct amdgpu_device *adev, · 118 unmodified lines
412{412{
413 if (!render)413 if (!render)
414 WREG32(mmVGA_RENDER_CONTROL,414 WREG32(mmVGA_RENDER_CONTROL,
415 RREG32(mmVGA_RENDER_CONTROL) & VGA_VSTATUS_CNTL);415 RREG32(mmVGA_RENDER_CONTROL) & ~VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK);
416}416}
417 417
418static int dce_v6_0_get_num_crtc(struct amdgpu_device *adev)418static int dce_v6_0_get_num_crtc(struct amdgpu_device *adev)
@@ -1011,16 +1011,16 @@ static void dce_v6_0_program_watermarks(struct amdgpu_device *adev, · 592 unmodified lines
1011 /* select wm A */1011 /* select wm A */
1012 arb_control3 = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset);1012 arb_control3 = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset);
1013 tmp = arb_control3;1013 tmp = arb_control3;
1014 tmp &= ~LATENCY_WATERMARK_MASK(3);1014 tmp &= ~(3 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
1015 tmp |= LATENCY_WATERMARK_MASK(1);1015 tmp |= (1 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
1016 WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp);1016 WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp);
1017 WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset,1017 WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset,
1018 ((latency_watermark_a << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) |1018 ((latency_watermark_a << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) |
1019 (line_time << DPG_PIPE_URGENCY_CONTROL__URGENCY_HIGH_WATERMARK__SHIFT)));1019 (line_time << DPG_PIPE_URGENCY_CONTROL__URGENCY_HIGH_WATERMARK__SHIFT)));
1020 /* select wm B */1020 /* select wm B */
1021 tmp = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset);1021 tmp = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset);
1022 tmp &= ~LATENCY_WATERMARK_MASK(3);1022 tmp &= ~(3 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
1023 tmp |= LATENCY_WATERMARK_MASK(2);1023 tmp |= (2 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT);
1024 WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp);1024 WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp);
1025 WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset,1025 WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset,
1026 ((latency_watermark_b << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) |1026 ((latency_watermark_b << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) |
@@ -1089,7 +1089,7 @@ static u32 dce_v6_0_line_buffer_adjust(struct amdgpu_device *adev, · 62 unmodified lines
1089 }1089 }
1090 1090
1091 WREG32(mmDC_LB_MEMORY_SPLIT + amdgpu_crtc->crtc_offset,1091 WREG32(mmDC_LB_MEMORY_SPLIT + amdgpu_crtc->crtc_offset,
1092 DC_LB_MEMORY_CONFIG(tmp));1092 (tmp << DC_LB_MEMORY_SPLIT__DC_LB_MEMORY_CONFIG__SHIFT));
1093 1093
1094 WREG32(mmPIPE0_DMIF_BUFFER_CONTROL + pipe_offset,1094 WREG32(mmPIPE0_DMIF_BUFFER_CONTROL + pipe_offset,
1095 (buffer_alloc << PIPE0_DMIF_BUFFER_CONTROL__DMIF_BUFFERS_ALLOCATED__SHIFT));1095 (buffer_alloc << PIPE0_DMIF_BUFFER_CONTROL__DMIF_BUFFERS_ALLOCATED__SHIFT));
@@ -1306,6 +1306,7 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 210 unmodified lines
1306 struct amdgpu_device *adev = drm_to_adev(dev);1306 struct amdgpu_device *adev = drm_to_adev(dev);
1307 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);1307 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
1308 struct amdgpu_encoder_atom_dig *dig = amdgpu_encoder->enc_priv;1308 struct amdgpu_encoder_atom_dig *dig = amdgpu_encoder->enc_priv;
1309 u32 offset;
1309 struct drm_connector *connector;1310 struct drm_connector *connector;
1310 struct drm_connector_list_iter iter;1311 struct drm_connector_list_iter iter;
1311 struct amdgpu_connector *amdgpu_connector = NULL;1312 struct amdgpu_connector *amdgpu_connector = NULL;
@@ -1327,6 +1328,11 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 15 unmodified lines
1327 { ixAZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13, HDMI_AUDIO_CODING_TYPE_WMA_PRO },1328 { ixAZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13, HDMI_AUDIO_CODING_TYPE_WMA_PRO },
1328 };1329 };
1329 1330
1331 if (!dig || !dig->afmt || !dig->afmt->pin)
1332 return;
1333
1334 offset = dig->afmt->pin->offset;
1335
1330 drm_connector_list_iter_begin(dev, &iter);1336 drm_connector_list_iter_begin(dev, &iter);
1331 drm_for_each_connector_iter(connector, &iter) {1337 drm_for_each_connector_iter(connector, &iter) {
1332 if (connector->encoder == encoder) {1338 if (connector->encoder == encoder) {
@@ -1348,7 +1354,7 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 15 unmodified lines
1348 return;1354 return;
1349 1355
1350 for (i = 0; i < ARRAY_SIZE(eld_reg_to_type); i++) {1356 for (i = 0; i < ARRAY_SIZE(eld_reg_to_type); i++) {
1351 u32 tmp = 0;1357 u32 value = 0;
1352 u8 stereo_freqs = 0;1358 u8 stereo_freqs = 0;
1353 int max_channels = -1;1359 int max_channels = -1;
1354 int j;1360 int j;
@@ -1358,12 +1364,12 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 3 unmodified lines
1358 1364
1359 if (sad->format == eld_reg_to_type[i][1]) {1365 if (sad->format == eld_reg_to_type[i][1]) {
1360 if (sad->channels > max_channels) {1366 if (sad->channels > max_channels) {
1361 tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,1367 value = (sad->channels <<
1362 MAX_CHANNELS, sad->channels);1368 AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__MAX_CHANNELS__SHIFT) |
1363 tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,1369 (sad->byte2 <<
1364 DESCRIPTOR_BYTE_2, sad->byte2);1370 AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__DESCRIPTOR_BYTE_2__SHIFT) |
1365 tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,1371 (sad->freq <<
1366 SUPPORTED_FREQUENCIES, sad->freq);1372 AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__SUPPORTED_FREQUENCIES__SHIFT);
1367 max_channels = sad->channels;1373 max_channels = sad->channels;
1368 }1374 }
1369 1375
@@ -1374,13 +1380,13 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 4 unmodified lines
1374 }1380 }
1375 }1381 }
1376 1382
1377 tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0,1383 value |= (stereo_freqs <<
1378 SUPPORTED_FREQUENCIES_STEREO, stereo_freqs);1384 AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__SUPPORTED_FREQUENCIES_STEREO__SHIFT);
1379 WREG32_AUDIO_ENDPT(dig->afmt->pin->offset, eld_reg_to_type[i][0], tmp);1385
1386 WREG32_AUDIO_ENDPT(offset, eld_reg_to_type[i][0], value);
1380 }1387 }
1381 1388
1382 kfree(sads);1389 kfree(sads);
1383
1384}1390}
1385 1391
1386static void dce_v6_0_audio_enable(struct amdgpu_device *adev,1392static void dce_v6_0_audio_enable(struct amdgpu_device *adev,
@@ -1886,7 +1892,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 499 unmodified lines
1886 struct amdgpu_bo *abo;1892 struct amdgpu_bo *abo;
1887 uint64_t fb_location, tiling_flags;1893 uint64_t fb_location, tiling_flags;
1888 uint32_t fb_format, fb_pitch_pixels, pipe_config;1894 uint32_t fb_format, fb_pitch_pixels, pipe_config;
1889 u32 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_NONE);1895 u32 fb_swap = (GRPH_ENDIAN_NONE << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1890 u32 viewport_w, viewport_h;1896 u32 viewport_w, viewport_h;
1891 int r;1897 int r;
1892 bool bypass_lut = false;1898 bool bypass_lut = false;
@@ -1926,76 +1932,76 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 33 unmodified lines
1926 1932
1927 switch (target_fb->format->format) {1933 switch (target_fb->format->format) {
1928 case DRM_FORMAT_C8:1934 case DRM_FORMAT_C8:
1929 fb_format = (GRPH_DEPTH(GRPH_DEPTH_8BPP) |1935 fb_format = ((GRPH_DEPTH_8BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1930 GRPH_FORMAT(GRPH_FORMAT_INDEXED));1936 (GRPH_FORMAT_INDEXED << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1931 break;1937 break;
1932 case DRM_FORMAT_XRGB4444:1938 case DRM_FORMAT_XRGB4444:
1933 case DRM_FORMAT_ARGB4444:1939 case DRM_FORMAT_ARGB4444:
1934 fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |1940 fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1935 GRPH_FORMAT(GRPH_FORMAT_ARGB4444));1941 (GRPH_FORMAT_ARGB4444 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1936#ifdef __BIG_ENDIAN1942#ifdef __BIG_ENDIAN
1937 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);1943 fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1938#endif1944#endif
1939 break;1945 break;
1940 case DRM_FORMAT_XRGB1555:1946 case DRM_FORMAT_XRGB1555:
1941 case DRM_FORMAT_ARGB1555:1947 case DRM_FORMAT_ARGB1555:
1942 fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |1948 fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1943 GRPH_FORMAT(GRPH_FORMAT_ARGB1555));1949 (GRPH_FORMAT_ARGB1555 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1944#ifdef __BIG_ENDIAN1950#ifdef __BIG_ENDIAN
1945 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);1951 fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1946#endif1952#endif
1947 break;1953 break;
1948 case DRM_FORMAT_BGRX5551:1954 case DRM_FORMAT_BGRX5551:
1949 case DRM_FORMAT_BGRA5551:1955 case DRM_FORMAT_BGRA5551:
1950 fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |1956 fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1951 GRPH_FORMAT(GRPH_FORMAT_BGRA5551));1957 (GRPH_FORMAT_BGRA5551 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1952#ifdef __BIG_ENDIAN1958#ifdef __BIG_ENDIAN
1953 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);1959 fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1954#endif1960#endif
1955 break;1961 break;
1956 case DRM_FORMAT_RGB565:1962 case DRM_FORMAT_RGB565:
1957 fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) |1963 fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1958 GRPH_FORMAT(GRPH_FORMAT_ARGB565));1964 (GRPH_FORMAT_ARGB565 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1959#ifdef __BIG_ENDIAN1965#ifdef __BIG_ENDIAN
1960 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16);1966 fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1961#endif1967#endif
1962 break;1968 break;
1963 case DRM_FORMAT_XRGB8888:1969 case DRM_FORMAT_XRGB8888:
1964 case DRM_FORMAT_ARGB8888:1970 case DRM_FORMAT_ARGB8888:
1965 fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |1971 fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1966 GRPH_FORMAT(GRPH_FORMAT_ARGB8888));1972 (GRPH_FORMAT_ARGB8888 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1967#ifdef __BIG_ENDIAN1973#ifdef __BIG_ENDIAN
1968 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);1974 fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1969#endif1975#endif
1970 break;1976 break;
1971 case DRM_FORMAT_XRGB2101010:1977 case DRM_FORMAT_XRGB2101010:
1972 case DRM_FORMAT_ARGB2101010:1978 case DRM_FORMAT_ARGB2101010:
1973 fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |1979 fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1974 GRPH_FORMAT(GRPH_FORMAT_ARGB2101010));1980 (GRPH_FORMAT_ARGB2101010 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1975#ifdef __BIG_ENDIAN1981#ifdef __BIG_ENDIAN
1976 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);1982 fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1977#endif1983#endif
1978 /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */1984 /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */
1979 bypass_lut = true;1985 bypass_lut = true;
1980 break;1986 break;
1981 case DRM_FORMAT_BGRX1010102:1987 case DRM_FORMAT_BGRX1010102:
1982 case DRM_FORMAT_BGRA1010102:1988 case DRM_FORMAT_BGRA1010102:
1983 fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |1989 fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1984 GRPH_FORMAT(GRPH_FORMAT_BGRA1010102));1990 (GRPH_FORMAT_BGRA1010102 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1985#ifdef __BIG_ENDIAN1991#ifdef __BIG_ENDIAN
1986 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);1992 fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1987#endif1993#endif
1988 /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */1994 /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */
1989 bypass_lut = true;1995 bypass_lut = true;
1990 break;1996 break;
1991 case DRM_FORMAT_XBGR8888:1997 case DRM_FORMAT_XBGR8888:
1992 case DRM_FORMAT_ABGR8888:1998 case DRM_FORMAT_ABGR8888:
1993 fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) |1999 fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) |
1994 GRPH_FORMAT(GRPH_FORMAT_ARGB8888));2000 (GRPH_FORMAT_ARGB8888 << GRPH_CONTROL__GRPH_FORMAT__SHIFT));
1995 fb_swap = (GRPH_RED_CROSSBAR(GRPH_RED_SEL_B) |2001 fb_swap = ((GRPH_RED_SEL_B << GRPH_SWAP_CNTL__GRPH_RED_CROSSBAR__SHIFT) |
1996 GRPH_BLUE_CROSSBAR(GRPH_BLUE_SEL_R));2002 (GRPH_BLUE_SEL_R << GRPH_SWAP_CNTL__GRPH_BLUE_CROSSBAR__SHIFT));
1997#ifdef __BIG_ENDIAN2003#ifdef __BIG_ENDIAN
1998 fb_swap |= GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32);2004 fb_swap |= (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT);
1999#endif2005#endif
2000 break;2006 break;
2001 default:2007 default:
@@ -2013,18 +2019,18 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 11 unmodified lines
2013 tile_split = AMDGPU_TILING_GET(tiling_flags, TILE_SPLIT);2019 tile_split = AMDGPU_TILING_GET(tiling_flags, TILE_SPLIT);
2014 num_banks = AMDGPU_TILING_GET(tiling_flags, NUM_BANKS);2020 num_banks = AMDGPU_TILING_GET(tiling_flags, NUM_BANKS);
2015 2021
2016 fb_format |= GRPH_NUM_BANKS(num_banks);2022 fb_format |= (num_banks << GRPH_CONTROL__GRPH_NUM_BANKS__SHIFT);
2017 fb_format |= GRPH_ARRAY_MODE(GRPH_ARRAY_2D_TILED_THIN1);2023 fb_format |= (GRPH_ARRAY_2D_TILED_THIN1 << GRPH_CONTROL__GRPH_ARRAY_MODE__SHIFT);
2018 fb_format |= GRPH_TILE_SPLIT(tile_split);2024 fb_format |= (tile_split << GRPH_CONTROL__GRPH_TILE_SPLIT__SHIFT);
2019 fb_format |= GRPH_BANK_WIDTH(bankw);2025 fb_format |= (bankw << GRPH_CONTROL__GRPH_BANK_WIDTH__SHIFT);
2020 fb_format |= GRPH_BANK_HEIGHT(bankh);2026 fb_format |= (bankh << GRPH_CONTROL__GRPH_BANK_HEIGHT__SHIFT);
2021 fb_format |= GRPH_MACRO_TILE_ASPECT(mtaspect);2027 fb_format |= (mtaspect << GRPH_CONTROL__GRPH_MACRO_TILE_ASPECT__SHIFT);
2022 } else if (AMDGPU_TILING_GET(tiling_flags, ARRAY_MODE) == ARRAY_1D_TILED_THIN1) {2028 } else if (AMDGPU_TILING_GET(tiling_flags, ARRAY_MODE) == ARRAY_1D_TILED_THIN1) {
2023 fb_format |= GRPH_ARRAY_MODE(GRPH_ARRAY_1D_TILED_THIN1);2029 fb_format |= (GRPH_ARRAY_1D_TILED_THIN1 << GRPH_CONTROL__GRPH_ARRAY_MODE__SHIFT);
2024 }2030 }
2025 2031
2026 pipe_config = AMDGPU_TILING_GET(tiling_flags, PIPE_CONFIG);2032 pipe_config = AMDGPU_TILING_GET(tiling_flags, PIPE_CONFIG);
2027 fb_format |= GRPH_PIPE_CONFIG(pipe_config);2033 fb_format |= (pipe_config << GRPH_CONTROL__GRPH_PIPE_CONFIG__SHIFT);
2028 2034
2029 dce_v6_0_vga_enable(crtc, false);2035 dce_v6_0_vga_enable(crtc, false);
2030 2036
@@ -2040,7 +2046,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 9 unmodified lines
2040 WREG32(mmGRPH_PRIMARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset,2046 WREG32(mmGRPH_PRIMARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset,
2041 (u32)fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK);2047 (u32)fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK);
2042 WREG32(mmGRPH_SECONDARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset,2048 WREG32(mmGRPH_SECONDARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset,
2043 (u32) fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK);2049 (u32) fb_location & GRPH_SECONDARY_SURFACE_ADDRESS__GRPH_SECONDARY_SURFACE_ADDRESS_MASK);
2044 WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format);2050 WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format);
2045 WREG32(mmGRPH_SWAP_CNTL + amdgpu_crtc->crtc_offset, fb_swap);2051 WREG32(mmGRPH_SWAP_CNTL + amdgpu_crtc->crtc_offset, fb_swap);
2046 2052
@@ -2108,14 +2114,13 @@ static void dce_v6_0_set_interleave(struct drm_crtc *crtc, · 61 unmodified lines
2108 2114
2109 if (mode->flags & DRM_MODE_FLAG_INTERLACE)2115 if (mode->flags & DRM_MODE_FLAG_INTERLACE)
2110 WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset,2116 WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset,
2111 INTERLEAVE_EN);2117 DATA_FORMAT__INTERLEAVE_EN_MASK);
2112 else2118 else
2113 WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, 0);2119 WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, 0);
2114}2120}
2115 2121
2116static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc)2122static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc)
2117{2123{
2118
2119 struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc);2124 struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc);
2120 struct drm_device *dev = crtc->dev;2125 struct drm_device *dev = crtc->dev;
2121 struct amdgpu_device *adev = drm_to_adev(dev);2126 struct amdgpu_device *adev = drm_to_adev(dev);
@@ -2125,15 +2130,15 @@ static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) · 3 unmodified lines
2125 DRM_DEBUG_KMS("%d\n", amdgpu_crtc->crtc_id);2130 DRM_DEBUG_KMS("%d\n", amdgpu_crtc->crtc_id);
2126 2131
2127 WREG32(mmINPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset,2132 WREG32(mmINPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset,
2128 ((0 << INPUT_CSC_CONTROL__INPUT_CSC_GRPH_MODE__SHIFT) |2133 ((INPUT_CSC_BYPASS << INPUT_CSC_CONTROL__INPUT_CSC_GRPH_MODE__SHIFT) |
2129 (0 << INPUT_CSC_CONTROL__INPUT_CSC_OVL_MODE__SHIFT)));2134 (INPUT_CSC_BYPASS << INPUT_CSC_CONTROL__INPUT_CSC_OVL_MODE__SHIFT)));
2130 WREG32(mmPRESCALE_GRPH_CONTROL + amdgpu_crtc->crtc_offset,2135 WREG32(mmPRESCALE_GRPH_CONTROL + amdgpu_crtc->crtc_offset,
2131 PRESCALE_GRPH_CONTROL__GRPH_PRESCALE_BYPASS_MASK);2136 PRESCALE_GRPH_CONTROL__GRPH_PRESCALE_BYPASS_MASK);
2132 WREG32(mmPRESCALE_OVL_CONTROL + amdgpu_crtc->crtc_offset,2137 WREG32(mmPRESCALE_OVL_CONTROL + amdgpu_crtc->crtc_offset,
2133 PRESCALE_OVL_CONTROL__OVL_PRESCALE_BYPASS_MASK);2138 PRESCALE_OVL_CONTROL__OVL_PRESCALE_BYPASS_MASK);
2134 WREG32(mmINPUT_GAMMA_CONTROL + amdgpu_crtc->crtc_offset,2139 WREG32(mmINPUT_GAMMA_CONTROL + amdgpu_crtc->crtc_offset,
2135 ((0 << INPUT_GAMMA_CONTROL__GRPH_INPUT_GAMMA_MODE__SHIFT) |2140 ((INPUT_GAMMA_USE_LUT << INPUT_GAMMA_CONTROL__GRPH_INPUT_GAMMA_MODE__SHIFT) |
2136 (0 << INPUT_GAMMA_CONTROL__OVL_INPUT_GAMMA_MODE__SHIFT)));2141 (INPUT_GAMMA_USE_LUT << INPUT_GAMMA_CONTROL__OVL_INPUT_GAMMA_MODE__SHIFT)));
2137 2142
2138 WREG32(mmDC_LUT_CONTROL + amdgpu_crtc->crtc_offset, 0);2143 WREG32(mmDC_LUT_CONTROL + amdgpu_crtc->crtc_offset, 0);
2139 2144
@@ -2160,19 +2165,19 @@ static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) · 20 unmodified lines
2160 }2165 }
2161 2166
2162 WREG32(mmDEGAMMA_CONTROL + amdgpu_crtc->crtc_offset,2167 WREG32(mmDEGAMMA_CONTROL + amdgpu_crtc->crtc_offset,
2163 ((0 << DEGAMMA_CONTROL__GRPH_DEGAMMA_MODE__SHIFT) |2168 ((DEGAMMA_BYPASS << DEGAMMA_CONTROL__GRPH_DEGAMMA_MODE__SHIFT) |
2164 (0 << DEGAMMA_CONTROL__OVL_DEGAMMA_MODE__SHIFT) |2169 (DEGAMMA_BYPASS << DEGAMMA_CONTROL__OVL_DEGAMMA_MODE__SHIFT) |
2165 ICON_DEGAMMA_MODE(0) |2170 (DEGAMMA_BYPASS << DEGAMMA_CONTROL__ICON_DEGAMMA_MODE__SHIFT) |
2166 (0 << DEGAMMA_CONTROL__CURSOR_DEGAMMA_MODE__SHIFT)));2171 (DEGAMMA_BYPASS << DEGAMMA_CONTROL__CURSOR_DEGAMMA_MODE__SHIFT)));
2167 WREG32(mmGAMUT_REMAP_CONTROL + amdgpu_crtc->crtc_offset,2172 WREG32(mmGAMUT_REMAP_CONTROL + amdgpu_crtc->crtc_offset,
2168 ((0 << GAMUT_REMAP_CONTROL__GRPH_GAMUT_REMAP_MODE__SHIFT) |2173 ((GAMUT_REMAP_BYPASS << GAMUT_REMAP_CONTROL__GRPH_GAMUT_REMAP_MODE__SHIFT) |
2169 (0 << GAMUT_REMAP_CONTROL__OVL_GAMUT_REMAP_MODE__SHIFT)));2174 (GAMUT_REMAP_BYPASS << GAMUT_REMAP_CONTROL__OVL_GAMUT_REMAP_MODE__SHIFT)));
2170 WREG32(mmREGAMMA_CONTROL + amdgpu_crtc->crtc_offset,2175 WREG32(mmREGAMMA_CONTROL + amdgpu_crtc->crtc_offset,
2171 ((0 << REGAMMA_CONTROL__GRPH_REGAMMA_MODE__SHIFT) |2176 ((REGAMMA_BYPASS << REGAMMA_CONTROL__GRPH_REGAMMA_MODE__SHIFT) |
2172 (0 << REGAMMA_CONTROL__OVL_REGAMMA_MODE__SHIFT)));2177 (REGAMMA_BYPASS << REGAMMA_CONTROL__OVL_REGAMMA_MODE__SHIFT)));
2173 WREG32(mmOUTPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset,2178 WREG32(mmOUTPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset,
2174 ((0 << OUTPUT_CSC_CONTROL__OUTPUT_CSC_GRPH_MODE__SHIFT) |2179 ((OUTPUT_CSC_BYPASS << OUTPUT_CSC_CONTROL__OUTPUT_CSC_GRPH_MODE__SHIFT) |
2175 (0 << OUTPUT_CSC_CONTROL__OUTPUT_CSC_OVL_MODE__SHIFT)));2180 (OUTPUT_CSC_BYPASS << OUTPUT_CSC_CONTROL__OUTPUT_CSC_OVL_MODE__SHIFT)));
2176 /* XXX match this to the depth of the crtc fmt block, move to modeset? */2181 /* XXX match this to the depth of the crtc fmt block, move to modeset? */
2177 WREG32(0x1a50 + amdgpu_crtc->crtc_offset, 0);2182 WREG32(0x1a50 + amdgpu_crtc->crtc_offset, 0);
2178 2183
@@ -2267,8 +2272,6 @@ static void dce_v6_0_hide_cursor(struct drm_crtc *crtc) · 88 unmodified lines
2267 WREG32(mmCUR_CONTROL + amdgpu_crtc->crtc_offset,2272 WREG32(mmCUR_CONTROL + amdgpu_crtc->crtc_offset,
2268 (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) |2273 (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) |
2269 (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT));2274 (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT));
2270
2271
2272}2275}
2273 2276
2274static void dce_v6_0_show_cursor(struct drm_crtc *crtc)2277static void dce_v6_0_show_cursor(struct drm_crtc *crtc)
@@ -2285,7 +2288,6 @@ static void dce_v6_0_show_cursor(struct drm_crtc *crtc) · 10 unmodified lines
2285 CUR_CONTROL__CURSOR_EN_MASK |2288 CUR_CONTROL__CURSOR_EN_MASK |
2286 (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) |2289 (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) |
2287 (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT));2290 (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT));
2288
2289}2291}
2290 2292
2291static int dce_v6_0_cursor_move_locked(struct drm_crtc *crtc,2293static int dce_v6_0_cursor_move_locked(struct drm_crtc *crtc,
@@ -2596,7 +2598,6 @@ static bool dce_v6_0_crtc_mode_fixup(struct drm_crtc *crtc, · 304 unmodified lines
2596 const struct drm_display_mode *mode,2598 const struct drm_display_mode *mode,
2597 struct drm_display_mode *adjusted_mode)2599 struct drm_display_mode *adjusted_mode)
2598{2600{
2599
2600 struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc);2601 struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc);
2601 struct drm_device *dev = crtc->dev;2602 struct drm_device *dev = crtc->dev;
2602 struct drm_encoder *encoder;2603 struct drm_encoder *encoder;
@@ -2669,7 +2670,7 @@ static void dce_v6_0_panic_flush(struct drm_plane *plane) · 66 unmodified lines
2669 2670
2670 /* Disable DC tiling */2671 /* Disable DC tiling */
2671 fb_format = RREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset);2672 fb_format = RREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset);
2672 fb_format &= ~GRPH_ARRAY_MODE(0x7);2673 fb_format &= ~GRPH_CONTROL__GRPH_ARRAY_MODE_MASK;
2673 WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format);2674 WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format);
2674 2675
2675}2676}
@@ -2745,7 +2746,6 @@ static int dce_v6_0_early_init(struct amdgpu_ip_block *ip_block) · 69 unmodified lines
2745static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block)2746static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block)
2746{2747{
2747 int r, i;2748 int r, i;
2748 bool ret;
2749 struct amdgpu_device *adev = ip_block->adev;2749 struct amdgpu_device *adev = ip_block->adev;
2750 2750
2751 for (i = 0; i < adev->mode_info.num_crtc; i++) {2751 for (i = 0; i < adev->mode_info.num_crtc; i++) {
@@ -2789,8 +2789,7 @@ static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 37 unmodified lines
2789 return r;2789 return r;
2790 }2790 }
2791 2791
2792 ret = amdgpu_atombios_get_connector_info_from_object_table(adev);2792 if (amdgpu_atombios_get_connector_info_from_object_table(adev))
2793 if (ret)
2794 amdgpu_display_print_display_setup(adev_to_drm(adev));2793 amdgpu_display_print_display_setup(adev_to_drm(adev));
2795 else2794 else
2796 return -EINVAL;2795 return -EINVAL;
@@ -2986,12 +2985,12 @@ static void dce_v6_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev, · 189 unmodified lines
2986 switch (state) {2985 switch (state) {
2987 case AMDGPU_IRQ_STATE_DISABLE:2986 case AMDGPU_IRQ_STATE_DISABLE:
2988 interrupt_mask = RREG32(mmINT_MASK + reg_block);2987 interrupt_mask = RREG32(mmINT_MASK + reg_block);
2989 interrupt_mask &= ~VBLANK_INT_MASK;2988 interrupt_mask &= ~INT_MASK__VBLANK_INT_MASK;
2990 WREG32(mmINT_MASK + reg_block, interrupt_mask);2989 WREG32(mmINT_MASK + reg_block, interrupt_mask);
2991 break;2990 break;
2992 case AMDGPU_IRQ_STATE_ENABLE:2991 case AMDGPU_IRQ_STATE_ENABLE:
2993 interrupt_mask = RREG32(mmINT_MASK + reg_block);2992 interrupt_mask = RREG32(mmINT_MASK + reg_block);
2994 interrupt_mask |= VBLANK_INT_MASK;2993 interrupt_mask |= INT_MASK__VBLANK_INT_MASK;
2995 WREG32(mmINT_MASK + reg_block, interrupt_mask);2994 WREG32(mmINT_MASK + reg_block, interrupt_mask);
2996 break;2995 break;
2997 default:2996 default:
@@ -3006,28 +3005,28 @@ static void dce_v6_0_set_crtc_vline_interrupt_state(struct amdgpu_device *adev, · 8 unmodified lines
3006 3005
3007}3006}
3008 3007
3009static int dce_v6_0_set_hpd_interrupt_state(struct amdgpu_device *adev,3008static int dce_v6_0_set_hpd_irq_state(struct amdgpu_device *adev,
3010 struct amdgpu_irq_src *src,3009 struct amdgpu_irq_src *src,
3011 unsigned type,3010 unsigned hpd,
3012 enum amdgpu_interrupt_state state)3011 enum amdgpu_interrupt_state state)
3013{3012{
3014 u32 dc_hpd_int_cntl;3013 u32 dc_hpd_int_cntl;
3015 3014
3016 if (type >= adev->mode_info.num_hpd) {3015 if (hpd >= adev->mode_info.num_hpd) {
3017 DRM_DEBUG("invalid hdp %d\n", type);3016 DRM_DEBUG("invalid hpd %d\n", hpd);
3018 return 0;3017 return 0;
3019 }3018 }
3020 3019
3021 switch (state) {3020 switch (state) {
3022 case AMDGPU_IRQ_STATE_DISABLE:3021 case AMDGPU_IRQ_STATE_DISABLE:
3023 dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type]);3022 dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd]);
3024 dc_hpd_int_cntl &= ~DC_HPDx_INT_EN;3023 dc_hpd_int_cntl &= ~DC_HPD1_INT_CONTROL__DC_HPD1_INT_EN_MASK;
3025 WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type], dc_hpd_int_cntl);3024 WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], dc_hpd_int_cntl);
3026 break;3025 break;
3027 case AMDGPU_IRQ_STATE_ENABLE:3026 case AMDGPU_IRQ_STATE_ENABLE:
3028 dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type]);3027 dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd]);
3029 dc_hpd_int_cntl |= DC_HPDx_INT_EN;3028 dc_hpd_int_cntl |= DC_HPD1_INT_CONTROL__DC_HPD1_INT_EN_MASK;
3030 WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type], dc_hpd_int_cntl);3029 WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], dc_hpd_int_cntl);
3031 break;3030 break;
3032 default:3031 default:
3033 break;3032 break;
@@ -3036,7 +3035,7 @@ static int dce_v6_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 2 unmodified lines
3036 return 0;3035 return 0;
3037}3036}
3038 3037
3039static int dce_v6_0_set_crtc_interrupt_state(struct amdgpu_device *adev,3038static int dce_v6_0_set_crtc_irq_state(struct amdgpu_device *adev,
3040 struct amdgpu_irq_src *src,3039 struct amdgpu_irq_src *src,
3041 unsigned type,3040 unsigned type,
3042 enum amdgpu_interrupt_state state)3041 enum amdgpu_interrupt_state state)
@@ -3096,7 +3095,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 53 unmodified lines
3096 switch (entry->src_data[0]) {3095 switch (entry->src_data[0]) {
3097 case 0: /* vblank */3096 case 0: /* vblank */
3098 if (disp_int & interrupt_status_offsets[crtc].vblank)3097 if (disp_int & interrupt_status_offsets[crtc].vblank)
3099 WREG32(mmVBLANK_STATUS + crtc_offsets[crtc], VBLANK_ACK);3098 WREG32(mmVBLANK_STATUS + crtc_offsets[crtc], VBLANK_STATUS__VBLANK_ACK_MASK);
3100 else3099 else
3101 DRM_DEBUG("IH: IH event w/o asserted irq bit?\n");3100 DRM_DEBUG("IH: IH event w/o asserted irq bit?\n");
3102 3101
@@ -3107,7 +3106,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 4 unmodified lines
3107 break;3106 break;
3108 case 1: /* vline */3107 case 1: /* vline */
3109 if (disp_int & interrupt_status_offsets[crtc].vline)3108 if (disp_int & interrupt_status_offsets[crtc].vline)
3110 WREG32(mmVLINE_STATUS + crtc_offsets[crtc], VLINE_ACK);3109 WREG32(mmVLINE_STATUS + crtc_offsets[crtc], VLINE_STATUS__VLINE_ACK_MASK);
3111 else3110 else
3112 DRM_DEBUG("IH: IH event w/o asserted irq bit?\n");3111 DRM_DEBUG("IH: IH event w/o asserted irq bit?\n");
3113 3112
@@ -3121,7 +3120,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 7 unmodified lines
3121 return 0;3120 return 0;
3122}3121}
3123 3122
3124static int dce_v6_0_set_pageflip_interrupt_state(struct amdgpu_device *adev,3123static int dce_v6_0_set_pageflip_irq_state(struct amdgpu_device *adev,
3125 struct amdgpu_irq_src *src,3124 struct amdgpu_irq_src *src,
3126 unsigned type,3125 unsigned type,
3127 enum amdgpu_interrupt_state state)3126 enum amdgpu_interrupt_state state)
@@ -3172,7 +3171,7 @@ static int dce_v6_0_pageflip_irq(struct amdgpu_device *adev, · 44 unmodified lines
3172 3171
3173 spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);3172 spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);
3174 works = amdgpu_crtc->pflip_works;3173 works = amdgpu_crtc->pflip_works;
3175 if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED){3174 if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED) {
3176 DRM_DEBUG_DRIVER("amdgpu_crtc->pflip_status = %d != "3175 DRM_DEBUG_DRIVER("amdgpu_crtc->pflip_status = %d != "
3177 "AMDGPU_FLIP_SUBMITTED(%d)\n",3176 "AMDGPU_FLIP_SUBMITTED(%d)\n",
3178 amdgpu_crtc->pflip_status,3177 amdgpu_crtc->pflip_status,
@@ -3249,12 +3248,10 @@ static const struct amd_ip_funcs dce_v6_0_ip_funcs = { · 70 unmodified lines
3249 .set_powergating_state = dce_v6_0_set_powergating_state,3248 .set_powergating_state = dce_v6_0_set_powergating_state,
3250};3249};
3251 3250
3252static void3251static void dce_v6_0_encoder_mode_set(struct drm_encoder *encoder,
3253dce_v6_0_encoder_mode_set(struct drm_encoder *encoder,
3254 struct drm_display_mode *mode,3252 struct drm_display_mode *mode,
3255 struct drm_display_mode *adjusted_mode)3253 struct drm_display_mode *adjusted_mode)
3256{3254{
3257
3258 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);3255 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
3259 int em = amdgpu_atombios_encoder_get_encoder_mode(encoder);3256 int em = amdgpu_atombios_encoder_get_encoder_mode(encoder);
3260 3257
@@ -3274,7 +3271,6 @@ dce_v6_0_encoder_mode_set(struct drm_encoder *encoder, · 13 unmodified lines
3274 3271
3275static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder)3272static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder)
3276{3273{
3277
3278 struct amdgpu_device *adev = drm_to_adev(encoder->dev);3274 struct amdgpu_device *adev = drm_to_adev(encoder->dev);
3279 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);3275 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
3280 struct drm_connector *connector = amdgpu_get_connector_for_encoder(encoder);3276 struct drm_connector *connector = amdgpu_get_connector_for_encoder(encoder);
@@ -3314,7 +3310,6 @@ static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder) · 33 unmodified lines
3314 3310
3315static void dce_v6_0_encoder_commit(struct drm_encoder *encoder)3311static void dce_v6_0_encoder_commit(struct drm_encoder *encoder)
3316{3312{
3317
3318 struct drm_device *dev = encoder->dev;3313 struct drm_device *dev = encoder->dev;
3319 struct amdgpu_device *adev = drm_to_adev(dev);3314 struct amdgpu_device *adev = drm_to_adev(dev);
3320 3315
@@ -3325,7 +3320,6 @@ static void dce_v6_0_encoder_commit(struct drm_encoder *encoder) · 4 unmodified lines
3325 3320
3326static void dce_v6_0_encoder_disable(struct drm_encoder *encoder)3321static void dce_v6_0_encoder_disable(struct drm_encoder *encoder)
3327{3322{
3328
3329 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);3323 struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder);
3330 struct amdgpu_encoder_atom_dig *dig;3324 struct amdgpu_encoder_atom_dig *dig;
3331 int em = amdgpu_atombios_encoder_get_encoder_mode(encoder);3325 int em = amdgpu_atombios_encoder_get_encoder_mode(encoder);
@@ -3541,17 +3535,17 @@ static void dce_v6_0_set_display_funcs(struct amdgpu_device *adev) · 209 unmodified lines
3541}3535}
3542 3536
3543static const struct amdgpu_irq_src_funcs dce_v6_0_crtc_irq_funcs = {3537static const struct amdgpu_irq_src_funcs dce_v6_0_crtc_irq_funcs = {
3544 .set = dce_v6_0_set_crtc_interrupt_state,3538 .set = dce_v6_0_set_crtc_irq_state,
3545 .process = dce_v6_0_crtc_irq,3539 .process = dce_v6_0_crtc_irq,
3546};3540};
3547 3541
3548static const struct amdgpu_irq_src_funcs dce_v6_0_pageflip_irq_funcs = {3542static const struct amdgpu_irq_src_funcs dce_v6_0_pageflip_irq_funcs = {
3549 .set = dce_v6_0_set_pageflip_interrupt_state,3543 .set = dce_v6_0_set_pageflip_irq_state,
3550 .process = dce_v6_0_pageflip_irq,3544 .process = dce_v6_0_pageflip_irq,
3551};3545};
3552 3546
3553static const struct amdgpu_irq_src_funcs dce_v6_0_hpd_irq_funcs = {3547static const struct amdgpu_irq_src_funcs dce_v6_0_hpd_irq_funcs = {
3554 .set = dce_v6_0_set_hpd_interrupt_state,3548 .set = dce_v6_0_set_hpd_irq_state,
3555 .process = dce_v6_0_hpd_irq,3549 .process = dce_v6_0_hpd_irq,
3556};3550};
3557 3551
drivers/gpu/drm/amd/amdgpu/dce_v8_0.c+8−8
@@ -271,7 +271,7 @@ static void dce_v8_0_hpd_int_ack(struct amdgpu_device *adev, · 270 unmodified lines
271 u32 tmp;271 u32 tmp;
272 272
273 if (hpd >= adev->mode_info.num_hpd) {273 if (hpd >= adev->mode_info.num_hpd) {
274 DRM_DEBUG("invalid hdp %d\n", hpd);274 DRM_DEBUG("invalid hpd %d\n", hpd);
275 return;275 return;
276 }276 }
277 277
@@ -3021,7 +3021,7 @@ static void dce_v8_0_set_crtc_vline_interrupt_state(struct amdgpu_device *adev, · 2743 unmodified lines
3021 }3021 }
3022}3022}
3023 3023
3024static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev,3024static int dce_v8_0_set_hpd_irq_state(struct amdgpu_device *adev,
3025 struct amdgpu_irq_src *src,3025 struct amdgpu_irq_src *src,
3026 unsigned type,3026 unsigned type,
3027 enum amdgpu_interrupt_state state)3027 enum amdgpu_interrupt_state state)
@@ -3029,7 +3029,7 @@ static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 1 unmodified line
3029 u32 dc_hpd_int_cntl;3029 u32 dc_hpd_int_cntl;
3030 3030
3031 if (type >= adev->mode_info.num_hpd) {3031 if (type >= adev->mode_info.num_hpd) {
3032 DRM_DEBUG("invalid hdp %d\n", type);3032 DRM_DEBUG("invalid hpd %d\n", type);
3033 return 0;3033 return 0;
3034 }3034 }
3035 3035
@@ -3051,7 +3051,7 @@ static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 15 unmodified lines
3051 return 0;3051 return 0;
3052}3052}
3053 3053
3054static int dce_v8_0_set_crtc_interrupt_state(struct amdgpu_device *adev,3054static int dce_v8_0_set_crtc_irq_state(struct amdgpu_device *adev,
3055 struct amdgpu_irq_src *src,3055 struct amdgpu_irq_src *src,
3056 unsigned type,3056 unsigned type,
3057 enum amdgpu_interrupt_state state)3057 enum amdgpu_interrupt_state state)
@@ -3136,7 +3136,7 @@ static int dce_v8_0_crtc_irq(struct amdgpu_device *adev, · 78 unmodified lines
3136 return 0;3136 return 0;
3137}3137}
3138 3138
3139static int dce_v8_0_set_pageflip_interrupt_state(struct amdgpu_device *adev,3139static int dce_v8_0_set_pageflip_irq_state(struct amdgpu_device *adev,
3140 struct amdgpu_irq_src *src,3140 struct amdgpu_irq_src *src,
3141 unsigned type,3141 unsigned type,
3142 enum amdgpu_interrupt_state state)3142 enum amdgpu_interrupt_state state)
@@ -3547,17 +3547,17 @@ static void dce_v8_0_set_display_funcs(struct amdgpu_device *adev) · 404 unmodified lines
3547}3547}
3548 3548
3549static const struct amdgpu_irq_src_funcs dce_v8_0_crtc_irq_funcs = {3549static const struct amdgpu_irq_src_funcs dce_v8_0_crtc_irq_funcs = {
3550 .set = dce_v8_0_set_crtc_interrupt_state,3550 .set = dce_v8_0_set_crtc_irq_state,
3551 .process = dce_v8_0_crtc_irq,3551 .process = dce_v8_0_crtc_irq,
3552};3552};
3553 3553
3554static const struct amdgpu_irq_src_funcs dce_v8_0_pageflip_irq_funcs = {3554static const struct amdgpu_irq_src_funcs dce_v8_0_pageflip_irq_funcs = {
3555 .set = dce_v8_0_set_pageflip_interrupt_state,3555 .set = dce_v8_0_set_pageflip_irq_state,
3556 .process = dce_v8_0_pageflip_irq,3556 .process = dce_v8_0_pageflip_irq,
3557};3557};
3558 3558
3559static const struct amdgpu_irq_src_funcs dce_v8_0_hpd_irq_funcs = {3559static const struct amdgpu_irq_src_funcs dce_v8_0_hpd_irq_funcs = {
3560 .set = dce_v8_0_set_hpd_interrupt_state,3560 .set = dce_v8_0_set_hpd_irq_state,
3561 .process = dce_v8_0_hpd_irq,3561 .process = dce_v8_0_hpd_irq,
3562};3562};
3563 3563
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c+91−48
@@ -368,11 +368,6 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_10_1[] = { · 367 unmodified lines
368 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_DEBUG_INST_ADDR),368 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_DEBUG_INST_ADDR),
369 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_LX6_CORE_PDEBUG_INST),369 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_LX6_CORE_PDEBUG_INST),
370 /* cp header registers */370 /* cp header registers */
371 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
372 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
373 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME2_HEADER_DUMP),
374 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
375 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
376 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MES_HEADER_DUMP),371 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MES_HEADER_DUMP),
377 /* SE status registers */372 /* SE status registers */
378 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0),373 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0),
@@ -421,7 +416,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_10[] = { · 42 unmodified lines
421 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_OFFSET),416 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_OFFSET),
422 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),417 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),
423 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_WG_STATE_OFFSET),418 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_WG_STATE_OFFSET),
424 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_DEQUEUE_STATUS)419 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_DEQUEUE_STATUS),
420 /* cp header registers */
421 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
422 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
423 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
424 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
425 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
426 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
427 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
428 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
425};429};
426 430
427static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = {431static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = {
@@ -448,7 +452,32 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = { · 20 unmodified lines
448 SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR),452 SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR),
449 SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR_HI),453 SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR_HI),
450 SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO),454 SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO),
451 SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI)455 SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI),
456 /* gfx header registers */
457 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
458 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
459 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
460 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
461 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
462 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
463 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
464 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
465 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
466 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
467 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
468 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
469 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
470 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
471 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
472 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
473 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
474 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
475 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
476 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
477 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
478 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
479 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
480 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
452};481};
453 482
454static const struct soc15_reg_golden golden_settings_gc_10_1[] = {483static const struct soc15_reg_golden golden_settings_gc_10_1[] = {
@@ -4296,9 +4325,7 @@ static u32 gfx_v10_0_get_csb_size(struct amdgpu_device *adev) · 3841 unmodified lines
4296static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev,4325static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev,
4297 volatile u32 *buffer)4326 volatile u32 *buffer)
4298{4327{
4299 u32 count = 0, i;4328 u32 count = 0;
4300 const struct cs_section_def *sect = NULL;
4301 const struct cs_extent_def *ext = NULL;
4302 int ctx_reg_offset;4329 int ctx_reg_offset;
4303 4330
4304 if (adev->gfx.rlc.cs_data == NULL)4331 if (adev->gfx.rlc.cs_data == NULL)
@@ -4306,39 +4333,15 @@ static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev, · 1 unmodified line
4306 if (buffer == NULL)4333 if (buffer == NULL)
4307 return;4334 return;
4308 4335
4309 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));4336 count = amdgpu_gfx_csb_preamble_start(buffer);
4310 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);4337 count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
4311 4338
4312 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));4339 ctx_reg_offset = SOC15_REG_OFFSET(GC, 0, mmPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
4313 buffer[count++] = cpu_to_le32(0x80000000);
4314 buffer[count++] = cpu_to_le32(0x80000000);
4315
4316 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
4317 for (ext = sect->section; ext->extent != NULL; ++ext) {
4318 if (sect->id == SECT_CONTEXT) {
4319 buffer[count++] =
4320 cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
4321 buffer[count++] = cpu_to_le32(ext->reg_index -
4322 PACKET3_SET_CONTEXT_REG_START);
4323 for (i = 0; i < ext->reg_count; i++)
4324 buffer[count++] = cpu_to_le32(ext->extent[i]);
4325 } else {
4326 return;
4327 }
4328 }
4329 }
4330
4331 ctx_reg_offset =
4332 SOC15_REG_OFFSET(GC, 0, mmPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
4333 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));4340 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));
4334 buffer[count++] = cpu_to_le32(ctx_reg_offset);4341 buffer[count++] = cpu_to_le32(ctx_reg_offset);
4335 buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override);4342 buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override);
4336 4343
4337 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));4344 amdgpu_gfx_csb_preamble_end(buffer, count);
4338 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
4339
4340 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
4341 buffer[count++] = cpu_to_le32(0);
4342}4345}
4343 4346
4344static void gfx_v10_0_rlc_fini(struct amdgpu_device *adev)4347static void gfx_v10_0_rlc_fini(struct amdgpu_device *adev)
@@ -4752,6 +4755,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 407 unmodified lines
4752 int i, j, k, r, ring_id = 0;4755 int i, j, k, r, ring_id = 0;
4753 int xcc_id = 0;4756 int xcc_id = 0;
4754 struct amdgpu_device *adev = ip_block->adev;4757 struct amdgpu_device *adev = ip_block->adev;
4758 int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
4755 4759
4756 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);4760 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);
4757 4761
@@ -4763,7 +4767,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
4763 case IP_VERSION(10, 1, 4):4767 case IP_VERSION(10, 1, 4):
4764 adev->gfx.me.num_me = 1;4768 adev->gfx.me.num_me = 1;
4765 adev->gfx.me.num_pipe_per_me = 1;4769 adev->gfx.me.num_pipe_per_me = 1;
4766 adev->gfx.me.num_queue_per_pipe = 1;4770 adev->gfx.me.num_queue_per_pipe = 8;
4767 adev->gfx.mec.num_mec = 2;4771 adev->gfx.mec.num_mec = 2;
4768 adev->gfx.mec.num_pipe_per_mec = 4;4772 adev->gfx.mec.num_pipe_per_mec = 4;
4769 adev->gfx.mec.num_queue_per_pipe = 8;4773 adev->gfx.mec.num_queue_per_pipe = 8;
@@ -4778,7 +4782,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
4778 case IP_VERSION(10, 3, 7):4782 case IP_VERSION(10, 3, 7):
4779 adev->gfx.me.num_me = 1;4783 adev->gfx.me.num_me = 1;
4780 adev->gfx.me.num_pipe_per_me = 2;4784 adev->gfx.me.num_pipe_per_me = 2;
4781 adev->gfx.me.num_queue_per_pipe = 1;4785 adev->gfx.me.num_queue_per_pipe = 2;
4782 adev->gfx.mec.num_mec = 2;4786 adev->gfx.mec.num_mec = 2;
4783 adev->gfx.mec.num_pipe_per_mec = 4;4787 adev->gfx.mec.num_pipe_per_mec = 4;
4784 adev->gfx.mec.num_queue_per_pipe = 4;4788 adev->gfx.mec.num_queue_per_pipe = 4;
@@ -4800,7 +4804,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 15 unmodified lines
4800 adev->gfx.cleaner_shader_size = sizeof(gfx_10_1_10_cleaner_shader_hex);4804 adev->gfx.cleaner_shader_size = sizeof(gfx_10_1_10_cleaner_shader_hex);
4801 if (adev->gfx.me_fw_version >= 101 &&4805 if (adev->gfx.me_fw_version >= 101 &&
4802 adev->gfx.pfp_fw_version >= 158 &&4806 adev->gfx.pfp_fw_version >= 158 &&
4803 adev->gfx.mec_fw_version >= 152) {4807 adev->gfx.mec_fw_version >= 151) {
4804 adev->gfx.enable_cleaner_shader = true;4808 adev->gfx.enable_cleaner_shader = true;
4805 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);4809 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
4806 if (r) {4810 if (r) {
@@ -4810,7 +4814,9 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
4810 }4814 }
4811 break;4815 break;
4812 case IP_VERSION(10, 3, 0):4816 case IP_VERSION(10, 3, 0):
4817 case IP_VERSION(10, 3, 1):
4813 case IP_VERSION(10, 3, 2):4818 case IP_VERSION(10, 3, 2):
4819 case IP_VERSION(10, 3, 3):
4814 case IP_VERSION(10, 3, 4):4820 case IP_VERSION(10, 3, 4):
4815 case IP_VERSION(10, 3, 5):4821 case IP_VERSION(10, 3, 5):
4816 adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;4822 adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;
@@ -4826,6 +4832,34 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 9 unmodified lines
4826 }4832 }
4827 }4833 }
4828 break;4834 break;
4835 case IP_VERSION(10, 3, 6):
4836 adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;
4837 adev->gfx.cleaner_shader_size = sizeof(gfx_10_3_0_cleaner_shader_hex);
4838 if (adev->gfx.me_fw_version >= 14 &&
4839 adev->gfx.pfp_fw_version >= 17 &&
4840 adev->gfx.mec_fw_version >= 24) {
4841 adev->gfx.enable_cleaner_shader = true;
4842 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
4843 if (r) {
4844 adev->gfx.enable_cleaner_shader = false;
4845 dev_err(adev->dev, "Failed to initialize cleaner shader\n");
4846 }
4847 }
4848 break;
4849 case IP_VERSION(10, 3, 7):
4850 adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex;
4851 adev->gfx.cleaner_shader_size = sizeof(gfx_10_3_0_cleaner_shader_hex);
4852 if (adev->gfx.me_fw_version >= 4 &&
4853 adev->gfx.pfp_fw_version >= 9 &&
4854 adev->gfx.mec_fw_version >= 12) {
4855 adev->gfx.enable_cleaner_shader = true;
4856 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
4857 if (r) {
4858 adev->gfx.enable_cleaner_shader = false;
4859 dev_err(adev->dev, "Failed to initialize cleaner shader\n");
4860 }
4861 }
4862 break;
4829 default:4863 default:
4830 adev->gfx.enable_cleaner_shader = false;4864 adev->gfx.enable_cleaner_shader = false;
4831 break;4865 break;
@@ -4886,7 +4920,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 54 unmodified lines
4886 4920
4887 /* set up the gfx ring */4921 /* set up the gfx ring */
4888 for (i = 0; i < adev->gfx.me.num_me; i++) {4922 for (i = 0; i < adev->gfx.me.num_me; i++) {
4889 for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {4923 for (j = 0; j < num_queue_per_pipe; j++) {
4890 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {4924 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
4891 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))4925 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
4892 continue;4926 continue;
@@ -9645,9 +9679,14 @@ static void gfx_v10_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printe · 4752 unmodified lines
9645 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {9679 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {
9646 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);9680 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);
9647 for (reg = 0; reg < reg_count; reg++) {9681 for (reg = 0; reg < reg_count; reg++) {
9648 drm_printf(p, "%-50s \t 0x%08x\n",9682 if (i && gc_cp_reg_list_10[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
9649 gc_cp_reg_list_10[reg].reg_name,9683 drm_printf(p, "%-50s \t 0x%08x\n",
9650 adev->gfx.ip_dump_compute_queues[index + reg]);9684 "mmCP_MEC_ME2_HEADER_DUMP",
9685 adev->gfx.ip_dump_compute_queues[index + reg]);
9686 else
9687 drm_printf(p, "%-50s \t 0x%08x\n",
9688 gc_cp_reg_list_10[reg].reg_name,
9689 adev->gfx.ip_dump_compute_queues[index + reg]);
9651 }9690 }
9652 index += reg_count;9691 index += reg_count;
9653 }9692 }
@@ -9708,9 +9747,13 @@ static void gfx_v10_ip_dump(struct amdgpu_ip_block *ip_block) · 54 unmodified lines
9708 nv_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0);9747 nv_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0);
9709 9748
9710 for (reg = 0; reg < reg_count; reg++) {9749 for (reg = 0; reg < reg_count; reg++) {
9711 adev->gfx.ip_dump_compute_queues[index + reg] =9750 if (i && gc_cp_reg_list_10[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
9712 RREG32(SOC15_REG_ENTRY_OFFSET(9751 adev->gfx.ip_dump_compute_queues[index + reg] =
9713 gc_cp_reg_list_10[reg]));9752 RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_MEC_ME2_HEADER_DUMP));
9753 else
9754 adev->gfx.ip_dump_compute_queues[index + reg] =
9755 RREG32(SOC15_REG_ENTRY_OFFSET(
9756 gc_cp_reg_list_10[reg]));
9714 }9757 }
9715 index += reg_count;9758 index += reg_count;
9716 }9759 }
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c+539−193
@@ -48,6 +48,8 @@ · 47 unmodified lines
48#include "gfx_v11_0_3.h"48#include "gfx_v11_0_3.h"
49#include "nbio_v4_3.h"49#include "nbio_v4_3.h"
50#include "mes_v11_0.h"50#include "mes_v11_0.h"
51#include "mes_userqueue.h"
52#include "amdgpu_userq_fence.h"
51 53
52#define GFX11_NUM_GFX_RINGS 154#define GFX11_NUM_GFX_RINGS 1
53#define GFX11_MEC_HPD_SIZE 204855#define GFX11_MEC_HPD_SIZE 2048
@@ -177,9 +179,13 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_11_0[] = { · 123 unmodified lines
177 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_INSTR_PNTR),179 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_INSTR_PNTR),
178 SOC15_REG_ENTRY_STR(GC, 0, regCP_CPC_STATUS),180 SOC15_REG_ENTRY_STR(GC, 0, regCP_CPC_STATUS),
179 /* cp header registers */181 /* cp header registers */
180 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),182 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
181 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),183 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
182 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),184 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
185 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
186 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
187 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
188 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
183 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),189 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
184 /* SE status registers */190 /* SE status registers */
185 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),191 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),
@@ -230,7 +236,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_11[] = { · 44 unmodified lines
230 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET),236 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET),
231 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),237 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),
232 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET),238 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET),
233 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS)239 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS),
240 /* cp header registers */
241 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
242 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
243 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
244 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
245 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
246 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
247 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
248 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
234};249};
235 250
236static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = {251static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = {
@@ -259,7 +274,24 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = { · 22 unmodified lines
259 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO),274 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO),
260 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI),275 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI),
261 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ),276 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ),
262 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ)277 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ),
278 /* cp header registers */
279 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
280 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
281 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
282 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
283 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
284 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
285 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
286 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
287 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
288 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
289 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
290 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
291 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
292 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
293 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
294 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
263};295};
264 296
265static const struct soc15_reg_golden golden_settings_gc_11_0[] = {297static const struct soc15_reg_golden golden_settings_gc_11_0[] = {
@@ -580,33 +612,18 @@ static int gfx_v11_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 314 unmodified lines
580 612
581 memset(&ib, 0, sizeof(ib));613 memset(&ib, 0, sizeof(ib));
582 614
583 if (ring->is_mes_queue) {615 r = amdgpu_device_wb_get(adev, &index);
584 uint32_t padding, offset;616 if (r)
585 617 return r;
586 offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
587 padding = amdgpu_mes_ctx_get_offs(ring,
588 AMDGPU_MES_CTX_PADDING_OFFS);
589
590 ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
591 ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
592
593 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding);
594 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding);
595 *cpu_ptr = cpu_to_le32(0xCAFEDEAD);
596 } else {
597 r = amdgpu_device_wb_get(adev, &index);
598 if (r)
599 return r;
600 618
601 gpu_addr = adev->wb.gpu_addr + (index * 4);619 gpu_addr = adev->wb.gpu_addr + (index * 4);
602 adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);620 adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
603 cpu_ptr = &adev->wb.wb[index];621 cpu_ptr = &adev->wb.wb[index];
604 622
605 r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib);623 r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib);
606 if (r) {624 if (r) {
607 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);625 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
608 goto err1;626 goto err1;
609 }
610 }627 }
611 628
612 ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);629 ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);
@@ -633,12 +650,10 @@ static int gfx_v11_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 20 unmodified lines
633 else650 else
634 r = -EINVAL;651 r = -EINVAL;
635err2:652err2:
636 if (!ring->is_mes_queue)653 amdgpu_ib_free(&ib, NULL);
637 amdgpu_ib_free(&ib, NULL);
638 dma_fence_put(f);654 dma_fence_put(f);
639err1:655err1:
640 if (!ring->is_mes_queue)656 amdgpu_device_wb_free(adev, index);
641 amdgpu_device_wb_free(adev, index);
642 return r;657 return r;
643}658}
644 659
@@ -833,9 +848,7 @@ static u32 gfx_v11_0_get_csb_size(struct amdgpu_device *adev) · 188 unmodified lines
833static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev,848static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev,
834 volatile u32 *buffer)849 volatile u32 *buffer)
835{850{
836 u32 count = 0, i;851 u32 count = 0;
837 const struct cs_section_def *sect = NULL;
838 const struct cs_extent_def *ext = NULL;
839 int ctx_reg_offset;852 int ctx_reg_offset;
840 853
841 if (adev->gfx.rlc.cs_data == NULL)854 if (adev->gfx.rlc.cs_data == NULL)
@@ -843,39 +856,15 @@ static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev, · 1 unmodified line
843 if (buffer == NULL)856 if (buffer == NULL)
844 return;857 return;
845 858
846 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));859 count = amdgpu_gfx_csb_preamble_start(buffer);
847 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);860 count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
848
849 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
850 buffer[count++] = cpu_to_le32(0x80000000);
851 buffer[count++] = cpu_to_le32(0x80000000);
852
853 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
854 for (ext = sect->section; ext->extent != NULL; ++ext) {
855 if (sect->id == SECT_CONTEXT) {
856 buffer[count++] =
857 cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
858 buffer[count++] = cpu_to_le32(ext->reg_index -
859 PACKET3_SET_CONTEXT_REG_START);
860 for (i = 0; i < ext->reg_count; i++)
861 buffer[count++] = cpu_to_le32(ext->extent[i]);
862 } else {
863 return;
864 }
865 }
866 }
867 861
868 ctx_reg_offset =862 ctx_reg_offset = SOC15_REG_OFFSET(GC, 0, regPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
869 SOC15_REG_OFFSET(GC, 0, regPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START;
870 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));863 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));
871 buffer[count++] = cpu_to_le32(ctx_reg_offset);864 buffer[count++] = cpu_to_le32(ctx_reg_offset);
872 buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override);865 buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override);
873 866
874 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));867 amdgpu_gfx_csb_preamble_end(buffer, count);
875 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
876
877 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
878 buffer[count++] = cpu_to_le32(0);
879}868}
880 869
881static void gfx_v11_0_rlc_fini(struct amdgpu_device *adev)870static void gfx_v11_0_rlc_fini(struct amdgpu_device *adev)
@@ -1056,14 +1045,21 @@ static void gfx_v11_0_select_me_pipe_q(struct amdgpu_device *adev, · 174 unmodified lines
1056#define MQD_FWWORKAREA_SIZE 4841045#define MQD_FWWORKAREA_SIZE 484
1057#define MQD_FWWORKAREA_ALIGNMENT 2561046#define MQD_FWWORKAREA_ALIGNMENT 256
1058 1047
1059static int gfx_v11_0_get_gfx_shadow_info(struct amdgpu_device *adev,1048static void gfx_v11_0_get_gfx_shadow_info_nocheck(struct amdgpu_device *adev,
1060 struct amdgpu_gfx_shadow_info *shadow_info)1049 struct amdgpu_gfx_shadow_info *shadow_info)
1061{1050{
1062 if (adev->gfx.cp_gfx_shadow) {1051 shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE;
1063 shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE;1052 shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT;
1064 shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT;1053 shadow_info->csa_size = MQD_FWWORKAREA_SIZE;
1065 shadow_info->csa_size = MQD_FWWORKAREA_SIZE;1054 shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT;
1066 shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT;1055}
1056
1057static int gfx_v11_0_get_gfx_shadow_info(struct amdgpu_device *adev,
1058 struct amdgpu_gfx_shadow_info *shadow_info,
1059 bool skip_check)
1060{
1061 if (adev->gfx.cp_gfx_shadow || skip_check) {
1062 gfx_v11_0_get_gfx_shadow_info_nocheck(adev, shadow_info);
1067 return 0;1063 return 0;
1068 } else {1064 } else {
1069 memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info));1065 memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info));
@@ -1136,6 +1132,10 @@ static int gfx_v11_0_gfx_ring_init(struct amdgpu_device *adev, int ring_id, · 66 unmodified lines
1136 1132
1137 ring->ring_obj = NULL;1133 ring->ring_obj = NULL;
1138 ring->use_doorbell = true;1134 ring->use_doorbell = true;
1135 if (adev->gfx.disable_kq) {
1136 ring->no_scheduler = true;
1137 ring->no_user_submission = true;
1138 }
1139 1139
1140 if (!ring_id)1140 if (!ring_id)
1141 ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1;1141 ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1;
@@ -1568,24 +1568,18 @@ static void gfx_v11_0_alloc_ip_dump(struct amdgpu_device *adev) · 426 unmodified lines
1568 1568
1569static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block)1569static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block)
1570{1570{
1571 int i, j, k, r, ring_id = 0;1571 int i, j, k, r, ring_id;
1572 int xcc_id = 0;1572 int xcc_id = 0;
1573 struct amdgpu_device *adev = ip_block->adev;1573 struct amdgpu_device *adev = ip_block->adev;
1574 int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
1574 1575
1575 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);1576 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);
1576 1577
1577 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {1578 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1578 case IP_VERSION(11, 0, 0):1579 case IP_VERSION(11, 0, 0):
1580 case IP_VERSION(11, 0, 1):
1579 case IP_VERSION(11, 0, 2):1581 case IP_VERSION(11, 0, 2):
1580 case IP_VERSION(11, 0, 3):1582 case IP_VERSION(11, 0, 3):
1581 adev->gfx.me.num_me = 1;
1582 adev->gfx.me.num_pipe_per_me = 1;
1583 adev->gfx.me.num_queue_per_pipe = 1;
1584 adev->gfx.mec.num_mec = 1;
1585 adev->gfx.mec.num_pipe_per_mec = 4;
1586 adev->gfx.mec.num_queue_per_pipe = 4;
1587 break;
1588 case IP_VERSION(11, 0, 1):
1589 case IP_VERSION(11, 0, 4):1583 case IP_VERSION(11, 0, 4):
1590 case IP_VERSION(11, 5, 0):1584 case IP_VERSION(11, 5, 0):
1591 case IP_VERSION(11, 5, 1):1585 case IP_VERSION(11, 5, 1):
@@ -1593,7 +1587,7 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 1 unmodified line
1593 case IP_VERSION(11, 5, 3):1587 case IP_VERSION(11, 5, 3):
1594 adev->gfx.me.num_me = 1;1588 adev->gfx.me.num_me = 1;
1595 adev->gfx.me.num_pipe_per_me = 1;1589 adev->gfx.me.num_pipe_per_me = 1;
1596 adev->gfx.me.num_queue_per_pipe = 1;1590 adev->gfx.me.num_queue_per_pipe = 2;
1597 adev->gfx.mec.num_mec = 1;1591 adev->gfx.mec.num_mec = 1;
1598 adev->gfx.mec.num_pipe_per_mec = 4;1592 adev->gfx.mec.num_pipe_per_mec = 4;
1599 adev->gfx.mec.num_queue_per_pipe = 4;1593 adev->gfx.mec.num_queue_per_pipe = 4;
@@ -1608,6 +1602,35 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
1608 break;1602 break;
1609 }1603 }
1610 1604
1605 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1606 case IP_VERSION(11, 0, 0):
1607 case IP_VERSION(11, 0, 2):
1608 case IP_VERSION(11, 0, 3):
1609 if (!adev->gfx.disable_uq &&
1610 adev->gfx.me_fw_version >= 2390 &&
1611 adev->gfx.pfp_fw_version >= 2530 &&
1612 adev->gfx.mec_fw_version >= 2600 &&
1613 adev->mes.fw_version[0] >= 120) {
1614 adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs;
1615 adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs;
1616 }
1617 break;
1618 case IP_VERSION(11, 0, 1):
1619 case IP_VERSION(11, 0, 4):
1620 case IP_VERSION(11, 5, 0):
1621 case IP_VERSION(11, 5, 1):
1622 case IP_VERSION(11, 5, 2):
1623 case IP_VERSION(11, 5, 3):
1624 /* add firmware version checks here */
1625 if (0 && !adev->gfx.disable_uq) {
1626 adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs;
1627 adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs;
1628 }
1629 break;
1630 default:
1631 break;
1632 }
1633
1611 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {1634 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1612 case IP_VERSION(11, 0, 0):1635 case IP_VERSION(11, 0, 0):
1613 case IP_VERSION(11, 0, 2):1636 case IP_VERSION(11, 0, 2):
@@ -1640,6 +1663,34 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 26 unmodified lines
1640 }1663 }
1641 }1664 }
1642 break;1665 break;
1666 case IP_VERSION(11, 5, 2):
1667 adev->gfx.cleaner_shader_ptr = gfx_11_0_3_cleaner_shader_hex;
1668 adev->gfx.cleaner_shader_size = sizeof(gfx_11_0_3_cleaner_shader_hex);
1669 if (adev->gfx.me_fw_version >= 12 &&
1670 adev->gfx.pfp_fw_version >= 15 &&
1671 adev->gfx.mec_fw_version >= 15) {
1672 adev->gfx.enable_cleaner_shader = true;
1673 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
1674 if (r) {
1675 adev->gfx.enable_cleaner_shader = false;
1676 dev_err(adev->dev, "Failed to initialize cleaner shader\n");
1677 }
1678 }
1679 break;
1680 case IP_VERSION(11, 5, 3):
1681 adev->gfx.cleaner_shader_ptr = gfx_11_0_3_cleaner_shader_hex;
1682 adev->gfx.cleaner_shader_size = sizeof(gfx_11_0_3_cleaner_shader_hex);
1683 if (adev->gfx.me_fw_version >= 7 &&
1684 adev->gfx.pfp_fw_version >= 8 &&
1685 adev->gfx.mec_fw_version >= 8) {
1686 adev->gfx.enable_cleaner_shader = true;
1687 r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size);
1688 if (r) {
1689 adev->gfx.enable_cleaner_shader = false;
1690 dev_err(adev->dev, "Failed to initialize cleaner shader\n");
1691 }
1692 }
1693 break;
1643 default:1694 default:
1644 adev->gfx.enable_cleaner_shader = false;1695 adev->gfx.enable_cleaner_shader = false;
1645 break;1696 break;
@@ -1701,37 +1752,42 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 55 unmodified lines
1701 return r;1752 return r;
1702 }1753 }
1703 1754
1704 /* set up the gfx ring */1755 if (adev->gfx.num_gfx_rings) {
1705 for (i = 0; i < adev->gfx.me.num_me; i++) {1756 ring_id = 0;
1706 for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {1757 /* set up the gfx ring */
1707 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {1758 for (i = 0; i < adev->gfx.me.num_me; i++) {
1708 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))1759 for (j = 0; j < num_queue_per_pipe; j++) {
1709 continue;1760 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
1710 1761 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
1711 r = gfx_v11_0_gfx_ring_init(adev, ring_id,1762 continue;
1712 i, k, j);1763
1713 if (r)1764 r = gfx_v11_0_gfx_ring_init(adev, ring_id,
1714 return r;1765 i, k, j);
1715 ring_id++;1766 if (r)
1767 return r;
1768 ring_id++;
1769 }
1716 }1770 }
1717 }1771 }
1718 }1772 }
1719 1773
1720 ring_id = 0;1774 if (adev->gfx.num_compute_rings) {
1721 /* set up the compute queues - allocate horizontally across pipes */1775 ring_id = 0;
1722 for (i = 0; i < adev->gfx.mec.num_mec; ++i) {1776 /* set up the compute queues - allocate horizontally across pipes */
1723 for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {1777 for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1724 for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {1778 for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1725 if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,1779 for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
1726 k, j))1780 if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i,
1727 continue;1781 k, j))
1782 continue;
1728 1783
1729 r = gfx_v11_0_compute_ring_init(adev, ring_id,1784 r = gfx_v11_0_compute_ring_init(adev, ring_id,
1730 i, k, j);1785 i, k, j);
1731 if (r)1786 if (r)
1732 return r;1787 return r;
1733 1788
1734 ring_id++;1789 ring_id++;
1790 }
1735 }1791 }
1736 }1792 }
1737 }1793 }
@@ -4061,6 +4117,8 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 2323 unmodified lines
4061#ifdef __BIG_ENDIAN4117#ifdef __BIG_ENDIAN
4062 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1);4118 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1);
4063#endif4119#endif
4120 if (prop->tmz_queue)
4121 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, TMZ_MATCH, 1);
4064 mqd->cp_gfx_hqd_cntl = tmp;4122 mqd->cp_gfx_hqd_cntl = tmp;
4065 4123
4066 /* set up cp_doorbell_control */4124 /* set up cp_doorbell_control */
@@ -4081,6 +4139,16 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 14 unmodified lines
4081 /* active the queue */4139 /* active the queue */
4082 mqd->cp_gfx_hqd_active = 1;4140 mqd->cp_gfx_hqd_active = 1;
4083 4141
4142 /* set gfx UQ items */
4143 mqd->shadow_base_lo = lower_32_bits(prop->shadow_addr);
4144 mqd->shadow_base_hi = upper_32_bits(prop->shadow_addr);
4145 mqd->gds_bkup_base_lo = lower_32_bits(prop->gds_bkup_addr);
4146 mqd->gds_bkup_base_hi = upper_32_bits(prop->gds_bkup_addr);
4147 mqd->fw_work_area_base_lo = lower_32_bits(prop->csa_addr);
4148 mqd->fw_work_area_base_hi = upper_32_bits(prop->csa_addr);
4149 mqd->fence_address_lo = lower_32_bits(prop->fence_address);
4150 mqd->fence_address_hi = upper_32_bits(prop->fence_address);
4151
4084 return 0;4152 return 0;
4085}4153}
4086 4154
@@ -4205,6 +4273,8 @@ static int gfx_v11_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 118 unmodified lines
4205 prop->allow_tunneling);4273 prop->allow_tunneling);
4206 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);4274 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
4207 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);4275 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
4276 if (prop->tmz_queue)
4277 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TMZ, 1);
4208 mqd->cp_hqd_pq_control = tmp;4278 mqd->cp_hqd_pq_control = tmp;
4209 4279
4210 /* set the wb address whether it's enabled or not */4280 /* set the wb address whether it's enabled or not */
@@ -4256,6 +4326,10 @@ static int gfx_v11_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 45 unmodified lines
4256 4326
4257 mqd->cp_hqd_active = prop->hqd_active;4327 mqd->cp_hqd_active = prop->hqd_active;
4258 4328
4329 /* set UQ fenceaddress */
4330 mqd->fence_address_lo = lower_32_bits(prop->fence_address);
4331 mqd->fence_address_hi = upper_32_bits(prop->fence_address);
4332
4259 return 0;4333 return 0;
4260}4334}
4261 4335
@@ -4509,11 +4583,23 @@ static int gfx_v11_0_cp_resume(struct amdgpu_device *adev) · 247 unmodified lines
4509 return r;4583 return r;
4510 }4584 }
4511 4585
4512 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {4586 if (adev->gfx.disable_kq) {
4513 ring = &adev->gfx.gfx_ring[i];4587 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4514 r = amdgpu_ring_test_helper(ring);4588 ring = &adev->gfx.gfx_ring[i];
4515 if (r)4589 /* we don't want to set ring->ready */
4516 return r;4590 r = amdgpu_ring_test_ring(ring);
4591 if (r)
4592 return r;
4593 }
4594 if (amdgpu_async_gfx_ring)
4595 amdgpu_gfx_disable_kgq(adev, 0);
4596 } else {
4597 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4598 ring = &adev->gfx.gfx_ring[i];
4599 r = amdgpu_ring_test_helper(ring);
4600 if (r)
4601 return r;
4602 }
4517 }4603 }
4518 4604
4519 for (i = 0; i < adev->gfx.num_compute_rings; i++) {4605 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
@@ -4722,6 +4808,49 @@ static int gfx_v11_0_hw_init(struct amdgpu_ip_block *ip_block) · 202 unmodified lines
4722 return r;4808 return r;
4723}4809}
4724 4810
4811static int gfx_v11_0_set_userq_eop_interrupts(struct amdgpu_device *adev,
4812 bool enable)
4813{
4814 unsigned int irq_type;
4815 int m, p, r;
4816
4817 if (adev->userq_funcs[AMDGPU_HW_IP_GFX]) {
4818 for (m = 0; m < adev->gfx.me.num_me; m++) {
4819 for (p = 0; p < adev->gfx.me.num_pipe_per_me; p++) {
4820 irq_type = AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP + p;
4821 if (enable)
4822 r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
4823 irq_type);
4824 else
4825 r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
4826 irq_type);
4827 if (r)
4828 return r;
4829 }
4830 }
4831 }
4832
4833 if (adev->userq_funcs[AMDGPU_HW_IP_COMPUTE]) {
4834 for (m = 0; m < adev->gfx.mec.num_mec; ++m) {
4835 for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) {
4836 irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
4837 + (m * adev->gfx.mec.num_pipe_per_mec)
4838 + p;
4839 if (enable)
4840 r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
4841 irq_type);
4842 else
4843 r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
4844 irq_type);
4845 if (r)
4846 return r;
4847 }
4848 }
4849 }
4850
4851 return 0;
4852}
4853
4725static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block)4854static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block)
4726{4855{
4727 struct amdgpu_device *adev = ip_block->adev;4856 struct amdgpu_device *adev = ip_block->adev;
@@ -4731,9 +4860,11 @@ static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
4731 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);4860 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
4732 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);4861 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
4733 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);4862 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
4863 gfx_v11_0_set_userq_eop_interrupts(adev, false);
4734 4864
4735 if (!adev->no_hw_access) {4865 if (!adev->no_hw_access) {
4736 if (amdgpu_async_gfx_ring) {4866 if (amdgpu_async_gfx_ring &&
4867 !adev->gfx.disable_kq) {
4737 if (amdgpu_gfx_disable_kgq(adev, 0))4868 if (amdgpu_gfx_disable_kgq(adev, 0))
4738 DRM_ERROR("KGQ disable failed\n");4869 DRM_ERROR("KGQ disable failed\n");
4739 }4870 }
@@ -5059,11 +5190,36 @@ static int gfx_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 319 unmodified lines
5059{5190{
5060 struct amdgpu_device *adev = ip_block->adev;5191 struct amdgpu_device *adev = ip_block->adev;
5061 5192
5193 switch (amdgpu_user_queue) {
5194 case -1:
5195 case 0:
5196 default:
5197 adev->gfx.disable_kq = false;
5198 adev->gfx.disable_uq = true;
5199 break;
5200 case 1:
5201 adev->gfx.disable_kq = false;
5202 adev->gfx.disable_uq = false;
5203 break;
5204 case 2:
5205 adev->gfx.disable_kq = true;
5206 adev->gfx.disable_uq = false;
5207 break;
5208 }
5209
5062 adev->gfx.funcs = &gfx_v11_0_gfx_funcs;5210 adev->gfx.funcs = &gfx_v11_0_gfx_funcs;
5063 5211
5064 adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS;5212 if (adev->gfx.disable_kq) {
5065 adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),5213 /* We need one GFX ring temporarily to set up
5066 AMDGPU_MAX_COMPUTE_RINGS);5214 * the clear state.
5215 */
5216 adev->gfx.num_gfx_rings = 1;
5217 adev->gfx.num_compute_rings = 0;
5218 } else {
5219 adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS;
5220 adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
5221 AMDGPU_MAX_COMPUTE_RINGS);
5222 }
5067 5223
5068 gfx_v11_0_set_kiq_pm4_funcs(adev);5224 gfx_v11_0_set_kiq_pm4_funcs(adev);
5069 gfx_v11_0_set_ring_funcs(adev);5225 gfx_v11_0_set_ring_funcs(adev);
@@ -5094,6 +5250,11 @@ static int gfx_v11_0_late_init(struct amdgpu_ip_block *ip_block) · 24 unmodified lines
5094 r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0);5250 r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0);
5095 if (r)5251 if (r)
5096 return r;5252 return r;
5253
5254 r = gfx_v11_0_set_userq_eop_interrupts(adev, true);
5255 if (r)
5256 return r;
5257
5097 return 0;5258 return 0;
5098}5259}
5099 5260
@@ -5691,10 +5852,6 @@ static void gfx_v11_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, · 591 unmodified lines
5691 (!amdgpu_sriov_vf(ring->adev) && flags & AMDGPU_IB_PREEMPTED) ? true : false);5852 (!amdgpu_sriov_vf(ring->adev) && flags & AMDGPU_IB_PREEMPTED) ? true : false);
5692 }5853 }
5693 5854
5694 if (ring->is_mes_queue)
5695 /* inherit vmid from mqd */
5696 control |= 0x400000;
5697
5698 amdgpu_ring_write(ring, header);5855 amdgpu_ring_write(ring, header);
5699 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */5856 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
5700 amdgpu_ring_write(ring,5857 amdgpu_ring_write(ring,
@@ -5714,10 +5871,6 @@ static void gfx_v11_0_ring_emit_ib_compute(struct amdgpu_ring *ring, · 13 unmodified lines
5714 unsigned vmid = AMDGPU_JOB_GET_VMID(job);5871 unsigned vmid = AMDGPU_JOB_GET_VMID(job);
5715 u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);5872 u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);
5716 5873
5717 if (ring->is_mes_queue)
5718 /* inherit vmid from mqd */
5719 control |= 0x40000000;
5720
5721 /* Currently, there is a high possibility to get wave ID mismatch5874 /* Currently, there is a high possibility to get wave ID mismatch
5722 * between ME and GDS, leading to a hw deadlock, because ME generates5875 * between ME and GDS, leading to a hw deadlock, because ME generates
5723 * different wave IDs than the GDS expects. This situation happens5876 * different wave IDs than the GDS expects. This situation happens
@@ -5775,8 +5928,7 @@ static void gfx_v11_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, · 51 unmodified lines
5775 amdgpu_ring_write(ring, upper_32_bits(addr));5928 amdgpu_ring_write(ring, upper_32_bits(addr));
5776 amdgpu_ring_write(ring, lower_32_bits(seq));5929 amdgpu_ring_write(ring, lower_32_bits(seq));
5777 amdgpu_ring_write(ring, upper_32_bits(seq));5930 amdgpu_ring_write(ring, upper_32_bits(seq));
5778 amdgpu_ring_write(ring, ring->is_mes_queue ?5931 amdgpu_ring_write(ring, 0);
5779 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0);
5780}5932}
5781 5933
5782static void gfx_v11_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)5934static void gfx_v11_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
@@ -5804,10 +5956,7 @@ static void gfx_v11_0_ring_invalidate_tlbs(struct amdgpu_ring *ring, · 21 unmodified lines
5804static void gfx_v11_0_ring_emit_vm_flush(struct amdgpu_ring *ring,5956static void gfx_v11_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
5805 unsigned vmid, uint64_t pd_addr)5957 unsigned vmid, uint64_t pd_addr)
5806{5958{
5807 if (ring->is_mes_queue)5959 amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
5808 gfx_v11_0_ring_invalidate_tlbs(ring, 0, 0, false, 0);
5809 else
5810 amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
5811 5960
5812 /* compute doesn't have PFP */5961 /* compute doesn't have PFP */
5813 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {5962 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {
@@ -6036,28 +6185,13 @@ static void gfx_v11_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume) · 222 unmodified lines
6036 void *de_payload_cpu_addr;6185 void *de_payload_cpu_addr;
6037 int cnt;6186 int cnt;
6038 6187
6039 if (ring->is_mes_queue) {6188 offset = offsetof(struct v10_gfx_meta_data, de_payload);
6040 offset = offsetof(struct amdgpu_mes_ctx_meta_data,6189 de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
6041 gfx[0].gfx_meta_data) +6190 de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
6042 offsetof(struct v10_gfx_meta_data, de_payload);
6043 de_payload_gpu_addr =
6044 amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
6045 de_payload_cpu_addr =
6046 amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
6047
6048 offset = offsetof(struct amdgpu_mes_ctx_meta_data,
6049 gfx[0].gds_backup) +
6050 offsetof(struct v10_gfx_meta_data, de_payload);
6051 gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
6052 } else {
6053 offset = offsetof(struct v10_gfx_meta_data, de_payload);
6054 de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
6055 de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
6056 6191
6057 gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +6192 gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
6058 AMDGPU_CSA_SIZE - adev->gds.gds_size,6193 AMDGPU_CSA_SIZE - adev->gds.gds_size,
6059 PAGE_SIZE);6194 PAGE_SIZE);
6060 }
6061 6195
6062 de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);6196 de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);
6063 de_payload.gds_backup_addrhi = upper_32_bits(gds_addr);6197 de_payload.gds_backup_addrhi = upper_32_bits(gds_addr);
@@ -6296,25 +6430,23 @@ static int gfx_v11_0_eop_irq(struct amdgpu_device *adev, · 232 unmodified lines
6296 struct amdgpu_irq_src *source,6430 struct amdgpu_irq_src *source,
6297 struct amdgpu_iv_entry *entry)6431 struct amdgpu_iv_entry *entry)
6298{6432{
6299 int i;6433 u32 doorbell_offset = entry->src_data[0];
6300 u8 me_id, pipe_id, queue_id;6434 u8 me_id, pipe_id, queue_id;
6301 struct amdgpu_ring *ring;6435 struct amdgpu_ring *ring;
6302 uint32_t mes_queue_id = entry->src_data[0];6436 int i;
6303 6437
6304 DRM_DEBUG("IH: CP EOP\n");6438 DRM_DEBUG("IH: CP EOP\n");
6305 6439
6306 if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {6440 if (adev->enable_mes && doorbell_offset) {
6307 struct amdgpu_mes_queue *queue;6441 struct amdgpu_userq_fence_driver *fence_drv = NULL;
6308 6442 struct xarray *xa = &adev->userq_xa;
6309 mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;6443 unsigned long flags;
6310 6444
6311 spin_lock(&adev->mes.queue_id_lock);6445 xa_lock_irqsave(xa, flags);
6312 queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);6446 fence_drv = xa_load(xa, doorbell_offset);
6313 if (queue) {6447 if (fence_drv)
6314 DRM_DEBUG("process mes queue id = %d\n", mes_queue_id);6448 amdgpu_userq_fence_driver_process(fence_drv);
6315 amdgpu_fence_process(queue->ring);6449 xa_unlock_irqrestore(xa, flags);
6316 }
6317 spin_unlock(&adev->mes.queue_id_lock);
6318 } else {6450 } else {
6319 me_id = (entry->ring_id & 0x0c) >> 2;6451 me_id = (entry->ring_id & 0x0c) >> 2;
6320 pipe_id = (entry->ring_id & 0x03) >> 0;6452 pipe_id = (entry->ring_id & 0x03) >> 0;
@@ -6481,27 +6613,29 @@ static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev, · 160 unmodified lines
6481 pipe_id = (entry->ring_id & 0x03) >> 0;6613 pipe_id = (entry->ring_id & 0x03) >> 0;
6482 queue_id = (entry->ring_id & 0x70) >> 4;6614 queue_id = (entry->ring_id & 0x70) >> 4;
6483 6615
6484 switch (me_id) {6616 if (!adev->gfx.disable_kq) {
6485 case 0:6617 switch (me_id) {
6486 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {6618 case 0:
6487 ring = &adev->gfx.gfx_ring[i];6619 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
6488 if (ring->me == me_id && ring->pipe == pipe_id &&6620 ring = &adev->gfx.gfx_ring[i];
6489 ring->queue == queue_id)6621 if (ring->me == me_id && ring->pipe == pipe_id &&
6490 drm_sched_fault(&ring->sched);6622 ring->queue == queue_id)
6491 }6623 drm_sched_fault(&ring->sched);
6492 break;6624 }
6493 case 1:6625 break;
6494 case 2:6626 case 1:
6495 for (i = 0; i < adev->gfx.num_compute_rings; i++) {6627 case 2:
6496 ring = &adev->gfx.compute_ring[i];6628 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
6497 if (ring->me == me_id && ring->pipe == pipe_id &&6629 ring = &adev->gfx.compute_ring[i];
6498 ring->queue == queue_id)6630 if (ring->me == me_id && ring->pipe == pipe_id &&
6499 drm_sched_fault(&ring->sched);6631 ring->queue == queue_id)
6632 drm_sched_fault(&ring->sched);
6633 }
6634 break;
6635 default:
6636 BUG();
6637 break;
6500 }6638 }
6501 break;
6502 default:
6503 BUG();
6504 break;
6505 }6639 }
6506}6640}
6507 6641
@@ -6609,6 +6743,69 @@ static void gfx_v11_0_emit_mem_sync(struct amdgpu_ring *ring) · 101 unmodified lines
6609 amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */6743 amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */
6610}6744}
6611 6745
6746static bool gfx_v11_pipe_reset_support(struct amdgpu_device *adev)
6747{
6748 /* Disable the pipe reset until the CPFW fully support it.*/
6749 dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
6750 return false;
6751}
6752
6753
6754static int gfx_v11_reset_gfx_pipe(struct amdgpu_ring *ring)
6755{
6756 struct amdgpu_device *adev = ring->adev;
6757 uint32_t reset_pipe = 0, clean_pipe = 0;
6758 int r;
6759
6760 if (!gfx_v11_pipe_reset_support(adev))
6761 return -EOPNOTSUPP;
6762
6763 gfx_v11_0_set_safe_mode(adev, 0);
6764 mutex_lock(&adev->srbm_mutex);
6765 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
6766
6767 switch (ring->pipe) {
6768 case 0:
6769 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6770 PFP_PIPE0_RESET, 1);
6771 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6772 ME_PIPE0_RESET, 1);
6773 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6774 PFP_PIPE0_RESET, 0);
6775 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6776 ME_PIPE0_RESET, 0);
6777 break;
6778 case 1:
6779 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6780 PFP_PIPE1_RESET, 1);
6781 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6782 ME_PIPE1_RESET, 1);
6783 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6784 PFP_PIPE1_RESET, 0);
6785 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6786 ME_PIPE1_RESET, 0);
6787 break;
6788 default:
6789 break;
6790 }
6791
6792 WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
6793 WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
6794
6795 r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
6796 RS64_FW_UC_START_ADDR_LO;
6797 soc21_grbm_select(adev, 0, 0, 0, 0);
6798 mutex_unlock(&adev->srbm_mutex);
6799 gfx_v11_0_unset_safe_mode(adev, 0);
6800
6801 dev_info(adev->dev, "The ring %s pipe reset to the ME firmware start PC: %s\n", ring->name,
6802 r == 0 ? "successfully" : "failed");
6803 /* FIXME: Sometimes driver can't cache the ME firmware start PC correctly,
6804 * so the pipe reset status relies on the later gfx ring test result.
6805 */
6806 return 0;
6807}
6808
6612static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)6809static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
6613{6810{
6614 struct amdgpu_device *adev = ring->adev;6811 struct amdgpu_device *adev = ring->adev;
@@ -6618,8 +6815,13 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 3 unmodified lines
6618 return -EINVAL;6815 return -EINVAL;
6619 6816
6620 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false);6817 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false);
6621 if (r)6818 if (r) {
6622 return r;6819
6820 dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r);
6821 r = gfx_v11_reset_gfx_pipe(ring);
6822 if (r)
6823 return r;
6824 }
6623 6825
6624 r = gfx_v11_0_kgq_init_queue(ring, true);6826 r = gfx_v11_0_kgq_init_queue(ring, true);
6625 if (r) {6827 if (r) {
@@ -6636,6 +6838,136 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 10 unmodified lines
6636 return amdgpu_ring_test_ring(ring);6838 return amdgpu_ring_test_ring(ring);
6637}6839}
6638 6840
6841static int gfx_v11_0_reset_compute_pipe(struct amdgpu_ring *ring)
6842{
6843
6844 struct amdgpu_device *adev = ring->adev;
6845 uint32_t reset_pipe = 0, clean_pipe = 0;
6846 int r;
6847
6848 if (!gfx_v11_pipe_reset_support(adev))
6849 return -EOPNOTSUPP;
6850
6851 gfx_v11_0_set_safe_mode(adev, 0);
6852 mutex_lock(&adev->srbm_mutex);
6853 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
6854
6855 reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
6856 clean_pipe = reset_pipe;
6857
6858 if (adev->gfx.rs64_enable) {
6859
6860 switch (ring->pipe) {
6861 case 0:
6862 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6863 MEC_PIPE0_RESET, 1);
6864 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6865 MEC_PIPE0_RESET, 0);
6866 break;
6867 case 1:
6868 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6869 MEC_PIPE1_RESET, 1);
6870 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6871 MEC_PIPE1_RESET, 0);
6872 break;
6873 case 2:
6874 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6875 MEC_PIPE2_RESET, 1);
6876 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6877 MEC_PIPE2_RESET, 0);
6878 break;
6879 case 3:
6880 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6881 MEC_PIPE3_RESET, 1);
6882 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6883 MEC_PIPE3_RESET, 0);
6884 break;
6885 default:
6886 break;
6887 }
6888 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe);
6889 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe);
6890 r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
6891 RS64_FW_UC_START_ADDR_LO;
6892 } else {
6893 if (ring->me == 1) {
6894 switch (ring->pipe) {
6895 case 0:
6896 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6897 MEC_ME1_PIPE0_RESET, 1);
6898 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6899 MEC_ME1_PIPE0_RESET, 0);
6900 break;
6901 case 1:
6902 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6903 MEC_ME1_PIPE1_RESET, 1);
6904 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6905 MEC_ME1_PIPE1_RESET, 0);
6906 break;
6907 case 2:
6908 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6909 MEC_ME1_PIPE2_RESET, 1);
6910 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6911 MEC_ME1_PIPE2_RESET, 0);
6912 break;
6913 case 3:
6914 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6915 MEC_ME1_PIPE3_RESET, 1);
6916 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6917 MEC_ME1_PIPE3_RESET, 0);
6918 break;
6919 default:
6920 break;
6921 }
6922 /* mec1 fw pc: CP_MEC1_INSTR_PNTR */
6923 } else {
6924 switch (ring->pipe) {
6925 case 0:
6926 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6927 MEC_ME2_PIPE0_RESET, 1);
6928 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6929 MEC_ME2_PIPE0_RESET, 0);
6930 break;
6931 case 1:
6932 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6933 MEC_ME2_PIPE1_RESET, 1);
6934 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6935 MEC_ME2_PIPE1_RESET, 0);
6936 break;
6937 case 2:
6938 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6939 MEC_ME2_PIPE2_RESET, 1);
6940 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6941 MEC_ME2_PIPE2_RESET, 0);
6942 break;
6943 case 3:
6944 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
6945 MEC_ME2_PIPE3_RESET, 1);
6946 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
6947 MEC_ME2_PIPE3_RESET, 0);
6948 break;
6949 default:
6950 break;
6951 }
6952 /* mec2 fw pc: CP:CP_MEC2_INSTR_PNTR */
6953 }
6954 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe);
6955 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe);
6956 r = RREG32(SOC15_REG_OFFSET(GC, 0, regCP_MEC1_INSTR_PNTR));
6957 }
6958
6959 soc21_grbm_select(adev, 0, 0, 0, 0);
6960 mutex_unlock(&adev->srbm_mutex);
6961 gfx_v11_0_unset_safe_mode(adev, 0);
6962
6963 dev_info(adev->dev, "The ring %s pipe resets to MEC FW start PC: %s\n", ring->name,
6964 r == 0 ? "successfully" : "failed");
6965 /*FIXME:Sometimes driver can't cache the MEC firmware start PC correctly, so the pipe
6966 * reset status relies on the compute ring test result.
6967 */
6968 return 0;
6969}
6970
6639static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)6971static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)
6640{6972{
6641 struct amdgpu_device *adev = ring->adev;6973 struct amdgpu_device *adev = ring->adev;
@@ -6646,8 +6978,10 @@ static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines
6646 6978
6647 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true);6979 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true);
6648 if (r) {6980 if (r) {
6649 dev_err(adev->dev, "reset via MMIO failed %d\n", r);6981 dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r);
6650 return r;6982 r = gfx_v11_0_reset_compute_pipe(ring);
6983 if (r)
6984 return r;
6651 }6985 }
6652 6986
6653 r = gfx_v11_0_kcq_init_queue(ring, true);6987 r = gfx_v11_0_kcq_init_queue(ring, true);
@@ -6693,9 +7027,14 @@ static void gfx_v11_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printe · 39 unmodified lines
6693 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {7027 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {
6694 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);7028 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);
6695 for (reg = 0; reg < reg_count; reg++) {7029 for (reg = 0; reg < reg_count; reg++) {
6696 drm_printf(p, "%-50s \t 0x%08x\n",7030 if (i && gc_cp_reg_list_11[reg].reg_offset == regCP_MEC_ME1_HEADER_DUMP)
6697 gc_cp_reg_list_11[reg].reg_name,7031 drm_printf(p, "%-50s \t 0x%08x\n",
6698 adev->gfx.ip_dump_compute_queues[index + reg]);7032 "regCP_MEC_ME2_HEADER_DUMP",
7033 adev->gfx.ip_dump_compute_queues[index + reg]);
7034 else
7035 drm_printf(p, "%-50s \t 0x%08x\n",
7036 gc_cp_reg_list_11[reg].reg_name,
7037 adev->gfx.ip_dump_compute_queues[index + reg]);
6699 }7038 }
6700 index += reg_count;7039 index += reg_count;
6701 }7040 }
@@ -6755,9 +7094,16 @@ static void gfx_v11_ip_dump(struct amdgpu_ip_block *ip_block) · 53 unmodified lines
6755 /* ME0 is for GFX so start from 1 for CP */7094 /* ME0 is for GFX so start from 1 for CP */
6756 soc21_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0);7095 soc21_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0);
6757 for (reg = 0; reg < reg_count; reg++) {7096 for (reg = 0; reg < reg_count; reg++) {
6758 adev->gfx.ip_dump_compute_queues[index + reg] =7097 if (i &&
6759 RREG32(SOC15_REG_ENTRY_OFFSET(7098 gc_cp_reg_list_11[reg].reg_offset ==
6760 gc_cp_reg_list_11[reg]));7099 regCP_MEC_ME1_HEADER_DUMP)
7100 adev->gfx.ip_dump_compute_queues[index + reg] =
7101 RREG32(SOC15_REG_OFFSET(GC, 0,
7102 regCP_MEC_ME2_HEADER_DUMP));
7103 else
7104 adev->gfx.ip_dump_compute_queues[index + reg] =
7105 RREG32(SOC15_REG_ENTRY_OFFSET(
7106 gc_cp_reg_list_11[reg]));
6761 }7107 }
6762 index += reg_count;7108 index += reg_count;
6763 }7109 }
drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c+425−191
@@ -44,6 +44,8 @@ · 43 unmodified lines
44#include "gfx_v12_0.h"44#include "gfx_v12_0.h"
45#include "nbif_v6_3_1.h"45#include "nbif_v6_3_1.h"
46#include "mes_v12_0.h"46#include "mes_v12_0.h"
47#include "mes_userqueue.h"
48#include "amdgpu_userq_fence.h"
47 49
48#define GFX12_NUM_GFX_RINGS 150#define GFX12_NUM_GFX_RINGS 1
49#define GFX12_MEC_HPD_SIZE 204851#define GFX12_MEC_HPD_SIZE 2048
@@ -133,11 +135,14 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_12_0[] = { · 83 unmodified lines
133 SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR0),135 SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR0),
134 SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR1),136 SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR1),
135 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_RS64_INSTR_PNTR),137 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_RS64_INSTR_PNTR),
136
137 /* cp header registers */138 /* cp header registers */
138 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),139 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
139 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),140 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
140 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),141 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
142 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
143 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
144 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
145 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
141 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),146 SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP),
142 /* SE status registers */147 /* SE status registers */
143 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),148 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),
@@ -186,7 +191,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_12[] = { · 42 unmodified lines
186 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET),191 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET),
187 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),192 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT),
188 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET),193 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET),
189 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS)194 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS),
195 /* cp header registers */
196 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
197 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
198 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
199 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
200 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
201 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
202 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
203 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
190};204};
191 205
192static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = {206static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = {
@@ -215,7 +229,24 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = { · 22 unmodified lines
215 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO),229 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO),
216 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI),230 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI),
217 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ),231 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ),
218 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ)232 SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ),
233 /* cp header registers */
234 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
235 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
236 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
237 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
238 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
239 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
240 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
241 SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP),
242 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
243 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
244 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
245 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
246 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
247 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
248 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
249 SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP),
219};250};
220 251
221static const struct soc15_reg_golden golden_settings_gc_12_0_rev0[] = {252static const struct soc15_reg_golden golden_settings_gc_12_0_rev0[] = {
@@ -475,33 +506,18 @@ static int gfx_v12_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 253 unmodified lines
475 506
476 memset(&ib, 0, sizeof(ib));507 memset(&ib, 0, sizeof(ib));
477 508
478 if (ring->is_mes_queue) {509 r = amdgpu_device_wb_get(adev, &index);
479 uint32_t padding, offset;510 if (r)
480 511 return r;
481 offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);
482 padding = amdgpu_mes_ctx_get_offs(ring,
483 AMDGPU_MES_CTX_PADDING_OFFS);
484
485 ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
486 ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
487
488 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding);
489 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding);
490 *cpu_ptr = cpu_to_le32(0xCAFEDEAD);
491 } else {
492 r = amdgpu_device_wb_get(adev, &index);
493 if (r)
494 return r;
495 512
496 gpu_addr = adev->wb.gpu_addr + (index * 4);513 gpu_addr = adev->wb.gpu_addr + (index * 4);
497 adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);514 adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD);
498 cpu_ptr = &adev->wb.wb[index];515 cpu_ptr = &adev->wb.wb[index];
499 516
500 r = amdgpu_ib_get(adev, NULL, 16, AMDGPU_IB_POOL_DIRECT, &ib);517 r = amdgpu_ib_get(adev, NULL, 16, AMDGPU_IB_POOL_DIRECT, &ib);
501 if (r) {518 if (r) {
502 dev_err(adev->dev, "amdgpu: failed to get ib (%ld).\n", r);519 dev_err(adev->dev, "amdgpu: failed to get ib (%ld).\n", r);
503 goto err1;520 goto err1;
504 }
505 }521 }
506 522
507 ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);523 ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3);
@@ -528,12 +544,10 @@ static int gfx_v12_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 20 unmodified lines
528 else544 else
529 r = -EINVAL;545 r = -EINVAL;
530err2:546err2:
531 if (!ring->is_mes_queue)547 amdgpu_ib_free(&ib, NULL);
532 amdgpu_ib_free(&ib, NULL);
533 dma_fence_put(f);548 dma_fence_put(f);
534err1:549err1:
535 if (!ring->is_mes_queue)550 amdgpu_device_wb_free(adev, index);
536 amdgpu_device_wb_free(adev, index);
537 return r;551 return r;
538}552}
539 553
@@ -881,6 +895,34 @@ static void gfx_v12_0_select_me_pipe_q(struct amdgpu_device *adev, · 341 unmodified lines
881 soc24_grbm_select(adev, me, pipe, q, vm);895 soc24_grbm_select(adev, me, pipe, q, vm);
882}896}
883 897
898/* all sizes are in bytes */
899#define MQD_SHADOW_BASE_SIZE 73728
900#define MQD_SHADOW_BASE_ALIGNMENT 256
901#define MQD_FWWORKAREA_SIZE 484
902#define MQD_FWWORKAREA_ALIGNMENT 256
903
904static void gfx_v12_0_get_gfx_shadow_info_nocheck(struct amdgpu_device *adev,
905 struct amdgpu_gfx_shadow_info *shadow_info)
906{
907 shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE;
908 shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT;
909 shadow_info->csa_size = MQD_FWWORKAREA_SIZE;
910 shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT;
911}
912
913static int gfx_v12_0_get_gfx_shadow_info(struct amdgpu_device *adev,
914 struct amdgpu_gfx_shadow_info *shadow_info,
915 bool skip_check)
916{
917 if (adev->gfx.cp_gfx_shadow || skip_check) {
918 gfx_v12_0_get_gfx_shadow_info_nocheck(adev, shadow_info);
919 return 0;
920 }
921
922 memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info));
923 return -EINVAL;
924}
925
884static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = {926static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = {
885 .get_gpu_clock_counter = &gfx_v12_0_get_gpu_clock_counter,927 .get_gpu_clock_counter = &gfx_v12_0_get_gpu_clock_counter,
886 .select_se_sh = &gfx_v12_0_select_se_sh,928 .select_se_sh = &gfx_v12_0_select_se_sh,
@@ -889,6 +931,7 @@ static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = { · 2 unmodified lines
889 .read_wave_vgprs = &gfx_v12_0_read_wave_vgprs,931 .read_wave_vgprs = &gfx_v12_0_read_wave_vgprs,
890 .select_me_pipe_q = &gfx_v12_0_select_me_pipe_q,932 .select_me_pipe_q = &gfx_v12_0_select_me_pipe_q,
891 .update_perfmon_mgcg = &gfx_v12_0_update_perf_clk,933 .update_perfmon_mgcg = &gfx_v12_0_update_perf_clk,
934 .get_gfx_shadow_info = &gfx_v12_0_get_gfx_shadow_info,
892};935};
893 936
894static int gfx_v12_0_gpu_early_init(struct amdgpu_device *adev)937static int gfx_v12_0_gpu_early_init(struct amdgpu_device *adev)
@@ -1346,6 +1389,7 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 451 unmodified lines
1346 unsigned num_compute_rings;1389 unsigned num_compute_rings;
1347 int xcc_id = 0;1390 int xcc_id = 0;
1348 struct amdgpu_device *adev = ip_block->adev;1391 struct amdgpu_device *adev = ip_block->adev;
1392 int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */
1349 1393
1350 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);1394 INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler);
1351 1395
@@ -1354,7 +1398,7 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 2 unmodified lines
1354 case IP_VERSION(12, 0, 1):1398 case IP_VERSION(12, 0, 1):
1355 adev->gfx.me.num_me = 1;1399 adev->gfx.me.num_me = 1;
1356 adev->gfx.me.num_pipe_per_me = 1;1400 adev->gfx.me.num_pipe_per_me = 1;
1357 adev->gfx.me.num_queue_per_pipe = 1;1401 adev->gfx.me.num_queue_per_pipe = 8;
1358 adev->gfx.mec.num_mec = 1;1402 adev->gfx.mec.num_mec = 1;
1359 adev->gfx.mec.num_pipe_per_mec = 2;1403 adev->gfx.mec.num_pipe_per_mec = 2;
1360 adev->gfx.mec.num_queue_per_pipe = 4;1404 adev->gfx.mec.num_queue_per_pipe = 4;
@@ -1369,6 +1413,22 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
1369 break;1413 break;
1370 }1414 }
1371 1415
1416 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1417 case IP_VERSION(12, 0, 0):
1418 case IP_VERSION(12, 0, 1):
1419 if (!adev->gfx.disable_uq &&
1420 adev->gfx.me_fw_version >= 2780 &&
1421 adev->gfx.pfp_fw_version >= 2840 &&
1422 adev->gfx.mec_fw_version >= 3050 &&
1423 adev->mes.fw_version[0] >= 123) {
1424 adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs;
1425 adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs;
1426 }
1427 break;
1428 default:
1429 break;
1430 }
1431
1372 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {1432 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
1373 case IP_VERSION(12, 0, 0):1433 case IP_VERSION(12, 0, 0):
1374 case IP_VERSION(12, 0, 1):1434 case IP_VERSION(12, 0, 1):
@@ -1383,11 +1443,13 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines
1383 break;1443 break;
1384 }1444 }
1385 1445
1386 /* recalculate compute rings to use based on hardware configuration */1446 if (adev->gfx.num_compute_rings) {
1387 num_compute_rings = (adev->gfx.mec.num_pipe_per_mec *1447 /* recalculate compute rings to use based on hardware configuration */
1388 adev->gfx.mec.num_queue_per_pipe) / 2;1448 num_compute_rings = (adev->gfx.mec.num_pipe_per_mec *
1389 adev->gfx.num_compute_rings = min(adev->gfx.num_compute_rings,1449 adev->gfx.mec.num_queue_per_pipe) / 2;
1390 num_compute_rings);1450 adev->gfx.num_compute_rings = min(adev->gfx.num_compute_rings,
1451 num_compute_rings);
1452 }
1391 1453
1392 /* EOP Event */1454 /* EOP Event */
1393 r = amdgpu_irq_add_id(adev, SOC21_IH_CLIENTID_GRBM_CP,1455 r = amdgpu_irq_add_id(adev, SOC21_IH_CLIENTID_GRBM_CP,
@@ -1433,37 +1495,41 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 39 unmodified lines
1433 return r;1495 return r;
1434 }1496 }
1435 1497
1436 /* set up the gfx ring */1498 if (adev->gfx.num_gfx_rings) {
1437 for (i = 0; i < adev->gfx.me.num_me; i++) {1499 /* set up the gfx ring */
1438 for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) {1500 for (i = 0; i < adev->gfx.me.num_me; i++) {
1439 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {1501 for (j = 0; j < num_queue_per_pipe; j++) {
1440 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))1502 for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) {
1441 continue;1503 if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j))
1442 1504 continue;
1443 r = gfx_v12_0_gfx_ring_init(adev, ring_id,1505
1444 i, k, j);1506 r = gfx_v12_0_gfx_ring_init(adev, ring_id,
1445 if (r)1507 i, k, j);
1446 return r;1508 if (r)
1447 ring_id++;1509 return r;
1510 ring_id++;
1511 }
1448 }1512 }
1449 }1513 }
1450 }1514 }
1451 1515
1452 ring_id = 0;1516 if (adev->gfx.num_compute_rings) {
1453 /* set up the compute queues - allocate horizontally across pipes */1517 ring_id = 0;
1454 for (i = 0; i < adev->gfx.mec.num_mec; ++i) {1518 /* set up the compute queues - allocate horizontally across pipes */
1455 for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {1519 for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1456 for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {1520 for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1457 if (!amdgpu_gfx_is_mec_queue_enabled(adev,1521 for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) {
1458 0, i, k, j))1522 if (!amdgpu_gfx_is_mec_queue_enabled(adev,
1459 continue;1523 0, i, k, j))
1524 continue;
1460 1525
1461 r = gfx_v12_0_compute_ring_init(adev, ring_id,1526 r = gfx_v12_0_compute_ring_init(adev, ring_id,
1462 i, k, j);1527 i, k, j);
1463 if (r)1528 if (r)
1464 return r;1529 return r;
1465 1530
1466 ring_id++;1531 ring_id++;
1532 }
1467 }1533 }
1468 }1534 }
1469 }1535 }
@@ -2948,6 +3014,8 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 1478 unmodified lines
2948#ifdef __BIG_ENDIAN3014#ifdef __BIG_ENDIAN
2949 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1);3015 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1);
2950#endif3016#endif
3017 if (prop->tmz_queue)
3018 tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, TMZ_MATCH, 1);
2951 mqd->cp_gfx_hqd_cntl = tmp;3019 mqd->cp_gfx_hqd_cntl = tmp;
2952 3020
2953 /* set up cp_doorbell_control */3021 /* set up cp_doorbell_control */
@@ -2968,6 +3036,14 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 14 unmodified lines
2968 /* active the queue */3036 /* active the queue */
2969 mqd->cp_gfx_hqd_active = 1;3037 mqd->cp_gfx_hqd_active = 1;
2970 3038
3039 /* set gfx UQ items */
3040 mqd->shadow_base_lo = lower_32_bits(prop->shadow_addr);
3041 mqd->shadow_base_hi = upper_32_bits(prop->shadow_addr);
3042 mqd->fw_work_area_base_lo = lower_32_bits(prop->csa_addr);
3043 mqd->fw_work_area_base_hi = upper_32_bits(prop->csa_addr);
3044 mqd->fence_address_lo = lower_32_bits(prop->fence_address);
3045 mqd->fence_address_hi = upper_32_bits(prop->fence_address);
3046
2971 return 0;3047 return 0;
2972}3048}
2973 3049
@@ -3091,6 +3167,8 @@ static int gfx_v12_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 117 unmodified lines
3091 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TUNNEL_DISPATCH, 0);3167 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TUNNEL_DISPATCH, 0);
3092 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);3168 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1);
3093 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);3169 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1);
3170 if (prop->tmz_queue)
3171 tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TMZ, 1);
3094 mqd->cp_hqd_pq_control = tmp;3172 mqd->cp_hqd_pq_control = tmp;
3095 3173
3096 /* set the wb address whether it's enabled or not */3174 /* set the wb address whether it's enabled or not */
@@ -3142,6 +3220,10 @@ static int gfx_v12_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 45 unmodified lines
3142 3220
3143 mqd->cp_hqd_active = prop->hqd_active;3221 mqd->cp_hqd_active = prop->hqd_active;
3144 3222
3223 /* set UQ fenceaddress */
3224 mqd->fence_address_lo = lower_32_bits(prop->fence_address);
3225 mqd->fence_address_hi = upper_32_bits(prop->fence_address);
3226
3145 return 0;3227 return 0;
3146}3228}
3147 3229
@@ -3600,6 +3682,49 @@ static int gfx_v12_0_hw_init(struct amdgpu_ip_block *ip_block) · 452 unmodified lines
3600 return r;3682 return r;
3601}3683}
3602 3684
3685static int gfx_v12_0_set_userq_eop_interrupts(struct amdgpu_device *adev,
3686 bool enable)
3687{
3688 unsigned int irq_type;
3689 int m, p, r;
3690
3691 if (adev->userq_funcs[AMDGPU_HW_IP_GFX]) {
3692 for (m = 0; m < adev->gfx.me.num_me; m++) {
3693 for (p = 0; p < adev->gfx.me.num_pipe_per_me; p++) {
3694 irq_type = AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP + p;
3695 if (enable)
3696 r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
3697 irq_type);
3698 else
3699 r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
3700 irq_type);
3701 if (r)
3702 return r;
3703 }
3704 }
3705 }
3706
3707 if (adev->userq_funcs[AMDGPU_HW_IP_COMPUTE]) {
3708 for (m = 0; m < adev->gfx.mec.num_mec; ++m) {
3709 for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) {
3710 irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
3711 + (m * adev->gfx.mec.num_pipe_per_mec)
3712 + p;
3713 if (enable)
3714 r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
3715 irq_type);
3716 else
3717 r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
3718 irq_type);
3719 if (r)
3720 return r;
3721 }
3722 }
3723 }
3724
3725 return 0;
3726}
3727
3603static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block)3728static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block)
3604{3729{
3605 struct amdgpu_device *adev = ip_block->adev;3730 struct amdgpu_device *adev = ip_block->adev;
@@ -3610,6 +3735,7 @@ static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block) · 4 unmodified lines
3610 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);3735 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
3611 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);3736 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
3612 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);3737 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
3738 gfx_v12_0_set_userq_eop_interrupts(adev, false);
3613 3739
3614 if (!adev->no_hw_access) {3740 if (!adev->no_hw_access) {
3615 if (amdgpu_async_gfx_ring) {3741 if (amdgpu_async_gfx_ring) {
@@ -3698,11 +3824,33 @@ static int gfx_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 82 unmodified lines
3698{3824{
3699 struct amdgpu_device *adev = ip_block->adev;3825 struct amdgpu_device *adev = ip_block->adev;
3700 3826
3827 switch (amdgpu_user_queue) {
3828 case -1:
3829 case 0:
3830 default:
3831 adev->gfx.disable_kq = false;
3832 adev->gfx.disable_uq = true;
3833 break;
3834 case 1:
3835 adev->gfx.disable_kq = false;
3836 adev->gfx.disable_uq = false;
3837 break;
3838 case 2:
3839 adev->gfx.disable_kq = true;
3840 adev->gfx.disable_uq = false;
3841 break;
3842 }
3843
3701 adev->gfx.funcs = &gfx_v12_0_gfx_funcs;3844 adev->gfx.funcs = &gfx_v12_0_gfx_funcs;
3702 3845
3703 adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS;3846 if (adev->gfx.disable_kq) {
3704 adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),3847 adev->gfx.num_gfx_rings = 0;
3705 AMDGPU_MAX_COMPUTE_RINGS);3848 adev->gfx.num_compute_rings = 0;
3849 } else {
3850 adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS;
3851 adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
3852 AMDGPU_MAX_COMPUTE_RINGS);
3853 }
3706 3854
3707 gfx_v12_0_set_kiq_pm4_funcs(adev);3855 gfx_v12_0_set_kiq_pm4_funcs(adev);
3708 gfx_v12_0_set_ring_funcs(adev);3856 gfx_v12_0_set_ring_funcs(adev);
@@ -3733,6 +3881,10 @@ static int gfx_v12_0_late_init(struct amdgpu_ip_block *ip_block) · 24 unmodified lines
3733 if (r)3881 if (r)
3734 return r;3882 return r;
3735 3883
3884 r = gfx_v12_0_set_userq_eop_interrupts(adev, true);
3885 if (r)
3886 return r;
3887
3736 return 0;3888 return 0;
3737}3889}
3738 3890
@@ -4172,45 +4324,17 @@ static u64 gfx_v12_0_ring_get_wptr_gfx(struct amdgpu_ring *ring) · 433 unmodified lines
4172static void gfx_v12_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)4324static void gfx_v12_0_ring_set_wptr_gfx(struct amdgpu_ring *ring)
4173{4325{
4174 struct amdgpu_device *adev = ring->adev;4326 struct amdgpu_device *adev = ring->adev;
4175 uint32_t *wptr_saved;
4176 uint32_t *is_queue_unmap;
4177 uint64_t aggregated_db_index;
4178 uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_GFX].mqd_size;
4179 uint64_t wptr_tmp;
4180
4181 if (ring->is_mes_queue) {
4182 wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);
4183 is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +
4184 sizeof(uint32_t));
4185 aggregated_db_index =
4186 amdgpu_mes_get_aggregated_doorbell_index(adev,
4187 ring->hw_prio);
4188
4189 wptr_tmp = ring->wptr & ring->buf_mask;
4190 atomic64_set((atomic64_t *)ring->wptr_cpu_addr, wptr_tmp);
4191 *wptr_saved = wptr_tmp;
4192 /* assume doorbell always being used by mes mapped queue */
4193 if (*is_queue_unmap) {
4194 WDOORBELL64(aggregated_db_index, wptr_tmp);
4195 WDOORBELL64(ring->doorbell_index, wptr_tmp);
4196 } else {
4197 WDOORBELL64(ring->doorbell_index, wptr_tmp);
4198 4327
4199 if (*is_queue_unmap)4328 if (ring->use_doorbell) {
4200 WDOORBELL64(aggregated_db_index, wptr_tmp);4329 /* XXX check if swapping is necessary on BE */
4201 }4330 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4331 ring->wptr);
4332 WDOORBELL64(ring->doorbell_index, ring->wptr);
4202 } else {4333 } else {
4203 if (ring->use_doorbell) {4334 WREG32_SOC15(GC, 0, regCP_RB0_WPTR,
4204 /* XXX check if swapping is necessary on BE */4335 lower_32_bits(ring->wptr));
4205 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,4336 WREG32_SOC15(GC, 0, regCP_RB0_WPTR_HI,
4206 ring->wptr);4337 upper_32_bits(ring->wptr));
4207 WDOORBELL64(ring->doorbell_index, ring->wptr);
4208 } else {
4209 WREG32_SOC15(GC, 0, regCP_RB0_WPTR,
4210 lower_32_bits(ring->wptr));
4211 WREG32_SOC15(GC, 0, regCP_RB0_WPTR_HI,
4212 upper_32_bits(ring->wptr));
4213 }
4214 }4338 }
4215}4339}
4216 4340
@@ -4235,42 +4359,14 @@ static u64 gfx_v12_0_ring_get_wptr_compute(struct amdgpu_ring *ring) · 18 unmodified lines
4235static void gfx_v12_0_ring_set_wptr_compute(struct amdgpu_ring *ring)4359static void gfx_v12_0_ring_set_wptr_compute(struct amdgpu_ring *ring)
4236{4360{
4237 struct amdgpu_device *adev = ring->adev;4361 struct amdgpu_device *adev = ring->adev;
4238 uint32_t *wptr_saved;
4239 uint32_t *is_queue_unmap;
4240 uint64_t aggregated_db_index;
4241 uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_COMPUTE].mqd_size;
4242 uint64_t wptr_tmp;
4243
4244 if (ring->is_mes_queue) {
4245 wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);
4246 is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +
4247 sizeof(uint32_t));
4248 aggregated_db_index =
4249 amdgpu_mes_get_aggregated_doorbell_index(adev,
4250 ring->hw_prio);
4251
4252 wptr_tmp = ring->wptr & ring->buf_mask;
4253 atomic64_set((atomic64_t *)ring->wptr_cpu_addr, wptr_tmp);
4254 *wptr_saved = wptr_tmp;
4255 /* assume doorbell always used by mes mapped queue */
4256 if (*is_queue_unmap) {
4257 WDOORBELL64(aggregated_db_index, wptr_tmp);
4258 WDOORBELL64(ring->doorbell_index, wptr_tmp);
4259 } else {
4260 WDOORBELL64(ring->doorbell_index, wptr_tmp);
4261 4362
4262 if (*is_queue_unmap)4363 /* XXX check if swapping is necessary on BE */
4263 WDOORBELL64(aggregated_db_index, wptr_tmp);4364 if (ring->use_doorbell) {
4264 }4365 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4366 ring->wptr);
4367 WDOORBELL64(ring->doorbell_index, ring->wptr);
4265 } else {4368 } else {
4266 /* XXX check if swapping is necessary on BE */4369 BUG(); /* only DOORBELL method supported on gfx12 now */
4267 if (ring->use_doorbell) {
4268 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
4269 ring->wptr);
4270 WDOORBELL64(ring->doorbell_index, ring->wptr);
4271 } else {
4272 BUG(); /* only DOORBELL method supported on gfx12 now */
4273 }
4274 }4370 }
4275}4371}
4276 4372
@@ -4317,10 +4413,6 @@ static void gfx_v12_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, · 40 unmodified lines
4317 4413
4318 control |= ib->length_dw | (vmid << 24);4414 control |= ib->length_dw | (vmid << 24);
4319 4415
4320 if (ring->is_mes_queue)
4321 /* inherit vmid from mqd */
4322 control |= 0x400000;
4323
4324 amdgpu_ring_write(ring, header);4416 amdgpu_ring_write(ring, header);
4325 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */4417 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
4326 amdgpu_ring_write(ring,4418 amdgpu_ring_write(ring,
@@ -4340,10 +4432,6 @@ static void gfx_v12_0_ring_emit_ib_compute(struct amdgpu_ring *ring, · 13 unmodified lines
4340 unsigned vmid = AMDGPU_JOB_GET_VMID(job);4432 unsigned vmid = AMDGPU_JOB_GET_VMID(job);
4341 u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);4433 u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24);
4342 4434
4343 if (ring->is_mes_queue)
4344 /* inherit vmid from mqd */
4345 control |= 0x40000000;
4346
4347 amdgpu_ring_write(ring, PACKET3(PACKET3_INDIRECT_BUFFER, 2));4435 amdgpu_ring_write(ring, PACKET3(PACKET3_INDIRECT_BUFFER, 2));
4348 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */4436 BUG_ON(ib->gpu_addr & 0x3); /* Dword align */
4349 amdgpu_ring_write(ring,4437 amdgpu_ring_write(ring,
@@ -4383,8 +4471,7 @@ static void gfx_v12_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, · 33 unmodified lines
4383 amdgpu_ring_write(ring, upper_32_bits(addr));4471 amdgpu_ring_write(ring, upper_32_bits(addr));
4384 amdgpu_ring_write(ring, lower_32_bits(seq));4472 amdgpu_ring_write(ring, lower_32_bits(seq));
4385 amdgpu_ring_write(ring, upper_32_bits(seq));4473 amdgpu_ring_write(ring, upper_32_bits(seq));
4386 amdgpu_ring_write(ring, ring->is_mes_queue ?4474 amdgpu_ring_write(ring, 0);
4387 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0);
4388}4475}
4389 4476
4390static void gfx_v12_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)4477static void gfx_v12_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
@@ -4412,10 +4499,7 @@ static void gfx_v12_0_ring_invalidate_tlbs(struct amdgpu_ring *ring, · 21 unmodified lines
4412static void gfx_v12_0_ring_emit_vm_flush(struct amdgpu_ring *ring,4499static void gfx_v12_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
4413 unsigned vmid, uint64_t pd_addr)4500 unsigned vmid, uint64_t pd_addr)
4414{4501{
4415 if (ring->is_mes_queue)4502 amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
4416 gfx_v12_0_ring_invalidate_tlbs(ring, 0, 0, false, 0);
4417 else
4418 amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
4419 4503
4420 /* compute doesn't have PFP */4504 /* compute doesn't have PFP */
4421 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {4505 if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {
@@ -4749,25 +4833,23 @@ static int gfx_v12_0_eop_irq(struct amdgpu_device *adev, · 327 unmodified lines
4749 struct amdgpu_irq_src *source,4833 struct amdgpu_irq_src *source,
4750 struct amdgpu_iv_entry *entry)4834 struct amdgpu_iv_entry *entry)
4751{4835{
4752 int i;4836 u32 doorbell_offset = entry->src_data[0];
4753 u8 me_id, pipe_id, queue_id;4837 u8 me_id, pipe_id, queue_id;
4754 struct amdgpu_ring *ring;4838 struct amdgpu_ring *ring;
4755 uint32_t mes_queue_id = entry->src_data[0];4839 int i;
4756 4840
4757 DRM_DEBUG("IH: CP EOP\n");4841 DRM_DEBUG("IH: CP EOP\n");
4758 4842
4759 if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) {4843 if (adev->enable_mes && doorbell_offset) {
4760 struct amdgpu_mes_queue *queue;4844 struct amdgpu_userq_fence_driver *fence_drv = NULL;
4845 struct xarray *xa = &adev->userq_xa;
4846 unsigned long flags;
4761 4847
4762 mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK;4848 xa_lock_irqsave(xa, flags);
4763 4849 fence_drv = xa_load(xa, doorbell_offset);
4764 spin_lock(&adev->mes.queue_id_lock);4850 if (fence_drv)
4765 queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id);4851 amdgpu_userq_fence_driver_process(fence_drv);
4766 if (queue) {4852 xa_unlock_irqrestore(xa, flags);
4767 DRM_DEBUG("process mes queue id = %d\n", mes_queue_id);
4768 amdgpu_fence_process(queue->ring);
4769 }
4770 spin_unlock(&adev->mes.queue_id_lock);
4771 } else {4853 } else {
4772 me_id = (entry->ring_id & 0x0c) >> 2;4854 me_id = (entry->ring_id & 0x0c) >> 2;
4773 pipe_id = (entry->ring_id & 0x03) >> 0;4855 pipe_id = (entry->ring_id & 0x03) >> 0;
@@ -4934,27 +5016,29 @@ static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev, · 160 unmodified lines
4934 pipe_id = (entry->ring_id & 0x03) >> 0;5016 pipe_id = (entry->ring_id & 0x03) >> 0;
4935 queue_id = (entry->ring_id & 0x70) >> 4;5017 queue_id = (entry->ring_id & 0x70) >> 4;
4936 5018
4937 switch (me_id) {5019 if (!adev->gfx.disable_kq) {
4938 case 0:5020 switch (me_id) {
4939 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {5021 case 0:
4940 ring = &adev->gfx.gfx_ring[i];5022 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4941 if (ring->me == me_id && ring->pipe == pipe_id &&5023 ring = &adev->gfx.gfx_ring[i];
4942 ring->queue == queue_id)5024 if (ring->me == me_id && ring->pipe == pipe_id &&
4943 drm_sched_fault(&ring->sched);5025 ring->queue == queue_id)
4944 }5026 drm_sched_fault(&ring->sched);
4945 break;5027 }
4946 case 1:5028 break;
4947 case 2:5029 case 1:
4948 for (i = 0; i < adev->gfx.num_compute_rings; i++) {5030 case 2:
4949 ring = &adev->gfx.compute_ring[i];5031 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
4950 if (ring->me == me_id && ring->pipe == pipe_id &&5032 ring = &adev->gfx.compute_ring[i];
4951 ring->queue == queue_id)5033 if (ring->me == me_id && ring->pipe == pipe_id &&
4952 drm_sched_fault(&ring->sched);5034 ring->queue == queue_id)
5035 drm_sched_fault(&ring->sched);
5036 }
5037 break;
5038 default:
5039 BUG();
5040 break;
4953 }5041 }
4954 break;
4955 default:
4956 BUG();
4957 break;
4958 }5042 }
4959}5043}
4960 5044
@@ -5160,6 +5244,69 @@ static void gfx_v12_ip_dump(struct amdgpu_ip_block *ip_block) · 199 unmodified lines
5160 amdgpu_gfx_off_ctrl(adev, true);5244 amdgpu_gfx_off_ctrl(adev, true);
5161}5245}
5162 5246
5247static bool gfx_v12_pipe_reset_support(struct amdgpu_device *adev)
5248{
5249 /* Disable the pipe reset until the CPFW fully support it.*/
5250 dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
5251 return false;
5252}
5253
5254static int gfx_v12_reset_gfx_pipe(struct amdgpu_ring *ring)
5255{
5256 struct amdgpu_device *adev = ring->adev;
5257 uint32_t reset_pipe = 0, clean_pipe = 0;
5258 int r;
5259
5260 if (!gfx_v12_pipe_reset_support(adev))
5261 return -EOPNOTSUPP;
5262
5263 gfx_v12_0_set_safe_mode(adev, 0);
5264 mutex_lock(&adev->srbm_mutex);
5265 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5266
5267 switch (ring->pipe) {
5268 case 0:
5269 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5270 PFP_PIPE0_RESET, 1);
5271 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5272 ME_PIPE0_RESET, 1);
5273 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5274 PFP_PIPE0_RESET, 0);
5275 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5276 ME_PIPE0_RESET, 0);
5277 break;
5278 case 1:
5279 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5280 PFP_PIPE1_RESET, 1);
5281 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5282 ME_PIPE1_RESET, 1);
5283 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5284 PFP_PIPE1_RESET, 0);
5285 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5286 ME_PIPE1_RESET, 0);
5287 break;
5288 default:
5289 break;
5290 }
5291
5292 WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
5293 WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
5294
5295 r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
5296 RS64_FW_UC_START_ADDR_LO;
5297 soc24_grbm_select(adev, 0, 0, 0, 0);
5298 mutex_unlock(&adev->srbm_mutex);
5299 gfx_v12_0_unset_safe_mode(adev, 0);
5300
5301 dev_info(adev->dev, "The ring %s pipe reset: %s\n", ring->name,
5302 r == 0 ? "successfully" : "failed");
5303 /* Sometimes the ME start pc counter can't cache correctly, so the
5304 * PC check only as a reference and pipe reset result rely on the
5305 * later ring test.
5306 */
5307 return 0;
5308}
5309
5163static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)5310static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid)
5164{5311{
5165 struct amdgpu_device *adev = ring->adev;5312 struct amdgpu_device *adev = ring->adev;
@@ -5170,8 +5317,10 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines
5170 5317
5171 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false);5318 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false);
5172 if (r) {5319 if (r) {
5173 dev_err(adev->dev, "reset via MES failed %d\n", r);5320 dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r);
5174 return r;5321 r = gfx_v12_reset_gfx_pipe(ring);
5322 if (r)
5323 return r;
5175 }5324 }
5176 5325
5177 r = gfx_v12_0_kgq_init_queue(ring, true);5326 r = gfx_v12_0_kgq_init_queue(ring, true);
@@ -5189,6 +5338,89 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 11 unmodified lines
5189 return amdgpu_ring_test_ring(ring);5338 return amdgpu_ring_test_ring(ring);
5190}5339}
5191 5340
5341static int gfx_v12_0_reset_compute_pipe(struct amdgpu_ring *ring)
5342{
5343 struct amdgpu_device *adev = ring->adev;
5344 uint32_t reset_pipe = 0, clean_pipe = 0;
5345 int r = 0;
5346
5347 if (!gfx_v12_pipe_reset_support(adev))
5348 return -EOPNOTSUPP;
5349
5350 gfx_v12_0_set_safe_mode(adev, 0);
5351 mutex_lock(&adev->srbm_mutex);
5352 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5353
5354 reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
5355 clean_pipe = reset_pipe;
5356
5357 if (adev->gfx.rs64_enable) {
5358 switch (ring->pipe) {
5359 case 0:
5360 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5361 MEC_PIPE0_RESET, 1);
5362 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5363 MEC_PIPE0_RESET, 0);
5364 break;
5365 case 1:
5366 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5367 MEC_PIPE1_RESET, 1);
5368 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5369 MEC_PIPE1_RESET, 0);
5370 break;
5371 case 2:
5372 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5373 MEC_PIPE2_RESET, 1);
5374 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5375 MEC_PIPE2_RESET, 0);
5376 break;
5377 case 3:
5378 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5379 MEC_PIPE3_RESET, 1);
5380 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5381 MEC_PIPE3_RESET, 0);
5382 break;
5383 default:
5384 break;
5385 }
5386 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe);
5387 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe);
5388 r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
5389 RS64_FW_UC_START_ADDR_LO;
5390 } else {
5391 switch (ring->pipe) {
5392 case 0:
5393 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
5394 MEC_ME1_PIPE0_RESET, 1);
5395 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
5396 MEC_ME1_PIPE0_RESET, 0);
5397 break;
5398 case 1:
5399 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
5400 MEC_ME1_PIPE1_RESET, 1);
5401 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
5402 MEC_ME1_PIPE1_RESET, 0);
5403 break;
5404 default:
5405 break;
5406 }
5407 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe);
5408 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe);
5409 /* Doesn't find the F32 MEC instruction pointer register, and suppose
5410 * the driver won't run into the F32 mode.
5411 */
5412 }
5413
5414 soc24_grbm_select(adev, 0, 0, 0, 0);
5415 mutex_unlock(&adev->srbm_mutex);
5416 gfx_v12_0_unset_safe_mode(adev, 0);
5417
5418 dev_info(adev->dev, "The ring %s pipe resets: %s\n", ring->name,
5419 r == 0 ? "successfully" : "failed");
5420 /* Need the ring test to verify the pipe reset result.*/
5421 return 0;
5422}
5423
5192static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)5424static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid)
5193{5425{
5194 struct amdgpu_device *adev = ring->adev;5426 struct amdgpu_device *adev = ring->adev;
@@ -5199,8 +5431,10 @@ static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines
5199 5431
5200 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true);5432 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true);
5201 if (r) {5433 if (r) {
5202 dev_err(adev->dev, "reset via MMIO failed %d\n", r);5434 dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r);
5203 return r;5435 r = gfx_v12_0_reset_compute_pipe(ring);
5436 if (r)
5437 return r;
5204 }5438 }
5205 5439
5206 r = gfx_v12_0_kcq_init_queue(ring, true);5440 r = gfx_v12_0_kcq_init_queue(ring, true);
drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c+15−31
@@ -53,6 +53,9 @@ · 52 unmodified lines
53#define VERDE_GB_ADDR_CONFIG_GOLDEN 0x1201000253#define VERDE_GB_ADDR_CONFIG_GOLDEN 0x12010002
54#define HAINAN_GB_ADDR_CONFIG_GOLDEN 0x0201000154#define HAINAN_GB_ADDR_CONFIG_GOLDEN 0x02010001
55 55
56#define GFX6_NUM_GFX_RINGS 1
57#define GFX6_NUM_COMPUTE_RINGS 2
58
56static void gfx_v6_0_set_ring_funcs(struct amdgpu_device *adev);59static void gfx_v6_0_set_ring_funcs(struct amdgpu_device *adev);
57static void gfx_v6_0_set_irq_funcs(struct amdgpu_device *adev);60static void gfx_v6_0_set_irq_funcs(struct amdgpu_device *adev);
58static void gfx_v6_0_get_cu_info(struct amdgpu_device *adev);61static void gfx_v6_0_get_cu_info(struct amdgpu_device *adev);
@@ -1732,10 +1735,14 @@ static void gfx_v6_0_constants_init(struct amdgpu_device *adev) · 1673 unmodified lines
1732 gfx_v6_0_get_cu_info(adev);1735 gfx_v6_0_get_cu_info(adev);
1733 gfx_v6_0_config_init(adev);1736 gfx_v6_0_config_init(adev);
1734 1737
1735 WREG32(mmCP_QUEUE_THRESHOLDS, ((0x16 << CP_QUEUE_THRESHOLDS__ROQ_IB1_START__SHIFT) |1738 WREG32(mmCP_QUEUE_THRESHOLDS,
1736 (0x2b << CP_QUEUE_THRESHOLDS__ROQ_IB2_START__SHIFT)));1739 ((0x16 << CP_QUEUE_THRESHOLDS__ROQ_IB1_START__SHIFT) |
1737 WREG32(mmCP_MEQ_THRESHOLDS, (0x30 << CP_MEQ_THRESHOLDS__MEQ1_START__SHIFT) |1740 (0x2b << CP_QUEUE_THRESHOLDS__ROQ_IB2_START__SHIFT)));
1738 (0x60 << CP_MEQ_THRESHOLDS__MEQ2_START__SHIFT));1741
1742 /* set HW defaults for 3D engine */
1743 WREG32(mmCP_MEQ_THRESHOLDS,
1744 (0x30 << CP_MEQ_THRESHOLDS__MEQ1_START__SHIFT) |
1745 (0x60 << CP_MEQ_THRESHOLDS__MEQ2_START__SHIFT));
1739 1746
1740 sx_debug_1 = RREG32(mmSX_DEBUG_1);1747 sx_debug_1 = RREG32(mmSX_DEBUG_1);
1741 WREG32(mmSX_DEBUG_1, sx_debug_1);1748 WREG32(mmSX_DEBUG_1, sx_debug_1);
@@ -2851,44 +2858,21 @@ static u32 gfx_v6_0_get_csb_size(struct amdgpu_device *adev) · 1109 unmodified lines
2851static void gfx_v6_0_get_csb_buffer(struct amdgpu_device *adev,2858static void gfx_v6_0_get_csb_buffer(struct amdgpu_device *adev,
2852 volatile u32 *buffer)2859 volatile u32 *buffer)
2853{2860{
2854 u32 count = 0, i;2861 u32 count = 0;
2855 const struct cs_section_def *sect = NULL;
2856 const struct cs_extent_def *ext = NULL;
2857 2862
2858 if (adev->gfx.rlc.cs_data == NULL)2863 if (adev->gfx.rlc.cs_data == NULL)
2859 return;2864 return;
2860 if (buffer == NULL)2865 if (buffer == NULL)
2861 return;2866 return;
2862 2867
2863 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));2868 count = amdgpu_gfx_csb_preamble_start(buffer);
2864 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);2869 count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
2865 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
2866 buffer[count++] = cpu_to_le32(0x80000000);
2867 buffer[count++] = cpu_to_le32(0x80000000);
2868
2869 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
2870 for (ext = sect->section; ext->extent != NULL; ++ext) {
2871 if (sect->id == SECT_CONTEXT) {
2872 buffer[count++] =
2873 cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
2874 buffer[count++] = cpu_to_le32(ext->reg_index - 0xa000);
2875 for (i = 0; i < ext->reg_count; i++)
2876 buffer[count++] = cpu_to_le32(ext->extent[i]);
2877 } else {
2878 return;
2879 }
2880 }
2881 }
2882 2870
2883 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));2871 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1));
2884 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);2872 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);
2885 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);2873 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);
2886 2874
2887 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));2875 amdgpu_gfx_csb_preamble_end(buffer, count);
2888 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
2889
2890 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
2891 buffer[count++] = cpu_to_le32(0);
2892}2876}
2893 2877
2894static void gfx_v6_0_init_pg(struct amdgpu_device *adev)2878static void gfx_v6_0_init_pg(struct amdgpu_device *adev)
drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c+9−51
@@ -55,6 +55,9 @@ · 54 unmodified lines
55#define GFX7_NUM_GFX_RINGS 155#define GFX7_NUM_GFX_RINGS 1
56#define GFX7_MEC_HPD_SIZE 204856#define GFX7_MEC_HPD_SIZE 2048
57 57
58#define BONAIRE_GB_ADDR_CONFIG_GOLDEN 0x12010001
59#define HAWAII_GB_ADDR_CONFIG_GOLDEN 0x12011003
60
58static void gfx_v7_0_set_ring_funcs(struct amdgpu_device *adev);61static void gfx_v7_0_set_ring_funcs(struct amdgpu_device *adev);
59static void gfx_v7_0_set_irq_funcs(struct amdgpu_device *adev);62static void gfx_v7_0_set_irq_funcs(struct amdgpu_device *adev);
60static void gfx_v7_0_set_gds_init(struct amdgpu_device *adev);63static void gfx_v7_0_set_gds_init(struct amdgpu_device *adev);
@@ -3882,67 +3885,22 @@ static u32 gfx_v7_0_get_csb_size(struct amdgpu_device *adev) · 3821 unmodified lines
3882static void gfx_v7_0_get_csb_buffer(struct amdgpu_device *adev,3885static void gfx_v7_0_get_csb_buffer(struct amdgpu_device *adev,
3883 volatile u32 *buffer)3886 volatile u32 *buffer)
3884{3887{
3885 u32 count = 0, i;3888 u32 count = 0;
3886 const struct cs_section_def *sect = NULL;
3887 const struct cs_extent_def *ext = NULL;
3888 3889
3889 if (adev->gfx.rlc.cs_data == NULL)3890 if (adev->gfx.rlc.cs_data == NULL)
3890 return;3891 return;
3891 if (buffer == NULL)3892 if (buffer == NULL)
3892 return;3893 return;
3893 3894
3894 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));3895 count = amdgpu_gfx_csb_preamble_start(buffer);
3895 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);3896 count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
3896
3897 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
3898 buffer[count++] = cpu_to_le32(0x80000000);
3899 buffer[count++] = cpu_to_le32(0x80000000);
3900
3901 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
3902 for (ext = sect->section; ext->extent != NULL; ++ext) {
3903 if (sect->id == SECT_CONTEXT) {
3904 buffer[count++] =
3905 cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
3906 buffer[count++] = cpu_to_le32(ext->reg_index - PACKET3_SET_CONTEXT_REG_START);
3907 for (i = 0; i < ext->reg_count; i++)
3908 buffer[count++] = cpu_to_le32(ext->extent[i]);
3909 } else {
3910 return;
3911 }
3912 }
3913 }
3914 3897
3915 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2));3898 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2));
3916 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);3899 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);
3917 switch (adev->asic_type) {3900 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);
3918 case CHIP_BONAIRE:3901 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1);
3919 buffer[count++] = cpu_to_le32(0x16000012);
3920 buffer[count++] = cpu_to_le32(0x00000000);
3921 break;
3922 case CHIP_KAVERI:
3923 buffer[count++] = cpu_to_le32(0x00000000); /* XXX */
3924 buffer[count++] = cpu_to_le32(0x00000000);
3925 break;
3926 case CHIP_KABINI:
3927 case CHIP_MULLINS:
3928 buffer[count++] = cpu_to_le32(0x00000000); /* XXX */
3929 buffer[count++] = cpu_to_le32(0x00000000);
3930 break;
3931 case CHIP_HAWAII:
3932 buffer[count++] = cpu_to_le32(0x3a00161a);
3933 buffer[count++] = cpu_to_le32(0x0000002e);
3934 break;
3935 default:
3936 buffer[count++] = cpu_to_le32(0x00000000);
3937 buffer[count++] = cpu_to_le32(0x00000000);
3938 break;
3939 }
3940
3941 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
3942 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
3943 3902
3944 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));3903 amdgpu_gfx_csb_preamble_end(buffer, count);
3945 buffer[count++] = cpu_to_le32(0);
3946}3904}
3947 3905
3948static void gfx_v7_0_init_pg(struct amdgpu_device *adev)3906static void gfx_v7_0_init_pg(struct amdgpu_device *adev)
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c+5−31
@@ -1223,48 +1223,22 @@ out: · 1222 unmodified lines
1223static void gfx_v8_0_get_csb_buffer(struct amdgpu_device *adev,1223static void gfx_v8_0_get_csb_buffer(struct amdgpu_device *adev,
1224 volatile u32 *buffer)1224 volatile u32 *buffer)
1225{1225{
1226 u32 count = 0, i;1226 u32 count = 0;
1227 const struct cs_section_def *sect = NULL;
1228 const struct cs_extent_def *ext = NULL;
1229 1227
1230 if (adev->gfx.rlc.cs_data == NULL)1228 if (adev->gfx.rlc.cs_data == NULL)
1231 return;1229 return;
1232 if (buffer == NULL)1230 if (buffer == NULL)
1233 return;1231 return;
1234 1232
1235 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));1233 count = amdgpu_gfx_csb_preamble_start(buffer);
1236 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);1234 count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
1237
1238 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
1239 buffer[count++] = cpu_to_le32(0x80000000);
1240 buffer[count++] = cpu_to_le32(0x80000000);
1241
1242 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
1243 for (ext = sect->section; ext->extent != NULL; ++ext) {
1244 if (sect->id == SECT_CONTEXT) {
1245 buffer[count++] =
1246 cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
1247 buffer[count++] = cpu_to_le32(ext->reg_index -
1248 PACKET3_SET_CONTEXT_REG_START);
1249 for (i = 0; i < ext->reg_count; i++)
1250 buffer[count++] = cpu_to_le32(ext->extent[i]);
1251 } else {
1252 return;
1253 }
1254 }
1255 }
1256 1235
1257 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2));1236 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2));
1258 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG -1237 buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START);
1259 PACKET3_SET_CONTEXT_REG_START);
1260 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);1238 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config);
1261 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1);1239 buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1);
1262 1240
1263 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));1241 amdgpu_gfx_csb_preamble_end(buffer, count);
1264 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
1265
1266 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
1267 buffer[count++] = cpu_to_le32(0);
1268}1242}
1269 1243
1270static int gfx_v8_0_cp_jump_table_num(struct amdgpu_device *adev)1244static int gfx_v8_0_cp_jump_table_num(struct amdgpu_device *adev)
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c+73−98
@@ -225,17 +225,36 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_9[] = { · 224 unmodified lines
225 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_SMU_SAFE_MODE),225 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_SMU_SAFE_MODE),
226 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_INT_STAT),226 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_INT_STAT),
227 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_GENERAL_6),227 SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_GENERAL_6),
228 /* cp header registers */
229 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
230 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
231 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME2_HEADER_DUMP),
232 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
233 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
234 /* SE status registers */228 /* SE status registers */
235 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0),229 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0),
236 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE1),230 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE1),
237 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE2),231 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE2),
238 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE3)232 SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE3),
233 /* packet headers */
234 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
235 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
236 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
237 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
238 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
239 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
240 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
241 SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP),
242 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
243 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
244 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
245 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
246 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
247 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
248 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
249 SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP),
250 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
251 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
252 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
253 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
254 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
255 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
256 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP),
257 SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP)
239};258};
240 259
241static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = {260static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = {
@@ -277,6 +296,14 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = { · 35 unmodified lines
277 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_LO),296 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_LO),
278 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_HI),297 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_HI),
279 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_GFX_STATUS),298 SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_GFX_STATUS),
299 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
300 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
301 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
302 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
303 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
304 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
305 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP),
306 SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP)
280};307};
281 308
282enum ta_ras_gfx_subblock {309enum ta_ras_gfx_subblock {
@@ -1624,42 +1651,16 @@ static u32 gfx_v9_0_get_csb_size(struct amdgpu_device *adev) · 1341 unmodified lines
1624static void gfx_v9_0_get_csb_buffer(struct amdgpu_device *adev,1651static void gfx_v9_0_get_csb_buffer(struct amdgpu_device *adev,
1625 volatile u32 *buffer)1652 volatile u32 *buffer)
1626{1653{
1627 u32 count = 0, i;1654 u32 count = 0;
1628 const struct cs_section_def *sect = NULL;
1629 const struct cs_extent_def *ext = NULL;
1630 1655
1631 if (adev->gfx.rlc.cs_data == NULL)1656 if (adev->gfx.rlc.cs_data == NULL)
1632 return;1657 return;
1633 if (buffer == NULL)1658 if (buffer == NULL)
1634 return;1659 return;
1635 1660
1636 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));1661 count = amdgpu_gfx_csb_preamble_start(buffer);
1637 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE);1662 count = amdgpu_gfx_csb_data_parser(adev, buffer, count);
1638 1663 amdgpu_gfx_csb_preamble_end(buffer, count);
1639 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1));
1640 buffer[count++] = cpu_to_le32(0x80000000);
1641 buffer[count++] = cpu_to_le32(0x80000000);
1642
1643 for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) {
1644 for (ext = sect->section; ext->extent != NULL; ++ext) {
1645 if (sect->id == SECT_CONTEXT) {
1646 buffer[count++] =
1647 cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count));
1648 buffer[count++] = cpu_to_le32(ext->reg_index -
1649 PACKET3_SET_CONTEXT_REG_START);
1650 for (i = 0; i < ext->reg_count; i++)
1651 buffer[count++] = cpu_to_le32(ext->extent[i]);
1652 } else {
1653 return;
1654 }
1655 }
1656 }
1657
1658 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0));
1659 buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE);
1660
1661 buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0));
1662 buffer[count++] = cpu_to_le32(0);
1663}1664}
1664 1665
1665static void gfx_v9_0_init_always_on_cu_mask(struct amdgpu_device *adev)1666static void gfx_v9_0_init_always_on_cu_mask(struct amdgpu_device *adev)
@@ -5441,16 +5442,8 @@ static void gfx_v9_0_ring_patch_ce_meta(struct amdgpu_ring *ring, · 3775 unmodified lines
5441 5442
5442 payload_size = sizeof(struct v9_ce_ib_state);5443 payload_size = sizeof(struct v9_ce_ib_state);
5443 5444
5444 if (ring->is_mes_queue) {5445 payload_offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5445 payload_offset = offsetof(struct amdgpu_mes_ctx_meta_data,5446 ce_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
5446 gfx[0].gfx_meta_data) +
5447 offsetof(struct v9_gfx_meta_data, ce_payload);
5448 ce_payload_cpu_addr =
5449 amdgpu_mes_ctx_get_offs_cpu_addr(ring, payload_offset);
5450 } else {
5451 payload_offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5452 ce_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
5453 }
5454 5447
5455 if (offset + (payload_size >> 2) <= ring->buf_mask + 1) {5448 if (offset + (payload_size >> 2) <= ring->buf_mask + 1) {
5456 memcpy((void *)&ring->ring[offset], ce_payload_cpu_addr, payload_size);5449 memcpy((void *)&ring->ring[offset], ce_payload_cpu_addr, payload_size);
@@ -5473,16 +5466,8 @@ static void gfx_v9_0_ring_patch_de_meta(struct amdgpu_ring *ring, · 16 unmodified lines
5473 5466
5474 payload_size = sizeof(struct v9_de_ib_state);5467 payload_size = sizeof(struct v9_de_ib_state);
5475 5468
5476 if (ring->is_mes_queue) {5469 payload_offset = offsetof(struct v9_gfx_meta_data, de_payload);
5477 payload_offset = offsetof(struct amdgpu_mes_ctx_meta_data,5470 de_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
5478 gfx[0].gfx_meta_data) +
5479 offsetof(struct v9_gfx_meta_data, de_payload);
5480 de_payload_cpu_addr =
5481 amdgpu_mes_ctx_get_offs_cpu_addr(ring, payload_offset);
5482 } else {
5483 payload_offset = offsetof(struct v9_gfx_meta_data, de_payload);
5484 de_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset;
5485 }
5486 5471
5487 ((struct v9_de_ib_state *)de_payload_cpu_addr)->ib_completion_status =5472 ((struct v9_de_ib_state *)de_payload_cpu_addr)->ib_completion_status =
5488 IB_COMPLETION_STATUS_PREEMPTED;5473 IB_COMPLETION_STATUS_PREEMPTED;
@@ -5672,19 +5657,9 @@ static void gfx_v9_0_ring_emit_ce_meta(struct amdgpu_ring *ring, bool resume) · 183 unmodified lines
5672 5657
5673 cnt = (sizeof(ce_payload) >> 2) + 4 - 2;5658 cnt = (sizeof(ce_payload) >> 2) + 4 - 2;
5674 5659
5675 if (ring->is_mes_queue) {5660 offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5676 offset = offsetof(struct amdgpu_mes_ctx_meta_data,5661 ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5677 gfx[0].gfx_meta_data) +5662 ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
5678 offsetof(struct v9_gfx_meta_data, ce_payload);
5679 ce_payload_gpu_addr =
5680 amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
5681 ce_payload_cpu_addr =
5682 amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
5683 } else {
5684 offset = offsetof(struct v9_gfx_meta_data, ce_payload);
5685 ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5686 ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
5687 }
5688 5663
5689 amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));5664 amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt));
5690 amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(2) |5665 amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(2) |
@@ -5770,28 +5745,13 @@ static void gfx_v9_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume, bo · 79 unmodified lines
5770 void *de_payload_cpu_addr;5745 void *de_payload_cpu_addr;
5771 int cnt;5746 int cnt;
5772 5747
5773 if (ring->is_mes_queue) {5748 offset = offsetof(struct v9_gfx_meta_data, de_payload);
5774 offset = offsetof(struct amdgpu_mes_ctx_meta_data,5749 de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5775 gfx[0].gfx_meta_data) +5750 de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
5776 offsetof(struct v9_gfx_meta_data, de_payload);
5777 de_payload_gpu_addr =
5778 amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
5779 de_payload_cpu_addr =
5780 amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
5781
5782 offset = offsetof(struct amdgpu_mes_ctx_meta_data,
5783 gfx[0].gds_backup) +
5784 offsetof(struct v9_gfx_meta_data, de_payload);
5785 gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
5786 } else {
5787 offset = offsetof(struct v9_gfx_meta_data, de_payload);
5788 de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset;
5789 de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset;
5790 5751
5791 gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +5752 gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) +
5792 AMDGPU_CSA_SIZE - adev->gds.gds_size,5753 AMDGPU_CSA_SIZE - adev->gds.gds_size,
5793 PAGE_SIZE);5754 PAGE_SIZE);
5794 }
5795 5755
5796 if (usegds) {5756 if (usegds) {
5797 de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);5757 de_payload.gds_backup_addrlo = lower_32_bits(gds_addr);
@@ -7339,9 +7299,14 @@ static void gfx_v9_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer · 1541 unmodified lines
7339 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {7299 for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) {
7340 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);7300 drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k);
7341 for (reg = 0; reg < reg_count; reg++) {7301 for (reg = 0; reg < reg_count; reg++) {
7342 drm_printf(p, "%-50s \t 0x%08x\n",7302 if (i && gc_cp_reg_list_9[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
7343 gc_cp_reg_list_9[reg].reg_name,7303 drm_printf(p, "%-50s \t 0x%08x\n",
7344 adev->gfx.ip_dump_compute_queues[index + reg]);7304 "mmCP_MEC_ME2_HEADER_DUMP",
7305 adev->gfx.ip_dump_compute_queues[index + reg]);
7306 else
7307 drm_printf(p, "%-50s \t 0x%08x\n",
7308 gc_cp_reg_list_9[reg].reg_name,
7309 adev->gfx.ip_dump_compute_queues[index + reg]);
7345 }7310 }
7346 index += reg_count;7311 index += reg_count;
7347 }7312 }
@@ -7378,9 +7343,13 @@ static void gfx_v9_ip_dump(struct amdgpu_ip_block *ip_block) · 30 unmodified lines
7378 soc15_grbm_select(adev, 1 + i, j, k, 0, 0);7343 soc15_grbm_select(adev, 1 + i, j, k, 0, 0);
7379 7344
7380 for (reg = 0; reg < reg_count; reg++) {7345 for (reg = 0; reg < reg_count; reg++) {
7381 adev->gfx.ip_dump_compute_queues[index + reg] =7346 if (i && gc_cp_reg_list_9[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP)
7382 RREG32(SOC15_REG_ENTRY_OFFSET(7347 adev->gfx.ip_dump_compute_queues[index + reg] =
7383 gc_cp_reg_list_9[reg]));7348 RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_MEC_ME2_HEADER_DUMP));
7349 else
7350 adev->gfx.ip_dump_compute_queues[index + reg] =
7351 RREG32(SOC15_REG_ENTRY_OFFSET(
7352 gc_cp_reg_list_9[reg]));
7384 }7353 }
7385 index += reg_count;7354 index += reg_count;
7386 }7355 }
@@ -7394,8 +7363,14 @@ static void gfx_v9_ip_dump(struct amdgpu_ip_block *ip_block) · 7 unmodified lines
7394 7363
7395static void gfx_v9_0_ring_emit_cleaner_shader(struct amdgpu_ring *ring)7364static void gfx_v9_0_ring_emit_cleaner_shader(struct amdgpu_ring *ring)
7396{7365{
7366 struct amdgpu_device *adev = ring->adev;
7367
7397 /* Emit the cleaner shader */7368 /* Emit the cleaner shader */
7398 amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER, 0));7369 if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 2))
7370 amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER, 0));
7371 else
7372 amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER_9_0, 0));
7373
7399 amdgpu_ring_write(ring, 0); /* RESERVED field, programmed to zero */7374 amdgpu_ring_write(ring, 0); /* RESERVED field, programmed to zero */
7400}7375}
7401 7376
drivers/gpu/drm/amd/amdgpu/gfx_v9_4_2.c+2−2
@@ -1547,7 +1547,7 @@ static void gfx_v9_4_2_log_utc_edc_count(struct amdgpu_device *adev, · 1546 unmodified lines
1547{1547{
1548 uint32_t bank, way, mem;1548 uint32_t bank, way, mem;
1549 static const char * const vml2_way_str[] = { "BIGK", "4K" };1549 static const char * const vml2_way_str[] = { "BIGK", "4K" };
1550 static const char * const utcl2_rounter_str[] = { "VMC", "APT" };1550 static const char * const utcl2_router_str[] = { "VMC", "APT" };
1551 1551
1552 mem = instance % blk->num_mem_blocks;1552 mem = instance % blk->num_mem_blocks;
1553 way = (instance / blk->num_mem_blocks) % blk->num_ways;1553 way = (instance / blk->num_mem_blocks) % blk->num_ways;
@@ -1568,7 +1568,7 @@ static void gfx_v9_4_2_log_utc_edc_count(struct amdgpu_device *adev, · 14 unmodified lines
1568 dev_info(1568 dev_info(
1569 adev->dev,1569 adev->dev,
1570 "GFX SubBlock UTCL2_ROUTER_IFIF%d_GROUP0_%s, SED %d, DED %d\n",1570 "GFX SubBlock UTCL2_ROUTER_IFIF%d_GROUP0_%s, SED %d, DED %d\n",
1571 bank, utcl2_rounter_str[mem], sec_cnt, ded_cnt);1571 bank, utcl2_router_str[mem], sec_cnt, ded_cnt);
1572 break;1572 break;
1573 case ATC_L2_CACHE_2M:1573 case ATC_L2_CACHE_2M:
1574 dev_info(1574 dev_info(
drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c+37−15
@@ -105,9 +105,6 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_9_4_3[] = { · 104 unmodified lines
105 SOC15_REG_ENTRY_STR(GC, 0, regRLC_SMU_SAFE_MODE),105 SOC15_REG_ENTRY_STR(GC, 0, regRLC_SMU_SAFE_MODE),
106 SOC15_REG_ENTRY_STR(GC, 0, regRLC_INT_STAT),106 SOC15_REG_ENTRY_STR(GC, 0, regRLC_INT_STAT),
107 SOC15_REG_ENTRY_STR(GC, 0, regRLC_GPM_GENERAL_6),107 SOC15_REG_ENTRY_STR(GC, 0, regRLC_GPM_GENERAL_6),
108 /* cp header registers */
109 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
110 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME2_HEADER_DUMP),
111 /* SE status registers */108 /* SE status registers */
112 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),109 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0),
113 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE1),110 SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE1),
@@ -154,6 +151,14 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9_4_3[] = { · 40 unmodified lines
154 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_LO),151 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_LO),
155 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_HI),152 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_HI),
156 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_GFX_STATUS),153 SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_GFX_STATUS),
154 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
155 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
156 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
157 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
158 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
159 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
160 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
161 SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP),
157};162};
158 163
159struct amdgpu_gfx_ras gfx_v9_4_3_ras;164struct amdgpu_gfx_ras gfx_v9_4_3_ras;
@@ -4558,12 +4563,21 @@ static void gfx_v9_4_3_ip_print(struct amdgpu_ip_block *ip_block, struct drm_pri · 4398 unmodified lines
4558 "\nxcc:%d mec:%d, pipe:%d, queue:%d\n",4563 "\nxcc:%d mec:%d, pipe:%d, queue:%d\n",
4559 xcc_id, i, j, k);4564 xcc_id, i, j, k);
4560 for (reg = 0; reg < reg_count; reg++) {4565 for (reg = 0; reg < reg_count; reg++) {
4561 drm_printf(p,4566 if (i && gc_cp_reg_list_9_4_3[reg].reg_offset ==
4562 "%-50s \t 0x%08x\n",4567 regCP_MEC_ME1_HEADER_DUMP)
4563 gc_cp_reg_list_9_4_3[reg].reg_name,4568 drm_printf(p,
4564 adev->gfx.ip_dump_compute_queues4569 "%-50s \t 0x%08x\n",
4565 [xcc_offset + inst_offset +4570 "regCP_MEC_ME2_HEADER_DUMP",
4566 reg]);4571 adev->gfx.ip_dump_compute_queues
4572 [xcc_offset + inst_offset +
4573 reg]);
4574 else
4575 drm_printf(p,
4576 "%-50s \t 0x%08x\n",
4577 gc_cp_reg_list_9_4_3[reg].reg_name,
4578 adev->gfx.ip_dump_compute_queues
4579 [xcc_offset + inst_offset +
4580 reg]);
4567 }4581 }
4568 inst_offset += reg_count;4582 inst_offset += reg_count;
4569 }4583 }
@@ -4612,12 +4626,20 @@ static void gfx_v9_4_3_ip_dump(struct amdgpu_ip_block *ip_block) · 42 unmodified lines
4612 GET_INST(GC, xcc_id));4626 GET_INST(GC, xcc_id));
4613 4627
4614 for (reg = 0; reg < reg_count; reg++) {4628 for (reg = 0; reg < reg_count; reg++) {
4615 adev->gfx.ip_dump_compute_queues4629 if (i && gc_cp_reg_list_9_4_3[reg].reg_offset ==
4616 [xcc_offset +4630 regCP_MEC_ME1_HEADER_DUMP)
4617 inst_offset + reg] =4631 adev->gfx.ip_dump_compute_queues
4618 RREG32(SOC15_REG_ENTRY_OFFSET_INST(4632 [xcc_offset +
4619 gc_cp_reg_list_9_4_3[reg],4633 inst_offset + reg] =
4620 GET_INST(GC, xcc_id)));4634 RREG32(SOC15_REG_OFFSET(GC, GET_INST(GC, xcc_id),
4635 regCP_MEC_ME2_HEADER_DUMP));
4636 else
4637 adev->gfx.ip_dump_compute_queues
4638 [xcc_offset +
4639 inst_offset + reg] =
4640 RREG32(SOC15_REG_ENTRY_OFFSET_INST(
4641 gc_cp_reg_list_9_4_3[reg],
4642 GET_INST(GC, xcc_id)));
4621 }4643 }
4622 inst_offset += reg_count;4644 inst_offset += reg_count;
4623 }4645 }
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c+0−4
@@ -428,10 +428,6 @@ static void gmc_v10_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned int · 427 unmodified lines
428 struct amdgpu_device *adev = ring->adev;428 struct amdgpu_device *adev = ring->adev;
429 uint32_t reg;429 uint32_t reg;
430 430
431 /* MES fw manages IH_VMID_x_LUT updating */
432 if (ring->is_mes_queue)
433 return;
434
435 if (ring->vm_hub == AMDGPU_GFXHUB(0))431 if (ring->vm_hub == AMDGPU_GFXHUB(0))
436 reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid;432 reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid;
437 else433 else
drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c+1−5
@@ -393,10 +393,6 @@ static void gmc_v11_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned int · 392 unmodified lines
393 struct amdgpu_device *adev = ring->adev;393 struct amdgpu_device *adev = ring->adev;
394 uint32_t reg;394 uint32_t reg;
395 395
396 /* MES fw manages IH_VMID_x_LUT updating */
397 if (ring->is_mes_queue)
398 return;
399
400 if (ring->vm_hub == AMDGPU_GFXHUB(0))396 if (ring->vm_hub == AMDGPU_GFXHUB(0))
401 reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid;397 reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid;
402 else398 else
@@ -832,7 +828,7 @@ static int gmc_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 429 unmodified lines
832 * amdgpu graphics/compute will use VMIDs 1-7828 * amdgpu graphics/compute will use VMIDs 1-7
833 * amdkfd will use VMIDs 8-15829 * amdkfd will use VMIDs 8-15
834 */830 */
835 adev->vm_manager.first_kfd_vmid = 8;831 adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
836 832
837 amdgpu_vm_manager_init(adev);833 amdgpu_vm_manager_init(adev);
838 834
drivers/gpu/drm/amd/amdgpu/gmc_v12_0.c+1−5
@@ -413,10 +413,6 @@ static void gmc_v12_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned vmid · 412 unmodified lines
413 struct amdgpu_device *adev = ring->adev;413 struct amdgpu_device *adev = ring->adev;
414 uint32_t reg;414 uint32_t reg;
415 415
416 /* MES fw manages IH_VMID_x_LUT updating */
417 if (ring->is_mes_queue)
418 return;
419
420 if (ring->vm_hub == AMDGPU_GFXHUB(0))416 if (ring->vm_hub == AMDGPU_GFXHUB(0))
421 reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid;417 reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid;
422 else418 else
@@ -820,7 +816,7 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 397 unmodified lines
820 * amdgpu graphics/compute will use VMIDs 1-7816 * amdgpu graphics/compute will use VMIDs 1-7
821 * amdkfd will use VMIDs 8-15817 * amdkfd will use VMIDs 8-15
822 */818 */
823 adev->vm_manager.first_kfd_vmid = 8;819 adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8;
824 820
825 amdgpu_vm_manager_init(adev);821 amdgpu_vm_manager_init(adev);
826 822
drivers/gpu/drm/amd/amdgpu/gmc_v6_0.c+29−17
@@ -249,7 +249,7 @@ static void gmc_v6_0_mc_program(struct amdgpu_device *adev) · 248 unmodified lines
249 249
250 /* disable VGA render */250 /* disable VGA render */
251 tmp = RREG32(mmVGA_RENDER_CONTROL);251 tmp = RREG32(mmVGA_RENDER_CONTROL);
252 tmp &= ~VGA_VSTATUS_CNTL;252 tmp &= VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK;
253 WREG32(mmVGA_RENDER_CONTROL, tmp);253 WREG32(mmVGA_RENDER_CONTROL, tmp);
254 }254 }
255 /* Update configuration */255 /* Update configuration */
@@ -627,17 +627,16 @@ static void gmc_v6_0_vm_decode_fault(struct amdgpu_device *adev, · 371 unmodified lines
627 "write" : "read", block, mc_client, mc_id);627 "write" : "read", block, mc_client, mc_id);
628}628}
629 629
630/*
631static const u32 mc_cg_registers[] = {630static const u32 mc_cg_registers[] = {
632 MC_HUB_MISC_HUB_CG,631 mmMC_HUB_MISC_HUB_CG,
633 MC_HUB_MISC_SIP_CG,632 mmMC_HUB_MISC_SIP_CG,
634 MC_HUB_MISC_VM_CG,633 mmMC_HUB_MISC_VM_CG,
635 MC_XPB_CLK_GAT,634 mmMC_XPB_CLK_GAT,
636 ATC_MISC_CG,635 mmATC_MISC_CG,
637 MC_CITF_MISC_WR_CG,636 mmMC_CITF_MISC_WR_CG,
638 MC_CITF_MISC_RD_CG,637 mmMC_CITF_MISC_RD_CG,
639 MC_CITF_MISC_VM_CG,638 mmMC_CITF_MISC_VM_CG,
640 VM_L2_CG,639 mmVM_L2_CG,
641};640};
642 641
643static const u32 mc_cg_ls_en[] = {642static const u32 mc_cg_ls_en[] = {
@@ -672,7 +671,7 @@ static void gmc_v6_0_enable_mc_ls(struct amdgpu_device *adev, · 28 unmodified lines
672 671
673 for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {672 for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {
674 orig = data = RREG32(mc_cg_registers[i]);673 orig = data = RREG32(mc_cg_registers[i]);
675 if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_LS))674 if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_LS))
676 data |= mc_cg_ls_en[i];675 data |= mc_cg_ls_en[i];
677 else676 else
678 data &= ~mc_cg_ls_en[i];677 data &= ~mc_cg_ls_en[i];
@@ -689,7 +688,7 @@ static void gmc_v6_0_enable_mc_mgcg(struct amdgpu_device *adev, · 10 unmodified lines
689 688
690 for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {689 for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) {
691 orig = data = RREG32(mc_cg_registers[i]);690 orig = data = RREG32(mc_cg_registers[i]);
692 if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_MGCG))691 if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_MGCG))
693 data |= mc_cg_en[i];692 data |= mc_cg_en[i];
694 else693 else
695 data &= ~mc_cg_en[i];694 data &= ~mc_cg_en[i];
@@ -705,7 +704,7 @@ static void gmc_v6_0_enable_bif_mgls(struct amdgpu_device *adev, · 9 unmodified lines
705 704
706 orig = data = RREG32_PCIE(ixPCIE_CNTL2);705 orig = data = RREG32_PCIE(ixPCIE_CNTL2);
707 706
708 if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_BIF_LS)) {707 if (enable && (adev->cg_flags & AMD_CG_SUPPORT_BIF_LS)) {
709 data = REG_SET_FIELD(data, PCIE_CNTL2, SLV_MEM_LS_EN, 1);708 data = REG_SET_FIELD(data, PCIE_CNTL2, SLV_MEM_LS_EN, 1);
710 data = REG_SET_FIELD(data, PCIE_CNTL2, MST_MEM_LS_EN, 1);709 data = REG_SET_FIELD(data, PCIE_CNTL2, MST_MEM_LS_EN, 1);
711 data = REG_SET_FIELD(data, PCIE_CNTL2, REPLAY_MEM_LS_EN, 1);710 data = REG_SET_FIELD(data, PCIE_CNTL2, REPLAY_MEM_LS_EN, 1);
@@ -728,7 +727,7 @@ static void gmc_v6_0_enable_hdp_mgcg(struct amdgpu_device *adev, · 16 unmodified lines
728 727
729 orig = data = RREG32(mmHDP_HOST_PATH_CNTL);728 orig = data = RREG32(mmHDP_HOST_PATH_CNTL);
730 729
731 if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_MGCG))730 if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_MGCG))
732 data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 0);731 data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 0);
733 else732 else
734 data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 1);733 data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 1);
@@ -744,7 +743,7 @@ static void gmc_v6_0_enable_hdp_ls(struct amdgpu_device *adev, · 9 unmodified lines
744 743
745 orig = data = RREG32(mmHDP_MEM_POWER_LS);744 orig = data = RREG32(mmHDP_MEM_POWER_LS);
746 745
747 if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_LS))746 if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_LS))
748 data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 1);747 data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 1);
749 else748 else
750 data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 0);749 data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 0);
@@ -752,7 +751,6 @@ static void gmc_v6_0_enable_hdp_ls(struct amdgpu_device *adev, · 1 unmodified line
752 if (orig != data)751 if (orig != data)
753 WREG32(mmHDP_MEM_POWER_LS, data);752 WREG32(mmHDP_MEM_POWER_LS, data);
754}753}
755*/
756 754
757static int gmc_v6_0_convert_vram_type(int mc_seq_vram_type)755static int gmc_v6_0_convert_vram_type(int mc_seq_vram_type)
758{756{
@@ -1098,6 +1096,20 @@ static int gmc_v6_0_process_interrupt(struct amdgpu_device *adev, · 339 unmodified lines
1098static int gmc_v6_0_set_clockgating_state(struct amdgpu_ip_block *ip_block,1096static int gmc_v6_0_set_clockgating_state(struct amdgpu_ip_block *ip_block,
1099 enum amd_clockgating_state state)1097 enum amd_clockgating_state state)
1100{1098{
1099 struct amdgpu_device *adev = ip_block->adev;
1100 bool gate = false;
1101
1102 if (state == AMD_CG_STATE_GATE)
1103 gate = true;
1104
1105 if (!(adev->flags & AMD_IS_APU)) {
1106 gmc_v6_0_enable_mc_mgcg(adev, gate);
1107 gmc_v6_0_enable_mc_ls(adev, gate);
1108 }
1109 gmc_v6_0_enable_bif_mgls(adev, gate);
1110 gmc_v6_0_enable_hdp_mgcg(adev, gate);
1111 gmc_v6_0_enable_hdp_ls(adev, gate);
1112
1101 return 0;1113 return 0;
1102}1114}
1103 1115
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c+1−8
@@ -1157,17 +1157,10 @@ static bool gmc_v7_0_is_idle(struct amdgpu_ip_block *ip_block) · 1156 unmodified lines
1157static int gmc_v7_0_wait_for_idle(struct amdgpu_ip_block *ip_block)1157static int gmc_v7_0_wait_for_idle(struct amdgpu_ip_block *ip_block)
1158{1158{
1159 unsigned int i;1159 unsigned int i;
1160 u32 tmp;
1161 struct amdgpu_device *adev = ip_block->adev;1160 struct amdgpu_device *adev = ip_block->adev;
1162 1161
1163 for (i = 0; i < adev->usec_timeout; i++) {1162 for (i = 0; i < adev->usec_timeout; i++) {
1164 /* read MC_STATUS */1163 if (gmc_v7_0_is_idle(ip_block))
1165 tmp = RREG32(mmSRBM_STATUS) & (SRBM_STATUS__MCB_BUSY_MASK |
1166 SRBM_STATUS__MCB_NON_DISPLAY_BUSY_MASK |
1167 SRBM_STATUS__MCC_BUSY_MASK |
1168 SRBM_STATUS__MCD_BUSY_MASK |
1169 SRBM_STATUS__VMC_BUSY_MASK);
1170 if (!tmp)
1171 return 0;1164 return 0;
1172 udelay(1);1165 udelay(1);
1173 }1166 }
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c+2−12
@@ -1213,10 +1213,7 @@ static void gmc_v9_0_get_coherence_flags(struct amdgpu_device *adev, · 1212 unmodified lines
1213 if (uncached) {1213 if (uncached) {
1214 mtype = MTYPE_UC;1214 mtype = MTYPE_UC;
1215 } else if (ext_coherent) {1215 } else if (ext_coherent) {
1216 if (gc_ip_version == IP_VERSION(9, 5, 0) || adev->rev_id)1216 mtype = is_local ? MTYPE_CC : MTYPE_UC;
1217 mtype = is_local ? MTYPE_CC : MTYPE_UC;
1218 else
1219 mtype = MTYPE_UC;
1220 } else if (adev->flags & AMD_IS_APU) {1217 } else if (adev->flags & AMD_IS_APU) {
1221 mtype = is_local ? mtype_local : MTYPE_NC;1218 mtype = is_local ? mtype_local : MTYPE_NC;
1222 } else {1219 } else {
@@ -1336,7 +1333,7 @@ static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev, · 113 unmodified lines
1336 mtype_local = MTYPE_CC;1333 mtype_local = MTYPE_CC;
1337 1334
1338 *flags = AMDGPU_PTE_MTYPE_VG10(*flags, mtype_local);1335 *flags = AMDGPU_PTE_MTYPE_VG10(*flags, mtype_local);
1339 } else if (adev->rev_id) {1336 } else {
1340 /* MTYPE_UC case */1337 /* MTYPE_UC case */
1341 *flags = AMDGPU_PTE_MTYPE_VG10(*flags, MTYPE_CC);1338 *flags = AMDGPU_PTE_MTYPE_VG10(*flags, MTYPE_CC);
1342 }1339 }
@@ -2411,13 +2408,6 @@ static int gmc_v9_0_hw_init(struct amdgpu_ip_block *ip_block) · 1068 unmodified lines
2411 adev->gmc.flush_tlb_needs_extra_type_2 =2408 adev->gmc.flush_tlb_needs_extra_type_2 =
2412 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 0) &&2409 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 0) &&
2413 adev->gmc.xgmi.num_physical_nodes;2410 adev->gmc.xgmi.num_physical_nodes;
2414 /*
2415 * TODO: This workaround is badly documented and had a buggy
2416 * implementation. We should probably verify what we do here.
2417 */
2418 adev->gmc.flush_tlb_needs_extra_type_0 =
2419 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) &&
2420 adev->rev_id == 0;
2421 2411
2422 /* The sequence of these two function calls matters.*/2412 /* The sequence of these two function calls matters.*/
2423 gmc_v9_0_init_golden_registers(adev);2413 gmc_v9_0_init_golden_registers(adev);
drivers/gpu/drm/amd/amdgpu/hdp_v4_0.c+1−12
@@ -36,17 +36,6 @@ · 35 unmodified lines
36#define HDP_MEM_POWER_CTRL__RC_MEM_POWER_LS_EN_MASK 0x00020000L36#define HDP_MEM_POWER_CTRL__RC_MEM_POWER_LS_EN_MASK 0x00020000L
37#define mmHDP_MEM_POWER_CTRL_BASE_IDX 037#define mmHDP_MEM_POWER_CTRL_BASE_IDX 0
38 38
39static void hdp_v4_0_flush_hdp(struct amdgpu_device *adev,
40 struct amdgpu_ring *ring)
41{
42 if (!ring || !ring->funcs->emit_wreg) {
43 WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
44 RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
45 } else {
46 amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
47 }
48}
49
50static void hdp_v4_0_invalidate_hdp(struct amdgpu_device *adev,39static void hdp_v4_0_invalidate_hdp(struct amdgpu_device *adev,
51 struct amdgpu_ring *ring)40 struct amdgpu_ring *ring)
52{41{
@@ -180,7 +169,7 @@ struct amdgpu_hdp_ras hdp_v4_0_ras = { · 127 unmodified lines
180};169};
181 170
182const struct amdgpu_hdp_funcs hdp_v4_0_funcs = {171const struct amdgpu_hdp_funcs hdp_v4_0_funcs = {
183 .flush_hdp = hdp_v4_0_flush_hdp,172 .flush_hdp = amdgpu_hdp_generic_flush,
184 .invalidate_hdp = hdp_v4_0_invalidate_hdp,173 .invalidate_hdp = hdp_v4_0_invalidate_hdp,
185 .update_clock_gating = hdp_v4_0_update_clock_gating,174 .update_clock_gating = hdp_v4_0_update_clock_gating,
186 .get_clock_gating_state = hdp_v4_0_get_clockgating_state,175 .get_clock_gating_state = hdp_v4_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/hdp_v5_0.c+1−12
@@ -27,17 +27,6 @@ · 26 unmodified lines
27#include "hdp/hdp_5_0_0_sh_mask.h"27#include "hdp/hdp_5_0_0_sh_mask.h"
28#include <uapi/linux/kfd_ioctl.h>28#include <uapi/linux/kfd_ioctl.h>
29 29
30static void hdp_v5_0_flush_hdp(struct amdgpu_device *adev,
31 struct amdgpu_ring *ring)
32{
33 if (!ring || !ring->funcs->emit_wreg) {
34 WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
35 RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
36 } else {
37 amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
38 }
39}
40
41static void hdp_v5_0_invalidate_hdp(struct amdgpu_device *adev,30static void hdp_v5_0_invalidate_hdp(struct amdgpu_device *adev,
42 struct amdgpu_ring *ring)31 struct amdgpu_ring *ring)
43{32{
@@ -217,7 +206,7 @@ static void hdp_v5_0_init_registers(struct amdgpu_device *adev) · 173 unmodified lines
217}206}
218 207
219const struct amdgpu_hdp_funcs hdp_v5_0_funcs = {208const struct amdgpu_hdp_funcs hdp_v5_0_funcs = {
220 .flush_hdp = hdp_v5_0_flush_hdp,209 .flush_hdp = amdgpu_hdp_generic_flush,
221 .invalidate_hdp = hdp_v5_0_invalidate_hdp,210 .invalidate_hdp = hdp_v5_0_invalidate_hdp,
222 .update_clock_gating = hdp_v5_0_update_clock_gating,211 .update_clock_gating = hdp_v5_0_update_clock_gating,
223 .get_clock_gating_state = hdp_v5_0_get_clockgating_state,212 .get_clock_gating_state = hdp_v5_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/hdp_v5_2.c+11−1
@@ -33,7 +33,17 @@ static void hdp_v5_2_flush_hdp(struct amdgpu_device *adev, · 32 unmodified lines
33 if (!ring || !ring->funcs->emit_wreg) {33 if (!ring || !ring->funcs->emit_wreg) {
34 WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2,34 WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2,
35 0);35 0);
36 RREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);36 if (amdgpu_sriov_vf(adev)) {
37 /* this is fine because SR_IOV doesn't remap the register */
38 RREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
39 } else {
40 /* We just need to read back a register to post the write.
41 * Reading back the remapped register causes problems on
42 * some platforms so just read back the memory size register.
43 */
44 if (adev->nbio.funcs->get_memsize)
45 adev->nbio.funcs->get_memsize(adev);
46 }
37 } else {47 } else {
38 amdgpu_ring_emit_wreg(ring,48 amdgpu_ring_emit_wreg(ring,
39 (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2,49 (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2,
drivers/gpu/drm/amd/amdgpu/hdp_v6_0.c+1−12
@@ -30,17 +30,6 @@ · 29 unmodified lines
30#define regHDP_CLK_CNTL_V6_1 0xd530#define regHDP_CLK_CNTL_V6_1 0xd5
31#define regHDP_CLK_CNTL_V6_1_BASE_IDX 031#define regHDP_CLK_CNTL_V6_1_BASE_IDX 0
32 32
33static void hdp_v6_0_flush_hdp(struct amdgpu_device *adev,
34 struct amdgpu_ring *ring)
35{
36 if (!ring || !ring->funcs->emit_wreg) {
37 WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
38 RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
39 } else {
40 amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
41 }
42}
43
44static void hdp_v6_0_update_clock_gating(struct amdgpu_device *adev,33static void hdp_v6_0_update_clock_gating(struct amdgpu_device *adev,
45 bool enable)34 bool enable)
46{35{
@@ -149,7 +138,7 @@ static void hdp_v6_0_get_clockgating_state(struct amdgpu_device *adev, · 102 unmodified lines
149}138}
150 139
151const struct amdgpu_hdp_funcs hdp_v6_0_funcs = {140const struct amdgpu_hdp_funcs hdp_v6_0_funcs = {
152 .flush_hdp = hdp_v6_0_flush_hdp,141 .flush_hdp = amdgpu_hdp_generic_flush,
153 .update_clock_gating = hdp_v6_0_update_clock_gating,142 .update_clock_gating = hdp_v6_0_update_clock_gating,
154 .get_clock_gating_state = hdp_v6_0_get_clockgating_state,143 .get_clock_gating_state = hdp_v6_0_get_clockgating_state,
155};144};
drivers/gpu/drm/amd/amdgpu/hdp_v7_0.c+1−12
@@ -27,17 +27,6 @@ · 26 unmodified lines
27#include "hdp/hdp_7_0_0_sh_mask.h"27#include "hdp/hdp_7_0_0_sh_mask.h"
28#include <uapi/linux/kfd_ioctl.h>28#include <uapi/linux/kfd_ioctl.h>
29 29
30static void hdp_v7_0_flush_hdp(struct amdgpu_device *adev,
31 struct amdgpu_ring *ring)
32{
33 if (!ring || !ring->funcs->emit_wreg) {
34 WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
35 RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2);
36 } else {
37 amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);
38 }
39}
40
41static void hdp_v7_0_update_clock_gating(struct amdgpu_device *adev,30static void hdp_v7_0_update_clock_gating(struct amdgpu_device *adev,
42 bool enable)31 bool enable)
43{32{
@@ -137,7 +126,7 @@ static void hdp_v7_0_get_clockgating_state(struct amdgpu_device *adev, · 93 unmodified lines
137}126}
138 127
139const struct amdgpu_hdp_funcs hdp_v7_0_funcs = {128const struct amdgpu_hdp_funcs hdp_v7_0_funcs = {
140 .flush_hdp = hdp_v7_0_flush_hdp,129 .flush_hdp = amdgpu_hdp_generic_flush,
141 .update_clock_gating = hdp_v7_0_update_clock_gating,130 .update_clock_gating = hdp_v7_0_update_clock_gating,
142 .get_clock_gating_state = hdp_v7_0_get_clockgating_state,131 .get_clock_gating_state = hdp_v7_0_get_clockgating_state,
143};132};
drivers/gpu/drm/amd/amdgpu/ih_v6_0.c+5−1
@@ -349,6 +349,7 @@ static int ih_v6_0_irq_init(struct amdgpu_device *adev) · 348 unmodified lines
349 if (ret)349 if (ret)
350 return ret;350 return ret;
351 }351 }
352 ih[i]->overflow = false;
352 }353 }
353 354
354 /* update doorbell range for ih ring 0 */355 /* update doorbell range for ih ring 0 */
@@ -446,7 +447,10 @@ static u32 ih_v6_0_get_wptr(struct amdgpu_device *adev, · 91 unmodified lines
446 wptr = RREG32_NO_KIQ(ih_regs->ih_rb_wptr);447 wptr = RREG32_NO_KIQ(ih_regs->ih_rb_wptr);
447 if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW))448 if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW))
448 goto out;449 goto out;
449 wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);450 if (!amdgpu_sriov_vf(adev))
451 wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);
452 else
453 ih->overflow = true;
450 454
451 /* When a ring buffer overflow happen start parsing interrupt455 /* When a ring buffer overflow happen start parsing interrupt
452 * from the last not overwritten vector (wptr + 32). Hopefully456 * from the last not overwritten vector (wptr + 32). Hopefully
drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c+2−2
@@ -156,7 +156,7 @@ static int jpeg_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 155 unmodified lines
156 156
157 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {157 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {
158 ring = &adev->jpeg.inst[i].ring_dec[j];158 ring = &adev->jpeg.inst[i].ring_dec[j];
159 ring->use_doorbell = false;159 ring->use_doorbell = true;
160 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);160 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);
161 if (!amdgpu_sriov_vf(adev)) {161 if (!amdgpu_sriov_vf(adev)) {
162 ring->doorbell_index =162 ring->doorbell_index =
@@ -264,7 +264,7 @@ static int jpeg_v5_0_1_hw_init(struct amdgpu_ip_block *ip_block) · 101 unmodified lines
264 ring = &adev->jpeg.inst[i].ring_dec[j];264 ring = &adev->jpeg.inst[i].ring_dec[j];
265 if (ring->use_doorbell)265 if (ring->use_doorbell)
266 WREG32_SOC15_OFFSET(VCN, GET_INST(VCN, i), regVCN_JPEG_DB_CTRL,266 WREG32_SOC15_OFFSET(VCN, GET_INST(VCN, i), regVCN_JPEG_DB_CTRL,
267 (ring->pipe ? (ring->pipe - 0x15) : 0),267 ring->pipe,
268 ring->doorbell_index <<268 ring->doorbell_index <<
269 VCN_JPEG_DB_CTRL__OFFSET__SHIFT |269 VCN_JPEG_DB_CTRL__OFFSET__SHIFT |
270 VCN_JPEG_DB_CTRL__EN_MASK);270 VCN_JPEG_DB_CTRL__EN_MASK);
drivers/gpu/drm/amd/amdgpu/mes_userqueue.c+355−0
@@ -0,0 +1,355 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright 2024 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24#include "amdgpu.h"
25#include "amdgpu_gfx.h"
26#include "mes_userqueue.h"
27#include "amdgpu_userq_fence.h"
28
29#define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE
30#define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE
31
32static int
33mes_userq_map_gtt_bo_to_gart(struct amdgpu_bo *bo)
34{
35 int ret;
36
37 ret = amdgpu_bo_reserve(bo, true);
38 if (ret) {
39 DRM_ERROR("Failed to reserve bo. ret %d\n", ret);
40 goto err_reserve_bo_failed;
41 }
42
43 ret = amdgpu_ttm_alloc_gart(&bo->tbo);
44 if (ret) {
45 DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret);
46 goto err_map_bo_gart_failed;
47 }
48
49 amdgpu_bo_unreserve(bo);
50 bo = amdgpu_bo_ref(bo);
51
52 return 0;
53
54err_map_bo_gart_failed:
55 amdgpu_bo_unreserve(bo);
56err_reserve_bo_failed:
57 return ret;
58}
59
60static int
61mes_userq_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr,
62 struct amdgpu_usermode_queue *queue,
63 uint64_t wptr)
64{
65 struct amdgpu_bo_va_mapping *wptr_mapping;
66 struct amdgpu_vm *wptr_vm;
67 struct amdgpu_userq_obj *wptr_obj = &queue->wptr_obj;
68 int ret;
69
70 wptr_vm = queue->vm;
71 ret = amdgpu_bo_reserve(wptr_vm->root.bo, false);
72 if (ret)
73 return ret;
74
75 wptr &= AMDGPU_GMC_HOLE_MASK;
76 wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> PAGE_SHIFT);
77 amdgpu_bo_unreserve(wptr_vm->root.bo);
78 if (!wptr_mapping) {
79 DRM_ERROR("Failed to lookup wptr bo\n");
80 return -EINVAL;
81 }
82
83 wptr_obj->obj = wptr_mapping->bo_va->base.bo;
84 if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) {
85 DRM_ERROR("Requested GART mapping for wptr bo larger than one page\n");
86 return -EINVAL;
87 }
88
89 ret = mes_userq_map_gtt_bo_to_gart(wptr_obj->obj);
90 if (ret) {
91 DRM_ERROR("Failed to map wptr bo to GART\n");
92 return ret;
93 }
94
95 queue->wptr_obj.gpu_addr = amdgpu_bo_gpu_offset_no_check(wptr_obj->obj);
96 return 0;
97}
98
99static int convert_to_mes_priority(int priority)
100{
101 switch (priority) {
102 case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_NORMAL_LOW:
103 default:
104 return AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
105 case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_LOW:
106 return AMDGPU_MES_PRIORITY_LEVEL_LOW;
107 case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_NORMAL_HIGH:
108 return AMDGPU_MES_PRIORITY_LEVEL_MEDIUM;
109 case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_HIGH:
110 return AMDGPU_MES_PRIORITY_LEVEL_HIGH;
111 }
112}
113
114static int mes_userq_map(struct amdgpu_userq_mgr *uq_mgr,
115 struct amdgpu_usermode_queue *queue)
116{
117 struct amdgpu_device *adev = uq_mgr->adev;
118 struct amdgpu_userq_obj *ctx = &queue->fw_obj;
119 struct amdgpu_mqd_prop *userq_props = queue->userq_prop;
120 struct mes_add_queue_input queue_input;
121 int r;
122
123 memset(&queue_input, 0x0, sizeof(struct mes_add_queue_input));
124
125 queue_input.process_va_start = 0;
126 queue_input.process_va_end = adev->vm_manager.max_pfn - 1;
127
128 /* set process quantum to 10 ms and gang quantum to 1 ms as default */
129 queue_input.process_quantum = 100000;
130 queue_input.gang_quantum = 10000;
131 queue_input.paging = false;
132
133 queue_input.process_context_addr = ctx->gpu_addr;
134 queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;
135 queue_input.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
136 queue_input.gang_global_priority_level = convert_to_mes_priority(queue->priority);
137
138 queue_input.process_id = queue->vm->pasid;
139 queue_input.queue_type = queue->queue_type;
140 queue_input.mqd_addr = queue->mqd.gpu_addr;
141 queue_input.wptr_addr = userq_props->wptr_gpu_addr;
142 queue_input.queue_size = userq_props->queue_size >> 2;
143 queue_input.doorbell_offset = userq_props->doorbell_index;
144 queue_input.page_table_base_addr = amdgpu_gmc_pd_addr(queue->vm->root.bo);
145 queue_input.wptr_mc_addr = queue->wptr_obj.gpu_addr;
146
147 amdgpu_mes_lock(&adev->mes);
148 r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input);
149 amdgpu_mes_unlock(&adev->mes);
150 if (r) {
151 DRM_ERROR("Failed to map queue in HW, err (%d)\n", r);
152 return r;
153 }
154
155 DRM_DEBUG_DRIVER("Queue (doorbell:%d) mapped successfully\n", userq_props->doorbell_index);
156 return 0;
157}
158
159static int mes_userq_unmap(struct amdgpu_userq_mgr *uq_mgr,
160 struct amdgpu_usermode_queue *queue)
161{
162 struct amdgpu_device *adev = uq_mgr->adev;
163 struct mes_remove_queue_input queue_input;
164 struct amdgpu_userq_obj *ctx = &queue->fw_obj;
165 int r;
166
167 memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));
168 queue_input.doorbell_offset = queue->doorbell_index;
169 queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;
170
171 amdgpu_mes_lock(&adev->mes);
172 r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);
173 amdgpu_mes_unlock(&adev->mes);
174 if (r)
175 DRM_ERROR("Failed to unmap queue in HW, err (%d)\n", r);
176 return r;
177}
178
179static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr,
180 struct amdgpu_usermode_queue *queue,
181 struct drm_amdgpu_userq_in *mqd_user)
182{
183 struct amdgpu_userq_obj *ctx = &queue->fw_obj;
184 int r, size;
185
186 /*
187 * The FW expects at least one page space allocated for
188 * process ctx and gang ctx each. Create an object
189 * for the same.
190 */
191 size = AMDGPU_USERQ_PROC_CTX_SZ + AMDGPU_USERQ_GANG_CTX_SZ;
192 r = amdgpu_userq_create_object(uq_mgr, ctx, size);
193 if (r) {
194 DRM_ERROR("Failed to allocate ctx space bo for userqueue, err:%d\n", r);
195 return r;
196 }
197
198 return 0;
199}
200
201static int mes_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr,
202 struct drm_amdgpu_userq_in *args_in,
203 struct amdgpu_usermode_queue *queue)
204{
205 struct amdgpu_device *adev = uq_mgr->adev;
206 struct amdgpu_mqd *mqd_hw_default = &adev->mqds[queue->queue_type];
207 struct drm_amdgpu_userq_in *mqd_user = args_in;
208 struct amdgpu_mqd_prop *userq_props;
209 int r;
210
211 /* Structure to initialize MQD for userqueue using generic MQD init function */
212 userq_props = kzalloc(sizeof(struct amdgpu_mqd_prop), GFP_KERNEL);
213 if (!userq_props) {
214 DRM_ERROR("Failed to allocate memory for userq_props\n");
215 return -ENOMEM;
216 }
217
218 if (!mqd_user->wptr_va || !mqd_user->rptr_va ||
219 !mqd_user->queue_va || mqd_user->queue_size == 0) {
220 DRM_ERROR("Invalid MQD parameters for userqueue\n");
221 r = -EINVAL;
222 goto free_props;
223 }
224
225 r = amdgpu_userq_create_object(uq_mgr, &queue->mqd, mqd_hw_default->mqd_size);
226 if (r) {
227 DRM_ERROR("Failed to create MQD object for userqueue\n");
228 goto free_props;
229 }
230
231 /* Initialize the MQD BO with user given values */
232 userq_props->wptr_gpu_addr = mqd_user->wptr_va;
233 userq_props->rptr_gpu_addr = mqd_user->rptr_va;
234 userq_props->queue_size = mqd_user->queue_size;
235 userq_props->hqd_base_gpu_addr = mqd_user->queue_va;
236 userq_props->mqd_gpu_addr = queue->mqd.gpu_addr;
237 userq_props->use_doorbell = true;
238 userq_props->doorbell_index = queue->doorbell_index;
239 userq_props->fence_address = queue->fence_drv->gpu_addr;
240
241 if (queue->queue_type == AMDGPU_HW_IP_COMPUTE) {
242 struct drm_amdgpu_userq_mqd_compute_gfx11 *compute_mqd;
243
244 if (mqd_user->mqd_size != sizeof(*compute_mqd)) {
245 DRM_ERROR("Invalid compute IP MQD size\n");
246 r = -EINVAL;
247 goto free_mqd;
248 }
249
250 compute_mqd = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
251 if (IS_ERR(compute_mqd)) {
252 DRM_ERROR("Failed to read user MQD\n");
253 r = -ENOMEM;
254 goto free_mqd;
255 }
256
257 userq_props->eop_gpu_addr = compute_mqd->eop_va;
258 userq_props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL;
259 userq_props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM;
260 userq_props->hqd_active = false;
261 userq_props->tmz_queue =
262 mqd_user->flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE;
263 kfree(compute_mqd);
264 } else if (queue->queue_type == AMDGPU_HW_IP_GFX) {
265 struct drm_amdgpu_userq_mqd_gfx11 *mqd_gfx_v11;
266
267 if (mqd_user->mqd_size != sizeof(*mqd_gfx_v11) || !mqd_user->mqd) {
268 DRM_ERROR("Invalid GFX MQD\n");
269 r = -EINVAL;
270 goto free_mqd;
271 }
272
273 mqd_gfx_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
274 if (IS_ERR(mqd_gfx_v11)) {
275 DRM_ERROR("Failed to read user MQD\n");
276 r = -ENOMEM;
277 goto free_mqd;
278 }
279
280 userq_props->shadow_addr = mqd_gfx_v11->shadow_va;
281 userq_props->csa_addr = mqd_gfx_v11->csa_va;
282 userq_props->tmz_queue =
283 mqd_user->flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE;
284 kfree(mqd_gfx_v11);
285 } else if (queue->queue_type == AMDGPU_HW_IP_DMA) {
286 struct drm_amdgpu_userq_mqd_sdma_gfx11 *mqd_sdma_v11;
287
288 if (mqd_user->mqd_size != sizeof(*mqd_sdma_v11) || !mqd_user->mqd) {
289 DRM_ERROR("Invalid SDMA MQD\n");
290 r = -EINVAL;
291 goto free_mqd;
292 }
293
294 mqd_sdma_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size);
295 if (IS_ERR(mqd_sdma_v11)) {
296 DRM_ERROR("Failed to read sdma user MQD\n");
297 r = -ENOMEM;
298 goto free_mqd;
299 }
300
301 userq_props->csa_addr = mqd_sdma_v11->csa_va;
302 kfree(mqd_sdma_v11);
303 }
304
305 queue->userq_prop = userq_props;
306
307 r = mqd_hw_default->init_mqd(adev, (void *)queue->mqd.cpu_ptr, userq_props);
308 if (r) {
309 DRM_ERROR("Failed to initialize MQD for userqueue\n");
310 goto free_mqd;
311 }
312
313 /* Create BO for FW operations */
314 r = mes_userq_create_ctx_space(uq_mgr, queue, mqd_user);
315 if (r) {
316 DRM_ERROR("Failed to allocate BO for userqueue (%d)", r);
317 goto free_mqd;
318 }
319
320 /* FW expects WPTR BOs to be mapped into GART */
321 r = mes_userq_create_wptr_mapping(uq_mgr, queue, userq_props->wptr_gpu_addr);
322 if (r) {
323 DRM_ERROR("Failed to create WPTR mapping\n");
324 goto free_ctx;
325 }
326
327 return 0;
328
329free_ctx:
330 amdgpu_userq_destroy_object(uq_mgr, &queue->fw_obj);
331
332free_mqd:
333 amdgpu_userq_destroy_object(uq_mgr, &queue->mqd);
334
335free_props:
336 kfree(userq_props);
337
338 return r;
339}
340
341static void
342mes_userq_mqd_destroy(struct amdgpu_userq_mgr *uq_mgr,
343 struct amdgpu_usermode_queue *queue)
344{
345 amdgpu_userq_destroy_object(uq_mgr, &queue->fw_obj);
346 kfree(queue->userq_prop);
347 amdgpu_userq_destroy_object(uq_mgr, &queue->mqd);
348}
349
350const struct amdgpu_userq_funcs userq_mes_funcs = {
351 .mqd_create = mes_userq_mqd_create,
352 .mqd_destroy = mes_userq_mqd_destroy,
353 .unmap = mes_userq_unmap,
354 .map = mes_userq_map,
355};
drivers/gpu/drm/amd/amdgpu/mes_userqueue.h+30−0
@@ -0,0 +1,30 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright 2024 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#ifndef MES_USERQ_H
26#define MES_USERQ_H
27#include "amdgpu_userq.h"
28
29extern const struct amdgpu_userq_funcs userq_mes_funcs;
30#endif
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c+27−47
@@ -287,6 +287,23 @@ static int convert_to_mes_queue_type(int queue_type) · 286 unmodified lines
287 return -1;287 return -1;
288}288}
289 289
290static int convert_to_mes_priority_level(int priority_level)
291{
292 switch (priority_level) {
293 case AMDGPU_MES_PRIORITY_LEVEL_LOW:
294 return AMD_PRIORITY_LEVEL_LOW;
295 case AMDGPU_MES_PRIORITY_LEVEL_NORMAL:
296 default:
297 return AMD_PRIORITY_LEVEL_NORMAL;
298 case AMDGPU_MES_PRIORITY_LEVEL_MEDIUM:
299 return AMD_PRIORITY_LEVEL_MEDIUM;
300 case AMDGPU_MES_PRIORITY_LEVEL_HIGH:
301 return AMD_PRIORITY_LEVEL_HIGH;
302 case AMDGPU_MES_PRIORITY_LEVEL_REALTIME:
303 return AMD_PRIORITY_LEVEL_REALTIME;
304 }
305}
306
290static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes,307static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes,
291 struct mes_add_queue_input *input)308 struct mes_add_queue_input *input)
292{309{
@@ -310,9 +327,9 @@ static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines
310 mes_add_queue_pkt.gang_quantum = input->gang_quantum;327 mes_add_queue_pkt.gang_quantum = input->gang_quantum;
311 mes_add_queue_pkt.gang_context_addr = input->gang_context_addr;328 mes_add_queue_pkt.gang_context_addr = input->gang_context_addr;
312 mes_add_queue_pkt.inprocess_gang_priority =329 mes_add_queue_pkt.inprocess_gang_priority =
313 input->inprocess_gang_priority;330 convert_to_mes_priority_level(input->inprocess_gang_priority);
314 mes_add_queue_pkt.gang_global_priority_level =331 mes_add_queue_pkt.gang_global_priority_level =
315 input->gang_global_priority_level;332 convert_to_mes_priority_level(input->gang_global_priority_level);
316 mes_add_queue_pkt.doorbell_offset = input->doorbell_offset;333 mes_add_queue_pkt.doorbell_offset = input->doorbell_offset;
317 mes_add_queue_pkt.mqd_addr = input->mqd_addr;334 mes_add_queue_pkt.mqd_addr = input->mqd_addr;
318 335
@@ -458,31 +475,6 @@ static int mes_v11_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 139 unmodified lines
458 return r;475 return r;
459}476}
460 477
461static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes,
462 struct mes_reset_queue_input *input)
463{
464 if (input->use_mmio)
465 return mes_v11_0_reset_queue_mmio(mes, input->queue_type,
466 input->me_id, input->pipe_id,
467 input->queue_id, input->vmid);
468
469 union MESAPI__RESET mes_reset_queue_pkt;
470
471 memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));
472
473 mes_reset_queue_pkt.header.type = MES_API_TYPE_SCHEDULER;
474 mes_reset_queue_pkt.header.opcode = MES_SCH_API_RESET;
475 mes_reset_queue_pkt.header.dwsize = API_FRAME_SIZE_IN_DWORDS;
476
477 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
478 mes_reset_queue_pkt.gang_context_addr = input->gang_context_addr;
479 /*mes_reset_queue_pkt.reset_queue_only = 1;*/
480
481 return mes_v11_0_submit_pkt_and_poll_completion(mes,
482 &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt),
483 offsetof(union MESAPI__REMOVE_QUEUE, api_status));
484}
485
486static int mes_v11_0_map_legacy_queue(struct amdgpu_mes *mes,478static int mes_v11_0_map_legacy_queue(struct amdgpu_mes *mes,
487 struct mes_map_legacy_queue_input *input)479 struct mes_map_legacy_queue_input *input)
488{480{
@@ -649,7 +641,7 @@ static int mes_v11_0_misc_op(struct amdgpu_mes *mes, · 160 unmodified lines
649 break;641 break;
650 case MES_MISC_OP_CHANGE_CONFIG:642 case MES_MISC_OP_CHANGE_CONFIG:
651 if ((mes->adev->mes.sched_version & AMDGPU_MES_VERSION_MASK) < 0x63) {643 if ((mes->adev->mes.sched_version & AMDGPU_MES_VERSION_MASK) < 0x63) {
652 dev_err(mes->adev->dev, "MES FW versoin must be larger than 0x63 to support limit single process feature.\n");644 dev_err(mes->adev->dev, "MES FW version must be larger than 0x63 to support limit single process feature.\n");
653 return -EINVAL;645 return -EINVAL;
654 }646 }
655 misc_pkt.opcode = MESAPI_MISC__CHANGE_CONFIG;647 misc_pkt.opcode = MESAPI_MISC__CHANGE_CONFIG;
@@ -694,7 +686,8 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes) · 38 unmodified lines
694 mes->compute_hqd_mask[i];686 mes->compute_hqd_mask[i];
695 687
696 for (i = 0; i < MAX_GFX_PIPES; i++)688 for (i = 0; i < MAX_GFX_PIPES; i++)
697 mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i];689 mes_set_hw_res_pkt.gfx_hqd_mask[i] =
690 mes->gfx_hqd_mask[i];
698 691
699 for (i = 0; i < MAX_SDMA_PIPES; i++)692 for (i = 0; i < MAX_SDMA_PIPES; i++)
700 mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];693 mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i];
@@ -723,7 +716,7 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes) · 22 unmodified lines
723 mes->event_log_gpu_addr;716 mes->event_log_gpu_addr;
724 }717 }
725 718
726 if (enforce_isolation)719 if (adev->enforce_isolation[0] == AMDGPU_ENFORCE_ISOLATION_ENABLE)
727 mes_set_hw_res_pkt.limit_single_process = 1;720 mes_set_hw_res_pkt.limit_single_process = 1;
728 721
729 return mes_v11_0_submit_pkt_and_poll_completion(mes,722 return mes_v11_0_submit_pkt_and_poll_completion(mes,
@@ -753,8 +746,8 @@ static int mes_v11_0_set_hw_resources_1(struct amdgpu_mes *mes) · 23 unmodified lines
753 offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status));746 offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status));
754}747}
755 748
756static int mes_v11_0_reset_legacy_queue(struct amdgpu_mes *mes,749static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes,
757 struct mes_reset_legacy_queue_input *input)750 struct mes_reset_queue_input *input)
758{751{
759 union MESAPI__RESET mes_reset_queue_pkt;752 union MESAPI__RESET mes_reset_queue_pkt;
760 753
@@ -772,7 +765,7 @@ static int mes_v11_0_reset_legacy_queue(struct amdgpu_mes *mes, · 11 unmodified lines
772 mes_reset_queue_pkt.queue_type =765 mes_reset_queue_pkt.queue_type =
773 convert_to_mes_queue_type(input->queue_type);766 convert_to_mes_queue_type(input->queue_type);
774 767
775 if (mes_reset_queue_pkt.queue_type == MES_QUEUE_TYPE_GFX) {768 if (input->legacy_gfx) {
776 mes_reset_queue_pkt.reset_legacy_gfx = 1;769 mes_reset_queue_pkt.reset_legacy_gfx = 1;
777 mes_reset_queue_pkt.pipe_id_lp = input->pipe_id;770 mes_reset_queue_pkt.pipe_id_lp = input->pipe_id;
778 mes_reset_queue_pkt.queue_id_lp = input->queue_id;771 mes_reset_queue_pkt.queue_id_lp = input->queue_id;
@@ -798,7 +791,6 @@ static const struct amdgpu_mes_funcs mes_v11_0_funcs = { · 19 unmodified lines
798 .suspend_gang = mes_v11_0_suspend_gang,791 .suspend_gang = mes_v11_0_suspend_gang,
799 .resume_gang = mes_v11_0_resume_gang,792 .resume_gang = mes_v11_0_resume_gang,
800 .misc_op = mes_v11_0_misc_op,793 .misc_op = mes_v11_0_misc_op,
801 .reset_legacy_queue = mes_v11_0_reset_legacy_queue,
802 .reset_hw_queue = mes_v11_0_reset_hw_queue,794 .reset_hw_queue = mes_v11_0_reset_hw_queue,
803};795};
804 796
@@ -1701,22 +1693,10 @@ static int mes_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 896 unmodified lines
1701 return 0;1693 return 0;
1702}1694}
1703 1695
1704static int mes_v11_0_late_init(struct amdgpu_ip_block *ip_block)
1705{
1706 struct amdgpu_device *adev = ip_block->adev;
1707
1708 /* it's only intended for use in mes_self_test case, not for s0ix and reset */
1709 if (!amdgpu_in_reset(adev) && !adev->in_s0ix && !adev->in_suspend &&
1710 (amdgpu_ip_version(adev, GC_HWIP, 0) != IP_VERSION(11, 0, 3)))
1711 amdgpu_mes_self_test(adev);
1712
1713 return 0;
1714}
1715
1716static const struct amd_ip_funcs mes_v11_0_ip_funcs = {1696static const struct amd_ip_funcs mes_v11_0_ip_funcs = {
1717 .name = "mes_v11_0",1697 .name = "mes_v11_0",
1718 .early_init = mes_v11_0_early_init,1698 .early_init = mes_v11_0_early_init,
1719 .late_init = mes_v11_0_late_init,1699 .late_init = NULL,
1720 .sw_init = mes_v11_0_sw_init,1700 .sw_init = mes_v11_0_sw_init,
1721 .sw_fini = mes_v11_0_sw_fini,1701 .sw_fini = mes_v11_0_sw_fini,
1722 .hw_init = mes_v11_0_hw_init,1702 .hw_init = mes_v11_0_hw_init,
drivers/gpu/drm/amd/amdgpu/mes_v12_0.c+25−46
@@ -274,6 +274,23 @@ static int convert_to_mes_queue_type(int queue_type) · 273 unmodified lines
274 return -1;274 return -1;
275}275}
276 276
277static int convert_to_mes_priority_level(int priority_level)
278{
279 switch (priority_level) {
280 case AMDGPU_MES_PRIORITY_LEVEL_LOW:
281 return AMD_PRIORITY_LEVEL_LOW;
282 case AMDGPU_MES_PRIORITY_LEVEL_NORMAL:
283 default:
284 return AMD_PRIORITY_LEVEL_NORMAL;
285 case AMDGPU_MES_PRIORITY_LEVEL_MEDIUM:
286 return AMD_PRIORITY_LEVEL_MEDIUM;
287 case AMDGPU_MES_PRIORITY_LEVEL_HIGH:
288 return AMD_PRIORITY_LEVEL_HIGH;
289 case AMDGPU_MES_PRIORITY_LEVEL_REALTIME:
290 return AMD_PRIORITY_LEVEL_REALTIME;
291 }
292}
293
277static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes,294static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes,
278 struct mes_add_queue_input *input)295 struct mes_add_queue_input *input)
279{296{
@@ -297,9 +314,9 @@ static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines
297 mes_add_queue_pkt.gang_quantum = input->gang_quantum;314 mes_add_queue_pkt.gang_quantum = input->gang_quantum;
298 mes_add_queue_pkt.gang_context_addr = input->gang_context_addr;315 mes_add_queue_pkt.gang_context_addr = input->gang_context_addr;
299 mes_add_queue_pkt.inprocess_gang_priority =316 mes_add_queue_pkt.inprocess_gang_priority =
300 input->inprocess_gang_priority;317 convert_to_mes_priority_level(input->inprocess_gang_priority);
301 mes_add_queue_pkt.gang_global_priority_level =318 mes_add_queue_pkt.gang_global_priority_level =
302 input->gang_global_priority_level;319 convert_to_mes_priority_level(input->gang_global_priority_level);
303 mes_add_queue_pkt.doorbell_offset = input->doorbell_offset;320 mes_add_queue_pkt.doorbell_offset = input->doorbell_offset;
304 mes_add_queue_pkt.mqd_addr = input->mqd_addr;321 mes_add_queue_pkt.mqd_addr = input->mqd_addr;
305 322
@@ -477,32 +494,6 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 171 unmodified lines
477 return r;494 return r;
478}495}
479 496
480static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes,
481 struct mes_reset_queue_input *input)
482{
483 union MESAPI__RESET mes_reset_queue_pkt;
484 int pipe;
485
486 memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));
487
488 mes_reset_queue_pkt.header.type = MES_API_TYPE_SCHEDULER;
489 mes_reset_queue_pkt.header.opcode = MES_SCH_API_RESET;
490 mes_reset_queue_pkt.header.dwsize = API_FRAME_SIZE_IN_DWORDS;
491
492 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
493 mes_reset_queue_pkt.gang_context_addr = input->gang_context_addr;
494 /*mes_reset_queue_pkt.reset_queue_only = 1;*/
495
496 if (mes->adev->enable_uni_mes)
497 pipe = AMDGPU_MES_KIQ_PIPE;
498 else
499 pipe = AMDGPU_MES_SCHED_PIPE;
500
501 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
502 &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt),
503 offsetof(union MESAPI__REMOVE_QUEUE, api_status));
504}
505
506static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes,497static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes,
507 struct mes_map_legacy_queue_input *input)498 struct mes_map_legacy_queue_input *input)
508{499{
@@ -762,7 +753,7 @@ static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe) · 253 unmodified lines
762 pipe * (AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE);753 pipe * (AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE);
763 }754 }
764 755
765 if (enforce_isolation)756 if (adev->enforce_isolation[0] == AMDGPU_ENFORCE_ISOLATION_ENABLE)
766 mes_set_hw_res_pkt.limit_single_process = 1;757 mes_set_hw_res_pkt.limit_single_process = 1;
767 758
768 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,759 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
@@ -845,8 +836,8 @@ static void mes_v12_0_enable_unmapped_doorbell_handling( · 76 unmodified lines
845 WREG32_SOC15(GC, 0, regCP_UNMAPPED_DOORBELL, data);836 WREG32_SOC15(GC, 0, regCP_UNMAPPED_DOORBELL, data);
846}837}
847 838
848static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes,839static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes,
849 struct mes_reset_legacy_queue_input *input)840 struct mes_reset_queue_input *input)
850{841{
851 union MESAPI__RESET mes_reset_queue_pkt;842 union MESAPI__RESET mes_reset_queue_pkt;
852 int pipe;843 int pipe;
@@ -865,7 +856,7 @@ static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, · 12 unmodified lines
865 mes_reset_queue_pkt.queue_type =856 mes_reset_queue_pkt.queue_type =
866 convert_to_mes_queue_type(input->queue_type);857 convert_to_mes_queue_type(input->queue_type);
867 858
868 if (mes_reset_queue_pkt.queue_type == MES_QUEUE_TYPE_GFX) {859 if (input->legacy_gfx) {
869 mes_reset_queue_pkt.reset_legacy_gfx = 1;860 mes_reset_queue_pkt.reset_legacy_gfx = 1;
870 mes_reset_queue_pkt.pipe_id_lp = input->pipe_id;861 mes_reset_queue_pkt.pipe_id_lp = input->pipe_id;
871 mes_reset_queue_pkt.queue_id_lp = input->queue_id;862 mes_reset_queue_pkt.queue_id_lp = input->queue_id;
@@ -878,7 +869,7 @@ static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, · 6 unmodified lines
878 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;869 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
879 }870 }
880 871
881 if (mes->adev->enable_uni_mes)872 if (input->is_kq)
882 pipe = AMDGPU_MES_KIQ_PIPE;873 pipe = AMDGPU_MES_KIQ_PIPE;
883 else874 else
884 pipe = AMDGPU_MES_SCHED_PIPE;875 pipe = AMDGPU_MES_SCHED_PIPE;
@@ -896,7 +887,6 @@ static const struct amdgpu_mes_funcs mes_v12_0_funcs = { · 11 unmodified lines
896 .suspend_gang = mes_v12_0_suspend_gang,887 .suspend_gang = mes_v12_0_suspend_gang,
897 .resume_gang = mes_v12_0_resume_gang,888 .resume_gang = mes_v12_0_resume_gang,
898 .misc_op = mes_v12_0_misc_op,889 .misc_op = mes_v12_0_misc_op,
899 .reset_legacy_queue = mes_v12_0_reset_legacy_queue,
900 .reset_hw_queue = mes_v12_0_reset_hw_queue,890 .reset_hw_queue = mes_v12_0_reset_hw_queue,
901};891};
902 892
@@ -1811,21 +1801,10 @@ static int mes_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 908 unmodified lines
1811 return 0;1801 return 0;
1812}1802}
1813 1803
1814static int mes_v12_0_late_init(struct amdgpu_ip_block *ip_block)
1815{
1816 struct amdgpu_device *adev = ip_block->adev;
1817
1818 /* it's only intended for use in mes_self_test case, not for s0ix and reset */
1819 if (!amdgpu_in_reset(adev) && !adev->in_s0ix && !adev->in_suspend)
1820 amdgpu_mes_self_test(adev);
1821
1822 return 0;
1823}
1824
1825static const struct amd_ip_funcs mes_v12_0_ip_funcs = {1804static const struct amd_ip_funcs mes_v12_0_ip_funcs = {
1826 .name = "mes_v12_0",1805 .name = "mes_v12_0",
1827 .early_init = mes_v12_0_early_init,1806 .early_init = mes_v12_0_early_init,
1828 .late_init = mes_v12_0_late_init,1807 .late_init = NULL,
1829 .sw_init = mes_v12_0_sw_init,1808 .sw_init = mes_v12_0_sw_init,
1830 .sw_fini = mes_v12_0_sw_fini,1809 .sw_fini = mes_v12_0_sw_fini,
1831 .hw_init = mes_v12_0_hw_init,1810 .hw_init = mes_v12_0_hw_init,
drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c+52−14
@@ -30,6 +30,7 @@ · 29 unmodified lines
30#include "soc15_common.h"30#include "soc15_common.h"
31#include "soc15.h"31#include "soc15.h"
32#include "amdgpu_ras.h"32#include "amdgpu_ras.h"
33#include "amdgpu_psp.h"
33 34
34#define regVM_L2_CNTL3_DEFAULT 0x8010000735#define regVM_L2_CNTL3_DEFAULT 0x80100007
35#define regVM_L2_CNTL4_DEFAULT 0x000000c136#define regVM_L2_CNTL4_DEFAULT 0x000000c1
@@ -192,10 +193,8 @@ static void mmhub_v1_8_init_tlb_regs(struct amdgpu_device *adev) · 156 unmodified lines
192 uint32_t tmp, inst_mask;193 uint32_t tmp, inst_mask;
193 int i;194 int i;
194 195
195 /* Setup TLB control */196 if (amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev)) {
196 inst_mask = adev->aid_mask;197 tmp = RREG32_SOC15(MMHUB, 0, regMC_VM_MX_L1_TLB_CNTL);
197 for_each_inst(i, inst_mask) {
198 tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL);
199 198
200 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,199 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
201 1);200 1);
@@ -209,7 +208,26 @@ static void mmhub_v1_8_init_tlb_regs(struct amdgpu_device *adev) · 7 unmodified lines
209 MTYPE, MTYPE_UC);/* XXX for emulation. */208 MTYPE, MTYPE_UC);/* XXX for emulation. */
210 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1);209 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1);
211 210
212 WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp);211 psp_reg_program_no_ring(&adev->psp, tmp, PSP_REG_MMHUB_L1_TLB_CNTL);
212 } else {
213 inst_mask = adev->aid_mask;
214 for_each_inst(i, inst_mask) {
215 tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL);
216
217 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
218 1);
219 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
220 SYSTEM_ACCESS_MODE, 3);
221 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
222 ENABLE_ADVANCED_DRIVER_MODEL, 1);
223 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
224 SYSTEM_APERTURE_UNMAPPED_ACCESS, 0);
225 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
226 MTYPE, MTYPE_UC);/* XXX for emulation. */
227 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1);
228
229 WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp);
230 }
213 }231 }
214}232}
215 233
@@ -221,6 +239,9 @@ static void mmhub_v1_8_init_snoop_override_regs(struct amdgpu_device *adev) · 5 unmodified lines
221 uint32_t distance = regDAGB1_WRCLI_GPU_SNOOP_OVERRIDE -239 uint32_t distance = regDAGB1_WRCLI_GPU_SNOOP_OVERRIDE -
222 regDAGB0_WRCLI_GPU_SNOOP_OVERRIDE;240 regDAGB0_WRCLI_GPU_SNOOP_OVERRIDE;
223 241
242 if (amdgpu_sriov_vf(adev))
243 return;
244
224 inst_mask = adev->aid_mask;245 inst_mask = adev->aid_mask;
225 for_each_inst(i, inst_mask) {246 for_each_inst(i, inst_mask) {
226 for (j = 0; j < 5; j++) { /* DAGB instances */247 for (j = 0; j < 5; j++) { /* DAGB instances */
@@ -454,6 +475,30 @@ static int mmhub_v1_8_gart_enable(struct amdgpu_device *adev) · 227 unmodified lines
454 return 0;475 return 0;
455}476}
456 477
478static void mmhub_v1_8_disable_l1_tlb(struct amdgpu_device *adev)
479{
480 u32 tmp;
481 u32 i, inst_mask;
482
483 if (amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev)) {
484 tmp = RREG32_SOC15(MMHUB, 0, regMC_VM_MX_L1_TLB_CNTL);
485 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, 0);
486 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
487 ENABLE_ADVANCED_DRIVER_MODEL, 0);
488 psp_reg_program_no_ring(&adev->psp, tmp, PSP_REG_MMHUB_L1_TLB_CNTL);
489 } else {
490 inst_mask = adev->aid_mask;
491 for_each_inst(i, inst_mask) {
492 tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL);
493 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
494 0);
495 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
496 ENABLE_ADVANCED_DRIVER_MODEL, 0);
497 WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp);
498 }
499 }
500}
501
457static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev)502static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev)
458{503{
459 struct amdgpu_vmhub *hub;504 struct amdgpu_vmhub *hub;
@@ -467,15 +512,6 @@ static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) · 7 unmodified lines
467 for (i = 0; i < 16; i++)512 for (i = 0; i < 16; i++)
468 WREG32_SOC15_OFFSET(MMHUB, j, regVM_CONTEXT0_CNTL,513 WREG32_SOC15_OFFSET(MMHUB, j, regVM_CONTEXT0_CNTL,
469 i * hub->ctx_distance, 0);514 i * hub->ctx_distance, 0);
470
471 /* Setup TLB control */
472 tmp = RREG32_SOC15(MMHUB, j, regMC_VM_MX_L1_TLB_CNTL);
473 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB,
474 0);
475 tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL,
476 ENABLE_ADVANCED_DRIVER_MODEL, 0);
477 WREG32_SOC15(MMHUB, j, regMC_VM_MX_L1_TLB_CNTL, tmp);
478
479 if (!amdgpu_sriov_vf(adev)) {515 if (!amdgpu_sriov_vf(adev)) {
480 /* Setup L2 cache */516 /* Setup L2 cache */
481 tmp = RREG32_SOC15(MMHUB, j, regVM_L2_CNTL);517 tmp = RREG32_SOC15(MMHUB, j, regVM_L2_CNTL);
@@ -485,6 +521,8 @@ static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) · 3 unmodified lines
485 WREG32_SOC15(MMHUB, j, regVM_L2_CNTL3, 0);521 WREG32_SOC15(MMHUB, j, regVM_L2_CNTL3, 0);
486 }522 }
487 }523 }
524
525 mmhub_v1_8_disable_l1_tlb(adev);
488}526}
489 527
490/**528/**
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c+46−17
@@ -274,6 +274,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 273 unmodified lines
274{274{
275 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work);275 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work);
276 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);276 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
277 struct amdgpu_reset_context reset_context = { 0 };
277 278
278 amdgpu_virt_fini_data_exchange(adev);279 amdgpu_virt_fini_data_exchange(adev);
279 280
@@ -281,8 +282,6 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 1 unmodified line
281 if (amdgpu_device_should_recover_gpu(adev)282 if (amdgpu_device_should_recover_gpu(adev)
282 && (!amdgpu_device_has_job_running(adev) ||283 && (!amdgpu_device_has_job_running(adev) ||
283 adev->sdma_timeout == MAX_SCHEDULE_TIMEOUT)) {284 adev->sdma_timeout == MAX_SCHEDULE_TIMEOUT)) {
284 struct amdgpu_reset_context reset_context;
285 memset(&reset_context, 0, sizeof(reset_context));
286 285
287 reset_context.method = AMD_RESET_METHOD_NONE;286 reset_context.method = AMD_RESET_METHOD_NONE;
288 reset_context.reset_req_dev = adev;287 reset_context.reset_req_dev = adev;
@@ -293,6 +292,19 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines
293 }292 }
294}293}
295 294
295static void xgpu_ai_mailbox_bad_pages_work(struct work_struct *work)
296{
297 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, bad_pages_work);
298 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
299
300 if (down_read_trylock(&adev->reset_domain->sem)) {
301 amdgpu_virt_fini_data_exchange(adev);
302 amdgpu_virt_request_bad_pages(adev);
303 amdgpu_virt_init_data_exchange(adev);
304 up_read(&adev->reset_domain->sem);
305 }
306}
307
296static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,308static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
297 struct amdgpu_irq_src *src,309 struct amdgpu_irq_src *src,
298 unsigned type,310 unsigned type,
@@ -312,26 +324,42 @@ static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev, · 13 unmodified lines
312 struct amdgpu_iv_entry *entry)324 struct amdgpu_iv_entry *entry)
313{325{
314 enum idh_event event = xgpu_ai_mailbox_peek_msg(adev);326 enum idh_event event = xgpu_ai_mailbox_peek_msg(adev);
327 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
315 328
316 switch (event) {329 switch (event) {
317 case IDH_FLR_NOTIFICATION:330 case IDH_RAS_BAD_PAGES_NOTIFICATION:
331 xgpu_ai_mailbox_send_ack(adev);
332 if (amdgpu_sriov_runtime(adev))
333 schedule_work(&adev->virt.bad_pages_work);
334 break;
335 case IDH_UNRECOV_ERR_NOTIFICATION:
336 xgpu_ai_mailbox_send_ack(adev);
337 ras->is_rma = true;
338 dev_err(adev->dev, "VF is in an unrecoverable state. Runtime Services are halted.\n");
318 if (amdgpu_sriov_runtime(adev))339 if (amdgpu_sriov_runtime(adev))
319 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,340 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
320 &adev->virt.flr_work),341 &adev->virt.flr_work),
321 "Failed to queue work! at %s",342 "Failed to queue work! at %s",
322 __func__);343 __func__);
323 break;344 break;
324 case IDH_QUERY_ALIVE:345 case IDH_FLR_NOTIFICATION:
325 xgpu_ai_mailbox_send_ack(adev);346 if (amdgpu_sriov_runtime(adev))
326 break;347 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
327 /* READY_TO_ACCESS_GPU is fetched by kernel polling, IRQ can ignore348 &adev->virt.flr_work),
328 * it byfar since that polling thread will handle it,349 "Failed to queue work! at %s",
329 * other msg like flr complete is not handled here.350 __func__);
330 */351 break;
331 case IDH_CLR_MSG_BUF:352 case IDH_QUERY_ALIVE:
332 case IDH_FLR_NOTIFICATION_CMPL:353 xgpu_ai_mailbox_send_ack(adev);
333 case IDH_READY_TO_ACCESS_GPU:354 break;
334 default:355 /* READY_TO_ACCESS_GPU is fetched by kernel polling, IRQ can ignore
356 * it byfar since that polling thread will handle it,
357 * other msg like flr complete is not handled here.
358 */
359 case IDH_CLR_MSG_BUF:
360 case IDH_FLR_NOTIFICATION_CMPL:
361 case IDH_READY_TO_ACCESS_GPU:
362 default:
335 break;363 break;
336 }364 }
337 365
@@ -387,6 +415,7 @@ int xgpu_ai_mailbox_get_irq(struct amdgpu_device *adev) · 49 unmodified lines
387 }415 }
388 416
389 INIT_WORK(&adev->virt.flr_work, xgpu_ai_mailbox_flr_work);417 INIT_WORK(&adev->virt.flr_work, xgpu_ai_mailbox_flr_work);
418 INIT_WORK(&adev->virt.bad_pages_work, xgpu_ai_mailbox_bad_pages_work);
390 419
391 return 0;420 return 0;
392}421}
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.h+4−0
@@ -40,6 +40,7 @@ enum idh_request { · 39 unmodified lines
40 IDH_LOG_VF_ERROR = 200,40 IDH_LOG_VF_ERROR = 200,
41 IDH_READY_TO_RESET = 201,41 IDH_READY_TO_RESET = 201,
42 IDH_RAS_POISON = 202,42 IDH_RAS_POISON = 202,
43 IDH_REQ_RAS_BAD_PAGES = 205,
43};44};
44 45
45enum idh_event {46enum idh_event {
@@ -54,6 +55,9 @@ enum idh_event { · 8 unmodified lines
54 IDH_RAS_POISON_READY,55 IDH_RAS_POISON_READY,
55 IDH_PF_SOFT_FLR_NOTIFICATION,56 IDH_PF_SOFT_FLR_NOTIFICATION,
56 IDH_RAS_ERROR_DETECTED,57 IDH_RAS_ERROR_DETECTED,
58 IDH_RAS_BAD_PAGES_READY = 15,
59 IDH_RAS_BAD_PAGES_NOTIFICATION = 16,
60 IDH_UNRECOV_ERR_NOTIFICATION = 17,
57 IDH_TEXT_MESSAGE = 255,61 IDH_TEXT_MESSAGE = 255,
58};62};
59 63
drivers/gpu/drm/amd/amdgpu/mxgpu_nv.c+59−3
@@ -67,6 +67,8 @@ static int xgpu_nv_mailbox_rcv_msg(struct amdgpu_device *adev, · 66 unmodified lines
67 reg = RREG32_NO_KIQ(mmMAILBOX_MSGBUF_RCV_DW0);67 reg = RREG32_NO_KIQ(mmMAILBOX_MSGBUF_RCV_DW0);
68 if (reg == IDH_FAIL)68 if (reg == IDH_FAIL)
69 r = -EINVAL;69 r = -EINVAL;
70 if (reg == IDH_UNRECOV_ERR_NOTIFICATION)
71 r = -ENODEV;
70 else if (reg != event)72 else if (reg != event)
71 return -ENOENT;73 return -ENOENT;
72 74
@@ -103,6 +105,7 @@ static int xgpu_nv_poll_msg(struct amdgpu_device *adev, enum idh_event event) · 30 unmodified lines
103{105{
104 int r;106 int r;
105 uint64_t timeout, now;107 uint64_t timeout, now;
108 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
106 109
107 now = (uint64_t)ktime_to_ms(ktime_get());110 now = (uint64_t)ktime_to_ms(ktime_get());
108 timeout = now + NV_MAILBOX_POLL_MSG_TIMEDOUT;111 timeout = now + NV_MAILBOX_POLL_MSG_TIMEDOUT;
@@ -110,8 +113,16 @@ static int xgpu_nv_poll_msg(struct amdgpu_device *adev, enum idh_event event) · 1 unmodified line
110 do {113 do {
111 r = xgpu_nv_mailbox_rcv_msg(adev, event);114 r = xgpu_nv_mailbox_rcv_msg(adev, event);
112 if (!r) {115 if (!r) {
113 dev_dbg(adev->dev, "rcv_msg 0x%x after %llu ms\n", event, NV_MAILBOX_POLL_MSG_TIMEDOUT - timeout + now);116 dev_dbg(adev->dev, "rcv_msg 0x%x after %llu ms\n",
117 event, NV_MAILBOX_POLL_MSG_TIMEDOUT - timeout + now);
114 return 0;118 return 0;
119 } else if (r == -ENODEV) {
120 if (!amdgpu_ras_is_rma(adev)) {
121 ras->is_rma = true;
122 dev_err(adev->dev, "VF is in an unrecoverable state. "
123 "Runtime Services are halted.\n");
124 }
125 return r;
115 }126 }
116 127
117 msleep(10);128 msleep(10);
@@ -166,6 +177,10 @@ static int xgpu_nv_send_access_requests_with_param(struct amdgpu_device *adev, · 48 unmodified lines
166 enum idh_event event = -1;177 enum idh_event event = -1;
167 178
168send_request:179send_request:
180
181 if (amdgpu_ras_is_rma(adev))
182 return -ENODEV;
183
169 xgpu_nv_mailbox_trans_msg(adev, req, data1, data2, data3);184 xgpu_nv_mailbox_trans_msg(adev, req, data1, data2, data3);
170 185
171 switch (req) {186 switch (req) {
@@ -187,6 +202,9 @@ send_request: · 15 unmodified lines
187 case IDH_REQ_RAS_CPER_DUMP:202 case IDH_REQ_RAS_CPER_DUMP:
188 event = IDH_RAS_CPER_DUMP_READY;203 event = IDH_RAS_CPER_DUMP_READY;
189 break;204 break;
205 case IDH_REQ_RAS_BAD_PAGES:
206 event = IDH_RAS_BAD_PAGES_READY;
207 break;
190 default:208 default:
191 break;209 break;
192 }210 }
@@ -320,6 +338,7 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 127 unmodified lines
320{338{
321 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work);339 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work);
322 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);340 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
341 struct amdgpu_reset_context reset_context = { 0 };
323 342
324 amdgpu_virt_fini_data_exchange(adev);343 amdgpu_virt_fini_data_exchange(adev);
325 344
@@ -330,8 +349,6 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines
330 adev->gfx_timeout == MAX_SCHEDULE_TIMEOUT ||349 adev->gfx_timeout == MAX_SCHEDULE_TIMEOUT ||
331 adev->compute_timeout == MAX_SCHEDULE_TIMEOUT ||350 adev->compute_timeout == MAX_SCHEDULE_TIMEOUT ||
332 adev->video_timeout == MAX_SCHEDULE_TIMEOUT)) {351 adev->video_timeout == MAX_SCHEDULE_TIMEOUT)) {
333 struct amdgpu_reset_context reset_context;
334 memset(&reset_context, 0, sizeof(reset_context));
335 352
336 reset_context.method = AMD_RESET_METHOD_NONE;353 reset_context.method = AMD_RESET_METHOD_NONE;
337 reset_context.reset_req_dev = adev;354 reset_context.reset_req_dev = adev;
@@ -342,6 +359,19 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines
342 }359 }
343}360}
344 361
362static void xgpu_nv_mailbox_bad_pages_work(struct work_struct *work)
363{
364 struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, bad_pages_work);
365 struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt);
366
367 if (down_read_trylock(&adev->reset_domain->sem)) {
368 amdgpu_virt_fini_data_exchange(adev);
369 amdgpu_virt_request_bad_pages(adev);
370 amdgpu_virt_init_data_exchange(adev);
371 up_read(&adev->reset_domain->sem);
372 }
373}
374
345static int xgpu_nv_set_mailbox_rcv_irq(struct amdgpu_device *adev,375static int xgpu_nv_set_mailbox_rcv_irq(struct amdgpu_device *adev,
346 struct amdgpu_irq_src *src,376 struct amdgpu_irq_src *src,
347 unsigned type,377 unsigned type,
@@ -364,8 +394,27 @@ static int xgpu_nv_mailbox_rcv_irq(struct amdgpu_device *adev, · 16 unmodified lines
364 struct amdgpu_iv_entry *entry)394 struct amdgpu_iv_entry *entry)
365{395{
366 enum idh_event event = xgpu_nv_mailbox_peek_msg(adev);396 enum idh_event event = xgpu_nv_mailbox_peek_msg(adev);
397 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
367 398
368 switch (event) {399 switch (event) {
400 case IDH_RAS_BAD_PAGES_NOTIFICATION:
401 xgpu_nv_mailbox_send_ack(adev);
402 if (amdgpu_sriov_runtime(adev))
403 schedule_work(&adev->virt.bad_pages_work);
404 break;
405 case IDH_UNRECOV_ERR_NOTIFICATION:
406 xgpu_nv_mailbox_send_ack(adev);
407 if (!amdgpu_ras_is_rma(adev)) {
408 ras->is_rma = true;
409 dev_err(adev->dev, "VF is in an unrecoverable state. Runtime Services are halted.\n");
410 }
411
412 if (amdgpu_sriov_runtime(adev))
413 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
414 &adev->virt.flr_work),
415 "Failed to queue work! at %s",
416 __func__);
417 break;
369 case IDH_FLR_NOTIFICATION:418 case IDH_FLR_NOTIFICATION:
370 if (amdgpu_sriov_runtime(adev))419 if (amdgpu_sriov_runtime(adev))
371 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,420 WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain,
@@ -436,6 +485,7 @@ int xgpu_nv_mailbox_get_irq(struct amdgpu_device *adev) · 64 unmodified lines
436 }485 }
437 486
438 INIT_WORK(&adev->virt.flr_work, xgpu_nv_mailbox_flr_work);487 INIT_WORK(&adev->virt.flr_work, xgpu_nv_mailbox_flr_work);
488 INIT_WORK(&adev->virt.bad_pages_work, xgpu_nv_mailbox_bad_pages_work);
439 489
440 return 0;490 return 0;
441}491}
@@ -480,6 +530,11 @@ static int xgpu_nv_req_ras_cper_dump(struct amdgpu_device *adev, u64 vf_rptr) · 38 unmodified lines
480 adev, IDH_REQ_RAS_CPER_DUMP, vf_rptr_hi, vf_rptr_lo, 0);530 adev, IDH_REQ_RAS_CPER_DUMP, vf_rptr_hi, vf_rptr_lo, 0);
481}531}
482 532
533static int xgpu_nv_req_ras_bad_pages(struct amdgpu_device *adev)
534{
535 return xgpu_nv_send_access_requests(adev, IDH_REQ_RAS_BAD_PAGES);
536}
537
483const struct amdgpu_virt_ops xgpu_nv_virt_ops = {538const struct amdgpu_virt_ops xgpu_nv_virt_ops = {
484 .req_full_gpu = xgpu_nv_request_full_gpu_access,539 .req_full_gpu = xgpu_nv_request_full_gpu_access,
485 .rel_full_gpu = xgpu_nv_release_full_gpu_access,540 .rel_full_gpu = xgpu_nv_release_full_gpu_access,
@@ -492,4 +547,5 @@ const struct amdgpu_virt_ops xgpu_nv_virt_ops = { · 6 unmodified lines
492 .rcvd_ras_intr = xgpu_nv_rcvd_ras_intr,547 .rcvd_ras_intr = xgpu_nv_rcvd_ras_intr,
493 .req_ras_err_count = xgpu_nv_req_ras_err_count,548 .req_ras_err_count = xgpu_nv_req_ras_err_count,
494 .req_ras_cper_dump = xgpu_nv_req_ras_cper_dump,549 .req_ras_cper_dump = xgpu_nv_req_ras_cper_dump,
550 .req_bad_pages = xgpu_nv_req_ras_bad_pages,
495};551};
drivers/gpu/drm/amd/amdgpu/mxgpu_nv.h+4−0
@@ -42,6 +42,7 @@ enum idh_request { · 41 unmodified lines
42 IDH_RAS_POISON = 202,42 IDH_RAS_POISON = 202,
43 IDH_REQ_RAS_ERROR_COUNT = 203,43 IDH_REQ_RAS_ERROR_COUNT = 203,
44 IDH_REQ_RAS_CPER_DUMP = 204,44 IDH_REQ_RAS_CPER_DUMP = 204,
45 IDH_REQ_RAS_BAD_PAGES = 205,
45};46};
46 47
47enum idh_event {48enum idh_event {
@@ -58,6 +59,9 @@ enum idh_event { · 10 unmodified lines
58 IDH_RAS_ERROR_DETECTED,59 IDH_RAS_ERROR_DETECTED,
59 IDH_RAS_ERROR_COUNT_READY = 11,60 IDH_RAS_ERROR_COUNT_READY = 11,
60 IDH_RAS_CPER_DUMP_READY = 14,61 IDH_RAS_CPER_DUMP_READY = 14,
62 IDH_RAS_BAD_PAGES_READY = 15,
63 IDH_RAS_BAD_PAGES_NOTIFICATION = 16,
64 IDH_UNRECOV_ERR_NOTIFICATION = 17,
61 65
62 IDH_TEXT_MESSAGE = 255,66 IDH_TEXT_MESSAGE = 255,
63};67};
drivers/gpu/drm/amd/amdgpu/nbio_v7_9.c+6−2
@@ -177,8 +177,12 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 176 unmodified lines
177{177{
178 u32 doorbell_range = 0, doorbell_ctrl = 0;178 u32 doorbell_range = 0, doorbell_ctrl = 0;
179 u32 aid_id = instance;179 u32 aid_id = instance;
180 u32 range_size;
180 181
181 if (use_doorbell) {182 if (use_doorbell) {
183 range_size = (amdgpu_ip_version(adev, GC_HWIP, 0) ==
184 IP_VERSION(9, 5, 0)) ?
185 0xb : 0x9;
182 doorbell_range = REG_SET_FIELD(doorbell_range,186 doorbell_range = REG_SET_FIELD(doorbell_range,
183 DOORBELL0_CTRL_ENTRY_0,187 DOORBELL0_CTRL_ENTRY_0,
184 BIF_DOORBELL0_RANGE_OFFSET_ENTRY,188 BIF_DOORBELL0_RANGE_OFFSET_ENTRY,
@@ -186,7 +190,7 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 1 unmodified line
186 doorbell_range = REG_SET_FIELD(doorbell_range,190 doorbell_range = REG_SET_FIELD(doorbell_range,
187 DOORBELL0_CTRL_ENTRY_0,191 DOORBELL0_CTRL_ENTRY_0,
188 BIF_DOORBELL0_RANGE_SIZE_ENTRY,192 BIF_DOORBELL0_RANGE_SIZE_ENTRY,
189 0x9);193 range_size);
190 if (aid_id)194 if (aid_id)
191 doorbell_range = REG_SET_FIELD(doorbell_range,195 doorbell_range = REG_SET_FIELD(doorbell_range,
192 DOORBELL0_CTRL_ENTRY_0,196 DOORBELL0_CTRL_ENTRY_0,
@@ -204,7 +208,7 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 11 unmodified lines
204 S2A_DOORBELL_PORT1_RANGE_OFFSET, 0x4);208 S2A_DOORBELL_PORT1_RANGE_OFFSET, 0x4);
205 doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl,209 doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl,
206 S2A_DOORBELL_ENTRY_1_CTRL,210 S2A_DOORBELL_ENTRY_1_CTRL,
207 S2A_DOORBELL_PORT1_RANGE_SIZE, 0x9);211 S2A_DOORBELL_PORT1_RANGE_SIZE, range_size);
208 doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl,212 doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl,
209 S2A_DOORBELL_ENTRY_1_CTRL,213 S2A_DOORBELL_ENTRY_1_CTRL,
210 S2A_DOORBELL_PORT1_AWADDR_31_28_VALUE, 0x4);214 S2A_DOORBELL_PORT1_AWADDR_31_28_VALUE, 0x4);
drivers/gpu/drm/amd/amdgpu/psp_v12_0.c+17−85
@@ -34,9 +34,6 @@ · 33 unmodified lines
34#include "sdma0/sdma0_4_0_offset.h"34#include "sdma0/sdma0_4_0_offset.h"
35#include "nbio/nbio_7_4_offset.h"35#include "nbio/nbio_7_4_offset.h"
36 36
37#include "oss/osssys_4_0_offset.h"
38#include "oss/osssys_4_0_sh_mask.h"
39
40MODULE_FIRMWARE("amdgpu/renoir_asd.bin");37MODULE_FIRMWARE("amdgpu/renoir_asd.bin");
41MODULE_FIRMWARE("amdgpu/renoir_ta.bin");38MODULE_FIRMWARE("amdgpu/renoir_ta.bin");
42MODULE_FIRMWARE("amdgpu/green_sardine_asd.bin");39MODULE_FIRMWARE("amdgpu/green_sardine_asd.bin");
@@ -99,9 +96,6 @@ static int psp_v12_0_bootloader_load_sysdrv(struct psp_context *psp) · 56 unmodified lines
99 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35,96 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35,
100 psp_gfxdrv_command_reg);97 psp_gfxdrv_command_reg);
101 98
102 /* there might be handshake issue with hardware which needs delay */
103 mdelay(20);
104
105 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_35),99 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_35),
106 0x80000000, 0x80000000, false);100 0x80000000, 0x80000000, false);
107 101
@@ -138,8 +132,6 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) · 30 unmodified lines
138 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35,132 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35,
139 psp_gfxdrv_command_reg);133 psp_gfxdrv_command_reg);
140 134
141 /* there might be handshake issue with hardware which needs delay */
142 mdelay(20);
143 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_81),135 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_81),
144 RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_81),136 RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_81),
145 0, true);137 0, true);
@@ -147,37 +139,6 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) · 1 unmodified line
147 return ret;139 return ret;
148}140}
149 141
150static void psp_v12_0_reroute_ih(struct psp_context *psp)
151{
152 struct amdgpu_device *adev = psp->adev;
153 uint32_t tmp;
154
155 /* Change IH ring for VMC */
156 tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1244b);
157 tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, CLIENT_TYPE, 1);
158 tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
159
160 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 3);
161 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
162 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
163
164 mdelay(20);
165 psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
166 0x80000000, 0x8000FFFF, false);
167
168 /* Change IH ring for UMC */
169 tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1216b);
170 tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1);
171
172 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 4);
173 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp);
174 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET);
175
176 mdelay(20);
177 psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
178 0x80000000, 0x8000FFFF, false);
179}
180
181static int psp_v12_0_ring_create(struct psp_context *psp,142static int psp_v12_0_ring_create(struct psp_context *psp,
182 enum psp_ring_type ring_type)143 enum psp_ring_type ring_type)
183{144{
@@ -186,49 +147,23 @@ static int psp_v12_0_ring_create(struct psp_context *psp, · 2 unmodified lines
186 struct psp_ring *ring = &psp->km_ring;147 struct psp_ring *ring = &psp->km_ring;
187 struct amdgpu_device *adev = psp->adev;148 struct amdgpu_device *adev = psp->adev;
188 149
189 psp_v12_0_reroute_ih(psp);150 /* Write low address of the ring to C2PMSG_69 */
190 151 psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);
191 if (amdgpu_sriov_vf(psp->adev)) {152 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg);
192 /* Write low address of the ring to C2PMSG_102 */153 /* Write high address of the ring to C2PMSG_70 */
193 psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);154 psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr);
194 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_102, psp_ring_reg);155 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, psp_ring_reg);
195 /* Write high address of the ring to C2PMSG_103 */156 /* Write size of ring to C2PMSG_71 */
196 psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr);157 psp_ring_reg = ring->ring_size;
197 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_103, psp_ring_reg);158 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_71, psp_ring_reg);
198 159 /* Write the ring initialization command to C2PMSG_64 */
199 /* Write the ring initialization command to C2PMSG_101 */160 psp_ring_reg = ring_type;
200 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_101,161 psp_ring_reg = psp_ring_reg << 16;
201 GFX_CTRL_CMD_ID_INIT_GPCOM_RING);162 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, psp_ring_reg);
202 163
203 /* there might be handshake issue with hardware which needs delay */164 /* Wait for response flag (bit 31) in C2PMSG_64 */
204 mdelay(20);165 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
205 166 0x80000000, 0x8000FFFF, false);
206 /* Wait for response flag (bit 31) in C2PMSG_101 */
207 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101),
208 0x80000000, 0x8000FFFF, false);
209
210 } else {
211 /* Write low address of the ring to C2PMSG_69 */
212 psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr);
213 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg);
214 /* Write high address of the ring to C2PMSG_70 */
215 psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr);
216 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, psp_ring_reg);
217 /* Write size of ring to C2PMSG_71 */
218 psp_ring_reg = ring->ring_size;
219 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_71, psp_ring_reg);
220 /* Write the ring initialization command to C2PMSG_64 */
221 psp_ring_reg = ring_type;
222 psp_ring_reg = psp_ring_reg << 16;
223 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, psp_ring_reg);
224
225 /* there might be handshake issue with hardware which needs delay */
226 mdelay(20);
227
228 /* Wait for response flag (bit 31) in C2PMSG_64 */
229 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64),
230 0x80000000, 0x8000FFFF, false);
231 }
232 167
233 return ret;168 return ret;
234}169}
@@ -247,9 +182,6 @@ static int psp_v12_0_ring_stop(struct psp_context *psp, · 12 unmodified lines
247 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64,182 WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64,
248 GFX_CTRL_CMD_ID_DESTROY_RINGS);183 GFX_CTRL_CMD_ID_DESTROY_RINGS);
249 184
250 /* there might be handshake issue with hardware which needs delay */
251 mdelay(20);
252
253 /* Wait for response flag (bit 31) */185 /* Wait for response flag (bit 31) */
254 if (amdgpu_sriov_vf(adev))186 if (amdgpu_sriov_vf(adev))
255 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101),187 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101),
drivers/gpu/drm/amd/amdgpu/psp_v13_0.c+51−0
@@ -85,6 +85,8 @@ MODULE_FIRMWARE("amdgpu/psp_14_0_4_ta.bin"); · 84 unmodified lines
85 85
86#define regMP1_PUB_SCRATCH0 0x3b1009086#define regMP1_PUB_SCRATCH0 0x3b10090
87 87
88#define PSP13_BL_STATUS_SIZE 100
89
88static int psp_v13_0_init_microcode(struct psp_context *psp)90static int psp_v13_0_init_microcode(struct psp_context *psp)
89{91{
90 struct amdgpu_device *adev = psp->adev;92 struct amdgpu_device *adev = psp->adev;
@@ -151,6 +153,32 @@ static bool psp_v13_0_is_sos_alive(struct psp_context *psp) · 60 unmodified lines
151 return sol_reg != 0x0;153 return sol_reg != 0x0;
152}154}
153 155
156static void psp_v13_0_bootloader_print_status(struct psp_context *psp,
157 const char *msg)
158{
159 struct amdgpu_device *adev = psp->adev;
160 u32 bl_status_reg;
161 char bl_status_msg[PSP13_BL_STATUS_SIZE];
162 int i, at;
163
164 if (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||
165 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||
166 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)) {
167 at = 0;
168 for_each_inst(i, adev->aid_mask) {
169 bl_status_reg =
170 (SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_92)
171 << 2) +
172 adev->asic_funcs->encode_ext_smn_addressing(i);
173 at += snprintf(bl_status_msg + at,
174 PSP13_BL_STATUS_SIZE - at,
175 " status(%02i): 0x%08x", i,
176 RREG32_PCIE_EXT(bl_status_reg));
177 }
178 dev_info(adev->dev, "%s - %s", msg, bl_status_msg);
179 }
180}
181
154static int psp_v13_0_wait_for_vmbx_ready(struct psp_context *psp)182static int psp_v13_0_wait_for_vmbx_ready(struct psp_context *psp)
155{183{
156 struct amdgpu_device *adev = psp->adev;184 struct amdgpu_device *adev = psp->adev;
@@ -196,6 +224,9 @@ static int psp_v13_0_wait_for_bootloader(struct psp_context *psp) · 39 unmodified lines
196 224
197 if (ret == 0)225 if (ret == 0)
198 return 0;226 return 0;
227 if (retry_loop && !(retry_loop % 10))
228 psp_v13_0_bootloader_print_status(
229 psp, "Waiting for bootloader completion");
199 }230 }
200 231
201 return ret;232 return ret;
@@ -858,6 +889,25 @@ static bool psp_v13_0_is_reload_needed(struct psp_context *psp) · 656 unmodified lines
858 return false;889 return false;
859}890}
860 891
892static int psp_v13_0_reg_program_no_ring(struct psp_context *psp, uint32_t val,
893 enum psp_reg_prog_id id)
894{
895 struct amdgpu_device *adev = psp->adev;
896 int ret = -EOPNOTSUPP;
897
898 /* PSP will broadcast the value to all instances */
899 if (amdgpu_sriov_vf(adev)) {
900 WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_101, GFX_CTRL_CMD_ID_GBR_IH_SET);
901 WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_102, id);
902 WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_103, val);
903
904 ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_101),
905 0x80000000, 0x80000000, false);
906 }
907
908 return ret;
909}
910
861static const struct psp_funcs psp_v13_0_funcs = {911static const struct psp_funcs psp_v13_0_funcs = {
862 .init_microcode = psp_v13_0_init_microcode,912 .init_microcode = psp_v13_0_init_microcode,
863 .wait_for_bootloader = psp_v13_0_wait_for_bootloader_steady_state,913 .wait_for_bootloader = psp_v13_0_wait_for_bootloader_steady_state,
@@ -884,6 +934,7 @@ static const struct psp_funcs psp_v13_0_funcs = { · 20 unmodified lines
884 .get_ras_capability = psp_v13_0_get_ras_capability,934 .get_ras_capability = psp_v13_0_get_ras_capability,
885 .is_aux_sos_load_required = psp_v13_0_is_aux_sos_load_required,935 .is_aux_sos_load_required = psp_v13_0_is_aux_sos_load_required,
886 .is_reload_needed = psp_v13_0_is_reload_needed,936 .is_reload_needed = psp_v13_0_is_reload_needed,
937 .reg_program_no_ring = psp_v13_0_reg_program_no_ring,
887};938};
888 939
889void psp_v13_0_set_psp_funcs(struct psp_context *psp)940void psp_v13_0_set_psp_funcs(struct psp_context *psp)
drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c+17−19
@@ -106,8 +106,9 @@ static void sdma_v4_4_2_set_buffer_funcs(struct amdgpu_device *adev); · 105 unmodified lines
106static void sdma_v4_4_2_set_vm_pte_funcs(struct amdgpu_device *adev);106static void sdma_v4_4_2_set_vm_pte_funcs(struct amdgpu_device *adev);
107static void sdma_v4_4_2_set_irq_funcs(struct amdgpu_device *adev);107static void sdma_v4_4_2_set_irq_funcs(struct amdgpu_device *adev);
108static void sdma_v4_4_2_set_ras_funcs(struct amdgpu_device *adev);108static void sdma_v4_4_2_set_ras_funcs(struct amdgpu_device *adev);
109static void sdma_v4_4_2_set_engine_reset_funcs(struct amdgpu_device *adev);
110static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev);109static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev);
110static int sdma_v4_4_2_stop_queue(struct amdgpu_ring *ring);
111static int sdma_v4_4_2_restore_queue(struct amdgpu_ring *ring);
111 112
112static u32 sdma_v4_4_2_get_reg_offset(struct amdgpu_device *adev,113static u32 sdma_v4_4_2_get_reg_offset(struct amdgpu_device *adev,
113 u32 instance, u32 offset)114 u32 instance, u32 offset)
@@ -1333,6 +1334,11 @@ static bool sdma_v4_4_2_fw_support_paging_queue(struct amdgpu_device *adev) · 1219 unmodified lines
1333 }1334 }
1334}1335}
1335 1336
1337static const struct amdgpu_sdma_funcs sdma_v4_4_2_sdma_funcs = {
1338 .stop_kernel_queue = &sdma_v4_4_2_stop_queue,
1339 .start_kernel_queue = &sdma_v4_4_2_restore_queue,
1340};
1341
1336static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block)1342static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block)
1337{1343{
1338 struct amdgpu_device *adev = ip_block->adev;1344 struct amdgpu_device *adev = ip_block->adev;
@@ -1351,8 +1357,6 @@ static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block) · 12 unmodified lines
1351 sdma_v4_4_2_set_vm_pte_funcs(adev);1357 sdma_v4_4_2_set_vm_pte_funcs(adev);
1352 sdma_v4_4_2_set_irq_funcs(adev);1358 sdma_v4_4_2_set_irq_funcs(adev);
1353 sdma_v4_4_2_set_ras_funcs(adev);1359 sdma_v4_4_2_set_ras_funcs(adev);
1354 sdma_v4_4_2_set_engine_reset_funcs(adev);
1355
1356 return 0;1360 return 0;
1357}1361}
1358 1362
@@ -1447,6 +1451,7 @@ static int sdma_v4_4_2_sw_init(struct amdgpu_ip_block *ip_block) · 88 unmodified lines
1447 /* Initialize guilty flags for GFX and PAGE queues */1451 /* Initialize guilty flags for GFX and PAGE queues */
1448 adev->sdma.instance[i].gfx_guilty = false;1452 adev->sdma.instance[i].gfx_guilty = false;
1449 adev->sdma.instance[i].page_guilty = false;1453 adev->sdma.instance[i].page_guilty = false;
1454 adev->sdma.instance[i].funcs = &sdma_v4_4_2_sdma_funcs;
1450 1455
1451 ring = &adev->sdma.instance[i].ring;1456 ring = &adev->sdma.instance[i].ring;
1452 ring->ring_obj = NULL;1457 ring->ring_obj = NULL;
@@ -1678,11 +1683,12 @@ static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 225 unmodified lines
1678 return r;1683 return r;
1679}1684}
1680 1685
1681static int sdma_v4_4_2_stop_queue(struct amdgpu_device *adev, uint32_t instance_id)1686static int sdma_v4_4_2_stop_queue(struct amdgpu_ring *ring)
1682{1687{
1688 struct amdgpu_device *adev = ring->adev;
1689 u32 instance_id = GET_INST(SDMA0, ring->me);
1683 u32 inst_mask;1690 u32 inst_mask;
1684 uint64_t rptr;1691 uint64_t rptr;
1685 struct amdgpu_ring *ring = &adev->sdma.instance[instance_id].ring;
1686 1692
1687 if (amdgpu_sriov_vf(adev))1693 if (amdgpu_sriov_vf(adev))
1688 return -EINVAL;1694 return -EINVAL;
@@ -1715,11 +1721,11 @@ static int sdma_v4_4_2_stop_queue(struct amdgpu_device *adev, uint32_t instance_ · 26 unmodified lines
1715 return 0;1721 return 0;
1716}1722}
1717 1723
1718static int sdma_v4_4_2_restore_queue(struct amdgpu_device *adev, uint32_t instance_id)1724static int sdma_v4_4_2_restore_queue(struct amdgpu_ring *ring)
1719{1725{
1720 int i;1726 struct amdgpu_device *adev = ring->adev;
1721 u32 inst_mask;1727 u32 inst_mask;
1722 struct amdgpu_ring *ring = &adev->sdma.instance[instance_id].ring;1728 int i;
1723 1729
1724 inst_mask = 1 << ring->me;1730 inst_mask = 1 << ring->me;
1725 udelay(50);1731 udelay(50);
@@ -1739,16 +1745,6 @@ static int sdma_v4_4_2_restore_queue(struct amdgpu_device *adev, uint32_t instan · 13 unmodified lines
1739 return sdma_v4_4_2_inst_start(adev, inst_mask, true);1745 return sdma_v4_4_2_inst_start(adev, inst_mask, true);
1740}1746}
1741 1747
1742static struct sdma_on_reset_funcs sdma_v4_4_2_engine_reset_funcs = {
1743 .pre_reset = sdma_v4_4_2_stop_queue,
1744 .post_reset = sdma_v4_4_2_restore_queue,
1745};
1746
1747static void sdma_v4_4_2_set_engine_reset_funcs(struct amdgpu_device *adev)
1748{
1749 amdgpu_sdma_register_on_reset_callbacks(adev, &sdma_v4_4_2_engine_reset_funcs);
1750}
1751
1752static int sdma_v4_4_2_set_trap_irq_state(struct amdgpu_device *adev,1748static int sdma_v4_4_2_set_trap_irq_state(struct amdgpu_device *adev,
1753 struct amdgpu_irq_src *source,1749 struct amdgpu_irq_src *source,
1754 unsigned type,1750 unsigned type,
@@ -2373,7 +2369,9 @@ static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev) · 618 unmodified lines
2373 adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;2369 adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
2374 break;2370 break;
2375 case IP_VERSION(9, 5, 0):2371 case IP_VERSION(9, 5, 0):
2376 /*TODO: enable the queue reset flag until fw supported */2372 if ((adev->gfx.mec_fw_version >= 0xf) && amdgpu_dpm_reset_sdma_is_supported(adev))
2373 adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
2374 break;
2377 default:2375 default:
2378 break;2376 break;
2379 }2377 }
drivers/gpu/drm/amd/amdgpu/sdma_v5_0.c+114−157
@@ -112,6 +112,8 @@ static void sdma_v5_0_set_ring_funcs(struct amdgpu_device *adev); · 111 unmodified lines
112static void sdma_v5_0_set_buffer_funcs(struct amdgpu_device *adev);112static void sdma_v5_0_set_buffer_funcs(struct amdgpu_device *adev);
113static void sdma_v5_0_set_vm_pte_funcs(struct amdgpu_device *adev);113static void sdma_v5_0_set_vm_pte_funcs(struct amdgpu_device *adev);
114static void sdma_v5_0_set_irq_funcs(struct amdgpu_device *adev);114static void sdma_v5_0_set_irq_funcs(struct amdgpu_device *adev);
115static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring);
116static int sdma_v5_0_restore_queue(struct amdgpu_ring *ring);
115 117
116static const struct soc15_reg_golden golden_settings_sdma_5[] = {118static const struct soc15_reg_golden golden_settings_sdma_5[] = {
117 SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_CHICKEN_BITS, 0xffbf1f0f, 0x03ab0107),119 SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_CHICKEN_BITS, 0xffbf1f0f, 0x03ab0107),
@@ -369,67 +371,36 @@ static uint64_t sdma_v5_0_ring_get_wptr(struct amdgpu_ring *ring) · 251 unmodified lines
369static void sdma_v5_0_ring_set_wptr(struct amdgpu_ring *ring)371static void sdma_v5_0_ring_set_wptr(struct amdgpu_ring *ring)
370{372{
371 struct amdgpu_device *adev = ring->adev;373 struct amdgpu_device *adev = ring->adev;
372 uint32_t *wptr_saved;
373 uint32_t *is_queue_unmap;
374 uint64_t aggregated_db_index;
375 uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_DMA].mqd_size;
376 374
377 DRM_DEBUG("Setting write pointer\n");375 DRM_DEBUG("Setting write pointer\n");
378 if (ring->is_mes_queue) {376 if (ring->use_doorbell) {
379 wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);377 DRM_DEBUG("Using doorbell -- "
380 is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +378 "wptr_offs == 0x%08x "
381 sizeof(uint32_t));379 "lower_32_bits(ring->wptr) << 2 == 0x%08x "
382 aggregated_db_index =380 "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",
383 amdgpu_mes_get_aggregated_doorbell_index(adev,381 ring->wptr_offs,
384 AMDGPU_MES_PRIORITY_LEVEL_NORMAL);382 lower_32_bits(ring->wptr << 2),
385 383 upper_32_bits(ring->wptr << 2));
384 /* XXX check if swapping is necessary on BE */
386 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,385 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
387 ring->wptr << 2);386 ring->wptr << 2);
388 *wptr_saved = ring->wptr << 2;387 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
389 if (*is_queue_unmap) {388 ring->doorbell_index, ring->wptr << 2);
390 WDOORBELL64(aggregated_db_index, ring->wptr << 2);389 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
391 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
392 ring->doorbell_index, ring->wptr << 2);
393 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
394 } else {
395 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
396 ring->doorbell_index, ring->wptr << 2);
397 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
398
399 if (*is_queue_unmap)
400 WDOORBELL64(aggregated_db_index,
401 ring->wptr << 2);
402 }
403 } else {390 } else {
404 if (ring->use_doorbell) {391 DRM_DEBUG("Not using doorbell -- "
405 DRM_DEBUG("Using doorbell -- "392 "mmSDMA%i_GFX_RB_WPTR == 0x%08x "
406 "wptr_offs == 0x%08x "393 "mmSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
407 "lower_32_bits(ring->wptr) << 2 == 0x%08x "394 ring->me,
408 "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",395 lower_32_bits(ring->wptr << 2),
409 ring->wptr_offs,396 ring->me,
410 lower_32_bits(ring->wptr << 2),397 upper_32_bits(ring->wptr << 2));
411 upper_32_bits(ring->wptr << 2));398 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
412 /* XXX check if swapping is necessary on BE */399 ring->me, mmSDMA0_GFX_RB_WPTR),
413 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,400 lower_32_bits(ring->wptr << 2));
414 ring->wptr << 2);401 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
415 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",402 ring->me, mmSDMA0_GFX_RB_WPTR_HI),
416 ring->doorbell_index, ring->wptr << 2);403 upper_32_bits(ring->wptr << 2));
417 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
418 } else {
419 DRM_DEBUG("Not using doorbell -- "
420 "mmSDMA%i_GFX_RB_WPTR == 0x%08x "
421 "mmSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
422 ring->me,
423 lower_32_bits(ring->wptr << 2),
424 ring->me,
425 upper_32_bits(ring->wptr << 2));
426 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
427 ring->me, mmSDMA0_GFX_RB_WPTR),
428 lower_32_bits(ring->wptr << 2));
429 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev,
430 ring->me, mmSDMA0_GFX_RB_WPTR_HI),
431 upper_32_bits(ring->wptr << 2));
432 }
433 }404 }
434}405}
435 406
@@ -575,11 +546,9 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 139 unmodified lines
575 }546 }
576 547
577 if (flags & AMDGPU_FENCE_FLAG_INT) {548 if (flags & AMDGPU_FENCE_FLAG_INT) {
578 uint32_t ctx = ring->is_mes_queue ?
579 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
580 /* generate an interrupt */549 /* generate an interrupt */
581 amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));550 amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));
582 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));551 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
583 }552 }
584}553}
585 554
@@ -588,15 +557,15 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 2 unmodified lines
588 * sdma_v5_0_gfx_stop - stop the gfx async dma engines557 * sdma_v5_0_gfx_stop - stop the gfx async dma engines
589 *558 *
590 * @adev: amdgpu_device pointer559 * @adev: amdgpu_device pointer
591 *560 * @inst_mask: mask of dma engine instances to be disabled
592 * Stop the gfx async dma ring buffers (NAVI10).561 * Stop the gfx async dma ring buffers (NAVI10).
593 */562 */
594static void sdma_v5_0_gfx_stop(struct amdgpu_device *adev)563static void sdma_v5_0_gfx_stop(struct amdgpu_device *adev, uint32_t inst_mask)
595{564{
596 u32 rb_cntl, ib_cntl;565 u32 rb_cntl, ib_cntl;
597 int i;566 int i;
598 567
599 for (i = 0; i < adev->sdma.num_instances; i++) {568 for_each_inst(i, inst_mask) {
600 rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));569 rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
601 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);570 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
602 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);571 WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
@@ -688,9 +657,11 @@ static void sdma_v5_0_enable(struct amdgpu_device *adev, bool enable) · 85 unmodified lines
688{657{
689 u32 f32_cntl;658 u32 f32_cntl;
690 int i;659 int i;
660 uint32_t inst_mask;
691 661
662 inst_mask = GENMASK(adev->sdma.num_instances - 1, 0);
692 if (!enable) {663 if (!enable) {
693 sdma_v5_0_gfx_stop(adev);664 sdma_v5_0_gfx_stop(adev, 1 << inst_mask);
694 sdma_v5_0_rlc_stop(adev);665 sdma_v5_0_rlc_stop(adev);
695 }666 }
696 667
@@ -1046,33 +1017,22 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 349 unmodified lines
1046 int r;1017 int r;
1047 u32 tmp;1018 u32 tmp;
1048 u64 gpu_addr;1019 u64 gpu_addr;
1049 volatile uint32_t *cpu_ptr = NULL;
1050 1020
1051 tmp = 0xCAFEDEAD;1021 tmp = 0xCAFEDEAD;
1052 1022
1053 if (ring->is_mes_queue) {1023 r = amdgpu_device_wb_get(adev, &index);
1054 uint32_t offset = 0;1024 if (r) {
1055 offset = amdgpu_mes_ctx_get_offs(ring,1025 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
1056 AMDGPU_MES_CTX_PADDING_OFFS);1026 return r;
1057 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1058 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1059 *cpu_ptr = tmp;
1060 } else {
1061 r = amdgpu_device_wb_get(adev, &index);
1062 if (r) {
1063 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
1064 return r;
1065 }
1066
1067 gpu_addr = adev->wb.gpu_addr + (index * 4);
1068 adev->wb.wb[index] = cpu_to_le32(tmp);
1069 }1027 }
1070 1028
1029 gpu_addr = adev->wb.gpu_addr + (index * 4);
1030 adev->wb.wb[index] = cpu_to_le32(tmp);
1031
1071 r = amdgpu_ring_alloc(ring, 20);1032 r = amdgpu_ring_alloc(ring, 20);
1072 if (r) {1033 if (r) {
1073 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);1034 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
1074 if (!ring->is_mes_queue)1035 amdgpu_device_wb_free(adev, index);
1075 amdgpu_device_wb_free(adev, index);
1076 return r;1036 return r;
1077 }1037 }
1078 1038
@@ -1085,10 +1045,7 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
1085 amdgpu_ring_commit(ring);1045 amdgpu_ring_commit(ring);
1086 1046
1087 for (i = 0; i < adev->usec_timeout; i++) {1047 for (i = 0; i < adev->usec_timeout; i++) {
1088 if (ring->is_mes_queue)1048 tmp = le32_to_cpu(adev->wb.wb[index]);
1089 tmp = le32_to_cpu(*cpu_ptr);
1090 else
1091 tmp = le32_to_cpu(adev->wb.wb[index]);
1092 if (tmp == 0xDEADBEEF)1049 if (tmp == 0xDEADBEEF)
1093 break;1050 break;
1094 if (amdgpu_emu_mode == 1)1051 if (amdgpu_emu_mode == 1)
@@ -1100,8 +1057,7 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
1100 if (i >= adev->usec_timeout)1057 if (i >= adev->usec_timeout)
1101 r = -ETIMEDOUT;1058 r = -ETIMEDOUT;
1102 1059
1103 if (!ring->is_mes_queue)1060 amdgpu_device_wb_free(adev, index);
1104 amdgpu_device_wb_free(adev, index);
1105 1061
1106 return r;1062 return r;
1107}1063}
@@ -1124,38 +1080,24 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
1124 long r;1080 long r;
1125 u32 tmp = 0;1081 u32 tmp = 0;
1126 u64 gpu_addr;1082 u64 gpu_addr;
1127 volatile uint32_t *cpu_ptr = NULL;
1128 1083
1129 tmp = 0xCAFEDEAD;1084 tmp = 0xCAFEDEAD;
1130 memset(&ib, 0, sizeof(ib));1085 memset(&ib, 0, sizeof(ib));
1131 1086
1132 if (ring->is_mes_queue) {1087 r = amdgpu_device_wb_get(adev, &index);
1133 uint32_t offset = 0;1088 if (r) {
1134 offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);1089 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1135 ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);1090 return r;
1136 ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);1091 }
1137
1138 offset = amdgpu_mes_ctx_get_offs(ring,
1139 AMDGPU_MES_CTX_PADDING_OFFS);
1140 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1141 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1142 *cpu_ptr = tmp;
1143 } else {
1144 r = amdgpu_device_wb_get(adev, &index);
1145 if (r) {
1146 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1147 return r;
1148 }
1149 1092
1150 gpu_addr = adev->wb.gpu_addr + (index * 4);1093 gpu_addr = adev->wb.gpu_addr + (index * 4);
1151 adev->wb.wb[index] = cpu_to_le32(tmp);1094 adev->wb.wb[index] = cpu_to_le32(tmp);
1152 1095
1153 r = amdgpu_ib_get(adev, NULL, 256,1096 r = amdgpu_ib_get(adev, NULL, 256,
1154 AMDGPU_IB_POOL_DIRECT, &ib);1097 AMDGPU_IB_POOL_DIRECT, &ib);
1155 if (r) {1098 if (r) {
1156 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);1099 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1157 goto err0;1100 goto err0;
1158 }
1159 }1101 }
1160 1102
1161 ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |1103 ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |
@@ -1183,10 +1125,7 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
1183 goto err1;1125 goto err1;
1184 }1126 }
1185 1127
1186 if (ring->is_mes_queue)1128 tmp = le32_to_cpu(adev->wb.wb[index]);
1187 tmp = le32_to_cpu(*cpu_ptr);
1188 else
1189 tmp = le32_to_cpu(adev->wb.wb[index]);
1190 1129
1191 if (tmp == 0xDEADBEEF)1130 if (tmp == 0xDEADBEEF)
1192 r = 0;1131 r = 0;
@@ -1197,8 +1136,7 @@ err1: · 4 unmodified lines
1197 amdgpu_ib_free(&ib, NULL);1136 amdgpu_ib_free(&ib, NULL);
1198 dma_fence_put(f);1137 dma_fence_put(f);
1199err0:1138err0:
1200 if (!ring->is_mes_queue)1139 amdgpu_device_wb_free(adev, index);
1201 amdgpu_device_wb_free(adev, index);
1202 return r;1140 return r;
1203}1141}
1204 1142
@@ -1389,6 +1327,36 @@ static void sdma_v5_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring, · 184 unmodified lines
1389 amdgpu_ring_emit_reg_wait(ring, reg1, mask, mask);1327 amdgpu_ring_emit_reg_wait(ring, reg1, mask, mask);
1390}1328}
1391 1329
1330static int sdma_v5_0_soft_reset_engine(struct amdgpu_device *adev, u32 instance_id)
1331{
1332 u32 grbm_soft_reset;
1333 u32 tmp;
1334
1335 grbm_soft_reset = REG_SET_FIELD(0,
1336 GRBM_SOFT_RESET, SOFT_RESET_SDMA0,
1337 1);
1338 grbm_soft_reset <<= instance_id;
1339
1340 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1341 tmp |= grbm_soft_reset;
1342 DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp);
1343 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
1344 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1345
1346 udelay(50);
1347
1348 tmp &= ~grbm_soft_reset;
1349 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
1350 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1351 return 0;
1352}
1353
1354static const struct amdgpu_sdma_funcs sdma_v5_0_sdma_funcs = {
1355 .stop_kernel_queue = &sdma_v5_0_stop_queue,
1356 .start_kernel_queue = &sdma_v5_0_restore_queue,
1357 .soft_reset_kernel_queue = &sdma_v5_0_soft_reset_engine,
1358};
1359
1392static int sdma_v5_0_early_init(struct amdgpu_ip_block *ip_block)1360static int sdma_v5_0_early_init(struct amdgpu_ip_block *ip_block)
1393{1361{
1394 struct amdgpu_device *adev = ip_block->adev;1362 struct amdgpu_device *adev = ip_block->adev;
@@ -1431,6 +1399,7 @@ static int sdma_v5_0_sw_init(struct amdgpu_ip_block *ip_block) · 36 unmodified lines
1431 return r;1399 return r;
1432 1400
1433 for (i = 0; i < adev->sdma.num_instances; i++) {1401 for (i = 0; i < adev->sdma.num_instances; i++) {
1402 adev->sdma.instance[i].funcs = &sdma_v5_0_sdma_funcs;
1434 ring = &adev->sdma.instance[i].ring;1403 ring = &adev->sdma.instance[i].ring;
1435 ring->ring_obj = NULL;1404 ring->ring_obj = NULL;
1436 ring->use_doorbell = true;1405 ring->use_doorbell = true;
@@ -1572,32 +1541,25 @@ static int sdma_v5_0_soft_reset(struct amdgpu_ip_block *ip_block) · 135 unmodified lines
1572static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)1541static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
1573{1542{
1574 struct amdgpu_device *adev = ring->adev;1543 struct amdgpu_device *adev = ring->adev;
1575 int i, j, r;1544 u32 inst_id = ring->me;
1576 u32 rb_cntl, ib_cntl, f32_cntl, freeze, cntl, preempt, soft_reset, stat1_reg;
1577 1545
1578 if (amdgpu_sriov_vf(adev))1546 return amdgpu_sdma_reset_engine(adev, inst_id);
1579 return -EINVAL;1547}
1580 1548
1581 for (i = 0; i < adev->sdma.num_instances; i++) {1549static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring)
1582 if (ring == &adev->sdma.instance[i].ring)1550{
1583 break;1551 u32 f32_cntl, freeze, cntl, stat1_reg;
1584 }1552 struct amdgpu_device *adev = ring->adev;
1553 int i, j, r = 0;
1585 1554
1586 if (i == adev->sdma.num_instances) {1555 if (amdgpu_sriov_vf(adev))
1587 DRM_ERROR("sdma instance not found\n");
1588 return -EINVAL;1556 return -EINVAL;
1589 }
1590 1557
1558 i = ring->me;
1591 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);1559 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
1592 1560
1593 /* stop queue */1561 /* stop queue */
1594 ib_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL));1562 sdma_v5_0_gfx_stop(adev, 1 << i);
1595 ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0);
1596 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl);
1597
1598 rb_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
1599 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
1600 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
1601 1563
1602 /* engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */1564 /* engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */
1603 freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE));1565 freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE));
@@ -1628,30 +1590,25 @@ static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 24 unmodified lines
1628 cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL));1590 cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL));
1629 cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0);1591 cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0);
1630 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl);1592 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl);
1593err0:
1594 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
1595 return r;
1596}
1631 1597
1632 /* soft reset SDMA_GFX_PREEMPT.IB_PREEMPT = 0 mmGRBM_SOFT_RESET.SOFT_RESET_SDMA0/1 = 1 */1598static int sdma_v5_0_restore_queue(struct amdgpu_ring *ring)
1633 preempt = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT));1599{
1634 preempt = REG_SET_FIELD(preempt, SDMA0_GFX_PREEMPT, IB_PREEMPT, 0);1600 struct amdgpu_device *adev = ring->adev;
1635 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT), preempt);1601 u32 inst_id = ring->me;
1636 1602 u32 freeze;
1637 soft_reset = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);1603 int r;
1638 soft_reset |= 1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i;
1639
1640 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1641
1642 udelay(50);
1643
1644 soft_reset &= ~(1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i);
1645 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1646 1604
1605 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
1647 /* unfreeze*/1606 /* unfreeze*/
1648 freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE));1607 freeze = RREG32(sdma_v5_0_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE));
1649 freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0);1608 freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0);
1650 WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE), freeze);1609 WREG32(sdma_v5_0_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE), freeze);
1651 1610
1652 r = sdma_v5_0_gfx_resume_instance(adev, i, true);1611 r = sdma_v5_0_gfx_resume_instance(adev, inst_id, true);
1653
1654err0:
1655 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);1612 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
1656 return r;1613 return r;
1657}1614}
drivers/gpu/drm/amd/amdgpu/sdma_v5_2.c+87−117
@@ -113,6 +113,8 @@ static void sdma_v5_2_set_ring_funcs(struct amdgpu_device *adev); · 112 unmodified lines
113static void sdma_v5_2_set_buffer_funcs(struct amdgpu_device *adev);113static void sdma_v5_2_set_buffer_funcs(struct amdgpu_device *adev);
114static void sdma_v5_2_set_vm_pte_funcs(struct amdgpu_device *adev);114static void sdma_v5_2_set_vm_pte_funcs(struct amdgpu_device *adev);
115static void sdma_v5_2_set_irq_funcs(struct amdgpu_device *adev);115static void sdma_v5_2_set_irq_funcs(struct amdgpu_device *adev);
116static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring);
117static int sdma_v5_2_restore_queue(struct amdgpu_ring *ring);
116 118
117static u32 sdma_v5_2_get_reg_offset(struct amdgpu_device *adev, u32 instance, u32 internal_offset)119static u32 sdma_v5_2_get_reg_offset(struct amdgpu_device *adev, u32 instance, u32 internal_offset)
118{120{
@@ -394,11 +396,9 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 275 unmodified lines
394 }396 }
395 397
396 if ((flags & AMDGPU_FENCE_FLAG_INT)) {398 if ((flags & AMDGPU_FENCE_FLAG_INT)) {
397 uint32_t ctx = ring->is_mes_queue ?
398 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
399 /* generate an interrupt */399 /* generate an interrupt */
400 amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));400 amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP));
401 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));401 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
402 }402 }
403}403}
404 404
@@ -407,15 +407,15 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 2 unmodified lines
407 * sdma_v5_2_gfx_stop - stop the gfx async dma engines407 * sdma_v5_2_gfx_stop - stop the gfx async dma engines
408 *408 *
409 * @adev: amdgpu_device pointer409 * @adev: amdgpu_device pointer
410 *410 * @inst_mask: mask of dma engine instances to be disabled
411 * Stop the gfx async dma ring buffers.411 * Stop the gfx async dma ring buffers.
412 */412 */
413static void sdma_v5_2_gfx_stop(struct amdgpu_device *adev)413static void sdma_v5_2_gfx_stop(struct amdgpu_device *adev, uint32_t inst_mask)
414{414{
415 u32 rb_cntl, ib_cntl;415 u32 rb_cntl, ib_cntl;
416 int i;416 int i;
417 417
418 for (i = 0; i < adev->sdma.num_instances; i++) {418 for_each_inst(i, inst_mask) {
419 rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));419 rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
420 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);420 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
421 WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);421 WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
@@ -506,9 +506,11 @@ static void sdma_v5_2_enable(struct amdgpu_device *adev, bool enable) · 84 unmodified lines
506{506{
507 u32 f32_cntl;507 u32 f32_cntl;
508 int i;508 int i;
509 uint32_t inst_mask;
509 510
511 inst_mask = GENMASK(adev->sdma.num_instances - 1, 0);
510 if (!enable) {512 if (!enable) {
511 sdma_v5_2_gfx_stop(adev);513 sdma_v5_2_gfx_stop(adev, inst_mask);
512 sdma_v5_2_rlc_stop(adev);514 sdma_v5_2_rlc_stop(adev);
513 }515 }
514 516
@@ -761,37 +763,49 @@ static int sdma_v5_2_load_microcode(struct amdgpu_device *adev) · 246 unmodified lines
761 return 0;763 return 0;
762}764}
763 765
764static int sdma_v5_2_soft_reset(struct amdgpu_ip_block *ip_block)766static int sdma_v5_2_soft_reset_engine(struct amdgpu_device *adev, u32 instance_id)
765{767{
766 struct amdgpu_device *adev = ip_block->adev;
767 u32 grbm_soft_reset;768 u32 grbm_soft_reset;
768 u32 tmp;769 u32 tmp;
769 int i;
770 770
771 for (i = 0; i < adev->sdma.num_instances; i++) {771 grbm_soft_reset = REG_SET_FIELD(0,
772 grbm_soft_reset = REG_SET_FIELD(0,772 GRBM_SOFT_RESET, SOFT_RESET_SDMA0,
773 GRBM_SOFT_RESET, SOFT_RESET_SDMA0,773 1);
774 1);774 grbm_soft_reset <<= instance_id;
775 grbm_soft_reset <<= i;
776 775
777 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);776 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
778 tmp |= grbm_soft_reset;777 tmp |= grbm_soft_reset;
779 DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp);778 DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp);
780 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);779 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
781 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);780 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
782 781
783 udelay(50);782 udelay(50);
783
784 tmp &= ~grbm_soft_reset;
785 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);
786 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
787 return 0;
788}
784 789
785 tmp &= ~grbm_soft_reset;790static int sdma_v5_2_soft_reset(struct amdgpu_ip_block *ip_block)
786 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp);791{
787 tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);792 struct amdgpu_device *adev = ip_block->adev;
793 int i;
788 794
795 for (i = 0; i < adev->sdma.num_instances; i++) {
796 sdma_v5_2_soft_reset_engine(adev, i);
789 udelay(50);797 udelay(50);
790 }798 }
791 799
792 return 0;800 return 0;
793}801}
794 802
803static const struct amdgpu_sdma_funcs sdma_v5_2_sdma_funcs = {
804 .stop_kernel_queue = &sdma_v5_2_stop_queue,
805 .start_kernel_queue = &sdma_v5_2_restore_queue,
806 .soft_reset_kernel_queue = &sdma_v5_2_soft_reset_engine,
807};
808
795/**809/**
796 * sdma_v5_2_start - setup and start the async dma engines810 * sdma_v5_2_start - setup and start the async dma engines
797 *811 *
@@ -903,33 +917,22 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 105 unmodified lines
903 int r;917 int r;
904 u32 tmp;918 u32 tmp;
905 u64 gpu_addr;919 u64 gpu_addr;
906 volatile uint32_t *cpu_ptr = NULL;
907 920
908 tmp = 0xCAFEDEAD;921 tmp = 0xCAFEDEAD;
909 922
910 if (ring->is_mes_queue) {923 r = amdgpu_device_wb_get(adev, &index);
911 uint32_t offset = 0;924 if (r) {
912 offset = amdgpu_mes_ctx_get_offs(ring,925 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
913 AMDGPU_MES_CTX_PADDING_OFFS);926 return r;
914 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
915 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
916 *cpu_ptr = tmp;
917 } else {
918 r = amdgpu_device_wb_get(adev, &index);
919 if (r) {
920 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
921 return r;
922 }
923
924 gpu_addr = adev->wb.gpu_addr + (index * 4);
925 adev->wb.wb[index] = cpu_to_le32(tmp);
926 }927 }
927 928
929 gpu_addr = adev->wb.gpu_addr + (index * 4);
930 adev->wb.wb[index] = cpu_to_le32(tmp);
931
928 r = amdgpu_ring_alloc(ring, 20);932 r = amdgpu_ring_alloc(ring, 20);
929 if (r) {933 if (r) {
930 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);934 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
931 if (!ring->is_mes_queue)935 amdgpu_device_wb_free(adev, index);
932 amdgpu_device_wb_free(adev, index);
933 return r;936 return r;
934 }937 }
935 938
@@ -942,10 +945,7 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
942 amdgpu_ring_commit(ring);945 amdgpu_ring_commit(ring);
943 946
944 for (i = 0; i < adev->usec_timeout; i++) {947 for (i = 0; i < adev->usec_timeout; i++) {
945 if (ring->is_mes_queue)948 tmp = le32_to_cpu(adev->wb.wb[index]);
946 tmp = le32_to_cpu(*cpu_ptr);
947 else
948 tmp = le32_to_cpu(adev->wb.wb[index]);
949 if (tmp == 0xDEADBEEF)949 if (tmp == 0xDEADBEEF)
950 break;950 break;
951 if (amdgpu_emu_mode == 1)951 if (amdgpu_emu_mode == 1)
@@ -957,8 +957,7 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
957 if (i >= adev->usec_timeout)957 if (i >= adev->usec_timeout)
958 r = -ETIMEDOUT;958 r = -ETIMEDOUT;
959 959
960 if (!ring->is_mes_queue)960 amdgpu_device_wb_free(adev, index);
961 amdgpu_device_wb_free(adev, index);
962 961
963 return r;962 return r;
964}963}
@@ -981,37 +980,23 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
981 long r;980 long r;
982 u32 tmp = 0;981 u32 tmp = 0;
983 u64 gpu_addr;982 u64 gpu_addr;
984 volatile uint32_t *cpu_ptr = NULL;
985 983
986 tmp = 0xCAFEDEAD;984 tmp = 0xCAFEDEAD;
987 memset(&ib, 0, sizeof(ib));985 memset(&ib, 0, sizeof(ib));
988 986
989 if (ring->is_mes_queue) {987 r = amdgpu_device_wb_get(adev, &index);
990 uint32_t offset = 0;988 if (r) {
991 offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);989 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
992 ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);990 return r;
993 ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);991 }
994
995 offset = amdgpu_mes_ctx_get_offs(ring,
996 AMDGPU_MES_CTX_PADDING_OFFS);
997 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
998 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
999 *cpu_ptr = tmp;
1000 } else {
1001 r = amdgpu_device_wb_get(adev, &index);
1002 if (r) {
1003 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1004 return r;
1005 }
1006 992
1007 gpu_addr = adev->wb.gpu_addr + (index * 4);993 gpu_addr = adev->wb.gpu_addr + (index * 4);
1008 adev->wb.wb[index] = cpu_to_le32(tmp);994 adev->wb.wb[index] = cpu_to_le32(tmp);
1009 995
1010 r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);996 r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1011 if (r) {997 if (r) {
1012 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);998 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1013 goto err0;999 goto err0;
1014 }
1015 }1000 }
1016 1001
1017 ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |1002 ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) |
@@ -1039,10 +1024,7 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
1039 goto err1;1024 goto err1;
1040 }1025 }
1041 1026
1042 if (ring->is_mes_queue)1027 tmp = le32_to_cpu(adev->wb.wb[index]);
1043 tmp = le32_to_cpu(*cpu_ptr);
1044 else
1045 tmp = le32_to_cpu(adev->wb.wb[index]);
1046 1028
1047 if (tmp == 0xDEADBEEF)1029 if (tmp == 0xDEADBEEF)
1048 r = 0;1030 r = 0;
@@ -1053,8 +1035,7 @@ err1: · 4 unmodified lines
1053 amdgpu_ib_free(&ib, NULL);1035 amdgpu_ib_free(&ib, NULL);
1054 dma_fence_put(f);1036 dma_fence_put(f);
1055err0:1037err0:
1056 if (!ring->is_mes_queue)1038 amdgpu_device_wb_free(adev, index);
1057 amdgpu_device_wb_free(adev, index);
1058 return r;1039 return r;
1059}1040}
1060 1041
@@ -1337,6 +1318,7 @@ static int sdma_v5_2_sw_init(struct amdgpu_ip_block *ip_block) · 276 unmodified lines
1337 }1318 }
1338 1319
1339 for (i = 0; i < adev->sdma.num_instances; i++) {1320 for (i = 0; i < adev->sdma.num_instances; i++) {
1321 adev->sdma.instance[i].funcs = &sdma_v5_2_sdma_funcs;
1340 ring = &adev->sdma.instance[i].ring;1322 ring = &adev->sdma.instance[i].ring;
1341 ring->ring_obj = NULL;1323 ring->ring_obj = NULL;
1342 ring->use_doorbell = true;1324 ring->use_doorbell = true;
@@ -1472,32 +1454,25 @@ static int sdma_v5_2_wait_for_idle(struct amdgpu_ip_block *ip_block) · 129 unmodified lines
1472static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)1454static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid)
1473{1455{
1474 struct amdgpu_device *adev = ring->adev;1456 struct amdgpu_device *adev = ring->adev;
1475 int i, j, r;1457 u32 inst_id = ring->me;
1476 u32 rb_cntl, ib_cntl, f32_cntl, freeze, cntl, preempt, soft_reset, stat1_reg;
1477 1458
1478 if (amdgpu_sriov_vf(adev))1459 return amdgpu_sdma_reset_engine(adev, inst_id);
1479 return -EINVAL;1460}
1480 1461
1481 for (i = 0; i < adev->sdma.num_instances; i++) {1462static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring)
1482 if (ring == &adev->sdma.instance[i].ring)1463{
1483 break;1464 u32 f32_cntl, freeze, cntl, stat1_reg;
1484 }1465 struct amdgpu_device *adev = ring->adev;
1466 int i, j, r = 0;
1485 1467
1486 if (i == adev->sdma.num_instances) {1468 if (amdgpu_sriov_vf(adev))
1487 DRM_ERROR("sdma instance not found\n");
1488 return -EINVAL;1469 return -EINVAL;
1489 }
1490 1470
1471 i = ring->me;
1491 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);1472 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
1492 1473
1493 /* stop queue */1474 /* stop queue */
1494 ib_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL));1475 sdma_v5_2_gfx_stop(adev, 1 << i);
1495 ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0);
1496 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl);
1497
1498 rb_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL));
1499 rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0);
1500 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl);
1501 1476
1502 /*engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */1477 /*engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */
1503 freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE));1478 freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE));
@@ -1530,31 +1505,26 @@ static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 26 unmodified lines
1530 cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0);1505 cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0);
1531 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl);1506 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl);
1532 1507
1533 /* soft reset SDMA_GFX_PREEMPT.IB_PREEMPT = 0 mmGRBM_SOFT_RESET.SOFT_RESET_SDMA0/1 = 1 */1508err0:
1534 preempt = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT));1509 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
1535 preempt = REG_SET_FIELD(preempt, SDMA0_GFX_PREEMPT, IB_PREEMPT, 0);1510 return r;
1536 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT), preempt);1511}
1537
1538 soft_reset = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET);
1539 soft_reset |= 1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i;
1540
1541
1542 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);
1543
1544 udelay(50);
1545
1546 soft_reset &= ~(1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i);
1547 1512
1548 WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset);1513static int sdma_v5_2_restore_queue(struct amdgpu_ring *ring)
1514{
1515 struct amdgpu_device *adev = ring->adev;
1516 u32 inst_id = ring->me;
1517 u32 freeze;
1518 int r;
1549 1519
1520 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
1550 /* unfreeze and unhalt */1521 /* unfreeze and unhalt */
1551 freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE));1522 freeze = RREG32(sdma_v5_2_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE));
1552 freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0);1523 freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0);
1553 WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE), freeze);1524 WREG32(sdma_v5_2_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE), freeze);
1554 1525
1555 r = sdma_v5_2_gfx_resume_instance(adev, i, true);1526 r = sdma_v5_2_gfx_resume_instance(adev, inst_id, true);
1556 1527
1557err0:
1558 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);1528 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
1559 return r;1529 return r;
1560}1530}
drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c+80−60
@@ -43,6 +43,7 @@ · 42 unmodified lines
43#include "sdma_common.h"43#include "sdma_common.h"
44#include "sdma_v6_0.h"44#include "sdma_v6_0.h"
45#include "v11_structs.h"45#include "v11_structs.h"
46#include "mes_userqueue.h"
46 47
47MODULE_FIRMWARE("amdgpu/sdma_6_0_0.bin");48MODULE_FIRMWARE("amdgpu/sdma_6_0_0.bin");
48MODULE_FIRMWARE("amdgpu/sdma_6_0_1.bin");49MODULE_FIRMWARE("amdgpu/sdma_6_0_1.bin");
@@ -376,11 +377,9 @@ static void sdma_v6_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 327 unmodified lines
376 }377 }
377 378
378 if (flags & AMDGPU_FENCE_FLAG_INT) {379 if (flags & AMDGPU_FENCE_FLAG_INT) {
379 uint32_t ctx = ring->is_mes_queue ?
380 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
381 /* generate an interrupt */380 /* generate an interrupt */
382 amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP));381 amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP));
383 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));382 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
384 }383 }
385}384}
386 385
@@ -891,6 +890,9 @@ static int sdma_v6_0_mqd_init(struct amdgpu_device *adev, void *mqd, · 504 unmodified lines
891 m->sdmax_rlcx_rb_aql_cntl = regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT;890 m->sdmax_rlcx_rb_aql_cntl = regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT;
892 m->sdmax_rlcx_dummy_reg = regSDMA0_QUEUE0_DUMMY_REG_DEFAULT;891 m->sdmax_rlcx_dummy_reg = regSDMA0_QUEUE0_DUMMY_REG_DEFAULT;
893 892
893 m->sdmax_rlcx_csa_addr_lo = lower_32_bits(prop->csa_addr);
894 m->sdmax_rlcx_csa_addr_hi = upper_32_bits(prop->csa_addr);
895
894 return 0;896 return 0;
895}897}
896 898
@@ -917,33 +919,22 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 20 unmodified lines
917 int r;919 int r;
918 u32 tmp;920 u32 tmp;
919 u64 gpu_addr;921 u64 gpu_addr;
920 volatile uint32_t *cpu_ptr = NULL;
921 922
922 tmp = 0xCAFEDEAD;923 tmp = 0xCAFEDEAD;
923 924
924 if (ring->is_mes_queue) {925 r = amdgpu_device_wb_get(adev, &index);
925 uint32_t offset = 0;926 if (r) {
926 offset = amdgpu_mes_ctx_get_offs(ring,927 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
927 AMDGPU_MES_CTX_PADDING_OFFS);928 return r;
928 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
929 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
930 *cpu_ptr = tmp;
931 } else {
932 r = amdgpu_device_wb_get(adev, &index);
933 if (r) {
934 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
935 return r;
936 }
937
938 gpu_addr = adev->wb.gpu_addr + (index * 4);
939 adev->wb.wb[index] = cpu_to_le32(tmp);
940 }929 }
941 930
931 gpu_addr = adev->wb.gpu_addr + (index * 4);
932 adev->wb.wb[index] = cpu_to_le32(tmp);
933
942 r = amdgpu_ring_alloc(ring, 5);934 r = amdgpu_ring_alloc(ring, 5);
943 if (r) {935 if (r) {
944 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);936 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
945 if (!ring->is_mes_queue)937 amdgpu_device_wb_free(adev, index);
946 amdgpu_device_wb_free(adev, index);
947 return r;938 return r;
948 }939 }
949 940
@@ -956,10 +947,7 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
956 amdgpu_ring_commit(ring);947 amdgpu_ring_commit(ring);
957 948
958 for (i = 0; i < adev->usec_timeout; i++) {949 for (i = 0; i < adev->usec_timeout; i++) {
959 if (ring->is_mes_queue)950 tmp = le32_to_cpu(adev->wb.wb[index]);
960 tmp = le32_to_cpu(*cpu_ptr);
961 else
962 tmp = le32_to_cpu(adev->wb.wb[index]);
963 if (tmp == 0xDEADBEEF)951 if (tmp == 0xDEADBEEF)
964 break;952 break;
965 if (amdgpu_emu_mode == 1)953 if (amdgpu_emu_mode == 1)
@@ -971,8 +959,7 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
971 if (i >= adev->usec_timeout)959 if (i >= adev->usec_timeout)
972 r = -ETIMEDOUT;960 r = -ETIMEDOUT;
973 961
974 if (!ring->is_mes_queue)962 amdgpu_device_wb_free(adev, index);
975 amdgpu_device_wb_free(adev, index);
976 963
977 return r;964 return r;
978}965}
@@ -995,37 +982,23 @@ static int sdma_v6_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
995 long r;982 long r;
996 u32 tmp = 0;983 u32 tmp = 0;
997 u64 gpu_addr;984 u64 gpu_addr;
998 volatile uint32_t *cpu_ptr = NULL;
999 985
1000 tmp = 0xCAFEDEAD;986 tmp = 0xCAFEDEAD;
1001 memset(&ib, 0, sizeof(ib));987 memset(&ib, 0, sizeof(ib));
1002 988
1003 if (ring->is_mes_queue) {989 r = amdgpu_device_wb_get(adev, &index);
1004 uint32_t offset = 0;990 if (r) {
1005 offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);991 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1006 ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);992 return r;
1007 ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);993 }
1008
1009 offset = amdgpu_mes_ctx_get_offs(ring,
1010 AMDGPU_MES_CTX_PADDING_OFFS);
1011 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1012 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1013 *cpu_ptr = tmp;
1014 } else {
1015 r = amdgpu_device_wb_get(adev, &index);
1016 if (r) {
1017 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1018 return r;
1019 }
1020 994
1021 gpu_addr = adev->wb.gpu_addr + (index * 4);995 gpu_addr = adev->wb.gpu_addr + (index * 4);
1022 adev->wb.wb[index] = cpu_to_le32(tmp);996 adev->wb.wb[index] = cpu_to_le32(tmp);
1023 997
1024 r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);998 r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1025 if (r) {999 if (r) {
1026 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);1000 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1027 goto err0;1001 goto err0;
1028 }
1029 }1002 }
1030 1003
1031 ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) |1004 ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) |
@@ -1053,10 +1026,7 @@ static int sdma_v6_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
1053 goto err1;1026 goto err1;
1054 }1027 }
1055 1028
1056 if (ring->is_mes_queue)1029 tmp = le32_to_cpu(adev->wb.wb[index]);
1057 tmp = le32_to_cpu(*cpu_ptr);
1058 else
1059 tmp = le32_to_cpu(adev->wb.wb[index]);
1060 1030
1061 if (tmp == 0xDEADBEEF)1031 if (tmp == 0xDEADBEEF)
1062 r = 0;1032 r = 0;
@@ -1067,8 +1037,7 @@ err1: · 4 unmodified lines
1067 amdgpu_ib_free(&ib, NULL);1037 amdgpu_ib_free(&ib, NULL);
1068 dma_fence_put(f);1038 dma_fence_put(f);
1069err0:1039err0:
1070 if (!ring->is_mes_queue)1040 amdgpu_device_wb_free(adev, index);
1071 amdgpu_device_wb_free(adev, index);
1072 return r;1041 return r;
1073}1042}
1074 1043
@@ -1300,6 +1269,23 @@ static int sdma_v6_0_early_init(struct amdgpu_ip_block *ip_block) · 225 unmodified lines
1300 struct amdgpu_device *adev = ip_block->adev;1269 struct amdgpu_device *adev = ip_block->adev;
1301 int r;1270 int r;
1302 1271
1272 switch (amdgpu_user_queue) {
1273 case -1:
1274 case 0:
1275 default:
1276 adev->sdma.no_user_submission = false;
1277 adev->sdma.disable_uq = true;
1278 break;
1279 case 1:
1280 adev->sdma.no_user_submission = false;
1281 adev->sdma.disable_uq = false;
1282 break;
1283 case 2:
1284 adev->sdma.no_user_submission = true;
1285 adev->sdma.disable_uq = false;
1286 break;
1287 }
1288
1303 r = amdgpu_sdma_init_microcode(adev, 0, true);1289 r = amdgpu_sdma_init_microcode(adev, 0, true);
1304 if (r)1290 if (r)
1305 return r;1291 return r;
@@ -1334,6 +1320,7 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 28 unmodified lines
1334 ring->ring_obj = NULL;1320 ring->ring_obj = NULL;
1335 ring->use_doorbell = true;1321 ring->use_doorbell = true;
1336 ring->me = i;1322 ring->me = i;
1323 ring->no_user_submission = adev->sdma.no_user_submission;
1337 1324
1338 DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,1325 DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
1339 ring->use_doorbell?"true":"false");1326 ring->use_doorbell?"true":"false");
@@ -1376,6 +1363,10 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 36 unmodified lines
1376 else1363 else
1377 DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n");1364 DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n");
1378 1365
1366 /* add firmware version checks here */
1367 if (0 && !adev->sdma.disable_uq)
1368 adev->userq_funcs[AMDGPU_HW_IP_DMA] = &userq_mes_funcs;
1369
1379 r = amdgpu_sdma_sysfs_reset_mask_init(adev);1370 r = amdgpu_sdma_sysfs_reset_mask_init(adev);
1380 if (r)1371 if (r)
1381 return r;1372 return r;
@@ -1399,11 +1390,39 @@ static int sdma_v6_0_sw_fini(struct amdgpu_ip_block *ip_block) · 17 unmodified lines
1399 return 0;1390 return 0;
1400}1391}
1401 1392
1393static int sdma_v6_0_set_userq_trap_interrupts(struct amdgpu_device *adev,
1394 bool enable)
1395{
1396 unsigned int irq_type;
1397 int i, r;
1398
1399 if (adev->userq_funcs[AMDGPU_HW_IP_DMA]) {
1400 for (i = 0; i < adev->sdma.num_instances; i++) {
1401 irq_type = AMDGPU_SDMA_IRQ_INSTANCE0 + i;
1402 if (enable)
1403 r = amdgpu_irq_get(adev, &adev->sdma.trap_irq,
1404 irq_type);
1405 else
1406 r = amdgpu_irq_put(adev, &adev->sdma.trap_irq,
1407 irq_type);
1408 if (r)
1409 return r;
1410 }
1411 }
1412
1413 return 0;
1414}
1415
1402static int sdma_v6_0_hw_init(struct amdgpu_ip_block *ip_block)1416static int sdma_v6_0_hw_init(struct amdgpu_ip_block *ip_block)
1403{1417{
1404 struct amdgpu_device *adev = ip_block->adev;1418 struct amdgpu_device *adev = ip_block->adev;
1419 int r;
1405 1420
1406 return sdma_v6_0_start(adev);1421 r = sdma_v6_0_start(adev);
1422 if (r)
1423 return r;
1424
1425 return sdma_v6_0_set_userq_trap_interrupts(adev, true);
1407}1426}
1408 1427
1409static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block)1428static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block)
@@ -1415,6 +1434,7 @@ static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
1415 1434
1416 sdma_v6_0_ctxempty_int_enable(adev, false);1435 sdma_v6_0_ctxempty_int_enable(adev, false);
1417 sdma_v6_0_enable(adev, false);1436 sdma_v6_0_enable(adev, false);
1437 sdma_v6_0_set_userq_trap_interrupts(adev, false);
1418 1438
1419 return 0;1439 return 0;
1420}1440}
drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c+107−114
@@ -42,6 +42,7 @@ · 41 unmodified lines
42#include "sdma_common.h"42#include "sdma_common.h"
43#include "sdma_v7_0.h"43#include "sdma_v7_0.h"
44#include "v12_structs.h"44#include "v12_structs.h"
45#include "mes_userqueue.h"
45 46
46MODULE_FIRMWARE("amdgpu/sdma_7_0_0.bin");47MODULE_FIRMWARE("amdgpu/sdma_7_0_0.bin");
47MODULE_FIRMWARE("amdgpu/sdma_7_0_1.bin");48MODULE_FIRMWARE("amdgpu/sdma_7_0_1.bin");
@@ -204,66 +205,39 @@ static uint64_t sdma_v7_0_ring_get_wptr(struct amdgpu_ring *ring) · 156 unmodified lines
204static void sdma_v7_0_ring_set_wptr(struct amdgpu_ring *ring)205static void sdma_v7_0_ring_set_wptr(struct amdgpu_ring *ring)
205{206{
206 struct amdgpu_device *adev = ring->adev;207 struct amdgpu_device *adev = ring->adev;
207 uint32_t *wptr_saved;
208 uint32_t *is_queue_unmap;
209 uint64_t aggregated_db_index;
210 uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_DMA].mqd_size;
211 208
212 DRM_DEBUG("Setting write pointer\n");209 DRM_DEBUG("Setting write pointer\n");
213 210
214 if (ring->is_mes_queue) {211 if (ring->use_doorbell) {
215 wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size);212 DRM_DEBUG("Using doorbell -- "
216 is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size +213 "wptr_offs == 0x%08x "
217 sizeof(uint32_t));214 "lower_32_bits(ring->wptr) << 2 == 0x%08x "
218 aggregated_db_index =215 "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",
219 amdgpu_mes_get_aggregated_doorbell_index(adev,216 ring->wptr_offs,
220 ring->hw_prio);217 lower_32_bits(ring->wptr << 2),
221 218 upper_32_bits(ring->wptr << 2));
219 /* XXX check if swapping is necessary on BE */
222 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,220 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,
223 ring->wptr << 2);221 ring->wptr << 2);
224 *wptr_saved = ring->wptr << 2;222 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
225 if (*is_queue_unmap) {223 ring->doorbell_index, ring->wptr << 2);
226 WDOORBELL64(aggregated_db_index, ring->wptr << 2);224 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
227 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
228 ring->doorbell_index, ring->wptr << 2);
229 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
230 } else {
231 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",
232 ring->doorbell_index, ring->wptr << 2);
233 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);
234 }
235 } else {225 } else {
236 if (ring->use_doorbell) {226 DRM_DEBUG("Not using doorbell -- "
237 DRM_DEBUG("Using doorbell -- "227 "regSDMA%i_GFX_RB_WPTR == 0x%08x "
238 "wptr_offs == 0x%08x "228 "regSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
239 "lower_32_bits(ring->wptr) << 2 == 0x%08x "229 ring->me,
240 "upper_32_bits(ring->wptr) << 2 == 0x%08x\n",230 lower_32_bits(ring->wptr << 2),
241 ring->wptr_offs,231 ring->me,
242 lower_32_bits(ring->wptr << 2),232 upper_32_bits(ring->wptr << 2));
243 upper_32_bits(ring->wptr << 2));233 WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
244 /* XXX check if swapping is necessary on BE */234 ring->me,
245 atomic64_set((atomic64_t *)ring->wptr_cpu_addr,235 regSDMA0_QUEUE0_RB_WPTR),
246 ring->wptr << 2);236 lower_32_bits(ring->wptr << 2));
247 DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n",237 WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
248 ring->doorbell_index, ring->wptr << 2);238 ring->me,
249 WDOORBELL64(ring->doorbell_index, ring->wptr << 2);239 regSDMA0_QUEUE0_RB_WPTR_HI),
250 } else {240 upper_32_bits(ring->wptr << 2));
251 DRM_DEBUG("Not using doorbell -- "
252 "regSDMA%i_GFX_RB_WPTR == 0x%08x "
253 "regSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n",
254 ring->me,
255 lower_32_bits(ring->wptr << 2),
256 ring->me,
257 upper_32_bits(ring->wptr << 2));
258 WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
259 ring->me,
260 regSDMA0_QUEUE0_RB_WPTR),
261 lower_32_bits(ring->wptr << 2));
262 WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev,
263 ring->me,
264 regSDMA0_QUEUE0_RB_WPTR_HI),
265 upper_32_bits(ring->wptr << 2));
266 }
267 }241 }
268}242}
269 243
@@ -407,11 +381,9 @@ static void sdma_v7_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 137 unmodified lines
407 }381 }
408 382
409 if (flags & AMDGPU_FENCE_FLAG_INT) {383 if (flags & AMDGPU_FENCE_FLAG_INT) {
410 uint32_t ctx = ring->is_mes_queue ?
411 (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0;
412 /* generate an interrupt */384 /* generate an interrupt */
413 amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP));385 amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP));
414 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx));386 amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0));
415 }387 }
416}388}
417 389
@@ -935,6 +907,9 @@ static int sdma_v7_0_mqd_init(struct amdgpu_device *adev, void *mqd, · 517 unmodified lines
935 m->sdmax_rlcx_rb_aql_cntl = 0x4000; //regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT;907 m->sdmax_rlcx_rb_aql_cntl = 0x4000; //regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT;
936 m->sdmax_rlcx_dummy_reg = 0xf; //regSDMA0_QUEUE0_DUMMY_REG_DEFAULT;908 m->sdmax_rlcx_dummy_reg = 0xf; //regSDMA0_QUEUE0_DUMMY_REG_DEFAULT;
937 909
910 m->sdmax_rlcx_csa_addr_lo = lower_32_bits(prop->csa_addr);
911 m->sdmax_rlcx_csa_addr_hi = upper_32_bits(prop->csa_addr);
912
938 return 0;913 return 0;
939}914}
940 915
@@ -961,33 +936,22 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 20 unmodified lines
961 int r;936 int r;
962 u32 tmp;937 u32 tmp;
963 u64 gpu_addr;938 u64 gpu_addr;
964 volatile uint32_t *cpu_ptr = NULL;
965 939
966 tmp = 0xCAFEDEAD;940 tmp = 0xCAFEDEAD;
967 941
968 if (ring->is_mes_queue) {942 r = amdgpu_device_wb_get(adev, &index);
969 uint32_t offset = 0;943 if (r) {
970 offset = amdgpu_mes_ctx_get_offs(ring,944 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
971 AMDGPU_MES_CTX_PADDING_OFFS);945 return r;
972 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
973 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
974 *cpu_ptr = tmp;
975 } else {
976 r = amdgpu_device_wb_get(adev, &index);
977 if (r) {
978 dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r);
979 return r;
980 }
981
982 gpu_addr = adev->wb.gpu_addr + (index * 4);
983 adev->wb.wb[index] = cpu_to_le32(tmp);
984 }946 }
985 947
948 gpu_addr = adev->wb.gpu_addr + (index * 4);
949 adev->wb.wb[index] = cpu_to_le32(tmp);
950
986 r = amdgpu_ring_alloc(ring, 5);951 r = amdgpu_ring_alloc(ring, 5);
987 if (r) {952 if (r) {
988 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);953 DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r);
989 if (!ring->is_mes_queue)954 amdgpu_device_wb_free(adev, index);
990 amdgpu_device_wb_free(adev, index);
991 return r;955 return r;
992 }956 }
993 957
@@ -1000,10 +964,7 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines
1000 amdgpu_ring_commit(ring);964 amdgpu_ring_commit(ring);
1001 965
1002 for (i = 0; i < adev->usec_timeout; i++) {966 for (i = 0; i < adev->usec_timeout; i++) {
1003 if (ring->is_mes_queue)967 tmp = le32_to_cpu(adev->wb.wb[index]);
1004 tmp = le32_to_cpu(*cpu_ptr);
1005 else
1006 tmp = le32_to_cpu(adev->wb.wb[index]);
1007 if (tmp == 0xDEADBEEF)968 if (tmp == 0xDEADBEEF)
1008 break;969 break;
1009 if (amdgpu_emu_mode == 1)970 if (amdgpu_emu_mode == 1)
@@ -1015,8 +976,7 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines
1015 if (i >= adev->usec_timeout)976 if (i >= adev->usec_timeout)
1016 r = -ETIMEDOUT;977 r = -ETIMEDOUT;
1017 978
1018 if (!ring->is_mes_queue)979 amdgpu_device_wb_free(adev, index);
1019 amdgpu_device_wb_free(adev, index);
1020 980
1021 return r;981 return r;
1022}982}
@@ -1039,37 +999,23 @@ static int sdma_v7_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines
1039 long r;999 long r;
1040 u32 tmp = 0;1000 u32 tmp = 0;
1041 u64 gpu_addr;1001 u64 gpu_addr;
1042 volatile uint32_t *cpu_ptr = NULL;
1043 1002
1044 tmp = 0xCAFEDEAD;1003 tmp = 0xCAFEDEAD;
1045 memset(&ib, 0, sizeof(ib));1004 memset(&ib, 0, sizeof(ib));
1046 1005
1047 if (ring->is_mes_queue) {1006 r = amdgpu_device_wb_get(adev, &index);
1048 uint32_t offset = 0;1007 if (r) {
1049 offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS);1008 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1050 ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);1009 return r;
1051 ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);1010 }
1052
1053 offset = amdgpu_mes_ctx_get_offs(ring,
1054 AMDGPU_MES_CTX_PADDING_OFFS);
1055 gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset);
1056 cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset);
1057 *cpu_ptr = tmp;
1058 } else {
1059 r = amdgpu_device_wb_get(adev, &index);
1060 if (r) {
1061 dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r);
1062 return r;
1063 }
1064 1011
1065 gpu_addr = adev->wb.gpu_addr + (index * 4);1012 gpu_addr = adev->wb.gpu_addr + (index * 4);
1066 adev->wb.wb[index] = cpu_to_le32(tmp);1013 adev->wb.wb[index] = cpu_to_le32(tmp);
1067 1014
1068 r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);1015 r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib);
1069 if (r) {1016 if (r) {
1070 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);1017 DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r);
1071 goto err0;1018 goto err0;
1072 }
1073 }1019 }
1074 1020
1075 ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) |1021 ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) |
@@ -1097,10 +1043,7 @@ static int sdma_v7_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines
1097 goto err1;1043 goto err1;
1098 }1044 }
1099 1045
1100 if (ring->is_mes_queue)1046 tmp = le32_to_cpu(adev->wb.wb[index]);
1101 tmp = le32_to_cpu(*cpu_ptr);
1102 else
1103 tmp = le32_to_cpu(adev->wb.wb[index]);
1104 1047
1105 if (tmp == 0xDEADBEEF)1048 if (tmp == 0xDEADBEEF)
1106 r = 0;1049 r = 0;
@@ -1111,8 +1054,7 @@ err1: · 4 unmodified lines
1111 amdgpu_ib_free(&ib, NULL);1054 amdgpu_ib_free(&ib, NULL);
1112 dma_fence_put(f);1055 dma_fence_put(f);
1113err0:1056err0:
1114 if (!ring->is_mes_queue)1057 amdgpu_device_wb_free(adev, index);
1115 amdgpu_device_wb_free(adev, index);
1116 return r;1058 return r;
1117}1059}
1118 1060
@@ -1312,6 +1254,23 @@ static int sdma_v7_0_early_init(struct amdgpu_ip_block *ip_block) · 193 unmodified lines
1312 struct amdgpu_device *adev = ip_block->adev;1254 struct amdgpu_device *adev = ip_block->adev;
1313 int r;1255 int r;
1314 1256
1257 switch (amdgpu_user_queue) {
1258 case -1:
1259 case 0:
1260 default:
1261 adev->sdma.no_user_submission = false;
1262 adev->sdma.disable_uq = true;
1263 break;
1264 case 1:
1265 adev->sdma.no_user_submission = false;
1266 adev->sdma.disable_uq = false;
1267 break;
1268 case 2:
1269 adev->sdma.no_user_submission = true;
1270 adev->sdma.disable_uq = false;
1271 break;
1272 }
1273
1315 r = amdgpu_sdma_init_microcode(adev, 0, true);1274 r = amdgpu_sdma_init_microcode(adev, 0, true);
1316 if (r) {1275 if (r) {
1317 DRM_ERROR("Failed to init sdma firmware!\n");1276 DRM_ERROR("Failed to init sdma firmware!\n");
@@ -1347,6 +1306,7 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block) · 29 unmodified lines
1347 ring->ring_obj = NULL;1306 ring->ring_obj = NULL;
1348 ring->use_doorbell = true;1307 ring->use_doorbell = true;
1349 ring->me = i;1308 ring->me = i;
1309 ring->no_user_submission = adev->sdma.no_user_submission;
1350 1310
1351 DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,1311 DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i,
1352 ring->use_doorbell?"true":"false");1312 ring->use_doorbell?"true":"false");
@@ -1378,6 +1338,10 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block) · 25 unmodified lines
1378 else1338 else
1379 DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n");1339 DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n");
1380 1340
1341 /* add firmware version checks here */
1342 if (0 && !adev->sdma.disable_uq)
1343 adev->userq_funcs[AMDGPU_HW_IP_DMA] = &userq_mes_funcs;
1344
1381 return r;1345 return r;
1382}1346}
1383 1347
@@ -1400,11 +1364,39 @@ static int sdma_v7_0_sw_fini(struct amdgpu_ip_block *ip_block) · 16 unmodified lines
1400 return 0;1364 return 0;
1401}1365}
1402 1366
1367static int sdma_v7_0_set_userq_trap_interrupts(struct amdgpu_device *adev,
1368 bool enable)
1369{
1370 unsigned int irq_type;
1371 int i, r;
1372
1373 if (adev->userq_funcs[AMDGPU_HW_IP_DMA]) {
1374 for (i = 0; i < adev->sdma.num_instances; i++) {
1375 irq_type = AMDGPU_SDMA_IRQ_INSTANCE0 + i;
1376 if (enable)
1377 r = amdgpu_irq_get(adev, &adev->sdma.trap_irq,
1378 irq_type);
1379 else
1380 r = amdgpu_irq_put(adev, &adev->sdma.trap_irq,
1381 irq_type);
1382 if (r)
1383 return r;
1384 }
1385 }
1386
1387 return 0;
1388}
1389
1403static int sdma_v7_0_hw_init(struct amdgpu_ip_block *ip_block)1390static int sdma_v7_0_hw_init(struct amdgpu_ip_block *ip_block)
1404{1391{
1405 struct amdgpu_device *adev = ip_block->adev;1392 struct amdgpu_device *adev = ip_block->adev;
1393 int r;
1406 1394
1407 return sdma_v7_0_start(adev);1395 r = sdma_v7_0_start(adev);
1396 if (r)
1397 return r;
1398
1399 return sdma_v7_0_set_userq_trap_interrupts(adev, true);
1408}1400}
1409 1401
1410static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block)1402static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block)
@@ -1416,6 +1408,7 @@ static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
1416 1408
1417 sdma_v7_0_ctx_switch_enable(adev, false);1409 sdma_v7_0_ctx_switch_enable(adev, false);
1418 sdma_v7_0_enable(adev, false);1410 sdma_v7_0_enable(adev, false);
1411 sdma_v7_0_set_userq_trap_interrupts(adev, false);
1419 1412
1420 return 0;1413 return 0;
1421}1414}
drivers/gpu/drm/amd/amdgpu/si.c+214−199
@@ -35,6 +35,7 @@ · 34 unmodified lines
35#include "amdgpu_vce.h"35#include "amdgpu_vce.h"
36#include "atom.h"36#include "atom.h"
37#include "amd_pcie.h"37#include "amd_pcie.h"
38
38#include "si_dpm.h"39#include "si_dpm.h"
39#include "sid.h"40#include "sid.h"
40#include "si_ih.h"41#include "si_ih.h"
@@ -44,17 +45,30 @@ · 3 unmodified lines
44#include "dce_v6_0.h"45#include "dce_v6_0.h"
45#include "si.h"46#include "si.h"
46#include "uvd_v3_1.h"47#include "uvd_v3_1.h"
47#include "amdgpu_vkms.h"48
49#include "uvd/uvd_4_0_d.h"
50
51#include "smu/smu_6_0_d.h"
52#include "smu/smu_6_0_sh_mask.h"
53
48#include "gca/gfx_6_0_d.h"54#include "gca/gfx_6_0_d.h"
55#include "gca/gfx_6_0_sh_mask.h"
56
49#include "oss/oss_1_0_d.h"57#include "oss/oss_1_0_d.h"
50#include "oss/oss_1_0_sh_mask.h"58#include "oss/oss_1_0_sh_mask.h"
59
51#include "gmc/gmc_6_0_d.h"60#include "gmc/gmc_6_0_d.h"
61#include"gmc/gmc_6_0_sh_mask.h"
62
52#include "dce/dce_6_0_d.h"63#include "dce/dce_6_0_d.h"
53#include "uvd/uvd_4_0_d.h"64#include "dce/dce_6_0_sh_mask.h"
65
54#include "bif/bif_3_0_d.h"66#include "bif/bif_3_0_d.h"
55#include "bif/bif_3_0_sh_mask.h"67#include "bif/bif_3_0_sh_mask.h"
68#include "si_enums.h"
56 69
57#include "amdgpu_dm.h"70#include "amdgpu_dm.h"
71#include "amdgpu_vkms.h"
58 72
59static const u32 tahiti_golden_registers[] =73static const u32 tahiti_golden_registers[] =
60{74{
@@ -1071,8 +1085,8 @@ static u32 si_smc_rreg(struct amdgpu_device *adev, u32 reg) · 1010 unmodified lines
1071 u32 r;1085 u32 r;
1072 1086
1073 spin_lock_irqsave(&adev->smc_idx_lock, flags);1087 spin_lock_irqsave(&adev->smc_idx_lock, flags);
1074 WREG32(SMC_IND_INDEX_0, (reg));1088 WREG32(mmSMC_IND_INDEX_0, (reg));
1075 r = RREG32(SMC_IND_DATA_0);1089 r = RREG32(mmSMC_IND_DATA_0);
1076 spin_unlock_irqrestore(&adev->smc_idx_lock, flags);1090 spin_unlock_irqrestore(&adev->smc_idx_lock, flags);
1077 return r;1091 return r;
1078}1092}
@@ -1082,8 +1096,8 @@ static void si_smc_wreg(struct amdgpu_device *adev, u32 reg, u32 v) · 3 unmodified lines
1082 unsigned long flags;1096 unsigned long flags;
1083 1097
1084 spin_lock_irqsave(&adev->smc_idx_lock, flags);1098 spin_lock_irqsave(&adev->smc_idx_lock, flags);
1085 WREG32(SMC_IND_INDEX_0, (reg));1099 WREG32(mmSMC_IND_INDEX_0, (reg));
1086 WREG32(SMC_IND_DATA_0, (v));1100 WREG32(mmSMC_IND_DATA_0, (v));
1087 spin_unlock_irqrestore(&adev->smc_idx_lock, flags);1101 spin_unlock_irqrestore(&adev->smc_idx_lock, flags);
1088}1102}
1089 1103
@@ -1110,20 +1124,20 @@ static void si_uvd_ctx_wreg(struct amdgpu_device *adev, u32 reg, u32 v) · 20 unmodified lines
1110}1124}
1111 1125
1112static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = {1126static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = {
1113 {GRBM_STATUS},1127 {mmGRBM_STATUS},
1114 {mmGRBM_STATUS2},1128 {mmGRBM_STATUS2},
1115 {mmGRBM_STATUS_SE0},1129 {mmGRBM_STATUS_SE0},
1116 {mmGRBM_STATUS_SE1},1130 {mmGRBM_STATUS_SE1},
1117 {mmSRBM_STATUS},1131 {mmSRBM_STATUS},
1118 {mmSRBM_STATUS2},1132 {mmSRBM_STATUS2},
1119 {DMA_STATUS_REG + DMA0_REGISTER_OFFSET},1133 {mmDMA_STATUS_REG + DMA0_REGISTER_OFFSET},
1120 {DMA_STATUS_REG + DMA1_REGISTER_OFFSET},1134 {mmDMA_STATUS_REG + DMA1_REGISTER_OFFSET},
1121 {mmCP_STAT},1135 {mmCP_STAT},
1122 {mmCP_STALLED_STAT1},1136 {mmCP_STALLED_STAT1},
1123 {mmCP_STALLED_STAT2},1137 {mmCP_STALLED_STAT2},
1124 {mmCP_STALLED_STAT3},1138 {mmCP_STALLED_STAT3},
1125 {GB_ADDR_CONFIG},1139 {mmGB_ADDR_CONFIG},
1126 {MC_ARB_RAMCFG},1140 {mmMC_ARB_RAMCFG},
1127 {mmGB_TILE_MODE0},1141 {mmGB_TILE_MODE0},
1128 {mmGB_TILE_MODE1},1142 {mmGB_TILE_MODE1},
1129 {mmGB_TILE_MODE2},1143 {mmGB_TILE_MODE2},
@@ -1156,7 +1170,7 @@ static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = { · 26 unmodified lines
1156 {mmGB_TILE_MODE29},1170 {mmGB_TILE_MODE29},
1157 {mmGB_TILE_MODE30},1171 {mmGB_TILE_MODE30},
1158 {mmGB_TILE_MODE31},1172 {mmGB_TILE_MODE31},
1159 {CC_RB_BACKEND_DISABLE, true},1173 {mmCC_RB_BACKEND_DISABLE, true},
1160 {mmGC_USER_RB_BACKEND_DISABLE, true},1174 {mmGC_USER_RB_BACKEND_DISABLE, true},
1161 {mmPA_SC_RASTER_CONFIG, true},1175 {mmPA_SC_RASTER_CONFIG, true},
1162};1176};
@@ -1264,37 +1278,37 @@ static bool si_read_disabled_bios(struct amdgpu_device *adev) · 101 unmodified lines
1264 u32 rom_cntl;1278 u32 rom_cntl;
1265 bool r;1279 bool r;
1266 1280
1267 bus_cntl = RREG32(R600_BUS_CNTL);1281 bus_cntl = RREG32(mmBUS_CNTL);
1268 if (adev->mode_info.num_crtc) {1282 if (adev->mode_info.num_crtc) {
1269 d1vga_control = RREG32(AVIVO_D1VGA_CONTROL);1283 d1vga_control = RREG32(mmD1VGA_CONTROL);
1270 d2vga_control = RREG32(AVIVO_D2VGA_CONTROL);1284 d2vga_control = RREG32(mmD2VGA_CONTROL);
1271 vga_render_control = RREG32(VGA_RENDER_CONTROL);1285 vga_render_control = RREG32(mmVGA_RENDER_CONTROL);
1272 }1286 }
1273 rom_cntl = RREG32(R600_ROM_CNTL);1287 rom_cntl = RREG32(R600_ROM_CNTL);
1274 1288
1275 /* enable the rom */1289 /* enable the rom */
1276 WREG32(R600_BUS_CNTL, (bus_cntl & ~R600_BIOS_ROM_DIS));1290 WREG32(mmBUS_CNTL, (bus_cntl & ~BUS_CNTL__BIOS_ROM_DIS_MASK));
1277 if (adev->mode_info.num_crtc) {1291 if (adev->mode_info.num_crtc) {
1278 /* Disable VGA mode */1292 /* Disable VGA mode */
1279 WREG32(AVIVO_D1VGA_CONTROL,1293 WREG32(mmD1VGA_CONTROL,
1280 (d1vga_control & ~(AVIVO_DVGA_CONTROL_MODE_ENABLE |1294 (d1vga_control & ~(D1VGA_CONTROL__D1VGA_MODE_ENABLE_MASK |
1281 AVIVO_DVGA_CONTROL_TIMING_SELECT)));1295 D1VGA_CONTROL__D1VGA_TIMING_SELECT_MASK)));
1282 WREG32(AVIVO_D2VGA_CONTROL,1296 WREG32(mmD2VGA_CONTROL,
1283 (d2vga_control & ~(AVIVO_DVGA_CONTROL_MODE_ENABLE |1297 (d2vga_control & ~(D1VGA_CONTROL__D1VGA_MODE_ENABLE_MASK |
1284 AVIVO_DVGA_CONTROL_TIMING_SELECT)));1298 D1VGA_CONTROL__D1VGA_TIMING_SELECT_MASK)));
1285 WREG32(VGA_RENDER_CONTROL,1299 WREG32(mmVGA_RENDER_CONTROL,
1286 (vga_render_control & C_000300_VGA_VSTATUS_CNTL));1300 (vga_render_control & ~VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK));
1287 }1301 }
1288 WREG32(R600_ROM_CNTL, rom_cntl | R600_SCK_OVERWRITE);1302 WREG32(R600_ROM_CNTL, rom_cntl | R600_SCK_OVERWRITE);
1289 1303
1290 r = amdgpu_read_bios(adev);1304 r = amdgpu_read_bios(adev);
1291 1305
1292 /* restore regs */1306 /* restore regs */
1293 WREG32(R600_BUS_CNTL, bus_cntl);1307 WREG32(mmBUS_CNTL, bus_cntl);
1294 if (adev->mode_info.num_crtc) {1308 if (adev->mode_info.num_crtc) {
1295 WREG32(AVIVO_D1VGA_CONTROL, d1vga_control);1309 WREG32(mmD1VGA_CONTROL, d1vga_control);
1296 WREG32(AVIVO_D2VGA_CONTROL, d2vga_control);1310 WREG32(mmD2VGA_CONTROL, d2vga_control);
1297 WREG32(VGA_RENDER_CONTROL, vga_render_control);1311 WREG32(mmVGA_RENDER_CONTROL, vga_render_control);
1298 }1312 }
1299 WREG32(R600_ROM_CNTL, rom_cntl);1313 WREG32(R600_ROM_CNTL, rom_cntl);
1300 return r;1314 return r;
@@ -1331,23 +1345,24 @@ static void si_set_clk_bypass_mode(struct amdgpu_device *adev) · 30 unmodified lines
1331{1345{
1332 u32 tmp, i;1346 u32 tmp, i;
1333 1347
1334 tmp = RREG32(CG_SPLL_FUNC_CNTL);1348 tmp = RREG32(mmCG_SPLL_FUNC_CNTL);
1335 tmp |= SPLL_BYPASS_EN;1349 tmp |= CG_SPLL_FUNC_CNTL__SPLL_BYPASS_EN_MASK;
1336 WREG32(CG_SPLL_FUNC_CNTL, tmp);1350 WREG32(mmCG_SPLL_FUNC_CNTL, tmp);
1337 1351
1338 tmp = RREG32(CG_SPLL_FUNC_CNTL_2);1352 tmp = RREG32(mmCG_SPLL_FUNC_CNTL_2);
1339 tmp |= SPLL_CTLREQ_CHG;1353 tmp |= CG_SPLL_FUNC_CNTL_2__SPLL_CTLREQ_CHG_MASK;
1340 WREG32(CG_SPLL_FUNC_CNTL_2, tmp);1354 WREG32(mmCG_SPLL_FUNC_CNTL_2, tmp);
1341 1355
1342 for (i = 0; i < adev->usec_timeout; i++) {1356 for (i = 0; i < adev->usec_timeout; i++) {
1343 if (RREG32(SPLL_STATUS) & SPLL_CHG_STATUS)1357 if (RREG32(mmCG_SPLL_STATUS) & CG_SPLL_STATUS__SPLL_CHG_STATUS_MASK)
1344 break;1358 break;
1345 udelay(1);1359 udelay(1);
1346 }1360 }
1347 1361
1348 tmp = RREG32(CG_SPLL_FUNC_CNTL_2);1362 tmp = RREG32(mmCG_SPLL_FUNC_CNTL_2);
1349 tmp &= ~(SPLL_CTLREQ_CHG | SCLK_MUX_UPDATE);1363 tmp &= ~(CG_SPLL_FUNC_CNTL_2__SPLL_CTLREQ_CHG_MASK |
1350 WREG32(CG_SPLL_FUNC_CNTL_2, tmp);1364 CG_SPLL_FUNC_CNTL_2__SCLK_MUX_UPDATE_MASK);
1365 WREG32(mmCG_SPLL_FUNC_CNTL_2, tmp);
1351 1366
1352 tmp = RREG32(MPLL_CNTL_MODE);1367 tmp = RREG32(MPLL_CNTL_MODE);
1353 tmp &= ~MPLL_MCLK_SEL;1368 tmp &= ~MPLL_MCLK_SEL;
@@ -1358,21 +1373,21 @@ static void si_spll_powerdown(struct amdgpu_device *adev) · 4 unmodified lines
1358{1373{
1359 u32 tmp;1374 u32 tmp;
1360 1375
1361 tmp = RREG32(SPLL_CNTL_MODE);1376 tmp = RREG32(mmSPLL_CNTL_MODE);
1362 tmp |= SPLL_SW_DIR_CONTROL;1377 tmp |= SPLL_CNTL_MODE__SPLL_SW_DIR_CONTROL_MASK;
1363 WREG32(SPLL_CNTL_MODE, tmp);1378 WREG32(mmSPLL_CNTL_MODE, tmp);
1364 1379
1365 tmp = RREG32(CG_SPLL_FUNC_CNTL);1380 tmp = RREG32(mmCG_SPLL_FUNC_CNTL);
1366 tmp |= SPLL_RESET;1381 tmp |= CG_SPLL_FUNC_CNTL__SPLL_RESET_MASK;
1367 WREG32(CG_SPLL_FUNC_CNTL, tmp);1382 WREG32(mmCG_SPLL_FUNC_CNTL, tmp);
1368 1383
1369 tmp = RREG32(CG_SPLL_FUNC_CNTL);1384 tmp = RREG32(mmCG_SPLL_FUNC_CNTL);
1370 tmp |= SPLL_SLEEP;1385 tmp |= CG_SPLL_FUNC_CNTL__SPLL_SLEEP_MASK;
1371 WREG32(CG_SPLL_FUNC_CNTL, tmp);1386 WREG32(mmCG_SPLL_FUNC_CNTL, tmp);
1372 1387
1373 tmp = RREG32(SPLL_CNTL_MODE);1388 tmp = RREG32(mmSPLL_CNTL_MODE);
1374 tmp &= ~SPLL_SW_DIR_CONTROL;1389 tmp &= ~SPLL_CNTL_MODE__SPLL_SW_DIR_CONTROL_MASK;
1375 WREG32(SPLL_CNTL_MODE, tmp);1390 WREG32(mmSPLL_CNTL_MODE, tmp);
1376}1391}
1377 1392
1378static int si_gpu_pci_config_reset(struct amdgpu_device *adev)1393static int si_gpu_pci_config_reset(struct amdgpu_device *adev)
@@ -1454,14 +1469,14 @@ static void si_vga_set_state(struct amdgpu_device *adev, bool state) · 75 unmodified lines
1454{1469{
1455 uint32_t temp;1470 uint32_t temp;
1456 1471
1457 temp = RREG32(CONFIG_CNTL);1472 temp = RREG32(mmCONFIG_CNTL);
1458 if (!state) {1473 if (!state) {
1459 temp &= ~(1<<0);1474 temp &= ~(1<<0);
1460 temp |= (1<<1);1475 temp |= (1<<1);
1461 } else {1476 } else {
1462 temp &= ~(1<<1);1477 temp &= ~(1<<1);
1463 }1478 }
1464 WREG32(CONFIG_CNTL, temp);1479 WREG32(mmCONFIG_CNTL, temp);
1465}1480}
1466 1481
1467static u32 si_get_xclk(struct amdgpu_device *adev)1482static u32 si_get_xclk(struct amdgpu_device *adev)
@@ -1469,12 +1484,12 @@ static u32 si_get_xclk(struct amdgpu_device *adev) · 1 unmodified line
1469 u32 reference_clock = adev->clock.spll.reference_freq;1484 u32 reference_clock = adev->clock.spll.reference_freq;
1470 u32 tmp;1485 u32 tmp;
1471 1486
1472 tmp = RREG32(CG_CLKPIN_CNTL_2);1487 tmp = RREG32(mmCG_CLKPIN_CNTL_2);
1473 if (tmp & MUX_TCLK_TO_XCLK)1488 if (tmp & CG_CLKPIN_CNTL_2__MUX_TCLK_TO_XCLK_MASK)
1474 return TCLK;1489 return TCLK;
1475 1490
1476 tmp = RREG32(CG_CLKPIN_CNTL);1491 tmp = RREG32(mmCG_CLKPIN_CNTL);
1477 if (tmp & XTALIN_DIVIDE)1492 if (tmp & CG_CLKPIN_CNTL__XTALIN_DIVIDE_MASK)
1478 return reference_clock / 4;1493 return reference_clock / 4;
1479 1494
1480 return reference_clock;1495 return reference_clock;
@@ -1519,9 +1534,9 @@ static int si_get_pcie_lanes(struct amdgpu_device *adev) · 38 unmodified lines
1519 if (adev->flags & AMD_IS_APU)1534 if (adev->flags & AMD_IS_APU)
1520 return 0;1535 return 0;
1521 1536
1522 link_width_cntl = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);1537 link_width_cntl = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
1523 1538
1524 switch ((link_width_cntl & LC_LINK_WIDTH_RD_MASK) >> LC_LINK_WIDTH_RD_SHIFT) {1539 switch ((link_width_cntl & PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD_MASK) >> PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD__SHIFT) {
1525 case LC_LINK_WIDTH_X1:1540 case LC_LINK_WIDTH_X1:
1526 return 1;1541 return 1;
1527 case LC_LINK_WIDTH_X2:1542 case LC_LINK_WIDTH_X2:
@@ -1568,13 +1583,13 @@ static void si_set_pcie_lanes(struct amdgpu_device *adev, int lanes) · 40 unmodified lines
1568 return;1583 return;
1569 }1584 }
1570 1585
1571 link_width_cntl = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);1586 link_width_cntl = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
1572 link_width_cntl &= ~LC_LINK_WIDTH_MASK;1587 link_width_cntl &= ~PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_MASK;
1573 link_width_cntl |= mask << LC_LINK_WIDTH_SHIFT;1588 link_width_cntl |= mask << PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH__SHIFT;
1574 link_width_cntl |= (LC_RECONFIG_NOW |1589 link_width_cntl |= (PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_NOW_MASK |
1575 LC_RECONFIG_ARC_MISSING_ESCAPE);1590 PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_ARC_MISSING_ESCAPE_MASK);
1576 1591
1577 WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, link_width_cntl);1592 WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, link_width_cntl);
1578}1593}
1579 1594
1580static void si_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,1595static void si_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,
@@ -2018,7 +2033,7 @@ static const struct amdgpu_asic_funcs si_asic_funcs = · 437 unmodified lines
2018 2033
2019static uint32_t si_get_rev_id(struct amdgpu_device *adev)2034static uint32_t si_get_rev_id(struct amdgpu_device *adev)
2020{2035{
2021 return (RREG32(CC_DRM_ID_STRAPS) & CC_DRM_ID_STRAPS__ATI_REV_ID_MASK)2036 return (RREG32(mmCC_DRM_ID_STRAPS) & CC_DRM_ID_STRAPS__ATI_REV_ID_MASK)
2022 >> CC_DRM_ID_STRAPS__ATI_REV_ID__SHIFT;2037 >> CC_DRM_ID_STRAPS__ATI_REV_ID__SHIFT;
2023}2038}
2024 2039
@@ -2239,9 +2254,9 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 214 unmodified lines
2239 CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)))2254 CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)))
2240 return;2255 return;
2241 2256
2242 speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL);2257 speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL);
2243 current_data_rate = (speed_cntl & LC_CURRENT_DATA_RATE_MASK) >>2258 current_data_rate = (speed_cntl & PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE_MASK) >>
2244 LC_CURRENT_DATA_RATE_SHIFT;2259 PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE__SHIFT;
2245 if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) {2260 if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) {
2246 if (current_data_rate == 2) {2261 if (current_data_rate == 2) {
2247 DRM_INFO("PCIE gen 3 link speeds already enabled\n");2262 DRM_INFO("PCIE gen 3 link speeds already enabled\n");
@@ -2268,17 +2283,17 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 20 unmodified lines
2268 pcie_capability_set_word(root, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD);2283 pcie_capability_set_word(root, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD);
2269 pcie_capability_set_word(adev->pdev, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD);2284 pcie_capability_set_word(adev->pdev, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD);
2270 2285
2271 tmp = RREG32_PCIE(PCIE_LC_STATUS1);2286 tmp = RREG32_PCIE(ixPCIE_LC_STATUS1);
2272 max_lw = (tmp & LC_DETECTED_LINK_WIDTH_MASK) >> LC_DETECTED_LINK_WIDTH_SHIFT;2287 max_lw = (tmp & PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH_MASK) >> PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH__SHIFT;
2273 current_lw = (tmp & LC_OPERATING_LINK_WIDTH_MASK) >> LC_OPERATING_LINK_WIDTH_SHIFT;2288 current_lw = (tmp & PCIE_LC_STATUS1__LC_OPERATING_LINK_WIDTH_MASK) >> PCIE_LC_STATUS1__LC_OPERATING_LINK_WIDTH__SHIFT;
2274 2289
2275 if (current_lw < max_lw) {2290 if (current_lw < max_lw) {
2276 tmp = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);2291 tmp = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
2277 if (tmp & LC_RENEGOTIATION_SUPPORT) {2292 if (tmp & PCIE_LC_LINK_WIDTH_CNTL__LC_RENEGOTIATION_SUPPORT_MASK) {
2278 tmp &= ~(LC_LINK_WIDTH_MASK | LC_UPCONFIGURE_DIS);2293 tmp &= ~(PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_UPCONFIGURE_DIS_MASK);
2279 tmp |= (max_lw << LC_LINK_WIDTH_SHIFT);2294 tmp |= (max_lw << PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH__SHIFT);
2280 tmp |= LC_UPCONFIGURE_SUPPORT | LC_RENEGOTIATE_EN | LC_RECONFIG_NOW;2295 tmp |= PCIE_LC_LINK_WIDTH_CNTL__LC_UPCONFIGURE_SUPPORT_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_RENEGOTIATE_EN_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_NOW_MASK;
2281 WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, tmp);2296 WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, tmp);
2282 }2297 }
2283 }2298 }
2284 2299
@@ -2301,13 +2316,13 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 16 unmodified lines
2301 PCI_EXP_LNKCTL2,2316 PCI_EXP_LNKCTL2,
2302 &gpu_cfg2);2317 &gpu_cfg2);
2303 2318
2304 tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4);2319 tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4);
2305 tmp |= LC_SET_QUIESCE;2320 tmp |= PCIE_LC_CNTL4__LC_SET_QUIESCE_MASK;
2306 WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp);2321 WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp);
2307 2322
2308 tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4);2323 tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4);
2309 tmp |= LC_REDO_EQ;2324 tmp |= PCIE_LC_CNTL4__LC_REDO_EQ_MASK;
2310 WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp);2325 WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp);
2311 2326
2312 mdelay(100);2327 mdelay(100);
2313 2328
@@ -2333,16 +2348,16 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 19 unmodified lines
2333 (PCI_EXP_LNKCTL2_ENTER_COMP |2348 (PCI_EXP_LNKCTL2_ENTER_COMP |
2334 PCI_EXP_LNKCTL2_TX_MARGIN));2349 PCI_EXP_LNKCTL2_TX_MARGIN));
2335 2350
2336 tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4);2351 tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4);
2337 tmp &= ~LC_SET_QUIESCE;2352 tmp &= ~PCIE_LC_CNTL4__LC_SET_QUIESCE_MASK;
2338 WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp);2353 WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp);
2339 }2354 }
2340 }2355 }
2341 }2356 }
2342 2357
2343 speed_cntl |= LC_FORCE_EN_SW_SPEED_CHANGE | LC_FORCE_DIS_HW_SPEED_CHANGE;2358 speed_cntl |= PCIE_LC_SPEED_CNTL__LC_FORCE_EN_SW_SPEED_CHANGE_MASK | PCIE_LC_SPEED_CNTL__LC_FORCE_DIS_HW_SPEED_CHANGE_MASK;
2344 speed_cntl &= ~LC_FORCE_DIS_SW_SPEED_CHANGE;2359 speed_cntl &= ~PCIE_LC_SPEED_CNTL__LC_FORCE_DIS_SW_SPEED_CHANGE_MASK;
2345 WREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL, speed_cntl);2360 WREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL, speed_cntl);
2346 2361
2347 tmp16 = 0;2362 tmp16 = 0;
2348 if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)2363 if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3)
@@ -2354,13 +2369,13 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 5 unmodified lines
2354 pcie_capability_clear_and_set_word(adev->pdev, PCI_EXP_LNKCTL2,2369 pcie_capability_clear_and_set_word(adev->pdev, PCI_EXP_LNKCTL2,
2355 PCI_EXP_LNKCTL2_TLS, tmp16);2370 PCI_EXP_LNKCTL2_TLS, tmp16);
2356 2371
2357 speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL);2372 speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL);
2358 speed_cntl |= LC_INITIATE_LINK_SPEED_CHANGE;2373 speed_cntl |= PCIE_LC_SPEED_CNTL__LC_INITIATE_LINK_SPEED_CHANGE_MASK;
2359 WREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL, speed_cntl);2374 WREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL, speed_cntl);
2360 2375
2361 for (i = 0; i < adev->usec_timeout; i++) {2376 for (i = 0; i < adev->usec_timeout; i++) {
2362 speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL);2377 speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL);
2363 if ((speed_cntl & LC_INITIATE_LINK_SPEED_CHANGE) == 0)2378 if ((speed_cntl & PCIE_LC_SPEED_CNTL__LC_INITIATE_LINK_SPEED_CHANGE_MASK) == 0)
2364 break;2379 break;
2365 udelay(1);2380 udelay(1);
2366 }2381 }
@@ -2418,121 +2433,121 @@ static void si_program_aspm(struct amdgpu_device *adev) · 51 unmodified lines
2418 if (!amdgpu_device_should_use_aspm(adev))2433 if (!amdgpu_device_should_use_aspm(adev))
2419 return;2434 return;
2420 2435
2421 orig = data = RREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL);2436 orig = data = RREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL);
2422 data &= ~LC_XMIT_N_FTS_MASK;2437 data &= ~PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS_MASK;
2423 data |= LC_XMIT_N_FTS(0x24) | LC_XMIT_N_FTS_OVERRIDE_EN;2438 data |= (0x24 << PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS__SHIFT) | PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS_OVERRIDE_EN_MASK;
2424 if (orig != data)2439 if (orig != data)
2425 WREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL, data);2440 WREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL, data);
2426 2441
2427 orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL3);2442 orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL3);
2428 data |= LC_GO_TO_RECOVERY;2443 data |= PCIE_LC_CNTL3__LC_GO_TO_RECOVERY_MASK;
2429 if (orig != data)2444 if (orig != data)
2430 WREG32_PCIE_PORT(PCIE_LC_CNTL3, data);2445 WREG32_PCIE_PORT(ixPCIE_LC_CNTL3, data);
2431 2446
2432 orig = data = RREG32_PCIE(PCIE_P_CNTL);2447 orig = data = RREG32_PCIE(ixPCIE_P_CNTL);
2433 data |= P_IGNORE_EDB_ERR;2448 data |= PCIE_P_CNTL__P_IGNORE_EDB_ERR_MASK;
2434 if (orig != data)2449 if (orig != data)
2435 WREG32_PCIE(PCIE_P_CNTL, data);2450 WREG32_PCIE(ixPCIE_P_CNTL, data);
2436 2451
2437 orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL);2452 orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL);
2438 data &= ~(LC_L0S_INACTIVITY_MASK | LC_L1_INACTIVITY_MASK);2453 data &= ~(PCIE_LC_CNTL__LC_L0S_INACTIVITY_MASK | PCIE_LC_CNTL__LC_L1_INACTIVITY_MASK);
2439 data |= LC_PMI_TO_L1_DIS;2454 data |= PCIE_LC_CNTL__LC_PMI_TO_L1_DIS_MASK;
2440 if (!disable_l0s)2455 if (!disable_l0s)
2441 data |= LC_L0S_INACTIVITY(7);2456 data |= (7 << PCIE_LC_CNTL__LC_L0S_INACTIVITY__SHIFT);
2442 2457
2443 if (!disable_l1) {2458 if (!disable_l1) {
2444 data |= LC_L1_INACTIVITY(7);2459 data |= (7 << PCIE_LC_CNTL__LC_L1_INACTIVITY__SHIFT);
2445 data &= ~LC_PMI_TO_L1_DIS;2460 data &= ~PCIE_LC_CNTL__LC_PMI_TO_L1_DIS_MASK;
2446 if (orig != data)2461 if (orig != data)
2447 WREG32_PCIE_PORT(PCIE_LC_CNTL, data);2462 WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data);
2448 2463
2449 if (!disable_plloff_in_l1) {2464 if (!disable_plloff_in_l1) {
2450 bool clk_req_support;2465 bool clk_req_support;
2451 2466
2452 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_0);2467 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_0);
2453 data &= ~(PLL_POWER_STATE_IN_OFF_0_MASK | PLL_POWER_STATE_IN_TXS2_0_MASK);2468 data &= ~(PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0_MASK | PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0_MASK);
2454 data |= PLL_POWER_STATE_IN_OFF_0(7) | PLL_POWER_STATE_IN_TXS2_0(7);2469 data |= (7 << PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0__SHIFT) | (7 << PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0__SHIFT);
2455 if (orig != data)2470 if (orig != data)
2456 si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_0, data);2471 si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_0, data);
2457 2472
2458 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_1);2473 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_1);
2459 data &= ~(PLL_POWER_STATE_IN_OFF_1_MASK | PLL_POWER_STATE_IN_TXS2_1_MASK);2474 data &= ~(PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1_MASK | PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1_MASK);
2460 data |= PLL_POWER_STATE_IN_OFF_1(7) | PLL_POWER_STATE_IN_TXS2_1(7);2475 data |= (7 << PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1__SHIFT) | (7 << PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1__SHIFT);
2461 if (orig != data)2476 if (orig != data)
2462 si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_1, data);2477 si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_1, data);
2463 2478
2464 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_0);2479 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_0);
2465 data &= ~(PLL_POWER_STATE_IN_OFF_0_MASK | PLL_POWER_STATE_IN_TXS2_0_MASK);2480 data &= ~(PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0_MASK | PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0_MASK);
2466 data |= PLL_POWER_STATE_IN_OFF_0(7) | PLL_POWER_STATE_IN_TXS2_0(7);2481 data |= (7 << PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0__SHIFT) | (7 << PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0__SHIFT);
2467 if (orig != data)2482 if (orig != data)
2468 si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_0, data);2483 si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_0, data);
2469 2484
2470 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_1);2485 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_1);
2471 data &= ~(PLL_POWER_STATE_IN_OFF_1_MASK | PLL_POWER_STATE_IN_TXS2_1_MASK);2486 data &= ~(PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1_MASK | PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1_MASK);
2472 data |= PLL_POWER_STATE_IN_OFF_1(7) | PLL_POWER_STATE_IN_TXS2_1(7);2487 data |= (7 << PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1__SHIFT) | (7 << PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1__SHIFT);
2473 if (orig != data)2488 if (orig != data)
2474 si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_1, data);2489 si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_1, data);
2475 2490
2476 if ((adev->asic_type != CHIP_OLAND) && (adev->asic_type != CHIP_HAINAN)) {2491 if ((adev->asic_type != CHIP_OLAND) && (adev->asic_type != CHIP_HAINAN)) {
2477 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_0);2492 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_0);
2478 data &= ~PLL_RAMP_UP_TIME_0_MASK;2493 data &= ~PB0_PIF_PWRDOWN_0__PLL_RAMP_UP_TIME_0_MASK;
2479 if (orig != data)2494 if (orig != data)
2480 si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_0, data);2495 si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_0, data);
2481 2496
2482 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_1);2497 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_1);
2483 data &= ~PLL_RAMP_UP_TIME_1_MASK;2498 data &= ~PB0_PIF_PWRDOWN_1__PLL_RAMP_UP_TIME_1_MASK;
2484 if (orig != data)2499 if (orig != data)
2485 si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_1, data);2500 si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_1, data);
2486 2501
2487 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_2);2502 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_2);
2488 data &= ~PLL_RAMP_UP_TIME_2_MASK;2503 data &= ~PB0_PIF_PWRDOWN_2__PLL_RAMP_UP_TIME_2_MASK;
2489 if (orig != data)2504 if (orig != data)
2490 si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_2, data);2505 si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_2, data);
2491 2506
2492 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_3);2507 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_3);
2493 data &= ~PLL_RAMP_UP_TIME_3_MASK;2508 data &= ~PB0_PIF_PWRDOWN_3__PLL_RAMP_UP_TIME_3_MASK;
2494 if (orig != data)2509 if (orig != data)
2495 si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_3, data);2510 si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_3, data);
2496 2511
2497 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_0);2512 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_0);
2498 data &= ~PLL_RAMP_UP_TIME_0_MASK;2513 data &= ~PB1_PIF_PWRDOWN_0__PLL_RAMP_UP_TIME_0_MASK;
2499 if (orig != data)2514 if (orig != data)
2500 si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_0, data);2515 si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_0, data);
2501 2516
2502 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_1);2517 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_1);
2503 data &= ~PLL_RAMP_UP_TIME_1_MASK;2518 data &= ~PB1_PIF_PWRDOWN_1__PLL_RAMP_UP_TIME_1_MASK;
2504 if (orig != data)2519 if (orig != data)
2505 si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_1, data);2520 si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_1, data);
2506 2521
2507 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_2);2522 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_2);
2508 data &= ~PLL_RAMP_UP_TIME_2_MASK;2523 data &= ~PB1_PIF_PWRDOWN_2__PLL_RAMP_UP_TIME_2_MASK;
2509 if (orig != data)2524 if (orig != data)
2510 si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_2, data);2525 si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_2, data);
2511 2526
2512 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_3);2527 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_3);
2513 data &= ~PLL_RAMP_UP_TIME_3_MASK;2528 data &= ~PB1_PIF_PWRDOWN_3__PLL_RAMP_UP_TIME_3_MASK;
2514 if (orig != data)2529 if (orig != data)
2515 si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_3, data);2530 si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_3, data);
2516 }2531 }
2517 orig = data = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL);2532 orig = data = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL);
2518 data &= ~LC_DYN_LANES_PWR_STATE_MASK;2533 data &= ~PCIE_LC_LINK_WIDTH_CNTL__LC_DYN_LANES_PWR_STATE_MASK;
2519 data |= LC_DYN_LANES_PWR_STATE(3);2534 data |= (3 << PCIE_LC_LINK_WIDTH_CNTL__LC_DYN_LANES_PWR_STATE__SHIFT);
2520 if (orig != data)2535 if (orig != data)
2521 WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, data);2536 WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, data);
2522 2537
2523 orig = data = si_pif_phy0_rreg(adev,PB0_PIF_CNTL);2538 orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_CNTL);
2524 data &= ~LS2_EXIT_TIME_MASK;2539 data &= ~PB0_PIF_CNTL__LS2_EXIT_TIME_MASK;
2525 if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN))2540 if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN))
2526 data |= LS2_EXIT_TIME(5);2541 data |= (5 << PB0_PIF_CNTL__LS2_EXIT_TIME__SHIFT);
2527 if (orig != data)2542 if (orig != data)
2528 si_pif_phy0_wreg(adev,PB0_PIF_CNTL, data);2543 si_pif_phy0_wreg(adev,ixPB0_PIF_CNTL, data);
2529 2544
2530 orig = data = si_pif_phy1_rreg(adev,PB1_PIF_CNTL);2545 orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_CNTL);
2531 data &= ~LS2_EXIT_TIME_MASK;2546 data &= ~PB1_PIF_CNTL__LS2_EXIT_TIME_MASK;
2532 if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN))2547 if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN))
2533 data |= LS2_EXIT_TIME(5);2548 data |= (5 << PB1_PIF_CNTL__LS2_EXIT_TIME__SHIFT);
2534 if (orig != data)2549 if (orig != data)
2535 si_pif_phy1_wreg(adev,PB1_PIF_CNTL, data);2550 si_pif_phy1_wreg(adev,ixPB1_PIF_CNTL, data);
2536 2551
2537 if (!disable_clkreq &&2552 if (!disable_clkreq &&
2538 !pci_is_root_bus(adev->pdev->bus)) {2553 !pci_is_root_bus(adev->pdev->bus)) {
@@ -2548,64 +2563,64 @@ static void si_program_aspm(struct amdgpu_device *adev) · 9 unmodified lines
2548 }2563 }
2549 2564
2550 if (clk_req_support) {2565 if (clk_req_support) {
2551 orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL2);2566 orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL2);
2552 data |= LC_ALLOW_PDWN_IN_L1 | LC_ALLOW_PDWN_IN_L23;2567 data |= PCIE_LC_CNTL2__LC_ALLOW_PDWN_IN_L1_MASK | PCIE_LC_CNTL2__LC_ALLOW_PDWN_IN_L23_MASK;
2553 if (orig != data)2568 if (orig != data)
2554 WREG32_PCIE_PORT(PCIE_LC_CNTL2, data);2569 WREG32_PCIE_PORT(ixPCIE_LC_CNTL2, data);
2555 2570
2556 orig = data = RREG32(THM_CLK_CNTL);2571 orig = data = RREG32(mmTHM_CLK_CNTL);
2557 data &= ~(CMON_CLK_SEL_MASK | TMON_CLK_SEL_MASK);2572 data &= ~(THM_CLK_CNTL__CMON_CLK_SEL_MASK | THM_CLK_CNTL__TMON_CLK_SEL_MASK);
2558 data |= CMON_CLK_SEL(1) | TMON_CLK_SEL(1);2573 data |= (1 << THM_CLK_CNTL__CMON_CLK_SEL__SHIFT) | (1 << THM_CLK_CNTL__TMON_CLK_SEL__SHIFT);
2559 if (orig != data)2574 if (orig != data)
2560 WREG32(THM_CLK_CNTL, data);2575 WREG32(mmTHM_CLK_CNTL, data);
2561 2576
2562 orig = data = RREG32(MISC_CLK_CNTL);2577 orig = data = RREG32(mmMISC_CLK_CNTL);
2563 data &= ~(DEEP_SLEEP_CLK_SEL_MASK | ZCLK_SEL_MASK);2578 data &= ~(MISC_CLK_CNTL__DEEP_SLEEP_CLK_SEL_MASK | MISC_CLK_CNTL__ZCLK_SEL_MASK);
2564 data |= DEEP_SLEEP_CLK_SEL(1) | ZCLK_SEL(1);2579 data |= (1 << MISC_CLK_CNTL__DEEP_SLEEP_CLK_SEL__SHIFT) | (1 << MISC_CLK_CNTL__ZCLK_SEL__SHIFT);
2565 if (orig != data)2580 if (orig != data)
2566 WREG32(MISC_CLK_CNTL, data);2581 WREG32(mmMISC_CLK_CNTL, data);
2567 2582
2568 orig = data = RREG32(CG_CLKPIN_CNTL);2583 orig = data = RREG32(mmCG_CLKPIN_CNTL);
2569 data &= ~BCLK_AS_XCLK;2584 data &= ~CG_CLKPIN_CNTL__BCLK_AS_XCLK_MASK;
2570 if (orig != data)2585 if (orig != data)
2571 WREG32(CG_CLKPIN_CNTL, data);2586 WREG32(mmCG_CLKPIN_CNTL, data);
2572 2587
2573 orig = data = RREG32(CG_CLKPIN_CNTL_2);2588 orig = data = RREG32(mmCG_CLKPIN_CNTL_2);
2574 data &= ~FORCE_BIF_REFCLK_EN;2589 data &= ~CG_CLKPIN_CNTL_2__FORCE_BIF_REFCLK_EN_MASK;
2575 if (orig != data)2590 if (orig != data)
2576 WREG32(CG_CLKPIN_CNTL_2, data);2591 WREG32(mmCG_CLKPIN_CNTL_2, data);
2577 2592
2578 orig = data = RREG32(MPLL_BYPASSCLK_SEL);2593 orig = data = RREG32(mmMPLL_BYPASSCLK_SEL);
2579 data &= ~MPLL_CLKOUT_SEL_MASK;2594 data &= ~MPLL_BYPASSCLK_SEL__MPLL_CLKOUT_SEL_MASK;
2580 data |= MPLL_CLKOUT_SEL(4);2595 data |= 4 << MPLL_BYPASSCLK_SEL__MPLL_CLKOUT_SEL__SHIFT;
2581 if (orig != data)2596 if (orig != data)
2582 WREG32(MPLL_BYPASSCLK_SEL, data);2597 WREG32(mmMPLL_BYPASSCLK_SEL, data);
2583 2598
2584 orig = data = RREG32(SPLL_CNTL_MODE);2599 orig = data = RREG32(mmSPLL_CNTL_MODE);
2585 data &= ~SPLL_REFCLK_SEL_MASK;2600 data &= ~SPLL_CNTL_MODE__SPLL_REFCLK_SEL_MASK;
2586 if (orig != data)2601 if (orig != data)
2587 WREG32(SPLL_CNTL_MODE, data);2602 WREG32(mmSPLL_CNTL_MODE, data);
2588 }2603 }
2589 }2604 }
2590 } else {2605 } else {
2591 if (orig != data)2606 if (orig != data)
2592 WREG32_PCIE_PORT(PCIE_LC_CNTL, data);2607 WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data);
2593 }2608 }
2594 2609
2595 orig = data = RREG32_PCIE(PCIE_CNTL2);2610 orig = data = RREG32_PCIE(ixPCIE_CNTL2);
2596 data |= SLV_MEM_LS_EN | MST_MEM_LS_EN | REPLAY_MEM_LS_EN;2611 data |= PCIE_CNTL2__SLV_MEM_LS_EN_MASK | PCIE_CNTL2__MST_MEM_LS_EN_MASK | PCIE_CNTL2__REPLAY_MEM_LS_EN_MASK;
2597 if (orig != data)2612 if (orig != data)
2598 WREG32_PCIE(PCIE_CNTL2, data);2613 WREG32_PCIE(ixPCIE_CNTL2, data);
2599 2614
2600 if (!disable_l0s) {2615 if (!disable_l0s) {
2601 data = RREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL);2616 data = RREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL);
2602 if((data & LC_N_FTS_MASK) == LC_N_FTS_MASK) {2617 if((data & PCIE_LC_N_FTS_CNTL__LC_N_FTS_MASK) == PCIE_LC_N_FTS_CNTL__LC_N_FTS_MASK) {
2603 data = RREG32_PCIE(PCIE_LC_STATUS1);2618 data = RREG32_PCIE(ixPCIE_LC_STATUS1);
2604 if ((data & LC_REVERSE_XMIT) && (data & LC_REVERSE_RCVR)) {2619 if ((data & PCIE_LC_STATUS1__LC_REVERSE_XMIT_MASK) && (data & PCIE_LC_STATUS1__LC_REVERSE_RCVR_MASK)) {
2605 orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL);2620 orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL);
2606 data &= ~LC_L0S_INACTIVITY_MASK;2621 data &= ~PCIE_LC_CNTL__LC_L0S_INACTIVITY_MASK;
2607 if (orig != data)2622 if (orig != data)
2608 WREG32_PCIE_PORT(PCIE_LC_CNTL, data);2623 WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data);
2609 }2624 }
2610 }2625 }
2611 }2626 }
drivers/gpu/drm/amd/amdgpu/si_dma.c+66−50
@@ -27,6 +27,8 @@ · 26 unmodified lines
27#include "si.h"27#include "si.h"
28#include "sid.h"28#include "sid.h"
29 29
30#include "oss/oss_1_0_d.h"
31#include "oss/oss_1_0_sh_mask.h"
30const u32 sdma_offsets[SDMA_MAX_INSTANCE] =32const u32 sdma_offsets[SDMA_MAX_INSTANCE] =
31{33{
32 DMA0_REGISTER_OFFSET,34 DMA0_REGISTER_OFFSET,
@@ -38,17 +40,31 @@ static void si_dma_set_buffer_funcs(struct amdgpu_device *adev); · 5 unmodified lines
38static void si_dma_set_vm_pte_funcs(struct amdgpu_device *adev);40static void si_dma_set_vm_pte_funcs(struct amdgpu_device *adev);
39static void si_dma_set_irq_funcs(struct amdgpu_device *adev);41static void si_dma_set_irq_funcs(struct amdgpu_device *adev);
40 42
43/**
44 * si_dma_ring_get_rptr - get the current read pointer
45 *
46 * @ring: amdgpu ring pointer
47 *
48 * Get the current rptr from the hardware (SI).
49 */
41static uint64_t si_dma_ring_get_rptr(struct amdgpu_ring *ring)50static uint64_t si_dma_ring_get_rptr(struct amdgpu_ring *ring)
42{51{
43 return *ring->rptr_cpu_addr;52 return *ring->rptr_cpu_addr;
44}53}
45 54
55/**
56 * si_dma_ring_get_wptr - get the current write pointer
57 *
58 * @ring: amdgpu ring pointer
59 *
60 * Get the current wptr from the hardware (SI).
61 */
46static uint64_t si_dma_ring_get_wptr(struct amdgpu_ring *ring)62static uint64_t si_dma_ring_get_wptr(struct amdgpu_ring *ring)
47{63{
48 struct amdgpu_device *adev = ring->adev;64 struct amdgpu_device *adev = ring->adev;
49 u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1;65 u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1;
50 66
51 return (RREG32(DMA_RB_WPTR + sdma_offsets[me]) & 0x3fffc) >> 2;67 return (RREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[me]) & 0x3fffc) >> 2;
52}68}
53 69
54static void si_dma_ring_set_wptr(struct amdgpu_ring *ring)70static void si_dma_ring_set_wptr(struct amdgpu_ring *ring)
@@ -56,7 +72,7 @@ static void si_dma_ring_set_wptr(struct amdgpu_ring *ring) · 1 unmodified line
56 struct amdgpu_device *adev = ring->adev;72 struct amdgpu_device *adev = ring->adev;
57 u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1;73 u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1;
58 74
59 WREG32(DMA_RB_WPTR + sdma_offsets[me], (ring->wptr << 2) & 0x3fffc);75 WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[me], (ring->wptr << 2) & 0x3fffc);
60}76}
61 77
62static void si_dma_ring_emit_ib(struct amdgpu_ring *ring,78static void si_dma_ring_emit_ib(struct amdgpu_ring *ring,
@@ -117,9 +133,9 @@ static void si_dma_stop(struct amdgpu_device *adev) · 54 unmodified lines
117 133
118 for (i = 0; i < adev->sdma.num_instances; i++) {134 for (i = 0; i < adev->sdma.num_instances; i++) {
119 /* dma0 */135 /* dma0 */
120 rb_cntl = RREG32(DMA_RB_CNTL + sdma_offsets[i]);136 rb_cntl = RREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i]);
121 rb_cntl &= ~DMA_RB_ENABLE;137 rb_cntl &= ~DMA_GFX_RB_CNTL__RB_ENABLE_MASK;
122 WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl);138 WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl);
123 }139 }
124}140}
125 141
@@ -133,44 +149,44 @@ static int si_dma_start(struct amdgpu_device *adev) · 7 unmodified lines
133 for (i = 0; i < adev->sdma.num_instances; i++) {149 for (i = 0; i < adev->sdma.num_instances; i++) {
134 ring = &adev->sdma.instance[i].ring;150 ring = &adev->sdma.instance[i].ring;
135 151
136 WREG32(DMA_SEM_INCOMPLETE_TIMER_CNTL + sdma_offsets[i], 0);152 WREG32(mmDMA_SEM_INCOMPLETE_TIMER_CNTL + sdma_offsets[i], 0);
137 WREG32(DMA_SEM_WAIT_FAIL_TIMER_CNTL + sdma_offsets[i], 0);153 WREG32(mmDMA_SEM_WAIT_FAIL_TIMER_CNTL + sdma_offsets[i], 0);
138 154
139 /* Set ring buffer size in dwords */155 /* Set ring buffer size in dwords */
140 rb_bufsz = order_base_2(ring->ring_size / 4);156 rb_bufsz = order_base_2(ring->ring_size / 4);
141 rb_cntl = rb_bufsz << 1;157 rb_cntl = rb_bufsz << 1;
142#ifdef __BIG_ENDIAN158#ifdef __BIG_ENDIAN
143 rb_cntl |= DMA_RB_SWAP_ENABLE | DMA_RPTR_WRITEBACK_SWAP_ENABLE;159 rb_cntl |= DMA_GFX_RB_CNTL__RB_SWAP_ENABLE_MASK | DMA_GFX_RB_CNTL__RPTR_WRITEBACK_SWAP_ENABLE_MASK;
144#endif160#endif
145 WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl);161 WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl);
146 162
147 /* Initialize the ring buffer's read and write pointers */163 /* Initialize the ring buffer's read and write pointers */
148 WREG32(DMA_RB_RPTR + sdma_offsets[i], 0);164 WREG32(mmDMA_GFX_RB_RPTR + sdma_offsets[i], 0);
149 WREG32(DMA_RB_WPTR + sdma_offsets[i], 0);165 WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[i], 0);
150 166
151 rptr_addr = ring->rptr_gpu_addr;167 rptr_addr = ring->rptr_gpu_addr;
152 168
153 WREG32(DMA_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr));169 WREG32(mmDMA_GFX_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr));
154 WREG32(DMA_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF);170 WREG32(mmDMA_GFX_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF);
155 171
156 rb_cntl |= DMA_RPTR_WRITEBACK_ENABLE;172 rb_cntl |= DMA_GFX_RB_CNTL__RPTR_WRITEBACK_ENABLE_MASK;
157 173
158 WREG32(DMA_RB_BASE + sdma_offsets[i], ring->gpu_addr >> 8);174 WREG32(mmDMA_GFX_RB_BASE + sdma_offsets[i], ring->gpu_addr >> 8);
159 175
160 /* enable DMA IBs */176 /* enable DMA IBs */
161 ib_cntl = DMA_IB_ENABLE | CMD_VMID_FORCE;177 ib_cntl = DMA_GFX_IB_CNTL__IB_ENABLE_MASK | DMA_GFX_IB_CNTL__CMD_VMID_FORCE_MASK;
162#ifdef __BIG_ENDIAN178#ifdef __BIG_ENDIAN
163 ib_cntl |= DMA_IB_SWAP_ENABLE;179 ib_cntl |= DMA_GFX_IB_CNTL__IB_SWAP_ENABLE_MASK;
164#endif180#endif
165 WREG32(DMA_IB_CNTL + sdma_offsets[i], ib_cntl);181 WREG32(mmDMA_GFX_IB_CNTL + sdma_offsets[i], ib_cntl);
166 182
167 dma_cntl = RREG32(DMA_CNTL + sdma_offsets[i]);183 dma_cntl = RREG32(mmDMA_CNTL + sdma_offsets[i]);
168 dma_cntl &= ~CTXEMPTY_INT_ENABLE;184 dma_cntl &= ~DMA_CNTL__CTXEMPTY_INT_ENABLE_MASK;
169 WREG32(DMA_CNTL + sdma_offsets[i], dma_cntl);185 WREG32(mmDMA_CNTL + sdma_offsets[i], dma_cntl);
170 186
171 ring->wptr = 0;187 ring->wptr = 0;
172 WREG32(DMA_RB_WPTR + sdma_offsets[i], ring->wptr << 2);188 WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[i], ring->wptr << 2);
173 WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl | DMA_RB_ENABLE);189 WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl | DMA_GFX_RB_CNTL__RB_ENABLE_MASK);
174 190
175 r = amdgpu_ring_test_helper(ring);191 r = amdgpu_ring_test_helper(ring);
176 if (r)192 if (r)
@@ -461,7 +477,7 @@ static int si_dma_early_init(struct amdgpu_ip_block *ip_block) · 284 unmodified lines
461{477{
462 struct amdgpu_device *adev = ip_block->adev;478 struct amdgpu_device *adev = ip_block->adev;
463 479
464 adev->sdma.num_instances = 2;480 adev->sdma.num_instances = SDMA_MAX_INSTANCE;
465 481
466 si_dma_set_ring_funcs(adev);482 si_dma_set_ring_funcs(adev);
467 si_dma_set_buffer_funcs(adev);483 si_dma_set_buffer_funcs(adev);
@@ -545,9 +561,9 @@ static bool si_dma_is_idle(struct amdgpu_ip_block *ip_block) · 77 unmodified lines
545{561{
546 struct amdgpu_device *adev = ip_block->adev;562 struct amdgpu_device *adev = ip_block->adev;
547 563
548 u32 tmp = RREG32(SRBM_STATUS2);564 u32 tmp = RREG32(mmSRBM_STATUS2);
549 565
550 if (tmp & (DMA_BUSY_MASK | DMA1_BUSY_MASK))566 if (tmp & (SRBM_STATUS2__DMA_BUSY_MASK | SRBM_STATUS2__DMA1_BUSY_MASK))
551 return false;567 return false;
552 568
553 return true;569 return true;
@@ -583,14 +599,14 @@ static int si_dma_set_trap_irq_state(struct amdgpu_device *adev, · 29 unmodified lines
583 case AMDGPU_SDMA_IRQ_INSTANCE0:599 case AMDGPU_SDMA_IRQ_INSTANCE0:
584 switch (state) {600 switch (state) {
585 case AMDGPU_IRQ_STATE_DISABLE:601 case AMDGPU_IRQ_STATE_DISABLE:
586 sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET);602 sdma_cntl = RREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET);
587 sdma_cntl &= ~TRAP_ENABLE;603 sdma_cntl &= ~DMA_CNTL__TRAP_ENABLE_MASK;
588 WREG32(DMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);604 WREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);
589 break;605 break;
590 case AMDGPU_IRQ_STATE_ENABLE:606 case AMDGPU_IRQ_STATE_ENABLE:
591 sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET);607 sdma_cntl = RREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET);
592 sdma_cntl |= TRAP_ENABLE;608 sdma_cntl |= DMA_CNTL__TRAP_ENABLE_MASK;
593 WREG32(DMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);609 WREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl);
594 break;610 break;
595 default:611 default:
596 break;612 break;
@@ -599,14 +615,14 @@ static int si_dma_set_trap_irq_state(struct amdgpu_device *adev, · 2 unmodified lines
599 case AMDGPU_SDMA_IRQ_INSTANCE1:615 case AMDGPU_SDMA_IRQ_INSTANCE1:
600 switch (state) {616 switch (state) {
601 case AMDGPU_IRQ_STATE_DISABLE:617 case AMDGPU_IRQ_STATE_DISABLE:
602 sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET);618 sdma_cntl = RREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET);
603 sdma_cntl &= ~TRAP_ENABLE;619 sdma_cntl &= ~DMA_CNTL__TRAP_ENABLE_MASK;
604 WREG32(DMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);620 WREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);
605 break;621 break;
606 case AMDGPU_IRQ_STATE_ENABLE:622 case AMDGPU_IRQ_STATE_ENABLE:
607 sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET);623 sdma_cntl = RREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET);
608 sdma_cntl |= TRAP_ENABLE;624 sdma_cntl |= DMA_CNTL__TRAP_ENABLE_MASK;
609 WREG32(DMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);625 WREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl);
610 break;626 break;
611 default:627 default:
612 break;628 break;
@@ -645,11 +661,11 @@ static int si_dma_set_clockgating_state(struct amdgpu_ip_block *ip_block, · 32 unmodified lines
645 offset = DMA0_REGISTER_OFFSET;661 offset = DMA0_REGISTER_OFFSET;
646 else662 else
647 offset = DMA1_REGISTER_OFFSET;663 offset = DMA1_REGISTER_OFFSET;
648 orig = data = RREG32(DMA_POWER_CNTL + offset);664 orig = data = RREG32(mmDMA_POWER_CNTL + offset);
649 data &= ~MEM_POWER_OVERRIDE;665 data &= ~DMA_POWER_CNTL__MEM_POWER_OVERRIDE_MASK;
650 if (data != orig)666 if (data != orig)
651 WREG32(DMA_POWER_CNTL + offset, data);667 WREG32(mmDMA_POWER_CNTL + offset, data);
652 WREG32(DMA_CLK_CTRL + offset, 0x00000100);668 WREG32(mmDMA_CLK_CTRL + offset, 0x00000100);
653 }669 }
654 } else {670 } else {
655 for (i = 0; i < adev->sdma.num_instances; i++) {671 for (i = 0; i < adev->sdma.num_instances; i++) {
@@ -657,15 +673,15 @@ static int si_dma_set_clockgating_state(struct amdgpu_ip_block *ip_block, · 1 unmodified line
657 offset = DMA0_REGISTER_OFFSET;673 offset = DMA0_REGISTER_OFFSET;
658 else674 else
659 offset = DMA1_REGISTER_OFFSET;675 offset = DMA1_REGISTER_OFFSET;
660 orig = data = RREG32(DMA_POWER_CNTL + offset);676 orig = data = RREG32(mmDMA_POWER_CNTL + offset);
661 data |= MEM_POWER_OVERRIDE;677 data |= DMA_POWER_CNTL__MEM_POWER_OVERRIDE_MASK;
662 if (data != orig)678 if (data != orig)
663 WREG32(DMA_POWER_CNTL + offset, data);679 WREG32(mmDMA_POWER_CNTL + offset, data);
664 680
665 orig = data = RREG32(DMA_CLK_CTRL + offset);681 orig = data = RREG32(mmDMA_CLK_CTRL + offset);
666 data = 0xff000000;682 data = 0xff000000;
667 if (data != orig)683 if (data != orig)
668 WREG32(DMA_CLK_CTRL + offset, data);684 WREG32(mmDMA_CLK_CTRL + offset, data);
669 }685 }
670 }686 }
671 687
@@ -679,11 +695,11 @@ static int si_dma_set_powergating_state(struct amdgpu_ip_block *ip_block, · 7 unmodified lines
679 695
680 struct amdgpu_device *adev = ip_block->adev;696 struct amdgpu_device *adev = ip_block->adev;
681 697
682 WREG32(DMA_PGFSM_WRITE, 0x00002000);698 WREG32(mmDMA_PGFSM_WRITE, 0x00002000);
683 WREG32(DMA_PGFSM_CONFIG, 0x100010ff);699 WREG32(mmDMA_PGFSM_CONFIG, 0x100010ff);
684 700
685 for (tmp = 0; tmp < 5; tmp++)701 for (tmp = 0; tmp < 5; tmp++)
686 WREG32(DMA_PGFSM_WRITE, 0);702 WREG32(mmDMA_PGFSM_WRITE, 0);
687 703
688 return 0;704 return 0;
689}705}
drivers/gpu/drm/amd/amdgpu/si_enums.h+1−233
@@ -23,115 +23,15 @@ · 22 unmodified lines
23#ifndef SI_ENUMS_H23#ifndef SI_ENUMS_H
24#define SI_ENUMS_H24#define SI_ENUMS_H
25 25
26#define VBLANK_INT_MASK (1 << 0)
27#define DC_HPDx_INT_EN (1 << 16)
28#define VBLANK_ACK (1 << 4)
29#define VLINE_ACK (1 << 4)
30
31#define CURSOR_WIDTH 64
32#define CURSOR_HEIGHT 64
33
34#define VGA_VSTATUS_CNTL 0xFFFCFFFF
35#define PRIORITY_MARK_MASK 0x7fff26#define PRIORITY_MARK_MASK 0x7fff
36#define PRIORITY_OFF (1 << 16)27#define PRIORITY_OFF (1 << 16)
37#define PRIORITY_ALWAYS_ON (1 << 20)28#define PRIORITY_ALWAYS_ON (1 << 20)
38#define INTERLEAVE_EN (1 << 0)
39
40#define LATENCY_WATERMARK_MASK(x) ((x) << 16)
41#define DC_LB_MEMORY_CONFIG(x) ((x) << 20)
42#define ICON_DEGAMMA_MODE(x) (((x) & 0x3) << 8)
43
44#define GRPH_ENDIAN_SWAP(x) (((x) & 0x3) << 0)
45#define GRPH_ENDIAN_NONE 0
46#define GRPH_ENDIAN_8IN16 1
47#define GRPH_ENDIAN_8IN32 2
48#define GRPH_ENDIAN_8IN64 3
49#define GRPH_RED_CROSSBAR(x) (((x) & 0x3) << 4)
50#define GRPH_RED_SEL_R 0
51#define GRPH_RED_SEL_G 1
52#define GRPH_RED_SEL_B 2
53#define GRPH_RED_SEL_A 3
54#define GRPH_GREEN_CROSSBAR(x) (((x) & 0x3) << 6)
55#define GRPH_GREEN_SEL_G 0
56#define GRPH_GREEN_SEL_B 1
57#define GRPH_GREEN_SEL_A 2
58#define GRPH_GREEN_SEL_R 3
59#define GRPH_BLUE_CROSSBAR(x) (((x) & 0x3) << 8)
60#define GRPH_BLUE_SEL_B 0
61#define GRPH_BLUE_SEL_A 1
62#define GRPH_BLUE_SEL_R 2
63#define GRPH_BLUE_SEL_G 3
64#define GRPH_ALPHA_CROSSBAR(x) (((x) & 0x3) << 10)
65#define GRPH_ALPHA_SEL_A 0
66#define GRPH_ALPHA_SEL_R 1
67#define GRPH_ALPHA_SEL_G 2
68#define GRPH_ALPHA_SEL_B 3
69
70#define GRPH_DEPTH(x) (((x) & 0x3) << 0)
71#define GRPH_DEPTH_8BPP 0
72#define GRPH_DEPTH_16BPP 1
73#define GRPH_DEPTH_32BPP 2
74
75#define GRPH_FORMAT(x) (((x) & 0x7) << 8)
76#define GRPH_FORMAT_INDEXED 0
77#define GRPH_FORMAT_ARGB1555 0
78#define GRPH_FORMAT_ARGB565 1
79#define GRPH_FORMAT_ARGB4444 2
80#define GRPH_FORMAT_AI88 3
81#define GRPH_FORMAT_MONO16 4
82#define GRPH_FORMAT_BGRA5551 5
83#define GRPH_FORMAT_ARGB8888 0
84#define GRPH_FORMAT_ARGB2101010 1
85#define GRPH_FORMAT_32BPP_DIG 2
86#define GRPH_FORMAT_8B_ARGB2101010 3
87#define GRPH_FORMAT_BGRA1010102 4
88#define GRPH_FORMAT_8B_BGRA1010102 5
89#define GRPH_FORMAT_RGB111110 6
90#define GRPH_FORMAT_BGR101111 7
91
92#define GRPH_NUM_BANKS(x) (((x) & 0x3) << 2)
93#define GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20)
94#define GRPH_ARRAY_LINEAR_GENERAL 0
95#define GRPH_ARRAY_LINEAR_ALIGNED 1
96#define GRPH_ARRAY_1D_TILED_THIN1 2
97#define GRPH_ARRAY_2D_TILED_THIN1 4
98#define GRPH_TILE_SPLIT(x) (((x) & 0x7) << 13)
99#define GRPH_BANK_WIDTH(x) (((x) & 0x3) << 6)
100#define GRPH_BANK_HEIGHT(x) (((x) & 0x3) << 11)
101#define GRPH_MACRO_TILE_ASPECT(x) (((x) & 0x3) << 18)
102#define GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20)
103#define GRPH_PIPE_CONFIG(x) (((x) & 0x1f) << 24)
104
105#define CURSOR_EN (1 << 0)
106#define CURSOR_MODE(x) (((x) & 0x3) << 8)
107#define CURSOR_MONO 0
108#define CURSOR_24_1 1
109#define CURSOR_24_8_PRE_MULT 2
110#define CURSOR_24_8_UNPRE_MULT 3
111#define CURSOR_2X_MAGNIFY (1 << 16)
112#define CURSOR_FORCE_MC_ON (1 << 20)
113#define CURSOR_URGENT_CONTROL(x) (((x) & 0x7) << 24)
114#define CURSOR_URGENT_ALWAYS 0
115#define CURSOR_URGENT_1_8 1
116#define CURSOR_URGENT_1_4 2
117#define CURSOR_URGENT_3_8 3
118#define CURSOR_URGENT_1_2 4
119#define CURSOR_UPDATE_PENDING (1 << 0)
120#define CURSOR_UPDATE_TAKEN (1 << 1)
121#define CURSOR_UPDATE_LOCK (1 << 16)
122#define CURSOR_DISABLE_MULTIPLE_UPDATE (1 << 24)
123
124
125#define ES_AND_GS_AUTO 3
126#define RADEON_PACKET_TYPE3 3
127#define CE_PARTITION_BASE 3
128#define BUF_SWAP_32BIT (2 << 16)
129 29
130#define GFX_POWER_STATUS (1 << 1)30#define GFX_POWER_STATUS (1 << 1)
131#define GFX_CLOCK_STATUS (1 << 2)31#define GFX_CLOCK_STATUS (1 << 2)
132#define GFX_LS_STATUS (1 << 3)32#define GFX_LS_STATUS (1 << 3)
133#define RLC_BUSY_STATUS (1 << 0)
134 33
34#define RLC_BUSY_STATUS (1 << 0)
135#define RLC_PUD(x) ((x) << 0)35#define RLC_PUD(x) ((x) << 0)
136#define RLC_PUD_MASK (0xff << 0)36#define RLC_PUD_MASK (0xff << 0)
137#define RLC_PDD(x) ((x) << 8)37#define RLC_PDD(x) ((x) << 8)
@@ -140,140 +40,8 @@ · 2 unmodified lines
140#define RLC_TTPD_MASK (0xff << 16)40#define RLC_TTPD_MASK (0xff << 16)
141#define RLC_MSD(x) ((x) << 24)41#define RLC_MSD(x) ((x) << 24)
142#define RLC_MSD_MASK (0xff << 24)42#define RLC_MSD_MASK (0xff << 24)
143#define WRITE_DATA_ENGINE_SEL(x) ((x) << 30)
144#define WRITE_DATA_DST_SEL(x) ((x) << 8)
145#define EVENT_TYPE(x) ((x) << 0)
146#define EVENT_INDEX(x) ((x) << 8)
147#define WAIT_REG_MEM_MEM_SPACE(x) ((x) << 4)
148#define WAIT_REG_MEM_FUNCTION(x) ((x) << 0)
149#define WAIT_REG_MEM_ENGINE(x) ((x) << 8)
150 43
151#define GFX6_NUM_GFX_RINGS 1
152#define GFX6_NUM_COMPUTE_RINGS 2
153#define RLC_SAVE_AND_RESTORE_STARTING_OFFSET 0x9044#define RLC_SAVE_AND_RESTORE_STARTING_OFFSET 0x90
154#define RLC_CLEAR_STATE_DESCRIPTOR_OFFSET 0x3D45#define RLC_CLEAR_STATE_DESCRIPTOR_OFFSET 0x3D
155 46
156#define PACKET3(op, n) ((RADEON_PACKET_TYPE3 << 30) | \
157 (((op) & 0xFF) << 8) | \
158 ((n) & 0x3FFF) << 16)
159#define PACKET3_COMPUTE(op, n) (PACKET3(op, n) | 1 << 1)
160#define PACKET3_NOP 0x10
161#define PACKET3_SET_BASE 0x11
162#define PACKET3_BASE_INDEX(x) ((x) << 0)
163#define PACKET3_CLEAR_STATE 0x12
164#define PACKET3_INDEX_BUFFER_SIZE 0x13
165#define PACKET3_DISPATCH_DIRECT 0x15
166#define PACKET3_DISPATCH_INDIRECT 0x16
167#define PACKET3_ALLOC_GDS 0x1B
168#define PACKET3_WRITE_GDS_RAM 0x1C
169#define PACKET3_ATOMIC_GDS 0x1D
170#define PACKET3_ATOMIC 0x1E
171#define PACKET3_OCCLUSION_QUERY 0x1F
172#define PACKET3_SET_PREDICATION 0x20
173#define PACKET3_REG_RMW 0x21
174#define PACKET3_COND_EXEC 0x22
175#define PACKET3_PRED_EXEC 0x23
176#define PACKET3_DRAW_INDIRECT 0x24
177#define PACKET3_DRAW_INDEX_INDIRECT 0x25
178#define PACKET3_INDEX_BASE 0x26
179#define PACKET3_DRAW_INDEX_2 0x27
180#define PACKET3_CONTEXT_CONTROL 0x28
181#define PACKET3_INDEX_TYPE 0x2A
182#define PACKET3_DRAW_INDIRECT_MULTI 0x2C
183#define PACKET3_DRAW_INDEX_AUTO 0x2D
184#define PACKET3_DRAW_INDEX_IMMD 0x2E
185#define PACKET3_NUM_INSTANCES 0x2F
186#define PACKET3_DRAW_INDEX_MULTI_AUTO 0x30
187#define PACKET3_INDIRECT_BUFFER_CONST 0x31
188#define PACKET3_INDIRECT_BUFFER 0x3F
189#define PACKET3_STRMOUT_BUFFER_UPDATE 0x34
190#define PACKET3_DRAW_INDEX_OFFSET_2 0x35
191#define PACKET3_DRAW_INDEX_MULTI_ELEMENT 0x36
192#define PACKET3_WRITE_DATA 0x37
193#define PACKET3_DRAW_INDEX_INDIRECT_MULTI 0x38
194#define PACKET3_MEM_SEMAPHORE 0x39
195#define PACKET3_MPEG_INDEX 0x3A
196#define PACKET3_COPY_DW 0x3B
197#define PACKET3_WAIT_REG_MEM 0x3C
198#define PACKET3_MEM_WRITE 0x3D
199#define PACKET3_COPY_DATA 0x40
200#define PACKET3_CP_DMA 0x41
201# define PACKET3_CP_DMA_DST_SEL(x) ((x) << 20)
202# define PACKET3_CP_DMA_ENGINE(x) ((x) << 27)
203# define PACKET3_CP_DMA_SRC_SEL(x) ((x) << 29)
204# define PACKET3_CP_DMA_CP_SYNC (1 << 31)
205# define PACKET3_CP_DMA_DIS_WC (1 << 21)
206# define PACKET3_CP_DMA_CMD_SRC_SWAP(x) ((x) << 22)
207# define PACKET3_CP_DMA_CMD_DST_SWAP(x) ((x) << 24)
208# define PACKET3_CP_DMA_CMD_SAS (1 << 26)
209# define PACKET3_CP_DMA_CMD_DAS (1 << 27)
210# define PACKET3_CP_DMA_CMD_SAIC (1 << 28)
211# define PACKET3_CP_DMA_CMD_DAIC (1 << 29)
212# define PACKET3_CP_DMA_CMD_RAW_WAIT (1 << 30)
213#define PACKET3_PFP_SYNC_ME 0x42
214#define PACKET3_SURFACE_SYNC 0x43
215# define PACKET3_DEST_BASE_0_ENA (1 << 0)
216# define PACKET3_DEST_BASE_1_ENA (1 << 1)
217# define PACKET3_CB0_DEST_BASE_ENA (1 << 6)
218# define PACKET3_CB1_DEST_BASE_ENA (1 << 7)
219# define PACKET3_CB2_DEST_BASE_ENA (1 << 8)
220# define PACKET3_CB3_DEST_BASE_ENA (1 << 9)
221# define PACKET3_CB4_DEST_BASE_ENA (1 << 10)
222# define PACKET3_CB5_DEST_BASE_ENA (1 << 11)
223# define PACKET3_CB6_DEST_BASE_ENA (1 << 12)
224# define PACKET3_CB7_DEST_BASE_ENA (1 << 13)
225# define PACKET3_DB_DEST_BASE_ENA (1 << 14)
226# define PACKET3_DEST_BASE_2_ENA (1 << 19)
227# define PACKET3_DEST_BASE_3_ENA (1 << 21)
228# define PACKET3_TCL1_ACTION_ENA (1 << 22)
229# define PACKET3_TC_ACTION_ENA (1 << 23)
230# define PACKET3_CB_ACTION_ENA (1 << 25)
231# define PACKET3_DB_ACTION_ENA (1 << 26)
232# define PACKET3_SH_KCACHE_ACTION_ENA (1 << 27)
233# define PACKET3_SH_ICACHE_ACTION_ENA (1 << 29)
234#define PACKET3_ME_INITIALIZE 0x44
235#define PACKET3_ME_INITIALIZE_DEVICE_ID(x) ((x) << 16)
236#define PACKET3_COND_WRITE 0x45
237#define PACKET3_EVENT_WRITE 0x46
238#define PACKET3_EVENT_WRITE_EOP 0x47
239#define PACKET3_EVENT_WRITE_EOS 0x48
240#define PACKET3_PREAMBLE_CNTL 0x4A
241# define PACKET3_PREAMBLE_BEGIN_CLEAR_STATE (2 << 28)
242# define PACKET3_PREAMBLE_END_CLEAR_STATE (3 << 28)
243#define PACKET3_ONE_REG_WRITE 0x57
244#define PACKET3_LOAD_CONFIG_REG 0x5F
245#define PACKET3_LOAD_CONTEXT_REG 0x60
246#define PACKET3_LOAD_SH_REG 0x61
247#define PACKET3_SET_CONFIG_REG 0x68
248#define PACKET3_SET_CONFIG_REG_START 0x00002000
249#define PACKET3_SET_CONFIG_REG_END 0x00002c00
250#define PACKET3_SET_CONTEXT_REG 0x69
251#define PACKET3_SET_CONTEXT_REG_START 0x000a000
252#define PACKET3_SET_CONTEXT_REG_END 0x000a400
253#define PACKET3_SET_CONTEXT_REG_INDIRECT 0x73
254#define PACKET3_SET_RESOURCE_INDIRECT 0x74
255#define PACKET3_SET_SH_REG 0x76
256#define PACKET3_SET_SH_REG_START 0x00002c00
257#define PACKET3_SET_SH_REG_END 0x00003000
258#define PACKET3_SET_SH_REG_OFFSET 0x77
259#define PACKET3_ME_WRITE 0x7A
260#define PACKET3_SCRATCH_RAM_WRITE 0x7D
261#define PACKET3_SCRATCH_RAM_READ 0x7E
262#define PACKET3_CE_WRITE 0x7F
263#define PACKET3_LOAD_CONST_RAM 0x80
264#define PACKET3_WRITE_CONST_RAM 0x81
265#define PACKET3_WRITE_CONST_RAM_OFFSET 0x82
266#define PACKET3_DUMP_CONST_RAM 0x83
267#define PACKET3_INCREMENT_CE_COUNTER 0x84
268#define PACKET3_INCREMENT_DE_COUNTER 0x85
269#define PACKET3_WAIT_ON_CE_COUNTER 0x86
270#define PACKET3_WAIT_ON_DE_COUNTER 0x87
271#define PACKET3_WAIT_ON_DE_COUNTER_DIFF 0x88
272#define PACKET3_SET_CE_DE_COUNTERS 0x89
273#define PACKET3_WAIT_ON_AVAIL_BUFFER 0x8A
274#define PACKET3_SWITCH_BUFFER 0x8B
275#define PACKET3_SEM_WAIT_ON_SIGNAL (0x1 << 12)
276#define PACKET3_SEM_SEL_SIGNAL (0x6 << 29)
277#define PACKET3_SEM_SEL_WAIT (0x7 << 29)
278
279#endif47#endif
drivers/gpu/drm/amd/amdgpu/si_ih.c+9−8
@@ -27,6 +27,7 @@ · 26 unmodified lines
27#include "amdgpu_ih.h"27#include "amdgpu_ih.h"
28#include "sid.h"28#include "sid.h"
29#include "si_ih.h"29#include "si_ih.h"
30
30#include "oss/oss_1_0_d.h"31#include "oss/oss_1_0_d.h"
31#include "oss/oss_1_0_sh_mask.h"32#include "oss/oss_1_0_sh_mask.h"
32 33
@@ -213,7 +214,7 @@ static int si_ih_resume(struct amdgpu_ip_block *ip_block) · 180 unmodified lines
213static bool si_ih_is_idle(struct amdgpu_ip_block *ip_block)214static bool si_ih_is_idle(struct amdgpu_ip_block *ip_block)
214{215{
215 struct amdgpu_device *adev = ip_block->adev;216 struct amdgpu_device *adev = ip_block->adev;
216 u32 tmp = RREG32(SRBM_STATUS);217 u32 tmp = RREG32(mmSRBM_STATUS);
217 218
218 if (tmp & SRBM_STATUS__IH_BUSY_MASK)219 if (tmp & SRBM_STATUS__IH_BUSY_MASK)
219 return false;220 return false;
@@ -239,23 +240,23 @@ static int si_ih_soft_reset(struct amdgpu_ip_block *ip_block) · 19 unmodified lines
239 struct amdgpu_device *adev = ip_block->adev;240 struct amdgpu_device *adev = ip_block->adev;
240 241
241 u32 srbm_soft_reset = 0;242 u32 srbm_soft_reset = 0;
242 u32 tmp = RREG32(SRBM_STATUS);243 u32 tmp = RREG32(mmSRBM_STATUS);
243 244
244 if (tmp & SRBM_STATUS__IH_BUSY_MASK)245 if (tmp & SRBM_STATUS__IH_BUSY_MASK)
245 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_IH_MASK;246 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_IH_MASK;
246 247
247 if (srbm_soft_reset) {248 if (srbm_soft_reset) {
248 tmp = RREG32(SRBM_SOFT_RESET);249 tmp = RREG32(mmSRBM_SOFT_RESET);
249 tmp |= srbm_soft_reset;250 tmp |= srbm_soft_reset;
250 dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp);251 dev_info(adev->dev, "mmSRBM_SOFT_RESET=0x%08X\n", tmp);
251 WREG32(SRBM_SOFT_RESET, tmp);252 WREG32(mmSRBM_SOFT_RESET, tmp);
252 tmp = RREG32(SRBM_SOFT_RESET);253 tmp = RREG32(mmSRBM_SOFT_RESET);
253 254
254 udelay(50);255 udelay(50);
255 256
256 tmp &= ~srbm_soft_reset;257 tmp &= ~srbm_soft_reset;
257 WREG32(SRBM_SOFT_RESET, tmp);258 WREG32(mmSRBM_SOFT_RESET, tmp);
258 tmp = RREG32(SRBM_SOFT_RESET);259 tmp = RREG32(mmSRBM_SOFT_RESET);
259 260
260 udelay(50);261 udelay(50);
261 }262 }
drivers/gpu/drm/amd/amdgpu/sid.h+107−1448
@@ -24,43 +24,12 @@ · 23 unmodified lines
24#ifndef SI_H24#ifndef SI_H
25#define SI_H25#define SI_H
26 26
27#define TAHITI_RB_BITMAP_WIDTH_PER_SH 2
28
29#define SI_MAX_SH_GPRS 256
30#define SI_MAX_TEMP_GPRS 16
31#define SI_MAX_SH_THREADS 256
32#define SI_MAX_SH_STACK_ENTRIES 4096
33#define SI_MAX_FRC_EOV_CNT 16384
34#define SI_MAX_BACKENDS 8
35#define SI_MAX_BACKENDS_MASK 0xFF
36#define SI_MAX_BACKENDS_PER_SE_MASK 0x0F
37#define SI_MAX_SIMDS 12
38#define SI_MAX_SIMDS_MASK 0x0FFF
39#define SI_MAX_SIMDS_PER_SE_MASK 0x00FF
40#define SI_MAX_PIPES 8
41#define SI_MAX_PIPES_MASK 0xFF
42#define SI_MAX_PIPES_PER_SIMD_MASK 0x3F
43#define SI_MAX_LDS_NUM 0xFFFF
44#define SI_MAX_TCC 16
45#define SI_MAX_TCC_MASK 0xFFFF
46#define SI_MAX_CTLACKS_ASSERTION_WAIT 10027#define SI_MAX_CTLACKS_ASSERTION_WAIT 100
47 28
48/* SMC IND accessor regs */
49#define SMC_IND_INDEX_0 0x80
50#define SMC_IND_DATA_0 0x81
51
52#define SMC_IND_ACCESS_CNTL 0x8A
53# define AUTO_INCREMENT_IND_0 (1 << 0)
54#define SMC_MESSAGE_0 0x8B
55#define SMC_RESP_0 0x8C
56
57/* CG IND registers are accessed via SMC indirect space + SMC_CG_IND_START */29/* CG IND registers are accessed via SMC indirect space + SMC_CG_IND_START */
58#define SMC_CG_IND_START 0xc003000030#define SMC_CG_IND_START 0xc0030000
59#define SMC_CG_IND_END 0xc004000031#define SMC_CG_IND_END 0xc0040000
60 32
61#define CG_CGTT_LOCAL_0 0x400
62#define CG_CGTT_LOCAL_1 0x401
63
64/* SMC IND registers */33/* SMC IND registers */
65#define SMC_SYSCON_RESET_CNTL 0x8000000034#define SMC_SYSCON_RESET_CNTL 0x80000000
66# define RST_REG (1 << 0)35# define RST_REG (1 << 0)
@@ -68,9 +37,6 @@ · 1 unmodified line
68# define CK_DISABLE (1 << 0)37# define CK_DISABLE (1 << 0)
69# define CKEN (1 << 24)38# define CKEN (1 << 24)
70 39
71#define VGA_HDP_CONTROL 0xCA
72#define VGA_MEMORY_DISABLE (1 << 4)
73
74#define DCCG_DISP_SLOW_SELECT_REG 0x13F40#define DCCG_DISP_SLOW_SELECT_REG 0x13F
75#define DCCG_DISP1_SLOW_SELECT(x) ((x) << 0)41#define DCCG_DISP1_SLOW_SELECT(x) ((x) << 0)
76#define DCCG_DISP1_SLOW_SELECT_MASK (7 << 0)42#define DCCG_DISP1_SLOW_SELECT_MASK (7 << 0)
@@ -79,47 +45,6 @@ · 2 unmodified lines
79#define DCCG_DISP2_SLOW_SELECT_MASK (7 << 4)45#define DCCG_DISP2_SLOW_SELECT_MASK (7 << 4)
80#define DCCG_DISP2_SLOW_SELECT_SHIFT 446#define DCCG_DISP2_SLOW_SELECT_SHIFT 4
81 47
82#define CG_SPLL_FUNC_CNTL 0x180
83#define SPLL_RESET (1 << 0)
84#define SPLL_SLEEP (1 << 1)
85#define SPLL_BYPASS_EN (1 << 3)
86#define SPLL_REF_DIV(x) ((x) << 4)
87#define SPLL_REF_DIV_MASK (0x3f << 4)
88#define SPLL_PDIV_A(x) ((x) << 20)
89#define SPLL_PDIV_A_MASK (0x7f << 20)
90#define SPLL_PDIV_A_SHIFT 20
91#define CG_SPLL_FUNC_CNTL_2 0x181
92#define SCLK_MUX_SEL(x) ((x) << 0)
93#define SCLK_MUX_SEL_MASK (0x1ff << 0)
94#define SPLL_CTLREQ_CHG (1 << 23)
95#define SCLK_MUX_UPDATE (1 << 26)
96#define CG_SPLL_FUNC_CNTL_3 0x182
97#define SPLL_FB_DIV(x) ((x) << 0)
98#define SPLL_FB_DIV_MASK (0x3ffffff << 0)
99#define SPLL_FB_DIV_SHIFT 0
100#define SPLL_DITHEN (1 << 28)
101#define CG_SPLL_FUNC_CNTL_4 0x183
102
103#define SPLL_STATUS 0x185
104#define SPLL_CHG_STATUS (1 << 1)
105#define SPLL_CNTL_MODE 0x186
106#define SPLL_SW_DIR_CONTROL (1 << 0)
107# define SPLL_REFCLK_SEL(x) ((x) << 26)
108# define SPLL_REFCLK_SEL_MASK (3 << 26)
109
110#define CG_SPLL_SPREAD_SPECTRUM 0x188
111#define SSEN (1 << 0)
112#define CLK_S(x) ((x) << 4)
113#define CLK_S_MASK (0xfff << 4)
114#define CLK_S_SHIFT 4
115#define CG_SPLL_SPREAD_SPECTRUM_2 0x189
116#define CLK_V(x) ((x) << 0)
117#define CLK_V_MASK (0x3ffffff << 0)
118#define CLK_V_SHIFT 0
119
120#define CG_SPLL_AUTOSCALE_CNTL 0x18b
121# define AUTOSCALE_ON_SS_CLEAR (1 << 9)
122
123/* discrete uvd clocks */48/* discrete uvd clocks */
124#define CG_UPLL_FUNC_CNTL 0x18d49#define CG_UPLL_FUNC_CNTL 0x18d
125# define UPLL_RESET_MASK 0x0000000150# define UPLL_RESET_MASK 0x00000001
@@ -149,317 +74,13 @@ · 23 unmodified lines
149#define CG_UPLL_SPREAD_SPECTRUM 0x19474#define CG_UPLL_SPREAD_SPECTRUM 0x194
150# define SSEN_MASK 0x0000000175# define SSEN_MASK 0x00000001
151 76
152#define MPLL_BYPASSCLK_SEL 0x197
153# define MPLL_CLKOUT_SEL(x) ((x) << 8)
154# define MPLL_CLKOUT_SEL_MASK 0xFF00
155
156#define CG_CLKPIN_CNTL 0x198
157# define XTALIN_DIVIDE (1 << 1)
158# define BCLK_AS_XCLK (1 << 2)
159#define CG_CLKPIN_CNTL_2 0x199
160# define FORCE_BIF_REFCLK_EN (1 << 3)
161# define MUX_TCLK_TO_XCLK (1 << 8)
162
163#define THM_CLK_CNTL 0x19b
164# define CMON_CLK_SEL(x) ((x) << 0)
165# define CMON_CLK_SEL_MASK 0xFF
166# define TMON_CLK_SEL(x) ((x) << 8)
167# define TMON_CLK_SEL_MASK 0xFF00
168#define MISC_CLK_CNTL 0x19c
169# define DEEP_SLEEP_CLK_SEL(x) ((x) << 0)
170# define DEEP_SLEEP_CLK_SEL_MASK 0xFF
171# define ZCLK_SEL(x) ((x) << 8)
172# define ZCLK_SEL_MASK 0xFF00
173
174#define CG_THERMAL_CTRL 0x1c0
175#define DPM_EVENT_SRC(x) ((x) << 0)
176#define DPM_EVENT_SRC_MASK (7 << 0)
177#define DIG_THERM_DPM(x) ((x) << 14)
178#define DIG_THERM_DPM_MASK 0x003FC000
179#define DIG_THERM_DPM_SHIFT 14
180#define CG_THERMAL_STATUS 0x1c1
181#define FDO_PWM_DUTY(x) ((x) << 9)
182#define FDO_PWM_DUTY_MASK (0xff << 9)
183#define FDO_PWM_DUTY_SHIFT 9
184#define CG_THERMAL_INT 0x1c2
185#define DIG_THERM_INTH(x) ((x) << 8)
186#define DIG_THERM_INTH_MASK 0x0000FF00
187#define DIG_THERM_INTH_SHIFT 8
188#define DIG_THERM_INTL(x) ((x) << 16)
189#define DIG_THERM_INTL_MASK 0x00FF0000
190#define DIG_THERM_INTL_SHIFT 16
191#define THERM_INT_MASK_HIGH (1 << 24)
192#define THERM_INT_MASK_LOW (1 << 25)
193
194#define CG_MULT_THERMAL_CTRL 0x1c4
195#define TEMP_SEL(x) ((x) << 20)
196#define TEMP_SEL_MASK (0xff << 20)
197#define TEMP_SEL_SHIFT 20
198#define CG_MULT_THERMAL_STATUS 0x1c5
199#define ASIC_MAX_TEMP(x) ((x) << 0)
200#define ASIC_MAX_TEMP_MASK 0x000001ff
201#define ASIC_MAX_TEMP_SHIFT 0
202#define CTF_TEMP(x) ((x) << 9)
203#define CTF_TEMP_MASK 0x0003fe00
204#define CTF_TEMP_SHIFT 9
205
206#define CG_FDO_CTRL0 0x1d5
207#define FDO_STATIC_DUTY(x) ((x) << 0)
208#define FDO_STATIC_DUTY_MASK 0x000000FF
209#define FDO_STATIC_DUTY_SHIFT 0
210#define CG_FDO_CTRL1 0x1d6
211#define FMAX_DUTY100(x) ((x) << 0)
212#define FMAX_DUTY100_MASK 0x000000FF
213#define FMAX_DUTY100_SHIFT 0
214#define CG_FDO_CTRL2 0x1d7
215#define TMIN(x) ((x) << 0)
216#define TMIN_MASK 0x000000FF
217#define TMIN_SHIFT 0
218#define FDO_PWM_MODE(x) ((x) << 11)
219#define FDO_PWM_MODE_MASK (7 << 11)
220#define FDO_PWM_MODE_SHIFT 11
221#define TACH_PWM_RESP_RATE(x) ((x) << 25)
222#define TACH_PWM_RESP_RATE_MASK (0x7f << 25)
223#define TACH_PWM_RESP_RATE_SHIFT 25
224
225#define CG_TACH_CTRL 0x1dc
226# define EDGE_PER_REV(x) ((x) << 0)
227# define EDGE_PER_REV_MASK (0x7 << 0)
228# define EDGE_PER_REV_SHIFT 0
229# define TARGET_PERIOD(x) ((x) << 3)
230# define TARGET_PERIOD_MASK 0xfffffff8
231# define TARGET_PERIOD_SHIFT 3
232#define CG_TACH_STATUS 0x1dd
233# define TACH_PERIOD(x) ((x) << 0)
234# define TACH_PERIOD_MASK 0xffffffff
235# define TACH_PERIOD_SHIFT 0
236
237#define GENERAL_PWRMGT 0x1e0
238# define GLOBAL_PWRMGT_EN (1 << 0)
239# define STATIC_PM_EN (1 << 1)
240# define THERMAL_PROTECTION_DIS (1 << 2)
241# define THERMAL_PROTECTION_TYPE (1 << 3)
242# define SW_SMIO_INDEX(x) ((x) << 6)
243# define SW_SMIO_INDEX_MASK (1 << 6)
244# define SW_SMIO_INDEX_SHIFT 6
245# define VOLT_PWRMGT_EN (1 << 10)
246# define DYN_SPREAD_SPECTRUM_EN (1 << 23)
247#define CG_TPC 0x1e1
248#define SCLK_PWRMGT_CNTL 0x1e2
249# define SCLK_PWRMGT_OFF (1 << 0)
250# define SCLK_LOW_D1 (1 << 1)
251# define FIR_RESET (1 << 4)
252# define FIR_FORCE_TREND_SEL (1 << 5)
253# define FIR_TREND_MODE (1 << 6)
254# define DYN_GFX_CLK_OFF_EN (1 << 7)
255# define GFX_CLK_FORCE_ON (1 << 8)
256# define GFX_CLK_REQUEST_OFF (1 << 9)
257# define GFX_CLK_FORCE_OFF (1 << 10)
258# define GFX_CLK_OFF_ACPI_D1 (1 << 11)
259# define GFX_CLK_OFF_ACPI_D2 (1 << 12)
260# define GFX_CLK_OFF_ACPI_D3 (1 << 13)
261# define DYN_LIGHT_SLEEP_EN (1 << 14)
262
263#define TARGET_AND_CURRENT_PROFILE_INDEX 0x1e6
264# define CURRENT_STATE_INDEX_MASK (0xf << 4)
265# define CURRENT_STATE_INDEX_SHIFT 4
266
267#define CG_FTV 0x1ef
268
269#define CG_FFCT_0 0x1f0
270# define UTC_0(x) ((x) << 0)
271# define UTC_0_MASK (0x3ff << 0)
272# define DTC_0(x) ((x) << 10)
273# define DTC_0_MASK (0x3ff << 10)
274
275#define CG_BSP 0x1ff
276# define BSP(x) ((x) << 0)
277# define BSP_MASK (0xffff << 0)
278# define BSU(x) ((x) << 16)
279# define BSU_MASK (0xf << 16)
280#define CG_AT 0x200
281# define CG_R(x) ((x) << 0)
282# define CG_R_MASK (0xffff << 0)
283# define CG_L(x) ((x) << 16)
284# define CG_L_MASK (0xffff << 16)
285
286#define CG_GIT 0x201
287# define CG_GICST(x) ((x) << 0)
288# define CG_GICST_MASK (0xffff << 0)
289# define CG_GIPOT(x) ((x) << 16)
290# define CG_GIPOT_MASK (0xffff << 16)
291
292#define CG_SSP 0x203
293# define SST(x) ((x) << 0)
294# define SST_MASK (0xffff << 0)
295# define SSTU(x) ((x) << 16)
296# define SSTU_MASK (0xf << 16)
297
298#define CG_DISPLAY_GAP_CNTL 0x20a
299# define DISP1_GAP(x) ((x) << 0)
300# define DISP1_GAP_MASK (3 << 0)
301# define DISP2_GAP(x) ((x) << 2)
302# define DISP2_GAP_MASK (3 << 2)
303# define VBI_TIMER_COUNT(x) ((x) << 4)
304# define VBI_TIMER_COUNT_MASK (0x3fff << 4)
305# define VBI_TIMER_UNIT(x) ((x) << 20)
306# define VBI_TIMER_UNIT_MASK (7 << 20)
307# define DISP1_GAP_MCHG(x) ((x) << 24)
308# define DISP1_GAP_MCHG_MASK (3 << 24)
309# define DISP2_GAP_MCHG(x) ((x) << 26)
310# define DISP2_GAP_MCHG_MASK (3 << 26)
311
312#define CG_ULV_CONTROL 0x21e
313#define CG_ULV_PARAMETER 0x21f
314
315#define SMC_SCRATCH0 0x221
316
317#define CG_CAC_CTRL 0x22e
318# define CAC_WINDOW(x) ((x) << 0)
319# define CAC_WINDOW_MASK 0x00ffffff
320
321#define DMIF_ADDR_CONFIG 0x2F5
322
323#define DMIF_ADDR_CALC 0x300
324
325#define PIPE0_DMIF_BUFFER_CONTROL 0x0328
326# define DMIF_BUFFERS_ALLOCATED(x) ((x) << 0)
327# define DMIF_BUFFERS_ALLOCATED_COMPLETED (1 << 4)
328
329#define SRBM_STATUS 0x394
330#define GRBM_RQ_PENDING (1 << 5)
331#define VMC_BUSY (1 << 8)
332#define MCB_BUSY (1 << 9)
333#define MCB_NON_DISPLAY_BUSY (1 << 10)
334#define MCC_BUSY (1 << 11)
335#define MCD_BUSY (1 << 12)
336#define SEM_BUSY (1 << 14)
337#define IH_BUSY (1 << 17)
338
339#define SRBM_SOFT_RESET 0x398
340#define SOFT_RESET_BIF (1 << 1)
341#define SOFT_RESET_DC (1 << 5)
342#define SOFT_RESET_DMA1 (1 << 6)
343#define SOFT_RESET_GRBM (1 << 8)
344#define SOFT_RESET_HDP (1 << 9)
345#define SOFT_RESET_IH (1 << 10)
346#define SOFT_RESET_MC (1 << 11)
347#define SOFT_RESET_ROM (1 << 14)
348#define SOFT_RESET_SEM (1 << 15)
349#define SOFT_RESET_VMC (1 << 17)
350#define SOFT_RESET_DMA (1 << 20)
351#define SOFT_RESET_TST (1 << 21)
352#define SOFT_RESET_REGBB (1 << 22)
353#define SOFT_RESET_ORB (1 << 23)
354
355#define CC_SYS_RB_BACKEND_DISABLE 0x3A0
356#define GC_USER_SYS_RB_BACKEND_DISABLE 0x3A1
357
358#define SRBM_READ_ERROR 0x3A6
359#define SRBM_INT_CNTL 0x3A8
360#define SRBM_INT_ACK 0x3AA
361
362#define SRBM_STATUS2 0x3B1
363#define DMA_BUSY (1 << 5)
364#define DMA1_BUSY (1 << 6)
365
366#define VM_L2_CNTL 0x500
367#define ENABLE_L2_CACHE (1 << 0)
368#define ENABLE_L2_FRAGMENT_PROCESSING (1 << 1)
369#define L2_CACHE_PTE_ENDIAN_SWAP_MODE(x) ((x) << 2)
370#define L2_CACHE_PDE_ENDIAN_SWAP_MODE(x) ((x) << 4)
371#define ENABLE_L2_PTE_CACHE_LRU_UPDATE_BY_WRITE (1 << 9)
372#define ENABLE_L2_PDE0_CACHE_LRU_UPDATE_BY_WRITE (1 << 10)
373#define EFFECTIVE_L2_QUEUE_SIZE(x) (((x) & 7) << 15)
374#define CONTEXT1_IDENTITY_ACCESS_MODE(x) (((x) & 3) << 19)
375#define VM_L2_CNTL2 0x501
376#define INVALIDATE_ALL_L1_TLBS (1 << 0)
377#define INVALIDATE_L2_CACHE (1 << 1)
378#define INVALIDATE_CACHE_MODE(x) ((x) << 26)
379#define INVALIDATE_PTE_AND_PDE_CACHES 0
380#define INVALIDATE_ONLY_PTE_CACHES 1
381#define INVALIDATE_ONLY_PDE_CACHES 2
382#define VM_L2_CNTL3 0x502
383#define BANK_SELECT(x) ((x) << 0)
384#define L2_CACHE_UPDATE_MODE(x) ((x) << 6)
385#define L2_CACHE_BIGK_FRAGMENT_SIZE(x) ((x) << 15)
386#define L2_CACHE_BIGK_ASSOCIATIVITY (1 << 20)
387#define VM_L2_STATUS 0x503
388#define L2_BUSY (1 << 0)
389#define VM_CONTEXT0_CNTL 0x504
390#define ENABLE_CONTEXT (1 << 0)
391#define PAGE_TABLE_DEPTH(x) (((x) & 3) << 1)
392#define RANGE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 3)
393#define RANGE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 4)
394#define DUMMY_PAGE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 6)
395#define DUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 7)
396#define PDE0_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 9)
397#define PDE0_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 10)
398#define VALID_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 12)
399#define VALID_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 13)
400#define READ_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 15)
401#define READ_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 16)
402#define WRITE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 18)
403#define WRITE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 19)
404#define PAGE_TABLE_BLOCK_SIZE(x) (((x) & 0xF) << 24)
405#define VM_CONTEXT1_CNTL 0x505
406#define VM_CONTEXT0_CNTL2 0x50C
407#define VM_CONTEXT1_CNTL2 0x50D
408#define VM_CONTEXT8_PAGE_TABLE_BASE_ADDR 0x50E
409#define VM_CONTEXT9_PAGE_TABLE_BASE_ADDR 0x50F
410#define VM_CONTEXT10_PAGE_TABLE_BASE_ADDR 0x510
411#define VM_CONTEXT11_PAGE_TABLE_BASE_ADDR 0x511
412#define VM_CONTEXT12_PAGE_TABLE_BASE_ADDR 0x512
413#define VM_CONTEXT13_PAGE_TABLE_BASE_ADDR 0x513
414#define VM_CONTEXT14_PAGE_TABLE_BASE_ADDR 0x514
415#define VM_CONTEXT15_PAGE_TABLE_BASE_ADDR 0x515
416
417#define VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x53f
418#define VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x537
419#define PROTECTIONS_MASK (0xf << 0)
420#define PROTECTIONS_SHIFT 0
421 /* bit 0: range
422 * bit 1: pde0
423 * bit 2: valid
424 * bit 3: read
425 * bit 4: write
426 */
427#define MEMORY_CLIENT_ID_MASK (0xff << 12)
428#define MEMORY_CLIENT_ID_SHIFT 12
429#define MEMORY_CLIENT_RW_MASK (1 << 24)
430#define MEMORY_CLIENT_RW_SHIFT 24
431#define FAULT_VMID_MASK (0xf << 25)
432#define FAULT_VMID_SHIFT 25
433
434#define VM_INVALIDATE_REQUEST 0x51E77#define VM_INVALIDATE_REQUEST 0x51E
435#define VM_INVALIDATE_RESPONSE 0x51F78#define VM_INVALIDATE_RESPONSE 0x51F
436 79
437#define VM_CONTEXT0_PROTECTION_FAULT_DEFAULT_ADDR 0x546
438#define VM_CONTEXT1_PROTECTION_FAULT_DEFAULT_ADDR 0x547
439
440#define VM_CONTEXT0_PAGE_TABLE_BASE_ADDR 0x54F
441#define VM_CONTEXT1_PAGE_TABLE_BASE_ADDR 0x550
442#define VM_CONTEXT2_PAGE_TABLE_BASE_ADDR 0x551
443#define VM_CONTEXT3_PAGE_TABLE_BASE_ADDR 0x552
444#define VM_CONTEXT4_PAGE_TABLE_BASE_ADDR 0x553
445#define VM_CONTEXT5_PAGE_TABLE_BASE_ADDR 0x554
446#define VM_CONTEXT6_PAGE_TABLE_BASE_ADDR 0x555
447#define VM_CONTEXT7_PAGE_TABLE_BASE_ADDR 0x556
448#define VM_CONTEXT0_PAGE_TABLE_START_ADDR 0x557
449#define VM_CONTEXT1_PAGE_TABLE_START_ADDR 0x558
450
451#define VM_CONTEXT0_PAGE_TABLE_END_ADDR 0x55F
452#define VM_CONTEXT1_PAGE_TABLE_END_ADDR 0x560
453
454#define VM_L2_CG 0x57080#define VM_L2_CG 0x570
455#define MC_CG_ENABLE (1 << 18)81#define MC_CG_ENABLE (1 << 18)
456#define MC_LS_ENABLE (1 << 19)82#define MC_LS_ENABLE (1 << 19)
457 83
458#define MC_SHARED_CHMAP 0x801
459#define NOOFCHAN_SHIFT 12
460#define NOOFCHAN_MASK 0x0000f000
461#define MC_SHARED_CHREMAP 0x802
462
463#define MC_VM_FB_LOCATION 0x80984#define MC_VM_FB_LOCATION 0x809
464#define MC_VM_AGP_TOP 0x80A85#define MC_VM_AGP_TOP 0x80A
465#define MC_VM_AGP_BOT 0x80B86#define MC_VM_AGP_BOT 0x80B
@@ -491,21 +112,6 @@ · 25 unmodified lines
491#define MC_CITF_MISC_WR_CG 0x993112#define MC_CITF_MISC_WR_CG 0x993
492#define MC_CITF_MISC_VM_CG 0x994113#define MC_CITF_MISC_VM_CG 0x994
493 114
494#define MC_ARB_RAMCFG 0x9D8
495#define NOOFBANK_SHIFT 0
496#define NOOFBANK_MASK 0x00000003
497#define NOOFRANK_SHIFT 2
498#define NOOFRANK_MASK 0x00000004
499#define NOOFROWS_SHIFT 3
500#define NOOFROWS_MASK 0x00000038
501#define NOOFCOLS_SHIFT 6
502#define NOOFCOLS_MASK 0x000000C0
503#define CHANSIZE_SHIFT 8
504#define CHANSIZE_MASK 0x00000100
505#define CHANSIZE_OVERRIDE (1 << 11)
506#define NOOFGROUPS_SHIFT 12
507#define NOOFGROUPS_MASK 0x00001000
508
509#define MC_ARB_DRAM_TIMING 0x9DD115#define MC_ARB_DRAM_TIMING 0x9DD
510#define MC_ARB_DRAM_TIMING2 0x9DE116#define MC_ARB_DRAM_TIMING2 0x9DE
511 117
@@ -631,20 +237,6 @@ · 119 unmodified lines
631#define CLKS(x) ((x) << 0)237#define CLKS(x) ((x) << 0)
632#define CLKS_MASK (0xfff << 0)238#define CLKS_MASK (0xfff << 0)
633 239
634#define HDP_HOST_PATH_CNTL 0xB00
635#define CLOCK_GATING_DIS (1 << 23)
636#define HDP_NONSURFACE_BASE 0xB01
637#define HDP_NONSURFACE_INFO 0xB02
638#define HDP_NONSURFACE_SIZE 0xB03
639
640#define HDP_DEBUG0 0xBCC
641
642#define HDP_ADDR_CONFIG 0xBD2
643#define HDP_MISC_CNTL 0xBD3
644#define HDP_FLUSH_INVALIDATE_CACHE (1 << 0)
645#define HDP_MEM_POWER_LS 0xBD4
646#define HDP_LS_ENABLE (1 << 0)
647
648#define ATC_MISC_CG 0xCD4240#define ATC_MISC_CG 0xCD4
649 241
650#define IH_RB_CNTL 0xF80242#define IH_RB_CNTL 0xF80
@@ -674,8 +266,6 @@ · 23 unmodified lines
674# define MC_WR_CLEAN_CNT(x) ((x) << 20)266# define MC_WR_CLEAN_CNT(x) ((x) << 20)
675# define MC_VMID(x) ((x) << 25)267# define MC_VMID(x) ((x) << 25)
676 268
677#define CONFIG_MEMSIZE 0x150A
678
679#define INTERRUPT_CNTL 0x151A269#define INTERRUPT_CNTL 0x151A
680# define IH_DUMMY_RD_OVERRIDE (1 << 0)270# define IH_DUMMY_RD_OVERRIDE (1 << 0)
681# define IH_DUMMY_RD_EN (1 << 1)271# define IH_DUMMY_RD_EN (1 << 1)
@@ -683,486 +273,22 @@ · 1 unmodified line
683# define GEN_IH_INT_EN (1 << 8)273# define GEN_IH_INT_EN (1 << 8)
684#define INTERRUPT_CNTL2 0x151B274#define INTERRUPT_CNTL2 0x151B
685 275
686#define HDP_MEM_COHERENCY_FLUSH_CNTL 0x1520
687
688#define BIF_FB_EN 0x1524
689#define FB_READ_EN (1 << 0)
690#define FB_WRITE_EN (1 << 1)
691
692#define HDP_REG_COHERENCY_FLUSH_CNTL 0x1528
693
694/* DCE6 ELD audio interface */
695#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0 0x28 /* LPCM */
696#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR1 0x29 /* AC3 */
697#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR2 0x2A /* MPEG1 */
698#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR3 0x2B /* MP3 */
699#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR4 0x2C /* MPEG2 */
700#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR5 0x2D /* AAC */
701#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR6 0x2E /* DTS */
702#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR7 0x2F /* ATRAC */
703#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR8 0x30 /* one bit audio - leave at 0 (default) */
704#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR9 0x31 /* Dolby Digital */
705#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR10 0x32 /* DTS-HD */
706#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR11 0x33 /* MAT-MLP */
707#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR12 0x34 /* DTS */
708#define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13 0x35 /* WMA Pro */
709# define MAX_CHANNELS(x) (((x) & 0x7) << 0)
710/* max channels minus one. 7 = 8 channels */
711# define SUPPORTED_FREQUENCIES(x) (((x) & 0xff) << 8)
712# define DESCRIPTOR_BYTE_2(x) (((x) & 0xff) << 16)
713# define SUPPORTED_FREQUENCIES_STEREO(x) (((x) & 0xff) << 24) /* LPCM only */
714/* SUPPORTED_FREQUENCIES, SUPPORTED_FREQUENCIES_STEREO
715 * bit0 = 32 kHz
716 * bit1 = 44.1 kHz
717 * bit2 = 48 kHz
718 * bit3 = 88.2 kHz
719 * bit4 = 96 kHz
720 * bit5 = 176.4 kHz
721 * bit6 = 192 kHz
722 */
723
724#define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_LIPSYNC 0x37
725# define VIDEO_LIPSYNC(x) (((x) & 0xff) << 0)
726# define AUDIO_LIPSYNC(x) (((x) & 0xff) << 8)
727/* VIDEO_LIPSYNC, AUDIO_LIPSYNC
728 * 0 = invalid
729 * x = legal delay value
730 * 255 = sync not supported
731 */
732#define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_HBR 0x38
733# define HBR_CAPABLE (1 << 0) /* enabled by default */
734
735#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO0 0x3a
736# define MANUFACTURER_ID(x) (((x) & 0xffff) << 0)
737# define PRODUCT_ID(x) (((x) & 0xffff) << 16)
738#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO1 0x3b
739# define SINK_DESCRIPTION_LEN(x) (((x) & 0xff) << 0)
740#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO2 0x3c
741# define PORT_ID0(x) (((x) & 0xffffffff) << 0)
742#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO3 0x3d
743# define PORT_ID1(x) (((x) & 0xffffffff) << 0)
744#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO4 0x3e
745# define DESCRIPTION0(x) (((x) & 0xff) << 0)
746# define DESCRIPTION1(x) (((x) & 0xff) << 8)
747# define DESCRIPTION2(x) (((x) & 0xff) << 16)
748# define DESCRIPTION3(x) (((x) & 0xff) << 24)
749#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO5 0x3f
750# define DESCRIPTION4(x) (((x) & 0xff) << 0)
751# define DESCRIPTION5(x) (((x) & 0xff) << 8)
752# define DESCRIPTION6(x) (((x) & 0xff) << 16)
753# define DESCRIPTION7(x) (((x) & 0xff) << 24)
754#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO6 0x40
755# define DESCRIPTION8(x) (((x) & 0xff) << 0)
756# define DESCRIPTION9(x) (((x) & 0xff) << 8)
757# define DESCRIPTION10(x) (((x) & 0xff) << 16)
758# define DESCRIPTION11(x) (((x) & 0xff) << 24)
759#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO7 0x41
760# define DESCRIPTION12(x) (((x) & 0xff) << 0)
761# define DESCRIPTION13(x) (((x) & 0xff) << 8)
762# define DESCRIPTION14(x) (((x) & 0xff) << 16)
763# define DESCRIPTION15(x) (((x) & 0xff) << 24)
764#define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO8 0x42
765# define DESCRIPTION16(x) (((x) & 0xff) << 0)
766# define DESCRIPTION17(x) (((x) & 0xff) << 8)
767
768#define AZ_F0_CODEC_PIN_CONTROL_HOT_PLUG_CONTROL 0x54
769# define AUDIO_ENABLED (1 << 31)
770
771#define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_CONFIGURATION_DEFAULT 0x56
772#define PORT_CONNECTIVITY_MASK (3 << 30)
773#define PORT_CONNECTIVITY_SHIFT 30
774
775#define DC_LB_MEMORY_SPLIT 0x1AC3
776#define DC_LB_MEMORY_CONFIG(x) ((x) << 20)
777
778#define PRIORITY_A_CNT 0x1AC6
779#define PRIORITY_MARK_MASK 0x7fff
780#define PRIORITY_OFF (1 << 16)
781#define PRIORITY_ALWAYS_ON (1 << 20)
782#define PRIORITY_B_CNT 0x1AC7
783
784#define DPG_PIPE_ARBITRATION_CONTROL3 0x1B32
785# define LATENCY_WATERMARK_MASK(x) ((x) << 16)
786#define DPG_PIPE_LATENCY_CONTROL 0x1B33
787# define LATENCY_LOW_WATERMARK(x) ((x) << 0)
788# define LATENCY_HIGH_WATERMARK(x) ((x) << 16)
789
790/* 0x6bb8, 0x77b8, 0x103b8, 0x10fb8, 0x11bb8, 0x127b8 */
791#define VLINE_STATUS 0x1AEE
792# define VLINE_OCCURRED (1 << 0)
793# define VLINE_ACK (1 << 4)
794# define VLINE_STAT (1 << 12)
795# define VLINE_INTERRUPT (1 << 16)
796# define VLINE_INTERRUPT_TYPE (1 << 17)
797/* 0x6bbc, 0x77bc, 0x103bc, 0x10fbc, 0x11bbc, 0x127bc */
798#define VBLANK_STATUS 0x1AEF
799# define VBLANK_OCCURRED (1 << 0)
800# define VBLANK_ACK (1 << 4)
801# define VBLANK_STAT (1 << 12)
802# define VBLANK_INTERRUPT (1 << 16)
803# define VBLANK_INTERRUPT_TYPE (1 << 17)
804
805/* 0x6b40, 0x7740, 0x10340, 0x10f40, 0x11b40, 0x12740 */
806#define INT_MASK 0x1AD0
807# define VBLANK_INT_MASK (1 << 0)
808# define VLINE_INT_MASK (1 << 4)
809
810#define DISP_INTERRUPT_STATUS 0x183D
811# define LB_D1_VLINE_INTERRUPT (1 << 2)
812# define LB_D1_VBLANK_INTERRUPT (1 << 3)
813# define DC_HPD1_INTERRUPT (1 << 17)
814# define DC_HPD1_RX_INTERRUPT (1 << 18)
815# define DACA_AUTODETECT_INTERRUPT (1 << 22)
816# define DACB_AUTODETECT_INTERRUPT (1 << 23)
817# define DC_I2C_SW_DONE_INTERRUPT (1 << 24)
818# define DC_I2C_HW_DONE_INTERRUPT (1 << 25)
819#define DISP_INTERRUPT_STATUS_CONTINUE 0x183E
820# define LB_D2_VLINE_INTERRUPT (1 << 2)
821# define LB_D2_VBLANK_INTERRUPT (1 << 3)
822# define DC_HPD2_INTERRUPT (1 << 17)
823# define DC_HPD2_RX_INTERRUPT (1 << 18)
824# define DISP_TIMER_INTERRUPT (1 << 24)
825#define DISP_INTERRUPT_STATUS_CONTINUE2 0x183F
826# define LB_D3_VLINE_INTERRUPT (1 << 2)
827# define LB_D3_VBLANK_INTERRUPT (1 << 3)
828# define DC_HPD3_INTERRUPT (1 << 17)
829# define DC_HPD3_RX_INTERRUPT (1 << 18)
830#define DISP_INTERRUPT_STATUS_CONTINUE3 0x1840
831# define LB_D4_VLINE_INTERRUPT (1 << 2)
832# define LB_D4_VBLANK_INTERRUPT (1 << 3)
833# define DC_HPD4_INTERRUPT (1 << 17)
834# define DC_HPD4_RX_INTERRUPT (1 << 18)
835#define DISP_INTERRUPT_STATUS_CONTINUE4 0x1853
836# define LB_D5_VLINE_INTERRUPT (1 << 2)
837# define LB_D5_VBLANK_INTERRUPT (1 << 3)
838# define DC_HPD5_INTERRUPT (1 << 17)
839# define DC_HPD5_RX_INTERRUPT (1 << 18)
840#define DISP_INTERRUPT_STATUS_CONTINUE5 0x1854
841# define LB_D6_VLINE_INTERRUPT (1 << 2)
842# define LB_D6_VBLANK_INTERRUPT (1 << 3)
843# define DC_HPD6_INTERRUPT (1 << 17)
844# define DC_HPD6_RX_INTERRUPT (1 << 18)
845
846/* 0x6858, 0x7458, 0x10058, 0x10c58, 0x11858, 0x12458 */
847#define GRPH_INT_STATUS 0x1A16
848# define GRPH_PFLIP_INT_OCCURRED (1 << 0)
849# define GRPH_PFLIP_INT_CLEAR (1 << 8)
850/* 0x685c, 0x745c, 0x1005c, 0x10c5c, 0x1185c, 0x1245c */
851#define GRPH_INT_CONTROL 0x1A17
852# define GRPH_PFLIP_INT_MASK (1 << 0)
853# define GRPH_PFLIP_INT_TYPE (1 << 8)
854
855#define DAC_AUTODETECT_INT_CONTROL 0x19F2
856
857#define DC_HPD1_INT_STATUS 0x1807
858#define DC_HPD2_INT_STATUS 0x180A
859#define DC_HPD3_INT_STATUS 0x180D
860#define DC_HPD4_INT_STATUS 0x1810
861#define DC_HPD5_INT_STATUS 0x1813
862#define DC_HPD6_INT_STATUS 0x1816
863# define DC_HPDx_INT_STATUS (1 << 0)
864# define DC_HPDx_SENSE (1 << 1)
865# define DC_HPDx_RX_INT_STATUS (1 << 8)
866
867#define DC_HPD1_INT_CONTROL 0x1808
868#define DC_HPD2_INT_CONTROL 0x180B
869#define DC_HPD3_INT_CONTROL 0x180E
870#define DC_HPD4_INT_CONTROL 0x1811
871#define DC_HPD5_INT_CONTROL 0x1814
872#define DC_HPD6_INT_CONTROL 0x1817
873# define DC_HPDx_INT_ACK (1 << 0)
874# define DC_HPDx_INT_POLARITY (1 << 8)
875# define DC_HPDx_INT_EN (1 << 16)
876# define DC_HPDx_RX_INT_ACK (1 << 20)
877# define DC_HPDx_RX_INT_EN (1 << 24)
878
879#define DC_HPD1_CONTROL 0x1809
880#define DC_HPD2_CONTROL 0x180C
881#define DC_HPD3_CONTROL 0x180F
882#define DC_HPD4_CONTROL 0x1812
883#define DC_HPD5_CONTROL 0x1815
884#define DC_HPD6_CONTROL 0x1818
885# define DC_HPDx_CONNECTION_TIMER(x) ((x) << 0)
886# define DC_HPDx_RX_INT_TIMER(x) ((x) << 16)
887# define DC_HPDx_EN (1 << 28)
888
889#define DPG_PIPE_STUTTER_CONTROL 0x1B35
890# define STUTTER_ENABLE (1 << 0)
891
892/* 0x6e98, 0x7a98, 0x10698, 0x11298, 0x11e98, 0x12a98 */
893#define CRTC_STATUS_FRAME_COUNT 0x1BA6
894
895/* Audio clocks */
896#define DCCG_AUDIO_DTO0_PHASE 0x05b0
897#define DCCG_AUDIO_DTO0_MODULE 0x05b4
898#define DCCG_AUDIO_DTO1_PHASE 0x05c0
899#define DCCG_AUDIO_DTO1_MODULE 0x05c4
900
901#define GRBM_CNTL 0x2000
902#define GRBM_READ_TIMEOUT(x) ((x) << 0)
903
904#define GRBM_STATUS2 0x2002
905#define RLC_RQ_PENDING (1 << 0)
906#define RLC_BUSY (1 << 8)
907#define TC_BUSY (1 << 9)
908
909#define GRBM_STATUS 0x2004
910#define CMDFIFO_AVAIL_MASK 0x0000000F
911#define RING2_RQ_PENDING (1 << 4)
912#define SRBM_RQ_PENDING (1 << 5)
913#define RING1_RQ_PENDING (1 << 6)
914#define CF_RQ_PENDING (1 << 7)
915#define PF_RQ_PENDING (1 << 8)
916#define GDS_DMA_RQ_PENDING (1 << 9)
917#define GRBM_EE_BUSY (1 << 10)
918#define DB_CLEAN (1 << 12)
919#define CB_CLEAN (1 << 13)
920#define TA_BUSY (1 << 14)
921#define GDS_BUSY (1 << 15)
922#define VGT_BUSY (1 << 17)
923#define IA_BUSY_NO_DMA (1 << 18)
924#define IA_BUSY (1 << 19)
925#define SX_BUSY (1 << 20)
926#define SPI_BUSY (1 << 22)
927#define BCI_BUSY (1 << 23)
928#define SC_BUSY (1 << 24)
929#define PA_BUSY (1 << 25)
930#define DB_BUSY (1 << 26)
931#define CP_COHERENCY_BUSY (1 << 28)
932#define CP_BUSY (1 << 29)
933#define CB_BUSY (1 << 30)
934#define GUI_ACTIVE (1 << 31)
935#define GRBM_STATUS_SE0 0x2005
936#define GRBM_STATUS_SE1 0x2006
937#define SE_DB_CLEAN (1 << 1)
938#define SE_CB_CLEAN (1 << 2)
939#define SE_BCI_BUSY (1 << 22)
940#define SE_VGT_BUSY (1 << 23)
941#define SE_PA_BUSY (1 << 24)
942#define SE_TA_BUSY (1 << 25)
943#define SE_SX_BUSY (1 << 26)
944#define SE_SPI_BUSY (1 << 27)
945#define SE_SC_BUSY (1 << 29)
946#define SE_DB_BUSY (1 << 30)
947#define SE_CB_BUSY (1 << 31)
948
949#define GRBM_INT_CNTL 0x2018
950# define RDERR_INT_ENABLE (1 << 0)
951# define GUI_IDLE_INT_ENABLE (1 << 19)
952
953#define CP_STRMOUT_CNTL 0x213F
954#define SCRATCH_REG0 0x2140
955#define SCRATCH_REG1 0x2141
956#define SCRATCH_REG2 0x2142
957#define SCRATCH_REG3 0x2143
958#define SCRATCH_REG4 0x2144
959#define SCRATCH_REG5 0x2145
960#define SCRATCH_REG6 0x2146
961#define SCRATCH_REG7 0x2147
962
963#define SCRATCH_UMSK 0x2150
964#define SCRATCH_ADDR 0x2151
965
966#define CP_SEM_WAIT_TIMER 0x216F
967
968#define CP_SEM_INCOMPLETE_TIMER_CNTL 0x2172
969
970#define CP_ME_CNTL 0x21B6
971#define CP_CE_HALT (1 << 24)
972#define CP_PFP_HALT (1 << 26)
973#define CP_ME_HALT (1 << 28)
974
975#define CP_COHER_CNTL2 0x217A
976
977#define CP_RB2_RPTR 0x21BE
978#define CP_RB1_RPTR 0x21BF
979#define CP_RB0_RPTR 0x21C0
980#define CP_RB_WPTR_DELAY 0x21C1
981
982#define CP_QUEUE_THRESHOLDS 0x21D8
983#define ROQ_IB1_START(x) ((x) << 0)
984#define ROQ_IB2_START(x) ((x) << 8)
985#define CP_MEQ_THRESHOLDS 0x21D9
986#define MEQ1_START(x) ((x) << 0)
987#define MEQ2_START(x) ((x) << 8)
988
989#define CP_PERFMON_CNTL 0x21FF
990
991#define VGT_VTX_VECT_EJECT_REG 0x222C276#define VGT_VTX_VECT_EJECT_REG 0x222C
992
993#define VGT_ESGS_RING_SIZE 0x2232277#define VGT_ESGS_RING_SIZE 0x2232
994#define VGT_GSVS_RING_SIZE 0x2233278#define VGT_GSVS_RING_SIZE 0x2233
995
996#define VGT_GS_VERTEX_REUSE 0x2235279#define VGT_GS_VERTEX_REUSE 0x2235
997
998#define VGT_PRIMITIVE_TYPE 0x2256280#define VGT_PRIMITIVE_TYPE 0x2256
999#define VGT_INDEX_TYPE 0x2257281#define VGT_INDEX_TYPE 0x2257
1000
1001#define VGT_NUM_INDICES 0x225C282#define VGT_NUM_INDICES 0x225C
1002#define VGT_NUM_INSTANCES 0x225D283#define VGT_NUM_INSTANCES 0x225D
1003
1004#define VGT_TF_RING_SIZE 0x2262284#define VGT_TF_RING_SIZE 0x2262
1005
1006#define VGT_HS_OFFCHIP_PARAM 0x226C285#define VGT_HS_OFFCHIP_PARAM 0x226C
1007
1008#define VGT_TF_MEMORY_BASE 0x226E286#define VGT_TF_MEMORY_BASE 0x226E
1009 287
1010#define PA_CL_ENHANCE 0x2285
1011#define CLIP_VTX_REORDER_ENA (1 << 0)
1012#define NUM_CLIP_SEQ(x) ((x) << 1)
1013
1014#define PA_SU_LINE_STIPPLE_VALUE 0x2298
1015
1016#define PA_SC_LINE_STIPPLE_STATE 0x22C4
1017
1018#define PA_SC_FORCE_EOV_MAX_CNTS 0x22C9
1019#define FORCE_EOV_MAX_CLK_CNT(x) ((x) << 0)
1020#define FORCE_EOV_MAX_REZ_CNT(x) ((x) << 16)
1021
1022#define PA_SC_FIFO_SIZE 0x22F3
1023#define SC_FRONTEND_PRIM_FIFO_SIZE(x) ((x) << 0)
1024#define SC_BACKEND_PRIM_FIFO_SIZE(x) ((x) << 6)
1025#define SC_HIZ_TILE_FIFO_SIZE(x) ((x) << 15)
1026#define SC_EARLYZ_TILE_FIFO_SIZE(x) ((x) << 23)
1027
1028#define PA_SC_ENHANCE 0x22FC288#define PA_SC_ENHANCE 0x22FC
1029 289
1030#define SQ_CONFIG 0x2300
1031
1032#define SQC_CACHES 0x2302
1033
1034#define SQ_POWER_THROTTLE 0x2396
1035#define MIN_POWER(x) ((x) << 0)
1036#define MIN_POWER_MASK (0x3fff << 0)
1037#define MIN_POWER_SHIFT 0
1038#define MAX_POWER(x) ((x) << 16)
1039#define MAX_POWER_MASK (0x3fff << 16)
1040#define MAX_POWER_SHIFT 0
1041#define SQ_POWER_THROTTLE2 0x2397
1042#define MAX_POWER_DELTA(x) ((x) << 0)
1043#define MAX_POWER_DELTA_MASK (0x3fff << 0)
1044#define MAX_POWER_DELTA_SHIFT 0
1045#define STI_SIZE(x) ((x) << 16)
1046#define STI_SIZE_MASK (0x3ff << 16)
1047#define STI_SIZE_SHIFT 16
1048#define LTI_RATIO(x) ((x) << 27)
1049#define LTI_RATIO_MASK (0xf << 27)
1050#define LTI_RATIO_SHIFT 27
1051
1052#define SX_DEBUG_1 0x2418
1053
1054#define SPI_STATIC_THREAD_MGMT_1 0x2438
1055#define SPI_STATIC_THREAD_MGMT_2 0x2439
1056#define SPI_STATIC_THREAD_MGMT_3 0x243A
1057#define SPI_PS_MAX_WAVE_ID 0x243B
1058
1059#define SPI_CONFIG_CNTL 0x2440
1060
1061#define SPI_CONFIG_CNTL_1 0x244F
1062#define VTX_DONE_DELAY(x) ((x) << 0)
1063#define INTERP_ONE_PRIM_PER_ROW (1 << 4)
1064
1065#define CGTS_TCC_DISABLE 0x2452
1066#define CGTS_USER_TCC_DISABLE 0x2453
1067#define TCC_DISABLE_MASK 0xFFFF0000
1068#define TCC_DISABLE_SHIFT 16
1069#define CGTS_SM_CTRL_REG 0x2454
1070#define OVERRIDE (1 << 21)
1071#define LS_OVERRIDE (1 << 22)
1072
1073#define SPI_LB_CU_MASK 0x24D5
1074
1075#define TA_CNTL_AUX 0x2542290#define TA_CNTL_AUX 0x2542
1076 291
1077#define CC_RB_BACKEND_DISABLE 0x263D
1078#define BACKEND_DISABLE(x) ((x) << 16)
1079#define GB_ADDR_CONFIG 0x263E
1080#define NUM_PIPES(x) ((x) << 0)
1081#define NUM_PIPES_MASK 0x00000007
1082#define NUM_PIPES_SHIFT 0
1083#define PIPE_INTERLEAVE_SIZE(x) ((x) << 4)
1084#define PIPE_INTERLEAVE_SIZE_MASK 0x00000070
1085#define PIPE_INTERLEAVE_SIZE_SHIFT 4
1086#define NUM_SHADER_ENGINES(x) ((x) << 12)
1087#define NUM_SHADER_ENGINES_MASK 0x00003000
1088#define NUM_SHADER_ENGINES_SHIFT 12
1089#define SHADER_ENGINE_TILE_SIZE(x) ((x) << 16)
1090#define SHADER_ENGINE_TILE_SIZE_MASK 0x00070000
1091#define SHADER_ENGINE_TILE_SIZE_SHIFT 16
1092#define NUM_GPUS(x) ((x) << 20)
1093#define NUM_GPUS_MASK 0x00700000
1094#define NUM_GPUS_SHIFT 20
1095#define MULTI_GPU_TILE_SIZE(x) ((x) << 24)
1096#define MULTI_GPU_TILE_SIZE_MASK 0x03000000
1097#define MULTI_GPU_TILE_SIZE_SHIFT 24
1098#define ROW_SIZE(x) ((x) << 28)
1099#define ROW_SIZE_MASK 0x30000000
1100#define ROW_SIZE_SHIFT 28
1101
1102#define CB_PERFCOUNTER0_SELECT0 0x2688
1103#define CB_PERFCOUNTER0_SELECT1 0x2689
1104#define CB_PERFCOUNTER1_SELECT0 0x268A
1105#define CB_PERFCOUNTER1_SELECT1 0x268B
1106#define CB_PERFCOUNTER2_SELECT0 0x268C
1107#define CB_PERFCOUNTER2_SELECT1 0x268D
1108#define CB_PERFCOUNTER3_SELECT0 0x268E
1109#define CB_PERFCOUNTER3_SELECT1 0x268F
1110
1111#define CB_CGTT_SCLK_CTRL 0x2698
1112
1113#define TCP_CHAN_STEER_LO 0x2B03
1114#define TCP_CHAN_STEER_HI 0x2B94
1115
1116#define CP_RB0_BASE 0x3040
1117#define CP_RB0_CNTL 0x3041
1118#define RB_BUFSZ(x) ((x) << 0)
1119#define RB_BLKSZ(x) ((x) << 8)
1120#define BUF_SWAP_32BIT (2 << 16)
1121#define RB_NO_UPDATE (1 << 27)
1122#define RB_RPTR_WR_ENA (1 << 31)
1123
1124#define CP_RB0_RPTR_ADDR 0x3043
1125#define CP_RB0_RPTR_ADDR_HI 0x3044
1126#define CP_RB0_WPTR 0x3045
1127
1128#define CP_PFP_UCODE_ADDR 0x3054
1129#define CP_PFP_UCODE_DATA 0x3055
1130#define CP_ME_RAM_RADDR 0x3056
1131#define CP_ME_RAM_WADDR 0x3057
1132#define CP_ME_RAM_DATA 0x3058
1133
1134#define CP_CE_UCODE_ADDR 0x305A
1135#define CP_CE_UCODE_DATA 0x305B
1136
1137#define CP_RB1_BASE 0x3060
1138#define CP_RB1_CNTL 0x3061
1139#define CP_RB1_RPTR_ADDR 0x3062
1140#define CP_RB1_RPTR_ADDR_HI 0x3063
1141#define CP_RB1_WPTR 0x3064
1142#define CP_RB2_BASE 0x3065
1143#define CP_RB2_CNTL 0x3066
1144#define CP_RB2_RPTR_ADDR 0x3067
1145#define CP_RB2_RPTR_ADDR_HI 0x3068
1146#define CP_RB2_WPTR 0x3069
1147#define CP_INT_CNTL_RING0 0x306A
1148#define CP_INT_CNTL_RING1 0x306B
1149#define CP_INT_CNTL_RING2 0x306C
1150# define CNTX_BUSY_INT_ENABLE (1 << 19)
1151# define CNTX_EMPTY_INT_ENABLE (1 << 20)
1152# define WAIT_MEM_SEM_INT_ENABLE (1 << 21)
1153# define TIME_STAMP_INT_ENABLE (1 << 26)
1154# define CP_RINGID2_INT_ENABLE (1 << 29)
1155# define CP_RINGID1_INT_ENABLE (1 << 30)
1156# define CP_RINGID0_INT_ENABLE (1 << 31)
1157#define CP_INT_STATUS_RING0 0x306D
1158#define CP_INT_STATUS_RING1 0x306E
1159#define CP_INT_STATUS_RING2 0x306F
1160# define WAIT_MEM_SEM_INT_STAT (1 << 21)
1161# define TIME_STAMP_INT_STAT (1 << 26)
1162# define CP_RINGID2_INT_STAT (1 << 29)
1163# define CP_RINGID1_INT_STAT (1 << 30)
1164# define CP_RINGID0_INT_STAT (1 << 31)
1165
1166// #define PA_SC_RASTER_CONFIG 0xA0D4292// #define PA_SC_RASTER_CONFIG 0xA0D4
1167# define RB_XSEL2(x) ((x) << 4)293# define RB_XSEL2(x) ((x) << 4)
1168# define RB_XSEL2_MASK (0x3 << 4)294# define RB_XSEL2_MASK (0x3 << 4)
@@ -1185,171 +311,14 @@ · 16 unmodified lines
1185# define SE_YSEL(x) ((x) << 28)311# define SE_YSEL(x) ((x) << 28)
1186# define SE_YSEL_MASK (0x3 << 28)312# define SE_YSEL_MASK (0x3 << 28)
1187 313
1188/* PIF PHY0 registers idx/data 0x8/0xc */
1189#define PB0_PIF_CNTL 0x10
1190# define LS2_EXIT_TIME(x) ((x) << 17)
1191# define LS2_EXIT_TIME_MASK (0x7 << 17)
1192# define LS2_EXIT_TIME_SHIFT 17
1193#define PB0_PIF_PAIRING 0x11
1194# define MULTI_PIF (1 << 25)
1195#define PB0_PIF_PWRDOWN_0 0x12
1196# define PLL_POWER_STATE_IN_TXS2_0(x) ((x) << 7)
1197# define PLL_POWER_STATE_IN_TXS2_0_MASK (0x7 << 7)
1198# define PLL_POWER_STATE_IN_TXS2_0_SHIFT 7
1199# define PLL_POWER_STATE_IN_OFF_0(x) ((x) << 10)
1200# define PLL_POWER_STATE_IN_OFF_0_MASK (0x7 << 10)
1201# define PLL_POWER_STATE_IN_OFF_0_SHIFT 10
1202# define PLL_RAMP_UP_TIME_0(x) ((x) << 24)
1203# define PLL_RAMP_UP_TIME_0_MASK (0x7 << 24)
1204# define PLL_RAMP_UP_TIME_0_SHIFT 24
1205#define PB0_PIF_PWRDOWN_1 0x13
1206# define PLL_POWER_STATE_IN_TXS2_1(x) ((x) << 7)
1207# define PLL_POWER_STATE_IN_TXS2_1_MASK (0x7 << 7)
1208# define PLL_POWER_STATE_IN_TXS2_1_SHIFT 7
1209# define PLL_POWER_STATE_IN_OFF_1(x) ((x) << 10)
1210# define PLL_POWER_STATE_IN_OFF_1_MASK (0x7 << 10)
1211# define PLL_POWER_STATE_IN_OFF_1_SHIFT 10
1212# define PLL_RAMP_UP_TIME_1(x) ((x) << 24)
1213# define PLL_RAMP_UP_TIME_1_MASK (0x7 << 24)
1214# define PLL_RAMP_UP_TIME_1_SHIFT 24
1215
1216#define PB0_PIF_PWRDOWN_2 0x17
1217# define PLL_POWER_STATE_IN_TXS2_2(x) ((x) << 7)
1218# define PLL_POWER_STATE_IN_TXS2_2_MASK (0x7 << 7)
1219# define PLL_POWER_STATE_IN_TXS2_2_SHIFT 7
1220# define PLL_POWER_STATE_IN_OFF_2(x) ((x) << 10)
1221# define PLL_POWER_STATE_IN_OFF_2_MASK (0x7 << 10)
1222# define PLL_POWER_STATE_IN_OFF_2_SHIFT 10
1223# define PLL_RAMP_UP_TIME_2(x) ((x) << 24)
1224# define PLL_RAMP_UP_TIME_2_MASK (0x7 << 24)
1225# define PLL_RAMP_UP_TIME_2_SHIFT 24
1226#define PB0_PIF_PWRDOWN_3 0x18
1227# define PLL_POWER_STATE_IN_TXS2_3(x) ((x) << 7)
1228# define PLL_POWER_STATE_IN_TXS2_3_MASK (0x7 << 7)
1229# define PLL_POWER_STATE_IN_TXS2_3_SHIFT 7
1230# define PLL_POWER_STATE_IN_OFF_3(x) ((x) << 10)
1231# define PLL_POWER_STATE_IN_OFF_3_MASK (0x7 << 10)
1232# define PLL_POWER_STATE_IN_OFF_3_SHIFT 10
1233# define PLL_RAMP_UP_TIME_3(x) ((x) << 24)
1234# define PLL_RAMP_UP_TIME_3_MASK (0x7 << 24)
1235# define PLL_RAMP_UP_TIME_3_SHIFT 24
1236/* PIF PHY1 registers idx/data 0x10/0x14 */
1237#define PB1_PIF_CNTL 0x10
1238#define PB1_PIF_PAIRING 0x11
1239#define PB1_PIF_PWRDOWN_0 0x12
1240#define PB1_PIF_PWRDOWN_1 0x13
1241
1242#define PB1_PIF_PWRDOWN_2 0x17
1243#define PB1_PIF_PWRDOWN_3 0x18
1244/* PCIE registers idx/data 0x30/0x34 */
1245#define PCIE_CNTL2 0x1c /* PCIE */
1246# define SLV_MEM_LS_EN (1 << 16)
1247# define SLV_MEM_AGGRESSIVE_LS_EN (1 << 17)
1248# define MST_MEM_LS_EN (1 << 18)
1249# define REPLAY_MEM_LS_EN (1 << 19)
1250#define PCIE_LC_STATUS1 0x28 /* PCIE */
1251# define LC_REVERSE_RCVR (1 << 0)
1252# define LC_REVERSE_XMIT (1 << 1)
1253# define LC_OPERATING_LINK_WIDTH_MASK (0x7 << 2)
1254# define LC_OPERATING_LINK_WIDTH_SHIFT 2
1255# define LC_DETECTED_LINK_WIDTH_MASK (0x7 << 5)
1256# define LC_DETECTED_LINK_WIDTH_SHIFT 5
1257
1258#define PCIE_P_CNTL 0x40 /* PCIE */
1259# define P_IGNORE_EDB_ERR (1 << 6)
1260
1261/* PCIE PORT registers idx/data 0x38/0x3c */314/* PCIE PORT registers idx/data 0x38/0x3c */
1262#define PCIE_LC_CNTL 0xa0315// #define PCIE_LC_LINK_WIDTH_CNTL 0xa2 /* PCIE_P */
1263# define LC_L0S_INACTIVITY(x) ((x) << 8)
1264# define LC_L0S_INACTIVITY_MASK (0xf << 8)
1265# define LC_L0S_INACTIVITY_SHIFT 8
1266# define LC_L1_INACTIVITY(x) ((x) << 12)
1267# define LC_L1_INACTIVITY_MASK (0xf << 12)
1268# define LC_L1_INACTIVITY_SHIFT 12
1269# define LC_PMI_TO_L1_DIS (1 << 16)
1270# define LC_ASPM_TO_L1_DIS (1 << 24)
1271#define PCIE_LC_LINK_WIDTH_CNTL 0xa2 /* PCIE_P */
1272# define LC_LINK_WIDTH_SHIFT 0
1273# define LC_LINK_WIDTH_MASK 0x7
1274# define LC_LINK_WIDTH_X0 0316# define LC_LINK_WIDTH_X0 0
1275# define LC_LINK_WIDTH_X1 1317# define LC_LINK_WIDTH_X1 1
1276# define LC_LINK_WIDTH_X2 2318# define LC_LINK_WIDTH_X2 2
1277# define LC_LINK_WIDTH_X4 3319# define LC_LINK_WIDTH_X4 3
1278# define LC_LINK_WIDTH_X8 4320# define LC_LINK_WIDTH_X8 4
1279# define LC_LINK_WIDTH_X16 6321# define LC_LINK_WIDTH_X16 6
1280# define LC_LINK_WIDTH_RD_SHIFT 4
1281# define LC_LINK_WIDTH_RD_MASK 0x70
1282# define LC_RECONFIG_ARC_MISSING_ESCAPE (1 << 7)
1283# define LC_RECONFIG_NOW (1 << 8)
1284# define LC_RENEGOTIATION_SUPPORT (1 << 9)
1285# define LC_RENEGOTIATE_EN (1 << 10)
1286# define LC_SHORT_RECONFIG_EN (1 << 11)
1287# define LC_UPCONFIGURE_SUPPORT (1 << 12)
1288# define LC_UPCONFIGURE_DIS (1 << 13)
1289# define LC_DYN_LANES_PWR_STATE(x) ((x) << 21)
1290# define LC_DYN_LANES_PWR_STATE_MASK (0x3 << 21)
1291# define LC_DYN_LANES_PWR_STATE_SHIFT 21
1292#define PCIE_LC_N_FTS_CNTL 0xa3 /* PCIE_P */
1293# define LC_XMIT_N_FTS(x) ((x) << 0)
1294# define LC_XMIT_N_FTS_MASK (0xff << 0)
1295# define LC_XMIT_N_FTS_SHIFT 0
1296# define LC_XMIT_N_FTS_OVERRIDE_EN (1 << 8)
1297# define LC_N_FTS_MASK (0xff << 24)
1298#define PCIE_LC_SPEED_CNTL 0xa4 /* PCIE_P */
1299# define LC_GEN2_EN_STRAP (1 << 0)
1300# define LC_GEN3_EN_STRAP (1 << 1)
1301# define LC_TARGET_LINK_SPEED_OVERRIDE_EN (1 << 2)
1302# define LC_TARGET_LINK_SPEED_OVERRIDE_MASK (0x3 << 3)
1303# define LC_TARGET_LINK_SPEED_OVERRIDE_SHIFT 3
1304# define LC_FORCE_EN_SW_SPEED_CHANGE (1 << 5)
1305# define LC_FORCE_DIS_SW_SPEED_CHANGE (1 << 6)
1306# define LC_FORCE_EN_HW_SPEED_CHANGE (1 << 7)
1307# define LC_FORCE_DIS_HW_SPEED_CHANGE (1 << 8)
1308# define LC_INITIATE_LINK_SPEED_CHANGE (1 << 9)
1309# define LC_SPEED_CHANGE_ATTEMPTS_ALLOWED_MASK (0x3 << 10)
1310# define LC_SPEED_CHANGE_ATTEMPTS_ALLOWED_SHIFT 10
1311# define LC_CURRENT_DATA_RATE_MASK (0x3 << 13) /* 0/1/2 = gen1/2/3 */
1312# define LC_CURRENT_DATA_RATE_SHIFT 13
1313# define LC_CLR_FAILED_SPD_CHANGE_CNT (1 << 16)
1314# define LC_OTHER_SIDE_EVER_SENT_GEN2 (1 << 18)
1315# define LC_OTHER_SIDE_SUPPORTS_GEN2 (1 << 19)
1316# define LC_OTHER_SIDE_EVER_SENT_GEN3 (1 << 20)
1317# define LC_OTHER_SIDE_SUPPORTS_GEN3 (1 << 21)
1318
1319#define PCIE_LC_CNTL2 0xb1
1320# define LC_ALLOW_PDWN_IN_L1 (1 << 17)
1321# define LC_ALLOW_PDWN_IN_L23 (1 << 18)
1322
1323#define PCIE_LC_CNTL3 0xb5 /* PCIE_P */
1324# define LC_GO_TO_RECOVERY (1 << 30)
1325#define PCIE_LC_CNTL4 0xb6 /* PCIE_P */
1326# define LC_REDO_EQ (1 << 5)
1327# define LC_SET_QUIESCE (1 << 13)
1328
1329/*
1330 * UVD
1331 */
1332#define UVD_UDEC_ADDR_CONFIG 0x3bd3
1333#define UVD_UDEC_DB_ADDR_CONFIG 0x3bd4
1334#define UVD_UDEC_DBW_ADDR_CONFIG 0x3bd5
1335#define UVD_RBC_RB_RPTR 0x3da4
1336#define UVD_RBC_RB_WPTR 0x3da5
1337#define UVD_STATUS 0x3daf
1338
1339#define UVD_CGC_CTRL 0x3dc2
1340# define DCM (1 << 0)
1341# define CG_DT(x) ((x) << 2)
1342# define CG_DT_MASK (0xf << 2)
1343# define CLK_OD(x) ((x) << 6)
1344# define CLK_OD_MASK (0x1f << 6)
1345
1346 /* UVD CTX indirect */
1347#define UVD_CGC_MEM_CTRL 0xC0
1348#define UVD_CGC_CTRL2 0xC1
1349# define DYN_OR_EN (1 << 0)
1350# define DYN_RR_EN (1 << 1)
1351# define G_DIV_ID(x) ((x) << 2)
1352# define G_DIV_ID_MASK (0x7 << 2)
1353 322
1354/*323/*
1355 * PM4324 * PM4
@@ -1583,45 +552,7 @@ · 227 unmodified lines
1583/* ASYNC DMA - first instance at 0xd000, second at 0xd800 */552/* ASYNC DMA - first instance at 0xd000, second at 0xd800 */
1584#define DMA0_REGISTER_OFFSET 0x0 /* not a register */553#define DMA0_REGISTER_OFFSET 0x0 /* not a register */
1585#define DMA1_REGISTER_OFFSET 0x200 /* not a register */554#define DMA1_REGISTER_OFFSET 0x200 /* not a register */
1586 555#define SDMA_MAX_INSTANCE 2
1587#define DMA_RB_CNTL 0x3400
1588# define DMA_RB_ENABLE (1 << 0)
1589# define DMA_RB_SIZE(x) ((x) << 1) /* log2 */
1590# define DMA_RB_SWAP_ENABLE (1 << 9) /* 8IN32 */
1591# define DMA_RPTR_WRITEBACK_ENABLE (1 << 12)
1592# define DMA_RPTR_WRITEBACK_SWAP_ENABLE (1 << 13) /* 8IN32 */
1593# define DMA_RPTR_WRITEBACK_TIMER(x) ((x) << 16) /* log2 */
1594#define DMA_RB_BASE 0x3401
1595#define DMA_RB_RPTR 0x3402
1596#define DMA_RB_WPTR 0x3403
1597
1598#define DMA_RB_RPTR_ADDR_HI 0x3407
1599#define DMA_RB_RPTR_ADDR_LO 0x3408
1600
1601#define DMA_IB_CNTL 0x3409
1602# define DMA_IB_ENABLE (1 << 0)
1603# define DMA_IB_SWAP_ENABLE (1 << 4)
1604# define CMD_VMID_FORCE (1 << 31)
1605#define DMA_IB_RPTR 0x340a
1606#define DMA_CNTL 0x340b
1607# define TRAP_ENABLE (1 << 0)
1608# define SEM_INCOMPLETE_INT_ENABLE (1 << 1)
1609# define SEM_WAIT_INT_ENABLE (1 << 2)
1610# define DATA_SWAP_ENABLE (1 << 3)
1611# define FENCE_SWAP_ENABLE (1 << 4)
1612# define CTXEMPTY_INT_ENABLE (1 << 28)
1613#define DMA_STATUS_REG 0x340d
1614# define DMA_IDLE (1 << 0)
1615#define DMA_TILING_CONFIG 0x342e
1616
1617#define DMA_POWER_CNTL 0x342f
1618# define MEM_POWER_OVERRIDE (1 << 8)
1619#define DMA_CLK_CTRL 0x3430
1620
1621#define DMA_PG 0x3435
1622# define PG_CNTL_ENABLE (1 << 0)
1623#define DMA_PGFSM_CONFIG 0x3436
1624#define DMA_PGFSM_WRITE 0x3437
1625 556
1626#define DMA_PACKET(cmd, b, t, s, n) ((((cmd) & 0xF) << 28) | \557#define DMA_PACKET(cmd, b, t, s, n) ((((cmd) & 0xF) << 28) | \
1627 (((b) & 0x1) << 26) | \558 (((b) & 0x1) << 26) | \
@@ -1650,6 +581,7 @@ · 22 unmodified lines
1650#define DMA_PACKET_POLL_REG_MEM 0xe581#define DMA_PACKET_POLL_REG_MEM 0xe
1651#define DMA_PACKET_NOP 0xf582#define DMA_PACKET_NOP 0xf
1652 583
584/* VCE */
1653#define VCE_STATUS 0x20004585#define VCE_STATUS 0x20004
1654#define VCE_VCPU_CNTL 0x20014586#define VCE_VCPU_CNTL 0x20014
1655#define VCE_CLK_EN (1 << 0)587#define VCE_CLK_EN (1 << 0)
@@ -1726,378 +658,118 @@ · 70 unmodified lines
1726 658
1727#define CURSOR_WIDTH 64659#define CURSOR_WIDTH 64
1728#define CURSOR_HEIGHT 64660#define CURSOR_HEIGHT 64
1729#define AMDGPU_MM_INDEX 0x0000661
1730#define AMDGPU_MM_DATA 0x0001
1731
1732#define VERDE_NUM_CRTC 6
1733#define BLACKOUT_MODE_MASK 0x00000007
1734#define VGA_RENDER_CONTROL 0xC0
1735#define R_000300_VGA_RENDER_CONTROL 0xC0
1736#define C_000300_VGA_VSTATUS_CNTL 0xFFFCFFFF
1737#define EVERGREEN_CRTC_STATUS 0x1BA3
1738#define EVERGREEN_CRTC_V_BLANK (1 << 0)
1739#define EVERGREEN_CRTC_STATUS_POSITION 0x1BA4
1740/* CRTC blocks at 0x6df0, 0x79f0, 0x105f0, 0x111f0, 0x11df0, 0x129f0 */
1741#define EVERGREEN_CRTC_V_BLANK_START_END 0x1b8d
1742#define EVERGREEN_CRTC_CONTROL 0x1b9c
1743#define EVERGREEN_CRTC_MASTER_EN (1 << 0)
1744#define EVERGREEN_CRTC_DISP_READ_REQUEST_DISABLE (1 << 24)
1745#define EVERGREEN_CRTC_BLANK_CONTROL 0x1b9d
1746#define EVERGREEN_CRTC_BLANK_DATA_EN (1 << 8)
1747#define EVERGREEN_CRTC_V_BLANK (1 << 0)
1748#define EVERGREEN_CRTC_STATUS_HV_COUNT 0x1ba8
1749#define EVERGREEN_CRTC_UPDATE_LOCK 0x1bb5
1750#define EVERGREEN_MASTER_UPDATE_LOCK 0x1bbd
1751#define EVERGREEN_MASTER_UPDATE_MODE 0x1bbe
1752#define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16)
1753#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07
1754#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08
1755#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS 0x1a04
1756#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS 0x1a05
1757#define EVERGREEN_GRPH_UPDATE 0x1a11
1758#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS 0xc4
1759#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS_HIGH 0xc9
1760#define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2)
1761
1762#define EVERGREEN_DATA_FORMAT 0x1ac0
1763# define EVERGREEN_INTERLEAVE_EN (1 << 0)
1764
1765#define R600_D1GRPH_ARRAY_MODE_LINEAR_GENERAL (0 << 20)
1766#define R600_D1GRPH_ARRAY_MODE_LINEAR_ALIGNED (1 << 20)
1767#define R600_D1GRPH_ARRAY_MODE_1D_TILED_THIN1 (2 << 20)
1768#define R600_D1GRPH_ARRAY_MODE_2D_TILED_THIN1 (4 << 20)
1769
1770#define R700_D1GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a45
1771#define R700_D2GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1845
1772
1773#define R700_D2GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1847
1774#define R700_D1GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a47
1775
1776#define R600_D1GRPH_SWAP_CONTROL 0x1843
1777#define R600_D1GRPH_SWAP_ENDIAN_NONE (0 << 0)
1778#define R600_D1GRPH_SWAP_ENDIAN_16BIT (1 << 0)
1779#define R600_D1GRPH_SWAP_ENDIAN_32BIT (2 << 0)
1780#define R600_D1GRPH_SWAP_ENDIAN_64BIT (3 << 0)
1781
1782#define AVIVO_D1VGA_CONTROL 0x00cc
1783# define AVIVO_DVGA_CONTROL_MODE_ENABLE (1 << 0)
1784# define AVIVO_DVGA_CONTROL_TIMING_SELECT (1 << 8)
1785# define AVIVO_DVGA_CONTROL_SYNC_POLARITY_SELECT (1 << 9)
1786# define AVIVO_DVGA_CONTROL_OVERSCAN_TIMING_SELECT (1 << 10)
1787# define AVIVO_DVGA_CONTROL_OVERSCAN_COLOR_EN (1 << 16)
1788# define AVIVO_DVGA_CONTROL_ROTATE (1 << 24)
1789#define AVIVO_D2VGA_CONTROL 0x00ce
1790
1791#define R600_BUS_CNTL 0x1508
1792# define R600_BIOS_ROM_DIS (1 << 1)
1793 662
1794#define R600_ROM_CNTL 0x580663#define R600_ROM_CNTL 0x580
1795# define R600_SCK_OVERWRITE (1 << 1)664# define R600_SCK_OVERWRITE (1 << 1)
1796# define R600_SCK_PRESCALE_CRYSTAL_CLK_SHIFT 28665# define R600_SCK_PRESCALE_CRYSTAL_CLK_SHIFT 28
1797# define R600_SCK_PRESCALE_CRYSTAL_CLK_MASK (0xf << 28)666# define R600_SCK_PRESCALE_CRYSTAL_CLK_MASK (0xf << 28)
1798 667
1799#define FMT_BIT_DEPTH_CONTROL 0x1bf2668#define GRPH_ARRAY_LINEAR_GENERAL 0
1800#define FMT_TRUNCATE_EN (1 << 0)669#define GRPH_ARRAY_LINEAR_ALIGNED 1
1801#define FMT_TRUNCATE_DEPTH (1 << 4)670#define GRPH_ARRAY_1D_TILED_THIN1 2
1802#define FMT_SPATIAL_DITHER_EN (1 << 8)671#define GRPH_ARRAY_2D_TILED_THIN1 4
1803#define FMT_SPATIAL_DITHER_MODE(x) ((x) << 9)672
1804#define FMT_SPATIAL_DITHER_DEPTH (1 << 12)673#define ES_AND_GS_AUTO 3
1805#define FMT_FRAME_RANDOM_ENABLE (1 << 13)674#define BUF_SWAP_32BIT (2 << 16)
1806#define FMT_RGB_RANDOM_ENABLE (1 << 14)675
1807#define FMT_HIGHPASS_RANDOM_ENABLE (1 << 15)676#define GRPH_DEPTH_8BPP 0
1808#define FMT_TEMPORAL_DITHER_EN (1 << 16)677#define GRPH_DEPTH_16BPP 1
1809#define FMT_TEMPORAL_DITHER_DEPTH (1 << 20)678#define GRPH_DEPTH_32BPP 2
1810#define FMT_TEMPORAL_DITHER_OFFSET(x) ((x) << 21)679
1811#define FMT_TEMPORAL_LEVEL (1 << 24)680/* 8 BPP */
1812#define FMT_TEMPORAL_DITHER_RESET (1 << 25)681#define GRPH_FORMAT_INDEXED 0
1813#define FMT_25FRC_SEL(x) ((x) << 26)682
1814#define FMT_50FRC_SEL(x) ((x) << 28)683/* 16 BPP */
1815#define FMT_75FRC_SEL(x) ((x) << 30)684#define GRPH_FORMAT_ARGB1555 0
1816 685#define GRPH_FORMAT_ARGB565 1
1817#define EVERGREEN_DC_LUT_CONTROL 0x1a80686#define GRPH_FORMAT_ARGB4444 2
1818#define EVERGREEN_DC_LUT_BLACK_OFFSET_BLUE 0x1a81687#define GRPH_FORMAT_AI88 3
1819#define EVERGREEN_DC_LUT_BLACK_OFFSET_GREEN 0x1a82688#define GRPH_FORMAT_MONO16 4
1820#define EVERGREEN_DC_LUT_BLACK_OFFSET_RED 0x1a83689#define GRPH_FORMAT_BGRA5551 5
1821#define EVERGREEN_DC_LUT_WHITE_OFFSET_BLUE 0x1a84
1822#define EVERGREEN_DC_LUT_WHITE_OFFSET_GREEN 0x1a85
1823#define EVERGREEN_DC_LUT_WHITE_OFFSET_RED 0x1a86
1824#define EVERGREEN_DC_LUT_30_COLOR 0x1a7c
1825#define EVERGREEN_DC_LUT_RW_INDEX 0x1a79
1826#define EVERGREEN_DC_LUT_WRITE_EN_MASK 0x1a7e
1827#define EVERGREEN_DC_LUT_RW_MODE 0x1a78
1828
1829#define EVERGREEN_GRPH_ENABLE 0x1a00
1830#define EVERGREEN_GRPH_CONTROL 0x1a01
1831#define EVERGREEN_GRPH_DEPTH(x) (((x) & 0x3) << 0)
1832#define EVERGREEN_GRPH_DEPTH_8BPP 0
1833#define EVERGREEN_GRPH_DEPTH_16BPP 1
1834#define EVERGREEN_GRPH_DEPTH_32BPP 2
1835#define EVERGREEN_GRPH_NUM_BANKS(x) (((x) & 0x3) << 2)
1836#define EVERGREEN_ADDR_SURF_2_BANK 0
1837#define EVERGREEN_ADDR_SURF_4_BANK 1
1838#define EVERGREEN_ADDR_SURF_8_BANK 2
1839#define EVERGREEN_ADDR_SURF_16_BANK 3
1840#define EVERGREEN_GRPH_Z(x) (((x) & 0x3) << 4)
1841#define EVERGREEN_GRPH_BANK_WIDTH(x) (((x) & 0x3) << 6)
1842#define EVERGREEN_ADDR_SURF_BANK_WIDTH_1 0
1843#define EVERGREEN_ADDR_SURF_BANK_WIDTH_2 1
1844#define EVERGREEN_ADDR_SURF_BANK_WIDTH_4 2
1845#define EVERGREEN_ADDR_SURF_BANK_WIDTH_8 3
1846#define EVERGREEN_GRPH_FORMAT(x) (((x) & 0x7) << 8)
1847
1848#define EVERGREEN_GRPH_FORMAT_INDEXED 0
1849#define EVERGREEN_GRPH_FORMAT_ARGB1555 0
1850#define EVERGREEN_GRPH_FORMAT_ARGB565 1
1851#define EVERGREEN_GRPH_FORMAT_ARGB4444 2
1852#define EVERGREEN_GRPH_FORMAT_AI88 3
1853#define EVERGREEN_GRPH_FORMAT_MONO16 4
1854#define EVERGREEN_GRPH_FORMAT_BGRA5551 5
1855 690
1856/* 32 BPP */691/* 32 BPP */
1857#define EVERGREEN_GRPH_FORMAT_ARGB8888 0692#define GRPH_FORMAT_ARGB8888 0
1858#define EVERGREEN_GRPH_FORMAT_ARGB2101010 1693#define GRPH_FORMAT_ARGB2101010 1
1859#define EVERGREEN_GRPH_FORMAT_32BPP_DIG 2694#define GRPH_FORMAT_32BPP_DIG 2
1860#define EVERGREEN_GRPH_FORMAT_8B_ARGB2101010 3695#define GRPH_FORMAT_8B_ARGB2101010 3
1861#define EVERGREEN_GRPH_FORMAT_BGRA1010102 4696#define GRPH_FORMAT_BGRA1010102 4
1862#define EVERGREEN_GRPH_FORMAT_8B_BGRA1010102 5697#define GRPH_FORMAT_8B_BGRA1010102 5
1863#define EVERGREEN_GRPH_FORMAT_RGB111110 6698#define GRPH_FORMAT_RGB111110 6
1864#define EVERGREEN_GRPH_FORMAT_BGR101111 7699#define GRPH_FORMAT_BGR101111 7
1865#define EVERGREEN_GRPH_BANK_HEIGHT(x) (((x) & 0x3) << 11)700
1866#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_1 0701#define GRPH_ENDIAN_NONE 0
1867#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_2 1702#define GRPH_ENDIAN_8IN16 1
1868#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_4 2703#define GRPH_ENDIAN_8IN32 2
1869#define EVERGREEN_ADDR_SURF_BANK_HEIGHT_8 3704#define GRPH_ENDIAN_8IN64 3
1870#define EVERGREEN_GRPH_TILE_SPLIT(x) (((x) & 0x7) << 13)705#define GRPH_RED_SEL_R 0
1871#define EVERGREEN_ADDR_SURF_TILE_SPLIT_64B 0706#define GRPH_RED_SEL_G 1
1872#define EVERGREEN_ADDR_SURF_TILE_SPLIT_128B 1707#define GRPH_RED_SEL_B 2
1873#define EVERGREEN_ADDR_SURF_TILE_SPLIT_256B 2708#define GRPH_RED_SEL_A 3
1874#define EVERGREEN_ADDR_SURF_TILE_SPLIT_512B 3709
1875#define EVERGREEN_ADDR_SURF_TILE_SPLIT_1KB 4710#define GRPH_GREEN_SEL_G 0
1876#define EVERGREEN_ADDR_SURF_TILE_SPLIT_2KB 5711#define GRPH_GREEN_SEL_B 1
1877#define EVERGREEN_ADDR_SURF_TILE_SPLIT_4KB 6712#define GRPH_GREEN_SEL_A 2
1878#define EVERGREEN_GRPH_MACRO_TILE_ASPECT(x) (((x) & 0x3) << 18)713#define GRPH_GREEN_SEL_R 3
1879#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_1 0714
1880#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_2 1715#define GRPH_BLUE_SEL_B 0
1881#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_4 2716#define GRPH_BLUE_SEL_A 1
1882#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_8 3717#define GRPH_BLUE_SEL_R 2
1883#define EVERGREEN_GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20)718#define GRPH_BLUE_SEL_G 3
1884#define EVERGREEN_GRPH_ARRAY_LINEAR_GENERAL 0719
1885#define EVERGREEN_GRPH_ARRAY_LINEAR_ALIGNED 1720#define GRPH_ALPHA_SEL_A 0
1886#define EVERGREEN_GRPH_ARRAY_1D_TILED_THIN1 2721#define GRPH_ALPHA_SEL_R 1
1887#define EVERGREEN_GRPH_ARRAY_2D_TILED_THIN1 4722#define GRPH_ALPHA_SEL_G 2
1888#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_1 0723#define GRPH_ALPHA_SEL_B 3
1889#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_2 1724
1890#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_4 2725/* CUR_CONTROL */
1891#define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_8 3726 #define CURSOR_MONO 0
1892 727 #define CURSOR_24_1 1
1893#define EVERGREEN_GRPH_SWAP_CONTROL 0x1a03728 #define CURSOR_24_8_PRE_MULT 2
1894#define EVERGREEN_GRPH_ENDIAN_SWAP(x) (((x) & 0x3) << 0)729 #define CURSOR_24_8_UNPRE_MULT 3
1895# define EVERGREEN_GRPH_ENDIAN_NONE 0730 #define CURSOR_URGENT_ALWAYS 0
1896# define EVERGREEN_GRPH_ENDIAN_8IN16 1731 #define CURSOR_URGENT_1_8 1
1897# define EVERGREEN_GRPH_ENDIAN_8IN32 2732 #define CURSOR_URGENT_1_4 2
1898# define EVERGREEN_GRPH_ENDIAN_8IN64 3733 #define CURSOR_URGENT_3_8 3
1899#define EVERGREEN_GRPH_RED_CROSSBAR(x) (((x) & 0x3) << 4)734 #define CURSOR_URGENT_1_2 4
1900# define EVERGREEN_GRPH_RED_SEL_R 0735
1901# define EVERGREEN_GRPH_RED_SEL_G 1736/* INPUT_CSC_CONTROL */
1902# define EVERGREEN_GRPH_RED_SEL_B 2737# define INPUT_CSC_BYPASS 0
1903# define EVERGREEN_GRPH_RED_SEL_A 3738# define INPUT_CSC_PROG_COEFF 1
1904#define EVERGREEN_GRPH_GREEN_CROSSBAR(x) (((x) & 0x3) << 6)739# define INPUT_CSC_PROG_SHARED_MATRIXA 2
1905# define EVERGREEN_GRPH_GREEN_SEL_G 0740
1906# define EVERGREEN_GRPH_GREEN_SEL_B 1741/* OUTPUT_CSC_CONTROL */
1907# define EVERGREEN_GRPH_GREEN_SEL_A 2742# define OUTPUT_CSC_BYPASS 0
1908# define EVERGREEN_GRPH_GREEN_SEL_R 3743# define OUTPUT_CSC_TV_RGB 1
1909#define EVERGREEN_GRPH_BLUE_CROSSBAR(x) (((x) & 0x3) << 8)744# define OUTPUT_CSC_YCBCR_601 2
1910# define EVERGREEN_GRPH_BLUE_SEL_B 0745# define OUTPUT_CSC_YCBCR_709 3
1911# define EVERGREEN_GRPH_BLUE_SEL_A 1746# define OUTPUT_CSC_PROG_COEFF 4
1912# define EVERGREEN_GRPH_BLUE_SEL_R 2747# define OUTPUT_CSC_PROG_SHARED_MATRIXB 5
1913# define EVERGREEN_GRPH_BLUE_SEL_G 3748
1914#define EVERGREEN_GRPH_ALPHA_CROSSBAR(x) (((x) & 0x3) << 10)749/* DEGAMMA_CONTROL */
1915# define EVERGREEN_GRPH_ALPHA_SEL_A 0750# define DEGAMMA_BYPASS 0
1916# define EVERGREEN_GRPH_ALPHA_SEL_R 1751# define DEGAMMA_SRGB_24 1
1917# define EVERGREEN_GRPH_ALPHA_SEL_G 2752# define DEGAMMA_XVYCC_222 2
1918# define EVERGREEN_GRPH_ALPHA_SEL_B 3753
1919 754/* GAMUT_REMAP_CONTROL */
1920#define EVERGREEN_D3VGA_CONTROL 0xf8755# define GAMUT_REMAP_BYPASS 0
1921#define EVERGREEN_D4VGA_CONTROL 0xf9756# define GAMUT_REMAP_PROG_COEFF 1
1922#define EVERGREEN_D5VGA_CONTROL 0xfa757# define GAMUT_REMAP_PROG_SHARED_MATRIXA 2
1923#define EVERGREEN_D6VGA_CONTROL 0xfb758# define GAMUT_REMAP_PROG_SHARED_MATRIXB 3
1924 759
1925#define EVERGREEN_GRPH_SURFACE_ADDRESS_MASK 0xffffff00760/* REGAMMA_CONTROL */
1926 761# define REGAMMA_BYPASS 0
1927#define EVERGREEN_GRPH_LUT_10BIT_BYPASS_CONTROL 0x1a02762# define REGAMMA_SRGB_24 1
1928#define EVERGREEN_LUT_10BIT_BYPASS_EN (1 << 8)763# define REGAMMA_XVYCC_222 2
1929 764# define REGAMMA_PROG_A 3
1930#define EVERGREEN_GRPH_PITCH 0x1a06765# define REGAMMA_PROG_B 4
1931#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07766
1932#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08767
1933#define EVERGREEN_GRPH_SURFACE_OFFSET_X 0x1a09768/* INPUT_GAMMA_CONTROL */
1934#define EVERGREEN_GRPH_SURFACE_OFFSET_Y 0x1a0a769# define INPUT_GAMMA_USE_LUT 0
1935#define EVERGREEN_GRPH_X_START 0x1a0b770# define INPUT_GAMMA_BYPASS 1
1936#define EVERGREEN_GRPH_Y_START 0x1a0c771# define INPUT_GAMMA_SRGB_24 2
1937#define EVERGREEN_GRPH_X_END 0x1a0d772# define INPUT_GAMMA_XVYCC_222 3
1938#define EVERGREEN_GRPH_Y_END 0x1a0e
1939#define EVERGREEN_GRPH_UPDATE 0x1a11
1940#define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2)
1941#define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16)
1942#define EVERGREEN_GRPH_FLIP_CONTROL 0x1a12
1943#define EVERGREEN_GRPH_SURFACE_UPDATE_H_RETRACE_EN (1 << 0)
1944
1945#define EVERGREEN_VIEWPORT_START 0x1b5c
1946#define EVERGREEN_VIEWPORT_SIZE 0x1b5d
1947#define EVERGREEN_DESKTOP_HEIGHT 0x1ac1
1948
1949/* CUR blocks at 0x6998, 0x7598, 0x10198, 0x10d98, 0x11998, 0x12598 */
1950#define EVERGREEN_CUR_CONTROL 0x1a66
1951# define EVERGREEN_CURSOR_EN (1 << 0)
1952# define EVERGREEN_CURSOR_MODE(x) (((x) & 0x3) << 8)
1953# define EVERGREEN_CURSOR_MONO 0
1954# define EVERGREEN_CURSOR_24_1 1
1955# define EVERGREEN_CURSOR_24_8_PRE_MULT 2
1956# define EVERGREEN_CURSOR_24_8_UNPRE_MULT 3
1957# define EVERGREEN_CURSOR_2X_MAGNIFY (1 << 16)
1958# define EVERGREEN_CURSOR_FORCE_MC_ON (1 << 20)
1959# define EVERGREEN_CURSOR_URGENT_CONTROL(x) (((x) & 0x7) << 24)
1960# define EVERGREEN_CURSOR_URGENT_ALWAYS 0
1961# define EVERGREEN_CURSOR_URGENT_1_8 1
1962# define EVERGREEN_CURSOR_URGENT_1_4 2
1963# define EVERGREEN_CURSOR_URGENT_3_8 3
1964# define EVERGREEN_CURSOR_URGENT_1_2 4
1965#define EVERGREEN_CUR_SURFACE_ADDRESS 0x1a67
1966# define EVERGREEN_CUR_SURFACE_ADDRESS_MASK 0xfffff000
1967#define EVERGREEN_CUR_SIZE 0x1a68
1968#define EVERGREEN_CUR_SURFACE_ADDRESS_HIGH 0x1a69
1969#define EVERGREEN_CUR_POSITION 0x1a6a
1970#define EVERGREEN_CUR_HOT_SPOT 0x1a6b
1971#define EVERGREEN_CUR_COLOR1 0x1a6c
1972#define EVERGREEN_CUR_COLOR2 0x1a6d
1973#define EVERGREEN_CUR_UPDATE 0x1a6e
1974# define EVERGREEN_CURSOR_UPDATE_PENDING (1 << 0)
1975# define EVERGREEN_CURSOR_UPDATE_TAKEN (1 << 1)
1976# define EVERGREEN_CURSOR_UPDATE_LOCK (1 << 16)
1977# define EVERGREEN_CURSOR_DISABLE_MULTIPLE_UPDATE (1 << 24)
1978
1979
1980#define NI_INPUT_CSC_CONTROL 0x1a35
1981# define NI_INPUT_CSC_GRPH_MODE(x) (((x) & 0x3) << 0)
1982# define NI_INPUT_CSC_BYPASS 0
1983# define NI_INPUT_CSC_PROG_COEFF 1
1984# define NI_INPUT_CSC_PROG_SHARED_MATRIXA 2
1985# define NI_INPUT_CSC_OVL_MODE(x) (((x) & 0x3) << 4)
1986
1987#define NI_OUTPUT_CSC_CONTROL 0x1a3c
1988# define NI_OUTPUT_CSC_GRPH_MODE(x) (((x) & 0x7) << 0)
1989# define NI_OUTPUT_CSC_BYPASS 0
1990# define NI_OUTPUT_CSC_TV_RGB 1
1991# define NI_OUTPUT_CSC_YCBCR_601 2
1992# define NI_OUTPUT_CSC_YCBCR_709 3
1993# define NI_OUTPUT_CSC_PROG_COEFF 4
1994# define NI_OUTPUT_CSC_PROG_SHARED_MATRIXB 5
1995# define NI_OUTPUT_CSC_OVL_MODE(x) (((x) & 0x7) << 4)
1996
1997#define NI_DEGAMMA_CONTROL 0x1a58
1998# define NI_GRPH_DEGAMMA_MODE(x) (((x) & 0x3) << 0)
1999# define NI_DEGAMMA_BYPASS 0
2000# define NI_DEGAMMA_SRGB_24 1
2001# define NI_DEGAMMA_XVYCC_222 2
2002# define NI_OVL_DEGAMMA_MODE(x) (((x) & 0x3) << 4)
2003# define NI_ICON_DEGAMMA_MODE(x) (((x) & 0x3) << 8)
2004# define NI_CURSOR_DEGAMMA_MODE(x) (((x) & 0x3) << 12)
2005
2006#define NI_GAMUT_REMAP_CONTROL 0x1a59
2007# define NI_GRPH_GAMUT_REMAP_MODE(x) (((x) & 0x3) << 0)
2008# define NI_GAMUT_REMAP_BYPASS 0
2009# define NI_GAMUT_REMAP_PROG_COEFF 1
2010# define NI_GAMUT_REMAP_PROG_SHARED_MATRIXA 2
2011# define NI_GAMUT_REMAP_PROG_SHARED_MATRIXB 3
2012# define NI_OVL_GAMUT_REMAP_MODE(x) (((x) & 0x3) << 4)
2013
2014#define NI_REGAMMA_CONTROL 0x1aa0
2015# define NI_GRPH_REGAMMA_MODE(x) (((x) & 0x7) << 0)
2016# define NI_REGAMMA_BYPASS 0
2017# define NI_REGAMMA_SRGB_24 1
2018# define NI_REGAMMA_XVYCC_222 2
2019# define NI_REGAMMA_PROG_A 3
2020# define NI_REGAMMA_PROG_B 4
2021# define NI_OVL_REGAMMA_MODE(x) (((x) & 0x7) << 4)
2022
2023
2024#define NI_PRESCALE_GRPH_CONTROL 0x1a2d
2025# define NI_GRPH_PRESCALE_BYPASS (1 << 4)
2026
2027#define NI_PRESCALE_OVL_CONTROL 0x1a31
2028# define NI_OVL_PRESCALE_BYPASS (1 << 4)
2029
2030#define NI_INPUT_GAMMA_CONTROL 0x1a10
2031# define NI_GRPH_INPUT_GAMMA_MODE(x) (((x) & 0x3) << 0)
2032# define NI_INPUT_GAMMA_USE_LUT 0
2033# define NI_INPUT_GAMMA_BYPASS 1
2034# define NI_INPUT_GAMMA_SRGB_24 2
2035# define NI_INPUT_GAMMA_XVYCC_222 3
2036# define NI_OVL_INPUT_GAMMA_MODE(x) (((x) & 0x3) << 4)
2037
2038#define BLACKOUT_MODE_MASK 0x00000007
2039#define VGA_RENDER_CONTROL 0xC0
2040#define R_000300_VGA_RENDER_CONTROL 0xC0
2041#define C_000300_VGA_VSTATUS_CNTL 0xFFFCFFFF
2042#define EVERGREEN_CRTC_STATUS 0x1BA3
2043#define EVERGREEN_CRTC_V_BLANK (1 << 0)
2044#define EVERGREEN_CRTC_STATUS_POSITION 0x1BA4
2045/* CRTC blocks at 0x6df0, 0x79f0, 0x105f0, 0x111f0, 0x11df0, 0x129f0 */
2046#define EVERGREEN_CRTC_V_BLANK_START_END 0x1b8d
2047#define EVERGREEN_CRTC_CONTROL 0x1b9c
2048# define EVERGREEN_CRTC_MASTER_EN (1 << 0)
2049# define EVERGREEN_CRTC_DISP_READ_REQUEST_DISABLE (1 << 24)
2050#define EVERGREEN_CRTC_BLANK_CONTROL 0x1b9d
2051# define EVERGREEN_CRTC_BLANK_DATA_EN (1 << 8)
2052# define EVERGREEN_CRTC_V_BLANK (1 << 0)
2053#define EVERGREEN_CRTC_STATUS_HV_COUNT 0x1ba8
2054#define EVERGREEN_CRTC_UPDATE_LOCK 0x1bb5
2055#define EVERGREEN_MASTER_UPDATE_LOCK 0x1bbd
2056#define EVERGREEN_MASTER_UPDATE_MODE 0x1bbe
2057#define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16)
2058#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07
2059#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08
2060#define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS 0x1a04
2061#define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS 0x1a05
2062#define EVERGREEN_GRPH_UPDATE 0x1a11
2063#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS 0xc4
2064#define EVERGREEN_VGA_MEMORY_BASE_ADDRESS_HIGH 0xc9
2065#define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2)
2066
2067#define mmVM_CONTEXT1_CNTL__xxRANGE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x10
2068#define mmVM_CONTEXT1_CNTL__xxRANGE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x4
2069#define mmVM_CONTEXT1_CNTL__xxDUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x80
2070#define mmVM_CONTEXT1_CNTL__xxDUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x7
2071#define mmVM_CONTEXT1_CNTL__xxPDE0_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x400
2072#define mmVM_CONTEXT1_CNTL__xxPDE0_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0xa
2073#define mmVM_CONTEXT1_CNTL__xxVALID_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x2000
2074#define mmVM_CONTEXT1_CNTL__xxVALID_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0xd
2075#define mmVM_CONTEXT1_CNTL__xxREAD_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x10000
2076#define mmVM_CONTEXT1_CNTL__xxREAD_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x10
2077#define mmVM_CONTEXT1_CNTL__xxWRITE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x80000
2078#define mmVM_CONTEXT1_CNTL__xxWRITE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x13
2079
2080#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxVMID_MASK 0x1e000000
2081#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxVMID__SHIFT 0x19
2082#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxPROTECTIONS_MASK 0xff
2083#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxPROTECTIONS__SHIFT 0x0
2084#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_ID_MASK 0xff000
2085#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_ID__SHIFT 0xc
2086#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_RW_MASK 0x1000000
2087#define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_RW__SHIFT 0x18
2088
2089#define mmMC_SHARED_BLACKOUT_CNTL__xxBLACKOUT_MODE_MASK 0x7
2090#define mmMC_SHARED_BLACKOUT_CNTL__xxBLACKOUT_MODE__SHIFT 0x0
2091
2092#define mmBIF_FB_EN__xxFB_READ_EN_MASK 0x1
2093#define mmBIF_FB_EN__xxFB_READ_EN__SHIFT 0x0
2094#define mmBIF_FB_EN__xxFB_WRITE_EN_MASK 0x2
2095#define mmBIF_FB_EN__xxFB_WRITE_EN__SHIFT 0x1
2096
2097#define mmSRBM_SOFT_RESET__xxSOFT_RESET_VMC_MASK 0x20000
2098#define mmSRBM_SOFT_RESET__xxSOFT_RESET_VMC__SHIFT 0x11
2099#define mmSRBM_SOFT_RESET__xxSOFT_RESET_MC_MASK 0x800
2100#define mmSRBM_SOFT_RESET__xxSOFT_RESET_MC__SHIFT 0xb
2101 773
2102#define MC_SEQ_MISC0__MT__MASK 0xf0000000774#define MC_SEQ_MISC0__MT__MASK 0xf0000000
2103#define MC_SEQ_MISC0__MT__GDDR1 0x10000000775#define MC_SEQ_MISC0__MT__GDDR1 0x10000000
@@ -2113,20 +785,9 @@ · 9 unmodified lines
2113#define PACKET3_SEM_SEL_SIGNAL (0x6 << 29)785#define PACKET3_SEM_SEL_SIGNAL (0x6 << 29)
2114#define PACKET3_SEM_SEL_WAIT (0x7 << 29)786#define PACKET3_SEM_SEL_WAIT (0x7 << 29)
2115 787
2116#define CONFIG_CNTL 0x1509
2117#define CC_DRM_ID_STRAPS 0X1559
2118#define AMDGPU_PCIE_INDEX 0xc788#define AMDGPU_PCIE_INDEX 0xc
2119#define AMDGPU_PCIE_DATA 0xd789#define AMDGPU_PCIE_DATA 0xd
2120 790
2121#define DMA_SEM_INCOMPLETE_TIMER_CNTL 0x3411
2122#define DMA_SEM_WAIT_FAIL_TIMER_CNTL 0x3412
2123#define DMA_MODE 0x342f
2124#define DMA_RB_RPTR_ADDR_HI 0x3407
2125#define DMA_RB_RPTR_ADDR_LO 0x3408
2126#define DMA_BUSY_MASK 0x20
2127#define DMA1_BUSY_MASK 0X40
2128#define SDMA_MAX_INSTANCE 2
2129
2130#define PCIE_BUS_CLK 10000791#define PCIE_BUS_CLK 10000
2131#define TCLK (PCIE_BUS_CLK / 10)792#define TCLK (PCIE_BUS_CLK / 10)
2132#define PCIE_PORT_INDEX 0xe793#define PCIE_PORT_INDEX 0xe
@@ -2136,8 +797,6 @@ · 3 unmodified lines
2136#define EVERGREEN_PIF_PHY1_INDEX 0x10797#define EVERGREEN_PIF_PHY1_INDEX 0x10
2137#define EVERGREEN_PIF_PHY1_DATA 0x14798#define EVERGREEN_PIF_PHY1_DATA 0x14
2138 799
2139#define MC_VM_FB_OFFSET 0x81a
2140
2141/* Discrete VCE clocks */800/* Discrete VCE clocks */
2142#define CG_VCEPLL_FUNC_CNTL 0xc0030600801#define CG_VCEPLL_FUNC_CNTL 0xc0030600
2143#define VCEPLL_RESET_MASK 0x00000001802#define VCEPLL_RESET_MASK 0x00000001
drivers/gpu/drm/amd/amdgpu/soc15.c+5−0
@@ -584,6 +584,8 @@ soc15_asic_reset_method(struct amdgpu_device *adev) · 583 unmodified lines
584 * Enable triggering of GPU reset only if specified584 * Enable triggering of GPU reset only if specified
585 * by module parameter.585 * by module parameter.
586 */586 */
587 if (adev->pcie_reset_ctx.in_link_reset)
588 return AMD_RESET_METHOD_LINK;
587 if (amdgpu_gpu_recovery == 4 || amdgpu_gpu_recovery == 5)589 if (amdgpu_gpu_recovery == 4 || amdgpu_gpu_recovery == 5)
588 return AMD_RESET_METHOD_MODE2;590 return AMD_RESET_METHOD_MODE2;
589 else if (!(adev->flags & AMD_IS_APU))591 else if (!(adev->flags & AMD_IS_APU))
@@ -640,6 +642,9 @@ asic_reset: · 50 unmodified lines
640 case AMD_RESET_METHOD_MODE2:642 case AMD_RESET_METHOD_MODE2:
641 dev_info(adev->dev, "MODE2 reset\n");643 dev_info(adev->dev, "MODE2 reset\n");
642 return amdgpu_dpm_mode2_reset(adev);644 return amdgpu_dpm_mode2_reset(adev);
645 case AMD_RESET_METHOD_LINK:
646 dev_info(adev->dev, "Link reset\n");
647 return amdgpu_device_link_reset(adev);
643 default:648 default:
644 dev_info(adev->dev, "MODE1 reset\n");649 dev_info(adev->dev, "MODE1 reset\n");
645 return amdgpu_device_mode1_reset(adev);650 return amdgpu_device_mode1_reset(adev);
drivers/gpu/drm/amd/amdgpu/soc15d.h+5−0
@@ -552,6 +552,11 @@ · 551 unmodified lines
552# define PACKET3_QUERY_STATUS_DOORBELL_OFFSET(x) ((x) << 2)552# define PACKET3_QUERY_STATUS_DOORBELL_OFFSET(x) ((x) << 2)
553# define PACKET3_QUERY_STATUS_ENG_SEL(x) ((x) << 25)553# define PACKET3_QUERY_STATUS_ENG_SEL(x) ((x) << 25)
554 554
555#define PACKET3_RUN_CLEANER_SHADER_9_0 0xD7
556/* 1. header
557 * 2. RESERVED [31:0]
558 */
559
555#define PACKET3_RUN_CLEANER_SHADER 0xD2560#define PACKET3_RUN_CLEANER_SHADER 0xD2
556/* 1. header561/* 1. header
557 * 2. RESERVED [31:0]562 * 2. RESERVED [31:0]
drivers/gpu/drm/amd/amdgpu/umc_v12_0.c+10−0
@@ -220,6 +220,13 @@ static int umc_v12_0_convert_error_address(struct amdgpu_device *adev, · 219 unmodified lines
220 nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);220 nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);
221 221
222 /* other nps modes are taken as nps1 */222 /* other nps modes are taken as nps1 */
223 if (nps == AMDGPU_NPS2_PARTITION_MODE) {
224 loop_bits[0] = UMC_V12_0_PA_CH5_BIT;
225 loop_bits[1] = UMC_V12_0_PA_C2_BIT;
226 loop_bits[2] = UMC_V12_0_PA_B1_BIT;
227 loop_bits[3] = UMC_V12_0_PA_R12_BIT;
228 }
229
223 if (nps == AMDGPU_NPS4_PARTITION_MODE) {230 if (nps == AMDGPU_NPS4_PARTITION_MODE) {
224 loop_bits[0] = UMC_V12_0_PA_CH4_BIT;231 loop_bits[0] = UMC_V12_0_PA_CH4_BIT;
225 loop_bits[1] = UMC_V12_0_PA_CH5_BIT;232 loop_bits[1] = UMC_V12_0_PA_CH5_BIT;
@@ -517,6 +524,9 @@ static int umc_v12_0_update_ecc_status(struct amdgpu_device *adev, · 291 unmodified lines
517 524
518 if (adev->gmc.gmc_funcs->query_mem_partition_mode)525 if (adev->gmc.gmc_funcs->query_mem_partition_mode)
519 nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);526 nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev);
527
528 if (nps == AMDGPU_NPS2_PARTITION_MODE)
529 shift_bit = UMC_V12_0_PA_B1_BIT;
520 if (nps == AMDGPU_NPS4_PARTITION_MODE)530 if (nps == AMDGPU_NPS4_PARTITION_MODE)
521 shift_bit = UMC_V12_0_PA_B0_BIT;531 shift_bit = UMC_V12_0_PA_B0_BIT;
522 532
drivers/gpu/drm/amd/amdgpu/umc_v12_0.h+2−0
@@ -65,12 +65,14 @@ · 64 unmodified lines
65/* row bits in SOC physical address */65/* row bits in SOC physical address */
66#define UMC_V12_0_PA_R0_BIT 2266#define UMC_V12_0_PA_R0_BIT 22
67#define UMC_V12_0_PA_R11_BIT 3367#define UMC_V12_0_PA_R11_BIT 33
68#define UMC_V12_0_PA_R12_BIT 34
68#define UMC_V12_0_PA_R13_BIT 3569#define UMC_V12_0_PA_R13_BIT 35
69/* channel bit in SOC physical address */70/* channel bit in SOC physical address */
70#define UMC_V12_0_PA_CH4_BIT 1271#define UMC_V12_0_PA_CH4_BIT 12
71#define UMC_V12_0_PA_CH5_BIT 1372#define UMC_V12_0_PA_CH5_BIT 13
72/* bank bit in SOC physical address */73/* bank bit in SOC physical address */
73#define UMC_V12_0_PA_B0_BIT 1974#define UMC_V12_0_PA_B0_BIT 19
75#define UMC_V12_0_PA_B1_BIT 20
74/* row bits in MCA address */76/* row bits in MCA address */
75#define UMC_V12_0_MA_R0_BIT 1077#define UMC_V12_0_MA_R0_BIT 10
76 78
drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c+1−0
@@ -39,6 +39,7 @@ · 38 unmodified lines
39 39
40#define VCN_VID_SOC_ADDRESS_2_0 0x1fa0040#define VCN_VID_SOC_ADDRESS_2_0 0x1fa00
41#define VCN1_VID_SOC_ADDRESS_3_0 0x4820041#define VCN1_VID_SOC_ADDRESS_3_0 0x48200
42#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
42 43
43#define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x1fd44#define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x1fd
44#define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x50345#define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x503
drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c+1−0
@@ -39,6 +39,7 @@ · 38 unmodified lines
39 39
40#define VCN_VID_SOC_ADDRESS_2_0 0x1fa0040#define VCN_VID_SOC_ADDRESS_2_0 0x1fa00
41#define VCN1_VID_SOC_ADDRESS_3_0 0x4820041#define VCN1_VID_SOC_ADDRESS_3_0 0x48200
42#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
42 43
43#define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x2744#define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27
44#define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f45#define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f
drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c+1−0
@@ -40,6 +40,7 @@ · 39 unmodified lines
40 40
41#define VCN_VID_SOC_ADDRESS_2_0 0x1fa0041#define VCN_VID_SOC_ADDRESS_2_0 0x1fa00
42#define VCN1_VID_SOC_ADDRESS_3_0 0x4820042#define VCN1_VID_SOC_ADDRESS_3_0 0x48200
43#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
43 44
44#define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x2745#define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27
45#define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f46#define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f
drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c+19−2
@@ -46,6 +46,7 @@ · 45 unmodified lines
46 46
47#define VCN_VID_SOC_ADDRESS_2_0 0x1fb0047#define VCN_VID_SOC_ADDRESS_2_0 0x1fb00
48#define VCN1_VID_SOC_ADDRESS_3_0 0x4830048#define VCN1_VID_SOC_ADDRESS_3_0 0x48300
49#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
49 50
50#define VCN_HARVEST_MMSCH 051#define VCN_HARVEST_MMSCH 0
51 52
@@ -238,9 +239,9 @@ static int vcn_v4_0_sw_init(struct amdgpu_ip_block *ip_block) · 186 unmodified lines
238 adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_pause_dpg_mode;239 adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_pause_dpg_mode;
239 }240 }
240 241
241 /* TODO: Add queue reset mask when FW fully supports it */
242 adev->vcn.supported_reset =242 adev->vcn.supported_reset =
243 amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);243 amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);
244 adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
244 245
245 if (amdgpu_sriov_vf(adev)) {246 if (amdgpu_sriov_vf(adev)) {
246 r = amdgpu_virt_alloc_mm_table(adev);247 r = amdgpu_virt_alloc_mm_table(adev);
@@ -614,7 +615,8 @@ static void vcn_v4_0_mc_resume_dpg_mode(struct amdgpu_vcn_inst *vinst, · 367 unmodified lines
614 615
615 /* VCN global tiling registers */616 /* VCN global tiling registers */
616 WREG32_SOC15_DPG_MODE(inst_idx, SOC15_DPG_MODE_OFFSET(617 WREG32_SOC15_DPG_MODE(inst_idx, SOC15_DPG_MODE_OFFSET(
617 VCN, 0, regUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect);618 VCN, inst_idx, regUVD_GFX10_ADDR_CONFIG),
619 adev->gfx.config.gb_addr_config, 0, indirect);
618}620}
619 621
620/**622/**
@@ -1945,6 +1947,20 @@ static int vcn_v4_0_ring_patch_cs_in_place(struct amdgpu_cs_parser *p, · 1324 unmodified lines
1945 return 0;1947 return 0;
1946}1948}
1947 1949
1950static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1951{
1952 struct amdgpu_device *adev = ring->adev;
1953 struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1954
1955 if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1956 return -EOPNOTSUPP;
1957
1958 vcn_v4_0_stop(vinst);
1959 vcn_v4_0_start(vinst);
1960
1961 return amdgpu_ring_test_helper(ring);
1962}
1963
1948static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {1964static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
1949 .type = AMDGPU_RING_TYPE_VCN_ENC,1965 .type = AMDGPU_RING_TYPE_VCN_ENC,
1950 .align_mask = 0x3f,1966 .align_mask = 0x3f,
@@ -1974,6 +1990,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { · 23 unmodified lines
1974 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,1990 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
1975 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,1991 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
1976 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,1992 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1993 .reset = vcn_v4_0_ring_reset,
1977};1994};
1978 1995
1979/**1996/**
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c+62−22
@@ -45,6 +45,7 @@ · 44 unmodified lines
45 45
46#define VCN_VID_SOC_ADDRESS_2_0 0x1fb0046#define VCN_VID_SOC_ADDRESS_2_0 0x1fb00
47#define VCN1_VID_SOC_ADDRESS_3_0 0x4830047#define VCN1_VID_SOC_ADDRESS_3_0 0x48300
48#define VCN1_AON_SOC_ADDRESS_3_0 0x48000
48 49
49static const struct amdgpu_hwip_reg_entry vcn_reg_list_4_0_3[] = {50static const struct amdgpu_hwip_reg_entry vcn_reg_list_4_0_3[] = {
50 SOC15_REG_ENTRY_STR(VCN, 0, regUVD_POWER_STATUS),51 SOC15_REG_ENTRY_STR(VCN, 0, regUVD_POWER_STATUS),
@@ -287,6 +288,31 @@ static int vcn_v4_0_3_sw_fini(struct amdgpu_ip_block *ip_block) · 236 unmodified lines
287 return 0;288 return 0;
288}289}
289 290
291static int vcn_v4_0_3_hw_init_inst(struct amdgpu_vcn_inst *vinst)
292{
293 int vcn_inst;
294 struct amdgpu_device *adev = vinst->adev;
295 struct amdgpu_ring *ring;
296 int inst_idx = vinst->inst;
297
298 vcn_inst = GET_INST(VCN, inst_idx);
299 ring = &adev->vcn.inst[inst_idx].ring_enc[0];
300 if (ring->use_doorbell) {
301 adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell,
302 (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 9 * vcn_inst,
303 adev->vcn.inst[inst_idx].aid_id);
304
305 WREG32_SOC15(VCN, vcn_inst, regVCN_RB1_DB_CTRL,
306 ring->doorbell_index << VCN_RB1_DB_CTRL__OFFSET__SHIFT |
307 VCN_RB1_DB_CTRL__EN_MASK);
308
309 /* Read DB_CTRL to flush the write DB_CTRL command. */
310 RREG32_SOC15(VCN, vcn_inst, regVCN_RB1_DB_CTRL);
311 }
312
313 return 0;
314}
315
290/**316/**
291 * vcn_v4_0_3_hw_init - start and test VCN block317 * vcn_v4_0_3_hw_init - start and test VCN block
292 *318 *
@@ -298,7 +324,8 @@ static int vcn_v4_0_3_hw_init(struct amdgpu_ip_block *ip_block) · 5 unmodified lines
298{324{
299 struct amdgpu_device *adev = ip_block->adev;325 struct amdgpu_device *adev = ip_block->adev;
300 struct amdgpu_ring *ring;326 struct amdgpu_ring *ring;
301 int i, r, vcn_inst;327 struct amdgpu_vcn_inst *vinst;
328 int i, r;
302 329
303 if (amdgpu_sriov_vf(adev)) {330 if (amdgpu_sriov_vf(adev)) {
304 r = vcn_v4_0_3_start_sriov(adev);331 r = vcn_v4_0_3_start_sriov(adev);
@@ -321,28 +348,9 @@ static int vcn_v4_0_3_hw_init(struct amdgpu_ip_block *ip_block) · 16 unmodified lines
321 for (i = 0; i < adev->vcn.num_vcn_inst; ++i) {348 for (i = 0; i < adev->vcn.num_vcn_inst; ++i) {
322 struct amdgpu_vcn4_fw_shared *fw_shared;349 struct amdgpu_vcn4_fw_shared *fw_shared;
323 350
324 vcn_inst = GET_INST(VCN, i);
325 ring = &adev->vcn.inst[i].ring_enc[0];351 ring = &adev->vcn.inst[i].ring_enc[0];
326 352 vinst = &adev->vcn.inst[i];
327 if (ring->use_doorbell) {353 vcn_v4_0_3_hw_init_inst(vinst);
328 adev->nbio.funcs->vcn_doorbell_range(
329 adev, ring->use_doorbell,
330 (adev->doorbell_index.vcn.vcn_ring0_1 << 1) +
331 9 * vcn_inst,
332 adev->vcn.inst[i].aid_id);
333
334 WREG32_SOC15(
335 VCN, GET_INST(VCN, ring->me),
336 regVCN_RB1_DB_CTRL,
337 ring->doorbell_index
338 << VCN_RB1_DB_CTRL__OFFSET__SHIFT |
339 VCN_RB1_DB_CTRL__EN_MASK);
340
341 /* Read DB_CTRL to flush the write DB_CTRL command. */
342 RREG32_SOC15(
343 VCN, GET_INST(VCN, ring->me),
344 regVCN_RB1_DB_CTRL);
345 }
346 354
347 /* Re-init fw_shared when RAS fatal error occurred */355 /* Re-init fw_shared when RAS fatal error occurred */
348 fw_shared = adev->vcn.inst[i].fw_shared.cpu_addr;356 fw_shared = adev->vcn.inst[i].fw_shared.cpu_addr;
@@ -1563,6 +1571,37 @@ static void vcn_v4_0_3_unified_ring_set_wptr(struct amdgpu_ring *ring) · 1214 unmodified lines
1563 }1571 }
1564}1572}
1565 1573
1574static int vcn_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1575{
1576 int r = 0;
1577 int vcn_inst;
1578 struct amdgpu_device *adev = ring->adev;
1579 struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1580
1581 if (amdgpu_sriov_vf(ring->adev))
1582 return -EOPNOTSUPP;
1583
1584 if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1585 return -EOPNOTSUPP;
1586
1587 vcn_inst = GET_INST(VCN, ring->me);
1588 r = amdgpu_dpm_reset_vcn(adev, 1 << vcn_inst);
1589
1590 if (r) {
1591 DRM_DEV_ERROR(adev->dev, "VCN reset fail : %d\n", r);
1592 return r;
1593 }
1594
1595 /* This flag is not set for VF, assumed to be disabled always */
1596 if (RREG32_SOC15(VCN, GET_INST(VCN, 0), regVCN_RRMT_CNTL) & 0x100)
1597 adev->vcn.caps |= AMDGPU_VCN_CAPS(RRMT_ENABLED);
1598 vcn_v4_0_3_hw_init_inst(vinst);
1599 vcn_v4_0_3_start_dpg_mode(vinst, adev->vcn.inst[ring->me].indirect_sram);
1600 r = amdgpu_ring_test_helper(ring);
1601
1602 return r;
1603}
1604
1566static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = {1605static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = {
1567 .type = AMDGPU_RING_TYPE_VCN_ENC,1606 .type = AMDGPU_RING_TYPE_VCN_ENC,
1568 .align_mask = 0x3f,1607 .align_mask = 0x3f,
@@ -1591,6 +1630,7 @@ static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = { · 22 unmodified lines
1591 .emit_wreg = vcn_v4_0_3_enc_ring_emit_wreg,1630 .emit_wreg = vcn_v4_0_3_enc_ring_emit_wreg,
1592 .emit_reg_wait = vcn_v4_0_3_enc_ring_emit_reg_wait,1631 .emit_reg_wait = vcn_v4_0_3_enc_ring_emit_reg_wait,
1593 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,1632 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1633 .reset = vcn_v4_0_3_ring_reset,
1594};1634};
1595 1635
1596/**1636/**
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c+27−0
@@ -46,6 +46,7 @@ · 45 unmodified lines
46 46
47#define VCN_VID_SOC_ADDRESS_2_0 0x1fb0047#define VCN_VID_SOC_ADDRESS_2_0 0x1fb00
48#define VCN1_VID_SOC_ADDRESS_3_0 (0x48300 + 0x38000)48#define VCN1_VID_SOC_ADDRESS_3_0 (0x48300 + 0x38000)
49#define VCN1_AON_SOC_ADDRESS_3_0 (0x48000 + 0x38000)
49 50
50#define VCN_HARVEST_MMSCH 051#define VCN_HARVEST_MMSCH 0
51 52
@@ -207,6 +208,10 @@ static int vcn_v4_0_5_sw_init(struct amdgpu_ip_block *ip_block) · 155 unmodified lines
207 if (amdgpu_sriov_vf(adev))208 if (amdgpu_sriov_vf(adev))
208 fw_shared->present_flag_0 |= cpu_to_le32(AMDGPU_VCN_VF_RB_SETUP_FLAG);209 fw_shared->present_flag_0 |= cpu_to_le32(AMDGPU_VCN_VF_RB_SETUP_FLAG);
209 210
211 fw_shared->present_flag_0 |= AMDGPU_FW_SHARED_FLAG_0_DRM_KEY_INJECT;
212 fw_shared->drm_key_wa.method =
213 AMDGPU_DRM_KEY_INJECT_WORKAROUND_VCNFW_ASD_HANDSHAKING;
214
210 if (amdgpu_vcnfw_log)215 if (amdgpu_vcnfw_log)
211 amdgpu_vcn_fwlog_init(&adev->vcn.inst[i]);216 amdgpu_vcn_fwlog_init(&adev->vcn.inst[i]);
212 217
@@ -214,6 +219,13 @@ static int vcn_v4_0_5_sw_init(struct amdgpu_ip_block *ip_block) · 1 unmodified line
214 adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_5_pause_dpg_mode;219 adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_5_pause_dpg_mode;
215 }220 }
216 221
222 adev->vcn.supported_reset = amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);
223 adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
224
225 r = amdgpu_vcn_sysfs_reset_mask_init(adev);
226 if (r)
227 return r;
228
217 if (amdgpu_sriov_vf(adev)) {229 if (amdgpu_sriov_vf(adev)) {
218 r = amdgpu_virt_alloc_mm_table(adev);230 r = amdgpu_virt_alloc_mm_table(adev);
219 if (r)231 if (r)
@@ -1435,6 +1447,20 @@ static void vcn_v4_0_5_unified_ring_set_wptr(struct amdgpu_ring *ring) · 1215 unmodified lines
1435 }1447 }
1436}1448}
1437 1449
1450static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1451{
1452 struct amdgpu_device *adev = ring->adev;
1453 struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1454
1455 if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1456 return -EOPNOTSUPP;
1457
1458 vcn_v4_0_5_stop(vinst);
1459 vcn_v4_0_5_start(vinst);
1460
1461 return amdgpu_ring_test_helper(ring);
1462}
1463
1438static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = {1464static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = {
1439 .type = AMDGPU_RING_TYPE_VCN_ENC,1465 .type = AMDGPU_RING_TYPE_VCN_ENC,
1440 .align_mask = 0x3f,1466 .align_mask = 0x3f,
@@ -1462,6 +1488,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = { · 21 unmodified lines
1462 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,1488 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
1463 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,1489 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
1464 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,1490 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1491 .reset = vcn_v4_0_5_ring_reset,
1465};1492};
1466 1493
1467/**1494/**
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_0.c+18−2
@@ -196,9 +196,9 @@ static int vcn_v5_0_0_sw_init(struct amdgpu_ip_block *ip_block) · 195 unmodified lines
196 adev->vcn.inst[i].pause_dpg_mode = vcn_v5_0_0_pause_dpg_mode;196 adev->vcn.inst[i].pause_dpg_mode = vcn_v5_0_0_pause_dpg_mode;
197 }197 }
198 198
199 /* TODO: Add queue reset mask when FW fully supports it */
200 adev->vcn.supported_reset =199 adev->vcn.supported_reset =
201 amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);200 amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]);
201 adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE;
202 202
203 vcn_v5_0_0_alloc_ip_dump(adev);203 vcn_v5_0_0_alloc_ip_dump(adev);
204 204
@@ -533,7 +533,8 @@ static void vcn_v5_0_0_mc_resume_dpg_mode(struct amdgpu_vcn_inst *vinst, · 328 unmodified lines
533 533
534 /* VCN global tiling registers */534 /* VCN global tiling registers */
535 WREG32_SOC24_DPG_MODE(inst_idx, SOC24_DPG_MODE_OFFSET(535 WREG32_SOC24_DPG_MODE(inst_idx, SOC24_DPG_MODE_OFFSET(
536 VCN, 0, regUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect);536 VCN, inst_idx, regUVD_GFX10_ADDR_CONFIG),
537 adev->gfx.config.gb_addr_config, 0, indirect);
537 538
538 return;539 return;
539}540}
@@ -1171,6 +1172,20 @@ static void vcn_v5_0_0_unified_ring_set_wptr(struct amdgpu_ring *ring) · 631 unmodified lines
1171 }1172 }
1172}1173}
1173 1174
1175static int vcn_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid)
1176{
1177 struct amdgpu_device *adev = ring->adev;
1178 struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1179
1180 if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE))
1181 return -EOPNOTSUPP;
1182
1183 vcn_v5_0_0_stop(vinst);
1184 vcn_v5_0_0_start(vinst);
1185
1186 return amdgpu_ring_test_helper(ring);
1187}
1188
1174static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = {1189static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = {
1175 .type = AMDGPU_RING_TYPE_VCN_ENC,1190 .type = AMDGPU_RING_TYPE_VCN_ENC,
1176 .align_mask = 0x3f,1191 .align_mask = 0x3f,
@@ -1198,6 +1213,7 @@ static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = { · 21 unmodified lines
1198 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,1213 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
1199 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,1214 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
1200 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,1215 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
1216 .reset = vcn_v5_0_0_ring_reset,
1201};1217};
1202 1218
1203/**1219/**
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c+2−2
@@ -126,7 +126,7 @@ static int vcn_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 125 unmodified lines
126 126
127 ring = &adev->vcn.inst[i].ring_enc[0];127 ring = &adev->vcn.inst[i].ring_enc[0];
128 ring->use_doorbell = true;128 ring->use_doorbell = true;
129 ring->doorbell_index = (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 9 * vcn_inst;129 ring->doorbell_index = (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 11 * vcn_inst;
130 130
131 ring->vm_hub = AMDGPU_MMHUB0(adev->vcn.inst[i].aid_id);131 ring->vm_hub = AMDGPU_MMHUB0(adev->vcn.inst[i].aid_id);
132 sprintf(ring->name, "vcn_unified_%d", adev->vcn.inst[i].aid_id);132 sprintf(ring->name, "vcn_unified_%d", adev->vcn.inst[i].aid_id);
@@ -213,7 +213,7 @@ static int vcn_v5_0_1_hw_init(struct amdgpu_ip_block *ip_block) · 80 unmodified lines
213 if (ring->use_doorbell)213 if (ring->use_doorbell)
214 adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell,214 adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell,
215 ((adev->doorbell_index.vcn.vcn_ring0_1 << 1) +215 ((adev->doorbell_index.vcn.vcn_ring0_1 << 1) +
216 9 * vcn_inst),216 11 * vcn_inst),
217 adev->vcn.inst[i].aid_id);217 adev->vcn.inst[i].aid_id);
218 218
219 /* Re-init fw_shared, if required */219 /* Re-init fw_shared, if required */
drivers/gpu/drm/amd/amdgpu/vega20_ih.c+5−1
@@ -350,6 +350,7 @@ static int vega20_ih_irq_init(struct amdgpu_device *adev) · 349 unmodified lines
350 if (ret)350 if (ret)
351 return ret;351 return ret;
352 }352 }
353 ih[i]->overflow = false;
353 }354 }
354 355
355 if (!amdgpu_sriov_vf(adev))356 if (!amdgpu_sriov_vf(adev))
@@ -437,7 +438,10 @@ static u32 vega20_ih_get_wptr(struct amdgpu_device *adev, · 81 unmodified lines
437 if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW))438 if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW))
438 goto out;439 goto out;
439 440
440 wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);441 if (!amdgpu_sriov_vf(adev))
442 wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0);
443 else
444 ih->overflow = true;
441 445
442 /* When a ring buffer overflow happen start parsing interrupt446 /* When a ring buffer overflow happen start parsing interrupt
443 * from the last not overwritten vector (wptr + 32). Hopefully447 * from the last not overwritten vector (wptr + 32). Hopefully
drivers/gpu/drm/amd/amdkfd/kfd_debugfs.c+76−0
@@ -27,6 +27,16 @@ · 26 unmodified lines
27#include "kfd_priv.h"27#include "kfd_priv.h"
28 28
29static struct dentry *debugfs_root;29static struct dentry *debugfs_root;
30static struct dentry *debugfs_proc;
31static struct list_head procs;
32
33struct debugfs_proc_entry {
34 struct list_head list;
35 struct dentry *proc_dentry;
36 pid_t pid;
37};
38
39#define MAX_DEBUGFS_FILENAME_LEN 32
30 40
31static int kfd_debugfs_open(struct inode *inode, struct file *file)41static int kfd_debugfs_open(struct inode *inode, struct file *file)
32{42{
@@ -92,6 +102,8 @@ static const struct file_operations kfd_debugfs_hang_hws_fops = { · 59 unmodified lines
92void kfd_debugfs_init(void)102void kfd_debugfs_init(void)
93{103{
94 debugfs_root = debugfs_create_dir("kfd", NULL);104 debugfs_root = debugfs_create_dir("kfd", NULL);
105 debugfs_proc = debugfs_create_dir("proc", debugfs_root);
106 INIT_LIST_HEAD(&procs);
95 107
96 debugfs_create_file("mqds", S_IFREG | 0444, debugfs_root,108 debugfs_create_file("mqds", S_IFREG | 0444, debugfs_root,
97 kfd_debugfs_mqds_by_process, &kfd_debugfs_fops);109 kfd_debugfs_mqds_by_process, &kfd_debugfs_fops);
@@ -107,5 +119,69 @@ void kfd_debugfs_init(void) · 9 unmodified lines
107 119
108void kfd_debugfs_fini(void)120void kfd_debugfs_fini(void)
109{121{
122 debugfs_remove_recursive(debugfs_proc);
110 debugfs_remove_recursive(debugfs_root);123 debugfs_remove_recursive(debugfs_root);
111}124}
125
126static ssize_t kfd_debugfs_pasid_read(struct file *file, char __user *buf,
127 size_t count, loff_t *ppos)
128{
129 struct kfd_process_device *pdd = file_inode(file)->i_private;
130 char tmp[32];
131 int len;
132
133 len = snprintf(tmp, sizeof(tmp), "%u\n", pdd->pasid);
134
135 return simple_read_from_buffer(buf, count, ppos, tmp, len);
136}
137
138static const struct file_operations kfd_debugfs_pasid_fops = {
139 .owner = THIS_MODULE,
140 .read = kfd_debugfs_pasid_read,
141};
142
143void kfd_debugfs_add_process(struct kfd_process *p)
144{
145 int i;
146 char name[MAX_DEBUGFS_FILENAME_LEN];
147 struct debugfs_proc_entry *entry;
148
149 entry = kzalloc(sizeof(*entry), GFP_KERNEL);
150 if (!entry)
151 return;
152
153 list_add(&entry->list, &procs);
154 entry->pid = p->lead_thread->pid;
155 snprintf(name, MAX_DEBUGFS_FILENAME_LEN, "%d",
156 (int)entry->pid);
157 entry->proc_dentry = debugfs_create_dir(name, debugfs_proc);
158
159 /* Create debugfs files for each GPU:
160 * - proc/<pid>/pasid_<gpuid>
161 */
162 for (i = 0; i < p->n_pdds; i++) {
163 struct kfd_process_device *pdd = p->pdds[i];
164
165 snprintf(name, MAX_DEBUGFS_FILENAME_LEN, "pasid_%u",
166 pdd->dev->id);
167 debugfs_create_file((const char *)name, S_IFREG | 0444,
168 entry->proc_dentry, pdd,
169 &kfd_debugfs_pasid_fops);
170 }
171}
172
173void kfd_debugfs_remove_process(struct kfd_process *p)
174{
175 struct debugfs_proc_entry *entry, *next;
176
177 mutex_lock(&kfd_processes_mutex);
178 list_for_each_entry_safe(entry, next, &procs, list) {
179 if (entry->pid != p->lead_thread->pid)
180 continue;
181
182 debugfs_remove_recursive(entry->proc_dentry);
183 list_del(&entry->list);
184 kfree(entry);
185 }
186 mutex_unlock(&kfd_processes_mutex);
187}
drivers/gpu/drm/amd/amdkfd/kfd_device.c+0−5
@@ -352,11 +352,6 @@ struct kfd_dev *kgd2kfd_probe(struct amdgpu_device *adev, bool vf) · 351 unmodified lines
352 f2g = &aldebaran_kfd2kgd;352 f2g = &aldebaran_kfd2kgd;
353 break;353 break;
354 case IP_VERSION(9, 4, 3):354 case IP_VERSION(9, 4, 3):
355 gfx_target_version = adev->rev_id >= 1 ? 90402
356 : adev->flags & AMD_IS_APU ? 90400
357 : 90401;
358 f2g = &gc_9_4_3_kfd2kgd;
359 break;
360 case IP_VERSION(9, 4, 4):355 case IP_VERSION(9, 4, 4):
361 gfx_target_version = 90402;356 gfx_target_version = 90402;
362 f2g = &gc_9_4_3_kfd2kgd;357 f2g = &gc_9_4_3_kfd2kgd;
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c+8−6
@@ -1576,8 +1576,9 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 1575 unmodified lines
1576 int bit;1576 int bit;
1577 1577
1578 if (q->properties.type == KFD_QUEUE_TYPE_SDMA) {1578 if (q->properties.type == KFD_QUEUE_TYPE_SDMA) {
1579 if (bitmap_empty(dqm->sdma_bitmap, KFD_MAX_SDMA_QUEUES)) {1579 if (bitmap_empty(dqm->sdma_bitmap, get_num_sdma_queues(dqm))) {
1580 dev_err(dev, "No more SDMA queue to allocate\n");1580 dev_warn(dev, "No more SDMA queue to allocate (%d total queues)\n",
1581 get_num_sdma_queues(dqm));
1581 return -ENOMEM;1582 return -ENOMEM;
1582 }1583 }
1583 1584
@@ -1602,8 +1603,9 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 18 unmodified lines
1602 q->properties.sdma_queue_id = q->sdma_id /1603 q->properties.sdma_queue_id = q->sdma_id /
1603 kfd_get_num_sdma_engines(dqm->dev);1604 kfd_get_num_sdma_engines(dqm->dev);
1604 } else if (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI) {1605 } else if (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI) {
1605 if (bitmap_empty(dqm->xgmi_sdma_bitmap, KFD_MAX_SDMA_QUEUES)) {1606 if (bitmap_empty(dqm->xgmi_sdma_bitmap, get_num_xgmi_sdma_queues(dqm))) {
1606 dev_err(dev, "No more XGMI SDMA queue to allocate\n");1607 dev_warn(dev, "No more XGMI SDMA queue to allocate (%d total queues)\n",
1608 get_num_xgmi_sdma_queues(dqm));
1607 return -ENOMEM;1609 return -ENOMEM;
1608 }1610 }
1609 if (restore_sdma_id) {1611 if (restore_sdma_id) {
@@ -1662,8 +1664,8 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 52 unmodified lines
1662 }1664 }
1663 1665
1664 if (!free_bit_found) {1666 if (!free_bit_found) {
1665 dev_err(dev, "No more SDMA queue to allocate for target ID %i\n",1667 dev_warn(dev, "No more SDMA queue to allocate for target ID %i (%d total queues)\n",
1666 q->properties.sdma_engine_id);1668 q->properties.sdma_engine_id, num_queues);
1667 return -ENOMEM;1669 return -ENOMEM;
1668 }1670 }
1669 }1671 }
drivers/gpu/drm/amd/amdkfd/kfd_events.c+19−0
@@ -1177,6 +1177,25 @@ void kfd_signal_hw_exception_event(u32 pasid) · 1176 unmodified lines
1177 kfd_unref_process(p);1177 kfd_unref_process(p);
1178}1178}
1179 1179
1180void kfd_signal_vm_fault_event_with_userptr(struct kfd_process *p, uint64_t gpu_va)
1181{
1182 struct kfd_process_device *pdd;
1183 struct kfd_hsa_memory_exception_data exception_data;
1184 int i;
1185
1186 memset(&exception_data, 0, sizeof(exception_data));
1187 exception_data.va = gpu_va;
1188 exception_data.failure.NotPresent = 1;
1189
1190 // Send VM seg fault to all kfd process device
1191 for (i = 0; i < p->n_pdds; i++) {
1192 pdd = p->pdds[i];
1193 exception_data.gpu_id = pdd->user_gpu_id;
1194 kfd_evict_process_device(pdd);
1195 kfd_signal_vm_fault_event(pdd, NULL, &exception_data);
1196 }
1197}
1198
1180void kfd_signal_vm_fault_event(struct kfd_process_device *pdd,1199void kfd_signal_vm_fault_event(struct kfd_process_device *pdd,
1181 struct kfd_vm_fault_info *info,1200 struct kfd_vm_fault_info *info,
1182 struct kfd_hsa_memory_exception_data *data)1201 struct kfd_hsa_memory_exception_data *data)
drivers/gpu/drm/amd/amdkfd/kfd_int_process_v10.c+94−48
@@ -168,11 +168,12 @@ static bool event_interrupt_isr_v10(struct kfd_node *dev, · 167 unmodified lines
168 client_id != SOC15_IH_CLIENTID_SE3SH)168 client_id != SOC15_IH_CLIENTID_SE3SH)
169 return false;169 return false;
170 170
171 pr_debug("client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw data:\n",171 dev_dbg(dev->adev->dev,
172 client_id, source_id, vmid, pasid);172 "client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw data:\n",
173 pr_debug("%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n",173 client_id, source_id, vmid, pasid);
174 data[0], data[1], data[2], data[3],174 dev_dbg(dev->adev->dev, "%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n",
175 data[4], data[5], data[6], data[7]);175 data[0], data[1], data[2], data[3], data[4], data[5], data[6],
176 data[7]);
176 177
177 /* If there is no valid PASID, it's likely a bug */178 /* If there is no valid PASID, it's likely a bug */
178 if (WARN_ONCE(pasid == 0, "Bug: No PASID in KFD interrupt"))179 if (WARN_ONCE(pasid == 0, "Bug: No PASID in KFD interrupt"))
@@ -217,37 +218,66 @@ static void event_interrupt_wq_v10(struct kfd_node *dev, · 38 unmodified lines
217 SQ_INTERRUPT_WORD_WAVE_CTXID1, ENCODING);218 SQ_INTERRUPT_WORD_WAVE_CTXID1, ENCODING);
218 switch (encoding) {219 switch (encoding) {
219 case SQ_INTERRUPT_WORD_ENCODING_AUTO:220 case SQ_INTERRUPT_WORD_ENCODING_AUTO:
220 pr_debug_ratelimited(221 dev_dbg_ratelimited(
222 dev->adev->dev,
221 "sq_intr: auto, se %d, ttrace %d, wlt %d, ttrac_buf0_full %d, ttrac_buf1_full %d, ttrace_utc_err %d\n",223 "sq_intr: auto, se %d, ttrace %d, wlt %d, ttrac_buf0_full %d, ttrac_buf1_full %d, ttrace_utc_err %d\n",
222 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_AUTO_CTXID1,224 REG_GET_FIELD(
223 SE_ID),225 context_id1,
224 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,226 SQ_INTERRUPT_WORD_AUTO_CTXID1,
225 THREAD_TRACE),227 SE_ID),
226 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,228 REG_GET_FIELD(
227 WLT),229 context_id0,
228 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,230 SQ_INTERRUPT_WORD_AUTO_CTXID0,
229 THREAD_TRACE_BUF0_FULL),231 THREAD_TRACE),
230 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,232 REG_GET_FIELD(
231 THREAD_TRACE_BUF1_FULL),233 context_id0,
232 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,234 SQ_INTERRUPT_WORD_AUTO_CTXID0,
233 THREAD_TRACE_UTC_ERROR));235 WLT),
236 REG_GET_FIELD(
237 context_id0,
238 SQ_INTERRUPT_WORD_AUTO_CTXID0,
239 THREAD_TRACE_BUF0_FULL),
240 REG_GET_FIELD(
241 context_id0,
242 SQ_INTERRUPT_WORD_AUTO_CTXID0,
243 THREAD_TRACE_BUF1_FULL),
244 REG_GET_FIELD(
245 context_id0,
246 SQ_INTERRUPT_WORD_AUTO_CTXID0,
247 THREAD_TRACE_UTC_ERROR));
234 break;248 break;
235 case SQ_INTERRUPT_WORD_ENCODING_INST:249 case SQ_INTERRUPT_WORD_ENCODING_INST:
236 pr_debug_ratelimited("sq_intr: inst, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",250 dev_dbg_ratelimited(
237 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,251 dev->adev->dev,
238 SE_ID),252 "sq_intr: inst, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",
239 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,253 REG_GET_FIELD(
240 DATA),254 context_id1,
241 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,255 SQ_INTERRUPT_WORD_WAVE_CTXID1,
242 SA_ID),256 SE_ID),
243 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,257 REG_GET_FIELD(
244 PRIV),258 context_id0,
245 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,259 SQ_INTERRUPT_WORD_WAVE_CTXID0,
246 WAVE_ID),260 DATA),
247 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,261 REG_GET_FIELD(
248 SIMD_ID),262 context_id0,
249 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,263 SQ_INTERRUPT_WORD_WAVE_CTXID0,
250 WGP_ID));264 SA_ID),
265 REG_GET_FIELD(
266 context_id0,
267 SQ_INTERRUPT_WORD_WAVE_CTXID0,
268 PRIV),
269 REG_GET_FIELD(
270 context_id0,
271 SQ_INTERRUPT_WORD_WAVE_CTXID0,
272 WAVE_ID),
273 REG_GET_FIELD(
274 context_id0,
275 SQ_INTERRUPT_WORD_WAVE_CTXID0,
276 SIMD_ID),
277 REG_GET_FIELD(
278 context_id1,
279 SQ_INTERRUPT_WORD_WAVE_CTXID1,
280 WGP_ID));
251 if (context_id0 & SQ_INTERRUPT_WORD_WAVE_CTXID0__PRIV_MASK) {281 if (context_id0 & SQ_INTERRUPT_WORD_WAVE_CTXID0__PRIV_MASK) {
252 if (kfd_set_dbg_ev_from_interrupt(dev, pasid,282 if (kfd_set_dbg_ev_from_interrupt(dev, pasid,
253 KFD_DEBUG_DOORBELL_ID(context_id0),283 KFD_DEBUG_DOORBELL_ID(context_id0),
@@ -259,21 +289,37 @@ static void event_interrupt_wq_v10(struct kfd_node *dev, · 5 unmodified lines
259 case SQ_INTERRUPT_WORD_ENCODING_ERROR:289 case SQ_INTERRUPT_WORD_ENCODING_ERROR:
260 sq_intr_err_type = REG_GET_FIELD(context_id0, KFD_CTXID0,290 sq_intr_err_type = REG_GET_FIELD(context_id0, KFD_CTXID0,
261 ERR_TYPE);291 ERR_TYPE);
262 pr_warn_ratelimited("sq_intr: error, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d, err_type %d\n",292 dev_warn_ratelimited(
263 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,293 dev->adev->dev,
264 SE_ID),294 "sq_intr: error, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d, err_type %d\n",
265 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,295 REG_GET_FIELD(
266 DATA),296 context_id1,
267 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,297 SQ_INTERRUPT_WORD_WAVE_CTXID1,
268 SA_ID),298 SE_ID),
269 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,299 REG_GET_FIELD(
270 PRIV),300 context_id0,
271 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,301 SQ_INTERRUPT_WORD_WAVE_CTXID0,
272 WAVE_ID),302 DATA),
273 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,303 REG_GET_FIELD(
274 SIMD_ID),304 context_id0,
275 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,305 SQ_INTERRUPT_WORD_WAVE_CTXID0,
276 WGP_ID),306 SA_ID),
307 REG_GET_FIELD(
308 context_id0,
309 SQ_INTERRUPT_WORD_WAVE_CTXID0,
310 PRIV),
311 REG_GET_FIELD(
312 context_id0,
313 SQ_INTERRUPT_WORD_WAVE_CTXID0,
314 WAVE_ID),
315 REG_GET_FIELD(
316 context_id0,
317 SQ_INTERRUPT_WORD_WAVE_CTXID0,
318 SIMD_ID),
319 REG_GET_FIELD(
320 context_id1,
321 SQ_INTERRUPT_WORD_WAVE_CTXID1,
322 WGP_ID),
277 sq_intr_err_type);323 sq_intr_err_type);
278 break;324 break;
279 default:325 default:
drivers/gpu/drm/amd/amdkfd/kfd_int_process_v11.c+59−33
@@ -148,44 +148,69 @@ enum SQ_INTERRUPT_ERROR_TYPE { · 147 unmodified lines
148#define KFD_CTXID0_DOORBELL_ID(ctxid0) ((ctxid0) & \148#define KFD_CTXID0_DOORBELL_ID(ctxid0) ((ctxid0) & \
149 KFD_CTXID0_DOORBELL_ID_MASK)149 KFD_CTXID0_DOORBELL_ID_MASK)
150 150
151static void print_sq_intr_info_auto(uint32_t context_id0, uint32_t context_id1)151static void print_sq_intr_info_auto(struct kfd_node *dev, uint32_t context_id0,
152 uint32_t context_id1)
152{153{
153 pr_debug_ratelimited(154 dev_dbg_ratelimited(
155 dev->adev->dev,
154 "sq_intr: auto, ttrace %d, wlt %d, ttrace_buf_full %d, reg_tms %d, cmd_tms %d, host_cmd_ovf %d, host_reg_ovf %d, immed_ovf %d, ttrace_utc_err %d\n",156 "sq_intr: auto, ttrace %d, wlt %d, ttrace_buf_full %d, reg_tms %d, cmd_tms %d, host_cmd_ovf %d, host_reg_ovf %d, immed_ovf %d, ttrace_utc_err %d\n",
155 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE),157 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
158 THREAD_TRACE),
156 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, WLT),159 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, WLT),
157 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE_BUF_FULL),160 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
158 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, REG_TIMESTAMP),161 THREAD_TRACE_BUF_FULL),
159 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, CMD_TIMESTAMP),162 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
160 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, HOST_CMD_OVERFLOW),163 REG_TIMESTAMP),
161 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, HOST_REG_OVERFLOW),164 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
162 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, IMMED_OVERFLOW),165 CMD_TIMESTAMP),
163 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE_UTC_ERROR));166 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
167 HOST_CMD_OVERFLOW),
168 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
169 HOST_REG_OVERFLOW),
170 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
171 IMMED_OVERFLOW),
172 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0,
173 THREAD_TRACE_UTC_ERROR));
164}174}
165 175
166static void print_sq_intr_info_inst(uint32_t context_id0, uint32_t context_id1)176static void print_sq_intr_info_inst(struct kfd_node *dev, uint32_t context_id0,
177 uint32_t context_id1)
167{178{
168 pr_debug_ratelimited(179 dev_dbg_ratelimited(
180 dev->adev->dev,
169 "sq_intr: inst, data 0x%08x, sh %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",181 "sq_intr: inst, data 0x%08x, sh %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n",
170 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, DATA),182 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, DATA),
171 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, SH_ID),183 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
184 SH_ID),
172 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, PRIV),185 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, PRIV),
173 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, WAVE_ID),186 REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0,
174 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, SIMD_ID),187 WAVE_ID),
175 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, WGP_ID));188 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
189 SIMD_ID),
190 REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1,
191 WGP_ID));
176}192}
177 193

72 more lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_int_process_v9.c+110−32

This diff is large: its 166 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_packet_manager_v9.c+6−5

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_priv.h+7−0

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_process.c+7−0

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c+9−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_queue.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c+22−2

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_smi_events.h+1−0

This diff is large: its 6 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_svm.c+1−2

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/amdkfd/kfd_topology.c+24−17

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c+375−214

This diff is large: its 1707 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h+24−1

This diff is large: its 60 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c+49−7

This diff is large: its 84 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c+13−0

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c+33−12

This diff is large: its 80 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c+2−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.c+0−8

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.c+2−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/Makefile+20−21

This diff is large: its 52 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/basics/fixpt31_32.c+0−5

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/bios/bios_parser.c+2−2

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.c+0−14

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn10/rv1_clk_mgr_vbios_smu.h+0−1

This diff is large: its 7 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.c+0−14

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn21/rn_clk_mgr_vbios_smu.h+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn315/dcn315_smu.c+14−15

This diff is large: its 57 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn351_clk_mgr.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_clk_mgr.c+7−1

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn35/dcn35_smu.c+3−3

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc.c+131−11

This diff is large: its 254 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_debug.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_resource.c+6−42

This diff is large: its 76 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_state.c+107−4

This diff is large: its 141 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_stream.c+63−15

This diff is large: its 113 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc.h+182−165

This diff is large: its 421 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c+109−93

This diff is large: its 333 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_fused_io.c+148−0

This diff is large: its 149 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_fused_io.h+31−0

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_helper.c+7−1

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_state_priv.h+20−0

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_stream.h+11−0

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_types.h+15−2

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dccg/dcn35/dcn35_dccg.c+37−25

This diff is large: its 208 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c+5−4

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c+2−1

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dmub_replay.c+13−6

This diff is large: its 61 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce60/Makefile+1−2

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce80/dce80_timing_generator.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dm_helpers.h+8−0

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn30/display_mode_vba_30.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn31/display_mode_vba_31.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn314/display_mode_vba_314.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/Makefile+0−3

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_translation_helper.c+9−3

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/dml21_wrapper.c+4−4

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top.h+1−0

This diff is large: its 6 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_dchub_registers.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/inc/dml_top_display_cfg_types.h+2−1

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4.c+5−5

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_dcn4_calcs.c+2138−2208

This diff is large: its 6570 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_shared_types.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_core/dml2_core_utils.c+71−71

This diff is large: its 285 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_dpmm/dml2_dpmm_dcn4.c+4−0

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_mcg/dml2_mcg_dcn4.c+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_pmo/dml2_pmo_dcn4_fams2.c+4−4

This diff is large: its 36 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/dml2_top/dml2_top_soc15.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.c+0−31

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_debug.h+42−35

This diff is large: its 98 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml21/src/inc/dml2_internal_shared_types.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml2_dc_resource_mgmt.c+12−12

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml2_translation_helper.c+6−10

This diff is large: its 51 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2/dml2_wrapper.c+4−1

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dpp/dcn30/dcn30_dpp.c+3−6

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dpp/dcn35/dcn35_dpp.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.c+0−7

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dsc/dcn20/dcn20_dsc.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dsc/dcn401/dcn401_dsc.c+0−6

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/gpio_service.c+14−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hubbub/dcn21/dcn21_hubbub.c+3−3

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/Makefile+18−8

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce110/dce110_hwseq.c+4−4

This diff is large: its 33 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.crenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_hw_sequencer.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.hrenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_hw_sequencer.h+0−0

File renamed without changes.

drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.c+122−1

This diff is large: its 137 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.h+7−0

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn20/dcn20_hwseq.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn314/dcn314_hwseq.c+14−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn32/dcn32_hwseq.c+3−1

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c+14−7

This diff is large: its 68 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_init.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn351/dcn351_init.c+3−1

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c+317−37

This diff is large: its 444 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h+8−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h+6−2

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer_private.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/core_status.h+3−0

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/core_types.h+10−4

This diff is large: its 49 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h+2−1

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/mpc.h+29−17

This diff is large: its 60 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/optc.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/reg_helper.h+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/resource.h+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/irq/dcn32/irq_service_dcn32.c+46−15

This diff is large: its 104 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/irq/dcn401/irq_service_dcn401.c+46−14

This diff is large: its 103 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/irq_types.h+16−0

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/link_factory.c+5−1

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_dpia.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c+4−3

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c+7−4

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn32/dcn32_mpc.c+192−188

This diff is large: its 468 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.c+0−11

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn401/dcn401_mpc.h+0−14

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn35/dcn35_optc.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/Makefile+18−8

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce100/dce100_resource.c+3−3

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce110/dce110_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce112/dce112_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.crenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c+3−3

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dce60/dce60_resource.hrenamed from drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.h+0−0

File renamed without changes.

drivers/gpu/drm/amd/display/dc/resource/dce80/dce80_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn10/dcn10_resource.c+7−2

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.c+3−3

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn20/dcn20_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn21/dcn21_resource.c+3−3

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.c+2−4

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c+59−13

This diff is large: its 132 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.h+5−1

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn321/dcn321_resource.c+3−1

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn35/dcn35_resource.c+4−4

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn351/dcn351_resource.c+3−3

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn36/dcn36_resource.c+4−4

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.c+45−7

This diff is large: its 79 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn401/dcn401_resource.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/sspl/dc_spl.c+14−11

This diff is large: its 80 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/sspl/spl_fixpt31_32.c+0−4

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/dmub_srv.h+110−47

This diff is large: its 283 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h+73−2

This diff is large: its 138 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.c+83−38

This diff is large: its 155 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_dcn401.h+3−1

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_srv.c+247−98

This diff is large: its 484 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/src/dmub_srv_stat.c+4−17

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/display/include/gpio_service_interface.h+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_execution.c+43−10

This diff is large: its 72 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp2_transition.c+36−12

This diff is large: its 81 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/hdcp/hdcp_ddc.c+73−0

This diff is large: its 77 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/inc/mod_hdcp.h+36−2

This diff is large: its 68 lines aren’t shown here.

drivers/gpu/drm/amd/include/amd_shared.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/dce/dce_6_0_sh_mask.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/gca/gfx_6_0_d.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_d.h+20−3

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/oss/oss_1_0_sh_mask.h+41−0

This diff is large: its 55 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_d.h+44−0

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/smu/smu_6_0_sh_mask.h+185−3

This diff is large: its 235 lines aren’t shown here.

drivers/gpu/drm/amd/include/atombios.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/include/kgd_pp_interface.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/include/v11_structs.h+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/include/v12_structs.h+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/amdgpu_dpm.c+43−0

This diff is large: its 57 lines aren’t shown here.

drivers/gpu/drm/amd/pm/amdgpu_pm.c+34−0

This diff is large: its 69 lines aren’t shown here.

drivers/gpu/drm/amd/pm/inc/amdgpu_dpm.h+3−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c+190−168

This diff is large: its 806 lines aren’t shown here.

drivers/gpu/drm/amd/pm/legacy-dpm/si_smc.c+24−18

This diff is large: its 127 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c+8−0

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.c+0−36

This diff is large: its 43 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/smumgr/smu7_smumgr.h+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c+34−0

This diff is large: its 52 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h+26−3

This diff is large: its 77 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_pmfw.h+7−0

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/pmfw_if/smu_v13_0_6_ppsmc.h+3−1

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_types.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v11_0.h+0−5

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v13_0.h+2−12

This diff is large: its 42 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/smu_v11_0.c+0−39

This diff is large: its 46 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c+11−2

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c+10−101

This diff is large: its 136 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_12_ppt.c+14−24

This diff is large: its 73 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c+126−29

This diff is large: its 319 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.h+21−0

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/drm_file.c+34−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/radeon/atombios.h+1−2

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/radeon/r600_hdmi.c+0−22

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon.h+0−3

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon_asic.h+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon_cs.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/radeon/radeon_fence.c+0−42

This diff is large: its 49 lines aren’t shown here.

drivers/gpu/drm/radeon/sid.h+1−1

This diff is large: its 9 lines aren’t shown here.

include/drm/drm_file.h+3−0

This diff is large: its 10 lines aren’t shown here.

include/linux/dma-fence-unwrap.h+2−0

This diff is large: its 9 lines aren’t shown here.

include/uapi/drm/amdgpu_drm.h+316−0

This diff is large: its 372 lines aren’t shown here.

include/uapi/linux/kfd_ioctl.h+5−0

This diff is large: its 19 lines aren’t shown here.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t