History

Merge tag 'amd-drm-next-7.3-2026-07-02' of https://gitlab.freedesktop.org/agd5f/linux into drm-next

[airlied: had to reapply drm/amdgpu: Implement "color format" DRM property by hand to amdgpu_dm_connector.c] amd-drm-next-7.3-2026-07-02: amdgpu: - Queue reset updates - Initial compute pipe reset support - Improved boundary checking for bios parsing - Cleaned up sysfs input parsing - devcoredump fixes - RAS updates and rework - VCN secure submission fixes - 8K panel fix - Add display KUnit tests - Display CRC fixes - UserQ updates - Backlight fixes - Parse panel type info from DisplayID - Align IP discovery to pci device lifetime - IOCTL boundary check fixes - Convert amdgpu_vm_lock_by_pasid() to drm_exec - Ctx fixes and cleanup - SOC15 register macro cleanups - Memory placement fixes for UVD - Disable KQ support for MI3xx - GFX9 mode2 reset fix - BO list cleanup - Soc24 aborted suspend fix - Gfx8 soft reset rework - Enable soft reset on gfx8 - Drop unnecessary BUG() and BUG_ON() in error paths - Fix power reporting unit conversion - Improve vbios command table bounds checking - UVD bounds checking improvements - VCN bounds checking improvements - PSR and replay fixes - DCN 4.2 updates - Colorop updates - DC GPIO rework - ACP fixes - Fix aperture mapping leak - Ignore_damage_clips fix - Fixes for non-4K pages - JPEG idle check fixes - Userptr fixes - GPUVM fixes - GC 11.7 updates - SMU 13 fixes amdkfd: - Initial compute pipe reset support - Allow applications to opt out of sigbus on fatal errors - Fix doorbell/mmio BO cleanup - Improved CRIU boundary checking - MQD handling rework - SMI fixes - Reset event fixes - CRIU fixes - Sysfs teardown fixes - IOCTL boundary check fixes - SVM fixes - Soft IH ring fixes - Move TBA/TMA from system to device memory radeon: - Blit fix for large BOs - r600 dpm cleanup fix drm: - Extract EDID base section header processing into helper - Parse panel type from DisplayID 2.x Display Parameters UAPI: - KFD interface for applications to select sigbus behavior on fatal errors Proposed userspace: https://github.com/ROCm/rocm-systems/pull/6190

Dave Airlie committed 4 months ago

Commit 0461ba92 parents 0639cb2 50be7c9376 files · +35568 −23804Browse files

Unified · Split

376 files
  1. drivers/gpu/drm/amd/amdgpu/Makefile +1 −1
  2. drivers/gpu/drm/amd/amdgpu/amdgpu.h +10 −42
  3. drivers/gpu/drm/amd/amdgpu/amdgpu_aca.c +0 −985
  4. drivers/gpu/drm/amd/amdgpu/amdgpu_aca.h +0 −232
  5. drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c +34 −34
  6. drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +14 −0
  7. drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +24 −4
  8. drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gc_9_4_3.c +62 −1
  9. drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +14 −6
  10. drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c +1 −1
  11. drivers/gpu/drm/amd/amdgpu/amdgpu_atpx_handler.c +21 −12
  12. drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c +8 −2
  13. drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c +68 −102
  14. drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.h +8 −14
  15. drivers/gpu/drm/amd/amdgpu/amdgpu_cper.c +2 −114
  16. drivers/gpu/drm/amd/amdgpu/amdgpu_cper.h +0 −8
  17. drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +28 −39
  18. drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +63 −148
  19. drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h +10 −4
  20. drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +7 −0
  21. drivers/gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c +1 −0
  22. drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14 −174
  23. drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +270 −40
  24. drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.h +6 −0
  25. drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +23 −4
  26. drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +52 −1
  27. drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c +278 −6
  28. drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +22 −2
  29. drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +7 −2
  30. drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h +0 −1
  31. drivers/gpu/drm/amd/amdgpu/amdgpu_ip.c +126 −17
  32. drivers/gpu/drm/amd/amdgpu/amdgpu_ip.h +6 −2
  33. drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +12 −1
  34. drivers/gpu/drm/amd/amdgpu/amdgpu_jpeg.c +5 −4
  35. drivers/gpu/drm/amd/amdgpu/amdgpu_jpeg.h +3 −0
  36. drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +31 −7
  37. drivers/gpu/drm/amd/amdgpu/amdgpu_mca.c +0 −486
  38. drivers/gpu/drm/amd/amdgpu/amdgpu_mca.h +0 −107
  39. drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c +67 −3
  40. drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h +15 −0
  41. drivers/gpu/drm/amd/amdgpu/amdgpu_mmhub.h +0 −23
  42. drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +0 −40
  43. drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c +11 −3
  44. drivers/gpu/drm/amd/amdgpu/amdgpu_psp_ta.c +4 −1
  45. drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +47 −815
  46. drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +3 −37
  47. drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +119 −249
  48. drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +0 −3
  49. drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +188 −0
  50. drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +8 −0
  51. drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.c +39 −0
  52. drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h +10 −0
  53. drivers/gpu/drm/amd/amdgpu/amdgpu_sa.h +77 −0
  54. drivers/gpu/drm/amd/amdgpu/amdgpu_sched.c +4 −4
  55. drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c +15 −4
  56. drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h +0 −28
  57. drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h +150 −0
  58. drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 −1
  59. drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h +1 −0
  60. drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c +60 −209
  61. drivers/gpu/drm/amd/amdgpu/amdgpu_umc.h +0 −26
  62. drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +108 −57
  63. drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h +14 −2
  64. drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c +7 −3
  65. drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c +45 −31
  66. drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h +3 −0
  67. drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c +31 −11
  68. drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +15 −284
  69. drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.c +45 −18
  70. drivers/gpu/drm/amd/amdgpu/atom.c +48 −18
  71. drivers/gpu/drm/amd/amdgpu/atom.h +2 −1
  72. drivers/gpu/drm/amd/amdgpu/cik.c +0 −7
  73. drivers/gpu/drm/amd/amdgpu/dce_v10_0.c +0 −66
  74. drivers/gpu/drm/amd/amdgpu/dce_v6_0.c +0 −57
  75. drivers/gpu/drm/amd/amdgpu/dce_v8_0.c +0 −57
  76. drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +2 −0
  77. drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +43 −291
  78. drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c +40 −211
  79. drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c +4 −0
  80. drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +64 −160
  81. drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +2 −0
  82. drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c +104 −993
  83. drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +0 −125
  84. drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +14 −5
  85. drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c +17 −71
  86. drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c +17 −75
  87. drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_2.c +0 −3
  88. drivers/gpu/drm/amd/amdgpu/mes_userqueue.c +90 −57
  89. drivers/gpu/drm/amd/amdgpu/mes_userqueue.h +9 −0
  90. drivers/gpu/drm/amd/amdgpu/mes_v11_0.c +236 −4
  91. drivers/gpu/drm/amd/amdgpu/mes_v12_0.c +236 −31
  92. drivers/gpu/drm/amd/amdgpu/mes_v12_1.c +19 −6
  93. drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c +1 −230
  94. drivers/gpu/drm/amd/amdgpu/nv.c +0 −6
  95. drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c +0 −62
  96. drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c +15 −181
  97. drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c +0 −18
  98. drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c +0 −18
  99. drivers/gpu/drm/amd/amdgpu/sdma_v7_1.c +0 −18
  100. drivers/gpu/drm/amd/amdgpu/si.c +0 −7
  101. drivers/gpu/drm/amd/amdgpu/soc15.c +0 −9
  102. drivers/gpu/drm/amd/amdgpu/soc15_common.h +31 −34
  103. drivers/gpu/drm/amd/amdgpu/soc21.c +0 −12
  104. drivers/gpu/drm/amd/amdgpu/soc24.c +0 −11
  105. drivers/gpu/drm/amd/amdgpu/soc_v1_0.c +3 −11
  106. drivers/gpu/drm/amd/amdgpu/tonga_ih.c +0 −40
  107. drivers/gpu/drm/amd/amdgpu/umc_v12_0.c +1 −584
  108. drivers/gpu/drm/amd/amdgpu/umc_v12_0.h +0 −28
  109. drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c +0 −45
  110. drivers/gpu/drm/amd/amdgpu/vce_v3_0.c +0 −69
  111. drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c +5 −7
  112. drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c +40 −5
  113. drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c +17 −70
  114. drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c +2 −4
  115. drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c +17 −73
  116. drivers/gpu/drm/amd/amdgpu/vcn_v5_0_2.c +0 −3
  117. drivers/gpu/drm/amd/amdgpu/vi.c +0 −22
  118. drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +8 −4
  119. drivers/gpu/drm/amd/amdkfd/kfd_device.c +24 −0
  120. drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +91 −132
  121. drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h +2 −0
  122. drivers/gpu/drm/amd/amdkfd/kfd_events.c +86 −138
  123. drivers/gpu/drm/amd/amdkfd/kfd_events.h +0 −4
  124. drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c +16 −0
  125. drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.h +2 −0
  126. drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c +0 −14
  127. drivers/gpu/drm/amd/amdkfd/kfd_priv.h +23 −2
  128. drivers/gpu/drm/amd/amdkfd/kfd_process.c +67 −9
  129. drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +7 −1
  130. drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c +2 −2
  131. drivers/gpu/drm/amd/amdkfd/kfd_svm.c +7 −1
  132. drivers/gpu/drm/amd/amdkfd/kfd_topology.c +10 −1
  133. drivers/gpu/drm/amd/amdxcp/amdgpu_xcp_drv.c +12 −6
  134. drivers/gpu/drm/amd/display/amdgpu_dm/Makefile +5 −1
  135. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +2696 −9516
  136. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h +30 −49
  137. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_audio.c +323 −0
  138. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_audio.h +32 −15
  139. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_backlight.c +726 −0
  140. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_backlight.h +75 −0
  141. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c +104 −75
  142. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.h +4 −4
  143. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_colorop.c +1 −0
  144. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_connector.c +3663 −0
  145. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_connector.h +162 −0
  146. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c +80 −22
  147. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.h +6 −0
  148. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c +15 −7
  149. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.h +6 −0
  150. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_debugfs.c +71 −7
  151. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_dmub.c +943 −0
  152. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_dmub.h +68 −0
  153. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c +78 −37
  154. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h +12 −0
  155. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c +47 −17
  156. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.h +20 −0
  157. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c +1543 −5
  158. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.h +27 −0
  159. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_kunit_helpers.h +1 −0
  160. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c +119 −70
  161. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.h +25 −0
  162. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +83 −54
  163. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.h +51 −0
  164. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_pp_smu.c +172 −131
  165. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_pp_smu.h +61 −0
  166. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.c +49 −2
  167. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.h +5 −0
  168. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_quirks.c +2 −0
  169. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c +5 −5
  170. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_services.c +7 −0
  171. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.c +11 −5
  172. drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.h +13 −0
  173. drivers/gpu/drm/amd/display/amdgpu_dm/tests/.kunitconfig +6 −0
  174. drivers/gpu/drm/amd/display/amdgpu_dm/tests/Makefile +18 −0
  175. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_audio_test.c +490 −0
  176. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_backlight_test.c +1242 −0
  177. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_color_test.c +32 −32
  178. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_colorop_test.c +125 −15
  179. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_connector_test.c +2158 −0
  180. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_crc_test.c +122 −0
  181. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_crtc_test.c +523 −0
  182. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_dmub_test.c +582 −0
  183. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_hdcp_test.c +280 −17
  184. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_helpers_test.c +634 −0
  185. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_irq_test.c +909 −0
  186. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_ism_test.c +13 −26
  187. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_kunit_helpers.c +142 −0
  188. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_kunit_test_helpers.h +32 −0
  189. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_mst_types_test.c +1092 −0
  190. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_plane_test.c +1228 −0
  191. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_pp_smu_test.c +2454 −0
  192. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_psr_test.c +497 −21
  193. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_quirks_test.c +103 −0
  194. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_replay_test.c +426 −6
  195. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_services_test.c +313 −0
  196. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_test.c +949 −0
  197. drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_wb_test.c +392 −0
  198. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn401/dcn401_clk_mgr.c +54 −3
  199. drivers/gpu/drm/amd/display/dc/clk_mgr/dcn401/dcn401_clk_mgr.h +9 −0
  200. drivers/gpu/drm/amd/display/dc/core/dc.c +172 −119
  201. drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c +164 −10
  202. drivers/gpu/drm/amd/display/dc/core/dc_resource.c +5 −6
  203. drivers/gpu/drm/amd/display/dc/core/dc_surface.c +38 −6
  204. drivers/gpu/drm/amd/display/dc/dc.h +175 −63
  205. drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c +21 −20
  206. drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h +30 −13
  207. drivers/gpu/drm/amd/display/dc/dc_dp_types.h +1 −0
  208. drivers/gpu/drm/amd/display/dc/dc_helper.c +0 −226
  209. drivers/gpu/drm/amd/display/dc/dc_stream.h +29 −0
  210. drivers/gpu/drm/amd/display/dc/dc_types.h +49 −18
  211. drivers/gpu/drm/amd/display/dc/dccg/dcn42/dcn42_dccg.c +34 −25
  212. drivers/gpu/drm/amd/display/dc/dccg/dcn42/dcn42_dccg.h +1 −0
  213. drivers/gpu/drm/amd/display/dc/dce/dce_aux.c +5 −4
  214. drivers/gpu/drm/amd/display/dc/dce110/dce110_compressor.c +0 −2
  215. drivers/gpu/drm/amd/display/dc/dce110/dce110_mem_input_v.c +0 −2
  216. drivers/gpu/drm/amd/display/dc/dce112/dce112_compressor.c +6 −2
  217. drivers/gpu/drm/amd/display/dc/dcn30/dcn30_cm_common.c +184 −0
  218. drivers/gpu/drm/amd/display/dc/dio/dcn31/dcn31_dio_link_encoder.c +4 −0
  219. drivers/gpu/drm/amd/display/dc/dio/dcn31/dcn31_dio_link_encoder.h +2 −0
  220. drivers/gpu/drm/amd/display/dc/dm_services.h +0 −4
  221. drivers/gpu/drm/amd/display/dc/dml/calcs/dcn_calcs.c +1 −1
  222. drivers/gpu/drm/amd/display/dc/dml/dcn31/display_mode_vba_31.c +1 −1
  223. drivers/gpu/drm/amd/display/dc/dml/dcn314/display_mode_vba_314.c +1 −1
  224. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/dml21_wrapper_fpu.c +4 −4
  225. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/inc/bounding_boxes/dcn42b_soc_bb.h +39 −1
  226. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/inc/dml_top_soc_parameter_types.h +13 −0
  227. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/dml2_core/dml2_core_dcn4_calcs.c +28 −13
  228. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/dml2_core/dml2_core_factory.c +0 −2
  229. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/dml2_core/dml2_core_shared_types.h +0 −2
  230. drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/inc/dml2_internal_shared_types.h +0 −1
  231. drivers/gpu/drm/amd/display/dc/dpp/dcn10/dcn10_dpp_cm.c +0 −5
  232. drivers/gpu/drm/amd/display/dc/dwb/dcn30/dcn30_cm_common.h +4 −0
  233. drivers/gpu/drm/amd/display/dc/gpio/dce80/hw_translate_dce80.c +4 −1
  234. drivers/gpu/drm/amd/display/dc/gpio/dcn10/hw_translate_dcn10.c +173 −311
  235. drivers/gpu/drm/amd/display/dc/gpio/dcn20/hw_translate_dcn20.c +153 −279
  236. drivers/gpu/drm/amd/display/dc/gpio/dcn21/hw_translate_dcn21.c +147 −270
  237. drivers/gpu/drm/amd/display/dc/gpio/dcn30/hw_translate_dcn30.c +153 −279
  238. drivers/gpu/drm/amd/display/dc/gpio/dcn315/hw_translate_dcn315.c +148 −270
  239. drivers/gpu/drm/amd/display/dc/gpio/dcn32/hw_translate_dcn32.c +138 −248
  240. drivers/gpu/drm/amd/display/dc/gpio/dcn401/hw_translate_dcn401.c +137 −255
  241. drivers/gpu/drm/amd/display/dc/gpio/dcn42/hw_translate_dcn42.c +70 −123
  242. drivers/gpu/drm/amd/display/dc/gpio/hw_translate.c +86 −0
  243. drivers/gpu/drm/amd/display/dc/gpio/hw_translate.h +21 −0
  244. drivers/gpu/drm/amd/display/dc/hubbub/dcn31/dcn31_hubbub.c +4 −4
  245. drivers/gpu/drm/amd/display/dc/hubbub/dcn35/dcn35_hubbub.c +1 −1
  246. drivers/gpu/drm/amd/display/dc/hubp/dcn401/dcn401_hubp.c +1 −1
  247. drivers/gpu/drm/amd/display/dc/hubp/dcn42/dcn42_hubp.c +6 −0
  248. drivers/gpu/drm/amd/display/dc/hwss/dce110/dce110_hwseq.c +4 −4
  249. drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.c +4 −4
  250. drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.c +31 −31
  251. drivers/gpu/drm/amd/display/dc/hwss/dcn20/dcn20_hwseq.c +38 −38
  252. drivers/gpu/drm/amd/display/dc/hwss/dcn201/dcn201_hwseq.c +1 −1
  253. drivers/gpu/drm/amd/display/dc/hwss/dcn30/dcn30_hwseq.c +6 −4
  254. drivers/gpu/drm/amd/display/dc/hwss/dcn32/dcn32_hwseq.c +18 −18
  255. drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c +0 −3
  256. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c +214 −127
  257. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h +11 −1
  258. drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_init.c +2 −0
  259. drivers/gpu/drm/amd/display/dc/hwss/dcn42/dcn42_hwseq.c +166 −161
  260. drivers/gpu/drm/amd/display/dc/hwss/dcn42/dcn42_hwseq.h +2 −3
  261. drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h +82 −0
  262. drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer_private.h +2 −1
  263. drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr.h +12 −0
  264. drivers/gpu/drm/amd/display/dc/inc/hw/dccg.h +1 −0
  265. drivers/gpu/drm/amd/display/dc/inc/hw/hubp.h +1 −1
  266. drivers/gpu/drm/amd/display/dc/inc/link_service.h +1 −0
  267. drivers/gpu/drm/amd/display/dc/inc/reg_helper.h +0 −19
  268. drivers/gpu/drm/amd/display/dc/inc/soc_and_ip_translator.h +0 −14
  269. drivers/gpu/drm/amd/display/dc/link/link_detection.c +1 −1
  270. drivers/gpu/drm/amd/display/dc/link/link_factory.c +5 −2
  271. drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.c +17 −3
  272. drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.h +2 −0
  273. drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_capability.c +12 −2
  274. drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_panel_replay.c +2 −1
  275. drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c +3 −5
  276. drivers/gpu/drm/amd/display/dc/mpc/dcn20/dcn20_mpc.c +0 −4
  277. drivers/gpu/drm/amd/display/dc/opp/dcn20/dcn20_opp.c +43 −8
  278. drivers/gpu/drm/amd/display/dc/optc/dcn10/dcn10_optc.c +0 −5
  279. drivers/gpu/drm/amd/display/dc/optc/dcn20/dcn20_optc.c +0 −5
  280. drivers/gpu/drm/amd/display/dc/optc/dcn31/dcn31_optc.c +0 −5
  281. drivers/gpu/drm/amd/display/dc/optc/dcn314/dcn314_optc.c +0 −5
  282. drivers/gpu/drm/amd/display/dc/optc/dcn32/dcn32_optc.c +0 −5
  283. drivers/gpu/drm/amd/display/dc/optc/dcn35/dcn35_optc.c +0 −5
  284. drivers/gpu/drm/amd/display/dc/optc/dcn401/dcn401_optc.c +0 −5
  285. drivers/gpu/drm/amd/display/dc/pg/dcn42/dcn42_pg_cntl.c +125 −133
  286. drivers/gpu/drm/amd/display/dc/pg/dcn42/dcn42_pg_cntl.h +18 −4
  287. drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.c +0 −1
  288. drivers/gpu/drm/amd/display/dc/resource/dcn302/dcn302_resource.c +0 −1
  289. drivers/gpu/drm/amd/display/dc/resource/dcn303/dcn303_resource.c +0 −1
  290. drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.c +3 −2
  291. drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.c +3 −1
  292. drivers/gpu/drm/amd/display/dc/resource/dcn315/dcn315_resource.c +3 −2
  293. drivers/gpu/drm/amd/display/dc/resource/dcn316/dcn316_resource.c +3 −2
  294. drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c +0 −1
  295. drivers/gpu/drm/amd/display/dc/resource/dcn321/dcn321_resource.c +0 −1
  296. drivers/gpu/drm/amd/display/dc/resource/dcn35/dcn35_resource.c +3 −3
  297. drivers/gpu/drm/amd/display/dc/resource/dcn351/dcn351_resource.c +3 −3
  298. drivers/gpu/drm/amd/display/dc/resource/dcn36/dcn36_resource.c +3 −3
  299. drivers/gpu/drm/amd/display/dc/resource/dcn42/dcn42_resource.c +4 −1
  300. drivers/gpu/drm/amd/display/dc/resource/dcn42b/dcn42b_resource.c +4 −2
  301. drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/Makefile +3 −0
  302. drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42/dcn42_soc_and_ip_translator.c +6 −12
  303. drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42/dcn42_soc_and_ip_translator.h +1 −0
  304. drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42b/dcn42b_soc_and_ip_translator.c +42 −0
  305. drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42b/dcn42b_soc_and_ip_translator.h +17 −0
  306. drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/soc_and_ip_translator.c +4 −1
  307. drivers/gpu/drm/amd/display/dc/sspl/spl_debug.h +2 −2
  308. drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h +144 −145
  309. drivers/gpu/drm/amd/display/include/ddc_service_types.h +1 −0
  310. drivers/gpu/drm/amd/display/include/gpio_types.h +48 −0
  311. drivers/gpu/drm/amd/display/modules/power/Makefile +1 −1
  312. drivers/gpu/drm/amd/display/modules/power/power.c +6 −6
  313. drivers/gpu/drm/amd/display/modules/power/power_abm.c +2 −12
  314. drivers/gpu/drm/amd/display/modules/power/power_replay.c +4 −5
  315. drivers/gpu/drm/amd/include/amd_shared.h +0 −3
  316. drivers/gpu/drm/amd/include/asic_reg/sdma/sdma_4_4_2_offset.h +4 −0
  317. drivers/gpu/drm/amd/include/kgd_kfd_interface.h +3 −0
  318. drivers/gpu/drm/amd/include/mes_v11_api_def.h +3 −2
  319. drivers/gpu/drm/amd/include/mes_v12_api_def.h +3 −2
  320. drivers/gpu/drm/amd/include/soc15_hw_ip.h +1 −0
  321. drivers/gpu/drm/amd/include/v9_structs.h +2 −2
  322. drivers/gpu/drm/amd/pm/amdgpu_pm.c +48 −68
  323. drivers/gpu/drm/amd/pm/powerplay/amd_powerplay.c +8 −13
  324. drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c +30 −8
  325. drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c +8 −3
  326. drivers/gpu/drm/amd/pm/powerplay/hwmgr/process_pptables_v1_0.c +626 −129
  327. drivers/gpu/drm/amd/pm/powerplay/hwmgr/processpptables.c +316 −130
  328. drivers/gpu/drm/amd/pm/powerplay/hwmgr/smu7_hwmgr.c +44 −14
  329. drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c +66 −15
  330. drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_processpptables.c +599 −162
  331. drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega12_processpptables.c +7 −0
  332. drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega20_processpptables.c +7 −0
  333. drivers/gpu/drm/amd/pm/powerplay/inc/hwmgr.h +17 −0
  334. drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c +16 −18
  335. drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h +7 −30
  336. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v11_0.h +0 −2
  337. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v13_0.h +0 −6
  338. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v14_0.h +0 −4
  339. drivers/gpu/drm/amd/pm/swsmu/inc/smu_v15_0.h +0 −4
  340. drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c +4 −4
  341. drivers/gpu/drm/amd/pm/swsmu/smu11/navi10_ppt.c +4 −4
  342. drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c +4 −4
  343. drivers/gpu/drm/amd/pm/swsmu/smu11/smu_v11_0.c +10 −74
  344. drivers/gpu/drm/amd/pm/swsmu/smu12/renoir_ppt.c +0 −1
  345. drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c +0 −1
  346. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c +2 −92
  347. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c +4 −3
  348. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_12_ppt.c +1 −7
  349. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c +0 −653
  350. drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_7_ppt.c +4 −3
  351. drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0.c +2 −83
  352. drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c +4 −5
  353. drivers/gpu/drm/amd/pm/swsmu/smu15/smu_v15_0.c +2 −83
  354. drivers/gpu/drm/amd/pm/swsmu/smu_cmn.c +140 −0
  355. drivers/gpu/drm/amd/pm/swsmu/smu_cmn.h +13 −0
  356. drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_cmd.c +0 −3
  357. drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_eeprom_i2c.c +1 −1
  358. drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_mgr.c +74 −2
  359. drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_mgr.h +2 −0
  360. drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_mp1_v13_0.c +14 −0
  361. drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_sys.c +19 −0
  362. drivers/gpu/drm/amd/ras/ras_mgr/ras_sys.h +3 −0
  363. drivers/gpu/drm/amd/ras/rascore/ras.h +5 −0
  364. drivers/gpu/drm/amd/ras/rascore/ras_aca_v1_0.c +6 −7
  365. drivers/gpu/drm/amd/ras/rascore/ras_core.c +15 −0
  366. drivers/gpu/drm/amd/ras/rascore/ras_eeprom.c +3 −0
  367. drivers/gpu/drm/amd/ras/rascore/ras_eeprom_fw.c +1 −1
  368. drivers/gpu/drm/amd/ras/rascore/ras_mp1.c +18 −1
  369. drivers/gpu/drm/amd/ras/rascore/ras_mp1.h +3 −0
  370. drivers/gpu/drm/amd/ras/rascore/ras_mp1_v13_0.c +13 −0
  371. drivers/gpu/drm/amd/ras/rascore/ras_process.c +3 −2
  372. drivers/gpu/drm/amd/ras/rascore/ras_umc.c +28 −4
  373. drivers/gpu/drm/amd/ras/rascore/ras_umc.h +6 −0
  374. drivers/gpu/drm/amd/ras/rascore/ras_umc_v12_0.h +2 −0
  375. drivers/gpu/drm/radeon/r600_dpm.c +15 −6
  376. include/uapi/drm/amdgpu_drm.h +21 −0
drivers/gpu/drm/amd/amdgpu/Makefile+1−1
@@ -70,7 +70,7 @@ amdgpu-y += amdgpu_device.o amdgpu_reg_access.o amdgpu_doorbell_mgr.o amdgpu_kms · 69 unmodified lines
70 amdgpu_umc.o smu_v11_0_i2c.o amdgpu_fru_eeprom.o amdgpu_rap.o \70 amdgpu_umc.o smu_v11_0_i2c.o amdgpu_fru_eeprom.o amdgpu_rap.o \
71 amdgpu_fw_attestation.o amdgpu_securedisplay.o \71 amdgpu_fw_attestation.o amdgpu_securedisplay.o \
72 amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o amdgpu_lockdep.o \72 amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o amdgpu_lockdep.o \
73 amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_aca.o amdgpu_dev_coredump.o \73 amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_dev_coredump.o \
74 amdgpu_cper.o amdgpu_userq_fence.o amdgpu_eviction_fence.o amdgpu_ip.o74 amdgpu_cper.o amdgpu_userq_fence.o amdgpu_eviction_fence.o amdgpu_ip.o
75 75
76amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o76amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o
drivers/gpu/drm/amd/amdgpu/amdgpu.h+10−42
@@ -44,6 +44,7 @@ · 43 unmodified lines
44#include <linux/hashtable.h>44#include <linux/hashtable.h>
45#include <linux/dma-fence.h>45#include <linux/dma-fence.h>
46#include <linux/pci.h>46#include <linux/pci.h>
47#include <linux/xarray.h>
47 48
48#include <drm/ttm/ttm_bo.h>49#include <drm/ttm/ttm_bo.h>
49#include <drm/ttm/ttm_placement.h>50#include <drm/ttm/ttm_placement.h>
@@ -103,7 +104,6 @@ · 53 unmodified lines
103#include "amdgpu_smuio.h"104#include "amdgpu_smuio.h"
104#include "amdgpu_fdinfo.h"105#include "amdgpu_fdinfo.h"
105#include "amdgpu_mca.h"106#include "amdgpu_mca.h"
106#include "amdgpu_aca.h"
107#include "amdgpu_ras.h"107#include "amdgpu_ras.h"
108#include "amdgpu_lockdep.h"108#include "amdgpu_lockdep.h"
109#include "amdgpu_cper.h"109#include "amdgpu_cper.h"
@@ -113,6 +113,7 @@ · 3 unmodified lines
113#include "amdgpu_userq.h"113#include "amdgpu_userq.h"
114#include "amdgpu_eviction_fence.h"114#include "amdgpu_eviction_fence.h"
115#include "amdgpu_ip.h"115#include "amdgpu_ip.h"
116#include "amdgpu_sa.h"
116#if defined(CONFIG_DRM_AMD_ISP)117#if defined(CONFIG_DRM_AMD_ISP)
117#include "amdgpu_isp.h"118#include "amdgpu_isp.h"
118#endif119#endif
@@ -272,7 +273,6 @@ extern int amdgpu_ptl; · 153 unmodified lines
272 273
273extern uint amdgpu_hdmi_hpd_debounce_delay_ms;274extern uint amdgpu_hdmi_hpd_debounce_delay_ms;
274 275
275#define AMDGPU_VM_MAX_NUM_CTX 4096
276#define AMDGPU_SG_THRESHOLD (256*1024*1024)276#define AMDGPU_SG_THRESHOLD (256*1024*1024)
277#define AMDGPU_WAIT_IDLE_TIMEOUT_IN_MS 3000277#define AMDGPU_WAIT_IDLE_TIMEOUT_IN_MS 3000
278#define AMDGPU_MAX_USEC_TIMEOUT 100000 /* 100 ms */278#define AMDGPU_MAX_USEC_TIMEOUT 100000 /* 100 ms */
@@ -305,9 +305,10 @@ extern uint amdgpu_hdmi_hpd_debounce_delay_ms; · 26 unmodified lines
305 305
306/* reset mask */306/* reset mask */
307#define AMDGPU_RESET_TYPE_FULL (1 << 0) /* full adapter reset, mode1/mode2/BACO/etc. */307#define AMDGPU_RESET_TYPE_FULL (1 << 0) /* full adapter reset, mode1/mode2/BACO/etc. */
308#define AMDGPU_RESET_TYPE_SOFT_RESET (1 << 1) /* IP level soft reset */308#define AMDGPU_RESET_TYPE_SOFT_RECOVERY (1 << 1) /* soft recovery, eg. kill shaders */
309#define AMDGPU_RESET_TYPE_PER_QUEUE (1 << 2) /* per queue */309#define AMDGPU_RESET_TYPE_PER_QUEUE (1 << 2) /* per queue */
310#define AMDGPU_RESET_TYPE_PER_PIPE (1 << 3) /* per pipe */310#define AMDGPU_RESET_TYPE_PER_PIPE (1 << 3) /* per pipe */
311#define AMDGPU_RESET_TYPE_IP_BLOCK_SOFT_RESET (1 << 4) /* soft-resets an IP block */
311 312
312/* max cursor sizes (in pixels) */313/* max cursor sizes (in pixels) */
313#define CIK_CURSOR_WIDTH 128314#define CIK_CURSOR_WIDTH 128
@@ -387,37 +388,6 @@ struct amdgpu_clock { · 73 unmodified lines
387 uint32_t max_pixel_clock;388 uint32_t max_pixel_clock;
388};389};
389 390
390/* sub-allocation manager, it has to be protected by another lock.
391 * By conception this is an helper for other part of the driver
392 * like the indirect buffer or semaphore, which both have their
393 * locking.
394 *
395 * Principe is simple, we keep a list of sub allocation in offset
396 * order (first entry has offset == 0, last entry has the highest
397 * offset).
398 *
399 * When allocating new object we first check if there is room at
400 * the end total_size - (last_object_offset + last_object_size) >=
401 * alloc_size. If so we allocate new object there.
402 *
403 * When there is not enough room at the end, we start waiting for
404 * each sub object until we reach object_offset+object_size >=
405 * alloc_size, this object then become the sub object we return.
406 *
407 * Alignment can't be bigger than page size.
408 *
409 * Hole are not considered for allocation to keep things simple.
410 * Assumption is that there won't be hole (all object on same
411 * alignment).
412 */
413
414struct amdgpu_sa_manager {
415 struct drm_suballoc_manager base;
416 struct amdgpu_bo *bo;
417 uint64_t gpu_addr;
418 void *cpu_ptr;
419};
420
421/*391/*
422 * IRQS.392 * IRQS.
423 */393 */
@@ -446,8 +416,7 @@ struct amdgpu_fpriv { · 22 unmodified lines
446 struct amdgpu_bo_va *prt_va;416 struct amdgpu_bo_va *prt_va;
447 struct amdgpu_bo_va *csa_va;417 struct amdgpu_bo_va *csa_va;
448 struct amdgpu_bo_va *seq64_va;418 struct amdgpu_bo_va *seq64_va;
449 struct mutex bo_list_lock;419 struct xarray bo_list_handles;
450 struct idr bo_list_handles;
451 struct amdgpu_ctx_mgr ctx_mgr;420 struct amdgpu_ctx_mgr ctx_mgr;
452 struct amdgpu_userq_mgr userq_mgr;421 struct amdgpu_userq_mgr userq_mgr;
453 422
@@ -587,8 +556,6 @@ struct amdgpu_asic_funcs { · 133 unmodified lines
587 /* invalidate hdp read cache */556 /* invalidate hdp read cache */
588 void (*invalidate_hdp)(struct amdgpu_device *adev,557 void (*invalidate_hdp)(struct amdgpu_device *adev,
589 struct amdgpu_ring *ring);558 struct amdgpu_ring *ring);
590 /* check if the asic needs a full reset of if soft reset will work */
591 bool (*need_full_reset)(struct amdgpu_device *adev);
592 /* initialize doorbell layout for specific asic*/559 /* initialize doorbell layout for specific asic*/
593 void (*init_doorbell_index)(struct amdgpu_device *adev);560 void (*init_doorbell_index)(struct amdgpu_device *adev);
594 /* PCIe bandwidth usage */561 /* PCIe bandwidth usage */
@@ -851,6 +818,7 @@ struct amdgpu_device { · 256 unmodified lines
851 struct dev_pm_domain vga_pm_domain;818 struct dev_pm_domain vga_pm_domain;
852 bool have_disp_power_ref;819 bool have_disp_power_ref;
853 bool have_atomics_support;820 bool have_atomics_support;
821 bool is_sw_smu;
854 822
855 /* BIOS */823 /* BIOS */
856 bool is_atom_fw;824 bool is_atom_fw;
@@ -1022,9 +990,6 @@ struct amdgpu_device { · 165 unmodified lines
1022 /* MCA */990 /* MCA */
1023 struct amdgpu_mca mca;991 struct amdgpu_mca mca;
1024 992
1025 /* ACA */
1026 struct amdgpu_aca aca;
1027
1028 /* CPER */993 /* CPER */
1029 struct amdgpu_cper cper;994 struct amdgpu_cper cper;
1030 995
@@ -1136,6 +1101,8 @@ struct amdgpu_device { · 105 unmodified lines
1136 bool debug_vm_userptr;1101 bool debug_vm_userptr;
1137 bool debug_disable_ce_logs;1102 bool debug_disable_ce_logs;
1138 bool debug_enable_ce_cs;1103 bool debug_enable_ce_cs;
1104 bool debug_hibernation_thaw_resume_gpu;
1105 bool debug_disable_ip_block_soft_reset;
1139 1106
1140 /* Protection for the following isolation structure */1107 /* Protection for the following isolation structure */
1141 struct mutex enforce_isolation_mutex;1108 struct mutex enforce_isolation_mutex;
@@ -1356,7 +1323,6 @@ int emu_soc_asic_init(struct amdgpu_device *adev); · 214 unmodified lines
1356#define amdgpu_asic_read_bios_from_rom(adev, b, l) (adev)->asic_funcs->read_bios_from_rom((adev), (b), (l))1323#define amdgpu_asic_read_bios_from_rom(adev, b, l) (adev)->asic_funcs->read_bios_from_rom((adev), (b), (l))
1357#define amdgpu_asic_read_register(adev, se, sh, offset, v)((adev)->asic_funcs->read_register((adev), (se), (sh), (offset), (v)))1324#define amdgpu_asic_read_register(adev, se, sh, offset, v)((adev)->asic_funcs->read_register((adev), (se), (sh), (offset), (v)))
1358#define amdgpu_asic_get_config_memsize(adev) (adev)->asic_funcs->get_config_memsize((adev))1325#define amdgpu_asic_get_config_memsize(adev) (adev)->asic_funcs->get_config_memsize((adev))
1359#define amdgpu_asic_need_full_reset(adev) (adev)->asic_funcs->need_full_reset((adev))
1360#define amdgpu_asic_init_doorbell_index(adev) (adev)->asic_funcs->init_doorbell_index((adev))1326#define amdgpu_asic_init_doorbell_index(adev) (adev)->asic_funcs->init_doorbell_index((adev))
1361#define amdgpu_asic_get_pcie_usage(adev, cnt0, cnt1) ((adev)->asic_funcs->get_pcie_usage((adev), (cnt0), (cnt1)))1327#define amdgpu_asic_get_pcie_usage(adev, cnt0, cnt1) ((adev)->asic_funcs->get_pcie_usage((adev), (cnt0), (cnt1)))
1362#define amdgpu_asic_need_reset_on_init(adev) (adev)->asic_funcs->need_reset_on_init((adev))1328#define amdgpu_asic_need_reset_on_init(adev) (adev)->asic_funcs->need_reset_on_init((adev))
@@ -1468,6 +1434,8 @@ int amdgpu_enable_vblank_kms(struct drm_crtc *crtc); · 105 unmodified lines
1468void amdgpu_disable_vblank_kms(struct drm_crtc *crtc);1434void amdgpu_disable_vblank_kms(struct drm_crtc *crtc);
1469int amdgpu_info_ioctl(struct drm_device *dev, void *data,1435int amdgpu_info_ioctl(struct drm_device *dev, void *data,
1470 struct drm_file *filp);1436 struct drm_file *filp);
1437int amdgpu_proc_options_ioctl(struct drm_device *dev, void *data,
1438 struct drm_file *filp);
1471 1439
1472/*1440/*
1473 * functions used by amdgpu_encoder.c1441 * functions used by amdgpu_encoder.c
drivers/gpu/drm/amd/amdgpu/amdgpu_aca.c+0−985
@@ -1,985 +0,0 @@
1/*
2 * Copyright 2023 Advanced Micro Devices, Inc.
3 *
4 * Permission is hereby granted, free of charge, to any person obtaining a
5 * copy of this software and associated documentation files (the "Software"),
6 * to deal in the Software without restriction, including without limitation
7 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8 * and/or sell copies of the Software, and to permit persons to whom the
9 * Software is furnished to do so, subject to the following conditions:
10 *
11 * The above copyright notice and this permission notice shall be included in
12 * all copies or substantial portions of the Software.
13 *
14 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
17 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
18 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
19 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
20 * OTHER DEALINGS IN THE SOFTWARE.
21 *
22 */
23
24#include <linux/list.h>
25#include "amdgpu.h"
26#include "amdgpu_aca.h"
27#include "amdgpu_ras.h"
28
29#define ACA_BANK_HWID(type, hwid, mcatype) [ACA_HWIP_TYPE_##type] = {hwid, mcatype}
30
31typedef int bank_handler_t(struct aca_handle *handle, struct aca_bank *bank, enum aca_smu_type type, void *data);
32
33static struct aca_hwip aca_hwid_mcatypes[ACA_HWIP_TYPE_COUNT] = {
34 ACA_BANK_HWID(SMU, 0x01, 0x01),
35 ACA_BANK_HWID(PCS_XGMI, 0x50, 0x00),
36 ACA_BANK_HWID(UMC, 0x96, 0x00),
37};
38
39static void aca_banks_init(struct aca_banks *banks)
40{
41 if (!banks)
42 return;
43
44 memset(banks, 0, sizeof(*banks));
45 INIT_LIST_HEAD(&banks->list);
46}
47
48static int aca_banks_add_bank(struct aca_banks *banks, struct aca_bank *bank)
49{
50 struct aca_bank_node *node;
51
52 if (!bank)
53 return -EINVAL;
54
55 node = kvzalloc_obj(*node);
56 if (!node)
57 return -ENOMEM;
58
59 memcpy(&node->bank, bank, sizeof(*bank));
60
61 INIT_LIST_HEAD(&node->node);
62 list_add_tail(&node->node, &banks->list);
63
64 banks->nr_banks++;
65
66 return 0;
67}
68
69static void aca_banks_release(struct aca_banks *banks)
70{
71 struct aca_bank_node *node, *tmp;
72
73 if (list_empty(&banks->list))
74 return;
75
76 list_for_each_entry_safe(node, tmp, &banks->list, node) {
77 list_del(&node->node);
78 kvfree(node);
79 banks->nr_banks--;
80 }
81}
82
83static int aca_smu_get_valid_aca_count(struct amdgpu_device *adev, enum aca_smu_type type, u32 *count)
84{
85 struct amdgpu_aca *aca = &adev->aca;
86 const struct aca_smu_funcs *smu_funcs = aca->smu_funcs;
87
88 if (!count)
89 return -EINVAL;
90
91 if (!smu_funcs || !smu_funcs->get_valid_aca_count)
92 return -EOPNOTSUPP;
93
94 return smu_funcs->get_valid_aca_count(adev, type, count);
95}
96
97static struct aca_regs_dump {
98 const char *name;
99 int reg_idx;
100} aca_regs[] = {
101 {"CONTROL", ACA_REG_IDX_CTL},
102 {"STATUS", ACA_REG_IDX_STATUS},
103 {"ADDR", ACA_REG_IDX_ADDR},
104 {"MISC", ACA_REG_IDX_MISC0},
105 {"CONFIG", ACA_REG_IDX_CONFIG},
106 {"IPID", ACA_REG_IDX_IPID},
107 {"SYND", ACA_REG_IDX_SYND},
108 {"DESTAT", ACA_REG_IDX_DESTAT},
109 {"DEADDR", ACA_REG_IDX_DEADDR},
110 {"CONTROL_MASK", ACA_REG_IDX_CTL_MASK},
111};
112
113static void aca_smu_bank_dump(struct amdgpu_device *adev, int idx, int total, struct aca_bank *bank,
114 struct ras_query_context *qctx)
115{
116 u64 event_id = qctx ? qctx->evid.event_id : RAS_EVENT_INVALID_ID;
117 int i;
118
119 if (adev->debug_disable_ce_logs &&
120 bank->smu_err_type == ACA_SMU_TYPE_CE &&
121 !ACA_BANK_ERR_IS_DEFFERED(bank))
122 return;
123
124 RAS_EVENT_LOG(adev, event_id, HW_ERR "Accelerator Check Architecture events logged\n");
125 /* plus 1 for output format, e.g: ACA[08/08]: xxxx */
126 for (i = 0; i < ARRAY_SIZE(aca_regs); i++)
127 RAS_EVENT_LOG(adev, event_id, HW_ERR "ACA[%02d/%02d].%s=0x%016llx\n",
128 idx + 1, total, aca_regs[i].name, bank->regs[aca_regs[i].reg_idx]);
129
130 if (ACA_REG__STATUS__SCRUB(bank->regs[ACA_REG_IDX_STATUS]))
131 RAS_EVENT_LOG(adev, event_id, HW_ERR "hardware error logged by the scrubber\n");
132}
133
134static bool aca_bank_hwip_is_matched(struct aca_bank *bank, enum aca_hwip_type type)
135{
136
137 struct aca_hwip *hwip;
138 int hwid, mcatype;
139 u64 ipid;
140
141 if (!bank || type == ACA_HWIP_TYPE_UNKNOW)
142 return false;
143
144 hwip = &aca_hwid_mcatypes[type];
145 if (!hwip->hwid)
146 return false;
147
148 ipid = bank->regs[ACA_REG_IDX_IPID];
149 hwid = ACA_REG__IPID__HARDWAREID(ipid);
150 mcatype = ACA_REG__IPID__MCATYPE(ipid);
151
152 return hwip->hwid == hwid && hwip->mcatype == mcatype;
153}
154
155static int aca_smu_get_valid_aca_banks(struct amdgpu_device *adev, enum aca_smu_type type,
156 int start, int count,
157 struct aca_banks *banks, struct ras_query_context *qctx)
158{
159 struct amdgpu_aca *aca = &adev->aca;
160 const struct aca_smu_funcs *smu_funcs = aca->smu_funcs;
161 struct aca_bank bank;
162 int i, max_count, ret;
163
164 if (!count)
165 return 0;
166
167 if (!smu_funcs || !smu_funcs->get_valid_aca_bank)
168 return -EOPNOTSUPP;
169
170 switch (type) {
171 case ACA_SMU_TYPE_UE:
172 max_count = smu_funcs->max_ue_bank_count;
173 break;
174 case ACA_SMU_TYPE_CE:
175 max_count = smu_funcs->max_ce_bank_count;
176 break;
177 default:
178 return -EINVAL;
179 }
180
181 if (start + count > max_count)
182 return -EINVAL;
183
184 count = min_t(int, count, max_count);
185 for (i = 0; i < count; i++) {
186 memset(&bank, 0, sizeof(bank));
187 ret = smu_funcs->get_valid_aca_bank(adev, type, start + i, &bank);
188 if (ret)
189 return ret;
190
191 bank.smu_err_type = type;
192
193 /*
194 * Poison being consumed when injecting a UE while running background workloads,
195 * which are unexpected.
196 */
197 if (type == ACA_SMU_TYPE_UE &&
198 ACA_REG__STATUS__POISON(bank.regs[ACA_REG_IDX_STATUS]) &&
199 !aca_bank_hwip_is_matched(&bank, ACA_HWIP_TYPE_UMC))
200 continue;
201
202 aca_smu_bank_dump(adev, i, count, &bank, qctx);
203
204 ret = aca_banks_add_bank(banks, &bank);
205 if (ret)
206 return ret;
207 }
208
209 return 0;
210}
211
212static bool aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank, enum aca_smu_type type)
213{
214 const struct aca_bank_ops *bank_ops = handle->bank_ops;
215
216 /* Parse all deferred errors with UMC aca handle */
217 if (ACA_BANK_ERR_IS_DEFFERED(bank))
218 return handle->hwip == ACA_HWIP_TYPE_UMC;
219
220 if (!aca_bank_hwip_is_matched(bank, handle->hwip))
221 return false;
222
223 if (!bank_ops->aca_bank_is_valid)
224 return true;
225
226 return bank_ops->aca_bank_is_valid(handle, bank, type, handle->data);
227}
228
229static struct aca_bank_error *new_bank_error(struct aca_error *aerr, struct aca_bank_info *info)
230{
231 struct aca_bank_error *bank_error;
232
233 bank_error = kvzalloc_obj(*bank_error);
234 if (!bank_error)
235 return NULL;
236
237 INIT_LIST_HEAD(&bank_error->node);
238 memcpy(&bank_error->info, info, sizeof(*info));
239
240 mutex_lock(&aerr->lock);
241 list_add_tail(&bank_error->node, &aerr->list);
242 aerr->nr_errors++;
243 mutex_unlock(&aerr->lock);
244
245 return bank_error;
246}
247
248static struct aca_bank_error *find_bank_error(struct aca_error *aerr, struct aca_bank_info *info)
249{
250 struct aca_bank_error *bank_error = NULL;
251 struct aca_bank_info *tmp_info;
252 bool found = false;
253
254 mutex_lock(&aerr->lock);
255 list_for_each_entry(bank_error, &aerr->list, node) {
256 tmp_info = &bank_error->info;
257 if (tmp_info->socket_id == info->socket_id &&
258 tmp_info->die_id == info->die_id) {
259 found = true;
260 goto out_unlock;
261 }
262 }
263
264out_unlock:
265 mutex_unlock(&aerr->lock);
266
267 return found ? bank_error : NULL;
268}
269
270static void aca_bank_error_remove(struct aca_error *aerr, struct aca_bank_error *bank_error)
271{
272 if (!aerr || !bank_error)
273 return;
274
275 list_del(&bank_error->node);
276 aerr->nr_errors--;
277
278 kvfree(bank_error);
279}
280
281static struct aca_bank_error *get_bank_error(struct aca_error *aerr, struct aca_bank_info *info)
282{
283 struct aca_bank_error *bank_error;
284
285 if (!aerr || !info)
286 return NULL;
287
288 bank_error = find_bank_error(aerr, info);
289 if (bank_error)
290 return bank_error;
291
292 return new_bank_error(aerr, info);
293}
294
295int aca_error_cache_log_bank_error(struct aca_handle *handle, struct aca_bank_info *info,
296 enum aca_error_type type, u64 count)
297{
298 struct aca_error_cache *error_cache = &handle->error_cache;
299 struct aca_bank_error *bank_error;
300 struct aca_error *aerr;
301
302 if (!handle || !info || type >= ACA_ERROR_TYPE_COUNT)
303 return -EINVAL;
304
305 if (!count)
306 return 0;
307
308 aerr = &error_cache->errors[type];
309 bank_error = get_bank_error(aerr, info);
310 if (!bank_error)
311 return -ENOMEM;
312
313 bank_error->count += count;
314
315 return 0;
316}
317
318static int aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank, enum aca_smu_type type)
319{
320 const struct aca_bank_ops *bank_ops = handle->bank_ops;
321
322 if (!bank)
323 return -EINVAL;
324
325 if (!bank_ops->aca_bank_parser)
326 return -EOPNOTSUPP;
327
328 return bank_ops->aca_bank_parser(handle, bank, type,
329 handle->data);
330}
331
332static int handler_aca_log_bank_error(struct aca_handle *handle, struct aca_bank *bank,
333 enum aca_smu_type type, void *data)
334{
335 int ret;
336
337 ret = aca_bank_parser(handle, bank, type);
338 if (ret)
339 return ret;
340
341 return 0;
342}
343
344static int aca_dispatch_bank(struct aca_handle_manager *mgr, struct aca_bank *bank,
345 enum aca_smu_type type, bank_handler_t handler, void *data)
346{
347 struct aca_handle *handle;
348 int ret;
349
350 if (list_empty(&mgr->list))
351 return 0;
352
353 list_for_each_entry(handle, &mgr->list, node) {
354 if (!aca_bank_is_valid(handle, bank, type))
355 continue;
356
357 ret = handler(handle, bank, type, data);
358 if (ret)
359 return ret;
360 }
361
362 return 0;
363}
364
365static int aca_dispatch_banks(struct aca_handle_manager *mgr, struct aca_banks *banks,
366 enum aca_smu_type type, bank_handler_t handler, void *data)
367{
368 struct aca_bank_node *node;
369 struct aca_bank *bank;
370 int ret;
371
372 if (!mgr || !banks)
373 return -EINVAL;
374
375 /* pre check to avoid unnecessary operations */
376 if (list_empty(&mgr->list) || list_empty(&banks->list))
377 return 0;
378
379 list_for_each_entry(node, &banks->list, node) {
380 bank = &node->bank;
381
382 ret = aca_dispatch_bank(mgr, bank, type, handler, data);
383 if (ret)
384 return ret;
385 }
386
387 return 0;
388}
389
390static bool aca_bank_should_update(struct amdgpu_device *adev, enum aca_smu_type type)
391{
392 struct amdgpu_aca *aca = &adev->aca;
393 bool ret = true;
394
395 /*
396 * Because the UE Valid MCA count will only be cleared after reset,
397 * in order to avoid repeated counting of the error count,
398 * the aca bank is only updated once during the gpu recovery stage.
399 */
400 if (type == ACA_SMU_TYPE_UE) {
401 if (amdgpu_ras_intr_triggered())
402 ret = atomic_cmpxchg(&aca->ue_update_flag, 0, 1) == 0;
403 else
404 atomic_set(&aca->ue_update_flag, 0);
405 }
406
407 return ret;
408}
409
410static void aca_banks_generate_cper(struct amdgpu_device *adev,
411 enum aca_smu_type type,
412 struct aca_banks *banks,
413 int count)
414{
415 struct aca_bank_node *node;
416 struct aca_bank *bank;
417 int r;
418
419 if (!adev->cper.enabled)
420 return;
421
422 if (!banks || !count) {
423 dev_warn(adev->dev, "fail to generate cper records\n");
424 return;
425 }
426
427 /* UEs must be encoded into separate CPER entries */
428 if (type == ACA_SMU_TYPE_UE) {
429 struct aca_banks de_banks;
430
431 aca_banks_init(&de_banks);
432 list_for_each_entry(node, &banks->list, node) {
433 bank = &node->bank;
434 if (bank->aca_err_type == ACA_ERROR_TYPE_DEFERRED) {
435 r = aca_banks_add_bank(&de_banks, bank);
436 if (r)
437 dev_warn(adev->dev, "fail to add de banks, ret = %d\n", r);
438 } else {
439 if (amdgpu_cper_generate_ue_record(adev, bank))
440 dev_warn(adev->dev, "fail to generate ue cper records\n");
441 }
442 }
443
444 if (!list_empty(&de_banks.list)) {
445 if (amdgpu_cper_generate_ce_records(adev, &de_banks, de_banks.nr_banks))
446 dev_warn(adev->dev, "fail to generate de cper records\n");
447 }
448
449 aca_banks_release(&de_banks);
450 } else {
451 /*
452 * SMU_TYPE_CE banks are combined into 1 CPER entries,
453 * they could be CEs or DEs or both
454 */
455 if (amdgpu_cper_generate_ce_records(adev, banks, count))
456 dev_warn(adev->dev, "fail to generate ce cper records\n");
457 }
458}
459
460static int aca_banks_update(struct amdgpu_device *adev, enum aca_smu_type type,
461 bank_handler_t handler, struct ras_query_context *qctx, void *data)
462{
463 struct amdgpu_aca *aca = &adev->aca;
464 struct aca_banks banks;
465 u32 count = 0;
466 int ret;
467
468 if (list_empty(&aca->mgr.list))
469 return 0;
470
471 if (!aca_bank_should_update(adev, type))
472 return 0;
473
474 ret = aca_smu_get_valid_aca_count(adev, type, &count);
475 if (ret)
476 return ret;
477
478 if (!count)
479 return 0;
480
481 aca_banks_init(&banks);
482
483 ret = aca_smu_get_valid_aca_banks(adev, type, 0, count, &banks, qctx);
484 if (ret)
485 goto err_release_banks;
486
487 if (list_empty(&banks.list)) {
488 ret = 0;
489 goto err_release_banks;
490 }
491
492 ret = aca_dispatch_banks(&aca->mgr, &banks, type,
493 handler, data);
494 if (ret)
495 goto err_release_banks;
496
497 aca_banks_generate_cper(adev, type, &banks, count);
498
499err_release_banks:
500 aca_banks_release(&banks);
501
502 return ret;
503}
504
505static int aca_log_aca_error_data(struct aca_bank_error *bank_error, enum aca_error_type type, struct ras_err_data *err_data)
506{
507 struct aca_bank_info *info;
508 struct amdgpu_smuio_mcm_config_info mcm_info;
509 u64 count;
510
511 if (type >= ACA_ERROR_TYPE_COUNT)
512 return -EINVAL;
513
514 count = bank_error->count;
515 if (!count)
516 return 0;
517
518 info = &bank_error->info;
519 mcm_info.die_id = info->die_id;
520 mcm_info.socket_id = info->socket_id;
521
522 switch (type) {
523 case ACA_ERROR_TYPE_UE:
524 amdgpu_ras_error_statistic_ue_count(err_data, &mcm_info, count);
525 break;
526 case ACA_ERROR_TYPE_CE:
527 amdgpu_ras_error_statistic_ce_count(err_data, &mcm_info, count);
528 break;
529 case ACA_ERROR_TYPE_DEFERRED:
530 amdgpu_ras_error_statistic_de_count(err_data, &mcm_info, count);
531 break;
532 default:
533 break;
534 }
535
536 return 0;
537}
538
539static int aca_log_aca_error(struct aca_handle *handle, enum aca_error_type type, struct ras_err_data *err_data)
540{
541 struct aca_error_cache *error_cache = &handle->error_cache;
542 struct aca_error *aerr = &error_cache->errors[type];
543 struct aca_bank_error *bank_error, *tmp;
544
545 mutex_lock(&aerr->lock);
546
547 if (list_empty(&aerr->list))
548 goto out_unlock;
549
550 list_for_each_entry_safe(bank_error, tmp, &aerr->list, node) {
551 aca_log_aca_error_data(bank_error, type, err_data);
552 aca_bank_error_remove(aerr, bank_error);
553 }
554
555out_unlock:
556 mutex_unlock(&aerr->lock);
557
558 return 0;
559}
560
561static int __aca_get_error_data(struct amdgpu_device *adev, struct aca_handle *handle, enum aca_error_type type,
562 struct ras_err_data *err_data, struct ras_query_context *qctx)
563{
564 enum aca_smu_type smu_type;
565 int ret;
566
567 switch (type) {
568 case ACA_ERROR_TYPE_UE:
569 smu_type = ACA_SMU_TYPE_UE;
570 break;
571 case ACA_ERROR_TYPE_CE:
572 case ACA_ERROR_TYPE_DEFERRED:
573 smu_type = ACA_SMU_TYPE_CE;
574 break;
575 default:
576 return -EINVAL;
577 }
578
579 /* update aca bank to aca source error_cache first */
580 ret = aca_banks_update(adev, smu_type, handler_aca_log_bank_error, qctx, NULL);
581 if (ret)
582 return ret;
583
584 /* DEs may contain in CEs or UEs */
585 if (type != ACA_ERROR_TYPE_DEFERRED)
586 aca_log_aca_error(handle, ACA_ERROR_TYPE_DEFERRED, err_data);
587
588 return aca_log_aca_error(handle, type, err_data);
589}
590
591static bool aca_handle_is_valid(struct aca_handle *handle)
592{
593 if (!handle->mask || !list_empty(&handle->node))
594 return false;
595
596 return true;
597}
598
599int amdgpu_aca_get_error_data(struct amdgpu_device *adev, struct aca_handle *handle,
600 enum aca_error_type type, struct ras_err_data *err_data,
601 struct ras_query_context *qctx)
602{
603 if (!handle || !err_data)
604 return -EINVAL;
605
606 if (aca_handle_is_valid(handle))
607 return -EOPNOTSUPP;
608
609 if ((type < 0) || (!(BIT(type) & handle->mask)))
610 return 0;
611
612 return __aca_get_error_data(adev, handle, type, err_data, qctx);
613}
614
615static void aca_error_init(struct aca_error *aerr, enum aca_error_type type)
616{
617 mutex_init(&aerr->lock);
618 INIT_LIST_HEAD(&aerr->list);
619 aerr->type = type;
620 aerr->nr_errors = 0;
621}
622
623static void aca_init_error_cache(struct aca_handle *handle)
624{
625 struct aca_error_cache *error_cache = &handle->error_cache;
626 int type;
627
628 for (type = ACA_ERROR_TYPE_UE; type < ACA_ERROR_TYPE_COUNT; type++)
629 aca_error_init(&error_cache->errors[type], type);
630}
631
632static void aca_error_fini(struct aca_error *aerr)
633{
634 struct aca_bank_error *bank_error, *tmp;
635
636 mutex_lock(&aerr->lock);
637 if (list_empty(&aerr->list))
638 goto out_unlock;
639
640 list_for_each_entry_safe(bank_error, tmp, &aerr->list, node)
641 aca_bank_error_remove(aerr, bank_error);
642
643out_unlock:
644 mutex_unlock(&aerr->lock);
645 mutex_destroy(&aerr->lock);
646}
647
648static void aca_fini_error_cache(struct aca_handle *handle)
649{
650 struct aca_error_cache *error_cache = &handle->error_cache;
651 int type;
652
653 for (type = ACA_ERROR_TYPE_UE; type < ACA_ERROR_TYPE_COUNT; type++)
654 aca_error_fini(&error_cache->errors[type]);
655}
656
657static int add_aca_handle(struct amdgpu_device *adev, struct aca_handle_manager *mgr, struct aca_handle *handle,
658 const char *name, const struct aca_info *ras_info, void *data)
659{
660 memset(handle, 0, sizeof(*handle));
661
662 handle->adev = adev;
663 handle->mgr = mgr;
664 handle->name = name;
665 handle->hwip = ras_info->hwip;
666 handle->mask = ras_info->mask;
667 handle->bank_ops = ras_info->bank_ops;
668 handle->data = data;
669 aca_init_error_cache(handle);
670
671 INIT_LIST_HEAD(&handle->node);
672 list_add_tail(&handle->node, &mgr->list);
673 mgr->nr_handles++;
674
675 return 0;
676}
677
678static ssize_t aca_sysfs_read(struct device *dev,
679 struct device_attribute *attr, char *buf)
680{
681 struct aca_handle *handle = container_of(attr, struct aca_handle, aca_attr);
682
683 /* NOTE: the aca cache will be auto cleared once read,
684 * So the driver should unify the query entry point, forward request to ras query interface directly */
685 return amdgpu_ras_aca_sysfs_read(dev, attr, handle, buf, handle->data);
686}
687
688static int add_aca_sysfs(struct amdgpu_device *adev, struct aca_handle *handle)
689{
690 struct device_attribute *aca_attr = &handle->aca_attr;
691
692 snprintf(handle->attr_name, sizeof(handle->attr_name) - 1, "aca_%s", handle->name);
693 aca_attr->show = aca_sysfs_read;
694 aca_attr->attr.name = handle->attr_name;
695 aca_attr->attr.mode = S_IRUGO;
696 sysfs_attr_init(&aca_attr->attr);
697
698 return sysfs_add_file_to_group(&adev->dev->kobj,
699 &aca_attr->attr,
700 "ras");
701}
702
703int amdgpu_aca_add_handle(struct amdgpu_device *adev, struct aca_handle *handle,
704 const char *name, const struct aca_info *ras_info, void *data)
705{
706 struct amdgpu_aca *aca = &adev->aca;
707 int ret;
708
709 if (!amdgpu_aca_is_enabled(adev))
710 return 0;
711
712 ret = add_aca_handle(adev, &aca->mgr, handle, name, ras_info, data);
713 if (ret)
714 return ret;
715
716 return add_aca_sysfs(adev, handle);
717}
718
719static void remove_aca_handle(struct aca_handle *handle)
720{
721 struct aca_handle_manager *mgr = handle->mgr;
722
723 aca_fini_error_cache(handle);
724 list_del(&handle->node);
725 mgr->nr_handles--;
726}
727
728static void remove_aca_sysfs(struct aca_handle *handle)
729{
730 struct amdgpu_device *adev = handle->adev;
731 struct device_attribute *aca_attr = &handle->aca_attr;
732
733 if (adev->dev->kobj.sd)
734 sysfs_remove_file_from_group(&adev->dev->kobj,
735 &aca_attr->attr,
736 "ras");
737}
738
739void amdgpu_aca_remove_handle(struct aca_handle *handle)
740{
741 if (!handle || list_empty(&handle->node))
742 return;
743
744 remove_aca_sysfs(handle);
745 remove_aca_handle(handle);
746}
747
748static int aca_manager_init(struct aca_handle_manager *mgr)
749{
750 INIT_LIST_HEAD(&mgr->list);
751 mgr->nr_handles = 0;
752
753 return 0;
754}
755
756static void aca_manager_fini(struct aca_handle_manager *mgr)
757{
758 struct aca_handle *handle, *tmp;
759
760 if (list_empty(&mgr->list))
761 return;
762
763 list_for_each_entry_safe(handle, tmp, &mgr->list, node)
764 amdgpu_aca_remove_handle(handle);
765}
766
767bool amdgpu_aca_is_enabled(struct amdgpu_device *adev)
768{
769 return (adev->aca.is_enabled ||
770 adev->debug_enable_ras_aca);
771}
772
773int amdgpu_aca_init(struct amdgpu_device *adev)
774{
775 struct amdgpu_aca *aca = &adev->aca;
776 int ret;
777
778 atomic_set(&aca->ue_update_flag, 0);
779
780 ret = aca_manager_init(&aca->mgr);
781 if (ret)
782 return ret;
783
784 return 0;
785}
786
787void amdgpu_aca_fini(struct amdgpu_device *adev)
788{
789 struct amdgpu_aca *aca = &adev->aca;
790
791 aca_manager_fini(&aca->mgr);
792
793 atomic_set(&aca->ue_update_flag, 0);
794}
795
796int amdgpu_aca_reset(struct amdgpu_device *adev)
797{
798 struct amdgpu_aca *aca = &adev->aca;
799
800 atomic_set(&aca->ue_update_flag, 0);
801
802 return 0;
803}
804
805void amdgpu_aca_set_smu_funcs(struct amdgpu_device *adev, const struct aca_smu_funcs *smu_funcs)
806{
807 struct amdgpu_aca *aca = &adev->aca;
808
809 WARN_ON(aca->smu_funcs);
810 aca->smu_funcs = smu_funcs;
811}
812
813int aca_bank_info_decode(struct aca_bank *bank, struct aca_bank_info *info)
814{
815 u64 ipid;
816 u32 instidhi, instidlo;
817
818 if (!bank || !info)
819 return -EINVAL;
820
821 ipid = bank->regs[ACA_REG_IDX_IPID];
822 info->hwid = ACA_REG__IPID__HARDWAREID(ipid);
823 info->mcatype = ACA_REG__IPID__MCATYPE(ipid);
824 /*
825 * Unfied DieID Format: SAASS. A:AID, S:Socket.
826 * Unfied DieID[4:4] = InstanceId[0:0]
827 * Unfied DieID[0:3] = InstanceIdHi[0:3]
828 */
829 instidhi = ACA_REG__IPID__INSTANCEIDHI(ipid);
830 instidlo = ACA_REG__IPID__INSTANCEIDLO(ipid);
831 info->die_id = ((instidhi >> 2) & 0x03);
832 info->socket_id = ((instidlo & 0x1) << 2) | (instidhi & 0x03);
833
834 return 0;
835}
836
837static int aca_bank_get_error_code(struct amdgpu_device *adev, struct aca_bank *bank)
838{
839 struct amdgpu_aca *aca = &adev->aca;
840 const struct aca_smu_funcs *smu_funcs = aca->smu_funcs;
841
842 if (!smu_funcs || !smu_funcs->parse_error_code)
843 return -EOPNOTSUPP;
844
845 return smu_funcs->parse_error_code(adev, bank);
846}
847
848int aca_bank_check_error_codes(struct amdgpu_device *adev, struct aca_bank *bank, int *err_codes, int size)
849{
850 int i, error_code;
851
852 if (!bank || !err_codes)
853 return -EINVAL;
854
855 error_code = aca_bank_get_error_code(adev, bank);
856 if (error_code < 0)
857 return error_code;
858
859 for (i = 0; i < size; i++) {
860 if (err_codes[i] == error_code)
861 return 0;
862 }
863
864 return -EINVAL;
865}
866
867int amdgpu_aca_smu_set_debug_mode(struct amdgpu_device *adev, bool en)
868{
869 struct amdgpu_aca *aca = &adev->aca;
870 const struct aca_smu_funcs *smu_funcs = aca->smu_funcs;
871
872 if (!smu_funcs || !smu_funcs->set_debug_mode)
873 return -EOPNOTSUPP;
874
875 return smu_funcs->set_debug_mode(adev, en);
876}
877
878#if defined(CONFIG_DEBUG_FS)
879static int amdgpu_aca_smu_debug_mode_set(void *data, u64 val)
880{
881 struct amdgpu_device *adev = (struct amdgpu_device *)data;
882 int ret;
883
884 ret = amdgpu_ras_set_aca_debug_mode(adev, val ? true : false);
885 if (ret)
886 return ret;
887
888 dev_info(adev->dev, "amdgpu set smu aca debug mode %s success\n", val ? "on" : "off");
889
890 return 0;
891}
892
893static void aca_dump_entry(struct seq_file *m, struct aca_bank *bank, enum aca_smu_type type, int idx)
894{
895 struct aca_bank_info info;
896 int i, ret;
897
898 ret = aca_bank_info_decode(bank, &info);
899 if (ret)
900 return;
901
902 seq_printf(m, "aca entry[%d].type: %s\n", idx, type == ACA_SMU_TYPE_UE ? "UE" : "CE");
903 seq_printf(m, "aca entry[%d].info: socketid:%d aid:%d hwid:0x%03x mcatype:0x%04x\n",
904 idx, info.socket_id, info.die_id, info.hwid, info.mcatype);
905
906 for (i = 0; i < ARRAY_SIZE(aca_regs); i++)
907 seq_printf(m, "aca entry[%d].regs[%d]: 0x%016llx\n", idx, aca_regs[i].reg_idx, bank->regs[aca_regs[i].reg_idx]);
908}
909
910struct aca_dump_context {
911 struct seq_file *m;
912 int idx;
913};
914
915static int handler_aca_bank_dump(struct aca_handle *handle, struct aca_bank *bank,
916 enum aca_smu_type type, void *data)
917{
918 struct aca_dump_context *ctx = (struct aca_dump_context *)data;
919
920 aca_dump_entry(ctx->m, bank, type, ctx->idx++);
921
922 return handler_aca_log_bank_error(handle, bank, type, NULL);
923}
924
925static int aca_dump_show(struct seq_file *m, enum aca_smu_type type)
926{
927 struct amdgpu_device *adev = (struct amdgpu_device *)m->private;
928 struct aca_dump_context context = {
929 .m = m,
930 .idx = 0,
931 };
932
933 return aca_banks_update(adev, type, handler_aca_bank_dump, NULL, (void *)&context);
934}
935
936static int aca_dump_ce_show(struct seq_file *m, void *unused)
937{
938 return aca_dump_show(m, ACA_SMU_TYPE_CE);
939}
940
941static int aca_dump_ce_open(struct inode *inode, struct file *file)
942{
943 return single_open(file, aca_dump_ce_show, inode->i_private);
944}
945
946static const struct file_operations aca_ce_dump_debug_fops = {
947 .owner = THIS_MODULE,
948 .open = aca_dump_ce_open,
949 .read = seq_read,
950 .llseek = seq_lseek,
951 .release = single_release,
952};
953
954static int aca_dump_ue_show(struct seq_file *m, void *unused)
955{
956 return aca_dump_show(m, ACA_SMU_TYPE_UE);
957}
958
959static int aca_dump_ue_open(struct inode *inode, struct file *file)
960{
961 return single_open(file, aca_dump_ue_show, inode->i_private);
962}
963
964static const struct file_operations aca_ue_dump_debug_fops = {
965 .owner = THIS_MODULE,
966 .open = aca_dump_ue_open,
967 .read = seq_read,
968 .llseek = seq_lseek,
969 .release = single_release,
970};
971
972DEFINE_DEBUGFS_ATTRIBUTE(aca_debug_mode_fops, NULL, amdgpu_aca_smu_debug_mode_set, "%llu\n");
973#endif
974
975void amdgpu_aca_smu_debugfs_init(struct amdgpu_device *adev, struct dentry *root)
976{
977#if defined(CONFIG_DEBUG_FS)
978 if (!root)
979 return;
980
981 debugfs_create_file("aca_debug_mode", 0200, root, adev, &aca_debug_mode_fops);
982 debugfs_create_file("aca_ue_dump", 0400, root, adev, &aca_ue_dump_debug_fops);
983 debugfs_create_file("aca_ce_dump", 0400, root, adev, &aca_ce_dump_debug_fops);
984#endif
985}
drivers/gpu/drm/amd/amdgpu/amdgpu_aca.h+0−232
@@ -1,232 +0,0 @@
1/*
2 * Copyright 2023 Advanced Micro Devices, Inc.
3 *
4 * Permission is hereby granted, free of charge, to any person obtaining a
5 * copy of this software and associated documentation files (the "Software"),
6 * to deal in the Software without restriction, including without limitation
7 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8 * and/or sell copies of the Software, and to permit persons to whom the
9 * Software is furnished to do so, subject to the following conditions:
10 *
11 * The above copyright notice and this permission notice shall be included in
12 * all copies or substantial portions of the Software.
13 *
14 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
17 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
18 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
19 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
20 * OTHER DEALINGS IN THE SOFTWARE.
21 *
22 */
23
24#ifndef __AMDGPU_ACA_H__
25#define __AMDGPU_ACA_H__
26
27#include <linux/list.h>
28
29struct ras_err_data;
30struct ras_query_context;
31
32#define ACA_MAX_REGS_COUNT (16)
33
34#define ACA_REG_FIELD(x, h, l) (((x) & GENMASK_ULL(h, l)) >> l)
35#define ACA_REG__STATUS__VAL(x) ACA_REG_FIELD(x, 63, 63)
36#define ACA_REG__STATUS__OVERFLOW(x) ACA_REG_FIELD(x, 62, 62)
37#define ACA_REG__STATUS__UC(x) ACA_REG_FIELD(x, 61, 61)
38#define ACA_REG__STATUS__EN(x) ACA_REG_FIELD(x, 60, 60)
39#define ACA_REG__STATUS__MISCV(x) ACA_REG_FIELD(x, 59, 59)
40#define ACA_REG__STATUS__ADDRV(x) ACA_REG_FIELD(x, 58, 58)
41#define ACA_REG__STATUS__PCC(x) ACA_REG_FIELD(x, 57, 57)
42#define ACA_REG__STATUS__ERRCOREIDVAL(x) ACA_REG_FIELD(x, 56, 56)
43#define ACA_REG__STATUS__TCC(x) ACA_REG_FIELD(x, 55, 55)
44#define ACA_REG__STATUS__SYNDV(x) ACA_REG_FIELD(x, 53, 53)
45#define ACA_REG__STATUS__CECC(x) ACA_REG_FIELD(x, 46, 46)
46#define ACA_REG__STATUS__UECC(x) ACA_REG_FIELD(x, 45, 45)
47#define ACA_REG__STATUS__DEFERRED(x) ACA_REG_FIELD(x, 44, 44)
48#define ACA_REG__STATUS__POISON(x) ACA_REG_FIELD(x, 43, 43)
49#define ACA_REG__STATUS__SCRUB(x) ACA_REG_FIELD(x, 40, 40)
50#define ACA_REG__STATUS__ERRCOREID(x) ACA_REG_FIELD(x, 37, 32)
51#define ACA_REG__STATUS__ADDRLSB(x) ACA_REG_FIELD(x, 29, 24)
52#define ACA_REG__STATUS__ERRORCODEEXT(x) ACA_REG_FIELD(x, 21, 16)
53#define ACA_REG__STATUS__ERRORCODE(x) ACA_REG_FIELD(x, 15, 0)
54
55#define ACA_REG__IPID__MCATYPE(x) ACA_REG_FIELD(x, 63, 48)
56#define ACA_REG__IPID__INSTANCEIDHI(x) ACA_REG_FIELD(x, 47, 44)
57#define ACA_REG__IPID__HARDWAREID(x) ACA_REG_FIELD(x, 43, 32)
58#define ACA_REG__IPID__INSTANCEIDLO(x) ACA_REG_FIELD(x, 31, 0)
59
60#define ACA_REG__MISC0__VALID(x) ACA_REG_FIELD(x, 63, 63)
61#define ACA_REG__MISC0__OVRFLW(x) ACA_REG_FIELD(x, 48, 48)
62#define ACA_REG__MISC0__ERRCNT(x) ACA_REG_FIELD(x, 43, 32)
63
64#define ACA_REG__SYND__ERRORINFORMATION(x) ACA_REG_FIELD(x, 17, 0)
65
66/* NOTE: The following codes refers to the smu header file */
67#define ACA_EXTERROR_CODE_CE 0x3a
68#define ACA_EXTERROR_CODE_FAULT 0x3b
69
70#define ACA_ERROR_UE_MASK BIT_MASK(ACA_ERROR_TYPE_UE)
71#define ACA_ERROR_CE_MASK BIT_MASK(ACA_ERROR_TYPE_CE)
72#define ACA_ERROR_DEFERRED_MASK BIT_MASK(ACA_ERROR_TYPE_DEFERRED)
73
74#define mmSMNAID_AID0_MCA_SMU 0x03b30400 /* SMN AID AID0 */
75#define mmSMNAID_XCD0_MCA_SMU 0x36430400 /* SMN AID XCD0 */
76#define mmSMNAID_XCD1_MCA_SMU 0x38430400 /* SMN AID XCD1 */
77#define mmSMNXCD_XCD0_MCA_SMU 0x40430400 /* SMN XCD XCD0 */
78
79#define ACA_BANK_ERR_IS_DEFFERED(bank) \
80 (ACA_REG__STATUS__POISON((bank)->regs[ACA_REG_IDX_STATUS]) || \
81 ACA_REG__STATUS__DEFERRED((bank)->regs[ACA_REG_IDX_STATUS]))
82
83enum aca_reg_idx {
84 ACA_REG_IDX_CTL = 0,
85 ACA_REG_IDX_STATUS = 1,
86 ACA_REG_IDX_ADDR = 2,
87 ACA_REG_IDX_MISC0 = 3,
88 ACA_REG_IDX_CONFIG = 4,
89 ACA_REG_IDX_IPID = 5,
90 ACA_REG_IDX_SYND = 6,
91 ACA_REG_IDX_DESTAT = 8,
92 ACA_REG_IDX_DEADDR = 9,
93 ACA_REG_IDX_CTL_MASK = 10,
94 ACA_REG_IDX_COUNT = 16,
95};
96
97enum aca_hwip_type {
98 ACA_HWIP_TYPE_UNKNOW = -1,
99 ACA_HWIP_TYPE_PSP = 0,
100 ACA_HWIP_TYPE_UMC,
101 ACA_HWIP_TYPE_SMU,
102 ACA_HWIP_TYPE_PCS_XGMI,
103 ACA_HWIP_TYPE_COUNT,
104};
105
106enum aca_error_type {
107 ACA_ERROR_TYPE_INVALID = -1,
108 ACA_ERROR_TYPE_UE = 0,
109 ACA_ERROR_TYPE_CE,
110 ACA_ERROR_TYPE_DEFERRED,
111 ACA_ERROR_TYPE_COUNT
112};
113
114enum aca_smu_type {
115 ACA_SMU_TYPE_INVALID = -1,
116 ACA_SMU_TYPE_UE = 0,
117 ACA_SMU_TYPE_CE,
118 ACA_SMU_TYPE_COUNT,
119};
120
121struct aca_hwip {
122 int hwid;
123 int mcatype;
124};
125
126struct aca_bank {
127 enum aca_error_type aca_err_type;
128 enum aca_smu_type smu_err_type;
129 u64 regs[ACA_MAX_REGS_COUNT];
130};
131
132struct aca_bank_node {
133 struct aca_bank bank;
134 struct list_head node;
135};
136
137struct aca_banks {
138 int nr_banks;
139 struct list_head list;
140};
141
142struct aca_bank_info {
143 int die_id;
144 int socket_id;
145 int hwid;
146 int mcatype;
147};
148
149struct aca_bank_error {
150 struct list_head node;
151 struct aca_bank_info info;
152 u64 count;
153};
154
155struct aca_error {
156 struct list_head list;
157 struct mutex lock;
158 enum aca_error_type type;
159 int nr_errors;
160};
161
162struct aca_handle_manager {
163 struct list_head list;
164 int nr_handles;
165};
166
167struct aca_error_cache {
168 struct aca_error errors[ACA_ERROR_TYPE_COUNT];
169};
170
171struct aca_handle {
172 struct list_head node;
173 enum aca_hwip_type hwip;
174 struct amdgpu_device *adev;
175 struct aca_handle_manager *mgr;
176 struct aca_error_cache error_cache;
177 const struct aca_bank_ops *bank_ops;
178 struct device_attribute aca_attr;
179 char attr_name[64];
180 const char *name;
181 u32 mask;
182 void *data;
183};
184
185struct aca_bank_ops {
186 int (*aca_bank_parser)(struct aca_handle *handle, struct aca_bank *bank, enum aca_smu_type type, void *data);
187 bool (*aca_bank_is_valid)(struct aca_handle *handle, struct aca_bank *bank, enum aca_smu_type type,
188 void *data);
189};
190
191struct aca_smu_funcs {
192 int max_ue_bank_count;
193 int max_ce_bank_count;
194 int (*set_debug_mode)(struct amdgpu_device *adev, bool enable);
195 int (*get_valid_aca_count)(struct amdgpu_device *adev, enum aca_smu_type type, u32 *count);
196 int (*get_valid_aca_bank)(struct amdgpu_device *adev, enum aca_smu_type type, int idx, struct aca_bank *bank);
197 int (*parse_error_code)(struct amdgpu_device *adev, struct aca_bank *bank);
198};
199
200struct amdgpu_aca {
201 struct aca_handle_manager mgr;
202 const struct aca_smu_funcs *smu_funcs;
203 atomic_t ue_update_flag;
204 bool is_enabled;
205};
206
207struct aca_info {
208 enum aca_hwip_type hwip;
209 const struct aca_bank_ops *bank_ops;
210 u32 mask;
211};
212
213int amdgpu_aca_init(struct amdgpu_device *adev);
214void amdgpu_aca_fini(struct amdgpu_device *adev);
215int amdgpu_aca_reset(struct amdgpu_device *adev);
216void amdgpu_aca_set_smu_funcs(struct amdgpu_device *adev, const struct aca_smu_funcs *smu_funcs);
217bool amdgpu_aca_is_enabled(struct amdgpu_device *adev);
218
219int aca_bank_info_decode(struct aca_bank *bank, struct aca_bank_info *info);
220int aca_bank_check_error_codes(struct amdgpu_device *adev, struct aca_bank *bank, int *err_codes, int size);
221
222int amdgpu_aca_add_handle(struct amdgpu_device *adev, struct aca_handle *handle,
223 const char *name, const struct aca_info *aca_info, void *data);
224void amdgpu_aca_remove_handle(struct aca_handle *handle);
225int amdgpu_aca_get_error_data(struct amdgpu_device *adev, struct aca_handle *handle,
226 enum aca_error_type type, struct ras_err_data *err_data,
227 struct ras_query_context *qctx);
228int amdgpu_aca_smu_set_debug_mode(struct amdgpu_device *adev, bool en);
229void amdgpu_aca_smu_debugfs_init(struct amdgpu_device *adev, struct dentry *root);
230int aca_error_cache_log_bank_error(struct aca_handle *handle, struct aca_bank_info *info,
231 enum aca_error_type type, u64 count);
232#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c+34−34
@@ -140,13 +140,15 @@ static struct amdgpu_acpi_priv { · 139 unmodified lines
140 * @atif: atif structure140 * @atif: atif structure
141 * @function: the ATIF function to execute141 * @function: the ATIF function to execute
142 * @params: ATIF function params142 * @params: ATIF function params
143 * @min_size: minimum size of the expected output buffer in bytes
143 *144 *
144 * Executes the requested ATIF function (all asics).145 * Executes the requested ATIF function (all asics).
145 * Returns a pointer to the acpi output buffer.146 * Returns a pointer to the acpi output buffer.
146 */147 */
147static union acpi_object *amdgpu_atif_call(struct amdgpu_atif *atif,148static union acpi_object *amdgpu_atif_call(struct amdgpu_atif *atif,
148 int function,149 int function,
149 struct acpi_buffer *params)150 struct acpi_buffer *params,
151 size_t min_size)
150{152{
151 acpi_status status;153 acpi_status status;
152 union acpi_object *obj;154 union acpi_object *obj;
@@ -189,6 +191,28 @@ static union acpi_object *amdgpu_atif_call(struct amdgpu_atif *atif, · 36 unmodified lines
189 return NULL;191 return NULL;
190 }192 }
191 193
194 if (obj->buffer.length < sizeof(u16)) {
195 DRM_DEBUG_DRIVER("ATIF buffer too small to hold size field: %u\n",
196 obj->buffer.length);
197 kfree(obj);
198 return NULL;
199 }
200
201 if (obj->buffer.length < *(u16 *)obj->buffer.pointer) {
202 DRM_DEBUG_DRIVER("ATIF buffer length mismatch: reported %u, actual %u\n",
203 *(u16 *)obj->buffer.pointer,
204 obj->buffer.length);
205 kfree(obj);
206 return NULL;
207 }
208
209 if (*(u16 *)obj->buffer.pointer < min_size) {
210 DRM_DEBUG_DRIVER("ATIF buffer too small: expected %zu, got %u\n",
211 min_size, *(u16 *)obj->buffer.pointer);
212 kfree(obj);
213 return NULL;
214 }
215
192 return obj;216 return obj;
193}217}
194 218
@@ -251,19 +275,14 @@ int amdgpu_atif_verify_interface(struct amdgpu_atif *atif) · 56 unmodified lines
251 size_t size;275 size_t size;
252 int err = 0;276 int err = 0;
253 277
254 info = amdgpu_atif_call(atif, ATIF_FUNCTION_VERIFY_INTERFACE, NULL);278 info = amdgpu_atif_call(atif, ATIF_FUNCTION_VERIFY_INTERFACE, NULL,
279 sizeof(output));
255 if (!info)280 if (!info)
256 return -EIO;281 return -EIO;
257 282
258 memset(&output, 0, sizeof(output));283 memset(&output, 0, sizeof(output));
259 284
260 size = *(u16 *) info->buffer.pointer;285 size = min(sizeof(output), (size_t)*(u16 *)info->buffer.pointer);
261 if (size < 12) {
262 DRM_INFO("ATIF buffer is too small: %zu\n", size);
263 err = -EINVAL;
264 goto out;
265 }
266 size = min(sizeof(output), size);
267 286
268 memcpy(&output, info->buffer.pointer, size);287 memcpy(&output, info->buffer.pointer, size);
269 288
@@ -273,7 +292,6 @@ int amdgpu_atif_verify_interface(struct amdgpu_atif *atif) · 3 unmodified lines
273 amdgpu_atif_parse_notification(&atif->notifications, output.notification_mask);292 amdgpu_atif_parse_notification(&atif->notifications, output.notification_mask);
274 amdgpu_atif_parse_functions(&atif->functions, output.function_bits);293 amdgpu_atif_parse_functions(&atif->functions, output.function_bits);
275 294
276out:
277 kfree(info);295 kfree(info);
278 return err;296 return err;
279}297}
@@ -299,20 +317,14 @@ int amdgpu_atif_get_notification_params(struct amdgpu_atif *atif) · 19 unmodified lines
299 int err = 0;317 int err = 0;
300 318
301 info = amdgpu_atif_call(atif, ATIF_FUNCTION_GET_SYSTEM_PARAMETERS,319 info = amdgpu_atif_call(atif, ATIF_FUNCTION_GET_SYSTEM_PARAMETERS,
302 NULL);320 NULL, offsetof(struct atif_system_params, command_code));
303 if (!info) {321 if (!info) {
304 err = -EIO;322 err = -EIO;
305 goto out;323 goto out;
306 }324 }
307 325
308 size = *(u16 *) info->buffer.pointer;
309 if (size < 10) {
310 err = -EINVAL;
311 goto out;
312 }
313
314 memset(&params, 0, sizeof(params));326 memset(&params, 0, sizeof(params));
315 size = min(sizeof(params), size);327 size = min(sizeof(params), (size_t)*(u16 *)info->buffer.pointer);
316 memcpy(&params, info->buffer.pointer, size);328 memcpy(&params, info->buffer.pointer, size);
317 329
318 DRM_DEBUG_DRIVER("SYSTEM_PARAMS: mask = %#x, flags = %#x\n",330 DRM_DEBUG_DRIVER("SYSTEM_PARAMS: mask = %#x, flags = %#x\n",
@@ -376,20 +388,14 @@ int amdgpu_atif_query_backlight_caps(struct amdgpu_atif *atif) · 57 unmodified lines
376 388
377 info = amdgpu_atif_call(atif,389 info = amdgpu_atif_call(atif,
378 ATIF_FUNCTION_QUERY_BRIGHTNESS_TRANSFER_CHARACTERISTICS,390 ATIF_FUNCTION_QUERY_BRIGHTNESS_TRANSFER_CHARACTERISTICS,
379 &params);391 &params, offsetof(struct atif_qbtc_output, data_points));
380 if (!info) {392 if (!info) {
381 err = -EIO;393 err = -EIO;
382 goto out;394 goto out;
383 }395 }
384 396
385 size = *(u16 *) info->buffer.pointer;
386 if (size < 10) {
387 err = -EINVAL;
388 goto out;
389 }
390
391 memset(&characteristics, 0, sizeof(characteristics));397 memset(&characteristics, 0, sizeof(characteristics));
392 size = min(sizeof(characteristics), size);398 size = min(sizeof(characteristics), (size_t)*(u16 *)info->buffer.pointer);
393 memcpy(&characteristics, info->buffer.pointer, size);399 memcpy(&characteristics, info->buffer.pointer, size);
394 400
395 atif->backlight_caps.caps_valid = true;401 atif->backlight_caps.caps_valid = true;
@@ -427,24 +433,18 @@ static int amdgpu_atif_get_sbios_requests(struct amdgpu_atif *atif, · 31 unmodified lines
427 int count = 0;433 int count = 0;
428 434
429 info = amdgpu_atif_call(atif, ATIF_FUNCTION_GET_SYSTEM_BIOS_REQUESTS,435 info = amdgpu_atif_call(atif, ATIF_FUNCTION_GET_SYSTEM_BIOS_REQUESTS,
430 NULL);436 NULL, sizeof(*req));
431 if (!info)437 if (!info)
432 return -EIO;438 return -EIO;
433 439
434 size = *(u16 *)info->buffer.pointer;
435 if (size < 0xd) {
436 count = -EINVAL;
437 goto out;
438 }
439 memset(req, 0, sizeof(*req));440 memset(req, 0, sizeof(*req));
440 441
441 size = min(sizeof(*req), size);442 size = min(sizeof(*req), (size_t)*(u16 *)info->buffer.pointer);
442 memcpy(req, info->buffer.pointer, size);443 memcpy(req, info->buffer.pointer, size);
443 DRM_DEBUG_DRIVER("SBIOS pending requests: %#x\n", req->pending);444 DRM_DEBUG_DRIVER("SBIOS pending requests: %#x\n", req->pending);
444 445
445 count = hweight32(req->pending);446 count = hweight32(req->pending);
446 447
447out:
448 kfree(info);448 kfree(info);
449 return count;449 return count;
450}450}
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c+14−0
@@ -957,3 +957,17 @@ int amdgpu_amdkfd_config_sq_perfmon(struct amdgpu_device *adev, uint32_t xcp_id, · 956 unmodified lines
957 957
958 return r;958 return r;
959}959}
960
961/* Reset an MES queue */
962int amdgpu_amdkfd_reset_mes_queue(struct amdgpu_device *adev,
963 uint32_t node_id,
964 int queue_type,
965 int pipe, int queue,
966 unsigned int db)
967{
968 if (!adev->kfd.init_complete)
969 return 0;
970
971 return kgd2kfd_reset_mes_queue(adev->kfd.dev, node_id, queue_type,
972 pipe, queue, db);
973}
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h+24−4
@@ -210,6 +210,7 @@ int amdgpu_amdkfd_evict_userptr(struct mmu_interval_notifier *mni, · 209 unmodified lines
210int amdgpu_amdkfd_bo_validate_and_fence(struct amdgpu_bo *bo,210int amdgpu_amdkfd_bo_validate_and_fence(struct amdgpu_bo *bo,
211 uint32_t domain,211 uint32_t domain,
212 struct dma_fence *fence);212 struct dma_fence *fence);
213int amdgpu_amdkfd_set_sigbus_delay(struct task_struct *task, u32 ms);
213#else214#else
214static inline215static inline
215bool amdkfd_fence_check_mm(struct dma_fence *f, struct mm_struct *mm)216bool amdkfd_fence_check_mm(struct dma_fence *f, struct mm_struct *mm)
@@ -241,6 +242,11 @@ int amdgpu_amdkfd_bo_validate_and_fence(struct amdgpu_bo *bo, · 25 unmodified lines
241{242{
242 return 0;243 return 0;
243}244}
245static inline
246int amdgpu_amdkfd_set_sigbus_delay(struct task_struct *task, u32 ms)
247{
248 return -EOPNOTSUPP;
249}
244#endif250#endif
245/* Shared API */251/* Shared API */
246int amdgpu_amdkfd_alloc_kernel_mem(struct amdgpu_device *adev, size_t size,252int amdgpu_amdkfd_alloc_kernel_mem(struct amdgpu_device *adev, size_t size,
@@ -275,7 +281,11 @@ int amdgpu_amdkfd_stop_sched(struct amdgpu_device *adev, uint32_t node_id); · 28 unmodified lines
275int amdgpu_amdkfd_config_sq_perfmon(struct amdgpu_device *adev, uint32_t xcp_id,281int amdgpu_amdkfd_config_sq_perfmon(struct amdgpu_device *adev, uint32_t xcp_id,
276 bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable);282 bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable);
277bool amdgpu_amdkfd_compute_active(struct amdgpu_device *adev, uint32_t node_id);283bool amdgpu_amdkfd_compute_active(struct amdgpu_device *adev, uint32_t node_id);
278 284int amdgpu_amdkfd_reset_mes_queue(struct amdgpu_device *adev,
285 uint32_t node_id,
286 int queue_type,
287 int pipe, int queue,
288 unsigned int db);
279 289
280/* Read user wptr from a specified user address space with page fault290/* Read user wptr from a specified user address space with page fault
281 * disabled. The memory must be pinned and mapped to the hardware when291 * disabled. The memory must be pinned and mapped to the hardware when
@@ -326,9 +336,9 @@ int amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu( · 44 unmodified lines
326int amdgpu_amdkfd_gpuvm_dmaunmap_mem(struct kgd_mem *mem, void *drm_priv);336int amdgpu_amdkfd_gpuvm_dmaunmap_mem(struct kgd_mem *mem, void *drm_priv);
327int amdgpu_amdkfd_gpuvm_sync_memory(337int amdgpu_amdkfd_gpuvm_sync_memory(
328 struct amdgpu_device *adev, struct kgd_mem *mem, bool intr);338 struct amdgpu_device *adev, struct kgd_mem *mem, bool intr);
329int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct kgd_mem *mem,339int amdgpu_amdkfd_gpuvm_map_bo_to_kernel(struct kgd_mem *mem, void **kptr,
330 void **kptr, uint64_t *size);340 u64 *size, u32 domain);
331void amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(struct kgd_mem *mem);341void amdgpu_amdkfd_gpuvm_unmap_bo_from_kernel(struct kgd_mem *mem);
332 342
333int amdgpu_amdkfd_map_gtt_bo_to_gart(struct amdgpu_bo *bo, struct amdgpu_bo **bo_gart);343int amdgpu_amdkfd_map_gtt_bo_to_gart(struct amdgpu_bo *bo, struct amdgpu_bo **bo_gart);
334 344
@@ -446,6 +456,9 @@ bool kgd2kfd_vmfault_fast_path(struct amdgpu_device *adev, struct amdgpu_iv_entr · 111 unmodified lines
446 bool retry_fault);456 bool retry_fault);
447void kgd2kfd_lock_kfd(void);457void kgd2kfd_lock_kfd(void);
448void kgd2kfd_teardown_processes(struct amdgpu_device *adev);458void kgd2kfd_teardown_processes(struct amdgpu_device *adev);
459int kgd2kfd_reset_mes_queue(struct kfd_dev *kfd, uint32_t node_id,
460 int queue_type, int pipe, int queue,
461 unsigned int db);
449 462
450#else463#else
451static inline int kgd2kfd_init(void)464static inline int kgd2kfd_init(void)
@@ -576,5 +589,12 @@ static inline void kgd2kfd_teardown_processes(struct amdgpu_device *adev) · 124 unmodified lines
576{589{
577}590}
578 591
592static inline int kgd2kfd_reset_mes_queue(struct kfd_dev *kfd, uint32_t node_id,
593 int queue_type, int pipe, int queue,
594 unsigned int db)
595{
596 return 0;
597}
598
579#endif599#endif
580#endif /* AMDGPU_AMDKFD_H_INCLUDED */600#endif /* AMDGPU_AMDKFD_H_INCLUDED */
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gc_9_4_3.c+62−1
@@ -530,6 +530,66 @@ static uint32_t kgd_v9_4_3_ptl_ctrl(struct amdgpu_device *adev, · 529 unmodified lines
530 ptl_state, fmt1, fmt2);530 ptl_state, fmt1, fmt2);
531}531}
532 532
533static int kgd_gfx_v9_4_3_hqd_sdma_get_counter(struct amdgpu_device *adev,
534 void *mqd, uint32_t num_sdma_queues_per_eng,
535 uint64_t *val)
536{
537 struct v9_sdma_mqd *m = get_sdma_mqd(mqd);
538 uint32_t sdma_rlc_reg_offset = 0;
539 uint32_t sdma_rlc_rb_cntl;
540 uint32_t engine_id, queue_id;
541 uint32_t engines = adev->sdma.num_instances;
542 uint32_t sdma_rlcx_rb_base, sdma_rlcx_rb_base_hi;
543 bool found = false;
544
545 if (!m)
546 return -EINVAL;
547
548 if (((amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) ||
549 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 4)) &&
550 adev->gfx.mec_fw_version < 194) ||
551 (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 5, 0) &&
552 adev->gfx.mec_fw_version < 44)) {
553 pr_warn_once("MEC FW doesn't support SDMA counter!\n");
554 return -EOPNOTSUPP;
555 }
556
557 /* SDMA doesn't support over-subscription, there must be
558 * a HQD associated with a MQD, so found must be true in
559 * the finding loop.
560 */
561 for (engine_id = 0; engine_id < engines && !found; engine_id++) {
562 for (queue_id = 0; queue_id < num_sdma_queues_per_eng; queue_id++) {
563 sdma_rlc_reg_offset = get_sdma_rlc_reg_offset(adev,
564 engine_id, queue_id);
565 sdma_rlcx_rb_base = RREG32(sdma_rlc_reg_offset +
566 regSDMA_RLC0_RB_BASE);
567 sdma_rlcx_rb_base_hi = RREG32(sdma_rlc_reg_offset +
568 regSDMA_RLC0_RB_BASE_HI);
569
570 if (m->sdmax_rlcx_rb_base == sdma_rlcx_rb_base &&
571 m->sdmax_rlcx_rb_base_hi == sdma_rlcx_rb_base_hi) {
572 found = true;
573 break;
574 }
575 }
576 }
577
578 sdma_rlc_rb_cntl = RREG32(sdma_rlc_reg_offset + regSDMA_RLC0_RB_CNTL);
579
580 /* Read sdma activity counter from utilization register
581 * if hw queue is enabled, otherwise read from MQD.
582 */
583 if (sdma_rlc_rb_cntl & SDMA_RLC0_RB_CNTL__RB_ENABLE_MASK)
584 *val = (uint64_t)RREG32(sdma_rlc_reg_offset + regSDMA_RLC0_UTILIZATION_HI) << 32 |
585 RREG32(sdma_rlc_reg_offset + regSDMA_RLC0_UTILIZATION_LO);
586 else
587 *val = (uint64_t)m->sdmax_rlcx_utilization_hi << 32 |
588 m->sdmax_rlcx_utilization_lo;
589
590 return 0;
591}
592
533const struct kfd2kgd_calls gc_9_4_3_kfd2kgd = {593const struct kfd2kgd_calls gc_9_4_3_kfd2kgd = {
534 .program_sh_mem_settings = kgd_gfx_v9_program_sh_mem_settings,594 .program_sh_mem_settings = kgd_gfx_v9_program_sh_mem_settings,
535 .set_pasid_vmid_mapping = kgd_gfx_v9_4_3_set_pasid_vmid_mapping,595 .set_pasid_vmid_mapping = kgd_gfx_v9_4_3_set_pasid_vmid_mapping,
@@ -566,5 +626,6 @@ const struct kfd2kgd_calls gc_9_4_3_kfd2kgd = { · 30 unmodified lines
566 .hqd_get_pq_addr = kgd_gfx_v9_hqd_get_pq_addr,626 .hqd_get_pq_addr = kgd_gfx_v9_hqd_get_pq_addr,
567 .hqd_reset = kgd_gfx_v9_hqd_reset,627 .hqd_reset = kgd_gfx_v9_hqd_reset,
568 .hqd_sdma_get_doorbell = kgd_gfx_v9_4_3_hqd_sdma_get_doorbell,628 .hqd_sdma_get_doorbell = kgd_gfx_v9_4_3_hqd_sdma_get_doorbell,
569 .ptl_ctrl = kgd_v9_4_3_ptl_ctrl629 .ptl_ctrl = kgd_v9_4_3_ptl_ctrl,
630 .hqd_sdma_get_counter = kgd_gfx_v9_4_3_hqd_sdma_get_counter
570};631};
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c+14−6
@@ -2271,11 +2271,14 @@ err_reserve_bo_failed: · 2270 unmodified lines
2271 return ret;2271 return ret;
2272}2272}
2273 2273
2274/** amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel() - Map a GTT BO for kernel CPU access2274/** amdgpu_amdkfd_gpuvm_map_bo_to_kernel() - Map GTT or VRAM BO for kernel CPU access
2275 *2275 *
2276 * @mem: Buffer object to be mapped for CPU access2276 * @mem: Buffer object to be mapped for CPU access
2277 * @kptr[out]: pointer in kernel CPU address space2277 * @kptr[out]: pointer in kernel CPU address space
2278 * @size[out]: size of the buffer2278 * @size[out]: size of the buffer
2279 * @domain[IN]: domain for pinning (AMDGPU_GEM_DOMAIN_GTT, AMDGPU_GEM_DOMAIN_VRAM,
2280 * or their combination to let the driver choose). CPU visibility is
2281 * automatically enforced by amdgpu_bo_pin()
2279 *2282 *
2280 * Pins the BO and maps it for kernel CPU access. The eviction fence is removed2283 * Pins the BO and maps it for kernel CPU access. The eviction fence is removed
2281 * from the BO, since pinned BOs cannot be evicted. The bo must remain on the2284 * from the BO, since pinned BOs cannot be evicted. The bo must remain on the
@@ -2284,8 +2287,8 @@ err_reserve_bo_failed: · 2 unmodified lines
2284 *2287 *
2285 * Return: 0 on success, error code on failure2288 * Return: 0 on success, error code on failure
2286 */2289 */
2287int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct kgd_mem *mem,2290int amdgpu_amdkfd_gpuvm_map_bo_to_kernel(struct kgd_mem *mem, void **kptr,
2288 void **kptr, uint64_t *size)2291 u64 *size, u32 domain)
2289{2292{
2290 int ret;2293 int ret;
2291 struct amdgpu_bo *bo = mem->bo;2294 struct amdgpu_bo *bo = mem->bo;
@@ -2295,6 +2298,11 @@ int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct kgd_mem *mem, · 3 unmodified lines
2295 return -EINVAL;2298 return -EINVAL;
2296 }2299 }
2297 2300
2301 if (!(domain & (AMDGPU_GEM_DOMAIN_GTT | AMDGPU_GEM_DOMAIN_VRAM))) {
2302 pr_debug("Invalid domain 0x%x for kernel mapping\n", domain);
2303 return -EINVAL;
2304 }
2305
2298 mutex_lock(&mem->process_info->lock);2306 mutex_lock(&mem->process_info->lock);
2299 2307
2300 ret = amdgpu_bo_reserve(bo, true);2308 ret = amdgpu_bo_reserve(bo, true);
@@ -2303,7 +2311,7 @@ int amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(struct kgd_mem *mem, · 2 unmodified lines
2303 goto bo_reserve_failed;2311 goto bo_reserve_failed;
2304 }2312 }
2305 2313
2306 ret = amdgpu_bo_pin(bo, AMDGPU_GEM_DOMAIN_GTT);2314 ret = amdgpu_bo_pin(bo, domain);
2307 if (ret) {2315 if (ret) {
2308 pr_err("Failed to pin bo. ret %d\n", ret);2316 pr_err("Failed to pin bo. ret %d\n", ret);
2309 goto pin_failed;2317 goto pin_failed;
@@ -2336,7 +2344,7 @@ bo_reserve_failed: · 26 unmodified lines
2336 return ret;2344 return ret;
2337}2345}
2338 2346
2339/** amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel() - Unmap a GTT BO for kernel CPU access2347/** amdgpu_amdkfd_gpuvm_unmap_bo_from_kernel() - Unmap GTT or VRAM BO for kernel CPU access
2340 *2348 *
2341 * @mem: Buffer object to be unmapped for CPU access2349 * @mem: Buffer object to be unmapped for CPU access
2342 *2350 *
@@ -2344,7 +2352,7 @@ bo_reserve_failed: · 1 unmodified line
2344 * eviction fence, so this function should only be used for cleanup before the2352 * eviction fence, so this function should only be used for cleanup before the
2345 * BO is destroyed.2353 * BO is destroyed.
2346 */2354 */
2347void amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(struct kgd_mem *mem)2355void amdgpu_amdkfd_gpuvm_unmap_bo_from_kernel(struct kgd_mem *mem)
2348{2356{
2349 struct amdgpu_bo *bo = mem->bo;2357 struct amdgpu_bo *bo = mem->bo;
2350 2358
drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c+1−1
@@ -1923,7 +1923,7 @@ int amdgpu_atombios_init(struct amdgpu_device *adev) · 1922 unmodified lines
1923 atom_card_info->pll_read = cail_pll_read;1923 atom_card_info->pll_read = cail_pll_read;
1924 atom_card_info->pll_write = cail_pll_write;1924 atom_card_info->pll_write = cail_pll_write;
1925 1925
1926 adev->mode_info.atom_context = amdgpu_atom_parse(atom_card_info, adev->bios);1926 adev->mode_info.atom_context = amdgpu_atom_parse(atom_card_info, adev->bios, adev->bios_size);
1927 if (!adev->mode_info.atom_context) {1927 if (!adev->mode_info.atom_context) {
1928 amdgpu_atombios_fini(adev);1928 amdgpu_atombios_fini(adev);
1929 return -ENOMEM;1929 return -ENOMEM;
drivers/gpu/drm/amd/amdgpu/amdgpu_atpx_handler.c+21−12
@@ -89,6 +89,15 @@ bool amdgpu_is_atpx_hybrid(void) · 88 unmodified lines
89 return amdgpu_atpx_priv.atpx.is_hybrid;89 return amdgpu_atpx_priv.atpx.is_hybrid;
90}90}
91 91
92static bool amdgpu_atpx_buffer_validate(const union acpi_object *obj,
93 size_t min_size)
94{
95 return obj && obj->type == ACPI_TYPE_BUFFER &&
96 obj->buffer.length >= sizeof(u16) &&
97 obj->buffer.length >= *(u16 *)obj->buffer.pointer &&
98 *(u16 *)obj->buffer.pointer >= min_size;
99}
100
92/**101/**
93 * amdgpu_atpx_call - call an ATPX method102 * amdgpu_atpx_call - call an ATPX method
94 *103 *
@@ -179,15 +188,15 @@ static int amdgpu_atpx_validate(struct amdgpu_atpx *atpx) · 84 unmodified lines
179 if (!info)188 if (!info)
180 return -EIO;189 return -EIO;
181 190
182 memset(&output, 0, sizeof(output));191 if (!amdgpu_atpx_buffer_validate(info, sizeof(output))) {
183 192 pr_err("Invalid ATPX GET_PX_PARAMETERS response\n");
184 size = *(u16 *) info->buffer.pointer;
185 if (size < 10) {
186 pr_err("ATPX buffer is too small: %zu\n", size);
187 kfree(info);193 kfree(info);
188 return -EINVAL;194 return -EINVAL;
189 }195 }
190 size = min(sizeof(output), size);196
197 memset(&output, 0, sizeof(output));
198
199 size = min(sizeof(output), (size_t)*(u16 *)info->buffer.pointer);
191 200
192 memcpy(&output, info->buffer.pointer, size);201 memcpy(&output, info->buffer.pointer, size);
193 202
@@ -258,15 +267,15 @@ static int amdgpu_atpx_verify_interface(struct amdgpu_atpx *atpx) · 64 unmodified lines
258 if (!info)267 if (!info)
259 return -EIO;268 return -EIO;
260 269
261 memset(&output, 0, sizeof(output));270 if (!amdgpu_atpx_buffer_validate(info, sizeof(output))) {
262 271 pr_err("Invalid ATPX VERIFY_INTERFACE response\n");
263 size = *(u16 *) info->buffer.pointer;
264 if (size < 8) {
265 pr_err("ATPX buffer is too small: %zu\n", size);
266 err = -EINVAL;272 err = -EINVAL;
267 goto out;273 goto out;
268 }274 }
269 size = min(sizeof(output), size);275
276 memset(&output, 0, sizeof(output));
277
278 size = min(sizeof(output), (size_t)*(u16 *)info->buffer.pointer);
270 279
271 memcpy(&output, info->buffer.pointer, size);280 memcpy(&output, info->buffer.pointer, size);
272 281
drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c+8−2
@@ -296,8 +296,14 @@ static int amdgpu_atrm_call(acpi_handle atrm_handle, uint8_t *bios, · 295 unmodified lines
296 }296 }
297 297
298 obj = (union acpi_object *)buffer.pointer;298 obj = (union acpi_object *)buffer.pointer;
299 memcpy(bios+offset, obj->buffer.pointer, obj->buffer.length);299 if (!obj || obj->type != ACPI_TYPE_BUFFER) {
300 len = obj->buffer.length;300 DRM_ERROR("ATRM returned an invalid object\n");
301 kfree(buffer.pointer);
302 return -EINVAL;
303 }
304
305 len = min_t(size_t, obj->buffer.length, len);
306 memcpy(bios+offset, obj->buffer.pointer, len);
301 kfree(buffer.pointer);307 kfree(buffer.pointer);
302 return len;308 return len;
303}309}
drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.c+68−102
@@ -38,14 +38,6 @@ · 37 unmodified lines
38#define AMDGPU_BO_LIST_NUM_BUCKETS (AMDGPU_BO_LIST_MAX_PRIORITY + 1)38#define AMDGPU_BO_LIST_NUM_BUCKETS (AMDGPU_BO_LIST_MAX_PRIORITY + 1)
39#define AMDGPU_BO_LIST_MAX_ENTRIES (128 * 1024)39#define AMDGPU_BO_LIST_MAX_ENTRIES (128 * 1024)
40 40
41static void amdgpu_bo_list_free_rcu(struct rcu_head *rcu)
42{
43 struct amdgpu_bo_list *list = container_of(rcu, struct amdgpu_bo_list,
44 rhead);
45 mutex_destroy(&list->bo_list_mutex);
46 kvfree(list);
47}
48
49static void amdgpu_bo_list_free(struct kref *ref)41static void amdgpu_bo_list_free(struct kref *ref)
50{42{
51 struct amdgpu_bo_list *list = container_of(ref, struct amdgpu_bo_list,43 struct amdgpu_bo_list *list = container_of(ref, struct amdgpu_bo_list,
@@ -54,7 +46,8 @@ static void amdgpu_bo_list_free(struct kref *ref) · 2 unmodified lines
54 46
55 amdgpu_bo_list_for_each_entry(e, list)47 amdgpu_bo_list_for_each_entry(e, list)
56 amdgpu_bo_unref(&e->bo);48 amdgpu_bo_unref(&e->bo);
57 call_rcu(&list->rhead, amdgpu_bo_list_free_rcu);49
50 kvfree(list);
58}51}
59 52
60static int amdgpu_bo_list_entry_cmp(const void *_a, const void *_b)53static int amdgpu_bo_list_entry_cmp(const void *_a, const void *_b)
@@ -66,9 +59,9 @@ static int amdgpu_bo_list_entry_cmp(const void *_a, const void *_b) · 5 unmodified lines
66 return (int)a->priority - (int)b->priority;59 return (int)a->priority - (int)b->priority;
67}60}
68 61
69int amdgpu_bo_list_create(struct amdgpu_device *adev, struct drm_file *filp,62struct amdgpu_bo_list *
70 struct drm_amdgpu_bo_list_entry *info,63amdgpu_bo_list_create(struct amdgpu_device *adev, struct drm_file *filp,
71 size_t num_entries, struct amdgpu_bo_list **result)64 struct drm_amdgpu_bo_list_entry *info, size_t num_entries)
72{65{
73 unsigned last_entry = 0, first_userptr = num_entries;66 unsigned last_entry = 0, first_userptr = num_entries;
74 struct amdgpu_bo_list_entry *array;67 struct amdgpu_bo_list_entry *array;
@@ -79,7 +72,7 @@ int amdgpu_bo_list_create(struct amdgpu_device *adev, struct drm_file *filp, · 4 unmodified lines
79 72
80 list = kvzalloc_flex(*list, entries, num_entries);73 list = kvzalloc_flex(*list, entries, num_entries);
81 if (!list)74 if (!list)
82 return -ENOMEM;75 return ERR_PTR(-ENOMEM);
83 76
84 kref_init(&list->refcount);77 kref_init(&list->refcount);
85 78
@@ -134,9 +127,7 @@ int amdgpu_bo_list_create(struct amdgpu_device *adev, struct drm_file *filp, · 48 unmodified lines
134 127
135 trace_amdgpu_cs_bo_status(list->num_entries, total_size);128 trace_amdgpu_cs_bo_status(list->num_entries, total_size);
136 129
137 mutex_init(&list->bo_list_mutex);130 return list;
138 *result = list;
139 return 0;
140 131
141error_free:132error_free:
142 for (i = 0; i < last_entry; ++i)133 for (i = 0; i < last_entry; ++i)
@@ -144,150 +135,125 @@ error_free: · 1 unmodified line
144 for (i = first_userptr; i < num_entries; ++i)135 for (i = first_userptr; i < num_entries; ++i)
145 amdgpu_bo_unref(&array[i].bo);136 amdgpu_bo_unref(&array[i].bo);
146 kvfree(list);137 kvfree(list);
147 return r;138 return ERR_PTR(r);
148 139
149}140}
150 141
151static void amdgpu_bo_list_destroy(struct amdgpu_fpriv *fpriv, int id)142struct amdgpu_bo_list *amdgpu_bo_list_get(struct amdgpu_fpriv *fpriv, u32 id)
152{143{
153 struct amdgpu_bo_list *list;144 struct amdgpu_bo_list *list;
154 145
155 mutex_lock(&fpriv->bo_list_lock);146 xa_lock(&fpriv->bo_list_handles);
156 list = idr_remove(&fpriv->bo_list_handles, id);147 list = xa_load(&fpriv->bo_list_handles, id);
157 mutex_unlock(&fpriv->bo_list_lock);
158 if (list)148 if (list)
159 kref_put(&list->refcount, amdgpu_bo_list_free);149 kref_get(&list->refcount);
160}150 else
161 151 list = ERR_PTR(-ENOENT);
162int amdgpu_bo_list_get(struct amdgpu_fpriv *fpriv, int id,152 xa_unlock(&fpriv->bo_list_handles);
163 struct amdgpu_bo_list **result)
164{
165 rcu_read_lock();
166 *result = idr_find(&fpriv->bo_list_handles, id);
167
168 if (*result && kref_get_unless_zero(&(*result)->refcount)) {
169 rcu_read_unlock();
170 return 0;
171 }
172 153
173 rcu_read_unlock();154 return list;
174 *result = NULL;
175 return -ENOENT;
176}155}
177 156
178void amdgpu_bo_list_put(struct amdgpu_bo_list *list)157void amdgpu_bo_list_put(struct amdgpu_bo_list *list)
179{158{
180 kref_put(&list->refcount, amdgpu_bo_list_free);159 if (list)
160 kref_put(&list->refcount, amdgpu_bo_list_free);
181}161}
182 162
183int amdgpu_bo_create_list_entry_array(struct drm_amdgpu_bo_list_in *in,163struct drm_amdgpu_bo_list_entry *
184 struct drm_amdgpu_bo_list_entry **info_param)164amdgpu_bo_create_list_entry_array(struct drm_amdgpu_bo_list_in *in)
185{165{
186 const uint32_t info_size = sizeof(struct drm_amdgpu_bo_list_entry);
187 const void __user *uptr = u64_to_user_ptr(in->bo_info_ptr);166 const void __user *uptr = u64_to_user_ptr(in->bo_info_ptr);
188 const uint32_t bo_info_size = in->bo_info_size;
189 const uint32_t bo_number = in->bo_number;167 const uint32_t bo_number = in->bo_number;
190 struct drm_amdgpu_bo_list_entry *info;
191 168
192 if (bo_number > AMDGPU_BO_LIST_MAX_ENTRIES)169 if (bo_number > AMDGPU_BO_LIST_MAX_ENTRIES)
193 return -EINVAL;170 return ERR_PTR(-EINVAL);
194 171
195 /* copy the handle array from userspace to a kernel buffer */172 if (in->bo_info_size != sizeof(struct drm_amdgpu_bo_list_entry))
196 if (likely(info_size == bo_info_size)) {173 return ERR_PTR(-EINVAL);
197 info = vmemdup_array_user(uptr, bo_number, info_size);
198 if (IS_ERR(info))
199 return PTR_ERR(info);
200 } else {
201 const uint32_t bytes = min(bo_info_size, info_size);
202 unsigned i;
203
204 info = kvmalloc_array(bo_number, info_size, GFP_KERNEL);
205 if (!info)
206 return -ENOMEM;
207
208 memset(info, 0, bo_number * info_size);
209 for (i = 0; i < bo_number; ++i, uptr += bo_info_size) {
210 if (copy_from_user(&info[i], uptr, bytes)) {
211 kvfree(info);
212 return -EFAULT;
213 }
214 }
215 }
216 174
217 *info_param = info;175 return vmemdup_array_user(uptr, bo_number,
218 return 0;176 sizeof(struct drm_amdgpu_bo_list_entry));
219}177}
220 178
221int amdgpu_bo_list_ioctl(struct drm_device *dev, void *data,179int amdgpu_bo_list_ioctl(struct drm_device *dev, void *data,
222 struct drm_file *filp)180 struct drm_file *filp)
223{181{
224 struct amdgpu_device *adev = drm_to_adev(dev);
225 struct amdgpu_fpriv *fpriv = filp->driver_priv;182 struct amdgpu_fpriv *fpriv = filp->driver_priv;
183 struct amdgpu_device *adev = drm_to_adev(dev);
184 struct amdgpu_bo_list *list, *prev, *curr;
226 union drm_amdgpu_bo_list *args = data;185 union drm_amdgpu_bo_list *args = data;
227 uint32_t handle = args->in.list_handle;186 uint32_t handle = args->in.list_handle;
228 struct drm_amdgpu_bo_list_entry *info = NULL;187 struct drm_amdgpu_bo_list_entry *info;
229 struct amdgpu_bo_list *list, *old;
230 int r;188 int r;
231 189
232 r = amdgpu_bo_create_list_entry_array(&args->in, &info);
233 if (r)
234 return r;
235
236 switch (args->in.operation) {190 switch (args->in.operation) {
237 case AMDGPU_BO_LIST_OP_CREATE:191 case AMDGPU_BO_LIST_OP_CREATE:
238 r = amdgpu_bo_list_create(adev, filp, info, args->in.bo_number,192 case AMDGPU_BO_LIST_OP_UPDATE:
239 &list);193 info = amdgpu_bo_create_list_entry_array(&args->in);
240 if (r)194 if (IS_ERR(info))
241 goto error_free;195 return PTR_ERR(info);
242 196
243 mutex_lock(&fpriv->bo_list_lock);197 list = amdgpu_bo_list_create(adev, filp, info,
244 r = idr_alloc(&fpriv->bo_list_handles, list, 1, 0, GFP_KERNEL);198 args->in.bo_number);
245 mutex_unlock(&fpriv->bo_list_lock);199 kvfree(info);
246 if (r < 0) {200 if (IS_ERR(list))
247 goto error_put_list;201 return PTR_ERR(list);
248 }
249 202
250 handle = r;
251 break;203 break;
252 204
253 case AMDGPU_BO_LIST_OP_DESTROY:205 case AMDGPU_BO_LIST_OP_DESTROY:
254 amdgpu_bo_list_destroy(fpriv, handle);206 list = xa_erase(&fpriv->bo_list_handles, handle);
207 amdgpu_bo_list_put(list);
255 handle = 0;208 handle = 0;
209
256 break;210 break;
257 211
258 case AMDGPU_BO_LIST_OP_UPDATE:212 default:
259 r = amdgpu_bo_list_create(adev, filp, info, args->in.bo_number,213 return -EINVAL;
260 &list);214 };
215
216 switch (args->in.operation) {
217 case AMDGPU_BO_LIST_OP_CREATE:
218 r = xa_alloc(&fpriv->bo_list_handles, &handle, list,
219 xa_limit_32b, GFP_KERNEL);
261 if (r)220 if (r)
262 goto error_free;221 goto error_put_list;
222
223 break;
263 224
264 mutex_lock(&fpriv->bo_list_lock);225 case AMDGPU_BO_LIST_OP_UPDATE:
265 old = idr_replace(&fpriv->bo_list_handles, list, handle);226 curr = xa_load(&fpriv->bo_list_handles, handle);
266 mutex_unlock(&fpriv->bo_list_lock);227 if (!curr) {
228 r = -ENOENT;
229 goto error_put_list;
230 }
267 231
268 if (IS_ERR(old)) {232 prev = xa_cmpxchg(&fpriv->bo_list_handles, handle, curr, list,
269 r = PTR_ERR(old);233 GFP_KERNEL);
234 if (xa_is_err(prev)) {
235 r = xa_err(prev);
236 goto error_put_list;
237 } else if (prev != curr) {
238 r = -ENOENT;
270 goto error_put_list;239 goto error_put_list;
271 }240 }
272 241
273 amdgpu_bo_list_put(old);242 amdgpu_bo_list_put(curr);
274 break;243 break;
275 244
245 case AMDGPU_BO_LIST_OP_DESTROY:
276 default:246 default:
277 r = -EINVAL;247 /* Handled above. */
278 goto error_free;248 break;
279 }249 }
280 250
281 memset(args, 0, sizeof(*args));251 memset(args, 0, sizeof(*args));
282 args->out.list_handle = handle;252 args->out.list_handle = handle;
283 kvfree(info);
284 253
285 return 0;254 return 0;
286 255
287error_put_list:256error_put_list:
288 amdgpu_bo_list_put(list);257 amdgpu_bo_list_put(list);
289
290error_free:
291 kvfree(info);
292 return r;258 return r;
293}259}
drivers/gpu/drm/amd/amdgpu/amdgpu_bo_list.h+8−14
@@ -43,7 +43,6 @@ struct amdgpu_bo_list_entry { · 42 unmodified lines
43};43};
44 44
45struct amdgpu_bo_list {45struct amdgpu_bo_list {
46 struct rcu_head rhead;
47 struct kref refcount;46 struct kref refcount;
48 struct amdgpu_bo *gds_obj;47 struct amdgpu_bo *gds_obj;
49 struct amdgpu_bo *gws_obj;48 struct amdgpu_bo *gws_obj;
@@ -51,24 +50,19 @@ struct amdgpu_bo_list { · 1 unmodified line
51 unsigned first_userptr;50 unsigned first_userptr;
52 unsigned num_entries;51 unsigned num_entries;
53 52
54 /* Protect access during command submission.
55 */
56 struct mutex bo_list_mutex;
57
58 struct amdgpu_bo_list_entry entries[] __counted_by(num_entries);53 struct amdgpu_bo_list_entry entries[] __counted_by(num_entries);
59};54};
60 55
61int amdgpu_bo_list_get(struct amdgpu_fpriv *fpriv, int id,56struct amdgpu_bo_list *amdgpu_bo_list_get(struct amdgpu_fpriv *fpriv, u32 id);
62 struct amdgpu_bo_list **result);
63void amdgpu_bo_list_put(struct amdgpu_bo_list *list);57void amdgpu_bo_list_put(struct amdgpu_bo_list *list);
64int amdgpu_bo_create_list_entry_array(struct drm_amdgpu_bo_list_in *in,58struct drm_amdgpu_bo_list_entry *
65 struct drm_amdgpu_bo_list_entry **info_param);59amdgpu_bo_create_list_entry_array(struct drm_amdgpu_bo_list_in *in);
66 60
67int amdgpu_bo_list_create(struct amdgpu_device *adev,61struct amdgpu_bo_list *
68 struct drm_file *filp,62amdgpu_bo_list_create(struct amdgpu_device *adev,
69 struct drm_amdgpu_bo_list_entry *info,63 struct drm_file *filp,
70 size_t num_entries,64 struct drm_amdgpu_bo_list_entry *info,
71 struct amdgpu_bo_list **list);65 size_t num_entries);
72 66
73#define amdgpu_bo_list_for_each_entry(e, list) \67#define amdgpu_bo_list_for_each_entry(e, list) \
74 for (e = list->entries; \68 for (e = list->entries; \
drivers/gpu/drm/amd/amdgpu/amdgpu_cper.c+2−114
@@ -289,40 +289,6 @@ struct cper_hdr *amdgpu_cper_alloc_entry(struct amdgpu_device *adev, · 288 unmodified lines
289 return hdr;289 return hdr;
290}290}
291 291
292int amdgpu_cper_generate_ue_record(struct amdgpu_device *adev,
293 struct aca_bank *bank)
294{
295 struct cper_hdr *fatal = NULL;
296 struct cper_sec_crashdump_reg_data reg_data = { 0 };
297 struct amdgpu_ring *ring = &adev->cper.ring_buf;
298 int ret;
299
300 fatal = amdgpu_cper_alloc_entry(adev, AMDGPU_CPER_TYPE_FATAL, 1);
301 if (!fatal) {
302 dev_err(adev->dev, "fail to alloc cper entry for ue record\n");
303 return -ENOMEM;
304 }
305
306 reg_data.status_lo = lower_32_bits(bank->regs[ACA_REG_IDX_STATUS]);
307 reg_data.status_hi = upper_32_bits(bank->regs[ACA_REG_IDX_STATUS]);
308 reg_data.addr_lo = lower_32_bits(bank->regs[ACA_REG_IDX_ADDR]);
309 reg_data.addr_hi = upper_32_bits(bank->regs[ACA_REG_IDX_ADDR]);
310 reg_data.ipid_lo = lower_32_bits(bank->regs[ACA_REG_IDX_IPID]);
311 reg_data.ipid_hi = upper_32_bits(bank->regs[ACA_REG_IDX_IPID]);
312 reg_data.synd_lo = lower_32_bits(bank->regs[ACA_REG_IDX_SYND]);
313 reg_data.synd_hi = upper_32_bits(bank->regs[ACA_REG_IDX_SYND]);
314
315 amdgpu_cper_entry_fill_hdr(adev, fatal, AMDGPU_CPER_TYPE_FATAL, CPER_SEV_FATAL_UNCORRECTED);
316 ret = amdgpu_cper_entry_fill_fatal_section(adev, fatal, 0, reg_data);
317 if (ret)
318 return ret;
319
320 amdgpu_cper_ring_write(ring, fatal, fatal->record_length);
321 kfree(fatal);
322
323 return 0;
324}
325
326int amdgpu_cper_generate_bp_threshold_record(struct amdgpu_device *adev)292int amdgpu_cper_generate_bp_threshold_record(struct amdgpu_device *adev)
327{293{
328 struct cper_hdr *bp_threshold = NULL;294 struct cper_hdr *bp_threshold = NULL;
@@ -348,83 +314,6 @@ int amdgpu_cper_generate_bp_threshold_record(struct amdgpu_device *adev) · 19 unmodified lines
348 return 0;314 return 0;
349}315}
350 316
351static enum cper_error_severity amdgpu_aca_err_type_to_cper_sev(struct amdgpu_device *adev,
352 enum aca_error_type aca_err_type)
353{
354 switch (aca_err_type) {
355 case ACA_ERROR_TYPE_UE:
356 return CPER_SEV_FATAL_UNCORRECTED;
357 case ACA_ERROR_TYPE_CE:
358 return CPER_SEV_NON_FATAL_CORRECTED;
359 case ACA_ERROR_TYPE_DEFERRED:
360 return CPER_SEV_NON_FATAL_UNCORRECTED;
361 default:
362 dev_err(adev->dev, "Unknown ACA error type!\n");
363 return CPER_SEV_FATAL_UNCORRECTED;
364 }
365}
366
367int amdgpu_cper_generate_ce_records(struct amdgpu_device *adev,
368 struct aca_banks *banks,
369 uint16_t bank_count)
370{
371 struct cper_hdr *corrected = NULL;
372 enum cper_error_severity sev = CPER_SEV_NON_FATAL_CORRECTED;
373 struct amdgpu_ring *ring = &adev->cper.ring_buf;
374 uint32_t reg_data[CPER_ACA_REG_COUNT] = { 0 };
375 struct aca_bank_node *node;
376 struct aca_bank *bank;
377 uint32_t i = 0;
378 int ret;
379
380 corrected = amdgpu_cper_alloc_entry(adev, AMDGPU_CPER_TYPE_RUNTIME, bank_count);
381 if (!corrected) {
382 dev_err(adev->dev, "fail to allocate cper entry for ce records\n");
383 return -ENOMEM;
384 }
385
386 /* Raise severity if any DE is detected in the ACA bank list */
387 list_for_each_entry(node, &banks->list, node) {
388 bank = &node->bank;
389 if (bank->aca_err_type == ACA_ERROR_TYPE_DEFERRED) {
390 sev = CPER_SEV_NON_FATAL_UNCORRECTED;
391 break;
392 }
393 }
394
395 amdgpu_cper_entry_fill_hdr(adev, corrected, AMDGPU_CPER_TYPE_RUNTIME, sev);
396
397 /* Combine CE and DE in cper record */
398 list_for_each_entry(node, &banks->list, node) {
399 bank = &node->bank;
400 reg_data[CPER_ACA_REG_CTL_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_CTL]);
401 reg_data[CPER_ACA_REG_CTL_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_CTL]);
402 reg_data[CPER_ACA_REG_STATUS_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_STATUS]);
403 reg_data[CPER_ACA_REG_STATUS_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_STATUS]);
404 reg_data[CPER_ACA_REG_ADDR_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_ADDR]);
405 reg_data[CPER_ACA_REG_ADDR_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_ADDR]);
406 reg_data[CPER_ACA_REG_MISC0_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_MISC0]);
407 reg_data[CPER_ACA_REG_MISC0_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_MISC0]);
408 reg_data[CPER_ACA_REG_CONFIG_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_CONFIG]);
409 reg_data[CPER_ACA_REG_CONFIG_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_CONFIG]);
410 reg_data[CPER_ACA_REG_IPID_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_IPID]);
411 reg_data[CPER_ACA_REG_IPID_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_IPID]);
412 reg_data[CPER_ACA_REG_SYND_LO] = lower_32_bits(bank->regs[ACA_REG_IDX_SYND]);
413 reg_data[CPER_ACA_REG_SYND_HI] = upper_32_bits(bank->regs[ACA_REG_IDX_SYND]);
414
415 ret = amdgpu_cper_entry_fill_runtime_section(adev, corrected, i++,
416 amdgpu_aca_err_type_to_cper_sev(adev, bank->aca_err_type),
417 reg_data, CPER_ACA_REG_COUNT);
418 if (ret)
419 return ret;
420 }
421
422 amdgpu_cper_ring_write(ring, corrected, corrected->record_length);
423 kfree(corrected);
424
425 return 0;
426}
427
428static bool amdgpu_cper_is_hdr(struct amdgpu_ring *ring, u64 pos)317static bool amdgpu_cper_is_hdr(struct amdgpu_ring *ring, u64 pos)
429{318{
430 char signature[CPER_SIGNATURE_SZ];319 char signature[CPER_SIGNATURE_SZ];
@@ -592,8 +481,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 161 unmodified lines
592 481
593 if (amdgpu_sriov_vf(adev) && !amdgpu_sriov_ras_cper_en(adev))482 if (amdgpu_sriov_vf(adev) && !amdgpu_sriov_ras_cper_en(adev))
594 return 0;483 return 0;
595 else if (!amdgpu_sriov_vf(adev) && !amdgpu_uniras_enabled(adev) &&484 else if (!amdgpu_sriov_vf(adev) && !amdgpu_uniras_enabled(adev))
596 !amdgpu_aca_is_enabled(adev))
597 return 0;485 return 0;
598 486
599 r = amdgpu_cper_ring_init(adev);487 r = amdgpu_cper_ring_init(adev);
@@ -612,7 +500,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 12 unmodified lines
612 500
613int amdgpu_cper_fini(struct amdgpu_device *adev)501int amdgpu_cper_fini(struct amdgpu_device *adev)
614{502{
615 if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev))503 if (amdgpu_sriov_vf(adev))
616 return 0;504 return 0;
617 505
618 adev->cper.enabled = false;506 adev->cper.enabled = false;
drivers/gpu/drm/amd/amdgpu/amdgpu_cper.h+0−8
@@ -26,7 +26,6 @@ · 25 unmodified lines
26#define __AMDGPU_CPER_H__26#define __AMDGPU_CPER_H__
27 27
28#include "amd_cper.h"28#include "amd_cper.h"
29#include "amdgpu_aca.h"
30 29
31#define CPER_MAX_ALLOWED_COUNT 0x100030#define CPER_MAX_ALLOWED_COUNT 0x1000
32#define CPER_MAX_RING_SIZE 0X10000031#define CPER_MAX_RING_SIZE 0X100000
@@ -88,13 +87,6 @@ int amdgpu_cper_entry_fill_bad_page_threshold_section(struct amdgpu_device *adev · 55 unmodified lines
88struct cper_hdr *amdgpu_cper_alloc_entry(struct amdgpu_device *adev,87struct cper_hdr *amdgpu_cper_alloc_entry(struct amdgpu_device *adev,
89 enum amdgpu_cper_type type,88 enum amdgpu_cper_type type,
90 uint16_t section_count);89 uint16_t section_count);
91/* UE must be encoded into separated cper entries, 1 UE 1 cper */
92int amdgpu_cper_generate_ue_record(struct amdgpu_device *adev,
93 struct aca_bank *bank);
94/* CEs and DEs are combined into 1 cper entry */
95int amdgpu_cper_generate_ce_records(struct amdgpu_device *adev,
96 struct aca_banks *banks,
97 uint16_t bank_count);
98/* Bad page threshold is encoded into separated cper entry */90/* Bad page threshold is encoded into separated cper entry */
99int amdgpu_cper_generate_bp_threshold_record(struct amdgpu_device *adev);91int amdgpu_cper_generate_bp_threshold_record(struct amdgpu_device *adev);
100void amdgpu_cper_ring_write(struct amdgpu_ring *ring,92void amdgpu_cper_ring_write(struct amdgpu_ring *ring,
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c+28−39
@@ -140,24 +140,19 @@ static int amdgpu_cs_p1_bo_handles(struct amdgpu_cs_parser *p, · 139 unmodified lines
140 struct drm_amdgpu_bo_list_in *data)140 struct drm_amdgpu_bo_list_in *data)
141{141{
142 struct drm_amdgpu_bo_list_entry *info;142 struct drm_amdgpu_bo_list_entry *info;
143 int r;143 struct amdgpu_bo_list *list;
144
145 r = amdgpu_bo_create_list_entry_array(data, &info);
146 if (r)
147 return r;
148
149 r = amdgpu_bo_list_create(p->adev, p->filp, info, data->bo_number,
150 &p->bo_list);
151 if (r)
152 goto error_free;
153 144
154 kvfree(info);145 info = amdgpu_bo_create_list_entry_array(data);
155 return 0;146 if (IS_ERR(info))
147 return PTR_ERR(info);
156 148
157error_free:149 list = amdgpu_bo_list_create(p->adev, p->filp, info, data->bo_number);
158 kvfree(info);150 kvfree(info);
151 if (IS_ERR(list))
152 return PTR_ERR(list);
159 153
160 return r;154 p->bo_list = list;
155 return 0;
161}156}
162 157
163/* Copy the data from userspace and go over it the first time */158/* Copy the data from userspace and go over it the first time */
@@ -846,6 +841,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 682 unmodified lines
846{841{
847 struct amdgpu_fpriv *fpriv = p->filp->driver_priv;842 struct amdgpu_fpriv *fpriv = p->filp->driver_priv;
848 struct ttm_operation_ctx ctx = { true, false };843 struct ttm_operation_ctx ctx = { true, false };
844 struct amdgpu_bo_list *list = NULL;
849 struct amdgpu_vm *vm = &fpriv->vm;845 struct amdgpu_vm *vm = &fpriv->vm;
850 struct amdgpu_bo_list_entry *e;846 struct amdgpu_bo_list_entry *e;
851 struct drm_gem_object *obj;847 struct drm_gem_object *obj;
@@ -857,25 +853,24 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 5 unmodified lines
857 if (p->bo_list)853 if (p->bo_list)
858 return -EINVAL;854 return -EINVAL;
859 855
860 r = amdgpu_bo_list_get(fpriv, cs->in.bo_list_handle,856 list = amdgpu_bo_list_get(fpriv, cs->in.bo_list_handle);
861 &p->bo_list);
862 if (r)
863 return r;
864 } else if (!p->bo_list) {857 } else if (!p->bo_list) {
865 /* Create a empty bo_list when no handle is provided */858 /* Create a empty bo_list when no handle is provided */
866 r = amdgpu_bo_list_create(p->adev, p->filp, NULL, 0,859 list = amdgpu_bo_list_create(p->adev, p->filp, NULL, 0);
867 &p->bo_list);
868 if (r)
869 return r;
870 }860 }
871 861
872 mutex_lock(&p->bo_list->bo_list_mutex);862 if (IS_ERR(list))
863 return PTR_ERR(list);
864 else if (list)
865 p->bo_list = list;
866 else
867 list = p->bo_list;
873 868
874 /* Get userptr backing pages. If pages are updated after registered869 /* Get userptr backing pages. If pages are updated after registered
875 * in amdgpu_gem_userptr_ioctl(), amdgpu_cs_list_validate() will do870 * in amdgpu_gem_userptr_ioctl(), amdgpu_cs_list_validate() will do
876 * amdgpu_ttm_backend_bind() to flush and invalidate new pages871 * amdgpu_ttm_backend_bind() to flush and invalidate new pages
877 */872 */
878 amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) {873 amdgpu_bo_list_for_each_userptr_entry(e, list) {
879 bool userpage_invalidated = false;874 bool userpage_invalidated = false;
880 struct amdgpu_bo *bo = e->bo;875 struct amdgpu_bo *bo = e->bo;
881 876
@@ -905,7 +900,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 23 unmodified lines
905 if (unlikely(r))900 if (unlikely(r))
906 goto out_free_user_pages;901 goto out_free_user_pages;
907 902
908 amdgpu_bo_list_for_each_entry(e, p->bo_list) {903 amdgpu_bo_list_for_each_entry(e, list) {
909 r = drm_exec_prepare_obj(&p->exec, &e->bo->tbo.base,904 r = drm_exec_prepare_obj(&p->exec, &e->bo->tbo.base,
910 TTM_NUM_MOVE_FENCES + p->gang_size);905 TTM_NUM_MOVE_FENCES + p->gang_size);
911 drm_exec_retry_on_contention(&p->exec);906 drm_exec_retry_on_contention(&p->exec);
@@ -924,7 +919,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 12 unmodified lines
924 }919 }
925 }920 }
926 921
927 amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) {922 amdgpu_bo_list_for_each_userptr_entry(e, list) {
928 struct mm_struct *usermm;923 struct mm_struct *usermm;
929 924
930 usermm = amdgpu_ttm_tt_get_usermm(e->bo->tbo.ttm);925 usermm = amdgpu_ttm_tt_get_usermm(e->bo->tbo.ttm);
@@ -977,17 +972,15 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 46 unmodified lines
977 p->bytes_moved_vis);972 p->bytes_moved_vis);
978 973
979 for (i = 0; i < p->gang_size; ++i)974 for (i = 0; i < p->gang_size; ++i)
980 amdgpu_job_set_resources(p->jobs[i], p->bo_list->gds_obj,975 amdgpu_job_set_resources(p->jobs[i], list->gds_obj,
981 p->bo_list->gws_obj,976 list->gws_obj, list->oa_obj);
982 p->bo_list->oa_obj);
983 return 0;977 return 0;
984 978
985out_free_user_pages:979out_free_user_pages:
986 amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) {980 amdgpu_bo_list_for_each_userptr_entry(e, list) {
987 amdgpu_hmm_range_free(e->range);981 amdgpu_hmm_range_free(e->range);
988 e->range = NULL;982 e->range = NULL;
989 }983 }
990 mutex_unlock(&p->bo_list->bo_list_mutex);
991 return r;984 return r;
992}985}
993 986
@@ -1371,7 +1364,6 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · 377 unmodified lines
1371 amdgpu_vm_move_to_lru_tail(p->adev, &fpriv->vm);1364 amdgpu_vm_move_to_lru_tail(p->adev, &fpriv->vm);
1372 1365
1373 mutex_unlock(&p->adev->notifier_lock);1366 mutex_unlock(&p->adev->notifier_lock);
1374 mutex_unlock(&p->bo_list->bo_list_mutex);
1375 return 0;1367 return 0;
1376}1368}
1377 1369
@@ -1443,28 +1435,25 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · 65 unmodified lines
1443 1435
1444 r = amdgpu_cs_patch_jobs(&parser);1436 r = amdgpu_cs_patch_jobs(&parser);
1445 if (r)1437 if (r)
1446 goto error_backoff;1438 goto error_fini;
1447 1439
1448 r = amdgpu_cs_vm_handling(&parser);1440 r = amdgpu_cs_vm_handling(&parser);
1449 if (r)1441 if (r)
1450 goto error_backoff;1442 goto error_fini;
1451 1443
1452 r = amdgpu_cs_sync_rings(&parser);1444 r = amdgpu_cs_sync_rings(&parser);
1453 if (r)1445 if (r)
1454 goto error_backoff;1446 goto error_fini;
1455 1447
1456 trace_amdgpu_cs_ibs(&parser);1448 trace_amdgpu_cs_ibs(&parser);
1457 1449
1458 r = amdgpu_cs_submit(&parser, data);1450 r = amdgpu_cs_submit(&parser, data);
1459 if (r)1451 if (r)
1460 goto error_backoff;1452 goto error_fini;
1461 1453
1462 amdgpu_cs_parser_fini(&parser);1454 amdgpu_cs_parser_fini(&parser);
1463 return 0;1455 return 0;
1464 1456
1465error_backoff:
1466 mutex_unlock(&parser.bo_list->bo_list_mutex);
1467
1468error_fini:1457error_fini:
1469 amdgpu_cs_parser_fini(&parser);1458 amdgpu_cs_parser_fini(&parser);
1470 return r;1459 return r;
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c+63−148
@@ -283,6 +283,8 @@ static ktime_t amdgpu_ctx_fini_entity(struct amdgpu_device *adev, · 282 unmodified lines
283 if (!entity)283 if (!entity)
284 return res;284 return res;
285 285
286 drm_sched_entity_destroy(&entity->entity);
287
286 for (i = 0; i < amdgpu_sched_jobs; ++i) {288 for (i = 0; i < amdgpu_sched_jobs; ++i) {
287 res = ktime_add(res, amdgpu_ctx_fence_time(entity->fences[i]));289 res = ktime_add(res, amdgpu_ctx_fence_time(entity->fences[i]));
288 dma_fence_put(entity->fences[i]);290 dma_fence_put(entity->fences[i]);
@@ -294,32 +296,20 @@ static ktime_t amdgpu_ctx_fini_entity(struct amdgpu_device *adev, · 5 unmodified lines
294 return res;296 return res;
295}297}
296 298
297static int amdgpu_ctx_get_stable_pstate(struct amdgpu_ctx *ctx,299static u32 amdgpu_get_stable_pstate(struct amdgpu_device *adev)
298 u32 *stable_pstate)
299{300{
300 struct amdgpu_device *adev = ctx->mgr->adev;301 switch (amdgpu_dpm_get_performance_level(adev)) {
301 enum amd_dpm_forced_level current_level;
302
303 current_level = amdgpu_dpm_get_performance_level(adev);
304
305 switch (current_level) {
306 case AMD_DPM_FORCED_LEVEL_PROFILE_STANDARD:302 case AMD_DPM_FORCED_LEVEL_PROFILE_STANDARD:
307 *stable_pstate = AMDGPU_CTX_STABLE_PSTATE_STANDARD;303 return AMDGPU_CTX_STABLE_PSTATE_STANDARD;
308 break;
309 case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK:304 case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_SCLK:
310 *stable_pstate = AMDGPU_CTX_STABLE_PSTATE_MIN_SCLK;305 return AMDGPU_CTX_STABLE_PSTATE_MIN_SCLK;
311 break;
312 case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK:306 case AMD_DPM_FORCED_LEVEL_PROFILE_MIN_MCLK:
313 *stable_pstate = AMDGPU_CTX_STABLE_PSTATE_MIN_MCLK;307 return AMDGPU_CTX_STABLE_PSTATE_MIN_MCLK;
314 break;
315 case AMD_DPM_FORCED_LEVEL_PROFILE_PEAK:308 case AMD_DPM_FORCED_LEVEL_PROFILE_PEAK:
316 *stable_pstate = AMDGPU_CTX_STABLE_PSTATE_PEAK;309 return AMDGPU_CTX_STABLE_PSTATE_PEAK;
317 break;
318 default:310 default:
319 *stable_pstate = AMDGPU_CTX_STABLE_PSTATE_NONE;311 return AMDGPU_CTX_STABLE_PSTATE_NONE;
320 break;
321 }312 }
322 return 0;
323}313}
324 314
325static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority,315static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority,
@@ -383,9 +373,9 @@ static int __amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx, · 57 unmodified lines
383 if (current_ctx && current_ctx != ctx)373 if (current_ctx && current_ctx != ctx)
384 return -EBUSY;374 return -EBUSY;
385 375
386 r = amdgpu_ctx_get_stable_pstate(ctx, &current_stable_pstate);376 current_stable_pstate = amdgpu_get_stable_pstate(adev);
387 if (r || current_stable_pstate == stable_pstate)377 if (current_stable_pstate == stable_pstate)
388 return r;378 return 0;
389 379
390 r = amdgpu_dpm_force_performance_level(adev, level);380 r = amdgpu_dpm_force_performance_level(adev, level);
391 if (r)381 if (r)
@@ -416,7 +406,7 @@ static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx, · 24 unmodified lines
416 return r;406 return r;
417}407}
418 408
419static void amdgpu_ctx_fini(struct kref *ref)409void amdgpu_ctx_fini(struct kref *ref)
420{410{
421 struct amdgpu_ctx *ctx = container_of(ref, struct amdgpu_ctx, refcount);411 struct amdgpu_ctx *ctx = container_of(ref, struct amdgpu_ctx, refcount);
422 struct amdgpu_ctx_mgr *mgr = ctx->mgr;412 struct amdgpu_ctx_mgr *mgr = ctx->mgr;
@@ -504,53 +494,26 @@ static int amdgpu_ctx_alloc(struct amdgpu_device *adev, · 81 unmodified lines
504 if (!ctx)494 if (!ctx)
505 return -ENOMEM;495 return -ENOMEM;
506 496
507 mutex_lock(&mgr->lock);
508 r = idr_alloc(&mgr->ctx_handles, ctx, 1, AMDGPU_VM_MAX_NUM_CTX, GFP_KERNEL);
509 if (r < 0) {
510 mutex_unlock(&mgr->lock);
511 kfree(ctx);
512 return r;
513 }
514
515 *id = (uint32_t)r;
516 r = amdgpu_ctx_init(mgr, priority, filp, ctx);497 r = amdgpu_ctx_init(mgr, priority, filp, ctx);
517 if (r) {498 if (r) {
518 idr_remove(&mgr->ctx_handles, *id);
519 *id = 0;
520 kfree(ctx);499 kfree(ctx);
500 return r;
521 }501 }
522 mutex_unlock(&mgr->lock);
523 return r;
524}
525
526static void amdgpu_ctx_do_release(struct kref *ref)
527{
528 struct amdgpu_ctx *ctx;
529 u32 i, j;
530 502
531 ctx = container_of(ref, struct amdgpu_ctx, refcount);503 r = xa_alloc(&mgr->ctx_handles, id, ctx, xa_limit_32b, GFP_KERNEL);
532 for (i = 0; i < AMDGPU_HW_IP_NUM; ++i) {504 if (r)
533 for (j = 0; j < amdgpu_ctx_num_entities[i]; ++j) {505 amdgpu_ctx_put(ctx);
534 if (!ctx->entities[i][j])
535 continue;
536
537 drm_sched_entity_destroy(&ctx->entities[i][j]->entity);
538 }
539 }
540 506
541 amdgpu_ctx_fini(ref);507 return r;
542}508}
543 509
544static int amdgpu_ctx_free(struct amdgpu_fpriv *fpriv, uint32_t id)510static int amdgpu_ctx_free(struct amdgpu_fpriv *fpriv, uint32_t id)
545{511{
546 struct amdgpu_ctx_mgr *mgr = &fpriv->ctx_mgr;
547 struct amdgpu_ctx *ctx;512 struct amdgpu_ctx *ctx;
548 513
549 mutex_lock(&mgr->lock);514 ctx = xa_erase(&fpriv->ctx_mgr.ctx_handles, id);
550 ctx = idr_remove(&mgr->ctx_handles, id);515 amdgpu_ctx_put(ctx);
551 if (ctx)516
552 kref_put(&ctx->refcount, amdgpu_ctx_do_release);
553 mutex_unlock(&mgr->lock);
554 return ctx ? 0 : -EINVAL;517 return ctx ? 0 : -EINVAL;
555}518}
556 519
@@ -559,19 +522,11 @@ static int amdgpu_ctx_query(struct amdgpu_device *adev, · 2 unmodified lines
559 union drm_amdgpu_ctx_out *out)522 union drm_amdgpu_ctx_out *out)
560{523{
561 struct amdgpu_ctx *ctx;524 struct amdgpu_ctx *ctx;
562 struct amdgpu_ctx_mgr *mgr;
563 unsigned reset_counter;525 unsigned reset_counter;
564 526
565 if (!fpriv)527 ctx = amdgpu_ctx_get(fpriv, id);
566 return -EINVAL;528 if (!ctx)
567
568 mgr = &fpriv->ctx_mgr;
569 mutex_lock(&mgr->lock);
570 ctx = idr_find(&mgr->ctx_handles, id);
571 if (!ctx) {
572 mutex_unlock(&mgr->lock);
573 return -EINVAL;529 return -EINVAL;
574 }
575 530
576 /* TODO: these two are always zero */531 /* TODO: these two are always zero */
577 out->state.flags = 0x0;532 out->state.flags = 0x0;
@@ -586,7 +541,8 @@ static int amdgpu_ctx_query(struct amdgpu_device *adev, · 8 unmodified lines
586 out->state.reset_status = AMDGPU_CTX_UNKNOWN_RESET;541 out->state.reset_status = AMDGPU_CTX_UNKNOWN_RESET;
587 ctx->reset_counter_query = reset_counter;542 ctx->reset_counter_query = reset_counter;
588 543
589 mutex_unlock(&mgr->lock);544 amdgpu_ctx_put(ctx);
545
590 return 0;546 return 0;
591}547}
592 548
@@ -619,18 +575,10 @@ static int amdgpu_ctx_query2(struct amdgpu_device *adev, · 26 unmodified lines
619{575{
620 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);576 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
621 struct amdgpu_ctx *ctx;577 struct amdgpu_ctx *ctx;
622 struct amdgpu_ctx_mgr *mgr;
623 578
624 if (!fpriv)579 ctx = amdgpu_ctx_get(fpriv, id);
625 return -EINVAL;580 if (!ctx)
626
627 mgr = &fpriv->ctx_mgr;
628 mutex_lock(&mgr->lock);
629 ctx = idr_find(&mgr->ctx_handles, id);
630 if (!ctx) {
631 mutex_unlock(&mgr->lock);
632 return -EINVAL;581 return -EINVAL;
633 }
634 582
635 out->state.flags = 0x0;583 out->state.flags = 0x0;
636 out->state.hangs = 0x0;584 out->state.hangs = 0x0;
@@ -671,7 +619,8 @@ static int amdgpu_ctx_query2(struct amdgpu_device *adev, · 34 unmodified lines
671 msecs_to_jiffies(AMDGPU_RAS_COUNTE_DELAY_MS));619 msecs_to_jiffies(AMDGPU_RAS_COUNTE_DELAY_MS));
672 }620 }
673 621
674 mutex_unlock(&mgr->lock);622 amdgpu_ctx_put(ctx);
623
675 return 0;624 return 0;
676}625}
677 626
@@ -680,26 +629,26 @@ static int amdgpu_ctx_stable_pstate(struct amdgpu_device *adev, · 2 unmodified lines
680 bool set, u32 *stable_pstate)629 bool set, u32 *stable_pstate)
681{630{
682 struct amdgpu_ctx *ctx;631 struct amdgpu_ctx *ctx;
683 struct amdgpu_ctx_mgr *mgr;632 int r = 0;
684 int r;
685 633
686 if (!fpriv)634 ctx = amdgpu_ctx_get(fpriv, id);
635 if (!ctx)
687 return -EINVAL;636 return -EINVAL;
688 637
689 mgr = &fpriv->ctx_mgr;638 /*
690 mutex_lock(&mgr->lock);639 * The get path is odd in this uapi - it will check whether the context
691 ctx = idr_find(&mgr->ctx_handles, id);640 * id exist, but otherwise does nothing with it. In other words, the
692 if (!ctx) {641 * uapi has historically been implemented as being able to query the
693 mutex_unlock(&mgr->lock);642 * global device state, as long as the caller supplies a random valid
694 return -EINVAL;643 * context id.
695 }644 */
696 645
697 if (set)646 if (set)
698 r = amdgpu_ctx_set_stable_pstate(ctx, *stable_pstate);647 r = amdgpu_ctx_set_stable_pstate(ctx, *stable_pstate);
699 else648 else
700 r = amdgpu_ctx_get_stable_pstate(ctx, stable_pstate);649 *stable_pstate = amdgpu_get_stable_pstate(adev);
701 650
702 mutex_unlock(&mgr->lock);651 amdgpu_ctx_put(ctx);
703 return r;652 return r;
704}653}
705 654
@@ -778,23 +727,14 @@ struct amdgpu_ctx *amdgpu_ctx_get(struct amdgpu_fpriv *fpriv, uint32_t id) · 72 unmodified lines
778 727
779 mgr = &fpriv->ctx_mgr;728 mgr = &fpriv->ctx_mgr;
780 729
781 mutex_lock(&mgr->lock);730 xa_lock(&mgr->ctx_handles);
782 ctx = idr_find(&mgr->ctx_handles, id);731 ctx = xa_load(&mgr->ctx_handles, id);
783 if (ctx)732 if (ctx)
784 kref_get(&ctx->refcount);733 kref_get(&ctx->refcount);
785 mutex_unlock(&mgr->lock);734 xa_unlock(&mgr->ctx_handles);
786 return ctx;735 return ctx;
787}736}
788 737
789int amdgpu_ctx_put(struct amdgpu_ctx *ctx)
790{
791 if (ctx == NULL)
792 return -EINVAL;
793
794 kref_put(&ctx->refcount, amdgpu_ctx_do_release);
795 return 0;
796}
797
798uint64_t amdgpu_ctx_add_fence(struct amdgpu_ctx *ctx,738uint64_t amdgpu_ctx_add_fence(struct amdgpu_ctx *ctx,
799 struct drm_sched_entity *entity,739 struct drm_sched_entity *entity,
800 struct dma_fence *fence)740 struct dma_fence *fence)
@@ -928,8 +868,7 @@ void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr, · 127 unmodified lines
928 unsigned int i;868 unsigned int i;
929 869
930 mgr->adev = adev;870 mgr->adev = adev;
931 mutex_init(&mgr->lock);871 xa_init_flags(&mgr->ctx_handles, XA_FLAGS_ALLOC1);
932 idr_init_base(&mgr->ctx_handles, 1);
933 872
934 for (i = 0; i < AMDGPU_HW_IP_NUM; ++i)873 for (i = 0; i < AMDGPU_HW_IP_NUM; ++i)
935 atomic64_set(&mgr->time_spend[i], 0);874 atomic64_set(&mgr->time_spend[i], 0);
@@ -938,13 +877,13 @@ void amdgpu_ctx_mgr_init(struct amdgpu_ctx_mgr *mgr, · 2 unmodified lines
938long amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr, long timeout)877long amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr, long timeout)
939{878{
940 struct amdgpu_ctx *ctx;879 struct amdgpu_ctx *ctx;
941 struct idr *idp;880 unsigned long id;
942 uint32_t id, i, j;881 int i, j;
943 882
944 idp = &mgr->ctx_handles;883 xa_lock(&mgr->ctx_handles);
945 884 xa_for_each(&mgr->ctx_handles, id, ctx) {
946 mutex_lock(&mgr->lock);885 kref_get(&ctx->refcount);
947 idr_for_each_entry(idp, ctx, id) {886 xa_unlock(&mgr->ctx_handles);
948 for (i = 0; i < AMDGPU_HW_IP_NUM; ++i) {887 for (i = 0; i < AMDGPU_HW_IP_NUM; ++i) {
949 for (j = 0; j < amdgpu_ctx_num_entities[i]; ++j) {888 for (j = 0; j < amdgpu_ctx_num_entities[i]; ++j) {
950 struct drm_sched_entity *entity;889 struct drm_sched_entity *entity;
@@ -956,45 +895,21 @@ long amdgpu_ctx_mgr_entity_flush(struct amdgpu_ctx_mgr *mgr, long timeout) · 5 unmodified lines
956 timeout = drm_sched_entity_flush(entity, timeout);895 timeout = drm_sched_entity_flush(entity, timeout);
957 }896 }
958 }897 }
898 amdgpu_ctx_put(ctx);
899 xa_lock(&mgr->ctx_handles);
959 }900 }
960 mutex_unlock(&mgr->lock);901 xa_unlock(&mgr->ctx_handles);
961 return timeout;902 return timeout;
962}903}
963 904
964static void amdgpu_ctx_mgr_entity_fini(struct amdgpu_ctx_mgr *mgr)905void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr)
965{906{
966 struct amdgpu_ctx *ctx;907 struct amdgpu_ctx *ctx;
967 struct idr *idp;908 unsigned long id;
968 uint32_t id, i, j;
969
970 idp = &mgr->ctx_handles;
971
972 idr_for_each_entry(idp, ctx, id) {
973 if (kref_read(&ctx->refcount) != 1) {
974 drm_err(adev_to_drm(mgr->adev), "ctx %p is still alive\n", ctx);
975 continue;
976 }
977 909
978 for (i = 0; i < AMDGPU_HW_IP_NUM; ++i) {910 xa_for_each(&mgr->ctx_handles, id, ctx)
979 for (j = 0; j < amdgpu_ctx_num_entities[i]; ++j) {911 amdgpu_ctx_put(ctx);
980 struct drm_sched_entity *entity;912 xa_destroy(&mgr->ctx_handles);
981
982 if (!ctx->entities[i][j])
983 continue;
984
985 entity = &ctx->entities[i][j]->entity;
986 drm_sched_entity_fini(entity);
987 }
988 }
989 kref_put(&ctx->refcount, amdgpu_ctx_fini);
990 }
991}
992
993void amdgpu_ctx_mgr_fini(struct amdgpu_ctx_mgr *mgr)
994{
995 amdgpu_ctx_mgr_entity_fini(mgr);
996 idr_destroy(&mgr->ctx_handles);
997 mutex_destroy(&mgr->lock);
998}913}
999 914
1000void amdgpu_ctx_mgr_usage(struct amdgpu_ctx_mgr *mgr,915void amdgpu_ctx_mgr_usage(struct amdgpu_ctx_mgr *mgr,
@@ -1002,21 +917,21 @@ void amdgpu_ctx_mgr_usage(struct amdgpu_ctx_mgr *mgr, · 1 unmodified line
1002{917{
1003 struct amdgpu_ctx *ctx;918 struct amdgpu_ctx *ctx;
1004 unsigned int hw_ip, i;919 unsigned int hw_ip, i;
1005 uint32_t id;920 unsigned long id;
1006 921
1007 /*922 /*
1008 * This is a little bit racy because it can be that a ctx or a fence are923 * This is a little bit racy because it can be that a ctx or a fence are
1009 * destroyed just in the moment we try to account them. But that is ok924 * destroyed just in the moment we try to account them. But that is ok
1010 * since exactly that case is explicitely allowed by the interface.925 * since exactly that case is explicitely allowed by the interface.
1011 */926 */
1012 mutex_lock(&mgr->lock);
1013 for (hw_ip = 0; hw_ip < AMDGPU_HW_IP_NUM; ++hw_ip) {927 for (hw_ip = 0; hw_ip < AMDGPU_HW_IP_NUM; ++hw_ip) {
1014 uint64_t ns = atomic64_read(&mgr->time_spend[hw_ip]);928 uint64_t ns = atomic64_read(&mgr->time_spend[hw_ip]);
1015 929
1016 usage[hw_ip] = ns_to_ktime(ns);930 usage[hw_ip] = ns_to_ktime(ns);
1017 }931 }
1018 932
1019 idr_for_each_entry(&mgr->ctx_handles, ctx, id) {933 xa_lock(&mgr->ctx_handles);
934 xa_for_each(&mgr->ctx_handles, id, ctx) {
1020 for (hw_ip = 0; hw_ip < AMDGPU_HW_IP_NUM; ++hw_ip) {935 for (hw_ip = 0; hw_ip < AMDGPU_HW_IP_NUM; ++hw_ip) {
1021 for (i = 0; i < amdgpu_ctx_num_entities[hw_ip]; ++i) {936 for (i = 0; i < amdgpu_ctx_num_entities[hw_ip]; ++i) {
1022 struct amdgpu_ctx_entity *centity;937 struct amdgpu_ctx_entity *centity;
@@ -1030,5 +945,5 @@ void amdgpu_ctx_mgr_usage(struct amdgpu_ctx_mgr *mgr, · 7 unmodified lines
1030 }945 }
1031 }946 }
1032 }947 }
1033 mutex_unlock(&mgr->lock);948 xa_unlock(&mgr->ctx_handles);
1034}949}
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h+10−4
@@ -25,6 +25,7 @@ · 24 unmodified lines
25 25
26#include <linux/ktime.h>26#include <linux/ktime.h>
27#include <linux/types.h>27#include <linux/types.h>
28#include <linux/xarray.h>
28 29
29#include "amdgpu_ring.h"30#include "amdgpu_ring.h"
30 31
@@ -60,16 +61,21 @@ struct amdgpu_ctx { · 29 unmodified lines
60 61
61struct amdgpu_ctx_mgr {62struct amdgpu_ctx_mgr {
62 struct amdgpu_device *adev;63 struct amdgpu_device *adev;
63 struct mutex lock;64 struct xarray ctx_handles;
64 /* protected by lock */
65 struct idr ctx_handles;
66 atomic64_t time_spend[AMDGPU_HW_IP_NUM];65 atomic64_t time_spend[AMDGPU_HW_IP_NUM];
67};66};
68 67
69extern const unsigned int amdgpu_ctx_num_entities[AMDGPU_HW_IP_NUM];68extern const unsigned int amdgpu_ctx_num_entities[AMDGPU_HW_IP_NUM];
70 69
71struct amdgpu_ctx *amdgpu_ctx_get(struct amdgpu_fpriv *fpriv, uint32_t id);70struct amdgpu_ctx *amdgpu_ctx_get(struct amdgpu_fpriv *fpriv, uint32_t id);
72int amdgpu_ctx_put(struct amdgpu_ctx *ctx);71
72void amdgpu_ctx_fini(struct kref *kref);
73
74static inline void amdgpu_ctx_put(struct amdgpu_ctx *ctx)
75{
76 if (ctx)
77 kref_put(&ctx->refcount, amdgpu_ctx_fini);
78}
73 79
74int amdgpu_ctx_get_entity(struct amdgpu_ctx *ctx, u32 hw_ip, u32 instance,80int amdgpu_ctx_get_entity(struct amdgpu_ctx *ctx, u32 hw_ip, u32 instance,
75 u32 ring, struct drm_sched_entity **entity);81 u32 ring, struct drm_sched_entity **entity);
drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c+7−0
@@ -26,6 +26,7 @@ · 25 unmodified lines
26#include <linux/kthread.h>26#include <linux/kthread.h>
27#include <linux/pci.h>27#include <linux/pci.h>
28#include <linux/uaccess.h>28#include <linux/uaccess.h>
29#include <linux/security.h>
29#include <linux/pm_runtime.h>30#include <linux/pm_runtime.h>
30 31
31#include "amdgpu.h"32#include "amdgpu.h"
@@ -1739,6 +1740,12 @@ int amdgpu_debugfs_regs_init(struct amdgpu_device *adev) · 1707 unmodified lines
1739 struct dentry *ent, *root = minor->debugfs_root;1740 struct dentry *ent, *root = minor->debugfs_root;
1740 unsigned int i;1741 unsigned int i;
1741 1742
1743 if (security_locked_down(LOCKDOWN_PCI_ACCESS)) {
1744 drm_info(adev_to_drm(adev),
1745 "amdgpu: HW debugfs nodes disabled (kernel lockdown)\n");
1746 return 0;
1747 }
1748
1742 for (i = 0; i < ARRAY_SIZE(debugfs_regs); i++) {1749 for (i = 0; i < ARRAY_SIZE(debugfs_regs); i++) {
1743 ent = debugfs_create_file(debugfs_regs_names[i],1750 ent = debugfs_create_file(debugfs_regs_names[i],
1744 S_IFREG | 0400, root,1751 S_IFREG | 0400, root,
drivers/gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c+1−0
@@ -64,6 +64,7 @@ const char *hw_ip_names[MAX_HWIP] = { · 63 unmodified lines
64 [VCN1_HWIP] = "VCN1",64 [VCN1_HWIP] = "VCN1",
65 [VCE_HWIP] = "VCE",65 [VCE_HWIP] = "VCE",
66 [VPE_HWIP] = "VPE",66 [VPE_HWIP] = "VPE",
67 [UMSCH_HWIP] = "UMSCH",
67 [DF_HWIP] = "DF",68 [DF_HWIP] = "DF",
68 [DCE_HWIP] = "DCE",69 [DCE_HWIP] = "DCE",
69 [OSSSYS_HWIP] = "OSSSYS",70 [OSSSYS_HWIP] = "OSSSYS",
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c+14−174
@@ -74,6 +74,7 @@ · 73 unmodified lines
74#include "amdgpu_ras.h"74#include "amdgpu_ras.h"
75#include "amdgpu_ras_mgr.h"75#include "amdgpu_ras_mgr.h"
76#include "amdgpu_pmu.h"76#include "amdgpu_pmu.h"
77#include "amdgpu_smu.h"
77#include "amdgpu_fru_eeprom.h"78#include "amdgpu_fru_eeprom.h"
78#include "amdgpu_reset.h"79#include "amdgpu_reset.h"
79#include "amdgpu_virt.h"80#include "amdgpu_virt.h"
@@ -2130,6 +2131,8 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 2050 unmodified lines
2130 adev->cg_flags &= amdgpu_cg_mask;2131 adev->cg_flags &= amdgpu_cg_mask;
2131 adev->pg_flags &= amdgpu_pg_mask;2132 adev->pg_flags &= amdgpu_pg_mask;
2132 2133
2134 amdgpu_smu_early_init(adev);
2135
2133 return 0;2136 return 0;
2134}2137}
2135 2138
@@ -3677,6 +3680,10 @@ static void amdgpu_device_sys_interface_fini(struct amdgpu_device *adev) · 1541 unmodified lines
3677 amdgpu_pm_sysfs_fini(adev);3680 amdgpu_pm_sysfs_fini(adev);
3678 if (adev->ucode_sysfs_en)3681 if (adev->ucode_sysfs_en)
3679 amdgpu_ucode_sysfs_fini(adev);3682 amdgpu_ucode_sysfs_fini(adev);
3683
3684 amdgpu_discovery_sysfs_fini(adev);
3685 amdgpu_preempt_mgr_sysfs_fini(adev);
3686
3680 amdgpu_device_attr_sysfs_fini(adev);3687 amdgpu_device_attr_sysfs_fini(adev);
3681 amdgpu_fru_sysfs_fini(adev);3688 amdgpu_fru_sysfs_fini(adev);
3682 3689
@@ -3773,6 +3780,7 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 90 unmodified lines
3773 3780
3774 spin_lock_init(&adev->irq.lock);3781 spin_lock_init(&adev->irq.lock);
3775 3782
3783 amdgpu_early_init_rlc_reg_funcs(adev);
3776 amdgpu_device_init_apu_flags(adev);3784 amdgpu_device_init_apu_flags(adev);
3777 3785
3778 r = amdgpu_device_check_arguments(adev);3786 r = amdgpu_device_check_arguments(adev);
@@ -4208,6 +4216,7 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev) · 429 unmodified lines
4208 4216
4209 if (adev->mman.initialized)4217 if (adev->mman.initialized)
4210 drain_workqueue(adev->mman.bdev.wq);4218 drain_workqueue(adev->mman.bdev.wq);
4219
4211 adev->shutdown = true;4220 adev->shutdown = true;
4212 4221
4213 unregister_pm_notifier(&adev->pm_nb);4222 unregister_pm_notifier(&adev->pm_nb);
@@ -4706,161 +4715,6 @@ exit: · 492 unmodified lines
4706 return 0;4715 return 0;
4707}4716}
4708 4717
4709/**
4710 * amdgpu_device_ip_check_soft_reset - did soft reset succeed
4711 *
4712 * @adev: amdgpu_device pointer
4713 *
4714 * The list of all the hardware IPs that make up the asic is walked and
4715 * the check_soft_reset callbacks are run. check_soft_reset determines
4716 * if the asic is still hung or not.
4717 * Returns true if any of the IPs are still in a hung state, false if not.
4718 */
4719static bool amdgpu_device_ip_check_soft_reset(struct amdgpu_device *adev)
4720{
4721 int i;
4722 bool asic_hang = false;
4723
4724 if (amdgpu_sriov_vf(adev))
4725 return true;
4726
4727 if (amdgpu_asic_need_full_reset(adev))
4728 return true;
4729
4730 for (i = 0; i < adev->num_ip_blocks; i++) {
4731 if (!adev->ip_blocks[i].status.valid)
4732 continue;
4733 if (adev->ip_blocks[i].version->funcs->check_soft_reset)
4734 adev->ip_blocks[i].status.hang =
4735 adev->ip_blocks[i].version->funcs->check_soft_reset(
4736 &adev->ip_blocks[i]);
4737 if (adev->ip_blocks[i].status.hang) {
4738 dev_info(adev->dev, "IP block:%s is hung!\n", adev->ip_blocks[i].version->funcs->name);
4739 asic_hang = true;
4740 }
4741 }
4742 return asic_hang;
4743}
4744
4745/**
4746 * amdgpu_device_ip_pre_soft_reset - prepare for soft reset
4747 *
4748 * @adev: amdgpu_device pointer
4749 *
4750 * The list of all the hardware IPs that make up the asic is walked and the
4751 * pre_soft_reset callbacks are run if the block is hung. pre_soft_reset
4752 * handles any IP specific hardware or software state changes that are
4753 * necessary for a soft reset to succeed.
4754 * Returns 0 on success, negative error code on failure.
4755 */
4756static int amdgpu_device_ip_pre_soft_reset(struct amdgpu_device *adev)
4757{
4758 int i, r = 0;
4759
4760 for (i = 0; i < adev->num_ip_blocks; i++) {
4761 if (!adev->ip_blocks[i].status.valid)
4762 continue;
4763 if (adev->ip_blocks[i].status.hang &&
4764 adev->ip_blocks[i].version->funcs->pre_soft_reset) {
4765 r = adev->ip_blocks[i].version->funcs->pre_soft_reset(&adev->ip_blocks[i]);
4766 if (r)
4767 return r;
4768 }
4769 }
4770
4771 return 0;
4772}
4773
4774/**
4775 * amdgpu_device_ip_need_full_reset - check if a full asic reset is needed
4776 *
4777 * @adev: amdgpu_device pointer
4778 *
4779 * Some hardware IPs cannot be soft reset. If they are hung, a full gpu
4780 * reset is necessary to recover.
4781 * Returns true if a full asic reset is required, false if not.
4782 */
4783static bool amdgpu_device_ip_need_full_reset(struct amdgpu_device *adev)
4784{
4785 int i;
4786
4787 if (amdgpu_asic_need_full_reset(adev))
4788 return true;
4789
4790 for (i = 0; i < adev->num_ip_blocks; i++) {
4791 if (!adev->ip_blocks[i].status.valid)
4792 continue;
4793 if ((adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_GMC) ||
4794 (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_SMC) ||
4795 (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_ACP) ||
4796 (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_DCE) ||
4797 adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_PSP) {
4798 if (adev->ip_blocks[i].status.hang) {
4799 dev_info(adev->dev, "Some block need full reset!\n");
4800 return true;
4801 }
4802 }
4803 }
4804 return false;
4805}
4806
4807/**
4808 * amdgpu_device_ip_soft_reset - do a soft reset
4809 *
4810 * @adev: amdgpu_device pointer
4811 *
4812 * The list of all the hardware IPs that make up the asic is walked and the
4813 * soft_reset callbacks are run if the block is hung. soft_reset handles any
4814 * IP specific hardware or software state changes that are necessary to soft
4815 * reset the IP.
4816 * Returns 0 on success, negative error code on failure.
4817 */
4818static int amdgpu_device_ip_soft_reset(struct amdgpu_device *adev)
4819{
4820 int i, r = 0;
4821
4822 for (i = 0; i < adev->num_ip_blocks; i++) {
4823 if (!adev->ip_blocks[i].status.valid)
4824 continue;
4825 if (adev->ip_blocks[i].status.hang &&
4826 adev->ip_blocks[i].version->funcs->soft_reset) {
4827 r = adev->ip_blocks[i].version->funcs->soft_reset(&adev->ip_blocks[i]);
4828 if (r)
4829 return r;
4830 }
4831 }
4832
4833 return 0;
4834}
4835
4836/**
4837 * amdgpu_device_ip_post_soft_reset - clean up from soft reset
4838 *
4839 * @adev: amdgpu_device pointer
4840 *
4841 * The list of all the hardware IPs that make up the asic is walked and the
4842 * post_soft_reset callbacks are run if the asic was hung. post_soft_reset
4843 * handles any IP specific hardware or software state changes that are
4844 * necessary after the IP has been soft reset.
4845 * Returns 0 on success, negative error code on failure.
4846 */
4847static int amdgpu_device_ip_post_soft_reset(struct amdgpu_device *adev)
4848{
4849 int i, r = 0;
4850
4851 for (i = 0; i < adev->num_ip_blocks; i++) {
4852 if (!adev->ip_blocks[i].status.valid)
4853 continue;
4854 if (adev->ip_blocks[i].status.hang &&
4855 adev->ip_blocks[i].version->funcs->post_soft_reset)
4856 r = adev->ip_blocks[i].version->funcs->post_soft_reset(&adev->ip_blocks[i]);
4857 if (r)
4858 return r;
4859 }
4860
4861 return 0;
4862}
4863
4864/**4718/**
4865 * amdgpu_device_reset_sriov - reset ASIC for SR-IOV vf4719 * amdgpu_device_reset_sriov - reset ASIC for SR-IOV vf
4866 *4720 *
@@ -5152,20 +5006,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · 285 unmodified lines
5152 5006
5153 /* Don't suspend on bare metal if we are not going to HW reset the ASIC */5007 /* Don't suspend on bare metal if we are not going to HW reset the ASIC */
5154 if (!amdgpu_sriov_vf(adev)) {5008 if (!amdgpu_sriov_vf(adev)) {
5155 5009 need_full_reset = true;
5156 if (!need_full_reset)
5157 need_full_reset = amdgpu_device_ip_need_full_reset(adev);
5158
5159 if (!need_full_reset && amdgpu_gpu_recovery &&
5160 amdgpu_device_ip_check_soft_reset(adev)) {
5161 amdgpu_device_ip_pre_soft_reset(adev);
5162 r = amdgpu_device_ip_soft_reset(adev);
5163 amdgpu_device_ip_post_soft_reset(adev);
5164 if (r || amdgpu_device_ip_check_soft_reset(adev)) {
5165 dev_info(adev->dev, "soft reset failed, will fallback to full reset!\n");
5166 need_full_reset = true;
5167 }
5168 }
5169 5010
5170 if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)) {5011 if (!test_bit(AMDGPU_SKIP_COREDUMP, &reset_context->flags)) {
5171 dev_info(tmp_adev->dev, "Dumping IP State\n");5012 dev_info(tmp_adev->dev, "Dumping IP State\n");
@@ -5618,8 +5459,7 @@ static void amdgpu_device_halt_activities(struct amdgpu_device *adev, · 446 unmodified lines
5618 drm_client_dev_suspend(adev_to_drm(tmp_adev));5459 drm_client_dev_suspend(adev_to_drm(tmp_adev));
5619 5460
5620 /* disable ras on ALL IPs */5461 /* disable ras on ALL IPs */
5621 if (!need_emergency_restart && !amdgpu_reset_in_dpc(adev) &&5462 if (!need_emergency_restart && !amdgpu_reset_in_dpc(adev))
5622 amdgpu_device_ip_need_full_reset(tmp_adev))
5623 amdgpu_ras_suspend(tmp_adev);5463 amdgpu_ras_suspend(tmp_adev);
5624 5464
5625 amdgpu_userq_pre_reset(tmp_adev);5465 amdgpu_userq_pre_reset(tmp_adev);
@@ -6891,7 +6731,7 @@ ssize_t amdgpu_get_soft_full_reset_mask(struct amdgpu_ring *ring) · 1265 unmodified lines
6891 6731
6892 if (unlikely(!ring->adev->debug_disable_soft_recovery) &&6732 if (unlikely(!ring->adev->debug_disable_soft_recovery) &&
6893 !amdgpu_sriov_vf(ring->adev) && ring->funcs->soft_recovery)6733 !amdgpu_sriov_vf(ring->adev) && ring->funcs->soft_recovery)
6894 size |= AMDGPU_RESET_TYPE_SOFT_RESET;6734 size |= AMDGPU_RESET_TYPE_SOFT_RECOVERY;
6895 6735
6896 return size;6736 return size;
6897}6737}
@@ -6907,8 +6747,8 @@ ssize_t amdgpu_show_reset_mask(char *buf, uint32_t supported_reset) · 9 unmodified lines
6907 6747
6908 }6748 }
6909 6749
6910 if (supported_reset & AMDGPU_RESET_TYPE_SOFT_RESET)6750 if (supported_reset & AMDGPU_RESET_TYPE_SOFT_RECOVERY)
6911 size += sysfs_emit_at(buf, size, "soft ");6751 size += sysfs_emit_at(buf, size, "soft_recovery ");
6912 6752
6913 if (supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE)6753 if (supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE)
6914 size += sysfs_emit_at(buf, size, "queue ");6754 size += sysfs_emit_at(buf, size, "queue ");
drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c+270−40
@@ -22,6 +22,7 @@ · 21 unmodified lines
22 */22 */
23 23
24#include <linux/firmware.h>24#include <linux/firmware.h>
25#include <linux/kernfs.h>
25 26
26#include "amdgpu.h"27#include "amdgpu.h"
27#include "amdgpu_discovery.h"28#include "amdgpu_discovery.h"
@@ -148,6 +149,26 @@ MODULE_FIRMWARE("amdgpu/aldebaran_ip_discovery.bin"); · 120 unmodified lines
148#define mmDRIVER_SCRATCH_1 0x95149#define mmDRIVER_SCRATCH_1 0x95
149#define mmDRIVER_SCRATCH_2 0x96150#define mmDRIVER_SCRATCH_2 0x96
150 151
152struct ip_discovery_top {
153 struct kobject kobj;
154 struct kset die_kset;
155 struct pci_dev *pdev;
156 struct amdgpu_device *adev;
157 uint8_t *discovery_bin;
158 uint32_t bin_size;
159 bool standalone_mode;
160};
161
162/* List to track early-initialized ip_discovery_top entries */
163struct early_ip_discovery {
164 struct list_head list;
165 struct pci_dev *pdev;
166 struct ip_discovery_top *ip_top;
167};
168
169static LIST_HEAD(early_ip_discovery_list);
170static DEFINE_MUTEX(early_ip_discovery_mutex);
171
151static const char *hw_id_names[HW_ID_MAX] = {172static const char *hw_id_names[HW_ID_MAX] = {
152 [MP1_HWID] = "MP1",173 [MP1_HWID] = "MP1",
153 [MP2_HWID] = "MP2",174 [MP2_HWID] = "MP2",
@@ -226,6 +247,7 @@ static const char *hw_id_names[HW_ID_MAX] = { · 72 unmodified lines
226 [XGBE_HWID] = "XGBE",247 [XGBE_HWID] = "XGBE",
227 [MP0_HWID] = "MP0",248 [MP0_HWID] = "MP0",
228 [VPE_HWID] = "VPE",249 [VPE_HWID] = "VPE",
250 [UMSCH_HWID] = "UMSCH",
229 [ATU_HWID] = "ATU",251 [ATU_HWID] = "ATU",
230 [AIGC_HWID] = "AIGC",252 [AIGC_HWID] = "AIGC",
231};253};
@@ -258,6 +280,7 @@ static int hw_id_map[MAX_HWIP] = { · 26 unmodified lines
258 [DCI_HWIP] = DCI_HWID,280 [DCI_HWIP] = DCI_HWID,
259 [PCIE_HWIP] = PCIE_HWID,281 [PCIE_HWIP] = PCIE_HWID,
260 [VPE_HWIP] = VPE_HWID,282 [VPE_HWIP] = VPE_HWID,
283 [UMSCH_HWIP] = UMSCH_HWID,
261 [ISP_HWIP] = ISP_HWID,284 [ISP_HWIP] = ISP_HWID,
262 [ATU_HWIP] = ATU_HWID,285 [ATU_HWIP] = ATU_HWID,
263};286};
@@ -542,25 +565,37 @@ static const char *amdgpu_discovery_get_fw_name(struct amdgpu_device *adev) · 278 unmodified lines
542 }565 }
543}566}
544 567
545static int amdgpu_discovery_get_table_info(struct amdgpu_device *adev,568static struct table_info *
546 struct table_info **info,569amdgpu_discovery_get_table_info_from_bin(uint8_t *discovery_bin,
547 uint16_t table_id)570 uint16_t table_id)
548{571{
549 struct binary_header *bhdr =572 struct binary_header *bhdr = (struct binary_header *)discovery_bin;
550 (struct binary_header *)adev->discovery.bin;
551 struct binary_header_v2 *bhdrv2;573 struct binary_header_v2 *bhdrv2;
552 574
553 switch (bhdr->version_major) {575 switch (bhdr->version_major) {
554 case 2:576 case 2:
555 bhdrv2 = (struct binary_header_v2 *)adev->discovery.bin;577 bhdrv2 = (struct binary_header_v2 *)discovery_bin;
556 *info = &bhdrv2->table_list[table_id];578 return &bhdrv2->table_list[table_id];
557 break;
558 case 1:579 case 1:
559 case 0:580 case 0:
560 *info = &bhdr->table_list[table_id];581 return &bhdr->table_list[table_id];
561 break;
562 default:582 default:
563 dev_err(adev->dev, "Invalid ip discovery table version %d\n",bhdr->version_major);583 return NULL;
584 }
585}
586
587static int amdgpu_discovery_get_table_info(struct amdgpu_device *adev,
588 struct table_info **info,
589 uint16_t table_id)
590{
591 struct binary_header *bhdr =
592 (struct binary_header *)adev->discovery.bin;
593
594 *info = amdgpu_discovery_get_table_info_from_bin(adev->discovery.bin,
595 table_id);
596 if (!*info) {
597 dev_err(adev->dev, "Invalid ip discovery table version %d\n",
598 bhdr->version_major);
564 return -EINVAL;599 return -EINVAL;
565 }600 }
566 601
@@ -724,11 +759,11 @@ out: · 157 unmodified lines
724 return r;759 return r;
725}760}
726 761
727static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev);
728
729void amdgpu_discovery_fini(struct amdgpu_device *adev)762void amdgpu_discovery_fini(struct amdgpu_device *adev)
730{763{
731 amdgpu_discovery_sysfs_fini(adev);764 if (adev->discovery.ip_top && !adev->discovery.ip_top->standalone_mode)
765 amdgpu_discovery_sysfs_fini(adev);
766
732 kfree(adev->discovery.bin);767 kfree(adev->discovery.bin);
733 adev->discovery.bin = NULL;768 adev->discovery.bin = NULL;
734}769}
@@ -737,15 +772,17 @@ static int amdgpu_discovery_validate_ip(struct amdgpu_device *adev, · 2 unmodified lines
737 uint8_t instance, uint16_t hw_id)772 uint8_t instance, uint16_t hw_id)
738{773{
739 if (instance >= HWIP_MAX_INSTANCE) {774 if (instance >= HWIP_MAX_INSTANCE) {
740 dev_err(adev->dev,775 if (adev)
741 "Unexpected instance_number (%d) from ip discovery blob\n",776 dev_err(adev->dev,
742 instance);777 "Unexpected instance_number (%d) from ip discovery blob\n",
778 instance);
743 return -EINVAL;779 return -EINVAL;
744 }780 }
745 if (hw_id >= HW_ID_MAX) {781 if (hw_id >= HW_ID_MAX) {
746 dev_err(adev->dev,782 if (adev)
747 "Unexpected hw_id (%d) from ip discovery blob\n",783 dev_err(adev->dev,
748 hw_id);784 "Unexpected hw_id (%d) from ip discovery blob\n",
785 hw_id);
749 return -EINVAL;786 return -EINVAL;
750 }787 }
751 788
@@ -1111,12 +1148,6 @@ static const struct kobj_type ip_discovery_ktype = { · 359 unmodified lines
1111 .sysfs_ops = &kobj_sysfs_ops,1148 .sysfs_ops = &kobj_sysfs_ops,
1112};1149};
1113 1150
1114struct ip_discovery_top {
1115 struct kobject kobj; /* ip_discovery/ */
1116 struct kset die_kset; /* ip_discovery/die/, contains ip_die_entry */
1117 struct amdgpu_device *adev;
1118};
1119
1120static void die_kobj_release(struct kobject *kobj)1151static void die_kobj_release(struct kobject *kobj)
1121{1152{
1122 struct ip_discovery_top *ip_top = container_of(to_kset(kobj),1153 struct ip_discovery_top *ip_top = container_of(to_kset(kobj),
@@ -1132,8 +1163,14 @@ static void ip_disc_release(struct kobject *kobj) · 9 unmodified lines
1132 kobj);1163 kobj);
1133 struct amdgpu_device *adev = ip_top->adev;1164 struct amdgpu_device *adev = ip_top->adev;
1134 1165
1166 /* In standalone mode, discovery_bin is managed by devm and will be
1167 * freed automatically when the PCI device is removed. Do not manually
1168 * free it here to avoid double-free.
1169 */
1170
1135 kfree(ip_top);1171 kfree(ip_top);
1136 adev->discovery.ip_top = NULL;1172 if (adev)
1173 adev->discovery.ip_top = NULL;
1137}1174}
1138 1175
1139static uint8_t amdgpu_discovery_get_harvest_info(struct amdgpu_device *adev,1176static uint8_t amdgpu_discovery_get_harvest_info(struct amdgpu_device *adev,
@@ -1141,6 +1178,10 @@ static uint8_t amdgpu_discovery_get_harvest_info(struct amdgpu_device *adev, · 1 unmodified line
1141{1178{
1142 uint8_t harvest = 0;1179 uint8_t harvest = 0;
1143 1180
1181 /* In early init mode (adev == NULL), harvest info is not available */
1182 if (!adev)
1183 return 0;
1184
1144 /* Until a uniform way is figured, get mask based on hwid */1185 /* Until a uniform way is figured, get mask based on hwid */
1145 switch (hw_id) {1186 switch (hw_id) {
1146 case VCN_HWID:1187 case VCN_HWID:
@@ -1169,11 +1210,14 @@ static uint8_t amdgpu_discovery_get_harvest_info(struct amdgpu_device *adev, · 22 unmodified lines
1169}1210}
1170 1211
1171static int amdgpu_discovery_sysfs_ips(struct amdgpu_device *adev,1212static int amdgpu_discovery_sysfs_ips(struct amdgpu_device *adev,
1213 struct ip_discovery_top *ip_top,
1172 struct ip_die_entry *ip_die_entry,1214 struct ip_die_entry *ip_die_entry,
1173 const size_t _ip_offset, const int num_ips,1215 const size_t _ip_offset, const int num_ips,
1174 bool reg_base_64)1216 bool reg_base_64)
1175{1217{
1176 uint8_t *discovery_bin = adev->discovery.bin;1218 uint8_t *discovery_bin = ip_top->standalone_mode ?
1219 ip_top->discovery_bin :
1220 adev->discovery.bin;
1177 int ii, jj, kk, res;1221 int ii, jj, kk, res;
1178 uint16_t hw_id;1222 uint16_t hw_id;
1179 uint8_t inst;1223 uint8_t inst;
@@ -1270,10 +1314,12 @@ next_ip: · 90 unmodified lines
1270 return 0;1314 return 0;
1271}1315}
1272 1316
1273static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev)1317static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev,
1318 struct ip_discovery_top *ip_top)
1274{1319{
1275 struct ip_discovery_top *ip_top = adev->discovery.ip_top;1320 uint8_t *discovery_bin = ip_top->standalone_mode ?
1276 uint8_t *discovery_bin = adev->discovery.bin;1321 ip_top->discovery_bin :
1322 adev->discovery.bin;
1277 struct table_info *info;1323 struct table_info *info;
1278 struct ip_discovery_header *ihdr;1324 struct ip_discovery_header *ihdr;
1279 struct die_header *dhdr;1325 struct die_header *dhdr;
@@ -1282,9 +1328,10 @@ static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev) · 2 unmodified lines
1282 size_t ip_offset;1328 size_t ip_offset;
1283 int ii, res;1329 int ii, res;
1284 1330
1285 res = amdgpu_discovery_get_table_info(adev, &info, IP_DISCOVERY);1331 info = amdgpu_discovery_get_table_info_from_bin(discovery_bin,
1286 if (res)1332 IP_DISCOVERY);
1287 return res;1333 if (!info)
1334 return -EINVAL;
1288 ihdr = (struct ip_discovery_header1335 ihdr = (struct ip_discovery_header
1289 *)(discovery_bin +1336 *)(discovery_bin +
1290 le16_to_cpu(info->offset));1337 le16_to_cpu(info->offset));
@@ -1322,7 +1369,8 @@ static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev) · 31 unmodified lines
1322 return res;1369 return res;
1323 }1370 }
1324 1371
1325 amdgpu_discovery_sysfs_ips(adev, ip_die_entry, ip_offset, num_ips, !!ihdr->base_addr_64_bit);1372 amdgpu_discovery_sysfs_ips(adev, ip_top, ip_die_entry, ip_offset,
1373 num_ips, !!ihdr->base_addr_64_bit);
1326 }1374 }
1327 1375
1328 return 0;1376 return 0;
@@ -1338,12 +1386,30 @@ static int amdgpu_discovery_sysfs_init(struct amdgpu_device *adev) · 9 unmodified lines
1338 if (!discovery_bin)1386 if (!discovery_bin)
1339 return -EINVAL;1387 return -EINVAL;
1340 1388
1389 /* If early init already created sysfs in standalone mode, skip normal init */
1390 if (adev->discovery.ip_top && adev->discovery.ip_top->standalone_mode)
1391 return 0;
1392
1341 ip_top = kzalloc_obj(*ip_top);1393 ip_top = kzalloc_obj(*ip_top);
1342 if (!ip_top)1394 if (!ip_top)
1343 return -ENOMEM;1395 return -ENOMEM;
1344 1396
1345 ip_top->adev = adev;1397 ip_top->adev = adev;
1346 adev->discovery.ip_top = ip_top;1398
1399 /* Check if ip_discovery already exists before creating.
1400 * This shouldn't normally happen but handle it gracefully.
1401 */
1402 if (adev->dev->kobj.sd) {
1403 struct kernfs_node *existing;
1404
1405 existing = kernfs_find_and_get(adev->dev->kobj.sd, "ip_discovery");
1406 if (existing) {
1407 kernfs_put(existing);
1408 kfree(ip_top);
1409 return 0;
1410 }
1411 }
1412
1347 res = kobject_init_and_add(&ip_top->kobj, &ip_discovery_ktype,1413 res = kobject_init_and_add(&ip_top->kobj, &ip_discovery_ktype,
1348 &adev->dev->kobj, "ip_discovery");1414 &adev->dev->kobj, "ip_discovery");
1349 if (res) {1415 if (res) {
@@ -1351,6 +1417,8 @@ static int amdgpu_discovery_sysfs_init(struct amdgpu_device *adev) · 1 unmodified line
1351 goto Err;1417 goto Err;
1352 }1418 }
1353 1419
1420 adev->discovery.ip_top = ip_top;
1421
1354 die_kset = &ip_top->die_kset;1422 die_kset = &ip_top->die_kset;
1355 kobject_set_name(&die_kset->kobj, "%s", "die");1423 kobject_set_name(&die_kset->kobj, "%s", "die");
1356 die_kset->kobj.parent = &ip_top->kobj;1424 die_kset->kobj.parent = &ip_top->kobj;
@@ -1365,7 +1433,7 @@ static int amdgpu_discovery_sysfs_init(struct amdgpu_device *adev) · 8 unmodified lines
1365 ip_hw_instance_attrs[ii] = &ip_hw_attr[ii].attr;1433 ip_hw_instance_attrs[ii] = &ip_hw_attr[ii].attr;
1366 ip_hw_instance_attrs[ii] = NULL;1434 ip_hw_instance_attrs[ii] = NULL;
1367 1435
1368 res = amdgpu_discovery_sysfs_recurse(adev);1436 res = amdgpu_discovery_sysfs_recurse(adev, ip_top);
1369 1437
1370 return res;1438 return res;
1371Err:1439Err:
@@ -1412,7 +1480,7 @@ static void amdgpu_discovery_sysfs_die_free(struct ip_die_entry *ip_die_entry) · 40 unmodified lines
1412 kobject_put(&ip_die_entry->ip_kset.kobj);1480 kobject_put(&ip_die_entry->ip_kset.kobj);
1413}1481}
1414 1482
1415static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev)1483void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev)
1416{1484{
1417 struct ip_discovery_top *ip_top = adev->discovery.ip_top;1485 struct ip_discovery_top *ip_top = adev->discovery.ip_top;
1418 struct list_head *el, *tmp;1486 struct list_head *el, *tmp;
@@ -1421,6 +1489,16 @@ static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev) · 2 unmodified lines
1421 if (!ip_top)1489 if (!ip_top)
1422 return;1490 return;
1423 1491
1492 /*
1493 * In standalone mode the sysfs hierarchy is tied to the PCI device
1494 * lifetime and is torn down by amdgpu_discovery_sysfs_early_fini().
1495 * Freeing it here would leave a dangling pointer in the early
1496 * discovery list, causing a use-after-free on driver unbind.
1497 */
1498 if (ip_top->standalone_mode)
1499 return;
1500
1501 adev->discovery.ip_top = NULL;
1424 die_kset = &ip_top->die_kset;1502 die_kset = &ip_top->die_kset;
1425 spin_lock(&die_kset->list_lock);1503 spin_lock(&die_kset->list_lock);
1426 list_for_each_prev_safe(el, tmp, &die_kset->list) {1504 list_for_each_prev_safe(el, tmp, &die_kset->list) {
@@ -1479,6 +1557,150 @@ void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p) · 52 unmodified lines
1479 spin_unlock(&die_kset->list_lock);1557 spin_unlock(&die_kset->list_lock);
1480}1558}
1481 1559
1560int amdgpu_discovery_sysfs_early_init(struct amdgpu_device *adev, struct pci_dev *pdev)
1561{
1562 struct ip_discovery_top *ip_top;
1563 struct early_ip_discovery *early_entry, *tmp;
1564 struct kset *die_kset;
1565 uint8_t *discovery_bin;
1566 int res, ii;
1567
1568 if (!adev || !adev->discovery.bin)
1569 return -EINVAL;
1570
1571 if (adev->discovery.ip_top)
1572 return 0;
1573
1574 mutex_lock(&early_ip_discovery_mutex);
1575 list_for_each_entry_safe(early_entry, tmp, &early_ip_discovery_list, list) {
1576 if (early_entry->pdev == pdev) {
1577 adev->discovery.ip_top = early_entry->ip_top;
1578 early_entry->ip_top->adev = adev;
1579 mutex_unlock(&early_ip_discovery_mutex);
1580 return 0;
1581 }
1582 }
1583 mutex_unlock(&early_ip_discovery_mutex);
1584
1585 discovery_bin = adev->discovery.bin;
1586
1587 early_entry = kzalloc(sizeof(*early_entry), GFP_KERNEL);
1588 if (!early_entry)
1589 return -ENOMEM;
1590
1591 ip_top = kzalloc(sizeof(*ip_top), GFP_KERNEL);
1592 if (!ip_top) {
1593 kfree(early_entry);
1594 return -ENOMEM;
1595 }
1596
1597 ip_top->discovery_bin = devm_kmemdup(&pdev->dev, discovery_bin,
1598 DISCOVERY_TMR_SIZE, GFP_KERNEL);
1599 if (!ip_top->discovery_bin) {
1600 kfree(ip_top);
1601 kfree(early_entry);
1602 return -ENOMEM;
1603 }
1604
1605 ip_top->bin_size = DISCOVERY_TMR_SIZE;
1606 ip_top->pdev = pdev;
1607 ip_top->adev = adev;
1608 ip_top->standalone_mode = true;
1609
1610 /* Check if ip_discovery already exists (from previous probe attempt).
1611 * This can happen if the module was unloaded and reloaded but the
1612 * sysfs persisted (tied to PCI device lifetime).
1613 */
1614 if (pdev->dev.kobj.sd) {
1615 struct kernfs_node *existing;
1616
1617 existing = kernfs_find_and_get(pdev->dev.kobj.sd, "ip_discovery");
1618 if (existing) {
1619 kernfs_put(existing);
1620 kfree(ip_top);
1621 kfree(early_entry);
1622 return 0;
1623 }
1624 }
1625
1626 res = kobject_init_and_add(&ip_top->kobj, &ip_discovery_ktype,
1627 &pdev->dev.kobj, "ip_discovery");
1628 if (res)
1629 goto err_put_kobj;
1630
1631 adev->discovery.ip_top = ip_top;
1632
1633 die_kset = &ip_top->die_kset;
1634 kobject_set_name(&die_kset->kobj, "%s", "die");
1635 die_kset->kobj.parent = &ip_top->kobj;
1636 die_kset->kobj.ktype = &die_kobj_ktype;
1637 res = kset_register(&ip_top->die_kset);
1638 if (res)
1639 goto err_put_die_kset;
1640
1641 for (ii = 0; ii < ARRAY_SIZE(ip_hw_attr); ii++)
1642 ip_hw_instance_attrs[ii] = &ip_hw_attr[ii].attr;
1643 ip_hw_instance_attrs[ii] = NULL;
1644
1645 res = amdgpu_discovery_sysfs_recurse(NULL, ip_top);
1646 if (res)
1647 goto err_put_die_kset;
1648
1649 early_entry->pdev = pdev;
1650 early_entry->ip_top = ip_top;
1651 mutex_lock(&early_ip_discovery_mutex);
1652 list_add(&early_entry->list, &early_ip_discovery_list);
1653 mutex_unlock(&early_ip_discovery_mutex);
1654
1655 return 0;
1656
1657err_put_die_kset:
1658 kobject_put(&ip_top->die_kset.kobj);
1659err_put_kobj:
1660 kobject_put(&ip_top->kobj);
1661 kfree(early_entry);
1662 adev->discovery.ip_top = NULL;
1663 return res;
1664}
1665
1666void amdgpu_discovery_sysfs_early_fini(struct pci_dev *pdev)
1667{
1668 struct early_ip_discovery *entry, *tmp_entry;
1669 struct ip_discovery_top *ip_top = NULL;
1670 struct list_head *el, *tmp;
1671 struct kset *die_kset;
1672
1673 /* Find the entry in our tracking list */
1674 mutex_lock(&early_ip_discovery_mutex);
1675 list_for_each_entry_safe(entry, tmp_entry, &early_ip_discovery_list, list) {
1676 if (entry->pdev == pdev) {
1677 ip_top = entry->ip_top;
1678 list_del(&entry->list);
1679 kfree(entry);
1680 break;
1681 }
1682 }
1683 mutex_unlock(&early_ip_discovery_mutex);
1684
1685 if (!ip_top)
1686 return;
1687
1688 /* Clean up sysfs hierarchy */
1689 die_kset = &ip_top->die_kset;
1690
1691 spin_lock(&die_kset->list_lock);
1692 list_for_each_prev_safe(el, tmp, &die_kset->list) {
1693 list_del_init(el);
1694 spin_unlock(&die_kset->list_lock);
1695 amdgpu_discovery_sysfs_die_free(to_ip_die_entry(list_to_kobj(el)));
1696 spin_lock(&die_kset->list_lock);
1697 }
1698 spin_unlock(&die_kset->list_lock);
1699
1700 kobject_put(&ip_top->die_kset.kobj);
1701 kobject_put(&ip_top->kobj);
1702 /* ip_top itself will be freed by kobject_put via ip_disc_release */
1703}
1482 1704
1483/* ================================================== */1705/* ================================================== */
1484 1706
@@ -1504,6 +1726,9 @@ static int amdgpu_discovery_reg_base_init(struct amdgpu_device *adev) · 19 unmodified lines
1504 r = amdgpu_discovery_init(adev);1726 r = amdgpu_discovery_init(adev);
1505 if (r)1727 if (r)
1506 return r;1728 return r;
1729
1730 amdgpu_discovery_sysfs_early_init(adev, adev->pdev);
1731
1507 discovery_bin = adev->discovery.bin;1732 discovery_bin = adev->discovery.bin;
1508 wafl_ver = 0;1733 wafl_ver = 0;
1509 adev->gfx.xcc_mask = 0;1734 adev->gfx.xcc_mask = 0;
@@ -2636,7 +2861,12 @@ static int amdgpu_discovery_set_mm_ip_blocks(struct amdgpu_device *adev) · 1126 unmodified lines
2636 return -EINVAL;2861 return -EINVAL;
2637 }2862 }
2638 } else {2863 } else {
2639 switch (amdgpu_ip_version(adev, UVD_HWIP, 0)) {2864 uint32_t vcn_version = amdgpu_ip_version(adev, UVD_HWIP, 0);
2865
2866 /* no VCN discovered; nothing to add */
2867 if (!vcn_version)
2868 return 0;
2869 switch (vcn_version) {
2640 case IP_VERSION(1, 0, 0):2870 case IP_VERSION(1, 0, 0):
2641 case IP_VERSION(1, 0, 1):2871 case IP_VERSION(1, 0, 1):
2642 amdgpu_device_ip_block_add(adev, &vcn_v1_0_ip_block);2872 amdgpu_device_ip_block_add(adev, &vcn_v1_0_ip_block);
@@ -2704,7 +2934,7 @@ static int amdgpu_discovery_set_mm_ip_blocks(struct amdgpu_device *adev) · 61 unmodified lines
2704 default:2934 default:
2705 dev_err(adev->dev,2935 dev_err(adev->dev,
2706 "Failed to add vcn/jpeg ip block(UVD_HWIP:0x%x)\n",2936 "Failed to add vcn/jpeg ip block(UVD_HWIP:0x%x)\n",
2707 amdgpu_ip_version(adev, UVD_HWIP, 0));2937 vcn_version);
2708 return -EINVAL;2938 return -EINVAL;
2709 }2939 }
2710 }2940 }
drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.h+6−0
@@ -41,6 +41,7 @@ struct amdgpu_discovery_info { · 40 unmodified lines
41 bool reserve_tmr;41 bool reserve_tmr;
42};42};
43 43
44void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev);
44void amdgpu_discovery_fini(struct amdgpu_device *adev);45void amdgpu_discovery_fini(struct amdgpu_device *adev);
45int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev);46int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev);
46 47
@@ -53,4 +54,9 @@ int amdgpu_discovery_get_gc_major_minor_version(struct amdgpu_device *adev, · 6 unmodified lines
53 54
54void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p);55void amdgpu_discovery_dump(struct amdgpu_device *adev, struct drm_printer *p);
55 56
57/* Early sysfs functions for persistent ip_discovery export */
58int amdgpu_discovery_sysfs_early_init(struct amdgpu_device *adev,
59 struct pci_dev *pdev);
60void amdgpu_discovery_sysfs_early_fini(struct pci_dev *pdev);
61
56#endif /* __AMDGPU_DISCOVERY__ */62#endif /* __AMDGPU_DISCOVERY__ */
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c+23−4
@@ -33,7 +33,6 @@ · 32 unmodified lines
33#include <drm/drm_vblank.h>33#include <drm/drm_vblank.h>
34 34
35#include <linux/cc_platform.h>35#include <linux/cc_platform.h>
36#include <linux/console.h>
37#include <linux/dynamic_debug.h>36#include <linux/dynamic_debug.h>
38#include <linux/module.h>37#include <linux/module.h>
39#include <linux/mmu_notifier.h>38#include <linux/mmu_notifier.h>
@@ -146,7 +145,9 @@ enum AMDGPU_DEBUG_MASK { · 106 unmodified lines
146 AMDGPU_DEBUG_SMU_POOL = BIT(7),145 AMDGPU_DEBUG_SMU_POOL = BIT(7),
147 AMDGPU_DEBUG_VM_USERPTR = BIT(8),146 AMDGPU_DEBUG_VM_USERPTR = BIT(8),
148 AMDGPU_DEBUG_DISABLE_RAS_CE_LOG = BIT(9),147 AMDGPU_DEBUG_DISABLE_RAS_CE_LOG = BIT(9),
149 AMDGPU_DEBUG_ENABLE_CE_CS = BIT(10)148 AMDGPU_DEBUG_ENABLE_CE_CS = BIT(10),
149 AMDGPU_DEBUG_HIBERNATION_THAW_RESUME_GPU = BIT(11),
150 AMDGPU_DEBUG_DISABLE_IP_BLOCK_SOFT_RESET = BIT(12),
150};151};
151 152
152unsigned int amdgpu_vram_limit = UINT_MAX;153unsigned int amdgpu_vram_limit = UINT_MAX;
@@ -1939,6 +1940,7 @@ static const struct pci_device_id pciidlist[] = { · 1786 unmodified lines
1939 {0x1002, 0x6646, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE|AMD_IS_MOBILITY},1940 {0x1002, 0x6646, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE|AMD_IS_MOBILITY},
1940 {0x1002, 0x6647, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE|AMD_IS_MOBILITY},1941 {0x1002, 0x6647, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE|AMD_IS_MOBILITY},
1941 {0x1002, 0x6649, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},1942 {0x1002, 0x6649, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},
1943 {0x1002, 0x664D, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},
1942 {0x1002, 0x6650, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},1944 {0x1002, 0x6650, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},
1943 {0x1002, 0x6651, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},1945 {0x1002, 0x6651, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},
1944 {0x1002, 0x6658, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},1946 {0x1002, 0x6658, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_BONAIRE},
@@ -2008,6 +2010,7 @@ static const struct pci_device_id pciidlist[] = { · 63 unmodified lines
2008 {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},2010 {0x1002, 0x6930, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
2009 {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},2011 {0x1002, 0x6938, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
2010 {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},2012 {0x1002, 0x6939, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
2013 {0x1002, 0x693B, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TONGA},
2011 /* fiji */2014 /* fiji */
2012 {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},2015 {0x1002, 0x7300, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
2013 {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},2016 {0x1002, 0x730F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_FIJI},
@@ -2036,6 +2039,7 @@ static const struct pci_device_id pciidlist[] = { · 22 unmodified lines
2036 {0x1002, 0x67C4, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},2039 {0x1002, 0x67C4, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
2037 {0x1002, 0x67C7, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},2040 {0x1002, 0x67C7, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
2038 {0x1002, 0x67D0, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},2041 {0x1002, 0x67D0, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
2042 {0x1002, 0x67D4, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
2039 {0x1002, 0x67DF, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},2043 {0x1002, 0x67DF, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
2040 {0x1002, 0x67C8, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},2044 {0x1002, 0x67C8, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
2041 {0x1002, 0x67C9, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},2045 {0x1002, 0x67C9, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS10},
@@ -2049,6 +2053,7 @@ static const struct pci_device_id pciidlist[] = { · 7 unmodified lines
2049 {0x1002, 0x6985, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},2053 {0x1002, 0x6985, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
2050 {0x1002, 0x6986, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},2054 {0x1002, 0x6986, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
2051 {0x1002, 0x6987, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},2055 {0x1002, 0x6987, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
2056 {0x1002, 0x698F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
2052 {0x1002, 0x6995, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},2057 {0x1002, 0x6995, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
2053 {0x1002, 0x6997, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},2058 {0x1002, 0x6997, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
2054 {0x1002, 0x699F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},2059 {0x1002, 0x699F, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_POLARIS12},
@@ -2250,7 +2255,7 @@ static void amdgpu_init_debug_options(struct amdgpu_device *adev) · 195 unmodified lines
2250 }2255 }
2251 2256
2252 if (amdgpu_debug_mask & AMDGPU_DEBUG_DISABLE_GPU_SOFT_RECOVERY) {2257 if (amdgpu_debug_mask & AMDGPU_DEBUG_DISABLE_GPU_SOFT_RECOVERY) {
2253 pr_info("debug: soft reset for GPU recovery disabled\n");2258 pr_info("debug: soft recovery disabled\n");
2254 adev->debug_disable_soft_recovery = true;2259 adev->debug_disable_soft_recovery = true;
2255 }2260 }
2256 2261
@@ -2291,6 +2296,16 @@ static void amdgpu_init_debug_options(struct amdgpu_device *adev) · 34 unmodified lines
2291 pr_info("debug: allowing command submission to CE engine\n");2296 pr_info("debug: allowing command submission to CE engine\n");
2292 adev->debug_enable_ce_cs = true;2297 adev->debug_enable_ce_cs = true;
2293 }2298 }
2299
2300 if (amdgpu_debug_mask & AMDGPU_DEBUG_HIBERNATION_THAW_RESUME_GPU) {
2301 pr_info("debug: resume gpu in thaw() of hibernation\n");
2302 adev->debug_hibernation_thaw_resume_gpu = true;
2303 }
2304
2305 if (amdgpu_debug_mask & AMDGPU_DEBUG_DISABLE_IP_BLOCK_SOFT_RESET) {
2306 pr_info("debug: IP block soft reset disabled\n");
2307 adev->debug_disable_ip_block_soft_reset = true;
2308 }
2294}2309}
2295 2310
2296static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags)2311static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags)
@@ -2552,6 +2567,8 @@ amdgpu_pci_remove(struct pci_dev *pdev) · 255 unmodified lines
2552 2567
2553 amdgpu_driver_unload_kms(dev);2568 amdgpu_driver_unload_kms(dev);
2554 2569
2570 amdgpu_discovery_sysfs_early_fini(pdev);
2571
2555 /*2572 /*
2556 * Flush any in flight DMA operations from device.2573 * Flush any in flight DMA operations from device.
2557 * Clear the Bus Master Enable bit and then wait on the PCIe Device2574 * Clear the Bus Master Enable bit and then wait on the PCIe Device
@@ -2705,9 +2722,10 @@ static int amdgpu_pmops_freeze(struct device *dev) · 147 unmodified lines
2705static int amdgpu_pmops_thaw(struct device *dev)2722static int amdgpu_pmops_thaw(struct device *dev)
2706{2723{
2707 struct drm_device *drm_dev = dev_get_drvdata(dev);2724 struct drm_device *drm_dev = dev_get_drvdata(dev);
2725 struct amdgpu_device *adev = drm_to_adev(drm_dev);
2708 2726
2709 /* do not resume device if it's normal hibernation */2727 /* do not resume device if it's normal hibernation */
2710 if (console_suspend_enabled &&2728 if (!adev->debug_hibernation_thaw_resume_gpu &&
2711 !pm_hibernate_is_recovering() &&2729 !pm_hibernate_is_recovering() &&
2712 !pm_hibernation_mode_is_suspend())2730 !pm_hibernation_mode_is_suspend())
2713 return 0;2731 return 0;
@@ -3076,6 +3094,7 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = { · 362 unmodified lines
3076 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_SIGNAL, amdgpu_userq_signal_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),3094 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_SIGNAL, amdgpu_userq_signal_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3077 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),3095 DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3078 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_LIST_HANDLES, amdgpu_gem_list_handles_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),3096 DRM_IOCTL_DEF_DRV(AMDGPU_GEM_LIST_HANDLES, amdgpu_gem_list_handles_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3097 DRM_IOCTL_DEF_DRV(AMDGPU_PROC_OPTIONS, amdgpu_proc_options_ioctl, DRM_AUTH|DRM_RENDER_ALLOW),
3079};3098};
3080 3099
3081static const struct drm_driver amdgpu_kms_driver = {3100static const struct drm_driver amdgpu_kms_driver = {
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c+52−1
@@ -727,6 +727,15 @@ void amdgpu_ring_set_fence_errors_and_reemit(struct amdgpu_ring *ring, · 726 unmodified lines
727 last_seq = amdgpu_fence_read(ring) & ring->fence_drv.num_fences_mask;727 last_seq = amdgpu_fence_read(ring) & ring->fence_drv.num_fences_mask;
728 seq = ring->fence_drv.sync_seq & ring->fence_drv.num_fences_mask;728 seq = ring->fence_drv.sync_seq & ring->fence_drv.num_fences_mask;
729 729
730 /* If there is nothing to reemit, return early and set an error on the fence
731 * if applicable. If all of the fences are siganlled, this will be a nop.
732 * if there are still fences and ring_backup_entries_to_copy is 0, then
733 * we are skipping it on purpose.
734 */
735 if (!ring->ring_backup_entries_to_copy) {
736 amdgpu_fence_driver_force_completion(ring, &guilty_fence->base);
737 return;
738 }
730 ring->reemit = true;739 ring->reemit = true;
731 amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy);740 amdgpu_ring_alloc(ring, ring->ring_backup_entries_to_copy);
732 spin_lock_irqsave(&ring->fence_drv.lock, flags);741 spin_lock_irqsave(&ring->fence_drv.lock, flags);
@@ -741,7 +750,8 @@ void amdgpu_ring_set_fence_errors_and_reemit(struct amdgpu_ring *ring, · 8 unmodified lines
741 if (unprocessed && !dma_fence_is_signaled_locked(unprocessed)) {750 if (unprocessed && !dma_fence_is_signaled_locked(unprocessed)) {
742 fence = container_of(unprocessed, struct amdgpu_fence, base);751 fence = container_of(unprocessed, struct amdgpu_fence, base);
743 is_guilty_fence = fence == guilty_fence;752 is_guilty_fence = fence == guilty_fence;
744 is_guilty_context = fence->context == guilty_fence->context;753 is_guilty_context = guilty_fence ?
754 (fence->context == guilty_fence->context) : false;
745 755
746 /* mark all fences from the guilty context with an error */756 /* mark all fences from the guilty context with an error */
747 if (is_guilty_fence)757 if (is_guilty_fence)
@@ -794,6 +804,17 @@ void amdgpu_ring_backup_unprocessed_commands(struct amdgpu_ring *ring, · 46 unmodified lines
794 seq = ring->fence_drv.sync_seq & ring->fence_drv.num_fences_mask;804 seq = ring->fence_drv.sync_seq & ring->fence_drv.num_fences_mask;
795 ring->ring_backup_entries_to_copy = 0;805 ring->ring_backup_entries_to_copy = 0;
796 806
807 /* if we've already seen this fence, return early.
808 * ring->ring_backup_entries_to_copy is set to 0 so
809 * the reemit helper will return early as well to
810 * avoid getting stuck in a reemit loop.
811 */
812 if (ring->guilty_fence == guilty_fence) {
813 ring->guilty_fence = NULL;
814 return;
815 }
816 ring->guilty_fence = guilty_fence;
817
797 do {818 do {
798 last_seq++;819 last_seq++;
799 last_seq &= ring->fence_drv.num_fences_mask;820 last_seq &= ring->fence_drv.num_fences_mask;
@@ -811,6 +832,36 @@ void amdgpu_ring_backup_unprocessed_commands(struct amdgpu_ring *ring, · 11 unmodified lines
811 } while (last_seq != seq);832 } while (last_seq != seq);
812}833}
813 834
835struct amdgpu_fence *
836amdgpu_ring_find_guilty_fence(struct amdgpu_ring *ring)
837{
838 struct dma_fence *unprocessed;
839 struct dma_fence __rcu **ptr;
840 struct amdgpu_fence *fence;
841 u32 seq, last_seq;
842
843 last_seq = amdgpu_fence_read(ring) & ring->fence_drv.num_fences_mask;
844 seq = ring->fence_drv.sync_seq & ring->fence_drv.num_fences_mask;
845
846 do {
847 last_seq++;
848 last_seq &= ring->fence_drv.num_fences_mask;
849
850 ptr = &ring->fence_drv.fences[last_seq];
851 rcu_read_lock();
852 unprocessed = rcu_dereference(*ptr);
853
854 if (unprocessed && !dma_fence_is_signaled(unprocessed)) {
855 fence = container_of(unprocessed, struct amdgpu_fence, base);
856 rcu_read_unlock();
857 return fence;
858 }
859 rcu_read_unlock();
860 } while (last_seq != seq);
861
862 return NULL;
863}
864
814/*865/*
815 * Common fence implementation866 * Common fence implementation
816 */867 */
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c+278−6
@@ -34,6 +34,7 @@ · 33 unmodified lines
34#include "amdgpu_xcp.h"34#include "amdgpu_xcp.h"
35#include "amdgpu_xgmi.h"35#include "amdgpu_xgmi.h"
36#include "amdgpu_mes.h"36#include "amdgpu_mes.h"
37#include "mes_userqueue.h"
37#include "nvd.h"38#include "nvd.h"
38 39
39/* delay 0.1 second to enable gfx off feature */40/* delay 0.1 second to enable gfx off feature */
@@ -377,6 +378,30 @@ int amdgpu_gfx_kiq_init(struct amdgpu_device *adev, · 337 unmodified lines
377 return 0;378 return 0;
378}379}
379 380
381static void amdgpu_gfx_mqd_reset_restore(struct amdgpu_ring *ring)
382{
383 struct amdgpu_device *adev = ring->adev;
384 int mqd_idx, mqd_size;
385
386 /* restore mqd with the backup copy */
387 if (ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) {
388 mqd_idx = ring - &adev->gfx.compute_ring[0];
389 mqd_size = adev->mqds[AMDGPU_HW_IP_COMPUTE].mqd_size;
390 if (adev->gfx.mec.mqd_backup[mqd_idx])
391 memcpy_toio(ring->mqd_ptr, adev->gfx.mec.mqd_backup[mqd_idx], mqd_size);
392 } else if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) {
393 mqd_size = adev->mqds[AMDGPU_HW_IP_GFX].mqd_size;
394 mqd_idx = ring - &adev->gfx.gfx_ring[0];
395
396 if (adev->gfx.me.mqd_backup[mqd_idx])
397 memcpy_toio(ring->mqd_ptr, adev->gfx.me.mqd_backup[mqd_idx], mqd_size);
398 }
399 /* reset the ring */
400 ring->wptr = 0;
401 atomic64_set((atomic64_t *)ring->wptr_cpu_addr, 0);
402 amdgpu_ring_clear_ring(ring);
403}
404
380/* create MQD for each compute/gfx queue */405/* create MQD for each compute/gfx queue */
381int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,406int amdgpu_gfx_mqd_sw_init(struct amdgpu_device *adev,
382 unsigned int mqd_size, int xcc_id)407 unsigned int mqd_size, int xcc_id)
@@ -1964,6 +1989,60 @@ static ssize_t amdgpu_gfx_get_compute_reset_mask(struct device *dev, · 1581 unmodified lines
1964 return amdgpu_show_reset_mask(buf, adev->gfx.compute_supported_reset);1989 return amdgpu_show_reset_mask(buf, adev->gfx.compute_supported_reset);
1965}1990}
1966 1991
1992static int amdgpu_gfx_mes_reset_queue_start(struct amdgpu_ring *ring,
1993 unsigned int vmid,
1994 struct amdgpu_fence *timedout_fence,
1995 bool use_mmio)
1996{
1997 struct amdgpu_device *adev = ring->adev;
1998 bool reinit_queue;
1999 int r;
2000
2001 if ((ring->funcs->type == AMDGPU_RING_TYPE_COMPUTE) &&
2002 adev->mes.compute_pipe_reset_enabled)
2003 reinit_queue = true;
2004 else if ((ring->funcs->type == AMDGPU_RING_TYPE_GFX) &&
2005 adev->mes.gfx_pipe_reset_enabled)
2006 reinit_queue = true;
2007 else
2008 reinit_queue = use_mmio;
2009
2010 amdgpu_ring_reset_helper_begin(ring, timedout_fence);
2011
2012 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, use_mmio, 0);
2013 if (r)
2014 return r;
2015
2016 if (reinit_queue) {
2017 r = amdgpu_mes_unmap_legacy_queue(adev, ring,
2018 RESET_QUEUES, 0, 0, 0);
2019 if (r)
2020 return r;
2021 amdgpu_gfx_mqd_reset_restore(ring);
2022
2023 r = amdgpu_mes_map_legacy_queue(adev, ring, 0);
2024 if (r) {
2025 dev_err(adev->dev, "failed to remap kgq\n");
2026 return r;
2027 }
2028 }
2029 return 0;
2030}
2031
2032int amdgpu_gfx_mes_reset_queue(struct amdgpu_ring *ring,
2033 unsigned int vmid,
2034 struct amdgpu_fence *timedout_fence,
2035 bool use_mmio)
2036{
2037 int r;
2038
2039 r = amdgpu_gfx_mes_reset_queue_start(ring, vmid, timedout_fence,
2040 use_mmio);
2041 if (r)
2042 return r;
2043 return amdgpu_ring_reset_helper_end(ring, timedout_fence);
2044}
2045
1967static DEVICE_ATTR(run_cleaner_shader, 0200,2046static DEVICE_ATTR(run_cleaner_shader, 0200,
1968 NULL, amdgpu_gfx_set_run_cleaner_shader);2047 NULL, amdgpu_gfx_set_run_cleaner_shader);
1969 2048
@@ -2122,6 +2201,200 @@ void amdgpu_gfx_sysfs_fini(struct amdgpu_device *adev) · 152 unmodified lines
2122 }2201 }
2123}2202}
2124 2203
2204static void amdgpu_gfx_reset_start_compute_scheds(struct amdgpu_device *adev,
2205 struct amdgpu_ring *guilty_ring)
2206{
2207 struct amdgpu_ring *ring;
2208 int i;
2209
2210 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
2211 ring = &adev->gfx.compute_ring[i];
2212 if (ring == guilty_ring)
2213 continue;
2214 drm_sched_wqueue_start(&ring->sched);
2215 }
2216}
2217
2218static void amdgpu_gfx_reset_stop_compute_scheds(struct amdgpu_device *adev,
2219 struct amdgpu_ring *guilty_ring)
2220{
2221 struct amdgpu_ring *ring;
2222 int i;
2223
2224 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
2225 ring = &adev->gfx.compute_ring[i];
2226 if (ring == guilty_ring)
2227 continue;
2228 drm_sched_wqueue_stop(&ring->sched);
2229 }
2230}
2231
2232/*
2233 * Match the MES-reported hung doorbell against a compute ring and run
2234 * the reset. On hit, the matched ring and its guilty fence are returned
2235 * via *out_ring / *out_fence so the caller can defer reset end until
2236 * after MES has resumed all gangs.
2237 */
2238static int amdgpu_gfx_reset_mes_kcq(struct amdgpu_device *adev,
2239 struct amdgpu_ring *guilty_ring,
2240 unsigned int db,
2241 struct amdgpu_ring **out_ring,
2242 struct amdgpu_fence **out_fence)
2243{
2244 bool use_mmio = adev->gfx.mec.use_mmio_for_reset;
2245 struct amdgpu_fence *fence;
2246 struct amdgpu_ring *ring;
2247 int i, r;
2248
2249 *out_ring = NULL;
2250 *out_fence = NULL;
2251 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
2252 ring = &adev->gfx.compute_ring[i];
2253 if (ring == guilty_ring)
2254 continue;
2255 if (ring->doorbell_index == db) {
2256 fence = amdgpu_ring_find_guilty_fence(ring);
2257 r = amdgpu_gfx_mes_reset_queue_start(ring, 0, fence,
2258 use_mmio);
2259 if (r)
2260 return r;
2261 *out_ring = ring;
2262 *out_fence = fence;
2263 break;
2264 }
2265 }
2266 return 0;
2267}
2268
2269int amdgpu_gfx_reset_mes_compute(struct amdgpu_device *adev,
2270 struct amdgpu_ring *ring,
2271 struct amdgpu_fence *guilty_fence,
2272 struct amdgpu_usermode_queue *uq,
2273 unsigned int *hung_queue_count,
2274 void *faulty_queue_input)
2275{
2276 struct amdgpu_mes_hung_queue_hqd_info *hqd_info =
2277 (struct amdgpu_mes_hung_queue_hqd_info *)
2278 &adev->gfx.mec.mes_hung_db_array[adev->mes.hung_queue_hqd_info_offset];
2279 int i, r, pipe, queue, queue_type;
2280 unsigned int num_hung = 0;
2281 bool use_mmio = adev->gfx.mec.use_mmio_for_reset;
2282 struct mes_remove_queue_input *queue_input = (struct mes_remove_queue_input *)faulty_queue_input;
2283 struct amdgpu_gfx_deferred_entry deferred_end[AMDGPU_MAX_COMPUTE_RINGS + 1];
2284 int n_deferred = 0;
2285 int ring_err;
2286
2287 guard(mutex)(&adev->gfx.mec.reset_mutex);
2288 /* stop the drm schedulers for all compute queues */
2289 amdgpu_gfx_reset_stop_compute_scheds(adev, ring);
2290 /* suspend all will determine which queues are hung.
2291 * reset detect will return the array of bad queue doorbells
2292 */
2293 r = amdgpu_mes_suspend(adev, 0);
2294 /* if suspend all success, it should no hang queue */
2295 if (!r)
2296 /* always reset the KCQ/userq since we need to signal the fence
2297 * and we could be stuck in a loop which is preemptable.
2298 */
2299 goto fence_reset;
2300 r = amdgpu_mes_detect_and_reset_hung_queues(adev, AMDGPU_RING_TYPE_COMPUTE,
2301 true, &num_hung, adev->gfx.mec.mes_hung_db_array, 0);
2302 if (r)
2303 goto out;
2304 if (hung_queue_count)
2305 *hung_queue_count = num_hung;
2306
2307fence_reset:
2308 /* reset the queue this came from if specified */
2309 if (ring) {
2310 r = amdgpu_gfx_mes_reset_queue_start(ring, 0, guilty_fence,
2311 use_mmio);
2312 if (r)
2313 goto out;
2314 deferred_end[n_deferred].ring = ring;
2315 deferred_end[n_deferred].fence = guilty_fence;
2316 n_deferred++;
2317 }
2318 if (uq) {
2319 r = mes_userq_reset(uq);
2320 if (r)
2321 goto out;
2322 }
2323 for (i = 0; i < num_hung; i++) {
2324 struct amdgpu_ring *hr = NULL;
2325 struct amdgpu_fence *hf = NULL;
2326
2327 pipe = hqd_info[i].pipe_index;
2328 queue = hqd_info[i].queue_index;
2329 queue_type = hqd_info[i].queue_type;
2330
2331 /* reset any KCQs */
2332 r = amdgpu_gfx_reset_mes_kcq(adev, ring,
2333 adev->gfx.mec.mes_hung_db_array[i],
2334 &hr, &hf);
2335 if (r)
2336 goto out;
2337 if (hr) {
2338 deferred_end[n_deferred].ring = hr;
2339 deferred_end[n_deferred].fence = hf;
2340 n_deferred++;
2341 }
2342 /* reset any KFD queues */
2343 r = amdgpu_amdkfd_reset_mes_queue(adev, 0, queue_type, pipe, queue,
2344 adev->gfx.mec.mes_hung_db_array[i]);
2345 if (r)
2346 goto out;
2347 /* reset KGD user queues */
2348 r = mes_userq_reset_queue(adev, uq, queue_type, pipe, queue,
2349 adev->gfx.mec.mes_hung_db_array[i]);
2350 if (r)
2351 goto out;
2352 }
2353
2354 /* MES doesn't detect any hung queue but we have a known bad queue
2355 * and it is not KCQ
2356 */
2357 if (!num_hung && queue_input && !ring) {
2358 /* MES suspend_all is successful means this bad queue is
2359 * preempted successfuly. Remove it before resume all so it
2360 * doesn't get mapped back
2361 */
2362 if (!down_read_trylock(&adev->reset_domain->sem)) {
2363 r = -EIO;
2364 goto out;
2365 }
2366 amdgpu_mes_lock(&adev->mes);
2367 r = adev->mes.funcs->remove_hw_queue(&adev->mes, queue_input);
2368 amdgpu_mes_unlock(&adev->mes);
2369 up_read(&adev->reset_domain->sem);
2370 }
2371
2372out:
2373 /* resume all will enable the non-hung queues */
2374 amdgpu_mes_resume(adev, 0);
2375
2376 /* Now CP is running again — replay backed-up commands and ring
2377 * doorbells on each reset queue.
2378 */
2379 ring_err = r;
2380 for (i = 0; i < n_deferred; i++) {
2381 int er = amdgpu_ring_reset_helper_end(deferred_end[i].ring,
2382 deferred_end[i].fence);
2383
2384 if (er && !ring_err)
2385 ring_err = er;
2386 }
2387
2388 if (!ring_err)
2389 amdgpu_gfx_reset_start_compute_scheds(adev, ring);
2390
2391 /* If this reset is triggered by non-KCQ, the KCQ result after resume must
2392 * not override the reset result; otherwise a false reset failure is returned
2393 * to the non-KCQ caller
2394 */
2395 return ring ? ring_err : r;
2396}
2397
2125int amdgpu_gfx_cleaner_shader_sw_init(struct amdgpu_device *adev,2398int amdgpu_gfx_cleaner_shader_sw_init(struct amdgpu_device *adev,
2126 unsigned int cleaner_shader_size)2399 unsigned int cleaner_shader_size)
2127{2400{
@@ -2460,9 +2733,8 @@ void amdgpu_gfx_profile_ring_begin_use(struct amdgpu_ring *ring) · 332 unmodified lines
2460 else2733 else
2461 profile = PP_SMC_POWER_PROFILE_COMPUTE;2734 profile = PP_SMC_POWER_PROFILE_COMPUTE;
2462 2735
2463 atomic_inc(&adev->gfx.total_submission_cnt);2736 if (!atomic_fetch_inc(&adev->gfx.total_submission_cnt))
2464 2737 cancel_delayed_work_sync(&adev->gfx.idle_work);
2465 cancel_delayed_work_sync(&adev->gfx.idle_work);
2466 2738
2467 /* We can safely return early here because we've cancelled the2739 /* We can safely return early here because we've cancelled the
2468 * the delayed work so there is no one else to set it to false2740 * the delayed work so there is no one else to set it to false
@@ -2490,9 +2762,9 @@ void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring) · 21 unmodified lines
2490 if (amdgpu_dpm_is_overdrive_enabled(adev))2762 if (amdgpu_dpm_is_overdrive_enabled(adev))
2491 return;2763 return;
2492 2764
2493 atomic_dec(&ring->adev->gfx.total_submission_cnt);2765 if (atomic_dec_and_test(&ring->adev->gfx.total_submission_cnt))
2494 2766 schedule_delayed_work(&ring->adev->gfx.idle_work,
2495 schedule_delayed_work(&ring->adev->gfx.idle_work, GFX_PROFILE_IDLE_TIMEOUT);2767 GFX_PROFILE_IDLE_TIMEOUT);
2496}2768}
2497 2769
2498/**2770/**
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h+22−2
@@ -36,6 +36,8 @@ · 35 unmodified lines
36#include "amdgpu_ring_mux.h"36#include "amdgpu_ring_mux.h"
37#include "amdgpu_xcp.h"37#include "amdgpu_xcp.h"
38 38
39struct amdgpu_usermode_queue;
40
39/* GFX current status */41/* GFX current status */
40#define AMDGPU_GFX_NORMAL_MODE 0x00000000L42#define AMDGPU_GFX_NORMAL_MODE 0x00000000L
41#define AMDGPU_GFX_SAFE_MODE 0x00000001L43#define AMDGPU_GFX_SAFE_MODE 0x00000001L
@@ -116,6 +118,9 @@ struct amdgpu_mec { · 74 unmodified lines
116 u32 num_pipe_per_mec;118 u32 num_pipe_per_mec;
117 u32 num_queue_per_pipe;119 u32 num_queue_per_pipe;
118 void *mqd_backup[AMDGPU_MAX_COMPUTE_RINGS * AMDGPU_MAX_GC_INSTANCES];120 void *mqd_backup[AMDGPU_MAX_COMPUTE_RINGS * AMDGPU_MAX_GC_INSTANCES];
121 bool use_mmio_for_reset;
122 u32 *mes_hung_db_array;
123 struct mutex reset_mutex;
119};124};
120 125
121struct amdgpu_mec_bitmap {126struct amdgpu_mec_bitmap {
@@ -401,6 +406,7 @@ struct amdgpu_me { · 279 unmodified lines
401 uint32_t num_pipe_per_me;406 uint32_t num_pipe_per_me;
402 uint32_t num_queue_per_pipe;407 uint32_t num_queue_per_pipe;
403 void *mqd_backup[AMDGPU_MAX_GFX_RINGS];408 void *mqd_backup[AMDGPU_MAX_GFX_RINGS];
409 bool use_mmio_for_reset;
404 410
405 /* These are the resources for which amdgpu takes ownership */411 /* These are the resources for which amdgpu takes ownership */
406 DECLARE_BITMAP(queue_bitmap, AMDGPU_MAX_GFX_QUEUES);412 DECLARE_BITMAP(queue_bitmap, AMDGPU_MAX_GFX_QUEUES);
@@ -479,8 +485,6 @@ struct amdgpu_gfx { · 72 unmodified lines
479 const struct amdgpu_gfx_funcs *funcs;485 const struct amdgpu_gfx_funcs *funcs;
480 486
481 /* reset mask */487 /* reset mask */
482 uint32_t grbm_soft_reset;
483 uint32_t srbm_soft_reset;
484 uint32_t gfx_supported_reset;488 uint32_t gfx_supported_reset;
485 uint32_t compute_supported_reset;489 uint32_t compute_supported_reset;
486 490
@@ -543,6 +547,11 @@ struct amdgpu_gfx { · 56 unmodified lines
543 bool disable_uq;547 bool disable_uq;
544};548};
545 549
550struct amdgpu_gfx_deferred_entry {
551 struct amdgpu_ring *ring;
552 struct amdgpu_fence *fence;
553};
554
546struct amdgpu_gfx_ras_reg_entry {555struct amdgpu_gfx_ras_reg_entry {
547 struct amdgpu_ras_err_status_reg_entry reg_entry;556 struct amdgpu_ras_err_status_reg_entry reg_entry;
548 enum amdgpu_gfx_ras_mem_id_type mem_id_type;557 enum amdgpu_gfx_ras_mem_id_type mem_id_type;
@@ -641,6 +650,12 @@ int amdgpu_gfx_poison_consumption_handler(struct amdgpu_device *adev, · 92 unmodified lines
641bool amdgpu_gfx_is_master_xcc(struct amdgpu_device *adev, int xcc_id);650bool amdgpu_gfx_is_master_xcc(struct amdgpu_device *adev, int xcc_id);
642int amdgpu_gfx_sysfs_init(struct amdgpu_device *adev);651int amdgpu_gfx_sysfs_init(struct amdgpu_device *adev);
643void amdgpu_gfx_sysfs_fini(struct amdgpu_device *adev);652void amdgpu_gfx_sysfs_fini(struct amdgpu_device *adev);
653int amdgpu_gfx_reset_mes_compute(struct amdgpu_device *adev,
654 struct amdgpu_ring *ring,
655 struct amdgpu_fence *guilty_fence,
656 struct amdgpu_usermode_queue *uq,
657 unsigned int *hung_queue_count,
658 void *faulty_queue_input);
644void amdgpu_gfx_ras_error_func(struct amdgpu_device *adev,659void amdgpu_gfx_ras_error_func(struct amdgpu_device *adev,
645 void *ras_error_status,660 void *ras_error_status,
646 void (*func)(struct amdgpu_device *adev, void *ras_error_status,661 void (*func)(struct amdgpu_device *adev, void *ras_error_status,
@@ -667,6 +682,11 @@ void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev); · 20 unmodified lines
667 682
668int amdgpu_gfx_ring_preempt_ib(struct amdgpu_ring *ring);683int amdgpu_gfx_ring_preempt_ib(struct amdgpu_ring *ring);
669 684
685int amdgpu_gfx_mes_reset_queue(struct amdgpu_ring *ring,
686 unsigned int vmid,
687 struct amdgpu_fence *timedout_fence,
688 bool use_mmio);
689
670static inline const char *amdgpu_gfx_compute_mode_desc(int mode)690static inline const char *amdgpu_gfx_compute_mode_desc(int mode)
671{691{
672 switch (mode) {692 switch (mode) {
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c+7−2
@@ -1763,10 +1763,15 @@ int amdgpu_gmc_init_mem_ranges(struct amdgpu_device *adev) · 1762 unmodified lines
1763 valid = true;1763 valid = true;
1764 else1764 else
1765 valid = amdgpu_gmc_validate_partition_info(adev);1765 valid = amdgpu_gmc_validate_partition_info(adev);
1766 if (!valid) {1766 if (!valid)
1767 /* TODO: handle invalid case */
1768 dev_warn(adev->dev,1767 dev_warn(adev->dev,
1769 "Mem ranges not matching with hardware config\n");1768 "Mem ranges not matching with hardware config\n");
1769
1770 if (!adev->gmc.num_mem_partitions) {
1771 dev_err(adev->dev, "num_mem_partitions is zero\n");
1772 kfree(adev->gmc.mem_partitions);
1773 adev->gmc.mem_partitions = NULL;
1774 return -EINVAL;
1770 }1775 }
1771 1776
1772 return 0;1777 return 0;
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h+0−1
@@ -286,7 +286,6 @@ struct amdgpu_gmc { · 285 unmodified lines
286 struct amdgpu_irq_src vm_fault;286 struct amdgpu_irq_src vm_fault;
287 uint32_t vram_type;287 uint32_t vram_type;
288 uint8_t vram_vendor;288 uint8_t vram_vendor;
289 uint32_t srbm_soft_reset;
290 bool prt_warning;289 bool prt_warning;
291 uint32_t sdpif_register;290 uint32_t sdpif_register;
292 /* apertures */291 /* apertures */
drivers/gpu/drm/amd/amdgpu/amdgpu_ip.c+126−17
@@ -369,43 +369,152 @@ int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, · 368 unmodified lines
369}369}
370 370
371/**371/**
372 * amdgpu_device_ip_is_hw - is the hardware IP enabled372 * amdgpu_device_ip_is_valid - is the hardware IP valid
373 *373 *
374 * @adev: amdgpu_device pointer374 * @adev: amdgpu_device pointer
375 * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.)375 * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.)
376 *376 *
377 * Check if the hardware IP is enable or not.377 * Check if the hardware IP is valid or not.
378 * Returns true if it the IP is enable, false if not.378 * Returns true if it the IP is valid, false if not.
379 */379 */
380bool amdgpu_device_ip_is_hw(struct amdgpu_device *adev,380bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev,
381 enum amd_ip_block_type block_type)381 enum amd_ip_block_type block_type)
382{382{
383 struct amdgpu_ip_block *ip_block;383 struct amdgpu_ip_block *ip_block;
384 384
385 ip_block = amdgpu_device_ip_get_ip_block(adev, block_type);385 ip_block = amdgpu_device_ip_get_ip_block(adev, block_type);
386 if (ip_block)386 if (ip_block)
387 return ip_block->status.hw;387 return ip_block->status.valid;
388 388
389 return false;389 return false;
390}390}
391 391
392/**392/**
393 * amdgpu_device_ip_is_valid - is the hardware IP valid393 * amdgpu_ip_from_ring() - Find IP block type corresponding to ring type.
394 *394 *
395 * @adev: amdgpu_device pointer395 * @ring_type: The ring type whose IP block you are looking for.
396 * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.)396 */
397static enum amd_ip_block_type amdgpu_ip_from_ring(const enum amdgpu_ring_type ring_type)
398{
399 switch (ring_type) {
400 case AMDGPU_RING_TYPE_GFX:
401 case AMDGPU_RING_TYPE_COMPUTE:
402 return AMD_IP_BLOCK_TYPE_GFX;
403
404 case AMDGPU_RING_TYPE_SDMA:
405 return AMD_IP_BLOCK_TYPE_SDMA;
406
407 case AMDGPU_RING_TYPE_UVD:
408 case AMDGPU_RING_TYPE_UVD_ENC:
409 return AMD_IP_BLOCK_TYPE_UVD;
410
411 case AMDGPU_RING_TYPE_VCE:
412 return AMD_IP_BLOCK_TYPE_VCE;
413
414 case AMDGPU_RING_TYPE_VCN_DEC:
415 case AMDGPU_RING_TYPE_VCN_ENC:
416 return AMD_IP_BLOCK_TYPE_VCN;
417
418 case AMDGPU_RING_TYPE_VCN_JPEG:
419 return AMD_IP_BLOCK_TYPE_JPEG;
420
421 case AMDGPU_RING_TYPE_VPE:
422 return AMD_IP_BLOCK_TYPE_VPE;
423
424 default:
425 return AMD_IP_BLOCK_TYPE_NUM;
426 }
427}
428
429/**
430 * amdgpu_ring_mask_from_ip() - Find mask of ring types corresponding to an IP block type.
397 *431 *
398 * Check if the hardware IP is valid or not.432 * @ip_type: The IP block type whose rings you are looking for.
399 * Returns true if it the IP is valid, false if not.
400 */433 */
401bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev,434static u32 amdgpu_ring_mask_from_ip(const enum amd_ip_block_type ip_type)
402 enum amd_ip_block_type block_type)435{
436 switch (ip_type) {
437 case AMD_IP_BLOCK_TYPE_GFX:
438 return BIT(AMDGPU_RING_TYPE_GFX) | BIT(AMDGPU_RING_TYPE_COMPUTE);
439
440 case AMD_IP_BLOCK_TYPE_SDMA:
441 return BIT(AMDGPU_RING_TYPE_SDMA);
442
443 case AMD_IP_BLOCK_TYPE_UVD:
444 return BIT(AMDGPU_RING_TYPE_UVD) | BIT(AMDGPU_RING_TYPE_UVD_ENC);
445
446 case AMD_IP_BLOCK_TYPE_VCE:
447 return BIT(AMD_IP_BLOCK_TYPE_VCE);
448
449 case AMD_IP_BLOCK_TYPE_VCN:
450 return BIT(AMDGPU_RING_TYPE_VCN_DEC) | BIT(AMDGPU_RING_TYPE_VCN_ENC);
451
452 case AMD_IP_BLOCK_TYPE_JPEG:
453 return BIT(AMDGPU_RING_TYPE_VCN_JPEG);
454
455 case AMD_IP_BLOCK_TYPE_VPE:
456 return BIT(AMDGPU_RING_TYPE_VPE);
457
458 default:
459 return 0;
460 }
461}
462
463/**
464 * amdgpu_device_ip_soft_reset() - Perform a graceful soft reset on an IP block.
465 *
466 * @guilty_ring: The ring which is guilty of causing a reset.
467 * @guilty_fence: The fence which didn't signal.
468 *
469 * IP block soft reset is used when attempting to recover
470 * from a GPU hang in a situation where a more fine grained
471 * reset type isn't available or didn't work. This effectively
472 * resets all rings that belong to the same device IP block
473 * and re-initializes the device IP block.
474 *
475 * The reset is handled gracefully, meaning that we try to
476 * minimize collateral damage (ie. avoid rejecting non-guilty jobs)
477 * as well as back up and restore the contents of all rings
478 * so that the system can move on from the hang.
479 */
480int amdgpu_device_ip_soft_reset(struct amdgpu_ring *guilty_ring,
481 struct amdgpu_fence *guilty_fence)
403{482{
483 struct amdgpu_device *adev = guilty_ring->adev;
404 struct amdgpu_ip_block *ip_block;484 struct amdgpu_ip_block *ip_block;
485 enum amd_ip_block_type ip_type;
486 u32 ring_type_mask;
487 int r;
405 488
406 ip_block = amdgpu_device_ip_get_ip_block(adev, block_type);489 ip_type = amdgpu_ip_from_ring(guilty_ring->funcs->type);
407 if (ip_block)490 ip_block = amdgpu_device_ip_get_ip_block(adev, ip_type);
408 return ip_block->status.valid;
409 491
410 return false;492 if (!ip_block || !ip_block->version->funcs->soft_reset) {
493 dev_warn(adev->dev, "IP block soft reset not supported on %s\n",
494 ip_block->version->funcs->name);
495 return -EOPNOTSUPP;
496 }
497
498 dev_err(adev->dev, "Starting %s IP block soft reset\n",
499 ip_block->version->funcs->name);
500
501 ring_type_mask = amdgpu_ring_mask_from_ip(ip_type);
502
503 amdgpu_device_lock_reset_domain(adev->reset_domain);
504 amdgpu_multi_ring_reset_helper_begin(ring_type_mask, guilty_ring, guilty_fence);
505
506 r = ip_block->version->funcs->soft_reset(ip_block);
507
508 r = amdgpu_multi_ring_reset_helper_end(ring_type_mask, guilty_ring, r);
509 amdgpu_device_unlock_reset_domain(adev->reset_domain);
510
511 if (r) {
512 dev_err(adev->dev, "Failed %s IP block soft reset: %d\n",
513 ip_block->version->funcs->name, r);
514 return r;
515 }
516
517 dev_err(adev->dev, "Successful %s IP block soft reset\n",
518 ip_block->version->funcs->name);
519 return 0;
411}520}
drivers/gpu/drm/amd/amdgpu/amdgpu_ip.h+6−2
@@ -68,6 +68,7 @@ enum amd_hw_ip_block_type { · 67 unmodified lines
68 ISP_HWIP,68 ISP_HWIP,
69 ATU_HWIP,69 ATU_HWIP,
70 AIGC_HWIP,70 AIGC_HWIP,
71 UMSCH_HWIP,
71 MAX_HWIP72 MAX_HWIP
72};73};
73 74
@@ -84,6 +85,9 @@ enum amd_hw_ip_block_type { · 10 unmodified lines
84#define IP_VERSION_SUBREV(ver) ((ver) & 0xF)85#define IP_VERSION_SUBREV(ver) ((ver) & 0xF)
85#define IP_VERSION_MAJ_MIN_REV(ver) ((ver) >> 8)86#define IP_VERSION_MAJ_MIN_REV(ver) ((ver) >> 8)
86 87
88struct amdgpu_ring;
89struct amdgpu_fence;
90
87struct amdgpu_ip_map_info {91struct amdgpu_ip_map_info {
88 /* Map of logical to actual dev instances/mask */92 /* Map of logical to actual dev instances/mask */
89 uint32_t dev_inst[MAX_HWIP][HWIP_MAX_INSTANCE];93 uint32_t dev_inst[MAX_HWIP][HWIP_MAX_INSTANCE];
@@ -146,9 +150,9 @@ void amdgpu_device_ip_get_clockgating_state(struct amdgpu_device *adev, · 56 unmodified lines
146 u64 *flags);150 u64 *flags);
147int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev,151int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev,
148 enum amd_ip_block_type block_type);152 enum amd_ip_block_type block_type);
149bool amdgpu_device_ip_is_hw(struct amdgpu_device *adev,
150 enum amd_ip_block_type block_type);
151bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev,153bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev,
152 enum amd_ip_block_type block_type);154 enum amd_ip_block_type block_type);
155int amdgpu_device_ip_soft_reset(struct amdgpu_ring *guilty_ring,
156 struct amdgpu_fence *guilty_fence);
153 157
154#endif /* __AMDGPU_IP_H__ */158#endif /* __AMDGPU_IP_H__ */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c+12−1
@@ -112,7 +112,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) · 111 unmodified lines
112 amdgpu_job_core_dump(adev, job);112 amdgpu_job_core_dump(adev, job);
113 113
114 if (amdgpu_gpu_recovery &&114 if (amdgpu_gpu_recovery &&
115 amdgpu_ring_is_reset_type_supported(ring, AMDGPU_RESET_TYPE_SOFT_RESET) &&115 amdgpu_ring_is_reset_type_supported(ring, AMDGPU_RESET_TYPE_SOFT_RECOVERY) &&
116 amdgpu_ring_soft_recovery(ring, job->vmid, s_job->s_fence->parent)) {116 amdgpu_ring_soft_recovery(ring, job->vmid, s_job->s_fence->parent)) {
117 dev_err(adev->dev, "ring %s timeout, but soft recovered\n",117 dev_err(adev->dev, "ring %s timeout, but soft recovered\n",
118 s_job->sched->name);118 s_job->sched->name);
@@ -151,6 +151,17 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) · 32 unmodified lines
151 dev_err(adev->dev, "Ring %s reset failed\n", ring->sched.name);151 dev_err(adev->dev, "Ring %s reset failed\n", ring->sched.name);
152 }152 }
153 153
154 /* Attempt an IP block soft reset, if supported. */
155 if (amdgpu_gpu_recovery &&
156 amdgpu_ring_is_reset_type_supported(ring, AMDGPU_RESET_TYPE_IP_BLOCK_SOFT_RESET)) {
157 r = amdgpu_device_ip_soft_reset(ring, job->hw_fence);
158 if (!r) {
159 atomic_inc(&ring->adev->gpu_reset_counter);
160 drm_dev_wedged_event(adev_to_drm(adev), DRM_WEDGE_RECOVERY_NONE, info);
161 goto exit;
162 }
163 }
164
154 if (dma_fence_get_status(&s_job->s_fence->finished) == 0)165 if (dma_fence_get_status(&s_job->s_fence->finished) == 0)
155 dma_fence_set_error(&s_job->s_fence->finished, -ETIME);166 dma_fence_set_error(&s_job->s_fence->finished, -ETIME);
156 167
drivers/gpu/drm/amd/amdgpu/amdgpu_jpeg.c+5−4
@@ -134,8 +134,8 @@ void amdgpu_jpeg_ring_begin_use(struct amdgpu_ring *ring) · 133 unmodified lines
134{134{
135 struct amdgpu_device *adev = ring->adev;135 struct amdgpu_device *adev = ring->adev;
136 136
137 atomic_inc(&adev->jpeg.total_submission_cnt);137 if (!atomic_fetch_inc(&adev->jpeg.total_submission_cnt))
138 cancel_delayed_work_sync(&adev->jpeg.idle_work);138 cancel_delayed_work_sync(&adev->jpeg.idle_work);
139 139
140 mutex_lock(&adev->jpeg.jpeg_pg_lock);140 mutex_lock(&adev->jpeg.jpeg_pg_lock);
141 amdgpu_device_ip_set_powergating_state(adev, AMD_IP_BLOCK_TYPE_JPEG,141 amdgpu_device_ip_set_powergating_state(adev, AMD_IP_BLOCK_TYPE_JPEG,
@@ -145,8 +145,9 @@ void amdgpu_jpeg_ring_begin_use(struct amdgpu_ring *ring) · 3 unmodified lines
145 145
146void amdgpu_jpeg_ring_end_use(struct amdgpu_ring *ring)146void amdgpu_jpeg_ring_end_use(struct amdgpu_ring *ring)
147{147{
148 atomic_dec(&ring->adev->jpeg.total_submission_cnt);148 if (atomic_dec_and_test(&ring->adev->jpeg.total_submission_cnt))
149 schedule_delayed_work(&ring->adev->jpeg.idle_work, JPEG_IDLE_TIMEOUT);149 schedule_delayed_work(&ring->adev->jpeg.idle_work,
150 JPEG_IDLE_TIMEOUT);
150}151}
151 152
152int amdgpu_jpeg_dec_ring_test_ring(struct amdgpu_ring *ring)153int amdgpu_jpeg_dec_ring_test_ring(struct amdgpu_ring *ring)
drivers/gpu/drm/amd/amdgpu/amdgpu_jpeg.h+3−0
@@ -149,6 +149,9 @@ struct amdgpu_jpeg { · 148 unmodified lines
149 u32 *ip_dump;149 u32 *ip_dump;
150 u32 reg_count;150 u32 reg_count;
151 const struct amdgpu_hwip_reg_entry *reg_list;151 const struct amdgpu_hwip_reg_entry *reg_list;
152
153 bool disable_uq;
154 bool disable_kq;
152};155};
153 156
154int amdgpu_jpeg_sw_init(struct amdgpu_device *adev);157int amdgpu_jpeg_sw_init(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c+31−7
@@ -1423,6 +1423,33 @@ int amdgpu_info_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) · 1422 unmodified lines
1423 return 0;1423 return 0;
1424}1424}
1425 1425
1426/**
1427 * amdgpu_proc_options_ioctl - set per-fd user options
1428 *
1429 * @dev: drm dev pointer
1430 * @data: pointer to struct drm_amdgpu_proc_options
1431 * @filp: drm file
1432 *
1433 * Sets options stored on the per-file amdgpu_fpriv. Currently the only
1434 * supported option is %AMDGPU_PROC_OPTIONS_OP_KFD_SIGBUS_DELAY which
1435 * controls how KFD delivers SIGBUS for poison/RAS events to the calling
1436 * process (immediate, suppressed, or delayed by N milliseconds).
1437 */
1438int amdgpu_proc_options_ioctl(struct drm_device *dev, void *data,
1439 struct drm_file *filp)
1440{
1441 struct drm_amdgpu_proc_options *args = data;
1442
1443 switch (args->op) {
1444 case AMDGPU_PROC_OPTIONS_OP_KFD_SIGBUS_DELAY:
1445 return amdgpu_amdkfd_set_sigbus_delay(current,
1446 args->kfd_sigbus_delay.value);
1447 default:
1448 DRM_DEBUG_KMS("Invalid user option op %u\n", args->op);
1449 return -EINVAL;
1450 }
1451}
1452
1426/**1453/**
1427 * amdgpu_driver_open_kms - drm callback for open1454 * amdgpu_driver_open_kms - drm callback for open
1428 *1455 *
@@ -1504,8 +1531,7 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · 75 unmodified lines
1504 if (r)1531 if (r)
1505 goto error_vm;1532 goto error_vm;
1506 1533
1507 mutex_init(&fpriv->bo_list_lock);1534 xa_init_flags(&fpriv->bo_list_handles, XA_FLAGS_ALLOC1);
1508 idr_init_base(&fpriv->bo_list_handles, 1);
1509 1535
1510 r = amdgpu_userq_mgr_init(&fpriv->userq_mgr, file_priv, adev);1536 r = amdgpu_userq_mgr_init(&fpriv->userq_mgr, file_priv, adev);
1511 if (r)1537 if (r)
@@ -1550,8 +1576,8 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 38 unmodified lines
1550 struct amdgpu_fpriv *fpriv = file_priv->driver_priv;1576 struct amdgpu_fpriv *fpriv = file_priv->driver_priv;
1551 struct amdgpu_bo_list *list;1577 struct amdgpu_bo_list *list;
1552 struct amdgpu_bo *pd;1578 struct amdgpu_bo *pd;
1579 unsigned long handle;
1553 u32 pasid;1580 u32 pasid;
1554 int handle;
1555 1581
1556 if (!fpriv)1582 if (!fpriv)
1557 return;1583 return;
@@ -1587,11 +1613,9 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 29 unmodified lines
1587 amdgpu_pasid_free_delayed(pd->tbo.base.resv, pasid);1613 amdgpu_pasid_free_delayed(pd->tbo.base.resv, pasid);
1588 amdgpu_bo_unref(&pd);1614 amdgpu_bo_unref(&pd);
1589 1615
1590 idr_for_each_entry(&fpriv->bo_list_handles, list, handle)1616 xa_for_each(&fpriv->bo_list_handles, handle, list)
1591 amdgpu_bo_list_put(list);1617 amdgpu_bo_list_put(list);
1592 1618 xa_destroy(&fpriv->bo_list_handles);
1593 idr_destroy(&fpriv->bo_list_handles);
1594 mutex_destroy(&fpriv->bo_list_lock);
1595 1619
1596 kfree(fpriv);1620 kfree(fpriv);
1597 file_priv->driver_priv = NULL;1621 file_priv->driver_priv = NULL;
drivers/gpu/drm/amd/amdgpu/amdgpu_mca.c+0−486
@@ -27,16 +27,6 @@ · 26 unmodified lines
27#include "umc/umc_6_7_0_offset.h"27#include "umc/umc_6_7_0_offset.h"
28#include "umc/umc_6_7_0_sh_mask.h"28#include "umc/umc_6_7_0_sh_mask.h"
29 29
30static bool amdgpu_mca_is_deferred_error(struct amdgpu_device *adev,
31 uint64_t mc_status)
32{
33 if (adev->umc.ras->check_ecc_err_status)
34 return adev->umc.ras->check_ecc_err_status(adev,
35 AMDGPU_MCA_ERROR_TYPE_DE, &mc_status);
36
37 return false;
38}
39
40void amdgpu_mca_query_correctable_error_count(struct amdgpu_device *adev,30void amdgpu_mca_query_correctable_error_count(struct amdgpu_device *adev,
41 uint64_t mc_status_addr,31 uint64_t mc_status_addr,
42 unsigned long *error_count)32 unsigned long *error_count)
@@ -155,479 +145,3 @@ int amdgpu_mca_mpio_ras_sw_init(struct amdgpu_device *adev) · 112 unmodified lines
155 145
156 return 0;146 return 0;
157}147}
158
159static void amdgpu_mca_bank_set_init(struct mca_bank_set *mca_set)
160{
161 if (!mca_set)
162 return;
163
164 memset(mca_set, 0, sizeof(*mca_set));
165 INIT_LIST_HEAD(&mca_set->list);
166}
167
168static int amdgpu_mca_bank_set_add_entry(struct mca_bank_set *mca_set, struct mca_bank_entry *entry)
169{
170 struct mca_bank_node *node;
171
172 if (!entry)
173 return -EINVAL;
174
175 node = kvzalloc_obj(*node);
176 if (!node)
177 return -ENOMEM;
178
179 memcpy(&node->entry, entry, sizeof(*entry));
180
181 INIT_LIST_HEAD(&node->node);
182 list_add_tail(&node->node, &mca_set->list);
183
184 mca_set->nr_entries++;
185
186 return 0;
187}
188
189static int amdgpu_mca_bank_set_merge(struct mca_bank_set *mca_set, struct mca_bank_set *new)
190{
191 struct mca_bank_node *node;
192
193 list_for_each_entry(node, &new->list, node)
194 amdgpu_mca_bank_set_add_entry(mca_set, &node->entry);
195
196 return 0;
197}
198
199static void amdgpu_mca_bank_set_remove_node(struct mca_bank_set *mca_set, struct mca_bank_node *node)
200{
201 if (!node)
202 return;
203
204 list_del(&node->node);
205 kvfree(node);
206
207 mca_set->nr_entries--;
208}
209
210static void amdgpu_mca_bank_set_release(struct mca_bank_set *mca_set)
211{
212 struct mca_bank_node *node, *tmp;
213
214 if (list_empty(&mca_set->list))
215 return;
216
217 list_for_each_entry_safe(node, tmp, &mca_set->list, node)
218 amdgpu_mca_bank_set_remove_node(mca_set, node);
219}
220
221void amdgpu_mca_smu_init_funcs(struct amdgpu_device *adev, const struct amdgpu_mca_smu_funcs *mca_funcs)
222{
223 struct amdgpu_mca *mca = &adev->mca;
224
225 mca->mca_funcs = mca_funcs;
226}
227
228int amdgpu_mca_init(struct amdgpu_device *adev)
229{
230 struct amdgpu_mca *mca = &adev->mca;
231 struct mca_bank_cache *mca_cache;
232 int i;
233
234 atomic_set(&mca->ue_update_flag, 0);
235
236 for (i = 0; i < ARRAY_SIZE(mca->mca_caches); i++) {
237 mca_cache = &mca->mca_caches[i];
238 mutex_init(&mca_cache->lock);
239 amdgpu_mca_bank_set_init(&mca_cache->mca_set);
240 }
241
242 return 0;
243}
244
245void amdgpu_mca_fini(struct amdgpu_device *adev)
246{
247 struct amdgpu_mca *mca = &adev->mca;
248 struct mca_bank_cache *mca_cache;
249 int i;
250
251 atomic_set(&mca->ue_update_flag, 0);
252
253 for (i = 0; i < ARRAY_SIZE(mca->mca_caches); i++) {
254 mca_cache = &mca->mca_caches[i];
255 amdgpu_mca_bank_set_release(&mca_cache->mca_set);
256 mutex_destroy(&mca_cache->lock);
257 }
258}
259
260int amdgpu_mca_reset(struct amdgpu_device *adev)
261{
262 amdgpu_mca_fini(adev);
263
264 return amdgpu_mca_init(adev);
265}
266
267int amdgpu_mca_smu_set_debug_mode(struct amdgpu_device *adev, bool enable)
268{
269 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
270
271 if (mca_funcs && mca_funcs->mca_set_debug_mode)
272 return mca_funcs->mca_set_debug_mode(adev, enable);
273
274 return -EOPNOTSUPP;
275}
276
277static void amdgpu_mca_smu_mca_bank_dump(struct amdgpu_device *adev, int idx, struct mca_bank_entry *entry,
278 struct ras_query_context *qctx)
279{
280 u64 event_id = qctx ? qctx->evid.event_id : RAS_EVENT_INVALID_ID;
281
282 RAS_EVENT_LOG(adev, event_id, HW_ERR "Accelerator Check Architecture events logged\n");
283 RAS_EVENT_LOG(adev, event_id, HW_ERR "aca entry[%02d].STATUS=0x%016llx\n",
284 idx, entry->regs[MCA_REG_IDX_STATUS]);
285 RAS_EVENT_LOG(adev, event_id, HW_ERR "aca entry[%02d].ADDR=0x%016llx\n",
286 idx, entry->regs[MCA_REG_IDX_ADDR]);
287 RAS_EVENT_LOG(adev, event_id, HW_ERR "aca entry[%02d].MISC0=0x%016llx\n",
288 idx, entry->regs[MCA_REG_IDX_MISC0]);
289 RAS_EVENT_LOG(adev, event_id, HW_ERR "aca entry[%02d].IPID=0x%016llx\n",
290 idx, entry->regs[MCA_REG_IDX_IPID]);
291 RAS_EVENT_LOG(adev, event_id, HW_ERR "aca entry[%02d].SYND=0x%016llx\n",
292 idx, entry->regs[MCA_REG_IDX_SYND]);
293}
294
295static int amdgpu_mca_smu_get_valid_mca_count(struct amdgpu_device *adev, enum amdgpu_mca_error_type type, uint32_t *count)
296{
297 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
298
299 if (!count)
300 return -EINVAL;
301
302 if (mca_funcs && mca_funcs->mca_get_valid_mca_count)
303 return mca_funcs->mca_get_valid_mca_count(adev, type, count);
304
305 return -EOPNOTSUPP;
306}
307
308static int amdgpu_mca_smu_get_mca_entry(struct amdgpu_device *adev, enum amdgpu_mca_error_type type,
309 int idx, struct mca_bank_entry *entry)
310{
311 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
312 int count;
313
314 if (!mca_funcs || !mca_funcs->mca_get_mca_entry)
315 return -EOPNOTSUPP;
316
317 switch (type) {
318 case AMDGPU_MCA_ERROR_TYPE_UE:
319 count = mca_funcs->max_ue_count;
320 break;
321 case AMDGPU_MCA_ERROR_TYPE_CE:
322 count = mca_funcs->max_ce_count;
323 break;
324 default:
325 return -EINVAL;
326 }
327
328 if (idx >= count)
329 return -EINVAL;
330
331 return mca_funcs->mca_get_mca_entry(adev, type, idx, entry);
332}
333
334static bool amdgpu_mca_bank_should_update(struct amdgpu_device *adev, enum amdgpu_mca_error_type type)
335{
336 struct amdgpu_mca *mca = &adev->mca;
337 bool ret = true;
338
339 /*
340 * Because the UE Valid MCA count will only be cleared after reset,
341 * in order to avoid repeated counting of the error count,
342 * the aca bank is only updated once during the gpu recovery stage.
343 */
344 if (type == AMDGPU_MCA_ERROR_TYPE_UE) {
345 if (amdgpu_ras_intr_triggered())
346 ret = atomic_cmpxchg(&mca->ue_update_flag, 0, 1) == 0;
347 else
348 atomic_set(&mca->ue_update_flag, 0);
349 }
350
351 return ret;
352}
353
354static bool amdgpu_mca_bank_should_dump(struct amdgpu_device *adev, enum amdgpu_mca_error_type type,
355 struct mca_bank_entry *entry)
356{
357 bool ret;
358
359 switch (type) {
360 case AMDGPU_MCA_ERROR_TYPE_CE:
361 ret = amdgpu_mca_is_deferred_error(adev, entry->regs[MCA_REG_IDX_STATUS]);
362 break;
363 case AMDGPU_MCA_ERROR_TYPE_UE:
364 default:
365 ret = true;
366 break;
367 }
368
369 return ret;
370}
371
372static int amdgpu_mca_smu_get_mca_set(struct amdgpu_device *adev, enum amdgpu_mca_error_type type, struct mca_bank_set *mca_set,
373 struct ras_query_context *qctx)
374{
375 struct mca_bank_entry entry;
376 uint32_t count = 0, i;
377 int ret;
378
379 if (!mca_set)
380 return -EINVAL;
381
382 if (!amdgpu_mca_bank_should_update(adev, type))
383 return 0;
384
385 ret = amdgpu_mca_smu_get_valid_mca_count(adev, type, &count);
386 if (ret)
387 return ret;
388
389 for (i = 0; i < count; i++) {
390 memset(&entry, 0, sizeof(entry));
391 ret = amdgpu_mca_smu_get_mca_entry(adev, type, i, &entry);
392 if (ret)
393 return ret;
394
395 amdgpu_mca_bank_set_add_entry(mca_set, &entry);
396
397 if (amdgpu_mca_bank_should_dump(adev, type, &entry))
398 amdgpu_mca_smu_mca_bank_dump(adev, i, &entry, qctx);
399 }
400
401 return 0;
402}
403
404static int amdgpu_mca_smu_parse_mca_error_count(struct amdgpu_device *adev, enum amdgpu_ras_block blk,
405 enum amdgpu_mca_error_type type, struct mca_bank_entry *entry, uint32_t *count)
406{
407 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
408
409 if (!count || !entry)
410 return -EINVAL;
411
412 if (!mca_funcs || !mca_funcs->mca_parse_mca_error_count)
413 return -EOPNOTSUPP;
414
415 return mca_funcs->mca_parse_mca_error_count(adev, blk, type, entry, count);
416}
417
418static int amdgpu_mca_dispatch_mca_set(struct amdgpu_device *adev, enum amdgpu_ras_block blk, enum amdgpu_mca_error_type type,
419 struct mca_bank_set *mca_set, struct ras_err_data *err_data)
420{
421 struct amdgpu_smuio_mcm_config_info mcm_info;
422 struct mca_bank_node *node, *tmp;
423 struct mca_bank_entry *entry;
424 uint32_t count;
425 int ret;
426
427 if (!mca_set)
428 return -EINVAL;
429
430 if (!mca_set->nr_entries)
431 return 0;
432
433 list_for_each_entry_safe(node, tmp, &mca_set->list, node) {
434 entry = &node->entry;
435
436 count = 0;
437 ret = amdgpu_mca_smu_parse_mca_error_count(adev, blk, type, entry, &count);
438 if (ret && ret != -EOPNOTSUPP)
439 return ret;
440
441 if (!count)
442 continue;
443
444 memset(&mcm_info, 0, sizeof(mcm_info));
445
446 mcm_info.socket_id = entry->info.socket_id;
447 mcm_info.die_id = entry->info.aid;
448
449 if (type == AMDGPU_MCA_ERROR_TYPE_UE) {
450 amdgpu_ras_error_statistic_ue_count(err_data,
451 &mcm_info, (uint64_t)count);
452 } else {
453 if (amdgpu_mca_is_deferred_error(adev, entry->regs[MCA_REG_IDX_STATUS]))
454 amdgpu_ras_error_statistic_de_count(err_data,
455 &mcm_info, (uint64_t)count);
456 else
457 amdgpu_ras_error_statistic_ce_count(err_data,
458 &mcm_info, (uint64_t)count);
459 }
460
461 amdgpu_mca_bank_set_remove_node(mca_set, node);
462 }
463
464 return 0;
465}
466
467static int amdgpu_mca_add_mca_set_to_cache(struct amdgpu_device *adev, enum amdgpu_mca_error_type type, struct mca_bank_set *new)
468{
469 struct mca_bank_cache *mca_cache = &adev->mca.mca_caches[type];
470 int ret;
471
472 mutex_lock(&mca_cache->lock);
473 ret = amdgpu_mca_bank_set_merge(&mca_cache->mca_set, new);
474 mutex_unlock(&mca_cache->lock);
475
476 return ret;
477}
478
479int amdgpu_mca_smu_log_ras_error(struct amdgpu_device *adev, enum amdgpu_ras_block blk, enum amdgpu_mca_error_type type,
480 struct ras_err_data *err_data, struct ras_query_context *qctx)
481{
482 struct mca_bank_set mca_set;
483 struct mca_bank_cache *mca_cache = &adev->mca.mca_caches[type];
484 int ret;
485
486 amdgpu_mca_bank_set_init(&mca_set);
487
488 ret = amdgpu_mca_smu_get_mca_set(adev, type, &mca_set, qctx);
489 if (ret)
490 goto out_mca_release;
491
492 ret = amdgpu_mca_dispatch_mca_set(adev, blk, type, &mca_set, err_data);
493 if (ret)
494 goto out_mca_release;
495
496 /* add remain mca bank to mca cache */
497 if (mca_set.nr_entries) {
498 ret = amdgpu_mca_add_mca_set_to_cache(adev, type, &mca_set);
499 if (ret)
500 goto out_mca_release;
501 }
502
503 /* dispatch mca set again if mca cache has valid data */
504 mutex_lock(&mca_cache->lock);
505 if (mca_cache->mca_set.nr_entries)
506 ret = amdgpu_mca_dispatch_mca_set(adev, blk, type, &mca_cache->mca_set, err_data);
507 mutex_unlock(&mca_cache->lock);
508
509out_mca_release:
510 amdgpu_mca_bank_set_release(&mca_set);
511
512 return ret;
513}
514
515#if defined(CONFIG_DEBUG_FS)
516static int amdgpu_mca_smu_debug_mode_set(void *data, u64 val)
517{
518 struct amdgpu_device *adev = (struct amdgpu_device *)data;
519 int ret;
520
521 ret = amdgpu_ras_set_mca_debug_mode(adev, val ? true : false);
522 if (ret)
523 return ret;
524
525 dev_info(adev->dev, "amdgpu set smu mca debug mode %s success\n", val ? "on" : "off");
526
527 return 0;
528}
529
530static void mca_dump_entry(struct seq_file *m, struct mca_bank_entry *entry)
531{
532 int i, idx = entry->idx;
533 int reg_idx_array[] = {
534 MCA_REG_IDX_STATUS,
535 MCA_REG_IDX_ADDR,
536 MCA_REG_IDX_MISC0,
537 MCA_REG_IDX_IPID,
538 MCA_REG_IDX_SYND,
539 };
540
541 seq_printf(m, "mca entry[%d].type: %s\n", idx, entry->type == AMDGPU_MCA_ERROR_TYPE_UE ? "UE" : "CE");
542 seq_printf(m, "mca entry[%d].ip: %d\n", idx, entry->ip);
543 seq_printf(m, "mca entry[%d].info: socketid:%d aid:%d hwid:0x%03x mcatype:0x%04x\n",
544 idx, entry->info.socket_id, entry->info.aid, entry->info.hwid, entry->info.mcatype);
545
546 for (i = 0; i < ARRAY_SIZE(reg_idx_array); i++)
547 seq_printf(m, "mca entry[%d].regs[%d]: 0x%016llx\n", idx, reg_idx_array[i], entry->regs[reg_idx_array[i]]);
548}
549
550static int mca_dump_show(struct seq_file *m, enum amdgpu_mca_error_type type)
551{
552 struct amdgpu_device *adev = (struct amdgpu_device *)m->private;
553 struct mca_bank_node *node;
554 struct mca_bank_set mca_set;
555 struct ras_query_context qctx;
556 int ret;
557
558 amdgpu_mca_bank_set_init(&mca_set);
559
560 qctx.evid.event_id = RAS_EVENT_INVALID_ID;
561 ret = amdgpu_mca_smu_get_mca_set(adev, type, &mca_set, &qctx);
562 if (ret)
563 goto err_free_mca_set;
564
565 seq_printf(m, "amdgpu smu %s valid mca count: %d\n",
566 type == AMDGPU_MCA_ERROR_TYPE_UE ? "UE" : "CE", mca_set.nr_entries);
567
568 if (!mca_set.nr_entries)
569 goto err_free_mca_set;
570
571 list_for_each_entry(node, &mca_set.list, node)
572 mca_dump_entry(m, &node->entry);
573
574 /* add mca bank to mca bank cache */
575 ret = amdgpu_mca_add_mca_set_to_cache(adev, type, &mca_set);
576
577err_free_mca_set:
578 amdgpu_mca_bank_set_release(&mca_set);
579
580 return ret;
581}
582
583static int mca_dump_ce_show(struct seq_file *m, void *unused)
584{
585 return mca_dump_show(m, AMDGPU_MCA_ERROR_TYPE_CE);
586}
587
588static int mca_dump_ce_open(struct inode *inode, struct file *file)
589{
590 return single_open(file, mca_dump_ce_show, inode->i_private);
591}
592
593static const struct file_operations mca_ce_dump_debug_fops = {
594 .owner = THIS_MODULE,
595 .open = mca_dump_ce_open,
596 .read = seq_read,
597 .llseek = seq_lseek,
598 .release = single_release,
599};
600
601static int mca_dump_ue_show(struct seq_file *m, void *unused)
602{
603 return mca_dump_show(m, AMDGPU_MCA_ERROR_TYPE_UE);
604}
605
606static int mca_dump_ue_open(struct inode *inode, struct file *file)
607{
608 return single_open(file, mca_dump_ue_show, inode->i_private);
609}
610
611static const struct file_operations mca_ue_dump_debug_fops = {
612 .owner = THIS_MODULE,
613 .open = mca_dump_ue_open,
614 .read = seq_read,
615 .llseek = seq_lseek,
616 .release = single_release,
617};
618
619DEFINE_DEBUGFS_ATTRIBUTE(mca_debug_mode_fops, NULL, amdgpu_mca_smu_debug_mode_set, "%llu\n");
620#endif
621
622void amdgpu_mca_smu_debugfs_init(struct amdgpu_device *adev, struct dentry *root)
623{
624#if defined(CONFIG_DEBUG_FS)
625 if (!root)
626 return;
627
628 debugfs_create_file("mca_debug_mode", 0200, root, adev, &mca_debug_mode_fops);
629 debugfs_create_file("mca_ue_dump", 0400, root, adev, &mca_ue_dump_debug_fops);
630 debugfs_create_file("mca_ce_dump", 0400, root, adev, &mca_ce_dump_debug_fops);
631#endif
632}
633
drivers/gpu/drm/amd/amdgpu/amdgpu_mca.h+0−107
@@ -23,45 +23,6 @@ · 22 unmodified lines
23 23
24#include "amdgpu_ras.h"24#include "amdgpu_ras.h"
25 25
26#define MCA_MAX_REGS_COUNT (16)
27
28#define MCA_REG_FIELD(x, h, l) (((x) & GENMASK_ULL(h, l)) >> l)
29#define MCA_REG__STATUS__VAL(x) MCA_REG_FIELD(x, 63, 63)
30#define MCA_REG__STATUS__OVERFLOW(x) MCA_REG_FIELD(x, 62, 62)
31#define MCA_REG__STATUS__UC(x) MCA_REG_FIELD(x, 61, 61)
32#define MCA_REG__STATUS__EN(x) MCA_REG_FIELD(x, 60, 60)
33#define MCA_REG__STATUS__MISCV(x) MCA_REG_FIELD(x, 59, 59)
34#define MCA_REG__STATUS__ADDRV(x) MCA_REG_FIELD(x, 58, 58)
35#define MCA_REG__STATUS__PCC(x) MCA_REG_FIELD(x, 57, 57)
36#define MCA_REG__STATUS__ERRCOREIDVAL(x) MCA_REG_FIELD(x, 56, 56)
37#define MCA_REG__STATUS__TCC(x) MCA_REG_FIELD(x, 55, 55)
38#define MCA_REG__STATUS__SYNDV(x) MCA_REG_FIELD(x, 53, 53)
39#define MCA_REG__STATUS__CECC(x) MCA_REG_FIELD(x, 46, 46)
40#define MCA_REG__STATUS__UECC(x) MCA_REG_FIELD(x, 45, 45)
41#define MCA_REG__STATUS__DEFERRED(x) MCA_REG_FIELD(x, 44, 44)
42#define MCA_REG__STATUS__POISON(x) MCA_REG_FIELD(x, 43, 43)
43#define MCA_REG__STATUS__SCRUB(x) MCA_REG_FIELD(x, 40, 40)
44#define MCA_REG__STATUS__ERRCOREID(x) MCA_REG_FIELD(x, 37, 32)
45#define MCA_REG__STATUS__ADDRLSB(x) MCA_REG_FIELD(x, 29, 24)
46#define MCA_REG__STATUS__ERRORCODEEXT(x) MCA_REG_FIELD(x, 21, 16)
47#define MCA_REG__STATUS__ERRORCODE(x) MCA_REG_FIELD(x, 15, 0)
48
49#define MCA_REG__MISC0__ERRCNT(x) MCA_REG_FIELD(x, 43, 32)
50
51#define MCA_REG__SYND__ERRORINFORMATION(x) MCA_REG_FIELD(x, 17, 0)
52
53enum amdgpu_mca_ip {
54 AMDGPU_MCA_IP_UNKNOW = -1,
55 AMDGPU_MCA_IP_PSP = 0,
56 AMDGPU_MCA_IP_SDMA,
57 AMDGPU_MCA_IP_GC,
58 AMDGPU_MCA_IP_SMU,
59 AMDGPU_MCA_IP_MP5,
60 AMDGPU_MCA_IP_UMC,
61 AMDGPU_MCA_IP_PCS_XGMI,
62 AMDGPU_MCA_IP_COUNT,
63};
64
65enum amdgpu_mca_error_type {26enum amdgpu_mca_error_type {
66 AMDGPU_MCA_ERROR_TYPE_UE = 0,27 AMDGPU_MCA_ERROR_TYPE_UE = 0,
67 AMDGPU_MCA_ERROR_TYPE_CE,28 AMDGPU_MCA_ERROR_TYPE_CE,
@@ -77,77 +38,20 @@ struct amdgpu_mca_ras { · 9 unmodified lines
77 struct amdgpu_mca_ras_block *ras;38 struct amdgpu_mca_ras_block *ras;
78};39};
79 40
80struct mca_bank_set {
81 int nr_entries;
82 struct list_head list;
83};
84
85struct mca_bank_cache {
86 struct mca_bank_set mca_set;
87 struct mutex lock;
88};
89
90struct amdgpu_mca {41struct amdgpu_mca {
91 struct amdgpu_mca_ras mp0;42 struct amdgpu_mca_ras mp0;
92 struct amdgpu_mca_ras mp1;43 struct amdgpu_mca_ras mp1;
93 struct amdgpu_mca_ras mpio;44 struct amdgpu_mca_ras mpio;
94 const struct amdgpu_mca_smu_funcs *mca_funcs;
95 struct mca_bank_cache mca_caches[AMDGPU_MCA_ERROR_TYPE_DE];
96 atomic_t ue_update_flag;
97};
98
99enum mca_reg_idx {
100 MCA_REG_IDX_STATUS = 1,
101 MCA_REG_IDX_ADDR = 2,
102 MCA_REG_IDX_MISC0 = 3,
103 MCA_REG_IDX_IPID = 5,
104 MCA_REG_IDX_SYND = 6,
105 MCA_REG_IDX_COUNT = 16,
106};
107
108struct mca_bank_info {
109 int socket_id;
110 int aid;
111 int hwid;
112 int mcatype;
113};
114
115struct mca_bank_entry {
116 int idx;
117 enum amdgpu_mca_error_type type;
118 enum amdgpu_mca_ip ip;
119 struct mca_bank_info info;
120 uint64_t regs[MCA_MAX_REGS_COUNT];
121};
122
123struct mca_bank_node {
124 struct mca_bank_entry entry;
125 struct list_head node;
126};
127
128struct amdgpu_mca_smu_funcs {
129 int max_ue_count;
130 int max_ce_count;
131 int (*mca_set_debug_mode)(struct amdgpu_device *adev, bool enable);
132 int (*mca_parse_mca_error_count)(struct amdgpu_device *adev, enum amdgpu_ras_block blk, enum amdgpu_mca_error_type type,
133 struct mca_bank_entry *entry, uint32_t *count);
134 int (*mca_get_valid_mca_count)(struct amdgpu_device *adev, enum amdgpu_mca_error_type type,
135 uint32_t *count);
136 int (*mca_get_mca_entry)(struct amdgpu_device *adev, enum amdgpu_mca_error_type type,
137 int idx, struct mca_bank_entry *entry);
138};45};
139 46
140void amdgpu_mca_query_correctable_error_count(struct amdgpu_device *adev,47void amdgpu_mca_query_correctable_error_count(struct amdgpu_device *adev,
141 uint64_t mc_status_addr,48 uint64_t mc_status_addr,
142 unsigned long *error_count);49 unsigned long *error_count);
143
144void amdgpu_mca_query_uncorrectable_error_count(struct amdgpu_device *adev,50void amdgpu_mca_query_uncorrectable_error_count(struct amdgpu_device *adev,
145 uint64_t mc_status_addr,51 uint64_t mc_status_addr,
146 unsigned long *error_count);52 unsigned long *error_count);
147
148void amdgpu_mca_reset_error_count(struct amdgpu_device *adev,53void amdgpu_mca_reset_error_count(struct amdgpu_device *adev,
149 uint64_t mc_status_addr);54 uint64_t mc_status_addr);
150
151void amdgpu_mca_query_ras_error_count(struct amdgpu_device *adev,55void amdgpu_mca_query_ras_error_count(struct amdgpu_device *adev,
152 uint64_t mc_status_addr,56 uint64_t mc_status_addr,
153 void *ras_error_status);57 void *ras_error_status);
@@ -155,15 +59,4 @@ int amdgpu_mca_mp0_ras_sw_init(struct amdgpu_device *adev); · 1 unmodified line
155int amdgpu_mca_mp1_ras_sw_init(struct amdgpu_device *adev);59int amdgpu_mca_mp1_ras_sw_init(struct amdgpu_device *adev);
156int amdgpu_mca_mpio_ras_sw_init(struct amdgpu_device *adev);60int amdgpu_mca_mpio_ras_sw_init(struct amdgpu_device *adev);
157 61
158void amdgpu_mca_smu_init_funcs(struct amdgpu_device *adev, const struct amdgpu_mca_smu_funcs *mca_funcs);
159int amdgpu_mca_init(struct amdgpu_device *adev);
160void amdgpu_mca_fini(struct amdgpu_device *adev);
161int amdgpu_mca_reset(struct amdgpu_device *adev);
162int amdgpu_mca_smu_set_debug_mode(struct amdgpu_device *adev, bool enable);
163int amdgpu_mca_smu_get_mca_set_error_count(struct amdgpu_device *adev, enum amdgpu_ras_block blk,
164 enum amdgpu_mca_error_type type, uint32_t *total);
165void amdgpu_mca_smu_debugfs_init(struct amdgpu_device *adev, struct dentry *root);
166int amdgpu_mca_smu_log_ras_error(struct amdgpu_device *adev, enum amdgpu_ras_block blk, enum amdgpu_mca_error_type type,
167 struct ras_err_data *err_data, struct ras_query_context *qctx);
168
169#endif62#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.c+67−3
@@ -217,7 +217,7 @@ int amdgpu_mes_init(struct amdgpu_device *adev) · 216 unmodified lines
217 if (r)217 if (r)
218 goto error_doorbell;218 goto error_doorbell;
219 219
220 if (amdgpu_ip_version(adev, GC_HWIP, 0) >= IP_VERSION(12, 1, 0)) {220 if (amdgpu_ip_version(adev, GC_HWIP, 0) >= IP_VERSION(11, 0, 0)) {
221 /* When queue/pipe reset is done in MES instead of in the221 /* When queue/pipe reset is done in MES instead of in the
222 * driver, MES passes hung queues information to the driver in222 * driver, MES passes hung queues information to the driver in
223 * hung_queue_hqd_info. Calculate required space to store this223 * hung_queue_hqd_info. Calculate required space to store this
@@ -252,6 +252,10 @@ int amdgpu_mes_init(struct amdgpu_device *adev) · 28 unmodified lines
252 }252 }
253 }253 }
254 254
255 adev->gfx.mec.mes_hung_db_array =
256 kcalloc(amdgpu_mes_get_hung_queue_db_array_size(adev),
257 sizeof(u32), GFP_KERNEL);
258
255 return 0;259 return 0;
256 260
257error_doorbell:261error_doorbell:
@@ -279,6 +283,8 @@ void amdgpu_mes_fini(struct amdgpu_device *adev) · 21 unmodified lines
279 int i;283 int i;
280 int num_xcc = adev->gfx.xcc_mask ? NUM_XCC(adev->gfx.xcc_mask) : 1;284 int num_xcc = adev->gfx.xcc_mask ? NUM_XCC(adev->gfx.xcc_mask) : 1;
281 285
286 kfree(adev->gfx.mec.mes_hung_db_array);
287
282 amdgpu_bo_free_kernel(&adev->mes.event_log_gpu_obj,288 amdgpu_bo_free_kernel(&adev->mes.event_log_gpu_obj,
283 &adev->mes.event_log_gpu_addr,289 &adev->mes.event_log_gpu_addr,
284 &adev->mes.event_log_cpu_addr);290 &adev->mes.event_log_cpu_addr);
@@ -439,6 +445,59 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 154 unmodified lines
439 return r;445 return r;
440}446}
441 447
448int amdgpu_mes_reset_queue_mmio(struct amdgpu_device *adev,
449 int queue_type,
450 unsigned int vmid,
451 unsigned int me,
452 unsigned int pipe,
453 unsigned int queue,
454 uint32_t xcc_id)
455{
456 struct mes_reset_queue_input queue_input;
457 int r;
458
459 memset(&queue_input, 0, sizeof(queue_input));
460
461 queue_input.xcc_id = xcc_id;
462 queue_input.me_id = me;
463 queue_input.pipe_id = pipe;
464 queue_input.queue_id = queue;
465 queue_input.vmid = vmid;
466 queue_input.queue_type = queue_type;
467 queue_input.use_mmio = true;
468
469 amdgpu_mes_lock(&adev->mes);
470 r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
471 amdgpu_mes_unlock(&adev->mes);
472 if (r)
473 dev_err(adev->dev, "failed to reset legacy queue\n");
474
475 return r;
476}
477
478int amdgpu_mes_reset_user_queue(struct amdgpu_device *adev,
479 int queue_type,
480 unsigned int doorbell_index,
481 unsigned int xcc_id)
482{
483 struct mes_reset_queue_input queue_input;
484 int r;
485
486 memset(&queue_input, 0, sizeof(queue_input));
487
488 queue_input.xcc_id = xcc_id;
489 queue_input.queue_type = queue_type;
490 queue_input.doorbell_offset = doorbell_index;
491
492 amdgpu_mes_lock(&adev->mes);
493 r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
494 amdgpu_mes_unlock(&adev->mes);
495 if (r)
496 dev_err(adev->dev, "failed to reset user queue\n");
497
498 return r;
499}
500
442int amdgpu_mes_get_hung_queue_db_array_size(struct amdgpu_device *adev)501int amdgpu_mes_get_hung_queue_db_array_size(struct amdgpu_device *adev)
443{502{
444 return adev->mes.hung_queue_db_array_size;503 return adev->mes.hung_queue_db_array_size;
@@ -805,8 +864,13 @@ bool amdgpu_mes_suspend_resume_all_supported(struct amdgpu_device *adev) · 360 unmodified lines
805 864
806bool amdgpu_mes_queue_reset_by_mes_supported(struct amdgpu_device *adev)865bool amdgpu_mes_queue_reset_by_mes_supported(struct amdgpu_device *adev)
807{866{
808 return (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 1, 0) &&867 u32 ip_maj = IP_VERSION_MAJ(amdgpu_ip_version(adev, GC_HWIP, 0));
809 (adev->mes.sched_version & AMDGPU_MES_VERSION_MASK) >= 0x73);868 u32 ip_min = IP_VERSION_MIN(amdgpu_ip_version(adev, GC_HWIP, 0));
869 u32 mes_sched = adev->mes.sched_version & AMDGPU_MES_VERSION_MASK;
870
871 return (ip_maj == 11 && mes_sched >= 0x8c) ||
872 ((ip_maj == 12 && ip_min == 0) && mes_sched >= 0x8d) ||
873 ((ip_maj == 12 && ip_min == 1) && mes_sched >= 0x73);
810}874}
811 875
812/* Fix me -- node_id is used to identify the correct MES instances in the future */876/* Fix me -- node_id is used to identify the correct MES instances in the future */
drivers/gpu/drm/amd/amdgpu/amdgpu_mes.h+15−0
@@ -168,6 +168,9 @@ struct amdgpu_mes { · 167 unmodified lines
168 int master_xcc_ids[AMDGPU_MAX_MES_INST_PIPES];168 int master_xcc_ids[AMDGPU_MAX_MES_INST_PIPES];
169 struct amdgpu_bo *shared_cmd_buf_obj[AMDGPU_MAX_MES_INST_PIPES];169 struct amdgpu_bo *shared_cmd_buf_obj[AMDGPU_MAX_MES_INST_PIPES];
170 uint64_t shared_cmd_buf_gpu_addr[AMDGPU_MAX_MES_INST_PIPES];170 uint64_t shared_cmd_buf_gpu_addr[AMDGPU_MAX_MES_INST_PIPES];
171
172 bool compute_pipe_reset_enabled;
173 bool gfx_pipe_reset_enabled;
171};174};
172 175
173struct amdgpu_mes_hung_queue_hqd_info {176struct amdgpu_mes_hung_queue_hqd_info {
@@ -271,6 +274,7 @@ struct mes_remove_queue_input { · 97 unmodified lines
271 uint32_t xcc_id;274 uint32_t xcc_id;
272 uint32_t doorbell_offset;275 uint32_t doorbell_offset;
273 uint64_t gang_context_addr;276 uint64_t gang_context_addr;
277 uint32_t queue_type;
274 bool remove_queue_after_reset;278 bool remove_queue_after_reset;
275};279};
276 280
@@ -461,6 +465,17 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 184 unmodified lines
461 unsigned int vmid,465 unsigned int vmid,
462 bool use_mmio,466 bool use_mmio,
463 uint32_t xcc_id);467 uint32_t xcc_id);
468int amdgpu_mes_reset_queue_mmio(struct amdgpu_device *adev,
469 int queue_type,
470 unsigned int vmid,
471 unsigned int me,
472 unsigned int pipe,
473 unsigned int queue,
474 uint32_t xcc_id);
475int amdgpu_mes_reset_user_queue(struct amdgpu_device *adev,
476 int queue_type,
477 unsigned int doorbell_index,
478 unsigned int xcc_id);
464 479
465int amdgpu_mes_get_hung_queue_db_array_size(struct amdgpu_device *adev);480int amdgpu_mes_get_hung_queue_db_array_size(struct amdgpu_device *adev);
466int amdgpu_mes_detect_and_reset_hung_queues(struct amdgpu_device *adev,481int amdgpu_mes_detect_and_reset_hung_queues(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/amdgpu_mmhub.h+0−23
@@ -21,29 +21,6 @@ · 20 unmodified lines
21#ifndef __AMDGPU_MMHUB_H__21#ifndef __AMDGPU_MMHUB_H__
22#define __AMDGPU_MMHUB_H__22#define __AMDGPU_MMHUB_H__
23 23
24enum amdgpu_mmhub_ras_memory_id {
25 AMDGPU_MMHUB_WGMI_PAGEMEM = 0,
26 AMDGPU_MMHUB_RGMI_PAGEMEM = 1,
27 AMDGPU_MMHUB_WDRAM_PAGEMEM = 2,
28 AMDGPU_MMHUB_RDRAM_PAGEMEM = 3,
29 AMDGPU_MMHUB_WIO_CMDMEM = 4,
30 AMDGPU_MMHUB_RIO_CMDMEM = 5,
31 AMDGPU_MMHUB_WGMI_CMDMEM = 6,
32 AMDGPU_MMHUB_RGMI_CMDMEM = 7,
33 AMDGPU_MMHUB_WDRAM_CMDMEM = 8,
34 AMDGPU_MMHUB_RDRAM_CMDMEM = 9,
35 AMDGPU_MMHUB_MAM_DMEM0 = 10,
36 AMDGPU_MMHUB_MAM_DMEM1 = 11,
37 AMDGPU_MMHUB_MAM_DMEM2 = 12,
38 AMDGPU_MMHUB_MAM_DMEM3 = 13,
39 AMDGPU_MMHUB_WRET_TAGMEM = 19,
40 AMDGPU_MMHUB_RRET_TAGMEM = 20,
41 AMDGPU_MMHUB_WIO_DATAMEM = 21,
42 AMDGPU_MMHUB_WGMI_DATAMEM = 22,
43 AMDGPU_MMHUB_WDRAM_DATAMEM = 23,
44 AMDGPU_MMHUB_MEMORY_BLOCK_LAST,
45};
46
47struct amdgpu_mmhub_ras {24struct amdgpu_mmhub_ras {
48 struct amdgpu_ras_block_object ras_block;25 struct amdgpu_ras_block_object ras_block;
49};26};
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h+0−40
@@ -312,46 +312,6 @@ uint32_t amdgpu_bo_mem_stats_placement(struct amdgpu_bo *bo); · 311 unmodified lines
312uint32_t amdgpu_bo_get_preferred_domain(struct amdgpu_device *adev,312uint32_t amdgpu_bo_get_preferred_domain(struct amdgpu_device *adev,
313 uint32_t domain);313 uint32_t domain);
314 314
315/*
316 * sub allocation
317 */
318static inline struct amdgpu_sa_manager *
319to_amdgpu_sa_manager(struct drm_suballoc_manager *manager)
320{
321 return container_of(manager, struct amdgpu_sa_manager, base);
322}
323
324static inline uint64_t amdgpu_sa_bo_gpu_addr(struct drm_suballoc *sa_bo)
325{
326 return to_amdgpu_sa_manager(sa_bo->manager)->gpu_addr +
327 drm_suballoc_soffset(sa_bo);
328}
329
330static inline void *amdgpu_sa_bo_cpu_addr(struct drm_suballoc *sa_bo)
331{
332 return to_amdgpu_sa_manager(sa_bo->manager)->cpu_ptr +
333 drm_suballoc_soffset(sa_bo);
334}
335
336int amdgpu_sa_bo_manager_init(struct amdgpu_device *adev,
337 struct amdgpu_sa_manager *sa_manager,
338 unsigned size, u32 align, u32 domain);
339void amdgpu_sa_bo_manager_fini(struct amdgpu_device *adev,
340 struct amdgpu_sa_manager *sa_manager);
341int amdgpu_sa_bo_manager_start(struct amdgpu_device *adev,
342 struct amdgpu_sa_manager *sa_manager);
343int amdgpu_sa_bo_new(struct amdgpu_sa_manager *sa_manager,
344 struct drm_suballoc **sa_bo,
345 unsigned int size);
346void amdgpu_sa_bo_free(struct drm_suballoc **sa_bo,
347 struct dma_fence *fence);
348#if defined(CONFIG_DEBUG_FS)
349void amdgpu_sa_bo_dump_debug_info(struct amdgpu_sa_manager *sa_manager,
350 struct seq_file *m);
351u64 amdgpu_bo_print_info(int id, struct amdgpu_bo *bo, struct seq_file *m);
352#endif
353void amdgpu_debugfs_sa_init(struct amdgpu_device *adev);
354
355bool amdgpu_bo_support_uswc(u64 bo_flags);315bool amdgpu_bo_support_uswc(u64 bo_flags);
356 316
357 317
drivers/gpu/drm/amd/amdgpu/amdgpu_preempt_mgr.c+11−3
@@ -46,6 +46,17 @@ static ssize_t mem_info_preempt_used_show(struct device *dev, · 45 unmodified lines
46 46
47static DEVICE_ATTR_RO(mem_info_preempt_used);47static DEVICE_ATTR_RO(mem_info_preempt_used);
48 48
49/**
50 * amdgpu_preempt_mgr_sysfs_fini - remove PREEMPT manager sysfs attributes
51 *
52 * @adev: amdgpu_device pointer
53 */
54void amdgpu_preempt_mgr_sysfs_fini(struct amdgpu_device *adev)
55{
56 if (adev->dev->kobj.sd)
57 device_remove_file(adev->dev, &dev_attr_mem_info_preempt_used);
58}
59
49/**60/**
50 * amdgpu_preempt_mgr_new - allocate a new node61 * amdgpu_preempt_mgr_new - allocate a new node
51 *62 *
@@ -137,9 +148,6 @@ void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev) · 85 unmodified lines
137 if (ret)148 if (ret)
138 return;149 return;
139 150
140 if (adev->dev->kobj.sd)
141 device_remove_file(adev->dev, &dev_attr_mem_info_preempt_used);
142
143 ttm_resource_manager_cleanup(man);151 ttm_resource_manager_cleanup(man);
144 ttm_set_driver_manager(&adev->mman.bdev, AMDGPU_PL_PREEMPT, NULL);152 ttm_set_driver_manager(&adev->mman.bdev, AMDGPU_PL_PREEMPT, NULL);
145}153}
drivers/gpu/drm/amd/amdgpu/amdgpu_psp_ta.c+4−1
@@ -166,7 +166,8 @@ static ssize_t ta_if_load_debugfs_write(struct file *fp, const char *buf, size_t · 165 unmodified lines
166 if (ret)166 if (ret)
167 return -EFAULT;167 return -EFAULT;
168 168
169 if (ta_bin_len > PSP_1_MEG)169 if (ta_bin_len < sizeof(struct common_firmware_header) ||
170 ta_bin_len > PSP_1_MEG)
170 return -EINVAL;171 return -EINVAL;
171 172
172 copy_pos += sizeof(uint32_t);173 copy_pos += sizeof(uint32_t);
@@ -321,6 +322,8 @@ static ssize_t ta_if_invoke_debugfs_write(struct file *fp, const char *buf, size · 148 unmodified lines
321 ret = copy_from_user((void *)&shared_buf_len, &buf[copy_pos], sizeof(uint32_t));322 ret = copy_from_user((void *)&shared_buf_len, &buf[copy_pos], sizeof(uint32_t));
322 if (ret)323 if (ret)
323 return -EFAULT;324 return -EFAULT;
325 if (!shared_buf_len || shared_buf_len > PSP_1_MEG)
326 return -EINVAL;
324 copy_pos += sizeof(uint32_t);327 copy_pos += sizeof(uint32_t);
325 328
326 shared_buf = memdup_user(&buf[copy_pos], shared_buf_len);329 shared_buf = memdup_user(&buf[copy_pos], shared_buf_len);
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c+47−815
@@ -128,12 +128,6 @@ const char *get_ras_block_str(struct ras_common_if *ras_block) · 127 unmodified lines
128/* typical ECC bad page rate is 1 bad page per 100MB VRAM */128/* typical ECC bad page rate is 1 bad page per 100MB VRAM */
129#define RAS_BAD_PAGE_COVER (100 * 1024 * 1024ULL)129#define RAS_BAD_PAGE_COVER (100 * 1024 * 1024ULL)
130 130
131#define MAX_UMC_POISON_POLLING_TIME_ASYNC 10
132
133#define AMDGPU_RAS_RETIRE_PAGE_INTERVAL 100 //ms
134
135#define MAX_FLUSH_RETIRE_DWORK_TIMES 100
136
137#define BYPASS_ALLOCATED_ADDRESS 0x0131#define BYPASS_ALLOCATED_ADDRESS 0x0
138#define BYPASS_INITIALIZATION_ADDRESS 0x1132#define BYPASS_INITIALIZATION_ADDRESS 0x1
139 133
@@ -249,16 +243,12 @@ static int amdgpu_check_address_validity(struct amdgpu_device *adev, · 109 unmodified lines
249 (address >= RAS_UMC_INJECT_ADDR_LIMIT))243 (address >= RAS_UMC_INJECT_ADDR_LIMIT))
250 return -EFAULT;244 return -EFAULT;
251 245
252 if (amdgpu_uniras_enabled(adev)) {246 if (amdgpu_sriov_vf(adev))
253 if (amdgpu_sriov_vf(adev))247 count = amdgpu_virt_ras_convert_retired_address(adev, address,
254 count = amdgpu_virt_ras_convert_retired_address(adev, address,248 page_pfns, ARRAY_SIZE(page_pfns));
255 page_pfns, ARRAY_SIZE(page_pfns));249 else
256 else250 count = amdgpu_ras_mgr_lookup_bad_pages_in_a_row(adev, address,
257 count = amdgpu_ras_mgr_lookup_bad_pages_in_a_row(adev, address,251 page_pfns, ARRAY_SIZE(page_pfns));
258 page_pfns, ARRAY_SIZE(page_pfns));
259 } else
260 count = amdgpu_umc_lookup_bad_pages_in_a_row(adev,
261 address, page_pfns, ARRAY_SIZE(page_pfns));
262 252
263 if (count <= 0)253 if (count <= 0)
264 return -EPERM;254 return -EPERM;
@@ -1381,76 +1371,6 @@ static void amdgpu_ras_mgr_virt_error_data_statistics_update(struct ras_manager · 1116 unmodified lines
1381 obj->err_data.de_count = err_data->de_count;1371 obj->err_data.de_count = err_data->de_count;
1382}1372}
1383 1373
1384static struct ras_manager *get_ras_manager(struct amdgpu_device *adev, enum amdgpu_ras_block blk)
1385{
1386 struct ras_common_if head;
1387
1388 memset(&head, 0, sizeof(head));
1389 head.block = blk;
1390
1391 return amdgpu_ras_find_obj(adev, &head);
1392}
1393
1394int amdgpu_ras_bind_aca(struct amdgpu_device *adev, enum amdgpu_ras_block blk,
1395 const struct aca_info *aca_info, void *data)
1396{
1397 struct ras_manager *obj;
1398
1399 /* in resume phase, no need to create aca fs node */
1400 if (adev->in_suspend || amdgpu_reset_in_recovery(adev))
1401 return 0;
1402
1403 obj = get_ras_manager(adev, blk);
1404 if (!obj)
1405 return -EINVAL;
1406
1407 return amdgpu_aca_add_handle(adev, &obj->aca_handle, ras_block_str(blk), aca_info, data);
1408}
1409
1410int amdgpu_ras_unbind_aca(struct amdgpu_device *adev, enum amdgpu_ras_block blk)
1411{
1412 struct ras_manager *obj;
1413
1414 obj = get_ras_manager(adev, blk);
1415 if (!obj)
1416 return -EINVAL;
1417
1418 amdgpu_aca_remove_handle(&obj->aca_handle);
1419
1420 return 0;
1421}
1422
1423static int amdgpu_aca_log_ras_error_data(struct amdgpu_device *adev, enum amdgpu_ras_block blk,
1424 enum aca_error_type type, struct ras_err_data *err_data,
1425 struct ras_query_context *qctx)
1426{
1427 struct ras_manager *obj;
1428
1429 obj = get_ras_manager(adev, blk);
1430 if (!obj)
1431 return -EINVAL;
1432
1433 return amdgpu_aca_get_error_data(adev, &obj->aca_handle, type, err_data, qctx);
1434}
1435
1436ssize_t amdgpu_ras_aca_sysfs_read(struct device *dev, struct device_attribute *attr,
1437 struct aca_handle *handle, char *buf, void *data)
1438{
1439 struct ras_manager *obj = container_of(handle, struct ras_manager, aca_handle);
1440 struct ras_query_if info = {
1441 .head = obj->head,
1442 };
1443
1444 if (!amdgpu_ras_get_error_query_ready(obj->adev))
1445 return sysfs_emit(buf, "Query currently inaccessible\n");
1446
1447 if (amdgpu_ras_query_error_status(obj->adev, &info))
1448 return -EINVAL;
1449
1450 return sysfs_emit(buf, "%s: %lu\n%s: %lu\n%s: %lu\n", "ue", info.ue_count,
1451 "ce", info.ce_count, "de", info.de_count);
1452}
1453
1454static int amdgpu_ras_query_error_status_helper(struct amdgpu_device *adev,1374static int amdgpu_ras_query_error_status_helper(struct amdgpu_device *adev,
1455 struct ras_query_if *info,1375 struct ras_query_if *info,
1456 struct ras_err_data *err_data,1376 struct ras_err_data *err_data,
@@ -1459,7 +1379,6 @@ static int amdgpu_ras_query_error_status_helper(struct amdgpu_device *adev, · 2 unmodified lines
1459{1379{
1460 enum amdgpu_ras_block blk = info ? info->head.block : AMDGPU_RAS_BLOCK_COUNT;1380 enum amdgpu_ras_block blk = info ? info->head.block : AMDGPU_RAS_BLOCK_COUNT;
1461 struct amdgpu_ras_block_object *block_obj = NULL;1381 struct amdgpu_ras_block_object *block_obj = NULL;
1462 int ret;
1463 1382
1464 if (blk == AMDGPU_RAS_BLOCK_COUNT)1383 if (blk == AMDGPU_RAS_BLOCK_COUNT)
1465 return -EINVAL;1384 return -EINVAL;
@@ -1469,7 +1388,7 @@ static int amdgpu_ras_query_error_status_helper(struct amdgpu_device *adev, · 3 unmodified lines
1469 1388
1470 if (error_query_mode == AMDGPU_RAS_VIRT_ERROR_COUNT_QUERY) {1389 if (error_query_mode == AMDGPU_RAS_VIRT_ERROR_COUNT_QUERY) {
1471 return amdgpu_virt_req_ras_err_count(adev, blk, err_data);1390 return amdgpu_virt_req_ras_err_count(adev, blk, err_data);
1472 } else if (error_query_mode == AMDGPU_RAS_DIRECT_ERROR_QUERY) {1391 } else {
1473 if (info->head.block == AMDGPU_RAS_BLOCK__UMC) {1392 if (info->head.block == AMDGPU_RAS_BLOCK__UMC) {
1474 amdgpu_ras_get_ecc_info(adev, err_data);1393 amdgpu_ras_get_ecc_info(adev, err_data);
1475 } else {1394 } else {
@@ -1490,24 +1409,6 @@ static int amdgpu_ras_query_error_status_helper(struct amdgpu_device *adev, · 14 unmodified lines
1490 block_obj->hw_ops->query_ras_error_status(adev);1409 block_obj->hw_ops->query_ras_error_status(adev);
1491 }1410 }
1492 }1411 }
1493 } else {
1494 if (amdgpu_aca_is_enabled(adev)) {
1495 ret = amdgpu_aca_log_ras_error_data(adev, blk, ACA_ERROR_TYPE_UE, err_data, qctx);
1496 if (ret)
1497 return ret;
1498
1499 ret = amdgpu_aca_log_ras_error_data(adev, blk, ACA_ERROR_TYPE_CE, err_data, qctx);
1500 if (ret)
1501 return ret;
1502
1503 ret = amdgpu_aca_log_ras_error_data(adev, blk, ACA_ERROR_TYPE_DEFERRED, err_data, qctx);
1504 if (ret)
1505 return ret;
1506 } else {
1507 /* FIXME: add code to check return value later */
1508 amdgpu_mca_smu_log_ras_error(adev, blk, AMDGPU_MCA_ERROR_TYPE_UE, err_data, qctx);
1509 amdgpu_mca_smu_log_ras_error(adev, blk, AMDGPU_MCA_ERROR_TYPE_CE, err_data, qctx);
1510 }
1511 }1412 }
1512 1413
1513 return 0;1414 return 0;
@@ -1624,8 +1525,6 @@ int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, · 110 unmodified lines
1624 enum amdgpu_ras_block block)1525 enum amdgpu_ras_block block)
1625{1526{
1626 struct amdgpu_ras_block_object *block_obj = amdgpu_ras_get_ras_block(adev, block, 0);1527 struct amdgpu_ras_block_object *block_obj = amdgpu_ras_get_ras_block(adev, block, 0);
1627 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
1628 const struct aca_smu_funcs *smu_funcs = adev->aca.smu_funcs;
1629 1528
1630 if (!block_obj || !block_obj->hw_ops) {1529 if (!block_obj || !block_obj->hw_ops) {
1631 dev_dbg_once(adev->dev, "%s doesn't config RAS function\n",1530 dev_dbg_once(adev->dev, "%s doesn't config RAS function\n",
@@ -1633,17 +1532,14 @@ int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, · 1 unmodified line
1633 return -EOPNOTSUPP;1532 return -EOPNOTSUPP;
1634 }1533 }
1635 1534
1636 if (!amdgpu_ras_is_supported(adev, block) ||1535 if (!amdgpu_ras_is_supported(adev, block))
1637 !amdgpu_ras_get_aca_debug_mode(adev))
1638 return -EOPNOTSUPP;1536 return -EOPNOTSUPP;
1639 1537
1640 if (amdgpu_sriov_vf(adev))1538 if (amdgpu_sriov_vf(adev))
1641 return -EOPNOTSUPP;1539 return -EOPNOTSUPP;
1642 1540
1643 /* skip ras error reset in gpu reset */1541 /* skip ras error reset in gpu reset */
1644 if ((amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) &&1542 if (amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev))
1645 ((smu_funcs && smu_funcs->set_debug_mode) ||
1646 (mca_funcs && mca_funcs->mca_set_debug_mode)))
1647 return -EOPNOTSUPP;1543 return -EOPNOTSUPP;
1648 1544
1649 if (block_obj->hw_ops->reset_ras_error_count)1545 if (block_obj->hw_ops->reset_ras_error_count)
@@ -2090,9 +1986,6 @@ int amdgpu_ras_sysfs_create(struct amdgpu_device *adev, · 440 unmodified lines
2090{1986{
2091 struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);1987 struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);
2092 1988
2093 if (amdgpu_aca_is_enabled(adev))
2094 return 0;
2095
2096 if (!obj || obj->attr_inuse)1989 if (!obj || obj->attr_inuse)
2097 return -EINVAL;1990 return -EINVAL;
2098 1991
@@ -2130,9 +2023,6 @@ int amdgpu_ras_sysfs_remove(struct amdgpu_device *adev, · 31 unmodified lines
2130{2023{
2131 struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);2024 struct ras_manager *obj = amdgpu_ras_find_obj(adev, head);
2132 2025
2133 if (amdgpu_aca_is_enabled(adev))
2134 return 0;
2135
2136 if (!obj || !obj->attr_inuse)2026 if (!obj || !obj->attr_inuse)
2137 return -EINVAL;2027 return -EINVAL;
2138 2028
@@ -2245,25 +2135,6 @@ static void amdgpu_ras_debugfs_create(struct amdgpu_device *adev, · 106 unmodified lines
2245 obj, &amdgpu_ras_debugfs_ops);2135 obj, &amdgpu_ras_debugfs_ops);
2246}2136}
2247 2137
2248static bool amdgpu_ras_aca_is_supported(struct amdgpu_device *adev)
2249{
2250 bool ret;
2251
2252 switch (amdgpu_ip_version(adev, MP0_HWIP, 0)) {
2253 case IP_VERSION(13, 0, 6):
2254 case IP_VERSION(13, 0, 12):
2255 case IP_VERSION(13, 0, 14):
2256 case IP_VERSION(13, 0, 15):
2257 ret = true;
2258 break;
2259 default:
2260 ret = false;
2261 break;
2262 }
2263
2264 return ret;
2265}
2266
2267void amdgpu_ras_debugfs_create_all(struct amdgpu_device *adev)2138void amdgpu_ras_debugfs_create_all(struct amdgpu_device *adev)
2268{2139{
2269 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);2140 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
@@ -2290,13 +2161,6 @@ void amdgpu_ras_debugfs_create_all(struct amdgpu_device *adev) · 20 unmodified lines
2290 amdgpu_ras_debugfs_create(adev, &fs_info, dir);2161 amdgpu_ras_debugfs_create(adev, &fs_info, dir);
2291 }2162 }
2292 }2163 }
2293
2294 if (amdgpu_ras_aca_is_supported(adev)) {
2295 if (amdgpu_aca_is_enabled(adev))
2296 amdgpu_aca_smu_debugfs_init(adev, dir);
2297 else
2298 amdgpu_mca_smu_debugfs_init(adev, dir);
2299 }
2300}2164}
2301 2165
2302/* debugfs end */2166/* debugfs end */
@@ -2489,14 +2353,6 @@ static void amdgpu_ras_interrupt_poison_creation_handler(struct ras_manager *obj · 186 unmodified lines
2489 event_id = amdgpu_ras_acquire_event_id(adev, type);2353 event_id = amdgpu_ras_acquire_event_id(adev, type);
2490 RAS_EVENT_LOG(adev, event_id, "Poison is created\n");2354 RAS_EVENT_LOG(adev, event_id, "Poison is created\n");
2491 2355
2492 if (amdgpu_ip_version(obj->adev, UMC_HWIP, 0) >= IP_VERSION(12, 0, 0)) {
2493 struct amdgpu_ras *con = amdgpu_ras_get_context(obj->adev);
2494
2495 atomic_inc(&con->page_retirement_req_cnt);
2496 atomic_inc(&con->poison_creation_count);
2497
2498 wake_up(&con->page_retirement_wq);
2499 }
2500}2356}
2501 2357
2502static void amdgpu_ras_interrupt_umc_handler(struct ras_manager *obj,2358static void amdgpu_ras_interrupt_umc_handler(struct ras_manager *obj,
@@ -3026,77 +2882,6 @@ static int amdgpu_ras_realloc_eh_data_space(struct amdgpu_device *adev, · 523 unmodified lines
3026 return 0;2882 return 0;
3027}2883}
3028 2884
3029static int amdgpu_ras_mca2pa_by_idx(struct amdgpu_device *adev,
3030 struct eeprom_table_record *bps,
3031 struct ras_err_data *err_data)
3032{
3033 struct ta_ras_query_address_input addr_in;
3034 uint32_t socket = 0;
3035 int ret = 0;
3036
3037 if (adev->smuio.funcs && adev->smuio.funcs->get_socket_id)
3038 socket = adev->smuio.funcs->get_socket_id(adev);
3039
3040 /* reinit err_data */
3041 err_data->err_addr_cnt = 0;
3042 err_data->err_addr_len = adev->umc.retire_unit;
3043
3044 memset(&addr_in, 0, sizeof(addr_in));
3045 addr_in.ma.err_addr = bps->address;
3046 addr_in.ma.socket_id = socket;
3047 addr_in.ma.ch_inst = bps->mem_channel;
3048 if (!amdgpu_ras_smu_eeprom_supported(adev)) {
3049 /* tell RAS TA the node instance is not used */
3050 addr_in.ma.node_inst = TA_RAS_INV_NODE;
3051 } else {
3052 addr_in.ma.umc_inst = bps->mcumc_id;
3053 addr_in.ma.node_inst = bps->cu;
3054 }
3055
3056 if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr)
3057 ret = adev->umc.ras->convert_ras_err_addr(adev, err_data,
3058 &addr_in, NULL, false);
3059
3060 return ret;
3061}
3062
3063static int amdgpu_ras_mca2pa(struct amdgpu_device *adev,
3064 struct eeprom_table_record *bps,
3065 struct ras_err_data *err_data)
3066{
3067 struct ta_ras_query_address_input addr_in;
3068 uint32_t die_id, socket = 0;
3069
3070 if (adev->smuio.funcs && adev->smuio.funcs->get_socket_id)
3071 socket = adev->smuio.funcs->get_socket_id(adev);
3072
3073 /* although die id is gotten from PA in nps1 mode, the id is
3074 * fitable for any nps mode
3075 */
3076 if (adev->umc.ras && adev->umc.ras->get_die_id_from_pa)
3077 die_id = adev->umc.ras->get_die_id_from_pa(adev, bps->address,
3078 bps->retired_page << AMDGPU_GPU_PAGE_SHIFT);
3079 else
3080 return -EINVAL;
3081
3082 /* reinit err_data */
3083 err_data->err_addr_cnt = 0;
3084 err_data->err_addr_len = adev->umc.retire_unit;
3085
3086 memset(&addr_in, 0, sizeof(addr_in));
3087 addr_in.ma.err_addr = bps->address;
3088 addr_in.ma.ch_inst = bps->mem_channel;
3089 addr_in.ma.umc_inst = bps->mcumc_id;
3090 addr_in.ma.node_inst = die_id;
3091 addr_in.ma.socket_id = socket;
3092
3093 if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr)
3094 return adev->umc.ras->convert_ras_err_addr(adev, err_data,
3095 &addr_in, NULL, false);
3096 else
3097 return -EINVAL;
3098}
3099
3100static bool __check_record_in_range(struct amdgpu_device *adev,2885static bool __check_record_in_range(struct amdgpu_device *adev,
3101 struct eeprom_table_record *bps, int count)2886 struct eeprom_table_record *bps, int count)
3102{2887{
@@ -3157,117 +2942,13 @@ static int __amdgpu_ras_convert_rec_array_from_rom(struct amdgpu_device *adev, · 54 unmodified lines
3157 struct eeprom_table_record *bps, struct ras_err_data *err_data,2942 struct eeprom_table_record *bps, struct ras_err_data *err_data,
3158 enum amdgpu_memory_partition nps)2943 enum amdgpu_memory_partition nps)
3159{2944{
3160 int i = 0;
3161 uint64_t chan_idx_v2;
3162 enum amdgpu_memory_partition save_nps;
3163
3164 save_nps = (bps[0].retired_page >> UMC_NPS_SHIFT) & UMC_NPS_MASK;
3165 chan_idx_v2 = bps[0].retired_page & UMC_CHANNEL_IDX_V2;
3166
3167 /*old asics just have pa in eeprom*/2945 /*old asics just have pa in eeprom*/
3168 if (IP_VERSION_MAJ(amdgpu_ip_version(adev, UMC_HWIP, 0)) < 12) {2946 memcpy(err_data->err_addr, bps,
3169 memcpy(err_data->err_addr, bps,2947 sizeof(struct eeprom_table_record) * adev->umc.retire_unit);
3170 sizeof(struct eeprom_table_record) * adev->umc.retire_unit);
3171 goto out;
3172 }
3173
3174 for (i = 0; i < adev->umc.retire_unit; i++)
3175 bps[i].retired_page &= ~(UMC_NPS_MASK << UMC_NPS_SHIFT);
3176 2948
3177 if (save_nps || chan_idx_v2) {
3178 if (save_nps == nps) {
3179 if (amdgpu_umc_pages_in_a_row(adev, err_data,
3180 bps[0].retired_page << AMDGPU_GPU_PAGE_SHIFT))
3181 return -EINVAL;
3182 for (i = 0; i < adev->umc.retire_unit; i++) {
3183 err_data->err_addr[i].address = bps[0].address;
3184 err_data->err_addr[i].mem_channel = bps[0].mem_channel;
3185 err_data->err_addr[i].bank = bps[0].bank;
3186 err_data->err_addr[i].err_type = bps[0].err_type;
3187 err_data->err_addr[i].mcumc_id = bps[0].mcumc_id;
3188 }
3189 } else {
3190 if (amdgpu_ras_mca2pa_by_idx(adev, &bps[0], err_data))
3191 return -EINVAL;
3192 }
3193 } else {
3194 if (bps[0].address == 0) {
3195 /* for specific old eeprom data, mca address is not stored,
3196 * calc it from pa
3197 */
3198 if (amdgpu_umc_pa2mca(adev, bps[0].retired_page << AMDGPU_GPU_PAGE_SHIFT,
3199 &(bps[0].address), AMDGPU_NPS1_PARTITION_MODE))
3200 return -EINVAL;
3201 }
3202
3203 if (amdgpu_ras_mca2pa(adev, &bps[0], err_data)) {
3204 if (nps == AMDGPU_NPS1_PARTITION_MODE)
3205 memcpy(err_data->err_addr, bps,
3206 sizeof(struct eeprom_table_record) * adev->umc.retire_unit);
3207 else
3208 return -EOPNOTSUPP;
3209 }
3210 }
3211
3212out:
3213 return __amdgpu_ras_restore_bad_pages(adev, err_data->err_addr, adev->umc.retire_unit);2949 return __amdgpu_ras_restore_bad_pages(adev, err_data->err_addr, adev->umc.retire_unit);
3214}2950}
3215 2951
3216static int __amdgpu_ras_convert_rec_from_rom(struct amdgpu_device *adev,
3217 struct eeprom_table_record *bps, struct ras_err_data *err_data,
3218 enum amdgpu_memory_partition nps)
3219{
3220 int i = 0;
3221 uint64_t chan_idx_v2;
3222 enum amdgpu_memory_partition save_nps;
3223
3224 if (!amdgpu_ras_smu_eeprom_supported(adev)) {
3225 save_nps = (bps->retired_page >> UMC_NPS_SHIFT) & UMC_NPS_MASK;
3226 chan_idx_v2 = bps->retired_page & UMC_CHANNEL_IDX_V2;
3227 bps->retired_page &= ~(UMC_NPS_MASK << UMC_NPS_SHIFT);
3228 } else {
3229 /* if pmfw manages eeprom, save_nps is not stored on eeprom,
3230 * we should always convert mca address into physical address,
3231 * make save_nps different from nps
3232 */
3233 save_nps = nps + 1;
3234 }
3235
3236 if (save_nps == nps) {
3237 if (amdgpu_umc_pages_in_a_row(adev, err_data,
3238 bps->retired_page << AMDGPU_GPU_PAGE_SHIFT))
3239 return -EINVAL;
3240 for (i = 0; i < adev->umc.retire_unit; i++) {
3241 err_data->err_addr[i].address = bps->address;
3242 err_data->err_addr[i].mem_channel = bps->mem_channel;
3243 err_data->err_addr[i].bank = bps->bank;
3244 err_data->err_addr[i].err_type = bps->err_type;
3245 err_data->err_addr[i].mcumc_id = bps->mcumc_id;
3246 }
3247 } else {
3248 if (save_nps || chan_idx_v2) {
3249 if (amdgpu_ras_mca2pa_by_idx(adev, bps, err_data))
3250 return -EINVAL;
3251 } else {
3252 /* for specific old eeprom data, mca address is not stored,
3253 * calc it from pa
3254 */
3255 if (bps->address == 0)
3256 if (amdgpu_umc_pa2mca(adev,
3257 bps->retired_page << AMDGPU_GPU_PAGE_SHIFT,
3258 &(bps->address),
3259 AMDGPU_NPS1_PARTITION_MODE))
3260 return -EINVAL;
3261
3262 if (amdgpu_ras_mca2pa(adev, bps, err_data))
3263 return -EOPNOTSUPP;
3264 }
3265 }
3266
3267 return __amdgpu_ras_restore_bad_pages(adev, err_data->err_addr,
3268 adev->umc.retire_unit);
3269}
3270
3271/* it deal with vram only. */2952/* it deal with vram only. */
3272int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev,2953int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev,
3273 struct eeprom_table_record *bps, int pages, bool from_rom)2954 struct eeprom_table_record *bps, int pages, bool from_rom)
@@ -3300,8 +2981,7 @@ int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev, · 26 unmodified lines
3300 2981
3301 if (from_rom) {2982 if (from_rom) {
3302 /* there is no pa recs in V3, so skip pa recs processing */2983 /* there is no pa recs in V3, so skip pa recs processing */
3303 if ((control->tbl_hdr.version < RAS_TABLE_VER_V3) &&2984 if (control->tbl_hdr.version < RAS_TABLE_VER_V3) {
3304 !amdgpu_ras_smu_eeprom_supported(adev)) {
3305 for (i = 0; i < pages; i++) {2985 for (i = 0; i < pages; i++) {
3306 if (control->ras_num_recs - i >= adev->umc.retire_unit) {2986 if (control->ras_num_recs - i >= adev->umc.retire_unit) {
3307 if ((bps[i].address == bps[i + 1].address) &&2987 if ((bps[i].address == bps[i + 1].address) &&
@@ -3318,10 +2998,8 @@ int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev, · 10 unmodified lines
3318 }2998 }
3319 }2999 }
3320 }3000 }
3321 for (; i < pages; i++) {3001 for (; i < pages; i++)
3322 ret = __amdgpu_ras_convert_rec_from_rom(adev,3002 bps[i].retired_page &= ~(UMC_NPS_MASK << UMC_NPS_SHIFT);
3323 &bps[i], &err_data, nps);
3324 }
3325 3003
3326 con->eh_data->count_saved = con->eh_data->count;3004 con->eh_data->count_saved = con->eh_data->count;
3327 } else {3005 } else {
@@ -3346,7 +3024,7 @@ int amdgpu_ras_save_bad_pages(struct amdgpu_device *adev, · 18 unmodified lines
3346 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);3024 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3347 struct ras_err_handler_data *data;3025 struct ras_err_handler_data *data;
3348 struct amdgpu_ras_eeprom_control *control;3026 struct amdgpu_ras_eeprom_control *control;
3349 int save_count, unit_num, i;3027 int save_count, unit_num;
3350 3028
3351 if (!con || !con->eh_data) {3029 if (!con || !con->eh_data) {
3352 if (new_cnt)3030 if (new_cnt)
@@ -3367,12 +3045,7 @@ int amdgpu_ras_save_bad_pages(struct amdgpu_device *adev, · 14 unmodified lines
3367 mutex_lock(&con->recovery_lock);3045 mutex_lock(&con->recovery_lock);
3368 control = &con->eeprom_control;3046 control = &con->eeprom_control;
3369 data = con->eh_data;3047 data = con->eh_data;
3370 if (amdgpu_ras_smu_eeprom_supported(adev))3048 unit_num = data->count / adev->umc.retire_unit - control->ras_num_recs;
3371 unit_num = control->ras_num_recs -
3372 control->ras_num_recs_old;
3373 else
3374 unit_num = data->count / adev->umc.retire_unit -
3375 control->ras_num_recs;
3376 3049
3377 save_count = con->bad_page_num - control->ras_num_bad_pages;3050 save_count = con->bad_page_num - control->ras_num_bad_pages;
3378 mutex_unlock(&con->recovery_lock);3051 mutex_unlock(&con->recovery_lock);
@@ -3383,21 +3056,10 @@ int amdgpu_ras_save_bad_pages(struct amdgpu_device *adev, · 4 unmodified lines
3383 /* only new entries are saved */3056 /* only new entries are saved */
3384 if (unit_num && save_count) {3057 if (unit_num && save_count) {
3385 /*old asics only save pa to eeprom like before*/3058 /*old asics only save pa to eeprom like before*/
3386 if (IP_VERSION_MAJ(amdgpu_ip_version(adev, UMC_HWIP, 0)) < 12) {3059 if (amdgpu_ras_eeprom_append(control,
3387 if (amdgpu_ras_eeprom_append(control,3060 &data->bps[data->count_saved], unit_num)) {
3388 &data->bps[data->count_saved], unit_num)) {3061 dev_err(adev->dev, "Failed to save EEPROM table data!");
3389 dev_err(adev->dev, "Failed to save EEPROM table data!");3062 return -EIO;
3390 return -EIO;
3391 }
3392 } else {
3393 for (i = 0; i < unit_num; i++) {
3394 if (amdgpu_ras_eeprom_append(control,
3395 &data->bps[data->count_saved +
3396 i * adev->umc.retire_unit], 1)) {
3397 dev_err(adev->dev, "Failed to save EEPROM table data!");
3398 return -EIO;
3399 }
3400 }
3401 }3063 }
3402 3064
3403 dev_info(adev->dev, "Saved %d pages to EEPROM table.\n", save_count);3065 dev_info(adev->dev, "Saved %d pages to EEPROM table.\n", save_count);
@@ -3416,7 +3078,7 @@ static int amdgpu_ras_load_bad_pages(struct amdgpu_device *adev) · 12 unmodified lines
3416 struct amdgpu_ras_eeprom_control *control =3078 struct amdgpu_ras_eeprom_control *control =
3417 &adev->psp.ras_context.ras->eeprom_control;3079 &adev->psp.ras_context.ras->eeprom_control;
3418 struct eeprom_table_record *bps;3080 struct eeprom_table_record *bps;
3419 int ret, i = 0;3081 int ret;
3420 3082
3421 /* no bad page record, skip eeprom access */3083 /* no bad page record, skip eeprom access */
3422 if (control->ras_num_recs == 0 || amdgpu_bad_page_threshold == 0)3084 if (control->ras_num_recs == 0 || amdgpu_bad_page_threshold == 0)
@@ -3430,33 +3092,6 @@ static int amdgpu_ras_load_bad_pages(struct amdgpu_device *adev) · 7 unmodified lines
3430 if (ret) {3092 if (ret) {
3431 dev_err(adev->dev, "Failed to load EEPROM table records!");3093 dev_err(adev->dev, "Failed to load EEPROM table records!");
3432 } else {3094 } else {
3433 if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr) {
3434 /*In V3, there is no pa recs, and some cases(when address==0) may be parsed
3435 as pa recs, so add verion check to avoid it.
3436 */
3437 if ((control->tbl_hdr.version < RAS_TABLE_VER_V3) &&
3438 !amdgpu_ras_smu_eeprom_supported(adev)) {
3439 for (i = 0; i < control->ras_num_recs; i++) {
3440 if ((control->ras_num_recs - i) >= adev->umc.retire_unit) {
3441 if ((bps[i].address == bps[i + 1].address) &&
3442 (bps[i].mem_channel == bps[i + 1].mem_channel)) {
3443 control->ras_num_pa_recs += adev->umc.retire_unit;
3444 i += (adev->umc.retire_unit - 1);
3445 } else {
3446 control->ras_num_mca_recs +=
3447 (control->ras_num_recs - i);
3448 break;
3449 }
3450 } else {
3451 control->ras_num_mca_recs += (control->ras_num_recs - i);
3452 break;
3453 }
3454 }
3455 } else {
3456 control->ras_num_mca_recs = control->ras_num_recs;
3457 }
3458 }
3459
3460 ret = amdgpu_ras_add_bad_pages(adev, bps, control->ras_num_recs, true);3095 ret = amdgpu_ras_add_bad_pages(adev, bps, control->ras_num_recs, true);
3461 if (ret)3096 if (ret)
3462 goto out;3097 goto out;
@@ -3550,293 +3185,6 @@ static void amdgpu_ras_validate_threshold(struct amdgpu_device *adev, · 87 unmodified lines
3550 }3185 }
3551}3186}
3552 3187
3553int amdgpu_ras_put_poison_req(struct amdgpu_device *adev,
3554 enum amdgpu_ras_block block, uint16_t pasid,
3555 pasid_notify pasid_fn, void *data, uint32_t reset)
3556{
3557 int ret = 0;
3558 struct ras_poison_msg poison_msg;
3559 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3560
3561 memset(&poison_msg, 0, sizeof(poison_msg));
3562 poison_msg.block = block;
3563 poison_msg.pasid = pasid;
3564 poison_msg.reset = reset;
3565 poison_msg.pasid_fn = pasid_fn;
3566 poison_msg.data = data;
3567
3568 ret = kfifo_put(&con->poison_fifo, poison_msg);
3569 if (!ret) {
3570 dev_err(adev->dev, "Poison message fifo is full!\n");
3571 return -ENOSPC;
3572 }
3573
3574 return 0;
3575}
3576
3577static int amdgpu_ras_get_poison_req(struct amdgpu_device *adev,
3578 struct ras_poison_msg *poison_msg)
3579{
3580 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3581
3582 return kfifo_get(&con->poison_fifo, poison_msg);
3583}
3584
3585static void amdgpu_ras_ecc_log_init(struct ras_ecc_log_info *ecc_log)
3586{
3587 mutex_init(&ecc_log->lock);
3588
3589 INIT_RADIX_TREE(&ecc_log->de_page_tree, GFP_KERNEL);
3590 ecc_log->de_queried_count = 0;
3591 ecc_log->consumption_q_count = 0;
3592}
3593
3594static void amdgpu_ras_ecc_log_fini(struct ras_ecc_log_info *ecc_log)
3595{
3596 struct radix_tree_iter iter;
3597 void __rcu **slot;
3598 struct ras_ecc_err *ecc_err;
3599
3600 mutex_lock(&ecc_log->lock);
3601 radix_tree_for_each_slot(slot, &ecc_log->de_page_tree, &iter, 0) {
3602 ecc_err = radix_tree_deref_slot(slot);
3603 kfree(ecc_err->err_pages.pfn);
3604 kfree(ecc_err);
3605 radix_tree_iter_delete(&ecc_log->de_page_tree, &iter, slot);
3606 }
3607 mutex_unlock(&ecc_log->lock);
3608
3609 mutex_destroy(&ecc_log->lock);
3610 ecc_log->de_queried_count = 0;
3611 ecc_log->consumption_q_count = 0;
3612}
3613
3614static bool amdgpu_ras_schedule_retirement_dwork(struct amdgpu_ras *con,
3615 uint32_t delayed_ms)
3616{
3617 int ret;
3618
3619 mutex_lock(&con->umc_ecc_log.lock);
3620 ret = radix_tree_tagged(&con->umc_ecc_log.de_page_tree,
3621 UMC_ECC_NEW_DETECTED_TAG);
3622 mutex_unlock(&con->umc_ecc_log.lock);
3623
3624 if (ret)
3625 schedule_delayed_work(&con->page_retirement_dwork,
3626 msecs_to_jiffies(delayed_ms));
3627
3628 return ret ? true : false;
3629}
3630
3631static void amdgpu_ras_do_page_retirement(struct work_struct *work)
3632{
3633 struct amdgpu_ras *con = container_of(work, struct amdgpu_ras,
3634 page_retirement_dwork.work);
3635 struct amdgpu_device *adev = con->adev;
3636 struct ras_err_data err_data;
3637
3638 /* If gpu reset is ongoing, delay retiring the bad pages */
3639 if (amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) {
3640 amdgpu_ras_schedule_retirement_dwork(con,
3641 AMDGPU_RAS_RETIRE_PAGE_INTERVAL * 3);
3642 return;
3643 }
3644
3645 amdgpu_ras_error_data_init(&err_data);
3646
3647 amdgpu_umc_handle_bad_pages(adev, &err_data);
3648
3649 amdgpu_ras_error_data_fini(&err_data);
3650
3651 amdgpu_ras_schedule_retirement_dwork(con,
3652 AMDGPU_RAS_RETIRE_PAGE_INTERVAL);
3653}
3654
3655static int amdgpu_ras_poison_creation_handler(struct amdgpu_device *adev,
3656 uint32_t poison_creation_count)
3657{
3658 int ret = 0;
3659 struct ras_ecc_log_info *ecc_log;
3660 struct ras_query_if info;
3661 u32 timeout = MAX_UMC_POISON_POLLING_TIME_ASYNC;
3662 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
3663 u64 de_queried_count;
3664 u64 consumption_q_count;
3665 enum ras_event_type type = RAS_EVENT_TYPE_POISON_CREATION;
3666
3667 memset(&info, 0, sizeof(info));
3668 info.head.block = AMDGPU_RAS_BLOCK__UMC;
3669
3670 ecc_log = &ras->umc_ecc_log;
3671 ecc_log->de_queried_count = 0;
3672 ecc_log->consumption_q_count = 0;
3673
3674 do {
3675 ret = amdgpu_ras_query_error_status_with_event(adev, &info, type);
3676 if (ret)
3677 return ret;
3678
3679 de_queried_count = ecc_log->de_queried_count;
3680 consumption_q_count = ecc_log->consumption_q_count;
3681
3682 if (de_queried_count && consumption_q_count)
3683 break;
3684
3685 msleep(100);
3686 } while (--timeout);
3687
3688 if (de_queried_count)
3689 schedule_delayed_work(&ras->page_retirement_dwork, 0);
3690
3691 if (amdgpu_ras_is_rma(adev) && atomic_cmpxchg(&ras->rma_in_recovery, 0, 1) == 0)
3692 amdgpu_ras_reset_gpu(adev);
3693
3694 return 0;
3695}
3696
3697static void amdgpu_ras_clear_poison_fifo(struct amdgpu_device *adev)
3698{
3699 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3700 struct ras_poison_msg msg;
3701 int ret;
3702
3703 do {
3704 ret = kfifo_get(&con->poison_fifo, &msg);
3705 } while (ret);
3706}
3707
3708static int amdgpu_ras_poison_consumption_handler(struct amdgpu_device *adev,
3709 uint32_t msg_count, uint32_t *gpu_reset)
3710{
3711 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3712 uint32_t reset_flags = 0, reset = 0;
3713 struct ras_poison_msg msg;
3714 int ret, i;
3715
3716 kgd2kfd_set_sram_ecc_flag(adev->kfd.dev);
3717
3718 for (i = 0; i < msg_count; i++) {
3719 ret = amdgpu_ras_get_poison_req(adev, &msg);
3720 if (!ret)
3721 continue;
3722
3723 if (msg.pasid_fn)
3724 msg.pasid_fn(adev, msg.pasid, msg.data);
3725
3726 reset_flags |= msg.reset;
3727 }
3728
3729 /*
3730 * Try to ensure poison creation handler is completed first
3731 * to set rma if bad page exceed threshold.
3732 */
3733 flush_delayed_work(&con->page_retirement_dwork);
3734
3735 /* for RMA, amdgpu_ras_poison_creation_handler will trigger gpu reset */
3736 if (reset_flags && !amdgpu_ras_is_rma(adev)) {
3737 if (reset_flags & AMDGPU_RAS_GPU_RESET_MODE1_RESET)
3738 reset = AMDGPU_RAS_GPU_RESET_MODE1_RESET;
3739 else if (reset_flags & AMDGPU_RAS_GPU_RESET_MODE2_RESET)
3740 reset = AMDGPU_RAS_GPU_RESET_MODE2_RESET;
3741 else
3742 reset = reset_flags;
3743
3744 con->gpu_reset_flags |= reset;
3745 amdgpu_ras_reset_gpu(adev);
3746
3747 *gpu_reset = reset;
3748
3749 /* Wait for gpu recovery to complete */
3750 flush_work(&con->recovery_work);
3751 }
3752
3753 return 0;
3754}
3755
3756static int amdgpu_ras_page_retirement_thread(void *param)
3757{
3758 struct amdgpu_device *adev = (struct amdgpu_device *)param;
3759 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3760 uint32_t poison_creation_count, msg_count;
3761 uint32_t gpu_reset;
3762 int ret;
3763
3764 while (!kthread_should_stop()) {
3765
3766 wait_event_interruptible(con->page_retirement_wq,
3767 kthread_should_stop() ||
3768 atomic_read(&con->page_retirement_req_cnt));
3769
3770 if (kthread_should_stop())
3771 break;
3772
3773 mutex_lock(&con->poison_lock);
3774 gpu_reset = 0;
3775
3776 do {
3777 poison_creation_count = atomic_read(&con->poison_creation_count);
3778 ret = amdgpu_ras_poison_creation_handler(adev, poison_creation_count);
3779 if (ret == -EIO)
3780 break;
3781
3782 if (poison_creation_count) {
3783 atomic_sub(poison_creation_count, &con->poison_creation_count);
3784 atomic_sub(poison_creation_count, &con->page_retirement_req_cnt);
3785 }
3786 } while (atomic_read(&con->poison_creation_count) &&
3787 !atomic_read(&con->poison_consumption_count));
3788
3789 if (ret != -EIO) {
3790 msg_count = kfifo_len(&con->poison_fifo);
3791 if (msg_count) {
3792 ret = amdgpu_ras_poison_consumption_handler(adev,
3793 msg_count, &gpu_reset);
3794 if ((ret != -EIO) &&
3795 (gpu_reset != AMDGPU_RAS_GPU_RESET_MODE1_RESET))
3796 atomic_sub(msg_count, &con->page_retirement_req_cnt);
3797 }
3798 }
3799
3800 if ((ret == -EIO) || (gpu_reset == AMDGPU_RAS_GPU_RESET_MODE1_RESET)) {
3801 /* gpu mode-1 reset is ongoing or just completed ras mode-1 reset */
3802 /* Clear poison creation request */
3803 atomic_set(&con->poison_creation_count, 0);
3804 atomic_set(&con->poison_consumption_count, 0);
3805
3806 /* Clear poison fifo */
3807 amdgpu_ras_clear_poison_fifo(adev);
3808
3809 /* Clear all poison requests */
3810 atomic_set(&con->page_retirement_req_cnt, 0);
3811
3812 if (ret == -EIO) {
3813 /* Wait for mode-1 reset to complete */
3814 down_read(&adev->reset_domain->sem);
3815 up_read(&adev->reset_domain->sem);
3816 }
3817
3818 /* Wake up work to save bad pages to eeprom */
3819 schedule_delayed_work(&con->page_retirement_dwork, 0);
3820 } else if (gpu_reset) {
3821 /* gpu just completed mode-2 reset or other reset */
3822 /* Clear poison consumption messages cached in fifo */
3823 msg_count = kfifo_len(&con->poison_fifo);
3824 if (msg_count) {
3825 amdgpu_ras_clear_poison_fifo(adev);
3826 atomic_sub(msg_count, &con->page_retirement_req_cnt);
3827 }
3828
3829 atomic_set(&con->poison_consumption_count, 0);
3830
3831 /* Wake up work to save bad pages to eeprom */
3832 schedule_delayed_work(&con->page_retirement_dwork, 0);
3833 }
3834 mutex_unlock(&con->poison_lock);
3835 }
3836
3837 return 0;
3838}
3839
3840int amdgpu_ras_init_badpage_info(struct amdgpu_device *adev)3188int amdgpu_ras_init_badpage_info(struct amdgpu_device *adev)
3841{3189{
3842 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);3190 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
@@ -3846,7 +3194,14 @@ int amdgpu_ras_init_badpage_info(struct amdgpu_device *adev) · 3 unmodified lines
3846 if (!con || amdgpu_sriov_vf(adev))3194 if (!con || amdgpu_sriov_vf(adev))
3847 return 0;3195 return 0;
3848 3196
3849 if (amdgpu_uniras_enabled(adev))3197 /*
3198 * For the reset-on-init path (e.g. an NPS memory partition,
3199 * switch) the RAS IP block hw_init has not been enabled and
3200 * the amdgpu_uniras_enabled return false, check amdgpu ras
3201 * context uniras_enabled flag, eeprom init will be called
3202 * during RAS IP block hw_init.
3203 */
3204 if (amdgpu_uniras_enabled(adev) || con->uniras_enabled)
3850 return 0;3205 return 0;
3851 3206
3852 control = &con->eeprom_control;3207 control = &con->eeprom_control;
@@ -3855,9 +3210,6 @@ int amdgpu_ras_init_badpage_info(struct amdgpu_device *adev) · 2 unmodified lines
3855 ret = amdgpu_ras_eeprom_init(control);3210 ret = amdgpu_ras_eeprom_init(control);
3856 control->is_eeprom_valid = !ret;3211 control->is_eeprom_valid = !ret;
3857 3212
3858 if (!adev->umc.ras || !adev->umc.ras->convert_ras_err_addr)
3859 control->ras_num_pa_recs = control->ras_num_recs;
3860
3861 if (adev->umc.ras &&3213 if (adev->umc.ras &&
3862 adev->umc.ras->get_retire_flip_bits)3214 adev->umc.ras->get_retire_flip_bits)
3863 adev->umc.ras->get_retire_flip_bits(adev);3215 adev->umc.ras->get_retire_flip_bits(adev);
@@ -3877,13 +3229,6 @@ int amdgpu_ras_init_badpage_info(struct amdgpu_device *adev) · 13 unmodified lines
3877 adev, control->bad_channel_bitmap);3229 adev, control->bad_channel_bitmap);
3878 con->update_channel_flag = false;3230 con->update_channel_flag = false;
3879 }3231 }
3880
3881 /* The format action is only applied to new ASICs */
3882 if (IP_VERSION_MAJ(amdgpu_ip_version(adev, UMC_HWIP, 0)) >= 12 &&
3883 control->tbl_hdr.version < RAS_TABLE_VER_V3)
3884 if (!amdgpu_ras_eeprom_reset_table(control))
3885 if (amdgpu_ras_save_bad_pages(adev, NULL))
3886 dev_warn(adev->dev, "Failed to format RAS EEPROM data in V3 version!\n");
3887 }3232 }
3888 3233
3889 return 0;3234 return 0;
@@ -3917,10 +3262,8 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev, bool init_bp_info) · 27 unmodified lines
3917 }3262 }
3918 3263
3919 mutex_init(&con->recovery_lock);3264 mutex_init(&con->recovery_lock);
3920 mutex_init(&con->poison_lock);
3921 INIT_WORK(&con->recovery_work, amdgpu_ras_do_recovery);3265 INIT_WORK(&con->recovery_work, amdgpu_ras_do_recovery);
3922 atomic_set(&con->in_recovery, 0);3266 atomic_set(&con->in_recovery, 0);
3923 atomic_set(&con->rma_in_recovery, 0);
3924 con->eeprom_control.bad_channel_bitmap = 0;3267 con->eeprom_control.bad_channel_bitmap = 0;
3925 3268
3926 max_eeprom_records_count = amdgpu_ras_eeprom_max_record_count(&con->eeprom_control);3269 max_eeprom_records_count = amdgpu_ras_eeprom_max_record_count(&con->eeprom_control);
@@ -3933,21 +3276,8 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev, bool init_bp_info) · 6 unmodified lines
3933 }3276 }
3934 3277
3935 mutex_init(&con->page_rsv_lock);3278 mutex_init(&con->page_rsv_lock);
3936 INIT_KFIFO(con->poison_fifo);
3937 mutex_init(&con->page_retirement_lock);3279 mutex_init(&con->page_retirement_lock);
3938 init_waitqueue_head(&con->page_retirement_wq);3280
3939 atomic_set(&con->page_retirement_req_cnt, 0);
3940 atomic_set(&con->poison_creation_count, 0);
3941 atomic_set(&con->poison_consumption_count, 0);
3942 con->page_retirement_thread =
3943 kthread_run(amdgpu_ras_page_retirement_thread, adev, "umc_page_retirement");
3944 if (IS_ERR(con->page_retirement_thread)) {
3945 con->page_retirement_thread = NULL;
3946 dev_warn(adev->dev, "Failed to create umc_page_retirement thread!!!\n");
3947 }
3948
3949 INIT_DELAYED_WORK(&con->page_retirement_dwork, amdgpu_ras_do_page_retirement);
3950 amdgpu_ras_ecc_log_init(&con->umc_ecc_log);
3951#ifdef CONFIG_X86_MCE_AMD3281#ifdef CONFIG_X86_MCE_AMD
3952 if ((adev->asic_type == CHIP_ALDEBARAN) &&3282 if ((adev->asic_type == CHIP_ALDEBARAN) &&
3953 (adev->gmc.xgmi.connected_to_cpu))3283 (adev->gmc.xgmi.connected_to_cpu))
@@ -3978,33 +3308,15 @@ static int amdgpu_ras_recovery_fini(struct amdgpu_device *adev) · 24 unmodified lines
3978{3308{
3979 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);3309 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
3980 struct ras_err_handler_data *data = con->eh_data;3310 struct ras_err_handler_data *data = con->eh_data;
3981 int max_flush_timeout = MAX_FLUSH_RETIRE_DWORK_TIMES;
3982 bool ret;
3983 3311
3984 /* recovery_init failed to init it, fini is useless */3312 /* recovery_init failed to init it, fini is useless */
3985 if (!data)3313 if (!data)
3986 return 0;3314 return 0;
3987 3315
3988 /* Save all cached bad pages to eeprom */
3989 do {
3990 flush_delayed_work(&con->page_retirement_dwork);
3991 ret = amdgpu_ras_schedule_retirement_dwork(con, 0);
3992 } while (ret && max_flush_timeout--);
3993
3994 if (con->page_retirement_thread)
3995 kthread_stop(con->page_retirement_thread);
3996
3997 atomic_set(&con->page_retirement_req_cnt, 0);
3998 atomic_set(&con->poison_creation_count, 0);
3999
4000 mutex_destroy(&con->page_rsv_lock);3316 mutex_destroy(&con->page_rsv_lock);
4001 3317
4002 cancel_work_sync(&con->recovery_work);3318 cancel_work_sync(&con->recovery_work);
4003 3319
4004 cancel_delayed_work_sync(&con->page_retirement_dwork);
4005
4006 amdgpu_ras_ecc_log_fini(&con->umc_ecc_log);
4007
4008 mutex_lock(&con->recovery_lock);3320 mutex_lock(&con->recovery_lock);
4009 con->eh_data = NULL;3321 con->eh_data = NULL;
4010 kfree(data->bps);3322 kfree(data->bps);
@@ -4206,15 +3518,6 @@ init_ras_enabled_flag: · 195 unmodified lines
4206 adev->ras_enabled = amdgpu_ras_enable == 0 ? 0 :3518 adev->ras_enabled = amdgpu_ras_enable == 0 ? 0 :
4207 adev->ras_hw_enabled & amdgpu_ras_mask;3519 adev->ras_hw_enabled & amdgpu_ras_mask;
4208 3520
4209 /* aca is disabled by default except for psp v13_0_6/v13_0_12/v13_0_14 */
4210 if (!amdgpu_sriov_vf(adev)) {
4211 adev->aca.is_enabled =
4212 (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) ||
4213 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) ||
4214 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14) ||
4215 amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 15));
4216 }
4217
4218 /* bad page feature is not applicable to specific app platform */3521 /* bad page feature is not applicable to specific app platform */
4219 if (adev->gmc.is_app_apu &&3522 if (adev->gmc.is_app_apu &&
4220 amdgpu_ip_version(adev, UMC_HWIP, 0) == IP_VERSION(12, 0, 0))3523 amdgpu_ip_version(adev, UMC_HWIP, 0) == IP_VERSION(12, 0, 0))
@@ -4435,15 +3738,6 @@ int amdgpu_ras_init(struct amdgpu_device *adev) · 214 unmodified lines
4435 goto release_con;3738 goto release_con;
4436 }3739 }
4437 3740
4438 if (amdgpu_ras_aca_is_supported(adev)) {
4439 if (amdgpu_aca_is_enabled(adev))
4440 r = amdgpu_aca_init(adev);
4441 else
4442 r = amdgpu_mca_init(adev);
4443 if (r)
4444 goto release_con;
4445 }
4446
4447 con->init_task_pid = task_pid_nr(current);3741 con->init_task_pid = task_pid_nr(current);
4448 get_task_comm(con->init_task_comm, current);3742 get_task_comm(con->init_task_comm, current);
4449 3743
@@ -4541,9 +3835,9 @@ int amdgpu_ras_block_late_init(struct amdgpu_device *adev, · 91 unmodified lines
4541 goto cleanup;3835 goto cleanup;
4542 }3836 }
4543 3837
4544 if (ras_obj->hw_ops &&3838 if (amdgpu_uniras_enabled(adev) || (ras_obj->hw_ops &&
4545 (ras_obj->hw_ops->query_ras_error_count ||3839 (ras_obj->hw_ops->query_ras_error_count ||
4546 ras_obj->hw_ops->query_ras_error_status)) {3840 ras_obj->hw_ops->query_ras_error_status))) {
4547 r = amdgpu_ras_sysfs_create(adev, ras_block);3841 r = amdgpu_ras_sysfs_create(adev, ras_block);
4548 if (r)3842 if (r)
4549 goto interrupt;3843 goto interrupt;
@@ -4671,28 +3965,13 @@ int amdgpu_ras_late_init(struct amdgpu_device *adev) · 121 unmodified lines
4671 3965
4672 amdgpu_ras_event_mgr_init(adev);3966 amdgpu_ras_event_mgr_init(adev);
4673 3967
4674 if (amdgpu_ras_aca_is_supported(adev)) {
4675 if (amdgpu_reset_in_recovery(adev)) {
4676 if (amdgpu_aca_is_enabled(adev))
4677 r = amdgpu_aca_reset(adev);
4678 else
4679 r = amdgpu_mca_reset(adev);
4680 if (r)
4681 return r;
4682 }
4683
4684 if (!amdgpu_sriov_vf(adev)) {
4685 if (amdgpu_aca_is_enabled(adev))
4686 amdgpu_ras_set_aca_debug_mode(adev, false);
4687 else
4688 amdgpu_ras_set_mca_debug_mode(adev, false);
4689 }
4690 }
4691
4692 /* Guest side doesn't need init ras feature */3968 /* Guest side doesn't need init ras feature */
4693 if (amdgpu_sriov_vf(adev) && !amdgpu_sriov_ras_telemetry_en(adev))3969 if (amdgpu_sriov_vf(adev) && !amdgpu_sriov_ras_telemetry_en(adev))
4694 return 0;3970 return 0;
4695 3971
3972 if (amdgpu_uniras_enabled(adev))
3973 amdgpu_ras_mgr_set_debug_mode(adev, false);
3974
4696 list_for_each_entry_safe(node, tmp, &adev->ras_list, node) {3975 list_for_each_entry_safe(node, tmp, &adev->ras_list, node) {
4697 obj = node->ras_obj;3976 obj = node->ras_obj;
4698 if (!obj) {3977 if (!obj) {
@@ -4773,13 +4052,6 @@ int amdgpu_ras_fini(struct amdgpu_device *adev) · 74 unmodified lines
4773 amdgpu_ras_fs_fini(adev);4052 amdgpu_ras_fs_fini(adev);
4774 amdgpu_ras_interrupt_remove_all(adev);4053 amdgpu_ras_interrupt_remove_all(adev);
4775 4054
4776 if (amdgpu_ras_aca_is_supported(adev)) {
4777 if (amdgpu_aca_is_enabled(adev))
4778 amdgpu_aca_fini(adev);
4779 else
4780 amdgpu_mca_fini(adev);
4781 }
4782
4783 WARN(AMDGPU_RAS_GET_FEATURES(con->features), "Feature mask is not cleared");4055 WARN(AMDGPU_RAS_GET_FEATURES(con->features), "Feature mask is not cleared");
4784 4056
4785 if (AMDGPU_RAS_GET_FEATURES(con->features))4057 if (AMDGPU_RAS_GET_FEATURES(con->features))
@@ -5064,6 +4336,13 @@ static void amdgpu_register_bad_pages_mca_notifier(struct amdgpu_device *adev) · 278 unmodified lines
5064 * Use this list instead of mgpu_info to find the amdgpu4336 * Use this list instead of mgpu_info to find the amdgpu
5065 * device on which the UMC error was reported.4337 * device on which the UMC error was reported.
5066 */4338 */
4339 if (mce_adev_list.num_gpu >= MAX_GPU_INSTANCE) {
4340 dev_warn_ratelimited(adev->dev,
4341 "mce_adev_list full, skip notifier registration (max=%d)\n",
4342 MAX_GPU_INSTANCE);
4343 return;
4344 }
4345
5067 mce_adev_list.devs[mce_adev_list.num_gpu++] = adev;4346 mce_adev_list.devs[mce_adev_list.num_gpu++] = adev;
5068 4347
5069 /*4348 /*
@@ -5181,59 +4460,10 @@ int amdgpu_ras_reset_gpu(struct amdgpu_device *adev) · 111 unmodified lines
5181 return 0;4460 return 0;
5182}4461}
5183 4462
5184int amdgpu_ras_set_mca_debug_mode(struct amdgpu_device *adev, bool enable)
5185{
5186 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
5187 int ret = 0;
5188
5189 if (con) {
5190 ret = amdgpu_mca_smu_set_debug_mode(adev, enable);
5191 if (!ret)
5192 con->is_aca_debug_mode = enable;
5193 }
5194
5195 return ret;
5196}
5197
5198int amdgpu_ras_set_aca_debug_mode(struct amdgpu_device *adev, bool enable)
5199{
5200 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
5201 int ret = 0;
5202
5203 if (con) {
5204 if (amdgpu_aca_is_enabled(adev))
5205 ret = amdgpu_aca_smu_set_debug_mode(adev, enable);
5206 else
5207 ret = amdgpu_mca_smu_set_debug_mode(adev, enable);
5208 if (!ret)
5209 con->is_aca_debug_mode = enable;
5210 }
5211
5212 return ret;
5213}
5214
5215bool amdgpu_ras_get_aca_debug_mode(struct amdgpu_device *adev)
5216{
5217 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
5218 const struct aca_smu_funcs *smu_funcs = adev->aca.smu_funcs;
5219 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
5220
5221 if (!con)
5222 return false;
5223
5224 if ((amdgpu_aca_is_enabled(adev) && smu_funcs && smu_funcs->set_debug_mode) ||
5225 (!amdgpu_aca_is_enabled(adev) && mca_funcs && mca_funcs->mca_set_debug_mode))
5226 return con->is_aca_debug_mode;
5227 else
5228 return true;
5229}
5230
5231bool amdgpu_ras_get_error_query_mode(struct amdgpu_device *adev,4463bool amdgpu_ras_get_error_query_mode(struct amdgpu_device *adev,
5232 unsigned int *error_query_mode)4464 unsigned int *error_query_mode)
5233{4465{
5234 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);4466 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
5235 const struct amdgpu_mca_smu_funcs *mca_funcs = adev->mca.mca_funcs;
5236 const struct aca_smu_funcs *smu_funcs = adev->aca.smu_funcs;
5237 4467
5238 if (!con) {4468 if (!con) {
5239 *error_query_mode = AMDGPU_RAS_INVALID_ERROR_QUERY;4469 *error_query_mode = AMDGPU_RAS_INVALID_ERROR_QUERY;
@@ -5242,9 +4472,6 @@ bool amdgpu_ras_get_error_query_mode(struct amdgpu_device *adev, · 2 unmodified lines
5242 4472
5243 if (amdgpu_sriov_vf(adev)) {4473 if (amdgpu_sriov_vf(adev)) {
5244 *error_query_mode = AMDGPU_RAS_VIRT_ERROR_COUNT_QUERY;4474 *error_query_mode = AMDGPU_RAS_VIRT_ERROR_COUNT_QUERY;
5245 } else if ((smu_funcs && smu_funcs->set_debug_mode) || (mca_funcs && mca_funcs->mca_set_debug_mode)) {
5246 *error_query_mode =
5247 (con->is_aca_debug_mode) ? AMDGPU_RAS_DIRECT_ERROR_QUERY : AMDGPU_RAS_FIRMWARE_ERROR_QUERY;
5248 } else {4475 } else {
5249 *error_query_mode = AMDGPU_RAS_DIRECT_ERROR_QUERY;4476 *error_query_mode = AMDGPU_RAS_DIRECT_ERROR_QUERY;
5250 }4477 }
@@ -5834,3 +5061,8 @@ void amdgpu_ras_post_reset(struct amdgpu_device *adev, · 583 unmodified lines
5834 amdgpu_ras_mgr_post_reset(tmp_adev);5061 amdgpu_ras_mgr_post_reset(tmp_adev);
5835 }5062 }
5836}5063}
5064
5065void amdgpu_ras_resume_after_reset(struct amdgpu_device *adev)
5066{
5067 amdgpu_ras_mgr_resume_after_reset(adev);
5068}
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h+3−37
@@ -31,7 +31,6 @@ · 30 unmodified lines
31#include "ta_ras_if.h"31#include "ta_ras_if.h"
32#include "amdgpu_ras_eeprom.h"32#include "amdgpu_ras_eeprom.h"
33#include "amdgpu_smuio.h"33#include "amdgpu_smuio.h"
34#include "amdgpu_aca.h"
35 34
36struct amdgpu_iv_entry;35struct amdgpu_iv_entry;
37 36
@@ -466,14 +465,6 @@ struct ras_query_context { · 428 unmodified lines
466typedef int (*pasid_notify)(struct amdgpu_device *adev,465typedef int (*pasid_notify)(struct amdgpu_device *adev,
467 uint16_t pasid, void *data);466 uint16_t pasid, void *data);
468 467
469struct ras_poison_msg {
470 enum amdgpu_ras_block block;
471 uint16_t pasid;
472 uint32_t reset;
473 pasid_notify pasid_fn;
474 void *data;
475};
476
477struct ras_err_pages {468struct ras_err_pages {
478 uint32_t count;469 uint32_t count;
479 uint64_t *pfn;470 uint64_t *pfn;
@@ -492,8 +483,6 @@ struct ras_ecc_err { · 12 unmodified lines
492struct ras_ecc_log_info {483struct ras_ecc_log_info {
493 struct mutex lock;484 struct mutex lock;
494 struct radix_tree_root de_page_tree;485 struct radix_tree_root de_page_tree;
495 uint64_t de_queried_count;
496 uint64_t consumption_q_count;
497};486};
498 487
499struct ras_critical_region {488struct ras_critical_region {
@@ -549,7 +538,6 @@ struct amdgpu_ras { · 49 unmodified lines
549 /* gpu recovery */538 /* gpu recovery */
550 struct work_struct recovery_work;539 struct work_struct recovery_work;
551 atomic_t in_recovery;540 atomic_t in_recovery;
552 atomic_t rma_in_recovery;
553 struct amdgpu_device *adev;541 struct amdgpu_device *adev;
554 /* error handler data */542 /* error handler data */
555 struct ras_err_handler_data *eh_data;543 struct ras_err_handler_data *eh_data;
@@ -581,22 +569,15 @@ struct amdgpu_ras { · 25 unmodified lines
581 /* Indicates smu whether need update bad channel info */569 /* Indicates smu whether need update bad channel info */
582 bool update_channel_flag;570 bool update_channel_flag;
583 /* Record status of smu mca debug mode */571 /* Record status of smu mca debug mode */
584 bool is_aca_debug_mode;572 bool is_mca_debug_mode;
585 bool is_rma;573 bool is_rma;
586 574
587 /* Record special requirements of gpu reset caller */575 /* Record special requirements of gpu reset caller */
588 uint32_t gpu_reset_flags;576 uint32_t gpu_reset_flags;
589 577
590 struct task_struct *page_retirement_thread;
591 wait_queue_head_t page_retirement_wq;
592 struct mutex page_retirement_lock;578 struct mutex page_retirement_lock;
593 atomic_t page_retirement_req_cnt;
594 atomic_t poison_creation_count;
595 atomic_t poison_consumption_count;
596 struct mutex page_rsv_lock;579 struct mutex page_rsv_lock;
597 DECLARE_KFIFO(poison_fifo, struct ras_poison_msg, 128);
598 struct ras_ecc_log_info umc_ecc_log;580 struct ras_ecc_log_info umc_ecc_log;
599 struct delayed_work page_retirement_dwork;
600 581
601 /* ras errors detected */582 /* ras errors detected */
602 unsigned long ras_err_state;583 unsigned long ras_err_state;
@@ -615,9 +596,6 @@ struct amdgpu_ras { · 12 unmodified lines
615 struct list_head critical_region_head;596 struct list_head critical_region_head;
616 struct mutex critical_region_lock;597 struct mutex critical_region_lock;
617 598
618 /* Protect poison injection */
619 struct mutex poison_lock;
620
621 /* Disable/Enable uniras switch */599 /* Disable/Enable uniras switch */
622 bool uniras_enabled;600 bool uniras_enabled;
623 const struct ras_smu_drv *ras_smu_drv;601 const struct ras_smu_drv *ras_smu_drv;
@@ -702,8 +680,6 @@ struct ras_manager { · 78 unmodified lines
702 struct ras_ih_data ih_data;680 struct ras_ih_data ih_data;
703 681
704 struct ras_err_data err_data;682 struct ras_err_data err_data;
705
706 struct aca_handle aca_handle;
707};683};
708 684
709struct ras_badpage {685struct ras_badpage {
@@ -964,8 +940,7 @@ struct amdgpu_ras* amdgpu_ras_get_context(struct amdgpu_device *adev); · 254 unmodified lines
964int amdgpu_ras_set_context(struct amdgpu_device *adev, struct amdgpu_ras *ras_con);940int amdgpu_ras_set_context(struct amdgpu_device *adev, struct amdgpu_ras *ras_con);
965 941
966int amdgpu_ras_set_mca_debug_mode(struct amdgpu_device *adev, bool enable);942int amdgpu_ras_set_mca_debug_mode(struct amdgpu_device *adev, bool enable);
967int amdgpu_ras_set_aca_debug_mode(struct amdgpu_device *adev, bool enable);943bool amdgpu_ras_get_mca_debug_mode(struct amdgpu_device *adev);
968bool amdgpu_ras_get_aca_debug_mode(struct amdgpu_device *adev);
969bool amdgpu_ras_get_error_query_mode(struct amdgpu_device *adev,944bool amdgpu_ras_get_error_query_mode(struct amdgpu_device *adev,
970 unsigned int *mode);945 unsigned int *mode);
971 946
@@ -1006,12 +981,6 @@ int amdgpu_ras_error_statistic_de_count(struct ras_err_data *err_data, · 34 unmodified lines
1006 struct amdgpu_smuio_mcm_config_info *mcm_info,981 struct amdgpu_smuio_mcm_config_info *mcm_info,
1007 u64 count);982 u64 count);
1008void amdgpu_ras_query_boot_status(struct amdgpu_device *adev, u32 num_instances);983void amdgpu_ras_query_boot_status(struct amdgpu_device *adev, u32 num_instances);
1009int amdgpu_ras_bind_aca(struct amdgpu_device *adev, enum amdgpu_ras_block blk,
1010 const struct aca_info *aca_info, void *data);
1011int amdgpu_ras_unbind_aca(struct amdgpu_device *adev, enum amdgpu_ras_block blk);
1012
1013ssize_t amdgpu_ras_aca_sysfs_read(struct device *dev, struct device_attribute *attr,
1014 struct aca_handle *handle, char *buf, void *data);
1015 984
1016void amdgpu_ras_set_fed(struct amdgpu_device *adev, bool status);985void amdgpu_ras_set_fed(struct amdgpu_device *adev, bool status);
1017bool amdgpu_ras_get_fed_status(struct amdgpu_device *adev);986bool amdgpu_ras_get_fed_status(struct amdgpu_device *adev);
@@ -1029,10 +998,6 @@ int amdgpu_ras_reserve_page(struct amdgpu_device *adev, uint64_t pfn); · 11 unmodified lines
1029int amdgpu_ras_add_critical_region(struct amdgpu_device *adev, struct amdgpu_bo *bo);998int amdgpu_ras_add_critical_region(struct amdgpu_device *adev, struct amdgpu_bo *bo);
1030bool amdgpu_ras_check_critical_address(struct amdgpu_device *adev, uint64_t addr);999bool amdgpu_ras_check_critical_address(struct amdgpu_device *adev, uint64_t addr);
1031 1000
1032int amdgpu_ras_put_poison_req(struct amdgpu_device *adev,
1033 enum amdgpu_ras_block block, uint16_t pasid,
1034 pasid_notify pasid_fn, void *data, uint32_t reset);
1035
1036bool amdgpu_ras_in_recovery(struct amdgpu_device *adev);1001bool amdgpu_ras_in_recovery(struct amdgpu_device *adev);
1037 1002
1038__printf(3, 4)1003__printf(3, 4)
@@ -1045,4 +1010,5 @@ void amdgpu_ras_pre_reset(struct amdgpu_device *adev, · 6 unmodified lines
1045 struct list_head *device_list);1010 struct list_head *device_list);
1046void amdgpu_ras_post_reset(struct amdgpu_device *adev,1011void amdgpu_ras_post_reset(struct amdgpu_device *adev,
1047 struct list_head *device_list);1012 struct list_head *device_list);
1013void amdgpu_ras_resume_after_reset(struct amdgpu_device *adev);
1048#endif1014#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c+119−249
@@ -124,8 +124,6 @@ · 123 unmodified lines
124 RAS_TABLE_V2_1_INFO_SIZE) \124 RAS_TABLE_V2_1_INFO_SIZE) \
125 / RAS_TABLE_RECORD_SIZE)125 / RAS_TABLE_RECORD_SIZE)
126 126
127#define RAS_SMU_MESSAGE_TIMEOUT_MS 1000 /* 1s */
128
129/* Given a zero-based index of an EEPROM RAS record, yields the EEPROM127/* Given a zero-based index of an EEPROM RAS record, yields the EEPROM
130 * offset off of RAS_TABLE_START. That is, this is something you can128 * offset off of RAS_TABLE_START. That is, this is something you can
131 * add to control->i2c_address, and then tell I2C layer to read129 * add to control->i2c_address, and then tell I2C layer to read
@@ -159,6 +157,9 @@ · 27 unmodified lines
159 157
160static bool __is_ras_eeprom_supported(struct amdgpu_device *adev)158static bool __is_ras_eeprom_supported(struct amdgpu_device *adev)
161{159{
160 if (amdgpu_sriov_vf(adev))
161 return false;
162
162 switch (amdgpu_ip_version(adev, MP1_HWIP, 0)) {163 switch (amdgpu_ip_version(adev, MP1_HWIP, 0)) {
163 case IP_VERSION(11, 0, 2): /* VEGA20 and ARCTURUS */164 case IP_VERSION(11, 0, 2): /* VEGA20 and ARCTURUS */
164 case IP_VERSION(11, 0, 7): /* Sienna cichlid */165 case IP_VERSION(11, 0, 7): /* Sienna cichlid */
@@ -449,57 +450,46 @@ int amdgpu_ras_eeprom_reset_table(struct amdgpu_ras_eeprom_control *control) · 284 unmodified lines
449 struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr;450 struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr;
450 struct amdgpu_ras_eeprom_table_ras_info *rai = &control->tbl_rai;451 struct amdgpu_ras_eeprom_table_ras_info *rai = &control->tbl_rai;
451 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);452 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
452 u32 erase_res = 0;
453 u8 csum;453 u8 csum;
454 int res;454 int res;
455 455
456 mutex_lock(&control->ras_tbl_mutex);456 mutex_lock(&control->ras_tbl_mutex);
457 457
458 if (!amdgpu_ras_smu_eeprom_supported(adev)) {458 hdr->header = RAS_TABLE_HDR_VAL;
459 hdr->header = RAS_TABLE_HDR_VAL;459 amdgpu_ras_set_eeprom_table_version(control);
460 amdgpu_ras_set_eeprom_table_version(control);
461
462 if (hdr->version >= RAS_TABLE_VER_V2_1) {
463 hdr->first_rec_offset = RAS_RECORD_START_V2_1;
464 hdr->tbl_size = RAS_TABLE_HEADER_SIZE +
465 RAS_TABLE_V2_1_INFO_SIZE;
466 rai->rma_status = GPU_HEALTH_USABLE;
467
468 control->ras_record_offset = RAS_RECORD_START_V2_1;
469 control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1;
470 /**
471 * GPU health represented as a percentage.
472 * 0 means worst health, 100 means fully health.
473 */
474 rai->health_percent = 100;
475 /* ecc_page_threshold = 0 means disable bad page retirement */
476 rai->ecc_page_threshold = con->bad_page_cnt_threshold;
477 } else {
478 hdr->first_rec_offset = RAS_RECORD_START;
479 hdr->tbl_size = RAS_TABLE_HEADER_SIZE;
480 460
481 control->ras_record_offset = RAS_RECORD_START;461 if (hdr->version >= RAS_TABLE_VER_V2_1) {
482 control->ras_max_record_count = RAS_MAX_RECORD_COUNT;462 hdr->first_rec_offset = RAS_RECORD_START_V2_1;
483 }463 hdr->tbl_size = RAS_TABLE_HEADER_SIZE +
464 RAS_TABLE_V2_1_INFO_SIZE;
465 rai->rma_status = GPU_HEALTH_USABLE;
484 466
485 csum = __calc_hdr_byte_sum(control);467 control->ras_record_offset = RAS_RECORD_START_V2_1;
486 if (hdr->version >= RAS_TABLE_VER_V2_1)468 control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1;
487 csum += __calc_ras_info_byte_sum(control);469 /**
488 csum = -csum;470 * GPU health represented as a percentage.
489 hdr->checksum = csum;471 * 0 means worst health, 100 means fully health.
490 res = __write_table_header(control);472 */
491 if (!res && hdr->version > RAS_TABLE_VER_V1)473 rai->health_percent = 100;
492 res = __write_table_ras_info(control);474 /* ecc_page_threshold = 0 means disable bad page retirement */
475 rai->ecc_page_threshold = con->bad_page_cnt_threshold;
493 } else {476 } else {
494 res = amdgpu_ras_smu_erase_ras_table(adev, &erase_res);477 hdr->first_rec_offset = RAS_RECORD_START;
495 if (res || erase_res) {478 hdr->tbl_size = RAS_TABLE_HEADER_SIZE;
496 dev_warn(adev->dev, "RAS EEPROM reset failed, res:%d result:%d",479
497 res, erase_res);480 control->ras_record_offset = RAS_RECORD_START;
498 if (!res)481 control->ras_max_record_count = RAS_MAX_RECORD_COUNT;
499 res = -EIO;
500 }
501 }482 }
502 483
484 csum = __calc_hdr_byte_sum(control);
485 if (hdr->version >= RAS_TABLE_VER_V2_1)
486 csum += __calc_ras_info_byte_sum(control);
487 csum = -csum;
488 hdr->checksum = csum;
489 res = __write_table_header(control);
490 if (!res && hdr->version > RAS_TABLE_VER_V1)
491 res = __write_table_ras_info(control);
492
503 control->ras_num_recs = 0;493 control->ras_num_recs = 0;
504 control->ras_num_bad_pages = 0;494 control->ras_num_bad_pages = 0;
505 control->ras_num_mca_recs = 0;495 control->ras_num_mca_recs = 0;
@@ -662,7 +652,6 @@ amdgpu_ras_eeprom_append_table(struct amdgpu_ras_eeprom_control *control, · 156 unmodified lines
662 const u32 num)652 const u32 num)
663{653{
664 struct amdgpu_ras *con = amdgpu_ras_get_context(to_amdgpu_device(control));654 struct amdgpu_ras *con = amdgpu_ras_get_context(to_amdgpu_device(control));
665 struct amdgpu_device *adev = to_amdgpu_device(control);
666 u32 a, b, i;655 u32 a, b, i;
667 u8 *buf, *pp;656 u8 *buf, *pp;
668 int res;657 int res;
@@ -767,10 +756,7 @@ amdgpu_ras_eeprom_append_table(struct amdgpu_ras_eeprom_control *control, · 98 unmodified lines
767 % control->ras_max_record_count;756 % control->ras_max_record_count;
768 757
769 /*old asics only save pa to eeprom like before*/758 /*old asics only save pa to eeprom like before*/
770 if (IP_VERSION_MAJ(amdgpu_ip_version(adev, UMC_HWIP, 0)) < 12)759 control->ras_num_pa_recs += num;
771 control->ras_num_pa_recs += num;
772 else
773 control->ras_num_mca_recs += num;
774 760
775 control->ras_num_bad_pages = con->bad_page_num;761 control->ras_num_bad_pages = con->bad_page_num;
776Out:762Out:
@@ -879,71 +865,6 @@ Out: · 102 unmodified lines
879 return res;865 return res;
880}866}
881 867
882int amdgpu_ras_eeprom_update_record_num(struct amdgpu_ras_eeprom_control *control)
883{
884 struct amdgpu_device *adev = to_amdgpu_device(control);
885 int ret, retry = 20;
886
887 if (!amdgpu_ras_smu_eeprom_supported(adev))
888 return 0;
889
890 control->ras_num_recs_old = control->ras_num_recs;
891
892 do {
893 /* 1000ms timeout is long enough, smu_get_badpage_count won't
894 * return -EBUSY before timeout.
895 */
896 ret = amdgpu_ras_smu_get_badpage_count(adev,
897 &(control->ras_num_recs), RAS_SMU_MESSAGE_TIMEOUT_MS);
898 if (!ret &&
899 (control->ras_num_recs_old == control->ras_num_recs)) {
900 /* record number update in PMFW needs some time,
901 * smu_get_badpage_count may return immediately without
902 * count update, sleep for a while and retry again.
903 */
904 msleep(50);
905 retry--;
906 } else {
907 break;
908 }
909 } while (retry);
910
911 /* no update of record number is not a real failure,
912 * don't print warning here
913 */
914 if (!ret && (control->ras_num_recs_old == control->ras_num_recs))
915 ret = -EINVAL;
916
917 return ret;
918}
919
920static int amdgpu_ras_smu_eeprom_append(struct amdgpu_ras_eeprom_control *control)
921{
922 struct amdgpu_device *adev = to_amdgpu_device(control);
923 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
924
925 if (!amdgpu_ras_smu_eeprom_supported(adev) || !con)
926 return 0;
927
928 control->ras_num_bad_pages = con->bad_page_num;
929
930 if (amdgpu_bad_page_threshold != 0 &&
931 control->ras_num_bad_pages > con->bad_page_cnt_threshold) {
932 dev_warn(adev->dev,
933 "Saved bad pages %d reaches threshold value %d\n",
934 control->ras_num_bad_pages, con->bad_page_cnt_threshold);
935
936 if (adev->cper.enabled && amdgpu_cper_generate_bp_threshold_record(adev))
937 dev_warn(adev->dev, "fail to generate bad page threshold cper records\n");
938
939 if ((amdgpu_bad_page_threshold != -1) &&
940 (amdgpu_bad_page_threshold != -2))
941 con->is_rma = true;
942 }
943
944 return 0;
945}
946
947/**868/**
948 * amdgpu_ras_eeprom_append -- append records to the EEPROM RAS table869 * amdgpu_ras_eeprom_append -- append records to the EEPROM RAS table
949 * @control: pointer to control structure870 * @control: pointer to control structure
@@ -968,9 +889,6 @@ int amdgpu_ras_eeprom_append(struct amdgpu_ras_eeprom_control *control, · 18 unmodified lines
968 if (!__is_ras_eeprom_supported(adev))889 if (!__is_ras_eeprom_supported(adev))
969 return 0;890 return 0;
970 891
971 if (amdgpu_ras_smu_eeprom_supported(adev))
972 return amdgpu_ras_smu_eeprom_append(control);
973
974 if (num == 0) {892 if (num == 0) {
975 dev_err(adev->dev, "will not append 0 records\n");893 dev_err(adev->dev, "will not append 0 records\n");
976 return -EINVAL;894 return -EINVAL;
@@ -1046,52 +964,6 @@ static int __amdgpu_ras_eeprom_read(struct amdgpu_ras_eeprom_control *control, · 69 unmodified lines
1046 return res;964 return res;
1047}965}
1048 966
1049int amdgpu_ras_eeprom_read_idx(struct amdgpu_ras_eeprom_control *control,
1050 struct eeprom_table_record *record, u32 rec_idx,
1051 const u32 num)
1052{
1053 struct amdgpu_device *adev = to_amdgpu_device(control);
1054 uint64_t ts, end_idx;
1055 int i, ret;
1056 u64 mca, ipid;
1057 u32 cu, mem_channel, mcumc_id;
1058
1059 if (!amdgpu_ras_smu_eeprom_supported(adev))
1060 return 0;
1061
1062 if (!adev->umc.ras || !adev->umc.ras->mca_ipid_parse)
1063 return -EOPNOTSUPP;
1064
1065 end_idx = rec_idx + num;
1066 for (i = rec_idx; i < end_idx; i++) {
1067 ret = amdgpu_ras_smu_get_badpage_mca_addr(adev, i, &mca);
1068 if (ret)
1069 return ret;
1070
1071 ret = amdgpu_ras_smu_get_badpage_ipid(adev, i, &ipid);
1072 if (ret)
1073 return ret;
1074
1075 ret = amdgpu_ras_smu_get_timestamp(adev, i, &ts);
1076 if (ret)
1077 return ret;
1078
1079 record[i - rec_idx].address = mca;
1080 /* retired_page (pa) is unused now */
1081 record[i - rec_idx].retired_page = 0x1ULL;
1082 record[i - rec_idx].ts = ts;
1083 record[i - rec_idx].err_type = AMDGPU_RAS_EEPROM_ERR_NON_RECOVERABLE;
1084
1085 adev->umc.ras->mca_ipid_parse(adev, ipid,
1086 &cu, &mem_channel, &mcumc_id, NULL);
1087 record[i - rec_idx].cu = (u8)cu;
1088 record[i - rec_idx].mem_channel = (u8)mem_channel;
1089 record[i - rec_idx].mcumc_id = (u8)mcumc_id;
1090 }
1091
1092 return 0;
1093}
1094
1095/**967/**
1096 * amdgpu_ras_eeprom_read -- read EEPROM968 * amdgpu_ras_eeprom_read -- read EEPROM
1097 * @control: pointer to control structure969 * @control: pointer to control structure
@@ -1113,9 +985,6 @@ int amdgpu_ras_eeprom_read(struct amdgpu_ras_eeprom_control *control, · 15 unmodified lines
1113 u8 *buf, *pp;985 u8 *buf, *pp;
1114 u32 g0, g1;986 u32 g0, g1;
1115 987
1116 if (amdgpu_ras_smu_eeprom_supported(adev))
1117 return amdgpu_ras_eeprom_read_idx(control, record, 0, num);
1118
1119 if (!__is_ras_eeprom_supported(adev))988 if (!__is_ras_eeprom_supported(adev))
1120 return 0;989 return 0;
1121 990
@@ -1395,6 +1264,86 @@ Out: · 273 unmodified lines
1395 return res < 0 ? res : orig_size - size;1264 return res < 0 ? res : orig_size - size;
1396}1265}
1397 1266
1267static ssize_t
1268amdgpu_ras_debugfs_table_read_uniras(struct amdgpu_device *adev,
1269 char __user *buf,
1270 size_t size, loff_t *pos)
1271{
1272 struct amdgpu_ras_mgr *ras_mgr = amdgpu_ras_mgr_get_context(adev);
1273 struct ras_core_context *ras_core = ras_mgr ? ras_mgr->ras_core : NULL;
1274 struct eeprom_umc_record *records = NULL;
1275 struct ras_eeprom_control *control;
1276 size_t bufsz, len = 0;
1277 u32 num_recs;
1278 char *kbuf;
1279 ssize_t res;
1280 int i;
1281
1282 if (!ras_core)
1283 return 0;
1284
1285 /* pmfw manages eeprom data by itself */
1286 if (ras_fw_eeprom_supported(ras_core))
1287 return 0;
1288
1289 control = &ras_core->ras_eeprom;
1290 num_recs = ras_eeprom_get_record_count(ras_core);
1291
1292 bufsz = strlen(tbl_hdr_str) + tbl_hdr_fmt_size +
1293 strlen(rec_hdr_str) + (size_t)rec_hdr_fmt_size * num_recs + 1;
1294
1295 kbuf = kvmalloc(bufsz, GFP_KERNEL);
1296 if (!kbuf)
1297 return -ENOMEM;
1298
1299 if (num_recs) {
1300 records = kvcalloc(num_recs, sizeof(*records), GFP_KERNEL);
1301 if (!records) {
1302 res = -ENOMEM;
1303 goto out;
1304 }
1305
1306 res = ras_eeprom_read(ras_core, records, num_recs);
1307 if (res)
1308 goto out;
1309 }
1310
1311 len += scnprintf(kbuf + len, bufsz - len, "%s", tbl_hdr_str);
1312 len += scnprintf(kbuf + len, bufsz - len, tbl_hdr_fmt,
1313 control->tbl_hdr.header,
1314 control->tbl_hdr.version,
1315 control->tbl_hdr.first_rec_offset,
1316 control->tbl_hdr.tbl_size,
1317 control->tbl_hdr.checksum);
1318 len += scnprintf(kbuf + len, bufsz - len, "%s", rec_hdr_str);
1319
1320 for (i = 0; i < num_recs; i++) {
1321 u32 ai = RAS_RI_TO_AI(control, i);
1322 int et = records[i].err_type;
1323 const char *ets = (et >= 0 && et < AMDGPU_RAS_EEPROM_ERR_COUNT) ?
1324 record_err_type_str[et] : "na";
1325
1326 len += scnprintf(kbuf + len, bufsz - len, rec_hdr_fmt,
1327 i,
1328 RAS_INDEX_TO_OFFSET(control, ai),
1329 ets,
1330 records[i].bank,
1331 records[i].ts,
1332 records[i].offset,
1333 records[i].mem_channel,
1334 records[i].mcumc_id,
1335 records[i].retired_row_pfn);
1336 }
1337
1338 res = simple_read_from_buffer(buf, size, pos, kbuf, len);
1339
1340out:
1341 kvfree(records);
1342 kvfree(kbuf);
1343
1344 return res;
1345}
1346
1398static ssize_t1347static ssize_t
1399amdgpu_ras_debugfs_eeprom_table_read(struct file *f, char __user *buf,1348amdgpu_ras_debugfs_eeprom_table_read(struct file *f, char __user *buf,
1400 size_t size, loff_t *pos)1349 size_t size, loff_t *pos)
@@ -1408,6 +1357,10 @@ amdgpu_ras_debugfs_eeprom_table_read(struct file *f, char __user *buf, · 7 unmodified lines
1408 if (!size)1357 if (!size)
1409 return size;1358 return size;
1410 1359
1360 if (amdgpu_uniras_enabled(adev))
1361 return amdgpu_ras_debugfs_table_read_uniras(adev, buf,
1362 size, pos);
1363
1411 if (!ras || !control) {1364 if (!ras || !control) {
1412 res = snprintf(data, sizeof(data), "Not supported\n");1365 res = snprintf(data, sizeof(data), "Not supported\n");
1413 if (*pos >= res)1366 if (*pos >= res)
@@ -1521,42 +1474,6 @@ Out: · 107 unmodified lines
1521 return res == RAS_TABLE_V2_1_INFO_SIZE ? 0 : res;1474 return res == RAS_TABLE_V2_1_INFO_SIZE ? 0 : res;
1522}1475}
1523 1476
1524static int amdgpu_ras_smu_eeprom_init(struct amdgpu_ras_eeprom_control *control)
1525{
1526 struct amdgpu_device *adev = to_amdgpu_device(control);
1527 struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr;
1528 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
1529 uint64_t local_time;
1530 int res;
1531
1532 ras->is_rma = false;
1533
1534 if (!__is_ras_eeprom_supported(adev))
1535 return 0;
1536 mutex_init(&control->ras_tbl_mutex);
1537
1538 res = amdgpu_ras_smu_get_table_version(adev, &(hdr->version));
1539 if (res)
1540 return res;
1541
1542 res = amdgpu_ras_smu_get_badpage_count(adev,
1543 &(control->ras_num_recs), 100);
1544 if (res)
1545 return res;
1546
1547 local_time = (uint64_t)ktime_get_real_seconds();
1548 res = amdgpu_ras_smu_set_timestamp(adev, local_time);
1549 if (res)
1550 return res;
1551
1552 control->ras_max_record_count = 4000;
1553
1554 control->ras_num_mca_recs = 0;
1555 control->ras_num_pa_recs = 0;
1556
1557 return 0;
1558}
1559
1560int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control)1477int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control)
1561{1478{
1562 struct amdgpu_device *adev = to_amdgpu_device(control);1479 struct amdgpu_device *adev = to_amdgpu_device(control);
@@ -1567,9 +1484,6 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 4 unmodified lines
1567 uint32_t vram_type = adev->gmc.vram_type;1484 uint32_t vram_type = adev->gmc.vram_type;
1568 int res;1485 int res;
1569 1486
1570 if (amdgpu_ras_smu_eeprom_supported(adev))
1571 return amdgpu_ras_smu_eeprom_init(control);
1572
1573 ras->is_rma = false;1487 ras->is_rma = false;
1574 1488
1575 if (!__is_ras_eeprom_supported(adev))1489 if (!__is_ras_eeprom_supported(adev))
@@ -1663,47 +1577,6 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 87 unmodified lines
1663 return 0;1577 return 0;
1664}1578}
1665 1579
1666static int amdgpu_ras_smu_eeprom_check(struct amdgpu_ras_eeprom_control *control)
1667{
1668 struct amdgpu_device *adev = to_amdgpu_device(control);
1669 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
1670
1671 if (!__is_ras_eeprom_supported(adev))
1672 return 0;
1673
1674 control->ras_num_bad_pages = ras->bad_page_num;
1675
1676 if ((ras->bad_page_cnt_threshold < control->ras_num_bad_pages) &&
1677 amdgpu_bad_page_threshold != 0) {
1678 dev_warn(adev->dev,
1679 "RAS records:%d exceed threshold:%d\n",
1680 control->ras_num_bad_pages, ras->bad_page_cnt_threshold);
1681 if ((amdgpu_bad_page_threshold == -1) ||
1682 (amdgpu_bad_page_threshold == -2)) {
1683 dev_warn(adev->dev,
1684 "Please consult AMD Service Action Guide (SAG) for appropriate service procedures\n");
1685 } else {
1686 ras->is_rma = true;
1687 dev_warn(adev->dev,
1688 "User defined threshold is set, runtime service will be halt when threshold is reached\n");
1689 }
1690
1691 return 0;
1692 }
1693
1694 dev_dbg(adev->dev,
1695 "Found existing EEPROM table with %d records",
1696 control->ras_num_bad_pages);
1697
1698 /* Warn if we are at 90% of the threshold or above
1699 */
1700 if (10 * control->ras_num_bad_pages >= 9 * ras->bad_page_cnt_threshold)
1701 dev_warn(adev->dev, "RAS records:%u exceeds 90%% of threshold:%d",
1702 control->ras_num_bad_pages,
1703 ras->bad_page_cnt_threshold);
1704 return 0;
1705}
1706
1707int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control)1580int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control)
1708{1581{
1709 struct amdgpu_device *adev = to_amdgpu_device(control);1582 struct amdgpu_device *adev = to_amdgpu_device(control);
@@ -1711,9 +1584,6 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 1 unmodified line
1711 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);1584 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
1712 int res = 0;1585 int res = 0;
1713 1586
1714 if (amdgpu_ras_smu_eeprom_supported(adev))
1715 return amdgpu_ras_smu_eeprom_check(control);
1716
1717 if (!__is_ras_eeprom_supported(adev))1587 if (!__is_ras_eeprom_supported(adev))
1718 return 0;1588 return 0;
1719 1589
@@ -1973,7 +1843,7 @@ void amdgpu_ras_check_bad_page_status(struct amdgpu_device *adev) · 253 unmodified lines
1973 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);1843 struct amdgpu_ras *ras = amdgpu_ras_get_context(adev);
1974 struct amdgpu_ras_eeprom_control *control = ras ? &ras->eeprom_control : NULL;1844 struct amdgpu_ras_eeprom_control *control = ras ? &ras->eeprom_control : NULL;
1975 1845
1976 if (!control || amdgpu_bad_page_threshold == 0)1846 if (!__is_ras_eeprom_supported(adev) || !control || amdgpu_bad_page_threshold == 0)
1977 return;1847 return;
1978 1848
1979 if (control->ras_num_bad_pages > ras->bad_page_cnt_threshold) {1849 if (control->ras_num_bad_pages > ras->bad_page_cnt_threshold) {
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h+0−3
@@ -82,7 +82,6 @@ struct amdgpu_ras_eeprom_control { · 81 unmodified lines
82 /* Number of records in the table.82 /* Number of records in the table.
83 */83 */
84 u32 ras_num_recs;84 u32 ras_num_recs;
85 u32 ras_num_recs_old;
86 85
87 /* the bad page number is ras_num_recs or86 /* the bad page number is ras_num_recs or
88 * ras_num_recs * umc.retire_unit87 * ras_num_recs * umc.retire_unit
@@ -191,8 +190,6 @@ int amdgpu_ras_eeprom_read_idx(struct amdgpu_ras_eeprom_control *control, · 102 unmodified lines
191 struct eeprom_table_record *record, u32 rec_idx,190 struct eeprom_table_record *record, u32 rec_idx,
192 const u32 num);191 const u32 num);
193 192
194int amdgpu_ras_eeprom_update_record_num(struct amdgpu_ras_eeprom_control *control);
195
196void amdgpu_ras_check_bad_page_status(struct amdgpu_device *adev);193void amdgpu_ras_check_bad_page_status(struct amdgpu_device *adev);
197 194
198extern const struct file_operations amdgpu_ras_debugfs_eeprom_size_ops;195extern const struct file_operations amdgpu_ras_debugfs_eeprom_size_ops;
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c+188−0
@@ -935,6 +935,194 @@ int amdgpu_ring_reset_helper_end(struct amdgpu_ring *ring, · 934 unmodified lines
935 return 0;935 return 0;
936}936}
937 937
938/**
939 * amdgpu_multi_ring_reset_helper_begin() - Prepare multiple rings for a reset.
940 *
941 * @ring_type_mask: Bitmask of affected ring types
942 * @guilty_ring: The ring which is guilty of causing a reset.
943 * @guilty_fence: The fence which didn't signal on the guilty ring.
944 *
945 * Useful when performing a GPU reset method that affects
946 * multiple rings at the same time, such as an IP block soft
947 * reset. For example, a GFX IP block soft reset will affect
948 * every graphics and compute queue.
949 *
950 * This function should be called before such a reset.
951 *
952 * Prepare the affected rings before the reset, make sure to
953 * minimize collateral damage, and backup the contents of
954 * the rings. Then the caller can call the actual HW specific
955 * reset function.
956 *
957 * After the reset is complete, the caller should then call
958 * amdgpu_multi_ring_reset_helper_end() to restore the rings.
959 */
960void amdgpu_multi_ring_reset_helper_begin(const u32 ring_type_mask,
961 struct amdgpu_ring *guilty_ring,
962 struct amdgpu_fence *guilty_fence)
963{
964 struct amdgpu_device *adev = guilty_ring->adev;
965 struct amdgpu_fence *ring_guilty_fence;
966 struct amdgpu_ring *ring;
967 bool rings_busy;
968 int i;
969 u32 t;
970
971 for (i = 0; i < adev->num_rings; ++i) {
972 ring = adev->rings[i];
973
974 if (!(BIT(ring->funcs->type) & ring_type_mask))
975 continue;
976
977 /* Don't accept new submissions on the ring. */
978 if (amdgpu_ring_sched_ready(ring) && !drm_sched_is_stopped(&ring->sched))
979 drm_sched_wqueue_stop(&ring->sched);
980
981 /*
982 * Clear the preempt condition to stop the ring
983 * from starting its next submission. This ensures
984 * that only the currently executing submission
985 * can be rejected because of the reset and helps
986 * minimize collateral damage.
987 */
988 if (ring->funcs->init_cond_exec)
989 amdgpu_ring_set_preempt_cond_exec(ring, false);
990 }
991
992 /* Flush HDP cache so the GPU can see the updated COND_EXEC values */
993 amdgpu_device_flush_hdp(adev, NULL);
994
995 /*
996 * Give some time for non-guilty rings to finish their
997 * current submission, to try to minimize collateral damage.
998 *
999 * Note that this is just a best effort, but really there
1000 * is no way to really know which ring is actually responsible
1001 * because different rings may share resources, eg. a compute
1002 * ring may hog shader engines, causing a graphics ring to hang.
1003 */
1004 for (t = 0; t < adev->usec_timeout; t += 10000) {
1005 rings_busy = false;
1006
1007 /* Check if any of the non-guilty rings are busy */
1008 for (i = 0; i < adev->num_rings; ++i) {
1009 ring = adev->rings[i];
1010
1011 if (!(BIT(ring->funcs->type) & ring_type_mask))
1012 continue;
1013
1014 if (ring == guilty_ring)
1015 continue;
1016
1017 rings_busy |=
1018 atomic_read(&ring->fence_drv.last_seq) !=
1019 READ_ONCE(ring->fence_drv.sync_seq);
1020 }
1021
1022 if (!rings_busy)
1023 break;
1024
1025 mdelay(10);
1026 }
1027
1028 for (i = 0; i < adev->num_rings; ++i) {
1029 ring = adev->rings[i];
1030
1031 if (!(BIT(ring->funcs->type) & ring_type_mask))
1032 continue;
1033
1034 /*
1035 * Find guilty fences, ie. the fences that didn't signal
1036 * on each ring. At this point there is no way to know
1037 * which one is really responsible for the hang, and no
1038 * way to save any of them, so we treat all of them as guilty.
1039 */
1040 ring_guilty_fence =
1041 ring == guilty_ring ? guilty_fence :
1042 amdgpu_ring_find_guilty_fence(ring);
1043
1044 /*
1045 * Backup current contents of the ring.
1046 * The helper takes care to only reemit unsignalled fences
1047 * so we don't have to worry about that here.
1048 */
1049 amdgpu_ring_reset_helper_begin(ring, ring_guilty_fence);
1050 }
1051}
1052
1053/**
1054 * amdgpu_multi_ring_reset_helper_end() - Prepare multiple rings for a reset.
1055 *
1056 * @ring_type_mask: Bitmask of affected ring types
1057 * @guilty_ring: The ring which is guilty of causing a reset.
1058 * @ret: Return code from the reset function.
1059 *
1060 * After calling amdgpu_multi_ring_reset_helper_begin()
1061 * and executing the actual reset method, call this
1062 * function to restore normal operation.
1063 *
1064 * In case the reset failed, this function should still
1065 * be called to restore preemption state, but it won't attempt to
1066 * fully restore the ring contents.
1067 */
1068int amdgpu_multi_ring_reset_helper_end(const u32 ring_type_mask,
1069 struct amdgpu_ring *guilty_ring, int ret)
1070{
1071 struct amdgpu_device *adev = guilty_ring->adev;
1072 struct amdgpu_ring *ring;
1073 int i, r;
1074
1075 /* Set preempt condition, rings are now allowed to execute submissions */
1076 for (i = 0; i < adev->num_rings; ++i) {
1077 ring = adev->rings[i];
1078
1079 if (!(BIT(ring->funcs->type) & ring_type_mask))
1080 continue;
1081
1082 if (ring->funcs->init_cond_exec)
1083 amdgpu_ring_set_preempt_cond_exec(ring, true);
1084 }
1085
1086 /* Flush HDP cache so the GPU can see the updated COND_EXEC values */
1087 amdgpu_device_flush_hdp(adev, NULL);
1088
1089 /* If the reset was unsuccessful, return without restoring anything else. */
1090 if (ret)
1091 return ret;
1092
1093 /* Restore contents of all rings */
1094 for (i = 0; i < adev->num_rings; ++i) {
1095 ring = adev->rings[i];
1096
1097 if (!(BIT(ring->funcs->type) & ring_type_mask))
1098 continue;
1099
1100 /* Restore contents of the ring */
1101 r = amdgpu_ring_reset_helper_end(ring, ring->guilty_fence);
1102 if (r) {
1103 dev_err(adev->dev,
1104 "Failed to recover ring %s after soft reset\n",
1105 ring->name);
1106 return r;
1107 }
1108 }
1109
1110 /* Accept submissions on all rings again */
1111 for (i = 0; i < adev->num_rings; ++i) {
1112 ring = adev->rings[i];
1113
1114 if (!(BIT(ring->funcs->type) & ring_type_mask))
1115 continue;
1116
1117 if (!amdgpu_ring_sched_ready(ring))
1118 continue;
1119
1120 drm_sched_wqueue_start(&ring->sched);
1121 }
1122
1123 return 0;
1124}
1125
938bool amdgpu_ring_is_reset_type_supported(struct amdgpu_ring *ring,1126bool amdgpu_ring_is_reset_type_supported(struct amdgpu_ring *ring,
939 u32 reset_type)1127 u32 reset_type)
940{1128{
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h+8−0
@@ -314,6 +314,7 @@ struct amdgpu_ring { · 313 unmodified lines
314 uint32_t *ring_backup;314 uint32_t *ring_backup;
315 unsigned int ring_backup_entries_to_copy;315 unsigned int ring_backup_entries_to_copy;
316 bool reemit;316 bool reemit;
317 struct amdgpu_fence *guilty_fence;
317 unsigned rptr_offs;318 unsigned rptr_offs;
318 u64 rptr_gpu_addr;319 u64 rptr_gpu_addr;
319 u32 *rptr_cpu_addr;320 u32 *rptr_cpu_addr;
@@ -588,10 +589,17 @@ int amdgpu_ib_ring_tests(struct amdgpu_device *adev); · 268 unmodified lines
588bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);589bool amdgpu_ring_sched_ready(struct amdgpu_ring *ring);
589void amdgpu_ring_backup_unprocessed_commands(struct amdgpu_ring *ring,590void amdgpu_ring_backup_unprocessed_commands(struct amdgpu_ring *ring,
590 struct amdgpu_fence *guilty_fence);591 struct amdgpu_fence *guilty_fence);
592struct amdgpu_fence *
593amdgpu_ring_find_guilty_fence(struct amdgpu_ring *ring);
591void amdgpu_ring_reset_helper_begin(struct amdgpu_ring *ring,594void amdgpu_ring_reset_helper_begin(struct amdgpu_ring *ring,
592 struct amdgpu_fence *guilty_fence);595 struct amdgpu_fence *guilty_fence);
593int amdgpu_ring_reset_helper_end(struct amdgpu_ring *ring,596int amdgpu_ring_reset_helper_end(struct amdgpu_ring *ring,
594 struct amdgpu_fence *guilty_fence);597 struct amdgpu_fence *guilty_fence);
598void amdgpu_multi_ring_reset_helper_begin(const u32 ring_type_mask,
599 struct amdgpu_ring *guilty_ring,
600 struct amdgpu_fence *guilty_fence);
601int amdgpu_multi_ring_reset_helper_end(const u32 ring_type_mask,
602 struct amdgpu_ring *guilty_ring, int ret);
595bool amdgpu_ring_is_reset_type_supported(struct amdgpu_ring *ring,603bool amdgpu_ring_is_reset_type_supported(struct amdgpu_ring *ring,
596 u32 reset_type);604 u32 reset_type);
597#endif605#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.c+39−0
@@ -583,3 +583,42 @@ int amdgpu_gfx_rlc_init_microcode(struct amdgpu_device *adev, · 582 unmodified lines
583 amdgpu_gfx_rlc_init_microcode_v2_5(adev);583 amdgpu_gfx_rlc_init_microcode_v2_5(adev);
584 return 0;584 return 0;
585}585}
586
587static const struct amdgpu_rlc_reg_funcs amdgpu_sriov_rlc_reg_funcs = {
588 .rreg32 = amdgpu_sriov_rreg,
589 .wreg32 = amdgpu_sriov_wreg,
590};
591
592static u32
593amdgpu_rlc_rreg(struct amdgpu_device *adev, u32 reg, u32 acc_flags, u32 hwip,
594 u32 xcc_id)
595{
596 return amdgpu_device_rreg(adev, reg, 0);
597}
598
599static void
600amdgpu_rlc_wreg(struct amdgpu_device *adev, u32 reg, u32 value, u32 acc_flags,
601 u32 hwip, u32 xcc_id)
602{
603 amdgpu_device_wreg(adev, reg, value, 0);
604}
605
606static const struct amdgpu_rlc_reg_funcs amdgpu_rlc_reg_funcs = {
607 .rreg32 = amdgpu_rlc_rreg,
608 .wreg32 = amdgpu_rlc_wreg,
609};
610
611void amdgpu_early_init_rlc_reg_funcs(struct amdgpu_device *adev)
612{
613 adev->gfx.rlc.reg_funcs = &amdgpu_rlc_reg_funcs;
614}
615
616void amdgpu_init_rlc_reg_funcs(struct amdgpu_device *adev)
617{
618 if (amdgpu_sriov_vf(adev) &&
619 adev->gfx.rlc.funcs &&
620 adev->gfx.rlc.rlcg_reg_access_supported)
621 adev->gfx.rlc.reg_funcs = &amdgpu_sriov_rlc_reg_funcs;
622 else
623 adev->gfx.rlc.reg_funcs = &amdgpu_rlc_reg_funcs;
624}
drivers/gpu/drm/amd/amdgpu/amdgpu_rlc.h+10−0
@@ -262,6 +262,11 @@ struct amdgpu_rlc_funcs { · 261 unmodified lines
262 bool (*is_rlcg_access_range)(struct amdgpu_device *adev, uint32_t reg);262 bool (*is_rlcg_access_range)(struct amdgpu_device *adev, uint32_t reg);
263};263};
264 264
265struct amdgpu_rlc_reg_funcs {
266 u32 (*rreg32)(struct amdgpu_device *adev, u32 reg, u32 acc_flags, u32 hwip, u32 xcc_id);
267 void (*wreg32)(struct amdgpu_device *adev, u32 reg, u32 val, u32 acc_flags, u32 hwip, u32 xcc_id);
268};
269
265struct amdgpu_rlcg_reg_access_ctrl {270struct amdgpu_rlcg_reg_access_ctrl {
266 uint32_t scratch_reg0;271 uint32_t scratch_reg0;
267 uint32_t scratch_reg1;272 uint32_t scratch_reg1;
@@ -303,6 +308,7 @@ struct amdgpu_rlc { · 35 unmodified lines
303 /* safe mode for updating CG/PG state */308 /* safe mode for updating CG/PG state */
304 bool in_safe_mode[AMDGPU_MAX_RLC_INSTANCES];309 bool in_safe_mode[AMDGPU_MAX_RLC_INSTANCES];
305 const struct amdgpu_rlc_funcs *funcs;310 const struct amdgpu_rlc_funcs *funcs;
311 const struct amdgpu_rlc_reg_funcs *reg_funcs;
306 312
307 /* for firmware data */313 /* for firmware data */
308 u32 save_and_restore_offset;314 u32 save_and_restore_offset;
@@ -374,4 +380,8 @@ void amdgpu_gfx_rlc_fini(struct amdgpu_device *adev); · 65 unmodified lines
374int amdgpu_gfx_rlc_init_microcode(struct amdgpu_device *adev,380int amdgpu_gfx_rlc_init_microcode(struct amdgpu_device *adev,
375 uint16_t version_major,381 uint16_t version_major,
376 uint16_t version_minor);382 uint16_t version_minor);
383
384void amdgpu_early_init_rlc_reg_funcs(struct amdgpu_device *adev);
385void amdgpu_init_rlc_reg_funcs(struct amdgpu_device *adev);
386
377#endif387#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_sa.h+77−0
@@ -0,0 +1,77 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright 2026 Advanced Micro Devices, Inc.
4 *
5 * Permission is hereby granted, free of charge, to any person obtaining a
6 * copy of this software and associated documentation files (the "Software"),
7 * to deal in the Software without restriction, including without limitation
8 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
9 * and/or sell copies of the Software, and to permit persons to whom the
10 * Software is furnished to do so, subject to the following conditions:
11 *
12 * The above copyright notice and this permission notice shall be included in
13 * all copies or substantial portions of the Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
19 * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
20 * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
21 * OTHER DEALINGS IN THE SOFTWARE.
22 *
23 */
24
25#ifndef AMDGPU_SA_H_
26#define AMDGPU_SA_H_
27
28#include <drm/drm_suballoc.h>
29
30struct amdgpu_device;
31struct amdgpu_bo;
32
33struct amdgpu_sa_manager {
34 struct drm_suballoc_manager base;
35 struct amdgpu_bo *bo;
36 uint64_t gpu_addr;
37 void *cpu_ptr;
38};
39
40static inline struct amdgpu_sa_manager *
41to_amdgpu_sa_manager(struct drm_suballoc_manager *manager)
42{
43 return container_of(manager, struct amdgpu_sa_manager, base);
44}
45
46static inline uint64_t amdgpu_sa_bo_gpu_addr(struct drm_suballoc *sa_bo)
47{
48 return to_amdgpu_sa_manager(sa_bo->manager)->gpu_addr +
49 drm_suballoc_soffset(sa_bo);
50}
51
52static inline void *amdgpu_sa_bo_cpu_addr(struct drm_suballoc *sa_bo)
53{
54 return to_amdgpu_sa_manager(sa_bo->manager)->cpu_ptr +
55 drm_suballoc_soffset(sa_bo);
56}
57
58int amdgpu_sa_bo_manager_init(struct amdgpu_device *adev,
59 struct amdgpu_sa_manager *sa_manager,
60 unsigned size, u32 align, u32 domain);
61void amdgpu_sa_bo_manager_fini(struct amdgpu_device *adev,
62 struct amdgpu_sa_manager *sa_manager);
63int amdgpu_sa_bo_manager_start(struct amdgpu_device *adev,
64 struct amdgpu_sa_manager *sa_manager);
65int amdgpu_sa_bo_new(struct amdgpu_sa_manager *sa_manager,
66 struct drm_suballoc **sa_bo,
67 unsigned int size);
68void amdgpu_sa_bo_free(struct drm_suballoc **sa_bo,
69 struct dma_fence *fence);
70#if defined(CONFIG_DEBUG_FS)
71void amdgpu_sa_bo_dump_debug_info(struct amdgpu_sa_manager *sa_manager,
72 struct seq_file *m);
73u64 amdgpu_bo_print_info(int id, struct amdgpu_bo *bo, struct seq_file *m);
74#endif
75void amdgpu_debugfs_sa_init(struct amdgpu_device *adev);
76
77#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_sched.c+4−4
@@ -39,7 +39,7 @@ static int amdgpu_sched_process_priority_override(struct amdgpu_device *adev, · 38 unmodified lines
39 struct amdgpu_fpriv *fpriv;39 struct amdgpu_fpriv *fpriv;
40 struct amdgpu_ctx_mgr *mgr;40 struct amdgpu_ctx_mgr *mgr;
41 struct amdgpu_ctx *ctx;41 struct amdgpu_ctx *ctx;
42 uint32_t id;42 unsigned long id;
43 int r;43 int r;
44 44
45 if (fd_empty(f))45 if (fd_empty(f))
@@ -50,10 +50,10 @@ static int amdgpu_sched_process_priority_override(struct amdgpu_device *adev, · 4 unmodified lines
50 return r;50 return r;
51 51
52 mgr = &fpriv->ctx_mgr;52 mgr = &fpriv->ctx_mgr;
53 mutex_lock(&mgr->lock);53 xa_lock(&mgr->ctx_handles);
54 idr_for_each_entry(&mgr->ctx_handles, ctx, id)54 xa_for_each(&mgr->ctx_handles, id, ctx)
55 amdgpu_ctx_priority_override(ctx, priority);55 amdgpu_ctx_priority_override(ctx, priority);
56 mutex_unlock(&mgr->lock);56 xa_unlock(&mgr->ctx_handles);
57 57
58 return 0;58 return 0;
59}59}
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.c+15−4
@@ -553,10 +553,11 @@ static int amdgpu_sdma_soft_reset(struct amdgpu_device *adev, u32 instance_id) · 552 unmodified lines
553int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id,553int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id,
554 bool caller_handles_kernel_queues)554 bool caller_handles_kernel_queues)
555{555{
556 int ret = 0;
557 struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];556 struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id];
558 struct amdgpu_ring *gfx_ring = &sdma_instance->ring;557 struct amdgpu_ring *gfx_ring = &sdma_instance->ring;
559 struct amdgpu_ring *page_ring = &sdma_instance->page;558 struct amdgpu_ring *page_ring = &sdma_instance->page;
559 struct amdgpu_fence *gfx_fence, *page_fence;
560 int ret = 0;
560 561
561 if (amdgpu_sriov_vf(adev))562 if (amdgpu_sriov_vf(adev))
562 return -EOPNOTSUPP;563 return -EOPNOTSUPP;
@@ -569,9 +570,14 @@ int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id, · 6 unmodified lines
569 * the reset is in progress.570 * the reset is in progress.
570 */571 */
571 drm_sched_wqueue_stop(&gfx_ring->sched);572 drm_sched_wqueue_stop(&gfx_ring->sched);
573 gfx_fence = amdgpu_ring_find_guilty_fence(gfx_ring);
574 amdgpu_ring_reset_helper_begin(gfx_ring, gfx_fence);
572 575
573 if (adev->sdma.has_page_queue)576 if (adev->sdma.has_page_queue) {
574 drm_sched_wqueue_stop(&page_ring->sched);577 drm_sched_wqueue_stop(&page_ring->sched);
578 page_fence = amdgpu_ring_find_guilty_fence(page_ring);
579 amdgpu_ring_reset_helper_begin(page_ring, page_fence);
580 }
575 }581 }
576 582
577 if (sdma_instance->funcs->stop_kernel_queue) {583 if (sdma_instance->funcs->stop_kernel_queue) {
@@ -600,14 +606,19 @@ exit: · 22 unmodified lines
600 * to be submitted to the queues after the reset is complete.606 * to be submitted to the queues after the reset is complete.
601 */607 */
602 if (!ret) {608 if (!ret) {
603 amdgpu_fence_driver_force_completion(gfx_ring, NULL);609 ret = amdgpu_ring_reset_helper_end(gfx_ring, gfx_fence);
610 if (ret)
611 goto unlock;
604 drm_sched_wqueue_start(&gfx_ring->sched);612 drm_sched_wqueue_start(&gfx_ring->sched);
605 if (adev->sdma.has_page_queue) {613 if (adev->sdma.has_page_queue) {
606 amdgpu_fence_driver_force_completion(page_ring, NULL);614 ret = amdgpu_ring_reset_helper_end(page_ring, page_fence);
615 if (ret)
616 goto unlock;
607 drm_sched_wqueue_start(&page_ring->sched);617 drm_sched_wqueue_start(&page_ring->sched);
608 }618 }
609 }619 }
610 }620 }
621unlock:
611 mutex_unlock(&sdma_instance->engine_reset_mutex);622 mutex_unlock(&sdma_instance->engine_reset_mutex);
612 623
613 return ret;624 return ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_sdma.h+0−28
@@ -85,34 +85,6 @@ struct amdgpu_sdma_instance { · 84 unmodified lines
85 const struct amdgpu_sdma_funcs *funcs;85 const struct amdgpu_sdma_funcs *funcs;
86};86};
87 87
88enum amdgpu_sdma_ras_memory_id {
89 AMDGPU_SDMA_MBANK_DATA_BUF0 = 1,
90 AMDGPU_SDMA_MBANK_DATA_BUF1 = 2,
91 AMDGPU_SDMA_MBANK_DATA_BUF2 = 3,
92 AMDGPU_SDMA_MBANK_DATA_BUF3 = 4,
93 AMDGPU_SDMA_MBANK_DATA_BUF4 = 5,
94 AMDGPU_SDMA_MBANK_DATA_BUF5 = 6,
95 AMDGPU_SDMA_MBANK_DATA_BUF6 = 7,
96 AMDGPU_SDMA_MBANK_DATA_BUF7 = 8,
97 AMDGPU_SDMA_MBANK_DATA_BUF8 = 9,
98 AMDGPU_SDMA_MBANK_DATA_BUF9 = 10,
99 AMDGPU_SDMA_MBANK_DATA_BUF10 = 11,
100 AMDGPU_SDMA_MBANK_DATA_BUF11 = 12,
101 AMDGPU_SDMA_MBANK_DATA_BUF12 = 13,
102 AMDGPU_SDMA_MBANK_DATA_BUF13 = 14,
103 AMDGPU_SDMA_MBANK_DATA_BUF14 = 15,
104 AMDGPU_SDMA_MBANK_DATA_BUF15 = 16,
105 AMDGPU_SDMA_UCODE_BUF = 17,
106 AMDGPU_SDMA_RB_CMD_BUF = 18,
107 AMDGPU_SDMA_IB_CMD_BUF = 19,
108 AMDGPU_SDMA_UTCL1_RD_FIFO = 20,
109 AMDGPU_SDMA_UTCL1_RDBST_FIFO = 21,
110 AMDGPU_SDMA_UTCL1_WR_FIFO = 22,
111 AMDGPU_SDMA_DATA_LUT_FIFO = 23,
112 AMDGPU_SDMA_SPLIT_DAT_BUF = 24,
113 AMDGPU_SDMA_MEMORY_BLOCK_LAST,
114};
115
116struct amdgpu_sdma_ras {88struct amdgpu_sdma_ras {
117 struct amdgpu_ras_block_object ras_block;89 struct amdgpu_ras_block_object ras_block;
118};90};
drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h+150−0
@@ -28,6 +28,8 @@ · 27 unmodified lines
28#include <linux/types.h>28#include <linux/types.h>
29#include <linux/tracepoint.h>29#include <linux/tracepoint.h>
30 30
31#include "amdgpu_userq_fence.h"
32
31#undef TRACE_SYSTEM33#undef TRACE_SYSTEM
32#define TRACE_SYSTEM amdgpu34#define TRACE_SYSTEM amdgpu
33#define TRACE_INCLUDE_FILE amdgpu_trace35#define TRACE_INCLUDE_FILE amdgpu_trace
@@ -582,6 +584,154 @@ TRACE_EVENT(amdgpu_reset_reg_dumps, · 548 unmodified lines
582 __entry->value)584 __entry->value)
583);585);
584 586
587DECLARE_EVENT_CLASS(amdgpu_userq_queue,
588 TP_PROTO(struct amdgpu_usermode_queue *queue),
589 TP_ARGS(queue),
590 TP_STRUCT__entry(
591 __field(void *, queue)
592 __field(u64, doorbell_index)
593 __field(int, queue_type)
594 __field(int, state)
595 __field(u32, xcp_id)
596 ),
597 TP_fast_assign(
598 __entry->queue = queue;
599 __entry->doorbell_index = queue->doorbell_index;
600 __entry->queue_type = queue->queue_type;
601 __entry->state = queue->state;
602 __entry->xcp_id = queue->xcp_id;
603 ),
604 TP_printk("queue=%p, doorbell=%llu, type=%d, state=%d, xcp_id=%u",
605 __entry->queue, __entry->doorbell_index,
606 __entry->queue_type, __entry->state, __entry->xcp_id)
607);
608DEFINE_EVENT(amdgpu_userq_queue, amdgpu_userq_create_start,
609 TP_PROTO(struct amdgpu_usermode_queue *queue),
610 TP_ARGS(queue));
611DEFINE_EVENT(amdgpu_userq_queue, amdgpu_userq_destroy_start,
612 TP_PROTO(struct amdgpu_usermode_queue *queue),
613 TP_ARGS(queue));
614DECLARE_EVENT_CLASS(amdgpu_userq_queue_result,
615 TP_PROTO(struct amdgpu_usermode_queue *queue, int result),
616 TP_ARGS(queue, result),
617 TP_STRUCT__entry(
618 __field(void *, queue)
619 __field(u64, doorbell_index)
620 __field(int, queue_type)
621 __field(int, state)
622 __field(u32, xcp_id)
623 __field(int, result)
624 ),
625 TP_fast_assign(
626 __entry->queue = queue;
627 __entry->doorbell_index = queue->doorbell_index;
628 __entry->queue_type = queue->queue_type;
629 __entry->state = queue->state;
630 __entry->xcp_id = queue->xcp_id;
631 __entry->result = result;
632 ),
633 TP_printk("queue=%p, doorbell=%llu, type=%d, state=%d, xcp_id=%u, result=%d",
634 __entry->queue, __entry->doorbell_index,
635 __entry->queue_type, __entry->state,
636 __entry->xcp_id, __entry->result)
637);
638DEFINE_EVENT(amdgpu_userq_queue_result, amdgpu_userq_create_end,
639 TP_PROTO(struct amdgpu_usermode_queue *queue, int result),
640 TP_ARGS(queue, result));
641DEFINE_EVENT(amdgpu_userq_queue_result, amdgpu_userq_destroy_end,
642 TP_PROTO(struct amdgpu_usermode_queue *queue, int result),
643 TP_ARGS(queue, result));
644
645TRACE_EVENT(amdgpu_userq_emit_fence,
646 TP_PROTO(struct device *device, struct amdgpu_usermode_queue *queue, struct amdgpu_userq_fence *fence),
647 TP_ARGS(device, queue, fence),
648 TP_STRUCT__entry(
649 __field(u64, fence_context)
650 __field(u64, fence_seqno)
651 __string(dev, dev_name(device))
652 __field(u64, doorbell_index)
653 __field(u64, client_id)
654 __field(u32, queue_type)
655 ),
656 TP_fast_assign(
657 __entry->fence_context = fence->base.context;
658 __entry->fence_seqno = fence->base.seqno;
659 __assign_str(dev);
660 __entry->doorbell_index = queue->doorbell_index;
661 __entry->client_id = queue->userq_mgr->file->client_id;
662 __entry->queue_type = queue->queue_type;
663 ),
664 TP_printk("dev=%s, client_id=%llu, type=%u, doorbell=%llu, fence=%llu:%llu",
665 __get_str(dev), __entry->client_id, __entry->queue_type, __entry->doorbell_index,
666 __entry->fence_context,
667 __entry->fence_seqno)
668);
669
670TRACE_EVENT(amdgpu_userq_wait_deps,
671 TP_PROTO(struct device *device, struct amdgpu_usermode_queue *queue, struct amdgpu_userq_fence *dep),
672 TP_ARGS(device, queue, dep),
673 TP_STRUCT__entry(
674 __field(u64, context)
675 __field(u64, dep_context)
676 __field(u64, dep_seqno)
677 __string(dev, dev_name(device))
678 __field(u64, doorbell_index)
679 __field(u64, client_id)
680 __field(u32, queue_type)
681 ),
682 TP_fast_assign(
683 __assign_str(dev);
684 __entry->doorbell_index = queue->doorbell_index;
685 __entry->queue_type = queue->queue_type;
686 __entry->client_id = queue->userq_mgr->file->client_id;
687 __entry->context = queue->fence_drv->context;
688 __entry->dep_context = dep->base.context;
689 __entry->dep_seqno = dep->base.seqno;
690 ),
691 TP_printk("dev=%s, client_id=%llu, type=%u, doorbell=%llu, context=%llu depends on fence=%llu:%llu",
692 __get_str(dev), __entry->client_id, __entry->queue_type, __entry->doorbell_index, __entry->context,
693 __entry->dep_context,
694 __entry->dep_seqno)
695);
696
697TRACE_EVENT(amdgpu_userq_state_start,
698 TP_PROTO(struct amdgpu_usermode_queue *queue),
699 TP_ARGS(queue),
700 TP_STRUCT__entry(
701 __field(u64, doorbell_index)
702 __field(u64, client_id)
703 __field(u32, queue_type)
704 __field(u32, from)
705 ),
706 TP_fast_assign(
707 __entry->doorbell_index = queue->doorbell_index;
708 __entry->queue_type = queue->queue_type;
709 __entry->client_id = queue->userq_mgr->file->client_id;
710 __entry->from = queue->state;
711 ),
712 TP_printk("client_id=%llu, type=%u, doorbell=%llu, from=%d",
713 __entry->client_id, __entry->queue_type, __entry->doorbell_index, __entry->from)
714);
715
716TRACE_EVENT(amdgpu_userq_state_changed,
717 TP_PROTO(struct amdgpu_usermode_queue *queue, enum amdgpu_userq_state new_state),
718 TP_ARGS(queue, new_state),
719 TP_STRUCT__entry(
720 __field(u64, doorbell_index)
721 __field(u64, client_id)
722 __field(u32, queue_type)
723 __field(u32, to)
724 ),
725 TP_fast_assign(
726 __entry->doorbell_index = queue->doorbell_index;
727 __entry->queue_type = queue->queue_type;
728 __entry->client_id = queue->userq_mgr->file->client_id;
729 __entry->to = new_state;
730 ),
731 TP_printk("client_id=%llu, type=%u, doorbell=%llu, to=%d",
732 __entry->client_id, __entry->queue_type, __entry->doorbell_index, __entry->to)
733);
734
585#undef AMDGPU_JOB_GET_TIMELINE_NAME735#undef AMDGPU_JOB_GET_TIMELINE_NAME
586#endif736#endif
587 737
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c+1−1
@@ -2194,7 +2194,7 @@ int amdgpu_ttm_init(struct amdgpu_device *adev) · 2193 unmodified lines
2194 return r;2194 return r;
2195 }2195 }
2196 2196
2197 /* Create a boorbell page for kernel usages */2197 /* Create a doorbell page for kernel usages */
2198 r = amdgpu_doorbell_create_kernel_doorbells(adev);2198 r = amdgpu_doorbell_create_kernel_doorbells(adev);
2199 if (r) {2199 if (r) {
2200 dev_err(adev->dev, "Failed to initialize kernel doorbells.\n");2200 dev_err(adev->dev, "Failed to initialize kernel doorbells.\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.h+1−0
@@ -140,6 +140,7 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size); · 139 unmodified lines
140void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev);140void amdgpu_gtt_mgr_fini(struct amdgpu_device *adev);
141int amdgpu_preempt_mgr_init(struct amdgpu_device *adev);141int amdgpu_preempt_mgr_init(struct amdgpu_device *adev);
142void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev);142void amdgpu_preempt_mgr_fini(struct amdgpu_device *adev);
143void amdgpu_preempt_mgr_sysfs_fini(struct amdgpu_device *adev);
143int amdgpu_vram_mgr_init(struct amdgpu_device *adev);144int amdgpu_vram_mgr_init(struct amdgpu_device *adev);
144void amdgpu_vram_mgr_fini(struct amdgpu_device *adev);145void amdgpu_vram_mgr_fini(struct amdgpu_device *adev);
145 146
drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c+60−209
@@ -97,7 +97,6 @@ void amdgpu_umc_handle_bad_pages(struct amdgpu_device *adev, · 96 unmodified lines
97{97{
98 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;98 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;
99 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);99 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
100 struct amdgpu_ras_eeprom_control *control = &con->eeprom_control;
101 unsigned int error_query_mode;100 unsigned int error_query_mode;
102 int ret = 0;101 int ret = 0;
103 unsigned long err_count;102 unsigned long err_count;
@@ -118,77 +117,66 @@ void amdgpu_umc_handle_bad_pages(struct amdgpu_device *adev, · 14 unmodified lines
118 err_data->err_addr_len = adev->umc.max_ras_err_cnt_per_query;117 err_data->err_addr_len = adev->umc.max_ras_err_cnt_per_query;
119 118
120 mutex_lock(&con->page_retirement_lock);119 mutex_lock(&con->page_retirement_lock);
121 if (!amdgpu_ras_smu_eeprom_supported(adev)) {120 ret = amdgpu_dpm_get_ecc_info(adev, (void *)&(con->umc_ecc));
122 ret = amdgpu_dpm_get_ecc_info(adev, (void *)&(con->umc_ecc));121 if (ret == -EOPNOTSUPP &&
123 if (ret == -EOPNOTSUPP &&122 error_query_mode == AMDGPU_RAS_DIRECT_ERROR_QUERY) {
124 error_query_mode == AMDGPU_RAS_DIRECT_ERROR_QUERY) {123 if (adev->umc.ras && adev->umc.ras->ras_block.hw_ops &&
125 if (adev->umc.ras && adev->umc.ras->ras_block.hw_ops &&124 adev->umc.ras->ras_block.hw_ops->query_ras_error_count)
126 adev->umc.ras->ras_block.hw_ops->query_ras_error_count)125 adev->umc.ras->ras_block.hw_ops->query_ras_error_count(adev,
127 adev->umc.ras->ras_block.hw_ops->query_ras_error_count(adev,126 ras_error_status);
128 ras_error_status);127
129 128 if (adev->umc.ras && adev->umc.ras->ras_block.hw_ops &&
130 if (adev->umc.ras && adev->umc.ras->ras_block.hw_ops &&129 adev->umc.ras->ras_block.hw_ops->query_ras_error_address &&
131 adev->umc.ras->ras_block.hw_ops->query_ras_error_address &&130 adev->umc.max_ras_err_cnt_per_query) {
132 adev->umc.max_ras_err_cnt_per_query) {131 err_data->err_addr =
133 kfree(err_data->err_addr);132 kzalloc_objs(struct eeprom_table_record,
134 err_data->err_addr =133 adev->umc.max_ras_err_cnt_per_query);
135 kzalloc_objs(struct eeprom_table_record,134
136 adev->umc.max_ras_err_cnt_per_query);135 /* still call query_ras_error_address to clear error status
137 136 * even NOMEM error is encountered
138 /* still call query_ras_error_address to clear error status137 */
139 * even NOMEM error is encountered138 if (!err_data->err_addr)
140 */139 dev_warn(adev->dev,
141 if (!err_data->err_addr)140 "Failed to alloc memory for umc error address record!\n");
142 dev_warn(adev->dev,141 else
143 "Failed to alloc memory for umc error address record!\n");142 err_data->err_addr_len =
144 else143 adev->umc.max_ras_err_cnt_per_query;
145 err_data->err_addr_len =144
146 adev->umc.max_ras_err_cnt_per_query;145 /* umc query_ras_error_address is also responsible for clearing
147 146 * error status
148 /* umc query_ras_error_address is also responsible for clearing147 */
149 * error status148 adev->umc.ras->ras_block.hw_ops->query_ras_error_address(adev,
150 */149 ras_error_status);
151 adev->umc.ras->ras_block.hw_ops->query_ras_error_address(adev,
152 ras_error_status);
153 }
154 } else if (error_query_mode == AMDGPU_RAS_FIRMWARE_ERROR_QUERY ||
155 (!ret && error_query_mode == AMDGPU_RAS_DIRECT_ERROR_QUERY)) {
156 if (adev->umc.ras &&
157 adev->umc.ras->ecc_info_query_ras_error_count)
158 adev->umc.ras->ecc_info_query_ras_error_count(adev,
159 ras_error_status);
160
161 if (adev->umc.ras &&
162 adev->umc.ras->ecc_info_query_ras_error_address &&
163 adev->umc.max_ras_err_cnt_per_query) {
164 kfree(err_data->err_addr);
165 err_data->err_addr =
166 kzalloc_objs(struct eeprom_table_record,
167 adev->umc.max_ras_err_cnt_per_query);
168
169 /* still call query_ras_error_address to clear error status
170 * even NOMEM error is encountered
171 */
172 if (!err_data->err_addr)
173 dev_warn(adev->dev,
174 "Failed to alloc memory for umc error address record!\n");
175 else
176 err_data->err_addr_len =
177 adev->umc.max_ras_err_cnt_per_query;
178
179 /* umc query_ras_error_address is also responsible for clearing
180 * error status
181 */
182 adev->umc.ras->ecc_info_query_ras_error_address(adev,
183 ras_error_status);
184 }
185 }150 }
186 } else {151 } else if (error_query_mode == AMDGPU_RAS_FIRMWARE_ERROR_QUERY ||
187 if (!amdgpu_ras_eeprom_update_record_num(control)) {152 (!ret && error_query_mode == AMDGPU_RAS_DIRECT_ERROR_QUERY)) {
188 err_data->err_addr_cnt = err_data->de_count =153 if (adev->umc.ras &&
189 control->ras_num_recs - control->ras_num_recs_old;154 adev->umc.ras->ecc_info_query_ras_error_count)
190 amdgpu_ras_eeprom_read_idx(control, err_data->err_addr,155 adev->umc.ras->ecc_info_query_ras_error_count(adev,
191 control->ras_num_recs_old, err_data->de_count);156 ras_error_status);
157
158 if (adev->umc.ras &&
159 adev->umc.ras->ecc_info_query_ras_error_address &&
160 adev->umc.max_ras_err_cnt_per_query) {
161 err_data->err_addr =
162 kcalloc(adev->umc.max_ras_err_cnt_per_query,
163 sizeof(struct eeprom_table_record), GFP_KERNEL);
164
165 /* still call query_ras_error_address to clear error status
166 * even NOMEM error is encountered
167 */
168 if (!err_data->err_addr)
169 dev_warn(adev->dev,
170 "Failed to alloc memory for umc error address record!\n");
171 else
172 err_data->err_addr_len =
173 adev->umc.max_ras_err_cnt_per_query;
174
175 /* umc query_ras_error_address is also responsible for clearing
176 * error status
177 */
178 adev->umc.ras->ecc_info_query_ras_error_address(adev,
179 ras_error_status);
192 }180 }
193 }181 }
194 182
@@ -276,7 +264,7 @@ int amdgpu_umc_pasid_poison_handler(struct amdgpu_device *adev, · 81 unmodified lines
276 }264 }
277 265
278 amdgpu_ras_error_data_fini(&err_data);266 amdgpu_ras_error_data_fini(&err_data);
279 } else if (amdgpu_uniras_enabled(adev)) {267 } else {
280 struct ras_ih_info ih_info = {0};268 struct ras_ih_info ih_info = {0};
281 269
282 ih_info.block = block;270 ih_info.block = block;
@@ -285,17 +273,6 @@ int amdgpu_umc_pasid_poison_handler(struct amdgpu_device *adev, · 2 unmodified lines
285 ih_info.pasid_fn = pasid_fn;273 ih_info.pasid_fn = pasid_fn;
286 ih_info.data = data;274 ih_info.data = data;
287 amdgpu_ras_mgr_handle_consumer_interrupt(adev, &ih_info);275 amdgpu_ras_mgr_handle_consumer_interrupt(adev, &ih_info);
288 } else {
289 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
290 int ret;
291
292 ret = amdgpu_ras_put_poison_req(adev,
293 block, pasid, pasid_fn, data, reset);
294 if (!ret) {
295 atomic_inc(&con->page_retirement_req_cnt);
296 atomic_inc(&con->poison_consumption_count);
297 wake_up(&con->page_retirement_wq);
298 }
299 }276 }
300 } else {277 } else {
301 if (adev->virt.ops && adev->virt.ops->ras_poison_handler)278 if (adev->virt.ops && adev->virt.ops->ras_poison_handler)
@@ -512,129 +489,3 @@ int amdgpu_umc_loop_channels(struct amdgpu_device *adev, · 210 unmodified lines
512 489
513 return 0;490 return 0;
514}491}
515
516int amdgpu_umc_update_ecc_status(struct amdgpu_device *adev,
517 uint64_t status, uint64_t ipid, uint64_t addr)
518{
519 if (adev->umc.ras->update_ecc_status)
520 return adev->umc.ras->update_ecc_status(adev,
521 status, ipid, addr);
522 return 0;
523}
524
525int amdgpu_umc_logs_ecc_err(struct amdgpu_device *adev,
526 struct radix_tree_root *ecc_tree, struct ras_ecc_err *ecc_err)
527{
528 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
529 struct ras_ecc_log_info *ecc_log;
530 int ret;
531
532 ecc_log = &con->umc_ecc_log;
533
534 mutex_lock(&ecc_log->lock);
535 ret = radix_tree_insert(ecc_tree, ecc_err->pa_pfn, ecc_err);
536 if (!ret)
537 radix_tree_tag_set(ecc_tree,
538 ecc_err->pa_pfn, UMC_ECC_NEW_DETECTED_TAG);
539 mutex_unlock(&ecc_log->lock);
540
541 return ret;
542}
543
544int amdgpu_umc_pages_in_a_row(struct amdgpu_device *adev,
545 struct ras_err_data *err_data, uint64_t pa_addr)
546{
547 struct ta_ras_query_address_output addr_out;
548
549 /* reinit err_data */
550 err_data->err_addr_cnt = 0;
551 err_data->err_addr_len = adev->umc.retire_unit;
552
553 addr_out.pa.pa = pa_addr;
554 if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr)
555 return adev->umc.ras->convert_ras_err_addr(adev, err_data, NULL,
556 &addr_out, false);
557 else
558 return -EINVAL;
559}
560
561int amdgpu_umc_lookup_bad_pages_in_a_row(struct amdgpu_device *adev,
562 uint64_t pa_addr, uint64_t *pfns, int len)
563{
564 int i, ret;
565 struct ras_err_data err_data;
566
567 err_data.err_addr = kzalloc_objs(struct eeprom_table_record,
568 adev->umc.retire_unit);
569 if (!err_data.err_addr) {
570 dev_warn(adev->dev, "Failed to alloc memory in bad page lookup!\n");
571 return 0;
572 }
573
574 ret = amdgpu_umc_pages_in_a_row(adev, &err_data, pa_addr);
575 if (ret)
576 goto out;
577
578 for (i = 0; i < adev->umc.retire_unit; i++) {
579 if (i >= len)
580 goto out;
581
582 pfns[i] = err_data.err_addr[i].retired_page;
583 }
584 ret = i;
585 adev->umc.err_addr_cnt = err_data.err_addr_cnt;
586
587out:
588 kfree(err_data.err_addr);
589 return ret;
590}
591
592int amdgpu_umc_mca_to_addr(struct amdgpu_device *adev,
593 uint64_t err_addr, uint32_t ch, uint32_t umc,
594 uint32_t node, uint32_t socket,
595 struct ta_ras_query_address_output *addr_out, bool dump_addr)
596{
597 struct ta_ras_query_address_input addr_in;
598 int ret;
599
600 memset(&addr_in, 0, sizeof(addr_in));
601 addr_in.ma.err_addr = err_addr;
602 addr_in.ma.ch_inst = ch;
603 addr_in.ma.umc_inst = umc;
604 addr_in.ma.node_inst = node;
605 addr_in.ma.socket_id = socket;
606
607 if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr) {
608 ret = adev->umc.ras->convert_ras_err_addr(adev, NULL, &addr_in,
609 addr_out, dump_addr);
610 if (ret)
611 return ret;
612 } else {
613 return 0;
614 }
615
616 return 0;
617}
618
619int amdgpu_umc_pa2mca(struct amdgpu_device *adev,
620 uint64_t pa, uint64_t *mca, enum amdgpu_memory_partition nps)
621{
622 struct ta_ras_query_address_input addr_in;
623 struct ta_ras_query_address_output addr_out;
624 int ret;
625
626 /* nps: the pa belongs to */
627 addr_in.pa.pa = pa | ((uint64_t)nps << 58);
628 addr_in.addr_type = TA_RAS_PA_TO_MCA;
629 ret = psp_ras_query_address(&adev->psp, &addr_in, &addr_out);
630 if (ret) {
631 dev_warn(adev->dev, "Failed to query RAS MCA address for 0x%llx",
632 pa);
633
634 return ret;
635 }
636
637 *mca = addr_out.ma.err_addr;
638
639 return 0;
640}
drivers/gpu/drm/amd/amdgpu/amdgpu_umc.h+0−26
@@ -103,18 +103,7 @@ struct amdgpu_umc_ras { · 102 unmodified lines
103 void *ras_error_status);103 void *ras_error_status);
104 bool (*check_ecc_err_status)(struct amdgpu_device *adev,104 bool (*check_ecc_err_status)(struct amdgpu_device *adev,
105 enum amdgpu_mca_error_type type, void *ras_error_status);105 enum amdgpu_mca_error_type type, void *ras_error_status);
106 int (*update_ecc_status)(struct amdgpu_device *adev,
107 uint64_t status, uint64_t ipid, uint64_t addr);
108 int (*convert_ras_err_addr)(struct amdgpu_device *adev,
109 struct ras_err_data *err_data,
110 struct ta_ras_query_address_input *addr_in,
111 struct ta_ras_query_address_output *addr_out,
112 bool dump_addr);
113 uint32_t (*get_die_id_from_pa)(struct amdgpu_device *adev,
114 uint64_t mca_addr, uint64_t retired_page);
115 void (*get_retire_flip_bits)(struct amdgpu_device *adev);106 void (*get_retire_flip_bits)(struct amdgpu_device *adev);
116 void (*mca_ipid_parse)(struct amdgpu_device *adev, uint64_t ipid,
117 uint32_t *did, uint32_t *ch, uint32_t *umc_inst, uint32_t *sid);
118};107};
119 108
120struct amdgpu_umc_funcs {109struct amdgpu_umc_funcs {
@@ -179,21 +168,6 @@ int amdgpu_umc_page_retirement_mca(struct amdgpu_device *adev, · 58 unmodified lines
179int amdgpu_umc_loop_channels(struct amdgpu_device *adev,168int amdgpu_umc_loop_channels(struct amdgpu_device *adev,
180 umc_func func, void *data);169 umc_func func, void *data);
181 170
182int amdgpu_umc_update_ecc_status(struct amdgpu_device *adev,
183 uint64_t status, uint64_t ipid, uint64_t addr);
184int amdgpu_umc_logs_ecc_err(struct amdgpu_device *adev,
185 struct radix_tree_root *ecc_tree, struct ras_ecc_err *ecc_err);
186
187void amdgpu_umc_handle_bad_pages(struct amdgpu_device *adev,171void amdgpu_umc_handle_bad_pages(struct amdgpu_device *adev,
188 void *ras_error_status);172 void *ras_error_status);
189int amdgpu_umc_pages_in_a_row(struct amdgpu_device *adev,
190 struct ras_err_data *err_data, uint64_t pa_addr);
191int amdgpu_umc_lookup_bad_pages_in_a_row(struct amdgpu_device *adev,
192 uint64_t pa_addr, uint64_t *pfns, int len);
193int amdgpu_umc_mca_to_addr(struct amdgpu_device *adev,
194 uint64_t err_addr, uint32_t ch, uint32_t umc,
195 uint32_t node, uint32_t socket,
196 struct ta_ras_query_address_output *addr_out, bool dump_addr);
197int amdgpu_umc_pa2mca(struct amdgpu_device *adev,
198 uint64_t pa, uint64_t *mca, enum amdgpu_memory_partition nps);
199#endif173#endif
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c+108−57
@@ -33,6 +33,7 @@ · 32 unmodified lines
33#include "amdgpu_userq.h"33#include "amdgpu_userq.h"
34#include "amdgpu_hmm.h"34#include "amdgpu_hmm.h"
35#include "amdgpu_userq_fence.h"35#include "amdgpu_userq_fence.h"
36#include "amdgpu_trace.h"
36 37
37u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev)38u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev)
38{39{
@@ -88,14 +89,7 @@ static void amdgpu_userq_mgr_reset_work(struct work_struct *work) · 49 unmodified lines
88 container_of(work, struct amdgpu_userq_mgr,89 container_of(work, struct amdgpu_userq_mgr,
89 reset_work);90 reset_work);
90 struct amdgpu_device *adev = uq_mgr->adev;91 struct amdgpu_device *adev = uq_mgr->adev;
91 const int queue_types[] = {92 struct amdgpu_reset_context reset_context;
92 AMDGPU_RING_TYPE_COMPUTE,
93 AMDGPU_RING_TYPE_GFX,
94 AMDGPU_RING_TYPE_SDMA
95 };
96 const int num_queue_types = ARRAY_SIZE(queue_types);
97 bool gpu_reset = false;
98 int i, r;
99 93
100 if (unlikely(adev->debug_disable_gpu_ring_reset)) {94 if (unlikely(adev->debug_disable_gpu_ring_reset)) {
101 dev_err(adev->dev, "userq reset disabled by debug mask\n");95 dev_err(adev->dev, "userq reset disabled by debug mask\n");
@@ -109,42 +103,15 @@ static void amdgpu_userq_mgr_reset_work(struct work_struct *work) · 7 unmodified lines
109 if (!amdgpu_gpu_recovery)103 if (!amdgpu_gpu_recovery)
110 return;104 return;
111 105
112 /*106 memset(&reset_context, 0, sizeof(reset_context));
113 * Iterate through all queue types to detect and reset problematic queues
114 * Process each queue type in the defined order
115 */
116 for (i = 0; i < num_queue_types; i++) {
117 int ring_type = queue_types[i];
118 const struct amdgpu_userq_funcs *funcs =
119 adev->userq_funcs[ring_type];
120
121 if (!amdgpu_userq_is_reset_type_supported(adev, ring_type,
122 AMDGPU_RESET_TYPE_PER_QUEUE))
123 continue;
124 107
125 if (atomic_read(&uq_mgr->userq_count[ring_type]) > 0 &&108 reset_context.method = AMD_RESET_METHOD_NONE;
126 funcs && funcs->detect_and_reset) {109 reset_context.reset_req_dev = adev;
127 r = funcs->detect_and_reset(adev, ring_type);110 reset_context.src = AMDGPU_RESET_SRC_USERQ;
128 if (r) {111 set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);
129 gpu_reset = true;112 /*set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags);*/
130 break;
131 }
132 }
133 }
134 113
135 if (gpu_reset) {114 amdgpu_device_gpu_recover(adev, NULL, &reset_context);
136 struct amdgpu_reset_context reset_context;
137
138 memset(&reset_context, 0, sizeof(reset_context));
139
140 reset_context.method = AMD_RESET_METHOD_NONE;
141 reset_context.reset_req_dev = adev;
142 reset_context.src = AMDGPU_RESET_SRC_USERQ;
143 set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags);
144 /*set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags);*/
145
146 amdgpu_device_gpu_recover(adev, NULL, &reset_context);
147 }
148}115}
149 116
150static void amdgpu_userq_hang_detect_work(struct work_struct *work)117static void amdgpu_userq_hang_detect_work(struct work_struct *work)
@@ -152,12 +119,45 @@ static void amdgpu_userq_hang_detect_work(struct work_struct *work) · 1 unmodified line
152 struct amdgpu_usermode_queue *queue =119 struct amdgpu_usermode_queue *queue =
153 container_of(work, struct amdgpu_usermode_queue,120 container_of(work, struct amdgpu_usermode_queue,
154 hang_detect_work.work);121 hang_detect_work.work);
122 struct amdgpu_userq_mgr *uq_mgr = queue->userq_mgr;
123 struct amdgpu_device *adev = uq_mgr->adev;
124 const struct amdgpu_userq_funcs *userq_funcs =
125 adev->userq_funcs[queue->queue_type];
126 bool gpu_reset = false;
127
128 if (unlikely(adev->debug_disable_gpu_ring_reset)) {
129 dev_err(adev->dev, "userq reset disabled by debug mask\n");
130 return;
131 }
132
133 /*
134 * If GPU recovery feature is disabled system-wide,
135 * skip all reset detection logic
136 */
137 if (!amdgpu_gpu_recovery)
138 return;
139
140 if (amdgpu_userq_is_reset_type_supported(adev, queue->queue_type,
141 AMDGPU_RESET_TYPE_PER_QUEUE)) {
142 int r;
143
144 if (queue->queue_type == AMDGPU_HW_IP_COMPUTE)
145 r = amdgpu_gfx_reset_mes_compute(adev, NULL, NULL,
146 queue, NULL, NULL);
147 else
148 r = userq_funcs->reset(queue);
149 if (r)
150 gpu_reset = true;
151 } else {
152 gpu_reset = true;
153 }
155 154
156 /*155 /*
157 * Don't schedule the work here! Scheduling or queue work from one reset156 * Don't schedule the work here! Scheduling or queue work from one reset
158 * handler to another is illegal if you don't take extra precautions!157 * handler to another is illegal if you don't take extra precautions!
159 */158 */
160 amdgpu_userq_mgr_reset_work(&queue->userq_mgr->reset_work);159 if (gpu_reset)
160 amdgpu_userq_mgr_reset_work(&queue->userq_mgr->reset_work);
161}161}
162 162
163/*163/*
@@ -293,11 +293,15 @@ static int amdgpu_userq_preempt_helper(struct amdgpu_usermode_queue *queue) · 129 unmodified lines
293 int r;293 int r;
294 294
295 if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {295 if (queue->state == AMDGPU_USERQ_STATE_MAPPED) {
296 trace_amdgpu_userq_state_start(queue);
297
296 r = userq_funcs->preempt(queue);298 r = userq_funcs->preempt(queue);
297 if (r) {299 if (r) {
300 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
298 queue->state = AMDGPU_USERQ_STATE_HUNG;301 queue->state = AMDGPU_USERQ_STATE_HUNG;
299 return r;302 return r;
300 } else {303 } else {
304 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_PREEMPTED);
301 queue->state = AMDGPU_USERQ_STATE_PREEMPTED;305 queue->state = AMDGPU_USERQ_STATE_PREEMPTED;
302 }306 }
303 }307 }
@@ -313,10 +317,14 @@ static int amdgpu_userq_restore_helper(struct amdgpu_usermode_queue *queue) · 9 unmodified lines
313 int r = 0;317 int r = 0;
314 318
315 if (queue->state == AMDGPU_USERQ_STATE_PREEMPTED) {319 if (queue->state == AMDGPU_USERQ_STATE_PREEMPTED) {
320 trace_amdgpu_userq_state_start(queue);
321
316 r = userq_funcs->restore(queue);322 r = userq_funcs->restore(queue);
317 if (r) {323 if (r) {
324 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
318 queue->state = AMDGPU_USERQ_STATE_HUNG;325 queue->state = AMDGPU_USERQ_STATE_HUNG;
319 } else {326 } else {
327 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_MAPPED);
320 queue->state = AMDGPU_USERQ_STATE_MAPPED;328 queue->state = AMDGPU_USERQ_STATE_MAPPED;
321 }329 }
322 }330 }
@@ -334,12 +342,15 @@ static int amdgpu_userq_unmap_helper(struct amdgpu_usermode_queue *queue) · 11 unmodified lines
334 342
335 if ((queue->state == AMDGPU_USERQ_STATE_MAPPED) ||343 if ((queue->state == AMDGPU_USERQ_STATE_MAPPED) ||
336 (queue->state == AMDGPU_USERQ_STATE_PREEMPTED)) {344 (queue->state == AMDGPU_USERQ_STATE_PREEMPTED)) {
345 trace_amdgpu_userq_state_start(queue);
337 346
338 r = userq_funcs->unmap(queue);347 r = userq_funcs->unmap(queue);
339 if (r) {348 if (r) {
349 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
340 queue->state = AMDGPU_USERQ_STATE_HUNG;350 queue->state = AMDGPU_USERQ_STATE_HUNG;
341 return r;351 return r;
342 } else {352 } else {
353 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_UNMAPPED);
343 queue->state = AMDGPU_USERQ_STATE_UNMAPPED;354 queue->state = AMDGPU_USERQ_STATE_UNMAPPED;
344 }355 }
345 }356 }
@@ -356,11 +367,15 @@ static int amdgpu_userq_map_helper(struct amdgpu_usermode_queue *queue) · 10 unmodified lines
356 int r;367 int r;
357 368
358 if (queue->state == AMDGPU_USERQ_STATE_UNMAPPED) {369 if (queue->state == AMDGPU_USERQ_STATE_UNMAPPED) {
370 trace_amdgpu_userq_state_start(queue);
371
359 r = userq_funcs->map(queue);372 r = userq_funcs->map(queue);
360 if (r) {373 if (r) {
374 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
361 queue->state = AMDGPU_USERQ_STATE_HUNG;375 queue->state = AMDGPU_USERQ_STATE_HUNG;
362 return r;376 return r;
363 } else {377 } else {
378 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_MAPPED);
364 queue->state = AMDGPU_USERQ_STATE_MAPPED;379 queue->state = AMDGPU_USERQ_STATE_MAPPED;
365 }380 }
366 }381 }
@@ -507,6 +522,8 @@ amdgpu_userq_destroy(struct amdgpu_userq_mgr *uq_mgr, struct amdgpu_usermode_que · 140 unmodified lines
507 const struct amdgpu_userq_funcs *uq_funcs = adev->userq_funcs[queue->queue_type];522 const struct amdgpu_userq_funcs *uq_funcs = adev->userq_funcs[queue->queue_type];
508 int r = 0;523 int r = 0;
509 524
525 trace_amdgpu_userq_destroy_start(queue);
526
510 cancel_delayed_work_sync(&uq_mgr->resume_work);527 cancel_delayed_work_sync(&uq_mgr->resume_work);
511 528
512 /* Cancel any pending hang detection work and cleanup */529 /* Cancel any pending hang detection work and cleanup */
@@ -532,6 +549,7 @@ amdgpu_userq_destroy(struct amdgpu_userq_mgr *uq_mgr, struct amdgpu_usermode_que · 19 unmodified lines
532 amdgpu_bo_unreserve(queue->db_obj.obj);549 amdgpu_bo_unreserve(queue->db_obj.obj);
533 amdgpu_bo_unref(&queue->db_obj.obj);550 amdgpu_bo_unref(&queue->db_obj.obj);
534 551
552 trace_amdgpu_userq_destroy_end(queue, r);
535 kfree(queue);553 kfree(queue);
536 554
537 pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);555 pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -629,6 +647,8 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args) · 91 unmodified lines
629 queue->queue_type = args->in.ip_type;647 queue->queue_type = args->in.ip_type;
630 queue->vm = &fpriv->vm;648 queue->vm = &fpriv->vm;
631 queue->priority = priority;649 queue->priority = priority;
650 queue->xcp_id = (fpriv->xcp_id != AMDGPU_XCP_NO_PARTITION) ?
651 fpriv->xcp_id : 0;
632 queue->userq_mgr = uq_mgr;652 queue->userq_mgr = uq_mgr;
633 INIT_DELAYED_WORK(&queue->hang_detect_work,653 INIT_DELAYED_WORK(&queue->hang_detect_work,
634 amdgpu_userq_hang_detect_work);654 amdgpu_userq_hang_detect_work);
@@ -671,6 +691,8 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args) · 36 unmodified lines
671 }691 }
672 692
673 queue->doorbell_index = index;693 queue->doorbell_index = index;
694 queue->doorbell_offset = (u32)args->in.doorbell_offset;
695 trace_amdgpu_userq_create_start(queue);
674 r = uq_funcs->mqd_create(queue, &args->in);696 r = uq_funcs->mqd_create(queue, &args->in);
675 if (r) {697 if (r) {
676 drm_file_err(uq_mgr->file, "Failed to create Queue\n");698 drm_file_err(uq_mgr->file, "Failed to create Queue\n");
@@ -694,6 +716,7 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args) · 17 unmodified lines
694 r = amdgpu_userq_map_helper(queue);716 r = amdgpu_userq_map_helper(queue);
695 if (r) {717 if (r) {
696 drm_file_err(uq_mgr->file, "Failed to map Queue\n");718 drm_file_err(uq_mgr->file, "Failed to map Queue\n");
719 trace_amdgpu_userq_create_end(queue, r);
697 mutex_unlock(&uq_mgr->userq_mutex);720 mutex_unlock(&uq_mgr->userq_mutex);
698 goto erase_doorbell;721 goto erase_doorbell;
699 }722 }
@@ -710,11 +733,13 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args) · 10 unmodified lines
710 * This drops the last reference which should take care of733 * This drops the last reference which should take care of
711 * all cleanup.734 * all cleanup.
712 */735 */
736 trace_amdgpu_userq_create_end(queue, r);
713 amdgpu_userq_put(queue);737 amdgpu_userq_put(queue);
714 return r;738 return r;
715 }739 }
716 740
717 amdgpu_debugfs_userq_init(filp, queue, qid);741 amdgpu_debugfs_userq_init(filp, queue, qid);
742 trace_amdgpu_userq_create_end(queue, 0);
718 args->out.queue_id = qid;743 args->out.queue_id = qid;
719 return 0;744 return 0;
720 745
@@ -730,6 +755,7 @@ clean_doorbell_bo: · 9 unmodified lines
730free_fence_drv:755free_fence_drv:
731 amdgpu_userq_fence_driver_free(queue);756 amdgpu_userq_fence_driver_free(queue);
732free_queue:757free_queue:
758 trace_amdgpu_userq_create_end(queue, r);
733 kfree(queue);759 kfree(queue);
734err_pm_runtime:760err_pm_runtime:
735 pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);761 pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -862,16 +888,10 @@ int amdgpu_userq_ioctl(struct drm_device *dev, void *data, · 126 unmodified lines
862static int888static int
863amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)889amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
864{890{
865 struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
866 struct amdgpu_vm *vm = &fpriv->vm;
867 struct amdgpu_usermode_queue *queue;891 struct amdgpu_usermode_queue *queue;
868 unsigned long queue_id;892 unsigned long queue_id;
869 int ret = 0, r;893 int ret = 0, r;
870 894
871
872 if (amdgpu_bo_reserve(vm->root.bo, false))
873 return false;
874
875 mutex_lock(&uq_mgr->userq_mutex);895 mutex_lock(&uq_mgr->userq_mutex);
876 /* Resume all the queues for this process */896 /* Resume all the queues for this process */
877 xa_for_each(&uq_mgr->userq_xa, queue_id, queue) {897 xa_for_each(&uq_mgr->userq_xa, queue_id, queue) {
@@ -879,6 +899,7 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · 1 unmodified line
879 if (!amdgpu_userq_buffer_vas_mapped(queue)) {899 if (!amdgpu_userq_buffer_vas_mapped(queue)) {
880 drm_file_err(uq_mgr->file,900 drm_file_err(uq_mgr->file,
881 "trying restore queue without va mapping\n");901 "trying restore queue without va mapping\n");
902 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_INVALID_VA);
882 queue->state = AMDGPU_USERQ_STATE_INVALID_VA;903 queue->state = AMDGPU_USERQ_STATE_INVALID_VA;
883 continue;904 continue;
884 }905 }
@@ -886,10 +907,8 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · 1 unmodified line
886 r = amdgpu_userq_map_helper(queue);907 r = amdgpu_userq_map_helper(queue);
887 if (r)908 if (r)
888 ret = r;909 ret = r;
889
890 }910 }
891 mutex_unlock(&uq_mgr->userq_mutex);911 mutex_unlock(&uq_mgr->userq_mutex);
892 amdgpu_bo_unreserve(vm->root.bo);
893 912
894 if (ret)913 if (ret)
895 drm_file_err(uq_mgr->file,914 drm_file_err(uq_mgr->file,
@@ -923,7 +942,8 @@ amdgpu_userq_bo_validate(struct amdgpu_device *adev, struct drm_exec *exec, · 27 unmodified lines
923 spin_unlock(&vm->individual_lock);942 spin_unlock(&vm->individual_lock);
924 943
925 bo = bo_va->base.bo;944 bo = bo_va->base.bo;
926 ret = drm_exec_prepare_obj(exec, &bo->tbo.base, 2);945 ret = drm_exec_prepare_obj(exec, &bo->tbo.base,
946 TTM_NUM_MOVE_FENCES + 1);
927 if (unlikely(ret))947 if (unlikely(ret))
928 return ret;948 return ret;
929 949
@@ -946,7 +966,7 @@ amdgpu_userq_bo_validate(struct amdgpu_device *adev, struct drm_exec *exec, · 16 unmodified lines
946 966
947/* Make sure the whole VM is ready to be used */967/* Make sure the whole VM is ready to be used */
948static int968static int
949amdgpu_userq_vm_validate(struct amdgpu_userq_mgr *uq_mgr)969amdgpu_userq_vm_validate_and_restore_queue(struct amdgpu_userq_mgr *uq_mgr)
950{970{
951 struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);971 struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr);
952 bool invalidated = false, new_addition = false;972 bool invalidated = false, new_addition = false;
@@ -1072,8 +1092,12 @@ retry_lock: · 119 unmodified lines
1072 dma_fence_wait(vm->last_update, false);1092 dma_fence_wait(vm->last_update, false);
1073 1093
1074 ret = amdgpu_evf_mgr_rearm(&fpriv->evf_mgr, &exec);1094 ret = amdgpu_evf_mgr_rearm(&fpriv->evf_mgr, &exec);
1075 if (ret)1095 if (ret) {
1076 drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n");1096 drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n");
1097 goto unlock_all;
1098 }
1099
1100 ret = amdgpu_userq_restore_all(uq_mgr);
1077 1101
1078unlock_all:1102unlock_all:
1079 drm_exec_fini(&exec);1103 drm_exec_fini(&exec);
@@ -1099,18 +1123,34 @@ static void amdgpu_userq_restore_worker(struct work_struct *work) · 19 unmodified lines
1099 if (!dma_fence_is_signaled(ev_fence))1123 if (!dma_fence_is_signaled(ev_fence))
1100 goto put_fence;1124 goto put_fence;
1101 1125
1102 ret = amdgpu_userq_vm_validate(uq_mgr);1126 ret = amdgpu_userq_vm_validate_and_restore_queue(uq_mgr);
1103 if (ret) {1127 if (ret) {
1104 drm_file_err(uq_mgr->file, "Failed to validate BOs to restore ret=%d\n", ret);1128 drm_file_err(uq_mgr->file, "Failed to validate BOs to restore ret=%d\n", ret);
1105 goto put_fence;1129 goto put_fence;
1106 }1130 }
1107 1131
1108 amdgpu_userq_restore_all(uq_mgr);
1109
1110put_fence:1132put_fence:
1111 dma_fence_put(ev_fence);1133 dma_fence_put(ev_fence);
1112}1134}
1113 1135
1136void amdgpu_userq_process_reset_irq(struct amdgpu_device *adev,
1137 u32 pasid, u32 doorbell_offset)
1138{
1139 struct xarray *xa = &adev->userq_doorbell_xa;
1140 struct amdgpu_usermode_queue *queue;
1141 unsigned long flags, idx;
1142
1143 xa_lock_irqsave(xa, flags);
1144 xa_for_each(xa, idx, queue) {
1145 if (queue->vm && queue->vm->pasid == pasid &&
1146 queue->doorbell_offset == doorbell_offset) {
1147 amdgpu_userq_start_hang_detect_work(queue);
1148 break;
1149 }
1150 }
1151 xa_unlock_irqrestore(xa, flags);
1152}
1153
1114static int1154static int
1115amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr)1155amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr)
1116{1156{
@@ -1166,6 +1206,7 @@ int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *f · 49 unmodified lines
1166 xa_init_flags(&userq_mgr->userq_xa, XA_FLAGS_ALLOC);1206 xa_init_flags(&userq_mgr->userq_xa, XA_FLAGS_ALLOC);
1167 userq_mgr->adev = adev;1207 userq_mgr->adev = adev;
1168 userq_mgr->file = file_priv;1208 userq_mgr->file = file_priv;
1209 mutex_init(&userq_mgr->proc_ctx_lock);
1169 1210
1170 INIT_DELAYED_WORK(&userq_mgr->resume_work, amdgpu_userq_restore_worker);1211 INIT_DELAYED_WORK(&userq_mgr->resume_work, amdgpu_userq_restore_worker);
1171 INIT_WORK(&userq_mgr->reset_work, amdgpu_userq_mgr_reset_work);1212 INIT_WORK(&userq_mgr->reset_work, amdgpu_userq_mgr_reset_work);
@@ -1219,6 +1260,11 @@ void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr) · 47 unmodified lines
1219 */1260 */
1220 cancel_work_sync(&userq_mgr->reset_work);1261 cancel_work_sync(&userq_mgr->reset_work);
1221 1262
1263 amdgpu_bo_free_kernel(&userq_mgr->proc_ctx_obj.obj,
1264 &userq_mgr->proc_ctx_obj.gpu_addr,
1265 &userq_mgr->proc_ctx_obj.cpu_ptr);
1266
1267 mutex_destroy(&userq_mgr->proc_ctx_lock);
1222 mutex_destroy(&userq_mgr->userq_mutex);1268 mutex_destroy(&userq_mgr->userq_mutex);
1223}1269}
1224 1270
@@ -1370,12 +1416,14 @@ void amdgpu_userq_pre_reset(struct amdgpu_device *adev) · 145 unmodified lines
1370 if (queue->state != AMDGPU_USERQ_STATE_MAPPED)1416 if (queue->state != AMDGPU_USERQ_STATE_MAPPED)
1371 continue;1417 continue;
1372 1418
1419 trace_amdgpu_userq_state_start(queue);
1373 userq_funcs = adev->userq_funcs[queue->queue_type];1420 userq_funcs = adev->userq_funcs[queue->queue_type];
1374 userq_funcs->unmap(queue);1421 userq_funcs->unmap(queue);
1375 /* just mark all queues as hung at this point.1422 /* just mark all queues as hung at this point.
1376 * if unmap succeeds, we could map again1423 * if unmap succeeds, we could map again
1377 * in amdgpu_userq_post_reset() if vram is not lost1424 * in amdgpu_userq_post_reset() if vram is not lost
1378 */1425 */
1426 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_HUNG);
1379 queue->state = AMDGPU_USERQ_STATE_HUNG;1427 queue->state = AMDGPU_USERQ_STATE_HUNG;
1380 amdgpu_userq_fence_driver_force_completion(queue);1428 amdgpu_userq_fence_driver_force_completion(queue);
1381 }1429 }
@@ -1394,6 +1442,8 @@ int amdgpu_userq_post_reset(struct amdgpu_device *adev, bool vram_lost) · 12 unmodified lines
1394 1442
1395 xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {1443 xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {
1396 if (queue->state == AMDGPU_USERQ_STATE_HUNG && !vram_lost) {1444 if (queue->state == AMDGPU_USERQ_STATE_HUNG && !vram_lost) {
1445 trace_amdgpu_userq_state_start(queue);
1446
1397 userq_funcs = adev->userq_funcs[queue->queue_type];1447 userq_funcs = adev->userq_funcs[queue->queue_type];
1398 /* Re-map queue */1448 /* Re-map queue */
1399 r = userq_funcs->map(queue);1449 r = userq_funcs->map(queue);
@@ -1401,6 +1451,7 @@ int amdgpu_userq_post_reset(struct amdgpu_device *adev, bool vram_lost) · 1 unmodified line
1401 dev_err(adev->dev, "Failed to remap queue %ld\n", queue_id);1451 dev_err(adev->dev, "Failed to remap queue %ld\n", queue_id);
1402 continue;1452 continue;
1403 }1453 }
1454 trace_amdgpu_userq_state_changed(queue, AMDGPU_USERQ_STATE_MAPPED);
1404 queue->state = AMDGPU_USERQ_STATE_MAPPED;1455 queue->state = AMDGPU_USERQ_STATE_MAPPED;
1405 }1456 }
1406 }1457 }
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.h+14−2
@@ -53,6 +53,7 @@ struct amdgpu_usermode_queue { · 52 unmodified lines
53 enum amdgpu_userq_state state;53 enum amdgpu_userq_state state;
54 uint64_t doorbell_handle;54 uint64_t doorbell_handle;
55 uint64_t doorbell_index;55 uint64_t doorbell_index;
56 u32 doorbell_offset;
56 uint64_t flags;57 uint64_t flags;
57 struct amdgpu_mqd_prop *userq_prop;58 struct amdgpu_mqd_prop *userq_prop;
58 struct amdgpu_userq_mgr *userq_mgr;59 struct amdgpu_userq_mgr *userq_mgr;
@@ -111,8 +112,7 @@ struct amdgpu_userq_funcs { · 52 unmodified lines
111 int (*map)(struct amdgpu_usermode_queue *queue);112 int (*map)(struct amdgpu_usermode_queue *queue);
112 int (*preempt)(struct amdgpu_usermode_queue *queue);113 int (*preempt)(struct amdgpu_usermode_queue *queue);
113 int (*restore)(struct amdgpu_usermode_queue *queue);114 int (*restore)(struct amdgpu_usermode_queue *queue);
114 int (*detect_and_reset)(struct amdgpu_device *adev,115 int (*reset)(struct amdgpu_usermode_queue *queue);
115 int queue_type);
116};116};
117 117
118/* Usermode queues for gfx */118/* Usermode queues for gfx */
@@ -127,6 +127,8 @@ struct amdgpu_userq_mgr { · 8 unmodified lines
127 struct amdgpu_device *adev;127 struct amdgpu_device *adev;
128 struct delayed_work resume_work;128 struct delayed_work resume_work;
129 struct drm_file *file;129 struct drm_file *file;
130 struct mutex proc_ctx_lock;
131 struct amdgpu_userq_obj proc_ctx_obj;
130 132
131 /**133 /**
132 * @reset_work:134 * @reset_work:
@@ -177,6 +179,16 @@ int amdgpu_userq_post_reset(struct amdgpu_device *adev, bool vram_lost); · 44 unmodified lines
177void amdgpu_userq_start_hang_detect_work(struct amdgpu_usermode_queue *queue);179void amdgpu_userq_start_hang_detect_work(struct amdgpu_usermode_queue *queue);
178void amdgpu_userq_process_fence_irq(struct amdgpu_device *adev, u32 doorbell);180void amdgpu_userq_process_fence_irq(struct amdgpu_device *adev, u32 doorbell);
179 181
182/*
183 * CP packs the per-process doorbell_id of the queue in
184 * CTXID0[9:0] on priv-fault (same encoding KFD uses via
185 * KFD_CTXID0_DOORBELL_ID_MASK)
186 */
187#define AMDGPU_CTXID0_DOORBELL_ID_MASK 0x3ff
188
189void amdgpu_userq_process_reset_irq(struct amdgpu_device *adev,
190 u32 pasid, u32 doorbell_offset);
191
180int amdgpu_userq_input_va_validate(struct amdgpu_device *adev,192int amdgpu_userq_input_va_validate(struct amdgpu_device *adev,
181 struct amdgpu_usermode_queue *queue,193 struct amdgpu_usermode_queue *queue,
182 u64 addr, u64 expected_size, u64 *va_out);194 u64 addr, u64 expected_size, u64 *va_out);
drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c+7−3
@@ -30,7 +30,7 @@ · 29 unmodified lines
30#include <drm/drm_syncobj.h>30#include <drm/drm_syncobj.h>
31 31
32#include "amdgpu.h"32#include "amdgpu.h"
33#include "amdgpu_userq_fence.h"33#include "amdgpu_trace.h"
34 34
35#define AMDGPU_USERQ_MAX_HANDLES (1U << 16)35#define AMDGPU_USERQ_MAX_HANDLES (1U << 16)
36 36
@@ -528,6 +528,8 @@ int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data, · 491 unmodified lines
528 /* Create the new fence */528 /* Create the new fence */
529 amdgpu_userq_fence_init(queue, fence, wptr);529 amdgpu_userq_fence_init(queue, fence, wptr);
530 530
531 trace_amdgpu_userq_emit_fence(dev->dev, queue, fence);
532
531 mutex_unlock(&userq_mgr->userq_mutex);533 mutex_unlock(&userq_mgr->userq_mutex);
532 534
533 /*535 /*
@@ -701,7 +703,7 @@ amdgpu_userq_wait_add_fence(struct drm_amdgpu_userq_wait *wait_info, · 167 unmodified lines
701}703}
702 704
703static int705static int
704amdgpu_userq_wait_return_fence_info(struct drm_file *filp,706amdgpu_userq_wait_return_fence_info(struct drm_device *dev, struct drm_file *filp,
705 struct drm_amdgpu_userq_wait *wait_info,707 struct drm_amdgpu_userq_wait *wait_info,
706 u32 *syncobj_handles, u64 *timeline_points,708 u32 *syncobj_handles, u64 *timeline_points,
707 u32 *timeline_handles,709 u32 *timeline_handles,
@@ -869,6 +871,8 @@ amdgpu_userq_wait_return_fence_info(struct drm_file *filp, · 161 unmodified lines
869 871
870 amdgpu_userq_fence_driver_get(fence_drv);872 amdgpu_userq_fence_driver_get(fence_drv);
871 873
874 trace_amdgpu_userq_wait_deps(dev->dev, waitq, userq_fence);
875
872 /* Store drm syncobj's gpu va address and value */876 /* Store drm syncobj's gpu va address and value */
873 fence_info[cnt].va = fence_drv->va;877 fence_info[cnt].va = fence_drv->va;
874 fence_info[cnt].value = fences[i]->seqno;878 fence_info[cnt].value = fences[i]->seqno;
@@ -969,7 +973,7 @@ int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data, · 94 unmodified lines
969 gobj_write,973 gobj_write,
970 gobj_read);974 gobj_read);
971 } else {975 } else {
972 r = amdgpu_userq_wait_return_fence_info(filp, wait_info,976 r = amdgpu_userq_wait_return_fence_info(dev, filp, wait_info,
973 syncobj_handles,977 syncobj_handles,
974 timeline_points,978 timeline_points,
975 timeline_handles,979 timeline_handles,
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.c+45−31
@@ -506,9 +506,8 @@ void amdgpu_vcn_ring_begin_use(struct amdgpu_ring *ring) · 505 unmodified lines
506 struct amdgpu_device *adev = ring->adev;506 struct amdgpu_device *adev = ring->adev;
507 struct amdgpu_vcn_inst *vcn_inst = &adev->vcn.inst[ring->me];507 struct amdgpu_vcn_inst *vcn_inst = &adev->vcn.inst[ring->me];
508 508
509 atomic_inc(&vcn_inst->total_submission_cnt);509 if (!atomic_fetch_inc(&vcn_inst->total_submission_cnt))
510 510 cancel_delayed_work_sync(&vcn_inst->idle_work);
511 cancel_delayed_work_sync(&vcn_inst->idle_work);
512 511
513 mutex_lock(&vcn_inst->vcn_pg_lock);512 mutex_lock(&vcn_inst->vcn_pg_lock);
514 vcn_inst->set_pg_state(vcn_inst, AMD_PG_STATE_UNGATE);513 vcn_inst->set_pg_state(vcn_inst, AMD_PG_STATE_UNGATE);
@@ -550,10 +549,9 @@ void amdgpu_vcn_ring_end_use(struct amdgpu_ring *ring) · 35 unmodified lines
550 !adev->vcn.inst[ring->me].using_unified_queue)549 !adev->vcn.inst[ring->me].using_unified_queue)
551 atomic_dec(&ring->adev->vcn.inst[ring->me].dpg_enc_submission_cnt);550 atomic_dec(&ring->adev->vcn.inst[ring->me].dpg_enc_submission_cnt);
552 551
553 atomic_dec(&ring->adev->vcn.inst[ring->me].total_submission_cnt);552 if (atomic_dec_and_test(&ring->adev->vcn.inst[ring->me].total_submission_cnt))
554 553 schedule_delayed_work(&ring->adev->vcn.inst[ring->me].idle_work,
555 schedule_delayed_work(&ring->adev->vcn.inst[ring->me].idle_work,554 VCN_IDLE_TIMEOUT);
556 VCN_IDLE_TIMEOUT);
557}555}
558 556
559int amdgpu_vcn_dec_ring_test_ring(struct amdgpu_ring *ring)557int amdgpu_vcn_dec_ring_test_ring(struct amdgpu_ring *ring)
@@ -1485,6 +1483,37 @@ int vcn_set_powergating_state(struct amdgpu_ip_block *ip_block, · 925 unmodified lines
1485 return ret;1483 return ret;
1486}1484}
1487 1485
1486static struct amdgpu_fence *
1487amdgpu_vcn_ring_reset_begin_helper(struct amdgpu_ring *ring,
1488 struct amdgpu_ring *guilty_ring,
1489 struct amdgpu_fence *timedout_fence)
1490{
1491 struct amdgpu_fence *fence;
1492
1493 drm_sched_wqueue_stop(&ring->sched);
1494 if (ring == guilty_ring)
1495 fence = timedout_fence;
1496 else
1497 fence = amdgpu_ring_find_guilty_fence(ring);
1498 amdgpu_ring_reset_helper_begin(ring, fence);
1499
1500 return fence;
1501}
1502
1503static int
1504amdgpu_vcn_ring_reset_end_helper(struct amdgpu_ring *ring,
1505 struct amdgpu_fence *fence)
1506{
1507 int r;
1508
1509 r = amdgpu_ring_reset_helper_end(ring, fence);
1510 if (r)
1511 return r;
1512
1513 drm_sched_wqueue_start(&ring->sched);
1514 return 0;
1515}
1516
1488/**1517/**
1489 * amdgpu_vcn_ring_reset - Reset a VCN ring1518 * amdgpu_vcn_ring_reset - Reset a VCN ring
1490 * @ring: ring to reset1519 * @ring: ring to reset
@@ -1502,48 +1531,33 @@ int amdgpu_vcn_ring_reset(struct amdgpu_ring *ring, · 11 unmodified lines
1502{1531{
1503 struct amdgpu_device *adev = ring->adev;1532 struct amdgpu_device *adev = ring->adev;
1504 struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];1533 struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me];
1534 struct amdgpu_fence *dec_fence;
1535 struct amdgpu_fence *enc_fence[AMDGPU_VCN_MAX_ENC_RINGS];
1505 int r, i;1536 int r, i;
1506 1537
1507 if (adev->vcn.inst[ring->me].using_unified_queue)1538 if (adev->vcn.inst[ring->me].using_unified_queue)
1508 return -EINVAL;1539 return -EINVAL;
1509 1540
1510 mutex_lock(&vinst->engine_reset_mutex);1541 mutex_lock(&vinst->engine_reset_mutex);
1511 /* Stop the scheduler's work queue for the dec and enc rings if they are running.1542 dec_fence = amdgpu_vcn_ring_reset_begin_helper(&vinst->ring_dec, ring,
1512 * This ensures that no new tasks are submitted to the queues while1543 timedout_fence);
1513 * the reset is in progress.
1514 */
1515 drm_sched_wqueue_stop(&vinst->ring_dec.sched);
1516 for (i = 0; i < vinst->num_enc_rings; i++)1544 for (i = 0; i < vinst->num_enc_rings; i++)
1517 drm_sched_wqueue_stop(&vinst->ring_enc[i].sched);1545 enc_fence[i] = amdgpu_vcn_ring_reset_begin_helper(&vinst->ring_enc[i], ring,
1546 timedout_fence);
1518 1547
1519 /* Perform the VCN reset for the specified instance */1548 /* Perform the VCN reset for the specified instance */
1520 r = vinst->reset(vinst);1549 r = vinst->reset(vinst);
1521 if (r)1550 if (r)
1522 goto unlock;1551 goto unlock;
1523 r = amdgpu_ring_test_ring(&vinst->ring_dec);1552
1553 r = amdgpu_vcn_ring_reset_end_helper(&vinst->ring_dec, dec_fence);
1524 if (r)1554 if (r)
1525 goto unlock;1555 goto unlock;
1526 for (i = 0; i < vinst->num_enc_rings; i++) {1556 for (i = 0; i < vinst->num_enc_rings; i++) {
1527 r = amdgpu_ring_test_ring(&vinst->ring_enc[i]);1557 r = amdgpu_vcn_ring_reset_end_helper(&vinst->ring_enc[i], enc_fence[i]);
1528 if (r)1558 if (r)
1529 goto unlock;1559 goto unlock;
1530 }1560 }
1531 amdgpu_fence_driver_force_completion(&vinst->ring_dec,
1532 (&vinst->ring_dec == ring) ?
1533 &timedout_fence->base : NULL);
1534 for (i = 0; i < vinst->num_enc_rings; i++)
1535 amdgpu_fence_driver_force_completion(&vinst->ring_enc[i],
1536 (&vinst->ring_enc[i] == ring) ?
1537 &timedout_fence->base : NULL);
1538
1539 /* Restart the scheduler's work queue for the dec and enc rings
1540 * if they were stopped by this function. This allows new tasks
1541 * to be submitted to the queues after the reset is complete.
1542 */
1543 drm_sched_wqueue_start(&vinst->ring_dec.sched);
1544 for (i = 0; i < vinst->num_enc_rings; i++)
1545 drm_sched_wqueue_start(&vinst->ring_enc[i].sched);
1546
1547unlock:1561unlock:
1548 mutex_unlock(&vinst->engine_reset_mutex);1562 mutex_unlock(&vinst->engine_reset_mutex);
1549 1563
drivers/gpu/drm/amd/amdgpu/amdgpu_vcn.h+3−0
@@ -368,6 +368,9 @@ struct amdgpu_vcn { · 367 unmodified lines
368 struct mutex workload_profile_mutex;368 struct mutex workload_profile_mutex;
369 u32 reg_count;369 u32 reg_count;
370 const struct amdgpu_hwip_reg_entry *reg_list;370 const struct amdgpu_hwip_reg_entry *reg_list;
371
372 bool disable_uq;
373 bool disable_kq;
371};374};
372 375
373struct amdgpu_fw_shared_rb_ptrs_struct {376struct amdgpu_fw_shared_rb_ptrs_struct {
drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c+31−11
@@ -381,7 +381,8 @@ int amdgpu_xcp_get_inst_details(struct amdgpu_xcp *xcp, · 380 unmodified lines
381 enum AMDGPU_XCP_IP_BLOCK ip,381 enum AMDGPU_XCP_IP_BLOCK ip,
382 uint32_t *inst_mask)382 uint32_t *inst_mask)
383{383{
384 if (!xcp->valid || !inst_mask || !(xcp->ip[ip].valid))384 if (!xcp->valid || !inst_mask || ip >= AMDGPU_XCP_MAX_BLOCKS ||
385 !(xcp->ip[ip].valid))
385 return -EINVAL;386 return -EINVAL;
386 387
387 *inst_mask = xcp->ip[ip].inst_mask;388 *inst_mask = xcp->ip[ip].inst_mask;
@@ -468,14 +469,18 @@ void amdgpu_xcp_release_sched(struct amdgpu_device *adev, · 80 unmodified lines
468{469{
469 struct drm_gpu_scheduler *sched =470 struct drm_gpu_scheduler *sched =
470 container_of(entity->entity.rq, typeof(*sched), rq);471 container_of(entity->entity.rq, typeof(*sched), rq);
472 struct amdgpu_xcp_mgr *xcp_mgr = adev->xcp_mgr;
471 473
472 if (!adev->xcp_mgr)474 if (!xcp_mgr)
473 return;475 return;
474 476
475 if (drm_sched_wqueue_ready(sched)) {477 if (drm_sched_wqueue_ready(sched)) {
476 struct amdgpu_ring *ring = to_amdgpu_ring(sched);478 struct amdgpu_ring *ring = to_amdgpu_ring(sched);
477 479
478 atomic_dec(&adev->xcp_mgr->xcp[ring->xcp_id].ref_cnt);480 mutex_lock(&xcp_mgr->xcp_lock);
481 if (ring->xcp_id < xcp_mgr->num_xcps && xcp_mgr->xcp[ring->xcp_id].valid)
482 atomic_dec(&xcp_mgr->xcp[ring->xcp_id].ref_cnt);
483 mutex_unlock(&xcp_mgr->xcp_lock);
479 }484 }
480}485}
481 486
@@ -488,7 +493,9 @@ int amdgpu_xcp_select_scheds(struct amdgpu_device *adev, · 6 unmodified lines
488 u32 sel_xcp_id;493 u32 sel_xcp_id;
489 int i;494 int i;
490 struct amdgpu_xcp_mgr *xcp_mgr = adev->xcp_mgr;495 struct amdgpu_xcp_mgr *xcp_mgr = adev->xcp_mgr;
496 int r = 0;
491 497
498 mutex_lock(&xcp_mgr->xcp_lock);
492 if (fpriv->xcp_id == AMDGPU_XCP_NO_PARTITION) {499 if (fpriv->xcp_id == AMDGPU_XCP_NO_PARTITION) {
493 u32 least_ref_cnt = ~0;500 u32 least_ref_cnt = ~0;
494 501
@@ -505,19 +512,27 @@ int amdgpu_xcp_select_scheds(struct amdgpu_device *adev, · 10 unmodified lines
505 }512 }
506 sel_xcp_id = fpriv->xcp_id;513 sel_xcp_id = fpriv->xcp_id;
507 514
515 if (sel_xcp_id >= xcp_mgr->num_xcps || !xcp_mgr->xcp[sel_xcp_id].valid) {
516 dev_err(adev->dev, "Selected partition #%d is not valid.", sel_xcp_id);
517 r = -ENODEV;
518 goto out;
519 }
520
508 if (xcp_mgr->xcp[sel_xcp_id].gpu_sched[hw_ip][hw_prio].num_scheds) {521 if (xcp_mgr->xcp[sel_xcp_id].gpu_sched[hw_ip][hw_prio].num_scheds) {
509 *num_scheds =522 *num_scheds =
510 xcp_mgr->xcp[fpriv->xcp_id].gpu_sched[hw_ip][hw_prio].num_scheds;523 xcp_mgr->xcp[sel_xcp_id].gpu_sched[hw_ip][hw_prio].num_scheds;
511 *scheds =524 *scheds =
512 xcp_mgr->xcp[fpriv->xcp_id].gpu_sched[hw_ip][hw_prio].sched;525 xcp_mgr->xcp[sel_xcp_id].gpu_sched[hw_ip][hw_prio].sched;
513 atomic_inc(&adev->xcp_mgr->xcp[sel_xcp_id].ref_cnt);526 atomic_inc(&xcp_mgr->xcp[sel_xcp_id].ref_cnt);
514 dev_dbg(adev->dev, "Selected partition #%d", sel_xcp_id);527 dev_dbg(adev->dev, "Selected partition #%d", sel_xcp_id);
515 } else {528 } else {
516 dev_err(adev->dev, "Failed to schedule partition #%d.", sel_xcp_id);529 dev_err(adev->dev, "Failed to schedule partition #%d.", sel_xcp_id);
517 return -ENOENT;530 r = -ENOENT;
518 }531 }
519 532
520 return 0;533out:
534 mutex_unlock(&xcp_mgr->xcp_lock);
535 return r;
521}536}
522 537
523static void amdgpu_set_xcp_id(struct amdgpu_device *adev,538static void amdgpu_set_xcp_id(struct amdgpu_device *adev,
@@ -574,6 +589,9 @@ static void amdgpu_xcp_gpu_sched_update(struct amdgpu_device *adev, · 50 unmodified lines
574{589{
575 unsigned int *num_gpu_sched;590 unsigned int *num_gpu_sched;
576 591
592 if (sel_xcp_id >= MAX_XCP || sel_xcp_id == AMDGPU_XCP_NO_PARTITION)
593 return;
594
577 num_gpu_sched = &adev->xcp_mgr->xcp[sel_xcp_id]595 num_gpu_sched = &adev->xcp_mgr->xcp[sel_xcp_id]
578 .gpu_sched[ring->funcs->type][ring->hw_prio].num_scheds;596 .gpu_sched[ring->funcs->type][ring->hw_prio].num_scheds;
579 adev->xcp_mgr->xcp[sel_xcp_id].gpu_sched[ring->funcs->type][ring->hw_prio]597 adev->xcp_mgr->xcp[sel_xcp_id].gpu_sched[ring->funcs->type][ring->hw_prio]
@@ -903,7 +921,7 @@ static void amdgpu_xcp_cfg_sysfs_init(struct amdgpu_device *adev) · 323 unmodified lines
903{921{
904 struct amdgpu_xcp_res_details *xcp_res;922 struct amdgpu_xcp_res_details *xcp_res;
905 struct amdgpu_xcp_cfg *xcp_cfg;923 struct amdgpu_xcp_cfg *xcp_cfg;
906 int i, r, j, rid, mode;924 int i, r, rid, mode;
907 925
908 if (!adev->xcp_mgr)926 if (!adev->xcp_mgr)
909 return;927 return;
@@ -949,14 +967,16 @@ static void amdgpu_xcp_cfg_sysfs_init(struct amdgpu_device *adev) · 39 unmodified lines
949 &xcp_cfg_res_sysfs_ktype,967 &xcp_cfg_res_sysfs_ktype,
950 &xcp_cfg->kobj, "%s",968 &xcp_cfg->kobj, "%s",
951 xcp_res_names[rid]);969 xcp_res_names[rid]);
952 if (r)970 if (r) {
971 kobject_put(&xcp_res->kobj);
953 goto err;972 goto err;
973 }
954 }974 }
955 975
956 adev->xcp_mgr->xcp_cfg = xcp_cfg;976 adev->xcp_mgr->xcp_cfg = xcp_cfg;
957 return;977 return;
958err:978err:
959 for (j = 0; j < i; j++) {979 while (i--) {
960 xcp_res = &xcp_cfg->xcp_res[i];980 xcp_res = &xcp_cfg->xcp_res[i];
961 kobject_put(&xcp_res->kobj);981 kobject_put(&xcp_res->kobj);
962 }982 }
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c+15−284
@@ -106,53 +106,6 @@ static const int walf_pcs_err_noncorrectable_mask_reg_aldebaran[] = { · 105 unmodified lines
106 smnPCS_GOPX1_PCS_ERROR_NONCORRECTABLE_MASK + 0x100000106 smnPCS_GOPX1_PCS_ERROR_NONCORRECTABLE_MASK + 0x100000
107};107};
108 108
109static const int xgmi3x16_pcs_err_status_reg_v6_4[] = {
110 smnPCS_XGMI3X16_PCS_ERROR_STATUS,
111 smnPCS_XGMI3X16_PCS_ERROR_STATUS + 0x100000
112};
113
114static const int xgmi3x16_pcs_err_noncorrectable_mask_reg_v6_4[] = {
115 smnPCS_XGMI3X16_PCS_ERROR_NONCORRECTABLE_MASK,
116 smnPCS_XGMI3X16_PCS_ERROR_NONCORRECTABLE_MASK + 0x100000
117};
118
119static const u64 xgmi_v6_4_0_mca_base_array[] = {
120 0x11a09200,
121 0x11b09200,
122};
123
124static const char *xgmi_v6_4_0_ras_error_code_ext[32] = {
125 [0x00] = "XGMI PCS DataLossErr",
126 [0x01] = "XGMI PCS TrainingErr",
127 [0x02] = "XGMI PCS FlowCtrlAckErr",
128 [0x03] = "XGMI PCS RxFifoUnderflowErr",
129 [0x04] = "XGMI PCS RxFifoOverflowErr",
130 [0x05] = "XGMI PCS CRCErr",
131 [0x06] = "XGMI PCS BERExceededErr",
132 [0x07] = "XGMI PCS TxMetaDataErr",
133 [0x08] = "XGMI PCS ReplayBufParityErr",
134 [0x09] = "XGMI PCS DataParityErr",
135 [0x0a] = "XGMI PCS ReplayFifoOverflowErr",
136 [0x0b] = "XGMI PCS ReplayFifoUnderflowErr",
137 [0x0c] = "XGMI PCS ElasticFifoOverflowErr",
138 [0x0d] = "XGMI PCS DeskewErr",
139 [0x0e] = "XGMI PCS FlowCtrlCRCErr",
140 [0x0f] = "XGMI PCS DataStartupLimitErr",
141 [0x10] = "XGMI PCS FCInitTimeoutErr",
142 [0x11] = "XGMI PCS RecoveryTimeoutErr",
143 [0x12] = "XGMI PCS ReadySerialTimeoutErr",
144 [0x13] = "XGMI PCS ReadySerialAttemptErr",
145 [0x14] = "XGMI PCS RecoveryAttemptErr",
146 [0x15] = "XGMI PCS RecoveryRelockAttemptErr",
147 [0x16] = "XGMI PCS ReplayAttemptErr",
148 [0x17] = "XGMI PCS SyncHdrErr",
149 [0x18] = "XGMI PCS TxReplayTimeoutErr",
150 [0x19] = "XGMI PCS RxReplayTimeoutErr",
151 [0x1a] = "XGMI PCS LinkSubTxTimeoutErr",
152 [0x1b] = "XGMI PCS LinkSubRxTimeoutErr",
153 [0x1c] = "XGMI PCS RxCMDPktErr",
154};
155
156static const struct amdgpu_pcs_ras_field xgmi_pcs_ras_fields[] = {109static const struct amdgpu_pcs_ras_field xgmi_pcs_ras_fields[] = {
157 {"XGMI PCS DataLossErr",110 {"XGMI PCS DataLossErr",
158 SOC15_REG_FIELD(XGMI0_PCS_GOPX16_PCS_ERROR_STATUS, DataLossErr)},111 SOC15_REG_FIELD(XGMI0_PCS_GOPX16_PCS_ERROR_STATUS, DataLossErr)},
@@ -1152,91 +1105,15 @@ int amdgpu_xgmi_remove_device(struct amdgpu_device *adev) · 993 unmodified lines
1152 return 0;1105 return 0;
1153}1106}
1154 1107
1155static int xgmi_v6_4_0_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
1156 enum aca_smu_type type, void *data)
1157{
1158 struct amdgpu_device *adev = handle->adev;
1159 struct aca_bank_info info;
1160 const char *error_str;
1161 u64 status, count;
1162 int ret, ext_error_code;
1163
1164 ret = aca_bank_info_decode(bank, &info);
1165 if (ret)
1166 return ret;
1167
1168 status = bank->regs[ACA_REG_IDX_STATUS];
1169 ext_error_code = ACA_REG__STATUS__ERRORCODEEXT(status);
1170
1171 error_str = ext_error_code < ARRAY_SIZE(xgmi_v6_4_0_ras_error_code_ext) ?
1172 xgmi_v6_4_0_ras_error_code_ext[ext_error_code] : NULL;
1173 if (error_str)
1174 dev_info(adev->dev, "%s detected\n", error_str);
1175
1176 count = ACA_REG__MISC0__ERRCNT(bank->regs[ACA_REG_IDX_MISC0]);
1177
1178 switch (type) {
1179 case ACA_SMU_TYPE_UE:
1180 if (ext_error_code != 0 && ext_error_code != 1 && ext_error_code != 9)
1181 count = 0ULL;
1182
1183 bank->aca_err_type = ACA_ERROR_TYPE_UE;
1184 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE, count);
1185 break;
1186 case ACA_SMU_TYPE_CE:
1187 count = ext_error_code == 6 ? count : 0ULL;
1188 bank->aca_err_type = ACA_ERROR_TYPE_CE;
1189 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type, count);
1190 break;
1191 default:
1192 return -EINVAL;
1193 }
1194
1195 return ret;
1196}
1197
1198static const struct aca_bank_ops xgmi_v6_4_0_aca_bank_ops = {
1199 .aca_bank_parser = xgmi_v6_4_0_aca_bank_parser,
1200};
1201
1202static const struct aca_info xgmi_v6_4_0_aca_info = {
1203 .hwip = ACA_HWIP_TYPE_PCS_XGMI,
1204 .mask = ACA_ERROR_UE_MASK | ACA_ERROR_CE_MASK,
1205 .bank_ops = &xgmi_v6_4_0_aca_bank_ops,
1206};
1207
1208static int amdgpu_xgmi_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)1108static int amdgpu_xgmi_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
1209{1109{
1210 int r;
1211
1212 if (!adev->gmc.xgmi.supported ||1110 if (!adev->gmc.xgmi.supported ||
1213 adev->gmc.xgmi.num_physical_nodes == 0)1111 adev->gmc.xgmi.num_physical_nodes == 0)
1214 return 0;1112 return 0;
1215 1113
1216 amdgpu_ras_reset_error_count(adev, AMDGPU_RAS_BLOCK__XGMI_WAFL);1114 amdgpu_ras_reset_error_count(adev, AMDGPU_RAS_BLOCK__XGMI_WAFL);
1217 1115
1218 r = amdgpu_ras_block_late_init(adev, ras_block);1116 return amdgpu_ras_block_late_init(adev, ras_block);
1219 if (r)
1220 return r;
1221
1222 switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
1223 case IP_VERSION(6, 4, 0):
1224 case IP_VERSION(6, 4, 1):
1225 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__XGMI_WAFL,
1226 &xgmi_v6_4_0_aca_info, NULL);
1227 if (r)
1228 goto late_fini;
1229 break;
1230 default:
1231 break;
1232 }
1233
1234 return 0;
1235
1236late_fini:
1237 amdgpu_ras_block_late_fini(adev, ras_block);
1238
1239 return r;
1240}1117}
1241 1118
1242uint64_t amdgpu_xgmi_get_relative_phy_addr(struct amdgpu_device *adev,1119uint64_t amdgpu_xgmi_get_relative_phy_addr(struct amdgpu_device *adev,
@@ -1252,7 +1129,7 @@ static void pcs_clear_status(struct amdgpu_device *adev, uint32_t pcs_status_reg · 9 unmodified lines
1252 WREG32_PCIE(pcs_status_reg, 0);1129 WREG32_PCIE(pcs_status_reg, 0);
1253}1130}
1254 1131
1255static void amdgpu_xgmi_legacy_reset_ras_error_count(struct amdgpu_device *adev)1132static void amdgpu_xgmi_reset_ras_error_count(struct amdgpu_device *adev)
1256{1133{
1257 uint32_t i;1134 uint32_t i;
1258 1135
@@ -1278,54 +1155,6 @@ static void amdgpu_xgmi_legacy_reset_ras_error_count(struct amdgpu_device *adev) · 19 unmodified lines
1278 default:1155 default:
1279 break;1156 break;
1280 }1157 }
1281
1282 switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
1283 case IP_VERSION(6, 4, 0):
1284 case IP_VERSION(6, 4, 1):
1285 for (i = 0; i < ARRAY_SIZE(xgmi3x16_pcs_err_status_reg_v6_4); i++)
1286 pcs_clear_status(adev,
1287 xgmi3x16_pcs_err_status_reg_v6_4[i]);
1288 break;
1289 default:
1290 break;
1291 }
1292}
1293
1294static void __xgmi_v6_4_0_reset_error_count(struct amdgpu_device *adev, int xgmi_inst, u64 mca_base)
1295{
1296 uint64_t smn_base =
1297 amdgpu_reg_get_smn_base64(adev, XGMI_HWIP, xgmi_inst);
1298
1299 WREG64_MCA(smn_base, mca_base, ACA_REG_IDX_STATUS, 0ULL);
1300}
1301
1302static void xgmi_v6_4_0_reset_error_count(struct amdgpu_device *adev, int xgmi_inst)
1303{
1304 int i;
1305
1306 for (i = 0; i < ARRAY_SIZE(xgmi_v6_4_0_mca_base_array); i++)
1307 __xgmi_v6_4_0_reset_error_count(adev, xgmi_inst, xgmi_v6_4_0_mca_base_array[i]);
1308}
1309
1310static void xgmi_v6_4_0_reset_ras_error_count(struct amdgpu_device *adev)
1311{
1312 int i;
1313
1314 for_each_inst(i, adev->aid_mask)
1315 xgmi_v6_4_0_reset_error_count(adev, i);
1316}
1317
1318static void amdgpu_xgmi_reset_ras_error_count(struct amdgpu_device *adev)
1319{
1320 switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
1321 case IP_VERSION(6, 4, 0):
1322 case IP_VERSION(6, 4, 1):
1323 xgmi_v6_4_0_reset_ras_error_count(adev);
1324 break;
1325 default:
1326 amdgpu_xgmi_legacy_reset_ras_error_count(adev);
1327 break;
1328 }
1329}1158}
1330 1159
1331static int amdgpu_xgmi_query_pcs_error_status(struct amdgpu_device *adev,1160static int amdgpu_xgmi_query_pcs_error_status(struct amdgpu_device *adev,
@@ -1343,11 +1172,7 @@ static int amdgpu_xgmi_query_pcs_error_status(struct amdgpu_device *adev, · 11 unmodified lines
1343 1172
1344 if (is_xgmi_pcs) {1173 if (is_xgmi_pcs) {
1345 if (amdgpu_ip_version(adev, XGMI_HWIP, 0) ==1174 if (amdgpu_ip_version(adev, XGMI_HWIP, 0) ==
1346 IP_VERSION(6, 1, 0) ||1175 IP_VERSION(6, 1, 0)) {
1347 amdgpu_ip_version(adev, XGMI_HWIP, 0) ==
1348 IP_VERSION(6, 4, 0) ||
1349 amdgpu_ip_version(adev, XGMI_HWIP, 0) ==
1350 IP_VERSION(6, 4, 1)) {
1351 pcs_ras_fields = &xgmi3x16_pcs_ras_fields[0];1176 pcs_ras_fields = &xgmi3x16_pcs_ras_fields[0];
1352 field_array_size = ARRAY_SIZE(xgmi3x16_pcs_ras_fields);1177 field_array_size = ARRAY_SIZE(xgmi3x16_pcs_ras_fields);
1353 } else {1178 } else {
@@ -1381,11 +1206,11 @@ static int amdgpu_xgmi_query_pcs_error_status(struct amdgpu_device *adev, · 27 unmodified lines
1381 return 0;1206 return 0;
1382}1207}
1383 1208
1384static void amdgpu_xgmi_legacy_query_ras_error_count(struct amdgpu_device *adev,1209static void amdgpu_xgmi_query_ras_error_count(struct amdgpu_device *adev,
1385 void *ras_error_status)1210 void *ras_error_status)
1386{1211{
1387 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;1212 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;
1388 int i, supported = 1;1213 int i;
1389 uint32_t data, mask_data = 0;1214 uint32_t data, mask_data = 0;
1390 uint32_t ue_cnt = 0, ce_cnt = 0;1215 uint32_t ue_cnt = 0, ce_cnt = 0;
1391 1216
@@ -1449,26 +1274,6 @@ static void amdgpu_xgmi_legacy_query_ras_error_count(struct amdgpu_device *adev, · 57 unmodified lines
1449 }1274 }
1450 break;1275 break;
1451 default:1276 default:
1452 supported = 0;
1453 break;
1454 }
1455
1456 switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
1457 case IP_VERSION(6, 4, 0):
1458 case IP_VERSION(6, 4, 1):
1459 /* check xgmi3x16 pcs error */
1460 for (i = 0; i < ARRAY_SIZE(xgmi3x16_pcs_err_status_reg_v6_4); i++) {
1461 data = RREG32_PCIE(xgmi3x16_pcs_err_status_reg_v6_4[i]);
1462 mask_data =
1463 RREG32_PCIE(xgmi3x16_pcs_err_noncorrectable_mask_reg_v6_4[i]);
1464 if (data)
1465 amdgpu_xgmi_query_pcs_error_status(adev, data,
1466 mask_data, &ue_cnt, &ce_cnt, true, true);
1467 }
1468 break;
1469 default:
1470 if (!supported)
1471 dev_warn(adev->dev, "XGMI RAS error query not supported");
1472 break;1277 break;
1473 }1278 }
1474 1279
@@ -1478,90 +1283,6 @@ static void amdgpu_xgmi_legacy_query_ras_error_count(struct amdgpu_device *adev, · 3 unmodified lines
1478 err_data->ce_count += ce_cnt;1283 err_data->ce_count += ce_cnt;
1479}1284}
1480 1285
1481static enum aca_error_type xgmi_v6_4_0_pcs_mca_get_error_type(struct amdgpu_device *adev, u64 status)
1482{
1483 const char *error_str;
1484 int ext_error_code;
1485
1486 ext_error_code = ACA_REG__STATUS__ERRORCODEEXT(status);
1487
1488 error_str = ext_error_code < ARRAY_SIZE(xgmi_v6_4_0_ras_error_code_ext) ?
1489 xgmi_v6_4_0_ras_error_code_ext[ext_error_code] : NULL;
1490 if (error_str)
1491 dev_info(adev->dev, "%s detected\n", error_str);
1492
1493 switch (ext_error_code) {
1494 case 0:
1495 return ACA_ERROR_TYPE_UE;
1496 case 6:
1497 return ACA_ERROR_TYPE_CE;
1498 default:
1499 return -EINVAL;
1500 }
1501
1502 return -EINVAL;
1503}
1504
1505static void __xgmi_v6_4_0_query_error_count(struct amdgpu_device *adev, struct amdgpu_smuio_mcm_config_info *mcm_info,
1506 u64 mca_base, struct ras_err_data *err_data)
1507{
1508 int xgmi_inst = mcm_info->die_id;
1509 uint64_t smn_base;
1510 u64 status = 0;
1511
1512 status = RREG64_MCA(xgmi_inst, mca_base, ACA_REG_IDX_STATUS);
1513 if (!ACA_REG__STATUS__VAL(status))
1514 return;
1515
1516 switch (xgmi_v6_4_0_pcs_mca_get_error_type(adev, status)) {
1517 case ACA_ERROR_TYPE_UE:
1518 amdgpu_ras_error_statistic_ue_count(err_data, mcm_info, 1ULL);
1519 break;
1520 case ACA_ERROR_TYPE_CE:
1521 amdgpu_ras_error_statistic_ce_count(err_data, mcm_info, 1ULL);
1522 break;
1523 default:
1524 break;
1525 }
1526 smn_base = amdgpu_reg_get_smn_base64(adev, XGMI_HWIP, xgmi_inst);
1527 WREG64_MCA(smn_base, mca_base, ACA_REG_IDX_STATUS, 0ULL);
1528}
1529
1530static void xgmi_v6_4_0_query_error_count(struct amdgpu_device *adev, int xgmi_inst, struct ras_err_data *err_data)
1531{
1532 struct amdgpu_smuio_mcm_config_info mcm_info = {
1533 .socket_id = adev->smuio.funcs->get_socket_id(adev),
1534 .die_id = xgmi_inst,
1535 };
1536 int i;
1537
1538 for (i = 0; i < ARRAY_SIZE(xgmi_v6_4_0_mca_base_array); i++)
1539 __xgmi_v6_4_0_query_error_count(adev, &mcm_info, xgmi_v6_4_0_mca_base_array[i], err_data);
1540}
1541
1542static void xgmi_v6_4_0_query_ras_error_count(struct amdgpu_device *adev, void *ras_error_status)
1543{
1544 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;
1545 int i;
1546
1547 for_each_inst(i, adev->aid_mask)
1548 xgmi_v6_4_0_query_error_count(adev, i, err_data);
1549}
1550
1551static void amdgpu_xgmi_query_ras_error_count(struct amdgpu_device *adev,
1552 void *ras_error_status)
1553{
1554 switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) {
1555 case IP_VERSION(6, 4, 0):
1556 case IP_VERSION(6, 4, 1):
1557 xgmi_v6_4_0_query_ras_error_count(adev, ras_error_status);
1558 break;
1559 default:
1560 amdgpu_xgmi_legacy_query_ras_error_count(adev, ras_error_status);
1561 break;
1562 }
1563}
1564
1565/* Trigger XGMI/WAFL error */1286/* Trigger XGMI/WAFL error */
1566static int amdgpu_ras_error_inject_xgmi(struct amdgpu_device *adev,1287static int amdgpu_ras_error_inject_xgmi(struct amdgpu_device *adev,
1567 void *inject_if, uint32_t instance_mask)1288 void *inject_if, uint32_t instance_mask)
@@ -1663,6 +1384,16 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work) · 95 unmodified lines
1663 if (r && r != -EHWPOISON)1384 if (r && r != -EHWPOISON)
1664 dev_err(tmp_adev->dev,1385 dev_err(tmp_adev->dev,
1665 "error during bad page data initialization");1386 "error during bad page data initialization");
1387
1388 /*
1389 * For the reset-on-init path (e.g. an NPS memory partition
1390 * switch) the RAS IP block hw_init was skipped under the
1391 * minimal init level, so uniras was never enabled. Bring it
1392 * up now that the reset domain has been unlocked. This is a
1393 * no-op for any other reset path where RAS is already
1394 * initialized, and for non-uniras devices.
1395 */
1396 amdgpu_ras_resume_after_reset(tmp_adev);
1666 }1397 }
1667}1398}
1668 1399
drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.c+45−18
@@ -273,8 +273,10 @@ static int aqua_vanjaram_get_xcp_res_info(struct amdgpu_xcp_mgr *xcp_mgr, · 272 unmodified lines
273 xcp_cfg->num_res = ARRAY_SIZE(max_res);273 xcp_cfg->num_res = ARRAY_SIZE(max_res);
274 274
275 for (i = 0; i < xcp_cfg->num_res; i++) {275 for (i = 0; i < xcp_cfg->num_res; i++) {
276 res_lt_xcp = max_res[i] < num_xcp;
277 xcp_cfg->xcp_res[i].id = i;276 xcp_cfg->xcp_res[i].id = i;
277 if (!max_res[i])
278 continue;
279 res_lt_xcp = max_res[i] < num_xcp;
278 xcp_cfg->xcp_res[i].num_inst =280 xcp_cfg->xcp_res[i].num_inst =
279 res_lt_xcp ? 1 : max_res[i] / num_xcp;281 res_lt_xcp ? 1 : max_res[i] / num_xcp;
280 xcp_cfg->xcp_res[i].num_inst =282 xcp_cfg->xcp_res[i].num_inst =
@@ -589,6 +591,29 @@ static struct aqua_reg_list pcie_reg_addrs[] = { · 308 unmodified lines
589 { smreg_0x1A380088, 6, DW_ADDR_INCR },591 { smreg_0x1A380088, 6, DW_ADDR_INCR },
590};592};
591 593
594/*
595 * Return the GPU's internal US switch port, or NULL if it is not visible
596 * (e.g. passthrough) or the EP is parented under an unrelated bridge.
597 */
598static struct pci_dev *aqua_vanjaram_get_us_pdev(struct amdgpu_device *adev)
599{
600 struct pci_dev *ds_pdev, *us_pdev;
601
602 ds_pdev = pci_upstream_bridge(adev->pdev);
603 if (!ds_pdev || ds_pdev->vendor != PCI_VENDOR_ID_ATI ||
604 pci_pcie_type(ds_pdev) != PCI_EXP_TYPE_DOWNSTREAM)
605 return NULL;
606
607 us_pdev = pci_upstream_bridge(ds_pdev);
608 if (!us_pdev ||
609 (us_pdev->vendor != PCI_VENDOR_ID_ATI &&
610 us_pdev->vendor != PCI_VENDOR_ID_AMD) ||
611 pci_pcie_type(us_pdev) != PCI_EXP_TYPE_UPSTREAM)
612 return NULL;
613
614 return us_pdev;
615}
616
592static ssize_t aqua_vanjaram_read_pcie_state(struct amdgpu_device *adev,617static ssize_t aqua_vanjaram_read_pcie_state(struct amdgpu_device *adev,
593 void *buf, size_t max_size)618 void *buf, size_t max_size)
594{619{
@@ -596,7 +621,7 @@ static ssize_t aqua_vanjaram_read_pcie_state(struct amdgpu_device *adev, · 1 unmodified line
596 uint32_t start_addr, incrx, num_regs, szbuf;621 uint32_t start_addr, incrx, num_regs, szbuf;
597 struct amdgpu_regs_pcie_v1_0 *pcie_regs;622 struct amdgpu_regs_pcie_v1_0 *pcie_regs;
598 struct amdgpu_smn_reg_data *reg_data;623 struct amdgpu_smn_reg_data *reg_data;
599 struct pci_dev *us_pdev, *ds_pdev;624 struct pci_dev *us_pdev;
600 int aer_cap, r, n;625 int aer_cap, r, n;
601 626
602 if (!buf || !max_size)627 if (!buf || !max_size)
@@ -628,25 +653,27 @@ static ssize_t aqua_vanjaram_read_pcie_state(struct amdgpu_device *adev, · 25 unmodified lines
628 }653 }
629 }654 }
630 655
631 ds_pdev = pci_upstream_bridge(adev->pdev);656 us_pdev = aqua_vanjaram_get_us_pdev(adev);
632 us_pdev = pci_upstream_bridge(ds_pdev);657 if (us_pdev) {
658 pcie_capability_read_word(us_pdev, PCI_EXP_DEVSTA,
659 &pcie_regs->device_status);
660 pcie_capability_read_word(us_pdev, PCI_EXP_LNKSTA,
661 &pcie_regs->link_status);
662
663 aer_cap = pci_find_ext_capability(us_pdev, PCI_EXT_CAP_ID_ERR);
664 if (aer_cap) {
665 pci_read_config_dword(us_pdev,
666 aer_cap + PCI_ERR_COR_STATUS,
667 &pcie_regs->pcie_corr_err_status);
668 pci_read_config_dword(us_pdev,
669 aer_cap + PCI_ERR_UNCOR_STATUS,
670 &pcie_regs->pcie_uncorr_err_status);
671 }
633 672
634 pcie_capability_read_word(us_pdev, PCI_EXP_DEVSTA,673 pci_read_config_dword(us_pdev, PCI_PRIMARY_BUS,
635 &pcie_regs->device_status);674 &pcie_regs->sub_bus_number_latency);
636 pcie_capability_read_word(us_pdev, PCI_EXP_LNKSTA,
637 &pcie_regs->link_status);
638
639 aer_cap = pci_find_ext_capability(us_pdev, PCI_EXT_CAP_ID_ERR);
640 if (aer_cap) {
641 pci_read_config_dword(us_pdev, aer_cap + PCI_ERR_COR_STATUS,
642 &pcie_regs->pcie_corr_err_status);
643 pci_read_config_dword(us_pdev, aer_cap + PCI_ERR_UNCOR_STATUS,
644 &pcie_regs->pcie_uncorr_err_status);
645 }675 }
646 676
647 pci_read_config_dword(us_pdev, PCI_PRIMARY_BUS,
648 &pcie_regs->sub_bus_number_latency);
649
650 pcie_reg_state->common_header.structure_size = szbuf;677 pcie_reg_state->common_header.structure_size = szbuf;
651 pcie_reg_state->common_header.format_revision = 1;678 pcie_reg_state->common_header.format_revision = 1;
652 pcie_reg_state->common_header.content_revision = 0;679 pcie_reg_state->common_header.content_revision = 0;
drivers/gpu/drm/amd/amdgpu/atom.c+48−18
@@ -114,8 +114,10 @@ static uint32_t atom_iio_execute(struct atom_context *ctx, int base, · 113 unmodified lines
114 uint32_t index, uint32_t data)114 uint32_t index, uint32_t data)
115{115{
116 uint32_t temp = 0xCDCDCDCD;116 uint32_t temp = 0xCDCDCDCD;
117 int start = base;
117 118
118 while (1)119 /* IIO opcodes read up to base+3; keep within the BIOS image */
120 while (base + 3 < ctx->bios_size)
119 switch (CU8(base)) {121 switch (CU8(base)) {
120 case ATOM_IIO_NOP:122 case ATOM_IIO_NOP:
121 base++;123 base++;
@@ -180,6 +182,9 @@ static uint32_t atom_iio_execute(struct atom_context *ctx, int base, · 58 unmodified lines
180 pr_info("Unknown IIO opcode\n");182 pr_info("Unknown IIO opcode\n");
181 return 0;183 return 0;
182 }184 }
185
186 pr_info("IIO method starting at offset %d runs past BIOS image\n", start);
187 return 0;
183}188}
184 189
185static uint32_t atom_get_src_int(atom_exec_context *ctx, uint8_t attr,190static uint32_t atom_get_src_int(atom_exec_context *ctx, uint8_t attr,
@@ -1327,11 +1332,25 @@ static void atom_index_iio(struct atom_context *ctx, int base) · 1141 unmodified lines
1327 ctx->iio = kzalloc(2 * 256, GFP_KERNEL);1332 ctx->iio = kzalloc(2 * 256, GFP_KERNEL);
1328 if (!ctx->iio)1333 if (!ctx->iio)
1329 return;1334 return;
1330 while (CU8(base) == ATOM_IIO_START) {1335 while (base + 1 < ctx->bios_size && CU8(base) == ATOM_IIO_START) {
1331 ctx->iio[CU8(base + 1)] = base + 2;1336 uint8_t index = CU8(base + 1);
1337 int start = base + 2;
1332 base += 2;1338 base += 2;
1333 while (CU8(base) != ATOM_IIO_END)1339 while (base < ctx->bios_size && CU8(base) != ATOM_IIO_END) {
1334 base += atom_iio_len[CU8(base)];1340 uint8_t op = CU8(base);
1341
1342 /*
1343 * Unknown opcode: its length is unknown so the byte
1344 * stream cannot be resynced reliably.
1345 */
1346 if (op >= ARRAY_SIZE(atom_iio_len))
1347 return;
1348 base += atom_iio_len[op];
1349 }
1350 if (base >= ctx->bios_size)
1351 return;
1352 /* Only index well-formed methods, others stay 0 */
1353 ctx->iio[index] = start;
1335 base += 3;1354 base += 3;
1336 }1355 }
1337}1356}
@@ -1339,6 +1358,7 @@ static void atom_index_iio(struct atom_context *ctx, int base) · 1 unmodified line
1339static void atom_get_vbios_name(struct atom_context *ctx)1358static void atom_get_vbios_name(struct atom_context *ctx)
1340{1359{
1341 unsigned char *p_rom;1360 unsigned char *p_rom;
1361 unsigned char *p_end;
1342 unsigned char str_num;1362 unsigned char str_num;
1343 unsigned short off_to_vbios_str;1363 unsigned short off_to_vbios_str;
1344 unsigned char *c_ptr;1364 unsigned char *c_ptr;
@@ -1349,39 +1369,48 @@ static void atom_get_vbios_name(struct atom_context *ctx) · 4 unmodified lines
1349 char *back;1369 char *back;
1350 1370
1351 p_rom = ctx->bios;1371 p_rom = ctx->bios;
1372 p_end = p_rom + ctx->bios_size;
1373
1374 if (p_rom + OFFSET_TO_GET_ATOMBIOS_STRING_START + 1 >= p_end)
1375 goto no_name;
1352 1376
1353 str_num = *(p_rom + OFFSET_TO_GET_ATOMBIOS_NUMBER_OF_STRINGS);1377 str_num = *(p_rom + OFFSET_TO_GET_ATOMBIOS_NUMBER_OF_STRINGS);
1354 if (str_num != 0) {1378 if (!str_num)
1355 off_to_vbios_str =1379 goto no_name;
1356 *(unsigned short *)(p_rom + OFFSET_TO_GET_ATOMBIOS_STRING_START);
1357 1380
1358 c_ptr = (unsigned char *)(p_rom + off_to_vbios_str);1381 off_to_vbios_str =
1359 } else {1382 *(unsigned short *)(p_rom + OFFSET_TO_GET_ATOMBIOS_STRING_START);
1360 /* do not know where to find name */1383
1361 memcpy(ctx->name, na, 7);1384 c_ptr = (unsigned char *)(p_rom + off_to_vbios_str);
1362 ctx->name[7] = 0;1385 if (c_ptr >= p_end)
1363 return;1386 goto no_name;
1364 }
1365 1387
1366 /*1388 /*
1367 * skip the atombios strings, usually 41389 * skip the atombios strings, usually 4
1368 * 1st is P/N, 2nd is ASIC, 3rd is PCI type, 4th is Memory type1390 * 1st is P/N, 2nd is ASIC, 3rd is PCI type, 4th is Memory type
1369 */1391 */
1370 for (i = 0; i < str_num; i++) {1392 for (i = 0; i < str_num; i++) {
1371 while (*c_ptr != 0)1393 while (c_ptr < p_end && *c_ptr != 0)
1372 c_ptr++;1394 c_ptr++;
1373 c_ptr++;1395 c_ptr++;
1374 }1396 }
1375 1397
1376 /* skip the following 2 chars: 0x0D 0x0A */1398 /* skip the following 2 chars: 0x0D 0x0A */
1377 c_ptr += 2;1399 c_ptr += 2;
1400 if (c_ptr >= p_end)
1401 goto no_name;
1378 1402
1379 name_size = strnlen(c_ptr, STRLEN_LONG - 1);1403 name_size = strnlen(c_ptr, min(STRLEN_LONG - 1, (int)(p_end - c_ptr)));
1380 memcpy(ctx->name, c_ptr, name_size);1404 memcpy(ctx->name, c_ptr, name_size);
1381 back = ctx->name + name_size;1405 back = ctx->name + name_size;
1382 while ((*--back) == ' ')1406 while ((*--back) == ' ')
1383 ;1407 ;
1384 *(back + 1) = '\0';1408 *(back + 1) = '\0';
1409 return;
1410
1411no_name:
1412 /* do not know where to find name */
1413 strscpy(ctx->name, na, sizeof(ctx->name));
1385}1414}
1386 1415
1387static void atom_get_vbios_date(struct atom_context *ctx)1416static void atom_get_vbios_date(struct atom_context *ctx)
@@ -1553,7 +1582,7 @@ static inline void atom_print_vbios_info(struct atom_context *ctx) · 165 unmodified lines
1553 drm_info(ctx->card->dev, "ATOM BIOS: %s\n", vbios_info);1582 drm_info(ctx->card->dev, "ATOM BIOS: %s\n", vbios_info);
1554}1583}
1555 1584
1556struct atom_context *amdgpu_atom_parse(struct card_info *card, void *bios)1585struct atom_context *amdgpu_atom_parse(struct card_info *card, void *bios, uint32_t bios_size)
1557{1586{
1558 int base;1587 int base;
1559 struct atom_context *ctx =1588 struct atom_context *ctx =
@@ -1567,6 +1596,7 @@ struct atom_context *amdgpu_atom_parse(struct card_info *card, void *bios) · 7 unmodified lines
1567 1596
1568 ctx->card = card;1597 ctx->card = card;
1569 ctx->bios = bios;1598 ctx->bios = bios;
1599 ctx->bios_size = bios_size;
1570 1600
1571 if (CU16(0) != ATOM_BIOS_MAGIC) {1601 if (CU16(0) != ATOM_BIOS_MAGIC) {
1572 pr_info("Invalid BIOS magic\n");1602 pr_info("Invalid BIOS magic\n");
drivers/gpu/drm/amd/amdgpu/atom.h+2−1
@@ -133,6 +133,7 @@ struct atom_context { · 132 unmodified lines
133 struct card_info *card;133 struct card_info *card;
134 struct mutex mutex;134 struct mutex mutex;
135 void *bios;135 void *bios;
136 uint32_t bios_size;
136 uint32_t cmd_table, data_table;137 uint32_t cmd_table, data_table;
137 uint16_t *iio;138 uint16_t *iio;
138 139
@@ -160,7 +161,7 @@ struct atom_context { · 21 unmodified lines
160 161
161extern int amdgpu_atom_debug;162extern int amdgpu_atom_debug;
162 163
163struct atom_context *amdgpu_atom_parse(struct card_info *card, void *bios);164struct atom_context *amdgpu_atom_parse(struct card_info *card, void *bios, uint32_t bios_size);
164int amdgpu_atom_execute_table(struct atom_context *ctx, int index, uint32_t *params, int params_size);165int amdgpu_atom_execute_table(struct atom_context *ctx, int index, uint32_t *params, int params_size);
165int amdgpu_atom_asic_init(struct atom_context *ctx);166int amdgpu_atom_asic_init(struct atom_context *ctx);
166void amdgpu_atom_destroy(struct atom_context *ctx);167void amdgpu_atom_destroy(struct atom_context *ctx);
drivers/gpu/drm/amd/amdgpu/cik.c+0−7
@@ -1876,12 +1876,6 @@ static void cik_invalidate_hdp(struct amdgpu_device *adev, · 1875 unmodified lines
1876 }1876 }
1877}1877}
1878 1878
1879static bool cik_need_full_reset(struct amdgpu_device *adev)
1880{
1881 /* change this when we support soft reset */
1882 return true;
1883}
1884
1885static void cik_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,1879static void cik_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,
1886 uint64_t *count1)1880 uint64_t *count1)
1887{1881{
@@ -1971,7 +1965,6 @@ static const struct amdgpu_asic_funcs cik_asic_funcs = · 83 unmodified lines
1971 .get_config_memsize = &cik_get_config_memsize,1965 .get_config_memsize = &cik_get_config_memsize,
1972 .flush_hdp = &cik_flush_hdp,1966 .flush_hdp = &cik_flush_hdp,
1973 .invalidate_hdp = &cik_invalidate_hdp,1967 .invalidate_hdp = &cik_invalidate_hdp,
1974 .need_full_reset = &cik_need_full_reset,
1975 .init_doorbell_index = &legacy_doorbell_index_init,1968 .init_doorbell_index = &legacy_doorbell_index_init,
1976 .get_pcie_usage = &cik_get_pcie_usage,1969 .get_pcie_usage = &cik_get_pcie_usage,
1977 .need_reset_on_init = &cik_need_reset_on_init,1970 .need_reset_on_init = &cik_need_reset_on_init,
drivers/gpu/drm/amd/amdgpu/dce_v10_0.c+0−66
@@ -410,36 +410,6 @@ static u32 dce_v10_0_hpd_get_gpio_reg(struct amdgpu_device *adev) · 409 unmodified lines
410 return mmDC_GPIO_HPD_A;410 return mmDC_GPIO_HPD_A;
411}411}
412 412
413static bool dce_v10_0_is_display_hung(struct amdgpu_device *adev)
414{
415 u32 crtc_hung = 0;
416 u32 crtc_status[6];
417 u32 i, j, tmp;
418
419 for (i = 0; i < adev->mode_info.num_crtc; i++) {
420 tmp = RREG32(mmCRTC_CONTROL + crtc_offsets[i]);
421 if (REG_GET_FIELD(tmp, CRTC_CONTROL, CRTC_MASTER_EN)) {
422 crtc_status[i] = RREG32(mmCRTC_STATUS_HV_COUNT + crtc_offsets[i]);
423 crtc_hung |= (1 << i);
424 }
425 }
426
427 for (j = 0; j < 10; j++) {
428 for (i = 0; i < adev->mode_info.num_crtc; i++) {
429 if (crtc_hung & (1 << i)) {
430 tmp = RREG32(mmCRTC_STATUS_HV_COUNT + crtc_offsets[i]);
431 if (tmp != crtc_status[i])
432 crtc_hung &= ~(1 << i);
433 }
434 }
435 if (crtc_hung == 0)
436 return false;
437 udelay(100);
438 }
439
440 return true;
441}
442
443static void dce_v10_0_set_vga_render_state(struct amdgpu_device *adev,413static void dce_v10_0_set_vga_render_state(struct amdgpu_device *adev,
444 bool render)414 bool render)
445{415{
@@ -2956,40 +2926,6 @@ static bool dce_v10_0_is_idle(struct amdgpu_ip_block *ip_block) · 2510 unmodified lines
2956 return true;2926 return true;
2957}2927}
2958 2928
2959static bool dce_v10_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
2960{
2961 struct amdgpu_device *adev = ip_block->adev;
2962
2963 return dce_v10_0_is_display_hung(adev);
2964}
2965
2966static int dce_v10_0_soft_reset(struct amdgpu_ip_block *ip_block)
2967{
2968 u32 srbm_soft_reset = 0, tmp;
2969 struct amdgpu_device *adev = ip_block->adev;
2970
2971 if (dce_v10_0_is_display_hung(adev))
2972 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_DC_MASK;
2973
2974 if (srbm_soft_reset) {
2975 tmp = RREG32(mmSRBM_SOFT_RESET);
2976 tmp |= srbm_soft_reset;
2977 dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp);
2978 WREG32(mmSRBM_SOFT_RESET, tmp);
2979 tmp = RREG32(mmSRBM_SOFT_RESET);
2980
2981 udelay(50);
2982
2983 tmp &= ~srbm_soft_reset;
2984 WREG32(mmSRBM_SOFT_RESET, tmp);
2985 tmp = RREG32(mmSRBM_SOFT_RESET);
2986
2987 /* Wait a little for things to settle down */
2988 udelay(50);
2989 }
2990 return 0;
2991}
2992
2993static void dce_v10_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev,2929static void dce_v10_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev,
2994 int crtc,2930 int crtc,
2995 enum amdgpu_interrupt_state state)2931 enum amdgpu_interrupt_state state)
@@ -3332,8 +3268,6 @@ static const struct amd_ip_funcs dce_v10_0_ip_funcs = { · 336 unmodified lines
3332 .suspend = dce_v10_0_suspend,3268 .suspend = dce_v10_0_suspend,
3333 .resume = dce_v10_0_resume,3269 .resume = dce_v10_0_resume,
3334 .is_idle = dce_v10_0_is_idle,3270 .is_idle = dce_v10_0_is_idle,
3335 .check_soft_reset = dce_v10_0_check_soft_reset,
3336 .soft_reset = dce_v10_0_soft_reset,
3337 .set_clockgating_state = dce_v10_0_set_clockgating_state,3271 .set_clockgating_state = dce_v10_0_set_clockgating_state,
3338 .set_powergating_state = dce_v10_0_set_powergating_state,3272 .set_powergating_state = dce_v10_0_set_powergating_state,
3339};3273};
drivers/gpu/drm/amd/amdgpu/dce_v6_0.c+0−57
@@ -378,35 +378,6 @@ static u32 dce_v6_0_hpd_get_gpio_reg(struct amdgpu_device *adev) · 377 unmodified lines
378 return mmDC_GPIO_HPD_A;378 return mmDC_GPIO_HPD_A;
379}379}
380 380
381static bool dce_v6_0_is_display_hung(struct amdgpu_device *adev)
382{
383 u32 crtc_hung = 0;
384 u32 crtc_status[6];
385 u32 i, j, tmp;
386
387 for (i = 0; i < adev->mode_info.num_crtc; i++) {
388 if (RREG32(mmCRTC_CONTROL + crtc_offsets[i]) & CRTC_CONTROL__CRTC_MASTER_EN_MASK) {
389 crtc_status[i] = RREG32(mmCRTC_STATUS_HV_COUNT + crtc_offsets[i]);
390 crtc_hung |= (1 << i);
391 }
392 }
393
394 for (j = 0; j < 10; j++) {
395 for (i = 0; i < adev->mode_info.num_crtc; i++) {
396 if (crtc_hung & (1 << i)) {
397 tmp = RREG32(mmCRTC_STATUS_HV_COUNT + crtc_offsets[i]);
398 if (tmp != crtc_status[i])
399 crtc_hung &= ~(1 << i);
400 }
401 }
402 if (crtc_hung == 0)
403 return false;
404 udelay(100);
405 }
406
407 return true;
408}
409
410static void dce_v6_0_set_vga_render_state(struct amdgpu_device *adev,381static void dce_v6_0_set_vga_render_state(struct amdgpu_device *adev,
411 bool render)382 bool render)
412{383{
@@ -2901,33 +2872,6 @@ static bool dce_v6_0_is_idle(struct amdgpu_ip_block *ip_block) · 2488 unmodified lines
2901 return true;2872 return true;
2902}2873}
2903 2874
2904static int dce_v6_0_soft_reset(struct amdgpu_ip_block *ip_block)
2905{
2906 u32 srbm_soft_reset = 0, tmp;
2907 struct amdgpu_device *adev = ip_block->adev;
2908
2909 if (dce_v6_0_is_display_hung(adev))
2910 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_DC_MASK;
2911
2912 if (srbm_soft_reset) {
2913 tmp = RREG32(mmSRBM_SOFT_RESET);
2914 tmp |= srbm_soft_reset;
2915 dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp);
2916 WREG32(mmSRBM_SOFT_RESET, tmp);
2917 tmp = RREG32(mmSRBM_SOFT_RESET);
2918
2919 udelay(50);
2920
2921 tmp &= ~srbm_soft_reset;
2922 WREG32(mmSRBM_SOFT_RESET, tmp);
2923 tmp = RREG32(mmSRBM_SOFT_RESET);
2924
2925 /* Wait a little for things to settle down */
2926 udelay(50);
2927 }
2928 return 0;
2929}
2930
2931static void dce_v6_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev,2875static void dce_v6_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev,
2932 int crtc,2876 int crtc,
2933 enum amdgpu_interrupt_state state)2877 enum amdgpu_interrupt_state state)
@@ -3224,7 +3168,6 @@ static const struct amd_ip_funcs dce_v6_0_ip_funcs = { · 290 unmodified lines
3224 .suspend = dce_v6_0_suspend,3168 .suspend = dce_v6_0_suspend,
3225 .resume = dce_v6_0_resume,3169 .resume = dce_v6_0_resume,
3226 .is_idle = dce_v6_0_is_idle,3170 .is_idle = dce_v6_0_is_idle,
3227 .soft_reset = dce_v6_0_soft_reset,
3228 .set_clockgating_state = dce_v6_0_set_clockgating_state,3171 .set_clockgating_state = dce_v6_0_set_clockgating_state,
3229 .set_powergating_state = dce_v6_0_set_powergating_state,3172 .set_powergating_state = dce_v6_0_set_powergating_state,
3230};3173};
drivers/gpu/drm/amd/amdgpu/dce_v8_0.c+0−57
@@ -362,35 +362,6 @@ static u32 dce_v8_0_hpd_get_gpio_reg(struct amdgpu_device *adev) · 361 unmodified lines
362 return mmDC_GPIO_HPD_A;362 return mmDC_GPIO_HPD_A;
363}363}
364 364
365static bool dce_v8_0_is_display_hung(struct amdgpu_device *adev)
366{
367 u32 crtc_hung = 0;
368 u32 crtc_status[6];
369 u32 i, j, tmp;
370
371 for (i = 0; i < adev->mode_info.num_crtc; i++) {
372 if (RREG32(mmCRTC_CONTROL + crtc_offsets[i]) & CRTC_CONTROL__CRTC_MASTER_EN_MASK) {
373 crtc_status[i] = RREG32(mmCRTC_STATUS_HV_COUNT + crtc_offsets[i]);
374 crtc_hung |= (1 << i);
375 }
376 }
377
378 for (j = 0; j < 10; j++) {
379 for (i = 0; i < adev->mode_info.num_crtc; i++) {
380 if (crtc_hung & (1 << i)) {
381 tmp = RREG32(mmCRTC_STATUS_HV_COUNT + crtc_offsets[i]);
382 if (tmp != crtc_status[i])
383 crtc_hung &= ~(1 << i);
384 }
385 }
386 if (crtc_hung == 0)
387 return false;
388 udelay(100);
389 }
390
391 return true;
392}
393
394static void dce_v8_0_set_vga_render_state(struct amdgpu_device *adev,365static void dce_v8_0_set_vga_render_state(struct amdgpu_device *adev,
395 bool render)366 bool render)
396{367{
@@ -2873,33 +2844,6 @@ static bool dce_v8_0_is_idle(struct amdgpu_ip_block *ip_block) · 2476 unmodified lines
2873 return true;2844 return true;
2874}2845}
2875 2846
2876static int dce_v8_0_soft_reset(struct amdgpu_ip_block *ip_block)
2877{
2878 u32 srbm_soft_reset = 0, tmp;
2879 struct amdgpu_device *adev = ip_block->adev;
2880
2881 if (dce_v8_0_is_display_hung(adev))
2882 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_DC_MASK;
2883
2884 if (srbm_soft_reset) {
2885 tmp = RREG32(mmSRBM_SOFT_RESET);
2886 tmp |= srbm_soft_reset;
2887 dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp);
2888 WREG32(mmSRBM_SOFT_RESET, tmp);
2889 tmp = RREG32(mmSRBM_SOFT_RESET);
2890
2891 udelay(50);
2892
2893 tmp &= ~srbm_soft_reset;
2894 WREG32(mmSRBM_SOFT_RESET, tmp);
2895 tmp = RREG32(mmSRBM_SOFT_RESET);
2896
2897 /* Wait a little for things to settle down */
2898 udelay(50);
2899 }
2900 return 0;
2901}
2902
2903static void dce_v8_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev,2847static void dce_v8_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev,
2904 int crtc,2848 int crtc,
2905 enum amdgpu_interrupt_state state)2849 enum amdgpu_interrupt_state state)
@@ -3241,7 +3185,6 @@ static const struct amd_ip_funcs dce_v8_0_ip_funcs = { · 335 unmodified lines
3241 .suspend = dce_v8_0_suspend,3185 .suspend = dce_v8_0_suspend,
3242 .resume = dce_v8_0_resume,3186 .resume = dce_v8_0_resume,
3243 .is_idle = dce_v8_0_is_idle,3187 .is_idle = dce_v8_0_is_idle,
3244 .soft_reset = dce_v8_0_soft_reset,
3245 .set_clockgating_state = dce_v8_0_set_clockgating_state,3188 .set_clockgating_state = dce_v8_0_set_clockgating_state,
3246 .set_powergating_state = dce_v8_0_set_powergating_state,3189 .set_powergating_state = dce_v8_0_set_powergating_state,
3247};3190};
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c+2−0
@@ -7852,6 +7852,8 @@ static int gfx_v10_0_early_init(struct amdgpu_ip_block *ip_block) · 7851 unmodified lines
7852 /* init rlcg reg access ctrl */7852 /* init rlcg reg access ctrl */
7853 gfx_v10_0_init_rlcg_reg_access_ctrl(adev);7853 gfx_v10_0_init_rlcg_reg_access_ctrl(adev);
7854 7854
7855 amdgpu_init_rlc_reg_funcs(adev);
7856
7855 return gfx_v10_0_init_microcode(adev);7857 return gfx_v10_0_init_microcode(adev);
7856}7858}
7857 7859
drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c+43−291
@@ -1923,6 +1923,11 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 1922 unmodified lines
1923 if (r)1923 if (r)
1924 return r;1924 return r;
1925 1925
1926 adev->gfx.me.use_mmio_for_reset = false;
1927 adev->gfx.mec.use_mmio_for_reset = true;
1928
1929 mutex_init(&adev->gfx.mec.reset_mutex);
1930
1926 return 0;1931 return 0;
1927}1932}
1928 1933
@@ -4233,13 +4238,13 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 2304 unmodified lines
4233 return 0;4238 return 0;
4234}4239}
4235 4240
4236static int gfx_v11_0_kgq_init_queue(struct amdgpu_ring *ring, bool reset)4241static int gfx_v11_0_kgq_init_queue(struct amdgpu_ring *ring)
4237{4242{
4238 struct amdgpu_device *adev = ring->adev;4243 struct amdgpu_device *adev = ring->adev;
4239 struct v11_gfx_mqd *mqd = ring->mqd_ptr;4244 struct v11_gfx_mqd *mqd = ring->mqd_ptr;
4240 int mqd_idx = ring - &adev->gfx.gfx_ring[0];4245 int mqd_idx = ring - &adev->gfx.gfx_ring[0];
4241 4246
4242 if (!reset && !amdgpu_in_reset(adev) && !adev->in_suspend) {4247 if (!amdgpu_in_reset(adev) && !adev->in_suspend) {
4243 memset((void *)mqd, 0, sizeof(*mqd));4248 memset((void *)mqd, 0, sizeof(*mqd));
4244 mutex_lock(&adev->srbm_mutex);4249 mutex_lock(&adev->srbm_mutex);
4245 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);4250 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
@@ -4266,7 +4271,7 @@ static int gfx_v11_0_cp_async_gfx_ring_resume(struct amdgpu_device *adev) · 20 unmodified lines
4266 int r, i;4271 int r, i;
4267 4272
4268 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {4273 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
4269 r = gfx_v11_0_kgq_init_queue(&adev->gfx.gfx_ring[i], false);4274 r = gfx_v11_0_kgq_init_queue(&adev->gfx.gfx_ring[i]);
4270 if (r)4275 if (r)
4271 return r;4276 return r;
4272 }4277 }
@@ -4603,13 +4608,13 @@ static int gfx_v11_0_kiq_init_queue(struct amdgpu_ring *ring) · 330 unmodified lines
4603 return 0;4608 return 0;
4604}4609}
4605 4610
4606static int gfx_v11_0_kcq_init_queue(struct amdgpu_ring *ring, bool reset)4611static int gfx_v11_0_kcq_init_queue(struct amdgpu_ring *ring)
4607{4612{
4608 struct amdgpu_device *adev = ring->adev;4613 struct amdgpu_device *adev = ring->adev;
4609 struct v11_compute_mqd *mqd = ring->mqd_ptr;4614 struct v11_compute_mqd *mqd = ring->mqd_ptr;
4610 int mqd_idx = ring - &adev->gfx.compute_ring[0];4615 int mqd_idx = ring - &adev->gfx.compute_ring[0];
4611 4616
4612 if (!reset && !amdgpu_in_reset(adev) && !adev->in_suspend) {4617 if (!amdgpu_in_reset(adev) && !adev->in_suspend) {
4613 memset((void *)mqd, 0, sizeof(*mqd));4618 memset((void *)mqd, 0, sizeof(*mqd));
4614 mutex_lock(&adev->srbm_mutex);4619 mutex_lock(&adev->srbm_mutex);
4615 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);4620 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
@@ -4646,7 +4651,7 @@ static int gfx_v11_0_kcq_resume(struct amdgpu_device *adev) · 30 unmodified lines
4646 gfx_v11_0_cp_compute_enable(adev, true);4651 gfx_v11_0_cp_compute_enable(adev, true);
4647 4652
4648 for (i = 0; i < adev->gfx.num_compute_rings; i++) {4653 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
4649 r = gfx_v11_0_kcq_init_queue(&adev->gfx.compute_ring[i], false);4654 r = gfx_v11_0_kcq_init_queue(&adev->gfx.compute_ring[i]);
4650 if (r)4655 if (r)
4651 return r;4656 return r;
4652 }4657 }
@@ -5265,38 +5270,12 @@ static int gfx_v11_0_soft_reset(struct amdgpu_ip_block *ip_block) · 612 unmodified lines
5265 5270
5266 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);5271 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
5267 5272
5268 return gfx_v11_0_cp_resume(adev);5273 r = gfx_v11_0_cp_resume(adev);
5269}5274 if (r)
5270 5275 return r;
5271static bool gfx_v11_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
5272{
5273 int i, r;
5274 struct amdgpu_device *adev = ip_block->adev;
5275 struct amdgpu_ring *ring;
5276 long tmo = msecs_to_jiffies(1000);
5277
5278 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
5279 ring = &adev->gfx.gfx_ring[i];
5280 r = amdgpu_ring_test_ib(ring, tmo);
5281 if (r)
5282 return true;
5283 }
5284
5285 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
5286 ring = &adev->gfx.compute_ring[i];
5287 r = amdgpu_ring_test_ib(ring, tmo);
5288 if (r)
5289 return true;
5290 }
5291
5292 return false;
5293}
5294 5276
5295static int gfx_v11_0_post_soft_reset(struct amdgpu_ip_block *ip_block)
5296{
5297 struct amdgpu_device *adev = ip_block->adev;
5298 /**5277 /**
5299 * GFX soft reset will impact MES, need resume MES when do GFX soft reset5278 * GFX soft reset impacts MES, resume MES after GFX soft reset is finished
5300 */5279 */
5301 return amdgpu_mes_resume(adev, 0);5280 return amdgpu_mes_resume(adev, 0);
5302}5281}
@@ -5420,6 +5399,8 @@ static int gfx_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 117 unmodified lines
5420 5399
5421 gfx_v11_0_init_rlcg_reg_access_ctrl(adev);5400 gfx_v11_0_init_rlcg_reg_access_ctrl(adev);
5422 5401
5402 amdgpu_init_rlc_reg_funcs(adev);
5403
5423 return gfx_v11_0_init_microcode(adev);5404 return gfx_v11_0_init_microcode(adev);
5424}5405}
5425 5406
@@ -6708,22 +6689,29 @@ static int gfx_v11_0_set_priv_inst_fault_state(struct amdgpu_device *adev, · 1282 unmodified lines
6708static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev,6689static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev,
6709 struct amdgpu_iv_entry *entry)6690 struct amdgpu_iv_entry *entry)
6710{6691{
6711 u8 me_id, pipe_id, queue_id;6692 u32 doorbell_offset = entry->src_data[0] & AMDGPU_CTXID0_DOORBELL_ID_MASK;
6712 struct amdgpu_ring *ring;
6713 int i;
6714
6715 me_id = (entry->ring_id & 0x0c) >> 2;
6716 pipe_id = (entry->ring_id & 0x03) >> 0;
6717 queue_id = (entry->ring_id & 0x70) >> 4;
6718 6693
6694 /*
6695 * Try KQ first by ring_id (HW slot is authoritative). The
6696 * KMD compute_hqd_mask contract guarantees KCQ and user queues
6697 * never share a HW slot.
6698 */
6719 if (!adev->gfx.disable_kq) {6699 if (!adev->gfx.disable_kq) {
6700 u8 me_id = (entry->ring_id & 0x0c) >> 2;
6701 u8 pipe_id = (entry->ring_id & 0x03) >> 0;
6702 u8 queue_id = (entry->ring_id & 0x70) >> 4;
6703 struct amdgpu_ring *ring;
6704 int i;
6705
6720 switch (me_id) {6706 switch (me_id) {
6721 case 0:6707 case 0:
6722 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {6708 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
6723 ring = &adev->gfx.gfx_ring[i];6709 ring = &adev->gfx.gfx_ring[i];
6724 if (ring->me == me_id && ring->pipe == pipe_id &&6710 if (ring->me == me_id && ring->pipe == pipe_id &&
6725 ring->queue == queue_id)6711 ring->queue == queue_id) {
6726 drm_sched_fault(&ring->sched);6712 drm_sched_fault(&ring->sched);
6713 return;
6714 }
6727 }6715 }
6728 break;6716 break;
6729 case 1:6717 case 1:
@@ -6731,8 +6719,10 @@ static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev, · 1 unmodified line
6731 for (i = 0; i < adev->gfx.num_compute_rings; i++) {6719 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
6732 ring = &adev->gfx.compute_ring[i];6720 ring = &adev->gfx.compute_ring[i];
6733 if (ring->me == me_id && ring->pipe == pipe_id &&6721 if (ring->me == me_id && ring->pipe == pipe_id &&
6734 ring->queue == queue_id)6722 ring->queue == queue_id) {
6735 drm_sched_fault(&ring->sched);6723 drm_sched_fault(&ring->sched);
6724 return;
6725 }
6736 }6726 }
6737 break;6727 break;
6738 default:6728 default:
@@ -6740,6 +6730,11 @@ static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev, · 1 unmodified line
6740 break;6730 break;
6741 }6731 }
6742 }6732 }
6733
6734 /* No KQ matched: HW slot is a MES-scheduled user queue. */
6735 if (adev->enable_mes && doorbell_offset)
6736 amdgpu_userq_process_reset_irq(adev, entry->pasid,
6737 doorbell_offset);
6743}6738}
6744 6739
6745static int gfx_v11_0_priv_reg_irq(struct amdgpu_device *adev,6740static int gfx_v11_0_priv_reg_irq(struct amdgpu_device *adev,
@@ -6846,233 +6841,14 @@ static void gfx_v11_0_emit_mem_sync(struct amdgpu_ring *ring) · 100 unmodified lines
6846 amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */6841 amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */
6847}6842}
6848 6843
6849static bool gfx_v11_pipe_reset_support(struct amdgpu_device *adev)
6850{
6851 /* Disable the pipe reset until the CPFW fully support it.*/
6852 dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
6853 return false;
6854}
6855
6856
6857static int gfx_v11_reset_gfx_pipe(struct amdgpu_ring *ring)
6858{
6859 struct amdgpu_device *adev = ring->adev;
6860 uint32_t reset_pipe = 0, clean_pipe = 0;
6861 int r;
6862
6863 if (!gfx_v11_pipe_reset_support(adev))
6864 return -EOPNOTSUPP;
6865
6866 gfx_v11_0_set_safe_mode(adev, 0);
6867 mutex_lock(&adev->srbm_mutex);
6868 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
6869
6870 switch (ring->pipe) {
6871 case 0:
6872 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6873 PFP_PIPE0_RESET, 1);
6874 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6875 ME_PIPE0_RESET, 1);
6876 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6877 PFP_PIPE0_RESET, 0);
6878 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6879 ME_PIPE0_RESET, 0);
6880 break;
6881 case 1:
6882 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6883 PFP_PIPE1_RESET, 1);
6884 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
6885 ME_PIPE1_RESET, 1);
6886 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6887 PFP_PIPE1_RESET, 0);
6888 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
6889 ME_PIPE1_RESET, 0);
6890 break;
6891 default:
6892 break;
6893 }
6894
6895 WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
6896 WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
6897
6898 r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
6899 RS64_FW_UC_START_ADDR_LO;
6900 soc21_grbm_select(adev, 0, 0, 0, 0);
6901 mutex_unlock(&adev->srbm_mutex);
6902 gfx_v11_0_unset_safe_mode(adev, 0);
6903
6904 dev_info(adev->dev, "The ring %s pipe reset to the ME firmware start PC: %s\n", ring->name,
6905 r == 0 ? "successfully" : "failed");
6906 /* FIXME: Sometimes driver can't cache the ME firmware start PC correctly,
6907 * so the pipe reset status relies on the later gfx ring test result.
6908 */
6909 return 0;
6910}
6911
6912static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring,6844static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring,
6913 unsigned int vmid,6845 unsigned int vmid,
6914 struct amdgpu_fence *timedout_fence)6846 struct amdgpu_fence *timedout_fence)
6915{6847{
6916 struct amdgpu_device *adev = ring->adev;6848 struct amdgpu_device *adev = ring->adev;
6917 bool use_mmio = false;6849 bool use_mmio = adev->gfx.me.use_mmio_for_reset;
6918 int r;
6919 6850
6920 amdgpu_ring_reset_helper_begin(ring, timedout_fence);6851 return amdgpu_gfx_mes_reset_queue(ring, vmid, timedout_fence, use_mmio);
6921
6922 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, use_mmio, 0);
6923 if (r) {
6924
6925 dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r);
6926 r = gfx_v11_reset_gfx_pipe(ring);
6927 if (r)
6928 return r;
6929 }
6930
6931 if (use_mmio) {
6932 r = gfx_v11_0_kgq_init_queue(ring, true);
6933 if (r) {
6934 dev_err(adev->dev, "failed to init kgq\n");
6935 return r;
6936 }
6937
6938 r = amdgpu_mes_map_legacy_queue(adev, ring, 0);
6939 if (r) {
6940 dev_err(adev->dev, "failed to remap kgq\n");
6941 return r;
6942 }
6943 }
6944
6945 return amdgpu_ring_reset_helper_end(ring, timedout_fence);
6946}
6947
6948static int gfx_v11_0_reset_compute_pipe(struct amdgpu_ring *ring)
6949{
6950
6951 struct amdgpu_device *adev = ring->adev;
6952 uint32_t reset_pipe = 0, clean_pipe = 0;
6953 int r;
6954
6955 if (!gfx_v11_pipe_reset_support(adev))
6956 return -EOPNOTSUPP;
6957
6958 gfx_v11_0_set_safe_mode(adev, 0);
6959 mutex_lock(&adev->srbm_mutex);
6960 soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
6961
6962 reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
6963 clean_pipe = reset_pipe;
6964
6965 if (adev->gfx.rs64_enable) {
6966
6967 switch (ring->pipe) {
6968 case 0:
6969 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6970 MEC_PIPE0_RESET, 1);
6971 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6972 MEC_PIPE0_RESET, 0);
6973 break;
6974 case 1:
6975 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6976 MEC_PIPE1_RESET, 1);
6977 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6978 MEC_PIPE1_RESET, 0);
6979 break;
6980 case 2:
6981 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6982 MEC_PIPE2_RESET, 1);
6983 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6984 MEC_PIPE2_RESET, 0);
6985 break;
6986 case 3:
6987 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
6988 MEC_PIPE3_RESET, 1);
6989 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
6990 MEC_PIPE3_RESET, 0);
6991 break;
6992 default:
6993 break;
6994 }
6995 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe);
6996 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe);
6997 r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
6998 RS64_FW_UC_START_ADDR_LO;
6999 } else {
7000 if (ring->me == 1) {
7001 switch (ring->pipe) {
7002 case 0:
7003 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7004 MEC_ME1_PIPE0_RESET, 1);
7005 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7006 MEC_ME1_PIPE0_RESET, 0);
7007 break;
7008 case 1:
7009 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7010 MEC_ME1_PIPE1_RESET, 1);
7011 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7012 MEC_ME1_PIPE1_RESET, 0);
7013 break;
7014 case 2:
7015 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7016 MEC_ME1_PIPE2_RESET, 1);
7017 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7018 MEC_ME1_PIPE2_RESET, 0);
7019 break;
7020 case 3:
7021 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7022 MEC_ME1_PIPE3_RESET, 1);
7023 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7024 MEC_ME1_PIPE3_RESET, 0);
7025 break;
7026 default:
7027 break;
7028 }
7029 /* mec1 fw pc: CP_MEC1_INSTR_PNTR */
7030 } else {
7031 switch (ring->pipe) {
7032 case 0:
7033 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7034 MEC_ME2_PIPE0_RESET, 1);
7035 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7036 MEC_ME2_PIPE0_RESET, 0);
7037 break;
7038 case 1:
7039 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7040 MEC_ME2_PIPE1_RESET, 1);
7041 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7042 MEC_ME2_PIPE1_RESET, 0);
7043 break;
7044 case 2:
7045 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7046 MEC_ME2_PIPE2_RESET, 1);
7047 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7048 MEC_ME2_PIPE2_RESET, 0);
7049 break;
7050 case 3:
7051 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
7052 MEC_ME2_PIPE3_RESET, 1);
7053 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
7054 MEC_ME2_PIPE3_RESET, 0);
7055 break;
7056 default:
7057 break;
7058 }
7059 /* mec2 fw pc: CP:CP_MEC2_INSTR_PNTR */
7060 }
7061 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe);
7062 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe);
7063 r = RREG32(SOC15_REG_OFFSET(GC, 0, regCP_MEC1_INSTR_PNTR));
7064 }
7065
7066 soc21_grbm_select(adev, 0, 0, 0, 0);
7067 mutex_unlock(&adev->srbm_mutex);
7068 gfx_v11_0_unset_safe_mode(adev, 0);
7069
7070 dev_info(adev->dev, "The ring %s pipe resets to MEC FW start PC: %s\n", ring->name,
7071 r == 0 ? "successfully" : "failed");
7072 /*FIXME:Sometimes driver can't cache the MEC firmware start PC correctly, so the pipe
7073 * reset status relies on the compute ring test result.
7074 */
7075 return 0;
7076}6852}
7077 6853
7078static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring,6854static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring,
@@ -7080,30 +6856,8 @@ static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, · 1 unmodified line
7080 struct amdgpu_fence *timedout_fence)6856 struct amdgpu_fence *timedout_fence)
7081{6857{
7082 struct amdgpu_device *adev = ring->adev;6858 struct amdgpu_device *adev = ring->adev;
7083 int r = 0;
7084
7085 amdgpu_ring_reset_helper_begin(ring, timedout_fence);
7086
7087 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true, 0);
7088 if (r) {
7089 dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r);
7090 r = gfx_v11_0_reset_compute_pipe(ring);
7091 if (r)
7092 return r;
7093 }
7094
7095 r = gfx_v11_0_kcq_init_queue(ring, true);
7096 if (r) {
7097 dev_err(adev->dev, "fail to init kcq\n");
7098 return r;
7099 }
7100 r = amdgpu_mes_map_legacy_queue(adev, ring, 0);
7101 if (r) {
7102 dev_err(adev->dev, "failed to remap kcq\n");
7103 return r;
7104 }
7105 6859
7106 return amdgpu_ring_reset_helper_end(ring, timedout_fence);6860 return amdgpu_gfx_reset_mes_compute(adev, ring, timedout_fence, NULL, NULL, NULL);
7107}6861}
7108 6862
7109static void gfx_v11_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer *p)6863static void gfx_v11_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer *p)
@@ -7281,8 +7035,6 @@ static const struct amd_ip_funcs gfx_v11_0_ip_funcs = { · 171 unmodified lines
7281 .is_idle = gfx_v11_0_is_idle,7035 .is_idle = gfx_v11_0_is_idle,
7282 .wait_for_idle = gfx_v11_0_wait_for_idle,7036 .wait_for_idle = gfx_v11_0_wait_for_idle,
7283 .soft_reset = gfx_v11_0_soft_reset,7037 .soft_reset = gfx_v11_0_soft_reset,
7284 .check_soft_reset = gfx_v11_0_check_soft_reset,
7285 .post_soft_reset = gfx_v11_0_post_soft_reset,
7286 .set_clockgating_state = gfx_v11_0_set_clockgating_state,7038 .set_clockgating_state = gfx_v11_0_set_clockgating_state,
7287 .set_powergating_state = gfx_v11_0_set_powergating_state,7039 .set_powergating_state = gfx_v11_0_set_powergating_state,
7288 .get_clockgating_state = gfx_v11_0_get_clockgating_state,7040 .get_clockgating_state = gfx_v11_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/gfx_v12_0.c+40−211
@@ -1603,6 +1603,11 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 1602 unmodified lines
1603 if (r)1603 if (r)
1604 return r;1604 return r;
1605 1605
1606 adev->gfx.me.use_mmio_for_reset = false;
1607 adev->gfx.mec.use_mmio_for_reset = true;
1608
1609 mutex_init(&adev->gfx.mec.reset_mutex);
1610
1606 return 0;1611 return 0;
1607}1612}
1608 1613
@@ -3071,13 +3076,13 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 1462 unmodified lines
3071 return 0;3076 return 0;
3072}3077}
3073 3078
3074static int gfx_v12_0_kgq_init_queue(struct amdgpu_ring *ring, bool reset)3079static int gfx_v12_0_kgq_init_queue(struct amdgpu_ring *ring)
3075{3080{
3076 struct amdgpu_device *adev = ring->adev;3081 struct amdgpu_device *adev = ring->adev;
3077 struct v12_gfx_mqd *mqd = ring->mqd_ptr;3082 struct v12_gfx_mqd *mqd = ring->mqd_ptr;
3078 int mqd_idx = ring - &adev->gfx.gfx_ring[0];3083 int mqd_idx = ring - &adev->gfx.gfx_ring[0];
3079 3084
3080 if (!reset && !amdgpu_in_reset(adev) && !adev->in_suspend) {3085 if (!amdgpu_in_reset(adev) && !adev->in_suspend) {
3081 memset((void *)mqd, 0, sizeof(*mqd));3086 memset((void *)mqd, 0, sizeof(*mqd));
3082 mutex_lock(&adev->srbm_mutex);3087 mutex_lock(&adev->srbm_mutex);
3083 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);3088 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
@@ -3104,7 +3109,7 @@ static int gfx_v12_0_cp_async_gfx_ring_resume(struct amdgpu_device *adev) · 20 unmodified lines
3104 int i, r;3109 int i, r;
3105 3110
3106 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {3111 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
3107 r = gfx_v12_0_kgq_init_queue(&adev->gfx.gfx_ring[i], false);3112 r = gfx_v12_0_kgq_init_queue(&adev->gfx.gfx_ring[i]);
3108 if (r)3113 if (r)
3109 return r;3114 return r;
3110 }3115 }
@@ -3441,13 +3446,13 @@ static int gfx_v12_0_kiq_init_queue(struct amdgpu_ring *ring) · 330 unmodified lines
3441 return 0;3446 return 0;
3442}3447}
3443 3448
3444static int gfx_v12_0_kcq_init_queue(struct amdgpu_ring *ring, bool reset)3449static int gfx_v12_0_kcq_init_queue(struct amdgpu_ring *ring)
3445{3450{
3446 struct amdgpu_device *adev = ring->adev;3451 struct amdgpu_device *adev = ring->adev;
3447 struct v12_compute_mqd *mqd = ring->mqd_ptr;3452 struct v12_compute_mqd *mqd = ring->mqd_ptr;
3448 int mqd_idx = ring - &adev->gfx.compute_ring[0];3453 int mqd_idx = ring - &adev->gfx.compute_ring[0];
3449 3454
3450 if (!reset && !amdgpu_in_reset(adev) && !adev->in_suspend) {3455 if (!amdgpu_in_reset(adev) && !adev->in_suspend) {
3451 memset((void *)mqd, 0, sizeof(*mqd));3456 memset((void *)mqd, 0, sizeof(*mqd));
3452 mutex_lock(&adev->srbm_mutex);3457 mutex_lock(&adev->srbm_mutex);
3453 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);3458 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
@@ -3485,7 +3490,7 @@ static int gfx_v12_0_kcq_resume(struct amdgpu_device *adev) · 31 unmodified lines
3485 gfx_v12_0_cp_compute_enable(adev, true);3490 gfx_v12_0_cp_compute_enable(adev, true);
3486 3491
3487 for (i = 0; i < adev->gfx.num_compute_rings; i++) {3492 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
3488 r = gfx_v12_0_kcq_init_queue(&adev->gfx.compute_ring[i], false);3493 r = gfx_v12_0_kcq_init_queue(&adev->gfx.compute_ring[i]);
3489 if (r)3494 if (r)
3490 return r;3495 return r;
3491 }3496 }
@@ -3986,6 +3991,8 @@ static int gfx_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 494 unmodified lines
3986 3991
3987 gfx_v12_0_init_rlcg_reg_access_ctrl(adev);3992 gfx_v12_0_init_rlcg_reg_access_ctrl(adev);
3988 3993
3994 amdgpu_init_rlc_reg_funcs(adev);
3995
3989 return gfx_v12_0_init_microcode(adev);3996 return gfx_v12_0_init_microcode(adev);
3990}3997}
3991 3998
@@ -5025,22 +5032,30 @@ static int gfx_v12_0_set_priv_inst_fault_state(struct amdgpu_device *adev, · 1033 unmodified lines
5025static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev,5032static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev,
5026 struct amdgpu_iv_entry *entry)5033 struct amdgpu_iv_entry *entry)
5027{5034{
5028 u8 me_id, pipe_id, queue_id;5035 u32 doorbell_offset = entry->src_data[0] & AMDGPU_CTXID0_DOORBELL_ID_MASK;
5029 struct amdgpu_ring *ring;
5030 int i;
5031
5032 me_id = (entry->ring_id & 0x0c) >> 2;
5033 pipe_id = (entry->ring_id & 0x03) >> 0;
5034 queue_id = (entry->ring_id & 0x70) >> 4;
5035 5036
5037 /*
5038 * Try KQ first by ring_id; UQ as fallback. KCQ and UQ never share
5039 * a HW slot (compute_hqd_mask contract).
5040 */
5036 if (!adev->gfx.disable_kq) {5041 if (!adev->gfx.disable_kq) {
5042 u8 me_id, pipe_id, queue_id;
5043 struct amdgpu_ring *ring;
5044 int i;
5045
5046 me_id = (entry->ring_id & 0x0c) >> 2;
5047 pipe_id = (entry->ring_id & 0x03) >> 0;
5048 queue_id = (entry->ring_id & 0x70) >> 4;
5049
5037 switch (me_id) {5050 switch (me_id) {
5038 case 0:5051 case 0:
5039 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {5052 for (i = 0; i < adev->gfx.num_gfx_rings; i++) {
5040 ring = &adev->gfx.gfx_ring[i];5053 ring = &adev->gfx.gfx_ring[i];
5041 if (ring->me == me_id && ring->pipe == pipe_id &&5054 if (ring->me == me_id && ring->pipe == pipe_id &&
5042 ring->queue == queue_id)5055 ring->queue == queue_id) {
5043 drm_sched_fault(&ring->sched);5056 drm_sched_fault(&ring->sched);
5057 return;
5058 }
5044 }5059 }
5045 break;5060 break;
5046 case 1:5061 case 1:
@@ -5048,8 +5063,10 @@ static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev, · 1 unmodified line
5048 for (i = 0; i < adev->gfx.num_compute_rings; i++) {5063 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
5049 ring = &adev->gfx.compute_ring[i];5064 ring = &adev->gfx.compute_ring[i];
5050 if (ring->me == me_id && ring->pipe == pipe_id &&5065 if (ring->me == me_id && ring->pipe == pipe_id &&
5051 ring->queue == queue_id)5066 ring->queue == queue_id) {
5052 drm_sched_fault(&ring->sched);5067 drm_sched_fault(&ring->sched);
5068 return;
5069 }
5053 }5070 }
5054 break;5071 break;
5055 default:5072 default:
@@ -5057,6 +5074,11 @@ static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev, · 1 unmodified line
5057 break;5074 break;
5058 }5075 }
5059 }5076 }
5077
5078 /* No KQ matched: HW slot is a MES-scheduled user queue. */
5079 if (adev->enable_mes && doorbell_offset)
5080 amdgpu_userq_process_reset_irq(adev, entry->pasid,
5081 doorbell_offset);
5060}5082}
5061 5083
5062static int gfx_v12_0_priv_reg_irq(struct amdgpu_device *adev,5084static int gfx_v12_0_priv_reg_irq(struct amdgpu_device *adev,
@@ -5261,185 +5283,14 @@ static void gfx_v12_ip_dump(struct amdgpu_ip_block *ip_block) · 198 unmodified lines
5261 amdgpu_gfx_off_ctrl(adev, true);5283 amdgpu_gfx_off_ctrl(adev, true);
5262}5284}
5263 5285
5264static bool gfx_v12_pipe_reset_support(struct amdgpu_device *adev)
5265{
5266 /* Disable the pipe reset until the CPFW fully support it.*/
5267 dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
5268 return false;
5269}
5270
5271static int gfx_v12_reset_gfx_pipe(struct amdgpu_ring *ring)
5272{
5273 struct amdgpu_device *adev = ring->adev;
5274 uint32_t reset_pipe = 0, clean_pipe = 0;
5275 int r;
5276
5277 if (!gfx_v12_pipe_reset_support(adev))
5278 return -EOPNOTSUPP;
5279
5280 gfx_v12_0_set_safe_mode(adev, 0);
5281 mutex_lock(&adev->srbm_mutex);
5282 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5283
5284 switch (ring->pipe) {
5285 case 0:
5286 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5287 PFP_PIPE0_RESET, 1);
5288 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5289 ME_PIPE0_RESET, 1);
5290 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5291 PFP_PIPE0_RESET, 0);
5292 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5293 ME_PIPE0_RESET, 0);
5294 break;
5295 case 1:
5296 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5297 PFP_PIPE1_RESET, 1);
5298 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
5299 ME_PIPE1_RESET, 1);
5300 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5301 PFP_PIPE1_RESET, 0);
5302 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
5303 ME_PIPE1_RESET, 0);
5304 break;
5305 default:
5306 break;
5307 }
5308
5309 WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
5310 WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
5311
5312 r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
5313 RS64_FW_UC_START_ADDR_LO;
5314 soc24_grbm_select(adev, 0, 0, 0, 0);
5315 mutex_unlock(&adev->srbm_mutex);
5316 gfx_v12_0_unset_safe_mode(adev, 0);
5317
5318 dev_info(adev->dev, "The ring %s pipe reset: %s\n", ring->name,
5319 r == 0 ? "successfully" : "failed");
5320 /* Sometimes the ME start pc counter can't cache correctly, so the
5321 * PC check only as a reference and pipe reset result rely on the
5322 * later ring test.
5323 */
5324 return 0;
5325}
5326
5327static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring,5286static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring,
5328 unsigned int vmid,5287 unsigned int vmid,
5329 struct amdgpu_fence *timedout_fence)5288 struct amdgpu_fence *timedout_fence)
5330{5289{
5331 struct amdgpu_device *adev = ring->adev;5290 struct amdgpu_device *adev = ring->adev;
5332 bool use_mmio = false;5291 bool use_mmio = adev->gfx.me.use_mmio_for_reset;
5333 int r;
5334
5335 amdgpu_ring_reset_helper_begin(ring, timedout_fence);
5336
5337 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, use_mmio, 0);
5338 if (r) {
5339 dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r);
5340 r = gfx_v12_reset_gfx_pipe(ring);
5341 if (r)
5342 return r;
5343 }
5344
5345 if (use_mmio) {
5346 r = gfx_v12_0_kgq_init_queue(ring, true);
5347 if (r) {
5348 dev_err(adev->dev, "failed to init kgq\n");
5349 return r;
5350 }
5351
5352 r = amdgpu_mes_map_legacy_queue(adev, ring, 0);
5353 if (r) {
5354 dev_err(adev->dev, "failed to remap kgq\n");
5355 return r;
5356 }
5357 }
5358
5359 return amdgpu_ring_reset_helper_end(ring, timedout_fence);
5360}
5361
5362static int gfx_v12_0_reset_compute_pipe(struct amdgpu_ring *ring)
5363{
5364 struct amdgpu_device *adev = ring->adev;
5365 uint32_t reset_pipe = 0, clean_pipe = 0;
5366 int r = 0;
5367
5368 if (!gfx_v12_pipe_reset_support(adev))
5369 return -EOPNOTSUPP;
5370
5371 gfx_v12_0_set_safe_mode(adev, 0);
5372 mutex_lock(&adev->srbm_mutex);
5373 soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5374
5375 reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
5376 clean_pipe = reset_pipe;
5377 5292
5378 if (adev->gfx.rs64_enable) {5293 return amdgpu_gfx_mes_reset_queue(ring, vmid, timedout_fence, use_mmio);
5379 switch (ring->pipe) {
5380 case 0:
5381 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5382 MEC_PIPE0_RESET, 1);
5383 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5384 MEC_PIPE0_RESET, 0);
5385 break;
5386 case 1:
5387 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5388 MEC_PIPE1_RESET, 1);
5389 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5390 MEC_PIPE1_RESET, 0);
5391 break;
5392 case 2:
5393 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5394 MEC_PIPE2_RESET, 1);
5395 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5396 MEC_PIPE2_RESET, 0);
5397 break;
5398 case 3:
5399 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL,
5400 MEC_PIPE3_RESET, 1);
5401 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL,
5402 MEC_PIPE3_RESET, 0);
5403 break;
5404 default:
5405 break;
5406 }
5407 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe);
5408 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe);
5409 r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
5410 RS64_FW_UC_START_ADDR_LO;
5411 } else {
5412 switch (ring->pipe) {
5413 case 0:
5414 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
5415 MEC_ME1_PIPE0_RESET, 1);
5416 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
5417 MEC_ME1_PIPE0_RESET, 0);
5418 break;
5419 case 1:
5420 reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL,
5421 MEC_ME1_PIPE1_RESET, 1);
5422 clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL,
5423 MEC_ME1_PIPE1_RESET, 0);
5424 break;
5425 default:
5426 break;
5427 }
5428 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe);
5429 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe);
5430 /* Doesn't find the F32 MEC instruction pointer register, and suppose
5431 * the driver won't run into the F32 mode.
5432 */
5433 }
5434
5435 soc24_grbm_select(adev, 0, 0, 0, 0);
5436 mutex_unlock(&adev->srbm_mutex);
5437 gfx_v12_0_unset_safe_mode(adev, 0);
5438
5439 dev_info(adev->dev, "The ring %s pipe resets: %s\n", ring->name,
5440 r == 0 ? "successfully" : "failed");
5441 /* Need the ring test to verify the pipe reset result.*/
5442 return 0;
5443}5294}
5444 5295
5445static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring,5296static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring,
@@ -5447,30 +5298,8 @@ static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, · 1 unmodified line
5447 struct amdgpu_fence *timedout_fence)5298 struct amdgpu_fence *timedout_fence)
5448{5299{
5449 struct amdgpu_device *adev = ring->adev;5300 struct amdgpu_device *adev = ring->adev;
5450 int r;
5451
5452 amdgpu_ring_reset_helper_begin(ring, timedout_fence);
5453
5454 r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true, 0);
5455 if (r) {
5456 dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r);
5457 r = gfx_v12_0_reset_compute_pipe(ring);
5458 if (r)
5459 return r;
5460 }
5461
5462 r = gfx_v12_0_kcq_init_queue(ring, true);
5463 if (r) {
5464 dev_err(adev->dev, "failed to init kcq\n");
5465 return r;
5466 }
5467 r = amdgpu_mes_map_legacy_queue(adev, ring, 0);
5468 if (r) {
5469 dev_err(adev->dev, "failed to remap kcq\n");
5470 return r;
5471 }
5472 5301
5473 return amdgpu_ring_reset_helper_end(ring, timedout_fence);5302 return amdgpu_gfx_reset_mes_compute(adev, ring, timedout_fence, NULL, NULL, NULL);
5474}5303}
5475 5304
5476static void gfx_v12_0_ring_begin_use(struct amdgpu_ring *ring)5305static void gfx_v12_0_ring_begin_use(struct amdgpu_ring *ring)
drivers/gpu/drm/amd/amdgpu/gfx_v12_1.c+4−0
@@ -1287,6 +1287,8 @@ static int gfx_v12_1_sw_init(struct amdgpu_ip_block *ip_block) · 1286 unmodified lines
1287 if (r)1287 if (r)
1288 return r;1288 return r;
1289 1289
1290 mutex_init(&adev->gfx.mec.reset_mutex);
1291
1290 return 0;1292 return 0;
1291}1293}
1292 1294
@@ -3004,6 +3006,8 @@ static int gfx_v12_1_early_init(struct amdgpu_ip_block *ip_block) · 1711 unmodified lines
3004 3006
3005 gfx_v12_1_init_rlcg_reg_access_ctrl(adev);3007 gfx_v12_1_init_rlcg_reg_access_ctrl(adev);
3006 3008
3009 amdgpu_init_rlc_reg_funcs(adev);
3010
3007 return gfx_v12_1_init_microcode(adev);3011 return gfx_v12_1_init_microcode(adev);
3008}3012}
3009 3013
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c+64−160
@@ -1487,7 +1487,14 @@ static int gfx_v8_0_do_edc_gpr_workarounds(struct amdgpu_device *adev) · 1486 unmodified lines
1487 1487
1488 /* bail if the compute ring is not ready */1488 /* bail if the compute ring is not ready */
1489 if (!ring->sched.ready)1489 if (!ring->sched.ready)
1490 return 0;1490 return -EBUSY;
1491
1492 if (amdgpu_in_reset(adev)) {
1493 /* Set preempt condition to execute IB */
1494 amdgpu_ring_set_preempt_cond_exec(ring, true);
1495 /* Flush HDP cache so the GPU can see the updated COND_EXEC value */
1496 amdgpu_device_flush_hdp(adev, NULL);
1497 }
1491 1498
1492 tmp = RREG32(mmGB_EDC_MODE);1499 tmp = RREG32(mmGB_EDC_MODE);
1493 WREG32(mmGB_EDC_MODE, 0);1500 WREG32(mmGB_EDC_MODE, 0);
@@ -2028,6 +2035,11 @@ static int gfx_v8_0_sw_init(struct amdgpu_ip_block *ip_block) · 534 unmodified lines
2028 adev->gfx.compute_supported_reset =2035 adev->gfx.compute_supported_reset =
2029 amdgpu_get_soft_full_reset_mask(&adev->gfx.compute_ring[0]);2036 amdgpu_get_soft_full_reset_mask(&adev->gfx.compute_ring[0]);
2030 2037
2038 if (!amdgpu_sriov_vf(adev) && !adev->debug_disable_ip_block_soft_reset) {
2039 adev->gfx.compute_supported_reset |= AMDGPU_RESET_TYPE_IP_BLOCK_SOFT_RESET;
2040 adev->gfx.gfx_supported_reset |= AMDGPU_RESET_TYPE_IP_BLOCK_SOFT_RESET;
2041 }
2042
2031 return 0;2043 return 0;
2032}2044}
2033 2045
@@ -4703,12 +4715,14 @@ static int gfx_v8_0_cp_test_all_rings(struct amdgpu_device *adev) · 2669 unmodified lines
4703 if (r)4715 if (r)
4704 return r;4716 return r;
4705 4717
4718 r = 0;
4719
4706 for (i = 0; i < adev->gfx.num_compute_rings; i++) {4720 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
4707 ring = &adev->gfx.compute_ring[i];4721 ring = &adev->gfx.compute_ring[i];
4708 amdgpu_ring_test_helper(ring);4722 r |= amdgpu_ring_test_helper(ring);
4709 }4723 }
4710 4724
4711 return 0;4725 return r;
4712}4726}
4713 4727
4714static int gfx_v8_0_cp_resume(struct amdgpu_device *adev)4728static int gfx_v8_0_cp_resume(struct amdgpu_device *adev)
@@ -4868,14 +4882,12 @@ static int gfx_v8_0_hw_fini(struct amdgpu_ip_block *ip_block) · 153 unmodified lines
4868 }4882 }
4869 4883
4870 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);4884 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
4871 if (!gfx_v8_0_wait_for_idle(ip_block))4885 if (!amdgpu_in_reset(adev) && gfx_v8_0_wait_for_idle(ip_block))
4872 gfx_v8_0_cp_enable(adev, false);
4873 else
4874 pr_err("cp is busy, skip halt cp\n");4886 pr_err("cp is busy, skip halt cp\n");
4875 if (!gfx_v8_0_wait_for_rlc_idle(adev))4887 if (!amdgpu_in_reset(adev) && gfx_v8_0_wait_for_rlc_idle(adev))
4876 adev->gfx.rlc.funcs->stop(adev);4888 pr_err("rlc is busy\n");
4877 else4889 gfx_v8_0_cp_enable(adev, false);
4878 pr_err("rlc is busy, skip halt rlc\n");4890 adev->gfx.rlc.funcs->stop(adev);
4879 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);4891 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
4880 4892
4881 return 0;4893 return 0;
@@ -4891,128 +4903,49 @@ static int gfx_v8_0_resume(struct amdgpu_ip_block *ip_block) · 9 unmodified lines
4891 return gfx_v8_0_hw_init(ip_block);4903 return gfx_v8_0_hw_init(ip_block);
4892}4904}
4893 4905
4894static bool gfx_v8_0_check_soft_reset(struct amdgpu_ip_block *ip_block)4906static int gfx_v8_0_soft_reset(struct amdgpu_ip_block *ip_block)
4895{4907{
4896 struct amdgpu_device *adev = ip_block->adev;4908 struct amdgpu_device *adev = ip_block->adev;
4897 u32 grbm_soft_reset = 0, srbm_soft_reset = 0;4909 u32 grbm_soft_reset = 0, srbm_soft_reset = 0;
4898 u32 tmp;4910 u32 tmp;
4911 int i;
4912 int r;
4899 4913
4900 /* GRBM_STATUS */4914 grbm_soft_reset =
4901 tmp = RREG32(mmGRBM_STATUS);4915 REG_SET_FIELD(0, GRBM_SOFT_RESET, SOFT_RESET_RLC, 1) |
4902 if (tmp & (GRBM_STATUS__PA_BUSY_MASK | GRBM_STATUS__SC_BUSY_MASK |4916 REG_SET_FIELD(0, GRBM_SOFT_RESET, SOFT_RESET_GFX, 1) |
4903 GRBM_STATUS__BCI_BUSY_MASK | GRBM_STATUS__SX_BUSY_MASK |4917 REG_SET_FIELD(0, GRBM_SOFT_RESET, SOFT_RESET_CP, 1) |
4904 GRBM_STATUS__TA_BUSY_MASK | GRBM_STATUS__VGT_BUSY_MASK |4918 REG_SET_FIELD(0, GRBM_SOFT_RESET, SOFT_RESET_CPF, 1) |
4905 GRBM_STATUS__DB_BUSY_MASK | GRBM_STATUS__CB_BUSY_MASK |4919 REG_SET_FIELD(0, GRBM_SOFT_RESET, SOFT_RESET_CPC, 1) |
4906 GRBM_STATUS__GDS_BUSY_MASK | GRBM_STATUS__SPI_BUSY_MASK |4920 REG_SET_FIELD(0, GRBM_SOFT_RESET, SOFT_RESET_CPG, 1);
4907 GRBM_STATUS__IA_BUSY_MASK | GRBM_STATUS__IA_BUSY_NO_DMA_MASK |
4908 GRBM_STATUS__CP_BUSY_MASK | GRBM_STATUS__CP_COHERENCY_BUSY_MASK)) {
4909 grbm_soft_reset = REG_SET_FIELD(grbm_soft_reset,
4910 GRBM_SOFT_RESET, SOFT_RESET_CP, 1);
4911 grbm_soft_reset = REG_SET_FIELD(grbm_soft_reset,
4912 GRBM_SOFT_RESET, SOFT_RESET_GFX, 1);
4913 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset,
4914 SRBM_SOFT_RESET, SOFT_RESET_GRBM, 1);
4915 }
4916
4917 /* GRBM_STATUS2 */
4918 tmp = RREG32(mmGRBM_STATUS2);
4919 if (REG_GET_FIELD(tmp, GRBM_STATUS2, RLC_BUSY))
4920 grbm_soft_reset = REG_SET_FIELD(grbm_soft_reset,
4921 GRBM_SOFT_RESET, SOFT_RESET_RLC, 1);
4922
4923 if (REG_GET_FIELD(tmp, GRBM_STATUS2, CPF_BUSY) ||
4924 REG_GET_FIELD(tmp, GRBM_STATUS2, CPC_BUSY) ||
4925 REG_GET_FIELD(tmp, GRBM_STATUS2, CPG_BUSY)) {
4926 grbm_soft_reset = REG_SET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET,
4927 SOFT_RESET_CPF, 1);
4928 grbm_soft_reset = REG_SET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET,
4929 SOFT_RESET_CPC, 1);
4930 grbm_soft_reset = REG_SET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET,
4931 SOFT_RESET_CPG, 1);
4932 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET,
4933 SOFT_RESET_GRBM, 1);
4934 }
4935
4936 /* SRBM_STATUS */
4937 tmp = RREG32(mmSRBM_STATUS);
4938 if (REG_GET_FIELD(tmp, SRBM_STATUS, GRBM_RQ_PENDING))
4939 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset,
4940 SRBM_SOFT_RESET, SOFT_RESET_GRBM, 1);
4941 if (REG_GET_FIELD(tmp, SRBM_STATUS, SEM_BUSY))
4942 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset,
4943 SRBM_SOFT_RESET, SOFT_RESET_SEM, 1);
4944
4945 if (grbm_soft_reset || srbm_soft_reset) {
4946 adev->gfx.grbm_soft_reset = grbm_soft_reset;
4947 adev->gfx.srbm_soft_reset = srbm_soft_reset;
4948 return true;
4949 } else {
4950 adev->gfx.grbm_soft_reset = 0;
4951 adev->gfx.srbm_soft_reset = 0;
4952 return false;
4953 }
4954}
4955
4956static int gfx_v8_0_pre_soft_reset(struct amdgpu_ip_block *ip_block)
4957{
4958 struct amdgpu_device *adev = ip_block->adev;
4959 u32 grbm_soft_reset = 0;
4960
4961 if ((!adev->gfx.grbm_soft_reset) &&
4962 (!adev->gfx.srbm_soft_reset))
4963 return 0;
4964
4965 grbm_soft_reset = adev->gfx.grbm_soft_reset;
4966
4967 /* stop the rlc */
4968 adev->gfx.rlc.funcs->stop(adev);
4969 4921
4970 if (REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CP) ||4922 srbm_soft_reset =
4971 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_GFX))4923 REG_SET_FIELD(0, SRBM_SOFT_RESET, SOFT_RESET_GRBM, 1) |
4972 /* Disable GFX parsing/prefetching */4924 REG_SET_FIELD(0, SRBM_SOFT_RESET, SOFT_RESET_SEM, 1);
4973 gfx_v8_0_cp_gfx_enable(adev, false);
4974 4925
4975 if (REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CP) ||4926 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
4976 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CPF) ||4927 struct amdgpu_ring *ring = &adev->gfx.compute_ring[i];
4977 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CPC) ||
4978 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CPG)) {
4979 int i;
4980 4928
4981 for (i = 0; i < adev->gfx.num_compute_rings; i++) {4929 mutex_lock(&adev->srbm_mutex);
4982 struct amdgpu_ring *ring = &adev->gfx.compute_ring[i];4930 vi_srbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
4931 gfx_v8_0_deactivate_hqd(adev, 2);
4932 vi_srbm_select(adev, 0, 0, 0, 0);
4933 mutex_unlock(&adev->srbm_mutex);
4983 4934
4984 mutex_lock(&adev->srbm_mutex);4935 udelay(50);
4985 vi_srbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
4986 gfx_v8_0_deactivate_hqd(adev, 2);
4987 vi_srbm_select(adev, 0, 0, 0, 0);
4988 mutex_unlock(&adev->srbm_mutex);
4989 }
4990 /* Disable MEC parsing/prefetching */
4991 gfx_v8_0_cp_compute_enable(adev, false);
4992 }4936 }
4993 4937
4994 return 0;4938 ip_block->version->funcs->set_clockgating_state(ip_block, AMD_CG_STATE_UNGATE);
4995}4939 ip_block->version->funcs->set_powergating_state(ip_block, AMD_PG_STATE_UNGATE);
4996 4940 ip_block->version->funcs->suspend(ip_block);
4997static int gfx_v8_0_soft_reset(struct amdgpu_ip_block *ip_block)
4998{
4999 struct amdgpu_device *adev = ip_block->adev;
5000 u32 grbm_soft_reset = 0, srbm_soft_reset = 0;
5001 u32 tmp;
5002
5003 if ((!adev->gfx.grbm_soft_reset) &&
5004 (!adev->gfx.srbm_soft_reset))
5005 return 0;
5006
5007 grbm_soft_reset = adev->gfx.grbm_soft_reset;
5008 srbm_soft_reset = adev->gfx.srbm_soft_reset;
5009 4941
5010 if (grbm_soft_reset || srbm_soft_reset) {4942 if (grbm_soft_reset || srbm_soft_reset) {
5011 tmp = RREG32(mmGMCON_DEBUG);4943 tmp = RREG32(mmGMCON_DEBUG);
5012 tmp = REG_SET_FIELD(tmp, GMCON_DEBUG, GFX_STALL, 1);4944 tmp = REG_SET_FIELD(tmp, GMCON_DEBUG, GFX_STALL, 1);
5013 tmp = REG_SET_FIELD(tmp, GMCON_DEBUG, GFX_CLEAR, 1);4945 tmp = REG_SET_FIELD(tmp, GMCON_DEBUG, GFX_CLEAR, 1);
5014 WREG32(mmGMCON_DEBUG, tmp);4946 WREG32(mmGMCON_DEBUG, tmp);
5015 udelay(50);4947
4948 udelay(100);
5016 }4949 }
5017 4950
5018 if (grbm_soft_reset) {4951 if (grbm_soft_reset) {
@@ -5022,11 +4955,13 @@ static int gfx_v8_0_soft_reset(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
5022 WREG32(mmGRBM_SOFT_RESET, tmp);4955 WREG32(mmGRBM_SOFT_RESET, tmp);
5023 tmp = RREG32(mmGRBM_SOFT_RESET);4956 tmp = RREG32(mmGRBM_SOFT_RESET);
5024 4957
5025 udelay(50);4958 udelay(100);
5026 4959
5027 tmp &= ~grbm_soft_reset;4960 tmp &= ~grbm_soft_reset;
5028 WREG32(mmGRBM_SOFT_RESET, tmp);4961 WREG32(mmGRBM_SOFT_RESET, tmp);
5029 tmp = RREG32(mmGRBM_SOFT_RESET);4962 tmp = RREG32(mmGRBM_SOFT_RESET);
4963
4964 udelay(100);
5030 }4965 }
5031 4966
5032 if (srbm_soft_reset) {4967 if (srbm_soft_reset) {
@@ -5036,11 +4971,13 @@ static int gfx_v8_0_soft_reset(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
5036 WREG32(mmSRBM_SOFT_RESET, tmp);4971 WREG32(mmSRBM_SOFT_RESET, tmp);
5037 tmp = RREG32(mmSRBM_SOFT_RESET);4972 tmp = RREG32(mmSRBM_SOFT_RESET);
5038 4973
5039 udelay(50);4974 udelay(100);
5040 4975
5041 tmp &= ~srbm_soft_reset;4976 tmp &= ~srbm_soft_reset;
5042 WREG32(mmSRBM_SOFT_RESET, tmp);4977 WREG32(mmSRBM_SOFT_RESET, tmp);
5043 tmp = RREG32(mmSRBM_SOFT_RESET);4978 tmp = RREG32(mmSRBM_SOFT_RESET);
4979
4980 udelay(100);
5044 }4981 }
5045 4982
5046 if (grbm_soft_reset || srbm_soft_reset) {4983 if (grbm_soft_reset || srbm_soft_reset) {
@@ -5051,48 +4988,15 @@ static int gfx_v8_0_soft_reset(struct amdgpu_ip_block *ip_block) · 4 unmodified lines
5051 }4988 }
5052 4989
5053 /* Wait a little for things to settle down */4990 /* Wait a little for things to settle down */
5054 udelay(50);4991 udelay(100);
5055 4992
5056 return 0;4993 r = ip_block->version->funcs->resume(ip_block);
5057}4994 r |= ip_block->version->funcs->late_init(ip_block);
5058 4995 if (r)
5059static int gfx_v8_0_post_soft_reset(struct amdgpu_ip_block *ip_block)4996 return r;
5060{
5061 struct amdgpu_device *adev = ip_block->adev;
5062 u32 grbm_soft_reset = 0;
5063
5064 if ((!adev->gfx.grbm_soft_reset) &&
5065 (!adev->gfx.srbm_soft_reset))
5066 return 0;
5067
5068 grbm_soft_reset = adev->gfx.grbm_soft_reset;
5069
5070 if (REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CP) ||
5071 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CPF) ||
5072 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CPC) ||
5073 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CPG)) {
5074 int i;
5075
5076 for (i = 0; i < adev->gfx.num_compute_rings; i++) {
5077 struct amdgpu_ring *ring = &adev->gfx.compute_ring[i];
5078
5079 mutex_lock(&adev->srbm_mutex);
5080 vi_srbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
5081 gfx_v8_0_deactivate_hqd(adev, 2);
5082 vi_srbm_select(adev, 0, 0, 0, 0);
5083 mutex_unlock(&adev->srbm_mutex);
5084 }
5085 gfx_v8_0_kiq_resume(adev);
5086 gfx_v8_0_kcq_resume(adev);
5087 }
5088
5089 if (REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_CP) ||
5090 REG_GET_FIELD(grbm_soft_reset, GRBM_SOFT_RESET, SOFT_RESET_GFX))
5091 gfx_v8_0_cp_gfx_resume(adev);
5092 4997
5093 gfx_v8_0_cp_test_all_rings(adev);4998 ip_block->version->funcs->set_clockgating_state(ip_block, AMD_CG_STATE_GATE);
5094 4999 ip_block->version->funcs->set_powergating_state(ip_block, AMD_PG_STATE_GATE);
5095 adev->gfx.rlc.funcs->start(adev);
5096 5000
5097 return 0;5001 return 0;
5098}5002}
@@ -6859,10 +6763,7 @@ static const struct amd_ip_funcs gfx_v8_0_ip_funcs = { · 1760 unmodified lines
6859 .resume = gfx_v8_0_resume,6763 .resume = gfx_v8_0_resume,
6860 .is_idle = gfx_v8_0_is_idle,6764 .is_idle = gfx_v8_0_is_idle,
6861 .wait_for_idle = gfx_v8_0_wait_for_idle,6765 .wait_for_idle = gfx_v8_0_wait_for_idle,
6862 .check_soft_reset = gfx_v8_0_check_soft_reset,
6863 .pre_soft_reset = gfx_v8_0_pre_soft_reset,
6864 .soft_reset = gfx_v8_0_soft_reset,6766 .soft_reset = gfx_v8_0_soft_reset,
6865 .post_soft_reset = gfx_v8_0_post_soft_reset,
6866 .set_clockgating_state = gfx_v8_0_set_clockgating_state,6767 .set_clockgating_state = gfx_v8_0_set_clockgating_state,
6867 .set_powergating_state = gfx_v8_0_set_powergating_state,6768 .set_powergating_state = gfx_v8_0_set_powergating_state,
6868 .get_clockgating_state = gfx_v8_0_get_clockgating_state,6769 .get_clockgating_state = gfx_v8_0_get_clockgating_state,
@@ -6923,10 +6824,12 @@ static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_compute = { · 54 unmodified lines
6923 .get_wptr = gfx_v8_0_ring_get_wptr_compute,6824 .get_wptr = gfx_v8_0_ring_get_wptr_compute,
6924 .set_wptr = gfx_v8_0_ring_set_wptr_compute,6825 .set_wptr = gfx_v8_0_ring_set_wptr_compute,
6925 .emit_frame_size =6826 .emit_frame_size =
6827 5 + /* gfx_v8_0_ring_emit_init_cond_exec (from amdgpu_ib_schedule) */
6926 20 + /* gfx_v8_0_ring_emit_gds_switch */6828 20 + /* gfx_v8_0_ring_emit_gds_switch */
6927 7 + /* gfx_v8_0_ring_emit_hdp_flush */6829 7 + /* gfx_v8_0_ring_emit_hdp_flush */
6928 5 + /* hdp_invalidate */6830 5 + /* hdp_invalidate */
6929 7 + /* gfx_v8_0_ring_emit_pipeline_sync */6831 7 + /* gfx_v8_0_ring_emit_pipeline_sync */
6832 5 + /* gfx_v8_0_ring_emit_init_cond_exec (from amdgpu_vm_flush) */
6930 VI_FLUSH_GPU_TLB_NUM_WREG * 5 + 7 + /* gfx_v8_0_ring_emit_vm_flush */6833 VI_FLUSH_GPU_TLB_NUM_WREG * 5 + 7 + /* gfx_v8_0_ring_emit_vm_flush */
6931 7 + 7 + 7 + /* gfx_v8_0_ring_emit_fence_compute x3 for user fence, vm fence */6834 7 + 7 + 7 + /* gfx_v8_0_ring_emit_fence_compute x3 for user fence, vm fence */
6932 7 + /* gfx_v8_0_emit_mem_sync_compute */6835 7 + /* gfx_v8_0_emit_mem_sync_compute */
@@ -6947,6 +6850,7 @@ static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_compute = { · 14 unmodified lines
6947 .soft_recovery = gfx_v8_0_ring_soft_recovery,6850 .soft_recovery = gfx_v8_0_ring_soft_recovery,
6948 .emit_mem_sync = gfx_v8_0_emit_mem_sync_compute,6851 .emit_mem_sync = gfx_v8_0_emit_mem_sync_compute,
6949 .emit_wave_limit = gfx_v8_0_emit_wave_limit,6852 .emit_wave_limit = gfx_v8_0_emit_wave_limit,
6853 .init_cond_exec = gfx_v8_0_ring_emit_init_cond_exec,
6950};6854};
6951 6855
6952static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_kiq = {6856static const struct amdgpu_ring_funcs gfx_v8_0_ring_funcs_kiq = {
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c+2−0
@@ -4875,6 +4875,8 @@ static int gfx_v9_0_early_init(struct amdgpu_ip_block *ip_block) · 4874 unmodified lines
4875 /* init rlcg reg access ctrl */4875 /* init rlcg reg access ctrl */
4876 gfx_v9_0_init_rlcg_reg_access_ctrl(adev);4876 gfx_v9_0_init_rlcg_reg_access_ctrl(adev);
4877 4877
4878 amdgpu_init_rlc_reg_funcs(adev);
4879
4878 return gfx_v9_0_init_microcode(adev);4880 return gfx_v9_0_init_microcode(adev);
4879}4881}
4880 4882
drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c+104−993
@@ -39,7 +39,6 @@ · 38 unmodified lines
39#include "gfx_v9_4_3.h"39#include "gfx_v9_4_3.h"
40#include "gfx_v9_4_3_cleaner_shader.h"40#include "gfx_v9_4_3_cleaner_shader.h"
41#include "amdgpu_xcp.h"41#include "amdgpu_xcp.h"
42#include "amdgpu_aca.h"
43 42
44MODULE_FIRMWARE("amdgpu/gc_9_4_3_mec.bin");43MODULE_FIRMWARE("amdgpu/gc_9_4_3_mec.bin");
45MODULE_FIRMWARE("amdgpu/gc_9_4_4_mec.bin");44MODULE_FIRMWARE("amdgpu/gc_9_4_4_mec.bin");
@@ -851,73 +850,6 @@ static const struct amdgpu_gfx_funcs gfx_v9_4_3_gfx_funcs = { · 805 unmodified lines
851 .get_hdp_flush_mask = &amdgpu_gfx_get_hdp_flush_mask,850 .get_hdp_flush_mask = &amdgpu_gfx_get_hdp_flush_mask,
852};851};
853 852
854static int gfx_v9_4_3_aca_bank_parser(struct aca_handle *handle,
855 struct aca_bank *bank, enum aca_smu_type type,
856 void *data)
857{
858 struct aca_bank_info info;
859 u64 misc0;
860 u32 instlo;
861 int ret;
862
863 ret = aca_bank_info_decode(bank, &info);
864 if (ret)
865 return ret;
866
867 /* NOTE: overwrite info.die_id with xcd id for gfx */
868 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
869 instlo &= GENMASK(31, 1);
870 info.die_id = instlo == mmSMNAID_XCD0_MCA_SMU ? 0 : 1;
871
872 misc0 = bank->regs[ACA_REG_IDX_MISC0];
873
874 switch (type) {
875 case ACA_SMU_TYPE_UE:
876 bank->aca_err_type = ACA_ERROR_TYPE_UE;
877 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type, 1ULL);
878 break;
879 case ACA_SMU_TYPE_CE:
880 bank->aca_err_type = ACA_ERROR_TYPE_CE;
881 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
882 ACA_REG__MISC0__ERRCNT(misc0));
883 break;
884 default:
885 return -EINVAL;
886 }
887
888 return ret;
889}
890
891static bool gfx_v9_4_3_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
892 enum aca_smu_type type, void *data)
893{
894 u32 instlo;
895
896 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
897 instlo &= GENMASK(31, 1);
898 switch (instlo) {
899 case mmSMNAID_XCD0_MCA_SMU:
900 case mmSMNAID_XCD1_MCA_SMU:
901 case mmSMNXCD_XCD0_MCA_SMU:
902 return true;
903 default:
904 break;
905 }
906
907 return false;
908}
909
910static const struct aca_bank_ops gfx_v9_4_3_aca_bank_ops = {
911 .aca_bank_parser = gfx_v9_4_3_aca_bank_parser,
912 .aca_bank_is_valid = gfx_v9_4_3_aca_bank_is_valid,
913};
914
915static const struct aca_info gfx_v9_4_3_aca_info = {
916 .hwip = ACA_HWIP_TYPE_SMU,
917 .mask = ACA_ERROR_UE_MASK | ACA_ERROR_CE_MASK,
918 .bank_ops = &gfx_v9_4_3_aca_bank_ops,
919};
920
921static int gfx_v9_4_3_gpu_early_init(struct amdgpu_device *adev)853static int gfx_v9_4_3_gpu_early_init(struct amdgpu_device *adev)
922{854{
923 adev->gfx.funcs = &gfx_v9_4_3_gfx_funcs;855 adev->gfx.funcs = &gfx_v9_4_3_gfx_funcs;
@@ -1107,22 +1039,24 @@ static int gfx_v9_4_3_sw_init(struct amdgpu_ip_block *ip_block) · 183 unmodified lines
1107 /* set up the compute queues - allocate horizontally across pipes */1039 /* set up the compute queues - allocate horizontally across pipes */
1108 for (xcc_id = 0; xcc_id < num_xcc; xcc_id++) {1040 for (xcc_id = 0; xcc_id < num_xcc; xcc_id++) {
1109 ring_id = 0;1041 ring_id = 0;
1110 for (i = 0; i < adev->gfx.mec.num_mec; ++i) {1042 if (!adev->gfx.disable_kq) {
1111 for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {1043 for (i = 0; i < adev->gfx.mec.num_mec; ++i) {
1112 for (k = 0; k < adev->gfx.mec.num_pipe_per_mec;1044 for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) {
1113 k++) {1045 for (k = 0; k < adev->gfx.mec.num_pipe_per_mec;
1114 if (!amdgpu_gfx_is_mec_queue_enabled(1046 k++) {
1115 adev, xcc_id, i, k, j))1047 if (!amdgpu_gfx_is_mec_queue_enabled(
1116 continue;1048 adev, xcc_id, i, k, j))
1117 1049 continue;
1118 r = gfx_v9_4_3_compute_ring_init(adev,1050
1119 ring_id,1051 r = gfx_v9_4_3_compute_ring_init(adev,
1120 xcc_id,1052 ring_id,
1121 i, k, j);1053 xcc_id,
1122 if (r)1054 i, k, j);
1123 return r;1055 if (r)
1124 1056 return r;
1125 ring_id++;1057
1058 ring_id++;
1059 }
1126 }1060 }
1127 }1061 }
1128 }1062 }
@@ -2350,6 +2284,65 @@ static void gfx_v9_4_3_xcc_fini(struct amdgpu_device *adev, int xcc_id) · 1221 unmodified lines
2350 gfx_v9_4_3_xcc_cp_compute_enable(adev, false, xcc_id);2284 gfx_v9_4_3_xcc_cp_compute_enable(adev, false, xcc_id);
2351}2285}
2352 2286
2287static int gfx_v9_4_3_set_userq_eop_interrupts(struct amdgpu_device *adev,
2288 bool enable)
2289{
2290 int num_xcc = NUM_XCC(adev->gfx.xcc_mask);
2291 unsigned int irq_type;
2292 int m, p, xcc_id, r;
2293
2294 if (adev->gfx.disable_kq) {
2295 for (xcc_id = 0; xcc_id < num_xcc; xcc_id++) {
2296 for (m = 0; m < adev->gfx.mec.num_mec; ++m) {
2297 for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) {
2298 irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
2299 + (m * adev->gfx.mec.num_pipe_per_mec)
2300 + p;
2301
2302 if (enable)
2303 r = amdgpu_irq_get(adev, &adev->gfx.eop_irq,
2304 irq_type);
2305 else
2306 r = amdgpu_irq_put(adev, &adev->gfx.eop_irq,
2307 irq_type);
2308 if (r) {
2309 if (!enable)
2310 return r;
2311 goto err_compute;
2312 }
2313 }
2314 }
2315 }
2316 }
2317
2318 return 0;
2319
2320err_compute:
2321 for (p--; p >= 0; p--) {
2322 irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
2323 + (m * adev->gfx.mec.num_pipe_per_mec) + p;
2324 amdgpu_irq_put(adev, &adev->gfx.eop_irq, irq_type);
2325 }
2326 for (m--; m >= 0; m--) {
2327 for (p = adev->gfx.mec.num_pipe_per_mec - 1; p >= 0; p--) {
2328 irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
2329 + (m * adev->gfx.mec.num_pipe_per_mec) + p;
2330 amdgpu_irq_put(adev, &adev->gfx.eop_irq, irq_type);
2331 }
2332 }
2333 for (xcc_id--; xcc_id >= 0; xcc_id--) {
2334 for (m = adev->gfx.mec.num_mec - 1; m <= 0; m--) {
2335 for (p = adev->gfx.mec.num_pipe_per_mec - 1; p >= 0; p--) {
2336 irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP
2337 + (m * adev->gfx.mec.num_pipe_per_mec) + p;
2338 amdgpu_irq_put(adev, &adev->gfx.eop_irq, irq_type);
2339 }
2340 }
2341 }
2342
2343 return r;
2344}
2345
2353static int gfx_v9_4_3_hw_init(struct amdgpu_ip_block *ip_block)2346static int gfx_v9_4_3_hw_init(struct amdgpu_ip_block *ip_block)
2354{2347{
2355 int r;2348 int r;
@@ -2382,9 +2375,14 @@ static int gfx_v9_4_3_hw_init(struct amdgpu_ip_block *ip_block) · 26 unmodified lines
2382 r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0);2375 r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0);
2383 if (r)2376 if (r)
2384 goto err_bad_op;2377 goto err_bad_op;
2378 r = gfx_v9_4_3_set_userq_eop_interrupts(adev, true);
2379 if (r)
2380 goto err_bad_eop;
2385 2381
2386 return 0;2382 return 0;
2387 2383
2384err_bad_eop:
2385 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
2388err_bad_op:2386err_bad_op:
2389 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);2387 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
2390err_priv_inst:2388err_priv_inst:
@@ -2467,6 +2465,7 @@ static int gfx_v9_4_3_hw_fini(struct amdgpu_ip_block *ip_block) · 76 unmodified lines
2467 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);2465 amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0);
2468 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);2466 amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
2469 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);2467 amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
2468 gfx_v9_4_3_set_userq_eop_interrupts(adev, false);
2470 2469
2471 num_xcc = NUM_XCC(adev->gfx.xcc_mask);2470 num_xcc = NUM_XCC(adev->gfx.xcc_mask);
2472 for (i = 0; i < num_xcc; i++) {2471 for (i = 0; i < num_xcc; i++) {
@@ -2612,8 +2611,24 @@ static int gfx_v9_4_3_early_init(struct amdgpu_ip_block *ip_block) · 139 unmodified lines
2612{2611{
2613 struct amdgpu_device *adev = ip_block->adev;2612 struct amdgpu_device *adev = ip_block->adev;
2614 2613
2615 adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),2614 switch (amdgpu_user_queue) {
2616 AMDGPU_MAX_COMPUTE_RINGS);2615 case -1:
2616 case 0:
2617 default:
2618 adev->gfx.disable_kq = false;
2619 adev->gfx.disable_uq = true;
2620 break;
2621 case 2:
2622 adev->gfx.disable_kq = true;
2623 adev->gfx.disable_uq = true;
2624 break;
2625 }
2626
2627 if (adev->gfx.disable_kq)
2628 adev->gfx.num_compute_rings = 0;
2629 else
2630 adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev),
2631 AMDGPU_MAX_COMPUTE_RINGS);
2617 gfx_v9_4_3_set_kiq_pm4_funcs(adev);2632 gfx_v9_4_3_set_kiq_pm4_funcs(adev);
2618 gfx_v9_4_3_set_ring_funcs(adev);2633 gfx_v9_4_3_set_ring_funcs(adev);
2619 gfx_v9_4_3_set_irq_funcs(adev);2634 gfx_v9_4_3_set_irq_funcs(adev);
@@ -2623,6 +2638,8 @@ static int gfx_v9_4_3_early_init(struct amdgpu_ip_block *ip_block) · 3 unmodified lines
2623 /* init rlcg reg access ctrl */2638 /* init rlcg reg access ctrl */
2624 gfx_v9_4_3_init_rlcg_reg_access_ctrl(adev);2639 gfx_v9_4_3_init_rlcg_reg_access_ctrl(adev);
2625 2640
2641 amdgpu_init_rlc_reg_funcs(adev);
2642
2626 return gfx_v9_4_3_init_microcode(adev);2643 return gfx_v9_4_3_init_microcode(adev);
2627}2644}
2628 2645
@@ -3709,872 +3726,6 @@ pipe_reset: · 1080 unmodified lines
3709 return amdgpu_ring_reset_helper_end(ring, timedout_fence);3726 return amdgpu_ring_reset_helper_end(ring, timedout_fence);
3710}3727}
3711 3728
3712enum amdgpu_gfx_cp_ras_mem_id {
3713 AMDGPU_GFX_CP_MEM1 = 1,
3714 AMDGPU_GFX_CP_MEM2,
3715 AMDGPU_GFX_CP_MEM3,
3716 AMDGPU_GFX_CP_MEM4,
3717 AMDGPU_GFX_CP_MEM5,
3718};
3719
3720enum amdgpu_gfx_gcea_ras_mem_id {
3721 AMDGPU_GFX_GCEA_IOWR_CMDMEM = 4,
3722 AMDGPU_GFX_GCEA_IORD_CMDMEM,
3723 AMDGPU_GFX_GCEA_GMIWR_CMDMEM,
3724 AMDGPU_GFX_GCEA_GMIRD_CMDMEM,
3725 AMDGPU_GFX_GCEA_DRAMWR_CMDMEM,
3726 AMDGPU_GFX_GCEA_DRAMRD_CMDMEM,
3727 AMDGPU_GFX_GCEA_MAM_DMEM0,
3728 AMDGPU_GFX_GCEA_MAM_DMEM1,
3729 AMDGPU_GFX_GCEA_MAM_DMEM2,
3730 AMDGPU_GFX_GCEA_MAM_DMEM3,
3731 AMDGPU_GFX_GCEA_MAM_AMEM0,
3732 AMDGPU_GFX_GCEA_MAM_AMEM1,
3733 AMDGPU_GFX_GCEA_MAM_AMEM2,
3734 AMDGPU_GFX_GCEA_MAM_AMEM3,
3735 AMDGPU_GFX_GCEA_MAM_AFLUSH_BUFFER,
3736 AMDGPU_GFX_GCEA_WRET_TAGMEM,
3737 AMDGPU_GFX_GCEA_RRET_TAGMEM,
3738 AMDGPU_GFX_GCEA_IOWR_DATAMEM,
3739 AMDGPU_GFX_GCEA_GMIWR_DATAMEM,
3740 AMDGPU_GFX_GCEA_DRAM_DATAMEM,
3741};
3742
3743enum amdgpu_gfx_gc_cane_ras_mem_id {
3744 AMDGPU_GFX_GC_CANE_MEM0 = 0,
3745};
3746
3747enum amdgpu_gfx_gcutcl2_ras_mem_id {
3748 AMDGPU_GFX_GCUTCL2_MEM2P512X95 = 160,
3749};
3750
3751enum amdgpu_gfx_gds_ras_mem_id {
3752 AMDGPU_GFX_GDS_MEM0 = 0,
3753};
3754
3755enum amdgpu_gfx_lds_ras_mem_id {
3756 AMDGPU_GFX_LDS_BANK0 = 0,
3757 AMDGPU_GFX_LDS_BANK1,
3758 AMDGPU_GFX_LDS_BANK2,
3759 AMDGPU_GFX_LDS_BANK3,
3760 AMDGPU_GFX_LDS_BANK4,
3761 AMDGPU_GFX_LDS_BANK5,
3762 AMDGPU_GFX_LDS_BANK6,
3763 AMDGPU_GFX_LDS_BANK7,
3764 AMDGPU_GFX_LDS_BANK8,
3765 AMDGPU_GFX_LDS_BANK9,
3766 AMDGPU_GFX_LDS_BANK10,
3767 AMDGPU_GFX_LDS_BANK11,
3768 AMDGPU_GFX_LDS_BANK12,
3769 AMDGPU_GFX_LDS_BANK13,
3770 AMDGPU_GFX_LDS_BANK14,
3771 AMDGPU_GFX_LDS_BANK15,
3772 AMDGPU_GFX_LDS_BANK16,
3773 AMDGPU_GFX_LDS_BANK17,
3774 AMDGPU_GFX_LDS_BANK18,
3775 AMDGPU_GFX_LDS_BANK19,
3776 AMDGPU_GFX_LDS_BANK20,
3777 AMDGPU_GFX_LDS_BANK21,
3778 AMDGPU_GFX_LDS_BANK22,
3779 AMDGPU_GFX_LDS_BANK23,
3780 AMDGPU_GFX_LDS_BANK24,
3781 AMDGPU_GFX_LDS_BANK25,
3782 AMDGPU_GFX_LDS_BANK26,
3783 AMDGPU_GFX_LDS_BANK27,
3784 AMDGPU_GFX_LDS_BANK28,
3785 AMDGPU_GFX_LDS_BANK29,
3786 AMDGPU_GFX_LDS_BANK30,
3787 AMDGPU_GFX_LDS_BANK31,
3788 AMDGPU_GFX_LDS_SP_BUFFER_A,
3789 AMDGPU_GFX_LDS_SP_BUFFER_B,
3790};
3791
3792enum amdgpu_gfx_rlc_ras_mem_id {
3793 AMDGPU_GFX_RLC_GPMF32 = 1,
3794 AMDGPU_GFX_RLC_RLCVF32,
3795 AMDGPU_GFX_RLC_SCRATCH,
3796 AMDGPU_GFX_RLC_SRM_ARAM,
3797 AMDGPU_GFX_RLC_SRM_DRAM,
3798 AMDGPU_GFX_RLC_TCTAG,
3799 AMDGPU_GFX_RLC_SPM_SE,
3800 AMDGPU_GFX_RLC_SPM_GRBMT,
3801};
3802
3803enum amdgpu_gfx_sp_ras_mem_id {
3804 AMDGPU_GFX_SP_SIMDID0 = 0,
3805};
3806
3807enum amdgpu_gfx_spi_ras_mem_id {
3808 AMDGPU_GFX_SPI_MEM0 = 0,
3809 AMDGPU_GFX_SPI_MEM1,
3810 AMDGPU_GFX_SPI_MEM2,
3811 AMDGPU_GFX_SPI_MEM3,
3812};
3813
3814enum amdgpu_gfx_sqc_ras_mem_id {
3815 AMDGPU_GFX_SQC_INST_CACHE_A = 100,
3816 AMDGPU_GFX_SQC_INST_CACHE_B = 101,
3817 AMDGPU_GFX_SQC_INST_CACHE_TAG_A = 102,
3818 AMDGPU_GFX_SQC_INST_CACHE_TAG_B = 103,
3819 AMDGPU_GFX_SQC_INST_CACHE_MISS_FIFO_A = 104,
3820 AMDGPU_GFX_SQC_INST_CACHE_MISS_FIFO_B = 105,
3821 AMDGPU_GFX_SQC_INST_CACHE_GATCL1_MISS_FIFO_A = 106,
3822 AMDGPU_GFX_SQC_INST_CACHE_GATCL1_MISS_FIFO_B = 107,
3823 AMDGPU_GFX_SQC_DATA_CACHE_A = 200,
3824 AMDGPU_GFX_SQC_DATA_CACHE_B = 201,
3825 AMDGPU_GFX_SQC_DATA_CACHE_TAG_A = 202,
3826 AMDGPU_GFX_SQC_DATA_CACHE_TAG_B = 203,
3827 AMDGPU_GFX_SQC_DATA_CACHE_MISS_FIFO_A = 204,
3828 AMDGPU_GFX_SQC_DATA_CACHE_MISS_FIFO_B = 205,
3829 AMDGPU_GFX_SQC_DATA_CACHE_HIT_FIFO_A = 206,
3830 AMDGPU_GFX_SQC_DATA_CACHE_HIT_FIFO_B = 207,
3831 AMDGPU_GFX_SQC_DIRTY_BIT_A = 208,
3832 AMDGPU_GFX_SQC_DIRTY_BIT_B = 209,
3833 AMDGPU_GFX_SQC_WRITE_DATA_BUFFER_CU0 = 210,
3834 AMDGPU_GFX_SQC_WRITE_DATA_BUFFER_CU1 = 211,
3835 AMDGPU_GFX_SQC_UTCL1_MISS_LFIFO_DATA_CACHE_A = 212,
3836 AMDGPU_GFX_SQC_UTCL1_MISS_LFIFO_DATA_CACHE_B = 213,
3837 AMDGPU_GFX_SQC_UTCL1_MISS_LFIFO_INST_CACHE = 108,
3838};
3839
3840enum amdgpu_gfx_sq_ras_mem_id {
3841 AMDGPU_GFX_SQ_SGPR_MEM0 = 0,
3842 AMDGPU_GFX_SQ_SGPR_MEM1,
3843 AMDGPU_GFX_SQ_SGPR_MEM2,
3844 AMDGPU_GFX_SQ_SGPR_MEM3,
3845};
3846
3847enum amdgpu_gfx_ta_ras_mem_id {
3848 AMDGPU_GFX_TA_FS_AFIFO_RAM_LO = 1,
3849 AMDGPU_GFX_TA_FS_AFIFO_RAM_HI,
3850 AMDGPU_GFX_TA_FS_CFIFO_RAM,
3851 AMDGPU_GFX_TA_FSX_LFIFO,
3852 AMDGPU_GFX_TA_FS_DFIFO_RAM,
3853};
3854
3855enum amdgpu_gfx_tcc_ras_mem_id {
3856 AMDGPU_GFX_TCC_MEM1 = 1,
3857};
3858
3859enum amdgpu_gfx_tca_ras_mem_id {
3860 AMDGPU_GFX_TCA_MEM1 = 1,
3861};
3862
3863enum amdgpu_gfx_tci_ras_mem_id {
3864 AMDGPU_GFX_TCIW_MEM = 1,
3865};
3866
3867enum amdgpu_gfx_tcp_ras_mem_id {
3868 AMDGPU_GFX_TCP_LFIFO0 = 1,
3869 AMDGPU_GFX_TCP_SET0BANK0_RAM,
3870 AMDGPU_GFX_TCP_SET0BANK1_RAM,
3871 AMDGPU_GFX_TCP_SET0BANK2_RAM,
3872 AMDGPU_GFX_TCP_SET0BANK3_RAM,
3873 AMDGPU_GFX_TCP_SET1BANK0_RAM,
3874 AMDGPU_GFX_TCP_SET1BANK1_RAM,
3875 AMDGPU_GFX_TCP_SET1BANK2_RAM,
3876 AMDGPU_GFX_TCP_SET1BANK3_RAM,
3877 AMDGPU_GFX_TCP_SET2BANK0_RAM,
3878 AMDGPU_GFX_TCP_SET2BANK1_RAM,
3879 AMDGPU_GFX_TCP_SET2BANK2_RAM,
3880 AMDGPU_GFX_TCP_SET2BANK3_RAM,
3881 AMDGPU_GFX_TCP_SET3BANK0_RAM,
3882 AMDGPU_GFX_TCP_SET3BANK1_RAM,
3883 AMDGPU_GFX_TCP_SET3BANK2_RAM,
3884 AMDGPU_GFX_TCP_SET3BANK3_RAM,
3885 AMDGPU_GFX_TCP_VM_FIFO,
3886 AMDGPU_GFX_TCP_DB_TAGRAM0,
3887 AMDGPU_GFX_TCP_DB_TAGRAM1,
3888 AMDGPU_GFX_TCP_DB_TAGRAM2,
3889 AMDGPU_GFX_TCP_DB_TAGRAM3,
3890 AMDGPU_GFX_TCP_UTCL1_LFIFO_PROBE0,
3891 AMDGPU_GFX_TCP_UTCL1_LFIFO_PROBE1,
3892 AMDGPU_GFX_TCP_CMD_FIFO,
3893};
3894
3895enum amdgpu_gfx_td_ras_mem_id {
3896 AMDGPU_GFX_TD_UTD_CS_FIFO_MEM = 1,
3897 AMDGPU_GFX_TD_UTD_SS_FIFO_LO_MEM,
3898 AMDGPU_GFX_TD_UTD_SS_FIFO_HI_MEM,
3899};
3900
3901enum amdgpu_gfx_tcx_ras_mem_id {
3902 AMDGPU_GFX_TCX_FIFOD0 = 0,
3903 AMDGPU_GFX_TCX_FIFOD1,
3904 AMDGPU_GFX_TCX_FIFOD2,
3905 AMDGPU_GFX_TCX_FIFOD3,
3906 AMDGPU_GFX_TCX_FIFOD4,
3907 AMDGPU_GFX_TCX_FIFOD5,
3908 AMDGPU_GFX_TCX_FIFOD6,
3909 AMDGPU_GFX_TCX_FIFOD7,
3910 AMDGPU_GFX_TCX_FIFOB0,
3911 AMDGPU_GFX_TCX_FIFOB1,
3912 AMDGPU_GFX_TCX_FIFOB2,
3913 AMDGPU_GFX_TCX_FIFOB3,
3914 AMDGPU_GFX_TCX_FIFOB4,
3915 AMDGPU_GFX_TCX_FIFOB5,
3916 AMDGPU_GFX_TCX_FIFOB6,
3917 AMDGPU_GFX_TCX_FIFOB7,
3918 AMDGPU_GFX_TCX_FIFOA0,
3919 AMDGPU_GFX_TCX_FIFOA1,
3920 AMDGPU_GFX_TCX_FIFOA2,
3921 AMDGPU_GFX_TCX_FIFOA3,
3922 AMDGPU_GFX_TCX_FIFOA4,
3923 AMDGPU_GFX_TCX_FIFOA5,
3924 AMDGPU_GFX_TCX_FIFOA6,
3925 AMDGPU_GFX_TCX_FIFOA7,
3926 AMDGPU_GFX_TCX_CFIFO0,
3927 AMDGPU_GFX_TCX_CFIFO1,
3928 AMDGPU_GFX_TCX_CFIFO2,
3929 AMDGPU_GFX_TCX_CFIFO3,
3930 AMDGPU_GFX_TCX_CFIFO4,
3931 AMDGPU_GFX_TCX_CFIFO5,
3932 AMDGPU_GFX_TCX_CFIFO6,
3933 AMDGPU_GFX_TCX_CFIFO7,
3934 AMDGPU_GFX_TCX_FIFO_ACKB0,
3935 AMDGPU_GFX_TCX_FIFO_ACKB1,
3936 AMDGPU_GFX_TCX_FIFO_ACKB2,
3937 AMDGPU_GFX_TCX_FIFO_ACKB3,
3938 AMDGPU_GFX_TCX_FIFO_ACKB4,
3939 AMDGPU_GFX_TCX_FIFO_ACKB5,
3940 AMDGPU_GFX_TCX_FIFO_ACKB6,
3941 AMDGPU_GFX_TCX_FIFO_ACKB7,
3942 AMDGPU_GFX_TCX_FIFO_ACKD0,
3943 AMDGPU_GFX_TCX_FIFO_ACKD1,
3944 AMDGPU_GFX_TCX_FIFO_ACKD2,
3945 AMDGPU_GFX_TCX_FIFO_ACKD3,
3946 AMDGPU_GFX_TCX_FIFO_ACKD4,
3947 AMDGPU_GFX_TCX_FIFO_ACKD5,
3948 AMDGPU_GFX_TCX_FIFO_ACKD6,
3949 AMDGPU_GFX_TCX_FIFO_ACKD7,
3950 AMDGPU_GFX_TCX_DST_FIFOA0,
3951 AMDGPU_GFX_TCX_DST_FIFOA1,
3952 AMDGPU_GFX_TCX_DST_FIFOA2,
3953 AMDGPU_GFX_TCX_DST_FIFOA3,
3954 AMDGPU_GFX_TCX_DST_FIFOA4,
3955 AMDGPU_GFX_TCX_DST_FIFOA5,
3956 AMDGPU_GFX_TCX_DST_FIFOA6,
3957 AMDGPU_GFX_TCX_DST_FIFOA7,
3958 AMDGPU_GFX_TCX_DST_FIFOB0,
3959 AMDGPU_GFX_TCX_DST_FIFOB1,
3960 AMDGPU_GFX_TCX_DST_FIFOB2,
3961 AMDGPU_GFX_TCX_DST_FIFOB3,
3962 AMDGPU_GFX_TCX_DST_FIFOB4,
3963 AMDGPU_GFX_TCX_DST_FIFOB5,
3964 AMDGPU_GFX_TCX_DST_FIFOB6,
3965 AMDGPU_GFX_TCX_DST_FIFOB7,
3966 AMDGPU_GFX_TCX_DST_FIFOD0,
3967 AMDGPU_GFX_TCX_DST_FIFOD1,
3968 AMDGPU_GFX_TCX_DST_FIFOD2,
3969 AMDGPU_GFX_TCX_DST_FIFOD3,
3970 AMDGPU_GFX_TCX_DST_FIFOD4,
3971 AMDGPU_GFX_TCX_DST_FIFOD5,
3972 AMDGPU_GFX_TCX_DST_FIFOD6,
3973 AMDGPU_GFX_TCX_DST_FIFOD7,
3974 AMDGPU_GFX_TCX_DST_FIFO_ACKB0,
3975 AMDGPU_GFX_TCX_DST_FIFO_ACKB1,
3976 AMDGPU_GFX_TCX_DST_FIFO_ACKB2,
3977 AMDGPU_GFX_TCX_DST_FIFO_ACKB3,
3978 AMDGPU_GFX_TCX_DST_FIFO_ACKB4,
3979 AMDGPU_GFX_TCX_DST_FIFO_ACKB5,
3980 AMDGPU_GFX_TCX_DST_FIFO_ACKB6,
3981 AMDGPU_GFX_TCX_DST_FIFO_ACKB7,
3982 AMDGPU_GFX_TCX_DST_FIFO_ACKD0,
3983 AMDGPU_GFX_TCX_DST_FIFO_ACKD1,
3984 AMDGPU_GFX_TCX_DST_FIFO_ACKD2,
3985 AMDGPU_GFX_TCX_DST_FIFO_ACKD3,
3986 AMDGPU_GFX_TCX_DST_FIFO_ACKD4,
3987 AMDGPU_GFX_TCX_DST_FIFO_ACKD5,
3988 AMDGPU_GFX_TCX_DST_FIFO_ACKD6,
3989 AMDGPU_GFX_TCX_DST_FIFO_ACKD7,
3990};
3991
3992enum amdgpu_gfx_atc_l2_ras_mem_id {
3993 AMDGPU_GFX_ATC_L2_MEM0 = 0,
3994};
3995
3996enum amdgpu_gfx_utcl2_ras_mem_id {
3997 AMDGPU_GFX_UTCL2_MEM0 = 0,
3998};
3999
4000enum amdgpu_gfx_vml2_ras_mem_id {
4001 AMDGPU_GFX_VML2_MEM0 = 0,
4002};
4003
4004enum amdgpu_gfx_vml2_walker_ras_mem_id {
4005 AMDGPU_GFX_VML2_WALKER_MEM0 = 0,
4006};
4007
4008static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_cp_mem_list[] = {
4009 {AMDGPU_GFX_CP_MEM1, "CP_MEM1"},
4010 {AMDGPU_GFX_CP_MEM2, "CP_MEM2"},
4011 {AMDGPU_GFX_CP_MEM3, "CP_MEM3"},
4012 {AMDGPU_GFX_CP_MEM4, "CP_MEM4"},
4013 {AMDGPU_GFX_CP_MEM5, "CP_MEM5"},
4014};
4015
4016static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_gcea_mem_list[] = {
4017 {AMDGPU_GFX_GCEA_IOWR_CMDMEM, "GCEA_IOWR_CMDMEM"},
4018 {AMDGPU_GFX_GCEA_IORD_CMDMEM, "GCEA_IORD_CMDMEM"},
4019 {AMDGPU_GFX_GCEA_GMIWR_CMDMEM, "GCEA_GMIWR_CMDMEM"},
4020 {AMDGPU_GFX_GCEA_GMIRD_CMDMEM, "GCEA_GMIRD_CMDMEM"},
4021 {AMDGPU_GFX_GCEA_DRAMWR_CMDMEM, "GCEA_DRAMWR_CMDMEM"},
4022 {AMDGPU_GFX_GCEA_DRAMRD_CMDMEM, "GCEA_DRAMRD_CMDMEM"},
4023 {AMDGPU_GFX_GCEA_MAM_DMEM0, "GCEA_MAM_DMEM0"},
4024 {AMDGPU_GFX_GCEA_MAM_DMEM1, "GCEA_MAM_DMEM1"},
4025 {AMDGPU_GFX_GCEA_MAM_DMEM2, "GCEA_MAM_DMEM2"},
4026 {AMDGPU_GFX_GCEA_MAM_DMEM3, "GCEA_MAM_DMEM3"},
4027 {AMDGPU_GFX_GCEA_MAM_AMEM0, "GCEA_MAM_AMEM0"},
4028 {AMDGPU_GFX_GCEA_MAM_AMEM1, "GCEA_MAM_AMEM1"},
4029 {AMDGPU_GFX_GCEA_MAM_AMEM2, "GCEA_MAM_AMEM2"},
4030 {AMDGPU_GFX_GCEA_MAM_AMEM3, "GCEA_MAM_AMEM3"},
4031 {AMDGPU_GFX_GCEA_MAM_AFLUSH_BUFFER, "GCEA_MAM_AFLUSH_BUFFER"},
4032 {AMDGPU_GFX_GCEA_WRET_TAGMEM, "GCEA_WRET_TAGMEM"},
4033 {AMDGPU_GFX_GCEA_RRET_TAGMEM, "GCEA_RRET_TAGMEM"},
4034 {AMDGPU_GFX_GCEA_IOWR_DATAMEM, "GCEA_IOWR_DATAMEM"},
4035 {AMDGPU_GFX_GCEA_GMIWR_DATAMEM, "GCEA_GMIWR_DATAMEM"},
4036 {AMDGPU_GFX_GCEA_DRAM_DATAMEM, "GCEA_DRAM_DATAMEM"},
4037};
4038
4039static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_gc_cane_mem_list[] = {
4040 {AMDGPU_GFX_GC_CANE_MEM0, "GC_CANE_MEM0"},
4041};
4042
4043static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_gcutcl2_mem_list[] = {
4044 {AMDGPU_GFX_GCUTCL2_MEM2P512X95, "GCUTCL2_MEM2P512X95"},
4045};
4046
4047static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_gds_mem_list[] = {
4048 {AMDGPU_GFX_GDS_MEM0, "GDS_MEM"},
4049};
4050
4051static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_lds_mem_list[] = {
4052 {AMDGPU_GFX_LDS_BANK0, "LDS_BANK0"},
4053 {AMDGPU_GFX_LDS_BANK1, "LDS_BANK1"},
4054 {AMDGPU_GFX_LDS_BANK2, "LDS_BANK2"},
4055 {AMDGPU_GFX_LDS_BANK3, "LDS_BANK3"},
4056 {AMDGPU_GFX_LDS_BANK4, "LDS_BANK4"},
4057 {AMDGPU_GFX_LDS_BANK5, "LDS_BANK5"},
4058 {AMDGPU_GFX_LDS_BANK6, "LDS_BANK6"},
4059 {AMDGPU_GFX_LDS_BANK7, "LDS_BANK7"},
4060 {AMDGPU_GFX_LDS_BANK8, "LDS_BANK8"},
4061 {AMDGPU_GFX_LDS_BANK9, "LDS_BANK9"},
4062 {AMDGPU_GFX_LDS_BANK10, "LDS_BANK10"},
4063 {AMDGPU_GFX_LDS_BANK11, "LDS_BANK11"},
4064 {AMDGPU_GFX_LDS_BANK12, "LDS_BANK12"},
4065 {AMDGPU_GFX_LDS_BANK13, "LDS_BANK13"},
4066 {AMDGPU_GFX_LDS_BANK14, "LDS_BANK14"},
4067 {AMDGPU_GFX_LDS_BANK15, "LDS_BANK15"},
4068 {AMDGPU_GFX_LDS_BANK16, "LDS_BANK16"},
4069 {AMDGPU_GFX_LDS_BANK17, "LDS_BANK17"},
4070 {AMDGPU_GFX_LDS_BANK18, "LDS_BANK18"},
4071 {AMDGPU_GFX_LDS_BANK19, "LDS_BANK19"},
4072 {AMDGPU_GFX_LDS_BANK20, "LDS_BANK20"},
4073 {AMDGPU_GFX_LDS_BANK21, "LDS_BANK21"},
4074 {AMDGPU_GFX_LDS_BANK22, "LDS_BANK22"},
4075 {AMDGPU_GFX_LDS_BANK23, "LDS_BANK23"},
4076 {AMDGPU_GFX_LDS_BANK24, "LDS_BANK24"},
4077 {AMDGPU_GFX_LDS_BANK25, "LDS_BANK25"},
4078 {AMDGPU_GFX_LDS_BANK26, "LDS_BANK26"},
4079 {AMDGPU_GFX_LDS_BANK27, "LDS_BANK27"},
4080 {AMDGPU_GFX_LDS_BANK28, "LDS_BANK28"},
4081 {AMDGPU_GFX_LDS_BANK29, "LDS_BANK29"},
4082 {AMDGPU_GFX_LDS_BANK30, "LDS_BANK30"},
4083 {AMDGPU_GFX_LDS_BANK31, "LDS_BANK31"},
4084 {AMDGPU_GFX_LDS_SP_BUFFER_A, "LDS_SP_BUFFER_A"},
4085 {AMDGPU_GFX_LDS_SP_BUFFER_B, "LDS_SP_BUFFER_B"},
4086};
4087
4088static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_rlc_mem_list[] = {
4089 {AMDGPU_GFX_RLC_GPMF32, "RLC_GPMF32"},
4090 {AMDGPU_GFX_RLC_RLCVF32, "RLC_RLCVF32"},
4091 {AMDGPU_GFX_RLC_SCRATCH, "RLC_SCRATCH"},
4092 {AMDGPU_GFX_RLC_SRM_ARAM, "RLC_SRM_ARAM"},
4093 {AMDGPU_GFX_RLC_SRM_DRAM, "RLC_SRM_DRAM"},
4094 {AMDGPU_GFX_RLC_TCTAG, "RLC_TCTAG"},
4095 {AMDGPU_GFX_RLC_SPM_SE, "RLC_SPM_SE"},
4096 {AMDGPU_GFX_RLC_SPM_GRBMT, "RLC_SPM_GRBMT"},
4097};
4098
4099static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_sp_mem_list[] = {
4100 {AMDGPU_GFX_SP_SIMDID0, "SP_SIMDID0"},
4101};
4102
4103static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_spi_mem_list[] = {
4104 {AMDGPU_GFX_SPI_MEM0, "SPI_MEM0"},
4105 {AMDGPU_GFX_SPI_MEM1, "SPI_MEM1"},
4106 {AMDGPU_GFX_SPI_MEM2, "SPI_MEM2"},
4107 {AMDGPU_GFX_SPI_MEM3, "SPI_MEM3"},
4108};
4109
4110static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_sqc_mem_list[] = {
4111 {AMDGPU_GFX_SQC_INST_CACHE_A, "SQC_INST_CACHE_A"},
4112 {AMDGPU_GFX_SQC_INST_CACHE_B, "SQC_INST_CACHE_B"},
4113 {AMDGPU_GFX_SQC_INST_CACHE_TAG_A, "SQC_INST_CACHE_TAG_A"},
4114 {AMDGPU_GFX_SQC_INST_CACHE_TAG_B, "SQC_INST_CACHE_TAG_B"},
4115 {AMDGPU_GFX_SQC_INST_CACHE_MISS_FIFO_A, "SQC_INST_CACHE_MISS_FIFO_A"},
4116 {AMDGPU_GFX_SQC_INST_CACHE_MISS_FIFO_B, "SQC_INST_CACHE_MISS_FIFO_B"},
4117 {AMDGPU_GFX_SQC_INST_CACHE_GATCL1_MISS_FIFO_A, "SQC_INST_CACHE_GATCL1_MISS_FIFO_A"},
4118 {AMDGPU_GFX_SQC_INST_CACHE_GATCL1_MISS_FIFO_B, "SQC_INST_CACHE_GATCL1_MISS_FIFO_B"},
4119 {AMDGPU_GFX_SQC_DATA_CACHE_A, "SQC_DATA_CACHE_A"},
4120 {AMDGPU_GFX_SQC_DATA_CACHE_B, "SQC_DATA_CACHE_B"},
4121 {AMDGPU_GFX_SQC_DATA_CACHE_TAG_A, "SQC_DATA_CACHE_TAG_A"},
4122 {AMDGPU_GFX_SQC_DATA_CACHE_TAG_B, "SQC_DATA_CACHE_TAG_B"},
4123 {AMDGPU_GFX_SQC_DATA_CACHE_MISS_FIFO_A, "SQC_DATA_CACHE_MISS_FIFO_A"},
4124 {AMDGPU_GFX_SQC_DATA_CACHE_MISS_FIFO_B, "SQC_DATA_CACHE_MISS_FIFO_B"},
4125 {AMDGPU_GFX_SQC_DATA_CACHE_HIT_FIFO_A, "SQC_DATA_CACHE_HIT_FIFO_A"},
4126 {AMDGPU_GFX_SQC_DATA_CACHE_HIT_FIFO_B, "SQC_DATA_CACHE_HIT_FIFO_B"},
4127 {AMDGPU_GFX_SQC_DIRTY_BIT_A, "SQC_DIRTY_BIT_A"},
4128 {AMDGPU_GFX_SQC_DIRTY_BIT_B, "SQC_DIRTY_BIT_B"},
4129 {AMDGPU_GFX_SQC_WRITE_DATA_BUFFER_CU0, "SQC_WRITE_DATA_BUFFER_CU0"},
4130 {AMDGPU_GFX_SQC_WRITE_DATA_BUFFER_CU1, "SQC_WRITE_DATA_BUFFER_CU1"},
4131 {AMDGPU_GFX_SQC_UTCL1_MISS_LFIFO_DATA_CACHE_A, "SQC_UTCL1_MISS_LFIFO_DATA_CACHE_A"},
4132 {AMDGPU_GFX_SQC_UTCL1_MISS_LFIFO_DATA_CACHE_B, "SQC_UTCL1_MISS_LFIFO_DATA_CACHE_B"},
4133 {AMDGPU_GFX_SQC_UTCL1_MISS_LFIFO_INST_CACHE, "SQC_UTCL1_MISS_LFIFO_INST_CACHE"},
4134};
4135
4136static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_sq_mem_list[] = {
4137 {AMDGPU_GFX_SQ_SGPR_MEM0, "SQ_SGPR_MEM0"},
4138 {AMDGPU_GFX_SQ_SGPR_MEM1, "SQ_SGPR_MEM1"},
4139 {AMDGPU_GFX_SQ_SGPR_MEM2, "SQ_SGPR_MEM2"},
4140 {AMDGPU_GFX_SQ_SGPR_MEM3, "SQ_SGPR_MEM3"},
4141};
4142
4143static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_ta_mem_list[] = {
4144 {AMDGPU_GFX_TA_FS_AFIFO_RAM_LO, "TA_FS_AFIFO_RAM_LO"},
4145 {AMDGPU_GFX_TA_FS_AFIFO_RAM_HI, "TA_FS_AFIFO_RAM_HI"},
4146 {AMDGPU_GFX_TA_FS_CFIFO_RAM, "TA_FS_CFIFO_RAM"},
4147 {AMDGPU_GFX_TA_FSX_LFIFO, "TA_FSX_LFIFO"},
4148 {AMDGPU_GFX_TA_FS_DFIFO_RAM, "TA_FS_DFIFO_RAM"},
4149};
4150
4151static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_tcc_mem_list[] = {
4152 {AMDGPU_GFX_TCC_MEM1, "TCC_MEM1"},
4153};
4154
4155static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_tca_mem_list[] = {
4156 {AMDGPU_GFX_TCA_MEM1, "TCA_MEM1"},
4157};
4158
4159static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_tci_mem_list[] = {
4160 {AMDGPU_GFX_TCIW_MEM, "TCIW_MEM"},
4161};
4162
4163static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_tcp_mem_list[] = {
4164 {AMDGPU_GFX_TCP_LFIFO0, "TCP_LFIFO0"},
4165 {AMDGPU_GFX_TCP_SET0BANK0_RAM, "TCP_SET0BANK0_RAM"},
4166 {AMDGPU_GFX_TCP_SET0BANK1_RAM, "TCP_SET0BANK1_RAM"},
4167 {AMDGPU_GFX_TCP_SET0BANK2_RAM, "TCP_SET0BANK2_RAM"},
4168 {AMDGPU_GFX_TCP_SET0BANK3_RAM, "TCP_SET0BANK3_RAM"},
4169 {AMDGPU_GFX_TCP_SET1BANK0_RAM, "TCP_SET1BANK0_RAM"},
4170 {AMDGPU_GFX_TCP_SET1BANK1_RAM, "TCP_SET1BANK1_RAM"},
4171 {AMDGPU_GFX_TCP_SET1BANK2_RAM, "TCP_SET1BANK2_RAM"},
4172 {AMDGPU_GFX_TCP_SET1BANK3_RAM, "TCP_SET1BANK3_RAM"},
4173 {AMDGPU_GFX_TCP_SET2BANK0_RAM, "TCP_SET2BANK0_RAM"},
4174 {AMDGPU_GFX_TCP_SET2BANK1_RAM, "TCP_SET2BANK1_RAM"},
4175 {AMDGPU_GFX_TCP_SET2BANK2_RAM, "TCP_SET2BANK2_RAM"},
4176 {AMDGPU_GFX_TCP_SET2BANK3_RAM, "TCP_SET2BANK3_RAM"},
4177 {AMDGPU_GFX_TCP_SET3BANK0_RAM, "TCP_SET3BANK0_RAM"},
4178 {AMDGPU_GFX_TCP_SET3BANK1_RAM, "TCP_SET3BANK1_RAM"},
4179 {AMDGPU_GFX_TCP_SET3BANK2_RAM, "TCP_SET3BANK2_RAM"},
4180 {AMDGPU_GFX_TCP_SET3BANK3_RAM, "TCP_SET3BANK3_RAM"},
4181 {AMDGPU_GFX_TCP_VM_FIFO, "TCP_VM_FIFO"},
4182 {AMDGPU_GFX_TCP_DB_TAGRAM0, "TCP_DB_TAGRAM0"},
4183 {AMDGPU_GFX_TCP_DB_TAGRAM1, "TCP_DB_TAGRAM1"},
4184 {AMDGPU_GFX_TCP_DB_TAGRAM2, "TCP_DB_TAGRAM2"},
4185 {AMDGPU_GFX_TCP_DB_TAGRAM3, "TCP_DB_TAGRAM3"},
4186 {AMDGPU_GFX_TCP_UTCL1_LFIFO_PROBE0, "TCP_UTCL1_LFIFO_PROBE0"},
4187 {AMDGPU_GFX_TCP_UTCL1_LFIFO_PROBE1, "TCP_UTCL1_LFIFO_PROBE1"},
4188 {AMDGPU_GFX_TCP_CMD_FIFO, "TCP_CMD_FIFO"},
4189};
4190
4191static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_td_mem_list[] = {
4192 {AMDGPU_GFX_TD_UTD_CS_FIFO_MEM, "TD_UTD_CS_FIFO_MEM"},
4193 {AMDGPU_GFX_TD_UTD_SS_FIFO_LO_MEM, "TD_UTD_SS_FIFO_LO_MEM"},
4194 {AMDGPU_GFX_TD_UTD_SS_FIFO_HI_MEM, "TD_UTD_SS_FIFO_HI_MEM"},
4195};
4196
4197static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_tcx_mem_list[] = {
4198 {AMDGPU_GFX_TCX_FIFOD0, "TCX_FIFOD0"},
4199 {AMDGPU_GFX_TCX_FIFOD1, "TCX_FIFOD1"},
4200 {AMDGPU_GFX_TCX_FIFOD2, "TCX_FIFOD2"},
4201 {AMDGPU_GFX_TCX_FIFOD3, "TCX_FIFOD3"},
4202 {AMDGPU_GFX_TCX_FIFOD4, "TCX_FIFOD4"},
4203 {AMDGPU_GFX_TCX_FIFOD5, "TCX_FIFOD5"},
4204 {AMDGPU_GFX_TCX_FIFOD6, "TCX_FIFOD6"},
4205 {AMDGPU_GFX_TCX_FIFOD7, "TCX_FIFOD7"},
4206 {AMDGPU_GFX_TCX_FIFOB0, "TCX_FIFOB0"},
4207 {AMDGPU_GFX_TCX_FIFOB1, "TCX_FIFOB1"},
4208 {AMDGPU_GFX_TCX_FIFOB2, "TCX_FIFOB2"},
4209 {AMDGPU_GFX_TCX_FIFOB3, "TCX_FIFOB3"},
4210 {AMDGPU_GFX_TCX_FIFOB4, "TCX_FIFOB4"},
4211 {AMDGPU_GFX_TCX_FIFOB5, "TCX_FIFOB5"},
4212 {AMDGPU_GFX_TCX_FIFOB6, "TCX_FIFOB6"},
4213 {AMDGPU_GFX_TCX_FIFOB7, "TCX_FIFOB7"},
4214 {AMDGPU_GFX_TCX_FIFOA0, "TCX_FIFOA0"},
4215 {AMDGPU_GFX_TCX_FIFOA1, "TCX_FIFOA1"},
4216 {AMDGPU_GFX_TCX_FIFOA2, "TCX_FIFOA2"},
4217 {AMDGPU_GFX_TCX_FIFOA3, "TCX_FIFOA3"},
4218 {AMDGPU_GFX_TCX_FIFOA4, "TCX_FIFOA4"},
4219 {AMDGPU_GFX_TCX_FIFOA5, "TCX_FIFOA5"},
4220 {AMDGPU_GFX_TCX_FIFOA6, "TCX_FIFOA6"},
4221 {AMDGPU_GFX_TCX_FIFOA7, "TCX_FIFOA7"},
4222 {AMDGPU_GFX_TCX_CFIFO0, "TCX_CFIFO0"},
4223 {AMDGPU_GFX_TCX_CFIFO1, "TCX_CFIFO1"},
4224 {AMDGPU_GFX_TCX_CFIFO2, "TCX_CFIFO2"},
4225 {AMDGPU_GFX_TCX_CFIFO3, "TCX_CFIFO3"},
4226 {AMDGPU_GFX_TCX_CFIFO4, "TCX_CFIFO4"},
4227 {AMDGPU_GFX_TCX_CFIFO5, "TCX_CFIFO5"},
4228 {AMDGPU_GFX_TCX_CFIFO6, "TCX_CFIFO6"},
4229 {AMDGPU_GFX_TCX_CFIFO7, "TCX_CFIFO7"},
4230 {AMDGPU_GFX_TCX_FIFO_ACKB0, "TCX_FIFO_ACKB0"},
4231 {AMDGPU_GFX_TCX_FIFO_ACKB1, "TCX_FIFO_ACKB1"},
4232 {AMDGPU_GFX_TCX_FIFO_ACKB2, "TCX_FIFO_ACKB2"},
4233 {AMDGPU_GFX_TCX_FIFO_ACKB3, "TCX_FIFO_ACKB3"},
4234 {AMDGPU_GFX_TCX_FIFO_ACKB4, "TCX_FIFO_ACKB4"},
4235 {AMDGPU_GFX_TCX_FIFO_ACKB5, "TCX_FIFO_ACKB5"},
4236 {AMDGPU_GFX_TCX_FIFO_ACKB6, "TCX_FIFO_ACKB6"},
4237 {AMDGPU_GFX_TCX_FIFO_ACKB7, "TCX_FIFO_ACKB7"},
4238 {AMDGPU_GFX_TCX_FIFO_ACKD0, "TCX_FIFO_ACKD0"},
4239 {AMDGPU_GFX_TCX_FIFO_ACKD1, "TCX_FIFO_ACKD1"},
4240 {AMDGPU_GFX_TCX_FIFO_ACKD2, "TCX_FIFO_ACKD2"},
4241 {AMDGPU_GFX_TCX_FIFO_ACKD3, "TCX_FIFO_ACKD3"},
4242 {AMDGPU_GFX_TCX_FIFO_ACKD4, "TCX_FIFO_ACKD4"},
4243 {AMDGPU_GFX_TCX_FIFO_ACKD5, "TCX_FIFO_ACKD5"},
4244 {AMDGPU_GFX_TCX_FIFO_ACKD6, "TCX_FIFO_ACKD6"},
4245 {AMDGPU_GFX_TCX_FIFO_ACKD7, "TCX_FIFO_ACKD7"},
4246 {AMDGPU_GFX_TCX_DST_FIFOA0, "TCX_DST_FIFOA0"},
4247 {AMDGPU_GFX_TCX_DST_FIFOA1, "TCX_DST_FIFOA1"},
4248 {AMDGPU_GFX_TCX_DST_FIFOA2, "TCX_DST_FIFOA2"},
4249 {AMDGPU_GFX_TCX_DST_FIFOA3, "TCX_DST_FIFOA3"},
4250 {AMDGPU_GFX_TCX_DST_FIFOA4, "TCX_DST_FIFOA4"},
4251 {AMDGPU_GFX_TCX_DST_FIFOA5, "TCX_DST_FIFOA5"},
4252 {AMDGPU_GFX_TCX_DST_FIFOA6, "TCX_DST_FIFOA6"},
4253 {AMDGPU_GFX_TCX_DST_FIFOA7, "TCX_DST_FIFOA7"},
4254 {AMDGPU_GFX_TCX_DST_FIFOB0, "TCX_DST_FIFOB0"},
4255 {AMDGPU_GFX_TCX_DST_FIFOB1, "TCX_DST_FIFOB1"},
4256 {AMDGPU_GFX_TCX_DST_FIFOB2, "TCX_DST_FIFOB2"},
4257 {AMDGPU_GFX_TCX_DST_FIFOB3, "TCX_DST_FIFOB3"},
4258 {AMDGPU_GFX_TCX_DST_FIFOB4, "TCX_DST_FIFOB4"},
4259 {AMDGPU_GFX_TCX_DST_FIFOB5, "TCX_DST_FIFOB5"},
4260 {AMDGPU_GFX_TCX_DST_FIFOB6, "TCX_DST_FIFOB6"},
4261 {AMDGPU_GFX_TCX_DST_FIFOB7, "TCX_DST_FIFOB7"},
4262 {AMDGPU_GFX_TCX_DST_FIFOD0, "TCX_DST_FIFOD0"},
4263 {AMDGPU_GFX_TCX_DST_FIFOD1, "TCX_DST_FIFOD1"},
4264 {AMDGPU_GFX_TCX_DST_FIFOD2, "TCX_DST_FIFOD2"},
4265 {AMDGPU_GFX_TCX_DST_FIFOD3, "TCX_DST_FIFOD3"},
4266 {AMDGPU_GFX_TCX_DST_FIFOD4, "TCX_DST_FIFOD4"},
4267 {AMDGPU_GFX_TCX_DST_FIFOD5, "TCX_DST_FIFOD5"},
4268 {AMDGPU_GFX_TCX_DST_FIFOD6, "TCX_DST_FIFOD6"},
4269 {AMDGPU_GFX_TCX_DST_FIFOD7, "TCX_DST_FIFOD7"},
4270 {AMDGPU_GFX_TCX_DST_FIFO_ACKB0, "TCX_DST_FIFO_ACKB0"},
4271 {AMDGPU_GFX_TCX_DST_FIFO_ACKB1, "TCX_DST_FIFO_ACKB1"},
4272 {AMDGPU_GFX_TCX_DST_FIFO_ACKB2, "TCX_DST_FIFO_ACKB2"},
4273 {AMDGPU_GFX_TCX_DST_FIFO_ACKB3, "TCX_DST_FIFO_ACKB3"},
4274 {AMDGPU_GFX_TCX_DST_FIFO_ACKB4, "TCX_DST_FIFO_ACKB4"},
4275 {AMDGPU_GFX_TCX_DST_FIFO_ACKB5, "TCX_DST_FIFO_ACKB5"},
4276 {AMDGPU_GFX_TCX_DST_FIFO_ACKB6, "TCX_DST_FIFO_ACKB6"},
4277 {AMDGPU_GFX_TCX_DST_FIFO_ACKB7, "TCX_DST_FIFO_ACKB7"},
4278 {AMDGPU_GFX_TCX_DST_FIFO_ACKD0, "TCX_DST_FIFO_ACKD0"},
4279 {AMDGPU_GFX_TCX_DST_FIFO_ACKD1, "TCX_DST_FIFO_ACKD1"},
4280 {AMDGPU_GFX_TCX_DST_FIFO_ACKD2, "TCX_DST_FIFO_ACKD2"},
4281 {AMDGPU_GFX_TCX_DST_FIFO_ACKD3, "TCX_DST_FIFO_ACKD3"},
4282 {AMDGPU_GFX_TCX_DST_FIFO_ACKD4, "TCX_DST_FIFO_ACKD4"},
4283 {AMDGPU_GFX_TCX_DST_FIFO_ACKD5, "TCX_DST_FIFO_ACKD5"},
4284 {AMDGPU_GFX_TCX_DST_FIFO_ACKD6, "TCX_DST_FIFO_ACKD6"},
4285 {AMDGPU_GFX_TCX_DST_FIFO_ACKD7, "TCX_DST_FIFO_ACKD7"},
4286};
4287
4288static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_atc_l2_mem_list[] = {
4289 {AMDGPU_GFX_ATC_L2_MEM, "ATC_L2_MEM"},
4290};
4291
4292static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_utcl2_mem_list[] = {
4293 {AMDGPU_GFX_UTCL2_MEM, "UTCL2_MEM"},
4294};
4295
4296static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_vml2_mem_list[] = {
4297 {AMDGPU_GFX_VML2_MEM, "VML2_MEM"},
4298};
4299
4300static const struct amdgpu_ras_memory_id_entry gfx_v9_4_3_ras_vml2_walker_mem_list[] = {
4301 {AMDGPU_GFX_VML2_WALKER_MEM, "VML2_WALKER_MEM"},
4302};
4303
4304static const struct amdgpu_gfx_ras_mem_id_entry gfx_v9_4_3_ras_mem_list_array[AMDGPU_GFX_MEM_TYPE_NUM] = {
4305 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_cp_mem_list)
4306 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_gcea_mem_list)
4307 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_gc_cane_mem_list)
4308 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_gcutcl2_mem_list)
4309 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_gds_mem_list)
4310 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_lds_mem_list)
4311 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_rlc_mem_list)
4312 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_sp_mem_list)
4313 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_spi_mem_list)
4314 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_sqc_mem_list)
4315 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_sq_mem_list)
4316 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_ta_mem_list)
4317 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_tcc_mem_list)
4318 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_tca_mem_list)
4319 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_tci_mem_list)
4320 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_tcp_mem_list)
4321 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_td_mem_list)
4322 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_tcx_mem_list)
4323 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_atc_l2_mem_list)
4324 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_utcl2_mem_list)
4325 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_vml2_mem_list)
4326 AMDGPU_GFX_MEMID_ENT(gfx_v9_4_3_ras_vml2_walker_mem_list)
4327};
4328
4329static const struct amdgpu_gfx_ras_reg_entry gfx_v9_4_3_ce_reg_list[] = {
4330 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regRLC_CE_ERR_STATUS_LOW, regRLC_CE_ERR_STATUS_HIGH),
4331 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "RLC"},
4332 AMDGPU_GFX_RLC_MEM, 1},
4333 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regCPC_CE_ERR_STATUS_LO, regCPC_CE_ERR_STATUS_HI),
4334 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CPC"},
4335 AMDGPU_GFX_CP_MEM, 1},
4336 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regCPF_CE_ERR_STATUS_LO, regCPF_CE_ERR_STATUS_HI),
4337 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CPF"},
4338 AMDGPU_GFX_CP_MEM, 1},
4339 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regCPG_CE_ERR_STATUS_LO, regCPG_CE_ERR_STATUS_HI),
4340 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CPG"},
4341 AMDGPU_GFX_CP_MEM, 1},
4342 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regGDS_CE_ERR_STATUS_LO, regGDS_CE_ERR_STATUS_HI),
4343 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "GDS"},
4344 AMDGPU_GFX_GDS_MEM, 1},
4345 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regGC_CANE_CE_ERR_STATUS_LO, regGC_CANE_CE_ERR_STATUS_HI),
4346 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CANE"},
4347 AMDGPU_GFX_GC_CANE_MEM, 1},
4348 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSPI_CE_ERR_STATUS_LO, regSPI_CE_ERR_STATUS_HI),
4349 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SPI"},
4350 AMDGPU_GFX_SPI_MEM, 1},
4351 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSP0_CE_ERR_STATUS_LO, regSP0_CE_ERR_STATUS_HI),
4352 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SP0"},
4353 AMDGPU_GFX_SP_MEM, 4},
4354 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSP1_CE_ERR_STATUS_LO, regSP1_CE_ERR_STATUS_HI),
4355 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SP1"},
4356 AMDGPU_GFX_SP_MEM, 4},
4357 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSQ_CE_ERR_STATUS_LO, regSQ_CE_ERR_STATUS_HI),
4358 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SQ"},
4359 AMDGPU_GFX_SQ_MEM, 4},
4360 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSQC_CE_EDC_LO, regSQC_CE_EDC_HI),
4361 5, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SQC"},
4362 AMDGPU_GFX_SQC_MEM, 4},
4363 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCX_CE_ERR_STATUS_LO, regTCX_CE_ERR_STATUS_HI),
4364 2, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCX"},
4365 AMDGPU_GFX_TCX_MEM, 1},
4366 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCC_CE_ERR_STATUS_LO, regTCC_CE_ERR_STATUS_HI),
4367 16, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCC"},
4368 AMDGPU_GFX_TCC_MEM, 1},
4369 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTA_CE_EDC_LO, regTA_CE_EDC_HI),
4370 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TA"},
4371 AMDGPU_GFX_TA_MEM, 4},
4372 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCI_CE_EDC_LO_REG, regTCI_CE_EDC_HI_REG),
4373 27, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCI"},
4374 AMDGPU_GFX_TCI_MEM, 1},
4375 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCP_CE_EDC_LO_REG, regTCP_CE_EDC_HI_REG),
4376 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCP"},
4377 AMDGPU_GFX_TCP_MEM, 4},
4378 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTD_CE_EDC_LO, regTD_CE_EDC_HI),
4379 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TD"},
4380 AMDGPU_GFX_TD_MEM, 4},
4381 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regGCEA_CE_ERR_STATUS_LO, regGCEA_CE_ERR_STATUS_HI),
4382 16, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "GCEA"},
4383 AMDGPU_GFX_GCEA_MEM, 1},
4384 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regLDS_CE_ERR_STATUS_LO, regLDS_CE_ERR_STATUS_HI),
4385 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "LDS"},
4386 AMDGPU_GFX_LDS_MEM, 4},
4387};
4388
4389static const struct amdgpu_gfx_ras_reg_entry gfx_v9_4_3_ue_reg_list[] = {
4390 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regRLC_UE_ERR_STATUS_LOW, regRLC_UE_ERR_STATUS_HIGH),
4391 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "RLC"},
4392 AMDGPU_GFX_RLC_MEM, 1},
4393 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regCPC_UE_ERR_STATUS_LO, regCPC_UE_ERR_STATUS_HI),
4394 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CPC"},
4395 AMDGPU_GFX_CP_MEM, 1},
4396 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regCPF_UE_ERR_STATUS_LO, regCPF_UE_ERR_STATUS_HI),
4397 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CPF"},
4398 AMDGPU_GFX_CP_MEM, 1},
4399 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regCPG_UE_ERR_STATUS_LO, regCPG_UE_ERR_STATUS_HI),
4400 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CPG"},
4401 AMDGPU_GFX_CP_MEM, 1},
4402 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regGDS_UE_ERR_STATUS_LO, regGDS_UE_ERR_STATUS_HI),
4403 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "GDS"},
4404 AMDGPU_GFX_GDS_MEM, 1},
4405 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regGC_CANE_UE_ERR_STATUS_LO, regGC_CANE_UE_ERR_STATUS_HI),
4406 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "CANE"},
4407 AMDGPU_GFX_GC_CANE_MEM, 1},
4408 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSPI_UE_ERR_STATUS_LO, regSPI_UE_ERR_STATUS_HI),
4409 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SPI"},
4410 AMDGPU_GFX_SPI_MEM, 1},
4411 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSP0_UE_ERR_STATUS_LO, regSP0_UE_ERR_STATUS_HI),
4412 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SP0"},
4413 AMDGPU_GFX_SP_MEM, 4},
4414 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSP1_UE_ERR_STATUS_LO, regSP1_UE_ERR_STATUS_HI),
4415 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SP1"},
4416 AMDGPU_GFX_SP_MEM, 4},
4417 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSQ_UE_ERR_STATUS_LO, regSQ_UE_ERR_STATUS_HI),
4418 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SQ"},
4419 AMDGPU_GFX_SQ_MEM, 4},
4420 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regSQC_UE_EDC_LO, regSQC_UE_EDC_HI),
4421 5, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SQC"},
4422 AMDGPU_GFX_SQC_MEM, 4},
4423 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCX_UE_ERR_STATUS_LO, regTCX_UE_ERR_STATUS_HI),
4424 2, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCX"},
4425 AMDGPU_GFX_TCX_MEM, 1},
4426 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCC_UE_ERR_STATUS_LO, regTCC_UE_ERR_STATUS_HI),
4427 16, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCC"},
4428 AMDGPU_GFX_TCC_MEM, 1},
4429 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTA_UE_EDC_LO, regTA_UE_EDC_HI),
4430 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TA"},
4431 AMDGPU_GFX_TA_MEM, 4},
4432 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCI_UE_EDC_LO_REG, regTCI_UE_EDC_HI_REG),
4433 27, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCI"},
4434 AMDGPU_GFX_TCI_MEM, 1},
4435 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCP_UE_EDC_LO_REG, regTCP_UE_EDC_HI_REG),
4436 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCP"},
4437 AMDGPU_GFX_TCP_MEM, 4},
4438 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTD_UE_EDC_LO, regTD_UE_EDC_HI),
4439 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TD"},
4440 AMDGPU_GFX_TD_MEM, 4},
4441 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regTCA_UE_ERR_STATUS_LO, regTCA_UE_ERR_STATUS_HI),
4442 2, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "TCA"},
4443 AMDGPU_GFX_TCA_MEM, 1},
4444 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regGCEA_UE_ERR_STATUS_LO, regGCEA_UE_ERR_STATUS_HI),
4445 16, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "GCEA"},
4446 AMDGPU_GFX_GCEA_MEM, 1},
4447 {{AMDGPU_RAS_REG_ENTRY(GC, 0, regLDS_UE_ERR_STATUS_LO, regLDS_UE_ERR_STATUS_HI),
4448 10, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "LDS"},
4449 AMDGPU_GFX_LDS_MEM, 4},
4450};
4451
4452static void gfx_v9_4_3_inst_query_ras_err_count(struct amdgpu_device *adev,
4453 void *ras_error_status, int xcc_id)
4454{
4455 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;
4456 unsigned long ce_count = 0, ue_count = 0;
4457 uint32_t i, j, k;
4458
4459 /* NOTE: convert xcc_id to physical XCD ID (XCD0 or XCD1) */
4460 struct amdgpu_smuio_mcm_config_info mcm_info = {
4461 .socket_id = adev->smuio.funcs->get_socket_id(adev),
4462 .die_id = xcc_id & 0x01 ? 1 : 0,
4463 };
4464
4465 mutex_lock(&adev->grbm_idx_mutex);
4466
4467 for (i = 0; i < ARRAY_SIZE(gfx_v9_4_3_ce_reg_list); i++) {
4468 for (j = 0; j < gfx_v9_4_3_ce_reg_list[i].se_num; j++) {
4469 for (k = 0; k < gfx_v9_4_3_ce_reg_list[i].reg_entry.reg_inst; k++) {
4470 /* no need to select if instance number is 1 */
4471 if (gfx_v9_4_3_ce_reg_list[i].se_num > 1 ||
4472 gfx_v9_4_3_ce_reg_list[i].reg_entry.reg_inst > 1)
4473 gfx_v9_4_3_xcc_select_se_sh(adev, j, 0, k, xcc_id);
4474
4475 amdgpu_ras_inst_query_ras_error_count(adev,
4476 &(gfx_v9_4_3_ce_reg_list[i].reg_entry),
4477 1,
4478 gfx_v9_4_3_ras_mem_list_array[gfx_v9_4_3_ce_reg_list[i].mem_id_type].mem_id_ent,
4479 gfx_v9_4_3_ras_mem_list_array[gfx_v9_4_3_ce_reg_list[i].mem_id_type].size,
4480 GET_INST(GC, xcc_id),
4481 AMDGPU_RAS_ERROR__SINGLE_CORRECTABLE,
4482 &ce_count);
4483
4484 amdgpu_ras_inst_query_ras_error_count(adev,
4485 &(gfx_v9_4_3_ue_reg_list[i].reg_entry),
4486 1,
4487 gfx_v9_4_3_ras_mem_list_array[gfx_v9_4_3_ue_reg_list[i].mem_id_type].mem_id_ent,
4488 gfx_v9_4_3_ras_mem_list_array[gfx_v9_4_3_ue_reg_list[i].mem_id_type].size,
4489 GET_INST(GC, xcc_id),
4490 AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
4491 &ue_count);
4492 }
4493 }
4494 }
4495
4496 /* handle extra register entries of UE */
4497 for (; i < ARRAY_SIZE(gfx_v9_4_3_ue_reg_list); i++) {
4498 for (j = 0; j < gfx_v9_4_3_ue_reg_list[i].se_num; j++) {
4499 for (k = 0; k < gfx_v9_4_3_ue_reg_list[i].reg_entry.reg_inst; k++) {
4500 /* no need to select if instance number is 1 */
4501 if (gfx_v9_4_3_ue_reg_list[i].se_num > 1 ||
4502 gfx_v9_4_3_ue_reg_list[i].reg_entry.reg_inst > 1)
4503 gfx_v9_4_3_xcc_select_se_sh(adev, j, 0, k, xcc_id);
4504
4505 amdgpu_ras_inst_query_ras_error_count(adev,
4506 &(gfx_v9_4_3_ue_reg_list[i].reg_entry),
4507 1,
4508 gfx_v9_4_3_ras_mem_list_array[gfx_v9_4_3_ue_reg_list[i].mem_id_type].mem_id_ent,
4509 gfx_v9_4_3_ras_mem_list_array[gfx_v9_4_3_ue_reg_list[i].mem_id_type].size,
4510 GET_INST(GC, xcc_id),
4511 AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
4512 &ue_count);
4513 }
4514 }
4515 }
4516
4517 gfx_v9_4_3_xcc_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff,
4518 xcc_id);
4519 mutex_unlock(&adev->grbm_idx_mutex);
4520
4521 /* the caller should make sure initialize value of
4522 * err_data->ue_count and err_data->ce_count
4523 */
4524 amdgpu_ras_error_statistic_ue_count(err_data, &mcm_info, ue_count);
4525 amdgpu_ras_error_statistic_ce_count(err_data, &mcm_info, ce_count);
4526}
4527
4528static void gfx_v9_4_3_inst_reset_ras_err_count(struct amdgpu_device *adev,
4529 void *ras_error_status, int xcc_id)
4530{
4531 uint32_t i, j, k;
4532
4533 mutex_lock(&adev->grbm_idx_mutex);
4534
4535 for (i = 0; i < ARRAY_SIZE(gfx_v9_4_3_ce_reg_list); i++) {
4536 for (j = 0; j < gfx_v9_4_3_ce_reg_list[i].se_num; j++) {
4537 for (k = 0; k < gfx_v9_4_3_ce_reg_list[i].reg_entry.reg_inst; k++) {
4538 /* no need to select if instance number is 1 */
4539 if (gfx_v9_4_3_ce_reg_list[i].se_num > 1 ||
4540 gfx_v9_4_3_ce_reg_list[i].reg_entry.reg_inst > 1)
4541 gfx_v9_4_3_xcc_select_se_sh(adev, j, 0, k, xcc_id);
4542
4543 amdgpu_ras_inst_reset_ras_error_count(adev,
4544 &(gfx_v9_4_3_ce_reg_list[i].reg_entry),
4545 1,
4546 GET_INST(GC, xcc_id));
4547
4548 amdgpu_ras_inst_reset_ras_error_count(adev,
4549 &(gfx_v9_4_3_ue_reg_list[i].reg_entry),
4550 1,
4551 GET_INST(GC, xcc_id));
4552 }
4553 }
4554 }
4555
4556 /* handle extra register entries of UE */
4557 for (; i < ARRAY_SIZE(gfx_v9_4_3_ue_reg_list); i++) {
4558 for (j = 0; j < gfx_v9_4_3_ue_reg_list[i].se_num; j++) {
4559 for (k = 0; k < gfx_v9_4_3_ue_reg_list[i].reg_entry.reg_inst; k++) {
4560 /* no need to select if instance number is 1 */
4561 if (gfx_v9_4_3_ue_reg_list[i].se_num > 1 ||
4562 gfx_v9_4_3_ue_reg_list[i].reg_entry.reg_inst > 1)
4563 gfx_v9_4_3_xcc_select_se_sh(adev, j, 0, k, xcc_id);
4564
4565 amdgpu_ras_inst_reset_ras_error_count(adev,
4566 &(gfx_v9_4_3_ue_reg_list[i].reg_entry),
4567 1,
4568 GET_INST(GC, xcc_id));
4569 }
4570 }
4571 }
4572
4573 gfx_v9_4_3_xcc_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff,
4574 xcc_id);
4575 mutex_unlock(&adev->grbm_idx_mutex);
4576}
4577
4578static void gfx_v9_4_3_inst_enable_watchdog_timer(struct amdgpu_device *adev,3729static void gfx_v9_4_3_inst_enable_watchdog_timer(struct amdgpu_device *adev,
4579 void *ras_error_status, int xcc_id)3730 void *ras_error_status, int xcc_id)
4580{3731{
@@ -4607,18 +3758,6 @@ static void gfx_v9_4_3_inst_enable_watchdog_timer(struct amdgpu_device *adev, · 26 unmodified lines
4607 mutex_unlock(&adev->grbm_idx_mutex);3758 mutex_unlock(&adev->grbm_idx_mutex);
4608}3759}
4609 3760
4610static void gfx_v9_4_3_query_ras_error_count(struct amdgpu_device *adev,
4611 void *ras_error_status)
4612{
4613 amdgpu_gfx_ras_error_func(adev, ras_error_status,
4614 gfx_v9_4_3_inst_query_ras_err_count);
4615}
4616
4617static void gfx_v9_4_3_reset_ras_error_count(struct amdgpu_device *adev)
4618{
4619 amdgpu_gfx_ras_error_func(adev, NULL, gfx_v9_4_3_inst_reset_ras_err_count);
4620}
4621
4622static void gfx_v9_4_3_enable_watchdog_timer(struct amdgpu_device *adev)3761static void gfx_v9_4_3_enable_watchdog_timer(struct amdgpu_device *adev)
4623{3762{
4624 amdgpu_gfx_ras_error_func(adev, NULL, gfx_v9_4_3_inst_enable_watchdog_timer);3763 amdgpu_gfx_ras_error_func(adev, NULL, gfx_v9_4_3_inst_enable_watchdog_timer);
@@ -5099,37 +4238,9 @@ struct amdgpu_xcp_ip_funcs gfx_v9_4_3_xcp_funcs = { · 474 unmodified lines
5099 .resume = &gfx_v9_4_3_xcp_resume4238 .resume = &gfx_v9_4_3_xcp_resume
5100};4239};
5101 4240
5102struct amdgpu_ras_block_hw_ops gfx_v9_4_3_ras_ops = {
5103 .query_ras_error_count = &gfx_v9_4_3_query_ras_error_count,
5104 .reset_ras_error_count = &gfx_v9_4_3_reset_ras_error_count,
5105};
5106
5107static int gfx_v9_4_3_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
5108{
5109 int r;
5110
5111 r = amdgpu_ras_block_late_init(adev, ras_block);
5112 if (r)
5113 return r;
5114
5115 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__GFX,
5116 &gfx_v9_4_3_aca_info,
5117 NULL);
5118 if (r)
5119 goto late_fini;
5120
5121 return 0;
5122
5123late_fini:
5124 amdgpu_ras_block_late_fini(adev, ras_block);
5125
5126 return r;
5127}
5128
5129struct amdgpu_gfx_ras gfx_v9_4_3_ras = {4241struct amdgpu_gfx_ras gfx_v9_4_3_ras = {
5130 .ras_block = {4242 .ras_block = {
5131 .hw_ops = &gfx_v9_4_3_ras_ops,4243 .hw_ops = NULL,
5132 .ras_late_init = &gfx_v9_4_3_ras_late_init,
5133 },4244 },
5134 .enable_watchdog_timer = &gfx_v9_4_3_enable_watchdog_timer,4245 .enable_watchdog_timer = &gfx_v9_4_3_enable_watchdog_timer,
5135};4246};
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c+0−125
@@ -167,44 +167,6 @@ static void gmc_v8_0_init_golden_registers(struct amdgpu_device *adev) · 166 unmodified lines
167 }167 }
168}168}
169 169
170static void gmc_v8_0_mc_stop(struct amdgpu_device *adev)
171{
172 u32 blackout;
173 struct amdgpu_ip_block *ip_block;
174
175 ip_block = amdgpu_device_ip_get_ip_block(adev, AMD_IP_BLOCK_TYPE_GMC);
176 if (!ip_block)
177 return;
178
179 gmc_v8_0_wait_for_idle(ip_block);
180
181 blackout = RREG32(mmMC_SHARED_BLACKOUT_CNTL);
182 if (REG_GET_FIELD(blackout, MC_SHARED_BLACKOUT_CNTL, BLACKOUT_MODE) != 1) {
183 /* Block CPU access */
184 WREG32(mmBIF_FB_EN, 0);
185 /* blackout the MC */
186 blackout = REG_SET_FIELD(blackout,
187 MC_SHARED_BLACKOUT_CNTL, BLACKOUT_MODE, 1);
188 WREG32(mmMC_SHARED_BLACKOUT_CNTL, blackout);
189 }
190 /* wait for the MC to settle */
191 udelay(100);
192}
193
194static void gmc_v8_0_mc_resume(struct amdgpu_device *adev)
195{
196 u32 tmp;
197
198 /* unblackout the MC */
199 tmp = RREG32(mmMC_SHARED_BLACKOUT_CNTL);
200 tmp = REG_SET_FIELD(tmp, MC_SHARED_BLACKOUT_CNTL, BLACKOUT_MODE, 0);
201 WREG32(mmMC_SHARED_BLACKOUT_CNTL, tmp);
202 /* allow CPU access */
203 tmp = REG_SET_FIELD(0, BIF_FB_EN, FB_READ_EN, 1);
204 tmp = REG_SET_FIELD(tmp, BIF_FB_EN, FB_WRITE_EN, 1);
205 WREG32(mmBIF_FB_EN, tmp);
206}
207
208/**170/**
209 * gmc_v8_0_init_microcode - load ucode images from disk171 * gmc_v8_0_init_microcode - load ucode images from disk
210 *172 *
@@ -1293,89 +1255,6 @@ static int gmc_v8_0_wait_for_idle(struct amdgpu_ip_block *ip_block) · 1082 unmodified lines
1293 1255
1294}1256}
1295 1257
1296static bool gmc_v8_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
1297{
1298 u32 srbm_soft_reset = 0;
1299 struct amdgpu_device *adev = ip_block->adev;
1300 u32 tmp = RREG32(mmSRBM_STATUS);
1301
1302 if (tmp & SRBM_STATUS__VMC_BUSY_MASK)
1303 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset,
1304 SRBM_SOFT_RESET, SOFT_RESET_VMC, 1);
1305
1306 if (tmp & (SRBM_STATUS__MCB_BUSY_MASK | SRBM_STATUS__MCB_NON_DISPLAY_BUSY_MASK |
1307 SRBM_STATUS__MCC_BUSY_MASK | SRBM_STATUS__MCD_BUSY_MASK)) {
1308 if (!(adev->flags & AMD_IS_APU))
1309 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset,
1310 SRBM_SOFT_RESET, SOFT_RESET_MC, 1);
1311 }
1312
1313 if (srbm_soft_reset) {
1314 adev->gmc.srbm_soft_reset = srbm_soft_reset;
1315 return true;
1316 }
1317
1318 adev->gmc.srbm_soft_reset = 0;
1319
1320 return false;
1321}
1322
1323static int gmc_v8_0_pre_soft_reset(struct amdgpu_ip_block *ip_block)
1324{
1325 struct amdgpu_device *adev = ip_block->adev;
1326
1327 if (!adev->gmc.srbm_soft_reset)
1328 return 0;
1329
1330 gmc_v8_0_mc_stop(adev);
1331 if (gmc_v8_0_wait_for_idle(ip_block))
1332 dev_warn(adev->dev, "Wait for GMC idle timed out !\n");
1333
1334 return 0;
1335}
1336
1337static int gmc_v8_0_soft_reset(struct amdgpu_ip_block *ip_block)
1338{
1339 struct amdgpu_device *adev = ip_block->adev;
1340 u32 srbm_soft_reset;
1341
1342 if (!adev->gmc.srbm_soft_reset)
1343 return 0;
1344 srbm_soft_reset = adev->gmc.srbm_soft_reset;
1345
1346 if (srbm_soft_reset) {
1347 u32 tmp;
1348
1349 tmp = RREG32(mmSRBM_SOFT_RESET);
1350 tmp |= srbm_soft_reset;
1351 dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp);
1352 WREG32(mmSRBM_SOFT_RESET, tmp);
1353 tmp = RREG32(mmSRBM_SOFT_RESET);
1354
1355 udelay(50);
1356
1357 tmp &= ~srbm_soft_reset;
1358 WREG32(mmSRBM_SOFT_RESET, tmp);
1359 tmp = RREG32(mmSRBM_SOFT_RESET);
1360
1361 /* Wait a little for things to settle down */
1362 udelay(50);
1363 }
1364
1365 return 0;
1366}
1367
1368static int gmc_v8_0_post_soft_reset(struct amdgpu_ip_block *ip_block)
1369{
1370 struct amdgpu_device *adev = ip_block->adev;
1371
1372 if (!adev->gmc.srbm_soft_reset)
1373 return 0;
1374
1375 gmc_v8_0_mc_resume(adev);
1376 return 0;
1377}
1378
1379static int gmc_v8_0_vm_fault_interrupt_state(struct amdgpu_device *adev,1258static int gmc_v8_0_vm_fault_interrupt_state(struct amdgpu_device *adev,
1380 struct amdgpu_irq_src *src,1259 struct amdgpu_irq_src *src,
1381 unsigned int type,1260 unsigned int type,
@@ -1715,10 +1594,6 @@ static const struct amd_ip_funcs gmc_v8_0_ip_funcs = { · 333 unmodified lines
1715 .resume = gmc_v8_0_resume,1594 .resume = gmc_v8_0_resume,
1716 .is_idle = gmc_v8_0_is_idle,1595 .is_idle = gmc_v8_0_is_idle,
1717 .wait_for_idle = gmc_v8_0_wait_for_idle,1596 .wait_for_idle = gmc_v8_0_wait_for_idle,
1718 .check_soft_reset = gmc_v8_0_check_soft_reset,
1719 .pre_soft_reset = gmc_v8_0_pre_soft_reset,
1720 .soft_reset = gmc_v8_0_soft_reset,
1721 .post_soft_reset = gmc_v8_0_post_soft_reset,
1722 .set_clockgating_state = gmc_v8_0_set_clockgating_state,1597 .set_clockgating_state = gmc_v8_0_set_clockgating_state,
1723 .set_powergating_state = gmc_v8_0_set_powergating_state,1598 .set_powergating_state = gmc_v8_0_set_powergating_state,
1724 .get_clockgating_state = gmc_v8_0_get_clockgating_state,1599 .get_clockgating_state = gmc_v8_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c+14−5
@@ -57,6 +57,7 @@ · 56 unmodified lines
57#include "umc_v6_0.h"57#include "umc_v6_0.h"
58#include "umc_v6_7.h"58#include "umc_v6_7.h"
59#include "umc_v12_0.h"59#include "umc_v12_0.h"
60#include "ras_umc_v12_0.h"
60#include "hdp_v4_0.h"61#include "hdp_v4_0.h"
61#include "mca_v3_0.h"62#include "mca_v3_0.h"
62 63
@@ -1382,7 +1383,7 @@ static void gmc_v9_0_set_umc_funcs(struct amdgpu_device *adev) · 1319 unmodified lines
1382 case IP_VERSION(12, 0, 0):1383 case IP_VERSION(12, 0, 0):
1383 case IP_VERSION(12, 5, 0):1384 case IP_VERSION(12, 5, 0):
1384 adev->umc.max_ras_err_cnt_per_query =1385 adev->umc.max_ras_err_cnt_per_query =
1385 UMC_V12_0_TOTAL_CHANNEL_NUM(adev) * UMC_V12_0_BAD_PAGE_NUM_PER_CHANNEL;1386 UMC_V12_0_TOTAL_CHANNEL_NUM * UMC_V12_0_BAD_PAGE_NUM_PER_CHANNEL;
1386 adev->umc.channel_inst_num = UMC_V12_0_CHANNEL_INSTANCE_NUM;1387 adev->umc.channel_inst_num = UMC_V12_0_CHANNEL_INSTANCE_NUM;
1387 adev->umc.umc_inst_num = UMC_V12_0_UMC_INSTANCE_NUM;1388 adev->umc.umc_inst_num = UMC_V12_0_UMC_INSTANCE_NUM;
1388 adev->umc.node_inst_num /= UMC_V12_0_UMC_INSTANCE_NUM;1389 adev->umc.node_inst_num /= UMC_V12_0_UMC_INSTANCE_NUM;
@@ -2025,11 +2026,19 @@ static int gmc_v9_0_sw_init(struct amdgpu_ip_block *ip_block) · 636 unmodified lines
2025 * The first KFD VMID is 8 for GPUs with graphics, 3 for2026 * The first KFD VMID is 8 for GPUs with graphics, 3 for
2026 * compute-only GPUs. On compute-only GPUs that leaves 2 VMIDs2027 * compute-only GPUs. On compute-only GPUs that leaves 2 VMIDs
2027 * for video processing.2028 * for video processing.
2029 *
2030 * If kernel queues are disabled, allow KFD to use all vmids.
2028 */2031 */
2029 adev->vm_manager.first_kfd_vmid =2032 if (adev->gfx.disable_kq &&
2030 (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 1) ||2033 adev->jpeg.disable_kq &&
2031 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 2) ||2034 adev->vcn.disable_kq &&
2032 amdgpu_is_multi_aid(adev)) ?2035 adev->sdma.no_user_submission)
2036 adev->vm_manager.first_kfd_vmid = 1;
2037 else
2038 adev->vm_manager.first_kfd_vmid =
2039 (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 1) ||
2040 amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 2) ||
2041 amdgpu_is_multi_aid(adev)) ?
2033 3 :2042 3 :
2034 8;2043 8;
2035 2044
drivers/gpu/drm/amd/amdgpu/jpeg_v4_0_3.c+17−71
@@ -119,6 +119,19 @@ static int jpeg_v4_0_3_early_init(struct amdgpu_ip_block *ip_block) · 118 unmodified lines
119{119{
120 struct amdgpu_device *adev = ip_block->adev;120 struct amdgpu_device *adev = ip_block->adev;
121 121
122 switch (amdgpu_user_queue) {
123 case -1:
124 case 0:
125 default:
126 adev->jpeg.disable_kq = false;
127 adev->jpeg.disable_uq = true;
128 break;
129 case 2:
130 adev->jpeg.disable_kq = true;
131 adev->jpeg.disable_uq = true;
132 break;
133 }
134
122 adev->jpeg.num_jpeg_rings = AMDGPU_MAX_JPEG_RINGS_4_0_3;135 adev->jpeg.num_jpeg_rings = AMDGPU_MAX_JPEG_RINGS_4_0_3;
123 136
124 jpeg_v4_0_3_set_dec_ring_funcs(adev);137 jpeg_v4_0_3_set_dec_ring_funcs(adev);
@@ -175,6 +188,10 @@ static int jpeg_v4_0_3_sw_init(struct amdgpu_ip_block *ip_block) · 50 unmodified lines
175 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {188 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {
176 ring = &adev->jpeg.inst[i].ring_dec[j];189 ring = &adev->jpeg.inst[i].ring_dec[j];
177 ring->use_doorbell = true;190 ring->use_doorbell = true;
191 if (adev->jpeg.disable_kq) {
192 ring->no_scheduler = true;
193 ring->no_user_submission = true;
194 }
178 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);195 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);
179 if (!amdgpu_sriov_vf(adev)) {196 if (!amdgpu_sriov_vf(adev)) {
180 ring->doorbell_index =197 ring->doorbell_index =
@@ -1425,72 +1442,6 @@ static const struct amdgpu_ras_block_hw_ops jpeg_v4_0_3_ras_hw_ops = { · 1244 unmodified lines
1425 .query_poison_status = jpeg_v4_0_3_query_ras_poison_status,1442 .query_poison_status = jpeg_v4_0_3_query_ras_poison_status,
1426};1443};
1427 1444
1428static int jpeg_v4_0_3_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
1429 enum aca_smu_type type, void *data)
1430{
1431 struct aca_bank_info info;
1432 u64 misc0;
1433 int ret;
1434
1435 ret = aca_bank_info_decode(bank, &info);
1436 if (ret)
1437 return ret;
1438
1439 misc0 = bank->regs[ACA_REG_IDX_MISC0];
1440 switch (type) {
1441 case ACA_SMU_TYPE_UE:
1442 bank->aca_err_type = ACA_ERROR_TYPE_UE;
1443 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE,
1444 1ULL);
1445 break;
1446 case ACA_SMU_TYPE_CE:
1447 bank->aca_err_type = ACA_ERROR_TYPE_CE;
1448 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
1449 ACA_REG__MISC0__ERRCNT(misc0));
1450 break;
1451 default:
1452 return -EINVAL;
1453 }
1454
1455 return ret;
1456}
1457
1458/* reference to smu driver if header file */
1459static int jpeg_v4_0_3_err_codes[] = {
1460 16, 17, 18, 19, 20, 21, 22, 23, /* JPEG[0-7][S|D] */
1461 24, 25, 26, 27, 28, 29, 30, 31
1462};
1463
1464static bool jpeg_v4_0_3_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
1465 enum aca_smu_type type, void *data)
1466{
1467 u32 instlo;
1468
1469 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
1470 instlo &= GENMASK(31, 1);
1471
1472 if (instlo != mmSMNAID_AID0_MCA_SMU)
1473 return false;
1474
1475 if (aca_bank_check_error_codes(handle->adev, bank,
1476 jpeg_v4_0_3_err_codes,
1477 ARRAY_SIZE(jpeg_v4_0_3_err_codes)))
1478 return false;
1479
1480 return true;
1481}
1482
1483static const struct aca_bank_ops jpeg_v4_0_3_aca_bank_ops = {
1484 .aca_bank_parser = jpeg_v4_0_3_aca_bank_parser,
1485 .aca_bank_is_valid = jpeg_v4_0_3_aca_bank_is_valid,
1486};
1487
1488static const struct aca_info jpeg_v4_0_3_aca_info = {
1489 .hwip = ACA_HWIP_TYPE_SMU,
1490 .mask = ACA_ERROR_UE_MASK,
1491 .bank_ops = &jpeg_v4_0_3_aca_bank_ops,
1492};
1493
1494static int jpeg_v4_0_3_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)1445static int jpeg_v4_0_3_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
1495{1446{
1496 int r;1447 int r;
@@ -1506,11 +1457,6 @@ static int jpeg_v4_0_3_ras_late_init(struct amdgpu_device *adev, struct ras_comm · 9 unmodified lines
1506 goto late_fini;1457 goto late_fini;
1507 }1458 }
1508 1459
1509 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__JPEG,
1510 &jpeg_v4_0_3_aca_info, NULL);
1511 if (r)
1512 goto late_fini;
1513
1514 return 0;1460 return 0;
1515 1461
1516late_fini:1462late_fini:
drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_1.c+17−75
@@ -118,6 +118,19 @@ static int jpeg_v5_0_1_early_init(struct amdgpu_ip_block *ip_block) · 117 unmodified lines
118 if (!adev->jpeg.num_jpeg_inst || adev->jpeg.num_jpeg_inst > AMDGPU_MAX_JPEG_INSTANCES)118 if (!adev->jpeg.num_jpeg_inst || adev->jpeg.num_jpeg_inst > AMDGPU_MAX_JPEG_INSTANCES)
119 return -ENOENT;119 return -ENOENT;
120 120
121 switch (amdgpu_user_queue) {
122 case -1:
123 case 0:
124 default:
125 adev->jpeg.disable_kq = false;
126 adev->jpeg.disable_uq = true;
127 break;
128 case 2:
129 adev->jpeg.disable_kq = true;
130 adev->jpeg.disable_uq = true;
131 break;
132 }
133
121 adev->jpeg.num_jpeg_rings = AMDGPU_MAX_JPEG_RINGS;134 adev->jpeg.num_jpeg_rings = AMDGPU_MAX_JPEG_RINGS;
122 jpeg_v5_0_1_set_dec_ring_funcs(adev);135 jpeg_v5_0_1_set_dec_ring_funcs(adev);
123 jpeg_v5_0_1_set_irq_funcs(adev);136 jpeg_v5_0_1_set_irq_funcs(adev);
@@ -172,6 +185,10 @@ static int jpeg_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 48 unmodified lines
172 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {185 for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) {
173 ring = &adev->jpeg.inst[i].ring_dec[j];186 ring = &adev->jpeg.inst[i].ring_dec[j];
174 ring->use_doorbell = true;187 ring->use_doorbell = true;
188 if (adev->jpeg.disable_kq) {
189 ring->no_scheduler = true;
190 ring->no_user_submission = true;
191 }
175 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);192 ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id);
176 if (!amdgpu_sriov_vf(adev)) {193 if (!amdgpu_sriov_vf(adev)) {
177 ring->doorbell_index =194 ring->doorbell_index =
@@ -871,10 +888,7 @@ static const struct amd_ip_funcs jpeg_v5_0_1_ip_funcs = { · 693 unmodified lines
871 .resume = jpeg_v5_0_1_resume,888 .resume = jpeg_v5_0_1_resume,
872 .is_idle = jpeg_v5_0_1_is_idle,889 .is_idle = jpeg_v5_0_1_is_idle,
873 .wait_for_idle = jpeg_v5_0_1_wait_for_idle,890 .wait_for_idle = jpeg_v5_0_1_wait_for_idle,
874 .check_soft_reset = NULL,
875 .pre_soft_reset = NULL,
876 .soft_reset = NULL,891 .soft_reset = NULL,
877 .post_soft_reset = NULL,
878 .set_clockgating_state = jpeg_v5_0_1_set_clockgating_state,892 .set_clockgating_state = jpeg_v5_0_1_set_clockgating_state,
879 .set_powergating_state = jpeg_v5_0_1_set_powergating_state,893 .set_powergating_state = jpeg_v5_0_1_set_powergating_state,
880 .dump_ip_state = amdgpu_jpeg_dump_ip_state,894 .dump_ip_state = amdgpu_jpeg_dump_ip_state,
@@ -1003,73 +1017,6 @@ static const struct amdgpu_ras_block_hw_ops jpeg_v5_0_1_ras_hw_ops = { · 122 unmodified lines
1003 .query_poison_status = jpeg_v5_0_1_query_ras_poison_status,1017 .query_poison_status = jpeg_v5_0_1_query_ras_poison_status,
1004};1018};
1005 1019
1006static int jpeg_v5_0_1_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
1007 enum aca_smu_type type, void *data)
1008{
1009 struct aca_bank_info info;
1010 u64 misc0;
1011 int ret;
1012
1013 ret = aca_bank_info_decode(bank, &info);
1014 if (ret)
1015 return ret;
1016
1017 misc0 = bank->regs[ACA_REG_IDX_MISC0];
1018 switch (type) {
1019 case ACA_SMU_TYPE_UE:
1020 bank->aca_err_type = ACA_ERROR_TYPE_UE;
1021 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE,
1022 1ULL);
1023 break;
1024 case ACA_SMU_TYPE_CE:
1025 bank->aca_err_type = ACA_ERROR_TYPE_CE;
1026 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
1027 ACA_REG__MISC0__ERRCNT(misc0));
1028 break;
1029 default:
1030 return -EINVAL;
1031 }
1032
1033 return ret;
1034}
1035
1036/* reference to smu driver if header file */
1037static int jpeg_v5_0_1_err_codes[] = {
1038 16, 17, 18, 19, 20, 21, 22, 23, /* JPEG[0-9][S|D] */
1039 24, 25, 26, 27, 28, 29, 30, 31,
1040 48, 49, 50, 51,
1041};
1042
1043static bool jpeg_v5_0_1_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
1044 enum aca_smu_type type, void *data)
1045{
1046 u32 instlo;
1047
1048 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
1049 instlo &= GENMASK(31, 1);
1050
1051 if (instlo != mmSMNAID_AID0_MCA_SMU)
1052 return false;
1053
1054 if (aca_bank_check_error_codes(handle->adev, bank,
1055 jpeg_v5_0_1_err_codes,
1056 ARRAY_SIZE(jpeg_v5_0_1_err_codes)))
1057 return false;
1058
1059 return true;
1060}
1061
1062static const struct aca_bank_ops jpeg_v5_0_1_aca_bank_ops = {
1063 .aca_bank_parser = jpeg_v5_0_1_aca_bank_parser,
1064 .aca_bank_is_valid = jpeg_v5_0_1_aca_bank_is_valid,
1065};
1066
1067static const struct aca_info jpeg_v5_0_1_aca_info = {
1068 .hwip = ACA_HWIP_TYPE_SMU,
1069 .mask = ACA_ERROR_UE_MASK,
1070 .bank_ops = &jpeg_v5_0_1_aca_bank_ops,
1071};
1072
1073static int jpeg_v5_0_1_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)1020static int jpeg_v5_0_1_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
1074{1021{
1075 int r;1022 int r;
@@ -1078,11 +1025,6 @@ static int jpeg_v5_0_1_ras_late_init(struct amdgpu_device *adev, struct ras_comm · 2 unmodified lines
1078 if (r)1025 if (r)
1079 return r;1026 return r;
1080 1027
1081 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__JPEG,
1082 &jpeg_v5_0_1_aca_info, NULL);
1083 if (r)
1084 goto late_fini;
1085
1086 if (amdgpu_ras_is_supported(adev, ras_block->block) &&1028 if (amdgpu_ras_is_supported(adev, ras_block->block) &&
1087 adev->jpeg.inst->ras_poison_irq.funcs) {1029 adev->jpeg.inst->ras_poison_irq.funcs) {
1088 r = amdgpu_irq_get(adev, &adev->jpeg.inst->ras_poison_irq, 0);1030 r = amdgpu_irq_get(adev, &adev->jpeg.inst->ras_poison_irq, 0);
drivers/gpu/drm/amd/amdgpu/jpeg_v5_0_2.c+0−3
@@ -690,10 +690,7 @@ static const struct amd_ip_funcs jpeg_v5_0_2_ip_funcs = { · 689 unmodified lines
690 .resume = jpeg_v5_0_2_resume,690 .resume = jpeg_v5_0_2_resume,
691 .is_idle = jpeg_v5_0_2_is_idle,691 .is_idle = jpeg_v5_0_2_is_idle,
692 .wait_for_idle = jpeg_v5_0_2_wait_for_idle,692 .wait_for_idle = jpeg_v5_0_2_wait_for_idle,
693 .check_soft_reset = NULL,
694 .pre_soft_reset = NULL,
695 .soft_reset = NULL,693 .soft_reset = NULL,
696 .post_soft_reset = NULL,
697 .set_clockgating_state = jpeg_v5_0_2_set_clockgating_state,694 .set_clockgating_state = jpeg_v5_0_2_set_clockgating_state,
698 .set_powergating_state = jpeg_v5_0_2_set_powergating_state,695 .set_powergating_state = jpeg_v5_0_2_set_powergating_state,
699 .dump_ip_state = amdgpu_jpeg_dump_ip_state,696 .dump_ip_state = amdgpu_jpeg_dump_ip_state,
drivers/gpu/drm/amd/amdgpu/mes_userqueue.c+90−57
@@ -133,8 +133,8 @@ static int mes_userq_map(struct amdgpu_usermode_queue *queue) · 132 unmodified lines
133 queue_input.gang_quantum = 10000;133 queue_input.gang_quantum = 10000;
134 queue_input.paging = false;134 queue_input.paging = false;
135 135
136 queue_input.process_context_addr = ctx->gpu_addr;136 queue_input.process_context_addr = uq_mgr->proc_ctx_obj.gpu_addr;
137 queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;137 queue_input.gang_context_addr = ctx->gpu_addr;
138 queue_input.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;138 queue_input.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL;
139 queue_input.gang_global_priority_level = convert_to_mes_priority(queue->priority);139 queue_input.gang_global_priority_level = convert_to_mes_priority(queue->priority);
140 140
@@ -169,7 +169,8 @@ static int mes_userq_unmap(struct amdgpu_usermode_queue *queue) · 28 unmodified lines
169 169
170 memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));170 memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));
171 queue_input.doorbell_offset = queue->doorbell_index;171 queue_input.doorbell_offset = queue->doorbell_index;
172 queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;172 queue_input.gang_context_addr = ctx->gpu_addr;
173 queue_input.queue_type = queue->queue_type;
173 174
174 amdgpu_mes_lock(&adev->mes);175 amdgpu_mes_lock(&adev->mes);
175 r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);176 r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input);
@@ -179,6 +180,63 @@ static int mes_userq_unmap(struct amdgpu_usermode_queue *queue) · 3 unmodified lines
179 return r;180 return r;
180}181}
181 182
183int mes_userq_reset(struct amdgpu_usermode_queue *queue)
184{
185 struct amdgpu_userq_mgr *uq_mgr = queue->userq_mgr;
186 struct amdgpu_device *adev = uq_mgr->adev;
187 struct mes_reset_queue_input queue_input;
188 int r;
189
190 /* XXX: add a FW version check for SDMA per queue reset */
191 memset(&queue_input, 0x0, sizeof(struct mes_reset_queue_input));
192 queue_input.doorbell_offset = queue->doorbell_index;
193 queue_input.queue_type = queue->queue_type;
194
195 amdgpu_mes_lock(&adev->mes);
196 r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input);
197 amdgpu_mes_unlock(&adev->mes);
198 if (r)
199 return r;
200 return mes_userq_unmap(queue);
201}
202
203int mes_userq_reset_queue(struct amdgpu_device *adev,
204 struct amdgpu_usermode_queue *guilty_uq,
205 int queue_type,
206 unsigned int pipe,
207 unsigned int queue,
208 unsigned int db)
209{
210 struct amdgpu_usermode_queue *uq;
211 bool use_mmio = adev->gfx.mec.use_mmio_for_reset;
212 unsigned long uq_id;
213 int r;
214
215 xa_for_each(&adev->userq_doorbell_xa, uq_id, uq) {
216 if (uq->queue_type == queue_type) {
217 if (uq == guilty_uq)
218 continue;
219 if (uq->doorbell_index == db) {
220 uq->state = AMDGPU_USERQ_STATE_HUNG;
221 if (use_mmio)
222 r = amdgpu_mes_reset_queue_mmio(adev, queue_type, 0, 1, pipe, queue, 0);
223 else
224 r = amdgpu_mes_reset_user_queue(adev, queue_type, db, 0);
225 if (r)
226 return r;
227 r = mes_userq_unmap(uq);
228 if (r)
229 return r;
230 atomic_inc(&adev->gpu_reset_counter);
231 amdgpu_userq_fence_driver_force_completion(uq);
232 drm_dev_wedged_event(adev_to_drm(adev), DRM_WEDGE_RECOVERY_NONE, NULL);
233 break;
234 }
235 }
236 }
237 return 0;
238}
239
182static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr,240static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr,
183 struct amdgpu_usermode_queue *queue,241 struct amdgpu_usermode_queue *queue,
184 struct drm_amdgpu_userq_in *mqd_user)242 struct drm_amdgpu_userq_in *mqd_user)
@@ -186,12 +244,8 @@ static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr, · 1 unmodified line
186 struct amdgpu_userq_obj *ctx = &queue->fw_obj;244 struct amdgpu_userq_obj *ctx = &queue->fw_obj;
187 int r, size;245 int r, size;
188 246
189 /*247 /* The FW expects at least one page space allocated for gang ctx. */
190 * The FW expects at least one page space allocated for248 size = AMDGPU_USERQ_GANG_CTX_SZ;
191 * process ctx and gang ctx each. Create an object
192 * for the same.
193 */
194 size = AMDGPU_USERQ_PROC_CTX_SZ + AMDGPU_USERQ_GANG_CTX_SZ;
195 r = amdgpu_bo_create_kernel(uq_mgr->adev, size, 0,249 r = amdgpu_bo_create_kernel(uq_mgr->adev, size, 0,
196 AMDGPU_GEM_DOMAIN_GTT,250 AMDGPU_GEM_DOMAIN_GTT,
197 &ctx->obj, &ctx->gpu_addr,251 &ctx->obj, &ctx->gpu_addr,
@@ -205,54 +259,26 @@ static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr, · 7 unmodified lines
205 return 0;259 return 0;
206}260}
207 261
208static int mes_userq_detect_and_reset(struct amdgpu_device *adev,262static int mes_userq_create_proc_ctx_space(struct amdgpu_userq_mgr *uq_mgr)
209 int queue_type)
210{263{
211 int db_array_size = amdgpu_mes_get_hung_queue_db_array_size(adev);264 int r = 0;
212 struct mes_detect_and_reset_queue_input input;
213 struct amdgpu_usermode_queue *queue;
214 unsigned int hung_db_num = 0;
215 unsigned long queue_id;
216 u32 db_array[8];
217 bool found_hung_queue = false;
218 int r, i;
219
220 if (db_array_size > 8) {
221 dev_err(adev->dev, "DB array size (%d vs 8) too small\n",
222 db_array_size);
223 return -EINVAL;
224 }
225
226 memset(&input, 0x0, sizeof(struct mes_detect_and_reset_queue_input));
227 265
228 input.queue_type = queue_type;266 mutex_lock(&uq_mgr->proc_ctx_lock);
229 267 /* This check is a necessary because amdgpu_bo_create_kernel()
230 amdgpu_mes_lock(&adev->mes);268 * calls helpers like amdgpu_bo_pin() and memset() unconditionally
231 r = amdgpu_mes_detect_and_reset_hung_queues(adev, queue_type, false,269 */
232 &hung_db_num, db_array, 0);270 if (!uq_mgr->proc_ctx_obj.obj) {
233 amdgpu_mes_unlock(&adev->mes);271 r = amdgpu_bo_create_kernel(uq_mgr->adev, AMDGPU_USERQ_PROC_CTX_SZ,
234 if (r) {272 0, AMDGPU_GEM_DOMAIN_GTT,
235 dev_err(adev->dev, "Failed to detect and reset queues, err (%d)\n", r);273 &uq_mgr->proc_ctx_obj.obj,
236 } else if (hung_db_num) {274 &uq_mgr->proc_ctx_obj.gpu_addr,
237 xa_for_each(&adev->userq_doorbell_xa, queue_id, queue) {275 &uq_mgr->proc_ctx_obj.cpu_ptr);
238 if (queue->queue_type == queue_type) {276
239 for (i = 0; i < hung_db_num; i++) {277 if (!r)
240 if (queue->doorbell_index == db_array[i]) {278 memset(uq_mgr->proc_ctx_obj.cpu_ptr, 0, AMDGPU_USERQ_PROC_CTX_SZ);
241 queue->state = AMDGPU_USERQ_STATE_HUNG;
242 found_hung_queue = true;
243 atomic_inc(&adev->gpu_reset_counter);
244 amdgpu_userq_fence_driver_force_completion(queue);
245 drm_dev_wedged_event(adev_to_drm(adev), DRM_WEDGE_RECOVERY_NONE, NULL);
246 }
247 }
248 }
249 }
250 }279 }
251 280
252 if (found_hung_queue) {281 mutex_unlock(&uq_mgr->proc_ctx_lock);
253 /* Resume scheduling after hang recovery */
254 r = amdgpu_mes_resume(adev, input.xcc_id);
255 }
256 282
257 return r;283 return r;
258}284}
@@ -429,7 +455,14 @@ static int mes_userq_mqd_create(struct amdgpu_usermode_queue *queue, · 170 unmodified lines
429 goto free_mqd;455 goto free_mqd;
430 }456 }
431 457
432 /* Create BO for FW operations */458 /* Create per-process MES process context BO */
459 r = mes_userq_create_proc_ctx_space(uq_mgr);
460 if (r) {
461 DRM_ERROR("Failed to allocate MES process context space bo, error: %d\n", r);
462 goto free_mqd;
463 }
464
465 /* Create BO of a gang for FW operations */
433 r = mes_userq_create_ctx_space(uq_mgr, queue, mqd_user);466 r = mes_userq_create_ctx_space(uq_mgr, queue, mqd_user);
434 if (r) {467 if (r) {
435 DRM_ERROR("Failed to allocate BO for userqueue (%d)", r);468 DRM_ERROR("Failed to allocate BO for userqueue (%d)", r);
@@ -497,7 +530,7 @@ static int mes_userq_preempt(struct amdgpu_usermode_queue *queue) · 61 unmodified lines
497 *fence_ptr = 0;530 *fence_ptr = 0;
498 531
499 memset(&queue_input, 0x0, sizeof(struct mes_suspend_gang_input));532 memset(&queue_input, 0x0, sizeof(struct mes_suspend_gang_input));
500 queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;533 queue_input.gang_context_addr = ctx->gpu_addr;
501 queue_input.suspend_fence_addr = fence_gpu_addr;534 queue_input.suspend_fence_addr = fence_gpu_addr;
502 queue_input.suspend_fence_value = 1;535 queue_input.suspend_fence_value = 1;
503 amdgpu_mes_lock(&adev->mes);536 amdgpu_mes_lock(&adev->mes);
@@ -534,7 +567,7 @@ static int mes_userq_restore(struct amdgpu_usermode_queue *queue) · 30 unmodified lines
534 return 0;567 return 0;
535 568
536 memset(&queue_input, 0x0, sizeof(struct mes_resume_gang_input));569 memset(&queue_input, 0x0, sizeof(struct mes_resume_gang_input));
537 queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ;570 queue_input.gang_context_addr = ctx->gpu_addr;
538 571
539 amdgpu_mes_lock(&adev->mes);572 amdgpu_mes_lock(&adev->mes);
540 r = adev->mes.funcs->resume_gang(&adev->mes, &queue_input);573 r = adev->mes.funcs->resume_gang(&adev->mes, &queue_input);
@@ -549,7 +582,7 @@ const struct amdgpu_userq_funcs userq_mes_funcs = { · 8 unmodified lines
549 .mqd_destroy = mes_userq_mqd_destroy,582 .mqd_destroy = mes_userq_mqd_destroy,
550 .unmap = mes_userq_unmap,583 .unmap = mes_userq_unmap,
551 .map = mes_userq_map,584 .map = mes_userq_map,
552 .detect_and_reset = mes_userq_detect_and_reset,
553 .preempt = mes_userq_preempt,585 .preempt = mes_userq_preempt,
554 .restore = mes_userq_restore,586 .restore = mes_userq_restore,
587 .reset = mes_userq_reset,
555};588};
drivers/gpu/drm/amd/amdgpu/mes_userqueue.h+9−0
@@ -27,4 +27,13 @@ · 26 unmodified lines
27#include "amdgpu_userq.h"27#include "amdgpu_userq.h"
28 28
29extern const struct amdgpu_userq_funcs userq_mes_funcs;29extern const struct amdgpu_userq_funcs userq_mes_funcs;
30
31int mes_userq_reset(struct amdgpu_usermode_queue *queue);
32int mes_userq_reset_queue(struct amdgpu_device *adev,
33 struct amdgpu_usermode_queue *guilty_uq,
34 int queue_type,
35 unsigned int pipe,
36 unsigned int queue,
37 unsigned int db);
38
30#endif39#endif
drivers/gpu/drm/amd/amdgpu/mes_v11_0.c+236−4
@@ -387,6 +387,8 @@ static int mes_v11_0_remove_hw_queue(struct amdgpu_mes *mes, · 386 unmodified lines
387 387
388 mes_remove_queue_pkt.doorbell_offset = input->doorbell_offset;388 mes_remove_queue_pkt.doorbell_offset = input->doorbell_offset;
389 mes_remove_queue_pkt.gang_context_addr = input->gang_context_addr;389 mes_remove_queue_pkt.gang_context_addr = input->gang_context_addr;
390 mes_remove_queue_pkt.queue_type =
391 convert_to_mes_queue_type(input->queue_type);
390 392
391 if (mes_rev >= 0x60)393 if (mes_rev >= 0x60)
392 mes_remove_queue_pkt.remove_queue_after_reset = input->remove_queue_after_reset;394 mes_remove_queue_pkt.remove_queue_after_reset = input->remove_queue_after_reset;
@@ -396,6 +398,230 @@ static int mes_v11_0_remove_hw_queue(struct amdgpu_mes *mes, · 3 unmodified lines
396 offsetof(union MESAPI__REMOVE_QUEUE, api_status));398 offsetof(union MESAPI__REMOVE_QUEUE, api_status));
397}399}
398 400
401static bool mes_v11_0_pipe_reset_support(struct amdgpu_device *adev)
402{
403 /* Disable the pipe reset until the CPFW fully support it.*/
404 dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
405 return false;
406}
407static int mes_v11_0_reset_gfx_pipe_mmio(struct amdgpu_device *adev,
408 u32 me, u32 pipe, u32 queue)
409{
410 uint32_t reset_pipe = 0, clean_pipe = 0;
411 int r;
412
413 if (!mes_v11_0_pipe_reset_support(adev))
414 return -EOPNOTSUPP;
415
416 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
417 mutex_lock(&adev->srbm_mutex);
418 soc21_grbm_select(adev, me, pipe, queue, 0);
419
420 switch (pipe) {
421 case 0:
422 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
423 PFP_PIPE0_RESET, 1);
424 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
425 ME_PIPE0_RESET, 1);
426 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
427 PFP_PIPE0_RESET, 0);
428 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
429 ME_PIPE0_RESET, 0);
430 break;
431 case 1:
432 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
433 PFP_PIPE1_RESET, 1);
434 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
435 ME_PIPE1_RESET, 1);
436 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
437 PFP_PIPE1_RESET, 0);
438 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
439 ME_PIPE1_RESET, 0);
440 break;
441 default:
442 break;
443 }
444
445 WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
446 WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
447
448 r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
449 RS64_FW_UC_START_ADDR_LO;
450 soc21_grbm_select(adev, 0, 0, 0, 0);
451 mutex_unlock(&adev->srbm_mutex);
452 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
453
454 dev_info(adev->dev, "The gfx pipe reset to the ME firmware start PC: %s\n",
455 r == 0 ? "successfully" : "failed");
456 /* FIXME: Sometimes driver can't cache the ME firmware start PC correctly,
457 * so the pipe reset status relies on the later gfx ring test result.
458 */
459 return 0;
460}
461
462/*
463 * With MEC pipe reset asserted, clear CP_HQD_ACTIVE / CP_HQD_DEQUEUE_REQUEST for
464 * every queue on (me, pipe). HQDs must be torn down while pipe reset stays
465 * asserted; only then clear the pipe reset bit.
466 * Caller must hold adev->srbm_mutex.
467 */
468static void mes_v11_0_clear_hqds_on_mec_pipe(struct amdgpu_device *adev, u32 me,
469 u32 pipe)
470{
471 unsigned int q;
472
473 for (q = 0; q < adev->gfx.mec.num_queue_per_pipe; q++) {
474 soc21_grbm_select(adev, me, pipe, q, 0);
475 /* Start from a clean HQD dequeue state before forcing HQD inactive. */
476 WREG32_SOC15(GC, 0, regCP_HQD_ACTIVE, 0);
477 WREG32_SOC15(GC, 0, regCP_HQD_DEQUEUE_REQUEST, 0);
478 }
479}
480
481static int mes_v11_0_reset_compute_pipe_mmio(struct amdgpu_device *adev,
482 u32 me, u32 pipe, u32 queue)
483{
484 uint32_t reset_val, clean_val;
485 int r;
486
487 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
488 mutex_lock(&adev->srbm_mutex);
489 soc21_grbm_select(adev, me, pipe, queue, 0);
490
491 if (adev->gfx.rs64_enable) {
492 reset_val = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
493 clean_val = reset_val;
494
495 switch (pipe) {
496 case 0:
497 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
498 MEC_PIPE0_RESET, 1);
499 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
500 MEC_PIPE0_RESET, 0);
501 break;
502 case 1:
503 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
504 MEC_PIPE1_RESET, 1);
505 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
506 MEC_PIPE1_RESET, 0);
507 break;
508 case 2:
509 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
510 MEC_PIPE2_RESET, 1);
511 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
512 MEC_PIPE2_RESET, 0);
513 break;
514 case 3:
515 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
516 MEC_PIPE3_RESET, 1);
517 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
518 MEC_PIPE3_RESET, 0);
519 break;
520 default:
521 break;
522 }
523 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_val);
524 mes_v11_0_clear_hqds_on_mec_pipe(adev, me, pipe);
525 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_val);
526 r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
527 RS64_FW_UC_START_ADDR_LO;
528 } else {
529 reset_val = RREG32_SOC15(GC, 0, regCP_MEC_CNTL);
530 clean_val = reset_val;
531
532 if (me == 1) {
533 switch (pipe) {
534 case 0:
535 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
536 MEC_ME1_PIPE0_RESET, 1);
537 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
538 MEC_ME1_PIPE0_RESET, 0);
539 break;
540 case 1:
541 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
542 MEC_ME1_PIPE1_RESET, 1);
543 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
544 MEC_ME1_PIPE1_RESET, 0);
545 break;
546 case 2:
547 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
548 MEC_ME1_PIPE2_RESET, 1);
549 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
550 MEC_ME1_PIPE2_RESET, 0);
551 break;
552 case 3:
553 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
554 MEC_ME1_PIPE3_RESET, 1);
555 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
556 MEC_ME1_PIPE3_RESET, 0);
557 break;
558 default:
559 break;
560 }
561 /* mec1 fw pc: CP_MEC1_INSTR_PNTR */
562 } else {
563 switch (pipe) {
564 case 0:
565 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
566 MEC_ME2_PIPE0_RESET, 1);
567 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
568 MEC_ME2_PIPE0_RESET, 0);
569 break;
570 case 1:
571 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
572 MEC_ME2_PIPE1_RESET, 1);
573 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
574 MEC_ME2_PIPE1_RESET, 0);
575 break;
576 case 2:
577 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
578 MEC_ME2_PIPE2_RESET, 1);
579 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
580 MEC_ME2_PIPE2_RESET, 0);
581 break;
582 case 3:
583 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
584 MEC_ME2_PIPE3_RESET, 1);
585 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
586 MEC_ME2_PIPE3_RESET, 0);
587 break;
588 default:
589 break;
590 }
591 /* mec2 fw pc: CP:CP_MEC2_INSTR_PNTR */
592 }
593 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_val);
594 mes_v11_0_clear_hqds_on_mec_pipe(adev, me, pipe);
595 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_val);
596 r = RREG32(SOC15_REG_OFFSET(GC, 0, regCP_MEC1_INSTR_PNTR));
597 }
598
599 soc21_grbm_select(adev, 0, 0, 0, 0);
600 mutex_unlock(&adev->srbm_mutex);
601 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
602
603 dev_dbg(adev->dev, "MEC pipe me%u pipe%u queue%u resets to MEC FW start PC: %s\n",
604 me, pipe, queue, r == 0 ? "successfully" : "failed");
605 /*FIXME:Sometimes driver can't cache the MEC firmware start PC correctly, so the pipe
606 * reset status relies on the compute ring test result.
607 */
608 return 0;
609}
610
611static int mes_v11_0_reset_pipe_mmio(struct amdgpu_mes *mes, uint32_t queue_type,
612 uint32_t me_id, uint32_t pipe_id,
613 uint32_t queue_id, uint32_t vmid)
614{
615 struct amdgpu_device *adev = mes->adev;
616
617 if (queue_type == AMDGPU_RING_TYPE_GFX)
618 return mes_v11_0_reset_gfx_pipe_mmio(adev, me_id, pipe_id, queue_id);
619 else if (queue_type == AMDGPU_RING_TYPE_COMPUTE)
620 return mes_v11_0_reset_compute_pipe_mmio(adev, me_id, pipe_id, queue_id);
621 else
622 return -EOPNOTSUPP;
623}
624
399static int mes_v11_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_type,625static int mes_v11_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_type,
400 uint32_t me_id, uint32_t pipe_id,626 uint32_t me_id, uint32_t pipe_id,
401 uint32_t queue_id, uint32_t vmid)627 uint32_t queue_id, uint32_t vmid)
@@ -770,10 +996,16 @@ static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes, · 368 unmodified lines
770{996{
771 union MESAPI__RESET mes_reset_queue_pkt;997 union MESAPI__RESET mes_reset_queue_pkt;
772 998
773 if (input->use_mmio)999 if (input->use_mmio) {
774 return mes_v11_0_reset_queue_mmio(mes, input->queue_type,1000 int r = mes_v11_0_reset_queue_mmio(mes, input->queue_type,
775 input->me_id, input->pipe_id,1001 input->me_id, input->pipe_id,
776 input->queue_id, input->vmid);1002 input->queue_id, input->vmid);
1003 if (r)
1004 return mes_v11_0_reset_pipe_mmio(mes, input->queue_type,
1005 input->me_id, input->pipe_id,
1006 input->queue_id, input->vmid);
1007 return 0;
1008 }
777 1009
778 memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));1010 memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));
779 1011
drivers/gpu/drm/amd/amdgpu/mes_v12_0.c+236−31
@@ -26,7 +26,7 @@ · 25 unmodified lines
26#include "amdgpu.h"26#include "amdgpu.h"
27#include "gfx_v12_0.h"27#include "gfx_v12_0.h"
28#include "soc15_common.h"28#include "soc15_common.h"
29#include "soc21.h"29#include "soc24.h"
30#include "gc/gc_12_0_0_offset.h"30#include "gc/gc_12_0_0_offset.h"
31#include "gc/gc_12_0_0_sh_mask.h"31#include "gc/gc_12_0_0_sh_mask.h"
32#include "gc/gc_11_0_0_default.h"32#include "gc/gc_11_0_0_default.h"
@@ -371,6 +371,8 @@ static int mes_v12_0_remove_hw_queue(struct amdgpu_mes *mes, · 338 unmodified lines
371 371
372 mes_remove_queue_pkt.doorbell_offset = input->doorbell_offset;372 mes_remove_queue_pkt.doorbell_offset = input->doorbell_offset;
373 mes_remove_queue_pkt.gang_context_addr = input->gang_context_addr;373 mes_remove_queue_pkt.gang_context_addr = input->gang_context_addr;
374 mes_remove_queue_pkt.queue_type =
375 convert_to_mes_queue_type(input->queue_type);
374 376
375 if (mes_rev >= 0x5a)377 if (mes_rev >= 0x5a)
376 mes_remove_queue_pkt.remove_queue_after_reset = input->remove_queue_after_reset;378 mes_remove_queue_pkt.remove_queue_after_reset = input->remove_queue_after_reset;
@@ -413,6 +415,171 @@ int gfx_v12_0_request_gfx_index_mutex(struct amdgpu_device *adev, · 36 unmodified lines
413 return 0;415 return 0;
414}416}
415 417
418static bool mes_v12_0_pipe_reset_support(struct amdgpu_device *adev)
419{
420 /* Disable the pipe reset until the CPFW fully support it.*/
421 dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n");
422 return false;
423}
424
425static int mes_v12_0_reset_gfx_pipe_mmio(struct amdgpu_device *adev,
426 u32 me, u32 pipe, u32 queue)
427{
428 uint32_t reset_pipe = 0, clean_pipe = 0;
429 int r;
430
431 if (!mes_v12_0_pipe_reset_support(adev))
432 return -EOPNOTSUPP;
433
434 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
435 mutex_lock(&adev->srbm_mutex);
436 soc24_grbm_select(adev, me, pipe, queue, 0);
437
438 switch (pipe) {
439 case 0:
440 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
441 PFP_PIPE0_RESET, 1);
442 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
443 ME_PIPE0_RESET, 1);
444 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
445 PFP_PIPE0_RESET, 0);
446 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
447 ME_PIPE0_RESET, 0);
448 break;
449 case 1:
450 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
451 PFP_PIPE1_RESET, 1);
452 reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL,
453 ME_PIPE1_RESET, 1);
454 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
455 PFP_PIPE1_RESET, 0);
456 clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL,
457 ME_PIPE1_RESET, 0);
458 break;
459 default:
460 break;
461 }
462
463 WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe);
464 WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe);
465
466 r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) -
467 RS64_FW_UC_START_ADDR_LO;
468 soc24_grbm_select(adev, 0, 0, 0, 0);
469 mutex_unlock(&adev->srbm_mutex);
470 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
471
472 dev_info(adev->dev, "The gfx pipe reset: %s\n",
473 r == 0 ? "successfully" : "failed");
474 /* Sometimes the ME start pc counter can't cache correctly, so the
475 * PC check only as a reference and pipe reset result rely on the
476 * later ring test.
477 */
478 return 0;
479}
480
481/*
482 * With MEC pipe reset asserted, clear CP_HQD_ACTIVE / CP_HQD_DEQUEUE_REQUEST for
483 * every queue on (me, pipe). HQDs must be torn down while pipe reset stays
484 * asserted; only then clear the pipe reset bit.
485 * Caller must hold adev->srbm_mutex.
486 */
487static void mes_v12_0_clear_hqds_on_mec_pipe(struct amdgpu_device *adev, u32 me,
488 u32 pipe)
489{
490 unsigned int q;
491
492 for (q = 0; q < adev->gfx.mec.num_queue_per_pipe; q++) {
493 soc24_grbm_select(adev, me, pipe, q, 0);
494 /* Start from a clean HQD dequeue state before forcing HQD inactive. */
495 WREG32_SOC15(GC, 0, regCP_HQD_ACTIVE, 0);
496 WREG32_SOC15(GC, 0, regCP_HQD_DEQUEUE_REQUEST, 0);
497 }
498}
499
500static int mes_v12_0_reset_compute_pipe_mmio(struct amdgpu_device *adev,
501 u32 me, u32 pipe, u32 queue)
502{
503 uint32_t reset_val, clean_val;
504 int r = 0;
505
506 amdgpu_gfx_rlc_enter_safe_mode(adev, 0);
507 mutex_lock(&adev->srbm_mutex);
508 soc24_grbm_select(adev, me, pipe, queue, 0);
509 if (adev->gfx.rs64_enable) {
510 reset_val = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL);
511 clean_val = reset_val;
512
513 switch (pipe) {
514 case 0:
515 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
516 MEC_PIPE0_RESET, 1);
517 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
518 MEC_PIPE0_RESET, 0);
519 break;
520 case 1:
521 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
522 MEC_PIPE1_RESET, 1);
523 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
524 MEC_PIPE1_RESET, 0);
525 break;
526 case 2:
527 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
528 MEC_PIPE2_RESET, 1);
529 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
530 MEC_PIPE2_RESET, 0);
531 break;
532 case 3:
533 reset_val = REG_SET_FIELD(reset_val, CP_MEC_RS64_CNTL,
534 MEC_PIPE3_RESET, 1);
535 clean_val = REG_SET_FIELD(clean_val, CP_MEC_RS64_CNTL,
536 MEC_PIPE3_RESET, 0);
537 break;
538 default:
539 break;
540 }
541 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_val);
542 mes_v12_0_clear_hqds_on_mec_pipe(adev, me, pipe);
543 soc24_grbm_select(adev, me, pipe, queue, 0);
544 WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_val);
545 r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) -
546 RS64_FW_UC_START_ADDR_LO;
547 } else {
548 reset_val = RREG32_SOC15(GC, 0, regCP_MEC_CNTL);
549 clean_val = reset_val;
550
551 switch (pipe) {
552 case 0:
553 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
554 MEC_ME1_PIPE0_RESET, 1);
555 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
556 MEC_ME1_PIPE0_RESET, 0);
557 break;
558 case 1:
559 reset_val = REG_SET_FIELD(reset_val, CP_MEC_CNTL,
560 MEC_ME1_PIPE1_RESET, 1);
561 clean_val = REG_SET_FIELD(clean_val, CP_MEC_CNTL,
562 MEC_ME1_PIPE1_RESET, 0);
563 break;
564 default:
565 break;
566 }
567
568 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_val);
569 mes_v12_0_clear_hqds_on_mec_pipe(adev, me, pipe);
570 soc24_grbm_select(adev, me, pipe, queue, 0);
571 WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_val);
572 }
573
574 soc24_grbm_select(adev, 0, 0, 0, 0);
575 mutex_unlock(&adev->srbm_mutex);
576 amdgpu_gfx_rlc_exit_safe_mode(adev, 0);
577
578 dev_dbg(adev->dev, "MEC pipe me%u pipe%u queue%u resets to MEC FW start PC: %s\n",
579 me, pipe, queue, r == 0 ? "successfully" : "failed");
580 return 0;
581}
582
416static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_type,583static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_type,
417 uint32_t me_id, uint32_t pipe_id,584 uint32_t me_id, uint32_t pipe_id,
418 uint32_t queue_id, uint32_t vmid)585 uint32_t queue_id, uint32_t vmid)
@@ -442,7 +609,7 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 23 unmodified lines
442 mutex_unlock(&adev->gfx.reset_sem_mutex);609 mutex_unlock(&adev->gfx.reset_sem_mutex);
443 610
444 mutex_lock(&adev->srbm_mutex);611 mutex_lock(&adev->srbm_mutex);
445 soc21_grbm_select(adev, me_id, pipe_id, queue_id, 0);612 soc24_grbm_select(adev, me_id, pipe_id, queue_id, 0);
446 /* wait till dequeue take effects */613 /* wait till dequeue take effects */
447 for (i = 0; i < adev->usec_timeout; i++) {614 for (i = 0; i < adev->usec_timeout; i++) {
448 if (!(RREG32_SOC15(GC, 0, regCP_GFX_HQD_ACTIVE) & 1))615 if (!(RREG32_SOC15(GC, 0, regCP_GFX_HQD_ACTIVE) & 1))
@@ -454,13 +621,13 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 5 unmodified lines
454 r = -ETIMEDOUT;621 r = -ETIMEDOUT;
455 }622 }
456 623
457 soc21_grbm_select(adev, 0, 0, 0, 0);624 soc24_grbm_select(adev, 0, 0, 0, 0);
458 mutex_unlock(&adev->srbm_mutex);625 mutex_unlock(&adev->srbm_mutex);
459 } else if (queue_type == AMDGPU_RING_TYPE_COMPUTE) {626 } else if (queue_type == AMDGPU_RING_TYPE_COMPUTE) {
460 dev_info(adev->dev, "reset compute queue (%d:%d:%d)\n",627 dev_info(adev->dev, "reset compute queue (%d:%d:%d)\n",
461 me_id, pipe_id, queue_id);628 me_id, pipe_id, queue_id);
462 mutex_lock(&adev->srbm_mutex);629 mutex_lock(&adev->srbm_mutex);
463 soc21_grbm_select(adev, me_id, pipe_id, queue_id, 0);630 soc24_grbm_select(adev, me_id, pipe_id, queue_id, 0);
464 WREG32_SOC15(GC, 0, regCP_HQD_DEQUEUE_REQUEST, 0x2);631 WREG32_SOC15(GC, 0, regCP_HQD_DEQUEUE_REQUEST, 0x2);
465 WREG32_SOC15(GC, 0, regSPI_COMPUTE_QUEUE_RESET, 0x1);632 WREG32_SOC15(GC, 0, regSPI_COMPUTE_QUEUE_RESET, 0x1);
466 633
@@ -474,7 +641,7 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 7 unmodified lines
474 dev_err(adev->dev, "failed to wait on hqd deactivate\n");641 dev_err(adev->dev, "failed to wait on hqd deactivate\n");
475 r = -ETIMEDOUT;642 r = -ETIMEDOUT;
476 }643 }
477 soc21_grbm_select(adev, 0, 0, 0, 0);644 soc24_grbm_select(adev, 0, 0, 0, 0);
478 mutex_unlock(&adev->srbm_mutex);645 mutex_unlock(&adev->srbm_mutex);
479 } else if (queue_type == AMDGPU_RING_TYPE_SDMA) {646 } else if (queue_type == AMDGPU_RING_TYPE_SDMA) {
480 dev_info(adev->dev, "reset sdma queue (%d:%d:%d)\n",647 dev_info(adev->dev, "reset sdma queue (%d:%d:%d)\n",
@@ -507,6 +674,20 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 26 unmodified lines
507 return r;674 return r;
508}675}
509 676
677static int mes_v12_0_reset_pipe_mmio(struct amdgpu_mes *mes, uint32_t queue_type,
678 uint32_t me_id, uint32_t pipe_id,
679 uint32_t queue_id, uint32_t vmid)
680{
681 struct amdgpu_device *adev = mes->adev;
682
683 if (queue_type == AMDGPU_RING_TYPE_GFX)
684 return mes_v12_0_reset_gfx_pipe_mmio(adev, me_id, pipe_id, queue_id);
685 else if (queue_type == AMDGPU_RING_TYPE_COMPUTE)
686 return mes_v12_0_reset_compute_pipe_mmio(adev, me_id, pipe_id, queue_id);
687 else
688 return -EOPNOTSUPP;
689}
690
510static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes,691static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes,
511 struct mes_map_legacy_queue_input *input)692 struct mes_map_legacy_queue_input *input)
512{693{
@@ -528,10 +709,15 @@ static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes, · 15 unmodified lines
528 convert_to_mes_queue_type(input->queue_type);709 convert_to_mes_queue_type(input->queue_type);
529 mes_add_queue_pkt.map_legacy_kq = 1;710 mes_add_queue_pkt.map_legacy_kq = 1;
530 711
531 if (mes->adev->enable_uni_mes)712 if (mes->adev->enable_uni_mes) {
532 pipe = AMDGPU_MES_KIQ_PIPE;713 /* Keep scheduler queue on KIQ pipe; map all other kernel queues on sched pipe. */
533 else714 if (input->queue_type == AMDGPU_RING_TYPE_MES)
715 pipe = AMDGPU_MES_KIQ_PIPE;
716 else
717 pipe = AMDGPU_MES_SCHED_PIPE;
718 } else {
534 pipe = AMDGPU_MES_SCHED_PIPE;719 pipe = AMDGPU_MES_SCHED_PIPE;
720 }
535 721
536 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,722 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
537 &mes_add_queue_pkt, sizeof(mes_add_queue_pkt),723 &mes_add_queue_pkt, sizeof(mes_add_queue_pkt),
@@ -565,12 +751,28 @@ static int mes_v12_0_unmap_legacy_queue(struct amdgpu_mes *mes, · 27 unmodified lines
565 mes_remove_queue_pkt.unmap_legacy_queue = 1;751 mes_remove_queue_pkt.unmap_legacy_queue = 1;
566 mes_remove_queue_pkt.queue_type =752 mes_remove_queue_pkt.queue_type =
567 convert_to_mes_queue_type(input->queue_type);753 convert_to_mes_queue_type(input->queue_type);
754 /*
755 * A reset-time unmap: the queue was already reset via MMIO while
756 * gangs are suspended and it is on the MES hung/fail list. Tell
757 * MES to just drop its internal state for it. Without this flag
758 * MES asks CP to unmap the already-reset (still wedged) queue
759 * again, which times out and forces a GPU reset.
760 */
761 if (input->action == RESET_QUEUES &&
762 (mes->sched_version & AMDGPU_MES_VERSION_MASK) >= 0x5a)
763 mes_remove_queue_pkt.remove_queue_after_reset = 1;
764
568 }765 }
569 766
570 if (mes->adev->enable_uni_mes)767 if (mes->adev->enable_uni_mes) {
571 pipe = AMDGPU_MES_KIQ_PIPE;768 /* Keep scheduler queue on KIQ pipe; unmap all other kernel queues on sched pipe. */
572 else769 if (input->queue_type == AMDGPU_RING_TYPE_MES)
770 pipe = AMDGPU_MES_KIQ_PIPE;
771 else
772 pipe = AMDGPU_MES_SCHED_PIPE;
773 } else {
573 pipe = AMDGPU_MES_SCHED_PIPE;774 pipe = AMDGPU_MES_SCHED_PIPE;
775 }
574 776
575 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,777 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
576 &mes_remove_queue_pkt, sizeof(mes_remove_queue_pkt),778 &mes_remove_queue_pkt, sizeof(mes_remove_queue_pkt),
@@ -888,10 +1090,16 @@ static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes, · 311 unmodified lines
888 union MESAPI__RESET mes_reset_queue_pkt;1090 union MESAPI__RESET mes_reset_queue_pkt;
889 int pipe;1091 int pipe;
890 1092
891 if (input->use_mmio)1093 if (input->use_mmio) {
892 return mes_v12_0_reset_queue_mmio(mes, input->queue_type,1094 int r = mes_v12_0_reset_queue_mmio(mes, input->queue_type,
893 input->me_id, input->pipe_id,1095 input->me_id, input->pipe_id,
894 input->queue_id, input->vmid);1096 input->queue_id, input->vmid);
1097 if (r)
1098 return mes_v12_0_reset_pipe_mmio(mes, input->queue_type,
1099 input->me_id, input->pipe_id,
1100 input->queue_id, input->vmid);
1101 return 0;
1102 }
895 1103
896 memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));1104 memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt));
897 1105
@@ -915,10 +1123,7 @@ static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines
915 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;1123 mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset;
916 }1124 }
917 1125
918 if (input->is_kq)1126 pipe = AMDGPU_MES_SCHED_PIPE;
919 pipe = AMDGPU_MES_KIQ_PIPE;
920 else
921 pipe = AMDGPU_MES_SCHED_PIPE;
922 1127
923 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,1128 return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe,
924 &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt),1129 &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt),
@@ -1094,7 +1299,7 @@ static void mes_v12_0_enable(struct amdgpu_device *adev, bool enable) · 169 unmodified lines
1094 if (enable) {1299 if (enable) {
1095 mutex_lock(&adev->srbm_mutex);1300 mutex_lock(&adev->srbm_mutex);
1096 for (pipe = 0; pipe < AMDGPU_MAX_MES_PIPES; pipe++) {1301 for (pipe = 0; pipe < AMDGPU_MAX_MES_PIPES; pipe++) {
1097 soc21_grbm_select(adev, 3, pipe, 0, 0);1302 soc24_grbm_select(adev, 3, pipe, 0, 0);
1098 if (amdgpu_mes_log_enable) {1303 if (amdgpu_mes_log_enable) {
1099 u32 log_size = AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE;1304 u32 log_size = AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE;
1100 /* In case uni mes is not enabled, only program for pipe 0 */1305 /* In case uni mes is not enabled, only program for pipe 0 */
@@ -1133,7 +1338,7 @@ static void mes_v12_0_enable(struct amdgpu_device *adev, bool enable) · 32 unmodified lines
1133 WREG32_SOC15(GC, 0, regCP_MES_CNTL, data);1338 WREG32_SOC15(GC, 0, regCP_MES_CNTL, data);
1134 1339
1135 }1340 }
1136 soc21_grbm_select(adev, 0, 0, 0, 0);1341 soc24_grbm_select(adev, 0, 0, 0, 0);
1137 mutex_unlock(&adev->srbm_mutex);1342 mutex_unlock(&adev->srbm_mutex);
1138 1343
1139 if (amdgpu_emu_mode)1344 if (amdgpu_emu_mode)
@@ -1165,7 +1370,7 @@ static void mes_v12_0_set_ucode_start_addr(struct amdgpu_device *adev) · 25 unmodified lines
1165 mutex_lock(&adev->srbm_mutex);1370 mutex_lock(&adev->srbm_mutex);
1166 for (pipe = 0; pipe < AMDGPU_MAX_MES_PIPES; pipe++) {1371 for (pipe = 0; pipe < AMDGPU_MAX_MES_PIPES; pipe++) {
1167 /* me=3, queue=0 */1372 /* me=3, queue=0 */
1168 soc21_grbm_select(adev, 3, pipe, 0, 0);1373 soc24_grbm_select(adev, 3, pipe, 0, 0);
1169 1374
1170 /* set ucode start address */1375 /* set ucode start address */
1171 ucode_addr = adev->mes.uc_start_addr[pipe] >> 2;1376 ucode_addr = adev->mes.uc_start_addr[pipe] >> 2;
@@ -1174,7 +1379,7 @@ static void mes_v12_0_set_ucode_start_addr(struct amdgpu_device *adev) · 2 unmodified lines
1174 WREG32_SOC15(GC, 0, regCP_MES_PRGRM_CNTR_START_HI,1379 WREG32_SOC15(GC, 0, regCP_MES_PRGRM_CNTR_START_HI,
1175 upper_32_bits(ucode_addr));1380 upper_32_bits(ucode_addr));
1176 1381
1177 soc21_grbm_select(adev, 0, 0, 0, 0);1382 soc24_grbm_select(adev, 0, 0, 0, 0);
1178 }1383 }
1179 mutex_unlock(&adev->srbm_mutex);1384 mutex_unlock(&adev->srbm_mutex);
1180}1385}
@@ -1203,7 +1408,7 @@ static int mes_v12_0_load_microcode(struct amdgpu_device *adev, · 22 unmodified lines
1203 1408
1204 mutex_lock(&adev->srbm_mutex);1409 mutex_lock(&adev->srbm_mutex);
1205 /* me=3, pipe=0, queue=0 */1410 /* me=3, pipe=0, queue=0 */
1206 soc21_grbm_select(adev, 3, pipe, 0, 0);1411 soc24_grbm_select(adev, 3, pipe, 0, 0);
1207 1412
1208 WREG32_SOC15(GC, 0, regCP_MES_IC_BASE_CNTL, 0);1413 WREG32_SOC15(GC, 0, regCP_MES_IC_BASE_CNTL, 0);
1209 1414
@@ -1238,7 +1443,7 @@ static int mes_v12_0_load_microcode(struct amdgpu_device *adev, · 28 unmodified lines
1238 WREG32_SOC15(GC, 0, regCP_MES_IC_OP_CNTL, data);1443 WREG32_SOC15(GC, 0, regCP_MES_IC_OP_CNTL, data);
1239 }1444 }
1240 1445
1241 soc21_grbm_select(adev, 0, 0, 0, 0);1446 soc24_grbm_select(adev, 0, 0, 0, 0);
1242 mutex_unlock(&adev->srbm_mutex);1447 mutex_unlock(&adev->srbm_mutex);
1243 1448
1244 return 0;1449 return 0;
@@ -1385,7 +1590,7 @@ static void mes_v12_0_queue_init_register(struct amdgpu_ring *ring) · 140 unmodified lines
1385 uint32_t data = 0;1590 uint32_t data = 0;
1386 1591
1387 mutex_lock(&adev->srbm_mutex);1592 mutex_lock(&adev->srbm_mutex);
1388 soc21_grbm_select(adev, 3, ring->pipe, 0, 0);1593 soc24_grbm_select(adev, 3, ring->pipe, 0, 0);
1389 1594
1390 /* set CP_HQD_VMID.VMID = 0. */1595 /* set CP_HQD_VMID.VMID = 0. */
1391 data = RREG32_SOC15(GC, 0, regCP_HQD_VMID);1596 data = RREG32_SOC15(GC, 0, regCP_HQD_VMID);
@@ -1436,7 +1641,7 @@ static void mes_v12_0_queue_init_register(struct amdgpu_ring *ring) · 44 unmodified lines
1436 /* set CP_HQD_ACTIVE.ACTIVE=1 */1641 /* set CP_HQD_ACTIVE.ACTIVE=1 */
1437 WREG32_SOC15(GC, 0, regCP_HQD_ACTIVE, mqd->cp_hqd_active);1642 WREG32_SOC15(GC, 0, regCP_HQD_ACTIVE, mqd->cp_hqd_active);
1438 1643
1439 soc21_grbm_select(adev, 0, 0, 0, 0);1644 soc24_grbm_select(adev, 0, 0, 0, 0);
1440 mutex_unlock(&adev->srbm_mutex);1645 mutex_unlock(&adev->srbm_mutex);
1441}1646}
1442 1647
@@ -1502,14 +1707,14 @@ static int mes_v12_0_queue_init(struct amdgpu_device *adev, · 59 unmodified lines
1502 ((pipe == AMDGPU_MES_KIQ_PIPE) && !adev->mes.kiq_version)) {1707 ((pipe == AMDGPU_MES_KIQ_PIPE) && !adev->mes.kiq_version)) {
1503 /* get MES scheduler/KIQ versions */1708 /* get MES scheduler/KIQ versions */
1504 mutex_lock(&adev->srbm_mutex);1709 mutex_lock(&adev->srbm_mutex);
1505 soc21_grbm_select(adev, 3, pipe, 0, 0);1710 soc24_grbm_select(adev, 3, pipe, 0, 0);
1506 1711
1507 if (pipe == AMDGPU_MES_SCHED_PIPE)1712 if (pipe == AMDGPU_MES_SCHED_PIPE)
1508 adev->mes.sched_version = RREG32_SOC15(GC, 0, regCP_MES_GP3_LO);1713 adev->mes.sched_version = RREG32_SOC15(GC, 0, regCP_MES_GP3_LO);
1509 else if (pipe == AMDGPU_MES_KIQ_PIPE && adev->enable_mes_kiq)1714 else if (pipe == AMDGPU_MES_KIQ_PIPE && adev->enable_mes_kiq)
1510 adev->mes.kiq_version = RREG32_SOC15(GC, 0, regCP_MES_GP3_LO);1715 adev->mes.kiq_version = RREG32_SOC15(GC, 0, regCP_MES_GP3_LO);
1511 1716
1512 soc21_grbm_select(adev, 0, 0, 0, 0);1717 soc24_grbm_select(adev, 0, 0, 0, 0);
1513 mutex_unlock(&adev->srbm_mutex);1718 mutex_unlock(&adev->srbm_mutex);
1514 }1719 }
1515 1720
@@ -1697,7 +1902,7 @@ static void mes_v12_0_kiq_dequeue_sched(struct amdgpu_device *adev) · 181 unmodified lines
1697 int i;1902 int i;
1698 1903
1699 mutex_lock(&adev->srbm_mutex);1904 mutex_lock(&adev->srbm_mutex);
1700 soc21_grbm_select(adev, 3, AMDGPU_MES_SCHED_PIPE, 0, 0);1905 soc24_grbm_select(adev, 3, AMDGPU_MES_SCHED_PIPE, 0, 0);
1701 1906
1702 /* disable the queue if it's active */1907 /* disable the queue if it's active */
1703 if (RREG32_SOC15(GC, 0, regCP_HQD_ACTIVE) & 1) {1908 if (RREG32_SOC15(GC, 0, regCP_HQD_ACTIVE) & 1) {
@@ -1721,7 +1926,7 @@ static void mes_v12_0_kiq_dequeue_sched(struct amdgpu_device *adev) · 17 unmodified lines
1721 WREG32_SOC15(GC, 0, regCP_HQD_PQ_WPTR_HI, 0);1926 WREG32_SOC15(GC, 0, regCP_HQD_PQ_WPTR_HI, 0);
1722 WREG32_SOC15(GC, 0, regCP_HQD_PQ_RPTR, 0);1927 WREG32_SOC15(GC, 0, regCP_HQD_PQ_RPTR, 0);
1723 1928
1724 soc21_grbm_select(adev, 0, 0, 0, 0);1929 soc24_grbm_select(adev, 0, 0, 0, 0);
1725 mutex_unlock(&adev->srbm_mutex);1930 mutex_unlock(&adev->srbm_mutex);
1726 1931
1727 adev->mes.ring[0].sched.ready = false;1932 adev->mes.ring[0].sched.ready = false;
drivers/gpu/drm/amd/amdgpu/mes_v12_1.c+19−6
@@ -362,6 +362,8 @@ static int mes_v12_1_remove_hw_queue(struct amdgpu_mes *mes, · 361 unmodified lines
362 362
363 mes_remove_queue_pkt.doorbell_offset = input->doorbell_offset;363 mes_remove_queue_pkt.doorbell_offset = input->doorbell_offset;
364 mes_remove_queue_pkt.gang_context_addr = input->gang_context_addr;364 mes_remove_queue_pkt.gang_context_addr = input->gang_context_addr;
365 mes_remove_queue_pkt.queue_type =
366 convert_to_mes_queue_type(input->queue_type);
365 367
366 return mes_v12_1_submit_pkt_and_poll_completion(mes,368 return mes_v12_1_submit_pkt_and_poll_completion(mes,
367 xcc_id, AMDGPU_MES_SCHED_PIPE,369 xcc_id, AMDGPU_MES_SCHED_PIPE,
@@ -417,10 +419,15 @@ static int mes_v12_1_map_legacy_queue(struct amdgpu_mes *mes, · 49 unmodified lines
417 convert_to_mes_queue_type(input->queue_type);419 convert_to_mes_queue_type(input->queue_type);
418 mes_add_queue_pkt.map_legacy_kq = 1;420 mes_add_queue_pkt.map_legacy_kq = 1;
419 421
420 if (mes->adev->enable_uni_mes)422 if (mes->adev->enable_uni_mes) {
421 pipe = AMDGPU_MES_KIQ_PIPE;423 /* Keep scheduler queue on KIQ pipe; map all other kernel queues on sched pipe. */
422 else424 if (input->queue_type == AMDGPU_RING_TYPE_MES)
425 pipe = AMDGPU_MES_KIQ_PIPE;
426 else
427 pipe = AMDGPU_MES_SCHED_PIPE;
428 } else {
423 pipe = AMDGPU_MES_SCHED_PIPE;429 pipe = AMDGPU_MES_SCHED_PIPE;
430 }
424 431
425 return mes_v12_1_submit_pkt_and_poll_completion(mes,432 return mes_v12_1_submit_pkt_and_poll_completion(mes,
426 input->xcc_id, pipe,433 input->xcc_id, pipe,
@@ -457,10 +464,15 @@ static int mes_v12_1_unmap_legacy_queue(struct amdgpu_mes *mes, · 30 unmodified lines
457 convert_to_mes_queue_type(input->queue_type);464 convert_to_mes_queue_type(input->queue_type);
458 }465 }
459 466
460 if (mes->adev->enable_uni_mes)467 if (mes->adev->enable_uni_mes) {
461 pipe = AMDGPU_MES_KIQ_PIPE;468 /* Keep scheduler queue on KIQ pipe; map all other kernel queues on sched pipe. */
462 else469 if (input->queue_type == AMDGPU_RING_TYPE_MES)
470 pipe = AMDGPU_MES_KIQ_PIPE;
471 else
472 pipe = AMDGPU_MES_SCHED_PIPE;
473 } else {
463 pipe = AMDGPU_MES_SCHED_PIPE;474 pipe = AMDGPU_MES_SCHED_PIPE;
475 }
464 476
465 return mes_v12_1_submit_pkt_and_poll_completion(mes,477 return mes_v12_1_submit_pkt_and_poll_completion(mes,
466 input->xcc_id, pipe,478 input->xcc_id, pipe,
@@ -2262,6 +2274,7 @@ static int mes_v12_1_test_queue(struct amdgpu_device *adev, int xcc_id, · 1795 unmodified lines
2262 remove_queue.xcc_id = xcc_id;2274 remove_queue.xcc_id = xcc_id;
2263 remove_queue.doorbell_offset = doorbell_idx;2275 remove_queue.doorbell_offset = doorbell_idx;
2264 remove_queue.gang_context_addr = add_queue.gang_context_addr;2276 remove_queue.gang_context_addr = add_queue.gang_context_addr;
2277 remove_queue.queue_type = queue_type;
2265 r = mes_v12_1_remove_hw_queue(&adev->mes, &remove_queue);2278 r = mes_v12_1_remove_hw_queue(&adev->mes, &remove_queue);
2266 2279
2267error:2280error:
drivers/gpu/drm/amd/amdgpu/mmhub_v1_8.c+1−230
@@ -29,7 +29,6 @@ · 28 unmodified lines
29 29
30#include "soc15_common.h"30#include "soc15_common.h"
31#include "soc15.h"31#include "soc15.h"
32#include "amdgpu_ras.h"
33#include "amdgpu_psp.h"32#include "amdgpu_psp.h"
34 33
35#define regVM_L2_CNTL3_DEFAULT 0x8010000734#define regVM_L2_CNTL3_DEFAULT 0x80100007
@@ -636,236 +635,8 @@ const struct amdgpu_mmhub_funcs mmhub_v1_8_funcs = { · 600 unmodified lines
636 .get_clockgating = mmhub_v1_8_get_clockgating,635 .get_clockgating = mmhub_v1_8_get_clockgating,
637};636};
638 637
639static const struct amdgpu_ras_err_status_reg_entry mmhub_v1_8_ce_reg_list[] = {
640 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA0_CE_ERR_STATUS_LO, regMMEA0_CE_ERR_STATUS_HI),
641 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA0"},
642 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA1_CE_ERR_STATUS_LO, regMMEA1_CE_ERR_STATUS_HI),
643 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA1"},
644 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA2_CE_ERR_STATUS_LO, regMMEA2_CE_ERR_STATUS_HI),
645 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA2"},
646 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA3_CE_ERR_STATUS_LO, regMMEA3_CE_ERR_STATUS_HI),
647 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA3"},
648 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA4_CE_ERR_STATUS_LO, regMMEA4_CE_ERR_STATUS_HI),
649 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA4"},
650 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMM_CANE_CE_ERR_STATUS_LO, regMM_CANE_CE_ERR_STATUS_HI),
651 1, 0, "MM_CANE"},
652};
653
654static const struct amdgpu_ras_err_status_reg_entry mmhub_v1_8_ue_reg_list[] = {
655 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA0_UE_ERR_STATUS_LO, regMMEA0_UE_ERR_STATUS_HI),
656 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA0"},
657 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA1_UE_ERR_STATUS_LO, regMMEA1_UE_ERR_STATUS_HI),
658 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA1"},
659 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA2_UE_ERR_STATUS_LO, regMMEA2_UE_ERR_STATUS_HI),
660 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA2"},
661 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA3_UE_ERR_STATUS_LO, regMMEA3_UE_ERR_STATUS_HI),
662 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA3"},
663 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMMEA4_UE_ERR_STATUS_LO, regMMEA4_UE_ERR_STATUS_HI),
664 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "MMEA4"},
665 {AMDGPU_RAS_REG_ENTRY(MMHUB, 0, regMM_CANE_UE_ERR_STATUS_LO, regMM_CANE_UE_ERR_STATUS_HI),
666 1, 0, "MM_CANE"},
667};
668
669static const struct amdgpu_ras_memory_id_entry mmhub_v1_8_ras_memory_list[] = {
670 {AMDGPU_MMHUB_WGMI_PAGEMEM, "MMEA_WGMI_PAGEMEM"},
671 {AMDGPU_MMHUB_RGMI_PAGEMEM, "MMEA_RGMI_PAGEMEM"},
672 {AMDGPU_MMHUB_WDRAM_PAGEMEM, "MMEA_WDRAM_PAGEMEM"},
673 {AMDGPU_MMHUB_RDRAM_PAGEMEM, "MMEA_RDRAM_PAGEMEM"},
674 {AMDGPU_MMHUB_WIO_CMDMEM, "MMEA_WIO_CMDMEM"},
675 {AMDGPU_MMHUB_RIO_CMDMEM, "MMEA_RIO_CMDMEM"},
676 {AMDGPU_MMHUB_WGMI_CMDMEM, "MMEA_WGMI_CMDMEM"},
677 {AMDGPU_MMHUB_RGMI_CMDMEM, "MMEA_RGMI_CMDMEM"},
678 {AMDGPU_MMHUB_WDRAM_CMDMEM, "MMEA_WDRAM_CMDMEM"},
679 {AMDGPU_MMHUB_RDRAM_CMDMEM, "MMEA_RDRAM_CMDMEM"},
680 {AMDGPU_MMHUB_MAM_DMEM0, "MMEA_MAM_DMEM0"},
681 {AMDGPU_MMHUB_MAM_DMEM1, "MMEA_MAM_DMEM1"},
682 {AMDGPU_MMHUB_MAM_DMEM2, "MMEA_MAM_DMEM2"},
683 {AMDGPU_MMHUB_MAM_DMEM3, "MMEA_MAM_DMEM3"},
684 {AMDGPU_MMHUB_WRET_TAGMEM, "MMEA_WRET_TAGMEM"},
685 {AMDGPU_MMHUB_RRET_TAGMEM, "MMEA_RRET_TAGMEM"},
686 {AMDGPU_MMHUB_WIO_DATAMEM, "MMEA_WIO_DATAMEM"},
687 {AMDGPU_MMHUB_WGMI_DATAMEM, "MMEA_WGMI_DATAMEM"},
688 {AMDGPU_MMHUB_WDRAM_DATAMEM, "MMEA_WDRAM_DATAMEM"},
689};
690
691static void mmhub_v1_8_inst_query_ras_error_count(struct amdgpu_device *adev,
692 uint32_t mmhub_inst,
693 void *ras_err_status)
694{
695 struct ras_err_data *err_data = (struct ras_err_data *)ras_err_status;
696 unsigned long ue_count = 0, ce_count = 0;
697
698 /* NOTE: mmhub is converted by aid_mask and the range is 0-3,
699 * which can be used as die ID directly */
700 struct amdgpu_smuio_mcm_config_info mcm_info = {
701 .socket_id = adev->smuio.funcs->get_socket_id(adev),
702 .die_id = mmhub_inst,
703 };
704
705 amdgpu_ras_inst_query_ras_error_count(adev,
706 mmhub_v1_8_ce_reg_list,
707 ARRAY_SIZE(mmhub_v1_8_ce_reg_list),
708 mmhub_v1_8_ras_memory_list,
709 ARRAY_SIZE(mmhub_v1_8_ras_memory_list),
710 mmhub_inst,
711 AMDGPU_RAS_ERROR__SINGLE_CORRECTABLE,
712 &ce_count);
713 amdgpu_ras_inst_query_ras_error_count(adev,
714 mmhub_v1_8_ue_reg_list,
715 ARRAY_SIZE(mmhub_v1_8_ue_reg_list),
716 mmhub_v1_8_ras_memory_list,
717 ARRAY_SIZE(mmhub_v1_8_ras_memory_list),
718 mmhub_inst,
719 AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
720 &ue_count);
721
722 amdgpu_ras_error_statistic_ce_count(err_data, &mcm_info, ce_count);
723 amdgpu_ras_error_statistic_ue_count(err_data, &mcm_info, ue_count);
724}
725
726static void mmhub_v1_8_query_ras_error_count(struct amdgpu_device *adev,
727 void *ras_err_status)
728{
729 uint32_t inst_mask;
730 uint32_t i;
731
732 if (!amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__MMHUB)) {
733 dev_warn(adev->dev, "MMHUB RAS is not supported\n");
734 return;
735 }
736
737 inst_mask = adev->aid_mask;
738 for_each_inst(i, inst_mask)
739 mmhub_v1_8_inst_query_ras_error_count(adev, i, ras_err_status);
740}
741
742static void mmhub_v1_8_inst_reset_ras_error_count(struct amdgpu_device *adev,
743 uint32_t mmhub_inst)
744{
745 amdgpu_ras_inst_reset_ras_error_count(adev,
746 mmhub_v1_8_ce_reg_list,
747 ARRAY_SIZE(mmhub_v1_8_ce_reg_list),
748 mmhub_inst);
749 amdgpu_ras_inst_reset_ras_error_count(adev,
750 mmhub_v1_8_ue_reg_list,
751 ARRAY_SIZE(mmhub_v1_8_ue_reg_list),
752 mmhub_inst);
753}
754
755static void mmhub_v1_8_reset_ras_error_count(struct amdgpu_device *adev)
756{
757 uint32_t inst_mask;
758 uint32_t i;
759
760 if (!amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__MMHUB)) {
761 dev_warn(adev->dev, "MMHUB RAS is not supported\n");
762 return;
763 }
764
765 inst_mask = adev->aid_mask;
766 for_each_inst(i, inst_mask)
767 mmhub_v1_8_inst_reset_ras_error_count(adev, i);
768}
769
770static const struct amdgpu_ras_block_hw_ops mmhub_v1_8_ras_hw_ops = {
771 .query_ras_error_count = mmhub_v1_8_query_ras_error_count,
772 .reset_ras_error_count = mmhub_v1_8_reset_ras_error_count,
773};
774
775static int mmhub_v1_8_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
776 enum aca_smu_type type, void *data)
777{
778 struct aca_bank_info info;
779 u64 misc0;
780 int ret;
781
782 ret = aca_bank_info_decode(bank, &info);
783 if (ret)
784 return ret;
785
786 misc0 = bank->regs[ACA_REG_IDX_MISC0];
787 switch (type) {
788 case ACA_SMU_TYPE_UE:
789 bank->aca_err_type = ACA_ERROR_TYPE_UE;
790 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE,
791 1ULL);
792 break;
793 case ACA_SMU_TYPE_CE:
794 bank->aca_err_type = ACA_ERROR_TYPE_CE;
795 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
796 ACA_REG__MISC0__ERRCNT(misc0));
797 break;
798 default:
799 return -EINVAL;
800 }
801
802 return ret;
803}
804
805/* reference to smu driver if header file */
806static int mmhub_v1_8_err_codes[] = {
807 0, 1, 2, 3, 4, /* CODE_DAGB0 - 4 */
808 5, 6, 7, 8, 9, /* CODE_EA0 - 4 */
809 10, /* CODE_UTCL2_ROUTER */
810 11, /* CODE_VML2 */
811 12, /* CODE_VML2_WALKER */
812 13, /* CODE_MMCANE */
813};
814
815static bool mmhub_v1_8_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
816 enum aca_smu_type type, void *data)
817{
818 u32 instlo;
819
820 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
821 instlo &= GENMASK(31, 1);
822
823 if (instlo != mmSMNAID_AID0_MCA_SMU)
824 return false;
825
826 if (aca_bank_check_error_codes(handle->adev, bank,
827 mmhub_v1_8_err_codes,
828 ARRAY_SIZE(mmhub_v1_8_err_codes)))
829 return false;
830
831 return true;
832}
833
834static const struct aca_bank_ops mmhub_v1_8_aca_bank_ops = {
835 .aca_bank_parser = mmhub_v1_8_aca_bank_parser,
836 .aca_bank_is_valid = mmhub_v1_8_aca_bank_is_valid,
837};
838
839static const struct aca_info mmhub_v1_8_aca_info = {
840 .hwip = ACA_HWIP_TYPE_SMU,
841 .mask = ACA_ERROR_UE_MASK,
842 .bank_ops = &mmhub_v1_8_aca_bank_ops,
843};
844
845static int mmhub_v1_8_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
846{
847 int r;
848
849 r = amdgpu_ras_block_late_init(adev, ras_block);
850 if (r)
851 return r;
852
853 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__MMHUB,
854 &mmhub_v1_8_aca_info, NULL);
855 if (r)
856 goto late_fini;
857
858 return 0;
859
860late_fini:
861 amdgpu_ras_block_late_fini(adev, ras_block);
862
863 return r;
864}
865
866struct amdgpu_mmhub_ras mmhub_v1_8_ras = {638struct amdgpu_mmhub_ras mmhub_v1_8_ras = {
867 .ras_block = {639 .ras_block = {
868 .hw_ops = &mmhub_v1_8_ras_hw_ops,640 .hw_ops = NULL,
869 .ras_late_init = mmhub_v1_8_ras_late_init,
870 },641 },
871};642};
drivers/gpu/drm/amd/amdgpu/nv.c+0−6
@@ -507,11 +507,6 @@ void nv_set_virt_ops(struct amdgpu_device *adev) · 506 unmodified lines
507 adev->virt.ops = &xgpu_nv_virt_ops;507 adev->virt.ops = &xgpu_nv_virt_ops;
508}508}
509 509
510static bool nv_need_full_reset(struct amdgpu_device *adev)
511{
512 return true;
513}
514
515static bool nv_need_reset_on_init(struct amdgpu_device *adev)510static bool nv_need_reset_on_init(struct amdgpu_device *adev)
516{511{
517 u32 sol_reg;512 u32 sol_reg;
@@ -595,7 +590,6 @@ static const struct amdgpu_asic_funcs nv_asic_funcs = { · 77 unmodified lines
595 .set_vce_clocks = &nv_set_vce_clocks,590 .set_vce_clocks = &nv_set_vce_clocks,
596 .get_config_memsize = &nv_get_config_memsize,591 .get_config_memsize = &nv_get_config_memsize,
597 .init_doorbell_index = &nv_init_doorbell_index,592 .init_doorbell_index = &nv_init_doorbell_index,
598 .need_full_reset = &nv_need_full_reset,
599 .need_reset_on_init = &nv_need_reset_on_init,593 .need_reset_on_init = &nv_need_reset_on_init,
600 .get_pcie_replay_count = &amdgpu_nbio_get_pcie_replay_count,594 .get_pcie_replay_count = &amdgpu_nbio_get_pcie_replay_count,
601 .supports_baco = &amdgpu_dpm_is_baco_supported,595 .supports_baco = &amdgpu_dpm_is_baco_supported,
drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c+0−62
@@ -1237,65 +1237,6 @@ static int sdma_v3_0_wait_for_idle(struct amdgpu_ip_block *ip_block) · 1236 unmodified lines
1237 return -ETIMEDOUT;1237 return -ETIMEDOUT;
1238}1238}
1239 1239
1240static bool sdma_v3_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
1241{
1242 struct amdgpu_device *adev = ip_block->adev;
1243 u32 srbm_soft_reset = 0;
1244 u32 tmp = RREG32(mmSRBM_STATUS2);
1245
1246 if ((tmp & SRBM_STATUS2__SDMA_BUSY_MASK) ||
1247 (tmp & SRBM_STATUS2__SDMA1_BUSY_MASK)) {
1248 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_SDMA_MASK;
1249 srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_SDMA1_MASK;
1250 }
1251
1252 if (srbm_soft_reset) {
1253 adev->sdma.srbm_soft_reset = srbm_soft_reset;
1254 return true;
1255 } else {
1256 adev->sdma.srbm_soft_reset = 0;
1257 return false;
1258 }
1259}
1260
1261static int sdma_v3_0_pre_soft_reset(struct amdgpu_ip_block *ip_block)
1262{
1263 struct amdgpu_device *adev = ip_block->adev;
1264 u32 srbm_soft_reset = 0;
1265
1266 if (!adev->sdma.srbm_soft_reset)
1267 return 0;
1268
1269 srbm_soft_reset = adev->sdma.srbm_soft_reset;
1270
1271 if (REG_GET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_SDMA) ||
1272 REG_GET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_SDMA1)) {
1273 sdma_v3_0_ctx_switch_enable(adev, false);
1274 sdma_v3_0_enable(adev, false);
1275 }
1276
1277 return 0;
1278}
1279
1280static int sdma_v3_0_post_soft_reset(struct amdgpu_ip_block *ip_block)
1281{
1282 struct amdgpu_device *adev = ip_block->adev;
1283 u32 srbm_soft_reset = 0;
1284
1285 if (!adev->sdma.srbm_soft_reset)
1286 return 0;
1287
1288 srbm_soft_reset = adev->sdma.srbm_soft_reset;
1289
1290 if (REG_GET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_SDMA) ||
1291 REG_GET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_SDMA1)) {
1292 sdma_v3_0_gfx_resume(adev);
1293 sdma_v3_0_rlc_resume(adev);
1294 }
1295
1296 return 0;
1297}
1298
1299static int sdma_v3_0_soft_reset(struct amdgpu_ip_block *ip_block)1240static int sdma_v3_0_soft_reset(struct amdgpu_ip_block *ip_block)
1300{1241{
1301 struct amdgpu_device *adev = ip_block->adev;1242 struct amdgpu_device *adev = ip_block->adev;
@@ -1552,9 +1493,6 @@ static const struct amd_ip_funcs sdma_v3_0_ip_funcs = { · 250 unmodified lines
1552 .resume = sdma_v3_0_resume,1493 .resume = sdma_v3_0_resume,
1553 .is_idle = sdma_v3_0_is_idle,1494 .is_idle = sdma_v3_0_is_idle,
1554 .wait_for_idle = sdma_v3_0_wait_for_idle,1495 .wait_for_idle = sdma_v3_0_wait_for_idle,
1555 .check_soft_reset = sdma_v3_0_check_soft_reset,
1556 .pre_soft_reset = sdma_v3_0_pre_soft_reset,
1557 .post_soft_reset = sdma_v3_0_post_soft_reset,
1558 .soft_reset = sdma_v3_0_soft_reset,1496 .soft_reset = sdma_v3_0_soft_reset,
1559 .set_clockgating_state = sdma_v3_0_set_clockgating_state,1497 .set_clockgating_state = sdma_v3_0_set_clockgating_state,
1560 .set_powergating_state = sdma_v3_0_set_powergating_state,1498 .set_powergating_state = sdma_v3_0_set_powergating_state,
drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c+15−181
@@ -95,8 +95,6 @@ static const struct amdgpu_hwip_reg_entry sdma_reg_list_4_4_2[] = { · 94 unmodified lines
95 SOC15_REG_ENTRY_STR(GC, 0, regSDMA_VM_CNTL)95 SOC15_REG_ENTRY_STR(GC, 0, regSDMA_VM_CNTL)
96};96};
97 97
98#define mmSMNAID_AID0_MCA_SMU 0x03b30400
99
100#define WREG32_SDMA(instance, offset, value) \98#define WREG32_SDMA(instance, offset, value) \
101 WREG32(sdma_v4_4_2_get_reg_offset(adev, (instance), (offset)), value)99 WREG32(sdma_v4_4_2_get_reg_offset(adev, (instance), (offset)), value)
102#define RREG32_SDMA(instance, offset) \100#define RREG32_SDMA(instance, offset) \
@@ -1359,6 +1357,19 @@ static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block) · 1256 unmodified lines
1359 struct amdgpu_device *adev = ip_block->adev;1357 struct amdgpu_device *adev = ip_block->adev;
1360 int r;1358 int r;
1361 1359
1360 switch (amdgpu_user_queue) {
1361 case -1:
1362 case 0:
1363 default:
1364 adev->sdma.no_user_submission = false;
1365 adev->sdma.disable_uq = true;
1366 break;
1367 case 2:
1368 adev->sdma.no_user_submission = true;
1369 adev->sdma.disable_uq = true;
1370 break;
1371 }
1372
1362 r = sdma_v4_4_2_init_microcode(adev);1373 r = sdma_v4_4_2_init_microcode(adev);
1363 if (r)1374 if (r)
1364 return r;1375 return r;
@@ -1478,6 +1489,7 @@ static int sdma_v4_4_2_sw_init(struct amdgpu_ip_block *ip_block) · 113 unmodified lines
1478 /* doorbell size is 2 dwords, get DWORD offset */1489 /* doorbell size is 2 dwords, get DWORD offset */
1479 ring->doorbell_index = adev->doorbell_index.sdma_engine[i] << 1;1490 ring->doorbell_index = adev->doorbell_index.sdma_engine[i] << 1;
1480 ring->vm_hub = AMDGPU_MMHUB0(aid_id);1491 ring->vm_hub = AMDGPU_MMHUB0(aid_id);
1492 ring->no_user_submission = adev->sdma.no_user_submission;
1481 1493
1482 sprintf(ring->name, "sdma%d.%d", aid_id,1494 sprintf(ring->name, "sdma%d.%d", aid_id,
1483 i % adev->sdma.num_inst_per_aid);1495 i % adev->sdma.num_inst_per_aid);
@@ -2404,187 +2416,9 @@ struct amdgpu_xcp_ip_funcs sdma_v4_4_2_xcp_funcs = { · 920 unmodified lines
2404 .resume = &sdma_v4_4_2_xcp_resume2416 .resume = &sdma_v4_4_2_xcp_resume
2405};2417};
2406 2418
2407static const struct amdgpu_ras_err_status_reg_entry sdma_v4_2_2_ue_reg_list[] = {
2408 {AMDGPU_RAS_REG_ENTRY(SDMA0, 0, regSDMA_UE_ERR_STATUS_LO, regSDMA_UE_ERR_STATUS_HI),
2409 1, (AMDGPU_RAS_ERR_INFO_VALID | AMDGPU_RAS_ERR_STATUS_VALID), "SDMA"},
2410};
2411
2412static const struct amdgpu_ras_memory_id_entry sdma_v4_4_2_ras_memory_list[] = {
2413 {AMDGPU_SDMA_MBANK_DATA_BUF0, "SDMA_MBANK_DATA_BUF0"},
2414 {AMDGPU_SDMA_MBANK_DATA_BUF1, "SDMA_MBANK_DATA_BUF1"},
2415 {AMDGPU_SDMA_MBANK_DATA_BUF2, "SDMA_MBANK_DATA_BUF2"},
2416 {AMDGPU_SDMA_MBANK_DATA_BUF3, "SDMA_MBANK_DATA_BUF3"},
2417 {AMDGPU_SDMA_MBANK_DATA_BUF4, "SDMA_MBANK_DATA_BUF4"},
2418 {AMDGPU_SDMA_MBANK_DATA_BUF5, "SDMA_MBANK_DATA_BUF5"},
2419 {AMDGPU_SDMA_MBANK_DATA_BUF6, "SDMA_MBANK_DATA_BUF6"},
2420 {AMDGPU_SDMA_MBANK_DATA_BUF7, "SDMA_MBANK_DATA_BUF7"},
2421 {AMDGPU_SDMA_MBANK_DATA_BUF8, "SDMA_MBANK_DATA_BUF8"},
2422 {AMDGPU_SDMA_MBANK_DATA_BUF9, "SDMA_MBANK_DATA_BUF9"},
2423 {AMDGPU_SDMA_MBANK_DATA_BUF10, "SDMA_MBANK_DATA_BUF10"},
2424 {AMDGPU_SDMA_MBANK_DATA_BUF11, "SDMA_MBANK_DATA_BUF11"},
2425 {AMDGPU_SDMA_MBANK_DATA_BUF12, "SDMA_MBANK_DATA_BUF12"},
2426 {AMDGPU_SDMA_MBANK_DATA_BUF13, "SDMA_MBANK_DATA_BUF13"},
2427 {AMDGPU_SDMA_MBANK_DATA_BUF14, "SDMA_MBANK_DATA_BUF14"},
2428 {AMDGPU_SDMA_MBANK_DATA_BUF15, "SDMA_MBANK_DATA_BUF15"},
2429 {AMDGPU_SDMA_UCODE_BUF, "SDMA_UCODE_BUF"},
2430 {AMDGPU_SDMA_RB_CMD_BUF, "SDMA_RB_CMD_BUF"},
2431 {AMDGPU_SDMA_IB_CMD_BUF, "SDMA_IB_CMD_BUF"},
2432 {AMDGPU_SDMA_UTCL1_RD_FIFO, "SDMA_UTCL1_RD_FIFO"},
2433 {AMDGPU_SDMA_UTCL1_RDBST_FIFO, "SDMA_UTCL1_RDBST_FIFO"},
2434 {AMDGPU_SDMA_UTCL1_WR_FIFO, "SDMA_UTCL1_WR_FIFO"},
2435 {AMDGPU_SDMA_DATA_LUT_FIFO, "SDMA_DATA_LUT_FIFO"},
2436 {AMDGPU_SDMA_SPLIT_DAT_BUF, "SDMA_SPLIT_DAT_BUF"},
2437};
2438
2439static void sdma_v4_4_2_inst_query_ras_error_count(struct amdgpu_device *adev,
2440 uint32_t sdma_inst,
2441 void *ras_err_status)
2442{
2443 struct ras_err_data *err_data = (struct ras_err_data *)ras_err_status;
2444 uint32_t sdma_dev_inst = GET_INST(SDMA0, sdma_inst);
2445 unsigned long ue_count = 0;
2446 struct amdgpu_smuio_mcm_config_info mcm_info = {
2447 .socket_id = adev->smuio.funcs->get_socket_id(adev),
2448 .die_id = adev->sdma.instance[sdma_inst].aid_id,
2449 };
2450
2451 /* sdma v4_4_2 doesn't support query ce counts */
2452 amdgpu_ras_inst_query_ras_error_count(adev,
2453 sdma_v4_2_2_ue_reg_list,
2454 ARRAY_SIZE(sdma_v4_2_2_ue_reg_list),
2455 sdma_v4_4_2_ras_memory_list,
2456 ARRAY_SIZE(sdma_v4_4_2_ras_memory_list),
2457 sdma_dev_inst,
2458 AMDGPU_RAS_ERROR__MULTI_UNCORRECTABLE,
2459 &ue_count);
2460
2461 amdgpu_ras_error_statistic_ue_count(err_data, &mcm_info, ue_count);
2462}
2463
2464static void sdma_v4_4_2_query_ras_error_count(struct amdgpu_device *adev,
2465 void *ras_err_status)
2466{
2467 uint32_t inst_mask;
2468 int i = 0;
2469
2470 inst_mask = GENMASK(adev->sdma.num_instances - 1, 0);
2471 if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__SDMA)) {
2472 for_each_inst(i, inst_mask)
2473 sdma_v4_4_2_inst_query_ras_error_count(adev, i, ras_err_status);
2474 } else {
2475 dev_warn(adev->dev, "SDMA RAS is not supported\n");
2476 }
2477}
2478
2479static void sdma_v4_4_2_inst_reset_ras_error_count(struct amdgpu_device *adev,
2480 uint32_t sdma_inst)
2481{
2482 uint32_t sdma_dev_inst = GET_INST(SDMA0, sdma_inst);
2483
2484 amdgpu_ras_inst_reset_ras_error_count(adev,
2485 sdma_v4_2_2_ue_reg_list,
2486 ARRAY_SIZE(sdma_v4_2_2_ue_reg_list),
2487 sdma_dev_inst);
2488}
2489
2490static void sdma_v4_4_2_reset_ras_error_count(struct amdgpu_device *adev)
2491{
2492 uint32_t inst_mask;
2493 int i = 0;
2494
2495 inst_mask = GENMASK(adev->sdma.num_instances - 1, 0);
2496 if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__SDMA)) {
2497 for_each_inst(i, inst_mask)
2498 sdma_v4_4_2_inst_reset_ras_error_count(adev, i);
2499 } else {
2500 dev_warn(adev->dev, "SDMA RAS is not supported\n");
2501 }
2502}
2503
2504static const struct amdgpu_ras_block_hw_ops sdma_v4_4_2_ras_hw_ops = {
2505 .query_ras_error_count = sdma_v4_4_2_query_ras_error_count,
2506 .reset_ras_error_count = sdma_v4_4_2_reset_ras_error_count,
2507};
2508
2509static int sdma_v4_4_2_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
2510 enum aca_smu_type type, void *data)
2511{
2512 struct aca_bank_info info;
2513 u64 misc0;
2514 int ret;
2515
2516 ret = aca_bank_info_decode(bank, &info);
2517 if (ret)
2518 return ret;
2519
2520 misc0 = bank->regs[ACA_REG_IDX_MISC0];
2521 switch (type) {
2522 case ACA_SMU_TYPE_UE:
2523 bank->aca_err_type = ACA_ERROR_TYPE_UE;
2524 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE,
2525 1ULL);
2526 break;
2527 case ACA_SMU_TYPE_CE:
2528 bank->aca_err_type = ACA_ERROR_TYPE_CE;
2529 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
2530 ACA_REG__MISC0__ERRCNT(misc0));
2531 break;
2532 default:
2533 return -EINVAL;
2534 }
2535
2536 return ret;
2537}
2538
2539/* CODE_SDMA0 - CODE_SDMA4, reference to smu driver if header file */
2540static int sdma_v4_4_2_err_codes[] = { 33, 34, 35, 36 };
2541
2542static bool sdma_v4_4_2_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
2543 enum aca_smu_type type, void *data)
2544{
2545 u32 instlo;
2546
2547 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
2548 instlo &= GENMASK(31, 1);
2549
2550 if (instlo != mmSMNAID_AID0_MCA_SMU)
2551 return false;
2552
2553 if (aca_bank_check_error_codes(handle->adev, bank,
2554 sdma_v4_4_2_err_codes,
2555 ARRAY_SIZE(sdma_v4_4_2_err_codes)))
2556 return false;
2557
2558 return true;
2559}
2560
2561static const struct aca_bank_ops sdma_v4_4_2_aca_bank_ops = {
2562 .aca_bank_parser = sdma_v4_4_2_aca_bank_parser,
2563 .aca_bank_is_valid = sdma_v4_4_2_aca_bank_is_valid,
2564};
2565
2566static const struct aca_info sdma_v4_4_2_aca_info = {
2567 .hwip = ACA_HWIP_TYPE_SMU,
2568 .mask = ACA_ERROR_UE_MASK,
2569 .bank_ops = &sdma_v4_4_2_aca_bank_ops,
2570};
2571
2572static int sdma_v4_4_2_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
2573{
2574 int r;
2575
2576 r = amdgpu_sdma_ras_late_init(adev, ras_block);
2577 if (r)
2578 return r;
2579
2580 return amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__SDMA,
2581 &sdma_v4_4_2_aca_info, NULL);
2582}
2583
2584static struct amdgpu_sdma_ras sdma_v4_4_2_ras = {2419static struct amdgpu_sdma_ras sdma_v4_4_2_ras = {
2585 .ras_block = {2420 .ras_block = {
2586 .hw_ops = &sdma_v4_4_2_ras_hw_ops,2421 .hw_ops = NULL,
2587 .ras_late_init = sdma_v4_4_2_ras_late_init,
2588 },2422 },
2589};2423};
2590 2424
drivers/gpu/drm/amd/amdgpu/sdma_v6_0.c+0−18
@@ -793,23 +793,6 @@ static int sdma_v6_0_soft_reset(struct amdgpu_ip_block *ip_block) · 792 unmodified lines
793 return sdma_v6_0_start(adev);793 return sdma_v6_0_start(adev);
794}794}
795 795
796static bool sdma_v6_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
797{
798 struct amdgpu_device *adev = ip_block->adev;
799 struct amdgpu_ring *ring;
800 int i, r;
801 long tmo = msecs_to_jiffies(1000);
802
803 for (i = 0; i < adev->sdma.num_instances; i++) {
804 ring = &adev->sdma.instance[i].ring;
805 r = amdgpu_ring_test_ib(ring, tmo);
806 if (r)
807 return true;
808 }
809
810 return false;
811}
812
813/**796/**
814 * sdma_v6_0_start - setup and start the async dma engines797 * sdma_v6_0_start - setup and start the async dma engines
815 *798 *
@@ -1747,7 +1730,6 @@ const struct amd_ip_funcs sdma_v6_0_ip_funcs = { · 931 unmodified lines
1747 .is_idle = sdma_v6_0_is_idle,1730 .is_idle = sdma_v6_0_is_idle,
1748 .wait_for_idle = sdma_v6_0_wait_for_idle,1731 .wait_for_idle = sdma_v6_0_wait_for_idle,
1749 .soft_reset = sdma_v6_0_soft_reset,1732 .soft_reset = sdma_v6_0_soft_reset,
1750 .check_soft_reset = sdma_v6_0_check_soft_reset,
1751 .set_clockgating_state = sdma_v6_0_set_clockgating_state,1733 .set_clockgating_state = sdma_v6_0_set_clockgating_state,
1752 .set_powergating_state = sdma_v6_0_set_powergating_state,1734 .set_powergating_state = sdma_v6_0_set_powergating_state,
1753 .get_clockgating_state = sdma_v6_0_get_clockgating_state,1735 .get_clockgating_state = sdma_v6_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/sdma_v7_0.c+0−18
@@ -784,23 +784,6 @@ static int sdma_v7_0_soft_reset(struct amdgpu_ip_block *ip_block) · 783 unmodified lines
784 return sdma_v7_0_start(adev);784 return sdma_v7_0_start(adev);
785}785}
786 786
787static bool sdma_v7_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
788{
789 struct amdgpu_device *adev = ip_block->adev;
790 struct amdgpu_ring *ring;
791 int i, r;
792 long tmo = msecs_to_jiffies(1000);
793
794 for (i = 0; i < adev->sdma.num_instances; i++) {
795 ring = &adev->sdma.instance[i].ring;
796 r = amdgpu_ring_test_ib(ring, tmo);
797 if (r)
798 return true;
799 }
800
801 return false;
802}
803
804static int sdma_v7_0_reset_queue(struct amdgpu_ring *ring,787static int sdma_v7_0_reset_queue(struct amdgpu_ring *ring,
805 unsigned int vmid,788 unsigned int vmid,
806 struct amdgpu_fence *timedout_fence)789 struct amdgpu_fence *timedout_fence)
@@ -1679,7 +1662,6 @@ const struct amd_ip_funcs sdma_v7_0_ip_funcs = { · 872 unmodified lines
1679 .is_idle = sdma_v7_0_is_idle,1662 .is_idle = sdma_v7_0_is_idle,
1680 .wait_for_idle = sdma_v7_0_wait_for_idle,1663 .wait_for_idle = sdma_v7_0_wait_for_idle,
1681 .soft_reset = sdma_v7_0_soft_reset,1664 .soft_reset = sdma_v7_0_soft_reset,
1682 .check_soft_reset = sdma_v7_0_check_soft_reset,
1683 .set_clockgating_state = sdma_v7_0_set_clockgating_state,1665 .set_clockgating_state = sdma_v7_0_set_clockgating_state,
1684 .set_powergating_state = sdma_v7_0_set_powergating_state,1666 .set_powergating_state = sdma_v7_0_set_powergating_state,
1685 .get_clockgating_state = sdma_v7_0_get_clockgating_state,1667 .get_clockgating_state = sdma_v7_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/sdma_v7_1.c+0−18
@@ -775,23 +775,6 @@ static int sdma_v7_1_soft_reset(struct amdgpu_ip_block *ip_block) · 774 unmodified lines
775 return sdma_v7_1_inst_start(adev, inst_mask);775 return sdma_v7_1_inst_start(adev, inst_mask);
776}776}
777 777
778static bool sdma_v7_1_check_soft_reset(struct amdgpu_ip_block *ip_block)
779{
780 struct amdgpu_device *adev = ip_block->adev;
781 struct amdgpu_ring *ring;
782 int i, r;
783 long tmo = msecs_to_jiffies(1000);
784
785 for (i = 0; i < adev->sdma.num_instances; i++) {
786 ring = &adev->sdma.instance[i].ring;
787 r = amdgpu_ring_test_ib(ring, tmo);
788 if (r)
789 return true;
790 }
791
792 return false;
793}
794
795static int sdma_v7_1_reset_queue(struct amdgpu_ring *ring,778static int sdma_v7_1_reset_queue(struct amdgpu_ring *ring,
796 unsigned int vmid,779 unsigned int vmid,
797 struct amdgpu_fence *timedout_fence)780 struct amdgpu_fence *timedout_fence)
@@ -1644,7 +1627,6 @@ const struct amd_ip_funcs sdma_v7_1_ip_funcs = { · 846 unmodified lines
1644 .is_idle = sdma_v7_1_is_idle,1627 .is_idle = sdma_v7_1_is_idle,
1645 .wait_for_idle = sdma_v7_1_wait_for_idle,1628 .wait_for_idle = sdma_v7_1_wait_for_idle,
1646 .soft_reset = sdma_v7_1_soft_reset,1629 .soft_reset = sdma_v7_1_soft_reset,
1647 .check_soft_reset = sdma_v7_1_check_soft_reset,
1648 .set_clockgating_state = sdma_v7_1_set_clockgating_state,1630 .set_clockgating_state = sdma_v7_1_set_clockgating_state,
1649 .set_powergating_state = sdma_v7_1_set_powergating_state,1631 .set_powergating_state = sdma_v7_1_set_powergating_state,
1650 .get_clockgating_state = sdma_v7_1_get_clockgating_state,1632 .get_clockgating_state = sdma_v7_1_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/si.c+0−7
@@ -1509,12 +1509,6 @@ static void si_invalidate_hdp(struct amdgpu_device *adev, · 1508 unmodified lines
1509 }1509 }
1510}1510}
1511 1511
1512static bool si_need_full_reset(struct amdgpu_device *adev)
1513{
1514 /* change this when we support soft reset */
1515 return true;
1516}
1517
1518static bool si_need_reset_on_init(struct amdgpu_device *adev)1512static bool si_need_reset_on_init(struct amdgpu_device *adev)
1519{1513{
1520 return false;1514 return false;
@@ -2019,7 +2013,6 @@ static const struct amdgpu_asic_funcs si_asic_funcs = · 498 unmodified lines
2019 .get_config_memsize = &si_get_config_memsize,2013 .get_config_memsize = &si_get_config_memsize,
2020 .flush_hdp = &si_flush_hdp,2014 .flush_hdp = &si_flush_hdp,
2021 .invalidate_hdp = &si_invalidate_hdp,2015 .invalidate_hdp = &si_invalidate_hdp,
2022 .need_full_reset = &si_need_full_reset,
2023 .get_pcie_usage = &si_get_pcie_usage,2016 .get_pcie_usage = &si_get_pcie_usage,
2024 .need_reset_on_init = &si_need_reset_on_init,2017 .need_reset_on_init = &si_need_reset_on_init,
2025 .get_pcie_replay_count = &si_get_pcie_replay_count,2018 .get_pcie_replay_count = &si_get_pcie_replay_count,
drivers/gpu/drm/amd/amdgpu/soc15.c+0−9
@@ -721,12 +721,6 @@ void soc15_set_virt_ops(struct amdgpu_device *adev) · 720 unmodified lines
721 soc15_reg_base_init(adev);721 soc15_reg_base_init(adev);
722}722}
723 723
724static bool soc15_need_full_reset(struct amdgpu_device *adev)
725{
726 /* change this when we implement soft reset */
727 return true;
728}
729
730static void soc15_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,724static void soc15_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,
731 uint64_t *count1)725 uint64_t *count1)
732{726{
@@ -878,7 +872,6 @@ static const struct amdgpu_asic_funcs soc15_asic_funcs = · 145 unmodified lines
878 .set_uvd_clocks = &soc15_set_uvd_clocks,872 .set_uvd_clocks = &soc15_set_uvd_clocks,
879 .set_vce_clocks = &soc15_set_vce_clocks,873 .set_vce_clocks = &soc15_set_vce_clocks,
880 .get_config_memsize = &soc15_get_config_memsize,874 .get_config_memsize = &soc15_get_config_memsize,
881 .need_full_reset = &soc15_need_full_reset,
882 .init_doorbell_index = &vega10_doorbell_index_init,875 .init_doorbell_index = &vega10_doorbell_index_init,
883 .get_pcie_usage = &soc15_get_pcie_usage,876 .get_pcie_usage = &soc15_get_pcie_usage,
884 .need_reset_on_init = &soc15_need_reset_on_init,877 .need_reset_on_init = &soc15_need_reset_on_init,
@@ -899,7 +892,6 @@ static const struct amdgpu_asic_funcs vega20_asic_funcs = · 14 unmodified lines
899 .set_uvd_clocks = &soc15_set_uvd_clocks,892 .set_uvd_clocks = &soc15_set_uvd_clocks,
900 .set_vce_clocks = &soc15_set_vce_clocks,893 .set_vce_clocks = &soc15_set_vce_clocks,
901 .get_config_memsize = &soc15_get_config_memsize,894 .get_config_memsize = &soc15_get_config_memsize,
902 .need_full_reset = &soc15_need_full_reset,
903 .init_doorbell_index = &vega20_doorbell_index_init,895 .init_doorbell_index = &vega20_doorbell_index_init,
904 .get_pcie_usage = &vega20_get_pcie_usage,896 .get_pcie_usage = &vega20_get_pcie_usage,
905 .need_reset_on_init = &soc15_need_reset_on_init,897 .need_reset_on_init = &soc15_need_reset_on_init,
@@ -920,7 +912,6 @@ static const struct amdgpu_asic_funcs aqua_vanjaram_asic_funcs = · 14 unmodified lines
920 .set_uvd_clocks = &soc15_set_uvd_clocks,912 .set_uvd_clocks = &soc15_set_uvd_clocks,
921 .set_vce_clocks = &soc15_set_vce_clocks,913 .set_vce_clocks = &soc15_set_vce_clocks,
922 .get_config_memsize = &soc15_get_config_memsize,914 .get_config_memsize = &soc15_get_config_memsize,
923 .need_full_reset = &soc15_need_full_reset,
924 .init_doorbell_index = &aqua_vanjaram_doorbell_index_init,915 .init_doorbell_index = &aqua_vanjaram_doorbell_index_init,
925 .need_reset_on_init = &soc15_need_reset_on_init,916 .need_reset_on_init = &soc15_need_reset_on_init,
926 .get_pcie_replay_count = &amdgpu_nbio_get_pcie_replay_count,917 .get_pcie_replay_count = &amdgpu_nbio_get_pcie_replay_count,
drivers/gpu/drm/amd/amdgpu/soc15_common.h+31−34
@@ -38,30 +38,30 @@ · 37 unmodified lines
38 (adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + (reg)+(offset))38 (adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + (reg)+(offset))
39 39
40#define __WREG32_SOC15_RLC__(reg, value, flag, hwip, inst) \40#define __WREG32_SOC15_RLC__(reg, value, flag, hwip, inst) \
41 ((amdgpu_sriov_vf(adev) && adev->gfx.rlc.funcs && adev->gfx.rlc.rlcg_reg_access_supported) ? \41 adev->gfx.rlc.reg_funcs->wreg32(adev, reg, value, flag, hwip, inst)
42 amdgpu_sriov_wreg(adev, reg, value, flag, hwip, inst) : \
43 WREG32(reg, value))
44 42
45#define __RREG32_SOC15_RLC__(reg, flag, hwip, inst) \43#define __RREG32_SOC15_RLC__(reg, flag, hwip, inst) \
46 ((amdgpu_sriov_vf(adev) && adev->gfx.rlc.funcs && adev->gfx.rlc.rlcg_reg_access_supported) ? \44 adev->gfx.rlc.reg_funcs->rreg32(adev, reg, flag, hwip, inst)
47 amdgpu_sriov_rreg(adev, reg, flag, hwip, inst) : \45
48 RREG32(reg))46#define WREG32_FIELD15(ip, idx, reg_name, field, val) \
49 47do { \
50#define WREG32_FIELD15(ip, idx, reg, field, val) \48 u32 reg__ = adev->reg_offset[ip##_HWIP][idx][mm##reg_name##_BASE_IDX] + mm##reg_name; \
51 __WREG32_SOC15_RLC__(adev->reg_offset[ip##_HWIP][idx][mm##reg##_BASE_IDX] + mm##reg, \49 u32 val__ = __RREG32_SOC15_RLC__(reg__, 0, ip##_HWIP, idx); \
52 (__RREG32_SOC15_RLC__( \50\
53 adev->reg_offset[ip##_HWIP][idx][mm##reg##_BASE_IDX] + mm##reg, \51 val__ &= ~REG_FIELD_MASK(reg_name, field); \
54 0, ip##_HWIP, idx) & \52 val__ |= (val) << REG_FIELD_SHIFT(reg_name, field); \
55 ~REG_FIELD_MASK(reg, field)) | (val) << REG_FIELD_SHIFT(reg, field), \53 __WREG32_SOC15_RLC__(reg__, val__, 0, ip##_HWIP, idx); \
56 0, ip##_HWIP, idx)54} while (0)
57 55
58#define WREG32_FIELD15_PREREG(ip, idx, reg_name, field, val) \56#define WREG32_FIELD15_PREREG(ip, idx, reg_name, field, val) \
59 __WREG32_SOC15_RLC__(adev->reg_offset[ip##_HWIP][idx][reg##reg_name##_BASE_IDX] + reg##reg_name, \57do { \
60 (__RREG32_SOC15_RLC__( \58 u32 reg__ = adev->reg_offset[ip##_HWIP][idx][reg##reg_name##_BASE_IDX] + reg##reg_name; \
61 adev->reg_offset[ip##_HWIP][idx][reg##reg_name##_BASE_IDX] + reg##reg_name, \59 u32 val__ = __RREG32_SOC15_RLC__(reg__, 0, ip##_HWIP, idx); \
62 0, ip##_HWIP, idx) & \60\
63 ~REG_FIELD_MASK(reg_name, field)) | (val) << REG_FIELD_SHIFT(reg_name, field), \61 val__ &= ~REG_FIELD_MASK(reg_name, field); \
64 0, ip##_HWIP, idx)62 val__ |= (val) << REG_FIELD_SHIFT(reg_name, field); \
63 __WREG32_SOC15_RLC__(reg__, val__, 0, ip##_HWIP, idx); \
64} while (0)
65 65
66#define RREG32_SOC15(ip, inst, reg) \66#define RREG32_SOC15(ip, inst, reg) \
67 __RREG32_SOC15_RLC__(adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg, \67 __RREG32_SOC15_RLC__(adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg, \
@@ -181,12 +181,15 @@ · 113 unmodified lines
181 WREG32_RLC_EX(prefix, target_reg, value, inst); \181 WREG32_RLC_EX(prefix, target_reg, value, inst); \
182 } while (0)182 } while (0)
183 183
184#define WREG32_FIELD15_RLC(ip, idx, reg, field, val) \184#define WREG32_FIELD15_RLC(ip, idx, reg_name, field, val) \
185 __WREG32_SOC15_RLC__((adev->reg_offset[ip##_HWIP][idx][mm##reg##_BASE_IDX] + mm##reg), \185do { \
186 (__RREG32_SOC15_RLC__(adev->reg_offset[ip##_HWIP][idx][mm##reg##_BASE_IDX] + mm##reg, \186 u32 reg__ = adev->reg_offset[ip##_HWIP][idx][mm##reg_name##_BASE_IDX] + mm##reg_name; \
187 AMDGPU_REGS_RLC, ip##_HWIP, idx) & \187 u32 val__ = __RREG32_SOC15_RLC__(reg__, AMDGPU_REGS_RLC, ip##_HWIP, idx); \
188 ~REG_FIELD_MASK(reg, field)) | (val) << REG_FIELD_SHIFT(reg, field), \188\
189 AMDGPU_REGS_RLC, ip##_HWIP, idx)189 val__ &= ~REG_FIELD_MASK(reg_name, field); \
190 val__ |= (val) << REG_FIELD_SHIFT(reg_name, field); \
191 __WREG32_SOC15_RLC__(reg__, val__, AMDGPU_REGS_RLC, ip##_HWIP, idx); \
192} while (0)
190 193
191#define WREG32_SOC15_OFFSET_RLC(ip, inst, reg, offset, value) \194#define WREG32_SOC15_OFFSET_RLC(ip, inst, reg, offset, value) \
192 __WREG32_SOC15_RLC__((adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg) + offset, value, AMDGPU_REGS_RLC, ip##_HWIP, inst)195 __WREG32_SOC15_RLC__((adev->reg_offset[ip##_HWIP][inst][reg##_BASE_IDX] + reg) + offset, value, AMDGPU_REGS_RLC, ip##_HWIP, inst)
@@ -207,10 +210,4 @@ · 14 unmodified lines
207 amdgpu_reg_get_smn_base64(adev, ip##_HWIP, inst), \210 amdgpu_reg_get_smn_base64(adev, ip##_HWIP, inst), \
208 value)211 value)
209 212
210#define RREG64_MCA(smn_base, mca_base, idx) \
211 RREG64_PCIE_EXT(smn_base + mca_base + (idx * 8))
212
213#define WREG64_MCA(smn_base, mca_base, idx, val) \
214 WREG64_PCIE_EXT(smn_base + mca_base + (idx * 8), val)
215
216#endif213#endif
drivers/gpu/drm/amd/amdgpu/soc21.c+0−12
@@ -461,17 +461,6 @@ const struct amdgpu_ip_block_version soc21_common_ip_block = { · 460 unmodified lines
461 .funcs = &soc21_common_ip_funcs,461 .funcs = &soc21_common_ip_funcs,
462};462};
463 463
464static bool soc21_need_full_reset(struct amdgpu_device *adev)
465{
466 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
467 case IP_VERSION(11, 0, 0):
468 case IP_VERSION(11, 0, 2):
469 case IP_VERSION(11, 0, 3):
470 default:
471 return true;
472 }
473}
474
475static bool soc21_need_reset_on_init(struct amdgpu_device *adev)464static bool soc21_need_reset_on_init(struct amdgpu_device *adev)
476{465{
477 u32 sol_reg;466 u32 sol_reg;
@@ -550,7 +539,6 @@ static const struct amdgpu_asic_funcs soc21_asic_funcs = { · 72 unmodified lines
550 .set_vce_clocks = &soc21_set_vce_clocks,539 .set_vce_clocks = &soc21_set_vce_clocks,
551 .get_config_memsize = &soc21_get_config_memsize,540 .get_config_memsize = &soc21_get_config_memsize,
552 .init_doorbell_index = &soc21_init_doorbell_index,541 .init_doorbell_index = &soc21_init_doorbell_index,
553 .need_full_reset = &soc21_need_full_reset,
554 .need_reset_on_init = &soc21_need_reset_on_init,542 .need_reset_on_init = &soc21_need_reset_on_init,
555 .get_pcie_replay_count = &amdgpu_nbio_get_pcie_replay_count,543 .get_pcie_replay_count = &amdgpu_nbio_get_pcie_replay_count,
556 .supports_baco = &amdgpu_dpm_is_baco_supported,544 .supports_baco = &amdgpu_dpm_is_baco_supported,
drivers/gpu/drm/amd/amdgpu/soc24.c+0−11
@@ -238,16 +238,6 @@ const struct amdgpu_ip_block_version soc24_common_ip_block = { · 237 unmodified lines
238 .funcs = &soc24_common_ip_funcs,238 .funcs = &soc24_common_ip_funcs,
239};239};
240 240
241static bool soc24_need_full_reset(struct amdgpu_device *adev)
242{
243 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
244 case IP_VERSION(12, 0, 0):
245 case IP_VERSION(12, 0, 1):
246 default:
247 return true;
248 }
249}
250
251static bool soc24_need_reset_on_init(struct amdgpu_device *adev)241static bool soc24_need_reset_on_init(struct amdgpu_device *adev)
252{242{
253 u32 sol_reg;243 u32 sol_reg;
@@ -330,7 +320,6 @@ static const struct amdgpu_asic_funcs soc24_asic_funcs = { · 76 unmodified lines
330 .get_xclk = &soc24_get_xclk,320 .get_xclk = &soc24_get_xclk,
331 .get_config_memsize = &soc24_get_config_memsize,321 .get_config_memsize = &soc24_get_config_memsize,
332 .init_doorbell_index = &soc24_init_doorbell_index,322 .init_doorbell_index = &soc24_init_doorbell_index,
333 .need_full_reset = &soc24_need_full_reset,
334 .need_reset_on_init = &soc24_need_reset_on_init,323 .need_reset_on_init = &soc24_need_reset_on_init,
335 .get_pcie_replay_count = &soc24_get_pcie_replay_count,324 .get_pcie_replay_count = &soc24_get_pcie_replay_count,
336 .supports_baco = &amdgpu_dpm_is_baco_supported,325 .supports_baco = &amdgpu_dpm_is_baco_supported,
drivers/gpu/drm/amd/amdgpu/soc_v1_0.c+3−11
@@ -223,15 +223,6 @@ static int soc_v1_0_read_register(struct amdgpu_device *adev, · 222 unmodified lines
223 return -EINVAL;223 return -EINVAL;
224}224}
225 225
226static bool soc_v1_0_need_full_reset(struct amdgpu_device *adev)
227{
228 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
229 case IP_VERSION(12, 1, 0):
230 default:
231 return true;
232 }
233}
234
235static bool soc_v1_0_need_reset_on_init(struct amdgpu_device *adev)226static bool soc_v1_0_need_reset_on_init(struct amdgpu_device *adev)
236{227{
237 228
@@ -271,7 +262,6 @@ static const struct amdgpu_asic_funcs soc_v1_0_asic_funcs = { · 33 unmodified lines
271 .read_register = &soc_v1_0_read_register,262 .read_register = &soc_v1_0_read_register,
272 .get_config_memsize = &soc_v1_0_get_config_memsize,263 .get_config_memsize = &soc_v1_0_get_config_memsize,
273 .get_xclk = &soc_v1_0_get_xclk,264 .get_xclk = &soc_v1_0_get_xclk,
274 .need_full_reset = &soc_v1_0_need_full_reset,
275 .init_doorbell_index = &soc_v1_0_doorbell_index_init,265 .init_doorbell_index = &soc_v1_0_doorbell_index_init,
276 .need_reset_on_init = &soc_v1_0_need_reset_on_init,266 .need_reset_on_init = &soc_v1_0_need_reset_on_init,
277 .encode_ext_smn_addressing = &soc_v1_0_encode_ext_smn_addressing,267 .encode_ext_smn_addressing = &soc_v1_0_encode_ext_smn_addressing,
@@ -600,8 +590,10 @@ static int soc_v1_0_get_xcp_res_info(struct amdgpu_xcp_mgr *xcp_mgr, · 322 unmodified lines
600 xcp_cfg->num_res = ARRAY_SIZE(max_res);590 xcp_cfg->num_res = ARRAY_SIZE(max_res);
601 591
602 for (i = 0; i < xcp_cfg->num_res; i++) {592 for (i = 0; i < xcp_cfg->num_res; i++) {
603 res_lt_xcp = max_res[i] < num_xcp;
604 xcp_cfg->xcp_res[i].id = i;593 xcp_cfg->xcp_res[i].id = i;
594 if (!max_res[i])
595 continue;
596 res_lt_xcp = max_res[i] < num_xcp;
605 xcp_cfg->xcp_res[i].num_inst =597 xcp_cfg->xcp_res[i].num_inst =
606 res_lt_xcp ? 1 : max_res[i] / num_xcp;598 res_lt_xcp ? 1 : max_res[i] / num_xcp;
607 xcp_cfg->xcp_res[i].num_inst =599 xcp_cfg->xcp_res[i].num_inst =
drivers/gpu/drm/amd/amdgpu/tonga_ih.c+0−40
@@ -390,43 +390,6 @@ static int tonga_ih_wait_for_idle(struct amdgpu_ip_block *ip_block) · 389 unmodified lines
390 return -ETIMEDOUT;390 return -ETIMEDOUT;
391}391}
392 392
393static bool tonga_ih_check_soft_reset(struct amdgpu_ip_block *ip_block)
394{
395 struct amdgpu_device *adev = ip_block->adev;
396 u32 srbm_soft_reset = 0;
397 u32 tmp = RREG32(mmSRBM_STATUS);
398
399 if (tmp & SRBM_STATUS__IH_BUSY_MASK)
400 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET,
401 SOFT_RESET_IH, 1);
402
403 if (srbm_soft_reset) {
404 adev->irq.srbm_soft_reset = srbm_soft_reset;
405 return true;
406 } else {
407 adev->irq.srbm_soft_reset = 0;
408 return false;
409 }
410}
411
412static int tonga_ih_pre_soft_reset(struct amdgpu_ip_block *ip_block)
413{
414 if (!ip_block->adev->irq.srbm_soft_reset)
415 return 0;
416
417 return tonga_ih_hw_fini(ip_block);
418}
419
420static int tonga_ih_post_soft_reset(struct amdgpu_ip_block *ip_block)
421{
422 struct amdgpu_device *adev = ip_block->adev;
423
424 if (!adev->irq.srbm_soft_reset)
425 return 0;
426
427 return tonga_ih_hw_init(ip_block);
428}
429
430static int tonga_ih_soft_reset(struct amdgpu_ip_block *ip_block)393static int tonga_ih_soft_reset(struct amdgpu_ip_block *ip_block)
431{394{
432 struct amdgpu_device *adev = ip_block->adev;395 struct amdgpu_device *adev = ip_block->adev;
@@ -481,10 +444,7 @@ static const struct amd_ip_funcs tonga_ih_ip_funcs = { · 48 unmodified lines
481 .resume = tonga_ih_resume,444 .resume = tonga_ih_resume,
482 .is_idle = tonga_ih_is_idle,445 .is_idle = tonga_ih_is_idle,
483 .wait_for_idle = tonga_ih_wait_for_idle,446 .wait_for_idle = tonga_ih_wait_for_idle,
484 .check_soft_reset = tonga_ih_check_soft_reset,
485 .pre_soft_reset = tonga_ih_pre_soft_reset,
486 .soft_reset = tonga_ih_soft_reset,447 .soft_reset = tonga_ih_soft_reset,
487 .post_soft_reset = tonga_ih_post_soft_reset,
488 .set_clockgating_state = tonga_ih_set_clockgating_state,448 .set_clockgating_state = tonga_ih_set_clockgating_state,
489 .set_powergating_state = tonga_ih_set_powergating_state,449 .set_powergating_state = tonga_ih_set_powergating_state,
490};450};
drivers/gpu/drm/amd/amdgpu/umc_v12_0.c+1−584
@@ -28,48 +28,6 @@ · 27 unmodified lines
28#include "umc/umc_12_0_0_sh_mask.h"28#include "umc/umc_12_0_0_sh_mask.h"
29#include "mp/mp_13_0_6_sh_mask.h"29#include "mp/mp_13_0_6_sh_mask.h"
30 30
31#define MAX_ECC_NUM_PER_RETIREMENT 32
32#define DELAYED_TIME_FOR_GPU_RESET 1000 //ms
33
34static inline uint64_t get_umc_v12_0_reg_offset(struct amdgpu_device *adev,
35 uint32_t node_inst,
36 uint32_t umc_inst,
37 uint32_t ch_inst)
38{
39 uint32_t index = umc_inst * adev->umc.channel_inst_num + ch_inst;
40 uint64_t cross_node_offset = (node_inst == 0) ? 0 : UMC_V12_0_CROSS_NODE_OFFSET;
41
42 umc_inst = index / 4;
43 ch_inst = index % 4;
44
45 return adev->umc.channel_offs * ch_inst + UMC_V12_0_INST_DIST * umc_inst +
46 UMC_V12_0_NODE_DIST * node_inst + cross_node_offset;
47}
48
49static int umc_v12_0_reset_error_count_per_channel(struct amdgpu_device *adev,
50 uint32_t node_inst, uint32_t umc_inst,
51 uint32_t ch_inst, void *data)
52{
53 uint64_t odecc_err_cnt_addr;
54 uint64_t umc_reg_offset =
55 get_umc_v12_0_reg_offset(adev, node_inst, umc_inst, ch_inst);
56
57 odecc_err_cnt_addr =
58 SOC15_REG_OFFSET(UMC, 0, regUMCCH0_OdEccErrCnt);
59
60 /* clear error count */
61 WREG32_PCIE_EXT((odecc_err_cnt_addr + umc_reg_offset) * 4,
62 UMC_V12_0_CE_CNT_INIT);
63
64 return 0;
65}
66
67static void umc_v12_0_reset_error_count(struct amdgpu_device *adev)
68{
69 amdgpu_umc_loop_channels(adev,
70 umc_v12_0_reset_error_count_per_channel, NULL);
71}
72
73bool umc_v12_0_is_deferred_error(struct amdgpu_device *adev, uint64_t mc_umc_status)31bool umc_v12_0_is_deferred_error(struct amdgpu_device *adev, uint64_t mc_umc_status)
74{32{
75 dev_dbg(adev->dev,33 dev_dbg(adev->dev,
@@ -115,65 +73,6 @@ bool umc_v12_0_is_correctable_error(struct amdgpu_device *adev, uint64_t mc_umc_ · 39 unmodified lines
115 !(umc_v12_0_is_uncorrectable_error(adev, mc_umc_status)))));73 !(umc_v12_0_is_uncorrectable_error(adev, mc_umc_status)))));
116}74}
117 75
118static void umc_v12_0_query_error_count_per_type(struct amdgpu_device *adev,
119 uint64_t umc_reg_offset,
120 unsigned long *error_count,
121 check_error_type_func error_type_func)
122{
123 uint64_t mc_umc_status;
124 uint64_t mc_umc_status_addr;
125
126 mc_umc_status_addr =
127 SOC15_REG_OFFSET(UMC, 0, regMCA_UMC_UMC0_MCUMC_STATUST0);
128
129 /* Check MCUMC_STATUS */
130 mc_umc_status =
131 RREG64_PCIE_EXT((mc_umc_status_addr + umc_reg_offset) * 4);
132
133 if (error_type_func(adev, mc_umc_status))
134 *error_count += 1;
135}
136
137static int umc_v12_0_query_error_count(struct amdgpu_device *adev,
138 uint32_t node_inst, uint32_t umc_inst,
139 uint32_t ch_inst, void *data)
140{
141 struct ras_err_data *err_data = (struct ras_err_data *)data;
142 unsigned long ue_count = 0, ce_count = 0, de_count = 0;
143
144 /* NOTE: node_inst is converted by adev->umc.active_mask and the range is [0-3],
145 * which can be used as die ID directly */
146 struct amdgpu_smuio_mcm_config_info mcm_info = {
147 .socket_id = adev->smuio.funcs->get_socket_id(adev),
148 .die_id = node_inst,
149 };
150
151 uint64_t umc_reg_offset =
152 get_umc_v12_0_reg_offset(adev, node_inst, umc_inst, ch_inst);
153
154 umc_v12_0_query_error_count_per_type(adev, umc_reg_offset,
155 &ce_count, umc_v12_0_is_correctable_error);
156 umc_v12_0_query_error_count_per_type(adev, umc_reg_offset,
157 &ue_count, umc_v12_0_is_uncorrectable_error);
158 umc_v12_0_query_error_count_per_type(adev, umc_reg_offset,
159 &de_count, umc_v12_0_is_deferred_error);
160
161 amdgpu_ras_error_statistic_ue_count(err_data, &mcm_info, ue_count);
162 amdgpu_ras_error_statistic_ce_count(err_data, &mcm_info, ce_count);
163 amdgpu_ras_error_statistic_de_count(err_data, &mcm_info, de_count);
164
165 return 0;
166}
167
168static void umc_v12_0_query_ras_error_count(struct amdgpu_device *adev,
169 void *ras_error_status)
170{
171 amdgpu_umc_loop_channels(adev,
172 umc_v12_0_query_error_count, ras_error_status);
173
174 umc_v12_0_reset_error_count(adev);
175}
176
177static void umc_v12_0_get_retire_flip_bits(struct amdgpu_device *adev)76static void umc_v12_0_get_retire_flip_bits(struct amdgpu_device *adev)
178{77{
179 enum amdgpu_memory_partition nps = AMDGPU_NPS1_PARTITION_MODE;78 enum amdgpu_memory_partition nps = AMDGPU_NPS1_PARTITION_MODE;
@@ -279,190 +178,6 @@ static void umc_v12_0_get_retire_flip_bits(struct amdgpu_device *adev) · 99 unmodified lines
279 adev->umc.retire_unit = 0x1 << flip_bits->bit_num;178 adev->umc.retire_unit = 0x1 << flip_bits->bit_num;
280}179}
281 180
282static int umc_v12_0_convert_error_address(struct amdgpu_device *adev,
283 struct ras_err_data *err_data,
284 struct ta_ras_query_address_input *addr_in,
285 struct ta_ras_query_address_output *addr_out,
286 bool dump_addr)
287{
288 uint32_t row = 0, row_lower = 0, row_high = 0;
289 uint32_t col = 0, col_lower = 0, bank = 0;
290 uint32_t channel_index = 0, umc_inst = 0;
291 uint32_t i, bit_num, retire_unit, *flip_bits;
292 uint64_t soc_pa, column, err_addr;
293 struct ta_ras_query_address_output addr_out_tmp;
294 struct ta_ras_query_address_output *paddr_out;
295 int ret = 0;
296
297 if (!addr_out)
298 paddr_out = &addr_out_tmp;
299 else
300 paddr_out = addr_out;
301
302 err_addr = bank = 0;
303 if (addr_in) {
304 err_addr = addr_in->ma.err_addr;
305 addr_in->addr_type = TA_RAS_MCA_TO_PA;
306 ret = psp_ras_query_address(&adev->psp, addr_in, paddr_out);
307 if (ret) {
308 dev_warn(adev->dev, "Failed to query RAS physical address for 0x%llx",
309 err_addr);
310
311 goto out;
312 }
313
314 bank = paddr_out->pa.bank;
315 /* no need to care about umc inst if addr_in is NULL */
316 umc_inst = addr_in->ma.umc_inst;
317 }
318
319 flip_bits = adev->umc.flip_bits.flip_bits_in_pa;
320 bit_num = adev->umc.flip_bits.bit_num;
321 retire_unit = adev->umc.retire_unit;
322
323 soc_pa = paddr_out->pa.pa;
324 channel_index = paddr_out->pa.channel_idx;
325 /* clear loop bits in soc physical address */
326 for (i = 0; i < bit_num; i++)
327 soc_pa &= ~BIT_ULL(flip_bits[i]);
328
329 paddr_out->pa.pa = soc_pa;
330 /* get column bit 0 and 1 in mca address */
331 col_lower = (err_addr >> 1) & 0x3ULL;
332 /* extra row bit will be handled later */
333 row_lower = (err_addr >> UMC_V12_0_MA_R0_BIT) & 0x1fffULL;
334 row_lower &= ~BIT_ULL(adev->umc.flip_bits.flip_row_bit);
335
336 if (amdgpu_ip_version(adev, GC_HWIP, 0) >= IP_VERSION(9, 5, 0)) {
337 row_high = (soc_pa >> adev->umc.flip_bits.r13_in_pa) & 0x3ULL;
338 /* it's 2.25GB in each channel, from MCA address to PA
339 * [R14 R13] is converted if the two bits value are 0x3,
340 * get them from PA instead of MCA address.
341 */
342 row_lower |= (row_high << 13);
343 }
344
345 if (!err_data && !dump_addr)
346 goto out;
347
348 /* loop for all possibilities of retired bits */
349 for (column = 0; column < retire_unit; column++) {
350 soc_pa = paddr_out->pa.pa;
351 for (i = 0; i < bit_num; i++)
352 soc_pa |= (((column >> i) & 0x1ULL) << flip_bits[i]);
353
354 col = ((column & 0x7) << 2) | col_lower;
355 /* handle extra row bit */
356 if (bit_num == RETIRE_FLIP_BITS_NUM)
357 row = ((column >> 3) << adev->umc.flip_bits.flip_row_bit) |
358 row_lower;
359
360 if (dump_addr)
361 dev_info(adev->dev,
362 "Error Address(PA):0x%-10llx Row:0x%-4x Col:0x%-2x Bank:0x%x Channel:0x%x\n",
363 soc_pa, row, col, bank, channel_index);
364
365 if (err_data)
366 amdgpu_umc_fill_error_record(err_data, err_addr,
367 soc_pa, channel_index, umc_inst);
368 }
369
370out:
371 return ret;
372}
373
374static int umc_v12_0_query_error_address(struct amdgpu_device *adev,
375 uint32_t node_inst, uint32_t umc_inst,
376 uint32_t ch_inst, void *data)
377{
378 struct ras_err_data *err_data = (struct ras_err_data *)data;
379 struct ta_ras_query_address_input addr_in;
380 uint64_t mc_umc_status_addr;
381 uint64_t mc_umc_status, err_addr;
382 uint64_t mc_umc_addrt0;
383 uint64_t umc_reg_offset =
384 get_umc_v12_0_reg_offset(adev, node_inst, umc_inst, ch_inst);
385
386 mc_umc_status_addr =
387 SOC15_REG_OFFSET(UMC, 0, regMCA_UMC_UMC0_MCUMC_STATUST0);
388
389 mc_umc_status = RREG64_PCIE_EXT((mc_umc_status_addr + umc_reg_offset) * 4);
390
391 if (mc_umc_status == 0)
392 return 0;
393
394 if (!err_data->err_addr) {
395 /* clear umc status */
396 WREG64_PCIE_EXT((mc_umc_status_addr + umc_reg_offset) * 4, 0x0ULL);
397
398 return 0;
399 }
400
401 /* calculate error address if ue error is detected */
402 if (umc_v12_0_is_uncorrectable_error(adev, mc_umc_status) ||
403 umc_v12_0_is_deferred_error(adev, mc_umc_status)) {
404 mc_umc_addrt0 =
405 SOC15_REG_OFFSET(UMC, 0, regMCA_UMC_UMC0_MCUMC_ADDRT0);
406
407 err_addr = RREG64_PCIE_EXT((mc_umc_addrt0 + umc_reg_offset) * 4);
408
409 err_addr = REG_GET_FIELD(err_addr, MCA_UMC_UMC0_MCUMC_ADDRT0, ErrorAddr);
410
411 if (!adev->aid_mask &&
412 adev->smuio.funcs &&
413 adev->smuio.funcs->get_socket_id)
414 addr_in.ma.socket_id = adev->smuio.funcs->get_socket_id(adev);
415 else
416 addr_in.ma.socket_id = 0;
417
418 addr_in.ma.err_addr = err_addr;
419 addr_in.ma.ch_inst = ch_inst;
420 addr_in.ma.umc_inst = umc_inst;
421 addr_in.ma.node_inst = node_inst;
422
423 umc_v12_0_convert_error_address(adev, err_data, &addr_in, NULL, true);
424 }
425
426 /* clear umc status */
427 WREG64_PCIE_EXT((mc_umc_status_addr + umc_reg_offset) * 4, 0x0ULL);
428
429 return 0;
430}
431
432static void umc_v12_0_query_ras_error_address(struct amdgpu_device *adev,
433 void *ras_error_status)
434{
435 amdgpu_umc_loop_channels(adev,
436 umc_v12_0_query_error_address, ras_error_status);
437}
438
439static int umc_v12_0_err_cnt_init_per_channel(struct amdgpu_device *adev,
440 uint32_t node_inst, uint32_t umc_inst,
441 uint32_t ch_inst, void *data)
442{
443 uint32_t odecc_cnt_sel;
444 uint64_t odecc_cnt_sel_addr, odecc_err_cnt_addr;
445 uint64_t umc_reg_offset =
446 get_umc_v12_0_reg_offset(adev, node_inst, umc_inst, ch_inst);
447
448 odecc_cnt_sel_addr =
449 SOC15_REG_OFFSET(UMC, 0, regUMCCH0_OdEccCntSel);
450 odecc_err_cnt_addr =
451 SOC15_REG_OFFSET(UMC, 0, regUMCCH0_OdEccErrCnt);
452
453 odecc_cnt_sel = RREG32_PCIE_EXT((odecc_cnt_sel_addr + umc_reg_offset) * 4);
454
455 /* set ce error interrupt type to APIC based interrupt */
456 odecc_cnt_sel = REG_SET_FIELD(odecc_cnt_sel, UMCCH0_OdEccCntSel,
457 OdEccErrInt, 0x1);
458 WREG32_PCIE_EXT((odecc_cnt_sel_addr + umc_reg_offset) * 4, odecc_cnt_sel);
459
460 /* set error count to initial value */
461 WREG32_PCIE_EXT((odecc_err_cnt_addr + umc_reg_offset) * 4, UMC_V12_0_CE_CNT_INIT);
462
463 return 0;
464}
465
466static bool umc_v12_0_check_ecc_err_status(struct amdgpu_device *adev,181static bool umc_v12_0_check_ecc_err_status(struct amdgpu_device *adev,
467 enum amdgpu_mca_error_type type, void *ras_error_status)182 enum amdgpu_mca_error_type type, void *ras_error_status)
468{183{
@@ -482,309 +197,11 @@ static bool umc_v12_0_check_ecc_err_status(struct amdgpu_device *adev, · 13 unmodified lines
482 return false;197 return false;
483}198}
484 199
485static void umc_v12_0_err_cnt_init(struct amdgpu_device *adev)
486{
487 amdgpu_umc_loop_channels(adev,
488 umc_v12_0_err_cnt_init_per_channel, NULL);
489}
490
491static bool umc_v12_0_query_ras_poison_mode(struct amdgpu_device *adev)
492{
493 /*
494 * Force return true, because regUMCCH0_EccCtrl
495 * is not accessible from host side
496 */
497 return true;
498}
499
500const struct amdgpu_ras_block_hw_ops umc_v12_0_ras_hw_ops = {
501 .query_ras_error_count = umc_v12_0_query_ras_error_count,
502 .query_ras_error_address = umc_v12_0_query_ras_error_address,
503};
504
505static int umc_v12_0_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
506 enum aca_smu_type type, void *data)
507{
508 struct amdgpu_device *adev = handle->adev;
509 struct aca_bank_info info;
510 enum aca_error_type err_type;
511 u64 status, count;
512 u32 ext_error_code;
513 int ret;
514
515 status = bank->regs[ACA_REG_IDX_STATUS];
516 if (umc_v12_0_is_deferred_error(adev, status))
517 err_type = ACA_ERROR_TYPE_DEFERRED;
518 else if (umc_v12_0_is_uncorrectable_error(adev, status))
519 err_type = ACA_ERROR_TYPE_UE;
520 else if (umc_v12_0_is_correctable_error(adev, status))
521 err_type = ACA_ERROR_TYPE_CE;
522 else
523 return 0;
524 bank->aca_err_type = err_type;
525
526 ret = aca_bank_info_decode(bank, &info);
527 if (ret)
528 return ret;
529
530 amdgpu_umc_update_ecc_status(adev,
531 bank->regs[ACA_REG_IDX_STATUS],
532 bank->regs[ACA_REG_IDX_IPID],
533 bank->regs[ACA_REG_IDX_ADDR]);
534
535 ext_error_code = ACA_REG__STATUS__ERRORCODEEXT(status);
536 if (umc_v12_0_is_deferred_error(adev, status))
537 count = ext_error_code == 0 ?
538 adev->umc.err_addr_cnt / adev->umc.retire_unit : 1ULL;
539 else
540 count = ext_error_code == 0 ?
541 ACA_REG__MISC0__ERRCNT(bank->regs[ACA_REG_IDX_MISC0]) : 1ULL;
542
543 return aca_error_cache_log_bank_error(handle, &info, err_type, count);
544}
545
546static const struct aca_bank_ops umc_v12_0_aca_bank_ops = {
547 .aca_bank_parser = umc_v12_0_aca_bank_parser,
548};
549
550const struct aca_info umc_v12_0_aca_info = {
551 .hwip = ACA_HWIP_TYPE_UMC,
552 .mask = ACA_ERROR_UE_MASK | ACA_ERROR_CE_MASK | ACA_ERROR_DEFERRED_MASK,
553 .bank_ops = &umc_v12_0_aca_bank_ops,
554};
555
556static int umc_v12_0_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
557{
558 int ret;
559
560 ret = amdgpu_umc_ras_late_init(adev, ras_block);
561 if (ret)
562 return ret;
563
564 ret = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__UMC,
565 &umc_v12_0_aca_info, NULL);
566 if (ret)
567 return ret;
568
569 return 0;
570}
571
572static int umc_v12_0_update_ecc_status(struct amdgpu_device *adev,
573 uint64_t status, uint64_t ipid, uint64_t addr)
574{
575 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
576 uint16_t hwid, mcatype;
577 uint64_t page_pfn[UMC_V12_0_BAD_PAGE_NUM_PER_CHANNEL];
578 uint64_t err_addr, pa_addr = 0;
579 struct ras_ecc_err *ecc_err;
580 struct ta_ras_query_address_output addr_out;
581 uint32_t shift_bit = adev->umc.flip_bits.flip_bits_in_pa[2];
582 int count, ret, i;
583
584 hwid = REG_GET_FIELD(ipid, MCMP1_IPIDT0, HardwareID);
585 mcatype = REG_GET_FIELD(ipid, MCMP1_IPIDT0, McaType);
586
587 /* The IP block decode of consumption is SMU */
588 if (hwid != MCA_UMC_HWID_V12_0 || mcatype != MCA_UMC_MCATYPE_V12_0) {
589 con->umc_ecc_log.consumption_q_count++;
590 return 0;
591 }
592
593 if (!status)
594 return 0;
595
596 if (!umc_v12_0_is_deferred_error(adev, status))
597 return 0;
598
599 err_addr = REG_GET_FIELD(addr,
600 MCA_UMC_UMC0_MCUMC_ADDRT0, ErrorAddr);
601
602 dev_dbg(adev->dev,
603 "UMC:IPID:0x%llx, socket:%llu, aid:%llu, inst:%llu, ch:%llu, err_addr:0x%llx\n",
604 ipid,
605 MCA_IPID_2_SOCKET_ID(ipid),
606 MCA_IPID_2_DIE_ID(ipid),
607 MCA_IPID_2_UMC_INST(ipid),
608 MCA_IPID_2_UMC_CH(ipid),
609 err_addr);
610
611 ret = amdgpu_umc_mca_to_addr(adev,
612 err_addr, MCA_IPID_2_UMC_CH(ipid),
613 MCA_IPID_2_UMC_INST(ipid), MCA_IPID_2_DIE_ID(ipid),
614 MCA_IPID_2_SOCKET_ID(ipid), &addr_out, true);
615 if (ret)
616 return ret;
617
618 ecc_err = kzalloc_obj(*ecc_err);
619 if (!ecc_err)
620 return -ENOMEM;
621
622 pa_addr = addr_out.pa.pa;
623 ecc_err->status = status;
624 ecc_err->ipid = ipid;
625 ecc_err->addr = addr;
626 ecc_err->pa_pfn = pa_addr >> AMDGPU_GPU_PAGE_SHIFT;
627 ecc_err->channel_idx = addr_out.pa.channel_idx;
628
629 /* If converted pa_pfn is 0, use pa C4 pfn. */
630 if (!ecc_err->pa_pfn)
631 ecc_err->pa_pfn = BIT_ULL(shift_bit) >> AMDGPU_GPU_PAGE_SHIFT;
632
633 ret = amdgpu_umc_logs_ecc_err(adev, &con->umc_ecc_log.de_page_tree, ecc_err);
634 if (ret) {
635 if (ret == -EEXIST)
636 con->umc_ecc_log.de_queried_count++;
637 else
638 dev_err(adev->dev, "Fail to log ecc error! ret:%d\n", ret);
639
640 kfree(ecc_err);
641 return ret;
642 }
643
644 con->umc_ecc_log.de_queried_count++;
645
646 memset(page_pfn, 0, sizeof(page_pfn));
647 count = amdgpu_umc_lookup_bad_pages_in_a_row(adev,
648 pa_addr,
649 page_pfn, ARRAY_SIZE(page_pfn));
650 if (count <= 0) {
651 dev_warn(adev->dev, "Fail to convert error address! count:%d\n", count);
652 return 0;
653 }
654
655 /* Reserve memory */
656 for (i = 0; i < count; i++)
657 amdgpu_ras_reserve_page(adev, page_pfn[i]);
658
659 /* The problem case is as follows:
660 * 1. GPU A triggers a gpu ras reset, and GPU A drives
661 * GPU B to also perform a gpu ras reset.
662 * 2. After gpu B ras reset started, gpu B queried a DE
663 * data. Since the DE data was queried in the ras reset
664 * thread instead of the page retirement thread, bad
665 * page retirement work would not be triggered. Then
666 * even if all gpu resets are completed, the bad pages
667 * will be cached in RAM until GPU B's bad page retirement
668 * work is triggered again and then saved to eeprom.
669 * Trigger delayed work to save the bad pages to eeprom in time
670 * after gpu ras reset is completed.
671 */
672 if (amdgpu_ras_in_recovery(adev))
673 schedule_delayed_work(&con->page_retirement_dwork,
674 msecs_to_jiffies(DELAYED_TIME_FOR_GPU_RESET));
675
676 return 0;
677}
678
679static int umc_v12_0_fill_error_record(struct amdgpu_device *adev,
680 struct ras_ecc_err *ecc_err, void *ras_error_status)
681{
682 struct ras_err_data *err_data = (struct ras_err_data *)ras_error_status;
683 uint64_t page_pfn[UMC_V12_0_BAD_PAGE_NUM_PER_CHANNEL];
684 int ret, i, count;
685
686 if (!err_data || !ecc_err)
687 return -EINVAL;
688
689 memset(page_pfn, 0, sizeof(page_pfn));
690 count = amdgpu_umc_lookup_bad_pages_in_a_row(adev,
691 ecc_err->pa_pfn << AMDGPU_GPU_PAGE_SHIFT,
692 page_pfn, ARRAY_SIZE(page_pfn));
693
694 for (i = 0; i < count; i++) {
695 ret = amdgpu_umc_fill_error_record(err_data,
696 ecc_err->addr,
697 page_pfn[i] << AMDGPU_GPU_PAGE_SHIFT,
698 ecc_err->channel_idx,
699 MCA_IPID_2_UMC_INST(ecc_err->ipid));
700 if (ret)
701 break;
702 }
703
704 err_data->de_count++;
705
706 return ret;
707}
708
709static void umc_v12_0_query_ras_ecc_err_addr(struct amdgpu_device *adev,
710 void *ras_error_status)
711{
712 struct amdgpu_ras *con = amdgpu_ras_get_context(adev);
713 struct ras_ecc_err *entries[MAX_ECC_NUM_PER_RETIREMENT];
714 struct radix_tree_root *ecc_tree;
715 int new_detected, ret, i;
716
717 ecc_tree = &con->umc_ecc_log.de_page_tree;
718
719 mutex_lock(&con->umc_ecc_log.lock);
720 new_detected = radix_tree_gang_lookup_tag(ecc_tree, (void **)entries,
721 0, ARRAY_SIZE(entries), UMC_ECC_NEW_DETECTED_TAG);
722 for (i = 0; i < new_detected; i++) {
723 if (!entries[i])
724 continue;
725
726 ret = umc_v12_0_fill_error_record(adev, entries[i], ras_error_status);
727 if (ret) {
728 dev_err(adev->dev, "Fail to fill umc error record, ret:%d\n", ret);
729 break;
730 }
731 radix_tree_tag_clear(ecc_tree,
732 entries[i]->pa_pfn, UMC_ECC_NEW_DETECTED_TAG);
733 }
734 mutex_unlock(&con->umc_ecc_log.lock);
735}
736
737static uint32_t umc_v12_0_get_die_id(struct amdgpu_device *adev,
738 uint64_t mca_addr, uint64_t retired_page)
739{
740 uint32_t die = 0;
741
742 /* we only calculate die id for nps1 mode right now */
743 die += ((((retired_page >> 12) & 0x1ULL)^
744 ((retired_page >> 20) & 0x1ULL) ^
745 ((retired_page >> 27) & 0x1ULL) ^
746 ((retired_page >> 34) & 0x1ULL) ^
747 ((retired_page >> 41) & 0x1ULL)) << 0);
748
749 /* the original PA_C4 and PA_R13 may be cleared in retired_page, so
750 * get them from mca_addr.
751 */
752 die += ((((retired_page >> 13) & 0x1ULL) ^
753 ((mca_addr >> 5) & 0x1ULL) ^
754 ((retired_page >> 28) & 0x1ULL) ^
755 ((mca_addr >> 23) & 0x1ULL) ^
756 ((retired_page >> 42) & 0x1ULL)) << 1);
757 die &= 3;
758
759 return die;
760}
761
762static void umc_v12_0_mca_ipid_parse(struct amdgpu_device *adev, uint64_t ipid,
763 uint32_t *did, uint32_t *ch, uint32_t *umc_inst, uint32_t *sid)
764{
765 if (did)
766 *did = MCA_IPID_2_DIE_ID(ipid);
767 if (ch)
768 *ch = MCA_IPID_2_UMC_CH(ipid);
769 if (umc_inst)
770 *umc_inst = MCA_IPID_2_UMC_INST(ipid);
771 if (sid)
772 *sid = MCA_IPID_2_SOCKET_ID(ipid);
773}
774
775struct amdgpu_umc_ras umc_v12_0_ras = {200struct amdgpu_umc_ras umc_v12_0_ras = {
776 .ras_block = {201 .ras_block = {
777 .hw_ops = &umc_v12_0_ras_hw_ops,202 .hw_ops = NULL,
778 .ras_late_init = umc_v12_0_ras_late_init,
779 },203 },
780 .err_cnt_init = umc_v12_0_err_cnt_init,
781 .query_ras_poison_mode = umc_v12_0_query_ras_poison_mode,
782 .ecc_info_query_ras_error_address = umc_v12_0_query_ras_ecc_err_addr,
783 .check_ecc_err_status = umc_v12_0_check_ecc_err_status,204 .check_ecc_err_status = umc_v12_0_check_ecc_err_status,
784 .update_ecc_status = umc_v12_0_update_ecc_status,
785 .convert_ras_err_addr = umc_v12_0_convert_error_address,
786 .get_die_id_from_pa = umc_v12_0_get_die_id,
787 .get_retire_flip_bits = umc_v12_0_get_retire_flip_bits,205 .get_retire_flip_bits = umc_v12_0_get_retire_flip_bits,
788 .mca_ipid_parse = umc_v12_0_mca_ipid_parse,
789};206};
790 207
drivers/gpu/drm/amd/amdgpu/umc_v12_0.h+0−28
@@ -26,31 +26,6 @@ · 25 unmodified lines
26#include "soc15_common.h"26#include "soc15_common.h"
27#include "amdgpu.h"27#include "amdgpu.h"
28 28
29#define UMC_V12_0_NODE_DIST 0x40000000
30#define UMC_V12_0_INST_DIST 0x40000
31
32/* UMC register per channel offset */
33#define UMC_V12_0_PER_CHANNEL_OFFSET 0x400
34
35/* UMC cross node offset */
36#define UMC_V12_0_CROSS_NODE_OFFSET 0x100000000
37
38/* OdEccErrCnt max value */
39#define UMC_V12_0_CE_CNT_MAX 0xffff
40/* umc ce interrupt threshold */
41#define UMC_V12_0_CE_INT_THRESHOLD 0xffff
42/* umc ce count initial value */
43#define UMC_V12_0_CE_CNT_INIT (UMC_V12_0_CE_CNT_MAX - UMC_V12_0_CE_INT_THRESHOLD)
44
45/* number of umc channel instance with memory map register access */
46#define UMC_V12_0_CHANNEL_INSTANCE_NUM 8
47/* number of umc instance with memory map register access */
48#define UMC_V12_0_UMC_INSTANCE_NUM 4
49
50/* Total channel instances for all available umc nodes */
51#define UMC_V12_0_TOTAL_CHANNEL_NUM(adev) \
52 (UMC_V12_0_CHANNEL_INSTANCE_NUM * (adev)->gmc.num_umc)
53
54/* one piece of normalized address is mapped to 8 pieces of physical address */29/* one piece of normalized address is mapped to 8 pieces of physical address */
55#define UMC_V12_0_NA_MAP_PA_NUM 830#define UMC_V12_0_NA_MAP_PA_NUM 8
56/* R13 bit shift should be considered, double the number */31/* R13 bit shift should be considered, double the number */
@@ -75,9 +50,6 @@ · 18 unmodified lines
75/* row bits in MCA address */50/* row bits in MCA address */
76#define UMC_V12_0_MA_R0_BIT 1051#define UMC_V12_0_MA_R0_BIT 10
77 52
78#define MCA_UMC_HWID_V12_0 0x96
79#define MCA_UMC_MCATYPE_V12_0 0x0
80
81#define MCA_IPID_LO_2_UMC_CH(_ipid_lo) (((((_ipid_lo) >> 20) & 0x1) * 4) + \53#define MCA_IPID_LO_2_UMC_CH(_ipid_lo) (((((_ipid_lo) >> 20) & 0x1) * 4) + \
82 (((_ipid_lo) >> 12) & 0xF))54 (((_ipid_lo) >> 12) & 0xF))
83#define MCA_IPID_LO_2_UMC_INST(_ipid_lo) (((_ipid_lo) >> 21) & 0x7)55#define MCA_IPID_LO_2_UMC_INST(_ipid_lo) (((_ipid_lo) >> 21) & 0x7)
drivers/gpu/drm/amd/amdgpu/uvd_v6_0.c+0−45
@@ -1165,36 +1165,6 @@ static int uvd_v6_0_wait_for_idle(struct amdgpu_ip_block *ip_block) · 1164 unmodified lines
1165}1165}
1166 1166
1167#define AMDGPU_UVD_STATUS_BUSY_MASK 0xfd1167#define AMDGPU_UVD_STATUS_BUSY_MASK 0xfd
1168static bool uvd_v6_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
1169{
1170 struct amdgpu_device *adev = ip_block->adev;
1171 u32 srbm_soft_reset = 0;
1172 u32 tmp = RREG32(mmSRBM_STATUS);
1173
1174 if (REG_GET_FIELD(tmp, SRBM_STATUS, UVD_RQ_PENDING) ||
1175 REG_GET_FIELD(tmp, SRBM_STATUS, UVD_BUSY) ||
1176 (RREG32(mmUVD_STATUS) & AMDGPU_UVD_STATUS_BUSY_MASK))
1177 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_UVD, 1);
1178
1179 if (srbm_soft_reset) {
1180 adev->uvd.inst->srbm_soft_reset = srbm_soft_reset;
1181 return true;
1182 } else {
1183 adev->uvd.inst->srbm_soft_reset = 0;
1184 return false;
1185 }
1186}
1187
1188static int uvd_v6_0_pre_soft_reset(struct amdgpu_ip_block *ip_block)
1189{
1190 struct amdgpu_device *adev = ip_block->adev;
1191
1192 if (!adev->uvd.inst->srbm_soft_reset)
1193 return 0;
1194
1195 uvd_v6_0_stop(adev);
1196 return 0;
1197}
1198 1168
1199static int uvd_v6_0_soft_reset(struct amdgpu_ip_block *ip_block)1169static int uvd_v6_0_soft_reset(struct amdgpu_ip_block *ip_block)
1200{1170{
@@ -1227,18 +1197,6 @@ static int uvd_v6_0_soft_reset(struct amdgpu_ip_block *ip_block) · 26 unmodified lines
1227 return 0;1197 return 0;
1228}1198}
1229 1199
1230static int uvd_v6_0_post_soft_reset(struct amdgpu_ip_block *ip_block)
1231{
1232 struct amdgpu_device *adev = ip_block->adev;
1233
1234 if (!adev->uvd.inst->srbm_soft_reset)
1235 return 0;
1236
1237 mdelay(5);
1238
1239 return uvd_v6_0_start(adev);
1240}
1241
1242static int uvd_v6_0_set_interrupt_state(struct amdgpu_device *adev,1200static int uvd_v6_0_set_interrupt_state(struct amdgpu_device *adev,
1243 struct amdgpu_irq_src *source,1201 struct amdgpu_irq_src *source,
1244 unsigned type,1202 unsigned type,
@@ -1538,10 +1496,7 @@ static const struct amd_ip_funcs uvd_v6_0_ip_funcs = { · 293 unmodified lines
1538 .resume = uvd_v6_0_resume,1496 .resume = uvd_v6_0_resume,
1539 .is_idle = uvd_v6_0_is_idle,1497 .is_idle = uvd_v6_0_is_idle,
1540 .wait_for_idle = uvd_v6_0_wait_for_idle,1498 .wait_for_idle = uvd_v6_0_wait_for_idle,
1541 .check_soft_reset = uvd_v6_0_check_soft_reset,
1542 .pre_soft_reset = uvd_v6_0_pre_soft_reset,
1543 .soft_reset = uvd_v6_0_soft_reset,1499 .soft_reset = uvd_v6_0_soft_reset,
1544 .post_soft_reset = uvd_v6_0_post_soft_reset,
1545 .set_clockgating_state = uvd_v6_0_set_clockgating_state,1500 .set_clockgating_state = uvd_v6_0_set_clockgating_state,
1546 .set_powergating_state = uvd_v6_0_set_powergating_state,1501 .set_powergating_state = uvd_v6_0_set_powergating_state,
1547 .get_clockgating_state = uvd_v6_0_get_clockgating_state,1502 .get_clockgating_state = uvd_v6_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/vce_v3_0.c+0−69
@@ -631,47 +631,6 @@ static int vce_v3_0_wait_for_idle(struct amdgpu_ip_block *ip_block) · 630 unmodified lines
631#define AMDGPU_VCE_STATUS_BUSY_MASK (VCE_STATUS_VCPU_REPORT_AUTO_BUSY_MASK | \631#define AMDGPU_VCE_STATUS_BUSY_MASK (VCE_STATUS_VCPU_REPORT_AUTO_BUSY_MASK | \
632 VCE_STATUS_VCPU_REPORT_RB0_BUSY_MASK)632 VCE_STATUS_VCPU_REPORT_RB0_BUSY_MASK)
633 633
634static bool vce_v3_0_check_soft_reset(struct amdgpu_ip_block *ip_block)
635{
636 struct amdgpu_device *adev = ip_block->adev;
637 u32 srbm_soft_reset = 0;
638
639 /* According to VCE team , we should use VCE_STATUS instead
640 * SRBM_STATUS.VCE_BUSY bit for busy status checking.
641 * GRBM_GFX_INDEX.INSTANCE_INDEX is used to specify which VCE
642 * instance's registers are accessed
643 * (0 for 1st instance, 10 for 2nd instance).
644 *
645 *VCE_STATUS
646 *|UENC|ACPI|AUTO ACTIVE|RB1 |RB0 |RB2 | |FW_LOADED|JOB |
647 *|----+----+-----------+----+----+----+----------+---------+----|
648 *|bit8|bit7| bit6 |bit5|bit4|bit3| bit2 | bit1 |bit0|
649 *
650 * VCE team suggest use bit 3--bit 6 for busy status check
651 */
652 mutex_lock(&adev->grbm_idx_mutex);
653 WREG32(mmGRBM_GFX_INDEX, GET_VCE_INSTANCE(0));
654 if (RREG32(mmVCE_STATUS) & AMDGPU_VCE_STATUS_BUSY_MASK) {
655 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_VCE0, 1);
656 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_VCE1, 1);
657 }
658 WREG32(mmGRBM_GFX_INDEX, GET_VCE_INSTANCE(1));
659 if (RREG32(mmVCE_STATUS) & AMDGPU_VCE_STATUS_BUSY_MASK) {
660 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_VCE0, 1);
661 srbm_soft_reset = REG_SET_FIELD(srbm_soft_reset, SRBM_SOFT_RESET, SOFT_RESET_VCE1, 1);
662 }
663 WREG32(mmGRBM_GFX_INDEX, GET_VCE_INSTANCE(0));
664 mutex_unlock(&adev->grbm_idx_mutex);
665
666 if (srbm_soft_reset) {
667 adev->vce.srbm_soft_reset = srbm_soft_reset;
668 return true;
669 } else {
670 adev->vce.srbm_soft_reset = 0;
671 return false;
672 }
673}
674
675static int vce_v3_0_soft_reset(struct amdgpu_ip_block *ip_block)634static int vce_v3_0_soft_reset(struct amdgpu_ip_block *ip_block)
676{635{
677 struct amdgpu_device *adev = ip_block->adev;636 struct amdgpu_device *adev = ip_block->adev;
@@ -703,31 +662,6 @@ static int vce_v3_0_soft_reset(struct amdgpu_ip_block *ip_block) · 25 unmodified lines
703 return 0;662 return 0;
704}663}
705 664
706static int vce_v3_0_pre_soft_reset(struct amdgpu_ip_block *ip_block)
707{
708 struct amdgpu_device *adev = ip_block->adev;
709
710 if (!adev->vce.srbm_soft_reset)
711 return 0;
712
713 mdelay(5);
714
715 return vce_v3_0_suspend(ip_block);
716}
717
718
719static int vce_v3_0_post_soft_reset(struct amdgpu_ip_block *ip_block)
720{
721 struct amdgpu_device *adev = ip_block->adev;
722
723 if (!adev->vce.srbm_soft_reset)
724 return 0;
725
726 mdelay(5);
727
728 return vce_v3_0_resume(ip_block);
729}
730
731static int vce_v3_0_set_interrupt_state(struct amdgpu_device *adev,665static int vce_v3_0_set_interrupt_state(struct amdgpu_device *adev,
732 struct amdgpu_irq_src *source,666 struct amdgpu_irq_src *source,
733 unsigned type,667 unsigned type,
@@ -909,10 +843,7 @@ static const struct amd_ip_funcs vce_v3_0_ip_funcs = { · 175 unmodified lines
909 .resume = vce_v3_0_resume,843 .resume = vce_v3_0_resume,
910 .is_idle = vce_v3_0_is_idle,844 .is_idle = vce_v3_0_is_idle,
911 .wait_for_idle = vce_v3_0_wait_for_idle,845 .wait_for_idle = vce_v3_0_wait_for_idle,
912 .check_soft_reset = vce_v3_0_check_soft_reset,
913 .pre_soft_reset = vce_v3_0_pre_soft_reset,
914 .soft_reset = vce_v3_0_soft_reset,846 .soft_reset = vce_v3_0_soft_reset,
915 .post_soft_reset = vce_v3_0_post_soft_reset,
916 .set_clockgating_state = vce_v3_0_set_clockgating_state,847 .set_clockgating_state = vce_v3_0_set_clockgating_state,
917 .set_powergating_state = vce_v3_0_set_powergating_state,848 .set_powergating_state = vce_v3_0_set_powergating_state,
918 .get_clockgating_state = vce_v3_0_get_clockgating_state,849 .get_clockgating_state = vce_v3_0_get_clockgating_state,
drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c+5−7
@@ -159,9 +159,8 @@ static void vcn_v2_5_ring_begin_use(struct amdgpu_ring *ring) · 158 unmodified lines
159 struct amdgpu_device *adev = ring->adev;159 struct amdgpu_device *adev = ring->adev;
160 struct amdgpu_vcn_inst *v = &adev->vcn.inst[ring->me];160 struct amdgpu_vcn_inst *v = &adev->vcn.inst[ring->me];
161 161
162 atomic_inc(&adev->vcn.inst[0].total_submission_cnt);162 if (!atomic_fetch_inc(&adev->vcn.inst[0].total_submission_cnt))
163 163 cancel_delayed_work_sync(&adev->vcn.inst[0].idle_work);
164 cancel_delayed_work_sync(&adev->vcn.inst[0].idle_work);
165 164
166 /* We can safely return early here because we've cancelled the165 /* We can safely return early here because we've cancelled the
167 * the delayed work so there is no one else to set it to false166 * the delayed work so there is no one else to set it to false
@@ -207,10 +206,9 @@ static void vcn_v2_5_ring_end_use(struct amdgpu_ring *ring) · 39 unmodified lines
207 !adev->vcn.inst[ring->me].using_unified_queue)206 !adev->vcn.inst[ring->me].using_unified_queue)
208 atomic_dec(&adev->vcn.inst[ring->me].dpg_enc_submission_cnt);207 atomic_dec(&adev->vcn.inst[ring->me].dpg_enc_submission_cnt);
209 208
210 atomic_dec(&adev->vcn.inst[0].total_submission_cnt);209 if (atomic_dec_and_test(&adev->vcn.inst[0].total_submission_cnt))
211 210 schedule_delayed_work(&adev->vcn.inst[0].idle_work,
212 schedule_delayed_work(&adev->vcn.inst[0].idle_work,211 VCN_IDLE_TIMEOUT);
213 VCN_IDLE_TIMEOUT);
214}212}
215 213
216/**214/**
drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c+40−5
@@ -1995,7 +1995,7 @@ static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, · 1994 unmodified lines
1995 return amdgpu_ring_reset_helper_end(ring, timedout_fence);1995 return amdgpu_ring_reset_helper_end(ring, timedout_fence);
1996}1996}
1997 1997
1998static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {1998static const struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
1999 .type = AMDGPU_RING_TYPE_VCN_ENC,1999 .type = AMDGPU_RING_TYPE_VCN_ENC,
2000 .align_mask = 0x3f,2000 .align_mask = 0x3f,
2001 .nop = VCN_ENC_CMD_NO_OP,2001 .nop = VCN_ENC_CMD_NO_OP,
@@ -2028,6 +2028,40 @@ static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { · 26 unmodified lines
2028 .reset = vcn_v4_0_ring_reset,2028 .reset = vcn_v4_0_ring_reset,
2029};2029};
2030 2030
2031static const struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs_secure = {
2032 .type = AMDGPU_RING_TYPE_VCN_ENC,
2033 .align_mask = 0x3f,
2034 .nop = VCN_ENC_CMD_NO_OP,
2035 .secure_submission_supported = true,
2036 .no_user_fence = true,
2037 .extra_bytes = sizeof(struct amdgpu_vcn_rb_metadata),
2038 .get_rptr = vcn_v4_0_unified_ring_get_rptr,
2039 .get_wptr = vcn_v4_0_unified_ring_get_wptr,
2040 .set_wptr = vcn_v4_0_unified_ring_set_wptr,
2041 .patch_cs_in_place = vcn_v4_0_ring_patch_cs_in_place,
2042 .emit_frame_size =
2043 SOC15_FLUSH_GPU_TLB_NUM_WREG * 3 +
2044 SOC15_FLUSH_GPU_TLB_NUM_REG_WAIT * 4 +
2045 4 + /* vcn_v2_0_enc_ring_emit_vm_flush */
2046 5 + 5 + /* vcn_v2_0_enc_ring_emit_fence x2 vm fence */
2047 1, /* vcn_v2_0_enc_ring_insert_end */
2048 .emit_ib_size = 5, /* vcn_v2_0_enc_ring_emit_ib */
2049 .emit_ib = vcn_v2_0_enc_ring_emit_ib,
2050 .emit_fence = vcn_v2_0_enc_ring_emit_fence,
2051 .emit_vm_flush = vcn_v2_0_enc_ring_emit_vm_flush,
2052 .test_ring = amdgpu_vcn_enc_ring_test_ring,
2053 .test_ib = amdgpu_vcn_unified_ring_test_ib,
2054 .insert_nop = amdgpu_ring_insert_nop,
2055 .insert_end = vcn_v2_0_enc_ring_insert_end,
2056 .pad_ib = amdgpu_ring_generic_pad_ib,
2057 .begin_use = amdgpu_vcn_ring_begin_use,
2058 .end_use = amdgpu_vcn_ring_end_use,
2059 .emit_wreg = vcn_v2_0_enc_ring_emit_wreg,
2060 .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait,
2061 .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper,
2062 .reset = vcn_v4_0_ring_reset,
2063};
2064
2031/**2065/**
2032 * vcn_v4_0_set_unified_ring_funcs - set unified ring functions2066 * vcn_v4_0_set_unified_ring_funcs - set unified ring functions
2033 *2067 *
@@ -2044,10 +2078,11 @@ static void vcn_v4_0_set_unified_ring_funcs(struct amdgpu_device *adev) · 10 unmodified lines
2044 continue;2078 continue;
2045 2079
2046 if (amdgpu_ip_version(adev, VCN_HWIP, 0) == IP_VERSION(4, 0, 2))2080 if (amdgpu_ip_version(adev, VCN_HWIP, 0) == IP_VERSION(4, 0, 2))
2047 vcn_v4_0_unified_ring_vm_funcs.secure_submission_supported = true;2081 adev->vcn.inst[i].ring_enc[0].funcs =
2048 2082 &vcn_v4_0_unified_ring_vm_funcs_secure;
2049 adev->vcn.inst[i].ring_enc[0].funcs =2083 else
2050 (const struct amdgpu_ring_funcs *)&vcn_v4_0_unified_ring_vm_funcs;2084 adev->vcn.inst[i].ring_enc[0].funcs =
2085 &vcn_v4_0_unified_ring_vm_funcs;
2051 adev->vcn.inst[i].ring_enc[0].me = i;2086 adev->vcn.inst[i].ring_enc[0].me = i;
2052 }2087 }
2053}2088}
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_3.c+17−70
@@ -115,6 +115,19 @@ static int vcn_v4_0_3_early_init(struct amdgpu_ip_block *ip_block) · 114 unmodified lines
115 struct amdgpu_device *adev = ip_block->adev;115 struct amdgpu_device *adev = ip_block->adev;
116 int i, r;116 int i, r;
117 117
118 switch (amdgpu_user_queue) {
119 case -1:
120 case 0:
121 default:
122 adev->vcn.disable_kq = false;
123 adev->vcn.disable_uq = true;
124 break;
125 case 2:
126 adev->vcn.disable_kq = true;
127 adev->vcn.disable_uq = true;
128 break;
129 }
130
118 for (i = 0; i < adev->vcn.num_vcn_inst; ++i)131 for (i = 0; i < adev->vcn.num_vcn_inst; ++i)
119 /* re-use enc ring as unified ring */132 /* re-use enc ring as unified ring */
120 adev->vcn.inst[i].num_enc_rings = 1;133 adev->vcn.inst[i].num_enc_rings = 1;
@@ -217,6 +230,10 @@ static int vcn_v4_0_3_sw_init(struct amdgpu_ip_block *ip_block) · 96 unmodified lines
217 230
218 ring = &adev->vcn.inst[i].ring_enc[0];231 ring = &adev->vcn.inst[i].ring_enc[0];
219 ring->use_doorbell = true;232 ring->use_doorbell = true;
233 if (adev->vcn.disable_kq) {
234 ring->no_scheduler = true;
235 ring->no_user_submission = true;
236 }
220 237
221 if (!amdgpu_sriov_vf(adev))238 if (!amdgpu_sriov_vf(adev))
222 ring->doorbell_index =239 ring->doorbell_index =
@@ -2146,71 +2163,6 @@ static const struct amdgpu_ras_block_hw_ops vcn_v4_0_3_ras_hw_ops = { · 1923 unmodified lines
2146 .query_poison_status = vcn_v4_0_3_query_poison_status,2163 .query_poison_status = vcn_v4_0_3_query_poison_status,
2147};2164};
2148 2165
2149static int vcn_v4_0_3_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
2150 enum aca_smu_type type, void *data)
2151{
2152 struct aca_bank_info info;
2153 u64 misc0;
2154 int ret;
2155
2156 ret = aca_bank_info_decode(bank, &info);
2157 if (ret)
2158 return ret;
2159
2160 misc0 = bank->regs[ACA_REG_IDX_MISC0];
2161 switch (type) {
2162 case ACA_SMU_TYPE_UE:
2163 bank->aca_err_type = ACA_ERROR_TYPE_UE;
2164 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE,
2165 1ULL);
2166 break;
2167 case ACA_SMU_TYPE_CE:
2168 bank->aca_err_type = ACA_ERROR_TYPE_CE;
2169 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
2170 ACA_REG__MISC0__ERRCNT(misc0));
2171 break;
2172 default:
2173 return -EINVAL;
2174 }
2175
2176 return ret;
2177}
2178
2179/* reference to smu driver if header file */
2180static int vcn_v4_0_3_err_codes[] = {
2181 14, 15, /* VCN */
2182};
2183
2184static bool vcn_v4_0_3_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
2185 enum aca_smu_type type, void *data)
2186{
2187 u32 instlo;
2188
2189 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
2190 instlo &= GENMASK(31, 1);
2191
2192 if (instlo != mmSMNAID_AID0_MCA_SMU)
2193 return false;
2194
2195 if (aca_bank_check_error_codes(handle->adev, bank,
2196 vcn_v4_0_3_err_codes,
2197 ARRAY_SIZE(vcn_v4_0_3_err_codes)))
2198 return false;
2199
2200 return true;
2201}
2202
2203static const struct aca_bank_ops vcn_v4_0_3_aca_bank_ops = {
2204 .aca_bank_parser = vcn_v4_0_3_aca_bank_parser,
2205 .aca_bank_is_valid = vcn_v4_0_3_aca_bank_is_valid,
2206};
2207
2208static const struct aca_info vcn_v4_0_3_aca_info = {
2209 .hwip = ACA_HWIP_TYPE_SMU,
2210 .mask = ACA_ERROR_UE_MASK,
2211 .bank_ops = &vcn_v4_0_3_aca_bank_ops,
2212};
2213
2214static int vcn_v4_0_3_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)2166static int vcn_v4_0_3_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
2215{2167{
2216 int r;2168 int r;
@@ -2226,11 +2178,6 @@ static int vcn_v4_0_3_ras_late_init(struct amdgpu_device *adev, struct ras_commo · 9 unmodified lines
2226 goto late_fini;2178 goto late_fini;
2227 }2179 }
2228 2180
2229 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__VCN,
2230 &vcn_v4_0_3_aca_info, NULL);
2231 if (r)
2232 goto late_fini;
2233
2234 return 0;2181 return 0;
2235 2182
2236late_fini:2183late_fini:
drivers/gpu/drm/amd/amdgpu/vcn_v4_0_5.c+2−4
@@ -1479,10 +1479,11 @@ static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, · 1478 unmodified lines
1479 return amdgpu_ring_reset_helper_end(ring, timedout_fence);1479 return amdgpu_ring_reset_helper_end(ring, timedout_fence);
1480}1480}
1481 1481
1482static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = {1482static const struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = {
1483 .type = AMDGPU_RING_TYPE_VCN_ENC,1483 .type = AMDGPU_RING_TYPE_VCN_ENC,
1484 .align_mask = 0x3f,1484 .align_mask = 0x3f,
1485 .nop = VCN_ENC_CMD_NO_OP,1485 .nop = VCN_ENC_CMD_NO_OP,
1486 .secure_submission_supported = true,
1486 .no_user_fence = true,1487 .no_user_fence = true,
1487 .get_rptr = vcn_v4_0_5_unified_ring_get_rptr,1488 .get_rptr = vcn_v4_0_5_unified_ring_get_rptr,
1488 .get_wptr = vcn_v4_0_5_unified_ring_get_wptr,1489 .get_wptr = vcn_v4_0_5_unified_ring_get_wptr,
@@ -1525,9 +1526,6 @@ static void vcn_v4_0_5_set_unified_ring_funcs(struct amdgpu_device *adev) · 36 unmodified lines
1525 if (adev->vcn.harvest_config & (1 << i))1526 if (adev->vcn.harvest_config & (1 << i))
1526 continue;1527 continue;
1527 1528
1528 if (amdgpu_ip_version(adev, VCN_HWIP, 0) == IP_VERSION(4, 0, 5))
1529 vcn_v4_0_5_unified_ring_vm_funcs.secure_submission_supported = true;
1530
1531 adev->vcn.inst[i].ring_enc[0].funcs = &vcn_v4_0_5_unified_ring_vm_funcs;1529 adev->vcn.inst[i].ring_enc[0].funcs = &vcn_v4_0_5_unified_ring_vm_funcs;
1532 adev->vcn.inst[i].ring_enc[0].me = i;1530 adev->vcn.inst[i].ring_enc[0].me = i;
1533 }1531 }
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_1.c+17−73
@@ -94,6 +94,19 @@ static int vcn_v5_0_1_early_init(struct amdgpu_ip_block *ip_block) · 93 unmodified lines
94 struct amdgpu_device *adev = ip_block->adev;94 struct amdgpu_device *adev = ip_block->adev;
95 int i, r;95 int i, r;
96 96
97 switch (amdgpu_user_queue) {
98 case -1:
99 case 0:
100 default:
101 adev->vcn.disable_kq = false;
102 adev->vcn.disable_uq = true;
103 break;
104 case 2:
105 adev->vcn.disable_kq = true;
106 adev->vcn.disable_uq = true;
107 break;
108 }
109
97 for (i = 0; i < adev->vcn.num_vcn_inst; ++i)110 for (i = 0; i < adev->vcn.num_vcn_inst; ++i)
98 /* re-use enc ring as unified ring */111 /* re-use enc ring as unified ring */
99 adev->vcn.inst[i].num_enc_rings = 1;112 adev->vcn.inst[i].num_enc_rings = 1;
@@ -188,6 +201,10 @@ static int vcn_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 88 unmodified lines
188 201
189 ring = &adev->vcn.inst[i].ring_enc[0];202 ring = &adev->vcn.inst[i].ring_enc[0];
190 ring->use_doorbell = true;203 ring->use_doorbell = true;
204 if (adev->vcn.disable_kq) {
205 ring->no_scheduler = true;
206 ring->no_user_submission = true;
207 }
191 if (!amdgpu_sriov_vf(adev))208 if (!amdgpu_sriov_vf(adev))
192 ring->doorbell_index =209 ring->doorbell_index =
193 (adev->doorbell_index.vcn.vcn_ring0_1 << 1) +210 (adev->doorbell_index.vcn.vcn_ring0_1 << 1) +
@@ -1657,10 +1674,7 @@ static const struct amd_ip_funcs vcn_v5_0_1_ip_funcs = { · 1463 unmodified lines
1657 .resume = vcn_v5_0_1_resume,1674 .resume = vcn_v5_0_1_resume,
1658 .is_idle = vcn_v5_0_1_is_idle,1675 .is_idle = vcn_v5_0_1_is_idle,
1659 .wait_for_idle = vcn_v5_0_1_wait_for_idle,1676 .wait_for_idle = vcn_v5_0_1_wait_for_idle,
1660 .check_soft_reset = NULL,
1661 .pre_soft_reset = NULL,
1662 .soft_reset = NULL,1677 .soft_reset = NULL,
1663 .post_soft_reset = NULL,
1664 .set_clockgating_state = vcn_v5_0_1_set_clockgating_state,1678 .set_clockgating_state = vcn_v5_0_1_set_clockgating_state,
1665 .set_powergating_state = vcn_set_powergating_state,1679 .set_powergating_state = vcn_set_powergating_state,
1666 .dump_ip_state = amdgpu_vcn_dump_ip_state,1680 .dump_ip_state = amdgpu_vcn_dump_ip_state,
@@ -1713,71 +1727,6 @@ static const struct amdgpu_ras_block_hw_ops vcn_v5_0_1_ras_hw_ops = { · 46 unmodified lines
1713 .query_poison_status = vcn_v5_0_1_query_poison_status,1727 .query_poison_status = vcn_v5_0_1_query_poison_status,
1714};1728};
1715 1729
1716static int vcn_v5_0_1_aca_bank_parser(struct aca_handle *handle, struct aca_bank *bank,
1717 enum aca_smu_type type, void *data)
1718{
1719 struct aca_bank_info info;
1720 u64 misc0;
1721 int ret;
1722
1723 ret = aca_bank_info_decode(bank, &info);
1724 if (ret)
1725 return ret;
1726
1727 misc0 = bank->regs[ACA_REG_IDX_MISC0];
1728 switch (type) {
1729 case ACA_SMU_TYPE_UE:
1730 bank->aca_err_type = ACA_ERROR_TYPE_UE;
1731 ret = aca_error_cache_log_bank_error(handle, &info, ACA_ERROR_TYPE_UE,
1732 1ULL);
1733 break;
1734 case ACA_SMU_TYPE_CE:
1735 bank->aca_err_type = ACA_ERROR_TYPE_CE;
1736 ret = aca_error_cache_log_bank_error(handle, &info, bank->aca_err_type,
1737 ACA_REG__MISC0__ERRCNT(misc0));
1738 break;
1739 default:
1740 return -EINVAL;
1741 }
1742
1743 return ret;
1744}
1745
1746/* reference to smu driver if header file */
1747static int vcn_v5_0_1_err_codes[] = {
1748 14, 15, 47, /* VCN [D|V|S] */
1749};
1750
1751static bool vcn_v5_0_1_aca_bank_is_valid(struct aca_handle *handle, struct aca_bank *bank,
1752 enum aca_smu_type type, void *data)
1753{
1754 u32 instlo;
1755
1756 instlo = ACA_REG__IPID__INSTANCEIDLO(bank->regs[ACA_REG_IDX_IPID]);
1757 instlo &= GENMASK(31, 1);
1758
1759 if (instlo != mmSMNAID_AID0_MCA_SMU)
1760 return false;
1761
1762 if (aca_bank_check_error_codes(handle->adev, bank,
1763 vcn_v5_0_1_err_codes,
1764 ARRAY_SIZE(vcn_v5_0_1_err_codes)))
1765 return false;
1766
1767 return true;
1768}
1769
1770static const struct aca_bank_ops vcn_v5_0_1_aca_bank_ops = {
1771 .aca_bank_parser = vcn_v5_0_1_aca_bank_parser,
1772 .aca_bank_is_valid = vcn_v5_0_1_aca_bank_is_valid,
1773};
1774
1775static const struct aca_info vcn_v5_0_1_aca_info = {
1776 .hwip = ACA_HWIP_TYPE_SMU,
1777 .mask = ACA_ERROR_UE_MASK,
1778 .bank_ops = &vcn_v5_0_1_aca_bank_ops,
1779};
1780
1781static int vcn_v5_0_1_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)1730static int vcn_v5_0_1_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block)
1782{1731{
1783 int r;1732 int r;
@@ -1786,11 +1735,6 @@ static int vcn_v5_0_1_ras_late_init(struct amdgpu_device *adev, struct ras_commo · 2 unmodified lines
1786 if (r)1735 if (r)
1787 return r;1736 return r;
1788 1737
1789 r = amdgpu_ras_bind_aca(adev, AMDGPU_RAS_BLOCK__VCN,
1790 &vcn_v5_0_1_aca_info, NULL);
1791 if (r)
1792 goto late_fini;
1793
1794 if (amdgpu_ras_is_supported(adev, ras_block->block) &&1738 if (amdgpu_ras_is_supported(adev, ras_block->block) &&
1795 adev->vcn.inst->ras_poison_irq.funcs) {1739 adev->vcn.inst->ras_poison_irq.funcs) {
1796 r = amdgpu_irq_get(adev, &adev->vcn.inst->ras_poison_irq, 0);1740 r = amdgpu_irq_get(adev, &adev->vcn.inst->ras_poison_irq, 0);
drivers/gpu/drm/amd/amdgpu/vcn_v5_0_2.c+0−3
@@ -1203,10 +1203,7 @@ static const struct amd_ip_funcs vcn_v5_0_2_ip_funcs = { · 1202 unmodified lines
1203 .resume = vcn_v5_0_2_resume,1203 .resume = vcn_v5_0_2_resume,
1204 .is_idle = vcn_v5_0_2_is_idle,1204 .is_idle = vcn_v5_0_2_is_idle,
1205 .wait_for_idle = vcn_v5_0_2_wait_for_idle,1205 .wait_for_idle = vcn_v5_0_2_wait_for_idle,
1206 .check_soft_reset = NULL,
1207 .pre_soft_reset = NULL,
1208 .soft_reset = NULL,1206 .soft_reset = NULL,
1209 .post_soft_reset = NULL,
1210 .set_clockgating_state = vcn_v5_0_2_set_clockgating_state,1207 .set_clockgating_state = vcn_v5_0_2_set_clockgating_state,
1211 .set_powergating_state = vcn_set_powergating_state,1208 .set_powergating_state = vcn_set_powergating_state,
1212};1209};
drivers/gpu/drm/amd/amdgpu/vi.c+0−22
@@ -1328,27 +1328,6 @@ static void vi_invalidate_hdp(struct amdgpu_device *adev, · 1327 unmodified lines
1328 }1328 }
1329}1329}
1330 1330
1331static bool vi_need_full_reset(struct amdgpu_device *adev)
1332{
1333 switch (adev->asic_type) {
1334 case CHIP_CARRIZO:
1335 case CHIP_STONEY:
1336 /* CZ has hang issues with full reset at the moment */
1337 return false;
1338 case CHIP_FIJI:
1339 case CHIP_TONGA:
1340 /* XXX: soft reset should work on fiji and tonga */
1341 return true;
1342 case CHIP_POLARIS10:
1343 case CHIP_POLARIS11:
1344 case CHIP_POLARIS12:
1345 case CHIP_TOPAZ:
1346 default:
1347 /* change this when we support soft reset */
1348 return true;
1349 }
1350}
1351
1352static void vi_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,1331static void vi_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0,
1353 uint64_t *count1)1332 uint64_t *count1)
1354{1333{
@@ -1437,7 +1416,6 @@ static const struct amdgpu_asic_funcs vi_asic_funcs = · 82 unmodified lines
1437 .get_config_memsize = &vi_get_config_memsize,1416 .get_config_memsize = &vi_get_config_memsize,
1438 .flush_hdp = &vi_flush_hdp,1417 .flush_hdp = &vi_flush_hdp,
1439 .invalidate_hdp = &vi_invalidate_hdp,1418 .invalidate_hdp = &vi_invalidate_hdp,
1440 .need_full_reset = &vi_need_full_reset,
1441 .init_doorbell_index = &legacy_doorbell_index_init,1419 .init_doorbell_index = &legacy_doorbell_index_init,
1442 .get_pcie_usage = &vi_get_pcie_usage,1420 .get_pcie_usage = &vi_get_pcie_usage,
1443 .need_reset_on_init = &vi_need_reset_on_init,1421 .need_reset_on_init = &vi_need_reset_on_init,
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c+8−4
@@ -1783,9 +1783,6 @@ static int kfd_ptl_control(struct kfd_process_device *pdd, bool enable) · 1782 unmodified lines
1783 uint32_t ptl_state = enable ? 1 : 0;1783 uint32_t ptl_state = enable ? 1 : 0;
1784 int ret;1784 int ret;
1785 1785
1786 if (!ptl->hw_supported)
1787 return -EOPNOTSUPP;
1788
1789 if (!pdd->dev->kfd2kgd || !pdd->dev->kfd2kgd->ptl_ctrl)1786 if (!pdd->dev->kfd2kgd || !pdd->dev->kfd2kgd->ptl_ctrl)
1790 return -EOPNOTSUPP;1787 return -EOPNOTSUPP;
1791 1788
@@ -1804,6 +1801,9 @@ int kfd_ptl_disable_request(struct kfd_process_device *pdd, · 12 unmodified lines
1804 struct amdgpu_ptl *ptl = &adev->psp.ptl;1801 struct amdgpu_ptl *ptl = &adev->psp.ptl;
1805 int ret = 0;1802 int ret = 0;
1806 1803
1804 if (!ptl->hw_supported)
1805 return -EOPNOTSUPP;
1806
1807 mutex_lock(&ptl->mutex);1807 mutex_lock(&ptl->mutex);
1808 1808
1809 if (pdd->ptl_disable_req)1809 if (pdd->ptl_disable_req)
@@ -1833,6 +1833,9 @@ int kfd_ptl_disable_release(struct kfd_process_device *pdd, · 23 unmodified lines
1833 struct amdgpu_ptl *ptl = &adev->psp.ptl;1833 struct amdgpu_ptl *ptl = &adev->psp.ptl;
1834 int ret = 0;1834 int ret = 0;
1835 1835
1836 if (!ptl->hw_supported)
1837 return -EOPNOTSUPP;
1838
1836 mutex_lock(&ptl->mutex);1839 mutex_lock(&ptl->mutex);
1837 1840
1838 if (!pdd->ptl_disable_req)1841 if (!pdd->ptl_disable_req)
@@ -3734,7 +3737,8 @@ static int kfd_mmap(struct file *filep, struct vm_area_struct *vma) · 1895 unmodified lines
3734 return kfd_doorbell_mmap(dev, process, vma);3737 return kfd_doorbell_mmap(dev, process, vma);
3735 3738
3736 case KFD_MMAP_TYPE_EVENTS:3739 case KFD_MMAP_TYPE_EVENTS:
3737 return kfd_event_mmap(process, vma);3740 pr_warn("KFD_MMAP_TYPE_EVENTS is no longer supported\n");
3741 return -EINVAL;
3738 3742
3739 case KFD_MMAP_TYPE_RESERVED_MEM:3743 case KFD_MMAP_TYPE_RESERVED_MEM:
3740 pr_warn("KFD_MMAP_TYPE_RESERVED_MEM is no longer supported\n");3744 pr_warn("KFD_MMAP_TYPE_RESERVED_MEM is no longer supported\n");
drivers/gpu/drm/amd/amdkfd/kfd_device.c+24−0
@@ -1806,6 +1806,30 @@ void kgd2kfd_teardown_processes(struct amdgpu_device *adev) · 1805 unmodified lines
1806 cond_resched();1806 cond_resched();
1807}1807}
1808 1808
1809int kgd2kfd_reset_mes_queue(struct kfd_dev *kfd, uint32_t node_id,
1810 int queue_type, int pipe, int queue,
1811 unsigned int db)
1812{
1813 struct kfd_node *node;
1814 int ret;
1815
1816 if (!kfd->init_complete)
1817 return 0;
1818
1819 if (node_id >= kfd->num_nodes) {
1820 dev_warn(kfd->adev->dev, "Invalid node ID: %u exceeds %u\n",
1821 node_id, kfd->num_nodes - 1);
1822 return -EINVAL;
1823 }
1824 node = kfd->nodes[node_id];
1825
1826 ret = kfd_reset_queue_mes(node->dqm, queue_type, pipe, queue, db);
1827 if (ret)
1828 dev_err(kfd_device, "Error resetting queue\n");
1829
1830 return ret;
1831}
1832
1809#if defined(CONFIG_DEBUG_FS)1833#if defined(CONFIG_DEBUG_FS)
1810 1834
1811/* This function will send a package to HIQ to hang the HWS1835/* This function will send a package to HIQ to hang the HWS
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c+91−132
@@ -37,7 +37,8 @@ · 36 unmodified lines
37#include "amdgpu_amdkfd.h"37#include "amdgpu_amdkfd.h"
38#include "amdgpu_reset.h"38#include "amdgpu_reset.h"
39#include "amdgpu_sdma.h"39#include "amdgpu_sdma.h"
40#include "mes_v11_api_def.h"40#include "amdgpu_ring.h"
41#include "amdgpu_mes.h"
41#include "kfd_debug.h"42#include "kfd_debug.h"
42 43
43/* Size of the per-pipe EOP queue */44/* Size of the per-pipe EOP queue */
@@ -71,12 +72,11 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 27 unmodified lines
71 struct queue *q, const uint32_t *restore_sdma_id);72 struct queue *q, const uint32_t *restore_sdma_id);
72 73
73static int reset_queues_on_hws_hang(struct device_queue_manager *dqm, bool is_sdma);74static int reset_queues_on_hws_hang(struct device_queue_manager *dqm, bool is_sdma);
74static int resume_all_queues_mes(struct device_queue_manager *dqm);
75static int suspend_all_queues_mes(struct device_queue_manager *dqm);
76static struct queue *find_queue_by_doorbell_offset(struct device_queue_manager *dqm,75static struct queue *find_queue_by_doorbell_offset(struct device_queue_manager *dqm,
77 u32 doorbell_offset);76 u32 doorbell_offset);
78static void set_queue_as_reset(struct device_queue_manager *dqm, struct queue *q,77static void set_queue_as_reset(struct device_queue_manager *dqm, struct queue *q,
79 struct qcm_process_device *qpd);78 struct qcm_process_device *qpd);
79static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q);
80 80
81static inline81static inline
82enum KFD_MQD_TYPE get_mqd_type_from_queue_type(enum kfd_queue_type type)82enum KFD_MQD_TYPE get_mqd_type_from_queue_type(enum kfd_queue_type type)
@@ -184,24 +184,24 @@ static void kfd_hws_hang(struct device_queue_manager *dqm) · 101 unmodified lines
184 amdgpu_amdkfd_gpu_reset(dqm->dev->adev);184 amdgpu_amdkfd_gpu_reset(dqm->dev->adev);
185}185}
186 186
187static int convert_to_mes_queue_type(int queue_type)187static int convert_to_amdgpu_ring_type(int queue_type)
188{188{
189 int mes_queue_type;189 int amdgpu_ring_type;
190 190
191 switch (queue_type) {191 switch (queue_type) {
192 case KFD_QUEUE_TYPE_COMPUTE:192 case KFD_QUEUE_TYPE_COMPUTE:
193 mes_queue_type = MES_QUEUE_TYPE_COMPUTE;193 amdgpu_ring_type = AMDGPU_RING_TYPE_COMPUTE;
194 break;194 break;
195 case KFD_QUEUE_TYPE_SDMA:195 case KFD_QUEUE_TYPE_SDMA:
196 mes_queue_type = MES_QUEUE_TYPE_SDMA;196 amdgpu_ring_type = AMDGPU_RING_TYPE_SDMA;
197 break;197 break;
198 default:198 default:
199 WARN(1, "Invalid queue type %d", queue_type);199 WARN(1, "Invalid queue type %d", queue_type);
200 mes_queue_type = -EINVAL;200 amdgpu_ring_type = -EINVAL;
201 break;201 break;
202 }202 }
203 203
204 return mes_queue_type;204 return amdgpu_ring_type;
205}205}
206 206
207static int add_queue_mes(struct device_queue_manager *dqm, struct queue *q,207static int add_queue_mes(struct device_queue_manager *dqm, struct queue *q,
@@ -251,7 +251,7 @@ static int add_queue_mes(struct device_queue_manager *dqm, struct queue *q, · 43 unmodified lines
251 (qpd->pqm->process->debug_trap_enabled ||251 (qpd->pqm->process->debug_trap_enabled ||
252 kfd_dbg_has_ttmps_always_setup(q->device));252 kfd_dbg_has_ttmps_always_setup(q->device));
253 253
254 queue_type = convert_to_mes_queue_type(q->properties.type);254 queue_type = convert_to_amdgpu_ring_type(q->properties.type);
255 if (queue_type < 0) {255 if (queue_type < 0) {
256 dev_err(adev->dev, "Queue type not supported with MES, queue:%d\n",256 dev_err(adev->dev, "Queue type not supported with MES, queue:%d\n",
257 q->properties.type);257 q->properties.type);
@@ -300,6 +300,7 @@ static int remove_queue_mes_on_reset_option(struct device_queue_manager *dqm, st · 42 unmodified lines
300 memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));300 memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));
301 queue_input.doorbell_offset = q->properties.doorbell_off;301 queue_input.doorbell_offset = q->properties.doorbell_off;
302 queue_input.gang_context_addr = q->gang_ctx_gpu_addr;302 queue_input.gang_context_addr = q->gang_ctx_gpu_addr;
303 queue_input.queue_type = convert_to_amdgpu_ring_type(q->properties.type);
303 queue_input.remove_queue_after_reset = flush_mes_queue;304 queue_input.remove_queue_after_reset = flush_mes_queue;
304 queue_input.xcc_id = ffs(dqm->dev->xcc_mask) - 1;305 queue_input.xcc_id = ffs(dqm->dev->xcc_mask) - 1;
305 306
@@ -308,14 +309,14 @@ static int remove_queue_mes_on_reset_option(struct device_queue_manager *dqm, st · 2 unmodified lines
308 amdgpu_mes_unlock(&adev->mes);309 amdgpu_mes_unlock(&adev->mes);
309 up_read(&adev->reset_domain->sem);310 up_read(&adev->reset_domain->sem);
310 311
311 if (is_for_reset)312 /* If is_for_reset set, it is a mes internal cleanup */
313 if (!r || is_for_reset)
312 return r;314 return r;
313 315
314 if (r) {316 /* remove_hw_queue failure indicates a queue hang. reset the queue */
315 if (!suspend_all_queues_mes(dqm))317 r = reset_queues_mes(dqm, q);
316 return resume_all_queues_mes(dqm);318 if (r && amdgpu_gpu_recovery) {
317 319 dev_err(adev->dev, "failed to remove queue from MES, doorbell=0x%x\n",
318 dev_err(adev->dev, "failed to remove hardware queue from MES, doorbell=0x%x\n",
319 q->properties.doorbell_off);320 q->properties.doorbell_off);
320 dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");321 dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
321 kfd_hws_hang(dqm);322 kfd_hws_hang(dqm);
@@ -407,16 +408,50 @@ static int add_all_kfd_queues_mes(struct device_queue_manager *dqm) · 85 unmodified lines
407 return retval;408 return retval;
408}409}
409 410
410static int reset_queues_mes(struct device_queue_manager *dqm)411static int reset_queue_mes(struct device_queue_manager *dqm, struct queue *q,
412 int queue_type, int pipe, int queue, unsigned int db)
411{413{
412 struct amdgpu_device *adev = (struct amdgpu_device *)dqm->dev->adev;414 struct amdgpu_device *adev = (struct amdgpu_device *)dqm->dev->adev;
413 int hqd_info_size = adev->mes.hung_queue_hqd_info_offset;
414 int num_hung = 0, r = 0, i, pipe, queue, queue_type;
415 u32 *hung_array = dqm->hung_db_array;
416 struct amdgpu_mes_hung_queue_hqd_info *hqd_info = dqm->hqd_info;
417 struct kfd_process_device *pdd;415 struct kfd_process_device *pdd;
416 bool use_mmio = adev->gfx.mec.use_mmio_for_reset;
417 int r;
418
419 pdd = kfd_get_process_device_data(q->device, q->process);
420 if (!pdd)
421 return -ENODEV;
422
423 if (use_mmio)
424 r = amdgpu_mes_reset_queue_mmio(adev, queue_type, 0, 1, pipe, queue,
425 ffs(dqm->dev->xcc_mask) - 1);
426 else
427 r = amdgpu_mes_reset_user_queue(adev, queue_type, db,
428 ffs(dqm->dev->xcc_mask) - 1);
429 if (r)
430 return r;
431 /* Proceed remove_queue with reset=true */
432 remove_queue_mes_on_reset_option(dqm, q, &pdd->qpd, true, true);
433 set_queue_as_reset(dqm, q, &pdd->qpd);
434 return 0;
435}
436
437int kfd_reset_queue_mes(struct device_queue_manager *dqm, int queue_type,
438 int pipe, int queue, unsigned int db)
439{
418 struct queue *q;440 struct queue *q;
419 441
442 q = find_queue_by_doorbell_offset(dqm, db);
443 if (!q)
444 return 0;
445 return reset_queue_mes(dqm, q, queue_type, pipe, queue, db);
446}
447
448static int reset_queues_mes(struct device_queue_manager *dqm, struct queue *q)
449{
450 struct amdgpu_device *adev = (struct amdgpu_device *)dqm->dev->adev;
451 unsigned int num_hung = 0;
452 int r = 0;
453 struct mes_remove_queue_input queue_input;
454
420 if (!amdgpu_mes_queue_reset_by_mes_supported(adev)) {455 if (!amdgpu_mes_queue_reset_by_mes_supported(adev)) {
421 r = -ENOTRECOVERABLE;456 r = -ENOTRECOVERABLE;
422 goto fail;457 goto fail;
@@ -431,111 +466,29 @@ static int reset_queues_mes(struct device_queue_manager *dqm) · 8 unmodified lines
431 goto fail;466 goto fail;
432 }467 }
433 468
434 if (!hung_array || !hqd_info) {469 memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input));
435 r = -ENOMEM;470 queue_input.doorbell_offset = q->properties.doorbell_off;
436 goto fail;471 queue_input.gang_context_addr = q->gang_ctx_gpu_addr;
437 }472 queue_input.queue_type = convert_to_amdgpu_ring_type(q->properties.type);
438 473 queue_input.remove_queue_after_reset = false;
439 memset(hqd_info, 0, hqd_info_size * sizeof(struct amdgpu_mes_hung_queue_hqd_info));474 queue_input.xcc_id = ffs(dqm->dev->xcc_mask) - 1;
440 475 /* pass the known bad queue info to the reset function */
441 /*476 r = amdgpu_gfx_reset_mes_compute(adev, NULL, NULL, NULL, &num_hung, &queue_input);
442 * AMDGPU_RING_TYPE_COMPUTE parameter does not matter if called477 if (r)
443 * post suspend_all as reset & detect will return all hung queue types.
444 *
445 * Passed parameter is for targeting queues not scheduled by MES add_queue.
446 */
447 r = amdgpu_mes_detect_and_reset_hung_queues(adev, AMDGPU_RING_TYPE_COMPUTE,
448 false, &num_hung, hung_array, ffs(dqm->dev->xcc_mask) - 1);
449
450 if (!num_hung || r) {
451 r = -ENOTRECOVERABLE;
452 goto fail;478 goto fail;
453 }
454
455 /* MES resets queue/pipe and cleans up internally */
456 for (i = 0; i < num_hung; i++) {
457 hqd_info[i].bit0_31 = hung_array[i + hqd_info_size];
458 pipe = hqd_info[i].pipe_index;
459 queue = hqd_info[i].queue_index;
460 queue_type = hqd_info[i].queue_type;
461
462 if (queue_type != MES_QUEUE_TYPE_COMPUTE &&
463 queue_type != MES_QUEUE_TYPE_SDMA) {
464 pr_warn("Unsupported hung queue reset type: %d\n", queue_type);
465 hung_array[i] = AMDGPU_MES_INVALID_DB_OFFSET;
466 continue;
467 }
468
469 q = find_queue_by_doorbell_offset(dqm, hung_array[i]);
470 if (!q) {
471 r = -ENOTRECOVERABLE;
472 goto fail;
473 }
474
475 pdd = kfd_get_process_device_data(q->device, q->process);
476 if (!pdd) {
477 r = -ENODEV;
478 goto fail;
479 }
480
481 pr_warn("Hang detected doorbell %x pipe %d queue %d type %d\n",
482 hung_array[i], pipe, queue, queue_type);
483 /* Proceed remove_queue with reset=true */
484 remove_queue_mes_on_reset_option(dqm, q, &pdd->qpd, true, false);
485 set_queue_as_reset(dqm, q, &pdd->qpd);
486 }
487 479
488 dqm->detect_hang_count = num_hung;480 dqm->detect_hang_count = num_hung;
489 kfd_signal_reset_event(dqm->dev);481 /* When MES doesn't detect any queue hang, no reset happens. Don't signal reset
482 * event.
483 */
484 if (dqm->detect_hang_count)
485 kfd_signal_reset_event(dqm->dev);
490 486
491fail:487fail:
492 dqm->detect_hang_count = 0;488 dqm->detect_hang_count = 0;
493 return r;489 return r;
494}490}
495 491
496static int suspend_all_queues_mes(struct device_queue_manager *dqm)
497{
498 struct amdgpu_device *adev = (struct amdgpu_device *)dqm->dev->adev;
499 int r = 0;
500
501 if (!down_read_trylock(&adev->reset_domain->sem))
502 return -EIO;
503
504 r = amdgpu_mes_suspend(adev, ffs(dqm->dev->xcc_mask) - 1);
505 up_read(&adev->reset_domain->sem);
506
507 if (r) {
508 if (!reset_queues_mes(dqm))
509 return 0;
510
511 dev_err(adev->dev, "failed to suspend gangs from MES\n");
512 dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
513 kfd_hws_hang(dqm);
514 }
515
516 return r;
517}
518
519static int resume_all_queues_mes(struct device_queue_manager *dqm)
520{
521 struct amdgpu_device *adev = (struct amdgpu_device *)dqm->dev->adev;
522 int r = 0;
523
524 if (!down_read_trylock(&adev->reset_domain->sem))
525 return -EIO;
526
527 r = amdgpu_mes_resume(adev, ffs(dqm->dev->xcc_mask) - 1);
528 up_read(&adev->reset_domain->sem);
529
530 if (r) {
531 dev_err(adev->dev, "failed to resume gangs from MES\n");
532 dev_err(adev->dev, "MES might be in unrecoverable state, issue a GPU reset\n");
533 kfd_hws_hang(dqm);
534 }
535
536 return r;
537}
538
539static void increment_queue_count(struct device_queue_manager *dqm,492static void increment_queue_count(struct device_queue_manager *dqm,
540 struct qcm_process_device *qpd,493 struct qcm_process_device *qpd,
541 struct queue *q)494 struct queue *q)
@@ -1064,8 +1017,15 @@ static int destroy_queue_nocpsch(struct device_queue_manager *dqm, · 522 unmodified lines
1064 /* Get the SDMA queue stats */1017 /* Get the SDMA queue stats */
1065 if ((q->properties.type == KFD_QUEUE_TYPE_SDMA) ||1018 if ((q->properties.type == KFD_QUEUE_TYPE_SDMA) ||
1066 (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI)) {1019 (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI)) {
1067 retval = read_sdma_queue_counter((uint64_t __user *)q->properties.read_ptr,1020 if (dqm->dev->kfd2kgd->hqd_sdma_get_counter)
1068 &sdma_val);1021 retval = dqm->dev->kfd2kgd->hqd_sdma_get_counter(
1022 dqm->dev->adev, q->mqd,
1023 dqm->dev->kfd->device_info.num_sdma_queues_per_engine,
1024 &sdma_val);
1025 else
1026 retval = read_sdma_queue_counter(
1027 (uint64_t __user *)q->properties.read_ptr,
1028 &sdma_val);
1069 if (retval)1029 if (retval)
1070 dev_err(dev, "Failed to read SDMA queue counter for queue: %d\n",1030 dev_err(dev, "Failed to read SDMA queue counter for queue: %d\n",
1071 q->properties.queue_id);1031 q->properties.queue_id);
@@ -2703,8 +2663,16 @@ static int destroy_queue_cpsch(struct device_queue_manager *dqm, · 1631 unmodified lines
2703 /* Get the SDMA queue stats */2663 /* Get the SDMA queue stats */
2704 if ((q->properties.type == KFD_QUEUE_TYPE_SDMA) ||2664 if ((q->properties.type == KFD_QUEUE_TYPE_SDMA) ||
2705 (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI)) {2665 (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI)) {
2706 retval = read_sdma_queue_counter((uint64_t __user *)q->properties.read_ptr,2666 if (dqm->dev->kfd2kgd->hqd_sdma_get_counter)
2707 &sdma_val);2667 retval = dqm->dev->kfd2kgd->hqd_sdma_get_counter(
2668 dqm->dev->adev, q->mqd,
2669 dqm->dev->kfd->device_info.num_sdma_queues_per_engine,
2670 &sdma_val);
2671 else
2672 retval = read_sdma_queue_counter(
2673 (uint64_t __user *)q->properties.read_ptr,
2674 &sdma_val);
2675
2708 if (retval)2676 if (retval)
2709 dev_err(dev, "Failed to read SDMA queue counter for queue: %d\n",2677 dev_err(dev, "Failed to read SDMA queue counter for queue: %d\n",
2710 q->properties.queue_id);2678 q->properties.queue_id);
@@ -3244,12 +3212,12 @@ void device_queue_manager_uninit(struct device_queue_manager *dqm) · 533 unmodified lines
3244 kfree(dqm);3212 kfree(dqm);
3245}3213}
3246 3214
3215/* bad queue notified by interrupt from CP */
3247int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbell_id)3216int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbell_id)
3248{3217{
3249 struct kfd_process_device *pdd = NULL;3218 struct kfd_process_device *pdd = NULL;
3250 struct kfd_process *p = kfd_lookup_process_by_pasid(pasid, &pdd);3219 struct kfd_process *p = kfd_lookup_process_by_pasid(pasid, &pdd);
3251 struct device_queue_manager *dqm = knode->dqm;3220 struct device_queue_manager *dqm = knode->dqm;
3252 struct device *dev = dqm->dev->adev->dev;
3253 struct qcm_process_device *qpd;3221 struct qcm_process_device *qpd;
3254 struct queue *q = NULL;3222 struct queue *q = NULL;
3255 int ret = 0;3223 int ret = 0;
@@ -3264,19 +3232,10 @@ int kfd_dqm_suspend_bad_queue_mes(struct kfd_node *knode, u32 pasid, u32 doorbel · 8 unmodified lines
3264 3232
3265 list_for_each_entry(q, &qpd->queues_list, list) {3233 list_for_each_entry(q, &qpd->queues_list, list) {
3266 if (q->doorbell_id == doorbell_id && q->properties.is_active) {3234 if (q->doorbell_id == doorbell_id && q->properties.is_active) {
3267 /* suspend all queues will save any good queues and mark the rest as bad */3235 reset_queues_mes(dqm, q);
3268 suspend_all_queues_mes(dqm);
3269
3270 q->properties.is_evicted = true;3236 q->properties.is_evicted = true;
3271 q->properties.is_active = false;3237 q->properties.is_active = false;
3272 decrement_queue_count(dqm, qpd, q);3238 decrement_queue_count(dqm, qpd, q);
3273
3274 /* this will remove the bad queue and sched a GPU reset if needed */
3275 ret = remove_queue_mes(dqm, q, qpd);
3276 if (ret)
3277 dev_err(dev, "Removing bad queue failed");
3278 /* resume the good queues */
3279 resume_all_queues_mes(dqm);
3280 break;3239 break;
3281 }3240 }
3282 }3241 }
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h+2−0
@@ -333,6 +333,8 @@ int debug_refresh_runlist(struct device_queue_manager *dqm); · 332 unmodified lines
333bool kfd_dqm_is_queue_in_process(struct device_queue_manager *dqm,333bool kfd_dqm_is_queue_in_process(struct device_queue_manager *dqm,
334 struct qcm_process_device *qpd,334 struct qcm_process_device *qpd,
335 int doorbell_off, u32 *queue_format);335 int doorbell_off, u32 *queue_format);
336int kfd_reset_queue_mes(struct device_queue_manager *dqm, int queue_type,
337 int pipe, int queue, unsigned int db);
336 338
337static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd)339static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd)
338{340{
drivers/gpu/drm/amd/amdkfd/kfd_events.c+86−138
@@ -29,10 +29,12 @@ · 28 unmodified lines
29#include <linux/uaccess.h>29#include <linux/uaccess.h>
30#include <linux/mman.h>30#include <linux/mman.h>
31#include <linux/memory.h>31#include <linux/memory.h>
32#include <linux/workqueue.h>
32#include "kfd_priv.h"33#include "kfd_priv.h"
33#include "kfd_events.h"34#include "kfd_events.h"
34#include "kfd_device_queue_manager.h"35#include "kfd_device_queue_manager.h"
35#include <linux/device.h>36#include <linux/device.h>
37#include <drm/amdgpu_drm.h>
36 38
37/*39/*
38 * Wrapper around wait_queue_entry_t40 * Wrapper around wait_queue_entry_t
@@ -44,67 +46,19 @@ struct kfd_event_waiter { · 5 unmodified lines
44 bool event_age_enabled; /* set to true when last_event_age is non-zero */46 bool event_age_enabled; /* set to true when last_event_age is non-zero */
45};47};
46 48
47/*
48 * Each signal event needs a 64-bit signal slot where the signaler will write
49 * a 1 before sending an interrupt. (This is needed because some interrupts
50 * do not contain enough spare data bits to identify an event.)
51 * We get whole pages and map them to the process VA.
52 * Individual signal events use their event_id as slot index.
53 */
54struct kfd_signal_page {
55 uint64_t *kernel_address;
56 uint64_t __user *user_address;
57 bool need_to_free_pages;
58};
59
60static uint64_t *page_slots(struct kfd_signal_page *page)
61{
62 return page->kernel_address;
63}
64
65static struct kfd_signal_page *allocate_signal_page(struct kfd_process *p)
66{
67 void *backing_store;
68 struct kfd_signal_page *page;
69
70 page = kzalloc_obj(*page);
71 if (!page)
72 return NULL;
73
74 backing_store = (void *) __get_free_pages(GFP_KERNEL,
75 get_order(KFD_SIGNAL_EVENT_LIMIT * 8));
76 if (!backing_store)
77 goto fail_alloc_signal_store;
78
79 /* Initialize all events to unsignaled */
80 memset(backing_store, (uint8_t) UNSIGNALED_EVENT_SLOT,
81 KFD_SIGNAL_EVENT_LIMIT * 8);
82
83 page->kernel_address = backing_store;
84 page->need_to_free_pages = true;
85 pr_debug("Allocated new event signal page at %p, for process %p\n",
86 page, p);
87
88 return page;
89
90fail_alloc_signal_store:
91 kfree(page);
92 return NULL;
93}
94
95static int allocate_event_notification_slot(struct kfd_process *p,49static int allocate_event_notification_slot(struct kfd_process *p,
96 struct kfd_event *ev,50 struct kfd_event *ev,
97 const int *restore_id)51 const int *restore_id)
98{52{
99 int id;53 int id;
100 54
101 if (!p->signal_page) {55 /*
102 p->signal_page = allocate_signal_page(p);56 * The signal page is allocated in user mode and mapped to the kernel
103 if (!p->signal_page)57 * via the event_page_offset of the create event IOCTL. Without it no
104 return -ENOMEM;58 * signal events can be created.
105 /* Oldest user mode expects 256 event slots */59 */
106 p->signal_mapped_size = 256*8;60 if (!p->signal_page)
107 }61 return -ENOMEM;
108 62
109 if (restore_id) {63 if (restore_id) {
110 id = idr_alloc(&p->event_idr, ev, *restore_id, *restore_id + 1,64 id = idr_alloc(&p->event_idr, ev, *restore_id, *restore_id + 1,
@@ -123,7 +77,7 @@ static int allocate_event_notification_slot(struct kfd_process *p, · 12 unmodified lines
123 return id;77 return id;
124 78
125 ev->event_id = id;79 ev->event_id = id;
126 page_slots(p->signal_page)[id] = UNSIGNALED_EVENT_SLOT;80 p->signal_page[id] = UNSIGNALED_EVENT_SLOT;
127 81
128 return 0;82 return 0;
129}83}
@@ -169,7 +123,7 @@ static struct kfd_event *lookup_signaled_event_by_partial_id( · 39 unmodified lines
169 */123 */
170 if (bits > 31 || (1U << bits) >= KFD_SIGNAL_EVENT_LIMIT) {124 if (bits > 31 || (1U << bits) >= KFD_SIGNAL_EVENT_LIMIT) {
171 if (signal_mailbox_updated &&125 if (signal_mailbox_updated &&
172 page_slots(p->signal_page)[id] == UNSIGNALED_EVENT_SLOT)126 p->signal_page[id] == UNSIGNALED_EVENT_SLOT)
173 return NULL;127 return NULL;
174 128
175 return idr_find(&p->event_idr, id);129 return idr_find(&p->event_idr, id);
@@ -179,7 +133,7 @@ static struct kfd_event *lookup_signaled_event_by_partial_id( · 3 unmodified lines
179 * and find the first one that has signaled.133 * and find the first one that has signaled.
180 */134 */
181 for (ev = NULL; id < KFD_SIGNAL_EVENT_LIMIT && !ev; id += 1U << bits) {135 for (ev = NULL; id < KFD_SIGNAL_EVENT_LIMIT && !ev; id += 1U << bits) {
182 if (page_slots(p->signal_page)[id] == UNSIGNALED_EVENT_SLOT)136 if (p->signal_page[id] == UNSIGNALED_EVENT_SLOT)
183 continue;137 continue;
184 138
185 ev = idr_find(&p->event_idr, id);139 ev = idr_find(&p->event_idr, id);
@@ -210,10 +164,8 @@ static int create_signal_event(struct file *devkfd, struct kfd_process *p, · 24 unmodified lines
210 164
211 p->signal_event_count++;165 p->signal_event_count++;
212 166
213 ev->user_signal_address = &p->signal_page->user_address[ev->event_id];167 pr_debug("Signal event number %zu created with id %d\n",
214 pr_debug("Signal event number %zu created with id %d, address %p\n",168 p->signal_event_count, ev->event_id);
215 p->signal_event_count, ev->event_id,
216 ev->user_signal_address);
217 169
218 return 0;170 return 0;
219}171}
@@ -293,26 +245,9 @@ static void destroy_events(struct kfd_process *p) · 73 unmodified lines
293 mutex_destroy(&p->event_mutex);245 mutex_destroy(&p->event_mutex);
294}246}
295 247
296/*
297 * We assume that the process is being destroyed and there is no need to
298 * unmap the pages or keep bookkeeping data in order.
299 */
300static void shutdown_signal_page(struct kfd_process *p)
301{
302 struct kfd_signal_page *page = p->signal_page;
303
304 if (page) {
305 if (page->need_to_free_pages)
306 free_pages((unsigned long)page->kernel_address,
307 get_order(KFD_SIGNAL_EVENT_LIMIT * 8));
308 kfree(page);
309 }
310}
311
312void kfd_event_free_process(struct kfd_process *p)248void kfd_event_free_process(struct kfd_process *p)
313{249{
314 destroy_events(p);250 destroy_events(p);
315 shutdown_signal_page(p);
316}251}
317 252
318static bool event_can_be_gpu_signaled(const struct kfd_event *ev)253static bool event_can_be_gpu_signaled(const struct kfd_event *ev)
@@ -329,8 +264,6 @@ static bool event_can_be_cpu_signaled(const struct kfd_event *ev) · 10 unmodified lines
329static int kfd_event_page_set(struct kfd_process *p, void *kernel_address,264static int kfd_event_page_set(struct kfd_process *p, void *kernel_address,
330 uint64_t size, uint64_t user_handle)265 uint64_t size, uint64_t user_handle)
331{266{
332 struct kfd_signal_page *page;
333
334 if (p->signal_page)267 if (p->signal_page)
335 return -EBUSY;268 return -EBUSY;
336 269
@@ -340,17 +273,11 @@ static int kfd_event_page_set(struct kfd_process *p, void *kernel_address, · 3 unmodified lines
340 return -EINVAL;273 return -EINVAL;
341 }274 }
342 275
343 page = kzalloc_obj(*page);
344 if (!page)
345 return -ENOMEM;
346
347 /* Initialize all events to unsignaled */276 /* Initialize all events to unsignaled */
348 memset(kernel_address, (uint8_t) UNSIGNALED_EVENT_SLOT,277 memset(kernel_address, (uint8_t) UNSIGNALED_EVENT_SLOT,
349 KFD_SIGNAL_EVENT_LIMIT * 8);278 KFD_SIGNAL_EVENT_LIMIT * 8);
350 279
351 page->kernel_address = kernel_address;280 p->signal_page = kernel_address;
352
353 p->signal_page = page;
354 p->signal_mapped_size = size;281 p->signal_mapped_size = size;
355 p->signal_handle = user_handle;282 p->signal_handle = user_handle;
356 return 0;283 return 0;
@@ -387,7 +314,8 @@ int kfd_kmap_event_page(struct kfd_process *p, uint64_t event_page_offset) · 30 unmodified lines
387 return -EINVAL;314 return -EINVAL;
388 }315 }
389 316
390 err = amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(mem, &kern_addr, &size);317 err = amdgpu_amdkfd_gpuvm_map_bo_to_kernel(mem, &kern_addr, &size,
318 AMDGPU_GEM_DOMAIN_GTT);
391 if (err) {319 if (err) {
392 pr_err("Failed to map event page to kernel\n");320 pr_err("Failed to map event page to kernel\n");
393 return err;321 return err;
@@ -396,7 +324,7 @@ int kfd_kmap_event_page(struct kfd_process *p, uint64_t event_page_offset) · 2 unmodified lines
396 err = kfd_event_page_set(p, kern_addr, size, event_page_offset);324 err = kfd_event_page_set(p, kern_addr, size, event_page_offset);
397 if (err) {325 if (err) {
398 pr_err("Failed to set event page\n");326 pr_err("Failed to set event page\n");
399 amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(mem);327 amdgpu_amdkfd_gpuvm_unmap_bo_from_kernel(mem);
400 return err;328 return err;
401 }329 }
402 return err;330 return err;
@@ -717,7 +645,7 @@ unlock_rcu: · 314 unmodified lines
717 645
718static void acknowledge_signal(struct kfd_process *p, struct kfd_event *ev)646static void acknowledge_signal(struct kfd_process *p, struct kfd_event *ev)
719{647{
720 WRITE_ONCE(page_slots(p->signal_page)[ev->event_id], UNSIGNALED_EVENT_SLOT);648 WRITE_ONCE(p->signal_page[ev->event_id], UNSIGNALED_EVENT_SLOT);
721}649}
722 650
723static void set_event_from_interrupt(struct kfd_process *p,651static void set_event_from_interrupt(struct kfd_process *p,
@@ -760,7 +688,7 @@ void kfd_signal_event_interrupt(u32 pasid, uint32_t partial_id, · 36 unmodified lines
760 * in the interrupt payload was invalid and do an688 * in the interrupt payload was invalid and do an
761 * exhaustive search of signaled events.689 * exhaustive search of signaled events.
762 */690 */
763 uint64_t *slots = page_slots(p->signal_page);691 uint64_t *slots = p->signal_page;
764 uint32_t id;692 uint32_t id;
765 693
766 if (valid_id_bits)694 if (valid_id_bits)
@@ -1068,51 +996,6 @@ out: · 301 unmodified lines
1068 return ret;996 return ret;
1069}997}
1070 998
1071int kfd_event_mmap(struct kfd_process *p, struct vm_area_struct *vma)
1072{
1073 unsigned long pfn;
1074 struct kfd_signal_page *page;
1075 int ret;
1076
1077 /* check required size doesn't exceed the allocated size */
1078 if (get_order(KFD_SIGNAL_EVENT_LIMIT * 8) <
1079 get_order(vma->vm_end - vma->vm_start)) {
1080 pr_err("Event page mmap requested illegal size\n");
1081 return -EINVAL;
1082 }
1083
1084 page = p->signal_page;
1085 if (!page) {
1086 /* Probably KFD bug, but mmap is user-accessible. */
1087 pr_debug("Signal page could not be found\n");
1088 return -EINVAL;
1089 }
1090
1091 pfn = __pa(page->kernel_address);
1092 pfn >>= PAGE_SHIFT;
1093
1094 vm_flags_set(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE
1095 | VM_DONTDUMP | VM_PFNMAP);
1096
1097 pr_debug("Mapping signal page\n");
1098 pr_debug(" start user address == 0x%08lx\n", vma->vm_start);
1099 pr_debug(" end user address == 0x%08lx\n", vma->vm_end);
1100 pr_debug(" pfn == 0x%016lX\n", pfn);
1101 pr_debug(" vm_flags == 0x%08lX\n", vma->vm_flags);
1102 pr_debug(" size == 0x%08lX\n",
1103 vma->vm_end - vma->vm_start);
1104
1105 page->user_address = (uint64_t __user *)vma->vm_start;
1106
1107 /* mapping the page to user process */
1108 ret = remap_pfn_range(vma, vma->vm_start, pfn,
1109 vma->vm_end - vma->vm_start, vma->vm_page_prot);
1110 if (!ret)
1111 p->signal_mapped_size = vma->vm_end - vma->vm_start;
1112
1113 return ret;
1114}
1115
1116/*999/*
1117 * Assumes that p is not going away.1000 * Assumes that p is not going away.
1118 */1001 */
@@ -1338,6 +1221,71 @@ void kfd_signal_reset_event(struct kfd_node *dev) · 219 unmodified lines
1338 srcu_read_unlock(&kfd_processes_srcu, idx);1221 srcu_read_unlock(&kfd_processes_srcu, idx);
1339}1222}
1340 1223
1224/*
1225 * Per-process opt-in for poison-consumption SIGBUS handling.
1226 *
1227 * Default: kernel sends SIGBUS to the process immediately when poison is
1228 * consumed, in addition to delivering the KFD HW/MEMORY exception events.
1229 *
1230 * Userspace (ROCr) can opt-in per-process via the
1231 * DRM_IOCTL_AMDGPU_PROC_OPTIONS / AMDGPU_PROC_OPTIONS_OP_KFD_SIGBUS_DELAY
1232 * option. This lets the app's registered system-event callback handle the
1233 * RAS error first, instead of being killed by SIGBUS.
1234 *
1235 * Encoded value (stored on the kfd_process):
1236 * 0 - default: SIGBUS immediately (no opt-in)
1237 * 0xFFFFFFFF - opt-in, never escalate to SIGBUS
1238 * N (other) - opt-in, escalate to SIGBUS after N ms if app does not
1239 * handle the error in time (safety timeout)
1240 */
1241
1242void kfd_signal_sigbus_delayed_fn(struct work_struct *work)
1243{
1244 struct kfd_process *p = container_of(to_delayed_work(work),
1245 struct kfd_process, signal_work);
1246
1247 if (p->lead_thread)
1248 send_sig(SIGBUS, p->lead_thread, 0);
1249
1250 kfd_unref_process(p);
1251}
1252
1253static void kfd_signal_sigbus_with_delay(struct kfd_node *dev,
1254 struct kfd_process *p)
1255{
1256 u32 delay_ms = atomic_read(&p->kfd_sigbus_delay_ms);
1257
1258 if (delay_ms == AMDGPU_PROC_OPTIONS_KFD_SIGBUS_DELAY_DISABLED) {
1259 dev_info(dev->adev->dev,
1260 "SIGBUS suppressed for process %s(pid:%d): app opted in to handle RAS error\n",
1261 p->lead_thread->comm, p->lead_thread->pid);
1262 return;
1263 }
1264
1265 if (delay_ms == 0)
1266 goto send_now;
1267
1268 /*
1269 * Take an extra reference for the delayed worker. If the work is
1270 * already pending (e.g. another device of this process consumed poison
1271 * just before), drop the reference and skip rescheduling - the process
1272 * only needs to be notified once.
1273 */
1274 kref_get(&p->ref);
1275 if (!schedule_delayed_work(&p->signal_work, msecs_to_jiffies(delay_ms))) {
1276 kfd_unref_process(p);
1277 return;
1278 }
1279
1280 dev_info(dev->adev->dev,
1281 "Deferring SIGBUS to process %s(pid:%d) by %u ms (RAS error opt-in safety timeout)\n",
1282 p->lead_thread->comm, p->lead_thread->pid, delay_ms);
1283 return;
1284
1285send_now:
1286 send_sig(SIGBUS, p->lead_thread, 0);
1287}
1288
1341void kfd_signal_poison_consumed_event(struct kfd_node *dev, u32 pasid)1289void kfd_signal_poison_consumed_event(struct kfd_node *dev, u32 pasid)
1342{1290{
1343 struct kfd_process *p = kfd_lookup_process_by_pasid(pasid, NULL);1291 struct kfd_process *p = kfd_lookup_process_by_pasid(pasid, NULL);
@@ -1392,7 +1340,7 @@ void kfd_signal_poison_consumed_event(struct kfd_node *dev, u32 pasid) · 48 unmodified lines
1392 rcu_read_unlock();1340 rcu_read_unlock();
1393 1341
1394 /* user application will handle SIGBUS signal */1342 /* user application will handle SIGBUS signal */
1395 send_sig(SIGBUS, p->lead_thread, 0);1343 kfd_signal_sigbus_with_delay(dev, p);
1396 1344
1397 kfd_unref_process(p);1345 kfd_unref_process(p);
1398}1346}
drivers/gpu/drm/amd/amdkfd/kfd_events.h+0−4
@@ -49,7 +49,6 @@ · 48 unmodified lines
49#define UNSIGNALED_EVENT_SLOT ((uint64_t)-1)49#define UNSIGNALED_EVENT_SLOT ((uint64_t)-1)
50 50
51struct kfd_event_waiter;51struct kfd_event_waiter;
52struct signal_page;
53 52
54struct kfd_event {53struct kfd_event {
55 u32 event_id;54 u32 event_id;
@@ -63,9 +62,6 @@ struct kfd_event { · 7 unmodified lines
63 spinlock_t lock;62 spinlock_t lock;
64 wait_queue_head_t wq; /* List of event waiters. */63 wait_queue_head_t wq; /* List of event waiters. */
65 64
66 /* Only for signal events. */
67 uint64_t __user *user_signal_address;
68
69 /* type specific data */65 /* type specific data */
70 union {66 union {
71 struct kfd_hsa_memory_exception_data memory_exception_data;67 struct kfd_hsa_memory_exception_data memory_exception_data;
drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.c+16−0
@@ -315,3 +315,19 @@ bool kfd_check_hiq_mqd_doorbell_id(struct kfd_node *node, uint32_t doorbell_id, · 314 unmodified lines
315 315
316 return false;316 return false;
317}317}
318
319bool mqd_on_vram(struct amdgpu_device *adev)
320{
321 if (adev->apu_prefer_gtt)
322 return false;
323
324 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
325 case IP_VERSION(9, 4, 2):
326 case IP_VERSION(9, 4, 3):
327 case IP_VERSION(9, 4, 4):
328 case IP_VERSION(9, 5, 0):
329 return true;
330 default:
331 return false;
332 }
333}
drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager.h+2−0
@@ -202,4 +202,6 @@ uint64_t kfd_mqd_stride(struct mqd_manager *mm, · 201 unmodified lines
202 struct queue_properties *q);202 struct queue_properties *q);
203bool kfd_check_hiq_mqd_doorbell_id(struct kfd_node *node, uint32_t doorbell_id,203bool kfd_check_hiq_mqd_doorbell_id(struct kfd_node *node, uint32_t doorbell_id,
204 uint32_t inst);204 uint32_t inst);
205bool mqd_on_vram(struct amdgpu_device *adev);
206
205#endif /* KFD_MQD_MANAGER_H_ */207#endif /* KFD_MQD_MANAGER_H_ */
drivers/gpu/drm/amd/amdkfd/kfd_mqd_manager_v9.c+0−14
@@ -116,20 +116,6 @@ static void set_priority(struct v9_mqd *m, struct queue_properties *q) · 115 unmodified lines
116 m->cp_hqd_pipe_priority = pipe_priority_map[q->priority];116 m->cp_hqd_pipe_priority = pipe_priority_map[q->priority];
117}117}
118 118
119static bool mqd_on_vram(struct amdgpu_device *adev)
120{
121 if (adev->apu_prefer_gtt)
122 return false;
123
124 switch (amdgpu_ip_version(adev, GC_HWIP, 0)) {
125 case IP_VERSION(9, 4, 3):
126 case IP_VERSION(9, 5, 0):
127 return true;
128 default:
129 return false;
130 }
131}
132
133static struct kfd_mem_obj *allocate_mqd(struct mqd_manager *mm,119static struct kfd_mem_obj *allocate_mqd(struct mqd_manager *mm,
134 struct queue_properties *q)120 struct queue_properties *q)
135{121{
drivers/gpu/drm/amd/amdkfd/kfd_priv.h+23−2
@@ -952,11 +952,32 @@ struct kfd_process { · 951 unmodified lines
952 struct idr event_idr;952 struct idr event_idr;
953 /* Event page */953 /* Event page */
954 u64 signal_handle;954 u64 signal_handle;
955 struct kfd_signal_page *signal_page;955 /*
956 * Each signal event needs a 64-bit signal slot where the signaler will
957 * write a 1 before sending an interrupt. (This is needed because some
958 * interrupts do not contain enough spare data bits to identify an
959 * event.) The signal page is allocated in user mode and mapped to the
960 * kernel; individual signal events use their event_id as slot index.
961 */
962 uint64_t *signal_page;
956 size_t signal_mapped_size;963 size_t signal_mapped_size;
957 size_t signal_event_count;964 size_t signal_event_count;
958 bool signal_event_limit_reached;965 bool signal_event_limit_reached;
959 966
967 /**
968 * @kfd_sigbus_delay_ms: Per-process KFD SIGBUS delivery option for
969 * poison/RAS events (set via DRM_IOCTL_AMDGPU_PROC_OPTIONS /
970 * AMDGPU_PROC_OPTIONS_OP_KFD_SIGBUS_DELAY).
971 *
972 * 0 - send SIGBUS immediately (default)
973 * 0xFFFFFFFF - suppress SIGBUS delivery
974 * other - delay SIGBUS delivery by this many milliseconds
975 */
976 atomic_t kfd_sigbus_delay_ms;
977
978 /* Delayed signal delivery to user */
979 struct delayed_work signal_work;
980
960 /* Information used for memory eviction */981 /* Information used for memory eviction */
961 void *kgd_process_info;982 void *kgd_process_info;
962 /* Eviction fence that is attached to all the BOs of this process. The983 /* Eviction fence that is attached to all the BOs of this process. The
@@ -1525,7 +1546,6 @@ extern const struct kfd_device_global_init_class device_global_init_class_cik; · 562 unmodified lines
1525 1546
1526int kfd_event_init_process(struct kfd_process *p);1547int kfd_event_init_process(struct kfd_process *p);
1527void kfd_event_free_process(struct kfd_process *p);1548void kfd_event_free_process(struct kfd_process *p);
1528int kfd_event_mmap(struct kfd_process *process, struct vm_area_struct *vma);
1529int kfd_wait_on_events(struct kfd_process *p,1549int kfd_wait_on_events(struct kfd_process *p,
1530 uint32_t num_events, void __user *data,1550 uint32_t num_events, void __user *data,
1531 bool all, uint32_t *user_timeout_ms,1551 bool all, uint32_t *user_timeout_ms,
@@ -1554,6 +1574,7 @@ void kfd_signal_vm_fault_event(struct kfd_process_device *pdd, · 22 unmodified lines
1554void kfd_signal_reset_event(struct kfd_node *dev);1574void kfd_signal_reset_event(struct kfd_node *dev);
1555 1575
1556void kfd_signal_poison_consumed_event(struct kfd_node *dev, u32 pasid);1576void kfd_signal_poison_consumed_event(struct kfd_node *dev, u32 pasid);
1577void kfd_signal_sigbus_delayed_fn(struct work_struct *work);
1557void kfd_signal_process_terminate_event(struct kfd_process *p);1578void kfd_signal_process_terminate_event(struct kfd_process *p);
1558 1579
1559static inline void kfd_flush_tlb(struct kfd_process_device *pdd)1580static inline void kfd_flush_tlb(struct kfd_process_device *pdd)
drivers/gpu/drm/amd/amdkfd/kfd_process.c+67−9
@@ -153,6 +153,21 @@ static void kfd_sdma_activity_worker(struct work_struct *work) · 152 unmodified lines
153 (q->properties.type != KFD_QUEUE_TYPE_SDMA_XGMI))153 (q->properties.type != KFD_QUEUE_TYPE_SDMA_XGMI))
154 continue;154 continue;
155 155
156 if (dqm->dev->kfd2kgd->hqd_sdma_get_counter) {
157 val = 0;
158 ret = dqm->dev->kfd2kgd->hqd_sdma_get_counter(
159 dqm->dev->adev, q->mqd,
160 dqm->dev->kfd->device_info.num_sdma_queues_per_engine,
161 &val);
162
163 if (ret)
164 pr_debug("Failed to read SDMA queue active counter %i\n", ret);
165 else
166 workarea->sdma_activity_counter += val;
167
168 continue;
169 }
170
156 sdma_q = kzalloc_obj(struct temp_sdma_queue_list);171 sdma_q = kzalloc_obj(struct temp_sdma_queue_list);
157 if (!sdma_q) {172 if (!sdma_q) {
158 dqm_unlock(dqm);173 dqm_unlock(dqm);
@@ -171,7 +186,7 @@ static void kfd_sdma_activity_worker(struct work_struct *work) · 12 unmodified lines
171 * count186 * count
172 */187 */
173 if (list_empty(&sdma_q_list.list)) {188 if (list_empty(&sdma_q_list.list)) {
174 workarea->sdma_activity_counter = pdd->sdma_past_activity_counter;189 workarea->sdma_activity_counter += pdd->sdma_past_activity_counter;
175 dqm_unlock(dqm);190 dqm_unlock(dqm);
176 return;191 return;
177 }192 }
@@ -721,7 +736,7 @@ static void kfd_process_free_gpuvm(struct kgd_mem *mem, · 543 unmodified lines
721 struct kfd_node *dev = pdd->dev;736 struct kfd_node *dev = pdd->dev;
722 737
723 if (kptr && *kptr) {738 if (kptr && *kptr) {
724 amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(mem);739 amdgpu_amdkfd_gpuvm_unmap_bo_from_kernel(mem);
725 *kptr = NULL;740 *kptr = NULL;
726 }741 }
727 742
@@ -761,10 +776,16 @@ static int kfd_process_alloc_gpuvm(struct kfd_process_device *pdd, · 33 unmodified lines
761 }776 }
762 777
763 if (kptr) {778 if (kptr) {
764 err = amdgpu_amdkfd_gpuvm_map_gtt_bo_to_kernel(779 u32 domain;
765 (struct kgd_mem *)*mem, kptr, NULL);780
781 if (flags & KFD_IOC_ALLOC_MEM_FLAGS_VRAM)
782 domain = AMDGPU_GEM_DOMAIN_VRAM;
783 else
784 domain = AMDGPU_GEM_DOMAIN_GTT;
785 err = amdgpu_amdkfd_gpuvm_map_bo_to_kernel((struct kgd_mem *)*mem,
786 kptr, NULL, domain);
766 if (err) {787 if (err) {
767 pr_debug("Map GTT BO to kernel failed\n");788 pr_debug("Map BO to kernel failed err %d\n", err);
768 goto sync_memory_failed;789 goto sync_memory_failed;
769 }790 }
770 }791 }
@@ -986,6 +1007,33 @@ out: · 215 unmodified lines
986 return process;1007 return process;
987}1008}
988 1009
1010/**
1011 * amdgpu_amdkfd_set_sigbus_delay - Set per-process KFD SIGBUS delay
1012 * @task: task in the target process
1013 * @ms: encoded delay value (0 = immediate, 0xFFFFFFFF = suppress,
1014 * otherwise delay in milliseconds)
1015 *
1016 * Stores the SIGBUS delivery option on the kfd_process associated with
1017 * @task. If the calling process has not opened /dev/kfd yet (no
1018 * kfd_process exists), this is a no-op - the option only applies to
1019 * processes that actually use KFD.
1020 */
1021int amdgpu_amdkfd_set_sigbus_delay(struct task_struct *task, u32 ms)
1022{
1023 struct kfd_process *p;
1024
1025 if (!task->mm)
1026 return -EINVAL;
1027
1028 p = kfd_lookup_process_by_mm(task->mm);
1029 if (!p)
1030 return 0;
1031
1032 atomic_set(&p->kfd_sigbus_delay_ms, ms);
1033 kfd_unref_process(p);
1034 return 0;
1035}
1036
989static struct kfd_process *find_process_by_mm(const struct mm_struct *mm)1037static struct kfd_process *find_process_by_mm(const struct mm_struct *mm)
990{1038{
991 struct kfd_process *process;1039 struct kfd_process *process;
@@ -1092,7 +1140,7 @@ static void kfd_process_kunmap_signal_bo(struct kfd_process *p) · 100 unmodified lines
1092 if (!mem)1140 if (!mem)
1093 goto out;1141 goto out;
1094 1142
1095 amdgpu_amdkfd_gpuvm_unmap_gtt_bo_from_kernel(mem);1143 amdgpu_amdkfd_gpuvm_unmap_bo_from_kernel(mem);
1096 1144
1097out:1145out:
1098 mutex_unlock(&p->mutex);1146 mutex_unlock(&p->mutex);
@@ -1330,6 +1378,11 @@ void kfd_process_notifier_release_internal(struct kfd_process *p) · 231 unmodified lines
1330 kfd_process_table_remove(p);1378 kfd_process_table_remove(p);
1331 cancel_delayed_work_sync(&p->eviction_work);1379 cancel_delayed_work_sync(&p->eviction_work);
1332 cancel_delayed_work_sync(&p->restore_work);1380 cancel_delayed_work_sync(&p->restore_work);
1381 /*
1382 * If work pending, cancel it and drop the extra ref
1383 */
1384 if (cancel_delayed_work_sync(&p->signal_work))
1385 kfd_unref_process(p);
1333 1386
1334 /*1387 /*
1335 * Dequeue and destroy user queues, it is not safe for GPU to access1388 * Dequeue and destroy user queues, it is not safe for GPU to access
@@ -1436,8 +1489,7 @@ static int kfd_process_device_init_cwsr_dgpu(struct kfd_process_device *pdd) · 100 unmodified lines
1436{1489{
1437 struct kfd_node *dev = pdd->dev;1490 struct kfd_node *dev = pdd->dev;
1438 struct qcm_process_device *qpd = &pdd->qpd;1491 struct qcm_process_device *qpd = &pdd->qpd;
1439 uint32_t flags = KFD_IOC_ALLOC_MEM_FLAGS_GTT1492 u32 flags = KFD_IOC_ALLOC_MEM_FLAGS_NO_SUBSTITUTE
1440 | KFD_IOC_ALLOC_MEM_FLAGS_NO_SUBSTITUTE
1441 | KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE;1493 | KFD_IOC_ALLOC_MEM_FLAGS_EXECUTABLE;
1442 struct kgd_mem *mem;1494 struct kgd_mem *mem;
1443 void *kaddr;1495 void *kaddr;
@@ -1446,7 +1498,12 @@ static int kfd_process_device_init_cwsr_dgpu(struct kfd_process_device *pdd) · 2 unmodified lines
1446 if (!dev->kfd->cwsr_enabled || qpd->cwsr_kaddr || !qpd->cwsr_base)1498 if (!dev->kfd->cwsr_enabled || qpd->cwsr_kaddr || !qpd->cwsr_base)
1447 return 0;1499 return 0;
1448 1500
1449 /* cwsr_base is only set for dGPU */1501 if (KFD_GC_VERSION(dev) >= IP_VERSION(9, 4, 2) && !dev->adev->apu_prefer_gtt)
1502 flags |= KFD_IOC_ALLOC_MEM_FLAGS_VRAM;
1503 else
1504 flags |= KFD_IOC_ALLOC_MEM_FLAGS_GTT;
1505
1506 /* Allocate CWSR TBA/TMA buffers */
1450 ret = kfd_process_alloc_gpuvm(pdd, qpd->cwsr_base,1507 ret = kfd_process_alloc_gpuvm(pdd, qpd->cwsr_base,
1451 KFD_CWSR_TBA_TMA_SIZE, flags, &mem, &kaddr);1508 KFD_CWSR_TBA_TMA_SIZE, flags, &mem, &kaddr);
1452 if (ret)1509 if (ret)
@@ -1586,6 +1643,7 @@ struct kfd_process *create_process(const struct task_struct *thread, bool primar · 133 unmodified lines
1586 1643
1587 INIT_DELAYED_WORK(&process->eviction_work, evict_process_worker);1644 INIT_DELAYED_WORK(&process->eviction_work, evict_process_worker);
1588 INIT_DELAYED_WORK(&process->restore_work, restore_process_worker);1645 INIT_DELAYED_WORK(&process->restore_work, restore_process_worker);
1646 INIT_DELAYED_WORK(&process->signal_work, kfd_signal_sigbus_delayed_fn);
1589 process->last_restore_timestamp = get_jiffies_64();1647 process->last_restore_timestamp = get_jiffies_64();
1590 err = kfd_event_init_process(process);1648 err = kfd_event_init_process(process);
1591 if (err)1649 if (err)
drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c+7−1
@@ -1040,7 +1040,13 @@ int kfd_criu_restore_queue(struct kfd_process *p, · 1039 unmodified lines
1040 ctl_stack = mqd + q_data->mqd_size;1040 ctl_stack = mqd + q_data->mqd_size;
1041 1041
1042 memset(&qp, 0, sizeof(qp));1042 memset(&qp, 0, sizeof(qp));
1043 set_queue_properties_from_criu(&qp, q_data, NUM_XCC(pdd->dev->adev->gfx.xcc_mask));1043 set_queue_properties_from_criu(&qp, q_data, NUM_XCC(pdd->dev->xcc_mask));
1044
1045 ret = kfd_queue_acquire_buffers(pdd, &qp);
1046 if (ret) {
1047 pr_debug("failed to acquire user queue buffers for CRIU\n");
1048 goto exit;
1049 }
1044 1050
1045 ret = kfd_queue_acquire_buffers(pdd, &qp);1051 ret = kfd_queue_acquire_buffers(pdd, &qp);
1046 if (ret) {1052 if (ret) {
drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c+2−2
@@ -224,10 +224,10 @@ static void kfd_smi_event_add(struct task_struct *task, struct kfd_node *dev, · 223 unmodified lines
224 224
225 pid = kfd_smi_task_to_pid(task);225 pid = kfd_smi_task_to_pid(task);
226 226
227 len = snprintf(fifo_in, sizeof(fifo_in), "%x ", event);227 len = scnprintf(fifo_in, sizeof(fifo_in), "%x ", event);
228 228
229 va_start(args, fmt);229 va_start(args, fmt);
230 len += vsnprintf(fifo_in + len, sizeof(fifo_in) - len, fmt, args);230 len += vscnprintf(fifo_in + len, sizeof(fifo_in) - len, fmt, args);
231 va_end(args);231 va_end(args);
232 232
233 add_event_to_kfifo(pid, dev, event, fifo_in, len);233 add_event_to_kfifo(pid, dev, event, fifo_in, len);
drivers/gpu/drm/amd/amdkfd/kfd_svm.c+7−1
@@ -3473,7 +3473,13 @@ svm_range_is_valid(struct kfd_process *p, uint64_t start, uint64_t size) · 3472 unmodified lines
3473 unsigned long start_unchg = start;3473 unsigned long start_unchg = start;
3474 3474
3475 start <<= PAGE_SHIFT;3475 start <<= PAGE_SHIFT;
3476 end = start + (size << PAGE_SHIFT);3476
3477 if (size == 0)
3478 return -EINVAL;
3479
3480 if (check_add_overflow(start, size << PAGE_SHIFT, &end))
3481 return -EOVERFLOW;
3482
3477 do {3483 do {
3478 vma = vma_lookup(p->mm, start);3484 vma = vma_lookup(p->mm, start);
3479 if (!vma || (vma->vm_flags & device_vma))3485 if (!vma || (vma->vm_flags & device_vma))
drivers/gpu/drm/amd/amdkfd/kfd_topology.c+10−1
@@ -2013,12 +2013,21 @@ static void kfd_topology_set_capabilities(struct kfd_topology_device *dev) · 2012 unmodified lines
2013 dev->node_props.capability |=2013 dev->node_props.capability |=
2014 HSA_CAP_TRAP_DEBUG_PRECISE_MEMORY_OPERATIONS_SUPPORTED;2014 HSA_CAP_TRAP_DEBUG_PRECISE_MEMORY_OPERATIONS_SUPPORTED;
2015 2015
2016 if (!amdgpu_sriov_vf(dev->gpu->adev))2016 if (KFD_GC_VERSION(dev->gpu) >= IP_VERSION(9, 4, 3) &&
2017 !amdgpu_sriov_vf(dev->gpu->adev))
2017 dev->node_props.capability |= HSA_CAP_PER_QUEUE_RESET_SUPPORTED;2018 dev->node_props.capability |= HSA_CAP_PER_QUEUE_RESET_SUPPORTED;
2018 2019
2019 } else {2020 } else {
2020 dev->node_props.debug_prop |= HSA_DBG_WATCH_ADDR_MASK_LO_BIT_GFX10 |2021 dev->node_props.debug_prop |= HSA_DBG_WATCH_ADDR_MASK_LO_BIT_GFX10 |
2021 HSA_DBG_WATCH_ADDR_MASK_HI_BIT;2022 HSA_DBG_WATCH_ADDR_MASK_HI_BIT;
2023 /* gfx11 dGPU and gfx12.0 */
2024 if ((KFD_GC_VERSION(dev->gpu) == IP_VERSION(11, 0, 0) ||
2025 KFD_GC_VERSION(dev->gpu) == IP_VERSION(11, 0, 2) ||
2026 KFD_GC_VERSION(dev->gpu) == IP_VERSION(11, 0, 3) ||
2027 KFD_GC_VERSION(dev->gpu) == IP_VERSION(12, 0, 0) ||
2028 KFD_GC_VERSION(dev->gpu) == IP_VERSION(12, 0, 1)) &&
2029 !amdgpu_sriov_vf(dev->gpu->adev))
2030 dev->node_props.capability |= HSA_CAP_PER_QUEUE_RESET_SUPPORTED;
2022 2031
2023 if (KFD_GC_VERSION(dev->gpu) >= IP_VERSION(12, 0, 0))2032 if (KFD_GC_VERSION(dev->gpu) >= IP_VERSION(12, 0, 0))
2024 dev->node_props.capability |=2033 dev->node_props.capability |=
drivers/gpu/drm/amd/amdxcp/amdgpu_xcp_drv.c+12−6
@@ -45,7 +45,7 @@ static const struct drm_driver amdgpu_xcp_driver = { · 44 unmodified lines
45 .minor = 0,45 .minor = 0,
46};46};
47 47
48static int8_t pdev_num;48static u8 pdev_num;
49static struct xcp_device *xcp_dev[MAX_XCP_PLATFORM_DEVICE];49static struct xcp_device *xcp_dev[MAX_XCP_PLATFORM_DEVICE];
50static DEFINE_MUTEX(xcp_mutex);50static DEFINE_MUTEX(xcp_mutex);
51 51
@@ -56,6 +56,11 @@ int amdgpu_xcp_drm_dev_alloc(struct drm_device **ddev) · 4 unmodified lines
56 char *dev_name;56 char *dev_name;
57 int ret, i;57 int ret, i;
58 58
59 if (!ddev)
60 return -EINVAL;
61
62 BUILD_BUG_ON(MAX_XCP_PLATFORM_DEVICE >= U8_MAX);
63
59 guard(mutex)(&xcp_mutex);64 guard(mutex)(&xcp_mutex);
60 65
61 if (pdev_num >= MAX_XCP_PLATFORM_DEVICE)66 if (pdev_num >= MAX_XCP_PLATFORM_DEVICE)
@@ -105,7 +110,7 @@ out_unregister: · 43 unmodified lines
105}110}
106EXPORT_SYMBOL(amdgpu_xcp_drm_dev_alloc);111EXPORT_SYMBOL(amdgpu_xcp_drm_dev_alloc);
107 112
108static void free_xcp_dev(int8_t index)113static void free_xcp_dev(uint8_t index)
109{114{
110 if ((index < MAX_XCP_PLATFORM_DEVICE) && (xcp_dev[index])) {115 if ((index < MAX_XCP_PLATFORM_DEVICE) && (xcp_dev[index])) {
111 struct platform_device *pdev = xcp_dev[index]->pdev;116 struct platform_device *pdev = xcp_dev[index]->pdev;
@@ -114,17 +119,18 @@ static void free_xcp_dev(int8_t index) · 2 unmodified lines
114 platform_device_unregister(pdev);119 platform_device_unregister(pdev);
115 120
116 xcp_dev[index] = NULL;121 xcp_dev[index] = NULL;
117 pdev_num--;122 if (pdev_num > 0)
123 pdev_num--;
118 }124 }
119}125}
120 126
121void amdgpu_xcp_drm_dev_free(struct drm_device *ddev)127void amdgpu_xcp_drm_dev_free(struct drm_device *ddev)
122{128{
123 int8_t i;129 uint8_t i;
124 130
125 guard(mutex)(&xcp_mutex);131 guard(mutex)(&xcp_mutex);
126 132
127 for (i = 0; i < MAX_XCP_PLATFORM_DEVICE; i++) {133 for (i = 0; pdev_num && i < MAX_XCP_PLATFORM_DEVICE; i++) {
128 if ((xcp_dev[i]) && (&xcp_dev[i]->drm == ddev)) {134 if ((xcp_dev[i]) && (&xcp_dev[i]->drm == ddev)) {
129 free_xcp_dev(i);135 free_xcp_dev(i);
130 break;136 break;
@@ -135,7 +141,7 @@ EXPORT_SYMBOL(amdgpu_xcp_drm_dev_free); · 4 unmodified lines
135 141
136void amdgpu_xcp_drv_release(void)142void amdgpu_xcp_drv_release(void)
137{143{
138 int8_t i;144 uint8_t i;
139 145
140 guard(mutex)(&xcp_mutex);146 guard(mutex)(&xcp_mutex);
141 147
drivers/gpu/drm/amd/display/amdgpu_dm/Makefile+5−1
@@ -41,7 +41,11 @@ AMDGPUDM = \ · 40 unmodified lines
41 amdgpu_dm_quirks.o \41 amdgpu_dm_quirks.o \
42 amdgpu_dm_wb.o \42 amdgpu_dm_wb.o \
43 amdgpu_dm_colorop.o \43 amdgpu_dm_colorop.o \
44 amdgpu_dm_ism.o44 amdgpu_dm_ism.o \
45 amdgpu_dm_backlight.o \
46 amdgpu_dm_audio.o \
47 amdgpu_dm_dmub.o \
48 amdgpu_dm_connector.o
45 49
46ifdef CONFIG_DRM_AMD_DC_FP50ifdef CONFIG_DRM_AMD_DC_FP
47AMDGPUDM += dc_fpu.o51AMDGPUDM += dc_fpu.o
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c+2696−9516
@@ -66,6 +66,11 @@ · 65 unmodified lines
66#endif66#endif
67#include "amdgpu_dm_psr.h"67#include "amdgpu_dm_psr.h"
68#include "amdgpu_dm_replay.h"68#include "amdgpu_dm_replay.h"
69#include "amdgpu_dm_backlight.h"
70#include "amdgpu_dm_audio.h"
71#include "amdgpu_dm_dmub.h"
72#include "amdgpu_dm_connector.h"
73#include "amdgpu_dm_kunit_helpers.h"
69 74
70#include "ivsrcid/ivsrcid_vislands30.h"75#include "ivsrcid/ivsrcid_vislands30.h"
71 76
@@ -94,7 +99,6 @@ · 22 unmodified lines
94#include <drm/drm_mode.h>99#include <drm/drm_mode.h>
95#include <drm/drm_utils.h>100#include <drm/drm_utils.h>
96#include <drm/drm_vblank.h>101#include <drm/drm_vblank.h>
97#include <drm/drm_audio_component.h>
98#include <drm/drm_colorop.h>102#include <drm/drm_colorop.h>
99#include <drm/drm_gem_atomic_helper.h>103#include <drm/drm_gem_atomic_helper.h>
100 104
@@ -107,60 +111,9 @@ · 6 unmodified lines
107#include "modules/inc/mod_power.h"111#include "modules/inc/mod_power.h"
108#include "modules/power/power_helpers.h"112#include "modules/power/power_helpers.h"
109 113
110static_assert(AMDGPU_DMUB_NOTIFICATION_MAX == DMUB_NOTIFICATION_MAX, "AMDGPU_DMUB_NOTIFICATION_MAX mismatch");
111
112#define FIRMWARE_RENOIR_DMUB "amdgpu/renoir_dmcub.bin"
113MODULE_FIRMWARE(FIRMWARE_RENOIR_DMUB);
114#define FIRMWARE_SIENNA_CICHLID_DMUB "amdgpu/sienna_cichlid_dmcub.bin"
115MODULE_FIRMWARE(FIRMWARE_SIENNA_CICHLID_DMUB);
116#define FIRMWARE_NAVY_FLOUNDER_DMUB "amdgpu/navy_flounder_dmcub.bin"
117MODULE_FIRMWARE(FIRMWARE_NAVY_FLOUNDER_DMUB);
118#define FIRMWARE_GREEN_SARDINE_DMUB "amdgpu/green_sardine_dmcub.bin"
119MODULE_FIRMWARE(FIRMWARE_GREEN_SARDINE_DMUB);
120#define FIRMWARE_VANGOGH_DMUB "amdgpu/vangogh_dmcub.bin"
121MODULE_FIRMWARE(FIRMWARE_VANGOGH_DMUB);
122#define FIRMWARE_DIMGREY_CAVEFISH_DMUB "amdgpu/dimgrey_cavefish_dmcub.bin"
123MODULE_FIRMWARE(FIRMWARE_DIMGREY_CAVEFISH_DMUB);
124#define FIRMWARE_BEIGE_GOBY_DMUB "amdgpu/beige_goby_dmcub.bin"
125MODULE_FIRMWARE(FIRMWARE_BEIGE_GOBY_DMUB);
126#define FIRMWARE_YELLOW_CARP_DMUB "amdgpu/yellow_carp_dmcub.bin"
127MODULE_FIRMWARE(FIRMWARE_YELLOW_CARP_DMUB);
128#define FIRMWARE_DCN_314_DMUB "amdgpu/dcn_3_1_4_dmcub.bin"
129MODULE_FIRMWARE(FIRMWARE_DCN_314_DMUB);
130#define FIRMWARE_DCN_315_DMUB "amdgpu/dcn_3_1_5_dmcub.bin"
131MODULE_FIRMWARE(FIRMWARE_DCN_315_DMUB);
132#define FIRMWARE_DCN316_DMUB "amdgpu/dcn_3_1_6_dmcub.bin"
133MODULE_FIRMWARE(FIRMWARE_DCN316_DMUB);
134
135#define FIRMWARE_DCN_V3_2_0_DMCUB "amdgpu/dcn_3_2_0_dmcub.bin"
136MODULE_FIRMWARE(FIRMWARE_DCN_V3_2_0_DMCUB);
137#define FIRMWARE_DCN_V3_2_1_DMCUB "amdgpu/dcn_3_2_1_dmcub.bin"
138MODULE_FIRMWARE(FIRMWARE_DCN_V3_2_1_DMCUB);
139
140#define FIRMWARE_RAVEN_DMCU "amdgpu/raven_dmcu.bin"
141MODULE_FIRMWARE(FIRMWARE_RAVEN_DMCU);114MODULE_FIRMWARE(FIRMWARE_RAVEN_DMCU);
142
143#define FIRMWARE_NAVI12_DMCU "amdgpu/navi12_dmcu.bin"
144MODULE_FIRMWARE(FIRMWARE_NAVI12_DMCU);115MODULE_FIRMWARE(FIRMWARE_NAVI12_DMCU);
145 116
146#define FIRMWARE_DCN_35_DMUB "amdgpu/dcn_3_5_dmcub.bin"
147MODULE_FIRMWARE(FIRMWARE_DCN_35_DMUB);
148
149#define FIRMWARE_DCN_351_DMUB "amdgpu/dcn_3_5_1_dmcub.bin"
150MODULE_FIRMWARE(FIRMWARE_DCN_351_DMUB);
151
152#define FIRMWARE_DCN_36_DMUB "amdgpu/dcn_3_6_dmcub.bin"
153MODULE_FIRMWARE(FIRMWARE_DCN_36_DMUB);
154
155#define FIRMWARE_DCN_401_DMUB "amdgpu/dcn_4_0_1_dmcub.bin"
156MODULE_FIRMWARE(FIRMWARE_DCN_401_DMUB);
157
158#define FIRMWARE_DCN_42_DMUB "amdgpu/dcn_4_2_dmcub.bin"
159MODULE_FIRMWARE(FIRMWARE_DCN_42_DMUB);
160
161#define FIRMWARE_DCN_42B_DMUB "amdgpu/dcn_4_2_1_dmcub.bin"
162MODULE_FIRMWARE(FIRMWARE_DCN_42B_DMUB);
163
164/**117/**
165 * DOC: overview118 * DOC: overview
166 *119 *
@@ -174,46 +127,7 @@ MODULE_FIRMWARE(FIRMWARE_DCN_42B_DMUB); · 7 unmodified lines
174/* basic init/fini API */127/* basic init/fini API */
175static int amdgpu_dm_init(struct amdgpu_device *adev);128static int amdgpu_dm_init(struct amdgpu_device *adev);
176static void amdgpu_dm_fini(struct amdgpu_device *adev);129static void amdgpu_dm_fini(struct amdgpu_device *adev);
177static bool is_freesync_video_mode(const struct drm_display_mode *mode, struct amdgpu_dm_connector *aconnector);
178static void reset_freesync_config_for_crtc(struct dm_crtc_state *new_crtc_state);130static void reset_freesync_config_for_crtc(struct dm_crtc_state *new_crtc_state);
179static struct amdgpu_i2c_adapter *
180create_i2c(struct ddc_service *ddc_service, bool oem);
181
182static enum drm_mode_subconnector get_subconnector_type(struct dc_link *link)
183{
184 switch (link->dpcd_caps.dongle_type) {
185 case DISPLAY_DONGLE_NONE:
186 return DRM_MODE_SUBCONNECTOR_Native;
187 case DISPLAY_DONGLE_DP_VGA_CONVERTER:
188 return DRM_MODE_SUBCONNECTOR_VGA;
189 case DISPLAY_DONGLE_DP_DVI_CONVERTER:
190 case DISPLAY_DONGLE_DP_DVI_DONGLE:
191 return DRM_MODE_SUBCONNECTOR_DVID;
192 case DISPLAY_DONGLE_DP_HDMI_CONVERTER:
193 case DISPLAY_DONGLE_DP_HDMI_DONGLE:
194 return DRM_MODE_SUBCONNECTOR_HDMIA;
195 case DISPLAY_DONGLE_DP_HDMI_MISMATCHED_DONGLE:
196 default:
197 return DRM_MODE_SUBCONNECTOR_Unknown;
198 }
199}
200
201static void update_subconnector_property(struct amdgpu_dm_connector *aconnector)
202{
203 struct dc_link *link = aconnector->dc_link;
204 struct drm_connector *connector = &aconnector->base;
205 enum drm_mode_subconnector subconnector = DRM_MODE_SUBCONNECTOR_Unknown;
206
207 if (connector->connector_type != DRM_MODE_CONNECTOR_DisplayPort)
208 return;
209
210 if (aconnector->dc_sink)
211 subconnector = get_subconnector_type(link);
212
213 drm_object_property_set_value(&connector->base,
214 connector->dev->mode_config.dp_subconnector_property,
215 subconnector);
216}
217 131
218/*132/*
219 * initializes drm_device display related structures, based on the information133 * initializes drm_device display related structures, based on the information
@@ -226,32 +140,23 @@ static int amdgpu_dm_initialize_drm_device(struct amdgpu_device *adev); · 6 unmodified lines
226/* removes and deallocates the drm structures, created by the above function */140/* removes and deallocates the drm structures, created by the above function */
227static void amdgpu_dm_destroy_drm_device(struct amdgpu_display_manager *dm);141static void amdgpu_dm_destroy_drm_device(struct amdgpu_display_manager *dm);
228 142
229static int amdgpu_dm_connector_init(struct amdgpu_display_manager *dm,
230 struct amdgpu_dm_connector *amdgpu_dm_connector,
231 u32 link_index,
232 struct amdgpu_encoder *amdgpu_encoder);
233static int amdgpu_dm_encoder_init(struct drm_device *dev,
234 struct amdgpu_encoder *aencoder,
235 uint32_t link_index);
236
237static int amdgpu_dm_connector_get_modes(struct drm_connector *connector);
238
239static int amdgpu_dm_atomic_setup_commit(struct drm_atomic_commit *state);143static int amdgpu_dm_atomic_setup_commit(struct drm_atomic_commit *state);
240static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_commit *state);144static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_commit *state);
145static void dm_enable_per_frame_crtc_master_sync(struct dc_state *context);
241 146
242static int amdgpu_dm_atomic_check(struct drm_device *dev,147static int amdgpu_dm_atomic_check(struct drm_device *dev,
243 struct drm_atomic_commit *state);148 struct drm_atomic_commit *state);
244 149
245static void handle_hpd_irq_helper(struct amdgpu_dm_connector *aconnector);150STATIC_IFN_KUNIT bool
246static void handle_hpd_rx_irq(void *param);
247
248static void amdgpu_dm_backlight_set_level(struct amdgpu_display_manager *dm,
249 int bl_idx,
250 u32 user_brightness);
251
252static bool
253is_timing_unchanged_for_freesync(struct drm_crtc_state *old_crtc_state,151is_timing_unchanged_for_freesync(struct drm_crtc_state *old_crtc_state,
254 struct drm_crtc_state *new_crtc_state);152 struct drm_crtc_state *new_crtc_state);
153
154static inline void amdgpu_dm_exit_ips_for_hw_access(struct dc *dc)
155{
156 if (dc->ctx->dmub_srv && !dc->ctx->dmub_srv->idle_exit_counter)
157 dc_exit_ips_for_hw_access(dc);
158}
159
255/*160/*
256 * dm_vblank_get_counter161 * dm_vblank_get_counter
257 *162 *
@@ -332,40 +237,14 @@ static int dm_wait_for_idle(struct amdgpu_ip_block *ip_block) · 74 unmodified lines
332 return 0;237 return 0;
333}238}
334 239
335static bool dm_check_soft_reset(struct amdgpu_ip_block *ip_block)
336{
337 return false;
338}
339
340static int dm_soft_reset(struct amdgpu_ip_block *ip_block)240static int dm_soft_reset(struct amdgpu_ip_block *ip_block)
341{241{
342 /* XXX todo */242 /* XXX todo */
343 return 0;243 return 0;
344}244}
345 245
346static struct amdgpu_crtc *246STATIC_IFN_KUNIT bool is_dc_timing_adjust_needed(struct dm_crtc_state *old_state,
347get_crtc_by_otg_inst(struct amdgpu_device *adev,247 struct dm_crtc_state *new_state)
348 int otg_inst)
349{
350 struct drm_device *dev = adev_to_drm(adev);
351 struct drm_crtc *crtc;
352 struct amdgpu_crtc *amdgpu_crtc;
353
354 if (WARN_ON(otg_inst == -1))
355 return adev->mode_info.crtcs[0];
356
357 list_for_each_entry(crtc, &dev->mode_config.crtc_list, head) {
358 amdgpu_crtc = to_amdgpu_crtc(crtc);
359
360 if (amdgpu_crtc->otg_inst == otg_inst)
361 return amdgpu_crtc;
362 }
363
364 return NULL;
365}
366
367static inline bool is_dc_timing_adjust_needed(struct dm_crtc_state *old_state,
368 struct dm_crtc_state *new_state)
369{248{
370 if (new_state->stream->adjust.timing_adjust_pending)249 if (new_state->stream->adjust.timing_adjust_pending)
371 return true;250 return true;
@@ -376,13 +255,14 @@ static inline bool is_dc_timing_adjust_needed(struct dm_crtc_state *old_state, · 4 unmodified lines
376 else255 else
377 return false;256 return false;
378}257}
258EXPORT_IF_KUNIT(is_dc_timing_adjust_needed);
379 259
380/*260/*
381 * DC will program planes with their z-order determined by their ordering261 * DC will program planes with their z-order determined by their ordering
382 * in the dc_surface_updates array. This comparator is used to sort them262 * in the dc_surface_updates array. This comparator is used to sort them
383 * by descending zpos.263 * by descending zpos.
384 */264 */
385static int dm_plane_layer_index_cmp(const void *a, const void *b)265STATIC_IFN_KUNIT int dm_plane_layer_index_cmp(const void *a, const void *b)
386{266{
387 const struct dc_surface_update *sa = (struct dc_surface_update *)a;267 const struct dc_surface_update *sa = (struct dc_surface_update *)a;
388 const struct dc_surface_update *sb = (struct dc_surface_update *)b;268 const struct dc_surface_update *sb = (struct dc_surface_update *)b;
@@ -390,6 +270,7 @@ static int dm_plane_layer_index_cmp(const void *a, const void *b) · 1 unmodified line
390 /* Sort by descending dc_plane layer_index (i.e. normalized_zpos) */270 /* Sort by descending dc_plane layer_index (i.e. normalized_zpos) */
391 return sb->surface->layer_index - sa->surface->layer_index;271 return sb->surface->layer_index - sa->surface->layer_index;
392}272}
273EXPORT_IF_KUNIT(dm_plane_layer_index_cmp);
393 274
394/**275/**
395 * update_planes_and_stream_adapter() - Send planes to be updated in DC276 * update_planes_and_stream_adapter() - Send planes to be updated in DC
@@ -430,9291 +311,3195 @@ static inline bool update_planes_and_stream_adapter(struct dc *dc, · 34 unmodified lines
430 stream_update);311 stream_update);
431}312}
432 313
433/**314static int dm_set_clockgating_state(struct amdgpu_ip_block *ip_block,
434 * dm_pflip_high_irq() - Handle pageflip interrupt315 enum amd_clockgating_state state)
435 * @interrupt_params: ignored
436 *
437 * Handles the pageflip interrupt by notifying all interested parties
438 * that the pageflip has been completed.
439 */
440static void dm_pflip_high_irq(void *interrupt_params)
441{316{
442 struct amdgpu_crtc *amdgpu_crtc;317 return 0;
443 struct common_irq_params *irq_params = interrupt_params;318}
444 struct amdgpu_device *adev = irq_params->adev;
445 struct drm_device *dev = adev_to_drm(adev);
446 unsigned long flags;
447 struct drm_pending_vblank_event *e;
448 u32 vpos, hpos, v_blank_start, v_blank_end;
449 bool vrr_active;
450 319
451 amdgpu_crtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_PFLIP);320static int dm_set_powergating_state(struct amdgpu_ip_block *ip_block,
321 enum amd_powergating_state state)
322{
323 return 0;
324}
452 325
453 /* IRQ could occur when in initial stage */326/* Prototypes of private functions */
454 /* TODO work and BO cleanup */327static int dm_early_init(struct amdgpu_ip_block *ip_block);
455 if (amdgpu_crtc == NULL) {
456 drm_dbg_state(dev, "CRTC is null, returning.\n");
457 return;
458 }
459 328
460 spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);329/* Allocate memory for FBC compressed data */
330static void mmhub_read_system_context(struct amdgpu_device *adev, struct dc_phy_addr_space_config *pa_config)
331{
332 u64 pt_base;
333 u32 logical_addr_low;
334 u32 logical_addr_high;
335 u32 agp_base, agp_bot, agp_top;
336 PHYSICAL_ADDRESS_LOC page_table_start, page_table_end, page_table_base;
461 337
462 if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED) {338 memset(pa_config, 0, sizeof(*pa_config));
463 drm_dbg_state(dev,
464 "amdgpu_crtc->pflip_status = %d != AMDGPU_FLIP_SUBMITTED(%d) on crtc:%d[%p]\n",
465 amdgpu_crtc->pflip_status, AMDGPU_FLIP_SUBMITTED,
466 amdgpu_crtc->crtc_id, amdgpu_crtc);
467 spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);
468 return;
469 }
470 339
471 /* page flip completed. */340 agp_base = 0;
472 e = amdgpu_crtc->event;341 agp_bot = adev->gmc.agp_start >> 24;
473 amdgpu_crtc->event = NULL;342 agp_top = adev->gmc.agp_end >> 24;
474 343
475 WARN_ON(!e);344 /* AGP aperture is disabled */
345 if (agp_bot > agp_top) {
346 logical_addr_low = adev->gmc.fb_start >> 18;
347 if (adev->apu_flags & (AMD_APU_IS_RAVEN2 |
348 AMD_APU_IS_RENOIR |
349 AMD_APU_IS_GREEN_SARDINE))
350 /*
351 * Raven2 has a HW issue that it is unable to use the vram which
352 * is out of MC_VM_SYSTEM_APERTURE_HIGH_ADDR. So here is the
353 * workaround that increase system aperture high address (add 1)
354 * to get rid of the VM fault and hardware hang.
355 */
356 logical_addr_high = (adev->gmc.fb_end >> 18) + 0x1;
357 else
358 logical_addr_high = adev->gmc.fb_end >> 18;
359 } else {
360 logical_addr_low = min(adev->gmc.fb_start, adev->gmc.agp_start) >> 18;
361 if (adev->apu_flags & (AMD_APU_IS_RAVEN2 |
362 AMD_APU_IS_RENOIR |
363 AMD_APU_IS_GREEN_SARDINE))
364 /*
365 * Raven2 has a HW issue that it is unable to use the vram which
366 * is out of MC_VM_SYSTEM_APERTURE_HIGH_ADDR. So here is the
367 * workaround that increase system aperture high address (add 1)
368 * to get rid of the VM fault and hardware hang.
369 */
370 logical_addr_high = max((adev->gmc.fb_end >> 18) + 0x1, adev->gmc.agp_end >> 18);
371 else
372 logical_addr_high = max(adev->gmc.fb_end, adev->gmc.agp_end) >> 18;
373 }
476 374
477 vrr_active = amdgpu_dm_crtc_vrr_active_irq(amdgpu_crtc);375 pt_base = amdgpu_gmc_pd_addr(adev->gart.bo);
478 376
479 /* Fixed refresh rate, or VRR scanout position outside front-porch? */377 page_table_start.high_part = upper_32_bits(adev->gmc.gart_start >>
480 if (!vrr_active ||378 AMDGPU_GPU_PAGE_SHIFT);
481 !dc_stream_get_scanoutpos(amdgpu_crtc->dm_irq_params.stream, &v_blank_start,379 page_table_start.low_part = lower_32_bits(adev->gmc.gart_start >>
482 &v_blank_end, &hpos, &vpos) ||380 AMDGPU_GPU_PAGE_SHIFT);
483 (vpos < v_blank_start)) {381 page_table_end.high_part = upper_32_bits(adev->gmc.gart_end >>
484 /* Update to correct count and vblank timestamp if racing with382 AMDGPU_GPU_PAGE_SHIFT);
485 * vblank irq. This also updates to the correct vblank timestamp383 page_table_end.low_part = lower_32_bits(adev->gmc.gart_end >>
486 * even in VRR mode, as scanout is past the front-porch atm.384 AMDGPU_GPU_PAGE_SHIFT);
487 */385 page_table_base.high_part = upper_32_bits(pt_base);
488 drm_crtc_accurate_vblank_count(&amdgpu_crtc->base);386 page_table_base.low_part = lower_32_bits(pt_base);
489 387
490 /* Wake up userspace by sending the pageflip event with proper388 pa_config->system_aperture.start_addr = (uint64_t)logical_addr_low << 18;
491 * count and timestamp of vblank of flip completion.389 pa_config->system_aperture.end_addr = (uint64_t)logical_addr_high << 18;
492 */
493 if (e) {
494 drm_crtc_send_vblank_event(&amdgpu_crtc->base, e);
495
496 /* Event sent, so done with vblank for this flip */
497 drm_crtc_vblank_put(&amdgpu_crtc->base);
498 }
499 } else if (e) {
500 /* VRR active and inside front-porch: vblank count and
501 * timestamp for pageflip event will only be up to date after
502 * drm_crtc_handle_vblank() has been executed from late vblank
503 * irq handler after start of back-porch (vline 0). We queue the
504 * pageflip event for send-out by drm_crtc_handle_vblank() with
505 * updated timestamp and count, once it runs after us.
506 *
507 * We need to open-code this instead of using the helper
508 * drm_crtc_arm_vblank_event(), as that helper would
509 * call drm_crtc_accurate_vblank_count(), which we must
510 * not call in VRR mode while we are in front-porch!
511 */
512 390
513 /* sequence will be replaced by real count during send-out. */391 pa_config->system_aperture.agp_base = (uint64_t)agp_base << 24;
514 e->sequence = drm_crtc_vblank_count(&amdgpu_crtc->base);392 pa_config->system_aperture.agp_bot = (uint64_t)agp_bot << 24;
515 e->pipe = amdgpu_crtc->crtc_id;393 pa_config->system_aperture.agp_top = (uint64_t)agp_top << 24;
516 394
517 list_add_tail(&e->base.link, &adev_to_drm(adev)->vblank_event_list);395 pa_config->system_aperture.fb_base = adev->gmc.fb_start;
518 e = NULL;396 pa_config->system_aperture.fb_offset = adev->vm_manager.vram_base_offset;
519 }397 pa_config->system_aperture.fb_top = adev->gmc.fb_end;
520 398
521 /* Keep track of vblank of this flip for flip throttling. We use the399 pa_config->gart_config.page_table_start_addr = page_table_start.quad_part << 12;
522 * cooked hw counter, as that one incremented at start of this vblank400 pa_config->gart_config.page_table_end_addr = page_table_end.quad_part << 12;
523 * of pageflip completion, so last_flip_vblank is the forbidden count401 pa_config->gart_config.page_table_base_addr = page_table_base.quad_part;
524 * for queueing new pageflips if vsync + VRR is enabled.
525 */
526 amdgpu_crtc->dm_irq_params.last_flip_vblank =
527 amdgpu_get_vblank_counter_kms(&amdgpu_crtc->base);
528 402
529 amdgpu_crtc->pflip_status = AMDGPU_FLIP_NONE;403 pa_config->is_hvm_enabled = adev->mode_info.gpu_vm_support;
530 spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);
531 404
532 drm_dbg_state(dev,
533 "crtc:%d[%p], pflip_stat:AMDGPU_FLIP_NONE, vrr[%d]-fp %d\n",
534 amdgpu_crtc->crtc_id, amdgpu_crtc, vrr_active, (int)!e);
535}405}
536 406
537static void dm_handle_vmin_vmax_update(struct work_struct *offload_work)407struct amdgpu_stutter_quirk {
538{408 u16 chip_vendor;
539 struct vupdate_offload_work *work = container_of(offload_work, struct vupdate_offload_work, work);409 u16 chip_device;
540 struct amdgpu_device *adev = work->adev;410 u16 subsys_vendor;
541 struct dc_stream_state *stream = work->stream;411 u16 subsys_device;
542 struct dc_crtc_timing_adjust *adjust = work->adjust;412 u8 revision;
543 413};
544 mutex_lock(&adev->dm.dc_lock);
545 dc_stream_adjust_vmin_vmax(adev->dm.dc, stream, adjust);
546 mutex_unlock(&adev->dm.dc_lock);
547 414
548 dc_stream_release(stream);415static const struct amdgpu_stutter_quirk amdgpu_stutter_quirk_list[] = {
549 kfree(work->adjust);416 /* https://bugzilla.kernel.org/show_bug.cgi?id=214417 */
550 kfree(work);417 { 0x1002, 0x15dd, 0x1002, 0x15dd, 0xc8 },
551}418 { 0, 0, 0, 0, 0 },
419};
552 420
553static void schedule_dc_vmin_vmax(struct amdgpu_device *adev,421static bool dm_should_disable_stutter(struct pci_dev *pdev)
554 struct dc_stream_state *stream,
555 struct dc_crtc_timing_adjust *adjust)
556{422{
557 struct vupdate_offload_work *offload_work = kzalloc_obj(*offload_work,423 const struct amdgpu_stutter_quirk *p = amdgpu_stutter_quirk_list;
558 GFP_NOWAIT);
559 if (!offload_work) {
560 drm_dbg_driver(adev_to_drm(adev), "Failed to allocate vupdate_offload_work\n");
561 return;
562 }
563 424
564 struct dc_crtc_timing_adjust *adjust_copy = kzalloc_obj(*adjust_copy,425 while (p && p->chip_device != 0) {
565 GFP_NOWAIT);426 if (pdev->vendor == p->chip_vendor &&
566 if (!adjust_copy) {427 pdev->device == p->chip_device &&
567 drm_dbg_driver(adev_to_drm(adev), "Failed to allocate adjust_copy\n");428 pdev->subsystem_vendor == p->subsys_vendor &&
568 kfree(offload_work);429 pdev->subsystem_device == p->subsys_device &&
569 return;430 pdev->revision == p->revision) {
431 return true;
432 }
433 ++p;
570 }434 }
571 435 return false;
572 dc_stream_retain(stream);
573 memcpy(adjust_copy, adjust, sizeof(*adjust_copy));
574
575 INIT_WORK(&offload_work->work, dm_handle_vmin_vmax_update);
576 offload_work->adev = adev;
577 offload_work->stream = stream;
578 offload_work->adjust = adjust_copy;
579
580 queue_work(system_percpu_wq, &offload_work->work);
581}436}
582 437
583static void dm_vupdate_high_irq(void *interrupt_params)
584{
585 struct common_irq_params *irq_params = interrupt_params;
586 struct amdgpu_device *adev = irq_params->adev;
587 struct amdgpu_crtc *acrtc;
588 struct drm_device *drm_dev;
589 struct drm_vblank_crtc *vblank;
590 ktime_t frame_duration_ns, previous_timestamp;
591 unsigned long flags;
592 int vrr_active;
593 438
594 acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VUPDATE);439void*
440dm_allocate_gpu_mem(
441 struct amdgpu_device *adev,
442 enum dc_gpu_mem_alloc_type type,
443 size_t size,
444 long long *addr)
445{
446 struct dal_allocation *da;
447 u32 domain = (type == DC_MEM_ALLOC_TYPE_GART) ?
448 AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM;
449 int ret;
595 450
596 if (acrtc) {451 da = kzalloc_obj(struct dal_allocation);
597 vrr_active = amdgpu_dm_crtc_vrr_active_irq(acrtc);452 if (!da)
598 drm_dev = acrtc->base.dev;453 return NULL;
599 vblank = drm_crtc_vblank_crtc(&acrtc->base);
600 previous_timestamp = atomic64_read(&irq_params->previous_timestamp);
601 frame_duration_ns = vblank->time - previous_timestamp;
602 454
603 if (frame_duration_ns > 0) {455 ret = amdgpu_bo_create_kernel(adev, size, PAGE_SIZE,
604 trace_amdgpu_refresh_rate_track(acrtc->base.index,456 domain, &da->bo,
605 frame_duration_ns,457 &da->gpu_addr, &da->cpu_ptr);
606 ktime_divns(NSEC_PER_SEC, frame_duration_ns));
607 atomic64_set(&irq_params->previous_timestamp, vblank->time);
608 }
609 458
610 drm_dbg_vbl(drm_dev,459 *addr = da->gpu_addr;
611 "crtc:%d, vupdate-vrr:%d\n", acrtc->crtc_id,
612 vrr_active);
613 460
614 /* Core vblank handling is done here after end of front-porch in461 if (ret) {
615 * vrr mode, as vblank timestamping will give valid results462 kfree(da);
616 * while now done after front-porch. This will also deliver463 return NULL;
617 * page-flip completion events that have been queued to us
618 * if a pageflip happened inside front-porch.
619 */
620 if (vrr_active && acrtc->dm_irq_params.stream) {
621 bool replay_en = acrtc->dm_irq_params.stream->link->replay_settings.replay_feature_enabled;
622 bool psr_en = acrtc->dm_irq_params.stream->link->psr_settings.psr_feature_enabled;
623 bool fs_active_var_en = acrtc->dm_irq_params.freesync_config.state
624 == VRR_STATE_ACTIVE_VARIABLE;
625
626 amdgpu_dm_crtc_handle_vblank(acrtc);
627
628 /* BTR processing for pre-DCE12 ASICs */
629 if (adev->family < AMDGPU_FAMILY_AI) {
630 spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);
631 mod_freesync_handle_v_update(
632 adev->dm.freesync_module,
633 acrtc->dm_irq_params.stream,
634 &acrtc->dm_irq_params.vrr_params);
635
636 if (fs_active_var_en || (!fs_active_var_en && !replay_en && !psr_en)) {
637 schedule_dc_vmin_vmax(adev,
638 acrtc->dm_irq_params.stream,
639 &acrtc->dm_irq_params.vrr_params.adjust);
640 }
641 spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);
642 }
643 }
644 }464 }
465
466 /* add da to list in dm */
467 list_add(&da->list, &adev->dm.da_list);
468
469 return da->cpu_ptr;
645}470}
646 471
647/**472void
648 * dm_crtc_high_irq() - Handles CRTC interrupt473dm_free_gpu_mem(
649 * @interrupt_params: used for determining the CRTC instance474 struct amdgpu_device *adev,
650 *475 enum dc_gpu_mem_alloc_type type,
651 * Handles the CRTC/VSYNC interrupt by notfying DRM's VBLANK476 void *pvMem)
652 * event handler.
653 */
654static void dm_crtc_high_irq(void *interrupt_params)
655{477{
656 struct common_irq_params *irq_params = interrupt_params;478 struct dal_allocation *da;
657 struct amdgpu_device *adev = irq_params->adev;
658 struct drm_writeback_job *job;
659 struct amdgpu_crtc *acrtc;
660 unsigned long flags;
661 int vrr_active;
662
663 acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VBLANK);
664 if (!acrtc)
665 return;
666 479
667 if (acrtc->wb_conn) {480 /* walk the da list in DM */
668 spin_lock_irqsave(&acrtc->wb_conn->job_lock, flags);481 list_for_each_entry(da, &adev->dm.da_list, list) {
669 482 if (pvMem == da->cpu_ptr) {
670 if (acrtc->wb_pending) {483 amdgpu_bo_free_kernel(&da->bo, &da->gpu_addr, &da->cpu_ptr);
671 job = list_first_entry_or_null(&acrtc->wb_conn->job_queue,484 list_del(&da->list);
672 struct drm_writeback_job,485 kfree(da);
673 list_entry);486 break;
674 acrtc->wb_pending = false;487 }
675 spin_unlock_irqrestore(&acrtc->wb_conn->job_lock, flags);
676
677 if (job) {
678 unsigned int v_total, refresh_hz;
679 struct dc_stream_state *stream = acrtc->dm_irq_params.stream;
680
681 v_total = stream->adjust.v_total_max ?
682 stream->adjust.v_total_max : stream->timing.v_total;
683 refresh_hz = div_u64((uint64_t) stream->timing.pix_clk_100hz *
684 100LL, (v_total * stream->timing.h_total));
685 mdelay(1000 / refresh_hz);
686
687 drm_writeback_signal_completion(acrtc->wb_conn, 0);
688 dc_stream_fc_disable_writeback(adev->dm.dc,
689 acrtc->dm_irq_params.stream, 0);
690 }
691 } else
692 spin_unlock_irqrestore(&acrtc->wb_conn->job_lock, flags);
693 }488 }
694 489
695 vrr_active = amdgpu_dm_crtc_vrr_active_irq(acrtc);490}
696
697 drm_dbg_vbl(adev_to_drm(adev),
698 "crtc:%d, vupdate-vrr:%d, planes:%d\n", acrtc->crtc_id,
699 vrr_active, acrtc->dm_irq_params.active_planes);
700
701 /**
702 * Core vblank handling at start of front-porch is only possible
703 * in non-vrr mode, as only there vblank timestamping will give
704 * valid results while done in front-porch. Otherwise defer it
705 * to dm_vupdate_high_irq after end of front-porch.
706 */
707 if (!vrr_active)
708 amdgpu_dm_crtc_handle_vblank(acrtc);
709
710 /**
711 * Following stuff must happen at start of vblank, for crc
712 * computation and below-the-range btr support in vrr mode.
713 */
714 amdgpu_dm_crtc_handle_crc_irq(&acrtc->base);
715
716 /* BTR updates need to happen before VUPDATE on Vega and above. */
717 if (adev->family < AMDGPU_FAMILY_AI)
718 return;
719
720 spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags);
721
722 if (acrtc->dm_irq_params.stream &&
723 acrtc->dm_irq_params.vrr_params.supported) {
724 bool replay_en = acrtc->dm_irq_params.stream->link->replay_settings.replay_feature_enabled;
725 bool psr_en = acrtc->dm_irq_params.stream->link->psr_settings.psr_feature_enabled;
726 bool fs_active_var_en = acrtc->dm_irq_params.freesync_config.state == VRR_STATE_ACTIVE_VARIABLE;
727 491
728 mod_freesync_handle_v_update(adev->dm.freesync_module,492static int amdgpu_dm_init_power_module(struct amdgpu_display_manager *dm)
729 acrtc->dm_irq_params.stream,493{
730 &acrtc->dm_irq_params.vrr_params);494 struct mod_power_init_params init_data[MAX_NUM_EDP];
731 495
732 /* update vmin_vmax only if freesync is enabled, or only if PSR and REPLAY are disabled */496 if (dm->num_of_edps == 0) {
733 if (fs_active_var_en || (!fs_active_var_en && !replay_en && !psr_en)) {497 drm_dbg_driver(
734 schedule_dc_vmin_vmax(adev, acrtc->dm_irq_params.stream,498 dm->ddev,
735 &acrtc->dm_irq_params.vrr_params.adjust);499 "amdgpu: No eDP detected, skip initializing power module\n");
736 }500 return 0;
737 }501 }
738 502
739 /*503 /* Initialize all the power module parameters */
740 * If there aren't any active_planes then DCH HUBP may be clock-gated.504 for (int i = 0; i < dm->num_of_edps; i++) {
741 * In that case, pageflip completion interrupts won't fire and pageflip505 init_data[i].allow_psr_smu_optimizations =
742 * completion events won't get delivered. Prevent this by sending506 !!(amdgpu_dc_feature_mask & DC_PSR_ALLOW_SMU_OPT);
743 * pending pageflip events from here if a flip is still pending.507 init_data[i].allow_psr_multi_disp_optimizations =
744 *508 !!(amdgpu_dc_feature_mask & DC_PSR_ALLOW_MULTI_DISP_OPT);
745 * If any planes are enabled, use dm_pflip_high_irq() instead, to509 /* See dm_late_init */
746 * avoid race conditions between flip programming and completion,510 init_data[i].backlight_ramping_override = false;
747 * which could cause too early flip completion events.511 init_data[i].backlight_ramping_start = 0xCCCC;
748 */512 init_data[i].backlight_ramping_reduction = 0xCCCCCCCC;
749 if (adev->family >= AMDGPU_FAMILY_RV &&513 init_data[i].def_varibright_level = 0;
750 acrtc->pflip_status == AMDGPU_FLIP_SUBMITTED &&514 init_data[i].abm_config_setting = 0;
751 acrtc->dm_irq_params.active_planes == 0) {515 init_data[i].num_backlight_levels = 101;
752 if (acrtc->event) {516 init_data[i].use_nits_based_brightness = false;
753 drm_crtc_send_vblank_event(&acrtc->base, acrtc->event);517 init_data[i].panel_max_millinits = 0;
754 acrtc->event = NULL;518 init_data[i].panel_min_millinits = 0;
755 drm_crtc_vblank_put(&acrtc->base);519 init_data[i].disable_fractional_pwm =
756 }520 !(amdgpu_dc_feature_mask & DC_DISABLE_FRACTIONAL_PWM_MASK);
757 acrtc->pflip_status = AMDGPU_FLIP_NONE;521 init_data[i].use_custom_backlight_caps = false;
758 }522 init_data[i].custom_backlight_caps_config_no = 0;
759 523 init_data[i].use_linear_backlight_curve = false;
760 spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags);524 init_data[i].def_varibright_enable = 0;
761}525 init_data[i].varibright_level = 0;
762 526 /*
763#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)527 * Power module uses 16-bit backlight levels (0xFFFF max) rather
764/**528 * than 8-bit(0XFF max)
765 * dm_dcn_vertical_interrupt0_high_irq() - Handles OTG Vertical interrupt0 for529 */
766 * DCN generation ASICs530 init_data[i].min_backlight_pwm =
767 * @interrupt_params: interrupt parameters531 dm->backlight_caps[i].min_input_signal * 0x101;
768 *532 init_data[i].max_backlight_pwm =
769 * Used to set crc window/read out crc value at vertical line 0 position533 dm->backlight_caps[i].max_input_signal * 0x101;
770 */534 init_data[i].min_abm_backlight =
771static void dm_dcn_vertical_interrupt0_high_irq(void *interrupt_params)535 dm->backlight_caps[i].min_input_signal * 0x101;
772{
773 struct common_irq_params *irq_params = interrupt_params;
774 struct amdgpu_device *adev = irq_params->adev;
775 struct amdgpu_crtc *acrtc;
776 536
777 acrtc = get_crtc_by_otg_inst(adev, irq_params->irq_src - IRQ_TYPE_VLINE0);537 /* Min backlight level after ABM reduction, Don't allow below 1%
538 * 0xFFFF x 0.01 = 0x28F
539 */
540 init_data[i].min_abm_backlight = (init_data[i].min_abm_backlight < 0x28F) ?
541 0x28F : init_data[i].min_abm_backlight;
542 }
778 543
779 if (!acrtc)544 dm->power_module = mod_power_create(dm->dc, init_data, dm->num_of_edps);
780 return;545 if (!dm->power_module) {
546 drm_err(dm->ddev, "amdgpu: Error allocating memory for power module\n");
547 return -ENOMEM;
548 }
781 549
782 amdgpu_dm_crtc_handle_crc_window_irq(&acrtc->base);550 mod_power_hw_init(dm->power_module);
783}551 drm_dbg_driver(dm->ddev, "amdgpu: Power module init done\n");
784#endif /* CONFIG_DRM_AMD_SECURE_DISPLAY */
785 552
786/**553 return 0;
787 * dmub_aux_setconfig_callback - Callback for AUX or SET_CONFIG command.
788 * @adev: amdgpu_device pointer
789 * @notify: dmub notification structure
790 *
791 * Dmub AUX or SET_CONFIG command completion processing callback
792 * Copies dmub notification to DM which is to be read by AUX command.
793 * issuing thread and also signals the event to wake up the thread.
794 */
795static void dmub_aux_setconfig_callback(struct amdgpu_device *adev,
796 struct dmub_notification *notify)
797{
798 if (adev->dm.dmub_notify)
799 memcpy(adev->dm.dmub_notify, notify, sizeof(struct dmub_notification));
800 if (notify->type == DMUB_NOTIFICATION_AUX_REPLY)
801 complete(&adev->dm.dmub_aux_transfer_done);
802}554}
803 555
804static void dmub_aux_fused_io_callback(struct amdgpu_device *adev,556static int amdgpu_dm_init(struct amdgpu_device *adev)
805 struct dmub_notification *notify)
806{557{
807 if (!adev || !notify) {558 struct dc_init_data init_data;
808 ASSERT(false);559 struct dc_callback_init init_params;
809 return;560 int r;
810 }
811
812 const struct dmub_cmd_fused_request *req = &notify->fused_request;
813 const uint8_t ddc_line = req->u.aux.ddc_line;
814
815 if (ddc_line >= ARRAY_SIZE(adev->dm.fused_io)) {
816 ASSERT(false);
817 return;
818 }
819 561
820 struct fused_io_sync *sync = &adev->dm.fused_io[ddc_line];562 adev->dm.ddev = adev_to_drm(adev);
563 adev->dm.adev = adev;
821 564
822 static_assert(sizeof(*req) <= sizeof(sync->reply_data), "Size mismatch");565 /* Zero all the fields */
823 memcpy(sync->reply_data, req, sizeof(*req));566 memset(&init_data, 0, sizeof(init_data));
824 complete(&sync->replied);567 memset(&init_params, 0, sizeof(init_params));
825}
826 568
827/**569 mutex_init(&adev->dm.dpia_aux_lock);
828 * dmub_hpd_callback - DMUB HPD interrupt processing callback.570 mutex_init(&adev->dm.dc_lock);
829 * @adev: amdgpu_device pointer571 mutex_init(&adev->dm.audio_lock);
830 * @notify: dmub notification structure
831 *
832 * Dmub Hpd interrupt processing callback. Gets displayindex through the
833 * ink index and calls helper to do the processing.
834 */
835static void dmub_hpd_callback(struct amdgpu_device *adev,
836 struct dmub_notification *notify)
837{
838 struct amdgpu_dm_connector *aconnector;
839 struct amdgpu_dm_connector *hpd_aconnector = NULL;
840 struct drm_connector *connector;
841 struct drm_connector_list_iter iter;
842 struct dc_link *link;
843 u8 link_index = 0;
844 struct drm_device *dev;
845 572
846 if (adev == NULL)573 spin_lock_init(&adev->dm.dmub_lock);
847 return;
848 574
849 if (notify == NULL) {575 if (amdgpu_dm_irq_init(adev)) {
850 drm_err(adev_to_drm(adev), "DMUB HPD callback notification was NULL");576 drm_err(adev_to_drm(adev), "failed to initialize DM IRQ support.\n");
851 return;577 goto error;
852 }578 }
853 579
854 if (notify->link_index > adev->dm.dc->link_count) {580 /* special handling for early revisions of GC 11.5.4 */
855 drm_err(adev_to_drm(adev), "DMUB HPD index (%u)is abnormal", notify->link_index);581 if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(11, 5, 4))
856 return;582 init_data.asic_id.chip_family = AMDGPU_FAMILY_GC_11_5_4;
857 }583 else
584 init_data.asic_id.chip_family = adev->family;
858 585
859 /* Skip DMUB HPD IRQ in suspend/resume. We will probe them later. */586 init_data.asic_id.pci_revision_id = adev->pdev->revision;
860 if (notify->type == DMUB_NOTIFICATION_HPD && adev->in_suspend) {587 init_data.asic_id.hw_internal_rev = adev->external_rev_id;
861 drm_info(adev_to_drm(adev), "Skip DMUB HPD IRQ callback in suspend/resume\n");588 init_data.asic_id.chip_id = adev->pdev->device;
862 return;
863 }
864 589
865 link_index = notify->link_index;590 init_data.asic_id.vram_width = adev->gmc.vram_width;
866 link = adev->dm.dc->links[link_index];591 /* TODO: initialize init_data.asic_id.vram_type here!!!! */
867 dev = adev->dm.ddev;592 init_data.asic_id.atombios_base_address =
593 adev->mode_info.atom_context->bios;
868 594
869 drm_connector_list_iter_begin(dev, &iter);595 init_data.driver = adev;
870 drm_for_each_connector_iter(connector, &iter) {
871 596
872 if (connector->connector_type == DRM_MODE_CONNECTOR_WRITEBACK)597 /* cgs_device was created in dm_sw_init() */
873 continue;598 init_data.cgs_device = adev->dm.cgs_device;
874 599
875 aconnector = to_amdgpu_dm_connector(connector);600 init_data.dce_environment = DCE_ENV_PRODUCTION_DRV;
876 if (link && aconnector->dc_link == link) {
877 if (notify->type == DMUB_NOTIFICATION_HPD)
878 drm_info(adev_to_drm(adev), "DMUB HPD IRQ callback: link_index=%u\n", link_index);
879 else if (notify->type == DMUB_NOTIFICATION_HPD_IRQ)
880 drm_info(adev_to_drm(adev), "DMUB HPD RX IRQ callback: link_index=%u\n", link_index);
881 else
882 drm_warn(adev_to_drm(adev), "DMUB Unknown HPD callback type %d, link_index=%u\n",
883 notify->type, link_index);
884 601
885 hpd_aconnector = aconnector;602 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {
603 case IP_VERSION(2, 1, 0):
604 switch (adev->dm.dmcub_fw_version) {
605 case 0: /* development */
606 case 0x1: /* linux-firmware.git hash 6d9f399 */
607 case 0x01000000: /* linux-firmware.git hash 9a0b0f4 */
608 init_data.flags.disable_dmcu = false;
886 break;609 break;
610 default:
611 init_data.flags.disable_dmcu = true;
887 }612 }
613 break;
614 case IP_VERSION(2, 0, 3):
615 init_data.flags.disable_dmcu = true;
616 break;
617 default:
618 break;
888 }619 }
889 drm_connector_list_iter_end(&iter);
890 620
891 if (hpd_aconnector) {621 /* APU support S/G display by default except:
892 if (notify->type == DMUB_NOTIFICATION_HPD) {622 * ASICs before Carrizo,
893 if (hpd_aconnector->dc_link->hpd_status == (notify->hpd_status == DP_HPD_PLUG))623 * RAVEN1 (Users reported stability issue)
894 drm_warn(adev_to_drm(adev), "DMUB reported hpd status unchanged. link_index=%u\n", link_index);624 */
895 handle_hpd_irq_helper(hpd_aconnector);
896 } else if (notify->type == DMUB_NOTIFICATION_HPD_IRQ) {
897 handle_hpd_rx_irq(hpd_aconnector);
898 }
899 }
900}
901 625
902/**626 if (adev->asic_type < CHIP_CARRIZO) {
903 * dmub_hpd_sense_callback - DMUB HPD sense processing callback.627 init_data.flags.gpu_vm_support = false;
904 * @adev: amdgpu_device pointer628 } else if (adev->asic_type == CHIP_RAVEN) {
905 * @notify: dmub notification structure629 if (adev->apu_flags & AMD_APU_IS_RAVEN)
906 *630 init_data.flags.gpu_vm_support = false;
907 * HPD sense changes can occur during low power states and need to be631 else
908 * notified from firmware to driver.632 init_data.flags.gpu_vm_support = (amdgpu_sg_display != 0);
909 */633 } else {
910static void dmub_hpd_sense_callback(struct amdgpu_device *adev,634 if (amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(2, 0, 3))
911 struct dmub_notification *notify)635 init_data.flags.gpu_vm_support = (amdgpu_sg_display == 1);
912{636 else
913 drm_dbg_driver(adev_to_drm(adev), "DMUB HPD SENSE callback.\n");637 init_data.flags.gpu_vm_support =
914}638 (amdgpu_sg_display != 0) && (adev->flags & AMD_IS_APU);
639 }
915 640
916/**641 adev->mode_info.gpu_vm_support = init_data.flags.gpu_vm_support;
917 * register_dmub_notify_callback - Sets callback for DMUB notify
918 * @adev: amdgpu_device pointer
919 * @type: Type of dmub notification
920 * @callback: Dmub interrupt callback function
921 * @dmub_int_thread_offload: offload indicator
922 *
923 * API to register a dmub callback handler for a dmub notification
924 * Also sets indicator whether callback processing to be offloaded.
925 * to dmub interrupt handling thread
926 * Return: true if successfully registered, false if there is existing registration
927 */
928static bool register_dmub_notify_callback(struct amdgpu_device *adev,
929 enum dmub_notification_type type,
930 dmub_notify_interrupt_callback_t callback,
931 bool dmub_int_thread_offload)
932{
933 if (callback != NULL && type < ARRAY_SIZE(adev->dm.dmub_thread_offload)) {
934 adev->dm.dmub_callback[type] = callback;
935 adev->dm.dmub_thread_offload[type] = dmub_int_thread_offload;
936 } else
937 return false;
938 642
939 return true;643 if (amdgpu_dc_feature_mask & DC_FBC_MASK)
940}644 init_data.flags.fbc_support = true;
941 645
942static void dm_handle_hpd_work(struct work_struct *work)646 if (amdgpu_dc_feature_mask & DC_MULTI_MON_PP_MCLK_SWITCH_MASK)
943{647 init_data.flags.multi_mon_pp_mclk_switch = true;
944 struct dmub_hpd_work *dmub_hpd_wrk;
945 648
946 dmub_hpd_wrk = container_of(work, struct dmub_hpd_work, handle_hpd_work);649 if (amdgpu_dc_feature_mask & DC_DISABLE_FRACTIONAL_PWM_MASK)
650 init_data.flags.disable_fractional_pwm = true;
947 651
948 if (!dmub_hpd_wrk->dmub_notify) {652 if (amdgpu_dc_feature_mask & DC_EDP_NO_POWER_SEQUENCING)
949 drm_err(adev_to_drm(dmub_hpd_wrk->adev), "dmub_hpd_wrk dmub_notify is NULL");653 init_data.flags.edp_no_power_sequencing = true;
950 return;
951 }
952 654
953 if (dmub_hpd_wrk->dmub_notify->type < ARRAY_SIZE(dmub_hpd_wrk->adev->dm.dmub_callback)) {655 if (amdgpu_dc_feature_mask & DC_DISABLE_LTTPR_DP1_4A)
954 dmub_hpd_wrk->adev->dm.dmub_callback[dmub_hpd_wrk->dmub_notify->type](dmub_hpd_wrk->adev,656 init_data.flags.allow_lttpr_non_transparent_mode.bits.DP1_4A = true;
955 dmub_hpd_wrk->dmub_notify);657 if (amdgpu_dc_feature_mask & DC_DISABLE_LTTPR_DP2_0)
956 }658 init_data.flags.allow_lttpr_non_transparent_mode.bits.DP2_0 = true;
957 659
958 kfree(dmub_hpd_wrk->dmub_notify);660 if (amdgpu_dc_feature_mask & DC_FRL_MASK)
959 kfree(dmub_hpd_wrk);661 init_data.flags.enable_frl = true;
960 662
961}663 init_data.flags.seamless_boot_edp_requested = false;
962 664
963static const char *dmub_notification_type_str(enum dmub_notification_type e)665 if (amdgpu_device_seamless_boot_supported(adev)) {
964{666 init_data.flags.seamless_boot_edp_requested = true;
965 switch (e) {667 init_data.flags.allow_seamless_boot_optimization = true;
966 case DMUB_NOTIFICATION_NO_DATA:668 drm_dbg(adev->dm.ddev, "Seamless boot requested\n");
967 return "NO_DATA";
968 case DMUB_NOTIFICATION_AUX_REPLY:
969 return "AUX_REPLY";
970 case DMUB_NOTIFICATION_HPD:
971 return "HPD";
972 case DMUB_NOTIFICATION_HPD_IRQ:
973 return "HPD_IRQ";
974 case DMUB_NOTIFICATION_SET_CONFIG_REPLY:
975 return "SET_CONFIG_REPLY";
976 case DMUB_NOTIFICATION_DPIA_NOTIFICATION:
977 return "DPIA_NOTIFICATION";
978 case DMUB_NOTIFICATION_HPD_SENSE_NOTIFY:
979 return "HPD_SENSE_NOTIFY";
980 case DMUB_NOTIFICATION_FUSED_IO:
981 return "FUSED_IO";
982 default:
983 return "<unknown>";
984 }669 }
985}
986
987#define DMUB_TRACE_MAX_READ 64
988/**
989 * dm_dmub_outbox1_low_irq() - Handles Outbox interrupt
990 * @interrupt_params: used for determining the Outbox instance
991 *
992 * Handles the Outbox Interrupt
993 * event handler.
994 */
995static void dm_dmub_outbox1_low_irq(void *interrupt_params)
996{
997 struct dmub_notification notify = {0};
998 struct common_irq_params *irq_params = interrupt_params;
999 struct amdgpu_device *adev = irq_params->adev;
1000 struct amdgpu_display_manager *dm = &adev->dm;
1001 struct dmcub_trace_buf_entry entry = { 0 };
1002 u32 count = 0;
1003 struct dmub_hpd_work *dmub_hpd_wrk;
1004 670
1005 do {671 init_data.flags.enable_mipi_converter_optimization = true;
1006 if (dc_dmub_srv_get_dmub_outbox0_msg(dm->dc, &entry)) {
1007 trace_amdgpu_dmub_trace_high_irq(entry.trace_code, entry.tick_count,
1008 entry.param0, entry.param1);
1009 672
1010 drm_dbg_driver(adev_to_drm(adev), "trace_code:%u, tick_count:%u, param0:%u, param1:%u\n",673 init_data.dcn_reg_offsets = adev->reg_offset[DCE_HWIP][0];
1011 entry.trace_code, entry.tick_count, entry.param0, entry.param1);674 init_data.nbio_reg_offsets = adev->reg_offset[NBIO_HWIP][0];
1012 } else675 init_data.clk_reg_offsets = adev->reg_offset[CLK_HWIP][0];
1013 break;
1014 676
1015 count++;677 if (amdgpu_dc_debug_mask & DC_DISABLE_IPS)
678 init_data.flags.disable_ips = DMUB_IPS_DISABLE_ALL;
679 else if (amdgpu_dc_debug_mask & DC_DISABLE_IPS_DYNAMIC)
680 init_data.flags.disable_ips = DMUB_IPS_DISABLE_DYNAMIC;
681 else if (amdgpu_dc_debug_mask & DC_DISABLE_IPS2_DYNAMIC)
682 init_data.flags.disable_ips = DMUB_IPS_RCG_IN_ACTIVE_IPS2_IN_OFF;
683 else if (amdgpu_dc_debug_mask & DC_FORCE_IPS_ENABLE)
684 init_data.flags.disable_ips = DMUB_IPS_ENABLE;
685 else
686 init_data.flags.disable_ips = dm_get_default_ips_mode(adev);
1016 687
1017 } while (count <= DMUB_TRACE_MAX_READ);688 init_data.flags.disable_ips_in_vpb = 0;
1018 689
1019 if (count > DMUB_TRACE_MAX_READ)690 /* DCN35 and above supports dynamic DTBCLK switch */
1020 drm_dbg_driver(adev_to_drm(adev), "Warning : count > DMUB_TRACE_MAX_READ");691 if (amdgpu_ip_version(adev, DCE_HWIP, 0) >= IP_VERSION(3, 5, 0))
692 init_data.flags.allow_0_dtb_clk = true;
1021 693
1022 if (dc_enable_dmub_notifications(adev->dm.dc) &&694 /* Enable DWB for tested platforms only */
1023 irq_params->irq_src == DC_IRQ_SOURCE_DMCUB_OUTBOX) {695 if (amdgpu_ip_version(adev, DCE_HWIP, 0) >= IP_VERSION(3, 0, 0))
696 init_data.num_virtual_links = 1;
1024 697
1025 do {698 /* DCN42 and above dpia switch to unified link training path */
1026 dc_stat_get_dmub_notification(adev->dm.dc, &notify);699 if (amdgpu_ip_version(adev, DCE_HWIP, 0) >= IP_VERSION(4, 2, 0)) {
1027 if (notify.type >= ARRAY_SIZE(dm->dmub_thread_offload)) {700 init_data.flags.consolidated_dpia_dp_lt = true;
1028 drm_err(adev_to_drm(adev), "DM: notify type %d invalid!", notify.type);701 init_data.flags.enable_dpia_pre_training = true;
1029 continue;702 init_data.flags.unify_link_enc_assignment = true;
1030 }703 init_data.flags.usb4_bw_alloc_support = true;
1031 if (!dm->dmub_callback[notify.type]) {
1032 drm_warn(adev_to_drm(adev), "DMUB notification skipped due to no handler: type=%s\n",
1033 dmub_notification_type_str(notify.type));
1034 continue;
1035 }
1036 if (dm->dmub_thread_offload[notify.type] == true) {
1037 dmub_hpd_wrk = kzalloc_obj(*dmub_hpd_wrk,
1038 GFP_ATOMIC);
1039 if (!dmub_hpd_wrk) {
1040 drm_err(adev_to_drm(adev), "Failed to allocate dmub_hpd_wrk");
1041 return;
1042 }
1043 dmub_hpd_wrk->dmub_notify = kmemdup(&notify, sizeof(struct dmub_notification),
1044 GFP_ATOMIC);
1045 if (!dmub_hpd_wrk->dmub_notify) {
1046 kfree(dmub_hpd_wrk);
1047 drm_err(adev_to_drm(adev), "Failed to allocate dmub_hpd_wrk->dmub_notify");
1048 return;
1049 }
1050 INIT_WORK(&dmub_hpd_wrk->handle_hpd_work, dm_handle_hpd_work);
1051 dmub_hpd_wrk->adev = adev;
1052 queue_work(adev->dm.delayed_hpd_wq, &dmub_hpd_wrk->handle_hpd_work);
1053 } else {
1054 dm->dmub_callback[notify.type](adev, &notify);
1055 }
1056 } while (notify.pending_notification);
1057 }704 }
1058}705 retrieve_dmi_info(&adev->dm);
706 if (adev->dm.edp0_on_dp1_quirk)
707 init_data.flags.support_edp0_on_dp1 = true;
1059 708
1060static int dm_set_clockgating_state(struct amdgpu_ip_block *ip_block,709 if (adev->dm.bb_from_dmub)
1061 enum amd_clockgating_state state)710 init_data.bb_from_dmub = adev->dm.bb_from_dmub;
1062{711 else
1063 return 0;712 init_data.bb_from_dmub = NULL;
1064}
1065
1066static int dm_set_powergating_state(struct amdgpu_ip_block *ip_block,
1067 enum amd_powergating_state state)
1068{
1069 return 0;
1070}
1071
1072/* Prototypes of private functions */
1073static int dm_early_init(struct amdgpu_ip_block *ip_block);
1074
1075/* Allocate memory for FBC compressed data */
1076static void amdgpu_dm_fbc_init(struct drm_connector *connector)
1077{
1078 struct amdgpu_device *adev = drm_to_adev(connector->dev);
1079 struct dm_compressor_info *compressor = &adev->dm.compressor;
1080 struct amdgpu_dm_connector *aconn = to_amdgpu_dm_connector(connector);
1081 struct drm_display_mode *mode;
1082 unsigned long max_size = 0;
1083
1084 if (adev->dm.dc->fbc_compressor == NULL)
1085 return;
1086
1087 if (aconn->dc_link->connector_signal != SIGNAL_TYPE_EDP)
1088 return;
1089
1090 if (compressor->bo_ptr)
1091 return;
1092 713
714 /* Display Core create. */
715 adev->dm.dc = dc_create(&init_data);
1093 716
1094 list_for_each_entry(mode, &connector->modes, head) {717 if (adev->dm.dc) {
1095 if (max_size < (unsigned long) mode->htotal * mode->vtotal)718 drm_info(adev_to_drm(adev), "Display Core v%s initialized on %s\n", DC_VER,
1096 max_size = (unsigned long) mode->htotal * mode->vtotal;719 dce_version_to_string(adev->dm.dc->ctx->dce_version));
720 } else {
721 drm_info(adev_to_drm(adev), "Display Core failed to initialize with v%s!\n", DC_VER);
722 goto error;
1097 }723 }
1098 724
1099 if (max_size) {725 if (amdgpu_dc_debug_mask & DC_DISABLE_PIPE_SPLIT) {
1100 int r = amdgpu_bo_create_kernel(adev, max_size * 4, PAGE_SIZE,726 adev->dm.dc->debug.force_single_disp_pipe_split = false;
1101 AMDGPU_GEM_DOMAIN_GTT, &compressor->bo_ptr,727 adev->dm.dc->debug.pipe_split_policy = MPC_SPLIT_AVOID;
1102 &compressor->gpu_addr, &compressor->cpu_addr);
1103
1104 if (r)
1105 drm_err(adev_to_drm(adev), "DM: Failed to initialize FBC\n");
1106 else {
1107 adev->dm.dc->ctx->fbc_gpu_addr = compressor->gpu_addr;
1108 drm_info(adev_to_drm(adev), "DM: FBC alloc %lu\n", max_size*4);
1109 }
1110
1111 }728 }
1112 729
1113}730 if (adev->asic_type != CHIP_CARRIZO && adev->asic_type != CHIP_STONEY)
1114 731 adev->dm.dc->debug.disable_stutter = amdgpu_pp_feature_mask & PP_STUTTER_MODE ? false : true;
1115static int amdgpu_dm_audio_component_get_eld(struct device *kdev, int port,732 if (dm_should_disable_stutter(adev->pdev))
1116 int pipe, bool *enabled,733 adev->dm.dc->debug.disable_stutter = true;
1117 unsigned char *buf, int max_bytes)
1118{
1119 struct drm_device *dev = dev_get_drvdata(kdev);
1120 struct amdgpu_device *adev = drm_to_adev(dev);
1121 struct drm_connector *connector;
1122 struct drm_connector_list_iter conn_iter;
1123 struct amdgpu_dm_connector *aconnector;
1124 int ret = 0;
1125
1126 *enabled = false;
1127
1128 mutex_lock(&adev->dm.audio_lock);
1129 734
1130 drm_connector_list_iter_begin(dev, &conn_iter);735 if (amdgpu_dc_debug_mask & DC_DISABLE_STUTTER)
1131 drm_for_each_connector_iter(connector, &conn_iter) {736 adev->dm.dc->debug.disable_stutter = true;
1132 737
1133 if (connector->connector_type == DRM_MODE_CONNECTOR_WRITEBACK)738 if (amdgpu_dc_debug_mask & DC_DISABLE_DSC)
1134 continue;739 adev->dm.dc->debug.disable_dsc = true;
1135 740
1136 aconnector = to_amdgpu_dm_connector(connector);741 if (amdgpu_dc_debug_mask & DC_DISABLE_CLOCK_GATING)
1137 if (aconnector->audio_inst != port)742 adev->dm.dc->debug.disable_clock_gate = true;
1138 continue;
1139 743
1140 *enabled = true;744 if (amdgpu_dc_debug_mask & DC_FORCE_SUBVP_MCLK_SWITCH)
1141 mutex_lock(&connector->eld_mutex);745 adev->dm.dc->debug.force_subvp_mclk_switch = true;
1142 ret = drm_eld_size(connector->eld);
1143 memcpy(buf, connector->eld, min(max_bytes, ret));
1144 mutex_unlock(&connector->eld_mutex);
1145 746
1146 break;747 if (amdgpu_dc_debug_mask & DC_DISABLE_SUBVP_FAMS) {
748 adev->dm.dc->debug.force_disable_subvp = true;
749 adev->dm.dc->debug.fams2_config.bits.enable = false;
1147 }750 }
1148 drm_connector_list_iter_end(&conn_iter);
1149
1150 mutex_unlock(&adev->dm.audio_lock);
1151
1152 drm_dbg_kms(adev_to_drm(adev), "Get ELD : idx=%d ret=%d en=%d\n", port, ret, *enabled);
1153
1154 return ret;
1155}
1156
1157static const struct drm_audio_component_ops amdgpu_dm_audio_component_ops = {
1158 .get_eld = amdgpu_dm_audio_component_get_eld,
1159};
1160
1161static int amdgpu_dm_audio_component_bind(struct device *kdev,
1162 struct device *hda_kdev, void *data)
1163{
1164 struct drm_device *dev = dev_get_drvdata(kdev);
1165 struct amdgpu_device *adev = drm_to_adev(dev);
1166 struct drm_audio_component *acomp = data;
1167 751
1168 acomp->ops = &amdgpu_dm_audio_component_ops;752 if (amdgpu_dc_debug_mask & DC_ENABLE_DML2) {
1169 acomp->dev = kdev;753 adev->dm.dc->debug.using_dml2 = true;
1170 adev->dm.audio_component = acomp;754 adev->dm.dc->debug.using_dml21 = true;
755 }
1171 756
1172 return 0;757 if (amdgpu_dc_debug_mask & DC_HDCP_LC_FORCE_FW_ENABLE)
1173}758 adev->dm.dc->debug.hdcp_lc_force_fw_enable = true;
1174 759
1175static void amdgpu_dm_audio_component_unbind(struct device *kdev,760 if (amdgpu_dc_debug_mask & DC_HDCP_LC_ENABLE_SW_FALLBACK)
1176 struct device *hda_kdev, void *data)761 adev->dm.dc->debug.hdcp_lc_enable_sw_fallback = true;
1177{
1178 struct amdgpu_device *adev = drm_to_adev(dev_get_drvdata(kdev));
1179 struct drm_audio_component *acomp = data;
1180 762
1181 acomp->ops = NULL;763 if (amdgpu_dc_debug_mask & DC_SKIP_DETECTION_LT)
1182 acomp->dev = NULL;764 adev->dm.dc->debug.skip_detection_link_training = true;
1183 adev->dm.audio_component = NULL;
1184}
1185 765
1186static const struct component_ops amdgpu_dm_audio_component_bind_ops = {766 adev->dm.dc->debug.visual_confirm = amdgpu_dc_visual_confirm;
1187 .bind = amdgpu_dm_audio_component_bind,
1188 .unbind = amdgpu_dm_audio_component_unbind,
1189};
1190 767
1191static int amdgpu_dm_audio_init(struct amdgpu_device *adev)768 /* TODO: Remove after DP2 receiver gets proper support of Cable ID feature */
1192{769 adev->dm.dc->debug.ignore_cable_id = true;
1193 int i, ret;
1194 770
1195 if (!amdgpu_audio)771 if (adev->dm.dc->caps.dp_hdmi21_pcon_support)
1196 return 0;772 drm_info(adev_to_drm(adev), "DP-HDMI FRL PCON supported\n");
1197 773
1198 adev->mode_info.audio.enabled = true;774 r = dm_dmub_hw_init(adev);
775 if (r) {
776 drm_err(adev_to_drm(adev), "DMUB interface failed to initialize: status=%d\n", r);
777 goto error;
778 }
1199 779
1200 adev->mode_info.audio.num_pins = adev->dm.dc->res_pool->audio_count;780 dc_hardware_init(adev->dm.dc);
1201 781
1202 for (i = 0; i < adev->mode_info.audio.num_pins; i++) {782 adev->dm.hpd_rx_offload_wq = amdgpu_dm_hpd_rx_irq_create_workqueue(adev);
1203 adev->mode_info.audio.pin[i].channels = -1;783 if (!adev->dm.hpd_rx_offload_wq) {
1204 adev->mode_info.audio.pin[i].rate = -1;784 drm_err(adev_to_drm(adev), "failed to create hpd rx offload workqueue.\n");
1205 adev->mode_info.audio.pin[i].bits_per_sample = -1;785 goto error;
1206 adev->mode_info.audio.pin[i].status_bits = 0;
1207 adev->mode_info.audio.pin[i].category_code = 0;
1208 adev->mode_info.audio.pin[i].connected = false;
1209 adev->mode_info.audio.pin[i].id =
1210 adev->dm.dc->res_pool->audios[i]->inst;
1211 adev->mode_info.audio.pin[i].offset = 0;
1212 }786 }
1213 787
1214 ret = component_add(adev->dev, &amdgpu_dm_audio_component_bind_ops);788 if ((adev->flags & AMD_IS_APU) && (adev->asic_type >= CHIP_CARRIZO)) {
1215 if (ret < 0)789 struct dc_phy_addr_space_config pa_config;
1216 return ret;
1217 790
1218 adev->dm.audio_registered = true;791 mmhub_read_system_context(adev, &pa_config);
1219 792
1220 return 0;793 // Call the DC init_memory func
1221}794 dc_setup_system_context(adev->dm.dc, &pa_config);
795 }
1222 796
1223static void amdgpu_dm_audio_fini(struct amdgpu_device *adev)797 adev->dm.freesync_module = mod_freesync_create(adev->dm.dc);
1224{798 if (!adev->dm.freesync_module) {
1225 if (!amdgpu_audio)799 drm_err(adev_to_drm(adev),
1226 return;800 "failed to initialize freesync_module.\n");
801 } else
802 drm_dbg_driver(adev_to_drm(adev), "freesync_module init done %p.\n",
803 adev->dm.freesync_module);
1227 804
1228 if (!adev->mode_info.audio.enabled)805 amdgpu_dm_init_color_mod();
1229 return;
1230 806
1231 if (adev->dm.audio_registered) {807 if (adev->dm.dc->caps.max_links > 0) {
1232 component_del(adev->dev, &amdgpu_dm_audio_component_bind_ops);808 adev->dm.vblank_control_workqueue =
1233 adev->dm.audio_registered = false;809 create_singlethread_workqueue("dm_vblank_control_workqueue");
810 if (!adev->dm.vblank_control_workqueue)
811 drm_err(adev_to_drm(adev), "failed to initialize vblank_workqueue.\n");
1234 }812 }
1235 813
1236 /* TODO: Disable audio? */814 if (adev->dm.dc->caps.ips_support &&
1237 815 adev->dm.dc->config.disable_ips != DMUB_IPS_DISABLE_ALL)
1238 adev->mode_info.audio.enabled = false;816 adev->dm.idle_workqueue = idle_create_workqueue(adev);
1239}
1240 817
1241static void amdgpu_dm_audio_eld_notify(struct amdgpu_device *adev, int pin)818 if (adev->dm.dc->caps.max_links > 0 && adev->family >= AMDGPU_FAMILY_RV) {
1242{819 adev->dm.hdcp_workqueue = hdcp_create_workqueue(adev, &init_params.cp_psp, adev->dm.dc);
1243 struct drm_audio_component *acomp = adev->dm.audio_component;
1244 820
1245 if (acomp && acomp->audio_ops && acomp->audio_ops->pin_eld_notify) {821 if (!adev->dm.hdcp_workqueue)
1246 drm_dbg_kms(adev_to_drm(adev), "Notify ELD: %d\n", pin);822 drm_err(adev_to_drm(adev), "failed to initialize hdcp_workqueue.\n");
823 else
824 drm_dbg_driver(adev_to_drm(adev),
825 "hdcp_workqueue init done %p.\n",
826 adev->dm.hdcp_workqueue);
1247 827
1248 acomp->audio_ops->pin_eld_notify(acomp->audio_ops->audio_ptr,828 dc_init_callbacks(adev->dm.dc, &init_params);
1249 pin, -1);
1250 }829 }
1251}830 if (adev->dm.dc->caps.max_links > 0) {
831 adev->dm.hdmi_frl_status_polling_wq =
832 create_singlethread_workqueue("hdmi_frl_status_polling_workqueue");
833 if (!adev->dm.hdmi_frl_status_polling_wq)
834 drm_err(adev_to_drm(adev), "failed to initialize hdmi_frl_status_polling_workqueue\n");
835 }
836 if (dc_is_dmub_outbox_supported(adev->dm.dc)) {
837 init_completion(&adev->dm.dmub_aux_transfer_done);
838 adev->dm.dmub_notify = kzalloc_obj(struct dmub_notification);
839 if (!adev->dm.dmub_notify) {
840 drm_info(adev_to_drm(adev), "fail to allocate adev->dm.dmub_notify");
841 goto error;
842 }
1252 843
1253static int dm_dmub_hw_init(struct amdgpu_device *adev)844 adev->dm.delayed_hpd_wq = create_singlethread_workqueue("amdgpu_dm_hpd_wq");
1254{845 if (!adev->dm.delayed_hpd_wq) {
1255 const struct dmcub_firmware_header_v1_0 *hdr;846 drm_err(adev_to_drm(adev), "failed to create hpd offload workqueue.\n");
1256 struct dmub_srv *dmub_srv = adev->dm.dmub_srv;847 goto error;
1257 struct dmub_srv_fb_info *fb_info = adev->dm.dmub_fb_info;848 }
1258 const struct firmware *dmub_fw = adev->dm.dmub_fw;
1259 struct dc *dc = adev->dm.dc;
1260 struct dmcu *dmcu = adev->dm.dc->res_pool->dmcu;
1261 struct abm *abm = adev->dm.dc->res_pool->abm;
1262 struct dc_context *ctx = adev->dm.dc->ctx;
1263 struct dmub_srv_hw_params hw_params;
1264 enum dmub_status status;
1265 const unsigned char *fw_inst_const, *fw_bss_data;
1266 u32 i, fw_inst_const_size, fw_bss_data_size;
1267 bool has_hw_support;
1268
1269 if (!dmub_srv)
1270 /* DMUB isn't supported on the ASIC. */
1271 return 0;
1272 849
1273 if (!fb_info) {850 amdgpu_dm_outbox_init(adev);
1274 drm_err(adev_to_drm(adev), "No framebuffer info for DMUB service.\n");851 if (!dm_register_dmub_notify_callback(adev, DMUB_NOTIFICATION_AUX_REPLY,
1275 return -EINVAL;852 dm_dmub_aux_setconfig_callback, false)) {
1276 }853 drm_err(adev_to_drm(adev), "fail to register dmub aux callback");
854 goto error;
855 }
1277 856
1278 if (!dmub_fw) {857 for (size_t i = 0; i < ARRAY_SIZE(adev->dm.fused_io); i++)
1279 /* Firmware required for DMUB support. */858 init_completion(&adev->dm.fused_io[i].replied);
1280 drm_err(adev_to_drm(adev), "No firmware provided for DMUB.\n");
1281 return -EINVAL;
1282 }
1283 859
1284 /* initialize register offsets for ASICs with runtime initialization available */860 if (!dm_register_dmub_notify_callback(adev, DMUB_NOTIFICATION_FUSED_IO,
1285 if (dmub_srv->hw_funcs.init_reg_offsets)861 dm_dmub_aux_fused_io_callback, false)) {
1286 dmub_srv->hw_funcs.init_reg_offsets(dmub_srv, ctx);862 drm_err(adev_to_drm(adev), "fail to register dmub fused io callback");
863 goto error;
864 }
865 /* Enable outbox notification only after IRQ handlers are registered and DMUB is alive.
866 * It is expected that DMUB will resend any pending notifications at this point. Note
867 * that hpd and hpd_irq handler registration are deferred to
868 * amdgpu_dm_register_hpd_handlers() to align legacy interface initialization
869 * sequence. Connection status will be proactivly detected once in the
870 * amdgpu_dm_initialize_drm_device.
871 */
872 dc_enable_dmub_outbox(adev->dm.dc);
1287 873
1288 status = dmub_srv_has_hw_support(dmub_srv, &has_hw_support);874 /* DPIA trace goes to dmesg logs only if outbox is enabled */
1289 if (status != DMUB_STATUS_OK) {875 if (amdgpu_dc_debug_mask & DC_ENABLE_DPIA_TRACE)
1290 drm_err(adev_to_drm(adev), "Error checking HW support for DMUB: %d\n", status);876 dc_dmub_srv_enable_dpia_trace(adev->dm.dc);
1291 return -EINVAL;
1292 }877 }
1293 878
1294 if (!has_hw_support) {879 if (amdgpu_dm_initialize_drm_device(adev)) {
1295 drm_info(adev_to_drm(adev), "DMUB unsupported on ASIC\n");880 drm_err(adev_to_drm(adev),
1296 return 0;881 "failed to initialize sw for display support.\n");
882 goto error;
1297 }883 }
1298 884
1299 /* Reset DMCUB if it was previously running - before we overwrite its memory. */885 if (amdgpu_dm_init_power_module(&adev->dm))
1300 status = dmub_srv_hw_reset(dmub_srv);886 goto error;
1301 if (status != DMUB_STATUS_OK)
1302 drm_warn(adev_to_drm(adev), "Error resetting DMUB HW: %d\n", status);
1303
1304 hdr = (const struct dmcub_firmware_header_v1_0 *)dmub_fw->data;
1305
1306 fw_inst_const = dmub_fw->data +
1307 le32_to_cpu(hdr->header.ucode_array_offset_bytes) +
1308 PSP_HEADER_BYTES_256;
1309 887
1310 fw_bss_data = dmub_fw->data +888 /* create fake encoders for MST */
1311 le32_to_cpu(hdr->header.ucode_array_offset_bytes) +889 dm_dp_create_fake_mst_encoders(adev);
1312 le32_to_cpu(hdr->inst_const_bytes);
1313 890
1314 /* Copy firmware and bios info into FB memory. */891 /* TODO: Add_display_info? */
1315 fw_inst_const_size = adev->dm.fw_inst_size;
1316 892
1317 fw_bss_data_size = le32_to_cpu(hdr->bss_data_bytes);893 /* TODO use dynamic cursor width */
894 adev_to_drm(adev)->mode_config.cursor_width = adev->dm.dc->caps.max_cursor_size;
895 adev_to_drm(adev)->mode_config.cursor_height = adev->dm.dc->caps.max_cursor_size;
1318 896
1319 /* if adev->firmware.load_type == AMDGPU_FW_LOAD_PSP,897 if (drm_vblank_init(adev_to_drm(adev), adev->dm.display_indexes_num)) {
1320 * amdgpu_ucode_init_single_fw will load dmub firmware898 drm_err(adev_to_drm(adev),
1321 * fw_inst_const part to cw0; otherwise, the firmware back door load899 "failed to initialize vblank for display support.\n");
1322 * will be done by dm_dmub_hw_init900 goto error;
1323 */
1324 if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) {
1325 memcpy(fb_info->fb[DMUB_WINDOW_0_INST_CONST].cpu_addr, fw_inst_const,
1326 fw_inst_const_size);
1327 }901 }
1328 902
1329 if (fw_bss_data_size)903#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)
1330 memcpy(fb_info->fb[DMUB_WINDOW_2_BSS_DATA].cpu_addr,904 amdgpu_dm_crtc_secure_display_create_contexts(adev);
1331 fw_bss_data, fw_bss_data_size);905 if (!adev->dm.secure_display_ctx.crtc_ctx)
906 drm_err(adev_to_drm(adev), "failed to initialize secure display contexts.\n");
1332 907
1333 /* Copy firmware bios info into FB memory. */908 if (amdgpu_ip_version(adev, DCE_HWIP, 0) >= IP_VERSION(4, 0, 1))
1334 memcpy(fb_info->fb[DMUB_WINDOW_3_VBIOS].cpu_addr, adev->bios,909 adev->dm.secure_display_ctx.support_mul_roi = true;
1335 adev->bios_size);
1336 910
1337 /* Reset regions that need to be reset. */911#endif
1338 memset(fb_info->fb[DMUB_WINDOW_4_MAILBOX].cpu_addr, 0,
1339 fb_info->fb[DMUB_WINDOW_4_MAILBOX].size);
1340 912
1341 memset(fb_info->fb[DMUB_WINDOW_5_TRACEBUFF].cpu_addr, 0,913 drm_dbg_driver(adev_to_drm(adev), "KMS initialized.\n");
1342 fb_info->fb[DMUB_WINDOW_5_TRACEBUFF].size);
1343 914
1344 memset(fb_info->fb[DMUB_WINDOW_6_FW_STATE].cpu_addr, 0,915 return 0;
1345 fb_info->fb[DMUB_WINDOW_6_FW_STATE].size);916error:
917 amdgpu_dm_fini(adev);
1346 918
1347 memset(fb_info->fb[DMUB_WINDOW_SHARED_STATE].cpu_addr, 0,919 return -EINVAL;
1348 fb_info->fb[DMUB_WINDOW_SHARED_STATE].size);920}
1349 921
1350 /* Initialize hardware. */922static int amdgpu_dm_early_fini(struct amdgpu_ip_block *ip_block)
1351 memset(&hw_params, 0, sizeof(hw_params));923{
1352 hw_params.soc_fb_info.fb_base = adev->gmc.fb_start;924 struct amdgpu_device *adev = ip_block->adev;
1353 hw_params.soc_fb_info.fb_offset = adev->vm_manager.vram_base_offset;
1354 925
1355 /* backdoor load firmware and trigger dmub running */926 amdgpu_dm_audio_fini(adev);
1356 if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP)
1357 hw_params.load_inst_const = true;
1358 927
1359 if (dmcu)928 return 0;
1360 hw_params.psp_version = dmcu->psp_version;929}
1361 930
1362 for (i = 0; i < fb_info->num_fb; ++i)931static void amdgpu_dm_fini(struct amdgpu_device *adev)
1363 hw_params.fb[i] = &fb_info->fb[i];932{
933 int i;
1364 934
1365 /* Enable usb4 dpia in the FW APU */935 if (adev->dm.vblank_control_workqueue) {
1366 if (dc->caps.is_apu &&936 destroy_workqueue(adev->dm.vblank_control_workqueue);
1367 dc->res_pool->usb4_dpia_count != 0 &&937 adev->dm.vblank_control_workqueue = NULL;
1368 !dc->debug.dpia_debug.bits.disable_dpia) {
1369 hw_params.dpia_supported = true;
1370 hw_params.disable_dpia = dc->debug.dpia_debug.bits.disable_dpia;
1371 hw_params.dpia_hpd_int_enable_supported = false;
1372 hw_params.enable_non_transparent_setconfig = dc->config.consolidated_dpia_dp_lt;
1373 hw_params.disable_dpia_bw_allocation = !dc->config.usb4_bw_alloc_support;
1374 }938 }
1375 939
1376 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {940 if (adev->dm.idle_workqueue) {
1377 case IP_VERSION(3, 5, 0):941 if (adev->dm.idle_workqueue->running) {
1378 case IP_VERSION(3, 5, 1):942 adev->dm.idle_workqueue->enable = false;
1379 case IP_VERSION(3, 6, 0):943 flush_work(&adev->dm.idle_workqueue->work);
1380 case IP_VERSION(4, 2, 0):944 }
1381 case IP_VERSION(4, 2, 1):945
1382 hw_params.ips_sequential_ono = adev->external_rev_id > 0x10;946 kfree(adev->dm.idle_workqueue);
1383 hw_params.lower_hbr3_phy_ssc = true;947 adev->dm.idle_workqueue = NULL;
1384 break;
1385 default:
1386 break;
1387 }948 }
1388 949
1389 status = dmub_srv_hw_init(dmub_srv, &hw_params);950 /*
1390 if (status != DMUB_STATUS_OK) {951 * Disable ISM before dc_destroy() invalidates dm->dc.
1391 drm_err(adev_to_drm(adev), "Error initializing DMUB HW: %d\n", status);952 *
1392 return -EINVAL;953 * Quiesce workers first without dc_lock (they take dc_lock
1393 }954 * themselves, so syncing under it would deadlock), then drive the
955 * FSM back to FULL_POWER_RUNNING under dc_lock.
956 */
957 amdgpu_dm_ism_disable(&adev->dm);
958 scoped_guard(mutex, &adev->dm.dc_lock)
959 amdgpu_dm_ism_force_full_power(&adev->dm);
1394 960
1395 /* Wait for firmware load to finish. */961 amdgpu_dm_destroy_drm_device(&adev->dm);
1396 status = dmub_srv_wait_for_auto_load(dmub_srv, 100000);
1397 if (status != DMUB_STATUS_OK)
1398 drm_warn(adev_to_drm(adev), "Wait for DMUB auto-load failed: %d\n", status);
1399 962
1400 /* Init DMCU and ABM if available. */963#if defined(CONFIG_DRM_AMD_SECURE_DISPLAY)
1401 if (dmcu && abm) {964 if (adev->dm.secure_display_ctx.crtc_ctx) {
1402 dmcu->funcs->dmcu_init(dmcu);965 for (i = 0; i < adev->mode_info.num_crtc; i++) {
1403 abm->dmcu_is_running = dmcu->funcs->is_dmcu_initialized(dmcu);966 if (adev->dm.secure_display_ctx.crtc_ctx[i].crtc) {
967 flush_work(&adev->dm.secure_display_ctx.crtc_ctx[i].notify_ta_work);
968 flush_work(&adev->dm.secure_display_ctx.crtc_ctx[i].forward_roi_work);
969 }
970 }
971 kfree(adev->dm.secure_display_ctx.crtc_ctx);
972 adev->dm.secure_display_ctx.crtc_ctx = NULL;
973 }
974#endif
975 if (adev->dm.hdcp_workqueue) {
976 hdcp_destroy(&adev->dev->kobj, adev->dm.hdcp_workqueue);
977 adev->dm.hdcp_workqueue = NULL;
1404 }978 }
1405 979
1406 if (!adev->dm.dc->ctx->dmub_srv)980 if (adev->dm.dc) {
1407 adev->dm.dc->ctx->dmub_srv = dc_dmub_srv_create(adev->dm.dc, dmub_srv);981 dc_deinit_callbacks(adev->dm.dc);
1408 if (!adev->dm.dc->ctx->dmub_srv) {982 dc_dmub_srv_destroy(&adev->dm.dc->ctx->dmub_srv);
1409 drm_err(adev_to_drm(adev), "Couldn't allocate DC DMUB server!\n");983 if (dc_enable_dmub_notifications(adev->dm.dc)) {
1410 return -ENOMEM;984 kfree(adev->dm.dmub_notify);
985 adev->dm.dmub_notify = NULL;
986 destroy_workqueue(adev->dm.delayed_hpd_wq);
987 adev->dm.delayed_hpd_wq = NULL;
988 }
1411 }989 }
1412 990
1413 drm_info(adev_to_drm(adev), "DMUB hardware initialized: version=0x%08X\n",991 if (adev->dm.dmub_bo)
1414 adev->dm.dmcub_fw_version);992 amdgpu_bo_free_kernel(&adev->dm.dmub_bo,
993 &adev->dm.dmub_bo_gpu_addr,
994 &adev->dm.dmub_bo_cpu_addr);
995
996 if (adev->dm.boot_time_crc_info.bo_ptr)
997 amdgpu_bo_free_kernel(&adev->dm.boot_time_crc_info.bo_ptr,
998 &adev->dm.boot_time_crc_info.gpu_addr,
999 &adev->dm.boot_time_crc_info.cpu_addr);
1000
1001 if (adev->dm.hpd_rx_offload_wq && adev->dm.dc) {
1002 for (i = 0; i < adev->dm.dc->caps.max_links; i++) {
1003 if (adev->dm.hpd_rx_offload_wq[i].wq) {
1004 destroy_workqueue(adev->dm.hpd_rx_offload_wq[i].wq);
1005 adev->dm.hpd_rx_offload_wq[i].wq = NULL;
1006 }
1007 }
1008
1009 kfree(adev->dm.hpd_rx_offload_wq);
1010 adev->dm.hpd_rx_offload_wq = NULL;
1011 }
1415 1012
1416 /* Keeping sanity checks off if1013 /* DC Destroy TODO: Replace destroy DAL */
1417 * DCN31 >= 4.0.59.01014 if (adev->dm.dc)
1418 * DCN314 >= 8.0.16.01015 dc_destroy(&adev->dm.dc);
1419 * Otherwise, turn on sanity checks1016 /*
1017 * TODO: pageflip, vlank interrupt
1018 *
1019 * amdgpu_dm_irq_fini(adev);
1420 */1020 */
1421 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {1021
1422 case IP_VERSION(3, 1, 2):1022 if (adev->dm.cgs_device) {
1423 case IP_VERSION(3, 1, 3):1023 amdgpu_cgs_destroy_device(adev->dm.cgs_device);
1424 if (adev->dm.dmcub_fw_version &&1024 adev->dm.cgs_device = NULL;
1425 adev->dm.dmcub_fw_version >= DMUB_FW_VERSION(4, 0, 0) &&1025 }
1426 adev->dm.dmcub_fw_version < DMUB_FW_VERSION(4, 0, 59))1026 if (adev->dm.freesync_module) {
1427 adev->dm.dc->debug.sanity_checks = true;1027 mod_freesync_destroy(adev->dm.freesync_module);
1028 adev->dm.freesync_module = NULL;
1029 }
1030
1031 if (adev->dm.power_module) {
1032 mod_power_destroy(adev->dm.power_module);
1033 adev->dm.power_module = NULL;
1034 }
1035 mutex_destroy(&adev->dm.audio_lock);
1036 mutex_destroy(&adev->dm.dc_lock);
1037 mutex_destroy(&adev->dm.dpia_aux_lock);
1038}
1039
1040static int load_dmcu_fw(struct amdgpu_device *adev)
1041{
1042 const char *fw_name_dmcu = NULL;
1043 int r;
1044 const struct dmcu_firmware_header_v1_0 *hdr;
1045
1046 switch (adev->asic_type) {
1047#if defined(CONFIG_DRM_AMD_DC_SI)
1048 case CHIP_TAHITI:
1049 case CHIP_PITCAIRN:
1050 case CHIP_VERDE:
1051 case CHIP_OLAND:
1052#endif
1053 case CHIP_BONAIRE:
1054 case CHIP_HAWAII:
1055 case CHIP_KAVERI:
1056 case CHIP_KABINI:
1057 case CHIP_MULLINS:
1058 case CHIP_TONGA:
1059 case CHIP_FIJI:
1060 case CHIP_CARRIZO:
1061 case CHIP_STONEY:
1062 case CHIP_POLARIS11:
1063 case CHIP_POLARIS10:
1064 case CHIP_POLARIS12:
1065 case CHIP_VEGAM:
1066 case CHIP_VEGA10:
1067 case CHIP_VEGA12:
1068 case CHIP_VEGA20:
1069 return 0;
1070 case CHIP_NAVI12:
1071 fw_name_dmcu = FIRMWARE_NAVI12_DMCU;
1428 break;1072 break;
1429 case IP_VERSION(3, 1, 4):1073 case CHIP_RAVEN:
1430 if (adev->dm.dmcub_fw_version &&1074 if (ASICREV_IS_PICASSO(adev->external_rev_id))
1431 adev->dm.dmcub_fw_version >= DMUB_FW_VERSION(4, 0, 0) &&1075 fw_name_dmcu = FIRMWARE_RAVEN_DMCU;
1432 adev->dm.dmcub_fw_version < DMUB_FW_VERSION(8, 0, 16))1076 else if (ASICREV_IS_RAVEN2(adev->external_rev_id))
1433 adev->dm.dc->debug.sanity_checks = true;1077 fw_name_dmcu = FIRMWARE_RAVEN_DMCU;
1078 else
1079 return 0;
1434 break;1080 break;
1435 default:1081 default:
1436 break;1082 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {
1083 case IP_VERSION(2, 0, 2):
1084 case IP_VERSION(2, 0, 3):
1085 case IP_VERSION(2, 0, 0):
1086 case IP_VERSION(2, 1, 0):
1087 case IP_VERSION(3, 0, 0):
1088 case IP_VERSION(3, 0, 2):
1089 case IP_VERSION(3, 0, 3):
1090 case IP_VERSION(3, 0, 1):
1091 case IP_VERSION(3, 1, 2):
1092 case IP_VERSION(3, 1, 3):
1093 case IP_VERSION(3, 1, 4):
1094 case IP_VERSION(3, 1, 5):
1095 case IP_VERSION(3, 1, 6):
1096 case IP_VERSION(3, 2, 0):
1097 case IP_VERSION(3, 2, 1):
1098 case IP_VERSION(3, 5, 0):
1099 case IP_VERSION(3, 5, 1):
1100 case IP_VERSION(3, 6, 0):
1101 case IP_VERSION(4, 0, 1):
1102 case IP_VERSION(4, 2, 0):
1103 case IP_VERSION(4, 2, 1):
1104 return 0;
1105 default:
1106 break;
1107 }
1108 drm_err(adev_to_drm(adev), "Unsupported ASIC type: 0x%X\n", adev->asic_type);
1109 return -EINVAL;
1110 }
1111
1112 if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) {
1113 drm_dbg_kms(adev_to_drm(adev), "dm: DMCU firmware not supported on direct or SMU loading\n");
1114 return 0;
1115 }
1116
1117 r = amdgpu_ucode_request(adev, &adev->dm.fw_dmcu, AMDGPU_UCODE_REQUIRED,
1118 "%s", fw_name_dmcu);
1119 if (r == -ENODEV) {
1120 /* DMCU firmware is not necessary, so don't raise a fuss if it's missing */
1121 drm_dbg_kms(adev_to_drm(adev), "dm: DMCU firmware not found\n");
1122 adev->dm.fw_dmcu = NULL;
1123 return 0;
1124 }
1125 if (r) {
1126 drm_err(adev_to_drm(adev), "amdgpu_dm: Can't validate firmware \"%s\"\n",
1127 fw_name_dmcu);
1128 amdgpu_ucode_release(&adev->dm.fw_dmcu);
1129 return r;
1437 }1130 }
1438 1131
1132 hdr = (const struct dmcu_firmware_header_v1_0 *)adev->dm.fw_dmcu->data;
1133 adev->firmware.ucode[AMDGPU_UCODE_ID_DMCU_ERAM].ucode_id = AMDGPU_UCODE_ID_DMCU_ERAM;
1134 adev->firmware.ucode[AMDGPU_UCODE_ID_DMCU_ERAM].fw = adev->dm.fw_dmcu;
1135 adev->firmware.fw_size +=
1136 ALIGN(le32_to_cpu(hdr->header.ucode_size_bytes) - le32_to_cpu(hdr->intv_size_bytes), PAGE_SIZE);
1137
1138 adev->firmware.ucode[AMDGPU_UCODE_ID_DMCU_INTV].ucode_id = AMDGPU_UCODE_ID_DMCU_INTV;
1139 adev->firmware.ucode[AMDGPU_UCODE_ID_DMCU_INTV].fw = adev->dm.fw_dmcu;
1140 adev->firmware.fw_size +=
1141 ALIGN(le32_to_cpu(hdr->intv_size_bytes), PAGE_SIZE);
1142
1143 adev->dm.dmcu_fw_version = le32_to_cpu(hdr->header.ucode_version);
1144
1145 drm_dbg_kms(adev_to_drm(adev), "PSP loading DMCU firmware\n");
1146
1439 return 0;1147 return 0;
1440}1148}
1441 1149
1442static void dm_dmub_hw_resume(struct amdgpu_device *adev)1150static int dm_sw_init(struct amdgpu_ip_block *ip_block)
1443{1151{
1444 struct dmub_srv *dmub_srv = adev->dm.dmub_srv;1152 struct amdgpu_device *adev = ip_block->adev;
1445 enum dmub_status status;
1446 bool init;
1447 int r;1153 int r;
1448 1154
1449 if (!dmub_srv) {1155 adev->dm.cgs_device = amdgpu_cgs_create_device(adev);
1450 /* DMUB isn't supported on the ASIC. */1156
1451 return;1157 if (!adev->dm.cgs_device) {
1158 drm_err(adev_to_drm(adev), "failed to create cgs device.\n");
1159 return -EINVAL;
1452 }1160 }
1453 1161
1454 status = dmub_srv_is_hw_init(dmub_srv, &init);1162 /* Moved from dm init since we need to use allocations for storing bounding box data */
1455 if (status != DMUB_STATUS_OK)1163 INIT_LIST_HEAD(&adev->dm.da_list);
1456 drm_warn(adev_to_drm(adev), "DMUB hardware init check failed: %d\n", status);
1457 1164
1458 if (status == DMUB_STATUS_OK && init) {1165 r = dm_dmub_sw_init(adev);
1459 /* Wait for firmware load to finish. */1166 if (r)
1460 status = dmub_srv_wait_for_auto_load(dmub_srv, 100000);1167 return r;
1461 if (status != DMUB_STATUS_OK)1168
1462 drm_warn(adev_to_drm(adev), "Wait for DMUB auto-load failed: %d\n", status);1169 return load_dmcu_fw(adev);
1463 } else {
1464 /* Perform the full hardware initialization. */
1465 r = dm_dmub_hw_init(adev);
1466 if (r)
1467 drm_err(adev_to_drm(adev), "DMUB interface failed to initialize: status=%d\n", r);
1468 }
1469}1170}
1470 1171
1471static void mmhub_read_system_context(struct amdgpu_device *adev, struct dc_phy_addr_space_config *pa_config)1172static int dm_sw_fini(struct amdgpu_ip_block *ip_block)
1472{1173{
1473 u64 pt_base;1174 struct amdgpu_device *adev = ip_block->adev;
1474 u32 logical_addr_low;1175 struct dal_allocation *da;
1475 u32 logical_addr_high;
1476 u32 agp_base, agp_bot, agp_top;
1477 PHYSICAL_ADDRESS_LOC page_table_start, page_table_end, page_table_base;
1478 1176
1479 memset(pa_config, 0, sizeof(*pa_config));1177 list_for_each_entry(da, &adev->dm.da_list, list) {
1178 if (adev->dm.bb_from_dmub == (void *) da->cpu_ptr) {
1179 amdgpu_bo_free_kernel(&da->bo, &da->gpu_addr, &da->cpu_ptr);
1180 list_del(&da->list);
1181 kfree(da);
1182 adev->dm.bb_from_dmub = NULL;
1183 break;
1184 }
1185 }
1480 1186
1481 agp_base = 0;
1482 agp_bot = adev->gmc.agp_start >> 24;
1483 agp_top = adev->gmc.agp_end >> 24;
1484 1187
1485 /* AGP aperture is disabled */1188 kfree(adev->dm.dmub_fb_info);
1486 if (agp_bot > agp_top) {1189 adev->dm.dmub_fb_info = NULL;
1487 logical_addr_low = adev->gmc.fb_start >> 18;
1488 if (adev->apu_flags & (AMD_APU_IS_RAVEN2 |
1489 AMD_APU_IS_RENOIR |
1490 AMD_APU_IS_GREEN_SARDINE))
1491 /*
1492 * Raven2 has a HW issue that it is unable to use the vram which
1493 * is out of MC_VM_SYSTEM_APERTURE_HIGH_ADDR. So here is the
1494 * workaround that increase system aperture high address (add 1)
1495 * to get rid of the VM fault and hardware hang.
1496 */
1497 logical_addr_high = (adev->gmc.fb_end >> 18) + 0x1;
1498 else
1499 logical_addr_high = adev->gmc.fb_end >> 18;
1500 } else {
1501 logical_addr_low = min(adev->gmc.fb_start, adev->gmc.agp_start) >> 18;
1502 if (adev->apu_flags & (AMD_APU_IS_RAVEN2 |
1503 AMD_APU_IS_RENOIR |
1504 AMD_APU_IS_GREEN_SARDINE))
1505 /*
1506 * Raven2 has a HW issue that it is unable to use the vram which
1507 * is out of MC_VM_SYSTEM_APERTURE_HIGH_ADDR. So here is the
1508 * workaround that increase system aperture high address (add 1)
1509 * to get rid of the VM fault and hardware hang.
1510 */
1511 logical_addr_high = max((adev->gmc.fb_end >> 18) + 0x1, adev->gmc.agp_end >> 18);
1512 else
1513 logical_addr_high = max(adev->gmc.fb_end, adev->gmc.agp_end) >> 18;
1514 }
1515
1516 pt_base = amdgpu_gmc_pd_addr(adev->gart.bo);
1517
1518 page_table_start.high_part = upper_32_bits(adev->gmc.gart_start >>
1519 AMDGPU_GPU_PAGE_SHIFT);
1520 page_table_start.low_part = lower_32_bits(adev->gmc.gart_start >>
1521 AMDGPU_GPU_PAGE_SHIFT);
1522 page_table_end.high_part = upper_32_bits(adev->gmc.gart_end >>
1523 AMDGPU_GPU_PAGE_SHIFT);
1524 page_table_end.low_part = lower_32_bits(adev->gmc.gart_end >>
1525 AMDGPU_GPU_PAGE_SHIFT);
1526 page_table_base.high_part = upper_32_bits(pt_base);
1527 page_table_base.low_part = lower_32_bits(pt_base);
1528
1529 pa_config->system_aperture.start_addr = (uint64_t)logical_addr_low << 18;
1530 pa_config->system_aperture.end_addr = (uint64_t)logical_addr_high << 18;
1531
1532 pa_config->system_aperture.agp_base = (uint64_t)agp_base << 24;
1533 pa_config->system_aperture.agp_bot = (uint64_t)agp_bot << 24;
1534 pa_config->system_aperture.agp_top = (uint64_t)agp_top << 24;
1535
1536 pa_config->system_aperture.fb_base = adev->gmc.fb_start;
1537 pa_config->system_aperture.fb_offset = adev->vm_manager.vram_base_offset;
1538 pa_config->system_aperture.fb_top = adev->gmc.fb_end;
1539 1190
1540 pa_config->gart_config.page_table_start_addr = page_table_start.quad_part << 12;1191 if (adev->dm.dmub_srv) {
1541 pa_config->gart_config.page_table_end_addr = page_table_end.quad_part << 12;1192 dmub_srv_destroy(adev->dm.dmub_srv);
1542 pa_config->gart_config.page_table_base_addr = page_table_base.quad_part;1193 kfree(adev->dm.dmub_srv);
1194 adev->dm.dmub_srv = NULL;
1195 }
1543 1196
1544 pa_config->is_hvm_enabled = adev->mode_info.gpu_vm_support;1197 amdgpu_ucode_release(&adev->dm.dmub_fw);
1198 amdgpu_ucode_release(&adev->dm.fw_dmcu);
1545 1199
1200 return 0;
1546}1201}
1547 1202
1548static void force_connector_state(
1549 struct amdgpu_dm_connector *aconnector,
1550 enum drm_connector_force force_state)
1551{
1552 struct drm_connector *connector = &aconnector->base;
1553
1554 mutex_lock(&connector->dev->mode_config.mutex);
1555 aconnector->base.force = force_state;
1556 mutex_unlock(&connector->dev->mode_config.mutex);
1557 1203
1558 mutex_lock(&aconnector->hpd_lock);1204static void amdgpu_dm_boot_time_crc_init(struct amdgpu_device *adev)
1559 drm_kms_helper_connector_hotplug_event(connector);
1560 mutex_unlock(&aconnector->hpd_lock);
1561}
1562
1563static void dm_handle_hpd_rx_offload_work(struct work_struct *work)
1564{1205{
1565 struct hpd_rx_irq_offload_work *offload_work;1206 struct dm_boot_time_crc_info *bootcrc_info = NULL;
1566 struct amdgpu_dm_connector *aconnector;1207 struct dmub_srv *dmub = NULL;
1567 struct dc_link *dc_link;1208 union dmub_fw_boot_options option = {0};
1568 struct amdgpu_device *adev;1209 int ret = 0;
1569 enum dc_connection_type new_connection_type = dc_connection_none;1210 const uint32_t fb_size = 3 * 1024 * 1024; /* 3MB for DCC pattern */
1570 unsigned long flags;
1571 union test_response test_response;
1572
1573 memset(&test_response, 0, sizeof(test_response));
1574
1575 offload_work = container_of(work, struct hpd_rx_irq_offload_work, work);
1576 aconnector = offload_work->offload_wq->aconnector;
1577 adev = offload_work->adev;
1578 1211
1579 if (!aconnector) {1212 if (!adev || !adev->dm.dc || !adev->dm.dc->ctx ||
1580 drm_err(adev_to_drm(adev), "Can't retrieve aconnector in hpd_rx_irq_offload_work");1213 !adev->dm.dc->ctx->dmub_srv) {
1581 goto skip;1214 return;
1582 }1215 }
1583 1216
1584 dc_link = aconnector->dc_link;1217 dmub = adev->dm.dc->ctx->dmub_srv->dmub;
1585 1218 bootcrc_info = &adev->dm.boot_time_crc_info;
1586 mutex_lock(&aconnector->hpd_lock);
1587 if (!dc_link_detect_connection_type(dc_link, &new_connection_type))
1588 drm_err(adev_to_drm(adev), "KMS: Failed to detect connector\n");
1589 mutex_unlock(&aconnector->hpd_lock);
1590
1591 if (new_connection_type == dc_connection_none)
1592 goto skip;
1593
1594 if (amdgpu_in_reset(adev))
1595 goto skip;
1596 1219
1597 if (offload_work->data.bytes.device_service_irq.bits.UP_REQ_MSG_RDY ||1220 if (!dmub || !dmub->hw_funcs.get_fw_boot_option) {
1598 offload_work->data.bytes.device_service_irq.bits.DOWN_REP_MSG_RDY) {1221 drm_dbg(adev_to_drm(adev), "failed to init boot time crc buffer\n");
1599 dm_handle_mst_sideband_msg_ready_event(&aconnector->mst_mgr, DOWN_OR_UP_MSG_RDY_EVENT);1222 return;
1600 spin_lock_irqsave(&offload_work->offload_wq->offload_lock, flags);
1601 offload_work->offload_wq->is_handling_mst_msg_rdy_event = false;
1602 spin_unlock_irqrestore(&offload_work->offload_wq->offload_lock, flags);
1603 goto skip;
1604 }1223 }
1605 1224
1606 mutex_lock(&adev->dm.dc_lock);1225 option = dmub->hw_funcs.get_fw_boot_option(dmub);
1607 if (offload_work->data.bytes.device_service_irq.bits.AUTOMATED_TEST) {
1608 dc_link_dp_handle_automated_test(dc_link);
1609
1610 if (aconnector->timing_changed) {
1611 /* force connector disconnect and reconnect */
1612 force_connector_state(aconnector, DRM_FORCE_OFF);
1613 msleep(100);
1614 force_connector_state(aconnector, DRM_FORCE_UNSPECIFIED);
1615 }
1616
1617 test_response.bits.ACK = 1;
1618
1619 core_link_write_dpcd(
1620 dc_link,
1621 DP_TEST_RESPONSE,
1622 &test_response.raw,
1623 sizeof(test_response));
1624 } else if ((dc_link->connector_signal != SIGNAL_TYPE_EDP) &&
1625 dc_link_check_link_loss_status(dc_link, &offload_work->data) &&
1626 dc_link_dp_allow_hpd_rx_irq(dc_link)) {
1627 /* offload_work->data is from handle_hpd_rx_irq->
1628 * schedule_hpd_rx_offload_work.this is defer handle
1629 * for hpd short pulse. upon here, link status may be
1630 * changed, need get latest link status from dpcd
1631 * registers. if link status is good, skip run link
1632 * training again.
1633 */
1634 union hpd_irq_data irq_data;
1635 1226
1636 memset(&irq_data, 0, sizeof(irq_data));1227 /* Return if boot time CRC is not enabled */
1228 if (option.bits.bootcrc_en_at_S0i3 == 0)
1229 return;
1637 1230
1638 /* before dc_link_dp_handle_link_loss, allow new link lost handle1231 /* Create a buffer for boot time CRC */
1639 * request be added to work queue if link lost at end of dc_link_1232 ret = amdgpu_bo_create_kernel(adev, fb_size, PAGE_SIZE,
1640 * dp_handle_link_loss1233 AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT,
1641 */1234 &bootcrc_info->bo_ptr,
1642 spin_lock_irqsave(&offload_work->offload_wq->offload_lock, flags);1235 &bootcrc_info->gpu_addr,
1643 offload_work->offload_wq->is_handling_link_loss = false;1236 &bootcrc_info->cpu_addr);
1644 spin_unlock_irqrestore(&offload_work->offload_wq->offload_lock, flags);
1645 1237
1646 if ((dc_link_dp_read_hpd_rx_irq_data(dc_link, &irq_data) == DC_OK) &&1238 if (ret) {
1647 dc_link_check_link_loss_status(dc_link, &irq_data))1239 drm_dbg(adev_to_drm(adev), "failed to create boot time crc buffer\n");
1648 dc_link_dp_handle_link_loss(dc_link);1240 } else {
1649 }1241 bootcrc_info->size = fb_size;
1650 mutex_unlock(&adev->dm.dc_lock);
1651 1242
1652skip:1243 drm_dbg(adev_to_drm(adev), "boot time crc buffer created addr 0x%llx, size %u\n",
1653 kfree(offload_work);1244 bootcrc_info->gpu_addr, bootcrc_info->size);
1654 1245
1246 /* Send the buffer info to DMUB */
1247 dc_dmub_srv_boot_time_crc_init(adev->dm.dc,
1248 bootcrc_info->gpu_addr, bootcrc_info->size);
1249 }
1655}1250}
1656 1251
1657static struct hpd_rx_irq_offload_work_queue *hpd_rx_irq_create_workqueue(struct amdgpu_device *adev)1252static int dm_late_init(struct amdgpu_ip_block *ip_block)
1658{1253{
1659 struct dc *dc = adev->dm.dc;1254 struct amdgpu_device *adev = ip_block->adev;
1660 int max_caps = dc->caps.max_links;
1661 int i = 0;
1662 struct hpd_rx_irq_offload_work_queue *hpd_rx_offload_wq = NULL;
1663
1664 hpd_rx_offload_wq = kzalloc_objs(*hpd_rx_offload_wq, max_caps);
1665
1666 if (!hpd_rx_offload_wq)
1667 return NULL;
1668
1669
1670 for (i = 0; i < max_caps; i++) {
1671 hpd_rx_offload_wq[i].wq =
1672 create_singlethread_workqueue("amdgpu_dm_hpd_rx_offload_wq");
1673
1674 if (hpd_rx_offload_wq[i].wq == NULL) {
1675 drm_err(adev_to_drm(adev), "create amdgpu_dm_hpd_rx_offload_wq fail!");
1676 goto out_err;
1677 }
1678 1255
1679 spin_lock_init(&hpd_rx_offload_wq[i].offload_lock);1256 struct dmcu_iram_parameters params;
1680 }1257 unsigned int linear_lut[16];
1258 int i;
1259 struct dmcu *dmcu = NULL;
1681 1260
1682 return hpd_rx_offload_wq;1261 dmcu = adev->dm.dc->res_pool->dmcu;
1683 1262
1684out_err:1263 /* Init the boot time CRC (skip in resume) */
1685 for (i = 0; i < max_caps; i++) {1264 if ((adev->in_suspend == 0) &&
1686 if (hpd_rx_offload_wq[i].wq)1265 (amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(3, 6, 0)))
1687 destroy_workqueue(hpd_rx_offload_wq[i].wq);1266 amdgpu_dm_boot_time_crc_init(adev);
1688 }
1689 kfree(hpd_rx_offload_wq);
1690 return NULL;
1691}
1692 1267
1693struct amdgpu_stutter_quirk {1268 for (i = 0; i < 16; i++)
1694 u16 chip_vendor;1269 linear_lut[i] = 0xFFFF * i / 15;
1695 u16 chip_device;
1696 u16 subsys_vendor;
1697 u16 subsys_device;
1698 u8 revision;
1699};
1700 1270
1701static const struct amdgpu_stutter_quirk amdgpu_stutter_quirk_list[] = {1271 params.set = 0;
1702 /* https://bugzilla.kernel.org/show_bug.cgi?id=214417 */1272 params.backlight_ramping_override = false;
1703 { 0x1002, 0x15dd, 0x1002, 0x15dd, 0xc8 },1273 params.backlight_ramping_start = 0xCCCC;
1704 { 0, 0, 0, 0, 0 },1274 params.backlight_ramping_reduction = 0xCCCCCCCC;
1705};1275 params.backlight_lut_array_size = 16;
1276 params.backlight_lut_array = linear_lut;
1706 1277
1707static bool dm_should_disable_stutter(struct pci_dev *pdev)1278 /* Min backlight level after ABM reduction, Don't allow below 1%
1708{1279 * 0xFFFF x 0.01 = 0x28F
1709 const struct amdgpu_stutter_quirk *p = amdgpu_stutter_quirk_list;1280 */
1281 params.min_abm_backlight = 0x28F;
1282 /* In the case where abm is implemented on dmcub,
1283 * dmcu object will be null.
1284 * ABM 2.4 and up are implemented on dmcub.
1285 */
1286 if (dmcu) {
1287 if (!dmcu_load_iram(dmcu, params))
1288 return -EINVAL;
1289 } else if (adev->dm.dc->ctx->dmub_srv) {
1290 struct dc_link *edp_links[MAX_NUM_EDP];
1291 int edp_num;
1710 1292
1711 while (p && p->chip_device != 0) {1293 dc_get_edp_links(adev->dm.dc, edp_links, &edp_num);
1712 if (pdev->vendor == p->chip_vendor &&1294 for (i = 0; i < edp_num; i++) {
1713 pdev->device == p->chip_device &&1295 if (!dmub_init_abm_config(adev->dm.dc->res_pool, params, i))
1714 pdev->subsystem_vendor == p->subsys_vendor &&1296 return -EINVAL;
1715 pdev->subsystem_device == p->subsys_device &&
1716 pdev->revision == p->revision) {
1717 return true;
1718 }1297 }
1719 ++p;
1720 }1298 }
1721 return false;
1722}
1723 1299
1300 return amdgpu_dm_detect_mst_link_for_all_connectors(adev_to_drm(adev));
1301}
1724 1302
1725void*1303static void resume_mst_branch_status(struct drm_dp_mst_topology_mgr *mgr)
1726dm_allocate_gpu_mem(
1727 struct amdgpu_device *adev,
1728 enum dc_gpu_mem_alloc_type type,
1729 size_t size,
1730 long long *addr)
1731{1304{
1732 struct dal_allocation *da;1305 u8 buf[UUID_SIZE];
1733 u32 domain = (type == DC_MEM_ALLOC_TYPE_GART) ?1306 guid_t guid;
1734 AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM;
1735 int ret;1307 int ret;
1736 1308
1737 da = kzalloc_obj(struct dal_allocation);1309 mutex_lock(&mgr->lock);
1738 if (!da)1310 if (!mgr->mst_primary)
1739 return NULL;1311 goto out_fail;
1740 1312
1741 ret = amdgpu_bo_create_kernel(adev, size, PAGE_SIZE,1313 if (drm_dp_read_dpcd_caps(mgr->aux, mgr->dpcd) < 0) {
1742 domain, &da->bo,1314 drm_dbg_kms(mgr->dev, "dpcd read failed - undocked during suspend?\n");
1743 &da->gpu_addr, &da->cpu_ptr);1315 goto out_fail;
1316 }
1744 1317
1745 *addr = da->gpu_addr;1318 ret = drm_dp_dpcd_writeb(mgr->aux, DP_MSTM_CTRL,
1319 DP_MST_EN |
1320 DP_UP_REQ_EN |
1321 DP_UPSTREAM_IS_SRC);
1322 if (ret < 0) {
1323 drm_dbg_kms(mgr->dev, "mst write failed - undocked during suspend?\n");
1324 goto out_fail;
1325 }
1746 1326
1747 if (ret) {1327 /* Some hubs forget their guids after they resume */
1748 kfree(da);1328 ret = drm_dp_dpcd_read(mgr->aux, DP_GUID, buf, sizeof(buf));
1749 return NULL;1329 if (ret != sizeof(buf)) {
1330 drm_dbg_kms(mgr->dev, "dpcd read failed - undocked during suspend?\n");
1331 goto out_fail;
1750 }1332 }
1751 1333
1752 /* add da to list in dm */1334 import_guid(&guid, buf);
1753 list_add(&da->list, &adev->dm.da_list);
1754 1335
1755 return da->cpu_ptr;1336 if (guid_is_null(&guid)) {
1756}1337 guid_gen(&guid);
1338 export_guid(buf, &guid);
1757 1339
1758void1340 ret = drm_dp_dpcd_write(mgr->aux, DP_GUID, buf, sizeof(buf));
1759dm_free_gpu_mem(
1760 struct amdgpu_device *adev,
1761 enum dc_gpu_mem_alloc_type type,
1762 void *pvMem)
1763{
1764 struct dal_allocation *da;
1765 1341
1766 /* walk the da list in DM */1342 if (ret != sizeof(buf)) {
1767 list_for_each_entry(da, &adev->dm.da_list, list) {1343 drm_dbg_kms(mgr->dev, "check mstb guid failed - undocked during suspend?\n");
1768 if (pvMem == da->cpu_ptr) {1344 goto out_fail;
1769 amdgpu_bo_free_kernel(&da->bo, &da->gpu_addr, &da->cpu_ptr);
1770 list_del(&da->list);
1771 kfree(da);
1772 break;
1773 }1345 }
1774 }1346 }
1775 1347
1348 guid_copy(&mgr->mst_primary->guid, &guid);
1349
1350out_fail:
1351 mutex_unlock(&mgr->lock);
1776}1352}
1777 1353
1778static enum dmub_status1354static void s3_handle_mst(struct drm_device *dev, bool suspend)
1779dm_dmub_send_vbios_gpint_command(struct amdgpu_device *adev,
1780 enum dmub_gpint_command command_code,
1781 uint16_t param,
1782 uint32_t timeout_us)
1783{1355{
1784 union dmub_gpint_data_register reg, test;1356 struct amdgpu_dm_connector *aconnector;
1785 uint32_t i;1357 struct drm_connector *connector;
1786 1358 struct drm_connector_list_iter iter;
1787 /* Assume that VBIOS DMUB is ready to take commands */1359 struct drm_dp_mst_topology_mgr *mgr;
1788
1789 reg.bits.status = 1;
1790 reg.bits.command_code = command_code;
1791 reg.bits.param = param;
1792 1360
1793 cgs_write_register(adev->dm.cgs_device, 0x34c0 + 0x01f8, reg.all);1361 drm_connector_list_iter_begin(dev, &iter);
1362 drm_for_each_connector_iter(connector, &iter) {
1794 1363
1795 for (i = 0; i < timeout_us; ++i) {1364 if (connector->connector_type == DRM_MODE_CONNECTOR_WRITEBACK)
1796 udelay(1);1365 continue;
1797 1366
1798 /* Check if our GPINT got acked */1367 aconnector = to_amdgpu_dm_connector(connector);
1799 reg.bits.status = 0;1368 if (aconnector->dc_link->type != dc_connection_mst_branch ||
1800 test = (union dmub_gpint_data_register)1369 aconnector->mst_root)
1801 cgs_read_register(adev->dm.cgs_device, 0x34c0 + 0x01f8);1370 continue;
1802 1371
1803 if (test.all == reg.all)1372 mgr = &aconnector->mst_mgr;
1804 return DMUB_STATUS_OK;1373
1805 }1374 if (suspend) {
1375 drm_dp_mst_topology_mgr_suspend(mgr);
1376 } else {
1377 /* if extended timeout is supported in hardware,
1378 * default to LTTPR timeout (3.2ms) first as a W/A for DP link layer
1379 * CTS 4.2.1.1 regression introduced by CTS specs requirement update.
1380 */
1381 try_to_configure_aux_timeout(aconnector->dc_link->ddc, LINK_AUX_DEFAULT_LTTPR_TIMEOUT_PERIOD);
1382 if (!dp_is_lttpr_present(aconnector->dc_link))
1383 try_to_configure_aux_timeout(aconnector->dc_link->ddc, LINK_AUX_DEFAULT_TIMEOUT_PERIOD);
1806 1384
1807 return DMUB_STATUS_TIMEOUT;1385 /* TODO: move resume_mst_branch_status() into drm mst resume again
1386 * once topology probing work is pulled out from mst resume into mst
1387 * resume 2nd step. mst resume 2nd step should be called after old
1388 * state getting restored (i.e. drm_atomic_helper_resume()).
1389 */
1390 resume_mst_branch_status(mgr);
1391 }
1392 }
1393 drm_connector_list_iter_end(&iter);
1808}1394}
1809 1395
1810static void *dm_dmub_get_vbios_bounding_box(struct amdgpu_device *adev)1396static int amdgpu_dm_smu_write_watermarks_table(struct amdgpu_device *adev)
1811{1397{
1812 void *bb;1398 int ret = 0;
1813 long long addr;
1814 unsigned int bb_size;
1815 int i = 0;
1816 uint16_t chunk;
1817 enum dmub_gpint_command send_addrs[] = {
1818 DMUB_GPINT__SET_BB_ADDR_WORD0,
1819 DMUB_GPINT__SET_BB_ADDR_WORD1,
1820 DMUB_GPINT__SET_BB_ADDR_WORD2,
1821 DMUB_GPINT__SET_BB_ADDR_WORD3,
1822 };
1823 enum dmub_status ret;
1824 1399
1400 /* This interface is for dGPU Navi1x.Linux dc-pplib interface depends
1401 * on window driver dc implementation.
1402 * For Navi1x, clock settings of dcn watermarks are fixed. the settings
1403 * should be passed to smu during boot up and resume from s3.
1404 * boot up: dc calculate dcn watermark clock settings within dc_create,
1405 * dcn20_resource_construct
1406 * then call pplib functions below to pass the settings to smu:
1407 * smu_set_watermarks_for_clock_ranges
1408 * smu_set_watermarks_table
1409 * navi10_set_watermarks_table
1410 * smu_write_watermarks_table
1411 *
1412 * For Renoir, clock settings of dcn watermark are also fixed values.
1413 * dc has implemented different flow for window driver:
1414 * dc_hardware_init / dc_set_power_state
1415 * dcn10_init_hw
1416 * notify_wm_ranges
1417 * set_wm_ranges
1418 * -- Linux
1419 * smu_set_watermarks_for_clock_ranges
1420 * renoir_set_watermarks_table
1421 * smu_write_watermarks_table
1422 *
1423 * For Linux,
1424 * dc_hardware_init -> amdgpu_dm_init
1425 * dc_set_power_state --> dm_resume
1426 *
1427 * therefore, this function apply to navi10/12/14 but not Renoir
1428 * *
1429 */
1825 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {1430 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {
1826 case IP_VERSION(4, 0, 1):1431 case IP_VERSION(2, 0, 2):
1827 bb_size = sizeof(struct dml2_soc_bb);1432 case IP_VERSION(2, 0, 0):
1828 break;
1829 case IP_VERSION(4, 2, 0):
1830 case IP_VERSION(4, 2, 1):
1831 bb_size = sizeof(struct dml2_soc_bb);
1832 break;1433 break;
1833 default:1434 default:
1834 return NULL;1435 return 0;
1835 }1436 }
1836 1437
1837 bb = dm_allocate_gpu_mem(adev,1438 ret = amdgpu_dpm_write_watermarks_table(adev);
1838 DC_MEM_ALLOC_TYPE_GART,1439 if (ret) {
1839 bb_size,1440 drm_err(adev_to_drm(adev), "Failed to update WMTABLE!\n");
1840 &addr);1441 return ret;
1841 if (!bb)
1842 return NULL;
1843
1844 for (i = 0; i < 4; i++) {
1845 /* Extract 16-bit chunk */
1846 chunk = ((uint64_t) addr >> (i * 16)) & 0xFFFF;
1847 /* Send the chunk */
1848 ret = dm_dmub_send_vbios_gpint_command(adev, send_addrs[i], chunk, 30000);
1849 if (ret != DMUB_STATUS_OK)
1850 goto free_bb;
1851 }1442 }
1852 1443
1853 /* Now ask DMUB to copy the bb */1444 return 0;
1854 ret = dm_dmub_send_vbios_gpint_command(adev, DMUB_GPINT__BB_COPY, 1, 200000);
1855 if (ret != DMUB_STATUS_OK)
1856 goto free_bb;
1857
1858 return bb;
1859
1860free_bb:
1861 dm_free_gpu_mem(adev, DC_MEM_ALLOC_TYPE_GART, (void *) bb);
1862 return NULL;
1863
1864}1445}
1865 1446
1866static enum dmub_ips_disable_type dm_get_default_ips_mode(1447static int dm_oem_i2c_hw_init(struct amdgpu_device *adev)
1867 struct amdgpu_device *adev)
1868{1448{
1869 enum dmub_ips_disable_type ret = DMUB_IPS_ENABLE;1449 struct amdgpu_display_manager *dm = &adev->dm;
1450 struct amdgpu_i2c_adapter *oem_i2c;
1451 struct ddc_service *oem_ddc_service;
1452 int r;
1870 1453
1871 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {1454 oem_ddc_service = dc_get_oem_i2c_device(adev->dm.dc);
1872 case IP_VERSION(3, 5, 0):1455 if (oem_ddc_service) {
1873 case IP_VERSION(3, 6, 0):1456 oem_i2c = amdgpu_dm_create_i2c(oem_ddc_service, true);
1874 case IP_VERSION(3, 5, 1):1457 if (!oem_i2c) {
1875 ret = DMUB_IPS_RCG_IN_ACTIVE_IPS2_IN_OFF;1458 drm_info(adev_to_drm(adev), "Failed to create oem i2c adapter data\n");
1876 break;1459 return -ENOMEM;
1877 case IP_VERSION(4, 2, 0):1460 }
1878 case IP_VERSION(4, 2, 1):1461
1879 ret = DMUB_IPS_ENABLE;1462 r = devm_i2c_add_adapter(adev->dev, &oem_i2c->base);
1880 break;1463 if (r) {
1881 default:1464 drm_info(adev_to_drm(adev), "Failed to register oem i2c\n");
1882 /* ASICs older than DCN35 do not have IPSs */1465 kfree(oem_i2c);
1883 if (amdgpu_ip_version(adev, DCE_HWIP, 0) < IP_VERSION(3, 5, 0))1466 return r;
1884 ret = DMUB_IPS_DISABLE_ALL;1467 }
1885 break;1468 dm->oem_i2c = oem_i2c;
1886 }1469 }
1887 1470
1888 return ret;1471 return 0;
1889}1472}
1890 1473
1891static int amdgpu_dm_init_power_module(struct amdgpu_display_manager *dm)1474/**
1475 * dm_hw_init() - Initialize DC device
1476 * @ip_block: Pointer to the amdgpu_ip_block for this hw instance.
1477 *
1478 * Initialize the &struct amdgpu_display_manager device. This involves calling
1479 * the initializers of each DM component, then populating the struct with them.
1480 *
1481 * Although the function implies hardware initialization, both hardware and
1482 * software are initialized here. Splitting them out to their relevant init
1483 * hooks is a future TODO item.
1484 *
1485 * Some notable things that are initialized here:
1486 *
1487 * - Display Core, both software and hardware
1488 * - DC modules that we need (freesync and color management)
1489 * - DRM software states
1490 * - Interrupt sources and handlers
1491 * - Vblank support
1492 * - Debug FS entries, if enabled
1493 */
1494static int dm_hw_init(struct amdgpu_ip_block *ip_block)
1892{1495{
1893 struct mod_power_init_params init_data[MAX_NUM_EDP];1496 struct amdgpu_device *adev = ip_block->adev;
1497 int r;
1894 1498
1895 if (dm->num_of_edps == 0) {1499 /* Create DAL display manager */
1896 drm_dbg_driver(1500 r = amdgpu_dm_init(adev);
1897 dm->ddev,1501 if (r)
1898 "amdgpu: No eDP detected, skip initializing power module\n");1502 return r;
1899 return 0;1503 amdgpu_dm_hpd_init(adev);
1900 }
1901 1504
1902 /* Initialize all the power module parameters */1505 r = dm_oem_i2c_hw_init(adev);
1903 for (int i = 0; i < dm->num_of_edps; i++) {1506 if (r)
1904 init_data[i].allow_psr_smu_optimizations =1507 drm_info(adev_to_drm(adev), "Failed to add OEM i2c bus\n");
1905 !!(amdgpu_dc_feature_mask & DC_PSR_ALLOW_SMU_OPT);
1906 init_data[i].allow_psr_multi_disp_optimizations =
1907 !!(amdgpu_dc_feature_mask & DC_PSR_ALLOW_MULTI_DISP_OPT);
1908 /* See dm_late_init */
1909 init_data[i].backlight_ramping_override = false;
1910 init_data[i].backlight_ramping_start = 0xCCCC;
1911 init_data[i].backlight_ramping_reduction = 0xCCCCCCCC;
1912 init_data[i].def_varibright_level = 0;
1913 init_data[i].abm_config_setting = 0;
1914 init_data[i].num_backlight_levels = 101;
1915 init_data[i].use_nits_based_brightness = false;
1916 init_data[i].panel_max_millinits = 0;
1917 init_data[i].panel_min_millinits = 0;
1918 init_data[i].disable_fractional_pwm =
1919 !(amdgpu_dc_feature_mask & DC_DISABLE_FRACTIONAL_PWM_MASK);
1920 init_data[i].use_custom_backlight_caps = false;
1921 init_data[i].custom_backlight_caps_config_no = 0;
1922 init_data[i].use_linear_backlight_curve = false;
1923 init_data[i].def_varibright_enable = 0;
1924 init_data[i].varibright_level = 0;
1925 /*
1926 * Power module uses 16-bit backlight levels (0xFFFF max) rather
1927 * than 8-bit(0XFF max)
1928 */
1929 init_data[i].min_backlight_pwm =
1930 dm->backlight_caps[i].min_input_signal * 0x101;
1931 init_data[i].max_backlight_pwm =
1932 dm->backlight_caps[i].max_input_signal * 0x101;
1933 init_data[i].min_abm_backlight =
1934 dm->backlight_caps[i].min_input_signal * 0x101;
1935 1508
1936 /* Min backlight level after ABM reduction, Don't allow below 1%1509 return 0;
1937 * 0xFFFF x 0.01 = 0x28F1510}
1938 */
1939 init_data[i].min_abm_backlight = (init_data[i].min_abm_backlight < 0x28F) ?
1940 0x28F : init_data[i].min_abm_backlight;
1941 }
1942 1511
1943 dm->power_module = mod_power_create(dm->dc, init_data, dm->num_of_edps);1512/**
1944 if (!dm->power_module) {1513 * dm_hw_fini() - Teardown DC device
1945 drm_err(dm->ddev, "amdgpu: Error allocating memory for power module\n");1514 * @ip_block: Pointer to the amdgpu_ip_block for this hw instance.
1946 return -ENOMEM;1515 *
1947 }1516 * Teardown components within &struct amdgpu_display_manager that require
1517 * cleanup. This involves cleaning up the DRM device, DC, and any modules that
1518 * were loaded. Also flush IRQ workqueues and disable them.
1519 */
1520static int dm_hw_fini(struct amdgpu_ip_block *ip_block)
1521{
1522 struct amdgpu_device *adev = ip_block->adev;
1948 1523
1949 mod_power_hw_init(dm->power_module);1524 amdgpu_dm_hpd_fini(adev);
1950 drm_dbg_driver(dm->ddev, "amdgpu: Power module init done\n");
1951 1525
1526 amdgpu_dm_irq_fini(adev);
1527 amdgpu_dm_fini(adev);
1952 return 0;1528 return 0;
1953}1529}
1954 1530
1955static void hdmi_frl_status_polling_work(struct work_struct *work)1531
1532static void dm_gpureset_toggle_interrupts(struct amdgpu_device *adev,
1533 struct dc_state *state, bool enable)
1956{1534{
1957 struct amdgpu_display_manager *dm =1535 enum dc_irq_source irq_source;
1958 container_of(to_delayed_work(work), struct amdgpu_display_manager,1536 struct amdgpu_crtc *acrtc;
1959 hdmi_frl_status_polling_work);1537 int rc = -EBUSY;
1960 struct dc *dc = dm->dc;1538 int i = 0;
1961 struct dc_link *dc_link;
1962 bool link_update = false;
1963 1539
1964 for (int i = 0; i < MAX_LINKS; i++) {1540 for (i = 0; i < state->stream_count; i++) {
1965 dc_link = dc->links[i];1541 acrtc = amdgpu_dm_get_crtc_by_otg_inst(
1542 adev, state->stream_status[i].primary_otg_inst);
1966 1543
1967 if (!dc_link || !dc_link->local_sink)1544 if (acrtc && state->stream_status[i].plane_count != 0) {
1968 continue;1545 irq_source = IRQ_TYPE_PFLIP + acrtc->otg_inst;
1546 rc = dc_interrupt_set(adev->dm.dc, irq_source, enable) ? 0 : -EBUSY;
1547 if (rc)
1548 drm_warn(adev_to_drm(adev), "Failed to %s pflip interrupts\n",
1549 enable ? "enable" : "disable");
1969 1550
1970 if (!dc_is_hdmi_signal(dc_link->connector_signal))1551 if (dc_supports_vrr(adev->dm.dc->ctx->dce_version)) {
1971 continue;1552 if (enable) {
1553 if (amdgpu_dm_crtc_vrr_active(
1554 to_dm_crtc_state(acrtc->base.state)))
1555 rc = amdgpu_dm_crtc_set_vupdate_irq(
1556 &acrtc->base, true);
1557 } else
1558 rc = amdgpu_dm_crtc_set_vupdate_irq(
1559 &acrtc->base, false);
1972 1560
1973 if (dc_link->connector_signal != SIGNAL_TYPE_HDMI_FRL)1561 if (rc)
1974 continue;1562 drm_warn(adev_to_drm(adev), "Failed to %sable vupdate interrupt\n",
1563 enable ? "en" : "dis");
1564 }
1975 1565
1976 link_update = dc_link_frl_poll_status_flag(dc_link);1566 irq_source = IRQ_TYPE_VBLANK + acrtc->otg_inst;
1977 if (link_update) {1567 /* During gpu-reset we disable and then enable vblank irq, so
1978 mutex_lock(&dm->dc_lock);1568 * don't use amdgpu_irq_get/put() to avoid refcount change.
1979 dc_link_detect(dc_link, DETECT_REASON_RETRAIN);1569 */
1980 mutex_unlock(&dm->dc_lock);1570 if (!dc_interrupt_set(adev->dm.dc, irq_source, enable))
1571 drm_warn(adev_to_drm(adev), "Failed to %sable vblank interrupt\n", enable ? "en" : "dis");
1981 }1572 }
1982 }1573 }
1983 1574
1984 queue_delayed_work(dm->hdmi_frl_status_polling_wq,
1985 &dm->hdmi_frl_status_polling_work,
1986 msecs_to_jiffies(dm->hdmi_frl_status_polling_delay_ms));
1987}1575}
1988 1576
1989static int amdgpu_dm_init(struct amdgpu_device *adev)1577DEFINE_FREE(state_release, struct dc_state *, if (_T) dc_state_release(_T))
1990{
1991 struct dc_init_data init_data;
1992 struct dc_callback_init init_params;
1993 int r;
1994 1578
1995 adev->dm.ddev = adev_to_drm(adev);1579static enum dc_status amdgpu_dm_commit_zero_streams(struct dc *dc)
1996 adev->dm.adev = adev;1580{
1581 struct dc_state *context __free(state_release) = NULL;
1582 int i;
1583 struct dc_stream_state *del_streams[MAX_PIPES];
1584 int del_streams_count = 0;
1585 struct dc_commit_streams_params params = {};
1997 1586
1998 /* Zero all the fields */1587 memset(del_streams, 0, sizeof(del_streams));
1999 memset(&init_data, 0, sizeof(init_data));
2000 memset(&init_params, 0, sizeof(init_params));
2001 1588
2002 mutex_init(&adev->dm.dpia_aux_lock);1589 context = dc_state_create_current_copy(dc);
2003 mutex_init(&adev->dm.dc_lock);1590 if (context == NULL)
2004 mutex_init(&adev->dm.audio_lock);1591 return DC_ERROR_UNEXPECTED;
2005 1592
2006 spin_lock_init(&adev->dm.dmub_lock);1593 /* First remove from context all streams */
1594 for (i = 0; i < context->stream_count; i++) {
1595 struct dc_stream_state *stream = context->streams[i];
2007 1596
2008 if (amdgpu_dm_irq_init(adev)) {1597 del_streams[del_streams_count++] = stream;
2009 drm_err(adev_to_drm(adev), "failed to initialize DM IRQ support.\n");
2010 goto error;
2011 }1598 }
2012 1599
2013 /* special handling for early revisions of GC 11.5.4 */1600 /* Remove all planes for removed streams and then remove the streams */
2014 if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(11, 5, 4))1601 for (i = 0; i < del_streams_count; i++) {
2015 init_data.asic_id.chip_family = AMDGPU_FAMILY_GC_11_5_4;1602 enum dc_status res;
2016 else
2017 init_data.asic_id.chip_family = adev->family;
2018 1603
2019 init_data.asic_id.pci_revision_id = adev->pdev->revision;1604 if (!dc_state_rem_all_planes_for_stream(dc, del_streams[i], context))
2020 init_data.asic_id.hw_internal_rev = adev->external_rev_id;1605 return DC_FAIL_DETACH_SURFACES;
2021 init_data.asic_id.chip_id = adev->pdev->device;
2022 1606
2023 init_data.asic_id.vram_width = adev->gmc.vram_width;1607 res = dc_state_remove_stream(dc, context, del_streams[i]);
2024 /* TODO: initialize init_data.asic_id.vram_type here!!!! */1608 if (res != DC_OK)
2025 init_data.asic_id.atombios_base_address =1609 return res;
2026 adev->mode_info.atom_context->bios;1610 }
2027 1611
2028 init_data.driver = adev;1612 params.streams = context->streams;
1613 params.stream_count = context->stream_count;
2029 1614
2030 /* cgs_device was created in dm_sw_init() */1615 return dc_commit_streams(dc, &params);
2031 init_data.cgs_device = adev->dm.cgs_device;1616}
2032 1617
2033 init_data.dce_environment = DCE_ENV_PRODUCTION_DRV;1618static int dm_cache_state(struct amdgpu_device *adev)
1619{
1620 int r;
2034 1621
2035 switch (amdgpu_ip_version(adev, DCE_HWIP, 0)) {1622 adev->dm.cached_state = drm_atomic_helper_suspend(adev_to_drm(adev));
2036 case IP_VERSION(2, 1, 0):1623 if (IS_ERR(adev->dm.cached_state)) {
2037 switch (adev->dm.dmcub_fw_version) {1624 r = PTR_ERR(adev->dm.cached_state);
2038 case 0: /* development */1625 adev->dm.cached_state = NULL;
2039 case 0x1: /* linux-firmware.git hash 6d9f399 */
2040 case 0x01000000: /* linux-firmware.git hash 9a0b0f4 */
2041 init_data.flags.disable_dmcu = false;
2042 break;
2043 default:
2044 init_data.flags.disable_dmcu = true;
2045 }
2046 break;
2047 case IP_VERSION(2, 0, 3):
2048 init_data.flags.disable_dmcu = true;
2049 break;
2050 default:
2051 break;
2052 }1626 }
2053 1627
2054 /* APU support S/G display by default except:1628 return adev->dm.cached_state ? 0 : r;
2055 * ASICs before Carrizo,1629}
2056 * RAVEN1 (Users reported stability issue)
2057 */
2058 1630
2059 if (adev->asic_type < CHIP_CARRIZO) {1631static void dm_destroy_cached_state(struct amdgpu_device *adev)
2060 init_data.flags.gpu_vm_support = false;1632{
2061 } else if (adev->asic_type == CHIP_RAVEN) {1633 struct amdgpu_display_manager *dm = &adev->dm;
2062 if (adev->apu_flags & AMD_APU_IS_RAVEN)1634 struct drm_device *ddev = adev_to_drm(adev);
2063 init_data.flags.gpu_vm_support = false;1635 struct dm_plane_state *dm_new_plane_state;
2064 else1636 struct drm_plane_state *new_plane_state;
2065 init_data.flags.gpu_vm_support = (amdgpu_sg_display != 0);1637 struct dm_crtc_state *dm_new_crtc_state;
2066 } else {1638 struct drm_crtc_state *new_crtc_state;
2067 if (amdgpu_ip_version(adev, DCE_HWIP, 0) == IP_VERSION(2, 0, 3))1639 struct drm_plane *plane;
2068 init_data.flags.gpu_vm_support = (amdgpu_sg_display == 1);
2069 else
2070 init_data.flags.gpu_vm_support =
2071 (amdgpu_sg_display != 0) && (adev->flags & AMD_IS_APU);
2072 }

10262 more lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h+30−49

This diff is large: its 121 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_audio.c+323−0

This diff is large: its 324 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_audio.hrenamed from drivers/gpu/drm/amd/display/modules/power/power_helpers.c+32−15

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_backlight.c+726−0

This diff is large: its 727 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_backlight.h+75−0

This diff is large: its 76 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.c+104−75

This diff is large: its 374 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_color.h+4−4

This diff is large: its 28 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_colorop.c+1−0

This diff is large: its 5 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_connector.c+3663−0

This diff is large: its 3664 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_connector.h+162−0

This diff is large: its 163 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.c+80−22

This diff is large: its 155 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crc.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c+15−7

This diff is large: its 90 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_debugfs.c+71−7

This diff is large: its 137 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_dmub.c+943−0

This diff is large: its 944 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_dmub.h+68−0

This diff is large: its 69 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c+78−37

This diff is large: its 154 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h+12−0

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c+47−17

This diff is large: its 196 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.h+20−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c+1543−5

This diff is large: its 1633 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.h+27−0

This diff is large: its 39 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_kunit_helpers.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c+119−70

This diff is large: its 386 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.h+25−0

This diff is large: its 39 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c+83−54

This diff is large: its 324 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.h+51−0

This diff is large: its 63 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_pp_smu.c+172−131

This diff is large: its 585 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_pp_smu.h+61−0

This diff is large: its 62 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.c+49−2

This diff is large: its 83 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_psr.h+5−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_quirks.c+2−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_replay.c+5−5

This diff is large: its 49 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_services.c+7−0

This diff is large: its 43 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.c+11−5

This diff is large: its 62 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_wb.h+13−0

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/.kunitconfig+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/Makefile+18−0

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_audio_test.c+490−0

This diff is large: its 491 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_backlight_test.c+1242−0

This diff is large: its 1243 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_color_test.c+32−32

This diff is large: its 145 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_colorop_test.c+125−15

This diff is large: its 177 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_connector_test.c+2158−0

This diff is large: its 2159 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_crc_test.c+122−0

This diff is large: its 140 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_crtc_test.c+523−0

This diff is large: its 524 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_dmub_test.c+582−0

This diff is large: its 583 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_hdcp_test.c+280−17

This diff is large: its 356 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_helpers_test.c+634−0

This diff is large: its 635 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_irq_test.c+909−0

This diff is large: its 910 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_ism_test.c+13−26

This diff is large: its 130 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_kunit_helpers.c+142−0

This diff is large: its 143 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_kunit_test_helpers.h+32−0

This diff is large: its 33 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_mst_types_test.c+1092−0

This diff is large: its 1093 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_plane_test.c+1228−0

This diff is large: its 1229 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_pp_smu_test.c+2454−0

This diff is large: its 2455 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_psr_test.c+497−21

This diff is large: its 642 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_quirks_test.c+103−0

This diff is large: its 104 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_replay_test.c+426−6

This diff is large: its 484 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_services_test.c+313−0

This diff is large: its 314 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_test.c+949−0

This diff is large: its 950 lines aren’t shown here.

drivers/gpu/drm/amd/display/amdgpu_dm/tests/amdgpu_dm_wb_test.c+392−0

This diff is large: its 393 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn401/dcn401_clk_mgr.c+54−3

This diff is large: its 106 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/clk_mgr/dcn401/dcn401_clk_mgr.h+9−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc.c+172−119

This diff is large: its 665 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c+164−10

This diff is large: its 289 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_resource.c+5−6

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/core/dc_surface.c+38−6

This diff is large: its 60 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc.h+175−63

This diff is large: its 345 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c+21−20

This diff is large: its 90 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h+30−13

This diff is large: its 68 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_dp_types.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_helper.c+0−226

This diff is large: its 275 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_stream.h+29−0

This diff is large: its 36 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dc_types.h+49−18

This diff is large: its 115 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dccg/dcn42/dcn42_dccg.c+34−25

This diff is large: its 97 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dccg/dcn42/dcn42_dccg.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce/dce_aux.c+5−4

This diff is large: its 44 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce110/dce110_compressor.c+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce110/dce110_mem_input_v.c+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dce112/dce112_compressor.c+6−2

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dcn30/dcn30_cm_common.c+184−0

This diff is large: its 191 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dio/dcn31/dcn31_dio_link_encoder.c+4−0

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dio/dcn31/dcn31_dio_link_encoder.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dm_services.h+0−4

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/calcs/dcn_calcs.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn31/display_mode_vba_31.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml/dcn314/display_mode_vba_314.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/dml21_wrapper_fpu.c+4−4

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/inc/bounding_boxes/dcn42b_soc_bb.h+39−1

This diff is large: its 52 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/inc/dml_top_soc_parameter_types.h+13−0

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/dml2_core/dml2_core_dcn4_calcs.c+28−13

This diff is large: its 107 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/dml2_core/dml2_core_factory.c+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/dml2_core/dml2_core_shared_types.h+0−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dml2_0/dml21/src/inc/dml2_internal_shared_types.h+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dpp/dcn10/dcn10_dpp_cm.c+0−5

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/dwb/dcn30/dcn30_cm_common.h+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dce80/hw_translate_dce80.c+4−1

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn10/hw_translate_dcn10.c+173−311

This diff is large: its 507 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn20/hw_translate_dcn20.c+153−279

This diff is large: its 456 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn21/hw_translate_dcn21.c+147−270

This diff is large: its 442 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn30/hw_translate_dcn30.c+153−279

This diff is large: its 456 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn315/hw_translate_dcn315.c+148−270

This diff is large: its 442 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn32/hw_translate_dcn32.c+138−248

This diff is large: its 409 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn401/hw_translate_dcn401.c+137−255

This diff is large: its 416 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/dcn42/hw_translate_dcn42.c+70−123

This diff is large: its 215 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/hw_translate.c+86−0

This diff is large: its 90 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/gpio/hw_translate.h+21−0

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hubbub/dcn31/dcn31_hubbub.c+4−4

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hubbub/dcn35/dcn35_hubbub.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hubp/dcn401/dcn401_hubp.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hubp/dcn42/dcn42_hubp.c+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce110/dce110_hwseq.c+4−4

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dce60/dce60_hwseq.c+4−4

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn10/dcn10_hwseq.c+31−31

This diff is large: its 131 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn20/dcn20_hwseq.c+38−38

This diff is large: its 173 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn201/dcn201_hwseq.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn30/dcn30_hwseq.c+6−4

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn32/dcn32_hwseq.c+18−18

This diff is large: its 79 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn35/dcn35_hwseq.c+0−3

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.c+214−127

This diff is large: its 639 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_hwseq.h+11−1

This diff is large: its 30 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn401/dcn401_init.c+2−0

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn42/dcn42_hwseq.c+166−161

This diff is large: its 635 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/dcn42/dcn42_hwseq.h+2−3

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer.h+82−0

This diff is large: its 117 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/hwss/hw_sequencer_private.h+2−1

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr.h+12−0

This diff is large: its 47 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/dccg.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/hw/hubp.h+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/link_service.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/reg_helper.h+0−19

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/inc/soc_and_ip_translator.h+0−14

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/link_detection.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/link_factory.c+5−2

This diff is large: its 35 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.c+17−3

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_capability.c+12−2

This diff is large: its 38 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_panel_replay.c+2−1

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c+3−5

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/mpc/dcn20/dcn20_mpc.c+0−4

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/opp/dcn20/dcn20_opp.c+43−8

This diff is large: its 90 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn10/dcn10_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn20/dcn20_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn31/dcn31_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn314/dcn314_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn32/dcn32_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn35/dcn35_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/optc/dcn401/dcn401_optc.c+0−5

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/pg/dcn42/dcn42_pg_cntl.c+125−133

This diff is large: its 387 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/pg/dcn42/dcn42_pg_cntl.h+18−4

This diff is large: its 71 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn30/dcn30_resource.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn302/dcn302_resource.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn303/dcn303_resource.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn31/dcn31_resource.c+3−2

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn314/dcn314_resource.c+3−1

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn315/dcn315_resource.c+3−2

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn316/dcn316_resource.c+3−2

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn32/dcn32_resource.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn321/dcn321_resource.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn35/dcn35_resource.c+3−3

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn351/dcn351_resource.c+3−3

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn36/dcn36_resource.c+3−3

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn42/dcn42_resource.c+4−1

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/resource/dcn42b/dcn42b_resource.c+4−2

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/Makefile+3−0

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42/dcn42_soc_and_ip_translator.c+6−12

This diff is large: its 47 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42/dcn42_soc_and_ip_translator.h+1−0

This diff is large: its 7 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42b/dcn42b_soc_and_ip_translator.c+42−0

This diff is large: its 43 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/dcn42b/dcn42b_soc_and_ip_translator.h+17−0

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/soc_and_ip_translator/soc_and_ip_translator.c+4−1

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/display/dc/sspl/spl_debug.h+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h+144−145

This diff is large: its 388 lines aren’t shown here.

drivers/gpu/drm/amd/display/include/ddc_service_types.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/display/include/gpio_types.h+48−0

This diff is large: its 62 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/power/Makefile+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/power/power.c+6−6

This diff is large: its 23 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/power/power_abm.c+2−12

This diff is large: its 28 lines aren’t shown here.

drivers/gpu/drm/amd/display/modules/power/power_replay.c+4−5

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/amd/include/amd_shared.h+0−3

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/include/asic_reg/sdma/sdma_4_4_2_offset.h+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/include/kgd_kfd_interface.h+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/include/mes_v11_api_def.h+3−2

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/include/mes_v12_api_def.h+3−2

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/amd/include/soc15_hw_ip.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/include/v9_structs.h+2−2

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/pm/amdgpu_pm.c+48−68

This diff is large: its 178 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/amd_powerplay.c+8−13

This diff is large: its 34 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c+30−8

This diff is large: its 126 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c+8−3

This diff is large: its 25 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/process_pptables_v1_0.c+626−129

This diff is large: its 1004 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/processpptables.c+316−130

This diff is large: its 660 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/smu7_hwmgr.c+44−14

This diff is large: its 94 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_hwmgr.c+66−15

This diff is large: its 149 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega10_processpptables.c+599−162

This diff is large: its 1032 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega12_processpptables.c+7−0

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/hwmgr/vega20_processpptables.c+7−0

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/pm/powerplay/inc/hwmgr.h+17−0

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/amdgpu_smu.c+16−18

This diff is large: its 68 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/amdgpu_smu.h+7−30

This diff is large: its 72 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v11_0.h+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v13_0.h+0−6

This diff is large: its 20 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v14_0.h+0−4

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/inc/smu_v15_0.h+0−4

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/navi10_ppt.c+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c+4−4

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu11/smu_v11_0.c+10−74

This diff is large: its 104 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu12/renoir_ppt.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0.c+2−92

This diff is large: its 129 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c+4−3

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_12_ppt.c+1−7

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c+0−653

This diff is large: its 694 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_7_ppt.c+4−3

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0.c+2−83

This diff is large: its 113 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c+4−5

This diff is large: its 23 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu15/smu_v15_0.c+2−83

This diff is large: its 113 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu_cmn.c+140−0

This diff is large: its 144 lines aren’t shown here.

drivers/gpu/drm/amd/pm/swsmu/smu_cmn.h+13−0

This diff is large: its 27 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_cmd.c+0−3

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_eeprom_i2c.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_mgr.c+74−2

This diff is large: its 120 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_mgr.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_mp1_v13_0.c+14−0

This diff is large: its 32 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/amdgpu_ras_sys.c+19−0

This diff is large: its 31 lines aren’t shown here.

drivers/gpu/drm/amd/ras/ras_mgr/ras_sys.h+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras.h+5−0

This diff is large: its 24 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_aca_v1_0.c+6−7

This diff is large: its 38 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_core.c+15−0

This diff is large: its 26 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_eeprom.c+3−0

This diff is large: its 10 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_eeprom_fw.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_mp1.c+18−1

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_mp1.h+3−0

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_mp1_v13_0.c+13−0

This diff is large: its 21 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_process.c+3−2

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_umc.c+28−4

This diff is large: its 85 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_umc.h+6−0

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/amd/ras/rascore/ras_umc_v12_0.h+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/radeon/r600_dpm.c+15−6

This diff is large: its 60 lines aren’t shown here.

include/uapi/drm/amdgpu_drm.h+21−0

This diff is large: its 42 lines aren’t shown here.