amd-drm-next-6.16-2025-05-09: amdgpu: - IPS fixes - DSC cleanup - DC Scaling updates - DC FP fixes - Fused I2C-over-AUX updates - SubVP fixes - Freesync fix - DMUB AUX fixes - VCN fix - Hibernation fixes - HDP fixes - DCN 2.1 fixes - DPIA fixes - DMUB updates - Use drm_file_err in amdgpu - Enforce isolation updates - Use new dma_fence helpers - USERQ fixes - Documentation updates - Misc code cleanups - SR-IOV updates - RAS updates - PSP 12 cleanups amdkfd: - Update error messages for SDMA - Userptr updates drm: - Add drm_file_err function dma-buf: - Add a helper to sort and deduplicate dma_fence arrays
Unified · Split
| @@ -0,0 +1,23 @@ | |||
| 1 | ================================================= | ||
| 2 | AMD Hardware Components Information per Product | ||
| 3 | ================================================= | ||
| 4 | |||
| 5 | On this page, you can find the AMD product name and which component version is | ||
| 6 | part of it. | ||
| 7 | |||
| 8 | Accelerated Processing Units (APU) Info | ||
| 9 | --------------------------------------- | ||
| 10 | |||
| 11 | .. csv-table:: | ||
| 12 | :header-rows: 1 | ||
| 13 | :widths: 3, 2, 2, 1, 1, 1, 1 | ||
| 14 | :file: ./apu-asic-info-table.csv | ||
| 15 | |||
| 16 | Discrete GPU Info | ||
| 17 | ----------------- | ||
| 18 | |||
| 19 | .. csv-table:: | ||
| 20 | :header-rows: 1 | ||
| 21 | :widths: 3, 2, 2, 1, 1, 1 | ||
| 22 | :file: ./dgpu-asic-info-table.csv | ||
| 23 | |||
| @@ -12,18 +12,39 @@ we have a dedicated glossary for Display Core at · 11 unmodified lines | |||
| 12 | The number of CUs that are active on the system. The number of active | 12 | The number of CUs that are active on the system. The number of active |
| 13 | CUs may be less than SE * SH * CU depending on the board configuration. | 13 | CUs may be less than SE * SH * CU depending on the board configuration. |
| 14 | 14 | ||
| 15 | BACO | ||
| 16 | Bus Alive, Chip Off | ||
| 17 | |||
| 18 | BOCO | ||
| 19 | Bus Off, Chip Off | ||
| 20 | |||
| 15 | CE | 21 | CE |
| 16 | Constant Engine | 22 | Constant Engine |
| 17 | 23 | ||
| 24 | CIK | ||
| 25 | Sea Islands | ||
| 26 | |||
| 27 | CB | ||
| 28 | Color Buffer | ||
| 29 | |||
| 18 | CP | 30 | CP |
| 19 | Command Processor | 31 | Command Processor |
| 20 | 32 | ||
| 21 | CPLIB | 33 | CPLIB |
| 22 | Content Protection Library | 34 | Content Protection Library |
| 23 | 35 | ||
| 36 | CS | ||
| 37 | Command Submission | ||
| 38 | |||
| 39 | CSB | ||
| 40 | Clear State Indirect Buffer | ||
| 41 | |||
| 24 | CU | 42 | CU |
| 25 | Compute Unit | 43 | Compute Unit |
| 26 | 44 | ||
| 45 | DB | ||
| 46 | Depth Buffer | ||
| 47 | |||
| 27 | DFS | 48 | DFS |
| 28 | Digital Frequency Synthesizer | 49 | Digital Frequency Synthesizer |
| 29 | 50 | ||
| @@ -33,6 +54,9 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines | |||
| 33 | EOP | 54 | EOP |
| 34 | End Of Pipe/Pipeline | 55 | End Of Pipe/Pipeline |
| 35 | 56 | ||
| 57 | FLR | ||
| 58 | Function Level Reset | ||
| 59 | |||
| 36 | GART | 60 | GART |
| 37 | Graphics Address Remapping Table. This is the name we use for the GPUVM | 61 | Graphics Address Remapping Table. This is the name we use for the GPUVM |
| 38 | page table used by the GPU kernel driver. It remaps system resources | 62 | page table used by the GPU kernel driver. It remaps system resources |
| @@ -45,6 +69,12 @@ we have a dedicated glossary for Display Core at · 6 unmodified lines | |||
| 45 | GC | 69 | GC |
| 46 | Graphics and Compute | 70 | Graphics and Compute |
| 47 | 71 | ||
| 72 | GDS | ||
| 73 | Global Data Share | ||
| 74 | |||
| 75 | GE | ||
| 76 | Geometry Engine | ||
| 77 | |||
| 48 | GMC | 78 | GMC |
| 49 | Graphic Memory Controller | 79 | Graphic Memory Controller |
| 50 | 80 | ||
| @@ -80,6 +110,9 @@ we have a dedicated glossary for Display Core at · 29 unmodified lines | |||
| 80 | KCQ | 110 | KCQ |
| 81 | Kernel Compute Queue | 111 | Kernel Compute Queue |
| 82 | 112 | ||
| 113 | KFD | ||
| 114 | Kernel Fusion Driver | ||
| 115 | |||
| 83 | KGQ | 116 | KGQ |
| 84 | Kernel Graphics Queue | 117 | Kernel Graphics Queue |
| 85 | 118 | ||
| @@ -89,6 +122,9 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines | |||
| 89 | MC | 122 | MC |
| 90 | Memory Controller | 123 | Memory Controller |
| 91 | 124 | ||
| 125 | MCBP | ||
| 126 | Mid Command Buffer Preemption | ||
| 127 | |||
| 92 | ME | 128 | ME |
| 93 | MicroEngine (Graphics) | 129 | MicroEngine (Graphics) |
| 94 | 130 | ||
| @@ -104,6 +140,9 @@ we have a dedicated glossary for Display Core at · 9 unmodified lines | |||
| 104 | MQD | 140 | MQD |
| 105 | Memory Queue Descriptor | 141 | Memory Queue Descriptor |
| 106 | 142 | ||
| 143 | PA | ||
| 144 | Primitive Assembler / Physical Address | ||
| 145 | |||
| 107 | PFP | 146 | PFP |
| 108 | Pre-Fetch Parser (Graphics) | 147 | Pre-Fetch Parser (Graphics) |
| 109 | 148 | ||
| @@ -113,24 +152,39 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines | |||
| 113 | PSP | 152 | PSP |
| 114 | Platform Security Processor | 153 | Platform Security Processor |
| 115 | 154 | ||
| 155 | RB | ||
| 156 | Render Backends. Some people called it ROPs. | ||
| 157 | |||
| 116 | RLC | 158 | RLC |
| 117 | RunList Controller. This name is a remnant of past ages and doesn't have | 159 | RunList Controller. This name is a remnant of past ages and doesn't have |
| 118 | much meaning today. It's a group of general-purpose helper engines for | 160 | much meaning today. It's a group of general-purpose helper engines for |
| 119 | the GFX block. It's involved in GFX power management and SR-IOV, among | 161 | the GFX block. It's involved in GFX power management and SR-IOV, among |
| 120 | other things. | 162 | other things. |
| 121 | 163 | ||
| 164 | SC | ||
| 165 | Scan Converter | ||
| 166 | |||
| 122 | SDMA | 167 | SDMA |
| 123 | System DMA | 168 | System DMA |
| 124 | 169 | ||
| 125 | SE | 170 | SE |
| 126 | Shader Engine | 171 | Shader Engine |
| 127 | 172 | ||
| 173 | SGPR | ||
| 174 | Scalar General-Purpose Registers | ||
| 175 | |||
| 128 | SH | 176 | SH |
| 129 | SHader array | 177 | SHader array |
| 130 | 178 | ||
| 179 | SI | ||
| 180 | Southern Islands | ||
| 181 | |||
| 131 | SMU/SMC | 182 | SMU/SMC |
| 132 | System Management Unit / System Management Controller | 183 | System Management Unit / System Management Controller |
| 133 | 184 | ||
| 185 | SPI | ||
| 186 | Shader Processor Input | ||
| 187 | |||
| 134 | SRLC | 188 | SRLC |
| 135 | Save/Restore List Control | 189 | Save/Restore List Control |
| 136 | 190 | ||
| @@ -143,12 +197,21 @@ we have a dedicated glossary for Display Core at · 6 unmodified lines | |||
| 143 | SS | 197 | SS |
| 144 | Spread Spectrum | 198 | Spread Spectrum |
| 145 | 199 | ||
| 200 | SX | ||
| 201 | Shader Export | ||
| 202 | |||
| 146 | TA | 203 | TA |
| 147 | Trusted Application | 204 | Trusted Application |
| 148 | 205 | ||
| 206 | TC | ||
| 207 | Texture Cache | ||
| 208 | |||
| 149 | TOC | 209 | TOC |
| 150 | Table of Contents | 210 | Table of Contents |
| 151 | 211 | ||
| 212 | UMSCH | ||
| 213 | User Mode Scheduler | ||
| 214 | |||
| 152 | UVD | 215 | UVD |
| 153 | Unified Video Decoder | 216 | Unified Video Decoder |
| 154 | 217 | ||
| @@ -158,5 +221,17 @@ we have a dedicated glossary for Display Core at · 3 unmodified lines | |||
| 158 | VCN | 221 | VCN |
| 159 | Video Codec Next | 222 | Video Codec Next |
| 160 | 223 | ||
| 224 | VGPR | ||
| 225 | Vector General-Purpose Registers | ||
| 226 | |||
| 227 | VMID | ||
| 228 | Virtual Memory ID | ||
| 229 | |||
| 161 | VPE | 230 | VPE |
| 162 | Video Processing Engine | 231 | Video Processing Engine |
| 232 | |||
| 233 | XCC | ||
| 234 | Accelerator Core Complex | ||
| 235 | |||
| 236 | XCP | ||
| 237 | Accelerator Core Partition | ||
| @@ -13,3 +13,5 @@ Ryzen 7x20 series, Mendocino, 3.1.6, 10.3.7, 3.1.1, 5.2.7, 13.0.8 · 12 unmodified lines | |||
| 13 | Ryzen 7x40 series, Phoenix, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11 | 13 | Ryzen 7x40 series, Phoenix, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11 |
| 14 | Ryzen 8x40 series, Hawk Point, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11 | 14 | Ryzen 8x40 series, Hawk Point, 3.1.4, 11.0.1 / 11.0.4, 4.0.2, 6.0.1, 13.0.4 / 13.0.11 |
| 15 | Ryzen AI 300 series, Strix Point, 3.5.0, 11.5.0, 4.0.5, 6.1.0, 14.0.0 | 15 | Ryzen AI 300 series, Strix Point, 3.5.0, 11.5.0, 4.0.5, 6.1.0, 14.0.0 |
| 16 | Ryzen AI 350 series, Krackan Point, 3.5.0, 11.5.2, 4.0.5, 6.1.2, 14.0.4 | ||
| 17 | Ryzen AI Max 300 series, Strix Halo, 3.5.1, 11.5.1, 4.0.6, 6.1.1, 14.0.1 | ||
| @@ -0,0 +1,210 @@ | |||
| 1 | ============== | ||
| 2 | AMDGPU DebugFS | ||
| 3 | ============== | ||
| 4 | |||
| 5 | The amdgpu driver provides a number of debugfs files to aid in debugging | ||
| 6 | issues in the driver. These are usually found in | ||
| 7 | /sys/kernel/debug/dri/<num>. | ||
| 8 | |||
| 9 | DebugFS Files | ||
| 10 | ============= | ||
| 11 | |||
| 12 | amdgpu_benchmark | ||
| 13 | ---------------- | ||
| 14 | |||
| 15 | Run benchmarks using the DMA engine the driver uses for GPU memory paging. | ||
| 16 | Write a number to the file to run the test. The results are written to the | ||
| 17 | kernel log. VRAM is on device memory (dGPUs) or carve out (APUs) and GTT | ||
| 18 | (Graphics Translation Tables) is system memory that is accessible by the GPU. | ||
| 19 | The following tests are available: | ||
| 20 | |||
| 21 | - 1: simple test, VRAM to GTT and GTT to VRAM | ||
| 22 | - 2: simple test, VRAM to VRAM | ||
| 23 | - 3: GTT to VRAM, buffer size sweep, powers of 2 | ||
| 24 | - 4: VRAM to GTT, buffer size sweep, powers of 2 | ||
| 25 | - 5: VRAM to VRAM, buffer size sweep, powers of 2 | ||
| 26 | - 6: GTT to VRAM, buffer size sweep, common display sizes | ||
| 27 | - 7: VRAM to GTT, buffer size sweep, common display sizes | ||
| 28 | - 8: VRAM to VRAM, buffer size sweep, common display sizes | ||
| 29 | |||
| 30 | amdgpu_test_ib | ||
| 31 | -------------- | ||
| 32 | |||
| 33 | Read this file to run simple IB (Indirect Buffer) tests on all kernel managed | ||
| 34 | rings. IBs are command buffers usually generated by userspace applications | ||
| 35 | which are submitted to the kernel for execution on an particular GPU engine. | ||
| 36 | This just runs the simple IB tests included in the kernel. These tests | ||
| 37 | are engine specific and verify that IB submission works. | ||
| 38 | |||
| 39 | amdgpu_discovery | ||
| 40 | ---------------- | ||
| 41 | |||
| 42 | Provides raw access to the IP discovery binary provided by the GPU. Read this | ||
| 43 | file to access the raw binary. This is useful for verifying the contents of | ||
| 44 | the IP discovery table. It is chip specific. | ||
| 45 | |||
| 46 | amdgpu_vbios | ||
| 47 | ------------ | ||
| 48 | |||
| 49 | Provides raw access to the ROM binary image from the GPU. Read this file to | ||
| 50 | access the raw binary. This is useful for verifying the contents of the | ||
| 51 | video BIOS ROM. It is board specific. | ||
| 52 | |||
| 53 | amdgpu_evict_gtt | ||
| 54 | ---------------- | ||
| 55 | |||
| 56 | Evict all buffers from the GTT memory pool. Read this file to evict all | ||
| 57 | buffers from this pool. | ||
| 58 | |||
| 59 | amdgpu_evict_vram | ||
| 60 | ----------------- | ||
| 61 | |||
| 62 | Evict all buffers from the VRAM memory pool. Read this file to evict all | ||
| 63 | buffers from this pool. | ||
| 64 | |||
| 65 | amdgpu_gpu_recover | ||
| 66 | ------------------ | ||
| 67 | |||
| 68 | Trigger a GPU reset. Read this file to trigger reset the entire GPU. | ||
| 69 | All work currently running on the GPU will be lost. | ||
| 70 | |||
| 71 | amdgpu_ring_<name> | ||
| 72 | ------------------ | ||
| 73 | |||
| 74 | Provides read access to the kernel managed ring buffers for each ring <name>. | ||
| 75 | These are useful for debugging problems on a particular ring. The ring buffer | ||
| 76 | is how the CPU sends commands to the GPU. The CPU writes commands into the | ||
| 77 | buffer and then asks the GPU engine to process it. This is the raw binary | ||
| 78 | contents of the ring buffer. Use a tool like UMR to decode the rings into human | ||
| 79 | readable form. | ||
| 80 | |||
| 81 | amdgpu_mqd_<name> | ||
| 82 | ----------------- | ||
| 83 | |||
| 84 | Provides read access to the kernel managed MQD (Memory Queue Descriptor) for | ||
| 85 | ring <name> managed by the kernel driver. MQDs define the features of the ring | ||
| 86 | and are used to store the ring's state when it is not connected to hardware. | ||
| 87 | The driver writes the requested ring features and metadata (GPU addresses of | ||
| 88 | the ring itself and associated buffers) to the MQD and the firmware uses the MQD | ||
| 89 | to populate the hardware when the ring is mapped to a hardware slot. Only | ||
| 90 | available on engines which use MQDs. This provides access to the raw MQD | ||
| 91 | binary. | ||
| 92 | |||
| 93 | amdgpu_error_<name> | ||
| 94 | ------------------- | ||
| 95 | |||
| 96 | Provides an interface to set an error code on the dma fences associated with | ||
| 97 | ring <name>. The error code specified is propogated to all fences associated | ||
| 98 | with the ring. Use this to inject a fence error into a ring. | ||
| 99 | |||
| 100 | amdgpu_pm_info | ||
| 101 | -------------- | ||
| 102 | |||
| 103 | Provides human readable information about the power management features | ||
| 104 | and state of the GPU. This includes current GFX clock, Memory clock, | ||
| 105 | voltages, average SoC power, temperature, GFX load, Memory load, SMU | ||
| 106 | feature mask, VCN power state, clock and power gating features. | ||
| 107 | |||
| 108 | amdgpu_firmware_info | ||
| 109 | -------------------- | ||
| 110 | |||
| 111 | Lists the firmware versions for all firmwares used by the GPU. Only | ||
| 112 | entries with a non-0 version are valid. If the version is 0, the firmware | ||
| 113 | is not valid for the GPU. | ||
| 114 | |||
| 115 | amdgpu_fence_info | ||
| 116 | ----------------- | ||
| 117 | |||
| 118 | Shows the last signalled and emitted fence sequence numbers for each | ||
| 119 | kernel driver managed ring. Fences are associated with submissions | ||
| 120 | to the engine. Emitted fences have been submitted to the ring | ||
| 121 | and signalled fences have been signalled by the GPU. Rings with a | ||
| 122 | larger emitted fence value have outstanding work that is still being | ||
| 123 | processed by the engine that owns that ring. When the emitted and | ||
| 124 | signalled fence values are equal, the ring is idle. | ||
| 125 | |||
| 126 | amdgpu_gem_info | ||
| 127 | --------------- | ||
| 128 | |||
| 129 | Lists all of the PIDs using the GPU and the GPU buffers that they have | ||
| 130 | allocated. This lists the buffer size, pool (VRAM, GTT, etc.), and buffer | ||
| 131 | attributes (CPU access required, CPU cache attributes, etc.). | ||
| 132 | |||
| 133 | amdgpu_vm_info | ||
| 134 | -------------- | ||
| 135 | |||
| 136 | Lists all of the PIDs using the GPU and the GPU buffers that they have | ||
| 137 | allocated as well as the status of those buffers relative to that process' | ||
| 138 | GPU virtual address space (e.g., evicted, idle, invalidated, etc.). | ||
| 139 | |||
| 140 | amdgpu_sa_info | ||
| 141 | -------------- | ||
| 142 | |||
| 143 | Prints out all of the suballocations (sa) by the suballocation manager in the | ||
| 144 | kernel driver. Prints the GPU address, size, and fence info associated | ||
| 145 | with each suballocation. The suballocations are used internally within | ||
| 146 | the kernel driver for various things. | ||
| 147 | |||
| 148 | amdgpu_<pool>_mm | ||
| 149 | ---------------- | ||
| 150 | |||
| 151 | Prints TTM information about the memory pool <pool>. | ||
| 152 | |||
| 153 | amdgpu_vram | ||
| 154 | ----------- | ||
| 155 | |||
| 156 | Provides direct access to VRAM. Used by tools like UMR to inspect | ||
| 157 | objects in VRAM. | ||
| 158 | |||
| 159 | amdgpu_iomem | ||
| 160 | ------------ | ||
| 161 | |||
| 162 | Provides direct access to GTT memory. Used by tools like UMR to inspect | ||
| 163 | GTT memory. | ||
| 164 | |||
| 165 | amdgpu_regs_* | ||
| 166 | ------------- | ||
| 167 | |||
| 168 | Provides direct access to various register aperatures on the GPU. Used | ||
| 169 | by tools like UMR to access GPU registers. | ||
| 170 | |||
| 171 | amdgpu_regs2 | ||
| 172 | ------------ | ||
| 173 | |||
| 174 | Provides an IOCTL interface used by UMR for interacting with GPU registers. | ||
| 175 | |||
| 176 | |||
| 177 | amdgpu_sensors | ||
| 178 | -------------- | ||
| 179 | |||
| 180 | Provides an interface to query GPU power metrics (temperature, average | ||
| 181 | power, etc.). Used by tools like UMR to query GPU power metrics. | ||
| 182 | |||
| 183 | |||
| 184 | amdgpu_gca_config | ||
| 185 | ----------------- | ||
| 186 | |||
| 187 | Provides an interface to query GPU details (Graphics/Compute Array config, | ||
| 188 | PCI config, GPU family, etc.). Used by tools like UMR to query GPU details. | ||
| 189 | |||
| 190 | amdgpu_wave | ||
| 191 | ----------- | ||
| 192 | |||
| 193 | Used to query GFX/compute wave information from the hardware. Used by tools | ||
| 194 | like UMR to query GFX/compute wave information. | ||
| 195 | |||
| 196 | amdgpu_gpr | ||
| 197 | ---------- | ||
| 198 | |||
| 199 | Used to query GFX/compute GPR (General Purpose Register) information from the | ||
| 200 | hardware. Used by tools like UMR to query GPRs when debugging shaders. | ||
| 201 | |||
| 202 | amdgpu_gprwave | ||
| 203 | -------------- | ||
| 204 | |||
| 205 | Provides an IOCTL interface used by UMR for interacting with shader waves. | ||
| 206 | |||
| 207 | amdgpu_fw_attestation | ||
| 208 | --------------------- | ||
| 209 | |||
| 210 | Provides an interface for reading back firmware attestation records. | ||
| @@ -2,6 +2,13 @@ · 1 unmodified line | |||
| 2 | GPU Debugging | 2 | GPU Debugging |
| 3 | =============== | 3 | =============== |
| 4 | 4 | ||
| 5 | General Debugging Options | ||
| 6 | ========================= | ||
| 7 | |||
| 8 | The DebugFS section provides documentation on a number files to aid in debugging | ||
| 9 | issues on the GPU. | ||
| 10 | |||
| 11 | |||
| 5 | GPUVM Debugging | 12 | GPUVM Debugging |
| 6 | =============== | 13 | =============== |
| 7 | 14 | ||
| @@ -154,7 +154,7 @@ of the display parameters, but the userspace might also cause this issue. One · 153 unmodified lines | |||
| 154 | way to identify the source of the problem is to take a screenshot or make a | 154 | way to identify the source of the problem is to take a screenshot or make a |
| 155 | desktop video capture when the problem happens; after checking the | 155 | desktop video capture when the problem happens; after checking the |
| 156 | screenshot/video recording, if you don't see any of the artifacts, it means | 156 | screenshot/video recording, if you don't see any of the artifacts, it means |
| 157 | that the issue is likely on the the driver side. If you can still see the | 157 | that the issue is likely on the driver side. If you can still see the |
| 158 | problem in the data collected, it is an issue that probably happened during | 158 | problem in the data collected, it is an issue that probably happened during |
| 159 | rendering, and the display code just got the framebuffer already corrupted. | 159 | rendering, and the display code just got the framebuffer already corrupted. |
| 160 | 160 | ||
| @@ -67,36 +67,66 @@ GC (Graphics and Compute) · 66 unmodified lines | |||
| 67 | This is the graphics and compute engine, i.e., the block that | 67 | This is the graphics and compute engine, i.e., the block that |
| 68 | encompasses the 3D pipeline and and shader blocks. This is by far the | 68 | encompasses the 3D pipeline and and shader blocks. This is by far the |
| 69 | largest block on the GPU. The 3D pipeline has tons of sub-blocks. In | 69 | largest block on the GPU. The 3D pipeline has tons of sub-blocks. In |
| 70 | addition to that, it also contains the CP microcontrollers (ME, PFP, | 70 | addition to that, it also contains the CP microcontrollers (ME, PFP, CE, |
| 71 | CE, MEC) and the RLC microcontroller. It's exposed to userspace for | 71 | MEC) and the RLC microcontroller. It's exposed to userspace for user mode |
| 72 | user mode drivers (OpenGL, Vulkan, OpenCL, etc.) | 72 | drivers (OpenGL, Vulkan, OpenCL, etc.). More details in :ref:`Graphics (GFX) |
| 73 | and Compute <amdgpu-gc>`. | ||
| 73 | 74 | ||
| 74 | VCN (Video Core Next) | 75 | VCN (Video Core Next) |
| 75 | This is the multi-media engine. It handles video and image encode and | 76 | This is the multi-media engine. It handles video and image encode and |
| 76 | decode. It's exposed to userspace for user mode drivers (VA-API, | 77 | decode. It's exposed to userspace for user mode drivers (VA-API, |
| 77 | OpenMAX, etc.) | 78 | OpenMAX, etc.) |
| 78 | 79 | ||
| 79 | Graphics and Compute Microcontrollers | 80 | .. _pipes-and-queues-description: |
| 80 | ------------------------------------- | 81 | |
| 81 | 82 | GFX, Compute, and SDMA Overall Behavior | |
| 82 | CP (Command Processor) | 83 | ======================================= |
| 83 | The name for the hardware block that encompasses the front end of the | 84 | |
| 84 | GFX/Compute pipeline. Consists mainly of a bunch of microcontrollers | 85 | .. note:: For simplicity, whenever the term block is used in this section, it |
| 85 | (PFP, ME, CE, MEC). The firmware that runs on these microcontrollers | 86 | means GFX, Compute, and SDMA. |
| 86 | provides the driver interface to interact with the GFX/Compute engine. | 87 | |
| 87 | 88 | GFX, Compute and SDMA share a similar form of operation that can be abstracted | |
| 88 | MEC (MicroEngine Compute) | 89 | to facilitate understanding of the behavior of these blocks. See the figure |
| 89 | This is the microcontroller that controls the compute queues on the | 90 | below illustrating the common components of these blocks: |
| 90 | GFX/compute engine. | 91 | |
| 91 | 92 | .. kernel-figure:: pipe_and_queue_abstraction.svg | |
| 92 | MES (MicroEngine Scheduler) | 93 | |
| 93 | This is a new engine for managing queues. This is currently unused. | 94 | In the central part of this figure, you can see two hardware elements, one called |
| 94 | 95 | **Pipes** and another called **Queues**; it is important to highlight that Queues | |
| 95 | RLC (RunList Controller) | 96 | must be associated with a Pipe and vice-versa. Every specific hardware IP may have |
| 96 | This is another microcontroller in the GFX/Compute engine. It handles | 97 | a different number of Pipes and, in turn, a different number of Queues; for |
| 97 | power management related functionality within the GFX/Compute engine. | 98 | example, GFX 11 has two Pipes and two Queues per Pipe for the GFX front end. |
| 98 | The name is a vestige of old hardware where it was originally added | 99 | |
| 99 | and doesn't really have much relation to what the engine does now. | 100 | Pipe is the hardware that processes the instructions available in the Queues; |
| 101 | in other words, it is a thread executing the operations inserted in the Queue. | ||
| 102 | One crucial characteristic of Pipes is that they can only execute one Queue at | ||
| 103 | a time; no matter if the hardware has multiple Queues in the Pipe, it only runs | ||
| 104 | one Queue per Pipe. | ||
| 105 | |||
| 106 | Pipes have the mechanics of swapping between queues at the hardware level. | ||
| 107 | Nonetheless, they only make use of Queues that are considered mapped. Pipes can | ||
| 108 | switch between queues based on any of the following inputs: | ||
| 109 | |||
| 110 | 1. Command Stream; | ||
| 111 | 2. Packet by Packet; | ||
| 112 | 3. Other hardware requests the change (e.g., MES). | ||
| 113 | |||
| 114 | Queues within Pipes are defined by the Hardware Queue Descriptors (HQD). | ||
| 115 | Associated with the HQD concept, we have the Memory Queue Descriptor (MQD), | ||
| 116 | which is responsible for storing information about the state of each of the | ||
| 117 | available Queues in the memory. The state of a Queue contains information such | ||
| 118 | as the GPU virtual address of the queue itself, save areas, doorbell, etc. The | ||
| 119 | MQD also stores the HQD registers, which are vital for activating or | ||
| 120 | deactivating a given Queue. The scheduling firmware (e.g., MES) is responsible | ||
| 121 | for loading HQDs from MQDs and vice versa. | ||
| 122 | |||
| 123 | The Queue-switching process can also happen with the firmware requesting the | ||
| 124 | preemption or unmapping of a Queue. The firmware waits for the HQD_ACTIVE bit | ||
| 125 | to change to low before saving the state into the MQD. To make a different | ||
| 126 | Queue become active, the firmware copies the MQD state into the HQD registers | ||
| 127 | and loads any additional state. Finally, it sets the HQD_ACTIVE bit to high to | ||
| 128 | indicate that the queue is active. The Pipe will then execute work from active | ||
| 129 | Queues. | ||
| 100 | 130 | ||
| 101 | Driver Structure | 131 | Driver Structure |
| 102 | ================ | 132 | ================ |
| @@ -110,7 +140,8 @@ Some useful constructs: · 7 unmodified lines | |||
| 110 | KIQ (Kernel Interface Queue) | 140 | KIQ (Kernel Interface Queue) |
| 111 | This is a control queue used by the kernel driver to manage other gfx | 141 | This is a control queue used by the kernel driver to manage other gfx |
| 112 | and compute queues on the GFX/compute engine. You can use it to | 142 | and compute queues on the GFX/compute engine. You can use it to |
| 113 | map/unmap additional queues, etc. | 143 | map/unmap additional queues, etc. This is replaced by MES on |
| 144 | GFX 11 and newer hardware. | ||
| 114 | 145 | ||
| 115 | IB (Indirect Buffer) | 146 | IB (Indirect Buffer) |
| 116 | A command buffer for a particular engine. Rather than writing | 147 | A command buffer for a particular engine. Rather than writing |
| @@ -50,23 +50,6 @@ board_info · 49 unmodified lines | |||
| 50 | .. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 50 | .. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |
| 51 | :doc: board_info | 51 | :doc: board_info |
| 52 | 52 | ||
| 53 | Accelerated Processing Units (APU) Info | ||
| 54 | --------------------------------------- | ||
| 55 | |||
| 56 | .. csv-table:: | ||
| 57 | :header-rows: 1 | ||
| 58 | :widths: 3, 2, 2, 1, 1, 1, 1 | ||
| 59 | :file: ./apu-asic-info-table.csv | ||
| 60 | |||
| 61 | Discrete GPU Info | ||
| 62 | ----------------- | ||
| 63 | |||
| 64 | .. csv-table:: | ||
| 65 | :header-rows: 1 | ||
| 66 | :widths: 3, 2, 2, 1, 1, 1 | ||
| 67 | :file: ./dgpu-asic-info-table.csv | ||
| 68 | |||
| 69 | |||
| 70 | GPU Memory Usage Information | 53 | GPU Memory Usage Information |
| 71 | ============================ | 54 | ============================ |
| 72 | 55 | ||
| @@ -0,0 +1,52 @@ | |||
| 1 | .. _amdgpu-gc: | ||
| 2 | |||
| 3 | ======================================== | ||
| 4 | drm/amdgpu - Graphics and Compute (GC) | ||
| 5 | ======================================== | ||
| 6 | |||
| 7 | The relationship between the CPU and GPU can be described as the | ||
| 8 | producer-consumer problem, where the CPU fills out a buffer with operations | ||
| 9 | (producer) to be executed by the GPU (consumer). The requested operations in | ||
| 10 | the buffer are called Command Packets, which can be summarized as a compressed | ||
| 11 | way of transmitting command information to the graphics controller. | ||
| 12 | |||
| 13 | The component that acts as the front end between the CPU and the GPU is called | ||
| 14 | the Command Processor (CP). This component is responsible for providing greater | ||
| 15 | flexibility to the GC since CP makes it possible to program various aspects of | ||
| 16 | the GPU pipeline. CP also coordinates the communication between the CPU and GPU | ||
| 17 | via a mechanism named **Ring Buffers**, where the CPU appends information to | ||
| 18 | the buffer while the GPU removes operations. It is relevant to highlight that a | ||
| 19 | CPU can add a pointer to the Ring Buffer that points to another region of | ||
| 20 | memory outside the Ring Buffer, and CP can handle it; this mechanism is called | ||
| 21 | **Indirect Buffer (IB)**. CP receives and parses the Command Streams (CS), and | ||
| 22 | writes the operations to the correct hardware blocks. | ||
| 23 | |||
| 24 | Graphics (GFX) and Compute Microcontrollers | ||
| 25 | ------------------------------------------- | ||
| 26 | |||
| 27 | GC is a large block, and as a result, it has multiple firmware associated with | ||
| 28 | it. Some of them are: | ||
| 29 | |||
| 30 | CP (Command Processor) | ||
| 31 | The name for the hardware block that encompasses the front end of the | ||
| 32 | GFX/Compute pipeline. Consists mainly of a bunch of microcontrollers | ||
| 33 | (PFP, ME, CE, MEC). The firmware that runs on these microcontrollers | ||
| 34 | provides the driver interface to interact with the GFX/Compute engine. | ||
| 35 | |||
| 36 | MEC (MicroEngine Compute) | ||
| 37 | This is the microcontroller that controls the compute queues on the | ||
| 38 | GFX/compute engine. | ||
| 39 | |||
| 40 | MES (MicroEngine Scheduler) | ||
| 41 | This is the engine for managing queues. For more details check | ||
| 42 | :ref:`MicroEngine Scheduler (MES) <amdgpu-mes>`. | ||
| 43 | |||
| 44 | RLC (RunList Controller) | ||
| 45 | This is another microcontroller in the GFX/Compute engine. It handles | ||
| 46 | power management related functionality within the GFX/Compute engine. | ||
| 47 | The name is a vestige of old hardware where it was originally added | ||
| 48 | and doesn't really have much relation to what the engine does now. | ||
| 49 | |||
| 50 | .. toctree:: | ||
| 51 | |||
| 52 | mes.rst | ||
| @@ -0,0 +1,38 @@ | |||
| 1 | .. _amdgpu-mes: | ||
| 2 | |||
| 3 | ============================= | ||
| 4 | MicroEngine Scheduler (MES) | ||
| 5 | ============================= | ||
| 6 | |||
| 7 | .. note:: | ||
| 8 | Queue and ring buffer are used as a synonymous. | ||
| 9 | |||
| 10 | .. note:: | ||
| 11 | This section assumes that you are familiar with the concept of Pipes, Queues, and GC. | ||
| 12 | If not, check :ref:`GFX, Compute, and SDMA Overall Behavior<pipes-and-queues-description>` | ||
| 13 | and :ref:`drm/amdgpu - Graphics and Compute (GC) <amdgpu-gc>`. | ||
| 14 | |||
| 15 | Every GFX has a pipe component with one or more hardware queues. Pipes can | ||
| 16 | switch between queues depending on certain conditions, and one of the | ||
| 17 | components that can request a queue switch to a pipe is the MicroEngine | ||
| 18 | Scheduler (MES). Whenever the driver is initialized, it creates one MQD per | ||
| 19 | hardware queue, and then the MQDs are handed to the MES firmware for mapping | ||
| 20 | to: | ||
| 21 | |||
| 22 | 1. Kernel Queues (legacy): This queue is statically mapped to HQDs and never | ||
| 23 | preempted. Even though this is a legacy feature, it is the current default, and | ||
| 24 | most existing hardware supports it. When an application submits work to the | ||
| 25 | kernel driver, it submits all of the application command buffers to the kernel | ||
| 26 | queues. The CS IOCTL takes the command buffer from the applications and | ||
| 27 | schedules them on the kernel queue. | ||
| 28 | |||
| 29 | 2. User Queues: These queues are dynamically mapped to the HQDs. Regarding the | ||
| 30 | utilization of User Queues, the userspace application will create its user | ||
| 31 | queues and submit work directly to its user queues with no need to IOCTL for | ||
| 32 | each submission and no need to share a single kernel queue. | ||
| 33 | |||
| 34 | In terms of User Queues, MES can dynamically map them to the HQD. If there are | ||
| 35 | more MQDs than HQDs, the MES firmware will preempt other user queues to make | ||
| 36 | sure each queues get a time slice; in other words, MES is a microcontroller | ||
| 37 | that handles the mapping and unmapping of MQDs into HQDs, as well as the | ||
| 38 | priorities and oversubscription of MQDs. | ||
| @@ -7,8 +7,10 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures. · 6 unmodified lines | |||
| 7 | 7 | ||
| 8 | .. toctree:: | 8 | .. toctree:: |
| 9 | 9 | ||
| 10 | module-parameters | ||
| 11 | driver-core | 10 | driver-core |
| 11 | amd-hardware-list-info | ||
| 12 | module-parameters | ||
| 13 | gc/index | ||
| 12 | display/index | 14 | display/index |
| 13 | flashing | 15 | flashing |
| 14 | xgmi | 16 | xgmi |
| @@ -16,5 +18,6 @@ Next (GCN), Radeon DNA (RDNA), and Compute DNA (CDNA) architectures. · 1 unmodified line | |||
| 16 | thermal | 18 | thermal |
| 17 | driver-misc | 19 | driver-misc |
| 18 | debugging | 20 | debugging |
| 21 | debugfs | ||
| 19 | process-isolation | 22 | process-isolation |
| 20 | amdgpu-glossary | 23 | amdgpu-glossary |
| @@ -0,0 +1,1279 @@ | |||
| 1 | <?xml version="1.0" encoding="UTF-8" standalone="no"?> | ||
| 2 | <!-- Created with Inkscape (http://www.inkscape.org/) --> | ||
| 3 | |||
| 4 | <svg | ||
| 5 | width="395.47891mm" | ||
| 6 | height="234.73715mm" | ||
| 7 | viewBox="0 0 395.47891 234.73714" | ||
| 8 | version="1.1" | ||
| 9 | id="svg1" | ||
| 10 | inkscape:version="1.4 (e7c3feb100, 2024-10-09)" | ||
| 11 | sodipodi:docname="pipe_and_queue_abstraction.svg" | ||
| 12 | xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" | ||
| 13 | xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" | ||
| 14 | xmlns="http://www.w3.org/2000/svg" | ||
| 15 | xmlns:svg="http://www.w3.org/2000/svg"> | ||
| 16 | <sodipodi:namedview | ||
| 17 | id="namedview1" | ||
| 18 | pagecolor="#ffffff" | ||
| 19 | bordercolor="#000000" | ||
| 20 | borderopacity="0.25" | ||
| 21 | inkscape:showpageshadow="2" | ||
| 22 | inkscape:pageopacity="0.0" | ||
| 23 | inkscape:pagecheckerboard="0" | ||
| 24 | inkscape:deskcolor="#d1d1d1" | ||
| 25 | inkscape:document-units="mm" | ||
| 26 | inkscape:zoom="1.6489689" | ||
| 27 | inkscape:cx="713.17296" | ||
| 28 | inkscape:cy="466.65527" | ||
| 29 | inkscape:window-width="3840" | ||
| 30 | inkscape:window-height="2083" | ||
| 31 | inkscape:window-x="0" | ||
| 32 | inkscape:window-y="0" | ||
| 33 | inkscape:window-maximized="1" | ||
| 34 | inkscape:current-layer="layer1" /> | ||
| 35 | <defs | ||
| 36 | id="defs1"> | ||
| 37 | <marker | ||
| 38 | style="overflow:visible" | ||
| 39 | id="Dot" | ||
| 40 | refX="0" | ||
| 41 | refY="0" | ||
| 42 | orient="auto" | ||
| 43 | inkscape:stockid="Dot" | ||
| 44 | markerWidth="1" | ||
| 45 | markerHeight="1" | ||
| 46 | viewBox="0 0 1 1" | ||
| 47 | inkscape:isstock="true" | ||
| 48 | inkscape:collect="always" | ||
| 49 | preserveAspectRatio="xMidYMid"> | ||
| 50 | <path | ||
| 51 | transform="scale(0.5)" | ||
| 52 | style="fill:context-stroke;fill-rule:evenodd;stroke:none" | ||
| 53 | d="M 5,0 C 5,2.76 2.76,5 0,5 -2.76,5 -5,2.76 -5,0 c 0,-2.76 2.3,-5 5,-5 2.76,0 5,2.24 5,5 z" | ||
| 54 | sodipodi:nodetypes="sssss" | ||
| 55 | id="path110" /> | ||
| 56 | </marker> | ||
| 57 | <marker | ||
| 58 | style="overflow:visible" | ||
| 59 | id="marker109" | ||
| 60 | refX="0" | ||
| 61 | refY="0" | ||
| 62 | orient="auto-start-reverse" | ||
| 63 | inkscape:stockid="Stylized triangle arrow" | ||
| 64 | markerWidth="1" | ||
| 65 | markerHeight="1" | ||
| 66 | viewBox="0 0 1 1" | ||
| 67 | inkscape:isstock="true" | ||
| 68 | inkscape:collect="always" | ||
| 69 | preserveAspectRatio="xMidYMid"> | ||
| 70 | <path | ||
| 71 | transform="scale(0.5)" | ||
| 72 | style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt" | ||
| 73 | d="m 6,0 c -3,1 -7,3 -9,5 0,0 0,-4 2,-5 -2,-1 -2,-5 -2,-5 2,2 6,4 9,5 z" | ||
| 74 | id="path109" /> | ||
| 75 | </marker> | ||
| 76 | <marker | ||
| 77 | style="overflow:visible" | ||
| 78 | id="ArrowTriangleStylized" | ||
| 79 | refX="0" | ||
| 80 | refY="0" | ||
| 81 | orient="auto-start-reverse" | ||
| 82 | inkscape:stockid="Stylized triangle arrow" | ||
| 83 | markerWidth="1" | ||
| 84 | markerHeight="1" | ||
| 85 | viewBox="0 0 1 1" | ||
| 86 | inkscape:isstock="true" | ||
| 87 | inkscape:collect="always" | ||
| 88 | preserveAspectRatio="xMidYMid"> | ||
| 89 | <path | ||
| 90 | transform="scale(0.5)" | ||
| 91 | style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt" | ||
| 92 | d="m 6,0 c -3,1 -7,3 -9,5 0,0 0,-4 2,-5 -2,-1 -2,-5 -2,-5 2,2 6,4 9,5 z" | ||
| 93 | id="path108" /> | ||
| 94 | </marker> | ||
| 95 | <marker | ||
| 96 | style="overflow:visible" | ||
| 97 | id="ArrowWide" | ||
| 98 | refX="0" | ||
| 99 | refY="0" | ||
| 100 | orient="auto-start-reverse" | ||
| 101 | inkscape:stockid="Wide arrow" | ||
| 102 | markerWidth="1" | ||
| 103 | markerHeight="1" | ||
| 104 | viewBox="0 0 1 1" | ||
| 105 | inkscape:isstock="true" | ||
| 106 | inkscape:collect="always" | ||
| 107 | preserveAspectRatio="xMidYMid"> | ||
| 108 | <path | ||
| 109 | style="fill:none;stroke:context-stroke;stroke-width:1;stroke-linecap:butt" | ||
| 110 | d="M 3,-3 0,0 3,3" | ||
| 111 | transform="rotate(180,0.125,0)" | ||
| 112 | sodipodi:nodetypes="ccc" | ||
| 113 | id="path1" /> | ||
| 114 | </marker> | ||
| 115 | <marker | ||
| 116 | style="overflow:visible" | ||
| 117 | id="Triangle" | ||
| 118 | refX="0" | ||
| 119 | refY="0" | ||
| 120 | orient="auto-start-reverse" | ||
| 121 | inkscape:stockid="Triangle arrow" | ||
| 122 | markerWidth="1" | ||
| 123 | markerHeight="1" | ||
| 124 | viewBox="0 0 1 1" | ||
| 125 | inkscape:isstock="true" | ||
| 126 | inkscape:collect="always" | ||
| 127 | preserveAspectRatio="xMidYMid"> | ||
| 128 | <path | ||
| 129 | transform="scale(0.5)" | ||
| 130 | style="fill:context-stroke;fill-rule:evenodd;stroke:context-stroke;stroke-width:1pt" | ||
| 131 | d="M 5.77,0 -2.88,5 V -5 Z" | ||
| 132 | id="path135" /> | ||
| 133 | </marker> | ||
| 134 | <marker | ||
| 135 | style="overflow:visible" | ||
| 136 | id="ArrowWideHeavy" | ||
| 137 | refX="0" | ||
| 138 | refY="0" | ||
| 139 | orient="auto-start-reverse" | ||
| 140 | inkscape:stockid="Wide, heavy arrow" | ||
| 141 | markerWidth="1" | ||
| 142 | markerHeight="1" | ||
| 143 | viewBox="0 0 1 1" | ||
| 144 | inkscape:isstock="true" | ||
| 145 | inkscape:collect="always" | ||
| 146 | preserveAspectRatio="xMidYMid"> | ||
| 147 | <path | ||
| 148 | style="fill:context-stroke;fill-rule:evenodd;stroke:none" | ||
| 149 | d="m 1,0 -3,3 h -2 l 3,-3 -3,-3 h 2 z" | ||
| 150 | id="path71" /> | ||
| 151 | </marker> | ||
| 152 | </defs> | ||
| 153 | <g | ||
| 154 | inkscape:label="Layer 1" | ||
| 155 | inkscape:groupmode="layer" | ||
| 156 | id="layer1" | ||
| 157 | transform="translate(149.03517,55.110629)"> | ||
| 158 | <circle | ||
| 159 | style="fill:#ffeeaa;fill-opacity:1;stroke:#1a1a1a;stroke-width:0.733436;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 160 | id="path98" | ||
| 161 | cx="-35.757576" | ||
| 162 | cy="-10.495151" | ||
| 163 | r="44.24876" /> | ||
| 164 | <rect | ||
| 165 | style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none" | ||
| 166 | id="rect1" | ||
| 167 | width="167.79619" | ||
| 168 | height="24.831829" | ||
| 169 | x="14.21942" | ||
| 170 | y="57.862854" /> | ||
| 171 | <g | ||
| 172 | id="g11" | ||
| 173 | transform="translate(24.021362,-46.545299)"> | ||
| 174 | <rect | ||
| 175 | style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none" | ||
| 176 | id="rect2" | ||
| 177 | width="131.09708" | ||
| 178 | height="5.8163381" | ||
| 179 | x="23.245802" | ||
| 180 | y="107.16314" /> | ||
| 181 | <path | ||
| 182 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 183 | d="m 37.884707,107.26337 v 5.53565" | ||
| 184 | id="path2" | ||
| 185 | sodipodi:nodetypes="cc" /> | ||
| 186 | <path | ||
| 187 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 188 | d="m 52.466233,107.26337 v 5.53565" | ||
| 189 | id="path3" | ||
| 190 | sodipodi:nodetypes="cc" /> | ||
| 191 | <path | ||
| 192 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 193 | d="m 67.047758,107.26337 v 5.53565" | ||
| 194 | id="path4" | ||
| 195 | sodipodi:nodetypes="cc" /> | ||
| 196 | <path | ||
| 197 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 198 | d="m 81.629283,107.26337 v 5.53565" | ||
| 199 | id="path5" | ||
| 200 | sodipodi:nodetypes="cc" /> | ||
| 201 | <path | ||
| 202 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 203 | d="m 96.210809,107.26337 v 5.53565" | ||
| 204 | id="path6" | ||
| 205 | sodipodi:nodetypes="cc" /> | ||
| 206 | <path | ||
| 207 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 208 | d="m 110.79233,107.26337 v 5.53565" | ||
| 209 | id="path7" | ||
| 210 | sodipodi:nodetypes="cc" /> | ||
| 211 | <path | ||
| 212 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 213 | d="m 125.37386,107.26337 v 5.53565" | ||
| 214 | id="path8" | ||
| 215 | sodipodi:nodetypes="cc" /> | ||
| 216 | <path | ||
| 217 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 218 | d="m 139.95538,107.26337 v 5.53565" | ||
| 219 | id="path9" | ||
| 220 | sodipodi:nodetypes="cc" /> | ||
| 221 | <text | ||
| 222 | xml:space="preserve" | ||
| 223 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 224 | x="86.713425" | ||
| 225 | y="110.1963" | ||
| 226 | id="text11"><tspan | ||
| 227 | sodipodi:role="line" | ||
| 228 | id="tspan11" | ||
| 229 | style="stroke:#006680;stroke-width:0.5" | ||
| 230 | x="86.713425" | ||
| 231 | y="110.1963">. . .</tspan></text> | ||
| 232 | </g> | ||
| 233 | <g | ||
| 234 | id="g18" | ||
| 235 | transform="translate(24.021362,-32.25779)"> | ||
| 236 | <rect | ||
| 237 | style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none" | ||
| 238 | id="rect11" | ||
| 239 | width="131.09708" | ||
| 240 | height="5.8163381" | ||
| 241 | x="23.245802" | ||
| 242 | y="107.16314" /> | ||
| 243 | <path | ||
| 244 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 245 | d="m 37.884707,107.26337 v 5.53565" | ||
| 246 | id="path11" | ||
| 247 | sodipodi:nodetypes="cc" /> | ||
| 248 | <path | ||
| 249 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 250 | d="m 52.466233,107.26337 v 5.53565" | ||
| 251 | id="path12" | ||
| 252 | sodipodi:nodetypes="cc" /> | ||
| 253 | <path | ||
| 254 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 255 | d="m 67.047758,107.26337 v 5.53565" | ||
| 256 | id="path13" | ||
| 257 | sodipodi:nodetypes="cc" /> | ||
| 258 | <path | ||
| 259 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 260 | d="m 81.629283,107.26337 v 5.53565" | ||
| 261 | id="path14" | ||
| 262 | sodipodi:nodetypes="cc" /> | ||
| 263 | <path | ||
| 264 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 265 | d="m 96.210809,107.26337 v 5.53565" | ||
| 266 | id="path15" | ||
| 267 | sodipodi:nodetypes="cc" /> | ||
| 268 | <path | ||
| 269 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 270 | d="m 110.79233,107.26337 v 5.53565" | ||
| 271 | id="path16" | ||
| 272 | sodipodi:nodetypes="cc" /> | ||
| 273 | <path | ||
| 274 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 275 | d="m 125.37386,107.26337 v 5.53565" | ||
| 276 | id="path17" | ||
| 277 | sodipodi:nodetypes="cc" /> | ||
| 278 | <path | ||
| 279 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 280 | d="m 139.95538,107.26337 v 5.53565" | ||
| 281 | id="path18" | ||
| 282 | sodipodi:nodetypes="cc" /> | ||
| 283 | <text | ||
| 284 | xml:space="preserve" | ||
| 285 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 286 | x="86.713425" | ||
| 287 | y="110.1963" | ||
| 288 | id="text18"><tspan | ||
| 289 | sodipodi:role="line" | ||
| 290 | id="tspan18" | ||
| 291 | style="stroke:#006680;stroke-width:0.5" | ||
| 292 | x="86.713425" | ||
| 293 | y="110.1963">. . .</tspan></text> | ||
| 294 | </g> | ||
| 295 | <text | ||
| 296 | xml:space="preserve" | ||
| 297 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 298 | x="113.37768" | ||
| 299 | y="68.463142" | ||
| 300 | id="text19"><tspan | ||
| 301 | sodipodi:role="line" | ||
| 302 | id="tspan19" | ||
| 303 | style="writing-mode:tb-rl;stroke-width:0.5" | ||
| 304 | x="113.37768" | ||
| 305 | y="68.463142">. . .</tspan></text> | ||
| 306 | <text | ||
| 307 | xml:space="preserve" | ||
| 308 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 309 | x="25.005701" | ||
| 310 | y="55.308445" | ||
| 311 | id="text844-2-9"><tspan | ||
| 312 | sodipodi:role="line" | ||
| 313 | x="25.005701" | ||
| 314 | y="55.308445" | ||
| 315 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 316 | id="tspan2868">Pipe[0]</tspan></text> | ||
| 317 | <text | ||
| 318 | xml:space="preserve" | ||
| 319 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 320 | x="-74.441521" | ||
| 321 | y="63.075123" | ||
| 322 | id="text844-2-9-4"><tspan | ||
| 323 | sodipodi:role="line" | ||
| 324 | x="-74.441521" | ||
| 325 | y="63.075123" | ||
| 326 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 327 | id="tspan2868-7">MQD</tspan></text> | ||
| 328 | <text | ||
| 329 | xml:space="preserve" | ||
| 330 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 331 | x="30.264952" | ||
| 332 | y="65.490654" | ||
| 333 | id="text20"><tspan | ||
| 334 | sodipodi:role="line" | ||
| 335 | x="30.264952" | ||
| 336 | y="65.490654" | ||
| 337 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 338 | id="tspan20">Queue[0]</tspan></text> | ||
| 339 | <text | ||
| 340 | xml:space="preserve" | ||
| 341 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 342 | x="30.264952" | ||
| 343 | y="79.249001" | ||
| 344 | id="text21"><tspan | ||
| 345 | sodipodi:role="line" | ||
| 346 | x="30.264952" | ||
| 347 | y="79.249001" | ||
| 348 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 349 | id="tspan21">Queue[n]</tspan></text> | ||
| 350 | <text | ||
| 351 | xml:space="preserve" | ||
| 352 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 353 | x="30.264952" | ||
| 354 | y="71.84066" | ||
| 355 | id="text22"><tspan | ||
| 356 | sodipodi:role="line" | ||
| 357 | x="30.264952" | ||
| 358 | y="71.84066" | ||
| 359 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 360 | id="tspan22">...</tspan></text> | ||
| 361 | <g | ||
| 362 | id="g71" | ||
| 363 | transform="translate(-23.283342)"> | ||
| 364 | <rect | ||
| 365 | style="fill:#ffffff;fill-opacity:0;stroke:#000000;stroke-width:0.5;stroke-dasharray:0.5, 0.5;stroke-dashoffset:0" | ||
| 366 | id="rect67" | ||
| 367 | width="18.533583" | ||
| 368 | height="114.96632" | ||
| 369 | x="250.9435" | ||
| 370 | y="54.754276" | ||
| 371 | ry="6.0427966" /> | ||
| 372 | </g> | ||
| 373 | <rect | ||
| 374 | style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 375 | id="rect68" | ||
| 376 | width="188.21231" | ||
| 377 | height="139.5948" | ||
| 378 | x="4.0113592" | ||
| 379 | y="37.597778" | ||
| 380 | ry="0" /> | ||
| 381 | <g | ||
| 382 | id="g43" | ||
| 383 | transform="translate(0,40.745853)"> | ||
| 384 | <rect | ||
| 385 | style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none" | ||
| 386 | id="rect22" | ||
| 387 | width="167.79619" | ||
| 388 | height="24.831829" | ||
| 389 | x="14.21942" | ||
| 390 | y="57.862854" /> | ||
| 391 | <g | ||
| 392 | id="g30" | ||
| 393 | transform="translate(24.021362,-46.545299)"> | ||
| 394 | <rect | ||
| 395 | style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none" | ||
| 396 | id="rect23" | ||
| 397 | width="131.09708" | ||
| 398 | height="5.8163381" | ||
| 399 | x="23.245802" | ||
| 400 | y="107.16314" /> | ||
| 401 | <path | ||
| 402 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 403 | d="m 37.884707,107.26337 v 5.53565" | ||
| 404 | id="path23" | ||
| 405 | sodipodi:nodetypes="cc" /> | ||
| 406 | <path | ||
| 407 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 408 | d="m 52.466233,107.26337 v 5.53565" | ||
| 409 | id="path24" | ||
| 410 | sodipodi:nodetypes="cc" /> | ||
| 411 | <path | ||
| 412 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 413 | d="m 67.047758,107.26337 v 5.53565" | ||
| 414 | id="path25" | ||
| 415 | sodipodi:nodetypes="cc" /> | ||
| 416 | <path | ||
| 417 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 418 | d="m 81.629283,107.26337 v 5.53565" | ||
| 419 | id="path26" | ||
| 420 | sodipodi:nodetypes="cc" /> | ||
| 421 | <path | ||
| 422 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 423 | d="m 96.210809,107.26337 v 5.53565" | ||
| 424 | id="path27" | ||
| 425 | sodipodi:nodetypes="cc" /> | ||
| 426 | <path | ||
| 427 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 428 | d="m 110.79233,107.26337 v 5.53565" | ||
| 429 | id="path28" | ||
| 430 | sodipodi:nodetypes="cc" /> | ||
| 431 | <path | ||
| 432 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 433 | d="m 125.37386,107.26337 v 5.53565" | ||
| 434 | id="path29" | ||
| 435 | sodipodi:nodetypes="cc" /> | ||
| 436 | <path | ||
| 437 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 438 | d="m 139.95538,107.26337 v 5.53565" | ||
| 439 | id="path30" | ||
| 440 | sodipodi:nodetypes="cc" /> | ||
| 441 | <text | ||
| 442 | xml:space="preserve" | ||
| 443 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 444 | x="86.713425" | ||
| 445 | y="110.1963" | ||
| 446 | id="text30"><tspan | ||
| 447 | sodipodi:role="line" | ||
| 448 | id="tspan30" | ||
| 449 | style="stroke:#006680;stroke-width:0.5" | ||
| 450 | x="86.713425" | ||
| 451 | y="110.1963">. . .</tspan></text> | ||
| 452 | </g> | ||
| 453 | <g | ||
| 454 | id="g38" | ||
| 455 | transform="translate(24.021362,-32.25779)"> | ||
| 456 | <rect | ||
| 457 | style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none" | ||
| 458 | id="rect30" | ||
| 459 | width="131.09708" | ||
| 460 | height="5.8163381" | ||
| 461 | x="23.245802" | ||
| 462 | y="107.16314" /> | ||
| 463 | <path | ||
| 464 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 465 | d="m 37.884707,107.26337 v 5.53565" | ||
| 466 | id="path31" | ||
| 467 | sodipodi:nodetypes="cc" /> | ||
| 468 | <path | ||
| 469 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 470 | d="m 52.466233,107.26337 v 5.53565" | ||
| 471 | id="path32" | ||
| 472 | sodipodi:nodetypes="cc" /> | ||
| 473 | <path | ||
| 474 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 475 | d="m 67.047758,107.26337 v 5.53565" | ||
| 476 | id="path33" | ||
| 477 | sodipodi:nodetypes="cc" /> | ||
| 478 | <path | ||
| 479 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 480 | d="m 81.629283,107.26337 v 5.53565" | ||
| 481 | id="path34" | ||
| 482 | sodipodi:nodetypes="cc" /> | ||
| 483 | <path | ||
| 484 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 485 | d="m 96.210809,107.26337 v 5.53565" | ||
| 486 | id="path35" | ||
| 487 | sodipodi:nodetypes="cc" /> | ||
| 488 | <path | ||
| 489 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 490 | d="m 110.79233,107.26337 v 5.53565" | ||
| 491 | id="path36" | ||
| 492 | sodipodi:nodetypes="cc" /> | ||
| 493 | <path | ||
| 494 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 495 | d="m 125.37386,107.26337 v 5.53565" | ||
| 496 | id="path37" | ||
| 497 | sodipodi:nodetypes="cc" /> | ||
| 498 | <path | ||
| 499 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 500 | d="m 139.95538,107.26337 v 5.53565" | ||
| 501 | id="path38" | ||
| 502 | sodipodi:nodetypes="cc" /> | ||
| 503 | <text | ||
| 504 | xml:space="preserve" | ||
| 505 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 506 | x="86.713425" | ||
| 507 | y="110.1963" | ||
| 508 | id="text38"><tspan | ||
| 509 | sodipodi:role="line" | ||
| 510 | id="tspan38" | ||
| 511 | style="stroke:#006680;stroke-width:0.5" | ||
| 512 | x="86.713425" | ||
| 513 | y="110.1963">. . .</tspan></text> | ||
| 514 | </g> | ||
| 515 | <text | ||
| 516 | xml:space="preserve" | ||
| 517 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 518 | x="113.37768" | ||
| 519 | y="68.463142" | ||
| 520 | id="text39"><tspan | ||
| 521 | sodipodi:role="line" | ||
| 522 | id="tspan39" | ||
| 523 | style="writing-mode:tb-rl;stroke-width:0.5" | ||
| 524 | x="113.37768" | ||
| 525 | y="68.463142">. . .</tspan></text> | ||
| 526 | <text | ||
| 527 | xml:space="preserve" | ||
| 528 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 529 | x="25.005701" | ||
| 530 | y="55.308445" | ||
| 531 | id="text40"><tspan | ||
| 532 | sodipodi:role="line" | ||
| 533 | x="25.005701" | ||
| 534 | y="55.308445" | ||
| 535 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 536 | id="tspan40">Pipe[1]</tspan></text> | ||
| 537 | <text | ||
| 538 | xml:space="preserve" | ||
| 539 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 540 | x="30.264952" | ||
| 541 | y="65.490654" | ||
| 542 | id="text41"><tspan | ||
| 543 | sodipodi:role="line" | ||
| 544 | x="30.264952" | ||
| 545 | y="65.490654" | ||
| 546 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 547 | id="tspan41">Queue[0]</tspan></text> | ||
| 548 | <text | ||
| 549 | xml:space="preserve" | ||
| 550 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 551 | x="30.264952" | ||
| 552 | y="79.249001" | ||
| 553 | id="text42"><tspan | ||
| 554 | sodipodi:role="line" | ||
| 555 | x="30.264952" | ||
| 556 | y="79.249001" | ||
| 557 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 558 | id="tspan42">Queue[n]</tspan></text> | ||
| 559 | <text | ||
| 560 | xml:space="preserve" | ||
| 561 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 562 | x="30.264952" | ||
| 563 | y="71.84066" | ||
| 564 | id="text43"><tspan | ||
| 565 | sodipodi:role="line" | ||
| 566 | x="30.264952" | ||
| 567 | y="71.84066" | ||
| 568 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 569 | id="tspan43">...</tspan></text> | ||
| 570 | </g> | ||
| 571 | <g | ||
| 572 | id="g64" | ||
| 573 | transform="translate(0,85.195881)"> | ||
| 574 | <rect | ||
| 575 | style="fill:none;stroke:#000000;stroke-width:0.878057;stroke-dasharray:none" | ||
| 576 | id="rect43" | ||
| 577 | width="167.79619" | ||
| 578 | height="24.831829" | ||
| 579 | x="14.21942" | ||
| 580 | y="57.862854" /> | ||
| 581 | <g | ||
| 582 | id="g51" | ||
| 583 | transform="translate(24.021362,-46.545299)"> | ||
| 584 | <rect | ||
| 585 | style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none" | ||
| 586 | id="rect44" | ||
| 587 | width="131.09708" | ||
| 588 | height="5.8163381" | ||
| 589 | x="23.245802" | ||
| 590 | y="107.16314" /> | ||
| 591 | <path | ||
| 592 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 593 | d="m 37.884707,107.26337 v 5.53565" | ||
| 594 | id="path44" | ||
| 595 | sodipodi:nodetypes="cc" /> | ||
| 596 | <path | ||
| 597 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 598 | d="m 52.466233,107.26337 v 5.53565" | ||
| 599 | id="path45" | ||
| 600 | sodipodi:nodetypes="cc" /> | ||
| 601 | <path | ||
| 602 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 603 | d="m 67.047758,107.26337 v 5.53565" | ||
| 604 | id="path46" | ||
| 605 | sodipodi:nodetypes="cc" /> | ||
| 606 | <path | ||
| 607 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 608 | d="m 81.629283,107.26337 v 5.53565" | ||
| 609 | id="path47" | ||
| 610 | sodipodi:nodetypes="cc" /> | ||
| 611 | <path | ||
| 612 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 613 | d="m 96.210809,107.26337 v 5.53565" | ||
| 614 | id="path48" | ||
| 615 | sodipodi:nodetypes="cc" /> | ||
| 616 | <path | ||
| 617 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 618 | d="m 110.79233,107.26337 v 5.53565" | ||
| 619 | id="path49" | ||
| 620 | sodipodi:nodetypes="cc" /> | ||
| 621 | <path | ||
| 622 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 623 | d="m 125.37386,107.26337 v 5.53565" | ||
| 624 | id="path50" | ||
| 625 | sodipodi:nodetypes="cc" /> | ||
| 626 | <path | ||
| 627 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 628 | d="m 139.95538,107.26337 v 5.53565" | ||
| 629 | id="path51" | ||
| 630 | sodipodi:nodetypes="cc" /> | ||
| 631 | <text | ||
| 632 | xml:space="preserve" | ||
| 633 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 634 | x="86.713425" | ||
| 635 | y="110.1963" | ||
| 636 | id="text51"><tspan | ||
| 637 | sodipodi:role="line" | ||
| 638 | id="tspan51" | ||
| 639 | style="stroke:#006680;stroke-width:0.5" | ||
| 640 | x="86.713425" | ||
| 641 | y="110.1963">. . .</tspan></text> | ||
| 642 | </g> | ||
| 643 | <g | ||
| 644 | id="g59" | ||
| 645 | transform="translate(24.021362,-32.25779)"> | ||
| 646 | <rect | ||
| 647 | style="fill:none;stroke:#006680;stroke-width:0.459999;stroke-dasharray:none" | ||
| 648 | id="rect51" | ||
| 649 | width="131.09708" | ||
| 650 | height="5.8163381" | ||
| 651 | x="23.245802" | ||
| 652 | y="107.16314" /> | ||
| 653 | <path | ||
| 654 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 655 | d="m 37.884707,107.26337 v 5.53565" | ||
| 656 | id="path52" | ||
| 657 | sodipodi:nodetypes="cc" /> | ||
| 658 | <path | ||
| 659 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 660 | d="m 52.466233,107.26337 v 5.53565" | ||
| 661 | id="path53" | ||
| 662 | sodipodi:nodetypes="cc" /> | ||
| 663 | <path | ||
| 664 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 665 | d="m 67.047758,107.26337 v 5.53565" | ||
| 666 | id="path54" | ||
| 667 | sodipodi:nodetypes="cc" /> | ||
| 668 | <path | ||
| 669 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 670 | d="m 81.629283,107.26337 v 5.53565" | ||
| 671 | id="path55" | ||
| 672 | sodipodi:nodetypes="cc" /> | ||
| 673 | <path | ||
| 674 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 675 | d="m 96.210809,107.26337 v 5.53565" | ||
| 676 | id="path56" | ||
| 677 | sodipodi:nodetypes="cc" /> | ||
| 678 | <path | ||
| 679 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 680 | d="m 110.79233,107.26337 v 5.53565" | ||
| 681 | id="path57" | ||
| 682 | sodipodi:nodetypes="cc" /> | ||
| 683 | <path | ||
| 684 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 685 | d="m 125.37386,107.26337 v 5.53565" | ||
| 686 | id="path58" | ||
| 687 | sodipodi:nodetypes="cc" /> | ||
| 688 | <path | ||
| 689 | style="fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 690 | d="m 139.95538,107.26337 v 5.53565" | ||
| 691 | id="path59" | ||
| 692 | sodipodi:nodetypes="cc" /> | ||
| 693 | <text | ||
| 694 | xml:space="preserve" | ||
| 695 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 696 | x="86.713425" | ||
| 697 | y="110.1963" | ||
| 698 | id="text59"><tspan | ||
| 699 | sodipodi:role="line" | ||
| 700 | id="tspan59" | ||
| 701 | style="stroke:#006680;stroke-width:0.5" | ||
| 702 | x="86.713425" | ||
| 703 | y="110.1963">. . .</tspan></text> | ||
| 704 | </g> | ||
| 705 | <text | ||
| 706 | xml:space="preserve" | ||
| 707 | style="font-size:3.175px;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:none;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 708 | x="113.37768" | ||
| 709 | y="68.463142" | ||
| 710 | id="text60"><tspan | ||
| 711 | sodipodi:role="line" | ||
| 712 | id="tspan60" | ||
| 713 | style="writing-mode:tb-rl;stroke-width:0.5" | ||
| 714 | x="113.37768" | ||
| 715 | y="68.463142">. . .</tspan></text> | ||
| 716 | <text | ||
| 717 | xml:space="preserve" | ||
| 718 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 719 | x="25.005701" | ||
| 720 | y="55.308445" | ||
| 721 | id="text61"><tspan | ||
| 722 | sodipodi:role="line" | ||
| 723 | x="25.005701" | ||
| 724 | y="55.308445" | ||
| 725 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 726 | id="tspan61">Pipe[n]</tspan></text> | ||
| 727 | <text | ||
| 728 | xml:space="preserve" | ||
| 729 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 730 | x="30.264952" | ||
| 731 | y="65.490654" | ||
| 732 | id="text62"><tspan | ||
| 733 | sodipodi:role="line" | ||
| 734 | x="30.264952" | ||
| 735 | y="65.490654" | ||
| 736 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 737 | id="tspan62">Queue[0]</tspan></text> | ||
| 738 | <text | ||
| 739 | xml:space="preserve" | ||
| 740 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 741 | x="30.264952" | ||
| 742 | y="79.249001" | ||
| 743 | id="text63"><tspan | ||
| 744 | sodipodi:role="line" | ||
| 745 | x="30.264952" | ||
| 746 | y="79.249001" | ||
| 747 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 748 | id="tspan63">Queue[n]</tspan></text> | ||
| 749 | <text | ||
| 750 | xml:space="preserve" | ||
| 751 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 752 | x="30.264952" | ||
| 753 | y="71.84066" | ||
| 754 | id="text64"><tspan | ||
| 755 | sodipodi:role="line" | ||
| 756 | x="30.264952" | ||
| 757 | y="71.84066" | ||
| 758 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 759 | id="tspan64">...</tspan></text> | ||
| 760 | </g> | ||
| 761 | <text | ||
| 762 | xml:space="preserve" | ||
| 763 | style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.76111px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:lr-tb;direction:ltr;text-anchor:start;fill:#000000;fill-opacity:0;stroke:#006680;stroke-width:0.499999;stroke-dasharray:none" | ||
| 764 | x="92.18071" | ||
| 765 | y="128.21965" | ||
| 766 | id="text65"><tspan | ||
| 767 | sodipodi:role="line" | ||
| 768 | id="tspan65" | ||
| 769 | style="font-size:7.76111px;writing-mode:tb-rl;fill:#1a1a1a;stroke:#000000;stroke-width:0.5" | ||
| 770 | x="92.18071" | ||
| 771 | y="128.21965">...</tspan></text> | ||
| 772 | <text | ||
| 773 | xml:space="preserve" | ||
| 774 | style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:7.7611px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none" | ||
| 775 | x="198.72205" | ||
| 776 | y="80.708267" | ||
| 777 | id="text66"><tspan | ||
| 778 | sodipodi:role="line" | ||
| 779 | id="tspan66" | ||
| 780 | style="stroke-width:0.5" | ||
| 781 | x="198.72205" | ||
| 782 | y="80.708267" /></text> | ||
| 783 | <text | ||
| 784 | xml:space="preserve" | ||
| 785 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 786 | x="97.905846" | ||
| 787 | y="44.725101" | ||
| 788 | id="text68"><tspan | ||
| 789 | sodipodi:role="line" | ||
| 790 | x="97.905846" | ||
| 791 | y="44.725101" | ||
| 792 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 793 | id="tspan68">Hardware Block</tspan></text> | ||
| 794 | <text | ||
| 795 | xml:space="preserve" | ||
| 796 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;writing-mode:vertical-lr;text-orientation:upright;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 797 | x="236.36934" | ||
| 798 | y="112.10503" | ||
| 799 | id="text68-7"><tspan | ||
| 800 | sodipodi:role="line" | ||
| 801 | x="236.36934" | ||
| 802 | y="112.10503" | ||
| 803 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;writing-mode:vertical-lr;text-orientation:upright;text-anchor:middle;stroke-width:0.0690111" | ||
| 804 | id="tspan68-4">EXECUTION</tspan></text> | ||
| 805 | <rect | ||
| 806 | style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.940575;stroke-dasharray:7.5246, 0.940575;stroke-dashoffset:0" | ||
| 807 | id="rect68-1" | ||
| 808 | width="68.749969" | ||
| 809 | height="141.2751" | ||
| 810 | x="-129.49162" | ||
| 811 | y="37.881134" | ||
| 812 | ry="0" /> | ||
| 813 | <text | ||
| 814 | xml:space="preserve" | ||
| 815 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 816 | x="-114.4223" | ||
| 817 | y="44.966106" | ||
| 818 | id="text68-2"><tspan | ||
| 819 | sodipodi:role="line" | ||
| 820 | x="-114.4223" | ||
| 821 | y="44.966106" | ||
| 822 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 823 | id="tspan68-9">Memory</tspan></text> | ||
| 824 | <text | ||
| 825 | xml:space="preserve" | ||
| 826 | style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:12.7px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-orientation:upright;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:0.499999;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 827 | x="212.6013" | ||
| 828 | y="64.823341" | ||
| 829 | id="text69"><tspan | ||
| 830 | sodipodi:role="line" | ||
| 831 | id="tspan69" | ||
| 832 | style="font-size:12.7px;stroke-width:0.5" | ||
| 833 | x="212.6013" | ||
| 834 | y="64.823341" /></text> | ||
| 835 | <g | ||
| 836 | id="g72"> | ||
| 837 | <path | ||
| 838 | style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111" | ||
| 839 | d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z" | ||
| 840 | id="text70" | ||
| 841 | aria-label="}" | ||
| 842 | sodipodi:nodetypes="ccccsccccccccscscscscc" /> | ||
| 843 | <path | ||
| 844 | style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)" | ||
| 845 | d="M 190.40199,70.278769 H 224.6663" | ||
| 846 | id="path70" /> | ||
| 847 | </g> | ||
| 848 | <path | ||
| 849 | style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111" | ||
| 850 | d="m -88.44114,74.337891 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 V 62.57347 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 v -9.376826 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z" | ||
| 851 | id="text70-1" | ||
| 852 | aria-label="}" | ||
| 853 | sodipodi:nodetypes="ccccsccccccccscscscscc" /> | ||
| 854 | <text | ||
| 855 | xml:space="preserve" | ||
| 856 | style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16.9333px;font-family:Serif;-inkscape-font-specification:Serif;text-align:start;writing-mode:tb-rl;direction:ltr;text-orientation:upright;text-anchor:start;fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 857 | x="200.88817" | ||
| 858 | y="38.990276" | ||
| 859 | id="text72"><tspan | ||
| 860 | sodipodi:role="line" | ||
| 861 | id="tspan72" | ||
| 862 | style="stroke-width:1.25" | ||
| 863 | x="200.88817" | ||
| 864 | y="38.990276" /></text> | ||
| 865 | <text | ||
| 866 | xml:space="preserve" | ||
| 867 | style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 868 | x="205.12828" | ||
| 869 | y="60.090775" | ||
| 870 | id="text73"><tspan | ||
| 871 | sodipodi:role="line" | ||
| 872 | x="205.12828" | ||
| 873 | y="60.090775" | ||
| 874 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 875 | id="tspan73">e.g.,:</tspan><tspan | ||
| 876 | sodipodi:role="line" | ||
| 877 | x="205.12828" | ||
| 878 | y="68.028275" | ||
| 879 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 880 | id="tspan74">queue[0]</tspan></text> | ||
| 881 | <g | ||
| 882 | id="g75" | ||
| 883 | transform="translate(0,40.745853)"> | ||
| 884 | <path | ||
| 885 | style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111" | ||
| 886 | d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z" | ||
| 887 | id="path74" | ||
| 888 | aria-label="}" | ||
| 889 | sodipodi:nodetypes="ccccsccccccccscscscscc" /> | ||
| 890 | <path | ||
| 891 | style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)" | ||
| 892 | d="M 190.40199,70.278769 H 224.6663" | ||
| 893 | id="path75" /> | ||
| 894 | </g> | ||
| 895 | <text | ||
| 896 | xml:space="preserve" | ||
| 897 | style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 898 | x="205.12828" | ||
| 899 | y="100.83664" | ||
| 900 | id="text76"><tspan | ||
| 901 | sodipodi:role="line" | ||
| 902 | x="205.12828" | ||
| 903 | y="100.83664" | ||
| 904 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 905 | id="tspan75">e.g.,:</tspan><tspan | ||
| 906 | sodipodi:role="line" | ||
| 907 | x="205.12828" | ||
| 908 | y="108.77414" | ||
| 909 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 910 | id="tspan76">queue[4]</tspan></text> | ||
| 911 | <g | ||
| 912 | id="g77" | ||
| 913 | transform="translate(0,85.725048)"> | ||
| 914 | <path | ||
| 915 | style="font-weight:bold;font-size:16.9333px;line-height:1.25;-inkscape-font-specification:'sans-serif Bold';text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;stroke-width:0.0690111" | ||
| 916 | d="m 185.24734,83.96512 h 1.28693 c 1.20226,0.01693 2.09973,-1.015998 2.0828,-2.370662 v -9.393759 c -0.0169,-0.711199 0.33866,-1.066798 1.10066,-1.066798 h 0.2032 0.0677 v -1.557864 c -1.0668,0.01693 -1.38853,-0.270933 -1.37159,-1.236131 V 58.96308 c 0.0339,-1.354664 -0.88054,-2.387595 -2.0828,-2.370662 h -1.28693 v 1.676397 h 0.62653 c 0.57573,0 0.77893,0.321733 0.762,1.100664 v 9.122827 c 0,1.219198 0.44027,1.710263 1.64253,1.862663 -1.20226,0.135466 -1.64253,0.626532 -1.64253,1.84573 v 9.122826 c 0.0169,0.660399 -0.23707,0.965198 -0.762,0.965198 h -0.62653 z" | ||
| 917 | id="path76" | ||
| 918 | aria-label="}" | ||
| 919 | sodipodi:nodetypes="ccccsccccccccscscscscc" /> | ||
| 920 | <path | ||
| 921 | style="fill:#1a1a1a;fill-opacity:0;stroke:#000000;stroke-width:1.25;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#ArrowWideHeavy)" | ||
| 922 | d="M 190.40199,70.278769 H 224.6663" | ||
| 923 | id="path77" /> | ||
| 924 | </g> | ||
| 925 | <text | ||
| 926 | xml:space="preserve" | ||
| 927 | style="font-style:normal;font-weight:normal;font-size:6.35px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 928 | x="205.12828" | ||
| 929 | y="145.81558" | ||
| 930 | id="text78"><tspan | ||
| 931 | sodipodi:role="line" | ||
| 932 | x="205.12828" | ||
| 933 | y="145.81558" | ||
| 934 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 935 | id="tspan77">e.g.,:</tspan><tspan | ||
| 936 | sodipodi:role="line" | ||
| 937 | x="205.12828" | ||
| 938 | y="153.75308" | ||
| 939 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:6.35px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 940 | id="tspan78">queue[n]</tspan></text> | ||
| 941 | <g | ||
| 942 | id="g81"> | ||
| 943 | <text | ||
| 944 | xml:space="preserve" | ||
| 945 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 946 | x="-18.502264" | ||
| 947 | y="65.642387" | ||
| 948 | id="text79"><tspan | ||
| 949 | sodipodi:role="line" | ||
| 950 | x="-18.502264" | ||
| 951 | y="65.642387" | ||
| 952 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111" | ||
| 953 | id="tspan79">HQD</tspan></text> | ||
| 954 | <ellipse | ||
| 955 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 956 | id="path79" | ||
| 957 | cx="-18.407015" | ||
| 958 | cy="63.2188" | ||
| 959 | rx="13.317666" | ||
| 960 | ry="4.4124799" /> | ||
| 961 | <path | ||
| 962 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)" | ||
| 963 | d="M -4.6361224,63.2188 H 10.606988" | ||
| 964 | id="path80" /> | ||
| 965 | </g> | ||
| 966 | <g | ||
| 967 | id="g82" | ||
| 968 | transform="translate(0,14.287503)"> | ||
| 969 | <text | ||
| 970 | xml:space="preserve" | ||
| 971 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 972 | x="-18.502264" | ||
| 973 | y="65.642387" | ||
| 974 | id="text81"><tspan | ||
| 975 | sodipodi:role="line" | ||
| 976 | x="-18.502264" | ||
| 977 | y="65.642387" | ||
| 978 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111" | ||
| 979 | id="tspan81">HQD</tspan></text> | ||
| 980 | <ellipse | ||
| 981 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 982 | id="ellipse81" | ||
| 983 | cx="-18.407015" | ||
| 984 | cy="63.2188" | ||
| 985 | rx="13.317666" | ||
| 986 | ry="4.4124799" /> | ||
| 987 | <path | ||
| 988 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)" | ||
| 989 | d="M -4.6361224,63.2188 H 10.606988" | ||
| 990 | id="path81" /> | ||
| 991 | </g> | ||
| 992 | <g | ||
| 993 | id="g83" | ||
| 994 | transform="translate(0,40.745853)"> | ||
| 995 | <text | ||
| 996 | xml:space="preserve" | ||
| 997 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 998 | x="-18.502264" | ||
| 999 | y="65.642387" | ||
| 1000 | id="text82"><tspan | ||
| 1001 | sodipodi:role="line" | ||
| 1002 | x="-18.502264" | ||
| 1003 | y="65.642387" | ||
| 1004 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111" | ||
| 1005 | id="tspan82">HQD</tspan></text> | ||
| 1006 | <ellipse | ||
| 1007 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 1008 | id="ellipse82" | ||
| 1009 | cx="-18.407015" | ||
| 1010 | cy="63.2188" | ||
| 1011 | rx="13.317666" | ||
| 1012 | ry="4.4124799" /> | ||
| 1013 | <path | ||
| 1014 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)" | ||
| 1015 | d="M -4.6361224,63.2188 H 10.606988" | ||
| 1016 | id="path82" /> | ||
| 1017 | </g> | ||
| 1018 | <g | ||
| 1019 | id="g84" | ||
| 1020 | transform="translate(0,55.033362)"> | ||
| 1021 | <text | ||
| 1022 | xml:space="preserve" | ||
| 1023 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1024 | x="-18.502264" | ||
| 1025 | y="65.642387" | ||
| 1026 | id="text83"><tspan | ||
| 1027 | sodipodi:role="line" | ||
| 1028 | x="-18.502264" | ||
| 1029 | y="65.642387" | ||
| 1030 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111" | ||
| 1031 | id="tspan83">HQD</tspan></text> | ||
| 1032 | <ellipse | ||
| 1033 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 1034 | id="ellipse83" | ||
| 1035 | cx="-18.407015" | ||
| 1036 | cy="63.2188" | ||
| 1037 | rx="13.317666" | ||
| 1038 | ry="4.4124799" /> | ||
| 1039 | <path | ||
| 1040 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)" | ||
| 1041 | d="M -4.6361224,63.2188 H 10.606988" | ||
| 1042 | id="path83" /> | ||
| 1043 | </g> | ||
| 1044 | <g | ||
| 1045 | id="g85" | ||
| 1046 | transform="translate(0,85.195881)"> | ||
| 1047 | <text | ||
| 1048 | xml:space="preserve" | ||
| 1049 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1050 | x="-18.502264" | ||
| 1051 | y="65.642387" | ||
| 1052 | id="text84"><tspan | ||
| 1053 | sodipodi:role="line" | ||
| 1054 | x="-18.502264" | ||
| 1055 | y="65.642387" | ||
| 1056 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111" | ||
| 1057 | id="tspan84">HQD</tspan></text> | ||
| 1058 | <ellipse | ||
| 1059 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 1060 | id="ellipse84" | ||
| 1061 | cx="-18.407015" | ||
| 1062 | cy="63.2188" | ||
| 1063 | rx="13.317666" | ||
| 1064 | ry="4.4124799" /> | ||
| 1065 | <path | ||
| 1066 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)" | ||
| 1067 | d="M -4.6361224,63.2188 H 10.606988" | ||
| 1068 | id="path84" /> | ||
| 1069 | </g> | ||
| 1070 | <g | ||
| 1071 | id="g86" | ||
| 1072 | transform="translate(0,99.48339)"> | ||
| 1073 | <text | ||
| 1074 | xml:space="preserve" | ||
| 1075 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1076 | x="-18.502264" | ||
| 1077 | y="65.642387" | ||
| 1078 | id="text85"><tspan | ||
| 1079 | sodipodi:role="line" | ||
| 1080 | x="-18.502264" | ||
| 1081 | y="65.642387" | ||
| 1082 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;fill:#800000;stroke-width:0.0690111" | ||
| 1083 | id="tspan85">HQD</tspan></text> | ||
| 1084 | <ellipse | ||
| 1085 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 1086 | id="ellipse85" | ||
| 1087 | cx="-18.407015" | ||
| 1088 | cy="63.2188" | ||
| 1089 | rx="13.317666" | ||
| 1090 | ry="4.4124799" /> | ||
| 1091 | <path | ||
| 1092 | style="fill:#800000;fill-opacity:0;stroke:#800000;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Triangle)" | ||
| 1093 | d="M -4.6361224,63.2188 H 10.606988" | ||
| 1094 | id="path85" /> | ||
| 1095 | </g> | ||
| 1096 | <text | ||
| 1097 | xml:space="preserve" | ||
| 1098 | style="font-style:normal;font-weight:normal;font-size:5.64444px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1099 | x="-35.2131" | ||
| 1100 | y="54.673237" | ||
| 1101 | id="text86"><tspan | ||
| 1102 | sodipodi:role="line" | ||
| 1103 | x="-35.2131" | ||
| 1104 | y="54.673237" | ||
| 1105 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1106 | id="tspan86">Registers</tspan></text> | ||
| 1107 | <path | ||
| 1108 | style="fill:#800000;fill-opacity:0;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:1, 1;stroke-dashoffset:0;marker-end:url(#ArrowWide)" | ||
| 1109 | d="m -45.247972,57.442462 v 5.888987 h 11.344412" | ||
| 1110 | id="path86" | ||
| 1111 | sodipodi:nodetypes="ccc" /> | ||
| 1112 | <text | ||
| 1113 | xml:space="preserve" | ||
| 1114 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1115 | x="-74.441521" | ||
| 1116 | y="82.441582" | ||
| 1117 | id="text87"><tspan | ||
| 1118 | sodipodi:role="line" | ||
| 1119 | x="-74.441521" | ||
| 1120 | y="82.441582" | ||
| 1121 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1122 | id="tspan87">MQD</tspan></text> | ||
| 1123 | <text | ||
| 1124 | xml:space="preserve" | ||
| 1125 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1126 | x="-74.441521" | ||
| 1127 | y="99.240776" | ||
| 1128 | id="text88"><tspan | ||
| 1129 | sodipodi:role="line" | ||
| 1130 | x="-74.441521" | ||
| 1131 | y="99.240776" | ||
| 1132 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1133 | id="tspan88">MQD</tspan></text> | ||
| 1134 | <text | ||
| 1135 | xml:space="preserve" | ||
| 1136 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1137 | x="-74.441521" | ||
| 1138 | y="116.03998" | ||
| 1139 | id="text89"><tspan | ||
| 1140 | sodipodi:role="line" | ||
| 1141 | x="-74.441521" | ||
| 1142 | y="116.03998" | ||
| 1143 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1144 | id="tspan89">MQD</tspan></text> | ||
| 1145 | <text | ||
| 1146 | xml:space="preserve" | ||
| 1147 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1148 | x="-74.441521" | ||
| 1149 | y="132.83917" | ||
| 1150 | id="text90"><tspan | ||
| 1151 | sodipodi:role="line" | ||
| 1152 | x="-74.441521" | ||
| 1153 | y="132.83917" | ||
| 1154 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1155 | id="tspan90">MQD</tspan></text> | ||
| 1156 | <text | ||
| 1157 | xml:space="preserve" | ||
| 1158 | style="font-style:normal;font-weight:normal;font-size:7.05556px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1159 | x="-74.441521" | ||
| 1160 | y="168.3002" | ||
| 1161 | id="text91"><tspan | ||
| 1162 | sodipodi:role="line" | ||
| 1163 | x="-74.441521" | ||
| 1164 | y="168.3002" | ||
| 1165 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:7.05556px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1166 | id="tspan91">MQD</tspan></text> | ||
| 1167 | <text | ||
| 1168 | xml:space="preserve" | ||
| 1169 | style="font-style:normal;font-weight:normal;font-size:10.5833px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1170 | x="-74.441521" | ||
| 1171 | y="148.1461" | ||
| 1172 | id="text92"><tspan | ||
| 1173 | sodipodi:role="line" | ||
| 1174 | x="-74.441521" | ||
| 1175 | y="148.1461" | ||
| 1176 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:10.5833px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;writing-mode:tb-rl;text-anchor:middle;stroke-width:0.0690111" | ||
| 1177 | id="tspan92">...</tspan></text> | ||
| 1178 | <g | ||
| 1179 | id="g97" | ||
| 1180 | transform="translate(-5.8208336)"> | ||
| 1181 | <rect | ||
| 1182 | style="fill:#aaffcc;fill-opacity:1;stroke:#1a1a1a;stroke-width:0.5;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 1183 | id="rect96" | ||
| 1184 | width="58.726093" | ||
| 1185 | height="27.598055" | ||
| 1186 | x="-142.96434" | ||
| 1187 | y="46.852512" /> | ||
| 1188 | <text | ||
| 1189 | xml:space="preserve" | ||
| 1190 | style="font-style:normal;font-weight:normal;font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:start;letter-spacing:0px;word-spacing:0px;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1191 | x="-139.74622" | ||
| 1192 | y="52.62756" | ||
| 1193 | id="text93"><tspan | ||
| 1194 | sodipodi:role="line" | ||
| 1195 | x="-139.74622" | ||
| 1196 | y="52.62756" | ||
| 1197 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1198 | id="tspan93">HQD Registers</tspan><tspan | ||
| 1199 | sodipodi:role="line" | ||
| 1200 | x="-139.74622" | ||
| 1201 | y="57.91922" | ||
| 1202 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1203 | id="tspan95">Queue Address in the GPU</tspan><tspan | ||
| 1204 | sodipodi:role="line" | ||
| 1205 | x="-139.74622" | ||
| 1206 | y="63.210884" | ||
| 1207 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1208 | id="tspan96">Doorbell</tspan><tspan | ||
| 1209 | sodipodi:role="line" | ||
| 1210 | x="-139.74622" | ||
| 1211 | y="68.502548" | ||
| 1212 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1213 | id="tspan97">...</tspan><tspan | ||
| 1214 | sodipodi:role="line" | ||
| 1215 | x="-139.74622" | ||
| 1216 | y="73.794212" | ||
| 1217 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1218 | id="tspan94" /></text> | ||
| 1219 | </g> | ||
| 1220 | <text | ||
| 1221 | xml:space="preserve" | ||
| 1222 | style="font-style:normal;font-weight:normal;font-size:4.23333px;line-height:1.25;font-family:sans-serif;text-align:start;letter-spacing:0px;word-spacing:0px;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1223 | x="-61.293022" | ||
| 1224 | y="-19.380915" | ||
| 1225 | id="text93-1"><tspan | ||
| 1226 | sodipodi:role="line" | ||
| 1227 | x="-61.293022" | ||
| 1228 | y="-19.380915" | ||
| 1229 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1230 | id="tspan104">SWITCH QUEUE:</tspan><tspan | ||
| 1231 | sodipodi:role="line" | ||
| 1232 | x="-61.293022" | ||
| 1233 | y="-14.089252" | ||
| 1234 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1235 | id="tspan101">WAIT FOR HQD_ACTIVE = 0</tspan><tspan | ||
| 1236 | sodipodi:role="line" | ||
| 1237 | x="-61.293022" | ||
| 1238 | y="-8.7975903" | ||
| 1239 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1240 | id="tspan105">SAVE QUEUE STATE TO THE MQD</tspan><tspan | ||
| 1241 | sodipodi:role="line" | ||
| 1242 | x="-61.293022" | ||
| 1243 | y="-3.505928" | ||
| 1244 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1245 | id="tspan102">COPY NEW MQD STATE</tspan><tspan | ||
| 1246 | sodipodi:role="line" | ||
| 1247 | x="-61.293022" | ||
| 1248 | y="1.7857342" | ||
| 1249 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:4.23333px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:start;text-anchor:start;stroke-width:0.0690111" | ||
| 1250 | id="tspan103">SET HQD_ACTIVE = 1</tspan></text> | ||
| 1251 | <circle | ||
| 1252 | style="fill:#ffeeaa;fill-opacity:1;stroke:#1a1a1a;stroke-width:1.88976;stroke-dasharray:none;stroke-dashoffset:0" | ||
| 1253 | id="path97" | ||
| 1254 | cx="0" | ||
| 1255 | cy="0" | ||
| 1256 | r="0" | ||
| 1257 | transform="matrix(0.26458333,0,0,0.26458333,-149.03517,37.347779)" /> | ||
| 1258 | <text | ||
| 1259 | xml:space="preserve" | ||
| 1260 | style="font-style:normal;font-weight:normal;font-size:5.64444px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.0690111" | ||
| 1261 | x="-36.659206" | ||
| 1262 | y="-44.828983" | ||
| 1263 | id="text106"><tspan | ||
| 1264 | sodipodi:role="line" | ||
| 1265 | x="-36.659206" | ||
| 1266 | y="-44.828983" | ||
| 1267 | style="font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;font-size:5.64444px;font-family:sans-serif;-inkscape-font-specification:'sans-serif Bold';text-align:center;text-anchor:middle;stroke-width:0.0690111" | ||
| 1268 | id="tspan106">Firmware</tspan></text> | ||
| 1269 | <path | ||
| 1270 | style="fill:none;fill-opacity:1;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-start:url(#marker109);marker-end:url(#ArrowTriangleStylized)" | ||
| 1271 | d="M -84.242601,-9.1838245 H -98.041629 V 33.17598" | ||
| 1272 | id="path106" /> | ||
| 1273 | <path | ||
| 1274 | style="fill:none;fill-opacity:1;stroke:#1a1a1a;stroke-width:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Dot)" | ||
| 1275 | d="M 9.9745536,-9.3442784 H 29.549918 V 37.170287" | ||
| 1276 | id="path107" | ||
| 1277 | sodipodi:nodetypes="ccc" /> | ||
| 1278 | </g> | ||
| 1279 | </svg> | ||
| @@ -79,6 +79,41 @@ static int fence_cmp(const void *_a, const void *_b) · 78 unmodified lines | |||
| 79 | return 0; | 79 | return 0; |
| 80 | } | 80 | } |
| 81 | 81 | ||
| 82 | /** | ||
| 83 | * dma_fence_dedup_array - Sort and deduplicate an array of dma_fence pointers | ||
| 84 | * @fences: Array of dma_fence pointers to be deduplicated | ||
| 85 | * @num_fences: Number of entries in the @fences array | ||
| 86 | * | ||
| 87 | * Sorts the input array by context, then removes duplicate | ||
| 88 | * fences with the same context, keeping only the most recent one. | ||
| 89 | * | ||
| 90 | * The array is modified in-place and unreferenced duplicate fences are released | ||
| 91 | * via dma_fence_put(). The function returns the new number of fences after | ||
| 92 | * deduplication. | ||
| 93 | * | ||
| 94 | * Return: Number of unique fences remaining in the array. | ||
| 95 | */ | ||
| 96 | int dma_fence_dedup_array(struct dma_fence **fences, int num_fences) | ||
| 97 | { | ||
| 98 | int i, j; | ||
| 99 | |||
| 100 | sort(fences, num_fences, sizeof(*fences), fence_cmp, NULL); | ||
| 101 | |||
| 102 | /* | ||
| 103 | * Only keep the most recent fence for each context. | ||
| 104 | */ | ||
| 105 | j = 0; | ||
| 106 | for (i = 1; i < num_fences; i++) { | ||
| 107 | if (fences[i]->context == fences[j]->context) | ||
| 108 | dma_fence_put(fences[i]); | ||
| 109 | else | ||
| 110 | fences[++j] = fences[i]; | ||
| 111 | } | ||
| 112 | |||
| 113 | return ++j; | ||
| 114 | } | ||
| 115 | EXPORT_SYMBOL_GPL(dma_fence_dedup_array); | ||
| 116 | |||
| 82 | /* Implementation for the dma_fence_merge() marco, don't use directly */ | 117 | /* Implementation for the dma_fence_merge() marco, don't use directly */ |
| 83 | struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, | 118 | struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, |
| 84 | struct dma_fence **fences, | 119 | struct dma_fence **fences, |
| @@ -87,7 +122,7 @@ struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, · 2 unmodified lines | |||
| 87 | struct dma_fence *tmp, *unsignaled = NULL, **array; | 122 | struct dma_fence *tmp, *unsignaled = NULL, **array; |
| 88 | struct dma_fence_array *result; | 123 | struct dma_fence_array *result; |
| 89 | ktime_t timestamp; | 124 | ktime_t timestamp; |
| 90 | int i, j, count; | 125 | int i, count; |
| 91 | 126 | ||
| 92 | count = 0; | 127 | count = 0; |
| 93 | timestamp = ns_to_ktime(0); | 128 | timestamp = ns_to_ktime(0); |
| @@ -141,19 +176,7 @@ struct dma_fence *__dma_fence_unwrap_merge(unsigned int num_fences, · 47 unmodified lines | |||
| 141 | if (count == 0 || count == 1) | 176 | if (count == 0 || count == 1) |
| 142 | goto return_fastpath; | 177 | goto return_fastpath; |
| 143 | 178 | ||
| 144 | sort(array, count, sizeof(*array), fence_cmp, NULL); | 179 | count = dma_fence_dedup_array(array, count); |
| 145 | |||
| 146 | /* | ||
| 147 | * Only keep the most recent fence for each context. | ||
| 148 | */ | ||
| 149 | j = 0; | ||
| 150 | for (i = 1; i < count; i++) { | ||
| 151 | if (array[i]->context == array[j]->context) | ||
| 152 | dma_fence_put(array[i]); | ||
| 153 | else | ||
| 154 | array[++j] = array[i]; | ||
| 155 | } | ||
| 156 | count = ++j; | ||
| 157 | 180 | ||
| 158 | if (count > 1) { | 181 | if (count > 1) { |
| 159 | result = dma_fence_array_create(count, array, | 182 | result = dma_fence_array_create(count, array, |
| @@ -66,7 +66,7 @@ amdgpu-y += amdgpu_device.o amdgpu_doorbell_mgr.o amdgpu_kms.o \ · 65 unmodified lines | |||
| 66 | amdgpu_fw_attestation.o amdgpu_securedisplay.o \ | 66 | amdgpu_fw_attestation.o amdgpu_securedisplay.o \ |
| 67 | amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o \ | 67 | amdgpu_eeprom.o amdgpu_mca.o amdgpu_psp_ta.o amdgpu_lsdma.o \ |
| 68 | amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_aca.o amdgpu_dev_coredump.o \ | 68 | amdgpu_ring_mux.o amdgpu_xcp.o amdgpu_seq64.o amdgpu_aca.o amdgpu_dev_coredump.o \ |
| 69 | amdgpu_cper.o | 69 | amdgpu_cper.o amdgpu_userq_fence.o amdgpu_eviction_fence.o |
| 70 | 70 | ||
| 71 | amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o | 71 | amdgpu-$(CONFIG_PROC_FS) += amdgpu_fdinfo.o |
| 72 | 72 | ||
| @@ -174,7 +174,10 @@ amdgpu-y += \ · 101 unmodified lines | |||
| 174 | amdgpu-y += \ | 174 | amdgpu-y += \ |
| 175 | amdgpu_mes.o \ | 175 | amdgpu_mes.o \ |
| 176 | mes_v11_0.o \ | 176 | mes_v11_0.o \ |
| 177 | mes_v12_0.o | 177 | mes_v12_0.o \ |
| 178 | |||
| 179 | # add GFX userqueue support | ||
| 180 | amdgpu-y += mes_userqueue.o | ||
| 178 | 181 | ||
| 179 | # add UVD block | 182 | # add UVD block |
| 180 | amdgpu-y += \ | 183 | amdgpu-y += \ |
| @@ -253,6 +256,8 @@ amdgpu-y += \ · 72 unmodified lines | |||
| 253 | # add amdkfd interfaces | 256 | # add amdkfd interfaces |
| 254 | amdgpu-y += amdgpu_amdkfd.o | 257 | amdgpu-y += amdgpu_amdkfd.o |
| 255 | 258 | ||
| 259 | # add gfx usermode queue | ||
| 260 | amdgpu-y += amdgpu_userq.o | ||
| 256 | 261 | ||
| 257 | ifneq ($(CONFIG_HSA_AMD),) | 262 | ifneq ($(CONFIG_HSA_AMD),) |
| 258 | AMDKFD_PATH := ../amdkfd | 263 | AMDKFD_PATH := ../amdkfd |
| @@ -113,6 +113,8 @@ · 112 unmodified lines | |||
| 113 | #include "amdgpu_xcp.h" | 113 | #include "amdgpu_xcp.h" |
| 114 | #include "amdgpu_seq64.h" | 114 | #include "amdgpu_seq64.h" |
| 115 | #include "amdgpu_reg_state.h" | 115 | #include "amdgpu_reg_state.h" |
| 116 | #include "amdgpu_userq.h" | ||
| 117 | #include "amdgpu_eviction_fence.h" | ||
| 116 | #if defined(CONFIG_DRM_AMD_ISP) | 118 | #if defined(CONFIG_DRM_AMD_ISP) |
| 117 | #include "amdgpu_isp.h" | 119 | #include "amdgpu_isp.h" |
| 118 | #endif | 120 | #endif |
| @@ -228,7 +230,7 @@ extern int amdgpu_force_asic_type; · 109 unmodified lines | |||
| 228 | extern int amdgpu_smartshift_bias; | 230 | extern int amdgpu_smartshift_bias; |
| 229 | extern int amdgpu_use_xgmi_p2p; | 231 | extern int amdgpu_use_xgmi_p2p; |
| 230 | extern int amdgpu_mtype_local; | 232 | extern int amdgpu_mtype_local; |
| 231 | extern bool enforce_isolation; | 233 | extern int amdgpu_enforce_isolation; |
| 232 | #ifdef CONFIG_HSA_AMD | 234 | #ifdef CONFIG_HSA_AMD |
| 233 | extern int sched_policy; | 235 | extern int sched_policy; |
| 234 | extern bool debug_evictions; | 236 | extern bool debug_evictions; |
| @@ -266,8 +268,10 @@ extern int amdgpu_umsch_mm_fwlog; · 31 unmodified lines | |||
| 266 | 268 | ||
| 267 | extern int amdgpu_user_partt_mode; | 269 | extern int amdgpu_user_partt_mode; |
| 268 | extern int amdgpu_agp; | 270 | extern int amdgpu_agp; |
| 271 | extern int amdgpu_rebar; | ||
| 269 | 272 | ||
| 270 | extern int amdgpu_wbrf; | 273 | extern int amdgpu_wbrf; |
| 274 | extern int amdgpu_user_queue; | ||
| 271 | 275 | ||
| 272 | #define AMDGPU_VM_MAX_NUM_CTX 4096 | 276 | #define AMDGPU_VM_MAX_NUM_CTX 4096 |
| 273 | #define AMDGPU_SG_THRESHOLD (256*1024*1024) | 277 | #define AMDGPU_SG_THRESHOLD (256*1024*1024) |
| @@ -488,7 +492,6 @@ struct amdgpu_flip_work { · 214 unmodified lines | |||
| 488 | bool async; | 492 | bool async; |
| 489 | }; | 493 | }; |
| 490 | 494 | ||
| 491 | |||
| 492 | /* | 495 | /* |
| 493 | * file private structure | 496 | * file private structure |
| 494 | */ | 497 | */ |
| @@ -501,6 +504,11 @@ struct amdgpu_fpriv { · 6 unmodified lines | |||
| 501 | struct mutex bo_list_lock; | 504 | struct mutex bo_list_lock; |
| 502 | struct idr bo_list_handles; | 505 | struct idr bo_list_handles; |
| 503 | struct amdgpu_ctx_mgr ctx_mgr; | 506 | struct amdgpu_ctx_mgr ctx_mgr; |
| 507 | struct amdgpu_userq_mgr userq_mgr; | ||
| 508 | |||
| 509 | /* Eviction fence infra */ | ||
| 510 | struct amdgpu_eviction_fence_mgr evf_mgr; | ||
| 511 | |||
| 504 | /** GPU partition selection */ | 512 | /** GPU partition selection */ |
| 505 | uint32_t xcp_id; | 513 | uint32_t xcp_id; |
| 506 | }; | 514 | }; |
| @@ -512,12 +520,62 @@ int amdgpu_file_to_fpriv(struct file *filp, struct amdgpu_fpriv **fpriv); · 5 unmodified lines | |||
| 512 | */ | 520 | */ |
| 513 | #define AMDGPU_MAX_WB 1024 /* Reserve at most 1024 WB slots for amdgpu-owned rings. */ | 521 | #define AMDGPU_MAX_WB 1024 /* Reserve at most 1024 WB slots for amdgpu-owned rings. */ |
| 514 | 522 | ||
| 523 | /** | ||
| 524 | * amdgpu_wb - This struct is used for small GPU memory allocation. | ||
| 525 | * | ||
| 526 | * This struct is used to allocate a small amount of GPU memory that can be | ||
| 527 | * used to shadow certain states into the memory. This is especially useful for | ||
| 528 | * providing easy CPU access to some states without requiring register access | ||
| 529 | * (e.g., if some block is power gated, reading register may be problematic). | ||
| 530 | * | ||
| 531 | * Note: the term writeback was initially used because many of the amdgpu | ||
| 532 | * components had some level of writeback memory, and this struct initially | ||
| 533 | * described those components. | ||
| 534 | */ | ||
| 515 | struct amdgpu_wb { | 535 | struct amdgpu_wb { |
| 536 | |||
| 537 | /** | ||
| 538 | * @wb_obj: | ||
| 539 | * | ||
| 540 | * Buffer Object used for the writeback memory. | ||
| 541 | */ | ||
| 516 | struct amdgpu_bo *wb_obj; | 542 | struct amdgpu_bo *wb_obj; |
| 543 | |||
| 544 | /** | ||
| 545 | * @wb: | ||
| 546 | * | ||
| 547 | * Pointer to the first writeback slot. In terms of CPU address | ||
| 548 | * this value can be accessed directly by using the offset as an index. | ||
| 549 | * For the GPU address, it is necessary to use gpu_addr and the offset. | ||
| 550 | */ | ||
| 517 | volatile uint32_t *wb; | 551 | volatile uint32_t *wb; |
| 552 | |||
| 553 | /** | ||
| 554 | * @gpu_addr: | ||
| 555 | * | ||
| 556 | * Writeback base address in the GPU. | ||
| 557 | */ | ||
| 518 | uint64_t gpu_addr; | 558 | uint64_t gpu_addr; |
| 519 | u32 num_wb; /* Number of wb slots actually reserved for amdgpu. */ | 559 | |
| 560 | /** | ||
| 561 | * @num_wb: | ||
| 562 | * | ||
| 563 | * Number of writeback slots reserved for amdgpu. | ||
| 564 | */ | ||
| 565 | u32 num_wb; | ||
| 566 | |||
| 567 | /** | ||
| 568 | * @used: | ||
| 569 | * | ||
| 570 | * Track the writeback slot already used. | ||
| 571 | */ | ||
| 520 | unsigned long used[DIV_ROUND_UP(AMDGPU_MAX_WB, BITS_PER_LONG)]; | 572 | unsigned long used[DIV_ROUND_UP(AMDGPU_MAX_WB, BITS_PER_LONG)]; |
| 573 | |||
| 574 | /** | ||
| 575 | * @lock: | ||
| 576 | * | ||
| 577 | * Protects read and write of the used field array. | ||
| 578 | */ | ||
| 521 | spinlock_t lock; | 579 | spinlock_t lock; |
| 522 | }; | 580 | }; |
| 523 | 581 | ||
| @@ -551,6 +609,7 @@ struct amdgpu_allowed_register_entry { · 27 unmodified lines | |||
| 551 | * are reset depends on the ASIC. Notably doesn't reset IPs | 609 | * are reset depends on the ASIC. Notably doesn't reset IPs |
| 552 | * shared with the CPU on APUs or the memory controllers (so | 610 | * shared with the CPU on APUs or the memory controllers (so |
| 553 | * VRAM is not lost). Not available on all ASICs. | 611 | * VRAM is not lost). Not available on all ASICs. |
| 612 | * @AMD_RESET_LINK: Triggers SW-UP link reset on other GPUs | ||
| 554 | * @AMD_RESET_BACO: BACO (Bus Alive, Chip Off) method powers off and on the card | 613 | * @AMD_RESET_BACO: BACO (Bus Alive, Chip Off) method powers off and on the card |
| 555 | * but without powering off the PCI bus. Suitable only for | 614 | * but without powering off the PCI bus. Suitable only for |
| 556 | * discrete GPUs. | 615 | * discrete GPUs. |
| @@ -568,6 +627,7 @@ enum amd_reset_method { · 11 unmodified lines | |||
| 568 | AMD_RESET_METHOD_MODE0, | 627 | AMD_RESET_METHOD_MODE0, |
| 569 | AMD_RESET_METHOD_MODE1, | 628 | AMD_RESET_METHOD_MODE1, |
| 570 | AMD_RESET_METHOD_MODE2, | 629 | AMD_RESET_METHOD_MODE2, |
| 630 | AMD_RESET_METHOD_LINK, | ||
| 571 | AMD_RESET_METHOD_BACO, | 631 | AMD_RESET_METHOD_BACO, |
| 572 | AMD_RESET_METHOD_PCI, | 632 | AMD_RESET_METHOD_PCI, |
| 573 | AMD_RESET_METHOD_ON_INIT, | 633 | AMD_RESET_METHOD_ON_INIT, |
| @@ -821,6 +881,11 @@ struct amdgpu_mqd_prop { · 247 unmodified lines | |||
| 821 | uint32_t hqd_queue_priority; | 881 | uint32_t hqd_queue_priority; |
| 822 | bool allow_tunneling; | 882 | bool allow_tunneling; |
| 823 | bool hqd_active; | 883 | bool hqd_active; |
| 884 | uint64_t shadow_addr; | ||
| 885 | uint64_t gds_bkup_addr; | ||
| 886 | uint64_t csa_addr; | ||
| 887 | uint64_t fence_address; | ||
| 888 | bool tmz_queue; | ||
| 824 | }; | 889 | }; |
| 825 | 890 | ||
| 826 | struct amdgpu_mqd { | 891 | struct amdgpu_mqd { |
| @@ -829,6 +894,12 @@ struct amdgpu_mqd { · 2 unmodified lines | |||
| 829 | struct amdgpu_mqd_prop *p); | 894 | struct amdgpu_mqd_prop *p); |
| 830 | }; | 895 | }; |
| 831 | 896 | ||
| 897 | struct amdgpu_pcie_reset_ctx { | ||
| 898 | bool in_link_reset; | ||
| 899 | bool occurs_dpc; | ||
| 900 | bool audio_suspended; | ||
| 901 | }; | ||
| 902 | |||
| 832 | /* | 903 | /* |
| 833 | * Custom Init levels could be defined for different situations where a full | 904 | * Custom Init levels could be defined for different situations where a full |
| 834 | * initialization of all hardware blocks are not expected. Sample cases are | 905 | * initialization of all hardware blocks are not expected. Sample cases are |
| @@ -853,6 +924,14 @@ struct amdgpu_init_level { · 18 unmodified lines | |||
| 853 | struct amdgpu_reset_domain; | 924 | struct amdgpu_reset_domain; |
| 854 | struct amdgpu_fru_info; | 925 | struct amdgpu_fru_info; |
| 855 | 926 | ||
| 927 | enum amdgpu_enforce_isolation_mode { | ||
| 928 | AMDGPU_ENFORCE_ISOLATION_DISABLE = 0, | ||
| 929 | AMDGPU_ENFORCE_ISOLATION_ENABLE = 1, | ||
| 930 | AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY = 2, | ||
| 931 | AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER = 3, | ||
| 932 | }; | ||
| 933 | |||
| 934 | |||
| 856 | /* | 935 | /* |
| 857 | * Non-zero (true) if the GPU has VRAM. Zero (false) otherwise. | 936 | * Non-zero (true) if the GPU has VRAM. Zero (false) otherwise. |
| 858 | */ | 937 | */ |
| @@ -1081,6 +1160,13 @@ struct amdgpu_device { · 222 unmodified lines | |||
| 1081 | bool enable_uni_mes; | 1160 | bool enable_uni_mes; |
| 1082 | struct amdgpu_mes mes; | 1161 | struct amdgpu_mes mes; |
| 1083 | struct amdgpu_mqd mqds[AMDGPU_HW_IP_NUM]; | 1162 | struct amdgpu_mqd mqds[AMDGPU_HW_IP_NUM]; |
| 1163 | const struct amdgpu_userq_funcs *userq_funcs[AMDGPU_HW_IP_NUM]; | ||
| 1164 | |||
| 1165 | /* xarray used to retrieve the user queue fence driver reference | ||
| 1166 | * in the EOP interrupt handler to signal the particular user | ||
| 1167 | * queue fence. | ||
| 1168 | */ | ||
| 1169 | struct xarray userq_xa; | ||
| 1084 | 1170 | ||
| 1085 | /* df */ | 1171 | /* df */ |
| 1086 | struct amdgpu_df df; | 1172 | struct amdgpu_df df; |
| @@ -1160,6 +1246,8 @@ struct amdgpu_device { · 73 unmodified lines | |||
| 1160 | struct pci_saved_state *pci_state; | 1246 | struct pci_saved_state *pci_state; |
| 1161 | pci_channel_state_t pci_channel_state; | 1247 | pci_channel_state_t pci_channel_state; |
| 1162 | 1248 | ||
| 1249 | struct amdgpu_pcie_reset_ctx pcie_reset_ctx; | ||
| 1250 | |||
| 1163 | /* Track auto wait count on s_barrier settings */ | 1251 | /* Track auto wait count on s_barrier settings */ |
| 1164 | bool barrier_has_auto_waitcnt; | 1252 | bool barrier_has_auto_waitcnt; |
| 1165 | 1253 | ||
| @@ -1193,10 +1281,11 @@ struct amdgpu_device { · 27 unmodified lines | |||
| 1193 | bool debug_enable_ras_aca; | 1281 | bool debug_enable_ras_aca; |
| 1194 | bool debug_exp_resets; | 1282 | bool debug_exp_resets; |
| 1195 | bool debug_disable_gpu_ring_reset; | 1283 | bool debug_disable_gpu_ring_reset; |
| 1284 | bool debug_vm_userptr; | ||
| 1196 | 1285 | ||
| 1197 | /* Protection for the following isolation structure */ | 1286 | /* Protection for the following isolation structure */ |
| 1198 | struct mutex enforce_isolation_mutex; | 1287 | struct mutex enforce_isolation_mutex; |
| 1199 | bool enforce_isolation[MAX_XCP]; | 1288 | enum amdgpu_enforce_isolation_mode enforce_isolation[MAX_XCP]; |
| 1200 | struct amdgpu_isolation { | 1289 | struct amdgpu_isolation { |
| 1201 | void *owner; | 1290 | void *owner; |
| 1202 | struct dma_fence *spearhead; | 1291 | struct dma_fence *spearhead; |
| @@ -1210,6 +1299,10 @@ struct amdgpu_device { · 7 unmodified lines | |||
| 1210 | * in KFD: VRAM or GTT. | 1299 | * in KFD: VRAM or GTT. |
| 1211 | */ | 1300 | */ |
| 1212 | bool apu_prefer_gtt; | 1301 | bool apu_prefer_gtt; |
| 1302 | |||
| 1303 | struct list_head userq_mgr_list; | ||
| 1304 | struct mutex userq_mutex; | ||
| 1305 | bool userq_halt_for_enforce_isolation; | ||
| 1213 | }; | 1306 | }; |
| 1214 | 1307 | ||
| 1215 | static inline uint32_t amdgpu_ip_version(const struct amdgpu_device *adev, | 1308 | static inline uint32_t amdgpu_ip_version(const struct amdgpu_device *adev, |
| @@ -1464,6 +1557,7 @@ void amdgpu_device_program_register_sequence(struct amdgpu_device *adev, · 248 unmodified lines | |||
| 1464 | const u32 array_size); | 1557 | const u32 array_size); |
| 1465 | 1558 | ||
| 1466 | int amdgpu_device_mode1_reset(struct amdgpu_device *adev); | 1559 | int amdgpu_device_mode1_reset(struct amdgpu_device *adev); |
| 1560 | int amdgpu_device_link_reset(struct amdgpu_device *adev); | ||
| 1467 | bool amdgpu_device_supports_atpx(struct drm_device *dev); | 1561 | bool amdgpu_device_supports_atpx(struct drm_device *dev); |
| 1468 | bool amdgpu_device_supports_px(struct drm_device *dev); | 1562 | bool amdgpu_device_supports_px(struct drm_device *dev); |
| 1469 | bool amdgpu_device_supports_boco(struct drm_device *dev); | 1563 | bool amdgpu_device_supports_boco(struct drm_device *dev); |
| @@ -1614,11 +1708,9 @@ static inline void amdgpu_acpi_get_backlight_caps(struct amdgpu_dm_backlight_cap · 144 unmodified lines | |||
| 1614 | #if defined(CONFIG_ACPI) && defined(CONFIG_SUSPEND) | 1708 | #if defined(CONFIG_ACPI) && defined(CONFIG_SUSPEND) |
| 1615 | bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev); | 1709 | bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev); |
| 1616 | bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev); | 1710 | bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev); |
| 1617 | void amdgpu_choose_low_power_state(struct amdgpu_device *adev); | ||
| 1618 | #else | 1711 | #else |
| 1619 | static inline bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) { return false; } | 1712 | static inline bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) { return false; } |
| 1620 | static inline bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev) { return false; } | 1713 | static inline bool amdgpu_acpi_is_s3_active(struct amdgpu_device *adev) { return false; } |
| 1621 | static inline void amdgpu_choose_low_power_state(struct amdgpu_device *adev) { } | ||
| 1622 | #endif | 1714 | #endif |
| 1623 | 1715 | ||
| 1624 | void amdgpu_register_gpu_instance(struct amdgpu_device *adev); | 1716 | void amdgpu_register_gpu_instance(struct amdgpu_device *adev); |
| @@ -120,6 +120,9 @@ static void aca_smu_bank_dump(struct amdgpu_device *adev, int idx, int total, st · 119 unmodified lines | |||
| 120 | for (i = 0; i < ARRAY_SIZE(aca_regs); i++) | 120 | for (i = 0; i < ARRAY_SIZE(aca_regs); i++) |
| 121 | RAS_EVENT_LOG(adev, event_id, HW_ERR "ACA[%02d/%02d].%s=0x%016llx\n", | 121 | RAS_EVENT_LOG(adev, event_id, HW_ERR "ACA[%02d/%02d].%s=0x%016llx\n", |
| 122 | idx + 1, total, aca_regs[i].name, bank->regs[aca_regs[i].reg_idx]); | 122 | idx + 1, total, aca_regs[i].name, bank->regs[aca_regs[i].reg_idx]); |
| 123 | |||
| 124 | if (ACA_REG__STATUS__SCRUB(bank->regs[ACA_REG_IDX_STATUS])) | ||
| 125 | RAS_EVENT_LOG(adev, event_id, HW_ERR "hardware error logged by the scrubber\n"); | ||
| 123 | } | 126 | } |
| 124 | 127 | ||
| 125 | static int aca_smu_get_valid_aca_banks(struct amdgpu_device *adev, enum aca_smu_type type, | 128 | static int aca_smu_get_valid_aca_banks(struct amdgpu_device *adev, enum aca_smu_type type, |
| @@ -1533,22 +1533,4 @@ bool amdgpu_acpi_is_s0ix_active(struct amdgpu_device *adev) · 1532 unmodified lines | |||
| 1533 | #endif /* CONFIG_AMD_PMC */ | 1533 | #endif /* CONFIG_AMD_PMC */ |
| 1534 | } | 1534 | } |
| 1535 | 1535 | ||
| 1536 | /** | ||
| 1537 | * amdgpu_choose_low_power_state | ||
| 1538 | * | ||
| 1539 | * @adev: amdgpu_device_pointer | ||
| 1540 | * | ||
| 1541 | * Choose the target low power state for the GPU | ||
| 1542 | */ | ||
| 1543 | void amdgpu_choose_low_power_state(struct amdgpu_device *adev) | ||
| 1544 | { | ||
| 1545 | if (adev->in_runpm) | ||
| 1546 | return; | ||
| 1547 | |||
| 1548 | if (amdgpu_acpi_is_s0ix_active(adev)) | ||
| 1549 | adev->in_s0ix = true; | ||
| 1550 | else if (amdgpu_acpi_is_s3_active(adev)) | ||
| 1551 | adev->in_s3 = true; | ||
| 1552 | } | ||
| 1553 | |||
| 1554 | #endif /* CONFIG_SUSPEND */ | 1536 | #endif /* CONFIG_SUSPEND */ |
| @@ -2559,6 +2559,18 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info, · 2558 unmodified lines | |||
| 2559 | if (ret != -EFAULT) | 2559 | if (ret != -EFAULT) |
| 2560 | return ret; | 2560 | return ret; |
| 2561 | 2561 | ||
| 2562 | /* If applications unmap memory before destroying the userptr | ||
| 2563 | * from the KFD, trigger a segmentation fault in VM debug mode. | ||
| 2564 | */ | ||
| 2565 | if (amdgpu_ttm_adev(bo->tbo.bdev)->debug_vm_userptr) { | ||
| 2566 | pr_err("Pid %d unmapped memory before destroying userptr at GPU addr 0x%llx\n", | ||
| 2567 | pid_nr(process_info->pid), mem->va); | ||
| 2568 | |||
| 2569 | // Send GPU VM fault to user space | ||
| 2570 | kfd_signal_vm_fault_event_with_userptr(kfd_lookup_process_by_pid(process_info->pid), | ||
| 2571 | mem->va); | ||
| 2572 | } | ||
| 2573 | |||
| 2562 | ret = 0; | 2574 | ret = 0; |
| 2563 | } | 2575 | } |
| 2564 | 2576 | ||
| @@ -252,83 +252,22 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 251 unmodified lines | |||
| 252 | 252 | ||
| 253 | if (!adev->pm.fw) { | 253 | if (!adev->pm.fw) { |
| 254 | switch (adev->asic_type) { | 254 | switch (adev->asic_type) { |
| 255 | case CHIP_TAHITI: | ||
| 256 | strcpy(fw_name, "radeon/tahiti_smc.bin"); | ||
| 257 | break; | ||
| 258 | case CHIP_PITCAIRN: | ||
| 259 | if ((adev->pdev->revision == 0x81) && | ||
| 260 | ((adev->pdev->device == 0x6810) || | ||
| 261 | (adev->pdev->device == 0x6811))) { | ||
| 262 | info->is_kicker = true; | ||
| 263 | strcpy(fw_name, "radeon/pitcairn_k_smc.bin"); | ||
| 264 | } else { | ||
| 265 | strcpy(fw_name, "radeon/pitcairn_smc.bin"); | ||
| 266 | } | ||
| 267 | break; | ||
| 268 | case CHIP_VERDE: | ||
| 269 | if (((adev->pdev->device == 0x6820) && | ||
| 270 | ((adev->pdev->revision == 0x81) || | ||
| 271 | (adev->pdev->revision == 0x83))) || | ||
| 272 | ((adev->pdev->device == 0x6821) && | ||
| 273 | ((adev->pdev->revision == 0x83) || | ||
| 274 | (adev->pdev->revision == 0x87))) || | ||
| 275 | ((adev->pdev->revision == 0x87) && | ||
| 276 | ((adev->pdev->device == 0x6823) || | ||
| 277 | (adev->pdev->device == 0x682b)))) { | ||
| 278 | info->is_kicker = true; | ||
| 279 | strcpy(fw_name, "radeon/verde_k_smc.bin"); | ||
| 280 | } else { | ||
| 281 | strcpy(fw_name, "radeon/verde_smc.bin"); | ||
| 282 | } | ||
| 283 | break; | ||
| 284 | case CHIP_OLAND: | ||
| 285 | if (((adev->pdev->revision == 0x81) && | ||
| 286 | ((adev->pdev->device == 0x6600) || | ||
| 287 | (adev->pdev->device == 0x6604) || | ||
| 288 | (adev->pdev->device == 0x6605) || | ||
| 289 | (adev->pdev->device == 0x6610))) || | ||
| 290 | ((adev->pdev->revision == 0x83) && | ||
| 291 | (adev->pdev->device == 0x6610))) { | ||
| 292 | info->is_kicker = true; | ||
| 293 | strcpy(fw_name, "radeon/oland_k_smc.bin"); | ||
| 294 | } else { | ||
| 295 | strcpy(fw_name, "radeon/oland_smc.bin"); | ||
| 296 | } | ||
| 297 | break; | ||
| 298 | case CHIP_HAINAN: | ||
| 299 | if (((adev->pdev->revision == 0x81) && | ||
| 300 | (adev->pdev->device == 0x6660)) || | ||
| 301 | ((adev->pdev->revision == 0x83) && | ||
| 302 | ((adev->pdev->device == 0x6660) || | ||
| 303 | (adev->pdev->device == 0x6663) || | ||
| 304 | (adev->pdev->device == 0x6665) || | ||
| 305 | (adev->pdev->device == 0x6667)))) { | ||
| 306 | info->is_kicker = true; | ||
| 307 | strcpy(fw_name, "radeon/hainan_k_smc.bin"); | ||
| 308 | } else if ((adev->pdev->revision == 0xc3) && | ||
| 309 | (adev->pdev->device == 0x6665)) { | ||
| 310 | info->is_kicker = true; | ||
| 311 | strcpy(fw_name, "radeon/banks_k_2_smc.bin"); | ||
| 312 | } else { | ||
| 313 | strcpy(fw_name, "radeon/hainan_smc.bin"); | ||
| 314 | } | ||
| 315 | break; | ||
| 316 | case CHIP_BONAIRE: | 255 | case CHIP_BONAIRE: |
| 317 | if ((adev->pdev->revision == 0x80) || | 256 | if ((adev->pdev->revision == 0x80) || |
| 318 | (adev->pdev->revision == 0x81) || | 257 | (adev->pdev->revision == 0x81) || |
| 319 | (adev->pdev->device == 0x665f)) { | 258 | (adev->pdev->device == 0x665f)) { |
| 320 | info->is_kicker = true; | 259 | info->is_kicker = true; |
| 321 | strcpy(fw_name, "amdgpu/bonaire_k_smc.bin"); | 260 | strscpy(fw_name, "amdgpu/bonaire_k_smc.bin"); |
| 322 | } else { | 261 | } else { |
| 323 | strcpy(fw_name, "amdgpu/bonaire_smc.bin"); | 262 | strscpy(fw_name, "amdgpu/bonaire_smc.bin"); |
| 324 | } | 263 | } |
| 325 | break; | 264 | break; |
| 326 | case CHIP_HAWAII: | 265 | case CHIP_HAWAII: |
| 327 | if (adev->pdev->revision == 0x80) { | 266 | if (adev->pdev->revision == 0x80) { |
| 328 | info->is_kicker = true; | 267 | info->is_kicker = true; |
| 329 | strcpy(fw_name, "amdgpu/hawaii_k_smc.bin"); | 268 | strscpy(fw_name, "amdgpu/hawaii_k_smc.bin"); |
| 330 | } else { | 269 | } else { |
| 331 | strcpy(fw_name, "amdgpu/hawaii_smc.bin"); | 270 | strscpy(fw_name, "amdgpu/hawaii_smc.bin"); |
| 332 | } | 271 | } |
| 333 | break; | 272 | break; |
| 334 | case CHIP_TOPAZ: | 273 | case CHIP_TOPAZ: |
| @@ -338,76 +277,76 @@ static int amdgpu_cgs_get_firmware_info(struct cgs_device *cgs_device, · 3 unmodified lines | |||
| 338 | ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD1)) || | 277 | ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD1)) || |
| 339 | ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD3))) { | 278 | ((adev->pdev->device == 0x6900) && (adev->pdev->revision == 0xD3))) { |
| 340 | info->is_kicker = true; | 279 | info->is_kicker = true; |
| 341 | strcpy(fw_name, "amdgpu/topaz_k_smc.bin"); | 280 | strscpy(fw_name, "amdgpu/topaz_k_smc.bin"); |
| 342 | } else | 281 | } else |
| 343 | strcpy(fw_name, "amdgpu/topaz_smc.bin"); | 282 | strscpy(fw_name, "amdgpu/topaz_smc.bin"); |
| 344 | break; | 283 | break; |
| 345 | case CHIP_TONGA: | 284 | case CHIP_TONGA: |
| 346 | if (((adev->pdev->device == 0x6939) && (adev->pdev->revision == 0xf1)) || | 285 | if (((adev->pdev->device == 0x6939) && (adev->pdev->revision == 0xf1)) || |
| 347 | ((adev->pdev->device == 0x6938) && (adev->pdev->revision == 0xf1))) { | 286 | ((adev->pdev->device == 0x6938) && (adev->pdev->revision == 0xf1))) { |
| 348 | info->is_kicker = true; | 287 | info->is_kicker = true; |
| 349 | strcpy(fw_name, "amdgpu/tonga_k_smc.bin"); | 288 | strscpy(fw_name, "amdgpu/tonga_k_smc.bin"); |
| 350 | } else | 289 | } else |
| 351 | strcpy(fw_name, "amdgpu/tonga_smc.bin"); | 290 | strscpy(fw_name, "amdgpu/tonga_smc.bin"); |
| 352 | break; | 291 | break; |
| 353 | case CHIP_FIJI: | 292 | case CHIP_FIJI: |
| 354 | strcpy(fw_name, "amdgpu/fiji_smc.bin"); | 293 | strscpy(fw_name, "amdgpu/fiji_smc.bin"); |
| 355 | break; | 294 | break; |
| 356 | case CHIP_POLARIS11: | 295 | case CHIP_POLARIS11: |
| 357 | if (type == CGS_UCODE_ID_SMU) { | 296 | if (type == CGS_UCODE_ID_SMU) { |
| 358 | if (ASICID_IS_P21(adev->pdev->device, adev->pdev->revision)) { | 297 | if (ASICID_IS_P21(adev->pdev->device, adev->pdev->revision)) { |
| 359 | info->is_kicker = true; | 298 | info->is_kicker = true; |
| 360 | strcpy(fw_name, "amdgpu/polaris11_k_smc.bin"); | 299 | strscpy(fw_name, "amdgpu/polaris11_k_smc.bin"); |
| 361 | } else if (ASICID_IS_P31(adev->pdev->device, adev->pdev->revision)) { | 300 | } else if (ASICID_IS_P31(adev->pdev->device, adev->pdev->revision)) { |
| 362 | info->is_kicker = true; | 301 | info->is_kicker = true; |
| 363 | strcpy(fw_name, "amdgpu/polaris11_k2_smc.bin"); | 302 | strscpy(fw_name, "amdgpu/polaris11_k2_smc.bin"); |
| 364 | } else { | 303 | } else { |
| 365 | strcpy(fw_name, "amdgpu/polaris11_smc.bin"); | 304 | strscpy(fw_name, "amdgpu/polaris11_smc.bin"); |
| 366 | } | 305 | } |
| 367 | } else if (type == CGS_UCODE_ID_SMU_SK) { | 306 | } else if (type == CGS_UCODE_ID_SMU_SK) { |
| 368 | strcpy(fw_name, "amdgpu/polaris11_smc_sk.bin"); | 307 | strscpy(fw_name, "amdgpu/polaris11_smc_sk.bin"); |
| 369 | } | 308 | } |
| 370 | break; | 309 | break; |
| 371 | case CHIP_POLARIS10: | 310 | case CHIP_POLARIS10: |
| 372 | if (type == CGS_UCODE_ID_SMU) { | 311 | if (type == CGS_UCODE_ID_SMU) { |
| 373 | if (ASICID_IS_P20(adev->pdev->device, adev->pdev->revision)) { | 312 | if (ASICID_IS_P20(adev->pdev->device, adev->pdev->revision)) { |
| 374 | info->is_kicker = true; | 313 | info->is_kicker = true; |
| 375 | strcpy(fw_name, "amdgpu/polaris10_k_smc.bin"); | 314 | strscpy(fw_name, "amdgpu/polaris10_k_smc.bin"); |
| 376 | } else if (ASICID_IS_P30(adev->pdev->device, adev->pdev->revision)) { | 315 | } else if (ASICID_IS_P30(adev->pdev->device, adev->pdev->revision)) { |
| 377 | info->is_kicker = true; | 316 | info->is_kicker = true; |
| 378 | strcpy(fw_name, "amdgpu/polaris10_k2_smc.bin"); | 317 | strscpy(fw_name, "amdgpu/polaris10_k2_smc.bin"); |
| 379 | } else { | 318 | } else { |
| 380 | strcpy(fw_name, "amdgpu/polaris10_smc.bin"); | 319 | strscpy(fw_name, "amdgpu/polaris10_smc.bin"); |
| 381 | } | 320 | } |
| 382 | } else if (type == CGS_UCODE_ID_SMU_SK) { | 321 | } else if (type == CGS_UCODE_ID_SMU_SK) { |
| 383 | strcpy(fw_name, "amdgpu/polaris10_smc_sk.bin"); | 322 | strscpy(fw_name, "amdgpu/polaris10_smc_sk.bin"); |
| 384 | } | 323 | } |
| 385 | break; | 324 | break; |
| 386 | case CHIP_POLARIS12: | 325 | case CHIP_POLARIS12: |
| 387 | if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision)) { | 326 | if (ASICID_IS_P23(adev->pdev->device, adev->pdev->revision)) { |
| 388 | info->is_kicker = true; | 327 | info->is_kicker = true; |
| 389 | strcpy(fw_name, "amdgpu/polaris12_k_smc.bin"); | 328 | strscpy(fw_name, "amdgpu/polaris12_k_smc.bin"); |
| 390 | } else { | 329 | } else { |
| 391 | strcpy(fw_name, "amdgpu/polaris12_smc.bin"); | 330 | strscpy(fw_name, "amdgpu/polaris12_smc.bin"); |
| 392 | } | 331 | } |
| 393 | break; | 332 | break; |
| 394 | case CHIP_VEGAM: | 333 | case CHIP_VEGAM: |
| 395 | strcpy(fw_name, "amdgpu/vegam_smc.bin"); | 334 | strscpy(fw_name, "amdgpu/vegam_smc.bin"); |
| 396 | break; | 335 | break; |
| 397 | case CHIP_VEGA10: | 336 | case CHIP_VEGA10: |
| 398 | if ((adev->pdev->device == 0x687f) && | 337 | if ((adev->pdev->device == 0x687f) && |
| 399 | ((adev->pdev->revision == 0xc0) || | 338 | ((adev->pdev->revision == 0xc0) || |
| 400 | (adev->pdev->revision == 0xc1) || | 339 | (adev->pdev->revision == 0xc1) || |
| 401 | (adev->pdev->revision == 0xc3))) | 340 | (adev->pdev->revision == 0xc3))) |
| 402 | strcpy(fw_name, "amdgpu/vega10_acg_smc.bin"); | 341 | strscpy(fw_name, "amdgpu/vega10_acg_smc.bin"); |
| 403 | else | 342 | else |
| 404 | strcpy(fw_name, "amdgpu/vega10_smc.bin"); | 343 | strscpy(fw_name, "amdgpu/vega10_smc.bin"); |
| 405 | break; | 344 | break; |
| 406 | case CHIP_VEGA12: | 345 | case CHIP_VEGA12: |
| 407 | strcpy(fw_name, "amdgpu/vega12_smc.bin"); | 346 | strscpy(fw_name, "amdgpu/vega12_smc.bin"); |
| 408 | break; | 347 | break; |
| 409 | case CHIP_VEGA20: | 348 | case CHIP_VEGA20: |
| 410 | strcpy(fw_name, "amdgpu/vega20_smc.bin"); | 349 | strscpy(fw_name, "amdgpu/vega20_smc.bin"); |
| 411 | break; | 350 | break; |
| 412 | default: | 351 | default: |
| 413 | DRM_ERROR("SMC firmware not supported\n"); | 352 | DRM_ERROR("SMC firmware not supported\n"); |
| @@ -549,7 +549,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 548 unmodified lines | |||
| 549 | { | 549 | { |
| 550 | int r; | 550 | int r; |
| 551 | 551 | ||
| 552 | if (!amdgpu_aca_is_enabled(adev)) | 552 | if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev)) |
| 553 | return 0; | 553 | return 0; |
| 554 | 554 | ||
| 555 | r = amdgpu_cper_ring_init(adev); | 555 | r = amdgpu_cper_ring_init(adev); |
| @@ -568,7 +568,7 @@ int amdgpu_cper_init(struct amdgpu_device *adev) · 12 unmodified lines | |||
| 568 | 568 | ||
| 569 | int amdgpu_cper_fini(struct amdgpu_device *adev) | 569 | int amdgpu_cper_fini(struct amdgpu_device *adev) |
| 570 | { | 570 | { |
| 571 | if (!amdgpu_aca_is_enabled(adev)) | 571 | if (!amdgpu_aca_is_enabled(adev) && !amdgpu_sriov_ras_cper_en(adev)) |
| 572 | return 0; | 572 | return 0; |
| 573 | 573 | ||
| 574 | adev->cper.enabled = false; | 574 | adev->cper.enabled = false; |
| @@ -296,7 +296,25 @@ static int amdgpu_cs_pass1(struct amdgpu_cs_parser *p, · 295 unmodified lines | |||
| 296 | num_ibs[i], &p->jobs[i]); | 296 | num_ibs[i], &p->jobs[i]); |
| 297 | if (ret) | 297 | if (ret) |
| 298 | goto free_all_kdata; | 298 | goto free_all_kdata; |
| 299 | p->jobs[i]->enforce_isolation = p->adev->enforce_isolation[fpriv->xcp_id]; | 299 | switch (p->adev->enforce_isolation[fpriv->xcp_id]) { |
| 300 | case AMDGPU_ENFORCE_ISOLATION_DISABLE: | ||
| 301 | default: | ||
| 302 | p->jobs[i]->enforce_isolation = false; | ||
| 303 | p->jobs[i]->run_cleaner_shader = false; | ||
| 304 | break; | ||
| 305 | case AMDGPU_ENFORCE_ISOLATION_ENABLE: | ||
| 306 | p->jobs[i]->enforce_isolation = true; | ||
| 307 | p->jobs[i]->run_cleaner_shader = true; | ||
| 308 | break; | ||
| 309 | case AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY: | ||
| 310 | p->jobs[i]->enforce_isolation = true; | ||
| 311 | p->jobs[i]->run_cleaner_shader = false; | ||
| 312 | break; | ||
| 313 | case AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER: | ||
| 314 | p->jobs[i]->enforce_isolation = true; | ||
| 315 | p->jobs[i]->run_cleaner_shader = false; | ||
| 316 | break; | ||
| 317 | } | ||
| 300 | } | 318 | } |
| 301 | p->gang_leader = p->jobs[p->gang_leader_idx]; | 319 | p->gang_leader = p->jobs[p->gang_leader_idx]; |
| 302 | 320 | ||
| @@ -349,6 +367,10 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p, · 46 unmodified lines | |||
| 349 | ring = amdgpu_job_ring(job); | 367 | ring = amdgpu_job_ring(job); |
| 350 | ib = &job->ibs[job->num_ibs++]; | 368 | ib = &job->ibs[job->num_ibs++]; |
| 351 | 369 | ||
| 370 | /* submissions to kernel queues are disabled */ | ||
| 371 | if (ring->no_user_submission) | ||
| 372 | return -EINVAL; | ||
| 373 | |||
| 352 | /* MM engine doesn't support user fences */ | 374 | /* MM engine doesn't support user fences */ |
| 353 | if (p->uf_bo && ring->funcs->no_user_fence) | 375 | if (p->uf_bo && ring->funcs->no_user_fence) |
| 354 | return -EINVAL; | 376 | return -EINVAL; |
| @@ -85,6 +85,7 @@ · 84 unmodified lines | |||
| 85 | 85 | ||
| 86 | #if IS_ENABLED(CONFIG_X86) | 86 | #if IS_ENABLED(CONFIG_X86) |
| 87 | #include <asm/intel-family.h> | 87 | #include <asm/intel-family.h> |
| 88 | #include <asm/cpu_device_id.h> | ||
| 88 | #endif | 89 | #endif |
| 89 | 90 | ||
| 90 | MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin"); | 91 | MODULE_FIRMWARE("amdgpu/vega10_gpu_info.bin"); |
| @@ -1680,6 +1681,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev) · 1589 unmodified lines | |||
| 1680 | if (amdgpu_sriov_vf(adev)) | 1681 | if (amdgpu_sriov_vf(adev)) |
| 1681 | return 0; | 1682 | return 0; |
| 1682 | 1683 | ||
| 1684 | if (!amdgpu_rebar) | ||
| 1685 | return 0; | ||
| 1686 | |||
| 1683 | /* resizing on Dell G5 SE platforms causes problems with runtime pm */ | 1687 | /* resizing on Dell G5 SE platforms causes problems with runtime pm */ |
| 1684 | if ((amdgpu_runtime_pm != 0) && | 1688 | if ((amdgpu_runtime_pm != 0) && |
| 1685 | adev->pdev->vendor == PCI_VENDOR_ID_ATI && | 1689 | adev->pdev->vendor == PCI_VENDOR_ID_ATI && |
| @@ -1870,6 +1874,35 @@ static bool amdgpu_device_pcie_dynamic_switching_supported(struct amdgpu_device · 184 unmodified lines | |||
| 1870 | return true; | 1874 | return true; |
| 1871 | } | 1875 | } |
| 1872 | 1876 | ||
| 1877 | static bool amdgpu_device_aspm_support_quirk(struct amdgpu_device *adev) | ||
| 1878 | { | ||
| 1879 | #if IS_ENABLED(CONFIG_X86) | ||
| 1880 | struct cpuinfo_x86 *c = &cpu_data(0); | ||
| 1881 | |||
| 1882 | if (!(amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 0, 0) || | ||
| 1883 | amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(12, 0, 1))) | ||
| 1884 | return false; | ||
| 1885 | |||
| 1886 | if (c->x86 == 6 && | ||
| 1887 | adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN5) { | ||
| 1888 | switch (c->x86_model) { | ||
| 1889 | case VFM_MODEL(INTEL_ALDERLAKE): | ||
| 1890 | case VFM_MODEL(INTEL_ALDERLAKE_L): | ||
| 1891 | case VFM_MODEL(INTEL_RAPTORLAKE): | ||
| 1892 | case VFM_MODEL(INTEL_RAPTORLAKE_P): | ||
| 1893 | case VFM_MODEL(INTEL_RAPTORLAKE_S): | ||
| 1894 | return true; | ||
| 1895 | default: | ||
| 1896 | return false; | ||
| 1897 | } | ||
| 1898 | } else { | ||
| 1899 | return false; | ||
| 1900 | } | ||
| 1901 | #else | ||
| 1902 | return false; | ||
| 1903 | #endif | ||
| 1904 | } | ||
| 1905 | |||
| 1873 | /** | 1906 | /** |
| 1874 | * amdgpu_device_should_use_aspm - check if the device should program ASPM | 1907 | * amdgpu_device_should_use_aspm - check if the device should program ASPM |
| 1875 | * | 1908 | * |
| @@ -1894,7 +1927,7 @@ bool amdgpu_device_should_use_aspm(struct amdgpu_device *adev) · 18 unmodified lines | |||
| 1894 | } | 1927 | } |
| 1895 | if (adev->flags & AMD_IS_APU) | 1928 | if (adev->flags & AMD_IS_APU) |
| 1896 | return false; | 1929 | return false; |
| 1897 | if (!(adev->pm.pp_feature & PP_PCIE_DPM_MASK)) | 1930 | if (amdgpu_device_aspm_support_quirk(adev)) |
| 1898 | return false; | 1931 | return false; |
| 1899 | return pcie_aspm_enabled(adev->pdev); | 1932 | return pcie_aspm_enabled(adev->pdev); |
| 1900 | } | 1933 | } |
| @@ -2112,8 +2145,31 @@ static int amdgpu_device_check_arguments(struct amdgpu_device *adev) · 211 unmodified lines | |||
| 2112 | 2145 | ||
| 2113 | adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, amdgpu_fw_load_type); | 2146 | adev->firmware.load_type = amdgpu_ucode_get_load_type(adev, amdgpu_fw_load_type); |
| 2114 | 2147 | ||
| 2115 | for (i = 0; i < MAX_XCP; i++) | 2148 | for (i = 0; i < MAX_XCP; i++) { |
| 2116 | adev->enforce_isolation[i] = !!enforce_isolation; | 2149 | switch (amdgpu_enforce_isolation) { |
| 2150 | case -1: | ||
| 2151 | case 0: | ||
| 2152 | default: | ||
| 2153 | /* disable */ | ||
| 2154 | adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE; | ||
| 2155 | break; | ||
| 2156 | case 1: | ||
| 2157 | /* enable */ | ||
| 2158 | adev->enforce_isolation[i] = | ||
| 2159 | AMDGPU_ENFORCE_ISOLATION_ENABLE; | ||
| 2160 | break; | ||
| 2161 | case 2: | ||
| 2162 | /* enable legacy mode */ | ||
| 2163 | adev->enforce_isolation[i] = | ||
| 2164 | AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY; | ||
| 2165 | break; | ||
| 2166 | case 3: | ||
| 2167 | /* enable only process isolation without submitting cleaner shader */ | ||
| 2168 | adev->enforce_isolation[i] = | ||
| 2169 | AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER; | ||
| 2170 | break; | ||
| 2171 | } | ||
| 2172 | } | ||
| 2117 | 2173 | ||
| 2118 | return 0; | 2174 | return 0; |
| 2119 | } | 2175 | } |
| @@ -2689,6 +2745,13 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 569 unmodified lines | |||
| 2689 | break; | 2745 | break; |
| 2690 | } | 2746 | } |
| 2691 | 2747 | ||
| 2748 | /* Check for IP version 9.4.3 with A0 hardware */ | ||
| 2749 | if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) && | ||
| 2750 | !amdgpu_device_get_rev_id(adev)) { | ||
| 2751 | dev_err(adev->dev, "Unsupported A0 hardware\n"); | ||
| 2752 | return -ENODEV; /* device unsupported - no device error */ | ||
| 2753 | } | ||
| 2754 | |||
| 2692 | if (amdgpu_has_atpx() && | 2755 | if (amdgpu_has_atpx() && |
| 2693 | (amdgpu_is_atpx_hybrid() || | 2756 | (amdgpu_is_atpx_hybrid() || |
| 2694 | amdgpu_has_atpx_dgpu_power_cntl()) && | 2757 | amdgpu_has_atpx_dgpu_power_cntl()) && |
| @@ -2701,7 +2764,6 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 2701 | adev->has_pr3 = parent ? pci_pr3_present(parent) : false; | 2764 | adev->has_pr3 = parent ? pci_pr3_present(parent) : false; |
| 2702 | } | 2765 | } |
| 2703 | 2766 | ||
| 2704 | |||
| 2705 | adev->pm.pp_feature = amdgpu_pp_feature_mask; | 2767 | adev->pm.pp_feature = amdgpu_pp_feature_mask; |
| 2706 | if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS) | 2768 | if (amdgpu_sriov_vf(adev) || sched_policy == KFD_SCHED_POLICY_NO_HWS) |
| 2707 | adev->pm.pp_feature &= ~PP_GFXOFF_MASK; | 2769 | adev->pm.pp_feature &= ~PP_GFXOFF_MASK; |
| @@ -3172,6 +3234,7 @@ static bool amdgpu_device_check_vram_lost(struct amdgpu_device *adev) · 464 unmodified lines | |||
| 3172 | * always assumed to be lost. | 3234 | * always assumed to be lost. |
| 3173 | */ | 3235 | */ |
| 3174 | switch (amdgpu_asic_reset_method(adev)) { | 3236 | switch (amdgpu_asic_reset_method(adev)) { |
| 3237 | case AMD_RESET_METHOD_LINK: | ||
| 3175 | case AMD_RESET_METHOD_BACO: | 3238 | case AMD_RESET_METHOD_BACO: |
| 3176 | case AMD_RESET_METHOD_MODE1: | 3239 | case AMD_RESET_METHOD_MODE1: |
| 3177 | return true; | 3240 | return true; |
| @@ -3455,6 +3518,7 @@ static int amdgpu_device_ip_fini_early(struct amdgpu_device *adev) · 277 unmodified lines | |||
| 3455 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE); | 3518 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE); |
| 3456 | 3519 | ||
| 3457 | amdgpu_amdkfd_suspend(adev, false); | 3520 | amdgpu_amdkfd_suspend(adev, false); |
| 3521 | amdgpu_userq_suspend(adev); | ||
| 3458 | 3522 | ||
| 3459 | /* Workaround for ASICs need to disable SMC first */ | 3523 | /* Workaround for ASICs need to disable SMC first */ |
| 3460 | amdgpu_device_smu_fini_early(adev); | 3524 | amdgpu_device_smu_fini_early(adev); |
| @@ -4307,9 +4371,10 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 846 unmodified lines | |||
| 4307 | amdgpu_sync_create(&adev->isolation[i].active); | 4371 | amdgpu_sync_create(&adev->isolation[i].active); |
| 4308 | amdgpu_sync_create(&adev->isolation[i].prev); | 4372 | amdgpu_sync_create(&adev->isolation[i].prev); |
| 4309 | } | 4373 | } |
| 4310 | mutex_init(&adev->gfx.kfd_sch_mutex); | 4374 | mutex_init(&adev->gfx.userq_sch_mutex); |
| 4311 | mutex_init(&adev->gfx.workload_profile_mutex); | 4375 | mutex_init(&adev->gfx.workload_profile_mutex); |
| 4312 | mutex_init(&adev->vcn.workload_profile_mutex); | 4376 | mutex_init(&adev->vcn.workload_profile_mutex); |
| 4377 | mutex_init(&adev->userq_mutex); | ||
| 4313 | 4378 | ||
| 4314 | amdgpu_device_init_apu_flags(adev); | 4379 | amdgpu_device_init_apu_flags(adev); |
| 4315 | 4380 | ||
| @@ -4329,12 +4394,16 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 13 unmodified lines | |||
| 4329 | spin_lock_init(&adev->virt.rlcg_reg_lock); | 4394 | spin_lock_init(&adev->virt.rlcg_reg_lock); |
| 4330 | spin_lock_init(&adev->wb.lock); | 4395 | spin_lock_init(&adev->wb.lock); |
| 4331 | 4396 | ||
| 4397 | xa_init_flags(&adev->userq_xa, XA_FLAGS_LOCK_IRQ); | ||
| 4398 | |||
| 4332 | INIT_LIST_HEAD(&adev->reset_list); | 4399 | INIT_LIST_HEAD(&adev->reset_list); |
| 4333 | 4400 | ||
| 4334 | INIT_LIST_HEAD(&adev->ras_list); | 4401 | INIT_LIST_HEAD(&adev->ras_list); |
| 4335 | 4402 | ||
| 4336 | INIT_LIST_HEAD(&adev->pm.od_kobj_list); | 4403 | INIT_LIST_HEAD(&adev->pm.od_kobj_list); |
| 4337 | 4404 | ||
| 4405 | INIT_LIST_HEAD(&adev->userq_mgr_list); | ||
| 4406 | |||
| 4338 | INIT_DELAYED_WORK(&adev->delayed_init_work, | 4407 | INIT_DELAYED_WORK(&adev->delayed_init_work, |
| 4339 | amdgpu_device_delayed_init_work_handler); | 4408 | amdgpu_device_delayed_init_work_handler); |
| 4340 | INIT_DELAYED_WORK(&adev->gfx.gfx_off_delay_work, | 4409 | INIT_DELAYED_WORK(&adev->gfx.gfx_off_delay_work, |
| @@ -4907,28 +4976,20 @@ static int amdgpu_device_evict_resources(struct amdgpu_device *adev) · 566 unmodified lines | |||
| 4907 | * @data: data | 4976 | * @data: data |
| 4908 | * | 4977 | * |
| 4909 | * This function is called when the system is about to suspend or hibernate. | 4978 | * This function is called when the system is about to suspend or hibernate. |
| 4910 | * It is used to evict resources from the device before the system goes to | 4979 | * It is used to set the appropriate flags so that eviction can be optimized |
| 4911 | * sleep while there is still access to swap. | 4980 | * in the pm prepare callback. |
| 4912 | */ | 4981 | */ |
| 4913 | static int amdgpu_device_pm_notifier(struct notifier_block *nb, unsigned long mode, | 4982 | static int amdgpu_device_pm_notifier(struct notifier_block *nb, unsigned long mode, |
| 4914 | void *data) | 4983 | void *data) |
| 4915 | { | 4984 | { |
| 4916 | struct amdgpu_device *adev = container_of(nb, struct amdgpu_device, pm_nb); | 4985 | struct amdgpu_device *adev = container_of(nb, struct amdgpu_device, pm_nb); |
| 4917 | int r; | ||
| 4918 | 4986 | ||
| 4919 | switch (mode) { | 4987 | switch (mode) { |
| 4920 | case PM_HIBERNATION_PREPARE: | 4988 | case PM_HIBERNATION_PREPARE: |
| 4921 | adev->in_s4 = true; | 4989 | adev->in_s4 = true; |
| 4922 | fallthrough; | 4990 | break; |
| 4923 | case PM_SUSPEND_PREPARE: | 4991 | case PM_POST_HIBERNATION: |
| 4924 | r = amdgpu_device_evict_resources(adev); | 4992 | adev->in_s4 = false; |
| 4925 | /* | ||
| 4926 | * This is considered non-fatal at this time because | ||
| 4927 | * amdgpu_device_prepare() will also fatally evict resources. | ||
| 4928 | * See https://gitlab.freedesktop.org/drm/amd/-/issues/3781 | ||
| 4929 | */ | ||
| 4930 | if (r) | ||
| 4931 | drm_warn(adev_to_drm(adev), "Failed to evict resources, freeze active processes if problems occur: %d\n", r); | ||
| 4932 | break; | 4993 | break; |
| 4933 | } | 4994 | } |
| 4934 | 4995 | ||
| @@ -4949,15 +5010,13 @@ int amdgpu_device_prepare(struct drm_device *dev) · 14 unmodified lines | |||
| 4949 | struct amdgpu_device *adev = drm_to_adev(dev); | 5010 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 4950 | int i, r; | 5011 | int i, r; |
| 4951 | 5012 | ||
| 4952 | amdgpu_choose_low_power_state(adev); | ||
| 4953 | |||
| 4954 | if (dev->switch_power_state == DRM_SWITCH_POWER_OFF) | 5013 | if (dev->switch_power_state == DRM_SWITCH_POWER_OFF) |
| 4955 | return 0; | 5014 | return 0; |
| 4956 | 5015 | ||
| 4957 | /* Evict the majority of BOs before starting suspend sequence */ | 5016 | /* Evict the majority of BOs before starting suspend sequence */ |
| 4958 | r = amdgpu_device_evict_resources(adev); | 5017 | r = amdgpu_device_evict_resources(adev); |
| 4959 | if (r) | 5018 | if (r) |
| 4960 | goto unprepare; | 5019 | return r; |
| 4961 | 5020 | ||
| 4962 | flush_delayed_work(&adev->gfx.gfx_off_delay_work); | 5021 | flush_delayed_work(&adev->gfx.gfx_off_delay_work); |
| 4963 | 5022 | ||
| @@ -4968,15 +5027,10 @@ int amdgpu_device_prepare(struct drm_device *dev) · 4 unmodified lines | |||
| 4968 | continue; | 5027 | continue; |
| 4969 | r = adev->ip_blocks[i].version->funcs->prepare_suspend(&adev->ip_blocks[i]); | 5028 | r = adev->ip_blocks[i].version->funcs->prepare_suspend(&adev->ip_blocks[i]); |
| 4970 | if (r) | 5029 | if (r) |
| 4971 | goto unprepare; | 5030 | return r; |
| 4972 | } | 5031 | } |
| 4973 | 5032 | ||
| 4974 | return 0; | 5033 | return 0; |
| 4975 | |||
| 4976 | unprepare: | ||
| 4977 | adev->in_s0ix = adev->in_s3 = adev->in_s4 = false; | ||
| 4978 | |||
| 4979 | return r; | ||
| 4980 | } | 5034 | } |
| 4981 | 5035 | ||
| 4982 | /** | 5036 | /** |
| @@ -5018,8 +5072,10 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients) · 35 unmodified lines | |||
| 5018 | 5072 | ||
| 5019 | amdgpu_device_ip_suspend_phase1(adev); | 5073 | amdgpu_device_ip_suspend_phase1(adev); |
| 5020 | 5074 | ||
| 5021 | if (!adev->in_s0ix) | 5075 | if (!adev->in_s0ix) { |
| 5022 | amdgpu_amdkfd_suspend(adev, adev->in_runpm); | 5076 | amdgpu_amdkfd_suspend(adev, adev->in_runpm); |
| 5077 | amdgpu_userq_suspend(adev); | ||
| 5078 | } | ||
| 5023 | 5079 | ||
| 5024 | r = amdgpu_device_evict_resources(adev); | 5080 | r = amdgpu_device_evict_resources(adev); |
| 5025 | if (r) | 5081 | if (r) |
| @@ -5086,6 +5142,10 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) · 60 unmodified lines | |||
| 5086 | r = amdgpu_amdkfd_resume(adev, adev->in_runpm); | 5142 | r = amdgpu_amdkfd_resume(adev, adev->in_runpm); |
| 5087 | if (r) | 5143 | if (r) |
| 5088 | goto exit; | 5144 | goto exit; |
| 5145 | |||
| 5146 | r = amdgpu_userq_resume(adev); | ||
| 5147 | if (r) | ||
| 5148 | goto exit; | ||
| 5089 | } | 5149 | } |
| 5090 | 5150 | ||
| 5091 | r = amdgpu_device_ip_late_init(adev); | 5151 | r = amdgpu_device_ip_late_init(adev); |
| @@ -5134,9 +5194,6 @@ exit: · 42 unmodified lines | |||
| 5134 | } | 5194 | } |
| 5135 | adev->in_suspend = false; | 5195 | adev->in_suspend = false; |
| 5136 | 5196 | ||
| 5137 | if (adev->enable_mes) | ||
| 5138 | amdgpu_mes_self_test(adev); | ||
| 5139 | |||
| 5140 | if (amdgpu_acpi_smart_shift_update(dev, AMDGPU_SS_DEV_D0)) | 5197 | if (amdgpu_acpi_smart_shift_update(dev, AMDGPU_SS_DEV_D0)) |
| 5141 | DRM_WARN("smart shift update failed\n"); | 5198 | DRM_WARN("smart shift update failed\n"); |
| 5142 | 5199 | ||
| @@ -5517,6 +5574,29 @@ mode1_reset_failed: · 374 unmodified lines | |||
| 5517 | return ret; | 5574 | return ret; |
| 5518 | } | 5575 | } |
| 5519 | 5576 | ||
| 5577 | int amdgpu_device_link_reset(struct amdgpu_device *adev) | ||
| 5578 | { | ||
| 5579 | int ret = 0; | ||
| 5580 | |||
| 5581 | dev_info(adev->dev, "GPU link reset\n"); | ||
| 5582 | |||
| 5583 | if (!adev->pcie_reset_ctx.occurs_dpc) | ||
| 5584 | ret = amdgpu_dpm_link_reset(adev); | ||
| 5585 | |||
| 5586 | if (ret) | ||
| 5587 | goto link_reset_failed; | ||
| 5588 | |||
| 5589 | ret = amdgpu_psp_wait_for_bootloader(adev); | ||
| 5590 | if (ret) | ||
| 5591 | goto link_reset_failed; | ||
| 5592 | |||
| 5593 | return 0; | ||
| 5594 | |||
| 5595 | link_reset_failed: | ||
| 5596 | dev_err(adev->dev, "GPU link reset failed\n"); | ||
| 5597 | return ret; | ||
| 5598 | } | ||
| 5599 | |||
| 5520 | int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, | 5600 | int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, |
| 5521 | struct amdgpu_reset_context *reset_context) | 5601 | struct amdgpu_reset_context *reset_context) |
| 5522 | { | 5602 | { |
| @@ -5821,6 +5901,7 @@ static void amdgpu_device_set_mp1_state(struct amdgpu_device *adev) · 298 unmodified lines | |||
| 5821 | 5901 | ||
| 5822 | switch (amdgpu_asic_reset_method(adev)) { | 5902 | switch (amdgpu_asic_reset_method(adev)) { |
| 5823 | case AMD_RESET_METHOD_MODE1: | 5903 | case AMD_RESET_METHOD_MODE1: |
| 5904 | case AMD_RESET_METHOD_LINK: | ||
| 5824 | adev->mp1_state = PP_MP1_STATE_SHUTDOWN; | 5905 | adev->mp1_state = PP_MP1_STATE_SHUTDOWN; |
| 5825 | break; | 5906 | break; |
| 5826 | case AMD_RESET_METHOD_MODE2: | 5907 | case AMD_RESET_METHOD_MODE2: |
| @@ -5937,94 +6018,42 @@ static int amdgpu_device_health_check(struct list_head *device_list_handle) · 110 unmodified lines | |||
| 5937 | return ret; | 6018 | return ret; |
| 5938 | } | 6019 | } |
| 5939 | 6020 | ||
| 5940 | /** | 6021 | static int amdgpu_device_halt_activities(struct amdgpu_device *adev, |
| 5941 | * amdgpu_device_gpu_recover - reset the asic and recover scheduler | ||
| 5942 | * | ||
| 5943 | * @adev: amdgpu_device pointer | ||
| 5944 | * @job: which job trigger hang | ||
| 5945 | * @reset_context: amdgpu reset context pointer | ||
| 5946 | * | ||
| 5947 | * Attempt to reset the GPU if it has hung (all asics). | ||
| 5948 | * Attempt to do soft-reset or full-reset and reinitialize Asic | ||
| 5949 | * Returns 0 for success or an error on failure. | ||
| 5950 | */ | ||
| 5951 | |||
| 5952 | int amdgpu_device_gpu_recover(struct amdgpu_device *adev, | ||
| 5953 | struct amdgpu_job *job, | 6022 | struct amdgpu_job *job, |
| 5954 | struct amdgpu_reset_context *reset_context) | 6023 | struct amdgpu_reset_context *reset_context, |
| 6024 | struct list_head *device_list, | ||
| 6025 | struct amdgpu_hive_info *hive, | ||
| 6026 | bool need_emergency_restart) | ||
| 5955 | { | 6027 | { |
| 5956 | struct list_head device_list, *device_list_handle = NULL; | 6028 | struct list_head *device_list_handle = NULL; |
| 5957 | bool job_signaled = false; | ||
| 5958 | struct amdgpu_hive_info *hive = NULL; | ||
| 5959 | struct amdgpu_device *tmp_adev = NULL; | 6029 | struct amdgpu_device *tmp_adev = NULL; |
| 5960 | int i, r = 0; | 6030 | int i, r = 0; |
| 5961 | bool need_emergency_restart = false; | ||
| 5962 | bool audio_suspended = false; | ||
| 5963 | int retry_limit = AMDGPU_MAX_RETRY_LIMIT; | ||
| 5964 | 6031 | ||
| 5965 | /* | ||
| 5966 | * If it reaches here because of hang/timeout and a RAS error is | ||
| 5967 | * detected at the same time, let RAS recovery take care of it. | ||
| 5968 | */ | ||
| 5969 | if (amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY) && | ||
| 5970 | !amdgpu_sriov_vf(adev) && | ||
| 5971 | reset_context->src != AMDGPU_RESET_SRC_RAS) { | ||
| 5972 | dev_dbg(adev->dev, | ||
| 5973 | "Gpu recovery from source: %d yielding to RAS error recovery handling", | ||
| 5974 | reset_context->src); | ||
| 5975 | return 0; | ||
| 5976 | } | ||
| 5977 | /* | ||
| 5978 | * Special case: RAS triggered and full reset isn't supported | ||
| 5979 | */ | ||
| 5980 | need_emergency_restart = amdgpu_ras_need_emergency_restart(adev); | ||
| 5981 | |||
| 5982 | /* | ||
| 5983 | * Flush RAM to disk so that after reboot | ||
| 5984 | * the user can read log and see why the system rebooted. | ||
| 5985 | */ | ||
| 5986 | if (need_emergency_restart && amdgpu_ras_get_context(adev) && | ||
| 5987 | amdgpu_ras_get_context(adev)->reboot) { | ||
| 5988 | DRM_WARN("Emergency reboot."); | ||
| 5989 | |||
| 5990 | ksys_sync_helper(); | ||
| 5991 | emergency_restart(); | ||
| 5992 | } | ||
| 5993 | |||
| 5994 | dev_info(adev->dev, "GPU %s begin!\n", | ||
| 5995 | need_emergency_restart ? "jobs stop":"reset"); | ||
| 5996 | |||
| 5997 | if (!amdgpu_sriov_vf(adev)) | ||
| 5998 | hive = amdgpu_get_xgmi_hive(adev); | ||
| 5999 | if (hive) | ||
| 6000 | mutex_lock(&hive->hive_lock); | ||
| 6001 | |||
| 6002 | reset_context->job = job; | ||
| 6003 | reset_context->hive = hive; | ||
| 6004 | /* | 6032 | /* |
| 6005 | * Build list of devices to reset. | 6033 | * Build list of devices to reset. |
| 6006 | * In case we are in XGMI hive mode, resort the device list | 6034 | * In case we are in XGMI hive mode, resort the device list |
| 6007 | * to put adev in the 1st position. | 6035 | * to put adev in the 1st position. |
| 6008 | */ | 6036 | */ |
| 6009 | INIT_LIST_HEAD(&device_list); | ||
| 6010 | if (!amdgpu_sriov_vf(adev) && (adev->gmc.xgmi.num_physical_nodes > 1) && hive) { | 6037 | if (!amdgpu_sriov_vf(adev) && (adev->gmc.xgmi.num_physical_nodes > 1) && hive) { |
| 6011 | list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) { | 6038 | list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) { |
| 6012 | list_add_tail(&tmp_adev->reset_list, &device_list); | 6039 | list_add_tail(&tmp_adev->reset_list, device_list); |
| 6013 | if (adev->shutdown) | 6040 | if (adev->shutdown) |
| 6014 | tmp_adev->shutdown = true; | 6041 | tmp_adev->shutdown = true; |
| 6042 | if (adev->pcie_reset_ctx.occurs_dpc) | ||
| 6043 | tmp_adev->pcie_reset_ctx.in_link_reset = true; | ||
| 6015 | } | 6044 | } |
| 6016 | if (!list_is_first(&adev->reset_list, &device_list)) | 6045 | if (!list_is_first(&adev->reset_list, device_list)) |
| 6017 | list_rotate_to_front(&adev->reset_list, &device_list); | 6046 | list_rotate_to_front(&adev->reset_list, device_list); |
| 6018 | device_list_handle = &device_list; | 6047 | device_list_handle = device_list; |
| 6019 | } else { | 6048 | } else { |
| 6020 | list_add_tail(&adev->reset_list, &device_list); | 6049 | list_add_tail(&adev->reset_list, device_list); |
| 6021 | device_list_handle = &device_list; | 6050 | device_list_handle = device_list; |
| 6022 | } | 6051 | } |
| 6023 | 6052 | ||
| 6024 | if (!amdgpu_sriov_vf(adev)) { | 6053 | if (!amdgpu_sriov_vf(adev) && (!adev->pcie_reset_ctx.occurs_dpc)) { |
| 6025 | r = amdgpu_device_health_check(device_list_handle); | 6054 | r = amdgpu_device_health_check(device_list_handle); |
| 6026 | if (r) | 6055 | if (r) |
| 6027 | goto end_reset; | 6056 | return r; |
| 6028 | } | 6057 | } |
| 6029 | 6058 | ||
| 6030 | /* We need to lock reset domain only once both for XGMI and single device */ | 6059 | /* We need to lock reset domain only once both for XGMI and single device */ |
| @@ -6048,7 +6077,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 17 unmodified lines | |||
| 6048 | * some audio codec errors. | 6077 | * some audio codec errors. |
| 6049 | */ | 6078 | */ |
| 6050 | if (!amdgpu_device_suspend_display_audio(tmp_adev)) | 6079 | if (!amdgpu_device_suspend_display_audio(tmp_adev)) |
| 6051 | audio_suspended = true; | 6080 | tmp_adev->pcie_reset_ctx.audio_suspended = true; |
| 6052 | 6081 | ||
| 6053 | amdgpu_ras_set_error_query_ready(tmp_adev, false); | 6082 | amdgpu_ras_set_error_query_ready(tmp_adev, false); |
| 6054 | 6083 | ||
| @@ -6066,6 +6095,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 11 unmodified lines | |||
| 6066 | 6095 | ||
| 6067 | /* disable ras on ALL IPs */ | 6096 | /* disable ras on ALL IPs */ |
| 6068 | if (!need_emergency_restart && | 6097 | if (!need_emergency_restart && |
| 6098 | (!adev->pcie_reset_ctx.occurs_dpc) && | ||
| 6069 | amdgpu_device_ip_need_full_reset(tmp_adev)) | 6099 | amdgpu_device_ip_need_full_reset(tmp_adev)) |
| 6070 | amdgpu_ras_suspend(tmp_adev); | 6100 | amdgpu_ras_suspend(tmp_adev); |
| 6071 | 6101 | ||
| @@ -6083,24 +6113,24 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 11 unmodified lines | |||
| 6083 | atomic_inc(&tmp_adev->gpu_reset_counter); | 6113 | atomic_inc(&tmp_adev->gpu_reset_counter); |
| 6084 | } | 6114 | } |
| 6085 | 6115 | ||
| 6086 | if (need_emergency_restart) | 6116 | return r; |
| 6087 | goto skip_sched_resume; | 6117 | } |
| 6088 | 6118 | ||
| 6089 | /* | 6119 | static int amdgpu_device_asic_reset(struct amdgpu_device *adev, |
| 6090 | * Must check guilty signal here since after this point all old | 6120 | struct list_head *device_list, |
| 6091 | * HW fences are force signaled. | 6121 | struct amdgpu_reset_context *reset_context) |
| 6092 | * | 6122 | { |
| 6093 | * job->base holds a reference to parent fence | 6123 | struct amdgpu_device *tmp_adev = NULL; |
| 6094 | */ | 6124 | int retry_limit = AMDGPU_MAX_RETRY_LIMIT; |
| 6095 | if (job && dma_fence_is_signaled(&job->hw_fence)) { | 6125 | int r = 0; |
| 6096 | job_signaled = true; | ||
| 6097 | dev_info(adev->dev, "Guilty job already signaled, skipping HW reset"); | ||
| 6098 | goto skip_hw_reset; | ||
| 6099 | } | ||
| 6100 | 6126 | ||
| 6101 | retry: /* Rest of adevs pre asic reset from XGMI hive. */ | 6127 | retry: /* Rest of adevs pre asic reset from XGMI hive. */ |
| 6102 | list_for_each_entry(tmp_adev, device_list_handle, reset_list) { | 6128 | list_for_each_entry(tmp_adev, device_list, reset_list) { |
| 6129 | if (adev->pcie_reset_ctx.occurs_dpc) | ||
| 6130 | tmp_adev->no_hw_access = true; | ||
| 6103 | r = amdgpu_device_pre_asic_reset(tmp_adev, reset_context); | 6131 | r = amdgpu_device_pre_asic_reset(tmp_adev, reset_context); |
| 6132 | if (adev->pcie_reset_ctx.occurs_dpc) | ||
| 6133 | tmp_adev->no_hw_access = false; | ||
| 6104 | /*TODO Should we stop ?*/ | 6134 | /*TODO Should we stop ?*/ |
| 6105 | if (r) { | 6135 | if (r) { |
| 6106 | dev_err(tmp_adev->dev, "GPU pre asic reset failed with err, %d for drm dev, %s ", | 6136 | dev_err(tmp_adev->dev, "GPU pre asic reset failed with err, %d for drm dev, %s ", |
| @@ -6112,6 +6142,11 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 5 unmodified lines | |||
| 6112 | /* Actual ASIC resets if needed.*/ | 6142 | /* Actual ASIC resets if needed.*/ |
| 6113 | /* Host driver will handle XGMI hive reset for SRIOV */ | 6143 | /* Host driver will handle XGMI hive reset for SRIOV */ |
| 6114 | if (amdgpu_sriov_vf(adev)) { | 6144 | if (amdgpu_sriov_vf(adev)) { |
| 6145 | |||
| 6146 | /* Bail out of reset early */ | ||
| 6147 | if (amdgpu_ras_is_rma(adev)) | ||
| 6148 | return -ENODEV; | ||
| 6149 | |||
| 6115 | if (amdgpu_ras_get_fed_status(adev) || amdgpu_virt_rcvd_ras_interrupt(adev)) { | 6150 | if (amdgpu_ras_get_fed_status(adev) || amdgpu_virt_rcvd_ras_interrupt(adev)) { |
| 6116 | dev_dbg(adev->dev, "Detected RAS error, wait for FLR completion\n"); | 6151 | dev_dbg(adev->dev, "Detected RAS error, wait for FLR completion\n"); |
| 6117 | amdgpu_ras_set_fed(adev, true); | 6152 | amdgpu_ras_set_fed(adev, true); |
| @@ -6126,12 +6161,12 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 8 unmodified lines | |||
| 6126 | if (r) | 6161 | if (r) |
| 6127 | adev->asic_reset_res = r; | 6162 | adev->asic_reset_res = r; |
| 6128 | } else { | 6163 | } else { |
| 6129 | r = amdgpu_do_asic_reset(device_list_handle, reset_context); | 6164 | r = amdgpu_do_asic_reset(device_list, reset_context); |
| 6130 | if (r && r == -EAGAIN) | 6165 | if (r && r == -EAGAIN) |
| 6131 | goto retry; | 6166 | goto retry; |
| 6132 | } | 6167 | } |
| 6133 | 6168 | ||
| 6134 | list_for_each_entry(tmp_adev, device_list_handle, reset_list) { | 6169 | list_for_each_entry(tmp_adev, device_list, reset_list) { |
| 6135 | /* | 6170 | /* |
| 6136 | * Drop any pending non scheduler resets queued before reset is done. | 6171 | * Drop any pending non scheduler resets queued before reset is done. |
| 6137 | * Any reset scheduled after this point would be valid. Scheduler resets | 6172 | * Any reset scheduled after this point would be valid. Scheduler resets |
| @@ -6141,10 +6176,18 @@ retry: /* Rest of adevs pre asic reset from XGMI hive. */ · 3 unmodified lines | |||
| 6141 | amdgpu_device_stop_pending_resets(tmp_adev); | 6176 | amdgpu_device_stop_pending_resets(tmp_adev); |
| 6142 | } | 6177 | } |
| 6143 | 6178 | ||
| 6144 | skip_hw_reset: | 6179 | return r; |
| 6180 | } | ||
| 6181 | |||
| 6182 | static int amdgpu_device_sched_resume(struct list_head *device_list, | ||
| 6183 | struct amdgpu_reset_context *reset_context, | ||
| 6184 | bool job_signaled) | ||
| 6185 | { | ||
| 6186 | struct amdgpu_device *tmp_adev = NULL; | ||
| 6187 | int i, r = 0; | ||
| 6145 | 6188 | ||
| 6146 | /* Post ASIC reset for all devs .*/ | 6189 | /* Post ASIC reset for all devs .*/ |
| 6147 | list_for_each_entry(tmp_adev, device_list_handle, reset_list) { | 6190 | list_for_each_entry(tmp_adev, device_list, reset_list) { |
| 6148 | 6191 | ||
| 6149 | for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { | 6192 | for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { |
| 6150 | struct amdgpu_ring *ring = tmp_adev->rings[i]; | 6193 | struct amdgpu_ring *ring = tmp_adev->rings[i]; |
| @@ -6180,8 +6223,16 @@ skip_hw_reset: · 29 unmodified lines | |||
| 6180 | } | 6223 | } |
| 6181 | } | 6224 | } |
| 6182 | 6225 | ||
| 6183 | skip_sched_resume: | 6226 | return r; |
| 6184 | list_for_each_entry(tmp_adev, device_list_handle, reset_list) { | 6227 | } |
| 6228 | |||
| 6229 | static void amdgpu_device_gpu_resume(struct amdgpu_device *adev, | ||
| 6230 | struct list_head *device_list, | ||
| 6231 | bool need_emergency_restart) | ||
| 6232 | { | ||
| 6233 | struct amdgpu_device *tmp_adev = NULL; | ||
| 6234 | |||
| 6235 | list_for_each_entry(tmp_adev, device_list, reset_list) { | ||
| 6185 | /* unlock kfd: SRIOV would do it separately */ | 6236 | /* unlock kfd: SRIOV would do it separately */ |
| 6186 | if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev)) | 6237 | if (!need_emergency_restart && !amdgpu_sriov_vf(tmp_adev)) |
| 6187 | amdgpu_amdkfd_post_reset(tmp_adev); | 6238 | amdgpu_amdkfd_post_reset(tmp_adev); |
| @@ -6192,18 +6243,114 @@ skip_sched_resume: · 4 unmodified lines | |||
| 6192 | if (!adev->kfd.init_complete) | 6243 | if (!adev->kfd.init_complete) |
| 6193 | amdgpu_amdkfd_device_init(adev); | 6244 | amdgpu_amdkfd_device_init(adev); |
| 6194 | 6245 | ||
| 6195 | if (audio_suspended) | 6246 | if (tmp_adev->pcie_reset_ctx.audio_suspended) |
| 6196 | amdgpu_device_resume_display_audio(tmp_adev); | 6247 | amdgpu_device_resume_display_audio(tmp_adev); |
| 6197 | 6248 | ||
| 6198 | amdgpu_device_unset_mp1_state(tmp_adev); | 6249 | amdgpu_device_unset_mp1_state(tmp_adev); |
| 6199 | 6250 | ||
| 6200 | amdgpu_ras_set_error_query_ready(tmp_adev, true); | 6251 | amdgpu_ras_set_error_query_ready(tmp_adev, true); |
| 6252 | |||
| 6201 | } | 6253 | } |
| 6202 | 6254 | ||
| 6203 | tmp_adev = list_first_entry(device_list_handle, struct amdgpu_device, | 6255 | tmp_adev = list_first_entry(device_list, struct amdgpu_device, |
| 6204 | reset_list); | 6256 | reset_list); |
| 6205 | amdgpu_device_unlock_reset_domain(tmp_adev->reset_domain); | 6257 | amdgpu_device_unlock_reset_domain(tmp_adev->reset_domain); |
| 6206 | 6258 | ||
| 6259 | } | ||
| 6260 | |||
| 6261 | |||
| 6262 | /** | ||
| 6263 | * amdgpu_device_gpu_recover - reset the asic and recover scheduler | ||
| 6264 | * | ||
| 6265 | * @adev: amdgpu_device pointer | ||
| 6266 | * @job: which job trigger hang | ||
| 6267 | * @reset_context: amdgpu reset context pointer | ||
| 6268 | * | ||
| 6269 | * Attempt to reset the GPU if it has hung (all asics). | ||
| 6270 | * Attempt to do soft-reset or full-reset and reinitialize Asic | ||
| 6271 | * Returns 0 for success or an error on failure. | ||
| 6272 | */ | ||
| 6273 | |||
| 6274 | int amdgpu_device_gpu_recover(struct amdgpu_device *adev, | ||
| 6275 | struct amdgpu_job *job, | ||
| 6276 | struct amdgpu_reset_context *reset_context) | ||
| 6277 | { | ||
| 6278 | struct list_head device_list; | ||
| 6279 | bool job_signaled = false; | ||
| 6280 | struct amdgpu_hive_info *hive = NULL; | ||
| 6281 | int r = 0; | ||
| 6282 | bool need_emergency_restart = false; | ||
| 6283 | |||
| 6284 | /* | ||
| 6285 | * If it reaches here because of hang/timeout and a RAS error is | ||
| 6286 | * detected at the same time, let RAS recovery take care of it. | ||
| 6287 | */ | ||
| 6288 | if (amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY) && | ||
| 6289 | !amdgpu_sriov_vf(adev) && | ||
| 6290 | reset_context->src != AMDGPU_RESET_SRC_RAS) { | ||
| 6291 | dev_dbg(adev->dev, | ||
| 6292 | "Gpu recovery from source: %d yielding to RAS error recovery handling", | ||
| 6293 | reset_context->src); | ||
| 6294 | return 0; | ||
| 6295 | } | ||
| 6296 | |||
| 6297 | /* | ||
| 6298 | * Special case: RAS triggered and full reset isn't supported | ||
| 6299 | */ | ||
| 6300 | need_emergency_restart = amdgpu_ras_need_emergency_restart(adev); | ||
| 6301 | |||
| 6302 | /* | ||
| 6303 | * Flush RAM to disk so that after reboot | ||
| 6304 | * the user can read log and see why the system rebooted. | ||
| 6305 | */ | ||
| 6306 | if (need_emergency_restart && amdgpu_ras_get_context(adev) && | ||
| 6307 | amdgpu_ras_get_context(adev)->reboot) { | ||
| 6308 | DRM_WARN("Emergency reboot."); | ||
| 6309 | |||
| 6310 | ksys_sync_helper(); | ||
| 6311 | emergency_restart(); | ||
| 6312 | } | ||
| 6313 | |||
| 6314 | dev_info(adev->dev, "GPU %s begin!\n", | ||
| 6315 | need_emergency_restart ? "jobs stop":"reset"); | ||
| 6316 | |||
| 6317 | if (!amdgpu_sriov_vf(adev)) | ||
| 6318 | hive = amdgpu_get_xgmi_hive(adev); | ||
| 6319 | if (hive) | ||
| 6320 | mutex_lock(&hive->hive_lock); | ||
| 6321 | |||
| 6322 | reset_context->job = job; | ||
| 6323 | reset_context->hive = hive; | ||
| 6324 | INIT_LIST_HEAD(&device_list); | ||
| 6325 | |||
| 6326 | r = amdgpu_device_halt_activities(adev, job, reset_context, &device_list, | ||
| 6327 | hive, need_emergency_restart); | ||
| 6328 | if (r) | ||
| 6329 | goto end_reset; | ||
| 6330 | |||
| 6331 | if (need_emergency_restart) | ||
| 6332 | goto skip_sched_resume; | ||
| 6333 | /* | ||
| 6334 | * Must check guilty signal here since after this point all old | ||
| 6335 | * HW fences are force signaled. | ||
| 6336 | * | ||
| 6337 | * job->base holds a reference to parent fence | ||
| 6338 | */ | ||
| 6339 | if (job && dma_fence_is_signaled(&job->hw_fence)) { | ||
| 6340 | job_signaled = true; | ||
| 6341 | dev_info(adev->dev, "Guilty job already signaled, skipping HW reset"); | ||
| 6342 | goto skip_hw_reset; | ||
| 6343 | } | ||
| 6344 | |||
| 6345 | r = amdgpu_device_asic_reset(adev, &device_list, reset_context); | ||
| 6346 | if (r) | ||
| 6347 | goto end_reset; | ||
| 6348 | skip_hw_reset: | ||
| 6349 | r = amdgpu_device_sched_resume(&device_list, reset_context, job_signaled); | ||
| 6350 | if (r) | ||
| 6351 | goto end_reset; | ||
| 6352 | skip_sched_resume: | ||
| 6353 | amdgpu_device_gpu_resume(adev, &device_list, need_emergency_restart); | ||
| 6207 | end_reset: | 6354 | end_reset: |
| 6208 | if (hive) { | 6355 | if (hive) { |
| 6209 | mutex_unlock(&hive->hive_lock); | 6356 | mutex_unlock(&hive->hive_lock); |
| @@ -6587,12 +6734,15 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 377 unmodified lines | |||
| 6587 | { | 6734 | { |
| 6588 | struct drm_device *dev = pci_get_drvdata(pdev); | 6735 | struct drm_device *dev = pci_get_drvdata(pdev); |
| 6589 | struct amdgpu_device *adev = drm_to_adev(dev); | 6736 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 6590 | int i; | 6737 | struct amdgpu_hive_info *hive = amdgpu_get_xgmi_hive(adev); |
| 6738 | struct amdgpu_reset_context reset_context; | ||
| 6739 | struct list_head device_list; | ||
| 6740 | int r = 0; | ||
| 6591 | 6741 | ||
| 6592 | DRM_INFO("PCI error: detected callback, state(%d)!!\n", state); | 6742 | dev_info(adev->dev, "PCI error: detected callback!!\n"); |
| 6593 | 6743 | ||
| 6594 | if (adev->gmc.xgmi.num_physical_nodes > 1) { | 6744 | if (!amdgpu_dpm_is_link_reset_supported(adev)) { |
| 6595 | DRM_WARN("No support for XGMI hive yet..."); | 6745 | dev_warn(adev->dev, "No support for XGMI hive yet...\n"); |
| 6596 | return PCI_ERS_RESULT_DISCONNECT; | 6746 | return PCI_ERS_RESULT_DISCONNECT; |
| 6597 | } | 6747 | } |
| 6598 | 6748 | ||
| @@ -6600,32 +6750,30 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 1 unmodified line | |||
| 6600 | 6750 | ||
| 6601 | switch (state) { | 6751 | switch (state) { |
| 6602 | case pci_channel_io_normal: | 6752 | case pci_channel_io_normal: |
| 6753 | dev_info(adev->dev, "pci_channel_io_normal: state(%d)!!\n", state); | ||
| 6603 | return PCI_ERS_RESULT_CAN_RECOVER; | 6754 | return PCI_ERS_RESULT_CAN_RECOVER; |
| 6604 | /* Fatal error, prepare for slot reset */ | ||
| 6605 | case pci_channel_io_frozen: | 6755 | case pci_channel_io_frozen: |
| 6606 | /* | 6756 | /* Fatal error, prepare for slot reset */ |
| 6607 | * Locking adev->reset_domain->sem will prevent any external access | 6757 | dev_info(adev->dev, "pci_channel_io_frozen: state(%d)!!\n", state); |
| 6608 | * to GPU during PCI error recovery | 6758 | |
| 6609 | */ | 6759 | if (hive) |
| 6610 | amdgpu_device_lock_reset_domain(adev->reset_domain); | 6760 | mutex_lock(&hive->hive_lock); |
| 6611 | amdgpu_device_set_mp1_state(adev); | 6761 | adev->pcie_reset_ctx.occurs_dpc = true; |
| 6612 | 6762 | memset(&reset_context, 0, sizeof(reset_context)); | |
| 6613 | /* | 6763 | INIT_LIST_HEAD(&device_list); |
| 6614 | * Block any work scheduling as we do for regular GPU reset | 6764 | |
| 6615 | * for the duration of the recovery | 6765 | r = amdgpu_device_halt_activities(adev, NULL, &reset_context, &device_list, |
| 6616 | */ | 6766 | hive, false); |
| 6617 | for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { | 6767 | if (hive) { |
| 6618 | struct amdgpu_ring *ring = adev->rings[i]; | 6768 | mutex_unlock(&hive->hive_lock); |
| 6619 | 6769 | amdgpu_put_xgmi_hive(hive); | |
| 6620 | if (!amdgpu_ring_sched_ready(ring)) | ||
| 6621 | continue; | ||
| 6622 | |||
| 6623 | drm_sched_stop(&ring->sched, NULL); | ||
| 6624 | } | 6770 | } |
| 6625 | atomic_inc(&adev->gpu_reset_counter); | 6771 | if (r) |
| 6772 | return PCI_ERS_RESULT_DISCONNECT; | ||
| 6626 | return PCI_ERS_RESULT_NEED_RESET; | 6773 | return PCI_ERS_RESULT_NEED_RESET; |
| 6627 | case pci_channel_io_perm_failure: | 6774 | case pci_channel_io_perm_failure: |
| 6628 | /* Permanent error, prepare for device removal */ | 6775 | /* Permanent error, prepare for device removal */ |
| 6776 | dev_info(adev->dev, "pci_channel_io_perm_failure: state(%d)!!\n", state); | ||
| 6629 | return PCI_ERS_RESULT_DISCONNECT; | 6777 | return PCI_ERS_RESULT_DISCONNECT; |
| 6630 | } | 6778 | } |
| 6631 | 6779 | ||
| @@ -6638,8 +6786,10 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta · 6 unmodified lines | |||
| 6638 | */ | 6786 | */ |
| 6639 | pci_ers_result_t amdgpu_pci_mmio_enabled(struct pci_dev *pdev) | 6787 | pci_ers_result_t amdgpu_pci_mmio_enabled(struct pci_dev *pdev) |
| 6640 | { | 6788 | { |
| 6789 | struct drm_device *dev = pci_get_drvdata(pdev); | ||
| 6790 | struct amdgpu_device *adev = drm_to_adev(dev); | ||
| 6641 | 6791 | ||
| 6642 | DRM_INFO("PCI error: mmio enabled callback!!\n"); | 6792 | dev_info(adev->dev, "PCI error: mmio enabled callback!!\n"); |
| 6643 | 6793 | ||
| 6644 | /* TODO - dump whatever for debugging purposes */ | 6794 | /* TODO - dump whatever for debugging purposes */ |
| 6645 | 6795 | ||
| @@ -6663,10 +6813,12 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 17 unmodified lines | |||
| 6663 | { | 6813 | { |
| 6664 | struct drm_device *dev = pci_get_drvdata(pdev); | 6814 | struct drm_device *dev = pci_get_drvdata(pdev); |
| 6665 | struct amdgpu_device *adev = drm_to_adev(dev); | 6815 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 6666 | int r, i; | ||
| 6667 | struct amdgpu_reset_context reset_context; | 6816 | struct amdgpu_reset_context reset_context; |
| 6668 | u32 memsize; | 6817 | struct amdgpu_device *tmp_adev; |
| 6818 | struct amdgpu_hive_info *hive; | ||
| 6669 | struct list_head device_list; | 6819 | struct list_head device_list; |
| 6820 | int r = 0, i; | ||
| 6821 | u32 memsize; | ||
| 6670 | 6822 | ||
| 6671 | /* PCI error slot reset should be skipped During RAS recovery */ | 6823 | /* PCI error slot reset should be skipped During RAS recovery */ |
| 6672 | if ((amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) || | 6824 | if ((amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) || |
| @@ -6674,15 +6826,12 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 1 unmodified line | |||
| 6674 | amdgpu_ras_in_recovery(adev)) | 6826 | amdgpu_ras_in_recovery(adev)) |
| 6675 | return PCI_ERS_RESULT_RECOVERED; | 6827 | return PCI_ERS_RESULT_RECOVERED; |
| 6676 | 6828 | ||
| 6677 | DRM_INFO("PCI error: slot reset callback!!\n"); | 6829 | dev_info(adev->dev, "PCI error: slot reset callback!!\n"); |
| 6678 | 6830 | ||
| 6679 | memset(&reset_context, 0, sizeof(reset_context)); | 6831 | memset(&reset_context, 0, sizeof(reset_context)); |
| 6680 | 6832 | ||
| 6681 | INIT_LIST_HEAD(&device_list); | ||
| 6682 | list_add_tail(&adev->reset_list, &device_list); | ||
| 6683 | |||
| 6684 | /* wait for asic to come out of reset */ | 6833 | /* wait for asic to come out of reset */ |
| 6685 | msleep(500); | 6834 | msleep(700); |
| 6686 | 6835 | ||
| 6687 | /* Restore PCI confspace */ | 6836 | /* Restore PCI confspace */ |
| 6688 | amdgpu_device_load_pci_state(pdev); | 6837 | amdgpu_device_load_pci_state(pdev); |
| @@ -6703,26 +6852,40 @@ pci_ers_result_t amdgpu_pci_slot_reset(struct pci_dev *pdev) · 14 unmodified lines | |||
| 6703 | reset_context.method = AMD_RESET_METHOD_NONE; | 6852 | reset_context.method = AMD_RESET_METHOD_NONE; |
| 6704 | reset_context.reset_req_dev = adev; | 6853 | reset_context.reset_req_dev = adev; |
| 6705 | set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags); | 6854 | set_bit(AMDGPU_NEED_FULL_RESET, &reset_context.flags); |
| 6706 | set_bit(AMDGPU_SKIP_HW_RESET, &reset_context.flags); | 6855 | set_bit(AMDGPU_SKIP_COREDUMP, &reset_context.flags); |
| 6707 | 6856 | INIT_LIST_HEAD(&device_list); | |
| 6708 | adev->no_hw_access = true; | ||
| 6709 | r = amdgpu_device_pre_asic_reset(adev, &reset_context); | ||
| 6710 | adev->no_hw_access = false; | ||
| 6711 | if (r) | ||
| 6712 | goto out; | ||
| 6713 | 6857 | ||
| 6714 | r = amdgpu_do_asic_reset(&device_list, &reset_context); | 6858 | hive = amdgpu_get_xgmi_hive(adev); |
| 6859 | if (hive) { | ||
| 6860 | mutex_lock(&hive->hive_lock); | ||
| 6861 | reset_context.hive = hive; | ||
| 6862 | list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) { | ||
| 6863 | tmp_adev->pcie_reset_ctx.in_link_reset = true; | ||
| 6864 | list_add_tail(&tmp_adev->reset_list, &device_list); | ||
| 6865 | } | ||
| 6866 | } else { | ||
| 6867 | set_bit(AMDGPU_SKIP_HW_RESET, &reset_context.flags); | ||
| 6868 | list_add_tail(&adev->reset_list, &device_list); | ||
| 6869 | } | ||
| 6715 | 6870 | ||
| 6871 | r = amdgpu_device_asic_reset(adev, &device_list, &reset_context); | ||
| 6716 | out: | 6872 | out: |
| 6717 | if (!r) { | 6873 | if (!r) { |
| 6718 | if (amdgpu_device_cache_pci_state(adev->pdev)) | 6874 | if (amdgpu_device_cache_pci_state(adev->pdev)) |
| 6719 | pci_restore_state(adev->pdev); | 6875 | pci_restore_state(adev->pdev); |
| 6720 | 6876 | dev_info(adev->dev, "PCIe error recovery succeeded\n"); | |
| 6721 | DRM_INFO("PCIe error recovery succeeded\n"); | ||
| 6722 | } else { | 6877 | } else { |
| 6723 | DRM_ERROR("PCIe error recovery failed, err:%d", r); | 6878 | dev_err(adev->dev, "PCIe error recovery failed, err:%d\n", r); |
| 6724 | amdgpu_device_unset_mp1_state(adev); | 6879 | if (hive) { |
| 6725 | amdgpu_device_unlock_reset_domain(adev->reset_domain); | 6880 | list_for_each_entry(tmp_adev, &device_list, reset_list) |
| 6881 | amdgpu_device_unset_mp1_state(tmp_adev); | ||
| 6882 | amdgpu_device_unlock_reset_domain(adev->reset_domain); | ||
| 6883 | } | ||
| 6884 | } | ||
| 6885 | |||
| 6886 | if (hive) { | ||
| 6887 | mutex_unlock(&hive->hive_lock); | ||
| 6888 | amdgpu_put_xgmi_hive(hive); | ||
| 6726 | } | 6889 | } |
| 6727 | 6890 | ||
| 6728 | return r ? PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_RECOVERED; | 6891 | return r ? PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_RECOVERED; |
| @@ -6739,26 +6902,36 @@ void amdgpu_pci_resume(struct pci_dev *pdev) · 10 unmodified lines | |||
| 6739 | { | 6902 | { |
| 6740 | struct drm_device *dev = pci_get_drvdata(pdev); | 6903 | struct drm_device *dev = pci_get_drvdata(pdev); |
| 6741 | struct amdgpu_device *adev = drm_to_adev(dev); | 6904 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 6742 | int i; | 6905 | struct list_head device_list; |
| 6743 | 6906 | struct amdgpu_hive_info *hive = NULL; | |
| 6907 | struct amdgpu_device *tmp_adev = NULL; | ||
| 6744 | 6908 | ||
| 6745 | DRM_INFO("PCI error: resume callback!!\n"); | 6909 | dev_info(adev->dev, "PCI error: resume callback!!\n"); |
| 6746 | 6910 | ||
| 6747 | /* Only continue execution for the case of pci_channel_io_frozen */ | 6911 | /* Only continue execution for the case of pci_channel_io_frozen */ |
| 6748 | if (adev->pci_channel_state != pci_channel_io_frozen) | 6912 | if (adev->pci_channel_state != pci_channel_io_frozen) |
| 6749 | return; | 6913 | return; |
| 6750 | 6914 | ||
| 6751 | for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { | 6915 | INIT_LIST_HEAD(&device_list); |
| 6752 | struct amdgpu_ring *ring = adev->rings[i]; | ||
| 6753 | 6916 | ||
| 6754 | if (!amdgpu_ring_sched_ready(ring)) | 6917 | hive = amdgpu_get_xgmi_hive(adev); |
| 6755 | continue; | 6918 | if (hive) { |
| 6919 | mutex_lock(&hive->hive_lock); | ||
| 6920 | list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) { | ||
| 6921 | tmp_adev->pcie_reset_ctx.in_link_reset = false; | ||
| 6922 | list_add_tail(&tmp_adev->reset_list, &device_list); | ||
| 6923 | } | ||
| 6924 | } else | ||
| 6925 | list_add_tail(&adev->reset_list, &device_list); | ||
| 6756 | 6926 | ||
| 6757 | drm_sched_start(&ring->sched, 0); | 6927 | amdgpu_device_sched_resume(&device_list, NULL, NULL); |
| 6758 | } | 6928 | amdgpu_device_gpu_resume(adev, &device_list, false); |
| 6929 | adev->pcie_reset_ctx.occurs_dpc = false; | ||
| 6759 | 6930 | ||
| 6760 | amdgpu_device_unset_mp1_state(adev); | 6931 | if (hive) { |
| 6761 | amdgpu_device_unlock_reset_domain(adev->reset_domain); | 6932 | mutex_unlock(&hive->hive_lock); |
| 6933 | amdgpu_put_xgmi_hive(hive); | ||
| 6934 | } | ||
| 6762 | } | 6935 | } |
| 6763 | 6936 | ||
| 6764 | bool amdgpu_device_cache_pci_state(struct pci_dev *pdev) | 6937 | bool amdgpu_device_cache_pci_state(struct pci_dev *pdev) |
| @@ -51,6 +51,8 @@ · 50 unmodified lines | |||
| 51 | #include "amdgpu_reset.h" | 51 | #include "amdgpu_reset.h" |
| 52 | #include "amdgpu_sched.h" | 52 | #include "amdgpu_sched.h" |
| 53 | #include "amdgpu_xgmi.h" | 53 | #include "amdgpu_xgmi.h" |
| 54 | #include "amdgpu_userq.h" | ||
| 55 | #include "amdgpu_userq_fence.h" | ||
| 54 | #include "../amdxcp/amdgpu_xcp_drv.h" | 56 | #include "../amdxcp/amdgpu_xcp_drv.h" |
| 55 | 57 | ||
| 56 | /* | 58 | /* |
| @@ -123,9 +125,10 @@ · 66 unmodified lines | |||
| 123 | * - 3.61.0 - Contains fix for RV/PCO compute queues | 125 | * - 3.61.0 - Contains fix for RV/PCO compute queues |
| 124 | * - 3.62.0 - Add AMDGPU_IDS_FLAGS_MODE_PF, AMDGPU_IDS_FLAGS_MODE_VF & AMDGPU_IDS_FLAGS_MODE_PT | 126 | * - 3.62.0 - Add AMDGPU_IDS_FLAGS_MODE_PF, AMDGPU_IDS_FLAGS_MODE_VF & AMDGPU_IDS_FLAGS_MODE_PT |
| 125 | * - 3.63.0 - GFX12 display DCC supports 256B max compressed block size | 127 | * - 3.63.0 - GFX12 display DCC supports 256B max compressed block size |
| 128 | * - 3.64.0 - Userq IP support query | ||
| 126 | */ | 129 | */ |
| 127 | #define KMS_DRIVER_MAJOR 3 | 130 | #define KMS_DRIVER_MAJOR 3 |
| 128 | #define KMS_DRIVER_MINOR 63 | 131 | #define KMS_DRIVER_MINOR 64 |
| 129 | #define KMS_DRIVER_PATCHLEVEL 0 | 132 | #define KMS_DRIVER_PATCHLEVEL 0 |
| 130 | 133 | ||
| 131 | /* | 134 | /* |
| @@ -140,6 +143,7 @@ enum AMDGPU_DEBUG_MASK { · 8 unmodified lines | |||
| 140 | AMDGPU_DEBUG_ENABLE_EXP_RESETS = BIT(5), | 143 | AMDGPU_DEBUG_ENABLE_EXP_RESETS = BIT(5), |
| 141 | AMDGPU_DEBUG_DISABLE_GPU_RING_RESET = BIT(6), | 144 | AMDGPU_DEBUG_DISABLE_GPU_RING_RESET = BIT(6), |
| 142 | AMDGPU_DEBUG_SMU_POOL = BIT(7), | 145 | AMDGPU_DEBUG_SMU_POOL = BIT(7), |
| 146 | AMDGPU_DEBUG_VM_USERPTR = BIT(8), | ||
| 143 | }; | 147 | }; |
| 144 | 148 | ||
| 145 | unsigned int amdgpu_vram_limit = UINT_MAX; | 149 | unsigned int amdgpu_vram_limit = UINT_MAX; |
| @@ -176,7 +180,7 @@ uint amdgpu_pg_mask = 0xffffffff; · 30 unmodified lines | |||
| 176 | uint amdgpu_sdma_phase_quantum = 32; | 180 | uint amdgpu_sdma_phase_quantum = 32; |
| 177 | char *amdgpu_disable_cu; | 181 | char *amdgpu_disable_cu; |
| 178 | char *amdgpu_virtual_display; | 182 | char *amdgpu_virtual_display; |
| 179 | bool enforce_isolation; | 183 | int amdgpu_enforce_isolation = -1; |
| 180 | int amdgpu_modeset = -1; | 184 | int amdgpu_modeset = -1; |
| 181 | 185 | ||
| 182 | /* Specifies the default granularity for SVM, used in buffer | 186 | /* Specifies the default granularity for SVM, used in buffer |
| @@ -238,6 +242,8 @@ int amdgpu_agp = -1; /* auto */ · 55 unmodified lines | |||
| 238 | int amdgpu_wbrf = -1; | 242 | int amdgpu_wbrf = -1; |
| 239 | int amdgpu_damage_clips = -1; /* auto */ | 243 | int amdgpu_damage_clips = -1; /* auto */ |
| 240 | int amdgpu_umsch_mm_fwlog; | 244 | int amdgpu_umsch_mm_fwlog; |
| 245 | int amdgpu_rebar = -1; /* auto */ | ||
| 246 | int amdgpu_user_queue = -1; | ||
| 241 | 247 | ||
| 242 | DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0, | 248 | DECLARE_DYNDBG_CLASSMAP(drm_debug_classes, DD_CLASS_TYPE_DISJOINT_BITS, 0, |
| 243 | "DRM_UT_CORE", | 249 | "DRM_UT_CORE", |
| @@ -1033,11 +1039,13 @@ module_param_named(user_partt_mode, amdgpu_user_partt_mode, uint, 0444); · 789 unmodified lines | |||
| 1033 | 1039 | ||
| 1034 | 1040 | ||
| 1035 | /** | 1041 | /** |
| 1036 | * DOC: enforce_isolation (bool) | 1042 | * DOC: enforce_isolation (int) |
| 1037 | * enforce process isolation between graphics and compute via using the same reserved vmid. | 1043 | * enforce process isolation between graphics and compute. |
| 1044 | * (-1 = auto, 0 = disable, 1 = enable, 2 = enable legacy mode, 3 = enable without cleaner shader) | ||
| 1038 | */ | 1045 | */ |
| 1039 | module_param(enforce_isolation, bool, 0444); | 1046 | module_param_named(enforce_isolation, amdgpu_enforce_isolation, int, 0444); |
| 1040 | MODULE_PARM_DESC(enforce_isolation, "enforce process isolation between graphics and compute . enforce_isolation = on"); | 1047 | MODULE_PARM_DESC(enforce_isolation, |
| 1048 | "enforce process isolation between graphics and compute. (-1 = auto, 0 = disable, 1 = enable, 2 = enable legacy mode, 3 = enable without cleaner shader)"); | ||
| 1041 | 1049 | ||
| 1042 | /** | 1050 | /** |
| 1043 | * DOC: modeset (int) | 1051 | * DOC: modeset (int) |
| @@ -1096,6 +1104,28 @@ MODULE_PARM_DESC(wbrf, · 52 unmodified lines | |||
| 1096 | "Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)"); | 1104 | "Enable Wifi RFI interference mitigation (0 = disabled, 1 = enabled, -1 = auto(default)"); |
| 1097 | module_param_named(wbrf, amdgpu_wbrf, int, 0444); | 1105 | module_param_named(wbrf, amdgpu_wbrf, int, 0444); |
| 1098 | 1106 | ||
| 1107 | /** | ||
| 1108 | * DOC: rebar (int) | ||
| 1109 | * Allow BAR resizing. Disable this to prevent the driver from attempting | ||
| 1110 | * to resize the BAR if the GPU supports it and there is available MMIO space. | ||
| 1111 | * Note that this just prevents the driver from resizing the BAR. The BIOS | ||
| 1112 | * may have already resized the BAR at boot time. | ||
| 1113 | */ | ||
| 1114 | MODULE_PARM_DESC(rebar, "Resizable BAR (-1 = auto (default), 0 = disable, 1 = enable)"); | ||
| 1115 | module_param_named(rebar, amdgpu_rebar, int, 0444); | ||
| 1116 | |||
| 1117 | /** | ||
| 1118 | * DOC: user_queue (int) | ||
| 1119 | * Enable user queues on systems that support user queues. Possible values: | ||
| 1120 | * | ||
| 1121 | * - -1 = auto (ASIC specific default) | ||
| 1122 | * - 0 = user queues disabled | ||
| 1123 | * - 1 = user queues enabled and kernel queues enabled (if supported) | ||
| 1124 | * - 2 = user queues enabled and kernel queues disabled | ||
| 1125 | */ | ||
| 1126 | MODULE_PARM_DESC(user_queue, "Enable user queues (-1 = auto (default), 0 = disable, 1 = enable, 2 = enable UQs and disable KQs)"); | ||
| 1127 | module_param_named(user_queue, amdgpu_user_queue, int, 0444); | ||
| 1128 | |||
| 1099 | /* These devices are not supported by amdgpu. | 1129 | /* These devices are not supported by amdgpu. |
| 1100 | * They are supported by the mach64, r128, radeon drivers | 1130 | * They are supported by the mach64, r128, radeon drivers |
| 1101 | */ | 1131 | */ |
| @@ -2244,6 +2274,10 @@ static void amdgpu_init_debug_options(struct amdgpu_device *adev) · 1142 unmodified lines | |||
| 2244 | pr_info("debug: use vram for smu pool\n"); | 2274 | pr_info("debug: use vram for smu pool\n"); |
| 2245 | adev->pm.smu_debug_mask |= SMU_DEBUG_POOL_USE_VRAM; | 2275 | adev->pm.smu_debug_mask |= SMU_DEBUG_POOL_USE_VRAM; |
| 2246 | } | 2276 | } |
| 2277 | if (amdgpu_debug_mask & AMDGPU_DEBUG_VM_USERPTR) { | ||
| 2278 | pr_info("debug: VM mode debug for userptr is enabled\n"); | ||
| 2279 | adev->debug_vm_userptr = true; | ||
| 2280 | } | ||
| 2247 | } | 2281 | } |
| 2248 | 2282 | ||
| 2249 | static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags) | 2283 | static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long flags) |
| @@ -2615,13 +2649,8 @@ static int amdgpu_pmops_freeze(struct device *dev) · 365 unmodified lines | |||
| 2615 | static int amdgpu_pmops_thaw(struct device *dev) | 2649 | static int amdgpu_pmops_thaw(struct device *dev) |
| 2616 | { | 2650 | { |
| 2617 | struct drm_device *drm_dev = dev_get_drvdata(dev); | 2651 | struct drm_device *drm_dev = dev_get_drvdata(dev); |
| 2618 | struct amdgpu_device *adev = drm_to_adev(drm_dev); | ||
| 2619 | int r; | ||
| 2620 | 2652 | ||
| 2621 | r = amdgpu_device_resume(drm_dev, true); | 2653 | return amdgpu_device_resume(drm_dev, true); |
| 2622 | adev->in_s4 = false; | ||
| 2623 | |||
| 2624 | return r; | ||
| 2625 | } | 2654 | } |
| 2626 | 2655 | ||
| 2627 | static int amdgpu_pmops_poweroff(struct device *dev) | 2656 | static int amdgpu_pmops_poweroff(struct device *dev) |
| @@ -2634,9 +2663,6 @@ static int amdgpu_pmops_poweroff(struct device *dev) · 6 unmodified lines | |||
| 2634 | static int amdgpu_pmops_restore(struct device *dev) | 2663 | static int amdgpu_pmops_restore(struct device *dev) |
| 2635 | { | 2664 | { |
| 2636 | struct drm_device *drm_dev = dev_get_drvdata(dev); | 2665 | struct drm_device *drm_dev = dev_get_drvdata(dev); |
| 2637 | struct amdgpu_device *adev = drm_to_adev(drm_dev); | ||
| 2638 | |||
| 2639 | adev->in_s4 = false; | ||
| 2640 | 2666 | ||
| 2641 | return amdgpu_device_resume(drm_dev, true); | 2667 | return amdgpu_device_resume(drm_dev, true); |
| 2642 | } | 2668 | } |
| @@ -2708,6 +2734,29 @@ static int amdgpu_runtime_idle_check_display(struct device *dev) · 65 unmodified lines | |||
| 2708 | return 0; | 2734 | return 0; |
| 2709 | } | 2735 | } |
| 2710 | 2736 | ||
| 2737 | static int amdgpu_runtime_idle_check_userq(struct device *dev) | ||
| 2738 | { | ||
| 2739 | struct pci_dev *pdev = to_pci_dev(dev); | ||
| 2740 | struct drm_device *drm_dev = pci_get_drvdata(pdev); | ||
| 2741 | struct amdgpu_device *adev = drm_to_adev(drm_dev); | ||
| 2742 | struct amdgpu_usermode_queue *queue; | ||
| 2743 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 2744 | int queue_id; | ||
| 2745 | int ret = 0; | ||
| 2746 | |||
| 2747 | mutex_lock(&adev->userq_mutex); | ||
| 2748 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 2749 | idr_for_each_entry(&uqm->userq_idr, queue, queue_id) { | ||
| 2750 | ret = -EBUSY; | ||
| 2751 | goto done; | ||
| 2752 | } | ||
| 2753 | } | ||
| 2754 | done: | ||
| 2755 | mutex_unlock(&adev->userq_mutex); | ||
| 2756 | |||
| 2757 | return ret; | ||
| 2758 | } | ||
| 2759 | |||
| 2711 | static int amdgpu_pmops_runtime_suspend(struct device *dev) | 2760 | static int amdgpu_pmops_runtime_suspend(struct device *dev) |
| 2712 | { | 2761 | { |
| 2713 | struct pci_dev *pdev = to_pci_dev(dev); | 2762 | struct pci_dev *pdev = to_pci_dev(dev); |
| @@ -2721,6 +2770,9 @@ static int amdgpu_pmops_runtime_suspend(struct device *dev) · 7 unmodified lines | |||
| 2721 | } | 2770 | } |
| 2722 | 2771 | ||
| 2723 | ret = amdgpu_runtime_idle_check_display(dev); | 2772 | ret = amdgpu_runtime_idle_check_display(dev); |
| 2773 | if (ret) | ||
| 2774 | return ret; | ||
| 2775 | ret = amdgpu_runtime_idle_check_userq(dev); | ||
| 2724 | if (ret) | 2776 | if (ret) |
| 2725 | return ret; | 2777 | return ret; |
| 2726 | 2778 | ||
| @@ -2844,12 +2896,30 @@ static int amdgpu_pmops_runtime_idle(struct device *dev) · 117 unmodified lines | |||
| 2844 | } | 2896 | } |
| 2845 | 2897 | ||
| 2846 | ret = amdgpu_runtime_idle_check_display(dev); | 2898 | ret = amdgpu_runtime_idle_check_display(dev); |
| 2899 | if (ret) | ||
| 2900 | goto done; | ||
| 2847 | 2901 | ||
| 2902 | ret = amdgpu_runtime_idle_check_userq(dev); | ||
| 2903 | done: | ||
| 2848 | pm_runtime_mark_last_busy(dev); | 2904 | pm_runtime_mark_last_busy(dev); |
| 2849 | pm_runtime_autosuspend(dev); | 2905 | pm_runtime_autosuspend(dev); |
| 2850 | return ret; | 2906 | return ret; |
| 2851 | } | 2907 | } |
| 2852 | 2908 | ||
| 2909 | static int amdgpu_drm_release(struct inode *inode, struct file *filp) | ||
| 2910 | { | ||
| 2911 | struct drm_file *file_priv = filp->private_data; | ||
| 2912 | struct amdgpu_fpriv *fpriv = file_priv->driver_priv; | ||
| 2913 | |||
| 2914 | if (fpriv) { | ||
| 2915 | fpriv->evf_mgr.fd_closing = true; | ||
| 2916 | amdgpu_userq_mgr_fini(&fpriv->userq_mgr); | ||
| 2917 | amdgpu_eviction_fence_destroy(&fpriv->evf_mgr); | ||
| 2918 | } | ||
| 2919 | |||
| 2920 | return drm_release(inode, filp); | ||
| 2921 | } | ||
| 2922 | |||
| 2853 | long amdgpu_drm_ioctl(struct file *filp, | 2923 | long amdgpu_drm_ioctl(struct file *filp, |
| 2854 | unsigned int cmd, unsigned long arg) | 2924 | unsigned int cmd, unsigned long arg) |
| 2855 | { | 2925 | { |
| @@ -2901,7 +2971,7 @@ static const struct file_operations amdgpu_driver_kms_fops = { · 45 unmodified lines | |||
| 2901 | .owner = THIS_MODULE, | 2971 | .owner = THIS_MODULE, |
| 2902 | .open = drm_open, | 2972 | .open = drm_open, |
| 2903 | .flush = amdgpu_flush, | 2973 | .flush = amdgpu_flush, |
| 2904 | .release = drm_release, | 2974 | .release = amdgpu_drm_release, |
| 2905 | .unlocked_ioctl = amdgpu_drm_ioctl, | 2975 | .unlocked_ioctl = amdgpu_drm_ioctl, |
| 2906 | .mmap = drm_gem_mmap, | 2976 | .mmap = drm_gem_mmap, |
| 2907 | .poll = drm_poll, | 2977 | .poll = drm_poll, |
| @@ -2948,6 +3018,9 @@ const struct drm_ioctl_desc amdgpu_ioctls_kms[] = { · 40 unmodified lines | |||
| 2948 | DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), | 3018 | DRM_IOCTL_DEF_DRV(AMDGPU_GEM_VA, amdgpu_gem_va_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), |
| 2949 | DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), | 3019 | DRM_IOCTL_DEF_DRV(AMDGPU_GEM_OP, amdgpu_gem_op_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), |
| 2950 | DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), | 3020 | DRM_IOCTL_DEF_DRV(AMDGPU_GEM_USERPTR, amdgpu_gem_userptr_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), |
| 3021 | DRM_IOCTL_DEF_DRV(AMDGPU_USERQ, amdgpu_userq_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), | ||
| 3022 | DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_SIGNAL, amdgpu_userq_signal_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), | ||
| 3023 | DRM_IOCTL_DEF_DRV(AMDGPU_USERQ_WAIT, amdgpu_userq_wait_ioctl, DRM_AUTH|DRM_RENDER_ALLOW), | ||
| 2951 | }; | 3024 | }; |
| 2952 | 3025 | ||
| 2953 | static const struct drm_driver amdgpu_kms_driver = { | 3026 | static const struct drm_driver amdgpu_kms_driver = { |
| @@ -3038,6 +3111,10 @@ static int __init amdgpu_init(void) · 84 unmodified lines | |||
| 3038 | if (r) | 3111 | if (r) |
| 3039 | goto error_fence; | 3112 | goto error_fence; |
| 3040 | 3113 | ||
| 3114 | r = amdgpu_userq_fence_slab_init(); | ||
| 3115 | if (r) | ||
| 3116 | goto error_fence; | ||
| 3117 | |||
| 3041 | DRM_INFO("amdgpu kernel modesetting enabled.\n"); | 3118 | DRM_INFO("amdgpu kernel modesetting enabled.\n"); |
| 3042 | amdgpu_register_atpx_handler(); | 3119 | amdgpu_register_atpx_handler(); |
| 3043 | amdgpu_acpi_detect(); | 3120 | amdgpu_acpi_detect(); |
| @@ -3069,6 +3146,7 @@ static void __exit amdgpu_exit(void) · 25 unmodified lines | |||
| 3069 | amdgpu_acpi_release(); | 3146 | amdgpu_acpi_release(); |
| 3070 | amdgpu_sync_fini(); | 3147 | amdgpu_sync_fini(); |
| 3071 | amdgpu_fence_slab_fini(); | 3148 | amdgpu_fence_slab_fini(); |
| 3149 | amdgpu_userq_fence_slab_fini(); | ||
| 3072 | mmu_notifier_synchronize(); | 3150 | mmu_notifier_synchronize(); |
| 3073 | amdgpu_xcp_drv_release(); | 3151 | amdgpu_xcp_drv_release(); |
| 3074 | } | 3152 | } |
| @@ -0,0 +1,233 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 2 | /* | ||
| 3 | * Copyright 2024 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | #include <linux/sched.h> | ||
| 25 | #include <drm/drm_exec.h> | ||
| 26 | #include "amdgpu.h" | ||
| 27 | |||
| 28 | #define work_to_evf_mgr(w, name) container_of(w, struct amdgpu_eviction_fence_mgr, name) | ||
| 29 | #define evf_mgr_to_fpriv(e) container_of(e, struct amdgpu_fpriv, evf_mgr) | ||
| 30 | |||
| 31 | static const char * | ||
| 32 | amdgpu_eviction_fence_get_driver_name(struct dma_fence *fence) | ||
| 33 | { | ||
| 34 | return "amdgpu_eviction_fence"; | ||
| 35 | } | ||
| 36 | |||
| 37 | static const char * | ||
| 38 | amdgpu_eviction_fence_get_timeline_name(struct dma_fence *f) | ||
| 39 | { | ||
| 40 | struct amdgpu_eviction_fence *ef; | ||
| 41 | |||
| 42 | ef = container_of(f, struct amdgpu_eviction_fence, base); | ||
| 43 | return ef->timeline_name; | ||
| 44 | } | ||
| 45 | |||
| 46 | int | ||
| 47 | amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 48 | struct drm_exec *exec) | ||
| 49 | { | ||
| 50 | struct amdgpu_eviction_fence *old_ef, *new_ef; | ||
| 51 | struct drm_gem_object *obj; | ||
| 52 | unsigned long index; | ||
| 53 | int ret; | ||
| 54 | |||
| 55 | if (evf_mgr->ev_fence && | ||
| 56 | !dma_fence_is_signaled(&evf_mgr->ev_fence->base)) | ||
| 57 | return 0; | ||
| 58 | /* | ||
| 59 | * Steps to replace eviction fence: | ||
| 60 | * * lock all objects in exec (caller) | ||
| 61 | * * create a new eviction fence | ||
| 62 | * * update new eviction fence in evf_mgr | ||
| 63 | * * attach the new eviction fence to BOs | ||
| 64 | * * release the old fence | ||
| 65 | * * unlock the objects (caller) | ||
| 66 | */ | ||
| 67 | new_ef = amdgpu_eviction_fence_create(evf_mgr); | ||
| 68 | if (!new_ef) { | ||
| 69 | DRM_ERROR("Failed to create new eviction fence\n"); | ||
| 70 | return -ENOMEM; | ||
| 71 | } | ||
| 72 | |||
| 73 | /* Update the eviction fence now */ | ||
| 74 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 75 | old_ef = evf_mgr->ev_fence; | ||
| 76 | evf_mgr->ev_fence = new_ef; | ||
| 77 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 78 | |||
| 79 | /* Attach the new fence */ | ||
| 80 | drm_exec_for_each_locked_object(exec, index, obj) { | ||
| 81 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | ||
| 82 | |||
| 83 | if (!bo) | ||
| 84 | continue; | ||
| 85 | ret = amdgpu_eviction_fence_attach(evf_mgr, bo); | ||
| 86 | if (ret) { | ||
| 87 | DRM_ERROR("Failed to attch new eviction fence\n"); | ||
| 88 | goto free_err; | ||
| 89 | } | ||
| 90 | } | ||
| 91 | |||
| 92 | /* Free old fence */ | ||
| 93 | if (old_ef) | ||
| 94 | dma_fence_put(&old_ef->base); | ||
| 95 | return 0; | ||
| 96 | |||
| 97 | free_err: | ||
| 98 | kfree(new_ef); | ||
| 99 | return ret; | ||
| 100 | } | ||
| 101 | |||
| 102 | static void | ||
| 103 | amdgpu_eviction_fence_suspend_worker(struct work_struct *work) | ||
| 104 | { | ||
| 105 | struct amdgpu_eviction_fence_mgr *evf_mgr = work_to_evf_mgr(work, suspend_work.work); | ||
| 106 | struct amdgpu_fpriv *fpriv = evf_mgr_to_fpriv(evf_mgr); | ||
| 107 | struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr; | ||
| 108 | struct amdgpu_eviction_fence *ev_fence; | ||
| 109 | |||
| 110 | mutex_lock(&uq_mgr->userq_mutex); | ||
| 111 | ev_fence = evf_mgr->ev_fence; | ||
| 112 | if (!ev_fence) | ||
| 113 | goto unlock; | ||
| 114 | |||
| 115 | amdgpu_userq_evict(uq_mgr, ev_fence); | ||
| 116 | |||
| 117 | unlock: | ||
| 118 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 119 | } | ||
| 120 | |||
| 121 | static bool amdgpu_eviction_fence_enable_signaling(struct dma_fence *f) | ||
| 122 | { | ||
| 123 | struct amdgpu_eviction_fence_mgr *evf_mgr; | ||
| 124 | struct amdgpu_eviction_fence *ev_fence; | ||
| 125 | |||
| 126 | if (!f) | ||
| 127 | return true; | ||
| 128 | |||
| 129 | ev_fence = to_ev_fence(f); | ||
| 130 | evf_mgr = ev_fence->evf_mgr; | ||
| 131 | |||
| 132 | schedule_delayed_work(&evf_mgr->suspend_work, 0); | ||
| 133 | return true; | ||
| 134 | } | ||
| 135 | |||
| 136 | static const struct dma_fence_ops amdgpu_eviction_fence_ops = { | ||
| 137 | .use_64bit_seqno = true, | ||
| 138 | .get_driver_name = amdgpu_eviction_fence_get_driver_name, | ||
| 139 | .get_timeline_name = amdgpu_eviction_fence_get_timeline_name, | ||
| 140 | .enable_signaling = amdgpu_eviction_fence_enable_signaling, | ||
| 141 | }; | ||
| 142 | |||
| 143 | void amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 144 | struct amdgpu_eviction_fence *ev_fence) | ||
| 145 | { | ||
| 146 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 147 | dma_fence_signal(&ev_fence->base); | ||
| 148 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 149 | } | ||
| 150 | |||
| 151 | struct amdgpu_eviction_fence * | ||
| 152 | amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 153 | { | ||
| 154 | struct amdgpu_eviction_fence *ev_fence; | ||
| 155 | |||
| 156 | ev_fence = kzalloc(sizeof(*ev_fence), GFP_KERNEL); | ||
| 157 | if (!ev_fence) | ||
| 158 | return NULL; | ||
| 159 | |||
| 160 | ev_fence->evf_mgr = evf_mgr; | ||
| 161 | get_task_comm(ev_fence->timeline_name, current); | ||
| 162 | spin_lock_init(&ev_fence->lock); | ||
| 163 | dma_fence_init(&ev_fence->base, &amdgpu_eviction_fence_ops, | ||
| 164 | &ev_fence->lock, evf_mgr->ev_fence_ctx, | ||
| 165 | atomic_inc_return(&evf_mgr->ev_fence_seq)); | ||
| 166 | return ev_fence; | ||
| 167 | } | ||
| 168 | |||
| 169 | void amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 170 | { | ||
| 171 | struct amdgpu_eviction_fence *ev_fence; | ||
| 172 | |||
| 173 | /* Wait for any pending work to execute */ | ||
| 174 | flush_delayed_work(&evf_mgr->suspend_work); | ||
| 175 | |||
| 176 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 177 | ev_fence = evf_mgr->ev_fence; | ||
| 178 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 179 | |||
| 180 | if (!ev_fence) | ||
| 181 | return; | ||
| 182 | |||
| 183 | dma_fence_wait(&ev_fence->base, false); | ||
| 184 | |||
| 185 | /* Last unref of ev_fence */ | ||
| 186 | dma_fence_put(&evf_mgr->ev_fence->base); | ||
| 187 | } | ||
| 188 | |||
| 189 | int amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 190 | struct amdgpu_bo *bo) | ||
| 191 | { | ||
| 192 | struct amdgpu_eviction_fence *ev_fence; | ||
| 193 | struct dma_resv *resv = bo->tbo.base.resv; | ||
| 194 | int ret; | ||
| 195 | |||
| 196 | if (!resv) | ||
| 197 | return 0; | ||
| 198 | |||
| 199 | ret = dma_resv_reserve_fences(resv, 1); | ||
| 200 | if (ret) { | ||
| 201 | DRM_DEBUG_DRIVER("Failed to resv fence space\n"); | ||
| 202 | return ret; | ||
| 203 | } | ||
| 204 | |||
| 205 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 206 | ev_fence = evf_mgr->ev_fence; | ||
| 207 | if (ev_fence) | ||
| 208 | dma_resv_add_fence(resv, &ev_fence->base, DMA_RESV_USAGE_BOOKKEEP); | ||
| 209 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 210 | |||
| 211 | return 0; | ||
| 212 | } | ||
| 213 | |||
| 214 | void amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 215 | struct amdgpu_bo *bo) | ||
| 216 | { | ||
| 217 | struct dma_fence *stub = dma_fence_get_stub(); | ||
| 218 | |||
| 219 | dma_resv_replace_fences(bo->tbo.base.resv, evf_mgr->ev_fence_ctx, | ||
| 220 | stub, DMA_RESV_USAGE_BOOKKEEP); | ||
| 221 | dma_fence_put(stub); | ||
| 222 | } | ||
| 223 | |||
| 224 | int amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 225 | { | ||
| 226 | /* This needs to be done one time per open */ | ||
| 227 | atomic_set(&evf_mgr->ev_fence_seq, 0); | ||
| 228 | evf_mgr->ev_fence_ctx = dma_fence_context_alloc(1); | ||
| 229 | spin_lock_init(&evf_mgr->ev_fence_lock); | ||
| 230 | |||
| 231 | INIT_DELAYED_WORK(&evf_mgr->suspend_work, amdgpu_eviction_fence_suspend_worker); | ||
| 232 | return 0; | ||
| 233 | } | ||
| @@ -0,0 +1,69 @@ | |||
| 1 | /* SPDX-License-Identifier: MIT */ | ||
| 2 | /* | ||
| 3 | * Copyright 2023 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | |||
| 25 | #ifndef AMDGPU_EV_FENCE_H_ | ||
| 26 | #define AMDGPU_EV_FENCE_H_ | ||
| 27 | |||
| 28 | struct amdgpu_eviction_fence { | ||
| 29 | struct dma_fence base; | ||
| 30 | spinlock_t lock; | ||
| 31 | char timeline_name[TASK_COMM_LEN]; | ||
| 32 | struct amdgpu_eviction_fence_mgr *evf_mgr; | ||
| 33 | }; | ||
| 34 | |||
| 35 | struct amdgpu_eviction_fence_mgr { | ||
| 36 | u64 ev_fence_ctx; | ||
| 37 | atomic_t ev_fence_seq; | ||
| 38 | spinlock_t ev_fence_lock; | ||
| 39 | struct amdgpu_eviction_fence *ev_fence; | ||
| 40 | struct delayed_work suspend_work; | ||
| 41 | uint8_t fd_closing; | ||
| 42 | }; | ||
| 43 | |||
| 44 | /* Eviction fence helper functions */ | ||
| 45 | struct amdgpu_eviction_fence * | ||
| 46 | amdgpu_eviction_fence_create(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 47 | |||
| 48 | void | ||
| 49 | amdgpu_eviction_fence_destroy(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 50 | |||
| 51 | int | ||
| 52 | amdgpu_eviction_fence_attach(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 53 | struct amdgpu_bo *bo); | ||
| 54 | |||
| 55 | void | ||
| 56 | amdgpu_eviction_fence_detach(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 57 | struct amdgpu_bo *bo); | ||
| 58 | |||
| 59 | int | ||
| 60 | amdgpu_eviction_fence_init(struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 61 | |||
| 62 | void | ||
| 63 | amdgpu_eviction_fence_signal(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 64 | struct amdgpu_eviction_fence *ev_fence); | ||
| 65 | |||
| 66 | int | ||
| 67 | amdgpu_eviction_fence_replace_fence(struct amdgpu_eviction_fence_mgr *evf_mgr, | ||
| 68 | struct drm_exec *exec); | ||
| 69 | #endif | ||
| @@ -36,6 +36,7 @@ · 35 unmodified lines | |||
| 36 | #include <drm/drm_exec.h> | 36 | #include <drm/drm_exec.h> |
| 37 | #include <drm/drm_gem_ttm_helper.h> | 37 | #include <drm/drm_gem_ttm_helper.h> |
| 38 | #include <drm/ttm/ttm_tt.h> | 38 | #include <drm/ttm/ttm_tt.h> |
| 39 | #include <drm/drm_syncobj.h> | ||
| 39 | 40 | ||
| 40 | #include "amdgpu.h" | 41 | #include "amdgpu.h" |
| 41 | #include "amdgpu_display.h" | 42 | #include "amdgpu_display.h" |
| @@ -44,6 +45,114 @@ · 2 unmodified lines | |||
| 44 | #include "amdgpu_xgmi.h" | 45 | #include "amdgpu_xgmi.h" |
| 45 | #include "amdgpu_vm.h" | 46 | #include "amdgpu_vm.h" |
| 46 | 47 | ||
| 48 | static int | ||
| 49 | amdgpu_gem_add_input_fence(struct drm_file *filp, | ||
| 50 | uint64_t syncobj_handles_array, | ||
| 51 | uint32_t num_syncobj_handles) | ||
| 52 | { | ||
| 53 | struct dma_fence *fence; | ||
| 54 | uint32_t *syncobj_handles; | ||
| 55 | int ret, i; | ||
| 56 | |||
| 57 | if (!num_syncobj_handles) | ||
| 58 | return 0; | ||
| 59 | |||
| 60 | syncobj_handles = memdup_user(u64_to_user_ptr(syncobj_handles_array), | ||
| 61 | sizeof(uint32_t) * num_syncobj_handles); | ||
| 62 | if (IS_ERR(syncobj_handles)) | ||
| 63 | return PTR_ERR(syncobj_handles); | ||
| 64 | |||
| 65 | for (i = 0; i < num_syncobj_handles; i++) { | ||
| 66 | |||
| 67 | if (!syncobj_handles[i]) { | ||
| 68 | ret = -EINVAL; | ||
| 69 | goto free_memdup; | ||
| 70 | } | ||
| 71 | |||
| 72 | ret = drm_syncobj_find_fence(filp, syncobj_handles[i], 0, 0, &fence); | ||
| 73 | if (ret) | ||
| 74 | goto free_memdup; | ||
| 75 | |||
| 76 | dma_fence_wait(fence, false); | ||
| 77 | |||
| 78 | /* TODO: optimize async handling */ | ||
| 79 | dma_fence_put(fence); | ||
| 80 | } | ||
| 81 | |||
| 82 | free_memdup: | ||
| 83 | kfree(syncobj_handles); | ||
| 84 | return ret; | ||
| 85 | } | ||
| 86 | |||
| 87 | static int | ||
| 88 | amdgpu_gem_update_timeline_node(struct drm_file *filp, | ||
| 89 | uint32_t syncobj_handle, | ||
| 90 | uint64_t point, | ||
| 91 | struct drm_syncobj **syncobj, | ||
| 92 | struct dma_fence_chain **chain) | ||
| 93 | { | ||
| 94 | if (!syncobj_handle) | ||
| 95 | return 0; | ||
| 96 | |||
| 97 | /* Find the sync object */ | ||
| 98 | *syncobj = drm_syncobj_find(filp, syncobj_handle); | ||
| 99 | if (!*syncobj) | ||
| 100 | return -ENOENT; | ||
| 101 | |||
| 102 | if (!point) | ||
| 103 | return 0; | ||
| 104 | |||
| 105 | /* Allocate the chain node */ | ||
| 106 | *chain = dma_fence_chain_alloc(); | ||
| 107 | if (!*chain) { | ||
| 108 | drm_syncobj_put(*syncobj); | ||
| 109 | return -ENOMEM; | ||
| 110 | } | ||
| 111 | |||
| 112 | return 0; | ||
| 113 | } | ||
| 114 | |||
| 115 | static void | ||
| 116 | amdgpu_gem_update_bo_mapping(struct drm_file *filp, | ||
| 117 | struct amdgpu_bo_va *bo_va, | ||
| 118 | uint32_t operation, | ||
| 119 | uint64_t point, | ||
| 120 | struct dma_fence *fence, | ||
| 121 | struct drm_syncobj *syncobj, | ||
| 122 | struct dma_fence_chain *chain) | ||
| 123 | { | ||
| 124 | struct amdgpu_bo *bo = bo_va ? bo_va->base.bo : NULL; | ||
| 125 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | ||
| 126 | struct amdgpu_vm *vm = &fpriv->vm; | ||
| 127 | struct dma_fence *last_update; | ||
| 128 | |||
| 129 | if (!syncobj) | ||
| 130 | return; | ||
| 131 | |||
| 132 | /* Find the last update fence */ | ||
| 133 | switch (operation) { | ||
| 134 | case AMDGPU_VA_OP_MAP: | ||
| 135 | case AMDGPU_VA_OP_REPLACE: | ||
| 136 | if (bo && (bo->tbo.base.resv == vm->root.bo->tbo.base.resv)) | ||
| 137 | last_update = vm->last_update; | ||
| 138 | else | ||
| 139 | last_update = bo_va->last_pt_update; | ||
| 140 | break; | ||
| 141 | case AMDGPU_VA_OP_UNMAP: | ||
| 142 | case AMDGPU_VA_OP_CLEAR: | ||
| 143 | last_update = fence; | ||
| 144 | break; | ||
| 145 | default: | ||
| 146 | return; | ||
| 147 | } | ||
| 148 | |||
| 149 | /* Add fence to timeline */ | ||
| 150 | if (!point) | ||
| 151 | drm_syncobj_replace_fence(syncobj, last_update); | ||
| 152 | else | ||
| 153 | drm_syncobj_add_point(syncobj, chain, last_update, point); | ||
| 154 | } | ||
| 155 | |||
| 47 | static vm_fault_t amdgpu_gem_fault(struct vm_fault *vmf) | 156 | static vm_fault_t amdgpu_gem_fault(struct vm_fault *vmf) |
| 48 | { | 157 | { |
| 49 | struct ttm_buffer_object *bo = vmf->vma->vm_private_data; | 158 | struct ttm_buffer_object *bo = vmf->vma->vm_private_data; |
| @@ -184,6 +293,15 @@ static int amdgpu_gem_object_open(struct drm_gem_object *obj, · 134 unmodified lines | |||
| 184 | bo_va = amdgpu_vm_bo_add(adev, vm, abo); | 293 | bo_va = amdgpu_vm_bo_add(adev, vm, abo); |
| 185 | else | 294 | else |
| 186 | ++bo_va->ref_count; | 295 | ++bo_va->ref_count; |
| 296 | |||
| 297 | /* attach gfx eviction fence */ | ||
| 298 | r = amdgpu_eviction_fence_attach(&fpriv->evf_mgr, abo); | ||
| 299 | if (r) { | ||
| 300 | DRM_DEBUG_DRIVER("Failed to attach eviction fence to BO\n"); | ||
| 301 | amdgpu_bo_unreserve(abo); | ||
| 302 | return r; | ||
| 303 | } | ||
| 304 | |||
| 187 | amdgpu_bo_unreserve(abo); | 305 | amdgpu_bo_unreserve(abo); |
| 188 | 306 | ||
| 189 | /* Validate and add eviction fence to DMABuf imports with dynamic | 307 | /* Validate and add eviction fence to DMABuf imports with dynamic |
| @@ -247,6 +365,9 @@ static void amdgpu_gem_object_close(struct drm_gem_object *obj, · 57 unmodified lines | |||
| 247 | goto out_unlock; | 365 | goto out_unlock; |
| 248 | } | 366 | } |
| 249 | 367 | ||
| 368 | if (!amdgpu_vm_is_bo_always_valid(vm, bo)) | ||
| 369 | amdgpu_eviction_fence_detach(&fpriv->evf_mgr, bo); | ||
| 370 | |||
| 250 | bo_va = amdgpu_vm_bo_find(vm, bo); | 371 | bo_va = amdgpu_vm_bo_find(vm, bo); |
| 251 | if (!bo_va || --bo_va->ref_count) | 372 | if (!bo_va || --bo_va->ref_count) |
| 252 | goto out_unlock; | 373 | goto out_unlock; |
| @@ -321,10 +442,6 @@ int amdgpu_gem_create_ioctl(struct drm_device *dev, void *data, · 68 unmodified lines | |||
| 321 | uint32_t handle, initial_domain; | 442 | uint32_t handle, initial_domain; |
| 322 | int r; | 443 | int r; |
| 323 | 444 | ||
| 324 | /* reject DOORBELLs until userspace code to use it is available */ | ||
| 325 | if (args->in.domains & AMDGPU_GEM_DOMAIN_DOORBELL) | ||
| 326 | return -EINVAL; | ||
| 327 | |||
| 328 | /* reject invalid gem flags */ | 445 | /* reject invalid gem flags */ |
| 329 | if (flags & ~(AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | | 446 | if (flags & ~(AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED | |
| 330 | AMDGPU_GEM_CREATE_NO_CPU_ACCESS | | 447 | AMDGPU_GEM_CREATE_NO_CPU_ACCESS | |
| @@ -638,18 +755,23 @@ out: · 307 unmodified lines | |||
| 638 | * | 755 | * |
| 639 | * Update the bo_va directly after setting its address. Errors are not | 756 | * Update the bo_va directly after setting its address. Errors are not |
| 640 | * vital here, so they are not reported back to userspace. | 757 | * vital here, so they are not reported back to userspace. |
| 758 | * | ||
| 759 | * Returns resulting fence if freed BO(s) got cleared from the PT. | ||
| 760 | * otherwise stub fence in case of error. | ||
| 641 | */ | 761 | */ |
| 642 | static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev, | 762 | static struct dma_fence * |
| 643 | struct amdgpu_vm *vm, | 763 | amdgpu_gem_va_update_vm(struct amdgpu_device *adev, |
| 644 | struct amdgpu_bo_va *bo_va, | 764 | struct amdgpu_vm *vm, |
| 645 | uint32_t operation) | 765 | struct amdgpu_bo_va *bo_va, |
| 766 | uint32_t operation) | ||
| 646 | { | 767 | { |
| 768 | struct dma_fence *fence = dma_fence_get_stub(); | ||
| 647 | int r; | 769 | int r; |
| 648 | 770 | ||
| 649 | if (!amdgpu_vm_ready(vm)) | 771 | if (!amdgpu_vm_ready(vm)) |
| 650 | return; | 772 | return fence; |
| 651 | 773 | ||
| 652 | r = amdgpu_vm_clear_freed(adev, vm, NULL); | 774 | r = amdgpu_vm_clear_freed(adev, vm, &fence); |
| 653 | if (r) | 775 | if (r) |
| 654 | goto error; | 776 | goto error; |
| 655 | 777 | ||
| @@ -665,6 +787,8 @@ static void amdgpu_gem_va_update_vm(struct amdgpu_device *adev, · 9 unmodified lines | |||
| 665 | error: | 787 | error: |
| 666 | if (r && r != -ERESTARTSYS) | 788 | if (r && r != -ERESTARTSYS) |
| 667 | DRM_ERROR("Couldn't update BO_VA (%d)\n", r); | 789 | DRM_ERROR("Couldn't update BO_VA (%d)\n", r); |
| 790 | |||
| 791 | return fence; | ||
| 668 | } | 792 | } |
| 669 | 793 | ||
| 670 | /** | 794 | /** |
| @@ -713,6 +837,9 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 42 unmodified lines | |||
| 713 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | 837 | struct amdgpu_fpriv *fpriv = filp->driver_priv; |
| 714 | struct amdgpu_bo *abo; | 838 | struct amdgpu_bo *abo; |
| 715 | struct amdgpu_bo_va *bo_va; | 839 | struct amdgpu_bo_va *bo_va; |
| 840 | struct drm_syncobj *timeline_syncobj = NULL; | ||
| 841 | struct dma_fence_chain *timeline_chain = NULL; | ||
| 842 | struct dma_fence *fence; | ||
| 716 | struct drm_exec exec; | 843 | struct drm_exec exec; |
| 717 | uint64_t va_flags; | 844 | uint64_t va_flags; |
| 718 | uint64_t vm_size; | 845 | uint64_t vm_size; |
| @@ -774,6 +901,12 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 55 unmodified lines | |||
| 774 | abo = NULL; | 901 | abo = NULL; |
| 775 | } | 902 | } |
| 776 | 903 | ||
| 904 | r = amdgpu_gem_add_input_fence(filp, | ||
| 905 | args->input_fence_syncobj_handles, | ||
| 906 | args->num_syncobj_handles); | ||
| 907 | if (r) | ||
| 908 | goto error_put_gobj; | ||
| 909 | |||
| 777 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT | | 910 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT | |
| 778 | DRM_EXEC_IGNORE_DUPLICATES, 0); | 911 | DRM_EXEC_IGNORE_DUPLICATES, 0); |
| 779 | drm_exec_until_all_locked(&exec) { | 912 | drm_exec_until_all_locked(&exec) { |
| @@ -802,6 +935,14 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 22 unmodified lines | |||
| 802 | bo_va = NULL; | 935 | bo_va = NULL; |
| 803 | } | 936 | } |
| 804 | 937 | ||
| 938 | r = amdgpu_gem_update_timeline_node(filp, | ||
| 939 | args->vm_timeline_syncobj_out, | ||
| 940 | args->vm_timeline_point, | ||
| 941 | &timeline_syncobj, | ||
| 942 | &timeline_chain); | ||
| 943 | if (r) | ||
| 944 | goto error; | ||
| 945 | |||
| 805 | switch (args->operation) { | 946 | switch (args->operation) { |
| 806 | case AMDGPU_VA_OP_MAP: | 947 | case AMDGPU_VA_OP_MAP: |
| 807 | va_flags = amdgpu_gem_va_map_flags(adev, args->flags); | 948 | va_flags = amdgpu_gem_va_map_flags(adev, args->flags); |
| @@ -827,12 +968,24 @@ int amdgpu_gem_va_ioctl(struct drm_device *dev, void *data, · 19 unmodified lines | |||
| 827 | default: | 968 | default: |
| 828 | break; | 969 | break; |
| 829 | } | 970 | } |
| 830 | if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm) | 971 | if (!r && !(args->flags & AMDGPU_VM_DELAY_UPDATE) && !adev->debug_vm) { |
| 831 | amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va, | 972 | fence = amdgpu_gem_va_update_vm(adev, &fpriv->vm, bo_va, |
| 832 | args->operation); | 973 | args->operation); |
| 974 | |||
| 975 | if (timeline_syncobj) | ||
| 976 | amdgpu_gem_update_bo_mapping(filp, bo_va, | ||
| 977 | args->operation, | ||
| 978 | args->vm_timeline_point, | ||
| 979 | fence, timeline_syncobj, | ||
| 980 | timeline_chain); | ||
| 981 | else | ||
| 982 | dma_fence_put(fence); | ||
| 983 | |||
| 984 | } | ||
| 833 | 985 | ||
| 834 | error: | 986 | error: |
| 835 | drm_exec_fini(&exec); | 987 | drm_exec_fini(&exec); |
| 988 | error_put_gobj: | ||
| 836 | drm_gem_object_put(gobj); | 989 | drm_gem_object_put(gobj); |
| 837 | return r; | 990 | return r; |
| 838 | } | 991 | } |
| @@ -33,6 +33,7 @@ · 32 unmodified lines | |||
| 33 | #include "amdgpu_reset.h" | 33 | #include "amdgpu_reset.h" |
| 34 | #include "amdgpu_xcp.h" | 34 | #include "amdgpu_xcp.h" |
| 35 | #include "amdgpu_xgmi.h" | 35 | #include "amdgpu_xgmi.h" |
| 36 | #include "nvd.h" | ||
| 36 | 37 | ||
| 37 | /* delay 0.1 second to enable gfx off feature */ | 38 | /* delay 0.1 second to enable gfx off feature */ |
| 38 | #define GFX_OFF_DELAY_ENABLE msecs_to_jiffies(100) | 39 | #define GFX_OFF_DELAY_ENABLE msecs_to_jiffies(100) |
| @@ -74,14 +75,15 @@ bool amdgpu_gfx_is_mec_queue_enabled(struct amdgpu_device *adev, · 35 unmodified lines | |||
| 74 | adev->gfx.mec_bitmap[xcc_id].queue_bitmap); | 75 | adev->gfx.mec_bitmap[xcc_id].queue_bitmap); |
| 75 | } | 76 | } |
| 76 | 77 | ||
| 77 | int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev, | 78 | static int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev, |
| 78 | int me, int pipe, int queue) | 79 | int me, int pipe, int queue) |
| 79 | { | 80 | { |
| 81 | int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */ | ||
| 80 | int bit = 0; | 82 | int bit = 0; |
| 81 | 83 | ||
| 82 | bit += me * adev->gfx.me.num_pipe_per_me | 84 | bit += me * adev->gfx.me.num_pipe_per_me |
| 83 | * adev->gfx.me.num_queue_per_pipe; | 85 | * num_queue_per_pipe; |
| 84 | bit += pipe * adev->gfx.me.num_queue_per_pipe; | 86 | bit += pipe * num_queue_per_pipe; |
| 85 | bit += queue; | 87 | bit += queue; |
| 86 | 88 | ||
| 87 | return bit; | 89 | return bit; |
| @@ -238,8 +240,8 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 150 unmodified lines | |||
| 238 | { | 240 | { |
| 239 | int i, queue, pipe; | 241 | int i, queue, pipe; |
| 240 | bool multipipe_policy = amdgpu_gfx_is_graphics_multipipe_capable(adev); | 242 | bool multipipe_policy = amdgpu_gfx_is_graphics_multipipe_capable(adev); |
| 241 | int max_queues_per_me = adev->gfx.me.num_pipe_per_me * | 243 | int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */ |
| 242 | adev->gfx.me.num_queue_per_pipe; | 244 | int max_queues_per_me = adev->gfx.me.num_pipe_per_me * num_queue_per_pipe; |
| 243 | 245 | ||
| 244 | if (multipipe_policy) { | 246 | if (multipipe_policy) { |
| 245 | /* policy: amdgpu owns the first queue per pipe at this stage | 247 | /* policy: amdgpu owns the first queue per pipe at this stage |
| @@ -247,9 +249,9 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 1 unmodified line | |||
| 247 | for (i = 0; i < max_queues_per_me; i++) { | 249 | for (i = 0; i < max_queues_per_me; i++) { |
| 248 | pipe = i % adev->gfx.me.num_pipe_per_me; | 250 | pipe = i % adev->gfx.me.num_pipe_per_me; |
| 249 | queue = (i / adev->gfx.me.num_pipe_per_me) % | 251 | queue = (i / adev->gfx.me.num_pipe_per_me) % |
| 250 | adev->gfx.me.num_queue_per_pipe; | 252 | num_queue_per_pipe; |
| 251 | 253 | ||
| 252 | set_bit(pipe * adev->gfx.me.num_queue_per_pipe + queue, | 254 | set_bit(pipe * num_queue_per_pipe + queue, |
| 253 | adev->gfx.me.queue_bitmap); | 255 | adev->gfx.me.queue_bitmap); |
| 254 | } | 256 | } |
| 255 | } else { | 257 | } else { |
| @@ -258,8 +260,9 @@ void amdgpu_gfx_graphics_queue_acquire(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 258 | } | 260 | } |
| 259 | 261 | ||
| 260 | /* update the number of active graphics rings */ | 262 | /* update the number of active graphics rings */ |
| 261 | adev->gfx.num_gfx_rings = | 263 | if (adev->gfx.num_gfx_rings) |
| 262 | bitmap_weight(adev->gfx.me.queue_bitmap, AMDGPU_MAX_GFX_QUEUES); | 264 | adev->gfx.num_gfx_rings = |
| 265 | bitmap_weight(adev->gfx.me.queue_bitmap, AMDGPU_MAX_GFX_QUEUES); | ||
| 263 | } | 266 | } |
| 264 | 267 | ||
| 265 | static int amdgpu_gfx_kiq_acquire(struct amdgpu_device *adev, | 268 | static int amdgpu_gfx_kiq_acquire(struct amdgpu_device *adev, |
| @@ -1351,6 +1354,10 @@ static ssize_t amdgpu_gfx_get_current_compute_partition(struct device *dev, · 1085 unmodified lines | |||
| 1351 | struct amdgpu_device *adev = drm_to_adev(ddev); | 1354 | struct amdgpu_device *adev = drm_to_adev(ddev); |
| 1352 | int mode; | 1355 | int mode; |
| 1353 | 1356 | ||
| 1357 | /* Only minimal precaution taken to reject requests while in reset.*/ | ||
| 1358 | if (amdgpu_in_reset(adev)) | ||
| 1359 | return -EPERM; | ||
| 1360 | |||
| 1354 | mode = amdgpu_xcp_query_partition_mode(adev->xcp_mgr, | 1361 | mode = amdgpu_xcp_query_partition_mode(adev->xcp_mgr, |
| 1355 | AMDGPU_XCP_FL_NONE); | 1362 | AMDGPU_XCP_FL_NONE); |
| 1356 | 1363 | ||
| @@ -1394,8 +1401,14 @@ static ssize_t amdgpu_gfx_set_compute_partition(struct device *dev, · 37 unmodified lines | |||
| 1394 | return -EINVAL; | 1401 | return -EINVAL; |
| 1395 | } | 1402 | } |
| 1396 | 1403 | ||
| 1404 | /* Don't allow a switch while under reset */ | ||
| 1405 | if (!down_read_trylock(&adev->reset_domain->sem)) | ||
| 1406 | return -EPERM; | ||
| 1407 | |||
| 1397 | ret = amdgpu_xcp_switch_partition_mode(adev->xcp_mgr, mode); | 1408 | ret = amdgpu_xcp_switch_partition_mode(adev->xcp_mgr, mode); |
| 1398 | 1409 | ||
| 1410 | up_read(&adev->reset_domain->sem); | ||
| 1411 | |||
| 1399 | if (ret) | 1412 | if (ret) |
| 1400 | return ret; | 1413 | return ret; |
| 1401 | 1414 | ||
| @@ -1466,6 +1479,8 @@ static int amdgpu_gfx_run_cleaner_shader_job(struct amdgpu_ring *ring) · 64 unmodified lines | |||
| 1466 | goto err; | 1479 | goto err; |
| 1467 | 1480 | ||
| 1468 | job->enforce_isolation = true; | 1481 | job->enforce_isolation = true; |
| 1482 | /* always run the cleaner shader */ | ||
| 1483 | job->run_cleaner_shader = true; | ||
| 1469 | 1484 | ||
| 1470 | ib = &job->ibs[0]; | 1485 | ib = &job->ibs[0]; |
| 1471 | for (i = 0; i <= ring->funcs->align_mask; ++i) | 1486 | for (i = 0; i <= ring->funcs->align_mask; ++i) |
| @@ -1552,6 +1567,9 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 80 unmodified lines | |||
| 1552 | if (adev->in_suspend && !adev->in_runpm) | 1567 | if (adev->in_suspend && !adev->in_runpm) |
| 1553 | return -EPERM; | 1568 | return -EPERM; |
| 1554 | 1569 | ||
| 1570 | if (adev->gfx.disable_kq) | ||
| 1571 | return -EPERM; | ||
| 1572 | |||
| 1555 | ret = kstrtol(buf, 0, &value); | 1573 | ret = kstrtol(buf, 0, &value); |
| 1556 | 1574 | ||
| 1557 | if (ret) | 1575 | if (ret) |
| @@ -1594,7 +1612,8 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 36 unmodified lines | |||
| 1594 | * Provides the sysfs read interface to get the current settings of the 'enforce_isolation' | 1612 | * Provides the sysfs read interface to get the current settings of the 'enforce_isolation' |
| 1595 | * feature for each GPU partition. Reading from the 'enforce_isolation' | 1613 | * feature for each GPU partition. Reading from the 'enforce_isolation' |
| 1596 | * sysfs file returns the isolation settings for all partitions, where '0' | 1614 | * sysfs file returns the isolation settings for all partitions, where '0' |
| 1597 | * indicates disabled and '1' indicates enabled. | 1615 | * indicates disabled, '1' indicates enabled, and '2' indicates enabled in legacy mode, |
| 1616 | * and '3' indicates enabled without cleaner shader. | ||
| 1598 | * | 1617 | * |
| 1599 | * Return: The number of bytes read from the sysfs file. | 1618 | * Return: The number of bytes read from the sysfs file. |
| 1600 | */ | 1619 | */ |
| @@ -1629,9 +1648,12 @@ static ssize_t amdgpu_gfx_get_enforce_isolation(struct device *dev, · 28 unmodified lines | |||
| 1629 | * @count: The size of the input data | 1648 | * @count: The size of the input data |
| 1630 | * | 1649 | * |
| 1631 | * This function allows control over the 'enforce_isolation' feature, which | 1650 | * This function allows control over the 'enforce_isolation' feature, which |
| 1632 | * serializes access to the graphics engine. Writing '1' or '0' to the | 1651 | * serializes access to the graphics engine. Writing '0' to disable, '1' to |
| 1633 | * 'enforce_isolation' sysfs file enables or disables process isolation for | 1652 | * enable isolation with cleaner shader, '2' to enable legacy isolation without |
| 1634 | * each partition. The input should specify the setting for all partitions. | 1653 | * cleaner shader, or '3' to enable process isolation without submitting the |
| 1654 | * cleaner shader to the 'enforce_isolation' sysfs file sets the isolation mode | ||
| 1655 | * for each partition. The input should specify the setting for all | ||
| 1656 | * partitions. | ||
| 1635 | * | 1657 | * |
| 1636 | * Return: The number of bytes written to the sysfs file. | 1658 | * Return: The number of bytes written to the sysfs file. |
| 1637 | */ | 1659 | */ |
| @@ -1668,13 +1690,34 @@ static ssize_t amdgpu_gfx_set_enforce_isolation(struct device *dev, · 30 unmodified lines | |||
| 1668 | return -EINVAL; | 1690 | return -EINVAL; |
| 1669 | 1691 | ||
| 1670 | for (i = 0; i < num_partitions; i++) { | 1692 | for (i = 0; i < num_partitions; i++) { |
| 1671 | if (partition_values[i] != 0 && partition_values[i] != 1) | 1693 | if (partition_values[i] != 0 && |
| 1694 | partition_values[i] != 1 && | ||
| 1695 | partition_values[i] != 2 && | ||
| 1696 | partition_values[i] != 3) | ||
| 1672 | return -EINVAL; | 1697 | return -EINVAL; |
| 1673 | } | 1698 | } |
| 1674 | 1699 | ||
| 1675 | mutex_lock(&adev->enforce_isolation_mutex); | 1700 | mutex_lock(&adev->enforce_isolation_mutex); |
| 1676 | for (i = 0; i < num_partitions; i++) | 1701 | for (i = 0; i < num_partitions; i++) { |
| 1677 | adev->enforce_isolation[i] = partition_values[i]; | 1702 | switch (partition_values[i]) { |
| 1703 | case 0: | ||
| 1704 | default: | ||
| 1705 | adev->enforce_isolation[i] = AMDGPU_ENFORCE_ISOLATION_DISABLE; | ||
| 1706 | break; | ||
| 1707 | case 1: | ||
| 1708 | adev->enforce_isolation[i] = | ||
| 1709 | AMDGPU_ENFORCE_ISOLATION_ENABLE; | ||
| 1710 | break; | ||
| 1711 | case 2: | ||
| 1712 | adev->enforce_isolation[i] = | ||
| 1713 | AMDGPU_ENFORCE_ISOLATION_ENABLE_LEGACY; | ||
| 1714 | break; | ||
| 1715 | case 3: | ||
| 1716 | adev->enforce_isolation[i] = | ||
| 1717 | AMDGPU_ENFORCE_ISOLATION_NO_CLEANER_SHADER; | ||
| 1718 | break; | ||
| 1719 | } | ||
| 1720 | } | ||
| 1678 | mutex_unlock(&adev->enforce_isolation_mutex); | 1721 | mutex_unlock(&adev->enforce_isolation_mutex); |
| 1679 | 1722 | ||
| 1680 | amdgpu_mes_update_enforce_isolation(adev); | 1723 | amdgpu_mes_update_enforce_isolation(adev); |
| @@ -1923,39 +1966,41 @@ void amdgpu_gfx_cleaner_shader_init(struct amdgpu_device *adev, · 242 unmodified lines | |||
| 1923 | static void amdgpu_gfx_kfd_sch_ctrl(struct amdgpu_device *adev, u32 idx, | 1966 | static void amdgpu_gfx_kfd_sch_ctrl(struct amdgpu_device *adev, u32 idx, |
| 1924 | bool enable) | 1967 | bool enable) |
| 1925 | { | 1968 | { |
| 1926 | mutex_lock(&adev->gfx.kfd_sch_mutex); | 1969 | mutex_lock(&adev->gfx.userq_sch_mutex); |
| 1927 | 1970 | ||
| 1928 | if (enable) { | 1971 | if (enable) { |
| 1929 | /* If the count is already 0, it means there's an imbalance bug somewhere. | 1972 | /* If the count is already 0, it means there's an imbalance bug somewhere. |
| 1930 | * Note that the bug may be in a different caller than the one which triggers the | 1973 | * Note that the bug may be in a different caller than the one which triggers the |
| 1931 | * WARN_ON_ONCE. | 1974 | * WARN_ON_ONCE. |
| 1932 | */ | 1975 | */ |
| 1933 | if (WARN_ON_ONCE(adev->gfx.kfd_sch_req_count[idx] == 0)) { | 1976 | if (WARN_ON_ONCE(adev->gfx.userq_sch_req_count[idx] == 0)) { |
| 1934 | dev_err(adev->dev, "Attempted to enable KFD scheduler when reference count is already zero\n"); | 1977 | dev_err(adev->dev, "Attempted to enable KFD scheduler when reference count is already zero\n"); |
| 1935 | goto unlock; | 1978 | goto unlock; |
| 1936 | } | 1979 | } |
| 1937 | 1980 | ||
| 1938 | adev->gfx.kfd_sch_req_count[idx]--; | 1981 | adev->gfx.userq_sch_req_count[idx]--; |
| 1939 | 1982 | ||
| 1940 | if (adev->gfx.kfd_sch_req_count[idx] == 0 && | 1983 | if (adev->gfx.userq_sch_req_count[idx] == 0 && |
| 1941 | adev->gfx.kfd_sch_inactive[idx]) { | 1984 | adev->gfx.userq_sch_inactive[idx]) { |
| 1942 | schedule_delayed_work(&adev->gfx.enforce_isolation[idx].work, | 1985 | schedule_delayed_work(&adev->gfx.enforce_isolation[idx].work, |
| 1943 | msecs_to_jiffies(adev->gfx.enforce_isolation_time[idx])); | 1986 | msecs_to_jiffies(adev->gfx.enforce_isolation_time[idx])); |
| 1944 | } | 1987 | } |
| 1945 | } else { | 1988 | } else { |
| 1946 | if (adev->gfx.kfd_sch_req_count[idx] == 0) { | 1989 | if (adev->gfx.userq_sch_req_count[idx] == 0) { |
| 1947 | cancel_delayed_work_sync(&adev->gfx.enforce_isolation[idx].work); | 1990 | cancel_delayed_work_sync(&adev->gfx.enforce_isolation[idx].work); |
| 1948 | if (!adev->gfx.kfd_sch_inactive[idx]) { | 1991 | if (!adev->gfx.userq_sch_inactive[idx]) { |
| 1949 | amdgpu_amdkfd_stop_sched(adev, idx); | 1992 | amdgpu_userq_stop_sched_for_enforce_isolation(adev, idx); |
| 1950 | adev->gfx.kfd_sch_inactive[idx] = true; | 1993 | if (adev->kfd.init_complete) |
| 1994 | amdgpu_amdkfd_stop_sched(adev, idx); | ||
| 1995 | adev->gfx.userq_sch_inactive[idx] = true; | ||
| 1951 | } | 1996 | } |
| 1952 | } | 1997 | } |
| 1953 | 1998 | ||
| 1954 | adev->gfx.kfd_sch_req_count[idx]++; | 1999 | adev->gfx.userq_sch_req_count[idx]++; |
| 1955 | } | 2000 | } |
| 1956 | 2001 | ||
| 1957 | unlock: | 2002 | unlock: |
| 1958 | mutex_unlock(&adev->gfx.kfd_sch_mutex); | 2003 | mutex_unlock(&adev->gfx.userq_sch_mutex); |
| 1959 | } | 2004 | } |
| 1960 | 2005 | ||
| 1961 | /** | 2006 | /** |
| @@ -2000,12 +2045,13 @@ void amdgpu_gfx_enforce_isolation_handler(struct work_struct *work) · 38 unmodified lines | |||
| 2000 | msecs_to_jiffies(1)); | 2045 | msecs_to_jiffies(1)); |
| 2001 | } else { | 2046 | } else { |
| 2002 | /* Tell KFD to resume the runqueue */ | 2047 | /* Tell KFD to resume the runqueue */ |
| 2003 | if (adev->kfd.init_complete) { | 2048 | WARN_ON_ONCE(!adev->gfx.userq_sch_inactive[idx]); |
| 2004 | WARN_ON_ONCE(!adev->gfx.kfd_sch_inactive[idx]); | 2049 | WARN_ON_ONCE(adev->gfx.userq_sch_req_count[idx]); |
| 2005 | WARN_ON_ONCE(adev->gfx.kfd_sch_req_count[idx]); | 2050 | |
| 2051 | amdgpu_userq_start_sched_for_enforce_isolation(adev, idx); | ||
| 2052 | if (adev->kfd.init_complete) | ||
| 2006 | amdgpu_amdkfd_start_sched(adev, idx); | 2053 | amdgpu_amdkfd_start_sched(adev, idx); |
| 2007 | adev->gfx.kfd_sch_inactive[idx] = false; | 2054 | adev->gfx.userq_sch_inactive[idx] = false; |
| 2008 | } | ||
| 2009 | } | 2055 | } |
| 2010 | mutex_unlock(&adev->enforce_isolation_mutex); | 2056 | mutex_unlock(&adev->enforce_isolation_mutex); |
| 2011 | } | 2057 | } |
| @@ -2029,7 +2075,7 @@ amdgpu_gfx_enforce_isolation_wait_for_kfd(struct amdgpu_device *adev, · 17 unmodified lines | |||
| 2029 | bool wait = false; | 2075 | bool wait = false; |
| 2030 | 2076 | ||
| 2031 | mutex_lock(&adev->enforce_isolation_mutex); | 2077 | mutex_lock(&adev->enforce_isolation_mutex); |
| 2032 | if (adev->enforce_isolation[idx]) { | 2078 | if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) { |
| 2033 | /* set the initial values if nothing is set */ | 2079 | /* set the initial values if nothing is set */ |
| 2034 | if (!adev->gfx.enforce_isolation_jiffies[idx]) { | 2080 | if (!adev->gfx.enforce_isolation_jiffies[idx]) { |
| 2035 | adev->gfx.enforce_isolation_jiffies[idx] = jiffies; | 2081 | adev->gfx.enforce_isolation_jiffies[idx] = jiffies; |
| @@ -2096,7 +2142,7 @@ void amdgpu_gfx_enforce_isolation_ring_begin_use(struct amdgpu_ring *ring) · 60 unmodified lines | |||
| 2096 | amdgpu_gfx_enforce_isolation_wait_for_kfd(adev, idx); | 2142 | amdgpu_gfx_enforce_isolation_wait_for_kfd(adev, idx); |
| 2097 | 2143 | ||
| 2098 | mutex_lock(&adev->enforce_isolation_mutex); | 2144 | mutex_lock(&adev->enforce_isolation_mutex); |
| 2099 | if (adev->enforce_isolation[idx]) { | 2145 | if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) { |
| 2100 | if (adev->kfd.init_complete) | 2146 | if (adev->kfd.init_complete) |
| 2101 | sched_work = true; | 2147 | sched_work = true; |
| 2102 | } | 2148 | } |
| @@ -2133,7 +2179,7 @@ void amdgpu_gfx_enforce_isolation_ring_end_use(struct amdgpu_ring *ring) · 30 unmodified lines | |||
| 2133 | return; | 2179 | return; |
| 2134 | 2180 | ||
| 2135 | mutex_lock(&adev->enforce_isolation_mutex); | 2181 | mutex_lock(&adev->enforce_isolation_mutex); |
| 2136 | if (adev->enforce_isolation[idx]) { | 2182 | if (adev->enforce_isolation[idx] == AMDGPU_ENFORCE_ISOLATION_ENABLE) { |
| 2137 | if (adev->kfd.init_complete) | 2183 | if (adev->kfd.init_complete) |
| 2138 | sched_work = true; | 2184 | sched_work = true; |
| 2139 | } | 2185 | } |
| @@ -2217,6 +2263,74 @@ void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring) · 77 unmodified lines | |||
| 2217 | schedule_delayed_work(&ring->adev->gfx.idle_work, GFX_PROFILE_IDLE_TIMEOUT); | 2263 | schedule_delayed_work(&ring->adev->gfx.idle_work, GFX_PROFILE_IDLE_TIMEOUT); |
| 2218 | } | 2264 | } |
| 2219 | 2265 | ||
| 2266 | /** | ||
| 2267 | * amdgpu_gfx_csb_preamble_start - Set CSB preamble start | ||
| 2268 | * | ||
| 2269 | * @buffer: This is an output variable that gets the PACKET3 preamble setup. | ||
| 2270 | * | ||
| 2271 | * Return: | ||
| 2272 | * return the latest index. | ||
| 2273 | */ | ||
| 2274 | u32 amdgpu_gfx_csb_preamble_start(volatile u32 *buffer) | ||
| 2275 | { | ||
| 2276 | u32 count = 0; | ||
| 2277 | |||
| 2278 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | ||
| 2279 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | ||
| 2280 | |||
| 2281 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | ||
| 2282 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 2283 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 2284 | |||
| 2285 | return count; | ||
| 2286 | } | ||
| 2287 | |||
| 2288 | /** | ||
| 2289 | * amdgpu_gfx_csb_data_parser - Parser CS data | ||
| 2290 | * | ||
| 2291 | * @adev: amdgpu_device pointer used to get the CS data and other gfx info. | ||
| 2292 | * @buffer: This is an output variable that gets the PACKET3 preamble end. | ||
| 2293 | * @count: Index to start set the preemble end. | ||
| 2294 | * | ||
| 2295 | * Return: | ||
| 2296 | * return the latest index. | ||
| 2297 | */ | ||
| 2298 | u32 amdgpu_gfx_csb_data_parser(struct amdgpu_device *adev, volatile u32 *buffer, u32 count) | ||
| 2299 | { | ||
| 2300 | const struct cs_section_def *sect = NULL; | ||
| 2301 | const struct cs_extent_def *ext = NULL; | ||
| 2302 | u32 i; | ||
| 2303 | |||
| 2304 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 2305 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 2306 | if (sect->id == SECT_CONTEXT) { | ||
| 2307 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 2308 | buffer[count++] = cpu_to_le32(ext->reg_index - PACKET3_SET_CONTEXT_REG_START); | ||
| 2309 | |||
| 2310 | for (i = 0; i < ext->reg_count; i++) | ||
| 2311 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 2312 | } | ||
| 2313 | } | ||
| 2314 | } | ||
| 2315 | |||
| 2316 | return count; | ||
| 2317 | } | ||
| 2318 | |||
| 2319 | /** | ||
| 2320 | * amdgpu_gfx_csb_preamble_end - Set CSB preamble end | ||
| 2321 | * | ||
| 2322 | * @buffer: This is an output variable that gets the PACKET3 preamble end. | ||
| 2323 | * @count: Index to start set the preemble end. | ||
| 2324 | */ | ||
| 2325 | void amdgpu_gfx_csb_preamble_end(volatile u32 *buffer, u32 count) | ||
| 2326 | { | ||
| 2327 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | ||
| 2328 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 2329 | |||
| 2330 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | ||
| 2331 | buffer[count++] = cpu_to_le32(0); | ||
| 2332 | } | ||
| 2333 | |||
| 2220 | /* | 2334 | /* |
| 2221 | * debugfs for to enable/disable gfx job submission to specific core. | 2335 | * debugfs for to enable/disable gfx job submission to specific core. |
| 2222 | */ | 2336 | */ |
| @@ -170,10 +170,46 @@ struct amdgpu_kiq { · 169 unmodified lines | |||
| 170 | #define AMDGPU_GFX_MAX_SE 4 | 170 | #define AMDGPU_GFX_MAX_SE 4 |
| 171 | #define AMDGPU_GFX_MAX_SH_PER_SE 2 | 171 | #define AMDGPU_GFX_MAX_SH_PER_SE 2 |
| 172 | 172 | ||
| 173 | /** | ||
| 174 | * amdgpu_rb_config - Configure a single Render Backend (RB) | ||
| 175 | * | ||
| 176 | * Bad RBs are fused off and there is a harvest register the driver reads to | ||
| 177 | * determine which RB(s) are fused off so that the driver can configure the | ||
| 178 | * hardware state so that nothing gets sent to them. There are also user | ||
| 179 | * harvest registers that the driver can program to disable additional RBs, | ||
| 180 | * etc., for testing purposes. | ||
| 181 | */ | ||
| 173 | struct amdgpu_rb_config { | 182 | struct amdgpu_rb_config { |
| 183 | /** | ||
| 184 | * @rb_backend_disable: | ||
| 185 | * | ||
| 186 | * The value captured from register RB_BACKEND_DISABLE indicates if the | ||
| 187 | * RB backend is disabled or not. | ||
| 188 | */ | ||
| 174 | uint32_t rb_backend_disable; | 189 | uint32_t rb_backend_disable; |
| 190 | |||
| 191 | /** | ||
| 192 | * @user_rb_backend_disable: | ||
| 193 | * | ||
| 194 | * The value captured from register USER_RB_BACKEND_DISABLE indicates | ||
| 195 | * if the User RB backend is disabled or not. | ||
| 196 | */ | ||
| 175 | uint32_t user_rb_backend_disable; | 197 | uint32_t user_rb_backend_disable; |
| 198 | |||
| 199 | /** | ||
| 200 | * @raster_config: | ||
| 201 | * | ||
| 202 | * To set up all of the states, it is necessary to have two registers | ||
| 203 | * to keep all of the states. This field holds the first register. | ||
| 204 | */ | ||
| 176 | uint32_t raster_config; | 205 | uint32_t raster_config; |
| 206 | |||
| 207 | /** | ||
| 208 | * @raster_config_1: | ||
| 209 | * | ||
| 210 | * To set up all of the states, it is necessary to have two registers | ||
| 211 | * to keep all of the states. This field holds the second register. | ||
| 212 | */ | ||
| 177 | uint32_t raster_config_1; | 213 | uint32_t raster_config_1; |
| 178 | }; | 214 | }; |
| 179 | 215 | ||
| @@ -221,6 +257,13 @@ struct amdgpu_gfx_config { · 41 unmodified lines | |||
| 221 | uint32_t macrotile_mode_array[16]; | 257 | uint32_t macrotile_mode_array[16]; |
| 222 | 258 | ||
| 223 | struct gb_addr_config gb_addr_config_fields; | 259 | struct gb_addr_config gb_addr_config_fields; |
| 260 | |||
| 261 | /** | ||
| 262 | * @rb_config: | ||
| 263 | * | ||
| 264 | * Matrix that keeps all the Render Backend (color and depth buffer | ||
| 265 | * handling) configuration on the 3D engine. | ||
| 266 | */ | ||
| 224 | struct amdgpu_rb_config rb_config[AMDGPU_GFX_MAX_SE][AMDGPU_GFX_MAX_SH_PER_SE]; | 267 | struct amdgpu_rb_config rb_config[AMDGPU_GFX_MAX_SE][AMDGPU_GFX_MAX_SH_PER_SE]; |
| 225 | 268 | ||
| 226 | /* gfx configure feature */ | 269 | /* gfx configure feature */ |
| @@ -305,7 +348,8 @@ struct amdgpu_gfx_funcs { · 78 unmodified lines | |||
| 305 | void (*init_spm_golden)(struct amdgpu_device *adev); | 348 | void (*init_spm_golden)(struct amdgpu_device *adev); |
| 306 | void (*update_perfmon_mgcg)(struct amdgpu_device *adev, bool enable); | 349 | void (*update_perfmon_mgcg)(struct amdgpu_device *adev, bool enable); |
| 307 | int (*get_gfx_shadow_info)(struct amdgpu_device *adev, | 350 | int (*get_gfx_shadow_info)(struct amdgpu_device *adev, |
| 308 | struct amdgpu_gfx_shadow_info *shadow_info); | 351 | struct amdgpu_gfx_shadow_info *shadow_info, |
| 352 | bool skip_check); | ||
| 309 | enum amdgpu_gfx_partition | 353 | enum amdgpu_gfx_partition |
| 310 | (*query_partition_mode)(struct amdgpu_device *adev); | 354 | (*query_partition_mode)(struct amdgpu_device *adev); |
| 311 | int (*switch_partition_mode)(struct amdgpu_device *adev, | 355 | int (*switch_partition_mode)(struct amdgpu_device *adev, |
| @@ -474,9 +518,9 @@ struct amdgpu_gfx { · 162 unmodified lines | |||
| 474 | bool enable_cleaner_shader; | 518 | bool enable_cleaner_shader; |
| 475 | struct amdgpu_isolation_work enforce_isolation[MAX_XCP]; | 519 | struct amdgpu_isolation_work enforce_isolation[MAX_XCP]; |
| 476 | /* Mutex for synchronizing KFD scheduler operations */ | 520 | /* Mutex for synchronizing KFD scheduler operations */ |
| 477 | struct mutex kfd_sch_mutex; | 521 | struct mutex userq_sch_mutex; |
| 478 | u64 kfd_sch_req_count[MAX_XCP]; | 522 | u64 userq_sch_req_count[MAX_XCP]; |
| 479 | bool kfd_sch_inactive[MAX_XCP]; | 523 | bool userq_sch_inactive[MAX_XCP]; |
| 480 | unsigned long enforce_isolation_jiffies[MAX_XCP]; | 524 | unsigned long enforce_isolation_jiffies[MAX_XCP]; |
| 481 | unsigned long enforce_isolation_time[MAX_XCP]; | 525 | unsigned long enforce_isolation_time[MAX_XCP]; |
| 482 | 526 | ||
| @@ -484,6 +528,9 @@ struct amdgpu_gfx { · 1 unmodified line | |||
| 484 | struct delayed_work idle_work; | 528 | struct delayed_work idle_work; |
| 485 | bool workload_profile_active; | 529 | bool workload_profile_active; |
| 486 | struct mutex workload_profile_mutex; | 530 | struct mutex workload_profile_mutex; |
| 531 | |||
| 532 | bool disable_kq; | ||
| 533 | bool disable_uq; | ||
| 487 | }; | 534 | }; |
| 488 | 535 | ||
| 489 | struct amdgpu_gfx_ras_reg_entry { | 536 | struct amdgpu_gfx_ras_reg_entry { |
| @@ -503,7 +550,7 @@ struct amdgpu_gfx_ras_mem_id_entry { · 13 unmodified lines | |||
| 503 | #define amdgpu_gfx_select_se_sh(adev, se, sh, instance, xcc_id) ((adev)->gfx.funcs->select_se_sh((adev), (se), (sh), (instance), (xcc_id))) | 550 | #define amdgpu_gfx_select_se_sh(adev, se, sh, instance, xcc_id) ((adev)->gfx.funcs->select_se_sh((adev), (se), (sh), (instance), (xcc_id))) |
| 504 | #define amdgpu_gfx_select_me_pipe_q(adev, me, pipe, q, vmid, xcc_id) ((adev)->gfx.funcs->select_me_pipe_q((adev), (me), (pipe), (q), (vmid), (xcc_id))) | 551 | #define amdgpu_gfx_select_me_pipe_q(adev, me, pipe, q, vmid, xcc_id) ((adev)->gfx.funcs->select_me_pipe_q((adev), (me), (pipe), (q), (vmid), (xcc_id))) |
| 505 | #define amdgpu_gfx_init_spm_golden(adev) (adev)->gfx.funcs->init_spm_golden((adev)) | 552 | #define amdgpu_gfx_init_spm_golden(adev) (adev)->gfx.funcs->init_spm_golden((adev)) |
| 506 | #define amdgpu_gfx_get_gfx_shadow_info(adev, si) ((adev)->gfx.funcs->get_gfx_shadow_info((adev), (si))) | 553 | #define amdgpu_gfx_get_gfx_shadow_info(adev, si) ((adev)->gfx.funcs->get_gfx_shadow_info((adev), (si), false)) |
| 507 | 554 | ||
| 508 | /** | 555 | /** |
| 509 | * amdgpu_gfx_create_bitmask - create a bitmask | 556 | * amdgpu_gfx_create_bitmask - create a bitmask |
| @@ -550,8 +597,6 @@ bool amdgpu_gfx_is_high_priority_compute_queue(struct amdgpu_device *adev, · 40 unmodified lines | |||
| 550 | struct amdgpu_ring *ring); | 597 | struct amdgpu_ring *ring); |
| 551 | bool amdgpu_gfx_is_high_priority_graphics_queue(struct amdgpu_device *adev, | 598 | bool amdgpu_gfx_is_high_priority_graphics_queue(struct amdgpu_device *adev, |
| 552 | struct amdgpu_ring *ring); | 599 | struct amdgpu_ring *ring); |
| 553 | int amdgpu_gfx_me_queue_to_bit(struct amdgpu_device *adev, int me, | ||
| 554 | int pipe, int queue); | ||
| 555 | bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me, | 600 | bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me, |
| 556 | int pipe, int queue); | 601 | int pipe, int queue); |
| 557 | void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable); | 602 | void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable); |
| @@ -597,6 +642,9 @@ void amdgpu_gfx_enforce_isolation_ring_end_use(struct amdgpu_ring *ring); · 39 unmodified lines | |||
| 597 | void amdgpu_gfx_profile_idle_work_handler(struct work_struct *work); | 642 | void amdgpu_gfx_profile_idle_work_handler(struct work_struct *work); |
| 598 | void amdgpu_gfx_profile_ring_begin_use(struct amdgpu_ring *ring); | 643 | void amdgpu_gfx_profile_ring_begin_use(struct amdgpu_ring *ring); |
| 599 | void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring); | 644 | void amdgpu_gfx_profile_ring_end_use(struct amdgpu_ring *ring); |
| 645 | u32 amdgpu_gfx_csb_preamble_start(volatile u32 *buffer); | ||
| 646 | u32 amdgpu_gfx_csb_data_parser(struct amdgpu_device *adev, volatile u32 *buffer, u32 count); | ||
| 647 | void amdgpu_gfx_csb_preamble_end(volatile u32 *buffer, u32 count); | ||
| 600 | 648 | ||
| 601 | void amdgpu_debugfs_gfx_sched_mask_init(struct amdgpu_device *adev); | 649 | void amdgpu_debugfs_gfx_sched_mask_init(struct amdgpu_device *adev); |
| 602 | void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev); | 650 | void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev); |
| @@ -1230,6 +1230,10 @@ static ssize_t current_memory_partition_show( · 1229 unmodified lines | |||
| 1230 | struct amdgpu_device *adev = drm_to_adev(ddev); | 1230 | struct amdgpu_device *adev = drm_to_adev(ddev); |
| 1231 | enum amdgpu_memory_partition mode; | 1231 | enum amdgpu_memory_partition mode; |
| 1232 | 1232 | ||
| 1233 | /* Only minimal precaution taken to reject requests while in reset */ | ||
| 1234 | if (amdgpu_in_reset(adev)) | ||
| 1235 | return -EPERM; | ||
| 1236 | |||
| 1233 | mode = adev->gmc.gmc_funcs->query_mem_partition_mode(adev); | 1237 | mode = adev->gmc.gmc_funcs->query_mem_partition_mode(adev); |
| 1234 | if ((mode >= ARRAY_SIZE(nps_desc)) || | 1238 | if ((mode >= ARRAY_SIZE(nps_desc)) || |
| 1235 | (BIT(mode) & AMDGPU_ALL_NPS_MASK) != BIT(mode)) | 1239 | (BIT(mode) & AMDGPU_ALL_NPS_MASK) != BIT(mode)) |
| @@ -22,6 +22,7 @@ · 21 unmodified lines | |||
| 22 | */ | 22 | */ |
| 23 | #include "amdgpu.h" | 23 | #include "amdgpu.h" |
| 24 | #include "amdgpu_ras.h" | 24 | #include "amdgpu_ras.h" |
| 25 | #include <uapi/linux/kfd_ioctl.h> | ||
| 25 | 26 | ||
| 26 | int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev) | 27 | int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev) |
| 27 | { | 28 | { |
| @@ -46,3 +47,22 @@ int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev) · 18 unmodified lines | |||
| 46 | /* hdp ras follows amdgpu_ras_block_late_init_default for late init */ | 47 | /* hdp ras follows amdgpu_ras_block_late_init_default for late init */ |
| 47 | return 0; | 48 | return 0; |
| 48 | } | 49 | } |
| 50 | |||
| 51 | void amdgpu_hdp_generic_flush(struct amdgpu_device *adev, | ||
| 52 | struct amdgpu_ring *ring) | ||
| 53 | { | ||
| 54 | if (!ring || !ring->funcs->emit_wreg) { | ||
| 55 | WREG32((adev->rmmio_remap.reg_offset + | ||
| 56 | KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> | ||
| 57 | 2, | ||
| 58 | 0); | ||
| 59 | if (adev->nbio.funcs->get_memsize) | ||
| 60 | adev->nbio.funcs->get_memsize(adev); | ||
| 61 | } else { | ||
| 62 | amdgpu_ring_emit_wreg(ring, | ||
| 63 | (adev->rmmio_remap.reg_offset + | ||
| 64 | KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> | ||
| 65 | 2, | ||
| 66 | 0); | ||
| 67 | } | ||
| 68 | } | ||
| @@ -44,4 +44,6 @@ struct amdgpu_hdp { · 43 unmodified lines | |||
| 44 | }; | 44 | }; |
| 45 | 45 | ||
| 46 | int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev); | 46 | int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev); |
| 47 | void amdgpu_hdp_generic_flush(struct amdgpu_device *adev, | ||
| 48 | struct amdgpu_ring *ring); | ||
| 47 | #endif /* __AMDGPU_HDP_H__ */ | 49 | #endif /* __AMDGPU_HDP_H__ */ |
| @@ -163,12 +163,12 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 162 unmodified lines | |||
| 163 | init_shadow = false; | 163 | init_shadow = false; |
| 164 | } | 164 | } |
| 165 | 165 | ||
| 166 | if (!ring->sched.ready && !ring->is_mes_queue) { | 166 | if (!ring->sched.ready) { |
| 167 | dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name); | 167 | dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name); |
| 168 | return -EINVAL; | 168 | return -EINVAL; |
| 169 | } | 169 | } |
| 170 | 170 | ||
| 171 | if (vm && !job->vmid && !ring->is_mes_queue) { | 171 | if (vm && !job->vmid) { |
| 172 | dev_err(adev->dev, "VM IB without ID\n"); | 172 | dev_err(adev->dev, "VM IB without ID\n"); |
| 173 | return -EINVAL; | 173 | return -EINVAL; |
| 174 | } | 174 | } |
| @@ -576,8 +576,16 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev) · 575 unmodified lines | |||
| 576 | INIT_LIST_HEAD(&id_mgr->ids_lru); | 576 | INIT_LIST_HEAD(&id_mgr->ids_lru); |
| 577 | id_mgr->reserved_use_count = 0; | 577 | id_mgr->reserved_use_count = 0; |
| 578 | 578 | ||
| 579 | /* manage only VMIDs not used by KFD */ | 579 | /* for GC <10, SDMA uses MMHUB so use first_kfd_vmid for both GC and MM */ |
| 580 | id_mgr->num_ids = adev->vm_manager.first_kfd_vmid; | 580 | if (amdgpu_ip_version(adev, GC_HWIP, 0) < IP_VERSION(10, 0, 0)) |
| 581 | /* manage only VMIDs not used by KFD */ | ||
| 582 | id_mgr->num_ids = adev->vm_manager.first_kfd_vmid; | ||
| 583 | else if (AMDGPU_IS_MMHUB0(i) || | ||
| 584 | AMDGPU_IS_MMHUB1(i)) | ||
| 585 | id_mgr->num_ids = 16; | ||
| 586 | else | ||
| 587 | /* manage only VMIDs not used by KFD */ | ||
| 588 | id_mgr->num_ids = adev->vm_manager.first_kfd_vmid; | ||
| 581 | 589 | ||
| 582 | /* skip over VMID 0, since it is the system VM */ | 590 | /* skip over VMID 0, since it is the system VM */ |
| 583 | for (j = 1; j < id_mgr->num_ids; ++j) { | 591 | for (j = 1; j < id_mgr->num_ids; ++j) { |
| @@ -588,7 +596,7 @@ void amdgpu_vmid_mgr_init(struct amdgpu_device *adev) · 4 unmodified lines | |||
| 588 | } | 596 | } |
| 589 | /* alloc a default reserved vmid to enforce isolation */ | 597 | /* alloc a default reserved vmid to enforce isolation */ |
| 590 | for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) { | 598 | for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) { |
| 591 | if (adev->enforce_isolation[i]) | 599 | if (adev->enforce_isolation[i] != AMDGPU_ENFORCE_ISOLATION_DISABLE) |
| 592 | amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(i)); | 600 | amdgpu_vmid_alloc_reserved(adev, AMDGPU_GFXHUB(i)); |
| 593 | } | 601 | } |
| 594 | } | 602 | } |
| @@ -25,6 +25,7 @@ · 24 unmodified lines | |||
| 25 | 25 | ||
| 26 | #include "amdgpu.h" | 26 | #include "amdgpu.h" |
| 27 | #include "amdgpu_ih.h" | 27 | #include "amdgpu_ih.h" |
| 28 | #include "amdgpu_reset.h" | ||
| 28 | 29 | ||
| 29 | /** | 30 | /** |
| 30 | * amdgpu_ih_ring_init - initialize the IH state | 31 | * amdgpu_ih_ring_init - initialize the IH state |
| @@ -227,13 +228,23 @@ restart_ih: · 196 unmodified lines | |||
| 227 | ih->rptr &= ih->ptr_mask; | 228 | ih->rptr &= ih->ptr_mask; |
| 228 | } | 229 | } |
| 229 | 230 | ||
| 230 | amdgpu_ih_set_rptr(adev, ih); | 231 | if (!ih->overflow) |
| 232 | amdgpu_ih_set_rptr(adev, ih); | ||
| 233 | |||
| 231 | wake_up_all(&ih->wait_process); | 234 | wake_up_all(&ih->wait_process); |
| 232 | 235 | ||
| 233 | /* make sure wptr hasn't changed while processing */ | 236 | /* make sure wptr hasn't changed while processing */ |
| 234 | wptr = amdgpu_ih_get_wptr(adev, ih); | 237 | wptr = amdgpu_ih_get_wptr(adev, ih); |
| 235 | if (wptr != ih->rptr) | 238 | if (wptr != ih->rptr) |
| 236 | goto restart_ih; | 239 | if (!ih->overflow) |
| 240 | goto restart_ih; | ||
| 241 | |||
| 242 | if (ih->overflow) | ||
| 243 | if (amdgpu_sriov_runtime(adev)) | ||
| 244 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, | ||
| 245 | &adev->virt.flr_work), | ||
| 246 | "Failed to queue work! at %s", | ||
| 247 | __func__); | ||
| 237 | 248 | ||
| 238 | return IRQ_HANDLED; | 249 | return IRQ_HANDLED; |
| 239 | } | 250 | } |
| @@ -72,6 +72,7 @@ struct amdgpu_ih_ring { · 71 unmodified lines | |||
| 72 | /* For waiting on IH processing at checkpoint. */ | 72 | /* For waiting on IH processing at checkpoint. */ |
| 73 | wait_queue_head_t wait_process; | 73 | wait_queue_head_t wait_process; |
| 74 | uint64_t processed_timestamp; | 74 | uint64_t processed_timestamp; |
| 75 | bool overflow; | ||
| 75 | }; | 76 | }; |
| 76 | 77 | ||
| 77 | /* return true if time stamp t2 is after t1 with 48bit wrap around */ | 78 | /* return true if time stamp t2 is after t1 with 48bit wrap around */ |
| @@ -78,6 +78,7 @@ struct amdgpu_job { · 77 unmodified lines | |||
| 78 | 78 | ||
| 79 | /* enforce isolation */ | 79 | /* enforce isolation */ |
| 80 | bool enforce_isolation; | 80 | bool enforce_isolation; |
| 81 | bool run_cleaner_shader; | ||
| 81 | 82 | ||
| 82 | uint32_t num_ibs; | 83 | uint32_t num_ibs; |
| 83 | struct amdgpu_ib ibs[]; | 84 | struct amdgpu_ib ibs[]; |
| @@ -45,6 +45,7 @@ · 44 unmodified lines | |||
| 45 | #include "amdgpu_ras.h" | 45 | #include "amdgpu_ras.h" |
| 46 | #include "amdgpu_reset.h" | 46 | #include "amdgpu_reset.h" |
| 47 | #include "amd_pcie.h" | 47 | #include "amd_pcie.h" |
| 48 | #include "amdgpu_userq.h" | ||
| 48 | 49 | ||
| 49 | void amdgpu_unregister_gpu_instance(struct amdgpu_device *adev) | 50 | void amdgpu_unregister_gpu_instance(struct amdgpu_device *adev) |
| 50 | { | 51 | { |
| @@ -370,6 +371,26 @@ static int amdgpu_firmware_info(struct drm_amdgpu_info_firmware *fw_info, · 319 unmodified lines | |||
| 370 | return 0; | 371 | return 0; |
| 371 | } | 372 | } |
| 372 | 373 | ||
| 374 | static int amdgpu_userq_metadata_info_gfx(struct amdgpu_device *adev, | ||
| 375 | struct drm_amdgpu_info *info, | ||
| 376 | struct drm_amdgpu_info_uq_metadata_gfx *meta) | ||
| 377 | { | ||
| 378 | int ret = -EOPNOTSUPP; | ||
| 379 | |||
| 380 | if (adev->gfx.funcs->get_gfx_shadow_info) { | ||
| 381 | struct amdgpu_gfx_shadow_info shadow = {}; | ||
| 382 | |||
| 383 | adev->gfx.funcs->get_gfx_shadow_info(adev, &shadow, true); | ||
| 384 | meta->shadow_size = shadow.shadow_size; | ||
| 385 | meta->shadow_alignment = shadow.shadow_alignment; | ||
| 386 | meta->csa_size = shadow.csa_size; | ||
| 387 | meta->csa_alignment = shadow.csa_alignment; | ||
| 388 | ret = 0; | ||
| 389 | } | ||
| 390 | |||
| 391 | return ret; | ||
| 392 | } | ||
| 393 | |||
| 373 | static int amdgpu_hw_ip_info(struct amdgpu_device *adev, | 394 | static int amdgpu_hw_ip_info(struct amdgpu_device *adev, |
| 374 | struct drm_amdgpu_info *info, | 395 | struct drm_amdgpu_info *info, |
| 375 | struct drm_amdgpu_info_hw_ip *result) | 396 | struct drm_amdgpu_info_hw_ip *result) |
| @@ -387,7 +408,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 11 unmodified lines | |||
| 387 | case AMDGPU_HW_IP_GFX: | 408 | case AMDGPU_HW_IP_GFX: |
| 388 | type = AMD_IP_BLOCK_TYPE_GFX; | 409 | type = AMD_IP_BLOCK_TYPE_GFX; |
| 389 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) | 410 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) |
| 390 | if (adev->gfx.gfx_ring[i].sched.ready) | 411 | if (adev->gfx.gfx_ring[i].sched.ready && |
| 412 | !adev->gfx.gfx_ring[i].no_user_submission) | ||
| 391 | ++num_rings; | 413 | ++num_rings; |
| 392 | ib_start_alignment = 32; | 414 | ib_start_alignment = 32; |
| 393 | ib_size_alignment = 32; | 415 | ib_size_alignment = 32; |
| @@ -395,7 +417,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line | |||
| 395 | case AMDGPU_HW_IP_COMPUTE: | 417 | case AMDGPU_HW_IP_COMPUTE: |
| 396 | type = AMD_IP_BLOCK_TYPE_GFX; | 418 | type = AMD_IP_BLOCK_TYPE_GFX; |
| 397 | for (i = 0; i < adev->gfx.num_compute_rings; i++) | 419 | for (i = 0; i < adev->gfx.num_compute_rings; i++) |
| 398 | if (adev->gfx.compute_ring[i].sched.ready) | 420 | if (adev->gfx.compute_ring[i].sched.ready && |
| 421 | !adev->gfx.compute_ring[i].no_user_submission) | ||
| 399 | ++num_rings; | 422 | ++num_rings; |
| 400 | ib_start_alignment = 32; | 423 | ib_start_alignment = 32; |
| 401 | ib_size_alignment = 32; | 424 | ib_size_alignment = 32; |
| @@ -403,7 +426,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line | |||
| 403 | case AMDGPU_HW_IP_DMA: | 426 | case AMDGPU_HW_IP_DMA: |
| 404 | type = AMD_IP_BLOCK_TYPE_SDMA; | 427 | type = AMD_IP_BLOCK_TYPE_SDMA; |
| 405 | for (i = 0; i < adev->sdma.num_instances; i++) | 428 | for (i = 0; i < adev->sdma.num_instances; i++) |
| 406 | if (adev->sdma.instance[i].ring.sched.ready) | 429 | if (adev->sdma.instance[i].ring.sched.ready && |
| 430 | !adev->sdma.instance[i].ring.no_user_submission) | ||
| 407 | ++num_rings; | 431 | ++num_rings; |
| 408 | ib_start_alignment = 256; | 432 | ib_start_alignment = 256; |
| 409 | ib_size_alignment = 4; | 433 | ib_size_alignment = 4; |
| @@ -414,7 +438,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 4 unmodified lines | |||
| 414 | if (adev->uvd.harvest_config & (1 << i)) | 438 | if (adev->uvd.harvest_config & (1 << i)) |
| 415 | continue; | 439 | continue; |
| 416 | 440 | ||
| 417 | if (adev->uvd.inst[i].ring.sched.ready) | 441 | if (adev->uvd.inst[i].ring.sched.ready && |
| 442 | !adev->uvd.inst[i].ring.no_user_submission) | ||
| 418 | ++num_rings; | 443 | ++num_rings; |
| 419 | } | 444 | } |
| 420 | ib_start_alignment = 256; | 445 | ib_start_alignment = 256; |
| @@ -423,7 +448,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 2 unmodified lines | |||
| 423 | case AMDGPU_HW_IP_VCE: | 448 | case AMDGPU_HW_IP_VCE: |
| 424 | type = AMD_IP_BLOCK_TYPE_VCE; | 449 | type = AMD_IP_BLOCK_TYPE_VCE; |
| 425 | for (i = 0; i < adev->vce.num_rings; i++) | 450 | for (i = 0; i < adev->vce.num_rings; i++) |
| 426 | if (adev->vce.ring[i].sched.ready) | 451 | if (adev->vce.ring[i].sched.ready && |
| 452 | !adev->vce.ring[i].no_user_submission) | ||
| 427 | ++num_rings; | 453 | ++num_rings; |
| 428 | ib_start_alignment = 256; | 454 | ib_start_alignment = 256; |
| 429 | ib_size_alignment = 4; | 455 | ib_size_alignment = 4; |
| @@ -435,7 +461,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 5 unmodified lines | |||
| 435 | continue; | 461 | continue; |
| 436 | 462 | ||
| 437 | for (j = 0; j < adev->uvd.num_enc_rings; j++) | 463 | for (j = 0; j < adev->uvd.num_enc_rings; j++) |
| 438 | if (adev->uvd.inst[i].ring_enc[j].sched.ready) | 464 | if (adev->uvd.inst[i].ring_enc[j].sched.ready && |
| 465 | !adev->uvd.inst[i].ring_enc[j].no_user_submission) | ||
| 439 | ++num_rings; | 466 | ++num_rings; |
| 440 | } | 467 | } |
| 441 | ib_start_alignment = 256; | 468 | ib_start_alignment = 256; |
| @@ -447,7 +474,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 5 unmodified lines | |||
| 447 | if (adev->vcn.harvest_config & (1 << i)) | 474 | if (adev->vcn.harvest_config & (1 << i)) |
| 448 | continue; | 475 | continue; |
| 449 | 476 | ||
| 450 | if (adev->vcn.inst[i].ring_dec.sched.ready) | 477 | if (adev->vcn.inst[i].ring_dec.sched.ready && |
| 478 | !adev->vcn.inst[i].ring_dec.no_user_submission) | ||
| 451 | ++num_rings; | 479 | ++num_rings; |
| 452 | } | 480 | } |
| 453 | ib_start_alignment = 256; | 481 | ib_start_alignment = 256; |
| @@ -460,7 +488,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 6 unmodified lines | |||
| 460 | continue; | 488 | continue; |
| 461 | 489 | ||
| 462 | for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++) | 490 | for (j = 0; j < adev->vcn.inst[i].num_enc_rings; j++) |
| 463 | if (adev->vcn.inst[i].ring_enc[j].sched.ready) | 491 | if (adev->vcn.inst[i].ring_enc[j].sched.ready && |
| 492 | !adev->vcn.inst[i].ring_enc[j].no_user_submission) | ||
| 464 | ++num_rings; | 493 | ++num_rings; |
| 465 | } | 494 | } |
| 466 | ib_start_alignment = 256; | 495 | ib_start_alignment = 256; |
| @@ -475,7 +504,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 8 unmodified lines | |||
| 475 | continue; | 504 | continue; |
| 476 | 505 | ||
| 477 | for (j = 0; j < adev->jpeg.num_jpeg_rings; j++) | 506 | for (j = 0; j < adev->jpeg.num_jpeg_rings; j++) |
| 478 | if (adev->jpeg.inst[i].ring_dec[j].sched.ready) | 507 | if (adev->jpeg.inst[i].ring_dec[j].sched.ready && |
| 508 | !adev->jpeg.inst[i].ring_dec[j].no_user_submission) | ||
| 479 | ++num_rings; | 509 | ++num_rings; |
| 480 | } | 510 | } |
| 481 | ib_start_alignment = 256; | 511 | ib_start_alignment = 256; |
| @@ -483,7 +513,8 @@ static int amdgpu_hw_ip_info(struct amdgpu_device *adev, · 1 unmodified line | |||
| 483 | break; | 513 | break; |
| 484 | case AMDGPU_HW_IP_VPE: | 514 | case AMDGPU_HW_IP_VPE: |
| 485 | type = AMD_IP_BLOCK_TYPE_VPE; | 515 | type = AMD_IP_BLOCK_TYPE_VPE; |
| 486 | if (adev->vpe.ring.sched.ready) | 516 | if (adev->vpe.ring.sched.ready && |
| 517 | !adev->vpe.ring.no_user_submission) | ||
| 487 | ++num_rings; | 518 | ++num_rings; |
| 488 | ib_start_alignment = 256; | 519 | ib_start_alignment = 256; |
| 489 | ib_size_alignment = 4; | 520 | ib_size_alignment = 4; |
| @@ -978,6 +1009,8 @@ out: · 488 unmodified lines | |||
| 978 | } | 1009 | } |
| 979 | } | 1010 | } |
| 980 | 1011 | ||
| 1012 | dev_info->userq_ip_mask = amdgpu_userq_get_supported_ip_mask(adev); | ||
| 1013 | |||
| 981 | ret = copy_to_user(out, dev_info, | 1014 | ret = copy_to_user(out, dev_info, |
| 982 | min((size_t)size, sizeof(*dev_info))) ? -EFAULT : 0; | 1015 | min((size_t)size, sizeof(*dev_info))) ? -EFAULT : 0; |
| 983 | kfree(dev_info); | 1016 | kfree(dev_info); |
| @@ -1293,6 +1326,22 @@ out: · 309 unmodified lines | |||
| 1293 | return copy_to_user(out, &gpuvm_fault, | 1326 | return copy_to_user(out, &gpuvm_fault, |
| 1294 | min((size_t)size, sizeof(gpuvm_fault))) ? -EFAULT : 0; | 1327 | min((size_t)size, sizeof(gpuvm_fault))) ? -EFAULT : 0; |
| 1295 | } | 1328 | } |
| 1329 | case AMDGPU_INFO_UQ_FW_AREAS: { | ||
| 1330 | struct drm_amdgpu_info_uq_metadata meta_info = {}; | ||
| 1331 | |||
| 1332 | switch (info->query_hw_ip.type) { | ||
| 1333 | case AMDGPU_HW_IP_GFX: | ||
| 1334 | ret = amdgpu_userq_metadata_info_gfx(adev, info, &meta_info.gfx); | ||
| 1335 | if (ret) | ||
| 1336 | return ret; | ||
| 1337 | |||
| 1338 | ret = copy_to_user(out, &meta_info, | ||
| 1339 | min((size_t)size, sizeof(meta_info))) ? -EFAULT : 0; | ||
| 1340 | return 0; | ||
| 1341 | default: | ||
| 1342 | return -EINVAL; | ||
| 1343 | } | ||
| 1344 | } | ||
| 1296 | default: | 1345 | default: |
| 1297 | DRM_DEBUG_KMS("Invalid request %d\n", info->query); | 1346 | DRM_DEBUG_KMS("Invalid request %d\n", info->query); |
| 1298 | return -EINVAL; | 1347 | return -EINVAL; |
| @@ -1376,8 +1425,16 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · 77 unmodified lines | |||
| 1376 | mutex_init(&fpriv->bo_list_lock); | 1425 | mutex_init(&fpriv->bo_list_lock); |
| 1377 | idr_init_base(&fpriv->bo_list_handles, 1); | 1426 | idr_init_base(&fpriv->bo_list_handles, 1); |
| 1378 | 1427 | ||
| 1428 | r = amdgpu_eviction_fence_init(&fpriv->evf_mgr); | ||
| 1429 | if (r) | ||
| 1430 | goto error_vm; | ||
| 1431 | |||
| 1379 | amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev); | 1432 | amdgpu_ctx_mgr_init(&fpriv->ctx_mgr, adev); |
| 1380 | 1433 | ||
| 1434 | r = amdgpu_userq_mgr_init(&fpriv->userq_mgr, file_priv, adev); | ||
| 1435 | if (r) | ||
| 1436 | DRM_WARN("Can't setup usermode queues, use legacy workload submission only\n"); | ||
| 1437 | |||
| 1381 | file_priv->driver_priv = fpriv; | 1438 | file_priv->driver_priv = fpriv; |
| 1382 | goto out_suspend; | 1439 | goto out_suspend; |
| 1383 | 1440 | ||
| @@ -1445,6 +1502,10 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 61 unmodified lines | |||
| 1445 | amdgpu_bo_unreserve(pd); | 1502 | amdgpu_bo_unreserve(pd); |
| 1446 | } | 1503 | } |
| 1447 | 1504 | ||
| 1505 | fpriv->evf_mgr.fd_closing = true; | ||
| 1506 | amdgpu_userq_mgr_fini(&fpriv->userq_mgr); | ||
| 1507 | amdgpu_eviction_fence_destroy(&fpriv->evf_mgr); | ||
| 1508 | |||
| 1448 | amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr); | 1509 | amdgpu_ctx_mgr_fini(&fpriv->ctx_mgr); |
| 1449 | amdgpu_vm_fini(adev, &fpriv->vm); | 1510 | amdgpu_vm_fini(adev, &fpriv->vm); |
| 1450 | 1511 | ||
| @@ -39,42 +39,6 @@ int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev) · 38 unmodified lines | |||
| 39 | PAGE_SIZE); | 39 | PAGE_SIZE); |
| 40 | } | 40 | } |
| 41 | 41 | ||
| 42 | static int amdgpu_mes_kernel_doorbell_get(struct amdgpu_device *adev, | ||
| 43 | int ip_type, uint64_t *doorbell_index) | ||
| 44 | { | ||
| 45 | unsigned int offset, found; | ||
| 46 | struct amdgpu_mes *mes = &adev->mes; | ||
| 47 | |||
| 48 | if (ip_type == AMDGPU_RING_TYPE_SDMA) | ||
| 49 | offset = adev->doorbell_index.sdma_engine[0]; | ||
| 50 | else | ||
| 51 | offset = 0; | ||
| 52 | |||
| 53 | found = find_next_zero_bit(mes->doorbell_bitmap, mes->num_mes_dbs, offset); | ||
| 54 | if (found >= mes->num_mes_dbs) { | ||
| 55 | DRM_WARN("No doorbell available\n"); | ||
| 56 | return -ENOSPC; | ||
| 57 | } | ||
| 58 | |||
| 59 | set_bit(found, mes->doorbell_bitmap); | ||
| 60 | |||
| 61 | /* Get the absolute doorbell index on BAR */ | ||
| 62 | *doorbell_index = mes->db_start_dw_offset + found * 2; | ||
| 63 | return 0; | ||
| 64 | } | ||
| 65 | |||
| 66 | static void amdgpu_mes_kernel_doorbell_free(struct amdgpu_device *adev, | ||
| 67 | uint32_t doorbell_index) | ||
| 68 | { | ||
| 69 | unsigned int old, rel_index; | ||
| 70 | struct amdgpu_mes *mes = &adev->mes; | ||
| 71 | |||
| 72 | /* Find the relative index of the doorbell in this object */ | ||
| 73 | rel_index = (doorbell_index - mes->db_start_dw_offset) / 2; | ||
| 74 | old = test_and_clear_bit(rel_index, mes->doorbell_bitmap); | ||
| 75 | WARN_ON(!old); | ||
| 76 | } | ||
| 77 | |||
| 78 | static int amdgpu_mes_doorbell_init(struct amdgpu_device *adev) | 42 | static int amdgpu_mes_doorbell_init(struct amdgpu_device *adev) |
| 79 | { | 43 | { |
| 80 | int i; | 44 | int i; |
| @@ -126,7 +90,7 @@ static void amdgpu_mes_doorbell_free(struct amdgpu_device *adev) · 45 unmodified lines | |||
| 126 | 90 | ||
| 127 | int amdgpu_mes_init(struct amdgpu_device *adev) | 91 | int amdgpu_mes_init(struct amdgpu_device *adev) |
| 128 | { | 92 | { |
| 129 | int i, r; | 93 | int i, r, num_pipes; |
| 130 | 94 | ||
| 131 | adev->mes.adev = adev; | 95 | adev->mes.adev = adev; |
| 132 | 96 | ||
| @@ -142,19 +106,52 @@ int amdgpu_mes_init(struct amdgpu_device *adev) · 9 unmodified lines | |||
| 142 | 106 | ||
| 143 | adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK; | 107 | adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK; |
| 144 | adev->mes.vmid_mask_mmhub = 0xffffff00; | 108 | adev->mes.vmid_mask_mmhub = 0xffffff00; |
| 145 | adev->mes.vmid_mask_gfxhub = 0xffffff00; | 109 | adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xfffffffe : 0xffffff00; |
| 110 | |||
| 111 | num_pipes = adev->gfx.me.num_pipe_per_me * adev->gfx.me.num_me; | ||
| 112 | if (num_pipes > AMDGPU_MES_MAX_GFX_PIPES) | ||
| 113 | dev_warn(adev->dev, "more gfx pipes than supported by MES! (%d vs %d)\n", | ||
| 114 | num_pipes, AMDGPU_MES_MAX_GFX_PIPES); | ||
| 115 | |||
| 116 | for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) { | ||
| 117 | if (i >= num_pipes) | ||
| 118 | break; | ||
| 119 | if (amdgpu_ip_version(adev, GC_HWIP, 0) >= | ||
| 120 | IP_VERSION(12, 0, 0)) | ||
| 121 | /* | ||
| 122 | * GFX V12 has only one GFX pipe, but 8 queues in it. | ||
| 123 | * GFX pipe 0 queue 0 is being used by Kernel queue. | ||
| 124 | * Set GFX pipe 0 queue 1-7 for MES scheduling | ||
| 125 | * mask = 1111 1110b | ||
| 126 | */ | ||
| 127 | adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0xFF : 0xFE; | ||
| 128 | else | ||
| 129 | /* | ||
| 130 | * GFX pipe 0 queue 0 is being used by Kernel queue. | ||
| 131 | * Set GFX pipe 0 queue 1 for MES scheduling | ||
| 132 | * mask = 10b | ||
| 133 | */ | ||
| 134 | adev->mes.gfx_hqd_mask[i] = adev->gfx.disable_kq ? 0x3 : 0x2; | ||
| 135 | } | ||
| 136 | |||
| 137 | num_pipes = adev->gfx.mec.num_pipe_per_mec * adev->gfx.mec.num_mec; | ||
| 138 | if (num_pipes > AMDGPU_MES_MAX_COMPUTE_PIPES) | ||
| 139 | dev_warn(adev->dev, "more compute pipes than supported by MES! (%d vs %d)\n", | ||
| 140 | num_pipes, AMDGPU_MES_MAX_COMPUTE_PIPES); | ||
| 146 | 141 | ||
| 147 | for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) { | 142 | for (i = 0; i < AMDGPU_MES_MAX_COMPUTE_PIPES; i++) { |
| 148 | if (i >= (adev->gfx.mec.num_pipe_per_mec * adev->gfx.mec.num_mec)) | 143 | if (i >= num_pipes) |
| 149 | break; | 144 | break; |
| 150 | adev->mes.compute_hqd_mask[i] = 0xc; | 145 | adev->mes.compute_hqd_mask[i] = adev->gfx.disable_kq ? 0xF : 0xC; |
| 151 | } | 146 | } |
| 152 | 147 | ||
| 153 | for (i = 0; i < AMDGPU_MES_MAX_GFX_PIPES; i++) | 148 | num_pipes = adev->sdma.num_instances; |
| 154 | adev->mes.gfx_hqd_mask[i] = i ? 0 : 0xfffffffe; | 149 | if (num_pipes > AMDGPU_MES_MAX_SDMA_PIPES) |
| 150 | dev_warn(adev->dev, "more SDMA pipes than supported by MES! (%d vs %d)\n", | ||
| 151 | num_pipes, AMDGPU_MES_MAX_SDMA_PIPES); | ||
| 155 | 152 | ||
| 156 | for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) { | 153 | for (i = 0; i < AMDGPU_MES_MAX_SDMA_PIPES; i++) { |
| 157 | if (i >= adev->sdma.num_instances) | 154 | if (i >= num_pipes) |
| 158 | break; | 155 | break; |
| 159 | adev->mes.sdma_hqd_mask[i] = 0xfc; | 156 | adev->mes.sdma_hqd_mask[i] = 0xfc; |
| 160 | } | 157 | } |
| @@ -240,244 +237,6 @@ void amdgpu_mes_fini(struct amdgpu_device *adev) · 79 unmodified lines | |||
| 240 | mutex_destroy(&adev->mes.mutex_hidden); | 237 | mutex_destroy(&adev->mes.mutex_hidden); |
| 241 | } | 238 | } |
| 242 | 239 | ||
| 243 | static void amdgpu_mes_queue_free_mqd(struct amdgpu_mes_queue *q) | ||
| 244 | { | ||
| 245 | amdgpu_bo_free_kernel(&q->mqd_obj, | ||
| 246 | &q->mqd_gpu_addr, | ||
| 247 | &q->mqd_cpu_ptr); | ||
| 248 | } | ||
| 249 | |||
| 250 | int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid, | ||
| 251 | struct amdgpu_vm *vm) | ||
| 252 | { | ||
| 253 | struct amdgpu_mes_process *process; | ||
| 254 | int r; | ||
| 255 | |||
| 256 | /* allocate the mes process buffer */ | ||
| 257 | process = kzalloc(sizeof(struct amdgpu_mes_process), GFP_KERNEL); | ||
| 258 | if (!process) { | ||
| 259 | DRM_ERROR("no more memory to create mes process\n"); | ||
| 260 | return -ENOMEM; | ||
| 261 | } | ||
| 262 | |||
| 263 | /* allocate the process context bo and map it */ | ||
| 264 | r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_PROC_CTX_SIZE, PAGE_SIZE, | ||
| 265 | AMDGPU_GEM_DOMAIN_GTT, | ||
| 266 | &process->proc_ctx_bo, | ||
| 267 | &process->proc_ctx_gpu_addr, | ||
| 268 | &process->proc_ctx_cpu_ptr); | ||
| 269 | if (r) { | ||
| 270 | DRM_ERROR("failed to allocate process context bo\n"); | ||
| 271 | goto clean_up_memory; | ||
| 272 | } | ||
| 273 | memset(process->proc_ctx_cpu_ptr, 0, AMDGPU_MES_PROC_CTX_SIZE); | ||
| 274 | |||
| 275 | /* | ||
| 276 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 277 | * lock dependencies. | ||
| 278 | */ | ||
| 279 | amdgpu_mes_lock(&adev->mes); | ||
| 280 | |||
| 281 | /* add the mes process to idr list */ | ||
| 282 | r = idr_alloc(&adev->mes.pasid_idr, process, pasid, pasid + 1, | ||
| 283 | GFP_KERNEL); | ||
| 284 | if (r < 0) { | ||
| 285 | DRM_ERROR("failed to lock pasid=%d\n", pasid); | ||
| 286 | goto clean_up_ctx; | ||
| 287 | } | ||
| 288 | |||
| 289 | INIT_LIST_HEAD(&process->gang_list); | ||
| 290 | process->vm = vm; | ||
| 291 | process->pasid = pasid; | ||
| 292 | process->process_quantum = adev->mes.default_process_quantum; | ||
| 293 | process->pd_gpu_addr = amdgpu_bo_gpu_offset(vm->root.bo); | ||
| 294 | |||
| 295 | amdgpu_mes_unlock(&adev->mes); | ||
| 296 | return 0; | ||
| 297 | |||
| 298 | clean_up_ctx: | ||
| 299 | amdgpu_mes_unlock(&adev->mes); | ||
| 300 | amdgpu_bo_free_kernel(&process->proc_ctx_bo, | ||
| 301 | &process->proc_ctx_gpu_addr, | ||
| 302 | &process->proc_ctx_cpu_ptr); | ||
| 303 | clean_up_memory: | ||
| 304 | kfree(process); | ||
| 305 | return r; | ||
| 306 | } | ||
| 307 | |||
| 308 | void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid) | ||
| 309 | { | ||
| 310 | struct amdgpu_mes_process *process; | ||
| 311 | struct amdgpu_mes_gang *gang, *tmp1; | ||
| 312 | struct amdgpu_mes_queue *queue, *tmp2; | ||
| 313 | struct mes_remove_queue_input queue_input; | ||
| 314 | unsigned long flags; | ||
| 315 | int r; | ||
| 316 | |||
| 317 | /* | ||
| 318 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 319 | * lock dependencies. | ||
| 320 | */ | ||
| 321 | amdgpu_mes_lock(&adev->mes); | ||
| 322 | |||
| 323 | process = idr_find(&adev->mes.pasid_idr, pasid); | ||
| 324 | if (!process) { | ||
| 325 | DRM_WARN("pasid %d doesn't exist\n", pasid); | ||
| 326 | amdgpu_mes_unlock(&adev->mes); | ||
| 327 | return; | ||
| 328 | } | ||
| 329 | |||
| 330 | /* Remove all queues from hardware */ | ||
| 331 | list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) { | ||
| 332 | list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) { | ||
| 333 | spin_lock_irqsave(&adev->mes.queue_id_lock, flags); | ||
| 334 | idr_remove(&adev->mes.queue_id_idr, queue->queue_id); | ||
| 335 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 336 | |||
| 337 | queue_input.doorbell_offset = queue->doorbell_off; | ||
| 338 | queue_input.gang_context_addr = gang->gang_ctx_gpu_addr; | ||
| 339 | |||
| 340 | r = adev->mes.funcs->remove_hw_queue(&adev->mes, | ||
| 341 | &queue_input); | ||
| 342 | if (r) | ||
| 343 | DRM_WARN("failed to remove hardware queue\n"); | ||
| 344 | } | ||
| 345 | |||
| 346 | idr_remove(&adev->mes.gang_id_idr, gang->gang_id); | ||
| 347 | } | ||
| 348 | |||
| 349 | idr_remove(&adev->mes.pasid_idr, pasid); | ||
| 350 | amdgpu_mes_unlock(&adev->mes); | ||
| 351 | |||
| 352 | /* free all memory allocated by the process */ | ||
| 353 | list_for_each_entry_safe(gang, tmp1, &process->gang_list, list) { | ||
| 354 | /* free all queues in the gang */ | ||
| 355 | list_for_each_entry_safe(queue, tmp2, &gang->queue_list, list) { | ||
| 356 | amdgpu_mes_queue_free_mqd(queue); | ||
| 357 | list_del(&queue->list); | ||
| 358 | kfree(queue); | ||
| 359 | } | ||
| 360 | amdgpu_bo_free_kernel(&gang->gang_ctx_bo, | ||
| 361 | &gang->gang_ctx_gpu_addr, | ||
| 362 | &gang->gang_ctx_cpu_ptr); | ||
| 363 | list_del(&gang->list); | ||
| 364 | kfree(gang); | ||
| 365 | |||
| 366 | } | ||
| 367 | amdgpu_bo_free_kernel(&process->proc_ctx_bo, | ||
| 368 | &process->proc_ctx_gpu_addr, | ||
| 369 | &process->proc_ctx_cpu_ptr); | ||
| 370 | kfree(process); | ||
| 371 | } | ||
| 372 | |||
| 373 | int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid, | ||
| 374 | struct amdgpu_mes_gang_properties *gprops, | ||
| 375 | int *gang_id) | ||
| 376 | { | ||
| 377 | struct amdgpu_mes_process *process; | ||
| 378 | struct amdgpu_mes_gang *gang; | ||
| 379 | int r; | ||
| 380 | |||
| 381 | /* allocate the mes gang buffer */ | ||
| 382 | gang = kzalloc(sizeof(struct amdgpu_mes_gang), GFP_KERNEL); | ||
| 383 | if (!gang) { | ||
| 384 | return -ENOMEM; | ||
| 385 | } | ||
| 386 | |||
| 387 | /* allocate the gang context bo and map it to cpu space */ | ||
| 388 | r = amdgpu_bo_create_kernel(adev, AMDGPU_MES_GANG_CTX_SIZE, PAGE_SIZE, | ||
| 389 | AMDGPU_GEM_DOMAIN_GTT, | ||
| 390 | &gang->gang_ctx_bo, | ||
| 391 | &gang->gang_ctx_gpu_addr, | ||
| 392 | &gang->gang_ctx_cpu_ptr); | ||
| 393 | if (r) { | ||
| 394 | DRM_ERROR("failed to allocate process context bo\n"); | ||
| 395 | goto clean_up_mem; | ||
| 396 | } | ||
| 397 | memset(gang->gang_ctx_cpu_ptr, 0, AMDGPU_MES_GANG_CTX_SIZE); | ||
| 398 | |||
| 399 | /* | ||
| 400 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 401 | * lock dependencies. | ||
| 402 | */ | ||
| 403 | amdgpu_mes_lock(&adev->mes); | ||
| 404 | |||
| 405 | process = idr_find(&adev->mes.pasid_idr, pasid); | ||
| 406 | if (!process) { | ||
| 407 | DRM_ERROR("pasid %d doesn't exist\n", pasid); | ||
| 408 | r = -EINVAL; | ||
| 409 | goto clean_up_ctx; | ||
| 410 | } | ||
| 411 | |||
| 412 | /* add the mes gang to idr list */ | ||
| 413 | r = idr_alloc(&adev->mes.gang_id_idr, gang, 1, 0, | ||
| 414 | GFP_KERNEL); | ||
| 415 | if (r < 0) { | ||
| 416 | DRM_ERROR("failed to allocate idr for gang\n"); | ||
| 417 | goto clean_up_ctx; | ||
| 418 | } | ||
| 419 | |||
| 420 | gang->gang_id = r; | ||
| 421 | *gang_id = r; | ||
| 422 | |||
| 423 | INIT_LIST_HEAD(&gang->queue_list); | ||
| 424 | gang->process = process; | ||
| 425 | gang->priority = gprops->priority; | ||
| 426 | gang->gang_quantum = gprops->gang_quantum ? | ||
| 427 | gprops->gang_quantum : adev->mes.default_gang_quantum; | ||
| 428 | gang->global_priority_level = gprops->global_priority_level; | ||
| 429 | gang->inprocess_gang_priority = gprops->inprocess_gang_priority; | ||
| 430 | list_add_tail(&gang->list, &process->gang_list); | ||
| 431 | |||
| 432 | amdgpu_mes_unlock(&adev->mes); | ||
| 433 | return 0; | ||
| 434 | |||
| 435 | clean_up_ctx: | ||
| 436 | amdgpu_mes_unlock(&adev->mes); | ||
| 437 | amdgpu_bo_free_kernel(&gang->gang_ctx_bo, | ||
| 438 | &gang->gang_ctx_gpu_addr, | ||
| 439 | &gang->gang_ctx_cpu_ptr); | ||
| 440 | clean_up_mem: | ||
| 441 | kfree(gang); | ||
| 442 | return r; | ||
| 443 | } | ||
| 444 | |||
| 445 | int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id) | ||
| 446 | { | ||
| 447 | struct amdgpu_mes_gang *gang; | ||
| 448 | |||
| 449 | /* | ||
| 450 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 451 | * lock dependencies. | ||
| 452 | */ | ||
| 453 | amdgpu_mes_lock(&adev->mes); | ||
| 454 | |||
| 455 | gang = idr_find(&adev->mes.gang_id_idr, gang_id); | ||
| 456 | if (!gang) { | ||
| 457 | DRM_ERROR("gang id %d doesn't exist\n", gang_id); | ||
| 458 | amdgpu_mes_unlock(&adev->mes); | ||
| 459 | return -EINVAL; | ||
| 460 | } | ||
| 461 | |||
| 462 | if (!list_empty(&gang->queue_list)) { | ||
| 463 | DRM_ERROR("queue list is not empty\n"); | ||
| 464 | amdgpu_mes_unlock(&adev->mes); | ||
| 465 | return -EBUSY; | ||
| 466 | } | ||
| 467 | |||
| 468 | idr_remove(&adev->mes.gang_id_idr, gang->gang_id); | ||
| 469 | list_del(&gang->list); | ||
| 470 | amdgpu_mes_unlock(&adev->mes); | ||
| 471 | |||
| 472 | amdgpu_bo_free_kernel(&gang->gang_ctx_bo, | ||
| 473 | &gang->gang_ctx_gpu_addr, | ||
| 474 | &gang->gang_ctx_cpu_ptr); | ||
| 475 | |||
| 476 | kfree(gang); | ||
| 477 | |||
| 478 | return 0; | ||
| 479 | } | ||
| 480 | |||
| 481 | int amdgpu_mes_suspend(struct amdgpu_device *adev) | 240 | int amdgpu_mes_suspend(struct amdgpu_device *adev) |
| 482 | { | 241 | { |
| 483 | struct mes_suspend_gang_input input; | 242 | struct mes_suspend_gang_input input; |
| @@ -526,304 +285,6 @@ int amdgpu_mes_resume(struct amdgpu_device *adev) · 42 unmodified lines | |||
| 526 | return r; | 285 | return r; |
| 527 | } | 286 | } |
| 528 | 287 | ||
| 529 | static int amdgpu_mes_queue_alloc_mqd(struct amdgpu_device *adev, | ||
| 530 | struct amdgpu_mes_queue *q, | ||
| 531 | struct amdgpu_mes_queue_properties *p) | ||
| 532 | { | ||
| 533 | struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type]; | ||
| 534 | u32 mqd_size = mqd_mgr->mqd_size; | ||
| 535 | int r; | ||
| 536 | |||
| 537 | r = amdgpu_bo_create_kernel(adev, mqd_size, PAGE_SIZE, | ||
| 538 | AMDGPU_GEM_DOMAIN_GTT, | ||
| 539 | &q->mqd_obj, | ||
| 540 | &q->mqd_gpu_addr, &q->mqd_cpu_ptr); | ||
| 541 | if (r) { | ||
| 542 | dev_warn(adev->dev, "failed to create queue mqd bo (%d)", r); | ||
| 543 | return r; | ||
| 544 | } | ||
| 545 | memset(q->mqd_cpu_ptr, 0, mqd_size); | ||
| 546 | |||
| 547 | r = amdgpu_bo_reserve(q->mqd_obj, false); | ||
| 548 | if (unlikely(r != 0)) | ||
| 549 | goto clean_up; | ||
| 550 | |||
| 551 | return 0; | ||
| 552 | |||
| 553 | clean_up: | ||
| 554 | amdgpu_bo_free_kernel(&q->mqd_obj, | ||
| 555 | &q->mqd_gpu_addr, | ||
| 556 | &q->mqd_cpu_ptr); | ||
| 557 | return r; | ||
| 558 | } | ||
| 559 | |||
| 560 | static void amdgpu_mes_queue_init_mqd(struct amdgpu_device *adev, | ||
| 561 | struct amdgpu_mes_queue *q, | ||
| 562 | struct amdgpu_mes_queue_properties *p) | ||
| 563 | { | ||
| 564 | struct amdgpu_mqd *mqd_mgr = &adev->mqds[p->queue_type]; | ||
| 565 | struct amdgpu_mqd_prop mqd_prop = {0}; | ||
| 566 | |||
| 567 | mqd_prop.mqd_gpu_addr = q->mqd_gpu_addr; | ||
| 568 | mqd_prop.hqd_base_gpu_addr = p->hqd_base_gpu_addr; | ||
| 569 | mqd_prop.rptr_gpu_addr = p->rptr_gpu_addr; | ||
| 570 | mqd_prop.wptr_gpu_addr = p->wptr_gpu_addr; | ||
| 571 | mqd_prop.queue_size = p->queue_size; | ||
| 572 | mqd_prop.use_doorbell = true; | ||
| 573 | mqd_prop.doorbell_index = p->doorbell_off; | ||
| 574 | mqd_prop.eop_gpu_addr = p->eop_gpu_addr; | ||
| 575 | mqd_prop.hqd_pipe_priority = p->hqd_pipe_priority; | ||
| 576 | mqd_prop.hqd_queue_priority = p->hqd_queue_priority; | ||
| 577 | mqd_prop.hqd_active = false; | ||
| 578 | |||
| 579 | if (p->queue_type == AMDGPU_RING_TYPE_GFX || | ||
| 580 | p->queue_type == AMDGPU_RING_TYPE_COMPUTE) { | ||
| 581 | mutex_lock(&adev->srbm_mutex); | ||
| 582 | amdgpu_gfx_select_me_pipe_q(adev, p->ring->me, p->ring->pipe, 0, 0, 0); | ||
| 583 | } | ||
| 584 | |||
| 585 | mqd_mgr->init_mqd(adev, q->mqd_cpu_ptr, &mqd_prop); | ||
| 586 | |||
| 587 | if (p->queue_type == AMDGPU_RING_TYPE_GFX || | ||
| 588 | p->queue_type == AMDGPU_RING_TYPE_COMPUTE) { | ||
| 589 | amdgpu_gfx_select_me_pipe_q(adev, 0, 0, 0, 0, 0); | ||
| 590 | mutex_unlock(&adev->srbm_mutex); | ||
| 591 | } | ||
| 592 | |||
| 593 | amdgpu_bo_unreserve(q->mqd_obj); | ||
| 594 | } | ||
| 595 | |||
| 596 | int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id, | ||
| 597 | struct amdgpu_mes_queue_properties *qprops, | ||
| 598 | int *queue_id) | ||
| 599 | { | ||
| 600 | struct amdgpu_mes_queue *queue; | ||
| 601 | struct amdgpu_mes_gang *gang; | ||
| 602 | struct mes_add_queue_input queue_input; | ||
| 603 | unsigned long flags; | ||
| 604 | int r; | ||
| 605 | |||
| 606 | memset(&queue_input, 0, sizeof(struct mes_add_queue_input)); | ||
| 607 | |||
| 608 | /* allocate the mes queue buffer */ | ||
| 609 | queue = kzalloc(sizeof(struct amdgpu_mes_queue), GFP_KERNEL); | ||
| 610 | if (!queue) { | ||
| 611 | DRM_ERROR("Failed to allocate memory for queue\n"); | ||
| 612 | return -ENOMEM; | ||
| 613 | } | ||
| 614 | |||
| 615 | /* Allocate the queue mqd */ | ||
| 616 | r = amdgpu_mes_queue_alloc_mqd(adev, queue, qprops); | ||
| 617 | if (r) | ||
| 618 | goto clean_up_memory; | ||
| 619 | |||
| 620 | /* | ||
| 621 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 622 | * lock dependencies. | ||
| 623 | */ | ||
| 624 | amdgpu_mes_lock(&adev->mes); | ||
| 625 | |||
| 626 | gang = idr_find(&adev->mes.gang_id_idr, gang_id); | ||
| 627 | if (!gang) { | ||
| 628 | DRM_ERROR("gang id %d doesn't exist\n", gang_id); | ||
| 629 | r = -EINVAL; | ||
| 630 | goto clean_up_mqd; | ||
| 631 | } | ||
| 632 | |||
| 633 | /* add the mes gang to idr list */ | ||
| 634 | spin_lock_irqsave(&adev->mes.queue_id_lock, flags); | ||
| 635 | r = idr_alloc(&adev->mes.queue_id_idr, queue, 1, 0, | ||
| 636 | GFP_ATOMIC); | ||
| 637 | if (r < 0) { | ||
| 638 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 639 | goto clean_up_mqd; | ||
| 640 | } | ||
| 641 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 642 | *queue_id = queue->queue_id = r; | ||
| 643 | |||
| 644 | /* allocate a doorbell index for the queue */ | ||
| 645 | r = amdgpu_mes_kernel_doorbell_get(adev, | ||
| 646 | qprops->queue_type, | ||
| 647 | &qprops->doorbell_off); | ||
| 648 | if (r) | ||
| 649 | goto clean_up_queue_id; | ||
| 650 | |||
| 651 | /* initialize the queue mqd */ | ||
| 652 | amdgpu_mes_queue_init_mqd(adev, queue, qprops); | ||
| 653 | |||
| 654 | /* add hw queue to mes */ | ||
| 655 | queue_input.process_id = gang->process->pasid; | ||
| 656 | |||
| 657 | queue_input.page_table_base_addr = | ||
| 658 | adev->vm_manager.vram_base_offset + gang->process->pd_gpu_addr - | ||
| 659 | adev->gmc.vram_start; | ||
| 660 | |||
| 661 | queue_input.process_va_start = 0; | ||
| 662 | queue_input.process_va_end = | ||
| 663 | (adev->vm_manager.max_pfn - 1) << AMDGPU_GPU_PAGE_SHIFT; | ||
| 664 | queue_input.process_quantum = gang->process->process_quantum; | ||
| 665 | queue_input.process_context_addr = gang->process->proc_ctx_gpu_addr; | ||
| 666 | queue_input.gang_quantum = gang->gang_quantum; | ||
| 667 | queue_input.gang_context_addr = gang->gang_ctx_gpu_addr; | ||
| 668 | queue_input.inprocess_gang_priority = gang->inprocess_gang_priority; | ||
| 669 | queue_input.gang_global_priority_level = gang->global_priority_level; | ||
| 670 | queue_input.doorbell_offset = qprops->doorbell_off; | ||
| 671 | queue_input.mqd_addr = queue->mqd_gpu_addr; | ||
| 672 | queue_input.wptr_addr = qprops->wptr_gpu_addr; | ||
| 673 | queue_input.wptr_mc_addr = qprops->wptr_mc_addr; | ||
| 674 | queue_input.queue_type = qprops->queue_type; | ||
| 675 | queue_input.paging = qprops->paging; | ||
| 676 | queue_input.is_kfd_process = 0; | ||
| 677 | |||
| 678 | r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input); | ||
| 679 | if (r) { | ||
| 680 | DRM_ERROR("failed to add hardware queue to MES, doorbell=0x%llx\n", | ||
| 681 | qprops->doorbell_off); | ||
| 682 | goto clean_up_doorbell; | ||
| 683 | } | ||
| 684 | |||
| 685 | DRM_DEBUG("MES hw queue was added, pasid=%d, gang id=%d, " | ||
| 686 | "queue type=%d, doorbell=0x%llx\n", | ||
| 687 | gang->process->pasid, gang_id, qprops->queue_type, | ||
| 688 | qprops->doorbell_off); | ||
| 689 | |||
| 690 | queue->ring = qprops->ring; | ||
| 691 | queue->doorbell_off = qprops->doorbell_off; | ||
| 692 | queue->wptr_gpu_addr = qprops->wptr_gpu_addr; | ||
| 693 | queue->queue_type = qprops->queue_type; | ||
| 694 | queue->paging = qprops->paging; | ||
| 695 | queue->gang = gang; | ||
| 696 | queue->ring->mqd_ptr = queue->mqd_cpu_ptr; | ||
| 697 | list_add_tail(&queue->list, &gang->queue_list); | ||
| 698 | |||
| 699 | amdgpu_mes_unlock(&adev->mes); | ||
| 700 | return 0; | ||
| 701 | |||
| 702 | clean_up_doorbell: | ||
| 703 | amdgpu_mes_kernel_doorbell_free(adev, qprops->doorbell_off); | ||
| 704 | clean_up_queue_id: | ||
| 705 | spin_lock_irqsave(&adev->mes.queue_id_lock, flags); | ||
| 706 | idr_remove(&adev->mes.queue_id_idr, queue->queue_id); | ||
| 707 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 708 | clean_up_mqd: | ||
| 709 | amdgpu_mes_unlock(&adev->mes); | ||
| 710 | amdgpu_mes_queue_free_mqd(queue); | ||
| 711 | clean_up_memory: | ||
| 712 | kfree(queue); | ||
| 713 | return r; | ||
| 714 | } | ||
| 715 | |||
| 716 | int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id) | ||
| 717 | { | ||
| 718 | unsigned long flags; | ||
| 719 | struct amdgpu_mes_queue *queue; | ||
| 720 | struct amdgpu_mes_gang *gang; | ||
| 721 | struct mes_remove_queue_input queue_input; | ||
| 722 | int r; | ||
| 723 | |||
| 724 | /* | ||
| 725 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 726 | * lock dependencies. | ||
| 727 | */ | ||
| 728 | amdgpu_mes_lock(&adev->mes); | ||
| 729 | |||
| 730 | /* remove the mes gang from idr list */ | ||
| 731 | spin_lock_irqsave(&adev->mes.queue_id_lock, flags); | ||
| 732 | |||
| 733 | queue = idr_find(&adev->mes.queue_id_idr, queue_id); | ||
| 734 | if (!queue) { | ||
| 735 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 736 | amdgpu_mes_unlock(&adev->mes); | ||
| 737 | DRM_ERROR("queue id %d doesn't exist\n", queue_id); | ||
| 738 | return -EINVAL; | ||
| 739 | } | ||
| 740 | |||
| 741 | idr_remove(&adev->mes.queue_id_idr, queue_id); | ||
| 742 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 743 | |||
| 744 | DRM_DEBUG("try to remove queue, doorbell off = 0x%llx\n", | ||
| 745 | queue->doorbell_off); | ||
| 746 | |||
| 747 | gang = queue->gang; | ||
| 748 | queue_input.doorbell_offset = queue->doorbell_off; | ||
| 749 | queue_input.gang_context_addr = gang->gang_ctx_gpu_addr; | ||
| 750 | |||
| 751 | r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input); | ||
| 752 | if (r) | ||
| 753 | DRM_ERROR("failed to remove hardware queue, queue id = %d\n", | ||
| 754 | queue_id); | ||
| 755 | |||
| 756 | list_del(&queue->list); | ||
| 757 | amdgpu_mes_kernel_doorbell_free(adev, queue->doorbell_off); | ||
| 758 | amdgpu_mes_unlock(&adev->mes); | ||
| 759 | |||
| 760 | amdgpu_mes_queue_free_mqd(queue); | ||
| 761 | kfree(queue); | ||
| 762 | return 0; | ||
| 763 | } | ||
| 764 | |||
| 765 | int amdgpu_mes_reset_hw_queue(struct amdgpu_device *adev, int queue_id) | ||
| 766 | { | ||
| 767 | unsigned long flags; | ||
| 768 | struct amdgpu_mes_queue *queue; | ||
| 769 | struct amdgpu_mes_gang *gang; | ||
| 770 | struct mes_reset_queue_input queue_input; | ||
| 771 | int r; | ||
| 772 | |||
| 773 | /* | ||
| 774 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 775 | * lock dependencies. | ||
| 776 | */ | ||
| 777 | amdgpu_mes_lock(&adev->mes); | ||
| 778 | |||
| 779 | /* remove the mes gang from idr list */ | ||
| 780 | spin_lock_irqsave(&adev->mes.queue_id_lock, flags); | ||
| 781 | |||
| 782 | queue = idr_find(&adev->mes.queue_id_idr, queue_id); | ||
| 783 | if (!queue) { | ||
| 784 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 785 | amdgpu_mes_unlock(&adev->mes); | ||
| 786 | DRM_ERROR("queue id %d doesn't exist\n", queue_id); | ||
| 787 | return -EINVAL; | ||
| 788 | } | ||
| 789 | spin_unlock_irqrestore(&adev->mes.queue_id_lock, flags); | ||
| 790 | |||
| 791 | DRM_DEBUG("try to reset queue, doorbell off = 0x%llx\n", | ||
| 792 | queue->doorbell_off); | ||
| 793 | |||
| 794 | gang = queue->gang; | ||
| 795 | queue_input.doorbell_offset = queue->doorbell_off; | ||
| 796 | queue_input.gang_context_addr = gang->gang_ctx_gpu_addr; | ||
| 797 | |||
| 798 | r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input); | ||
| 799 | if (r) | ||
| 800 | DRM_ERROR("failed to reset hardware queue, queue id = %d\n", | ||
| 801 | queue_id); | ||
| 802 | |||
| 803 | amdgpu_mes_unlock(&adev->mes); | ||
| 804 | |||
| 805 | return 0; | ||
| 806 | } | ||
| 807 | |||
| 808 | int amdgpu_mes_reset_hw_queue_mmio(struct amdgpu_device *adev, int queue_type, | ||
| 809 | int me_id, int pipe_id, int queue_id, int vmid) | ||
| 810 | { | ||
| 811 | struct mes_reset_queue_input queue_input; | ||
| 812 | int r; | ||
| 813 | |||
| 814 | queue_input.queue_type = queue_type; | ||
| 815 | queue_input.use_mmio = true; | ||
| 816 | queue_input.me_id = me_id; | ||
| 817 | queue_input.pipe_id = pipe_id; | ||
| 818 | queue_input.queue_id = queue_id; | ||
| 819 | queue_input.vmid = vmid; | ||
| 820 | r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input); | ||
| 821 | if (r) | ||
| 822 | DRM_ERROR("failed to reset hardware queue by mmio, queue id = %d\n", | ||
| 823 | queue_id); | ||
| 824 | return r; | ||
| 825 | } | ||
| 826 | |||
| 827 | int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev, | 288 | int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev, |
| 828 | struct amdgpu_ring *ring) | 289 | struct amdgpu_ring *ring) |
| 829 | { | 290 | { |
| @@ -874,7 +335,7 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 44 unmodified lines | |||
| 874 | unsigned int vmid, | 335 | unsigned int vmid, |
| 875 | bool use_mmio) | 336 | bool use_mmio) |
| 876 | { | 337 | { |
| 877 | struct mes_reset_legacy_queue_input queue_input; | 338 | struct mes_reset_queue_input queue_input; |
| 878 | int r; | 339 | int r; |
| 879 | 340 | ||
| 880 | memset(&queue_input, 0, sizeof(queue_input)); | 341 | memset(&queue_input, 0, sizeof(queue_input)); |
| @@ -888,8 +349,11 @@ int amdgpu_mes_reset_legacy_queue(struct amdgpu_device *adev, · 7 unmodified lines | |||
| 888 | queue_input.wptr_addr = ring->wptr_gpu_addr; | 349 | queue_input.wptr_addr = ring->wptr_gpu_addr; |
| 889 | queue_input.vmid = vmid; | 350 | queue_input.vmid = vmid; |
| 890 | queue_input.use_mmio = use_mmio; | 351 | queue_input.use_mmio = use_mmio; |
| 352 | queue_input.is_kq = true; | ||
| 353 | if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) | ||
| 354 | queue_input.legacy_gfx = true; | ||
| 891 | 355 | ||
| 892 | r = adev->mes.funcs->reset_legacy_queue(&adev->mes, &queue_input); | 356 | r = adev->mes.funcs->reset_hw_queue(&adev->mes, &queue_input); |
| 893 | if (r) | 357 | if (r) |
| 894 | DRM_ERROR("failed to reset legacy queue\n"); | 358 | DRM_ERROR("failed to reset legacy queue\n"); |
| 895 | 359 | ||
| @@ -905,7 +369,7 @@ uint32_t amdgpu_mes_rreg(struct amdgpu_device *adev, uint32_t reg) · 9 unmodified lines | |||
| 905 | uint32_t *read_val_ptr; | 369 | uint32_t *read_val_ptr; |
| 906 | 370 | ||
| 907 | if (amdgpu_device_wb_get(adev, &addr_offset)) { | 371 | if (amdgpu_device_wb_get(adev, &addr_offset)) { |
| 908 | DRM_ERROR("critical bug! too many mes readers\n"); | 372 | dev_err(adev->dev, "critical bug! too many mes readers\n"); |
| 909 | goto error; | 373 | goto error; |
| 910 | } | 374 | } |
| 911 | read_val_gpu_addr = adev->wb.gpu_addr + (addr_offset * 4); | 375 | read_val_gpu_addr = adev->wb.gpu_addr + (addr_offset * 4); |
| @@ -915,13 +379,13 @@ uint32_t amdgpu_mes_rreg(struct amdgpu_device *adev, uint32_t reg) · 3 unmodified lines | |||
| 915 | op_input.read_reg.buffer_addr = read_val_gpu_addr; | 379 | op_input.read_reg.buffer_addr = read_val_gpu_addr; |
| 916 | 380 | ||
| 917 | if (!adev->mes.funcs->misc_op) { | 381 | if (!adev->mes.funcs->misc_op) { |
| 918 | DRM_ERROR("mes rreg is not supported!\n"); | 382 | dev_err(adev->dev, "mes rreg is not supported!\n"); |
| 919 | goto error; | 383 | goto error; |
| 920 | } | 384 | } |
| 921 | 385 | ||
| 922 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); | 386 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); |
| 923 | if (r) | 387 | if (r) |
| 924 | DRM_ERROR("failed to read reg (0x%x)\n", reg); | 388 | dev_err(adev->dev, "failed to read reg (0x%x)\n", reg); |
| 925 | else | 389 | else |
| 926 | val = *(read_val_ptr); | 390 | val = *(read_val_ptr); |
| 927 | 391 | ||
| @@ -942,14 +406,14 @@ int amdgpu_mes_wreg(struct amdgpu_device *adev, · 14 unmodified lines | |||
| 942 | op_input.write_reg.reg_value = val; | 406 | op_input.write_reg.reg_value = val; |
| 943 | 407 | ||
| 944 | if (!adev->mes.funcs->misc_op) { | 408 | if (!adev->mes.funcs->misc_op) { |
| 945 | DRM_ERROR("mes wreg is not supported!\n"); | 409 | dev_err(adev->dev, "mes wreg is not supported!\n"); |
| 946 | r = -EINVAL; | 410 | r = -EINVAL; |
| 947 | goto error; | 411 | goto error; |
| 948 | } | 412 | } |
| 949 | 413 | ||
| 950 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); | 414 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); |
| 951 | if (r) | 415 | if (r) |
| 952 | DRM_ERROR("failed to write reg (0x%x)\n", reg); | 416 | dev_err(adev->dev, "failed to write reg (0x%x)\n", reg); |
| 953 | 417 | ||
| 954 | error: | 418 | error: |
| 955 | return r; | 419 | return r; |
| @@ -969,14 +433,14 @@ int amdgpu_mes_reg_write_reg_wait(struct amdgpu_device *adev, · 13 unmodified lines | |||
| 969 | op_input.wrm_reg.mask = mask; | 433 | op_input.wrm_reg.mask = mask; |
| 970 | 434 | ||
| 971 | if (!adev->mes.funcs->misc_op) { | 435 | if (!adev->mes.funcs->misc_op) { |
| 972 | DRM_ERROR("mes reg_write_reg_wait is not supported!\n"); | 436 | dev_err(adev->dev, "mes reg_write_reg_wait is not supported!\n"); |
| 973 | r = -EINVAL; | 437 | r = -EINVAL; |
| 974 | goto error; | 438 | goto error; |
| 975 | } | 439 | } |
| 976 | 440 | ||
| 977 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); | 441 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); |
| 978 | if (r) | 442 | if (r) |
| 979 | DRM_ERROR("failed to reg_write_reg_wait\n"); | 443 | dev_err(adev->dev, "failed to reg_write_reg_wait\n"); |
| 980 | 444 | ||
| 981 | error: | 445 | error: |
| 982 | return r; | 446 | return r; |
| @@ -994,14 +458,14 @@ int amdgpu_mes_reg_wait(struct amdgpu_device *adev, uint32_t reg, · 11 unmodified lines | |||
| 994 | op_input.wrm_reg.mask = mask; | 458 | op_input.wrm_reg.mask = mask; |
| 995 | 459 | ||
| 996 | if (!adev->mes.funcs->misc_op) { | 460 | if (!adev->mes.funcs->misc_op) { |
| 997 | DRM_ERROR("mes reg wait is not supported!\n"); | 461 | dev_err(adev->dev, "mes reg wait is not supported!\n"); |
| 998 | r = -EINVAL; | 462 | r = -EINVAL; |
| 999 | goto error; | 463 | goto error; |
| 1000 | } | 464 | } |
| 1001 | 465 | ||
| 1002 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); | 466 | r = adev->mes.funcs->misc_op(&adev->mes, &op_input); |
| 1003 | if (r) | 467 | if (r) |
| 1004 | DRM_ERROR("failed to reg_write_reg_wait\n"); | 468 | dev_err(adev->dev, "failed to reg_write_reg_wait\n"); |
| 1005 | 469 | ||
| 1006 | error: | 470 | error: |
| 1007 | return r; | 471 | return r; |
| @@ -1075,25 +539,6 @@ int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev, · 67 unmodified lines | |||
| 1075 | return r; | 539 | return r; |
| 1076 | } | 540 | } |
| 1077 | 541 | ||
| 1078 | static void | ||
| 1079 | amdgpu_mes_ring_to_queue_props(struct amdgpu_device *adev, | ||
| 1080 | struct amdgpu_ring *ring, | ||
| 1081 | struct amdgpu_mes_queue_properties *props) | ||
| 1082 | { | ||
| 1083 | props->queue_type = ring->funcs->type; | ||
| 1084 | props->hqd_base_gpu_addr = ring->gpu_addr; | ||
| 1085 | props->rptr_gpu_addr = ring->rptr_gpu_addr; | ||
| 1086 | props->wptr_gpu_addr = ring->wptr_gpu_addr; | ||
| 1087 | props->wptr_mc_addr = | ||
| 1088 | ring->mes_ctx->meta_data_mc_addr + ring->wptr_offs; | ||
| 1089 | props->queue_size = ring->ring_size; | ||
| 1090 | props->eop_gpu_addr = ring->eop_gpu_addr; | ||
| 1091 | props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL; | ||
| 1092 | props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM; | ||
| 1093 | props->paging = false; | ||
| 1094 | props->ring = ring; | ||
| 1095 | } | ||
| 1096 | |||
| 1097 | #define DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(_eng) \ | 542 | #define DEFINE_AMDGPU_MES_CTX_GET_OFFS_ENG(_eng) \ |
| 1098 | do { \ | 543 | do { \ |
| 1099 | if (id_offs < AMDGPU_MES_CTX_MAX_OFFS) \ | 544 | if (id_offs < AMDGPU_MES_CTX_MAX_OFFS) \ |
| @@ -1130,453 +575,12 @@ int amdgpu_mes_ctx_get_offs(struct amdgpu_ring *ring, unsigned int id_offs) · 30 unmodified lines | |||
| 1130 | return -EINVAL; | 575 | return -EINVAL; |
| 1131 | } | 576 | } |
| 1132 | 577 | ||
| 1133 | int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id, | ||
| 1134 | int queue_type, int idx, | ||
| 1135 | struct amdgpu_mes_ctx_data *ctx_data, | ||
| 1136 | struct amdgpu_ring **out) | ||
| 1137 | { | ||
| 1138 | struct amdgpu_ring *ring; | ||
| 1139 | struct amdgpu_mes_gang *gang; | ||
| 1140 | struct amdgpu_mes_queue_properties qprops = {0}; | ||
| 1141 | int r, queue_id, pasid; | ||
| 1142 | |||
| 1143 | /* | ||
| 1144 | * Avoid taking any other locks under MES lock to avoid circular | ||
| 1145 | * lock dependencies. | ||
| 1146 | */ | ||
| 1147 | amdgpu_mes_lock(&adev->mes); | ||
| 1148 | gang = idr_find(&adev->mes.gang_id_idr, gang_id); | ||
| 1149 | if (!gang) { | ||
| 1150 | DRM_ERROR("gang id %d doesn't exist\n", gang_id); | ||
| 1151 | amdgpu_mes_unlock(&adev->mes); | ||
| 1152 | return -EINVAL; | ||
| 1153 | } | ||
| 1154 | pasid = gang->process->pasid; | ||
| 1155 | |||
| 1156 | ring = kzalloc(sizeof(struct amdgpu_ring), GFP_KERNEL); | ||
| 1157 | if (!ring) { | ||
| 1158 | amdgpu_mes_unlock(&adev->mes); | ||
| 1159 | return -ENOMEM; | ||
| 1160 | } | ||
| 1161 | |||
| 1162 | ring->ring_obj = NULL; | ||
| 1163 | ring->use_doorbell = true; | ||
| 1164 | ring->is_mes_queue = true; | ||
| 1165 | ring->mes_ctx = ctx_data; | ||
| 1166 | ring->idx = idx; | ||
| 1167 | ring->no_scheduler = true; | ||
| 1168 | |||
| 1169 | if (queue_type == AMDGPU_RING_TYPE_COMPUTE) { | ||
| 1170 | int offset = offsetof(struct amdgpu_mes_ctx_meta_data, | ||
| 1171 | compute[ring->idx].mec_hpd); | ||
| 1172 | ring->eop_gpu_addr = | ||
| 1173 | amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 1174 | } | ||
| 1175 | |||
| 1176 | switch (queue_type) { | ||
| 1177 | case AMDGPU_RING_TYPE_GFX: | ||
| 1178 | ring->funcs = adev->gfx.gfx_ring[0].funcs; | ||
| 1179 | ring->me = adev->gfx.gfx_ring[0].me; | ||
| 1180 | ring->pipe = adev->gfx.gfx_ring[0].pipe; | ||
| 1181 | break; | ||
| 1182 | case AMDGPU_RING_TYPE_COMPUTE: | ||
| 1183 | ring->funcs = adev->gfx.compute_ring[0].funcs; | ||
| 1184 | ring->me = adev->gfx.compute_ring[0].me; | ||
| 1185 | ring->pipe = adev->gfx.compute_ring[0].pipe; | ||
| 1186 | break; | ||
| 1187 | case AMDGPU_RING_TYPE_SDMA: | ||
| 1188 | ring->funcs = adev->sdma.instance[0].ring.funcs; | ||
| 1189 | break; | ||
| 1190 | default: | ||
| 1191 | BUG(); | ||
| 1192 | } | ||
| 1193 | |||
| 1194 | r = amdgpu_ring_init(adev, ring, 1024, NULL, 0, | ||
| 1195 | AMDGPU_RING_PRIO_DEFAULT, NULL); | ||
| 1196 | if (r) { | ||
| 1197 | amdgpu_mes_unlock(&adev->mes); | ||
| 1198 | goto clean_up_memory; | ||
| 1199 | } | ||
| 1200 | |||
| 1201 | amdgpu_mes_ring_to_queue_props(adev, ring, &qprops); | ||
| 1202 | |||
| 1203 | dma_fence_wait(gang->process->vm->last_update, false); | ||
| 1204 | dma_fence_wait(ctx_data->meta_data_va->last_pt_update, false); | ||
| 1205 | amdgpu_mes_unlock(&adev->mes); | ||
| 1206 | |||
| 1207 | r = amdgpu_mes_add_hw_queue(adev, gang_id, &qprops, &queue_id); | ||
| 1208 | if (r) | ||
| 1209 | goto clean_up_ring; | ||
| 1210 | |||
| 1211 | ring->hw_queue_id = queue_id; | ||
| 1212 | ring->doorbell_index = qprops.doorbell_off; | ||
| 1213 | |||
| 1214 | if (queue_type == AMDGPU_RING_TYPE_GFX) | ||
| 1215 | sprintf(ring->name, "gfx_%d.%d.%d", pasid, gang_id, queue_id); | ||
| 1216 | else if (queue_type == AMDGPU_RING_TYPE_COMPUTE) | ||
| 1217 | sprintf(ring->name, "compute_%d.%d.%d", pasid, gang_id, | ||
| 1218 | queue_id); | ||
| 1219 | else if (queue_type == AMDGPU_RING_TYPE_SDMA) | ||
| 1220 | sprintf(ring->name, "sdma_%d.%d.%d", pasid, gang_id, | ||
| 1221 | queue_id); | ||
| 1222 | else | ||
| 1223 | BUG(); | ||
| 1224 | |||
| 1225 | *out = ring; | ||
| 1226 | return 0; | ||
| 1227 | |||
| 1228 | clean_up_ring: | ||
| 1229 | amdgpu_ring_fini(ring); | ||
| 1230 | clean_up_memory: | ||
| 1231 | kfree(ring); | ||
| 1232 | return r; | ||
| 1233 | } | ||
| 1234 | |||
| 1235 | void amdgpu_mes_remove_ring(struct amdgpu_device *adev, | ||
| 1236 | struct amdgpu_ring *ring) | ||
| 1237 | { | ||
| 1238 | if (!ring) | ||
| 1239 | return; | ||
| 1240 | |||
| 1241 | amdgpu_mes_remove_hw_queue(adev, ring->hw_queue_id); | ||
| 1242 | timer_delete_sync(&ring->fence_drv.fallback_timer); | ||
| 1243 | amdgpu_ring_fini(ring); | ||
| 1244 | kfree(ring); | ||
| 1245 | } | ||
| 1246 | |||
| 1247 | uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev, | 578 | uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev, |
| 1248 | enum amdgpu_mes_priority_level prio) | 579 | enum amdgpu_mes_priority_level prio) |
| 1249 | { | 580 | { |
| 1250 | return adev->mes.aggregated_doorbells[prio]; | 581 | return adev->mes.aggregated_doorbells[prio]; |
| 1251 | } | 582 | } |
| 1252 | 583 | ||
| 1253 | int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev, | ||
| 1254 | struct amdgpu_mes_ctx_data *ctx_data) | ||
| 1255 | { | ||
| 1256 | int r; | ||
| 1257 | |||
| 1258 | r = amdgpu_bo_create_kernel(adev, | ||
| 1259 | sizeof(struct amdgpu_mes_ctx_meta_data), | ||
| 1260 | PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT, | ||
| 1261 | &ctx_data->meta_data_obj, | ||
| 1262 | &ctx_data->meta_data_mc_addr, | ||
| 1263 | &ctx_data->meta_data_ptr); | ||
| 1264 | if (r) { | ||
| 1265 | dev_warn(adev->dev, "(%d) create CTX bo failed\n", r); | ||
| 1266 | return r; | ||
| 1267 | } | ||
| 1268 | |||
| 1269 | if (!ctx_data->meta_data_obj) | ||
| 1270 | return -ENOMEM; | ||
| 1271 | |||
| 1272 | memset(ctx_data->meta_data_ptr, 0, | ||
| 1273 | sizeof(struct amdgpu_mes_ctx_meta_data)); | ||
| 1274 | |||
| 1275 | return 0; | ||
| 1276 | } | ||
| 1277 | |||
| 1278 | void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data) | ||
| 1279 | { | ||
| 1280 | if (ctx_data->meta_data_obj) | ||
| 1281 | amdgpu_bo_free_kernel(&ctx_data->meta_data_obj, | ||
| 1282 | &ctx_data->meta_data_mc_addr, | ||
| 1283 | &ctx_data->meta_data_ptr); | ||
| 1284 | } | ||
| 1285 | |||
| 1286 | int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev, | ||
| 1287 | struct amdgpu_vm *vm, | ||
| 1288 | struct amdgpu_mes_ctx_data *ctx_data) | ||
| 1289 | { | ||
| 1290 | struct amdgpu_bo_va *bo_va; | ||
| 1291 | struct amdgpu_sync sync; | ||
| 1292 | struct drm_exec exec; | ||
| 1293 | int r; | ||
| 1294 | |||
| 1295 | amdgpu_sync_create(&sync); | ||
| 1296 | |||
| 1297 | drm_exec_init(&exec, 0, 0); | ||
| 1298 | drm_exec_until_all_locked(&exec) { | ||
| 1299 | r = drm_exec_lock_obj(&exec, | ||
| 1300 | &ctx_data->meta_data_obj->tbo.base); | ||
| 1301 | drm_exec_retry_on_contention(&exec); | ||
| 1302 | if (unlikely(r)) | ||
| 1303 | goto error_fini_exec; | ||
| 1304 | |||
| 1305 | r = amdgpu_vm_lock_pd(vm, &exec, 0); | ||
| 1306 | drm_exec_retry_on_contention(&exec); | ||
| 1307 | if (unlikely(r)) | ||
| 1308 | goto error_fini_exec; | ||
| 1309 | } | ||
| 1310 | |||
| 1311 | bo_va = amdgpu_vm_bo_add(adev, vm, ctx_data->meta_data_obj); | ||
| 1312 | if (!bo_va) { | ||
| 1313 | DRM_ERROR("failed to create bo_va for meta data BO\n"); | ||
| 1314 | r = -ENOMEM; | ||
| 1315 | goto error_fini_exec; | ||
| 1316 | } | ||
| 1317 | |||
| 1318 | r = amdgpu_vm_bo_map(adev, bo_va, ctx_data->meta_data_gpu_addr, 0, | ||
| 1319 | sizeof(struct amdgpu_mes_ctx_meta_data), | ||
| 1320 | AMDGPU_PTE_READABLE | AMDGPU_PTE_WRITEABLE | | ||
| 1321 | AMDGPU_PTE_EXECUTABLE); | ||
| 1322 | |||
| 1323 | if (r) { | ||
| 1324 | DRM_ERROR("failed to do bo_map on meta data, err=%d\n", r); | ||
| 1325 | goto error_del_bo_va; | ||
| 1326 | } | ||
| 1327 | |||
| 1328 | r = amdgpu_vm_bo_update(adev, bo_va, false); | ||
| 1329 | if (r) { | ||
| 1330 | DRM_ERROR("failed to do vm_bo_update on meta data\n"); | ||
| 1331 | goto error_del_bo_va; | ||
| 1332 | } | ||
| 1333 | amdgpu_sync_fence(&sync, bo_va->last_pt_update, GFP_KERNEL); | ||
| 1334 | |||
| 1335 | r = amdgpu_vm_update_pdes(adev, vm, false); | ||
| 1336 | if (r) { | ||
| 1337 | DRM_ERROR("failed to update pdes on meta data\n"); | ||
| 1338 | goto error_del_bo_va; | ||
| 1339 | } | ||
| 1340 | amdgpu_sync_fence(&sync, vm->last_update, GFP_KERNEL); | ||
| 1341 | |||
| 1342 | amdgpu_sync_wait(&sync, false); | ||
| 1343 | drm_exec_fini(&exec); | ||
| 1344 | |||
| 1345 | amdgpu_sync_free(&sync); | ||
| 1346 | ctx_data->meta_data_va = bo_va; | ||
| 1347 | return 0; | ||
| 1348 | |||
| 1349 | error_del_bo_va: | ||
| 1350 | amdgpu_vm_bo_del(adev, bo_va); | ||
| 1351 | |||
| 1352 | error_fini_exec: | ||
| 1353 | drm_exec_fini(&exec); | ||
| 1354 | amdgpu_sync_free(&sync); | ||
| 1355 | return r; | ||
| 1356 | } | ||
| 1357 | |||
| 1358 | int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev, | ||
| 1359 | struct amdgpu_mes_ctx_data *ctx_data) | ||
| 1360 | { | ||
| 1361 | struct amdgpu_bo_va *bo_va = ctx_data->meta_data_va; | ||
| 1362 | struct amdgpu_bo *bo = ctx_data->meta_data_obj; | ||
| 1363 | struct amdgpu_vm *vm = bo_va->base.vm; | ||
| 1364 | struct dma_fence *fence; | ||
| 1365 | struct drm_exec exec; | ||
| 1366 | long r; | ||
| 1367 | |||
| 1368 | drm_exec_init(&exec, 0, 0); | ||
| 1369 | drm_exec_until_all_locked(&exec) { | ||
| 1370 | r = drm_exec_lock_obj(&exec, | ||
| 1371 | &ctx_data->meta_data_obj->tbo.base); | ||
| 1372 | drm_exec_retry_on_contention(&exec); | ||
| 1373 | if (unlikely(r)) | ||
| 1374 | goto out_unlock; | ||
| 1375 | |||
| 1376 | r = amdgpu_vm_lock_pd(vm, &exec, 0); | ||
| 1377 | drm_exec_retry_on_contention(&exec); | ||
| 1378 | if (unlikely(r)) | ||
| 1379 | goto out_unlock; | ||
| 1380 | } | ||
| 1381 | |||
| 1382 | amdgpu_vm_bo_del(adev, bo_va); | ||
| 1383 | if (!amdgpu_vm_ready(vm)) | ||
| 1384 | goto out_unlock; | ||
| 1385 | |||
| 1386 | r = dma_resv_get_singleton(bo->tbo.base.resv, DMA_RESV_USAGE_BOOKKEEP, | ||
| 1387 | &fence); | ||
| 1388 | if (r) | ||
| 1389 | goto out_unlock; | ||
| 1390 | if (fence) { | ||
| 1391 | amdgpu_bo_fence(bo, fence, true); | ||
| 1392 | fence = NULL; | ||
| 1393 | } | ||
| 1394 | |||
| 1395 | r = amdgpu_vm_clear_freed(adev, vm, &fence); | ||
| 1396 | if (r || !fence) | ||
| 1397 | goto out_unlock; | ||
| 1398 | |||
| 1399 | dma_fence_wait(fence, false); | ||
| 1400 | amdgpu_bo_fence(bo, fence, true); | ||
| 1401 | dma_fence_put(fence); | ||
| 1402 | |||
| 1403 | out_unlock: | ||
| 1404 | if (unlikely(r < 0)) | ||
| 1405 | dev_err(adev->dev, "failed to clear page tables (%ld)\n", r); | ||
| 1406 | drm_exec_fini(&exec); | ||
| 1407 | |||
| 1408 | return r; | ||
| 1409 | } | ||
| 1410 | |||
| 1411 | static int amdgpu_mes_test_create_gang_and_queues(struct amdgpu_device *adev, | ||
| 1412 | int pasid, int *gang_id, | ||
| 1413 | int queue_type, int num_queue, | ||
| 1414 | struct amdgpu_ring **added_rings, | ||
| 1415 | struct amdgpu_mes_ctx_data *ctx_data) | ||
| 1416 | { | ||
| 1417 | struct amdgpu_ring *ring; | ||
| 1418 | struct amdgpu_mes_gang_properties gprops = {0}; | ||
| 1419 | int r, j; | ||
| 1420 | |||
| 1421 | /* create a gang for the process */ | ||
| 1422 | gprops.priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL; | ||
| 1423 | gprops.gang_quantum = adev->mes.default_gang_quantum; | ||
| 1424 | gprops.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL; | ||
| 1425 | gprops.priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL; | ||
| 1426 | gprops.global_priority_level = AMDGPU_MES_PRIORITY_LEVEL_NORMAL; | ||
| 1427 | |||
| 1428 | r = amdgpu_mes_add_gang(adev, pasid, &gprops, gang_id); | ||
| 1429 | if (r) { | ||
| 1430 | DRM_ERROR("failed to add gang\n"); | ||
| 1431 | return r; | ||
| 1432 | } | ||
| 1433 | |||
| 1434 | /* create queues for the gang */ | ||
| 1435 | for (j = 0; j < num_queue; j++) { | ||
| 1436 | r = amdgpu_mes_add_ring(adev, *gang_id, queue_type, j, | ||
| 1437 | ctx_data, &ring); | ||
| 1438 | if (r) { | ||
| 1439 | DRM_ERROR("failed to add ring\n"); | ||
| 1440 | break; | ||
| 1441 | } | ||
| 1442 | |||
| 1443 | DRM_INFO("ring %s was added\n", ring->name); | ||
| 1444 | added_rings[j] = ring; | ||
| 1445 | } | ||
| 1446 | |||
| 1447 | return 0; | ||
| 1448 | } | ||
| 1449 | |||
| 1450 | static int amdgpu_mes_test_queues(struct amdgpu_ring **added_rings) | ||
| 1451 | { | ||
| 1452 | struct amdgpu_ring *ring; | ||
| 1453 | int i, r; | ||
| 1454 | |||
| 1455 | for (i = 0; i < AMDGPU_MES_CTX_MAX_RINGS; i++) { | ||
| 1456 | ring = added_rings[i]; | ||
| 1457 | if (!ring) | ||
| 1458 | continue; | ||
| 1459 | |||
| 1460 | r = amdgpu_ring_test_helper(ring); | ||
| 1461 | if (r) | ||
| 1462 | return r; | ||
| 1463 | |||
| 1464 | r = amdgpu_ring_test_ib(ring, 1000 * 10); | ||
| 1465 | if (r) { | ||
| 1466 | DRM_DEV_ERROR(ring->adev->dev, | ||
| 1467 | "ring %s ib test failed (%d)\n", | ||
| 1468 | ring->name, r); | ||
| 1469 | return r; | ||
| 1470 | } else | ||
| 1471 | DRM_INFO("ring %s ib test pass\n", ring->name); | ||
| 1472 | } | ||
| 1473 | |||
| 1474 | return 0; | ||
| 1475 | } | ||
| 1476 | |||
| 1477 | int amdgpu_mes_self_test(struct amdgpu_device *adev) | ||
| 1478 | { | ||
| 1479 | struct amdgpu_vm *vm = NULL; | ||
| 1480 | struct amdgpu_mes_ctx_data ctx_data = {0}; | ||
| 1481 | struct amdgpu_ring *added_rings[AMDGPU_MES_CTX_MAX_RINGS] = { NULL }; | ||
| 1482 | int gang_ids[3] = {0}; | ||
| 1483 | int queue_types[][2] = { { AMDGPU_RING_TYPE_GFX, 1 }, | ||
| 1484 | { AMDGPU_RING_TYPE_COMPUTE, 1 }, | ||
| 1485 | { AMDGPU_RING_TYPE_SDMA, 1} }; | ||
| 1486 | int i, r, pasid, k = 0; | ||
| 1487 | |||
| 1488 | pasid = amdgpu_pasid_alloc(16); | ||
| 1489 | if (pasid < 0) { | ||
| 1490 | dev_warn(adev->dev, "No more PASIDs available!"); | ||
| 1491 | pasid = 0; | ||
| 1492 | } | ||
| 1493 | |||
| 1494 | vm = kzalloc(sizeof(*vm), GFP_KERNEL); | ||
| 1495 | if (!vm) { | ||
| 1496 | r = -ENOMEM; | ||
| 1497 | goto error_pasid; | ||
| 1498 | } | ||
| 1499 | |||
| 1500 | r = amdgpu_vm_init(adev, vm, -1); | ||
| 1501 | if (r) { | ||
| 1502 | DRM_ERROR("failed to initialize vm\n"); | ||
| 1503 | goto error_pasid; | ||
| 1504 | } | ||
| 1505 | |||
| 1506 | r = amdgpu_mes_ctx_alloc_meta_data(adev, &ctx_data); | ||
| 1507 | if (r) { | ||
| 1508 | DRM_ERROR("failed to alloc ctx meta data\n"); | ||
| 1509 | goto error_fini; | ||
| 1510 | } | ||
| 1511 | |||
| 1512 | ctx_data.meta_data_gpu_addr = AMDGPU_VA_RESERVED_BOTTOM; | ||
| 1513 | r = amdgpu_mes_ctx_map_meta_data(adev, vm, &ctx_data); | ||
| 1514 | if (r) { | ||
| 1515 | DRM_ERROR("failed to map ctx meta data\n"); | ||
| 1516 | goto error_vm; | ||
| 1517 | } | ||
| 1518 | |||
| 1519 | r = amdgpu_mes_create_process(adev, pasid, vm); | ||
| 1520 | if (r) { | ||
| 1521 | DRM_ERROR("failed to create MES process\n"); | ||
| 1522 | goto error_vm; | ||
| 1523 | } | ||
| 1524 | |||
| 1525 | for (i = 0; i < ARRAY_SIZE(queue_types); i++) { | ||
| 1526 | /* On GFX v10.3, fw hasn't supported to map sdma queue. */ | ||
| 1527 | if (amdgpu_ip_version(adev, GC_HWIP, 0) >= | ||
| 1528 | IP_VERSION(10, 3, 0) && | ||
| 1529 | amdgpu_ip_version(adev, GC_HWIP, 0) < | ||
| 1530 | IP_VERSION(11, 0, 0) && | ||
| 1531 | queue_types[i][0] == AMDGPU_RING_TYPE_SDMA) | ||
| 1532 | continue; | ||
| 1533 | |||
| 1534 | r = amdgpu_mes_test_create_gang_and_queues(adev, pasid, | ||
| 1535 | &gang_ids[i], | ||
| 1536 | queue_types[i][0], | ||
| 1537 | queue_types[i][1], | ||
| 1538 | &added_rings[k], | ||
| 1539 | &ctx_data); | ||
| 1540 | if (r) | ||
| 1541 | goto error_queues; | ||
| 1542 | |||
| 1543 | k += queue_types[i][1]; | ||
| 1544 | } | ||
| 1545 | |||
| 1546 | /* start ring test and ib test for MES queues */ | ||
| 1547 | amdgpu_mes_test_queues(added_rings); | ||
| 1548 | |||
| 1549 | error_queues: | ||
| 1550 | /* remove all queues */ | ||
| 1551 | for (i = 0; i < ARRAY_SIZE(added_rings); i++) { | ||
| 1552 | if (!added_rings[i]) | ||
| 1553 | continue; | ||
| 1554 | amdgpu_mes_remove_ring(adev, added_rings[i]); | ||
| 1555 | } | ||
| 1556 | |||
| 1557 | for (i = 0; i < ARRAY_SIZE(gang_ids); i++) { | ||
| 1558 | if (!gang_ids[i]) | ||
| 1559 | continue; | ||
| 1560 | amdgpu_mes_remove_gang(adev, gang_ids[i]); | ||
| 1561 | } | ||
| 1562 | |||
| 1563 | amdgpu_mes_destroy_process(adev, pasid); | ||
| 1564 | |||
| 1565 | error_vm: | ||
| 1566 | amdgpu_mes_ctx_unmap_meta_data(adev, &ctx_data); | ||
| 1567 | |||
| 1568 | error_fini: | ||
| 1569 | amdgpu_vm_fini(adev, vm); | ||
| 1570 | |||
| 1571 | error_pasid: | ||
| 1572 | if (pasid) | ||
| 1573 | amdgpu_pasid_free(pasid); | ||
| 1574 | |||
| 1575 | amdgpu_mes_ctx_free_meta_data(&ctx_data); | ||
| 1576 | kfree(vm); | ||
| 1577 | return 0; | ||
| 1578 | } | ||
| 1579 | |||
| 1580 | int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe) | 584 | int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe) |
| 1581 | { | 585 | { |
| 1582 | const struct mes_firmware_header_v1_0 *mes_hdr; | 586 | const struct mes_firmware_header_v1_0 *mes_hdr; |
| @@ -1705,7 +709,7 @@ int amdgpu_mes_update_enforce_isolation(struct amdgpu_device *adev) · 122 unmodified lines | |||
| 1705 | if (adev->enable_mes && adev->gfx.enable_cleaner_shader) { | 709 | if (adev->enable_mes && adev->gfx.enable_cleaner_shader) { |
| 1706 | mutex_lock(&adev->enforce_isolation_mutex); | 710 | mutex_lock(&adev->enforce_isolation_mutex); |
| 1707 | for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) { | 711 | for (i = 0; i < (adev->xcp_mgr ? adev->xcp_mgr->num_xcps : 1); i++) { |
| 1708 | if (adev->enforce_isolation[i]) | 712 | if (adev->enforce_isolation[i] == AMDGPU_ENFORCE_ISOLATION_ENABLE) |
| 1709 | r |= amdgpu_mes_set_enforce_isolation(adev, i, true); | 713 | r |= amdgpu_mes_set_enforce_isolation(adev, i, true); |
| 1710 | else | 714 | else |
| 1711 | r |= amdgpu_mes_set_enforce_isolation(adev, i, false); | 715 | r |= amdgpu_mes_set_enforce_isolation(adev, i, false); |
| @@ -111,8 +111,8 @@ struct amdgpu_mes { · 110 unmodified lines | |||
| 111 | 111 | ||
| 112 | uint32_t vmid_mask_gfxhub; | 112 | uint32_t vmid_mask_gfxhub; |
| 113 | uint32_t vmid_mask_mmhub; | 113 | uint32_t vmid_mask_mmhub; |
| 114 | uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES]; | ||
| 115 | uint32_t gfx_hqd_mask[AMDGPU_MES_MAX_GFX_PIPES]; | 114 | uint32_t gfx_hqd_mask[AMDGPU_MES_MAX_GFX_PIPES]; |
| 115 | uint32_t compute_hqd_mask[AMDGPU_MES_MAX_COMPUTE_PIPES]; | ||
| 116 | uint32_t sdma_hqd_mask[AMDGPU_MES_MAX_SDMA_PIPES]; | 116 | uint32_t sdma_hqd_mask[AMDGPU_MES_MAX_SDMA_PIPES]; |
| 117 | uint32_t aggregated_doorbells[AMDGPU_MES_PRIORITY_NUM_LEVELS]; | 117 | uint32_t aggregated_doorbells[AMDGPU_MES_PRIORITY_NUM_LEVELS]; |
| 118 | uint32_t sch_ctx_offs[AMDGPU_MAX_MES_PIPES]; | 118 | uint32_t sch_ctx_offs[AMDGPU_MAX_MES_PIPES]; |
| @@ -149,19 +149,6 @@ struct amdgpu_mes { · 30 unmodified lines | |||
| 149 | 149 | ||
| 150 | }; | 150 | }; |
| 151 | 151 | ||
| 152 | struct amdgpu_mes_process { | ||
| 153 | int pasid; | ||
| 154 | struct amdgpu_vm *vm; | ||
| 155 | uint64_t pd_gpu_addr; | ||
| 156 | struct amdgpu_bo *proc_ctx_bo; | ||
| 157 | uint64_t proc_ctx_gpu_addr; | ||
| 158 | void *proc_ctx_cpu_ptr; | ||
| 159 | uint64_t process_quantum; | ||
| 160 | struct list_head gang_list; | ||
| 161 | uint32_t doorbell_index; | ||
| 162 | struct mutex doorbell_lock; | ||
| 163 | }; | ||
| 164 | |||
| 165 | struct amdgpu_mes_gang { | 152 | struct amdgpu_mes_gang { |
| 166 | int gang_id; | 153 | int gang_id; |
| 167 | int priority; | 154 | int priority; |
| @@ -248,18 +235,6 @@ struct mes_remove_queue_input { · 80 unmodified lines | |||
| 248 | uint64_t gang_context_addr; | 235 | uint64_t gang_context_addr; |
| 249 | }; | 236 | }; |
| 250 | 237 | ||
| 251 | struct mes_reset_queue_input { | ||
| 252 | uint32_t doorbell_offset; | ||
| 253 | uint64_t gang_context_addr; | ||
| 254 | bool use_mmio; | ||
| 255 | uint32_t queue_type; | ||
| 256 | uint32_t me_id; | ||
| 257 | uint32_t pipe_id; | ||
| 258 | uint32_t queue_id; | ||
| 259 | uint32_t xcc_id; | ||
| 260 | uint32_t vmid; | ||
| 261 | }; | ||
| 262 | |||
| 263 | struct mes_map_legacy_queue_input { | 238 | struct mes_map_legacy_queue_input { |
| 264 | uint32_t queue_type; | 239 | uint32_t queue_type; |
| 265 | uint32_t doorbell_offset; | 240 | uint32_t doorbell_offset; |
| @@ -291,7 +266,7 @@ struct mes_resume_gang_input { · 25 unmodified lines | |||
| 291 | uint64_t gang_context_addr; | 266 | uint64_t gang_context_addr; |
| 292 | }; | 267 | }; |
| 293 | 268 | ||
| 294 | struct mes_reset_legacy_queue_input { | 269 | struct mes_reset_queue_input { |
| 295 | uint32_t queue_type; | 270 | uint32_t queue_type; |
| 296 | uint32_t doorbell_offset; | 271 | uint32_t doorbell_offset; |
| 297 | bool use_mmio; | 272 | bool use_mmio; |
| @@ -301,6 +276,8 @@ struct mes_reset_legacy_queue_input { · 3 unmodified lines | |||
| 301 | uint64_t mqd_addr; | 276 | uint64_t mqd_addr; |
| 302 | uint64_t wptr_addr; | 277 | uint64_t wptr_addr; |
| 303 | uint32_t vmid; | 278 | uint32_t vmid; |
| 279 | bool legacy_gfx; | ||
| 280 | bool is_kq; | ||
| 304 | }; | 281 | }; |
| 305 | 282 | ||
| 306 | enum mes_misc_opcode { | 283 | enum mes_misc_opcode { |
| @@ -388,9 +365,6 @@ struct amdgpu_mes_funcs { · 81 unmodified lines | |||
| 388 | int (*misc_op)(struct amdgpu_mes *mes, | 365 | int (*misc_op)(struct amdgpu_mes *mes, |
| 389 | struct mes_misc_op_input *input); | 366 | struct mes_misc_op_input *input); |
| 390 | 367 | ||
| 391 | int (*reset_legacy_queue)(struct amdgpu_mes *mes, | ||
| 392 | struct mes_reset_legacy_queue_input *input); | ||
| 393 | |||
| 394 | int (*reset_hw_queue)(struct amdgpu_mes *mes, | 368 | int (*reset_hw_queue)(struct amdgpu_mes *mes, |
| 395 | struct mes_reset_queue_input *input); | 369 | struct mes_reset_queue_input *input); |
| 396 | }; | 370 | }; |
| @@ -404,26 +378,9 @@ int amdgpu_mes_init_microcode(struct amdgpu_device *adev, int pipe); · 7 unmodified lines | |||
| 404 | int amdgpu_mes_init(struct amdgpu_device *adev); | 378 | int amdgpu_mes_init(struct amdgpu_device *adev); |
| 405 | void amdgpu_mes_fini(struct amdgpu_device *adev); | 379 | void amdgpu_mes_fini(struct amdgpu_device *adev); |
| 406 | 380 | ||
| 407 | int amdgpu_mes_create_process(struct amdgpu_device *adev, int pasid, | ||
| 408 | struct amdgpu_vm *vm); | ||
| 409 | void amdgpu_mes_destroy_process(struct amdgpu_device *adev, int pasid); | ||
| 410 | |||
| 411 | int amdgpu_mes_add_gang(struct amdgpu_device *adev, int pasid, | ||
| 412 | struct amdgpu_mes_gang_properties *gprops, | ||
| 413 | int *gang_id); | ||
| 414 | int amdgpu_mes_remove_gang(struct amdgpu_device *adev, int gang_id); | ||
| 415 | |||
| 416 | int amdgpu_mes_suspend(struct amdgpu_device *adev); | 381 | int amdgpu_mes_suspend(struct amdgpu_device *adev); |
| 417 | int amdgpu_mes_resume(struct amdgpu_device *adev); | 382 | int amdgpu_mes_resume(struct amdgpu_device *adev); |
| 418 | 383 | ||
| 419 | int amdgpu_mes_add_hw_queue(struct amdgpu_device *adev, int gang_id, | ||
| 420 | struct amdgpu_mes_queue_properties *qprops, | ||
| 421 | int *queue_id); | ||
| 422 | int amdgpu_mes_remove_hw_queue(struct amdgpu_device *adev, int queue_id); | ||
| 423 | int amdgpu_mes_reset_hw_queue(struct amdgpu_device *adev, int queue_id); | ||
| 424 | int amdgpu_mes_reset_hw_queue_mmio(struct amdgpu_device *adev, int queue_type, | ||
| 425 | int me_id, int pipe_id, int queue_id, int vmid); | ||
| 426 | |||
| 427 | int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev, | 384 | int amdgpu_mes_map_legacy_queue(struct amdgpu_device *adev, |
| 428 | struct amdgpu_ring *ring); | 385 | struct amdgpu_ring *ring); |
| 429 | int amdgpu_mes_unmap_legacy_queue(struct amdgpu_device *adev, | 386 | int amdgpu_mes_unmap_legacy_queue(struct amdgpu_device *adev, |
| @@ -451,27 +408,10 @@ int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, · 21 unmodified lines | |||
| 451 | bool trap_en); | 408 | bool trap_en); |
| 452 | int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev, | 409 | int amdgpu_mes_flush_shader_debugger(struct amdgpu_device *adev, |
| 453 | uint64_t process_context_addr); | 410 | uint64_t process_context_addr); |
| 454 | int amdgpu_mes_add_ring(struct amdgpu_device *adev, int gang_id, | ||
| 455 | int queue_type, int idx, | ||
| 456 | struct amdgpu_mes_ctx_data *ctx_data, | ||
| 457 | struct amdgpu_ring **out); | ||
| 458 | void amdgpu_mes_remove_ring(struct amdgpu_device *adev, | ||
| 459 | struct amdgpu_ring *ring); | ||
| 460 | 411 | ||
| 461 | uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev, | 412 | uint32_t amdgpu_mes_get_aggregated_doorbell_index(struct amdgpu_device *adev, |
| 462 | enum amdgpu_mes_priority_level prio); | 413 | enum amdgpu_mes_priority_level prio); |
| 463 | 414 | ||
| 464 | int amdgpu_mes_ctx_alloc_meta_data(struct amdgpu_device *adev, | ||
| 465 | struct amdgpu_mes_ctx_data *ctx_data); | ||
| 466 | void amdgpu_mes_ctx_free_meta_data(struct amdgpu_mes_ctx_data *ctx_data); | ||
| 467 | int amdgpu_mes_ctx_map_meta_data(struct amdgpu_device *adev, | ||
| 468 | struct amdgpu_vm *vm, | ||
| 469 | struct amdgpu_mes_ctx_data *ctx_data); | ||
| 470 | int amdgpu_mes_ctx_unmap_meta_data(struct amdgpu_device *adev, | ||
| 471 | struct amdgpu_mes_ctx_data *ctx_data); | ||
| 472 | |||
| 473 | int amdgpu_mes_self_test(struct amdgpu_device *adev); | ||
| 474 | |||
| 475 | int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev); | 415 | int amdgpu_mes_doorbell_process_slice(struct amdgpu_device *adev); |
| 476 | 416 | ||
| 477 | /* | 417 | /* |
| @@ -1644,7 +1644,11 @@ u64 amdgpu_bo_print_info(int id, struct amdgpu_bo *bo, struct seq_file *m) · 1643 unmodified lines | |||
| 1644 | amdgpu_bo_print_flag(m, bo, VRAM_CONTIGUOUS); | 1644 | amdgpu_bo_print_flag(m, bo, VRAM_CONTIGUOUS); |
| 1645 | amdgpu_bo_print_flag(m, bo, VM_ALWAYS_VALID); | 1645 | amdgpu_bo_print_flag(m, bo, VM_ALWAYS_VALID); |
| 1646 | amdgpu_bo_print_flag(m, bo, EXPLICIT_SYNC); | 1646 | amdgpu_bo_print_flag(m, bo, EXPLICIT_SYNC); |
| 1647 | 1647 | /* Add the gem obj resv fence dump*/ | |
| 1648 | if (dma_resv_trylock(bo->tbo.base.resv)) { | ||
| 1649 | dma_resv_describe(bo->tbo.base.resv, m); | ||
| 1650 | dma_resv_unlock(bo->tbo.base.resv); | ||
| 1651 | } | ||
| 1648 | seq_puts(m, "\n"); | 1652 | seq_puts(m, "\n"); |
| 1649 | 1653 | ||
| 1650 | return size; | 1654 | return size; |
| @@ -2214,7 +2214,8 @@ static int psp_securedisplay_initialize(struct psp_context *psp) · 2213 unmodified lines | |||
| 2214 | 2214 | ||
| 2215 | if (!psp->securedisplay_context.context.bin_desc.size_bytes || | 2215 | if (!psp->securedisplay_context.context.bin_desc.size_bytes || |
| 2216 | !psp->securedisplay_context.context.bin_desc.start_addr) { | 2216 | !psp->securedisplay_context.context.bin_desc.start_addr) { |
| 2217 | dev_info(psp->adev->dev, "SECUREDISPLAY: securedisplay ta ucode is not available\n"); | 2217 | dev_info(psp->adev->dev, |
| 2218 | "SECUREDISPLAY: optional securedisplay ta ucode is not available\n"); | ||
| 2218 | return 0; | 2219 | return 0; |
| 2219 | } | 2220 | } |
| 2220 | 2221 | ||
| @@ -107,6 +107,7 @@ enum psp_reg_prog_id { · 106 unmodified lines | |||
| 107 | PSP_REG_IH_RB_CNTL = 0, /* register IH_RB_CNTL */ | 107 | PSP_REG_IH_RB_CNTL = 0, /* register IH_RB_CNTL */ |
| 108 | PSP_REG_IH_RB_CNTL_RING1 = 1, /* register IH_RB_CNTL_RING1 */ | 108 | PSP_REG_IH_RB_CNTL_RING1 = 1, /* register IH_RB_CNTL_RING1 */ |
| 109 | PSP_REG_IH_RB_CNTL_RING2 = 2, /* register IH_RB_CNTL_RING2 */ | 109 | PSP_REG_IH_RB_CNTL_RING2 = 2, /* register IH_RB_CNTL_RING2 */ |
| 110 | PSP_REG_MMHUB_L1_TLB_CNTL = 25, | ||
| 110 | PSP_REG_LAST | 111 | PSP_REG_LAST |
| 111 | }; | 112 | }; |
| 112 | 113 | ||
| @@ -142,6 +143,8 @@ struct psp_funcs { · 29 unmodified lines | |||
| 142 | bool (*get_ras_capability)(struct psp_context *psp); | 143 | bool (*get_ras_capability)(struct psp_context *psp); |
| 143 | bool (*is_aux_sos_load_required)(struct psp_context *psp); | 144 | bool (*is_aux_sos_load_required)(struct psp_context *psp); |
| 144 | bool (*is_reload_needed)(struct psp_context *psp); | 145 | bool (*is_reload_needed)(struct psp_context *psp); |
| 146 | int (*reg_program_no_ring)(struct psp_context *psp, uint32_t val, | ||
| 147 | enum psp_reg_prog_id id); | ||
| 145 | }; | 148 | }; |
| 146 | 149 | ||
| 147 | struct ta_funcs { | 150 | struct ta_funcs { |
| @@ -475,6 +478,10 @@ struct amdgpu_psp_funcs { · 327 unmodified lines | |||
| 475 | #define psp_is_aux_sos_load_required(psp) \ | 478 | #define psp_is_aux_sos_load_required(psp) \ |
| 476 | ((psp)->funcs->is_aux_sos_load_required ? (psp)->funcs->is_aux_sos_load_required((psp)) : 0) | 479 | ((psp)->funcs->is_aux_sos_load_required ? (psp)->funcs->is_aux_sos_load_required((psp)) : 0) |
| 477 | 480 | ||
| 481 | #define psp_reg_program_no_ring(psp, val, id) \ | ||
| 482 | ((psp)->funcs->reg_program_no_ring ? \ | ||
| 483 | (psp)->funcs->reg_program_no_ring((psp), val, id) : -EINVAL) | ||
| 484 | |||
| 478 | extern const struct amd_ip_funcs psp_ip_funcs; | 485 | extern const struct amd_ip_funcs psp_ip_funcs; |
| 479 | 486 | ||
| 480 | extern const struct amdgpu_ip_block_version psp_v3_1_ip_block; | 487 | extern const struct amdgpu_ip_block_version psp_v3_1_ip_block; |
| @@ -569,5 +576,8 @@ bool amdgpu_psp_get_ras_capability(struct psp_context *psp); · 88 unmodified lines | |||
| 569 | int psp_config_sq_perfmon(struct psp_context *psp, uint32_t xcp_id, | 576 | int psp_config_sq_perfmon(struct psp_context *psp, uint32_t xcp_id, |
| 570 | bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable); | 577 | bool core_override_enable, bool reg_override_enable, bool perfmon_override_enable); |
| 571 | bool amdgpu_psp_tos_reload_needed(struct amdgpu_device *adev); | 578 | bool amdgpu_psp_tos_reload_needed(struct amdgpu_device *adev); |
| 579 | int amdgpu_psp_reg_program_no_ring(struct psp_context *psp, uint32_t val, | ||
| 580 | enum psp_reg_prog_id id); | ||
| 581 | |||
| 572 | 582 | ||
| 573 | #endif | 583 | #endif |
| @@ -1498,6 +1498,9 @@ int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, · 1497 unmodified lines | |||
| 1498 | !amdgpu_ras_get_aca_debug_mode(adev)) | 1498 | !amdgpu_ras_get_aca_debug_mode(adev)) |
| 1499 | return -EOPNOTSUPP; | 1499 | return -EOPNOTSUPP; |
| 1500 | 1500 | ||
| 1501 | if (amdgpu_sriov_vf(adev)) | ||
| 1502 | return -EOPNOTSUPP; | ||
| 1503 | |||
| 1501 | /* skip ras error reset in gpu reset */ | 1504 | /* skip ras error reset in gpu reset */ |
| 1502 | if ((amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) && | 1505 | if ((amdgpu_in_reset(adev) || amdgpu_ras_in_recovery(adev)) && |
| 1503 | ((smu_funcs && smu_funcs->set_debug_mode) || | 1506 | ((smu_funcs && smu_funcs->set_debug_mode) || |
| @@ -2161,7 +2164,7 @@ void amdgpu_ras_interrupt_fatal_error_handler(struct amdgpu_device *adev) · 657 unmodified lines | |||
| 2161 | /* Fatal error events are handled on host side */ | 2164 | /* Fatal error events are handled on host side */ |
| 2162 | if (amdgpu_sriov_vf(adev)) | 2165 | if (amdgpu_sriov_vf(adev)) |
| 2163 | return; | 2166 | return; |
| 2164 | /** | 2167 | /* |
| 2165 | * If the current interrupt is caused by a non-fatal RAS error, skip | 2168 | * If the current interrupt is caused by a non-fatal RAS error, skip |
| 2166 | * check for fatal error. For fatal errors, FED status of all devices | 2169 | * check for fatal error. For fatal errors, FED status of all devices |
| 2167 | * in XGMI hive gets set when the first device gets fatal error | 2170 | * in XGMI hive gets set when the first device gets fatal error |
| @@ -3793,10 +3796,12 @@ init_ras_enabled_flag: · 1625 unmodified lines | |||
| 3793 | adev->ras_hw_enabled & amdgpu_ras_mask; | 3796 | adev->ras_hw_enabled & amdgpu_ras_mask; |
| 3794 | 3797 | ||
| 3795 | /* aca is disabled by default except for psp v13_0_6/v13_0_12/v13_0_14 */ | 3798 | /* aca is disabled by default except for psp v13_0_6/v13_0_12/v13_0_14 */ |
| 3796 | adev->aca.is_enabled = | 3799 | if (!amdgpu_sriov_vf(adev)) { |
| 3797 | (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) || | 3800 | adev->aca.is_enabled = |
| 3798 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) || | 3801 | (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) || |
| 3799 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)); | 3802 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) || |
| 3803 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)); | ||
| 3804 | } | ||
| 3800 | 3805 | ||
| 3801 | /* bad page feature is not applicable to specific app platform */ | 3806 | /* bad page feature is not applicable to specific app platform */ |
| 3802 | if (adev->gmc.is_app_apu && | 3807 | if (adev->gmc.is_app_apu && |
| @@ -418,6 +418,7 @@ static void amdgpu_ras_set_eeprom_table_version(struct amdgpu_ras_eeprom_control · 417 unmodified lines | |||
| 418 | hdr->version = RAS_TABLE_VER_V2_1; | 418 | hdr->version = RAS_TABLE_VER_V2_1; |
| 419 | return; | 419 | return; |
| 420 | case IP_VERSION(12, 0, 0): | 420 | case IP_VERSION(12, 0, 0): |
| 421 | case IP_VERSION(12, 5, 0): | ||
| 421 | hdr->version = RAS_TABLE_VER_V3; | 422 | hdr->version = RAS_TABLE_VER_V3; |
| 422 | return; | 423 | return; |
| 423 | default: | 424 | default: |
| @@ -1392,17 +1393,39 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 968 unmodified lines | |||
| 1392 | 1393 | ||
| 1393 | __decode_table_header_from_buf(hdr, buf); | 1394 | __decode_table_header_from_buf(hdr, buf); |
| 1394 | 1395 | ||
| 1395 | if (hdr->version >= RAS_TABLE_VER_V2_1) { | 1396 | if (hdr->header != RAS_TABLE_HDR_VAL && |
| 1397 | hdr->header != RAS_TABLE_HDR_BAD) { | ||
| 1398 | dev_info(adev->dev, "Creating a new EEPROM table"); | ||
| 1399 | return amdgpu_ras_eeprom_reset_table(control); | ||
| 1400 | } | ||
| 1401 | |||
| 1402 | switch (hdr->version) { | ||
| 1403 | case RAS_TABLE_VER_V2_1: | ||
| 1404 | case RAS_TABLE_VER_V3: | ||
| 1396 | control->ras_num_recs = RAS_NUM_RECS_V2_1(hdr); | 1405 | control->ras_num_recs = RAS_NUM_RECS_V2_1(hdr); |
| 1397 | control->ras_record_offset = RAS_RECORD_START_V2_1; | 1406 | control->ras_record_offset = RAS_RECORD_START_V2_1; |
| 1398 | control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1; | 1407 | control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1; |
| 1399 | } else { | 1408 | break; |
| 1409 | case RAS_TABLE_VER_V1: | ||
| 1400 | control->ras_num_recs = RAS_NUM_RECS(hdr); | 1410 | control->ras_num_recs = RAS_NUM_RECS(hdr); |
| 1401 | control->ras_record_offset = RAS_RECORD_START; | 1411 | control->ras_record_offset = RAS_RECORD_START; |
| 1402 | control->ras_max_record_count = RAS_MAX_RECORD_COUNT; | 1412 | control->ras_max_record_count = RAS_MAX_RECORD_COUNT; |
| 1413 | break; | ||
| 1414 | default: | ||
| 1415 | dev_err(adev->dev, | ||
| 1416 | "RAS header invalid, unsupported version: %u", | ||
| 1417 | hdr->version); | ||
| 1418 | return -EINVAL; | ||
| 1403 | } | 1419 | } |
| 1404 | control->ras_fri = RAS_OFFSET_TO_INDEX(control, hdr->first_rec_offset); | ||
| 1405 | 1420 | ||
| 1421 | if (control->ras_num_recs > control->ras_max_record_count) { | ||
| 1422 | dev_err(adev->dev, | ||
| 1423 | "RAS header invalid, records in header: %u max allowed :%u", | ||
| 1424 | control->ras_num_recs, control->ras_max_record_count); | ||
| 1425 | return -EINVAL; | ||
| 1426 | } | ||
| 1427 | |||
| 1428 | control->ras_fri = RAS_OFFSET_TO_INDEX(control, hdr->first_rec_offset); | ||
| 1406 | control->ras_num_mca_recs = 0; | 1429 | control->ras_num_mca_recs = 0; |
| 1407 | control->ras_num_pa_recs = 0; | 1430 | control->ras_num_pa_recs = 0; |
| 1408 | return 0; | 1431 | return 0; |
| @@ -1413,7 +1436,7 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 4 unmodified lines | |||
| 1413 | struct amdgpu_device *adev = to_amdgpu_device(control); | 1436 | struct amdgpu_device *adev = to_amdgpu_device(control); |
| 1414 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; | 1437 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; |
| 1415 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | 1438 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); |
| 1416 | int res; | 1439 | int res = 0; |
| 1417 | 1440 | ||
| 1418 | if (!__is_ras_eeprom_supported(adev)) | 1441 | if (!__is_ras_eeprom_supported(adev)) |
| 1419 | return 0; | 1442 | return 0; |
| @@ -1494,10 +1517,6 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 74 unmodified lines | |||
| 1494 | "User defined threshold is set, runtime service will be halt when threshold is reached\n"); | 1517 | "User defined threshold is set, runtime service will be halt when threshold is reached\n"); |
| 1495 | } | 1518 | } |
| 1496 | } | 1519 | } |
| 1497 | } else { | ||
| 1498 | DRM_INFO("Creating a new EEPROM table"); | ||
| 1499 | |||
| 1500 | res = amdgpu_ras_eeprom_reset_table(control); | ||
| 1501 | } | 1520 | } |
| 1502 | 1521 | ||
| 1503 | return res < 0 ? res : 0; | 1522 | return res < 0 ? res : 0; |
| @@ -187,14 +187,10 @@ void amdgpu_ring_undo(struct amdgpu_ring *ring) · 186 unmodified lines | |||
| 187 | } | 187 | } |
| 188 | 188 | ||
| 189 | #define amdgpu_ring_get_gpu_addr(ring, offset) \ | 189 | #define amdgpu_ring_get_gpu_addr(ring, offset) \ |
| 190 | (ring->is_mes_queue ? \ | 190 | (ring->adev->wb.gpu_addr + offset * 4) |
| 191 | (ring->mes_ctx->meta_data_gpu_addr + offset) : \ | ||
| 192 | (ring->adev->wb.gpu_addr + offset * 4)) | ||
| 193 | 191 | ||
| 194 | #define amdgpu_ring_get_cpu_addr(ring, offset) \ | 192 | #define amdgpu_ring_get_cpu_addr(ring, offset) \ |
| 195 | (ring->is_mes_queue ? \ | 193 | (&ring->adev->wb.wb[offset]) |
| 196 | (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \ | ||
| 197 | (&ring->adev->wb.wb[offset])) | ||
| 198 | 194 | ||
| 199 | /** | 195 | /** |
| 200 | * amdgpu_ring_init - init driver ring struct. | 196 | * amdgpu_ring_init - init driver ring struct. |
| @@ -243,57 +239,42 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · 42 unmodified lines | |||
| 243 | ring->sched_score = sched_score; | 239 | ring->sched_score = sched_score; |
| 244 | ring->vmid_wait = dma_fence_get_stub(); | 240 | ring->vmid_wait = dma_fence_get_stub(); |
| 245 | 241 | ||
| 246 | if (!ring->is_mes_queue) { | 242 | ring->idx = adev->num_rings++; |
| 247 | ring->idx = adev->num_rings++; | 243 | adev->rings[ring->idx] = ring; |
| 248 | adev->rings[ring->idx] = ring; | ||
| 249 | } | ||
| 250 | 244 | ||
| 251 | r = amdgpu_fence_driver_init_ring(ring); | 245 | r = amdgpu_fence_driver_init_ring(ring); |
| 252 | if (r) | 246 | if (r) |
| 253 | return r; | 247 | return r; |
| 254 | } | 248 | } |
| 255 | 249 | ||
| 256 | if (ring->is_mes_queue) { | 250 | r = amdgpu_device_wb_get(adev, &ring->rptr_offs); |
| 257 | ring->rptr_offs = amdgpu_mes_ctx_get_offs(ring, | 251 | if (r) { |
| 258 | AMDGPU_MES_CTX_RPTR_OFFS); | 252 | dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r); |
| 259 | ring->wptr_offs = amdgpu_mes_ctx_get_offs(ring, | 253 | return r; |
| 260 | AMDGPU_MES_CTX_WPTR_OFFS); | 254 | } |
| 261 | ring->fence_offs = amdgpu_mes_ctx_get_offs(ring, | ||
| 262 | AMDGPU_MES_CTX_FENCE_OFFS); | ||
| 263 | ring->trail_fence_offs = amdgpu_mes_ctx_get_offs(ring, | ||
| 264 | AMDGPU_MES_CTX_TRAIL_FENCE_OFFS); | ||
| 265 | ring->cond_exe_offs = amdgpu_mes_ctx_get_offs(ring, | ||
| 266 | AMDGPU_MES_CTX_COND_EXE_OFFS); | ||
| 267 | } else { | ||
| 268 | r = amdgpu_device_wb_get(adev, &ring->rptr_offs); | ||
| 269 | if (r) { | ||
| 270 | dev_err(adev->dev, "(%d) ring rptr_offs wb alloc failed\n", r); | ||
| 271 | return r; | ||
| 272 | } | ||
| 273 | 255 | ||
| 274 | r = amdgpu_device_wb_get(adev, &ring->wptr_offs); | 256 | r = amdgpu_device_wb_get(adev, &ring->wptr_offs); |
| 275 | if (r) { | 257 | if (r) { |
| 276 | dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r); | 258 | dev_err(adev->dev, "(%d) ring wptr_offs wb alloc failed\n", r); |
| 277 | return r; | 259 | return r; |
| 278 | } | 260 | } |
| 279 | 261 | ||
| 280 | r = amdgpu_device_wb_get(adev, &ring->fence_offs); | 262 | r = amdgpu_device_wb_get(adev, &ring->fence_offs); |
| 281 | if (r) { | 263 | if (r) { |
| 282 | dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r); | 264 | dev_err(adev->dev, "(%d) ring fence_offs wb alloc failed\n", r); |
| 283 | return r; | 265 | return r; |
| 284 | } | 266 | } |
| 285 | 267 | ||
| 286 | r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs); | 268 | r = amdgpu_device_wb_get(adev, &ring->trail_fence_offs); |
| 287 | if (r) { | 269 | if (r) { |
| 288 | dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r); | 270 | dev_err(adev->dev, "(%d) ring trail_fence_offs wb alloc failed\n", r); |
| 289 | return r; | 271 | return r; |
| 290 | } | 272 | } |
| 291 | 273 | ||
| 292 | r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs); | 274 | r = amdgpu_device_wb_get(adev, &ring->cond_exe_offs); |
| 293 | if (r) { | 275 | if (r) { |
| 294 | dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r); | 276 | dev_err(adev->dev, "(%d) ring cond_exec_polling wb alloc failed\n", r); |
| 295 | return r; | 277 | return r; |
| 296 | } | ||
| 297 | } | 278 | } |
| 298 | 279 | ||
| 299 | ring->fence_gpu_addr = | 280 | ring->fence_gpu_addr = |
| @@ -353,18 +334,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · 53 unmodified lines | |||
| 353 | ring->cached_rptr = 0; | 334 | ring->cached_rptr = 0; |
| 354 | 335 | ||
| 355 | /* Allocate ring buffer */ | 336 | /* Allocate ring buffer */ |
| 356 | if (ring->is_mes_queue) { | 337 | if (ring->ring_obj == NULL) { |
| 357 | int offset = 0; | ||
| 358 | |||
| 359 | BUG_ON(ring->ring_size > PAGE_SIZE*4); | ||
| 360 | |||
| 361 | offset = amdgpu_mes_ctx_get_offs(ring, | ||
| 362 | AMDGPU_MES_CTX_RING_OFFS); | ||
| 363 | ring->gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 364 | ring->ring = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 365 | amdgpu_ring_clear_ring(ring); | ||
| 366 | |||
| 367 | } else if (ring->ring_obj == NULL) { | ||
| 368 | r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE, | 338 | r = amdgpu_bo_create_kernel(adev, ring->ring_size + ring->funcs->extra_dw, PAGE_SIZE, |
| 369 | AMDGPU_GEM_DOMAIN_GTT, | 339 | AMDGPU_GEM_DOMAIN_GTT, |
| 370 | &ring->ring_obj, | 340 | &ring->ring_obj, |
| @@ -401,32 +371,26 @@ void amdgpu_ring_fini(struct amdgpu_ring *ring) · 30 unmodified lines | |||
| 401 | { | 371 | { |
| 402 | 372 | ||
| 403 | /* Not to finish a ring which is not initialized */ | 373 | /* Not to finish a ring which is not initialized */ |
| 404 | if (!(ring->adev) || | 374 | if (!(ring->adev) || !(ring->adev->rings[ring->idx])) |
| 405 | (!ring->is_mes_queue && !(ring->adev->rings[ring->idx]))) | ||
| 406 | return; | 375 | return; |
| 407 | 376 | ||
| 408 | ring->sched.ready = false; | 377 | ring->sched.ready = false; |
| 409 | 378 | ||
| 410 | if (!ring->is_mes_queue) { | 379 | amdgpu_device_wb_free(ring->adev, ring->rptr_offs); |
| 411 | amdgpu_device_wb_free(ring->adev, ring->rptr_offs); | 380 | amdgpu_device_wb_free(ring->adev, ring->wptr_offs); |
| 412 | amdgpu_device_wb_free(ring->adev, ring->wptr_offs); | ||
| 413 | 381 | ||
| 414 | amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs); | 382 | amdgpu_device_wb_free(ring->adev, ring->cond_exe_offs); |
| 415 | amdgpu_device_wb_free(ring->adev, ring->fence_offs); | 383 | amdgpu_device_wb_free(ring->adev, ring->fence_offs); |
| 416 | 384 | ||
| 417 | amdgpu_bo_free_kernel(&ring->ring_obj, | 385 | amdgpu_bo_free_kernel(&ring->ring_obj, |
| 418 | &ring->gpu_addr, | 386 | &ring->gpu_addr, |
| 419 | (void **)&ring->ring); | 387 | (void **)&ring->ring); |
| 420 | } else { | ||
| 421 | kfree(ring->fence_drv.fences); | ||
| 422 | } | ||
| 423 | 388 | ||
| 424 | dma_fence_put(ring->vmid_wait); | 389 | dma_fence_put(ring->vmid_wait); |
| 425 | ring->vmid_wait = NULL; | 390 | ring->vmid_wait = NULL; |
| 426 | ring->me = 0; | 391 | ring->me = 0; |
| 427 | 392 | ||
| 428 | if (!ring->is_mes_queue) | 393 | ring->adev->rings[ring->idx] = NULL; |
| 429 | ring->adev->rings[ring->idx] = NULL; | ||
| 430 | } | 394 | } |
| 431 | 395 | ||
| 432 | /** | 396 | /** |
| @@ -164,8 +164,24 @@ void amdgpu_fence_update_start_timestamp(struct amdgpu_ring *ring, uint32_t seq, · 163 unmodified lines | |||
| 164 | 164 | ||
| 165 | /* provided by hw blocks that expose a ring buffer for commands */ | 165 | /* provided by hw blocks that expose a ring buffer for commands */ |
| 166 | struct amdgpu_ring_funcs { | 166 | struct amdgpu_ring_funcs { |
| 167 | /** | ||
| 168 | * @type: | ||
| 169 | * | ||
| 170 | * GFX, Compute, SDMA, UVD, VCE, VCN, VPE, KIQ, MES, UMSCH, and CPER | ||
| 171 | * use ring buffers. The type field just identifies which component the | ||
| 172 | * ring buffer is associated with. | ||
| 173 | */ | ||
| 167 | enum amdgpu_ring_type type; | 174 | enum amdgpu_ring_type type; |
| 168 | uint32_t align_mask; | 175 | uint32_t align_mask; |
| 176 | |||
| 177 | /** | ||
| 178 | * @nop: | ||
| 179 | * | ||
| 180 | * Every block in the amdgpu has no-op instructions (e.g., GFX 10 | ||
| 181 | * uses PACKET3(PACKET3_NOP, 0x3FFF), VCN 5 uses VCN_ENC_CMD_NO_OP, | ||
| 182 | * etc). This field receives the specific no-op for the component | ||
| 183 | * that initializes the ring. | ||
| 184 | */ | ||
| 169 | u32 nop; | 185 | u32 nop; |
| 170 | bool support_64bit_ptrs; | 186 | bool support_64bit_ptrs; |
| 171 | bool no_user_fence; | 187 | bool no_user_fence; |
| @@ -241,6 +257,9 @@ struct amdgpu_ring_funcs { · 69 unmodified lines | |||
| 241 | bool (*is_guilty)(struct amdgpu_ring *ring); | 257 | bool (*is_guilty)(struct amdgpu_ring *ring); |
| 242 | }; | 258 | }; |
| 243 | 259 | ||
| 260 | /** | ||
| 261 | * amdgpu_ring - Holds ring information | ||
| 262 | */ | ||
| 244 | struct amdgpu_ring { | 263 | struct amdgpu_ring { |
| 245 | struct amdgpu_device *adev; | 264 | struct amdgpu_device *adev; |
| 246 | const struct amdgpu_ring_funcs *funcs; | 265 | const struct amdgpu_ring_funcs *funcs; |
| @@ -252,13 +271,61 @@ struct amdgpu_ring { · 5 unmodified lines | |||
| 252 | unsigned rptr_offs; | 271 | unsigned rptr_offs; |
| 253 | u64 rptr_gpu_addr; | 272 | u64 rptr_gpu_addr; |
| 254 | volatile u32 *rptr_cpu_addr; | 273 | volatile u32 *rptr_cpu_addr; |
| 274 | |||
| 275 | /** | ||
| 276 | * @wptr: | ||
| 277 | * | ||
| 278 | * This is part of the Ring buffer implementation and represents the | ||
| 279 | * write pointer. The wptr determines where the host has written. | ||
| 280 | */ | ||
| 255 | u64 wptr; | 281 | u64 wptr; |
| 282 | |||
| 283 | /** | ||
| 284 | * @wptr_old: | ||
| 285 | * | ||
| 286 | * Before update wptr with the new value, usually the old value is | ||
| 287 | * stored in the wptr_old. | ||
| 288 | */ | ||
| 256 | u64 wptr_old; | 289 | u64 wptr_old; |
| 257 | unsigned ring_size; | 290 | unsigned ring_size; |
| 291 | |||
| 292 | /** | ||
| 293 | * @max_dw: | ||
| 294 | * | ||
| 295 | * Maximum number of DWords for ring allocation. This information is | ||
| 296 | * provided at the ring initialization time, and each IP block can | ||
| 297 | * specify a specific value. Check places that invoke | ||
| 298 | * amdgpu_ring_init() to see the maximum size per block. | ||
| 299 | */ | ||
| 258 | unsigned max_dw; | 300 | unsigned max_dw; |
| 301 | |||
| 302 | /** | ||
| 303 | * @count_dw: | ||
| 304 | * | ||
| 305 | * This value starts with the maximum amount of DWords supported by the | ||
| 306 | * ring. This value is updated based on the ring manipulation. | ||
| 307 | */ | ||
| 259 | int count_dw; | 308 | int count_dw; |
| 260 | uint64_t gpu_addr; | 309 | uint64_t gpu_addr; |
| 310 | |||
| 311 | /** | ||
| 312 | * @ptr_mask: | ||
| 313 | * | ||
| 314 | * Some IPs provide support for 64-bit pointers and others for 32-bit | ||
| 315 | * only; this behavior is component-specific and defined by the field | ||
| 316 | * support_64bit_ptr. If the IP block supports 64-bits, the mask | ||
| 317 | * 0xffffffffffffffff is set; otherwise, this value assumes buf_mask. | ||
| 318 | * Notice that this field is used to keep wptr under a valid range. | ||
| 319 | */ | ||
| 261 | uint64_t ptr_mask; | 320 | uint64_t ptr_mask; |
| 321 | |||
| 322 | /** | ||
| 323 | * @buf_mask: | ||
| 324 | * | ||
| 325 | * Buffer mask is a value used to keep wptr count under its | ||
| 326 | * thresholding. Buffer mask initialized during the ring buffer | ||
| 327 | * initialization time, and it is defined as (ring_size / 4) -1. | ||
| 328 | */ | ||
| 262 | uint32_t buf_mask; | 329 | uint32_t buf_mask; |
| 263 | u32 idx; | 330 | u32 idx; |
| 264 | u32 xcc_id; | 331 | u32 xcc_id; |
| @@ -276,6 +343,13 @@ struct amdgpu_ring { · 11 unmodified lines | |||
| 276 | bool use_pollmem; | 343 | bool use_pollmem; |
| 277 | unsigned wptr_offs; | 344 | unsigned wptr_offs; |
| 278 | u64 wptr_gpu_addr; | 345 | u64 wptr_gpu_addr; |
| 346 | |||
| 347 | /** | ||
| 348 | * @wptr_cpu_addr: | ||
| 349 | * | ||
| 350 | * This is the CPU address pointer in the writeback slot. This is used | ||
| 351 | * to commit changes to the GPU. | ||
| 352 | */ | ||
| 279 | volatile u32 *wptr_cpu_addr; | 353 | volatile u32 *wptr_cpu_addr; |
| 280 | unsigned fence_offs; | 354 | unsigned fence_offs; |
| 281 | u64 fence_gpu_addr; | 355 | u64 fence_gpu_addr; |
| @@ -297,20 +371,15 @@ struct amdgpu_ring { · 15 unmodified lines | |||
| 297 | struct dma_fence *vmid_wait; | 371 | struct dma_fence *vmid_wait; |
| 298 | bool has_compute_vm_bug; | 372 | bool has_compute_vm_bug; |
| 299 | bool no_scheduler; | 373 | bool no_scheduler; |
| 374 | bool no_user_submission; | ||
| 300 | int hw_prio; | 375 | int hw_prio; |
| 301 | unsigned num_hw_submission; | 376 | unsigned num_hw_submission; |
| 302 | atomic_t *sched_score; | 377 | atomic_t *sched_score; |
| 303 | 378 | ||
| 304 | /* used for mes */ | ||
| 305 | bool is_mes_queue; | ||
| 306 | uint32_t hw_queue_id; | ||
| 307 | struct amdgpu_mes_ctx_data *mes_ctx; | ||
| 308 | |||
| 309 | bool is_sw_ring; | 379 | bool is_sw_ring; |
| 310 | unsigned int entry_index; | 380 | unsigned int entry_index; |
| 311 | /* store the cached rptr to restore after reset */ | 381 | /* store the cached rptr to restore after reset */ |
| 312 | uint64_t cached_rptr; | 382 | uint64_t cached_rptr; |
| 313 | |||
| 314 | }; | 383 | }; |
| 315 | 384 | ||
| 316 | #define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib))) | 385 | #define amdgpu_ring_parse_cs(r, p, job, ib) ((r)->funcs->parse_cs((p), (job), (ib))) |
| @@ -435,15 +504,6 @@ static inline void amdgpu_ring_patch_cond_exec(struct amdgpu_ring *ring, · 118 unmodified lines | |||
| 435 | ring->ring[offset] = cur - offset; | 504 | ring->ring[offset] = cur - offset; |
| 436 | } | 505 | } |
| 437 | 506 | ||
| 438 | #define amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset) \ | ||
| 439 | (ring->is_mes_queue && ring->mes_ctx ? \ | ||
| 440 | (ring->mes_ctx->meta_data_gpu_addr + offset) : 0) | ||
| 441 | |||
| 442 | #define amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset) \ | ||
| 443 | (ring->is_mes_queue && ring->mes_ctx ? \ | ||
| 444 | (void *)((uint8_t *)(ring->mes_ctx->meta_data_ptr) + offset) : \ | ||
| 445 | NULL) | ||
| 446 | |||
| 447 | int amdgpu_ring_test_helper(struct amdgpu_ring *ring); | 507 | int amdgpu_ring_test_helper(struct amdgpu_ring *ring); |
| 448 | 508 | ||
| 449 | void amdgpu_debugfs_ring_init(struct amdgpu_device *adev, | 509 | void amdgpu_debugfs_ring_init(struct amdgpu_device *adev, |
| @@ -237,6 +237,20 @@ struct amdgpu_rlc_funcs { · 236 unmodified lines | |||
| 237 | void (*unset_safe_mode)(struct amdgpu_device *adev, int xcc_id); | 237 | void (*unset_safe_mode)(struct amdgpu_device *adev, int xcc_id); |
| 238 | int (*init)(struct amdgpu_device *adev); | 238 | int (*init)(struct amdgpu_device *adev); |
| 239 | u32 (*get_csb_size)(struct amdgpu_device *adev); | 239 | u32 (*get_csb_size)(struct amdgpu_device *adev); |
| 240 | |||
| 241 | /** | ||
| 242 | * @get_csb_buffer: Get the clear state to be put into the hardware. | ||
| 243 | * | ||
| 244 | * The parameter adev is used to get the CS data and other gfx info, | ||
| 245 | * and buffer is the RLC CS pointer | ||
| 246 | * | ||
| 247 | * Sometimes, the user space puts a request to clear the state in the | ||
| 248 | * command buffer; this function provides the clear state that gets put | ||
| 249 | * into the hardware. Note that the driver programs Clear State | ||
| 250 | * Indirect Buffer (CSB) explicitly when it sets up the kernel rings, | ||
| 251 | * and it also provides a pointer to it which is used by the firmware | ||
| 252 | * to load the clear state in some cases. | ||
| 253 | */ | ||
| 240 | void (*get_csb_buffer)(struct amdgpu_device *adev, volatile u32 *buffer); | 254 | void (*get_csb_buffer)(struct amdgpu_device *adev, volatile u32 *buffer); |
| 241 | int (*get_cp_table_num)(struct amdgpu_device *adev); | 255 | int (*get_cp_table_num)(struct amdgpu_device *adev); |
| 242 | int (*resume)(struct amdgpu_device *adev); | 256 | int (*resume)(struct amdgpu_device *adev); |
| @@ -26,6 +26,8 @@ · 25 unmodified lines | |||
| 26 | #include "amdgpu_sdma.h" | 26 | #include "amdgpu_sdma.h" |
| 27 | #include "amdgpu_ras.h" | 27 | #include "amdgpu_ras.h" |
| 28 | #include "amdgpu_reset.h" | 28 | #include "amdgpu_reset.h" |
| 29 | #include "gc/gc_10_1_0_offset.h" | ||
| 30 | #include "gc/gc_10_3_0_sh_mask.h" | ||
| 29 | 31 | ||
| 30 | #define AMDGPU_CSA_SDMA_SIZE 64 | 32 | #define AMDGPU_CSA_SDMA_SIZE 64 |
| 31 | /* SDMA CSA reside in the 3rd page of CSA */ | 33 | /* SDMA CSA reside in the 3rd page of CSA */ |
| @@ -76,22 +78,14 @@ uint64_t amdgpu_sdma_get_csa_mc_addr(struct amdgpu_ring *ring, · 44 unmodified lines | |||
| 76 | if (amdgpu_sriov_vf(adev) || vmid == 0 || !adev->gfx.mcbp) | 78 | if (amdgpu_sriov_vf(adev) || vmid == 0 || !adev->gfx.mcbp) |
| 77 | return 0; | 79 | return 0; |
| 78 | 80 | ||
| 79 | if (ring->is_mes_queue) { | 81 | r = amdgpu_sdma_get_index_from_ring(ring, &index); |
| 80 | uint32_t offset = 0; | ||
| 81 | 82 | ||
| 82 | offset = offsetof(struct amdgpu_mes_ctx_meta_data, | 83 | if (r || index > 31) |
| 83 | sdma[ring->idx].sdma_meta_data); | 84 | csa_mc_addr = 0; |
| 84 | csa_mc_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | 85 | else |
| 85 | } else { | 86 | csa_mc_addr = amdgpu_csa_vaddr(adev) + |
| 86 | r = amdgpu_sdma_get_index_from_ring(ring, &index); | 87 | AMDGPU_CSA_SDMA_OFFSET + |
| 87 | 88 | index * AMDGPU_CSA_SDMA_SIZE; | |
| 88 | if (r || index > 31) | ||
| 89 | csa_mc_addr = 0; | ||
| 90 | else | ||
| 91 | csa_mc_addr = amdgpu_csa_vaddr(adev) + | ||
| 92 | AMDGPU_CSA_SDMA_OFFSET + | ||
| 93 | index * AMDGPU_CSA_SDMA_SIZE; | ||
| 94 | } | ||
| 95 | 89 | ||
| 96 | return csa_mc_addr; | 90 | return csa_mc_addr; |
| 97 | } | 91 | } |
| @@ -537,28 +531,38 @@ bool amdgpu_sdma_is_shared_inv_eng(struct amdgpu_device *adev, struct amdgpu_rin · 439 unmodified lines | |||
| 537 | return false; | 531 | return false; |
| 538 | } | 532 | } |
| 539 | 533 | ||
| 540 | /** | 534 | static int amdgpu_sdma_soft_reset(struct amdgpu_device *adev, u32 instance_id) |
| 541 | * amdgpu_sdma_register_on_reset_callbacks - Register SDMA reset callbacks | ||
| 542 | * @funcs: Pointer to the callback structure containing pre_reset and post_reset functions | ||
| 543 | * | ||
| 544 | * This function allows KFD and AMDGPU to register their own callbacks for handling | ||
| 545 | * pre-reset and post-reset operations for engine reset. These are needed because engine | ||
| 546 | * reset will stop all queues on that engine. | ||
| 547 | */ | ||
| 548 | void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct sdma_on_reset_funcs *funcs) | ||
| 549 | { | 535 | { |
| 550 | if (!funcs) | 536 | struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id]; |
| 551 | return; | 537 | int r = -EOPNOTSUPP; |
| 552 | 538 | ||
| 553 | /* Ensure the reset_callback_list is initialized */ | 539 | switch (amdgpu_ip_version(adev, SDMA0_HWIP, 0)) { |
| 554 | if (!adev->sdma.reset_callback_list.next) { | 540 | case IP_VERSION(4, 4, 2): |
| 555 | INIT_LIST_HEAD(&adev->sdma.reset_callback_list); | 541 | case IP_VERSION(4, 4, 4): |
| 542 | case IP_VERSION(4, 4, 5): | ||
| 543 | /* For SDMA 4.x, use the existing DPM interface for backward compatibility */ | ||
| 544 | r = amdgpu_dpm_reset_sdma(adev, 1 << instance_id); | ||
| 545 | break; | ||
| 546 | case IP_VERSION(5, 0, 0): | ||
| 547 | case IP_VERSION(5, 0, 1): | ||
| 548 | case IP_VERSION(5, 0, 2): | ||
| 549 | case IP_VERSION(5, 0, 5): | ||
| 550 | case IP_VERSION(5, 2, 0): | ||
| 551 | case IP_VERSION(5, 2, 2): | ||
| 552 | case IP_VERSION(5, 2, 4): | ||
| 553 | case IP_VERSION(5, 2, 5): | ||
| 554 | case IP_VERSION(5, 2, 6): | ||
| 555 | case IP_VERSION(5, 2, 3): | ||
| 556 | case IP_VERSION(5, 2, 1): | ||
| 557 | case IP_VERSION(5, 2, 7): | ||
| 558 | if (sdma_instance->funcs->soft_reset_kernel_queue) | ||
| 559 | r = sdma_instance->funcs->soft_reset_kernel_queue(adev, instance_id); | ||
| 560 | break; | ||
| 561 | default: | ||
| 562 | break; | ||
| 556 | } | 563 | } |
| 557 | /* Initialize the list node in the callback structure */ | ||
| 558 | INIT_LIST_HEAD(&funcs->list); | ||
| 559 | 564 | ||
| 560 | /* Add the callback structure to the global list */ | 565 | return r; |
| 561 | list_add_tail(&funcs->list, &adev->sdma.reset_callback_list); | ||
| 562 | } | 566 | } |
| 563 | 567 | ||
| 564 | /** | 568 | /** |
| @@ -566,16 +570,10 @@ void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct · 1 unmodified line | |||
| 566 | * @adev: Pointer to the AMDGPU device | 570 | * @adev: Pointer to the AMDGPU device |
| 567 | * @instance_id: ID of the SDMA engine instance to reset | 571 | * @instance_id: ID of the SDMA engine instance to reset |
| 568 | * | 572 | * |
| 569 | * This function performs the following steps: | ||
| 570 | * 1. Calls all registered pre_reset callbacks to allow KFD and AMDGPU to save their state. | ||
| 571 | * 2. Resets the specified SDMA engine instance. | ||
| 572 | * 3. Calls all registered post_reset callbacks to allow KFD and AMDGPU to restore their state. | ||
| 573 | * | ||
| 574 | * Returns: 0 on success, or a negative error code on failure. | 573 | * Returns: 0 on success, or a negative error code on failure. |
| 575 | */ | 574 | */ |
| 576 | int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id) | 575 | int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id) |
| 577 | { | 576 | { |
| 578 | struct sdma_on_reset_funcs *funcs; | ||
| 579 | int ret = 0; | 577 | int ret = 0; |
| 580 | struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id]; | 578 | struct amdgpu_sdma_instance *sdma_instance = &adev->sdma.instance[instance_id]; |
| 581 | struct amdgpu_ring *gfx_ring = &sdma_instance->ring; | 579 | struct amdgpu_ring *gfx_ring = &sdma_instance->ring; |
| @@ -597,38 +595,18 @@ int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id) · 15 unmodified lines | |||
| 597 | page_sched_stopped = true; | 595 | page_sched_stopped = true; |
| 598 | } | 596 | } |
| 599 | 597 | ||
| 600 | /* Invoke all registered pre_reset callbacks */ | 598 | if (sdma_instance->funcs->stop_kernel_queue) |
| 601 | list_for_each_entry(funcs, &adev->sdma.reset_callback_list, list) { | 599 | sdma_instance->funcs->stop_kernel_queue(gfx_ring); |
| 602 | if (funcs->pre_reset) { | ||
| 603 | ret = funcs->pre_reset(adev, instance_id); | ||
| 604 | if (ret) { | ||
| 605 | dev_err(adev->dev, | ||
| 606 | "beforeReset callback failed for instance %u: %d\n", | ||
| 607 | instance_id, ret); | ||
| 608 | goto exit; | ||
| 609 | } | ||
| 610 | } | ||
| 611 | } | ||
| 612 | 600 | ||
| 613 | /* Perform the SDMA reset for the specified instance */ | 601 | /* Perform the SDMA reset for the specified instance */ |
| 614 | ret = amdgpu_dpm_reset_sdma(adev, 1 << instance_id); | 602 | ret = amdgpu_sdma_soft_reset(adev, instance_id); |
| 615 | if (ret) { | 603 | if (ret) { |
| 616 | dev_err(adev->dev, "Failed to reset SDMA instance %u\n", instance_id); | 604 | dev_err(adev->dev, "Failed to reset SDMA instance %u\n", instance_id); |
| 617 | goto exit; | 605 | goto exit; |
| 618 | } | 606 | } |
| 619 | 607 | ||
| 620 | /* Invoke all registered post_reset callbacks */ | 608 | if (sdma_instance->funcs->start_kernel_queue) |
| 621 | list_for_each_entry(funcs, &adev->sdma.reset_callback_list, list) { | 609 | sdma_instance->funcs->start_kernel_queue(gfx_ring); |
| 622 | if (funcs->post_reset) { | ||
| 623 | ret = funcs->post_reset(adev, instance_id); | ||
| 624 | if (ret) { | ||
| 625 | dev_err(adev->dev, | ||
| 626 | "afterReset callback failed for instance %u: %d\n", | ||
| 627 | instance_id, ret); | ||
| 628 | goto exit; | ||
| 629 | } | ||
| 630 | } | ||
| 631 | } | ||
| 632 | 610 | ||
| 633 | exit: | 611 | exit: |
| 634 | /* Restart the scheduler's work queue for the GFX and page rings | 612 | /* Restart the scheduler's work queue for the GFX and page rings |
| @@ -50,6 +50,12 @@ enum amdgpu_sdma_irq { · 49 unmodified lines | |||
| 50 | 50 | ||
| 51 | #define NUM_SDMA(x) hweight32(x) | 51 | #define NUM_SDMA(x) hweight32(x) |
| 52 | 52 | ||
| 53 | struct amdgpu_sdma_funcs { | ||
| 54 | int (*stop_kernel_queue)(struct amdgpu_ring *ring); | ||
| 55 | int (*start_kernel_queue)(struct amdgpu_ring *ring); | ||
| 56 | int (*soft_reset_kernel_queue)(struct amdgpu_device *adev, u32 instance_id); | ||
| 57 | }; | ||
| 58 | |||
| 53 | struct amdgpu_sdma_instance { | 59 | struct amdgpu_sdma_instance { |
| 54 | /* SDMA firmware */ | 60 | /* SDMA firmware */ |
| 55 | const struct firmware *fw; | 61 | const struct firmware *fw; |
| @@ -68,7 +74,7 @@ struct amdgpu_sdma_instance { · 12 unmodified lines | |||
| 68 | /* track guilty state of GFX and PAGE queues */ | 74 | /* track guilty state of GFX and PAGE queues */ |
| 69 | bool gfx_guilty; | 75 | bool gfx_guilty; |
| 70 | bool page_guilty; | 76 | bool page_guilty; |
| 71 | 77 | const struct amdgpu_sdma_funcs *funcs; | |
| 72 | }; | 78 | }; |
| 73 | 79 | ||
| 74 | enum amdgpu_sdma_ras_memory_id { | 80 | enum amdgpu_sdma_ras_memory_id { |
| @@ -103,13 +109,6 @@ struct amdgpu_sdma_ras { · 28 unmodified lines | |||
| 103 | struct amdgpu_ras_block_object ras_block; | 109 | struct amdgpu_ras_block_object ras_block; |
| 104 | }; | 110 | }; |
| 105 | 111 | ||
| 106 | struct sdma_on_reset_funcs { | ||
| 107 | int (*pre_reset)(struct amdgpu_device *adev, uint32_t instance_id); | ||
| 108 | int (*post_reset)(struct amdgpu_device *adev, uint32_t instance_id); | ||
| 109 | /* Linked list node to store this structure in a list; */ | ||
| 110 | struct list_head list; | ||
| 111 | }; | ||
| 112 | |||
| 113 | struct amdgpu_sdma { | 112 | struct amdgpu_sdma { |
| 114 | struct amdgpu_sdma_instance instance[AMDGPU_MAX_SDMA_INSTANCES]; | 113 | struct amdgpu_sdma_instance instance[AMDGPU_MAX_SDMA_INSTANCES]; |
| 115 | struct amdgpu_irq_src trap_irq; | 114 | struct amdgpu_irq_src trap_irq; |
| @@ -131,6 +130,8 @@ struct amdgpu_sdma { · 15 unmodified lines | |||
| 131 | uint32_t *ip_dump; | 130 | uint32_t *ip_dump; |
| 132 | uint32_t supported_reset; | 131 | uint32_t supported_reset; |
| 133 | struct list_head reset_callback_list; | 132 | struct list_head reset_callback_list; |
| 133 | bool no_user_submission; | ||
| 134 | bool disable_uq; | ||
| 134 | }; | 135 | }; |
| 135 | 136 | ||
| 136 | /* | 137 | /* |
| @@ -170,7 +171,6 @@ struct amdgpu_buffer_funcs { · 33 unmodified lines | |||
| 170 | uint32_t byte_count); | 171 | uint32_t byte_count); |
| 171 | }; | 172 | }; |
| 172 | 173 | ||
| 173 | void amdgpu_sdma_register_on_reset_callbacks(struct amdgpu_device *adev, struct sdma_on_reset_funcs *funcs); | ||
| 174 | int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id); | 174 | int amdgpu_sdma_reset_engine(struct amdgpu_device *adev, uint32_t instance_id); |
| 175 | 175 | ||
| 176 | #define amdgpu_emit_copy_buffer(adev, ib, s, d, b, t) (adev)->mman.buffer_funcs->emit_copy_buffer((ib), (s), (d), (b), (t)) | 176 | #define amdgpu_emit_copy_buffer(adev, ib, s, d, b, t) (adev)->mman.buffer_funcs->emit_copy_buffer((ib), (s), (d), (b), (t)) |
| @@ -45,7 +45,11 @@ · 44 unmodified lines | |||
| 45 | */ | 45 | */ |
| 46 | static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) | 46 | static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) |
| 47 | { | 47 | { |
| 48 | return AMDGPU_VA_RESERVED_SEQ64_START(adev); | 48 | u64 addr = AMDGPU_VA_RESERVED_SEQ64_START(adev); |
| 49 | |||
| 50 | addr = amdgpu_gmc_sign_extend(addr); | ||
| 51 | |||
| 52 | return addr; | ||
| 49 | } | 53 | } |
| 50 | 54 | ||
| 51 | /** | 55 | /** |
| @@ -63,9 +67,9 @@ static inline u64 amdgpu_seq64_get_va_base(struct amdgpu_device *adev) · 11 unmodified lines | |||
| 63 | int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, | 67 | int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, |
| 64 | struct amdgpu_bo_va **bo_va) | 68 | struct amdgpu_bo_va **bo_va) |
| 65 | { | 69 | { |
| 70 | u64 seq64_addr, va_flags; | ||
| 66 | struct amdgpu_bo *bo; | 71 | struct amdgpu_bo *bo; |
| 67 | struct drm_exec exec; | 72 | struct drm_exec exec; |
| 68 | u64 seq64_addr; | ||
| 69 | int r; | 73 | int r; |
| 70 | 74 | ||
| 71 | bo = adev->seq64.sbo; | 75 | bo = adev->seq64.sbo; |
| @@ -88,9 +92,11 @@ int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 16 unmodified lines | |||
| 88 | goto error; | 92 | goto error; |
| 89 | } | 93 | } |
| 90 | 94 | ||
| 91 | seq64_addr = amdgpu_seq64_get_va_base(adev); | 95 | seq64_addr = amdgpu_seq64_get_va_base(adev) & AMDGPU_GMC_HOLE_MASK; |
| 96 | |||
| 97 | va_flags = amdgpu_gem_va_map_flags(adev, AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_MTYPE_UC); | ||
| 92 | r = amdgpu_vm_bo_map(adev, *bo_va, seq64_addr, 0, AMDGPU_VA_RESERVED_SEQ64_SIZE, | 98 | r = amdgpu_vm_bo_map(adev, *bo_va, seq64_addr, 0, AMDGPU_VA_RESERVED_SEQ64_SIZE, |
| 93 | AMDGPU_PTE_READABLE); | 99 | va_flags); |
| 94 | if (r) { | 100 | if (r) { |
| 95 | DRM_ERROR("failed to do bo_map on userq sem, err=%d\n", r); | 101 | DRM_ERROR("failed to do bo_map on userq sem, err=%d\n", r); |
| 96 | amdgpu_vm_bo_del(adev, *bo_va); | 102 | amdgpu_vm_bo_del(adev, *bo_va); |
| @@ -156,6 +162,7 @@ error: · 59 unmodified lines | |||
| 156 | * | 162 | * |
| 157 | * @adev: amdgpu_device pointer | 163 | * @adev: amdgpu_device pointer |
| 158 | * @va: VA to access the seq in process address space | 164 | * @va: VA to access the seq in process address space |
| 165 | * @gpu_addr: GPU address to access the seq | ||
| 159 | * @cpu_addr: CPU address to access the seq | 166 | * @cpu_addr: CPU address to access the seq |
| 160 | * | 167 | * |
| 161 | * Alloc a 64 bit memory from seq64 pool. | 168 | * Alloc a 64 bit memory from seq64 pool. |
| @@ -163,7 +170,8 @@ error: · 1 unmodified line | |||
| 163 | * Returns: | 170 | * Returns: |
| 164 | * 0 on success or a negative error code on failure | 171 | * 0 on success or a negative error code on failure |
| 165 | */ | 172 | */ |
| 166 | int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 **cpu_addr) | 173 | int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, |
| 174 | u64 *gpu_addr, u64 **cpu_addr) | ||
| 167 | { | 175 | { |
| 168 | unsigned long bit_pos; | 176 | unsigned long bit_pos; |
| 169 | 177 | ||
| @@ -172,7 +180,12 @@ int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 **cpu_addr) · 2 unmodified lines | |||
| 172 | return -ENOSPC; | 180 | return -ENOSPC; |
| 173 | 181 | ||
| 174 | __set_bit(bit_pos, adev->seq64.used); | 182 | __set_bit(bit_pos, adev->seq64.used); |
| 183 | |||
| 175 | *va = bit_pos * sizeof(u64) + amdgpu_seq64_get_va_base(adev); | 184 | *va = bit_pos * sizeof(u64) + amdgpu_seq64_get_va_base(adev); |
| 185 | |||
| 186 | if (gpu_addr) | ||
| 187 | *gpu_addr = bit_pos * sizeof(u64) + adev->seq64.gpu_addr; | ||
| 188 | |||
| 176 | *cpu_addr = bit_pos + adev->seq64.cpu_base_addr; | 189 | *cpu_addr = bit_pos + adev->seq64.cpu_base_addr; |
| 177 | 190 | ||
| 178 | return 0; | 191 | return 0; |
| @@ -233,7 +246,7 @@ int amdgpu_seq64_init(struct amdgpu_device *adev) · 54 unmodified lines | |||
| 233 | */ | 246 | */ |
| 234 | r = amdgpu_bo_create_kernel(adev, AMDGPU_VA_RESERVED_SEQ64_SIZE, | 247 | r = amdgpu_bo_create_kernel(adev, AMDGPU_VA_RESERVED_SEQ64_SIZE, |
| 235 | PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT, | 248 | PAGE_SIZE, AMDGPU_GEM_DOMAIN_GTT, |
| 236 | &adev->seq64.sbo, NULL, | 249 | &adev->seq64.sbo, &adev->seq64.gpu_addr, |
| 237 | (void **)&adev->seq64.cpu_base_addr); | 250 | (void **)&adev->seq64.cpu_base_addr); |
| 238 | if (r) { | 251 | if (r) { |
| 239 | dev_warn(adev->dev, "(%d) create seq64 failed\n", r); | 252 | dev_warn(adev->dev, "(%d) create seq64 failed\n", r); |
| @@ -32,13 +32,14 @@ · 31 unmodified lines | |||
| 32 | struct amdgpu_seq64 { | 32 | struct amdgpu_seq64 { |
| 33 | struct amdgpu_bo *sbo; | 33 | struct amdgpu_bo *sbo; |
| 34 | u32 num_sem; | 34 | u32 num_sem; |
| 35 | u64 gpu_addr; | ||
| 35 | u64 *cpu_base_addr; | 36 | u64 *cpu_base_addr; |
| 36 | DECLARE_BITMAP(used, AMDGPU_MAX_SEQ64_SLOTS); | 37 | DECLARE_BITMAP(used, AMDGPU_MAX_SEQ64_SLOTS); |
| 37 | }; | 38 | }; |
| 38 | 39 | ||
| 39 | void amdgpu_seq64_fini(struct amdgpu_device *adev); | 40 | void amdgpu_seq64_fini(struct amdgpu_device *adev); |
| 40 | int amdgpu_seq64_init(struct amdgpu_device *adev); | 41 | int amdgpu_seq64_init(struct amdgpu_device *adev); |
| 41 | int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *gpu_addr, u64 **cpu_addr); | 42 | int amdgpu_seq64_alloc(struct amdgpu_device *adev, u64 *va, u64 *gpu_addr, u64 **cpu_addr); |
| 42 | void amdgpu_seq64_free(struct amdgpu_device *adev, u64 gpu_addr); | 43 | void amdgpu_seq64_free(struct amdgpu_device *adev, u64 gpu_addr); |
| 43 | int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, | 44 | int amdgpu_seq64_map(struct amdgpu_device *adev, struct amdgpu_vm *vm, |
| 44 | struct amdgpu_bo_va **bo_va); | 45 | struct amdgpu_bo_va **bo_va); |
| @@ -249,9 +249,8 @@ int amdgpu_sync_resv(struct amdgpu_device *adev, struct amdgpu_sync *sync, · 248 unmodified lines | |||
| 249 | 249 | ||
| 250 | if (resv == NULL) | 250 | if (resv == NULL) |
| 251 | return -EINVAL; | 251 | return -EINVAL; |
| 252 | 252 | /* Implicitly sync only to KERNEL, WRITE and READ */ | |
| 253 | /* TODO: Use DMA_RESV_USAGE_READ here */ | 253 | dma_resv_for_each_fence(&cursor, resv, DMA_RESV_USAGE_READ, f) { |
| 254 | dma_resv_for_each_fence(&cursor, resv, DMA_RESV_USAGE_BOOKKEEP, f) { | ||
| 255 | dma_fence_chain_for_each(f, f) { | 254 | dma_fence_chain_for_each(f, f) { |
| 256 | struct dma_fence *tmp = dma_fence_chain_contained(f); | 255 | struct dma_fence *tmp = dma_fence_chain_contained(f); |
| 257 | 256 | ||
| @@ -25,6 +25,8 @@ · 24 unmodified lines | |||
| 25 | 25 | ||
| 26 | #include "amdgpu_socbb.h" | 26 | #include "amdgpu_socbb.h" |
| 27 | 27 | ||
| 28 | #define RS64_FW_UC_START_ADDR_LO 0x3000 | ||
| 29 | |||
| 28 | struct common_firmware_header { | 30 | struct common_firmware_header { |
| 29 | uint32_t size_bytes; /* size of the entire header+image(s) in bytes */ | 31 | uint32_t size_bytes; /* size of the entire header+image(s) in bytes */ |
| 30 | uint32_t header_size_bytes; /* size of just the header in bytes */ | 32 | uint32_t header_size_bytes; /* size of just the header in bytes */ |
| @@ -0,0 +1,919 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 2 | /* | ||
| 3 | * Copyright 2023 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | |||
| 25 | #include <drm/drm_auth.h> | ||
| 26 | #include <drm/drm_exec.h> | ||
| 27 | #include <linux/pm_runtime.h> | ||
| 28 | |||
| 29 | #include "amdgpu.h" | ||
| 30 | #include "amdgpu_vm.h" | ||
| 31 | #include "amdgpu_userq.h" | ||
| 32 | #include "amdgpu_userq_fence.h" | ||
| 33 | |||
| 34 | u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev) | ||
| 35 | { | ||
| 36 | int i; | ||
| 37 | u32 userq_ip_mask = 0; | ||
| 38 | |||
| 39 | for (i = 0; i < AMDGPU_HW_IP_NUM; i++) { | ||
| 40 | if (adev->userq_funcs[i]) | ||
| 41 | userq_ip_mask |= (1 << i); | ||
| 42 | } | ||
| 43 | |||
| 44 | return userq_ip_mask; | ||
| 45 | } | ||
| 46 | |||
| 47 | static int | ||
| 48 | amdgpu_userq_unmap_helper(struct amdgpu_userq_mgr *uq_mgr, | ||
| 49 | struct amdgpu_usermode_queue *queue) | ||
| 50 | { | ||
| 51 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 52 | const struct amdgpu_userq_funcs *userq_funcs = | ||
| 53 | adev->userq_funcs[queue->queue_type]; | ||
| 54 | int r = 0; | ||
| 55 | |||
| 56 | if (queue->state == AMDGPU_USERQ_STATE_MAPPED) { | ||
| 57 | r = userq_funcs->unmap(uq_mgr, queue); | ||
| 58 | if (r) | ||
| 59 | queue->state = AMDGPU_USERQ_STATE_HUNG; | ||
| 60 | else | ||
| 61 | queue->state = AMDGPU_USERQ_STATE_UNMAPPED; | ||
| 62 | } | ||
| 63 | return r; | ||
| 64 | } | ||
| 65 | |||
| 66 | static int | ||
| 67 | amdgpu_userq_map_helper(struct amdgpu_userq_mgr *uq_mgr, | ||
| 68 | struct amdgpu_usermode_queue *queue) | ||
| 69 | { | ||
| 70 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 71 | const struct amdgpu_userq_funcs *userq_funcs = | ||
| 72 | adev->userq_funcs[queue->queue_type]; | ||
| 73 | int r = 0; | ||
| 74 | |||
| 75 | if (queue->state == AMDGPU_USERQ_STATE_UNMAPPED) { | ||
| 76 | r = userq_funcs->map(uq_mgr, queue); | ||
| 77 | if (r) { | ||
| 78 | queue->state = AMDGPU_USERQ_STATE_HUNG; | ||
| 79 | } else { | ||
| 80 | queue->state = AMDGPU_USERQ_STATE_MAPPED; | ||
| 81 | } | ||
| 82 | } | ||
| 83 | return r; | ||
| 84 | } | ||
| 85 | |||
| 86 | static void | ||
| 87 | amdgpu_userq_wait_for_last_fence(struct amdgpu_userq_mgr *uq_mgr, | ||
| 88 | struct amdgpu_usermode_queue *queue) | ||
| 89 | { | ||
| 90 | struct dma_fence *f = queue->last_fence; | ||
| 91 | int ret; | ||
| 92 | |||
| 93 | if (f && !dma_fence_is_signaled(f)) { | ||
| 94 | ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100)); | ||
| 95 | if (ret <= 0) | ||
| 96 | drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n", | ||
| 97 | f->context, f->seqno); | ||
| 98 | } | ||
| 99 | } | ||
| 100 | |||
| 101 | static void | ||
| 102 | amdgpu_userq_cleanup(struct amdgpu_userq_mgr *uq_mgr, | ||
| 103 | struct amdgpu_usermode_queue *queue, | ||
| 104 | int queue_id) | ||
| 105 | { | ||
| 106 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 107 | const struct amdgpu_userq_funcs *uq_funcs = adev->userq_funcs[queue->queue_type]; | ||
| 108 | |||
| 109 | uq_funcs->mqd_destroy(uq_mgr, queue); | ||
| 110 | amdgpu_userq_fence_driver_free(queue); | ||
| 111 | idr_remove(&uq_mgr->userq_idr, queue_id); | ||
| 112 | kfree(queue); | ||
| 113 | } | ||
| 114 | |||
| 115 | int | ||
| 116 | amdgpu_userq_active(struct amdgpu_userq_mgr *uq_mgr) | ||
| 117 | { | ||
| 118 | struct amdgpu_usermode_queue *queue; | ||
| 119 | int queue_id; | ||
| 120 | int ret = 0; | ||
| 121 | |||
| 122 | mutex_lock(&uq_mgr->userq_mutex); | ||
| 123 | /* Resume all the queues for this process */ | ||
| 124 | idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) | ||
| 125 | ret += queue->state == AMDGPU_USERQ_STATE_MAPPED; | ||
| 126 | |||
| 127 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 128 | return ret; | ||
| 129 | } | ||
| 130 | |||
| 131 | static struct amdgpu_usermode_queue * | ||
| 132 | amdgpu_userq_find(struct amdgpu_userq_mgr *uq_mgr, int qid) | ||
| 133 | { | ||
| 134 | return idr_find(&uq_mgr->userq_idr, qid); | ||
| 135 | } | ||
| 136 | |||
| 137 | void | ||
| 138 | amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *uq_mgr, | ||
| 139 | struct amdgpu_eviction_fence_mgr *evf_mgr) | ||
| 140 | { | ||
| 141 | struct amdgpu_eviction_fence *ev_fence; | ||
| 142 | |||
| 143 | retry: | ||
| 144 | /* Flush any pending resume work to create ev_fence */ | ||
| 145 | flush_delayed_work(&uq_mgr->resume_work); | ||
| 146 | |||
| 147 | mutex_lock(&uq_mgr->userq_mutex); | ||
| 148 | spin_lock(&evf_mgr->ev_fence_lock); | ||
| 149 | ev_fence = evf_mgr->ev_fence; | ||
| 150 | spin_unlock(&evf_mgr->ev_fence_lock); | ||
| 151 | if (!ev_fence || dma_fence_is_signaled(&ev_fence->base)) { | ||
| 152 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 153 | /* | ||
| 154 | * Looks like there was no pending resume work, | ||
| 155 | * add one now to create a valid eviction fence | ||
| 156 | */ | ||
| 157 | schedule_delayed_work(&uq_mgr->resume_work, 0); | ||
| 158 | goto retry; | ||
| 159 | } | ||
| 160 | } | ||
| 161 | |||
| 162 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, | ||
| 163 | struct amdgpu_userq_obj *userq_obj, | ||
| 164 | int size) | ||
| 165 | { | ||
| 166 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 167 | struct amdgpu_bo_param bp; | ||
| 168 | int r; | ||
| 169 | |||
| 170 | memset(&bp, 0, sizeof(bp)); | ||
| 171 | bp.byte_align = PAGE_SIZE; | ||
| 172 | bp.domain = AMDGPU_GEM_DOMAIN_GTT; | ||
| 173 | bp.flags = AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS | | ||
| 174 | AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED; | ||
| 175 | bp.type = ttm_bo_type_kernel; | ||
| 176 | bp.size = size; | ||
| 177 | bp.resv = NULL; | ||
| 178 | bp.bo_ptr_size = sizeof(struct amdgpu_bo); | ||
| 179 | |||
| 180 | r = amdgpu_bo_create(adev, &bp, &userq_obj->obj); | ||
| 181 | if (r) { | ||
| 182 | drm_file_err(uq_mgr->file, "Failed to allocate BO for userqueue (%d)", r); | ||
| 183 | return r; | ||
| 184 | } | ||
| 185 | |||
| 186 | r = amdgpu_bo_reserve(userq_obj->obj, true); | ||
| 187 | if (r) { | ||
| 188 | drm_file_err(uq_mgr->file, "Failed to reserve BO to map (%d)", r); | ||
| 189 | goto free_obj; | ||
| 190 | } | ||
| 191 | |||
| 192 | r = amdgpu_ttm_alloc_gart(&(userq_obj->obj)->tbo); | ||
| 193 | if (r) { | ||
| 194 | drm_file_err(uq_mgr->file, "Failed to alloc GART for userqueue object (%d)", r); | ||
| 195 | goto unresv; | ||
| 196 | } | ||
| 197 | |||
| 198 | r = amdgpu_bo_kmap(userq_obj->obj, &userq_obj->cpu_ptr); | ||
| 199 | if (r) { | ||
| 200 | drm_file_err(uq_mgr->file, "Failed to map BO for userqueue (%d)", r); | ||
| 201 | goto unresv; | ||
| 202 | } | ||
| 203 | |||
| 204 | userq_obj->gpu_addr = amdgpu_bo_gpu_offset(userq_obj->obj); | ||
| 205 | amdgpu_bo_unreserve(userq_obj->obj); | ||
| 206 | memset(userq_obj->cpu_ptr, 0, size); | ||
| 207 | return 0; | ||
| 208 | |||
| 209 | unresv: | ||
| 210 | amdgpu_bo_unreserve(userq_obj->obj); | ||
| 211 | |||
| 212 | free_obj: | ||
| 213 | amdgpu_bo_unref(&userq_obj->obj); | ||
| 214 | return r; | ||
| 215 | } | ||
| 216 | |||
| 217 | void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr, | ||
| 218 | struct amdgpu_userq_obj *userq_obj) | ||
| 219 | { | ||
| 220 | amdgpu_bo_kunmap(userq_obj->obj); | ||
| 221 | amdgpu_bo_unref(&userq_obj->obj); | ||
| 222 | } | ||
| 223 | |||
| 224 | uint64_t | ||
| 225 | amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr, | ||
| 226 | struct amdgpu_db_info *db_info, | ||
| 227 | struct drm_file *filp) | ||
| 228 | { | ||
| 229 | uint64_t index; | ||
| 230 | struct drm_gem_object *gobj; | ||
| 231 | struct amdgpu_userq_obj *db_obj = db_info->db_obj; | ||
| 232 | int r, db_size; | ||
| 233 | |||
| 234 | gobj = drm_gem_object_lookup(filp, db_info->doorbell_handle); | ||
| 235 | if (gobj == NULL) { | ||
| 236 | drm_file_err(uq_mgr->file, "Can't find GEM object for doorbell\n"); | ||
| 237 | return -EINVAL; | ||
| 238 | } | ||
| 239 | |||
| 240 | db_obj->obj = amdgpu_bo_ref(gem_to_amdgpu_bo(gobj)); | ||
| 241 | drm_gem_object_put(gobj); | ||
| 242 | |||
| 243 | /* Pin the BO before generating the index, unpin in queue destroy */ | ||
| 244 | r = amdgpu_bo_pin(db_obj->obj, AMDGPU_GEM_DOMAIN_DOORBELL); | ||
| 245 | if (r) { | ||
| 246 | drm_file_err(uq_mgr->file, "[Usermode queues] Failed to pin doorbell object\n"); | ||
| 247 | goto unref_bo; | ||
| 248 | } | ||
| 249 | |||
| 250 | r = amdgpu_bo_reserve(db_obj->obj, true); | ||
| 251 | if (r) { | ||
| 252 | drm_file_err(uq_mgr->file, "[Usermode queues] Failed to pin doorbell object\n"); | ||
| 253 | goto unpin_bo; | ||
| 254 | } | ||
| 255 | |||
| 256 | switch (db_info->queue_type) { | ||
| 257 | case AMDGPU_HW_IP_GFX: | ||
| 258 | case AMDGPU_HW_IP_COMPUTE: | ||
| 259 | case AMDGPU_HW_IP_DMA: | ||
| 260 | db_size = sizeof(u64); | ||
| 261 | break; | ||
| 262 | |||
| 263 | case AMDGPU_HW_IP_VCN_ENC: | ||
| 264 | db_size = sizeof(u32); | ||
| 265 | db_info->doorbell_offset += AMDGPU_NAVI10_DOORBELL64_VCN0_1 << 1; | ||
| 266 | break; | ||
| 267 | |||
| 268 | case AMDGPU_HW_IP_VPE: | ||
| 269 | db_size = sizeof(u32); | ||
| 270 | db_info->doorbell_offset += AMDGPU_NAVI10_DOORBELL64_VPE << 1; | ||
| 271 | break; | ||
| 272 | |||
| 273 | default: | ||
| 274 | drm_file_err(uq_mgr->file, "[Usermode queues] IP %d not support\n", | ||
| 275 | db_info->queue_type); | ||
| 276 | r = -EINVAL; | ||
| 277 | goto unpin_bo; | ||
| 278 | } | ||
| 279 | |||
| 280 | index = amdgpu_doorbell_index_on_bar(uq_mgr->adev, db_obj->obj, | ||
| 281 | db_info->doorbell_offset, db_size); | ||
| 282 | drm_dbg_driver(adev_to_drm(uq_mgr->adev), | ||
| 283 | "[Usermode queues] doorbell index=%lld\n", index); | ||
| 284 | amdgpu_bo_unreserve(db_obj->obj); | ||
| 285 | return index; | ||
| 286 | |||
| 287 | unpin_bo: | ||
| 288 | amdgpu_bo_unpin(db_obj->obj); | ||
| 289 | |||
| 290 | unref_bo: | ||
| 291 | amdgpu_bo_unref(&db_obj->obj); | ||
| 292 | return r; | ||
| 293 | } | ||
| 294 | |||
| 295 | static int | ||
| 296 | amdgpu_userq_destroy(struct drm_file *filp, int queue_id) | ||
| 297 | { | ||
| 298 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | ||
| 299 | struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr; | ||
| 300 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 301 | struct amdgpu_usermode_queue *queue; | ||
| 302 | int r = 0; | ||
| 303 | |||
| 304 | cancel_delayed_work(&uq_mgr->resume_work); | ||
| 305 | mutex_lock(&uq_mgr->userq_mutex); | ||
| 306 | |||
| 307 | queue = amdgpu_userq_find(uq_mgr, queue_id); | ||
| 308 | if (!queue) { | ||
| 309 | drm_dbg_driver(adev_to_drm(uq_mgr->adev), "Invalid queue id to destroy\n"); | ||
| 310 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 311 | return -EINVAL; | ||
| 312 | } | ||
| 313 | amdgpu_userq_wait_for_last_fence(uq_mgr, queue); | ||
| 314 | r = amdgpu_userq_unmap_helper(uq_mgr, queue); | ||
| 315 | amdgpu_bo_unpin(queue->db_obj.obj); | ||
| 316 | amdgpu_bo_unref(&queue->db_obj.obj); | ||
| 317 | amdgpu_userq_cleanup(uq_mgr, queue, queue_id); | ||
| 318 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 319 | |||
| 320 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 321 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 322 | |||
| 323 | return r; | ||
| 324 | } | ||
| 325 | |||
| 326 | static int amdgpu_userq_priority_permit(struct drm_file *filp, | ||
| 327 | int priority) | ||
| 328 | { | ||
| 329 | if (priority < AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_HIGH) | ||
| 330 | return 0; | ||
| 331 | |||
| 332 | if (capable(CAP_SYS_NICE)) | ||
| 333 | return 0; | ||
| 334 | |||
| 335 | if (drm_is_current_master(filp)) | ||
| 336 | return 0; | ||
| 337 | |||
| 338 | return -EACCES; | ||
| 339 | } | ||
| 340 | |||
| 341 | static int | ||
| 342 | amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args) | ||
| 343 | { | ||
| 344 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | ||
| 345 | struct amdgpu_userq_mgr *uq_mgr = &fpriv->userq_mgr; | ||
| 346 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 347 | const struct amdgpu_userq_funcs *uq_funcs; | ||
| 348 | struct amdgpu_usermode_queue *queue; | ||
| 349 | struct amdgpu_db_info db_info; | ||
| 350 | bool skip_map_queue; | ||
| 351 | uint64_t index; | ||
| 352 | int qid, r = 0; | ||
| 353 | int priority = | ||
| 354 | (args->in.flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_MASK) >> | ||
| 355 | AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_SHIFT; | ||
| 356 | |||
| 357 | /* Usermode queues are only supported for GFX IP as of now */ | ||
| 358 | if (args->in.ip_type != AMDGPU_HW_IP_GFX && | ||
| 359 | args->in.ip_type != AMDGPU_HW_IP_DMA && | ||
| 360 | args->in.ip_type != AMDGPU_HW_IP_COMPUTE) { | ||
| 361 | drm_file_err(uq_mgr->file, "Usermode queue doesn't support IP type %u\n", | ||
| 362 | args->in.ip_type); | ||
| 363 | return -EINVAL; | ||
| 364 | } | ||
| 365 | |||
| 366 | r = amdgpu_userq_priority_permit(filp, priority); | ||
| 367 | if (r) | ||
| 368 | return r; | ||
| 369 | |||
| 370 | if ((args->in.flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE) && | ||
| 371 | (args->in.ip_type != AMDGPU_HW_IP_GFX) && | ||
| 372 | (args->in.ip_type != AMDGPU_HW_IP_COMPUTE) && | ||
| 373 | !amdgpu_is_tmz(adev)) { | ||
| 374 | drm_file_err(uq_mgr->file, "Secure only supported on GFX/Compute queues\n"); | ||
| 375 | return -EINVAL; | ||
| 376 | } | ||
| 377 | |||
| 378 | r = pm_runtime_get_sync(adev_to_drm(adev)->dev); | ||
| 379 | if (r < 0) { | ||
| 380 | drm_file_err(uq_mgr->file, "pm_runtime_get_sync() failed for userqueue create\n"); | ||
| 381 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | ||
| 382 | return r; | ||
| 383 | } | ||
| 384 | |||
| 385 | /* | ||
| 386 | * There could be a situation that we are creating a new queue while | ||
| 387 | * the other queues under this UQ_mgr are suspended. So if there is any | ||
| 388 | * resume work pending, wait for it to get done. | ||
| 389 | * | ||
| 390 | * This will also make sure we have a valid eviction fence ready to be used. | ||
| 391 | */ | ||
| 392 | amdgpu_userq_ensure_ev_fence(&fpriv->userq_mgr, &fpriv->evf_mgr); | ||
| 393 | |||
| 394 | uq_funcs = adev->userq_funcs[args->in.ip_type]; | ||
| 395 | if (!uq_funcs) { | ||
| 396 | drm_file_err(uq_mgr->file, "Usermode queue is not supported for this IP (%u)\n", | ||
| 397 | args->in.ip_type); | ||
| 398 | r = -EINVAL; | ||
| 399 | goto unlock; | ||
| 400 | } | ||
| 401 | |||
| 402 | queue = kzalloc(sizeof(struct amdgpu_usermode_queue), GFP_KERNEL); | ||
| 403 | if (!queue) { | ||
| 404 | drm_file_err(uq_mgr->file, "Failed to allocate memory for queue\n"); | ||
| 405 | r = -ENOMEM; | ||
| 406 | goto unlock; | ||
| 407 | } | ||
| 408 | queue->doorbell_handle = args->in.doorbell_handle; | ||
| 409 | queue->queue_type = args->in.ip_type; | ||
| 410 | queue->vm = &fpriv->vm; | ||
| 411 | queue->priority = priority; | ||
| 412 | |||
| 413 | db_info.queue_type = queue->queue_type; | ||
| 414 | db_info.doorbell_handle = queue->doorbell_handle; | ||
| 415 | db_info.db_obj = &queue->db_obj; | ||
| 416 | db_info.doorbell_offset = args->in.doorbell_offset; | ||
| 417 | |||
| 418 | /* Convert relative doorbell offset into absolute doorbell index */ | ||
| 419 | index = amdgpu_userq_get_doorbell_index(uq_mgr, &db_info, filp); | ||
| 420 | if (index == (uint64_t)-EINVAL) { | ||
| 421 | drm_file_err(uq_mgr->file, "Failed to get doorbell for queue\n"); | ||
| 422 | kfree(queue); | ||
| 423 | goto unlock; | ||
| 424 | } | ||
| 425 | |||
| 426 | queue->doorbell_index = index; | ||
| 427 | xa_init_flags(&queue->fence_drv_xa, XA_FLAGS_ALLOC); | ||
| 428 | r = amdgpu_userq_fence_driver_alloc(adev, queue); | ||
| 429 | if (r) { | ||
| 430 | drm_file_err(uq_mgr->file, "Failed to alloc fence driver\n"); | ||
| 431 | goto unlock; | ||
| 432 | } | ||
| 433 | |||
| 434 | r = uq_funcs->mqd_create(uq_mgr, &args->in, queue); | ||
| 435 | if (r) { | ||
| 436 | drm_file_err(uq_mgr->file, "Failed to create Queue\n"); | ||
| 437 | amdgpu_userq_fence_driver_free(queue); | ||
| 438 | kfree(queue); | ||
| 439 | goto unlock; | ||
| 440 | } | ||
| 441 | |||
| 442 | |||
| 443 | qid = idr_alloc(&uq_mgr->userq_idr, queue, 1, AMDGPU_MAX_USERQ_COUNT, GFP_KERNEL); | ||
| 444 | if (qid < 0) { | ||
| 445 | drm_file_err(uq_mgr->file, "Failed to allocate a queue id\n"); | ||
| 446 | amdgpu_userq_fence_driver_free(queue); | ||
| 447 | uq_funcs->mqd_destroy(uq_mgr, queue); | ||
| 448 | kfree(queue); | ||
| 449 | r = -ENOMEM; | ||
| 450 | goto unlock; | ||
| 451 | } | ||
| 452 | |||
| 453 | /* don't map the queue if scheduling is halted */ | ||
| 454 | mutex_lock(&adev->userq_mutex); | ||
| 455 | if (adev->userq_halt_for_enforce_isolation && | ||
| 456 | ((queue->queue_type == AMDGPU_HW_IP_GFX) || | ||
| 457 | (queue->queue_type == AMDGPU_HW_IP_COMPUTE))) | ||
| 458 | skip_map_queue = true; | ||
| 459 | else | ||
| 460 | skip_map_queue = false; | ||
| 461 | if (!skip_map_queue) { | ||
| 462 | r = amdgpu_userq_map_helper(uq_mgr, queue); | ||
| 463 | if (r) { | ||
| 464 | mutex_unlock(&adev->userq_mutex); | ||
| 465 | drm_file_err(uq_mgr->file, "Failed to map Queue\n"); | ||
| 466 | idr_remove(&uq_mgr->userq_idr, qid); | ||
| 467 | amdgpu_userq_fence_driver_free(queue); | ||
| 468 | uq_funcs->mqd_destroy(uq_mgr, queue); | ||
| 469 | kfree(queue); | ||
| 470 | goto unlock; | ||
| 471 | } | ||
| 472 | } | ||
| 473 | mutex_unlock(&adev->userq_mutex); | ||
| 474 | |||
| 475 | |||
| 476 | args->out.queue_id = qid; | ||
| 477 | |||
| 478 | unlock: | ||
| 479 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 480 | |||
| 481 | return r; | ||
| 482 | } | ||
| 483 | |||
| 484 | int amdgpu_userq_ioctl(struct drm_device *dev, void *data, | ||
| 485 | struct drm_file *filp) | ||
| 486 | { | ||
| 487 | union drm_amdgpu_userq *args = data; | ||
| 488 | int r; | ||
| 489 | |||
| 490 | switch (args->in.op) { | ||
| 491 | case AMDGPU_USERQ_OP_CREATE: | ||
| 492 | if (args->in.flags & ~(AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_MASK | | ||
| 493 | AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE)) | ||
| 494 | return -EINVAL; | ||
| 495 | r = amdgpu_userq_create(filp, args); | ||
| 496 | if (r) | ||
| 497 | drm_file_err(filp, "Failed to create usermode queue\n"); | ||
| 498 | break; | ||
| 499 | |||
| 500 | case AMDGPU_USERQ_OP_FREE: | ||
| 501 | if (args->in.ip_type || | ||
| 502 | args->in.doorbell_handle || | ||
| 503 | args->in.doorbell_offset || | ||
| 504 | args->in.flags || | ||
| 505 | args->in.queue_va || | ||
| 506 | args->in.queue_size || | ||
| 507 | args->in.rptr_va || | ||
| 508 | args->in.wptr_va || | ||
| 509 | args->in.wptr_va || | ||
| 510 | args->in.mqd || | ||
| 511 | args->in.mqd_size) | ||
| 512 | return -EINVAL; | ||
| 513 | r = amdgpu_userq_destroy(filp, args->in.queue_id); | ||
| 514 | if (r) | ||
| 515 | drm_file_err(filp, "Failed to destroy usermode queue\n"); | ||
| 516 | break; | ||
| 517 | |||
| 518 | default: | ||
| 519 | drm_dbg_driver(dev, "Invalid user queue op specified: %d\n", args->in.op); | ||
| 520 | return -EINVAL; | ||
| 521 | } | ||
| 522 | |||
| 523 | return r; | ||
| 524 | } | ||
| 525 | |||
| 526 | static int | ||
| 527 | amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) | ||
| 528 | { | ||
| 529 | struct amdgpu_usermode_queue *queue; | ||
| 530 | int queue_id; | ||
| 531 | int ret = 0, r; | ||
| 532 | |||
| 533 | /* Resume all the queues for this process */ | ||
| 534 | idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) { | ||
| 535 | r = amdgpu_userq_map_helper(uq_mgr, queue); | ||
| 536 | if (r) | ||
| 537 | ret = r; | ||
| 538 | } | ||
| 539 | |||
| 540 | if (ret) | ||
| 541 | drm_file_err(uq_mgr->file, "Failed to map all the queues\n"); | ||
| 542 | return ret; | ||
| 543 | } | ||
| 544 | |||
| 545 | static int | ||
| 546 | amdgpu_userq_validate_vm_bo(void *_unused, struct amdgpu_bo *bo) | ||
| 547 | { | ||
| 548 | struct ttm_operation_ctx ctx = { false, false }; | ||
| 549 | int ret; | ||
| 550 | |||
| 551 | amdgpu_bo_placement_from_domain(bo, bo->allowed_domains); | ||
| 552 | |||
| 553 | ret = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); | ||
| 554 | if (ret) | ||
| 555 | DRM_ERROR("Fail to validate\n"); | ||
| 556 | |||
| 557 | return ret; | ||
| 558 | } | ||
| 559 | |||
| 560 | static int | ||
| 561 | amdgpu_userq_validate_bos(struct amdgpu_userq_mgr *uq_mgr) | ||
| 562 | { | ||
| 563 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); | ||
| 564 | struct amdgpu_vm *vm = &fpriv->vm; | ||
| 565 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 566 | struct amdgpu_bo_va *bo_va; | ||
| 567 | struct ww_acquire_ctx *ticket; | ||
| 568 | struct drm_exec exec; | ||
| 569 | struct amdgpu_bo *bo; | ||
| 570 | struct dma_resv *resv; | ||
| 571 | bool clear, unlock; | ||
| 572 | int ret = 0; | ||
| 573 | |||
| 574 | drm_exec_init(&exec, DRM_EXEC_IGNORE_DUPLICATES, 0); | ||
| 575 | drm_exec_until_all_locked(&exec) { | ||
| 576 | ret = amdgpu_vm_lock_pd(vm, &exec, 2); | ||
| 577 | drm_exec_retry_on_contention(&exec); | ||
| 578 | if (unlikely(ret)) { | ||
| 579 | drm_file_err(uq_mgr->file, "Failed to lock PD\n"); | ||
| 580 | goto unlock_all; | ||
| 581 | } | ||
| 582 | |||
| 583 | /* Lock the done list */ | ||
| 584 | list_for_each_entry(bo_va, &vm->done, base.vm_status) { | ||
| 585 | bo = bo_va->base.bo; | ||
| 586 | if (!bo) | ||
| 587 | continue; | ||
| 588 | |||
| 589 | ret = drm_exec_lock_obj(&exec, &bo->tbo.base); | ||
| 590 | drm_exec_retry_on_contention(&exec); | ||
| 591 | if (unlikely(ret)) | ||
| 592 | goto unlock_all; | ||
| 593 | } | ||
| 594 | } | ||
| 595 | |||
| 596 | spin_lock(&vm->status_lock); | ||
| 597 | while (!list_empty(&vm->moved)) { | ||
| 598 | bo_va = list_first_entry(&vm->moved, struct amdgpu_bo_va, | ||
| 599 | base.vm_status); | ||
| 600 | spin_unlock(&vm->status_lock); | ||
| 601 | |||
| 602 | /* Per VM BOs never need to bo cleared in the page tables */ | ||
| 603 | ret = amdgpu_vm_bo_update(adev, bo_va, false); | ||
| 604 | if (ret) | ||
| 605 | goto unlock_all; | ||
| 606 | spin_lock(&vm->status_lock); | ||
| 607 | } | ||
| 608 | |||
| 609 | ticket = &exec.ticket; | ||
| 610 | while (!list_empty(&vm->invalidated)) { | ||
| 611 | bo_va = list_first_entry(&vm->invalidated, struct amdgpu_bo_va, | ||
| 612 | base.vm_status); | ||
| 613 | resv = bo_va->base.bo->tbo.base.resv; | ||
| 614 | spin_unlock(&vm->status_lock); | ||
| 615 | |||
| 616 | bo = bo_va->base.bo; | ||
| 617 | ret = amdgpu_userq_validate_vm_bo(NULL, bo); | ||
| 618 | if (ret) { | ||
| 619 | drm_file_err(uq_mgr->file, "Failed to validate BO\n"); | ||
| 620 | goto unlock_all; | ||
| 621 | } | ||
| 622 | |||
| 623 | /* Try to reserve the BO to avoid clearing its ptes */ | ||
| 624 | if (!adev->debug_vm && dma_resv_trylock(resv)) { | ||
| 625 | clear = false; | ||
| 626 | unlock = true; | ||
| 627 | /* The caller is already holding the reservation lock */ | ||
| 628 | } else if (dma_resv_locking_ctx(resv) == ticket) { | ||
| 629 | clear = false; | ||
| 630 | unlock = false; | ||
| 631 | /* Somebody else is using the BO right now */ | ||
| 632 | } else { | ||
| 633 | clear = true; | ||
| 634 | unlock = false; | ||
| 635 | } | ||
| 636 | |||
| 637 | ret = amdgpu_vm_bo_update(adev, bo_va, clear); | ||
| 638 | |||
| 639 | if (unlock) | ||
| 640 | dma_resv_unlock(resv); | ||
| 641 | if (ret) | ||
| 642 | goto unlock_all; | ||
| 643 | |||
| 644 | spin_lock(&vm->status_lock); | ||
| 645 | } | ||
| 646 | spin_unlock(&vm->status_lock); | ||
| 647 | |||
| 648 | ret = amdgpu_eviction_fence_replace_fence(&fpriv->evf_mgr, &exec); | ||
| 649 | if (ret) | ||
| 650 | drm_file_err(uq_mgr->file, "Failed to replace eviction fence\n"); | ||
| 651 | |||
| 652 | unlock_all: | ||
| 653 | drm_exec_fini(&exec); | ||
| 654 | return ret; | ||
| 655 | } | ||
| 656 | |||
| 657 | static void amdgpu_userq_restore_worker(struct work_struct *work) | ||
| 658 | { | ||
| 659 | struct amdgpu_userq_mgr *uq_mgr = work_to_uq_mgr(work, resume_work.work); | ||
| 660 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); | ||
| 661 | int ret; | ||
| 662 | |||
| 663 | flush_work(&fpriv->evf_mgr.suspend_work.work); | ||
| 664 | |||
| 665 | mutex_lock(&uq_mgr->userq_mutex); | ||
| 666 | |||
| 667 | ret = amdgpu_userq_validate_bos(uq_mgr); | ||
| 668 | if (ret) { | ||
| 669 | drm_file_err(uq_mgr->file, "Failed to validate BOs to restore\n"); | ||
| 670 | goto unlock; | ||
| 671 | } | ||
| 672 | |||
| 673 | ret = amdgpu_userq_restore_all(uq_mgr); | ||
| 674 | if (ret) { | ||
| 675 | drm_file_err(uq_mgr->file, "Failed to restore all queues\n"); | ||
| 676 | goto unlock; | ||
| 677 | } | ||
| 678 | |||
| 679 | unlock: | ||
| 680 | mutex_unlock(&uq_mgr->userq_mutex); | ||
| 681 | } | ||
| 682 | |||
| 683 | static int | ||
| 684 | amdgpu_userq_evict_all(struct amdgpu_userq_mgr *uq_mgr) | ||
| 685 | { | ||
| 686 | struct amdgpu_usermode_queue *queue; | ||
| 687 | int queue_id; | ||
| 688 | int ret = 0, r; | ||
| 689 | |||
| 690 | /* Try to unmap all the queues in this process ctx */ | ||
| 691 | idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) { | ||
| 692 | r = amdgpu_userq_unmap_helper(uq_mgr, queue); | ||
| 693 | if (r) | ||
| 694 | ret = r; | ||
| 695 | } | ||
| 696 | |||
| 697 | if (ret) | ||
| 698 | drm_file_err(uq_mgr->file, "Couldn't unmap all the queues\n"); | ||
| 699 | return ret; | ||
| 700 | } | ||
| 701 | |||
| 702 | static int | ||
| 703 | amdgpu_userq_wait_for_signal(struct amdgpu_userq_mgr *uq_mgr) | ||
| 704 | { | ||
| 705 | struct amdgpu_usermode_queue *queue; | ||
| 706 | int queue_id, ret; | ||
| 707 | |||
| 708 | idr_for_each_entry(&uq_mgr->userq_idr, queue, queue_id) { | ||
| 709 | struct dma_fence *f = queue->last_fence; | ||
| 710 | |||
| 711 | if (!f || dma_fence_is_signaled(f)) | ||
| 712 | continue; | ||
| 713 | ret = dma_fence_wait_timeout(f, true, msecs_to_jiffies(100)); | ||
| 714 | if (ret <= 0) { | ||
| 715 | drm_file_err(uq_mgr->file, "Timed out waiting for fence=%llu:%llu\n", | ||
| 716 | f->context, f->seqno); | ||
| 717 | return -ETIMEDOUT; | ||
| 718 | } | ||
| 719 | } | ||
| 720 | |||
| 721 | return 0; | ||
| 722 | } | ||
| 723 | |||
| 724 | void | ||
| 725 | amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, | ||
| 726 | struct amdgpu_eviction_fence *ev_fence) | ||
| 727 | { | ||
| 728 | int ret; | ||
| 729 | struct amdgpu_fpriv *fpriv = uq_mgr_to_fpriv(uq_mgr); | ||
| 730 | struct amdgpu_eviction_fence_mgr *evf_mgr = &fpriv->evf_mgr; | ||
| 731 | |||
| 732 | /* Wait for any pending userqueue fence work to finish */ | ||
| 733 | ret = amdgpu_userq_wait_for_signal(uq_mgr); | ||
| 734 | if (ret) { | ||
| 735 | drm_file_err(uq_mgr->file, "Not evicting userqueue, timeout waiting for work\n"); | ||
| 736 | return; | ||
| 737 | } | ||
| 738 | |||
| 739 | ret = amdgpu_userq_evict_all(uq_mgr); | ||
| 740 | if (ret) { | ||
| 741 | drm_file_err(uq_mgr->file, "Failed to evict userqueue\n"); | ||
| 742 | return; | ||
| 743 | } | ||
| 744 | |||
| 745 | /* Signal current eviction fence */ | ||
| 746 | amdgpu_eviction_fence_signal(evf_mgr, ev_fence); | ||
| 747 | |||
| 748 | if (evf_mgr->fd_closing) { | ||
| 749 | cancel_delayed_work(&uq_mgr->resume_work); | ||
| 750 | return; | ||
| 751 | } | ||
| 752 | |||
| 753 | /* Schedule a resume work */ | ||
| 754 | schedule_delayed_work(&uq_mgr->resume_work, 0); | ||
| 755 | } | ||
| 756 | |||
| 757 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, | ||
| 758 | struct amdgpu_device *adev) | ||
| 759 | { | ||
| 760 | mutex_init(&userq_mgr->userq_mutex); | ||
| 761 | idr_init_base(&userq_mgr->userq_idr, 1); | ||
| 762 | userq_mgr->adev = adev; | ||
| 763 | userq_mgr->file = file_priv; | ||
| 764 | |||
| 765 | mutex_lock(&adev->userq_mutex); | ||
| 766 | list_add(&userq_mgr->list, &adev->userq_mgr_list); | ||
| 767 | mutex_unlock(&adev->userq_mutex); | ||
| 768 | |||
| 769 | INIT_DELAYED_WORK(&userq_mgr->resume_work, amdgpu_userq_restore_worker); | ||
| 770 | return 0; | ||
| 771 | } | ||
| 772 | |||
| 773 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr) | ||
| 774 | { | ||
| 775 | struct amdgpu_device *adev = userq_mgr->adev; | ||
| 776 | struct amdgpu_usermode_queue *queue; | ||
| 777 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 778 | uint32_t queue_id; | ||
| 779 | |||
| 780 | cancel_delayed_work(&userq_mgr->resume_work); | ||
| 781 | |||
| 782 | mutex_lock(&userq_mgr->userq_mutex); | ||
| 783 | idr_for_each_entry(&userq_mgr->userq_idr, queue, queue_id) { | ||
| 784 | amdgpu_userq_wait_for_last_fence(userq_mgr, queue); | ||
| 785 | amdgpu_userq_unmap_helper(userq_mgr, queue); | ||
| 786 | amdgpu_userq_cleanup(userq_mgr, queue, queue_id); | ||
| 787 | } | ||
| 788 | mutex_lock(&adev->userq_mutex); | ||
| 789 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 790 | if (uqm == userq_mgr) { | ||
| 791 | list_del(&uqm->list); | ||
| 792 | break; | ||
| 793 | } | ||
| 794 | } | ||
| 795 | mutex_unlock(&adev->userq_mutex); | ||
| 796 | idr_destroy(&userq_mgr->userq_idr); | ||
| 797 | mutex_unlock(&userq_mgr->userq_mutex); | ||
| 798 | mutex_destroy(&userq_mgr->userq_mutex); | ||
| 799 | } | ||
| 800 | |||
| 801 | int amdgpu_userq_suspend(struct amdgpu_device *adev) | ||
| 802 | { | ||
| 803 | u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev); | ||
| 804 | struct amdgpu_usermode_queue *queue; | ||
| 805 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 806 | int queue_id; | ||
| 807 | int ret = 0, r; | ||
| 808 | |||
| 809 | if (!ip_mask) | ||
| 810 | return 0; | ||
| 811 | |||
| 812 | mutex_lock(&adev->userq_mutex); | ||
| 813 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 814 | cancel_delayed_work_sync(&uqm->resume_work); | ||
| 815 | mutex_lock(&uqm->userq_mutex); | ||
| 816 | idr_for_each_entry(&uqm->userq_idr, queue, queue_id) { | ||
| 817 | r = amdgpu_userq_unmap_helper(uqm, queue); | ||
| 818 | if (r) | ||
| 819 | ret = r; | ||
| 820 | } | ||
| 821 | mutex_unlock(&uqm->userq_mutex); | ||
| 822 | } | ||
| 823 | mutex_unlock(&adev->userq_mutex); | ||
| 824 | return ret; | ||
| 825 | } | ||
| 826 | |||
| 827 | int amdgpu_userq_resume(struct amdgpu_device *adev) | ||
| 828 | { | ||
| 829 | u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev); | ||
| 830 | struct amdgpu_usermode_queue *queue; | ||
| 831 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 832 | int queue_id; | ||
| 833 | int ret = 0, r; | ||
| 834 | |||
| 835 | if (!ip_mask) | ||
| 836 | return 0; | ||
| 837 | |||
| 838 | mutex_lock(&adev->userq_mutex); | ||
| 839 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 840 | mutex_lock(&uqm->userq_mutex); | ||
| 841 | idr_for_each_entry(&uqm->userq_idr, queue, queue_id) { | ||
| 842 | r = amdgpu_userq_map_helper(uqm, queue); | ||
| 843 | if (r) | ||
| 844 | ret = r; | ||
| 845 | } | ||
| 846 | mutex_unlock(&uqm->userq_mutex); | ||
| 847 | } | ||
| 848 | mutex_unlock(&adev->userq_mutex); | ||
| 849 | return ret; | ||
| 850 | } | ||
| 851 | |||
| 852 | int amdgpu_userq_stop_sched_for_enforce_isolation(struct amdgpu_device *adev, | ||
| 853 | u32 idx) | ||
| 854 | { | ||
| 855 | u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev); | ||
| 856 | struct amdgpu_usermode_queue *queue; | ||
| 857 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 858 | int queue_id; | ||
| 859 | int ret = 0, r; | ||
| 860 | |||
| 861 | /* only need to stop gfx/compute */ | ||
| 862 | if (!(ip_mask & ((1 << AMDGPU_HW_IP_GFX) | (1 << AMDGPU_HW_IP_COMPUTE)))) | ||
| 863 | return 0; | ||
| 864 | |||
| 865 | mutex_lock(&adev->userq_mutex); | ||
| 866 | if (adev->userq_halt_for_enforce_isolation) | ||
| 867 | dev_warn(adev->dev, "userq scheduling already stopped!\n"); | ||
| 868 | adev->userq_halt_for_enforce_isolation = true; | ||
| 869 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 870 | cancel_delayed_work_sync(&uqm->resume_work); | ||
| 871 | mutex_lock(&uqm->userq_mutex); | ||
| 872 | idr_for_each_entry(&uqm->userq_idr, queue, queue_id) { | ||
| 873 | if (((queue->queue_type == AMDGPU_HW_IP_GFX) || | ||
| 874 | (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) && | ||
| 875 | (queue->xcp_id == idx)) { | ||
| 876 | r = amdgpu_userq_unmap_helper(uqm, queue); | ||
| 877 | if (r) | ||
| 878 | ret = r; | ||
| 879 | } | ||
| 880 | } | ||
| 881 | mutex_unlock(&uqm->userq_mutex); | ||
| 882 | } | ||
| 883 | mutex_unlock(&adev->userq_mutex); | ||
| 884 | return ret; | ||
| 885 | } | ||
| 886 | |||
| 887 | int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev, | ||
| 888 | u32 idx) | ||
| 889 | { | ||
| 890 | u32 ip_mask = amdgpu_userq_get_supported_ip_mask(adev); | ||
| 891 | struct amdgpu_usermode_queue *queue; | ||
| 892 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 893 | int queue_id; | ||
| 894 | int ret = 0, r; | ||
| 895 | |||
| 896 | /* only need to stop gfx/compute */ | ||
| 897 | if (!(ip_mask & ((1 << AMDGPU_HW_IP_GFX) | (1 << AMDGPU_HW_IP_COMPUTE)))) | ||
| 898 | return 0; | ||
| 899 | |||
| 900 | mutex_lock(&adev->userq_mutex); | ||
| 901 | if (!adev->userq_halt_for_enforce_isolation) | ||
| 902 | dev_warn(adev->dev, "userq scheduling already started!\n"); | ||
| 903 | adev->userq_halt_for_enforce_isolation = false; | ||
| 904 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 905 | mutex_lock(&uqm->userq_mutex); | ||
| 906 | idr_for_each_entry(&uqm->userq_idr, queue, queue_id) { | ||
| 907 | if (((queue->queue_type == AMDGPU_HW_IP_GFX) || | ||
| 908 | (queue->queue_type == AMDGPU_HW_IP_COMPUTE)) && | ||
| 909 | (queue->xcp_id == idx)) { | ||
| 910 | r = amdgpu_userq_map_helper(uqm, queue); | ||
| 911 | if (r) | ||
| 912 | ret = r; | ||
| 913 | } | ||
| 914 | } | ||
| 915 | mutex_unlock(&uqm->userq_mutex); | ||
| 916 | } | ||
| 917 | mutex_unlock(&adev->userq_mutex); | ||
| 918 | return ret; | ||
| 919 | } | ||
| @@ -0,0 +1,135 @@ | |||
| 1 | /* SPDX-License-Identifier: MIT */ | ||
| 2 | /* | ||
| 3 | * Copyright 2023 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | |||
| 25 | #ifndef AMDGPU_USERQ_H_ | ||
| 26 | #define AMDGPU_USERQ_H_ | ||
| 27 | #include "amdgpu_eviction_fence.h" | ||
| 28 | |||
| 29 | #define AMDGPU_MAX_USERQ_COUNT 512 | ||
| 30 | |||
| 31 | #define to_ev_fence(f) container_of(f, struct amdgpu_eviction_fence, base) | ||
| 32 | #define uq_mgr_to_fpriv(u) container_of(u, struct amdgpu_fpriv, userq_mgr) | ||
| 33 | #define work_to_uq_mgr(w, name) container_of(w, struct amdgpu_userq_mgr, name) | ||
| 34 | |||
| 35 | enum amdgpu_userq_state { | ||
| 36 | AMDGPU_USERQ_STATE_UNMAPPED = 0, | ||
| 37 | AMDGPU_USERQ_STATE_MAPPED, | ||
| 38 | AMDGPU_USERQ_STATE_PREEMPTED, | ||
| 39 | AMDGPU_USERQ_STATE_HUNG, | ||
| 40 | }; | ||
| 41 | |||
| 42 | struct amdgpu_mqd_prop; | ||
| 43 | |||
| 44 | struct amdgpu_userq_obj { | ||
| 45 | void *cpu_ptr; | ||
| 46 | uint64_t gpu_addr; | ||
| 47 | struct amdgpu_bo *obj; | ||
| 48 | }; | ||
| 49 | |||
| 50 | struct amdgpu_usermode_queue { | ||
| 51 | int queue_type; | ||
| 52 | enum amdgpu_userq_state state; | ||
| 53 | uint64_t doorbell_handle; | ||
| 54 | uint64_t doorbell_index; | ||
| 55 | uint64_t flags; | ||
| 56 | struct amdgpu_mqd_prop *userq_prop; | ||
| 57 | struct amdgpu_userq_mgr *userq_mgr; | ||
| 58 | struct amdgpu_vm *vm; | ||
| 59 | struct amdgpu_userq_obj mqd; | ||
| 60 | struct amdgpu_userq_obj db_obj; | ||
| 61 | struct amdgpu_userq_obj fw_obj; | ||
| 62 | struct amdgpu_userq_obj wptr_obj; | ||
| 63 | struct xarray fence_drv_xa; | ||
| 64 | struct amdgpu_userq_fence_driver *fence_drv; | ||
| 65 | struct dma_fence *last_fence; | ||
| 66 | u32 xcp_id; | ||
| 67 | int priority; | ||
| 68 | }; | ||
| 69 | |||
| 70 | struct amdgpu_userq_funcs { | ||
| 71 | int (*mqd_create)(struct amdgpu_userq_mgr *uq_mgr, | ||
| 72 | struct drm_amdgpu_userq_in *args, | ||
| 73 | struct amdgpu_usermode_queue *queue); | ||
| 74 | void (*mqd_destroy)(struct amdgpu_userq_mgr *uq_mgr, | ||
| 75 | struct amdgpu_usermode_queue *uq); | ||
| 76 | int (*unmap)(struct amdgpu_userq_mgr *uq_mgr, | ||
| 77 | struct amdgpu_usermode_queue *queue); | ||
| 78 | int (*map)(struct amdgpu_userq_mgr *uq_mgr, | ||
| 79 | struct amdgpu_usermode_queue *queue); | ||
| 80 | }; | ||
| 81 | |||
| 82 | /* Usermode queues for gfx */ | ||
| 83 | struct amdgpu_userq_mgr { | ||
| 84 | struct idr userq_idr; | ||
| 85 | struct mutex userq_mutex; | ||
| 86 | struct amdgpu_device *adev; | ||
| 87 | struct delayed_work resume_work; | ||
| 88 | struct list_head list; | ||
| 89 | struct drm_file *file; | ||
| 90 | }; | ||
| 91 | |||
| 92 | struct amdgpu_db_info { | ||
| 93 | uint64_t doorbell_handle; | ||
| 94 | uint32_t queue_type; | ||
| 95 | uint32_t doorbell_offset; | ||
| 96 | struct amdgpu_userq_obj *db_obj; | ||
| 97 | }; | ||
| 98 | |||
| 99 | int amdgpu_userq_ioctl(struct drm_device *dev, void *data, struct drm_file *filp); | ||
| 100 | |||
| 101 | int amdgpu_userq_mgr_init(struct amdgpu_userq_mgr *userq_mgr, struct drm_file *file_priv, | ||
| 102 | struct amdgpu_device *adev); | ||
| 103 | |||
| 104 | void amdgpu_userq_mgr_fini(struct amdgpu_userq_mgr *userq_mgr); | ||
| 105 | |||
| 106 | int amdgpu_userq_create_object(struct amdgpu_userq_mgr *uq_mgr, | ||
| 107 | struct amdgpu_userq_obj *userq_obj, | ||
| 108 | int size); | ||
| 109 | |||
| 110 | void amdgpu_userq_destroy_object(struct amdgpu_userq_mgr *uq_mgr, | ||
| 111 | struct amdgpu_userq_obj *userq_obj); | ||
| 112 | |||
| 113 | void amdgpu_userq_evict(struct amdgpu_userq_mgr *uq_mgr, | ||
| 114 | struct amdgpu_eviction_fence *ev_fence); | ||
| 115 | |||
| 116 | int amdgpu_userq_active(struct amdgpu_userq_mgr *uq_mgr); | ||
| 117 | |||
| 118 | void amdgpu_userq_ensure_ev_fence(struct amdgpu_userq_mgr *userq_mgr, | ||
| 119 | struct amdgpu_eviction_fence_mgr *evf_mgr); | ||
| 120 | |||
| 121 | uint64_t amdgpu_userq_get_doorbell_index(struct amdgpu_userq_mgr *uq_mgr, | ||
| 122 | struct amdgpu_db_info *db_info, | ||
| 123 | struct drm_file *filp); | ||
| 124 | |||
| 125 | u32 amdgpu_userq_get_supported_ip_mask(struct amdgpu_device *adev); | ||
| 126 | |||
| 127 | int amdgpu_userq_suspend(struct amdgpu_device *adev); | ||
| 128 | int amdgpu_userq_resume(struct amdgpu_device *adev); | ||
| 129 | |||
| 130 | int amdgpu_userq_stop_sched_for_enforce_isolation(struct amdgpu_device *adev, | ||
| 131 | u32 idx); | ||
| 132 | int amdgpu_userq_start_sched_for_enforce_isolation(struct amdgpu_device *adev, | ||
| 133 | u32 idx); | ||
| 134 | |||
| 135 | #endif | ||
| @@ -0,0 +1,966 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 2 | /* | ||
| 3 | * Copyright 2023 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | |||
| 25 | #include <linux/kref.h> | ||
| 26 | #include <linux/slab.h> | ||
| 27 | #include <linux/dma-fence-unwrap.h> | ||
| 28 | |||
| 29 | #include <drm/drm_exec.h> | ||
| 30 | #include <drm/drm_syncobj.h> | ||
| 31 | |||
| 32 | #include "amdgpu.h" | ||
| 33 | #include "amdgpu_userq_fence.h" | ||
| 34 | |||
| 35 | static const struct dma_fence_ops amdgpu_userq_fence_ops; | ||
| 36 | static struct kmem_cache *amdgpu_userq_fence_slab; | ||
| 37 | |||
| 38 | int amdgpu_userq_fence_slab_init(void) | ||
| 39 | { | ||
| 40 | amdgpu_userq_fence_slab = kmem_cache_create("amdgpu_userq_fence", | ||
| 41 | sizeof(struct amdgpu_userq_fence), | ||
| 42 | 0, | ||
| 43 | SLAB_HWCACHE_ALIGN, | ||
| 44 | NULL); | ||
| 45 | if (!amdgpu_userq_fence_slab) | ||
| 46 | return -ENOMEM; | ||
| 47 | |||
| 48 | return 0; | ||
| 49 | } | ||
| 50 | |||
| 51 | void amdgpu_userq_fence_slab_fini(void) | ||
| 52 | { | ||
| 53 | rcu_barrier(); | ||
| 54 | kmem_cache_destroy(amdgpu_userq_fence_slab); | ||
| 55 | } | ||
| 56 | |||
| 57 | static inline struct amdgpu_userq_fence *to_amdgpu_userq_fence(struct dma_fence *f) | ||
| 58 | { | ||
| 59 | if (!f || f->ops != &amdgpu_userq_fence_ops) | ||
| 60 | return NULL; | ||
| 61 | |||
| 62 | return container_of(f, struct amdgpu_userq_fence, base); | ||
| 63 | } | ||
| 64 | |||
| 65 | static u64 amdgpu_userq_fence_read(struct amdgpu_userq_fence_driver *fence_drv) | ||
| 66 | { | ||
| 67 | return le64_to_cpu(*fence_drv->cpu_addr); | ||
| 68 | } | ||
| 69 | |||
| 70 | int amdgpu_userq_fence_driver_alloc(struct amdgpu_device *adev, | ||
| 71 | struct amdgpu_usermode_queue *userq) | ||
| 72 | { | ||
| 73 | struct amdgpu_userq_fence_driver *fence_drv; | ||
| 74 | unsigned long flags; | ||
| 75 | int r; | ||
| 76 | |||
| 77 | fence_drv = kzalloc(sizeof(*fence_drv), GFP_KERNEL); | ||
| 78 | if (!fence_drv) | ||
| 79 | return -ENOMEM; | ||
| 80 | |||
| 81 | /* Acquire seq64 memory */ | ||
| 82 | r = amdgpu_seq64_alloc(adev, &fence_drv->va, &fence_drv->gpu_addr, | ||
| 83 | &fence_drv->cpu_addr); | ||
| 84 | if (r) | ||
| 85 | goto free_fence_drv; | ||
| 86 | |||
| 87 | memset(fence_drv->cpu_addr, 0, sizeof(u64)); | ||
| 88 | |||
| 89 | kref_init(&fence_drv->refcount); | ||
| 90 | INIT_LIST_HEAD(&fence_drv->fences); | ||
| 91 | spin_lock_init(&fence_drv->fence_list_lock); | ||
| 92 | |||
| 93 | fence_drv->adev = adev; | ||
| 94 | fence_drv->context = dma_fence_context_alloc(1); | ||
| 95 | get_task_comm(fence_drv->timeline_name, current); | ||
| 96 | |||
| 97 | xa_lock_irqsave(&adev->userq_xa, flags); | ||
| 98 | r = xa_err(__xa_store(&adev->userq_xa, userq->doorbell_index, | ||
| 99 | fence_drv, GFP_KERNEL)); | ||
| 100 | xa_unlock_irqrestore(&adev->userq_xa, flags); | ||
| 101 | if (r) | ||
| 102 | goto free_seq64; | ||
| 103 | |||
| 104 | userq->fence_drv = fence_drv; | ||
| 105 | |||
| 106 | return 0; | ||
| 107 | |||
| 108 | free_seq64: | ||
| 109 | amdgpu_seq64_free(adev, fence_drv->va); | ||
| 110 | free_fence_drv: | ||
| 111 | kfree(fence_drv); | ||
| 112 | |||
| 113 | return r; | ||
| 114 | } | ||
| 115 | |||
| 116 | static void amdgpu_userq_walk_and_drop_fence_drv(struct xarray *xa) | ||
| 117 | { | ||
| 118 | struct amdgpu_userq_fence_driver *fence_drv; | ||
| 119 | unsigned long index; | ||
| 120 | |||
| 121 | if (xa_empty(xa)) | ||
| 122 | return; | ||
| 123 | |||
| 124 | xa_lock(xa); | ||
| 125 | xa_for_each(xa, index, fence_drv) { | ||
| 126 | __xa_erase(xa, index); | ||
| 127 | amdgpu_userq_fence_driver_put(fence_drv); | ||
| 128 | } | ||
| 129 | |||
| 130 | xa_unlock(xa); | ||
| 131 | } | ||
| 132 | |||
| 133 | void | ||
| 134 | amdgpu_userq_fence_driver_free(struct amdgpu_usermode_queue *userq) | ||
| 135 | { | ||
| 136 | amdgpu_userq_walk_and_drop_fence_drv(&userq->fence_drv_xa); | ||
| 137 | xa_destroy(&userq->fence_drv_xa); | ||
| 138 | /* Drop the fence_drv reference held by user queue */ | ||
| 139 | amdgpu_userq_fence_driver_put(userq->fence_drv); | ||
| 140 | } | ||
| 141 | |||
| 142 | void amdgpu_userq_fence_driver_process(struct amdgpu_userq_fence_driver *fence_drv) | ||
| 143 | { | ||
| 144 | struct amdgpu_userq_fence *userq_fence, *tmp; | ||
| 145 | struct dma_fence *fence; | ||
| 146 | u64 rptr; | ||
| 147 | int i; | ||
| 148 | |||
| 149 | if (!fence_drv) | ||
| 150 | return; | ||
| 151 | |||
| 152 | rptr = amdgpu_userq_fence_read(fence_drv); | ||
| 153 | |||
| 154 | spin_lock(&fence_drv->fence_list_lock); | ||
| 155 | list_for_each_entry_safe(userq_fence, tmp, &fence_drv->fences, link) { | ||
| 156 | fence = &userq_fence->base; | ||
| 157 | |||
| 158 | if (rptr < fence->seqno) | ||
| 159 | break; | ||
| 160 | |||
| 161 | dma_fence_signal(fence); | ||
| 162 | |||
| 163 | for (i = 0; i < userq_fence->fence_drv_array_count; i++) | ||
| 164 | amdgpu_userq_fence_driver_put(userq_fence->fence_drv_array[i]); | ||
| 165 | |||
| 166 | list_del(&userq_fence->link); | ||
| 167 | dma_fence_put(fence); | ||
| 168 | } | ||
| 169 | spin_unlock(&fence_drv->fence_list_lock); | ||
| 170 | } | ||
| 171 | |||
| 172 | void amdgpu_userq_fence_driver_destroy(struct kref *ref) | ||
| 173 | { | ||
| 174 | struct amdgpu_userq_fence_driver *fence_drv = container_of(ref, | ||
| 175 | struct amdgpu_userq_fence_driver, | ||
| 176 | refcount); | ||
| 177 | struct amdgpu_userq_fence_driver *xa_fence_drv; | ||
| 178 | struct amdgpu_device *adev = fence_drv->adev; | ||
| 179 | struct amdgpu_userq_fence *fence, *tmp; | ||
| 180 | struct xarray *xa = &adev->userq_xa; | ||
| 181 | unsigned long index, flags; | ||
| 182 | struct dma_fence *f; | ||
| 183 | |||
| 184 | spin_lock(&fence_drv->fence_list_lock); | ||
| 185 | list_for_each_entry_safe(fence, tmp, &fence_drv->fences, link) { | ||
| 186 | f = &fence->base; | ||
| 187 | |||
| 188 | if (!dma_fence_is_signaled(f)) { | ||
| 189 | dma_fence_set_error(f, -ECANCELED); | ||
| 190 | dma_fence_signal(f); | ||
| 191 | } | ||
| 192 | |||
| 193 | list_del(&fence->link); | ||
| 194 | dma_fence_put(f); | ||
| 195 | } | ||
| 196 | spin_unlock(&fence_drv->fence_list_lock); | ||
| 197 | |||
| 198 | xa_lock_irqsave(xa, flags); | ||
| 199 | xa_for_each(xa, index, xa_fence_drv) | ||
| 200 | if (xa_fence_drv == fence_drv) | ||
| 201 | __xa_erase(xa, index); | ||
| 202 | xa_unlock_irqrestore(xa, flags); | ||
| 203 | |||
| 204 | /* Free seq64 memory */ | ||
| 205 | amdgpu_seq64_free(adev, fence_drv->va); | ||
| 206 | kfree(fence_drv); | ||
| 207 | } | ||
| 208 | |||
| 209 | void amdgpu_userq_fence_driver_get(struct amdgpu_userq_fence_driver *fence_drv) | ||
| 210 | { | ||
| 211 | kref_get(&fence_drv->refcount); | ||
| 212 | } | ||
| 213 | |||
| 214 | void amdgpu_userq_fence_driver_put(struct amdgpu_userq_fence_driver *fence_drv) | ||
| 215 | { | ||
| 216 | kref_put(&fence_drv->refcount, amdgpu_userq_fence_driver_destroy); | ||
| 217 | } | ||
| 218 | |||
| 219 | static int amdgpu_userq_fence_alloc(struct amdgpu_userq_fence **userq_fence) | ||
| 220 | { | ||
| 221 | *userq_fence = kmem_cache_alloc(amdgpu_userq_fence_slab, GFP_ATOMIC); | ||
| 222 | return *userq_fence ? 0 : -ENOMEM; | ||
| 223 | } | ||
| 224 | |||
| 225 | static int amdgpu_userq_fence_create(struct amdgpu_usermode_queue *userq, | ||
| 226 | struct amdgpu_userq_fence *userq_fence, | ||
| 227 | u64 seq, struct dma_fence **f) | ||
| 228 | { | ||
| 229 | struct amdgpu_userq_fence_driver *fence_drv; | ||
| 230 | struct dma_fence *fence; | ||
| 231 | unsigned long flags; | ||
| 232 | |||
| 233 | fence_drv = userq->fence_drv; | ||
| 234 | if (!fence_drv) | ||
| 235 | return -EINVAL; | ||
| 236 | |||
| 237 | spin_lock_init(&userq_fence->lock); | ||
| 238 | INIT_LIST_HEAD(&userq_fence->link); | ||
| 239 | fence = &userq_fence->base; | ||
| 240 | userq_fence->fence_drv = fence_drv; | ||
| 241 | |||
| 242 | dma_fence_init(fence, &amdgpu_userq_fence_ops, &userq_fence->lock, | ||
| 243 | fence_drv->context, seq); | ||
| 244 | |||
| 245 | amdgpu_userq_fence_driver_get(fence_drv); | ||
| 246 | dma_fence_get(fence); | ||
| 247 | |||
| 248 | if (!xa_empty(&userq->fence_drv_xa)) { | ||
| 249 | struct amdgpu_userq_fence_driver *stored_fence_drv; | ||
| 250 | unsigned long index, count = 0; | ||
| 251 | int i = 0; | ||
| 252 | |||
| 253 | xa_lock(&userq->fence_drv_xa); | ||
| 254 | xa_for_each(&userq->fence_drv_xa, index, stored_fence_drv) | ||
| 255 | count++; | ||
| 256 | |||
| 257 | userq_fence->fence_drv_array = | ||
| 258 | kvmalloc_array(count, | ||
| 259 | sizeof(struct amdgpu_userq_fence_driver *), | ||
| 260 | GFP_ATOMIC); | ||
| 261 | |||
| 262 | if (userq_fence->fence_drv_array) { | ||
| 263 | xa_for_each(&userq->fence_drv_xa, index, stored_fence_drv) { | ||
| 264 | userq_fence->fence_drv_array[i] = stored_fence_drv; | ||
| 265 | __xa_erase(&userq->fence_drv_xa, index); | ||
| 266 | i++; | ||
| 267 | } | ||
| 268 | } | ||
| 269 | |||
| 270 | userq_fence->fence_drv_array_count = i; | ||
| 271 | xa_unlock(&userq->fence_drv_xa); | ||
| 272 | } else { | ||
| 273 | userq_fence->fence_drv_array = NULL; | ||
| 274 | userq_fence->fence_drv_array_count = 0; | ||
| 275 | } | ||
| 276 | |||
| 277 | /* Check if hardware has already processed the job */ | ||
| 278 | spin_lock_irqsave(&fence_drv->fence_list_lock, flags); | ||
| 279 | if (!dma_fence_is_signaled_locked(fence)) | ||
| 280 | list_add_tail(&userq_fence->link, &fence_drv->fences); | ||
| 281 | else | ||
| 282 | dma_fence_put(fence); | ||
| 283 | |||
| 284 | spin_unlock_irqrestore(&fence_drv->fence_list_lock, flags); | ||
| 285 | |||
| 286 | *f = fence; | ||
| 287 | |||
| 288 | return 0; | ||
| 289 | } | ||
| 290 | |||
| 291 | static const char *amdgpu_userq_fence_get_driver_name(struct dma_fence *f) | ||
| 292 | { | ||
| 293 | return "amdgpu_userq_fence"; | ||
| 294 | } | ||
| 295 | |||
| 296 | static const char *amdgpu_userq_fence_get_timeline_name(struct dma_fence *f) | ||
| 297 | { | ||
| 298 | struct amdgpu_userq_fence *fence = to_amdgpu_userq_fence(f); | ||
| 299 | |||
| 300 | return fence->fence_drv->timeline_name; | ||
| 301 | } | ||
| 302 | |||
| 303 | static bool amdgpu_userq_fence_signaled(struct dma_fence *f) | ||
| 304 | { | ||
| 305 | struct amdgpu_userq_fence *fence = to_amdgpu_userq_fence(f); | ||
| 306 | struct amdgpu_userq_fence_driver *fence_drv = fence->fence_drv; | ||
| 307 | u64 rptr, wptr; | ||
| 308 | |||
| 309 | rptr = amdgpu_userq_fence_read(fence_drv); | ||
| 310 | wptr = fence->base.seqno; | ||
| 311 | |||
| 312 | if (rptr >= wptr) | ||
| 313 | return true; | ||
| 314 | |||
| 315 | return false; | ||
| 316 | } | ||
| 317 | |||
| 318 | static void amdgpu_userq_fence_free(struct rcu_head *rcu) | ||
| 319 | { | ||
| 320 | struct dma_fence *fence = container_of(rcu, struct dma_fence, rcu); | ||
| 321 | struct amdgpu_userq_fence *userq_fence = to_amdgpu_userq_fence(fence); | ||
| 322 | struct amdgpu_userq_fence_driver *fence_drv = userq_fence->fence_drv; | ||
| 323 | |||
| 324 | /* Release the fence driver reference */ | ||
| 325 | amdgpu_userq_fence_driver_put(fence_drv); | ||
| 326 | |||
| 327 | kvfree(userq_fence->fence_drv_array); | ||
| 328 | kmem_cache_free(amdgpu_userq_fence_slab, userq_fence); | ||
| 329 | } | ||
| 330 | |||
| 331 | static void amdgpu_userq_fence_release(struct dma_fence *f) | ||
| 332 | { | ||
| 333 | call_rcu(&f->rcu, amdgpu_userq_fence_free); | ||
| 334 | } | ||
| 335 | |||
| 336 | static const struct dma_fence_ops amdgpu_userq_fence_ops = { | ||
| 337 | .use_64bit_seqno = true, | ||
| 338 | .get_driver_name = amdgpu_userq_fence_get_driver_name, | ||
| 339 | .get_timeline_name = amdgpu_userq_fence_get_timeline_name, | ||
| 340 | .signaled = amdgpu_userq_fence_signaled, | ||
| 341 | .release = amdgpu_userq_fence_release, | ||
| 342 | }; | ||
| 343 | |||
| 344 | /** | ||
| 345 | * amdgpu_userq_fence_read_wptr - Read the userq wptr value | ||
| 346 | * | ||
| 347 | * @queue: user mode queue structure pointer | ||
| 348 | * @wptr: write pointer value | ||
| 349 | * | ||
| 350 | * Read the wptr value from userq's MQD. The userq signal IOCTL | ||
| 351 | * creates a dma_fence for the shared buffers that expects the | ||
| 352 | * RPTR value written to seq64 memory >= WPTR. | ||
| 353 | * | ||
| 354 | * Returns wptr value on success, error on failure. | ||
| 355 | */ | ||
| 356 | static int amdgpu_userq_fence_read_wptr(struct amdgpu_usermode_queue *queue, | ||
| 357 | u64 *wptr) | ||
| 358 | { | ||
| 359 | struct amdgpu_bo_va_mapping *mapping; | ||
| 360 | struct amdgpu_bo *bo; | ||
| 361 | u64 addr, *ptr; | ||
| 362 | int r; | ||
| 363 | |||
| 364 | r = amdgpu_bo_reserve(queue->vm->root.bo, false); | ||
| 365 | if (r) | ||
| 366 | return r; | ||
| 367 | |||
| 368 | addr = queue->userq_prop->wptr_gpu_addr; | ||
| 369 | addr &= AMDGPU_GMC_HOLE_MASK; | ||
| 370 | |||
| 371 | mapping = amdgpu_vm_bo_lookup_mapping(queue->vm, addr >> PAGE_SHIFT); | ||
| 372 | if (!mapping) { | ||
| 373 | amdgpu_bo_unreserve(queue->vm->root.bo); | ||
| 374 | DRM_ERROR("Failed to lookup amdgpu_bo_va_mapping\n"); | ||
| 375 | return -EINVAL; | ||
| 376 | } | ||
| 377 | |||
| 378 | bo = amdgpu_bo_ref(mapping->bo_va->base.bo); | ||
| 379 | amdgpu_bo_unreserve(queue->vm->root.bo); | ||
| 380 | r = amdgpu_bo_reserve(bo, true); | ||
| 381 | if (r) { | ||
| 382 | DRM_ERROR("Failed to reserve userqueue wptr bo"); | ||
| 383 | return r; | ||
| 384 | } | ||
| 385 | |||
| 386 | r = amdgpu_bo_kmap(bo, (void **)&ptr); | ||
| 387 | if (r) { | ||
| 388 | DRM_ERROR("Failed mapping the userqueue wptr bo"); | ||
| 389 | goto map_error; | ||
| 390 | } | ||
| 391 | |||
| 392 | *wptr = le64_to_cpu(*ptr); | ||
| 393 | |||
| 394 | amdgpu_bo_kunmap(bo); | ||
| 395 | amdgpu_bo_unreserve(bo); | ||
| 396 | amdgpu_bo_unref(&bo); | ||
| 397 | |||
| 398 | return 0; | ||
| 399 | |||
| 400 | map_error: | ||
| 401 | amdgpu_bo_unreserve(bo); | ||
| 402 | amdgpu_bo_unref(&bo); | ||
| 403 | |||
| 404 | return r; | ||
| 405 | } | ||
| 406 | |||
| 407 | static void amdgpu_userq_fence_cleanup(struct dma_fence *fence) | ||
| 408 | { | ||
| 409 | dma_fence_put(fence); | ||
| 410 | } | ||
| 411 | |||
| 412 | int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data, | ||
| 413 | struct drm_file *filp) | ||
| 414 | { | ||
| 415 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | ||
| 416 | struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr; | ||
| 417 | struct drm_amdgpu_userq_signal *args = data; | ||
| 418 | struct drm_gem_object **gobj_write = NULL; | ||
| 419 | struct drm_gem_object **gobj_read = NULL; | ||
| 420 | struct amdgpu_usermode_queue *queue; | ||
| 421 | struct amdgpu_userq_fence *userq_fence; | ||
| 422 | struct drm_syncobj **syncobj = NULL; | ||
| 423 | u32 *bo_handles_write, num_write_bo_handles; | ||
| 424 | u32 *syncobj_handles, num_syncobj_handles; | ||
| 425 | u32 *bo_handles_read, num_read_bo_handles; | ||
| 426 | int r, i, entry, rentry, wentry; | ||
| 427 | struct dma_fence *fence; | ||
| 428 | struct drm_exec exec; | ||
| 429 | u64 wptr; | ||
| 430 | |||
| 431 | num_syncobj_handles = args->num_syncobj_handles; | ||
| 432 | syncobj_handles = memdup_user(u64_to_user_ptr(args->syncobj_handles), | ||
| 433 | sizeof(u32) * num_syncobj_handles); | ||
| 434 | if (IS_ERR(syncobj_handles)) | ||
| 435 | return PTR_ERR(syncobj_handles); | ||
| 436 | |||
| 437 | /* Array of pointers to the looked up syncobjs */ | ||
| 438 | syncobj = kmalloc_array(num_syncobj_handles, sizeof(*syncobj), GFP_KERNEL); | ||
| 439 | if (!syncobj) { | ||
| 440 | r = -ENOMEM; | ||
| 441 | goto free_syncobj_handles; | ||
| 442 | } | ||
| 443 | |||
| 444 | for (entry = 0; entry < num_syncobj_handles; entry++) { | ||
| 445 | syncobj[entry] = drm_syncobj_find(filp, syncobj_handles[entry]); | ||
| 446 | if (!syncobj[entry]) { | ||
| 447 | r = -ENOENT; | ||
| 448 | goto free_syncobj; | ||
| 449 | } | ||
| 450 | } | ||
| 451 | |||
| 452 | num_read_bo_handles = args->num_bo_read_handles; | ||
| 453 | bo_handles_read = memdup_user(u64_to_user_ptr(args->bo_read_handles), | ||
| 454 | sizeof(u32) * num_read_bo_handles); | ||
| 455 | if (IS_ERR(bo_handles_read)) { | ||
| 456 | r = PTR_ERR(bo_handles_read); | ||
| 457 | goto free_syncobj; | ||
| 458 | } | ||
| 459 | |||
| 460 | /* Array of pointers to the GEM read objects */ | ||
| 461 | gobj_read = kmalloc_array(num_read_bo_handles, sizeof(*gobj_read), GFP_KERNEL); | ||
| 462 | if (!gobj_read) { | ||
| 463 | r = -ENOMEM; | ||
| 464 | goto free_bo_handles_read; | ||
| 465 | } | ||
| 466 | |||
| 467 | for (rentry = 0; rentry < num_read_bo_handles; rentry++) { | ||
| 468 | gobj_read[rentry] = drm_gem_object_lookup(filp, bo_handles_read[rentry]); | ||
| 469 | if (!gobj_read[rentry]) { | ||
| 470 | r = -ENOENT; | ||
| 471 | goto put_gobj_read; | ||
| 472 | } | ||
| 473 | } | ||
| 474 | |||
| 475 | num_write_bo_handles = args->num_bo_write_handles; | ||
| 476 | bo_handles_write = memdup_user(u64_to_user_ptr(args->bo_write_handles), | ||
| 477 | sizeof(u32) * num_write_bo_handles); | ||
| 478 | if (IS_ERR(bo_handles_write)) { | ||
| 479 | r = PTR_ERR(bo_handles_write); | ||
| 480 | goto put_gobj_read; | ||
| 481 | } | ||
| 482 | |||
| 483 | /* Array of pointers to the GEM write objects */ | ||
| 484 | gobj_write = kmalloc_array(num_write_bo_handles, sizeof(*gobj_write), GFP_KERNEL); | ||
| 485 | if (!gobj_write) { | ||
| 486 | r = -ENOMEM; | ||
| 487 | goto free_bo_handles_write; | ||
| 488 | } | ||
| 489 | |||
| 490 | for (wentry = 0; wentry < num_write_bo_handles; wentry++) { | ||
| 491 | gobj_write[wentry] = drm_gem_object_lookup(filp, bo_handles_write[wentry]); | ||
| 492 | if (!gobj_write[wentry]) { | ||
| 493 | r = -ENOENT; | ||
| 494 | goto put_gobj_write; | ||
| 495 | } | ||
| 496 | } | ||
| 497 | |||
| 498 | /* Retrieve the user queue */ | ||
| 499 | queue = idr_find(&userq_mgr->userq_idr, args->queue_id); | ||
| 500 | if (!queue) { | ||
| 501 | r = -ENOENT; | ||
| 502 | goto put_gobj_write; | ||
| 503 | } | ||
| 504 | |||
| 505 | r = amdgpu_userq_fence_read_wptr(queue, &wptr); | ||
| 506 | if (r) | ||
| 507 | goto put_gobj_write; | ||
| 508 | |||
| 509 | r = amdgpu_userq_fence_alloc(&userq_fence); | ||
| 510 | if (r) | ||
| 511 | goto put_gobj_write; | ||
| 512 | |||
| 513 | /* We are here means UQ is active, make sure the eviction fence is valid */ | ||
| 514 | amdgpu_userq_ensure_ev_fence(&fpriv->userq_mgr, &fpriv->evf_mgr); | ||
| 515 | |||
| 516 | /* Create a new fence */ | ||
| 517 | r = amdgpu_userq_fence_create(queue, userq_fence, wptr, &fence); | ||
| 518 | if (r) { | ||
| 519 | mutex_unlock(&userq_mgr->userq_mutex); | ||
| 520 | kmem_cache_free(amdgpu_userq_fence_slab, userq_fence); | ||
| 521 | goto put_gobj_write; | ||
| 522 | } | ||
| 523 | |||
| 524 | dma_fence_put(queue->last_fence); | ||
| 525 | queue->last_fence = dma_fence_get(fence); | ||
| 526 | mutex_unlock(&userq_mgr->userq_mutex); | ||
| 527 | |||
| 528 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, | ||
| 529 | (num_read_bo_handles + num_write_bo_handles)); | ||
| 530 | |||
| 531 | /* Lock all BOs with retry handling */ | ||
| 532 | drm_exec_until_all_locked(&exec) { | ||
| 533 | r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1); | ||
| 534 | drm_exec_retry_on_contention(&exec); | ||
| 535 | if (r) { | ||
| 536 | amdgpu_userq_fence_cleanup(fence); | ||
| 537 | goto exec_fini; | ||
| 538 | } | ||
| 539 | |||
| 540 | r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1); | ||
| 541 | drm_exec_retry_on_contention(&exec); | ||
| 542 | if (r) { | ||
| 543 | amdgpu_userq_fence_cleanup(fence); | ||
| 544 | goto exec_fini; | ||
| 545 | } | ||
| 546 | } | ||
| 547 | |||
| 548 | for (i = 0; i < num_read_bo_handles; i++) { | ||
| 549 | if (!gobj_read || !gobj_read[i]->resv) | ||
| 550 | continue; | ||
| 551 | |||
| 552 | dma_resv_add_fence(gobj_read[i]->resv, fence, | ||
| 553 | DMA_RESV_USAGE_READ); | ||
| 554 | } | ||
| 555 | |||
| 556 | for (i = 0; i < num_write_bo_handles; i++) { | ||
| 557 | if (!gobj_write || !gobj_write[i]->resv) | ||
| 558 | continue; | ||
| 559 | |||
| 560 | dma_resv_add_fence(gobj_write[i]->resv, fence, | ||
| 561 | DMA_RESV_USAGE_WRITE); | ||
| 562 | } | ||
| 563 | |||
| 564 | /* Add the created fence to syncobj/BO's */ | ||
| 565 | for (i = 0; i < num_syncobj_handles; i++) | ||
| 566 | drm_syncobj_replace_fence(syncobj[i], fence); | ||
| 567 | |||
| 568 | /* drop the reference acquired in fence creation function */ | ||
| 569 | dma_fence_put(fence); | ||
| 570 | |||
| 571 | exec_fini: | ||
| 572 | drm_exec_fini(&exec); | ||
| 573 | put_gobj_write: | ||
| 574 | while (wentry-- > 0) | ||
| 575 | drm_gem_object_put(gobj_write[wentry]); | ||
| 576 | kfree(gobj_write); | ||
| 577 | free_bo_handles_write: | ||
| 578 | kfree(bo_handles_write); | ||
| 579 | put_gobj_read: | ||
| 580 | while (rentry-- > 0) | ||
| 581 | drm_gem_object_put(gobj_read[rentry]); | ||
| 582 | kfree(gobj_read); | ||
| 583 | free_bo_handles_read: | ||
| 584 | kfree(bo_handles_read); | ||
| 585 | free_syncobj: | ||
| 586 | while (entry-- > 0) | ||
| 587 | if (syncobj[entry]) | ||
| 588 | drm_syncobj_put(syncobj[entry]); | ||
| 589 | kfree(syncobj); | ||
| 590 | free_syncobj_handles: | ||
| 591 | kfree(syncobj_handles); | ||
| 592 | |||
| 593 | return r; | ||
| 594 | } | ||
| 595 | |||
| 596 | int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data, | ||
| 597 | struct drm_file *filp) | ||
| 598 | { | ||
| 599 | u32 *syncobj_handles, *timeline_points, *timeline_handles, *bo_handles_read, *bo_handles_write; | ||
| 600 | u32 num_syncobj, num_read_bo_handles, num_write_bo_handles; | ||
| 601 | struct drm_amdgpu_userq_fence_info *fence_info = NULL; | ||
| 602 | struct drm_amdgpu_userq_wait *wait_info = data; | ||
| 603 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | ||
| 604 | struct amdgpu_userq_mgr *userq_mgr = &fpriv->userq_mgr; | ||
| 605 | struct amdgpu_usermode_queue *waitq; | ||
| 606 | struct drm_gem_object **gobj_write; | ||
| 607 | struct drm_gem_object **gobj_read; | ||
| 608 | struct dma_fence **fences = NULL; | ||
| 609 | u16 num_points, num_fences = 0; | ||
| 610 | int r, i, rentry, wentry, cnt; | ||
| 611 | struct drm_exec exec; | ||
| 612 | |||
| 613 | num_read_bo_handles = wait_info->num_bo_read_handles; | ||
| 614 | bo_handles_read = memdup_user(u64_to_user_ptr(wait_info->bo_read_handles), | ||
| 615 | sizeof(u32) * num_read_bo_handles); | ||
| 616 | if (IS_ERR(bo_handles_read)) | ||
| 617 | return PTR_ERR(bo_handles_read); | ||
| 618 | |||
| 619 | num_write_bo_handles = wait_info->num_bo_write_handles; | ||
| 620 | bo_handles_write = memdup_user(u64_to_user_ptr(wait_info->bo_write_handles), | ||
| 621 | sizeof(u32) * num_write_bo_handles); | ||
| 622 | if (IS_ERR(bo_handles_write)) { | ||
| 623 | r = PTR_ERR(bo_handles_write); | ||
| 624 | goto free_bo_handles_read; | ||
| 625 | } | ||
| 626 | |||
| 627 | num_syncobj = wait_info->num_syncobj_handles; | ||
| 628 | syncobj_handles = memdup_user(u64_to_user_ptr(wait_info->syncobj_handles), | ||
| 629 | sizeof(u32) * num_syncobj); | ||
| 630 | if (IS_ERR(syncobj_handles)) { | ||
| 631 | r = PTR_ERR(syncobj_handles); | ||
| 632 | goto free_bo_handles_write; | ||
| 633 | } | ||
| 634 | |||
| 635 | num_points = wait_info->num_syncobj_timeline_handles; | ||
| 636 | timeline_handles = memdup_user(u64_to_user_ptr(wait_info->syncobj_timeline_handles), | ||
| 637 | sizeof(u32) * num_points); | ||
| 638 | if (IS_ERR(timeline_handles)) { | ||
| 639 | r = PTR_ERR(timeline_handles); | ||
| 640 | goto free_syncobj_handles; | ||
| 641 | } | ||
| 642 | |||
| 643 | timeline_points = memdup_user(u64_to_user_ptr(wait_info->syncobj_timeline_points), | ||
| 644 | sizeof(u32) * num_points); | ||
| 645 | if (IS_ERR(timeline_points)) { | ||
| 646 | r = PTR_ERR(timeline_points); | ||
| 647 | goto free_timeline_handles; | ||
| 648 | } | ||
| 649 | |||
| 650 | gobj_read = kmalloc_array(num_read_bo_handles, sizeof(*gobj_read), GFP_KERNEL); | ||
| 651 | if (!gobj_read) { | ||
| 652 | r = -ENOMEM; | ||
| 653 | goto free_timeline_points; | ||
| 654 | } | ||
| 655 | |||
| 656 | for (rentry = 0; rentry < num_read_bo_handles; rentry++) { | ||
| 657 | gobj_read[rentry] = drm_gem_object_lookup(filp, bo_handles_read[rentry]); | ||
| 658 | if (!gobj_read[rentry]) { | ||
| 659 | r = -ENOENT; | ||
| 660 | goto put_gobj_read; | ||
| 661 | } | ||
| 662 | } | ||
| 663 | |||
| 664 | gobj_write = kmalloc_array(num_write_bo_handles, sizeof(*gobj_write), GFP_KERNEL); | ||
| 665 | if (!gobj_write) { | ||
| 666 | r = -ENOMEM; | ||
| 667 | goto put_gobj_read; | ||
| 668 | } | ||
| 669 | |||
| 670 | for (wentry = 0; wentry < num_write_bo_handles; wentry++) { | ||
| 671 | gobj_write[wentry] = drm_gem_object_lookup(filp, bo_handles_write[wentry]); | ||
| 672 | if (!gobj_write[wentry]) { | ||
| 673 | r = -ENOENT; | ||
| 674 | goto put_gobj_write; | ||
| 675 | } | ||
| 676 | } | ||
| 677 | |||
| 678 | drm_exec_init(&exec, DRM_EXEC_INTERRUPTIBLE_WAIT, | ||
| 679 | (num_read_bo_handles + num_write_bo_handles)); | ||
| 680 | |||
| 681 | /* Lock all BOs with retry handling */ | ||
| 682 | drm_exec_until_all_locked(&exec) { | ||
| 683 | r = drm_exec_prepare_array(&exec, gobj_read, num_read_bo_handles, 1); | ||
| 684 | drm_exec_retry_on_contention(&exec); | ||
| 685 | if (r) { | ||
| 686 | drm_exec_fini(&exec); | ||
| 687 | goto put_gobj_write; | ||
| 688 | } | ||
| 689 | |||
| 690 | r = drm_exec_prepare_array(&exec, gobj_write, num_write_bo_handles, 1); | ||
| 691 | drm_exec_retry_on_contention(&exec); | ||
| 692 | if (r) { | ||
| 693 | drm_exec_fini(&exec); | ||
| 694 | goto put_gobj_write; | ||
| 695 | } | ||
| 696 | } | ||
| 697 | |||
| 698 | if (!wait_info->num_fences) { | ||
| 699 | if (num_points) { | ||
| 700 | struct dma_fence_unwrap iter; | ||
| 701 | struct dma_fence *fence; | ||
| 702 | struct dma_fence *f; | ||
| 703 | |||
| 704 | for (i = 0; i < num_points; i++) { | ||
| 705 | r = drm_syncobj_find_fence(filp, timeline_handles[i], | ||
| 706 | timeline_points[i], | ||
| 707 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | ||
| 708 | &fence); | ||
| 709 | if (r) | ||
| 710 | goto exec_fini; | ||
| 711 | |||
| 712 | dma_fence_unwrap_for_each(f, &iter, fence) | ||
| 713 | num_fences++; | ||
| 714 | |||
| 715 | dma_fence_put(fence); | ||
| 716 | } | ||
| 717 | } | ||
| 718 | |||
| 719 | /* Count syncobj's fence */ | ||
| 720 | for (i = 0; i < num_syncobj; i++) { | ||
| 721 | struct dma_fence *fence; | ||
| 722 | |||
| 723 | r = drm_syncobj_find_fence(filp, syncobj_handles[i], | ||
| 724 | 0, | ||
| 725 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | ||
| 726 | &fence); | ||
| 727 | if (r) | ||
| 728 | goto exec_fini; | ||
| 729 | |||
| 730 | num_fences++; | ||
| 731 | dma_fence_put(fence); | ||
| 732 | } | ||
| 733 | |||
| 734 | /* Count GEM objects fence */ | ||
| 735 | for (i = 0; i < num_read_bo_handles; i++) { | ||
| 736 | struct dma_resv_iter resv_cursor; | ||
| 737 | struct dma_fence *fence; | ||
| 738 | |||
| 739 | dma_resv_for_each_fence(&resv_cursor, gobj_read[i]->resv, | ||
| 740 | DMA_RESV_USAGE_READ, fence) | ||
| 741 | num_fences++; | ||
| 742 | } | ||
| 743 | |||
| 744 | for (i = 0; i < num_write_bo_handles; i++) { | ||
| 745 | struct dma_resv_iter resv_cursor; | ||
| 746 | struct dma_fence *fence; | ||
| 747 | |||
| 748 | dma_resv_for_each_fence(&resv_cursor, gobj_write[i]->resv, | ||
| 749 | DMA_RESV_USAGE_WRITE, fence) | ||
| 750 | num_fences++; | ||
| 751 | } | ||
| 752 | |||
| 753 | /* | ||
| 754 | * Passing num_fences = 0 means that userspace doesn't want to | ||
| 755 | * retrieve userq_fence_info. If num_fences = 0 we skip filling | ||
| 756 | * userq_fence_info and return the actual number of fences on | ||
| 757 | * args->num_fences. | ||
| 758 | */ | ||
| 759 | wait_info->num_fences = num_fences; | ||
| 760 | } else { | ||
| 761 | /* Array of fence info */ | ||
| 762 | fence_info = kmalloc_array(wait_info->num_fences, sizeof(*fence_info), GFP_KERNEL); | ||
| 763 | if (!fence_info) { | ||
| 764 | r = -ENOMEM; | ||
| 765 | goto exec_fini; | ||
| 766 | } | ||
| 767 | |||
| 768 | /* Array of fences */ | ||
| 769 | fences = kmalloc_array(wait_info->num_fences, sizeof(*fences), GFP_KERNEL); | ||
| 770 | if (!fences) { | ||
| 771 | r = -ENOMEM; | ||
| 772 | goto free_fence_info; | ||
| 773 | } | ||
| 774 | |||
| 775 | /* Retrieve GEM read objects fence */ | ||
| 776 | for (i = 0; i < num_read_bo_handles; i++) { | ||
| 777 | struct dma_resv_iter resv_cursor; | ||
| 778 | struct dma_fence *fence; | ||
| 779 | |||
| 780 | dma_resv_for_each_fence(&resv_cursor, gobj_read[i]->resv, | ||
| 781 | DMA_RESV_USAGE_READ, fence) { | ||
| 782 | if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) { | ||
| 783 | r = -EINVAL; | ||
| 784 | goto free_fences; | ||
| 785 | } | ||
| 786 | |||
| 787 | fences[num_fences++] = fence; | ||
| 788 | dma_fence_get(fence); | ||
| 789 | } | ||
| 790 | } | ||
| 791 | |||
| 792 | /* Retrieve GEM write objects fence */ | ||
| 793 | for (i = 0; i < num_write_bo_handles; i++) { | ||
| 794 | struct dma_resv_iter resv_cursor; | ||
| 795 | struct dma_fence *fence; | ||
| 796 | |||
| 797 | dma_resv_for_each_fence(&resv_cursor, gobj_write[i]->resv, | ||
| 798 | DMA_RESV_USAGE_WRITE, fence) { | ||
| 799 | if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) { | ||
| 800 | r = -EINVAL; | ||
| 801 | goto free_fences; | ||
| 802 | } | ||
| 803 | |||
| 804 | fences[num_fences++] = fence; | ||
| 805 | dma_fence_get(fence); | ||
| 806 | } | ||
| 807 | } | ||
| 808 | |||
| 809 | if (num_points) { | ||
| 810 | struct dma_fence_unwrap iter; | ||
| 811 | struct dma_fence *fence; | ||
| 812 | struct dma_fence *f; | ||
| 813 | |||
| 814 | for (i = 0; i < num_points; i++) { | ||
| 815 | r = drm_syncobj_find_fence(filp, timeline_handles[i], | ||
| 816 | timeline_points[i], | ||
| 817 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | ||
| 818 | &fence); | ||
| 819 | if (r) | ||
| 820 | goto free_fences; | ||
| 821 | |||
| 822 | dma_fence_unwrap_for_each(f, &iter, fence) { | ||
| 823 | if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) { | ||
| 824 | r = -EINVAL; | ||
| 825 | goto free_fences; | ||
| 826 | } | ||
| 827 | |||
| 828 | dma_fence_get(f); | ||
| 829 | fences[num_fences++] = f; | ||
| 830 | } | ||
| 831 | |||
| 832 | dma_fence_put(fence); | ||
| 833 | } | ||
| 834 | } | ||
| 835 | |||
| 836 | /* Retrieve syncobj's fence */ | ||
| 837 | for (i = 0; i < num_syncobj; i++) { | ||
| 838 | struct dma_fence *fence; | ||
| 839 | |||
| 840 | r = drm_syncobj_find_fence(filp, syncobj_handles[i], | ||
| 841 | 0, | ||
| 842 | DRM_SYNCOBJ_WAIT_FLAGS_WAIT_FOR_SUBMIT, | ||
| 843 | &fence); | ||
| 844 | if (r) | ||
| 845 | goto free_fences; | ||
| 846 | |||
| 847 | if (WARN_ON_ONCE(num_fences >= wait_info->num_fences)) { | ||
| 848 | r = -EINVAL; | ||
| 849 | goto free_fences; | ||
| 850 | } | ||
| 851 | |||
| 852 | fences[num_fences++] = fence; | ||
| 853 | } | ||
| 854 | |||
| 855 | /* | ||
| 856 | * Keep only the latest fences to reduce the number of values | ||
| 857 | * given back to userspace. | ||
| 858 | */ | ||
| 859 | num_fences = dma_fence_dedup_array(fences, num_fences); | ||
| 860 | |||
| 861 | waitq = idr_find(&userq_mgr->userq_idr, wait_info->waitq_id); | ||
| 862 | if (!waitq) | ||
| 863 | goto free_fences; | ||
| 864 | |||
| 865 | for (i = 0, cnt = 0; i < num_fences; i++) { | ||
| 866 | struct amdgpu_userq_fence_driver *fence_drv; | ||
| 867 | struct amdgpu_userq_fence *userq_fence; | ||
| 868 | u32 index; | ||
| 869 | |||
| 870 | userq_fence = to_amdgpu_userq_fence(fences[i]); | ||
| 871 | if (!userq_fence) { | ||
| 872 | /* | ||
| 873 | * Just waiting on other driver fences should | ||
| 874 | * be good for now | ||
| 875 | */ | ||
| 876 | r = dma_fence_wait(fences[i], true); | ||
| 877 | if (r) { | ||
| 878 | dma_fence_put(fences[i]); | ||
| 879 | goto free_fences; | ||
| 880 | } | ||
| 881 | |||
| 882 | dma_fence_put(fences[i]); | ||
| 883 | continue; | ||
| 884 | } | ||
| 885 | |||
| 886 | fence_drv = userq_fence->fence_drv; | ||
| 887 | /* | ||
| 888 | * We need to make sure the user queue release their reference | ||
| 889 | * to the fence drivers at some point before queue destruction. | ||
| 890 | * Otherwise, we would gather those references until we don't | ||
| 891 | * have any more space left and crash. | ||
| 892 | */ | ||
| 893 | r = xa_alloc(&waitq->fence_drv_xa, &index, fence_drv, | ||
| 894 | xa_limit_32b, GFP_KERNEL); | ||
| 895 | if (r) | ||
| 896 | goto free_fences; | ||
| 897 | |||
| 898 | amdgpu_userq_fence_driver_get(fence_drv); | ||
| 899 | |||
| 900 | /* Store drm syncobj's gpu va address and value */ | ||
| 901 | fence_info[cnt].va = fence_drv->va; | ||
| 902 | fence_info[cnt].value = fences[i]->seqno; | ||
| 903 | |||
| 904 | dma_fence_put(fences[i]); | ||
| 905 | /* Increment the actual userq fence count */ | ||
| 906 | cnt++; | ||
| 907 | } | ||
| 908 | |||
| 909 | wait_info->num_fences = cnt; | ||
| 910 | /* Copy userq fence info to user space */ | ||
| 911 | if (copy_to_user(u64_to_user_ptr(wait_info->out_fences), | ||
| 912 | fence_info, wait_info->num_fences * sizeof(*fence_info))) { | ||
| 913 | r = -EFAULT; | ||
| 914 | goto free_fences; | ||
| 915 | } | ||
| 916 | |||
| 917 | kfree(fences); | ||
| 918 | kfree(fence_info); | ||
| 919 | } | ||
| 920 | |||
| 921 | drm_exec_fini(&exec); | ||
| 922 | for (i = 0; i < num_read_bo_handles; i++) | ||
| 923 | drm_gem_object_put(gobj_read[i]); | ||
| 924 | kfree(gobj_read); | ||
| 925 | |||
| 926 | for (i = 0; i < num_write_bo_handles; i++) | ||
| 927 | drm_gem_object_put(gobj_write[i]); | ||
| 928 | kfree(gobj_write); | ||
| 929 | |||
| 930 | kfree(timeline_points); | ||
| 931 | kfree(timeline_handles); | ||
| 932 | kfree(syncobj_handles); | ||
| 933 | kfree(bo_handles_write); | ||
| 934 | kfree(bo_handles_read); | ||
| 935 | |||
| 936 | return 0; | ||
| 937 | |||
| 938 | free_fences: | ||
| 939 | while (num_fences-- > 0) | ||
| 940 | dma_fence_put(fences[num_fences]); | ||
| 941 | kfree(fences); | ||
| 942 | free_fence_info: | ||
| 943 | kfree(fence_info); | ||
| 944 | exec_fini: | ||
| 945 | drm_exec_fini(&exec); | ||
| 946 | put_gobj_write: | ||
| 947 | while (wentry-- > 0) | ||
| 948 | drm_gem_object_put(gobj_write[wentry]); | ||
| 949 | kfree(gobj_write); | ||
| 950 | put_gobj_read: | ||
| 951 | while (rentry-- > 0) | ||
| 952 | drm_gem_object_put(gobj_read[rentry]); | ||
| 953 | kfree(gobj_read); | ||
| 954 | free_timeline_points: | ||
| 955 | kfree(timeline_points); | ||
| 956 | free_timeline_handles: | ||
| 957 | kfree(timeline_handles); | ||
| 958 | free_syncobj_handles: | ||
| 959 | kfree(syncobj_handles); | ||
| 960 | free_bo_handles_write: | ||
| 961 | kfree(bo_handles_write); | ||
| 962 | free_bo_handles_read: | ||
| 963 | kfree(bo_handles_read); | ||
| 964 | |||
| 965 | return r; | ||
| 966 | } | ||
| @@ -0,0 +1,76 @@ | |||
| 1 | /* SPDX-License-Identifier: MIT */ | ||
| 2 | /* | ||
| 3 | * Copyright 2023 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | |||
| 25 | #ifndef __AMDGPU_USERQ_FENCE_H__ | ||
| 26 | #define __AMDGPU_USERQ_FENCE_H__ | ||
| 27 | |||
| 28 | #include <linux/types.h> | ||
| 29 | |||
| 30 | #include "amdgpu_userq.h" | ||
| 31 | |||
| 32 | struct amdgpu_userq_fence { | ||
| 33 | struct dma_fence base; | ||
| 34 | /* | ||
| 35 | * This lock is necessary to synchronize the | ||
| 36 | * userqueue dma fence operations. | ||
| 37 | */ | ||
| 38 | spinlock_t lock; | ||
| 39 | struct list_head link; | ||
| 40 | unsigned long fence_drv_array_count; | ||
| 41 | struct amdgpu_userq_fence_driver *fence_drv; | ||
| 42 | struct amdgpu_userq_fence_driver **fence_drv_array; | ||
| 43 | }; | ||
| 44 | |||
| 45 | struct amdgpu_userq_fence_driver { | ||
| 46 | struct kref refcount; | ||
| 47 | u64 va; | ||
| 48 | u64 gpu_addr; | ||
| 49 | u64 *cpu_addr; | ||
| 50 | u64 context; | ||
| 51 | /* | ||
| 52 | * This lock is necesaary to synchronize the access | ||
| 53 | * to the fences list by the fence driver. | ||
| 54 | */ | ||
| 55 | spinlock_t fence_list_lock; | ||
| 56 | struct list_head fences; | ||
| 57 | struct amdgpu_device *adev; | ||
| 58 | char timeline_name[TASK_COMM_LEN]; | ||
| 59 | }; | ||
| 60 | |||
| 61 | int amdgpu_userq_fence_slab_init(void); | ||
| 62 | void amdgpu_userq_fence_slab_fini(void); | ||
| 63 | |||
| 64 | void amdgpu_userq_fence_driver_get(struct amdgpu_userq_fence_driver *fence_drv); | ||
| 65 | void amdgpu_userq_fence_driver_put(struct amdgpu_userq_fence_driver *fence_drv); | ||
| 66 | int amdgpu_userq_fence_driver_alloc(struct amdgpu_device *adev, | ||
| 67 | struct amdgpu_usermode_queue *userq); | ||
| 68 | void amdgpu_userq_fence_driver_free(struct amdgpu_usermode_queue *userq); | ||
| 69 | void amdgpu_userq_fence_driver_process(struct amdgpu_userq_fence_driver *fence_drv); | ||
| 70 | void amdgpu_userq_fence_driver_destroy(struct kref *ref); | ||
| 71 | int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data, | ||
| 72 | struct drm_file *filp); | ||
| 73 | int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data, | ||
| 74 | struct drm_file *filp); | ||
| 75 | |||
| 76 | #endif | ||
| @@ -353,9 +353,9 @@ int amdgpu_vcn_suspend(struct amdgpu_device *adev, int i) · 352 unmodified lines | |||
| 353 | 353 | ||
| 354 | cancel_delayed_work_sync(&adev->vcn.inst[i].idle_work); | 354 | cancel_delayed_work_sync(&adev->vcn.inst[i].idle_work); |
| 355 | 355 | ||
| 356 | /* err_event_athub will corrupt VCPU buffer, so we need to | 356 | /* err_event_athub and dpc recovery will corrupt VCPU buffer, so we need to |
| 357 | * restore fw data and clear buffer in amdgpu_vcn_resume() */ | 357 | * restore fw data and clear buffer in amdgpu_vcn_resume() */ |
| 358 | if (in_ras_intr) | 358 | if (in_ras_intr || adev->pcie_reset_ctx.in_link_reset) |
| 359 | return 0; | 359 | return 0; |
| 360 | 360 | ||
| 361 | return amdgpu_vcn_save_vcpu_bo_inst(adev, i); | 361 | return amdgpu_vcn_save_vcpu_bo_inst(adev, i); |
| @@ -66,7 +66,6 @@ · 65 unmodified lines | |||
| 66 | #define VCN_ENC_CMD_REG_WAIT 0x0000000c | 66 | #define VCN_ENC_CMD_REG_WAIT 0x0000000c |
| 67 | 67 | ||
| 68 | #define VCN_AON_SOC_ADDRESS_2_0 0x1f800 | 68 | #define VCN_AON_SOC_ADDRESS_2_0 0x1f800 |
| 69 | #define VCN1_AON_SOC_ADDRESS_3_0 0x48000 | ||
| 70 | #define VCN_VID_IP_ADDRESS_2_0 0x0 | 69 | #define VCN_VID_IP_ADDRESS_2_0 0x0 |
| 71 | #define VCN_AON_IP_ADDRESS_2_0 0x30000 | 70 | #define VCN_AON_IP_ADDRESS_2_0 0x30000 |
| 72 | 71 | ||
| @@ -1323,6 +1323,9 @@ static int amdgpu_virt_req_ras_err_count_internal(struct amdgpu_device *adev, bo · 1322 unmodified lines | |||
| 1323 | { | 1323 | { |
| 1324 | struct amdgpu_virt *virt = &adev->virt; | 1324 | struct amdgpu_virt *virt = &adev->virt; |
| 1325 | 1325 | ||
| 1326 | if (!virt->ops || !virt->ops->req_ras_err_count) | ||
| 1327 | return -EOPNOTSUPP; | ||
| 1328 | |||
| 1326 | /* Host allows 15 ras telemetry requests per 60 seconds. Afterwhich, the Host | 1329 | /* Host allows 15 ras telemetry requests per 60 seconds. Afterwhich, the Host |
| 1327 | * will ignore incoming guest messages. Ratelimit the guest messages to | 1330 | * will ignore incoming guest messages. Ratelimit the guest messages to |
| 1328 | * prevent guest self DOS. | 1331 | * prevent guest self DOS. |
| @@ -1378,14 +1381,16 @@ amdgpu_virt_write_cpers_to_ring(struct amdgpu_device *adev, · 49 unmodified lines | |||
| 1378 | used_size = host_telemetry->header.used_size; | 1381 | used_size = host_telemetry->header.used_size; |
| 1379 | 1382 | ||
| 1380 | if (used_size > (AMD_SRIOV_RAS_TELEMETRY_SIZE_KB << 10)) | 1383 | if (used_size > (AMD_SRIOV_RAS_TELEMETRY_SIZE_KB << 10)) |
| 1381 | return 0; | 1384 | return -EINVAL; |
| 1382 | 1385 | ||
| 1383 | cper_dump = kmemdup(&host_telemetry->body.cper_dump, used_size, GFP_KERNEL); | 1386 | cper_dump = kmemdup(&host_telemetry->body.cper_dump, used_size, GFP_KERNEL); |
| 1384 | if (!cper_dump) | 1387 | if (!cper_dump) |
| 1385 | return -ENOMEM; | 1388 | return -ENOMEM; |
| 1386 | 1389 | ||
| 1387 | if (checksum != amd_sriov_msg_checksum(cper_dump, used_size, 0, 0)) | 1390 | if (checksum != amd_sriov_msg_checksum(cper_dump, used_size, 0, 0)) { |
| 1391 | ret = -EINVAL; | ||
| 1388 | goto out; | 1392 | goto out; |
| 1393 | } | ||
| 1389 | 1394 | ||
| 1390 | *more = cper_dump->more; | 1395 | *more = cper_dump->more; |
| 1391 | 1396 | ||
| @@ -1425,7 +1430,7 @@ static int amdgpu_virt_req_ras_cper_dump_internal(struct amdgpu_device *adev) · 33 unmodified lines | |||
| 1425 | int ret = 0; | 1430 | int ret = 0; |
| 1426 | uint32_t more = 0; | 1431 | uint32_t more = 0; |
| 1427 | 1432 | ||
| 1428 | if (!amdgpu_sriov_ras_cper_en(adev)) | 1433 | if (!virt->ops || !virt->ops->req_ras_cper_dump) |
| 1429 | return -EOPNOTSUPP; | 1434 | return -EOPNOTSUPP; |
| 1430 | 1435 | ||
| 1431 | do { | 1436 | do { |
| @@ -1434,7 +1439,7 @@ static int amdgpu_virt_req_ras_cper_dump_internal(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 1434 | adev, virt->fw_reserve.ras_telemetry, &more); | 1439 | adev, virt->fw_reserve.ras_telemetry, &more); |
| 1435 | else | 1440 | else |
| 1436 | ret = 0; | 1441 | ret = 0; |
| 1437 | } while (more); | 1442 | } while (more && !ret); |
| 1438 | 1443 | ||
| 1439 | return ret; | 1444 | return ret; |
| 1440 | } | 1445 | } |
| @@ -1444,6 +1449,9 @@ int amdgpu_virt_req_ras_cper_dump(struct amdgpu_device *adev, bool force_update) · 3 unmodified lines | |||
| 1444 | struct amdgpu_virt *virt = &adev->virt; | 1449 | struct amdgpu_virt *virt = &adev->virt; |
| 1445 | int ret = 0; | 1450 | int ret = 0; |
| 1446 | 1451 | ||
| 1452 | if (!amdgpu_sriov_ras_cper_en(adev)) | ||
| 1453 | return -EOPNOTSUPP; | ||
| 1454 | |||
| 1447 | if ((__ratelimit(&virt->ras.ras_cper_dump_rs) || force_update) && | 1455 | if ((__ratelimit(&virt->ras.ras_cper_dump_rs) || force_update) && |
| 1448 | down_read_trylock(&adev->reset_domain->sem)) { | 1456 | down_read_trylock(&adev->reset_domain->sem)) { |
| 1449 | mutex_lock(&virt->ras.ras_telemetry_mutex); | 1457 | mutex_lock(&virt->ras.ras_telemetry_mutex); |
| @@ -1480,3 +1488,16 @@ bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev, · 30 unmodified lines | |||
| 1480 | 1488 | ||
| 1481 | return true; | 1489 | return true; |
| 1482 | } | 1490 | } |
| 1491 | |||
| 1492 | /* | ||
| 1493 | * amdgpu_virt_request_bad_pages() - request bad pages | ||
| 1494 | * @adev: amdgpu device. | ||
| 1495 | * Send command to GPU hypervisor to write new bad pages into the shared PF2VF region | ||
| 1496 | */ | ||
| 1497 | void amdgpu_virt_request_bad_pages(struct amdgpu_device *adev) | ||
| 1498 | { | ||
| 1499 | struct amdgpu_virt *virt = &adev->virt; | ||
| 1500 | |||
| 1501 | if (virt->ops && virt->ops->req_bad_pages) | ||
| 1502 | virt->ops->req_bad_pages(adev); | ||
| 1503 | } | ||
| @@ -97,6 +97,7 @@ struct amdgpu_virt_ops { · 96 unmodified lines | |||
| 97 | bool (*rcvd_ras_intr)(struct amdgpu_device *adev); | 97 | bool (*rcvd_ras_intr)(struct amdgpu_device *adev); |
| 98 | int (*req_ras_err_count)(struct amdgpu_device *adev); | 98 | int (*req_ras_err_count)(struct amdgpu_device *adev); |
| 99 | int (*req_ras_cper_dump)(struct amdgpu_device *adev, u64 vf_rptr); | 99 | int (*req_ras_cper_dump)(struct amdgpu_device *adev, u64 vf_rptr); |
| 100 | int (*req_bad_pages)(struct amdgpu_device *adev); | ||
| 100 | }; | 101 | }; |
| 101 | 102 | ||
| 102 | /* | 103 | /* |
| @@ -146,11 +147,13 @@ enum AMDGIM_FEATURE_FLAG { · 43 unmodified lines | |||
| 146 | 147 | ||
| 147 | enum AMDGIM_REG_ACCESS_FLAG { | 148 | enum AMDGIM_REG_ACCESS_FLAG { |
| 148 | /* Use PSP to program IH_RB_CNTL */ | 149 | /* Use PSP to program IH_RB_CNTL */ |
| 149 | AMDGIM_FEATURE_IH_REG_PSP_EN = (1 << 0), | 150 | AMDGIM_FEATURE_IH_REG_PSP_EN = (1 << 0), |
| 150 | /* Use RLC to program MMHUB regs */ | 151 | /* Use RLC to program MMHUB regs */ |
| 151 | AMDGIM_FEATURE_MMHUB_REG_RLC_EN = (1 << 1), | 152 | AMDGIM_FEATURE_MMHUB_REG_RLC_EN = (1 << 1), |
| 152 | /* Use RLC to program GC regs */ | 153 | /* Use RLC to program GC regs */ |
| 153 | AMDGIM_FEATURE_GC_REG_RLC_EN = (1 << 2), | 154 | AMDGIM_FEATURE_GC_REG_RLC_EN = (1 << 2), |
| 155 | /* Use PSP to program L1_TLB_CNTL*/ | ||
| 156 | AMDGIM_FEATURE_L1_TLB_CNTL_PSP_EN = (1 << 3), | ||
| 154 | }; | 157 | }; |
| 155 | 158 | ||
| 156 | struct amdgim_pf2vf_info_v1 { | 159 | struct amdgim_pf2vf_info_v1 { |
| @@ -260,7 +263,10 @@ struct amdgpu_virt { · 103 unmodified lines | |||
| 260 | uint32_t reg_val_offs; | 263 | uint32_t reg_val_offs; |
| 261 | struct amdgpu_irq_src ack_irq; | 264 | struct amdgpu_irq_src ack_irq; |
| 262 | struct amdgpu_irq_src rcv_irq; | 265 | struct amdgpu_irq_src rcv_irq; |
| 266 | |||
| 263 | struct work_struct flr_work; | 267 | struct work_struct flr_work; |
| 268 | struct work_struct bad_pages_work; | ||
| 269 | |||
| 264 | struct amdgpu_mm_table mm_table; | 270 | struct amdgpu_mm_table mm_table; |
| 265 | const struct amdgpu_virt_ops *ops; | 271 | const struct amdgpu_virt_ops *ops; |
| 266 | struct amdgpu_vf_error_buffer vf_errors; | 272 | struct amdgpu_vf_error_buffer vf_errors; |
| @@ -330,6 +336,10 @@ struct amdgpu_video_codec_info; · 63 unmodified lines | |||
| 330 | (amdgpu_sriov_vf((adev)) && \ | 336 | (amdgpu_sriov_vf((adev)) && \ |
| 331 | ((adev)->virt.reg_access & (AMDGIM_FEATURE_GC_REG_RLC_EN))) | 337 | ((adev)->virt.reg_access & (AMDGIM_FEATURE_GC_REG_RLC_EN))) |
| 332 | 338 | ||
| 339 | #define amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev) \ | ||
| 340 | (amdgpu_sriov_vf((adev)) && \ | ||
| 341 | ((adev)->virt.reg_access & (AMDGIM_FEATURE_L1_TLB_CNTL_PSP_EN))) | ||
| 342 | |||
| 333 | #define amdgpu_sriov_rlcg_error_report_enabled(adev) \ | 343 | #define amdgpu_sriov_rlcg_error_report_enabled(adev) \ |
| 334 | (amdgpu_sriov_reg_indirect_mmhub(adev) || amdgpu_sriov_reg_indirect_gc(adev)) | 344 | (amdgpu_sriov_reg_indirect_mmhub(adev) || amdgpu_sriov_reg_indirect_gc(adev)) |
| 335 | 345 | ||
| @@ -423,4 +433,5 @@ int amdgpu_virt_req_ras_cper_dump(struct amdgpu_device *adev, bool force_update) · 87 unmodified lines | |||
| 423 | int amdgpu_virt_ras_telemetry_post_reset(struct amdgpu_device *adev); | 433 | int amdgpu_virt_ras_telemetry_post_reset(struct amdgpu_device *adev); |
| 424 | bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev, | 434 | bool amdgpu_virt_ras_telemetry_block_en(struct amdgpu_device *adev, |
| 425 | enum amdgpu_ras_block block); | 435 | enum amdgpu_ras_block block); |
| 436 | void amdgpu_virt_request_bad_pages(struct amdgpu_device *adev); | ||
| 426 | #endif | 437 | #endif |
| @@ -787,7 +787,8 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · 786 unmodified lines | |||
| 787 | pasid_mapping_needed &= adev->gmc.gmc_funcs->emit_pasid_mapping && | 787 | pasid_mapping_needed &= adev->gmc.gmc_funcs->emit_pasid_mapping && |
| 788 | ring->funcs->emit_wreg; | 788 | ring->funcs->emit_wreg; |
| 789 | 789 | ||
| 790 | cleaner_shader_needed = adev->gfx.enable_cleaner_shader && | 790 | cleaner_shader_needed = job->run_cleaner_shader && |
| 791 | adev->gfx.enable_cleaner_shader && | ||
| 791 | ring->funcs->emit_cleaner_shader && job->base.s_fence && | 792 | ring->funcs->emit_cleaner_shader && job->base.s_fence && |
| 792 | &job->base.s_fence->scheduled == isolation->spearhead; | 793 | &job->base.s_fence->scheduled == isolation->spearhead; |
| 793 | 794 | ||
| @@ -817,7 +818,7 @@ int amdgpu_vm_flush(struct amdgpu_ring *ring, struct amdgpu_job *job, · 23 unmodified lines | |||
| 817 | if (spm_update_needed && adev->gfx.rlc.funcs->update_spm_vmid) | 818 | if (spm_update_needed && adev->gfx.rlc.funcs->update_spm_vmid) |
| 818 | adev->gfx.rlc.funcs->update_spm_vmid(adev, ring, job->vmid); | 819 | adev->gfx.rlc.funcs->update_spm_vmid(adev, ring, job->vmid); |
| 819 | 820 | ||
| 820 | if (!ring->is_mes_queue && ring->funcs->emit_gds_switch && | 821 | if (ring->funcs->emit_gds_switch && |
| 821 | gds_switch_needed) { | 822 | gds_switch_needed) { |
| 822 | amdgpu_ring_emit_gds_switch(ring, job->vmid, job->gds_base, | 823 | amdgpu_ring_emit_gds_switch(ring, job->vmid, job->gds_base, |
| 823 | job->gds_size, job->gws_base, | 824 | job->gds_size, job->gws_base, |
| @@ -296,15 +296,27 @@ static const struct amdgpu_pcs_ras_field xgmi3x16_pcs_ras_fields[] = { · 295 unmodified lines | |||
| 296 | 296 | ||
| 297 | static u32 xgmi_v6_4_get_link_status(struct amdgpu_device *adev, int global_link_num) | 297 | static u32 xgmi_v6_4_get_link_status(struct amdgpu_device *adev, int global_link_num) |
| 298 | { | 298 | { |
| 299 | const u32 smnpcs_xgmi3x16_pcs_state_hist1 = 0x11a00070; | 299 | const u32 smn_xgmi_6_4_pcs_state_hist1[2] = { 0x11a00070, 0x11b00070 }; |
| 300 | const int xgmi_inst = 2; | 300 | const u32 smn_xgmi_6_4_1_pcs_state_hist1[2] = { 0x12100070, |
| 301 | u32 link_inst; | 301 | 0x11b00070 }; |
| 302 | u32 i, n; | ||
| 302 | u64 addr; | 303 | u64 addr; |
| 303 | 304 | ||
| 304 | link_inst = global_link_num % xgmi_inst; | 305 | switch (amdgpu_ip_version(adev, XGMI_HWIP, 0)) { |
| 306 | case IP_VERSION(6, 4, 0): | ||
| 307 | n = ARRAY_SIZE(smn_xgmi_6_4_pcs_state_hist1); | ||
| 308 | addr = smn_xgmi_6_4_pcs_state_hist1[global_link_num % n]; | ||
| 309 | break; | ||
| 310 | case IP_VERSION(6, 4, 1): | ||
| 311 | n = ARRAY_SIZE(smn_xgmi_6_4_1_pcs_state_hist1); | ||
| 312 | addr = smn_xgmi_6_4_1_pcs_state_hist1[global_link_num % n]; | ||
| 313 | break; | ||
| 314 | default: | ||
| 315 | return U32_MAX; | ||
| 316 | } | ||
| 305 | 317 | ||
| 306 | addr = (smnpcs_xgmi3x16_pcs_state_hist1 | (link_inst << 20)) + | 318 | i = global_link_num / n; |
| 307 | adev->asic_funcs->encode_ext_smn_addressing(global_link_num / xgmi_inst); | 319 | addr += adev->asic_funcs->encode_ext_smn_addressing(i); |
| 308 | 320 | ||
| 309 | return RREG32_PCIE_EXT(addr); | 321 | return RREG32_PCIE_EXT(addr); |
| 310 | } | 322 | } |
| @@ -109,10 +109,11 @@ union amd_sriov_msg_feature_flags { · 108 unmodified lines | |||
| 109 | 109 | ||
| 110 | union amd_sriov_reg_access_flags { | 110 | union amd_sriov_reg_access_flags { |
| 111 | struct { | 111 | struct { |
| 112 | uint32_t vf_reg_access_ih : 1; | 112 | uint32_t vf_reg_access_ih : 1; |
| 113 | uint32_t vf_reg_access_mmhub : 1; | 113 | uint32_t vf_reg_access_mmhub : 1; |
| 114 | uint32_t vf_reg_access_gc : 1; | 114 | uint32_t vf_reg_access_gc : 1; |
| 115 | uint32_t reserved : 29; | 115 | uint32_t vf_reg_access_l1_tlb_cntl : 1; |
| 116 | uint32_t reserved : 28; | ||
| 116 | } flags; | 117 | } flags; |
| 117 | uint32_t all; | 118 | uint32_t all; |
| 118 | }; | 119 | }; |
| @@ -330,6 +331,7 @@ enum amd_sriov_mailbox_request_message { · 211 unmodified lines | |||
| 330 | MB_REQ_MSG_RAS_POISON = 202, | 331 | MB_REQ_MSG_RAS_POISON = 202, |
| 331 | MB_REQ_RAS_ERROR_COUNT = 203, | 332 | MB_REQ_RAS_ERROR_COUNT = 203, |
| 332 | MB_REQ_RAS_CPER_DUMP = 204, | 333 | MB_REQ_RAS_CPER_DUMP = 204, |
| 334 | MB_REQ_RAS_BAD_PAGES = 205, | ||
| 333 | }; | 335 | }; |
| 334 | 336 | ||
| 335 | /* mailbox message send from host to guest */ | 337 | /* mailbox message send from host to guest */ |
| @@ -347,6 +349,9 @@ enum amd_sriov_mailbox_response_message { · 11 unmodified lines | |||
| 347 | MB_RES_MSG_GPU_RMA = 10, | 349 | MB_RES_MSG_GPU_RMA = 10, |
| 348 | MB_RES_MSG_RAS_ERROR_COUNT_READY = 11, | 350 | MB_RES_MSG_RAS_ERROR_COUNT_READY = 11, |
| 349 | MB_REQ_RAS_CPER_DUMP_READY = 14, | 351 | MB_REQ_RAS_CPER_DUMP_READY = 14, |
| 352 | MB_RES_MSG_RAS_BAD_PAGES_READY = 15, | ||
| 353 | MB_RES_MSG_RAS_BAD_PAGES_NOTIFICATION = 16, | ||
| 354 | MB_RES_MSG_UNRECOV_ERR_NOTIFICATION = 17, | ||
| 350 | MB_RES_MSG_TEXT_MESSAGE = 255 | 355 | MB_RES_MSG_TEXT_MESSAGE = 255 |
| 351 | }; | 356 | }; |
| 352 | 357 | ||
| @@ -1444,6 +1444,7 @@ static void atom_get_vbios_pn(struct atom_context *ctx) · 1443 unmodified lines | |||
| 1444 | if (vbios_str == NULL) | 1444 | if (vbios_str == NULL) |
| 1445 | vbios_str += sizeof(BIOS_ATOM_PREFIX) - 1; | 1445 | vbios_str += sizeof(BIOS_ATOM_PREFIX) - 1; |
| 1446 | } | 1446 | } |
| 1447 | OPTIMIZER_HIDE_VAR(vbios_str); | ||
| 1447 | if (vbios_str != NULL && *vbios_str == 0) | 1448 | if (vbios_str != NULL && *vbios_str == 0) |
| 1448 | vbios_str++; | 1449 | vbios_str++; |
| 1449 | 1450 | ||
| @@ -56,6 +56,8 @@ static void cik_sdma_set_buffer_funcs(struct amdgpu_device *adev); · 55 unmodified lines | |||
| 56 | static void cik_sdma_set_vm_pte_funcs(struct amdgpu_device *adev); | 56 | static void cik_sdma_set_vm_pte_funcs(struct amdgpu_device *adev); |
| 57 | static int cik_sdma_soft_reset(struct amdgpu_ip_block *ip_block); | 57 | static int cik_sdma_soft_reset(struct amdgpu_ip_block *ip_block); |
| 58 | 58 | ||
| 59 | u32 amdgpu_cik_gpu_check_soft_reset(struct amdgpu_device *adev); | ||
| 60 | |||
| 59 | MODULE_FIRMWARE("amdgpu/bonaire_sdma.bin"); | 61 | MODULE_FIRMWARE("amdgpu/bonaire_sdma.bin"); |
| 60 | MODULE_FIRMWARE("amdgpu/bonaire_sdma1.bin"); | 62 | MODULE_FIRMWARE("amdgpu/bonaire_sdma1.bin"); |
| 61 | MODULE_FIRMWARE("amdgpu/hawaii_sdma.bin"); | 63 | MODULE_FIRMWARE("amdgpu/hawaii_sdma.bin"); |
| @@ -67,9 +69,6 @@ MODULE_FIRMWARE("amdgpu/kabini_sdma1.bin"); · 5 unmodified lines | |||
| 67 | MODULE_FIRMWARE("amdgpu/mullins_sdma.bin"); | 69 | MODULE_FIRMWARE("amdgpu/mullins_sdma.bin"); |
| 68 | MODULE_FIRMWARE("amdgpu/mullins_sdma1.bin"); | 70 | MODULE_FIRMWARE("amdgpu/mullins_sdma1.bin"); |
| 69 | 71 | ||
| 70 | u32 amdgpu_cik_gpu_check_soft_reset(struct amdgpu_device *adev); | ||
| 71 | |||
| 72 | |||
| 73 | static void cik_sdma_free_microcode(struct amdgpu_device *adev) | 72 | static void cik_sdma_free_microcode(struct amdgpu_device *adev) |
| 74 | { | 73 | { |
| 75 | int i; | 74 | int i; |
| @@ -993,14 +992,9 @@ static int cik_sdma_sw_fini(struct amdgpu_ip_block *ip_block) · 917 unmodified lines | |||
| 993 | 992 | ||
| 994 | static int cik_sdma_hw_init(struct amdgpu_ip_block *ip_block) | 993 | static int cik_sdma_hw_init(struct amdgpu_ip_block *ip_block) |
| 995 | { | 994 | { |
| 996 | int r; | ||
| 997 | struct amdgpu_device *adev = ip_block->adev; | 995 | struct amdgpu_device *adev = ip_block->adev; |
| 998 | 996 | ||
| 999 | r = cik_sdma_start(adev); | 997 | return cik_sdma_start(adev); |
| 1000 | if (r) | ||
| 1001 | return r; | ||
| 1002 | |||
| 1003 | return r; | ||
| 1004 | } | 998 | } |
| 1005 | 999 | ||
| 1006 | static int cik_sdma_hw_fini(struct amdgpu_ip_block *ip_block) | 1000 | static int cik_sdma_hw_fini(struct amdgpu_ip_block *ip_block) |
| @@ -1040,14 +1034,10 @@ static bool cik_sdma_is_idle(struct amdgpu_ip_block *ip_block) · 33 unmodified lines | |||
| 1040 | static int cik_sdma_wait_for_idle(struct amdgpu_ip_block *ip_block) | 1034 | static int cik_sdma_wait_for_idle(struct amdgpu_ip_block *ip_block) |
| 1041 | { | 1035 | { |
| 1042 | unsigned i; | 1036 | unsigned i; |
| 1043 | u32 tmp; | ||
| 1044 | struct amdgpu_device *adev = ip_block->adev; | 1037 | struct amdgpu_device *adev = ip_block->adev; |
| 1045 | 1038 | ||
| 1046 | for (i = 0; i < adev->usec_timeout; i++) { | 1039 | for (i = 0; i < adev->usec_timeout; i++) { |
| 1047 | tmp = RREG32(mmSRBM_STATUS2) & (SRBM_STATUS2__SDMA_BUSY_MASK | | 1040 | if (cik_sdma_is_idle(ip_block)) |
| 1048 | SRBM_STATUS2__SDMA1_BUSY_MASK); | ||
| 1049 | |||
| 1050 | if (!tmp) | ||
| 1051 | return 0; | 1041 | return 0; |
| 1052 | udelay(1); | 1042 | udelay(1); |
| 1053 | } | 1043 | } |
| @@ -60,9 +60,6 @@ · 59 unmodified lines | |||
| 60 | #define AUD5_REGISTER_OFFSET (0x179d - 0x1780) | 60 | #define AUD5_REGISTER_OFFSET (0x179d - 0x1780) |
| 61 | #define AUD6_REGISTER_OFFSET (0x17a4 - 0x1780) | 61 | #define AUD6_REGISTER_OFFSET (0x17a4 - 0x1780) |
| 62 | 62 | ||
| 63 | #define BONAIRE_GB_ADDR_CONFIG_GOLDEN 0x12010001 | ||
| 64 | #define HAWAII_GB_ADDR_CONFIG_GOLDEN 0x12011003 | ||
| 65 | |||
| 66 | #define PIPEID(x) ((x) << 0) | 63 | #define PIPEID(x) ((x) << 0) |
| 67 | #define MEID(x) ((x) << 2) | 64 | #define MEID(x) ((x) << 2) |
| 68 | #define VMID(x) ((x) << 4) | 65 | #define VMID(x) ((x) << 4) |
| @@ -3075,7 +3075,7 @@ static int dce_v10_0_set_hpd_irq_state(struct amdgpu_device *adev, · 3074 unmodified lines | |||
| 3075 | u32 tmp; | 3075 | u32 tmp; |
| 3076 | 3076 | ||
| 3077 | if (hpd >= adev->mode_info.num_hpd) { | 3077 | if (hpd >= adev->mode_info.num_hpd) { |
| 3078 | DRM_DEBUG("invalid hdp %d\n", hpd); | 3078 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 3079 | return 0; | 3079 | return 0; |
| 3080 | } | 3080 | } |
| 3081 | 3081 | ||
| @@ -3227,7 +3227,7 @@ static void dce_v10_0_hpd_int_ack(struct amdgpu_device *adev, · 145 unmodified lines | |||
| 3227 | u32 tmp; | 3227 | u32 tmp; |
| 3228 | 3228 | ||
| 3229 | if (hpd >= adev->mode_info.num_hpd) { | 3229 | if (hpd >= adev->mode_info.num_hpd) { |
| 3230 | DRM_DEBUG("invalid hdp %d\n", hpd); | 3230 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 3231 | return; | 3231 | return; |
| 3232 | } | 3232 | } |
| 3233 | 3233 | ||
| @@ -3206,7 +3206,7 @@ static int dce_v11_0_set_hpd_irq_state(struct amdgpu_device *adev, · 3205 unmodified lines | |||
| 3206 | u32 tmp; | 3206 | u32 tmp; |
| 3207 | 3207 | ||
| 3208 | if (hpd >= adev->mode_info.num_hpd) { | 3208 | if (hpd >= adev->mode_info.num_hpd) { |
| 3209 | DRM_DEBUG("invalid hdp %d\n", hpd); | 3209 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 3210 | return 0; | 3210 | return 0; |
| 3211 | } | 3211 | } |
| 3212 | 3212 | ||
| @@ -3358,7 +3358,7 @@ static void dce_v11_0_hpd_int_ack(struct amdgpu_device *adev, · 145 unmodified lines | |||
| 3358 | u32 tmp; | 3358 | u32 tmp; |
| 3359 | 3359 | ||
| 3360 | if (hpd >= adev->mode_info.num_hpd) { | 3360 | if (hpd >= adev->mode_info.num_hpd) { |
| 3361 | DRM_DEBUG("invalid hdp %d\n", hpd); | 3361 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 3362 | return; | 3362 | return; |
| 3363 | } | 3363 | } |
| 3364 | 3364 | ||
| @@ -3488,8 +3488,7 @@ static const struct amd_ip_funcs dce_v11_0_ip_funcs = { · 123 unmodified lines | |||
| 3488 | .set_powergating_state = dce_v11_0_set_powergating_state, | 3488 | .set_powergating_state = dce_v11_0_set_powergating_state, |
| 3489 | }; | 3489 | }; |
| 3490 | 3490 | ||
| 3491 | static void | 3491 | static void dce_v11_0_encoder_mode_set(struct drm_encoder *encoder, |
| 3492 | dce_v11_0_encoder_mode_set(struct drm_encoder *encoder, | ||
| 3493 | struct drm_display_mode *mode, | 3492 | struct drm_display_mode *mode, |
| 3494 | struct drm_display_mode *adjusted_mode) | 3493 | struct drm_display_mode *adjusted_mode) |
| 3495 | { | 3494 | { |
| @@ -287,7 +287,7 @@ static void dce_v6_0_hpd_int_ack(struct amdgpu_device *adev, · 286 unmodified lines | |||
| 287 | u32 tmp; | 287 | u32 tmp; |
| 288 | 288 | ||
| 289 | if (hpd >= adev->mode_info.num_hpd) { | 289 | if (hpd >= adev->mode_info.num_hpd) { |
| 290 | DRM_DEBUG("invalid hdp %d\n", hpd); | 290 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 291 | return; | 291 | return; |
| 292 | } | 292 | } |
| 293 | 293 | ||
| @@ -412,7 +412,7 @@ static void dce_v6_0_set_vga_render_state(struct amdgpu_device *adev, · 118 unmodified lines | |||
| 412 | { | 412 | { |
| 413 | if (!render) | 413 | if (!render) |
| 414 | WREG32(mmVGA_RENDER_CONTROL, | 414 | WREG32(mmVGA_RENDER_CONTROL, |
| 415 | RREG32(mmVGA_RENDER_CONTROL) & VGA_VSTATUS_CNTL); | 415 | RREG32(mmVGA_RENDER_CONTROL) & ~VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK); |
| 416 | } | 416 | } |
| 417 | 417 | ||
| 418 | static int dce_v6_0_get_num_crtc(struct amdgpu_device *adev) | 418 | static int dce_v6_0_get_num_crtc(struct amdgpu_device *adev) |
| @@ -1011,16 +1011,16 @@ static void dce_v6_0_program_watermarks(struct amdgpu_device *adev, · 592 unmodified lines | |||
| 1011 | /* select wm A */ | 1011 | /* select wm A */ |
| 1012 | arb_control3 = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset); | 1012 | arb_control3 = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset); |
| 1013 | tmp = arb_control3; | 1013 | tmp = arb_control3; |
| 1014 | tmp &= ~LATENCY_WATERMARK_MASK(3); | 1014 | tmp &= ~(3 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT); |
| 1015 | tmp |= LATENCY_WATERMARK_MASK(1); | 1015 | tmp |= (1 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT); |
| 1016 | WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp); | 1016 | WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp); |
| 1017 | WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset, | 1017 | WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset, |
| 1018 | ((latency_watermark_a << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) | | 1018 | ((latency_watermark_a << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) | |
| 1019 | (line_time << DPG_PIPE_URGENCY_CONTROL__URGENCY_HIGH_WATERMARK__SHIFT))); | 1019 | (line_time << DPG_PIPE_URGENCY_CONTROL__URGENCY_HIGH_WATERMARK__SHIFT))); |
| 1020 | /* select wm B */ | 1020 | /* select wm B */ |
| 1021 | tmp = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset); | 1021 | tmp = RREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset); |
| 1022 | tmp &= ~LATENCY_WATERMARK_MASK(3); | 1022 | tmp &= ~(3 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT); |
| 1023 | tmp |= LATENCY_WATERMARK_MASK(2); | 1023 | tmp |= (2 << DPG_PIPE_ARBITRATION_CONTROL3__URGENCY_WATERMARK_MASK__SHIFT); |
| 1024 | WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp); | 1024 | WREG32(mmDPG_PIPE_ARBITRATION_CONTROL3 + amdgpu_crtc->crtc_offset, tmp); |
| 1025 | WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset, | 1025 | WREG32(mmDPG_PIPE_URGENCY_CONTROL + amdgpu_crtc->crtc_offset, |
| 1026 | ((latency_watermark_b << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) | | 1026 | ((latency_watermark_b << DPG_PIPE_URGENCY_CONTROL__URGENCY_LOW_WATERMARK__SHIFT) | |
| @@ -1089,7 +1089,7 @@ static u32 dce_v6_0_line_buffer_adjust(struct amdgpu_device *adev, · 62 unmodified lines | |||
| 1089 | } | 1089 | } |
| 1090 | 1090 | ||
| 1091 | WREG32(mmDC_LB_MEMORY_SPLIT + amdgpu_crtc->crtc_offset, | 1091 | WREG32(mmDC_LB_MEMORY_SPLIT + amdgpu_crtc->crtc_offset, |
| 1092 | DC_LB_MEMORY_CONFIG(tmp)); | 1092 | (tmp << DC_LB_MEMORY_SPLIT__DC_LB_MEMORY_CONFIG__SHIFT)); |
| 1093 | 1093 | ||
| 1094 | WREG32(mmPIPE0_DMIF_BUFFER_CONTROL + pipe_offset, | 1094 | WREG32(mmPIPE0_DMIF_BUFFER_CONTROL + pipe_offset, |
| 1095 | (buffer_alloc << PIPE0_DMIF_BUFFER_CONTROL__DMIF_BUFFERS_ALLOCATED__SHIFT)); | 1095 | (buffer_alloc << PIPE0_DMIF_BUFFER_CONTROL__DMIF_BUFFERS_ALLOCATED__SHIFT)); |
| @@ -1306,6 +1306,7 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 210 unmodified lines | |||
| 1306 | struct amdgpu_device *adev = drm_to_adev(dev); | 1306 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 1307 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); | 1307 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); |
| 1308 | struct amdgpu_encoder_atom_dig *dig = amdgpu_encoder->enc_priv; | 1308 | struct amdgpu_encoder_atom_dig *dig = amdgpu_encoder->enc_priv; |
| 1309 | u32 offset; | ||
| 1309 | struct drm_connector *connector; | 1310 | struct drm_connector *connector; |
| 1310 | struct drm_connector_list_iter iter; | 1311 | struct drm_connector_list_iter iter; |
| 1311 | struct amdgpu_connector *amdgpu_connector = NULL; | 1312 | struct amdgpu_connector *amdgpu_connector = NULL; |
| @@ -1327,6 +1328,11 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 15 unmodified lines | |||
| 1327 | { ixAZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13, HDMI_AUDIO_CODING_TYPE_WMA_PRO }, | 1328 | { ixAZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13, HDMI_AUDIO_CODING_TYPE_WMA_PRO }, |
| 1328 | }; | 1329 | }; |
| 1329 | 1330 | ||
| 1331 | if (!dig || !dig->afmt || !dig->afmt->pin) | ||
| 1332 | return; | ||
| 1333 | |||
| 1334 | offset = dig->afmt->pin->offset; | ||
| 1335 | |||
| 1330 | drm_connector_list_iter_begin(dev, &iter); | 1336 | drm_connector_list_iter_begin(dev, &iter); |
| 1331 | drm_for_each_connector_iter(connector, &iter) { | 1337 | drm_for_each_connector_iter(connector, &iter) { |
| 1332 | if (connector->encoder == encoder) { | 1338 | if (connector->encoder == encoder) { |
| @@ -1348,7 +1354,7 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 15 unmodified lines | |||
| 1348 | return; | 1354 | return; |
| 1349 | 1355 | ||
| 1350 | for (i = 0; i < ARRAY_SIZE(eld_reg_to_type); i++) { | 1356 | for (i = 0; i < ARRAY_SIZE(eld_reg_to_type); i++) { |
| 1351 | u32 tmp = 0; | 1357 | u32 value = 0; |
| 1352 | u8 stereo_freqs = 0; | 1358 | u8 stereo_freqs = 0; |
| 1353 | int max_channels = -1; | 1359 | int max_channels = -1; |
| 1354 | int j; | 1360 | int j; |
| @@ -1358,12 +1364,12 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 3 unmodified lines | |||
| 1358 | 1364 | ||
| 1359 | if (sad->format == eld_reg_to_type[i][1]) { | 1365 | if (sad->format == eld_reg_to_type[i][1]) { |
| 1360 | if (sad->channels > max_channels) { | 1366 | if (sad->channels > max_channels) { |
| 1361 | tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0, | 1367 | value = (sad->channels << |
| 1362 | MAX_CHANNELS, sad->channels); | 1368 | AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__MAX_CHANNELS__SHIFT) | |
| 1363 | tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0, | 1369 | (sad->byte2 << |
| 1364 | DESCRIPTOR_BYTE_2, sad->byte2); | 1370 | AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__DESCRIPTOR_BYTE_2__SHIFT) | |
| 1365 | tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0, | 1371 | (sad->freq << |
| 1366 | SUPPORTED_FREQUENCIES, sad->freq); | 1372 | AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__SUPPORTED_FREQUENCIES__SHIFT); |
| 1367 | max_channels = sad->channels; | 1373 | max_channels = sad->channels; |
| 1368 | } | 1374 | } |
| 1369 | 1375 | ||
| @@ -1374,13 +1380,13 @@ static void dce_v6_0_audio_write_sad_regs(struct drm_encoder *encoder) · 4 unmodified lines | |||
| 1374 | } | 1380 | } |
| 1375 | } | 1381 | } |
| 1376 | 1382 | ||
| 1377 | tmp = REG_SET_FIELD(tmp, AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0, | 1383 | value |= (stereo_freqs << |
| 1378 | SUPPORTED_FREQUENCIES_STEREO, stereo_freqs); | 1384 | AZALIA_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0__SUPPORTED_FREQUENCIES_STEREO__SHIFT); |
| 1379 | WREG32_AUDIO_ENDPT(dig->afmt->pin->offset, eld_reg_to_type[i][0], tmp); | 1385 | |
| 1386 | WREG32_AUDIO_ENDPT(offset, eld_reg_to_type[i][0], value); | ||
| 1380 | } | 1387 | } |
| 1381 | 1388 | ||
| 1382 | kfree(sads); | 1389 | kfree(sads); |
| 1383 | |||
| 1384 | } | 1390 | } |
| 1385 | 1391 | ||
| 1386 | static void dce_v6_0_audio_enable(struct amdgpu_device *adev, | 1392 | static void dce_v6_0_audio_enable(struct amdgpu_device *adev, |
| @@ -1886,7 +1892,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 499 unmodified lines | |||
| 1886 | struct amdgpu_bo *abo; | 1892 | struct amdgpu_bo *abo; |
| 1887 | uint64_t fb_location, tiling_flags; | 1893 | uint64_t fb_location, tiling_flags; |
| 1888 | uint32_t fb_format, fb_pitch_pixels, pipe_config; | 1894 | uint32_t fb_format, fb_pitch_pixels, pipe_config; |
| 1889 | u32 fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_NONE); | 1895 | u32 fb_swap = (GRPH_ENDIAN_NONE << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1890 | u32 viewport_w, viewport_h; | 1896 | u32 viewport_w, viewport_h; |
| 1891 | int r; | 1897 | int r; |
| 1892 | bool bypass_lut = false; | 1898 | bool bypass_lut = false; |
| @@ -1926,76 +1932,76 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 33 unmodified lines | |||
| 1926 | 1932 | ||
| 1927 | switch (target_fb->format->format) { | 1933 | switch (target_fb->format->format) { |
| 1928 | case DRM_FORMAT_C8: | 1934 | case DRM_FORMAT_C8: |
| 1929 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_8BPP) | | 1935 | fb_format = ((GRPH_DEPTH_8BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1930 | GRPH_FORMAT(GRPH_FORMAT_INDEXED)); | 1936 | (GRPH_FORMAT_INDEXED << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1931 | break; | 1937 | break; |
| 1932 | case DRM_FORMAT_XRGB4444: | 1938 | case DRM_FORMAT_XRGB4444: |
| 1933 | case DRM_FORMAT_ARGB4444: | 1939 | case DRM_FORMAT_ARGB4444: |
| 1934 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) | | 1940 | fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1935 | GRPH_FORMAT(GRPH_FORMAT_ARGB4444)); | 1941 | (GRPH_FORMAT_ARGB4444 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1936 | #ifdef __BIG_ENDIAN | 1942 | #ifdef __BIG_ENDIAN |
| 1937 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16); | 1943 | fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1938 | #endif | 1944 | #endif |
| 1939 | break; | 1945 | break; |
| 1940 | case DRM_FORMAT_XRGB1555: | 1946 | case DRM_FORMAT_XRGB1555: |
| 1941 | case DRM_FORMAT_ARGB1555: | 1947 | case DRM_FORMAT_ARGB1555: |
| 1942 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) | | 1948 | fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1943 | GRPH_FORMAT(GRPH_FORMAT_ARGB1555)); | 1949 | (GRPH_FORMAT_ARGB1555 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1944 | #ifdef __BIG_ENDIAN | 1950 | #ifdef __BIG_ENDIAN |
| 1945 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16); | 1951 | fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1946 | #endif | 1952 | #endif |
| 1947 | break; | 1953 | break; |
| 1948 | case DRM_FORMAT_BGRX5551: | 1954 | case DRM_FORMAT_BGRX5551: |
| 1949 | case DRM_FORMAT_BGRA5551: | 1955 | case DRM_FORMAT_BGRA5551: |
| 1950 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) | | 1956 | fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1951 | GRPH_FORMAT(GRPH_FORMAT_BGRA5551)); | 1957 | (GRPH_FORMAT_BGRA5551 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1952 | #ifdef __BIG_ENDIAN | 1958 | #ifdef __BIG_ENDIAN |
| 1953 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16); | 1959 | fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1954 | #endif | 1960 | #endif |
| 1955 | break; | 1961 | break; |
| 1956 | case DRM_FORMAT_RGB565: | 1962 | case DRM_FORMAT_RGB565: |
| 1957 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_16BPP) | | 1963 | fb_format = ((GRPH_DEPTH_16BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1958 | GRPH_FORMAT(GRPH_FORMAT_ARGB565)); | 1964 | (GRPH_FORMAT_ARGB565 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1959 | #ifdef __BIG_ENDIAN | 1965 | #ifdef __BIG_ENDIAN |
| 1960 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN16); | 1966 | fb_swap = (GRPH_ENDIAN_8IN16 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1961 | #endif | 1967 | #endif |
| 1962 | break; | 1968 | break; |
| 1963 | case DRM_FORMAT_XRGB8888: | 1969 | case DRM_FORMAT_XRGB8888: |
| 1964 | case DRM_FORMAT_ARGB8888: | 1970 | case DRM_FORMAT_ARGB8888: |
| 1965 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) | | 1971 | fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1966 | GRPH_FORMAT(GRPH_FORMAT_ARGB8888)); | 1972 | (GRPH_FORMAT_ARGB8888 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1967 | #ifdef __BIG_ENDIAN | 1973 | #ifdef __BIG_ENDIAN |
| 1968 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32); | 1974 | fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1969 | #endif | 1975 | #endif |
| 1970 | break; | 1976 | break; |
| 1971 | case DRM_FORMAT_XRGB2101010: | 1977 | case DRM_FORMAT_XRGB2101010: |
| 1972 | case DRM_FORMAT_ARGB2101010: | 1978 | case DRM_FORMAT_ARGB2101010: |
| 1973 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) | | 1979 | fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1974 | GRPH_FORMAT(GRPH_FORMAT_ARGB2101010)); | 1980 | (GRPH_FORMAT_ARGB2101010 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1975 | #ifdef __BIG_ENDIAN | 1981 | #ifdef __BIG_ENDIAN |
| 1976 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32); | 1982 | fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1977 | #endif | 1983 | #endif |
| 1978 | /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */ | 1984 | /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */ |
| 1979 | bypass_lut = true; | 1985 | bypass_lut = true; |
| 1980 | break; | 1986 | break; |
| 1981 | case DRM_FORMAT_BGRX1010102: | 1987 | case DRM_FORMAT_BGRX1010102: |
| 1982 | case DRM_FORMAT_BGRA1010102: | 1988 | case DRM_FORMAT_BGRA1010102: |
| 1983 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) | | 1989 | fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1984 | GRPH_FORMAT(GRPH_FORMAT_BGRA1010102)); | 1990 | (GRPH_FORMAT_BGRA1010102 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1985 | #ifdef __BIG_ENDIAN | 1991 | #ifdef __BIG_ENDIAN |
| 1986 | fb_swap = GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32); | 1992 | fb_swap = (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1987 | #endif | 1993 | #endif |
| 1988 | /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */ | 1994 | /* Greater 8 bpc fb needs to bypass hw-lut to retain precision */ |
| 1989 | bypass_lut = true; | 1995 | bypass_lut = true; |
| 1990 | break; | 1996 | break; |
| 1991 | case DRM_FORMAT_XBGR8888: | 1997 | case DRM_FORMAT_XBGR8888: |
| 1992 | case DRM_FORMAT_ABGR8888: | 1998 | case DRM_FORMAT_ABGR8888: |
| 1993 | fb_format = (GRPH_DEPTH(GRPH_DEPTH_32BPP) | | 1999 | fb_format = ((GRPH_DEPTH_32BPP << GRPH_CONTROL__GRPH_DEPTH__SHIFT) | |
| 1994 | GRPH_FORMAT(GRPH_FORMAT_ARGB8888)); | 2000 | (GRPH_FORMAT_ARGB8888 << GRPH_CONTROL__GRPH_FORMAT__SHIFT)); |
| 1995 | fb_swap = (GRPH_RED_CROSSBAR(GRPH_RED_SEL_B) | | 2001 | fb_swap = ((GRPH_RED_SEL_B << GRPH_SWAP_CNTL__GRPH_RED_CROSSBAR__SHIFT) | |
| 1996 | GRPH_BLUE_CROSSBAR(GRPH_BLUE_SEL_R)); | 2002 | (GRPH_BLUE_SEL_R << GRPH_SWAP_CNTL__GRPH_BLUE_CROSSBAR__SHIFT)); |
| 1997 | #ifdef __BIG_ENDIAN | 2003 | #ifdef __BIG_ENDIAN |
| 1998 | fb_swap |= GRPH_ENDIAN_SWAP(GRPH_ENDIAN_8IN32); | 2004 | fb_swap |= (GRPH_ENDIAN_8IN32 << GRPH_SWAP_CNTL__GRPH_ENDIAN_SWAP__SHIFT); |
| 1999 | #endif | 2005 | #endif |
| 2000 | break; | 2006 | break; |
| 2001 | default: | 2007 | default: |
| @@ -2013,18 +2019,18 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 11 unmodified lines | |||
| 2013 | tile_split = AMDGPU_TILING_GET(tiling_flags, TILE_SPLIT); | 2019 | tile_split = AMDGPU_TILING_GET(tiling_flags, TILE_SPLIT); |
| 2014 | num_banks = AMDGPU_TILING_GET(tiling_flags, NUM_BANKS); | 2020 | num_banks = AMDGPU_TILING_GET(tiling_flags, NUM_BANKS); |
| 2015 | 2021 | ||
| 2016 | fb_format |= GRPH_NUM_BANKS(num_banks); | 2022 | fb_format |= (num_banks << GRPH_CONTROL__GRPH_NUM_BANKS__SHIFT); |
| 2017 | fb_format |= GRPH_ARRAY_MODE(GRPH_ARRAY_2D_TILED_THIN1); | 2023 | fb_format |= (GRPH_ARRAY_2D_TILED_THIN1 << GRPH_CONTROL__GRPH_ARRAY_MODE__SHIFT); |
| 2018 | fb_format |= GRPH_TILE_SPLIT(tile_split); | 2024 | fb_format |= (tile_split << GRPH_CONTROL__GRPH_TILE_SPLIT__SHIFT); |
| 2019 | fb_format |= GRPH_BANK_WIDTH(bankw); | 2025 | fb_format |= (bankw << GRPH_CONTROL__GRPH_BANK_WIDTH__SHIFT); |
| 2020 | fb_format |= GRPH_BANK_HEIGHT(bankh); | 2026 | fb_format |= (bankh << GRPH_CONTROL__GRPH_BANK_HEIGHT__SHIFT); |
| 2021 | fb_format |= GRPH_MACRO_TILE_ASPECT(mtaspect); | 2027 | fb_format |= (mtaspect << GRPH_CONTROL__GRPH_MACRO_TILE_ASPECT__SHIFT); |
| 2022 | } else if (AMDGPU_TILING_GET(tiling_flags, ARRAY_MODE) == ARRAY_1D_TILED_THIN1) { | 2028 | } else if (AMDGPU_TILING_GET(tiling_flags, ARRAY_MODE) == ARRAY_1D_TILED_THIN1) { |
| 2023 | fb_format |= GRPH_ARRAY_MODE(GRPH_ARRAY_1D_TILED_THIN1); | 2029 | fb_format |= (GRPH_ARRAY_1D_TILED_THIN1 << GRPH_CONTROL__GRPH_ARRAY_MODE__SHIFT); |
| 2024 | } | 2030 | } |
| 2025 | 2031 | ||
| 2026 | pipe_config = AMDGPU_TILING_GET(tiling_flags, PIPE_CONFIG); | 2032 | pipe_config = AMDGPU_TILING_GET(tiling_flags, PIPE_CONFIG); |
| 2027 | fb_format |= GRPH_PIPE_CONFIG(pipe_config); | 2033 | fb_format |= (pipe_config << GRPH_CONTROL__GRPH_PIPE_CONFIG__SHIFT); |
| 2028 | 2034 | ||
| 2029 | dce_v6_0_vga_enable(crtc, false); | 2035 | dce_v6_0_vga_enable(crtc, false); |
| 2030 | 2036 | ||
| @@ -2040,7 +2046,7 @@ static int dce_v6_0_crtc_do_set_base(struct drm_crtc *crtc, · 9 unmodified lines | |||
| 2040 | WREG32(mmGRPH_PRIMARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset, | 2046 | WREG32(mmGRPH_PRIMARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset, |
| 2041 | (u32)fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK); | 2047 | (u32)fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK); |
| 2042 | WREG32(mmGRPH_SECONDARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset, | 2048 | WREG32(mmGRPH_SECONDARY_SURFACE_ADDRESS + amdgpu_crtc->crtc_offset, |
| 2043 | (u32) fb_location & GRPH_PRIMARY_SURFACE_ADDRESS__GRPH_PRIMARY_SURFACE_ADDRESS_MASK); | 2049 | (u32) fb_location & GRPH_SECONDARY_SURFACE_ADDRESS__GRPH_SECONDARY_SURFACE_ADDRESS_MASK); |
| 2044 | WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format); | 2050 | WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format); |
| 2045 | WREG32(mmGRPH_SWAP_CNTL + amdgpu_crtc->crtc_offset, fb_swap); | 2051 | WREG32(mmGRPH_SWAP_CNTL + amdgpu_crtc->crtc_offset, fb_swap); |
| 2046 | 2052 | ||
| @@ -2108,14 +2114,13 @@ static void dce_v6_0_set_interleave(struct drm_crtc *crtc, · 61 unmodified lines | |||
| 2108 | 2114 | ||
| 2109 | if (mode->flags & DRM_MODE_FLAG_INTERLACE) | 2115 | if (mode->flags & DRM_MODE_FLAG_INTERLACE) |
| 2110 | WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, | 2116 | WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, |
| 2111 | INTERLEAVE_EN); | 2117 | DATA_FORMAT__INTERLEAVE_EN_MASK); |
| 2112 | else | 2118 | else |
| 2113 | WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, 0); | 2119 | WREG32(mmDATA_FORMAT + amdgpu_crtc->crtc_offset, 0); |
| 2114 | } | 2120 | } |
| 2115 | 2121 | ||
| 2116 | static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) | 2122 | static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) |
| 2117 | { | 2123 | { |
| 2118 | |||
| 2119 | struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); | 2124 | struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); |
| 2120 | struct drm_device *dev = crtc->dev; | 2125 | struct drm_device *dev = crtc->dev; |
| 2121 | struct amdgpu_device *adev = drm_to_adev(dev); | 2126 | struct amdgpu_device *adev = drm_to_adev(dev); |
| @@ -2125,15 +2130,15 @@ static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) · 3 unmodified lines | |||
| 2125 | DRM_DEBUG_KMS("%d\n", amdgpu_crtc->crtc_id); | 2130 | DRM_DEBUG_KMS("%d\n", amdgpu_crtc->crtc_id); |
| 2126 | 2131 | ||
| 2127 | WREG32(mmINPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset, | 2132 | WREG32(mmINPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset, |
| 2128 | ((0 << INPUT_CSC_CONTROL__INPUT_CSC_GRPH_MODE__SHIFT) | | 2133 | ((INPUT_CSC_BYPASS << INPUT_CSC_CONTROL__INPUT_CSC_GRPH_MODE__SHIFT) | |
| 2129 | (0 << INPUT_CSC_CONTROL__INPUT_CSC_OVL_MODE__SHIFT))); | 2134 | (INPUT_CSC_BYPASS << INPUT_CSC_CONTROL__INPUT_CSC_OVL_MODE__SHIFT))); |
| 2130 | WREG32(mmPRESCALE_GRPH_CONTROL + amdgpu_crtc->crtc_offset, | 2135 | WREG32(mmPRESCALE_GRPH_CONTROL + amdgpu_crtc->crtc_offset, |
| 2131 | PRESCALE_GRPH_CONTROL__GRPH_PRESCALE_BYPASS_MASK); | 2136 | PRESCALE_GRPH_CONTROL__GRPH_PRESCALE_BYPASS_MASK); |
| 2132 | WREG32(mmPRESCALE_OVL_CONTROL + amdgpu_crtc->crtc_offset, | 2137 | WREG32(mmPRESCALE_OVL_CONTROL + amdgpu_crtc->crtc_offset, |
| 2133 | PRESCALE_OVL_CONTROL__OVL_PRESCALE_BYPASS_MASK); | 2138 | PRESCALE_OVL_CONTROL__OVL_PRESCALE_BYPASS_MASK); |
| 2134 | WREG32(mmINPUT_GAMMA_CONTROL + amdgpu_crtc->crtc_offset, | 2139 | WREG32(mmINPUT_GAMMA_CONTROL + amdgpu_crtc->crtc_offset, |
| 2135 | ((0 << INPUT_GAMMA_CONTROL__GRPH_INPUT_GAMMA_MODE__SHIFT) | | 2140 | ((INPUT_GAMMA_USE_LUT << INPUT_GAMMA_CONTROL__GRPH_INPUT_GAMMA_MODE__SHIFT) | |
| 2136 | (0 << INPUT_GAMMA_CONTROL__OVL_INPUT_GAMMA_MODE__SHIFT))); | 2141 | (INPUT_GAMMA_USE_LUT << INPUT_GAMMA_CONTROL__OVL_INPUT_GAMMA_MODE__SHIFT))); |
| 2137 | 2142 | ||
| 2138 | WREG32(mmDC_LUT_CONTROL + amdgpu_crtc->crtc_offset, 0); | 2143 | WREG32(mmDC_LUT_CONTROL + amdgpu_crtc->crtc_offset, 0); |
| 2139 | 2144 | ||
| @@ -2160,19 +2165,19 @@ static void dce_v6_0_crtc_load_lut(struct drm_crtc *crtc) · 20 unmodified lines | |||
| 2160 | } | 2165 | } |
| 2161 | 2166 | ||
| 2162 | WREG32(mmDEGAMMA_CONTROL + amdgpu_crtc->crtc_offset, | 2167 | WREG32(mmDEGAMMA_CONTROL + amdgpu_crtc->crtc_offset, |
| 2163 | ((0 << DEGAMMA_CONTROL__GRPH_DEGAMMA_MODE__SHIFT) | | 2168 | ((DEGAMMA_BYPASS << DEGAMMA_CONTROL__GRPH_DEGAMMA_MODE__SHIFT) | |
| 2164 | (0 << DEGAMMA_CONTROL__OVL_DEGAMMA_MODE__SHIFT) | | 2169 | (DEGAMMA_BYPASS << DEGAMMA_CONTROL__OVL_DEGAMMA_MODE__SHIFT) | |
| 2165 | ICON_DEGAMMA_MODE(0) | | 2170 | (DEGAMMA_BYPASS << DEGAMMA_CONTROL__ICON_DEGAMMA_MODE__SHIFT) | |
| 2166 | (0 << DEGAMMA_CONTROL__CURSOR_DEGAMMA_MODE__SHIFT))); | 2171 | (DEGAMMA_BYPASS << DEGAMMA_CONTROL__CURSOR_DEGAMMA_MODE__SHIFT))); |
| 2167 | WREG32(mmGAMUT_REMAP_CONTROL + amdgpu_crtc->crtc_offset, | 2172 | WREG32(mmGAMUT_REMAP_CONTROL + amdgpu_crtc->crtc_offset, |
| 2168 | ((0 << GAMUT_REMAP_CONTROL__GRPH_GAMUT_REMAP_MODE__SHIFT) | | 2173 | ((GAMUT_REMAP_BYPASS << GAMUT_REMAP_CONTROL__GRPH_GAMUT_REMAP_MODE__SHIFT) | |
| 2169 | (0 << GAMUT_REMAP_CONTROL__OVL_GAMUT_REMAP_MODE__SHIFT))); | 2174 | (GAMUT_REMAP_BYPASS << GAMUT_REMAP_CONTROL__OVL_GAMUT_REMAP_MODE__SHIFT))); |
| 2170 | WREG32(mmREGAMMA_CONTROL + amdgpu_crtc->crtc_offset, | 2175 | WREG32(mmREGAMMA_CONTROL + amdgpu_crtc->crtc_offset, |
| 2171 | ((0 << REGAMMA_CONTROL__GRPH_REGAMMA_MODE__SHIFT) | | 2176 | ((REGAMMA_BYPASS << REGAMMA_CONTROL__GRPH_REGAMMA_MODE__SHIFT) | |
| 2172 | (0 << REGAMMA_CONTROL__OVL_REGAMMA_MODE__SHIFT))); | 2177 | (REGAMMA_BYPASS << REGAMMA_CONTROL__OVL_REGAMMA_MODE__SHIFT))); |
| 2173 | WREG32(mmOUTPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset, | 2178 | WREG32(mmOUTPUT_CSC_CONTROL + amdgpu_crtc->crtc_offset, |
| 2174 | ((0 << OUTPUT_CSC_CONTROL__OUTPUT_CSC_GRPH_MODE__SHIFT) | | 2179 | ((OUTPUT_CSC_BYPASS << OUTPUT_CSC_CONTROL__OUTPUT_CSC_GRPH_MODE__SHIFT) | |
| 2175 | (0 << OUTPUT_CSC_CONTROL__OUTPUT_CSC_OVL_MODE__SHIFT))); | 2180 | (OUTPUT_CSC_BYPASS << OUTPUT_CSC_CONTROL__OUTPUT_CSC_OVL_MODE__SHIFT))); |
| 2176 | /* XXX match this to the depth of the crtc fmt block, move to modeset? */ | 2181 | /* XXX match this to the depth of the crtc fmt block, move to modeset? */ |
| 2177 | WREG32(0x1a50 + amdgpu_crtc->crtc_offset, 0); | 2182 | WREG32(0x1a50 + amdgpu_crtc->crtc_offset, 0); |
| 2178 | 2183 | ||
| @@ -2267,8 +2272,6 @@ static void dce_v6_0_hide_cursor(struct drm_crtc *crtc) · 88 unmodified lines | |||
| 2267 | WREG32(mmCUR_CONTROL + amdgpu_crtc->crtc_offset, | 2272 | WREG32(mmCUR_CONTROL + amdgpu_crtc->crtc_offset, |
| 2268 | (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) | | 2273 | (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) | |
| 2269 | (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT)); | 2274 | (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT)); |
| 2270 | |||
| 2271 | |||
| 2272 | } | 2275 | } |
| 2273 | 2276 | ||
| 2274 | static void dce_v6_0_show_cursor(struct drm_crtc *crtc) | 2277 | static void dce_v6_0_show_cursor(struct drm_crtc *crtc) |
| @@ -2285,7 +2288,6 @@ static void dce_v6_0_show_cursor(struct drm_crtc *crtc) · 10 unmodified lines | |||
| 2285 | CUR_CONTROL__CURSOR_EN_MASK | | 2288 | CUR_CONTROL__CURSOR_EN_MASK | |
| 2286 | (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) | | 2289 | (CURSOR_24_8_PRE_MULT << CUR_CONTROL__CURSOR_MODE__SHIFT) | |
| 2287 | (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT)); | 2290 | (CURSOR_URGENT_1_2 << CUR_CONTROL__CURSOR_URGENT_CONTROL__SHIFT)); |
| 2288 | |||
| 2289 | } | 2291 | } |
| 2290 | 2292 | ||
| 2291 | static int dce_v6_0_cursor_move_locked(struct drm_crtc *crtc, | 2293 | static int dce_v6_0_cursor_move_locked(struct drm_crtc *crtc, |
| @@ -2596,7 +2598,6 @@ static bool dce_v6_0_crtc_mode_fixup(struct drm_crtc *crtc, · 304 unmodified lines | |||
| 2596 | const struct drm_display_mode *mode, | 2598 | const struct drm_display_mode *mode, |
| 2597 | struct drm_display_mode *adjusted_mode) | 2599 | struct drm_display_mode *adjusted_mode) |
| 2598 | { | 2600 | { |
| 2599 | |||
| 2600 | struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); | 2601 | struct amdgpu_crtc *amdgpu_crtc = to_amdgpu_crtc(crtc); |
| 2601 | struct drm_device *dev = crtc->dev; | 2602 | struct drm_device *dev = crtc->dev; |
| 2602 | struct drm_encoder *encoder; | 2603 | struct drm_encoder *encoder; |
| @@ -2669,7 +2670,7 @@ static void dce_v6_0_panic_flush(struct drm_plane *plane) · 66 unmodified lines | |||
| 2669 | 2670 | ||
| 2670 | /* Disable DC tiling */ | 2671 | /* Disable DC tiling */ |
| 2671 | fb_format = RREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset); | 2672 | fb_format = RREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset); |
| 2672 | fb_format &= ~GRPH_ARRAY_MODE(0x7); | 2673 | fb_format &= ~GRPH_CONTROL__GRPH_ARRAY_MODE_MASK; |
| 2673 | WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format); | 2674 | WREG32(mmGRPH_CONTROL + amdgpu_crtc->crtc_offset, fb_format); |
| 2674 | 2675 | ||
| 2675 | } | 2676 | } |
| @@ -2745,7 +2746,6 @@ static int dce_v6_0_early_init(struct amdgpu_ip_block *ip_block) · 69 unmodified lines | |||
| 2745 | static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block) | 2746 | static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block) |
| 2746 | { | 2747 | { |
| 2747 | int r, i; | 2748 | int r, i; |
| 2748 | bool ret; | ||
| 2749 | struct amdgpu_device *adev = ip_block->adev; | 2749 | struct amdgpu_device *adev = ip_block->adev; |
| 2750 | 2750 | ||
| 2751 | for (i = 0; i < adev->mode_info.num_crtc; i++) { | 2751 | for (i = 0; i < adev->mode_info.num_crtc; i++) { |
| @@ -2789,8 +2789,7 @@ static int dce_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 37 unmodified lines | |||
| 2789 | return r; | 2789 | return r; |
| 2790 | } | 2790 | } |
| 2791 | 2791 | ||
| 2792 | ret = amdgpu_atombios_get_connector_info_from_object_table(adev); | 2792 | if (amdgpu_atombios_get_connector_info_from_object_table(adev)) |
| 2793 | if (ret) | ||
| 2794 | amdgpu_display_print_display_setup(adev_to_drm(adev)); | 2793 | amdgpu_display_print_display_setup(adev_to_drm(adev)); |
| 2795 | else | 2794 | else |
| 2796 | return -EINVAL; | 2795 | return -EINVAL; |
| @@ -2986,12 +2985,12 @@ static void dce_v6_0_set_crtc_vblank_interrupt_state(struct amdgpu_device *adev, · 189 unmodified lines | |||
| 2986 | switch (state) { | 2985 | switch (state) { |
| 2987 | case AMDGPU_IRQ_STATE_DISABLE: | 2986 | case AMDGPU_IRQ_STATE_DISABLE: |
| 2988 | interrupt_mask = RREG32(mmINT_MASK + reg_block); | 2987 | interrupt_mask = RREG32(mmINT_MASK + reg_block); |
| 2989 | interrupt_mask &= ~VBLANK_INT_MASK; | 2988 | interrupt_mask &= ~INT_MASK__VBLANK_INT_MASK; |
| 2990 | WREG32(mmINT_MASK + reg_block, interrupt_mask); | 2989 | WREG32(mmINT_MASK + reg_block, interrupt_mask); |
| 2991 | break; | 2990 | break; |
| 2992 | case AMDGPU_IRQ_STATE_ENABLE: | 2991 | case AMDGPU_IRQ_STATE_ENABLE: |
| 2993 | interrupt_mask = RREG32(mmINT_MASK + reg_block); | 2992 | interrupt_mask = RREG32(mmINT_MASK + reg_block); |
| 2994 | interrupt_mask |= VBLANK_INT_MASK; | 2993 | interrupt_mask |= INT_MASK__VBLANK_INT_MASK; |
| 2995 | WREG32(mmINT_MASK + reg_block, interrupt_mask); | 2994 | WREG32(mmINT_MASK + reg_block, interrupt_mask); |
| 2996 | break; | 2995 | break; |
| 2997 | default: | 2996 | default: |
| @@ -3006,28 +3005,28 @@ static void dce_v6_0_set_crtc_vline_interrupt_state(struct amdgpu_device *adev, · 8 unmodified lines | |||
| 3006 | 3005 | ||
| 3007 | } | 3006 | } |
| 3008 | 3007 | ||
| 3009 | static int dce_v6_0_set_hpd_interrupt_state(struct amdgpu_device *adev, | 3008 | static int dce_v6_0_set_hpd_irq_state(struct amdgpu_device *adev, |
| 3010 | struct amdgpu_irq_src *src, | 3009 | struct amdgpu_irq_src *src, |
| 3011 | unsigned type, | 3010 | unsigned hpd, |
| 3012 | enum amdgpu_interrupt_state state) | 3011 | enum amdgpu_interrupt_state state) |
| 3013 | { | 3012 | { |
| 3014 | u32 dc_hpd_int_cntl; | 3013 | u32 dc_hpd_int_cntl; |
| 3015 | 3014 | ||
| 3016 | if (type >= adev->mode_info.num_hpd) { | 3015 | if (hpd >= adev->mode_info.num_hpd) { |
| 3017 | DRM_DEBUG("invalid hdp %d\n", type); | 3016 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 3018 | return 0; | 3017 | return 0; |
| 3019 | } | 3018 | } |
| 3020 | 3019 | ||
| 3021 | switch (state) { | 3020 | switch (state) { |
| 3022 | case AMDGPU_IRQ_STATE_DISABLE: | 3021 | case AMDGPU_IRQ_STATE_DISABLE: |
| 3023 | dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type]); | 3022 | dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd]); |
| 3024 | dc_hpd_int_cntl &= ~DC_HPDx_INT_EN; | 3023 | dc_hpd_int_cntl &= ~DC_HPD1_INT_CONTROL__DC_HPD1_INT_EN_MASK; |
| 3025 | WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type], dc_hpd_int_cntl); | 3024 | WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], dc_hpd_int_cntl); |
| 3026 | break; | 3025 | break; |
| 3027 | case AMDGPU_IRQ_STATE_ENABLE: | 3026 | case AMDGPU_IRQ_STATE_ENABLE: |
| 3028 | dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type]); | 3027 | dc_hpd_int_cntl = RREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd]); |
| 3029 | dc_hpd_int_cntl |= DC_HPDx_INT_EN; | 3028 | dc_hpd_int_cntl |= DC_HPD1_INT_CONTROL__DC_HPD1_INT_EN_MASK; |
| 3030 | WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[type], dc_hpd_int_cntl); | 3029 | WREG32(mmDC_HPD1_INT_CONTROL + hpd_offsets[hpd], dc_hpd_int_cntl); |
| 3031 | break; | 3030 | break; |
| 3032 | default: | 3031 | default: |
| 3033 | break; | 3032 | break; |
| @@ -3036,7 +3035,7 @@ static int dce_v6_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 2 unmodified lines | |||
| 3036 | return 0; | 3035 | return 0; |
| 3037 | } | 3036 | } |
| 3038 | 3037 | ||
| 3039 | static int dce_v6_0_set_crtc_interrupt_state(struct amdgpu_device *adev, | 3038 | static int dce_v6_0_set_crtc_irq_state(struct amdgpu_device *adev, |
| 3040 | struct amdgpu_irq_src *src, | 3039 | struct amdgpu_irq_src *src, |
| 3041 | unsigned type, | 3040 | unsigned type, |
| 3042 | enum amdgpu_interrupt_state state) | 3041 | enum amdgpu_interrupt_state state) |
| @@ -3096,7 +3095,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 53 unmodified lines | |||
| 3096 | switch (entry->src_data[0]) { | 3095 | switch (entry->src_data[0]) { |
| 3097 | case 0: /* vblank */ | 3096 | case 0: /* vblank */ |
| 3098 | if (disp_int & interrupt_status_offsets[crtc].vblank) | 3097 | if (disp_int & interrupt_status_offsets[crtc].vblank) |
| 3099 | WREG32(mmVBLANK_STATUS + crtc_offsets[crtc], VBLANK_ACK); | 3098 | WREG32(mmVBLANK_STATUS + crtc_offsets[crtc], VBLANK_STATUS__VBLANK_ACK_MASK); |
| 3100 | else | 3099 | else |
| 3101 | DRM_DEBUG("IH: IH event w/o asserted irq bit?\n"); | 3100 | DRM_DEBUG("IH: IH event w/o asserted irq bit?\n"); |
| 3102 | 3101 | ||
| @@ -3107,7 +3106,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 4 unmodified lines | |||
| 3107 | break; | 3106 | break; |
| 3108 | case 1: /* vline */ | 3107 | case 1: /* vline */ |
| 3109 | if (disp_int & interrupt_status_offsets[crtc].vline) | 3108 | if (disp_int & interrupt_status_offsets[crtc].vline) |
| 3110 | WREG32(mmVLINE_STATUS + crtc_offsets[crtc], VLINE_ACK); | 3109 | WREG32(mmVLINE_STATUS + crtc_offsets[crtc], VLINE_STATUS__VLINE_ACK_MASK); |
| 3111 | else | 3110 | else |
| 3112 | DRM_DEBUG("IH: IH event w/o asserted irq bit?\n"); | 3111 | DRM_DEBUG("IH: IH event w/o asserted irq bit?\n"); |
| 3113 | 3112 | ||
| @@ -3121,7 +3120,7 @@ static int dce_v6_0_crtc_irq(struct amdgpu_device *adev, · 7 unmodified lines | |||
| 3121 | return 0; | 3120 | return 0; |
| 3122 | } | 3121 | } |
| 3123 | 3122 | ||
| 3124 | static int dce_v6_0_set_pageflip_interrupt_state(struct amdgpu_device *adev, | 3123 | static int dce_v6_0_set_pageflip_irq_state(struct amdgpu_device *adev, |
| 3125 | struct amdgpu_irq_src *src, | 3124 | struct amdgpu_irq_src *src, |
| 3126 | unsigned type, | 3125 | unsigned type, |
| 3127 | enum amdgpu_interrupt_state state) | 3126 | enum amdgpu_interrupt_state state) |
| @@ -3172,7 +3171,7 @@ static int dce_v6_0_pageflip_irq(struct amdgpu_device *adev, · 44 unmodified lines | |||
| 3172 | 3171 | ||
| 3173 | spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags); | 3172 | spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags); |
| 3174 | works = amdgpu_crtc->pflip_works; | 3173 | works = amdgpu_crtc->pflip_works; |
| 3175 | if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED){ | 3174 | if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED) { |
| 3176 | DRM_DEBUG_DRIVER("amdgpu_crtc->pflip_status = %d != " | 3175 | DRM_DEBUG_DRIVER("amdgpu_crtc->pflip_status = %d != " |
| 3177 | "AMDGPU_FLIP_SUBMITTED(%d)\n", | 3176 | "AMDGPU_FLIP_SUBMITTED(%d)\n", |
| 3178 | amdgpu_crtc->pflip_status, | 3177 | amdgpu_crtc->pflip_status, |
| @@ -3249,12 +3248,10 @@ static const struct amd_ip_funcs dce_v6_0_ip_funcs = { · 70 unmodified lines | |||
| 3249 | .set_powergating_state = dce_v6_0_set_powergating_state, | 3248 | .set_powergating_state = dce_v6_0_set_powergating_state, |
| 3250 | }; | 3249 | }; |
| 3251 | 3250 | ||
| 3252 | static void | 3251 | static void dce_v6_0_encoder_mode_set(struct drm_encoder *encoder, |
| 3253 | dce_v6_0_encoder_mode_set(struct drm_encoder *encoder, | ||
| 3254 | struct drm_display_mode *mode, | 3252 | struct drm_display_mode *mode, |
| 3255 | struct drm_display_mode *adjusted_mode) | 3253 | struct drm_display_mode *adjusted_mode) |
| 3256 | { | 3254 | { |
| 3257 | |||
| 3258 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); | 3255 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); |
| 3259 | int em = amdgpu_atombios_encoder_get_encoder_mode(encoder); | 3256 | int em = amdgpu_atombios_encoder_get_encoder_mode(encoder); |
| 3260 | 3257 | ||
| @@ -3274,7 +3271,6 @@ dce_v6_0_encoder_mode_set(struct drm_encoder *encoder, · 13 unmodified lines | |||
| 3274 | 3271 | ||
| 3275 | static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder) | 3272 | static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder) |
| 3276 | { | 3273 | { |
| 3277 | |||
| 3278 | struct amdgpu_device *adev = drm_to_adev(encoder->dev); | 3274 | struct amdgpu_device *adev = drm_to_adev(encoder->dev); |
| 3279 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); | 3275 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); |
| 3280 | struct drm_connector *connector = amdgpu_get_connector_for_encoder(encoder); | 3276 | struct drm_connector *connector = amdgpu_get_connector_for_encoder(encoder); |
| @@ -3314,7 +3310,6 @@ static void dce_v6_0_encoder_prepare(struct drm_encoder *encoder) · 33 unmodified lines | |||
| 3314 | 3310 | ||
| 3315 | static void dce_v6_0_encoder_commit(struct drm_encoder *encoder) | 3311 | static void dce_v6_0_encoder_commit(struct drm_encoder *encoder) |
| 3316 | { | 3312 | { |
| 3317 | |||
| 3318 | struct drm_device *dev = encoder->dev; | 3313 | struct drm_device *dev = encoder->dev; |
| 3319 | struct amdgpu_device *adev = drm_to_adev(dev); | 3314 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 3320 | 3315 | ||
| @@ -3325,7 +3320,6 @@ static void dce_v6_0_encoder_commit(struct drm_encoder *encoder) · 4 unmodified lines | |||
| 3325 | 3320 | ||
| 3326 | static void dce_v6_0_encoder_disable(struct drm_encoder *encoder) | 3321 | static void dce_v6_0_encoder_disable(struct drm_encoder *encoder) |
| 3327 | { | 3322 | { |
| 3328 | |||
| 3329 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); | 3323 | struct amdgpu_encoder *amdgpu_encoder = to_amdgpu_encoder(encoder); |
| 3330 | struct amdgpu_encoder_atom_dig *dig; | 3324 | struct amdgpu_encoder_atom_dig *dig; |
| 3331 | int em = amdgpu_atombios_encoder_get_encoder_mode(encoder); | 3325 | int em = amdgpu_atombios_encoder_get_encoder_mode(encoder); |
| @@ -3541,17 +3535,17 @@ static void dce_v6_0_set_display_funcs(struct amdgpu_device *adev) · 209 unmodified lines | |||
| 3541 | } | 3535 | } |
| 3542 | 3536 | ||
| 3543 | static const struct amdgpu_irq_src_funcs dce_v6_0_crtc_irq_funcs = { | 3537 | static const struct amdgpu_irq_src_funcs dce_v6_0_crtc_irq_funcs = { |
| 3544 | .set = dce_v6_0_set_crtc_interrupt_state, | 3538 | .set = dce_v6_0_set_crtc_irq_state, |
| 3545 | .process = dce_v6_0_crtc_irq, | 3539 | .process = dce_v6_0_crtc_irq, |
| 3546 | }; | 3540 | }; |
| 3547 | 3541 | ||
| 3548 | static const struct amdgpu_irq_src_funcs dce_v6_0_pageflip_irq_funcs = { | 3542 | static const struct amdgpu_irq_src_funcs dce_v6_0_pageflip_irq_funcs = { |
| 3549 | .set = dce_v6_0_set_pageflip_interrupt_state, | 3543 | .set = dce_v6_0_set_pageflip_irq_state, |
| 3550 | .process = dce_v6_0_pageflip_irq, | 3544 | .process = dce_v6_0_pageflip_irq, |
| 3551 | }; | 3545 | }; |
| 3552 | 3546 | ||
| 3553 | static const struct amdgpu_irq_src_funcs dce_v6_0_hpd_irq_funcs = { | 3547 | static const struct amdgpu_irq_src_funcs dce_v6_0_hpd_irq_funcs = { |
| 3554 | .set = dce_v6_0_set_hpd_interrupt_state, | 3548 | .set = dce_v6_0_set_hpd_irq_state, |
| 3555 | .process = dce_v6_0_hpd_irq, | 3549 | .process = dce_v6_0_hpd_irq, |
| 3556 | }; | 3550 | }; |
| 3557 | 3551 | ||
| @@ -271,7 +271,7 @@ static void dce_v8_0_hpd_int_ack(struct amdgpu_device *adev, · 270 unmodified lines | |||
| 271 | u32 tmp; | 271 | u32 tmp; |
| 272 | 272 | ||
| 273 | if (hpd >= adev->mode_info.num_hpd) { | 273 | if (hpd >= adev->mode_info.num_hpd) { |
| 274 | DRM_DEBUG("invalid hdp %d\n", hpd); | 274 | DRM_DEBUG("invalid hpd %d\n", hpd); |
| 275 | return; | 275 | return; |
| 276 | } | 276 | } |
| 277 | 277 | ||
| @@ -3021,7 +3021,7 @@ static void dce_v8_0_set_crtc_vline_interrupt_state(struct amdgpu_device *adev, · 2743 unmodified lines | |||
| 3021 | } | 3021 | } |
| 3022 | } | 3022 | } |
| 3023 | 3023 | ||
| 3024 | static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, | 3024 | static int dce_v8_0_set_hpd_irq_state(struct amdgpu_device *adev, |
| 3025 | struct amdgpu_irq_src *src, | 3025 | struct amdgpu_irq_src *src, |
| 3026 | unsigned type, | 3026 | unsigned type, |
| 3027 | enum amdgpu_interrupt_state state) | 3027 | enum amdgpu_interrupt_state state) |
| @@ -3029,7 +3029,7 @@ static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 1 unmodified line | |||
| 3029 | u32 dc_hpd_int_cntl; | 3029 | u32 dc_hpd_int_cntl; |
| 3030 | 3030 | ||
| 3031 | if (type >= adev->mode_info.num_hpd) { | 3031 | if (type >= adev->mode_info.num_hpd) { |
| 3032 | DRM_DEBUG("invalid hdp %d\n", type); | 3032 | DRM_DEBUG("invalid hpd %d\n", type); |
| 3033 | return 0; | 3033 | return 0; |
| 3034 | } | 3034 | } |
| 3035 | 3035 | ||
| @@ -3051,7 +3051,7 @@ static int dce_v8_0_set_hpd_interrupt_state(struct amdgpu_device *adev, · 15 unmodified lines | |||
| 3051 | return 0; | 3051 | return 0; |
| 3052 | } | 3052 | } |
| 3053 | 3053 | ||
| 3054 | static int dce_v8_0_set_crtc_interrupt_state(struct amdgpu_device *adev, | 3054 | static int dce_v8_0_set_crtc_irq_state(struct amdgpu_device *adev, |
| 3055 | struct amdgpu_irq_src *src, | 3055 | struct amdgpu_irq_src *src, |
| 3056 | unsigned type, | 3056 | unsigned type, |
| 3057 | enum amdgpu_interrupt_state state) | 3057 | enum amdgpu_interrupt_state state) |
| @@ -3136,7 +3136,7 @@ static int dce_v8_0_crtc_irq(struct amdgpu_device *adev, · 78 unmodified lines | |||
| 3136 | return 0; | 3136 | return 0; |
| 3137 | } | 3137 | } |
| 3138 | 3138 | ||
| 3139 | static int dce_v8_0_set_pageflip_interrupt_state(struct amdgpu_device *adev, | 3139 | static int dce_v8_0_set_pageflip_irq_state(struct amdgpu_device *adev, |
| 3140 | struct amdgpu_irq_src *src, | 3140 | struct amdgpu_irq_src *src, |
| 3141 | unsigned type, | 3141 | unsigned type, |
| 3142 | enum amdgpu_interrupt_state state) | 3142 | enum amdgpu_interrupt_state state) |
| @@ -3547,17 +3547,17 @@ static void dce_v8_0_set_display_funcs(struct amdgpu_device *adev) · 404 unmodified lines | |||
| 3547 | } | 3547 | } |
| 3548 | 3548 | ||
| 3549 | static const struct amdgpu_irq_src_funcs dce_v8_0_crtc_irq_funcs = { | 3549 | static const struct amdgpu_irq_src_funcs dce_v8_0_crtc_irq_funcs = { |
| 3550 | .set = dce_v8_0_set_crtc_interrupt_state, | 3550 | .set = dce_v8_0_set_crtc_irq_state, |
| 3551 | .process = dce_v8_0_crtc_irq, | 3551 | .process = dce_v8_0_crtc_irq, |
| 3552 | }; | 3552 | }; |
| 3553 | 3553 | ||
| 3554 | static const struct amdgpu_irq_src_funcs dce_v8_0_pageflip_irq_funcs = { | 3554 | static const struct amdgpu_irq_src_funcs dce_v8_0_pageflip_irq_funcs = { |
| 3555 | .set = dce_v8_0_set_pageflip_interrupt_state, | 3555 | .set = dce_v8_0_set_pageflip_irq_state, |
| 3556 | .process = dce_v8_0_pageflip_irq, | 3556 | .process = dce_v8_0_pageflip_irq, |
| 3557 | }; | 3557 | }; |
| 3558 | 3558 | ||
| 3559 | static const struct amdgpu_irq_src_funcs dce_v8_0_hpd_irq_funcs = { | 3559 | static const struct amdgpu_irq_src_funcs dce_v8_0_hpd_irq_funcs = { |
| 3560 | .set = dce_v8_0_set_hpd_interrupt_state, | 3560 | .set = dce_v8_0_set_hpd_irq_state, |
| 3561 | .process = dce_v8_0_hpd_irq, | 3561 | .process = dce_v8_0_hpd_irq, |
| 3562 | }; | 3562 | }; |
| 3563 | 3563 | ||
| @@ -368,11 +368,6 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_10_1[] = { · 367 unmodified lines | |||
| 368 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_DEBUG_INST_ADDR), | 368 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_DEBUG_INST_ADDR), |
| 369 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_LX6_CORE_PDEBUG_INST), | 369 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_LX6_CORE_PDEBUG_INST), |
| 370 | /* cp header registers */ | 370 | /* cp header registers */ |
| 371 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 372 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 373 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME2_HEADER_DUMP), | ||
| 374 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 375 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 376 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MES_HEADER_DUMP), | 371 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MES_HEADER_DUMP), |
| 377 | /* SE status registers */ | 372 | /* SE status registers */ |
| 378 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0), | 373 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0), |
| @@ -421,7 +416,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_10[] = { · 42 unmodified lines | |||
| 421 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_OFFSET), | 416 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_OFFSET), |
| 422 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_DW_CNT), | 417 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_CNTL_STACK_DW_CNT), |
| 423 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_WG_STATE_OFFSET), | 418 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_SUSPEND_WG_STATE_OFFSET), |
| 424 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_DEQUEUE_STATUS) | 419 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_DEQUEUE_STATUS), |
| 420 | /* cp header registers */ | ||
| 421 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 422 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 423 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 424 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 425 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 426 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 427 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 428 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 425 | }; | 429 | }; |
| 426 | 430 | ||
| 427 | static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = { | 431 | static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = { |
| @@ -448,7 +452,32 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_10[] = { · 20 unmodified lines | |||
| 448 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR), | 452 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR), |
| 449 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR_HI), | 453 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_GFX_MQD_BASE_ADDR_HI), |
| 450 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO), | 454 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_LO), |
| 451 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI) | 455 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_RB_WPTR_POLL_ADDR_HI), |
| 456 | /* gfx header registers */ | ||
| 457 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 458 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 459 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 460 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 461 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 462 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 463 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 464 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 465 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 466 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 467 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 468 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 469 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 470 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 471 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 472 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 473 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 474 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 475 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 476 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 477 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 478 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 479 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 480 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 452 | }; | 481 | }; |
| 453 | 482 | ||
| 454 | static const struct soc15_reg_golden golden_settings_gc_10_1[] = { | 483 | static const struct soc15_reg_golden golden_settings_gc_10_1[] = { |
| @@ -4296,9 +4325,7 @@ static u32 gfx_v10_0_get_csb_size(struct amdgpu_device *adev) · 3841 unmodified lines | |||
| 4296 | static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev, | 4325 | static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev, |
| 4297 | volatile u32 *buffer) | 4326 | volatile u32 *buffer) |
| 4298 | { | 4327 | { |
| 4299 | u32 count = 0, i; | 4328 | u32 count = 0; |
| 4300 | const struct cs_section_def *sect = NULL; | ||
| 4301 | const struct cs_extent_def *ext = NULL; | ||
| 4302 | int ctx_reg_offset; | 4329 | int ctx_reg_offset; |
| 4303 | 4330 | ||
| 4304 | if (adev->gfx.rlc.cs_data == NULL) | 4331 | if (adev->gfx.rlc.cs_data == NULL) |
| @@ -4306,39 +4333,15 @@ static void gfx_v10_0_get_csb_buffer(struct amdgpu_device *adev, · 1 unmodified line | |||
| 4306 | if (buffer == NULL) | 4333 | if (buffer == NULL) |
| 4307 | return; | 4334 | return; |
| 4308 | 4335 | ||
| 4309 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 4336 | count = amdgpu_gfx_csb_preamble_start(buffer); |
| 4310 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | 4337 | count = amdgpu_gfx_csb_data_parser(adev, buffer, count); |
| 4311 | 4338 | ||
| 4312 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | 4339 | ctx_reg_offset = SOC15_REG_OFFSET(GC, 0, mmPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START; |
| 4313 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 4314 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 4315 | |||
| 4316 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 4317 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 4318 | if (sect->id == SECT_CONTEXT) { | ||
| 4319 | buffer[count++] = | ||
| 4320 | cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 4321 | buffer[count++] = cpu_to_le32(ext->reg_index - | ||
| 4322 | PACKET3_SET_CONTEXT_REG_START); | ||
| 4323 | for (i = 0; i < ext->reg_count; i++) | ||
| 4324 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 4325 | } else { | ||
| 4326 | return; | ||
| 4327 | } | ||
| 4328 | } | ||
| 4329 | } | ||
| 4330 | |||
| 4331 | ctx_reg_offset = | ||
| 4332 | SOC15_REG_OFFSET(GC, 0, mmPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START; | ||
| 4333 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1)); | 4340 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1)); |
| 4334 | buffer[count++] = cpu_to_le32(ctx_reg_offset); | 4341 | buffer[count++] = cpu_to_le32(ctx_reg_offset); |
| 4335 | buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override); | 4342 | buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override); |
| 4336 | 4343 | ||
| 4337 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 4344 | amdgpu_gfx_csb_preamble_end(buffer, count); |
| 4338 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 4339 | |||
| 4340 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | ||
| 4341 | buffer[count++] = cpu_to_le32(0); | ||
| 4342 | } | 4345 | } |
| 4343 | 4346 | ||
| 4344 | static void gfx_v10_0_rlc_fini(struct amdgpu_device *adev) | 4347 | static void gfx_v10_0_rlc_fini(struct amdgpu_device *adev) |
| @@ -4752,6 +4755,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 407 unmodified lines | |||
| 4752 | int i, j, k, r, ring_id = 0; | 4755 | int i, j, k, r, ring_id = 0; |
| 4753 | int xcc_id = 0; | 4756 | int xcc_id = 0; |
| 4754 | struct amdgpu_device *adev = ip_block->adev; | 4757 | struct amdgpu_device *adev = ip_block->adev; |
| 4758 | int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */ | ||
| 4755 | 4759 | ||
| 4756 | INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler); | 4760 | INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler); |
| 4757 | 4761 | ||
| @@ -4763,7 +4767,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 5 unmodified lines | |||
| 4763 | case IP_VERSION(10, 1, 4): | 4767 | case IP_VERSION(10, 1, 4): |
| 4764 | adev->gfx.me.num_me = 1; | 4768 | adev->gfx.me.num_me = 1; |
| 4765 | adev->gfx.me.num_pipe_per_me = 1; | 4769 | adev->gfx.me.num_pipe_per_me = 1; |
| 4766 | adev->gfx.me.num_queue_per_pipe = 1; | 4770 | adev->gfx.me.num_queue_per_pipe = 8; |
| 4767 | adev->gfx.mec.num_mec = 2; | 4771 | adev->gfx.mec.num_mec = 2; |
| 4768 | adev->gfx.mec.num_pipe_per_mec = 4; | 4772 | adev->gfx.mec.num_pipe_per_mec = 4; |
| 4769 | adev->gfx.mec.num_queue_per_pipe = 8; | 4773 | adev->gfx.mec.num_queue_per_pipe = 8; |
| @@ -4778,7 +4782,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines | |||
| 4778 | case IP_VERSION(10, 3, 7): | 4782 | case IP_VERSION(10, 3, 7): |
| 4779 | adev->gfx.me.num_me = 1; | 4783 | adev->gfx.me.num_me = 1; |
| 4780 | adev->gfx.me.num_pipe_per_me = 2; | 4784 | adev->gfx.me.num_pipe_per_me = 2; |
| 4781 | adev->gfx.me.num_queue_per_pipe = 1; | 4785 | adev->gfx.me.num_queue_per_pipe = 2; |
| 4782 | adev->gfx.mec.num_mec = 2; | 4786 | adev->gfx.mec.num_mec = 2; |
| 4783 | adev->gfx.mec.num_pipe_per_mec = 4; | 4787 | adev->gfx.mec.num_pipe_per_mec = 4; |
| 4784 | adev->gfx.mec.num_queue_per_pipe = 4; | 4788 | adev->gfx.mec.num_queue_per_pipe = 4; |
| @@ -4800,7 +4804,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 15 unmodified lines | |||
| 4800 | adev->gfx.cleaner_shader_size = sizeof(gfx_10_1_10_cleaner_shader_hex); | 4804 | adev->gfx.cleaner_shader_size = sizeof(gfx_10_1_10_cleaner_shader_hex); |
| 4801 | if (adev->gfx.me_fw_version >= 101 && | 4805 | if (adev->gfx.me_fw_version >= 101 && |
| 4802 | adev->gfx.pfp_fw_version >= 158 && | 4806 | adev->gfx.pfp_fw_version >= 158 && |
| 4803 | adev->gfx.mec_fw_version >= 152) { | 4807 | adev->gfx.mec_fw_version >= 151) { |
| 4804 | adev->gfx.enable_cleaner_shader = true; | 4808 | adev->gfx.enable_cleaner_shader = true; |
| 4805 | r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size); | 4809 | r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size); |
| 4806 | if (r) { | 4810 | if (r) { |
| @@ -4810,7 +4814,9 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 3 unmodified lines | |||
| 4810 | } | 4814 | } |
| 4811 | break; | 4815 | break; |
| 4812 | case IP_VERSION(10, 3, 0): | 4816 | case IP_VERSION(10, 3, 0): |
| 4817 | case IP_VERSION(10, 3, 1): | ||
| 4813 | case IP_VERSION(10, 3, 2): | 4818 | case IP_VERSION(10, 3, 2): |
| 4819 | case IP_VERSION(10, 3, 3): | ||
| 4814 | case IP_VERSION(10, 3, 4): | 4820 | case IP_VERSION(10, 3, 4): |
| 4815 | case IP_VERSION(10, 3, 5): | 4821 | case IP_VERSION(10, 3, 5): |
| 4816 | adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex; | 4822 | adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex; |
| @@ -4826,6 +4832,34 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 9 unmodified lines | |||
| 4826 | } | 4832 | } |
| 4827 | } | 4833 | } |
| 4828 | break; | 4834 | break; |
| 4835 | case IP_VERSION(10, 3, 6): | ||
| 4836 | adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex; | ||
| 4837 | adev->gfx.cleaner_shader_size = sizeof(gfx_10_3_0_cleaner_shader_hex); | ||
| 4838 | if (adev->gfx.me_fw_version >= 14 && | ||
| 4839 | adev->gfx.pfp_fw_version >= 17 && | ||
| 4840 | adev->gfx.mec_fw_version >= 24) { | ||
| 4841 | adev->gfx.enable_cleaner_shader = true; | ||
| 4842 | r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size); | ||
| 4843 | if (r) { | ||
| 4844 | adev->gfx.enable_cleaner_shader = false; | ||
| 4845 | dev_err(adev->dev, "Failed to initialize cleaner shader\n"); | ||
| 4846 | } | ||
| 4847 | } | ||
| 4848 | break; | ||
| 4849 | case IP_VERSION(10, 3, 7): | ||
| 4850 | adev->gfx.cleaner_shader_ptr = gfx_10_3_0_cleaner_shader_hex; | ||
| 4851 | adev->gfx.cleaner_shader_size = sizeof(gfx_10_3_0_cleaner_shader_hex); | ||
| 4852 | if (adev->gfx.me_fw_version >= 4 && | ||
| 4853 | adev->gfx.pfp_fw_version >= 9 && | ||
| 4854 | adev->gfx.mec_fw_version >= 12) { | ||
| 4855 | adev->gfx.enable_cleaner_shader = true; | ||
| 4856 | r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size); | ||
| 4857 | if (r) { | ||
| 4858 | adev->gfx.enable_cleaner_shader = false; | ||
| 4859 | dev_err(adev->dev, "Failed to initialize cleaner shader\n"); | ||
| 4860 | } | ||
| 4861 | } | ||
| 4862 | break; | ||
| 4829 | default: | 4863 | default: |
| 4830 | adev->gfx.enable_cleaner_shader = false; | 4864 | adev->gfx.enable_cleaner_shader = false; |
| 4831 | break; | 4865 | break; |
| @@ -4886,7 +4920,7 @@ static int gfx_v10_0_sw_init(struct amdgpu_ip_block *ip_block) · 54 unmodified lines | |||
| 4886 | 4920 | ||
| 4887 | /* set up the gfx ring */ | 4921 | /* set up the gfx ring */ |
| 4888 | for (i = 0; i < adev->gfx.me.num_me; i++) { | 4922 | for (i = 0; i < adev->gfx.me.num_me; i++) { |
| 4889 | for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) { | 4923 | for (j = 0; j < num_queue_per_pipe; j++) { |
| 4890 | for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) { | 4924 | for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) { |
| 4891 | if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j)) | 4925 | if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j)) |
| 4892 | continue; | 4926 | continue; |
| @@ -9645,9 +9679,14 @@ static void gfx_v10_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printe · 4752 unmodified lines | |||
| 9645 | for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) { | 9679 | for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) { |
| 9646 | drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k); | 9680 | drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k); |
| 9647 | for (reg = 0; reg < reg_count; reg++) { | 9681 | for (reg = 0; reg < reg_count; reg++) { |
| 9648 | drm_printf(p, "%-50s \t 0x%08x\n", | 9682 | if (i && gc_cp_reg_list_10[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP) |
| 9649 | gc_cp_reg_list_10[reg].reg_name, | 9683 | drm_printf(p, "%-50s \t 0x%08x\n", |
| 9650 | adev->gfx.ip_dump_compute_queues[index + reg]); | 9684 | "mmCP_MEC_ME2_HEADER_DUMP", |
| 9685 | adev->gfx.ip_dump_compute_queues[index + reg]); | ||
| 9686 | else | ||
| 9687 | drm_printf(p, "%-50s \t 0x%08x\n", | ||
| 9688 | gc_cp_reg_list_10[reg].reg_name, | ||
| 9689 | adev->gfx.ip_dump_compute_queues[index + reg]); | ||
| 9651 | } | 9690 | } |
| 9652 | index += reg_count; | 9691 | index += reg_count; |
| 9653 | } | 9692 | } |
| @@ -9708,9 +9747,13 @@ static void gfx_v10_ip_dump(struct amdgpu_ip_block *ip_block) · 54 unmodified lines | |||
| 9708 | nv_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0); | 9747 | nv_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0); |
| 9709 | 9748 | ||
| 9710 | for (reg = 0; reg < reg_count; reg++) { | 9749 | for (reg = 0; reg < reg_count; reg++) { |
| 9711 | adev->gfx.ip_dump_compute_queues[index + reg] = | 9750 | if (i && gc_cp_reg_list_10[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP) |
| 9712 | RREG32(SOC15_REG_ENTRY_OFFSET( | 9751 | adev->gfx.ip_dump_compute_queues[index + reg] = |
| 9713 | gc_cp_reg_list_10[reg])); | 9752 | RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_MEC_ME2_HEADER_DUMP)); |
| 9753 | else | ||
| 9754 | adev->gfx.ip_dump_compute_queues[index + reg] = | ||
| 9755 | RREG32(SOC15_REG_ENTRY_OFFSET( | ||
| 9756 | gc_cp_reg_list_10[reg])); | ||
| 9714 | } | 9757 | } |
| 9715 | index += reg_count; | 9758 | index += reg_count; |
| 9716 | } | 9759 | } |
| @@ -48,6 +48,8 @@ · 47 unmodified lines | |||
| 48 | #include "gfx_v11_0_3.h" | 48 | #include "gfx_v11_0_3.h" |
| 49 | #include "nbio_v4_3.h" | 49 | #include "nbio_v4_3.h" |
| 50 | #include "mes_v11_0.h" | 50 | #include "mes_v11_0.h" |
| 51 | #include "mes_userqueue.h" | ||
| 52 | #include "amdgpu_userq_fence.h" | ||
| 51 | 53 | ||
| 52 | #define GFX11_NUM_GFX_RINGS 1 | 54 | #define GFX11_NUM_GFX_RINGS 1 |
| 53 | #define GFX11_MEC_HPD_SIZE 2048 | 55 | #define GFX11_MEC_HPD_SIZE 2048 |
| @@ -177,9 +179,13 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_11_0[] = { · 123 unmodified lines | |||
| 177 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_INSTR_PNTR), | 179 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_INSTR_PNTR), |
| 178 | SOC15_REG_ENTRY_STR(GC, 0, regCP_CPC_STATUS), | 180 | SOC15_REG_ENTRY_STR(GC, 0, regCP_CPC_STATUS), |
| 179 | /* cp header registers */ | 181 | /* cp header registers */ |
| 180 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | 182 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 181 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | 183 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 182 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | 184 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 185 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 186 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 187 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 188 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 183 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | 189 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 184 | /* SE status registers */ | 190 | /* SE status registers */ |
| 185 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0), | 191 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0), |
| @@ -230,7 +236,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_11[] = { · 44 unmodified lines | |||
| 230 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET), | 236 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET), |
| 231 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT), | 237 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT), |
| 232 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET), | 238 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET), |
| 233 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS) | 239 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS), |
| 240 | /* cp header registers */ | ||
| 241 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 242 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 243 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 244 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 245 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 246 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 247 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 248 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 234 | }; | 249 | }; |
| 235 | 250 | ||
| 236 | static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = { | 251 | static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = { |
| @@ -259,7 +274,24 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_11[] = { · 22 unmodified lines | |||
| 259 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO), | 274 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO), |
| 260 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI), | 275 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI), |
| 261 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ), | 276 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ), |
| 262 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ) | 277 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ), |
| 278 | /* cp header registers */ | ||
| 279 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 280 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 281 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 282 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 283 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 284 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 285 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 286 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 287 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 288 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 289 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 290 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 291 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 292 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 293 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 294 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 263 | }; | 295 | }; |
| 264 | 296 | ||
| 265 | static const struct soc15_reg_golden golden_settings_gc_11_0[] = { | 297 | static const struct soc15_reg_golden golden_settings_gc_11_0[] = { |
| @@ -580,33 +612,18 @@ static int gfx_v11_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 314 unmodified lines | |||
| 580 | 612 | ||
| 581 | memset(&ib, 0, sizeof(ib)); | 613 | memset(&ib, 0, sizeof(ib)); |
| 582 | 614 | ||
| 583 | if (ring->is_mes_queue) { | 615 | r = amdgpu_device_wb_get(adev, &index); |
| 584 | uint32_t padding, offset; | 616 | if (r) |
| 585 | 617 | return r; | |
| 586 | offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS); | ||
| 587 | padding = amdgpu_mes_ctx_get_offs(ring, | ||
| 588 | AMDGPU_MES_CTX_PADDING_OFFS); | ||
| 589 | |||
| 590 | ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 591 | ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 592 | |||
| 593 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding); | ||
| 594 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding); | ||
| 595 | *cpu_ptr = cpu_to_le32(0xCAFEDEAD); | ||
| 596 | } else { | ||
| 597 | r = amdgpu_device_wb_get(adev, &index); | ||
| 598 | if (r) | ||
| 599 | return r; | ||
| 600 | 618 | ||
| 601 | gpu_addr = adev->wb.gpu_addr + (index * 4); | 619 | gpu_addr = adev->wb.gpu_addr + (index * 4); |
| 602 | adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD); | 620 | adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD); |
| 603 | cpu_ptr = &adev->wb.wb[index]; | 621 | cpu_ptr = &adev->wb.wb[index]; |
| 604 | 622 | ||
| 605 | r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib); | 623 | r = amdgpu_ib_get(adev, NULL, 20, AMDGPU_IB_POOL_DIRECT, &ib); |
| 606 | if (r) { | 624 | if (r) { |
| 607 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); | 625 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); |
| 608 | goto err1; | 626 | goto err1; |
| 609 | } | ||
| 610 | } | 627 | } |
| 611 | 628 | ||
| 612 | ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3); | 629 | ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3); |
| @@ -633,12 +650,10 @@ static int gfx_v11_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 20 unmodified lines | |||
| 633 | else | 650 | else |
| 634 | r = -EINVAL; | 651 | r = -EINVAL; |
| 635 | err2: | 652 | err2: |
| 636 | if (!ring->is_mes_queue) | 653 | amdgpu_ib_free(&ib, NULL); |
| 637 | amdgpu_ib_free(&ib, NULL); | ||
| 638 | dma_fence_put(f); | 654 | dma_fence_put(f); |
| 639 | err1: | 655 | err1: |
| 640 | if (!ring->is_mes_queue) | 656 | amdgpu_device_wb_free(adev, index); |
| 641 | amdgpu_device_wb_free(adev, index); | ||
| 642 | return r; | 657 | return r; |
| 643 | } | 658 | } |
| 644 | 659 | ||
| @@ -833,9 +848,7 @@ static u32 gfx_v11_0_get_csb_size(struct amdgpu_device *adev) · 188 unmodified lines | |||
| 833 | static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev, | 848 | static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev, |
| 834 | volatile u32 *buffer) | 849 | volatile u32 *buffer) |
| 835 | { | 850 | { |
| 836 | u32 count = 0, i; | 851 | u32 count = 0; |
| 837 | const struct cs_section_def *sect = NULL; | ||
| 838 | const struct cs_extent_def *ext = NULL; | ||
| 839 | int ctx_reg_offset; | 852 | int ctx_reg_offset; |
| 840 | 853 | ||
| 841 | if (adev->gfx.rlc.cs_data == NULL) | 854 | if (adev->gfx.rlc.cs_data == NULL) |
| @@ -843,39 +856,15 @@ static void gfx_v11_0_get_csb_buffer(struct amdgpu_device *adev, · 1 unmodified line | |||
| 843 | if (buffer == NULL) | 856 | if (buffer == NULL) |
| 844 | return; | 857 | return; |
| 845 | 858 | ||
| 846 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 859 | count = amdgpu_gfx_csb_preamble_start(buffer); |
| 847 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | 860 | count = amdgpu_gfx_csb_data_parser(adev, buffer, count); |
| 848 | |||
| 849 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | ||
| 850 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 851 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 852 | |||
| 853 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 854 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 855 | if (sect->id == SECT_CONTEXT) { | ||
| 856 | buffer[count++] = | ||
| 857 | cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 858 | buffer[count++] = cpu_to_le32(ext->reg_index - | ||
| 859 | PACKET3_SET_CONTEXT_REG_START); | ||
| 860 | for (i = 0; i < ext->reg_count; i++) | ||
| 861 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 862 | } else { | ||
| 863 | return; | ||
| 864 | } | ||
| 865 | } | ||
| 866 | } | ||
| 867 | 861 | ||
| 868 | ctx_reg_offset = | 862 | ctx_reg_offset = SOC15_REG_OFFSET(GC, 0, regPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START; |
| 869 | SOC15_REG_OFFSET(GC, 0, regPA_SC_TILE_STEERING_OVERRIDE) - PACKET3_SET_CONTEXT_REG_START; | ||
| 870 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1)); | 863 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1)); |
| 871 | buffer[count++] = cpu_to_le32(ctx_reg_offset); | 864 | buffer[count++] = cpu_to_le32(ctx_reg_offset); |
| 872 | buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override); | 865 | buffer[count++] = cpu_to_le32(adev->gfx.config.pa_sc_tile_steering_override); |
| 873 | 866 | ||
| 874 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 867 | amdgpu_gfx_csb_preamble_end(buffer, count); |
| 875 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 876 | |||
| 877 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | ||
| 878 | buffer[count++] = cpu_to_le32(0); | ||
| 879 | } | 868 | } |
| 880 | 869 | ||
| 881 | static void gfx_v11_0_rlc_fini(struct amdgpu_device *adev) | 870 | static void gfx_v11_0_rlc_fini(struct amdgpu_device *adev) |
| @@ -1056,14 +1045,21 @@ static void gfx_v11_0_select_me_pipe_q(struct amdgpu_device *adev, · 174 unmodified lines | |||
| 1056 | #define MQD_FWWORKAREA_SIZE 484 | 1045 | #define MQD_FWWORKAREA_SIZE 484 |
| 1057 | #define MQD_FWWORKAREA_ALIGNMENT 256 | 1046 | #define MQD_FWWORKAREA_ALIGNMENT 256 |
| 1058 | 1047 | ||
| 1059 | static int gfx_v11_0_get_gfx_shadow_info(struct amdgpu_device *adev, | 1048 | static void gfx_v11_0_get_gfx_shadow_info_nocheck(struct amdgpu_device *adev, |
| 1060 | struct amdgpu_gfx_shadow_info *shadow_info) | 1049 | struct amdgpu_gfx_shadow_info *shadow_info) |
| 1061 | { | 1050 | { |
| 1062 | if (adev->gfx.cp_gfx_shadow) { | 1051 | shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE; |
| 1063 | shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE; | 1052 | shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT; |
| 1064 | shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT; | 1053 | shadow_info->csa_size = MQD_FWWORKAREA_SIZE; |
| 1065 | shadow_info->csa_size = MQD_FWWORKAREA_SIZE; | 1054 | shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT; |
| 1066 | shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT; | 1055 | } |
| 1056 | |||
| 1057 | static int gfx_v11_0_get_gfx_shadow_info(struct amdgpu_device *adev, | ||
| 1058 | struct amdgpu_gfx_shadow_info *shadow_info, | ||
| 1059 | bool skip_check) | ||
| 1060 | { | ||
| 1061 | if (adev->gfx.cp_gfx_shadow || skip_check) { | ||
| 1062 | gfx_v11_0_get_gfx_shadow_info_nocheck(adev, shadow_info); | ||
| 1067 | return 0; | 1063 | return 0; |
| 1068 | } else { | 1064 | } else { |
| 1069 | memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info)); | 1065 | memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info)); |
| @@ -1136,6 +1132,10 @@ static int gfx_v11_0_gfx_ring_init(struct amdgpu_device *adev, int ring_id, · 66 unmodified lines | |||
| 1136 | 1132 | ||
| 1137 | ring->ring_obj = NULL; | 1133 | ring->ring_obj = NULL; |
| 1138 | ring->use_doorbell = true; | 1134 | ring->use_doorbell = true; |
| 1135 | if (adev->gfx.disable_kq) { | ||
| 1136 | ring->no_scheduler = true; | ||
| 1137 | ring->no_user_submission = true; | ||
| 1138 | } | ||
| 1139 | 1139 | ||
| 1140 | if (!ring_id) | 1140 | if (!ring_id) |
| 1141 | ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1; | 1141 | ring->doorbell_index = adev->doorbell_index.gfx_ring0 << 1; |
| @@ -1568,24 +1568,18 @@ static void gfx_v11_0_alloc_ip_dump(struct amdgpu_device *adev) · 426 unmodified lines | |||
| 1568 | 1568 | ||
| 1569 | static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) | 1569 | static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) |
| 1570 | { | 1570 | { |
| 1571 | int i, j, k, r, ring_id = 0; | 1571 | int i, j, k, r, ring_id; |
| 1572 | int xcc_id = 0; | 1572 | int xcc_id = 0; |
| 1573 | struct amdgpu_device *adev = ip_block->adev; | 1573 | struct amdgpu_device *adev = ip_block->adev; |
| 1574 | int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */ | ||
| 1574 | 1575 | ||
| 1575 | INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler); | 1576 | INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler); |
| 1576 | 1577 | ||
| 1577 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | 1578 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { |
| 1578 | case IP_VERSION(11, 0, 0): | 1579 | case IP_VERSION(11, 0, 0): |
| 1580 | case IP_VERSION(11, 0, 1): | ||
| 1579 | case IP_VERSION(11, 0, 2): | 1581 | case IP_VERSION(11, 0, 2): |
| 1580 | case IP_VERSION(11, 0, 3): | 1582 | case IP_VERSION(11, 0, 3): |
| 1581 | adev->gfx.me.num_me = 1; | ||
| 1582 | adev->gfx.me.num_pipe_per_me = 1; | ||
| 1583 | adev->gfx.me.num_queue_per_pipe = 1; | ||
| 1584 | adev->gfx.mec.num_mec = 1; | ||
| 1585 | adev->gfx.mec.num_pipe_per_mec = 4; | ||
| 1586 | adev->gfx.mec.num_queue_per_pipe = 4; | ||
| 1587 | break; | ||
| 1588 | case IP_VERSION(11, 0, 1): | ||
| 1589 | case IP_VERSION(11, 0, 4): | 1583 | case IP_VERSION(11, 0, 4): |
| 1590 | case IP_VERSION(11, 5, 0): | 1584 | case IP_VERSION(11, 5, 0): |
| 1591 | case IP_VERSION(11, 5, 1): | 1585 | case IP_VERSION(11, 5, 1): |
| @@ -1593,7 +1587,7 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 1 unmodified line | |||
| 1593 | case IP_VERSION(11, 5, 3): | 1587 | case IP_VERSION(11, 5, 3): |
| 1594 | adev->gfx.me.num_me = 1; | 1588 | adev->gfx.me.num_me = 1; |
| 1595 | adev->gfx.me.num_pipe_per_me = 1; | 1589 | adev->gfx.me.num_pipe_per_me = 1; |
| 1596 | adev->gfx.me.num_queue_per_pipe = 1; | 1590 | adev->gfx.me.num_queue_per_pipe = 2; |
| 1597 | adev->gfx.mec.num_mec = 1; | 1591 | adev->gfx.mec.num_mec = 1; |
| 1598 | adev->gfx.mec.num_pipe_per_mec = 4; | 1592 | adev->gfx.mec.num_pipe_per_mec = 4; |
| 1599 | adev->gfx.mec.num_queue_per_pipe = 4; | 1593 | adev->gfx.mec.num_queue_per_pipe = 4; |
| @@ -1608,6 +1602,35 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines | |||
| 1608 | break; | 1602 | break; |
| 1609 | } | 1603 | } |
| 1610 | 1604 | ||
| 1605 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | ||
| 1606 | case IP_VERSION(11, 0, 0): | ||
| 1607 | case IP_VERSION(11, 0, 2): | ||
| 1608 | case IP_VERSION(11, 0, 3): | ||
| 1609 | if (!adev->gfx.disable_uq && | ||
| 1610 | adev->gfx.me_fw_version >= 2390 && | ||
| 1611 | adev->gfx.pfp_fw_version >= 2530 && | ||
| 1612 | adev->gfx.mec_fw_version >= 2600 && | ||
| 1613 | adev->mes.fw_version[0] >= 120) { | ||
| 1614 | adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs; | ||
| 1615 | adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs; | ||
| 1616 | } | ||
| 1617 | break; | ||
| 1618 | case IP_VERSION(11, 0, 1): | ||
| 1619 | case IP_VERSION(11, 0, 4): | ||
| 1620 | case IP_VERSION(11, 5, 0): | ||
| 1621 | case IP_VERSION(11, 5, 1): | ||
| 1622 | case IP_VERSION(11, 5, 2): | ||
| 1623 | case IP_VERSION(11, 5, 3): | ||
| 1624 | /* add firmware version checks here */ | ||
| 1625 | if (0 && !adev->gfx.disable_uq) { | ||
| 1626 | adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs; | ||
| 1627 | adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs; | ||
| 1628 | } | ||
| 1629 | break; | ||
| 1630 | default: | ||
| 1631 | break; | ||
| 1632 | } | ||
| 1633 | |||
| 1611 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | 1634 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { |
| 1612 | case IP_VERSION(11, 0, 0): | 1635 | case IP_VERSION(11, 0, 0): |
| 1613 | case IP_VERSION(11, 0, 2): | 1636 | case IP_VERSION(11, 0, 2): |
| @@ -1640,6 +1663,34 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 26 unmodified lines | |||
| 1640 | } | 1663 | } |
| 1641 | } | 1664 | } |
| 1642 | break; | 1665 | break; |
| 1666 | case IP_VERSION(11, 5, 2): | ||
| 1667 | adev->gfx.cleaner_shader_ptr = gfx_11_0_3_cleaner_shader_hex; | ||
| 1668 | adev->gfx.cleaner_shader_size = sizeof(gfx_11_0_3_cleaner_shader_hex); | ||
| 1669 | if (adev->gfx.me_fw_version >= 12 && | ||
| 1670 | adev->gfx.pfp_fw_version >= 15 && | ||
| 1671 | adev->gfx.mec_fw_version >= 15) { | ||
| 1672 | adev->gfx.enable_cleaner_shader = true; | ||
| 1673 | r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size); | ||
| 1674 | if (r) { | ||
| 1675 | adev->gfx.enable_cleaner_shader = false; | ||
| 1676 | dev_err(adev->dev, "Failed to initialize cleaner shader\n"); | ||
| 1677 | } | ||
| 1678 | } | ||
| 1679 | break; | ||
| 1680 | case IP_VERSION(11, 5, 3): | ||
| 1681 | adev->gfx.cleaner_shader_ptr = gfx_11_0_3_cleaner_shader_hex; | ||
| 1682 | adev->gfx.cleaner_shader_size = sizeof(gfx_11_0_3_cleaner_shader_hex); | ||
| 1683 | if (adev->gfx.me_fw_version >= 7 && | ||
| 1684 | adev->gfx.pfp_fw_version >= 8 && | ||
| 1685 | adev->gfx.mec_fw_version >= 8) { | ||
| 1686 | adev->gfx.enable_cleaner_shader = true; | ||
| 1687 | r = amdgpu_gfx_cleaner_shader_sw_init(adev, adev->gfx.cleaner_shader_size); | ||
| 1688 | if (r) { | ||
| 1689 | adev->gfx.enable_cleaner_shader = false; | ||
| 1690 | dev_err(adev->dev, "Failed to initialize cleaner shader\n"); | ||
| 1691 | } | ||
| 1692 | } | ||
| 1693 | break; | ||
| 1643 | default: | 1694 | default: |
| 1644 | adev->gfx.enable_cleaner_shader = false; | 1695 | adev->gfx.enable_cleaner_shader = false; |
| 1645 | break; | 1696 | break; |
| @@ -1701,37 +1752,42 @@ static int gfx_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 55 unmodified lines | |||
| 1701 | return r; | 1752 | return r; |
| 1702 | } | 1753 | } |
| 1703 | 1754 | ||
| 1704 | /* set up the gfx ring */ | 1755 | if (adev->gfx.num_gfx_rings) { |
| 1705 | for (i = 0; i < adev->gfx.me.num_me; i++) { | 1756 | ring_id = 0; |
| 1706 | for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) { | 1757 | /* set up the gfx ring */ |
| 1707 | for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) { | 1758 | for (i = 0; i < adev->gfx.me.num_me; i++) { |
| 1708 | if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j)) | 1759 | for (j = 0; j < num_queue_per_pipe; j++) { |
| 1709 | continue; | 1760 | for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) { |
| 1710 | 1761 | if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j)) | |
| 1711 | r = gfx_v11_0_gfx_ring_init(adev, ring_id, | 1762 | continue; |
| 1712 | i, k, j); | 1763 | |
| 1713 | if (r) | 1764 | r = gfx_v11_0_gfx_ring_init(adev, ring_id, |
| 1714 | return r; | 1765 | i, k, j); |
| 1715 | ring_id++; | 1766 | if (r) |
| 1767 | return r; | ||
| 1768 | ring_id++; | ||
| 1769 | } | ||
| 1716 | } | 1770 | } |
| 1717 | } | 1771 | } |
| 1718 | } | 1772 | } |
| 1719 | 1773 | ||
| 1720 | ring_id = 0; | 1774 | if (adev->gfx.num_compute_rings) { |
| 1721 | /* set up the compute queues - allocate horizontally across pipes */ | 1775 | ring_id = 0; |
| 1722 | for (i = 0; i < adev->gfx.mec.num_mec; ++i) { | 1776 | /* set up the compute queues - allocate horizontally across pipes */ |
| 1723 | for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) { | 1777 | for (i = 0; i < adev->gfx.mec.num_mec; ++i) { |
| 1724 | for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) { | 1778 | for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) { |
| 1725 | if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i, | 1779 | for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) { |
| 1726 | k, j)) | 1780 | if (!amdgpu_gfx_is_mec_queue_enabled(adev, 0, i, |
| 1727 | continue; | 1781 | k, j)) |
| 1782 | continue; | ||
| 1728 | 1783 | ||
| 1729 | r = gfx_v11_0_compute_ring_init(adev, ring_id, | 1784 | r = gfx_v11_0_compute_ring_init(adev, ring_id, |
| 1730 | i, k, j); | 1785 | i, k, j); |
| 1731 | if (r) | 1786 | if (r) |
| 1732 | return r; | 1787 | return r; |
| 1733 | 1788 | ||
| 1734 | ring_id++; | 1789 | ring_id++; |
| 1790 | } | ||
| 1735 | } | 1791 | } |
| 1736 | } | 1792 | } |
| 1737 | } | 1793 | } |
| @@ -4061,6 +4117,8 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 2323 unmodified lines | |||
| 4061 | #ifdef __BIG_ENDIAN | 4117 | #ifdef __BIG_ENDIAN |
| 4062 | tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1); | 4118 | tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1); |
| 4063 | #endif | 4119 | #endif |
| 4120 | if (prop->tmz_queue) | ||
| 4121 | tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, TMZ_MATCH, 1); | ||
| 4064 | mqd->cp_gfx_hqd_cntl = tmp; | 4122 | mqd->cp_gfx_hqd_cntl = tmp; |
| 4065 | 4123 | ||
| 4066 | /* set up cp_doorbell_control */ | 4124 | /* set up cp_doorbell_control */ |
| @@ -4081,6 +4139,16 @@ static int gfx_v11_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 14 unmodified lines | |||
| 4081 | /* active the queue */ | 4139 | /* active the queue */ |
| 4082 | mqd->cp_gfx_hqd_active = 1; | 4140 | mqd->cp_gfx_hqd_active = 1; |
| 4083 | 4141 | ||
| 4142 | /* set gfx UQ items */ | ||
| 4143 | mqd->shadow_base_lo = lower_32_bits(prop->shadow_addr); | ||
| 4144 | mqd->shadow_base_hi = upper_32_bits(prop->shadow_addr); | ||
| 4145 | mqd->gds_bkup_base_lo = lower_32_bits(prop->gds_bkup_addr); | ||
| 4146 | mqd->gds_bkup_base_hi = upper_32_bits(prop->gds_bkup_addr); | ||
| 4147 | mqd->fw_work_area_base_lo = lower_32_bits(prop->csa_addr); | ||
| 4148 | mqd->fw_work_area_base_hi = upper_32_bits(prop->csa_addr); | ||
| 4149 | mqd->fence_address_lo = lower_32_bits(prop->fence_address); | ||
| 4150 | mqd->fence_address_hi = upper_32_bits(prop->fence_address); | ||
| 4151 | |||
| 4084 | return 0; | 4152 | return 0; |
| 4085 | } | 4153 | } |
| 4086 | 4154 | ||
| @@ -4205,6 +4273,8 @@ static int gfx_v11_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 118 unmodified lines | |||
| 4205 | prop->allow_tunneling); | 4273 | prop->allow_tunneling); |
| 4206 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1); | 4274 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1); |
| 4207 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1); | 4275 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1); |
| 4276 | if (prop->tmz_queue) | ||
| 4277 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TMZ, 1); | ||
| 4208 | mqd->cp_hqd_pq_control = tmp; | 4278 | mqd->cp_hqd_pq_control = tmp; |
| 4209 | 4279 | ||
| 4210 | /* set the wb address whether it's enabled or not */ | 4280 | /* set the wb address whether it's enabled or not */ |
| @@ -4256,6 +4326,10 @@ static int gfx_v11_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 45 unmodified lines | |||
| 4256 | 4326 | ||
| 4257 | mqd->cp_hqd_active = prop->hqd_active; | 4327 | mqd->cp_hqd_active = prop->hqd_active; |
| 4258 | 4328 | ||
| 4329 | /* set UQ fenceaddress */ | ||
| 4330 | mqd->fence_address_lo = lower_32_bits(prop->fence_address); | ||
| 4331 | mqd->fence_address_hi = upper_32_bits(prop->fence_address); | ||
| 4332 | |||
| 4259 | return 0; | 4333 | return 0; |
| 4260 | } | 4334 | } |
| 4261 | 4335 | ||
| @@ -4509,11 +4583,23 @@ static int gfx_v11_0_cp_resume(struct amdgpu_device *adev) · 247 unmodified lines | |||
| 4509 | return r; | 4583 | return r; |
| 4510 | } | 4584 | } |
| 4511 | 4585 | ||
| 4512 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { | 4586 | if (adev->gfx.disable_kq) { |
| 4513 | ring = &adev->gfx.gfx_ring[i]; | 4587 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { |
| 4514 | r = amdgpu_ring_test_helper(ring); | 4588 | ring = &adev->gfx.gfx_ring[i]; |
| 4515 | if (r) | 4589 | /* we don't want to set ring->ready */ |
| 4516 | return r; | 4590 | r = amdgpu_ring_test_ring(ring); |
| 4591 | if (r) | ||
| 4592 | return r; | ||
| 4593 | } | ||
| 4594 | if (amdgpu_async_gfx_ring) | ||
| 4595 | amdgpu_gfx_disable_kgq(adev, 0); | ||
| 4596 | } else { | ||
| 4597 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { | ||
| 4598 | ring = &adev->gfx.gfx_ring[i]; | ||
| 4599 | r = amdgpu_ring_test_helper(ring); | ||
| 4600 | if (r) | ||
| 4601 | return r; | ||
| 4602 | } | ||
| 4517 | } | 4603 | } |
| 4518 | 4604 | ||
| 4519 | for (i = 0; i < adev->gfx.num_compute_rings; i++) { | 4605 | for (i = 0; i < adev->gfx.num_compute_rings; i++) { |
| @@ -4722,6 +4808,49 @@ static int gfx_v11_0_hw_init(struct amdgpu_ip_block *ip_block) · 202 unmodified lines | |||
| 4722 | return r; | 4808 | return r; |
| 4723 | } | 4809 | } |
| 4724 | 4810 | ||
| 4811 | static int gfx_v11_0_set_userq_eop_interrupts(struct amdgpu_device *adev, | ||
| 4812 | bool enable) | ||
| 4813 | { | ||
| 4814 | unsigned int irq_type; | ||
| 4815 | int m, p, r; | ||
| 4816 | |||
| 4817 | if (adev->userq_funcs[AMDGPU_HW_IP_GFX]) { | ||
| 4818 | for (m = 0; m < adev->gfx.me.num_me; m++) { | ||
| 4819 | for (p = 0; p < adev->gfx.me.num_pipe_per_me; p++) { | ||
| 4820 | irq_type = AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP + p; | ||
| 4821 | if (enable) | ||
| 4822 | r = amdgpu_irq_get(adev, &adev->gfx.eop_irq, | ||
| 4823 | irq_type); | ||
| 4824 | else | ||
| 4825 | r = amdgpu_irq_put(adev, &adev->gfx.eop_irq, | ||
| 4826 | irq_type); | ||
| 4827 | if (r) | ||
| 4828 | return r; | ||
| 4829 | } | ||
| 4830 | } | ||
| 4831 | } | ||
| 4832 | |||
| 4833 | if (adev->userq_funcs[AMDGPU_HW_IP_COMPUTE]) { | ||
| 4834 | for (m = 0; m < adev->gfx.mec.num_mec; ++m) { | ||
| 4835 | for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) { | ||
| 4836 | irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP | ||
| 4837 | + (m * adev->gfx.mec.num_pipe_per_mec) | ||
| 4838 | + p; | ||
| 4839 | if (enable) | ||
| 4840 | r = amdgpu_irq_get(adev, &adev->gfx.eop_irq, | ||
| 4841 | irq_type); | ||
| 4842 | else | ||
| 4843 | r = amdgpu_irq_put(adev, &adev->gfx.eop_irq, | ||
| 4844 | irq_type); | ||
| 4845 | if (r) | ||
| 4846 | return r; | ||
| 4847 | } | ||
| 4848 | } | ||
| 4849 | } | ||
| 4850 | |||
| 4851 | return 0; | ||
| 4852 | } | ||
| 4853 | |||
| 4725 | static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block) | 4854 | static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block) |
| 4726 | { | 4855 | { |
| 4727 | struct amdgpu_device *adev = ip_block->adev; | 4856 | struct amdgpu_device *adev = ip_block->adev; |
| @@ -4731,9 +4860,11 @@ static int gfx_v11_0_hw_fini(struct amdgpu_ip_block *ip_block) · 3 unmodified lines | |||
| 4731 | amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0); | 4860 | amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0); |
| 4732 | amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0); | 4861 | amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0); |
| 4733 | amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0); | 4862 | amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0); |
| 4863 | gfx_v11_0_set_userq_eop_interrupts(adev, false); | ||
| 4734 | 4864 | ||
| 4735 | if (!adev->no_hw_access) { | 4865 | if (!adev->no_hw_access) { |
| 4736 | if (amdgpu_async_gfx_ring) { | 4866 | if (amdgpu_async_gfx_ring && |
| 4867 | !adev->gfx.disable_kq) { | ||
| 4737 | if (amdgpu_gfx_disable_kgq(adev, 0)) | 4868 | if (amdgpu_gfx_disable_kgq(adev, 0)) |
| 4738 | DRM_ERROR("KGQ disable failed\n"); | 4869 | DRM_ERROR("KGQ disable failed\n"); |
| 4739 | } | 4870 | } |
| @@ -5059,11 +5190,36 @@ static int gfx_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 319 unmodified lines | |||
| 5059 | { | 5190 | { |
| 5060 | struct amdgpu_device *adev = ip_block->adev; | 5191 | struct amdgpu_device *adev = ip_block->adev; |
| 5061 | 5192 | ||
| 5193 | switch (amdgpu_user_queue) { | ||
| 5194 | case -1: | ||
| 5195 | case 0: | ||
| 5196 | default: | ||
| 5197 | adev->gfx.disable_kq = false; | ||
| 5198 | adev->gfx.disable_uq = true; | ||
| 5199 | break; | ||
| 5200 | case 1: | ||
| 5201 | adev->gfx.disable_kq = false; | ||
| 5202 | adev->gfx.disable_uq = false; | ||
| 5203 | break; | ||
| 5204 | case 2: | ||
| 5205 | adev->gfx.disable_kq = true; | ||
| 5206 | adev->gfx.disable_uq = false; | ||
| 5207 | break; | ||
| 5208 | } | ||
| 5209 | |||
| 5062 | adev->gfx.funcs = &gfx_v11_0_gfx_funcs; | 5210 | adev->gfx.funcs = &gfx_v11_0_gfx_funcs; |
| 5063 | 5211 | ||
| 5064 | adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS; | 5212 | if (adev->gfx.disable_kq) { |
| 5065 | adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev), | 5213 | /* We need one GFX ring temporarily to set up |
| 5066 | AMDGPU_MAX_COMPUTE_RINGS); | 5214 | * the clear state. |
| 5215 | */ | ||
| 5216 | adev->gfx.num_gfx_rings = 1; | ||
| 5217 | adev->gfx.num_compute_rings = 0; | ||
| 5218 | } else { | ||
| 5219 | adev->gfx.num_gfx_rings = GFX11_NUM_GFX_RINGS; | ||
| 5220 | adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev), | ||
| 5221 | AMDGPU_MAX_COMPUTE_RINGS); | ||
| 5222 | } | ||
| 5067 | 5223 | ||
| 5068 | gfx_v11_0_set_kiq_pm4_funcs(adev); | 5224 | gfx_v11_0_set_kiq_pm4_funcs(adev); |
| 5069 | gfx_v11_0_set_ring_funcs(adev); | 5225 | gfx_v11_0_set_ring_funcs(adev); |
| @@ -5094,6 +5250,11 @@ static int gfx_v11_0_late_init(struct amdgpu_ip_block *ip_block) · 24 unmodified lines | |||
| 5094 | r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0); | 5250 | r = amdgpu_irq_get(adev, &adev->gfx.bad_op_irq, 0); |
| 5095 | if (r) | 5251 | if (r) |
| 5096 | return r; | 5252 | return r; |
| 5253 | |||
| 5254 | r = gfx_v11_0_set_userq_eop_interrupts(adev, true); | ||
| 5255 | if (r) | ||
| 5256 | return r; | ||
| 5257 | |||
| 5097 | return 0; | 5258 | return 0; |
| 5098 | } | 5259 | } |
| 5099 | 5260 | ||
| @@ -5691,10 +5852,6 @@ static void gfx_v11_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, · 591 unmodified lines | |||
| 5691 | (!amdgpu_sriov_vf(ring->adev) && flags & AMDGPU_IB_PREEMPTED) ? true : false); | 5852 | (!amdgpu_sriov_vf(ring->adev) && flags & AMDGPU_IB_PREEMPTED) ? true : false); |
| 5692 | } | 5853 | } |
| 5693 | 5854 | ||
| 5694 | if (ring->is_mes_queue) | ||
| 5695 | /* inherit vmid from mqd */ | ||
| 5696 | control |= 0x400000; | ||
| 5697 | |||
| 5698 | amdgpu_ring_write(ring, header); | 5855 | amdgpu_ring_write(ring, header); |
| 5699 | BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ | 5856 | BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ |
| 5700 | amdgpu_ring_write(ring, | 5857 | amdgpu_ring_write(ring, |
| @@ -5714,10 +5871,6 @@ static void gfx_v11_0_ring_emit_ib_compute(struct amdgpu_ring *ring, · 13 unmodified lines | |||
| 5714 | unsigned vmid = AMDGPU_JOB_GET_VMID(job); | 5871 | unsigned vmid = AMDGPU_JOB_GET_VMID(job); |
| 5715 | u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24); | 5872 | u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24); |
| 5716 | 5873 | ||
| 5717 | if (ring->is_mes_queue) | ||
| 5718 | /* inherit vmid from mqd */ | ||
| 5719 | control |= 0x40000000; | ||
| 5720 | |||
| 5721 | /* Currently, there is a high possibility to get wave ID mismatch | 5874 | /* Currently, there is a high possibility to get wave ID mismatch |
| 5722 | * between ME and GDS, leading to a hw deadlock, because ME generates | 5875 | * between ME and GDS, leading to a hw deadlock, because ME generates |
| 5723 | * different wave IDs than the GDS expects. This situation happens | 5876 | * different wave IDs than the GDS expects. This situation happens |
| @@ -5775,8 +5928,7 @@ static void gfx_v11_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, · 51 unmodified lines | |||
| 5775 | amdgpu_ring_write(ring, upper_32_bits(addr)); | 5928 | amdgpu_ring_write(ring, upper_32_bits(addr)); |
| 5776 | amdgpu_ring_write(ring, lower_32_bits(seq)); | 5929 | amdgpu_ring_write(ring, lower_32_bits(seq)); |
| 5777 | amdgpu_ring_write(ring, upper_32_bits(seq)); | 5930 | amdgpu_ring_write(ring, upper_32_bits(seq)); |
| 5778 | amdgpu_ring_write(ring, ring->is_mes_queue ? | 5931 | amdgpu_ring_write(ring, 0); |
| 5779 | (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0); | ||
| 5780 | } | 5932 | } |
| 5781 | 5933 | ||
| 5782 | static void gfx_v11_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) | 5934 | static void gfx_v11_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) |
| @@ -5804,10 +5956,7 @@ static void gfx_v11_0_ring_invalidate_tlbs(struct amdgpu_ring *ring, · 21 unmodified lines | |||
| 5804 | static void gfx_v11_0_ring_emit_vm_flush(struct amdgpu_ring *ring, | 5956 | static void gfx_v11_0_ring_emit_vm_flush(struct amdgpu_ring *ring, |
| 5805 | unsigned vmid, uint64_t pd_addr) | 5957 | unsigned vmid, uint64_t pd_addr) |
| 5806 | { | 5958 | { |
| 5807 | if (ring->is_mes_queue) | 5959 | amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr); |
| 5808 | gfx_v11_0_ring_invalidate_tlbs(ring, 0, 0, false, 0); | ||
| 5809 | else | ||
| 5810 | amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr); | ||
| 5811 | 5960 | ||
| 5812 | /* compute doesn't have PFP */ | 5961 | /* compute doesn't have PFP */ |
| 5813 | if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) { | 5962 | if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) { |
| @@ -6036,28 +6185,13 @@ static void gfx_v11_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume) · 222 unmodified lines | |||
| 6036 | void *de_payload_cpu_addr; | 6185 | void *de_payload_cpu_addr; |
| 6037 | int cnt; | 6186 | int cnt; |
| 6038 | 6187 | ||
| 6039 | if (ring->is_mes_queue) { | 6188 | offset = offsetof(struct v10_gfx_meta_data, de_payload); |
| 6040 | offset = offsetof(struct amdgpu_mes_ctx_meta_data, | 6189 | de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset; |
| 6041 | gfx[0].gfx_meta_data) + | 6190 | de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset; |
| 6042 | offsetof(struct v10_gfx_meta_data, de_payload); | ||
| 6043 | de_payload_gpu_addr = | ||
| 6044 | amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 6045 | de_payload_cpu_addr = | ||
| 6046 | amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 6047 | |||
| 6048 | offset = offsetof(struct amdgpu_mes_ctx_meta_data, | ||
| 6049 | gfx[0].gds_backup) + | ||
| 6050 | offsetof(struct v10_gfx_meta_data, de_payload); | ||
| 6051 | gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 6052 | } else { | ||
| 6053 | offset = offsetof(struct v10_gfx_meta_data, de_payload); | ||
| 6054 | de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset; | ||
| 6055 | de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset; | ||
| 6056 | 6191 | ||
| 6057 | gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) + | 6192 | gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) + |
| 6058 | AMDGPU_CSA_SIZE - adev->gds.gds_size, | 6193 | AMDGPU_CSA_SIZE - adev->gds.gds_size, |
| 6059 | PAGE_SIZE); | 6194 | PAGE_SIZE); |
| 6060 | } | ||
| 6061 | 6195 | ||
| 6062 | de_payload.gds_backup_addrlo = lower_32_bits(gds_addr); | 6196 | de_payload.gds_backup_addrlo = lower_32_bits(gds_addr); |
| 6063 | de_payload.gds_backup_addrhi = upper_32_bits(gds_addr); | 6197 | de_payload.gds_backup_addrhi = upper_32_bits(gds_addr); |
| @@ -6296,25 +6430,23 @@ static int gfx_v11_0_eop_irq(struct amdgpu_device *adev, · 232 unmodified lines | |||
| 6296 | struct amdgpu_irq_src *source, | 6430 | struct amdgpu_irq_src *source, |
| 6297 | struct amdgpu_iv_entry *entry) | 6431 | struct amdgpu_iv_entry *entry) |
| 6298 | { | 6432 | { |
| 6299 | int i; | 6433 | u32 doorbell_offset = entry->src_data[0]; |
| 6300 | u8 me_id, pipe_id, queue_id; | 6434 | u8 me_id, pipe_id, queue_id; |
| 6301 | struct amdgpu_ring *ring; | 6435 | struct amdgpu_ring *ring; |
| 6302 | uint32_t mes_queue_id = entry->src_data[0]; | 6436 | int i; |
| 6303 | 6437 | ||
| 6304 | DRM_DEBUG("IH: CP EOP\n"); | 6438 | DRM_DEBUG("IH: CP EOP\n"); |
| 6305 | 6439 | ||
| 6306 | if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) { | 6440 | if (adev->enable_mes && doorbell_offset) { |
| 6307 | struct amdgpu_mes_queue *queue; | 6441 | struct amdgpu_userq_fence_driver *fence_drv = NULL; |
| 6308 | 6442 | struct xarray *xa = &adev->userq_xa; | |
| 6309 | mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK; | 6443 | unsigned long flags; |
| 6310 | 6444 | ||
| 6311 | spin_lock(&adev->mes.queue_id_lock); | 6445 | xa_lock_irqsave(xa, flags); |
| 6312 | queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id); | 6446 | fence_drv = xa_load(xa, doorbell_offset); |
| 6313 | if (queue) { | 6447 | if (fence_drv) |
| 6314 | DRM_DEBUG("process mes queue id = %d\n", mes_queue_id); | 6448 | amdgpu_userq_fence_driver_process(fence_drv); |
| 6315 | amdgpu_fence_process(queue->ring); | 6449 | xa_unlock_irqrestore(xa, flags); |
| 6316 | } | ||
| 6317 | spin_unlock(&adev->mes.queue_id_lock); | ||
| 6318 | } else { | 6450 | } else { |
| 6319 | me_id = (entry->ring_id & 0x0c) >> 2; | 6451 | me_id = (entry->ring_id & 0x0c) >> 2; |
| 6320 | pipe_id = (entry->ring_id & 0x03) >> 0; | 6452 | pipe_id = (entry->ring_id & 0x03) >> 0; |
| @@ -6481,27 +6613,29 @@ static void gfx_v11_0_handle_priv_fault(struct amdgpu_device *adev, · 160 unmodified lines | |||
| 6481 | pipe_id = (entry->ring_id & 0x03) >> 0; | 6613 | pipe_id = (entry->ring_id & 0x03) >> 0; |
| 6482 | queue_id = (entry->ring_id & 0x70) >> 4; | 6614 | queue_id = (entry->ring_id & 0x70) >> 4; |
| 6483 | 6615 | ||
| 6484 | switch (me_id) { | 6616 | if (!adev->gfx.disable_kq) { |
| 6485 | case 0: | 6617 | switch (me_id) { |
| 6486 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { | 6618 | case 0: |
| 6487 | ring = &adev->gfx.gfx_ring[i]; | 6619 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { |
| 6488 | if (ring->me == me_id && ring->pipe == pipe_id && | 6620 | ring = &adev->gfx.gfx_ring[i]; |
| 6489 | ring->queue == queue_id) | 6621 | if (ring->me == me_id && ring->pipe == pipe_id && |
| 6490 | drm_sched_fault(&ring->sched); | 6622 | ring->queue == queue_id) |
| 6491 | } | 6623 | drm_sched_fault(&ring->sched); |
| 6492 | break; | 6624 | } |
| 6493 | case 1: | 6625 | break; |
| 6494 | case 2: | 6626 | case 1: |
| 6495 | for (i = 0; i < adev->gfx.num_compute_rings; i++) { | 6627 | case 2: |
| 6496 | ring = &adev->gfx.compute_ring[i]; | 6628 | for (i = 0; i < adev->gfx.num_compute_rings; i++) { |
| 6497 | if (ring->me == me_id && ring->pipe == pipe_id && | 6629 | ring = &adev->gfx.compute_ring[i]; |
| 6498 | ring->queue == queue_id) | 6630 | if (ring->me == me_id && ring->pipe == pipe_id && |
| 6499 | drm_sched_fault(&ring->sched); | 6631 | ring->queue == queue_id) |
| 6632 | drm_sched_fault(&ring->sched); | ||
| 6633 | } | ||
| 6634 | break; | ||
| 6635 | default: | ||
| 6636 | BUG(); | ||
| 6637 | break; | ||
| 6500 | } | 6638 | } |
| 6501 | break; | ||
| 6502 | default: | ||
| 6503 | BUG(); | ||
| 6504 | break; | ||
| 6505 | } | 6639 | } |
| 6506 | } | 6640 | } |
| 6507 | 6641 | ||
| @@ -6609,6 +6743,69 @@ static void gfx_v11_0_emit_mem_sync(struct amdgpu_ring *ring) · 101 unmodified lines | |||
| 6609 | amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */ | 6743 | amdgpu_ring_write(ring, gcr_cntl); /* GCR_CNTL */ |
| 6610 | } | 6744 | } |
| 6611 | 6745 | ||
| 6746 | static bool gfx_v11_pipe_reset_support(struct amdgpu_device *adev) | ||
| 6747 | { | ||
| 6748 | /* Disable the pipe reset until the CPFW fully support it.*/ | ||
| 6749 | dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n"); | ||
| 6750 | return false; | ||
| 6751 | } | ||
| 6752 | |||
| 6753 | |||
| 6754 | static int gfx_v11_reset_gfx_pipe(struct amdgpu_ring *ring) | ||
| 6755 | { | ||
| 6756 | struct amdgpu_device *adev = ring->adev; | ||
| 6757 | uint32_t reset_pipe = 0, clean_pipe = 0; | ||
| 6758 | int r; | ||
| 6759 | |||
| 6760 | if (!gfx_v11_pipe_reset_support(adev)) | ||
| 6761 | return -EOPNOTSUPP; | ||
| 6762 | |||
| 6763 | gfx_v11_0_set_safe_mode(adev, 0); | ||
| 6764 | mutex_lock(&adev->srbm_mutex); | ||
| 6765 | soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0); | ||
| 6766 | |||
| 6767 | switch (ring->pipe) { | ||
| 6768 | case 0: | ||
| 6769 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 6770 | PFP_PIPE0_RESET, 1); | ||
| 6771 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 6772 | ME_PIPE0_RESET, 1); | ||
| 6773 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 6774 | PFP_PIPE0_RESET, 0); | ||
| 6775 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 6776 | ME_PIPE0_RESET, 0); | ||
| 6777 | break; | ||
| 6778 | case 1: | ||
| 6779 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 6780 | PFP_PIPE1_RESET, 1); | ||
| 6781 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 6782 | ME_PIPE1_RESET, 1); | ||
| 6783 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 6784 | PFP_PIPE1_RESET, 0); | ||
| 6785 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 6786 | ME_PIPE1_RESET, 0); | ||
| 6787 | break; | ||
| 6788 | default: | ||
| 6789 | break; | ||
| 6790 | } | ||
| 6791 | |||
| 6792 | WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe); | ||
| 6793 | WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe); | ||
| 6794 | |||
| 6795 | r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) - | ||
| 6796 | RS64_FW_UC_START_ADDR_LO; | ||
| 6797 | soc21_grbm_select(adev, 0, 0, 0, 0); | ||
| 6798 | mutex_unlock(&adev->srbm_mutex); | ||
| 6799 | gfx_v11_0_unset_safe_mode(adev, 0); | ||
| 6800 | |||
| 6801 | dev_info(adev->dev, "The ring %s pipe reset to the ME firmware start PC: %s\n", ring->name, | ||
| 6802 | r == 0 ? "successfully" : "failed"); | ||
| 6803 | /* FIXME: Sometimes driver can't cache the ME firmware start PC correctly, | ||
| 6804 | * so the pipe reset status relies on the later gfx ring test result. | ||
| 6805 | */ | ||
| 6806 | return 0; | ||
| 6807 | } | ||
| 6808 | |||
| 6612 | static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) | 6809 | static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) |
| 6613 | { | 6810 | { |
| 6614 | struct amdgpu_device *adev = ring->adev; | 6811 | struct amdgpu_device *adev = ring->adev; |
| @@ -6618,8 +6815,13 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 3 unmodified lines | |||
| 6618 | return -EINVAL; | 6815 | return -EINVAL; |
| 6619 | 6816 | ||
| 6620 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false); | 6817 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false); |
| 6621 | if (r) | 6818 | if (r) { |
| 6622 | return r; | 6819 | |
| 6820 | dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r); | ||
| 6821 | r = gfx_v11_reset_gfx_pipe(ring); | ||
| 6822 | if (r) | ||
| 6823 | return r; | ||
| 6824 | } | ||
| 6623 | 6825 | ||
| 6624 | r = gfx_v11_0_kgq_init_queue(ring, true); | 6826 | r = gfx_v11_0_kgq_init_queue(ring, true); |
| 6625 | if (r) { | 6827 | if (r) { |
| @@ -6636,6 +6838,136 @@ static int gfx_v11_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 10 unmodified lines | |||
| 6636 | return amdgpu_ring_test_ring(ring); | 6838 | return amdgpu_ring_test_ring(ring); |
| 6637 | } | 6839 | } |
| 6638 | 6840 | ||
| 6841 | static int gfx_v11_0_reset_compute_pipe(struct amdgpu_ring *ring) | ||
| 6842 | { | ||
| 6843 | |||
| 6844 | struct amdgpu_device *adev = ring->adev; | ||
| 6845 | uint32_t reset_pipe = 0, clean_pipe = 0; | ||
| 6846 | int r; | ||
| 6847 | |||
| 6848 | if (!gfx_v11_pipe_reset_support(adev)) | ||
| 6849 | return -EOPNOTSUPP; | ||
| 6850 | |||
| 6851 | gfx_v11_0_set_safe_mode(adev, 0); | ||
| 6852 | mutex_lock(&adev->srbm_mutex); | ||
| 6853 | soc21_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0); | ||
| 6854 | |||
| 6855 | reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL); | ||
| 6856 | clean_pipe = reset_pipe; | ||
| 6857 | |||
| 6858 | if (adev->gfx.rs64_enable) { | ||
| 6859 | |||
| 6860 | switch (ring->pipe) { | ||
| 6861 | case 0: | ||
| 6862 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 6863 | MEC_PIPE0_RESET, 1); | ||
| 6864 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 6865 | MEC_PIPE0_RESET, 0); | ||
| 6866 | break; | ||
| 6867 | case 1: | ||
| 6868 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 6869 | MEC_PIPE1_RESET, 1); | ||
| 6870 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 6871 | MEC_PIPE1_RESET, 0); | ||
| 6872 | break; | ||
| 6873 | case 2: | ||
| 6874 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 6875 | MEC_PIPE2_RESET, 1); | ||
| 6876 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 6877 | MEC_PIPE2_RESET, 0); | ||
| 6878 | break; | ||
| 6879 | case 3: | ||
| 6880 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 6881 | MEC_PIPE3_RESET, 1); | ||
| 6882 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 6883 | MEC_PIPE3_RESET, 0); | ||
| 6884 | break; | ||
| 6885 | default: | ||
| 6886 | break; | ||
| 6887 | } | ||
| 6888 | WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe); | ||
| 6889 | WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe); | ||
| 6890 | r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) - | ||
| 6891 | RS64_FW_UC_START_ADDR_LO; | ||
| 6892 | } else { | ||
| 6893 | if (ring->me == 1) { | ||
| 6894 | switch (ring->pipe) { | ||
| 6895 | case 0: | ||
| 6896 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6897 | MEC_ME1_PIPE0_RESET, 1); | ||
| 6898 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6899 | MEC_ME1_PIPE0_RESET, 0); | ||
| 6900 | break; | ||
| 6901 | case 1: | ||
| 6902 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6903 | MEC_ME1_PIPE1_RESET, 1); | ||
| 6904 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6905 | MEC_ME1_PIPE1_RESET, 0); | ||
| 6906 | break; | ||
| 6907 | case 2: | ||
| 6908 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6909 | MEC_ME1_PIPE2_RESET, 1); | ||
| 6910 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6911 | MEC_ME1_PIPE2_RESET, 0); | ||
| 6912 | break; | ||
| 6913 | case 3: | ||
| 6914 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6915 | MEC_ME1_PIPE3_RESET, 1); | ||
| 6916 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6917 | MEC_ME1_PIPE3_RESET, 0); | ||
| 6918 | break; | ||
| 6919 | default: | ||
| 6920 | break; | ||
| 6921 | } | ||
| 6922 | /* mec1 fw pc: CP_MEC1_INSTR_PNTR */ | ||
| 6923 | } else { | ||
| 6924 | switch (ring->pipe) { | ||
| 6925 | case 0: | ||
| 6926 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6927 | MEC_ME2_PIPE0_RESET, 1); | ||
| 6928 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6929 | MEC_ME2_PIPE0_RESET, 0); | ||
| 6930 | break; | ||
| 6931 | case 1: | ||
| 6932 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6933 | MEC_ME2_PIPE1_RESET, 1); | ||
| 6934 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6935 | MEC_ME2_PIPE1_RESET, 0); | ||
| 6936 | break; | ||
| 6937 | case 2: | ||
| 6938 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6939 | MEC_ME2_PIPE2_RESET, 1); | ||
| 6940 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6941 | MEC_ME2_PIPE2_RESET, 0); | ||
| 6942 | break; | ||
| 6943 | case 3: | ||
| 6944 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 6945 | MEC_ME2_PIPE3_RESET, 1); | ||
| 6946 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 6947 | MEC_ME2_PIPE3_RESET, 0); | ||
| 6948 | break; | ||
| 6949 | default: | ||
| 6950 | break; | ||
| 6951 | } | ||
| 6952 | /* mec2 fw pc: CP:CP_MEC2_INSTR_PNTR */ | ||
| 6953 | } | ||
| 6954 | WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe); | ||
| 6955 | WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe); | ||
| 6956 | r = RREG32(SOC15_REG_OFFSET(GC, 0, regCP_MEC1_INSTR_PNTR)); | ||
| 6957 | } | ||
| 6958 | |||
| 6959 | soc21_grbm_select(adev, 0, 0, 0, 0); | ||
| 6960 | mutex_unlock(&adev->srbm_mutex); | ||
| 6961 | gfx_v11_0_unset_safe_mode(adev, 0); | ||
| 6962 | |||
| 6963 | dev_info(adev->dev, "The ring %s pipe resets to MEC FW start PC: %s\n", ring->name, | ||
| 6964 | r == 0 ? "successfully" : "failed"); | ||
| 6965 | /*FIXME:Sometimes driver can't cache the MEC firmware start PC correctly, so the pipe | ||
| 6966 | * reset status relies on the compute ring test result. | ||
| 6967 | */ | ||
| 6968 | return 0; | ||
| 6969 | } | ||
| 6970 | |||
| 6639 | static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) | 6971 | static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) |
| 6640 | { | 6972 | { |
| 6641 | struct amdgpu_device *adev = ring->adev; | 6973 | struct amdgpu_device *adev = ring->adev; |
| @@ -6646,8 +6978,10 @@ static int gfx_v11_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines | |||
| 6646 | 6978 | ||
| 6647 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true); | 6979 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true); |
| 6648 | if (r) { | 6980 | if (r) { |
| 6649 | dev_err(adev->dev, "reset via MMIO failed %d\n", r); | 6981 | dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r); |
| 6650 | return r; | 6982 | r = gfx_v11_0_reset_compute_pipe(ring); |
| 6983 | if (r) | ||
| 6984 | return r; | ||
| 6651 | } | 6985 | } |
| 6652 | 6986 | ||
| 6653 | r = gfx_v11_0_kcq_init_queue(ring, true); | 6987 | r = gfx_v11_0_kcq_init_queue(ring, true); |
| @@ -6693,9 +7027,14 @@ static void gfx_v11_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printe · 39 unmodified lines | |||
| 6693 | for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) { | 7027 | for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) { |
| 6694 | drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k); | 7028 | drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k); |
| 6695 | for (reg = 0; reg < reg_count; reg++) { | 7029 | for (reg = 0; reg < reg_count; reg++) { |
| 6696 | drm_printf(p, "%-50s \t 0x%08x\n", | 7030 | if (i && gc_cp_reg_list_11[reg].reg_offset == regCP_MEC_ME1_HEADER_DUMP) |
| 6697 | gc_cp_reg_list_11[reg].reg_name, | 7031 | drm_printf(p, "%-50s \t 0x%08x\n", |
| 6698 | adev->gfx.ip_dump_compute_queues[index + reg]); | 7032 | "regCP_MEC_ME2_HEADER_DUMP", |
| 7033 | adev->gfx.ip_dump_compute_queues[index + reg]); | ||
| 7034 | else | ||
| 7035 | drm_printf(p, "%-50s \t 0x%08x\n", | ||
| 7036 | gc_cp_reg_list_11[reg].reg_name, | ||
| 7037 | adev->gfx.ip_dump_compute_queues[index + reg]); | ||
| 6699 | } | 7038 | } |
| 6700 | index += reg_count; | 7039 | index += reg_count; |
| 6701 | } | 7040 | } |
| @@ -6755,9 +7094,16 @@ static void gfx_v11_ip_dump(struct amdgpu_ip_block *ip_block) · 53 unmodified lines | |||
| 6755 | /* ME0 is for GFX so start from 1 for CP */ | 7094 | /* ME0 is for GFX so start from 1 for CP */ |
| 6756 | soc21_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0); | 7095 | soc21_grbm_select(adev, adev->gfx.me.num_me + i, j, k, 0); |
| 6757 | for (reg = 0; reg < reg_count; reg++) { | 7096 | for (reg = 0; reg < reg_count; reg++) { |
| 6758 | adev->gfx.ip_dump_compute_queues[index + reg] = | 7097 | if (i && |
| 6759 | RREG32(SOC15_REG_ENTRY_OFFSET( | 7098 | gc_cp_reg_list_11[reg].reg_offset == |
| 6760 | gc_cp_reg_list_11[reg])); | 7099 | regCP_MEC_ME1_HEADER_DUMP) |
| 7100 | adev->gfx.ip_dump_compute_queues[index + reg] = | ||
| 7101 | RREG32(SOC15_REG_OFFSET(GC, 0, | ||
| 7102 | regCP_MEC_ME2_HEADER_DUMP)); | ||
| 7103 | else | ||
| 7104 | adev->gfx.ip_dump_compute_queues[index + reg] = | ||
| 7105 | RREG32(SOC15_REG_ENTRY_OFFSET( | ||
| 7106 | gc_cp_reg_list_11[reg])); | ||
| 6761 | } | 7107 | } |
| 6762 | index += reg_count; | 7108 | index += reg_count; |
| 6763 | } | 7109 | } |
| @@ -44,6 +44,8 @@ · 43 unmodified lines | |||
| 44 | #include "gfx_v12_0.h" | 44 | #include "gfx_v12_0.h" |
| 45 | #include "nbif_v6_3_1.h" | 45 | #include "nbif_v6_3_1.h" |
| 46 | #include "mes_v12_0.h" | 46 | #include "mes_v12_0.h" |
| 47 | #include "mes_userqueue.h" | ||
| 48 | #include "amdgpu_userq_fence.h" | ||
| 47 | 49 | ||
| 48 | #define GFX12_NUM_GFX_RINGS 1 | 50 | #define GFX12_NUM_GFX_RINGS 1 |
| 49 | #define GFX12_MEC_HPD_SIZE 2048 | 51 | #define GFX12_MEC_HPD_SIZE 2048 |
| @@ -133,11 +135,14 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_12_0[] = { · 83 unmodified lines | |||
| 133 | SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR0), | 135 | SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR0), |
| 134 | SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR1), | 136 | SOC15_REG_ENTRY_STR(GC, 0, regCP_GFX_RS64_INSTR_PNTR1), |
| 135 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_RS64_INSTR_PNTR), | 137 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_RS64_INSTR_PNTR), |
| 136 | |||
| 137 | /* cp header registers */ | 138 | /* cp header registers */ |
| 138 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | 139 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 139 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | 140 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 140 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | 141 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 142 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 143 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 144 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 145 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | ||
| 141 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), | 146 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MES_HEADER_DUMP), |
| 142 | /* SE status registers */ | 147 | /* SE status registers */ |
| 143 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0), | 148 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0), |
| @@ -186,7 +191,16 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_12[] = { · 42 unmodified lines | |||
| 186 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET), | 191 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_OFFSET), |
| 187 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT), | 192 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_CNTL_STACK_DW_CNT), |
| 188 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET), | 193 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_SUSPEND_WG_STATE_OFFSET), |
| 189 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS) | 194 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_DEQUEUE_STATUS), |
| 195 | /* cp header registers */ | ||
| 196 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 197 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 198 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 199 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 200 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 201 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 202 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 203 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 190 | }; | 204 | }; |
| 191 | 205 | ||
| 192 | static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = { | 206 | static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = { |
| @@ -215,7 +229,24 @@ static const struct amdgpu_hwip_reg_entry gc_gfx_queue_reg_list_12[] = { · 22 unmodified lines | |||
| 215 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO), | 229 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_LO), |
| 216 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI), | 230 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BASE_HI), |
| 217 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ), | 231 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_CMD_BUFSZ), |
| 218 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ) | 232 | SOC15_REG_ENTRY_STR(GC, 0, regCP_IB1_BUFSZ), |
| 233 | /* cp header registers */ | ||
| 234 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 235 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 236 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 237 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 238 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 239 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 240 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 241 | SOC15_REG_ENTRY_STR(GC, 0, regCP_PFP_HEADER_DUMP), | ||
| 242 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 243 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 244 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 245 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 246 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 247 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 248 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 249 | SOC15_REG_ENTRY_STR(GC, 0, regCP_ME_HEADER_DUMP), | ||
| 219 | }; | 250 | }; |
| 220 | 251 | ||
| 221 | static const struct soc15_reg_golden golden_settings_gc_12_0_rev0[] = { | 252 | static const struct soc15_reg_golden golden_settings_gc_12_0_rev0[] = { |
| @@ -475,33 +506,18 @@ static int gfx_v12_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 253 unmodified lines | |||
| 475 | 506 | ||
| 476 | memset(&ib, 0, sizeof(ib)); | 507 | memset(&ib, 0, sizeof(ib)); |
| 477 | 508 | ||
| 478 | if (ring->is_mes_queue) { | 509 | r = amdgpu_device_wb_get(adev, &index); |
| 479 | uint32_t padding, offset; | 510 | if (r) |
| 480 | 511 | return r; | |
| 481 | offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS); | ||
| 482 | padding = amdgpu_mes_ctx_get_offs(ring, | ||
| 483 | AMDGPU_MES_CTX_PADDING_OFFS); | ||
| 484 | |||
| 485 | ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 486 | ib.ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 487 | |||
| 488 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, padding); | ||
| 489 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, padding); | ||
| 490 | *cpu_ptr = cpu_to_le32(0xCAFEDEAD); | ||
| 491 | } else { | ||
| 492 | r = amdgpu_device_wb_get(adev, &index); | ||
| 493 | if (r) | ||
| 494 | return r; | ||
| 495 | 512 | ||
| 496 | gpu_addr = adev->wb.gpu_addr + (index * 4); | 513 | gpu_addr = adev->wb.gpu_addr + (index * 4); |
| 497 | adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD); | 514 | adev->wb.wb[index] = cpu_to_le32(0xCAFEDEAD); |
| 498 | cpu_ptr = &adev->wb.wb[index]; | 515 | cpu_ptr = &adev->wb.wb[index]; |
| 499 | 516 | ||
| 500 | r = amdgpu_ib_get(adev, NULL, 16, AMDGPU_IB_POOL_DIRECT, &ib); | 517 | r = amdgpu_ib_get(adev, NULL, 16, AMDGPU_IB_POOL_DIRECT, &ib); |
| 501 | if (r) { | 518 | if (r) { |
| 502 | dev_err(adev->dev, "amdgpu: failed to get ib (%ld).\n", r); | 519 | dev_err(adev->dev, "amdgpu: failed to get ib (%ld).\n", r); |
| 503 | goto err1; | 520 | goto err1; |
| 504 | } | ||
| 505 | } | 521 | } |
| 506 | 522 | ||
| 507 | ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3); | 523 | ib.ptr[0] = PACKET3(PACKET3_WRITE_DATA, 3); |
| @@ -528,12 +544,10 @@ static int gfx_v12_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 20 unmodified lines | |||
| 528 | else | 544 | else |
| 529 | r = -EINVAL; | 545 | r = -EINVAL; |
| 530 | err2: | 546 | err2: |
| 531 | if (!ring->is_mes_queue) | 547 | amdgpu_ib_free(&ib, NULL); |
| 532 | amdgpu_ib_free(&ib, NULL); | ||
| 533 | dma_fence_put(f); | 548 | dma_fence_put(f); |
| 534 | err1: | 549 | err1: |
| 535 | if (!ring->is_mes_queue) | 550 | amdgpu_device_wb_free(adev, index); |
| 536 | amdgpu_device_wb_free(adev, index); | ||
| 537 | return r; | 551 | return r; |
| 538 | } | 552 | } |
| 539 | 553 | ||
| @@ -881,6 +895,34 @@ static void gfx_v12_0_select_me_pipe_q(struct amdgpu_device *adev, · 341 unmodified lines | |||
| 881 | soc24_grbm_select(adev, me, pipe, q, vm); | 895 | soc24_grbm_select(adev, me, pipe, q, vm); |
| 882 | } | 896 | } |
| 883 | 897 | ||
| 898 | /* all sizes are in bytes */ | ||
| 899 | #define MQD_SHADOW_BASE_SIZE 73728 | ||
| 900 | #define MQD_SHADOW_BASE_ALIGNMENT 256 | ||
| 901 | #define MQD_FWWORKAREA_SIZE 484 | ||
| 902 | #define MQD_FWWORKAREA_ALIGNMENT 256 | ||
| 903 | |||
| 904 | static void gfx_v12_0_get_gfx_shadow_info_nocheck(struct amdgpu_device *adev, | ||
| 905 | struct amdgpu_gfx_shadow_info *shadow_info) | ||
| 906 | { | ||
| 907 | shadow_info->shadow_size = MQD_SHADOW_BASE_SIZE; | ||
| 908 | shadow_info->shadow_alignment = MQD_SHADOW_BASE_ALIGNMENT; | ||
| 909 | shadow_info->csa_size = MQD_FWWORKAREA_SIZE; | ||
| 910 | shadow_info->csa_alignment = MQD_FWWORKAREA_ALIGNMENT; | ||
| 911 | } | ||
| 912 | |||
| 913 | static int gfx_v12_0_get_gfx_shadow_info(struct amdgpu_device *adev, | ||
| 914 | struct amdgpu_gfx_shadow_info *shadow_info, | ||
| 915 | bool skip_check) | ||
| 916 | { | ||
| 917 | if (adev->gfx.cp_gfx_shadow || skip_check) { | ||
| 918 | gfx_v12_0_get_gfx_shadow_info_nocheck(adev, shadow_info); | ||
| 919 | return 0; | ||
| 920 | } | ||
| 921 | |||
| 922 | memset(shadow_info, 0, sizeof(struct amdgpu_gfx_shadow_info)); | ||
| 923 | return -EINVAL; | ||
| 924 | } | ||
| 925 | |||
| 884 | static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = { | 926 | static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = { |
| 885 | .get_gpu_clock_counter = &gfx_v12_0_get_gpu_clock_counter, | 927 | .get_gpu_clock_counter = &gfx_v12_0_get_gpu_clock_counter, |
| 886 | .select_se_sh = &gfx_v12_0_select_se_sh, | 928 | .select_se_sh = &gfx_v12_0_select_se_sh, |
| @@ -889,6 +931,7 @@ static const struct amdgpu_gfx_funcs gfx_v12_0_gfx_funcs = { · 2 unmodified lines | |||
| 889 | .read_wave_vgprs = &gfx_v12_0_read_wave_vgprs, | 931 | .read_wave_vgprs = &gfx_v12_0_read_wave_vgprs, |
| 890 | .select_me_pipe_q = &gfx_v12_0_select_me_pipe_q, | 932 | .select_me_pipe_q = &gfx_v12_0_select_me_pipe_q, |
| 891 | .update_perfmon_mgcg = &gfx_v12_0_update_perf_clk, | 933 | .update_perfmon_mgcg = &gfx_v12_0_update_perf_clk, |
| 934 | .get_gfx_shadow_info = &gfx_v12_0_get_gfx_shadow_info, | ||
| 892 | }; | 935 | }; |
| 893 | 936 | ||
| 894 | static int gfx_v12_0_gpu_early_init(struct amdgpu_device *adev) | 937 | static int gfx_v12_0_gpu_early_init(struct amdgpu_device *adev) |
| @@ -1346,6 +1389,7 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 451 unmodified lines | |||
| 1346 | unsigned num_compute_rings; | 1389 | unsigned num_compute_rings; |
| 1347 | int xcc_id = 0; | 1390 | int xcc_id = 0; |
| 1348 | struct amdgpu_device *adev = ip_block->adev; | 1391 | struct amdgpu_device *adev = ip_block->adev; |
| 1392 | int num_queue_per_pipe = 1; /* we only enable 1 KGQ per pipe */ | ||
| 1349 | 1393 | ||
| 1350 | INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler); | 1394 | INIT_DELAYED_WORK(&adev->gfx.idle_work, amdgpu_gfx_profile_idle_work_handler); |
| 1351 | 1395 | ||
| @@ -1354,7 +1398,7 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 2 unmodified lines | |||
| 1354 | case IP_VERSION(12, 0, 1): | 1398 | case IP_VERSION(12, 0, 1): |
| 1355 | adev->gfx.me.num_me = 1; | 1399 | adev->gfx.me.num_me = 1; |
| 1356 | adev->gfx.me.num_pipe_per_me = 1; | 1400 | adev->gfx.me.num_pipe_per_me = 1; |
| 1357 | adev->gfx.me.num_queue_per_pipe = 1; | 1401 | adev->gfx.me.num_queue_per_pipe = 8; |
| 1358 | adev->gfx.mec.num_mec = 1; | 1402 | adev->gfx.mec.num_mec = 1; |
| 1359 | adev->gfx.mec.num_pipe_per_mec = 2; | 1403 | adev->gfx.mec.num_pipe_per_mec = 2; |
| 1360 | adev->gfx.mec.num_queue_per_pipe = 4; | 1404 | adev->gfx.mec.num_queue_per_pipe = 4; |
| @@ -1369,6 +1413,22 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines | |||
| 1369 | break; | 1413 | break; |
| 1370 | } | 1414 | } |
| 1371 | 1415 | ||
| 1416 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | ||
| 1417 | case IP_VERSION(12, 0, 0): | ||
| 1418 | case IP_VERSION(12, 0, 1): | ||
| 1419 | if (!adev->gfx.disable_uq && | ||
| 1420 | adev->gfx.me_fw_version >= 2780 && | ||
| 1421 | adev->gfx.pfp_fw_version >= 2840 && | ||
| 1422 | adev->gfx.mec_fw_version >= 3050 && | ||
| 1423 | adev->mes.fw_version[0] >= 123) { | ||
| 1424 | adev->userq_funcs[AMDGPU_HW_IP_GFX] = &userq_mes_funcs; | ||
| 1425 | adev->userq_funcs[AMDGPU_HW_IP_COMPUTE] = &userq_mes_funcs; | ||
| 1426 | } | ||
| 1427 | break; | ||
| 1428 | default: | ||
| 1429 | break; | ||
| 1430 | } | ||
| 1431 | |||
| 1372 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { | 1432 | switch (amdgpu_ip_version(adev, GC_HWIP, 0)) { |
| 1373 | case IP_VERSION(12, 0, 0): | 1433 | case IP_VERSION(12, 0, 0): |
| 1374 | case IP_VERSION(12, 0, 1): | 1434 | case IP_VERSION(12, 0, 1): |
| @@ -1383,11 +1443,13 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 8 unmodified lines | |||
| 1383 | break; | 1443 | break; |
| 1384 | } | 1444 | } |
| 1385 | 1445 | ||
| 1386 | /* recalculate compute rings to use based on hardware configuration */ | 1446 | if (adev->gfx.num_compute_rings) { |
| 1387 | num_compute_rings = (adev->gfx.mec.num_pipe_per_mec * | 1447 | /* recalculate compute rings to use based on hardware configuration */ |
| 1388 | adev->gfx.mec.num_queue_per_pipe) / 2; | 1448 | num_compute_rings = (adev->gfx.mec.num_pipe_per_mec * |
| 1389 | adev->gfx.num_compute_rings = min(adev->gfx.num_compute_rings, | 1449 | adev->gfx.mec.num_queue_per_pipe) / 2; |
| 1390 | num_compute_rings); | 1450 | adev->gfx.num_compute_rings = min(adev->gfx.num_compute_rings, |
| 1451 | num_compute_rings); | ||
| 1452 | } | ||
| 1391 | 1453 | ||
| 1392 | /* EOP Event */ | 1454 | /* EOP Event */ |
| 1393 | r = amdgpu_irq_add_id(adev, SOC21_IH_CLIENTID_GRBM_CP, | 1455 | r = amdgpu_irq_add_id(adev, SOC21_IH_CLIENTID_GRBM_CP, |
| @@ -1433,37 +1495,41 @@ static int gfx_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 39 unmodified lines | |||
| 1433 | return r; | 1495 | return r; |
| 1434 | } | 1496 | } |
| 1435 | 1497 | ||
| 1436 | /* set up the gfx ring */ | 1498 | if (adev->gfx.num_gfx_rings) { |
| 1437 | for (i = 0; i < adev->gfx.me.num_me; i++) { | 1499 | /* set up the gfx ring */ |
| 1438 | for (j = 0; j < adev->gfx.me.num_queue_per_pipe; j++) { | 1500 | for (i = 0; i < adev->gfx.me.num_me; i++) { |
| 1439 | for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) { | 1501 | for (j = 0; j < num_queue_per_pipe; j++) { |
| 1440 | if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j)) | 1502 | for (k = 0; k < adev->gfx.me.num_pipe_per_me; k++) { |
| 1441 | continue; | 1503 | if (!amdgpu_gfx_is_me_queue_enabled(adev, i, k, j)) |
| 1442 | 1504 | continue; | |
| 1443 | r = gfx_v12_0_gfx_ring_init(adev, ring_id, | 1505 | |
| 1444 | i, k, j); | 1506 | r = gfx_v12_0_gfx_ring_init(adev, ring_id, |
| 1445 | if (r) | 1507 | i, k, j); |
| 1446 | return r; | 1508 | if (r) |
| 1447 | ring_id++; | 1509 | return r; |
| 1510 | ring_id++; | ||
| 1511 | } | ||
| 1448 | } | 1512 | } |
| 1449 | } | 1513 | } |
| 1450 | } | 1514 | } |
| 1451 | 1515 | ||
| 1452 | ring_id = 0; | 1516 | if (adev->gfx.num_compute_rings) { |
| 1453 | /* set up the compute queues - allocate horizontally across pipes */ | 1517 | ring_id = 0; |
| 1454 | for (i = 0; i < adev->gfx.mec.num_mec; ++i) { | 1518 | /* set up the compute queues - allocate horizontally across pipes */ |
| 1455 | for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) { | 1519 | for (i = 0; i < adev->gfx.mec.num_mec; ++i) { |
| 1456 | for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) { | 1520 | for (j = 0; j < adev->gfx.mec.num_queue_per_pipe; j++) { |
| 1457 | if (!amdgpu_gfx_is_mec_queue_enabled(adev, | 1521 | for (k = 0; k < adev->gfx.mec.num_pipe_per_mec; k++) { |
| 1458 | 0, i, k, j)) | 1522 | if (!amdgpu_gfx_is_mec_queue_enabled(adev, |
| 1459 | continue; | 1523 | 0, i, k, j)) |
| 1524 | continue; | ||
| 1460 | 1525 | ||
| 1461 | r = gfx_v12_0_compute_ring_init(adev, ring_id, | 1526 | r = gfx_v12_0_compute_ring_init(adev, ring_id, |
| 1462 | i, k, j); | 1527 | i, k, j); |
| 1463 | if (r) | 1528 | if (r) |
| 1464 | return r; | 1529 | return r; |
| 1465 | 1530 | ||
| 1466 | ring_id++; | 1531 | ring_id++; |
| 1532 | } | ||
| 1467 | } | 1533 | } |
| 1468 | } | 1534 | } |
| 1469 | } | 1535 | } |
| @@ -2948,6 +3014,8 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 1478 unmodified lines | |||
| 2948 | #ifdef __BIG_ENDIAN | 3014 | #ifdef __BIG_ENDIAN |
| 2949 | tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1); | 3015 | tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, BUF_SWAP, 1); |
| 2950 | #endif | 3016 | #endif |
| 3017 | if (prop->tmz_queue) | ||
| 3018 | tmp = REG_SET_FIELD(tmp, CP_GFX_HQD_CNTL, TMZ_MATCH, 1); | ||
| 2951 | mqd->cp_gfx_hqd_cntl = tmp; | 3019 | mqd->cp_gfx_hqd_cntl = tmp; |
| 2952 | 3020 | ||
| 2953 | /* set up cp_doorbell_control */ | 3021 | /* set up cp_doorbell_control */ |
| @@ -2968,6 +3036,14 @@ static int gfx_v12_0_gfx_mqd_init(struct amdgpu_device *adev, void *m, · 14 unmodified lines | |||
| 2968 | /* active the queue */ | 3036 | /* active the queue */ |
| 2969 | mqd->cp_gfx_hqd_active = 1; | 3037 | mqd->cp_gfx_hqd_active = 1; |
| 2970 | 3038 | ||
| 3039 | /* set gfx UQ items */ | ||
| 3040 | mqd->shadow_base_lo = lower_32_bits(prop->shadow_addr); | ||
| 3041 | mqd->shadow_base_hi = upper_32_bits(prop->shadow_addr); | ||
| 3042 | mqd->fw_work_area_base_lo = lower_32_bits(prop->csa_addr); | ||
| 3043 | mqd->fw_work_area_base_hi = upper_32_bits(prop->csa_addr); | ||
| 3044 | mqd->fence_address_lo = lower_32_bits(prop->fence_address); | ||
| 3045 | mqd->fence_address_hi = upper_32_bits(prop->fence_address); | ||
| 3046 | |||
| 2971 | return 0; | 3047 | return 0; |
| 2972 | } | 3048 | } |
| 2973 | 3049 | ||
| @@ -3091,6 +3167,8 @@ static int gfx_v12_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 117 unmodified lines | |||
| 3091 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TUNNEL_DISPATCH, 0); | 3167 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TUNNEL_DISPATCH, 0); |
| 3092 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1); | 3168 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, PRIV_STATE, 1); |
| 3093 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1); | 3169 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, KMD_QUEUE, 1); |
| 3170 | if (prop->tmz_queue) | ||
| 3171 | tmp = REG_SET_FIELD(tmp, CP_HQD_PQ_CONTROL, TMZ, 1); | ||
| 3094 | mqd->cp_hqd_pq_control = tmp; | 3172 | mqd->cp_hqd_pq_control = tmp; |
| 3095 | 3173 | ||
| 3096 | /* set the wb address whether it's enabled or not */ | 3174 | /* set the wb address whether it's enabled or not */ |
| @@ -3142,6 +3220,10 @@ static int gfx_v12_0_compute_mqd_init(struct amdgpu_device *adev, void *m, · 45 unmodified lines | |||
| 3142 | 3220 | ||
| 3143 | mqd->cp_hqd_active = prop->hqd_active; | 3221 | mqd->cp_hqd_active = prop->hqd_active; |
| 3144 | 3222 | ||
| 3223 | /* set UQ fenceaddress */ | ||
| 3224 | mqd->fence_address_lo = lower_32_bits(prop->fence_address); | ||
| 3225 | mqd->fence_address_hi = upper_32_bits(prop->fence_address); | ||
| 3226 | |||
| 3145 | return 0; | 3227 | return 0; |
| 3146 | } | 3228 | } |
| 3147 | 3229 | ||
| @@ -3600,6 +3682,49 @@ static int gfx_v12_0_hw_init(struct amdgpu_ip_block *ip_block) · 452 unmodified lines | |||
| 3600 | return r; | 3682 | return r; |
| 3601 | } | 3683 | } |
| 3602 | 3684 | ||
| 3685 | static int gfx_v12_0_set_userq_eop_interrupts(struct amdgpu_device *adev, | ||
| 3686 | bool enable) | ||
| 3687 | { | ||
| 3688 | unsigned int irq_type; | ||
| 3689 | int m, p, r; | ||
| 3690 | |||
| 3691 | if (adev->userq_funcs[AMDGPU_HW_IP_GFX]) { | ||
| 3692 | for (m = 0; m < adev->gfx.me.num_me; m++) { | ||
| 3693 | for (p = 0; p < adev->gfx.me.num_pipe_per_me; p++) { | ||
| 3694 | irq_type = AMDGPU_CP_IRQ_GFX_ME0_PIPE0_EOP + p; | ||
| 3695 | if (enable) | ||
| 3696 | r = amdgpu_irq_get(adev, &adev->gfx.eop_irq, | ||
| 3697 | irq_type); | ||
| 3698 | else | ||
| 3699 | r = amdgpu_irq_put(adev, &adev->gfx.eop_irq, | ||
| 3700 | irq_type); | ||
| 3701 | if (r) | ||
| 3702 | return r; | ||
| 3703 | } | ||
| 3704 | } | ||
| 3705 | } | ||
| 3706 | |||
| 3707 | if (adev->userq_funcs[AMDGPU_HW_IP_COMPUTE]) { | ||
| 3708 | for (m = 0; m < adev->gfx.mec.num_mec; ++m) { | ||
| 3709 | for (p = 0; p < adev->gfx.mec.num_pipe_per_mec; p++) { | ||
| 3710 | irq_type = AMDGPU_CP_IRQ_COMPUTE_MEC1_PIPE0_EOP | ||
| 3711 | + (m * adev->gfx.mec.num_pipe_per_mec) | ||
| 3712 | + p; | ||
| 3713 | if (enable) | ||
| 3714 | r = amdgpu_irq_get(adev, &adev->gfx.eop_irq, | ||
| 3715 | irq_type); | ||
| 3716 | else | ||
| 3717 | r = amdgpu_irq_put(adev, &adev->gfx.eop_irq, | ||
| 3718 | irq_type); | ||
| 3719 | if (r) | ||
| 3720 | return r; | ||
| 3721 | } | ||
| 3722 | } | ||
| 3723 | } | ||
| 3724 | |||
| 3725 | return 0; | ||
| 3726 | } | ||
| 3727 | |||
| 3603 | static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block) | 3728 | static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block) |
| 3604 | { | 3729 | { |
| 3605 | struct amdgpu_device *adev = ip_block->adev; | 3730 | struct amdgpu_device *adev = ip_block->adev; |
| @@ -3610,6 +3735,7 @@ static int gfx_v12_0_hw_fini(struct amdgpu_ip_block *ip_block) · 4 unmodified lines | |||
| 3610 | amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0); | 3735 | amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0); |
| 3611 | amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0); | 3736 | amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0); |
| 3612 | amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0); | 3737 | amdgpu_irq_put(adev, &adev->gfx.bad_op_irq, 0); |
| 3738 | gfx_v12_0_set_userq_eop_interrupts(adev, false); | ||
| 3613 | 3739 | ||
| 3614 | if (!adev->no_hw_access) { | 3740 | if (!adev->no_hw_access) { |
| 3615 | if (amdgpu_async_gfx_ring) { | 3741 | if (amdgpu_async_gfx_ring) { |
| @@ -3698,11 +3824,33 @@ static int gfx_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 82 unmodified lines | |||
| 3698 | { | 3824 | { |
| 3699 | struct amdgpu_device *adev = ip_block->adev; | 3825 | struct amdgpu_device *adev = ip_block->adev; |
| 3700 | 3826 | ||
| 3827 | switch (amdgpu_user_queue) { | ||
| 3828 | case -1: | ||
| 3829 | case 0: | ||
| 3830 | default: | ||
| 3831 | adev->gfx.disable_kq = false; | ||
| 3832 | adev->gfx.disable_uq = true; | ||
| 3833 | break; | ||
| 3834 | case 1: | ||
| 3835 | adev->gfx.disable_kq = false; | ||
| 3836 | adev->gfx.disable_uq = false; | ||
| 3837 | break; | ||
| 3838 | case 2: | ||
| 3839 | adev->gfx.disable_kq = true; | ||
| 3840 | adev->gfx.disable_uq = false; | ||
| 3841 | break; | ||
| 3842 | } | ||
| 3843 | |||
| 3701 | adev->gfx.funcs = &gfx_v12_0_gfx_funcs; | 3844 | adev->gfx.funcs = &gfx_v12_0_gfx_funcs; |
| 3702 | 3845 | ||
| 3703 | adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS; | 3846 | if (adev->gfx.disable_kq) { |
| 3704 | adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev), | 3847 | adev->gfx.num_gfx_rings = 0; |
| 3705 | AMDGPU_MAX_COMPUTE_RINGS); | 3848 | adev->gfx.num_compute_rings = 0; |
| 3849 | } else { | ||
| 3850 | adev->gfx.num_gfx_rings = GFX12_NUM_GFX_RINGS; | ||
| 3851 | adev->gfx.num_compute_rings = min(amdgpu_gfx_get_num_kcq(adev), | ||
| 3852 | AMDGPU_MAX_COMPUTE_RINGS); | ||
| 3853 | } | ||
| 3706 | 3854 | ||
| 3707 | gfx_v12_0_set_kiq_pm4_funcs(adev); | 3855 | gfx_v12_0_set_kiq_pm4_funcs(adev); |
| 3708 | gfx_v12_0_set_ring_funcs(adev); | 3856 | gfx_v12_0_set_ring_funcs(adev); |
| @@ -3733,6 +3881,10 @@ static int gfx_v12_0_late_init(struct amdgpu_ip_block *ip_block) · 24 unmodified lines | |||
| 3733 | if (r) | 3881 | if (r) |
| 3734 | return r; | 3882 | return r; |
| 3735 | 3883 | ||
| 3884 | r = gfx_v12_0_set_userq_eop_interrupts(adev, true); | ||
| 3885 | if (r) | ||
| 3886 | return r; | ||
| 3887 | |||
| 3736 | return 0; | 3888 | return 0; |
| 3737 | } | 3889 | } |
| 3738 | 3890 | ||
| @@ -4172,45 +4324,17 @@ static u64 gfx_v12_0_ring_get_wptr_gfx(struct amdgpu_ring *ring) · 433 unmodified lines | |||
| 4172 | static void gfx_v12_0_ring_set_wptr_gfx(struct amdgpu_ring *ring) | 4324 | static void gfx_v12_0_ring_set_wptr_gfx(struct amdgpu_ring *ring) |
| 4173 | { | 4325 | { |
| 4174 | struct amdgpu_device *adev = ring->adev; | 4326 | struct amdgpu_device *adev = ring->adev; |
| 4175 | uint32_t *wptr_saved; | ||
| 4176 | uint32_t *is_queue_unmap; | ||
| 4177 | uint64_t aggregated_db_index; | ||
| 4178 | uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_GFX].mqd_size; | ||
| 4179 | uint64_t wptr_tmp; | ||
| 4180 | |||
| 4181 | if (ring->is_mes_queue) { | ||
| 4182 | wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size); | ||
| 4183 | is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size + | ||
| 4184 | sizeof(uint32_t)); | ||
| 4185 | aggregated_db_index = | ||
| 4186 | amdgpu_mes_get_aggregated_doorbell_index(adev, | ||
| 4187 | ring->hw_prio); | ||
| 4188 | |||
| 4189 | wptr_tmp = ring->wptr & ring->buf_mask; | ||
| 4190 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, wptr_tmp); | ||
| 4191 | *wptr_saved = wptr_tmp; | ||
| 4192 | /* assume doorbell always being used by mes mapped queue */ | ||
| 4193 | if (*is_queue_unmap) { | ||
| 4194 | WDOORBELL64(aggregated_db_index, wptr_tmp); | ||
| 4195 | WDOORBELL64(ring->doorbell_index, wptr_tmp); | ||
| 4196 | } else { | ||
| 4197 | WDOORBELL64(ring->doorbell_index, wptr_tmp); | ||
| 4198 | 4327 | ||
| 4199 | if (*is_queue_unmap) | 4328 | if (ring->use_doorbell) { |
| 4200 | WDOORBELL64(aggregated_db_index, wptr_tmp); | 4329 | /* XXX check if swapping is necessary on BE */ |
| 4201 | } | 4330 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, |
| 4331 | ring->wptr); | ||
| 4332 | WDOORBELL64(ring->doorbell_index, ring->wptr); | ||
| 4202 | } else { | 4333 | } else { |
| 4203 | if (ring->use_doorbell) { | 4334 | WREG32_SOC15(GC, 0, regCP_RB0_WPTR, |
| 4204 | /* XXX check if swapping is necessary on BE */ | 4335 | lower_32_bits(ring->wptr)); |
| 4205 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, | 4336 | WREG32_SOC15(GC, 0, regCP_RB0_WPTR_HI, |
| 4206 | ring->wptr); | 4337 | upper_32_bits(ring->wptr)); |
| 4207 | WDOORBELL64(ring->doorbell_index, ring->wptr); | ||
| 4208 | } else { | ||
| 4209 | WREG32_SOC15(GC, 0, regCP_RB0_WPTR, | ||
| 4210 | lower_32_bits(ring->wptr)); | ||
| 4211 | WREG32_SOC15(GC, 0, regCP_RB0_WPTR_HI, | ||
| 4212 | upper_32_bits(ring->wptr)); | ||
| 4213 | } | ||
| 4214 | } | 4338 | } |
| 4215 | } | 4339 | } |
| 4216 | 4340 | ||
| @@ -4235,42 +4359,14 @@ static u64 gfx_v12_0_ring_get_wptr_compute(struct amdgpu_ring *ring) · 18 unmodified lines | |||
| 4235 | static void gfx_v12_0_ring_set_wptr_compute(struct amdgpu_ring *ring) | 4359 | static void gfx_v12_0_ring_set_wptr_compute(struct amdgpu_ring *ring) |
| 4236 | { | 4360 | { |
| 4237 | struct amdgpu_device *adev = ring->adev; | 4361 | struct amdgpu_device *adev = ring->adev; |
| 4238 | uint32_t *wptr_saved; | ||
| 4239 | uint32_t *is_queue_unmap; | ||
| 4240 | uint64_t aggregated_db_index; | ||
| 4241 | uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_COMPUTE].mqd_size; | ||
| 4242 | uint64_t wptr_tmp; | ||
| 4243 | |||
| 4244 | if (ring->is_mes_queue) { | ||
| 4245 | wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size); | ||
| 4246 | is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size + | ||
| 4247 | sizeof(uint32_t)); | ||
| 4248 | aggregated_db_index = | ||
| 4249 | amdgpu_mes_get_aggregated_doorbell_index(adev, | ||
| 4250 | ring->hw_prio); | ||
| 4251 | |||
| 4252 | wptr_tmp = ring->wptr & ring->buf_mask; | ||
| 4253 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, wptr_tmp); | ||
| 4254 | *wptr_saved = wptr_tmp; | ||
| 4255 | /* assume doorbell always used by mes mapped queue */ | ||
| 4256 | if (*is_queue_unmap) { | ||
| 4257 | WDOORBELL64(aggregated_db_index, wptr_tmp); | ||
| 4258 | WDOORBELL64(ring->doorbell_index, wptr_tmp); | ||
| 4259 | } else { | ||
| 4260 | WDOORBELL64(ring->doorbell_index, wptr_tmp); | ||
| 4261 | 4362 | ||
| 4262 | if (*is_queue_unmap) | 4363 | /* XXX check if swapping is necessary on BE */ |
| 4263 | WDOORBELL64(aggregated_db_index, wptr_tmp); | 4364 | if (ring->use_doorbell) { |
| 4264 | } | 4365 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, |
| 4366 | ring->wptr); | ||
| 4367 | WDOORBELL64(ring->doorbell_index, ring->wptr); | ||
| 4265 | } else { | 4368 | } else { |
| 4266 | /* XXX check if swapping is necessary on BE */ | 4369 | BUG(); /* only DOORBELL method supported on gfx12 now */ |
| 4267 | if (ring->use_doorbell) { | ||
| 4268 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, | ||
| 4269 | ring->wptr); | ||
| 4270 | WDOORBELL64(ring->doorbell_index, ring->wptr); | ||
| 4271 | } else { | ||
| 4272 | BUG(); /* only DOORBELL method supported on gfx12 now */ | ||
| 4273 | } | ||
| 4274 | } | 4370 | } |
| 4275 | } | 4371 | } |
| 4276 | 4372 | ||
| @@ -4317,10 +4413,6 @@ static void gfx_v12_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, · 40 unmodified lines | |||
| 4317 | 4413 | ||
| 4318 | control |= ib->length_dw | (vmid << 24); | 4414 | control |= ib->length_dw | (vmid << 24); |
| 4319 | 4415 | ||
| 4320 | if (ring->is_mes_queue) | ||
| 4321 | /* inherit vmid from mqd */ | ||
| 4322 | control |= 0x400000; | ||
| 4323 | |||
| 4324 | amdgpu_ring_write(ring, header); | 4416 | amdgpu_ring_write(ring, header); |
| 4325 | BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ | 4417 | BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ |
| 4326 | amdgpu_ring_write(ring, | 4418 | amdgpu_ring_write(ring, |
| @@ -4340,10 +4432,6 @@ static void gfx_v12_0_ring_emit_ib_compute(struct amdgpu_ring *ring, · 13 unmodified lines | |||
| 4340 | unsigned vmid = AMDGPU_JOB_GET_VMID(job); | 4432 | unsigned vmid = AMDGPU_JOB_GET_VMID(job); |
| 4341 | u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24); | 4433 | u32 control = INDIRECT_BUFFER_VALID | ib->length_dw | (vmid << 24); |
| 4342 | 4434 | ||
| 4343 | if (ring->is_mes_queue) | ||
| 4344 | /* inherit vmid from mqd */ | ||
| 4345 | control |= 0x40000000; | ||
| 4346 | |||
| 4347 | amdgpu_ring_write(ring, PACKET3(PACKET3_INDIRECT_BUFFER, 2)); | 4435 | amdgpu_ring_write(ring, PACKET3(PACKET3_INDIRECT_BUFFER, 2)); |
| 4348 | BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ | 4436 | BUG_ON(ib->gpu_addr & 0x3); /* Dword align */ |
| 4349 | amdgpu_ring_write(ring, | 4437 | amdgpu_ring_write(ring, |
| @@ -4383,8 +4471,7 @@ static void gfx_v12_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, · 33 unmodified lines | |||
| 4383 | amdgpu_ring_write(ring, upper_32_bits(addr)); | 4471 | amdgpu_ring_write(ring, upper_32_bits(addr)); |
| 4384 | amdgpu_ring_write(ring, lower_32_bits(seq)); | 4472 | amdgpu_ring_write(ring, lower_32_bits(seq)); |
| 4385 | amdgpu_ring_write(ring, upper_32_bits(seq)); | 4473 | amdgpu_ring_write(ring, upper_32_bits(seq)); |
| 4386 | amdgpu_ring_write(ring, ring->is_mes_queue ? | 4474 | amdgpu_ring_write(ring, 0); |
| 4387 | (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0); | ||
| 4388 | } | 4475 | } |
| 4389 | 4476 | ||
| 4390 | static void gfx_v12_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) | 4477 | static void gfx_v12_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) |
| @@ -4412,10 +4499,7 @@ static void gfx_v12_0_ring_invalidate_tlbs(struct amdgpu_ring *ring, · 21 unmodified lines | |||
| 4412 | static void gfx_v12_0_ring_emit_vm_flush(struct amdgpu_ring *ring, | 4499 | static void gfx_v12_0_ring_emit_vm_flush(struct amdgpu_ring *ring, |
| 4413 | unsigned vmid, uint64_t pd_addr) | 4500 | unsigned vmid, uint64_t pd_addr) |
| 4414 | { | 4501 | { |
| 4415 | if (ring->is_mes_queue) | 4502 | amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr); |
| 4416 | gfx_v12_0_ring_invalidate_tlbs(ring, 0, 0, false, 0); | ||
| 4417 | else | ||
| 4418 | amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr); | ||
| 4419 | 4503 | ||
| 4420 | /* compute doesn't have PFP */ | 4504 | /* compute doesn't have PFP */ |
| 4421 | if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) { | 4505 | if (ring->funcs->type == AMDGPU_RING_TYPE_GFX) { |
| @@ -4749,25 +4833,23 @@ static int gfx_v12_0_eop_irq(struct amdgpu_device *adev, · 327 unmodified lines | |||
| 4749 | struct amdgpu_irq_src *source, | 4833 | struct amdgpu_irq_src *source, |
| 4750 | struct amdgpu_iv_entry *entry) | 4834 | struct amdgpu_iv_entry *entry) |
| 4751 | { | 4835 | { |
| 4752 | int i; | 4836 | u32 doorbell_offset = entry->src_data[0]; |
| 4753 | u8 me_id, pipe_id, queue_id; | 4837 | u8 me_id, pipe_id, queue_id; |
| 4754 | struct amdgpu_ring *ring; | 4838 | struct amdgpu_ring *ring; |
| 4755 | uint32_t mes_queue_id = entry->src_data[0]; | 4839 | int i; |
| 4756 | 4840 | ||
| 4757 | DRM_DEBUG("IH: CP EOP\n"); | 4841 | DRM_DEBUG("IH: CP EOP\n"); |
| 4758 | 4842 | ||
| 4759 | if (adev->enable_mes && (mes_queue_id & AMDGPU_FENCE_MES_QUEUE_FLAG)) { | 4843 | if (adev->enable_mes && doorbell_offset) { |
| 4760 | struct amdgpu_mes_queue *queue; | 4844 | struct amdgpu_userq_fence_driver *fence_drv = NULL; |
| 4845 | struct xarray *xa = &adev->userq_xa; | ||
| 4846 | unsigned long flags; | ||
| 4761 | 4847 | ||
| 4762 | mes_queue_id &= AMDGPU_FENCE_MES_QUEUE_ID_MASK; | 4848 | xa_lock_irqsave(xa, flags); |
| 4763 | 4849 | fence_drv = xa_load(xa, doorbell_offset); | |
| 4764 | spin_lock(&adev->mes.queue_id_lock); | 4850 | if (fence_drv) |
| 4765 | queue = idr_find(&adev->mes.queue_id_idr, mes_queue_id); | 4851 | amdgpu_userq_fence_driver_process(fence_drv); |
| 4766 | if (queue) { | 4852 | xa_unlock_irqrestore(xa, flags); |
| 4767 | DRM_DEBUG("process mes queue id = %d\n", mes_queue_id); | ||
| 4768 | amdgpu_fence_process(queue->ring); | ||
| 4769 | } | ||
| 4770 | spin_unlock(&adev->mes.queue_id_lock); | ||
| 4771 | } else { | 4853 | } else { |
| 4772 | me_id = (entry->ring_id & 0x0c) >> 2; | 4854 | me_id = (entry->ring_id & 0x0c) >> 2; |
| 4773 | pipe_id = (entry->ring_id & 0x03) >> 0; | 4855 | pipe_id = (entry->ring_id & 0x03) >> 0; |
| @@ -4934,27 +5016,29 @@ static void gfx_v12_0_handle_priv_fault(struct amdgpu_device *adev, · 160 unmodified lines | |||
| 4934 | pipe_id = (entry->ring_id & 0x03) >> 0; | 5016 | pipe_id = (entry->ring_id & 0x03) >> 0; |
| 4935 | queue_id = (entry->ring_id & 0x70) >> 4; | 5017 | queue_id = (entry->ring_id & 0x70) >> 4; |
| 4936 | 5018 | ||
| 4937 | switch (me_id) { | 5019 | if (!adev->gfx.disable_kq) { |
| 4938 | case 0: | 5020 | switch (me_id) { |
| 4939 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { | 5021 | case 0: |
| 4940 | ring = &adev->gfx.gfx_ring[i]; | 5022 | for (i = 0; i < adev->gfx.num_gfx_rings; i++) { |
| 4941 | if (ring->me == me_id && ring->pipe == pipe_id && | 5023 | ring = &adev->gfx.gfx_ring[i]; |
| 4942 | ring->queue == queue_id) | 5024 | if (ring->me == me_id && ring->pipe == pipe_id && |
| 4943 | drm_sched_fault(&ring->sched); | 5025 | ring->queue == queue_id) |
| 4944 | } | 5026 | drm_sched_fault(&ring->sched); |
| 4945 | break; | 5027 | } |
| 4946 | case 1: | 5028 | break; |
| 4947 | case 2: | 5029 | case 1: |
| 4948 | for (i = 0; i < adev->gfx.num_compute_rings; i++) { | 5030 | case 2: |
| 4949 | ring = &adev->gfx.compute_ring[i]; | 5031 | for (i = 0; i < adev->gfx.num_compute_rings; i++) { |
| 4950 | if (ring->me == me_id && ring->pipe == pipe_id && | 5032 | ring = &adev->gfx.compute_ring[i]; |
| 4951 | ring->queue == queue_id) | 5033 | if (ring->me == me_id && ring->pipe == pipe_id && |
| 4952 | drm_sched_fault(&ring->sched); | 5034 | ring->queue == queue_id) |
| 5035 | drm_sched_fault(&ring->sched); | ||
| 5036 | } | ||
| 5037 | break; | ||
| 5038 | default: | ||
| 5039 | BUG(); | ||
| 5040 | break; | ||
| 4953 | } | 5041 | } |
| 4954 | break; | ||
| 4955 | default: | ||
| 4956 | BUG(); | ||
| 4957 | break; | ||
| 4958 | } | 5042 | } |
| 4959 | } | 5043 | } |
| 4960 | 5044 | ||
| @@ -5160,6 +5244,69 @@ static void gfx_v12_ip_dump(struct amdgpu_ip_block *ip_block) · 199 unmodified lines | |||
| 5160 | amdgpu_gfx_off_ctrl(adev, true); | 5244 | amdgpu_gfx_off_ctrl(adev, true); |
| 5161 | } | 5245 | } |
| 5162 | 5246 | ||
| 5247 | static bool gfx_v12_pipe_reset_support(struct amdgpu_device *adev) | ||
| 5248 | { | ||
| 5249 | /* Disable the pipe reset until the CPFW fully support it.*/ | ||
| 5250 | dev_warn_once(adev->dev, "The CPFW hasn't support pipe reset yet.\n"); | ||
| 5251 | return false; | ||
| 5252 | } | ||
| 5253 | |||
| 5254 | static int gfx_v12_reset_gfx_pipe(struct amdgpu_ring *ring) | ||
| 5255 | { | ||
| 5256 | struct amdgpu_device *adev = ring->adev; | ||
| 5257 | uint32_t reset_pipe = 0, clean_pipe = 0; | ||
| 5258 | int r; | ||
| 5259 | |||
| 5260 | if (!gfx_v12_pipe_reset_support(adev)) | ||
| 5261 | return -EOPNOTSUPP; | ||
| 5262 | |||
| 5263 | gfx_v12_0_set_safe_mode(adev, 0); | ||
| 5264 | mutex_lock(&adev->srbm_mutex); | ||
| 5265 | soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0); | ||
| 5266 | |||
| 5267 | switch (ring->pipe) { | ||
| 5268 | case 0: | ||
| 5269 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 5270 | PFP_PIPE0_RESET, 1); | ||
| 5271 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 5272 | ME_PIPE0_RESET, 1); | ||
| 5273 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 5274 | PFP_PIPE0_RESET, 0); | ||
| 5275 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 5276 | ME_PIPE0_RESET, 0); | ||
| 5277 | break; | ||
| 5278 | case 1: | ||
| 5279 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 5280 | PFP_PIPE1_RESET, 1); | ||
| 5281 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_ME_CNTL, | ||
| 5282 | ME_PIPE1_RESET, 1); | ||
| 5283 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 5284 | PFP_PIPE1_RESET, 0); | ||
| 5285 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_ME_CNTL, | ||
| 5286 | ME_PIPE1_RESET, 0); | ||
| 5287 | break; | ||
| 5288 | default: | ||
| 5289 | break; | ||
| 5290 | } | ||
| 5291 | |||
| 5292 | WREG32_SOC15(GC, 0, regCP_ME_CNTL, reset_pipe); | ||
| 5293 | WREG32_SOC15(GC, 0, regCP_ME_CNTL, clean_pipe); | ||
| 5294 | |||
| 5295 | r = (RREG32(SOC15_REG_OFFSET(GC, 0, regCP_GFX_RS64_INSTR_PNTR1)) << 2) - | ||
| 5296 | RS64_FW_UC_START_ADDR_LO; | ||
| 5297 | soc24_grbm_select(adev, 0, 0, 0, 0); | ||
| 5298 | mutex_unlock(&adev->srbm_mutex); | ||
| 5299 | gfx_v12_0_unset_safe_mode(adev, 0); | ||
| 5300 | |||
| 5301 | dev_info(adev->dev, "The ring %s pipe reset: %s\n", ring->name, | ||
| 5302 | r == 0 ? "successfully" : "failed"); | ||
| 5303 | /* Sometimes the ME start pc counter can't cache correctly, so the | ||
| 5304 | * PC check only as a reference and pipe reset result rely on the | ||
| 5305 | * later ring test. | ||
| 5306 | */ | ||
| 5307 | return 0; | ||
| 5308 | } | ||
| 5309 | |||
| 5163 | static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) | 5310 | static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) |
| 5164 | { | 5311 | { |
| 5165 | struct amdgpu_device *adev = ring->adev; | 5312 | struct amdgpu_device *adev = ring->adev; |
| @@ -5170,8 +5317,10 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines | |||
| 5170 | 5317 | ||
| 5171 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false); | 5318 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, false); |
| 5172 | if (r) { | 5319 | if (r) { |
| 5173 | dev_err(adev->dev, "reset via MES failed %d\n", r); | 5320 | dev_warn(adev->dev, "reset via MES failed and try pipe reset %d\n", r); |
| 5174 | return r; | 5321 | r = gfx_v12_reset_gfx_pipe(ring); |
| 5322 | if (r) | ||
| 5323 | return r; | ||
| 5175 | } | 5324 | } |
| 5176 | 5325 | ||
| 5177 | r = gfx_v12_0_kgq_init_queue(ring, true); | 5326 | r = gfx_v12_0_kgq_init_queue(ring, true); |
| @@ -5189,6 +5338,89 @@ static int gfx_v12_0_reset_kgq(struct amdgpu_ring *ring, unsigned int vmid) · 11 unmodified lines | |||
| 5189 | return amdgpu_ring_test_ring(ring); | 5338 | return amdgpu_ring_test_ring(ring); |
| 5190 | } | 5339 | } |
| 5191 | 5340 | ||
| 5341 | static int gfx_v12_0_reset_compute_pipe(struct amdgpu_ring *ring) | ||
| 5342 | { | ||
| 5343 | struct amdgpu_device *adev = ring->adev; | ||
| 5344 | uint32_t reset_pipe = 0, clean_pipe = 0; | ||
| 5345 | int r = 0; | ||
| 5346 | |||
| 5347 | if (!gfx_v12_pipe_reset_support(adev)) | ||
| 5348 | return -EOPNOTSUPP; | ||
| 5349 | |||
| 5350 | gfx_v12_0_set_safe_mode(adev, 0); | ||
| 5351 | mutex_lock(&adev->srbm_mutex); | ||
| 5352 | soc24_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0); | ||
| 5353 | |||
| 5354 | reset_pipe = RREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL); | ||
| 5355 | clean_pipe = reset_pipe; | ||
| 5356 | |||
| 5357 | if (adev->gfx.rs64_enable) { | ||
| 5358 | switch (ring->pipe) { | ||
| 5359 | case 0: | ||
| 5360 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 5361 | MEC_PIPE0_RESET, 1); | ||
| 5362 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 5363 | MEC_PIPE0_RESET, 0); | ||
| 5364 | break; | ||
| 5365 | case 1: | ||
| 5366 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 5367 | MEC_PIPE1_RESET, 1); | ||
| 5368 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 5369 | MEC_PIPE1_RESET, 0); | ||
| 5370 | break; | ||
| 5371 | case 2: | ||
| 5372 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 5373 | MEC_PIPE2_RESET, 1); | ||
| 5374 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 5375 | MEC_PIPE2_RESET, 0); | ||
| 5376 | break; | ||
| 5377 | case 3: | ||
| 5378 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_RS64_CNTL, | ||
| 5379 | MEC_PIPE3_RESET, 1); | ||
| 5380 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_RS64_CNTL, | ||
| 5381 | MEC_PIPE3_RESET, 0); | ||
| 5382 | break; | ||
| 5383 | default: | ||
| 5384 | break; | ||
| 5385 | } | ||
| 5386 | WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, reset_pipe); | ||
| 5387 | WREG32_SOC15(GC, 0, regCP_MEC_RS64_CNTL, clean_pipe); | ||
| 5388 | r = (RREG32_SOC15(GC, 0, regCP_MEC_RS64_INSTR_PNTR) << 2) - | ||
| 5389 | RS64_FW_UC_START_ADDR_LO; | ||
| 5390 | } else { | ||
| 5391 | switch (ring->pipe) { | ||
| 5392 | case 0: | ||
| 5393 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 5394 | MEC_ME1_PIPE0_RESET, 1); | ||
| 5395 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 5396 | MEC_ME1_PIPE0_RESET, 0); | ||
| 5397 | break; | ||
| 5398 | case 1: | ||
| 5399 | reset_pipe = REG_SET_FIELD(reset_pipe, CP_MEC_CNTL, | ||
| 5400 | MEC_ME1_PIPE1_RESET, 1); | ||
| 5401 | clean_pipe = REG_SET_FIELD(clean_pipe, CP_MEC_CNTL, | ||
| 5402 | MEC_ME1_PIPE1_RESET, 0); | ||
| 5403 | break; | ||
| 5404 | default: | ||
| 5405 | break; | ||
| 5406 | } | ||
| 5407 | WREG32_SOC15(GC, 0, regCP_MEC_CNTL, reset_pipe); | ||
| 5408 | WREG32_SOC15(GC, 0, regCP_MEC_CNTL, clean_pipe); | ||
| 5409 | /* Doesn't find the F32 MEC instruction pointer register, and suppose | ||
| 5410 | * the driver won't run into the F32 mode. | ||
| 5411 | */ | ||
| 5412 | } | ||
| 5413 | |||
| 5414 | soc24_grbm_select(adev, 0, 0, 0, 0); | ||
| 5415 | mutex_unlock(&adev->srbm_mutex); | ||
| 5416 | gfx_v12_0_unset_safe_mode(adev, 0); | ||
| 5417 | |||
| 5418 | dev_info(adev->dev, "The ring %s pipe resets: %s\n", ring->name, | ||
| 5419 | r == 0 ? "successfully" : "failed"); | ||
| 5420 | /* Need the ring test to verify the pipe reset result.*/ | ||
| 5421 | return 0; | ||
| 5422 | } | ||
| 5423 | |||
| 5192 | static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) | 5424 | static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) |
| 5193 | { | 5425 | { |
| 5194 | struct amdgpu_device *adev = ring->adev; | 5426 | struct amdgpu_device *adev = ring->adev; |
| @@ -5199,8 +5431,10 @@ static int gfx_v12_0_reset_kcq(struct amdgpu_ring *ring, unsigned int vmid) · 4 unmodified lines | |||
| 5199 | 5431 | ||
| 5200 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true); | 5432 | r = amdgpu_mes_reset_legacy_queue(ring->adev, ring, vmid, true); |
| 5201 | if (r) { | 5433 | if (r) { |
| 5202 | dev_err(adev->dev, "reset via MMIO failed %d\n", r); | 5434 | dev_warn(adev->dev, "fail(%d) to reset kcq and try pipe reset\n", r); |
| 5203 | return r; | 5435 | r = gfx_v12_0_reset_compute_pipe(ring); |
| 5436 | if (r) | ||
| 5437 | return r; | ||
| 5204 | } | 5438 | } |
| 5205 | 5439 | ||
| 5206 | r = gfx_v12_0_kcq_init_queue(ring, true); | 5440 | r = gfx_v12_0_kcq_init_queue(ring, true); |
| @@ -53,6 +53,9 @@ · 52 unmodified lines | |||
| 53 | #define VERDE_GB_ADDR_CONFIG_GOLDEN 0x12010002 | 53 | #define VERDE_GB_ADDR_CONFIG_GOLDEN 0x12010002 |
| 54 | #define HAINAN_GB_ADDR_CONFIG_GOLDEN 0x02010001 | 54 | #define HAINAN_GB_ADDR_CONFIG_GOLDEN 0x02010001 |
| 55 | 55 | ||
| 56 | #define GFX6_NUM_GFX_RINGS 1 | ||
| 57 | #define GFX6_NUM_COMPUTE_RINGS 2 | ||
| 58 | |||
| 56 | static void gfx_v6_0_set_ring_funcs(struct amdgpu_device *adev); | 59 | static void gfx_v6_0_set_ring_funcs(struct amdgpu_device *adev); |
| 57 | static void gfx_v6_0_set_irq_funcs(struct amdgpu_device *adev); | 60 | static void gfx_v6_0_set_irq_funcs(struct amdgpu_device *adev); |
| 58 | static void gfx_v6_0_get_cu_info(struct amdgpu_device *adev); | 61 | static void gfx_v6_0_get_cu_info(struct amdgpu_device *adev); |
| @@ -1732,10 +1735,14 @@ static void gfx_v6_0_constants_init(struct amdgpu_device *adev) · 1673 unmodified lines | |||
| 1732 | gfx_v6_0_get_cu_info(adev); | 1735 | gfx_v6_0_get_cu_info(adev); |
| 1733 | gfx_v6_0_config_init(adev); | 1736 | gfx_v6_0_config_init(adev); |
| 1734 | 1737 | ||
| 1735 | WREG32(mmCP_QUEUE_THRESHOLDS, ((0x16 << CP_QUEUE_THRESHOLDS__ROQ_IB1_START__SHIFT) | | 1738 | WREG32(mmCP_QUEUE_THRESHOLDS, |
| 1736 | (0x2b << CP_QUEUE_THRESHOLDS__ROQ_IB2_START__SHIFT))); | 1739 | ((0x16 << CP_QUEUE_THRESHOLDS__ROQ_IB1_START__SHIFT) | |
| 1737 | WREG32(mmCP_MEQ_THRESHOLDS, (0x30 << CP_MEQ_THRESHOLDS__MEQ1_START__SHIFT) | | 1740 | (0x2b << CP_QUEUE_THRESHOLDS__ROQ_IB2_START__SHIFT))); |
| 1738 | (0x60 << CP_MEQ_THRESHOLDS__MEQ2_START__SHIFT)); | 1741 | |
| 1742 | /* set HW defaults for 3D engine */ | ||
| 1743 | WREG32(mmCP_MEQ_THRESHOLDS, | ||
| 1744 | (0x30 << CP_MEQ_THRESHOLDS__MEQ1_START__SHIFT) | | ||
| 1745 | (0x60 << CP_MEQ_THRESHOLDS__MEQ2_START__SHIFT)); | ||
| 1739 | 1746 | ||
| 1740 | sx_debug_1 = RREG32(mmSX_DEBUG_1); | 1747 | sx_debug_1 = RREG32(mmSX_DEBUG_1); |
| 1741 | WREG32(mmSX_DEBUG_1, sx_debug_1); | 1748 | WREG32(mmSX_DEBUG_1, sx_debug_1); |
| @@ -2851,44 +2858,21 @@ static u32 gfx_v6_0_get_csb_size(struct amdgpu_device *adev) · 1109 unmodified lines | |||
| 2851 | static void gfx_v6_0_get_csb_buffer(struct amdgpu_device *adev, | 2858 | static void gfx_v6_0_get_csb_buffer(struct amdgpu_device *adev, |
| 2852 | volatile u32 *buffer) | 2859 | volatile u32 *buffer) |
| 2853 | { | 2860 | { |
| 2854 | u32 count = 0, i; | 2861 | u32 count = 0; |
| 2855 | const struct cs_section_def *sect = NULL; | ||
| 2856 | const struct cs_extent_def *ext = NULL; | ||
| 2857 | 2862 | ||
| 2858 | if (adev->gfx.rlc.cs_data == NULL) | 2863 | if (adev->gfx.rlc.cs_data == NULL) |
| 2859 | return; | 2864 | return; |
| 2860 | if (buffer == NULL) | 2865 | if (buffer == NULL) |
| 2861 | return; | 2866 | return; |
| 2862 | 2867 | ||
| 2863 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 2868 | count = amdgpu_gfx_csb_preamble_start(buffer); |
| 2864 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | 2869 | count = amdgpu_gfx_csb_data_parser(adev, buffer, count); |
| 2865 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | ||
| 2866 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 2867 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 2868 | |||
| 2869 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 2870 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 2871 | if (sect->id == SECT_CONTEXT) { | ||
| 2872 | buffer[count++] = | ||
| 2873 | cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 2874 | buffer[count++] = cpu_to_le32(ext->reg_index - 0xa000); | ||
| 2875 | for (i = 0; i < ext->reg_count; i++) | ||
| 2876 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 2877 | } else { | ||
| 2878 | return; | ||
| 2879 | } | ||
| 2880 | } | ||
| 2881 | } | ||
| 2882 | 2870 | ||
| 2883 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1)); | 2871 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 1)); |
| 2884 | buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START); | 2872 | buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START); |
| 2885 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config); | 2873 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config); |
| 2886 | 2874 | ||
| 2887 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 2875 | amdgpu_gfx_csb_preamble_end(buffer, count); |
| 2888 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 2889 | |||
| 2890 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | ||
| 2891 | buffer[count++] = cpu_to_le32(0); | ||
| 2892 | } | 2876 | } |
| 2893 | 2877 | ||
| 2894 | static void gfx_v6_0_init_pg(struct amdgpu_device *adev) | 2878 | static void gfx_v6_0_init_pg(struct amdgpu_device *adev) |
| @@ -55,6 +55,9 @@ · 54 unmodified lines | |||
| 55 | #define GFX7_NUM_GFX_RINGS 1 | 55 | #define GFX7_NUM_GFX_RINGS 1 |
| 56 | #define GFX7_MEC_HPD_SIZE 2048 | 56 | #define GFX7_MEC_HPD_SIZE 2048 |
| 57 | 57 | ||
| 58 | #define BONAIRE_GB_ADDR_CONFIG_GOLDEN 0x12010001 | ||
| 59 | #define HAWAII_GB_ADDR_CONFIG_GOLDEN 0x12011003 | ||
| 60 | |||
| 58 | static void gfx_v7_0_set_ring_funcs(struct amdgpu_device *adev); | 61 | static void gfx_v7_0_set_ring_funcs(struct amdgpu_device *adev); |
| 59 | static void gfx_v7_0_set_irq_funcs(struct amdgpu_device *adev); | 62 | static void gfx_v7_0_set_irq_funcs(struct amdgpu_device *adev); |
| 60 | static void gfx_v7_0_set_gds_init(struct amdgpu_device *adev); | 63 | static void gfx_v7_0_set_gds_init(struct amdgpu_device *adev); |
| @@ -3882,67 +3885,22 @@ static u32 gfx_v7_0_get_csb_size(struct amdgpu_device *adev) · 3821 unmodified lines | |||
| 3882 | static void gfx_v7_0_get_csb_buffer(struct amdgpu_device *adev, | 3885 | static void gfx_v7_0_get_csb_buffer(struct amdgpu_device *adev, |
| 3883 | volatile u32 *buffer) | 3886 | volatile u32 *buffer) |
| 3884 | { | 3887 | { |
| 3885 | u32 count = 0, i; | 3888 | u32 count = 0; |
| 3886 | const struct cs_section_def *sect = NULL; | ||
| 3887 | const struct cs_extent_def *ext = NULL; | ||
| 3888 | 3889 | ||
| 3889 | if (adev->gfx.rlc.cs_data == NULL) | 3890 | if (adev->gfx.rlc.cs_data == NULL) |
| 3890 | return; | 3891 | return; |
| 3891 | if (buffer == NULL) | 3892 | if (buffer == NULL) |
| 3892 | return; | 3893 | return; |
| 3893 | 3894 | ||
| 3894 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 3895 | count = amdgpu_gfx_csb_preamble_start(buffer); |
| 3895 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | 3896 | count = amdgpu_gfx_csb_data_parser(adev, buffer, count); |
| 3896 | |||
| 3897 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | ||
| 3898 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 3899 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 3900 | |||
| 3901 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 3902 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 3903 | if (sect->id == SECT_CONTEXT) { | ||
| 3904 | buffer[count++] = | ||
| 3905 | cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 3906 | buffer[count++] = cpu_to_le32(ext->reg_index - PACKET3_SET_CONTEXT_REG_START); | ||
| 3907 | for (i = 0; i < ext->reg_count; i++) | ||
| 3908 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 3909 | } else { | ||
| 3910 | return; | ||
| 3911 | } | ||
| 3912 | } | ||
| 3913 | } | ||
| 3914 | 3897 | ||
| 3915 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2)); | 3898 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2)); |
| 3916 | buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START); | 3899 | buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START); |
| 3917 | switch (adev->asic_type) { | 3900 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config); |
| 3918 | case CHIP_BONAIRE: | 3901 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1); |
| 3919 | buffer[count++] = cpu_to_le32(0x16000012); | ||
| 3920 | buffer[count++] = cpu_to_le32(0x00000000); | ||
| 3921 | break; | ||
| 3922 | case CHIP_KAVERI: | ||
| 3923 | buffer[count++] = cpu_to_le32(0x00000000); /* XXX */ | ||
| 3924 | buffer[count++] = cpu_to_le32(0x00000000); | ||
| 3925 | break; | ||
| 3926 | case CHIP_KABINI: | ||
| 3927 | case CHIP_MULLINS: | ||
| 3928 | buffer[count++] = cpu_to_le32(0x00000000); /* XXX */ | ||
| 3929 | buffer[count++] = cpu_to_le32(0x00000000); | ||
| 3930 | break; | ||
| 3931 | case CHIP_HAWAII: | ||
| 3932 | buffer[count++] = cpu_to_le32(0x3a00161a); | ||
| 3933 | buffer[count++] = cpu_to_le32(0x0000002e); | ||
| 3934 | break; | ||
| 3935 | default: | ||
| 3936 | buffer[count++] = cpu_to_le32(0x00000000); | ||
| 3937 | buffer[count++] = cpu_to_le32(0x00000000); | ||
| 3938 | break; | ||
| 3939 | } | ||
| 3940 | |||
| 3941 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | ||
| 3942 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 3943 | 3902 | ||
| 3944 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | 3903 | amdgpu_gfx_csb_preamble_end(buffer, count); |
| 3945 | buffer[count++] = cpu_to_le32(0); | ||
| 3946 | } | 3904 | } |
| 3947 | 3905 | ||
| 3948 | static void gfx_v7_0_init_pg(struct amdgpu_device *adev) | 3906 | static void gfx_v7_0_init_pg(struct amdgpu_device *adev) |
| @@ -1223,48 +1223,22 @@ out: · 1222 unmodified lines | |||
| 1223 | static void gfx_v8_0_get_csb_buffer(struct amdgpu_device *adev, | 1223 | static void gfx_v8_0_get_csb_buffer(struct amdgpu_device *adev, |
| 1224 | volatile u32 *buffer) | 1224 | volatile u32 *buffer) |
| 1225 | { | 1225 | { |
| 1226 | u32 count = 0, i; | 1226 | u32 count = 0; |
| 1227 | const struct cs_section_def *sect = NULL; | ||
| 1228 | const struct cs_extent_def *ext = NULL; | ||
| 1229 | 1227 | ||
| 1230 | if (adev->gfx.rlc.cs_data == NULL) | 1228 | if (adev->gfx.rlc.cs_data == NULL) |
| 1231 | return; | 1229 | return; |
| 1232 | if (buffer == NULL) | 1230 | if (buffer == NULL) |
| 1233 | return; | 1231 | return; |
| 1234 | 1232 | ||
| 1235 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 1233 | count = amdgpu_gfx_csb_preamble_start(buffer); |
| 1236 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | 1234 | count = amdgpu_gfx_csb_data_parser(adev, buffer, count); |
| 1237 | |||
| 1238 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | ||
| 1239 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 1240 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 1241 | |||
| 1242 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 1243 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 1244 | if (sect->id == SECT_CONTEXT) { | ||
| 1245 | buffer[count++] = | ||
| 1246 | cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 1247 | buffer[count++] = cpu_to_le32(ext->reg_index - | ||
| 1248 | PACKET3_SET_CONTEXT_REG_START); | ||
| 1249 | for (i = 0; i < ext->reg_count; i++) | ||
| 1250 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 1251 | } else { | ||
| 1252 | return; | ||
| 1253 | } | ||
| 1254 | } | ||
| 1255 | } | ||
| 1256 | 1235 | ||
| 1257 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2)); | 1236 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, 2)); |
| 1258 | buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - | 1237 | buffer[count++] = cpu_to_le32(mmPA_SC_RASTER_CONFIG - PACKET3_SET_CONTEXT_REG_START); |
| 1259 | PACKET3_SET_CONTEXT_REG_START); | ||
| 1260 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config); | 1238 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config); |
| 1261 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1); | 1239 | buffer[count++] = cpu_to_le32(adev->gfx.config.rb_config[0][0].raster_config_1); |
| 1262 | 1240 | ||
| 1263 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 1241 | amdgpu_gfx_csb_preamble_end(buffer, count); |
| 1264 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 1265 | |||
| 1266 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | ||
| 1267 | buffer[count++] = cpu_to_le32(0); | ||
| 1268 | } | 1242 | } |
| 1269 | 1243 | ||
| 1270 | static int gfx_v8_0_cp_jump_table_num(struct amdgpu_device *adev) | 1244 | static int gfx_v8_0_cp_jump_table_num(struct amdgpu_device *adev) |
| @@ -225,17 +225,36 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_9[] = { · 224 unmodified lines | |||
| 225 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_SMU_SAFE_MODE), | 225 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_SMU_SAFE_MODE), |
| 226 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_INT_STAT), | 226 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_INT_STAT), |
| 227 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_GENERAL_6), | 227 | SOC15_REG_ENTRY_STR(GC, 0, mmRLC_GPM_GENERAL_6), |
| 228 | /* cp header registers */ | ||
| 229 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 230 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 231 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME2_HEADER_DUMP), | ||
| 232 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 233 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 234 | /* SE status registers */ | 228 | /* SE status registers */ |
| 235 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0), | 229 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE0), |
| 236 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE1), | 230 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE1), |
| 237 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE2), | 231 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE2), |
| 238 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE3) | 232 | SOC15_REG_ENTRY_STR(GC, 0, mmGRBM_STATUS_SE3), |
| 233 | /* packet headers */ | ||
| 234 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 235 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 236 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 237 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 238 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 239 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 240 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 241 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_CE_HEADER_DUMP), | ||
| 242 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 243 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 244 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 245 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 246 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 247 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 248 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 249 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_PFP_HEADER_DUMP), | ||
| 250 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 251 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 252 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 253 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 254 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 255 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 256 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP), | ||
| 257 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_ME_HEADER_DUMP) | ||
| 239 | }; | 258 | }; |
| 240 | 259 | ||
| 241 | static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = { | 260 | static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = { |
| @@ -277,6 +296,14 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9[] = { · 35 unmodified lines | |||
| 277 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_LO), | 296 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_LO), |
| 278 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_HI), | 297 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_PQ_WPTR_HI), |
| 279 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_GFX_STATUS), | 298 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_HQD_GFX_STATUS), |
| 299 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 300 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 301 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 302 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 303 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 304 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 305 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP), | ||
| 306 | SOC15_REG_ENTRY_STR(GC, 0, mmCP_MEC_ME1_HEADER_DUMP) | ||
| 280 | }; | 307 | }; |
| 281 | 308 | ||
| 282 | enum ta_ras_gfx_subblock { | 309 | enum ta_ras_gfx_subblock { |
| @@ -1624,42 +1651,16 @@ static u32 gfx_v9_0_get_csb_size(struct amdgpu_device *adev) · 1341 unmodified lines | |||
| 1624 | static void gfx_v9_0_get_csb_buffer(struct amdgpu_device *adev, | 1651 | static void gfx_v9_0_get_csb_buffer(struct amdgpu_device *adev, |
| 1625 | volatile u32 *buffer) | 1652 | volatile u32 *buffer) |
| 1626 | { | 1653 | { |
| 1627 | u32 count = 0, i; | 1654 | u32 count = 0; |
| 1628 | const struct cs_section_def *sect = NULL; | ||
| 1629 | const struct cs_extent_def *ext = NULL; | ||
| 1630 | 1655 | ||
| 1631 | if (adev->gfx.rlc.cs_data == NULL) | 1656 | if (adev->gfx.rlc.cs_data == NULL) |
| 1632 | return; | 1657 | return; |
| 1633 | if (buffer == NULL) | 1658 | if (buffer == NULL) |
| 1634 | return; | 1659 | return; |
| 1635 | 1660 | ||
| 1636 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | 1661 | count = amdgpu_gfx_csb_preamble_start(buffer); |
| 1637 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_BEGIN_CLEAR_STATE); | 1662 | count = amdgpu_gfx_csb_data_parser(adev, buffer, count); |
| 1638 | 1663 | amdgpu_gfx_csb_preamble_end(buffer, count); | |
| 1639 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CONTEXT_CONTROL, 1)); | ||
| 1640 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 1641 | buffer[count++] = cpu_to_le32(0x80000000); | ||
| 1642 | |||
| 1643 | for (sect = adev->gfx.rlc.cs_data; sect->section != NULL; ++sect) { | ||
| 1644 | for (ext = sect->section; ext->extent != NULL; ++ext) { | ||
| 1645 | if (sect->id == SECT_CONTEXT) { | ||
| 1646 | buffer[count++] = | ||
| 1647 | cpu_to_le32(PACKET3(PACKET3_SET_CONTEXT_REG, ext->reg_count)); | ||
| 1648 | buffer[count++] = cpu_to_le32(ext->reg_index - | ||
| 1649 | PACKET3_SET_CONTEXT_REG_START); | ||
| 1650 | for (i = 0; i < ext->reg_count; i++) | ||
| 1651 | buffer[count++] = cpu_to_le32(ext->extent[i]); | ||
| 1652 | } else { | ||
| 1653 | return; | ||
| 1654 | } | ||
| 1655 | } | ||
| 1656 | } | ||
| 1657 | |||
| 1658 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_PREAMBLE_CNTL, 0)); | ||
| 1659 | buffer[count++] = cpu_to_le32(PACKET3_PREAMBLE_END_CLEAR_STATE); | ||
| 1660 | |||
| 1661 | buffer[count++] = cpu_to_le32(PACKET3(PACKET3_CLEAR_STATE, 0)); | ||
| 1662 | buffer[count++] = cpu_to_le32(0); | ||
| 1663 | } | 1664 | } |
| 1664 | 1665 | ||
| 1665 | static void gfx_v9_0_init_always_on_cu_mask(struct amdgpu_device *adev) | 1666 | static void gfx_v9_0_init_always_on_cu_mask(struct amdgpu_device *adev) |
| @@ -5441,16 +5442,8 @@ static void gfx_v9_0_ring_patch_ce_meta(struct amdgpu_ring *ring, · 3775 unmodified lines | |||
| 5441 | 5442 | ||
| 5442 | payload_size = sizeof(struct v9_ce_ib_state); | 5443 | payload_size = sizeof(struct v9_ce_ib_state); |
| 5443 | 5444 | ||
| 5444 | if (ring->is_mes_queue) { | 5445 | payload_offset = offsetof(struct v9_gfx_meta_data, ce_payload); |
| 5445 | payload_offset = offsetof(struct amdgpu_mes_ctx_meta_data, | 5446 | ce_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset; |
| 5446 | gfx[0].gfx_meta_data) + | ||
| 5447 | offsetof(struct v9_gfx_meta_data, ce_payload); | ||
| 5448 | ce_payload_cpu_addr = | ||
| 5449 | amdgpu_mes_ctx_get_offs_cpu_addr(ring, payload_offset); | ||
| 5450 | } else { | ||
| 5451 | payload_offset = offsetof(struct v9_gfx_meta_data, ce_payload); | ||
| 5452 | ce_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset; | ||
| 5453 | } | ||
| 5454 | 5447 | ||
| 5455 | if (offset + (payload_size >> 2) <= ring->buf_mask + 1) { | 5448 | if (offset + (payload_size >> 2) <= ring->buf_mask + 1) { |
| 5456 | memcpy((void *)&ring->ring[offset], ce_payload_cpu_addr, payload_size); | 5449 | memcpy((void *)&ring->ring[offset], ce_payload_cpu_addr, payload_size); |
| @@ -5473,16 +5466,8 @@ static void gfx_v9_0_ring_patch_de_meta(struct amdgpu_ring *ring, · 16 unmodified lines | |||
| 5473 | 5466 | ||
| 5474 | payload_size = sizeof(struct v9_de_ib_state); | 5467 | payload_size = sizeof(struct v9_de_ib_state); |
| 5475 | 5468 | ||
| 5476 | if (ring->is_mes_queue) { | 5469 | payload_offset = offsetof(struct v9_gfx_meta_data, de_payload); |
| 5477 | payload_offset = offsetof(struct amdgpu_mes_ctx_meta_data, | 5470 | de_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset; |
| 5478 | gfx[0].gfx_meta_data) + | ||
| 5479 | offsetof(struct v9_gfx_meta_data, de_payload); | ||
| 5480 | de_payload_cpu_addr = | ||
| 5481 | amdgpu_mes_ctx_get_offs_cpu_addr(ring, payload_offset); | ||
| 5482 | } else { | ||
| 5483 | payload_offset = offsetof(struct v9_gfx_meta_data, de_payload); | ||
| 5484 | de_payload_cpu_addr = adev->virt.csa_cpu_addr + payload_offset; | ||
| 5485 | } | ||
| 5486 | 5471 | ||
| 5487 | ((struct v9_de_ib_state *)de_payload_cpu_addr)->ib_completion_status = | 5472 | ((struct v9_de_ib_state *)de_payload_cpu_addr)->ib_completion_status = |
| 5488 | IB_COMPLETION_STATUS_PREEMPTED; | 5473 | IB_COMPLETION_STATUS_PREEMPTED; |
| @@ -5672,19 +5657,9 @@ static void gfx_v9_0_ring_emit_ce_meta(struct amdgpu_ring *ring, bool resume) · 183 unmodified lines | |||
| 5672 | 5657 | ||
| 5673 | cnt = (sizeof(ce_payload) >> 2) + 4 - 2; | 5658 | cnt = (sizeof(ce_payload) >> 2) + 4 - 2; |
| 5674 | 5659 | ||
| 5675 | if (ring->is_mes_queue) { | 5660 | offset = offsetof(struct v9_gfx_meta_data, ce_payload); |
| 5676 | offset = offsetof(struct amdgpu_mes_ctx_meta_data, | 5661 | ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset; |
| 5677 | gfx[0].gfx_meta_data) + | 5662 | ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset; |
| 5678 | offsetof(struct v9_gfx_meta_data, ce_payload); | ||
| 5679 | ce_payload_gpu_addr = | ||
| 5680 | amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 5681 | ce_payload_cpu_addr = | ||
| 5682 | amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 5683 | } else { | ||
| 5684 | offset = offsetof(struct v9_gfx_meta_data, ce_payload); | ||
| 5685 | ce_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset; | ||
| 5686 | ce_payload_cpu_addr = adev->virt.csa_cpu_addr + offset; | ||
| 5687 | } | ||
| 5688 | 5663 | ||
| 5689 | amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt)); | 5664 | amdgpu_ring_write(ring, PACKET3(PACKET3_WRITE_DATA, cnt)); |
| 5690 | amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(2) | | 5665 | amdgpu_ring_write(ring, (WRITE_DATA_ENGINE_SEL(2) | |
| @@ -5770,28 +5745,13 @@ static void gfx_v9_0_ring_emit_de_meta(struct amdgpu_ring *ring, bool resume, bo · 79 unmodified lines | |||
| 5770 | void *de_payload_cpu_addr; | 5745 | void *de_payload_cpu_addr; |
| 5771 | int cnt; | 5746 | int cnt; |
| 5772 | 5747 | ||
| 5773 | if (ring->is_mes_queue) { | 5748 | offset = offsetof(struct v9_gfx_meta_data, de_payload); |
| 5774 | offset = offsetof(struct amdgpu_mes_ctx_meta_data, | 5749 | de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset; |
| 5775 | gfx[0].gfx_meta_data) + | 5750 | de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset; |
| 5776 | offsetof(struct v9_gfx_meta_data, de_payload); | ||
| 5777 | de_payload_gpu_addr = | ||
| 5778 | amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 5779 | de_payload_cpu_addr = | ||
| 5780 | amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 5781 | |||
| 5782 | offset = offsetof(struct amdgpu_mes_ctx_meta_data, | ||
| 5783 | gfx[0].gds_backup) + | ||
| 5784 | offsetof(struct v9_gfx_meta_data, de_payload); | ||
| 5785 | gds_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 5786 | } else { | ||
| 5787 | offset = offsetof(struct v9_gfx_meta_data, de_payload); | ||
| 5788 | de_payload_gpu_addr = amdgpu_csa_vaddr(ring->adev) + offset; | ||
| 5789 | de_payload_cpu_addr = adev->virt.csa_cpu_addr + offset; | ||
| 5790 | 5751 | ||
| 5791 | gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) + | 5752 | gds_addr = ALIGN(amdgpu_csa_vaddr(ring->adev) + |
| 5792 | AMDGPU_CSA_SIZE - adev->gds.gds_size, | 5753 | AMDGPU_CSA_SIZE - adev->gds.gds_size, |
| 5793 | PAGE_SIZE); | 5754 | PAGE_SIZE); |
| 5794 | } | ||
| 5795 | 5755 | ||
| 5796 | if (usegds) { | 5756 | if (usegds) { |
| 5797 | de_payload.gds_backup_addrlo = lower_32_bits(gds_addr); | 5757 | de_payload.gds_backup_addrlo = lower_32_bits(gds_addr); |
| @@ -7339,9 +7299,14 @@ static void gfx_v9_ip_print(struct amdgpu_ip_block *ip_block, struct drm_printer · 1541 unmodified lines | |||
| 7339 | for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) { | 7299 | for (k = 0; k < adev->gfx.mec.num_queue_per_pipe; k++) { |
| 7340 | drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k); | 7300 | drm_printf(p, "\nmec %d, pipe %d, queue %d\n", i, j, k); |
| 7341 | for (reg = 0; reg < reg_count; reg++) { | 7301 | for (reg = 0; reg < reg_count; reg++) { |
| 7342 | drm_printf(p, "%-50s \t 0x%08x\n", | 7302 | if (i && gc_cp_reg_list_9[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP) |
| 7343 | gc_cp_reg_list_9[reg].reg_name, | 7303 | drm_printf(p, "%-50s \t 0x%08x\n", |
| 7344 | adev->gfx.ip_dump_compute_queues[index + reg]); | 7304 | "mmCP_MEC_ME2_HEADER_DUMP", |
| 7305 | adev->gfx.ip_dump_compute_queues[index + reg]); | ||
| 7306 | else | ||
| 7307 | drm_printf(p, "%-50s \t 0x%08x\n", | ||
| 7308 | gc_cp_reg_list_9[reg].reg_name, | ||
| 7309 | adev->gfx.ip_dump_compute_queues[index + reg]); | ||
| 7345 | } | 7310 | } |
| 7346 | index += reg_count; | 7311 | index += reg_count; |
| 7347 | } | 7312 | } |
| @@ -7378,9 +7343,13 @@ static void gfx_v9_ip_dump(struct amdgpu_ip_block *ip_block) · 30 unmodified lines | |||
| 7378 | soc15_grbm_select(adev, 1 + i, j, k, 0, 0); | 7343 | soc15_grbm_select(adev, 1 + i, j, k, 0, 0); |
| 7379 | 7344 | ||
| 7380 | for (reg = 0; reg < reg_count; reg++) { | 7345 | for (reg = 0; reg < reg_count; reg++) { |
| 7381 | adev->gfx.ip_dump_compute_queues[index + reg] = | 7346 | if (i && gc_cp_reg_list_9[reg].reg_offset == mmCP_MEC_ME1_HEADER_DUMP) |
| 7382 | RREG32(SOC15_REG_ENTRY_OFFSET( | 7347 | adev->gfx.ip_dump_compute_queues[index + reg] = |
| 7383 | gc_cp_reg_list_9[reg])); | 7348 | RREG32(SOC15_REG_OFFSET(GC, 0, mmCP_MEC_ME2_HEADER_DUMP)); |
| 7349 | else | ||
| 7350 | adev->gfx.ip_dump_compute_queues[index + reg] = | ||
| 7351 | RREG32(SOC15_REG_ENTRY_OFFSET( | ||
| 7352 | gc_cp_reg_list_9[reg])); | ||
| 7384 | } | 7353 | } |
| 7385 | index += reg_count; | 7354 | index += reg_count; |
| 7386 | } | 7355 | } |
| @@ -7394,8 +7363,14 @@ static void gfx_v9_ip_dump(struct amdgpu_ip_block *ip_block) · 7 unmodified lines | |||
| 7394 | 7363 | ||
| 7395 | static void gfx_v9_0_ring_emit_cleaner_shader(struct amdgpu_ring *ring) | 7364 | static void gfx_v9_0_ring_emit_cleaner_shader(struct amdgpu_ring *ring) |
| 7396 | { | 7365 | { |
| 7366 | struct amdgpu_device *adev = ring->adev; | ||
| 7367 | |||
| 7397 | /* Emit the cleaner shader */ | 7368 | /* Emit the cleaner shader */ |
| 7398 | amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER, 0)); | 7369 | if (amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 2)) |
| 7370 | amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER, 0)); | ||
| 7371 | else | ||
| 7372 | amdgpu_ring_write(ring, PACKET3(PACKET3_RUN_CLEANER_SHADER_9_0, 0)); | ||
| 7373 | |||
| 7399 | amdgpu_ring_write(ring, 0); /* RESERVED field, programmed to zero */ | 7374 | amdgpu_ring_write(ring, 0); /* RESERVED field, programmed to zero */ |
| 7400 | } | 7375 | } |
| 7401 | 7376 | ||
| @@ -1547,7 +1547,7 @@ static void gfx_v9_4_2_log_utc_edc_count(struct amdgpu_device *adev, · 1546 unmodified lines | |||
| 1547 | { | 1547 | { |
| 1548 | uint32_t bank, way, mem; | 1548 | uint32_t bank, way, mem; |
| 1549 | static const char * const vml2_way_str[] = { "BIGK", "4K" }; | 1549 | static const char * const vml2_way_str[] = { "BIGK", "4K" }; |
| 1550 | static const char * const utcl2_rounter_str[] = { "VMC", "APT" }; | 1550 | static const char * const utcl2_router_str[] = { "VMC", "APT" }; |
| 1551 | 1551 | ||
| 1552 | mem = instance % blk->num_mem_blocks; | 1552 | mem = instance % blk->num_mem_blocks; |
| 1553 | way = (instance / blk->num_mem_blocks) % blk->num_ways; | 1553 | way = (instance / blk->num_mem_blocks) % blk->num_ways; |
| @@ -1568,7 +1568,7 @@ static void gfx_v9_4_2_log_utc_edc_count(struct amdgpu_device *adev, · 14 unmodified lines | |||
| 1568 | dev_info( | 1568 | dev_info( |
| 1569 | adev->dev, | 1569 | adev->dev, |
| 1570 | "GFX SubBlock UTCL2_ROUTER_IFIF%d_GROUP0_%s, SED %d, DED %d\n", | 1570 | "GFX SubBlock UTCL2_ROUTER_IFIF%d_GROUP0_%s, SED %d, DED %d\n", |
| 1571 | bank, utcl2_rounter_str[mem], sec_cnt, ded_cnt); | 1571 | bank, utcl2_router_str[mem], sec_cnt, ded_cnt); |
| 1572 | break; | 1572 | break; |
| 1573 | case ATC_L2_CACHE_2M: | 1573 | case ATC_L2_CACHE_2M: |
| 1574 | dev_info( | 1574 | dev_info( |
| @@ -105,9 +105,6 @@ static const struct amdgpu_hwip_reg_entry gc_reg_list_9_4_3[] = { · 104 unmodified lines | |||
| 105 | SOC15_REG_ENTRY_STR(GC, 0, regRLC_SMU_SAFE_MODE), | 105 | SOC15_REG_ENTRY_STR(GC, 0, regRLC_SMU_SAFE_MODE), |
| 106 | SOC15_REG_ENTRY_STR(GC, 0, regRLC_INT_STAT), | 106 | SOC15_REG_ENTRY_STR(GC, 0, regRLC_INT_STAT), |
| 107 | SOC15_REG_ENTRY_STR(GC, 0, regRLC_GPM_GENERAL_6), | 107 | SOC15_REG_ENTRY_STR(GC, 0, regRLC_GPM_GENERAL_6), |
| 108 | /* cp header registers */ | ||
| 109 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 110 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME2_HEADER_DUMP), | ||
| 111 | /* SE status registers */ | 108 | /* SE status registers */ |
| 112 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0), | 109 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE0), |
| 113 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE1), | 110 | SOC15_REG_ENTRY_STR(GC, 0, regGRBM_STATUS_SE1), |
| @@ -154,6 +151,14 @@ static const struct amdgpu_hwip_reg_entry gc_cp_reg_list_9_4_3[] = { · 40 unmodified lines | |||
| 154 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_LO), | 151 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_LO), |
| 155 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_HI), | 152 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_PQ_WPTR_HI), |
| 156 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_GFX_STATUS), | 153 | SOC15_REG_ENTRY_STR(GC, 0, regCP_HQD_GFX_STATUS), |
| 154 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 155 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 156 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 157 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 158 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 159 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 160 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 161 | SOC15_REG_ENTRY_STR(GC, 0, regCP_MEC_ME1_HEADER_DUMP), | ||
| 157 | }; | 162 | }; |
| 158 | 163 | ||
| 159 | struct amdgpu_gfx_ras gfx_v9_4_3_ras; | 164 | struct amdgpu_gfx_ras gfx_v9_4_3_ras; |
| @@ -4558,12 +4563,21 @@ static void gfx_v9_4_3_ip_print(struct amdgpu_ip_block *ip_block, struct drm_pri · 4398 unmodified lines | |||
| 4558 | "\nxcc:%d mec:%d, pipe:%d, queue:%d\n", | 4563 | "\nxcc:%d mec:%d, pipe:%d, queue:%d\n", |
| 4559 | xcc_id, i, j, k); | 4564 | xcc_id, i, j, k); |
| 4560 | for (reg = 0; reg < reg_count; reg++) { | 4565 | for (reg = 0; reg < reg_count; reg++) { |
| 4561 | drm_printf(p, | 4566 | if (i && gc_cp_reg_list_9_4_3[reg].reg_offset == |
| 4562 | "%-50s \t 0x%08x\n", | 4567 | regCP_MEC_ME1_HEADER_DUMP) |
| 4563 | gc_cp_reg_list_9_4_3[reg].reg_name, | 4568 | drm_printf(p, |
| 4564 | adev->gfx.ip_dump_compute_queues | 4569 | "%-50s \t 0x%08x\n", |
| 4565 | [xcc_offset + inst_offset + | 4570 | "regCP_MEC_ME2_HEADER_DUMP", |
| 4566 | reg]); | 4571 | adev->gfx.ip_dump_compute_queues |
| 4572 | [xcc_offset + inst_offset + | ||
| 4573 | reg]); | ||
| 4574 | else | ||
| 4575 | drm_printf(p, | ||
| 4576 | "%-50s \t 0x%08x\n", | ||
| 4577 | gc_cp_reg_list_9_4_3[reg].reg_name, | ||
| 4578 | adev->gfx.ip_dump_compute_queues | ||
| 4579 | [xcc_offset + inst_offset + | ||
| 4580 | reg]); | ||
| 4567 | } | 4581 | } |
| 4568 | inst_offset += reg_count; | 4582 | inst_offset += reg_count; |
| 4569 | } | 4583 | } |
| @@ -4612,12 +4626,20 @@ static void gfx_v9_4_3_ip_dump(struct amdgpu_ip_block *ip_block) · 42 unmodified lines | |||
| 4612 | GET_INST(GC, xcc_id)); | 4626 | GET_INST(GC, xcc_id)); |
| 4613 | 4627 | ||
| 4614 | for (reg = 0; reg < reg_count; reg++) { | 4628 | for (reg = 0; reg < reg_count; reg++) { |
| 4615 | adev->gfx.ip_dump_compute_queues | 4629 | if (i && gc_cp_reg_list_9_4_3[reg].reg_offset == |
| 4616 | [xcc_offset + | 4630 | regCP_MEC_ME1_HEADER_DUMP) |
| 4617 | inst_offset + reg] = | 4631 | adev->gfx.ip_dump_compute_queues |
| 4618 | RREG32(SOC15_REG_ENTRY_OFFSET_INST( | 4632 | [xcc_offset + |
| 4619 | gc_cp_reg_list_9_4_3[reg], | 4633 | inst_offset + reg] = |
| 4620 | GET_INST(GC, xcc_id))); | 4634 | RREG32(SOC15_REG_OFFSET(GC, GET_INST(GC, xcc_id), |
| 4635 | regCP_MEC_ME2_HEADER_DUMP)); | ||
| 4636 | else | ||
| 4637 | adev->gfx.ip_dump_compute_queues | ||
| 4638 | [xcc_offset + | ||
| 4639 | inst_offset + reg] = | ||
| 4640 | RREG32(SOC15_REG_ENTRY_OFFSET_INST( | ||
| 4641 | gc_cp_reg_list_9_4_3[reg], | ||
| 4642 | GET_INST(GC, xcc_id))); | ||
| 4621 | } | 4643 | } |
| 4622 | inst_offset += reg_count; | 4644 | inst_offset += reg_count; |
| 4623 | } | 4645 | } |
| @@ -428,10 +428,6 @@ static void gmc_v10_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned int · 427 unmodified lines | |||
| 428 | struct amdgpu_device *adev = ring->adev; | 428 | struct amdgpu_device *adev = ring->adev; |
| 429 | uint32_t reg; | 429 | uint32_t reg; |
| 430 | 430 | ||
| 431 | /* MES fw manages IH_VMID_x_LUT updating */ | ||
| 432 | if (ring->is_mes_queue) | ||
| 433 | return; | ||
| 434 | |||
| 435 | if (ring->vm_hub == AMDGPU_GFXHUB(0)) | 431 | if (ring->vm_hub == AMDGPU_GFXHUB(0)) |
| 436 | reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid; | 432 | reg = SOC15_REG_OFFSET(OSSSYS, 0, mmIH_VMID_0_LUT) + vmid; |
| 437 | else | 433 | else |
| @@ -393,10 +393,6 @@ static void gmc_v11_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned int · 392 unmodified lines | |||
| 393 | struct amdgpu_device *adev = ring->adev; | 393 | struct amdgpu_device *adev = ring->adev; |
| 394 | uint32_t reg; | 394 | uint32_t reg; |
| 395 | 395 | ||
| 396 | /* MES fw manages IH_VMID_x_LUT updating */ | ||
| 397 | if (ring->is_mes_queue) | ||
| 398 | return; | ||
| 399 | |||
| 400 | if (ring->vm_hub == AMDGPU_GFXHUB(0)) | 396 | if (ring->vm_hub == AMDGPU_GFXHUB(0)) |
| 401 | reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid; | 397 | reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid; |
| 402 | else | 398 | else |
| @@ -832,7 +828,7 @@ static int gmc_v11_0_sw_init(struct amdgpu_ip_block *ip_block) · 429 unmodified lines | |||
| 832 | * amdgpu graphics/compute will use VMIDs 1-7 | 828 | * amdgpu graphics/compute will use VMIDs 1-7 |
| 833 | * amdkfd will use VMIDs 8-15 | 829 | * amdkfd will use VMIDs 8-15 |
| 834 | */ | 830 | */ |
| 835 | adev->vm_manager.first_kfd_vmid = 8; | 831 | adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8; |
| 836 | 832 | ||
| 837 | amdgpu_vm_manager_init(adev); | 833 | amdgpu_vm_manager_init(adev); |
| 838 | 834 | ||
| @@ -413,10 +413,6 @@ static void gmc_v12_0_emit_pasid_mapping(struct amdgpu_ring *ring, unsigned vmid · 412 unmodified lines | |||
| 413 | struct amdgpu_device *adev = ring->adev; | 413 | struct amdgpu_device *adev = ring->adev; |
| 414 | uint32_t reg; | 414 | uint32_t reg; |
| 415 | 415 | ||
| 416 | /* MES fw manages IH_VMID_x_LUT updating */ | ||
| 417 | if (ring->is_mes_queue) | ||
| 418 | return; | ||
| 419 | |||
| 420 | if (ring->vm_hub == AMDGPU_GFXHUB(0)) | 416 | if (ring->vm_hub == AMDGPU_GFXHUB(0)) |
| 421 | reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid; | 417 | reg = SOC15_REG_OFFSET(OSSSYS, 0, regIH_VMID_0_LUT) + vmid; |
| 422 | else | 418 | else |
| @@ -820,7 +816,7 @@ static int gmc_v12_0_sw_init(struct amdgpu_ip_block *ip_block) · 397 unmodified lines | |||
| 820 | * amdgpu graphics/compute will use VMIDs 1-7 | 816 | * amdgpu graphics/compute will use VMIDs 1-7 |
| 821 | * amdkfd will use VMIDs 8-15 | 817 | * amdkfd will use VMIDs 8-15 |
| 822 | */ | 818 | */ |
| 823 | adev->vm_manager.first_kfd_vmid = 8; | 819 | adev->vm_manager.first_kfd_vmid = adev->gfx.disable_kq ? 1 : 8; |
| 824 | 820 | ||
| 825 | amdgpu_vm_manager_init(adev); | 821 | amdgpu_vm_manager_init(adev); |
| 826 | 822 | ||
| @@ -249,7 +249,7 @@ static void gmc_v6_0_mc_program(struct amdgpu_device *adev) · 248 unmodified lines | |||
| 249 | 249 | ||
| 250 | /* disable VGA render */ | 250 | /* disable VGA render */ |
| 251 | tmp = RREG32(mmVGA_RENDER_CONTROL); | 251 | tmp = RREG32(mmVGA_RENDER_CONTROL); |
| 252 | tmp &= ~VGA_VSTATUS_CNTL; | 252 | tmp &= VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK; |
| 253 | WREG32(mmVGA_RENDER_CONTROL, tmp); | 253 | WREG32(mmVGA_RENDER_CONTROL, tmp); |
| 254 | } | 254 | } |
| 255 | /* Update configuration */ | 255 | /* Update configuration */ |
| @@ -627,17 +627,16 @@ static void gmc_v6_0_vm_decode_fault(struct amdgpu_device *adev, · 371 unmodified lines | |||
| 627 | "write" : "read", block, mc_client, mc_id); | 627 | "write" : "read", block, mc_client, mc_id); |
| 628 | } | 628 | } |
| 629 | 629 | ||
| 630 | /* | ||
| 631 | static const u32 mc_cg_registers[] = { | 630 | static const u32 mc_cg_registers[] = { |
| 632 | MC_HUB_MISC_HUB_CG, | 631 | mmMC_HUB_MISC_HUB_CG, |
| 633 | MC_HUB_MISC_SIP_CG, | 632 | mmMC_HUB_MISC_SIP_CG, |
| 634 | MC_HUB_MISC_VM_CG, | 633 | mmMC_HUB_MISC_VM_CG, |
| 635 | MC_XPB_CLK_GAT, | 634 | mmMC_XPB_CLK_GAT, |
| 636 | ATC_MISC_CG, | 635 | mmATC_MISC_CG, |
| 637 | MC_CITF_MISC_WR_CG, | 636 | mmMC_CITF_MISC_WR_CG, |
| 638 | MC_CITF_MISC_RD_CG, | 637 | mmMC_CITF_MISC_RD_CG, |
| 639 | MC_CITF_MISC_VM_CG, | 638 | mmMC_CITF_MISC_VM_CG, |
| 640 | VM_L2_CG, | 639 | mmVM_L2_CG, |
| 641 | }; | 640 | }; |
| 642 | 641 | ||
| 643 | static const u32 mc_cg_ls_en[] = { | 642 | static const u32 mc_cg_ls_en[] = { |
| @@ -672,7 +671,7 @@ static void gmc_v6_0_enable_mc_ls(struct amdgpu_device *adev, · 28 unmodified lines | |||
| 672 | 671 | ||
| 673 | for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) { | 672 | for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) { |
| 674 | orig = data = RREG32(mc_cg_registers[i]); | 673 | orig = data = RREG32(mc_cg_registers[i]); |
| 675 | if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_LS)) | 674 | if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_LS)) |
| 676 | data |= mc_cg_ls_en[i]; | 675 | data |= mc_cg_ls_en[i]; |
| 677 | else | 676 | else |
| 678 | data &= ~mc_cg_ls_en[i]; | 677 | data &= ~mc_cg_ls_en[i]; |
| @@ -689,7 +688,7 @@ static void gmc_v6_0_enable_mc_mgcg(struct amdgpu_device *adev, · 10 unmodified lines | |||
| 689 | 688 | ||
| 690 | for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) { | 689 | for (i = 0; i < ARRAY_SIZE(mc_cg_registers); i++) { |
| 691 | orig = data = RREG32(mc_cg_registers[i]); | 690 | orig = data = RREG32(mc_cg_registers[i]); |
| 692 | if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_MC_MGCG)) | 691 | if (enable && (adev->cg_flags & AMD_CG_SUPPORT_MC_MGCG)) |
| 693 | data |= mc_cg_en[i]; | 692 | data |= mc_cg_en[i]; |
| 694 | else | 693 | else |
| 695 | data &= ~mc_cg_en[i]; | 694 | data &= ~mc_cg_en[i]; |
| @@ -705,7 +704,7 @@ static void gmc_v6_0_enable_bif_mgls(struct amdgpu_device *adev, · 9 unmodified lines | |||
| 705 | 704 | ||
| 706 | orig = data = RREG32_PCIE(ixPCIE_CNTL2); | 705 | orig = data = RREG32_PCIE(ixPCIE_CNTL2); |
| 707 | 706 | ||
| 708 | if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_BIF_LS)) { | 707 | if (enable && (adev->cg_flags & AMD_CG_SUPPORT_BIF_LS)) { |
| 709 | data = REG_SET_FIELD(data, PCIE_CNTL2, SLV_MEM_LS_EN, 1); | 708 | data = REG_SET_FIELD(data, PCIE_CNTL2, SLV_MEM_LS_EN, 1); |
| 710 | data = REG_SET_FIELD(data, PCIE_CNTL2, MST_MEM_LS_EN, 1); | 709 | data = REG_SET_FIELD(data, PCIE_CNTL2, MST_MEM_LS_EN, 1); |
| 711 | data = REG_SET_FIELD(data, PCIE_CNTL2, REPLAY_MEM_LS_EN, 1); | 710 | data = REG_SET_FIELD(data, PCIE_CNTL2, REPLAY_MEM_LS_EN, 1); |
| @@ -728,7 +727,7 @@ static void gmc_v6_0_enable_hdp_mgcg(struct amdgpu_device *adev, · 16 unmodified lines | |||
| 728 | 727 | ||
| 729 | orig = data = RREG32(mmHDP_HOST_PATH_CNTL); | 728 | orig = data = RREG32(mmHDP_HOST_PATH_CNTL); |
| 730 | 729 | ||
| 731 | if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_MGCG)) | 730 | if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_MGCG)) |
| 732 | data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 0); | 731 | data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 0); |
| 733 | else | 732 | else |
| 734 | data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 1); | 733 | data = REG_SET_FIELD(data, HDP_HOST_PATH_CNTL, CLOCK_GATING_DIS, 1); |
| @@ -744,7 +743,7 @@ static void gmc_v6_0_enable_hdp_ls(struct amdgpu_device *adev, · 9 unmodified lines | |||
| 744 | 743 | ||
| 745 | orig = data = RREG32(mmHDP_MEM_POWER_LS); | 744 | orig = data = RREG32(mmHDP_MEM_POWER_LS); |
| 746 | 745 | ||
| 747 | if (enable && (adev->cg_flags & AMDGPU_CG_SUPPORT_HDP_LS)) | 746 | if (enable && (adev->cg_flags & AMD_CG_SUPPORT_HDP_LS)) |
| 748 | data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 1); | 747 | data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 1); |
| 749 | else | 748 | else |
| 750 | data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 0); | 749 | data = REG_SET_FIELD(data, HDP_MEM_POWER_LS, LS_ENABLE, 0); |
| @@ -752,7 +751,6 @@ static void gmc_v6_0_enable_hdp_ls(struct amdgpu_device *adev, · 1 unmodified line | |||
| 752 | if (orig != data) | 751 | if (orig != data) |
| 753 | WREG32(mmHDP_MEM_POWER_LS, data); | 752 | WREG32(mmHDP_MEM_POWER_LS, data); |
| 754 | } | 753 | } |
| 755 | */ | ||
| 756 | 754 | ||
| 757 | static int gmc_v6_0_convert_vram_type(int mc_seq_vram_type) | 755 | static int gmc_v6_0_convert_vram_type(int mc_seq_vram_type) |
| 758 | { | 756 | { |
| @@ -1098,6 +1096,20 @@ static int gmc_v6_0_process_interrupt(struct amdgpu_device *adev, · 339 unmodified lines | |||
| 1098 | static int gmc_v6_0_set_clockgating_state(struct amdgpu_ip_block *ip_block, | 1096 | static int gmc_v6_0_set_clockgating_state(struct amdgpu_ip_block *ip_block, |
| 1099 | enum amd_clockgating_state state) | 1097 | enum amd_clockgating_state state) |
| 1100 | { | 1098 | { |
| 1099 | struct amdgpu_device *adev = ip_block->adev; | ||
| 1100 | bool gate = false; | ||
| 1101 | |||
| 1102 | if (state == AMD_CG_STATE_GATE) | ||
| 1103 | gate = true; | ||
| 1104 | |||
| 1105 | if (!(adev->flags & AMD_IS_APU)) { | ||
| 1106 | gmc_v6_0_enable_mc_mgcg(adev, gate); | ||
| 1107 | gmc_v6_0_enable_mc_ls(adev, gate); | ||
| 1108 | } | ||
| 1109 | gmc_v6_0_enable_bif_mgls(adev, gate); | ||
| 1110 | gmc_v6_0_enable_hdp_mgcg(adev, gate); | ||
| 1111 | gmc_v6_0_enable_hdp_ls(adev, gate); | ||
| 1112 | |||
| 1101 | return 0; | 1113 | return 0; |
| 1102 | } | 1114 | } |
| 1103 | 1115 | ||
| @@ -1157,17 +1157,10 @@ static bool gmc_v7_0_is_idle(struct amdgpu_ip_block *ip_block) · 1156 unmodified lines | |||
| 1157 | static int gmc_v7_0_wait_for_idle(struct amdgpu_ip_block *ip_block) | 1157 | static int gmc_v7_0_wait_for_idle(struct amdgpu_ip_block *ip_block) |
| 1158 | { | 1158 | { |
| 1159 | unsigned int i; | 1159 | unsigned int i; |
| 1160 | u32 tmp; | ||
| 1161 | struct amdgpu_device *adev = ip_block->adev; | 1160 | struct amdgpu_device *adev = ip_block->adev; |
| 1162 | 1161 | ||
| 1163 | for (i = 0; i < adev->usec_timeout; i++) { | 1162 | for (i = 0; i < adev->usec_timeout; i++) { |
| 1164 | /* read MC_STATUS */ | 1163 | if (gmc_v7_0_is_idle(ip_block)) |
| 1165 | tmp = RREG32(mmSRBM_STATUS) & (SRBM_STATUS__MCB_BUSY_MASK | | ||
| 1166 | SRBM_STATUS__MCB_NON_DISPLAY_BUSY_MASK | | ||
| 1167 | SRBM_STATUS__MCC_BUSY_MASK | | ||
| 1168 | SRBM_STATUS__MCD_BUSY_MASK | | ||
| 1169 | SRBM_STATUS__VMC_BUSY_MASK); | ||
| 1170 | if (!tmp) | ||
| 1171 | return 0; | 1164 | return 0; |
| 1172 | udelay(1); | 1165 | udelay(1); |
| 1173 | } | 1166 | } |
| @@ -1213,10 +1213,7 @@ static void gmc_v9_0_get_coherence_flags(struct amdgpu_device *adev, · 1212 unmodified lines | |||
| 1213 | if (uncached) { | 1213 | if (uncached) { |
| 1214 | mtype = MTYPE_UC; | 1214 | mtype = MTYPE_UC; |
| 1215 | } else if (ext_coherent) { | 1215 | } else if (ext_coherent) { |
| 1216 | if (gc_ip_version == IP_VERSION(9, 5, 0) || adev->rev_id) | 1216 | mtype = is_local ? MTYPE_CC : MTYPE_UC; |
| 1217 | mtype = is_local ? MTYPE_CC : MTYPE_UC; | ||
| 1218 | else | ||
| 1219 | mtype = MTYPE_UC; | ||
| 1220 | } else if (adev->flags & AMD_IS_APU) { | 1217 | } else if (adev->flags & AMD_IS_APU) { |
| 1221 | mtype = is_local ? mtype_local : MTYPE_NC; | 1218 | mtype = is_local ? mtype_local : MTYPE_NC; |
| 1222 | } else { | 1219 | } else { |
| @@ -1336,7 +1333,7 @@ static void gmc_v9_0_override_vm_pte_flags(struct amdgpu_device *adev, · 113 unmodified lines | |||
| 1336 | mtype_local = MTYPE_CC; | 1333 | mtype_local = MTYPE_CC; |
| 1337 | 1334 | ||
| 1338 | *flags = AMDGPU_PTE_MTYPE_VG10(*flags, mtype_local); | 1335 | *flags = AMDGPU_PTE_MTYPE_VG10(*flags, mtype_local); |
| 1339 | } else if (adev->rev_id) { | 1336 | } else { |
| 1340 | /* MTYPE_UC case */ | 1337 | /* MTYPE_UC case */ |
| 1341 | *flags = AMDGPU_PTE_MTYPE_VG10(*flags, MTYPE_CC); | 1338 | *flags = AMDGPU_PTE_MTYPE_VG10(*flags, MTYPE_CC); |
| 1342 | } | 1339 | } |
| @@ -2411,13 +2408,6 @@ static int gmc_v9_0_hw_init(struct amdgpu_ip_block *ip_block) · 1068 unmodified lines | |||
| 2411 | adev->gmc.flush_tlb_needs_extra_type_2 = | 2408 | adev->gmc.flush_tlb_needs_extra_type_2 = |
| 2412 | amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 0) && | 2409 | amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 0) && |
| 2413 | adev->gmc.xgmi.num_physical_nodes; | 2410 | adev->gmc.xgmi.num_physical_nodes; |
| 2414 | /* | ||
| 2415 | * TODO: This workaround is badly documented and had a buggy | ||
| 2416 | * implementation. We should probably verify what we do here. | ||
| 2417 | */ | ||
| 2418 | adev->gmc.flush_tlb_needs_extra_type_0 = | ||
| 2419 | amdgpu_ip_version(adev, GC_HWIP, 0) == IP_VERSION(9, 4, 3) && | ||
| 2420 | adev->rev_id == 0; | ||
| 2421 | 2411 | ||
| 2422 | /* The sequence of these two function calls matters.*/ | 2412 | /* The sequence of these two function calls matters.*/ |
| 2423 | gmc_v9_0_init_golden_registers(adev); | 2413 | gmc_v9_0_init_golden_registers(adev); |
| @@ -36,17 +36,6 @@ · 35 unmodified lines | |||
| 36 | #define HDP_MEM_POWER_CTRL__RC_MEM_POWER_LS_EN_MASK 0x00020000L | 36 | #define HDP_MEM_POWER_CTRL__RC_MEM_POWER_LS_EN_MASK 0x00020000L |
| 37 | #define mmHDP_MEM_POWER_CTRL_BASE_IDX 0 | 37 | #define mmHDP_MEM_POWER_CTRL_BASE_IDX 0 |
| 38 | 38 | ||
| 39 | static void hdp_v4_0_flush_hdp(struct amdgpu_device *adev, | ||
| 40 | struct amdgpu_ring *ring) | ||
| 41 | { | ||
| 42 | if (!ring || !ring->funcs->emit_wreg) { | ||
| 43 | WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 44 | RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2); | ||
| 45 | } else { | ||
| 46 | amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 47 | } | ||
| 48 | } | ||
| 49 | |||
| 50 | static void hdp_v4_0_invalidate_hdp(struct amdgpu_device *adev, | 39 | static void hdp_v4_0_invalidate_hdp(struct amdgpu_device *adev, |
| 51 | struct amdgpu_ring *ring) | 40 | struct amdgpu_ring *ring) |
| 52 | { | 41 | { |
| @@ -180,7 +169,7 @@ struct amdgpu_hdp_ras hdp_v4_0_ras = { · 127 unmodified lines | |||
| 180 | }; | 169 | }; |
| 181 | 170 | ||
| 182 | const struct amdgpu_hdp_funcs hdp_v4_0_funcs = { | 171 | const struct amdgpu_hdp_funcs hdp_v4_0_funcs = { |
| 183 | .flush_hdp = hdp_v4_0_flush_hdp, | 172 | .flush_hdp = amdgpu_hdp_generic_flush, |
| 184 | .invalidate_hdp = hdp_v4_0_invalidate_hdp, | 173 | .invalidate_hdp = hdp_v4_0_invalidate_hdp, |
| 185 | .update_clock_gating = hdp_v4_0_update_clock_gating, | 174 | .update_clock_gating = hdp_v4_0_update_clock_gating, |
| 186 | .get_clock_gating_state = hdp_v4_0_get_clockgating_state, | 175 | .get_clock_gating_state = hdp_v4_0_get_clockgating_state, |
| @@ -27,17 +27,6 @@ · 26 unmodified lines | |||
| 27 | #include "hdp/hdp_5_0_0_sh_mask.h" | 27 | #include "hdp/hdp_5_0_0_sh_mask.h" |
| 28 | #include <uapi/linux/kfd_ioctl.h> | 28 | #include <uapi/linux/kfd_ioctl.h> |
| 29 | 29 | ||
| 30 | static void hdp_v5_0_flush_hdp(struct amdgpu_device *adev, | ||
| 31 | struct amdgpu_ring *ring) | ||
| 32 | { | ||
| 33 | if (!ring || !ring->funcs->emit_wreg) { | ||
| 34 | WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 35 | RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2); | ||
| 36 | } else { | ||
| 37 | amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 38 | } | ||
| 39 | } | ||
| 40 | |||
| 41 | static void hdp_v5_0_invalidate_hdp(struct amdgpu_device *adev, | 30 | static void hdp_v5_0_invalidate_hdp(struct amdgpu_device *adev, |
| 42 | struct amdgpu_ring *ring) | 31 | struct amdgpu_ring *ring) |
| 43 | { | 32 | { |
| @@ -217,7 +206,7 @@ static void hdp_v5_0_init_registers(struct amdgpu_device *adev) · 173 unmodified lines | |||
| 217 | } | 206 | } |
| 218 | 207 | ||
| 219 | const struct amdgpu_hdp_funcs hdp_v5_0_funcs = { | 208 | const struct amdgpu_hdp_funcs hdp_v5_0_funcs = { |
| 220 | .flush_hdp = hdp_v5_0_flush_hdp, | 209 | .flush_hdp = amdgpu_hdp_generic_flush, |
| 221 | .invalidate_hdp = hdp_v5_0_invalidate_hdp, | 210 | .invalidate_hdp = hdp_v5_0_invalidate_hdp, |
| 222 | .update_clock_gating = hdp_v5_0_update_clock_gating, | 211 | .update_clock_gating = hdp_v5_0_update_clock_gating, |
| 223 | .get_clock_gating_state = hdp_v5_0_get_clockgating_state, | 212 | .get_clock_gating_state = hdp_v5_0_get_clockgating_state, |
| @@ -33,7 +33,17 @@ static void hdp_v5_2_flush_hdp(struct amdgpu_device *adev, · 32 unmodified lines | |||
| 33 | if (!ring || !ring->funcs->emit_wreg) { | 33 | if (!ring || !ring->funcs->emit_wreg) { |
| 34 | WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, | 34 | WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, |
| 35 | 0); | 35 | 0); |
| 36 | RREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2); | 36 | if (amdgpu_sriov_vf(adev)) { |
| 37 | /* this is fine because SR_IOV doesn't remap the register */ | ||
| 38 | RREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2); | ||
| 39 | } else { | ||
| 40 | /* We just need to read back a register to post the write. | ||
| 41 | * Reading back the remapped register causes problems on | ||
| 42 | * some platforms so just read back the memory size register. | ||
| 43 | */ | ||
| 44 | if (adev->nbio.funcs->get_memsize) | ||
| 45 | adev->nbio.funcs->get_memsize(adev); | ||
| 46 | } | ||
| 37 | } else { | 47 | } else { |
| 38 | amdgpu_ring_emit_wreg(ring, | 48 | amdgpu_ring_emit_wreg(ring, |
| 39 | (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, | 49 | (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, |
| @@ -30,17 +30,6 @@ · 29 unmodified lines | |||
| 30 | #define regHDP_CLK_CNTL_V6_1 0xd5 | 30 | #define regHDP_CLK_CNTL_V6_1 0xd5 |
| 31 | #define regHDP_CLK_CNTL_V6_1_BASE_IDX 0 | 31 | #define regHDP_CLK_CNTL_V6_1_BASE_IDX 0 |
| 32 | 32 | ||
| 33 | static void hdp_v6_0_flush_hdp(struct amdgpu_device *adev, | ||
| 34 | struct amdgpu_ring *ring) | ||
| 35 | { | ||
| 36 | if (!ring || !ring->funcs->emit_wreg) { | ||
| 37 | WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 38 | RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2); | ||
| 39 | } else { | ||
| 40 | amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 41 | } | ||
| 42 | } | ||
| 43 | |||
| 44 | static void hdp_v6_0_update_clock_gating(struct amdgpu_device *adev, | 33 | static void hdp_v6_0_update_clock_gating(struct amdgpu_device *adev, |
| 45 | bool enable) | 34 | bool enable) |
| 46 | { | 35 | { |
| @@ -149,7 +138,7 @@ static void hdp_v6_0_get_clockgating_state(struct amdgpu_device *adev, · 102 unmodified lines | |||
| 149 | } | 138 | } |
| 150 | 139 | ||
| 151 | const struct amdgpu_hdp_funcs hdp_v6_0_funcs = { | 140 | const struct amdgpu_hdp_funcs hdp_v6_0_funcs = { |
| 152 | .flush_hdp = hdp_v6_0_flush_hdp, | 141 | .flush_hdp = amdgpu_hdp_generic_flush, |
| 153 | .update_clock_gating = hdp_v6_0_update_clock_gating, | 142 | .update_clock_gating = hdp_v6_0_update_clock_gating, |
| 154 | .get_clock_gating_state = hdp_v6_0_get_clockgating_state, | 143 | .get_clock_gating_state = hdp_v6_0_get_clockgating_state, |
| 155 | }; | 144 | }; |
| @@ -27,17 +27,6 @@ · 26 unmodified lines | |||
| 27 | #include "hdp/hdp_7_0_0_sh_mask.h" | 27 | #include "hdp/hdp_7_0_0_sh_mask.h" |
| 28 | #include <uapi/linux/kfd_ioctl.h> | 28 | #include <uapi/linux/kfd_ioctl.h> |
| 29 | 29 | ||
| 30 | static void hdp_v7_0_flush_hdp(struct amdgpu_device *adev, | ||
| 31 | struct amdgpu_ring *ring) | ||
| 32 | { | ||
| 33 | if (!ring || !ring->funcs->emit_wreg) { | ||
| 34 | WREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 35 | RREG32((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2); | ||
| 36 | } else { | ||
| 37 | amdgpu_ring_emit_wreg(ring, (adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0); | ||
| 38 | } | ||
| 39 | } | ||
| 40 | |||
| 41 | static void hdp_v7_0_update_clock_gating(struct amdgpu_device *adev, | 30 | static void hdp_v7_0_update_clock_gating(struct amdgpu_device *adev, |
| 42 | bool enable) | 31 | bool enable) |
| 43 | { | 32 | { |
| @@ -137,7 +126,7 @@ static void hdp_v7_0_get_clockgating_state(struct amdgpu_device *adev, · 93 unmodified lines | |||
| 137 | } | 126 | } |
| 138 | 127 | ||
| 139 | const struct amdgpu_hdp_funcs hdp_v7_0_funcs = { | 128 | const struct amdgpu_hdp_funcs hdp_v7_0_funcs = { |
| 140 | .flush_hdp = hdp_v7_0_flush_hdp, | 129 | .flush_hdp = amdgpu_hdp_generic_flush, |
| 141 | .update_clock_gating = hdp_v7_0_update_clock_gating, | 130 | .update_clock_gating = hdp_v7_0_update_clock_gating, |
| 142 | .get_clock_gating_state = hdp_v7_0_get_clockgating_state, | 131 | .get_clock_gating_state = hdp_v7_0_get_clockgating_state, |
| 143 | }; | 132 | }; |
| @@ -349,6 +349,7 @@ static int ih_v6_0_irq_init(struct amdgpu_device *adev) · 348 unmodified lines | |||
| 349 | if (ret) | 349 | if (ret) |
| 350 | return ret; | 350 | return ret; |
| 351 | } | 351 | } |
| 352 | ih[i]->overflow = false; | ||
| 352 | } | 353 | } |
| 353 | 354 | ||
| 354 | /* update doorbell range for ih ring 0 */ | 355 | /* update doorbell range for ih ring 0 */ |
| @@ -446,7 +447,10 @@ static u32 ih_v6_0_get_wptr(struct amdgpu_device *adev, · 91 unmodified lines | |||
| 446 | wptr = RREG32_NO_KIQ(ih_regs->ih_rb_wptr); | 447 | wptr = RREG32_NO_KIQ(ih_regs->ih_rb_wptr); |
| 447 | if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW)) | 448 | if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW)) |
| 448 | goto out; | 449 | goto out; |
| 449 | wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0); | 450 | if (!amdgpu_sriov_vf(adev)) |
| 451 | wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0); | ||
| 452 | else | ||
| 453 | ih->overflow = true; | ||
| 450 | 454 | ||
| 451 | /* When a ring buffer overflow happen start parsing interrupt | 455 | /* When a ring buffer overflow happen start parsing interrupt |
| 452 | * from the last not overwritten vector (wptr + 32). Hopefully | 456 | * from the last not overwritten vector (wptr + 32). Hopefully |
| @@ -156,7 +156,7 @@ static int jpeg_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 155 unmodified lines | |||
| 156 | 156 | ||
| 157 | for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) { | 157 | for (j = 0; j < adev->jpeg.num_jpeg_rings; ++j) { |
| 158 | ring = &adev->jpeg.inst[i].ring_dec[j]; | 158 | ring = &adev->jpeg.inst[i].ring_dec[j]; |
| 159 | ring->use_doorbell = false; | 159 | ring->use_doorbell = true; |
| 160 | ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id); | 160 | ring->vm_hub = AMDGPU_MMHUB0(adev->jpeg.inst[i].aid_id); |
| 161 | if (!amdgpu_sriov_vf(adev)) { | 161 | if (!amdgpu_sriov_vf(adev)) { |
| 162 | ring->doorbell_index = | 162 | ring->doorbell_index = |
| @@ -264,7 +264,7 @@ static int jpeg_v5_0_1_hw_init(struct amdgpu_ip_block *ip_block) · 101 unmodified lines | |||
| 264 | ring = &adev->jpeg.inst[i].ring_dec[j]; | 264 | ring = &adev->jpeg.inst[i].ring_dec[j]; |
| 265 | if (ring->use_doorbell) | 265 | if (ring->use_doorbell) |
| 266 | WREG32_SOC15_OFFSET(VCN, GET_INST(VCN, i), regVCN_JPEG_DB_CTRL, | 266 | WREG32_SOC15_OFFSET(VCN, GET_INST(VCN, i), regVCN_JPEG_DB_CTRL, |
| 267 | (ring->pipe ? (ring->pipe - 0x15) : 0), | 267 | ring->pipe, |
| 268 | ring->doorbell_index << | 268 | ring->doorbell_index << |
| 269 | VCN_JPEG_DB_CTRL__OFFSET__SHIFT | | 269 | VCN_JPEG_DB_CTRL__OFFSET__SHIFT | |
| 270 | VCN_JPEG_DB_CTRL__EN_MASK); | 270 | VCN_JPEG_DB_CTRL__EN_MASK); |
| @@ -0,0 +1,355 @@ | |||
| 1 | // SPDX-License-Identifier: MIT | ||
| 2 | /* | ||
| 3 | * Copyright 2024 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | #include "amdgpu.h" | ||
| 25 | #include "amdgpu_gfx.h" | ||
| 26 | #include "mes_userqueue.h" | ||
| 27 | #include "amdgpu_userq_fence.h" | ||
| 28 | |||
| 29 | #define AMDGPU_USERQ_PROC_CTX_SZ PAGE_SIZE | ||
| 30 | #define AMDGPU_USERQ_GANG_CTX_SZ PAGE_SIZE | ||
| 31 | |||
| 32 | static int | ||
| 33 | mes_userq_map_gtt_bo_to_gart(struct amdgpu_bo *bo) | ||
| 34 | { | ||
| 35 | int ret; | ||
| 36 | |||
| 37 | ret = amdgpu_bo_reserve(bo, true); | ||
| 38 | if (ret) { | ||
| 39 | DRM_ERROR("Failed to reserve bo. ret %d\n", ret); | ||
| 40 | goto err_reserve_bo_failed; | ||
| 41 | } | ||
| 42 | |||
| 43 | ret = amdgpu_ttm_alloc_gart(&bo->tbo); | ||
| 44 | if (ret) { | ||
| 45 | DRM_ERROR("Failed to bind bo to GART. ret %d\n", ret); | ||
| 46 | goto err_map_bo_gart_failed; | ||
| 47 | } | ||
| 48 | |||
| 49 | amdgpu_bo_unreserve(bo); | ||
| 50 | bo = amdgpu_bo_ref(bo); | ||
| 51 | |||
| 52 | return 0; | ||
| 53 | |||
| 54 | err_map_bo_gart_failed: | ||
| 55 | amdgpu_bo_unreserve(bo); | ||
| 56 | err_reserve_bo_failed: | ||
| 57 | return ret; | ||
| 58 | } | ||
| 59 | |||
| 60 | static int | ||
| 61 | mes_userq_create_wptr_mapping(struct amdgpu_userq_mgr *uq_mgr, | ||
| 62 | struct amdgpu_usermode_queue *queue, | ||
| 63 | uint64_t wptr) | ||
| 64 | { | ||
| 65 | struct amdgpu_bo_va_mapping *wptr_mapping; | ||
| 66 | struct amdgpu_vm *wptr_vm; | ||
| 67 | struct amdgpu_userq_obj *wptr_obj = &queue->wptr_obj; | ||
| 68 | int ret; | ||
| 69 | |||
| 70 | wptr_vm = queue->vm; | ||
| 71 | ret = amdgpu_bo_reserve(wptr_vm->root.bo, false); | ||
| 72 | if (ret) | ||
| 73 | return ret; | ||
| 74 | |||
| 75 | wptr &= AMDGPU_GMC_HOLE_MASK; | ||
| 76 | wptr_mapping = amdgpu_vm_bo_lookup_mapping(wptr_vm, wptr >> PAGE_SHIFT); | ||
| 77 | amdgpu_bo_unreserve(wptr_vm->root.bo); | ||
| 78 | if (!wptr_mapping) { | ||
| 79 | DRM_ERROR("Failed to lookup wptr bo\n"); | ||
| 80 | return -EINVAL; | ||
| 81 | } | ||
| 82 | |||
| 83 | wptr_obj->obj = wptr_mapping->bo_va->base.bo; | ||
| 84 | if (wptr_obj->obj->tbo.base.size > PAGE_SIZE) { | ||
| 85 | DRM_ERROR("Requested GART mapping for wptr bo larger than one page\n"); | ||
| 86 | return -EINVAL; | ||
| 87 | } | ||
| 88 | |||
| 89 | ret = mes_userq_map_gtt_bo_to_gart(wptr_obj->obj); | ||
| 90 | if (ret) { | ||
| 91 | DRM_ERROR("Failed to map wptr bo to GART\n"); | ||
| 92 | return ret; | ||
| 93 | } | ||
| 94 | |||
| 95 | queue->wptr_obj.gpu_addr = amdgpu_bo_gpu_offset_no_check(wptr_obj->obj); | ||
| 96 | return 0; | ||
| 97 | } | ||
| 98 | |||
| 99 | static int convert_to_mes_priority(int priority) | ||
| 100 | { | ||
| 101 | switch (priority) { | ||
| 102 | case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_NORMAL_LOW: | ||
| 103 | default: | ||
| 104 | return AMDGPU_MES_PRIORITY_LEVEL_NORMAL; | ||
| 105 | case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_LOW: | ||
| 106 | return AMDGPU_MES_PRIORITY_LEVEL_LOW; | ||
| 107 | case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_NORMAL_HIGH: | ||
| 108 | return AMDGPU_MES_PRIORITY_LEVEL_MEDIUM; | ||
| 109 | case AMDGPU_USERQ_CREATE_FLAGS_QUEUE_PRIORITY_HIGH: | ||
| 110 | return AMDGPU_MES_PRIORITY_LEVEL_HIGH; | ||
| 111 | } | ||
| 112 | } | ||
| 113 | |||
| 114 | static int mes_userq_map(struct amdgpu_userq_mgr *uq_mgr, | ||
| 115 | struct amdgpu_usermode_queue *queue) | ||
| 116 | { | ||
| 117 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 118 | struct amdgpu_userq_obj *ctx = &queue->fw_obj; | ||
| 119 | struct amdgpu_mqd_prop *userq_props = queue->userq_prop; | ||
| 120 | struct mes_add_queue_input queue_input; | ||
| 121 | int r; | ||
| 122 | |||
| 123 | memset(&queue_input, 0x0, sizeof(struct mes_add_queue_input)); | ||
| 124 | |||
| 125 | queue_input.process_va_start = 0; | ||
| 126 | queue_input.process_va_end = adev->vm_manager.max_pfn - 1; | ||
| 127 | |||
| 128 | /* set process quantum to 10 ms and gang quantum to 1 ms as default */ | ||
| 129 | queue_input.process_quantum = 100000; | ||
| 130 | queue_input.gang_quantum = 10000; | ||
| 131 | queue_input.paging = false; | ||
| 132 | |||
| 133 | queue_input.process_context_addr = ctx->gpu_addr; | ||
| 134 | queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ; | ||
| 135 | queue_input.inprocess_gang_priority = AMDGPU_MES_PRIORITY_LEVEL_NORMAL; | ||
| 136 | queue_input.gang_global_priority_level = convert_to_mes_priority(queue->priority); | ||
| 137 | |||
| 138 | queue_input.process_id = queue->vm->pasid; | ||
| 139 | queue_input.queue_type = queue->queue_type; | ||
| 140 | queue_input.mqd_addr = queue->mqd.gpu_addr; | ||
| 141 | queue_input.wptr_addr = userq_props->wptr_gpu_addr; | ||
| 142 | queue_input.queue_size = userq_props->queue_size >> 2; | ||
| 143 | queue_input.doorbell_offset = userq_props->doorbell_index; | ||
| 144 | queue_input.page_table_base_addr = amdgpu_gmc_pd_addr(queue->vm->root.bo); | ||
| 145 | queue_input.wptr_mc_addr = queue->wptr_obj.gpu_addr; | ||
| 146 | |||
| 147 | amdgpu_mes_lock(&adev->mes); | ||
| 148 | r = adev->mes.funcs->add_hw_queue(&adev->mes, &queue_input); | ||
| 149 | amdgpu_mes_unlock(&adev->mes); | ||
| 150 | if (r) { | ||
| 151 | DRM_ERROR("Failed to map queue in HW, err (%d)\n", r); | ||
| 152 | return r; | ||
| 153 | } | ||
| 154 | |||
| 155 | DRM_DEBUG_DRIVER("Queue (doorbell:%d) mapped successfully\n", userq_props->doorbell_index); | ||
| 156 | return 0; | ||
| 157 | } | ||
| 158 | |||
| 159 | static int mes_userq_unmap(struct amdgpu_userq_mgr *uq_mgr, | ||
| 160 | struct amdgpu_usermode_queue *queue) | ||
| 161 | { | ||
| 162 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 163 | struct mes_remove_queue_input queue_input; | ||
| 164 | struct amdgpu_userq_obj *ctx = &queue->fw_obj; | ||
| 165 | int r; | ||
| 166 | |||
| 167 | memset(&queue_input, 0x0, sizeof(struct mes_remove_queue_input)); | ||
| 168 | queue_input.doorbell_offset = queue->doorbell_index; | ||
| 169 | queue_input.gang_context_addr = ctx->gpu_addr + AMDGPU_USERQ_PROC_CTX_SZ; | ||
| 170 | |||
| 171 | amdgpu_mes_lock(&adev->mes); | ||
| 172 | r = adev->mes.funcs->remove_hw_queue(&adev->mes, &queue_input); | ||
| 173 | amdgpu_mes_unlock(&adev->mes); | ||
| 174 | if (r) | ||
| 175 | DRM_ERROR("Failed to unmap queue in HW, err (%d)\n", r); | ||
| 176 | return r; | ||
| 177 | } | ||
| 178 | |||
| 179 | static int mes_userq_create_ctx_space(struct amdgpu_userq_mgr *uq_mgr, | ||
| 180 | struct amdgpu_usermode_queue *queue, | ||
| 181 | struct drm_amdgpu_userq_in *mqd_user) | ||
| 182 | { | ||
| 183 | struct amdgpu_userq_obj *ctx = &queue->fw_obj; | ||
| 184 | int r, size; | ||
| 185 | |||
| 186 | /* | ||
| 187 | * The FW expects at least one page space allocated for | ||
| 188 | * process ctx and gang ctx each. Create an object | ||
| 189 | * for the same. | ||
| 190 | */ | ||
| 191 | size = AMDGPU_USERQ_PROC_CTX_SZ + AMDGPU_USERQ_GANG_CTX_SZ; | ||
| 192 | r = amdgpu_userq_create_object(uq_mgr, ctx, size); | ||
| 193 | if (r) { | ||
| 194 | DRM_ERROR("Failed to allocate ctx space bo for userqueue, err:%d\n", r); | ||
| 195 | return r; | ||
| 196 | } | ||
| 197 | |||
| 198 | return 0; | ||
| 199 | } | ||
| 200 | |||
| 201 | static int mes_userq_mqd_create(struct amdgpu_userq_mgr *uq_mgr, | ||
| 202 | struct drm_amdgpu_userq_in *args_in, | ||
| 203 | struct amdgpu_usermode_queue *queue) | ||
| 204 | { | ||
| 205 | struct amdgpu_device *adev = uq_mgr->adev; | ||
| 206 | struct amdgpu_mqd *mqd_hw_default = &adev->mqds[queue->queue_type]; | ||
| 207 | struct drm_amdgpu_userq_in *mqd_user = args_in; | ||
| 208 | struct amdgpu_mqd_prop *userq_props; | ||
| 209 | int r; | ||
| 210 | |||
| 211 | /* Structure to initialize MQD for userqueue using generic MQD init function */ | ||
| 212 | userq_props = kzalloc(sizeof(struct amdgpu_mqd_prop), GFP_KERNEL); | ||
| 213 | if (!userq_props) { | ||
| 214 | DRM_ERROR("Failed to allocate memory for userq_props\n"); | ||
| 215 | return -ENOMEM; | ||
| 216 | } | ||
| 217 | |||
| 218 | if (!mqd_user->wptr_va || !mqd_user->rptr_va || | ||
| 219 | !mqd_user->queue_va || mqd_user->queue_size == 0) { | ||
| 220 | DRM_ERROR("Invalid MQD parameters for userqueue\n"); | ||
| 221 | r = -EINVAL; | ||
| 222 | goto free_props; | ||
| 223 | } | ||
| 224 | |||
| 225 | r = amdgpu_userq_create_object(uq_mgr, &queue->mqd, mqd_hw_default->mqd_size); | ||
| 226 | if (r) { | ||
| 227 | DRM_ERROR("Failed to create MQD object for userqueue\n"); | ||
| 228 | goto free_props; | ||
| 229 | } | ||
| 230 | |||
| 231 | /* Initialize the MQD BO with user given values */ | ||
| 232 | userq_props->wptr_gpu_addr = mqd_user->wptr_va; | ||
| 233 | userq_props->rptr_gpu_addr = mqd_user->rptr_va; | ||
| 234 | userq_props->queue_size = mqd_user->queue_size; | ||
| 235 | userq_props->hqd_base_gpu_addr = mqd_user->queue_va; | ||
| 236 | userq_props->mqd_gpu_addr = queue->mqd.gpu_addr; | ||
| 237 | userq_props->use_doorbell = true; | ||
| 238 | userq_props->doorbell_index = queue->doorbell_index; | ||
| 239 | userq_props->fence_address = queue->fence_drv->gpu_addr; | ||
| 240 | |||
| 241 | if (queue->queue_type == AMDGPU_HW_IP_COMPUTE) { | ||
| 242 | struct drm_amdgpu_userq_mqd_compute_gfx11 *compute_mqd; | ||
| 243 | |||
| 244 | if (mqd_user->mqd_size != sizeof(*compute_mqd)) { | ||
| 245 | DRM_ERROR("Invalid compute IP MQD size\n"); | ||
| 246 | r = -EINVAL; | ||
| 247 | goto free_mqd; | ||
| 248 | } | ||
| 249 | |||
| 250 | compute_mqd = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size); | ||
| 251 | if (IS_ERR(compute_mqd)) { | ||
| 252 | DRM_ERROR("Failed to read user MQD\n"); | ||
| 253 | r = -ENOMEM; | ||
| 254 | goto free_mqd; | ||
| 255 | } | ||
| 256 | |||
| 257 | userq_props->eop_gpu_addr = compute_mqd->eop_va; | ||
| 258 | userq_props->hqd_pipe_priority = AMDGPU_GFX_PIPE_PRIO_NORMAL; | ||
| 259 | userq_props->hqd_queue_priority = AMDGPU_GFX_QUEUE_PRIORITY_MINIMUM; | ||
| 260 | userq_props->hqd_active = false; | ||
| 261 | userq_props->tmz_queue = | ||
| 262 | mqd_user->flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE; | ||
| 263 | kfree(compute_mqd); | ||
| 264 | } else if (queue->queue_type == AMDGPU_HW_IP_GFX) { | ||
| 265 | struct drm_amdgpu_userq_mqd_gfx11 *mqd_gfx_v11; | ||
| 266 | |||
| 267 | if (mqd_user->mqd_size != sizeof(*mqd_gfx_v11) || !mqd_user->mqd) { | ||
| 268 | DRM_ERROR("Invalid GFX MQD\n"); | ||
| 269 | r = -EINVAL; | ||
| 270 | goto free_mqd; | ||
| 271 | } | ||
| 272 | |||
| 273 | mqd_gfx_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size); | ||
| 274 | if (IS_ERR(mqd_gfx_v11)) { | ||
| 275 | DRM_ERROR("Failed to read user MQD\n"); | ||
| 276 | r = -ENOMEM; | ||
| 277 | goto free_mqd; | ||
| 278 | } | ||
| 279 | |||
| 280 | userq_props->shadow_addr = mqd_gfx_v11->shadow_va; | ||
| 281 | userq_props->csa_addr = mqd_gfx_v11->csa_va; | ||
| 282 | userq_props->tmz_queue = | ||
| 283 | mqd_user->flags & AMDGPU_USERQ_CREATE_FLAGS_QUEUE_SECURE; | ||
| 284 | kfree(mqd_gfx_v11); | ||
| 285 | } else if (queue->queue_type == AMDGPU_HW_IP_DMA) { | ||
| 286 | struct drm_amdgpu_userq_mqd_sdma_gfx11 *mqd_sdma_v11; | ||
| 287 | |||
| 288 | if (mqd_user->mqd_size != sizeof(*mqd_sdma_v11) || !mqd_user->mqd) { | ||
| 289 | DRM_ERROR("Invalid SDMA MQD\n"); | ||
| 290 | r = -EINVAL; | ||
| 291 | goto free_mqd; | ||
| 292 | } | ||
| 293 | |||
| 294 | mqd_sdma_v11 = memdup_user(u64_to_user_ptr(mqd_user->mqd), mqd_user->mqd_size); | ||
| 295 | if (IS_ERR(mqd_sdma_v11)) { | ||
| 296 | DRM_ERROR("Failed to read sdma user MQD\n"); | ||
| 297 | r = -ENOMEM; | ||
| 298 | goto free_mqd; | ||
| 299 | } | ||
| 300 | |||
| 301 | userq_props->csa_addr = mqd_sdma_v11->csa_va; | ||
| 302 | kfree(mqd_sdma_v11); | ||
| 303 | } | ||
| 304 | |||
| 305 | queue->userq_prop = userq_props; | ||
| 306 | |||
| 307 | r = mqd_hw_default->init_mqd(adev, (void *)queue->mqd.cpu_ptr, userq_props); | ||
| 308 | if (r) { | ||
| 309 | DRM_ERROR("Failed to initialize MQD for userqueue\n"); | ||
| 310 | goto free_mqd; | ||
| 311 | } | ||
| 312 | |||
| 313 | /* Create BO for FW operations */ | ||
| 314 | r = mes_userq_create_ctx_space(uq_mgr, queue, mqd_user); | ||
| 315 | if (r) { | ||
| 316 | DRM_ERROR("Failed to allocate BO for userqueue (%d)", r); | ||
| 317 | goto free_mqd; | ||
| 318 | } | ||
| 319 | |||
| 320 | /* FW expects WPTR BOs to be mapped into GART */ | ||
| 321 | r = mes_userq_create_wptr_mapping(uq_mgr, queue, userq_props->wptr_gpu_addr); | ||
| 322 | if (r) { | ||
| 323 | DRM_ERROR("Failed to create WPTR mapping\n"); | ||
| 324 | goto free_ctx; | ||
| 325 | } | ||
| 326 | |||
| 327 | return 0; | ||
| 328 | |||
| 329 | free_ctx: | ||
| 330 | amdgpu_userq_destroy_object(uq_mgr, &queue->fw_obj); | ||
| 331 | |||
| 332 | free_mqd: | ||
| 333 | amdgpu_userq_destroy_object(uq_mgr, &queue->mqd); | ||
| 334 | |||
| 335 | free_props: | ||
| 336 | kfree(userq_props); | ||
| 337 | |||
| 338 | return r; | ||
| 339 | } | ||
| 340 | |||
| 341 | static void | ||
| 342 | mes_userq_mqd_destroy(struct amdgpu_userq_mgr *uq_mgr, | ||
| 343 | struct amdgpu_usermode_queue *queue) | ||
| 344 | { | ||
| 345 | amdgpu_userq_destroy_object(uq_mgr, &queue->fw_obj); | ||
| 346 | kfree(queue->userq_prop); | ||
| 347 | amdgpu_userq_destroy_object(uq_mgr, &queue->mqd); | ||
| 348 | } | ||
| 349 | |||
| 350 | const struct amdgpu_userq_funcs userq_mes_funcs = { | ||
| 351 | .mqd_create = mes_userq_mqd_create, | ||
| 352 | .mqd_destroy = mes_userq_mqd_destroy, | ||
| 353 | .unmap = mes_userq_unmap, | ||
| 354 | .map = mes_userq_map, | ||
| 355 | }; | ||
| @@ -0,0 +1,30 @@ | |||
| 1 | /* SPDX-License-Identifier: MIT */ | ||
| 2 | /* | ||
| 3 | * Copyright 2024 Advanced Micro Devices, Inc. | ||
| 4 | * | ||
| 5 | * Permission is hereby granted, free of charge, to any person obtaining a | ||
| 6 | * copy of this software and associated documentation files (the "Software"), | ||
| 7 | * to deal in the Software without restriction, including without limitation | ||
| 8 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, | ||
| 9 | * and/or sell copies of the Software, and to permit persons to whom the | ||
| 10 | * Software is furnished to do so, subject to the following conditions: | ||
| 11 | * | ||
| 12 | * The above copyright notice and this permission notice shall be included in | ||
| 13 | * all copies or substantial portions of the Software. | ||
| 14 | * | ||
| 15 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| 16 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| 17 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL | ||
| 18 | * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR | ||
| 19 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, | ||
| 20 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
| 21 | * OTHER DEALINGS IN THE SOFTWARE. | ||
| 22 | * | ||
| 23 | */ | ||
| 24 | |||
| 25 | #ifndef MES_USERQ_H | ||
| 26 | #define MES_USERQ_H | ||
| 27 | #include "amdgpu_userq.h" | ||
| 28 | |||
| 29 | extern const struct amdgpu_userq_funcs userq_mes_funcs; | ||
| 30 | #endif | ||
| @@ -287,6 +287,23 @@ static int convert_to_mes_queue_type(int queue_type) · 286 unmodified lines | |||
| 287 | return -1; | 287 | return -1; |
| 288 | } | 288 | } |
| 289 | 289 | ||
| 290 | static int convert_to_mes_priority_level(int priority_level) | ||
| 291 | { | ||
| 292 | switch (priority_level) { | ||
| 293 | case AMDGPU_MES_PRIORITY_LEVEL_LOW: | ||
| 294 | return AMD_PRIORITY_LEVEL_LOW; | ||
| 295 | case AMDGPU_MES_PRIORITY_LEVEL_NORMAL: | ||
| 296 | default: | ||
| 297 | return AMD_PRIORITY_LEVEL_NORMAL; | ||
| 298 | case AMDGPU_MES_PRIORITY_LEVEL_MEDIUM: | ||
| 299 | return AMD_PRIORITY_LEVEL_MEDIUM; | ||
| 300 | case AMDGPU_MES_PRIORITY_LEVEL_HIGH: | ||
| 301 | return AMD_PRIORITY_LEVEL_HIGH; | ||
| 302 | case AMDGPU_MES_PRIORITY_LEVEL_REALTIME: | ||
| 303 | return AMD_PRIORITY_LEVEL_REALTIME; | ||
| 304 | } | ||
| 305 | } | ||
| 306 | |||
| 290 | static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes, | 307 | static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes, |
| 291 | struct mes_add_queue_input *input) | 308 | struct mes_add_queue_input *input) |
| 292 | { | 309 | { |
| @@ -310,9 +327,9 @@ static int mes_v11_0_add_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines | |||
| 310 | mes_add_queue_pkt.gang_quantum = input->gang_quantum; | 327 | mes_add_queue_pkt.gang_quantum = input->gang_quantum; |
| 311 | mes_add_queue_pkt.gang_context_addr = input->gang_context_addr; | 328 | mes_add_queue_pkt.gang_context_addr = input->gang_context_addr; |
| 312 | mes_add_queue_pkt.inprocess_gang_priority = | 329 | mes_add_queue_pkt.inprocess_gang_priority = |
| 313 | input->inprocess_gang_priority; | 330 | convert_to_mes_priority_level(input->inprocess_gang_priority); |
| 314 | mes_add_queue_pkt.gang_global_priority_level = | 331 | mes_add_queue_pkt.gang_global_priority_level = |
| 315 | input->gang_global_priority_level; | 332 | convert_to_mes_priority_level(input->gang_global_priority_level); |
| 316 | mes_add_queue_pkt.doorbell_offset = input->doorbell_offset; | 333 | mes_add_queue_pkt.doorbell_offset = input->doorbell_offset; |
| 317 | mes_add_queue_pkt.mqd_addr = input->mqd_addr; | 334 | mes_add_queue_pkt.mqd_addr = input->mqd_addr; |
| 318 | 335 | ||
| @@ -458,31 +475,6 @@ static int mes_v11_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 139 unmodified lines | |||
| 458 | return r; | 475 | return r; |
| 459 | } | 476 | } |
| 460 | 477 | ||
| 461 | static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes, | ||
| 462 | struct mes_reset_queue_input *input) | ||
| 463 | { | ||
| 464 | if (input->use_mmio) | ||
| 465 | return mes_v11_0_reset_queue_mmio(mes, input->queue_type, | ||
| 466 | input->me_id, input->pipe_id, | ||
| 467 | input->queue_id, input->vmid); | ||
| 468 | |||
| 469 | union MESAPI__RESET mes_reset_queue_pkt; | ||
| 470 | |||
| 471 | memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt)); | ||
| 472 | |||
| 473 | mes_reset_queue_pkt.header.type = MES_API_TYPE_SCHEDULER; | ||
| 474 | mes_reset_queue_pkt.header.opcode = MES_SCH_API_RESET; | ||
| 475 | mes_reset_queue_pkt.header.dwsize = API_FRAME_SIZE_IN_DWORDS; | ||
| 476 | |||
| 477 | mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset; | ||
| 478 | mes_reset_queue_pkt.gang_context_addr = input->gang_context_addr; | ||
| 479 | /*mes_reset_queue_pkt.reset_queue_only = 1;*/ | ||
| 480 | |||
| 481 | return mes_v11_0_submit_pkt_and_poll_completion(mes, | ||
| 482 | &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt), | ||
| 483 | offsetof(union MESAPI__REMOVE_QUEUE, api_status)); | ||
| 484 | } | ||
| 485 | |||
| 486 | static int mes_v11_0_map_legacy_queue(struct amdgpu_mes *mes, | 478 | static int mes_v11_0_map_legacy_queue(struct amdgpu_mes *mes, |
| 487 | struct mes_map_legacy_queue_input *input) | 479 | struct mes_map_legacy_queue_input *input) |
| 488 | { | 480 | { |
| @@ -649,7 +641,7 @@ static int mes_v11_0_misc_op(struct amdgpu_mes *mes, · 160 unmodified lines | |||
| 649 | break; | 641 | break; |
| 650 | case MES_MISC_OP_CHANGE_CONFIG: | 642 | case MES_MISC_OP_CHANGE_CONFIG: |
| 651 | if ((mes->adev->mes.sched_version & AMDGPU_MES_VERSION_MASK) < 0x63) { | 643 | if ((mes->adev->mes.sched_version & AMDGPU_MES_VERSION_MASK) < 0x63) { |
| 652 | dev_err(mes->adev->dev, "MES FW versoin must be larger than 0x63 to support limit single process feature.\n"); | 644 | dev_err(mes->adev->dev, "MES FW version must be larger than 0x63 to support limit single process feature.\n"); |
| 653 | return -EINVAL; | 645 | return -EINVAL; |
| 654 | } | 646 | } |
| 655 | misc_pkt.opcode = MESAPI_MISC__CHANGE_CONFIG; | 647 | misc_pkt.opcode = MESAPI_MISC__CHANGE_CONFIG; |
| @@ -694,7 +686,8 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes) · 38 unmodified lines | |||
| 694 | mes->compute_hqd_mask[i]; | 686 | mes->compute_hqd_mask[i]; |
| 695 | 687 | ||
| 696 | for (i = 0; i < MAX_GFX_PIPES; i++) | 688 | for (i = 0; i < MAX_GFX_PIPES; i++) |
| 697 | mes_set_hw_res_pkt.gfx_hqd_mask[i] = mes->gfx_hqd_mask[i]; | 689 | mes_set_hw_res_pkt.gfx_hqd_mask[i] = |
| 690 | mes->gfx_hqd_mask[i]; | ||
| 698 | 691 | ||
| 699 | for (i = 0; i < MAX_SDMA_PIPES; i++) | 692 | for (i = 0; i < MAX_SDMA_PIPES; i++) |
| 700 | mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i]; | 693 | mes_set_hw_res_pkt.sdma_hqd_mask[i] = mes->sdma_hqd_mask[i]; |
| @@ -723,7 +716,7 @@ static int mes_v11_0_set_hw_resources(struct amdgpu_mes *mes) · 22 unmodified lines | |||
| 723 | mes->event_log_gpu_addr; | 716 | mes->event_log_gpu_addr; |
| 724 | } | 717 | } |
| 725 | 718 | ||
| 726 | if (enforce_isolation) | 719 | if (adev->enforce_isolation[0] == AMDGPU_ENFORCE_ISOLATION_ENABLE) |
| 727 | mes_set_hw_res_pkt.limit_single_process = 1; | 720 | mes_set_hw_res_pkt.limit_single_process = 1; |
| 728 | 721 | ||
| 729 | return mes_v11_0_submit_pkt_and_poll_completion(mes, | 722 | return mes_v11_0_submit_pkt_and_poll_completion(mes, |
| @@ -753,8 +746,8 @@ static int mes_v11_0_set_hw_resources_1(struct amdgpu_mes *mes) · 23 unmodified lines | |||
| 753 | offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status)); | 746 | offsetof(union MESAPI_SET_HW_RESOURCES_1, api_status)); |
| 754 | } | 747 | } |
| 755 | 748 | ||
| 756 | static int mes_v11_0_reset_legacy_queue(struct amdgpu_mes *mes, | 749 | static int mes_v11_0_reset_hw_queue(struct amdgpu_mes *mes, |
| 757 | struct mes_reset_legacy_queue_input *input) | 750 | struct mes_reset_queue_input *input) |
| 758 | { | 751 | { |
| 759 | union MESAPI__RESET mes_reset_queue_pkt; | 752 | union MESAPI__RESET mes_reset_queue_pkt; |
| 760 | 753 | ||
| @@ -772,7 +765,7 @@ static int mes_v11_0_reset_legacy_queue(struct amdgpu_mes *mes, · 11 unmodified lines | |||
| 772 | mes_reset_queue_pkt.queue_type = | 765 | mes_reset_queue_pkt.queue_type = |
| 773 | convert_to_mes_queue_type(input->queue_type); | 766 | convert_to_mes_queue_type(input->queue_type); |
| 774 | 767 | ||
| 775 | if (mes_reset_queue_pkt.queue_type == MES_QUEUE_TYPE_GFX) { | 768 | if (input->legacy_gfx) { |
| 776 | mes_reset_queue_pkt.reset_legacy_gfx = 1; | 769 | mes_reset_queue_pkt.reset_legacy_gfx = 1; |
| 777 | mes_reset_queue_pkt.pipe_id_lp = input->pipe_id; | 770 | mes_reset_queue_pkt.pipe_id_lp = input->pipe_id; |
| 778 | mes_reset_queue_pkt.queue_id_lp = input->queue_id; | 771 | mes_reset_queue_pkt.queue_id_lp = input->queue_id; |
| @@ -798,7 +791,6 @@ static const struct amdgpu_mes_funcs mes_v11_0_funcs = { · 19 unmodified lines | |||
| 798 | .suspend_gang = mes_v11_0_suspend_gang, | 791 | .suspend_gang = mes_v11_0_suspend_gang, |
| 799 | .resume_gang = mes_v11_0_resume_gang, | 792 | .resume_gang = mes_v11_0_resume_gang, |
| 800 | .misc_op = mes_v11_0_misc_op, | 793 | .misc_op = mes_v11_0_misc_op, |
| 801 | .reset_legacy_queue = mes_v11_0_reset_legacy_queue, | ||
| 802 | .reset_hw_queue = mes_v11_0_reset_hw_queue, | 794 | .reset_hw_queue = mes_v11_0_reset_hw_queue, |
| 803 | }; | 795 | }; |
| 804 | 796 | ||
| @@ -1701,22 +1693,10 @@ static int mes_v11_0_early_init(struct amdgpu_ip_block *ip_block) · 896 unmodified lines | |||
| 1701 | return 0; | 1693 | return 0; |
| 1702 | } | 1694 | } |
| 1703 | 1695 | ||
| 1704 | static int mes_v11_0_late_init(struct amdgpu_ip_block *ip_block) | ||
| 1705 | { | ||
| 1706 | struct amdgpu_device *adev = ip_block->adev; | ||
| 1707 | |||
| 1708 | /* it's only intended for use in mes_self_test case, not for s0ix and reset */ | ||
| 1709 | if (!amdgpu_in_reset(adev) && !adev->in_s0ix && !adev->in_suspend && | ||
| 1710 | (amdgpu_ip_version(adev, GC_HWIP, 0) != IP_VERSION(11, 0, 3))) | ||
| 1711 | amdgpu_mes_self_test(adev); | ||
| 1712 | |||
| 1713 | return 0; | ||
| 1714 | } | ||
| 1715 | |||
| 1716 | static const struct amd_ip_funcs mes_v11_0_ip_funcs = { | 1696 | static const struct amd_ip_funcs mes_v11_0_ip_funcs = { |
| 1717 | .name = "mes_v11_0", | 1697 | .name = "mes_v11_0", |
| 1718 | .early_init = mes_v11_0_early_init, | 1698 | .early_init = mes_v11_0_early_init, |
| 1719 | .late_init = mes_v11_0_late_init, | 1699 | .late_init = NULL, |
| 1720 | .sw_init = mes_v11_0_sw_init, | 1700 | .sw_init = mes_v11_0_sw_init, |
| 1721 | .sw_fini = mes_v11_0_sw_fini, | 1701 | .sw_fini = mes_v11_0_sw_fini, |
| 1722 | .hw_init = mes_v11_0_hw_init, | 1702 | .hw_init = mes_v11_0_hw_init, |
| @@ -274,6 +274,23 @@ static int convert_to_mes_queue_type(int queue_type) · 273 unmodified lines | |||
| 274 | return -1; | 274 | return -1; |
| 275 | } | 275 | } |
| 276 | 276 | ||
| 277 | static int convert_to_mes_priority_level(int priority_level) | ||
| 278 | { | ||
| 279 | switch (priority_level) { | ||
| 280 | case AMDGPU_MES_PRIORITY_LEVEL_LOW: | ||
| 281 | return AMD_PRIORITY_LEVEL_LOW; | ||
| 282 | case AMDGPU_MES_PRIORITY_LEVEL_NORMAL: | ||
| 283 | default: | ||
| 284 | return AMD_PRIORITY_LEVEL_NORMAL; | ||
| 285 | case AMDGPU_MES_PRIORITY_LEVEL_MEDIUM: | ||
| 286 | return AMD_PRIORITY_LEVEL_MEDIUM; | ||
| 287 | case AMDGPU_MES_PRIORITY_LEVEL_HIGH: | ||
| 288 | return AMD_PRIORITY_LEVEL_HIGH; | ||
| 289 | case AMDGPU_MES_PRIORITY_LEVEL_REALTIME: | ||
| 290 | return AMD_PRIORITY_LEVEL_REALTIME; | ||
| 291 | } | ||
| 292 | } | ||
| 293 | |||
| 277 | static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes, | 294 | static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes, |
| 278 | struct mes_add_queue_input *input) | 295 | struct mes_add_queue_input *input) |
| 279 | { | 296 | { |
| @@ -297,9 +314,9 @@ static int mes_v12_0_add_hw_queue(struct amdgpu_mes *mes, · 17 unmodified lines | |||
| 297 | mes_add_queue_pkt.gang_quantum = input->gang_quantum; | 314 | mes_add_queue_pkt.gang_quantum = input->gang_quantum; |
| 298 | mes_add_queue_pkt.gang_context_addr = input->gang_context_addr; | 315 | mes_add_queue_pkt.gang_context_addr = input->gang_context_addr; |
| 299 | mes_add_queue_pkt.inprocess_gang_priority = | 316 | mes_add_queue_pkt.inprocess_gang_priority = |
| 300 | input->inprocess_gang_priority; | 317 | convert_to_mes_priority_level(input->inprocess_gang_priority); |
| 301 | mes_add_queue_pkt.gang_global_priority_level = | 318 | mes_add_queue_pkt.gang_global_priority_level = |
| 302 | input->gang_global_priority_level; | 319 | convert_to_mes_priority_level(input->gang_global_priority_level); |
| 303 | mes_add_queue_pkt.doorbell_offset = input->doorbell_offset; | 320 | mes_add_queue_pkt.doorbell_offset = input->doorbell_offset; |
| 304 | mes_add_queue_pkt.mqd_addr = input->mqd_addr; | 321 | mes_add_queue_pkt.mqd_addr = input->mqd_addr; |
| 305 | 322 | ||
| @@ -477,32 +494,6 @@ static int mes_v12_0_reset_queue_mmio(struct amdgpu_mes *mes, uint32_t queue_typ · 171 unmodified lines | |||
| 477 | return r; | 494 | return r; |
| 478 | } | 495 | } |
| 479 | 496 | ||
| 480 | static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes, | ||
| 481 | struct mes_reset_queue_input *input) | ||
| 482 | { | ||
| 483 | union MESAPI__RESET mes_reset_queue_pkt; | ||
| 484 | int pipe; | ||
| 485 | |||
| 486 | memset(&mes_reset_queue_pkt, 0, sizeof(mes_reset_queue_pkt)); | ||
| 487 | |||
| 488 | mes_reset_queue_pkt.header.type = MES_API_TYPE_SCHEDULER; | ||
| 489 | mes_reset_queue_pkt.header.opcode = MES_SCH_API_RESET; | ||
| 490 | mes_reset_queue_pkt.header.dwsize = API_FRAME_SIZE_IN_DWORDS; | ||
| 491 | |||
| 492 | mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset; | ||
| 493 | mes_reset_queue_pkt.gang_context_addr = input->gang_context_addr; | ||
| 494 | /*mes_reset_queue_pkt.reset_queue_only = 1;*/ | ||
| 495 | |||
| 496 | if (mes->adev->enable_uni_mes) | ||
| 497 | pipe = AMDGPU_MES_KIQ_PIPE; | ||
| 498 | else | ||
| 499 | pipe = AMDGPU_MES_SCHED_PIPE; | ||
| 500 | |||
| 501 | return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe, | ||
| 502 | &mes_reset_queue_pkt, sizeof(mes_reset_queue_pkt), | ||
| 503 | offsetof(union MESAPI__REMOVE_QUEUE, api_status)); | ||
| 504 | } | ||
| 505 | |||
| 506 | static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes, | 497 | static int mes_v12_0_map_legacy_queue(struct amdgpu_mes *mes, |
| 507 | struct mes_map_legacy_queue_input *input) | 498 | struct mes_map_legacy_queue_input *input) |
| 508 | { | 499 | { |
| @@ -762,7 +753,7 @@ static int mes_v12_0_set_hw_resources(struct amdgpu_mes *mes, int pipe) · 253 unmodified lines | |||
| 762 | pipe * (AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE); | 753 | pipe * (AMDGPU_MES_LOG_BUFFER_SIZE + AMDGPU_MES_MSCRATCH_SIZE); |
| 763 | } | 754 | } |
| 764 | 755 | ||
| 765 | if (enforce_isolation) | 756 | if (adev->enforce_isolation[0] == AMDGPU_ENFORCE_ISOLATION_ENABLE) |
| 766 | mes_set_hw_res_pkt.limit_single_process = 1; | 757 | mes_set_hw_res_pkt.limit_single_process = 1; |
| 767 | 758 | ||
| 768 | return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe, | 759 | return mes_v12_0_submit_pkt_and_poll_completion(mes, pipe, |
| @@ -845,8 +836,8 @@ static void mes_v12_0_enable_unmapped_doorbell_handling( · 76 unmodified lines | |||
| 845 | WREG32_SOC15(GC, 0, regCP_UNMAPPED_DOORBELL, data); | 836 | WREG32_SOC15(GC, 0, regCP_UNMAPPED_DOORBELL, data); |
| 846 | } | 837 | } |
| 847 | 838 | ||
| 848 | static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, | 839 | static int mes_v12_0_reset_hw_queue(struct amdgpu_mes *mes, |
| 849 | struct mes_reset_legacy_queue_input *input) | 840 | struct mes_reset_queue_input *input) |
| 850 | { | 841 | { |
| 851 | union MESAPI__RESET mes_reset_queue_pkt; | 842 | union MESAPI__RESET mes_reset_queue_pkt; |
| 852 | int pipe; | 843 | int pipe; |
| @@ -865,7 +856,7 @@ static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, · 12 unmodified lines | |||
| 865 | mes_reset_queue_pkt.queue_type = | 856 | mes_reset_queue_pkt.queue_type = |
| 866 | convert_to_mes_queue_type(input->queue_type); | 857 | convert_to_mes_queue_type(input->queue_type); |
| 867 | 858 | ||
| 868 | if (mes_reset_queue_pkt.queue_type == MES_QUEUE_TYPE_GFX) { | 859 | if (input->legacy_gfx) { |
| 869 | mes_reset_queue_pkt.reset_legacy_gfx = 1; | 860 | mes_reset_queue_pkt.reset_legacy_gfx = 1; |
| 870 | mes_reset_queue_pkt.pipe_id_lp = input->pipe_id; | 861 | mes_reset_queue_pkt.pipe_id_lp = input->pipe_id; |
| 871 | mes_reset_queue_pkt.queue_id_lp = input->queue_id; | 862 | mes_reset_queue_pkt.queue_id_lp = input->queue_id; |
| @@ -878,7 +869,7 @@ static int mes_v12_0_reset_legacy_queue(struct amdgpu_mes *mes, · 6 unmodified lines | |||
| 878 | mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset; | 869 | mes_reset_queue_pkt.doorbell_offset = input->doorbell_offset; |
| 879 | } | 870 | } |
| 880 | 871 | ||
| 881 | if (mes->adev->enable_uni_mes) | 872 | if (input->is_kq) |
| 882 | pipe = AMDGPU_MES_KIQ_PIPE; | 873 | pipe = AMDGPU_MES_KIQ_PIPE; |
| 883 | else | 874 | else |
| 884 | pipe = AMDGPU_MES_SCHED_PIPE; | 875 | pipe = AMDGPU_MES_SCHED_PIPE; |
| @@ -896,7 +887,6 @@ static const struct amdgpu_mes_funcs mes_v12_0_funcs = { · 11 unmodified lines | |||
| 896 | .suspend_gang = mes_v12_0_suspend_gang, | 887 | .suspend_gang = mes_v12_0_suspend_gang, |
| 897 | .resume_gang = mes_v12_0_resume_gang, | 888 | .resume_gang = mes_v12_0_resume_gang, |
| 898 | .misc_op = mes_v12_0_misc_op, | 889 | .misc_op = mes_v12_0_misc_op, |
| 899 | .reset_legacy_queue = mes_v12_0_reset_legacy_queue, | ||
| 900 | .reset_hw_queue = mes_v12_0_reset_hw_queue, | 890 | .reset_hw_queue = mes_v12_0_reset_hw_queue, |
| 901 | }; | 891 | }; |
| 902 | 892 | ||
| @@ -1811,21 +1801,10 @@ static int mes_v12_0_early_init(struct amdgpu_ip_block *ip_block) · 908 unmodified lines | |||
| 1811 | return 0; | 1801 | return 0; |
| 1812 | } | 1802 | } |
| 1813 | 1803 | ||
| 1814 | static int mes_v12_0_late_init(struct amdgpu_ip_block *ip_block) | ||
| 1815 | { | ||
| 1816 | struct amdgpu_device *adev = ip_block->adev; | ||
| 1817 | |||
| 1818 | /* it's only intended for use in mes_self_test case, not for s0ix and reset */ | ||
| 1819 | if (!amdgpu_in_reset(adev) && !adev->in_s0ix && !adev->in_suspend) | ||
| 1820 | amdgpu_mes_self_test(adev); | ||
| 1821 | |||
| 1822 | return 0; | ||
| 1823 | } | ||
| 1824 | |||
| 1825 | static const struct amd_ip_funcs mes_v12_0_ip_funcs = { | 1804 | static const struct amd_ip_funcs mes_v12_0_ip_funcs = { |
| 1826 | .name = "mes_v12_0", | 1805 | .name = "mes_v12_0", |
| 1827 | .early_init = mes_v12_0_early_init, | 1806 | .early_init = mes_v12_0_early_init, |
| 1828 | .late_init = mes_v12_0_late_init, | 1807 | .late_init = NULL, |
| 1829 | .sw_init = mes_v12_0_sw_init, | 1808 | .sw_init = mes_v12_0_sw_init, |
| 1830 | .sw_fini = mes_v12_0_sw_fini, | 1809 | .sw_fini = mes_v12_0_sw_fini, |
| 1831 | .hw_init = mes_v12_0_hw_init, | 1810 | .hw_init = mes_v12_0_hw_init, |
| @@ -30,6 +30,7 @@ · 29 unmodified lines | |||
| 30 | #include "soc15_common.h" | 30 | #include "soc15_common.h" |
| 31 | #include "soc15.h" | 31 | #include "soc15.h" |
| 32 | #include "amdgpu_ras.h" | 32 | #include "amdgpu_ras.h" |
| 33 | #include "amdgpu_psp.h" | ||
| 33 | 34 | ||
| 34 | #define regVM_L2_CNTL3_DEFAULT 0x80100007 | 35 | #define regVM_L2_CNTL3_DEFAULT 0x80100007 |
| 35 | #define regVM_L2_CNTL4_DEFAULT 0x000000c1 | 36 | #define regVM_L2_CNTL4_DEFAULT 0x000000c1 |
| @@ -192,10 +193,8 @@ static void mmhub_v1_8_init_tlb_regs(struct amdgpu_device *adev) · 156 unmodified lines | |||
| 192 | uint32_t tmp, inst_mask; | 193 | uint32_t tmp, inst_mask; |
| 193 | int i; | 194 | int i; |
| 194 | 195 | ||
| 195 | /* Setup TLB control */ | 196 | if (amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev)) { |
| 196 | inst_mask = adev->aid_mask; | 197 | tmp = RREG32_SOC15(MMHUB, 0, regMC_VM_MX_L1_TLB_CNTL); |
| 197 | for_each_inst(i, inst_mask) { | ||
| 198 | tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL); | ||
| 199 | 198 | ||
| 200 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, | 199 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, |
| 201 | 1); | 200 | 1); |
| @@ -209,7 +208,26 @@ static void mmhub_v1_8_init_tlb_regs(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 209 | MTYPE, MTYPE_UC);/* XXX for emulation. */ | 208 | MTYPE, MTYPE_UC);/* XXX for emulation. */ |
| 210 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1); | 209 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1); |
| 211 | 210 | ||
| 212 | WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp); | 211 | psp_reg_program_no_ring(&adev->psp, tmp, PSP_REG_MMHUB_L1_TLB_CNTL); |
| 212 | } else { | ||
| 213 | inst_mask = adev->aid_mask; | ||
| 214 | for_each_inst(i, inst_mask) { | ||
| 215 | tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL); | ||
| 216 | |||
| 217 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, | ||
| 218 | 1); | ||
| 219 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 220 | SYSTEM_ACCESS_MODE, 3); | ||
| 221 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 222 | ENABLE_ADVANCED_DRIVER_MODEL, 1); | ||
| 223 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 224 | SYSTEM_APERTURE_UNMAPPED_ACCESS, 0); | ||
| 225 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 226 | MTYPE, MTYPE_UC);/* XXX for emulation. */ | ||
| 227 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ATC_EN, 1); | ||
| 228 | |||
| 229 | WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp); | ||
| 230 | } | ||
| 213 | } | 231 | } |
| 214 | } | 232 | } |
| 215 | 233 | ||
| @@ -221,6 +239,9 @@ static void mmhub_v1_8_init_snoop_override_regs(struct amdgpu_device *adev) · 5 unmodified lines | |||
| 221 | uint32_t distance = regDAGB1_WRCLI_GPU_SNOOP_OVERRIDE - | 239 | uint32_t distance = regDAGB1_WRCLI_GPU_SNOOP_OVERRIDE - |
| 222 | regDAGB0_WRCLI_GPU_SNOOP_OVERRIDE; | 240 | regDAGB0_WRCLI_GPU_SNOOP_OVERRIDE; |
| 223 | 241 | ||
| 242 | if (amdgpu_sriov_vf(adev)) | ||
| 243 | return; | ||
| 244 | |||
| 224 | inst_mask = adev->aid_mask; | 245 | inst_mask = adev->aid_mask; |
| 225 | for_each_inst(i, inst_mask) { | 246 | for_each_inst(i, inst_mask) { |
| 226 | for (j = 0; j < 5; j++) { /* DAGB instances */ | 247 | for (j = 0; j < 5; j++) { /* DAGB instances */ |
| @@ -454,6 +475,30 @@ static int mmhub_v1_8_gart_enable(struct amdgpu_device *adev) · 227 unmodified lines | |||
| 454 | return 0; | 475 | return 0; |
| 455 | } | 476 | } |
| 456 | 477 | ||
| 478 | static void mmhub_v1_8_disable_l1_tlb(struct amdgpu_device *adev) | ||
| 479 | { | ||
| 480 | u32 tmp; | ||
| 481 | u32 i, inst_mask; | ||
| 482 | |||
| 483 | if (amdgpu_sriov_reg_indirect_l1_tlb_cntl(adev)) { | ||
| 484 | tmp = RREG32_SOC15(MMHUB, 0, regMC_VM_MX_L1_TLB_CNTL); | ||
| 485 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, 0); | ||
| 486 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 487 | ENABLE_ADVANCED_DRIVER_MODEL, 0); | ||
| 488 | psp_reg_program_no_ring(&adev->psp, tmp, PSP_REG_MMHUB_L1_TLB_CNTL); | ||
| 489 | } else { | ||
| 490 | inst_mask = adev->aid_mask; | ||
| 491 | for_each_inst(i, inst_mask) { | ||
| 492 | tmp = RREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL); | ||
| 493 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, | ||
| 494 | 0); | ||
| 495 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 496 | ENABLE_ADVANCED_DRIVER_MODEL, 0); | ||
| 497 | WREG32_SOC15(MMHUB, i, regMC_VM_MX_L1_TLB_CNTL, tmp); | ||
| 498 | } | ||
| 499 | } | ||
| 500 | } | ||
| 501 | |||
| 457 | static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) | 502 | static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) |
| 458 | { | 503 | { |
| 459 | struct amdgpu_vmhub *hub; | 504 | struct amdgpu_vmhub *hub; |
| @@ -467,15 +512,6 @@ static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 467 | for (i = 0; i < 16; i++) | 512 | for (i = 0; i < 16; i++) |
| 468 | WREG32_SOC15_OFFSET(MMHUB, j, regVM_CONTEXT0_CNTL, | 513 | WREG32_SOC15_OFFSET(MMHUB, j, regVM_CONTEXT0_CNTL, |
| 469 | i * hub->ctx_distance, 0); | 514 | i * hub->ctx_distance, 0); |
| 470 | |||
| 471 | /* Setup TLB control */ | ||
| 472 | tmp = RREG32_SOC15(MMHUB, j, regMC_VM_MX_L1_TLB_CNTL); | ||
| 473 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, | ||
| 474 | 0); | ||
| 475 | tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, | ||
| 476 | ENABLE_ADVANCED_DRIVER_MODEL, 0); | ||
| 477 | WREG32_SOC15(MMHUB, j, regMC_VM_MX_L1_TLB_CNTL, tmp); | ||
| 478 | |||
| 479 | if (!amdgpu_sriov_vf(adev)) { | 515 | if (!amdgpu_sriov_vf(adev)) { |
| 480 | /* Setup L2 cache */ | 516 | /* Setup L2 cache */ |
| 481 | tmp = RREG32_SOC15(MMHUB, j, regVM_L2_CNTL); | 517 | tmp = RREG32_SOC15(MMHUB, j, regVM_L2_CNTL); |
| @@ -485,6 +521,8 @@ static void mmhub_v1_8_gart_disable(struct amdgpu_device *adev) · 3 unmodified lines | |||
| 485 | WREG32_SOC15(MMHUB, j, regVM_L2_CNTL3, 0); | 521 | WREG32_SOC15(MMHUB, j, regVM_L2_CNTL3, 0); |
| 486 | } | 522 | } |
| 487 | } | 523 | } |
| 524 | |||
| 525 | mmhub_v1_8_disable_l1_tlb(adev); | ||
| 488 | } | 526 | } |
| 489 | 527 | ||
| 490 | /** | 528 | /** |
| @@ -274,6 +274,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 273 unmodified lines | |||
| 274 | { | 274 | { |
| 275 | struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work); | 275 | struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work); |
| 276 | struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt); | 276 | struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt); |
| 277 | struct amdgpu_reset_context reset_context = { 0 }; | ||
| 277 | 278 | ||
| 278 | amdgpu_virt_fini_data_exchange(adev); | 279 | amdgpu_virt_fini_data_exchange(adev); |
| 279 | 280 | ||
| @@ -281,8 +282,6 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 1 unmodified line | |||
| 281 | if (amdgpu_device_should_recover_gpu(adev) | 282 | if (amdgpu_device_should_recover_gpu(adev) |
| 282 | && (!amdgpu_device_has_job_running(adev) || | 283 | && (!amdgpu_device_has_job_running(adev) || |
| 283 | adev->sdma_timeout == MAX_SCHEDULE_TIMEOUT)) { | 284 | adev->sdma_timeout == MAX_SCHEDULE_TIMEOUT)) { |
| 284 | struct amdgpu_reset_context reset_context; | ||
| 285 | memset(&reset_context, 0, sizeof(reset_context)); | ||
| 286 | 285 | ||
| 287 | reset_context.method = AMD_RESET_METHOD_NONE; | 286 | reset_context.method = AMD_RESET_METHOD_NONE; |
| 288 | reset_context.reset_req_dev = adev; | 287 | reset_context.reset_req_dev = adev; |
| @@ -293,6 +292,19 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines | |||
| 293 | } | 292 | } |
| 294 | } | 293 | } |
| 295 | 294 | ||
| 295 | static void xgpu_ai_mailbox_bad_pages_work(struct work_struct *work) | ||
| 296 | { | ||
| 297 | struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, bad_pages_work); | ||
| 298 | struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt); | ||
| 299 | |||
| 300 | if (down_read_trylock(&adev->reset_domain->sem)) { | ||
| 301 | amdgpu_virt_fini_data_exchange(adev); | ||
| 302 | amdgpu_virt_request_bad_pages(adev); | ||
| 303 | amdgpu_virt_init_data_exchange(adev); | ||
| 304 | up_read(&adev->reset_domain->sem); | ||
| 305 | } | ||
| 306 | } | ||
| 307 | |||
| 296 | static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev, | 308 | static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev, |
| 297 | struct amdgpu_irq_src *src, | 309 | struct amdgpu_irq_src *src, |
| 298 | unsigned type, | 310 | unsigned type, |
| @@ -312,26 +324,42 @@ static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev, · 13 unmodified lines | |||
| 312 | struct amdgpu_iv_entry *entry) | 324 | struct amdgpu_iv_entry *entry) |
| 313 | { | 325 | { |
| 314 | enum idh_event event = xgpu_ai_mailbox_peek_msg(adev); | 326 | enum idh_event event = xgpu_ai_mailbox_peek_msg(adev); |
| 327 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | ||
| 315 | 328 | ||
| 316 | switch (event) { | 329 | switch (event) { |
| 317 | case IDH_FLR_NOTIFICATION: | 330 | case IDH_RAS_BAD_PAGES_NOTIFICATION: |
| 331 | xgpu_ai_mailbox_send_ack(adev); | ||
| 332 | if (amdgpu_sriov_runtime(adev)) | ||
| 333 | schedule_work(&adev->virt.bad_pages_work); | ||
| 334 | break; | ||
| 335 | case IDH_UNRECOV_ERR_NOTIFICATION: | ||
| 336 | xgpu_ai_mailbox_send_ack(adev); | ||
| 337 | ras->is_rma = true; | ||
| 338 | dev_err(adev->dev, "VF is in an unrecoverable state. Runtime Services are halted.\n"); | ||
| 318 | if (amdgpu_sriov_runtime(adev)) | 339 | if (amdgpu_sriov_runtime(adev)) |
| 319 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, | 340 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, |
| 320 | &adev->virt.flr_work), | 341 | &adev->virt.flr_work), |
| 321 | "Failed to queue work! at %s", | 342 | "Failed to queue work! at %s", |
| 322 | __func__); | 343 | __func__); |
| 323 | break; | 344 | break; |
| 324 | case IDH_QUERY_ALIVE: | 345 | case IDH_FLR_NOTIFICATION: |
| 325 | xgpu_ai_mailbox_send_ack(adev); | 346 | if (amdgpu_sriov_runtime(adev)) |
| 326 | break; | 347 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, |
| 327 | /* READY_TO_ACCESS_GPU is fetched by kernel polling, IRQ can ignore | 348 | &adev->virt.flr_work), |
| 328 | * it byfar since that polling thread will handle it, | 349 | "Failed to queue work! at %s", |
| 329 | * other msg like flr complete is not handled here. | 350 | __func__); |
| 330 | */ | 351 | break; |
| 331 | case IDH_CLR_MSG_BUF: | 352 | case IDH_QUERY_ALIVE: |
| 332 | case IDH_FLR_NOTIFICATION_CMPL: | 353 | xgpu_ai_mailbox_send_ack(adev); |
| 333 | case IDH_READY_TO_ACCESS_GPU: | 354 | break; |
| 334 | default: | 355 | /* READY_TO_ACCESS_GPU is fetched by kernel polling, IRQ can ignore |
| 356 | * it byfar since that polling thread will handle it, | ||
| 357 | * other msg like flr complete is not handled here. | ||
| 358 | */ | ||
| 359 | case IDH_CLR_MSG_BUF: | ||
| 360 | case IDH_FLR_NOTIFICATION_CMPL: | ||
| 361 | case IDH_READY_TO_ACCESS_GPU: | ||
| 362 | default: | ||
| 335 | break; | 363 | break; |
| 336 | } | 364 | } |
| 337 | 365 | ||
| @@ -387,6 +415,7 @@ int xgpu_ai_mailbox_get_irq(struct amdgpu_device *adev) · 49 unmodified lines | |||
| 387 | } | 415 | } |
| 388 | 416 | ||
| 389 | INIT_WORK(&adev->virt.flr_work, xgpu_ai_mailbox_flr_work); | 417 | INIT_WORK(&adev->virt.flr_work, xgpu_ai_mailbox_flr_work); |
| 418 | INIT_WORK(&adev->virt.bad_pages_work, xgpu_ai_mailbox_bad_pages_work); | ||
| 390 | 419 | ||
| 391 | return 0; | 420 | return 0; |
| 392 | } | 421 | } |
| @@ -40,6 +40,7 @@ enum idh_request { · 39 unmodified lines | |||
| 40 | IDH_LOG_VF_ERROR = 200, | 40 | IDH_LOG_VF_ERROR = 200, |
| 41 | IDH_READY_TO_RESET = 201, | 41 | IDH_READY_TO_RESET = 201, |
| 42 | IDH_RAS_POISON = 202, | 42 | IDH_RAS_POISON = 202, |
| 43 | IDH_REQ_RAS_BAD_PAGES = 205, | ||
| 43 | }; | 44 | }; |
| 44 | 45 | ||
| 45 | enum idh_event { | 46 | enum idh_event { |
| @@ -54,6 +55,9 @@ enum idh_event { · 8 unmodified lines | |||
| 54 | IDH_RAS_POISON_READY, | 55 | IDH_RAS_POISON_READY, |
| 55 | IDH_PF_SOFT_FLR_NOTIFICATION, | 56 | IDH_PF_SOFT_FLR_NOTIFICATION, |
| 56 | IDH_RAS_ERROR_DETECTED, | 57 | IDH_RAS_ERROR_DETECTED, |
| 58 | IDH_RAS_BAD_PAGES_READY = 15, | ||
| 59 | IDH_RAS_BAD_PAGES_NOTIFICATION = 16, | ||
| 60 | IDH_UNRECOV_ERR_NOTIFICATION = 17, | ||
| 57 | IDH_TEXT_MESSAGE = 255, | 61 | IDH_TEXT_MESSAGE = 255, |
| 58 | }; | 62 | }; |
| 59 | 63 | ||
| @@ -67,6 +67,8 @@ static int xgpu_nv_mailbox_rcv_msg(struct amdgpu_device *adev, · 66 unmodified lines | |||
| 67 | reg = RREG32_NO_KIQ(mmMAILBOX_MSGBUF_RCV_DW0); | 67 | reg = RREG32_NO_KIQ(mmMAILBOX_MSGBUF_RCV_DW0); |
| 68 | if (reg == IDH_FAIL) | 68 | if (reg == IDH_FAIL) |
| 69 | r = -EINVAL; | 69 | r = -EINVAL; |
| 70 | if (reg == IDH_UNRECOV_ERR_NOTIFICATION) | ||
| 71 | r = -ENODEV; | ||
| 70 | else if (reg != event) | 72 | else if (reg != event) |
| 71 | return -ENOENT; | 73 | return -ENOENT; |
| 72 | 74 | ||
| @@ -103,6 +105,7 @@ static int xgpu_nv_poll_msg(struct amdgpu_device *adev, enum idh_event event) · 30 unmodified lines | |||
| 103 | { | 105 | { |
| 104 | int r; | 106 | int r; |
| 105 | uint64_t timeout, now; | 107 | uint64_t timeout, now; |
| 108 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | ||
| 106 | 109 | ||
| 107 | now = (uint64_t)ktime_to_ms(ktime_get()); | 110 | now = (uint64_t)ktime_to_ms(ktime_get()); |
| 108 | timeout = now + NV_MAILBOX_POLL_MSG_TIMEDOUT; | 111 | timeout = now + NV_MAILBOX_POLL_MSG_TIMEDOUT; |
| @@ -110,8 +113,16 @@ static int xgpu_nv_poll_msg(struct amdgpu_device *adev, enum idh_event event) · 1 unmodified line | |||
| 110 | do { | 113 | do { |
| 111 | r = xgpu_nv_mailbox_rcv_msg(adev, event); | 114 | r = xgpu_nv_mailbox_rcv_msg(adev, event); |
| 112 | if (!r) { | 115 | if (!r) { |
| 113 | dev_dbg(adev->dev, "rcv_msg 0x%x after %llu ms\n", event, NV_MAILBOX_POLL_MSG_TIMEDOUT - timeout + now); | 116 | dev_dbg(adev->dev, "rcv_msg 0x%x after %llu ms\n", |
| 117 | event, NV_MAILBOX_POLL_MSG_TIMEDOUT - timeout + now); | ||
| 114 | return 0; | 118 | return 0; |
| 119 | } else if (r == -ENODEV) { | ||
| 120 | if (!amdgpu_ras_is_rma(adev)) { | ||
| 121 | ras->is_rma = true; | ||
| 122 | dev_err(adev->dev, "VF is in an unrecoverable state. " | ||
| 123 | "Runtime Services are halted.\n"); | ||
| 124 | } | ||
| 125 | return r; | ||
| 115 | } | 126 | } |
| 116 | 127 | ||
| 117 | msleep(10); | 128 | msleep(10); |
| @@ -166,6 +177,10 @@ static int xgpu_nv_send_access_requests_with_param(struct amdgpu_device *adev, · 48 unmodified lines | |||
| 166 | enum idh_event event = -1; | 177 | enum idh_event event = -1; |
| 167 | 178 | ||
| 168 | send_request: | 179 | send_request: |
| 180 | |||
| 181 | if (amdgpu_ras_is_rma(adev)) | ||
| 182 | return -ENODEV; | ||
| 183 | |||
| 169 | xgpu_nv_mailbox_trans_msg(adev, req, data1, data2, data3); | 184 | xgpu_nv_mailbox_trans_msg(adev, req, data1, data2, data3); |
| 170 | 185 | ||
| 171 | switch (req) { | 186 | switch (req) { |
| @@ -187,6 +202,9 @@ send_request: · 15 unmodified lines | |||
| 187 | case IDH_REQ_RAS_CPER_DUMP: | 202 | case IDH_REQ_RAS_CPER_DUMP: |
| 188 | event = IDH_RAS_CPER_DUMP_READY; | 203 | event = IDH_RAS_CPER_DUMP_READY; |
| 189 | break; | 204 | break; |
| 205 | case IDH_REQ_RAS_BAD_PAGES: | ||
| 206 | event = IDH_RAS_BAD_PAGES_READY; | ||
| 207 | break; | ||
| 190 | default: | 208 | default: |
| 191 | break; | 209 | break; |
| 192 | } | 210 | } |
| @@ -320,6 +338,7 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 127 unmodified lines | |||
| 320 | { | 338 | { |
| 321 | struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work); | 339 | struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, flr_work); |
| 322 | struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt); | 340 | struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt); |
| 341 | struct amdgpu_reset_context reset_context = { 0 }; | ||
| 323 | 342 | ||
| 324 | amdgpu_virt_fini_data_exchange(adev); | 343 | amdgpu_virt_fini_data_exchange(adev); |
| 325 | 344 | ||
| @@ -330,8 +349,6 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines | |||
| 330 | adev->gfx_timeout == MAX_SCHEDULE_TIMEOUT || | 349 | adev->gfx_timeout == MAX_SCHEDULE_TIMEOUT || |
| 331 | adev->compute_timeout == MAX_SCHEDULE_TIMEOUT || | 350 | adev->compute_timeout == MAX_SCHEDULE_TIMEOUT || |
| 332 | adev->video_timeout == MAX_SCHEDULE_TIMEOUT)) { | 351 | adev->video_timeout == MAX_SCHEDULE_TIMEOUT)) { |
| 333 | struct amdgpu_reset_context reset_context; | ||
| 334 | memset(&reset_context, 0, sizeof(reset_context)); | ||
| 335 | 352 | ||
| 336 | reset_context.method = AMD_RESET_METHOD_NONE; | 353 | reset_context.method = AMD_RESET_METHOD_NONE; |
| 337 | reset_context.reset_req_dev = adev; | 354 | reset_context.reset_req_dev = adev; |
| @@ -342,6 +359,19 @@ static void xgpu_nv_mailbox_flr_work(struct work_struct *work) · 4 unmodified lines | |||
| 342 | } | 359 | } |
| 343 | } | 360 | } |
| 344 | 361 | ||
| 362 | static void xgpu_nv_mailbox_bad_pages_work(struct work_struct *work) | ||
| 363 | { | ||
| 364 | struct amdgpu_virt *virt = container_of(work, struct amdgpu_virt, bad_pages_work); | ||
| 365 | struct amdgpu_device *adev = container_of(virt, struct amdgpu_device, virt); | ||
| 366 | |||
| 367 | if (down_read_trylock(&adev->reset_domain->sem)) { | ||
| 368 | amdgpu_virt_fini_data_exchange(adev); | ||
| 369 | amdgpu_virt_request_bad_pages(adev); | ||
| 370 | amdgpu_virt_init_data_exchange(adev); | ||
| 371 | up_read(&adev->reset_domain->sem); | ||
| 372 | } | ||
| 373 | } | ||
| 374 | |||
| 345 | static int xgpu_nv_set_mailbox_rcv_irq(struct amdgpu_device *adev, | 375 | static int xgpu_nv_set_mailbox_rcv_irq(struct amdgpu_device *adev, |
| 346 | struct amdgpu_irq_src *src, | 376 | struct amdgpu_irq_src *src, |
| 347 | unsigned type, | 377 | unsigned type, |
| @@ -364,8 +394,27 @@ static int xgpu_nv_mailbox_rcv_irq(struct amdgpu_device *adev, · 16 unmodified lines | |||
| 364 | struct amdgpu_iv_entry *entry) | 394 | struct amdgpu_iv_entry *entry) |
| 365 | { | 395 | { |
| 366 | enum idh_event event = xgpu_nv_mailbox_peek_msg(adev); | 396 | enum idh_event event = xgpu_nv_mailbox_peek_msg(adev); |
| 397 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | ||
| 367 | 398 | ||
| 368 | switch (event) { | 399 | switch (event) { |
| 400 | case IDH_RAS_BAD_PAGES_NOTIFICATION: | ||
| 401 | xgpu_nv_mailbox_send_ack(adev); | ||
| 402 | if (amdgpu_sriov_runtime(adev)) | ||
| 403 | schedule_work(&adev->virt.bad_pages_work); | ||
| 404 | break; | ||
| 405 | case IDH_UNRECOV_ERR_NOTIFICATION: | ||
| 406 | xgpu_nv_mailbox_send_ack(adev); | ||
| 407 | if (!amdgpu_ras_is_rma(adev)) { | ||
| 408 | ras->is_rma = true; | ||
| 409 | dev_err(adev->dev, "VF is in an unrecoverable state. Runtime Services are halted.\n"); | ||
| 410 | } | ||
| 411 | |||
| 412 | if (amdgpu_sriov_runtime(adev)) | ||
| 413 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, | ||
| 414 | &adev->virt.flr_work), | ||
| 415 | "Failed to queue work! at %s", | ||
| 416 | __func__); | ||
| 417 | break; | ||
| 369 | case IDH_FLR_NOTIFICATION: | 418 | case IDH_FLR_NOTIFICATION: |
| 370 | if (amdgpu_sriov_runtime(adev)) | 419 | if (amdgpu_sriov_runtime(adev)) |
| 371 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, | 420 | WARN_ONCE(!amdgpu_reset_domain_schedule(adev->reset_domain, |
| @@ -436,6 +485,7 @@ int xgpu_nv_mailbox_get_irq(struct amdgpu_device *adev) · 64 unmodified lines | |||
| 436 | } | 485 | } |
| 437 | 486 | ||
| 438 | INIT_WORK(&adev->virt.flr_work, xgpu_nv_mailbox_flr_work); | 487 | INIT_WORK(&adev->virt.flr_work, xgpu_nv_mailbox_flr_work); |
| 488 | INIT_WORK(&adev->virt.bad_pages_work, xgpu_nv_mailbox_bad_pages_work); | ||
| 439 | 489 | ||
| 440 | return 0; | 490 | return 0; |
| 441 | } | 491 | } |
| @@ -480,6 +530,11 @@ static int xgpu_nv_req_ras_cper_dump(struct amdgpu_device *adev, u64 vf_rptr) · 38 unmodified lines | |||
| 480 | adev, IDH_REQ_RAS_CPER_DUMP, vf_rptr_hi, vf_rptr_lo, 0); | 530 | adev, IDH_REQ_RAS_CPER_DUMP, vf_rptr_hi, vf_rptr_lo, 0); |
| 481 | } | 531 | } |
| 482 | 532 | ||
| 533 | static int xgpu_nv_req_ras_bad_pages(struct amdgpu_device *adev) | ||
| 534 | { | ||
| 535 | return xgpu_nv_send_access_requests(adev, IDH_REQ_RAS_BAD_PAGES); | ||
| 536 | } | ||
| 537 | |||
| 483 | const struct amdgpu_virt_ops xgpu_nv_virt_ops = { | 538 | const struct amdgpu_virt_ops xgpu_nv_virt_ops = { |
| 484 | .req_full_gpu = xgpu_nv_request_full_gpu_access, | 539 | .req_full_gpu = xgpu_nv_request_full_gpu_access, |
| 485 | .rel_full_gpu = xgpu_nv_release_full_gpu_access, | 540 | .rel_full_gpu = xgpu_nv_release_full_gpu_access, |
| @@ -492,4 +547,5 @@ const struct amdgpu_virt_ops xgpu_nv_virt_ops = { · 6 unmodified lines | |||
| 492 | .rcvd_ras_intr = xgpu_nv_rcvd_ras_intr, | 547 | .rcvd_ras_intr = xgpu_nv_rcvd_ras_intr, |
| 493 | .req_ras_err_count = xgpu_nv_req_ras_err_count, | 548 | .req_ras_err_count = xgpu_nv_req_ras_err_count, |
| 494 | .req_ras_cper_dump = xgpu_nv_req_ras_cper_dump, | 549 | .req_ras_cper_dump = xgpu_nv_req_ras_cper_dump, |
| 550 | .req_bad_pages = xgpu_nv_req_ras_bad_pages, | ||
| 495 | }; | 551 | }; |
| @@ -42,6 +42,7 @@ enum idh_request { · 41 unmodified lines | |||
| 42 | IDH_RAS_POISON = 202, | 42 | IDH_RAS_POISON = 202, |
| 43 | IDH_REQ_RAS_ERROR_COUNT = 203, | 43 | IDH_REQ_RAS_ERROR_COUNT = 203, |
| 44 | IDH_REQ_RAS_CPER_DUMP = 204, | 44 | IDH_REQ_RAS_CPER_DUMP = 204, |
| 45 | IDH_REQ_RAS_BAD_PAGES = 205, | ||
| 45 | }; | 46 | }; |
| 46 | 47 | ||
| 47 | enum idh_event { | 48 | enum idh_event { |
| @@ -58,6 +59,9 @@ enum idh_event { · 10 unmodified lines | |||
| 58 | IDH_RAS_ERROR_DETECTED, | 59 | IDH_RAS_ERROR_DETECTED, |
| 59 | IDH_RAS_ERROR_COUNT_READY = 11, | 60 | IDH_RAS_ERROR_COUNT_READY = 11, |
| 60 | IDH_RAS_CPER_DUMP_READY = 14, | 61 | IDH_RAS_CPER_DUMP_READY = 14, |
| 62 | IDH_RAS_BAD_PAGES_READY = 15, | ||
| 63 | IDH_RAS_BAD_PAGES_NOTIFICATION = 16, | ||
| 64 | IDH_UNRECOV_ERR_NOTIFICATION = 17, | ||
| 61 | 65 | ||
| 62 | IDH_TEXT_MESSAGE = 255, | 66 | IDH_TEXT_MESSAGE = 255, |
| 63 | }; | 67 | }; |
| @@ -177,8 +177,12 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 176 unmodified lines | |||
| 177 | { | 177 | { |
| 178 | u32 doorbell_range = 0, doorbell_ctrl = 0; | 178 | u32 doorbell_range = 0, doorbell_ctrl = 0; |
| 179 | u32 aid_id = instance; | 179 | u32 aid_id = instance; |
| 180 | u32 range_size; | ||
| 180 | 181 | ||
| 181 | if (use_doorbell) { | 182 | if (use_doorbell) { |
| 183 | range_size = (amdgpu_ip_version(adev, GC_HWIP, 0) == | ||
| 184 | IP_VERSION(9, 5, 0)) ? | ||
| 185 | 0xb : 0x9; | ||
| 182 | doorbell_range = REG_SET_FIELD(doorbell_range, | 186 | doorbell_range = REG_SET_FIELD(doorbell_range, |
| 183 | DOORBELL0_CTRL_ENTRY_0, | 187 | DOORBELL0_CTRL_ENTRY_0, |
| 184 | BIF_DOORBELL0_RANGE_OFFSET_ENTRY, | 188 | BIF_DOORBELL0_RANGE_OFFSET_ENTRY, |
| @@ -186,7 +190,7 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 1 unmodified line | |||
| 186 | doorbell_range = REG_SET_FIELD(doorbell_range, | 190 | doorbell_range = REG_SET_FIELD(doorbell_range, |
| 187 | DOORBELL0_CTRL_ENTRY_0, | 191 | DOORBELL0_CTRL_ENTRY_0, |
| 188 | BIF_DOORBELL0_RANGE_SIZE_ENTRY, | 192 | BIF_DOORBELL0_RANGE_SIZE_ENTRY, |
| 189 | 0x9); | 193 | range_size); |
| 190 | if (aid_id) | 194 | if (aid_id) |
| 191 | doorbell_range = REG_SET_FIELD(doorbell_range, | 195 | doorbell_range = REG_SET_FIELD(doorbell_range, |
| 192 | DOORBELL0_CTRL_ENTRY_0, | 196 | DOORBELL0_CTRL_ENTRY_0, |
| @@ -204,7 +208,7 @@ static void nbio_v7_9_vcn_doorbell_range(struct amdgpu_device *adev, bool use_do · 11 unmodified lines | |||
| 204 | S2A_DOORBELL_PORT1_RANGE_OFFSET, 0x4); | 208 | S2A_DOORBELL_PORT1_RANGE_OFFSET, 0x4); |
| 205 | doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl, | 209 | doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl, |
| 206 | S2A_DOORBELL_ENTRY_1_CTRL, | 210 | S2A_DOORBELL_ENTRY_1_CTRL, |
| 207 | S2A_DOORBELL_PORT1_RANGE_SIZE, 0x9); | 211 | S2A_DOORBELL_PORT1_RANGE_SIZE, range_size); |
| 208 | doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl, | 212 | doorbell_ctrl = REG_SET_FIELD(doorbell_ctrl, |
| 209 | S2A_DOORBELL_ENTRY_1_CTRL, | 213 | S2A_DOORBELL_ENTRY_1_CTRL, |
| 210 | S2A_DOORBELL_PORT1_AWADDR_31_28_VALUE, 0x4); | 214 | S2A_DOORBELL_PORT1_AWADDR_31_28_VALUE, 0x4); |
| @@ -34,9 +34,6 @@ · 33 unmodified lines | |||
| 34 | #include "sdma0/sdma0_4_0_offset.h" | 34 | #include "sdma0/sdma0_4_0_offset.h" |
| 35 | #include "nbio/nbio_7_4_offset.h" | 35 | #include "nbio/nbio_7_4_offset.h" |
| 36 | 36 | ||
| 37 | #include "oss/osssys_4_0_offset.h" | ||
| 38 | #include "oss/osssys_4_0_sh_mask.h" | ||
| 39 | |||
| 40 | MODULE_FIRMWARE("amdgpu/renoir_asd.bin"); | 37 | MODULE_FIRMWARE("amdgpu/renoir_asd.bin"); |
| 41 | MODULE_FIRMWARE("amdgpu/renoir_ta.bin"); | 38 | MODULE_FIRMWARE("amdgpu/renoir_ta.bin"); |
| 42 | MODULE_FIRMWARE("amdgpu/green_sardine_asd.bin"); | 39 | MODULE_FIRMWARE("amdgpu/green_sardine_asd.bin"); |
| @@ -99,9 +96,6 @@ static int psp_v12_0_bootloader_load_sysdrv(struct psp_context *psp) · 56 unmodified lines | |||
| 99 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35, | 96 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35, |
| 100 | psp_gfxdrv_command_reg); | 97 | psp_gfxdrv_command_reg); |
| 101 | 98 | ||
| 102 | /* there might be handshake issue with hardware which needs delay */ | ||
| 103 | mdelay(20); | ||
| 104 | |||
| 105 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_35), | 99 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_35), |
| 106 | 0x80000000, 0x80000000, false); | 100 | 0x80000000, 0x80000000, false); |
| 107 | 101 | ||
| @@ -138,8 +132,6 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) · 30 unmodified lines | |||
| 138 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35, | 132 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_35, |
| 139 | psp_gfxdrv_command_reg); | 133 | psp_gfxdrv_command_reg); |
| 140 | 134 | ||
| 141 | /* there might be handshake issue with hardware which needs delay */ | ||
| 142 | mdelay(20); | ||
| 143 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_81), | 135 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_81), |
| 144 | RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_81), | 136 | RREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_81), |
| 145 | 0, true); | 137 | 0, true); |
| @@ -147,37 +139,6 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) · 1 unmodified line | |||
| 147 | return ret; | 139 | return ret; |
| 148 | } | 140 | } |
| 149 | 141 | ||
| 150 | static void psp_v12_0_reroute_ih(struct psp_context *psp) | ||
| 151 | { | ||
| 152 | struct amdgpu_device *adev = psp->adev; | ||
| 153 | uint32_t tmp; | ||
| 154 | |||
| 155 | /* Change IH ring for VMC */ | ||
| 156 | tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1244b); | ||
| 157 | tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, CLIENT_TYPE, 1); | ||
| 158 | tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1); | ||
| 159 | |||
| 160 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 3); | ||
| 161 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp); | ||
| 162 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET); | ||
| 163 | |||
| 164 | mdelay(20); | ||
| 165 | psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64), | ||
| 166 | 0x80000000, 0x8000FFFF, false); | ||
| 167 | |||
| 168 | /* Change IH ring for UMC */ | ||
| 169 | tmp = REG_SET_FIELD(0, IH_CLIENT_CFG_DATA, CREDIT_RETURN_ADDR, 0x1216b); | ||
| 170 | tmp = REG_SET_FIELD(tmp, IH_CLIENT_CFG_DATA, RING_ID, 1); | ||
| 171 | |||
| 172 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, 4); | ||
| 173 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, tmp); | ||
| 174 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, GFX_CTRL_CMD_ID_GBR_IH_SET); | ||
| 175 | |||
| 176 | mdelay(20); | ||
| 177 | psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64), | ||
| 178 | 0x80000000, 0x8000FFFF, false); | ||
| 179 | } | ||
| 180 | |||
| 181 | static int psp_v12_0_ring_create(struct psp_context *psp, | 142 | static int psp_v12_0_ring_create(struct psp_context *psp, |
| 182 | enum psp_ring_type ring_type) | 143 | enum psp_ring_type ring_type) |
| 183 | { | 144 | { |
| @@ -186,49 +147,23 @@ static int psp_v12_0_ring_create(struct psp_context *psp, · 2 unmodified lines | |||
| 186 | struct psp_ring *ring = &psp->km_ring; | 147 | struct psp_ring *ring = &psp->km_ring; |
| 187 | struct amdgpu_device *adev = psp->adev; | 148 | struct amdgpu_device *adev = psp->adev; |
| 188 | 149 | ||
| 189 | psp_v12_0_reroute_ih(psp); | 150 | /* Write low address of the ring to C2PMSG_69 */ |
| 190 | 151 | psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr); | |
| 191 | if (amdgpu_sriov_vf(psp->adev)) { | 152 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg); |
| 192 | /* Write low address of the ring to C2PMSG_102 */ | 153 | /* Write high address of the ring to C2PMSG_70 */ |
| 193 | psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr); | 154 | psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr); |
| 194 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_102, psp_ring_reg); | 155 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, psp_ring_reg); |
| 195 | /* Write high address of the ring to C2PMSG_103 */ | 156 | /* Write size of ring to C2PMSG_71 */ |
| 196 | psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr); | 157 | psp_ring_reg = ring->ring_size; |
| 197 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_103, psp_ring_reg); | 158 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_71, psp_ring_reg); |
| 198 | 159 | /* Write the ring initialization command to C2PMSG_64 */ | |
| 199 | /* Write the ring initialization command to C2PMSG_101 */ | 160 | psp_ring_reg = ring_type; |
| 200 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_101, | 161 | psp_ring_reg = psp_ring_reg << 16; |
| 201 | GFX_CTRL_CMD_ID_INIT_GPCOM_RING); | 162 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, psp_ring_reg); |
| 202 | 163 | ||
| 203 | /* there might be handshake issue with hardware which needs delay */ | 164 | /* Wait for response flag (bit 31) in C2PMSG_64 */ |
| 204 | mdelay(20); | 165 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64), |
| 205 | 166 | 0x80000000, 0x8000FFFF, false); | |
| 206 | /* Wait for response flag (bit 31) in C2PMSG_101 */ | ||
| 207 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101), | ||
| 208 | 0x80000000, 0x8000FFFF, false); | ||
| 209 | |||
| 210 | } else { | ||
| 211 | /* Write low address of the ring to C2PMSG_69 */ | ||
| 212 | psp_ring_reg = lower_32_bits(ring->ring_mem_mc_addr); | ||
| 213 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_69, psp_ring_reg); | ||
| 214 | /* Write high address of the ring to C2PMSG_70 */ | ||
| 215 | psp_ring_reg = upper_32_bits(ring->ring_mem_mc_addr); | ||
| 216 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_70, psp_ring_reg); | ||
| 217 | /* Write size of ring to C2PMSG_71 */ | ||
| 218 | psp_ring_reg = ring->ring_size; | ||
| 219 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_71, psp_ring_reg); | ||
| 220 | /* Write the ring initialization command to C2PMSG_64 */ | ||
| 221 | psp_ring_reg = ring_type; | ||
| 222 | psp_ring_reg = psp_ring_reg << 16; | ||
| 223 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, psp_ring_reg); | ||
| 224 | |||
| 225 | /* there might be handshake issue with hardware which needs delay */ | ||
| 226 | mdelay(20); | ||
| 227 | |||
| 228 | /* Wait for response flag (bit 31) in C2PMSG_64 */ | ||
| 229 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_64), | ||
| 230 | 0x80000000, 0x8000FFFF, false); | ||
| 231 | } | ||
| 232 | 167 | ||
| 233 | return ret; | 168 | return ret; |
| 234 | } | 169 | } |
| @@ -247,9 +182,6 @@ static int psp_v12_0_ring_stop(struct psp_context *psp, · 12 unmodified lines | |||
| 247 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, | 182 | WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_64, |
| 248 | GFX_CTRL_CMD_ID_DESTROY_RINGS); | 183 | GFX_CTRL_CMD_ID_DESTROY_RINGS); |
| 249 | 184 | ||
| 250 | /* there might be handshake issue with hardware which needs delay */ | ||
| 251 | mdelay(20); | ||
| 252 | |||
| 253 | /* Wait for response flag (bit 31) */ | 185 | /* Wait for response flag (bit 31) */ |
| 254 | if (amdgpu_sriov_vf(adev)) | 186 | if (amdgpu_sriov_vf(adev)) |
| 255 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101), | 187 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, mmMP0_SMN_C2PMSG_101), |
| @@ -85,6 +85,8 @@ MODULE_FIRMWARE("amdgpu/psp_14_0_4_ta.bin"); · 84 unmodified lines | |||
| 85 | 85 | ||
| 86 | #define regMP1_PUB_SCRATCH0 0x3b10090 | 86 | #define regMP1_PUB_SCRATCH0 0x3b10090 |
| 87 | 87 | ||
| 88 | #define PSP13_BL_STATUS_SIZE 100 | ||
| 89 | |||
| 88 | static int psp_v13_0_init_microcode(struct psp_context *psp) | 90 | static int psp_v13_0_init_microcode(struct psp_context *psp) |
| 89 | { | 91 | { |
| 90 | struct amdgpu_device *adev = psp->adev; | 92 | struct amdgpu_device *adev = psp->adev; |
| @@ -151,6 +153,32 @@ static bool psp_v13_0_is_sos_alive(struct psp_context *psp) · 60 unmodified lines | |||
| 151 | return sol_reg != 0x0; | 153 | return sol_reg != 0x0; |
| 152 | } | 154 | } |
| 153 | 155 | ||
| 156 | static void psp_v13_0_bootloader_print_status(struct psp_context *psp, | ||
| 157 | const char *msg) | ||
| 158 | { | ||
| 159 | struct amdgpu_device *adev = psp->adev; | ||
| 160 | u32 bl_status_reg; | ||
| 161 | char bl_status_msg[PSP13_BL_STATUS_SIZE]; | ||
| 162 | int i, at; | ||
| 163 | |||
| 164 | if (amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 6) || | ||
| 165 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 12) || | ||
| 166 | amdgpu_ip_version(adev, MP0_HWIP, 0) == IP_VERSION(13, 0, 14)) { | ||
| 167 | at = 0; | ||
| 168 | for_each_inst(i, adev->aid_mask) { | ||
| 169 | bl_status_reg = | ||
| 170 | (SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_92) | ||
| 171 | << 2) + | ||
| 172 | adev->asic_funcs->encode_ext_smn_addressing(i); | ||
| 173 | at += snprintf(bl_status_msg + at, | ||
| 174 | PSP13_BL_STATUS_SIZE - at, | ||
| 175 | " status(%02i): 0x%08x", i, | ||
| 176 | RREG32_PCIE_EXT(bl_status_reg)); | ||
| 177 | } | ||
| 178 | dev_info(adev->dev, "%s - %s", msg, bl_status_msg); | ||
| 179 | } | ||
| 180 | } | ||
| 181 | |||
| 154 | static int psp_v13_0_wait_for_vmbx_ready(struct psp_context *psp) | 182 | static int psp_v13_0_wait_for_vmbx_ready(struct psp_context *psp) |
| 155 | { | 183 | { |
| 156 | struct amdgpu_device *adev = psp->adev; | 184 | struct amdgpu_device *adev = psp->adev; |
| @@ -196,6 +224,9 @@ static int psp_v13_0_wait_for_bootloader(struct psp_context *psp) · 39 unmodified lines | |||
| 196 | 224 | ||
| 197 | if (ret == 0) | 225 | if (ret == 0) |
| 198 | return 0; | 226 | return 0; |
| 227 | if (retry_loop && !(retry_loop % 10)) | ||
| 228 | psp_v13_0_bootloader_print_status( | ||
| 229 | psp, "Waiting for bootloader completion"); | ||
| 199 | } | 230 | } |
| 200 | 231 | ||
| 201 | return ret; | 232 | return ret; |
| @@ -858,6 +889,25 @@ static bool psp_v13_0_is_reload_needed(struct psp_context *psp) · 656 unmodified lines | |||
| 858 | return false; | 889 | return false; |
| 859 | } | 890 | } |
| 860 | 891 | ||
| 892 | static int psp_v13_0_reg_program_no_ring(struct psp_context *psp, uint32_t val, | ||
| 893 | enum psp_reg_prog_id id) | ||
| 894 | { | ||
| 895 | struct amdgpu_device *adev = psp->adev; | ||
| 896 | int ret = -EOPNOTSUPP; | ||
| 897 | |||
| 898 | /* PSP will broadcast the value to all instances */ | ||
| 899 | if (amdgpu_sriov_vf(adev)) { | ||
| 900 | WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_101, GFX_CTRL_CMD_ID_GBR_IH_SET); | ||
| 901 | WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_102, id); | ||
| 902 | WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_103, val); | ||
| 903 | |||
| 904 | ret = psp_wait_for(psp, SOC15_REG_OFFSET(MP0, 0, regMP0_SMN_C2PMSG_101), | ||
| 905 | 0x80000000, 0x80000000, false); | ||
| 906 | } | ||
| 907 | |||
| 908 | return ret; | ||
| 909 | } | ||
| 910 | |||
| 861 | static const struct psp_funcs psp_v13_0_funcs = { | 911 | static const struct psp_funcs psp_v13_0_funcs = { |
| 862 | .init_microcode = psp_v13_0_init_microcode, | 912 | .init_microcode = psp_v13_0_init_microcode, |
| 863 | .wait_for_bootloader = psp_v13_0_wait_for_bootloader_steady_state, | 913 | .wait_for_bootloader = psp_v13_0_wait_for_bootloader_steady_state, |
| @@ -884,6 +934,7 @@ static const struct psp_funcs psp_v13_0_funcs = { · 20 unmodified lines | |||
| 884 | .get_ras_capability = psp_v13_0_get_ras_capability, | 934 | .get_ras_capability = psp_v13_0_get_ras_capability, |
| 885 | .is_aux_sos_load_required = psp_v13_0_is_aux_sos_load_required, | 935 | .is_aux_sos_load_required = psp_v13_0_is_aux_sos_load_required, |
| 886 | .is_reload_needed = psp_v13_0_is_reload_needed, | 936 | .is_reload_needed = psp_v13_0_is_reload_needed, |
| 937 | .reg_program_no_ring = psp_v13_0_reg_program_no_ring, | ||
| 887 | }; | 938 | }; |
| 888 | 939 | ||
| 889 | void psp_v13_0_set_psp_funcs(struct psp_context *psp) | 940 | void psp_v13_0_set_psp_funcs(struct psp_context *psp) |
| @@ -106,8 +106,9 @@ static void sdma_v4_4_2_set_buffer_funcs(struct amdgpu_device *adev); · 105 unmodified lines | |||
| 106 | static void sdma_v4_4_2_set_vm_pte_funcs(struct amdgpu_device *adev); | 106 | static void sdma_v4_4_2_set_vm_pte_funcs(struct amdgpu_device *adev); |
| 107 | static void sdma_v4_4_2_set_irq_funcs(struct amdgpu_device *adev); | 107 | static void sdma_v4_4_2_set_irq_funcs(struct amdgpu_device *adev); |
| 108 | static void sdma_v4_4_2_set_ras_funcs(struct amdgpu_device *adev); | 108 | static void sdma_v4_4_2_set_ras_funcs(struct amdgpu_device *adev); |
| 109 | static void sdma_v4_4_2_set_engine_reset_funcs(struct amdgpu_device *adev); | ||
| 110 | static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev); | 109 | static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev); |
| 110 | static int sdma_v4_4_2_stop_queue(struct amdgpu_ring *ring); | ||
| 111 | static int sdma_v4_4_2_restore_queue(struct amdgpu_ring *ring); | ||
| 111 | 112 | ||
| 112 | static u32 sdma_v4_4_2_get_reg_offset(struct amdgpu_device *adev, | 113 | static u32 sdma_v4_4_2_get_reg_offset(struct amdgpu_device *adev, |
| 113 | u32 instance, u32 offset) | 114 | u32 instance, u32 offset) |
| @@ -1333,6 +1334,11 @@ static bool sdma_v4_4_2_fw_support_paging_queue(struct amdgpu_device *adev) · 1219 unmodified lines | |||
| 1333 | } | 1334 | } |
| 1334 | } | 1335 | } |
| 1335 | 1336 | ||
| 1337 | static const struct amdgpu_sdma_funcs sdma_v4_4_2_sdma_funcs = { | ||
| 1338 | .stop_kernel_queue = &sdma_v4_4_2_stop_queue, | ||
| 1339 | .start_kernel_queue = &sdma_v4_4_2_restore_queue, | ||
| 1340 | }; | ||
| 1341 | |||
| 1336 | static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block) | 1342 | static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block) |
| 1337 | { | 1343 | { |
| 1338 | struct amdgpu_device *adev = ip_block->adev; | 1344 | struct amdgpu_device *adev = ip_block->adev; |
| @@ -1351,8 +1357,6 @@ static int sdma_v4_4_2_early_init(struct amdgpu_ip_block *ip_block) · 12 unmodified lines | |||
| 1351 | sdma_v4_4_2_set_vm_pte_funcs(adev); | 1357 | sdma_v4_4_2_set_vm_pte_funcs(adev); |
| 1352 | sdma_v4_4_2_set_irq_funcs(adev); | 1358 | sdma_v4_4_2_set_irq_funcs(adev); |
| 1353 | sdma_v4_4_2_set_ras_funcs(adev); | 1359 | sdma_v4_4_2_set_ras_funcs(adev); |
| 1354 | sdma_v4_4_2_set_engine_reset_funcs(adev); | ||
| 1355 | |||
| 1356 | return 0; | 1360 | return 0; |
| 1357 | } | 1361 | } |
| 1358 | 1362 | ||
| @@ -1447,6 +1451,7 @@ static int sdma_v4_4_2_sw_init(struct amdgpu_ip_block *ip_block) · 88 unmodified lines | |||
| 1447 | /* Initialize guilty flags for GFX and PAGE queues */ | 1451 | /* Initialize guilty flags for GFX and PAGE queues */ |
| 1448 | adev->sdma.instance[i].gfx_guilty = false; | 1452 | adev->sdma.instance[i].gfx_guilty = false; |
| 1449 | adev->sdma.instance[i].page_guilty = false; | 1453 | adev->sdma.instance[i].page_guilty = false; |
| 1454 | adev->sdma.instance[i].funcs = &sdma_v4_4_2_sdma_funcs; | ||
| 1450 | 1455 | ||
| 1451 | ring = &adev->sdma.instance[i].ring; | 1456 | ring = &adev->sdma.instance[i].ring; |
| 1452 | ring->ring_obj = NULL; | 1457 | ring->ring_obj = NULL; |
| @@ -1678,11 +1683,12 @@ static int sdma_v4_4_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 225 unmodified lines | |||
| 1678 | return r; | 1683 | return r; |
| 1679 | } | 1684 | } |
| 1680 | 1685 | ||
| 1681 | static int sdma_v4_4_2_stop_queue(struct amdgpu_device *adev, uint32_t instance_id) | 1686 | static int sdma_v4_4_2_stop_queue(struct amdgpu_ring *ring) |
| 1682 | { | 1687 | { |
| 1688 | struct amdgpu_device *adev = ring->adev; | ||
| 1689 | u32 instance_id = GET_INST(SDMA0, ring->me); | ||
| 1683 | u32 inst_mask; | 1690 | u32 inst_mask; |
| 1684 | uint64_t rptr; | 1691 | uint64_t rptr; |
| 1685 | struct amdgpu_ring *ring = &adev->sdma.instance[instance_id].ring; | ||
| 1686 | 1692 | ||
| 1687 | if (amdgpu_sriov_vf(adev)) | 1693 | if (amdgpu_sriov_vf(adev)) |
| 1688 | return -EINVAL; | 1694 | return -EINVAL; |
| @@ -1715,11 +1721,11 @@ static int sdma_v4_4_2_stop_queue(struct amdgpu_device *adev, uint32_t instance_ · 26 unmodified lines | |||
| 1715 | return 0; | 1721 | return 0; |
| 1716 | } | 1722 | } |
| 1717 | 1723 | ||
| 1718 | static int sdma_v4_4_2_restore_queue(struct amdgpu_device *adev, uint32_t instance_id) | 1724 | static int sdma_v4_4_2_restore_queue(struct amdgpu_ring *ring) |
| 1719 | { | 1725 | { |
| 1720 | int i; | 1726 | struct amdgpu_device *adev = ring->adev; |
| 1721 | u32 inst_mask; | 1727 | u32 inst_mask; |
| 1722 | struct amdgpu_ring *ring = &adev->sdma.instance[instance_id].ring; | 1728 | int i; |
| 1723 | 1729 | ||
| 1724 | inst_mask = 1 << ring->me; | 1730 | inst_mask = 1 << ring->me; |
| 1725 | udelay(50); | 1731 | udelay(50); |
| @@ -1739,16 +1745,6 @@ static int sdma_v4_4_2_restore_queue(struct amdgpu_device *adev, uint32_t instan · 13 unmodified lines | |||
| 1739 | return sdma_v4_4_2_inst_start(adev, inst_mask, true); | 1745 | return sdma_v4_4_2_inst_start(adev, inst_mask, true); |
| 1740 | } | 1746 | } |
| 1741 | 1747 | ||
| 1742 | static struct sdma_on_reset_funcs sdma_v4_4_2_engine_reset_funcs = { | ||
| 1743 | .pre_reset = sdma_v4_4_2_stop_queue, | ||
| 1744 | .post_reset = sdma_v4_4_2_restore_queue, | ||
| 1745 | }; | ||
| 1746 | |||
| 1747 | static void sdma_v4_4_2_set_engine_reset_funcs(struct amdgpu_device *adev) | ||
| 1748 | { | ||
| 1749 | amdgpu_sdma_register_on_reset_callbacks(adev, &sdma_v4_4_2_engine_reset_funcs); | ||
| 1750 | } | ||
| 1751 | |||
| 1752 | static int sdma_v4_4_2_set_trap_irq_state(struct amdgpu_device *adev, | 1748 | static int sdma_v4_4_2_set_trap_irq_state(struct amdgpu_device *adev, |
| 1753 | struct amdgpu_irq_src *source, | 1749 | struct amdgpu_irq_src *source, |
| 1754 | unsigned type, | 1750 | unsigned type, |
| @@ -2373,7 +2369,9 @@ static void sdma_v4_4_2_update_reset_mask(struct amdgpu_device *adev) · 618 unmodified lines | |||
| 2373 | adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE; | 2369 | adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE; |
| 2374 | break; | 2370 | break; |
| 2375 | case IP_VERSION(9, 5, 0): | 2371 | case IP_VERSION(9, 5, 0): |
| 2376 | /*TODO: enable the queue reset flag until fw supported */ | 2372 | if ((adev->gfx.mec_fw_version >= 0xf) && amdgpu_dpm_reset_sdma_is_supported(adev)) |
| 2373 | adev->sdma.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE; | ||
| 2374 | break; | ||
| 2377 | default: | 2375 | default: |
| 2378 | break; | 2376 | break; |
| 2379 | } | 2377 | } |
| @@ -112,6 +112,8 @@ static void sdma_v5_0_set_ring_funcs(struct amdgpu_device *adev); · 111 unmodified lines | |||
| 112 | static void sdma_v5_0_set_buffer_funcs(struct amdgpu_device *adev); | 112 | static void sdma_v5_0_set_buffer_funcs(struct amdgpu_device *adev); |
| 113 | static void sdma_v5_0_set_vm_pte_funcs(struct amdgpu_device *adev); | 113 | static void sdma_v5_0_set_vm_pte_funcs(struct amdgpu_device *adev); |
| 114 | static void sdma_v5_0_set_irq_funcs(struct amdgpu_device *adev); | 114 | static void sdma_v5_0_set_irq_funcs(struct amdgpu_device *adev); |
| 115 | static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring); | ||
| 116 | static int sdma_v5_0_restore_queue(struct amdgpu_ring *ring); | ||
| 115 | 117 | ||
| 116 | static const struct soc15_reg_golden golden_settings_sdma_5[] = { | 118 | static const struct soc15_reg_golden golden_settings_sdma_5[] = { |
| 117 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_CHICKEN_BITS, 0xffbf1f0f, 0x03ab0107), | 119 | SOC15_REG_GOLDEN_VALUE(GC, 0, mmSDMA0_CHICKEN_BITS, 0xffbf1f0f, 0x03ab0107), |
| @@ -369,67 +371,36 @@ static uint64_t sdma_v5_0_ring_get_wptr(struct amdgpu_ring *ring) · 251 unmodified lines | |||
| 369 | static void sdma_v5_0_ring_set_wptr(struct amdgpu_ring *ring) | 371 | static void sdma_v5_0_ring_set_wptr(struct amdgpu_ring *ring) |
| 370 | { | 372 | { |
| 371 | struct amdgpu_device *adev = ring->adev; | 373 | struct amdgpu_device *adev = ring->adev; |
| 372 | uint32_t *wptr_saved; | ||
| 373 | uint32_t *is_queue_unmap; | ||
| 374 | uint64_t aggregated_db_index; | ||
| 375 | uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_DMA].mqd_size; | ||
| 376 | 374 | ||
| 377 | DRM_DEBUG("Setting write pointer\n"); | 375 | DRM_DEBUG("Setting write pointer\n"); |
| 378 | if (ring->is_mes_queue) { | 376 | if (ring->use_doorbell) { |
| 379 | wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size); | 377 | DRM_DEBUG("Using doorbell -- " |
| 380 | is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size + | 378 | "wptr_offs == 0x%08x " |
| 381 | sizeof(uint32_t)); | 379 | "lower_32_bits(ring->wptr) << 2 == 0x%08x " |
| 382 | aggregated_db_index = | 380 | "upper_32_bits(ring->wptr) << 2 == 0x%08x\n", |
| 383 | amdgpu_mes_get_aggregated_doorbell_index(adev, | 381 | ring->wptr_offs, |
| 384 | AMDGPU_MES_PRIORITY_LEVEL_NORMAL); | 382 | lower_32_bits(ring->wptr << 2), |
| 385 | 383 | upper_32_bits(ring->wptr << 2)); | |
| 384 | /* XXX check if swapping is necessary on BE */ | ||
| 386 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, | 385 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, |
| 387 | ring->wptr << 2); | 386 | ring->wptr << 2); |
| 388 | *wptr_saved = ring->wptr << 2; | 387 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", |
| 389 | if (*is_queue_unmap) { | 388 | ring->doorbell_index, ring->wptr << 2); |
| 390 | WDOORBELL64(aggregated_db_index, ring->wptr << 2); | 389 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); |
| 391 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", | ||
| 392 | ring->doorbell_index, ring->wptr << 2); | ||
| 393 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); | ||
| 394 | } else { | ||
| 395 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", | ||
| 396 | ring->doorbell_index, ring->wptr << 2); | ||
| 397 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); | ||
| 398 | |||
| 399 | if (*is_queue_unmap) | ||
| 400 | WDOORBELL64(aggregated_db_index, | ||
| 401 | ring->wptr << 2); | ||
| 402 | } | ||
| 403 | } else { | 390 | } else { |
| 404 | if (ring->use_doorbell) { | 391 | DRM_DEBUG("Not using doorbell -- " |
| 405 | DRM_DEBUG("Using doorbell -- " | 392 | "mmSDMA%i_GFX_RB_WPTR == 0x%08x " |
| 406 | "wptr_offs == 0x%08x " | 393 | "mmSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n", |
| 407 | "lower_32_bits(ring->wptr) << 2 == 0x%08x " | 394 | ring->me, |
| 408 | "upper_32_bits(ring->wptr) << 2 == 0x%08x\n", | 395 | lower_32_bits(ring->wptr << 2), |
| 409 | ring->wptr_offs, | 396 | ring->me, |
| 410 | lower_32_bits(ring->wptr << 2), | 397 | upper_32_bits(ring->wptr << 2)); |
| 411 | upper_32_bits(ring->wptr << 2)); | 398 | WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, |
| 412 | /* XXX check if swapping is necessary on BE */ | 399 | ring->me, mmSDMA0_GFX_RB_WPTR), |
| 413 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, | 400 | lower_32_bits(ring->wptr << 2)); |
| 414 | ring->wptr << 2); | 401 | WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, |
| 415 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", | 402 | ring->me, mmSDMA0_GFX_RB_WPTR_HI), |
| 416 | ring->doorbell_index, ring->wptr << 2); | 403 | upper_32_bits(ring->wptr << 2)); |
| 417 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); | ||
| 418 | } else { | ||
| 419 | DRM_DEBUG("Not using doorbell -- " | ||
| 420 | "mmSDMA%i_GFX_RB_WPTR == 0x%08x " | ||
| 421 | "mmSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n", | ||
| 422 | ring->me, | ||
| 423 | lower_32_bits(ring->wptr << 2), | ||
| 424 | ring->me, | ||
| 425 | upper_32_bits(ring->wptr << 2)); | ||
| 426 | WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, | ||
| 427 | ring->me, mmSDMA0_GFX_RB_WPTR), | ||
| 428 | lower_32_bits(ring->wptr << 2)); | ||
| 429 | WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, | ||
| 430 | ring->me, mmSDMA0_GFX_RB_WPTR_HI), | ||
| 431 | upper_32_bits(ring->wptr << 2)); | ||
| 432 | } | ||
| 433 | } | 404 | } |
| 434 | } | 405 | } |
| 435 | 406 | ||
| @@ -575,11 +546,9 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 139 unmodified lines | |||
| 575 | } | 546 | } |
| 576 | 547 | ||
| 577 | if (flags & AMDGPU_FENCE_FLAG_INT) { | 548 | if (flags & AMDGPU_FENCE_FLAG_INT) { |
| 578 | uint32_t ctx = ring->is_mes_queue ? | ||
| 579 | (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0; | ||
| 580 | /* generate an interrupt */ | 549 | /* generate an interrupt */ |
| 581 | amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP)); | 550 | amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP)); |
| 582 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx)); | 551 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0)); |
| 583 | } | 552 | } |
| 584 | } | 553 | } |
| 585 | 554 | ||
| @@ -588,15 +557,15 @@ static void sdma_v5_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 2 unmodified lines | |||
| 588 | * sdma_v5_0_gfx_stop - stop the gfx async dma engines | 557 | * sdma_v5_0_gfx_stop - stop the gfx async dma engines |
| 589 | * | 558 | * |
| 590 | * @adev: amdgpu_device pointer | 559 | * @adev: amdgpu_device pointer |
| 591 | * | 560 | * @inst_mask: mask of dma engine instances to be disabled |
| 592 | * Stop the gfx async dma ring buffers (NAVI10). | 561 | * Stop the gfx async dma ring buffers (NAVI10). |
| 593 | */ | 562 | */ |
| 594 | static void sdma_v5_0_gfx_stop(struct amdgpu_device *adev) | 563 | static void sdma_v5_0_gfx_stop(struct amdgpu_device *adev, uint32_t inst_mask) |
| 595 | { | 564 | { |
| 596 | u32 rb_cntl, ib_cntl; | 565 | u32 rb_cntl, ib_cntl; |
| 597 | int i; | 566 | int i; |
| 598 | 567 | ||
| 599 | for (i = 0; i < adev->sdma.num_instances; i++) { | 568 | for_each_inst(i, inst_mask) { |
| 600 | rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL)); | 569 | rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL)); |
| 601 | rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0); | 570 | rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0); |
| 602 | WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl); | 571 | WREG32_SOC15_IP(GC, sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl); |
| @@ -688,9 +657,11 @@ static void sdma_v5_0_enable(struct amdgpu_device *adev, bool enable) · 85 unmodified lines | |||
| 688 | { | 657 | { |
| 689 | u32 f32_cntl; | 658 | u32 f32_cntl; |
| 690 | int i; | 659 | int i; |
| 660 | uint32_t inst_mask; | ||
| 691 | 661 | ||
| 662 | inst_mask = GENMASK(adev->sdma.num_instances - 1, 0); | ||
| 692 | if (!enable) { | 663 | if (!enable) { |
| 693 | sdma_v5_0_gfx_stop(adev); | 664 | sdma_v5_0_gfx_stop(adev, 1 << inst_mask); |
| 694 | sdma_v5_0_rlc_stop(adev); | 665 | sdma_v5_0_rlc_stop(adev); |
| 695 | } | 666 | } |
| 696 | 667 | ||
| @@ -1046,33 +1017,22 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 349 unmodified lines | |||
| 1046 | int r; | 1017 | int r; |
| 1047 | u32 tmp; | 1018 | u32 tmp; |
| 1048 | u64 gpu_addr; | 1019 | u64 gpu_addr; |
| 1049 | volatile uint32_t *cpu_ptr = NULL; | ||
| 1050 | 1020 | ||
| 1051 | tmp = 0xCAFEDEAD; | 1021 | tmp = 0xCAFEDEAD; |
| 1052 | 1022 | ||
| 1053 | if (ring->is_mes_queue) { | 1023 | r = amdgpu_device_wb_get(adev, &index); |
| 1054 | uint32_t offset = 0; | 1024 | if (r) { |
| 1055 | offset = amdgpu_mes_ctx_get_offs(ring, | 1025 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); |
| 1056 | AMDGPU_MES_CTX_PADDING_OFFS); | 1026 | return r; |
| 1057 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 1058 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 1059 | *cpu_ptr = tmp; | ||
| 1060 | } else { | ||
| 1061 | r = amdgpu_device_wb_get(adev, &index); | ||
| 1062 | if (r) { | ||
| 1063 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); | ||
| 1064 | return r; | ||
| 1065 | } | ||
| 1066 | |||
| 1067 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 1068 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 1069 | } | 1027 | } |
| 1070 | 1028 | ||
| 1029 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 1030 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 1031 | |||
| 1071 | r = amdgpu_ring_alloc(ring, 20); | 1032 | r = amdgpu_ring_alloc(ring, 20); |
| 1072 | if (r) { | 1033 | if (r) { |
| 1073 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); | 1034 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); |
| 1074 | if (!ring->is_mes_queue) | 1035 | amdgpu_device_wb_free(adev, index); |
| 1075 | amdgpu_device_wb_free(adev, index); | ||
| 1076 | return r; | 1036 | return r; |
| 1077 | } | 1037 | } |
| 1078 | 1038 | ||
| @@ -1085,10 +1045,7 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines | |||
| 1085 | amdgpu_ring_commit(ring); | 1045 | amdgpu_ring_commit(ring); |
| 1086 | 1046 | ||
| 1087 | for (i = 0; i < adev->usec_timeout; i++) { | 1047 | for (i = 0; i < adev->usec_timeout; i++) { |
| 1088 | if (ring->is_mes_queue) | 1048 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 1089 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 1090 | else | ||
| 1091 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 1092 | if (tmp == 0xDEADBEEF) | 1049 | if (tmp == 0xDEADBEEF) |
| 1093 | break; | 1050 | break; |
| 1094 | if (amdgpu_emu_mode == 1) | 1051 | if (amdgpu_emu_mode == 1) |
| @@ -1100,8 +1057,7 @@ static int sdma_v5_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines | |||
| 1100 | if (i >= adev->usec_timeout) | 1057 | if (i >= adev->usec_timeout) |
| 1101 | r = -ETIMEDOUT; | 1058 | r = -ETIMEDOUT; |
| 1102 | 1059 | ||
| 1103 | if (!ring->is_mes_queue) | 1060 | amdgpu_device_wb_free(adev, index); |
| 1104 | amdgpu_device_wb_free(adev, index); | ||
| 1105 | 1061 | ||
| 1106 | return r; | 1062 | return r; |
| 1107 | } | 1063 | } |
| @@ -1124,38 +1080,24 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines | |||
| 1124 | long r; | 1080 | long r; |
| 1125 | u32 tmp = 0; | 1081 | u32 tmp = 0; |
| 1126 | u64 gpu_addr; | 1082 | u64 gpu_addr; |
| 1127 | volatile uint32_t *cpu_ptr = NULL; | ||
| 1128 | 1083 | ||
| 1129 | tmp = 0xCAFEDEAD; | 1084 | tmp = 0xCAFEDEAD; |
| 1130 | memset(&ib, 0, sizeof(ib)); | 1085 | memset(&ib, 0, sizeof(ib)); |
| 1131 | 1086 | ||
| 1132 | if (ring->is_mes_queue) { | 1087 | r = amdgpu_device_wb_get(adev, &index); |
| 1133 | uint32_t offset = 0; | 1088 | if (r) { |
| 1134 | offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS); | 1089 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); |
| 1135 | ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | 1090 | return r; |
| 1136 | ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | 1091 | } |
| 1137 | |||
| 1138 | offset = amdgpu_mes_ctx_get_offs(ring, | ||
| 1139 | AMDGPU_MES_CTX_PADDING_OFFS); | ||
| 1140 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 1141 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 1142 | *cpu_ptr = tmp; | ||
| 1143 | } else { | ||
| 1144 | r = amdgpu_device_wb_get(adev, &index); | ||
| 1145 | if (r) { | ||
| 1146 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); | ||
| 1147 | return r; | ||
| 1148 | } | ||
| 1149 | 1092 | ||
| 1150 | gpu_addr = adev->wb.gpu_addr + (index * 4); | 1093 | gpu_addr = adev->wb.gpu_addr + (index * 4); |
| 1151 | adev->wb.wb[index] = cpu_to_le32(tmp); | 1094 | adev->wb.wb[index] = cpu_to_le32(tmp); |
| 1152 | 1095 | ||
| 1153 | r = amdgpu_ib_get(adev, NULL, 256, | 1096 | r = amdgpu_ib_get(adev, NULL, 256, |
| 1154 | AMDGPU_IB_POOL_DIRECT, &ib); | 1097 | AMDGPU_IB_POOL_DIRECT, &ib); |
| 1155 | if (r) { | 1098 | if (r) { |
| 1156 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); | 1099 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); |
| 1157 | goto err0; | 1100 | goto err0; |
| 1158 | } | ||
| 1159 | } | 1101 | } |
| 1160 | 1102 | ||
| 1161 | ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) | | 1103 | ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) | |
| @@ -1183,10 +1125,7 @@ static int sdma_v5_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines | |||
| 1183 | goto err1; | 1125 | goto err1; |
| 1184 | } | 1126 | } |
| 1185 | 1127 | ||
| 1186 | if (ring->is_mes_queue) | 1128 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 1187 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 1188 | else | ||
| 1189 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 1190 | 1129 | ||
| 1191 | if (tmp == 0xDEADBEEF) | 1130 | if (tmp == 0xDEADBEEF) |
| 1192 | r = 0; | 1131 | r = 0; |
| @@ -1197,8 +1136,7 @@ err1: · 4 unmodified lines | |||
| 1197 | amdgpu_ib_free(&ib, NULL); | 1136 | amdgpu_ib_free(&ib, NULL); |
| 1198 | dma_fence_put(f); | 1137 | dma_fence_put(f); |
| 1199 | err0: | 1138 | err0: |
| 1200 | if (!ring->is_mes_queue) | 1139 | amdgpu_device_wb_free(adev, index); |
| 1201 | amdgpu_device_wb_free(adev, index); | ||
| 1202 | return r; | 1140 | return r; |
| 1203 | } | 1141 | } |
| 1204 | 1142 | ||
| @@ -1389,6 +1327,36 @@ static void sdma_v5_0_ring_emit_reg_write_reg_wait(struct amdgpu_ring *ring, · 184 unmodified lines | |||
| 1389 | amdgpu_ring_emit_reg_wait(ring, reg1, mask, mask); | 1327 | amdgpu_ring_emit_reg_wait(ring, reg1, mask, mask); |
| 1390 | } | 1328 | } |
| 1391 | 1329 | ||
| 1330 | static int sdma_v5_0_soft_reset_engine(struct amdgpu_device *adev, u32 instance_id) | ||
| 1331 | { | ||
| 1332 | u32 grbm_soft_reset; | ||
| 1333 | u32 tmp; | ||
| 1334 | |||
| 1335 | grbm_soft_reset = REG_SET_FIELD(0, | ||
| 1336 | GRBM_SOFT_RESET, SOFT_RESET_SDMA0, | ||
| 1337 | 1); | ||
| 1338 | grbm_soft_reset <<= instance_id; | ||
| 1339 | |||
| 1340 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | ||
| 1341 | tmp |= grbm_soft_reset; | ||
| 1342 | DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp); | ||
| 1343 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp); | ||
| 1344 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | ||
| 1345 | |||
| 1346 | udelay(50); | ||
| 1347 | |||
| 1348 | tmp &= ~grbm_soft_reset; | ||
| 1349 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp); | ||
| 1350 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | ||
| 1351 | return 0; | ||
| 1352 | } | ||
| 1353 | |||
| 1354 | static const struct amdgpu_sdma_funcs sdma_v5_0_sdma_funcs = { | ||
| 1355 | .stop_kernel_queue = &sdma_v5_0_stop_queue, | ||
| 1356 | .start_kernel_queue = &sdma_v5_0_restore_queue, | ||
| 1357 | .soft_reset_kernel_queue = &sdma_v5_0_soft_reset_engine, | ||
| 1358 | }; | ||
| 1359 | |||
| 1392 | static int sdma_v5_0_early_init(struct amdgpu_ip_block *ip_block) | 1360 | static int sdma_v5_0_early_init(struct amdgpu_ip_block *ip_block) |
| 1393 | { | 1361 | { |
| 1394 | struct amdgpu_device *adev = ip_block->adev; | 1362 | struct amdgpu_device *adev = ip_block->adev; |
| @@ -1431,6 +1399,7 @@ static int sdma_v5_0_sw_init(struct amdgpu_ip_block *ip_block) · 36 unmodified lines | |||
| 1431 | return r; | 1399 | return r; |
| 1432 | 1400 | ||
| 1433 | for (i = 0; i < adev->sdma.num_instances; i++) { | 1401 | for (i = 0; i < adev->sdma.num_instances; i++) { |
| 1402 | adev->sdma.instance[i].funcs = &sdma_v5_0_sdma_funcs; | ||
| 1434 | ring = &adev->sdma.instance[i].ring; | 1403 | ring = &adev->sdma.instance[i].ring; |
| 1435 | ring->ring_obj = NULL; | 1404 | ring->ring_obj = NULL; |
| 1436 | ring->use_doorbell = true; | 1405 | ring->use_doorbell = true; |
| @@ -1572,32 +1541,25 @@ static int sdma_v5_0_soft_reset(struct amdgpu_ip_block *ip_block) · 135 unmodified lines | |||
| 1572 | static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) | 1541 | static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) |
| 1573 | { | 1542 | { |
| 1574 | struct amdgpu_device *adev = ring->adev; | 1543 | struct amdgpu_device *adev = ring->adev; |
| 1575 | int i, j, r; | 1544 | u32 inst_id = ring->me; |
| 1576 | u32 rb_cntl, ib_cntl, f32_cntl, freeze, cntl, preempt, soft_reset, stat1_reg; | ||
| 1577 | 1545 | ||
| 1578 | if (amdgpu_sriov_vf(adev)) | 1546 | return amdgpu_sdma_reset_engine(adev, inst_id); |
| 1579 | return -EINVAL; | 1547 | } |
| 1580 | 1548 | ||
| 1581 | for (i = 0; i < adev->sdma.num_instances; i++) { | 1549 | static int sdma_v5_0_stop_queue(struct amdgpu_ring *ring) |
| 1582 | if (ring == &adev->sdma.instance[i].ring) | 1550 | { |
| 1583 | break; | 1551 | u32 f32_cntl, freeze, cntl, stat1_reg; |
| 1584 | } | 1552 | struct amdgpu_device *adev = ring->adev; |
| 1553 | int i, j, r = 0; | ||
| 1585 | 1554 | ||
| 1586 | if (i == adev->sdma.num_instances) { | 1555 | if (amdgpu_sriov_vf(adev)) |
| 1587 | DRM_ERROR("sdma instance not found\n"); | ||
| 1588 | return -EINVAL; | 1556 | return -EINVAL; |
| 1589 | } | ||
| 1590 | 1557 | ||
| 1558 | i = ring->me; | ||
| 1591 | amdgpu_gfx_rlc_enter_safe_mode(adev, 0); | 1559 | amdgpu_gfx_rlc_enter_safe_mode(adev, 0); |
| 1592 | 1560 | ||
| 1593 | /* stop queue */ | 1561 | /* stop queue */ |
| 1594 | ib_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL)); | 1562 | sdma_v5_0_gfx_stop(adev, 1 << i); |
| 1595 | ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0); | ||
| 1596 | WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl); | ||
| 1597 | |||
| 1598 | rb_cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL)); | ||
| 1599 | rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0); | ||
| 1600 | WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl); | ||
| 1601 | 1563 | ||
| 1602 | /* engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */ | 1564 | /* engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */ |
| 1603 | freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE)); | 1565 | freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE)); |
| @@ -1628,30 +1590,25 @@ static int sdma_v5_0_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 24 unmodified lines | |||
| 1628 | cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL)); | 1590 | cntl = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL)); |
| 1629 | cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0); | 1591 | cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0); |
| 1630 | WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl); | 1592 | WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl); |
| 1593 | err0: | ||
| 1594 | amdgpu_gfx_rlc_exit_safe_mode(adev, 0); | ||
| 1595 | return r; | ||
| 1596 | } | ||
| 1631 | 1597 | ||
| 1632 | /* soft reset SDMA_GFX_PREEMPT.IB_PREEMPT = 0 mmGRBM_SOFT_RESET.SOFT_RESET_SDMA0/1 = 1 */ | 1598 | static int sdma_v5_0_restore_queue(struct amdgpu_ring *ring) |
| 1633 | preempt = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT)); | 1599 | { |
| 1634 | preempt = REG_SET_FIELD(preempt, SDMA0_GFX_PREEMPT, IB_PREEMPT, 0); | 1600 | struct amdgpu_device *adev = ring->adev; |
| 1635 | WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT), preempt); | 1601 | u32 inst_id = ring->me; |
| 1636 | 1602 | u32 freeze; | |
| 1637 | soft_reset = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | 1603 | int r; |
| 1638 | soft_reset |= 1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i; | ||
| 1639 | |||
| 1640 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset); | ||
| 1641 | |||
| 1642 | udelay(50); | ||
| 1643 | |||
| 1644 | soft_reset &= ~(1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i); | ||
| 1645 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset); | ||
| 1646 | 1604 | ||
| 1605 | amdgpu_gfx_rlc_enter_safe_mode(adev, 0); | ||
| 1647 | /* unfreeze*/ | 1606 | /* unfreeze*/ |
| 1648 | freeze = RREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE)); | 1607 | freeze = RREG32(sdma_v5_0_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE)); |
| 1649 | freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0); | 1608 | freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0); |
| 1650 | WREG32(sdma_v5_0_get_reg_offset(adev, i, mmSDMA0_FREEZE), freeze); | 1609 | WREG32(sdma_v5_0_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE), freeze); |
| 1651 | 1610 | ||
| 1652 | r = sdma_v5_0_gfx_resume_instance(adev, i, true); | 1611 | r = sdma_v5_0_gfx_resume_instance(adev, inst_id, true); |
| 1653 | |||
| 1654 | err0: | ||
| 1655 | amdgpu_gfx_rlc_exit_safe_mode(adev, 0); | 1612 | amdgpu_gfx_rlc_exit_safe_mode(adev, 0); |
| 1656 | return r; | 1613 | return r; |
| 1657 | } | 1614 | } |
| @@ -113,6 +113,8 @@ static void sdma_v5_2_set_ring_funcs(struct amdgpu_device *adev); · 112 unmodified lines | |||
| 113 | static void sdma_v5_2_set_buffer_funcs(struct amdgpu_device *adev); | 113 | static void sdma_v5_2_set_buffer_funcs(struct amdgpu_device *adev); |
| 114 | static void sdma_v5_2_set_vm_pte_funcs(struct amdgpu_device *adev); | 114 | static void sdma_v5_2_set_vm_pte_funcs(struct amdgpu_device *adev); |
| 115 | static void sdma_v5_2_set_irq_funcs(struct amdgpu_device *adev); | 115 | static void sdma_v5_2_set_irq_funcs(struct amdgpu_device *adev); |
| 116 | static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring); | ||
| 117 | static int sdma_v5_2_restore_queue(struct amdgpu_ring *ring); | ||
| 116 | 118 | ||
| 117 | static u32 sdma_v5_2_get_reg_offset(struct amdgpu_device *adev, u32 instance, u32 internal_offset) | 119 | static u32 sdma_v5_2_get_reg_offset(struct amdgpu_device *adev, u32 instance, u32 internal_offset) |
| 118 | { | 120 | { |
| @@ -394,11 +396,9 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 275 unmodified lines | |||
| 394 | } | 396 | } |
| 395 | 397 | ||
| 396 | if ((flags & AMDGPU_FENCE_FLAG_INT)) { | 398 | if ((flags & AMDGPU_FENCE_FLAG_INT)) { |
| 397 | uint32_t ctx = ring->is_mes_queue ? | ||
| 398 | (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0; | ||
| 399 | /* generate an interrupt */ | 399 | /* generate an interrupt */ |
| 400 | amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP)); | 400 | amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_TRAP)); |
| 401 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx)); | 401 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0)); |
| 402 | } | 402 | } |
| 403 | } | 403 | } |
| 404 | 404 | ||
| @@ -407,15 +407,15 @@ static void sdma_v5_2_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 2 unmodified lines | |||
| 407 | * sdma_v5_2_gfx_stop - stop the gfx async dma engines | 407 | * sdma_v5_2_gfx_stop - stop the gfx async dma engines |
| 408 | * | 408 | * |
| 409 | * @adev: amdgpu_device pointer | 409 | * @adev: amdgpu_device pointer |
| 410 | * | 410 | * @inst_mask: mask of dma engine instances to be disabled |
| 411 | * Stop the gfx async dma ring buffers. | 411 | * Stop the gfx async dma ring buffers. |
| 412 | */ | 412 | */ |
| 413 | static void sdma_v5_2_gfx_stop(struct amdgpu_device *adev) | 413 | static void sdma_v5_2_gfx_stop(struct amdgpu_device *adev, uint32_t inst_mask) |
| 414 | { | 414 | { |
| 415 | u32 rb_cntl, ib_cntl; | 415 | u32 rb_cntl, ib_cntl; |
| 416 | int i; | 416 | int i; |
| 417 | 417 | ||
| 418 | for (i = 0; i < adev->sdma.num_instances; i++) { | 418 | for_each_inst(i, inst_mask) { |
| 419 | rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL)); | 419 | rb_cntl = RREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL)); |
| 420 | rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0); | 420 | rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0); |
| 421 | WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl); | 421 | WREG32_SOC15_IP(GC, sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl); |
| @@ -506,9 +506,11 @@ static void sdma_v5_2_enable(struct amdgpu_device *adev, bool enable) · 84 unmodified lines | |||
| 506 | { | 506 | { |
| 507 | u32 f32_cntl; | 507 | u32 f32_cntl; |
| 508 | int i; | 508 | int i; |
| 509 | uint32_t inst_mask; | ||
| 509 | 510 | ||
| 511 | inst_mask = GENMASK(adev->sdma.num_instances - 1, 0); | ||
| 510 | if (!enable) { | 512 | if (!enable) { |
| 511 | sdma_v5_2_gfx_stop(adev); | 513 | sdma_v5_2_gfx_stop(adev, inst_mask); |
| 512 | sdma_v5_2_rlc_stop(adev); | 514 | sdma_v5_2_rlc_stop(adev); |
| 513 | } | 515 | } |
| 514 | 516 | ||
| @@ -761,37 +763,49 @@ static int sdma_v5_2_load_microcode(struct amdgpu_device *adev) · 246 unmodified lines | |||
| 761 | return 0; | 763 | return 0; |
| 762 | } | 764 | } |
| 763 | 765 | ||
| 764 | static int sdma_v5_2_soft_reset(struct amdgpu_ip_block *ip_block) | 766 | static int sdma_v5_2_soft_reset_engine(struct amdgpu_device *adev, u32 instance_id) |
| 765 | { | 767 | { |
| 766 | struct amdgpu_device *adev = ip_block->adev; | ||
| 767 | u32 grbm_soft_reset; | 768 | u32 grbm_soft_reset; |
| 768 | u32 tmp; | 769 | u32 tmp; |
| 769 | int i; | ||
| 770 | 770 | ||
| 771 | for (i = 0; i < adev->sdma.num_instances; i++) { | 771 | grbm_soft_reset = REG_SET_FIELD(0, |
| 772 | grbm_soft_reset = REG_SET_FIELD(0, | 772 | GRBM_SOFT_RESET, SOFT_RESET_SDMA0, |
| 773 | GRBM_SOFT_RESET, SOFT_RESET_SDMA0, | 773 | 1); |
| 774 | 1); | 774 | grbm_soft_reset <<= instance_id; |
| 775 | grbm_soft_reset <<= i; | ||
| 776 | 775 | ||
| 777 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | 776 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); |
| 778 | tmp |= grbm_soft_reset; | 777 | tmp |= grbm_soft_reset; |
| 779 | DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp); | 778 | DRM_DEBUG("GRBM_SOFT_RESET=0x%08X\n", tmp); |
| 780 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp); | 779 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp); |
| 781 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | 780 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); |
| 782 | 781 | ||
| 783 | udelay(50); | 782 | udelay(50); |
| 783 | |||
| 784 | tmp &= ~grbm_soft_reset; | ||
| 785 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp); | ||
| 786 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | ||
| 787 | return 0; | ||
| 788 | } | ||
| 784 | 789 | ||
| 785 | tmp &= ~grbm_soft_reset; | 790 | static int sdma_v5_2_soft_reset(struct amdgpu_ip_block *ip_block) |
| 786 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, tmp); | 791 | { |
| 787 | tmp = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | 792 | struct amdgpu_device *adev = ip_block->adev; |
| 793 | int i; | ||
| 788 | 794 | ||
| 795 | for (i = 0; i < adev->sdma.num_instances; i++) { | ||
| 796 | sdma_v5_2_soft_reset_engine(adev, i); | ||
| 789 | udelay(50); | 797 | udelay(50); |
| 790 | } | 798 | } |
| 791 | 799 | ||
| 792 | return 0; | 800 | return 0; |
| 793 | } | 801 | } |
| 794 | 802 | ||
| 803 | static const struct amdgpu_sdma_funcs sdma_v5_2_sdma_funcs = { | ||
| 804 | .stop_kernel_queue = &sdma_v5_2_stop_queue, | ||
| 805 | .start_kernel_queue = &sdma_v5_2_restore_queue, | ||
| 806 | .soft_reset_kernel_queue = &sdma_v5_2_soft_reset_engine, | ||
| 807 | }; | ||
| 808 | |||
| 795 | /** | 809 | /** |
| 796 | * sdma_v5_2_start - setup and start the async dma engines | 810 | * sdma_v5_2_start - setup and start the async dma engines |
| 797 | * | 811 | * |
| @@ -903,33 +917,22 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 105 unmodified lines | |||
| 903 | int r; | 917 | int r; |
| 904 | u32 tmp; | 918 | u32 tmp; |
| 905 | u64 gpu_addr; | 919 | u64 gpu_addr; |
| 906 | volatile uint32_t *cpu_ptr = NULL; | ||
| 907 | 920 | ||
| 908 | tmp = 0xCAFEDEAD; | 921 | tmp = 0xCAFEDEAD; |
| 909 | 922 | ||
| 910 | if (ring->is_mes_queue) { | 923 | r = amdgpu_device_wb_get(adev, &index); |
| 911 | uint32_t offset = 0; | 924 | if (r) { |
| 912 | offset = amdgpu_mes_ctx_get_offs(ring, | 925 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); |
| 913 | AMDGPU_MES_CTX_PADDING_OFFS); | 926 | return r; |
| 914 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 915 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 916 | *cpu_ptr = tmp; | ||
| 917 | } else { | ||
| 918 | r = amdgpu_device_wb_get(adev, &index); | ||
| 919 | if (r) { | ||
| 920 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); | ||
| 921 | return r; | ||
| 922 | } | ||
| 923 | |||
| 924 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 925 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 926 | } | 927 | } |
| 927 | 928 | ||
| 929 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 930 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 931 | |||
| 928 | r = amdgpu_ring_alloc(ring, 20); | 932 | r = amdgpu_ring_alloc(ring, 20); |
| 929 | if (r) { | 933 | if (r) { |
| 930 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); | 934 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); |
| 931 | if (!ring->is_mes_queue) | 935 | amdgpu_device_wb_free(adev, index); |
| 932 | amdgpu_device_wb_free(adev, index); | ||
| 933 | return r; | 936 | return r; |
| 934 | } | 937 | } |
| 935 | 938 | ||
| @@ -942,10 +945,7 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines | |||
| 942 | amdgpu_ring_commit(ring); | 945 | amdgpu_ring_commit(ring); |
| 943 | 946 | ||
| 944 | for (i = 0; i < adev->usec_timeout; i++) { | 947 | for (i = 0; i < adev->usec_timeout; i++) { |
| 945 | if (ring->is_mes_queue) | 948 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 946 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 947 | else | ||
| 948 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 949 | if (tmp == 0xDEADBEEF) | 949 | if (tmp == 0xDEADBEEF) |
| 950 | break; | 950 | break; |
| 951 | if (amdgpu_emu_mode == 1) | 951 | if (amdgpu_emu_mode == 1) |
| @@ -957,8 +957,7 @@ static int sdma_v5_2_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines | |||
| 957 | if (i >= adev->usec_timeout) | 957 | if (i >= adev->usec_timeout) |
| 958 | r = -ETIMEDOUT; | 958 | r = -ETIMEDOUT; |
| 959 | 959 | ||
| 960 | if (!ring->is_mes_queue) | 960 | amdgpu_device_wb_free(adev, index); |
| 961 | amdgpu_device_wb_free(adev, index); | ||
| 962 | 961 | ||
| 963 | return r; | 962 | return r; |
| 964 | } | 963 | } |
| @@ -981,37 +980,23 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines | |||
| 981 | long r; | 980 | long r; |
| 982 | u32 tmp = 0; | 981 | u32 tmp = 0; |
| 983 | u64 gpu_addr; | 982 | u64 gpu_addr; |
| 984 | volatile uint32_t *cpu_ptr = NULL; | ||
| 985 | 983 | ||
| 986 | tmp = 0xCAFEDEAD; | 984 | tmp = 0xCAFEDEAD; |
| 987 | memset(&ib, 0, sizeof(ib)); | 985 | memset(&ib, 0, sizeof(ib)); |
| 988 | 986 | ||
| 989 | if (ring->is_mes_queue) { | 987 | r = amdgpu_device_wb_get(adev, &index); |
| 990 | uint32_t offset = 0; | 988 | if (r) { |
| 991 | offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS); | 989 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); |
| 992 | ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | 990 | return r; |
| 993 | ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | 991 | } |
| 994 | |||
| 995 | offset = amdgpu_mes_ctx_get_offs(ring, | ||
| 996 | AMDGPU_MES_CTX_PADDING_OFFS); | ||
| 997 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 998 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 999 | *cpu_ptr = tmp; | ||
| 1000 | } else { | ||
| 1001 | r = amdgpu_device_wb_get(adev, &index); | ||
| 1002 | if (r) { | ||
| 1003 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); | ||
| 1004 | return r; | ||
| 1005 | } | ||
| 1006 | 992 | ||
| 1007 | gpu_addr = adev->wb.gpu_addr + (index * 4); | 993 | gpu_addr = adev->wb.gpu_addr + (index * 4); |
| 1008 | adev->wb.wb[index] = cpu_to_le32(tmp); | 994 | adev->wb.wb[index] = cpu_to_le32(tmp); |
| 1009 | 995 | ||
| 1010 | r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib); | 996 | r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib); |
| 1011 | if (r) { | 997 | if (r) { |
| 1012 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); | 998 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); |
| 1013 | goto err0; | 999 | goto err0; |
| 1014 | } | ||
| 1015 | } | 1000 | } |
| 1016 | 1001 | ||
| 1017 | ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) | | 1002 | ib.ptr[0] = SDMA_PKT_HEADER_OP(SDMA_OP_WRITE) | |
| @@ -1039,10 +1024,7 @@ static int sdma_v5_2_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines | |||
| 1039 | goto err1; | 1024 | goto err1; |
| 1040 | } | 1025 | } |
| 1041 | 1026 | ||
| 1042 | if (ring->is_mes_queue) | 1027 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 1043 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 1044 | else | ||
| 1045 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 1046 | 1028 | ||
| 1047 | if (tmp == 0xDEADBEEF) | 1029 | if (tmp == 0xDEADBEEF) |
| 1048 | r = 0; | 1030 | r = 0; |
| @@ -1053,8 +1035,7 @@ err1: · 4 unmodified lines | |||
| 1053 | amdgpu_ib_free(&ib, NULL); | 1035 | amdgpu_ib_free(&ib, NULL); |
| 1054 | dma_fence_put(f); | 1036 | dma_fence_put(f); |
| 1055 | err0: | 1037 | err0: |
| 1056 | if (!ring->is_mes_queue) | 1038 | amdgpu_device_wb_free(adev, index); |
| 1057 | amdgpu_device_wb_free(adev, index); | ||
| 1058 | return r; | 1039 | return r; |
| 1059 | } | 1040 | } |
| 1060 | 1041 | ||
| @@ -1337,6 +1318,7 @@ static int sdma_v5_2_sw_init(struct amdgpu_ip_block *ip_block) · 276 unmodified lines | |||
| 1337 | } | 1318 | } |
| 1338 | 1319 | ||
| 1339 | for (i = 0; i < adev->sdma.num_instances; i++) { | 1320 | for (i = 0; i < adev->sdma.num_instances; i++) { |
| 1321 | adev->sdma.instance[i].funcs = &sdma_v5_2_sdma_funcs; | ||
| 1340 | ring = &adev->sdma.instance[i].ring; | 1322 | ring = &adev->sdma.instance[i].ring; |
| 1341 | ring->ring_obj = NULL; | 1323 | ring->ring_obj = NULL; |
| 1342 | ring->use_doorbell = true; | 1324 | ring->use_doorbell = true; |
| @@ -1472,32 +1454,25 @@ static int sdma_v5_2_wait_for_idle(struct amdgpu_ip_block *ip_block) · 129 unmodified lines | |||
| 1472 | static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) | 1454 | static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) |
| 1473 | { | 1455 | { |
| 1474 | struct amdgpu_device *adev = ring->adev; | 1456 | struct amdgpu_device *adev = ring->adev; |
| 1475 | int i, j, r; | 1457 | u32 inst_id = ring->me; |
| 1476 | u32 rb_cntl, ib_cntl, f32_cntl, freeze, cntl, preempt, soft_reset, stat1_reg; | ||
| 1477 | 1458 | ||
| 1478 | if (amdgpu_sriov_vf(adev)) | 1459 | return amdgpu_sdma_reset_engine(adev, inst_id); |
| 1479 | return -EINVAL; | 1460 | } |
| 1480 | 1461 | ||
| 1481 | for (i = 0; i < adev->sdma.num_instances; i++) { | 1462 | static int sdma_v5_2_stop_queue(struct amdgpu_ring *ring) |
| 1482 | if (ring == &adev->sdma.instance[i].ring) | 1463 | { |
| 1483 | break; | 1464 | u32 f32_cntl, freeze, cntl, stat1_reg; |
| 1484 | } | 1465 | struct amdgpu_device *adev = ring->adev; |
| 1466 | int i, j, r = 0; | ||
| 1485 | 1467 | ||
| 1486 | if (i == adev->sdma.num_instances) { | 1468 | if (amdgpu_sriov_vf(adev)) |
| 1487 | DRM_ERROR("sdma instance not found\n"); | ||
| 1488 | return -EINVAL; | 1469 | return -EINVAL; |
| 1489 | } | ||
| 1490 | 1470 | ||
| 1471 | i = ring->me; | ||
| 1491 | amdgpu_gfx_rlc_enter_safe_mode(adev, 0); | 1472 | amdgpu_gfx_rlc_enter_safe_mode(adev, 0); |
| 1492 | 1473 | ||
| 1493 | /* stop queue */ | 1474 | /* stop queue */ |
| 1494 | ib_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL)); | 1475 | sdma_v5_2_gfx_stop(adev, 1 << i); |
| 1495 | ib_cntl = REG_SET_FIELD(ib_cntl, SDMA0_GFX_IB_CNTL, IB_ENABLE, 0); | ||
| 1496 | WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_IB_CNTL), ib_cntl); | ||
| 1497 | |||
| 1498 | rb_cntl = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL)); | ||
| 1499 | rb_cntl = REG_SET_FIELD(rb_cntl, SDMA0_GFX_RB_CNTL, RB_ENABLE, 0); | ||
| 1500 | WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_RB_CNTL), rb_cntl); | ||
| 1501 | 1476 | ||
| 1502 | /*engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */ | 1477 | /*engine stop SDMA1_F32_CNTL.HALT to 1 and SDMAx_FREEZE freeze bit to 1 */ |
| 1503 | freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE)); | 1478 | freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE)); |
| @@ -1530,31 +1505,26 @@ static int sdma_v5_2_reset_queue(struct amdgpu_ring *ring, unsigned int vmid) · 26 unmodified lines | |||
| 1530 | cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0); | 1505 | cntl = REG_SET_FIELD(cntl, SDMA0_CNTL, UTC_L1_ENABLE, 0); |
| 1531 | WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl); | 1506 | WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_CNTL), cntl); |
| 1532 | 1507 | ||
| 1533 | /* soft reset SDMA_GFX_PREEMPT.IB_PREEMPT = 0 mmGRBM_SOFT_RESET.SOFT_RESET_SDMA0/1 = 1 */ | 1508 | err0: |
| 1534 | preempt = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT)); | 1509 | amdgpu_gfx_rlc_exit_safe_mode(adev, 0); |
| 1535 | preempt = REG_SET_FIELD(preempt, SDMA0_GFX_PREEMPT, IB_PREEMPT, 0); | 1510 | return r; |
| 1536 | WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_GFX_PREEMPT), preempt); | 1511 | } |
| 1537 | |||
| 1538 | soft_reset = RREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET); | ||
| 1539 | soft_reset |= 1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i; | ||
| 1540 | |||
| 1541 | |||
| 1542 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset); | ||
| 1543 | |||
| 1544 | udelay(50); | ||
| 1545 | |||
| 1546 | soft_reset &= ~(1 << GRBM_SOFT_RESET__SOFT_RESET_SDMA0__SHIFT << i); | ||
| 1547 | 1512 | ||
| 1548 | WREG32_SOC15(GC, 0, mmGRBM_SOFT_RESET, soft_reset); | 1513 | static int sdma_v5_2_restore_queue(struct amdgpu_ring *ring) |
| 1514 | { | ||
| 1515 | struct amdgpu_device *adev = ring->adev; | ||
| 1516 | u32 inst_id = ring->me; | ||
| 1517 | u32 freeze; | ||
| 1518 | int r; | ||
| 1549 | 1519 | ||
| 1520 | amdgpu_gfx_rlc_enter_safe_mode(adev, 0); | ||
| 1550 | /* unfreeze and unhalt */ | 1521 | /* unfreeze and unhalt */ |
| 1551 | freeze = RREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE)); | 1522 | freeze = RREG32(sdma_v5_2_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE)); |
| 1552 | freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0); | 1523 | freeze = REG_SET_FIELD(freeze, SDMA0_FREEZE, FREEZE, 0); |
| 1553 | WREG32(sdma_v5_2_get_reg_offset(adev, i, mmSDMA0_FREEZE), freeze); | 1524 | WREG32(sdma_v5_2_get_reg_offset(adev, inst_id, mmSDMA0_FREEZE), freeze); |
| 1554 | 1525 | ||
| 1555 | r = sdma_v5_2_gfx_resume_instance(adev, i, true); | 1526 | r = sdma_v5_2_gfx_resume_instance(adev, inst_id, true); |
| 1556 | 1527 | ||
| 1557 | err0: | ||
| 1558 | amdgpu_gfx_rlc_exit_safe_mode(adev, 0); | 1528 | amdgpu_gfx_rlc_exit_safe_mode(adev, 0); |
| 1559 | return r; | 1529 | return r; |
| 1560 | } | 1530 | } |
| @@ -43,6 +43,7 @@ · 42 unmodified lines | |||
| 43 | #include "sdma_common.h" | 43 | #include "sdma_common.h" |
| 44 | #include "sdma_v6_0.h" | 44 | #include "sdma_v6_0.h" |
| 45 | #include "v11_structs.h" | 45 | #include "v11_structs.h" |
| 46 | #include "mes_userqueue.h" | ||
| 46 | 47 | ||
| 47 | MODULE_FIRMWARE("amdgpu/sdma_6_0_0.bin"); | 48 | MODULE_FIRMWARE("amdgpu/sdma_6_0_0.bin"); |
| 48 | MODULE_FIRMWARE("amdgpu/sdma_6_0_1.bin"); | 49 | MODULE_FIRMWARE("amdgpu/sdma_6_0_1.bin"); |
| @@ -376,11 +377,9 @@ static void sdma_v6_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 327 unmodified lines | |||
| 376 | } | 377 | } |
| 377 | 378 | ||
| 378 | if (flags & AMDGPU_FENCE_FLAG_INT) { | 379 | if (flags & AMDGPU_FENCE_FLAG_INT) { |
| 379 | uint32_t ctx = ring->is_mes_queue ? | ||
| 380 | (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0; | ||
| 381 | /* generate an interrupt */ | 380 | /* generate an interrupt */ |
| 382 | amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP)); | 381 | amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP)); |
| 383 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx)); | 382 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0)); |
| 384 | } | 383 | } |
| 385 | } | 384 | } |
| 386 | 385 | ||
| @@ -891,6 +890,9 @@ static int sdma_v6_0_mqd_init(struct amdgpu_device *adev, void *mqd, · 504 unmodified lines | |||
| 891 | m->sdmax_rlcx_rb_aql_cntl = regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT; | 890 | m->sdmax_rlcx_rb_aql_cntl = regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT; |
| 892 | m->sdmax_rlcx_dummy_reg = regSDMA0_QUEUE0_DUMMY_REG_DEFAULT; | 891 | m->sdmax_rlcx_dummy_reg = regSDMA0_QUEUE0_DUMMY_REG_DEFAULT; |
| 893 | 892 | ||
| 893 | m->sdmax_rlcx_csa_addr_lo = lower_32_bits(prop->csa_addr); | ||
| 894 | m->sdmax_rlcx_csa_addr_hi = upper_32_bits(prop->csa_addr); | ||
| 895 | |||
| 894 | return 0; | 896 | return 0; |
| 895 | } | 897 | } |
| 896 | 898 | ||
| @@ -917,33 +919,22 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 20 unmodified lines | |||
| 917 | int r; | 919 | int r; |
| 918 | u32 tmp; | 920 | u32 tmp; |
| 919 | u64 gpu_addr; | 921 | u64 gpu_addr; |
| 920 | volatile uint32_t *cpu_ptr = NULL; | ||
| 921 | 922 | ||
| 922 | tmp = 0xCAFEDEAD; | 923 | tmp = 0xCAFEDEAD; |
| 923 | 924 | ||
| 924 | if (ring->is_mes_queue) { | 925 | r = amdgpu_device_wb_get(adev, &index); |
| 925 | uint32_t offset = 0; | 926 | if (r) { |
| 926 | offset = amdgpu_mes_ctx_get_offs(ring, | 927 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); |
| 927 | AMDGPU_MES_CTX_PADDING_OFFS); | 928 | return r; |
| 928 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 929 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 930 | *cpu_ptr = tmp; | ||
| 931 | } else { | ||
| 932 | r = amdgpu_device_wb_get(adev, &index); | ||
| 933 | if (r) { | ||
| 934 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); | ||
| 935 | return r; | ||
| 936 | } | ||
| 937 | |||
| 938 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 939 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 940 | } | 929 | } |
| 941 | 930 | ||
| 931 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 932 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 933 | |||
| 942 | r = amdgpu_ring_alloc(ring, 5); | 934 | r = amdgpu_ring_alloc(ring, 5); |
| 943 | if (r) { | 935 | if (r) { |
| 944 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); | 936 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); |
| 945 | if (!ring->is_mes_queue) | 937 | amdgpu_device_wb_free(adev, index); |
| 946 | amdgpu_device_wb_free(adev, index); | ||
| 947 | return r; | 938 | return r; |
| 948 | } | 939 | } |
| 949 | 940 | ||
| @@ -956,10 +947,7 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines | |||
| 956 | amdgpu_ring_commit(ring); | 947 | amdgpu_ring_commit(ring); |
| 957 | 948 | ||
| 958 | for (i = 0; i < adev->usec_timeout; i++) { | 949 | for (i = 0; i < adev->usec_timeout; i++) { |
| 959 | if (ring->is_mes_queue) | 950 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 960 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 961 | else | ||
| 962 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 963 | if (tmp == 0xDEADBEEF) | 951 | if (tmp == 0xDEADBEEF) |
| 964 | break; | 952 | break; |
| 965 | if (amdgpu_emu_mode == 1) | 953 | if (amdgpu_emu_mode == 1) |
| @@ -971,8 +959,7 @@ static int sdma_v6_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines | |||
| 971 | if (i >= adev->usec_timeout) | 959 | if (i >= adev->usec_timeout) |
| 972 | r = -ETIMEDOUT; | 960 | r = -ETIMEDOUT; |
| 973 | 961 | ||
| 974 | if (!ring->is_mes_queue) | 962 | amdgpu_device_wb_free(adev, index); |
| 975 | amdgpu_device_wb_free(adev, index); | ||
| 976 | 963 | ||
| 977 | return r; | 964 | return r; |
| 978 | } | 965 | } |
| @@ -995,37 +982,23 @@ static int sdma_v6_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines | |||
| 995 | long r; | 982 | long r; |
| 996 | u32 tmp = 0; | 983 | u32 tmp = 0; |
| 997 | u64 gpu_addr; | 984 | u64 gpu_addr; |
| 998 | volatile uint32_t *cpu_ptr = NULL; | ||
| 999 | 985 | ||
| 1000 | tmp = 0xCAFEDEAD; | 986 | tmp = 0xCAFEDEAD; |
| 1001 | memset(&ib, 0, sizeof(ib)); | 987 | memset(&ib, 0, sizeof(ib)); |
| 1002 | 988 | ||
| 1003 | if (ring->is_mes_queue) { | 989 | r = amdgpu_device_wb_get(adev, &index); |
| 1004 | uint32_t offset = 0; | 990 | if (r) { |
| 1005 | offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS); | 991 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); |
| 1006 | ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | 992 | return r; |
| 1007 | ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | 993 | } |
| 1008 | |||
| 1009 | offset = amdgpu_mes_ctx_get_offs(ring, | ||
| 1010 | AMDGPU_MES_CTX_PADDING_OFFS); | ||
| 1011 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 1012 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 1013 | *cpu_ptr = tmp; | ||
| 1014 | } else { | ||
| 1015 | r = amdgpu_device_wb_get(adev, &index); | ||
| 1016 | if (r) { | ||
| 1017 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); | ||
| 1018 | return r; | ||
| 1019 | } | ||
| 1020 | 994 | ||
| 1021 | gpu_addr = adev->wb.gpu_addr + (index * 4); | 995 | gpu_addr = adev->wb.gpu_addr + (index * 4); |
| 1022 | adev->wb.wb[index] = cpu_to_le32(tmp); | 996 | adev->wb.wb[index] = cpu_to_le32(tmp); |
| 1023 | 997 | ||
| 1024 | r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib); | 998 | r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib); |
| 1025 | if (r) { | 999 | if (r) { |
| 1026 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); | 1000 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); |
| 1027 | goto err0; | 1001 | goto err0; |
| 1028 | } | ||
| 1029 | } | 1002 | } |
| 1030 | 1003 | ||
| 1031 | ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) | | 1004 | ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) | |
| @@ -1053,10 +1026,7 @@ static int sdma_v6_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines | |||
| 1053 | goto err1; | 1026 | goto err1; |
| 1054 | } | 1027 | } |
| 1055 | 1028 | ||
| 1056 | if (ring->is_mes_queue) | 1029 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 1057 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 1058 | else | ||
| 1059 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 1060 | 1030 | ||
| 1061 | if (tmp == 0xDEADBEEF) | 1031 | if (tmp == 0xDEADBEEF) |
| 1062 | r = 0; | 1032 | r = 0; |
| @@ -1067,8 +1037,7 @@ err1: · 4 unmodified lines | |||
| 1067 | amdgpu_ib_free(&ib, NULL); | 1037 | amdgpu_ib_free(&ib, NULL); |
| 1068 | dma_fence_put(f); | 1038 | dma_fence_put(f); |
| 1069 | err0: | 1039 | err0: |
| 1070 | if (!ring->is_mes_queue) | 1040 | amdgpu_device_wb_free(adev, index); |
| 1071 | amdgpu_device_wb_free(adev, index); | ||
| 1072 | return r; | 1041 | return r; |
| 1073 | } | 1042 | } |
| 1074 | 1043 | ||
| @@ -1300,6 +1269,23 @@ static int sdma_v6_0_early_init(struct amdgpu_ip_block *ip_block) · 225 unmodified lines | |||
| 1300 | struct amdgpu_device *adev = ip_block->adev; | 1269 | struct amdgpu_device *adev = ip_block->adev; |
| 1301 | int r; | 1270 | int r; |
| 1302 | 1271 | ||
| 1272 | switch (amdgpu_user_queue) { | ||
| 1273 | case -1: | ||
| 1274 | case 0: | ||
| 1275 | default: | ||
| 1276 | adev->sdma.no_user_submission = false; | ||
| 1277 | adev->sdma.disable_uq = true; | ||
| 1278 | break; | ||
| 1279 | case 1: | ||
| 1280 | adev->sdma.no_user_submission = false; | ||
| 1281 | adev->sdma.disable_uq = false; | ||
| 1282 | break; | ||
| 1283 | case 2: | ||
| 1284 | adev->sdma.no_user_submission = true; | ||
| 1285 | adev->sdma.disable_uq = false; | ||
| 1286 | break; | ||
| 1287 | } | ||
| 1288 | |||
| 1303 | r = amdgpu_sdma_init_microcode(adev, 0, true); | 1289 | r = amdgpu_sdma_init_microcode(adev, 0, true); |
| 1304 | if (r) | 1290 | if (r) |
| 1305 | return r; | 1291 | return r; |
| @@ -1334,6 +1320,7 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 28 unmodified lines | |||
| 1334 | ring->ring_obj = NULL; | 1320 | ring->ring_obj = NULL; |
| 1335 | ring->use_doorbell = true; | 1321 | ring->use_doorbell = true; |
| 1336 | ring->me = i; | 1322 | ring->me = i; |
| 1323 | ring->no_user_submission = adev->sdma.no_user_submission; | ||
| 1337 | 1324 | ||
| 1338 | DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i, | 1325 | DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i, |
| 1339 | ring->use_doorbell?"true":"false"); | 1326 | ring->use_doorbell?"true":"false"); |
| @@ -1376,6 +1363,10 @@ static int sdma_v6_0_sw_init(struct amdgpu_ip_block *ip_block) · 36 unmodified lines | |||
| 1376 | else | 1363 | else |
| 1377 | DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n"); | 1364 | DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n"); |
| 1378 | 1365 | ||
| 1366 | /* add firmware version checks here */ | ||
| 1367 | if (0 && !adev->sdma.disable_uq) | ||
| 1368 | adev->userq_funcs[AMDGPU_HW_IP_DMA] = &userq_mes_funcs; | ||
| 1369 | |||
| 1379 | r = amdgpu_sdma_sysfs_reset_mask_init(adev); | 1370 | r = amdgpu_sdma_sysfs_reset_mask_init(adev); |
| 1380 | if (r) | 1371 | if (r) |
| 1381 | return r; | 1372 | return r; |
| @@ -1399,11 +1390,39 @@ static int sdma_v6_0_sw_fini(struct amdgpu_ip_block *ip_block) · 17 unmodified lines | |||
| 1399 | return 0; | 1390 | return 0; |
| 1400 | } | 1391 | } |
| 1401 | 1392 | ||
| 1393 | static int sdma_v6_0_set_userq_trap_interrupts(struct amdgpu_device *adev, | ||
| 1394 | bool enable) | ||
| 1395 | { | ||
| 1396 | unsigned int irq_type; | ||
| 1397 | int i, r; | ||
| 1398 | |||
| 1399 | if (adev->userq_funcs[AMDGPU_HW_IP_DMA]) { | ||
| 1400 | for (i = 0; i < adev->sdma.num_instances; i++) { | ||
| 1401 | irq_type = AMDGPU_SDMA_IRQ_INSTANCE0 + i; | ||
| 1402 | if (enable) | ||
| 1403 | r = amdgpu_irq_get(adev, &adev->sdma.trap_irq, | ||
| 1404 | irq_type); | ||
| 1405 | else | ||
| 1406 | r = amdgpu_irq_put(adev, &adev->sdma.trap_irq, | ||
| 1407 | irq_type); | ||
| 1408 | if (r) | ||
| 1409 | return r; | ||
| 1410 | } | ||
| 1411 | } | ||
| 1412 | |||
| 1413 | return 0; | ||
| 1414 | } | ||
| 1415 | |||
| 1402 | static int sdma_v6_0_hw_init(struct amdgpu_ip_block *ip_block) | 1416 | static int sdma_v6_0_hw_init(struct amdgpu_ip_block *ip_block) |
| 1403 | { | 1417 | { |
| 1404 | struct amdgpu_device *adev = ip_block->adev; | 1418 | struct amdgpu_device *adev = ip_block->adev; |
| 1419 | int r; | ||
| 1405 | 1420 | ||
| 1406 | return sdma_v6_0_start(adev); | 1421 | r = sdma_v6_0_start(adev); |
| 1422 | if (r) | ||
| 1423 | return r; | ||
| 1424 | |||
| 1425 | return sdma_v6_0_set_userq_trap_interrupts(adev, true); | ||
| 1407 | } | 1426 | } |
| 1408 | 1427 | ||
| 1409 | static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block) | 1428 | static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block) |
| @@ -1415,6 +1434,7 @@ static int sdma_v6_0_hw_fini(struct amdgpu_ip_block *ip_block) · 5 unmodified lines | |||
| 1415 | 1434 | ||
| 1416 | sdma_v6_0_ctxempty_int_enable(adev, false); | 1435 | sdma_v6_0_ctxempty_int_enable(adev, false); |
| 1417 | sdma_v6_0_enable(adev, false); | 1436 | sdma_v6_0_enable(adev, false); |
| 1437 | sdma_v6_0_set_userq_trap_interrupts(adev, false); | ||
| 1418 | 1438 | ||
| 1419 | return 0; | 1439 | return 0; |
| 1420 | } | 1440 | } |
| @@ -42,6 +42,7 @@ · 41 unmodified lines | |||
| 42 | #include "sdma_common.h" | 42 | #include "sdma_common.h" |
| 43 | #include "sdma_v7_0.h" | 43 | #include "sdma_v7_0.h" |
| 44 | #include "v12_structs.h" | 44 | #include "v12_structs.h" |
| 45 | #include "mes_userqueue.h" | ||
| 45 | 46 | ||
| 46 | MODULE_FIRMWARE("amdgpu/sdma_7_0_0.bin"); | 47 | MODULE_FIRMWARE("amdgpu/sdma_7_0_0.bin"); |
| 47 | MODULE_FIRMWARE("amdgpu/sdma_7_0_1.bin"); | 48 | MODULE_FIRMWARE("amdgpu/sdma_7_0_1.bin"); |
| @@ -204,66 +205,39 @@ static uint64_t sdma_v7_0_ring_get_wptr(struct amdgpu_ring *ring) · 156 unmodified lines | |||
| 204 | static void sdma_v7_0_ring_set_wptr(struct amdgpu_ring *ring) | 205 | static void sdma_v7_0_ring_set_wptr(struct amdgpu_ring *ring) |
| 205 | { | 206 | { |
| 206 | struct amdgpu_device *adev = ring->adev; | 207 | struct amdgpu_device *adev = ring->adev; |
| 207 | uint32_t *wptr_saved; | ||
| 208 | uint32_t *is_queue_unmap; | ||
| 209 | uint64_t aggregated_db_index; | ||
| 210 | uint32_t mqd_size = adev->mqds[AMDGPU_HW_IP_DMA].mqd_size; | ||
| 211 | 208 | ||
| 212 | DRM_DEBUG("Setting write pointer\n"); | 209 | DRM_DEBUG("Setting write pointer\n"); |
| 213 | 210 | ||
| 214 | if (ring->is_mes_queue) { | 211 | if (ring->use_doorbell) { |
| 215 | wptr_saved = (uint32_t *)(ring->mqd_ptr + mqd_size); | 212 | DRM_DEBUG("Using doorbell -- " |
| 216 | is_queue_unmap = (uint32_t *)(ring->mqd_ptr + mqd_size + | 213 | "wptr_offs == 0x%08x " |
| 217 | sizeof(uint32_t)); | 214 | "lower_32_bits(ring->wptr) << 2 == 0x%08x " |
| 218 | aggregated_db_index = | 215 | "upper_32_bits(ring->wptr) << 2 == 0x%08x\n", |
| 219 | amdgpu_mes_get_aggregated_doorbell_index(adev, | 216 | ring->wptr_offs, |
| 220 | ring->hw_prio); | 217 | lower_32_bits(ring->wptr << 2), |
| 221 | 218 | upper_32_bits(ring->wptr << 2)); | |
| 219 | /* XXX check if swapping is necessary on BE */ | ||
| 222 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, | 220 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, |
| 223 | ring->wptr << 2); | 221 | ring->wptr << 2); |
| 224 | *wptr_saved = ring->wptr << 2; | 222 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", |
| 225 | if (*is_queue_unmap) { | 223 | ring->doorbell_index, ring->wptr << 2); |
| 226 | WDOORBELL64(aggregated_db_index, ring->wptr << 2); | 224 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); |
| 227 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", | ||
| 228 | ring->doorbell_index, ring->wptr << 2); | ||
| 229 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); | ||
| 230 | } else { | ||
| 231 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", | ||
| 232 | ring->doorbell_index, ring->wptr << 2); | ||
| 233 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); | ||
| 234 | } | ||
| 235 | } else { | 225 | } else { |
| 236 | if (ring->use_doorbell) { | 226 | DRM_DEBUG("Not using doorbell -- " |
| 237 | DRM_DEBUG("Using doorbell -- " | 227 | "regSDMA%i_GFX_RB_WPTR == 0x%08x " |
| 238 | "wptr_offs == 0x%08x " | 228 | "regSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n", |
| 239 | "lower_32_bits(ring->wptr) << 2 == 0x%08x " | 229 | ring->me, |
| 240 | "upper_32_bits(ring->wptr) << 2 == 0x%08x\n", | 230 | lower_32_bits(ring->wptr << 2), |
| 241 | ring->wptr_offs, | 231 | ring->me, |
| 242 | lower_32_bits(ring->wptr << 2), | 232 | upper_32_bits(ring->wptr << 2)); |
| 243 | upper_32_bits(ring->wptr << 2)); | 233 | WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev, |
| 244 | /* XXX check if swapping is necessary on BE */ | 234 | ring->me, |
| 245 | atomic64_set((atomic64_t *)ring->wptr_cpu_addr, | 235 | regSDMA0_QUEUE0_RB_WPTR), |
| 246 | ring->wptr << 2); | 236 | lower_32_bits(ring->wptr << 2)); |
| 247 | DRM_DEBUG("calling WDOORBELL64(0x%08x, 0x%016llx)\n", | 237 | WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev, |
| 248 | ring->doorbell_index, ring->wptr << 2); | 238 | ring->me, |
| 249 | WDOORBELL64(ring->doorbell_index, ring->wptr << 2); | 239 | regSDMA0_QUEUE0_RB_WPTR_HI), |
| 250 | } else { | 240 | upper_32_bits(ring->wptr << 2)); |
| 251 | DRM_DEBUG("Not using doorbell -- " | ||
| 252 | "regSDMA%i_GFX_RB_WPTR == 0x%08x " | ||
| 253 | "regSDMA%i_GFX_RB_WPTR_HI == 0x%08x\n", | ||
| 254 | ring->me, | ||
| 255 | lower_32_bits(ring->wptr << 2), | ||
| 256 | ring->me, | ||
| 257 | upper_32_bits(ring->wptr << 2)); | ||
| 258 | WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev, | ||
| 259 | ring->me, | ||
| 260 | regSDMA0_QUEUE0_RB_WPTR), | ||
| 261 | lower_32_bits(ring->wptr << 2)); | ||
| 262 | WREG32_SOC15_IP(GC, sdma_v7_0_get_reg_offset(adev, | ||
| 263 | ring->me, | ||
| 264 | regSDMA0_QUEUE0_RB_WPTR_HI), | ||
| 265 | upper_32_bits(ring->wptr << 2)); | ||
| 266 | } | ||
| 267 | } | 241 | } |
| 268 | } | 242 | } |
| 269 | 243 | ||
| @@ -407,11 +381,9 @@ static void sdma_v7_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se · 137 unmodified lines | |||
| 407 | } | 381 | } |
| 408 | 382 | ||
| 409 | if (flags & AMDGPU_FENCE_FLAG_INT) { | 383 | if (flags & AMDGPU_FENCE_FLAG_INT) { |
| 410 | uint32_t ctx = ring->is_mes_queue ? | ||
| 411 | (ring->hw_queue_id | AMDGPU_FENCE_MES_QUEUE_FLAG) : 0; | ||
| 412 | /* generate an interrupt */ | 384 | /* generate an interrupt */ |
| 413 | amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP)); | 385 | amdgpu_ring_write(ring, SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_TRAP)); |
| 414 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(ctx)); | 386 | amdgpu_ring_write(ring, SDMA_PKT_TRAP_INT_CONTEXT_INT_CONTEXT(0)); |
| 415 | } | 387 | } |
| 416 | } | 388 | } |
| 417 | 389 | ||
| @@ -935,6 +907,9 @@ static int sdma_v7_0_mqd_init(struct amdgpu_device *adev, void *mqd, · 517 unmodified lines | |||
| 935 | m->sdmax_rlcx_rb_aql_cntl = 0x4000; //regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT; | 907 | m->sdmax_rlcx_rb_aql_cntl = 0x4000; //regSDMA0_QUEUE0_RB_AQL_CNTL_DEFAULT; |
| 936 | m->sdmax_rlcx_dummy_reg = 0xf; //regSDMA0_QUEUE0_DUMMY_REG_DEFAULT; | 908 | m->sdmax_rlcx_dummy_reg = 0xf; //regSDMA0_QUEUE0_DUMMY_REG_DEFAULT; |
| 937 | 909 | ||
| 910 | m->sdmax_rlcx_csa_addr_lo = lower_32_bits(prop->csa_addr); | ||
| 911 | m->sdmax_rlcx_csa_addr_hi = upper_32_bits(prop->csa_addr); | ||
| 912 | |||
| 938 | return 0; | 913 | return 0; |
| 939 | } | 914 | } |
| 940 | 915 | ||
| @@ -961,33 +936,22 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 20 unmodified lines | |||
| 961 | int r; | 936 | int r; |
| 962 | u32 tmp; | 937 | u32 tmp; |
| 963 | u64 gpu_addr; | 938 | u64 gpu_addr; |
| 964 | volatile uint32_t *cpu_ptr = NULL; | ||
| 965 | 939 | ||
| 966 | tmp = 0xCAFEDEAD; | 940 | tmp = 0xCAFEDEAD; |
| 967 | 941 | ||
| 968 | if (ring->is_mes_queue) { | 942 | r = amdgpu_device_wb_get(adev, &index); |
| 969 | uint32_t offset = 0; | 943 | if (r) { |
| 970 | offset = amdgpu_mes_ctx_get_offs(ring, | 944 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); |
| 971 | AMDGPU_MES_CTX_PADDING_OFFS); | 945 | return r; |
| 972 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 973 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 974 | *cpu_ptr = tmp; | ||
| 975 | } else { | ||
| 976 | r = amdgpu_device_wb_get(adev, &index); | ||
| 977 | if (r) { | ||
| 978 | dev_err(adev->dev, "(%d) failed to allocate wb slot\n", r); | ||
| 979 | return r; | ||
| 980 | } | ||
| 981 | |||
| 982 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 983 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 984 | } | 946 | } |
| 985 | 947 | ||
| 948 | gpu_addr = adev->wb.gpu_addr + (index * 4); | ||
| 949 | adev->wb.wb[index] = cpu_to_le32(tmp); | ||
| 950 | |||
| 986 | r = amdgpu_ring_alloc(ring, 5); | 951 | r = amdgpu_ring_alloc(ring, 5); |
| 987 | if (r) { | 952 | if (r) { |
| 988 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); | 953 | DRM_ERROR("amdgpu: dma failed to lock ring %d (%d).\n", ring->idx, r); |
| 989 | if (!ring->is_mes_queue) | 954 | amdgpu_device_wb_free(adev, index); |
| 990 | amdgpu_device_wb_free(adev, index); | ||
| 991 | return r; | 955 | return r; |
| 992 | } | 956 | } |
| 993 | 957 | ||
| @@ -1000,10 +964,7 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 6 unmodified lines | |||
| 1000 | amdgpu_ring_commit(ring); | 964 | amdgpu_ring_commit(ring); |
| 1001 | 965 | ||
| 1002 | for (i = 0; i < adev->usec_timeout; i++) { | 966 | for (i = 0; i < adev->usec_timeout; i++) { |
| 1003 | if (ring->is_mes_queue) | 967 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 1004 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 1005 | else | ||
| 1006 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 1007 | if (tmp == 0xDEADBEEF) | 968 | if (tmp == 0xDEADBEEF) |
| 1008 | break; | 969 | break; |
| 1009 | if (amdgpu_emu_mode == 1) | 970 | if (amdgpu_emu_mode == 1) |
| @@ -1015,8 +976,7 @@ static int sdma_v7_0_ring_test_ring(struct amdgpu_ring *ring) · 5 unmodified lines | |||
| 1015 | if (i >= adev->usec_timeout) | 976 | if (i >= adev->usec_timeout) |
| 1016 | r = -ETIMEDOUT; | 977 | r = -ETIMEDOUT; |
| 1017 | 978 | ||
| 1018 | if (!ring->is_mes_queue) | 979 | amdgpu_device_wb_free(adev, index); |
| 1019 | amdgpu_device_wb_free(adev, index); | ||
| 1020 | 980 | ||
| 1021 | return r; | 981 | return r; |
| 1022 | } | 982 | } |
| @@ -1039,37 +999,23 @@ static int sdma_v7_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 16 unmodified lines | |||
| 1039 | long r; | 999 | long r; |
| 1040 | u32 tmp = 0; | 1000 | u32 tmp = 0; |
| 1041 | u64 gpu_addr; | 1001 | u64 gpu_addr; |
| 1042 | volatile uint32_t *cpu_ptr = NULL; | ||
| 1043 | 1002 | ||
| 1044 | tmp = 0xCAFEDEAD; | 1003 | tmp = 0xCAFEDEAD; |
| 1045 | memset(&ib, 0, sizeof(ib)); | 1004 | memset(&ib, 0, sizeof(ib)); |
| 1046 | 1005 | ||
| 1047 | if (ring->is_mes_queue) { | 1006 | r = amdgpu_device_wb_get(adev, &index); |
| 1048 | uint32_t offset = 0; | 1007 | if (r) { |
| 1049 | offset = amdgpu_mes_ctx_get_offs(ring, AMDGPU_MES_CTX_IB_OFFS); | 1008 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); |
| 1050 | ib.gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | 1009 | return r; |
| 1051 | ib.ptr = (void *)amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | 1010 | } |
| 1052 | |||
| 1053 | offset = amdgpu_mes_ctx_get_offs(ring, | ||
| 1054 | AMDGPU_MES_CTX_PADDING_OFFS); | ||
| 1055 | gpu_addr = amdgpu_mes_ctx_get_offs_gpu_addr(ring, offset); | ||
| 1056 | cpu_ptr = amdgpu_mes_ctx_get_offs_cpu_addr(ring, offset); | ||
| 1057 | *cpu_ptr = tmp; | ||
| 1058 | } else { | ||
| 1059 | r = amdgpu_device_wb_get(adev, &index); | ||
| 1060 | if (r) { | ||
| 1061 | dev_err(adev->dev, "(%ld) failed to allocate wb slot\n", r); | ||
| 1062 | return r; | ||
| 1063 | } | ||
| 1064 | 1011 | ||
| 1065 | gpu_addr = adev->wb.gpu_addr + (index * 4); | 1012 | gpu_addr = adev->wb.gpu_addr + (index * 4); |
| 1066 | adev->wb.wb[index] = cpu_to_le32(tmp); | 1013 | adev->wb.wb[index] = cpu_to_le32(tmp); |
| 1067 | 1014 | ||
| 1068 | r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib); | 1015 | r = amdgpu_ib_get(adev, NULL, 256, AMDGPU_IB_POOL_DIRECT, &ib); |
| 1069 | if (r) { | 1016 | if (r) { |
| 1070 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); | 1017 | DRM_ERROR("amdgpu: failed to get ib (%ld).\n", r); |
| 1071 | goto err0; | 1018 | goto err0; |
| 1072 | } | ||
| 1073 | } | 1019 | } |
| 1074 | 1020 | ||
| 1075 | ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) | | 1021 | ib.ptr[0] = SDMA_PKT_COPY_LINEAR_HEADER_OP(SDMA_OP_WRITE) | |
| @@ -1097,10 +1043,7 @@ static int sdma_v7_0_ring_test_ib(struct amdgpu_ring *ring, long timeout) · 21 unmodified lines | |||
| 1097 | goto err1; | 1043 | goto err1; |
| 1098 | } | 1044 | } |
| 1099 | 1045 | ||
| 1100 | if (ring->is_mes_queue) | 1046 | tmp = le32_to_cpu(adev->wb.wb[index]); |
| 1101 | tmp = le32_to_cpu(*cpu_ptr); | ||
| 1102 | else | ||
| 1103 | tmp = le32_to_cpu(adev->wb.wb[index]); | ||
| 1104 | 1047 | ||
| 1105 | if (tmp == 0xDEADBEEF) | 1048 | if (tmp == 0xDEADBEEF) |
| 1106 | r = 0; | 1049 | r = 0; |
| @@ -1111,8 +1054,7 @@ err1: · 4 unmodified lines | |||
| 1111 | amdgpu_ib_free(&ib, NULL); | 1054 | amdgpu_ib_free(&ib, NULL); |
| 1112 | dma_fence_put(f); | 1055 | dma_fence_put(f); |
| 1113 | err0: | 1056 | err0: |
| 1114 | if (!ring->is_mes_queue) | 1057 | amdgpu_device_wb_free(adev, index); |
| 1115 | amdgpu_device_wb_free(adev, index); | ||
| 1116 | return r; | 1058 | return r; |
| 1117 | } | 1059 | } |
| 1118 | 1060 | ||
| @@ -1312,6 +1254,23 @@ static int sdma_v7_0_early_init(struct amdgpu_ip_block *ip_block) · 193 unmodified lines | |||
| 1312 | struct amdgpu_device *adev = ip_block->adev; | 1254 | struct amdgpu_device *adev = ip_block->adev; |
| 1313 | int r; | 1255 | int r; |
| 1314 | 1256 | ||
| 1257 | switch (amdgpu_user_queue) { | ||
| 1258 | case -1: | ||
| 1259 | case 0: | ||
| 1260 | default: | ||
| 1261 | adev->sdma.no_user_submission = false; | ||
| 1262 | adev->sdma.disable_uq = true; | ||
| 1263 | break; | ||
| 1264 | case 1: | ||
| 1265 | adev->sdma.no_user_submission = false; | ||
| 1266 | adev->sdma.disable_uq = false; | ||
| 1267 | break; | ||
| 1268 | case 2: | ||
| 1269 | adev->sdma.no_user_submission = true; | ||
| 1270 | adev->sdma.disable_uq = false; | ||
| 1271 | break; | ||
| 1272 | } | ||
| 1273 | |||
| 1315 | r = amdgpu_sdma_init_microcode(adev, 0, true); | 1274 | r = amdgpu_sdma_init_microcode(adev, 0, true); |
| 1316 | if (r) { | 1275 | if (r) { |
| 1317 | DRM_ERROR("Failed to init sdma firmware!\n"); | 1276 | DRM_ERROR("Failed to init sdma firmware!\n"); |
| @@ -1347,6 +1306,7 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block) · 29 unmodified lines | |||
| 1347 | ring->ring_obj = NULL; | 1306 | ring->ring_obj = NULL; |
| 1348 | ring->use_doorbell = true; | 1307 | ring->use_doorbell = true; |
| 1349 | ring->me = i; | 1308 | ring->me = i; |
| 1309 | ring->no_user_submission = adev->sdma.no_user_submission; | ||
| 1350 | 1310 | ||
| 1351 | DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i, | 1311 | DRM_DEBUG("SDMA %d use_doorbell being set to: [%s]\n", i, |
| 1352 | ring->use_doorbell?"true":"false"); | 1312 | ring->use_doorbell?"true":"false"); |
| @@ -1378,6 +1338,10 @@ static int sdma_v7_0_sw_init(struct amdgpu_ip_block *ip_block) · 25 unmodified lines | |||
| 1378 | else | 1338 | else |
| 1379 | DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n"); | 1339 | DRM_ERROR("Failed to allocated memory for SDMA IP Dump\n"); |
| 1380 | 1340 | ||
| 1341 | /* add firmware version checks here */ | ||
| 1342 | if (0 && !adev->sdma.disable_uq) | ||
| 1343 | adev->userq_funcs[AMDGPU_HW_IP_DMA] = &userq_mes_funcs; | ||
| 1344 | |||
| 1381 | return r; | 1345 | return r; |
| 1382 | } | 1346 | } |
| 1383 | 1347 | ||
| @@ -1400,11 +1364,39 @@ static int sdma_v7_0_sw_fini(struct amdgpu_ip_block *ip_block) · 16 unmodified lines | |||
| 1400 | return 0; | 1364 | return 0; |
| 1401 | } | 1365 | } |
| 1402 | 1366 | ||
| 1367 | static int sdma_v7_0_set_userq_trap_interrupts(struct amdgpu_device *adev, | ||
| 1368 | bool enable) | ||
| 1369 | { | ||
| 1370 | unsigned int irq_type; | ||
| 1371 | int i, r; | ||
| 1372 | |||
| 1373 | if (adev->userq_funcs[AMDGPU_HW_IP_DMA]) { | ||
| 1374 | for (i = 0; i < adev->sdma.num_instances; i++) { | ||
| 1375 | irq_type = AMDGPU_SDMA_IRQ_INSTANCE0 + i; | ||
| 1376 | if (enable) | ||
| 1377 | r = amdgpu_irq_get(adev, &adev->sdma.trap_irq, | ||
| 1378 | irq_type); | ||
| 1379 | else | ||
| 1380 | r = amdgpu_irq_put(adev, &adev->sdma.trap_irq, | ||
| 1381 | irq_type); | ||
| 1382 | if (r) | ||
| 1383 | return r; | ||
| 1384 | } | ||
| 1385 | } | ||
| 1386 | |||
| 1387 | return 0; | ||
| 1388 | } | ||
| 1389 | |||
| 1403 | static int sdma_v7_0_hw_init(struct amdgpu_ip_block *ip_block) | 1390 | static int sdma_v7_0_hw_init(struct amdgpu_ip_block *ip_block) |
| 1404 | { | 1391 | { |
| 1405 | struct amdgpu_device *adev = ip_block->adev; | 1392 | struct amdgpu_device *adev = ip_block->adev; |
| 1393 | int r; | ||
| 1406 | 1394 | ||
| 1407 | return sdma_v7_0_start(adev); | 1395 | r = sdma_v7_0_start(adev); |
| 1396 | if (r) | ||
| 1397 | return r; | ||
| 1398 | |||
| 1399 | return sdma_v7_0_set_userq_trap_interrupts(adev, true); | ||
| 1408 | } | 1400 | } |
| 1409 | 1401 | ||
| 1410 | static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block) | 1402 | static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block) |
| @@ -1416,6 +1408,7 @@ static int sdma_v7_0_hw_fini(struct amdgpu_ip_block *ip_block) · 5 unmodified lines | |||
| 1416 | 1408 | ||
| 1417 | sdma_v7_0_ctx_switch_enable(adev, false); | 1409 | sdma_v7_0_ctx_switch_enable(adev, false); |
| 1418 | sdma_v7_0_enable(adev, false); | 1410 | sdma_v7_0_enable(adev, false); |
| 1411 | sdma_v7_0_set_userq_trap_interrupts(adev, false); | ||
| 1419 | 1412 | ||
| 1420 | return 0; | 1413 | return 0; |
| 1421 | } | 1414 | } |
| @@ -35,6 +35,7 @@ · 34 unmodified lines | |||
| 35 | #include "amdgpu_vce.h" | 35 | #include "amdgpu_vce.h" |
| 36 | #include "atom.h" | 36 | #include "atom.h" |
| 37 | #include "amd_pcie.h" | 37 | #include "amd_pcie.h" |
| 38 | |||
| 38 | #include "si_dpm.h" | 39 | #include "si_dpm.h" |
| 39 | #include "sid.h" | 40 | #include "sid.h" |
| 40 | #include "si_ih.h" | 41 | #include "si_ih.h" |
| @@ -44,17 +45,30 @@ · 3 unmodified lines | |||
| 44 | #include "dce_v6_0.h" | 45 | #include "dce_v6_0.h" |
| 45 | #include "si.h" | 46 | #include "si.h" |
| 46 | #include "uvd_v3_1.h" | 47 | #include "uvd_v3_1.h" |
| 47 | #include "amdgpu_vkms.h" | 48 | |
| 49 | #include "uvd/uvd_4_0_d.h" | ||
| 50 | |||
| 51 | #include "smu/smu_6_0_d.h" | ||
| 52 | #include "smu/smu_6_0_sh_mask.h" | ||
| 53 | |||
| 48 | #include "gca/gfx_6_0_d.h" | 54 | #include "gca/gfx_6_0_d.h" |
| 55 | #include "gca/gfx_6_0_sh_mask.h" | ||
| 56 | |||
| 49 | #include "oss/oss_1_0_d.h" | 57 | #include "oss/oss_1_0_d.h" |
| 50 | #include "oss/oss_1_0_sh_mask.h" | 58 | #include "oss/oss_1_0_sh_mask.h" |
| 59 | |||
| 51 | #include "gmc/gmc_6_0_d.h" | 60 | #include "gmc/gmc_6_0_d.h" |
| 61 | #include"gmc/gmc_6_0_sh_mask.h" | ||
| 62 | |||
| 52 | #include "dce/dce_6_0_d.h" | 63 | #include "dce/dce_6_0_d.h" |
| 53 | #include "uvd/uvd_4_0_d.h" | 64 | #include "dce/dce_6_0_sh_mask.h" |
| 65 | |||
| 54 | #include "bif/bif_3_0_d.h" | 66 | #include "bif/bif_3_0_d.h" |
| 55 | #include "bif/bif_3_0_sh_mask.h" | 67 | #include "bif/bif_3_0_sh_mask.h" |
| 68 | #include "si_enums.h" | ||
| 56 | 69 | ||
| 57 | #include "amdgpu_dm.h" | 70 | #include "amdgpu_dm.h" |
| 71 | #include "amdgpu_vkms.h" | ||
| 58 | 72 | ||
| 59 | static const u32 tahiti_golden_registers[] = | 73 | static const u32 tahiti_golden_registers[] = |
| 60 | { | 74 | { |
| @@ -1071,8 +1085,8 @@ static u32 si_smc_rreg(struct amdgpu_device *adev, u32 reg) · 1010 unmodified lines | |||
| 1071 | u32 r; | 1085 | u32 r; |
| 1072 | 1086 | ||
| 1073 | spin_lock_irqsave(&adev->smc_idx_lock, flags); | 1087 | spin_lock_irqsave(&adev->smc_idx_lock, flags); |
| 1074 | WREG32(SMC_IND_INDEX_0, (reg)); | 1088 | WREG32(mmSMC_IND_INDEX_0, (reg)); |
| 1075 | r = RREG32(SMC_IND_DATA_0); | 1089 | r = RREG32(mmSMC_IND_DATA_0); |
| 1076 | spin_unlock_irqrestore(&adev->smc_idx_lock, flags); | 1090 | spin_unlock_irqrestore(&adev->smc_idx_lock, flags); |
| 1077 | return r; | 1091 | return r; |
| 1078 | } | 1092 | } |
| @@ -1082,8 +1096,8 @@ static void si_smc_wreg(struct amdgpu_device *adev, u32 reg, u32 v) · 3 unmodified lines | |||
| 1082 | unsigned long flags; | 1096 | unsigned long flags; |
| 1083 | 1097 | ||
| 1084 | spin_lock_irqsave(&adev->smc_idx_lock, flags); | 1098 | spin_lock_irqsave(&adev->smc_idx_lock, flags); |
| 1085 | WREG32(SMC_IND_INDEX_0, (reg)); | 1099 | WREG32(mmSMC_IND_INDEX_0, (reg)); |
| 1086 | WREG32(SMC_IND_DATA_0, (v)); | 1100 | WREG32(mmSMC_IND_DATA_0, (v)); |
| 1087 | spin_unlock_irqrestore(&adev->smc_idx_lock, flags); | 1101 | spin_unlock_irqrestore(&adev->smc_idx_lock, flags); |
| 1088 | } | 1102 | } |
| 1089 | 1103 | ||
| @@ -1110,20 +1124,20 @@ static void si_uvd_ctx_wreg(struct amdgpu_device *adev, u32 reg, u32 v) · 20 unmodified lines | |||
| 1110 | } | 1124 | } |
| 1111 | 1125 | ||
| 1112 | static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = { | 1126 | static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = { |
| 1113 | {GRBM_STATUS}, | 1127 | {mmGRBM_STATUS}, |
| 1114 | {mmGRBM_STATUS2}, | 1128 | {mmGRBM_STATUS2}, |
| 1115 | {mmGRBM_STATUS_SE0}, | 1129 | {mmGRBM_STATUS_SE0}, |
| 1116 | {mmGRBM_STATUS_SE1}, | 1130 | {mmGRBM_STATUS_SE1}, |
| 1117 | {mmSRBM_STATUS}, | 1131 | {mmSRBM_STATUS}, |
| 1118 | {mmSRBM_STATUS2}, | 1132 | {mmSRBM_STATUS2}, |
| 1119 | {DMA_STATUS_REG + DMA0_REGISTER_OFFSET}, | 1133 | {mmDMA_STATUS_REG + DMA0_REGISTER_OFFSET}, |
| 1120 | {DMA_STATUS_REG + DMA1_REGISTER_OFFSET}, | 1134 | {mmDMA_STATUS_REG + DMA1_REGISTER_OFFSET}, |
| 1121 | {mmCP_STAT}, | 1135 | {mmCP_STAT}, |
| 1122 | {mmCP_STALLED_STAT1}, | 1136 | {mmCP_STALLED_STAT1}, |
| 1123 | {mmCP_STALLED_STAT2}, | 1137 | {mmCP_STALLED_STAT2}, |
| 1124 | {mmCP_STALLED_STAT3}, | 1138 | {mmCP_STALLED_STAT3}, |
| 1125 | {GB_ADDR_CONFIG}, | 1139 | {mmGB_ADDR_CONFIG}, |
| 1126 | {MC_ARB_RAMCFG}, | 1140 | {mmMC_ARB_RAMCFG}, |
| 1127 | {mmGB_TILE_MODE0}, | 1141 | {mmGB_TILE_MODE0}, |
| 1128 | {mmGB_TILE_MODE1}, | 1142 | {mmGB_TILE_MODE1}, |
| 1129 | {mmGB_TILE_MODE2}, | 1143 | {mmGB_TILE_MODE2}, |
| @@ -1156,7 +1170,7 @@ static struct amdgpu_allowed_register_entry si_allowed_read_registers[] = { · 26 unmodified lines | |||
| 1156 | {mmGB_TILE_MODE29}, | 1170 | {mmGB_TILE_MODE29}, |
| 1157 | {mmGB_TILE_MODE30}, | 1171 | {mmGB_TILE_MODE30}, |
| 1158 | {mmGB_TILE_MODE31}, | 1172 | {mmGB_TILE_MODE31}, |
| 1159 | {CC_RB_BACKEND_DISABLE, true}, | 1173 | {mmCC_RB_BACKEND_DISABLE, true}, |
| 1160 | {mmGC_USER_RB_BACKEND_DISABLE, true}, | 1174 | {mmGC_USER_RB_BACKEND_DISABLE, true}, |
| 1161 | {mmPA_SC_RASTER_CONFIG, true}, | 1175 | {mmPA_SC_RASTER_CONFIG, true}, |
| 1162 | }; | 1176 | }; |
| @@ -1264,37 +1278,37 @@ static bool si_read_disabled_bios(struct amdgpu_device *adev) · 101 unmodified lines | |||
| 1264 | u32 rom_cntl; | 1278 | u32 rom_cntl; |
| 1265 | bool r; | 1279 | bool r; |
| 1266 | 1280 | ||
| 1267 | bus_cntl = RREG32(R600_BUS_CNTL); | 1281 | bus_cntl = RREG32(mmBUS_CNTL); |
| 1268 | if (adev->mode_info.num_crtc) { | 1282 | if (adev->mode_info.num_crtc) { |
| 1269 | d1vga_control = RREG32(AVIVO_D1VGA_CONTROL); | 1283 | d1vga_control = RREG32(mmD1VGA_CONTROL); |
| 1270 | d2vga_control = RREG32(AVIVO_D2VGA_CONTROL); | 1284 | d2vga_control = RREG32(mmD2VGA_CONTROL); |
| 1271 | vga_render_control = RREG32(VGA_RENDER_CONTROL); | 1285 | vga_render_control = RREG32(mmVGA_RENDER_CONTROL); |
| 1272 | } | 1286 | } |
| 1273 | rom_cntl = RREG32(R600_ROM_CNTL); | 1287 | rom_cntl = RREG32(R600_ROM_CNTL); |
| 1274 | 1288 | ||
| 1275 | /* enable the rom */ | 1289 | /* enable the rom */ |
| 1276 | WREG32(R600_BUS_CNTL, (bus_cntl & ~R600_BIOS_ROM_DIS)); | 1290 | WREG32(mmBUS_CNTL, (bus_cntl & ~BUS_CNTL__BIOS_ROM_DIS_MASK)); |
| 1277 | if (adev->mode_info.num_crtc) { | 1291 | if (adev->mode_info.num_crtc) { |
| 1278 | /* Disable VGA mode */ | 1292 | /* Disable VGA mode */ |
| 1279 | WREG32(AVIVO_D1VGA_CONTROL, | 1293 | WREG32(mmD1VGA_CONTROL, |
| 1280 | (d1vga_control & ~(AVIVO_DVGA_CONTROL_MODE_ENABLE | | 1294 | (d1vga_control & ~(D1VGA_CONTROL__D1VGA_MODE_ENABLE_MASK | |
| 1281 | AVIVO_DVGA_CONTROL_TIMING_SELECT))); | 1295 | D1VGA_CONTROL__D1VGA_TIMING_SELECT_MASK))); |
| 1282 | WREG32(AVIVO_D2VGA_CONTROL, | 1296 | WREG32(mmD2VGA_CONTROL, |
| 1283 | (d2vga_control & ~(AVIVO_DVGA_CONTROL_MODE_ENABLE | | 1297 | (d2vga_control & ~(D1VGA_CONTROL__D1VGA_MODE_ENABLE_MASK | |
| 1284 | AVIVO_DVGA_CONTROL_TIMING_SELECT))); | 1298 | D1VGA_CONTROL__D1VGA_TIMING_SELECT_MASK))); |
| 1285 | WREG32(VGA_RENDER_CONTROL, | 1299 | WREG32(mmVGA_RENDER_CONTROL, |
| 1286 | (vga_render_control & C_000300_VGA_VSTATUS_CNTL)); | 1300 | (vga_render_control & ~VGA_RENDER_CONTROL__VGA_VSTATUS_CNTL_MASK)); |
| 1287 | } | 1301 | } |
| 1288 | WREG32(R600_ROM_CNTL, rom_cntl | R600_SCK_OVERWRITE); | 1302 | WREG32(R600_ROM_CNTL, rom_cntl | R600_SCK_OVERWRITE); |
| 1289 | 1303 | ||
| 1290 | r = amdgpu_read_bios(adev); | 1304 | r = amdgpu_read_bios(adev); |
| 1291 | 1305 | ||
| 1292 | /* restore regs */ | 1306 | /* restore regs */ |
| 1293 | WREG32(R600_BUS_CNTL, bus_cntl); | 1307 | WREG32(mmBUS_CNTL, bus_cntl); |
| 1294 | if (adev->mode_info.num_crtc) { | 1308 | if (adev->mode_info.num_crtc) { |
| 1295 | WREG32(AVIVO_D1VGA_CONTROL, d1vga_control); | 1309 | WREG32(mmD1VGA_CONTROL, d1vga_control); |
| 1296 | WREG32(AVIVO_D2VGA_CONTROL, d2vga_control); | 1310 | WREG32(mmD2VGA_CONTROL, d2vga_control); |
| 1297 | WREG32(VGA_RENDER_CONTROL, vga_render_control); | 1311 | WREG32(mmVGA_RENDER_CONTROL, vga_render_control); |
| 1298 | } | 1312 | } |
| 1299 | WREG32(R600_ROM_CNTL, rom_cntl); | 1313 | WREG32(R600_ROM_CNTL, rom_cntl); |
| 1300 | return r; | 1314 | return r; |
| @@ -1331,23 +1345,24 @@ static void si_set_clk_bypass_mode(struct amdgpu_device *adev) · 30 unmodified lines | |||
| 1331 | { | 1345 | { |
| 1332 | u32 tmp, i; | 1346 | u32 tmp, i; |
| 1333 | 1347 | ||
| 1334 | tmp = RREG32(CG_SPLL_FUNC_CNTL); | 1348 | tmp = RREG32(mmCG_SPLL_FUNC_CNTL); |
| 1335 | tmp |= SPLL_BYPASS_EN; | 1349 | tmp |= CG_SPLL_FUNC_CNTL__SPLL_BYPASS_EN_MASK; |
| 1336 | WREG32(CG_SPLL_FUNC_CNTL, tmp); | 1350 | WREG32(mmCG_SPLL_FUNC_CNTL, tmp); |
| 1337 | 1351 | ||
| 1338 | tmp = RREG32(CG_SPLL_FUNC_CNTL_2); | 1352 | tmp = RREG32(mmCG_SPLL_FUNC_CNTL_2); |
| 1339 | tmp |= SPLL_CTLREQ_CHG; | 1353 | tmp |= CG_SPLL_FUNC_CNTL_2__SPLL_CTLREQ_CHG_MASK; |
| 1340 | WREG32(CG_SPLL_FUNC_CNTL_2, tmp); | 1354 | WREG32(mmCG_SPLL_FUNC_CNTL_2, tmp); |
| 1341 | 1355 | ||
| 1342 | for (i = 0; i < adev->usec_timeout; i++) { | 1356 | for (i = 0; i < adev->usec_timeout; i++) { |
| 1343 | if (RREG32(SPLL_STATUS) & SPLL_CHG_STATUS) | 1357 | if (RREG32(mmCG_SPLL_STATUS) & CG_SPLL_STATUS__SPLL_CHG_STATUS_MASK) |
| 1344 | break; | 1358 | break; |
| 1345 | udelay(1); | 1359 | udelay(1); |
| 1346 | } | 1360 | } |
| 1347 | 1361 | ||
| 1348 | tmp = RREG32(CG_SPLL_FUNC_CNTL_2); | 1362 | tmp = RREG32(mmCG_SPLL_FUNC_CNTL_2); |
| 1349 | tmp &= ~(SPLL_CTLREQ_CHG | SCLK_MUX_UPDATE); | 1363 | tmp &= ~(CG_SPLL_FUNC_CNTL_2__SPLL_CTLREQ_CHG_MASK | |
| 1350 | WREG32(CG_SPLL_FUNC_CNTL_2, tmp); | 1364 | CG_SPLL_FUNC_CNTL_2__SCLK_MUX_UPDATE_MASK); |
| 1365 | WREG32(mmCG_SPLL_FUNC_CNTL_2, tmp); | ||
| 1351 | 1366 | ||
| 1352 | tmp = RREG32(MPLL_CNTL_MODE); | 1367 | tmp = RREG32(MPLL_CNTL_MODE); |
| 1353 | tmp &= ~MPLL_MCLK_SEL; | 1368 | tmp &= ~MPLL_MCLK_SEL; |
| @@ -1358,21 +1373,21 @@ static void si_spll_powerdown(struct amdgpu_device *adev) · 4 unmodified lines | |||
| 1358 | { | 1373 | { |
| 1359 | u32 tmp; | 1374 | u32 tmp; |
| 1360 | 1375 | ||
| 1361 | tmp = RREG32(SPLL_CNTL_MODE); | 1376 | tmp = RREG32(mmSPLL_CNTL_MODE); |
| 1362 | tmp |= SPLL_SW_DIR_CONTROL; | 1377 | tmp |= SPLL_CNTL_MODE__SPLL_SW_DIR_CONTROL_MASK; |
| 1363 | WREG32(SPLL_CNTL_MODE, tmp); | 1378 | WREG32(mmSPLL_CNTL_MODE, tmp); |
| 1364 | 1379 | ||
| 1365 | tmp = RREG32(CG_SPLL_FUNC_CNTL); | 1380 | tmp = RREG32(mmCG_SPLL_FUNC_CNTL); |
| 1366 | tmp |= SPLL_RESET; | 1381 | tmp |= CG_SPLL_FUNC_CNTL__SPLL_RESET_MASK; |
| 1367 | WREG32(CG_SPLL_FUNC_CNTL, tmp); | 1382 | WREG32(mmCG_SPLL_FUNC_CNTL, tmp); |
| 1368 | 1383 | ||
| 1369 | tmp = RREG32(CG_SPLL_FUNC_CNTL); | 1384 | tmp = RREG32(mmCG_SPLL_FUNC_CNTL); |
| 1370 | tmp |= SPLL_SLEEP; | 1385 | tmp |= CG_SPLL_FUNC_CNTL__SPLL_SLEEP_MASK; |
| 1371 | WREG32(CG_SPLL_FUNC_CNTL, tmp); | 1386 | WREG32(mmCG_SPLL_FUNC_CNTL, tmp); |
| 1372 | 1387 | ||
| 1373 | tmp = RREG32(SPLL_CNTL_MODE); | 1388 | tmp = RREG32(mmSPLL_CNTL_MODE); |
| 1374 | tmp &= ~SPLL_SW_DIR_CONTROL; | 1389 | tmp &= ~SPLL_CNTL_MODE__SPLL_SW_DIR_CONTROL_MASK; |
| 1375 | WREG32(SPLL_CNTL_MODE, tmp); | 1390 | WREG32(mmSPLL_CNTL_MODE, tmp); |
| 1376 | } | 1391 | } |
| 1377 | 1392 | ||
| 1378 | static int si_gpu_pci_config_reset(struct amdgpu_device *adev) | 1393 | static int si_gpu_pci_config_reset(struct amdgpu_device *adev) |
| @@ -1454,14 +1469,14 @@ static void si_vga_set_state(struct amdgpu_device *adev, bool state) · 75 unmodified lines | |||
| 1454 | { | 1469 | { |
| 1455 | uint32_t temp; | 1470 | uint32_t temp; |
| 1456 | 1471 | ||
| 1457 | temp = RREG32(CONFIG_CNTL); | 1472 | temp = RREG32(mmCONFIG_CNTL); |
| 1458 | if (!state) { | 1473 | if (!state) { |
| 1459 | temp &= ~(1<<0); | 1474 | temp &= ~(1<<0); |
| 1460 | temp |= (1<<1); | 1475 | temp |= (1<<1); |
| 1461 | } else { | 1476 | } else { |
| 1462 | temp &= ~(1<<1); | 1477 | temp &= ~(1<<1); |
| 1463 | } | 1478 | } |
| 1464 | WREG32(CONFIG_CNTL, temp); | 1479 | WREG32(mmCONFIG_CNTL, temp); |
| 1465 | } | 1480 | } |
| 1466 | 1481 | ||
| 1467 | static u32 si_get_xclk(struct amdgpu_device *adev) | 1482 | static u32 si_get_xclk(struct amdgpu_device *adev) |
| @@ -1469,12 +1484,12 @@ static u32 si_get_xclk(struct amdgpu_device *adev) · 1 unmodified line | |||
| 1469 | u32 reference_clock = adev->clock.spll.reference_freq; | 1484 | u32 reference_clock = adev->clock.spll.reference_freq; |
| 1470 | u32 tmp; | 1485 | u32 tmp; |
| 1471 | 1486 | ||
| 1472 | tmp = RREG32(CG_CLKPIN_CNTL_2); | 1487 | tmp = RREG32(mmCG_CLKPIN_CNTL_2); |
| 1473 | if (tmp & MUX_TCLK_TO_XCLK) | 1488 | if (tmp & CG_CLKPIN_CNTL_2__MUX_TCLK_TO_XCLK_MASK) |
| 1474 | return TCLK; | 1489 | return TCLK; |
| 1475 | 1490 | ||
| 1476 | tmp = RREG32(CG_CLKPIN_CNTL); | 1491 | tmp = RREG32(mmCG_CLKPIN_CNTL); |
| 1477 | if (tmp & XTALIN_DIVIDE) | 1492 | if (tmp & CG_CLKPIN_CNTL__XTALIN_DIVIDE_MASK) |
| 1478 | return reference_clock / 4; | 1493 | return reference_clock / 4; |
| 1479 | 1494 | ||
| 1480 | return reference_clock; | 1495 | return reference_clock; |
| @@ -1519,9 +1534,9 @@ static int si_get_pcie_lanes(struct amdgpu_device *adev) · 38 unmodified lines | |||
| 1519 | if (adev->flags & AMD_IS_APU) | 1534 | if (adev->flags & AMD_IS_APU) |
| 1520 | return 0; | 1535 | return 0; |
| 1521 | 1536 | ||
| 1522 | link_width_cntl = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL); | 1537 | link_width_cntl = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL); |
| 1523 | 1538 | ||
| 1524 | switch ((link_width_cntl & LC_LINK_WIDTH_RD_MASK) >> LC_LINK_WIDTH_RD_SHIFT) { | 1539 | switch ((link_width_cntl & PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD_MASK) >> PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_RD__SHIFT) { |
| 1525 | case LC_LINK_WIDTH_X1: | 1540 | case LC_LINK_WIDTH_X1: |
| 1526 | return 1; | 1541 | return 1; |
| 1527 | case LC_LINK_WIDTH_X2: | 1542 | case LC_LINK_WIDTH_X2: |
| @@ -1568,13 +1583,13 @@ static void si_set_pcie_lanes(struct amdgpu_device *adev, int lanes) · 40 unmodified lines | |||
| 1568 | return; | 1583 | return; |
| 1569 | } | 1584 | } |
| 1570 | 1585 | ||
| 1571 | link_width_cntl = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL); | 1586 | link_width_cntl = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL); |
| 1572 | link_width_cntl &= ~LC_LINK_WIDTH_MASK; | 1587 | link_width_cntl &= ~PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_MASK; |
| 1573 | link_width_cntl |= mask << LC_LINK_WIDTH_SHIFT; | 1588 | link_width_cntl |= mask << PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH__SHIFT; |
| 1574 | link_width_cntl |= (LC_RECONFIG_NOW | | 1589 | link_width_cntl |= (PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_NOW_MASK | |
| 1575 | LC_RECONFIG_ARC_MISSING_ESCAPE); | 1590 | PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_ARC_MISSING_ESCAPE_MASK); |
| 1576 | 1591 | ||
| 1577 | WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, link_width_cntl); | 1592 | WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, link_width_cntl); |
| 1578 | } | 1593 | } |
| 1579 | 1594 | ||
| 1580 | static void si_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0, | 1595 | static void si_get_pcie_usage(struct amdgpu_device *adev, uint64_t *count0, |
| @@ -2018,7 +2033,7 @@ static const struct amdgpu_asic_funcs si_asic_funcs = · 437 unmodified lines | |||
| 2018 | 2033 | ||
| 2019 | static uint32_t si_get_rev_id(struct amdgpu_device *adev) | 2034 | static uint32_t si_get_rev_id(struct amdgpu_device *adev) |
| 2020 | { | 2035 | { |
| 2021 | return (RREG32(CC_DRM_ID_STRAPS) & CC_DRM_ID_STRAPS__ATI_REV_ID_MASK) | 2036 | return (RREG32(mmCC_DRM_ID_STRAPS) & CC_DRM_ID_STRAPS__ATI_REV_ID_MASK) |
| 2022 | >> CC_DRM_ID_STRAPS__ATI_REV_ID__SHIFT; | 2037 | >> CC_DRM_ID_STRAPS__ATI_REV_ID__SHIFT; |
| 2023 | } | 2038 | } |
| 2024 | 2039 | ||
| @@ -2239,9 +2254,9 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 214 unmodified lines | |||
| 2239 | CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3))) | 2254 | CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3))) |
| 2240 | return; | 2255 | return; |
| 2241 | 2256 | ||
| 2242 | speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL); | 2257 | speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL); |
| 2243 | current_data_rate = (speed_cntl & LC_CURRENT_DATA_RATE_MASK) >> | 2258 | current_data_rate = (speed_cntl & PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE_MASK) >> |
| 2244 | LC_CURRENT_DATA_RATE_SHIFT; | 2259 | PCIE_LC_SPEED_CNTL__LC_CURRENT_DATA_RATE__SHIFT; |
| 2245 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) { | 2260 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) { |
| 2246 | if (current_data_rate == 2) { | 2261 | if (current_data_rate == 2) { |
| 2247 | DRM_INFO("PCIE gen 3 link speeds already enabled\n"); | 2262 | DRM_INFO("PCIE gen 3 link speeds already enabled\n"); |
| @@ -2268,17 +2283,17 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 20 unmodified lines | |||
| 2268 | pcie_capability_set_word(root, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD); | 2283 | pcie_capability_set_word(root, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD); |
| 2269 | pcie_capability_set_word(adev->pdev, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD); | 2284 | pcie_capability_set_word(adev->pdev, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_HAWD); |
| 2270 | 2285 | ||
| 2271 | tmp = RREG32_PCIE(PCIE_LC_STATUS1); | 2286 | tmp = RREG32_PCIE(ixPCIE_LC_STATUS1); |
| 2272 | max_lw = (tmp & LC_DETECTED_LINK_WIDTH_MASK) >> LC_DETECTED_LINK_WIDTH_SHIFT; | 2287 | max_lw = (tmp & PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH_MASK) >> PCIE_LC_STATUS1__LC_DETECTED_LINK_WIDTH__SHIFT; |
| 2273 | current_lw = (tmp & LC_OPERATING_LINK_WIDTH_MASK) >> LC_OPERATING_LINK_WIDTH_SHIFT; | 2288 | current_lw = (tmp & PCIE_LC_STATUS1__LC_OPERATING_LINK_WIDTH_MASK) >> PCIE_LC_STATUS1__LC_OPERATING_LINK_WIDTH__SHIFT; |
| 2274 | 2289 | ||
| 2275 | if (current_lw < max_lw) { | 2290 | if (current_lw < max_lw) { |
| 2276 | tmp = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL); | 2291 | tmp = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL); |
| 2277 | if (tmp & LC_RENEGOTIATION_SUPPORT) { | 2292 | if (tmp & PCIE_LC_LINK_WIDTH_CNTL__LC_RENEGOTIATION_SUPPORT_MASK) { |
| 2278 | tmp &= ~(LC_LINK_WIDTH_MASK | LC_UPCONFIGURE_DIS); | 2293 | tmp &= ~(PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_UPCONFIGURE_DIS_MASK); |
| 2279 | tmp |= (max_lw << LC_LINK_WIDTH_SHIFT); | 2294 | tmp |= (max_lw << PCIE_LC_LINK_WIDTH_CNTL__LC_LINK_WIDTH__SHIFT); |
| 2280 | tmp |= LC_UPCONFIGURE_SUPPORT | LC_RENEGOTIATE_EN | LC_RECONFIG_NOW; | 2295 | tmp |= PCIE_LC_LINK_WIDTH_CNTL__LC_UPCONFIGURE_SUPPORT_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_RENEGOTIATE_EN_MASK | PCIE_LC_LINK_WIDTH_CNTL__LC_RECONFIG_NOW_MASK; |
| 2281 | WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, tmp); | 2296 | WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, tmp); |
| 2282 | } | 2297 | } |
| 2283 | } | 2298 | } |
| 2284 | 2299 | ||
| @@ -2301,13 +2316,13 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 16 unmodified lines | |||
| 2301 | PCI_EXP_LNKCTL2, | 2316 | PCI_EXP_LNKCTL2, |
| 2302 | &gpu_cfg2); | 2317 | &gpu_cfg2); |
| 2303 | 2318 | ||
| 2304 | tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4); | 2319 | tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4); |
| 2305 | tmp |= LC_SET_QUIESCE; | 2320 | tmp |= PCIE_LC_CNTL4__LC_SET_QUIESCE_MASK; |
| 2306 | WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp); | 2321 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp); |
| 2307 | 2322 | ||
| 2308 | tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4); | 2323 | tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4); |
| 2309 | tmp |= LC_REDO_EQ; | 2324 | tmp |= PCIE_LC_CNTL4__LC_REDO_EQ_MASK; |
| 2310 | WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp); | 2325 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp); |
| 2311 | 2326 | ||
| 2312 | mdelay(100); | 2327 | mdelay(100); |
| 2313 | 2328 | ||
| @@ -2333,16 +2348,16 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 19 unmodified lines | |||
| 2333 | (PCI_EXP_LNKCTL2_ENTER_COMP | | 2348 | (PCI_EXP_LNKCTL2_ENTER_COMP | |
| 2334 | PCI_EXP_LNKCTL2_TX_MARGIN)); | 2349 | PCI_EXP_LNKCTL2_TX_MARGIN)); |
| 2335 | 2350 | ||
| 2336 | tmp = RREG32_PCIE_PORT(PCIE_LC_CNTL4); | 2351 | tmp = RREG32_PCIE_PORT(ixPCIE_LC_CNTL4); |
| 2337 | tmp &= ~LC_SET_QUIESCE; | 2352 | tmp &= ~PCIE_LC_CNTL4__LC_SET_QUIESCE_MASK; |
| 2338 | WREG32_PCIE_PORT(PCIE_LC_CNTL4, tmp); | 2353 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL4, tmp); |
| 2339 | } | 2354 | } |
| 2340 | } | 2355 | } |
| 2341 | } | 2356 | } |
| 2342 | 2357 | ||
| 2343 | speed_cntl |= LC_FORCE_EN_SW_SPEED_CHANGE | LC_FORCE_DIS_HW_SPEED_CHANGE; | 2358 | speed_cntl |= PCIE_LC_SPEED_CNTL__LC_FORCE_EN_SW_SPEED_CHANGE_MASK | PCIE_LC_SPEED_CNTL__LC_FORCE_DIS_HW_SPEED_CHANGE_MASK; |
| 2344 | speed_cntl &= ~LC_FORCE_DIS_SW_SPEED_CHANGE; | 2359 | speed_cntl &= ~PCIE_LC_SPEED_CNTL__LC_FORCE_DIS_SW_SPEED_CHANGE_MASK; |
| 2345 | WREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL, speed_cntl); | 2360 | WREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL, speed_cntl); |
| 2346 | 2361 | ||
| 2347 | tmp16 = 0; | 2362 | tmp16 = 0; |
| 2348 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) | 2363 | if (adev->pm.pcie_gen_mask & CAIL_PCIE_LINK_SPEED_SUPPORT_GEN3) |
| @@ -2354,13 +2369,13 @@ static void si_pcie_gen3_enable(struct amdgpu_device *adev) · 5 unmodified lines | |||
| 2354 | pcie_capability_clear_and_set_word(adev->pdev, PCI_EXP_LNKCTL2, | 2369 | pcie_capability_clear_and_set_word(adev->pdev, PCI_EXP_LNKCTL2, |
| 2355 | PCI_EXP_LNKCTL2_TLS, tmp16); | 2370 | PCI_EXP_LNKCTL2_TLS, tmp16); |
| 2356 | 2371 | ||
| 2357 | speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL); | 2372 | speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL); |
| 2358 | speed_cntl |= LC_INITIATE_LINK_SPEED_CHANGE; | 2373 | speed_cntl |= PCIE_LC_SPEED_CNTL__LC_INITIATE_LINK_SPEED_CHANGE_MASK; |
| 2359 | WREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL, speed_cntl); | 2374 | WREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL, speed_cntl); |
| 2360 | 2375 | ||
| 2361 | for (i = 0; i < adev->usec_timeout; i++) { | 2376 | for (i = 0; i < adev->usec_timeout; i++) { |
| 2362 | speed_cntl = RREG32_PCIE_PORT(PCIE_LC_SPEED_CNTL); | 2377 | speed_cntl = RREG32_PCIE_PORT(ixPCIE_LC_SPEED_CNTL); |
| 2363 | if ((speed_cntl & LC_INITIATE_LINK_SPEED_CHANGE) == 0) | 2378 | if ((speed_cntl & PCIE_LC_SPEED_CNTL__LC_INITIATE_LINK_SPEED_CHANGE_MASK) == 0) |
| 2364 | break; | 2379 | break; |
| 2365 | udelay(1); | 2380 | udelay(1); |
| 2366 | } | 2381 | } |
| @@ -2418,121 +2433,121 @@ static void si_program_aspm(struct amdgpu_device *adev) · 51 unmodified lines | |||
| 2418 | if (!amdgpu_device_should_use_aspm(adev)) | 2433 | if (!amdgpu_device_should_use_aspm(adev)) |
| 2419 | return; | 2434 | return; |
| 2420 | 2435 | ||
| 2421 | orig = data = RREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL); | 2436 | orig = data = RREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL); |
| 2422 | data &= ~LC_XMIT_N_FTS_MASK; | 2437 | data &= ~PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS_MASK; |
| 2423 | data |= LC_XMIT_N_FTS(0x24) | LC_XMIT_N_FTS_OVERRIDE_EN; | 2438 | data |= (0x24 << PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS__SHIFT) | PCIE_LC_N_FTS_CNTL__LC_XMIT_N_FTS_OVERRIDE_EN_MASK; |
| 2424 | if (orig != data) | 2439 | if (orig != data) |
| 2425 | WREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL, data); | 2440 | WREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL, data); |
| 2426 | 2441 | ||
| 2427 | orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL3); | 2442 | orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL3); |
| 2428 | data |= LC_GO_TO_RECOVERY; | 2443 | data |= PCIE_LC_CNTL3__LC_GO_TO_RECOVERY_MASK; |
| 2429 | if (orig != data) | 2444 | if (orig != data) |
| 2430 | WREG32_PCIE_PORT(PCIE_LC_CNTL3, data); | 2445 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL3, data); |
| 2431 | 2446 | ||
| 2432 | orig = data = RREG32_PCIE(PCIE_P_CNTL); | 2447 | orig = data = RREG32_PCIE(ixPCIE_P_CNTL); |
| 2433 | data |= P_IGNORE_EDB_ERR; | 2448 | data |= PCIE_P_CNTL__P_IGNORE_EDB_ERR_MASK; |
| 2434 | if (orig != data) | 2449 | if (orig != data) |
| 2435 | WREG32_PCIE(PCIE_P_CNTL, data); | 2450 | WREG32_PCIE(ixPCIE_P_CNTL, data); |
| 2436 | 2451 | ||
| 2437 | orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL); | 2452 | orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL); |
| 2438 | data &= ~(LC_L0S_INACTIVITY_MASK | LC_L1_INACTIVITY_MASK); | 2453 | data &= ~(PCIE_LC_CNTL__LC_L0S_INACTIVITY_MASK | PCIE_LC_CNTL__LC_L1_INACTIVITY_MASK); |
| 2439 | data |= LC_PMI_TO_L1_DIS; | 2454 | data |= PCIE_LC_CNTL__LC_PMI_TO_L1_DIS_MASK; |
| 2440 | if (!disable_l0s) | 2455 | if (!disable_l0s) |
| 2441 | data |= LC_L0S_INACTIVITY(7); | 2456 | data |= (7 << PCIE_LC_CNTL__LC_L0S_INACTIVITY__SHIFT); |
| 2442 | 2457 | ||
| 2443 | if (!disable_l1) { | 2458 | if (!disable_l1) { |
| 2444 | data |= LC_L1_INACTIVITY(7); | 2459 | data |= (7 << PCIE_LC_CNTL__LC_L1_INACTIVITY__SHIFT); |
| 2445 | data &= ~LC_PMI_TO_L1_DIS; | 2460 | data &= ~PCIE_LC_CNTL__LC_PMI_TO_L1_DIS_MASK; |
| 2446 | if (orig != data) | 2461 | if (orig != data) |
| 2447 | WREG32_PCIE_PORT(PCIE_LC_CNTL, data); | 2462 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data); |
| 2448 | 2463 | ||
| 2449 | if (!disable_plloff_in_l1) { | 2464 | if (!disable_plloff_in_l1) { |
| 2450 | bool clk_req_support; | 2465 | bool clk_req_support; |
| 2451 | 2466 | ||
| 2452 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_0); | 2467 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_0); |
| 2453 | data &= ~(PLL_POWER_STATE_IN_OFF_0_MASK | PLL_POWER_STATE_IN_TXS2_0_MASK); | 2468 | data &= ~(PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0_MASK | PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0_MASK); |
| 2454 | data |= PLL_POWER_STATE_IN_OFF_0(7) | PLL_POWER_STATE_IN_TXS2_0(7); | 2469 | data |= (7 << PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0__SHIFT) | (7 << PB0_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0__SHIFT); |
| 2455 | if (orig != data) | 2470 | if (orig != data) |
| 2456 | si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_0, data); | 2471 | si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_0, data); |
| 2457 | 2472 | ||
| 2458 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_1); | 2473 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_1); |
| 2459 | data &= ~(PLL_POWER_STATE_IN_OFF_1_MASK | PLL_POWER_STATE_IN_TXS2_1_MASK); | 2474 | data &= ~(PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1_MASK | PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1_MASK); |
| 2460 | data |= PLL_POWER_STATE_IN_OFF_1(7) | PLL_POWER_STATE_IN_TXS2_1(7); | 2475 | data |= (7 << PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1__SHIFT) | (7 << PB0_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1__SHIFT); |
| 2461 | if (orig != data) | 2476 | if (orig != data) |
| 2462 | si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_1, data); | 2477 | si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_1, data); |
| 2463 | 2478 | ||
| 2464 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_0); | 2479 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_0); |
| 2465 | data &= ~(PLL_POWER_STATE_IN_OFF_0_MASK | PLL_POWER_STATE_IN_TXS2_0_MASK); | 2480 | data &= ~(PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0_MASK | PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0_MASK); |
| 2466 | data |= PLL_POWER_STATE_IN_OFF_0(7) | PLL_POWER_STATE_IN_TXS2_0(7); | 2481 | data |= (7 << PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_OFF_0__SHIFT) | (7 << PB1_PIF_PWRDOWN_0__PLL_POWER_STATE_IN_TXS2_0__SHIFT); |
| 2467 | if (orig != data) | 2482 | if (orig != data) |
| 2468 | si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_0, data); | 2483 | si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_0, data); |
| 2469 | 2484 | ||
| 2470 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_1); | 2485 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_1); |
| 2471 | data &= ~(PLL_POWER_STATE_IN_OFF_1_MASK | PLL_POWER_STATE_IN_TXS2_1_MASK); | 2486 | data &= ~(PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1_MASK | PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1_MASK); |
| 2472 | data |= PLL_POWER_STATE_IN_OFF_1(7) | PLL_POWER_STATE_IN_TXS2_1(7); | 2487 | data |= (7 << PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_OFF_1__SHIFT) | (7 << PB1_PIF_PWRDOWN_1__PLL_POWER_STATE_IN_TXS2_1__SHIFT); |
| 2473 | if (orig != data) | 2488 | if (orig != data) |
| 2474 | si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_1, data); | 2489 | si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_1, data); |
| 2475 | 2490 | ||
| 2476 | if ((adev->asic_type != CHIP_OLAND) && (adev->asic_type != CHIP_HAINAN)) { | 2491 | if ((adev->asic_type != CHIP_OLAND) && (adev->asic_type != CHIP_HAINAN)) { |
| 2477 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_0); | 2492 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_0); |
| 2478 | data &= ~PLL_RAMP_UP_TIME_0_MASK; | 2493 | data &= ~PB0_PIF_PWRDOWN_0__PLL_RAMP_UP_TIME_0_MASK; |
| 2479 | if (orig != data) | 2494 | if (orig != data) |
| 2480 | si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_0, data); | 2495 | si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_0, data); |
| 2481 | 2496 | ||
| 2482 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_1); | 2497 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_1); |
| 2483 | data &= ~PLL_RAMP_UP_TIME_1_MASK; | 2498 | data &= ~PB0_PIF_PWRDOWN_1__PLL_RAMP_UP_TIME_1_MASK; |
| 2484 | if (orig != data) | 2499 | if (orig != data) |
| 2485 | si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_1, data); | 2500 | si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_1, data); |
| 2486 | 2501 | ||
| 2487 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_2); | 2502 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_2); |
| 2488 | data &= ~PLL_RAMP_UP_TIME_2_MASK; | 2503 | data &= ~PB0_PIF_PWRDOWN_2__PLL_RAMP_UP_TIME_2_MASK; |
| 2489 | if (orig != data) | 2504 | if (orig != data) |
| 2490 | si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_2, data); | 2505 | si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_2, data); |
| 2491 | 2506 | ||
| 2492 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_PWRDOWN_3); | 2507 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_PWRDOWN_3); |
| 2493 | data &= ~PLL_RAMP_UP_TIME_3_MASK; | 2508 | data &= ~PB0_PIF_PWRDOWN_3__PLL_RAMP_UP_TIME_3_MASK; |
| 2494 | if (orig != data) | 2509 | if (orig != data) |
| 2495 | si_pif_phy0_wreg(adev,PB0_PIF_PWRDOWN_3, data); | 2510 | si_pif_phy0_wreg(adev,ixPB0_PIF_PWRDOWN_3, data); |
| 2496 | 2511 | ||
| 2497 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_0); | 2512 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_0); |
| 2498 | data &= ~PLL_RAMP_UP_TIME_0_MASK; | 2513 | data &= ~PB1_PIF_PWRDOWN_0__PLL_RAMP_UP_TIME_0_MASK; |
| 2499 | if (orig != data) | 2514 | if (orig != data) |
| 2500 | si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_0, data); | 2515 | si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_0, data); |
| 2501 | 2516 | ||
| 2502 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_1); | 2517 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_1); |
| 2503 | data &= ~PLL_RAMP_UP_TIME_1_MASK; | 2518 | data &= ~PB1_PIF_PWRDOWN_1__PLL_RAMP_UP_TIME_1_MASK; |
| 2504 | if (orig != data) | 2519 | if (orig != data) |
| 2505 | si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_1, data); | 2520 | si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_1, data); |
| 2506 | 2521 | ||
| 2507 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_2); | 2522 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_2); |
| 2508 | data &= ~PLL_RAMP_UP_TIME_2_MASK; | 2523 | data &= ~PB1_PIF_PWRDOWN_2__PLL_RAMP_UP_TIME_2_MASK; |
| 2509 | if (orig != data) | 2524 | if (orig != data) |
| 2510 | si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_2, data); | 2525 | si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_2, data); |
| 2511 | 2526 | ||
| 2512 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_PWRDOWN_3); | 2527 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_PWRDOWN_3); |
| 2513 | data &= ~PLL_RAMP_UP_TIME_3_MASK; | 2528 | data &= ~PB1_PIF_PWRDOWN_3__PLL_RAMP_UP_TIME_3_MASK; |
| 2514 | if (orig != data) | 2529 | if (orig != data) |
| 2515 | si_pif_phy1_wreg(adev,PB1_PIF_PWRDOWN_3, data); | 2530 | si_pif_phy1_wreg(adev,ixPB1_PIF_PWRDOWN_3, data); |
| 2516 | } | 2531 | } |
| 2517 | orig = data = RREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL); | 2532 | orig = data = RREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL); |
| 2518 | data &= ~LC_DYN_LANES_PWR_STATE_MASK; | 2533 | data &= ~PCIE_LC_LINK_WIDTH_CNTL__LC_DYN_LANES_PWR_STATE_MASK; |
| 2519 | data |= LC_DYN_LANES_PWR_STATE(3); | 2534 | data |= (3 << PCIE_LC_LINK_WIDTH_CNTL__LC_DYN_LANES_PWR_STATE__SHIFT); |
| 2520 | if (orig != data) | 2535 | if (orig != data) |
| 2521 | WREG32_PCIE_PORT(PCIE_LC_LINK_WIDTH_CNTL, data); | 2536 | WREG32_PCIE_PORT(ixPCIE_LC_LINK_WIDTH_CNTL, data); |
| 2522 | 2537 | ||
| 2523 | orig = data = si_pif_phy0_rreg(adev,PB0_PIF_CNTL); | 2538 | orig = data = si_pif_phy0_rreg(adev,ixPB0_PIF_CNTL); |
| 2524 | data &= ~LS2_EXIT_TIME_MASK; | 2539 | data &= ~PB0_PIF_CNTL__LS2_EXIT_TIME_MASK; |
| 2525 | if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN)) | 2540 | if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN)) |
| 2526 | data |= LS2_EXIT_TIME(5); | 2541 | data |= (5 << PB0_PIF_CNTL__LS2_EXIT_TIME__SHIFT); |
| 2527 | if (orig != data) | 2542 | if (orig != data) |
| 2528 | si_pif_phy0_wreg(adev,PB0_PIF_CNTL, data); | 2543 | si_pif_phy0_wreg(adev,ixPB0_PIF_CNTL, data); |
| 2529 | 2544 | ||
| 2530 | orig = data = si_pif_phy1_rreg(adev,PB1_PIF_CNTL); | 2545 | orig = data = si_pif_phy1_rreg(adev,ixPB1_PIF_CNTL); |
| 2531 | data &= ~LS2_EXIT_TIME_MASK; | 2546 | data &= ~PB1_PIF_CNTL__LS2_EXIT_TIME_MASK; |
| 2532 | if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN)) | 2547 | if ((adev->asic_type == CHIP_OLAND) || (adev->asic_type == CHIP_HAINAN)) |
| 2533 | data |= LS2_EXIT_TIME(5); | 2548 | data |= (5 << PB1_PIF_CNTL__LS2_EXIT_TIME__SHIFT); |
| 2534 | if (orig != data) | 2549 | if (orig != data) |
| 2535 | si_pif_phy1_wreg(adev,PB1_PIF_CNTL, data); | 2550 | si_pif_phy1_wreg(adev,ixPB1_PIF_CNTL, data); |
| 2536 | 2551 | ||
| 2537 | if (!disable_clkreq && | 2552 | if (!disable_clkreq && |
| 2538 | !pci_is_root_bus(adev->pdev->bus)) { | 2553 | !pci_is_root_bus(adev->pdev->bus)) { |
| @@ -2548,64 +2563,64 @@ static void si_program_aspm(struct amdgpu_device *adev) · 9 unmodified lines | |||
| 2548 | } | 2563 | } |
| 2549 | 2564 | ||
| 2550 | if (clk_req_support) { | 2565 | if (clk_req_support) { |
| 2551 | orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL2); | 2566 | orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL2); |
| 2552 | data |= LC_ALLOW_PDWN_IN_L1 | LC_ALLOW_PDWN_IN_L23; | 2567 | data |= PCIE_LC_CNTL2__LC_ALLOW_PDWN_IN_L1_MASK | PCIE_LC_CNTL2__LC_ALLOW_PDWN_IN_L23_MASK; |
| 2553 | if (orig != data) | 2568 | if (orig != data) |
| 2554 | WREG32_PCIE_PORT(PCIE_LC_CNTL2, data); | 2569 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL2, data); |
| 2555 | 2570 | ||
| 2556 | orig = data = RREG32(THM_CLK_CNTL); | 2571 | orig = data = RREG32(mmTHM_CLK_CNTL); |
| 2557 | data &= ~(CMON_CLK_SEL_MASK | TMON_CLK_SEL_MASK); | 2572 | data &= ~(THM_CLK_CNTL__CMON_CLK_SEL_MASK | THM_CLK_CNTL__TMON_CLK_SEL_MASK); |
| 2558 | data |= CMON_CLK_SEL(1) | TMON_CLK_SEL(1); | 2573 | data |= (1 << THM_CLK_CNTL__CMON_CLK_SEL__SHIFT) | (1 << THM_CLK_CNTL__TMON_CLK_SEL__SHIFT); |
| 2559 | if (orig != data) | 2574 | if (orig != data) |
| 2560 | WREG32(THM_CLK_CNTL, data); | 2575 | WREG32(mmTHM_CLK_CNTL, data); |
| 2561 | 2576 | ||
| 2562 | orig = data = RREG32(MISC_CLK_CNTL); | 2577 | orig = data = RREG32(mmMISC_CLK_CNTL); |
| 2563 | data &= ~(DEEP_SLEEP_CLK_SEL_MASK | ZCLK_SEL_MASK); | 2578 | data &= ~(MISC_CLK_CNTL__DEEP_SLEEP_CLK_SEL_MASK | MISC_CLK_CNTL__ZCLK_SEL_MASK); |
| 2564 | data |= DEEP_SLEEP_CLK_SEL(1) | ZCLK_SEL(1); | 2579 | data |= (1 << MISC_CLK_CNTL__DEEP_SLEEP_CLK_SEL__SHIFT) | (1 << MISC_CLK_CNTL__ZCLK_SEL__SHIFT); |
| 2565 | if (orig != data) | 2580 | if (orig != data) |
| 2566 | WREG32(MISC_CLK_CNTL, data); | 2581 | WREG32(mmMISC_CLK_CNTL, data); |
| 2567 | 2582 | ||
| 2568 | orig = data = RREG32(CG_CLKPIN_CNTL); | 2583 | orig = data = RREG32(mmCG_CLKPIN_CNTL); |
| 2569 | data &= ~BCLK_AS_XCLK; | 2584 | data &= ~CG_CLKPIN_CNTL__BCLK_AS_XCLK_MASK; |
| 2570 | if (orig != data) | 2585 | if (orig != data) |
| 2571 | WREG32(CG_CLKPIN_CNTL, data); | 2586 | WREG32(mmCG_CLKPIN_CNTL, data); |
| 2572 | 2587 | ||
| 2573 | orig = data = RREG32(CG_CLKPIN_CNTL_2); | 2588 | orig = data = RREG32(mmCG_CLKPIN_CNTL_2); |
| 2574 | data &= ~FORCE_BIF_REFCLK_EN; | 2589 | data &= ~CG_CLKPIN_CNTL_2__FORCE_BIF_REFCLK_EN_MASK; |
| 2575 | if (orig != data) | 2590 | if (orig != data) |
| 2576 | WREG32(CG_CLKPIN_CNTL_2, data); | 2591 | WREG32(mmCG_CLKPIN_CNTL_2, data); |
| 2577 | 2592 | ||
| 2578 | orig = data = RREG32(MPLL_BYPASSCLK_SEL); | 2593 | orig = data = RREG32(mmMPLL_BYPASSCLK_SEL); |
| 2579 | data &= ~MPLL_CLKOUT_SEL_MASK; | 2594 | data &= ~MPLL_BYPASSCLK_SEL__MPLL_CLKOUT_SEL_MASK; |
| 2580 | data |= MPLL_CLKOUT_SEL(4); | 2595 | data |= 4 << MPLL_BYPASSCLK_SEL__MPLL_CLKOUT_SEL__SHIFT; |
| 2581 | if (orig != data) | 2596 | if (orig != data) |
| 2582 | WREG32(MPLL_BYPASSCLK_SEL, data); | 2597 | WREG32(mmMPLL_BYPASSCLK_SEL, data); |
| 2583 | 2598 | ||
| 2584 | orig = data = RREG32(SPLL_CNTL_MODE); | 2599 | orig = data = RREG32(mmSPLL_CNTL_MODE); |
| 2585 | data &= ~SPLL_REFCLK_SEL_MASK; | 2600 | data &= ~SPLL_CNTL_MODE__SPLL_REFCLK_SEL_MASK; |
| 2586 | if (orig != data) | 2601 | if (orig != data) |
| 2587 | WREG32(SPLL_CNTL_MODE, data); | 2602 | WREG32(mmSPLL_CNTL_MODE, data); |
| 2588 | } | 2603 | } |
| 2589 | } | 2604 | } |
| 2590 | } else { | 2605 | } else { |
| 2591 | if (orig != data) | 2606 | if (orig != data) |
| 2592 | WREG32_PCIE_PORT(PCIE_LC_CNTL, data); | 2607 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data); |
| 2593 | } | 2608 | } |
| 2594 | 2609 | ||
| 2595 | orig = data = RREG32_PCIE(PCIE_CNTL2); | 2610 | orig = data = RREG32_PCIE(ixPCIE_CNTL2); |
| 2596 | data |= SLV_MEM_LS_EN | MST_MEM_LS_EN | REPLAY_MEM_LS_EN; | 2611 | data |= PCIE_CNTL2__SLV_MEM_LS_EN_MASK | PCIE_CNTL2__MST_MEM_LS_EN_MASK | PCIE_CNTL2__REPLAY_MEM_LS_EN_MASK; |
| 2597 | if (orig != data) | 2612 | if (orig != data) |
| 2598 | WREG32_PCIE(PCIE_CNTL2, data); | 2613 | WREG32_PCIE(ixPCIE_CNTL2, data); |
| 2599 | 2614 | ||
| 2600 | if (!disable_l0s) { | 2615 | if (!disable_l0s) { |
| 2601 | data = RREG32_PCIE_PORT(PCIE_LC_N_FTS_CNTL); | 2616 | data = RREG32_PCIE_PORT(ixPCIE_LC_N_FTS_CNTL); |
| 2602 | if((data & LC_N_FTS_MASK) == LC_N_FTS_MASK) { | 2617 | if((data & PCIE_LC_N_FTS_CNTL__LC_N_FTS_MASK) == PCIE_LC_N_FTS_CNTL__LC_N_FTS_MASK) { |
| 2603 | data = RREG32_PCIE(PCIE_LC_STATUS1); | 2618 | data = RREG32_PCIE(ixPCIE_LC_STATUS1); |
| 2604 | if ((data & LC_REVERSE_XMIT) && (data & LC_REVERSE_RCVR)) { | 2619 | if ((data & PCIE_LC_STATUS1__LC_REVERSE_XMIT_MASK) && (data & PCIE_LC_STATUS1__LC_REVERSE_RCVR_MASK)) { |
| 2605 | orig = data = RREG32_PCIE_PORT(PCIE_LC_CNTL); | 2620 | orig = data = RREG32_PCIE_PORT(ixPCIE_LC_CNTL); |
| 2606 | data &= ~LC_L0S_INACTIVITY_MASK; | 2621 | data &= ~PCIE_LC_CNTL__LC_L0S_INACTIVITY_MASK; |
| 2607 | if (orig != data) | 2622 | if (orig != data) |
| 2608 | WREG32_PCIE_PORT(PCIE_LC_CNTL, data); | 2623 | WREG32_PCIE_PORT(ixPCIE_LC_CNTL, data); |
| 2609 | } | 2624 | } |
| 2610 | } | 2625 | } |
| 2611 | } | 2626 | } |
| @@ -27,6 +27,8 @@ · 26 unmodified lines | |||
| 27 | #include "si.h" | 27 | #include "si.h" |
| 28 | #include "sid.h" | 28 | #include "sid.h" |
| 29 | 29 | ||
| 30 | #include "oss/oss_1_0_d.h" | ||
| 31 | #include "oss/oss_1_0_sh_mask.h" | ||
| 30 | const u32 sdma_offsets[SDMA_MAX_INSTANCE] = | 32 | const u32 sdma_offsets[SDMA_MAX_INSTANCE] = |
| 31 | { | 33 | { |
| 32 | DMA0_REGISTER_OFFSET, | 34 | DMA0_REGISTER_OFFSET, |
| @@ -38,17 +40,31 @@ static void si_dma_set_buffer_funcs(struct amdgpu_device *adev); · 5 unmodified lines | |||
| 38 | static void si_dma_set_vm_pte_funcs(struct amdgpu_device *adev); | 40 | static void si_dma_set_vm_pte_funcs(struct amdgpu_device *adev); |
| 39 | static void si_dma_set_irq_funcs(struct amdgpu_device *adev); | 41 | static void si_dma_set_irq_funcs(struct amdgpu_device *adev); |
| 40 | 42 | ||
| 43 | /** | ||
| 44 | * si_dma_ring_get_rptr - get the current read pointer | ||
| 45 | * | ||
| 46 | * @ring: amdgpu ring pointer | ||
| 47 | * | ||
| 48 | * Get the current rptr from the hardware (SI). | ||
| 49 | */ | ||
| 41 | static uint64_t si_dma_ring_get_rptr(struct amdgpu_ring *ring) | 50 | static uint64_t si_dma_ring_get_rptr(struct amdgpu_ring *ring) |
| 42 | { | 51 | { |
| 43 | return *ring->rptr_cpu_addr; | 52 | return *ring->rptr_cpu_addr; |
| 44 | } | 53 | } |
| 45 | 54 | ||
| 55 | /** | ||
| 56 | * si_dma_ring_get_wptr - get the current write pointer | ||
| 57 | * | ||
| 58 | * @ring: amdgpu ring pointer | ||
| 59 | * | ||
| 60 | * Get the current wptr from the hardware (SI). | ||
| 61 | */ | ||
| 46 | static uint64_t si_dma_ring_get_wptr(struct amdgpu_ring *ring) | 62 | static uint64_t si_dma_ring_get_wptr(struct amdgpu_ring *ring) |
| 47 | { | 63 | { |
| 48 | struct amdgpu_device *adev = ring->adev; | 64 | struct amdgpu_device *adev = ring->adev; |
| 49 | u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1; | 65 | u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1; |
| 50 | 66 | ||
| 51 | return (RREG32(DMA_RB_WPTR + sdma_offsets[me]) & 0x3fffc) >> 2; | 67 | return (RREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[me]) & 0x3fffc) >> 2; |
| 52 | } | 68 | } |
| 53 | 69 | ||
| 54 | static void si_dma_ring_set_wptr(struct amdgpu_ring *ring) | 70 | static void si_dma_ring_set_wptr(struct amdgpu_ring *ring) |
| @@ -56,7 +72,7 @@ static void si_dma_ring_set_wptr(struct amdgpu_ring *ring) · 1 unmodified line | |||
| 56 | struct amdgpu_device *adev = ring->adev; | 72 | struct amdgpu_device *adev = ring->adev; |
| 57 | u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1; | 73 | u32 me = (ring == &adev->sdma.instance[0].ring) ? 0 : 1; |
| 58 | 74 | ||
| 59 | WREG32(DMA_RB_WPTR + sdma_offsets[me], (ring->wptr << 2) & 0x3fffc); | 75 | WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[me], (ring->wptr << 2) & 0x3fffc); |
| 60 | } | 76 | } |
| 61 | 77 | ||
| 62 | static void si_dma_ring_emit_ib(struct amdgpu_ring *ring, | 78 | static void si_dma_ring_emit_ib(struct amdgpu_ring *ring, |
| @@ -117,9 +133,9 @@ static void si_dma_stop(struct amdgpu_device *adev) · 54 unmodified lines | |||
| 117 | 133 | ||
| 118 | for (i = 0; i < adev->sdma.num_instances; i++) { | 134 | for (i = 0; i < adev->sdma.num_instances; i++) { |
| 119 | /* dma0 */ | 135 | /* dma0 */ |
| 120 | rb_cntl = RREG32(DMA_RB_CNTL + sdma_offsets[i]); | 136 | rb_cntl = RREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i]); |
| 121 | rb_cntl &= ~DMA_RB_ENABLE; | 137 | rb_cntl &= ~DMA_GFX_RB_CNTL__RB_ENABLE_MASK; |
| 122 | WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl); | 138 | WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl); |
| 123 | } | 139 | } |
| 124 | } | 140 | } |
| 125 | 141 | ||
| @@ -133,44 +149,44 @@ static int si_dma_start(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 133 | for (i = 0; i < adev->sdma.num_instances; i++) { | 149 | for (i = 0; i < adev->sdma.num_instances; i++) { |
| 134 | ring = &adev->sdma.instance[i].ring; | 150 | ring = &adev->sdma.instance[i].ring; |
| 135 | 151 | ||
| 136 | WREG32(DMA_SEM_INCOMPLETE_TIMER_CNTL + sdma_offsets[i], 0); | 152 | WREG32(mmDMA_SEM_INCOMPLETE_TIMER_CNTL + sdma_offsets[i], 0); |
| 137 | WREG32(DMA_SEM_WAIT_FAIL_TIMER_CNTL + sdma_offsets[i], 0); | 153 | WREG32(mmDMA_SEM_WAIT_FAIL_TIMER_CNTL + sdma_offsets[i], 0); |
| 138 | 154 | ||
| 139 | /* Set ring buffer size in dwords */ | 155 | /* Set ring buffer size in dwords */ |
| 140 | rb_bufsz = order_base_2(ring->ring_size / 4); | 156 | rb_bufsz = order_base_2(ring->ring_size / 4); |
| 141 | rb_cntl = rb_bufsz << 1; | 157 | rb_cntl = rb_bufsz << 1; |
| 142 | #ifdef __BIG_ENDIAN | 158 | #ifdef __BIG_ENDIAN |
| 143 | rb_cntl |= DMA_RB_SWAP_ENABLE | DMA_RPTR_WRITEBACK_SWAP_ENABLE; | 159 | rb_cntl |= DMA_GFX_RB_CNTL__RB_SWAP_ENABLE_MASK | DMA_GFX_RB_CNTL__RPTR_WRITEBACK_SWAP_ENABLE_MASK; |
| 144 | #endif | 160 | #endif |
| 145 | WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl); | 161 | WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl); |
| 146 | 162 | ||
| 147 | /* Initialize the ring buffer's read and write pointers */ | 163 | /* Initialize the ring buffer's read and write pointers */ |
| 148 | WREG32(DMA_RB_RPTR + sdma_offsets[i], 0); | 164 | WREG32(mmDMA_GFX_RB_RPTR + sdma_offsets[i], 0); |
| 149 | WREG32(DMA_RB_WPTR + sdma_offsets[i], 0); | 165 | WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[i], 0); |
| 150 | 166 | ||
| 151 | rptr_addr = ring->rptr_gpu_addr; | 167 | rptr_addr = ring->rptr_gpu_addr; |
| 152 | 168 | ||
| 153 | WREG32(DMA_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr)); | 169 | WREG32(mmDMA_GFX_RB_RPTR_ADDR_LO + sdma_offsets[i], lower_32_bits(rptr_addr)); |
| 154 | WREG32(DMA_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF); | 170 | WREG32(mmDMA_GFX_RB_RPTR_ADDR_HI + sdma_offsets[i], upper_32_bits(rptr_addr) & 0xFF); |
| 155 | 171 | ||
| 156 | rb_cntl |= DMA_RPTR_WRITEBACK_ENABLE; | 172 | rb_cntl |= DMA_GFX_RB_CNTL__RPTR_WRITEBACK_ENABLE_MASK; |
| 157 | 173 | ||
| 158 | WREG32(DMA_RB_BASE + sdma_offsets[i], ring->gpu_addr >> 8); | 174 | WREG32(mmDMA_GFX_RB_BASE + sdma_offsets[i], ring->gpu_addr >> 8); |
| 159 | 175 | ||
| 160 | /* enable DMA IBs */ | 176 | /* enable DMA IBs */ |
| 161 | ib_cntl = DMA_IB_ENABLE | CMD_VMID_FORCE; | 177 | ib_cntl = DMA_GFX_IB_CNTL__IB_ENABLE_MASK | DMA_GFX_IB_CNTL__CMD_VMID_FORCE_MASK; |
| 162 | #ifdef __BIG_ENDIAN | 178 | #ifdef __BIG_ENDIAN |
| 163 | ib_cntl |= DMA_IB_SWAP_ENABLE; | 179 | ib_cntl |= DMA_GFX_IB_CNTL__IB_SWAP_ENABLE_MASK; |
| 164 | #endif | 180 | #endif |
| 165 | WREG32(DMA_IB_CNTL + sdma_offsets[i], ib_cntl); | 181 | WREG32(mmDMA_GFX_IB_CNTL + sdma_offsets[i], ib_cntl); |
| 166 | 182 | ||
| 167 | dma_cntl = RREG32(DMA_CNTL + sdma_offsets[i]); | 183 | dma_cntl = RREG32(mmDMA_CNTL + sdma_offsets[i]); |
| 168 | dma_cntl &= ~CTXEMPTY_INT_ENABLE; | 184 | dma_cntl &= ~DMA_CNTL__CTXEMPTY_INT_ENABLE_MASK; |
| 169 | WREG32(DMA_CNTL + sdma_offsets[i], dma_cntl); | 185 | WREG32(mmDMA_CNTL + sdma_offsets[i], dma_cntl); |
| 170 | 186 | ||
| 171 | ring->wptr = 0; | 187 | ring->wptr = 0; |
| 172 | WREG32(DMA_RB_WPTR + sdma_offsets[i], ring->wptr << 2); | 188 | WREG32(mmDMA_GFX_RB_WPTR + sdma_offsets[i], ring->wptr << 2); |
| 173 | WREG32(DMA_RB_CNTL + sdma_offsets[i], rb_cntl | DMA_RB_ENABLE); | 189 | WREG32(mmDMA_GFX_RB_CNTL + sdma_offsets[i], rb_cntl | DMA_GFX_RB_CNTL__RB_ENABLE_MASK); |
| 174 | 190 | ||
| 175 | r = amdgpu_ring_test_helper(ring); | 191 | r = amdgpu_ring_test_helper(ring); |
| 176 | if (r) | 192 | if (r) |
| @@ -461,7 +477,7 @@ static int si_dma_early_init(struct amdgpu_ip_block *ip_block) · 284 unmodified lines | |||
| 461 | { | 477 | { |
| 462 | struct amdgpu_device *adev = ip_block->adev; | 478 | struct amdgpu_device *adev = ip_block->adev; |
| 463 | 479 | ||
| 464 | adev->sdma.num_instances = 2; | 480 | adev->sdma.num_instances = SDMA_MAX_INSTANCE; |
| 465 | 481 | ||
| 466 | si_dma_set_ring_funcs(adev); | 482 | si_dma_set_ring_funcs(adev); |
| 467 | si_dma_set_buffer_funcs(adev); | 483 | si_dma_set_buffer_funcs(adev); |
| @@ -545,9 +561,9 @@ static bool si_dma_is_idle(struct amdgpu_ip_block *ip_block) · 77 unmodified lines | |||
| 545 | { | 561 | { |
| 546 | struct amdgpu_device *adev = ip_block->adev; | 562 | struct amdgpu_device *adev = ip_block->adev; |
| 547 | 563 | ||
| 548 | u32 tmp = RREG32(SRBM_STATUS2); | 564 | u32 tmp = RREG32(mmSRBM_STATUS2); |
| 549 | 565 | ||
| 550 | if (tmp & (DMA_BUSY_MASK | DMA1_BUSY_MASK)) | 566 | if (tmp & (SRBM_STATUS2__DMA_BUSY_MASK | SRBM_STATUS2__DMA1_BUSY_MASK)) |
| 551 | return false; | 567 | return false; |
| 552 | 568 | ||
| 553 | return true; | 569 | return true; |
| @@ -583,14 +599,14 @@ static int si_dma_set_trap_irq_state(struct amdgpu_device *adev, · 29 unmodified lines | |||
| 583 | case AMDGPU_SDMA_IRQ_INSTANCE0: | 599 | case AMDGPU_SDMA_IRQ_INSTANCE0: |
| 584 | switch (state) { | 600 | switch (state) { |
| 585 | case AMDGPU_IRQ_STATE_DISABLE: | 601 | case AMDGPU_IRQ_STATE_DISABLE: |
| 586 | sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET); | 602 | sdma_cntl = RREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET); |
| 587 | sdma_cntl &= ~TRAP_ENABLE; | 603 | sdma_cntl &= ~DMA_CNTL__TRAP_ENABLE_MASK; |
| 588 | WREG32(DMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl); | 604 | WREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl); |
| 589 | break; | 605 | break; |
| 590 | case AMDGPU_IRQ_STATE_ENABLE: | 606 | case AMDGPU_IRQ_STATE_ENABLE: |
| 591 | sdma_cntl = RREG32(DMA_CNTL + DMA0_REGISTER_OFFSET); | 607 | sdma_cntl = RREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET); |
| 592 | sdma_cntl |= TRAP_ENABLE; | 608 | sdma_cntl |= DMA_CNTL__TRAP_ENABLE_MASK; |
| 593 | WREG32(DMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl); | 609 | WREG32(mmDMA_CNTL + DMA0_REGISTER_OFFSET, sdma_cntl); |
| 594 | break; | 610 | break; |
| 595 | default: | 611 | default: |
| 596 | break; | 612 | break; |
| @@ -599,14 +615,14 @@ static int si_dma_set_trap_irq_state(struct amdgpu_device *adev, · 2 unmodified lines | |||
| 599 | case AMDGPU_SDMA_IRQ_INSTANCE1: | 615 | case AMDGPU_SDMA_IRQ_INSTANCE1: |
| 600 | switch (state) { | 616 | switch (state) { |
| 601 | case AMDGPU_IRQ_STATE_DISABLE: | 617 | case AMDGPU_IRQ_STATE_DISABLE: |
| 602 | sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET); | 618 | sdma_cntl = RREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET); |
| 603 | sdma_cntl &= ~TRAP_ENABLE; | 619 | sdma_cntl &= ~DMA_CNTL__TRAP_ENABLE_MASK; |
| 604 | WREG32(DMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl); | 620 | WREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl); |
| 605 | break; | 621 | break; |
| 606 | case AMDGPU_IRQ_STATE_ENABLE: | 622 | case AMDGPU_IRQ_STATE_ENABLE: |
| 607 | sdma_cntl = RREG32(DMA_CNTL + DMA1_REGISTER_OFFSET); | 623 | sdma_cntl = RREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET); |
| 608 | sdma_cntl |= TRAP_ENABLE; | 624 | sdma_cntl |= DMA_CNTL__TRAP_ENABLE_MASK; |
| 609 | WREG32(DMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl); | 625 | WREG32(mmDMA_CNTL + DMA1_REGISTER_OFFSET, sdma_cntl); |
| 610 | break; | 626 | break; |
| 611 | default: | 627 | default: |
| 612 | break; | 628 | break; |
| @@ -645,11 +661,11 @@ static int si_dma_set_clockgating_state(struct amdgpu_ip_block *ip_block, · 32 unmodified lines | |||
| 645 | offset = DMA0_REGISTER_OFFSET; | 661 | offset = DMA0_REGISTER_OFFSET; |
| 646 | else | 662 | else |
| 647 | offset = DMA1_REGISTER_OFFSET; | 663 | offset = DMA1_REGISTER_OFFSET; |
| 648 | orig = data = RREG32(DMA_POWER_CNTL + offset); | 664 | orig = data = RREG32(mmDMA_POWER_CNTL + offset); |
| 649 | data &= ~MEM_POWER_OVERRIDE; | 665 | data &= ~DMA_POWER_CNTL__MEM_POWER_OVERRIDE_MASK; |
| 650 | if (data != orig) | 666 | if (data != orig) |
| 651 | WREG32(DMA_POWER_CNTL + offset, data); | 667 | WREG32(mmDMA_POWER_CNTL + offset, data); |
| 652 | WREG32(DMA_CLK_CTRL + offset, 0x00000100); | 668 | WREG32(mmDMA_CLK_CTRL + offset, 0x00000100); |
| 653 | } | 669 | } |
| 654 | } else { | 670 | } else { |
| 655 | for (i = 0; i < adev->sdma.num_instances; i++) { | 671 | for (i = 0; i < adev->sdma.num_instances; i++) { |
| @@ -657,15 +673,15 @@ static int si_dma_set_clockgating_state(struct amdgpu_ip_block *ip_block, · 1 unmodified line | |||
| 657 | offset = DMA0_REGISTER_OFFSET; | 673 | offset = DMA0_REGISTER_OFFSET; |
| 658 | else | 674 | else |
| 659 | offset = DMA1_REGISTER_OFFSET; | 675 | offset = DMA1_REGISTER_OFFSET; |
| 660 | orig = data = RREG32(DMA_POWER_CNTL + offset); | 676 | orig = data = RREG32(mmDMA_POWER_CNTL + offset); |
| 661 | data |= MEM_POWER_OVERRIDE; | 677 | data |= DMA_POWER_CNTL__MEM_POWER_OVERRIDE_MASK; |
| 662 | if (data != orig) | 678 | if (data != orig) |
| 663 | WREG32(DMA_POWER_CNTL + offset, data); | 679 | WREG32(mmDMA_POWER_CNTL + offset, data); |
| 664 | 680 | ||
| 665 | orig = data = RREG32(DMA_CLK_CTRL + offset); | 681 | orig = data = RREG32(mmDMA_CLK_CTRL + offset); |
| 666 | data = 0xff000000; | 682 | data = 0xff000000; |
| 667 | if (data != orig) | 683 | if (data != orig) |
| 668 | WREG32(DMA_CLK_CTRL + offset, data); | 684 | WREG32(mmDMA_CLK_CTRL + offset, data); |
| 669 | } | 685 | } |
| 670 | } | 686 | } |
| 671 | 687 | ||
| @@ -679,11 +695,11 @@ static int si_dma_set_powergating_state(struct amdgpu_ip_block *ip_block, · 7 unmodified lines | |||
| 679 | 695 | ||
| 680 | struct amdgpu_device *adev = ip_block->adev; | 696 | struct amdgpu_device *adev = ip_block->adev; |
| 681 | 697 | ||
| 682 | WREG32(DMA_PGFSM_WRITE, 0x00002000); | 698 | WREG32(mmDMA_PGFSM_WRITE, 0x00002000); |
| 683 | WREG32(DMA_PGFSM_CONFIG, 0x100010ff); | 699 | WREG32(mmDMA_PGFSM_CONFIG, 0x100010ff); |
| 684 | 700 | ||
| 685 | for (tmp = 0; tmp < 5; tmp++) | 701 | for (tmp = 0; tmp < 5; tmp++) |
| 686 | WREG32(DMA_PGFSM_WRITE, 0); | 702 | WREG32(mmDMA_PGFSM_WRITE, 0); |
| 687 | 703 | ||
| 688 | return 0; | 704 | return 0; |
| 689 | } | 705 | } |
| @@ -23,115 +23,15 @@ · 22 unmodified lines | |||
| 23 | #ifndef SI_ENUMS_H | 23 | #ifndef SI_ENUMS_H |
| 24 | #define SI_ENUMS_H | 24 | #define SI_ENUMS_H |
| 25 | 25 | ||
| 26 | #define VBLANK_INT_MASK (1 << 0) | ||
| 27 | #define DC_HPDx_INT_EN (1 << 16) | ||
| 28 | #define VBLANK_ACK (1 << 4) | ||
| 29 | #define VLINE_ACK (1 << 4) | ||
| 30 | |||
| 31 | #define CURSOR_WIDTH 64 | ||
| 32 | #define CURSOR_HEIGHT 64 | ||
| 33 | |||
| 34 | #define VGA_VSTATUS_CNTL 0xFFFCFFFF | ||
| 35 | #define PRIORITY_MARK_MASK 0x7fff | 26 | #define PRIORITY_MARK_MASK 0x7fff |
| 36 | #define PRIORITY_OFF (1 << 16) | 27 | #define PRIORITY_OFF (1 << 16) |
| 37 | #define PRIORITY_ALWAYS_ON (1 << 20) | 28 | #define PRIORITY_ALWAYS_ON (1 << 20) |
| 38 | #define INTERLEAVE_EN (1 << 0) | ||
| 39 | |||
| 40 | #define LATENCY_WATERMARK_MASK(x) ((x) << 16) | ||
| 41 | #define DC_LB_MEMORY_CONFIG(x) ((x) << 20) | ||
| 42 | #define ICON_DEGAMMA_MODE(x) (((x) & 0x3) << 8) | ||
| 43 | |||
| 44 | #define GRPH_ENDIAN_SWAP(x) (((x) & 0x3) << 0) | ||
| 45 | #define GRPH_ENDIAN_NONE 0 | ||
| 46 | #define GRPH_ENDIAN_8IN16 1 | ||
| 47 | #define GRPH_ENDIAN_8IN32 2 | ||
| 48 | #define GRPH_ENDIAN_8IN64 3 | ||
| 49 | #define GRPH_RED_CROSSBAR(x) (((x) & 0x3) << 4) | ||
| 50 | #define GRPH_RED_SEL_R 0 | ||
| 51 | #define GRPH_RED_SEL_G 1 | ||
| 52 | #define GRPH_RED_SEL_B 2 | ||
| 53 | #define GRPH_RED_SEL_A 3 | ||
| 54 | #define GRPH_GREEN_CROSSBAR(x) (((x) & 0x3) << 6) | ||
| 55 | #define GRPH_GREEN_SEL_G 0 | ||
| 56 | #define GRPH_GREEN_SEL_B 1 | ||
| 57 | #define GRPH_GREEN_SEL_A 2 | ||
| 58 | #define GRPH_GREEN_SEL_R 3 | ||
| 59 | #define GRPH_BLUE_CROSSBAR(x) (((x) & 0x3) << 8) | ||
| 60 | #define GRPH_BLUE_SEL_B 0 | ||
| 61 | #define GRPH_BLUE_SEL_A 1 | ||
| 62 | #define GRPH_BLUE_SEL_R 2 | ||
| 63 | #define GRPH_BLUE_SEL_G 3 | ||
| 64 | #define GRPH_ALPHA_CROSSBAR(x) (((x) & 0x3) << 10) | ||
| 65 | #define GRPH_ALPHA_SEL_A 0 | ||
| 66 | #define GRPH_ALPHA_SEL_R 1 | ||
| 67 | #define GRPH_ALPHA_SEL_G 2 | ||
| 68 | #define GRPH_ALPHA_SEL_B 3 | ||
| 69 | |||
| 70 | #define GRPH_DEPTH(x) (((x) & 0x3) << 0) | ||
| 71 | #define GRPH_DEPTH_8BPP 0 | ||
| 72 | #define GRPH_DEPTH_16BPP 1 | ||
| 73 | #define GRPH_DEPTH_32BPP 2 | ||
| 74 | |||
| 75 | #define GRPH_FORMAT(x) (((x) & 0x7) << 8) | ||
| 76 | #define GRPH_FORMAT_INDEXED 0 | ||
| 77 | #define GRPH_FORMAT_ARGB1555 0 | ||
| 78 | #define GRPH_FORMAT_ARGB565 1 | ||
| 79 | #define GRPH_FORMAT_ARGB4444 2 | ||
| 80 | #define GRPH_FORMAT_AI88 3 | ||
| 81 | #define GRPH_FORMAT_MONO16 4 | ||
| 82 | #define GRPH_FORMAT_BGRA5551 5 | ||
| 83 | #define GRPH_FORMAT_ARGB8888 0 | ||
| 84 | #define GRPH_FORMAT_ARGB2101010 1 | ||
| 85 | #define GRPH_FORMAT_32BPP_DIG 2 | ||
| 86 | #define GRPH_FORMAT_8B_ARGB2101010 3 | ||
| 87 | #define GRPH_FORMAT_BGRA1010102 4 | ||
| 88 | #define GRPH_FORMAT_8B_BGRA1010102 5 | ||
| 89 | #define GRPH_FORMAT_RGB111110 6 | ||
| 90 | #define GRPH_FORMAT_BGR101111 7 | ||
| 91 | |||
| 92 | #define GRPH_NUM_BANKS(x) (((x) & 0x3) << 2) | ||
| 93 | #define GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20) | ||
| 94 | #define GRPH_ARRAY_LINEAR_GENERAL 0 | ||
| 95 | #define GRPH_ARRAY_LINEAR_ALIGNED 1 | ||
| 96 | #define GRPH_ARRAY_1D_TILED_THIN1 2 | ||
| 97 | #define GRPH_ARRAY_2D_TILED_THIN1 4 | ||
| 98 | #define GRPH_TILE_SPLIT(x) (((x) & 0x7) << 13) | ||
| 99 | #define GRPH_BANK_WIDTH(x) (((x) & 0x3) << 6) | ||
| 100 | #define GRPH_BANK_HEIGHT(x) (((x) & 0x3) << 11) | ||
| 101 | #define GRPH_MACRO_TILE_ASPECT(x) (((x) & 0x3) << 18) | ||
| 102 | #define GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20) | ||
| 103 | #define GRPH_PIPE_CONFIG(x) (((x) & 0x1f) << 24) | ||
| 104 | |||
| 105 | #define CURSOR_EN (1 << 0) | ||
| 106 | #define CURSOR_MODE(x) (((x) & 0x3) << 8) | ||
| 107 | #define CURSOR_MONO 0 | ||
| 108 | #define CURSOR_24_1 1 | ||
| 109 | #define CURSOR_24_8_PRE_MULT 2 | ||
| 110 | #define CURSOR_24_8_UNPRE_MULT 3 | ||
| 111 | #define CURSOR_2X_MAGNIFY (1 << 16) | ||
| 112 | #define CURSOR_FORCE_MC_ON (1 << 20) | ||
| 113 | #define CURSOR_URGENT_CONTROL(x) (((x) & 0x7) << 24) | ||
| 114 | #define CURSOR_URGENT_ALWAYS 0 | ||
| 115 | #define CURSOR_URGENT_1_8 1 | ||
| 116 | #define CURSOR_URGENT_1_4 2 | ||
| 117 | #define CURSOR_URGENT_3_8 3 | ||
| 118 | #define CURSOR_URGENT_1_2 4 | ||
| 119 | #define CURSOR_UPDATE_PENDING (1 << 0) | ||
| 120 | #define CURSOR_UPDATE_TAKEN (1 << 1) | ||
| 121 | #define CURSOR_UPDATE_LOCK (1 << 16) | ||
| 122 | #define CURSOR_DISABLE_MULTIPLE_UPDATE (1 << 24) | ||
| 123 | |||
| 124 | |||
| 125 | #define ES_AND_GS_AUTO 3 | ||
| 126 | #define RADEON_PACKET_TYPE3 3 | ||
| 127 | #define CE_PARTITION_BASE 3 | ||
| 128 | #define BUF_SWAP_32BIT (2 << 16) | ||
| 129 | 29 | ||
| 130 | #define GFX_POWER_STATUS (1 << 1) | 30 | #define GFX_POWER_STATUS (1 << 1) |
| 131 | #define GFX_CLOCK_STATUS (1 << 2) | 31 | #define GFX_CLOCK_STATUS (1 << 2) |
| 132 | #define GFX_LS_STATUS (1 << 3) | 32 | #define GFX_LS_STATUS (1 << 3) |
| 133 | #define RLC_BUSY_STATUS (1 << 0) | ||
| 134 | 33 | ||
| 34 | #define RLC_BUSY_STATUS (1 << 0) | ||
| 135 | #define RLC_PUD(x) ((x) << 0) | 35 | #define RLC_PUD(x) ((x) << 0) |
| 136 | #define RLC_PUD_MASK (0xff << 0) | 36 | #define RLC_PUD_MASK (0xff << 0) |
| 137 | #define RLC_PDD(x) ((x) << 8) | 37 | #define RLC_PDD(x) ((x) << 8) |
| @@ -140,140 +40,8 @@ · 2 unmodified lines | |||
| 140 | #define RLC_TTPD_MASK (0xff << 16) | 40 | #define RLC_TTPD_MASK (0xff << 16) |
| 141 | #define RLC_MSD(x) ((x) << 24) | 41 | #define RLC_MSD(x) ((x) << 24) |
| 142 | #define RLC_MSD_MASK (0xff << 24) | 42 | #define RLC_MSD_MASK (0xff << 24) |
| 143 | #define WRITE_DATA_ENGINE_SEL(x) ((x) << 30) | ||
| 144 | #define WRITE_DATA_DST_SEL(x) ((x) << 8) | ||
| 145 | #define EVENT_TYPE(x) ((x) << 0) | ||
| 146 | #define EVENT_INDEX(x) ((x) << 8) | ||
| 147 | #define WAIT_REG_MEM_MEM_SPACE(x) ((x) << 4) | ||
| 148 | #define WAIT_REG_MEM_FUNCTION(x) ((x) << 0) | ||
| 149 | #define WAIT_REG_MEM_ENGINE(x) ((x) << 8) | ||
| 150 | 43 | ||
| 151 | #define GFX6_NUM_GFX_RINGS 1 | ||
| 152 | #define GFX6_NUM_COMPUTE_RINGS 2 | ||
| 153 | #define RLC_SAVE_AND_RESTORE_STARTING_OFFSET 0x90 | 44 | #define RLC_SAVE_AND_RESTORE_STARTING_OFFSET 0x90 |
| 154 | #define RLC_CLEAR_STATE_DESCRIPTOR_OFFSET 0x3D | 45 | #define RLC_CLEAR_STATE_DESCRIPTOR_OFFSET 0x3D |
| 155 | 46 | ||
| 156 | #define PACKET3(op, n) ((RADEON_PACKET_TYPE3 << 30) | \ | ||
| 157 | (((op) & 0xFF) << 8) | \ | ||
| 158 | ((n) & 0x3FFF) << 16) | ||
| 159 | #define PACKET3_COMPUTE(op, n) (PACKET3(op, n) | 1 << 1) | ||
| 160 | #define PACKET3_NOP 0x10 | ||
| 161 | #define PACKET3_SET_BASE 0x11 | ||
| 162 | #define PACKET3_BASE_INDEX(x) ((x) << 0) | ||
| 163 | #define PACKET3_CLEAR_STATE 0x12 | ||
| 164 | #define PACKET3_INDEX_BUFFER_SIZE 0x13 | ||
| 165 | #define PACKET3_DISPATCH_DIRECT 0x15 | ||
| 166 | #define PACKET3_DISPATCH_INDIRECT 0x16 | ||
| 167 | #define PACKET3_ALLOC_GDS 0x1B | ||
| 168 | #define PACKET3_WRITE_GDS_RAM 0x1C | ||
| 169 | #define PACKET3_ATOMIC_GDS 0x1D | ||
| 170 | #define PACKET3_ATOMIC 0x1E | ||
| 171 | #define PACKET3_OCCLUSION_QUERY 0x1F | ||
| 172 | #define PACKET3_SET_PREDICATION 0x20 | ||
| 173 | #define PACKET3_REG_RMW 0x21 | ||
| 174 | #define PACKET3_COND_EXEC 0x22 | ||
| 175 | #define PACKET3_PRED_EXEC 0x23 | ||
| 176 | #define PACKET3_DRAW_INDIRECT 0x24 | ||
| 177 | #define PACKET3_DRAW_INDEX_INDIRECT 0x25 | ||
| 178 | #define PACKET3_INDEX_BASE 0x26 | ||
| 179 | #define PACKET3_DRAW_INDEX_2 0x27 | ||
| 180 | #define PACKET3_CONTEXT_CONTROL 0x28 | ||
| 181 | #define PACKET3_INDEX_TYPE 0x2A | ||
| 182 | #define PACKET3_DRAW_INDIRECT_MULTI 0x2C | ||
| 183 | #define PACKET3_DRAW_INDEX_AUTO 0x2D | ||
| 184 | #define PACKET3_DRAW_INDEX_IMMD 0x2E | ||
| 185 | #define PACKET3_NUM_INSTANCES 0x2F | ||
| 186 | #define PACKET3_DRAW_INDEX_MULTI_AUTO 0x30 | ||
| 187 | #define PACKET3_INDIRECT_BUFFER_CONST 0x31 | ||
| 188 | #define PACKET3_INDIRECT_BUFFER 0x3F | ||
| 189 | #define PACKET3_STRMOUT_BUFFER_UPDATE 0x34 | ||
| 190 | #define PACKET3_DRAW_INDEX_OFFSET_2 0x35 | ||
| 191 | #define PACKET3_DRAW_INDEX_MULTI_ELEMENT 0x36 | ||
| 192 | #define PACKET3_WRITE_DATA 0x37 | ||
| 193 | #define PACKET3_DRAW_INDEX_INDIRECT_MULTI 0x38 | ||
| 194 | #define PACKET3_MEM_SEMAPHORE 0x39 | ||
| 195 | #define PACKET3_MPEG_INDEX 0x3A | ||
| 196 | #define PACKET3_COPY_DW 0x3B | ||
| 197 | #define PACKET3_WAIT_REG_MEM 0x3C | ||
| 198 | #define PACKET3_MEM_WRITE 0x3D | ||
| 199 | #define PACKET3_COPY_DATA 0x40 | ||
| 200 | #define PACKET3_CP_DMA 0x41 | ||
| 201 | # define PACKET3_CP_DMA_DST_SEL(x) ((x) << 20) | ||
| 202 | # define PACKET3_CP_DMA_ENGINE(x) ((x) << 27) | ||
| 203 | # define PACKET3_CP_DMA_SRC_SEL(x) ((x) << 29) | ||
| 204 | # define PACKET3_CP_DMA_CP_SYNC (1 << 31) | ||
| 205 | # define PACKET3_CP_DMA_DIS_WC (1 << 21) | ||
| 206 | # define PACKET3_CP_DMA_CMD_SRC_SWAP(x) ((x) << 22) | ||
| 207 | # define PACKET3_CP_DMA_CMD_DST_SWAP(x) ((x) << 24) | ||
| 208 | # define PACKET3_CP_DMA_CMD_SAS (1 << 26) | ||
| 209 | # define PACKET3_CP_DMA_CMD_DAS (1 << 27) | ||
| 210 | # define PACKET3_CP_DMA_CMD_SAIC (1 << 28) | ||
| 211 | # define PACKET3_CP_DMA_CMD_DAIC (1 << 29) | ||
| 212 | # define PACKET3_CP_DMA_CMD_RAW_WAIT (1 << 30) | ||
| 213 | #define PACKET3_PFP_SYNC_ME 0x42 | ||
| 214 | #define PACKET3_SURFACE_SYNC 0x43 | ||
| 215 | # define PACKET3_DEST_BASE_0_ENA (1 << 0) | ||
| 216 | # define PACKET3_DEST_BASE_1_ENA (1 << 1) | ||
| 217 | # define PACKET3_CB0_DEST_BASE_ENA (1 << 6) | ||
| 218 | # define PACKET3_CB1_DEST_BASE_ENA (1 << 7) | ||
| 219 | # define PACKET3_CB2_DEST_BASE_ENA (1 << 8) | ||
| 220 | # define PACKET3_CB3_DEST_BASE_ENA (1 << 9) | ||
| 221 | # define PACKET3_CB4_DEST_BASE_ENA (1 << 10) | ||
| 222 | # define PACKET3_CB5_DEST_BASE_ENA (1 << 11) | ||
| 223 | # define PACKET3_CB6_DEST_BASE_ENA (1 << 12) | ||
| 224 | # define PACKET3_CB7_DEST_BASE_ENA (1 << 13) | ||
| 225 | # define PACKET3_DB_DEST_BASE_ENA (1 << 14) | ||
| 226 | # define PACKET3_DEST_BASE_2_ENA (1 << 19) | ||
| 227 | # define PACKET3_DEST_BASE_3_ENA (1 << 21) | ||
| 228 | # define PACKET3_TCL1_ACTION_ENA (1 << 22) | ||
| 229 | # define PACKET3_TC_ACTION_ENA (1 << 23) | ||
| 230 | # define PACKET3_CB_ACTION_ENA (1 << 25) | ||
| 231 | # define PACKET3_DB_ACTION_ENA (1 << 26) | ||
| 232 | # define PACKET3_SH_KCACHE_ACTION_ENA (1 << 27) | ||
| 233 | # define PACKET3_SH_ICACHE_ACTION_ENA (1 << 29) | ||
| 234 | #define PACKET3_ME_INITIALIZE 0x44 | ||
| 235 | #define PACKET3_ME_INITIALIZE_DEVICE_ID(x) ((x) << 16) | ||
| 236 | #define PACKET3_COND_WRITE 0x45 | ||
| 237 | #define PACKET3_EVENT_WRITE 0x46 | ||
| 238 | #define PACKET3_EVENT_WRITE_EOP 0x47 | ||
| 239 | #define PACKET3_EVENT_WRITE_EOS 0x48 | ||
| 240 | #define PACKET3_PREAMBLE_CNTL 0x4A | ||
| 241 | # define PACKET3_PREAMBLE_BEGIN_CLEAR_STATE (2 << 28) | ||
| 242 | # define PACKET3_PREAMBLE_END_CLEAR_STATE (3 << 28) | ||
| 243 | #define PACKET3_ONE_REG_WRITE 0x57 | ||
| 244 | #define PACKET3_LOAD_CONFIG_REG 0x5F | ||
| 245 | #define PACKET3_LOAD_CONTEXT_REG 0x60 | ||
| 246 | #define PACKET3_LOAD_SH_REG 0x61 | ||
| 247 | #define PACKET3_SET_CONFIG_REG 0x68 | ||
| 248 | #define PACKET3_SET_CONFIG_REG_START 0x00002000 | ||
| 249 | #define PACKET3_SET_CONFIG_REG_END 0x00002c00 | ||
| 250 | #define PACKET3_SET_CONTEXT_REG 0x69 | ||
| 251 | #define PACKET3_SET_CONTEXT_REG_START 0x000a000 | ||
| 252 | #define PACKET3_SET_CONTEXT_REG_END 0x000a400 | ||
| 253 | #define PACKET3_SET_CONTEXT_REG_INDIRECT 0x73 | ||
| 254 | #define PACKET3_SET_RESOURCE_INDIRECT 0x74 | ||
| 255 | #define PACKET3_SET_SH_REG 0x76 | ||
| 256 | #define PACKET3_SET_SH_REG_START 0x00002c00 | ||
| 257 | #define PACKET3_SET_SH_REG_END 0x00003000 | ||
| 258 | #define PACKET3_SET_SH_REG_OFFSET 0x77 | ||
| 259 | #define PACKET3_ME_WRITE 0x7A | ||
| 260 | #define PACKET3_SCRATCH_RAM_WRITE 0x7D | ||
| 261 | #define PACKET3_SCRATCH_RAM_READ 0x7E | ||
| 262 | #define PACKET3_CE_WRITE 0x7F | ||
| 263 | #define PACKET3_LOAD_CONST_RAM 0x80 | ||
| 264 | #define PACKET3_WRITE_CONST_RAM 0x81 | ||
| 265 | #define PACKET3_WRITE_CONST_RAM_OFFSET 0x82 | ||
| 266 | #define PACKET3_DUMP_CONST_RAM 0x83 | ||
| 267 | #define PACKET3_INCREMENT_CE_COUNTER 0x84 | ||
| 268 | #define PACKET3_INCREMENT_DE_COUNTER 0x85 | ||
| 269 | #define PACKET3_WAIT_ON_CE_COUNTER 0x86 | ||
| 270 | #define PACKET3_WAIT_ON_DE_COUNTER 0x87 | ||
| 271 | #define PACKET3_WAIT_ON_DE_COUNTER_DIFF 0x88 | ||
| 272 | #define PACKET3_SET_CE_DE_COUNTERS 0x89 | ||
| 273 | #define PACKET3_WAIT_ON_AVAIL_BUFFER 0x8A | ||
| 274 | #define PACKET3_SWITCH_BUFFER 0x8B | ||
| 275 | #define PACKET3_SEM_WAIT_ON_SIGNAL (0x1 << 12) | ||
| 276 | #define PACKET3_SEM_SEL_SIGNAL (0x6 << 29) | ||
| 277 | #define PACKET3_SEM_SEL_WAIT (0x7 << 29) | ||
| 278 | |||
| 279 | #endif | 47 | #endif |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | |||
| 27 | #include "amdgpu_ih.h" | 27 | #include "amdgpu_ih.h" |
| 28 | #include "sid.h" | 28 | #include "sid.h" |
| 29 | #include "si_ih.h" | 29 | #include "si_ih.h" |
| 30 | |||
| 30 | #include "oss/oss_1_0_d.h" | 31 | #include "oss/oss_1_0_d.h" |
| 31 | #include "oss/oss_1_0_sh_mask.h" | 32 | #include "oss/oss_1_0_sh_mask.h" |
| 32 | 33 | ||
| @@ -213,7 +214,7 @@ static int si_ih_resume(struct amdgpu_ip_block *ip_block) · 180 unmodified lines | |||
| 213 | static bool si_ih_is_idle(struct amdgpu_ip_block *ip_block) | 214 | static bool si_ih_is_idle(struct amdgpu_ip_block *ip_block) |
| 214 | { | 215 | { |
| 215 | struct amdgpu_device *adev = ip_block->adev; | 216 | struct amdgpu_device *adev = ip_block->adev; |
| 216 | u32 tmp = RREG32(SRBM_STATUS); | 217 | u32 tmp = RREG32(mmSRBM_STATUS); |
| 217 | 218 | ||
| 218 | if (tmp & SRBM_STATUS__IH_BUSY_MASK) | 219 | if (tmp & SRBM_STATUS__IH_BUSY_MASK) |
| 219 | return false; | 220 | return false; |
| @@ -239,23 +240,23 @@ static int si_ih_soft_reset(struct amdgpu_ip_block *ip_block) · 19 unmodified lines | |||
| 239 | struct amdgpu_device *adev = ip_block->adev; | 240 | struct amdgpu_device *adev = ip_block->adev; |
| 240 | 241 | ||
| 241 | u32 srbm_soft_reset = 0; | 242 | u32 srbm_soft_reset = 0; |
| 242 | u32 tmp = RREG32(SRBM_STATUS); | 243 | u32 tmp = RREG32(mmSRBM_STATUS); |
| 243 | 244 | ||
| 244 | if (tmp & SRBM_STATUS__IH_BUSY_MASK) | 245 | if (tmp & SRBM_STATUS__IH_BUSY_MASK) |
| 245 | srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_IH_MASK; | 246 | srbm_soft_reset |= SRBM_SOFT_RESET__SOFT_RESET_IH_MASK; |
| 246 | 247 | ||
| 247 | if (srbm_soft_reset) { | 248 | if (srbm_soft_reset) { |
| 248 | tmp = RREG32(SRBM_SOFT_RESET); | 249 | tmp = RREG32(mmSRBM_SOFT_RESET); |
| 249 | tmp |= srbm_soft_reset; | 250 | tmp |= srbm_soft_reset; |
| 250 | dev_info(adev->dev, "SRBM_SOFT_RESET=0x%08X\n", tmp); | 251 | dev_info(adev->dev, "mmSRBM_SOFT_RESET=0x%08X\n", tmp); |
| 251 | WREG32(SRBM_SOFT_RESET, tmp); | 252 | WREG32(mmSRBM_SOFT_RESET, tmp); |
| 252 | tmp = RREG32(SRBM_SOFT_RESET); | 253 | tmp = RREG32(mmSRBM_SOFT_RESET); |
| 253 | 254 | ||
| 254 | udelay(50); | 255 | udelay(50); |
| 255 | 256 | ||
| 256 | tmp &= ~srbm_soft_reset; | 257 | tmp &= ~srbm_soft_reset; |
| 257 | WREG32(SRBM_SOFT_RESET, tmp); | 258 | WREG32(mmSRBM_SOFT_RESET, tmp); |
| 258 | tmp = RREG32(SRBM_SOFT_RESET); | 259 | tmp = RREG32(mmSRBM_SOFT_RESET); |
| 259 | 260 | ||
| 260 | udelay(50); | 261 | udelay(50); |
| 261 | } | 262 | } |
| @@ -24,43 +24,12 @@ · 23 unmodified lines | |||
| 24 | #ifndef SI_H | 24 | #ifndef SI_H |
| 25 | #define SI_H | 25 | #define SI_H |
| 26 | 26 | ||
| 27 | #define TAHITI_RB_BITMAP_WIDTH_PER_SH 2 | ||
| 28 | |||
| 29 | #define SI_MAX_SH_GPRS 256 | ||
| 30 | #define SI_MAX_TEMP_GPRS 16 | ||
| 31 | #define SI_MAX_SH_THREADS 256 | ||
| 32 | #define SI_MAX_SH_STACK_ENTRIES 4096 | ||
| 33 | #define SI_MAX_FRC_EOV_CNT 16384 | ||
| 34 | #define SI_MAX_BACKENDS 8 | ||
| 35 | #define SI_MAX_BACKENDS_MASK 0xFF | ||
| 36 | #define SI_MAX_BACKENDS_PER_SE_MASK 0x0F | ||
| 37 | #define SI_MAX_SIMDS 12 | ||
| 38 | #define SI_MAX_SIMDS_MASK 0x0FFF | ||
| 39 | #define SI_MAX_SIMDS_PER_SE_MASK 0x00FF | ||
| 40 | #define SI_MAX_PIPES 8 | ||
| 41 | #define SI_MAX_PIPES_MASK 0xFF | ||
| 42 | #define SI_MAX_PIPES_PER_SIMD_MASK 0x3F | ||
| 43 | #define SI_MAX_LDS_NUM 0xFFFF | ||
| 44 | #define SI_MAX_TCC 16 | ||
| 45 | #define SI_MAX_TCC_MASK 0xFFFF | ||
| 46 | #define SI_MAX_CTLACKS_ASSERTION_WAIT 100 | 27 | #define SI_MAX_CTLACKS_ASSERTION_WAIT 100 |
| 47 | 28 | ||
| 48 | /* SMC IND accessor regs */ | ||
| 49 | #define SMC_IND_INDEX_0 0x80 | ||
| 50 | #define SMC_IND_DATA_0 0x81 | ||
| 51 | |||
| 52 | #define SMC_IND_ACCESS_CNTL 0x8A | ||
| 53 | # define AUTO_INCREMENT_IND_0 (1 << 0) | ||
| 54 | #define SMC_MESSAGE_0 0x8B | ||
| 55 | #define SMC_RESP_0 0x8C | ||
| 56 | |||
| 57 | /* CG IND registers are accessed via SMC indirect space + SMC_CG_IND_START */ | 29 | /* CG IND registers are accessed via SMC indirect space + SMC_CG_IND_START */ |
| 58 | #define SMC_CG_IND_START 0xc0030000 | 30 | #define SMC_CG_IND_START 0xc0030000 |
| 59 | #define SMC_CG_IND_END 0xc0040000 | 31 | #define SMC_CG_IND_END 0xc0040000 |
| 60 | 32 | ||
| 61 | #define CG_CGTT_LOCAL_0 0x400 | ||
| 62 | #define CG_CGTT_LOCAL_1 0x401 | ||
| 63 | |||
| 64 | /* SMC IND registers */ | 33 | /* SMC IND registers */ |
| 65 | #define SMC_SYSCON_RESET_CNTL 0x80000000 | 34 | #define SMC_SYSCON_RESET_CNTL 0x80000000 |
| 66 | # define RST_REG (1 << 0) | 35 | # define RST_REG (1 << 0) |
| @@ -68,9 +37,6 @@ · 1 unmodified line | |||
| 68 | # define CK_DISABLE (1 << 0) | 37 | # define CK_DISABLE (1 << 0) |
| 69 | # define CKEN (1 << 24) | 38 | # define CKEN (1 << 24) |
| 70 | 39 | ||
| 71 | #define VGA_HDP_CONTROL 0xCA | ||
| 72 | #define VGA_MEMORY_DISABLE (1 << 4) | ||
| 73 | |||
| 74 | #define DCCG_DISP_SLOW_SELECT_REG 0x13F | 40 | #define DCCG_DISP_SLOW_SELECT_REG 0x13F |
| 75 | #define DCCG_DISP1_SLOW_SELECT(x) ((x) << 0) | 41 | #define DCCG_DISP1_SLOW_SELECT(x) ((x) << 0) |
| 76 | #define DCCG_DISP1_SLOW_SELECT_MASK (7 << 0) | 42 | #define DCCG_DISP1_SLOW_SELECT_MASK (7 << 0) |
| @@ -79,47 +45,6 @@ · 2 unmodified lines | |||
| 79 | #define DCCG_DISP2_SLOW_SELECT_MASK (7 << 4) | 45 | #define DCCG_DISP2_SLOW_SELECT_MASK (7 << 4) |
| 80 | #define DCCG_DISP2_SLOW_SELECT_SHIFT 4 | 46 | #define DCCG_DISP2_SLOW_SELECT_SHIFT 4 |
| 81 | 47 | ||
| 82 | #define CG_SPLL_FUNC_CNTL 0x180 | ||
| 83 | #define SPLL_RESET (1 << 0) | ||
| 84 | #define SPLL_SLEEP (1 << 1) | ||
| 85 | #define SPLL_BYPASS_EN (1 << 3) | ||
| 86 | #define SPLL_REF_DIV(x) ((x) << 4) | ||
| 87 | #define SPLL_REF_DIV_MASK (0x3f << 4) | ||
| 88 | #define SPLL_PDIV_A(x) ((x) << 20) | ||
| 89 | #define SPLL_PDIV_A_MASK (0x7f << 20) | ||
| 90 | #define SPLL_PDIV_A_SHIFT 20 | ||
| 91 | #define CG_SPLL_FUNC_CNTL_2 0x181 | ||
| 92 | #define SCLK_MUX_SEL(x) ((x) << 0) | ||
| 93 | #define SCLK_MUX_SEL_MASK (0x1ff << 0) | ||
| 94 | #define SPLL_CTLREQ_CHG (1 << 23) | ||
| 95 | #define SCLK_MUX_UPDATE (1 << 26) | ||
| 96 | #define CG_SPLL_FUNC_CNTL_3 0x182 | ||
| 97 | #define SPLL_FB_DIV(x) ((x) << 0) | ||
| 98 | #define SPLL_FB_DIV_MASK (0x3ffffff << 0) | ||
| 99 | #define SPLL_FB_DIV_SHIFT 0 | ||
| 100 | #define SPLL_DITHEN (1 << 28) | ||
| 101 | #define CG_SPLL_FUNC_CNTL_4 0x183 | ||
| 102 | |||
| 103 | #define SPLL_STATUS 0x185 | ||
| 104 | #define SPLL_CHG_STATUS (1 << 1) | ||
| 105 | #define SPLL_CNTL_MODE 0x186 | ||
| 106 | #define SPLL_SW_DIR_CONTROL (1 << 0) | ||
| 107 | # define SPLL_REFCLK_SEL(x) ((x) << 26) | ||
| 108 | # define SPLL_REFCLK_SEL_MASK (3 << 26) | ||
| 109 | |||
| 110 | #define CG_SPLL_SPREAD_SPECTRUM 0x188 | ||
| 111 | #define SSEN (1 << 0) | ||
| 112 | #define CLK_S(x) ((x) << 4) | ||
| 113 | #define CLK_S_MASK (0xfff << 4) | ||
| 114 | #define CLK_S_SHIFT 4 | ||
| 115 | #define CG_SPLL_SPREAD_SPECTRUM_2 0x189 | ||
| 116 | #define CLK_V(x) ((x) << 0) | ||
| 117 | #define CLK_V_MASK (0x3ffffff << 0) | ||
| 118 | #define CLK_V_SHIFT 0 | ||
| 119 | |||
| 120 | #define CG_SPLL_AUTOSCALE_CNTL 0x18b | ||
| 121 | # define AUTOSCALE_ON_SS_CLEAR (1 << 9) | ||
| 122 | |||
| 123 | /* discrete uvd clocks */ | 48 | /* discrete uvd clocks */ |
| 124 | #define CG_UPLL_FUNC_CNTL 0x18d | 49 | #define CG_UPLL_FUNC_CNTL 0x18d |
| 125 | # define UPLL_RESET_MASK 0x00000001 | 50 | # define UPLL_RESET_MASK 0x00000001 |
| @@ -149,317 +74,13 @@ · 23 unmodified lines | |||
| 149 | #define CG_UPLL_SPREAD_SPECTRUM 0x194 | 74 | #define CG_UPLL_SPREAD_SPECTRUM 0x194 |
| 150 | # define SSEN_MASK 0x00000001 | 75 | # define SSEN_MASK 0x00000001 |
| 151 | 76 | ||
| 152 | #define MPLL_BYPASSCLK_SEL 0x197 | ||
| 153 | # define MPLL_CLKOUT_SEL(x) ((x) << 8) | ||
| 154 | # define MPLL_CLKOUT_SEL_MASK 0xFF00 | ||
| 155 | |||
| 156 | #define CG_CLKPIN_CNTL 0x198 | ||
| 157 | # define XTALIN_DIVIDE (1 << 1) | ||
| 158 | # define BCLK_AS_XCLK (1 << 2) | ||
| 159 | #define CG_CLKPIN_CNTL_2 0x199 | ||
| 160 | # define FORCE_BIF_REFCLK_EN (1 << 3) | ||
| 161 | # define MUX_TCLK_TO_XCLK (1 << 8) | ||
| 162 | |||
| 163 | #define THM_CLK_CNTL 0x19b | ||
| 164 | # define CMON_CLK_SEL(x) ((x) << 0) | ||
| 165 | # define CMON_CLK_SEL_MASK 0xFF | ||
| 166 | # define TMON_CLK_SEL(x) ((x) << 8) | ||
| 167 | # define TMON_CLK_SEL_MASK 0xFF00 | ||
| 168 | #define MISC_CLK_CNTL 0x19c | ||
| 169 | # define DEEP_SLEEP_CLK_SEL(x) ((x) << 0) | ||
| 170 | # define DEEP_SLEEP_CLK_SEL_MASK 0xFF | ||
| 171 | # define ZCLK_SEL(x) ((x) << 8) | ||
| 172 | # define ZCLK_SEL_MASK 0xFF00 | ||
| 173 | |||
| 174 | #define CG_THERMAL_CTRL 0x1c0 | ||
| 175 | #define DPM_EVENT_SRC(x) ((x) << 0) | ||
| 176 | #define DPM_EVENT_SRC_MASK (7 << 0) | ||
| 177 | #define DIG_THERM_DPM(x) ((x) << 14) | ||
| 178 | #define DIG_THERM_DPM_MASK 0x003FC000 | ||
| 179 | #define DIG_THERM_DPM_SHIFT 14 | ||
| 180 | #define CG_THERMAL_STATUS 0x1c1 | ||
| 181 | #define FDO_PWM_DUTY(x) ((x) << 9) | ||
| 182 | #define FDO_PWM_DUTY_MASK (0xff << 9) | ||
| 183 | #define FDO_PWM_DUTY_SHIFT 9 | ||
| 184 | #define CG_THERMAL_INT 0x1c2 | ||
| 185 | #define DIG_THERM_INTH(x) ((x) << 8) | ||
| 186 | #define DIG_THERM_INTH_MASK 0x0000FF00 | ||
| 187 | #define DIG_THERM_INTH_SHIFT 8 | ||
| 188 | #define DIG_THERM_INTL(x) ((x) << 16) | ||
| 189 | #define DIG_THERM_INTL_MASK 0x00FF0000 | ||
| 190 | #define DIG_THERM_INTL_SHIFT 16 | ||
| 191 | #define THERM_INT_MASK_HIGH (1 << 24) | ||
| 192 | #define THERM_INT_MASK_LOW (1 << 25) | ||
| 193 | |||
| 194 | #define CG_MULT_THERMAL_CTRL 0x1c4 | ||
| 195 | #define TEMP_SEL(x) ((x) << 20) | ||
| 196 | #define TEMP_SEL_MASK (0xff << 20) | ||
| 197 | #define TEMP_SEL_SHIFT 20 | ||
| 198 | #define CG_MULT_THERMAL_STATUS 0x1c5 | ||
| 199 | #define ASIC_MAX_TEMP(x) ((x) << 0) | ||
| 200 | #define ASIC_MAX_TEMP_MASK 0x000001ff | ||
| 201 | #define ASIC_MAX_TEMP_SHIFT 0 | ||
| 202 | #define CTF_TEMP(x) ((x) << 9) | ||
| 203 | #define CTF_TEMP_MASK 0x0003fe00 | ||
| 204 | #define CTF_TEMP_SHIFT 9 | ||
| 205 | |||
| 206 | #define CG_FDO_CTRL0 0x1d5 | ||
| 207 | #define FDO_STATIC_DUTY(x) ((x) << 0) | ||
| 208 | #define FDO_STATIC_DUTY_MASK 0x000000FF | ||
| 209 | #define FDO_STATIC_DUTY_SHIFT 0 | ||
| 210 | #define CG_FDO_CTRL1 0x1d6 | ||
| 211 | #define FMAX_DUTY100(x) ((x) << 0) | ||
| 212 | #define FMAX_DUTY100_MASK 0x000000FF | ||
| 213 | #define FMAX_DUTY100_SHIFT 0 | ||
| 214 | #define CG_FDO_CTRL2 0x1d7 | ||
| 215 | #define TMIN(x) ((x) << 0) | ||
| 216 | #define TMIN_MASK 0x000000FF | ||
| 217 | #define TMIN_SHIFT 0 | ||
| 218 | #define FDO_PWM_MODE(x) ((x) << 11) | ||
| 219 | #define FDO_PWM_MODE_MASK (7 << 11) | ||
| 220 | #define FDO_PWM_MODE_SHIFT 11 | ||
| 221 | #define TACH_PWM_RESP_RATE(x) ((x) << 25) | ||
| 222 | #define TACH_PWM_RESP_RATE_MASK (0x7f << 25) | ||
| 223 | #define TACH_PWM_RESP_RATE_SHIFT 25 | ||
| 224 | |||
| 225 | #define CG_TACH_CTRL 0x1dc | ||
| 226 | # define EDGE_PER_REV(x) ((x) << 0) | ||
| 227 | # define EDGE_PER_REV_MASK (0x7 << 0) | ||
| 228 | # define EDGE_PER_REV_SHIFT 0 | ||
| 229 | # define TARGET_PERIOD(x) ((x) << 3) | ||
| 230 | # define TARGET_PERIOD_MASK 0xfffffff8 | ||
| 231 | # define TARGET_PERIOD_SHIFT 3 | ||
| 232 | #define CG_TACH_STATUS 0x1dd | ||
| 233 | # define TACH_PERIOD(x) ((x) << 0) | ||
| 234 | # define TACH_PERIOD_MASK 0xffffffff | ||
| 235 | # define TACH_PERIOD_SHIFT 0 | ||
| 236 | |||
| 237 | #define GENERAL_PWRMGT 0x1e0 | ||
| 238 | # define GLOBAL_PWRMGT_EN (1 << 0) | ||
| 239 | # define STATIC_PM_EN (1 << 1) | ||
| 240 | # define THERMAL_PROTECTION_DIS (1 << 2) | ||
| 241 | # define THERMAL_PROTECTION_TYPE (1 << 3) | ||
| 242 | # define SW_SMIO_INDEX(x) ((x) << 6) | ||
| 243 | # define SW_SMIO_INDEX_MASK (1 << 6) | ||
| 244 | # define SW_SMIO_INDEX_SHIFT 6 | ||
| 245 | # define VOLT_PWRMGT_EN (1 << 10) | ||
| 246 | # define DYN_SPREAD_SPECTRUM_EN (1 << 23) | ||
| 247 | #define CG_TPC 0x1e1 | ||
| 248 | #define SCLK_PWRMGT_CNTL 0x1e2 | ||
| 249 | # define SCLK_PWRMGT_OFF (1 << 0) | ||
| 250 | # define SCLK_LOW_D1 (1 << 1) | ||
| 251 | # define FIR_RESET (1 << 4) | ||
| 252 | # define FIR_FORCE_TREND_SEL (1 << 5) | ||
| 253 | # define FIR_TREND_MODE (1 << 6) | ||
| 254 | # define DYN_GFX_CLK_OFF_EN (1 << 7) | ||
| 255 | # define GFX_CLK_FORCE_ON (1 << 8) | ||
| 256 | # define GFX_CLK_REQUEST_OFF (1 << 9) | ||
| 257 | # define GFX_CLK_FORCE_OFF (1 << 10) | ||
| 258 | # define GFX_CLK_OFF_ACPI_D1 (1 << 11) | ||
| 259 | # define GFX_CLK_OFF_ACPI_D2 (1 << 12) | ||
| 260 | # define GFX_CLK_OFF_ACPI_D3 (1 << 13) | ||
| 261 | # define DYN_LIGHT_SLEEP_EN (1 << 14) | ||
| 262 | |||
| 263 | #define TARGET_AND_CURRENT_PROFILE_INDEX 0x1e6 | ||
| 264 | # define CURRENT_STATE_INDEX_MASK (0xf << 4) | ||
| 265 | # define CURRENT_STATE_INDEX_SHIFT 4 | ||
| 266 | |||
| 267 | #define CG_FTV 0x1ef | ||
| 268 | |||
| 269 | #define CG_FFCT_0 0x1f0 | ||
| 270 | # define UTC_0(x) ((x) << 0) | ||
| 271 | # define UTC_0_MASK (0x3ff << 0) | ||
| 272 | # define DTC_0(x) ((x) << 10) | ||
| 273 | # define DTC_0_MASK (0x3ff << 10) | ||
| 274 | |||
| 275 | #define CG_BSP 0x1ff | ||
| 276 | # define BSP(x) ((x) << 0) | ||
| 277 | # define BSP_MASK (0xffff << 0) | ||
| 278 | # define BSU(x) ((x) << 16) | ||
| 279 | # define BSU_MASK (0xf << 16) | ||
| 280 | #define CG_AT 0x200 | ||
| 281 | # define CG_R(x) ((x) << 0) | ||
| 282 | # define CG_R_MASK (0xffff << 0) | ||
| 283 | # define CG_L(x) ((x) << 16) | ||
| 284 | # define CG_L_MASK (0xffff << 16) | ||
| 285 | |||
| 286 | #define CG_GIT 0x201 | ||
| 287 | # define CG_GICST(x) ((x) << 0) | ||
| 288 | # define CG_GICST_MASK (0xffff << 0) | ||
| 289 | # define CG_GIPOT(x) ((x) << 16) | ||
| 290 | # define CG_GIPOT_MASK (0xffff << 16) | ||
| 291 | |||
| 292 | #define CG_SSP 0x203 | ||
| 293 | # define SST(x) ((x) << 0) | ||
| 294 | # define SST_MASK (0xffff << 0) | ||
| 295 | # define SSTU(x) ((x) << 16) | ||
| 296 | # define SSTU_MASK (0xf << 16) | ||
| 297 | |||
| 298 | #define CG_DISPLAY_GAP_CNTL 0x20a | ||
| 299 | # define DISP1_GAP(x) ((x) << 0) | ||
| 300 | # define DISP1_GAP_MASK (3 << 0) | ||
| 301 | # define DISP2_GAP(x) ((x) << 2) | ||
| 302 | # define DISP2_GAP_MASK (3 << 2) | ||
| 303 | # define VBI_TIMER_COUNT(x) ((x) << 4) | ||
| 304 | # define VBI_TIMER_COUNT_MASK (0x3fff << 4) | ||
| 305 | # define VBI_TIMER_UNIT(x) ((x) << 20) | ||
| 306 | # define VBI_TIMER_UNIT_MASK (7 << 20) | ||
| 307 | # define DISP1_GAP_MCHG(x) ((x) << 24) | ||
| 308 | # define DISP1_GAP_MCHG_MASK (3 << 24) | ||
| 309 | # define DISP2_GAP_MCHG(x) ((x) << 26) | ||
| 310 | # define DISP2_GAP_MCHG_MASK (3 << 26) | ||
| 311 | |||
| 312 | #define CG_ULV_CONTROL 0x21e | ||
| 313 | #define CG_ULV_PARAMETER 0x21f | ||
| 314 | |||
| 315 | #define SMC_SCRATCH0 0x221 | ||
| 316 | |||
| 317 | #define CG_CAC_CTRL 0x22e | ||
| 318 | # define CAC_WINDOW(x) ((x) << 0) | ||
| 319 | # define CAC_WINDOW_MASK 0x00ffffff | ||
| 320 | |||
| 321 | #define DMIF_ADDR_CONFIG 0x2F5 | ||
| 322 | |||
| 323 | #define DMIF_ADDR_CALC 0x300 | ||
| 324 | |||
| 325 | #define PIPE0_DMIF_BUFFER_CONTROL 0x0328 | ||
| 326 | # define DMIF_BUFFERS_ALLOCATED(x) ((x) << 0) | ||
| 327 | # define DMIF_BUFFERS_ALLOCATED_COMPLETED (1 << 4) | ||
| 328 | |||
| 329 | #define SRBM_STATUS 0x394 | ||
| 330 | #define GRBM_RQ_PENDING (1 << 5) | ||
| 331 | #define VMC_BUSY (1 << 8) | ||
| 332 | #define MCB_BUSY (1 << 9) | ||
| 333 | #define MCB_NON_DISPLAY_BUSY (1 << 10) | ||
| 334 | #define MCC_BUSY (1 << 11) | ||
| 335 | #define MCD_BUSY (1 << 12) | ||
| 336 | #define SEM_BUSY (1 << 14) | ||
| 337 | #define IH_BUSY (1 << 17) | ||
| 338 | |||
| 339 | #define SRBM_SOFT_RESET 0x398 | ||
| 340 | #define SOFT_RESET_BIF (1 << 1) | ||
| 341 | #define SOFT_RESET_DC (1 << 5) | ||
| 342 | #define SOFT_RESET_DMA1 (1 << 6) | ||
| 343 | #define SOFT_RESET_GRBM (1 << 8) | ||
| 344 | #define SOFT_RESET_HDP (1 << 9) | ||
| 345 | #define SOFT_RESET_IH (1 << 10) | ||
| 346 | #define SOFT_RESET_MC (1 << 11) | ||
| 347 | #define SOFT_RESET_ROM (1 << 14) | ||
| 348 | #define SOFT_RESET_SEM (1 << 15) | ||
| 349 | #define SOFT_RESET_VMC (1 << 17) | ||
| 350 | #define SOFT_RESET_DMA (1 << 20) | ||
| 351 | #define SOFT_RESET_TST (1 << 21) | ||
| 352 | #define SOFT_RESET_REGBB (1 << 22) | ||
| 353 | #define SOFT_RESET_ORB (1 << 23) | ||
| 354 | |||
| 355 | #define CC_SYS_RB_BACKEND_DISABLE 0x3A0 | ||
| 356 | #define GC_USER_SYS_RB_BACKEND_DISABLE 0x3A1 | ||
| 357 | |||
| 358 | #define SRBM_READ_ERROR 0x3A6 | ||
| 359 | #define SRBM_INT_CNTL 0x3A8 | ||
| 360 | #define SRBM_INT_ACK 0x3AA | ||
| 361 | |||
| 362 | #define SRBM_STATUS2 0x3B1 | ||
| 363 | #define DMA_BUSY (1 << 5) | ||
| 364 | #define DMA1_BUSY (1 << 6) | ||
| 365 | |||
| 366 | #define VM_L2_CNTL 0x500 | ||
| 367 | #define ENABLE_L2_CACHE (1 << 0) | ||
| 368 | #define ENABLE_L2_FRAGMENT_PROCESSING (1 << 1) | ||
| 369 | #define L2_CACHE_PTE_ENDIAN_SWAP_MODE(x) ((x) << 2) | ||
| 370 | #define L2_CACHE_PDE_ENDIAN_SWAP_MODE(x) ((x) << 4) | ||
| 371 | #define ENABLE_L2_PTE_CACHE_LRU_UPDATE_BY_WRITE (1 << 9) | ||
| 372 | #define ENABLE_L2_PDE0_CACHE_LRU_UPDATE_BY_WRITE (1 << 10) | ||
| 373 | #define EFFECTIVE_L2_QUEUE_SIZE(x) (((x) & 7) << 15) | ||
| 374 | #define CONTEXT1_IDENTITY_ACCESS_MODE(x) (((x) & 3) << 19) | ||
| 375 | #define VM_L2_CNTL2 0x501 | ||
| 376 | #define INVALIDATE_ALL_L1_TLBS (1 << 0) | ||
| 377 | #define INVALIDATE_L2_CACHE (1 << 1) | ||
| 378 | #define INVALIDATE_CACHE_MODE(x) ((x) << 26) | ||
| 379 | #define INVALIDATE_PTE_AND_PDE_CACHES 0 | ||
| 380 | #define INVALIDATE_ONLY_PTE_CACHES 1 | ||
| 381 | #define INVALIDATE_ONLY_PDE_CACHES 2 | ||
| 382 | #define VM_L2_CNTL3 0x502 | ||
| 383 | #define BANK_SELECT(x) ((x) << 0) | ||
| 384 | #define L2_CACHE_UPDATE_MODE(x) ((x) << 6) | ||
| 385 | #define L2_CACHE_BIGK_FRAGMENT_SIZE(x) ((x) << 15) | ||
| 386 | #define L2_CACHE_BIGK_ASSOCIATIVITY (1 << 20) | ||
| 387 | #define VM_L2_STATUS 0x503 | ||
| 388 | #define L2_BUSY (1 << 0) | ||
| 389 | #define VM_CONTEXT0_CNTL 0x504 | ||
| 390 | #define ENABLE_CONTEXT (1 << 0) | ||
| 391 | #define PAGE_TABLE_DEPTH(x) (((x) & 3) << 1) | ||
| 392 | #define RANGE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 3) | ||
| 393 | #define RANGE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 4) | ||
| 394 | #define DUMMY_PAGE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 6) | ||
| 395 | #define DUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 7) | ||
| 396 | #define PDE0_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 9) | ||
| 397 | #define PDE0_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 10) | ||
| 398 | #define VALID_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 12) | ||
| 399 | #define VALID_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 13) | ||
| 400 | #define READ_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 15) | ||
| 401 | #define READ_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 16) | ||
| 402 | #define WRITE_PROTECTION_FAULT_ENABLE_INTERRUPT (1 << 18) | ||
| 403 | #define WRITE_PROTECTION_FAULT_ENABLE_DEFAULT (1 << 19) | ||
| 404 | #define PAGE_TABLE_BLOCK_SIZE(x) (((x) & 0xF) << 24) | ||
| 405 | #define VM_CONTEXT1_CNTL 0x505 | ||
| 406 | #define VM_CONTEXT0_CNTL2 0x50C | ||
| 407 | #define VM_CONTEXT1_CNTL2 0x50D | ||
| 408 | #define VM_CONTEXT8_PAGE_TABLE_BASE_ADDR 0x50E | ||
| 409 | #define VM_CONTEXT9_PAGE_TABLE_BASE_ADDR 0x50F | ||
| 410 | #define VM_CONTEXT10_PAGE_TABLE_BASE_ADDR 0x510 | ||
| 411 | #define VM_CONTEXT11_PAGE_TABLE_BASE_ADDR 0x511 | ||
| 412 | #define VM_CONTEXT12_PAGE_TABLE_BASE_ADDR 0x512 | ||
| 413 | #define VM_CONTEXT13_PAGE_TABLE_BASE_ADDR 0x513 | ||
| 414 | #define VM_CONTEXT14_PAGE_TABLE_BASE_ADDR 0x514 | ||
| 415 | #define VM_CONTEXT15_PAGE_TABLE_BASE_ADDR 0x515 | ||
| 416 | |||
| 417 | #define VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x53f | ||
| 418 | #define VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x537 | ||
| 419 | #define PROTECTIONS_MASK (0xf << 0) | ||
| 420 | #define PROTECTIONS_SHIFT 0 | ||
| 421 | /* bit 0: range | ||
| 422 | * bit 1: pde0 | ||
| 423 | * bit 2: valid | ||
| 424 | * bit 3: read | ||
| 425 | * bit 4: write | ||
| 426 | */ | ||
| 427 | #define MEMORY_CLIENT_ID_MASK (0xff << 12) | ||
| 428 | #define MEMORY_CLIENT_ID_SHIFT 12 | ||
| 429 | #define MEMORY_CLIENT_RW_MASK (1 << 24) | ||
| 430 | #define MEMORY_CLIENT_RW_SHIFT 24 | ||
| 431 | #define FAULT_VMID_MASK (0xf << 25) | ||
| 432 | #define FAULT_VMID_SHIFT 25 | ||
| 433 | |||
| 434 | #define VM_INVALIDATE_REQUEST 0x51E | 77 | #define VM_INVALIDATE_REQUEST 0x51E |
| 435 | #define VM_INVALIDATE_RESPONSE 0x51F | 78 | #define VM_INVALIDATE_RESPONSE 0x51F |
| 436 | 79 | ||
| 437 | #define VM_CONTEXT0_PROTECTION_FAULT_DEFAULT_ADDR 0x546 | ||
| 438 | #define VM_CONTEXT1_PROTECTION_FAULT_DEFAULT_ADDR 0x547 | ||
| 439 | |||
| 440 | #define VM_CONTEXT0_PAGE_TABLE_BASE_ADDR 0x54F | ||
| 441 | #define VM_CONTEXT1_PAGE_TABLE_BASE_ADDR 0x550 | ||
| 442 | #define VM_CONTEXT2_PAGE_TABLE_BASE_ADDR 0x551 | ||
| 443 | #define VM_CONTEXT3_PAGE_TABLE_BASE_ADDR 0x552 | ||
| 444 | #define VM_CONTEXT4_PAGE_TABLE_BASE_ADDR 0x553 | ||
| 445 | #define VM_CONTEXT5_PAGE_TABLE_BASE_ADDR 0x554 | ||
| 446 | #define VM_CONTEXT6_PAGE_TABLE_BASE_ADDR 0x555 | ||
| 447 | #define VM_CONTEXT7_PAGE_TABLE_BASE_ADDR 0x556 | ||
| 448 | #define VM_CONTEXT0_PAGE_TABLE_START_ADDR 0x557 | ||
| 449 | #define VM_CONTEXT1_PAGE_TABLE_START_ADDR 0x558 | ||
| 450 | |||
| 451 | #define VM_CONTEXT0_PAGE_TABLE_END_ADDR 0x55F | ||
| 452 | #define VM_CONTEXT1_PAGE_TABLE_END_ADDR 0x560 | ||
| 453 | |||
| 454 | #define VM_L2_CG 0x570 | 80 | #define VM_L2_CG 0x570 |
| 455 | #define MC_CG_ENABLE (1 << 18) | 81 | #define MC_CG_ENABLE (1 << 18) |
| 456 | #define MC_LS_ENABLE (1 << 19) | 82 | #define MC_LS_ENABLE (1 << 19) |
| 457 | 83 | ||
| 458 | #define MC_SHARED_CHMAP 0x801 | ||
| 459 | #define NOOFCHAN_SHIFT 12 | ||
| 460 | #define NOOFCHAN_MASK 0x0000f000 | ||
| 461 | #define MC_SHARED_CHREMAP 0x802 | ||
| 462 | |||
| 463 | #define MC_VM_FB_LOCATION 0x809 | 84 | #define MC_VM_FB_LOCATION 0x809 |
| 464 | #define MC_VM_AGP_TOP 0x80A | 85 | #define MC_VM_AGP_TOP 0x80A |
| 465 | #define MC_VM_AGP_BOT 0x80B | 86 | #define MC_VM_AGP_BOT 0x80B |
| @@ -491,21 +112,6 @@ · 25 unmodified lines | |||
| 491 | #define MC_CITF_MISC_WR_CG 0x993 | 112 | #define MC_CITF_MISC_WR_CG 0x993 |
| 492 | #define MC_CITF_MISC_VM_CG 0x994 | 113 | #define MC_CITF_MISC_VM_CG 0x994 |
| 493 | 114 | ||
| 494 | #define MC_ARB_RAMCFG 0x9D8 | ||
| 495 | #define NOOFBANK_SHIFT 0 | ||
| 496 | #define NOOFBANK_MASK 0x00000003 | ||
| 497 | #define NOOFRANK_SHIFT 2 | ||
| 498 | #define NOOFRANK_MASK 0x00000004 | ||
| 499 | #define NOOFROWS_SHIFT 3 | ||
| 500 | #define NOOFROWS_MASK 0x00000038 | ||
| 501 | #define NOOFCOLS_SHIFT 6 | ||
| 502 | #define NOOFCOLS_MASK 0x000000C0 | ||
| 503 | #define CHANSIZE_SHIFT 8 | ||
| 504 | #define CHANSIZE_MASK 0x00000100 | ||
| 505 | #define CHANSIZE_OVERRIDE (1 << 11) | ||
| 506 | #define NOOFGROUPS_SHIFT 12 | ||
| 507 | #define NOOFGROUPS_MASK 0x00001000 | ||
| 508 | |||
| 509 | #define MC_ARB_DRAM_TIMING 0x9DD | 115 | #define MC_ARB_DRAM_TIMING 0x9DD |
| 510 | #define MC_ARB_DRAM_TIMING2 0x9DE | 116 | #define MC_ARB_DRAM_TIMING2 0x9DE |
| 511 | 117 | ||
| @@ -631,20 +237,6 @@ · 119 unmodified lines | |||
| 631 | #define CLKS(x) ((x) << 0) | 237 | #define CLKS(x) ((x) << 0) |
| 632 | #define CLKS_MASK (0xfff << 0) | 238 | #define CLKS_MASK (0xfff << 0) |
| 633 | 239 | ||
| 634 | #define HDP_HOST_PATH_CNTL 0xB00 | ||
| 635 | #define CLOCK_GATING_DIS (1 << 23) | ||
| 636 | #define HDP_NONSURFACE_BASE 0xB01 | ||
| 637 | #define HDP_NONSURFACE_INFO 0xB02 | ||
| 638 | #define HDP_NONSURFACE_SIZE 0xB03 | ||
| 639 | |||
| 640 | #define HDP_DEBUG0 0xBCC | ||
| 641 | |||
| 642 | #define HDP_ADDR_CONFIG 0xBD2 | ||
| 643 | #define HDP_MISC_CNTL 0xBD3 | ||
| 644 | #define HDP_FLUSH_INVALIDATE_CACHE (1 << 0) | ||
| 645 | #define HDP_MEM_POWER_LS 0xBD4 | ||
| 646 | #define HDP_LS_ENABLE (1 << 0) | ||
| 647 | |||
| 648 | #define ATC_MISC_CG 0xCD4 | 240 | #define ATC_MISC_CG 0xCD4 |
| 649 | 241 | ||
| 650 | #define IH_RB_CNTL 0xF80 | 242 | #define IH_RB_CNTL 0xF80 |
| @@ -674,8 +266,6 @@ · 23 unmodified lines | |||
| 674 | # define MC_WR_CLEAN_CNT(x) ((x) << 20) | 266 | # define MC_WR_CLEAN_CNT(x) ((x) << 20) |
| 675 | # define MC_VMID(x) ((x) << 25) | 267 | # define MC_VMID(x) ((x) << 25) |
| 676 | 268 | ||
| 677 | #define CONFIG_MEMSIZE 0x150A | ||
| 678 | |||
| 679 | #define INTERRUPT_CNTL 0x151A | 269 | #define INTERRUPT_CNTL 0x151A |
| 680 | # define IH_DUMMY_RD_OVERRIDE (1 << 0) | 270 | # define IH_DUMMY_RD_OVERRIDE (1 << 0) |
| 681 | # define IH_DUMMY_RD_EN (1 << 1) | 271 | # define IH_DUMMY_RD_EN (1 << 1) |
| @@ -683,486 +273,22 @@ · 1 unmodified line | |||
| 683 | # define GEN_IH_INT_EN (1 << 8) | 273 | # define GEN_IH_INT_EN (1 << 8) |
| 684 | #define INTERRUPT_CNTL2 0x151B | 274 | #define INTERRUPT_CNTL2 0x151B |
| 685 | 275 | ||
| 686 | #define HDP_MEM_COHERENCY_FLUSH_CNTL 0x1520 | ||
| 687 | |||
| 688 | #define BIF_FB_EN 0x1524 | ||
| 689 | #define FB_READ_EN (1 << 0) | ||
| 690 | #define FB_WRITE_EN (1 << 1) | ||
| 691 | |||
| 692 | #define HDP_REG_COHERENCY_FLUSH_CNTL 0x1528 | ||
| 693 | |||
| 694 | /* DCE6 ELD audio interface */ | ||
| 695 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR0 0x28 /* LPCM */ | ||
| 696 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR1 0x29 /* AC3 */ | ||
| 697 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR2 0x2A /* MPEG1 */ | ||
| 698 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR3 0x2B /* MP3 */ | ||
| 699 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR4 0x2C /* MPEG2 */ | ||
| 700 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR5 0x2D /* AAC */ | ||
| 701 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR6 0x2E /* DTS */ | ||
| 702 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR7 0x2F /* ATRAC */ | ||
| 703 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR8 0x30 /* one bit audio - leave at 0 (default) */ | ||
| 704 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR9 0x31 /* Dolby Digital */ | ||
| 705 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR10 0x32 /* DTS-HD */ | ||
| 706 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR11 0x33 /* MAT-MLP */ | ||
| 707 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR12 0x34 /* DTS */ | ||
| 708 | #define AZ_F0_CODEC_PIN_CONTROL_AUDIO_DESCRIPTOR13 0x35 /* WMA Pro */ | ||
| 709 | # define MAX_CHANNELS(x) (((x) & 0x7) << 0) | ||
| 710 | /* max channels minus one. 7 = 8 channels */ | ||
| 711 | # define SUPPORTED_FREQUENCIES(x) (((x) & 0xff) << 8) | ||
| 712 | # define DESCRIPTOR_BYTE_2(x) (((x) & 0xff) << 16) | ||
| 713 | # define SUPPORTED_FREQUENCIES_STEREO(x) (((x) & 0xff) << 24) /* LPCM only */ | ||
| 714 | /* SUPPORTED_FREQUENCIES, SUPPORTED_FREQUENCIES_STEREO | ||
| 715 | * bit0 = 32 kHz | ||
| 716 | * bit1 = 44.1 kHz | ||
| 717 | * bit2 = 48 kHz | ||
| 718 | * bit3 = 88.2 kHz | ||
| 719 | * bit4 = 96 kHz | ||
| 720 | * bit5 = 176.4 kHz | ||
| 721 | * bit6 = 192 kHz | ||
| 722 | */ | ||
| 723 | |||
| 724 | #define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_LIPSYNC 0x37 | ||
| 725 | # define VIDEO_LIPSYNC(x) (((x) & 0xff) << 0) | ||
| 726 | # define AUDIO_LIPSYNC(x) (((x) & 0xff) << 8) | ||
| 727 | /* VIDEO_LIPSYNC, AUDIO_LIPSYNC | ||
| 728 | * 0 = invalid | ||
| 729 | * x = legal delay value | ||
| 730 | * 255 = sync not supported | ||
| 731 | */ | ||
| 732 | #define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_HBR 0x38 | ||
| 733 | # define HBR_CAPABLE (1 << 0) /* enabled by default */ | ||
| 734 | |||
| 735 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO0 0x3a | ||
| 736 | # define MANUFACTURER_ID(x) (((x) & 0xffff) << 0) | ||
| 737 | # define PRODUCT_ID(x) (((x) & 0xffff) << 16) | ||
| 738 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO1 0x3b | ||
| 739 | # define SINK_DESCRIPTION_LEN(x) (((x) & 0xff) << 0) | ||
| 740 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO2 0x3c | ||
| 741 | # define PORT_ID0(x) (((x) & 0xffffffff) << 0) | ||
| 742 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO3 0x3d | ||
| 743 | # define PORT_ID1(x) (((x) & 0xffffffff) << 0) | ||
| 744 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO4 0x3e | ||
| 745 | # define DESCRIPTION0(x) (((x) & 0xff) << 0) | ||
| 746 | # define DESCRIPTION1(x) (((x) & 0xff) << 8) | ||
| 747 | # define DESCRIPTION2(x) (((x) & 0xff) << 16) | ||
| 748 | # define DESCRIPTION3(x) (((x) & 0xff) << 24) | ||
| 749 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO5 0x3f | ||
| 750 | # define DESCRIPTION4(x) (((x) & 0xff) << 0) | ||
| 751 | # define DESCRIPTION5(x) (((x) & 0xff) << 8) | ||
| 752 | # define DESCRIPTION6(x) (((x) & 0xff) << 16) | ||
| 753 | # define DESCRIPTION7(x) (((x) & 0xff) << 24) | ||
| 754 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO6 0x40 | ||
| 755 | # define DESCRIPTION8(x) (((x) & 0xff) << 0) | ||
| 756 | # define DESCRIPTION9(x) (((x) & 0xff) << 8) | ||
| 757 | # define DESCRIPTION10(x) (((x) & 0xff) << 16) | ||
| 758 | # define DESCRIPTION11(x) (((x) & 0xff) << 24) | ||
| 759 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO7 0x41 | ||
| 760 | # define DESCRIPTION12(x) (((x) & 0xff) << 0) | ||
| 761 | # define DESCRIPTION13(x) (((x) & 0xff) << 8) | ||
| 762 | # define DESCRIPTION14(x) (((x) & 0xff) << 16) | ||
| 763 | # define DESCRIPTION15(x) (((x) & 0xff) << 24) | ||
| 764 | #define AZ_F0_CODEC_PIN_CONTROL_SINK_INFO8 0x42 | ||
| 765 | # define DESCRIPTION16(x) (((x) & 0xff) << 0) | ||
| 766 | # define DESCRIPTION17(x) (((x) & 0xff) << 8) | ||
| 767 | |||
| 768 | #define AZ_F0_CODEC_PIN_CONTROL_HOT_PLUG_CONTROL 0x54 | ||
| 769 | # define AUDIO_ENABLED (1 << 31) | ||
| 770 | |||
| 771 | #define AZ_F0_CODEC_PIN_CONTROL_RESPONSE_CONFIGURATION_DEFAULT 0x56 | ||
| 772 | #define PORT_CONNECTIVITY_MASK (3 << 30) | ||
| 773 | #define PORT_CONNECTIVITY_SHIFT 30 | ||
| 774 | |||
| 775 | #define DC_LB_MEMORY_SPLIT 0x1AC3 | ||
| 776 | #define DC_LB_MEMORY_CONFIG(x) ((x) << 20) | ||
| 777 | |||
| 778 | #define PRIORITY_A_CNT 0x1AC6 | ||
| 779 | #define PRIORITY_MARK_MASK 0x7fff | ||
| 780 | #define PRIORITY_OFF (1 << 16) | ||
| 781 | #define PRIORITY_ALWAYS_ON (1 << 20) | ||
| 782 | #define PRIORITY_B_CNT 0x1AC7 | ||
| 783 | |||
| 784 | #define DPG_PIPE_ARBITRATION_CONTROL3 0x1B32 | ||
| 785 | # define LATENCY_WATERMARK_MASK(x) ((x) << 16) | ||
| 786 | #define DPG_PIPE_LATENCY_CONTROL 0x1B33 | ||
| 787 | # define LATENCY_LOW_WATERMARK(x) ((x) << 0) | ||
| 788 | # define LATENCY_HIGH_WATERMARK(x) ((x) << 16) | ||
| 789 | |||
| 790 | /* 0x6bb8, 0x77b8, 0x103b8, 0x10fb8, 0x11bb8, 0x127b8 */ | ||
| 791 | #define VLINE_STATUS 0x1AEE | ||
| 792 | # define VLINE_OCCURRED (1 << 0) | ||
| 793 | # define VLINE_ACK (1 << 4) | ||
| 794 | # define VLINE_STAT (1 << 12) | ||
| 795 | # define VLINE_INTERRUPT (1 << 16) | ||
| 796 | # define VLINE_INTERRUPT_TYPE (1 << 17) | ||
| 797 | /* 0x6bbc, 0x77bc, 0x103bc, 0x10fbc, 0x11bbc, 0x127bc */ | ||
| 798 | #define VBLANK_STATUS 0x1AEF | ||
| 799 | # define VBLANK_OCCURRED (1 << 0) | ||
| 800 | # define VBLANK_ACK (1 << 4) | ||
| 801 | # define VBLANK_STAT (1 << 12) | ||
| 802 | # define VBLANK_INTERRUPT (1 << 16) | ||
| 803 | # define VBLANK_INTERRUPT_TYPE (1 << 17) | ||
| 804 | |||
| 805 | /* 0x6b40, 0x7740, 0x10340, 0x10f40, 0x11b40, 0x12740 */ | ||
| 806 | #define INT_MASK 0x1AD0 | ||
| 807 | # define VBLANK_INT_MASK (1 << 0) | ||
| 808 | # define VLINE_INT_MASK (1 << 4) | ||
| 809 | |||
| 810 | #define DISP_INTERRUPT_STATUS 0x183D | ||
| 811 | # define LB_D1_VLINE_INTERRUPT (1 << 2) | ||
| 812 | # define LB_D1_VBLANK_INTERRUPT (1 << 3) | ||
| 813 | # define DC_HPD1_INTERRUPT (1 << 17) | ||
| 814 | # define DC_HPD1_RX_INTERRUPT (1 << 18) | ||
| 815 | # define DACA_AUTODETECT_INTERRUPT (1 << 22) | ||
| 816 | # define DACB_AUTODETECT_INTERRUPT (1 << 23) | ||
| 817 | # define DC_I2C_SW_DONE_INTERRUPT (1 << 24) | ||
| 818 | # define DC_I2C_HW_DONE_INTERRUPT (1 << 25) | ||
| 819 | #define DISP_INTERRUPT_STATUS_CONTINUE 0x183E | ||
| 820 | # define LB_D2_VLINE_INTERRUPT (1 << 2) | ||
| 821 | # define LB_D2_VBLANK_INTERRUPT (1 << 3) | ||
| 822 | # define DC_HPD2_INTERRUPT (1 << 17) | ||
| 823 | # define DC_HPD2_RX_INTERRUPT (1 << 18) | ||
| 824 | # define DISP_TIMER_INTERRUPT (1 << 24) | ||
| 825 | #define DISP_INTERRUPT_STATUS_CONTINUE2 0x183F | ||
| 826 | # define LB_D3_VLINE_INTERRUPT (1 << 2) | ||
| 827 | # define LB_D3_VBLANK_INTERRUPT (1 << 3) | ||
| 828 | # define DC_HPD3_INTERRUPT (1 << 17) | ||
| 829 | # define DC_HPD3_RX_INTERRUPT (1 << 18) | ||
| 830 | #define DISP_INTERRUPT_STATUS_CONTINUE3 0x1840 | ||
| 831 | # define LB_D4_VLINE_INTERRUPT (1 << 2) | ||
| 832 | # define LB_D4_VBLANK_INTERRUPT (1 << 3) | ||
| 833 | # define DC_HPD4_INTERRUPT (1 << 17) | ||
| 834 | # define DC_HPD4_RX_INTERRUPT (1 << 18) | ||
| 835 | #define DISP_INTERRUPT_STATUS_CONTINUE4 0x1853 | ||
| 836 | # define LB_D5_VLINE_INTERRUPT (1 << 2) | ||
| 837 | # define LB_D5_VBLANK_INTERRUPT (1 << 3) | ||
| 838 | # define DC_HPD5_INTERRUPT (1 << 17) | ||
| 839 | # define DC_HPD5_RX_INTERRUPT (1 << 18) | ||
| 840 | #define DISP_INTERRUPT_STATUS_CONTINUE5 0x1854 | ||
| 841 | # define LB_D6_VLINE_INTERRUPT (1 << 2) | ||
| 842 | # define LB_D6_VBLANK_INTERRUPT (1 << 3) | ||
| 843 | # define DC_HPD6_INTERRUPT (1 << 17) | ||
| 844 | # define DC_HPD6_RX_INTERRUPT (1 << 18) | ||
| 845 | |||
| 846 | /* 0x6858, 0x7458, 0x10058, 0x10c58, 0x11858, 0x12458 */ | ||
| 847 | #define GRPH_INT_STATUS 0x1A16 | ||
| 848 | # define GRPH_PFLIP_INT_OCCURRED (1 << 0) | ||
| 849 | # define GRPH_PFLIP_INT_CLEAR (1 << 8) | ||
| 850 | /* 0x685c, 0x745c, 0x1005c, 0x10c5c, 0x1185c, 0x1245c */ | ||
| 851 | #define GRPH_INT_CONTROL 0x1A17 | ||
| 852 | # define GRPH_PFLIP_INT_MASK (1 << 0) | ||
| 853 | # define GRPH_PFLIP_INT_TYPE (1 << 8) | ||
| 854 | |||
| 855 | #define DAC_AUTODETECT_INT_CONTROL 0x19F2 | ||
| 856 | |||
| 857 | #define DC_HPD1_INT_STATUS 0x1807 | ||
| 858 | #define DC_HPD2_INT_STATUS 0x180A | ||
| 859 | #define DC_HPD3_INT_STATUS 0x180D | ||
| 860 | #define DC_HPD4_INT_STATUS 0x1810 | ||
| 861 | #define DC_HPD5_INT_STATUS 0x1813 | ||
| 862 | #define DC_HPD6_INT_STATUS 0x1816 | ||
| 863 | # define DC_HPDx_INT_STATUS (1 << 0) | ||
| 864 | # define DC_HPDx_SENSE (1 << 1) | ||
| 865 | # define DC_HPDx_RX_INT_STATUS (1 << 8) | ||
| 866 | |||
| 867 | #define DC_HPD1_INT_CONTROL 0x1808 | ||
| 868 | #define DC_HPD2_INT_CONTROL 0x180B | ||
| 869 | #define DC_HPD3_INT_CONTROL 0x180E | ||
| 870 | #define DC_HPD4_INT_CONTROL 0x1811 | ||
| 871 | #define DC_HPD5_INT_CONTROL 0x1814 | ||
| 872 | #define DC_HPD6_INT_CONTROL 0x1817 | ||
| 873 | # define DC_HPDx_INT_ACK (1 << 0) | ||
| 874 | # define DC_HPDx_INT_POLARITY (1 << 8) | ||
| 875 | # define DC_HPDx_INT_EN (1 << 16) | ||
| 876 | # define DC_HPDx_RX_INT_ACK (1 << 20) | ||
| 877 | # define DC_HPDx_RX_INT_EN (1 << 24) | ||
| 878 | |||
| 879 | #define DC_HPD1_CONTROL 0x1809 | ||
| 880 | #define DC_HPD2_CONTROL 0x180C | ||
| 881 | #define DC_HPD3_CONTROL 0x180F | ||
| 882 | #define DC_HPD4_CONTROL 0x1812 | ||
| 883 | #define DC_HPD5_CONTROL 0x1815 | ||
| 884 | #define DC_HPD6_CONTROL 0x1818 | ||
| 885 | # define DC_HPDx_CONNECTION_TIMER(x) ((x) << 0) | ||
| 886 | # define DC_HPDx_RX_INT_TIMER(x) ((x) << 16) | ||
| 887 | # define DC_HPDx_EN (1 << 28) | ||
| 888 | |||
| 889 | #define DPG_PIPE_STUTTER_CONTROL 0x1B35 | ||
| 890 | # define STUTTER_ENABLE (1 << 0) | ||
| 891 | |||
| 892 | /* 0x6e98, 0x7a98, 0x10698, 0x11298, 0x11e98, 0x12a98 */ | ||
| 893 | #define CRTC_STATUS_FRAME_COUNT 0x1BA6 | ||
| 894 | |||
| 895 | /* Audio clocks */ | ||
| 896 | #define DCCG_AUDIO_DTO0_PHASE 0x05b0 | ||
| 897 | #define DCCG_AUDIO_DTO0_MODULE 0x05b4 | ||
| 898 | #define DCCG_AUDIO_DTO1_PHASE 0x05c0 | ||
| 899 | #define DCCG_AUDIO_DTO1_MODULE 0x05c4 | ||
| 900 | |||
| 901 | #define GRBM_CNTL 0x2000 | ||
| 902 | #define GRBM_READ_TIMEOUT(x) ((x) << 0) | ||
| 903 | |||
| 904 | #define GRBM_STATUS2 0x2002 | ||
| 905 | #define RLC_RQ_PENDING (1 << 0) | ||
| 906 | #define RLC_BUSY (1 << 8) | ||
| 907 | #define TC_BUSY (1 << 9) | ||
| 908 | |||
| 909 | #define GRBM_STATUS 0x2004 | ||
| 910 | #define CMDFIFO_AVAIL_MASK 0x0000000F | ||
| 911 | #define RING2_RQ_PENDING (1 << 4) | ||
| 912 | #define SRBM_RQ_PENDING (1 << 5) | ||
| 913 | #define RING1_RQ_PENDING (1 << 6) | ||
| 914 | #define CF_RQ_PENDING (1 << 7) | ||
| 915 | #define PF_RQ_PENDING (1 << 8) | ||
| 916 | #define GDS_DMA_RQ_PENDING (1 << 9) | ||
| 917 | #define GRBM_EE_BUSY (1 << 10) | ||
| 918 | #define DB_CLEAN (1 << 12) | ||
| 919 | #define CB_CLEAN (1 << 13) | ||
| 920 | #define TA_BUSY (1 << 14) | ||
| 921 | #define GDS_BUSY (1 << 15) | ||
| 922 | #define VGT_BUSY (1 << 17) | ||
| 923 | #define IA_BUSY_NO_DMA (1 << 18) | ||
| 924 | #define IA_BUSY (1 << 19) | ||
| 925 | #define SX_BUSY (1 << 20) | ||
| 926 | #define SPI_BUSY (1 << 22) | ||
| 927 | #define BCI_BUSY (1 << 23) | ||
| 928 | #define SC_BUSY (1 << 24) | ||
| 929 | #define PA_BUSY (1 << 25) | ||
| 930 | #define DB_BUSY (1 << 26) | ||
| 931 | #define CP_COHERENCY_BUSY (1 << 28) | ||
| 932 | #define CP_BUSY (1 << 29) | ||
| 933 | #define CB_BUSY (1 << 30) | ||
| 934 | #define GUI_ACTIVE (1 << 31) | ||
| 935 | #define GRBM_STATUS_SE0 0x2005 | ||
| 936 | #define GRBM_STATUS_SE1 0x2006 | ||
| 937 | #define SE_DB_CLEAN (1 << 1) | ||
| 938 | #define SE_CB_CLEAN (1 << 2) | ||
| 939 | #define SE_BCI_BUSY (1 << 22) | ||
| 940 | #define SE_VGT_BUSY (1 << 23) | ||
| 941 | #define SE_PA_BUSY (1 << 24) | ||
| 942 | #define SE_TA_BUSY (1 << 25) | ||
| 943 | #define SE_SX_BUSY (1 << 26) | ||
| 944 | #define SE_SPI_BUSY (1 << 27) | ||
| 945 | #define SE_SC_BUSY (1 << 29) | ||
| 946 | #define SE_DB_BUSY (1 << 30) | ||
| 947 | #define SE_CB_BUSY (1 << 31) | ||
| 948 | |||
| 949 | #define GRBM_INT_CNTL 0x2018 | ||
| 950 | # define RDERR_INT_ENABLE (1 << 0) | ||
| 951 | # define GUI_IDLE_INT_ENABLE (1 << 19) | ||
| 952 | |||
| 953 | #define CP_STRMOUT_CNTL 0x213F | ||
| 954 | #define SCRATCH_REG0 0x2140 | ||
| 955 | #define SCRATCH_REG1 0x2141 | ||
| 956 | #define SCRATCH_REG2 0x2142 | ||
| 957 | #define SCRATCH_REG3 0x2143 | ||
| 958 | #define SCRATCH_REG4 0x2144 | ||
| 959 | #define SCRATCH_REG5 0x2145 | ||
| 960 | #define SCRATCH_REG6 0x2146 | ||
| 961 | #define SCRATCH_REG7 0x2147 | ||
| 962 | |||
| 963 | #define SCRATCH_UMSK 0x2150 | ||
| 964 | #define SCRATCH_ADDR 0x2151 | ||
| 965 | |||
| 966 | #define CP_SEM_WAIT_TIMER 0x216F | ||
| 967 | |||
| 968 | #define CP_SEM_INCOMPLETE_TIMER_CNTL 0x2172 | ||
| 969 | |||
| 970 | #define CP_ME_CNTL 0x21B6 | ||
| 971 | #define CP_CE_HALT (1 << 24) | ||
| 972 | #define CP_PFP_HALT (1 << 26) | ||
| 973 | #define CP_ME_HALT (1 << 28) | ||
| 974 | |||
| 975 | #define CP_COHER_CNTL2 0x217A | ||
| 976 | |||
| 977 | #define CP_RB2_RPTR 0x21BE | ||
| 978 | #define CP_RB1_RPTR 0x21BF | ||
| 979 | #define CP_RB0_RPTR 0x21C0 | ||
| 980 | #define CP_RB_WPTR_DELAY 0x21C1 | ||
| 981 | |||
| 982 | #define CP_QUEUE_THRESHOLDS 0x21D8 | ||
| 983 | #define ROQ_IB1_START(x) ((x) << 0) | ||
| 984 | #define ROQ_IB2_START(x) ((x) << 8) | ||
| 985 | #define CP_MEQ_THRESHOLDS 0x21D9 | ||
| 986 | #define MEQ1_START(x) ((x) << 0) | ||
| 987 | #define MEQ2_START(x) ((x) << 8) | ||
| 988 | |||
| 989 | #define CP_PERFMON_CNTL 0x21FF | ||
| 990 | |||
| 991 | #define VGT_VTX_VECT_EJECT_REG 0x222C | 276 | #define VGT_VTX_VECT_EJECT_REG 0x222C |
| 992 | |||
| 993 | #define VGT_ESGS_RING_SIZE 0x2232 | 277 | #define VGT_ESGS_RING_SIZE 0x2232 |
| 994 | #define VGT_GSVS_RING_SIZE 0x2233 | 278 | #define VGT_GSVS_RING_SIZE 0x2233 |
| 995 | |||
| 996 | #define VGT_GS_VERTEX_REUSE 0x2235 | 279 | #define VGT_GS_VERTEX_REUSE 0x2235 |
| 997 | |||
| 998 | #define VGT_PRIMITIVE_TYPE 0x2256 | 280 | #define VGT_PRIMITIVE_TYPE 0x2256 |
| 999 | #define VGT_INDEX_TYPE 0x2257 | 281 | #define VGT_INDEX_TYPE 0x2257 |
| 1000 | |||
| 1001 | #define VGT_NUM_INDICES 0x225C | 282 | #define VGT_NUM_INDICES 0x225C |
| 1002 | #define VGT_NUM_INSTANCES 0x225D | 283 | #define VGT_NUM_INSTANCES 0x225D |
| 1003 | |||
| 1004 | #define VGT_TF_RING_SIZE 0x2262 | 284 | #define VGT_TF_RING_SIZE 0x2262 |
| 1005 | |||
| 1006 | #define VGT_HS_OFFCHIP_PARAM 0x226C | 285 | #define VGT_HS_OFFCHIP_PARAM 0x226C |
| 1007 | |||
| 1008 | #define VGT_TF_MEMORY_BASE 0x226E | 286 | #define VGT_TF_MEMORY_BASE 0x226E |
| 1009 | 287 | ||
| 1010 | #define PA_CL_ENHANCE 0x2285 | ||
| 1011 | #define CLIP_VTX_REORDER_ENA (1 << 0) | ||
| 1012 | #define NUM_CLIP_SEQ(x) ((x) << 1) | ||
| 1013 | |||
| 1014 | #define PA_SU_LINE_STIPPLE_VALUE 0x2298 | ||
| 1015 | |||
| 1016 | #define PA_SC_LINE_STIPPLE_STATE 0x22C4 | ||
| 1017 | |||
| 1018 | #define PA_SC_FORCE_EOV_MAX_CNTS 0x22C9 | ||
| 1019 | #define FORCE_EOV_MAX_CLK_CNT(x) ((x) << 0) | ||
| 1020 | #define FORCE_EOV_MAX_REZ_CNT(x) ((x) << 16) | ||
| 1021 | |||
| 1022 | #define PA_SC_FIFO_SIZE 0x22F3 | ||
| 1023 | #define SC_FRONTEND_PRIM_FIFO_SIZE(x) ((x) << 0) | ||
| 1024 | #define SC_BACKEND_PRIM_FIFO_SIZE(x) ((x) << 6) | ||
| 1025 | #define SC_HIZ_TILE_FIFO_SIZE(x) ((x) << 15) | ||
| 1026 | #define SC_EARLYZ_TILE_FIFO_SIZE(x) ((x) << 23) | ||
| 1027 | |||
| 1028 | #define PA_SC_ENHANCE 0x22FC | 288 | #define PA_SC_ENHANCE 0x22FC |
| 1029 | 289 | ||
| 1030 | #define SQ_CONFIG 0x2300 | ||
| 1031 | |||
| 1032 | #define SQC_CACHES 0x2302 | ||
| 1033 | |||
| 1034 | #define SQ_POWER_THROTTLE 0x2396 | ||
| 1035 | #define MIN_POWER(x) ((x) << 0) | ||
| 1036 | #define MIN_POWER_MASK (0x3fff << 0) | ||
| 1037 | #define MIN_POWER_SHIFT 0 | ||
| 1038 | #define MAX_POWER(x) ((x) << 16) | ||
| 1039 | #define MAX_POWER_MASK (0x3fff << 16) | ||
| 1040 | #define MAX_POWER_SHIFT 0 | ||
| 1041 | #define SQ_POWER_THROTTLE2 0x2397 | ||
| 1042 | #define MAX_POWER_DELTA(x) ((x) << 0) | ||
| 1043 | #define MAX_POWER_DELTA_MASK (0x3fff << 0) | ||
| 1044 | #define MAX_POWER_DELTA_SHIFT 0 | ||
| 1045 | #define STI_SIZE(x) ((x) << 16) | ||
| 1046 | #define STI_SIZE_MASK (0x3ff << 16) | ||
| 1047 | #define STI_SIZE_SHIFT 16 | ||
| 1048 | #define LTI_RATIO(x) ((x) << 27) | ||
| 1049 | #define LTI_RATIO_MASK (0xf << 27) | ||
| 1050 | #define LTI_RATIO_SHIFT 27 | ||
| 1051 | |||
| 1052 | #define SX_DEBUG_1 0x2418 | ||
| 1053 | |||
| 1054 | #define SPI_STATIC_THREAD_MGMT_1 0x2438 | ||
| 1055 | #define SPI_STATIC_THREAD_MGMT_2 0x2439 | ||
| 1056 | #define SPI_STATIC_THREAD_MGMT_3 0x243A | ||
| 1057 | #define SPI_PS_MAX_WAVE_ID 0x243B | ||
| 1058 | |||
| 1059 | #define SPI_CONFIG_CNTL 0x2440 | ||
| 1060 | |||
| 1061 | #define SPI_CONFIG_CNTL_1 0x244F | ||
| 1062 | #define VTX_DONE_DELAY(x) ((x) << 0) | ||
| 1063 | #define INTERP_ONE_PRIM_PER_ROW (1 << 4) | ||
| 1064 | |||
| 1065 | #define CGTS_TCC_DISABLE 0x2452 | ||
| 1066 | #define CGTS_USER_TCC_DISABLE 0x2453 | ||
| 1067 | #define TCC_DISABLE_MASK 0xFFFF0000 | ||
| 1068 | #define TCC_DISABLE_SHIFT 16 | ||
| 1069 | #define CGTS_SM_CTRL_REG 0x2454 | ||
| 1070 | #define OVERRIDE (1 << 21) | ||
| 1071 | #define LS_OVERRIDE (1 << 22) | ||
| 1072 | |||
| 1073 | #define SPI_LB_CU_MASK 0x24D5 | ||
| 1074 | |||
| 1075 | #define TA_CNTL_AUX 0x2542 | 290 | #define TA_CNTL_AUX 0x2542 |
| 1076 | 291 | ||
| 1077 | #define CC_RB_BACKEND_DISABLE 0x263D | ||
| 1078 | #define BACKEND_DISABLE(x) ((x) << 16) | ||
| 1079 | #define GB_ADDR_CONFIG 0x263E | ||
| 1080 | #define NUM_PIPES(x) ((x) << 0) | ||
| 1081 | #define NUM_PIPES_MASK 0x00000007 | ||
| 1082 | #define NUM_PIPES_SHIFT 0 | ||
| 1083 | #define PIPE_INTERLEAVE_SIZE(x) ((x) << 4) | ||
| 1084 | #define PIPE_INTERLEAVE_SIZE_MASK 0x00000070 | ||
| 1085 | #define PIPE_INTERLEAVE_SIZE_SHIFT 4 | ||
| 1086 | #define NUM_SHADER_ENGINES(x) ((x) << 12) | ||
| 1087 | #define NUM_SHADER_ENGINES_MASK 0x00003000 | ||
| 1088 | #define NUM_SHADER_ENGINES_SHIFT 12 | ||
| 1089 | #define SHADER_ENGINE_TILE_SIZE(x) ((x) << 16) | ||
| 1090 | #define SHADER_ENGINE_TILE_SIZE_MASK 0x00070000 | ||
| 1091 | #define SHADER_ENGINE_TILE_SIZE_SHIFT 16 | ||
| 1092 | #define NUM_GPUS(x) ((x) << 20) | ||
| 1093 | #define NUM_GPUS_MASK 0x00700000 | ||
| 1094 | #define NUM_GPUS_SHIFT 20 | ||
| 1095 | #define MULTI_GPU_TILE_SIZE(x) ((x) << 24) | ||
| 1096 | #define MULTI_GPU_TILE_SIZE_MASK 0x03000000 | ||
| 1097 | #define MULTI_GPU_TILE_SIZE_SHIFT 24 | ||
| 1098 | #define ROW_SIZE(x) ((x) << 28) | ||
| 1099 | #define ROW_SIZE_MASK 0x30000000 | ||
| 1100 | #define ROW_SIZE_SHIFT 28 | ||
| 1101 | |||
| 1102 | #define CB_PERFCOUNTER0_SELECT0 0x2688 | ||
| 1103 | #define CB_PERFCOUNTER0_SELECT1 0x2689 | ||
| 1104 | #define CB_PERFCOUNTER1_SELECT0 0x268A | ||
| 1105 | #define CB_PERFCOUNTER1_SELECT1 0x268B | ||
| 1106 | #define CB_PERFCOUNTER2_SELECT0 0x268C | ||
| 1107 | #define CB_PERFCOUNTER2_SELECT1 0x268D | ||
| 1108 | #define CB_PERFCOUNTER3_SELECT0 0x268E | ||
| 1109 | #define CB_PERFCOUNTER3_SELECT1 0x268F | ||
| 1110 | |||
| 1111 | #define CB_CGTT_SCLK_CTRL 0x2698 | ||
| 1112 | |||
| 1113 | #define TCP_CHAN_STEER_LO 0x2B03 | ||
| 1114 | #define TCP_CHAN_STEER_HI 0x2B94 | ||
| 1115 | |||
| 1116 | #define CP_RB0_BASE 0x3040 | ||
| 1117 | #define CP_RB0_CNTL 0x3041 | ||
| 1118 | #define RB_BUFSZ(x) ((x) << 0) | ||
| 1119 | #define RB_BLKSZ(x) ((x) << 8) | ||
| 1120 | #define BUF_SWAP_32BIT (2 << 16) | ||
| 1121 | #define RB_NO_UPDATE (1 << 27) | ||
| 1122 | #define RB_RPTR_WR_ENA (1 << 31) | ||
| 1123 | |||
| 1124 | #define CP_RB0_RPTR_ADDR 0x3043 | ||
| 1125 | #define CP_RB0_RPTR_ADDR_HI 0x3044 | ||
| 1126 | #define CP_RB0_WPTR 0x3045 | ||
| 1127 | |||
| 1128 | #define CP_PFP_UCODE_ADDR 0x3054 | ||
| 1129 | #define CP_PFP_UCODE_DATA 0x3055 | ||
| 1130 | #define CP_ME_RAM_RADDR 0x3056 | ||
| 1131 | #define CP_ME_RAM_WADDR 0x3057 | ||
| 1132 | #define CP_ME_RAM_DATA 0x3058 | ||
| 1133 | |||
| 1134 | #define CP_CE_UCODE_ADDR 0x305A | ||
| 1135 | #define CP_CE_UCODE_DATA 0x305B | ||
| 1136 | |||
| 1137 | #define CP_RB1_BASE 0x3060 | ||
| 1138 | #define CP_RB1_CNTL 0x3061 | ||
| 1139 | #define CP_RB1_RPTR_ADDR 0x3062 | ||
| 1140 | #define CP_RB1_RPTR_ADDR_HI 0x3063 | ||
| 1141 | #define CP_RB1_WPTR 0x3064 | ||
| 1142 | #define CP_RB2_BASE 0x3065 | ||
| 1143 | #define CP_RB2_CNTL 0x3066 | ||
| 1144 | #define CP_RB2_RPTR_ADDR 0x3067 | ||
| 1145 | #define CP_RB2_RPTR_ADDR_HI 0x3068 | ||
| 1146 | #define CP_RB2_WPTR 0x3069 | ||
| 1147 | #define CP_INT_CNTL_RING0 0x306A | ||
| 1148 | #define CP_INT_CNTL_RING1 0x306B | ||
| 1149 | #define CP_INT_CNTL_RING2 0x306C | ||
| 1150 | # define CNTX_BUSY_INT_ENABLE (1 << 19) | ||
| 1151 | # define CNTX_EMPTY_INT_ENABLE (1 << 20) | ||
| 1152 | # define WAIT_MEM_SEM_INT_ENABLE (1 << 21) | ||
| 1153 | # define TIME_STAMP_INT_ENABLE (1 << 26) | ||
| 1154 | # define CP_RINGID2_INT_ENABLE (1 << 29) | ||
| 1155 | # define CP_RINGID1_INT_ENABLE (1 << 30) | ||
| 1156 | # define CP_RINGID0_INT_ENABLE (1 << 31) | ||
| 1157 | #define CP_INT_STATUS_RING0 0x306D | ||
| 1158 | #define CP_INT_STATUS_RING1 0x306E | ||
| 1159 | #define CP_INT_STATUS_RING2 0x306F | ||
| 1160 | # define WAIT_MEM_SEM_INT_STAT (1 << 21) | ||
| 1161 | # define TIME_STAMP_INT_STAT (1 << 26) | ||
| 1162 | # define CP_RINGID2_INT_STAT (1 << 29) | ||
| 1163 | # define CP_RINGID1_INT_STAT (1 << 30) | ||
| 1164 | # define CP_RINGID0_INT_STAT (1 << 31) | ||
| 1165 | |||
| 1166 | // #define PA_SC_RASTER_CONFIG 0xA0D4 | 292 | // #define PA_SC_RASTER_CONFIG 0xA0D4 |
| 1167 | # define RB_XSEL2(x) ((x) << 4) | 293 | # define RB_XSEL2(x) ((x) << 4) |
| 1168 | # define RB_XSEL2_MASK (0x3 << 4) | 294 | # define RB_XSEL2_MASK (0x3 << 4) |
| @@ -1185,171 +311,14 @@ · 16 unmodified lines | |||
| 1185 | # define SE_YSEL(x) ((x) << 28) | 311 | # define SE_YSEL(x) ((x) << 28) |
| 1186 | # define SE_YSEL_MASK (0x3 << 28) | 312 | # define SE_YSEL_MASK (0x3 << 28) |
| 1187 | 313 | ||
| 1188 | /* PIF PHY0 registers idx/data 0x8/0xc */ | ||
| 1189 | #define PB0_PIF_CNTL 0x10 | ||
| 1190 | # define LS2_EXIT_TIME(x) ((x) << 17) | ||
| 1191 | # define LS2_EXIT_TIME_MASK (0x7 << 17) | ||
| 1192 | # define LS2_EXIT_TIME_SHIFT 17 | ||
| 1193 | #define PB0_PIF_PAIRING 0x11 | ||
| 1194 | # define MULTI_PIF (1 << 25) | ||
| 1195 | #define PB0_PIF_PWRDOWN_0 0x12 | ||
| 1196 | # define PLL_POWER_STATE_IN_TXS2_0(x) ((x) << 7) | ||
| 1197 | # define PLL_POWER_STATE_IN_TXS2_0_MASK (0x7 << 7) | ||
| 1198 | # define PLL_POWER_STATE_IN_TXS2_0_SHIFT 7 | ||
| 1199 | # define PLL_POWER_STATE_IN_OFF_0(x) ((x) << 10) | ||
| 1200 | # define PLL_POWER_STATE_IN_OFF_0_MASK (0x7 << 10) | ||
| 1201 | # define PLL_POWER_STATE_IN_OFF_0_SHIFT 10 | ||
| 1202 | # define PLL_RAMP_UP_TIME_0(x) ((x) << 24) | ||
| 1203 | # define PLL_RAMP_UP_TIME_0_MASK (0x7 << 24) | ||
| 1204 | # define PLL_RAMP_UP_TIME_0_SHIFT 24 | ||
| 1205 | #define PB0_PIF_PWRDOWN_1 0x13 | ||
| 1206 | # define PLL_POWER_STATE_IN_TXS2_1(x) ((x) << 7) | ||
| 1207 | # define PLL_POWER_STATE_IN_TXS2_1_MASK (0x7 << 7) | ||
| 1208 | # define PLL_POWER_STATE_IN_TXS2_1_SHIFT 7 | ||
| 1209 | # define PLL_POWER_STATE_IN_OFF_1(x) ((x) << 10) | ||
| 1210 | # define PLL_POWER_STATE_IN_OFF_1_MASK (0x7 << 10) | ||
| 1211 | # define PLL_POWER_STATE_IN_OFF_1_SHIFT 10 | ||
| 1212 | # define PLL_RAMP_UP_TIME_1(x) ((x) << 24) | ||
| 1213 | # define PLL_RAMP_UP_TIME_1_MASK (0x7 << 24) | ||
| 1214 | # define PLL_RAMP_UP_TIME_1_SHIFT 24 | ||
| 1215 | |||
| 1216 | #define PB0_PIF_PWRDOWN_2 0x17 | ||
| 1217 | # define PLL_POWER_STATE_IN_TXS2_2(x) ((x) << 7) | ||
| 1218 | # define PLL_POWER_STATE_IN_TXS2_2_MASK (0x7 << 7) | ||
| 1219 | # define PLL_POWER_STATE_IN_TXS2_2_SHIFT 7 | ||
| 1220 | # define PLL_POWER_STATE_IN_OFF_2(x) ((x) << 10) | ||
| 1221 | # define PLL_POWER_STATE_IN_OFF_2_MASK (0x7 << 10) | ||
| 1222 | # define PLL_POWER_STATE_IN_OFF_2_SHIFT 10 | ||
| 1223 | # define PLL_RAMP_UP_TIME_2(x) ((x) << 24) | ||
| 1224 | # define PLL_RAMP_UP_TIME_2_MASK (0x7 << 24) | ||
| 1225 | # define PLL_RAMP_UP_TIME_2_SHIFT 24 | ||
| 1226 | #define PB0_PIF_PWRDOWN_3 0x18 | ||
| 1227 | # define PLL_POWER_STATE_IN_TXS2_3(x) ((x) << 7) | ||
| 1228 | # define PLL_POWER_STATE_IN_TXS2_3_MASK (0x7 << 7) | ||
| 1229 | # define PLL_POWER_STATE_IN_TXS2_3_SHIFT 7 | ||
| 1230 | # define PLL_POWER_STATE_IN_OFF_3(x) ((x) << 10) | ||
| 1231 | # define PLL_POWER_STATE_IN_OFF_3_MASK (0x7 << 10) | ||
| 1232 | # define PLL_POWER_STATE_IN_OFF_3_SHIFT 10 | ||
| 1233 | # define PLL_RAMP_UP_TIME_3(x) ((x) << 24) | ||
| 1234 | # define PLL_RAMP_UP_TIME_3_MASK (0x7 << 24) | ||
| 1235 | # define PLL_RAMP_UP_TIME_3_SHIFT 24 | ||
| 1236 | /* PIF PHY1 registers idx/data 0x10/0x14 */ | ||
| 1237 | #define PB1_PIF_CNTL 0x10 | ||
| 1238 | #define PB1_PIF_PAIRING 0x11 | ||
| 1239 | #define PB1_PIF_PWRDOWN_0 0x12 | ||
| 1240 | #define PB1_PIF_PWRDOWN_1 0x13 | ||
| 1241 | |||
| 1242 | #define PB1_PIF_PWRDOWN_2 0x17 | ||
| 1243 | #define PB1_PIF_PWRDOWN_3 0x18 | ||
| 1244 | /* PCIE registers idx/data 0x30/0x34 */ | ||
| 1245 | #define PCIE_CNTL2 0x1c /* PCIE */ | ||
| 1246 | # define SLV_MEM_LS_EN (1 << 16) | ||
| 1247 | # define SLV_MEM_AGGRESSIVE_LS_EN (1 << 17) | ||
| 1248 | # define MST_MEM_LS_EN (1 << 18) | ||
| 1249 | # define REPLAY_MEM_LS_EN (1 << 19) | ||
| 1250 | #define PCIE_LC_STATUS1 0x28 /* PCIE */ | ||
| 1251 | # define LC_REVERSE_RCVR (1 << 0) | ||
| 1252 | # define LC_REVERSE_XMIT (1 << 1) | ||
| 1253 | # define LC_OPERATING_LINK_WIDTH_MASK (0x7 << 2) | ||
| 1254 | # define LC_OPERATING_LINK_WIDTH_SHIFT 2 | ||
| 1255 | # define LC_DETECTED_LINK_WIDTH_MASK (0x7 << 5) | ||
| 1256 | # define LC_DETECTED_LINK_WIDTH_SHIFT 5 | ||
| 1257 | |||
| 1258 | #define PCIE_P_CNTL 0x40 /* PCIE */ | ||
| 1259 | # define P_IGNORE_EDB_ERR (1 << 6) | ||
| 1260 | |||
| 1261 | /* PCIE PORT registers idx/data 0x38/0x3c */ | 314 | /* PCIE PORT registers idx/data 0x38/0x3c */ |
| 1262 | #define PCIE_LC_CNTL 0xa0 | 315 | // #define PCIE_LC_LINK_WIDTH_CNTL 0xa2 /* PCIE_P */ |
| 1263 | # define LC_L0S_INACTIVITY(x) ((x) << 8) | ||
| 1264 | # define LC_L0S_INACTIVITY_MASK (0xf << 8) | ||
| 1265 | # define LC_L0S_INACTIVITY_SHIFT 8 | ||
| 1266 | # define LC_L1_INACTIVITY(x) ((x) << 12) | ||
| 1267 | # define LC_L1_INACTIVITY_MASK (0xf << 12) | ||
| 1268 | # define LC_L1_INACTIVITY_SHIFT 12 | ||
| 1269 | # define LC_PMI_TO_L1_DIS (1 << 16) | ||
| 1270 | # define LC_ASPM_TO_L1_DIS (1 << 24) | ||
| 1271 | #define PCIE_LC_LINK_WIDTH_CNTL 0xa2 /* PCIE_P */ | ||
| 1272 | # define LC_LINK_WIDTH_SHIFT 0 | ||
| 1273 | # define LC_LINK_WIDTH_MASK 0x7 | ||
| 1274 | # define LC_LINK_WIDTH_X0 0 | 316 | # define LC_LINK_WIDTH_X0 0 |
| 1275 | # define LC_LINK_WIDTH_X1 1 | 317 | # define LC_LINK_WIDTH_X1 1 |
| 1276 | # define LC_LINK_WIDTH_X2 2 | 318 | # define LC_LINK_WIDTH_X2 2 |
| 1277 | # define LC_LINK_WIDTH_X4 3 | 319 | # define LC_LINK_WIDTH_X4 3 |
| 1278 | # define LC_LINK_WIDTH_X8 4 | 320 | # define LC_LINK_WIDTH_X8 4 |
| 1279 | # define LC_LINK_WIDTH_X16 6 | 321 | # define LC_LINK_WIDTH_X16 6 |
| 1280 | # define LC_LINK_WIDTH_RD_SHIFT 4 | ||
| 1281 | # define LC_LINK_WIDTH_RD_MASK 0x70 | ||
| 1282 | # define LC_RECONFIG_ARC_MISSING_ESCAPE (1 << 7) | ||
| 1283 | # define LC_RECONFIG_NOW (1 << 8) | ||
| 1284 | # define LC_RENEGOTIATION_SUPPORT (1 << 9) | ||
| 1285 | # define LC_RENEGOTIATE_EN (1 << 10) | ||
| 1286 | # define LC_SHORT_RECONFIG_EN (1 << 11) | ||
| 1287 | # define LC_UPCONFIGURE_SUPPORT (1 << 12) | ||
| 1288 | # define LC_UPCONFIGURE_DIS (1 << 13) | ||
| 1289 | # define LC_DYN_LANES_PWR_STATE(x) ((x) << 21) | ||
| 1290 | # define LC_DYN_LANES_PWR_STATE_MASK (0x3 << 21) | ||
| 1291 | # define LC_DYN_LANES_PWR_STATE_SHIFT 21 | ||
| 1292 | #define PCIE_LC_N_FTS_CNTL 0xa3 /* PCIE_P */ | ||
| 1293 | # define LC_XMIT_N_FTS(x) ((x) << 0) | ||
| 1294 | # define LC_XMIT_N_FTS_MASK (0xff << 0) | ||
| 1295 | # define LC_XMIT_N_FTS_SHIFT 0 | ||
| 1296 | # define LC_XMIT_N_FTS_OVERRIDE_EN (1 << 8) | ||
| 1297 | # define LC_N_FTS_MASK (0xff << 24) | ||
| 1298 | #define PCIE_LC_SPEED_CNTL 0xa4 /* PCIE_P */ | ||
| 1299 | # define LC_GEN2_EN_STRAP (1 << 0) | ||
| 1300 | # define LC_GEN3_EN_STRAP (1 << 1) | ||
| 1301 | # define LC_TARGET_LINK_SPEED_OVERRIDE_EN (1 << 2) | ||
| 1302 | # define LC_TARGET_LINK_SPEED_OVERRIDE_MASK (0x3 << 3) | ||
| 1303 | # define LC_TARGET_LINK_SPEED_OVERRIDE_SHIFT 3 | ||
| 1304 | # define LC_FORCE_EN_SW_SPEED_CHANGE (1 << 5) | ||
| 1305 | # define LC_FORCE_DIS_SW_SPEED_CHANGE (1 << 6) | ||
| 1306 | # define LC_FORCE_EN_HW_SPEED_CHANGE (1 << 7) | ||
| 1307 | # define LC_FORCE_DIS_HW_SPEED_CHANGE (1 << 8) | ||
| 1308 | # define LC_INITIATE_LINK_SPEED_CHANGE (1 << 9) | ||
| 1309 | # define LC_SPEED_CHANGE_ATTEMPTS_ALLOWED_MASK (0x3 << 10) | ||
| 1310 | # define LC_SPEED_CHANGE_ATTEMPTS_ALLOWED_SHIFT 10 | ||
| 1311 | # define LC_CURRENT_DATA_RATE_MASK (0x3 << 13) /* 0/1/2 = gen1/2/3 */ | ||
| 1312 | # define LC_CURRENT_DATA_RATE_SHIFT 13 | ||
| 1313 | # define LC_CLR_FAILED_SPD_CHANGE_CNT (1 << 16) | ||
| 1314 | # define LC_OTHER_SIDE_EVER_SENT_GEN2 (1 << 18) | ||
| 1315 | # define LC_OTHER_SIDE_SUPPORTS_GEN2 (1 << 19) | ||
| 1316 | # define LC_OTHER_SIDE_EVER_SENT_GEN3 (1 << 20) | ||
| 1317 | # define LC_OTHER_SIDE_SUPPORTS_GEN3 (1 << 21) | ||
| 1318 | |||
| 1319 | #define PCIE_LC_CNTL2 0xb1 | ||
| 1320 | # define LC_ALLOW_PDWN_IN_L1 (1 << 17) | ||
| 1321 | # define LC_ALLOW_PDWN_IN_L23 (1 << 18) | ||
| 1322 | |||
| 1323 | #define PCIE_LC_CNTL3 0xb5 /* PCIE_P */ | ||
| 1324 | # define LC_GO_TO_RECOVERY (1 << 30) | ||
| 1325 | #define PCIE_LC_CNTL4 0xb6 /* PCIE_P */ | ||
| 1326 | # define LC_REDO_EQ (1 << 5) | ||
| 1327 | # define LC_SET_QUIESCE (1 << 13) | ||
| 1328 | |||
| 1329 | /* | ||
| 1330 | * UVD | ||
| 1331 | */ | ||
| 1332 | #define UVD_UDEC_ADDR_CONFIG 0x3bd3 | ||
| 1333 | #define UVD_UDEC_DB_ADDR_CONFIG 0x3bd4 | ||
| 1334 | #define UVD_UDEC_DBW_ADDR_CONFIG 0x3bd5 | ||
| 1335 | #define UVD_RBC_RB_RPTR 0x3da4 | ||
| 1336 | #define UVD_RBC_RB_WPTR 0x3da5 | ||
| 1337 | #define UVD_STATUS 0x3daf | ||
| 1338 | |||
| 1339 | #define UVD_CGC_CTRL 0x3dc2 | ||
| 1340 | # define DCM (1 << 0) | ||
| 1341 | # define CG_DT(x) ((x) << 2) | ||
| 1342 | # define CG_DT_MASK (0xf << 2) | ||
| 1343 | # define CLK_OD(x) ((x) << 6) | ||
| 1344 | # define CLK_OD_MASK (0x1f << 6) | ||
| 1345 | |||
| 1346 | /* UVD CTX indirect */ | ||
| 1347 | #define UVD_CGC_MEM_CTRL 0xC0 | ||
| 1348 | #define UVD_CGC_CTRL2 0xC1 | ||
| 1349 | # define DYN_OR_EN (1 << 0) | ||
| 1350 | # define DYN_RR_EN (1 << 1) | ||
| 1351 | # define G_DIV_ID(x) ((x) << 2) | ||
| 1352 | # define G_DIV_ID_MASK (0x7 << 2) | ||
| 1353 | 322 | ||
| 1354 | /* | 323 | /* |
| 1355 | * PM4 | 324 | * PM4 |
| @@ -1583,45 +552,7 @@ · 227 unmodified lines | |||
| 1583 | /* ASYNC DMA - first instance at 0xd000, second at 0xd800 */ | 552 | /* ASYNC DMA - first instance at 0xd000, second at 0xd800 */ |
| 1584 | #define DMA0_REGISTER_OFFSET 0x0 /* not a register */ | 553 | #define DMA0_REGISTER_OFFSET 0x0 /* not a register */ |
| 1585 | #define DMA1_REGISTER_OFFSET 0x200 /* not a register */ | 554 | #define DMA1_REGISTER_OFFSET 0x200 /* not a register */ |
| 1586 | 555 | #define SDMA_MAX_INSTANCE 2 | |
| 1587 | #define DMA_RB_CNTL 0x3400 | ||
| 1588 | # define DMA_RB_ENABLE (1 << 0) | ||
| 1589 | # define DMA_RB_SIZE(x) ((x) << 1) /* log2 */ | ||
| 1590 | # define DMA_RB_SWAP_ENABLE (1 << 9) /* 8IN32 */ | ||
| 1591 | # define DMA_RPTR_WRITEBACK_ENABLE (1 << 12) | ||
| 1592 | # define DMA_RPTR_WRITEBACK_SWAP_ENABLE (1 << 13) /* 8IN32 */ | ||
| 1593 | # define DMA_RPTR_WRITEBACK_TIMER(x) ((x) << 16) /* log2 */ | ||
| 1594 | #define DMA_RB_BASE 0x3401 | ||
| 1595 | #define DMA_RB_RPTR 0x3402 | ||
| 1596 | #define DMA_RB_WPTR 0x3403 | ||
| 1597 | |||
| 1598 | #define DMA_RB_RPTR_ADDR_HI 0x3407 | ||
| 1599 | #define DMA_RB_RPTR_ADDR_LO 0x3408 | ||
| 1600 | |||
| 1601 | #define DMA_IB_CNTL 0x3409 | ||
| 1602 | # define DMA_IB_ENABLE (1 << 0) | ||
| 1603 | # define DMA_IB_SWAP_ENABLE (1 << 4) | ||
| 1604 | # define CMD_VMID_FORCE (1 << 31) | ||
| 1605 | #define DMA_IB_RPTR 0x340a | ||
| 1606 | #define DMA_CNTL 0x340b | ||
| 1607 | # define TRAP_ENABLE (1 << 0) | ||
| 1608 | # define SEM_INCOMPLETE_INT_ENABLE (1 << 1) | ||
| 1609 | # define SEM_WAIT_INT_ENABLE (1 << 2) | ||
| 1610 | # define DATA_SWAP_ENABLE (1 << 3) | ||
| 1611 | # define FENCE_SWAP_ENABLE (1 << 4) | ||
| 1612 | # define CTXEMPTY_INT_ENABLE (1 << 28) | ||
| 1613 | #define DMA_STATUS_REG 0x340d | ||
| 1614 | # define DMA_IDLE (1 << 0) | ||
| 1615 | #define DMA_TILING_CONFIG 0x342e | ||
| 1616 | |||
| 1617 | #define DMA_POWER_CNTL 0x342f | ||
| 1618 | # define MEM_POWER_OVERRIDE (1 << 8) | ||
| 1619 | #define DMA_CLK_CTRL 0x3430 | ||
| 1620 | |||
| 1621 | #define DMA_PG 0x3435 | ||
| 1622 | # define PG_CNTL_ENABLE (1 << 0) | ||
| 1623 | #define DMA_PGFSM_CONFIG 0x3436 | ||
| 1624 | #define DMA_PGFSM_WRITE 0x3437 | ||
| 1625 | 556 | ||
| 1626 | #define DMA_PACKET(cmd, b, t, s, n) ((((cmd) & 0xF) << 28) | \ | 557 | #define DMA_PACKET(cmd, b, t, s, n) ((((cmd) & 0xF) << 28) | \ |
| 1627 | (((b) & 0x1) << 26) | \ | 558 | (((b) & 0x1) << 26) | \ |
| @@ -1650,6 +581,7 @@ · 22 unmodified lines | |||
| 1650 | #define DMA_PACKET_POLL_REG_MEM 0xe | 581 | #define DMA_PACKET_POLL_REG_MEM 0xe |
| 1651 | #define DMA_PACKET_NOP 0xf | 582 | #define DMA_PACKET_NOP 0xf |
| 1652 | 583 | ||
| 584 | /* VCE */ | ||
| 1653 | #define VCE_STATUS 0x20004 | 585 | #define VCE_STATUS 0x20004 |
| 1654 | #define VCE_VCPU_CNTL 0x20014 | 586 | #define VCE_VCPU_CNTL 0x20014 |
| 1655 | #define VCE_CLK_EN (1 << 0) | 587 | #define VCE_CLK_EN (1 << 0) |
| @@ -1726,378 +658,118 @@ · 70 unmodified lines | |||
| 1726 | 658 | ||
| 1727 | #define CURSOR_WIDTH 64 | 659 | #define CURSOR_WIDTH 64 |
| 1728 | #define CURSOR_HEIGHT 64 | 660 | #define CURSOR_HEIGHT 64 |
| 1729 | #define AMDGPU_MM_INDEX 0x0000 | 661 | |
| 1730 | #define AMDGPU_MM_DATA 0x0001 | ||
| 1731 | |||
| 1732 | #define VERDE_NUM_CRTC 6 | ||
| 1733 | #define BLACKOUT_MODE_MASK 0x00000007 | ||
| 1734 | #define VGA_RENDER_CONTROL 0xC0 | ||
| 1735 | #define R_000300_VGA_RENDER_CONTROL 0xC0 | ||
| 1736 | #define C_000300_VGA_VSTATUS_CNTL 0xFFFCFFFF | ||
| 1737 | #define EVERGREEN_CRTC_STATUS 0x1BA3 | ||
| 1738 | #define EVERGREEN_CRTC_V_BLANK (1 << 0) | ||
| 1739 | #define EVERGREEN_CRTC_STATUS_POSITION 0x1BA4 | ||
| 1740 | /* CRTC blocks at 0x6df0, 0x79f0, 0x105f0, 0x111f0, 0x11df0, 0x129f0 */ | ||
| 1741 | #define EVERGREEN_CRTC_V_BLANK_START_END 0x1b8d | ||
| 1742 | #define EVERGREEN_CRTC_CONTROL 0x1b9c | ||
| 1743 | #define EVERGREEN_CRTC_MASTER_EN (1 << 0) | ||
| 1744 | #define EVERGREEN_CRTC_DISP_READ_REQUEST_DISABLE (1 << 24) | ||
| 1745 | #define EVERGREEN_CRTC_BLANK_CONTROL 0x1b9d | ||
| 1746 | #define EVERGREEN_CRTC_BLANK_DATA_EN (1 << 8) | ||
| 1747 | #define EVERGREEN_CRTC_V_BLANK (1 << 0) | ||
| 1748 | #define EVERGREEN_CRTC_STATUS_HV_COUNT 0x1ba8 | ||
| 1749 | #define EVERGREEN_CRTC_UPDATE_LOCK 0x1bb5 | ||
| 1750 | #define EVERGREEN_MASTER_UPDATE_LOCK 0x1bbd | ||
| 1751 | #define EVERGREEN_MASTER_UPDATE_MODE 0x1bbe | ||
| 1752 | #define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16) | ||
| 1753 | #define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07 | ||
| 1754 | #define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08 | ||
| 1755 | #define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS 0x1a04 | ||
| 1756 | #define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS 0x1a05 | ||
| 1757 | #define EVERGREEN_GRPH_UPDATE 0x1a11 | ||
| 1758 | #define EVERGREEN_VGA_MEMORY_BASE_ADDRESS 0xc4 | ||
| 1759 | #define EVERGREEN_VGA_MEMORY_BASE_ADDRESS_HIGH 0xc9 | ||
| 1760 | #define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2) | ||
| 1761 | |||
| 1762 | #define EVERGREEN_DATA_FORMAT 0x1ac0 | ||
| 1763 | # define EVERGREEN_INTERLEAVE_EN (1 << 0) | ||
| 1764 | |||
| 1765 | #define R600_D1GRPH_ARRAY_MODE_LINEAR_GENERAL (0 << 20) | ||
| 1766 | #define R600_D1GRPH_ARRAY_MODE_LINEAR_ALIGNED (1 << 20) | ||
| 1767 | #define R600_D1GRPH_ARRAY_MODE_1D_TILED_THIN1 (2 << 20) | ||
| 1768 | #define R600_D1GRPH_ARRAY_MODE_2D_TILED_THIN1 (4 << 20) | ||
| 1769 | |||
| 1770 | #define R700_D1GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a45 | ||
| 1771 | #define R700_D2GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1845 | ||
| 1772 | |||
| 1773 | #define R700_D2GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1847 | ||
| 1774 | #define R700_D1GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a47 | ||
| 1775 | |||
| 1776 | #define R600_D1GRPH_SWAP_CONTROL 0x1843 | ||
| 1777 | #define R600_D1GRPH_SWAP_ENDIAN_NONE (0 << 0) | ||
| 1778 | #define R600_D1GRPH_SWAP_ENDIAN_16BIT (1 << 0) | ||
| 1779 | #define R600_D1GRPH_SWAP_ENDIAN_32BIT (2 << 0) | ||
| 1780 | #define R600_D1GRPH_SWAP_ENDIAN_64BIT (3 << 0) | ||
| 1781 | |||
| 1782 | #define AVIVO_D1VGA_CONTROL 0x00cc | ||
| 1783 | # define AVIVO_DVGA_CONTROL_MODE_ENABLE (1 << 0) | ||
| 1784 | # define AVIVO_DVGA_CONTROL_TIMING_SELECT (1 << 8) | ||
| 1785 | # define AVIVO_DVGA_CONTROL_SYNC_POLARITY_SELECT (1 << 9) | ||
| 1786 | # define AVIVO_DVGA_CONTROL_OVERSCAN_TIMING_SELECT (1 << 10) | ||
| 1787 | # define AVIVO_DVGA_CONTROL_OVERSCAN_COLOR_EN (1 << 16) | ||
| 1788 | # define AVIVO_DVGA_CONTROL_ROTATE (1 << 24) | ||
| 1789 | #define AVIVO_D2VGA_CONTROL 0x00ce | ||
| 1790 | |||
| 1791 | #define R600_BUS_CNTL 0x1508 | ||
| 1792 | # define R600_BIOS_ROM_DIS (1 << 1) | ||
| 1793 | 662 | ||
| 1794 | #define R600_ROM_CNTL 0x580 | 663 | #define R600_ROM_CNTL 0x580 |
| 1795 | # define R600_SCK_OVERWRITE (1 << 1) | 664 | # define R600_SCK_OVERWRITE (1 << 1) |
| 1796 | # define R600_SCK_PRESCALE_CRYSTAL_CLK_SHIFT 28 | 665 | # define R600_SCK_PRESCALE_CRYSTAL_CLK_SHIFT 28 |
| 1797 | # define R600_SCK_PRESCALE_CRYSTAL_CLK_MASK (0xf << 28) | 666 | # define R600_SCK_PRESCALE_CRYSTAL_CLK_MASK (0xf << 28) |
| 1798 | 667 | ||
| 1799 | #define FMT_BIT_DEPTH_CONTROL 0x1bf2 | 668 | #define GRPH_ARRAY_LINEAR_GENERAL 0 |
| 1800 | #define FMT_TRUNCATE_EN (1 << 0) | 669 | #define GRPH_ARRAY_LINEAR_ALIGNED 1 |
| 1801 | #define FMT_TRUNCATE_DEPTH (1 << 4) | 670 | #define GRPH_ARRAY_1D_TILED_THIN1 2 |
| 1802 | #define FMT_SPATIAL_DITHER_EN (1 << 8) | 671 | #define GRPH_ARRAY_2D_TILED_THIN1 4 |
| 1803 | #define FMT_SPATIAL_DITHER_MODE(x) ((x) << 9) | 672 | |
| 1804 | #define FMT_SPATIAL_DITHER_DEPTH (1 << 12) | 673 | #define ES_AND_GS_AUTO 3 |
| 1805 | #define FMT_FRAME_RANDOM_ENABLE (1 << 13) | 674 | #define BUF_SWAP_32BIT (2 << 16) |
| 1806 | #define FMT_RGB_RANDOM_ENABLE (1 << 14) | 675 | |
| 1807 | #define FMT_HIGHPASS_RANDOM_ENABLE (1 << 15) | 676 | #define GRPH_DEPTH_8BPP 0 |
| 1808 | #define FMT_TEMPORAL_DITHER_EN (1 << 16) | 677 | #define GRPH_DEPTH_16BPP 1 |
| 1809 | #define FMT_TEMPORAL_DITHER_DEPTH (1 << 20) | 678 | #define GRPH_DEPTH_32BPP 2 |
| 1810 | #define FMT_TEMPORAL_DITHER_OFFSET(x) ((x) << 21) | 679 | |
| 1811 | #define FMT_TEMPORAL_LEVEL (1 << 24) | 680 | /* 8 BPP */ |
| 1812 | #define FMT_TEMPORAL_DITHER_RESET (1 << 25) | 681 | #define GRPH_FORMAT_INDEXED 0 |
| 1813 | #define FMT_25FRC_SEL(x) ((x) << 26) | 682 | |
| 1814 | #define FMT_50FRC_SEL(x) ((x) << 28) | 683 | /* 16 BPP */ |
| 1815 | #define FMT_75FRC_SEL(x) ((x) << 30) | 684 | #define GRPH_FORMAT_ARGB1555 0 |
| 1816 | 685 | #define GRPH_FORMAT_ARGB565 1 | |
| 1817 | #define EVERGREEN_DC_LUT_CONTROL 0x1a80 | 686 | #define GRPH_FORMAT_ARGB4444 2 |
| 1818 | #define EVERGREEN_DC_LUT_BLACK_OFFSET_BLUE 0x1a81 | 687 | #define GRPH_FORMAT_AI88 3 |
| 1819 | #define EVERGREEN_DC_LUT_BLACK_OFFSET_GREEN 0x1a82 | 688 | #define GRPH_FORMAT_MONO16 4 |
| 1820 | #define EVERGREEN_DC_LUT_BLACK_OFFSET_RED 0x1a83 | 689 | #define GRPH_FORMAT_BGRA5551 5 |
| 1821 | #define EVERGREEN_DC_LUT_WHITE_OFFSET_BLUE 0x1a84 | ||
| 1822 | #define EVERGREEN_DC_LUT_WHITE_OFFSET_GREEN 0x1a85 | ||
| 1823 | #define EVERGREEN_DC_LUT_WHITE_OFFSET_RED 0x1a86 | ||
| 1824 | #define EVERGREEN_DC_LUT_30_COLOR 0x1a7c | ||
| 1825 | #define EVERGREEN_DC_LUT_RW_INDEX 0x1a79 | ||
| 1826 | #define EVERGREEN_DC_LUT_WRITE_EN_MASK 0x1a7e | ||
| 1827 | #define EVERGREEN_DC_LUT_RW_MODE 0x1a78 | ||
| 1828 | |||
| 1829 | #define EVERGREEN_GRPH_ENABLE 0x1a00 | ||
| 1830 | #define EVERGREEN_GRPH_CONTROL 0x1a01 | ||
| 1831 | #define EVERGREEN_GRPH_DEPTH(x) (((x) & 0x3) << 0) | ||
| 1832 | #define EVERGREEN_GRPH_DEPTH_8BPP 0 | ||
| 1833 | #define EVERGREEN_GRPH_DEPTH_16BPP 1 | ||
| 1834 | #define EVERGREEN_GRPH_DEPTH_32BPP 2 | ||
| 1835 | #define EVERGREEN_GRPH_NUM_BANKS(x) (((x) & 0x3) << 2) | ||
| 1836 | #define EVERGREEN_ADDR_SURF_2_BANK 0 | ||
| 1837 | #define EVERGREEN_ADDR_SURF_4_BANK 1 | ||
| 1838 | #define EVERGREEN_ADDR_SURF_8_BANK 2 | ||
| 1839 | #define EVERGREEN_ADDR_SURF_16_BANK 3 | ||
| 1840 | #define EVERGREEN_GRPH_Z(x) (((x) & 0x3) << 4) | ||
| 1841 | #define EVERGREEN_GRPH_BANK_WIDTH(x) (((x) & 0x3) << 6) | ||
| 1842 | #define EVERGREEN_ADDR_SURF_BANK_WIDTH_1 0 | ||
| 1843 | #define EVERGREEN_ADDR_SURF_BANK_WIDTH_2 1 | ||
| 1844 | #define EVERGREEN_ADDR_SURF_BANK_WIDTH_4 2 | ||
| 1845 | #define EVERGREEN_ADDR_SURF_BANK_WIDTH_8 3 | ||
| 1846 | #define EVERGREEN_GRPH_FORMAT(x) (((x) & 0x7) << 8) | ||
| 1847 | |||
| 1848 | #define EVERGREEN_GRPH_FORMAT_INDEXED 0 | ||
| 1849 | #define EVERGREEN_GRPH_FORMAT_ARGB1555 0 | ||
| 1850 | #define EVERGREEN_GRPH_FORMAT_ARGB565 1 | ||
| 1851 | #define EVERGREEN_GRPH_FORMAT_ARGB4444 2 | ||
| 1852 | #define EVERGREEN_GRPH_FORMAT_AI88 3 | ||
| 1853 | #define EVERGREEN_GRPH_FORMAT_MONO16 4 | ||
| 1854 | #define EVERGREEN_GRPH_FORMAT_BGRA5551 5 | ||
| 1855 | 690 | ||
| 1856 | /* 32 BPP */ | 691 | /* 32 BPP */ |
| 1857 | #define EVERGREEN_GRPH_FORMAT_ARGB8888 0 | 692 | #define GRPH_FORMAT_ARGB8888 0 |
| 1858 | #define EVERGREEN_GRPH_FORMAT_ARGB2101010 1 | 693 | #define GRPH_FORMAT_ARGB2101010 1 |
| 1859 | #define EVERGREEN_GRPH_FORMAT_32BPP_DIG 2 | 694 | #define GRPH_FORMAT_32BPP_DIG 2 |
| 1860 | #define EVERGREEN_GRPH_FORMAT_8B_ARGB2101010 3 | 695 | #define GRPH_FORMAT_8B_ARGB2101010 3 |
| 1861 | #define EVERGREEN_GRPH_FORMAT_BGRA1010102 4 | 696 | #define GRPH_FORMAT_BGRA1010102 4 |
| 1862 | #define EVERGREEN_GRPH_FORMAT_8B_BGRA1010102 5 | 697 | #define GRPH_FORMAT_8B_BGRA1010102 5 |
| 1863 | #define EVERGREEN_GRPH_FORMAT_RGB111110 6 | 698 | #define GRPH_FORMAT_RGB111110 6 |
| 1864 | #define EVERGREEN_GRPH_FORMAT_BGR101111 7 | 699 | #define GRPH_FORMAT_BGR101111 7 |
| 1865 | #define EVERGREEN_GRPH_BANK_HEIGHT(x) (((x) & 0x3) << 11) | 700 | |
| 1866 | #define EVERGREEN_ADDR_SURF_BANK_HEIGHT_1 0 | 701 | #define GRPH_ENDIAN_NONE 0 |
| 1867 | #define EVERGREEN_ADDR_SURF_BANK_HEIGHT_2 1 | 702 | #define GRPH_ENDIAN_8IN16 1 |
| 1868 | #define EVERGREEN_ADDR_SURF_BANK_HEIGHT_4 2 | 703 | #define GRPH_ENDIAN_8IN32 2 |
| 1869 | #define EVERGREEN_ADDR_SURF_BANK_HEIGHT_8 3 | 704 | #define GRPH_ENDIAN_8IN64 3 |
| 1870 | #define EVERGREEN_GRPH_TILE_SPLIT(x) (((x) & 0x7) << 13) | 705 | #define GRPH_RED_SEL_R 0 |
| 1871 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_64B 0 | 706 | #define GRPH_RED_SEL_G 1 |
| 1872 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_128B 1 | 707 | #define GRPH_RED_SEL_B 2 |
| 1873 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_256B 2 | 708 | #define GRPH_RED_SEL_A 3 |
| 1874 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_512B 3 | 709 | |
| 1875 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_1KB 4 | 710 | #define GRPH_GREEN_SEL_G 0 |
| 1876 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_2KB 5 | 711 | #define GRPH_GREEN_SEL_B 1 |
| 1877 | #define EVERGREEN_ADDR_SURF_TILE_SPLIT_4KB 6 | 712 | #define GRPH_GREEN_SEL_A 2 |
| 1878 | #define EVERGREEN_GRPH_MACRO_TILE_ASPECT(x) (((x) & 0x3) << 18) | 713 | #define GRPH_GREEN_SEL_R 3 |
| 1879 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_1 0 | 714 | |
| 1880 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_2 1 | 715 | #define GRPH_BLUE_SEL_B 0 |
| 1881 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_4 2 | 716 | #define GRPH_BLUE_SEL_A 1 |
| 1882 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_8 3 | 717 | #define GRPH_BLUE_SEL_R 2 |
| 1883 | #define EVERGREEN_GRPH_ARRAY_MODE(x) (((x) & 0x7) << 20) | 718 | #define GRPH_BLUE_SEL_G 3 |
| 1884 | #define EVERGREEN_GRPH_ARRAY_LINEAR_GENERAL 0 | 719 | |
| 1885 | #define EVERGREEN_GRPH_ARRAY_LINEAR_ALIGNED 1 | 720 | #define GRPH_ALPHA_SEL_A 0 |
| 1886 | #define EVERGREEN_GRPH_ARRAY_1D_TILED_THIN1 2 | 721 | #define GRPH_ALPHA_SEL_R 1 |
| 1887 | #define EVERGREEN_GRPH_ARRAY_2D_TILED_THIN1 4 | 722 | #define GRPH_ALPHA_SEL_G 2 |
| 1888 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_1 0 | 723 | #define GRPH_ALPHA_SEL_B 3 |
| 1889 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_2 1 | 724 | |
| 1890 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_4 2 | 725 | /* CUR_CONTROL */ |
| 1891 | #define EVERGREEN_ADDR_SURF_MACRO_TILE_ASPECT_8 3 | 726 | #define CURSOR_MONO 0 |
| 1892 | 727 | #define CURSOR_24_1 1 | |
| 1893 | #define EVERGREEN_GRPH_SWAP_CONTROL 0x1a03 | 728 | #define CURSOR_24_8_PRE_MULT 2 |
| 1894 | #define EVERGREEN_GRPH_ENDIAN_SWAP(x) (((x) & 0x3) << 0) | 729 | #define CURSOR_24_8_UNPRE_MULT 3 |
| 1895 | # define EVERGREEN_GRPH_ENDIAN_NONE 0 | 730 | #define CURSOR_URGENT_ALWAYS 0 |
| 1896 | # define EVERGREEN_GRPH_ENDIAN_8IN16 1 | 731 | #define CURSOR_URGENT_1_8 1 |
| 1897 | # define EVERGREEN_GRPH_ENDIAN_8IN32 2 | 732 | #define CURSOR_URGENT_1_4 2 |
| 1898 | # define EVERGREEN_GRPH_ENDIAN_8IN64 3 | 733 | #define CURSOR_URGENT_3_8 3 |
| 1899 | #define EVERGREEN_GRPH_RED_CROSSBAR(x) (((x) & 0x3) << 4) | 734 | #define CURSOR_URGENT_1_2 4 |
| 1900 | # define EVERGREEN_GRPH_RED_SEL_R 0 | 735 | |
| 1901 | # define EVERGREEN_GRPH_RED_SEL_G 1 | 736 | /* INPUT_CSC_CONTROL */ |
| 1902 | # define EVERGREEN_GRPH_RED_SEL_B 2 | 737 | # define INPUT_CSC_BYPASS 0 |
| 1903 | # define EVERGREEN_GRPH_RED_SEL_A 3 | 738 | # define INPUT_CSC_PROG_COEFF 1 |
| 1904 | #define EVERGREEN_GRPH_GREEN_CROSSBAR(x) (((x) & 0x3) << 6) | 739 | # define INPUT_CSC_PROG_SHARED_MATRIXA 2 |
| 1905 | # define EVERGREEN_GRPH_GREEN_SEL_G 0 | 740 | |
| 1906 | # define EVERGREEN_GRPH_GREEN_SEL_B 1 | 741 | /* OUTPUT_CSC_CONTROL */ |
| 1907 | # define EVERGREEN_GRPH_GREEN_SEL_A 2 | 742 | # define OUTPUT_CSC_BYPASS 0 |
| 1908 | # define EVERGREEN_GRPH_GREEN_SEL_R 3 | 743 | # define OUTPUT_CSC_TV_RGB 1 |
| 1909 | #define EVERGREEN_GRPH_BLUE_CROSSBAR(x) (((x) & 0x3) << 8) | 744 | # define OUTPUT_CSC_YCBCR_601 2 |
| 1910 | # define EVERGREEN_GRPH_BLUE_SEL_B 0 | 745 | # define OUTPUT_CSC_YCBCR_709 3 |
| 1911 | # define EVERGREEN_GRPH_BLUE_SEL_A 1 | 746 | # define OUTPUT_CSC_PROG_COEFF 4 |
| 1912 | # define EVERGREEN_GRPH_BLUE_SEL_R 2 | 747 | # define OUTPUT_CSC_PROG_SHARED_MATRIXB 5 |
| 1913 | # define EVERGREEN_GRPH_BLUE_SEL_G 3 | 748 | |
| 1914 | #define EVERGREEN_GRPH_ALPHA_CROSSBAR(x) (((x) & 0x3) << 10) | 749 | /* DEGAMMA_CONTROL */ |
| 1915 | # define EVERGREEN_GRPH_ALPHA_SEL_A 0 | 750 | # define DEGAMMA_BYPASS 0 |
| 1916 | # define EVERGREEN_GRPH_ALPHA_SEL_R 1 | 751 | # define DEGAMMA_SRGB_24 1 |
| 1917 | # define EVERGREEN_GRPH_ALPHA_SEL_G 2 | 752 | # define DEGAMMA_XVYCC_222 2 |
| 1918 | # define EVERGREEN_GRPH_ALPHA_SEL_B 3 | 753 | |
| 1919 | 754 | /* GAMUT_REMAP_CONTROL */ | |
| 1920 | #define EVERGREEN_D3VGA_CONTROL 0xf8 | 755 | # define GAMUT_REMAP_BYPASS 0 |
| 1921 | #define EVERGREEN_D4VGA_CONTROL 0xf9 | 756 | # define GAMUT_REMAP_PROG_COEFF 1 |
| 1922 | #define EVERGREEN_D5VGA_CONTROL 0xfa | 757 | # define GAMUT_REMAP_PROG_SHARED_MATRIXA 2 |
| 1923 | #define EVERGREEN_D6VGA_CONTROL 0xfb | 758 | # define GAMUT_REMAP_PROG_SHARED_MATRIXB 3 |
| 1924 | 759 | ||
| 1925 | #define EVERGREEN_GRPH_SURFACE_ADDRESS_MASK 0xffffff00 | 760 | /* REGAMMA_CONTROL */ |
| 1926 | 761 | # define REGAMMA_BYPASS 0 | |
| 1927 | #define EVERGREEN_GRPH_LUT_10BIT_BYPASS_CONTROL 0x1a02 | 762 | # define REGAMMA_SRGB_24 1 |
| 1928 | #define EVERGREEN_LUT_10BIT_BYPASS_EN (1 << 8) | 763 | # define REGAMMA_XVYCC_222 2 |
| 1929 | 764 | # define REGAMMA_PROG_A 3 | |
| 1930 | #define EVERGREEN_GRPH_PITCH 0x1a06 | 765 | # define REGAMMA_PROG_B 4 |
| 1931 | #define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07 | 766 | |
| 1932 | #define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08 | 767 | |
| 1933 | #define EVERGREEN_GRPH_SURFACE_OFFSET_X 0x1a09 | 768 | /* INPUT_GAMMA_CONTROL */ |
| 1934 | #define EVERGREEN_GRPH_SURFACE_OFFSET_Y 0x1a0a | 769 | # define INPUT_GAMMA_USE_LUT 0 |
| 1935 | #define EVERGREEN_GRPH_X_START 0x1a0b | 770 | # define INPUT_GAMMA_BYPASS 1 |
| 1936 | #define EVERGREEN_GRPH_Y_START 0x1a0c | 771 | # define INPUT_GAMMA_SRGB_24 2 |
| 1937 | #define EVERGREEN_GRPH_X_END 0x1a0d | 772 | # define INPUT_GAMMA_XVYCC_222 3 |
| 1938 | #define EVERGREEN_GRPH_Y_END 0x1a0e | ||
| 1939 | #define EVERGREEN_GRPH_UPDATE 0x1a11 | ||
| 1940 | #define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2) | ||
| 1941 | #define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16) | ||
| 1942 | #define EVERGREEN_GRPH_FLIP_CONTROL 0x1a12 | ||
| 1943 | #define EVERGREEN_GRPH_SURFACE_UPDATE_H_RETRACE_EN (1 << 0) | ||
| 1944 | |||
| 1945 | #define EVERGREEN_VIEWPORT_START 0x1b5c | ||
| 1946 | #define EVERGREEN_VIEWPORT_SIZE 0x1b5d | ||
| 1947 | #define EVERGREEN_DESKTOP_HEIGHT 0x1ac1 | ||
| 1948 | |||
| 1949 | /* CUR blocks at 0x6998, 0x7598, 0x10198, 0x10d98, 0x11998, 0x12598 */ | ||
| 1950 | #define EVERGREEN_CUR_CONTROL 0x1a66 | ||
| 1951 | # define EVERGREEN_CURSOR_EN (1 << 0) | ||
| 1952 | # define EVERGREEN_CURSOR_MODE(x) (((x) & 0x3) << 8) | ||
| 1953 | # define EVERGREEN_CURSOR_MONO 0 | ||
| 1954 | # define EVERGREEN_CURSOR_24_1 1 | ||
| 1955 | # define EVERGREEN_CURSOR_24_8_PRE_MULT 2 | ||
| 1956 | # define EVERGREEN_CURSOR_24_8_UNPRE_MULT 3 | ||
| 1957 | # define EVERGREEN_CURSOR_2X_MAGNIFY (1 << 16) | ||
| 1958 | # define EVERGREEN_CURSOR_FORCE_MC_ON (1 << 20) | ||
| 1959 | # define EVERGREEN_CURSOR_URGENT_CONTROL(x) (((x) & 0x7) << 24) | ||
| 1960 | # define EVERGREEN_CURSOR_URGENT_ALWAYS 0 | ||
| 1961 | # define EVERGREEN_CURSOR_URGENT_1_8 1 | ||
| 1962 | # define EVERGREEN_CURSOR_URGENT_1_4 2 | ||
| 1963 | # define EVERGREEN_CURSOR_URGENT_3_8 3 | ||
| 1964 | # define EVERGREEN_CURSOR_URGENT_1_2 4 | ||
| 1965 | #define EVERGREEN_CUR_SURFACE_ADDRESS 0x1a67 | ||
| 1966 | # define EVERGREEN_CUR_SURFACE_ADDRESS_MASK 0xfffff000 | ||
| 1967 | #define EVERGREEN_CUR_SIZE 0x1a68 | ||
| 1968 | #define EVERGREEN_CUR_SURFACE_ADDRESS_HIGH 0x1a69 | ||
| 1969 | #define EVERGREEN_CUR_POSITION 0x1a6a | ||
| 1970 | #define EVERGREEN_CUR_HOT_SPOT 0x1a6b | ||
| 1971 | #define EVERGREEN_CUR_COLOR1 0x1a6c | ||
| 1972 | #define EVERGREEN_CUR_COLOR2 0x1a6d | ||
| 1973 | #define EVERGREEN_CUR_UPDATE 0x1a6e | ||
| 1974 | # define EVERGREEN_CURSOR_UPDATE_PENDING (1 << 0) | ||
| 1975 | # define EVERGREEN_CURSOR_UPDATE_TAKEN (1 << 1) | ||
| 1976 | # define EVERGREEN_CURSOR_UPDATE_LOCK (1 << 16) | ||
| 1977 | # define EVERGREEN_CURSOR_DISABLE_MULTIPLE_UPDATE (1 << 24) | ||
| 1978 | |||
| 1979 | |||
| 1980 | #define NI_INPUT_CSC_CONTROL 0x1a35 | ||
| 1981 | # define NI_INPUT_CSC_GRPH_MODE(x) (((x) & 0x3) << 0) | ||
| 1982 | # define NI_INPUT_CSC_BYPASS 0 | ||
| 1983 | # define NI_INPUT_CSC_PROG_COEFF 1 | ||
| 1984 | # define NI_INPUT_CSC_PROG_SHARED_MATRIXA 2 | ||
| 1985 | # define NI_INPUT_CSC_OVL_MODE(x) (((x) & 0x3) << 4) | ||
| 1986 | |||
| 1987 | #define NI_OUTPUT_CSC_CONTROL 0x1a3c | ||
| 1988 | # define NI_OUTPUT_CSC_GRPH_MODE(x) (((x) & 0x7) << 0) | ||
| 1989 | # define NI_OUTPUT_CSC_BYPASS 0 | ||
| 1990 | # define NI_OUTPUT_CSC_TV_RGB 1 | ||
| 1991 | # define NI_OUTPUT_CSC_YCBCR_601 2 | ||
| 1992 | # define NI_OUTPUT_CSC_YCBCR_709 3 | ||
| 1993 | # define NI_OUTPUT_CSC_PROG_COEFF 4 | ||
| 1994 | # define NI_OUTPUT_CSC_PROG_SHARED_MATRIXB 5 | ||
| 1995 | # define NI_OUTPUT_CSC_OVL_MODE(x) (((x) & 0x7) << 4) | ||
| 1996 | |||
| 1997 | #define NI_DEGAMMA_CONTROL 0x1a58 | ||
| 1998 | # define NI_GRPH_DEGAMMA_MODE(x) (((x) & 0x3) << 0) | ||
| 1999 | # define NI_DEGAMMA_BYPASS 0 | ||
| 2000 | # define NI_DEGAMMA_SRGB_24 1 | ||
| 2001 | # define NI_DEGAMMA_XVYCC_222 2 | ||
| 2002 | # define NI_OVL_DEGAMMA_MODE(x) (((x) & 0x3) << 4) | ||
| 2003 | # define NI_ICON_DEGAMMA_MODE(x) (((x) & 0x3) << 8) | ||
| 2004 | # define NI_CURSOR_DEGAMMA_MODE(x) (((x) & 0x3) << 12) | ||
| 2005 | |||
| 2006 | #define NI_GAMUT_REMAP_CONTROL 0x1a59 | ||
| 2007 | # define NI_GRPH_GAMUT_REMAP_MODE(x) (((x) & 0x3) << 0) | ||
| 2008 | # define NI_GAMUT_REMAP_BYPASS 0 | ||
| 2009 | # define NI_GAMUT_REMAP_PROG_COEFF 1 | ||
| 2010 | # define NI_GAMUT_REMAP_PROG_SHARED_MATRIXA 2 | ||
| 2011 | # define NI_GAMUT_REMAP_PROG_SHARED_MATRIXB 3 | ||
| 2012 | # define NI_OVL_GAMUT_REMAP_MODE(x) (((x) & 0x3) << 4) | ||
| 2013 | |||
| 2014 | #define NI_REGAMMA_CONTROL 0x1aa0 | ||
| 2015 | # define NI_GRPH_REGAMMA_MODE(x) (((x) & 0x7) << 0) | ||
| 2016 | # define NI_REGAMMA_BYPASS 0 | ||
| 2017 | # define NI_REGAMMA_SRGB_24 1 | ||
| 2018 | # define NI_REGAMMA_XVYCC_222 2 | ||
| 2019 | # define NI_REGAMMA_PROG_A 3 | ||
| 2020 | # define NI_REGAMMA_PROG_B 4 | ||
| 2021 | # define NI_OVL_REGAMMA_MODE(x) (((x) & 0x7) << 4) | ||
| 2022 | |||
| 2023 | |||
| 2024 | #define NI_PRESCALE_GRPH_CONTROL 0x1a2d | ||
| 2025 | # define NI_GRPH_PRESCALE_BYPASS (1 << 4) | ||
| 2026 | |||
| 2027 | #define NI_PRESCALE_OVL_CONTROL 0x1a31 | ||
| 2028 | # define NI_OVL_PRESCALE_BYPASS (1 << 4) | ||
| 2029 | |||
| 2030 | #define NI_INPUT_GAMMA_CONTROL 0x1a10 | ||
| 2031 | # define NI_GRPH_INPUT_GAMMA_MODE(x) (((x) & 0x3) << 0) | ||
| 2032 | # define NI_INPUT_GAMMA_USE_LUT 0 | ||
| 2033 | # define NI_INPUT_GAMMA_BYPASS 1 | ||
| 2034 | # define NI_INPUT_GAMMA_SRGB_24 2 | ||
| 2035 | # define NI_INPUT_GAMMA_XVYCC_222 3 | ||
| 2036 | # define NI_OVL_INPUT_GAMMA_MODE(x) (((x) & 0x3) << 4) | ||
| 2037 | |||
| 2038 | #define BLACKOUT_MODE_MASK 0x00000007 | ||
| 2039 | #define VGA_RENDER_CONTROL 0xC0 | ||
| 2040 | #define R_000300_VGA_RENDER_CONTROL 0xC0 | ||
| 2041 | #define C_000300_VGA_VSTATUS_CNTL 0xFFFCFFFF | ||
| 2042 | #define EVERGREEN_CRTC_STATUS 0x1BA3 | ||
| 2043 | #define EVERGREEN_CRTC_V_BLANK (1 << 0) | ||
| 2044 | #define EVERGREEN_CRTC_STATUS_POSITION 0x1BA4 | ||
| 2045 | /* CRTC blocks at 0x6df0, 0x79f0, 0x105f0, 0x111f0, 0x11df0, 0x129f0 */ | ||
| 2046 | #define EVERGREEN_CRTC_V_BLANK_START_END 0x1b8d | ||
| 2047 | #define EVERGREEN_CRTC_CONTROL 0x1b9c | ||
| 2048 | # define EVERGREEN_CRTC_MASTER_EN (1 << 0) | ||
| 2049 | # define EVERGREEN_CRTC_DISP_READ_REQUEST_DISABLE (1 << 24) | ||
| 2050 | #define EVERGREEN_CRTC_BLANK_CONTROL 0x1b9d | ||
| 2051 | # define EVERGREEN_CRTC_BLANK_DATA_EN (1 << 8) | ||
| 2052 | # define EVERGREEN_CRTC_V_BLANK (1 << 0) | ||
| 2053 | #define EVERGREEN_CRTC_STATUS_HV_COUNT 0x1ba8 | ||
| 2054 | #define EVERGREEN_CRTC_UPDATE_LOCK 0x1bb5 | ||
| 2055 | #define EVERGREEN_MASTER_UPDATE_LOCK 0x1bbd | ||
| 2056 | #define EVERGREEN_MASTER_UPDATE_MODE 0x1bbe | ||
| 2057 | #define EVERGREEN_GRPH_UPDATE_LOCK (1 << 16) | ||
| 2058 | #define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS_HIGH 0x1a07 | ||
| 2059 | #define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS_HIGH 0x1a08 | ||
| 2060 | #define EVERGREEN_GRPH_PRIMARY_SURFACE_ADDRESS 0x1a04 | ||
| 2061 | #define EVERGREEN_GRPH_SECONDARY_SURFACE_ADDRESS 0x1a05 | ||
| 2062 | #define EVERGREEN_GRPH_UPDATE 0x1a11 | ||
| 2063 | #define EVERGREEN_VGA_MEMORY_BASE_ADDRESS 0xc4 | ||
| 2064 | #define EVERGREEN_VGA_MEMORY_BASE_ADDRESS_HIGH 0xc9 | ||
| 2065 | #define EVERGREEN_GRPH_SURFACE_UPDATE_PENDING (1 << 2) | ||
| 2066 | |||
| 2067 | #define mmVM_CONTEXT1_CNTL__xxRANGE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x10 | ||
| 2068 | #define mmVM_CONTEXT1_CNTL__xxRANGE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x4 | ||
| 2069 | #define mmVM_CONTEXT1_CNTL__xxDUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x80 | ||
| 2070 | #define mmVM_CONTEXT1_CNTL__xxDUMMY_PAGE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x7 | ||
| 2071 | #define mmVM_CONTEXT1_CNTL__xxPDE0_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x400 | ||
| 2072 | #define mmVM_CONTEXT1_CNTL__xxPDE0_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0xa | ||
| 2073 | #define mmVM_CONTEXT1_CNTL__xxVALID_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x2000 | ||
| 2074 | #define mmVM_CONTEXT1_CNTL__xxVALID_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0xd | ||
| 2075 | #define mmVM_CONTEXT1_CNTL__xxREAD_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x10000 | ||
| 2076 | #define mmVM_CONTEXT1_CNTL__xxREAD_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x10 | ||
| 2077 | #define mmVM_CONTEXT1_CNTL__xxWRITE_PROTECTION_FAULT_ENABLE_DEFAULT_MASK 0x80000 | ||
| 2078 | #define mmVM_CONTEXT1_CNTL__xxWRITE_PROTECTION_FAULT_ENABLE_DEFAULT__SHIFT 0x13 | ||
| 2079 | |||
| 2080 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxVMID_MASK 0x1e000000 | ||
| 2081 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxVMID__SHIFT 0x19 | ||
| 2082 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxPROTECTIONS_MASK 0xff | ||
| 2083 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxPROTECTIONS__SHIFT 0x0 | ||
| 2084 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_ID_MASK 0xff000 | ||
| 2085 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_ID__SHIFT 0xc | ||
| 2086 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_RW_MASK 0x1000000 | ||
| 2087 | #define mmVM_CONTEXT1_PROTECTION_FAULT_STATUS__xxMEMORY_CLIENT_RW__SHIFT 0x18 | ||
| 2088 | |||
| 2089 | #define mmMC_SHARED_BLACKOUT_CNTL__xxBLACKOUT_MODE_MASK 0x7 | ||
| 2090 | #define mmMC_SHARED_BLACKOUT_CNTL__xxBLACKOUT_MODE__SHIFT 0x0 | ||
| 2091 | |||
| 2092 | #define mmBIF_FB_EN__xxFB_READ_EN_MASK 0x1 | ||
| 2093 | #define mmBIF_FB_EN__xxFB_READ_EN__SHIFT 0x0 | ||
| 2094 | #define mmBIF_FB_EN__xxFB_WRITE_EN_MASK 0x2 | ||
| 2095 | #define mmBIF_FB_EN__xxFB_WRITE_EN__SHIFT 0x1 | ||
| 2096 | |||
| 2097 | #define mmSRBM_SOFT_RESET__xxSOFT_RESET_VMC_MASK 0x20000 | ||
| 2098 | #define mmSRBM_SOFT_RESET__xxSOFT_RESET_VMC__SHIFT 0x11 | ||
| 2099 | #define mmSRBM_SOFT_RESET__xxSOFT_RESET_MC_MASK 0x800 | ||
| 2100 | #define mmSRBM_SOFT_RESET__xxSOFT_RESET_MC__SHIFT 0xb | ||
| 2101 | 773 | ||
| 2102 | #define MC_SEQ_MISC0__MT__MASK 0xf0000000 | 774 | #define MC_SEQ_MISC0__MT__MASK 0xf0000000 |
| 2103 | #define MC_SEQ_MISC0__MT__GDDR1 0x10000000 | 775 | #define MC_SEQ_MISC0__MT__GDDR1 0x10000000 |
| @@ -2113,20 +785,9 @@ · 9 unmodified lines | |||
| 2113 | #define PACKET3_SEM_SEL_SIGNAL (0x6 << 29) | 785 | #define PACKET3_SEM_SEL_SIGNAL (0x6 << 29) |
| 2114 | #define PACKET3_SEM_SEL_WAIT (0x7 << 29) | 786 | #define PACKET3_SEM_SEL_WAIT (0x7 << 29) |
| 2115 | 787 | ||
| 2116 | #define CONFIG_CNTL 0x1509 | ||
| 2117 | #define CC_DRM_ID_STRAPS 0X1559 | ||
| 2118 | #define AMDGPU_PCIE_INDEX 0xc | 788 | #define AMDGPU_PCIE_INDEX 0xc |
| 2119 | #define AMDGPU_PCIE_DATA 0xd | 789 | #define AMDGPU_PCIE_DATA 0xd |
| 2120 | 790 | ||
| 2121 | #define DMA_SEM_INCOMPLETE_TIMER_CNTL 0x3411 | ||
| 2122 | #define DMA_SEM_WAIT_FAIL_TIMER_CNTL 0x3412 | ||
| 2123 | #define DMA_MODE 0x342f | ||
| 2124 | #define DMA_RB_RPTR_ADDR_HI 0x3407 | ||
| 2125 | #define DMA_RB_RPTR_ADDR_LO 0x3408 | ||
| 2126 | #define DMA_BUSY_MASK 0x20 | ||
| 2127 | #define DMA1_BUSY_MASK 0X40 | ||
| 2128 | #define SDMA_MAX_INSTANCE 2 | ||
| 2129 | |||
| 2130 | #define PCIE_BUS_CLK 10000 | 791 | #define PCIE_BUS_CLK 10000 |
| 2131 | #define TCLK (PCIE_BUS_CLK / 10) | 792 | #define TCLK (PCIE_BUS_CLK / 10) |
| 2132 | #define PCIE_PORT_INDEX 0xe | 793 | #define PCIE_PORT_INDEX 0xe |
| @@ -2136,8 +797,6 @@ · 3 unmodified lines | |||
| 2136 | #define EVERGREEN_PIF_PHY1_INDEX 0x10 | 797 | #define EVERGREEN_PIF_PHY1_INDEX 0x10 |
| 2137 | #define EVERGREEN_PIF_PHY1_DATA 0x14 | 798 | #define EVERGREEN_PIF_PHY1_DATA 0x14 |
| 2138 | 799 | ||
| 2139 | #define MC_VM_FB_OFFSET 0x81a | ||
| 2140 | |||
| 2141 | /* Discrete VCE clocks */ | 800 | /* Discrete VCE clocks */ |
| 2142 | #define CG_VCEPLL_FUNC_CNTL 0xc0030600 | 801 | #define CG_VCEPLL_FUNC_CNTL 0xc0030600 |
| 2143 | #define VCEPLL_RESET_MASK 0x00000001 | 802 | #define VCEPLL_RESET_MASK 0x00000001 |
| @@ -584,6 +584,8 @@ soc15_asic_reset_method(struct amdgpu_device *adev) · 583 unmodified lines | |||
| 584 | * Enable triggering of GPU reset only if specified | 584 | * Enable triggering of GPU reset only if specified |
| 585 | * by module parameter. | 585 | * by module parameter. |
| 586 | */ | 586 | */ |
| 587 | if (adev->pcie_reset_ctx.in_link_reset) | ||
| 588 | return AMD_RESET_METHOD_LINK; | ||
| 587 | if (amdgpu_gpu_recovery == 4 || amdgpu_gpu_recovery == 5) | 589 | if (amdgpu_gpu_recovery == 4 || amdgpu_gpu_recovery == 5) |
| 588 | return AMD_RESET_METHOD_MODE2; | 590 | return AMD_RESET_METHOD_MODE2; |
| 589 | else if (!(adev->flags & AMD_IS_APU)) | 591 | else if (!(adev->flags & AMD_IS_APU)) |
| @@ -640,6 +642,9 @@ asic_reset: · 50 unmodified lines | |||
| 640 | case AMD_RESET_METHOD_MODE2: | 642 | case AMD_RESET_METHOD_MODE2: |
| 641 | dev_info(adev->dev, "MODE2 reset\n"); | 643 | dev_info(adev->dev, "MODE2 reset\n"); |
| 642 | return amdgpu_dpm_mode2_reset(adev); | 644 | return amdgpu_dpm_mode2_reset(adev); |
| 645 | case AMD_RESET_METHOD_LINK: | ||
| 646 | dev_info(adev->dev, "Link reset\n"); | ||
| 647 | return amdgpu_device_link_reset(adev); | ||
| 643 | default: | 648 | default: |
| 644 | dev_info(adev->dev, "MODE1 reset\n"); | 649 | dev_info(adev->dev, "MODE1 reset\n"); |
| 645 | return amdgpu_device_mode1_reset(adev); | 650 | return amdgpu_device_mode1_reset(adev); |
| @@ -552,6 +552,11 @@ · 551 unmodified lines | |||
| 552 | # define PACKET3_QUERY_STATUS_DOORBELL_OFFSET(x) ((x) << 2) | 552 | # define PACKET3_QUERY_STATUS_DOORBELL_OFFSET(x) ((x) << 2) |
| 553 | # define PACKET3_QUERY_STATUS_ENG_SEL(x) ((x) << 25) | 553 | # define PACKET3_QUERY_STATUS_ENG_SEL(x) ((x) << 25) |
| 554 | 554 | ||
| 555 | #define PACKET3_RUN_CLEANER_SHADER_9_0 0xD7 | ||
| 556 | /* 1. header | ||
| 557 | * 2. RESERVED [31:0] | ||
| 558 | */ | ||
| 559 | |||
| 555 | #define PACKET3_RUN_CLEANER_SHADER 0xD2 | 560 | #define PACKET3_RUN_CLEANER_SHADER 0xD2 |
| 556 | /* 1. header | 561 | /* 1. header |
| 557 | * 2. RESERVED [31:0] | 562 | * 2. RESERVED [31:0] |
| @@ -220,6 +220,13 @@ static int umc_v12_0_convert_error_address(struct amdgpu_device *adev, · 219 unmodified lines | |||
| 220 | nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev); | 220 | nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev); |
| 221 | 221 | ||
| 222 | /* other nps modes are taken as nps1 */ | 222 | /* other nps modes are taken as nps1 */ |
| 223 | if (nps == AMDGPU_NPS2_PARTITION_MODE) { | ||
| 224 | loop_bits[0] = UMC_V12_0_PA_CH5_BIT; | ||
| 225 | loop_bits[1] = UMC_V12_0_PA_C2_BIT; | ||
| 226 | loop_bits[2] = UMC_V12_0_PA_B1_BIT; | ||
| 227 | loop_bits[3] = UMC_V12_0_PA_R12_BIT; | ||
| 228 | } | ||
| 229 | |||
| 223 | if (nps == AMDGPU_NPS4_PARTITION_MODE) { | 230 | if (nps == AMDGPU_NPS4_PARTITION_MODE) { |
| 224 | loop_bits[0] = UMC_V12_0_PA_CH4_BIT; | 231 | loop_bits[0] = UMC_V12_0_PA_CH4_BIT; |
| 225 | loop_bits[1] = UMC_V12_0_PA_CH5_BIT; | 232 | loop_bits[1] = UMC_V12_0_PA_CH5_BIT; |
| @@ -517,6 +524,9 @@ static int umc_v12_0_update_ecc_status(struct amdgpu_device *adev, · 291 unmodified lines | |||
| 517 | 524 | ||
| 518 | if (adev->gmc.gmc_funcs->query_mem_partition_mode) | 525 | if (adev->gmc.gmc_funcs->query_mem_partition_mode) |
| 519 | nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev); | 526 | nps = adev->gmc.gmc_funcs->query_mem_partition_mode(adev); |
| 527 | |||
| 528 | if (nps == AMDGPU_NPS2_PARTITION_MODE) | ||
| 529 | shift_bit = UMC_V12_0_PA_B1_BIT; | ||
| 520 | if (nps == AMDGPU_NPS4_PARTITION_MODE) | 530 | if (nps == AMDGPU_NPS4_PARTITION_MODE) |
| 521 | shift_bit = UMC_V12_0_PA_B0_BIT; | 531 | shift_bit = UMC_V12_0_PA_B0_BIT; |
| 522 | 532 | ||
| @@ -65,12 +65,14 @@ · 64 unmodified lines | |||
| 65 | /* row bits in SOC physical address */ | 65 | /* row bits in SOC physical address */ |
| 66 | #define UMC_V12_0_PA_R0_BIT 22 | 66 | #define UMC_V12_0_PA_R0_BIT 22 |
| 67 | #define UMC_V12_0_PA_R11_BIT 33 | 67 | #define UMC_V12_0_PA_R11_BIT 33 |
| 68 | #define UMC_V12_0_PA_R12_BIT 34 | ||
| 68 | #define UMC_V12_0_PA_R13_BIT 35 | 69 | #define UMC_V12_0_PA_R13_BIT 35 |
| 69 | /* channel bit in SOC physical address */ | 70 | /* channel bit in SOC physical address */ |
| 70 | #define UMC_V12_0_PA_CH4_BIT 12 | 71 | #define UMC_V12_0_PA_CH4_BIT 12 |
| 71 | #define UMC_V12_0_PA_CH5_BIT 13 | 72 | #define UMC_V12_0_PA_CH5_BIT 13 |
| 72 | /* bank bit in SOC physical address */ | 73 | /* bank bit in SOC physical address */ |
| 73 | #define UMC_V12_0_PA_B0_BIT 19 | 74 | #define UMC_V12_0_PA_B0_BIT 19 |
| 75 | #define UMC_V12_0_PA_B1_BIT 20 | ||
| 74 | /* row bits in MCA address */ | 76 | /* row bits in MCA address */ |
| 75 | #define UMC_V12_0_MA_R0_BIT 10 | 77 | #define UMC_V12_0_MA_R0_BIT 10 |
| 76 | 78 | ||
| @@ -39,6 +39,7 @@ · 38 unmodified lines | |||
| 39 | 39 | ||
| 40 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00 | 40 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00 |
| 41 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48200 | 41 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48200 |
| 42 | #define VCN1_AON_SOC_ADDRESS_3_0 0x48000 | ||
| 42 | 43 | ||
| 43 | #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x1fd | 44 | #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x1fd |
| 44 | #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x503 | 45 | #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x503 |
| @@ -39,6 +39,7 @@ · 38 unmodified lines | |||
| 39 | 39 | ||
| 40 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00 | 40 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00 |
| 41 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48200 | 41 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48200 |
| 42 | #define VCN1_AON_SOC_ADDRESS_3_0 0x48000 | ||
| 42 | 43 | ||
| 43 | #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27 | 44 | #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27 |
| 44 | #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f | 45 | #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f |
| @@ -40,6 +40,7 @@ · 39 unmodified lines | |||
| 40 | 40 | ||
| 41 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00 | 41 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fa00 |
| 42 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48200 | 42 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48200 |
| 43 | #define VCN1_AON_SOC_ADDRESS_3_0 0x48000 | ||
| 43 | 44 | ||
| 44 | #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27 | 45 | #define mmUVD_CONTEXT_ID_INTERNAL_OFFSET 0x27 |
| 45 | #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f | 46 | #define mmUVD_GPCOM_VCPU_CMD_INTERNAL_OFFSET 0x0f |
| @@ -46,6 +46,7 @@ · 45 unmodified lines | |||
| 46 | 46 | ||
| 47 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00 | 47 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00 |
| 48 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48300 | 48 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48300 |
| 49 | #define VCN1_AON_SOC_ADDRESS_3_0 0x48000 | ||
| 49 | 50 | ||
| 50 | #define VCN_HARVEST_MMSCH 0 | 51 | #define VCN_HARVEST_MMSCH 0 |
| 51 | 52 | ||
| @@ -238,9 +239,9 @@ static int vcn_v4_0_sw_init(struct amdgpu_ip_block *ip_block) · 186 unmodified lines | |||
| 238 | adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_pause_dpg_mode; | 239 | adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_pause_dpg_mode; |
| 239 | } | 240 | } |
| 240 | 241 | ||
| 241 | /* TODO: Add queue reset mask when FW fully supports it */ | ||
| 242 | adev->vcn.supported_reset = | 242 | adev->vcn.supported_reset = |
| 243 | amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]); | 243 | amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]); |
| 244 | adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE; | ||
| 244 | 245 | ||
| 245 | if (amdgpu_sriov_vf(adev)) { | 246 | if (amdgpu_sriov_vf(adev)) { |
| 246 | r = amdgpu_virt_alloc_mm_table(adev); | 247 | r = amdgpu_virt_alloc_mm_table(adev); |
| @@ -614,7 +615,8 @@ static void vcn_v4_0_mc_resume_dpg_mode(struct amdgpu_vcn_inst *vinst, · 367 unmodified lines | |||
| 614 | 615 | ||
| 615 | /* VCN global tiling registers */ | 616 | /* VCN global tiling registers */ |
| 616 | WREG32_SOC15_DPG_MODE(inst_idx, SOC15_DPG_MODE_OFFSET( | 617 | WREG32_SOC15_DPG_MODE(inst_idx, SOC15_DPG_MODE_OFFSET( |
| 617 | VCN, 0, regUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect); | 618 | VCN, inst_idx, regUVD_GFX10_ADDR_CONFIG), |
| 619 | adev->gfx.config.gb_addr_config, 0, indirect); | ||
| 618 | } | 620 | } |
| 619 | 621 | ||
| 620 | /** | 622 | /** |
| @@ -1945,6 +1947,20 @@ static int vcn_v4_0_ring_patch_cs_in_place(struct amdgpu_cs_parser *p, · 1324 unmodified lines | |||
| 1945 | return 0; | 1947 | return 0; |
| 1946 | } | 1948 | } |
| 1947 | 1949 | ||
| 1950 | static int vcn_v4_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid) | ||
| 1951 | { | ||
| 1952 | struct amdgpu_device *adev = ring->adev; | ||
| 1953 | struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me]; | ||
| 1954 | |||
| 1955 | if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE)) | ||
| 1956 | return -EOPNOTSUPP; | ||
| 1957 | |||
| 1958 | vcn_v4_0_stop(vinst); | ||
| 1959 | vcn_v4_0_start(vinst); | ||
| 1960 | |||
| 1961 | return amdgpu_ring_test_helper(ring); | ||
| 1962 | } | ||
| 1963 | |||
| 1948 | static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { | 1964 | static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { |
| 1949 | .type = AMDGPU_RING_TYPE_VCN_ENC, | 1965 | .type = AMDGPU_RING_TYPE_VCN_ENC, |
| 1950 | .align_mask = 0x3f, | 1966 | .align_mask = 0x3f, |
| @@ -1974,6 +1990,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { · 23 unmodified lines | |||
| 1974 | .emit_wreg = vcn_v2_0_enc_ring_emit_wreg, | 1990 | .emit_wreg = vcn_v2_0_enc_ring_emit_wreg, |
| 1975 | .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait, | 1991 | .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait, |
| 1976 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, | 1992 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, |
| 1993 | .reset = vcn_v4_0_ring_reset, | ||
| 1977 | }; | 1994 | }; |
| 1978 | 1995 | ||
| 1979 | /** | 1996 | /** |
| @@ -45,6 +45,7 @@ · 44 unmodified lines | |||
| 45 | 45 | ||
| 46 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00 | 46 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00 |
| 47 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48300 | 47 | #define VCN1_VID_SOC_ADDRESS_3_0 0x48300 |
| 48 | #define VCN1_AON_SOC_ADDRESS_3_0 0x48000 | ||
| 48 | 49 | ||
| 49 | static const struct amdgpu_hwip_reg_entry vcn_reg_list_4_0_3[] = { | 50 | static const struct amdgpu_hwip_reg_entry vcn_reg_list_4_0_3[] = { |
| 50 | SOC15_REG_ENTRY_STR(VCN, 0, regUVD_POWER_STATUS), | 51 | SOC15_REG_ENTRY_STR(VCN, 0, regUVD_POWER_STATUS), |
| @@ -287,6 +288,31 @@ static int vcn_v4_0_3_sw_fini(struct amdgpu_ip_block *ip_block) · 236 unmodified lines | |||
| 287 | return 0; | 288 | return 0; |
| 288 | } | 289 | } |
| 289 | 290 | ||
| 291 | static int vcn_v4_0_3_hw_init_inst(struct amdgpu_vcn_inst *vinst) | ||
| 292 | { | ||
| 293 | int vcn_inst; | ||
| 294 | struct amdgpu_device *adev = vinst->adev; | ||
| 295 | struct amdgpu_ring *ring; | ||
| 296 | int inst_idx = vinst->inst; | ||
| 297 | |||
| 298 | vcn_inst = GET_INST(VCN, inst_idx); | ||
| 299 | ring = &adev->vcn.inst[inst_idx].ring_enc[0]; | ||
| 300 | if (ring->use_doorbell) { | ||
| 301 | adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell, | ||
| 302 | (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 9 * vcn_inst, | ||
| 303 | adev->vcn.inst[inst_idx].aid_id); | ||
| 304 | |||
| 305 | WREG32_SOC15(VCN, vcn_inst, regVCN_RB1_DB_CTRL, | ||
| 306 | ring->doorbell_index << VCN_RB1_DB_CTRL__OFFSET__SHIFT | | ||
| 307 | VCN_RB1_DB_CTRL__EN_MASK); | ||
| 308 | |||
| 309 | /* Read DB_CTRL to flush the write DB_CTRL command. */ | ||
| 310 | RREG32_SOC15(VCN, vcn_inst, regVCN_RB1_DB_CTRL); | ||
| 311 | } | ||
| 312 | |||
| 313 | return 0; | ||
| 314 | } | ||
| 315 | |||
| 290 | /** | 316 | /** |
| 291 | * vcn_v4_0_3_hw_init - start and test VCN block | 317 | * vcn_v4_0_3_hw_init - start and test VCN block |
| 292 | * | 318 | * |
| @@ -298,7 +324,8 @@ static int vcn_v4_0_3_hw_init(struct amdgpu_ip_block *ip_block) · 5 unmodified lines | |||
| 298 | { | 324 | { |
| 299 | struct amdgpu_device *adev = ip_block->adev; | 325 | struct amdgpu_device *adev = ip_block->adev; |
| 300 | struct amdgpu_ring *ring; | 326 | struct amdgpu_ring *ring; |
| 301 | int i, r, vcn_inst; | 327 | struct amdgpu_vcn_inst *vinst; |
| 328 | int i, r; | ||
| 302 | 329 | ||
| 303 | if (amdgpu_sriov_vf(adev)) { | 330 | if (amdgpu_sriov_vf(adev)) { |
| 304 | r = vcn_v4_0_3_start_sriov(adev); | 331 | r = vcn_v4_0_3_start_sriov(adev); |
| @@ -321,28 +348,9 @@ static int vcn_v4_0_3_hw_init(struct amdgpu_ip_block *ip_block) · 16 unmodified lines | |||
| 321 | for (i = 0; i < adev->vcn.num_vcn_inst; ++i) { | 348 | for (i = 0; i < adev->vcn.num_vcn_inst; ++i) { |
| 322 | struct amdgpu_vcn4_fw_shared *fw_shared; | 349 | struct amdgpu_vcn4_fw_shared *fw_shared; |
| 323 | 350 | ||
| 324 | vcn_inst = GET_INST(VCN, i); | ||
| 325 | ring = &adev->vcn.inst[i].ring_enc[0]; | 351 | ring = &adev->vcn.inst[i].ring_enc[0]; |
| 326 | 352 | vinst = &adev->vcn.inst[i]; | |
| 327 | if (ring->use_doorbell) { | 353 | vcn_v4_0_3_hw_init_inst(vinst); |
| 328 | adev->nbio.funcs->vcn_doorbell_range( | ||
| 329 | adev, ring->use_doorbell, | ||
| 330 | (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + | ||
| 331 | 9 * vcn_inst, | ||
| 332 | adev->vcn.inst[i].aid_id); | ||
| 333 | |||
| 334 | WREG32_SOC15( | ||
| 335 | VCN, GET_INST(VCN, ring->me), | ||
| 336 | regVCN_RB1_DB_CTRL, | ||
| 337 | ring->doorbell_index | ||
| 338 | << VCN_RB1_DB_CTRL__OFFSET__SHIFT | | ||
| 339 | VCN_RB1_DB_CTRL__EN_MASK); | ||
| 340 | |||
| 341 | /* Read DB_CTRL to flush the write DB_CTRL command. */ | ||
| 342 | RREG32_SOC15( | ||
| 343 | VCN, GET_INST(VCN, ring->me), | ||
| 344 | regVCN_RB1_DB_CTRL); | ||
| 345 | } | ||
| 346 | 354 | ||
| 347 | /* Re-init fw_shared when RAS fatal error occurred */ | 355 | /* Re-init fw_shared when RAS fatal error occurred */ |
| 348 | fw_shared = adev->vcn.inst[i].fw_shared.cpu_addr; | 356 | fw_shared = adev->vcn.inst[i].fw_shared.cpu_addr; |
| @@ -1563,6 +1571,37 @@ static void vcn_v4_0_3_unified_ring_set_wptr(struct amdgpu_ring *ring) · 1214 unmodified lines | |||
| 1563 | } | 1571 | } |
| 1564 | } | 1572 | } |
| 1565 | 1573 | ||
| 1574 | static int vcn_v4_0_3_ring_reset(struct amdgpu_ring *ring, unsigned int vmid) | ||
| 1575 | { | ||
| 1576 | int r = 0; | ||
| 1577 | int vcn_inst; | ||
| 1578 | struct amdgpu_device *adev = ring->adev; | ||
| 1579 | struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me]; | ||
| 1580 | |||
| 1581 | if (amdgpu_sriov_vf(ring->adev)) | ||
| 1582 | return -EOPNOTSUPP; | ||
| 1583 | |||
| 1584 | if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE)) | ||
| 1585 | return -EOPNOTSUPP; | ||
| 1586 | |||
| 1587 | vcn_inst = GET_INST(VCN, ring->me); | ||
| 1588 | r = amdgpu_dpm_reset_vcn(adev, 1 << vcn_inst); | ||
| 1589 | |||
| 1590 | if (r) { | ||
| 1591 | DRM_DEV_ERROR(adev->dev, "VCN reset fail : %d\n", r); | ||
| 1592 | return r; | ||
| 1593 | } | ||
| 1594 | |||
| 1595 | /* This flag is not set for VF, assumed to be disabled always */ | ||
| 1596 | if (RREG32_SOC15(VCN, GET_INST(VCN, 0), regVCN_RRMT_CNTL) & 0x100) | ||
| 1597 | adev->vcn.caps |= AMDGPU_VCN_CAPS(RRMT_ENABLED); | ||
| 1598 | vcn_v4_0_3_hw_init_inst(vinst); | ||
| 1599 | vcn_v4_0_3_start_dpg_mode(vinst, adev->vcn.inst[ring->me].indirect_sram); | ||
| 1600 | r = amdgpu_ring_test_helper(ring); | ||
| 1601 | |||
| 1602 | return r; | ||
| 1603 | } | ||
| 1604 | |||
| 1566 | static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = { | 1605 | static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = { |
| 1567 | .type = AMDGPU_RING_TYPE_VCN_ENC, | 1606 | .type = AMDGPU_RING_TYPE_VCN_ENC, |
| 1568 | .align_mask = 0x3f, | 1607 | .align_mask = 0x3f, |
| @@ -1591,6 +1630,7 @@ static const struct amdgpu_ring_funcs vcn_v4_0_3_unified_ring_vm_funcs = { · 22 unmodified lines | |||
| 1591 | .emit_wreg = vcn_v4_0_3_enc_ring_emit_wreg, | 1630 | .emit_wreg = vcn_v4_0_3_enc_ring_emit_wreg, |
| 1592 | .emit_reg_wait = vcn_v4_0_3_enc_ring_emit_reg_wait, | 1631 | .emit_reg_wait = vcn_v4_0_3_enc_ring_emit_reg_wait, |
| 1593 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, | 1632 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, |
| 1633 | .reset = vcn_v4_0_3_ring_reset, | ||
| 1594 | }; | 1634 | }; |
| 1595 | 1635 | ||
| 1596 | /** | 1636 | /** |
| @@ -46,6 +46,7 @@ · 45 unmodified lines | |||
| 46 | 46 | ||
| 47 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00 | 47 | #define VCN_VID_SOC_ADDRESS_2_0 0x1fb00 |
| 48 | #define VCN1_VID_SOC_ADDRESS_3_0 (0x48300 + 0x38000) | 48 | #define VCN1_VID_SOC_ADDRESS_3_0 (0x48300 + 0x38000) |
| 49 | #define VCN1_AON_SOC_ADDRESS_3_0 (0x48000 + 0x38000) | ||
| 49 | 50 | ||
| 50 | #define VCN_HARVEST_MMSCH 0 | 51 | #define VCN_HARVEST_MMSCH 0 |
| 51 | 52 | ||
| @@ -207,6 +208,10 @@ static int vcn_v4_0_5_sw_init(struct amdgpu_ip_block *ip_block) · 155 unmodified lines | |||
| 207 | if (amdgpu_sriov_vf(adev)) | 208 | if (amdgpu_sriov_vf(adev)) |
| 208 | fw_shared->present_flag_0 |= cpu_to_le32(AMDGPU_VCN_VF_RB_SETUP_FLAG); | 209 | fw_shared->present_flag_0 |= cpu_to_le32(AMDGPU_VCN_VF_RB_SETUP_FLAG); |
| 209 | 210 | ||
| 211 | fw_shared->present_flag_0 |= AMDGPU_FW_SHARED_FLAG_0_DRM_KEY_INJECT; | ||
| 212 | fw_shared->drm_key_wa.method = | ||
| 213 | AMDGPU_DRM_KEY_INJECT_WORKAROUND_VCNFW_ASD_HANDSHAKING; | ||
| 214 | |||
| 210 | if (amdgpu_vcnfw_log) | 215 | if (amdgpu_vcnfw_log) |
| 211 | amdgpu_vcn_fwlog_init(&adev->vcn.inst[i]); | 216 | amdgpu_vcn_fwlog_init(&adev->vcn.inst[i]); |
| 212 | 217 | ||
| @@ -214,6 +219,13 @@ static int vcn_v4_0_5_sw_init(struct amdgpu_ip_block *ip_block) · 1 unmodified line | |||
| 214 | adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_5_pause_dpg_mode; | 219 | adev->vcn.inst[i].pause_dpg_mode = vcn_v4_0_5_pause_dpg_mode; |
| 215 | } | 220 | } |
| 216 | 221 | ||
| 222 | adev->vcn.supported_reset = amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]); | ||
| 223 | adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE; | ||
| 224 | |||
| 225 | r = amdgpu_vcn_sysfs_reset_mask_init(adev); | ||
| 226 | if (r) | ||
| 227 | return r; | ||
| 228 | |||
| 217 | if (amdgpu_sriov_vf(adev)) { | 229 | if (amdgpu_sriov_vf(adev)) { |
| 218 | r = amdgpu_virt_alloc_mm_table(adev); | 230 | r = amdgpu_virt_alloc_mm_table(adev); |
| 219 | if (r) | 231 | if (r) |
| @@ -1435,6 +1447,20 @@ static void vcn_v4_0_5_unified_ring_set_wptr(struct amdgpu_ring *ring) · 1215 unmodified lines | |||
| 1435 | } | 1447 | } |
| 1436 | } | 1448 | } |
| 1437 | 1449 | ||
| 1450 | static int vcn_v4_0_5_ring_reset(struct amdgpu_ring *ring, unsigned int vmid) | ||
| 1451 | { | ||
| 1452 | struct amdgpu_device *adev = ring->adev; | ||
| 1453 | struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me]; | ||
| 1454 | |||
| 1455 | if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE)) | ||
| 1456 | return -EOPNOTSUPP; | ||
| 1457 | |||
| 1458 | vcn_v4_0_5_stop(vinst); | ||
| 1459 | vcn_v4_0_5_start(vinst); | ||
| 1460 | |||
| 1461 | return amdgpu_ring_test_helper(ring); | ||
| 1462 | } | ||
| 1463 | |||
| 1438 | static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = { | 1464 | static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = { |
| 1439 | .type = AMDGPU_RING_TYPE_VCN_ENC, | 1465 | .type = AMDGPU_RING_TYPE_VCN_ENC, |
| 1440 | .align_mask = 0x3f, | 1466 | .align_mask = 0x3f, |
| @@ -1462,6 +1488,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_5_unified_ring_vm_funcs = { · 21 unmodified lines | |||
| 1462 | .emit_wreg = vcn_v2_0_enc_ring_emit_wreg, | 1488 | .emit_wreg = vcn_v2_0_enc_ring_emit_wreg, |
| 1463 | .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait, | 1489 | .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait, |
| 1464 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, | 1490 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, |
| 1491 | .reset = vcn_v4_0_5_ring_reset, | ||
| 1465 | }; | 1492 | }; |
| 1466 | 1493 | ||
| 1467 | /** | 1494 | /** |
| @@ -196,9 +196,9 @@ static int vcn_v5_0_0_sw_init(struct amdgpu_ip_block *ip_block) · 195 unmodified lines | |||
| 196 | adev->vcn.inst[i].pause_dpg_mode = vcn_v5_0_0_pause_dpg_mode; | 196 | adev->vcn.inst[i].pause_dpg_mode = vcn_v5_0_0_pause_dpg_mode; |
| 197 | } | 197 | } |
| 198 | 198 | ||
| 199 | /* TODO: Add queue reset mask when FW fully supports it */ | ||
| 200 | adev->vcn.supported_reset = | 199 | adev->vcn.supported_reset = |
| 201 | amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]); | 200 | amdgpu_get_soft_full_reset_mask(&adev->vcn.inst[0].ring_enc[0]); |
| 201 | adev->vcn.supported_reset |= AMDGPU_RESET_TYPE_PER_QUEUE; | ||
| 202 | 202 | ||
| 203 | vcn_v5_0_0_alloc_ip_dump(adev); | 203 | vcn_v5_0_0_alloc_ip_dump(adev); |
| 204 | 204 | ||
| @@ -533,7 +533,8 @@ static void vcn_v5_0_0_mc_resume_dpg_mode(struct amdgpu_vcn_inst *vinst, · 328 unmodified lines | |||
| 533 | 533 | ||
| 534 | /* VCN global tiling registers */ | 534 | /* VCN global tiling registers */ |
| 535 | WREG32_SOC24_DPG_MODE(inst_idx, SOC24_DPG_MODE_OFFSET( | 535 | WREG32_SOC24_DPG_MODE(inst_idx, SOC24_DPG_MODE_OFFSET( |
| 536 | VCN, 0, regUVD_GFX10_ADDR_CONFIG), adev->gfx.config.gb_addr_config, 0, indirect); | 536 | VCN, inst_idx, regUVD_GFX10_ADDR_CONFIG), |
| 537 | adev->gfx.config.gb_addr_config, 0, indirect); | ||
| 537 | 538 | ||
| 538 | return; | 539 | return; |
| 539 | } | 540 | } |
| @@ -1171,6 +1172,20 @@ static void vcn_v5_0_0_unified_ring_set_wptr(struct amdgpu_ring *ring) · 631 unmodified lines | |||
| 1171 | } | 1172 | } |
| 1172 | } | 1173 | } |
| 1173 | 1174 | ||
| 1175 | static int vcn_v5_0_0_ring_reset(struct amdgpu_ring *ring, unsigned int vmid) | ||
| 1176 | { | ||
| 1177 | struct amdgpu_device *adev = ring->adev; | ||
| 1178 | struct amdgpu_vcn_inst *vinst = &adev->vcn.inst[ring->me]; | ||
| 1179 | |||
| 1180 | if (!(adev->vcn.supported_reset & AMDGPU_RESET_TYPE_PER_QUEUE)) | ||
| 1181 | return -EOPNOTSUPP; | ||
| 1182 | |||
| 1183 | vcn_v5_0_0_stop(vinst); | ||
| 1184 | vcn_v5_0_0_start(vinst); | ||
| 1185 | |||
| 1186 | return amdgpu_ring_test_helper(ring); | ||
| 1187 | } | ||
| 1188 | |||
| 1174 | static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = { | 1189 | static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = { |
| 1175 | .type = AMDGPU_RING_TYPE_VCN_ENC, | 1190 | .type = AMDGPU_RING_TYPE_VCN_ENC, |
| 1176 | .align_mask = 0x3f, | 1191 | .align_mask = 0x3f, |
| @@ -1198,6 +1213,7 @@ static const struct amdgpu_ring_funcs vcn_v5_0_0_unified_ring_vm_funcs = { · 21 unmodified lines | |||
| 1198 | .emit_wreg = vcn_v2_0_enc_ring_emit_wreg, | 1213 | .emit_wreg = vcn_v2_0_enc_ring_emit_wreg, |
| 1199 | .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait, | 1214 | .emit_reg_wait = vcn_v2_0_enc_ring_emit_reg_wait, |
| 1200 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, | 1215 | .emit_reg_write_reg_wait = amdgpu_ring_emit_reg_write_reg_wait_helper, |
| 1216 | .reset = vcn_v5_0_0_ring_reset, | ||
| 1201 | }; | 1217 | }; |
| 1202 | 1218 | ||
| 1203 | /** | 1219 | /** |
| @@ -126,7 +126,7 @@ static int vcn_v5_0_1_sw_init(struct amdgpu_ip_block *ip_block) · 125 unmodified lines | |||
| 126 | 126 | ||
| 127 | ring = &adev->vcn.inst[i].ring_enc[0]; | 127 | ring = &adev->vcn.inst[i].ring_enc[0]; |
| 128 | ring->use_doorbell = true; | 128 | ring->use_doorbell = true; |
| 129 | ring->doorbell_index = (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 9 * vcn_inst; | 129 | ring->doorbell_index = (adev->doorbell_index.vcn.vcn_ring0_1 << 1) + 11 * vcn_inst; |
| 130 | 130 | ||
| 131 | ring->vm_hub = AMDGPU_MMHUB0(adev->vcn.inst[i].aid_id); | 131 | ring->vm_hub = AMDGPU_MMHUB0(adev->vcn.inst[i].aid_id); |
| 132 | sprintf(ring->name, "vcn_unified_%d", adev->vcn.inst[i].aid_id); | 132 | sprintf(ring->name, "vcn_unified_%d", adev->vcn.inst[i].aid_id); |
| @@ -213,7 +213,7 @@ static int vcn_v5_0_1_hw_init(struct amdgpu_ip_block *ip_block) · 80 unmodified lines | |||
| 213 | if (ring->use_doorbell) | 213 | if (ring->use_doorbell) |
| 214 | adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell, | 214 | adev->nbio.funcs->vcn_doorbell_range(adev, ring->use_doorbell, |
| 215 | ((adev->doorbell_index.vcn.vcn_ring0_1 << 1) + | 215 | ((adev->doorbell_index.vcn.vcn_ring0_1 << 1) + |
| 216 | 9 * vcn_inst), | 216 | 11 * vcn_inst), |
| 217 | adev->vcn.inst[i].aid_id); | 217 | adev->vcn.inst[i].aid_id); |
| 218 | 218 | ||
| 219 | /* Re-init fw_shared, if required */ | 219 | /* Re-init fw_shared, if required */ |
| @@ -350,6 +350,7 @@ static int vega20_ih_irq_init(struct amdgpu_device *adev) · 349 unmodified lines | |||
| 350 | if (ret) | 350 | if (ret) |
| 351 | return ret; | 351 | return ret; |
| 352 | } | 352 | } |
| 353 | ih[i]->overflow = false; | ||
| 353 | } | 354 | } |
| 354 | 355 | ||
| 355 | if (!amdgpu_sriov_vf(adev)) | 356 | if (!amdgpu_sriov_vf(adev)) |
| @@ -437,7 +438,10 @@ static u32 vega20_ih_get_wptr(struct amdgpu_device *adev, · 81 unmodified lines | |||
| 437 | if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW)) | 438 | if (!REG_GET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW)) |
| 438 | goto out; | 439 | goto out; |
| 439 | 440 | ||
| 440 | wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0); | 441 | if (!amdgpu_sriov_vf(adev)) |
| 442 | wptr = REG_SET_FIELD(wptr, IH_RB_WPTR, RB_OVERFLOW, 0); | ||
| 443 | else | ||
| 444 | ih->overflow = true; | ||
| 441 | 445 | ||
| 442 | /* When a ring buffer overflow happen start parsing interrupt | 446 | /* When a ring buffer overflow happen start parsing interrupt |
| 443 | * from the last not overwritten vector (wptr + 32). Hopefully | 447 | * from the last not overwritten vector (wptr + 32). Hopefully |
| @@ -27,6 +27,16 @@ · 26 unmodified lines | |||
| 27 | #include "kfd_priv.h" | 27 | #include "kfd_priv.h" |
| 28 | 28 | ||
| 29 | static struct dentry *debugfs_root; | 29 | static struct dentry *debugfs_root; |
| 30 | static struct dentry *debugfs_proc; | ||
| 31 | static struct list_head procs; | ||
| 32 | |||
| 33 | struct debugfs_proc_entry { | ||
| 34 | struct list_head list; | ||
| 35 | struct dentry *proc_dentry; | ||
| 36 | pid_t pid; | ||
| 37 | }; | ||
| 38 | |||
| 39 | #define MAX_DEBUGFS_FILENAME_LEN 32 | ||
| 30 | 40 | ||
| 31 | static int kfd_debugfs_open(struct inode *inode, struct file *file) | 41 | static int kfd_debugfs_open(struct inode *inode, struct file *file) |
| 32 | { | 42 | { |
| @@ -92,6 +102,8 @@ static const struct file_operations kfd_debugfs_hang_hws_fops = { · 59 unmodified lines | |||
| 92 | void kfd_debugfs_init(void) | 102 | void kfd_debugfs_init(void) |
| 93 | { | 103 | { |
| 94 | debugfs_root = debugfs_create_dir("kfd", NULL); | 104 | debugfs_root = debugfs_create_dir("kfd", NULL); |
| 105 | debugfs_proc = debugfs_create_dir("proc", debugfs_root); | ||
| 106 | INIT_LIST_HEAD(&procs); | ||
| 95 | 107 | ||
| 96 | debugfs_create_file("mqds", S_IFREG | 0444, debugfs_root, | 108 | debugfs_create_file("mqds", S_IFREG | 0444, debugfs_root, |
| 97 | kfd_debugfs_mqds_by_process, &kfd_debugfs_fops); | 109 | kfd_debugfs_mqds_by_process, &kfd_debugfs_fops); |
| @@ -107,5 +119,69 @@ void kfd_debugfs_init(void) · 9 unmodified lines | |||
| 107 | 119 | ||
| 108 | void kfd_debugfs_fini(void) | 120 | void kfd_debugfs_fini(void) |
| 109 | { | 121 | { |
| 122 | debugfs_remove_recursive(debugfs_proc); | ||
| 110 | debugfs_remove_recursive(debugfs_root); | 123 | debugfs_remove_recursive(debugfs_root); |
| 111 | } | 124 | } |
| 125 | |||
| 126 | static ssize_t kfd_debugfs_pasid_read(struct file *file, char __user *buf, | ||
| 127 | size_t count, loff_t *ppos) | ||
| 128 | { | ||
| 129 | struct kfd_process_device *pdd = file_inode(file)->i_private; | ||
| 130 | char tmp[32]; | ||
| 131 | int len; | ||
| 132 | |||
| 133 | len = snprintf(tmp, sizeof(tmp), "%u\n", pdd->pasid); | ||
| 134 | |||
| 135 | return simple_read_from_buffer(buf, count, ppos, tmp, len); | ||
| 136 | } | ||
| 137 | |||
| 138 | static const struct file_operations kfd_debugfs_pasid_fops = { | ||
| 139 | .owner = THIS_MODULE, | ||
| 140 | .read = kfd_debugfs_pasid_read, | ||
| 141 | }; | ||
| 142 | |||
| 143 | void kfd_debugfs_add_process(struct kfd_process *p) | ||
| 144 | { | ||
| 145 | int i; | ||
| 146 | char name[MAX_DEBUGFS_FILENAME_LEN]; | ||
| 147 | struct debugfs_proc_entry *entry; | ||
| 148 | |||
| 149 | entry = kzalloc(sizeof(*entry), GFP_KERNEL); | ||
| 150 | if (!entry) | ||
| 151 | return; | ||
| 152 | |||
| 153 | list_add(&entry->list, &procs); | ||
| 154 | entry->pid = p->lead_thread->pid; | ||
| 155 | snprintf(name, MAX_DEBUGFS_FILENAME_LEN, "%d", | ||
| 156 | (int)entry->pid); | ||
| 157 | entry->proc_dentry = debugfs_create_dir(name, debugfs_proc); | ||
| 158 | |||
| 159 | /* Create debugfs files for each GPU: | ||
| 160 | * - proc/<pid>/pasid_<gpuid> | ||
| 161 | */ | ||
| 162 | for (i = 0; i < p->n_pdds; i++) { | ||
| 163 | struct kfd_process_device *pdd = p->pdds[i]; | ||
| 164 | |||
| 165 | snprintf(name, MAX_DEBUGFS_FILENAME_LEN, "pasid_%u", | ||
| 166 | pdd->dev->id); | ||
| 167 | debugfs_create_file((const char *)name, S_IFREG | 0444, | ||
| 168 | entry->proc_dentry, pdd, | ||
| 169 | &kfd_debugfs_pasid_fops); | ||
| 170 | } | ||
| 171 | } | ||
| 172 | |||
| 173 | void kfd_debugfs_remove_process(struct kfd_process *p) | ||
| 174 | { | ||
| 175 | struct debugfs_proc_entry *entry, *next; | ||
| 176 | |||
| 177 | mutex_lock(&kfd_processes_mutex); | ||
| 178 | list_for_each_entry_safe(entry, next, &procs, list) { | ||
| 179 | if (entry->pid != p->lead_thread->pid) | ||
| 180 | continue; | ||
| 181 | |||
| 182 | debugfs_remove_recursive(entry->proc_dentry); | ||
| 183 | list_del(&entry->list); | ||
| 184 | kfree(entry); | ||
| 185 | } | ||
| 186 | mutex_unlock(&kfd_processes_mutex); | ||
| 187 | } | ||
| @@ -352,11 +352,6 @@ struct kfd_dev *kgd2kfd_probe(struct amdgpu_device *adev, bool vf) · 351 unmodified lines | |||
| 352 | f2g = &aldebaran_kfd2kgd; | 352 | f2g = &aldebaran_kfd2kgd; |
| 353 | break; | 353 | break; |
| 354 | case IP_VERSION(9, 4, 3): | 354 | case IP_VERSION(9, 4, 3): |
| 355 | gfx_target_version = adev->rev_id >= 1 ? 90402 | ||
| 356 | : adev->flags & AMD_IS_APU ? 90400 | ||
| 357 | : 90401; | ||
| 358 | f2g = &gc_9_4_3_kfd2kgd; | ||
| 359 | break; | ||
| 360 | case IP_VERSION(9, 4, 4): | 355 | case IP_VERSION(9, 4, 4): |
| 361 | gfx_target_version = 90402; | 356 | gfx_target_version = 90402; |
| 362 | f2g = &gc_9_4_3_kfd2kgd; | 357 | f2g = &gc_9_4_3_kfd2kgd; |
| @@ -1576,8 +1576,9 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 1575 unmodified lines | |||
| 1576 | int bit; | 1576 | int bit; |
| 1577 | 1577 | ||
| 1578 | if (q->properties.type == KFD_QUEUE_TYPE_SDMA) { | 1578 | if (q->properties.type == KFD_QUEUE_TYPE_SDMA) { |
| 1579 | if (bitmap_empty(dqm->sdma_bitmap, KFD_MAX_SDMA_QUEUES)) { | 1579 | if (bitmap_empty(dqm->sdma_bitmap, get_num_sdma_queues(dqm))) { |
| 1580 | dev_err(dev, "No more SDMA queue to allocate\n"); | 1580 | dev_warn(dev, "No more SDMA queue to allocate (%d total queues)\n", |
| 1581 | get_num_sdma_queues(dqm)); | ||
| 1581 | return -ENOMEM; | 1582 | return -ENOMEM; |
| 1582 | } | 1583 | } |
| 1583 | 1584 | ||
| @@ -1602,8 +1603,9 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 18 unmodified lines | |||
| 1602 | q->properties.sdma_queue_id = q->sdma_id / | 1603 | q->properties.sdma_queue_id = q->sdma_id / |
| 1603 | kfd_get_num_sdma_engines(dqm->dev); | 1604 | kfd_get_num_sdma_engines(dqm->dev); |
| 1604 | } else if (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI) { | 1605 | } else if (q->properties.type == KFD_QUEUE_TYPE_SDMA_XGMI) { |
| 1605 | if (bitmap_empty(dqm->xgmi_sdma_bitmap, KFD_MAX_SDMA_QUEUES)) { | 1606 | if (bitmap_empty(dqm->xgmi_sdma_bitmap, get_num_xgmi_sdma_queues(dqm))) { |
| 1606 | dev_err(dev, "No more XGMI SDMA queue to allocate\n"); | 1607 | dev_warn(dev, "No more XGMI SDMA queue to allocate (%d total queues)\n", |
| 1608 | get_num_xgmi_sdma_queues(dqm)); | ||
| 1607 | return -ENOMEM; | 1609 | return -ENOMEM; |
| 1608 | } | 1610 | } |
| 1609 | if (restore_sdma_id) { | 1611 | if (restore_sdma_id) { |
| @@ -1662,8 +1664,8 @@ static int allocate_sdma_queue(struct device_queue_manager *dqm, · 52 unmodified lines | |||
| 1662 | } | 1664 | } |
| 1663 | 1665 | ||
| 1664 | if (!free_bit_found) { | 1666 | if (!free_bit_found) { |
| 1665 | dev_err(dev, "No more SDMA queue to allocate for target ID %i\n", | 1667 | dev_warn(dev, "No more SDMA queue to allocate for target ID %i (%d total queues)\n", |
| 1666 | q->properties.sdma_engine_id); | 1668 | q->properties.sdma_engine_id, num_queues); |
| 1667 | return -ENOMEM; | 1669 | return -ENOMEM; |
| 1668 | } | 1670 | } |
| 1669 | } | 1671 | } |
| @@ -1177,6 +1177,25 @@ void kfd_signal_hw_exception_event(u32 pasid) · 1176 unmodified lines | |||
| 1177 | kfd_unref_process(p); | 1177 | kfd_unref_process(p); |
| 1178 | } | 1178 | } |
| 1179 | 1179 | ||
| 1180 | void kfd_signal_vm_fault_event_with_userptr(struct kfd_process *p, uint64_t gpu_va) | ||
| 1181 | { | ||
| 1182 | struct kfd_process_device *pdd; | ||
| 1183 | struct kfd_hsa_memory_exception_data exception_data; | ||
| 1184 | int i; | ||
| 1185 | |||
| 1186 | memset(&exception_data, 0, sizeof(exception_data)); | ||
| 1187 | exception_data.va = gpu_va; | ||
| 1188 | exception_data.failure.NotPresent = 1; | ||
| 1189 | |||
| 1190 | // Send VM seg fault to all kfd process device | ||
| 1191 | for (i = 0; i < p->n_pdds; i++) { | ||
| 1192 | pdd = p->pdds[i]; | ||
| 1193 | exception_data.gpu_id = pdd->user_gpu_id; | ||
| 1194 | kfd_evict_process_device(pdd); | ||
| 1195 | kfd_signal_vm_fault_event(pdd, NULL, &exception_data); | ||
| 1196 | } | ||
| 1197 | } | ||
| 1198 | |||
| 1180 | void kfd_signal_vm_fault_event(struct kfd_process_device *pdd, | 1199 | void kfd_signal_vm_fault_event(struct kfd_process_device *pdd, |
| 1181 | struct kfd_vm_fault_info *info, | 1200 | struct kfd_vm_fault_info *info, |
| 1182 | struct kfd_hsa_memory_exception_data *data) | 1201 | struct kfd_hsa_memory_exception_data *data) |
| @@ -168,11 +168,12 @@ static bool event_interrupt_isr_v10(struct kfd_node *dev, · 167 unmodified lines | |||
| 168 | client_id != SOC15_IH_CLIENTID_SE3SH) | 168 | client_id != SOC15_IH_CLIENTID_SE3SH) |
| 169 | return false; | 169 | return false; |
| 170 | 170 | ||
| 171 | pr_debug("client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw data:\n", | 171 | dev_dbg(dev->adev->dev, |
| 172 | client_id, source_id, vmid, pasid); | 172 | "client id 0x%x, source id %d, vmid %d, pasid 0x%x. raw data:\n", |
| 173 | pr_debug("%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n", | 173 | client_id, source_id, vmid, pasid); |
| 174 | data[0], data[1], data[2], data[3], | 174 | dev_dbg(dev->adev->dev, "%8X, %8X, %8X, %8X, %8X, %8X, %8X, %8X.\n", |
| 175 | data[4], data[5], data[6], data[7]); | 175 | data[0], data[1], data[2], data[3], data[4], data[5], data[6], |
| 176 | data[7]); | ||
| 176 | 177 | ||
| 177 | /* If there is no valid PASID, it's likely a bug */ | 178 | /* If there is no valid PASID, it's likely a bug */ |
| 178 | if (WARN_ONCE(pasid == 0, "Bug: No PASID in KFD interrupt")) | 179 | if (WARN_ONCE(pasid == 0, "Bug: No PASID in KFD interrupt")) |
| @@ -217,37 +218,66 @@ static void event_interrupt_wq_v10(struct kfd_node *dev, · 38 unmodified lines | |||
| 217 | SQ_INTERRUPT_WORD_WAVE_CTXID1, ENCODING); | 218 | SQ_INTERRUPT_WORD_WAVE_CTXID1, ENCODING); |
| 218 | switch (encoding) { | 219 | switch (encoding) { |
| 219 | case SQ_INTERRUPT_WORD_ENCODING_AUTO: | 220 | case SQ_INTERRUPT_WORD_ENCODING_AUTO: |
| 220 | pr_debug_ratelimited( | 221 | dev_dbg_ratelimited( |
| 222 | dev->adev->dev, | ||
| 221 | "sq_intr: auto, se %d, ttrace %d, wlt %d, ttrac_buf0_full %d, ttrac_buf1_full %d, ttrace_utc_err %d\n", | 223 | "sq_intr: auto, se %d, ttrace %d, wlt %d, ttrac_buf0_full %d, ttrac_buf1_full %d, ttrace_utc_err %d\n", |
| 222 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_AUTO_CTXID1, | 224 | REG_GET_FIELD( |
| 223 | SE_ID), | 225 | context_id1, |
| 224 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | 226 | SQ_INTERRUPT_WORD_AUTO_CTXID1, |
| 225 | THREAD_TRACE), | 227 | SE_ID), |
| 226 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | 228 | REG_GET_FIELD( |
| 227 | WLT), | 229 | context_id0, |
| 228 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | 230 | SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 229 | THREAD_TRACE_BUF0_FULL), | 231 | THREAD_TRACE), |
| 230 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | 232 | REG_GET_FIELD( |
| 231 | THREAD_TRACE_BUF1_FULL), | 233 | context_id0, |
| 232 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | 234 | SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 233 | THREAD_TRACE_UTC_ERROR)); | 235 | WLT), |
| 236 | REG_GET_FIELD( | ||
| 237 | context_id0, | ||
| 238 | SQ_INTERRUPT_WORD_AUTO_CTXID0, | ||
| 239 | THREAD_TRACE_BUF0_FULL), | ||
| 240 | REG_GET_FIELD( | ||
| 241 | context_id0, | ||
| 242 | SQ_INTERRUPT_WORD_AUTO_CTXID0, | ||
| 243 | THREAD_TRACE_BUF1_FULL), | ||
| 244 | REG_GET_FIELD( | ||
| 245 | context_id0, | ||
| 246 | SQ_INTERRUPT_WORD_AUTO_CTXID0, | ||
| 247 | THREAD_TRACE_UTC_ERROR)); | ||
| 234 | break; | 248 | break; |
| 235 | case SQ_INTERRUPT_WORD_ENCODING_INST: | 249 | case SQ_INTERRUPT_WORD_ENCODING_INST: |
| 236 | pr_debug_ratelimited("sq_intr: inst, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n", | 250 | dev_dbg_ratelimited( |
| 237 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, | 251 | dev->adev->dev, |
| 238 | SE_ID), | 252 | "sq_intr: inst, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n", |
| 239 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 253 | REG_GET_FIELD( |
| 240 | DATA), | 254 | context_id1, |
| 241 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 255 | SQ_INTERRUPT_WORD_WAVE_CTXID1, |
| 242 | SA_ID), | 256 | SE_ID), |
| 243 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 257 | REG_GET_FIELD( |
| 244 | PRIV), | 258 | context_id0, |
| 245 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 259 | SQ_INTERRUPT_WORD_WAVE_CTXID0, |
| 246 | WAVE_ID), | 260 | DATA), |
| 247 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 261 | REG_GET_FIELD( |
| 248 | SIMD_ID), | 262 | context_id0, |
| 249 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, | 263 | SQ_INTERRUPT_WORD_WAVE_CTXID0, |
| 250 | WGP_ID)); | 264 | SA_ID), |
| 265 | REG_GET_FIELD( | ||
| 266 | context_id0, | ||
| 267 | SQ_INTERRUPT_WORD_WAVE_CTXID0, | ||
| 268 | PRIV), | ||
| 269 | REG_GET_FIELD( | ||
| 270 | context_id0, | ||
| 271 | SQ_INTERRUPT_WORD_WAVE_CTXID0, | ||
| 272 | WAVE_ID), | ||
| 273 | REG_GET_FIELD( | ||
| 274 | context_id0, | ||
| 275 | SQ_INTERRUPT_WORD_WAVE_CTXID0, | ||
| 276 | SIMD_ID), | ||
| 277 | REG_GET_FIELD( | ||
| 278 | context_id1, | ||
| 279 | SQ_INTERRUPT_WORD_WAVE_CTXID1, | ||
| 280 | WGP_ID)); | ||
| 251 | if (context_id0 & SQ_INTERRUPT_WORD_WAVE_CTXID0__PRIV_MASK) { | 281 | if (context_id0 & SQ_INTERRUPT_WORD_WAVE_CTXID0__PRIV_MASK) { |
| 252 | if (kfd_set_dbg_ev_from_interrupt(dev, pasid, | 282 | if (kfd_set_dbg_ev_from_interrupt(dev, pasid, |
| 253 | KFD_DEBUG_DOORBELL_ID(context_id0), | 283 | KFD_DEBUG_DOORBELL_ID(context_id0), |
| @@ -259,21 +289,37 @@ static void event_interrupt_wq_v10(struct kfd_node *dev, · 5 unmodified lines | |||
| 259 | case SQ_INTERRUPT_WORD_ENCODING_ERROR: | 289 | case SQ_INTERRUPT_WORD_ENCODING_ERROR: |
| 260 | sq_intr_err_type = REG_GET_FIELD(context_id0, KFD_CTXID0, | 290 | sq_intr_err_type = REG_GET_FIELD(context_id0, KFD_CTXID0, |
| 261 | ERR_TYPE); | 291 | ERR_TYPE); |
| 262 | pr_warn_ratelimited("sq_intr: error, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d, err_type %d\n", | 292 | dev_warn_ratelimited( |
| 263 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, | 293 | dev->adev->dev, |
| 264 | SE_ID), | 294 | "sq_intr: error, se %d, data 0x%x, sa %d, priv %d, wave_id %d, simd_id %d, wgp_id %d, err_type %d\n", |
| 265 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 295 | REG_GET_FIELD( |
| 266 | DATA), | 296 | context_id1, |
| 267 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 297 | SQ_INTERRUPT_WORD_WAVE_CTXID1, |
| 268 | SA_ID), | 298 | SE_ID), |
| 269 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 299 | REG_GET_FIELD( |
| 270 | PRIV), | 300 | context_id0, |
| 271 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 301 | SQ_INTERRUPT_WORD_WAVE_CTXID0, |
| 272 | WAVE_ID), | 302 | DATA), |
| 273 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, | 303 | REG_GET_FIELD( |
| 274 | SIMD_ID), | 304 | context_id0, |
| 275 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, | 305 | SQ_INTERRUPT_WORD_WAVE_CTXID0, |
| 276 | WGP_ID), | 306 | SA_ID), |
| 307 | REG_GET_FIELD( | ||
| 308 | context_id0, | ||
| 309 | SQ_INTERRUPT_WORD_WAVE_CTXID0, | ||
| 310 | PRIV), | ||
| 311 | REG_GET_FIELD( | ||
| 312 | context_id0, | ||
| 313 | SQ_INTERRUPT_WORD_WAVE_CTXID0, | ||
| 314 | WAVE_ID), | ||
| 315 | REG_GET_FIELD( | ||
| 316 | context_id0, | ||
| 317 | SQ_INTERRUPT_WORD_WAVE_CTXID0, | ||
| 318 | SIMD_ID), | ||
| 319 | REG_GET_FIELD( | ||
| 320 | context_id1, | ||
| 321 | SQ_INTERRUPT_WORD_WAVE_CTXID1, | ||
| 322 | WGP_ID), | ||
| 277 | sq_intr_err_type); | 323 | sq_intr_err_type); |
| 278 | break; | 324 | break; |
| 279 | default: | 325 | default: |
| @@ -148,44 +148,69 @@ enum SQ_INTERRUPT_ERROR_TYPE { · 147 unmodified lines | |||
| 148 | #define KFD_CTXID0_DOORBELL_ID(ctxid0) ((ctxid0) & \ | 148 | #define KFD_CTXID0_DOORBELL_ID(ctxid0) ((ctxid0) & \ |
| 149 | KFD_CTXID0_DOORBELL_ID_MASK) | 149 | KFD_CTXID0_DOORBELL_ID_MASK) |
| 150 | 150 | ||
| 151 | static void print_sq_intr_info_auto(uint32_t context_id0, uint32_t context_id1) | 151 | static void print_sq_intr_info_auto(struct kfd_node *dev, uint32_t context_id0, |
| 152 | uint32_t context_id1) | ||
| 152 | { | 153 | { |
| 153 | pr_debug_ratelimited( | 154 | dev_dbg_ratelimited( |
| 155 | dev->adev->dev, | ||
| 154 | "sq_intr: auto, ttrace %d, wlt %d, ttrace_buf_full %d, reg_tms %d, cmd_tms %d, host_cmd_ovf %d, host_reg_ovf %d, immed_ovf %d, ttrace_utc_err %d\n", | 156 | "sq_intr: auto, ttrace %d, wlt %d, ttrace_buf_full %d, reg_tms %d, cmd_tms %d, host_cmd_ovf %d, host_reg_ovf %d, immed_ovf %d, ttrace_utc_err %d\n", |
| 155 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE), | 157 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 158 | THREAD_TRACE), | ||
| 156 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, WLT), | 159 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, WLT), |
| 157 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE_BUF_FULL), | 160 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 158 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, REG_TIMESTAMP), | 161 | THREAD_TRACE_BUF_FULL), |
| 159 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, CMD_TIMESTAMP), | 162 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 160 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, HOST_CMD_OVERFLOW), | 163 | REG_TIMESTAMP), |
| 161 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, HOST_REG_OVERFLOW), | 164 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 162 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, IMMED_OVERFLOW), | 165 | CMD_TIMESTAMP), |
| 163 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, THREAD_TRACE_UTC_ERROR)); | 166 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, |
| 167 | HOST_CMD_OVERFLOW), | ||
| 168 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | ||
| 169 | HOST_REG_OVERFLOW), | ||
| 170 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | ||
| 171 | IMMED_OVERFLOW), | ||
| 172 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_AUTO_CTXID0, | ||
| 173 | THREAD_TRACE_UTC_ERROR)); | ||
| 164 | } | 174 | } |
| 165 | 175 | ||
| 166 | static void print_sq_intr_info_inst(uint32_t context_id0, uint32_t context_id1) | 176 | static void print_sq_intr_info_inst(struct kfd_node *dev, uint32_t context_id0, |
| 177 | uint32_t context_id1) | ||
| 167 | { | 178 | { |
| 168 | pr_debug_ratelimited( | 179 | dev_dbg_ratelimited( |
| 180 | dev->adev->dev, | ||
| 169 | "sq_intr: inst, data 0x%08x, sh %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n", | 181 | "sq_intr: inst, data 0x%08x, sh %d, priv %d, wave_id %d, simd_id %d, wgp_id %d\n", |
| 170 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, DATA), | 182 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, DATA), |
| 171 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, SH_ID), | 183 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, |
| 184 | SH_ID), | ||
| 172 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, PRIV), | 185 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, PRIV), |
| 173 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, WAVE_ID), | 186 | REG_GET_FIELD(context_id0, SQ_INTERRUPT_WORD_WAVE_CTXID0, |
| 174 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, SIMD_ID), | 187 | WAVE_ID), |
| 175 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, WGP_ID)); | 188 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, |
| 189 | SIMD_ID), | ||
| 190 | REG_GET_FIELD(context_id1, SQ_INTERRUPT_WORD_WAVE_CTXID1, | ||
| 191 | WGP_ID)); | ||
| 176 | } | 192 | } |
| 177 | 193 | ||
72 more lines aren’t shown here.
This diff is large: its 166 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 1707 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 254 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 421 lines aren’t shown here.
This diff is large: its 333 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 149 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 208 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 6570 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 285 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
File renamed without changes.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 444 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 468 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
File renamed without changes.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 283 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 484 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 235 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 806 lines aren’t shown here.
This diff is large: its 127 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 319 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 372 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.