Pull drm updates from Dave Airlie: "There was a rather late merge of a new color pipeline feature, that some userspace projects are blocked on, and has seen a lot of work in amdgpu. This should have seen some time in -next. There is additional support for this for Intel, that if it arrives in the next day or two I'll pass it on in another pull request and you can decide if you want to take it. Highlights: - Arm Ethos NPU accelerator driver - new DRM color pipeline support - amdgpu will now run discrete SI/CIK cards instead of radeon, which enables vulkan support in userspace - msm gets gen8 gpu support - initial Xe3P support in xe Full detail summary: New driver: - Arm Ethos-U65/U85 accel driver Core: - support the drm color pipeline in vkms/amdgfx - add support for drm colorop pipeline - add COLOR PIPELINE plane property - add DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE - throttle dirty worker with vblank - use drm_for_each_bridge_in_chain_scoped in drm's bridge code - Ensure drm_client_modeset tests are enabled in UML - add simulated vblank interrupt - use in drivers - dumb buffer sizing helper - move freeing of drm client memory to driver - crtc sharpness strength property - stop using system_wq in scheduler/drivers - support emergency restore in drm-client Rust: - make slice::as_flattened usable on all supported rustc - add FromBytes::from_bytes_prefix() method - remove redundant device ptr from Rust GEM object - Change how AlwaysRefCounted is implemented for GEM objects gpuvm: - Add deferred vm_bo cleanup to GPUVM (for rust) atomic: - cleanup and improve state handling interfaces buddy: - optimize block management dma-buf: - heaps: Create heap per CMA reserved location - improve userspace documentation dp: - add POST_LT_ADJ_REQ training sequence - DPCD dSC quirk for synaptics panamera devices - helpers to query branch DSC max throughput ttm: - Rename ttm_bo_put to ttm_bo_fini - allow page protection flags on risc-v - rework pipelined eviction fence handling amdgpu: - enable amdgpu by default for SI/CI dGPUs - enable DC by default on SI - refactor CIK/SI enablement - add ABM KMS property - Re-enable DM idle optimizations - DC Analog encoders support - Powerplay fixes for fiji/iceland - Enable DC on bonaire by default - HMM cleanup - Add new RAS framework - DML2.1 updates - YCbCr420 fixes - DC FP fixes - DMUB fixes - LTTPR fixes - DTBCLK fixes - DMU cursor offload handling - Userq validation improvements - Unify shutdown callback handling - Suspend improvements - Power limit code cleanup - SR-IOV fixes - AUX backlight fixes - DCN 3.5 fixes - HDMI compliance fixes - DCN 4.0.1 cursor updates - DCN interrupt fix - DC KMS full update improvements - Add additional HDCP traces - DCN 3.2 fixes - DP MST fixes - Add support for new SR-IOV mailbox interface - UQ reset support - HDP flush rework - VCE1 support amdkfd: - HMM cleanups - Relax checks on save area overallocations - Fix GPU mappings after prefetch radeon: - refactor CIK/SI enablement xe: - Initial Xe3P support - panic support on VRAM for display - fix stolen size check - Loosen used tracking restriction - New SR-IOV debugfs structure and debugfs updates - Hide the GPU madvise flag behind a VM_BIND flag - Always expose VRAM provisioning data on discrete GPUs - Allow VRAM mappings for userptr when used with SVM - Allow pinning of p2p dma-buf - Use per-tile debugfs where appropriate - Add documentation for Execution Queues - PF improvements - VF migration recovery redesign work - User / Kernel VRAM partitioning - Update Tile-based messages - Allow configfs to disable specific GT types - VF provisioning and migration improvements - use SVM range helpers in PT layer - Initial CRI support - access VF registers using dedicated MMIO view - limit number of jobs per exec queue - add sriov_admin sysfs tree - more crescent island specific support - debugfs residency counter - SRIOV migration work - runtime registers for GFX 35 i915: - add initial Xe3p_LPD display version 35 support - Enable LNL+ content adaptive sharpness filter - Use optimized VRR guardband - Enable Xe3p LT PHY - enable FBC support for Xe3p_LPD display - add display 30.02 firmware support - refactor SKL+ watermark latency setup - refactor fbdev handling - call i915/xe runtime PM via function pointers - refactor i915/xe stolen memory/display interfaces - use display version instead of gfx version in display code - extend i915_display_info with Type-C port details - lots of display cleanups/refactorings - set O_LARGEFILE in __create_shmem - skuip guc communication warning on reset - fix time conversions - defeature DRRS on LNL+ - refactor intel_frontbuffer split between i915/xe/display - convert inteL_rom interfaces to struct drm_device - unify display register polling interfaces - aovid lock inversion when pinning to GGTT on CHV/BXT+VTD panel: - Add KD116N3730A08/A12, chromebook mt8189 - JT101TM023, LQ079L1SX01, - GLD070WX3-SL01 MIPI DSI - Samsung LTL106AL0, Samsung LTL106AL01 - Raystar RFF500F-AWH-DNN - Winstar WF70A8SYJHLNGA - Wanchanglong w552946aaa - Samsung SOFEF00 - Lenovo X13s panel - ilitek-ili9881c - add rpi 5" support - visionx-rm69299 - add backlight support - edp - support AUI B116XAN02.0 bridge: - improve ref counting - ti-sn65dsi86 - add support for DP mode with HPD - synopsis: support CEC, init timer with correct freq - ASL CS5263 DP-to-HDMI bridge support nova-core: - introduce bitfield! macro - introduce safe integer converters - GSP inits to fully booted state on Ampere - Use more future-proof register for GPU identification nova-drm: - select NOVA_CORE - 64-bit only nouveau: - improve reclocking on tegra 186+ - add large page and compression support msm: - GPU: - Gen8 support: A840 (Kaanapali) and X2-85 (Glymur) - A612 support - MDSS: - Added support for Glymur and QCS8300 platforms - DPU: - Enabled Quad-Pipe support, unlocking higher resolutions support - Added support for Glymur platform - Documented DPU on QCS8300 platform as supported - DisplayPort: - Added support for Glymur platform - Added support lame remapping inside DP block - Documented DisplayPort controller on QCS8300 and SM6150/QCS615 as supported tegra: - NVJPG driver panfrost: - display JM contexts over debugfs - export JM contexts to userspace - improve error and job handling panthor: - support custom ASN_HASH for mt8196 - support mali-G1 GPU - flush shmem write before mapping buffers uncached - make timeout per-queue instead of per-job mediatek: - MT8195/88 HDMIv2/DDCv2 support rockchip: - dsi: add support for RK3368 amdxdna: - enhance runtime PM - last hardware error reading uapi - support firmware debug output - add resource and telemetry data uapi - preemption support imx: - add driver for HDMI TX Parallel audio interface ivpu: - add support for user-managed preemption buffer - add userptr support - update JSM firware API to 3.33.0 - add better alloc/free warnings - fix page fault in unbind all bos - rework bind/unbind of imported buffers - enable MCA ECC signalling - split fw runtime and global memory buffers - add fdinfo memory statistics tidss: - convert to drm logging - logging cleanup ast: - refactor generation init paths - add per chip generation detect_tx_chip - set quirks for each chip model atmel-hlcdc: - set LCDC_ATTRE register in plane disable - set correct values for plane scaler solomon: - use drm helper for get_modes and move_valid sitronix: - fix output position when clearing screens qaic: - support dma-buf exports - support new firmware's READ_DATA implementation - sahara AIC200 image table update - add sysfs support - add coredump support - add uevents support - PM support sun4i: - layer refactors to decouple plane from output - improve DE33 support vc4: - switch to generic CEC helpers komeda: - use drm_ logging functions vkms: - configfs support for display configuration vgem: - fix fence timer deadlock etnaviv: - add HWDB entry for GC8000 Nano Ultra VIP r6205" * tag 'drm-next-2025-12-03' of https://gitlab.freedesktop.org/drm/kernel: (1869 commits) Revert "drm/amd: Skip power ungate during suspend for VPE" drm/amdgpu: use common defines for HUB faults drm/amdgpu/gmc12: add amdgpu_vm_handle_fault() handling drm/amdgpu/gmc11: add amdgpu_vm_handle_fault() handling drm/amdgpu: use static ids for ACP platform devs drm/amdgpu/sdma6: Update SDMA 6.0.3 FW version to include UMQ protected-fence fix drm/amdgpu: Forward VMID reservation errors drm/amdgpu/gmc8: Delegate VM faults to soft IRQ handler ring drm/amdgpu/gmc7: Delegate VM faults to soft IRQ handler ring drm/amdgpu/gmc6: Delegate VM faults to soft IRQ handler ring drm/amdgpu/gmc6: Cache VM fault info drm/amdgpu/gmc6: Don't print MC client as it's unknown drm/amdgpu/cz_ih: Enable soft IRQ handler ring drm/amdgpu/tonga_ih: Enable soft IRQ handler ring drm/amdgpu/iceland_ih: Enable soft IRQ handler ring drm/amdgpu/cik_ih: Enable soft IRQ handler ring drm/amdgpu/si_ih: Enable soft IRQ handler ring drm/amd/display: fix typo in display_mode_core_structs.h drm/amd/display: fix Smart Power OLED not working after S4 drm/amd/display: Move RGB-type check for audio sync to DCE HW sequence ...
Unified · Split
| @@ -167,7 +167,7 @@ ForEachMacros: · 166 unmodified lines | |||
| 167 | - 'drm_connector_for_each_possible_encoder' | 167 | - 'drm_connector_for_each_possible_encoder' |
| 168 | - 'drm_exec_for_each_locked_object' | 168 | - 'drm_exec_for_each_locked_object' |
| 169 | - 'drm_exec_for_each_locked_object_reverse' | 169 | - 'drm_exec_for_each_locked_object_reverse' |
| 170 | - 'drm_for_each_bridge_in_chain' | 170 | - 'drm_for_each_bridge_in_chain_scoped' |
| 171 | - 'drm_for_each_connector_iter' | 171 | - 'drm_for_each_connector_iter' |
| 172 | - 'drm_for_each_crtc' | 172 | - 'drm_for_each_crtc' |
| 173 | - 'drm_for_each_crtc_reverse' | 173 | - 'drm_for_each_crtc_reverse' |
| @@ -174,6 +174,7 @@ Carlos Bilbao <carlos.bilbao@kernel.org> <bilbao@vt.edu> · 173 unmodified lines | |||
| 174 | Changbin Du <changbin.du@intel.com> <changbin.du@gmail.com> | 174 | Changbin Du <changbin.du@intel.com> <changbin.du@gmail.com> |
| 175 | Chao Yu <chao@kernel.org> <chao2.yu@samsung.com> | 175 | Chao Yu <chao@kernel.org> <chao2.yu@samsung.com> |
| 176 | Chao Yu <chao@kernel.org> <yuchao0@huawei.com> | 176 | Chao Yu <chao@kernel.org> <yuchao0@huawei.com> |
| 177 | Chen-Yu Tsai <wens@kernel.org> <wens@csie.org> | ||
| 177 | Chester Lin <chester62515@gmail.com> <clin@suse.com> | 178 | Chester Lin <chester62515@gmail.com> <clin@suse.com> |
| 178 | Chris Chiu <chris.chiu@canonical.com> <chiu@endlessm.com> | 179 | Chris Chiu <chris.chiu@canonical.com> <chiu@endlessm.com> |
| 179 | Chris Chiu <chris.chiu@canonical.com> <chiu@endlessos.org> | 180 | Chris Chiu <chris.chiu@canonical.com> <chiu@endlessos.org> |
| @@ -0,0 +1,19 @@ | |||
| 1 | What: /sys/bus/pci/drivers/qaic/XXXX:XX:XX.X/accel/accel<minor_nr>/dbc<N>_state | ||
| 2 | Date: October 2025 | ||
| 3 | KernelVersion: 6.19 | ||
| 4 | Contact: Jeff Hugo <jeff.hugo@oss.qualcomm.com> | ||
| 5 | Description: Represents the current state of DMA Bridge channel (DBC). Below are the possible | ||
| 6 | states: | ||
| 7 | |||
| 8 | =================== ========================================================== | ||
| 9 | IDLE (0) DBC is free and can be activated | ||
| 10 | ASSIGNED (1) DBC is activated and a workload is running on device | ||
| 11 | BEFORE_SHUTDOWN (2) Sub-system associated with this workload has crashed and | ||
| 12 | it will shutdown soon | ||
| 13 | AFTER_SHUTDOWN (3) Sub-system associated with this workload has crashed and | ||
| 14 | it has shutdown | ||
| 15 | BEFORE_POWER_UP (4) Sub-system associated with this workload is shutdown and | ||
| 16 | it will be powered up soon | ||
| 17 | AFTER_POWER_UP (5) Sub-system associated with this workload is now powered up | ||
| 18 | =================== ========================================================== | ||
| 19 | Users: Any userspace application or clients interested in DBC state. | ||
| @@ -0,0 +1,159 @@ | |||
| 1 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/ | ||
| 2 | Date: October 2025 | ||
| 3 | KernelVersion: 6.19 | ||
| 4 | Contact: intel-xe@lists.freedesktop.org | ||
| 5 | Description: | ||
| 6 | This directory appears for the particular Intel Xe device when: | ||
| 7 | |||
| 8 | - device supports SR-IOV, and | ||
| 9 | - device is a Physical Function (PF), and | ||
| 10 | - driver support for the SR-IOV PF is enabled on given device. | ||
| 11 | |||
| 12 | This directory is used as a root for all attributes required to | ||
| 13 | manage both Physical Function (PF) and Virtual Functions (VFs). | ||
| 14 | |||
| 15 | |||
| 16 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/pf/ | ||
| 17 | Date: October 2025 | ||
| 18 | KernelVersion: 6.19 | ||
| 19 | Contact: intel-xe@lists.freedesktop.org | ||
| 20 | Description: | ||
| 21 | This directory holds attributes related to the SR-IOV Physical | ||
| 22 | Function (PF). | ||
| 23 | |||
| 24 | |||
| 25 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf1/ | ||
| 26 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf2/ | ||
| 27 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf<N>/ | ||
| 28 | Date: October 2025 | ||
| 29 | KernelVersion: 6.19 | ||
| 30 | Contact: intel-xe@lists.freedesktop.org | ||
| 31 | Description: | ||
| 32 | These directories hold attributes related to the SR-IOV Virtual | ||
| 33 | Functions (VFs). | ||
| 34 | |||
| 35 | Note that the VF number <N> is 1-based as described in PCI SR-IOV | ||
| 36 | specification as the Xe driver follows that naming schema. | ||
| 37 | |||
| 38 | There could be "vf1", "vf2" and so on, up to "vf<N>", where <N> | ||
| 39 | matches the value of the "sriov_totalvfs" attribute. | ||
| 40 | |||
| 41 | |||
| 42 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/pf/profile/exec_quantum_ms | ||
| 43 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/pf/profile/preempt_timeout_us | ||
| 44 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/pf/profile/sched_priority | ||
| 45 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf<n>/profile/exec_quantum_ms | ||
| 46 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf<n>/profile/preempt_timeout_us | ||
| 47 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf<n>/profile/sched_priority | ||
| 48 | Date: October 2025 | ||
| 49 | KernelVersion: 6.19 | ||
| 50 | Contact: intel-xe@lists.freedesktop.org | ||
| 51 | Description: | ||
| 52 | These files expose scheduling parameters for the PF and its VFs, and | ||
| 53 | are visible only on Intel Xe platforms that use time-sliced GPU sharing. | ||
| 54 | They can be changed even if VFs are enabled and running and reflect the | ||
| 55 | settings of all tiles/GTs assigned to the given function. | ||
| 56 | |||
| 57 | exec_quantum_ms: (RW) unsigned integer | ||
| 58 | The GT execution quantum (EQ) in [ms] for the given function. | ||
| 59 | Actual quantum value might be aligned per HW/FW requirements. | ||
| 60 | |||
| 61 | Default is 0 (unlimited). | ||
| 62 | |||
| 63 | preempt_timeout_us: (RW) unsigned integer | ||
| 64 | The GT preemption timeout in [us] of the given function. | ||
| 65 | Actual timeout value might be aligned per HW/FW requirements. | ||
| 66 | |||
| 67 | Default is 0 (unlimited). | ||
| 68 | |||
| 69 | sched_priority: (RW/RO) string | ||
| 70 | The GT scheduling priority of the given function. | ||
| 71 | |||
| 72 | "low" - function will be scheduled on the GPU for its EQ/PT | ||
| 73 | only if function has any work already submitted. | ||
| 74 | |||
| 75 | "normal" - functions will be scheduled on the GPU for its EQ/PT | ||
| 76 | irrespective of whether it has submitted a work or not. | ||
| 77 | |||
| 78 | "high" - function will be scheduled on the GPU for its EQ/PT | ||
| 79 | in the next time-slice after the current one completes | ||
| 80 | and function has a work submitted. | ||
| 81 | |||
| 82 | Default is "low". | ||
| 83 | |||
| 84 | When read, this file will display the current and available | ||
| 85 | scheduling priorities. The currently active priority level will | ||
| 86 | be enclosed in square brackets, like: | ||
| 87 | |||
| 88 | [low] normal high | ||
| 89 | |||
| 90 | This file can be read-only if changing the priority is not | ||
| 91 | supported. | ||
| 92 | |||
| 93 | Writes to these attributes may fail with errors like: | ||
| 94 | -EINVAL if provided input is malformed or not recognized, | ||
| 95 | -EPERM if change is not applicable on given HW/FW, | ||
| 96 | -EIO if FW refuses to change the provisioning. | ||
| 97 | |||
| 98 | Reads from these attributes may fail with: | ||
| 99 | -EUCLEAN if value is not consistent across all tiles/GTs. | ||
| 100 | |||
| 101 | |||
| 102 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/.bulk_profile/exec_quantum_ms | ||
| 103 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/.bulk_profile/preempt_timeout_us | ||
| 104 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/.bulk_profile/sched_priority | ||
| 105 | Date: October 2025 | ||
| 106 | KernelVersion: 6.19 | ||
| 107 | Contact: intel-xe@lists.freedesktop.org | ||
| 108 | Description: | ||
| 109 | These files allows bulk reconfiguration of the scheduling parameters | ||
| 110 | of the PF or VFs and are available only for Intel Xe platforms with | ||
| 111 | GPU sharing based on the time-slice basis. These scheduling parameters | ||
| 112 | can be changed even if VFs are enabled and running. | ||
| 113 | |||
| 114 | exec_quantum_ms: (WO) unsigned integer | ||
| 115 | The GT execution quantum (EQ) in [ms] to be applied to all functions. | ||
| 116 | See sriov_admin/{pf,vf<N>}/profile/exec_quantum_ms for more details. | ||
| 117 | |||
| 118 | preempt_timeout_us: (WO) unsigned integer | ||
| 119 | The GT preemption timeout (PT) in [us] to be applied to all functions. | ||
| 120 | See sriov_admin/{pf,vf<N>}/profile/preempt_timeout_us for more details. | ||
| 121 | |||
| 122 | sched_priority: (RW/RO) string | ||
| 123 | The GT scheduling priority to be applied for all functions. | ||
| 124 | See sriov_admin/{pf,vf<N>}/profile/sched_priority for more details. | ||
| 125 | |||
| 126 | Writes to these attributes may fail with errors like: | ||
| 127 | -EINVAL if provided input is malformed or not recognized, | ||
| 128 | -EPERM if change is not applicable on given HW/FW, | ||
| 129 | -EIO if FW refuses to change the provisioning. | ||
| 130 | |||
| 131 | |||
| 132 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf<n>/stop | ||
| 133 | Date: October 2025 | ||
| 134 | KernelVersion: 6.19 | ||
| 135 | Contact: intel-xe@lists.freedesktop.org | ||
| 136 | Description: | ||
| 137 | This file allows to control scheduling of the VF on the Intel Xe GPU | ||
| 138 | platforms. It allows to implement custom policy mechanism in case VFs | ||
| 139 | are misbehaving or triggering adverse events above defined thresholds. | ||
| 140 | |||
| 141 | stop: (WO) bool | ||
| 142 | All GT executions of given function shall be immediately stopped. | ||
| 143 | To allow scheduling this VF again, the VF FLR must be triggered. | ||
| 144 | |||
| 145 | Writes to this attribute may fail with errors like: | ||
| 146 | -EINVAL if provided input is malformed or not recognized, | ||
| 147 | -EPERM if change is not applicable on given HW/FW, | ||
| 148 | -EIO if FW refuses to change the scheduling. | ||
| 149 | |||
| 150 | |||
| 151 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/pf/device | ||
| 152 | What: /sys/bus/pci/drivers/xe/.../sriov_admin/vf<n>/device | ||
| 153 | Date: October 2025 | ||
| 154 | KernelVersion: 6.19 | ||
| 155 | Contact: intel-xe@lists.freedesktop.org | ||
| 156 | Description: | ||
| 157 | These are symlinks to the underlying PCI device entry representing | ||
| 158 | given Xe SR-IOV function. For the PF, this link is always present. | ||
| 159 | For VFs, this link is present only for currently enabled VFs. | ||
| @@ -487,8 +487,8 @@ one user crashes, the fallout of that should be limited to that workload and not · 486 unmodified lines | |||
| 487 | impact other workloads. SSR accomplishes this. | 487 | impact other workloads. SSR accomplishes this. |
| 488 | 488 | ||
| 489 | If a particular workload crashes, QSM notifies the host via the QAIC_SSR MHI | 489 | If a particular workload crashes, QSM notifies the host via the QAIC_SSR MHI |
| 490 | channel. This notification identifies the workload by it's assigned DBC. A | 490 | channel. This notification identifies the workload by its assigned DBC. A |
| 491 | multi-stage recovery process is then used to cleanup both sides, and get the | 491 | multi-stage recovery process is then used to cleanup both sides, and gets the |
| 492 | DBC/NSPs into a working state. | 492 | DBC/NSPs into a working state. |
| 493 | 493 | ||
| 494 | When SSR occurs, any state in the workload is lost. Any inputs that were in | 494 | When SSR occurs, any state in the workload is lost. Any inputs that were in |
| @@ -496,6 +496,27 @@ process, or queued by not yet serviced, are lost. The loaded artifacts will · 1 unmodified line | |||
| 496 | remain in on-card DDR, but the host will need to re-activate the workload if | 496 | remain in on-card DDR, but the host will need to re-activate the workload if |
| 497 | it desires to recover the workload. | 497 | it desires to recover the workload. |
| 498 | 498 | ||
| 499 | When SSR occurs for a specific NSP, the assigned DBC goes through the | ||
| 500 | following state transactions in order: | ||
| 501 | |||
| 502 | DBC_STATE_BEFORE_SHUTDOWN | ||
| 503 | Indicates that the affected NSP was found in an unrecoverable error | ||
| 504 | condition. | ||
| 505 | DBC_STATE_AFTER_SHUTDOWN | ||
| 506 | Indicates that the NSP is under reset. | ||
| 507 | DBC_STATE_BEFORE_POWER_UP | ||
| 508 | Indicates that the NSP's debug information has been collected, and is | ||
| 509 | ready to be collected by the host (if desired). At that stage the NSP | ||
| 510 | is restarted by QSM. | ||
| 511 | DBC_STATE_AFTER_POWER_UP | ||
| 512 | Indicates that the NSP has been restarted, fully operational and is | ||
| 513 | in idle state. | ||
| 514 | |||
| 515 | SSR also has an optional crashdump collection feature. If enabled, the host can | ||
| 516 | collect the memory dump for the crashed NSP and dump it to the user space via | ||
| 517 | the dev_coredump subsystem. The host can also decline the crashdump collection | ||
| 518 | request from the device. | ||
| 519 | |||
| 499 | Reliability, Accessibility, Serviceability (RAS) | 520 | Reliability, Accessibility, Serviceability (RAS) |
| 500 | ================================================ | 521 | ================================================ |
| 501 | 522 | ||
| @@ -36,7 +36,7 @@ polling mode and reenables the IRQ line. · 35 unmodified lines | |||
| 36 | This mitigation in QAIC is very effective. The same lprnet usecase that | 36 | This mitigation in QAIC is very effective. The same lprnet usecase that |
| 37 | generates 100k IRQs per second (per /proc/interrupts) is reduced to roughly 64 | 37 | generates 100k IRQs per second (per /proc/interrupts) is reduced to roughly 64 |
| 38 | IRQs over 5 minutes while keeping the host system stable, and having the same | 38 | IRQs over 5 minutes while keeping the host system stable, and having the same |
| 39 | workload throughput performance (within run to run noise variation). | 39 | workload throughput performance (within run-to-run noise variation). |
| 40 | 40 | ||
| 41 | Single MSI Mode | 41 | Single MSI Mode |
| 42 | --------------- | 42 | --------------- |
| @@ -49,7 +49,7 @@ useful to be able to fall back to a single MSI when needed. · 6 unmodified lines | |||
| 49 | To support this fallback, we allow the case where only one MSI is able to be | 49 | To support this fallback, we allow the case where only one MSI is able to be |
| 50 | allocated, and share that one MSI between MHI and the DBCs. The device detects | 50 | allocated, and share that one MSI between MHI and the DBCs. The device detects |
| 51 | when only one MSI has been configured and directs the interrupts for the DBCs | 51 | when only one MSI has been configured and directs the interrupts for the DBCs |
| 52 | to the interrupt normally used for MHI. Unfortunately this means that the | 52 | to the interrupt normally used for MHI. Unfortunately, this means that the |
| 53 | interrupt handlers for every DBC and MHI wake up for every interrupt that | 53 | interrupt handlers for every DBC and MHI wake up for every interrupt that |
| 54 | arrives; however, the DBC threaded irq handlers only are started when work to be | 54 | arrives; however, the DBC threaded irq handlers only are started when work to be |
| 55 | done is detected (MHI will always start its threaded handler). | 55 | done is detected (MHI will always start its threaded handler). |
| @@ -62,9 +62,9 @@ never disabled, allowing each new entry to the FIFO to trigger a new interrupt. · 6 unmodified lines | |||
| 62 | Neural Network Control (NNC) Protocol | 62 | Neural Network Control (NNC) Protocol |
| 63 | ===================================== | 63 | ===================================== |
| 64 | 64 | ||
| 65 | The implementation of NNC is split between the KMD (QAIC) and UMD. In general | 65 | The implementation of NNC is split between the KMD (QAIC) and UMD. In general, |
| 66 | QAIC understands how to encode/decode NNC wire protocol, and elements of the | 66 | QAIC understands how to encode/decode NNC wire protocol, and elements of the |
| 67 | protocol which require kernel space knowledge to process (for example, mapping | 67 | protocol which requires kernel space knowledge to process (for example, mapping |
| 68 | host memory to device IOVAs). QAIC understands the structure of a message, and | 68 | host memory to device IOVAs). QAIC understands the structure of a message, and |
| 69 | all of the transactions. QAIC does not understand commands (the payload of a | 69 | all of the transactions. QAIC does not understand commands (the payload of a |
| 70 | passthrough transaction). | 70 | passthrough transaction). |
| @@ -49,6 +49,10 @@ properties: · 48 unmodified lines | |||
| 49 | $ref: /schemas/graph.yaml#/properties/port | 49 | $ref: /schemas/graph.yaml#/properties/port |
| 50 | description: HDMI output port | 50 | description: HDMI output port |
| 51 | 51 | ||
| 52 | port@2: | ||
| 53 | $ref: /schemas/graph.yaml#/properties/port | ||
| 54 | description: Parallel audio input port | ||
| 55 | |||
| 52 | required: | 56 | required: |
| 53 | - port@0 | 57 | - port@0 |
| 54 | - port@1 | 58 | - port@1 |
| @@ -98,5 +102,13 @@ examples: · 43 unmodified lines | |||
| 98 | remote-endpoint = <&hdmi0_con>; | 102 | remote-endpoint = <&hdmi0_con>; |
| 99 | }; | 103 | }; |
| 100 | }; | 104 | }; |
| 105 | |||
| 106 | port@2 { | ||
| 107 | reg = <2>; | ||
| 108 | |||
| 109 | endpoint { | ||
| 110 | remote-endpoint = <&pai_to_hdmi_tx>; | ||
| 111 | }; | ||
| 112 | }; | ||
| 101 | }; | 113 | }; |
| 102 | }; | 114 | }; |
| @@ -19,6 +19,7 @@ properties: · 18 unmodified lines | |||
| 19 | compatible: | 19 | compatible: |
| 20 | enum: | 20 | enum: |
| 21 | - ite,it66121 | 21 | - ite,it66121 |
| 22 | - ite,it66122 | ||
| 22 | - ite,it6610 | 23 | - ite,it6610 |
| 23 | 24 | ||
| 24 | reg: | 25 | reg: |
| @@ -14,6 +14,9 @@ description: | · 13 unmodified lines | |||
| 14 | R-Car Gen4 SoCs. The encoder can operate in either DSI or CSI-2 mode, with up | 14 | R-Car Gen4 SoCs. The encoder can operate in either DSI or CSI-2 mode, with up |
| 15 | to four data lanes. | 15 | to four data lanes. |
| 16 | 16 | ||
| 17 | allOf: | ||
| 18 | - $ref: /schemas/display/dsi-controller.yaml# | ||
| 19 | |||
| 17 | properties: | 20 | properties: |
| 18 | compatible: | 21 | compatible: |
| 19 | enum: | 22 | enum: |
| @@ -80,14 +83,14 @@ required: · 60 unmodified lines | |||
| 80 | - resets | 83 | - resets |
| 81 | - ports | 84 | - ports |
| 82 | 85 | ||
| 83 | additionalProperties: false | 86 | unevaluatedProperties: false |
| 84 | 87 | ||
| 85 | examples: | 88 | examples: |
| 86 | - | | 89 | - | |
| 87 | #include <dt-bindings/clock/r8a779a0-cpg-mssr.h> | 90 | #include <dt-bindings/clock/r8a779a0-cpg-mssr.h> |
| 88 | #include <dt-bindings/power/r8a779a0-sysc.h> | 91 | #include <dt-bindings/power/r8a779a0-sysc.h> |
| 89 | 92 | ||
| 90 | dsi0: dsi-encoder@fed80000 { | 93 | dsi@fed80000 { |
| 91 | compatible = "renesas,r8a779a0-dsi-csi2-tx"; | 94 | compatible = "renesas,r8a779a0-dsi-csi2-tx"; |
| 92 | reg = <0xfed80000 0x10000>; | 95 | reg = <0xfed80000 0x10000>; |
| 93 | power-domains = <&sysc R8A779A0_PD_ALWAYS_ON>; | 96 | power-domains = <&sysc R8A779A0_PD_ALWAYS_ON>; |
| @@ -117,4 +120,51 @@ examples: · 23 unmodified lines | |||
| 117 | }; | 120 | }; |
| 118 | }; | 121 | }; |
| 119 | }; | 122 | }; |
| 123 | |||
| 124 | - | | ||
| 125 | #include <dt-bindings/clock/r8a779g0-cpg-mssr.h> | ||
| 126 | #include <dt-bindings/power/r8a779g0-sysc.h> | ||
| 127 | |||
| 128 | dsi@fed80000 { | ||
| 129 | #address-cells = <1>; | ||
| 130 | #size-cells = <0>; | ||
| 131 | compatible = "renesas,r8a779g0-dsi-csi2-tx"; | ||
| 132 | reg = <0xfed80000 0x10000>; | ||
| 133 | clocks = <&cpg CPG_MOD 415>, | ||
| 134 | <&cpg CPG_CORE R8A779G0_CLK_DSIEXT>, | ||
| 135 | <&cpg CPG_CORE R8A779G0_CLK_DSIREF>; | ||
| 136 | clock-names = "fck", "dsi", "pll"; | ||
| 137 | power-domains = <&sysc R8A779G0_PD_ALWAYS_ON>; | ||
| 138 | resets = <&cpg 415>; | ||
| 139 | |||
| 140 | ports { | ||
| 141 | #address-cells = <1>; | ||
| 142 | #size-cells = <0>; | ||
| 143 | |||
| 144 | port@0 { | ||
| 145 | reg = <0>; | ||
| 146 | }; | ||
| 147 | |||
| 148 | port@1 { | ||
| 149 | reg = <1>; | ||
| 150 | |||
| 151 | dsi0port1_out: endpoint { | ||
| 152 | remote-endpoint = <&panel_in>; | ||
| 153 | data-lanes = <1 2>; | ||
| 154 | }; | ||
| 155 | }; | ||
| 156 | }; | ||
| 157 | |||
| 158 | panel@0 { | ||
| 159 | reg = <0>; | ||
| 160 | compatible = "raspberrypi,dsi-7inch", "ilitek,ili9881c"; | ||
| 161 | power-supply = <&vcc_lcd_reg>; | ||
| 162 | |||
| 163 | port { | ||
| 164 | panel_in: endpoint { | ||
| 165 | remote-endpoint = <&dsi0port1_out>; | ||
| 166 | }; | ||
| 167 | }; | ||
| 168 | }; | ||
| 169 | }; | ||
| 120 | ... | 170 | ... |
| @@ -27,7 +27,9 @@ properties: · 26 unmodified lines | |||
| 27 | - const: adi,adv7123 | 27 | - const: adi,adv7123 |
| 28 | - enum: | 28 | - enum: |
| 29 | - adi,adv7123 | 29 | - adi,adv7123 |
| 30 | - asl-tek,cs5263 | ||
| 30 | - dumb-vga-dac | 31 | - dumb-vga-dac |
| 32 | - parade,ps185hdm | ||
| 31 | - radxa,ra620 | 33 | - radxa,ra620 |
| 32 | - realtek,rtd2171 | 34 | - realtek,rtd2171 |
| 33 | - ti,opa362 | 35 | - ti,opa362 |
| @@ -0,0 +1,69 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/imx/fsl,imx8mp-hdmi-pai.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Freescale i.MX8MP HDMI Parallel Audio Interface | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Shengjiu Wang <shengjiu.wang@nxp.com> | ||
| 11 | |||
| 12 | description: | ||
| 13 | The HDMI TX Parallel Audio Interface (HTX_PAI) is a bridge between the | ||
| 14 | Audio Subsystem to the HDMI TX Controller. | ||
| 15 | |||
| 16 | properties: | ||
| 17 | compatible: | ||
| 18 | const: fsl,imx8mp-hdmi-pai | ||
| 19 | |||
| 20 | reg: | ||
| 21 | maxItems: 1 | ||
| 22 | |||
| 23 | interrupts: | ||
| 24 | maxItems: 1 | ||
| 25 | |||
| 26 | clocks: | ||
| 27 | maxItems: 1 | ||
| 28 | |||
| 29 | clock-names: | ||
| 30 | const: apb | ||
| 31 | |||
| 32 | power-domains: | ||
| 33 | maxItems: 1 | ||
| 34 | |||
| 35 | port: | ||
| 36 | $ref: /schemas/graph.yaml#/properties/port | ||
| 37 | description: Output to the HDMI TX controller. | ||
| 38 | |||
| 39 | required: | ||
| 40 | - compatible | ||
| 41 | - reg | ||
| 42 | - interrupts | ||
| 43 | - clocks | ||
| 44 | - clock-names | ||
| 45 | - power-domains | ||
| 46 | - port | ||
| 47 | |||
| 48 | additionalProperties: false | ||
| 49 | |||
| 50 | examples: | ||
| 51 | - | | ||
| 52 | #include <dt-bindings/clock/imx8mp-clock.h> | ||
| 53 | #include <dt-bindings/power/imx8mp-power.h> | ||
| 54 | |||
| 55 | audio-bridge@32fc4800 { | ||
| 56 | compatible = "fsl,imx8mp-hdmi-pai"; | ||
| 57 | reg = <0x32fc4800 0x800>; | ||
| 58 | interrupt-parent = <&irqsteer_hdmi>; | ||
| 59 | interrupts = <14>; | ||
| 60 | clocks = <&clk IMX8MP_CLK_HDMI_APB>; | ||
| 61 | clock-names = "apb"; | ||
| 62 | power-domains = <&hdmi_blk_ctrl IMX8MP_HDMIBLK_PD_PAI>; | ||
| 63 | |||
| 64 | port { | ||
| 65 | pai_to_hdmi_tx: endpoint { | ||
| 66 | remote-endpoint = <&hdmi_tx_from_pai>; | ||
| 67 | }; | ||
| 68 | }; | ||
| 69 | }; | ||
| @@ -18,6 +18,7 @@ properties: · 17 unmodified lines | |||
| 18 | compatible: | 18 | compatible: |
| 19 | oneOf: | 19 | oneOf: |
| 20 | - enum: | 20 | - enum: |
| 21 | - qcom,glymur-dp | ||
| 21 | - qcom,sa8775p-dp | 22 | - qcom,sa8775p-dp |
| 22 | - qcom,sc7180-dp | 23 | - qcom,sc7180-dp |
| 23 | - qcom,sc7280-dp | 24 | - qcom,sc7280-dp |
| @@ -31,6 +32,11 @@ properties: · 7 unmodified lines | |||
| 31 | - qcom,sm8650-dp | 32 | - qcom,sm8650-dp |
| 32 | - qcom,x1e80100-dp | 33 | - qcom,x1e80100-dp |
| 33 | 34 | ||
| 35 | - items: | ||
| 36 | - enum: | ||
| 37 | - qcom,qcs8300-dp | ||
| 38 | - const: qcom,sa8775p-dp | ||
| 39 | |||
| 34 | - items: | 40 | - items: |
| 35 | - enum: | 41 | - enum: |
| 36 | - qcom,sm6350-dp | 42 | - qcom,sm6350-dp |
| @@ -53,6 +59,12 @@ properties: · 16 unmodified lines | |||
| 53 | - qcom,sm8550-dp | 59 | - qcom,sm8550-dp |
| 54 | - const: qcom,sm8350-dp | 60 | - const: qcom,sm8350-dp |
| 55 | 61 | ||
| 62 | - items: | ||
| 63 | - enum: | ||
| 64 | - qcom,sm6150-dp | ||
| 65 | - const: qcom,sm8150-dp | ||
| 66 | - const: qcom,sm8350-dp | ||
| 67 | |||
| 56 | - items: | 68 | - items: |
| 57 | - enum: | 69 | - enum: |
| 58 | - qcom,sm8750-dp | 70 | - qcom,sm8750-dp |
| @@ -195,9 +207,11 @@ allOf: · 136 unmodified lines | |||
| 195 | compatible: | 207 | compatible: |
| 196 | contains: | 208 | contains: |
| 197 | enum: | 209 | enum: |
| 210 | - qcom,glymur-dp | ||
| 198 | - qcom,sa8775p-dp | 211 | - qcom,sa8775p-dp |
| 199 | - qcom,x1e80100-dp | 212 | - qcom,x1e80100-dp |
| 200 | then: | 213 | then: |
| 214 | $ref: /schemas/sound/dai-common.yaml# | ||
| 201 | oneOf: | 215 | oneOf: |
| 202 | - required: | 216 | - required: |
| 203 | - aux-bus | 217 | - aux-bus |
| @@ -239,6 +253,7 @@ allOf: · 35 unmodified lines | |||
| 239 | enum: | 253 | enum: |
| 240 | # these platforms support 2 streams MST on some interfaces, | 254 | # these platforms support 2 streams MST on some interfaces, |
| 241 | # others are SST only | 255 | # others are SST only |
| 256 | - qcom,glymur-dp | ||
| 242 | - qcom,sc8280xp-dp | 257 | - qcom,sc8280xp-dp |
| 243 | - qcom,x1e80100-dp | 258 | - qcom,x1e80100-dp |
| 244 | then: | 259 | then: |
| @@ -295,7 +310,7 @@ allOf: · 50 unmodified lines | |||
| 295 | minItems: 6 | 310 | minItems: 6 |
| 296 | maxItems: 8 | 311 | maxItems: 8 |
| 297 | 312 | ||
| 298 | additionalProperties: false | 313 | unevaluatedProperties: false |
| 299 | 314 | ||
| 300 | examples: | 315 | examples: |
| 301 | - | | 316 | - | |
| @@ -21,7 +21,7 @@ properties: · 20 unmodified lines | |||
| 21 | compatible: | 21 | compatible: |
| 22 | oneOf: | 22 | oneOf: |
| 23 | - items: | 23 | - items: |
| 24 | - pattern: '^qcom,adreno-gmu-[67][0-9][0-9]\.[0-9]$' | 24 | - pattern: '^qcom,adreno-gmu-[6-8][0-9][0-9]\.[0-9]$' |
| 25 | - const: qcom,adreno-gmu | 25 | - const: qcom,adreno-gmu |
| 26 | - items: | 26 | - items: |
| 27 | - pattern: '^qcom,adreno-gmu-x[1-9][0-9][0-9]\.[0-9]$' | 27 | - pattern: '^qcom,adreno-gmu-x[1-9][0-9][0-9]\.[0-9]$' |
| @@ -299,6 +299,64 @@ allOf: · 271 unmodified lines | |||
| 299 | required: | 299 | required: |
| 300 | - qcom,qmp | 300 | - qcom,qmp |
| 301 | 301 | ||
| 302 | - if: | ||
| 303 | properties: | ||
| 304 | compatible: | ||
| 305 | contains: | ||
| 306 | const: qcom,adreno-gmu-840.1 | ||
| 307 | then: | ||
| 308 | properties: | ||
| 309 | reg: | ||
| 310 | items: | ||
| 311 | - description: Core GMU registers | ||
| 312 | reg-names: | ||
| 313 | items: | ||
| 314 | - const: gmu | ||
| 315 | clocks: | ||
| 316 | items: | ||
| 317 | - description: GPU AHB clock | ||
| 318 | - description: GMU clock | ||
| 319 | - description: GPU CX clock | ||
| 320 | - description: GPU MEMNOC clock | ||
| 321 | - description: GMU HUB clock | ||
| 322 | clock-names: | ||
| 323 | items: | ||
| 324 | - const: ahb | ||
| 325 | - const: gmu | ||
| 326 | - const: cxo | ||
| 327 | - const: memnoc | ||
| 328 | - const: hub | ||
| 329 | |||
| 330 | - if: | ||
| 331 | properties: | ||
| 332 | compatible: | ||
| 333 | contains: | ||
| 334 | const: qcom,adreno-gmu-x285.1 | ||
| 335 | then: | ||
| 336 | properties: | ||
| 337 | reg: | ||
| 338 | items: | ||
| 339 | - description: Core GMU registers | ||
| 340 | reg-names: | ||
| 341 | items: | ||
| 342 | - const: gmu | ||
| 343 | clocks: | ||
| 344 | items: | ||
| 345 | - description: GPU AHB clock | ||
| 346 | - description: GMU clock | ||
| 347 | - description: GPU CX clock | ||
| 348 | - description: GPU MEMNOC clock | ||
| 349 | - description: GMU HUB clock | ||
| 350 | - description: GMU RSCC HUB clock | ||
| 351 | clock-names: | ||
| 352 | items: | ||
| 353 | - const: ahb | ||
| 354 | - const: gmu | ||
| 355 | - const: cxo | ||
| 356 | - const: memnoc | ||
| 357 | - const: hub | ||
| 358 | - const: rscc | ||
| 359 | |||
| 302 | - if: | 360 | - if: |
| 303 | properties: | 361 | properties: |
| 304 | compatible: | 362 | compatible: |
| @@ -0,0 +1,264 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/msm/qcom,glymur-mdss.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Qualcomm Glymur Display MDSS | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Abel Vesa <abel.vesa@linaro.org> | ||
| 11 | |||
| 12 | description: | ||
| 13 | Glymur MSM Mobile Display Subsystem(MDSS), which encapsulates sub-blocks like | ||
| 14 | DPU display controller, DP interfaces, etc. | ||
| 15 | |||
| 16 | $ref: /schemas/display/msm/mdss-common.yaml# | ||
| 17 | |||
| 18 | properties: | ||
| 19 | compatible: | ||
| 20 | const: qcom,glymur-mdss | ||
| 21 | |||
| 22 | clocks: | ||
| 23 | items: | ||
| 24 | - description: Display AHB | ||
| 25 | - description: Display hf AXI | ||
| 26 | - description: Display core | ||
| 27 | |||
| 28 | iommus: | ||
| 29 | maxItems: 1 | ||
| 30 | |||
| 31 | interconnects: | ||
| 32 | items: | ||
| 33 | - description: Interconnect path from mdp0 port to the data bus | ||
| 34 | - description: Interconnect path from CPU to the reg bus | ||
| 35 | |||
| 36 | interconnect-names: | ||
| 37 | items: | ||
| 38 | - const: mdp0-mem | ||
| 39 | - const: cpu-cfg | ||
| 40 | |||
| 41 | patternProperties: | ||
| 42 | "^display-controller@[0-9a-f]+$": | ||
| 43 | type: object | ||
| 44 | additionalProperties: true | ||
| 45 | properties: | ||
| 46 | compatible: | ||
| 47 | const: qcom,glymur-dpu | ||
| 48 | |||
| 49 | "^displayport-controller@[0-9a-f]+$": | ||
| 50 | type: object | ||
| 51 | additionalProperties: true | ||
| 52 | properties: | ||
| 53 | compatible: | ||
| 54 | const: qcom,glymur-dp | ||
| 55 | |||
| 56 | "^phy@[0-9a-f]+$": | ||
| 57 | type: object | ||
| 58 | additionalProperties: true | ||
| 59 | properties: | ||
| 60 | compatible: | ||
| 61 | const: qcom,glymur-dp-phy | ||
| 62 | |||
| 63 | required: | ||
| 64 | - compatible | ||
| 65 | |||
| 66 | unevaluatedProperties: false | ||
| 67 | |||
| 68 | examples: | ||
| 69 | - | | ||
| 70 | #include <dt-bindings/clock/qcom,rpmh.h> | ||
| 71 | #include <dt-bindings/interconnect/qcom,icc.h> | ||
| 72 | #include <dt-bindings/interconnect/qcom,glymur-rpmh.h> | ||
| 73 | #include <dt-bindings/interrupt-controller/arm-gic.h> | ||
| 74 | #include <dt-bindings/phy/phy-qcom-qmp.h> | ||
| 75 | #include <dt-bindings/power/qcom,rpmhpd.h> | ||
| 76 | |||
| 77 | display-subsystem@ae00000 { | ||
| 78 | compatible = "qcom,glymur-mdss"; | ||
| 79 | reg = <0x0ae00000 0x1000>; | ||
| 80 | reg-names = "mdss"; | ||
| 81 | |||
| 82 | interrupts = <GIC_SPI 83 IRQ_TYPE_LEVEL_HIGH>; | ||
| 83 | |||
| 84 | clocks = <&dispcc_ahb_clk>, | ||
| 85 | <&gcc_disp_hf_axi_clk>, | ||
| 86 | <&dispcc_mdp_clk>; | ||
| 87 | clock-names = "bus", "nrt_bus", "core"; | ||
| 88 | |||
| 89 | interconnects = <&mmss_noc MASTER_MDP QCOM_ICC_TAG_ALWAYS | ||
| 90 | &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ALWAYS>, | ||
| 91 | <&hsc_noc MASTER_APPSS_PROC QCOM_ICC_TAG_ACTIVE_ONLY | ||
| 92 | &config_noc SLAVE_DISPLAY_CFG QCOM_ICC_TAG_ACTIVE_ONLY>; | ||
| 93 | interconnect-names = "mdp0-mem", | ||
| 94 | "cpu-cfg"; | ||
| 95 | |||
| 96 | resets = <&disp_cc_mdss_core_bcr>; | ||
| 97 | |||
| 98 | power-domains = <&mdss_gdsc>; | ||
| 99 | |||
| 100 | iommus = <&apps_smmu 0x1c00 0x2>; | ||
| 101 | |||
| 102 | interrupt-controller; | ||
| 103 | #interrupt-cells = <1>; | ||
| 104 | |||
| 105 | #address-cells = <1>; | ||
| 106 | #size-cells = <1>; | ||
| 107 | ranges; | ||
| 108 | |||
| 109 | display-controller@ae01000 { | ||
| 110 | compatible = "qcom,glymur-dpu"; | ||
| 111 | reg = <0x0ae01000 0x8f000>, | ||
| 112 | <0x0aeb0000 0x2008>; | ||
| 113 | reg-names = "mdp", "vbif"; | ||
| 114 | |||
| 115 | clocks = <&gcc_axi_clk>, | ||
| 116 | <&dispcc_ahb_clk>, | ||
| 117 | <&dispcc_mdp_lut_clk>, | ||
| 118 | <&dispcc_mdp_clk>, | ||
| 119 | <&dispcc_mdp_vsync_clk>; | ||
| 120 | clock-names = "nrt_bus", | ||
| 121 | "iface", | ||
| 122 | "lut", | ||
| 123 | "core", | ||
| 124 | "vsync"; | ||
| 125 | |||
| 126 | assigned-clocks = <&dispcc_mdp_vsync_clk>; | ||
| 127 | assigned-clock-rates = <19200000>; | ||
| 128 | |||
| 129 | operating-points-v2 = <&mdp_opp_table>; | ||
| 130 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 131 | |||
| 132 | interrupt-parent = <&mdss>; | ||
| 133 | interrupts = <0>; | ||
| 134 | |||
| 135 | ports { | ||
| 136 | #address-cells = <1>; | ||
| 137 | #size-cells = <0>; | ||
| 138 | |||
| 139 | port@0 { | ||
| 140 | reg = <0>; | ||
| 141 | dpu_intf1_out: endpoint { | ||
| 142 | remote-endpoint = <&dsi0_in>; | ||
| 143 | }; | ||
| 144 | }; | ||
| 145 | |||
| 146 | port@1 { | ||
| 147 | reg = <1>; | ||
| 148 | dpu_intf2_out: endpoint { | ||
| 149 | remote-endpoint = <&dsi1_in>; | ||
| 150 | }; | ||
| 151 | }; | ||
| 152 | }; | ||
| 153 | |||
| 154 | mdp_opp_table: opp-table { | ||
| 155 | compatible = "operating-points-v2"; | ||
| 156 | |||
| 157 | opp-200000000 { | ||
| 158 | opp-hz = /bits/ 64 <200000000>; | ||
| 159 | required-opps = <&rpmhpd_opp_low_svs>; | ||
| 160 | }; | ||
| 161 | |||
| 162 | opp-325000000 { | ||
| 163 | opp-hz = /bits/ 64 <325000000>; | ||
| 164 | required-opps = <&rpmhpd_opp_svs>; | ||
| 165 | }; | ||
| 166 | |||
| 167 | opp-375000000 { | ||
| 168 | opp-hz = /bits/ 64 <375000000>; | ||
| 169 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 170 | }; | ||
| 171 | |||
| 172 | opp-514000000 { | ||
| 173 | opp-hz = /bits/ 64 <514000000>; | ||
| 174 | required-opps = <&rpmhpd_opp_nom>; | ||
| 175 | }; | ||
| 176 | }; | ||
| 177 | }; | ||
| 178 | |||
| 179 | displayport-controller@ae90000 { | ||
| 180 | compatible = "qcom,glymur-dp"; | ||
| 181 | reg = <0xae90000 0x200>, | ||
| 182 | <0xae90200 0x200>, | ||
| 183 | <0xae90400 0x600>, | ||
| 184 | <0xae91000 0x400>, | ||
| 185 | <0xae91400 0x400>; | ||
| 186 | |||
| 187 | interrupt-parent = <&mdss>; | ||
| 188 | interrupts = <12>; | ||
| 189 | |||
| 190 | clocks = <&dispcc_mdss_ahb_clk>, | ||
| 191 | <&dispcc_dptx0_aux_clk>, | ||
| 192 | <&dispcc_dptx0_link_clk>, | ||
| 193 | <&dispcc_dptx0_link_intf_clk>, | ||
| 194 | <&dispcc_dptx0_pixel0_clk>, | ||
| 195 | <&dispcc_dptx0_pixel1_clk>; | ||
| 196 | clock-names = "core_iface", | ||
| 197 | "core_aux", | ||
| 198 | "ctrl_link", | ||
| 199 | "ctrl_link_iface", | ||
| 200 | "stream_pixel", | ||
| 201 | "stream_1_pixel"; | ||
| 202 | |||
| 203 | assigned-clocks = <&dispcc_mdss_dptx0_link_clk_src>, | ||
| 204 | <&dispcc_mdss_dptx0_pixel0_clk_src>, | ||
| 205 | <&dispcc_mdss_dptx0_pixel1_clk_src>; | ||
| 206 | assigned-clock-parents = <&usb_1_ss0_qmpphy QMP_USB43DP_DP_LINK_CLK>, | ||
| 207 | <&usb_1_ss0_qmpphy QMP_USB43DP_DP_VCO_DIV_CLK>, | ||
| 208 | <&usb_1_ss0_qmpphy QMP_USB43DP_DP_VCO_DIV_CLK>; | ||
| 209 | |||
| 210 | operating-points-v2 = <&mdss_dp0_opp_table>; | ||
| 211 | |||
| 212 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 213 | |||
| 214 | phys = <&usb_1_ss0_qmpphy QMP_USB43DP_DP_PHY>; | ||
| 215 | phy-names = "dp"; | ||
| 216 | |||
| 217 | #sound-dai-cells = <0>; | ||
| 218 | |||
| 219 | ports { | ||
| 220 | #address-cells = <1>; | ||
| 221 | #size-cells = <0>; | ||
| 222 | |||
| 223 | port@0 { | ||
| 224 | reg = <0>; | ||
| 225 | |||
| 226 | mdss_dp0_in: endpoint { | ||
| 227 | remote-endpoint = <&mdss_intf0_out>; | ||
| 228 | }; | ||
| 229 | }; | ||
| 230 | |||
| 231 | port@1 { | ||
| 232 | reg = <1>; | ||
| 233 | |||
| 234 | mdss_dp0_out: endpoint { | ||
| 235 | }; | ||
| 236 | }; | ||
| 237 | }; | ||
| 238 | |||
| 239 | mdss_dp0_opp_table: opp-table { | ||
| 240 | compatible = "operating-points-v2"; | ||
| 241 | |||
| 242 | opp-160000000 { | ||
| 243 | opp-hz = /bits/ 64 <160000000>; | ||
| 244 | required-opps = <&rpmhpd_opp_low_svs>; | ||
| 245 | }; | ||
| 246 | |||
| 247 | opp-270000000 { | ||
| 248 | opp-hz = /bits/ 64 <270000000>; | ||
| 249 | required-opps = <&rpmhpd_opp_svs>; | ||
| 250 | }; | ||
| 251 | |||
| 252 | opp-540000000 { | ||
| 253 | opp-hz = /bits/ 64 <540000000>; | ||
| 254 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 255 | }; | ||
| 256 | |||
| 257 | opp-810000000 { | ||
| 258 | opp-hz = /bits/ 64 <810000000>; | ||
| 259 | required-opps = <&rpmhpd_opp_nom>; | ||
| 260 | }; | ||
| 261 | }; | ||
| 262 | }; | ||
| 263 | }; | ||
| 264 | ... | ||
| @@ -0,0 +1,286 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/msm/qcom,qcs8300-mdss.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Qualcomm Technologies, Inc. QCS8300 Display MDSS | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Yongxing Mou <yongxing.mou@oss.qualcomm.com> | ||
| 11 | |||
| 12 | description: | ||
| 13 | QCS8300 MSM Mobile Display Subsystem(MDSS), which encapsulates sub-blocks like | ||
| 14 | DPU display controller, DP interfaces and EDP etc. | ||
| 15 | |||
| 16 | $ref: /schemas/display/msm/mdss-common.yaml# | ||
| 17 | |||
| 18 | properties: | ||
| 19 | compatible: | ||
| 20 | const: qcom,qcs8300-mdss | ||
| 21 | |||
| 22 | clocks: | ||
| 23 | items: | ||
| 24 | - description: Display AHB | ||
| 25 | - description: Display hf AXI | ||
| 26 | - description: Display core | ||
| 27 | |||
| 28 | iommus: | ||
| 29 | maxItems: 1 | ||
| 30 | |||
| 31 | interconnects: | ||
| 32 | maxItems: 3 | ||
| 33 | |||
| 34 | interconnect-names: | ||
| 35 | maxItems: 3 | ||
| 36 | |||
| 37 | patternProperties: | ||
| 38 | "^display-controller@[0-9a-f]+$": | ||
| 39 | type: object | ||
| 40 | additionalProperties: true | ||
| 41 | |||
| 42 | properties: | ||
| 43 | compatible: | ||
| 44 | contains: | ||
| 45 | const: qcom,qcs8300-dpu | ||
| 46 | |||
| 47 | "^displayport-controller@[0-9a-f]+$": | ||
| 48 | type: object | ||
| 49 | additionalProperties: true | ||
| 50 | |||
| 51 | properties: | ||
| 52 | compatible: | ||
| 53 | contains: | ||
| 54 | const: qcom,qcs8300-dp | ||
| 55 | |||
| 56 | "^phy@[0-9a-f]+$": | ||
| 57 | type: object | ||
| 58 | additionalProperties: true | ||
| 59 | properties: | ||
| 60 | compatible: | ||
| 61 | contains: | ||
| 62 | const: qcom,qcs8300-edp-phy | ||
| 63 | |||
| 64 | required: | ||
| 65 | - compatible | ||
| 66 | |||
| 67 | unevaluatedProperties: false | ||
| 68 | |||
| 69 | examples: | ||
| 70 | - | | ||
| 71 | #include <dt-bindings/interconnect/qcom,icc.h> | ||
| 72 | #include <dt-bindings/interrupt-controller/arm-gic.h> | ||
| 73 | #include <dt-bindings/clock/qcom,qcs8300-gcc.h> | ||
| 74 | #include <dt-bindings/clock/qcom,sa8775p-dispcc.h> | ||
| 75 | #include <dt-bindings/interconnect/qcom,qcs8300-rpmh.h> | ||
| 76 | #include <dt-bindings/power/qcom,rpmhpd.h> | ||
| 77 | #include <dt-bindings/power/qcom-rpmpd.h> | ||
| 78 | |||
| 79 | mdss: display-subsystem@ae00000 { | ||
| 80 | compatible = "qcom,qcs8300-mdss"; | ||
| 81 | reg = <0x0ae00000 0x1000>; | ||
| 82 | reg-names = "mdss"; | ||
| 83 | |||
| 84 | interconnects = <&mmss_noc MASTER_MDP0 QCOM_ICC_TAG_ACTIVE_ONLY | ||
| 85 | &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ACTIVE_ONLY>, | ||
| 86 | <&mmss_noc MASTER_MDP1 QCOM_ICC_TAG_ACTIVE_ONLY | ||
| 87 | &mc_virt SLAVE_EBI1 QCOM_ICC_TAG_ACTIVE_ONLY>, | ||
| 88 | <&gem_noc MASTER_APPSS_PROC QCOM_ICC_TAG_ACTIVE_ONLY | ||
| 89 | &config_noc SLAVE_DISPLAY_CFG QCOM_ICC_TAG_ACTIVE_ONLY>; | ||
| 90 | interconnect-names = "mdp0-mem", | ||
| 91 | "mdp1-mem", | ||
| 92 | "cpu-cfg"; | ||
| 93 | |||
| 94 | resets = <&dispcc_core_bcr>; | ||
| 95 | power-domains = <&dispcc_gdsc>; | ||
| 96 | |||
| 97 | clocks = <&dispcc_ahb_clk>, | ||
| 98 | <&gcc GCC_DISP_HF_AXI_CLK>, | ||
| 99 | <&dispcc_mdp_clk>; | ||
| 100 | |||
| 101 | interrupts = <GIC_SPI 92 IRQ_TYPE_LEVEL_HIGH>; | ||
| 102 | interrupt-controller; | ||
| 103 | #interrupt-cells = <1>; | ||
| 104 | |||
| 105 | iommus = <&apps_smmu 0x1000 0x402>; | ||
| 106 | |||
| 107 | #address-cells = <1>; | ||
| 108 | #size-cells = <1>; | ||
| 109 | ranges; | ||
| 110 | |||
| 111 | display-controller@ae01000 { | ||
| 112 | compatible = "qcom,qcs8300-dpu", "qcom,sa8775p-dpu"; | ||
| 113 | reg = <0x0ae01000 0x8f000>, | ||
| 114 | <0x0aeb0000 0x2008>; | ||
| 115 | reg-names = "mdp", "vbif"; | ||
| 116 | |||
| 117 | clocks = <&gcc GCC_DISP_HF_AXI_CLK>, | ||
| 118 | <&dispcc0 MDSS_DISP_CC_MDSS_AHB_CLK>, | ||
| 119 | <&dispcc0 MDSS_DISP_CC_MDSS_MDP_LUT_CLK>, | ||
| 120 | <&dispcc0 MDSS_DISP_CC_MDSS_MDP_CLK>, | ||
| 121 | <&dispcc0 MDSS_DISP_CC_MDSS_VSYNC_CLK>; | ||
| 122 | clock-names = "nrt_bus", | ||
| 123 | "iface", | ||
| 124 | "lut", | ||
| 125 | "core", | ||
| 126 | "vsync"; | ||
| 127 | |||
| 128 | assigned-clocks = <&dispcc0 MDSS_DISP_CC_MDSS_VSYNC_CLK>; | ||
| 129 | assigned-clock-rates = <19200000>; | ||
| 130 | operating-points-v2 = <&mdp_opp_table>; | ||
| 131 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 132 | |||
| 133 | interrupt-parent = <&mdss>; | ||
| 134 | interrupts = <0>; | ||
| 135 | ports { | ||
| 136 | #address-cells = <1>; | ||
| 137 | #size-cells = <0>; | ||
| 138 | port@0 { | ||
| 139 | reg = <0>; | ||
| 140 | |||
| 141 | dpu_intf0_out: endpoint { | ||
| 142 | remote-endpoint = <&mdss_dp0_in>; | ||
| 143 | }; | ||
| 144 | }; | ||
| 145 | }; | ||
| 146 | |||
| 147 | mdp_opp_table: opp-table { | ||
| 148 | compatible = "operating-points-v2"; | ||
| 149 | |||
| 150 | opp-375000000 { | ||
| 151 | opp-hz = /bits/ 64 <375000000>; | ||
| 152 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 153 | }; | ||
| 154 | |||
| 155 | opp-500000000 { | ||
| 156 | opp-hz = /bits/ 64 <500000000>; | ||
| 157 | required-opps = <&rpmhpd_opp_nom>; | ||
| 158 | }; | ||
| 159 | |||
| 160 | opp-575000000 { | ||
| 161 | opp-hz = /bits/ 64 <575000000>; | ||
| 162 | required-opps = <&rpmhpd_opp_turbo>; | ||
| 163 | }; | ||
| 164 | |||
| 165 | opp-650000000 { | ||
| 166 | opp-hz = /bits/ 64 <650000000>; | ||
| 167 | required-opps = <&rpmhpd_opp_turbo_l1>; | ||
| 168 | }; | ||
| 169 | }; | ||
| 170 | }; | ||
| 171 | |||
| 172 | mdss_dp0_phy: phy@aec2a00 { | ||
| 173 | compatible = "qcom,qcs8300-edp-phy", "qcom,sa8775p-edp-phy"; | ||
| 174 | |||
| 175 | reg = <0x0aec2a00 0x200>, | ||
| 176 | <0x0aec2200 0xd0>, | ||
| 177 | <0x0aec2600 0xd0>, | ||
| 178 | <0x0aec2000 0x1c8>; | ||
| 179 | |||
| 180 | clocks = <&dispcc MDSS_DISP_CC_MDSS_DPTX0_AUX_CLK>, | ||
| 181 | <&dispcc MDSS_DISP_CC_MDSS_AHB_CLK>; | ||
| 182 | clock-names = "aux", | ||
| 183 | "cfg_ahb"; | ||
| 184 | |||
| 185 | #clock-cells = <1>; | ||
| 186 | #phy-cells = <0>; | ||
| 187 | |||
| 188 | vdda-phy-supply = <&vreg_l1c>; | ||
| 189 | vdda-pll-supply = <&vreg_l4a>; | ||
| 190 | }; | ||
| 191 | |||
| 192 | displayport-controller@af54000 { | ||
| 193 | compatible = "qcom,qcs8300-dp", "qcom,sa8775p-dp"; | ||
| 194 | |||
| 195 | pinctrl-0 = <&dp_hot_plug_det>; | ||
| 196 | pinctrl-names = "default"; | ||
| 197 | |||
| 198 | reg = <0xaf54000 0x104>, | ||
| 199 | <0xaf54200 0x0c0>, | ||
| 200 | <0xaf55000 0x770>, | ||
| 201 | <0xaf56000 0x09c>, | ||
| 202 | <0xaf57000 0x09c>, | ||
| 203 | <0xaf58000 0x09c>, | ||
| 204 | <0xaf59000 0x09c>, | ||
| 205 | <0xaf5a000 0x23c>, | ||
| 206 | <0xaf5b000 0x23c>; | ||
| 207 | |||
| 208 | interrupt-parent = <&mdss>; | ||
| 209 | interrupts = <12>; | ||
| 210 | clocks = <&dispcc0 MDSS_DISP_CC_MDSS_AHB_CLK>, | ||
| 211 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_AUX_CLK>, | ||
| 212 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_LINK_CLK>, | ||
| 213 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_LINK_INTF_CLK>, | ||
| 214 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL0_CLK>, | ||
| 215 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL1_CLK>, | ||
| 216 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL2_CLK>, | ||
| 217 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL3_CLK>; | ||
| 218 | clock-names = "core_iface", | ||
| 219 | "core_aux", | ||
| 220 | "ctrl_link", | ||
| 221 | "ctrl_link_iface", | ||
| 222 | "stream_pixel", | ||
| 223 | "stream_1_pixel", | ||
| 224 | "stream_2_pixel", | ||
| 225 | "stream_3_pixel"; | ||
| 226 | assigned-clocks = <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_LINK_CLK_SRC>, | ||
| 227 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL0_CLK_SRC>, | ||
| 228 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL1_CLK_SRC>, | ||
| 229 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL2_CLK_SRC>, | ||
| 230 | <&dispcc0 MDSS_DISP_CC_MDSS_DPTX0_PIXEL3_CLK_SRC>; | ||
| 231 | assigned-clock-parents = <&mdss_dp0_phy 0>, | ||
| 232 | <&mdss_dp0_phy 1>, | ||
| 233 | <&mdss_dp0_phy 1>, | ||
| 234 | <&mdss_dp0_phy 1>; | ||
| 235 | phys = <&mdss_dp0_phy>; | ||
| 236 | phy-names = "dp"; | ||
| 237 | operating-points-v2 = <&dp_opp_table>; | ||
| 238 | power-domains = <&rpmhpd RPMHPD_MMCX>; | ||
| 239 | |||
| 240 | #sound-dai-cells = <0>; | ||
| 241 | |||
| 242 | ports { | ||
| 243 | #address-cells = <1>; | ||
| 244 | #size-cells = <0>; | ||
| 245 | |||
| 246 | port@0 { | ||
| 247 | reg = <0>; | ||
| 248 | |||
| 249 | mdss_dp0_in: endpoint { | ||
| 250 | remote-endpoint = <&dpu_intf0_out>; | ||
| 251 | }; | ||
| 252 | }; | ||
| 253 | |||
| 254 | port@1 { | ||
| 255 | reg = <1>; | ||
| 256 | |||
| 257 | mdss_dp_out: endpoint { }; | ||
| 258 | }; | ||
| 259 | }; | ||
| 260 | |||
| 261 | dp_opp_table: opp-table { | ||
| 262 | compatible = "operating-points-v2"; | ||
| 263 | |||
| 264 | opp-160000000 { | ||
| 265 | opp-hz = /bits/ 64 <160000000>; | ||
| 266 | required-opps = <&rpmhpd_opp_low_svs>; | ||
| 267 | }; | ||
| 268 | |||
| 269 | opp-270000000 { | ||
| 270 | opp-hz = /bits/ 64 <270000000>; | ||
| 271 | required-opps = <&rpmhpd_opp_svs>; | ||
| 272 | }; | ||
| 273 | |||
| 274 | opp-540000000 { | ||
| 275 | opp-hz = /bits/ 64 <540000000>; | ||
| 276 | required-opps = <&rpmhpd_opp_svs_l1>; | ||
| 277 | }; | ||
| 278 | |||
| 279 | opp-810000000 { | ||
| 280 | opp-hz = /bits/ 64 <810000000>; | ||
| 281 | required-opps = <&rpmhpd_opp_nom>; | ||
| 282 | }; | ||
| 283 | }; | ||
| 284 | }; | ||
| 285 | }; | ||
| 286 | ... | ||
| @@ -51,6 +51,14 @@ patternProperties: · 50 unmodified lines | |||
| 51 | compatible: | 51 | compatible: |
| 52 | const: qcom,sm6150-dpu | 52 | const: qcom,sm6150-dpu |
| 53 | 53 | ||
| 54 | "^displayport-controller@[0-9a-f]+$": | ||
| 55 | type: object | ||
| 56 | additionalProperties: true | ||
| 57 | properties: | ||
| 58 | compatible: | ||
| 59 | contains: | ||
| 60 | const: qcom,sm6150-dp | ||
| 61 | |||
| 54 | "^dsi@[0-9a-f]+$": | 62 | "^dsi@[0-9a-f]+$": |
| 55 | type: object | 63 | type: object |
| 56 | additionalProperties: true | 64 | additionalProperties: true |
| @@ -130,35 +138,37 @@ examples: · 73 unmodified lines | |||
| 130 | #size-cells = <0>; | 138 | #size-cells = <0>; |
| 131 | 139 | ||
| 132 | port@0 { | 140 | port@0 { |
| 133 | reg = <0>; | 141 | reg = <0>; |
| 134 | dpu_intf0_out: endpoint { | 142 | |
| 135 | }; | 143 | dpu_intf0_out: endpoint { |
| 144 | }; | ||
| 136 | }; | 145 | }; |
| 137 | 146 | ||
| 138 | port@1 { | 147 | port@1 { |
| 139 | reg = <1>; | 148 | reg = <1>; |
| 140 | dpu_intf1_out: endpoint { | 149 | |
| 141 | remote-endpoint = <&mdss_dsi0_in>; | 150 | dpu_intf1_out: endpoint { |
| 142 | }; | 151 | remote-endpoint = <&mdss_dsi0_in>; |
| 152 | }; | ||
| 143 | }; | 153 | }; |
| 144 | }; | 154 | }; |
| 145 | 155 | ||
| 146 | mdp_opp_table: opp-table { | 156 | mdp_opp_table: opp-table { |
| 147 | compatible = "operating-points-v2"; | 157 | compatible = "operating-points-v2"; |
| 148 | 158 | ||
| 149 | opp-19200000 { | 159 | opp-192000000 { |
| 150 | opp-hz = /bits/ 64 <19200000>; | 160 | opp-hz = /bits/ 64 <192000000>; |
| 151 | required-opps = <&rpmhpd_opp_low_svs>; | 161 | required-opps = <&rpmhpd_opp_low_svs>; |
| 152 | }; | 162 | }; |
| 153 | 163 | ||
| 154 | opp-25600000 { | 164 | opp-256000000 { |
| 155 | opp-hz = /bits/ 64 <25600000>; | 165 | opp-hz = /bits/ 64 <256000000>; |
| 156 | required-opps = <&rpmhpd_opp_svs>; | 166 | required-opps = <&rpmhpd_opp_svs>; |
| 157 | }; | 167 | }; |
| 158 | 168 | ||
| 159 | opp-307200000 { | 169 | opp-307200000 { |
| 160 | opp-hz = /bits/ 64 <307200000>; | 170 | opp-hz = /bits/ 64 <307200000>; |
| 161 | required-opps = <&rpmhpd_opp_nom>; | 171 | required-opps = <&rpmhpd_opp_nom>; |
| 162 | }; | 172 | }; |
| 163 | }; | 173 | }; |
| 164 | }; | 174 | }; |
| @@ -13,11 +13,17 @@ $ref: /schemas/display/msm/dpu-common.yaml# · 12 unmodified lines | |||
| 13 | 13 | ||
| 14 | properties: | 14 | properties: |
| 15 | compatible: | 15 | compatible: |
| 16 | enum: | 16 | oneOf: |
| 17 | - qcom,sa8775p-dpu | 17 | - enum: |
| 18 | - qcom,sm8650-dpu | 18 | - qcom,glymur-dpu |
| 19 | - qcom,sm8750-dpu | 19 | - qcom,sa8775p-dpu |
| 20 | - qcom,x1e80100-dpu | 20 | - qcom,sm8650-dpu |
| 21 | - qcom,sm8750-dpu | ||
| 22 | - qcom,x1e80100-dpu | ||
| 23 | - items: | ||
| 24 | - enum: | ||
| 25 | - qcom,qcs8300-dpu | ||
| 26 | - const: qcom,sa8775p-dpu | ||
| 21 | 27 | ||
| 22 | reg: | 28 | reg: |
| 23 | items: | 29 | items: |
| @@ -0,0 +1,68 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/ilitek,il79900a.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Ilitek IL79900a based MIPI-DSI panels | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Langyan Ye <yelangyan@huaqin.corp-partner.google.com> | ||
| 11 | |||
| 12 | allOf: | ||
| 13 | - $ref: panel-common.yaml# | ||
| 14 | |||
| 15 | properties: | ||
| 16 | compatible: | ||
| 17 | items: | ||
| 18 | - enum: | ||
| 19 | - tianma,tl121bvms07-00 | ||
| 20 | - const: ilitek,il79900a | ||
| 21 | |||
| 22 | reg: | ||
| 23 | maxItems: 1 | ||
| 24 | description: DSI virtual channel used by the panel | ||
| 25 | |||
| 26 | enable-gpios: | ||
| 27 | maxItems: 1 | ||
| 28 | description: GPIO specifier for the enable pin | ||
| 29 | |||
| 30 | avdd-supply: | ||
| 31 | description: Positive analog voltage supply (AVDD) | ||
| 32 | |||
| 33 | avee-supply: | ||
| 34 | description: Negative analog voltage supply (AVEE) | ||
| 35 | |||
| 36 | pp1800-supply: | ||
| 37 | description: 1.8V logic voltage supply | ||
| 38 | |||
| 39 | backlight: true | ||
| 40 | |||
| 41 | required: | ||
| 42 | - compatible | ||
| 43 | - reg | ||
| 44 | - enable-gpios | ||
| 45 | - avdd-supply | ||
| 46 | - avee-supply | ||
| 47 | - pp1800-supply | ||
| 48 | |||
| 49 | additionalProperties: false | ||
| 50 | |||
| 51 | examples: | ||
| 52 | - | | ||
| 53 | dsi { | ||
| 54 | #address-cells = <1>; | ||
| 55 | #size-cells = <0>; | ||
| 56 | |||
| 57 | panel@0 { | ||
| 58 | compatible = "tianma,tl121bvms07-00", "ilitek,il79900a"; | ||
| 59 | reg = <0>; | ||
| 60 | enable-gpios = <&pio 25 0>; | ||
| 61 | avdd-supply = <®_avdd>; | ||
| 62 | avee-supply = <®_avee>; | ||
| 63 | pp1800-supply = <®_pp1800>; | ||
| 64 | backlight = <&backlight>; | ||
| 65 | }; | ||
| 66 | }; | ||
| 67 | |||
| 68 | ... | ||
| @@ -20,9 +20,11 @@ properties: · 19 unmodified lines | |||
| 20 | - bananapi,lhr050h41 | 20 | - bananapi,lhr050h41 |
| 21 | - bestar,bsd1218-a101kl68 | 21 | - bestar,bsd1218-a101kl68 |
| 22 | - feixin,k101-im2byl02 | 22 | - feixin,k101-im2byl02 |
| 23 | - raspberrypi,dsi-5inch | ||
| 23 | - raspberrypi,dsi-7inch | 24 | - raspberrypi,dsi-7inch |
| 24 | - startek,kd050hdfia020 | 25 | - startek,kd050hdfia020 |
| 25 | - tdo,tl050hdv35 | 26 | - tdo,tl050hdv35 |
| 27 | - wanchanglong,w552946aaa | ||
| 26 | - wanchanglong,w552946aba | 28 | - wanchanglong,w552946aba |
| 27 | - const: ilitek,ili9881c | 29 | - const: ilitek,ili9881c |
| 28 | 30 | ||
| @@ -30,6 +32,7 @@ properties: · 1 unmodified line | |||
| 30 | maxItems: 1 | 32 | maxItems: 1 |
| 31 | 33 | ||
| 32 | backlight: true | 34 | backlight: true |
| 35 | port: true | ||
| 33 | power-supply: true | 36 | power-supply: true |
| 34 | reset-gpios: true | 37 | reset-gpios: true |
| 35 | rotation: true | 38 | rotation: true |
| @@ -0,0 +1,60 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/lg,ld070wx3-sl01.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: LG Corporation 7" WXGA TFT LCD panel | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Svyatoslav Ryhel <clamor95@gmail.com> | ||
| 11 | |||
| 12 | allOf: | ||
| 13 | - $ref: panel-common.yaml# | ||
| 14 | |||
| 15 | properties: | ||
| 16 | compatible: | ||
| 17 | items: | ||
| 18 | - const: lg,ld070wx3-sl01 | ||
| 19 | |||
| 20 | reg: | ||
| 21 | maxItems: 1 | ||
| 22 | |||
| 23 | vdd-supply: true | ||
| 24 | vcc-supply: true | ||
| 25 | |||
| 26 | backlight: true | ||
| 27 | port: true | ||
| 28 | |||
| 29 | required: | ||
| 30 | - compatible | ||
| 31 | - vdd-supply | ||
| 32 | - vcc-supply | ||
| 33 | |||
| 34 | additionalProperties: false | ||
| 35 | |||
| 36 | examples: | ||
| 37 | - | | ||
| 38 | #include <dt-bindings/gpio/gpio.h> | ||
| 39 | |||
| 40 | dsi { | ||
| 41 | #address-cells = <1>; | ||
| 42 | #size-cells = <0>; | ||
| 43 | |||
| 44 | panel@0 { | ||
| 45 | compatible = "lg,ld070wx3-sl01"; | ||
| 46 | reg = <0>; | ||
| 47 | |||
| 48 | vdd-supply = <&vdd_3v3_lcd>; | ||
| 49 | vcc-supply = <&vcc_1v8_lcd>; | ||
| 50 | |||
| 51 | backlight = <&backlight>; | ||
| 52 | |||
| 53 | port { | ||
| 54 | endpoint { | ||
| 55 | remote-endpoint = <&dsi0_out>; | ||
| 56 | }; | ||
| 57 | }; | ||
| 58 | }; | ||
| 59 | }; | ||
| 60 | ... | ||
| @@ -59,6 +59,8 @@ properties: · 58 unmodified lines | |||
| 59 | # Jenson Display BL-JT60050-01A 7" WSVGA (1024x600) color TFT LCD LVDS panel | 59 | # Jenson Display BL-JT60050-01A 7" WSVGA (1024x600) color TFT LCD LVDS panel |
| 60 | - jenson,bl-jt60050-01a | 60 | - jenson,bl-jt60050-01a |
| 61 | - tbs,a711-panel | 61 | - tbs,a711-panel |
| 62 | # Winstar WF70A8SYJHLNGA 7" WSVGA (1024x600) color TFT LCD LVDS panel | ||
| 63 | - winstar,wf70a8syjhlnga | ||
| 62 | 64 | ||
| 63 | - const: panel-lvds | 65 | - const: panel-lvds |
| 64 | 66 | ||
| @@ -19,6 +19,9 @@ description: | · 18 unmodified lines | |||
| 19 | 19 | ||
| 20 | If the panel is more advanced a dedicated binding file is required. | 20 | If the panel is more advanced a dedicated binding file is required. |
| 21 | 21 | ||
| 22 | allOf: | ||
| 23 | - $ref: panel-common.yaml# | ||
| 24 | |||
| 22 | properties: | 25 | properties: |
| 23 | 26 | ||
| 24 | compatible: | 27 | compatible: |
| @@ -42,8 +45,6 @@ properties: · 17 unmodified lines | |||
| 42 | - kingdisplay,kd097d04 | 45 | - kingdisplay,kd097d04 |
| 43 | # LG ACX467AKM-7 4.95" 1080×1920 LCD Panel | 46 | # LG ACX467AKM-7 4.95" 1080×1920 LCD Panel |
| 44 | - lg,acx467akm-7 | 47 | - lg,acx467akm-7 |
| 45 | # LG Corporation 7" WXGA TFT LCD panel | ||
| 46 | - lg,ld070wx3-sl01 | ||
| 47 | # LG Corporation 5" HD TFT LCD panel | 48 | # LG Corporation 5" HD TFT LCD panel |
| 48 | - lg,lh500wx1-sd03 | 49 | - lg,lh500wx1-sd03 |
| 49 | # Lincoln LCD197 5" 1080x1920 LCD panel | 50 | # Lincoln LCD197 5" 1080x1920 LCD panel |
| @@ -56,10 +57,6 @@ properties: · 6 unmodified lines | |||
| 56 | - panasonic,vvx10f034n00 | 57 | - panasonic,vvx10f034n00 |
| 57 | # Samsung s6e3fa7 1080x2220 based AMS559NK06 AMOLED panel | 58 | # Samsung s6e3fa7 1080x2220 based AMS559NK06 AMOLED panel |
| 58 | - samsung,s6e3fa7-ams559nk06 | 59 | - samsung,s6e3fa7-ams559nk06 |
| 59 | # Samsung s6e3fc2x01 1080x2340 AMOLED panel | ||
| 60 | - samsung,s6e3fc2x01 | ||
| 61 | # Samsung sofef00 1080x2280 AMOLED panel | ||
| 62 | - samsung,sofef00 | ||
| 63 | # Shangai Top Display Optoelectronics 7" TL070WSH30 1024x600 TFT LCD panel | 60 | # Shangai Top Display Optoelectronics 7" TL070WSH30 1024x600 TFT LCD panel |
| 64 | - tdo,tl070wsh30 | 61 | - tdo,tl070wsh30 |
| 65 | 62 | ||
| @@ -72,31 +69,12 @@ properties: · 6 unmodified lines | |||
| 72 | reset-gpios: true | 69 | reset-gpios: true |
| 73 | port: true | 70 | port: true |
| 74 | power-supply: true | 71 | power-supply: true |
| 75 | vddio-supply: true | ||
| 76 | |||
| 77 | allOf: | ||
| 78 | - $ref: panel-common.yaml# | ||
| 79 | - if: | ||
| 80 | properties: | ||
| 81 | compatible: | ||
| 82 | enum: | ||
| 83 | - samsung,s6e3fc2x01 | ||
| 84 | - samsung,sofef00 | ||
| 85 | then: | ||
| 86 | properties: | ||
| 87 | power-supply: false | ||
| 88 | required: | ||
| 89 | - vddio-supply | ||
| 90 | else: | ||
| 91 | properties: | ||
| 92 | vddio-supply: false | ||
| 93 | required: | ||
| 94 | - power-supply | ||
| 95 | 72 | ||
| 96 | additionalProperties: false | 73 | additionalProperties: false |
| 97 | 74 | ||
| 98 | required: | 75 | required: |
| 99 | - compatible | 76 | - compatible |
| 77 | - power-supply | ||
| 100 | - reg | 78 | - reg |
| 101 | 79 | ||
| 102 | examples: | 80 | examples: |
| @@ -184,6 +184,8 @@ properties: · 183 unmodified lines | |||
| 184 | - innolux,n156bge-l21 | 184 | - innolux,n156bge-l21 |
| 185 | # Innolux Corporation 7.0" WSVGA (1024x600) TFT LCD panel | 185 | # Innolux Corporation 7.0" WSVGA (1024x600) TFT LCD panel |
| 186 | - innolux,zj070na-01p | 186 | - innolux,zj070na-01p |
| 187 | # JuTouch Technology Co.. 10" JT101TM023 WXGA (1280 x 800) LVDS panel | ||
| 188 | - jutouch,jt101tm023 | ||
| 187 | # Kaohsiung Opto-Electronics Inc. 5.7" QVGA (320 x 240) TFT LCD panel | 189 | # Kaohsiung Opto-Electronics Inc. 5.7" QVGA (320 x 240) TFT LCD panel |
| 188 | - koe,tx14d24vm1bpa | 190 | - koe,tx14d24vm1bpa |
| 189 | # Kaohsiung Opto-Electronics. TX31D200VM0BAA 12.3" HSXGA LVDS panel | 191 | # Kaohsiung Opto-Electronics. TX31D200VM0BAA 12.3" HSXGA LVDS panel |
| @@ -268,6 +270,8 @@ properties: · 78 unmodified lines | |||
| 268 | - qiaodian,qd43003c0-40 | 270 | - qiaodian,qd43003c0-40 |
| 269 | # Shenzhen QiShenglong Industrialist Co., Ltd. Gopher 2b 4.3" 480(RGB)x272 TFT LCD panel | 271 | # Shenzhen QiShenglong Industrialist Co., Ltd. Gopher 2b 4.3" 480(RGB)x272 TFT LCD panel |
| 270 | - qishenglong,gopher2b-lcd | 272 | - qishenglong,gopher2b-lcd |
| 273 | # Raystar Optronics, Inc. RFF500F-AWH-DNN 5.0" TFT 840x480 | ||
| 274 | - raystar,rff500f-awh-dnn | ||
| 271 | # Rocktech Displays Ltd. RK101II01D-CT 10.1" TFT 1280x800 | 275 | # Rocktech Displays Ltd. RK101II01D-CT 10.1" TFT 1280x800 |
| 272 | - rocktech,rk101ii01d-ct | 276 | - rocktech,rk101ii01d-ct |
| 273 | # Rocktech Display Ltd. RK070ER9427 800(RGB)x480 TFT LCD panel | 277 | # Rocktech Display Ltd. RK070ER9427 800(RGB)x480 TFT LCD panel |
| @@ -276,6 +280,8 @@ properties: · 2 unmodified lines | |||
| 276 | - rocktech,rk043fn48h | 280 | - rocktech,rk043fn48h |
| 277 | # Samsung Electronics 10.1" WXGA (1280x800) TFT LCD panel | 281 | # Samsung Electronics 10.1" WXGA (1280x800) TFT LCD panel |
| 278 | - samsung,ltl101al01 | 282 | - samsung,ltl101al01 |
| 283 | # Samsung Electronics 10.6" FWXGA (1366x768) TFT LCD panel | ||
| 284 | - samsung,ltl106al01 | ||
| 279 | # Samsung Electronics 10.1" WSVGA TFT LCD panel | 285 | # Samsung Electronics 10.1" WSVGA TFT LCD panel |
| 280 | - samsung,ltn101nt05 | 286 | - samsung,ltn101nt05 |
| 281 | # Satoz SAT050AT40H12R2 5.0" WVGA TFT LCD panel | 287 | # Satoz SAT050AT40H12R2 5.0" WVGA TFT LCD panel |
| @@ -9,6 +9,9 @@ title: Ronbo RB070D30 DSI Display Panel · 8 unmodified lines | |||
| 9 | maintainers: | 9 | maintainers: |
| 10 | - Maxime Ripard <mripard@kernel.org> | 10 | - Maxime Ripard <mripard@kernel.org> |
| 11 | 11 | ||
| 12 | allOf: | ||
| 13 | - $ref: panel-common.yaml# | ||
| 14 | |||
| 12 | properties: | 15 | properties: |
| 13 | compatible: | 16 | compatible: |
| 14 | const: ronbo,rb070d30 | 17 | const: ronbo,rb070d30 |
| @@ -20,10 +23,6 @@ properties: · 5 unmodified lines | |||
| 20 | description: GPIO used for the power pin | 23 | description: GPIO used for the power pin |
| 21 | maxItems: 1 | 24 | maxItems: 1 |
| 22 | 25 | ||
| 23 | reset-gpios: | ||
| 24 | description: GPIO used for the reset pin | ||
| 25 | maxItems: 1 | ||
| 26 | |||
| 27 | shlr-gpios: | 26 | shlr-gpios: |
| 28 | description: GPIO used for the shlr pin (horizontal flip) | 27 | description: GPIO used for the shlr pin (horizontal flip) |
| 29 | maxItems: 1 | 28 | maxItems: 1 |
| @@ -35,10 +34,6 @@ properties: · 5 unmodified lines | |||
| 35 | vcc-lcd-supply: | 34 | vcc-lcd-supply: |
| 36 | description: Power regulator | 35 | description: Power regulator |
| 37 | 36 | ||
| 38 | backlight: | ||
| 39 | description: Backlight used by the panel | ||
| 40 | $ref: /schemas/types.yaml#/definitions/phandle | ||
| 41 | |||
| 42 | required: | 37 | required: |
| 43 | - compatible | 38 | - compatible |
| 44 | - power-gpios | 39 | - power-gpios |
| @@ -47,5 +42,6 @@ required: · 2 unmodified lines | |||
| 47 | - shlr-gpios | 42 | - shlr-gpios |
| 48 | - updn-gpios | 43 | - updn-gpios |
| 49 | - vcc-lcd-supply | 44 | - vcc-lcd-supply |
| 45 | - port | ||
| 50 | 46 | ||
| 51 | additionalProperties: false | 47 | unevaluatedProperties: false |
| @@ -33,6 +33,8 @@ properties: · 32 unmodified lines | |||
| 33 | - samsung,atna45dc02 | 33 | - samsung,atna45dc02 |
| 34 | # Samsung 15.6" 3K (2880x1620 pixels) eDP AMOLED panel | 34 | # Samsung 15.6" 3K (2880x1620 pixels) eDP AMOLED panel |
| 35 | - samsung,atna56ac03 | 35 | - samsung,atna56ac03 |
| 36 | # Samsung 16.0" 3K (2880x1800 pixels) eDP AMOLED panel | ||
| 37 | - samsung,atna60cl08 | ||
| 36 | - const: samsung,atna33xc20 | 38 | - const: samsung,atna33xc20 |
| 37 | 39 | ||
| 38 | enable-gpios: true | 40 | enable-gpios: true |
| @@ -0,0 +1,81 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/samsung,s6e3fc2x01.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Samsung S6E3FC2X01 AMOLED DDIC | ||
| 8 | |||
| 9 | description: The S6E3FC2X01 is display driver IC with connected panel. | ||
| 10 | |||
| 11 | maintainers: | ||
| 12 | - David Heidelberg <david@ixit.cz> | ||
| 13 | |||
| 14 | allOf: | ||
| 15 | - $ref: panel-common.yaml# | ||
| 16 | |||
| 17 | properties: | ||
| 18 | compatible: | ||
| 19 | items: | ||
| 20 | - enum: | ||
| 21 | # Samsung 6.41 inch, 1080x2340 pixels, 19.5:9 ratio | ||
| 22 | - samsung,s6e3fc2x01-ams641rw | ||
| 23 | - const: samsung,s6e3fc2x01 | ||
| 24 | |||
| 25 | reg: | ||
| 26 | maxItems: 1 | ||
| 27 | |||
| 28 | reset-gpios: true | ||
| 29 | |||
| 30 | port: true | ||
| 31 | |||
| 32 | vddio-supply: | ||
| 33 | description: VDD regulator | ||
| 34 | |||
| 35 | vci-supply: | ||
| 36 | description: VCI regulator | ||
| 37 | |||
| 38 | poc-supply: | ||
| 39 | description: POC regulator | ||
| 40 | |||
| 41 | required: | ||
| 42 | - compatible | ||
| 43 | - reset-gpios | ||
| 44 | - vddio-supply | ||
| 45 | - vci-supply | ||
| 46 | - poc-supply | ||
| 47 | |||
| 48 | unevaluatedProperties: false | ||
| 49 | |||
| 50 | examples: | ||
| 51 | - | | ||
| 52 | #include <dt-bindings/gpio/gpio.h> | ||
| 53 | |||
| 54 | dsi { | ||
| 55 | #address-cells = <1>; | ||
| 56 | #size-cells = <0>; | ||
| 57 | |||
| 58 | panel@0 { | ||
| 59 | compatible = "samsung,s6e3fc2x01-ams641rw", "samsung,s6e3fc2x01"; | ||
| 60 | reg = <0>; | ||
| 61 | |||
| 62 | vddio-supply = <&vreg_l14a_1p88>; | ||
| 63 | vci-supply = <&s2dos05_buck1>; | ||
| 64 | poc-supply = <&s2dos05_ldo1>; | ||
| 65 | |||
| 66 | te-gpios = <&tlmm 10 GPIO_ACTIVE_HIGH>; | ||
| 67 | reset-gpios = <&tlmm 6 GPIO_ACTIVE_HIGH>; | ||
| 68 | |||
| 69 | pinctrl-0 = <&sde_dsi_active &sde_te_active_sleep>; | ||
| 70 | pinctrl-1 = <&sde_dsi_suspend &sde_te_active_sleep>; | ||
| 71 | pinctrl-names = "default", "sleep"; | ||
| 72 | |||
| 73 | port { | ||
| 74 | panel_in: endpoint { | ||
| 75 | remote-endpoint = <&mdss_dsi0_out>; | ||
| 76 | }; | ||
| 77 | }; | ||
| 78 | }; | ||
| 79 | }; | ||
| 80 | |||
| 81 | ... | ||
| @@ -0,0 +1,79 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/samsung,sofef00.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Samsung SOFEF00 AMOLED DDIC | ||
| 8 | |||
| 9 | description: The SOFEF00 is display driver IC with connected panel. | ||
| 10 | |||
| 11 | maintainers: | ||
| 12 | - David Heidelberg <david@ixit.cz> | ||
| 13 | |||
| 14 | allOf: | ||
| 15 | - $ref: panel-common.yaml# | ||
| 16 | |||
| 17 | properties: | ||
| 18 | compatible: | ||
| 19 | items: | ||
| 20 | - enum: | ||
| 21 | # Samsung 6.01 inch, 1080x2160 pixels, 18:9 ratio | ||
| 22 | - samsung,sofef00-ams601nt22 | ||
| 23 | # Samsung 6.28 inch, 1080x2280 pixels, 19:9 ratio | ||
| 24 | - samsung,sofef00-ams628nw01 | ||
| 25 | - const: samsung,sofef00 | ||
| 26 | |||
| 27 | reg: | ||
| 28 | maxItems: 1 | ||
| 29 | |||
| 30 | poc-supply: | ||
| 31 | description: POC regulator | ||
| 32 | |||
| 33 | vci-supply: | ||
| 34 | description: VCI regulator | ||
| 35 | |||
| 36 | vddio-supply: | ||
| 37 | description: VDD regulator | ||
| 38 | |||
| 39 | required: | ||
| 40 | - compatible | ||
| 41 | - reset-gpios | ||
| 42 | - poc-supply | ||
| 43 | - vci-supply | ||
| 44 | - vddio-supply | ||
| 45 | |||
| 46 | unevaluatedProperties: false | ||
| 47 | |||
| 48 | examples: | ||
| 49 | - | | ||
| 50 | #include <dt-bindings/gpio/gpio.h> | ||
| 51 | |||
| 52 | dsi { | ||
| 53 | #address-cells = <1>; | ||
| 54 | #size-cells = <0>; | ||
| 55 | |||
| 56 | panel@0 { | ||
| 57 | compatible = "samsung,sofef00-ams628nw01", "samsung,sofef00"; | ||
| 58 | reg = <0>; | ||
| 59 | |||
| 60 | vddio-supply = <&vreg_l14a_1p88>; | ||
| 61 | vci-supply = <&s2dos05_buck1>; | ||
| 62 | poc-supply = <&s2dos05_ldo1>; | ||
| 63 | |||
| 64 | te-gpios = <&tlmm 10 GPIO_ACTIVE_HIGH>; | ||
| 65 | reset-gpios = <&tlmm 6 GPIO_ACTIVE_HIGH>; | ||
| 66 | |||
| 67 | pinctrl-0 = <&panel_active>; | ||
| 68 | pinctrl-1 = <&panel_suspend>; | ||
| 69 | pinctrl-names = "default", "sleep"; | ||
| 70 | |||
| 71 | port { | ||
| 72 | panel_in: endpoint { | ||
| 73 | remote-endpoint = <&mdss_dsi0_out>; | ||
| 74 | }; | ||
| 75 | }; | ||
| 76 | }; | ||
| 77 | }; | ||
| 78 | |||
| 79 | ... | ||
| @@ -0,0 +1,99 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/sharp,lq079l1sx01.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Sharp Microelectronics 7.9" WQXGA TFT LCD panel | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Svyatoslav Ryhel <clamor95@gmail.com> | ||
| 11 | |||
| 12 | description: > | ||
| 13 | This panel requires a dual-channel DSI host to operate and it supports | ||
| 14 | only left-right split mode, where each channel drives the left or right | ||
| 15 | half of the screen and only video mode. | ||
| 16 | |||
| 17 | Each of the DSI channels controls a separate DSI peripheral. | ||
| 18 | The peripheral driven by the first link (DSI-LINK1), left one, is | ||
| 19 | considered the primary peripheral and controls the device. | ||
| 20 | |||
| 21 | allOf: | ||
| 22 | - $ref: panel-common-dual.yaml# | ||
| 23 | |||
| 24 | properties: | ||
| 25 | compatible: | ||
| 26 | const: sharp,lq079l1sx01 | ||
| 27 | |||
| 28 | reg: | ||
| 29 | maxItems: 1 | ||
| 30 | |||
| 31 | avdd-supply: | ||
| 32 | description: regulator that supplies the analog voltage | ||
| 33 | |||
| 34 | vddio-supply: | ||
| 35 | description: regulator that supplies the I/O voltage | ||
| 36 | |||
| 37 | vsp-supply: | ||
| 38 | description: positive boost supply regulator | ||
| 39 | |||
| 40 | vsn-supply: | ||
| 41 | description: negative boost supply regulator | ||
| 42 | |||
| 43 | reset-gpios: | ||
| 44 | maxItems: 1 | ||
| 45 | |||
| 46 | backlight: true | ||
| 47 | ports: true | ||
| 48 | |||
| 49 | required: | ||
| 50 | - compatible | ||
| 51 | - reg | ||
| 52 | - avdd-supply | ||
| 53 | - vddio-supply | ||
| 54 | - ports | ||
| 55 | |||
| 56 | additionalProperties: false | ||
| 57 | |||
| 58 | examples: | ||
| 59 | - | | ||
| 60 | #include <dt-bindings/gpio/gpio.h> | ||
| 61 | |||
| 62 | dsi { | ||
| 63 | #address-cells = <1>; | ||
| 64 | #size-cells = <0>; | ||
| 65 | |||
| 66 | panel@0 { | ||
| 67 | compatible = "sharp,lq079l1sx01"; | ||
| 68 | reg = <0>; | ||
| 69 | |||
| 70 | reset-gpios = <&gpio 59 GPIO_ACTIVE_LOW>; | ||
| 71 | |||
| 72 | avdd-supply = <&avdd_lcd>; | ||
| 73 | vddio-supply = <&vdd_lcd_io>; | ||
| 74 | vsp-supply = <&vsp_5v5_lcd>; | ||
| 75 | vsn-supply = <&vsn_5v5_lcd>; | ||
| 76 | |||
| 77 | backlight = <&backlight>; | ||
| 78 | |||
| 79 | ports { | ||
| 80 | #address-cells = <1>; | ||
| 81 | #size-cells = <0>; | ||
| 82 | |||
| 83 | port@0 { | ||
| 84 | reg = <0>; | ||
| 85 | panel_in0: endpoint { | ||
| 86 | remote-endpoint = <&dsi0_out>; | ||
| 87 | }; | ||
| 88 | }; | ||
| 89 | |||
| 90 | port@1 { | ||
| 91 | reg = <1>; | ||
| 92 | panel_in1: endpoint { | ||
| 93 | remote-endpoint = <&dsi1_out>; | ||
| 94 | }; | ||
| 95 | }; | ||
| 96 | }; | ||
| 97 | }; | ||
| 98 | }; | ||
| 99 | ... | ||
| @@ -0,0 +1,89 @@ | |||
| 1 | # SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/display/panel/synaptics,td4300-panel.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Synaptics TDDI Display Panel Controller | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Kaustabh Chakraborty <kauschluss@disroot.org> | ||
| 11 | |||
| 12 | allOf: | ||
| 13 | - $ref: panel-common.yaml# | ||
| 14 | |||
| 15 | properties: | ||
| 16 | compatible: | ||
| 17 | enum: | ||
| 18 | - syna,td4101-panel | ||
| 19 | - syna,td4300-panel | ||
| 20 | |||
| 21 | reg: | ||
| 22 | maxItems: 1 | ||
| 23 | |||
| 24 | vio-supply: | ||
| 25 | description: core I/O voltage supply | ||
| 26 | |||
| 27 | vsn-supply: | ||
| 28 | description: negative voltage supply for analog circuits | ||
| 29 | |||
| 30 | vsp-supply: | ||
| 31 | description: positive voltage supply for analog circuits | ||
| 32 | |||
| 33 | backlight-gpios: | ||
| 34 | maxItems: 1 | ||
| 35 | description: backlight enable GPIO | ||
| 36 | |||
| 37 | reset-gpios: true | ||
| 38 | width-mm: true | ||
| 39 | height-mm: true | ||
| 40 | panel-timing: true | ||
| 41 | |||
| 42 | required: | ||
| 43 | - compatible | ||
| 44 | - reg | ||
| 45 | - width-mm | ||
| 46 | - height-mm | ||
| 47 | - panel-timing | ||
| 48 | |||
| 49 | additionalProperties: false | ||
| 50 | |||
| 51 | examples: | ||
| 52 | - | | ||
| 53 | #include <dt-bindings/gpio/gpio.h> | ||
| 54 | |||
| 55 | dsi { | ||
| 56 | #address-cells = <1>; | ||
| 57 | #size-cells = <0>; | ||
| 58 | |||
| 59 | panel@0 { | ||
| 60 | compatible = "syna,td4300-panel"; | ||
| 61 | reg = <0>; | ||
| 62 | |||
| 63 | vio-supply = <&panel_vio_reg>; | ||
| 64 | vsn-supply = <&panel_vsn_reg>; | ||
| 65 | vsp-supply = <&panel_vsp_reg>; | ||
| 66 | |||
| 67 | backlight-gpios = <&gpd3 5 GPIO_ACTIVE_LOW>; | ||
| 68 | reset-gpios = <&gpd3 4 GPIO_ACTIVE_LOW>; | ||
| 69 | |||
| 70 | width-mm = <68>; | ||
| 71 | height-mm = <121>; | ||
| 72 | |||
| 73 | panel-timing { | ||
| 74 | clock-frequency = <144389520>; | ||
| 75 | |||
| 76 | hactive = <1080>; | ||
| 77 | hsync-len = <4>; | ||
| 78 | hfront-porch = <120>; | ||
| 79 | hback-porch = <32>; | ||
| 80 | |||
| 81 | vactive = <1920>; | ||
| 82 | vsync-len = <2>; | ||
| 83 | vfront-porch = <21>; | ||
| 84 | vback-porch = <4>; | ||
| 85 | }; | ||
| 86 | }; | ||
| 87 | }; | ||
| 88 | |||
| 89 | ... | ||
| @@ -25,6 +25,9 @@ properties: · 24 unmodified lines | |||
| 25 | - enum: | 25 | - enum: |
| 26 | - renesas,r9a07g054-du # RZ/V2L | 26 | - renesas,r9a07g054-du # RZ/V2L |
| 27 | - const: renesas,r9a07g044-du # RZ/G2L fallback | 27 | - const: renesas,r9a07g044-du # RZ/G2L fallback |
| 28 | - items: | ||
| 29 | - const: renesas,r9a09g056-du # RZ/V2N | ||
| 30 | - const: renesas,r9a09g057-du # RZ/V2H(P) fallback | ||
| 28 | 31 | ||
| 29 | reg: | 32 | reg: |
| 30 | maxItems: 1 | 33 | maxItems: 1 |
| @@ -17,6 +17,7 @@ properties: · 16 unmodified lines | |||
| 17 | - rockchip,px30-mipi-dsi | 17 | - rockchip,px30-mipi-dsi |
| 18 | - rockchip,rk3128-mipi-dsi | 18 | - rockchip,rk3128-mipi-dsi |
| 19 | - rockchip,rk3288-mipi-dsi | 19 | - rockchip,rk3288-mipi-dsi |
| 20 | - rockchip,rk3368-mipi-dsi | ||
| 20 | - rockchip,rk3399-mipi-dsi | 21 | - rockchip,rk3399-mipi-dsi |
| 21 | - rockchip,rk3568-mipi-dsi | 22 | - rockchip,rk3568-mipi-dsi |
| 22 | - rockchip,rv1126-mipi-dsi | 23 | - rockchip,rv1126-mipi-dsi |
| @@ -73,6 +74,7 @@ allOf: · 50 unmodified lines | |||
| 73 | enum: | 74 | enum: |
| 74 | - rockchip,px30-mipi-dsi | 75 | - rockchip,px30-mipi-dsi |
| 75 | - rockchip,rk3128-mipi-dsi | 76 | - rockchip,rk3128-mipi-dsi |
| 77 | - rockchip,rk3368-mipi-dsi | ||
| 76 | - rockchip,rk3568-mipi-dsi | 78 | - rockchip,rk3568-mipi-dsi |
| 77 | - rockchip,rv1126-mipi-dsi | 79 | - rockchip,rv1126-mipi-dsi |
| 78 | 80 | ||
| @@ -113,6 +113,14 @@ properties: · 112 unmodified lines | |||
| 113 | description: | 113 | description: |
| 114 | Additional HDMI QP related data is accessed through VO GRF regs. | 114 | Additional HDMI QP related data is accessed through VO GRF regs. |
| 115 | 115 | ||
| 116 | frl-enable-gpios: | ||
| 117 | description: | ||
| 118 | Optional GPIO line to be asserted when operating in HDMI 2.1 FRL mode and | ||
| 119 | deasserted for HDMI 1.4/2.0 TMDS. It can be used to control external | ||
| 120 | voltage bias for HDMI data lines. When not present the HDMI encoder will | ||
| 121 | operate in TMDS mode only. | ||
| 122 | maxItems: 1 | ||
| 123 | |||
| 116 | required: | 124 | required: |
| 117 | - compatible | 125 | - compatible |
| 118 | - reg | 126 | - reg |
| @@ -132,8 +140,10 @@ unevaluatedProperties: false · 13 unmodified lines | |||
| 132 | examples: | 140 | examples: |
| 133 | - | | 141 | - | |
| 134 | #include <dt-bindings/clock/rockchip,rk3588-cru.h> | 142 | #include <dt-bindings/clock/rockchip,rk3588-cru.h> |
| 143 | #include <dt-bindings/gpio/gpio.h> | ||
| 135 | #include <dt-bindings/interrupt-controller/arm-gic.h> | 144 | #include <dt-bindings/interrupt-controller/arm-gic.h> |
| 136 | #include <dt-bindings/interrupt-controller/irq.h> | 145 | #include <dt-bindings/interrupt-controller/irq.h> |
| 146 | #include <dt-bindings/pinctrl/rockchip.h> | ||
| 137 | #include <dt-bindings/power/rk3588-power.h> | 147 | #include <dt-bindings/power/rk3588-power.h> |
| 138 | #include <dt-bindings/reset/rockchip,rk3588-cru.h> | 148 | #include <dt-bindings/reset/rockchip,rk3588-cru.h> |
| 139 | 149 | ||
| @@ -164,6 +174,7 @@ examples: · 24 unmodified lines | |||
| 164 | rockchip,grf = <&sys_grf>; | 174 | rockchip,grf = <&sys_grf>; |
| 165 | rockchip,vo-grf = <&vo1_grf>; | 175 | rockchip,vo-grf = <&vo1_grf>; |
| 166 | #sound-dai-cells = <0>; | 176 | #sound-dai-cells = <0>; |
| 177 | frl-enable-gpios = <&gpio4 RK_PB1 GPIO_ACTIVE_LOW>; | ||
| 167 | 178 | ||
| 168 | ports { | 179 | ports { |
| 169 | #address-cells = <1>; | 180 | #address-cells = <1>; |
| @@ -18,6 +18,8 @@ properties: · 17 unmodified lines | |||
| 18 | oneOf: | 18 | oneOf: |
| 19 | - items: | 19 | - items: |
| 20 | - enum: | 20 | - enum: |
| 21 | - mediatek,mt8196-mali | ||
| 22 | - nxp,imx95-mali # G310 | ||
| 21 | - rockchip,rk3588-mali | 23 | - rockchip,rk3588-mali |
| 22 | - const: arm,mali-valhall-csf # Mali Valhall GPU model/revision is fully discoverable | 24 | - const: arm,mali-valhall-csf # Mali Valhall GPU model/revision is fully discoverable |
| 23 | 25 | ||
| @@ -44,7 +46,9 @@ properties: · 20 unmodified lines | |||
| 44 | minItems: 1 | 46 | minItems: 1 |
| 45 | items: | 47 | items: |
| 46 | - const: core | 48 | - const: core |
| 47 | - const: coregroup | 49 | - enum: |
| 50 | - coregroup | ||
| 51 | - stacks | ||
| 48 | - const: stacks | 52 | - const: stacks |
| 49 | 53 | ||
| 50 | mali-supply: true | 54 | mali-supply: true |
| @@ -91,7 +95,6 @@ required: · 40 unmodified lines | |||
| 91 | - interrupts | 95 | - interrupts |
| 92 | - interrupt-names | 96 | - interrupt-names |
| 93 | - clocks | 97 | - clocks |
| 94 | - mali-supply | ||
| 95 | 98 | ||
| 96 | additionalProperties: false | 99 | additionalProperties: false |
| 97 | 100 | ||
| @@ -108,6 +111,29 @@ allOf: · 10 unmodified lines | |||
| 108 | power-domains: | 111 | power-domains: |
| 109 | maxItems: 1 | 112 | maxItems: 1 |
| 110 | power-domain-names: false | 113 | power-domain-names: false |
| 114 | required: | ||
| 115 | - mali-supply | ||
| 116 | - if: | ||
| 117 | properties: | ||
| 118 | compatible: | ||
| 119 | contains: | ||
| 120 | const: mediatek,mt8196-mali | ||
| 121 | then: | ||
| 122 | properties: | ||
| 123 | mali-supply: false | ||
| 124 | sram-supply: false | ||
| 125 | operating-points-v2: false | ||
| 126 | power-domains: | ||
| 127 | maxItems: 1 | ||
| 128 | power-domain-names: false | ||
| 129 | clocks: | ||
| 130 | maxItems: 2 | ||
| 131 | clock-names: | ||
| 132 | items: | ||
| 133 | - const: core | ||
| 134 | - const: stacks | ||
| 135 | required: | ||
| 136 | - power-domains | ||
| 111 | 137 | ||
| 112 | examples: | 138 | examples: |
| 113 | - | | 139 | - | |
| @@ -143,5 +169,17 @@ examples: · 29 unmodified lines | |||
| 143 | }; | 169 | }; |
| 144 | }; | 170 | }; |
| 145 | }; | 171 | }; |
| 172 | - | | ||
| 173 | gpu@48000000 { | ||
| 174 | compatible = "mediatek,mt8196-mali", "arm,mali-valhall-csf"; | ||
| 175 | reg = <0x48000000 0x480000>; | ||
| 176 | clocks = <&gpufreq 0>, <&gpufreq 1>; | ||
| 177 | clock-names = "core", "stacks"; | ||
| 178 | interrupts = <GIC_SPI 606 IRQ_TYPE_LEVEL_HIGH 0>, | ||
| 179 | <GIC_SPI 605 IRQ_TYPE_LEVEL_HIGH 0>, | ||
| 180 | <GIC_SPI 604 IRQ_TYPE_LEVEL_HIGH 0>; | ||
| 181 | interrupt-names = "job", "mmu", "gpu"; | ||
| 182 | power-domains = <&gpufreq>; | ||
| 183 | }; | ||
| 146 | 184 | ||
| 147 | ... | 185 | ... |
| @@ -13,6 +13,16 @@ maintainers: · 12 unmodified lines | |||
| 13 | properties: | 13 | properties: |
| 14 | compatible: | 14 | compatible: |
| 15 | oneOf: | 15 | oneOf: |
| 16 | - items: | ||
| 17 | - enum: | ||
| 18 | - renesas,r8a7796-gpu | ||
| 19 | - renesas,r8a77961-gpu | ||
| 20 | - const: img,img-gx6250 | ||
| 21 | - const: img,img-rogue | ||
| 22 | - items: | ||
| 23 | - const: renesas,r8a77965-gpu | ||
| 24 | - const: img,img-ge7800 | ||
| 25 | - const: img,img-rogue | ||
| 16 | - items: | 26 | - items: |
| 17 | - enum: | 27 | - enum: |
| 18 | - ti,am62-gpu | 28 | - ti,am62-gpu |
| @@ -86,48 +96,56 @@ allOf: · 67 unmodified lines | |||
| 86 | properties: | 96 | properties: |
| 87 | compatible: | 97 | compatible: |
| 88 | contains: | 98 | contains: |
| 89 | const: img,img-axe-1-16m | 99 | enum: |
| 100 | - ti,am62-gpu | ||
| 101 | - ti,j721s2-gpu | ||
| 90 | then: | 102 | then: |
| 91 | properties: | 103 | properties: |
| 92 | power-domains: | 104 | clocks: |
| 93 | items: | ||
| 94 | - description: Power domain A | ||
| 95 | power-domain-names: | ||
| 96 | maxItems: 1 | 105 | maxItems: 1 |
| 97 | required: | ||
| 98 | - power-domains | ||
| 99 | - power-domain-names | ||
| 100 | 106 | ||
| 101 | - if: | 107 | - if: |
| 102 | properties: | 108 | properties: |
| 103 | compatible: | 109 | compatible: |
| 104 | contains: | 110 | contains: |
| 105 | const: thead,th1520-gpu | 111 | enum: |
| 112 | - img,img-ge7800 | ||
| 113 | - img,img-gx6250 | ||
| 114 | - thead,th1520-gpu | ||
| 106 | then: | 115 | then: |
| 107 | properties: | 116 | properties: |
| 108 | clocks: | 117 | clocks: |
| 109 | minItems: 3 | 118 | minItems: 3 |
| 110 | clock-names: | 119 | clock-names: |
| 111 | minItems: 3 | 120 | minItems: 3 |
| 121 | |||
| 122 | - if: | ||
| 123 | properties: | ||
| 124 | compatible: | ||
| 125 | contains: | ||
| 126 | const: img,img-axe-1-16m | ||
| 127 | then: | ||
| 128 | properties: | ||
| 112 | power-domains: | 129 | power-domains: |
| 113 | items: | 130 | maxItems: 1 |
| 114 | - description: The single, unified power domain for the GPU on the | 131 | power-domain-names: |
| 115 | TH1520 SoC, integrating all internal IP power domains. | 132 | maxItems: 1 |
| 116 | power-domain-names: false | ||
| 117 | required: | 133 | required: |
| 118 | - power-domains | 134 | - power-domains |
| 135 | - power-domain-names | ||
| 119 | 136 | ||
| 120 | - if: | 137 | - if: |
| 121 | properties: | 138 | properties: |
| 122 | compatible: | 139 | compatible: |
| 123 | contains: | 140 | contains: |
| 124 | const: img,img-bxs-4-64 | 141 | enum: |
| 142 | - img,img-bxs-4-64 | ||
| 143 | - img,img-ge7800 | ||
| 144 | - img,img-gx6250 | ||
| 125 | then: | 145 | then: |
| 126 | properties: | 146 | properties: |
| 127 | power-domains: | 147 | power-domains: |
| 128 | items: | 148 | minItems: 2 |
| 129 | - description: Power domain A | ||
| 130 | - description: Power domain B | ||
| 131 | power-domain-names: | 149 | power-domain-names: |
| 132 | minItems: 2 | 150 | minItems: 2 |
| 133 | required: | 151 | required: |
| @@ -138,13 +156,16 @@ allOf: · 4 unmodified lines | |||
| 138 | properties: | 156 | properties: |
| 139 | compatible: | 157 | compatible: |
| 140 | contains: | 158 | contains: |
| 141 | enum: | 159 | const: thead,th1520-gpu |
| 142 | - ti,am62-gpu | ||
| 143 | - ti,j721s2-gpu | ||
| 144 | then: | 160 | then: |
| 145 | properties: | 161 | properties: |
| 146 | clocks: | 162 | power-domains: |
| 147 | maxItems: 1 | 163 | items: |
| 164 | - description: The single, unified power domain for the GPU on the | ||
| 165 | TH1520 SoC, integrating all internal IP power domains. | ||
| 166 | power-domain-names: false | ||
| 167 | required: | ||
| 168 | - power-domains | ||
| 148 | 169 | ||
| 149 | examples: | 170 | examples: |
| 150 | - | | 171 | - | |
| @@ -89,6 +89,8 @@ properties: · 88 unmodified lines | |||
| 89 | - description: Qcom Adreno GPUs implementing "qcom,smmu-500" and "arm,mmu-500" | 89 | - description: Qcom Adreno GPUs implementing "qcom,smmu-500" and "arm,mmu-500" |
| 90 | items: | 90 | items: |
| 91 | - enum: | 91 | - enum: |
| 92 | - qcom,glymur-smmu-500 | ||
| 93 | - qcom,kaanapali-smmu-500 | ||
| 92 | - qcom,milos-smmu-500 | 94 | - qcom,milos-smmu-500 |
| 93 | - qcom,qcm2290-smmu-500 | 95 | - qcom,qcm2290-smmu-500 |
| 94 | - qcom,qcs615-smmu-500 | 96 | - qcom,qcs615-smmu-500 |
| @@ -0,0 +1,79 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause | ||
| 2 | %YAML 1.2 | ||
| 3 | --- | ||
| 4 | $id: http://devicetree.org/schemas/npu/arm,ethos.yaml# | ||
| 5 | $schema: http://devicetree.org/meta-schemas/core.yaml# | ||
| 6 | |||
| 7 | title: Arm Ethos U65/U85 | ||
| 8 | |||
| 9 | maintainers: | ||
| 10 | - Rob Herring <robh@kernel.org> | ||
| 11 | |||
| 12 | description: > | ||
| 13 | The Arm Ethos-U NPUs are designed for IoT inference applications. The NPUs | ||
| 14 | can accelerate 8-bit and 16-bit integer quantized networks: | ||
| 15 | |||
| 16 | Transformer networks (U85 only) | ||
| 17 | Convolutional Neural Networks (CNN) | ||
| 18 | Recurrent Neural Networks (RNN) | ||
| 19 | |||
| 20 | Further documentation is available here: | ||
| 21 | |||
| 22 | U65 TRM: https://developer.arm.com/documentation/102023/ | ||
| 23 | U85 TRM: https://developer.arm.com/documentation/102685/ | ||
| 24 | |||
| 25 | properties: | ||
| 26 | compatible: | ||
| 27 | oneOf: | ||
| 28 | - items: | ||
| 29 | - enum: | ||
| 30 | - fsl,imx93-npu | ||
| 31 | - const: arm,ethos-u65 | ||
| 32 | - items: | ||
| 33 | - {} | ||
| 34 | - const: arm,ethos-u85 | ||
| 35 | |||
| 36 | reg: | ||
| 37 | maxItems: 1 | ||
| 38 | |||
| 39 | interrupts: | ||
| 40 | maxItems: 1 | ||
| 41 | |||
| 42 | clocks: | ||
| 43 | maxItems: 2 | ||
| 44 | |||
| 45 | clock-names: | ||
| 46 | items: | ||
| 47 | - const: core | ||
| 48 | - const: apb | ||
| 49 | |||
| 50 | power-domains: | ||
| 51 | maxItems: 1 | ||
| 52 | |||
| 53 | sram: | ||
| 54 | maxItems: 1 | ||
| 55 | |||
| 56 | required: | ||
| 57 | - compatible | ||
| 58 | - reg | ||
| 59 | - interrupts | ||
| 60 | - clocks | ||
| 61 | |||
| 62 | additionalProperties: false | ||
| 63 | |||
| 64 | examples: | ||
| 65 | - | | ||
| 66 | #include <dt-bindings/interrupt-controller/irq.h> | ||
| 67 | #include <dt-bindings/interrupt-controller/arm-gic.h> | ||
| 68 | #include <dt-bindings/clock/imx93-clock.h> | ||
| 69 | |||
| 70 | npu@4a900000 { | ||
| 71 | compatible = "fsl,imx93-npu", "arm,ethos-u65"; | ||
| 72 | reg = <0x4a900000 0x1000>; | ||
| 73 | interrupts = <GIC_SPI 178 IRQ_TYPE_LEVEL_HIGH>; | ||
| 74 | power-domains = <&mlmix>; | ||
| 75 | clocks = <&clk IMX93_CLK_ML>, <&clk IMX93_CLK_ML_APB>; | ||
| 76 | clock-names = "core", "apb"; | ||
| 77 | sram = <&sram>; | ||
| 78 | }; | ||
| 79 | ... | ||
| @@ -176,6 +176,8 @@ patternProperties: · 175 unmodified lines | |||
| 176 | description: All Sensors Corporation | 176 | description: All Sensors Corporation |
| 177 | "^asix,.*": | 177 | "^asix,.*": |
| 178 | description: ASIX Electronics Corporation | 178 | description: ASIX Electronics Corporation |
| 179 | "^asl-tek,.*": | ||
| 180 | description: ASL Xiamen Technology Co., Ltd. | ||
| 179 | "^aspeed,.*": | 181 | "^aspeed,.*": |
| 180 | description: ASPEED Technology Inc. | 182 | description: ASPEED Technology Inc. |
| 181 | "^asrock,.*": | 183 | "^asrock,.*": |
| @@ -835,6 +837,8 @@ patternProperties: · 653 unmodified lines | |||
| 835 | description: JOZ BV | 837 | description: JOZ BV |
| 836 | "^jty,.*": | 838 | "^jty,.*": |
| 837 | description: JTY | 839 | description: JTY |
| 840 | "^jutouch,.*": | ||
| 841 | description: JuTouch Technology Co., Ltd. | ||
| 838 | "^kam,.*": | 842 | "^kam,.*": |
| 839 | description: Kamstrup A/S | 843 | description: Kamstrup A/S |
| 840 | "^karo,.*": | 844 | "^karo,.*": |
| @@ -1323,6 +1327,8 @@ patternProperties: · 482 unmodified lines | |||
| 1323 | description: Raumfeld GmbH | 1327 | description: Raumfeld GmbH |
| 1324 | "^raydium,.*": | 1328 | "^raydium,.*": |
| 1325 | description: Raydium Semiconductor Corp. | 1329 | description: Raydium Semiconductor Corp. |
| 1330 | "^raystar,.*": | ||
| 1331 | description: Raystar Optronics, Inc. | ||
| 1326 | "^rda,.*": | 1332 | "^rda,.*": |
| 1327 | description: Unisoc Communications, Inc. | 1333 | description: Unisoc Communications, Inc. |
| 1328 | "^realtek,.*": | 1334 | "^realtek,.*": |
| @@ -92,6 +92,18 @@ GEM Atomic Helper Reference · 91 unmodified lines | |||
| 92 | .. kernel-doc:: drivers/gpu/drm/drm_gem_atomic_helper.c | 92 | .. kernel-doc:: drivers/gpu/drm/drm_gem_atomic_helper.c |
| 93 | :export: | 93 | :export: |
| 94 | 94 | ||
| 95 | VBLANK Helper Reference | ||
| 96 | ----------------------- | ||
| 97 | |||
| 98 | .. kernel-doc:: drivers/gpu/drm/drm_vblank_helper.c | ||
| 99 | :doc: overview | ||
| 100 | |||
| 101 | .. kernel-doc:: include/drm/drm_vblank_helper.h | ||
| 102 | :internal: | ||
| 103 | |||
| 104 | .. kernel-doc:: drivers/gpu/drm/drm_vblank_helper.c | ||
| 105 | :export: | ||
| 106 | |||
| 95 | Simple KMS Helper Reference | 107 | Simple KMS Helper Reference |
| 96 | =========================== | 108 | =========================== |
| 97 | 109 | ||
| @@ -413,6 +413,21 @@ Plane Panic Functions Reference · 412 unmodified lines | |||
| 413 | .. kernel-doc:: drivers/gpu/drm/drm_panic.c | 413 | .. kernel-doc:: drivers/gpu/drm/drm_panic.c |
| 414 | :export: | 414 | :export: |
| 415 | 415 | ||
| 416 | Colorop Abstraction | ||
| 417 | =================== | ||
| 418 | |||
| 419 | .. kernel-doc:: drivers/gpu/drm/drm_colorop.c | ||
| 420 | :doc: overview | ||
| 421 | |||
| 422 | Colorop Functions Reference | ||
| 423 | --------------------------- | ||
| 424 | |||
| 425 | .. kernel-doc:: include/drm/drm_colorop.h | ||
| 426 | :internal: | ||
| 427 | |||
| 428 | .. kernel-doc:: drivers/gpu/drm/drm_colorop.c | ||
| 429 | :export: | ||
| 430 | |||
| 416 | Display Modes Function Reference | 431 | Display Modes Function Reference |
| 417 | ================================ | 432 | ================================ |
| 418 | 433 | ||
| @@ -44,25 +44,6 @@ automatically generates the corresponding mappings between a value and a number. · 43 unmodified lines | |||
| 44 | | Complexity: Beginner | 44 | | Complexity: Beginner |
| 45 | | Link: https://docs.rs/num/latest/num/trait.FromPrimitive.html | 45 | | Link: https://docs.rs/num/latest/num/trait.FromPrimitive.html |
| 46 | 46 | ||
| 47 | Conversion from byte slices for types implementing FromBytes [TRSM] | ||
| 48 | ------------------------------------------------------------------- | ||
| 49 | |||
| 50 | We retrieve several structures from byte streams coming from the BIOS or loaded | ||
| 51 | firmware. At the moment converting the bytes slice into the proper type require | ||
| 52 | an inelegant `unsafe` operation; this will go away once `FromBytes` implements | ||
| 53 | a proper `from_bytes` method. | ||
| 54 | |||
| 55 | | Complexity: Beginner | ||
| 56 | |||
| 57 | CoherentAllocation improvements [COHA] | ||
| 58 | -------------------------------------- | ||
| 59 | |||
| 60 | `CoherentAllocation` needs a safe way to write into the allocation, and to | ||
| 61 | obtain slices within the allocation. | ||
| 62 | |||
| 63 | | Complexity: Beginner | ||
| 64 | | Contact: Abdiel Janulgue | ||
| 65 | |||
| 66 | Generic register abstraction [REGA] | 47 | Generic register abstraction [REGA] |
| 67 | ----------------------------------- | 48 | ----------------------------------- |
| 68 | 49 | ||
| @@ -153,17 +134,6 @@ A `num` core kernel module is being designed to provide these operations. · 84 unmodified lines | |||
| 153 | | Complexity: Intermediate | 134 | | Complexity: Intermediate |
| 154 | | Contact: Alexandre Courbot | 135 | | Contact: Alexandre Courbot |
| 155 | 136 | ||
| 156 | Delay / Sleep abstractions [DLAY] | ||
| 157 | --------------------------------- | ||
| 158 | |||
| 159 | Rust abstractions for the kernel's delay() and sleep() functions. | ||
| 160 | |||
| 161 | FUJITA Tomonori plans to work on abstractions for read_poll_timeout_atomic() | ||
| 162 | (and friends) [1]. | ||
| 163 | |||
| 164 | | Complexity: Beginner | ||
| 165 | | Link: https://lore.kernel.org/netdev/20250228.080550.354359820929821928.fujita.tomonori@gmail.com/ [1] | ||
| 166 | |||
| 167 | IRQ abstractions | 137 | IRQ abstractions |
| 168 | ---------------- | 138 | ---------------- |
| 169 | 139 | ||
| @@ -0,0 +1,378 @@ | |||
| 1 | .. SPDX-License-Identifier: GPL-2.0 | ||
| 2 | |||
| 3 | ======================== | ||
| 4 | Linux Color Pipeline API | ||
| 5 | ======================== | ||
| 6 | |||
| 7 | What problem are we solving? | ||
| 8 | ============================ | ||
| 9 | |||
| 10 | We would like to support pre-, and post-blending complex color | ||
| 11 | transformations in display controller hardware in order to allow for | ||
| 12 | HW-supported HDR use-cases, as well as to provide support to | ||
| 13 | color-managed applications, such as video or image editors. | ||
| 14 | |||
| 15 | It is possible to support an HDR output on HW supporting the Colorspace | ||
| 16 | and HDR Metadata drm_connector properties, but that requires the | ||
| 17 | compositor or application to render and compose the content into one | ||
| 18 | final buffer intended for display. Doing so is costly. | ||
| 19 | |||
| 20 | Most modern display HW offers various 1D LUTs, 3D LUTs, matrices, and other | ||
| 21 | operations to support color transformations. These operations are often | ||
| 22 | implemented in fixed-function HW and therefore much more power efficient than | ||
| 23 | performing similar operations via shaders or CPU. | ||
| 24 | |||
| 25 | We would like to make use of this HW functionality to support complex color | ||
| 26 | transformations with no, or minimal CPU or shader load. The switch between HW | ||
| 27 | fixed-function blocks and shaders/CPU must be seamless with no visible | ||
| 28 | difference when fallback to shaders/CPU is neceesary at any time. | ||
| 29 | |||
| 30 | |||
| 31 | How are other OSes solving this problem? | ||
| 32 | ======================================== | ||
| 33 | |||
| 34 | The most widely supported use-cases regard HDR content, whether video or | ||
| 35 | gaming. | ||
| 36 | |||
| 37 | Most OSes will specify the source content format (color gamut, encoding transfer | ||
| 38 | function, and other metadata, such as max and average light levels) to a driver. | ||
| 39 | Drivers will then program their fixed-function HW accordingly to map from a | ||
| 40 | source content buffer's space to a display's space. | ||
| 41 | |||
| 42 | When fixed-function HW is not available the compositor will assemble a shader to | ||
| 43 | ask the GPU to perform the transformation from the source content format to the | ||
| 44 | display's format. | ||
| 45 | |||
| 46 | A compositor's mapping function and a driver's mapping function are usually | ||
| 47 | entirely separate concepts. On OSes where a HW vendor has no insight into | ||
| 48 | closed-source compositor code such a vendor will tune their color management | ||
| 49 | code to visually match the compositor's. On other OSes, where both mapping | ||
| 50 | functions are open to an implementer they will ensure both mappings match. | ||
| 51 | |||
| 52 | This results in mapping algorithm lock-in, meaning that no-one alone can | ||
| 53 | experiment with or introduce new mapping algorithms and achieve | ||
| 54 | consistent results regardless of which implementation path is taken. | ||
| 55 | |||
| 56 | Why is Linux different? | ||
| 57 | ======================= | ||
| 58 | |||
| 59 | Unlike other OSes, where there is one compositor for one or more drivers, on | ||
| 60 | Linux we have a many-to-many relationship. Many compositors; many drivers. | ||
| 61 | In addition each compositor vendor or community has their own view of how | ||
| 62 | color management should be done. This is what makes Linux so beautiful. | ||
| 63 | |||
| 64 | This means that a HW vendor can now no longer tune their driver to one | ||
| 65 | compositor, as tuning it to one could make it look fairly different from | ||
| 66 | another compositor's color mapping. | ||
| 67 | |||
| 68 | We need a better solution. | ||
| 69 | |||
| 70 | |||
| 71 | Descriptive API | ||
| 72 | =============== | ||
| 73 | |||
| 74 | An API that describes the source and destination colorspaces is a descriptive | ||
| 75 | API. It describes the input and output color spaces but does not describe | ||
| 76 | how precisely they should be mapped. Such a mapping includes many minute | ||
| 77 | design decision that can greatly affect the look of the final result. | ||
| 78 | |||
| 79 | It is not feasible to describe such mapping with enough detail to ensure the | ||
| 80 | same result from each implementation. In fact, these mappings are a very active | ||
| 81 | research area. | ||
| 82 | |||
| 83 | |||
| 84 | Prescriptive API | ||
| 85 | ================ | ||
| 86 | |||
| 87 | A prescriptive API describes not the source and destination colorspaces. It | ||
| 88 | instead prescribes a recipe for how to manipulate pixel values to arrive at the | ||
| 89 | desired outcome. | ||
| 90 | |||
| 91 | This recipe is generally an ordered list of straight-forward operations, | ||
| 92 | with clear mathematical definitions, such as 1D LUTs, 3D LUTs, matrices, | ||
| 93 | or other operations that can be described in a precise manner. | ||
| 94 | |||
| 95 | |||
| 96 | The Color Pipeline API | ||
| 97 | ====================== | ||
| 98 | |||
| 99 | HW color management pipelines can significantly differ between HW | ||
| 100 | vendors in terms of availability, ordering, and capabilities of HW | ||
| 101 | blocks. This makes a common definition of color management blocks and | ||
| 102 | their ordering nigh impossible. Instead we are defining an API that | ||
| 103 | allows user space to discover the HW capabilities in a generic manner, | ||
| 104 | agnostic of specific drivers and hardware. | ||
| 105 | |||
| 106 | |||
| 107 | drm_colorop Object | ||
| 108 | ================== | ||
| 109 | |||
| 110 | To support the definition of color pipelines we define the DRM core | ||
| 111 | object type drm_colorop. Individual drm_colorop objects will be chained | ||
| 112 | via the NEXT property of a drm_colorop to constitute a color pipeline. | ||
| 113 | Each drm_colorop object is unique, i.e., even if multiple color | ||
| 114 | pipelines have the same operation they won't share the same drm_colorop | ||
| 115 | object to describe that operation. | ||
| 116 | |||
| 117 | Note that drivers are not expected to map drm_colorop objects statically | ||
| 118 | to specific HW blocks. The mapping of drm_colorop objects is entirely a | ||
| 119 | driver-internal detail and can be as dynamic or static as a driver needs | ||
| 120 | it to be. See more in the Driver Implementation Guide section below. | ||
| 121 | |||
| 122 | Each drm_colorop has three core properties: | ||
| 123 | |||
| 124 | TYPE: An enumeration property, defining the type of transformation, such as | ||
| 125 | * enumerated curve | ||
| 126 | * custom (uniform) 1D LUT | ||
| 127 | * 3x3 matrix | ||
| 128 | * 3x4 matrix | ||
| 129 | * 3D LUT | ||
| 130 | * etc. | ||
| 131 | |||
| 132 | Depending on the type of transformation other properties will describe | ||
| 133 | more details. | ||
| 134 | |||
| 135 | BYPASS: A boolean property that can be used to easily put a block into | ||
| 136 | bypass mode. The BYPASS property is not mandatory for a colorop, as long | ||
| 137 | as the entire pipeline can get bypassed by setting the COLOR_PIPELINE on | ||
| 138 | a plane to '0'. | ||
| 139 | |||
| 140 | NEXT: The ID of the next drm_colorop in a color pipeline, or 0 if this | ||
| 141 | drm_colorop is the last in the chain. | ||
| 142 | |||
| 143 | An example of a drm_colorop object might look like one of these:: | ||
| 144 | |||
| 145 | /* 1D enumerated curve */ | ||
| 146 | Color operation 42 | ||
| 147 | ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D enumerated curve | ||
| 148 | ├─ "BYPASS": bool {true, false} | ||
| 149 | ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, sRGB inverse EOTF, PQ EOTF, PQ inverse EOTF, …} | ||
| 150 | └─ "NEXT": immutable color operation ID = 43 | ||
| 151 | |||
| 152 | /* custom 4k entry 1D LUT */ | ||
| 153 | Color operation 52 | ||
| 154 | ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 1D LUT | ||
| 155 | ├─ "BYPASS": bool {true, false} | ||
| 156 | ├─ "SIZE": immutable range = 4096 | ||
| 157 | ├─ "DATA": blob | ||
| 158 | └─ "NEXT": immutable color operation ID = 0 | ||
| 159 | |||
| 160 | /* 17^3 3D LUT */ | ||
| 161 | Color operation 72 | ||
| 162 | ├─ "TYPE": immutable enum {1D enumerated curve, 1D LUT, 3x3 matrix, 3x4 matrix, 3D LUT, etc.} = 3D LUT | ||
| 163 | ├─ "BYPASS": bool {true, false} | ||
| 164 | ├─ "SIZE": immutable range = 17 | ||
| 165 | ├─ "DATA": blob | ||
| 166 | └─ "NEXT": immutable color operation ID = 73 | ||
| 167 | |||
| 168 | drm_colorop extensibility | ||
| 169 | ------------------------- | ||
| 170 | |||
| 171 | Unlike existing DRM core objects, like &drm_plane, drm_colorop is not | ||
| 172 | extensible. This simplifies implementations and keeps all functionality | ||
| 173 | for managing &drm_colorop objects in the DRM core. | ||
| 174 | |||
| 175 | If there is a need one may introduce a simple &drm_colorop_funcs | ||
| 176 | function table in the future, for example to support an IN_FORMATS | ||
| 177 | property on a &drm_colorop. | ||
| 178 | |||
| 179 | If a driver requires the ability to create a driver-specific colorop | ||
| 180 | object they will need to add &drm_colorop func table support with | ||
| 181 | support for the usual functions, like destroy, atomic_duplicate_state, | ||
| 182 | and atomic_destroy_state. | ||
| 183 | |||
| 184 | |||
| 185 | COLOR_PIPELINE Plane Property | ||
| 186 | ============================= | ||
| 187 | |||
| 188 | Color Pipelines are created by a driver and advertised via a new | ||
| 189 | COLOR_PIPELINE enum property on each plane. Values of the property | ||
| 190 | always include object id 0, which is the default and means all color | ||
| 191 | processing is disabled. Additional values will be the object IDs of the | ||
| 192 | first drm_colorop in a pipeline. A driver can create and advertise none, | ||
| 193 | one, or more possible color pipelines. A DRM client will select a color | ||
| 194 | pipeline by setting the COLOR PIPELINE to the respective value. | ||
| 195 | |||
| 196 | NOTE: Many DRM clients will set enumeration properties via the string | ||
| 197 | value, often hard-coding it. Since this enumeration is generated based | ||
| 198 | on the colorop object IDs it is important to perform the Color Pipeline | ||
| 199 | Discovery, described below, instead of hard-coding color pipeline | ||
| 200 | assignment. Drivers might generate the enum strings dynamically. | ||
| 201 | Hard-coded strings might only work for specific drivers on a specific | ||
| 202 | pieces of HW. Color Pipeline Discovery can work universally, as long as | ||
| 203 | drivers implement the required color operations. | ||
| 204 | |||
| 205 | The COLOR_PIPELINE property is only exposed when the | ||
| 206 | DRM_CLIENT_CAP_PLANE_COLOR_PIPELINE is set. Drivers shall ignore any | ||
| 207 | existing pre-blend color operations when this cap is set, such as | ||
| 208 | COLOR_RANGE and COLOR_ENCODING. If drivers want to support COLOR_RANGE | ||
| 209 | or COLOR_ENCODING functionality when the color pipeline client cap is | ||
| 210 | set, they are expected to expose colorops in the pipeline to allow for | ||
| 211 | the appropriate color transformation. | ||
| 212 | |||
| 213 | Setting of the COLOR_PIPELINE plane property or drm_colorop properties | ||
| 214 | is only allowed for userspace that sets this client cap. | ||
| 215 | |||
| 216 | An example of a COLOR_PIPELINE property on a plane might look like this:: | ||
| 217 | |||
| 218 | Plane 10 | ||
| 219 | ├─ "TYPE": immutable enum {Overlay, Primary, Cursor} = Primary | ||
| 220 | ├─ … | ||
| 221 | └─ "COLOR_PIPELINE": enum {0, 42, 52} = 0 | ||
| 222 | |||
| 223 | |||
| 224 | Color Pipeline Discovery | ||
| 225 | ======================== | ||
| 226 | |||
| 227 | A DRM client wanting color management on a drm_plane will: | ||
| 228 | |||
| 229 | 1. Get the COLOR_PIPELINE property of the plane | ||
| 230 | 2. iterate all COLOR_PIPELINE enum values | ||
| 231 | 3. for each enum value walk the color pipeline (via the NEXT pointers) | ||
| 232 | and see if the available color operations are suitable for the | ||
| 233 | desired color management operations | ||
| 234 | |||
| 235 | If userspace encounters an unknown or unsuitable color operation during | ||
| 236 | discovery it does not need to reject the entire color pipeline outright, | ||
| 237 | as long as the unknown or unsuitable colorop has a "BYPASS" property. | ||
| 238 | Drivers will ensure that a bypassed block does not have any effect. | ||
| 239 | |||
| 240 | An example of chained properties to define an AMD pre-blending color | ||
| 241 | pipeline might look like this:: | ||
| 242 | |||
| 243 | Plane 10 | ||
| 244 | ├─ "TYPE" (immutable) = Primary | ||
| 245 | └─ "COLOR_PIPELINE": enum {0, 44} = 0 | ||
| 246 | |||
| 247 | Color operation 44 | ||
| 248 | ├─ "TYPE" (immutable) = 1D enumerated curve | ||
| 249 | ├─ "BYPASS": bool | ||
| 250 | ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF | ||
| 251 | └─ "NEXT" (immutable) = 45 | ||
| 252 | |||
| 253 | Color operation 45 | ||
| 254 | ├─ "TYPE" (immutable) = 3x4 Matrix | ||
| 255 | ├─ "BYPASS": bool | ||
| 256 | ├─ "DATA": blob | ||
| 257 | └─ "NEXT" (immutable) = 46 | ||
| 258 | |||
| 259 | Color operation 46 | ||
| 260 | ├─ "TYPE" (immutable) = 1D enumerated curve | ||
| 261 | ├─ "BYPASS": bool | ||
| 262 | ├─ "CURVE_1D_TYPE": enum {sRGB Inverse EOTF, PQ Inverse EOTF} = sRGB EOTF | ||
| 263 | └─ "NEXT" (immutable) = 47 | ||
| 264 | |||
| 265 | Color operation 47 | ||
| 266 | ├─ "TYPE" (immutable) = 1D LUT | ||
| 267 | ├─ "SIZE": immutable range = 4096 | ||
| 268 | ├─ "DATA": blob | ||
| 269 | └─ "NEXT" (immutable) = 48 | ||
| 270 | |||
| 271 | Color operation 48 | ||
| 272 | ├─ "TYPE" (immutable) = 3D LUT | ||
| 273 | ├─ "DATA": blob | ||
| 274 | └─ "NEXT" (immutable) = 49 | ||
| 275 | |||
| 276 | Color operation 49 | ||
| 277 | ├─ "TYPE" (immutable) = 1D enumerated curve | ||
| 278 | ├─ "BYPASS": bool | ||
| 279 | ├─ "CURVE_1D_TYPE": enum {sRGB EOTF, PQ EOTF} = sRGB EOTF | ||
| 280 | └─ "NEXT" (immutable) = 0 | ||
| 281 | |||
| 282 | |||
| 283 | Color Pipeline Programming | ||
| 284 | ========================== | ||
| 285 | |||
| 286 | Once a DRM client has found a suitable pipeline it will: | ||
| 287 | |||
| 288 | 1. Set the COLOR_PIPELINE enum value to the one pointing at the first | ||
| 289 | drm_colorop object of the desired pipeline | ||
| 290 | 2. Set the properties for all drm_colorop objects in the pipeline to the | ||
| 291 | desired values, setting BYPASS to true for unused drm_colorop blocks, | ||
| 292 | and false for enabled drm_colorop blocks | ||
| 293 | 3. Perform (TEST_ONLY or not) atomic commit with all the other KMS | ||
| 294 | states it wishes to change | ||
| 295 | |||
| 296 | To configure the pipeline for an HDR10 PQ plane and blending in linear | ||
| 297 | space, a compositor might perform an atomic commit with the following | ||
| 298 | property values:: | ||
| 299 | |||
| 300 | Plane 10 | ||
| 301 | └─ "COLOR_PIPELINE" = 42 | ||
| 302 | |||
| 303 | Color operation 42 | ||
| 304 | └─ "BYPASS" = true | ||
| 305 | |||
| 306 | Color operation 44 | ||
| 307 | └─ "BYPASS" = true | ||
| 308 | |||
| 309 | Color operation 45 | ||
| 310 | └─ "BYPASS" = true | ||
| 311 | |||
| 312 | Color operation 46 | ||
| 313 | └─ "BYPASS" = true | ||
| 314 | |||
| 315 | Color operation 47 | ||
| 316 | ├─ "DATA" = Gamut mapping + tone mapping + night mode | ||
| 317 | └─ "BYPASS" = false | ||
| 318 | |||
| 319 | Color operation 48 | ||
| 320 | ├─ "CURVE_1D_TYPE" = PQ EOTF | ||
| 321 | └─ "BYPASS" = false | ||
| 322 | |||
| 323 | |||
| 324 | Driver Implementer's Guide | ||
| 325 | ========================== | ||
| 326 | |||
| 327 | What does this all mean for driver implementations? As noted above the | ||
| 328 | colorops can map to HW directly but don't need to do so. Here are some | ||
| 329 | suggestions on how to think about creating your color pipelines: | ||
| 330 | |||
| 331 | - Try to expose pipelines that use already defined colorops, even if | ||
| 332 | your hardware pipeline is split differently. This allows existing | ||
| 333 | userspace to immediately take advantage of the hardware. | ||
| 334 | |||
| 335 | - Additionally, try to expose your actual hardware blocks as colorops. | ||
| 336 | Define new colorop types where you believe it can offer significant | ||
| 337 | benefits if userspace learns to program them. | ||
| 338 | |||
| 339 | - Avoid defining new colorops for compound operations with very narrow | ||
| 340 | scope. If you have a hardware block for a special operation that | ||
| 341 | cannot be split further, you can expose that as a new colorop type. | ||
| 342 | However, try to not define colorops for "use cases", especially if | ||
| 343 | they require you to combine multiple hardware blocks. | ||
| 344 | |||
| 345 | - Design new colorops as prescriptive, not descriptive; by the | ||
| 346 | mathematical formula, not by the assumed input and output. | ||
| 347 | |||
| 348 | A defined colorop type must be deterministic. The exact behavior of the | ||
| 349 | colorop must be documented entirely, whether via a mathematical formula | ||
| 350 | or some other description. Its operation can depend only on its | ||
| 351 | properties and input and nothing else, allowed error tolerance | ||
| 352 | notwithstanding. | ||
| 353 | |||
| 354 | |||
| 355 | Driver Forward/Backward Compatibility | ||
| 356 | ===================================== | ||
| 357 | |||
| 358 | As this is uAPI drivers can't regress color pipelines that have been | ||
| 359 | introduced for a given HW generation. New HW generations are free to | ||
| 360 | abandon color pipelines advertised for previous generations. | ||
| 361 | Nevertheless, it can be beneficial to carry support for existing color | ||
| 362 | pipelines forward as those will likely already have support in DRM | ||
| 363 | clients. | ||
| 364 | |||
| 365 | Introducing new colorops to a pipeline is fine, as long as they can be | ||
| 366 | bypassed or are purely informational. DRM clients implementing support | ||
| 367 | for the pipeline can always skip unknown properties as long as they can | ||
| 368 | be confident that doing so will not cause unexpected results. | ||
| 369 | |||
| 370 | If a new colorop doesn't fall into one of the above categories | ||
| 371 | (bypassable or informational) the modified pipeline would be unusable | ||
| 372 | for user space. In this case a new pipeline should be defined. | ||
| 373 | |||
| 374 | |||
| 375 | References | ||
| 376 | ========== | ||
| 377 | |||
| 378 | 1. https://lore.kernel.org/dri-devel/QMers3awXvNCQlyhWdTtsPwkp5ie9bze_hD5nAccFW7a_RXlWjYB7MoUW_8CKLT2bSQwIXVi5H6VULYIxCdgvryZoAoJnC5lZgyK1QWn488=@emersion.fr/ | ||
| @@ -35,3 +35,6 @@ host such documentation: · 34 unmodified lines | |||
| 35 | .. toctree:: | 35 | .. toctree:: |
| 36 | 36 | ||
| 37 | i915_vm_bind.rst | 37 | i915_vm_bind.rst |
| 38 | |||
| 39 | .. toctree:: | ||
| 40 | color_pipeline.rst | ||
| @@ -623,6 +623,43 @@ Contact: Thomas Zimmermann <tzimmermann@suse.de>, Simona Vetter · 622 unmodified lines | |||
| 623 | 623 | ||
| 624 | Level: Advanced | 624 | Level: Advanced |
| 625 | 625 | ||
| 626 | Implement a new DUMB_CREATE2 ioctl | ||
| 627 | ---------------------------------- | ||
| 628 | |||
| 629 | The current DUMB_CREATE ioctl is not well defined. Instead of a pixel and | ||
| 630 | framebuffer format, it only accepts a color mode of vague semantics. Assuming | ||
| 631 | a linear framebuffer, the color mode gives an idea of the supported pixel | ||
| 632 | format. But userspace effectively has to guess the correct values. It really | ||
| 633 | only works reliably with framebuffers in XRGB8888. Userspace has begun to | ||
| 634 | workaround these limitations by computing arbitrary format's buffer sizes and | ||
| 635 | calculating their sizes in terms of XRGB8888 pixels. | ||
| 636 | |||
| 637 | One possible solution is a new ioctl DUMB_CREATE2. It should accept a DRM | ||
| 638 | format and a format modifier to resolve the color mode's ambiguity. As | ||
| 639 | framebuffers can be multi-planar, the new ioctl has to return the buffer size, | ||
| 640 | pitch and GEM handle for each individual color plane. | ||
| 641 | |||
| 642 | In the first step, the new ioctl can be limited to the current features of | ||
| 643 | the existing DUMB_CREATE. Individual drivers can then be extended to support | ||
| 644 | multi-planar formats. Rockchip might require this and would be a good candidate. | ||
| 645 | |||
| 646 | It might also be helpful to userspace to query information about the size of | ||
| 647 | a potential buffer, if allocated. Userspace would supply geometry and format; | ||
| 648 | the kernel would return minimal allocation sizes and scanline pitch. There is | ||
| 649 | interest to allocate that memory from another device and provide it to the | ||
| 650 | DRM driver (say via dma-buf). | ||
| 651 | |||
| 652 | Another requested feature is the ability to allocate a buffer by size, without | ||
| 653 | format. Accelators use this for their buffer allocation and it could likely be | ||
| 654 | generalized. | ||
| 655 | |||
| 656 | In addition to the kernel implementation, there must be user-space support | ||
| 657 | for the new ioctl. There's code in Mesa that might be able to use the new | ||
| 658 | call. | ||
| 659 | |||
| 660 | Contact: Thomas Zimmermann <tzimmermann@suse.de> | ||
| 661 | |||
| 662 | Level: Advanced | ||
| 626 | 663 | ||
| 627 | Better Testing | 664 | Better Testing |
| 628 | ============== | 665 | ============== |
| @@ -51,6 +51,97 @@ To disable the driver, use :: · 50 unmodified lines | |||
| 51 | 51 | ||
| 52 | sudo modprobe -r vkms | 52 | sudo modprobe -r vkms |
| 53 | 53 | ||
| 54 | Configuring With Configfs | ||
| 55 | ========================= | ||
| 56 | |||
| 57 | It is possible to create and configure multiple VKMS instances via configfs. | ||
| 58 | |||
| 59 | Start by mounting configfs and loading VKMS:: | ||
| 60 | |||
| 61 | sudo mount -t configfs none /config | ||
| 62 | sudo modprobe vkms | ||
| 63 | |||
| 64 | Once VKMS is loaded, ``/config/vkms`` is created automatically. Each directory | ||
| 65 | under ``/config/vkms`` represents a VKMS instance, create a new one:: | ||
| 66 | |||
| 67 | sudo mkdir /config/vkms/my-vkms | ||
| 68 | |||
| 69 | By default, the instance is disabled:: | ||
| 70 | |||
| 71 | cat /config/vkms/my-vkms/enabled | ||
| 72 | 0 | ||
| 73 | |||
| 74 | And directories are created for each configurable item of the display pipeline:: | ||
| 75 | |||
| 76 | tree /config/vkms/my-vkms | ||
| 77 | ├── connectors | ||
| 78 | ├── crtcs | ||
| 79 | ├── enabled | ||
| 80 | ├── encoders | ||
| 81 | └── planes | ||
| 82 | |||
| 83 | To add items to the display pipeline, create one or more directories under the | ||
| 84 | available paths. | ||
| 85 | |||
| 86 | Start by creating one or more planes:: | ||
| 87 | |||
| 88 | sudo mkdir /config/vkms/my-vkms/planes/plane0 | ||
| 89 | |||
| 90 | Planes have 1 configurable attribute: | ||
| 91 | |||
| 92 | - type: Plane type: 0 overlay, 1 primary, 2 cursor (same values as those | ||
| 93 | exposed by the "type" property of a plane) | ||
| 94 | |||
| 95 | Continue by creating one or more CRTCs:: | ||
| 96 | |||
| 97 | sudo mkdir /config/vkms/my-vkms/crtcs/crtc0 | ||
| 98 | |||
| 99 | CRTCs have 1 configurable attribute: | ||
| 100 | |||
| 101 | - writeback: Enable or disable writeback connector support by writing 1 or 0 | ||
| 102 | |||
| 103 | Next, create one or more encoders:: | ||
| 104 | |||
| 105 | sudo mkdir /config/vkms/my-vkms/encoders/encoder0 | ||
| 106 | |||
| 107 | Last but not least, create one or more connectors:: | ||
| 108 | |||
| 109 | sudo mkdir /config/vkms/my-vkms/connectors/connector0 | ||
| 110 | |||
| 111 | Connectors have 1 configurable attribute: | ||
| 112 | |||
| 113 | - status: Connection status: 1 connected, 2 disconnected, 3 unknown (same values | ||
| 114 | as those exposed by the "status" property of a connector) | ||
| 115 | |||
| 116 | To finish the configuration, link the different pipeline items:: | ||
| 117 | |||
| 118 | sudo ln -s /config/vkms/my-vkms/crtcs/crtc0 /config/vkms/my-vkms/planes/plane0/possible_crtcs | ||
| 119 | sudo ln -s /config/vkms/my-vkms/crtcs/crtc0 /config/vkms/my-vkms/encoders/encoder0/possible_crtcs | ||
| 120 | sudo ln -s /config/vkms/my-vkms/encoders/encoder0 /config/vkms/my-vkms/connectors/connector0/possible_encoders | ||
| 121 | |||
| 122 | Since at least one primary plane is required, make sure to set the right type:: | ||
| 123 | |||
| 124 | echo "1" | sudo tee /config/vkms/my-vkms/planes/plane0/type | ||
| 125 | |||
| 126 | Once you are done configuring the VKMS instance, enable it:: | ||
| 127 | |||
| 128 | echo "1" | sudo tee /config/vkms/my-vkms/enabled | ||
| 129 | |||
| 130 | Finally, you can remove the VKMS instance disabling it:: | ||
| 131 | |||
| 132 | echo "0" | sudo tee /config/vkms/my-vkms/enabled | ||
| 133 | |||
| 134 | And removing the top level directory and its subdirectories:: | ||
| 135 | |||
| 136 | sudo rm /config/vkms/my-vkms/planes/*/possible_crtcs/* | ||
| 137 | sudo rm /config/vkms/my-vkms/encoders/*/possible_crtcs/* | ||
| 138 | sudo rm /config/vkms/my-vkms/connectors/*/possible_encoders/* | ||
| 139 | sudo rmdir /config/vkms/my-vkms/planes/* | ||
| 140 | sudo rmdir /config/vkms/my-vkms/crtcs/* | ||
| 141 | sudo rmdir /config/vkms/my-vkms/encoders/* | ||
| 142 | sudo rmdir /config/vkms/my-vkms/connectors/* | ||
| 143 | sudo rmdir /config/vkms/my-vkms | ||
| 144 | |||
| 54 | Testing With IGT | 145 | Testing With IGT |
| 55 | ================ | 146 | ================ |
| 56 | 147 | ||
| @@ -68,26 +159,23 @@ To return to graphical mode, do:: · 11 unmodified lines | |||
| 68 | 159 | ||
| 69 | sudo systemctl isolate graphical.target | 160 | sudo systemctl isolate graphical.target |
| 70 | 161 | ||
| 71 | Once you are in text only mode, you can run tests using the --device switch | 162 | Once you are in text only mode, you can run tests using the IGT_FORCE_DRIVER |
| 72 | or IGT_DEVICE variable to specify the device filter for the driver we want | 163 | variable to specify the device filter for the driver we want to test. |
| 73 | to test. IGT_DEVICE can also be used with the run-test.sh script to run the | 164 | IGT_FORCE_DRIVER can also be used with the run-tests.sh script to run the |
| 74 | tests for a specific driver:: | 165 | tests for a specific driver:: |
| 75 | 166 | ||
| 76 | sudo ./build/tests/<name of test> --device "sys:/sys/devices/platform/vkms" | 167 | sudo IGT_FORCE_DRIVER="vkms" ./build/tests/<name of test> |
| 77 | sudo IGT_DEVICE="sys:/sys/devices/platform/vkms" ./build/tests/<name of test> | 168 | sudo IGT_FORCE_DRIVER="vkms" ./scripts/run-tests.sh -t <name of test> |
| 78 | sudo IGT_DEVICE="sys:/sys/devices/platform/vkms" ./scripts/run-tests.sh -t <name of test> | ||
| 79 | 169 | ||
| 80 | For example, to test the functionality of the writeback library, | 170 | For example, to test the functionality of the writeback library, |
| 81 | we can run the kms_writeback test:: | 171 | we can run the kms_writeback test:: |
| 82 | 172 | ||
| 83 | sudo ./build/tests/kms_writeback --device "sys:/sys/devices/platform/vkms" | 173 | sudo IGT_FORCE_DRIVER="vkms" ./build/tests/kms_writeback |
| 84 | sudo IGT_DEVICE="sys:/sys/devices/platform/vkms" ./build/tests/kms_writeback | 174 | sudo IGT_FORCE_DRIVER="vkms" ./scripts/run-tests.sh -t kms_writeback |
| 85 | sudo IGT_DEVICE="sys:/sys/devices/platform/vkms" ./scripts/run-tests.sh -t kms_writeback | ||
| 86 | 175 | ||
| 87 | You can also run subtests if you do not want to run the entire test:: | 176 | You can also run subtests if you do not want to run the entire test:: |
| 88 | 177 | ||
| 89 | sudo ./build/tests/kms_flip --run-subtest basic-plain-flip --device "sys:/sys/devices/platform/vkms" | 178 | sudo IGT_FORCE_DRIVER="vkms" ./build/tests/kms_flip --run-subtest basic-plain-flip |
| 90 | sudo IGT_DEVICE="sys:/sys/devices/platform/vkms" ./build/tests/kms_flip --run-subtest basic-plain-flip | ||
| 91 | 179 | ||
| 92 | Testing With KUnit | 180 | Testing With KUnit |
| 93 | ================== | 181 | ================== |
| @@ -147,21 +235,14 @@ Runtime Configuration · 53 unmodified lines | |||
| 147 | --------------------- | 235 | --------------------- |
| 148 | 236 | ||
| 149 | We want to be able to reconfigure vkms instance without having to reload the | 237 | We want to be able to reconfigure vkms instance without having to reload the |
| 150 | module. Use/Test-cases: | 238 | module through configfs. Use/Test-cases: |
| 151 | 239 | ||
| 152 | - Hotplug/hotremove connectors on the fly (to be able to test DP MST handling | 240 | - Hotplug/hotremove connectors on the fly (to be able to test DP MST handling |
| 153 | of compositors). | 241 | of compositors). |
| 154 | 242 | ||
| 155 | - Configure planes/crtcs/connectors (we'd need some code to have more than 1 of | ||
| 156 | them first). | ||
| 157 | |||
| 158 | - Change output configuration: Plug/unplug screens, change EDID, allow changing | 243 | - Change output configuration: Plug/unplug screens, change EDID, allow changing |
| 159 | the refresh rate. | 244 | the refresh rate. |
| 160 | 245 | ||
| 161 | The currently proposed solution is to expose vkms configuration through | ||
| 162 | configfs. All existing module options should be supported through configfs | ||
| 163 | too. | ||
| 164 | |||
| 165 | Writeback support | 246 | Writeback support |
| 166 | ----------------- | 247 | ----------------- |
| 167 | 248 | ||
| @@ -14,6 +14,7 @@ DG2, etc is provided to prototype the driver. · 13 unmodified lines | |||
| 14 | xe_mm | 14 | xe_mm |
| 15 | xe_map | 15 | xe_map |
| 16 | xe_migrate | 16 | xe_migrate |
| 17 | xe_exec_queue | ||
| 17 | xe_cs | 18 | xe_cs |
| 18 | xe_pm | 19 | xe_pm |
| 19 | xe_gt_freq | 20 | xe_gt_freq |
| @@ -0,0 +1,20 @@ | |||
| 1 | .. SPDX-License-Identifier: (GPL-2.0+ OR MIT) | ||
| 2 | |||
| 3 | =============== | ||
| 4 | Execution Queue | ||
| 5 | =============== | ||
| 6 | |||
| 7 | .. kernel-doc:: drivers/gpu/drm/xe/xe_exec_queue.c | ||
| 8 | :doc: Execution Queue | ||
| 9 | |||
| 10 | Internal API | ||
| 11 | ============ | ||
| 12 | |||
| 13 | .. kernel-doc:: drivers/gpu/drm/xe/xe_exec_queue_types.h | ||
| 14 | :internal: | ||
| 15 | |||
| 16 | .. kernel-doc:: drivers/gpu/drm/xe/xe_exec_queue.h | ||
| 17 | :internal: | ||
| 18 | |||
| 19 | .. kernel-doc:: drivers/gpu/drm/xe/xe_exec_queue.c | ||
| 20 | :internal: | ||
| @@ -7,6 +7,9 @@ Xe GT Frequency Management · 6 unmodified lines | |||
| 7 | .. kernel-doc:: drivers/gpu/drm/xe/xe_gt_freq.c | 7 | .. kernel-doc:: drivers/gpu/drm/xe/xe_gt_freq.c |
| 8 | :doc: Xe GT Frequency Management | 8 | :doc: Xe GT Frequency Management |
| 9 | 9 | ||
| 10 | .. kernel-doc:: drivers/gpu/drm/xe/xe_gt_throttle.c | ||
| 11 | :doc: Xe GT Throttle | ||
| 12 | |||
| 10 | Internal API | 13 | Internal API |
| 11 | ============ | 14 | ============ |
| 12 | 15 | ||
| @@ -16,13 +16,52 @@ following heaps: · 15 unmodified lines | |||
| 16 | 16 | ||
| 17 | - The ``system`` heap allocates virtually contiguous, cacheable, buffers. | 17 | - The ``system`` heap allocates virtually contiguous, cacheable, buffers. |
| 18 | 18 | ||
| 19 | - The ``cma`` heap allocates physically contiguous, cacheable, | 19 | - The ``default_cma_region`` heap allocates physically contiguous, |
| 20 | buffers. Only present if a CMA region is present. Such a region is | 20 | cacheable, buffers. Only present if a CMA region is present. Such a |
| 21 | usually created either through the kernel commandline through the | 21 | region is usually created either through the kernel commandline |
| 22 | ``cma`` parameter, a memory region Device-Tree node with the | 22 | through the ``cma`` parameter, a memory region Device-Tree node with |
| 23 | ``linux,cma-default`` property set, or through the ``CMA_SIZE_MBYTES`` or | 23 | the ``linux,cma-default`` property set, or through the |
| 24 | ``CMA_SIZE_PERCENTAGE`` Kconfig options. The heap's name in devtmpfs is | 24 | ``CMA_SIZE_MBYTES`` or ``CMA_SIZE_PERCENTAGE`` Kconfig options. Prior |
| 25 | ``default_cma_region``. For backwards compatibility, when the | 25 | to Linux 6.17, its name wasn't stable and could be called |
| 26 | ``DMABUF_HEAPS_CMA_LEGACY`` Kconfig option is set, a duplicate node is | 26 | ``reserved``, ``linux,cma``, or ``default-pool``, depending on the |
| 27 | created following legacy naming conventions; the legacy name might be | 27 | platform. |
| 28 | ``reserved``, ``linux,cma``, or ``default-pool``. | 28 | |
| 29 | - A heap will be created for each reusable region in the device tree | ||
| 30 | with the ``shared-dma-pool`` compatible, using the full device tree | ||
| 31 | node name as its name. The buffer semantics are identical to | ||
| 32 | ``default-cma-region``. | ||
| 33 | |||
| 34 | Naming Convention | ||
| 35 | ================= | ||
| 36 | |||
| 37 | ``dma-buf`` heaps name should meet a number of constraints: | ||
| 38 | |||
| 39 | - The name must be stable, and must not change from one version to the other. | ||
| 40 | Userspace identifies heaps by their name, so if the names ever change, we | ||
| 41 | would be likely to introduce regressions. | ||
| 42 | |||
| 43 | - The name must describe the memory region the heap will allocate from, and | ||
| 44 | must uniquely identify it in a given platform. Since userspace applications | ||
| 45 | use the heap name as the discriminant, it must be able to tell which heap it | ||
| 46 | wants to use reliably if there's multiple heaps. | ||
| 47 | |||
| 48 | - The name must not mention implementation details, such as the allocator. The | ||
| 49 | heap driver will change over time, and implementation details when it was | ||
| 50 | introduced might not be relevant in the future. | ||
| 51 | |||
| 52 | - The name should describe properties of the buffers that would be allocated. | ||
| 53 | Doing so will make heap identification easier for userspace. Such properties | ||
| 54 | are: | ||
| 55 | |||
| 56 | - ``contiguous`` for physically contiguous buffers; | ||
| 57 | |||
| 58 | - ``protected`` for encrypted buffers not accessible the OS; | ||
| 59 | |||
| 60 | - The name may describe intended usage. Doing so will make heap identification | ||
| 61 | easier for userspace applications and users. | ||
| 62 | |||
| 63 | For example, assuming a platform with a reserved memory region located | ||
| 64 | at the RAM address 0x42000000, intended to allocate video framebuffers, | ||
| 65 | physically contiguous, and backed by the CMA kernel allocator, good | ||
| 66 | names would be ``memory@42000000-contiguous`` or ``video@42000000``, but | ||
| 67 | ``cma-video`` wouldn't. | ||
| @@ -1081,7 +1081,7 @@ M: Austin Zheng <austin.zheng@amd.com> · 1080 unmodified lines | |||
| 1081 | M: Jun Lei <jun.lei@amd.com> | 1081 | M: Jun Lei <jun.lei@amd.com> |
| 1082 | S: Supported | 1082 | S: Supported |
| 1083 | F: drivers/gpu/drm/amd/display/dc/dml/ | 1083 | F: drivers/gpu/drm/amd/display/dc/dml/ |
| 1084 | F: drivers/gpu/drm/amd/display/dc/dml2/ | 1084 | F: drivers/gpu/drm/amd/display/dc/dml2_0/ |
| 1085 | 1085 | ||
| 1086 | AMD FAM15H PROCESSOR POWER MONITORING DRIVER | 1086 | AMD FAM15H PROCESSOR POWER MONITORING DRIVER |
| 1087 | M: Huang Rui <ray.huang@amd.com> | 1087 | M: Huang Rui <ray.huang@amd.com> |
| @@ -2022,6 +2022,15 @@ F: arch/arm64/include/asm/arch_timer.h · 934 unmodified lines | |||
| 2022 | F: drivers/clocksource/arm_arch_timer.c | 2022 | F: drivers/clocksource/arm_arch_timer.c |
| 2023 | F: drivers/clocksource/arm_arch_timer_mmio.c | 2023 | F: drivers/clocksource/arm_arch_timer_mmio.c |
| 2024 | 2024 | ||
| 2025 | ARM ETHOS-U NPU DRIVER | ||
| 2026 | M: Rob Herring (Arm) <robh@kernel.org> | ||
| 2027 | M: Tomeu Vizoso <tomeu@tomeuvizoso.net> | ||
| 2028 | L: dri-devel@lists.freedesktop.org | ||
| 2029 | S: Supported | ||
| 2030 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | ||
| 2031 | F: drivers/accel/ethosu/ | ||
| 2032 | F: include/uapi/drm/ethosu_accel.h | ||
| 2033 | |||
| 2025 | ARM GENERIC INTERRUPT CONTROLLER DRIVERS | 2034 | ARM GENERIC INTERRUPT CONTROLLER DRIVERS |
| 2026 | M: Marc Zyngier <maz@kernel.org> | 2035 | M: Marc Zyngier <maz@kernel.org> |
| 2027 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 2036 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| @@ -2097,7 +2106,8 @@ F: drivers/gpu/drm/arm/display/komeda/ · 69 unmodified lines | |||
| 2097 | ARM MALI PANFROST DRM DRIVER | 2106 | ARM MALI PANFROST DRM DRIVER |
| 2098 | M: Boris Brezillon <boris.brezillon@collabora.com> | 2107 | M: Boris Brezillon <boris.brezillon@collabora.com> |
| 2099 | M: Rob Herring <robh@kernel.org> | 2108 | M: Rob Herring <robh@kernel.org> |
| 2100 | R: Steven Price <steven.price@arm.com> | 2109 | M: Steven Price <steven.price@arm.com> |
| 2110 | M: Adrián Larumbe <adrian.larumbe@collabora.com> | ||
| 2101 | L: dri-devel@lists.freedesktop.org | 2111 | L: dri-devel@lists.freedesktop.org |
| 2102 | S: Supported | 2112 | S: Supported |
| 2103 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 2113 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| @@ -2317,7 +2327,7 @@ S: Maintained · 213 unmodified lines | |||
| 2317 | F: drivers/clk/sunxi/ | 2327 | F: drivers/clk/sunxi/ |
| 2318 | 2328 | ||
| 2319 | ARM/Allwinner sunXi SoC support | 2329 | ARM/Allwinner sunXi SoC support |
| 2320 | M: Chen-Yu Tsai <wens@csie.org> | 2330 | M: Chen-Yu Tsai <wens@kernel.org> |
| 2321 | M: Jernej Skrabec <jernej.skrabec@gmail.com> | 2331 | M: Jernej Skrabec <jernej.skrabec@gmail.com> |
| 2322 | M: Samuel Holland <samuel@sholland.org> | 2332 | M: Samuel Holland <samuel@sholland.org> |
| 2323 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) | 2333 | L: linux-arm-kernel@lists.infradead.org (moderated for non-subscribers) |
| @@ -7331,6 +7341,7 @@ F: Documentation/userspace-api/dma-buf-alloc-exchange.rst · 5007 unmodified lines | |||
| 7331 | F: drivers/dma-buf/ | 7341 | F: drivers/dma-buf/ |
| 7332 | F: include/linux/*fence.h | 7342 | F: include/linux/*fence.h |
| 7333 | F: include/linux/dma-buf.h | 7343 | F: include/linux/dma-buf.h |
| 7344 | F: include/linux/dma-buf/ | ||
| 7334 | F: include/linux/dma-resv.h | 7345 | F: include/linux/dma-resv.h |
| 7335 | K: \bdma_(?:buf|fence|resv)\b | 7346 | K: \bdma_(?:buf|fence|resv)\b |
| 7336 | 7347 | ||
| @@ -7654,8 +7665,7 @@ F: drivers/accel/ · 317 unmodified lines | |||
| 7654 | F: include/drm/drm_accel.h | 7665 | F: include/drm/drm_accel.h |
| 7655 | 7666 | ||
| 7656 | DRM DRIVER FOR ALLWINNER DE2 AND DE3 ENGINE | 7667 | DRM DRIVER FOR ALLWINNER DE2 AND DE3 ENGINE |
| 7657 | M: Maxime Ripard <mripard@kernel.org> | 7668 | M: Chen-Yu Tsai <wens@kernel.org> |
| 7658 | M: Chen-Yu Tsai <wens@csie.org> | ||
| 7659 | R: Jernej Skrabec <jernej.skrabec@gmail.com> | 7669 | R: Jernej Skrabec <jernej.skrabec@gmail.com> |
| 7660 | L: dri-devel@lists.freedesktop.org | 7670 | L: dri-devel@lists.freedesktop.org |
| 7661 | S: Supported | 7671 | S: Supported |
| @@ -7764,7 +7774,8 @@ F: Documentation/devicetree/bindings/display/panel/panel-edp.yaml · 102 unmodified lines | |||
| 7764 | F: drivers/gpu/drm/panel/panel-edp.c | 7774 | F: drivers/gpu/drm/panel/panel-edp.c |
| 7765 | 7775 | ||
| 7766 | DRM DRIVER FOR GENERIC USB DISPLAY | 7776 | DRM DRIVER FOR GENERIC USB DISPLAY |
| 7767 | S: Orphan | 7777 | M: Ruben Wauters <rubenru09@aol.com> |
| 7778 | S: Maintained | ||
| 7768 | W: https://github.com/notro/gud/wiki | 7779 | W: https://github.com/notro/gud/wiki |
| 7769 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 7780 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| 7770 | F: drivers/gpu/drm/gud/ | 7781 | F: drivers/gpu/drm/gud/ |
| @@ -7886,6 +7897,7 @@ DRM DRIVER for Qualcomm Adreno GPUs · 115 unmodified lines | |||
| 7886 | M: Rob Clark <robin.clark@oss.qualcomm.com> | 7897 | M: Rob Clark <robin.clark@oss.qualcomm.com> |
| 7887 | R: Sean Paul <sean@poorly.run> | 7898 | R: Sean Paul <sean@poorly.run> |
| 7888 | R: Konrad Dybcio <konradybcio@kernel.org> | 7899 | R: Konrad Dybcio <konradybcio@kernel.org> |
| 7900 | R: Akhil P Oommen <akhilpo@oss.qualcomm.com> | ||
| 7889 | L: linux-arm-msm@vger.kernel.org | 7901 | L: linux-arm-msm@vger.kernel.org |
| 7890 | L: dri-devel@lists.freedesktop.org | 7902 | L: dri-devel@lists.freedesktop.org |
| 7891 | L: freedreno@lists.freedesktop.org | 7903 | L: freedreno@lists.freedesktop.org |
| @@ -7905,7 +7917,7 @@ DRM DRIVER for Qualcomm display hardware · 13 unmodified lines | |||
| 7905 | M: Rob Clark <robin.clark@oss.qualcomm.com> | 7917 | M: Rob Clark <robin.clark@oss.qualcomm.com> |
| 7906 | M: Dmitry Baryshkov <lumag@kernel.org> | 7918 | M: Dmitry Baryshkov <lumag@kernel.org> |
| 7907 | R: Abhinav Kumar <abhinav.kumar@linux.dev> | 7919 | R: Abhinav Kumar <abhinav.kumar@linux.dev> |
| 7908 | R: Jessica Zhang <jessica.zhang@oss.qualcomm.com> | 7920 | R: Jessica Zhang <jesszhan0024@gmail.com> |
| 7909 | R: Sean Paul <sean@poorly.run> | 7921 | R: Sean Paul <sean@poorly.run> |
| 7910 | R: Marijn Suijten <marijn.suijten@somainline.org> | 7922 | R: Marijn Suijten <marijn.suijten@somainline.org> |
| 7911 | L: linux-arm-msm@vger.kernel.org | 7923 | L: linux-arm-msm@vger.kernel.org |
| @@ -8074,12 +8086,25 @@ S: Maintained · 162 unmodified lines | |||
| 8074 | F: Documentation/devicetree/bindings/display/panel/samsung,s6d7aa0.yaml | 8086 | F: Documentation/devicetree/bindings/display/panel/samsung,s6d7aa0.yaml |
| 8075 | F: drivers/gpu/drm/panel/panel-samsung-s6d7aa0.c | 8087 | F: drivers/gpu/drm/panel/panel-samsung-s6d7aa0.c |
| 8076 | 8088 | ||
| 8089 | DRM DRIVER FOR SAMSUNG S6E3FC2X01 DDIC | ||
| 8090 | M: David Heidelberg <david@ixit.cz> | ||
| 8091 | S: Maintained | ||
| 8092 | F: Documentation/devicetree/bindings/display/panel/samsung,s6e3fc2x01.yaml | ||
| 8093 | F: drivers/gpu/drm/panel/panel-samsung-s6e3fc2x01.c | ||
| 8094 | |||
| 8077 | DRM DRIVER FOR SAMSUNG S6E3HA8 PANELS | 8095 | DRM DRIVER FOR SAMSUNG S6E3HA8 PANELS |
| 8078 | M: Dzmitry Sankouski <dsankouski@gmail.com> | 8096 | M: Dzmitry Sankouski <dsankouski@gmail.com> |
| 8079 | S: Maintained | 8097 | S: Maintained |
| 8080 | F: Documentation/devicetree/bindings/display/panel/samsung,s6e3ha8.yaml | 8098 | F: Documentation/devicetree/bindings/display/panel/samsung,s6e3ha8.yaml |
| 8081 | F: drivers/gpu/drm/panel/panel-samsung-s6e3ha8.c | 8099 | F: drivers/gpu/drm/panel/panel-samsung-s6e3ha8.c |
| 8082 | 8100 | ||
| 8101 | DRM DRIVER FOR SAMSUNG SOFEF00 DDIC | ||
| 8102 | M: David Heidelberg <david@ixit.cz> | ||
| 8103 | M: Casey Connolly <casey.connolly@linaro.org> | ||
| 8104 | S: Maintained | ||
| 8105 | F: Documentation/devicetree/bindings/display/panel/samsung,sofef00.yaml | ||
| 8106 | F: drivers/gpu/drm/panel/panel-samsung-sofef00.c | ||
| 8107 | |||
| 8083 | DRM DRIVER FOR SHARP MEMORY LCD | 8108 | DRM DRIVER FOR SHARP MEMORY LCD |
| 8084 | M: Alex Lanzano <lanzano.alex@gmail.com> | 8109 | M: Alex Lanzano <lanzano.alex@gmail.com> |
| 8085 | S: Maintained | 8110 | S: Maintained |
| @@ -8264,12 +8289,12 @@ S: Supported · 178 unmodified lines | |||
| 8264 | W: https://drm.pages.freedesktop.org/maintainer-tools/drm-rust.html | 8289 | W: https://drm.pages.freedesktop.org/maintainer-tools/drm-rust.html |
| 8265 | T: git https://gitlab.freedesktop.org/drm/rust/kernel.git | 8290 | T: git https://gitlab.freedesktop.org/drm/rust/kernel.git |
| 8266 | F: drivers/gpu/drm/nova/ | 8291 | F: drivers/gpu/drm/nova/ |
| 8292 | F: drivers/gpu/drm/tyr/ | ||
| 8267 | F: drivers/gpu/nova-core/ | 8293 | F: drivers/gpu/nova-core/ |
| 8268 | F: rust/kernel/drm/ | 8294 | F: rust/kernel/drm/ |
| 8269 | 8295 | ||
| 8270 | DRM DRIVERS FOR ALLWINNER A10 | 8296 | DRM DRIVERS FOR ALLWINNER A10 |
| 8271 | M: Maxime Ripard <mripard@kernel.org> | 8297 | M: Chen-Yu Tsai <wens@kernel.org> |
| 8272 | M: Chen-Yu Tsai <wens@csie.org> | ||
| 8273 | L: dri-devel@lists.freedesktop.org | 8298 | L: dri-devel@lists.freedesktop.org |
| 8274 | S: Supported | 8299 | S: Supported |
| 8275 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 8300 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| @@ -8596,6 +8621,7 @@ S: Supported · 320 unmodified lines | |||
| 8596 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 8621 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| 8597 | F: drivers/gpu/drm/scheduler/ | 8622 | F: drivers/gpu/drm/scheduler/ |
| 8598 | F: include/drm/gpu_scheduler.h | 8623 | F: include/drm/gpu_scheduler.h |
| 8624 | F: include/drm/spsc_queue.h | ||
| 8599 | 8625 | ||
| 8600 | DRM GPUVM | 8626 | DRM GPUVM |
| 8601 | M: Danilo Krummrich <dakr@kernel.org> | 8627 | M: Danilo Krummrich <dakr@kernel.org> |
| @@ -8618,7 +8644,7 @@ F: drivers/gpu/drm/clients/drm_log.c · 16 unmodified lines | |||
| 8618 | 8644 | ||
| 8619 | DRM PANEL DRIVERS | 8645 | DRM PANEL DRIVERS |
| 8620 | M: Neil Armstrong <neil.armstrong@linaro.org> | 8646 | M: Neil Armstrong <neil.armstrong@linaro.org> |
| 8621 | R: Jessica Zhang <jessica.zhang@oss.qualcomm.com> | 8647 | R: Jessica Zhang <jesszhan0024@gmail.com> |
| 8622 | L: dri-devel@lists.freedesktop.org | 8648 | L: dri-devel@lists.freedesktop.org |
| 8623 | S: Maintained | 8649 | S: Maintained |
| 8624 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git | 8650 | T: git https://gitlab.freedesktop.org/drm/misc/kernel.git |
| @@ -27832,7 +27858,7 @@ F: drivers/acpi/pmic/intel_pmic_xpower.c · 19207 unmodified lines | |||
| 27832 | N: axp288 | 27858 | N: axp288 |
| 27833 | 27859 | ||
| 27834 | X-POWERS MULTIFUNCTION PMIC DEVICE DRIVERS | 27860 | X-POWERS MULTIFUNCTION PMIC DEVICE DRIVERS |
| 27835 | M: Chen-Yu Tsai <wens@csie.org> | 27861 | M: Chen-Yu Tsai <wens@kernel.org> |
| 27836 | L: linux-kernel@vger.kernel.org | 27862 | L: linux-kernel@vger.kernel.org |
| 27837 | S: Maintained | 27863 | S: Maintained |
| 27838 | N: axp[128] | 27864 | N: axp[128] |
| @@ -34,6 +34,41 @@ · 33 unmodified lines | |||
| 34 | status = "disabled"; | 34 | status = "disabled"; |
| 35 | }; | 35 | }; |
| 36 | 36 | ||
| 37 | display-subsystem { | ||
| 38 | compatible = "st,sti-display-subsystem"; | ||
| 39 | ports = <&compositor>, <&hqvdp>, <&tvout>, <&sti_hdmi>; | ||
| 40 | |||
| 41 | assigned-clocks = <&clk_s_d2_quadfs 0>, | ||
| 42 | <&clk_s_d2_quadfs 1>, | ||
| 43 | <&clk_s_c0_pll1 0>, | ||
| 44 | <&clk_s_c0_flexgen CLK_COMPO_DVP>, | ||
| 45 | <&clk_s_c0_flexgen CLK_MAIN_DISP>, | ||
| 46 | <&clk_s_d2_flexgen CLK_PIX_MAIN_DISP>, | ||
| 47 | <&clk_s_d2_flexgen CLK_PIX_AUX_DISP>, | ||
| 48 | <&clk_s_d2_flexgen CLK_PIX_GDP1>, | ||
| 49 | <&clk_s_d2_flexgen CLK_PIX_GDP2>, | ||
| 50 | <&clk_s_d2_flexgen CLK_PIX_GDP3>, | ||
| 51 | <&clk_s_d2_flexgen CLK_PIX_GDP4>; | ||
| 52 | |||
| 53 | assigned-clock-parents = <0>, | ||
| 54 | <0>, | ||
| 55 | <0>, | ||
| 56 | <&clk_s_c0_pll1 0>, | ||
| 57 | <&clk_s_c0_pll1 0>, | ||
| 58 | <&clk_s_d2_quadfs 0>, | ||
| 59 | <&clk_s_d2_quadfs 1>, | ||
| 60 | <&clk_s_d2_quadfs 0>, | ||
| 61 | <&clk_s_d2_quadfs 0>, | ||
| 62 | <&clk_s_d2_quadfs 0>, | ||
| 63 | <&clk_s_d2_quadfs 0>; | ||
| 64 | |||
| 65 | assigned-clock-rates = <297000000>, | ||
| 66 | <297000000>, | ||
| 67 | <0>, | ||
| 68 | <400000000>, | ||
| 69 | <400000000>; | ||
| 70 | }; | ||
| 71 | |||
| 37 | soc { | 72 | soc { |
| 38 | ohci0: usb@9a03c00 { | 73 | ohci0: usb@9a03c00 { |
| 39 | compatible = "st,st-ohci-300x"; | 74 | compatible = "st,st-ohci-300x"; |
| @@ -99,151 +134,174 @@ · 59 unmodified lines | |||
| 99 | status = "disabled"; | 134 | status = "disabled"; |
| 100 | }; | 135 | }; |
| 101 | 136 | ||
| 102 | sti-display-subsystem@0 { | 137 | compositor: display-controller@9d11000 { |
| 103 | compatible = "st,sti-display-subsystem"; | 138 | compatible = "st,stih407-compositor"; |
| 104 | #address-cells = <1>; | 139 | reg = <0x9d11000 0x1000>; |
| 105 | #size-cells = <1>; | 140 | |
| 106 | 141 | clock-names = "compo_main", | |
| 107 | reg = <0 0>; | 142 | "compo_aux", |
| 108 | assigned-clocks = <&clk_s_d2_quadfs 0>, | 143 | "pix_main", |
| 109 | <&clk_s_d2_quadfs 1>, | 144 | "pix_aux", |
| 110 | <&clk_s_c0_pll1 0>, | 145 | "pix_gdp1", |
| 111 | <&clk_s_c0_flexgen CLK_COMPO_DVP>, | 146 | "pix_gdp2", |
| 112 | <&clk_s_c0_flexgen CLK_MAIN_DISP>, | 147 | "pix_gdp3", |
| 113 | <&clk_s_d2_flexgen CLK_PIX_MAIN_DISP>, | 148 | "pix_gdp4", |
| 114 | <&clk_s_d2_flexgen CLK_PIX_AUX_DISP>, | 149 | "main_parent", |
| 115 | <&clk_s_d2_flexgen CLK_PIX_GDP1>, | 150 | "aux_parent"; |
| 116 | <&clk_s_d2_flexgen CLK_PIX_GDP2>, | 151 | |
| 117 | <&clk_s_d2_flexgen CLK_PIX_GDP3>, | 152 | clocks = <&clk_s_c0_flexgen CLK_COMPO_DVP>, |
| 118 | <&clk_s_d2_flexgen CLK_PIX_GDP4>; | 153 | <&clk_s_c0_flexgen CLK_COMPO_DVP>, |
| 119 | 154 | <&clk_s_d2_flexgen CLK_PIX_MAIN_DISP>, | |
| 120 | assigned-clock-parents = <0>, | 155 | <&clk_s_d2_flexgen CLK_PIX_AUX_DISP>, |
| 121 | <0>, | 156 | <&clk_s_d2_flexgen CLK_PIX_GDP1>, |
| 122 | <0>, | 157 | <&clk_s_d2_flexgen CLK_PIX_GDP2>, |
| 123 | <&clk_s_c0_pll1 0>, | 158 | <&clk_s_d2_flexgen CLK_PIX_GDP3>, |
| 124 | <&clk_s_c0_pll1 0>, | 159 | <&clk_s_d2_flexgen CLK_PIX_GDP4>, |
| 125 | <&clk_s_d2_quadfs 0>, | 160 | <&clk_s_d2_quadfs 0>, |
| 126 | <&clk_s_d2_quadfs 1>, | 161 | <&clk_s_d2_quadfs 1>; |
| 127 | <&clk_s_d2_quadfs 0>, | 162 | |
| 163 | reset-names = "compo-main", "compo-aux"; | ||
| 164 | resets = <&softreset STIH407_COMPO_SOFTRESET>, | ||
| 165 | <&softreset STIH407_COMPO_SOFTRESET>; | ||
| 166 | st,vtg = <&vtg_main>, <&vtg_aux>; | ||
| 167 | |||
| 168 | ports { | ||
| 169 | #address-cells = <1>; | ||
| 170 | #size-cells = <0>; | ||
| 171 | |||
| 172 | port@0 { | ||
| 173 | reg = <0>; | ||
| 174 | compo_main_out: endpoint { | ||
| 175 | remote-endpoint = <&tvout_in0>; | ||
| 176 | }; | ||
| 177 | }; | ||
| 178 | |||
| 179 | port@1 { | ||
| 180 | reg = <1>; | ||
| 181 | compo_aux_out: endpoint { | ||
| 182 | remote-endpoint = <&tvout_in1>; | ||
| 183 | }; | ||
| 184 | }; | ||
| 185 | }; | ||
| 186 | }; | ||
| 187 | |||
| 188 | tvout: encoder@8d08000 { | ||
| 189 | compatible = "st,stih407-tvout"; | ||
| 190 | reg = <0x8d08000 0x1000>; | ||
| 191 | reg-names = "tvout-reg"; | ||
| 192 | reset-names = "tvout"; | ||
| 193 | resets = <&softreset STIH407_HDTVOUT_SOFTRESET>; | ||
| 194 | assigned-clocks = <&clk_s_d2_flexgen CLK_PIX_HDMI>, | ||
| 195 | <&clk_s_d2_flexgen CLK_TMDS_HDMI>, | ||
| 196 | <&clk_s_d2_flexgen CLK_REF_HDMIPHY>, | ||
| 197 | <&clk_s_d0_flexgen CLK_PCM_0>, | ||
| 198 | <&clk_s_d2_flexgen CLK_PIX_HDDAC>, | ||
| 199 | <&clk_s_d2_flexgen CLK_HDDAC>; | ||
| 200 | |||
| 201 | assigned-clock-parents = <&clk_s_d2_quadfs 0>, | ||
| 202 | <&clk_tmdsout_hdmi>, | ||
| 128 | <&clk_s_d2_quadfs 0>, | 203 | <&clk_s_d2_quadfs 0>, |
| 204 | <&clk_s_d0_quadfs 0>, | ||
| 129 | <&clk_s_d2_quadfs 0>, | 205 | <&clk_s_d2_quadfs 0>, |
| 130 | <&clk_s_d2_quadfs 0>; | 206 | <&clk_s_d2_quadfs 0>; |
| 131 | 207 | ||
| 132 | assigned-clock-rates = <297000000>, | 208 | ports { |
| 133 | <297000000>, | ||
| 134 | <0>, | ||
| 135 | <400000000>, | ||
| 136 | <400000000>; | ||
| 137 | |||
| 138 | ranges; | ||
| 139 | |||
| 140 | sti-compositor@9d11000 { | ||
| 141 | compatible = "st,stih407-compositor"; | ||
| 142 | reg = <0x9d11000 0x1000>; | ||
| 143 | |||
| 144 | clock-names = "compo_main", | ||
| 145 | "compo_aux", | ||
| 146 | "pix_main", | ||
| 147 | "pix_aux", | ||
| 148 | "pix_gdp1", | ||
| 149 | "pix_gdp2", | ||
| 150 | "pix_gdp3", | ||
| 151 | "pix_gdp4", | ||
| 152 | "main_parent", | ||
| 153 | "aux_parent"; | ||
| 154 | |||
| 155 | clocks = <&clk_s_c0_flexgen CLK_COMPO_DVP>, | ||
| 156 | <&clk_s_c0_flexgen CLK_COMPO_DVP>, | ||
| 157 | <&clk_s_d2_flexgen CLK_PIX_MAIN_DISP>, | ||
| 158 | <&clk_s_d2_flexgen CLK_PIX_AUX_DISP>, | ||
| 159 | <&clk_s_d2_flexgen CLK_PIX_GDP1>, | ||
| 160 | <&clk_s_d2_flexgen CLK_PIX_GDP2>, | ||
| 161 | <&clk_s_d2_flexgen CLK_PIX_GDP3>, | ||
| 162 | <&clk_s_d2_flexgen CLK_PIX_GDP4>, | ||
| 163 | <&clk_s_d2_quadfs 0>, | ||
| 164 | <&clk_s_d2_quadfs 1>; | ||
| 165 | |||
| 166 | reset-names = "compo-main", "compo-aux"; | ||
| 167 | resets = <&softreset STIH407_COMPO_SOFTRESET>, | ||
| 168 | <&softreset STIH407_COMPO_SOFTRESET>; | ||
| 169 | st,vtg = <&vtg_main>, <&vtg_aux>; | ||
| 170 | }; | ||
| 171 | |||
| 172 | sti-tvout@8d08000 { | ||
| 173 | compatible = "st,stih407-tvout"; | ||
| 174 | reg = <0x8d08000 0x1000>; | ||
| 175 | reg-names = "tvout-reg"; | ||
| 176 | reset-names = "tvout"; | ||
| 177 | resets = <&softreset STIH407_HDTVOUT_SOFTRESET>; | ||
| 178 | #address-cells = <1>; | 209 | #address-cells = <1>; |
| 179 | #size-cells = <1>; | 210 | #size-cells = <0>; |
| 180 | assigned-clocks = <&clk_s_d2_flexgen CLK_PIX_HDMI>, | 211 | |
| 181 | <&clk_s_d2_flexgen CLK_TMDS_HDMI>, | 212 | port@0 { |
| 182 | <&clk_s_d2_flexgen CLK_REF_HDMIPHY>, | 213 | reg = <0>; |
| 183 | <&clk_s_d0_flexgen CLK_PCM_0>, | 214 | tvout_in0: endpoint { |
| 184 | <&clk_s_d2_flexgen CLK_PIX_HDDAC>, | 215 | remote-endpoint = <&compo_main_out>; |
| 185 | <&clk_s_d2_flexgen CLK_HDDAC>; | 216 | }; |
| 186 | 217 | }; | |
| 187 | assigned-clock-parents = <&clk_s_d2_quadfs 0>, | 218 | |
| 188 | <&clk_tmdsout_hdmi>, | 219 | port@1 { |
| 189 | <&clk_s_d2_quadfs 0>, | 220 | reg = <1>; |
| 190 | <&clk_s_d0_quadfs 0>, | 221 | tvout_in1: endpoint { |
| 191 | <&clk_s_d2_quadfs 0>, | 222 | remote-endpoint = <&compo_aux_out>; |
| 192 | <&clk_s_d2_quadfs 0>; | 223 | }; |
| 224 | }; | ||
| 225 | |||
| 226 | port@2 { | ||
| 227 | reg = <2>; | ||
| 228 | tvout_out0: endpoint { | ||
| 229 | remote-endpoint = <&hdmi_in>; | ||
| 230 | }; | ||
| 231 | }; | ||
| 232 | |||
| 233 | port@3 { | ||
| 234 | reg = <3>; | ||
| 235 | tvout_out1: endpoint { | ||
| 236 | remote-endpoint = <&hda_in>; | ||
| 237 | }; | ||
| 238 | }; | ||
| 193 | }; | 239 | }; |
| 240 | }; | ||
| 194 | 241 | ||
| 195 | sti_hdmi: sti-hdmi@8d04000 { | 242 | sti_hdmi: hdmi@8d04000 { |
| 196 | compatible = "st,stih407-hdmi"; | 243 | compatible = "st,stih407-hdmi"; |
| 197 | reg = <0x8d04000 0x1000>; | 244 | reg = <0x8d04000 0x1000>; |
| 198 | reg-names = "hdmi-reg"; | 245 | reg-names = "hdmi-reg"; |
| 199 | #sound-dai-cells = <0>; | 246 | #sound-dai-cells = <0>; |
| 200 | interrupts = <GIC_SPI 106 IRQ_TYPE_LEVEL_HIGH>; | 247 | interrupts = <GIC_SPI 106 IRQ_TYPE_LEVEL_HIGH>; |
| 201 | interrupt-names = "irq"; | 248 | interrupt-names = "irq"; |
| 202 | clock-names = "pix", | 249 | clock-names = "pix", |
| 203 | "tmds", | 250 | "tmds", |
| 204 | "phy", | 251 | "phy", |
| 205 | "audio", | 252 | "audio", |
| 206 | "main_parent", | 253 | "main_parent", |
| 207 | "aux_parent"; | 254 | "aux_parent"; |
| 208 | 255 | ||
| 209 | clocks = <&clk_s_d2_flexgen CLK_PIX_HDMI>, | 256 | clocks = <&clk_s_d2_flexgen CLK_PIX_HDMI>, |
| 210 | <&clk_s_d2_flexgen CLK_TMDS_HDMI>, | 257 | <&clk_s_d2_flexgen CLK_TMDS_HDMI>, |
| 211 | <&clk_s_d2_flexgen CLK_REF_HDMIPHY>, | 258 | <&clk_s_d2_flexgen CLK_REF_HDMIPHY>, |
| 212 | <&clk_s_d0_flexgen CLK_PCM_0>, | 259 | <&clk_s_d0_flexgen CLK_PCM_0>, |
| 213 | <&clk_s_d2_quadfs 0>, | 260 | <&clk_s_d2_quadfs 0>, |
| 214 | <&clk_s_d2_quadfs 1>; | 261 | <&clk_s_d2_quadfs 1>; |
| 215 | 262 | ||
| 216 | hdmi,hpd-gpio = <&pio5 3 GPIO_ACTIVE_LOW>; | 263 | hdmi,hpd-gpio = <&pio5 3 GPIO_ACTIVE_LOW>; |
| 217 | reset-names = "hdmi"; | 264 | reset-names = "hdmi"; |
| 218 | resets = <&softreset STIH407_HDMI_TX_PHY_SOFTRESET>; | 265 | resets = <&softreset STIH407_HDMI_TX_PHY_SOFTRESET>; |
| 219 | ddc = <&hdmiddc>; | 266 | ddc = <&hdmiddc>; |
| 267 | |||
| 268 | port { | ||
| 269 | hdmi_in: endpoint { | ||
| 270 | remote-endpoint = <&tvout_out0>; | ||
| 271 | }; | ||
| 220 | }; | 272 | }; |
| 273 | }; | ||
| 221 | 274 | ||
| 222 | sti-hda@8d02000 { | 275 | analog@8d02000 { |
| 223 | compatible = "st,stih407-hda"; | 276 | compatible = "st,stih407-hda"; |
| 224 | status = "disabled"; | 277 | status = "disabled"; |
| 225 | reg = <0x8d02000 0x400>, <0x92b0120 0x4>; | 278 | reg = <0x8d02000 0x400>, <0x92b0120 0x4>; |
| 226 | reg-names = "hda-reg", "video-dacs-ctrl"; | 279 | reg-names = "hda-reg", "video-dacs-ctrl"; |
| 227 | clock-names = "pix", | 280 | clock-names = "pix", |
| 228 | "hddac", | 281 | "hddac", |
| 229 | "main_parent", | 282 | "main_parent", |
| 230 | "aux_parent"; | 283 | "aux_parent"; |
| 231 | clocks = <&clk_s_d2_flexgen CLK_PIX_HDDAC>, | 284 | clocks = <&clk_s_d2_flexgen CLK_PIX_HDDAC>, |
| 232 | <&clk_s_d2_flexgen CLK_HDDAC>, | 285 | <&clk_s_d2_flexgen CLK_HDDAC>, |
| 233 | <&clk_s_d2_quadfs 0>, | 286 | <&clk_s_d2_quadfs 0>, |
| 234 | <&clk_s_d2_quadfs 1>; | 287 | <&clk_s_d2_quadfs 1>; |
| 288 | |||
| 289 | port { | ||
| 290 | hda_in: endpoint { | ||
| 291 | remote-endpoint = <&tvout_out1>; | ||
| 292 | }; | ||
| 235 | }; | 293 | }; |
| 294 | }; | ||
| 236 | 295 | ||
| 237 | sti-hqvdp@9c00000 { | 296 | hqvdp: plane@9c00000 { |
| 238 | compatible = "st,stih407-hqvdp"; | 297 | compatible = "st,stih407-hqvdp"; |
| 239 | reg = <0x9C00000 0x100000>; | 298 | reg = <0x9C00000 0x100000>; |
| 240 | clock-names = "hqvdp", "pix_main"; | 299 | clock-names = "hqvdp", "pix_main"; |
| 241 | clocks = <&clk_s_c0_flexgen CLK_MAIN_DISP>, | 300 | clocks = <&clk_s_c0_flexgen CLK_MAIN_DISP>, |
| 242 | <&clk_s_d2_flexgen CLK_PIX_MAIN_DISP>; | 301 | <&clk_s_d2_flexgen CLK_PIX_MAIN_DISP>; |
| 243 | reset-names = "hqvdp"; | 302 | reset-names = "hqvdp"; |
| 244 | resets = <&softreset STIH407_HDQVDP_SOFTRESET>; | 303 | resets = <&softreset STIH407_HDQVDP_SOFTRESET>; |
| 245 | st,vtg = <&vtg_main>; | 304 | st,vtg = <&vtg_main>; |
| 246 | }; | ||
| 247 | }; | 305 | }; |
| 248 | 306 | ||
| 249 | bdisp0:bdisp@9f10000 { | 307 | bdisp0:bdisp@9f10000 { |
| @@ -250,6 +250,28 @@ · 249 unmodified lines | |||
| 250 | clock-output-names = "dummy"; | 250 | clock-output-names = "dummy"; |
| 251 | }; | 251 | }; |
| 252 | 252 | ||
| 253 | gpu_opp_table: opp-table { | ||
| 254 | compatible = "operating-points-v2"; | ||
| 255 | |||
| 256 | opp-500000000 { | ||
| 257 | opp-hz = /bits/ 64 <500000000>; | ||
| 258 | opp-hz-real = /bits/ 64 <500000000>; | ||
| 259 | opp-microvolt = <920000>; | ||
| 260 | }; | ||
| 261 | |||
| 262 | opp-800000000 { | ||
| 263 | opp-hz = /bits/ 64 <800000000>; | ||
| 264 | opp-hz-real = /bits/ 64 <800000000>; | ||
| 265 | opp-microvolt = <920000>; | ||
| 266 | }; | ||
| 267 | |||
| 268 | opp-1000000000 { | ||
| 269 | opp-hz = /bits/ 64 <1000000000>; | ||
| 270 | opp-hz-real = /bits/ 64 <1000000000>; | ||
| 271 | opp-microvolt = <920000>; | ||
| 272 | }; | ||
| 273 | }; | ||
| 274 | |||
| 253 | clk_ext1: clock-ext1 { | 275 | clk_ext1: clock-ext1 { |
| 254 | compatible = "fixed-clock"; | 276 | compatible = "fixed-clock"; |
| 255 | #clock-cells = <0>; | 277 | #clock-cells = <0>; |
| @@ -2139,6 +2161,21 @@ · 1883 unmodified lines | |||
| 2139 | }; | 2161 | }; |
| 2140 | }; | 2162 | }; |
| 2141 | 2163 | ||
| 2164 | gpu: gpu@4d900000 { | ||
| 2165 | compatible = "nxp,imx95-mali", "arm,mali-valhall-csf"; | ||
| 2166 | reg = <0 0x4d900000 0 0x480000>; | ||
| 2167 | clocks = <&scmi_clk IMX95_CLK_GPU>, <&scmi_clk IMX95_CLK_GPUAPB>; | ||
| 2168 | clock-names = "core", "coregroup"; | ||
| 2169 | interrupts = <GIC_SPI 289 IRQ_TYPE_LEVEL_HIGH>, | ||
| 2170 | <GIC_SPI 290 IRQ_TYPE_LEVEL_HIGH>, | ||
| 2171 | <GIC_SPI 288 IRQ_TYPE_LEVEL_HIGH>; | ||
| 2172 | interrupt-names = "job", "mmu", "gpu"; | ||
| 2173 | operating-points-v2 = <&gpu_opp_table>; | ||
| 2174 | power-domains = <&scmi_devpd IMX95_PD_GPU>; | ||
| 2175 | #cooling-cells = <2>; | ||
| 2176 | dynamic-power-coefficient = <1013>; | ||
| 2177 | }; | ||
| 2178 | |||
| 2142 | ddr-pmu@4e090dc0 { | 2179 | ddr-pmu@4e090dc0 { |
| 2143 | compatible = "fsl,imx95-ddr-pmu", "fsl,imx93-ddr-pmu"; | 2180 | compatible = "fsl,imx95-ddr-pmu", "fsl,imx93-ddr-pmu"; |
| 2144 | reg = <0x0 0x4e090dc0 0x0 0x200>; | 2181 | reg = <0x0 0x4e090dc0 0x0 0x200>; |
| @@ -25,6 +25,7 @@ menuconfig DRM_ACCEL · 24 unmodified lines | |||
| 25 | and debugfs). | 25 | and debugfs). |
| 26 | 26 | ||
| 27 | source "drivers/accel/amdxdna/Kconfig" | 27 | source "drivers/accel/amdxdna/Kconfig" |
| 28 | source "drivers/accel/ethosu/Kconfig" | ||
| 28 | source "drivers/accel/habanalabs/Kconfig" | 29 | source "drivers/accel/habanalabs/Kconfig" |
| 29 | source "drivers/accel/ivpu/Kconfig" | 30 | source "drivers/accel/ivpu/Kconfig" |
| 30 | source "drivers/accel/qaic/Kconfig" | 31 | source "drivers/accel/qaic/Kconfig" |
| @@ -1,6 +1,7 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only | 1 | # SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | ||
| 3 | obj-$(CONFIG_DRM_ACCEL_AMDXDNA) += amdxdna/ | 3 | obj-$(CONFIG_DRM_ACCEL_AMDXDNA) += amdxdna/ |
| 4 | obj-$(CONFIG_DRM_ACCEL_ARM_ETHOSU) += ethosu/ | ||
| 4 | obj-$(CONFIG_DRM_ACCEL_HABANALABS) += habanalabs/ | 5 | obj-$(CONFIG_DRM_ACCEL_HABANALABS) += habanalabs/ |
| 5 | obj-$(CONFIG_DRM_ACCEL_IVPU) += ivpu/ | 6 | obj-$(CONFIG_DRM_ACCEL_IVPU) += ivpu/ |
| 6 | obj-$(CONFIG_DRM_ACCEL_QAIC) += qaic/ | 7 | obj-$(CONFIG_DRM_ACCEL_QAIC) += qaic/ |
| @@ -14,6 +14,7 @@ amdxdna-y := \ · 13 unmodified lines | |||
| 14 | amdxdna_mailbox.o \ | 14 | amdxdna_mailbox.o \ |
| 15 | amdxdna_mailbox_helper.o \ | 15 | amdxdna_mailbox_helper.o \ |
| 16 | amdxdna_pci_drv.o \ | 16 | amdxdna_pci_drv.o \ |
| 17 | amdxdna_pm.o \ | ||
| 17 | amdxdna_sysfs.o \ | 18 | amdxdna_sysfs.o \ |
| 18 | amdxdna_ubuf.o \ | 19 | amdxdna_ubuf.o \ |
| 19 | npu1_regs.o \ | 20 | npu1_regs.o \ |
| @@ -1,2 +1 @@ | |||
| 1 | - Add debugfs support | 1 | - Add debugfs support |
| 2 | - Add debug BO support | ||
| @@ -21,6 +21,7 @@ · 20 unmodified lines | |||
| 21 | #include "amdxdna_gem.h" | 21 | #include "amdxdna_gem.h" |
| 22 | #include "amdxdna_mailbox.h" | 22 | #include "amdxdna_mailbox.h" |
| 23 | #include "amdxdna_pci_drv.h" | 23 | #include "amdxdna_pci_drv.h" |
| 24 | #include "amdxdna_pm.h" | ||
| 24 | 25 | ||
| 25 | static bool force_cmdlist; | 26 | static bool force_cmdlist; |
| 26 | module_param(force_cmdlist, bool, 0600); | 27 | module_param(force_cmdlist, bool, 0600); |
| @@ -88,7 +89,7 @@ static int aie2_hwctx_restart(struct amdxdna_dev *xdna, struct amdxdna_hwctx *hw · 61 unmodified lines | |||
| 88 | goto out; | 89 | goto out; |
| 89 | } | 90 | } |
| 90 | 91 | ||
| 91 | ret = aie2_config_cu(hwctx); | 92 | ret = aie2_config_cu(hwctx, NULL); |
| 92 | if (ret) { | 93 | if (ret) { |
| 93 | XDNA_ERR(xdna, "Config cu failed, ret %d", ret); | 94 | XDNA_ERR(xdna, "Config cu failed, ret %d", ret); |
| 94 | goto out; | 95 | goto out; |
| @@ -167,14 +168,11 @@ static int aie2_hwctx_resume_cb(struct amdxdna_hwctx *hwctx, void *arg) · 72 unmodified lines | |||
| 167 | 168 | ||
| 168 | int aie2_hwctx_resume(struct amdxdna_client *client) | 169 | int aie2_hwctx_resume(struct amdxdna_client *client) |
| 169 | { | 170 | { |
| 170 | struct amdxdna_dev *xdna = client->xdna; | ||
| 171 | |||
| 172 | /* | 171 | /* |
| 173 | * The resume path cannot guarantee that mailbox channel can be | 172 | * The resume path cannot guarantee that mailbox channel can be |
| 174 | * regenerated. If this happen, when submit message to this | 173 | * regenerated. If this happen, when submit message to this |
| 175 | * mailbox channel, error will return. | 174 | * mailbox channel, error will return. |
| 176 | */ | 175 | */ |
| 177 | drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); | ||
| 178 | return amdxdna_hwctx_walk(client, NULL, aie2_hwctx_resume_cb); | 176 | return amdxdna_hwctx_walk(client, NULL, aie2_hwctx_resume_cb); |
| 179 | } | 177 | } |
| 180 | 178 | ||
| @@ -184,12 +182,13 @@ aie2_sched_notify(struct amdxdna_sched_job *job) · 3 unmodified lines | |||
| 184 | struct dma_fence *fence = job->fence; | 182 | struct dma_fence *fence = job->fence; |
| 185 | 183 | ||
| 186 | trace_xdna_job(&job->base, job->hwctx->name, "signaled fence", job->seq); | 184 | trace_xdna_job(&job->base, job->hwctx->name, "signaled fence", job->seq); |
| 185 | |||
| 186 | amdxdna_pm_suspend_put(job->hwctx->client->xdna); | ||
| 187 | job->hwctx->priv->completed++; | 187 | job->hwctx->priv->completed++; |
| 188 | dma_fence_signal(fence); | 188 | dma_fence_signal(fence); |
| 189 | 189 | ||
| 190 | up(&job->hwctx->priv->job_sem); | 190 | up(&job->hwctx->priv->job_sem); |
| 191 | job->job_done = true; | 191 | job->job_done = true; |
| 192 | dma_fence_put(fence); | ||
| 193 | mmput_async(job->mm); | 192 | mmput_async(job->mm); |
| 194 | aie2_job_put(job); | 193 | aie2_job_put(job); |
| 195 | } | 194 | } |
| @@ -204,10 +203,13 @@ aie2_sched_resp_handler(void *handle, void __iomem *data, size_t size) · 8 unmodified lines | |||
| 204 | 203 | ||
| 205 | cmd_abo = job->cmd_bo; | 204 | cmd_abo = job->cmd_bo; |
| 206 | 205 | ||
| 207 | if (unlikely(!data)) | 206 | if (unlikely(job->job_timeout)) { |
| 207 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_TIMEOUT); | ||
| 208 | ret = -EINVAL; | ||
| 208 | goto out; | 209 | goto out; |
| 210 | } | ||
| 209 | 211 | ||
| 210 | if (unlikely(size != sizeof(u32))) { | 212 | if (unlikely(!data) || unlikely(size != sizeof(u32))) { |
| 211 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT); | 213 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT); |
| 212 | ret = -EINVAL; | 214 | ret = -EINVAL; |
| 213 | goto out; | 215 | goto out; |
| @@ -226,11 +228,10 @@ out: · 12 unmodified lines | |||
| 226 | } | 228 | } |
| 227 | 229 | ||
| 228 | static int | 230 | static int |
| 229 | aie2_sched_nocmd_resp_handler(void *handle, void __iomem *data, size_t size) | 231 | aie2_sched_drvcmd_resp_handler(void *handle, void __iomem *data, size_t size) |
| 230 | { | 232 | { |
| 231 | struct amdxdna_sched_job *job = handle; | 233 | struct amdxdna_sched_job *job = handle; |
| 232 | int ret = 0; | 234 | int ret = 0; |
| 233 | u32 status; | ||
| 234 | 235 | ||
| 235 | if (unlikely(!data)) | 236 | if (unlikely(!data)) |
| 236 | goto out; | 237 | goto out; |
| @@ -240,8 +241,7 @@ aie2_sched_nocmd_resp_handler(void *handle, void __iomem *data, size_t size) · 3 unmodified lines | |||
| 240 | goto out; | 241 | goto out; |
| 241 | } | 242 | } |
| 242 | 243 | ||
| 243 | status = readl(data); | 244 | job->drv_cmd->result = readl(data); |
| 244 | XDNA_DBG(job->hwctx->client->xdna, "Resp status 0x%x", status); | ||
| 245 | 245 | ||
| 246 | out: | 246 | out: |
| 247 | aie2_sched_notify(job); | 247 | aie2_sched_notify(job); |
| @@ -260,6 +260,13 @@ aie2_sched_cmdlist_resp_handler(void *handle, void __iomem *data, size_t size) · 12 unmodified lines | |||
| 260 | int ret = 0; | 260 | int ret = 0; |
| 261 | 261 | ||
| 262 | cmd_abo = job->cmd_bo; | 262 | cmd_abo = job->cmd_bo; |
| 263 | |||
| 264 | if (unlikely(job->job_timeout)) { | ||
| 265 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_TIMEOUT); | ||
| 266 | ret = -EINVAL; | ||
| 267 | goto out; | ||
| 268 | } | ||
| 269 | |||
| 263 | if (unlikely(!data) || unlikely(size != sizeof(u32) * 3)) { | 270 | if (unlikely(!data) || unlikely(size != sizeof(u32) * 3)) { |
| 264 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT); | 271 | amdxdna_cmd_set_state(cmd_abo, ERT_CMD_STATE_ABORT); |
| 265 | ret = -EINVAL; | 272 | ret = -EINVAL; |
| @@ -314,8 +321,18 @@ aie2_sched_job_run(struct drm_sched_job *sched_job) · 48 unmodified lines | |||
| 314 | kref_get(&job->refcnt); | 321 | kref_get(&job->refcnt); |
| 315 | fence = dma_fence_get(job->fence); | 322 | fence = dma_fence_get(job->fence); |
| 316 | 323 | ||
| 317 | if (unlikely(!cmd_abo)) { | 324 | if (job->drv_cmd) { |
| 318 | ret = aie2_sync_bo(hwctx, job, aie2_sched_nocmd_resp_handler); | 325 | switch (job->drv_cmd->opcode) { |
| 326 | case SYNC_DEBUG_BO: | ||
| 327 | ret = aie2_sync_bo(hwctx, job, aie2_sched_drvcmd_resp_handler); | ||
| 328 | break; | ||
| 329 | case ATTACH_DEBUG_BO: | ||
| 330 | ret = aie2_config_debug_bo(hwctx, job, aie2_sched_drvcmd_resp_handler); | ||
| 331 | break; | ||
| 332 | default: | ||
| 333 | ret = -EINVAL; | ||
| 334 | break; | ||
| 335 | } | ||
| 319 | goto out; | 336 | goto out; |
| 320 | } | 337 | } |
| 321 | 338 | ||
| @@ -362,6 +379,7 @@ aie2_sched_job_timedout(struct drm_sched_job *sched_job) · 40 unmodified lines | |||
| 362 | 379 | ||
| 363 | xdna = hwctx->client->xdna; | 380 | xdna = hwctx->client->xdna; |
| 364 | trace_xdna_job(sched_job, hwctx->name, "job timedout", job->seq); | 381 | trace_xdna_job(sched_job, hwctx->name, "job timedout", job->seq); |
| 382 | job->job_timeout = true; | ||
| 365 | mutex_lock(&xdna->dev_lock); | 383 | mutex_lock(&xdna->dev_lock); |
| 366 | aie2_hwctx_stop(xdna, hwctx, sched_job); | 384 | aie2_hwctx_stop(xdna, hwctx, sched_job); |
| 367 | 385 | ||
| @@ -531,13 +549,12 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 163 unmodified lines | |||
| 531 | .num_rqs = DRM_SCHED_PRIORITY_COUNT, | 549 | .num_rqs = DRM_SCHED_PRIORITY_COUNT, |
| 532 | .credit_limit = HWCTX_MAX_CMDS, | 550 | .credit_limit = HWCTX_MAX_CMDS, |
| 533 | .timeout = msecs_to_jiffies(HWCTX_MAX_TIMEOUT), | 551 | .timeout = msecs_to_jiffies(HWCTX_MAX_TIMEOUT), |
| 534 | .name = hwctx->name, | 552 | .name = "amdxdna_js", |
| 535 | .dev = xdna->ddev.dev, | 553 | .dev = xdna->ddev.dev, |
| 536 | }; | 554 | }; |
| 537 | struct drm_gpu_scheduler *sched; | 555 | struct drm_gpu_scheduler *sched; |
| 538 | struct amdxdna_hwctx_priv *priv; | 556 | struct amdxdna_hwctx_priv *priv; |
| 539 | struct amdxdna_gem_obj *heap; | 557 | struct amdxdna_gem_obj *heap; |
| 540 | struct amdxdna_dev_hdl *ndev; | ||
| 541 | int i, ret; | 558 | int i, ret; |
| 542 | 559 | ||
| 543 | priv = kzalloc(sizeof(*hwctx->priv), GFP_KERNEL); | 560 | priv = kzalloc(sizeof(*hwctx->priv), GFP_KERNEL); |
| @@ -610,10 +627,14 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 66 unmodified lines | |||
| 610 | goto free_entity; | 627 | goto free_entity; |
| 611 | } | 628 | } |
| 612 | 629 | ||
| 630 | ret = amdxdna_pm_resume_get(xdna); | ||
| 631 | if (ret) | ||
| 632 | goto free_col_list; | ||
| 633 | |||
| 613 | ret = aie2_alloc_resource(hwctx); | 634 | ret = aie2_alloc_resource(hwctx); |
| 614 | if (ret) { | 635 | if (ret) { |
| 615 | XDNA_ERR(xdna, "Alloc hw resource failed, ret %d", ret); | 636 | XDNA_ERR(xdna, "Alloc hw resource failed, ret %d", ret); |
| 616 | goto free_col_list; | 637 | goto suspend_put; |
| 617 | } | 638 | } |
| 618 | 639 | ||
| 619 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, | 640 | ret = aie2_map_host_buf(xdna->dev_handle, hwctx->fw_ctx_id, |
| @@ -628,10 +649,9 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 8 unmodified lines | |||
| 628 | XDNA_ERR(xdna, "Create syncobj failed, ret %d", ret); | 649 | XDNA_ERR(xdna, "Create syncobj failed, ret %d", ret); |
| 629 | goto release_resource; | 650 | goto release_resource; |
| 630 | } | 651 | } |
| 652 | amdxdna_pm_suspend_put(xdna); | ||
| 631 | 653 | ||
| 632 | hwctx->status = HWCTX_STAT_INIT; | 654 | hwctx->status = HWCTX_STAT_INIT; |
| 633 | ndev = xdna->dev_handle; | ||
| 634 | ndev->hwctx_num++; | ||
| 635 | init_waitqueue_head(&priv->job_free_wq); | 655 | init_waitqueue_head(&priv->job_free_wq); |
| 636 | 656 | ||
| 637 | XDNA_DBG(xdna, "hwctx %s init completed", hwctx->name); | 657 | XDNA_DBG(xdna, "hwctx %s init completed", hwctx->name); |
| @@ -640,6 +660,8 @@ int aie2_hwctx_init(struct amdxdna_hwctx *hwctx) · 2 unmodified lines | |||
| 640 | 660 | ||
| 641 | release_resource: | 661 | release_resource: |
| 642 | aie2_release_resource(hwctx); | 662 | aie2_release_resource(hwctx); |
| 663 | suspend_put: | ||
| 664 | amdxdna_pm_suspend_put(xdna); | ||
| 643 | free_col_list: | 665 | free_col_list: |
| 644 | kfree(hwctx->col_list); | 666 | kfree(hwctx->col_list); |
| 645 | free_entity: | 667 | free_entity: |
| @@ -662,26 +684,25 @@ free_priv: · 16 unmodified lines | |||
| 662 | 684 | ||
| 663 | void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx) | 685 | void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx) |
| 664 | { | 686 | { |
| 665 | struct amdxdna_dev_hdl *ndev; | ||
| 666 | struct amdxdna_dev *xdna; | 687 | struct amdxdna_dev *xdna; |
| 667 | int idx; | 688 | int idx; |
| 668 | 689 | ||
| 669 | xdna = hwctx->client->xdna; | 690 | xdna = hwctx->client->xdna; |
| 670 | ndev = xdna->dev_handle; | ||
| 671 | ndev->hwctx_num--; | ||
| 672 | 691 | ||
| 673 | XDNA_DBG(xdna, "%s sequence number %lld", hwctx->name, hwctx->priv->seq); | 692 | XDNA_DBG(xdna, "%s sequence number %lld", hwctx->name, hwctx->priv->seq); |
| 674 | drm_sched_entity_destroy(&hwctx->priv->entity); | ||
| 675 | |||
| 676 | aie2_hwctx_wait_for_idle(hwctx); | 693 | aie2_hwctx_wait_for_idle(hwctx); |
| 677 | 694 | ||
| 678 | /* Request fw to destroy hwctx and cancel the rest pending requests */ | 695 | /* Request fw to destroy hwctx and cancel the rest pending requests */ |
| 679 | aie2_release_resource(hwctx); | 696 | aie2_release_resource(hwctx); |
| 680 | 697 | ||
| 698 | mutex_unlock(&xdna->dev_lock); | ||
| 699 | drm_sched_entity_destroy(&hwctx->priv->entity); | ||
| 700 | |||
| 681 | /* Wait for all submitted jobs to be completed or canceled */ | 701 | /* Wait for all submitted jobs to be completed or canceled */ |
| 682 | wait_event(hwctx->priv->job_free_wq, | 702 | wait_event(hwctx->priv->job_free_wq, |
| 683 | atomic64_read(&hwctx->job_submit_cnt) == | 703 | atomic64_read(&hwctx->job_submit_cnt) == |
| 684 | atomic64_read(&hwctx->job_free_cnt)); | 704 | atomic64_read(&hwctx->job_free_cnt)); |
| 705 | mutex_lock(&xdna->dev_lock); | ||
| 685 | 706 | ||
| 686 | drm_sched_fini(&hwctx->priv->sched); | 707 | drm_sched_fini(&hwctx->priv->sched); |
| 687 | aie2_ctx_syncobj_destroy(hwctx); | 708 | aie2_ctx_syncobj_destroy(hwctx); |
| @@ -697,6 +718,14 @@ void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx) · 9 unmodified lines | |||
| 697 | kfree(hwctx->cus); | 718 | kfree(hwctx->cus); |
| 698 | } | 719 | } |
| 699 | 720 | ||
| 721 | static int aie2_config_cu_resp_handler(void *handle, void __iomem *data, size_t size) | ||
| 722 | { | ||
| 723 | struct amdxdna_hwctx *hwctx = handle; | ||
| 724 | |||
| 725 | amdxdna_pm_suspend_put(hwctx->client->xdna); | ||
| 726 | return 0; | ||
| 727 | } | ||
| 728 | |||
| 700 | static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size) | 729 | static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size) |
| 701 | { | 730 | { |
| 702 | struct amdxdna_hwctx_param_config_cu *config = buf; | 731 | struct amdxdna_hwctx_param_config_cu *config = buf; |
| @@ -728,10 +757,14 @@ static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size · 25 unmodified lines | |||
| 728 | if (!hwctx->cus) | 757 | if (!hwctx->cus) |
| 729 | return -ENOMEM; | 758 | return -ENOMEM; |
| 730 | 759 | ||
| 731 | ret = aie2_config_cu(hwctx); | 760 | ret = amdxdna_pm_resume_get(xdna); |
| 761 | if (ret) | ||
| 762 | goto free_cus; | ||
| 763 | |||
| 764 | ret = aie2_config_cu(hwctx, aie2_config_cu_resp_handler); | ||
| 732 | if (ret) { | 765 | if (ret) { |
| 733 | XDNA_ERR(xdna, "Config CU to firmware failed, ret %d", ret); | 766 | XDNA_ERR(xdna, "Config CU to firmware failed, ret %d", ret); |
| 734 | goto free_cus; | 767 | goto pm_suspend_put; |
| 735 | } | 768 | } |
| 736 | 769 | ||
| 737 | wmb(); /* To avoid locking in command submit when check status */ | 770 | wmb(); /* To avoid locking in command submit when check status */ |
| @@ -739,12 +772,82 @@ static int aie2_hwctx_cu_config(struct amdxdna_hwctx *hwctx, void *buf, u32 size · 1 unmodified line | |||
| 739 | 772 | ||
| 740 | return 0; | 773 | return 0; |
| 741 | 774 | ||
| 775 | pm_suspend_put: | ||
| 776 | amdxdna_pm_suspend_put(xdna); | ||
| 742 | free_cus: | 777 | free_cus: |
| 743 | kfree(hwctx->cus); | 778 | kfree(hwctx->cus); |
| 744 | hwctx->cus = NULL; | 779 | hwctx->cus = NULL; |
| 745 | return ret; | 780 | return ret; |
| 746 | } | 781 | } |
| 747 | 782 | ||
| 783 | static void aie2_cmd_wait(struct amdxdna_hwctx *hwctx, u64 seq) | ||
| 784 | { | ||
| 785 | struct dma_fence *out_fence = aie2_cmd_get_out_fence(hwctx, seq); | ||
| 786 | |||
| 787 | if (!out_fence) { | ||
| 788 | XDNA_ERR(hwctx->client->xdna, "Failed to get fence"); | ||
| 789 | return; | ||
| 790 | } | ||
| 791 | |||
| 792 | dma_fence_wait_timeout(out_fence, false, MAX_SCHEDULE_TIMEOUT); | ||
| 793 | dma_fence_put(out_fence); | ||
| 794 | } | ||
| 795 | |||
| 796 | static int aie2_hwctx_cfg_debug_bo(struct amdxdna_hwctx *hwctx, u32 bo_hdl, | ||
| 797 | bool attach) | ||
| 798 | { | ||
| 799 | struct amdxdna_client *client = hwctx->client; | ||
| 800 | struct amdxdna_dev *xdna = client->xdna; | ||
| 801 | struct amdxdna_drv_cmd cmd = { 0 }; | ||
| 802 | struct amdxdna_gem_obj *abo; | ||
| 803 | u64 seq; | ||
| 804 | int ret; | ||
| 805 | |||
| 806 | abo = amdxdna_gem_get_obj(client, bo_hdl, AMDXDNA_BO_DEV); | ||
| 807 | if (!abo) { | ||
| 808 | XDNA_ERR(xdna, "Get bo %d failed", bo_hdl); | ||
| 809 | return -EINVAL; | ||
| 810 | } | ||
| 811 | |||
| 812 | if (attach) { | ||
| 813 | if (abo->assigned_hwctx != AMDXDNA_INVALID_CTX_HANDLE) { | ||
| 814 | ret = -EBUSY; | ||
| 815 | goto put_obj; | ||
| 816 | } | ||
| 817 | cmd.opcode = ATTACH_DEBUG_BO; | ||
| 818 | } else { | ||
| 819 | if (abo->assigned_hwctx != hwctx->id) { | ||
| 820 | ret = -EINVAL; | ||
| 821 | goto put_obj; | ||
| 822 | } | ||
| 823 | cmd.opcode = DETACH_DEBUG_BO; | ||
| 824 | } | ||
| 825 | |||
| 826 | ret = amdxdna_cmd_submit(client, &cmd, AMDXDNA_INVALID_BO_HANDLE, | ||
| 827 | &bo_hdl, 1, hwctx->id, &seq); | ||
| 828 | if (ret) { | ||
| 829 | XDNA_ERR(xdna, "Submit command failed"); | ||
| 830 | goto put_obj; | ||
| 831 | } | ||
| 832 | |||
| 833 | aie2_cmd_wait(hwctx, seq); | ||
| 834 | if (cmd.result) { | ||
| 835 | XDNA_ERR(xdna, "Response failure 0x%x", cmd.result); | ||
| 836 | goto put_obj; | ||
| 837 | } | ||
| 838 | |||
| 839 | if (attach) | ||
| 840 | abo->assigned_hwctx = hwctx->id; | ||
| 841 | else | ||
| 842 | abo->assigned_hwctx = AMDXDNA_INVALID_CTX_HANDLE; | ||
| 843 | |||
| 844 | XDNA_DBG(xdna, "Config debug BO %d to %s", bo_hdl, hwctx->name); | ||
| 845 | |||
| 846 | put_obj: | ||
| 847 | amdxdna_gem_put_obj(abo); | ||
| 848 | return ret; | ||
| 849 | } | ||
| 850 | |||
| 748 | int aie2_hwctx_config(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size) | 851 | int aie2_hwctx_config(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size) |
| 749 | { | 852 | { |
| 750 | struct amdxdna_dev *xdna = hwctx->client->xdna; | 853 | struct amdxdna_dev *xdna = hwctx->client->xdna; |
| @@ -754,14 +857,40 @@ int aie2_hwctx_config(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *bu · 3 unmodified lines | |||
| 754 | case DRM_AMDXDNA_HWCTX_CONFIG_CU: | 857 | case DRM_AMDXDNA_HWCTX_CONFIG_CU: |
| 755 | return aie2_hwctx_cu_config(hwctx, buf, size); | 858 | return aie2_hwctx_cu_config(hwctx, buf, size); |
| 756 | case DRM_AMDXDNA_HWCTX_ASSIGN_DBG_BUF: | 859 | case DRM_AMDXDNA_HWCTX_ASSIGN_DBG_BUF: |
| 860 | return aie2_hwctx_cfg_debug_bo(hwctx, (u32)value, true); | ||
| 757 | case DRM_AMDXDNA_HWCTX_REMOVE_DBG_BUF: | 861 | case DRM_AMDXDNA_HWCTX_REMOVE_DBG_BUF: |
| 758 | return -EOPNOTSUPP; | 862 | return aie2_hwctx_cfg_debug_bo(hwctx, (u32)value, false); |
| 759 | default: | 863 | default: |
| 760 | XDNA_DBG(xdna, "Not supported type %d", type); | 864 | XDNA_DBG(xdna, "Not supported type %d", type); |
| 761 | return -EOPNOTSUPP; | 865 | return -EOPNOTSUPP; |
| 762 | } | 866 | } |
| 763 | } | 867 | } |
| 764 | 868 | ||
| 869 | int aie2_hwctx_sync_debug_bo(struct amdxdna_hwctx *hwctx, u32 debug_bo_hdl) | ||
| 870 | { | ||
| 871 | struct amdxdna_client *client = hwctx->client; | ||
| 872 | struct amdxdna_dev *xdna = client->xdna; | ||
| 873 | struct amdxdna_drv_cmd cmd = { 0 }; | ||
| 874 | u64 seq; | ||
| 875 | int ret; | ||
| 876 | |||
| 877 | cmd.opcode = SYNC_DEBUG_BO; | ||
| 878 | ret = amdxdna_cmd_submit(client, &cmd, AMDXDNA_INVALID_BO_HANDLE, | ||
| 879 | &debug_bo_hdl, 1, hwctx->id, &seq); | ||
| 880 | if (ret) { | ||
| 881 | XDNA_ERR(xdna, "Submit command failed"); | ||
| 882 | return ret; | ||
| 883 | } | ||
| 884 | |||
| 885 | aie2_cmd_wait(hwctx, seq); | ||
| 886 | if (cmd.result) { | ||
| 887 | XDNA_ERR(xdna, "Response failure 0x%x", cmd.result); | ||
| 888 | return -EINVAL; | ||
| 889 | } | ||
| 890 | |||
| 891 | return 0; | ||
| 892 | } | ||
| 893 | |||
| 765 | static int aie2_populate_range(struct amdxdna_gem_obj *abo) | 894 | static int aie2_populate_range(struct amdxdna_gem_obj *abo) |
| 766 | { | 895 | { |
| 767 | struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev); | 896 | struct amdxdna_dev *xdna = to_xdna_dev(to_gobj(abo)->dev); |
| @@ -862,11 +991,15 @@ int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, · 94 unmodified lines | |||
| 862 | goto free_chain; | 991 | goto free_chain; |
| 863 | } | 992 | } |
| 864 | 993 | ||
| 994 | ret = amdxdna_pm_resume_get(xdna); | ||
| 995 | if (ret) | ||
| 996 | goto cleanup_job; | ||
| 997 | |||
| 865 | retry: | 998 | retry: |
| 866 | ret = drm_gem_lock_reservations(job->bos, job->bo_cnt, &acquire_ctx); | 999 | ret = drm_gem_lock_reservations(job->bos, job->bo_cnt, &acquire_ctx); |
| 867 | if (ret) { | 1000 | if (ret) { |
| 868 | XDNA_WARN(xdna, "Failed to lock BOs, ret %d", ret); | 1001 | XDNA_WARN(xdna, "Failed to lock BOs, ret %d", ret); |
| 869 | goto cleanup_job; | 1002 | goto suspend_put; |
| 870 | } | 1003 | } |
| 871 | 1004 | ||
| 872 | for (i = 0; i < job->bo_cnt; i++) { | 1005 | for (i = 0; i < job->bo_cnt; i++) { |
| @@ -874,7 +1007,7 @@ retry: · 1 unmodified line | |||
| 874 | if (ret) { | 1007 | if (ret) { |
| 875 | XDNA_WARN(xdna, "Failed to reserve fences %d", ret); | 1008 | XDNA_WARN(xdna, "Failed to reserve fences %d", ret); |
| 876 | drm_gem_unlock_reservations(job->bos, job->bo_cnt, &acquire_ctx); | 1009 | drm_gem_unlock_reservations(job->bos, job->bo_cnt, &acquire_ctx); |
| 877 | goto cleanup_job; | 1010 | goto suspend_put; |
| 878 | } | 1011 | } |
| 879 | } | 1012 | } |
| 880 | 1013 | ||
| @@ -889,12 +1022,12 @@ retry: · 8 unmodified lines | |||
| 889 | msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); | 1022 | msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); |
| 890 | } else if (time_after(jiffies, timeout)) { | 1023 | } else if (time_after(jiffies, timeout)) { |
| 891 | ret = -ETIME; | 1024 | ret = -ETIME; |
| 892 | goto cleanup_job; | 1025 | goto suspend_put; |
| 893 | } | 1026 | } |
| 894 | 1027 | ||
| 895 | ret = aie2_populate_range(abo); | 1028 | ret = aie2_populate_range(abo); |
| 896 | if (ret) | 1029 | if (ret) |
| 897 | goto cleanup_job; | 1030 | goto suspend_put; |
| 898 | goto retry; | 1031 | goto retry; |
| 899 | } | 1032 | } |
| 900 | } | 1033 | } |
| @@ -920,6 +1053,8 @@ retry: · 19 unmodified lines | |||
| 920 | 1053 | ||
| 921 | return 0; | 1054 | return 0; |
| 922 | 1055 | ||
| 1056 | suspend_put: | ||
| 1057 | amdxdna_pm_suspend_put(xdna); | ||
| 923 | cleanup_job: | 1058 | cleanup_job: |
| 924 | drm_sched_job_cleanup(&job->base); | 1059 | drm_sched_job_cleanup(&job->base); |
| 925 | free_chain: | 1060 | free_chain: |
| @@ -13,6 +13,7 @@ · 12 unmodified lines | |||
| 13 | 13 | ||
| 14 | #include "aie2_msg_priv.h" | 14 | #include "aie2_msg_priv.h" |
| 15 | #include "aie2_pci.h" | 15 | #include "aie2_pci.h" |
| 16 | #include "amdxdna_error.h" | ||
| 16 | #include "amdxdna_mailbox.h" | 17 | #include "amdxdna_mailbox.h" |
| 17 | #include "amdxdna_pci_drv.h" | 18 | #include "amdxdna_pci_drv.h" |
| 18 | 19 | ||
| @@ -46,6 +47,7 @@ enum aie_module_type { · 27 unmodified lines | |||
| 46 | AIE_MEM_MOD = 0, | 47 | AIE_MEM_MOD = 0, |
| 47 | AIE_CORE_MOD, | 48 | AIE_CORE_MOD, |
| 48 | AIE_PL_MOD, | 49 | AIE_PL_MOD, |
| 50 | AIE_UNKNOWN_MOD, | ||
| 49 | }; | 51 | }; |
| 50 | 52 | ||
| 51 | enum aie_error_category { | 53 | enum aie_error_category { |
| @@ -143,6 +145,31 @@ static const struct aie_event_category aie_ml_shim_tile_event_cat[] = { · 91 unmodified lines | |||
| 143 | EVENT_CATEGORY(74U, AIE_ERROR_LOCK), | 145 | EVENT_CATEGORY(74U, AIE_ERROR_LOCK), |
| 144 | }; | 146 | }; |
| 145 | 147 | ||
| 148 | static const enum amdxdna_error_num aie_cat_err_num_map[] = { | ||
| 149 | [AIE_ERROR_SATURATION] = AMDXDNA_ERROR_NUM_AIE_SATURATION, | ||
| 150 | [AIE_ERROR_FP] = AMDXDNA_ERROR_NUM_AIE_FP, | ||
| 151 | [AIE_ERROR_STREAM] = AMDXDNA_ERROR_NUM_AIE_STREAM, | ||
| 152 | [AIE_ERROR_ACCESS] = AMDXDNA_ERROR_NUM_AIE_ACCESS, | ||
| 153 | [AIE_ERROR_BUS] = AMDXDNA_ERROR_NUM_AIE_BUS, | ||
| 154 | [AIE_ERROR_INSTRUCTION] = AMDXDNA_ERROR_NUM_AIE_INSTRUCTION, | ||
| 155 | [AIE_ERROR_ECC] = AMDXDNA_ERROR_NUM_AIE_ECC, | ||
| 156 | [AIE_ERROR_LOCK] = AMDXDNA_ERROR_NUM_AIE_LOCK, | ||
| 157 | [AIE_ERROR_DMA] = AMDXDNA_ERROR_NUM_AIE_DMA, | ||
| 158 | [AIE_ERROR_MEM_PARITY] = AMDXDNA_ERROR_NUM_AIE_MEM_PARITY, | ||
| 159 | [AIE_ERROR_UNKNOWN] = AMDXDNA_ERROR_NUM_UNKNOWN, | ||
| 160 | }; | ||
| 161 | |||
| 162 | static_assert(ARRAY_SIZE(aie_cat_err_num_map) == AIE_ERROR_UNKNOWN + 1); | ||
| 163 | |||
| 164 | static const enum amdxdna_error_module aie_err_mod_map[] = { | ||
| 165 | [AIE_MEM_MOD] = AMDXDNA_ERROR_MODULE_AIE_MEMORY, | ||
| 166 | [AIE_CORE_MOD] = AMDXDNA_ERROR_MODULE_AIE_CORE, | ||
| 167 | [AIE_PL_MOD] = AMDXDNA_ERROR_MODULE_AIE_PL, | ||
| 168 | [AIE_UNKNOWN_MOD] = AMDXDNA_ERROR_MODULE_UNKNOWN, | ||
| 169 | }; | ||
| 170 | |||
| 171 | static_assert(ARRAY_SIZE(aie_err_mod_map) == AIE_UNKNOWN_MOD + 1); | ||
| 172 | |||
| 146 | static enum aie_error_category | 173 | static enum aie_error_category |
| 147 | aie_get_error_category(u8 row, u8 event_id, enum aie_module_type mod_type) | 174 | aie_get_error_category(u8 row, u8 event_id, enum aie_module_type mod_type) |
| 148 | { | 175 | { |
| @@ -176,12 +203,40 @@ aie_get_error_category(u8 row, u8 event_id, enum aie_module_type mod_type) · 27 unmodified lines | |||
| 176 | if (event_id != lut[i].event_id) | 203 | if (event_id != lut[i].event_id) |
| 177 | continue; | 204 | continue; |
| 178 | 205 | ||
| 206 | if (lut[i].category > AIE_ERROR_UNKNOWN) | ||
| 207 | return AIE_ERROR_UNKNOWN; | ||
| 208 | |||
| 179 | return lut[i].category; | 209 | return lut[i].category; |
| 180 | } | 210 | } |
| 181 | 211 | ||
| 182 | return AIE_ERROR_UNKNOWN; | 212 | return AIE_ERROR_UNKNOWN; |
| 183 | } | 213 | } |
| 184 | 214 | ||
| 215 | static void aie2_update_last_async_error(struct amdxdna_dev_hdl *ndev, void *err_info, u32 num_err) | ||
| 216 | { | ||
| 217 | struct aie_error *errs = err_info; | ||
| 218 | enum amdxdna_error_module err_mod; | ||
| 219 | enum aie_error_category aie_err; | ||
| 220 | enum amdxdna_error_num err_num; | ||
| 221 | struct aie_error *last_err; | ||
| 222 | |||
| 223 | last_err = &errs[num_err - 1]; | ||
| 224 | if (last_err->mod_type >= AIE_UNKNOWN_MOD) { | ||
| 225 | err_num = aie_cat_err_num_map[AIE_ERROR_UNKNOWN]; | ||
| 226 | err_mod = aie_err_mod_map[AIE_UNKNOWN_MOD]; | ||
| 227 | } else { | ||
| 228 | aie_err = aie_get_error_category(last_err->row, | ||
| 229 | last_err->event_id, | ||
| 230 | last_err->mod_type); | ||
| 231 | err_num = aie_cat_err_num_map[aie_err]; | ||
| 232 | err_mod = aie_err_mod_map[last_err->mod_type]; | ||
| 233 | } | ||
| 234 | |||
| 235 | ndev->last_async_err.err_code = AMDXDNA_ERROR_ENCODE(err_num, err_mod); | ||
| 236 | ndev->last_async_err.ts_us = ktime_to_us(ktime_get_real()); | ||
| 237 | ndev->last_async_err.ex_err_code = AMDXDNA_EXTRA_ERR_ENCODE(last_err->row, last_err->col); | ||
| 238 | } | ||
| 239 | |||
| 185 | static u32 aie2_error_backtrack(struct amdxdna_dev_hdl *ndev, void *err_info, u32 num_err) | 240 | static u32 aie2_error_backtrack(struct amdxdna_dev_hdl *ndev, void *err_info, u32 num_err) |
| 186 | { | 241 | { |
| 187 | struct aie_error *errs = err_info; | 242 | struct aie_error *errs = err_info; |
| @@ -264,29 +319,14 @@ static void aie2_error_worker(struct work_struct *err_work) · 76 unmodified lines | |||
| 264 | } | 319 | } |
| 265 | 320 | ||
| 266 | mutex_lock(&xdna->dev_lock); | 321 | mutex_lock(&xdna->dev_lock); |
| 322 | aie2_update_last_async_error(e->ndev, info->payload, info->err_cnt); | ||
| 323 | |||
| 267 | /* Re-sent this event to firmware */ | 324 | /* Re-sent this event to firmware */ |
| 268 | if (aie2_error_event_send(e)) | 325 | if (aie2_error_event_send(e)) |
| 269 | XDNA_WARN(xdna, "Unable to register async event"); | 326 | XDNA_WARN(xdna, "Unable to register async event"); |
| 270 | mutex_unlock(&xdna->dev_lock); | 327 | mutex_unlock(&xdna->dev_lock); |
| 271 | } | 328 | } |
| 272 | 329 | ||
| 273 | int aie2_error_async_events_send(struct amdxdna_dev_hdl *ndev) | ||
| 274 | { | ||
| 275 | struct amdxdna_dev *xdna = ndev->xdna; | ||
| 276 | struct async_event *e; | ||
| 277 | int i, ret; | ||
| 278 | |||
| 279 | drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); | ||
| 280 | for (i = 0; i < ndev->async_events->event_cnt; i++) { | ||
| 281 | e = &ndev->async_events->event[i]; | ||
| 282 | ret = aie2_error_event_send(e); | ||
| 283 | if (ret) | ||
| 284 | return ret; | ||
| 285 | } | ||
| 286 | |||
| 287 | return 0; | ||
| 288 | } | ||
| 289 | |||
| 290 | void aie2_error_async_events_free(struct amdxdna_dev_hdl *ndev) | 330 | void aie2_error_async_events_free(struct amdxdna_dev_hdl *ndev) |
| 291 | { | 331 | { |
| 292 | struct amdxdna_dev *xdna = ndev->xdna; | 332 | struct amdxdna_dev *xdna = ndev->xdna; |
| @@ -341,6 +381,10 @@ int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev) · 48 unmodified lines | |||
| 341 | e->size = ASYNC_BUF_SIZE; | 381 | e->size = ASYNC_BUF_SIZE; |
| 342 | e->resp.status = MAX_AIE2_STATUS_CODE; | 382 | e->resp.status = MAX_AIE2_STATUS_CODE; |
| 343 | INIT_WORK(&e->work, aie2_error_worker); | 383 | INIT_WORK(&e->work, aie2_error_worker); |
| 384 | |||
| 385 | ret = aie2_error_event_send(e); | ||
| 386 | if (ret) | ||
| 387 | goto free_wq; | ||
| 344 | } | 388 | } |
| 345 | 389 | ||
| 346 | ndev->async_events = events; | 390 | ndev->async_events = events; |
| @@ -349,6 +393,8 @@ int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev) · 2 unmodified lines | |||
| 349 | events->event_cnt, events->size); | 393 | events->event_cnt, events->size); |
| 350 | return 0; | 394 | return 0; |
| 351 | 395 | ||
| 396 | free_wq: | ||
| 397 | destroy_workqueue(events->wq); | ||
| 352 | free_buf: | 398 | free_buf: |
| 353 | dma_free_noncoherent(xdna->ddev.dev, events->size, events->buf, | 399 | dma_free_noncoherent(xdna->ddev.dev, events->size, events->buf, |
| 354 | events->addr, DMA_FROM_DEVICE); | 400 | events->addr, DMA_FROM_DEVICE); |
| @@ -356,3 +402,18 @@ free_events: · 1 unmodified line | |||
| 356 | kfree(events); | 402 | kfree(events); |
| 357 | return ret; | 403 | return ret; |
| 358 | } | 404 | } |
| 405 | |||
| 406 | int aie2_get_array_async_error(struct amdxdna_dev_hdl *ndev, struct amdxdna_drm_get_array *args) | ||
| 407 | { | ||
| 408 | struct amdxdna_dev *xdna = ndev->xdna; | ||
| 409 | |||
| 410 | drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); | ||
| 411 | |||
| 412 | args->num_element = 1; | ||
| 413 | args->element_size = sizeof(ndev->last_async_err); | ||
| 414 | if (copy_to_user(u64_to_user_ptr(args->buffer), | ||
| 415 | &ndev->last_async_err, args->element_size)) | ||
| 416 | return -EFAULT; | ||
| 417 | |||
| 418 | return 0; | ||
| 419 | } | ||
| @@ -27,6 +27,8 @@ · 26 unmodified lines | |||
| 27 | #define DECLARE_AIE2_MSG(name, op) \ | 27 | #define DECLARE_AIE2_MSG(name, op) \ |
| 28 | DECLARE_XDNA_MSG_COMMON(name, op, MAX_AIE2_STATUS_CODE) | 28 | DECLARE_XDNA_MSG_COMMON(name, op, MAX_AIE2_STATUS_CODE) |
| 29 | 29 | ||
| 30 | #define EXEC_MSG_OPS(xdna) ((xdna)->dev_handle->exec_msg_ops) | ||
| 31 | |||
| 30 | static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev, | 32 | static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev, |
| 31 | struct xdna_mailbox_msg *msg) | 33 | struct xdna_mailbox_msg *msg) |
| 32 | { | 34 | { |
| @@ -37,7 +39,7 @@ static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev, · 4 unmodified lines | |||
| 37 | if (!ndev->mgmt_chann) | 39 | if (!ndev->mgmt_chann) |
| 38 | return -ENODEV; | 40 | return -ENODEV; |
| 39 | 41 | ||
| 40 | drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); | 42 | drm_WARN_ON(&xdna->ddev, xdna->rpm_on && !mutex_is_locked(&xdna->dev_lock)); |
| 41 | ret = xdna_send_msg_wait(xdna, ndev->mgmt_chann, msg); | 43 | ret = xdna_send_msg_wait(xdna, ndev->mgmt_chann, msg); |
| 42 | if (ret == -ETIME) { | 44 | if (ret == -ETIME) { |
| 43 | xdna_mailbox_stop_channel(ndev->mgmt_chann); | 45 | xdna_mailbox_stop_channel(ndev->mgmt_chann); |
| @@ -45,7 +47,7 @@ static int aie2_send_mgmt_msg_wait(struct amdxdna_dev_hdl *ndev, · 1 unmodified line | |||
| 45 | ndev->mgmt_chann = NULL; | 47 | ndev->mgmt_chann = NULL; |
| 46 | } | 48 | } |
| 47 | 49 | ||
| 48 | if (!ret && *hdl->data != AIE2_STATUS_SUCCESS) { | 50 | if (!ret && *hdl->status != AIE2_STATUS_SUCCESS) { |
| 49 | XDNA_ERR(xdna, "command opcode 0x%x failed, status 0x%x", | 51 | XDNA_ERR(xdna, "command opcode 0x%x failed, status 0x%x", |
| 50 | msg->opcode, *hdl->data); | 52 | msg->opcode, *hdl->data); |
| 51 | ret = -EINVAL; | 53 | ret = -EINVAL; |
| @@ -208,6 +210,14 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct · 156 unmodified lines | |||
| 208 | hwctx->fw_ctx_id = resp.context_id; | 210 | hwctx->fw_ctx_id = resp.context_id; |
| 209 | WARN_ONCE(hwctx->fw_ctx_id == -1, "Unexpected context id"); | 211 | WARN_ONCE(hwctx->fw_ctx_id == -1, "Unexpected context id"); |
| 210 | 212 | ||
| 213 | if (ndev->force_preempt_enabled) { | ||
| 214 | ret = aie2_runtime_cfg(ndev, AIE2_RT_CFG_FORCE_PREEMPT, &hwctx->fw_ctx_id); | ||
| 215 | if (ret) { | ||
| 216 | XDNA_ERR(xdna, "failed to enable force preempt %d", ret); | ||
| 217 | return ret; | ||
| 218 | } | ||
| 219 | } | ||
| 220 | |||
| 211 | cq_pair = &resp.cq_pair[0]; | 221 | cq_pair = &resp.cq_pair[0]; |
| 212 | x2i.mb_head_ptr_reg = AIE2_MBOX_OFF(ndev, cq_pair->x2i_q.head_addr); | 222 | x2i.mb_head_ptr_reg = AIE2_MBOX_OFF(ndev, cq_pair->x2i_q.head_addr); |
| 213 | x2i.mb_tail_ptr_reg = AIE2_MBOX_OFF(ndev, cq_pair->x2i_q.tail_addr); | 223 | x2i.mb_tail_ptr_reg = AIE2_MBOX_OFF(ndev, cq_pair->x2i_q.tail_addr); |
| @@ -233,6 +243,7 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct · 19 unmodified lines | |||
| 233 | ret = -EINVAL; | 243 | ret = -EINVAL; |
| 234 | goto out_destroy_context; | 244 | goto out_destroy_context; |
| 235 | } | 245 | } |
| 246 | ndev->hwctx_num++; | ||
| 236 | 247 | ||
| 237 | XDNA_DBG(xdna, "%s mailbox channel irq: %d, msix_id: %d", | 248 | XDNA_DBG(xdna, "%s mailbox channel irq: %d, msix_id: %d", |
| 238 | hwctx->name, ret, resp.msix_id); | 249 | hwctx->name, ret, resp.msix_id); |
| @@ -267,6 +278,7 @@ int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwc · 28 unmodified lines | |||
| 267 | hwctx->fw_ctx_id); | 278 | hwctx->fw_ctx_id); |
| 268 | hwctx->priv->mbox_chann = NULL; | 279 | hwctx->priv->mbox_chann = NULL; |
| 269 | hwctx->fw_ctx_id = -1; | 280 | hwctx->fw_ctx_id = -1; |
| 281 | ndev->hwctx_num--; | ||
| 270 | 282 | ||
| 271 | return ret; | 283 | return ret; |
| 272 | } | 284 | } |
| @@ -332,11 +344,6 @@ int aie2_query_status(struct amdxdna_dev_hdl *ndev, char __user *buf, · 59 unmodified lines | |||
| 332 | goto fail; | 344 | goto fail; |
| 333 | } | 345 | } |
| 334 | 346 | ||
| 335 | if (resp.status != AIE2_STATUS_SUCCESS) { | ||
| 336 | XDNA_ERR(xdna, "Query NPU status failed, status 0x%x", resp.status); | ||
| 337 | ret = -EINVAL; | ||
| 338 | goto fail; | ||
| 339 | } | ||
| 340 | XDNA_DBG(xdna, "Query NPU status completed"); | 347 | XDNA_DBG(xdna, "Query NPU status completed"); |
| 341 | 348 | ||
| 342 | if (size < resp.size) { | 349 | if (size < resp.size) { |
| @@ -358,6 +365,55 @@ fail: · 15 unmodified lines | |||
| 358 | return ret; | 365 | return ret; |
| 359 | } | 366 | } |
| 360 | 367 | ||
| 368 | int aie2_query_telemetry(struct amdxdna_dev_hdl *ndev, | ||
| 369 | char __user *buf, u32 size, | ||
| 370 | struct amdxdna_drm_query_telemetry_header *header) | ||
| 371 | { | ||
| 372 | DECLARE_AIE2_MSG(get_telemetry, MSG_OP_GET_TELEMETRY); | ||
| 373 | struct amdxdna_dev *xdna = ndev->xdna; | ||
| 374 | dma_addr_t dma_addr; | ||
| 375 | u8 *addr; | ||
| 376 | int ret; | ||
| 377 | |||
| 378 | if (header->type >= MAX_TELEMETRY_TYPE) | ||
| 379 | return -EINVAL; | ||
| 380 | |||
| 381 | addr = dma_alloc_noncoherent(xdna->ddev.dev, size, &dma_addr, | ||
| 382 | DMA_FROM_DEVICE, GFP_KERNEL); | ||
| 383 | if (!addr) | ||
| 384 | return -ENOMEM; | ||
| 385 | |||
| 386 | req.buf_addr = dma_addr; | ||
| 387 | req.buf_size = size; | ||
| 388 | req.type = header->type; | ||
| 389 | |||
| 390 | drm_clflush_virt_range(addr, size); /* device can access */ | ||
| 391 | ret = aie2_send_mgmt_msg_wait(ndev, &msg); | ||
| 392 | if (ret) { | ||
| 393 | XDNA_ERR(xdna, "Query telemetry failed, status %d", ret); | ||
| 394 | goto free_buf; | ||
| 395 | } | ||
| 396 | |||
| 397 | if (size < resp.size) { | ||
| 398 | ret = -EINVAL; | ||
| 399 | XDNA_ERR(xdna, "Bad buffer size. Available: %u. Needs: %u", size, resp.size); | ||
| 400 | goto free_buf; | ||
| 401 | } | ||
| 402 | |||
| 403 | if (copy_to_user(buf, addr, resp.size)) { | ||
| 404 | ret = -EFAULT; | ||
| 405 | XDNA_ERR(xdna, "Failed to copy telemetry to user space"); | ||
| 406 | goto free_buf; | ||
| 407 | } | ||
| 408 | |||
| 409 | header->major = resp.major; | ||
| 410 | header->minor = resp.minor; | ||
| 411 | |||
| 412 | free_buf: | ||
| 413 | dma_free_noncoherent(xdna->ddev.dev, size, addr, dma_addr, DMA_FROM_DEVICE); | ||
| 414 | return ret; | ||
| 415 | } | ||
| 416 | |||
| 361 | int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr, u32 size, | 417 | int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr, u32 size, |
| 362 | void *handle, int (*cb)(void*, void __iomem *, size_t)) | 418 | void *handle, int (*cb)(void*, void __iomem *, size_t)) |
| 363 | { | 419 | { |
| @@ -377,15 +433,17 @@ int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr, · 13 unmodified lines | |||
| 377 | return xdna_mailbox_send_msg(ndev->mgmt_chann, &msg, TX_TIMEOUT); | 433 | return xdna_mailbox_send_msg(ndev->mgmt_chann, &msg, TX_TIMEOUT); |
| 378 | } | 434 | } |
| 379 | 435 | ||
| 380 | int aie2_config_cu(struct amdxdna_hwctx *hwctx) | 436 | int aie2_config_cu(struct amdxdna_hwctx *hwctx, |
| 437 | int (*notify_cb)(void *, void __iomem *, size_t)) | ||
| 381 | { | 438 | { |
| 382 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | 439 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 383 | struct amdxdna_dev *xdna = hwctx->client->xdna; | 440 | struct amdxdna_dev *xdna = hwctx->client->xdna; |
| 384 | u32 shift = xdna->dev_info->dev_mem_buf_shift; | 441 | u32 shift = xdna->dev_info->dev_mem_buf_shift; |
| 385 | DECLARE_AIE2_MSG(config_cu, MSG_OP_CONFIG_CU); | 442 | struct config_cu_req req = { 0 }; |
| 443 | struct xdna_mailbox_msg msg; | ||
| 386 | struct drm_gem_object *gobj; | 444 | struct drm_gem_object *gobj; |
| 387 | struct amdxdna_gem_obj *abo; | 445 | struct amdxdna_gem_obj *abo; |
| 388 | int ret, i; | 446 | int i; |
| 389 | 447 | ||
| 390 | if (!chann) | 448 | if (!chann) |
| 391 | return -ENODEV; | 449 | return -ENODEV; |
| @@ -423,191 +481,386 @@ int aie2_config_cu(struct amdxdna_hwctx *hwctx) · 31 unmodified lines | |||
| 423 | } | 481 | } |
| 424 | req.num_cus = hwctx->cus->num_cus; | 482 | req.num_cus = hwctx->cus->num_cus; |
| 425 | 483 | ||
| 426 | ret = xdna_send_msg_wait(xdna, chann, &msg); | 484 | msg.send_data = (u8 *)&req; |
| 427 | if (ret == -ETIME) | 485 | msg.send_size = sizeof(req); |
| 428 | aie2_destroy_context(xdna->dev_handle, hwctx); | 486 | msg.handle = hwctx; |
| 487 | msg.opcode = MSG_OP_CONFIG_CU; | ||
| 488 | msg.notify_cb = notify_cb; | ||
| 489 | return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | ||
| 490 | } | ||
| 429 | 491 | ||
| 430 | if (resp.status == AIE2_STATUS_SUCCESS) { | 492 | static int aie2_init_exec_cu_req(struct amdxdna_gem_obj *cmd_bo, void *req, |
| 431 | XDNA_DBG(xdna, "Configure %d CUs, ret %d", req.num_cus, ret); | 493 | size_t *size, u32 *msg_op) |
| 432 | return 0; | 494 | { |
| 433 | } | 495 | struct execute_buffer_req *cu_req = req; |
| 496 | u32 cmd_len; | ||
| 497 | void *cmd; | ||
| 434 | 498 | ||
| 435 | XDNA_ERR(xdna, "Command opcode 0x%x failed, status 0x%x ret %d", | 499 | cmd = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); |
| 436 | msg.opcode, resp.status, ret); | 500 | if (cmd_len > sizeof(cu_req->payload)) |
| 437 | return ret; | 501 | return -EINVAL; |
| 502 | |||
| 503 | cu_req->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); | ||
| 504 | if (cu_req->cu_idx == INVALID_CU_IDX) | ||
| 505 | return -EINVAL; | ||
| 506 | |||
| 507 | memcpy(cu_req->payload, cmd, cmd_len); | ||
| 508 | |||
| 509 | *size = sizeof(*cu_req); | ||
| 510 | *msg_op = MSG_OP_EXECUTE_BUFFER_CF; | ||
| 511 | return 0; | ||
| 438 | } | 512 | } |
| 439 | 513 | ||
| 440 | int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, | 514 | static int aie2_init_exec_dpu_req(struct amdxdna_gem_obj *cmd_bo, void *req, |
| 441 | int (*notify_cb)(void *, void __iomem *, size_t)) | 515 | size_t *size, u32 *msg_op) |
| 442 | { | 516 | { |
| 443 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | 517 | struct exec_dpu_req *dpu_req = req; |
| 444 | struct amdxdna_dev *xdna = hwctx->client->xdna; | 518 | struct amdxdna_cmd_start_npu *sn; |
| 445 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | 519 | u32 cmd_len; |
| 446 | union { | ||
| 447 | struct execute_buffer_req ebuf; | ||
| 448 | struct exec_dpu_req dpu; | ||
| 449 | } req; | ||
| 450 | struct xdna_mailbox_msg msg; | ||
| 451 | u32 payload_len; | ||
| 452 | void *payload; | ||
| 453 | int cu_idx; | ||
| 454 | int ret; | ||
| 455 | u32 op; | ||
| 456 | 520 | ||
| 457 | if (!chann) | 521 | sn = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); |
| 458 | return -ENODEV; | 522 | if (cmd_len - sizeof(*sn) > sizeof(dpu_req->payload)) |
| 523 | return -EINVAL; | ||
| 459 | 524 | ||
| 460 | payload = amdxdna_cmd_get_payload(cmd_abo, &payload_len); | 525 | dpu_req->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); |
| 461 | if (!payload) { | 526 | if (dpu_req->cu_idx == INVALID_CU_IDX) |
| 462 | XDNA_ERR(xdna, "Invalid command, cannot get payload"); | ||
| 463 | return -EINVAL; | 527 | return -EINVAL; |
| 464 | } | ||
| 465 | 528 | ||
| 466 | cu_idx = amdxdna_cmd_get_cu_idx(cmd_abo); | 529 | dpu_req->inst_buf_addr = sn->buffer; |
| 467 | if (cu_idx < 0) { | 530 | dpu_req->inst_size = sn->buffer_size; |
| 468 | XDNA_DBG(xdna, "Invalid cu idx"); | 531 | dpu_req->inst_prop_cnt = sn->prop_count; |
| 532 | memcpy(dpu_req->payload, sn->prop_args, cmd_len - sizeof(*sn)); | ||
| 533 | |||
| 534 | *size = sizeof(*dpu_req); | ||
| 535 | *msg_op = MSG_OP_EXEC_DPU; | ||
| 536 | return 0; | ||
| 537 | } | ||
| 538 | |||
| 539 | static void aie2_init_exec_chain_req(void *req, u64 slot_addr, size_t size, u32 cmd_cnt) | ||
| 540 | { | ||
| 541 | struct cmd_chain_req *chain_req = req; | ||
| 542 | |||
| 543 | chain_req->buf_addr = slot_addr; | ||
| 544 | chain_req->buf_size = size; | ||
| 545 | chain_req->count = cmd_cnt; | ||
| 546 | } | ||
| 547 | |||
| 548 | static void aie2_init_npu_chain_req(void *req, u64 slot_addr, size_t size, u32 cmd_cnt) | ||
| 549 | { | ||
| 550 | struct cmd_chain_npu_req *npu_chain_req = req; | ||
| 551 | |||
| 552 | npu_chain_req->flags = 0; | ||
| 553 | npu_chain_req->reserved = 0; | ||
| 554 | npu_chain_req->buf_addr = slot_addr; | ||
| 555 | npu_chain_req->buf_size = size; | ||
| 556 | npu_chain_req->count = cmd_cnt; | ||
| 557 | } | ||
| 558 | |||
| 559 | static int | ||
| 560 | aie2_cmdlist_fill_cf(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) | ||
| 561 | { | ||
| 562 | struct cmd_chain_slot_execbuf_cf *cf_slot = slot; | ||
| 563 | u32 cmd_len; | ||
| 564 | void *cmd; | ||
| 565 | |||
| 566 | cmd = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); | ||
| 567 | if (*size < sizeof(*cf_slot) + cmd_len) | ||
| 469 | return -EINVAL; | 568 | return -EINVAL; |
| 470 | } | ||
| 471 | 569 | ||
| 472 | op = amdxdna_cmd_get_op(cmd_abo); | 570 | cf_slot->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); |
| 473 | switch (op) { | 571 | if (cf_slot->cu_idx == INVALID_CU_IDX) |
| 572 | return -EINVAL; | ||
| 573 | |||
| 574 | cf_slot->arg_cnt = cmd_len / sizeof(u32); | ||
| 575 | memcpy(cf_slot->args, cmd, cmd_len); | ||
| 576 | /* Accurate slot size to hint firmware to do necessary copy */ | ||
| 577 | *size = sizeof(*cf_slot) + cmd_len; | ||
| 578 | return 0; | ||
| 579 | } | ||
| 580 | |||
| 581 | static int | ||
| 582 | aie2_cmdlist_fill_dpu(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) | ||
| 583 | { | ||
| 584 | struct cmd_chain_slot_dpu *dpu_slot = slot; | ||
| 585 | struct amdxdna_cmd_start_npu *sn; | ||
| 586 | u32 cmd_len; | ||
| 587 | u32 arg_sz; | ||
| 588 | |||
| 589 | sn = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); | ||
| 590 | arg_sz = cmd_len - sizeof(*sn); | ||
| 591 | if (cmd_len < sizeof(*sn) || arg_sz > MAX_DPU_ARGS_SIZE) | ||
| 592 | return -EINVAL; | ||
| 593 | |||
| 594 | if (*size < sizeof(*dpu_slot) + arg_sz) | ||
| 595 | return -EINVAL; | ||
| 596 | |||
| 597 | dpu_slot->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); | ||
| 598 | if (dpu_slot->cu_idx == INVALID_CU_IDX) | ||
| 599 | return -EINVAL; | ||
| 600 | |||
| 601 | dpu_slot->inst_buf_addr = sn->buffer; | ||
| 602 | dpu_slot->inst_size = sn->buffer_size; | ||
| 603 | dpu_slot->inst_prop_cnt = sn->prop_count; | ||
| 604 | dpu_slot->arg_cnt = arg_sz / sizeof(u32); | ||
| 605 | memcpy(dpu_slot->args, sn->prop_args, arg_sz); | ||
| 606 | |||
| 607 | /* Accurate slot size to hint firmware to do necessary copy */ | ||
| 608 | *size = sizeof(*dpu_slot) + arg_sz; | ||
| 609 | return 0; | ||
| 610 | } | ||
| 611 | |||
| 612 | static int aie2_cmdlist_unsupp(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) | ||
| 613 | { | ||
| 614 | return -EOPNOTSUPP; | ||
| 615 | } | ||
| 616 | |||
| 617 | static u32 aie2_get_chain_msg_op(u32 cmd_op) | ||
| 618 | { | ||
| 619 | switch (cmd_op) { | ||
| 474 | case ERT_START_CU: | 620 | case ERT_START_CU: |
| 475 | if (unlikely(payload_len > sizeof(req.ebuf.payload))) | 621 | return MSG_OP_CHAIN_EXEC_BUFFER_CF; |
| 476 | XDNA_DBG(xdna, "Invalid ebuf payload len: %d", payload_len); | 622 | case ERT_START_NPU: |
| 477 | req.ebuf.cu_idx = cu_idx; | 623 | return MSG_OP_CHAIN_EXEC_DPU; |
| 478 | memcpy(req.ebuf.payload, payload, sizeof(req.ebuf.payload)); | 624 | default: |
| 479 | msg.send_size = sizeof(req.ebuf); | ||
| 480 | msg.opcode = MSG_OP_EXECUTE_BUFFER_CF; | ||
| 481 | break; | ||
| 482 | case ERT_START_NPU: { | ||
| 483 | struct amdxdna_cmd_start_npu *sn = payload; | ||
| 484 | |||
| 485 | if (unlikely(payload_len - sizeof(*sn) > sizeof(req.dpu.payload))) | ||
| 486 | XDNA_DBG(xdna, "Invalid dpu payload len: %d", payload_len); | ||
| 487 | req.dpu.inst_buf_addr = sn->buffer; | ||
| 488 | req.dpu.inst_size = sn->buffer_size; | ||
| 489 | req.dpu.inst_prop_cnt = sn->prop_count; | ||
| 490 | req.dpu.cu_idx = cu_idx; | ||
| 491 | memcpy(req.dpu.payload, sn->prop_args, sizeof(req.dpu.payload)); | ||
| 492 | msg.send_size = sizeof(req.dpu); | ||
| 493 | msg.opcode = MSG_OP_EXEC_DPU; | ||
| 494 | break; | 625 | break; |
| 495 | } | 626 | } |
| 496 | default: | 627 | |
| 497 | XDNA_DBG(xdna, "Invalid ERT cmd op code: %d", op); | 628 | return MSG_OP_MAX_OPCODE; |
| 629 | } | ||
| 630 | |||
| 631 | static struct aie2_exec_msg_ops legacy_exec_message_ops = { | ||
| 632 | .init_cu_req = aie2_init_exec_cu_req, | ||
| 633 | .init_dpu_req = aie2_init_exec_dpu_req, | ||
| 634 | .init_chain_req = aie2_init_exec_chain_req, | ||
| 635 | .fill_cf_slot = aie2_cmdlist_fill_cf, | ||
| 636 | .fill_dpu_slot = aie2_cmdlist_fill_dpu, | ||
| 637 | .fill_preempt_slot = aie2_cmdlist_unsupp, | ||
| 638 | .fill_elf_slot = aie2_cmdlist_unsupp, | ||
| 639 | .get_chain_msg_op = aie2_get_chain_msg_op, | ||
| 640 | }; | ||
| 641 | |||
| 642 | static int | ||
| 643 | aie2_cmdlist_fill_npu_cf(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) | ||
| 644 | { | ||
| 645 | struct cmd_chain_slot_npu *npu_slot = slot; | ||
| 646 | u32 cmd_len; | ||
| 647 | void *cmd; | ||
| 648 | |||
| 649 | cmd = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); | ||
| 650 | if (*size < sizeof(*npu_slot) + cmd_len) | ||
| 498 | return -EINVAL; | 651 | return -EINVAL; |
| 499 | } | ||
| 500 | msg.handle = job; | ||
| 501 | msg.notify_cb = notify_cb; | ||
| 502 | msg.send_data = (u8 *)&req; | ||
| 503 | print_hex_dump_debug("cmd: ", DUMP_PREFIX_OFFSET, 16, 4, &req, | ||
| 504 | 0x40, false); | ||
| 505 | 652 | ||
| 506 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | 653 | npu_slot->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); |
| 507 | if (ret) { | 654 | if (npu_slot->cu_idx == INVALID_CU_IDX) |
| 508 | XDNA_ERR(xdna, "Send message failed"); | 655 | return -EINVAL; |
| 509 | return ret; | ||
| 510 | } | ||
| 511 | 656 | ||
| 657 | memset(npu_slot, 0, sizeof(*npu_slot)); | ||
| 658 | npu_slot->type = EXEC_NPU_TYPE_NON_ELF; | ||
| 659 | npu_slot->arg_cnt = cmd_len / sizeof(u32); | ||
| 660 | memcpy(npu_slot->args, cmd, cmd_len); | ||
| 661 | |||
| 662 | *size = sizeof(*npu_slot) + cmd_len; | ||
| 512 | return 0; | 663 | return 0; |
| 513 | } | 664 | } |
| 514 | 665 | ||
| 515 | static int | 666 | static int |
| 516 | aie2_cmdlist_fill_one_slot_cf(void *cmd_buf, u32 offset, | 667 | aie2_cmdlist_fill_npu_dpu(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) |
| 517 | struct amdxdna_gem_obj *abo, u32 *size) | ||
| 518 | { | 668 | { |
| 519 | struct cmd_chain_slot_execbuf_cf *buf = cmd_buf + offset; | 669 | struct cmd_chain_slot_npu *npu_slot = slot; |
| 520 | int cu_idx = amdxdna_cmd_get_cu_idx(abo); | 670 | struct amdxdna_cmd_start_npu *sn; |
| 521 | u32 payload_len; | 671 | u32 cmd_len; |
| 522 | void *payload; | 672 | u32 arg_sz; |
| 523 | 673 | ||
| 524 | if (cu_idx < 0) | 674 | sn = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); |
| 675 | arg_sz = cmd_len - sizeof(*sn); | ||
| 676 | if (cmd_len < sizeof(*sn) || arg_sz > MAX_NPU_ARGS_SIZE) | ||
| 525 | return -EINVAL; | 677 | return -EINVAL; |
| 526 | 678 | ||
| 527 | payload = amdxdna_cmd_get_payload(abo, &payload_len); | 679 | if (*size < sizeof(*npu_slot) + arg_sz) |
| 528 | if (!payload) | ||
| 529 | return -EINVAL; | 680 | return -EINVAL; |
| 530 | 681 | ||
| 531 | if (!slot_has_space(*buf, offset, payload_len)) | 682 | npu_slot->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); |
| 532 | return -ENOSPC; | 683 | if (npu_slot->cu_idx == INVALID_CU_IDX) |
| 684 | return -EINVAL; | ||
| 685 | |||
| 686 | memset(npu_slot, 0, sizeof(*npu_slot)); | ||
| 687 | npu_slot->type = EXEC_NPU_TYPE_PARTIAL_ELF; | ||
| 688 | npu_slot->inst_buf_addr = sn->buffer; | ||
| 689 | npu_slot->inst_size = sn->buffer_size; | ||
| 690 | npu_slot->inst_prop_cnt = sn->prop_count; | ||
| 691 | npu_slot->arg_cnt = arg_sz / sizeof(u32); | ||
| 692 | memcpy(npu_slot->args, sn->prop_args, arg_sz); | ||
| 533 | 693 | ||
| 534 | buf->cu_idx = cu_idx; | 694 | *size = sizeof(*npu_slot) + arg_sz; |
| 535 | buf->arg_cnt = payload_len / sizeof(u32); | ||
| 536 | memcpy(buf->args, payload, payload_len); | ||
| 537 | /* Accurate buf size to hint firmware to do necessary copy */ | ||
| 538 | *size = sizeof(*buf) + payload_len; | ||
| 539 | return 0; | 695 | return 0; |
| 540 | } | 696 | } |
| 541 | 697 | ||
| 542 | static int | 698 | static int |
| 543 | aie2_cmdlist_fill_one_slot_dpu(void *cmd_buf, u32 offset, | 699 | aie2_cmdlist_fill_npu_preempt(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) |
| 544 | struct amdxdna_gem_obj *abo, u32 *size) | ||
| 545 | { | 700 | { |
| 546 | struct cmd_chain_slot_dpu *buf = cmd_buf + offset; | 701 | struct cmd_chain_slot_npu *npu_slot = slot; |
| 547 | int cu_idx = amdxdna_cmd_get_cu_idx(abo); | 702 | struct amdxdna_cmd_preempt_data *pd; |
| 548 | struct amdxdna_cmd_start_npu *sn; | 703 | u32 cmd_len; |
| 549 | u32 payload_len; | ||
| 550 | void *payload; | ||
| 551 | u32 arg_sz; | 704 | u32 arg_sz; |
| 552 | 705 | ||
| 553 | if (cu_idx < 0) | 706 | pd = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); |
| 707 | arg_sz = cmd_len - sizeof(*pd); | ||
| 708 | if (cmd_len < sizeof(*pd) || arg_sz > MAX_NPU_ARGS_SIZE) | ||
| 554 | return -EINVAL; | 709 | return -EINVAL; |
| 555 | 710 | ||
| 556 | payload = amdxdna_cmd_get_payload(abo, &payload_len); | 711 | if (*size < sizeof(*npu_slot) + arg_sz) |
| 557 | if (!payload) | ||
| 558 | return -EINVAL; | 712 | return -EINVAL; |
| 559 | sn = payload; | 713 | |
| 560 | arg_sz = payload_len - sizeof(*sn); | 714 | npu_slot->cu_idx = amdxdna_cmd_get_cu_idx(cmd_bo); |
| 561 | if (payload_len < sizeof(*sn) || arg_sz > MAX_DPU_ARGS_SIZE) | 715 | if (npu_slot->cu_idx == INVALID_CU_IDX) |
| 562 | return -EINVAL; | 716 | return -EINVAL; |
| 563 | 717 | ||
| 564 | if (!slot_has_space(*buf, offset, arg_sz)) | 718 | memset(npu_slot, 0, sizeof(*npu_slot)); |
| 565 | return -ENOSPC; | 719 | npu_slot->type = EXEC_NPU_TYPE_PREEMPT; |
| 720 | npu_slot->inst_buf_addr = pd->inst_buf; | ||
| 721 | npu_slot->save_buf_addr = pd->save_buf; | ||
| 722 | npu_slot->restore_buf_addr = pd->restore_buf; | ||
| 723 | npu_slot->inst_size = pd->inst_size; | ||
| 724 | npu_slot->save_size = pd->save_size; | ||
| 725 | npu_slot->restore_size = pd->restore_size; | ||
| 726 | npu_slot->inst_prop_cnt = pd->inst_prop_cnt; | ||
| 727 | npu_slot->arg_cnt = arg_sz / sizeof(u32); | ||
| 728 | memcpy(npu_slot->args, pd->prop_args, arg_sz); | ||
| 729 | |||
| 730 | *size = sizeof(*npu_slot) + arg_sz; | ||
| 731 | return 0; | ||
| 732 | } | ||
| 733 | |||
| 734 | static int | ||
| 735 | aie2_cmdlist_fill_npu_elf(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size) | ||
| 736 | { | ||
| 737 | struct cmd_chain_slot_npu *npu_slot = slot; | ||
| 738 | struct amdxdna_cmd_preempt_data *pd; | ||
| 739 | u32 cmd_len; | ||
| 740 | u32 arg_sz; | ||
| 741 | |||
| 742 | pd = amdxdna_cmd_get_payload(cmd_bo, &cmd_len); | ||
| 743 | arg_sz = cmd_len - sizeof(*pd); | ||
| 744 | if (cmd_len < sizeof(*pd) || arg_sz > MAX_NPU_ARGS_SIZE) | ||
| 745 | return -EINVAL; | ||
| 566 | 746 | ||
| 567 | buf->inst_buf_addr = sn->buffer; | 747 | if (*size < sizeof(*npu_slot) + arg_sz) |
| 568 | buf->inst_size = sn->buffer_size; | 748 | return -EINVAL; |
| 569 | buf->inst_prop_cnt = sn->prop_count; | ||
| 570 | buf->cu_idx = cu_idx; | ||
| 571 | buf->arg_cnt = arg_sz / sizeof(u32); | ||
| 572 | memcpy(buf->args, sn->prop_args, arg_sz); | ||
| 573 | 749 | ||
| 574 | /* Accurate buf size to hint firmware to do necessary copy */ | 750 | memset(npu_slot, 0, sizeof(*npu_slot)); |
| 575 | *size = sizeof(*buf) + arg_sz; | 751 | npu_slot->type = EXEC_NPU_TYPE_ELF; |
| 752 | npu_slot->inst_buf_addr = pd->inst_buf; | ||
| 753 | npu_slot->save_buf_addr = pd->save_buf; | ||
| 754 | npu_slot->restore_buf_addr = pd->restore_buf; | ||
| 755 | npu_slot->inst_size = pd->inst_size; | ||
| 756 | npu_slot->save_size = pd->save_size; | ||
| 757 | npu_slot->restore_size = pd->restore_size; | ||
| 758 | npu_slot->inst_prop_cnt = pd->inst_prop_cnt; | ||
| 759 | npu_slot->arg_cnt = 1; | ||
| 760 | npu_slot->args[0] = AIE2_EXEC_BUFFER_KERNEL_OP_TXN; | ||
| 761 | |||
| 762 | *size = struct_size(npu_slot, args, npu_slot->arg_cnt); | ||
| 576 | return 0; | 763 | return 0; |
| 577 | } | 764 | } |
| 578 | 765 | ||
| 579 | static int | 766 | static u32 aie2_get_npu_chain_msg_op(u32 cmd_op) |
| 580 | aie2_cmdlist_fill_one_slot(u32 op, struct amdxdna_gem_obj *cmdbuf_abo, u32 offset, | 767 | { |
| 581 | struct amdxdna_gem_obj *abo, u32 *size) | 768 | return MSG_OP_CHAIN_EXEC_NPU; |
| 769 | } | ||
| 770 | |||
| 771 | static struct aie2_exec_msg_ops npu_exec_message_ops = { | ||
| 772 | .init_cu_req = aie2_init_exec_cu_req, | ||
| 773 | .init_dpu_req = aie2_init_exec_dpu_req, | ||
| 774 | .init_chain_req = aie2_init_npu_chain_req, | ||
| 775 | .fill_cf_slot = aie2_cmdlist_fill_npu_cf, | ||
| 776 | .fill_dpu_slot = aie2_cmdlist_fill_npu_dpu, | ||
| 777 | .fill_preempt_slot = aie2_cmdlist_fill_npu_preempt, | ||
| 778 | .fill_elf_slot = aie2_cmdlist_fill_npu_elf, | ||
| 779 | .get_chain_msg_op = aie2_get_npu_chain_msg_op, | ||
| 780 | }; | ||
| 781 | |||
| 782 | static int aie2_init_exec_req(void *req, struct amdxdna_gem_obj *cmd_abo, | ||
| 783 | size_t *size, u32 *msg_op) | ||
| 582 | { | 784 | { |
| 583 | u32 this_op = amdxdna_cmd_get_op(abo); | 785 | struct amdxdna_dev *xdna = cmd_abo->client->xdna; |
| 584 | void *cmd_buf = cmdbuf_abo->mem.kva; | ||
| 585 | int ret; | 786 | int ret; |
| 787 | u32 op; | ||
| 586 | 788 | ||
| 587 | if (this_op != op) { | ||
| 588 | ret = -EINVAL; | ||
| 589 | goto done; | ||
| 590 | } | ||
| 591 | 789 | ||
| 790 | op = amdxdna_cmd_get_op(cmd_abo); | ||
| 592 | switch (op) { | 791 | switch (op) { |
| 593 | case ERT_START_CU: | 792 | case ERT_START_CU: |
| 594 | ret = aie2_cmdlist_fill_one_slot_cf(cmd_buf, offset, abo, size); | 793 | ret = EXEC_MSG_OPS(xdna)->init_cu_req(cmd_abo, req, size, msg_op); |
| 794 | if (ret) { | ||
| 795 | XDNA_DBG(xdna, "Init CU req failed ret %d", ret); | ||
| 796 | return ret; | ||
| 797 | } | ||
| 595 | break; | 798 | break; |
| 596 | case ERT_START_NPU: | 799 | case ERT_START_NPU: |
| 597 | ret = aie2_cmdlist_fill_one_slot_dpu(cmd_buf, offset, abo, size); | 800 | ret = EXEC_MSG_OPS(xdna)->init_dpu_req(cmd_abo, req, size, msg_op); |
| 801 | if (ret) { | ||
| 802 | XDNA_DBG(xdna, "Init DPU req failed ret %d", ret); | ||
| 803 | return ret; | ||
| 804 | } | ||
| 805 | |||
| 598 | break; | 806 | break; |
| 599 | default: | 807 | default: |
| 808 | XDNA_ERR(xdna, "Unsupported op %d", op); | ||
| 600 | ret = -EOPNOTSUPP; | 809 | ret = -EOPNOTSUPP; |
| 810 | break; | ||
| 601 | } | 811 | } |
| 602 | 812 | ||
| 603 | done: | 813 | return ret; |
| 604 | if (ret) { | 814 | } |
| 605 | XDNA_ERR(abo->client->xdna, "Can't fill slot for cmd op %d ret %d", | 815 | |
| 606 | op, ret); | 816 | static int |
| 817 | aie2_cmdlist_fill_slot(void *slot, struct amdxdna_gem_obj *cmd_abo, | ||
| 818 | size_t *size, u32 *cmd_op) | ||
| 819 | { | ||
| 820 | struct amdxdna_dev *xdna = cmd_abo->client->xdna; | ||
| 821 | int ret; | ||
| 822 | u32 op; | ||
| 823 | |||
| 824 | op = amdxdna_cmd_get_op(cmd_abo); | ||
| 825 | if (*cmd_op == ERT_INVALID_CMD) | ||
| 826 | *cmd_op = op; | ||
| 827 | else if (op != *cmd_op) | ||
| 828 | return -EINVAL; | ||
| 829 | |||
| 830 | switch (op) { | ||
| 831 | case ERT_START_CU: | ||
| 832 | ret = EXEC_MSG_OPS(xdna)->fill_cf_slot(cmd_abo, slot, size); | ||
| 833 | break; | ||
| 834 | case ERT_START_NPU: | ||
| 835 | ret = EXEC_MSG_OPS(xdna)->fill_dpu_slot(cmd_abo, slot, size); | ||
| 836 | break; | ||
| 837 | case ERT_START_NPU_PREEMPT: | ||
| 838 | if (!AIE2_FEATURE_ON(xdna->dev_handle, AIE2_PREEMPT)) | ||
| 839 | return -EOPNOTSUPP; | ||
| 840 | ret = EXEC_MSG_OPS(xdna)->fill_preempt_slot(cmd_abo, slot, size); | ||
| 841 | break; | ||
| 842 | case ERT_START_NPU_PREEMPT_ELF: | ||
| 843 | if (!AIE2_FEATURE_ON(xdna->dev_handle, AIE2_PREEMPT)) | ||
| 844 | return -EOPNOTSUPP; | ||
| 845 | ret = EXEC_MSG_OPS(xdna)->fill_elf_slot(cmd_abo, slot, size); | ||
| 846 | break; | ||
| 847 | default: | ||
| 848 | XDNA_INFO(xdna, "Unsupported op %d", op); | ||
| 849 | ret = -EOPNOTSUPP; | ||
| 850 | break; | ||
| 607 | } | 851 | } |
| 852 | |||
| 608 | return ret; | 853 | return ret; |
| 609 | } | 854 | } |
| 610 | 855 | ||
| 856 | void aie2_msg_init(struct amdxdna_dev_hdl *ndev) | ||
| 857 | { | ||
| 858 | if (AIE2_FEATURE_ON(ndev, AIE2_NPU_COMMAND)) | ||
| 859 | ndev->exec_msg_ops = &npu_exec_message_ops; | ||
| 860 | else | ||
| 861 | ndev->exec_msg_ops = &legacy_exec_message_ops; | ||
| 862 | } | ||
| 863 | |||
| 611 | static inline struct amdxdna_gem_obj * | 864 | static inline struct amdxdna_gem_obj * |
| 612 | aie2_cmdlist_get_cmd_buf(struct amdxdna_sched_job *job) | 865 | aie2_cmdlist_get_cmd_buf(struct amdxdna_sched_job *job) |
| 613 | { | 866 | { |
| @@ -616,29 +869,36 @@ aie2_cmdlist_get_cmd_buf(struct amdxdna_sched_job *job) · 2 unmodified lines | |||
| 616 | return job->hwctx->priv->cmd_buf[idx]; | 869 | return job->hwctx->priv->cmd_buf[idx]; |
| 617 | } | 870 | } |
| 618 | 871 | ||
| 619 | static void | 872 | int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, |
| 620 | aie2_cmdlist_prepare_request(struct cmd_chain_req *req, | 873 | int (*notify_cb)(void *, void __iomem *, size_t)) |
| 621 | struct amdxdna_gem_obj *cmdbuf_abo, u32 size, u32 cnt) | ||
| 622 | { | 874 | { |
| 623 | req->buf_addr = cmdbuf_abo->mem.dev_addr; | 875 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 624 | req->buf_size = size; | 876 | struct amdxdna_dev *xdna = hwctx->client->xdna; |
| 625 | req->count = cnt; | 877 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 626 | drm_clflush_virt_range(cmdbuf_abo->mem.kva, size); | 878 | struct xdna_mailbox_msg msg; |
| 627 | XDNA_DBG(cmdbuf_abo->client->xdna, "Command buf addr 0x%llx size 0x%x count %d", | 879 | union exec_req req; |
| 628 | req->buf_addr, size, cnt); | 880 | int ret; |
| 629 | } | ||
| 630 | 881 | ||
| 631 | static inline u32 | 882 | if (!chann) |
| 632 | aie2_cmd_op_to_msg_op(u32 op) | 883 | return -ENODEV; |
| 633 | { | 884 | |
| 634 | switch (op) { | 885 | ret = aie2_init_exec_req(&req, cmd_abo, &msg.send_size, &msg.opcode); |
| 635 | case ERT_START_CU: | 886 | if (ret) |
| 636 | return MSG_OP_CHAIN_EXEC_BUFFER_CF; | 887 | return ret; |
| 637 | case ERT_START_NPU: | 888 | |
| 638 | return MSG_OP_CHAIN_EXEC_DPU; | 889 | msg.handle = job; |
| 639 | default: | 890 | msg.notify_cb = notify_cb; |
| 640 | return MSG_OP_MAX_OPCODE; | 891 | msg.send_data = (u8 *)&req; |
| 892 | print_hex_dump_debug("cmd: ", DUMP_PREFIX_OFFSET, 16, 4, &req, | ||
| 893 | 0x40, false); | ||
| 894 | |||
| 895 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | ||
| 896 | if (ret) { | ||
| 897 | XDNA_ERR(xdna, "Send message failed"); | ||
| 898 | return ret; | ||
| 641 | } | 899 | } |
| 900 | |||
| 901 | return 0; | ||
| 642 | } | 902 | } |
| 643 | 903 | ||
| 644 | int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, | 904 | int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, |
| @@ -649,12 +909,13 @@ int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, · 4 unmodified lines | |||
| 649 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | 909 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 650 | struct amdxdna_client *client = hwctx->client; | 910 | struct amdxdna_client *client = hwctx->client; |
| 651 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | 911 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 912 | struct amdxdna_dev *xdna = client->xdna; | ||
| 652 | struct amdxdna_cmd_chain *payload; | 913 | struct amdxdna_cmd_chain *payload; |
| 653 | struct xdna_mailbox_msg msg; | 914 | struct xdna_mailbox_msg msg; |
| 654 | struct cmd_chain_req req; | 915 | union exec_chain_req req; |
| 655 | u32 payload_len; | 916 | u32 payload_len; |
| 656 | u32 offset = 0; | 917 | u32 offset = 0; |
| 657 | u32 size; | 918 | size_t size; |
| 658 | int ret; | 919 | int ret; |
| 659 | u32 op; | 920 | u32 op; |
| 660 | u32 i; | 921 | u32 i; |
| @@ -665,41 +926,42 @@ int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, · 4 unmodified lines | |||
| 665 | payload_len < struct_size(payload, data, payload->command_count)) | 926 | payload_len < struct_size(payload, data, payload->command_count)) |
| 666 | return -EINVAL; | 927 | return -EINVAL; |
| 667 | 928 | ||
| 929 | op = ERT_INVALID_CMD; | ||
| 668 | for (i = 0; i < payload->command_count; i++) { | 930 | for (i = 0; i < payload->command_count; i++) { |
| 669 | u32 boh = (u32)(payload->data[i]); | 931 | u32 boh = (u32)(payload->data[i]); |
| 670 | struct amdxdna_gem_obj *abo; | 932 | struct amdxdna_gem_obj *abo; |
| 671 | 933 | ||
| 672 | abo = amdxdna_gem_get_obj(client, boh, AMDXDNA_BO_CMD); | 934 | abo = amdxdna_gem_get_obj(client, boh, AMDXDNA_BO_CMD); |
| 673 | if (!abo) { | 935 | if (!abo) { |
| 674 | XDNA_ERR(client->xdna, "Failed to find cmd BO %d", boh); | 936 | XDNA_ERR(xdna, "Failed to find cmd BO %d", boh); |
| 675 | return -ENOENT; | 937 | return -ENOENT; |
| 676 | } | 938 | } |
| 677 | 939 | ||
| 678 | /* All sub-cmd should have same op, use the first one. */ | 940 | size = cmdbuf_abo->mem.size - offset; |
| 679 | if (i == 0) | 941 | ret = aie2_cmdlist_fill_slot(cmdbuf_abo->mem.kva + offset, |
| 680 | op = amdxdna_cmd_get_op(abo); | 942 | abo, &size, &op); |
| 681 | |||
| 682 | ret = aie2_cmdlist_fill_one_slot(op, cmdbuf_abo, offset, abo, &size); | ||
| 683 | amdxdna_gem_put_obj(abo); | 943 | amdxdna_gem_put_obj(abo); |
| 684 | if (ret) | 944 | if (ret) |
| 685 | return -EINVAL; | 945 | return ret; |
| 686 | 946 | ||
| 687 | offset += size; | 947 | offset += size; |
| 688 | } | 948 | } |
| 949 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); | ||
| 950 | if (msg.opcode == MSG_OP_MAX_OPCODE) | ||
| 951 | return -EOPNOTSUPP; | ||
| 689 | 952 | ||
| 690 | /* The offset is the accumulated total size of the cmd buffer */ | 953 | /* The offset is the accumulated total size of the cmd buffer */ |
| 691 | aie2_cmdlist_prepare_request(&req, cmdbuf_abo, offset, payload->command_count); | 954 | EXEC_MSG_OPS(xdna)->init_chain_req(&req, cmdbuf_abo->mem.dev_addr, |
| 955 | offset, payload->command_count); | ||
| 956 | drm_clflush_virt_range(cmdbuf_abo->mem.kva, offset); | ||
| 692 | 957 | ||
| 693 | msg.opcode = aie2_cmd_op_to_msg_op(op); | ||
| 694 | if (msg.opcode == MSG_OP_MAX_OPCODE) | ||
| 695 | return -EOPNOTSUPP; | ||
| 696 | msg.handle = job; | 958 | msg.handle = job; |
| 697 | msg.notify_cb = notify_cb; | 959 | msg.notify_cb = notify_cb; |
| 698 | msg.send_data = (u8 *)&req; | 960 | msg.send_data = (u8 *)&req; |
| 699 | msg.send_size = sizeof(req); | 961 | msg.send_size = sizeof(req); |
| 700 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | 962 | ret = xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); |
| 701 | if (ret) { | 963 | if (ret) { |
| 702 | XDNA_ERR(hwctx->client->xdna, "Send message failed"); | 964 | XDNA_ERR(xdna, "Send message failed"); |
| 703 | return ret; | 965 | return ret; |
| 704 | } | 966 | } |
| 705 | 967 | ||
| @@ -712,23 +974,27 @@ int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx, · 6 unmodified lines | |||
| 712 | { | 974 | { |
| 713 | struct amdxdna_gem_obj *cmdbuf_abo = aie2_cmdlist_get_cmd_buf(job); | 975 | struct amdxdna_gem_obj *cmdbuf_abo = aie2_cmdlist_get_cmd_buf(job); |
| 714 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | 976 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; |
| 977 | struct amdxdna_dev *xdna = hwctx->client->xdna; | ||
| 715 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; | 978 | struct amdxdna_gem_obj *cmd_abo = job->cmd_bo; |
| 716 | struct xdna_mailbox_msg msg; | 979 | struct xdna_mailbox_msg msg; |
| 717 | struct cmd_chain_req req; | 980 | union exec_chain_req req; |
| 718 | u32 size; | 981 | u32 op = ERT_INVALID_CMD; |
| 982 | size_t size; | ||
| 719 | int ret; | 983 | int ret; |
| 720 | u32 op; | ||
| 721 | 984 | ||
| 722 | op = amdxdna_cmd_get_op(cmd_abo); | 985 | size = cmdbuf_abo->mem.size; |
| 723 | ret = aie2_cmdlist_fill_one_slot(op, cmdbuf_abo, 0, cmd_abo, &size); | 986 | ret = aie2_cmdlist_fill_slot(cmdbuf_abo->mem.kva, cmd_abo, &size, &op); |
| 724 | if (ret) | 987 | if (ret) |
| 725 | return ret; | 988 | return ret; |
| 726 | 989 | ||
| 727 | aie2_cmdlist_prepare_request(&req, cmdbuf_abo, size, 1); | 990 | msg.opcode = EXEC_MSG_OPS(xdna)->get_chain_msg_op(op); |
| 728 | |||
| 729 | msg.opcode = aie2_cmd_op_to_msg_op(op); | ||
| 730 | if (msg.opcode == MSG_OP_MAX_OPCODE) | 991 | if (msg.opcode == MSG_OP_MAX_OPCODE) |
| 731 | return -EOPNOTSUPP; | 992 | return -EOPNOTSUPP; |
| 993 | |||
| 994 | EXEC_MSG_OPS(xdna)->init_chain_req(&req, cmdbuf_abo->mem.dev_addr, | ||
| 995 | size, 1); | ||
| 996 | drm_clflush_virt_range(cmdbuf_abo->mem.kva, size); | ||
| 997 | |||
| 732 | msg.handle = job; | 998 | msg.handle = job; |
| 733 | msg.notify_cb = notify_cb; | 999 | msg.notify_cb = notify_cb; |
| 734 | msg.send_data = (u8 *)&req; | 1000 | msg.send_data = (u8 *)&req; |
| @@ -753,7 +1019,7 @@ int aie2_sync_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, · 18 unmodified lines | |||
| 753 | int ret = 0; | 1019 | int ret = 0; |
| 754 | 1020 | ||
| 755 | req.src_addr = 0; | 1021 | req.src_addr = 0; |
| 756 | req.dst_addr = abo->mem.dev_addr - hwctx->client->dev_heap->mem.dev_addr; | 1022 | req.dst_addr = amdxdna_dev_bo_offset(abo); |
| 757 | req.size = abo->mem.size; | 1023 | req.size = abo->mem.size; |
| 758 | 1024 | ||
| 759 | /* Device to Host */ | 1025 | /* Device to Host */ |
| @@ -777,3 +1043,32 @@ int aie2_sync_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, · 17 unmodified lines | |||
| 777 | 1043 | ||
| 778 | return 0; | 1044 | return 0; |
| 779 | } | 1045 | } |
| 1046 | |||
| 1047 | int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, | ||
| 1048 | int (*notify_cb)(void *, void __iomem *, size_t)) | ||
| 1049 | { | ||
| 1050 | struct mailbox_channel *chann = hwctx->priv->mbox_chann; | ||
| 1051 | struct amdxdna_gem_obj *abo = to_xdna_obj(job->bos[0]); | ||
| 1052 | struct amdxdna_dev *xdna = hwctx->client->xdna; | ||
| 1053 | struct config_debug_bo_req req; | ||
| 1054 | struct xdna_mailbox_msg msg; | ||
| 1055 | |||
| 1056 | if (job->drv_cmd->opcode == ATTACH_DEBUG_BO) | ||
| 1057 | req.config = DEBUG_BO_REGISTER; | ||
| 1058 | else | ||
| 1059 | req.config = DEBUG_BO_UNREGISTER; | ||
| 1060 | |||
| 1061 | req.offset = amdxdna_dev_bo_offset(abo); | ||
| 1062 | req.size = abo->mem.size; | ||
| 1063 | |||
| 1064 | XDNA_DBG(xdna, "offset 0x%llx size 0x%llx config %d", | ||
| 1065 | req.offset, req.size, req.config); | ||
| 1066 | |||
| 1067 | msg.handle = job; | ||
| 1068 | msg.notify_cb = notify_cb; | ||
| 1069 | msg.send_data = (u8 *)&req; | ||
| 1070 | msg.send_size = sizeof(req); | ||
| 1071 | msg.opcode = MSG_OP_CONFIG_DEBUG_BO; | ||
| 1072 | |||
| 1073 | return xdna_mailbox_send_msg(chann, &msg, TX_TIMEOUT); | ||
| 1074 | } | ||
| @@ -9,7 +9,8 @@ · 8 unmodified lines | |||
| 9 | enum aie2_msg_opcode { | 9 | enum aie2_msg_opcode { |
| 10 | MSG_OP_CREATE_CONTEXT = 0x2, | 10 | MSG_OP_CREATE_CONTEXT = 0x2, |
| 11 | MSG_OP_DESTROY_CONTEXT = 0x3, | 11 | MSG_OP_DESTROY_CONTEXT = 0x3, |
| 12 | MSG_OP_SYNC_BO = 0x7, | 12 | MSG_OP_GET_TELEMETRY = 0x4, |
| 13 | MSG_OP_SYNC_BO = 0x7, | ||
| 13 | MSG_OP_EXECUTE_BUFFER_CF = 0xC, | 14 | MSG_OP_EXECUTE_BUFFER_CF = 0xC, |
| 14 | MSG_OP_QUERY_COL_STATUS = 0xD, | 15 | MSG_OP_QUERY_COL_STATUS = 0xD, |
| 15 | MSG_OP_QUERY_AIE_TILE_INFO = 0xE, | 16 | MSG_OP_QUERY_AIE_TILE_INFO = 0xE, |
| @@ -18,6 +19,8 @@ enum aie2_msg_opcode { · 2 unmodified lines | |||
| 18 | MSG_OP_CONFIG_CU = 0x11, | 19 | MSG_OP_CONFIG_CU = 0x11, |
| 19 | MSG_OP_CHAIN_EXEC_BUFFER_CF = 0x12, | 20 | MSG_OP_CHAIN_EXEC_BUFFER_CF = 0x12, |
| 20 | MSG_OP_CHAIN_EXEC_DPU = 0x13, | 21 | MSG_OP_CHAIN_EXEC_DPU = 0x13, |
| 22 | MSG_OP_CONFIG_DEBUG_BO = 0x14, | ||
| 23 | MSG_OP_CHAIN_EXEC_NPU = 0x18, | ||
| 21 | MSG_OP_MAX_XRT_OPCODE, | 24 | MSG_OP_MAX_XRT_OPCODE, |
| 22 | MSG_OP_SUSPEND = 0x101, | 25 | MSG_OP_SUSPEND = 0x101, |
| 23 | MSG_OP_RESUME = 0x102, | 26 | MSG_OP_RESUME = 0x102, |
| @@ -135,6 +138,28 @@ struct destroy_ctx_resp { · 111 unmodified lines | |||
| 135 | enum aie2_msg_status status; | 138 | enum aie2_msg_status status; |
| 136 | } __packed; | 139 | } __packed; |
| 137 | 140 | ||
| 141 | enum telemetry_type { | ||
| 142 | TELEMETRY_TYPE_DISABLED, | ||
| 143 | TELEMETRY_TYPE_HEALTH, | ||
| 144 | TELEMETRY_TYPE_ERROR_INFO, | ||
| 145 | TELEMETRY_TYPE_PROFILING, | ||
| 146 | TELEMETRY_TYPE_DEBUG, | ||
| 147 | MAX_TELEMETRY_TYPE | ||
| 148 | }; | ||
| 149 | |||
| 150 | struct get_telemetry_req { | ||
| 151 | enum telemetry_type type; | ||
| 152 | __u64 buf_addr; | ||
| 153 | __u32 buf_size; | ||
| 154 | } __packed; | ||
| 155 | |||
| 156 | struct get_telemetry_resp { | ||
| 157 | __u32 major; | ||
| 158 | __u32 minor; | ||
| 159 | __u32 size; | ||
| 160 | enum aie2_msg_status status; | ||
| 161 | } __packed; | ||
| 162 | |||
| 138 | struct execute_buffer_req { | 163 | struct execute_buffer_req { |
| 139 | __u32 cu_idx; | 164 | __u32 cu_idx; |
| 140 | __u32 payload[19]; | 165 | __u32 payload[19]; |
| @@ -148,6 +173,18 @@ struct exec_dpu_req { · 7 unmodified lines | |||
| 148 | __u32 payload[35]; | 173 | __u32 payload[35]; |
| 149 | } __packed; | 174 | } __packed; |
| 150 | 175 | ||
| 176 | enum exec_npu_type { | ||
| 177 | EXEC_NPU_TYPE_NON_ELF = 0x1, | ||
| 178 | EXEC_NPU_TYPE_PARTIAL_ELF = 0x2, | ||
| 179 | EXEC_NPU_TYPE_PREEMPT = 0x3, | ||
| 180 | EXEC_NPU_TYPE_ELF = 0x4, | ||
| 181 | }; | ||
| 182 | |||
| 183 | union exec_req { | ||
| 184 | struct execute_buffer_req ebuf; | ||
| 185 | struct exec_dpu_req dpu_req; | ||
| 186 | }; | ||
| 187 | |||
| 151 | struct execute_buffer_resp { | 188 | struct execute_buffer_resp { |
| 152 | enum aie2_msg_status status; | 189 | enum aie2_msg_status status; |
| 153 | } __packed; | 190 | } __packed; |
| @@ -319,9 +356,6 @@ struct async_event_msg_resp { · 165 unmodified lines | |||
| 319 | } __packed; | 356 | } __packed; |
| 320 | 357 | ||
| 321 | #define MAX_CHAIN_CMDBUF_SIZE SZ_4K | 358 | #define MAX_CHAIN_CMDBUF_SIZE SZ_4K |
| 322 | #define slot_has_space(slot, offset, payload_size) \ | ||
| 323 | (MAX_CHAIN_CMDBUF_SIZE >= (offset) + (payload_size) + \ | ||
| 324 | sizeof(typeof(slot))) | ||
| 325 | 359 | ||
| 326 | struct cmd_chain_slot_execbuf_cf { | 360 | struct cmd_chain_slot_execbuf_cf { |
| 327 | __u32 cu_idx; | 361 | __u32 cu_idx; |
| @@ -339,12 +373,41 @@ struct cmd_chain_slot_dpu { · 11 unmodified lines | |||
| 339 | __u32 args[] __counted_by(arg_cnt); | 373 | __u32 args[] __counted_by(arg_cnt); |
| 340 | }; | 374 | }; |
| 341 | 375 | ||
| 376 | #define MAX_NPU_ARGS_SIZE (26 * sizeof(__u32)) | ||
| 377 | #define AIE2_EXEC_BUFFER_KERNEL_OP_TXN 3 | ||
| 378 | struct cmd_chain_slot_npu { | ||
| 379 | enum exec_npu_type type; | ||
| 380 | u64 inst_buf_addr; | ||
| 381 | u64 save_buf_addr; | ||
| 382 | u64 restore_buf_addr; | ||
| 383 | u32 inst_size; | ||
| 384 | u32 save_size; | ||
| 385 | u32 restore_size; | ||
| 386 | u32 inst_prop_cnt; | ||
| 387 | u32 cu_idx; | ||
| 388 | u32 arg_cnt; | ||
| 389 | u32 args[] __counted_by(arg_cnt); | ||
| 390 | } __packed; | ||
| 391 | |||
| 342 | struct cmd_chain_req { | 392 | struct cmd_chain_req { |
| 343 | __u64 buf_addr; | 393 | __u64 buf_addr; |
| 344 | __u32 buf_size; | 394 | __u32 buf_size; |
| 345 | __u32 count; | 395 | __u32 count; |
| 346 | } __packed; | 396 | } __packed; |
| 347 | 397 | ||
| 398 | struct cmd_chain_npu_req { | ||
| 399 | u32 flags; | ||
| 400 | u32 reserved; | ||
| 401 | u64 buf_addr; | ||
| 402 | u32 buf_size; | ||
| 403 | u32 count; | ||
| 404 | } __packed; | ||
| 405 | |||
| 406 | union exec_chain_req { | ||
| 407 | struct cmd_chain_npu_req npu_req; | ||
| 408 | struct cmd_chain_req req; | ||
| 409 | }; | ||
| 410 | |||
| 348 | struct cmd_chain_resp { | 411 | struct cmd_chain_resp { |
| 349 | enum aie2_msg_status status; | 412 | enum aie2_msg_status status; |
| 350 | __u32 fail_cmd_idx; | 413 | __u32 fail_cmd_idx; |
| @@ -365,4 +428,21 @@ struct sync_bo_req { · 14 unmodified lines | |||
| 365 | struct sync_bo_resp { | 428 | struct sync_bo_resp { |
| 366 | enum aie2_msg_status status; | 429 | enum aie2_msg_status status; |
| 367 | } __packed; | 430 | } __packed; |
| 431 | |||
| 432 | #define DEBUG_BO_UNREGISTER 0 | ||
| 433 | #define DEBUG_BO_REGISTER 1 | ||
| 434 | struct config_debug_bo_req { | ||
| 435 | __u64 offset; | ||
| 436 | __u64 size; | ||
| 437 | /* | ||
| 438 | * config operations. | ||
| 439 | * DEBUG_BO_REGISTER: Register debug buffer | ||
| 440 | * DEBUG_BO_UNREGISTER: Unregister debug buffer | ||
| 441 | */ | ||
| 442 | __u32 config; | ||
| 443 | } __packed; | ||
| 444 | |||
| 445 | struct config_debug_bo_resp { | ||
| 446 | enum aie2_msg_status status; | ||
| 447 | } __packed; | ||
| 368 | #endif /* _AIE2_MSG_PRIV_H_ */ | 448 | #endif /* _AIE2_MSG_PRIV_H_ */ |
| @@ -25,6 +25,7 @@ · 24 unmodified lines | |||
| 25 | #include "amdxdna_gem.h" | 25 | #include "amdxdna_gem.h" |
| 26 | #include "amdxdna_mailbox.h" | 26 | #include "amdxdna_mailbox.h" |
| 27 | #include "amdxdna_pci_drv.h" | 27 | #include "amdxdna_pci_drv.h" |
| 28 | #include "amdxdna_pm.h" | ||
| 28 | 29 | ||
| 29 | static int aie2_max_col = XRS_MAX_COL; | 30 | static int aie2_max_col = XRS_MAX_COL; |
| 30 | module_param(aie2_max_col, uint, 0600); | 31 | module_param(aie2_max_col, uint, 0600); |
| @@ -54,6 +55,7 @@ struct mgmt_mbox_chann_info { · 23 unmodified lines | |||
| 54 | 55 | ||
| 55 | static int aie2_check_protocol(struct amdxdna_dev_hdl *ndev, u32 fw_major, u32 fw_minor) | 56 | static int aie2_check_protocol(struct amdxdna_dev_hdl *ndev, u32 fw_major, u32 fw_minor) |
| 56 | { | 57 | { |
| 58 | const struct aie2_fw_feature_tbl *feature; | ||
| 57 | struct amdxdna_dev *xdna = ndev->xdna; | 59 | struct amdxdna_dev *xdna = ndev->xdna; |
| 58 | 60 | ||
| 59 | /* | 61 | /* |
| @@ -77,6 +79,17 @@ static int aie2_check_protocol(struct amdxdna_dev_hdl *ndev, u32 fw_major, u32 f · 17 unmodified lines | |||
| 77 | XDNA_ERR(xdna, "Firmware minor version smaller than supported"); | 79 | XDNA_ERR(xdna, "Firmware minor version smaller than supported"); |
| 78 | return -EINVAL; | 80 | return -EINVAL; |
| 79 | } | 81 | } |
| 82 | |||
| 83 | for (feature = ndev->priv->fw_feature_tbl; feature && feature->min_minor; | ||
| 84 | feature++) { | ||
| 85 | if (fw_minor < feature->min_minor) | ||
| 86 | continue; | ||
| 87 | if (feature->max_minor > 0 && fw_minor > feature->max_minor) | ||
| 88 | continue; | ||
| 89 | |||
| 90 | set_bit(feature->feature, &ndev->feature_mask); | ||
| 91 | } | ||
| 92 | |||
| 80 | return 0; | 93 | return 0; |
| 81 | } | 94 | } |
| 82 | 95 | ||
| @@ -170,6 +183,10 @@ int aie2_runtime_cfg(struct amdxdna_dev_hdl *ndev, · 87 unmodified lines | |||
| 170 | if (cfg->category != category) | 183 | if (cfg->category != category) |
| 171 | continue; | 184 | continue; |
| 172 | 185 | ||
| 186 | if (cfg->feature_mask && | ||
| 187 | bitmap_subset(&cfg->feature_mask, &ndev->feature_mask, AIE2_FEATURE_MAX)) | ||
| 188 | continue; | ||
| 189 | |||
| 173 | value = val ? *val : cfg->value; | 190 | value = val ? *val : cfg->value; |
| 174 | ret = aie2_set_runtime_cfg(ndev, cfg->type, value); | 191 | ret = aie2_set_runtime_cfg(ndev, cfg->type, value); |
| 175 | if (ret) { | 192 | if (ret) { |
| @@ -223,15 +240,6 @@ static int aie2_mgmt_fw_init(struct amdxdna_dev_hdl *ndev) · 47 unmodified lines | |||
| 223 | return ret; | 240 | return ret; |
| 224 | } | 241 | } |
| 225 | 242 | ||
| 226 | if (!ndev->async_events) | ||
| 227 | return 0; | ||
| 228 | |||
| 229 | ret = aie2_error_async_events_send(ndev); | ||
| 230 | if (ret) { | ||
| 231 | XDNA_ERR(ndev->xdna, "Send async events failed"); | ||
| 232 | return ret; | ||
| 233 | } | ||
| 234 | |||
| 235 | return 0; | 243 | return 0; |
| 236 | } | 244 | } |
| 237 | 245 | ||
| @@ -257,6 +265,8 @@ static int aie2_mgmt_fw_query(struct amdxdna_dev_hdl *ndev) · 19 unmodified lines | |||
| 257 | return ret; | 265 | return ret; |
| 258 | } | 266 | } |
| 259 | 267 | ||
| 268 | ndev->total_col = min(aie2_max_col, ndev->metadata.cols); | ||
| 269 | |||
| 260 | return 0; | 270 | return 0; |
| 261 | } | 271 | } |
| 262 | 272 | ||
| @@ -338,6 +348,7 @@ static void aie2_hw_stop(struct amdxdna_dev *xdna) · 75 unmodified lines | |||
| 338 | ndev->mbox = NULL; | 348 | ndev->mbox = NULL; |
| 339 | aie2_psp_stop(ndev->psp_hdl); | 349 | aie2_psp_stop(ndev->psp_hdl); |
| 340 | aie2_smu_fini(ndev); | 350 | aie2_smu_fini(ndev); |
| 351 | aie2_error_async_events_free(ndev); | ||
| 341 | pci_disable_device(pdev); | 352 | pci_disable_device(pdev); |
| 342 | 353 | ||
| 343 | ndev->dev_status = AIE2_DEV_INIT; | 354 | ndev->dev_status = AIE2_DEV_INIT; |
| @@ -424,6 +435,18 @@ static int aie2_hw_start(struct amdxdna_dev *xdna) · 80 unmodified lines | |||
| 424 | goto destroy_mgmt_chann; | 435 | goto destroy_mgmt_chann; |
| 425 | } | 436 | } |
| 426 | 437 | ||
| 438 | ret = aie2_mgmt_fw_query(ndev); | ||
| 439 | if (ret) { | ||
| 440 | XDNA_ERR(xdna, "failed to query fw, ret %d", ret); | ||
| 441 | goto destroy_mgmt_chann; | ||
| 442 | } | ||
| 443 | |||
| 444 | ret = aie2_error_async_events_alloc(ndev); | ||
| 445 | if (ret) { | ||
| 446 | XDNA_ERR(xdna, "Allocate async events failed, ret %d", ret); | ||
| 447 | goto destroy_mgmt_chann; | ||
| 448 | } | ||
| 449 | |||
| 427 | ndev->dev_status = AIE2_DEV_START; | 450 | ndev->dev_status = AIE2_DEV_START; |
| 428 | 451 | ||
| 429 | return 0; | 452 | return 0; |
| @@ -459,7 +482,6 @@ static int aie2_hw_resume(struct amdxdna_dev *xdna) · 29 unmodified lines | |||
| 459 | struct amdxdna_client *client; | 482 | struct amdxdna_client *client; |
| 460 | int ret; | 483 | int ret; |
| 461 | 484 | ||
| 462 | guard(mutex)(&xdna->dev_lock); | ||
| 463 | ret = aie2_hw_start(xdna); | 485 | ret = aie2_hw_start(xdna); |
| 464 | if (ret) { | 486 | if (ret) { |
| 465 | XDNA_ERR(xdna, "Start hardware failed, %d", ret); | 487 | XDNA_ERR(xdna, "Start hardware failed, %d", ret); |
| @@ -565,13 +587,6 @@ static int aie2_init(struct amdxdna_dev *xdna) · 99 unmodified lines | |||
| 565 | goto release_fw; | 587 | goto release_fw; |
| 566 | } | 588 | } |
| 567 | 589 | ||
| 568 | ret = aie2_mgmt_fw_query(ndev); | ||
| 569 | if (ret) { | ||
| 570 | XDNA_ERR(xdna, "Query firmware failed, ret %d", ret); | ||
| 571 | goto stop_hw; | ||
| 572 | } | ||
| 573 | ndev->total_col = min(aie2_max_col, ndev->metadata.cols); | ||
| 574 | |||
| 575 | xrs_cfg.clk_list.num_levels = ndev->max_dpm_level + 1; | 590 | xrs_cfg.clk_list.num_levels = ndev->max_dpm_level + 1; |
| 576 | for (i = 0; i < xrs_cfg.clk_list.num_levels; i++) | 591 | for (i = 0; i < xrs_cfg.clk_list.num_levels; i++) |
| 577 | xrs_cfg.clk_list.cu_clk_list[i] = ndev->priv->dpm_clk_tbl[i].hclk; | 592 | xrs_cfg.clk_list.cu_clk_list[i] = ndev->priv->dpm_clk_tbl[i].hclk; |
| @@ -587,30 +602,11 @@ static int aie2_init(struct amdxdna_dev *xdna) · 9 unmodified lines | |||
| 587 | goto stop_hw; | 602 | goto stop_hw; |
| 588 | } | 603 | } |
| 589 | 604 | ||
| 590 | ret = aie2_error_async_events_alloc(ndev); | ||
| 591 | if (ret) { | ||
| 592 | XDNA_ERR(xdna, "Allocate async events failed, ret %d", ret); | ||
| 593 | goto stop_hw; | ||
| 594 | } | ||
| 595 | |||
| 596 | ret = aie2_error_async_events_send(ndev); | ||
| 597 | if (ret) { | ||
| 598 | XDNA_ERR(xdna, "Send async events failed, ret %d", ret); | ||
| 599 | goto async_event_free; | ||
| 600 | } | ||
| 601 | |||
| 602 | /* Issue a command to make sure firmware handled async events */ | ||
| 603 | ret = aie2_query_firmware_version(ndev, &ndev->xdna->fw_ver); | ||
| 604 | if (ret) { | ||
| 605 | XDNA_ERR(xdna, "Re-query firmware version failed"); | ||
| 606 | goto async_event_free; | ||
| 607 | } | ||
| 608 | |||
| 609 | release_firmware(fw); | 605 | release_firmware(fw); |
| 606 | aie2_msg_init(ndev); | ||
| 607 | amdxdna_pm_init(xdna); | ||
| 610 | return 0; | 608 | return 0; |
| 611 | 609 | ||
| 612 | async_event_free: | ||
| 613 | aie2_error_async_events_free(ndev); | ||
| 614 | stop_hw: | 610 | stop_hw: |
| 615 | aie2_hw_stop(xdna); | 611 | aie2_hw_stop(xdna); |
| 616 | release_fw: | 612 | release_fw: |
| @@ -621,10 +617,8 @@ release_fw: · 4 unmodified lines | |||
| 621 | 617 | ||
| 622 | static void aie2_fini(struct amdxdna_dev *xdna) | 618 | static void aie2_fini(struct amdxdna_dev *xdna) |
| 623 | { | 619 | { |
| 624 | struct amdxdna_dev_hdl *ndev = xdna->dev_handle; | 620 | amdxdna_pm_fini(xdna); |
| 625 | |||
| 626 | aie2_hw_stop(xdna); | 621 | aie2_hw_stop(xdna); |
| 627 | aie2_error_async_events_free(ndev); | ||
| 628 | } | 622 | } |
| 629 | 623 | ||
| 630 | static int aie2_get_aie_status(struct amdxdna_client *client, | 624 | static int aie2_get_aie_status(struct amdxdna_client *client, |
| @@ -845,7 +839,120 @@ static int aie2_get_hwctx_status(struct amdxdna_client *client, · 214 unmodified lines | |||
| 845 | } | 839 | } |
| 846 | 840 | ||
| 847 | args->buffer_size -= (u32)(array_args.buffer - args->buffer); | 841 | args->buffer_size -= (u32)(array_args.buffer - args->buffer); |
| 848 | return ret; | 842 | return 0; |
| 843 | } | ||
| 844 | |||
| 845 | static int aie2_query_resource_info(struct amdxdna_client *client, | ||
| 846 | struct amdxdna_drm_get_info *args) | ||
| 847 | { | ||
| 848 | struct amdxdna_drm_get_resource_info res_info; | ||
| 849 | const struct amdxdna_dev_priv *priv; | ||
| 850 | struct amdxdna_dev_hdl *ndev; | ||
| 851 | struct amdxdna_dev *xdna; | ||
| 852 | |||
| 853 | xdna = client->xdna; | ||
| 854 | ndev = xdna->dev_handle; | ||
| 855 | priv = ndev->priv; | ||
| 856 | |||
| 857 | res_info.npu_clk_max = priv->dpm_clk_tbl[ndev->max_dpm_level].hclk; | ||
| 858 | res_info.npu_tops_max = ndev->max_tops; | ||
| 859 | res_info.npu_task_max = priv->hwctx_limit; | ||
| 860 | res_info.npu_tops_curr = ndev->curr_tops; | ||
| 861 | res_info.npu_task_curr = ndev->hwctx_num; | ||
| 862 | |||
| 863 | if (copy_to_user(u64_to_user_ptr(args->buffer), &res_info, sizeof(res_info))) | ||
| 864 | return -EFAULT; | ||
| 865 | |||
| 866 | return 0; | ||
| 867 | } | ||
| 868 | |||
| 869 | static int aie2_fill_hwctx_map(struct amdxdna_hwctx *hwctx, void *arg) | ||
| 870 | { | ||
| 871 | struct amdxdna_dev *xdna = hwctx->client->xdna; | ||
| 872 | u32 *map = arg; | ||
| 873 | |||
| 874 | if (hwctx->fw_ctx_id >= xdna->dev_handle->priv->hwctx_limit) { | ||
| 875 | XDNA_ERR(xdna, "Invalid fw ctx id %d/%d ", hwctx->fw_ctx_id, | ||
| 876 | xdna->dev_handle->priv->hwctx_limit); | ||
| 877 | return -EINVAL; | ||
| 878 | } | ||
| 879 | |||
| 880 | map[hwctx->fw_ctx_id] = hwctx->id; | ||
| 881 | return 0; | ||
| 882 | } | ||
| 883 | |||
| 884 | static int aie2_get_telemetry(struct amdxdna_client *client, | ||
| 885 | struct amdxdna_drm_get_info *args) | ||
| 886 | { | ||
| 887 | struct amdxdna_drm_query_telemetry_header *header __free(kfree) = NULL; | ||
| 888 | u32 telemetry_data_sz, header_sz, elem_num; | ||
| 889 | struct amdxdna_dev *xdna = client->xdna; | ||
| 890 | struct amdxdna_client *tmp_client; | ||
| 891 | int ret; | ||
| 892 | |||
| 893 | elem_num = xdna->dev_handle->priv->hwctx_limit; | ||
| 894 | header_sz = struct_size(header, map, elem_num); | ||
| 895 | if (args->buffer_size <= header_sz) { | ||
| 896 | XDNA_ERR(xdna, "Invalid buffer size"); | ||
| 897 | return -EINVAL; | ||
| 898 | } | ||
| 899 | |||
| 900 | telemetry_data_sz = args->buffer_size - header_sz; | ||
| 901 | if (telemetry_data_sz > SZ_4M) { | ||
| 902 | XDNA_ERR(xdna, "Buffer size is too big, %d", telemetry_data_sz); | ||
| 903 | return -EINVAL; | ||
| 904 | } | ||
| 905 | |||
| 906 | header = kzalloc(header_sz, GFP_KERNEL); | ||
| 907 | if (!header) | ||
| 908 | return -ENOMEM; | ||
| 909 | |||
| 910 | if (copy_from_user(header, u64_to_user_ptr(args->buffer), sizeof(*header))) { | ||
| 911 | XDNA_ERR(xdna, "Failed to copy telemetry header from user"); | ||
| 912 | return -EFAULT; | ||
| 913 | } | ||
| 914 | |||
| 915 | header->map_num_elements = elem_num; | ||
| 916 | list_for_each_entry(tmp_client, &xdna->client_list, node) { | ||
| 917 | ret = amdxdna_hwctx_walk(tmp_client, &header->map, | ||
| 918 | aie2_fill_hwctx_map); | ||
| 919 | if (ret) | ||
| 920 | return ret; | ||
| 921 | } | ||
| 922 | |||
| 923 | ret = aie2_query_telemetry(xdna->dev_handle, | ||
| 924 | u64_to_user_ptr(args->buffer + header_sz), | ||
| 925 | telemetry_data_sz, header); | ||
| 926 | if (ret) { | ||
| 927 | XDNA_ERR(xdna, "Query telemetry failed ret %d", ret); | ||
| 928 | return ret; | ||
| 929 | } | ||
| 930 | |||
| 931 | if (copy_to_user(u64_to_user_ptr(args->buffer), header, header_sz)) { | ||
| 932 | XDNA_ERR(xdna, "Copy header failed"); | ||
| 933 | return -EFAULT; | ||
| 934 | } | ||
| 935 | |||
| 936 | return 0; | ||
| 937 | } | ||
| 938 | |||
| 939 | static int aie2_get_preempt_state(struct amdxdna_client *client, | ||
| 940 | struct amdxdna_drm_get_info *args) | ||
| 941 | { | ||
| 942 | struct amdxdna_drm_attribute_state state = {}; | ||
| 943 | struct amdxdna_dev *xdna = client->xdna; | ||
| 944 | struct amdxdna_dev_hdl *ndev; | ||
| 945 | |||
| 946 | ndev = xdna->dev_handle; | ||
| 947 | if (args->param == DRM_AMDXDNA_GET_FORCE_PREEMPT_STATE) | ||
| 948 | state.state = ndev->force_preempt_enabled; | ||
| 949 | else if (args->param == DRM_AMDXDNA_GET_FRAME_BOUNDARY_PREEMPT_STATE) | ||
| 950 | state.state = ndev->frame_boundary_preempt; | ||
| 951 | |||
| 952 | if (copy_to_user(u64_to_user_ptr(args->buffer), &state, sizeof(state))) | ||
| 953 | return -EFAULT; | ||
| 954 | |||
| 955 | return 0; | ||
| 849 | } | 956 | } |
| 850 | 957 | ||
| 851 | static int aie2_get_info(struct amdxdna_client *client, struct amdxdna_drm_get_info *args) | 958 | static int aie2_get_info(struct amdxdna_client *client, struct amdxdna_drm_get_info *args) |
| @@ -856,6 +963,10 @@ static int aie2_get_info(struct amdxdna_client *client, struct amdxdna_drm_get_i · 4 unmodified lines | |||
| 856 | if (!drm_dev_enter(&xdna->ddev, &idx)) | 963 | if (!drm_dev_enter(&xdna->ddev, &idx)) |
| 857 | return -ENODEV; | 964 | return -ENODEV; |
| 858 | 965 | ||
| 966 | ret = amdxdna_pm_resume_get(xdna); | ||
| 967 | if (ret) | ||
| 968 | goto dev_exit; | ||
| 969 | |||
| 859 | switch (args->param) { | 970 | switch (args->param) { |
| 860 | case DRM_AMDXDNA_QUERY_AIE_STATUS: | 971 | case DRM_AMDXDNA_QUERY_AIE_STATUS: |
| 861 | ret = aie2_get_aie_status(client, args); | 972 | ret = aie2_get_aie_status(client, args); |
| @@ -878,12 +989,25 @@ static int aie2_get_info(struct amdxdna_client *client, struct amdxdna_drm_get_i · 16 unmodified lines | |||
| 878 | case DRM_AMDXDNA_GET_POWER_MODE: | 989 | case DRM_AMDXDNA_GET_POWER_MODE: |
| 879 | ret = aie2_get_power_mode(client, args); | 990 | ret = aie2_get_power_mode(client, args); |
| 880 | break; | 991 | break; |
| 992 | case DRM_AMDXDNA_QUERY_TELEMETRY: | ||
| 993 | ret = aie2_get_telemetry(client, args); | ||
| 994 | break; | ||
| 995 | case DRM_AMDXDNA_QUERY_RESOURCE_INFO: | ||
| 996 | ret = aie2_query_resource_info(client, args); | ||
| 997 | break; | ||
| 998 | case DRM_AMDXDNA_GET_FORCE_PREEMPT_STATE: | ||
| 999 | case DRM_AMDXDNA_GET_FRAME_BOUNDARY_PREEMPT_STATE: | ||
| 1000 | ret = aie2_get_preempt_state(client, args); | ||
| 1001 | break; | ||
| 881 | default: | 1002 | default: |
| 882 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); | 1003 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); |
| 883 | ret = -EOPNOTSUPP; | 1004 | ret = -EOPNOTSUPP; |
| 884 | } | 1005 | } |
| 1006 | |||
| 1007 | amdxdna_pm_suspend_put(xdna); | ||
| 885 | XDNA_DBG(xdna, "Got param %d", args->param); | 1008 | XDNA_DBG(xdna, "Got param %d", args->param); |
| 886 | 1009 | ||
| 1010 | dev_exit: | ||
| 887 | drm_dev_exit(idx); | 1011 | drm_dev_exit(idx); |
| 888 | return ret; | 1012 | return ret; |
| 889 | } | 1013 | } |
| @@ -898,6 +1022,12 @@ static int aie2_query_ctx_status_array(struct amdxdna_client *client, · 8 unmodified lines | |||
| 898 | 1022 | ||
| 899 | drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); | 1023 | drm_WARN_ON(&xdna->ddev, !mutex_is_locked(&xdna->dev_lock)); |
| 900 | 1024 | ||
| 1025 | if (args->element_size > SZ_4K || args->num_element > SZ_1K) { | ||
| 1026 | XDNA_DBG(xdna, "Invalid element size %d or number of element %d", | ||
| 1027 | args->element_size, args->num_element); | ||
| 1028 | return -EINVAL; | ||
| 1029 | } | ||
| 1030 | |||
| 901 | array_args.element_size = min(args->element_size, | 1031 | array_args.element_size = min(args->element_size, |
| 902 | sizeof(struct amdxdna_drm_hwctx_entry)); | 1032 | sizeof(struct amdxdna_drm_hwctx_entry)); |
| 903 | array_args.buffer = args->buffer; | 1033 | array_args.buffer = args->buffer; |
| @@ -914,7 +1044,7 @@ static int aie2_query_ctx_status_array(struct amdxdna_client *client, · 10 unmodified lines | |||
| 914 | args->num_element = (u32)((array_args.buffer - args->buffer) / | 1044 | args->num_element = (u32)((array_args.buffer - args->buffer) / |
| 915 | args->element_size); | 1045 | args->element_size); |
| 916 | 1046 | ||
| 917 | return ret; | 1047 | return 0; |
| 918 | } | 1048 | } |
| 919 | 1049 | ||
| 920 | static int aie2_get_array(struct amdxdna_client *client, | 1050 | static int aie2_get_array(struct amdxdna_client *client, |
| @@ -926,16 +1056,26 @@ static int aie2_get_array(struct amdxdna_client *client, · 5 unmodified lines | |||
| 926 | if (!drm_dev_enter(&xdna->ddev, &idx)) | 1056 | if (!drm_dev_enter(&xdna->ddev, &idx)) |
| 927 | return -ENODEV; | 1057 | return -ENODEV; |
| 928 | 1058 | ||
| 1059 | ret = amdxdna_pm_resume_get(xdna); | ||
| 1060 | if (ret) | ||
| 1061 | goto dev_exit; | ||
| 1062 | |||
| 929 | switch (args->param) { | 1063 | switch (args->param) { |
| 930 | case DRM_AMDXDNA_HW_CONTEXT_ALL: | 1064 | case DRM_AMDXDNA_HW_CONTEXT_ALL: |
| 931 | ret = aie2_query_ctx_status_array(client, args); | 1065 | ret = aie2_query_ctx_status_array(client, args); |
| 932 | break; | 1066 | break; |
| 1067 | case DRM_AMDXDNA_HW_LAST_ASYNC_ERR: | ||
| 1068 | ret = aie2_get_array_async_error(xdna->dev_handle, args); | ||
| 1069 | break; | ||
| 933 | default: | 1070 | default: |
| 934 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); | 1071 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); |
| 935 | ret = -EOPNOTSUPP; | 1072 | ret = -EOPNOTSUPP; |
| 936 | } | 1073 | } |
| 1074 | |||
| 1075 | amdxdna_pm_suspend_put(xdna); | ||
| 937 | XDNA_DBG(xdna, "Got param %d", args->param); | 1076 | XDNA_DBG(xdna, "Got param %d", args->param); |
| 938 | 1077 | ||
| 1078 | dev_exit: | ||
| 939 | drm_dev_exit(idx); | 1079 | drm_dev_exit(idx); |
| 940 | return ret; | 1080 | return ret; |
| 941 | } | 1081 | } |
| @@ -965,6 +1105,38 @@ static int aie2_set_power_mode(struct amdxdna_client *client, · 23 unmodified lines | |||
| 965 | return aie2_pm_set_mode(xdna->dev_handle, power_mode); | 1105 | return aie2_pm_set_mode(xdna->dev_handle, power_mode); |
| 966 | } | 1106 | } |
| 967 | 1107 | ||
| 1108 | static int aie2_set_preempt_state(struct amdxdna_client *client, | ||
| 1109 | struct amdxdna_drm_set_state *args) | ||
| 1110 | { | ||
| 1111 | struct amdxdna_dev_hdl *ndev = client->xdna->dev_handle; | ||
| 1112 | struct amdxdna_drm_attribute_state state; | ||
| 1113 | u32 val; | ||
| 1114 | int ret; | ||
| 1115 | |||
| 1116 | if (copy_from_user(&state, u64_to_user_ptr(args->buffer), sizeof(state))) | ||
| 1117 | return -EFAULT; | ||
| 1118 | |||
| 1119 | if (state.state > 1) | ||
| 1120 | return -EINVAL; | ||
| 1121 | |||
| 1122 | if (XDNA_MBZ_DBG(client->xdna, state.pad, sizeof(state.pad))) | ||
| 1123 | return -EINVAL; | ||
| 1124 | |||
| 1125 | if (args->param == DRM_AMDXDNA_SET_FORCE_PREEMPT) { | ||
| 1126 | ndev->force_preempt_enabled = state.state; | ||
| 1127 | } else if (args->param == DRM_AMDXDNA_SET_FRAME_BOUNDARY_PREEMPT) { | ||
| 1128 | val = state.state; | ||
| 1129 | ret = aie2_runtime_cfg(ndev, AIE2_RT_CFG_FRAME_BOUNDARY_PREEMPT, | ||
| 1130 | &val); | ||
| 1131 | if (ret) | ||
| 1132 | return ret; | ||
| 1133 | |||
| 1134 | ndev->frame_boundary_preempt = state.state; | ||
| 1135 | } | ||
| 1136 | |||
| 1137 | return 0; | ||
| 1138 | } | ||
| 1139 | |||
| 968 | static int aie2_set_state(struct amdxdna_client *client, | 1140 | static int aie2_set_state(struct amdxdna_client *client, |
| 969 | struct amdxdna_drm_set_state *args) | 1141 | struct amdxdna_drm_set_state *args) |
| 970 | { | 1142 | { |
| @@ -974,16 +1146,26 @@ static int aie2_set_state(struct amdxdna_client *client, · 3 unmodified lines | |||
| 974 | if (!drm_dev_enter(&xdna->ddev, &idx)) | 1146 | if (!drm_dev_enter(&xdna->ddev, &idx)) |
| 975 | return -ENODEV; | 1147 | return -ENODEV; |
| 976 | 1148 | ||
| 1149 | ret = amdxdna_pm_resume_get(xdna); | ||
| 1150 | if (ret) | ||
| 1151 | goto dev_exit; | ||
| 1152 | |||
| 977 | switch (args->param) { | 1153 | switch (args->param) { |
| 978 | case DRM_AMDXDNA_SET_POWER_MODE: | 1154 | case DRM_AMDXDNA_SET_POWER_MODE: |
| 979 | ret = aie2_set_power_mode(client, args); | 1155 | ret = aie2_set_power_mode(client, args); |
| 980 | break; | 1156 | break; |
| 1157 | case DRM_AMDXDNA_SET_FORCE_PREEMPT: | ||
| 1158 | case DRM_AMDXDNA_SET_FRAME_BOUNDARY_PREEMPT: | ||
| 1159 | ret = aie2_set_preempt_state(client, args); | ||
| 1160 | break; | ||
| 981 | default: | 1161 | default: |
| 982 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); | 1162 | XDNA_ERR(xdna, "Not supported request parameter %u", args->param); |
| 983 | ret = -EOPNOTSUPP; | 1163 | ret = -EOPNOTSUPP; |
| 984 | break; | 1164 | break; |
| 985 | } | 1165 | } |
| 986 | 1166 | ||
| 1167 | amdxdna_pm_suspend_put(xdna); | ||
| 1168 | dev_exit: | ||
| 987 | drm_dev_exit(idx); | 1169 | drm_dev_exit(idx); |
| 988 | return ret; | 1170 | return ret; |
| 989 | } | 1171 | } |
| @@ -998,6 +1180,7 @@ const struct amdxdna_dev_ops aie2_ops = { · 8 unmodified lines | |||
| 998 | .hwctx_init = aie2_hwctx_init, | 1180 | .hwctx_init = aie2_hwctx_init, |
| 999 | .hwctx_fini = aie2_hwctx_fini, | 1181 | .hwctx_fini = aie2_hwctx_fini, |
| 1000 | .hwctx_config = aie2_hwctx_config, | 1182 | .hwctx_config = aie2_hwctx_config, |
| 1183 | .hwctx_sync_debug_bo = aie2_hwctx_sync_debug_bo, | ||
| 1001 | .cmd_submit = aie2_cmd_submit, | 1184 | .cmd_submit = aie2_cmd_submit, |
| 1002 | .hmm_invalidate = aie2_hmm_invalidate, | 1185 | .hmm_invalidate = aie2_hmm_invalidate, |
| 1003 | .get_array = aie2_get_array, | 1186 | .get_array = aie2_get_array, |
| @@ -110,12 +110,15 @@ struct aie_metadata { · 109 unmodified lines | |||
| 110 | enum rt_config_category { | 110 | enum rt_config_category { |
| 111 | AIE2_RT_CFG_INIT, | 111 | AIE2_RT_CFG_INIT, |
| 112 | AIE2_RT_CFG_CLK_GATING, | 112 | AIE2_RT_CFG_CLK_GATING, |
| 113 | AIE2_RT_CFG_FORCE_PREEMPT, | ||
| 114 | AIE2_RT_CFG_FRAME_BOUNDARY_PREEMPT, | ||
| 113 | }; | 115 | }; |
| 114 | 116 | ||
| 115 | struct rt_config { | 117 | struct rt_config { |
| 116 | u32 type; | 118 | u32 type; |
| 117 | u32 value; | 119 | u32 value; |
| 118 | u32 category; | 120 | u32 category; |
| 121 | unsigned long feature_mask; | ||
| 119 | }; | 122 | }; |
| 120 | 123 | ||
| 121 | struct dpm_clk_freq { | 124 | struct dpm_clk_freq { |
| @@ -156,6 +159,19 @@ enum aie2_dev_status { · 34 unmodified lines | |||
| 156 | AIE2_DEV_START, | 159 | AIE2_DEV_START, |
| 157 | }; | 160 | }; |
| 158 | 161 | ||
| 162 | struct aie2_exec_msg_ops { | ||
| 163 | int (*init_cu_req)(struct amdxdna_gem_obj *cmd_bo, void *req, | ||
| 164 | size_t *size, u32 *msg_op); | ||
| 165 | int (*init_dpu_req)(struct amdxdna_gem_obj *cmd_bo, void *req, | ||
| 166 | size_t *size, u32 *msg_op); | ||
| 167 | void (*init_chain_req)(void *req, u64 slot_addr, size_t size, u32 cmd_cnt); | ||
| 168 | int (*fill_cf_slot)(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size); | ||
| 169 | int (*fill_dpu_slot)(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size); | ||
| 170 | int (*fill_preempt_slot)(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size); | ||
| 171 | int (*fill_elf_slot)(struct amdxdna_gem_obj *cmd_bo, void *slot, size_t *size); | ||
| 172 | u32 (*get_chain_msg_op)(u32 cmd_op); | ||
| 173 | }; | ||
| 174 | |||
| 159 | struct amdxdna_dev_hdl { | 175 | struct amdxdna_dev_hdl { |
| 160 | struct amdxdna_dev *xdna; | 176 | struct amdxdna_dev *xdna; |
| 161 | const struct amdxdna_dev_priv *priv; | 177 | const struct amdxdna_dev_priv *priv; |
| @@ -173,6 +189,8 @@ struct amdxdna_dev_hdl { · 11 unmodified lines | |||
| 173 | u32 total_col; | 189 | u32 total_col; |
| 174 | struct aie_version version; | 190 | struct aie_version version; |
| 175 | struct aie_metadata metadata; | 191 | struct aie_metadata metadata; |
| 192 | unsigned long feature_mask; | ||
| 193 | struct aie2_exec_msg_ops *exec_msg_ops; | ||
| 176 | 194 | ||
| 177 | /* power management and clock*/ | 195 | /* power management and clock*/ |
| 178 | enum amdxdna_power_mode_type pw_mode; | 196 | enum amdxdna_power_mode_type pw_mode; |
| @@ -182,6 +200,10 @@ struct amdxdna_dev_hdl { · 3 unmodified lines | |||
| 182 | u32 clk_gating; | 200 | u32 clk_gating; |
| 183 | u32 npuclk_freq; | 201 | u32 npuclk_freq; |
| 184 | u32 hclk_freq; | 202 | u32 hclk_freq; |
| 203 | u32 max_tops; | ||
| 204 | u32 curr_tops; | ||
| 205 | u32 force_preempt_enabled; | ||
| 206 | u32 frame_boundary_preempt; | ||
| 185 | 207 | ||
| 186 | /* Mailbox and the management channel */ | 208 | /* Mailbox and the management channel */ |
| 187 | struct mailbox *mbox; | 209 | struct mailbox *mbox; |
| @@ -190,6 +212,8 @@ struct amdxdna_dev_hdl { · 2 unmodified lines | |||
| 190 | 212 | ||
| 191 | enum aie2_dev_status dev_status; | 213 | enum aie2_dev_status dev_status; |
| 192 | u32 hwctx_num; | 214 | u32 hwctx_num; |
| 215 | |||
| 216 | struct amdxdna_async_error last_async_err; | ||
| 193 | }; | 217 | }; |
| 194 | 218 | ||
| 195 | #define DEFINE_BAR_OFFSET(reg_name, bar, reg_addr) \ | 219 | #define DEFINE_BAR_OFFSET(reg_name, bar, reg_addr) \ |
| @@ -204,12 +228,27 @@ struct aie2_hw_ops { · 8 unmodified lines | |||
| 204 | int (*set_dpm)(struct amdxdna_dev_hdl *ndev, u32 dpm_level); | 228 | int (*set_dpm)(struct amdxdna_dev_hdl *ndev, u32 dpm_level); |
| 205 | }; | 229 | }; |
| 206 | 230 | ||
| 231 | enum aie2_fw_feature { | ||
| 232 | AIE2_NPU_COMMAND, | ||
| 233 | AIE2_PREEMPT, | ||
| 234 | AIE2_FEATURE_MAX | ||
| 235 | }; | ||
| 236 | |||
| 237 | struct aie2_fw_feature_tbl { | ||
| 238 | enum aie2_fw_feature feature; | ||
| 239 | u32 max_minor; | ||
| 240 | u32 min_minor; | ||
| 241 | }; | ||
| 242 | |||
| 243 | #define AIE2_FEATURE_ON(ndev, feature) test_bit(feature, &(ndev)->feature_mask) | ||
| 244 | |||
| 207 | struct amdxdna_dev_priv { | 245 | struct amdxdna_dev_priv { |
| 208 | const char *fw_path; | 246 | const char *fw_path; |
| 209 | u64 protocol_major; | 247 | u64 protocol_major; |
| 210 | u64 protocol_minor; | 248 | u64 protocol_minor; |
| 211 | const struct rt_config *rt_config; | 249 | const struct rt_config *rt_config; |
| 212 | const struct dpm_clk_freq *dpm_clk_tbl; | 250 | const struct dpm_clk_freq *dpm_clk_tbl; |
| 251 | const struct aie2_fw_feature_tbl *fw_feature_tbl; | ||
| 213 | 252 | ||
| 214 | #define COL_ALIGN_NONE 0 | 253 | #define COL_ALIGN_NONE 0 |
| 215 | #define COL_ALIGN_NATURE 1 | 254 | #define COL_ALIGN_NATURE 1 |
| @@ -217,6 +256,7 @@ struct amdxdna_dev_priv { · 1 unmodified line | |||
| 217 | u32 mbox_dev_addr; | 256 | u32 mbox_dev_addr; |
| 218 | /* If mbox_size is 0, use BAR size. See MBOX_SIZE macro */ | 257 | /* If mbox_size is 0, use BAR size. See MBOX_SIZE macro */ |
| 219 | u32 mbox_size; | 258 | u32 mbox_size; |
| 259 | u32 hwctx_limit; | ||
| 220 | u32 sram_dev_addr; | 260 | u32 sram_dev_addr; |
| 221 | struct aie2_bar_off_pair sram_offs[SRAM_MAX_INDEX]; | 261 | struct aie2_bar_off_pair sram_offs[SRAM_MAX_INDEX]; |
| 222 | struct aie2_bar_off_pair psp_regs_off[PSP_MAX_REGS]; | 262 | struct aie2_bar_off_pair psp_regs_off[PSP_MAX_REGS]; |
| @@ -234,6 +274,7 @@ extern const struct dpm_clk_freq npu1_dpm_clk_table[]; · 11 unmodified lines | |||
| 234 | extern const struct dpm_clk_freq npu4_dpm_clk_table[]; | 274 | extern const struct dpm_clk_freq npu4_dpm_clk_table[]; |
| 235 | extern const struct rt_config npu1_default_rt_cfg[]; | 275 | extern const struct rt_config npu1_default_rt_cfg[]; |
| 236 | extern const struct rt_config npu4_default_rt_cfg[]; | 276 | extern const struct rt_config npu4_default_rt_cfg[]; |
| 277 | extern const struct aie2_fw_feature_tbl npu4_fw_feature_table[]; | ||
| 237 | 278 | ||
| 238 | /* aie2_smu.c */ | 279 | /* aie2_smu.c */ |
| 239 | int aie2_smu_init(struct amdxdna_dev_hdl *ndev); | 280 | int aie2_smu_init(struct amdxdna_dev_hdl *ndev); |
| @@ -253,10 +294,12 @@ void aie2_psp_stop(struct psp_device *psp); · 13 unmodified lines | |||
| 253 | /* aie2_error.c */ | 294 | /* aie2_error.c */ |
| 254 | int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev); | 295 | int aie2_error_async_events_alloc(struct amdxdna_dev_hdl *ndev); |
| 255 | void aie2_error_async_events_free(struct amdxdna_dev_hdl *ndev); | 296 | void aie2_error_async_events_free(struct amdxdna_dev_hdl *ndev); |
| 256 | int aie2_error_async_events_send(struct amdxdna_dev_hdl *ndev); | ||
| 257 | int aie2_error_async_msg_thread(void *data); | 297 | int aie2_error_async_msg_thread(void *data); |
| 298 | int aie2_get_array_async_error(struct amdxdna_dev_hdl *ndev, | ||
| 299 | struct amdxdna_drm_get_array *args); | ||
| 258 | 300 | ||
| 259 | /* aie2_message.c */ | 301 | /* aie2_message.c */ |
| 302 | void aie2_msg_init(struct amdxdna_dev_hdl *ndev); | ||
| 260 | int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev); | 303 | int aie2_suspend_fw(struct amdxdna_dev_hdl *ndev); |
| 261 | int aie2_resume_fw(struct amdxdna_dev_hdl *ndev); | 304 | int aie2_resume_fw(struct amdxdna_dev_hdl *ndev); |
| 262 | int aie2_set_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 value); | 305 | int aie2_set_runtime_cfg(struct amdxdna_dev_hdl *ndev, u32 type, u64 value); |
| @@ -270,9 +313,13 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct · 7 unmodified lines | |||
| 270 | int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); | 313 | int aie2_destroy_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwctx); |
| 271 | int aie2_map_host_buf(struct amdxdna_dev_hdl *ndev, u32 context_id, u64 addr, u64 size); | 314 | int aie2_map_host_buf(struct amdxdna_dev_hdl *ndev, u32 context_id, u64 addr, u64 size); |
| 272 | int aie2_query_status(struct amdxdna_dev_hdl *ndev, char __user *buf, u32 size, u32 *cols_filled); | 315 | int aie2_query_status(struct amdxdna_dev_hdl *ndev, char __user *buf, u32 size, u32 *cols_filled); |
| 316 | int aie2_query_telemetry(struct amdxdna_dev_hdl *ndev, | ||
| 317 | char __user *buf, u32 size, | ||
| 318 | struct amdxdna_drm_query_telemetry_header *header); | ||
| 273 | int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr, u32 size, | 319 | int aie2_register_asyn_event_msg(struct amdxdna_dev_hdl *ndev, dma_addr_t addr, u32 size, |
| 274 | void *handle, int (*cb)(void*, void __iomem *, size_t)); | 320 | void *handle, int (*cb)(void*, void __iomem *, size_t)); |
| 275 | int aie2_config_cu(struct amdxdna_hwctx *hwctx); | 321 | int aie2_config_cu(struct amdxdna_hwctx *hwctx, |
| 322 | int (*notify_cb)(void *, void __iomem *, size_t)); | ||
| 276 | int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, | 323 | int aie2_execbuf(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, |
| 277 | int (*notify_cb)(void *, void __iomem *, size_t)); | 324 | int (*notify_cb)(void *, void __iomem *, size_t)); |
| 278 | int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx, | 325 | int aie2_cmdlist_single_execbuf(struct amdxdna_hwctx *hwctx, |
| @@ -283,11 +330,14 @@ int aie2_cmdlist_multi_execbuf(struct amdxdna_hwctx *hwctx, · 4 unmodified lines | |||
| 283 | int (*notify_cb)(void *, void __iomem *, size_t)); | 330 | int (*notify_cb)(void *, void __iomem *, size_t)); |
| 284 | int aie2_sync_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, | 331 | int aie2_sync_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, |
| 285 | int (*notify_cb)(void *, void __iomem *, size_t)); | 332 | int (*notify_cb)(void *, void __iomem *, size_t)); |
| 333 | int aie2_config_debug_bo(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, | ||
| 334 | int (*notify_cb)(void *, void __iomem *, size_t)); | ||
| 286 | 335 | ||
| 287 | /* aie2_hwctx.c */ | 336 | /* aie2_hwctx.c */ |
| 288 | int aie2_hwctx_init(struct amdxdna_hwctx *hwctx); | 337 | int aie2_hwctx_init(struct amdxdna_hwctx *hwctx); |
| 289 | void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx); | 338 | void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx); |
| 290 | int aie2_hwctx_config(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size); | 339 | int aie2_hwctx_config(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size); |
| 340 | int aie2_hwctx_sync_debug_bo(struct amdxdna_hwctx *hwctx, u32 debug_bo_hdl); | ||
| 291 | void aie2_hwctx_suspend(struct amdxdna_client *client); | 341 | void aie2_hwctx_suspend(struct amdxdna_client *client); |
| 292 | int aie2_hwctx_resume(struct amdxdna_client *client); | 342 | int aie2_hwctx_resume(struct amdxdna_client *client); |
| 293 | int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, u64 *seq); | 343 | int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, u64 *seq); |
| @@ -11,6 +11,7 @@ · 10 unmodified lines | |||
| 11 | 11 | ||
| 12 | #include "aie2_pci.h" | 12 | #include "aie2_pci.h" |
| 13 | #include "amdxdna_pci_drv.h" | 13 | #include "amdxdna_pci_drv.h" |
| 14 | #include "amdxdna_pm.h" | ||
| 14 | 15 | ||
| 15 | #define SMU_RESULT_OK 1 | 16 | #define SMU_RESULT_OK 1 |
| 16 | 17 | ||
| @@ -22,6 +23,13 @@ · 5 unmodified lines | |||
| 22 | #define AIE2_SMU_SET_SOFT_DPMLEVEL 0x7 | 23 | #define AIE2_SMU_SET_SOFT_DPMLEVEL 0x7 |
| 23 | #define AIE2_SMU_SET_HARD_DPMLEVEL 0x8 | 24 | #define AIE2_SMU_SET_HARD_DPMLEVEL 0x8 |
| 24 | 25 | ||
| 26 | #define NPU4_DPM_TOPS(ndev, dpm_level) \ | ||
| 27 | ({ \ | ||
| 28 | typeof(ndev) _ndev = ndev; \ | ||
| 29 | (4096 * (_ndev)->total_col * \ | ||
| 30 | (_ndev)->priv->dpm_clk_tbl[dpm_level].hclk / 1000000); \ | ||
| 31 | }) | ||
| 32 | |||
| 25 | static int aie2_smu_exec(struct amdxdna_dev_hdl *ndev, u32 reg_cmd, | 33 | static int aie2_smu_exec(struct amdxdna_dev_hdl *ndev, u32 reg_cmd, |
| 26 | u32 reg_arg, u32 *out) | 34 | u32 reg_arg, u32 *out) |
| 27 | { | 35 | { |
| @@ -59,12 +67,16 @@ int npu1_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level) · 31 unmodified lines | |||
| 59 | u32 freq; | 67 | u32 freq; |
| 60 | int ret; | 68 | int ret; |
| 61 | 69 | ||
| 70 | ret = amdxdna_pm_resume_get(ndev->xdna); | ||
| 71 | if (ret) | ||
| 72 | return ret; | ||
| 73 | |||
| 62 | ret = aie2_smu_exec(ndev, AIE2_SMU_SET_MPNPUCLK_FREQ, | 74 | ret = aie2_smu_exec(ndev, AIE2_SMU_SET_MPNPUCLK_FREQ, |
| 63 | ndev->priv->dpm_clk_tbl[dpm_level].npuclk, &freq); | 75 | ndev->priv->dpm_clk_tbl[dpm_level].npuclk, &freq); |
| 64 | if (ret) { | 76 | if (ret) { |
| 65 | XDNA_ERR(ndev->xdna, "Set npu clock to %d failed, ret %d\n", | 77 | XDNA_ERR(ndev->xdna, "Set npu clock to %d failed, ret %d\n", |
| 66 | ndev->priv->dpm_clk_tbl[dpm_level].npuclk, ret); | 78 | ndev->priv->dpm_clk_tbl[dpm_level].npuclk, ret); |
| 67 | return ret; | 79 | goto suspend_put; |
| 68 | } | 80 | } |
| 69 | ndev->npuclk_freq = freq; | 81 | ndev->npuclk_freq = freq; |
| 70 | 82 | ||
| @@ -73,49 +85,78 @@ int npu1_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level) · 2 unmodified lines | |||
| 73 | if (ret) { | 85 | if (ret) { |
| 74 | XDNA_ERR(ndev->xdna, "Set h clock to %d failed, ret %d\n", | 86 | XDNA_ERR(ndev->xdna, "Set h clock to %d failed, ret %d\n", |
| 75 | ndev->priv->dpm_clk_tbl[dpm_level].hclk, ret); | 87 | ndev->priv->dpm_clk_tbl[dpm_level].hclk, ret); |
| 76 | return ret; | 88 | goto suspend_put; |
| 77 | } | 89 | } |
| 90 | |||
| 91 | amdxdna_pm_suspend_put(ndev->xdna); | ||
| 78 | ndev->hclk_freq = freq; | 92 | ndev->hclk_freq = freq; |
| 79 | ndev->dpm_level = dpm_level; | 93 | ndev->dpm_level = dpm_level; |
| 94 | ndev->max_tops = 2 * ndev->total_col; | ||
| 95 | ndev->curr_tops = ndev->max_tops * freq / 1028; | ||
| 80 | 96 | ||
| 81 | XDNA_DBG(ndev->xdna, "MP-NPU clock %d, H clock %d\n", | 97 | XDNA_DBG(ndev->xdna, "MP-NPU clock %d, H clock %d\n", |
| 82 | ndev->npuclk_freq, ndev->hclk_freq); | 98 | ndev->npuclk_freq, ndev->hclk_freq); |
| 83 | 99 | ||
| 84 | return 0; | 100 | return 0; |
| 101 | |||
| 102 | suspend_put: | ||
| 103 | amdxdna_pm_suspend_put(ndev->xdna); | ||
| 104 | return ret; | ||
| 85 | } | 105 | } |
| 86 | 106 | ||
| 87 | int npu4_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level) | 107 | int npu4_set_dpm(struct amdxdna_dev_hdl *ndev, u32 dpm_level) |
| 88 | { | 108 | { |
| 89 | int ret; | 109 | int ret; |
| 90 | 110 | ||
| 111 | ret = amdxdna_pm_resume_get(ndev->xdna); | ||
| 112 | if (ret) | ||
| 113 | return ret; | ||
| 114 | |||
| 91 | ret = aie2_smu_exec(ndev, AIE2_SMU_SET_HARD_DPMLEVEL, dpm_level, NULL); | 115 | ret = aie2_smu_exec(ndev, AIE2_SMU_SET_HARD_DPMLEVEL, dpm_level, NULL); |
| 92 | if (ret) { | 116 | if (ret) { |
| 93 | XDNA_ERR(ndev->xdna, "Set hard dpm level %d failed, ret %d ", | 117 | XDNA_ERR(ndev->xdna, "Set hard dpm level %d failed, ret %d ", |
| 94 | dpm_level, ret); | 118 | dpm_level, ret); |
| 95 | return ret; | 119 | goto suspend_put; |
| 96 | } | 120 | } |
| 97 | 121 | ||
| 98 | ret = aie2_smu_exec(ndev, AIE2_SMU_SET_SOFT_DPMLEVEL, dpm_level, NULL); | 122 | ret = aie2_smu_exec(ndev, AIE2_SMU_SET_SOFT_DPMLEVEL, dpm_level, NULL); |
| 99 | if (ret) { | 123 | if (ret) { |
| 100 | XDNA_ERR(ndev->xdna, "Set soft dpm level %d failed, ret %d", | 124 | XDNA_ERR(ndev->xdna, "Set soft dpm level %d failed, ret %d", |
| 101 | dpm_level, ret); | 125 | dpm_level, ret); |
| 102 | return ret; | 126 | goto suspend_put; |
| 103 | } | 127 | } |
| 104 | 128 | ||
| 129 | amdxdna_pm_suspend_put(ndev->xdna); | ||
| 105 | ndev->npuclk_freq = ndev->priv->dpm_clk_tbl[dpm_level].npuclk; | 130 | ndev->npuclk_freq = ndev->priv->dpm_clk_tbl[dpm_level].npuclk; |
| 106 | ndev->hclk_freq = ndev->priv->dpm_clk_tbl[dpm_level].hclk; | 131 | ndev->hclk_freq = ndev->priv->dpm_clk_tbl[dpm_level].hclk; |
| 107 | ndev->dpm_level = dpm_level; | 132 | ndev->dpm_level = dpm_level; |
| 133 | ndev->max_tops = NPU4_DPM_TOPS(ndev, ndev->max_dpm_level); | ||
| 134 | ndev->curr_tops = NPU4_DPM_TOPS(ndev, dpm_level); | ||
| 108 | 135 | ||
| 109 | XDNA_DBG(ndev->xdna, "MP-NPU clock %d, H clock %d\n", | 136 | XDNA_DBG(ndev->xdna, "MP-NPU clock %d, H clock %d\n", |
| 110 | ndev->npuclk_freq, ndev->hclk_freq); | 137 | ndev->npuclk_freq, ndev->hclk_freq); |
| 111 | 138 | ||
| 112 | return 0; | 139 | return 0; |
| 140 | |||
| 141 | suspend_put: | ||
| 142 | amdxdna_pm_suspend_put(ndev->xdna); | ||
| 143 | return ret; | ||
| 113 | } | 144 | } |
| 114 | 145 | ||
| 115 | int aie2_smu_init(struct amdxdna_dev_hdl *ndev) | 146 | int aie2_smu_init(struct amdxdna_dev_hdl *ndev) |
| 116 | { | 147 | { |
| 117 | int ret; | 148 | int ret; |
| 118 | 149 | ||
| 150 | /* | ||
| 151 | * Failing to set power off indicates an unrecoverable hardware or | ||
| 152 | * firmware error. | ||
| 153 | */ | ||
| 154 | ret = aie2_smu_exec(ndev, AIE2_SMU_POWER_OFF, 0, NULL); | ||
| 155 | if (ret) { | ||
| 156 | XDNA_ERR(ndev->xdna, "Access power failed, ret %d", ret); | ||
| 157 | return ret; | ||
| 158 | } | ||
| 159 | |||
| 119 | ret = aie2_smu_exec(ndev, AIE2_SMU_POWER_ON, 0, NULL); | 160 | ret = aie2_smu_exec(ndev, AIE2_SMU_POWER_ON, 0, NULL); |
| 120 | if (ret) { | 161 | if (ret) { |
| 121 | XDNA_ERR(ndev->xdna, "Power on failed, ret %d", ret); | 162 | XDNA_ERR(ndev->xdna, "Power on failed, ret %d", ret); |
| @@ -113,14 +113,14 @@ void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size) · 112 unmodified lines | |||
| 113 | return &cmd->data[num_masks]; | 113 | return &cmd->data[num_masks]; |
| 114 | } | 114 | } |
| 115 | 115 | ||
| 116 | int amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) | 116 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) |
| 117 | { | 117 | { |
| 118 | struct amdxdna_cmd *cmd = abo->mem.kva; | 118 | struct amdxdna_cmd *cmd = abo->mem.kva; |
| 119 | u32 num_masks, i; | 119 | u32 num_masks, i; |
| 120 | u32 *cu_mask; | 120 | u32 *cu_mask; |
| 121 | 121 | ||
| 122 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) | 122 | if (amdxdna_cmd_get_op(abo) == ERT_CMD_CHAIN) |
| 123 | return -1; | 123 | return INVALID_CU_IDX; |
| 124 | 124 | ||
| 125 | num_masks = 1 + FIELD_GET(AMDXDNA_CMD_EXTRA_CU_MASK, cmd->header); | 125 | num_masks = 1 + FIELD_GET(AMDXDNA_CMD_EXTRA_CU_MASK, cmd->header); |
| 126 | cu_mask = cmd->data; | 126 | cu_mask = cmd->data; |
| @@ -129,7 +129,7 @@ int amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo) · 2 unmodified lines | |||
| 129 | return ffs(cu_mask[i]) - 1; | 129 | return ffs(cu_mask[i]) - 1; |
| 130 | } | 130 | } |
| 131 | 131 | ||
| 132 | return -1; | 132 | return INVALID_CU_IDX; |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | /* | 135 | /* |
| @@ -161,19 +161,14 @@ int amdxdna_drm_create_hwctx_ioctl(struct drm_device *dev, void *data, struct dr · 25 unmodified lines | |||
| 161 | if (args->ext || args->ext_flags) | 161 | if (args->ext || args->ext_flags) |
| 162 | return -EINVAL; | 162 | return -EINVAL; |
| 163 | 163 | ||
| 164 | if (!drm_dev_enter(dev, &idx)) | ||
| 165 | return -ENODEV; | ||
| 166 | |||
| 167 | hwctx = kzalloc(sizeof(*hwctx), GFP_KERNEL); | 164 | hwctx = kzalloc(sizeof(*hwctx), GFP_KERNEL); |
| 168 | if (!hwctx) { | 165 | if (!hwctx) |
| 169 | ret = -ENOMEM; | 166 | return -ENOMEM; |
| 170 | goto exit; | ||
| 171 | } | ||
| 172 | 167 | ||
| 173 | if (copy_from_user(&hwctx->qos, u64_to_user_ptr(args->qos_p), sizeof(hwctx->qos))) { | 168 | if (copy_from_user(&hwctx->qos, u64_to_user_ptr(args->qos_p), sizeof(hwctx->qos))) { |
| 174 | XDNA_ERR(xdna, "Access QoS info failed"); | 169 | XDNA_ERR(xdna, "Access QoS info failed"); |
| 175 | ret = -EFAULT; | 170 | kfree(hwctx); |
| 176 | goto free_hwctx; | 171 | return -EFAULT; |
| 177 | } | 172 | } |
| 178 | 173 | ||
| 179 | hwctx->client = client; | 174 | hwctx->client = client; |
| @@ -181,30 +176,36 @@ int amdxdna_drm_create_hwctx_ioctl(struct drm_device *dev, void *data, struct dr · 1 unmodified line | |||
| 181 | hwctx->num_tiles = args->num_tiles; | 176 | hwctx->num_tiles = args->num_tiles; |
| 182 | hwctx->mem_size = args->mem_size; | 177 | hwctx->mem_size = args->mem_size; |
| 183 | hwctx->max_opc = args->max_opc; | 178 | hwctx->max_opc = args->max_opc; |
| 184 | ret = xa_alloc_cyclic(&client->hwctx_xa, &hwctx->id, hwctx, | 179 | |
| 185 | XA_LIMIT(AMDXDNA_INVALID_CTX_HANDLE + 1, MAX_HWCTX_ID), | 180 | guard(mutex)(&xdna->dev_lock); |
| 186 | &client->next_hwctxid, GFP_KERNEL); | 181 | |
| 187 | if (ret < 0) { | 182 | if (!drm_dev_enter(dev, &idx)) { |
| 188 | XDNA_ERR(xdna, "Allocate hwctx ID failed, ret %d", ret); | 183 | ret = -ENODEV; |
| 189 | goto free_hwctx; | 184 | goto free_hwctx; |
| 190 | } | 185 | } |
| 191 | 186 | ||
| 192 | hwctx->name = kasprintf(GFP_KERNEL, "hwctx.%d.%d", client->pid, hwctx->id); | 187 | ret = xdna->dev_info->ops->hwctx_init(hwctx); |
| 188 | if (ret) { | ||
| 189 | XDNA_ERR(xdna, "Init hwctx failed, ret %d", ret); | ||
| 190 | goto dev_exit; | ||
| 191 | } | ||
| 192 | |||
| 193 | hwctx->name = kasprintf(GFP_KERNEL, "hwctx.%d.%d", client->pid, hwctx->fw_ctx_id); | ||
| 193 | if (!hwctx->name) { | 194 | if (!hwctx->name) { |
| 194 | ret = -ENOMEM; | 195 | ret = -ENOMEM; |
| 195 | goto rm_id; | 196 | goto fini_hwctx; |
| 196 | } | 197 | } |
| 197 | 198 | ||
| 198 | mutex_lock(&xdna->dev_lock); | 199 | ret = xa_alloc_cyclic(&client->hwctx_xa, &hwctx->id, hwctx, |
| 199 | ret = xdna->dev_info->ops->hwctx_init(hwctx); | 200 | XA_LIMIT(AMDXDNA_INVALID_CTX_HANDLE + 1, MAX_HWCTX_ID), |
| 200 | if (ret) { | 201 | &client->next_hwctxid, GFP_KERNEL); |
| 201 | mutex_unlock(&xdna->dev_lock); | 202 | if (ret < 0) { |
| 202 | XDNA_ERR(xdna, "Init hwctx failed, ret %d", ret); | 203 | XDNA_ERR(xdna, "Allocate hwctx ID failed, ret %d", ret); |
| 203 | goto free_name; | 204 | goto free_name; |
| 204 | } | 205 | } |
| 206 | |||
| 205 | args->handle = hwctx->id; | 207 | args->handle = hwctx->id; |
| 206 | args->syncobj_handle = hwctx->syncobj_hdl; | 208 | args->syncobj_handle = hwctx->syncobj_hdl; |
| 207 | mutex_unlock(&xdna->dev_lock); | ||
| 208 | 209 | ||
| 209 | atomic64_set(&hwctx->job_submit_cnt, 0); | 210 | atomic64_set(&hwctx->job_submit_cnt, 0); |
| 210 | atomic64_set(&hwctx->job_free_cnt, 0); | 211 | atomic64_set(&hwctx->job_free_cnt, 0); |
| @@ -214,12 +215,12 @@ int amdxdna_drm_create_hwctx_ioctl(struct drm_device *dev, void *data, struct dr · 3 unmodified lines | |||
| 214 | 215 | ||
| 215 | free_name: | 216 | free_name: |
| 216 | kfree(hwctx->name); | 217 | kfree(hwctx->name); |
| 217 | rm_id: | 218 | fini_hwctx: |
| 218 | xa_erase(&client->hwctx_xa, hwctx->id); | 219 | xdna->dev_info->ops->hwctx_fini(hwctx); |
| 220 | dev_exit: | ||
| 221 | drm_dev_exit(idx); | ||
| 219 | free_hwctx: | 222 | free_hwctx: |
| 220 | kfree(hwctx); | 223 | kfree(hwctx); |
| 221 | exit: | ||
| 222 | drm_dev_exit(idx); | ||
| 223 | return ret; | 224 | return ret; |
| 224 | } | 225 | } |
| 225 | 226 | ||
| @@ -327,6 +328,38 @@ unlock_srcu: · 101 unmodified lines | |||
| 327 | return ret; | 328 | return ret; |
| 328 | } | 329 | } |
| 329 | 330 | ||
| 331 | int amdxdna_hwctx_sync_debug_bo(struct amdxdna_client *client, u32 debug_bo_hdl) | ||
| 332 | { | ||
| 333 | struct amdxdna_dev *xdna = client->xdna; | ||
| 334 | struct amdxdna_hwctx *hwctx; | ||
| 335 | struct amdxdna_gem_obj *abo; | ||
| 336 | struct drm_gem_object *gobj; | ||
| 337 | int ret, idx; | ||
| 338 | |||
| 339 | if (!xdna->dev_info->ops->hwctx_sync_debug_bo) | ||
| 340 | return -EOPNOTSUPP; | ||
| 341 | |||
| 342 | gobj = drm_gem_object_lookup(client->filp, debug_bo_hdl); | ||
| 343 | if (!gobj) | ||
| 344 | return -EINVAL; | ||
| 345 | |||
| 346 | abo = to_xdna_obj(gobj); | ||
| 347 | guard(mutex)(&xdna->dev_lock); | ||
| 348 | idx = srcu_read_lock(&client->hwctx_srcu); | ||
| 349 | hwctx = xa_load(&client->hwctx_xa, abo->assigned_hwctx); | ||
| 350 | if (!hwctx) { | ||
| 351 | ret = -EINVAL; | ||
| 352 | goto unlock_srcu; | ||
| 353 | } | ||
| 354 | |||
| 355 | ret = xdna->dev_info->ops->hwctx_sync_debug_bo(hwctx, debug_bo_hdl); | ||
| 356 | |||
| 357 | unlock_srcu: | ||
| 358 | srcu_read_unlock(&client->hwctx_srcu, idx); | ||
| 359 | drm_gem_object_put(gobj); | ||
| 360 | return ret; | ||
| 361 | } | ||
| 362 | |||
| 330 | static void | 363 | static void |
| 331 | amdxdna_arg_bos_put(struct amdxdna_sched_job *job) | 364 | amdxdna_arg_bos_put(struct amdxdna_sched_job *job) |
| 332 | { | 365 | { |
| @@ -389,9 +422,11 @@ void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job) · 56 unmodified lines | |||
| 389 | trace_amdxdna_debug_point(job->hwctx->name, job->seq, "job release"); | 422 | trace_amdxdna_debug_point(job->hwctx->name, job->seq, "job release"); |
| 390 | amdxdna_arg_bos_put(job); | 423 | amdxdna_arg_bos_put(job); |
| 391 | amdxdna_gem_put_obj(job->cmd_bo); | 424 | amdxdna_gem_put_obj(job->cmd_bo); |
| 425 | dma_fence_put(job->fence); | ||
| 392 | } | 426 | } |
| 393 | 427 | ||
| 394 | int amdxdna_cmd_submit(struct amdxdna_client *client, | 428 | int amdxdna_cmd_submit(struct amdxdna_client *client, |
| 429 | struct amdxdna_drv_cmd *drv_cmd, | ||
| 395 | u32 cmd_bo_hdl, u32 *arg_bo_hdls, u32 arg_bo_cnt, | 430 | u32 cmd_bo_hdl, u32 *arg_bo_hdls, u32 arg_bo_cnt, |
| 396 | u32 hwctx_hdl, u64 *seq) | 431 | u32 hwctx_hdl, u64 *seq) |
| 397 | { | 432 | { |
| @@ -405,6 +440,8 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 7 unmodified lines | |||
| 405 | if (!job) | 440 | if (!job) |
| 406 | return -ENOMEM; | 441 | return -ENOMEM; |
| 407 | 442 | ||
| 443 | job->drv_cmd = drv_cmd; | ||
| 444 | |||
| 408 | if (cmd_bo_hdl != AMDXDNA_INVALID_BO_HANDLE) { | 445 | if (cmd_bo_hdl != AMDXDNA_INVALID_BO_HANDLE) { |
| 409 | job->cmd_bo = amdxdna_gem_get_obj(client, cmd_bo_hdl, AMDXDNA_BO_CMD); | 446 | job->cmd_bo = amdxdna_gem_get_obj(client, cmd_bo_hdl, AMDXDNA_BO_CMD); |
| 410 | if (!job->cmd_bo) { | 447 | if (!job->cmd_bo) { |
| @@ -412,8 +449,6 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 1 unmodified line | |||
| 412 | ret = -EINVAL; | 449 | ret = -EINVAL; |
| 413 | goto free_job; | 450 | goto free_job; |
| 414 | } | 451 | } |
| 415 | } else { | ||
| 416 | job->cmd_bo = NULL; | ||
| 417 | } | 452 | } |
| 418 | 453 | ||
| 419 | ret = amdxdna_arg_bos_lookup(client, job, arg_bo_hdls, arg_bo_cnt); | 454 | ret = amdxdna_arg_bos_lookup(client, job, arg_bo_hdls, arg_bo_cnt); |
| @@ -431,11 +466,6 @@ int amdxdna_cmd_submit(struct amdxdna_client *client, · 11 unmodified lines | |||
| 431 | goto unlock_srcu; | 466 | goto unlock_srcu; |
| 432 | } | 467 | } |
| 433 | 468 | ||
| 434 | if (hwctx->status != HWCTX_STAT_READY) { | ||
| 435 | XDNA_ERR(xdna, "HW Context is not ready"); | ||
| 436 | ret = -EINVAL; | ||
| 437 | goto unlock_srcu; | ||
| 438 | } | ||
| 439 | 469 | ||
| 440 | job->hwctx = hwctx; | 470 | job->hwctx = hwctx; |
| 441 | job->mm = current->mm; | 471 | job->mm = current->mm; |
| @@ -512,7 +542,7 @@ static int amdxdna_drm_submit_execbuf(struct amdxdna_client *client, · 70 unmodified lines | |||
| 512 | } | 542 | } |
| 513 | } | 543 | } |
| 514 | 544 | ||
| 515 | ret = amdxdna_cmd_submit(client, cmd_bo_hdl, arg_bo_hdls, | 545 | ret = amdxdna_cmd_submit(client, NULL, cmd_bo_hdl, arg_bo_hdls, |
| 516 | args->arg_count, args->hwctx, &args->seq); | 546 | args->arg_count, args->hwctx, &args->seq); |
| 517 | if (ret) | 547 | if (ret) |
| 518 | XDNA_DBG(xdna, "Submit cmds failed, ret %d", ret); | 548 | XDNA_DBG(xdna, "Submit cmds failed, ret %d", ret); |
| @@ -13,9 +13,12 @@ · 12 unmodified lines | |||
| 13 | struct amdxdna_hwctx_priv; | 13 | struct amdxdna_hwctx_priv; |
| 14 | 14 | ||
| 15 | enum ert_cmd_opcode { | 15 | enum ert_cmd_opcode { |
| 16 | ERT_START_CU = 0, | 16 | ERT_START_CU = 0, |
| 17 | ERT_CMD_CHAIN = 19, | 17 | ERT_CMD_CHAIN = 19, |
| 18 | ERT_START_NPU = 20, | 18 | ERT_START_NPU = 20, |
| 19 | ERT_START_NPU_PREEMPT = 21, | ||
| 20 | ERT_START_NPU_PREEMPT_ELF = 22, | ||
| 21 | ERT_INVALID_CMD = ~0U, | ||
| 19 | }; | 22 | }; |
| 20 | 23 | ||
| 21 | enum ert_cmd_state { | 24 | enum ert_cmd_state { |
| @@ -54,6 +57,21 @@ struct amdxdna_cmd_chain { · 32 unmodified lines | |||
| 54 | u64 data[] __counted_by(command_count); | 57 | u64 data[] __counted_by(command_count); |
| 55 | }; | 58 | }; |
| 56 | 59 | ||
| 60 | /* | ||
| 61 | * Interpretation of the beginning of data payload for ERT_START_NPU_PREEMPT in | ||
| 62 | * amdxdna_cmd. The rest of the payload in amdxdna_cmd is regular kernel args. | ||
| 63 | */ | ||
| 64 | struct amdxdna_cmd_preempt_data { | ||
| 65 | u64 inst_buf; /* instruction buffer address */ | ||
| 66 | u64 save_buf; /* save buffer address */ | ||
| 67 | u64 restore_buf; /* restore buffer address */ | ||
| 68 | u32 inst_size; /* size of instruction buffer in bytes */ | ||
| 69 | u32 save_size; /* size of save buffer in bytes */ | ||
| 70 | u32 restore_size; /* size of restore buffer in bytes */ | ||
| 71 | u32 inst_prop_cnt; /* properties count */ | ||
| 72 | u32 prop_args[]; /* properties and regular kernel arguments */ | ||
| 73 | }; | ||
| 74 | |||
| 57 | /* Exec buffer command header format */ | 75 | /* Exec buffer command header format */ |
| 58 | #define AMDXDNA_CMD_STATE GENMASK(3, 0) | 76 | #define AMDXDNA_CMD_STATE GENMASK(3, 0) |
| 59 | #define AMDXDNA_CMD_EXTRA_CU_MASK GENMASK(11, 10) | 77 | #define AMDXDNA_CMD_EXTRA_CU_MASK GENMASK(11, 10) |
| @@ -64,6 +82,8 @@ struct amdxdna_cmd { · 4 unmodified lines | |||
| 64 | u32 data[]; | 82 | u32 data[]; |
| 65 | }; | 83 | }; |
| 66 | 84 | ||
| 85 | #define INVALID_CU_IDX (~0U) | ||
| 86 | |||
| 67 | struct amdxdna_hwctx { | 87 | struct amdxdna_hwctx { |
| 68 | struct amdxdna_client *client; | 88 | struct amdxdna_client *client; |
| 69 | struct amdxdna_hwctx_priv *priv; | 89 | struct amdxdna_hwctx_priv *priv; |
| @@ -95,6 +115,17 @@ struct amdxdna_hwctx { · 25 unmodified lines | |||
| 95 | #define drm_job_to_xdna_job(j) \ | 115 | #define drm_job_to_xdna_job(j) \ |
| 96 | container_of(j, struct amdxdna_sched_job, base) | 116 | container_of(j, struct amdxdna_sched_job, base) |
| 97 | 117 | ||
| 118 | enum amdxdna_job_opcode { | ||
| 119 | SYNC_DEBUG_BO, | ||
| 120 | ATTACH_DEBUG_BO, | ||
| 121 | DETACH_DEBUG_BO, | ||
| 122 | }; | ||
| 123 | |||
| 124 | struct amdxdna_drv_cmd { | ||
| 125 | enum amdxdna_job_opcode opcode; | ||
| 126 | u32 result; | ||
| 127 | }; | ||
| 128 | |||
| 98 | struct amdxdna_sched_job { | 129 | struct amdxdna_sched_job { |
| 99 | struct drm_sched_job base; | 130 | struct drm_sched_job base; |
| 100 | struct kref refcnt; | 131 | struct kref refcnt; |
| @@ -105,7 +136,9 @@ struct amdxdna_sched_job { · 4 unmodified lines | |||
| 105 | /* user can wait on this fence */ | 136 | /* user can wait on this fence */ |
| 106 | struct dma_fence *out_fence; | 137 | struct dma_fence *out_fence; |
| 107 | bool job_done; | 138 | bool job_done; |
| 139 | bool job_timeout; | ||
| 108 | u64 seq; | 140 | u64 seq; |
| 141 | struct amdxdna_drv_cmd *drv_cmd; | ||
| 109 | struct amdxdna_gem_obj *cmd_bo; | 142 | struct amdxdna_gem_obj *cmd_bo; |
| 110 | size_t bo_cnt; | 143 | size_t bo_cnt; |
| 111 | struct drm_gem_object *bos[] __counted_by(bo_cnt); | 144 | struct drm_gem_object *bos[] __counted_by(bo_cnt); |
| @@ -137,15 +170,17 @@ amdxdna_cmd_get_state(struct amdxdna_gem_obj *abo) · 25 unmodified lines | |||
| 137 | } | 170 | } |
| 138 | 171 | ||
| 139 | void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size); | 172 | void *amdxdna_cmd_get_payload(struct amdxdna_gem_obj *abo, u32 *size); |
| 140 | int amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo); | 173 | u32 amdxdna_cmd_get_cu_idx(struct amdxdna_gem_obj *abo); |
| 141 | 174 | ||
| 142 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job); | 175 | void amdxdna_sched_job_cleanup(struct amdxdna_sched_job *job); |
| 143 | void amdxdna_hwctx_remove_all(struct amdxdna_client *client); | 176 | void amdxdna_hwctx_remove_all(struct amdxdna_client *client); |
| 144 | int amdxdna_hwctx_walk(struct amdxdna_client *client, void *arg, | 177 | int amdxdna_hwctx_walk(struct amdxdna_client *client, void *arg, |
| 145 | int (*walk)(struct amdxdna_hwctx *hwctx, void *arg)); | 178 | int (*walk)(struct amdxdna_hwctx *hwctx, void *arg)); |
| 179 | int amdxdna_hwctx_sync_debug_bo(struct amdxdna_client *client, u32 debug_bo_hdl); | ||
| 146 | 180 | ||
| 147 | int amdxdna_cmd_submit(struct amdxdna_client *client, | 181 | int amdxdna_cmd_submit(struct amdxdna_client *client, |
| 148 | u32 cmd_bo_hdls, u32 *arg_bo_hdls, u32 arg_bo_cnt, | 182 | struct amdxdna_drv_cmd *drv_cmd, u32 cmd_bo_hdls, |
| 183 | u32 *arg_bo_hdls, u32 arg_bo_cnt, | ||
| 149 | u32 hwctx_hdl, u64 *seq); | 184 | u32 hwctx_hdl, u64 *seq); |
| 150 | 185 | ||
| 151 | int amdxdna_cmd_wait(struct amdxdna_client *client, u32 hwctx_hdl, | 186 | int amdxdna_cmd_wait(struct amdxdna_client *client, u32 hwctx_hdl, |
| @@ -0,0 +1,59 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* | ||
| 3 | * Copyright (C) 2025, Advanced Micro Devices, Inc. | ||
| 4 | */ | ||
| 5 | |||
| 6 | #ifndef _AMDXDNA_ERROR_H_ | ||
| 7 | #define _AMDXDNA_ERROR_H_ | ||
| 8 | |||
| 9 | #include <linux/bitfield.h> | ||
| 10 | #include <linux/bits.h> | ||
| 11 | |||
| 12 | #define AMDXDNA_ERR_DRV_AIE 4 | ||
| 13 | #define AMDXDNA_ERR_SEV_CRITICAL 3 | ||
| 14 | #define AMDXDNA_ERR_CLASS_AIE 2 | ||
| 15 | |||
| 16 | #define AMDXDNA_ERR_NUM_MASK GENMASK_U64(15, 0) | ||
| 17 | #define AMDXDNA_ERR_DRV_MASK GENMASK_U64(23, 16) | ||
| 18 | #define AMDXDNA_ERR_SEV_MASK GENMASK_U64(31, 24) | ||
| 19 | #define AMDXDNA_ERR_MOD_MASK GENMASK_U64(39, 32) | ||
| 20 | #define AMDXDNA_ERR_CLASS_MASK GENMASK_U64(47, 40) | ||
| 21 | |||
| 22 | enum amdxdna_error_num { | ||
| 23 | AMDXDNA_ERROR_NUM_AIE_SATURATION = 3, | ||
| 24 | AMDXDNA_ERROR_NUM_AIE_FP, | ||
| 25 | AMDXDNA_ERROR_NUM_AIE_STREAM, | ||
| 26 | AMDXDNA_ERROR_NUM_AIE_ACCESS, | ||
| 27 | AMDXDNA_ERROR_NUM_AIE_BUS, | ||
| 28 | AMDXDNA_ERROR_NUM_AIE_INSTRUCTION, | ||
| 29 | AMDXDNA_ERROR_NUM_AIE_ECC, | ||
| 30 | AMDXDNA_ERROR_NUM_AIE_LOCK, | ||
| 31 | AMDXDNA_ERROR_NUM_AIE_DMA, | ||
| 32 | AMDXDNA_ERROR_NUM_AIE_MEM_PARITY, | ||
| 33 | AMDXDNA_ERROR_NUM_UNKNOWN = 15, | ||
| 34 | }; | ||
| 35 | |||
| 36 | enum amdxdna_error_module { | ||
| 37 | AMDXDNA_ERROR_MODULE_AIE_CORE = 3, | ||
| 38 | AMDXDNA_ERROR_MODULE_AIE_MEMORY, | ||
| 39 | AMDXDNA_ERROR_MODULE_AIE_SHIM, | ||
| 40 | AMDXDNA_ERROR_MODULE_AIE_NOC, | ||
| 41 | AMDXDNA_ERROR_MODULE_AIE_PL, | ||
| 42 | AMDXDNA_ERROR_MODULE_UNKNOWN = 8, | ||
| 43 | }; | ||
| 44 | |||
| 45 | #define AMDXDNA_ERROR_ENCODE(err_num, err_mod) \ | ||
| 46 | (FIELD_PREP(AMDXDNA_ERR_NUM_MASK, err_num) | \ | ||
| 47 | FIELD_PREP_CONST(AMDXDNA_ERR_DRV_MASK, AMDXDNA_ERR_DRV_AIE) | \ | ||
| 48 | FIELD_PREP_CONST(AMDXDNA_ERR_SEV_MASK, AMDXDNA_ERR_SEV_CRITICAL) | \ | ||
| 49 | FIELD_PREP(AMDXDNA_ERR_MOD_MASK, err_mod) | \ | ||
| 50 | FIELD_PREP_CONST(AMDXDNA_ERR_CLASS_MASK, AMDXDNA_ERR_CLASS_AIE)) | ||
| 51 | |||
| 52 | #define AMDXDNA_EXTRA_ERR_COL_MASK GENMASK_U64(7, 0) | ||
| 53 | #define AMDXDNA_EXTRA_ERR_ROW_MASK GENMASK_U64(15, 8) | ||
| 54 | |||
| 55 | #define AMDXDNA_EXTRA_ERR_ENCODE(row, col) \ | ||
| 56 | (FIELD_PREP(AMDXDNA_EXTRA_ERR_COL_MASK, col) | \ | ||
| 57 | FIELD_PREP(AMDXDNA_EXTRA_ERR_ROW_MASK, row)) | ||
| 58 | |||
| 59 | #endif /* _AMDXDNA_ERROR_H_ */ | ||
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | #include <drm/drm_device.h> | 8 | #include <drm/drm_device.h> |
| 9 | #include <drm/drm_gem.h> | 9 | #include <drm/drm_gem.h> |
| 10 | #include <drm/drm_gem_shmem_helper.h> | 10 | #include <drm/drm_gem_shmem_helper.h> |
| 11 | #include <drm/drm_print.h> | ||
| 11 | #include <drm/gpu_scheduler.h> | 12 | #include <drm/gpu_scheduler.h> |
| 12 | #include <linux/dma-buf.h> | 13 | #include <linux/dma-buf.h> |
| 13 | #include <linux/dma-direct.h> | 14 | #include <linux/dma-direct.h> |
| @@ -392,35 +393,33 @@ static const struct dma_buf_ops amdxdna_dmabuf_ops = { · 378 unmodified lines | |||
| 392 | .vunmap = drm_gem_dmabuf_vunmap, | 393 | .vunmap = drm_gem_dmabuf_vunmap, |
| 393 | }; | 394 | }; |
| 394 | 395 | ||
| 395 | static int amdxdna_gem_obj_vmap(struct drm_gem_object *obj, struct iosys_map *map) | 396 | static int amdxdna_gem_obj_vmap(struct amdxdna_gem_obj *abo, void **vaddr) |
| 396 | { | 397 | { |
| 397 | struct amdxdna_gem_obj *abo = to_xdna_obj(obj); | 398 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); |
| 398 | 399 | int ret; | |
| 399 | iosys_map_clear(map); | ||
| 400 | |||
| 401 | dma_resv_assert_held(obj->resv); | ||
| 402 | 400 | ||
| 403 | if (is_import_bo(abo)) | 401 | if (is_import_bo(abo)) |
| 404 | dma_buf_vmap(abo->dma_buf, map); | 402 | ret = dma_buf_vmap_unlocked(abo->dma_buf, &map); |
| 405 | else | 403 | else |
| 406 | drm_gem_shmem_object_vmap(obj, map); | 404 | ret = drm_gem_vmap(to_gobj(abo), &map); |
| 407 | 405 | ||
| 408 | if (!map->vaddr) | 406 | *vaddr = map.vaddr; |
| 409 | return -ENOMEM; | 407 | return ret; |
| 410 | |||
| 411 | return 0; | ||
| 412 | } | 408 | } |
| 413 | 409 | ||
| 414 | static void amdxdna_gem_obj_vunmap(struct drm_gem_object *obj, struct iosys_map *map) | 410 | static void amdxdna_gem_obj_vunmap(struct amdxdna_gem_obj *abo) |
| 415 | { | 411 | { |
| 416 | struct amdxdna_gem_obj *abo = to_xdna_obj(obj); | 412 | struct iosys_map map; |
| 413 | |||
| 414 | if (!abo->mem.kva) | ||
| 415 | return; | ||
| 417 | 416 | ||
| 418 | dma_resv_assert_held(obj->resv); | 417 | iosys_map_set_vaddr(&map, abo->mem.kva); |
| 419 | 418 | ||
| 420 | if (is_import_bo(abo)) | 419 | if (is_import_bo(abo)) |
| 421 | dma_buf_vunmap(abo->dma_buf, map); | 420 | dma_buf_vunmap_unlocked(abo->dma_buf, &map); |
| 422 | else | 421 | else |
| 423 | drm_gem_shmem_object_vunmap(obj, map); | 422 | drm_gem_vunmap(to_gobj(abo), &map); |
| 424 | } | 423 | } |
| 425 | 424 | ||
| 426 | static struct dma_buf *amdxdna_gem_prime_export(struct drm_gem_object *gobj, int flags) | 425 | static struct dma_buf *amdxdna_gem_prime_export(struct drm_gem_object *gobj, int flags) |
| @@ -455,7 +454,6 @@ static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) · 28 unmodified lines | |||
| 455 | { | 454 | { |
| 456 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); | 455 | struct amdxdna_dev *xdna = to_xdna_dev(gobj->dev); |
| 457 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); | 456 | struct amdxdna_gem_obj *abo = to_xdna_obj(gobj); |
| 458 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(abo->mem.kva); | ||
| 459 | 457 | ||
| 460 | XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, abo->mem.dev_addr); | 458 | XDNA_DBG(xdna, "BO type %d xdna_addr 0x%llx", abo->type, abo->mem.dev_addr); |
| 461 | 459 | ||
| @@ -468,7 +466,7 @@ static void amdxdna_gem_obj_free(struct drm_gem_object *gobj) · 6 unmodified lines | |||
| 468 | if (abo->type == AMDXDNA_BO_DEV_HEAP) | 466 | if (abo->type == AMDXDNA_BO_DEV_HEAP) |
| 469 | drm_mm_takedown(&abo->mm); | 467 | drm_mm_takedown(&abo->mm); |
| 470 | 468 | ||
| 471 | drm_gem_vunmap(gobj, &map); | 469 | amdxdna_gem_obj_vunmap(abo); |
| 472 | mutex_destroy(&abo->lock); | 470 | mutex_destroy(&abo->lock); |
| 473 | 471 | ||
| 474 | if (is_import_bo(abo)) { | 472 | if (is_import_bo(abo)) { |
| @@ -489,8 +487,8 @@ static const struct drm_gem_object_funcs amdxdna_gem_shmem_funcs = { · 14 unmodified lines | |||
| 489 | .pin = drm_gem_shmem_object_pin, | 487 | .pin = drm_gem_shmem_object_pin, |
| 490 | .unpin = drm_gem_shmem_object_unpin, | 488 | .unpin = drm_gem_shmem_object_unpin, |
| 491 | .get_sg_table = drm_gem_shmem_object_get_sg_table, | 489 | .get_sg_table = drm_gem_shmem_object_get_sg_table, |
| 492 | .vmap = amdxdna_gem_obj_vmap, | 490 | .vmap = drm_gem_shmem_object_vmap, |
| 493 | .vunmap = amdxdna_gem_obj_vunmap, | 491 | .vunmap = drm_gem_shmem_object_vunmap, |
| 494 | .mmap = amdxdna_gem_obj_mmap, | 492 | .mmap = amdxdna_gem_obj_mmap, |
| 495 | .vm_ops = &drm_gem_shmem_vm_ops, | 493 | .vm_ops = &drm_gem_shmem_vm_ops, |
| 496 | .export = amdxdna_gem_prime_export, | 494 | .export = amdxdna_gem_prime_export, |
| @@ -663,7 +661,6 @@ amdxdna_drm_create_dev_heap(struct drm_device *dev, · 166 unmodified lines | |||
| 663 | struct drm_file *filp) | 661 | struct drm_file *filp) |
| 664 | { | 662 | { |
| 665 | struct amdxdna_client *client = filp->driver_priv; | 663 | struct amdxdna_client *client = filp->driver_priv; |
| 666 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); | ||
| 667 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | 664 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 668 | struct amdxdna_gem_obj *abo; | 665 | struct amdxdna_gem_obj *abo; |
| 669 | int ret; | 666 | int ret; |
| @@ -692,12 +689,11 @@ amdxdna_drm_create_dev_heap(struct drm_device *dev, · 22 unmodified lines | |||
| 692 | abo->mem.dev_addr = client->xdna->dev_info->dev_mem_base; | 689 | abo->mem.dev_addr = client->xdna->dev_info->dev_mem_base; |
| 693 | drm_mm_init(&abo->mm, abo->mem.dev_addr, abo->mem.size); | 690 | drm_mm_init(&abo->mm, abo->mem.dev_addr, abo->mem.size); |
| 694 | 691 | ||
| 695 | ret = drm_gem_vmap(to_gobj(abo), &map); | 692 | ret = amdxdna_gem_obj_vmap(abo, &abo->mem.kva); |
| 696 | if (ret) { | 693 | if (ret) { |
| 697 | XDNA_ERR(xdna, "Vmap heap bo failed, ret %d", ret); | 694 | XDNA_ERR(xdna, "Vmap heap bo failed, ret %d", ret); |
| 698 | goto release_obj; | 695 | goto release_obj; |
| 699 | } | 696 | } |
| 700 | abo->mem.kva = map.vaddr; | ||
| 701 | 697 | ||
| 702 | client->dev_heap = abo; | 698 | client->dev_heap = abo; |
| 703 | drm_gem_object_get(to_gobj(abo)); | 699 | drm_gem_object_get(to_gobj(abo)); |
| @@ -748,7 +744,6 @@ amdxdna_drm_create_cmd_bo(struct drm_device *dev, · 44 unmodified lines | |||
| 748 | struct amdxdna_drm_create_bo *args, | 744 | struct amdxdna_drm_create_bo *args, |
| 749 | struct drm_file *filp) | 745 | struct drm_file *filp) |
| 750 | { | 746 | { |
| 751 | struct iosys_map map = IOSYS_MAP_INIT_VADDR(NULL); | ||
| 752 | struct amdxdna_dev *xdna = to_xdna_dev(dev); | 747 | struct amdxdna_dev *xdna = to_xdna_dev(dev); |
| 753 | struct amdxdna_gem_obj *abo; | 748 | struct amdxdna_gem_obj *abo; |
| 754 | int ret; | 749 | int ret; |
| @@ -770,12 +765,11 @@ amdxdna_drm_create_cmd_bo(struct drm_device *dev, · 15 unmodified lines | |||
| 770 | abo->type = AMDXDNA_BO_CMD; | 765 | abo->type = AMDXDNA_BO_CMD; |
| 771 | abo->client = filp->driver_priv; | 766 | abo->client = filp->driver_priv; |
| 772 | 767 | ||
| 773 | ret = drm_gem_vmap(to_gobj(abo), &map); | 768 | ret = amdxdna_gem_obj_vmap(abo, &abo->mem.kva); |
| 774 | if (ret) { | 769 | if (ret) { |
| 775 | XDNA_ERR(xdna, "Vmap cmd bo failed, ret %d", ret); | 770 | XDNA_ERR(xdna, "Vmap cmd bo failed, ret %d", ret); |
| 776 | goto release_obj; | 771 | goto release_obj; |
| 777 | } | 772 | } |
| 778 | abo->mem.kva = map.vaddr; | ||
| 779 | 773 | ||
| 780 | return abo; | 774 | return abo; |
| 781 | 775 | ||
| @@ -969,6 +963,9 @@ int amdxdna_drm_sync_bo_ioctl(struct drm_device *dev, · 187 unmodified lines | |||
| 969 | XDNA_DBG(xdna, "Sync bo %d offset 0x%llx, size 0x%llx\n", | 963 | XDNA_DBG(xdna, "Sync bo %d offset 0x%llx, size 0x%llx\n", |
| 970 | args->handle, args->offset, args->size); | 964 | args->handle, args->offset, args->size); |
| 971 | 965 | ||
| 966 | if (args->direction == SYNC_DIRECT_FROM_DEVICE) | ||
| 967 | ret = amdxdna_hwctx_sync_debug_bo(abo->client, args->handle); | ||
| 968 | |||
| 972 | put_obj: | 969 | put_obj: |
| 973 | drm_gem_object_put(gobj); | 970 | drm_gem_object_put(gobj); |
| 974 | return ret; | 971 | return ret; |
| @@ -7,6 +7,7 @@ · 6 unmodified lines | |||
| 7 | #define _AMDXDNA_GEM_H_ | 7 | #define _AMDXDNA_GEM_H_ |
| 8 | 8 | ||
| 9 | #include <linux/hmm.h> | 9 | #include <linux/hmm.h> |
| 10 | #include "amdxdna_pci_drv.h" | ||
| 10 | 11 | ||
| 11 | struct amdxdna_umap { | 12 | struct amdxdna_umap { |
| 12 | struct vm_area_struct *vma; | 13 | struct vm_area_struct *vma; |
| @@ -62,6 +63,11 @@ static inline void amdxdna_gem_put_obj(struct amdxdna_gem_obj *abo) · 49 unmodified lines | |||
| 62 | drm_gem_object_put(to_gobj(abo)); | 63 | drm_gem_object_put(to_gobj(abo)); |
| 63 | } | 64 | } |
| 64 | 65 | ||
| 66 | static inline u64 amdxdna_dev_bo_offset(struct amdxdna_gem_obj *abo) | ||
| 67 | { | ||
| 68 | return abo->mem.dev_addr - abo->client->dev_heap->mem.dev_addr; | ||
| 69 | } | ||
| 70 | |||
| 65 | void amdxdna_umap_put(struct amdxdna_umap *mapp); | 71 | void amdxdna_umap_put(struct amdxdna_umap *mapp); |
| 66 | 72 | ||
| 67 | struct drm_gem_object * | 73 | struct drm_gem_object * |
| @@ -194,7 +194,8 @@ static void mailbox_release_msg(struct mailbox_channel *mb_chann, · 193 unmodified lines | |||
| 194 | { | 194 | { |
| 195 | MB_DBG(mb_chann, "msg_id 0x%x msg opcode 0x%x", | 195 | MB_DBG(mb_chann, "msg_id 0x%x msg opcode 0x%x", |
| 196 | mb_msg->pkg.header.id, mb_msg->pkg.header.opcode); | 196 | mb_msg->pkg.header.id, mb_msg->pkg.header.opcode); |
| 197 | mb_msg->notify_cb(mb_msg->handle, NULL, 0); | 197 | if (mb_msg->notify_cb) |
| 198 | mb_msg->notify_cb(mb_msg->handle, NULL, 0); | ||
| 198 | kfree(mb_msg); | 199 | kfree(mb_msg); |
| 199 | } | 200 | } |
| 200 | 201 | ||
| @@ -248,7 +249,7 @@ mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header *heade · 47 unmodified lines | |||
| 248 | { | 249 | { |
| 249 | struct mailbox_msg *mb_msg; | 250 | struct mailbox_msg *mb_msg; |
| 250 | int msg_id; | 251 | int msg_id; |
| 251 | int ret; | 252 | int ret = 0; |
| 252 | 253 | ||
| 253 | msg_id = header->id; | 254 | msg_id = header->id; |
| 254 | if (!mailbox_validate_msgid(msg_id)) { | 255 | if (!mailbox_validate_msgid(msg_id)) { |
| @@ -265,9 +266,11 @@ mailbox_get_resp(struct mailbox_channel *mb_chann, struct xdna_msg_header *heade · 10 unmodified lines | |||
| 265 | 266 | ||
| 266 | MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x", | 267 | MB_DBG(mb_chann, "opcode 0x%x size %d id 0x%x", |
| 267 | header->opcode, header->total_size, header->id); | 268 | header->opcode, header->total_size, header->id); |
| 268 | ret = mb_msg->notify_cb(mb_msg->handle, data, header->total_size); | 269 | if (mb_msg->notify_cb) { |
| 269 | if (unlikely(ret)) | 270 | ret = mb_msg->notify_cb(mb_msg->handle, data, header->total_size); |
| 270 | MB_ERR(mb_chann, "Message callback ret %d", ret); | 271 | if (unlikely(ret)) |
| 272 | MB_ERR(mb_chann, "Message callback ret %d", ret); | ||
| 273 | } | ||
| 271 | 274 | ||
| 272 | kfree(mb_msg); | 275 | kfree(mb_msg); |
| 273 | return ret; | 276 | return ret; |
| @@ -513,6 +516,7 @@ xdna_mailbox_create_channel(struct mailbox *mb, · 239 unmodified lines | |||
| 513 | } | 516 | } |
| 514 | 517 | ||
| 515 | mb_chann->bad_state = false; | 518 | mb_chann->bad_state = false; |
| 519 | mailbox_reg_write(mb_chann, mb_chann->iohub_int_addr, 0); | ||
| 516 | 520 | ||
| 517 | MB_DBG(mb_chann, "Mailbox channel created (irq: %d)", mb_chann->msix_irq); | 521 | MB_DBG(mb_chann, "Mailbox channel created (irq: %d)", mb_chann->msix_irq); |
| 518 | return mb_chann; | 522 | return mb_chann; |
| @@ -16,16 +16,18 @@ struct xdna_notify { · 15 unmodified lines | |||
| 16 | u32 *data; | 16 | u32 *data; |
| 17 | size_t size; | 17 | size_t size; |
| 18 | int error; | 18 | int error; |
| 19 | u32 *status; | ||
| 19 | }; | 20 | }; |
| 20 | 21 | ||
| 21 | #define DECLARE_XDNA_MSG_COMMON(name, op, status) \ | 22 | #define DECLARE_XDNA_MSG_COMMON(name, op, s) \ |
| 22 | struct name##_req req = { 0 }; \ | 23 | struct name##_req req = { 0 }; \ |
| 23 | struct name##_resp resp = { status }; \ | 24 | struct name##_resp resp = { .status = s }; \ |
| 24 | struct xdna_notify hdl = { \ | 25 | struct xdna_notify hdl = { \ |
| 25 | .error = 0, \ | 26 | .error = 0, \ |
| 26 | .data = (u32 *)&resp, \ | 27 | .data = (u32 *)&resp, \ |
| 27 | .size = sizeof(resp), \ | 28 | .size = sizeof(resp), \ |
| 28 | .comp = COMPLETION_INITIALIZER_ONSTACK(hdl.comp), \ | 29 | .comp = COMPLETION_INITIALIZER_ONSTACK(hdl.comp), \ |
| 30 | .status = (u32 *)&resp.status, \ | ||
| 29 | }; \ | 31 | }; \ |
| 30 | struct xdna_mailbox_msg msg = { \ | 32 | struct xdna_mailbox_msg msg = { \ |
| 31 | .send_data = (u8 *)&req, \ | 33 | .send_data = (u8 *)&req, \ |
| @@ -13,13 +13,11 @@ · 12 unmodified lines | |||
| 13 | #include <drm/gpu_scheduler.h> | 13 | #include <drm/gpu_scheduler.h> |
| 14 | #include <linux/iommu.h> | 14 | #include <linux/iommu.h> |
| 15 | #include <linux/pci.h> | 15 | #include <linux/pci.h> |
| 16 | #include <linux/pm_runtime.h> | ||
| 17 | 16 | ||
| 18 | #include "amdxdna_ctx.h" | 17 | #include "amdxdna_ctx.h" |
| 19 | #include "amdxdna_gem.h" | 18 | #include "amdxdna_gem.h" |
| 20 | #include "amdxdna_pci_drv.h" | 19 | #include "amdxdna_pci_drv.h" |
| 21 | 20 | #include "amdxdna_pm.h" | |
| 22 | #define AMDXDNA_AUTOSUSPEND_DELAY 5000 /* milliseconds */ | ||
| 23 | 21 | ||
| 24 | MODULE_FIRMWARE("amdnpu/1502_00/npu.sbin"); | 22 | MODULE_FIRMWARE("amdnpu/1502_00/npu.sbin"); |
| 25 | MODULE_FIRMWARE("amdnpu/17f0_10/npu.sbin"); | 23 | MODULE_FIRMWARE("amdnpu/17f0_10/npu.sbin"); |
| @@ -29,9 +27,14 @@ MODULE_FIRMWARE("amdnpu/17f0_20/npu.sbin"); · 3 unmodified lines | |||
| 29 | /* | 27 | /* |
| 30 | * 0.0: Initial version | 28 | * 0.0: Initial version |
| 31 | * 0.1: Support getting all hardware contexts by DRM_IOCTL_AMDXDNA_GET_ARRAY | 29 | * 0.1: Support getting all hardware contexts by DRM_IOCTL_AMDXDNA_GET_ARRAY |
| 30 | * 0.2: Support getting last error hardware error | ||
| 31 | * 0.3: Support firmware debug buffer | ||
| 32 | * 0.4: Support getting resource information | ||
| 33 | * 0.5: Support getting telemetry data | ||
| 34 | * 0.6: Support preemption | ||
| 32 | */ | 35 | */ |
| 33 | #define AMDXDNA_DRIVER_MAJOR 0 | 36 | #define AMDXDNA_DRIVER_MAJOR 0 |
| 34 | #define AMDXDNA_DRIVER_MINOR 1 | 37 | #define AMDXDNA_DRIVER_MINOR 6 |
| 35 | 38 | ||
| 36 | /* | 39 | /* |
| 37 | * Bind the driver base on (vendor_id, device_id) pair and later use the | 40 | * Bind the driver base on (vendor_id, device_id) pair and later use the |
| @@ -61,17 +64,9 @@ static int amdxdna_drm_open(struct drm_device *ddev, struct drm_file *filp) · 23 unmodified lines | |||
| 61 | struct amdxdna_client *client; | 64 | struct amdxdna_client *client; |
| 62 | int ret; | 65 | int ret; |
| 63 | 66 | ||
| 64 | ret = pm_runtime_resume_and_get(ddev->dev); | ||
| 65 | if (ret) { | ||
| 66 | XDNA_ERR(xdna, "Failed to get rpm, ret %d", ret); | ||
| 67 | return ret; | ||
| 68 | } | ||
| 69 | |||
| 70 | client = kzalloc(sizeof(*client), GFP_KERNEL); | 67 | client = kzalloc(sizeof(*client), GFP_KERNEL); |
| 71 | if (!client) { | 68 | if (!client) |
| 72 | ret = -ENOMEM; | 69 | return -ENOMEM; |
| 73 | goto put_rpm; | ||
| 74 | } | ||
| 75 | 70 | ||
| 76 | client->pid = pid_nr(rcu_access_pointer(filp->pid)); | 71 | client->pid = pid_nr(rcu_access_pointer(filp->pid)); |
| 77 | client->xdna = xdna; | 72 | client->xdna = xdna; |
| @@ -106,9 +101,6 @@ unbind_sva: · 28 unmodified lines | |||
| 106 | iommu_sva_unbind_device(client->sva); | 101 | iommu_sva_unbind_device(client->sva); |
| 107 | failed: | 102 | failed: |
| 108 | kfree(client); | 103 | kfree(client); |
| 109 | put_rpm: | ||
| 110 | pm_runtime_mark_last_busy(ddev->dev); | ||
| 111 | pm_runtime_put_autosuspend(ddev->dev); | ||
| 112 | 104 | ||
| 113 | return ret; | 105 | return ret; |
| 114 | } | 106 | } |
| @@ -130,8 +122,6 @@ static void amdxdna_drm_close(struct drm_device *ddev, struct drm_file *filp) · 15 unmodified lines | |||
| 130 | 122 | ||
| 131 | XDNA_DBG(xdna, "pid %d closed", client->pid); | 123 | XDNA_DBG(xdna, "pid %d closed", client->pid); |
| 132 | kfree(client); | 124 | kfree(client); |
| 133 | pm_runtime_mark_last_busy(ddev->dev); | ||
| 134 | pm_runtime_put_autosuspend(ddev->dev); | ||
| 135 | } | 125 | } |
| 136 | 126 | ||
| 137 | static int amdxdna_flush(struct file *f, fl_owner_t id) | 127 | static int amdxdna_flush(struct file *f, fl_owner_t id) |
| @@ -310,19 +300,12 @@ static int amdxdna_probe(struct pci_dev *pdev, const struct pci_device_id *id) · 172 unmodified lines | |||
| 310 | goto failed_dev_fini; | 300 | goto failed_dev_fini; |
| 311 | } | 301 | } |
| 312 | 302 | ||
| 313 | pm_runtime_set_autosuspend_delay(dev, AMDXDNA_AUTOSUSPEND_DELAY); | ||
| 314 | pm_runtime_use_autosuspend(dev); | ||
| 315 | pm_runtime_allow(dev); | ||
| 316 | |||
| 317 | ret = drm_dev_register(&xdna->ddev, 0); | 303 | ret = drm_dev_register(&xdna->ddev, 0); |
| 318 | if (ret) { | 304 | if (ret) { |
| 319 | XDNA_ERR(xdna, "DRM register failed, ret %d", ret); | 305 | XDNA_ERR(xdna, "DRM register failed, ret %d", ret); |
| 320 | pm_runtime_forbid(dev); | ||
| 321 | goto failed_sysfs_fini; | 306 | goto failed_sysfs_fini; |
| 322 | } | 307 | } |
| 323 | 308 | ||
| 324 | pm_runtime_mark_last_busy(dev); | ||
| 325 | pm_runtime_put_autosuspend(dev); | ||
| 326 | return 0; | 309 | return 0; |
| 327 | 310 | ||
| 328 | failed_sysfs_fini: | 311 | failed_sysfs_fini: |
| @@ -339,14 +322,10 @@ destroy_notifier_wq: · 10 unmodified lines | |||
| 339 | static void amdxdna_remove(struct pci_dev *pdev) | 322 | static void amdxdna_remove(struct pci_dev *pdev) |
| 340 | { | 323 | { |
| 341 | struct amdxdna_dev *xdna = pci_get_drvdata(pdev); | 324 | struct amdxdna_dev *xdna = pci_get_drvdata(pdev); |
| 342 | struct device *dev = &pdev->dev; | ||
| 343 | struct amdxdna_client *client; | 325 | struct amdxdna_client *client; |
| 344 | 326 | ||
| 345 | destroy_workqueue(xdna->notifier_wq); | 327 | destroy_workqueue(xdna->notifier_wq); |
| 346 | 328 | ||
| 347 | pm_runtime_get_noresume(dev); | ||
| 348 | pm_runtime_forbid(dev); | ||
| 349 | |||
| 350 | drm_dev_unplug(&xdna->ddev); | 329 | drm_dev_unplug(&xdna->ddev); |
| 351 | amdxdna_sysfs_fini(xdna); | 330 | amdxdna_sysfs_fini(xdna); |
| 352 | 331 | ||
| @@ -365,29 +344,9 @@ static void amdxdna_remove(struct pci_dev *pdev) · 12 unmodified lines | |||
| 365 | mutex_unlock(&xdna->dev_lock); | 344 | mutex_unlock(&xdna->dev_lock); |
| 366 | } | 345 | } |
| 367 | 346 | ||
| 368 | static int amdxdna_pmops_suspend(struct device *dev) | ||
| 369 | { | ||
| 370 | struct amdxdna_dev *xdna = pci_get_drvdata(to_pci_dev(dev)); | ||
| 371 | |||
| 372 | if (!xdna->dev_info->ops->suspend) | ||
| 373 | return -EOPNOTSUPP; | ||
| 374 | |||
| 375 | return xdna->dev_info->ops->suspend(xdna); | ||
| 376 | } | ||
| 377 | |||
| 378 | static int amdxdna_pmops_resume(struct device *dev) | ||
| 379 | { | ||
| 380 | struct amdxdna_dev *xdna = pci_get_drvdata(to_pci_dev(dev)); | ||
| 381 | |||
| 382 | if (!xdna->dev_info->ops->resume) | ||
| 383 | return -EOPNOTSUPP; | ||
| 384 | |||
| 385 | return xdna->dev_info->ops->resume(xdna); | ||
| 386 | } | ||
| 387 | |||
| 388 | static const struct dev_pm_ops amdxdna_pm_ops = { | 347 | static const struct dev_pm_ops amdxdna_pm_ops = { |
| 389 | SYSTEM_SLEEP_PM_OPS(amdxdna_pmops_suspend, amdxdna_pmops_resume) | 348 | SYSTEM_SLEEP_PM_OPS(amdxdna_pm_suspend, amdxdna_pm_resume) |
| 390 | RUNTIME_PM_OPS(amdxdna_pmops_suspend, amdxdna_pmops_resume, NULL) | 349 | RUNTIME_PM_OPS(amdxdna_pm_suspend, amdxdna_pm_resume, NULL) |
| 391 | }; | 350 | }; |
| 392 | 351 | ||
| 393 | static struct pci_driver amdxdna_pci_driver = { | 352 | static struct pci_driver amdxdna_pci_driver = { |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | |||
| 6 | #ifndef _AMDXDNA_PCI_DRV_H_ | 6 | #ifndef _AMDXDNA_PCI_DRV_H_ |
| 7 | #define _AMDXDNA_PCI_DRV_H_ | 7 | #define _AMDXDNA_PCI_DRV_H_ |
| 8 | 8 | ||
| 9 | #include <drm/drm_print.h> | ||
| 9 | #include <linux/workqueue.h> | 10 | #include <linux/workqueue.h> |
| 10 | #include <linux/xarray.h> | 11 | #include <linux/xarray.h> |
| 11 | 12 | ||
| @@ -54,6 +55,7 @@ struct amdxdna_dev_ops { · 42 unmodified lines | |||
| 54 | int (*hwctx_init)(struct amdxdna_hwctx *hwctx); | 55 | int (*hwctx_init)(struct amdxdna_hwctx *hwctx); |
| 55 | void (*hwctx_fini)(struct amdxdna_hwctx *hwctx); | 56 | void (*hwctx_fini)(struct amdxdna_hwctx *hwctx); |
| 56 | int (*hwctx_config)(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size); | 57 | int (*hwctx_config)(struct amdxdna_hwctx *hwctx, u32 type, u64 value, void *buf, u32 size); |
| 58 | int (*hwctx_sync_debug_bo)(struct amdxdna_hwctx *hwctx, u32 debug_bo_hdl); | ||
| 57 | void (*hmm_invalidate)(struct amdxdna_gem_obj *abo, unsigned long cur_seq); | 59 | void (*hmm_invalidate)(struct amdxdna_gem_obj *abo, unsigned long cur_seq); |
| 58 | int (*cmd_submit)(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, u64 *seq); | 60 | int (*cmd_submit)(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, u64 *seq); |
| 59 | int (*get_aie_info)(struct amdxdna_client *client, struct amdxdna_drm_get_info *args); | 61 | int (*get_aie_info)(struct amdxdna_client *client, struct amdxdna_drm_get_info *args); |
| @@ -99,6 +101,7 @@ struct amdxdna_dev { · 39 unmodified lines | |||
| 99 | struct amdxdna_fw_ver fw_ver; | 101 | struct amdxdna_fw_ver fw_ver; |
| 100 | struct rw_semaphore notifier_lock; /* for mmu notifier*/ | 102 | struct rw_semaphore notifier_lock; /* for mmu notifier*/ |
| 101 | struct workqueue_struct *notifier_wq; | 103 | struct workqueue_struct *notifier_wq; |
| 104 | bool rpm_on; | ||
| 102 | }; | 105 | }; |
| 103 | 106 | ||
| 104 | /* | 107 | /* |
| @@ -0,0 +1,94 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 | ||
| 2 | /* | ||
| 3 | * Copyright (C) 2025, Advanced Micro Devices, Inc. | ||
| 4 | */ | ||
| 5 | |||
| 6 | #include <drm/amdxdna_accel.h> | ||
| 7 | #include <drm/drm_drv.h> | ||
| 8 | #include <linux/pm_runtime.h> | ||
| 9 | |||
| 10 | #include "amdxdna_pm.h" | ||
| 11 | |||
| 12 | #define AMDXDNA_AUTOSUSPEND_DELAY 5000 /* milliseconds */ | ||
| 13 | |||
| 14 | int amdxdna_pm_suspend(struct device *dev) | ||
| 15 | { | ||
| 16 | struct amdxdna_dev *xdna = to_xdna_dev(dev_get_drvdata(dev)); | ||
| 17 | int ret = -EOPNOTSUPP; | ||
| 18 | bool rpm; | ||
| 19 | |||
| 20 | if (xdna->dev_info->ops->suspend) { | ||
| 21 | rpm = xdna->rpm_on; | ||
| 22 | xdna->rpm_on = false; | ||
| 23 | ret = xdna->dev_info->ops->suspend(xdna); | ||
| 24 | xdna->rpm_on = rpm; | ||
| 25 | } | ||
| 26 | |||
| 27 | XDNA_DBG(xdna, "Suspend done ret %d", ret); | ||
| 28 | return ret; | ||
| 29 | } | ||
| 30 | |||
| 31 | int amdxdna_pm_resume(struct device *dev) | ||
| 32 | { | ||
| 33 | struct amdxdna_dev *xdna = to_xdna_dev(dev_get_drvdata(dev)); | ||
| 34 | int ret = -EOPNOTSUPP; | ||
| 35 | bool rpm; | ||
| 36 | |||
| 37 | if (xdna->dev_info->ops->resume) { | ||
| 38 | rpm = xdna->rpm_on; | ||
| 39 | xdna->rpm_on = false; | ||
| 40 | ret = xdna->dev_info->ops->resume(xdna); | ||
| 41 | xdna->rpm_on = rpm; | ||
| 42 | } | ||
| 43 | |||
| 44 | XDNA_DBG(xdna, "Resume done ret %d", ret); | ||
| 45 | return ret; | ||
| 46 | } | ||
| 47 | |||
| 48 | int amdxdna_pm_resume_get(struct amdxdna_dev *xdna) | ||
| 49 | { | ||
| 50 | struct device *dev = xdna->ddev.dev; | ||
| 51 | int ret; | ||
| 52 | |||
| 53 | if (!xdna->rpm_on) | ||
| 54 | return 0; | ||
| 55 | |||
| 56 | ret = pm_runtime_resume_and_get(dev); | ||
| 57 | if (ret) { | ||
| 58 | XDNA_ERR(xdna, "Resume failed: %d", ret); | ||
| 59 | pm_runtime_set_suspended(dev); | ||
| 60 | } | ||
| 61 | |||
| 62 | return ret; | ||
| 63 | } | ||
| 64 | |||
| 65 | void amdxdna_pm_suspend_put(struct amdxdna_dev *xdna) | ||
| 66 | { | ||
| 67 | struct device *dev = xdna->ddev.dev; | ||
| 68 | |||
| 69 | if (!xdna->rpm_on) | ||
| 70 | return; | ||
| 71 | |||
| 72 | pm_runtime_put_autosuspend(dev); | ||
| 73 | } | ||
| 74 | |||
| 75 | void amdxdna_pm_init(struct amdxdna_dev *xdna) | ||
| 76 | { | ||
| 77 | struct device *dev = xdna->ddev.dev; | ||
| 78 | |||
| 79 | pm_runtime_set_active(dev); | ||
| 80 | pm_runtime_set_autosuspend_delay(dev, AMDXDNA_AUTOSUSPEND_DELAY); | ||
| 81 | pm_runtime_use_autosuspend(dev); | ||
| 82 | pm_runtime_allow(dev); | ||
| 83 | pm_runtime_put_autosuspend(dev); | ||
| 84 | xdna->rpm_on = true; | ||
| 85 | } | ||
| 86 | |||
| 87 | void amdxdna_pm_fini(struct amdxdna_dev *xdna) | ||
| 88 | { | ||
| 89 | struct device *dev = xdna->ddev.dev; | ||
| 90 | |||
| 91 | xdna->rpm_on = false; | ||
| 92 | pm_runtime_get_noresume(dev); | ||
| 93 | pm_runtime_forbid(dev); | ||
| 94 | } | ||
| @@ -0,0 +1,18 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 */ | ||
| 2 | /* | ||
| 3 | * Copyright (C) 2025, Advanced Micro Devices, Inc. | ||
| 4 | */ | ||
| 5 | |||
| 6 | #ifndef _AMDXDNA_PM_H_ | ||
| 7 | #define _AMDXDNA_PM_H_ | ||
| 8 | |||
| 9 | #include "amdxdna_pci_drv.h" | ||
| 10 | |||
| 11 | int amdxdna_pm_suspend(struct device *dev); | ||
| 12 | int amdxdna_pm_resume(struct device *dev); | ||
| 13 | int amdxdna_pm_resume_get(struct amdxdna_dev *xdna); | ||
| 14 | void amdxdna_pm_suspend_put(struct amdxdna_dev *xdna); | ||
| 15 | void amdxdna_pm_init(struct amdxdna_dev *xdna); | ||
| 16 | void amdxdna_pm_fini(struct amdxdna_dev *xdna); | ||
| 17 | |||
| 18 | #endif /* _AMDXDNA_PM_H_ */ | ||
| @@ -46,6 +46,7 @@ · 45 unmodified lines | |||
| 46 | 46 | ||
| 47 | const struct rt_config npu1_default_rt_cfg[] = { | 47 | const struct rt_config npu1_default_rt_cfg[] = { |
| 48 | { 2, 1, AIE2_RT_CFG_INIT }, /* PDI APP LOAD MODE */ | 48 | { 2, 1, AIE2_RT_CFG_INIT }, /* PDI APP LOAD MODE */ |
| 49 | { 4, 1, AIE2_RT_CFG_INIT }, /* Debug BO */ | ||
| 49 | { 1, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ | 50 | { 1, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ |
| 50 | { 0 }, | 51 | { 0 }, |
| 51 | }; | 52 | }; |
| @@ -62,16 +63,23 @@ const struct dpm_clk_freq npu1_dpm_clk_table[] = { · 10 unmodified lines | |||
| 62 | { 0 } | 63 | { 0 } |
| 63 | }; | 64 | }; |
| 64 | 65 | ||
| 66 | static const struct aie2_fw_feature_tbl npu1_fw_feature_table[] = { | ||
| 67 | { .feature = AIE2_NPU_COMMAND, .min_minor = 8 }, | ||
| 68 | { 0 } | ||
| 69 | }; | ||
| 70 | |||
| 65 | static const struct amdxdna_dev_priv npu1_dev_priv = { | 71 | static const struct amdxdna_dev_priv npu1_dev_priv = { |
| 66 | .fw_path = "amdnpu/1502_00/npu.sbin", | 72 | .fw_path = "amdnpu/1502_00/npu.sbin", |
| 67 | .protocol_major = 0x5, | 73 | .protocol_major = 0x5, |
| 68 | .protocol_minor = 0x7, | 74 | .protocol_minor = 0x7, |
| 69 | .rt_config = npu1_default_rt_cfg, | 75 | .rt_config = npu1_default_rt_cfg, |
| 70 | .dpm_clk_tbl = npu1_dpm_clk_table, | 76 | .dpm_clk_tbl = npu1_dpm_clk_table, |
| 77 | .fw_feature_tbl = npu1_fw_feature_table, | ||
| 71 | .col_align = COL_ALIGN_NONE, | 78 | .col_align = COL_ALIGN_NONE, |
| 72 | .mbox_dev_addr = NPU1_MBOX_BAR_BASE, | 79 | .mbox_dev_addr = NPU1_MBOX_BAR_BASE, |
| 73 | .mbox_size = 0, /* Use BAR size */ | 80 | .mbox_size = 0, /* Use BAR size */ |
| 74 | .sram_dev_addr = NPU1_SRAM_BAR_BASE, | 81 | .sram_dev_addr = NPU1_SRAM_BAR_BASE, |
| 82 | .hwctx_limit = 6, | ||
| 75 | .sram_offs = { | 83 | .sram_offs = { |
| 76 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU1_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), | 84 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU1_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), |
| 77 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU1_SRAM, MPNPU_SRAM_I2X_MAILBOX_15), | 85 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU1_SRAM, MPNPU_SRAM_I2X_MAILBOX_15), |
| @@ -67,10 +67,12 @@ static const struct amdxdna_dev_priv npu2_dev_priv = { · 66 unmodified lines | |||
| 67 | .protocol_minor = 0x6, | 67 | .protocol_minor = 0x6, |
| 68 | .rt_config = npu4_default_rt_cfg, | 68 | .rt_config = npu4_default_rt_cfg, |
| 69 | .dpm_clk_tbl = npu4_dpm_clk_table, | 69 | .dpm_clk_tbl = npu4_dpm_clk_table, |
| 70 | .fw_feature_tbl = npu4_fw_feature_table, | ||
| 70 | .col_align = COL_ALIGN_NATURE, | 71 | .col_align = COL_ALIGN_NATURE, |
| 71 | .mbox_dev_addr = NPU2_MBOX_BAR_BASE, | 72 | .mbox_dev_addr = NPU2_MBOX_BAR_BASE, |
| 72 | .mbox_size = 0, /* Use BAR size */ | 73 | .mbox_size = 0, /* Use BAR size */ |
| 73 | .sram_dev_addr = NPU2_SRAM_BAR_BASE, | 74 | .sram_dev_addr = NPU2_SRAM_BAR_BASE, |
| 75 | .hwctx_limit = 16, | ||
| 74 | .sram_offs = { | 76 | .sram_offs = { |
| 75 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU2_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), | 77 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU2_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), |
| 76 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU2_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), | 78 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU2_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), |
| @@ -63,10 +63,14 @@ · 62 unmodified lines | |||
| 63 | 63 | ||
| 64 | const struct rt_config npu4_default_rt_cfg[] = { | 64 | const struct rt_config npu4_default_rt_cfg[] = { |
| 65 | { 5, 1, AIE2_RT_CFG_INIT }, /* PDI APP LOAD MODE */ | 65 | { 5, 1, AIE2_RT_CFG_INIT }, /* PDI APP LOAD MODE */ |
| 66 | { 10, 1, AIE2_RT_CFG_INIT }, /* DEBUG BUF */ | ||
| 67 | { 14, 0, AIE2_RT_CFG_INIT, BIT_U64(AIE2_PREEMPT) }, /* Frame boundary preemption */ | ||
| 66 | { 1, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ | 68 | { 1, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ |
| 67 | { 2, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ | 69 | { 2, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ |
| 68 | { 3, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ | 70 | { 3, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ |
| 69 | { 4, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ | 71 | { 4, 1, AIE2_RT_CFG_CLK_GATING }, /* Clock gating on */ |
| 72 | { 13, 0, AIE2_RT_CFG_FORCE_PREEMPT }, | ||
| 73 | { 14, 0, AIE2_RT_CFG_FRAME_BOUNDARY_PREEMPT }, | ||
| 70 | { 0 }, | 74 | { 0 }, |
| 71 | }; | 75 | }; |
| 72 | 76 | ||
| @@ -82,16 +86,24 @@ const struct dpm_clk_freq npu4_dpm_clk_table[] = { · 9 unmodified lines | |||
| 82 | { 0 } | 86 | { 0 } |
| 83 | }; | 87 | }; |
| 84 | 88 | ||
| 89 | const struct aie2_fw_feature_tbl npu4_fw_feature_table[] = { | ||
| 90 | { .feature = AIE2_NPU_COMMAND, .min_minor = 15 }, | ||
| 91 | { .feature = AIE2_PREEMPT, .min_minor = 12 }, | ||
| 92 | { 0 } | ||
| 93 | }; | ||
| 94 | |||
| 85 | static const struct amdxdna_dev_priv npu4_dev_priv = { | 95 | static const struct amdxdna_dev_priv npu4_dev_priv = { |
| 86 | .fw_path = "amdnpu/17f0_10/npu.sbin", | 96 | .fw_path = "amdnpu/17f0_10/npu.sbin", |
| 87 | .protocol_major = 0x6, | 97 | .protocol_major = 0x6, |
| 88 | .protocol_minor = 12, | 98 | .protocol_minor = 12, |
| 89 | .rt_config = npu4_default_rt_cfg, | 99 | .rt_config = npu4_default_rt_cfg, |
| 90 | .dpm_clk_tbl = npu4_dpm_clk_table, | 100 | .dpm_clk_tbl = npu4_dpm_clk_table, |
| 101 | .fw_feature_tbl = npu4_fw_feature_table, | ||
| 91 | .col_align = COL_ALIGN_NATURE, | 102 | .col_align = COL_ALIGN_NATURE, |
| 92 | .mbox_dev_addr = NPU4_MBOX_BAR_BASE, | 103 | .mbox_dev_addr = NPU4_MBOX_BAR_BASE, |
| 93 | .mbox_size = 0, /* Use BAR size */ | 104 | .mbox_size = 0, /* Use BAR size */ |
| 94 | .sram_dev_addr = NPU4_SRAM_BAR_BASE, | 105 | .sram_dev_addr = NPU4_SRAM_BAR_BASE, |
| 106 | .hwctx_limit = 16, | ||
| 95 | .sram_offs = { | 107 | .sram_offs = { |
| 96 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU4_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), | 108 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU4_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), |
| 97 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU4_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), | 109 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU4_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), |
| @@ -67,10 +67,12 @@ static const struct amdxdna_dev_priv npu5_dev_priv = { · 66 unmodified lines | |||
| 67 | .protocol_minor = 12, | 67 | .protocol_minor = 12, |
| 68 | .rt_config = npu4_default_rt_cfg, | 68 | .rt_config = npu4_default_rt_cfg, |
| 69 | .dpm_clk_tbl = npu4_dpm_clk_table, | 69 | .dpm_clk_tbl = npu4_dpm_clk_table, |
| 70 | .fw_feature_tbl = npu4_fw_feature_table, | ||
| 70 | .col_align = COL_ALIGN_NATURE, | 71 | .col_align = COL_ALIGN_NATURE, |
| 71 | .mbox_dev_addr = NPU5_MBOX_BAR_BASE, | 72 | .mbox_dev_addr = NPU5_MBOX_BAR_BASE, |
| 72 | .mbox_size = 0, /* Use BAR size */ | 73 | .mbox_size = 0, /* Use BAR size */ |
| 73 | .sram_dev_addr = NPU5_SRAM_BAR_BASE, | 74 | .sram_dev_addr = NPU5_SRAM_BAR_BASE, |
| 75 | .hwctx_limit = 16, | ||
| 74 | .sram_offs = { | 76 | .sram_offs = { |
| 75 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU5_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), | 77 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU5_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), |
| 76 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU5_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), | 78 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU5_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), |
| @@ -67,10 +67,12 @@ static const struct amdxdna_dev_priv npu6_dev_priv = { · 66 unmodified lines | |||
| 67 | .protocol_minor = 12, | 67 | .protocol_minor = 12, |
| 68 | .rt_config = npu4_default_rt_cfg, | 68 | .rt_config = npu4_default_rt_cfg, |
| 69 | .dpm_clk_tbl = npu4_dpm_clk_table, | 69 | .dpm_clk_tbl = npu4_dpm_clk_table, |
| 70 | .fw_feature_tbl = npu4_fw_feature_table, | ||
| 70 | .col_align = COL_ALIGN_NATURE, | 71 | .col_align = COL_ALIGN_NATURE, |
| 71 | .mbox_dev_addr = NPU6_MBOX_BAR_BASE, | 72 | .mbox_dev_addr = NPU6_MBOX_BAR_BASE, |
| 72 | .mbox_size = 0, /* Use BAR size */ | 73 | .mbox_size = 0, /* Use BAR size */ |
| 73 | .sram_dev_addr = NPU6_SRAM_BAR_BASE, | 74 | .sram_dev_addr = NPU6_SRAM_BAR_BASE, |
| 75 | .hwctx_limit = 16, | ||
| 74 | .sram_offs = { | 76 | .sram_offs = { |
| 75 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU6_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), | 77 | DEFINE_BAR_OFFSET(MBOX_CHANN_OFF, NPU6_SRAM, MPNPU_SRAM_X2I_MAILBOX_0), |
| 76 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU6_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), | 78 | DEFINE_BAR_OFFSET(FW_ALIVE_OFF, NPU6_SRAM, MPNPU_SRAM_X2I_MAILBOX_15), |
| @@ -0,0 +1,11 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | |||
| 3 | config DRM_ACCEL_ARM_ETHOSU | ||
| 4 | tristate "Arm Ethos-U65/U85 NPU" | ||
| 5 | depends on HAS_IOMEM | ||
| 6 | depends on DRM_ACCEL | ||
| 7 | select DRM_GEM_DMA_HELPER | ||
| 8 | select DRM_SCHED | ||
| 9 | select GENERIC_ALLOCATOR | ||
| 10 | help | ||
| 11 | Enables driver for Arm Ethos-U65/U85 NPUs | ||
| @@ -0,0 +1,4 @@ | |||
| 1 | # SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | |||
| 3 | obj-$(CONFIG_DRM_ACCEL_ARM_ETHOSU) := ethosu.o | ||
| 4 | ethosu-y += ethosu_drv.o ethosu_gem.o ethosu_job.o | ||
| @@ -0,0 +1,197 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only or MIT */ | ||
| 2 | /* Copyright 2025 Arm, Ltd. */ | ||
| 3 | |||
| 4 | #ifndef __ETHOSU_DEVICE_H__ | ||
| 5 | #define __ETHOSU_DEVICE_H__ | ||
| 6 | |||
| 7 | #include <linux/bitfield.h> | ||
| 8 | #include <linux/bits.h> | ||
| 9 | #include <linux/types.h> | ||
| 10 | |||
| 11 | #include <drm/drm_device.h> | ||
| 12 | #include <drm/gpu_scheduler.h> | ||
| 13 | |||
| 14 | #include <drm/ethosu_accel.h> | ||
| 15 | |||
| 16 | struct clk; | ||
| 17 | struct gen_pool; | ||
| 18 | |||
| 19 | #define NPU_REG_ID 0x0000 | ||
| 20 | #define NPU_REG_STATUS 0x0004 | ||
| 21 | #define NPU_REG_CMD 0x0008 | ||
| 22 | #define NPU_REG_RESET 0x000c | ||
| 23 | #define NPU_REG_QBASE 0x0010 | ||
| 24 | #define NPU_REG_QBASE_HI 0x0014 | ||
| 25 | #define NPU_REG_QREAD 0x0018 | ||
| 26 | #define NPU_REG_QCONFIG 0x001c | ||
| 27 | #define NPU_REG_QSIZE 0x0020 | ||
| 28 | #define NPU_REG_PROT 0x0024 | ||
| 29 | #define NPU_REG_CONFIG 0x0028 | ||
| 30 | #define NPU_REG_REGIONCFG 0x003c | ||
| 31 | #define NPU_REG_AXILIMIT0 0x0040 // U65 | ||
| 32 | #define NPU_REG_AXILIMIT1 0x0044 // U65 | ||
| 33 | #define NPU_REG_AXILIMIT2 0x0048 // U65 | ||
| 34 | #define NPU_REG_AXILIMIT3 0x004c // U65 | ||
| 35 | #define NPU_REG_MEM_ATTR0 0x0040 // U85 | ||
| 36 | #define NPU_REG_MEM_ATTR1 0x0044 // U85 | ||
| 37 | #define NPU_REG_MEM_ATTR2 0x0048 // U85 | ||
| 38 | #define NPU_REG_MEM_ATTR3 0x004c // U85 | ||
| 39 | #define NPU_REG_AXI_SRAM 0x0050 // U85 | ||
| 40 | #define NPU_REG_AXI_EXT 0x0054 // U85 | ||
| 41 | |||
| 42 | #define NPU_REG_BASEP(x) (0x0080 + (x) * 8) | ||
| 43 | #define NPU_REG_BASEP_HI(x) (0x0084 + (x) * 8) | ||
| 44 | #define NPU_BASEP_REGION_MAX 8 | ||
| 45 | |||
| 46 | #define ID_ARCH_MAJOR_MASK GENMASK(31, 28) | ||
| 47 | #define ID_ARCH_MINOR_MASK GENMASK(27, 20) | ||
| 48 | #define ID_ARCH_PATCH_MASK GENMASK(19, 16) | ||
| 49 | #define ID_VER_MAJOR_MASK GENMASK(11, 8) | ||
| 50 | #define ID_VER_MINOR_MASK GENMASK(7, 4) | ||
| 51 | |||
| 52 | #define CONFIG_MACS_PER_CC_MASK GENMASK(3, 0) | ||
| 53 | #define CONFIG_CMD_STREAM_VER_MASK GENMASK(7, 4) | ||
| 54 | |||
| 55 | #define STATUS_STATE_RUNNING BIT(0) | ||
| 56 | #define STATUS_IRQ_RAISED BIT(1) | ||
| 57 | #define STATUS_BUS_STATUS BIT(2) | ||
| 58 | #define STATUS_RESET_STATUS BIT(3) | ||
| 59 | #define STATUS_CMD_PARSE_ERR BIT(4) | ||
| 60 | #define STATUS_CMD_END_REACHED BIT(5) | ||
| 61 | |||
| 62 | #define CMD_CLEAR_IRQ BIT(1) | ||
| 63 | #define CMD_TRANSITION_TO_RUN BIT(0) | ||
| 64 | |||
| 65 | #define RESET_PENDING_CSL BIT(1) | ||
| 66 | #define RESET_PENDING_CPL BIT(0) | ||
| 67 | |||
| 68 | #define PROT_ACTIVE_CSL BIT(1) | ||
| 69 | |||
| 70 | enum ethosu_cmds { | ||
| 71 | NPU_OP_CONV = 0x2, | ||
| 72 | NPU_OP_DEPTHWISE = 0x3, | ||
| 73 | NPU_OP_POOL = 0x5, | ||
| 74 | NPU_OP_ELEMENTWISE = 0x6, | ||
| 75 | NPU_OP_RESIZE = 0x7, // U85 only | ||
| 76 | NPU_OP_DMA_START = 0x10, | ||
| 77 | NPU_SET_IFM_PAD_TOP = 0x100, | ||
| 78 | NPU_SET_IFM_PAD_LEFT = 0x101, | ||
| 79 | NPU_SET_IFM_PAD_RIGHT = 0x102, | ||
| 80 | NPU_SET_IFM_PAD_BOTTOM = 0x103, | ||
| 81 | NPU_SET_IFM_DEPTH_M1 = 0x104, | ||
| 82 | NPU_SET_IFM_PRECISION = 0x105, | ||
| 83 | NPU_SET_IFM_BROADCAST = 0x108, | ||
| 84 | NPU_SET_IFM_WIDTH0_M1 = 0x10a, | ||
| 85 | NPU_SET_IFM_HEIGHT0_M1 = 0x10b, | ||
| 86 | NPU_SET_IFM_HEIGHT1_M1 = 0x10c, | ||
| 87 | NPU_SET_IFM_REGION = 0x10f, | ||
| 88 | NPU_SET_OFM_WIDTH_M1 = 0x111, | ||
| 89 | NPU_SET_OFM_HEIGHT_M1 = 0x112, | ||
| 90 | NPU_SET_OFM_DEPTH_M1 = 0x113, | ||
| 91 | NPU_SET_OFM_PRECISION = 0x114, | ||
| 92 | NPU_SET_OFM_WIDTH0_M1 = 0x11a, | ||
| 93 | NPU_SET_OFM_HEIGHT0_M1 = 0x11b, | ||
| 94 | NPU_SET_OFM_HEIGHT1_M1 = 0x11c, | ||
| 95 | NPU_SET_OFM_REGION = 0x11f, | ||
| 96 | NPU_SET_KERNEL_WIDTH_M1 = 0x120, | ||
| 97 | NPU_SET_KERNEL_HEIGHT_M1 = 0x121, | ||
| 98 | NPU_SET_KERNEL_STRIDE = 0x122, | ||
| 99 | NPU_SET_WEIGHT_REGION = 0x128, | ||
| 100 | NPU_SET_SCALE_REGION = 0x129, | ||
| 101 | NPU_SET_DMA0_SRC_REGION = 0x130, | ||
| 102 | NPU_SET_DMA0_DST_REGION = 0x131, | ||
| 103 | NPU_SET_DMA0_SIZE0 = 0x132, | ||
| 104 | NPU_SET_DMA0_SIZE1 = 0x133, | ||
| 105 | NPU_SET_IFM2_BROADCAST = 0x180, | ||
| 106 | NPU_SET_IFM2_PRECISION = 0x185, | ||
| 107 | NPU_SET_IFM2_WIDTH0_M1 = 0x18a, | ||
| 108 | NPU_SET_IFM2_HEIGHT0_M1 = 0x18b, | ||
| 109 | NPU_SET_IFM2_HEIGHT1_M1 = 0x18c, | ||
| 110 | NPU_SET_IFM2_REGION = 0x18f, | ||
| 111 | NPU_SET_IFM_BASE0 = 0x4000, | ||
| 112 | NPU_SET_IFM_BASE1 = 0x4001, | ||
| 113 | NPU_SET_IFM_BASE2 = 0x4002, | ||
| 114 | NPU_SET_IFM_BASE3 = 0x4003, | ||
| 115 | NPU_SET_IFM_STRIDE_X = 0x4004, | ||
| 116 | NPU_SET_IFM_STRIDE_Y = 0x4005, | ||
| 117 | NPU_SET_IFM_STRIDE_C = 0x4006, | ||
| 118 | NPU_SET_OFM_BASE0 = 0x4010, | ||
| 119 | NPU_SET_OFM_BASE1 = 0x4011, | ||
| 120 | NPU_SET_OFM_BASE2 = 0x4012, | ||
| 121 | NPU_SET_OFM_BASE3 = 0x4013, | ||
| 122 | NPU_SET_OFM_STRIDE_X = 0x4014, | ||
| 123 | NPU_SET_OFM_STRIDE_Y = 0x4015, | ||
| 124 | NPU_SET_OFM_STRIDE_C = 0x4016, | ||
| 125 | NPU_SET_WEIGHT_BASE = 0x4020, | ||
| 126 | NPU_SET_WEIGHT_LENGTH = 0x4021, | ||
| 127 | NPU_SET_SCALE_BASE = 0x4022, | ||
| 128 | NPU_SET_SCALE_LENGTH = 0x4023, | ||
| 129 | NPU_SET_DMA0_SRC = 0x4030, | ||
| 130 | NPU_SET_DMA0_DST = 0x4031, | ||
| 131 | NPU_SET_DMA0_LEN = 0x4032, | ||
| 132 | NPU_SET_DMA0_SRC_STRIDE0 = 0x4033, | ||
| 133 | NPU_SET_DMA0_SRC_STRIDE1 = 0x4034, | ||
| 134 | NPU_SET_DMA0_DST_STRIDE0 = 0x4035, | ||
| 135 | NPU_SET_DMA0_DST_STRIDE1 = 0x4036, | ||
| 136 | NPU_SET_IFM2_BASE0 = 0x4080, | ||
| 137 | NPU_SET_IFM2_BASE1 = 0x4081, | ||
| 138 | NPU_SET_IFM2_BASE2 = 0x4082, | ||
| 139 | NPU_SET_IFM2_BASE3 = 0x4083, | ||
| 140 | NPU_SET_IFM2_STRIDE_X = 0x4084, | ||
| 141 | NPU_SET_IFM2_STRIDE_Y = 0x4085, | ||
| 142 | NPU_SET_IFM2_STRIDE_C = 0x4086, | ||
| 143 | NPU_SET_WEIGHT1_BASE = 0x4090, | ||
| 144 | NPU_SET_WEIGHT1_LENGTH = 0x4091, | ||
| 145 | NPU_SET_SCALE1_BASE = 0x4092, | ||
| 146 | NPU_SET_WEIGHT2_BASE = 0x4092, | ||
| 147 | NPU_SET_SCALE1_LENGTH = 0x4093, | ||
| 148 | NPU_SET_WEIGHT2_LENGTH = 0x4093, | ||
| 149 | NPU_SET_WEIGHT3_BASE = 0x4094, | ||
| 150 | NPU_SET_WEIGHT3_LENGTH = 0x4095, | ||
| 151 | }; | ||
| 152 | |||
| 153 | #define ETHOSU_SRAM_REGION 2 /* Matching Vela compiler */ | ||
| 154 | |||
| 155 | /** | ||
| 156 | * struct ethosu_device - Ethosu device | ||
| 157 | */ | ||
| 158 | struct ethosu_device { | ||
| 159 | /** @base: Base drm_device. */ | ||
| 160 | struct drm_device base; | ||
| 161 | |||
| 162 | /** @iomem: CPU mapping of the registers. */ | ||
| 163 | void __iomem *regs; | ||
| 164 | |||
| 165 | void __iomem *sram; | ||
| 166 | struct gen_pool *srampool; | ||
| 167 | dma_addr_t sramphys; | ||
| 168 | |||
| 169 | struct clk_bulk_data *clks; | ||
| 170 | int num_clks; | ||
| 171 | int irq; | ||
| 172 | |||
| 173 | struct drm_ethosu_npu_info npu_info; | ||
| 174 | |||
| 175 | struct ethosu_job *in_flight_job; | ||
| 176 | /* For in_flight_job and ethosu_job_hw_submit() */ | ||
| 177 | struct mutex job_lock; | ||
| 178 | |||
| 179 | /* For dma_fence */ | ||
| 180 | spinlock_t fence_lock; | ||
| 181 | |||
| 182 | struct drm_gpu_scheduler sched; | ||
| 183 | /* For ethosu_job_do_push() */ | ||
| 184 | struct mutex sched_lock; | ||
| 185 | u64 fence_context; | ||
| 186 | u64 emit_seqno; | ||
| 187 | }; | ||
| 188 | |||
| 189 | #define to_ethosu_device(drm_dev) \ | ||
| 190 | ((struct ethosu_device *)container_of(drm_dev, struct ethosu_device, base)) | ||
| 191 | |||
| 192 | static inline bool ethosu_is_u65(const struct ethosu_device *ethosudev) | ||
| 193 | { | ||
| 194 | return FIELD_GET(ID_ARCH_MAJOR_MASK, ethosudev->npu_info.id) == 1; | ||
| 195 | } | ||
| 196 | |||
| 197 | #endif | ||
| @@ -0,0 +1,403 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only or MIT | ||
| 2 | // Copyright (C) 2025 Arm, Ltd. | ||
| 3 | |||
| 4 | #include <linux/bitfield.h> | ||
| 5 | #include <linux/clk.h> | ||
| 6 | #include <linux/genalloc.h> | ||
| 7 | #include <linux/io.h> | ||
| 8 | #include <linux/iopoll.h> | ||
| 9 | #include <linux/module.h> | ||
| 10 | #include <linux/mod_devicetable.h> | ||
| 11 | #include <linux/platform_device.h> | ||
| 12 | #include <linux/pm_runtime.h> | ||
| 13 | |||
| 14 | #include <drm/drm_drv.h> | ||
| 15 | #include <drm/drm_ioctl.h> | ||
| 16 | #include <drm/drm_utils.h> | ||
| 17 | #include <drm/drm_gem.h> | ||
| 18 | #include <drm/drm_accel.h> | ||
| 19 | #include <drm/ethosu_accel.h> | ||
| 20 | |||
| 21 | #include "ethosu_drv.h" | ||
| 22 | #include "ethosu_device.h" | ||
| 23 | #include "ethosu_gem.h" | ||
| 24 | #include "ethosu_job.h" | ||
| 25 | |||
| 26 | static int ethosu_ioctl_dev_query(struct drm_device *ddev, void *data, | ||
| 27 | struct drm_file *file) | ||
| 28 | { | ||
| 29 | struct ethosu_device *ethosudev = to_ethosu_device(ddev); | ||
| 30 | struct drm_ethosu_dev_query *args = data; | ||
| 31 | |||
| 32 | if (!args->pointer) { | ||
| 33 | switch (args->type) { | ||
| 34 | case DRM_ETHOSU_DEV_QUERY_NPU_INFO: | ||
| 35 | args->size = sizeof(ethosudev->npu_info); | ||
| 36 | return 0; | ||
| 37 | default: | ||
| 38 | return -EINVAL; | ||
| 39 | } | ||
| 40 | } | ||
| 41 | |||
| 42 | switch (args->type) { | ||
| 43 | case DRM_ETHOSU_DEV_QUERY_NPU_INFO: | ||
| 44 | if (args->size < offsetofend(struct drm_ethosu_npu_info, sram_size)) | ||
| 45 | return -EINVAL; | ||
| 46 | return copy_struct_to_user(u64_to_user_ptr(args->pointer), | ||
| 47 | args->size, | ||
| 48 | ðosudev->npu_info, | ||
| 49 | sizeof(ethosudev->npu_info), NULL); | ||
| 50 | default: | ||
| 51 | return -EINVAL; | ||
| 52 | } | ||
| 53 | } | ||
| 54 | |||
| 55 | #define ETHOSU_BO_FLAGS DRM_ETHOSU_BO_NO_MMAP | ||
| 56 | |||
| 57 | static int ethosu_ioctl_bo_create(struct drm_device *ddev, void *data, | ||
| 58 | struct drm_file *file) | ||
| 59 | { | ||
| 60 | struct drm_ethosu_bo_create *args = data; | ||
| 61 | int cookie, ret; | ||
| 62 | |||
| 63 | if (!drm_dev_enter(ddev, &cookie)) | ||
| 64 | return -ENODEV; | ||
| 65 | |||
| 66 | if (!args->size || (args->flags & ~ETHOSU_BO_FLAGS)) { | ||
| 67 | ret = -EINVAL; | ||
| 68 | goto out_dev_exit; | ||
| 69 | } | ||
| 70 | |||
| 71 | ret = ethosu_gem_create_with_handle(file, ddev, &args->size, | ||
| 72 | args->flags, &args->handle); | ||
| 73 | |||
| 74 | out_dev_exit: | ||
| 75 | drm_dev_exit(cookie); | ||
| 76 | return ret; | ||
| 77 | } | ||
| 78 | |||
| 79 | static int ethosu_ioctl_bo_wait(struct drm_device *ddev, void *data, | ||
| 80 | struct drm_file *file) | ||
| 81 | { | ||
| 82 | struct drm_ethosu_bo_wait *args = data; | ||
| 83 | int cookie, ret; | ||
| 84 | unsigned long timeout = drm_timeout_abs_to_jiffies(args->timeout_ns); | ||
| 85 | |||
| 86 | if (args->pad) | ||
| 87 | return -EINVAL; | ||
| 88 | |||
| 89 | if (!drm_dev_enter(ddev, &cookie)) | ||
| 90 | return -ENODEV; | ||
| 91 | |||
| 92 | ret = drm_gem_dma_resv_wait(file, args->handle, true, timeout); | ||
| 93 | |||
| 94 | drm_dev_exit(cookie); | ||
| 95 | return ret; | ||
| 96 | } | ||
| 97 | |||
| 98 | static int ethosu_ioctl_bo_mmap_offset(struct drm_device *ddev, void *data, | ||
| 99 | struct drm_file *file) | ||
| 100 | { | ||
| 101 | struct drm_ethosu_bo_mmap_offset *args = data; | ||
| 102 | struct drm_gem_object *obj; | ||
| 103 | |||
| 104 | if (args->pad) | ||
| 105 | return -EINVAL; | ||
| 106 | |||
| 107 | obj = drm_gem_object_lookup(file, args->handle); | ||
| 108 | if (!obj) | ||
| 109 | return -ENOENT; | ||
| 110 | |||
| 111 | args->offset = drm_vma_node_offset_addr(&obj->vma_node); | ||
| 112 | drm_gem_object_put(obj); | ||
| 113 | return 0; | ||
| 114 | } | ||
| 115 | |||
| 116 | static int ethosu_ioctl_cmdstream_bo_create(struct drm_device *ddev, void *data, | ||
| 117 | struct drm_file *file) | ||
| 118 | { | ||
| 119 | struct drm_ethosu_cmdstream_bo_create *args = data; | ||
| 120 | int cookie, ret; | ||
| 121 | |||
| 122 | if (!drm_dev_enter(ddev, &cookie)) | ||
| 123 | return -ENODEV; | ||
| 124 | |||
| 125 | if (!args->size || !args->data || args->pad || args->flags) { | ||
| 126 | ret = -EINVAL; | ||
| 127 | goto out_dev_exit; | ||
| 128 | } | ||
| 129 | |||
| 130 | args->flags |= DRM_ETHOSU_BO_NO_MMAP; | ||
| 131 | |||
| 132 | ret = ethosu_gem_cmdstream_create(file, ddev, args->size, args->data, | ||
| 133 | args->flags, &args->handle); | ||
| 134 | |||
| 135 | out_dev_exit: | ||
| 136 | drm_dev_exit(cookie); | ||
| 137 | return ret; | ||
| 138 | } | ||
| 139 | |||
| 140 | static int ethosu_open(struct drm_device *ddev, struct drm_file *file) | ||
| 141 | { | ||
| 142 | int ret = 0; | ||
| 143 | |||
| 144 | if (!try_module_get(THIS_MODULE)) | ||
| 145 | return -EINVAL; | ||
| 146 | |||
| 147 | struct ethosu_file_priv __free(kfree) *priv = kzalloc(sizeof(*priv), GFP_KERNEL); | ||
| 148 | if (!priv) { | ||
| 149 | ret = -ENOMEM; | ||
| 150 | goto err_put_mod; | ||
| 151 | } | ||
| 152 | priv->edev = to_ethosu_device(ddev); | ||
| 153 | |||
| 154 | ret = ethosu_job_open(priv); | ||
| 155 | if (ret) | ||
| 156 | goto err_put_mod; | ||
| 157 | |||
| 158 | file->driver_priv = no_free_ptr(priv); | ||
| 159 | return 0; | ||
| 160 | |||
| 161 | err_put_mod: | ||
| 162 | module_put(THIS_MODULE); | ||
| 163 | return ret; | ||
| 164 | } | ||
| 165 | |||
| 166 | static void ethosu_postclose(struct drm_device *ddev, struct drm_file *file) | ||
| 167 | { | ||
| 168 | ethosu_job_close(file->driver_priv); | ||
| 169 | kfree(file->driver_priv); | ||
| 170 | module_put(THIS_MODULE); | ||
| 171 | } | ||
| 172 | |||
| 173 | static const struct drm_ioctl_desc ethosu_drm_driver_ioctls[] = { | ||
| 174 | #define ETHOSU_IOCTL(n, func, flags) \ | ||
| 175 | DRM_IOCTL_DEF_DRV(ETHOSU_##n, ethosu_ioctl_##func, flags) | ||
| 176 | |||
| 177 | ETHOSU_IOCTL(DEV_QUERY, dev_query, 0), | ||
| 178 | ETHOSU_IOCTL(BO_CREATE, bo_create, 0), | ||
| 179 | ETHOSU_IOCTL(BO_WAIT, bo_wait, 0), | ||
| 180 | ETHOSU_IOCTL(BO_MMAP_OFFSET, bo_mmap_offset, 0), | ||
| 181 | ETHOSU_IOCTL(CMDSTREAM_BO_CREATE, cmdstream_bo_create, 0), | ||
| 182 | ETHOSU_IOCTL(SUBMIT, submit, 0), | ||
| 183 | }; | ||
| 184 | |||
| 185 | DEFINE_DRM_ACCEL_FOPS(ethosu_drm_driver_fops); | ||
| 186 | |||
| 187 | /* | ||
| 188 | * Ethosu driver version: | ||
| 189 | * - 1.0 - initial interface | ||
| 190 | */ | ||
| 191 | static const struct drm_driver ethosu_drm_driver = { | ||
| 192 | .driver_features = DRIVER_COMPUTE_ACCEL | DRIVER_GEM, | ||
| 193 | .open = ethosu_open, | ||
| 194 | .postclose = ethosu_postclose, | ||
| 195 | .ioctls = ethosu_drm_driver_ioctls, | ||
| 196 | .num_ioctls = ARRAY_SIZE(ethosu_drm_driver_ioctls), | ||
| 197 | .fops = ðosu_drm_driver_fops, | ||
| 198 | .name = "ethosu", | ||
| 199 | .desc = "Arm Ethos-U Accel driver", | ||
| 200 | .major = 1, | ||
| 201 | .minor = 0, | ||
| 202 | |||
| 203 | .gem_create_object = ethosu_gem_create_object, | ||
| 204 | }; | ||
| 205 | |||
| 206 | #define U65_DRAM_AXI_LIMIT_CFG 0x1f3f0002 | ||
| 207 | #define U65_SRAM_AXI_LIMIT_CFG 0x1f3f00b0 | ||
| 208 | #define U85_AXI_EXT_CFG 0x00021f3f | ||
| 209 | #define U85_AXI_SRAM_CFG 0x00021f3f | ||
| 210 | #define U85_MEM_ATTR0_CFG 0x00000000 | ||
| 211 | #define U85_MEM_ATTR2_CFG 0x000000b7 | ||
| 212 | |||
| 213 | static int ethosu_reset(struct ethosu_device *ethosudev) | ||
| 214 | { | ||
| 215 | int ret; | ||
| 216 | u32 reg; | ||
| 217 | |||
| 218 | writel_relaxed(RESET_PENDING_CSL, ethosudev->regs + NPU_REG_RESET); | ||
| 219 | ret = readl_poll_timeout(ethosudev->regs + NPU_REG_STATUS, reg, | ||
| 220 | !FIELD_GET(STATUS_RESET_STATUS, reg), | ||
| 221 | USEC_PER_MSEC, USEC_PER_SEC); | ||
| 222 | if (ret) | ||
| 223 | return ret; | ||
| 224 | |||
| 225 | if (!FIELD_GET(PROT_ACTIVE_CSL, readl_relaxed(ethosudev->regs + NPU_REG_PROT))) { | ||
| 226 | dev_warn(ethosudev->base.dev, "Could not reset to non-secure mode (PROT = %x)\n", | ||
| 227 | readl_relaxed(ethosudev->regs + NPU_REG_PROT)); | ||
| 228 | } | ||
| 229 | |||
| 230 | /* | ||
| 231 | * Assign region 2 (SRAM) to AXI M0 (AXILIMIT0), | ||
| 232 | * everything else to AXI M1 (AXILIMIT2) | ||
| 233 | */ | ||
| 234 | writel_relaxed(0x0000aa8a, ethosudev->regs + NPU_REG_REGIONCFG); | ||
| 235 | if (ethosu_is_u65(ethosudev)) { | ||
| 236 | writel_relaxed(U65_SRAM_AXI_LIMIT_CFG, ethosudev->regs + NPU_REG_AXILIMIT0); | ||
| 237 | writel_relaxed(U65_DRAM_AXI_LIMIT_CFG, ethosudev->regs + NPU_REG_AXILIMIT2); | ||
| 238 | } else { | ||
| 239 | writel_relaxed(U85_AXI_SRAM_CFG, ethosudev->regs + NPU_REG_AXI_SRAM); | ||
| 240 | writel_relaxed(U85_AXI_EXT_CFG, ethosudev->regs + NPU_REG_AXI_EXT); | ||
| 241 | writel_relaxed(U85_MEM_ATTR0_CFG, ethosudev->regs + NPU_REG_MEM_ATTR0); // SRAM | ||
| 242 | writel_relaxed(U85_MEM_ATTR2_CFG, ethosudev->regs + NPU_REG_MEM_ATTR2); // DRAM | ||
| 243 | } | ||
| 244 | |||
| 245 | if (ethosudev->sram) | ||
| 246 | memset_io(ethosudev->sram, 0, ethosudev->npu_info.sram_size); | ||
| 247 | |||
| 248 | return 0; | ||
| 249 | } | ||
| 250 | |||
| 251 | static int ethosu_device_resume(struct device *dev) | ||
| 252 | { | ||
| 253 | struct ethosu_device *ethosudev = dev_get_drvdata(dev); | ||
| 254 | int ret; | ||
| 255 | |||
| 256 | ret = clk_bulk_prepare_enable(ethosudev->num_clks, ethosudev->clks); | ||
| 257 | if (ret) | ||
| 258 | return ret; | ||
| 259 | |||
| 260 | ret = ethosu_reset(ethosudev); | ||
| 261 | if (!ret) | ||
| 262 | return 0; | ||
| 263 | |||
| 264 | clk_bulk_disable_unprepare(ethosudev->num_clks, ethosudev->clks); | ||
| 265 | return ret; | ||
| 266 | } | ||
| 267 | |||
| 268 | static int ethosu_device_suspend(struct device *dev) | ||
| 269 | { | ||
| 270 | struct ethosu_device *ethosudev = dev_get_drvdata(dev); | ||
| 271 | |||
| 272 | clk_bulk_disable_unprepare(ethosudev->num_clks, ethosudev->clks); | ||
| 273 | return 0; | ||
| 274 | } | ||
| 275 | |||
| 276 | static int ethosu_sram_init(struct ethosu_device *ethosudev) | ||
| 277 | { | ||
| 278 | ethosudev->npu_info.sram_size = 0; | ||
| 279 | |||
| 280 | ethosudev->srampool = of_gen_pool_get(ethosudev->base.dev->of_node, "sram", 0); | ||
| 281 | if (!ethosudev->srampool) | ||
| 282 | return 0; | ||
| 283 | |||
| 284 | ethosudev->npu_info.sram_size = gen_pool_size(ethosudev->srampool); | ||
| 285 | |||
| 286 | ethosudev->sram = (void __iomem *)gen_pool_dma_alloc(ethosudev->srampool, | ||
| 287 | ethosudev->npu_info.sram_size, | ||
| 288 | ðosudev->sramphys); | ||
| 289 | if (!ethosudev->sram) { | ||
| 290 | dev_err(ethosudev->base.dev, "failed to allocate from SRAM pool\n"); | ||
| 291 | return -ENOMEM; | ||
| 292 | } | ||
| 293 | |||
| 294 | return 0; | ||
| 295 | } | ||
| 296 | |||
| 297 | static int ethosu_init(struct ethosu_device *ethosudev) | ||
| 298 | { | ||
| 299 | int ret; | ||
| 300 | u32 id, config; | ||
| 301 | |||
| 302 | ret = ethosu_device_resume(ethosudev->base.dev); | ||
| 303 | if (ret) | ||
| 304 | return ret; | ||
| 305 | |||
| 306 | pm_runtime_set_autosuspend_delay(ethosudev->base.dev, 50); | ||
| 307 | pm_runtime_use_autosuspend(ethosudev->base.dev); | ||
| 308 | ret = devm_pm_runtime_set_active_enabled(ethosudev->base.dev); | ||
| 309 | if (ret) | ||
| 310 | return ret; | ||
| 311 | pm_runtime_get_noresume(ethosudev->base.dev); | ||
| 312 | |||
| 313 | ethosudev->npu_info.id = id = readl_relaxed(ethosudev->regs + NPU_REG_ID); | ||
| 314 | ethosudev->npu_info.config = config = readl_relaxed(ethosudev->regs + NPU_REG_CONFIG); | ||
| 315 | |||
| 316 | ethosu_sram_init(ethosudev); | ||
| 317 | |||
| 318 | dev_info(ethosudev->base.dev, | ||
| 319 | "Ethos-U NPU, arch v%ld.%ld.%ld, rev r%ldp%ld, cmd stream ver%ld, %d MACs, %dKB SRAM\n", | ||
| 320 | FIELD_GET(ID_ARCH_MAJOR_MASK, id), | ||
| 321 | FIELD_GET(ID_ARCH_MINOR_MASK, id), | ||
| 322 | FIELD_GET(ID_ARCH_PATCH_MASK, id), | ||
| 323 | FIELD_GET(ID_VER_MAJOR_MASK, id), | ||
| 324 | FIELD_GET(ID_VER_MINOR_MASK, id), | ||
| 325 | FIELD_GET(CONFIG_CMD_STREAM_VER_MASK, config), | ||
| 326 | 1 << FIELD_GET(CONFIG_MACS_PER_CC_MASK, config), | ||
| 327 | ethosudev->npu_info.sram_size / 1024); | ||
| 328 | |||
| 329 | return 0; | ||
| 330 | } | ||
| 331 | |||
| 332 | static int ethosu_probe(struct platform_device *pdev) | ||
| 333 | { | ||
| 334 | int ret; | ||
| 335 | struct ethosu_device *ethosudev; | ||
| 336 | |||
| 337 | ethosudev = devm_drm_dev_alloc(&pdev->dev, ðosu_drm_driver, | ||
| 338 | struct ethosu_device, base); | ||
| 339 | if (IS_ERR(ethosudev)) | ||
| 340 | return -ENOMEM; | ||
| 341 | platform_set_drvdata(pdev, ethosudev); | ||
| 342 | |||
| 343 | dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(40)); | ||
| 344 | |||
| 345 | ethosudev->regs = devm_platform_ioremap_resource(pdev, 0); | ||
| 346 | |||
| 347 | ethosudev->num_clks = devm_clk_bulk_get_all(&pdev->dev, ðosudev->clks); | ||
| 348 | if (ethosudev->num_clks < 0) | ||
| 349 | return ethosudev->num_clks; | ||
| 350 | |||
| 351 | ret = ethosu_job_init(ethosudev); | ||
| 352 | if (ret) | ||
| 353 | return ret; | ||
| 354 | |||
| 355 | ret = ethosu_init(ethosudev); | ||
| 356 | if (ret) | ||
| 357 | return ret; | ||
| 358 | |||
| 359 | ret = drm_dev_register(ðosudev->base, 0); | ||
| 360 | if (ret) | ||
| 361 | pm_runtime_dont_use_autosuspend(ethosudev->base.dev); | ||
| 362 | |||
| 363 | pm_runtime_put_autosuspend(ethosudev->base.dev); | ||
| 364 | return ret; | ||
| 365 | } | ||
| 366 | |||
| 367 | static void ethosu_remove(struct platform_device *pdev) | ||
| 368 | { | ||
| 369 | struct ethosu_device *ethosudev = dev_get_drvdata(&pdev->dev); | ||
| 370 | |||
| 371 | drm_dev_unregister(ðosudev->base); | ||
| 372 | ethosu_job_fini(ethosudev); | ||
| 373 | if (ethosudev->sram) | ||
| 374 | gen_pool_free(ethosudev->srampool, (unsigned long)ethosudev->sram, | ||
| 375 | ethosudev->npu_info.sram_size); | ||
| 376 | } | ||
| 377 | |||
| 378 | static const struct of_device_id dt_match[] = { | ||
| 379 | { .compatible = "arm,ethos-u65" }, | ||
| 380 | { .compatible = "arm,ethos-u85" }, | ||
| 381 | {} | ||
| 382 | }; | ||
| 383 | MODULE_DEVICE_TABLE(of, dt_match); | ||
| 384 | |||
| 385 | static DEFINE_RUNTIME_DEV_PM_OPS(ethosu_pm_ops, | ||
| 386 | ethosu_device_suspend, | ||
| 387 | ethosu_device_resume, | ||
| 388 | NULL); | ||
| 389 | |||
| 390 | static struct platform_driver ethosu_driver = { | ||
| 391 | .probe = ethosu_probe, | ||
| 392 | .remove = ethosu_remove, | ||
| 393 | .driver = { | ||
| 394 | .name = "ethosu", | ||
| 395 | .pm = pm_ptr(ðosu_pm_ops), | ||
| 396 | .of_match_table = dt_match, | ||
| 397 | }, | ||
| 398 | }; | ||
| 399 | module_platform_driver(ethosu_driver); | ||
| 400 | |||
| 401 | MODULE_AUTHOR("Rob Herring <robh@kernel.org>"); | ||
| 402 | MODULE_DESCRIPTION("Arm Ethos-U Accel Driver"); | ||
| 403 | MODULE_LICENSE("Dual MIT/GPL"); | ||
| @@ -0,0 +1,15 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only OR MIT */ | ||
| 2 | /* Copyright 2025 Arm, Ltd. */ | ||
| 3 | #ifndef __ETHOSU_DRV_H__ | ||
| 4 | #define __ETHOSU_DRV_H__ | ||
| 5 | |||
| 6 | #include <drm/gpu_scheduler.h> | ||
| 7 | |||
| 8 | struct ethosu_device; | ||
| 9 | |||
| 10 | struct ethosu_file_priv { | ||
| 11 | struct ethosu_device *edev; | ||
| 12 | struct drm_sched_entity sched_entity; | ||
| 13 | }; | ||
| 14 | |||
| 15 | #endif | ||
| @@ -0,0 +1,704 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only or MIT | ||
| 2 | /* Copyright 2025 Arm, Ltd. */ | ||
| 3 | |||
| 4 | #include <linux/err.h> | ||
| 5 | #include <linux/slab.h> | ||
| 6 | |||
| 7 | #include <drm/ethosu_accel.h> | ||
| 8 | |||
| 9 | #include "ethosu_device.h" | ||
| 10 | #include "ethosu_gem.h" | ||
| 11 | |||
| 12 | static void ethosu_gem_free_object(struct drm_gem_object *obj) | ||
| 13 | { | ||
| 14 | struct ethosu_gem_object *bo = to_ethosu_bo(obj); | ||
| 15 | |||
| 16 | kfree(bo->info); | ||
| 17 | drm_gem_free_mmap_offset(&bo->base.base); | ||
| 18 | drm_gem_dma_free(&bo->base); | ||
| 19 | } | ||
| 20 | |||
| 21 | static int ethosu_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma) | ||
| 22 | { | ||
| 23 | struct ethosu_gem_object *bo = to_ethosu_bo(obj); | ||
| 24 | |||
| 25 | /* Don't allow mmap on objects that have the NO_MMAP flag set. */ | ||
| 26 | if (bo->flags & DRM_ETHOSU_BO_NO_MMAP) | ||
| 27 | return -EINVAL; | ||
| 28 | |||
| 29 | return drm_gem_dma_object_mmap(obj, vma); | ||
| 30 | } | ||
| 31 | |||
| 32 | static const struct drm_gem_object_funcs ethosu_gem_funcs = { | ||
| 33 | .free = ethosu_gem_free_object, | ||
| 34 | .print_info = drm_gem_dma_object_print_info, | ||
| 35 | .get_sg_table = drm_gem_dma_object_get_sg_table, | ||
| 36 | .vmap = drm_gem_dma_object_vmap, | ||
| 37 | .mmap = ethosu_gem_mmap, | ||
| 38 | .vm_ops = &drm_gem_dma_vm_ops, | ||
| 39 | }; | ||
| 40 | |||
| 41 | /** | ||
| 42 | * ethosu_gem_create_object - Implementation of driver->gem_create_object. | ||
| 43 | * @ddev: DRM device | ||
| 44 | * @size: Size in bytes of the memory the object will reference | ||
| 45 | * | ||
| 46 | * This lets the GEM helpers allocate object structs for us, and keep | ||
| 47 | * our BO stats correct. | ||
| 48 | */ | ||
| 49 | struct drm_gem_object *ethosu_gem_create_object(struct drm_device *ddev, size_t size) | ||
| 50 | { | ||
| 51 | struct ethosu_gem_object *obj; | ||
| 52 | |||
| 53 | obj = kzalloc(sizeof(*obj), GFP_KERNEL); | ||
| 54 | if (!obj) | ||
| 55 | return ERR_PTR(-ENOMEM); | ||
| 56 | |||
| 57 | obj->base.base.funcs = ðosu_gem_funcs; | ||
| 58 | return &obj->base.base; | ||
| 59 | } | ||
| 60 | |||
| 61 | /** | ||
| 62 | * ethosu_gem_create_with_handle() - Create a GEM object and attach it to a handle. | ||
| 63 | * @file: DRM file. | ||
| 64 | * @ddev: DRM device. | ||
| 65 | * @size: Size of the GEM object to allocate. | ||
| 66 | * @flags: Combination of drm_ethosu_bo_flags flags. | ||
| 67 | * @handle: Pointer holding the handle pointing to the new GEM object. | ||
| 68 | * | ||
| 69 | * Return: Zero on success | ||
| 70 | */ | ||
| 71 | int ethosu_gem_create_with_handle(struct drm_file *file, | ||
| 72 | struct drm_device *ddev, | ||
| 73 | u64 *size, u32 flags, u32 *handle) | ||
| 74 | { | ||
| 75 | struct drm_gem_dma_object *mem; | ||
| 76 | struct ethosu_gem_object *bo; | ||
| 77 | int ret; | ||
| 78 | |||
| 79 | mem = drm_gem_dma_create(ddev, *size); | ||
| 80 | if (IS_ERR(mem)) | ||
| 81 | return PTR_ERR(mem); | ||
| 82 | |||
| 83 | bo = to_ethosu_bo(&mem->base); | ||
| 84 | bo->flags = flags; | ||
| 85 | |||
| 86 | /* | ||
| 87 | * Allocate an id of idr table where the obj is registered | ||
| 88 | * and handle has the id what user can see. | ||
| 89 | */ | ||
| 90 | ret = drm_gem_handle_create(file, &mem->base, handle); | ||
| 91 | if (!ret) | ||
| 92 | *size = bo->base.base.size; | ||
| 93 | |||
| 94 | /* drop reference from allocate - handle holds it now. */ | ||
| 95 | drm_gem_object_put(&mem->base); | ||
| 96 | |||
| 97 | return ret; | ||
| 98 | } | ||
| 99 | |||
| 100 | struct dma { | ||
| 101 | s8 region; | ||
| 102 | u64 len; | ||
| 103 | u64 offset; | ||
| 104 | s64 stride[2]; | ||
| 105 | }; | ||
| 106 | |||
| 107 | struct dma_state { | ||
| 108 | u16 size0; | ||
| 109 | u16 size1; | ||
| 110 | s8 mode; | ||
| 111 | struct dma src; | ||
| 112 | struct dma dst; | ||
| 113 | }; | ||
| 114 | |||
| 115 | struct buffer { | ||
| 116 | u64 base; | ||
| 117 | u32 length; | ||
| 118 | s8 region; | ||
| 119 | }; | ||
| 120 | |||
| 121 | struct feat_matrix { | ||
| 122 | u64 base[4]; | ||
| 123 | s64 stride_x; | ||
| 124 | s64 stride_y; | ||
| 125 | s64 stride_c; | ||
| 126 | s8 region; | ||
| 127 | u8 broadcast; | ||
| 128 | u16 stride_kernel; | ||
| 129 | u16 precision; | ||
| 130 | u16 depth; | ||
| 131 | u16 width; | ||
| 132 | u16 width0; | ||
| 133 | u16 height[3]; | ||
| 134 | u8 pad_top; | ||
| 135 | u8 pad_left; | ||
| 136 | u8 pad_bottom; | ||
| 137 | u8 pad_right; | ||
| 138 | }; | ||
| 139 | |||
| 140 | struct cmd_state { | ||
| 141 | struct dma_state dma; | ||
| 142 | struct buffer scale[2]; | ||
| 143 | struct buffer weight[4]; | ||
| 144 | struct feat_matrix ofm; | ||
| 145 | struct feat_matrix ifm; | ||
| 146 | struct feat_matrix ifm2; | ||
| 147 | }; | ||
| 148 | |||
| 149 | static void cmd_state_init(struct cmd_state *st) | ||
| 150 | { | ||
| 151 | /* Initialize to all 1s to detect missing setup */ | ||
| 152 | memset(st, 0xff, sizeof(*st)); | ||
| 153 | } | ||
| 154 | |||
| 155 | static u64 cmd_to_addr(u32 *cmd) | ||
| 156 | { | ||
| 157 | return ((u64)((cmd[0] & 0xff0000) << 16)) | cmd[1]; | ||
| 158 | } | ||
| 159 | |||
| 160 | static u64 dma_length(struct ethosu_validated_cmdstream_info *info, | ||
| 161 | struct dma_state *dma_st, struct dma *dma) | ||
| 162 | { | ||
| 163 | s8 mode = dma_st->mode; | ||
| 164 | u64 len = dma->len; | ||
| 165 | |||
| 166 | if (mode >= 1) { | ||
| 167 | len += dma->stride[0]; | ||
| 168 | len *= dma_st->size0; | ||
| 169 | } | ||
| 170 | if (mode == 2) { | ||
| 171 | len += dma->stride[1]; | ||
| 172 | len *= dma_st->size1; | ||
| 173 | } | ||
| 174 | if (dma->region >= 0) | ||
| 175 | info->region_size[dma->region] = max(info->region_size[dma->region], | ||
| 176 | len + dma->offset); | ||
| 177 | |||
| 178 | return len; | ||
| 179 | } | ||
| 180 | |||
| 181 | static u64 feat_matrix_length(struct ethosu_validated_cmdstream_info *info, | ||
| 182 | struct feat_matrix *fm, | ||
| 183 | u32 x, u32 y, u32 c) | ||
| 184 | { | ||
| 185 | u32 element_size, storage = fm->precision >> 14; | ||
| 186 | int tile = 0; | ||
| 187 | u64 addr; | ||
| 188 | |||
| 189 | if (fm->region < 0) | ||
| 190 | return U64_MAX; | ||
| 191 | |||
| 192 | switch (storage) { | ||
| 193 | case 0: | ||
| 194 | if (x >= fm->width0 + 1) { | ||
| 195 | x -= fm->width0 + 1; | ||
| 196 | tile += 1; | ||
| 197 | } | ||
| 198 | if (y >= fm->height[tile] + 1) { | ||
| 199 | y -= fm->height[tile] + 1; | ||
| 200 | tile += 2; | ||
| 201 | } | ||
| 202 | break; | ||
| 203 | case 1: | ||
| 204 | if (y >= fm->height[1] + 1) { | ||
| 205 | y -= fm->height[1] + 1; | ||
| 206 | tile = 2; | ||
| 207 | } else if (y >= fm->height[0] + 1) { | ||
| 208 | y -= fm->height[0] + 1; | ||
| 209 | tile = 1; | ||
| 210 | } | ||
| 211 | break; | ||
| 212 | } | ||
| 213 | if (fm->base[tile] == U64_MAX) | ||
| 214 | return U64_MAX; | ||
| 215 | |||
| 216 | addr = fm->base[tile] + y * fm->stride_y; | ||
| 217 | |||
| 218 | switch ((fm->precision >> 6) & 0x3) { // format | ||
| 219 | case 0: //nhwc: | ||
| 220 | addr += x * fm->stride_x + c; | ||
| 221 | break; | ||
| 222 | case 1: //nhcwb16: | ||
| 223 | element_size = BIT((fm->precision >> 1) & 0x3); | ||
| 224 | |||
| 225 | addr += (c / 16) * fm->stride_c + (16 * x + (c & 0xf)) * element_size; | ||
| 226 | break; | ||
| 227 | } | ||
| 228 | |||
| 229 | info->region_size[fm->region] = max(info->region_size[fm->region], addr + 1); | ||
| 230 | |||
| 231 | return addr; | ||
| 232 | } | ||
| 233 | |||
| 234 | static int calc_sizes(struct drm_device *ddev, | ||
| 235 | struct ethosu_validated_cmdstream_info *info, | ||
| 236 | u16 op, struct cmd_state *st, | ||
| 237 | bool ifm, bool ifm2, bool weight, bool scale) | ||
| 238 | { | ||
| 239 | u64 len; | ||
| 240 | |||
| 241 | if (ifm) { | ||
| 242 | if (st->ifm.stride_kernel == U16_MAX) | ||
| 243 | return -EINVAL; | ||
| 244 | u32 stride_y = ((st->ifm.stride_kernel >> 8) & 0x2) + | ||
| 245 | ((st->ifm.stride_kernel >> 1) & 0x1) + 1; | ||
| 246 | u32 stride_x = ((st->ifm.stride_kernel >> 5) & 0x2) + | ||
| 247 | (st->ifm.stride_kernel & 0x1) + 1; | ||
| 248 | u32 ifm_height = st->ofm.height[2] * stride_y + | ||
| 249 | st->ifm.height[2] - (st->ifm.pad_top + st->ifm.pad_bottom); | ||
| 250 | u32 ifm_width = st->ofm.width * stride_x + | ||
| 251 | st->ifm.width - (st->ifm.pad_left + st->ifm.pad_right); | ||
| 252 | |||
| 253 | len = feat_matrix_length(info, &st->ifm, ifm_width, | ||
| 254 | ifm_height, st->ifm.depth); | ||
| 255 | dev_dbg(ddev->dev, "op %d: IFM:%d:0x%llx-0x%llx\n", | ||
| 256 | op, st->ifm.region, st->ifm.base[0], len); | ||
| 257 | if (len == U64_MAX) | ||
| 258 | return -EINVAL; | ||
| 259 | } | ||
| 260 | |||
| 261 | if (ifm2) { | ||
| 262 | len = feat_matrix_length(info, &st->ifm2, st->ifm.depth, | ||
| 263 | 0, st->ofm.depth); | ||
| 264 | dev_dbg(ddev->dev, "op %d: IFM2:%d:0x%llx-0x%llx\n", | ||
| 265 | op, st->ifm2.region, st->ifm2.base[0], len); | ||
| 266 | if (len == U64_MAX) | ||
| 267 | return -EINVAL; | ||
| 268 | } | ||
| 269 | |||
| 270 | if (weight) { | ||
| 271 | dev_dbg(ddev->dev, "op %d: W:%d:0x%llx-0x%llx\n", | ||
| 272 | op, st->weight[0].region, st->weight[0].base, | ||
| 273 | st->weight[0].base + st->weight[0].length - 1); | ||
| 274 | if (st->weight[0].region < 0 || st->weight[0].base == U64_MAX || | ||
| 275 | st->weight[0].length == U32_MAX) | ||
| 276 | return -EINVAL; | ||
| 277 | info->region_size[st->weight[0].region] = | ||
| 278 | max(info->region_size[st->weight[0].region], | ||
| 279 | st->weight[0].base + st->weight[0].length); | ||
| 280 | } | ||
| 281 | |||
| 282 | if (scale) { | ||
| 283 | dev_dbg(ddev->dev, "op %d: S:%d:0x%llx-0x%llx\n", | ||
| 284 | op, st->scale[0].region, st->scale[0].base, | ||
| 285 | st->scale[0].base + st->scale[0].length - 1); | ||
| 286 | if (st->scale[0].region < 0 || st->scale[0].base == U64_MAX || | ||
| 287 | st->scale[0].length == U32_MAX) | ||
| 288 | return -EINVAL; | ||
| 289 | info->region_size[st->scale[0].region] = | ||
| 290 | max(info->region_size[st->scale[0].region], | ||
| 291 | st->scale[0].base + st->scale[0].length); | ||
| 292 | } | ||
| 293 | |||
| 294 | len = feat_matrix_length(info, &st->ofm, st->ofm.width, | ||
| 295 | st->ofm.height[2], st->ofm.depth); | ||
| 296 | dev_dbg(ddev->dev, "op %d: OFM:%d:0x%llx-0x%llx\n", | ||
| 297 | op, st->ofm.region, st->ofm.base[0], len); | ||
| 298 | if (len == U64_MAX) | ||
| 299 | return -EINVAL; | ||
| 300 | info->output_region[st->ofm.region] = true; | ||
| 301 | |||
| 302 | return 0; | ||
| 303 | } | ||
| 304 | |||
| 305 | static int calc_sizes_elemwise(struct drm_device *ddev, | ||
| 306 | struct ethosu_validated_cmdstream_info *info, | ||
| 307 | u16 op, struct cmd_state *st, | ||
| 308 | bool ifm, bool ifm2) | ||
| 309 | { | ||
| 310 | u32 height, width, depth; | ||
| 311 | u64 len; | ||
| 312 | |||
| 313 | if (ifm) { | ||
| 314 | height = st->ifm.broadcast & 0x1 ? 0 : st->ofm.height[2]; | ||
| 315 | width = st->ifm.broadcast & 0x2 ? 0 : st->ofm.width; | ||
| 316 | depth = st->ifm.broadcast & 0x4 ? 0 : st->ofm.depth; | ||
| 317 | |||
| 318 | len = feat_matrix_length(info, &st->ifm, width, | ||
| 319 | height, depth); | ||
| 320 | dev_dbg(ddev->dev, "op %d: IFM:%d:0x%llx-0x%llx\n", | ||
| 321 | op, st->ifm.region, st->ifm.base[0], len); | ||
| 322 | if (len == U64_MAX) | ||
| 323 | return -EINVAL; | ||
| 324 | } | ||
| 325 | |||
| 326 | if (ifm2) { | ||
| 327 | height = st->ifm2.broadcast & 0x1 ? 0 : st->ofm.height[2]; | ||
| 328 | width = st->ifm2.broadcast & 0x2 ? 0 : st->ofm.width; | ||
| 329 | depth = st->ifm2.broadcast & 0x4 ? 0 : st->ofm.depth; | ||
| 330 | |||
| 331 | len = feat_matrix_length(info, &st->ifm2, width, | ||
| 332 | height, depth); | ||
| 333 | dev_dbg(ddev->dev, "op %d: IFM2:%d:0x%llx-0x%llx\n", | ||
| 334 | op, st->ifm2.region, st->ifm2.base[0], len); | ||
| 335 | if (len == U64_MAX) | ||
| 336 | return -EINVAL; | ||
| 337 | } | ||
| 338 | |||
| 339 | len = feat_matrix_length(info, &st->ofm, st->ofm.width, | ||
| 340 | st->ofm.height[2], st->ofm.depth); | ||
| 341 | dev_dbg(ddev->dev, "op %d: OFM:%d:0x%llx-0x%llx\n", | ||
| 342 | op, st->ofm.region, st->ofm.base[0], len); | ||
| 343 | if (len == U64_MAX) | ||
| 344 | return -EINVAL; | ||
| 345 | info->output_region[st->ofm.region] = true; | ||
| 346 | |||
| 347 | return 0; | ||
| 348 | } | ||
| 349 | |||
| 350 | static int ethosu_gem_cmdstream_copy_and_validate(struct drm_device *ddev, | ||
| 351 | u32 __user *ucmds, | ||
| 352 | struct ethosu_gem_object *bo, | ||
| 353 | u32 size) | ||
| 354 | { | ||
| 355 | struct ethosu_validated_cmdstream_info __free(kfree) *info = kzalloc(sizeof(*info), GFP_KERNEL); | ||
| 356 | struct ethosu_device *edev = to_ethosu_device(ddev); | ||
| 357 | u32 *bocmds = bo->base.vaddr; | ||
| 358 | struct cmd_state st; | ||
| 359 | int i, ret; | ||
| 360 | |||
| 361 | if (!info) | ||
| 362 | return -ENOMEM; | ||
| 363 | info->cmd_size = size; | ||
| 364 | |||
| 365 | cmd_state_init(&st); | ||
| 366 | |||
| 367 | for (i = 0; i < size / 4; i++) { | ||
| 368 | bool use_ifm, use_ifm2, use_scale; | ||
| 369 | u64 dstlen, srclen; | ||
| 370 | u16 cmd, param; | ||
| 371 | u32 cmds[2]; | ||
| 372 | u64 addr; | ||
| 373 | |||
| 374 | if (get_user(cmds[0], ucmds++)) | ||
| 375 | return -EFAULT; | ||
| 376 | |||
| 377 | bocmds[i] = cmds[0]; | ||
| 378 | |||
| 379 | cmd = cmds[0]; | ||
| 380 | param = cmds[0] >> 16; | ||
| 381 | |||
| 382 | if (cmd & 0x4000) { | ||
| 383 | if (get_user(cmds[1], ucmds++)) | ||
| 384 | return -EFAULT; | ||
| 385 | |||
| 386 | i++; | ||
| 387 | bocmds[i] = cmds[1]; | ||
| 388 | addr = cmd_to_addr(cmds); | ||
| 389 | } | ||
| 390 | |||
| 391 | switch (cmd) { | ||
| 392 | case NPU_OP_DMA_START: | ||
| 393 | srclen = dma_length(info, &st.dma, &st.dma.src); | ||
| 394 | dstlen = dma_length(info, &st.dma, &st.dma.dst); | ||
| 395 | |||
| 396 | if (st.dma.dst.region >= 0) | ||
| 397 | info->output_region[st.dma.dst.region] = true; | ||
| 398 | dev_dbg(ddev->dev, "cmd: DMA SRC:%d:0x%llx+0x%llx DST:%d:0x%llx+0x%llx\n", | ||
| 399 | st.dma.src.region, st.dma.src.offset, srclen, | ||
| 400 | st.dma.dst.region, st.dma.dst.offset, dstlen); | ||
| 401 | break; | ||
| 402 | case NPU_OP_CONV: | ||
| 403 | case NPU_OP_DEPTHWISE: | ||
| 404 | use_ifm2 = param & 0x1; // weights_ifm2 | ||
| 405 | use_scale = !(st.ofm.precision & 0x100); | ||
| 406 | ret = calc_sizes(ddev, info, cmd, &st, true, use_ifm2, | ||
| 407 | !use_ifm2, use_scale); | ||
| 408 | if (ret) | ||
| 409 | return ret; | ||
| 410 | break; | ||
| 411 | case NPU_OP_POOL: | ||
| 412 | use_ifm = param != 0x4; // pooling mode | ||
| 413 | use_scale = !(st.ofm.precision & 0x100); | ||
| 414 | ret = calc_sizes(ddev, info, cmd, &st, use_ifm, false, | ||
| 415 | false, use_scale); | ||
| 416 | if (ret) | ||
| 417 | return ret; | ||
| 418 | break; | ||
| 419 | case NPU_OP_ELEMENTWISE: | ||
| 420 | use_ifm2 = !((st.ifm2.broadcast == 8) || (param == 5) || | ||
| 421 | (param == 6) || (param == 7) || (param == 0x24)); | ||
| 422 | use_ifm = st.ifm.broadcast != 8; | ||
| 423 | ret = calc_sizes_elemwise(ddev, info, cmd, &st, use_ifm, use_ifm2); | ||
| 424 | if (ret) | ||
| 425 | return ret; | ||
| 426 | break; | ||
| 427 | case NPU_OP_RESIZE: // U85 only | ||
| 428 | WARN_ON(1); // TODO | ||
| 429 | break; | ||
| 430 | case NPU_SET_KERNEL_WIDTH_M1: | ||
| 431 | st.ifm.width = param; | ||
| 432 | break; | ||
| 433 | case NPU_SET_KERNEL_HEIGHT_M1: | ||
| 434 | st.ifm.height[2] = param; | ||
| 435 | break; | ||
| 436 | case NPU_SET_KERNEL_STRIDE: | ||
| 437 | st.ifm.stride_kernel = param; | ||
| 438 | break; | ||
| 439 | case NPU_SET_IFM_PAD_TOP: | ||
| 440 | st.ifm.pad_top = param & 0x7f; | ||
| 441 | break; | ||
| 442 | case NPU_SET_IFM_PAD_LEFT: | ||
| 443 | st.ifm.pad_left = param & 0x7f; | ||
| 444 | break; | ||
| 445 | case NPU_SET_IFM_PAD_RIGHT: | ||
| 446 | st.ifm.pad_right = param & 0xff; | ||
| 447 | break; | ||
| 448 | case NPU_SET_IFM_PAD_BOTTOM: | ||
| 449 | st.ifm.pad_bottom = param & 0xff; | ||
| 450 | break; | ||
| 451 | case NPU_SET_IFM_DEPTH_M1: | ||
| 452 | st.ifm.depth = param; | ||
| 453 | break; | ||
| 454 | case NPU_SET_IFM_PRECISION: | ||
| 455 | st.ifm.precision = param; | ||
| 456 | break; | ||
| 457 | case NPU_SET_IFM_BROADCAST: | ||
| 458 | st.ifm.broadcast = param; | ||
| 459 | break; | ||
| 460 | case NPU_SET_IFM_REGION: | ||
| 461 | st.ifm.region = param & 0x7f; | ||
| 462 | break; | ||
| 463 | case NPU_SET_IFM_WIDTH0_M1: | ||
| 464 | st.ifm.width0 = param; | ||
| 465 | break; | ||
| 466 | case NPU_SET_IFM_HEIGHT0_M1: | ||
| 467 | st.ifm.height[0] = param; | ||
| 468 | break; | ||
| 469 | case NPU_SET_IFM_HEIGHT1_M1: | ||
| 470 | st.ifm.height[1] = param; | ||
| 471 | break; | ||
| 472 | case NPU_SET_IFM_BASE0: | ||
| 473 | case NPU_SET_IFM_BASE1: | ||
| 474 | case NPU_SET_IFM_BASE2: | ||
| 475 | case NPU_SET_IFM_BASE3: | ||
| 476 | st.ifm.base[cmd & 0x3] = addr; | ||
| 477 | break; | ||
| 478 | case NPU_SET_IFM_STRIDE_X: | ||
| 479 | st.ifm.stride_x = addr; | ||
| 480 | break; | ||
| 481 | case NPU_SET_IFM_STRIDE_Y: | ||
| 482 | st.ifm.stride_y = addr; | ||
| 483 | break; | ||
| 484 | case NPU_SET_IFM_STRIDE_C: | ||
| 485 | st.ifm.stride_c = addr; | ||
| 486 | break; | ||
| 487 | |||
| 488 | case NPU_SET_OFM_WIDTH_M1: | ||
| 489 | st.ofm.width = param; | ||
| 490 | break; | ||
| 491 | case NPU_SET_OFM_HEIGHT_M1: | ||
| 492 | st.ofm.height[2] = param; | ||
| 493 | break; | ||
| 494 | case NPU_SET_OFM_DEPTH_M1: | ||
| 495 | st.ofm.depth = param; | ||
| 496 | break; | ||
| 497 | case NPU_SET_OFM_PRECISION: | ||
| 498 | st.ofm.precision = param; | ||
| 499 | break; | ||
| 500 | case NPU_SET_OFM_REGION: | ||
| 501 | st.ofm.region = param & 0x7; | ||
| 502 | break; | ||
| 503 | case NPU_SET_OFM_WIDTH0_M1: | ||
| 504 | st.ofm.width0 = param; | ||
| 505 | break; | ||
| 506 | case NPU_SET_OFM_HEIGHT0_M1: | ||
| 507 | st.ofm.height[0] = param; | ||
| 508 | break; | ||
| 509 | case NPU_SET_OFM_HEIGHT1_M1: | ||
| 510 | st.ofm.height[1] = param; | ||
| 511 | break; | ||
| 512 | case NPU_SET_OFM_BASE0: | ||
| 513 | case NPU_SET_OFM_BASE1: | ||
| 514 | case NPU_SET_OFM_BASE2: | ||
| 515 | case NPU_SET_OFM_BASE3: | ||
| 516 | st.ofm.base[cmd & 0x3] = addr; | ||
| 517 | break; | ||
| 518 | case NPU_SET_OFM_STRIDE_X: | ||
| 519 | st.ofm.stride_x = addr; | ||
| 520 | break; | ||
| 521 | case NPU_SET_OFM_STRIDE_Y: | ||
| 522 | st.ofm.stride_y = addr; | ||
| 523 | break; | ||
| 524 | case NPU_SET_OFM_STRIDE_C: | ||
| 525 | st.ofm.stride_c = addr; | ||
| 526 | break; | ||
| 527 | |||
| 528 | case NPU_SET_IFM2_BROADCAST: | ||
| 529 | st.ifm2.broadcast = param; | ||
| 530 | break; | ||
| 531 | case NPU_SET_IFM2_PRECISION: | ||
| 532 | st.ifm2.precision = param; | ||
| 533 | break; | ||
| 534 | case NPU_SET_IFM2_REGION: | ||
| 535 | st.ifm2.region = param & 0x7; | ||
| 536 | break; | ||
| 537 | case NPU_SET_IFM2_WIDTH0_M1: | ||
| 538 | st.ifm2.width0 = param; | ||
| 539 | break; | ||
| 540 | case NPU_SET_IFM2_HEIGHT0_M1: | ||
| 541 | st.ifm2.height[0] = param; | ||
| 542 | break; | ||
| 543 | case NPU_SET_IFM2_HEIGHT1_M1: | ||
| 544 | st.ifm2.height[1] = param; | ||
| 545 | break; | ||
| 546 | case NPU_SET_IFM2_BASE0: | ||
| 547 | case NPU_SET_IFM2_BASE1: | ||
| 548 | case NPU_SET_IFM2_BASE2: | ||
| 549 | case NPU_SET_IFM2_BASE3: | ||
| 550 | st.ifm2.base[cmd & 0x3] = addr; | ||
| 551 | break; | ||
| 552 | case NPU_SET_IFM2_STRIDE_X: | ||
| 553 | st.ifm2.stride_x = addr; | ||
| 554 | break; | ||
| 555 | case NPU_SET_IFM2_STRIDE_Y: | ||
| 556 | st.ifm2.stride_y = addr; | ||
| 557 | break; | ||
| 558 | case NPU_SET_IFM2_STRIDE_C: | ||
| 559 | st.ifm2.stride_c = addr; | ||
| 560 | break; | ||
| 561 | |||
| 562 | case NPU_SET_WEIGHT_REGION: | ||
| 563 | st.weight[0].region = param & 0x7; | ||
| 564 | break; | ||
| 565 | case NPU_SET_SCALE_REGION: | ||
| 566 | st.scale[0].region = param & 0x7; | ||
| 567 | break; | ||
| 568 | case NPU_SET_WEIGHT_BASE: | ||
| 569 | st.weight[0].base = addr; | ||
| 570 | break; | ||
| 571 | case NPU_SET_WEIGHT_LENGTH: | ||
| 572 | st.weight[0].length = cmds[1]; | ||
| 573 | break; | ||
| 574 | case NPU_SET_SCALE_BASE: | ||
| 575 | st.scale[0].base = addr; | ||
| 576 | break; | ||
| 577 | case NPU_SET_SCALE_LENGTH: | ||
| 578 | st.scale[0].length = cmds[1]; | ||
| 579 | break; | ||
| 580 | case NPU_SET_WEIGHT1_BASE: | ||
| 581 | st.weight[1].base = addr; | ||
| 582 | break; | ||
| 583 | case NPU_SET_WEIGHT1_LENGTH: | ||
| 584 | st.weight[1].length = cmds[1]; | ||
| 585 | break; | ||
| 586 | case NPU_SET_SCALE1_BASE: // NPU_SET_WEIGHT2_BASE (U85) | ||
| 587 | if (ethosu_is_u65(edev)) | ||
| 588 | st.scale[1].base = addr; | ||
| 589 | else | ||
| 590 | st.weight[2].base = addr; | ||
| 591 | break; | ||
| 592 | case NPU_SET_SCALE1_LENGTH: // NPU_SET_WEIGHT2_LENGTH (U85) | ||
| 593 | if (ethosu_is_u65(edev)) | ||
| 594 | st.scale[1].length = cmds[1]; | ||
| 595 | else | ||
| 596 | st.weight[1].length = cmds[1]; | ||
| 597 | break; | ||
| 598 | case NPU_SET_WEIGHT3_BASE: | ||
| 599 | st.weight[3].base = addr; | ||
| 600 | break; | ||
| 601 | case NPU_SET_WEIGHT3_LENGTH: | ||
| 602 | st.weight[3].length = cmds[1]; | ||
| 603 | break; | ||
| 604 | |||
| 605 | case NPU_SET_DMA0_SRC_REGION: | ||
| 606 | if (param & 0x100) | ||
| 607 | st.dma.src.region = -1; | ||
| 608 | else | ||
| 609 | st.dma.src.region = param & 0x7; | ||
| 610 | st.dma.mode = (param >> 9) & 0x3; | ||
| 611 | break; | ||
| 612 | case NPU_SET_DMA0_DST_REGION: | ||
| 613 | if (param & 0x100) | ||
| 614 | st.dma.dst.region = -1; | ||
| 615 | else | ||
| 616 | st.dma.dst.region = param & 0x7; | ||
| 617 | break; | ||
| 618 | case NPU_SET_DMA0_SIZE0: | ||
| 619 | st.dma.size0 = param; | ||
| 620 | break; | ||
| 621 | case NPU_SET_DMA0_SIZE1: | ||
| 622 | st.dma.size1 = param; | ||
| 623 | break; | ||
| 624 | case NPU_SET_DMA0_SRC_STRIDE0: | ||
| 625 | st.dma.src.stride[0] = ((s64)addr << 24) >> 24; | ||
| 626 | break; | ||
| 627 | case NPU_SET_DMA0_SRC_STRIDE1: | ||
| 628 | st.dma.src.stride[1] = ((s64)addr << 24) >> 24; | ||
| 629 | break; | ||
| 630 | case NPU_SET_DMA0_DST_STRIDE0: | ||
| 631 | st.dma.dst.stride[0] = ((s64)addr << 24) >> 24; | ||
| 632 | break; | ||
| 633 | case NPU_SET_DMA0_DST_STRIDE1: | ||
| 634 | st.dma.dst.stride[1] = ((s64)addr << 24) >> 24; | ||
| 635 | break; | ||
| 636 | case NPU_SET_DMA0_SRC: | ||
| 637 | st.dma.src.offset = addr; | ||
| 638 | break; | ||
| 639 | case NPU_SET_DMA0_DST: | ||
| 640 | st.dma.dst.offset = addr; | ||
| 641 | break; | ||
| 642 | case NPU_SET_DMA0_LEN: | ||
| 643 | st.dma.src.len = st.dma.dst.len = addr; | ||
| 644 | break; | ||
| 645 | default: | ||
| 646 | break; | ||
| 647 | } | ||
| 648 | } | ||
| 649 | |||
| 650 | for (i = 0; i < NPU_BASEP_REGION_MAX; i++) { | ||
| 651 | if (!info->region_size[i]) | ||
| 652 | continue; | ||
| 653 | dev_dbg(ddev->dev, "region %d max size: 0x%llx\n", | ||
| 654 | i, info->region_size[i]); | ||
| 655 | } | ||
| 656 | |||
| 657 | bo->info = no_free_ptr(info); | ||
| 658 | return 0; | ||
| 659 | } | ||
| 660 | |||
| 661 | /** | ||
| 662 | * ethosu_gem_cmdstream_create() - Create a GEM object and attach it to a handle. | ||
| 663 | * @file: DRM file. | ||
| 664 | * @ddev: DRM device. | ||
| 665 | * @exclusive_vm: Exclusive VM. Not NULL if the GEM object can't be shared. | ||
| 666 | * @size: Size of the GEM object to allocate. | ||
| 667 | * @flags: Combination of drm_ethosu_bo_flags flags. | ||
| 668 | * @handle: Pointer holding the handle pointing to the new GEM object. | ||
| 669 | * | ||
| 670 | * Return: Zero on success | ||
| 671 | */ | ||
| 672 | int ethosu_gem_cmdstream_create(struct drm_file *file, | ||
| 673 | struct drm_device *ddev, | ||
| 674 | u32 size, u64 data, u32 flags, u32 *handle) | ||
| 675 | { | ||
| 676 | int ret; | ||
| 677 | struct drm_gem_dma_object *mem; | ||
| 678 | struct ethosu_gem_object *bo; | ||
| 679 | |||
| 680 | mem = drm_gem_dma_create(ddev, size); | ||
| 681 | if (IS_ERR(mem)) | ||
| 682 | return PTR_ERR(mem); | ||
| 683 | |||
| 684 | bo = to_ethosu_bo(&mem->base); | ||
| 685 | bo->flags = flags; | ||
| 686 | |||
| 687 | ret = ethosu_gem_cmdstream_copy_and_validate(ddev, | ||
| 688 | (void __user *)(uintptr_t)data, | ||
| 689 | bo, size); | ||
| 690 | if (ret) | ||
| 691 | goto fail; | ||
| 692 | |||
| 693 | /* | ||
| 694 | * Allocate an id of idr table where the obj is registered | ||
| 695 | * and handle has the id what user can see. | ||
| 696 | */ | ||
| 697 | ret = drm_gem_handle_create(file, &mem->base, handle); | ||
| 698 | |||
| 699 | fail: | ||
| 700 | /* drop reference from allocate - handle holds it now. */ | ||
| 701 | drm_gem_object_put(&mem->base); | ||
| 702 | |||
| 703 | return ret; | ||
| 704 | } | ||
| @@ -0,0 +1,46 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 or MIT */ | ||
| 2 | /* Copyright 2025 Arm, Ltd. */ | ||
| 3 | |||
| 4 | #ifndef __ETHOSU_GEM_H__ | ||
| 5 | #define __ETHOSU_GEM_H__ | ||
| 6 | |||
| 7 | #include "ethosu_device.h" | ||
| 8 | #include <drm/drm_gem_dma_helper.h> | ||
| 9 | |||
| 10 | struct ethosu_validated_cmdstream_info { | ||
| 11 | u32 cmd_size; | ||
| 12 | u64 region_size[NPU_BASEP_REGION_MAX]; | ||
| 13 | bool output_region[NPU_BASEP_REGION_MAX]; | ||
| 14 | }; | ||
| 15 | |||
| 16 | /** | ||
| 17 | * struct ethosu_gem_object - Driver specific GEM object. | ||
| 18 | */ | ||
| 19 | struct ethosu_gem_object { | ||
| 20 | /** @base: Inherit from drm_gem_shmem_object. */ | ||
| 21 | struct drm_gem_dma_object base; | ||
| 22 | |||
| 23 | struct ethosu_validated_cmdstream_info *info; | ||
| 24 | |||
| 25 | /** @flags: Combination of drm_ethosu_bo_flags flags. */ | ||
| 26 | u32 flags; | ||
| 27 | }; | ||
| 28 | |||
| 29 | static inline | ||
| 30 | struct ethosu_gem_object *to_ethosu_bo(struct drm_gem_object *obj) | ||
| 31 | { | ||
| 32 | return container_of(to_drm_gem_dma_obj(obj), struct ethosu_gem_object, base); | ||
| 33 | } | ||
| 34 | |||
| 35 | struct drm_gem_object *ethosu_gem_create_object(struct drm_device *ddev, | ||
| 36 | size_t size); | ||
| 37 | |||
| 38 | int ethosu_gem_create_with_handle(struct drm_file *file, | ||
| 39 | struct drm_device *ddev, | ||
| 40 | u64 *size, u32 flags, uint32_t *handle); | ||
| 41 | |||
| 42 | int ethosu_gem_cmdstream_create(struct drm_file *file, | ||
| 43 | struct drm_device *ddev, | ||
| 44 | u32 size, u64 data, u32 flags, u32 *handle); | ||
| 45 | |||
| 46 | #endif /* __ETHOSU_GEM_H__ */ | ||
| @@ -0,0 +1,497 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only OR MIT | ||
| 2 | /* Copyright 2024-2025 Tomeu Vizoso <tomeu@tomeuvizoso.net> */ | ||
| 3 | /* Copyright 2025 Arm, Ltd. */ | ||
| 4 | |||
| 5 | #include <linux/bitfield.h> | ||
| 6 | #include <linux/genalloc.h> | ||
| 7 | #include <linux/interrupt.h> | ||
| 8 | #include <linux/iopoll.h> | ||
| 9 | #include <linux/platform_device.h> | ||
| 10 | #include <linux/pm_runtime.h> | ||
| 11 | |||
| 12 | #include <drm/drm_file.h> | ||
| 13 | #include <drm/drm_gem.h> | ||
| 14 | #include <drm/drm_gem_dma_helper.h> | ||
| 15 | #include <drm/drm_print.h> | ||
| 16 | #include <drm/ethosu_accel.h> | ||
| 17 | |||
| 18 | #include "ethosu_device.h" | ||
| 19 | #include "ethosu_drv.h" | ||
| 20 | #include "ethosu_gem.h" | ||
| 21 | #include "ethosu_job.h" | ||
| 22 | |||
| 23 | #define JOB_TIMEOUT_MS 500 | ||
| 24 | |||
| 25 | static struct ethosu_job *to_ethosu_job(struct drm_sched_job *sched_job) | ||
| 26 | { | ||
| 27 | return container_of(sched_job, struct ethosu_job, base); | ||
| 28 | } | ||
| 29 | |||
| 30 | static const char *ethosu_fence_get_driver_name(struct dma_fence *fence) | ||
| 31 | { | ||
| 32 | return "ethosu"; | ||
| 33 | } | ||
| 34 | |||
| 35 | static const char *ethosu_fence_get_timeline_name(struct dma_fence *fence) | ||
| 36 | { | ||
| 37 | return "ethosu-npu"; | ||
| 38 | } | ||
| 39 | |||
| 40 | static const struct dma_fence_ops ethosu_fence_ops = { | ||
| 41 | .get_driver_name = ethosu_fence_get_driver_name, | ||
| 42 | .get_timeline_name = ethosu_fence_get_timeline_name, | ||
| 43 | }; | ||
| 44 | |||
| 45 | static void ethosu_job_hw_submit(struct ethosu_device *dev, struct ethosu_job *job) | ||
| 46 | { | ||
| 47 | struct drm_gem_dma_object *cmd_bo = to_drm_gem_dma_obj(job->cmd_bo); | ||
| 48 | struct ethosu_validated_cmdstream_info *cmd_info = to_ethosu_bo(job->cmd_bo)->info; | ||
| 49 | |||
| 50 | for (int i = 0; i < job->region_cnt; i++) { | ||
| 51 | struct drm_gem_dma_object *bo; | ||
| 52 | int region = job->region_bo_num[i]; | ||
| 53 | |||
| 54 | bo = to_drm_gem_dma_obj(job->region_bo[i]); | ||
| 55 | writel_relaxed(lower_32_bits(bo->dma_addr), dev->regs + NPU_REG_BASEP(region)); | ||
| 56 | writel_relaxed(upper_32_bits(bo->dma_addr), dev->regs + NPU_REG_BASEP_HI(region)); | ||
| 57 | dev_dbg(dev->base.dev, "Region %d base addr = %pad\n", region, &bo->dma_addr); | ||
| 58 | } | ||
| 59 | |||
| 60 | if (job->sram_size) { | ||
| 61 | writel_relaxed(lower_32_bits(dev->sramphys), | ||
| 62 | dev->regs + NPU_REG_BASEP(ETHOSU_SRAM_REGION)); | ||
| 63 | writel_relaxed(upper_32_bits(dev->sramphys), | ||
| 64 | dev->regs + NPU_REG_BASEP_HI(ETHOSU_SRAM_REGION)); | ||
| 65 | dev_dbg(dev->base.dev, "Region %d base addr = %pad (SRAM)\n", | ||
| 66 | ETHOSU_SRAM_REGION, &dev->sramphys); | ||
| 67 | } | ||
| 68 | |||
| 69 | writel_relaxed(lower_32_bits(cmd_bo->dma_addr), dev->regs + NPU_REG_QBASE); | ||
| 70 | writel_relaxed(upper_32_bits(cmd_bo->dma_addr), dev->regs + NPU_REG_QBASE_HI); | ||
| 71 | writel_relaxed(cmd_info->cmd_size, dev->regs + NPU_REG_QSIZE); | ||
| 72 | |||
| 73 | writel(CMD_TRANSITION_TO_RUN, dev->regs + NPU_REG_CMD); | ||
| 74 | |||
| 75 | dev_dbg(dev->base.dev, | ||
| 76 | "Submitted cmd at %pad to core\n", &cmd_bo->dma_addr); | ||
| 77 | } | ||
| 78 | |||
| 79 | static int ethosu_acquire_object_fences(struct ethosu_job *job) | ||
| 80 | { | ||
| 81 | int i, ret; | ||
| 82 | struct drm_gem_object **bos = job->region_bo; | ||
| 83 | struct ethosu_validated_cmdstream_info *info = to_ethosu_bo(job->cmd_bo)->info; | ||
| 84 | |||
| 85 | for (i = 0; i < job->region_cnt; i++) { | ||
| 86 | bool is_write; | ||
| 87 | |||
| 88 | if (!bos[i]) | ||
| 89 | break; | ||
| 90 | |||
| 91 | ret = dma_resv_reserve_fences(bos[i]->resv, 1); | ||
| 92 | if (ret) | ||
| 93 | return ret; | ||
| 94 | |||
| 95 | is_write = info->output_region[job->region_bo_num[i]]; | ||
| 96 | ret = drm_sched_job_add_implicit_dependencies(&job->base, bos[i], | ||
| 97 | is_write); | ||
| 98 | if (ret) | ||
| 99 | return ret; | ||
| 100 | } | ||
| 101 | |||
| 102 | return 0; | ||
| 103 | } | ||
| 104 | |||
| 105 | static void ethosu_attach_object_fences(struct ethosu_job *job) | ||
| 106 | { | ||
| 107 | int i; | ||
| 108 | struct dma_fence *fence = job->inference_done_fence; | ||
| 109 | struct drm_gem_object **bos = job->region_bo; | ||
| 110 | struct ethosu_validated_cmdstream_info *info = to_ethosu_bo(job->cmd_bo)->info; | ||
| 111 | |||
| 112 | for (i = 0; i < job->region_cnt; i++) | ||
| 113 | if (info->output_region[job->region_bo_num[i]]) | ||
| 114 | dma_resv_add_fence(bos[i]->resv, fence, DMA_RESV_USAGE_WRITE); | ||
| 115 | } | ||
| 116 | |||
| 117 | static int ethosu_job_push(struct ethosu_job *job) | ||
| 118 | { | ||
| 119 | struct ww_acquire_ctx acquire_ctx; | ||
| 120 | int ret; | ||
| 121 | |||
| 122 | ret = drm_gem_lock_reservations(job->region_bo, job->region_cnt, &acquire_ctx); | ||
| 123 | if (ret) | ||
| 124 | return ret; | ||
| 125 | |||
| 126 | ret = ethosu_acquire_object_fences(job); | ||
| 127 | if (ret) | ||
| 128 | goto out; | ||
| 129 | |||
| 130 | ret = pm_runtime_resume_and_get(job->dev->base.dev); | ||
| 131 | if (!ret) { | ||
| 132 | guard(mutex)(&job->dev->sched_lock); | ||
| 133 | |||
| 134 | drm_sched_job_arm(&job->base); | ||
| 135 | job->inference_done_fence = dma_fence_get(&job->base.s_fence->finished); | ||
| 136 | kref_get(&job->refcount); /* put by scheduler job completion */ | ||
| 137 | drm_sched_entity_push_job(&job->base); | ||
| 138 | ethosu_attach_object_fences(job); | ||
| 139 | } | ||
| 140 | |||
| 141 | out: | ||
| 142 | drm_gem_unlock_reservations(job->region_bo, job->region_cnt, &acquire_ctx); | ||
| 143 | return ret; | ||
| 144 | } | ||
| 145 | |||
| 146 | static void ethosu_job_cleanup(struct kref *ref) | ||
| 147 | { | ||
| 148 | struct ethosu_job *job = container_of(ref, struct ethosu_job, | ||
| 149 | refcount); | ||
| 150 | unsigned int i; | ||
| 151 | |||
| 152 | pm_runtime_put_autosuspend(job->dev->base.dev); | ||
| 153 | |||
| 154 | dma_fence_put(job->done_fence); | ||
| 155 | dma_fence_put(job->inference_done_fence); | ||
| 156 | |||
| 157 | for (i = 0; i < job->region_cnt; i++) | ||
| 158 | drm_gem_object_put(job->region_bo[i]); | ||
| 159 | |||
| 160 | drm_gem_object_put(job->cmd_bo); | ||
| 161 | |||
| 162 | kfree(job); | ||
| 163 | } | ||
| 164 | |||
| 165 | static void ethosu_job_put(struct ethosu_job *job) | ||
| 166 | { | ||
| 167 | kref_put(&job->refcount, ethosu_job_cleanup); | ||
| 168 | } | ||
| 169 | |||
| 170 | static void ethosu_job_free(struct drm_sched_job *sched_job) | ||
| 171 | { | ||
| 172 | struct ethosu_job *job = to_ethosu_job(sched_job); | ||
| 173 | |||
| 174 | drm_sched_job_cleanup(sched_job); | ||
| 175 | ethosu_job_put(job); | ||
| 176 | } | ||
| 177 | |||
| 178 | static struct dma_fence *ethosu_job_run(struct drm_sched_job *sched_job) | ||
| 179 | { | ||
| 180 | struct ethosu_job *job = to_ethosu_job(sched_job); | ||
| 181 | struct ethosu_device *dev = job->dev; | ||
| 182 | struct dma_fence *fence = job->done_fence; | ||
| 183 | |||
| 184 | if (unlikely(job->base.s_fence->finished.error)) | ||
| 185 | return NULL; | ||
| 186 | |||
| 187 | dma_fence_init(fence, ðosu_fence_ops, &dev->fence_lock, | ||
| 188 | dev->fence_context, ++dev->emit_seqno); | ||
| 189 | dma_fence_get(fence); | ||
| 190 | |||
| 191 | scoped_guard(mutex, &dev->job_lock) { | ||
| 192 | dev->in_flight_job = job; | ||
| 193 | ethosu_job_hw_submit(dev, job); | ||
| 194 | } | ||
| 195 | |||
| 196 | return fence; | ||
| 197 | } | ||
| 198 | |||
| 199 | static void ethosu_job_handle_irq(struct ethosu_device *dev) | ||
| 200 | { | ||
| 201 | u32 status = readl_relaxed(dev->regs + NPU_REG_STATUS); | ||
| 202 | |||
| 203 | if (status & (STATUS_BUS_STATUS | STATUS_CMD_PARSE_ERR)) { | ||
| 204 | dev_err(dev->base.dev, "Error IRQ - %x\n", status); | ||
| 205 | drm_sched_fault(&dev->sched); | ||
| 206 | return; | ||
| 207 | } | ||
| 208 | |||
| 209 | scoped_guard(mutex, &dev->job_lock) { | ||
| 210 | if (dev->in_flight_job) { | ||
| 211 | dma_fence_signal(dev->in_flight_job->done_fence); | ||
| 212 | dev->in_flight_job = NULL; | ||
| 213 | } | ||
| 214 | } | ||
| 215 | } | ||
| 216 | |||
| 217 | static irqreturn_t ethosu_job_irq_handler_thread(int irq, void *data) | ||
| 218 | { | ||
| 219 | struct ethosu_device *dev = data; | ||
| 220 | |||
| 221 | ethosu_job_handle_irq(dev); | ||
| 222 | |||
| 223 | return IRQ_HANDLED; | ||
| 224 | } | ||
| 225 | |||
| 226 | static irqreturn_t ethosu_job_irq_handler(int irq, void *data) | ||
| 227 | { | ||
| 228 | struct ethosu_device *dev = data; | ||
| 229 | u32 status = readl_relaxed(dev->regs + NPU_REG_STATUS); | ||
| 230 | |||
| 231 | if (!(status & STATUS_IRQ_RAISED)) | ||
| 232 | return IRQ_NONE; | ||
| 233 | |||
| 234 | writel_relaxed(CMD_CLEAR_IRQ, dev->regs + NPU_REG_CMD); | ||
| 235 | return IRQ_WAKE_THREAD; | ||
| 236 | } | ||
| 237 | |||
| 238 | static enum drm_gpu_sched_stat ethosu_job_timedout(struct drm_sched_job *bad) | ||
| 239 | { | ||
| 240 | struct ethosu_job *job = to_ethosu_job(bad); | ||
| 241 | struct ethosu_device *dev = job->dev; | ||
| 242 | bool running; | ||
| 243 | u32 *bocmds = to_drm_gem_dma_obj(job->cmd_bo)->vaddr; | ||
| 244 | u32 cmdaddr; | ||
| 245 | |||
| 246 | cmdaddr = readl_relaxed(dev->regs + NPU_REG_QREAD); | ||
| 247 | running = FIELD_GET(STATUS_STATE_RUNNING, readl_relaxed(dev->regs + NPU_REG_STATUS)); | ||
| 248 | |||
| 249 | if (running) { | ||
| 250 | int ret; | ||
| 251 | u32 reg; | ||
| 252 | |||
| 253 | ret = readl_relaxed_poll_timeout(dev->regs + NPU_REG_QREAD, | ||
| 254 | reg, | ||
| 255 | reg != cmdaddr, | ||
| 256 | USEC_PER_MSEC, 100 * USEC_PER_MSEC); | ||
| 257 | |||
| 258 | /* If still running and progress is being made, just return */ | ||
| 259 | if (!ret) | ||
| 260 | return DRM_GPU_SCHED_STAT_NO_HANG; | ||
| 261 | } | ||
| 262 | |||
| 263 | dev_err(dev->base.dev, "NPU sched timed out: NPU %s, cmdstream offset 0x%x: 0x%x\n", | ||
| 264 | running ? "running" : "stopped", | ||
| 265 | cmdaddr, bocmds[cmdaddr / 4]); | ||
| 266 | |||
| 267 | drm_sched_stop(&dev->sched, bad); | ||
| 268 | |||
| 269 | scoped_guard(mutex, &dev->job_lock) | ||
| 270 | dev->in_flight_job = NULL; | ||
| 271 | |||
| 272 | /* Proceed with reset now. */ | ||
| 273 | pm_runtime_force_suspend(dev->base.dev); | ||
| 274 | pm_runtime_force_resume(dev->base.dev); | ||
| 275 | |||
| 276 | /* Restart the scheduler */ | ||
| 277 | drm_sched_start(&dev->sched, 0); | ||
| 278 | |||
| 279 | return DRM_GPU_SCHED_STAT_RESET; | ||
| 280 | } | ||
| 281 | |||
| 282 | static const struct drm_sched_backend_ops ethosu_sched_ops = { | ||
| 283 | .run_job = ethosu_job_run, | ||
| 284 | .timedout_job = ethosu_job_timedout, | ||
| 285 | .free_job = ethosu_job_free | ||
| 286 | }; | ||
| 287 | |||
| 288 | int ethosu_job_init(struct ethosu_device *edev) | ||
| 289 | { | ||
| 290 | struct device *dev = edev->base.dev; | ||
| 291 | struct drm_sched_init_args args = { | ||
| 292 | .ops = ðosu_sched_ops, | ||
| 293 | .num_rqs = DRM_SCHED_PRIORITY_COUNT, | ||
| 294 | .credit_limit = 1, | ||
| 295 | .timeout = msecs_to_jiffies(JOB_TIMEOUT_MS), | ||
| 296 | .name = dev_name(dev), | ||
| 297 | .dev = dev, | ||
| 298 | }; | ||
| 299 | int ret; | ||
| 300 | |||
| 301 | spin_lock_init(&edev->fence_lock); | ||
| 302 | ret = devm_mutex_init(dev, &edev->job_lock); | ||
| 303 | if (ret) | ||
| 304 | return ret; | ||
| 305 | ret = devm_mutex_init(dev, &edev->sched_lock); | ||
| 306 | if (ret) | ||
| 307 | return ret; | ||
| 308 | |||
| 309 | edev->irq = platform_get_irq(to_platform_device(dev), 0); | ||
| 310 | if (edev->irq < 0) | ||
| 311 | return edev->irq; | ||
| 312 | |||
| 313 | ret = devm_request_threaded_irq(dev, edev->irq, | ||
| 314 | ethosu_job_irq_handler, | ||
| 315 | ethosu_job_irq_handler_thread, | ||
| 316 | IRQF_SHARED, KBUILD_MODNAME, | ||
| 317 | edev); | ||
| 318 | if (ret) { | ||
| 319 | dev_err(dev, "failed to request irq\n"); | ||
| 320 | return ret; | ||
| 321 | } | ||
| 322 | |||
| 323 | edev->fence_context = dma_fence_context_alloc(1); | ||
| 324 | |||
| 325 | ret = drm_sched_init(&edev->sched, &args); | ||
| 326 | if (ret) { | ||
| 327 | dev_err(dev, "Failed to create scheduler: %d\n", ret); | ||
| 328 | goto err_sched; | ||
| 329 | } | ||
| 330 | |||
| 331 | return 0; | ||
| 332 | |||
| 333 | err_sched: | ||
| 334 | drm_sched_fini(&edev->sched); | ||
| 335 | return ret; | ||
| 336 | } | ||
| 337 | |||
| 338 | void ethosu_job_fini(struct ethosu_device *dev) | ||
| 339 | { | ||
| 340 | drm_sched_fini(&dev->sched); | ||
| 341 | } | ||
| 342 | |||
| 343 | int ethosu_job_open(struct ethosu_file_priv *ethosu_priv) | ||
| 344 | { | ||
| 345 | struct ethosu_device *dev = ethosu_priv->edev; | ||
| 346 | struct drm_gpu_scheduler *sched = &dev->sched; | ||
| 347 | int ret; | ||
| 348 | |||
| 349 | ret = drm_sched_entity_init(ðosu_priv->sched_entity, | ||
| 350 | DRM_SCHED_PRIORITY_NORMAL, | ||
| 351 | &sched, 1, NULL); | ||
| 352 | return WARN_ON(ret); | ||
| 353 | } | ||
| 354 | |||
| 355 | void ethosu_job_close(struct ethosu_file_priv *ethosu_priv) | ||
| 356 | { | ||
| 357 | struct drm_sched_entity *entity = ðosu_priv->sched_entity; | ||
| 358 | |||
| 359 | drm_sched_entity_destroy(entity); | ||
| 360 | } | ||
| 361 | |||
| 362 | static int ethosu_ioctl_submit_job(struct drm_device *dev, struct drm_file *file, | ||
| 363 | struct drm_ethosu_job *job) | ||
| 364 | { | ||
| 365 | struct ethosu_device *edev = to_ethosu_device(dev); | ||
| 366 | struct ethosu_file_priv *file_priv = file->driver_priv; | ||
| 367 | struct ethosu_job *ejob = NULL; | ||
| 368 | struct ethosu_validated_cmdstream_info *cmd_info; | ||
| 369 | int ret = 0; | ||
| 370 | |||
| 371 | /* BO region 2 is reserved if SRAM is used */ | ||
| 372 | if (job->region_bo_handles[ETHOSU_SRAM_REGION] && job->sram_size) | ||
| 373 | return -EINVAL; | ||
| 374 | |||
| 375 | if (edev->npu_info.sram_size < job->sram_size) | ||
| 376 | return -EINVAL; | ||
| 377 | |||
| 378 | ejob = kzalloc(sizeof(*ejob), GFP_KERNEL); | ||
| 379 | if (!ejob) | ||
| 380 | return -ENOMEM; | ||
| 381 | |||
| 382 | kref_init(&ejob->refcount); | ||
| 383 | |||
| 384 | ejob->dev = edev; | ||
| 385 | ejob->sram_size = job->sram_size; | ||
| 386 | |||
| 387 | ejob->done_fence = kzalloc(sizeof(*ejob->done_fence), GFP_KERNEL); | ||
| 388 | if (!ejob->done_fence) { | ||
| 389 | ret = -ENOMEM; | ||
| 390 | goto out_cleanup_job; | ||
| 391 | } | ||
| 392 | |||
| 393 | ret = drm_sched_job_init(&ejob->base, | ||
| 394 | &file_priv->sched_entity, | ||
| 395 | 1, NULL, file->client_id); | ||
| 396 | if (ret) | ||
| 397 | goto out_put_job; | ||
| 398 | |||
| 399 | ejob->cmd_bo = drm_gem_object_lookup(file, job->cmd_bo); | ||
| 400 | if (!ejob->cmd_bo) { | ||
| 401 | ret = -ENOENT; | ||
| 402 | goto out_cleanup_job; | ||
| 403 | } | ||
| 404 | cmd_info = to_ethosu_bo(ejob->cmd_bo)->info; | ||
| 405 | if (!cmd_info) { | ||
| 406 | ret = -EINVAL; | ||
| 407 | goto out_cleanup_job; | ||
| 408 | } | ||
| 409 | |||
| 410 | for (int i = 0; i < NPU_BASEP_REGION_MAX; i++) { | ||
| 411 | struct drm_gem_object *gem; | ||
| 412 | |||
| 413 | /* Can only omit a BO handle if the region is not used or used for SRAM */ | ||
| 414 | if (!job->region_bo_handles[i] && | ||
| 415 | (!cmd_info->region_size[i] || (i == ETHOSU_SRAM_REGION && job->sram_size))) | ||
| 416 | continue; | ||
| 417 | |||
| 418 | if (job->region_bo_handles[i] && !cmd_info->region_size[i]) { | ||
| 419 | dev_err(dev->dev, | ||
| 420 | "Cmdstream BO handle %d set for unused region %d\n", | ||
| 421 | job->region_bo_handles[i], i); | ||
| 422 | ret = -EINVAL; | ||
| 423 | goto out_cleanup_job; | ||
| 424 | } | ||
| 425 | |||
| 426 | gem = drm_gem_object_lookup(file, job->region_bo_handles[i]); | ||
| 427 | if (!gem) { | ||
| 428 | dev_err(dev->dev, | ||
| 429 | "Invalid BO handle %d for region %d\n", | ||
| 430 | job->region_bo_handles[i], i); | ||
| 431 | ret = -ENOENT; | ||
| 432 | goto out_cleanup_job; | ||
| 433 | } | ||
| 434 | |||
| 435 | ejob->region_bo[ejob->region_cnt] = gem; | ||
| 436 | ejob->region_bo_num[ejob->region_cnt] = i; | ||
| 437 | ejob->region_cnt++; | ||
| 438 | |||
| 439 | if (to_ethosu_bo(gem)->info) { | ||
| 440 | dev_err(dev->dev, | ||
| 441 | "Cmdstream BO handle %d used for region %d\n", | ||
| 442 | job->region_bo_handles[i], i); | ||
| 443 | ret = -EINVAL; | ||
| 444 | goto out_cleanup_job; | ||
| 445 | } | ||
| 446 | |||
| 447 | /* Verify the command stream doesn't have accesses outside the BO */ | ||
| 448 | if (cmd_info->region_size[i] > gem->size) { | ||
| 449 | dev_err(dev->dev, | ||
| 450 | "cmd stream region %d size greater than BO size (%llu > %zu)\n", | ||
| 451 | i, cmd_info->region_size[i], gem->size); | ||
| 452 | ret = -EOVERFLOW; | ||
| 453 | goto out_cleanup_job; | ||
| 454 | } | ||
| 455 | } | ||
| 456 | ret = ethosu_job_push(ejob); | ||
| 457 | |||
| 458 | out_cleanup_job: | ||
| 459 | if (ret) | ||
| 460 | drm_sched_job_cleanup(&ejob->base); | ||
| 461 | out_put_job: | ||
| 462 | ethosu_job_put(ejob); | ||
| 463 | |||
| 464 | return ret; | ||
| 465 | } | ||
| 466 | |||
| 467 | int ethosu_ioctl_submit(struct drm_device *dev, void *data, struct drm_file *file) | ||
| 468 | { | ||
| 469 | struct drm_ethosu_submit *args = data; | ||
| 470 | int ret = 0; | ||
| 471 | unsigned int i = 0; | ||
| 472 | |||
| 473 | if (args->pad) { | ||
| 474 | drm_dbg(dev, "Reserved field in drm_ethosu_submit struct should be 0.\n"); | ||
| 475 | return -EINVAL; | ||
| 476 | } | ||
| 477 | |||
| 478 | struct drm_ethosu_job __free(kvfree) *jobs = | ||
| 479 | kvmalloc_array(args->job_count, sizeof(*jobs), GFP_KERNEL); | ||
| 480 | if (!jobs) | ||
| 481 | return -ENOMEM; | ||
| 482 | |||
| 483 | if (copy_from_user(jobs, | ||
| 484 | (void __user *)(uintptr_t)args->jobs, | ||
| 485 | args->job_count * sizeof(*jobs))) { | ||
| 486 | drm_dbg(dev, "Failed to copy incoming job array\n"); | ||
| 487 | return -EFAULT; | ||
| 488 | } | ||
| 489 | |||
| 490 | for (i = 0; i < args->job_count; i++) { | ||
| 491 | ret = ethosu_ioctl_submit_job(dev, file, &jobs[i]); | ||
| 492 | if (ret) | ||
| 493 | return ret; | ||
| 494 | } | ||
| 495 | |||
| 496 | return 0; | ||
| 497 | } | ||
| @@ -0,0 +1,40 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only OR MIT */ | ||
| 2 | /* Copyright 2024-2025 Tomeu Vizoso <tomeu@tomeuvizoso.net> */ | ||
| 3 | /* Copyright 2025 Arm, Ltd. */ | ||
| 4 | |||
| 5 | #ifndef __ETHOSU_JOB_H__ | ||
| 6 | #define __ETHOSU_JOB_H__ | ||
| 7 | |||
| 8 | #include <linux/kref.h> | ||
| 9 | #include <drm/gpu_scheduler.h> | ||
| 10 | |||
| 11 | struct ethosu_device; | ||
| 12 | struct ethosu_file_priv; | ||
| 13 | |||
| 14 | struct ethosu_job { | ||
| 15 | struct drm_sched_job base; | ||
| 16 | struct ethosu_device *dev; | ||
| 17 | |||
| 18 | struct drm_gem_object *cmd_bo; | ||
| 19 | struct drm_gem_object *region_bo[NPU_BASEP_REGION_MAX]; | ||
| 20 | u8 region_bo_num[NPU_BASEP_REGION_MAX]; | ||
| 21 | u8 region_cnt; | ||
| 22 | u32 sram_size; | ||
| 23 | |||
| 24 | /* Fence to be signaled by drm-sched once its done with the job */ | ||
| 25 | struct dma_fence *inference_done_fence; | ||
| 26 | |||
| 27 | /* Fence to be signaled by IRQ handler when the job is complete. */ | ||
| 28 | struct dma_fence *done_fence; | ||
| 29 | |||
| 30 | struct kref refcount; | ||
| 31 | }; | ||
| 32 | |||
| 33 | int ethosu_ioctl_submit(struct drm_device *dev, void *data, struct drm_file *file); | ||
| 34 | |||
| 35 | int ethosu_job_init(struct ethosu_device *dev); | ||
| 36 | void ethosu_job_fini(struct ethosu_device *dev); | ||
| 37 | int ethosu_job_open(struct ethosu_file_priv *ethosu_priv); | ||
| 38 | void ethosu_job_close(struct ethosu_file_priv *ethosu_priv); | ||
| 39 | |||
| 40 | #endif | ||
| @@ -6,6 +6,7 @@ intel_vpu-y := \ · 5 unmodified lines | |||
| 6 | ivpu_fw.o \ | 6 | ivpu_fw.o \ |
| 7 | ivpu_fw_log.o \ | 7 | ivpu_fw_log.o \ |
| 8 | ivpu_gem.o \ | 8 | ivpu_gem.o \ |
| 9 | ivpu_gem_userptr.o \ | ||
| 9 | ivpu_hw.o \ | 10 | ivpu_hw.o \ |
| 10 | ivpu_hw_btrs.o \ | 11 | ivpu_hw_btrs.o \ |
| 11 | ivpu_hw_ip.o \ | 12 | ivpu_hw_ip.o \ |
| @@ -398,35 +398,25 @@ static int dct_active_set(void *data, u64 active_percent) · 397 unmodified lines | |||
| 398 | 398 | ||
| 399 | DEFINE_DEBUGFS_ATTRIBUTE(ivpu_dct_fops, dct_active_get, dct_active_set, "%llu\n"); | 399 | DEFINE_DEBUGFS_ATTRIBUTE(ivpu_dct_fops, dct_active_get, dct_active_set, "%llu\n"); |
| 400 | 400 | ||
| 401 | static void print_priority_band(struct seq_file *s, struct ivpu_hw_info *hw, | ||
| 402 | int band, const char *name) | ||
| 403 | { | ||
| 404 | seq_printf(s, "%-9s: grace_period %9u process_grace_period %9u process_quantum %9u\n", | ||
| 405 | name, | ||
| 406 | hw->hws.grace_period[band], | ||
| 407 | hw->hws.process_grace_period[band], | ||
| 408 | hw->hws.process_quantum[band]); | ||
| 409 | } | ||
| 410 | |||
| 401 | static int priority_bands_show(struct seq_file *s, void *v) | 411 | static int priority_bands_show(struct seq_file *s, void *v) |
| 402 | { | 412 | { |
| 403 | struct ivpu_device *vdev = s->private; | 413 | struct ivpu_device *vdev = s->private; |
| 404 | struct ivpu_hw_info *hw = vdev->hw; | 414 | struct ivpu_hw_info *hw = vdev->hw; |
| 405 | 415 | ||
| 406 | for (int band = VPU_JOB_SCHEDULING_PRIORITY_BAND_IDLE; | 416 | print_priority_band(s, hw, VPU_JOB_SCHEDULING_PRIORITY_BAND_IDLE, "Idle"); |
| 407 | band < VPU_JOB_SCHEDULING_PRIORITY_BAND_COUNT; band++) { | 417 | print_priority_band(s, hw, VPU_JOB_SCHEDULING_PRIORITY_BAND_NORMAL, "Normal"); |
| 408 | switch (band) { | 418 | print_priority_band(s, hw, VPU_JOB_SCHEDULING_PRIORITY_BAND_FOCUS, "Focus"); |
| 409 | case VPU_JOB_SCHEDULING_PRIORITY_BAND_IDLE: | 419 | print_priority_band(s, hw, VPU_JOB_SCHEDULING_PRIORITY_BAND_REALTIME, "Realtime"); |
| 410 | seq_puts(s, "Idle: "); | ||
| 411 | break; | ||
| 412 | |||
| 413 | case VPU_JOB_SCHEDULING_PRIORITY_BAND_NORMAL: | ||
| 414 | seq_puts(s, "Normal: "); | ||
| 415 | break; | ||
| 416 | |||
| 417 | case VPU_JOB_SCHEDULING_PRIORITY_BAND_FOCUS: | ||
| 418 | seq_puts(s, "Focus: "); | ||
| 419 | break; | ||
| 420 | |||
| 421 | case VPU_JOB_SCHEDULING_PRIORITY_BAND_REALTIME: | ||
| 422 | seq_puts(s, "Realtime: "); | ||
| 423 | break; | ||
| 424 | } | ||
| 425 | |||
| 426 | seq_printf(s, "grace_period %9u process_grace_period %9u process_quantum %9u\n", | ||
| 427 | hw->hws.grace_period[band], hw->hws.process_grace_period[band], | ||
| 428 | hw->hws.process_quantum[band]); | ||
| 429 | } | ||
| 430 | 420 | ||
| 431 | return 0; | 421 | return 0; |
| 432 | } | 422 | } |
| @@ -57,7 +57,7 @@ MODULE_PARM_DESC(pll_max_ratio, "Maximum PLL ratio used to set NPU frequency"); · 56 unmodified lines | |||
| 57 | 57 | ||
| 58 | int ivpu_sched_mode = IVPU_SCHED_MODE_AUTO; | 58 | int ivpu_sched_mode = IVPU_SCHED_MODE_AUTO; |
| 59 | module_param_named(sched_mode, ivpu_sched_mode, int, 0444); | 59 | module_param_named(sched_mode, ivpu_sched_mode, int, 0444); |
| 60 | MODULE_PARM_DESC(sched_mode, "Scheduler mode: -1 - Use default scheduler, 0 - Use OS scheduler, 1 - Use HW scheduler"); | 60 | MODULE_PARM_DESC(sched_mode, "Scheduler mode: -1 - Use default scheduler, 0 - Use OS scheduler (supported on 27XX - 50XX), 1 - Use HW scheduler"); |
| 61 | 61 | ||
| 62 | bool ivpu_disable_mmu_cont_pages; | 62 | bool ivpu_disable_mmu_cont_pages; |
| 63 | module_param_named(disable_mmu_cont_pages, ivpu_disable_mmu_cont_pages, bool, 0444); | 63 | module_param_named(disable_mmu_cont_pages, ivpu_disable_mmu_cont_pages, bool, 0444); |
| @@ -134,6 +134,8 @@ bool ivpu_is_capable(struct ivpu_device *vdev, u32 capability) · 70 unmodified lines | |||
| 134 | return true; | 134 | return true; |
| 135 | case DRM_IVPU_CAP_DMA_MEMORY_RANGE: | 135 | case DRM_IVPU_CAP_DMA_MEMORY_RANGE: |
| 136 | return true; | 136 | return true; |
| 137 | case DRM_IVPU_CAP_BO_CREATE_FROM_USERPTR: | ||
| 138 | return true; | ||
| 137 | case DRM_IVPU_CAP_MANAGE_CMDQ: | 139 | case DRM_IVPU_CAP_MANAGE_CMDQ: |
| 138 | return vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW; | 140 | return vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW; |
| 139 | default: | 141 | default: |
| @@ -200,6 +202,9 @@ static int ivpu_get_param_ioctl(struct drm_device *dev, void *data, struct drm_f · 60 unmodified lines | |||
| 200 | case DRM_IVPU_PARAM_CAPABILITIES: | 202 | case DRM_IVPU_PARAM_CAPABILITIES: |
| 201 | args->value = ivpu_is_capable(vdev, args->index); | 203 | args->value = ivpu_is_capable(vdev, args->index); |
| 202 | break; | 204 | break; |
| 205 | case DRM_IVPU_PARAM_PREEMPT_BUFFER_SIZE: | ||
| 206 | args->value = ivpu_fw_preempt_buf_size(vdev); | ||
| 207 | break; | ||
| 203 | default: | 208 | default: |
| 204 | ret = -EINVAL; | 209 | ret = -EINVAL; |
| 205 | break; | 210 | break; |
| @@ -310,6 +315,7 @@ static const struct drm_ioctl_desc ivpu_drm_ioctls[] = { · 104 unmodified lines | |||
| 310 | DRM_IOCTL_DEF_DRV(IVPU_CMDQ_CREATE, ivpu_cmdq_create_ioctl, 0), | 315 | DRM_IOCTL_DEF_DRV(IVPU_CMDQ_CREATE, ivpu_cmdq_create_ioctl, 0), |
| 311 | DRM_IOCTL_DEF_DRV(IVPU_CMDQ_DESTROY, ivpu_cmdq_destroy_ioctl, 0), | 316 | DRM_IOCTL_DEF_DRV(IVPU_CMDQ_DESTROY, ivpu_cmdq_destroy_ioctl, 0), |
| 312 | DRM_IOCTL_DEF_DRV(IVPU_CMDQ_SUBMIT, ivpu_cmdq_submit_ioctl, 0), | 317 | DRM_IOCTL_DEF_DRV(IVPU_CMDQ_SUBMIT, ivpu_cmdq_submit_ioctl, 0), |
| 318 | DRM_IOCTL_DEF_DRV(IVPU_BO_CREATE_FROM_USERPTR, ivpu_bo_create_from_userptr_ioctl, 0), | ||
| 313 | }; | 319 | }; |
| 314 | 320 | ||
| 315 | static int ivpu_wait_for_ready(struct ivpu_device *vdev) | 321 | static int ivpu_wait_for_ready(struct ivpu_device *vdev) |
| @@ -377,8 +383,7 @@ int ivpu_boot(struct ivpu_device *vdev) · 61 unmodified lines | |||
| 377 | drm_WARN_ON(&vdev->drm, atomic_read(&vdev->job_timeout_counter)); | 383 | drm_WARN_ON(&vdev->drm, atomic_read(&vdev->job_timeout_counter)); |
| 378 | drm_WARN_ON(&vdev->drm, !xa_empty(&vdev->submitted_jobs_xa)); | 384 | drm_WARN_ON(&vdev->drm, !xa_empty(&vdev->submitted_jobs_xa)); |
| 379 | 385 | ||
| 380 | /* Update boot params located at first 4KB of FW memory */ | 386 | ivpu_fw_boot_params_setup(vdev, ivpu_bo_vaddr(vdev->fw->mem_bp)); |
| 381 | ivpu_fw_boot_params_setup(vdev, ivpu_bo_vaddr(vdev->fw->mem)); | ||
| 382 | 387 | ||
| 383 | ret = ivpu_hw_boot_fw(vdev); | 388 | ret = ivpu_hw_boot_fw(vdev); |
| 384 | if (ret) { | 389 | if (ret) { |
| @@ -450,6 +455,9 @@ int ivpu_shutdown(struct ivpu_device *vdev) · 65 unmodified lines | |||
| 450 | static const struct file_operations ivpu_fops = { | 455 | static const struct file_operations ivpu_fops = { |
| 451 | .owner = THIS_MODULE, | 456 | .owner = THIS_MODULE, |
| 452 | DRM_ACCEL_FOPS, | 457 | DRM_ACCEL_FOPS, |
| 458 | #ifdef CONFIG_PROC_FS | ||
| 459 | .show_fdinfo = drm_show_fdinfo, | ||
| 460 | #endif | ||
| 453 | }; | 461 | }; |
| 454 | 462 | ||
| 455 | static const struct drm_driver driver = { | 463 | static const struct drm_driver driver = { |
| @@ -464,6 +472,9 @@ static const struct drm_driver driver = { · 8 unmodified lines | |||
| 464 | .ioctls = ivpu_drm_ioctls, | 472 | .ioctls = ivpu_drm_ioctls, |
| 465 | .num_ioctls = ARRAY_SIZE(ivpu_drm_ioctls), | 473 | .num_ioctls = ARRAY_SIZE(ivpu_drm_ioctls), |
| 466 | .fops = &ivpu_fops, | 474 | .fops = &ivpu_fops, |
| 475 | #ifdef CONFIG_PROC_FS | ||
| 476 | .show_fdinfo = drm_show_memory_stats, | ||
| 477 | #endif | ||
| 467 | 478 | ||
| 468 | .name = DRIVER_NAME, | 479 | .name = DRIVER_NAME, |
| 469 | .desc = DRIVER_DESC, | 480 | .desc = DRIVER_DESC, |
| @@ -705,6 +716,7 @@ static struct pci_device_id ivpu_pci_ids[] = { · 235 unmodified lines | |||
| 705 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_LNL) }, | 716 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_LNL) }, |
| 706 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_PTL_P) }, | 717 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_PTL_P) }, |
| 707 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_WCL) }, | 718 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_WCL) }, |
| 719 | { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_NVL) }, | ||
| 708 | { } | 720 | { } |
| 709 | }; | 721 | }; |
| 710 | MODULE_DEVICE_TABLE(pci, ivpu_pci_ids); | 722 | MODULE_DEVICE_TABLE(pci, ivpu_pci_ids); |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | |||
| 27 | #define PCI_DEVICE_ID_LNL 0x643e | 27 | #define PCI_DEVICE_ID_LNL 0x643e |
| 28 | #define PCI_DEVICE_ID_PTL_P 0xb03e | 28 | #define PCI_DEVICE_ID_PTL_P 0xb03e |
| 29 | #define PCI_DEVICE_ID_WCL 0xfd3e | 29 | #define PCI_DEVICE_ID_WCL 0xfd3e |
| 30 | #define PCI_DEVICE_ID_NVL 0xd71d | ||
| 30 | 31 | ||
| 31 | #define IVPU_HW_IP_37XX 37 | 32 | #define IVPU_HW_IP_37XX 37 |
| 32 | #define IVPU_HW_IP_40XX 40 | 33 | #define IVPU_HW_IP_40XX 40 |
| @@ -78,6 +79,7 @@ · 45 unmodified lines | |||
| 78 | #define IVPU_DBG_KREF BIT(11) | 79 | #define IVPU_DBG_KREF BIT(11) |
| 79 | #define IVPU_DBG_RPM BIT(12) | 80 | #define IVPU_DBG_RPM BIT(12) |
| 80 | #define IVPU_DBG_MMU_MAP BIT(13) | 81 | #define IVPU_DBG_MMU_MAP BIT(13) |
| 82 | #define IVPU_DBG_IOCTL BIT(14) | ||
| 81 | 83 | ||
| 82 | #define ivpu_err(vdev, fmt, ...) \ | 84 | #define ivpu_err(vdev, fmt, ...) \ |
| 83 | drm_err(&(vdev)->drm, "%s(): " fmt, __func__, ##__VA_ARGS__) | 85 | drm_err(&(vdev)->drm, "%s(): " fmt, __func__, ##__VA_ARGS__) |
| @@ -245,6 +247,8 @@ static inline int ivpu_hw_ip_gen(struct ivpu_device *vdev) · 161 unmodified lines | |||
| 245 | case PCI_DEVICE_ID_PTL_P: | 247 | case PCI_DEVICE_ID_PTL_P: |
| 246 | case PCI_DEVICE_ID_WCL: | 248 | case PCI_DEVICE_ID_WCL: |
| 247 | return IVPU_HW_IP_50XX; | 249 | return IVPU_HW_IP_50XX; |
| 250 | case PCI_DEVICE_ID_NVL: | ||
| 251 | return IVPU_HW_IP_60XX; | ||
| 248 | default: | 252 | default: |
| 249 | dump_stack(); | 253 | dump_stack(); |
| 250 | ivpu_err(vdev, "Unknown NPU IP generation\n"); | 254 | ivpu_err(vdev, "Unknown NPU IP generation\n"); |
| @@ -261,6 +265,7 @@ static inline int ivpu_hw_btrs_gen(struct ivpu_device *vdev) · 10 unmodified lines | |||
| 261 | case PCI_DEVICE_ID_LNL: | 265 | case PCI_DEVICE_ID_LNL: |
| 262 | case PCI_DEVICE_ID_PTL_P: | 266 | case PCI_DEVICE_ID_PTL_P: |
| 263 | case PCI_DEVICE_ID_WCL: | 267 | case PCI_DEVICE_ID_WCL: |
| 268 | case PCI_DEVICE_ID_NVL: | ||
| 264 | return IVPU_HW_BTRS_LNL; | 269 | return IVPU_HW_BTRS_LNL; |
| 265 | default: | 270 | default: |
| 266 | dump_stack(); | 271 | dump_stack(); |
| @@ -17,15 +17,10 @@ · 16 unmodified lines | |||
| 17 | #include "ivpu_ipc.h" | 17 | #include "ivpu_ipc.h" |
| 18 | #include "ivpu_pm.h" | 18 | #include "ivpu_pm.h" |
| 19 | 19 | ||
| 20 | #define FW_GLOBAL_MEM_START (2ull * SZ_1G) | ||
| 21 | #define FW_GLOBAL_MEM_END (3ull * SZ_1G) | ||
| 22 | #define FW_SHARED_MEM_SIZE SZ_256M /* Must be aligned to FW_SHARED_MEM_ALIGNMENT */ | ||
| 23 | #define FW_SHARED_MEM_ALIGNMENT SZ_128K /* VPU MTRR limitation */ | ||
| 24 | #define FW_RUNTIME_MAX_SIZE SZ_512M | ||
| 25 | #define FW_SHAVE_NN_MAX_SIZE SZ_2M | 20 | #define FW_SHAVE_NN_MAX_SIZE SZ_2M |
| 26 | #define FW_RUNTIME_MIN_ADDR (FW_GLOBAL_MEM_START) | ||
| 27 | #define FW_RUNTIME_MAX_ADDR (FW_GLOBAL_MEM_END - FW_SHARED_MEM_SIZE) | ||
| 28 | #define FW_FILE_IMAGE_OFFSET (VPU_FW_HEADER_SIZE + FW_VERSION_HEADER_SIZE) | 21 | #define FW_FILE_IMAGE_OFFSET (VPU_FW_HEADER_SIZE + FW_VERSION_HEADER_SIZE) |
| 22 | #define FW_PREEMPT_BUF_MIN_SIZE SZ_4K | ||
| 23 | #define FW_PREEMPT_BUF_MAX_SIZE SZ_32M | ||
| 29 | 24 | ||
| 30 | #define WATCHDOG_MSS_REDIRECT 32 | 25 | #define WATCHDOG_MSS_REDIRECT 32 |
| 31 | #define WATCHDOG_NCE_REDIRECT 33 | 26 | #define WATCHDOG_NCE_REDIRECT 33 |
| @@ -61,12 +56,14 @@ static struct { · 29 unmodified lines | |||
| 61 | { IVPU_HW_IP_40XX, "intel/vpu/vpu_40xx_v0.0.bin" }, | 56 | { IVPU_HW_IP_40XX, "intel/vpu/vpu_40xx_v0.0.bin" }, |
| 62 | { IVPU_HW_IP_50XX, "intel/vpu/vpu_50xx_v1.bin" }, | 57 | { IVPU_HW_IP_50XX, "intel/vpu/vpu_50xx_v1.bin" }, |
| 63 | { IVPU_HW_IP_50XX, "intel/vpu/vpu_50xx_v0.0.bin" }, | 58 | { IVPU_HW_IP_50XX, "intel/vpu/vpu_50xx_v0.0.bin" }, |
| 59 | { IVPU_HW_IP_60XX, "intel/vpu/vpu_60xx_v1.bin" }, | ||
| 64 | }; | 60 | }; |
| 65 | 61 | ||
| 66 | /* Production fw_names from the table above */ | 62 | /* Production fw_names from the table above */ |
| 67 | MODULE_FIRMWARE("intel/vpu/vpu_37xx_v1.bin"); | 63 | MODULE_FIRMWARE("intel/vpu/vpu_37xx_v1.bin"); |
| 68 | MODULE_FIRMWARE("intel/vpu/vpu_40xx_v1.bin"); | 64 | MODULE_FIRMWARE("intel/vpu/vpu_40xx_v1.bin"); |
| 69 | MODULE_FIRMWARE("intel/vpu/vpu_50xx_v1.bin"); | 65 | MODULE_FIRMWARE("intel/vpu/vpu_50xx_v1.bin"); |
| 66 | MODULE_FIRMWARE("intel/vpu/vpu_60xx_v1.bin"); | ||
| 70 | 67 | ||
| 71 | static int ivpu_fw_request(struct ivpu_device *vdev) | 68 | static int ivpu_fw_request(struct ivpu_device *vdev) |
| 72 | { | 69 | { |
| @@ -131,9 +128,14 @@ ivpu_fw_check_api_ver_lt(struct ivpu_device *vdev, const struct vpu_firmware_hea · 58 unmodified lines | |||
| 131 | return false; | 128 | return false; |
| 132 | } | 129 | } |
| 133 | 130 | ||
| 134 | static bool is_within_range(u64 addr, size_t size, u64 range_start, size_t range_size) | 131 | bool ivpu_is_within_range(u64 addr, size_t size, struct ivpu_addr_range *range) |
| 135 | { | 132 | { |
| 136 | if (addr < range_start || addr + size > range_start + range_size) | 133 | u64 addr_end; |
| 134 | |||
| 135 | if (!range || check_add_overflow(addr, size, &addr_end)) | ||
| 136 | return false; | ||
| 137 | |||
| 138 | if (addr < range->start || addr_end > range->end) | ||
| 137 | return false; | 139 | return false; |
| 138 | 140 | ||
| 139 | return true; | 141 | return true; |
| @@ -142,6 +144,12 @@ static bool is_within_range(u64 addr, size_t size, u64 range_start, size_t range · 2 unmodified lines | |||
| 142 | static u32 | 144 | static u32 |
| 143 | ivpu_fw_sched_mode_select(struct ivpu_device *vdev, const struct vpu_firmware_header *fw_hdr) | 145 | ivpu_fw_sched_mode_select(struct ivpu_device *vdev, const struct vpu_firmware_header *fw_hdr) |
| 144 | { | 146 | { |
| 147 | if (ivpu_hw_ip_gen(vdev) >= IVPU_HW_IP_60XX && | ||
| 148 | ivpu_sched_mode == VPU_SCHEDULING_MODE_OS) { | ||
| 149 | ivpu_warn(vdev, "OS sched mode is not supported, using HW mode\n"); | ||
| 150 | return VPU_SCHEDULING_MODE_HW; | ||
| 151 | } | ||
| 152 | |||
| 145 | if (ivpu_sched_mode != IVPU_SCHED_MODE_AUTO) | 153 | if (ivpu_sched_mode != IVPU_SCHED_MODE_AUTO) |
| 146 | return ivpu_sched_mode; | 154 | return ivpu_sched_mode; |
| 147 | 155 | ||
| @@ -151,11 +159,56 @@ ivpu_fw_sched_mode_select(struct ivpu_device *vdev, const struct vpu_firmware_he · 3 unmodified lines | |||
| 151 | return VPU_SCHEDULING_MODE_HW; | 159 | return VPU_SCHEDULING_MODE_HW; |
| 152 | } | 160 | } |
| 153 | 161 | ||
| 162 | static void | ||
| 163 | ivpu_preemption_config_parse(struct ivpu_device *vdev, const struct vpu_firmware_header *fw_hdr) | ||
| 164 | { | ||
| 165 | struct ivpu_fw_info *fw = vdev->fw; | ||
| 166 | u32 primary_preempt_buf_size, secondary_preempt_buf_size; | ||
| 167 | |||
| 168 | if (fw_hdr->preemption_buffer_1_max_size) | ||
| 169 | primary_preempt_buf_size = fw_hdr->preemption_buffer_1_max_size; | ||
| 170 | else | ||
| 171 | primary_preempt_buf_size = fw_hdr->preemption_buffer_1_size; | ||
| 172 | |||
| 173 | if (fw_hdr->preemption_buffer_2_max_size) | ||
| 174 | secondary_preempt_buf_size = fw_hdr->preemption_buffer_2_max_size; | ||
| 175 | else | ||
| 176 | secondary_preempt_buf_size = fw_hdr->preemption_buffer_2_size; | ||
| 177 | |||
| 178 | ivpu_dbg(vdev, FW_BOOT, "Preemption buffer size, primary: %u, secondary: %u\n", | ||
| 179 | primary_preempt_buf_size, secondary_preempt_buf_size); | ||
| 180 | |||
| 181 | if (primary_preempt_buf_size < FW_PREEMPT_BUF_MIN_SIZE || | ||
| 182 | secondary_preempt_buf_size < FW_PREEMPT_BUF_MIN_SIZE) { | ||
| 183 | ivpu_warn(vdev, "Preemption buffers size too small\n"); | ||
| 184 | return; | ||
| 185 | } | ||
| 186 | |||
| 187 | if (primary_preempt_buf_size > FW_PREEMPT_BUF_MAX_SIZE || | ||
| 188 | secondary_preempt_buf_size > FW_PREEMPT_BUF_MAX_SIZE) { | ||
| 189 | ivpu_warn(vdev, "Preemption buffers size too big\n"); | ||
| 190 | return; | ||
| 191 | } | ||
| 192 | |||
| 193 | if (fw->sched_mode != VPU_SCHEDULING_MODE_HW) | ||
| 194 | return; | ||
| 195 | |||
| 196 | if (ivpu_test_mode & IVPU_TEST_MODE_MIP_DISABLE) | ||
| 197 | return; | ||
| 198 | |||
| 199 | vdev->fw->primary_preempt_buf_size = ALIGN(primary_preempt_buf_size, PAGE_SIZE); | ||
| 200 | vdev->fw->secondary_preempt_buf_size = ALIGN(secondary_preempt_buf_size, PAGE_SIZE); | ||
| 201 | } | ||
| 202 | |||
| 154 | static int ivpu_fw_parse(struct ivpu_device *vdev) | 203 | static int ivpu_fw_parse(struct ivpu_device *vdev) |
| 155 | { | 204 | { |
| 156 | struct ivpu_fw_info *fw = vdev->fw; | 205 | struct ivpu_fw_info *fw = vdev->fw; |
| 157 | const struct vpu_firmware_header *fw_hdr = (const void *)fw->file->data; | 206 | const struct vpu_firmware_header *fw_hdr = (const void *)fw->file->data; |
| 158 | u64 runtime_addr, image_load_addr, runtime_size, image_size; | 207 | struct ivpu_addr_range fw_image_range; |
| 208 | u64 boot_params_addr, boot_params_size; | ||
| 209 | u64 fw_version_addr, fw_version_size; | ||
| 210 | u64 runtime_addr, runtime_size; | ||
| 211 | u64 image_load_addr, image_size; | ||
| 159 | 212 | ||
| 160 | if (fw->file->size <= FW_FILE_IMAGE_OFFSET) { | 213 | if (fw->file->size <= FW_FILE_IMAGE_OFFSET) { |
| 161 | ivpu_err(vdev, "Firmware file is too small: %zu\n", fw->file->size); | 214 | ivpu_err(vdev, "Firmware file is too small: %zu\n", fw->file->size); |
| @@ -167,18 +220,37 @@ static int ivpu_fw_parse(struct ivpu_device *vdev) · 5 unmodified lines | |||
| 167 | return -EINVAL; | 220 | return -EINVAL; |
| 168 | } | 221 | } |
| 169 | 222 | ||
| 170 | runtime_addr = fw_hdr->boot_params_load_address; | 223 | boot_params_addr = fw_hdr->boot_params_load_address; |
| 171 | runtime_size = fw_hdr->runtime_size; | 224 | boot_params_size = SZ_4K; |
| 172 | image_load_addr = fw_hdr->image_load_address; | ||
| 173 | image_size = fw_hdr->image_size; | ||
| 174 | 225 | ||
| 175 | if (runtime_addr < FW_RUNTIME_MIN_ADDR || runtime_addr > FW_RUNTIME_MAX_ADDR) { | 226 | if (!ivpu_is_within_range(boot_params_addr, boot_params_size, &vdev->hw->ranges.runtime)) { |
| 176 | ivpu_err(vdev, "Invalid firmware runtime address: 0x%llx\n", runtime_addr); | 227 | ivpu_err(vdev, "Invalid boot params address: 0x%llx\n", boot_params_addr); |
| 177 | return -EINVAL; | 228 | return -EINVAL; |
| 178 | } | 229 | } |
| 179 | 230 | ||
| 180 | if (runtime_size < fw->file->size || runtime_size > FW_RUNTIME_MAX_SIZE) { | 231 | fw_version_addr = fw_hdr->firmware_version_load_address; |
| 181 | ivpu_err(vdev, "Invalid firmware runtime size: %llu\n", runtime_size); | 232 | fw_version_size = ALIGN(fw_hdr->firmware_version_size, SZ_4K); |
| 233 | |||
| 234 | if (fw_version_size != SZ_4K) { | ||
| 235 | ivpu_err(vdev, "Invalid firmware version size: %u\n", | ||
| 236 | fw_hdr->firmware_version_size); | ||
| 237 | return -EINVAL; | ||
| 238 | } | ||
| 239 | |||
| 240 | if (!ivpu_is_within_range(fw_version_addr, fw_version_size, &vdev->hw->ranges.runtime)) { | ||
| 241 | ivpu_err(vdev, "Invalid firmware version address: 0x%llx\n", fw_version_addr); | ||
| 242 | return -EINVAL; | ||
| 243 | } | ||
| 244 | |||
| 245 | runtime_addr = fw_hdr->image_load_address; | ||
| 246 | runtime_size = fw_hdr->runtime_size - boot_params_size - fw_version_size; | ||
| 247 | |||
| 248 | image_load_addr = fw_hdr->image_load_address; | ||
| 249 | image_size = fw_hdr->image_size; | ||
| 250 | |||
| 251 | if (!ivpu_is_within_range(runtime_addr, runtime_size, &vdev->hw->ranges.runtime)) { | ||
| 252 | ivpu_err(vdev, "Invalid firmware runtime address: 0x%llx and size %llu\n", | ||
| 253 | runtime_addr, runtime_size); | ||
| 182 | return -EINVAL; | 254 | return -EINVAL; |
| 183 | } | 255 | } |
| 184 | 256 | ||
| @@ -187,23 +259,25 @@ static int ivpu_fw_parse(struct ivpu_device *vdev) · 2 unmodified lines | |||
| 187 | return -EINVAL; | 259 | return -EINVAL; |
| 188 | } | 260 | } |
| 189 | 261 | ||
| 190 | if (image_load_addr < runtime_addr || | 262 | if (!ivpu_is_within_range(image_load_addr, image_size, &vdev->hw->ranges.runtime)) { |
| 191 | image_load_addr + image_size > runtime_addr + runtime_size) { | 263 | ivpu_err(vdev, "Invalid firmware load address: 0x%llx and size %llu\n", |
| 192 | ivpu_err(vdev, "Invalid firmware load address size: 0x%llx and size %llu\n", | ||
| 193 | image_load_addr, image_size); | 264 | image_load_addr, image_size); |
| 194 | return -EINVAL; | 265 | return -EINVAL; |
| 195 | } | 266 | } |
| 196 | 267 | ||
| 197 | if (fw_hdr->shave_nn_fw_size > FW_SHAVE_NN_MAX_SIZE) { | 268 | if (ivpu_hw_range_init(vdev, &fw_image_range, image_load_addr, image_size)) |
| 198 | ivpu_err(vdev, "SHAVE NN firmware is too big: %u\n", fw_hdr->shave_nn_fw_size); | ||
| 199 | return -EINVAL; | 269 | return -EINVAL; |
| 200 | } | ||
| 201 | 270 | ||
| 202 | if (fw_hdr->entry_point < image_load_addr || | 271 | if (!ivpu_is_within_range(fw_hdr->entry_point, SZ_4K, &fw_image_range)) { |
| 203 | fw_hdr->entry_point >= image_load_addr + image_size) { | ||
| 204 | ivpu_err(vdev, "Invalid entry point: 0x%llx\n", fw_hdr->entry_point); | 272 | ivpu_err(vdev, "Invalid entry point: 0x%llx\n", fw_hdr->entry_point); |
| 205 | return -EINVAL; | 273 | return -EINVAL; |
| 206 | } | 274 | } |
| 275 | |||
| 276 | if (fw_hdr->shave_nn_fw_size > FW_SHAVE_NN_MAX_SIZE) { | ||
| 277 | ivpu_err(vdev, "SHAVE NN firmware is too big: %u\n", fw_hdr->shave_nn_fw_size); | ||
| 278 | return -EINVAL; | ||
| 279 | } | ||
| 280 | |||
| 207 | ivpu_dbg(vdev, FW_BOOT, "Header version: 0x%x, format 0x%x\n", | 281 | ivpu_dbg(vdev, FW_BOOT, "Header version: 0x%x, format 0x%x\n", |
| 208 | fw_hdr->header_version, fw_hdr->image_format); | 282 | fw_hdr->header_version, fw_hdr->image_format); |
| 209 | 283 | ||
| @@ -217,6 +291,10 @@ static int ivpu_fw_parse(struct ivpu_device *vdev) · 7 unmodified lines | |||
| 217 | if (IVPU_FW_CHECK_API_COMPAT(vdev, fw_hdr, JSM, 3)) | 291 | if (IVPU_FW_CHECK_API_COMPAT(vdev, fw_hdr, JSM, 3)) |
| 218 | return -EINVAL; | 292 | return -EINVAL; |
| 219 | 293 | ||
| 294 | fw->boot_params_addr = boot_params_addr; | ||
| 295 | fw->boot_params_size = boot_params_size; | ||
| 296 | fw->fw_version_addr = fw_version_addr; | ||
| 297 | fw->fw_version_size = fw_version_size; | ||
| 220 | fw->runtime_addr = runtime_addr; | 298 | fw->runtime_addr = runtime_addr; |
| 221 | fw->runtime_size = runtime_size; | 299 | fw->runtime_size = runtime_size; |
| 222 | fw->image_load_offset = image_load_addr - runtime_addr; | 300 | fw->image_load_offset = image_load_addr - runtime_addr; |
| @@ -235,22 +313,13 @@ static int ivpu_fw_parse(struct ivpu_device *vdev) · 12 unmodified lines | |||
| 235 | fw->sched_mode = ivpu_fw_sched_mode_select(vdev, fw_hdr); | 313 | fw->sched_mode = ivpu_fw_sched_mode_select(vdev, fw_hdr); |
| 236 | ivpu_info(vdev, "Scheduler mode: %s\n", fw->sched_mode ? "HW" : "OS"); | 314 | ivpu_info(vdev, "Scheduler mode: %s\n", fw->sched_mode ? "HW" : "OS"); |
| 237 | 315 | ||
| 238 | if (fw_hdr->preemption_buffer_1_max_size) | 316 | ivpu_preemption_config_parse(vdev, fw_hdr); |
| 239 | fw->primary_preempt_buf_size = fw_hdr->preemption_buffer_1_max_size; | 317 | ivpu_dbg(vdev, FW_BOOT, "Mid-inference preemption %s supported\n", |
| 240 | else | 318 | ivpu_fw_preempt_buf_size(vdev) ? "is" : "is not"); |
| 241 | fw->primary_preempt_buf_size = fw_hdr->preemption_buffer_1_size; | ||
| 242 | 319 | ||
| 243 | if (fw_hdr->preemption_buffer_2_max_size) | 320 | if (fw_hdr->ro_section_start_address && |
| 244 | fw->secondary_preempt_buf_size = fw_hdr->preemption_buffer_2_max_size; | 321 | !ivpu_is_within_range(fw_hdr->ro_section_start_address, fw_hdr->ro_section_size, |
| 245 | else | 322 | &fw_image_range)) { |
| 246 | fw->secondary_preempt_buf_size = fw_hdr->preemption_buffer_2_size; | ||
| 247 | ivpu_dbg(vdev, FW_BOOT, "Preemption buffer sizes: primary %u, secondary %u\n", | ||
| 248 | fw->primary_preempt_buf_size, fw->secondary_preempt_buf_size); | ||
| 249 | |||
| 250 | if (fw_hdr->ro_section_start_address && !is_within_range(fw_hdr->ro_section_start_address, | ||
| 251 | fw_hdr->ro_section_size, | ||
| 252 | fw_hdr->image_load_address, | ||
| 253 | fw_hdr->image_size)) { | ||
| 254 | ivpu_err(vdev, "Invalid read-only section: start address 0x%llx, size %u\n", | 323 | ivpu_err(vdev, "Invalid read-only section: start address 0x%llx, size %u\n", |
| 255 | fw_hdr->ro_section_start_address, fw_hdr->ro_section_size); | 324 | fw_hdr->ro_section_start_address, fw_hdr->ro_section_size); |
| 256 | return -EINVAL; | 325 | return -EINVAL; |
| @@ -259,12 +328,18 @@ static int ivpu_fw_parse(struct ivpu_device *vdev) · 2 unmodified lines | |||
| 259 | fw->read_only_addr = fw_hdr->ro_section_start_address; | 328 | fw->read_only_addr = fw_hdr->ro_section_start_address; |
| 260 | fw->read_only_size = fw_hdr->ro_section_size; | 329 | fw->read_only_size = fw_hdr->ro_section_size; |
| 261 | 330 | ||
| 262 | ivpu_dbg(vdev, FW_BOOT, "Size: file %lu image %u runtime %u shavenn %u\n", | 331 | ivpu_dbg(vdev, FW_BOOT, "Boot params: address 0x%llx, size %llu\n", |
| 263 | fw->file->size, fw->image_size, fw->runtime_size, fw->shave_nn_size); | 332 | fw->boot_params_addr, fw->boot_params_size); |
| 264 | ivpu_dbg(vdev, FW_BOOT, "Address: runtime 0x%llx, load 0x%llx, entry point 0x%llx\n", | 333 | ivpu_dbg(vdev, FW_BOOT, "FW version: address 0x%llx, size %llu\n", |
| 265 | fw->runtime_addr, image_load_addr, fw->entry_point); | 334 | fw->fw_version_addr, fw->fw_version_size); |
| 335 | ivpu_dbg(vdev, FW_BOOT, "Runtime: address 0x%llx, size %u\n", | ||
| 336 | fw->runtime_addr, fw->runtime_size); | ||
| 337 | ivpu_dbg(vdev, FW_BOOT, "Image load offset: 0x%llx, size %u\n", | ||
| 338 | fw->image_load_offset, fw->image_size); | ||
| 266 | ivpu_dbg(vdev, FW_BOOT, "Read-only section: address 0x%llx, size %u\n", | 339 | ivpu_dbg(vdev, FW_BOOT, "Read-only section: address 0x%llx, size %u\n", |
| 267 | fw->read_only_addr, fw->read_only_size); | 340 | fw->read_only_addr, fw->read_only_size); |
| 341 | ivpu_dbg(vdev, FW_BOOT, "FW entry point: 0x%llx\n", fw->entry_point); | ||
| 342 | ivpu_dbg(vdev, FW_BOOT, "SHAVE NN size: %u\n", fw->shave_nn_size); | ||
| 268 | 343 | ||
| 269 | return 0; | 344 | return 0; |
| 270 | } | 345 | } |
| @@ -291,39 +366,33 @@ ivpu_fw_init_wa(struct ivpu_device *vdev) · 20 unmodified lines | |||
| 291 | IVPU_PRINT_WA(disable_d0i3_msg); | 366 | IVPU_PRINT_WA(disable_d0i3_msg); |
| 292 | } | 367 | } |
| 293 | 368 | ||
| 294 | static int ivpu_fw_update_global_range(struct ivpu_device *vdev) | ||
| 295 | { | ||
| 296 | struct ivpu_fw_info *fw = vdev->fw; | ||
| 297 | u64 start = ALIGN(fw->runtime_addr + fw->runtime_size, FW_SHARED_MEM_ALIGNMENT); | ||
| 298 | u64 size = FW_SHARED_MEM_SIZE; | ||
| 299 | |||
| 300 | if (start + size > FW_GLOBAL_MEM_END) { | ||
| 301 | ivpu_err(vdev, "No space for shared region, start %lld, size %lld\n", start, size); | ||
| 302 | return -EINVAL; | ||
| 303 | } | ||
| 304 | |||
| 305 | ivpu_hw_range_init(&vdev->hw->ranges.global, start, size); | ||
| 306 | return 0; | ||
| 307 | } | ||
| 308 | |||
| 309 | static int ivpu_fw_mem_init(struct ivpu_device *vdev) | 369 | static int ivpu_fw_mem_init(struct ivpu_device *vdev) |
| 310 | { | 370 | { |
| 311 | struct ivpu_fw_info *fw = vdev->fw; | 371 | struct ivpu_fw_info *fw = vdev->fw; |
| 312 | struct ivpu_addr_range fw_range; | ||
| 313 | int log_verb_size; | 372 | int log_verb_size; |
| 314 | int ret; | 373 | int ret; |
| 315 | 374 | ||
| 316 | ret = ivpu_fw_update_global_range(vdev); | 375 | fw->mem_bp = ivpu_bo_create_runtime(vdev, fw->boot_params_addr, fw->boot_params_size, |
| 317 | if (ret) | 376 | DRM_IVPU_BO_WC | DRM_IVPU_BO_MAPPABLE); |
| 318 | return ret; | 377 | if (!fw->mem_bp) { |
| 378 | ivpu_err(vdev, "Failed to create firmware boot params memory buffer\n"); | ||
| 379 | return -ENOMEM; | ||
| 380 | } | ||
| 319 | 381 | ||
| 320 | fw_range.start = fw->runtime_addr; | 382 | fw->mem_fw_ver = ivpu_bo_create_runtime(vdev, fw->fw_version_addr, fw->fw_version_size, |
| 321 | fw_range.end = fw->runtime_addr + fw->runtime_size; | 383 | DRM_IVPU_BO_WC | DRM_IVPU_BO_MAPPABLE); |
| 322 | fw->mem = ivpu_bo_create(vdev, &vdev->gctx, &fw_range, fw->runtime_size, | 384 | if (!fw->mem_fw_ver) { |
| 323 | DRM_IVPU_BO_WC | DRM_IVPU_BO_MAPPABLE); | 385 | ivpu_err(vdev, "Failed to create firmware version memory buffer\n"); |
| 386 | ret = -ENOMEM; | ||
| 387 | goto err_free_bp; | ||
| 388 | } | ||
| 389 | |||
| 390 | fw->mem = ivpu_bo_create_runtime(vdev, fw->runtime_addr, fw->runtime_size, | ||
| 391 | DRM_IVPU_BO_WC | DRM_IVPU_BO_MAPPABLE); | ||
| 324 | if (!fw->mem) { | 392 | if (!fw->mem) { |
| 325 | ivpu_err(vdev, "Failed to create firmware runtime memory buffer\n"); | 393 | ivpu_err(vdev, "Failed to create firmware runtime memory buffer\n"); |
| 326 | return -ENOMEM; | 394 | ret = -ENOMEM; |
| 395 | goto err_free_fw_ver; | ||
| 327 | } | 396 | } |
| 328 | 397 | ||
| 329 | ret = ivpu_mmu_context_set_pages_ro(vdev, &vdev->gctx, fw->read_only_addr, | 398 | ret = ivpu_mmu_context_set_pages_ro(vdev, &vdev->gctx, fw->read_only_addr, |
| @@ -372,6 +441,10 @@ err_free_log_crit: · 42 unmodified lines | |||
| 372 | ivpu_bo_free(fw->mem_log_crit); | 441 | ivpu_bo_free(fw->mem_log_crit); |
| 373 | err_free_fw_mem: | 442 | err_free_fw_mem: |
| 374 | ivpu_bo_free(fw->mem); | 443 | ivpu_bo_free(fw->mem); |
| 444 | err_free_fw_ver: | ||
| 445 | ivpu_bo_free(fw->mem_fw_ver); | ||
| 446 | err_free_bp: | ||
| 447 | ivpu_bo_free(fw->mem_bp); | ||
| 375 | return ret; | 448 | return ret; |
| 376 | } | 449 | } |
| 377 | 450 | ||
| @@ -387,10 +460,14 @@ static void ivpu_fw_mem_fini(struct ivpu_device *vdev) · 9 unmodified lines | |||
| 387 | ivpu_bo_free(fw->mem_log_verb); | 460 | ivpu_bo_free(fw->mem_log_verb); |
| 388 | ivpu_bo_free(fw->mem_log_crit); | 461 | ivpu_bo_free(fw->mem_log_crit); |
| 389 | ivpu_bo_free(fw->mem); | 462 | ivpu_bo_free(fw->mem); |
| 463 | ivpu_bo_free(fw->mem_fw_ver); | ||
| 464 | ivpu_bo_free(fw->mem_bp); | ||
| 390 | 465 | ||
| 391 | fw->mem_log_verb = NULL; | 466 | fw->mem_log_verb = NULL; |
| 392 | fw->mem_log_crit = NULL; | 467 | fw->mem_log_crit = NULL; |
| 393 | fw->mem = NULL; | 468 | fw->mem = NULL; |
| 469 | fw->mem_fw_ver = NULL; | ||
| 470 | fw->mem_bp = NULL; | ||
| 394 | } | 471 | } |
| 395 | 472 | ||
| 396 | int ivpu_fw_init(struct ivpu_device *vdev) | 473 | int ivpu_fw_init(struct ivpu_device *vdev) |
| @@ -483,11 +560,6 @@ static void ivpu_fw_boot_params_print(struct ivpu_device *vdev, struct vpu_boot_ · 86 unmodified lines | |||
| 483 | ivpu_dbg(vdev, FW_BOOT, "boot_params.cache_defaults[VPU_BOOT_L2_CACHE_CFG_NN].cfg = 0x%x\n", | 560 | ivpu_dbg(vdev, FW_BOOT, "boot_params.cache_defaults[VPU_BOOT_L2_CACHE_CFG_NN].cfg = 0x%x\n", |
| 484 | boot_params->cache_defaults[VPU_BOOT_L2_CACHE_CFG_NN].cfg); | 561 | boot_params->cache_defaults[VPU_BOOT_L2_CACHE_CFG_NN].cfg); |
| 485 | 562 | ||
| 486 | ivpu_dbg(vdev, FW_BOOT, "boot_params.global_memory_allocator_base = 0x%llx\n", | ||
| 487 | boot_params->global_memory_allocator_base); | ||
| 488 | ivpu_dbg(vdev, FW_BOOT, "boot_params.global_memory_allocator_size = 0x%x\n", | ||
| 489 | boot_params->global_memory_allocator_size); | ||
| 490 | |||
| 491 | ivpu_dbg(vdev, FW_BOOT, "boot_params.shave_nn_fw_base = 0x%llx\n", | 563 | ivpu_dbg(vdev, FW_BOOT, "boot_params.shave_nn_fw_base = 0x%llx\n", |
| 492 | boot_params->shave_nn_fw_base); | 564 | boot_params->shave_nn_fw_base); |
| 493 | 565 | ||
| @@ -495,10 +567,6 @@ static void ivpu_fw_boot_params_print(struct ivpu_device *vdev, struct vpu_boot_ · 1 unmodified line | |||
| 495 | boot_params->watchdog_irq_mss); | 567 | boot_params->watchdog_irq_mss); |
| 496 | ivpu_dbg(vdev, FW_BOOT, "boot_params.watchdog_irq_nce = 0x%x\n", | 568 | ivpu_dbg(vdev, FW_BOOT, "boot_params.watchdog_irq_nce = 0x%x\n", |
| 497 | boot_params->watchdog_irq_nce); | 569 | boot_params->watchdog_irq_nce); |
| 498 | ivpu_dbg(vdev, FW_BOOT, "boot_params.host_to_vpu_irq = 0x%x\n", | ||
| 499 | boot_params->host_to_vpu_irq); | ||
| 500 | ivpu_dbg(vdev, FW_BOOT, "boot_params.job_done_irq = 0x%x\n", | ||
| 501 | boot_params->job_done_irq); | ||
| 502 | 570 | ||
| 503 | ivpu_dbg(vdev, FW_BOOT, "boot_params.host_version_id = 0x%x\n", | 571 | ivpu_dbg(vdev, FW_BOOT, "boot_params.host_version_id = 0x%x\n", |
| 504 | boot_params->host_version_id); | 572 | boot_params->host_version_id); |
| @@ -546,6 +614,8 @@ static void ivpu_fw_boot_params_print(struct ivpu_device *vdev, struct vpu_boot_ · 41 unmodified lines | |||
| 546 | boot_params->system_time_us); | 614 | boot_params->system_time_us); |
| 547 | ivpu_dbg(vdev, FW_BOOT, "boot_params.power_profile = 0x%x\n", | 615 | ivpu_dbg(vdev, FW_BOOT, "boot_params.power_profile = 0x%x\n", |
| 548 | boot_params->power_profile); | 616 | boot_params->power_profile); |
| 617 | ivpu_dbg(vdev, FW_BOOT, "boot_params.vpu_uses_ecc_mca_signal = 0x%x\n", | ||
| 618 | boot_params->vpu_uses_ecc_mca_signal); | ||
| 549 | } | 619 | } |
| 550 | 620 | ||
| 551 | void ivpu_fw_boot_params_setup(struct ivpu_device *vdev, struct vpu_boot_params *boot_params) | 621 | void ivpu_fw_boot_params_setup(struct ivpu_device *vdev, struct vpu_boot_params *boot_params) |
| @@ -572,6 +642,7 @@ void ivpu_fw_boot_params_setup(struct ivpu_device *vdev, struct vpu_boot_params · 20 unmodified lines | |||
| 572 | return; | 642 | return; |
| 573 | } | 643 | } |
| 574 | 644 | ||
| 645 | memset(boot_params, 0, sizeof(*boot_params)); | ||
| 575 | vdev->pm->is_warmboot = false; | 646 | vdev->pm->is_warmboot = false; |
| 576 | 647 | ||
| 577 | boot_params->magic = VPU_BOOT_PARAMS_MAGIC; | 648 | boot_params->magic = VPU_BOOT_PARAMS_MAGIC; |
| @@ -647,6 +718,8 @@ void ivpu_fw_boot_params_setup(struct ivpu_device *vdev, struct vpu_boot_params · 69 unmodified lines | |||
| 647 | boot_params->d0i3_entry_vpu_ts = 0; | 718 | boot_params->d0i3_entry_vpu_ts = 0; |
| 648 | if (IVPU_WA(disable_d0i2)) | 719 | if (IVPU_WA(disable_d0i2)) |
| 649 | boot_params->power_profile |= BIT(1); | 720 | boot_params->power_profile |= BIT(1); |
| 721 | boot_params->vpu_uses_ecc_mca_signal = | ||
| 722 | ivpu_hw_uses_ecc_mca_signal(vdev) ? VPU_BOOT_MCA_ECC_BOTH : 0; | ||
| 650 | 723 | ||
| 651 | boot_params->system_time_us = ktime_to_us(ktime_get_real()); | 724 | boot_params->system_time_us = ktime_to_us(ktime_get_real()); |
| 652 | wmb(); /* Flush WC buffers after writing bootparams */ | 725 | wmb(); /* Flush WC buffers after writing bootparams */ |
| @@ -1,6 +1,6 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only */ | 1 | /* SPDX-License-Identifier: GPL-2.0-only */ |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2024 Intel Corporation | 3 | * Copyright (C) 2020-2025 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #ifndef __IVPU_FW_H__ | 6 | #ifndef __IVPU_FW_H__ |
| @@ -19,10 +19,16 @@ struct ivpu_fw_info { · 12 unmodified lines | |||
| 19 | const struct firmware *file; | 19 | const struct firmware *file; |
| 20 | const char *name; | 20 | const char *name; |
| 21 | char version[FW_VERSION_STR_SIZE]; | 21 | char version[FW_VERSION_STR_SIZE]; |
| 22 | struct ivpu_bo *mem_bp; | ||
| 23 | struct ivpu_bo *mem_fw_ver; | ||
| 22 | struct ivpu_bo *mem; | 24 | struct ivpu_bo *mem; |
| 23 | struct ivpu_bo *mem_shave_nn; | 25 | struct ivpu_bo *mem_shave_nn; |
| 24 | struct ivpu_bo *mem_log_crit; | 26 | struct ivpu_bo *mem_log_crit; |
| 25 | struct ivpu_bo *mem_log_verb; | 27 | struct ivpu_bo *mem_log_verb; |
| 28 | u64 boot_params_addr; | ||
| 29 | u64 boot_params_size; | ||
| 30 | u64 fw_version_addr; | ||
| 31 | u64 fw_version_size; | ||
| 26 | u64 runtime_addr; | 32 | u64 runtime_addr; |
| 27 | u32 runtime_size; | 33 | u32 runtime_size; |
| 28 | u64 image_load_offset; | 34 | u64 image_load_offset; |
| @@ -42,6 +48,7 @@ struct ivpu_fw_info { · 13 unmodified lines | |||
| 42 | u64 last_heartbeat; | 48 | u64 last_heartbeat; |
| 43 | }; | 49 | }; |
| 44 | 50 | ||
| 51 | bool ivpu_is_within_range(u64 addr, size_t size, struct ivpu_addr_range *range); | ||
| 45 | int ivpu_fw_init(struct ivpu_device *vdev); | 52 | int ivpu_fw_init(struct ivpu_device *vdev); |
| 46 | void ivpu_fw_fini(struct ivpu_device *vdev); | 53 | void ivpu_fw_fini(struct ivpu_device *vdev); |
| 47 | void ivpu_fw_load(struct ivpu_device *vdev); | 54 | void ivpu_fw_load(struct ivpu_device *vdev); |
| @@ -52,4 +59,9 @@ static inline bool ivpu_fw_is_cold_boot(struct ivpu_device *vdev) · 4 unmodified lines | |||
| 52 | return vdev->fw->entry_point == vdev->fw->cold_boot_entry_point; | 59 | return vdev->fw->entry_point == vdev->fw->cold_boot_entry_point; |
| 53 | } | 60 | } |
| 54 | 61 | ||
| 62 | static inline u32 ivpu_fw_preempt_buf_size(struct ivpu_device *vdev) | ||
| 63 | { | ||
| 64 | return vdev->fw->primary_preempt_buf_size + vdev->fw->secondary_preempt_buf_size; | ||
| 65 | } | ||
| 66 | |||
| 55 | #endif /* __IVPU_FW_H__ */ | 67 | #endif /* __IVPU_FW_H__ */ |
| @@ -15,6 +15,7 @@ · 14 unmodified lines | |||
| 15 | #include <drm/drm_utils.h> | 15 | #include <drm/drm_utils.h> |
| 16 | 16 | ||
| 17 | #include "ivpu_drv.h" | 17 | #include "ivpu_drv.h" |
| 18 | #include "ivpu_fw.h" | ||
| 18 | #include "ivpu_gem.h" | 19 | #include "ivpu_gem.h" |
| 19 | #include "ivpu_hw.h" | 20 | #include "ivpu_hw.h" |
| 20 | #include "ivpu_mmu.h" | 21 | #include "ivpu_mmu.h" |
| @@ -27,8 +28,8 @@ static const struct drm_gem_object_funcs ivpu_gem_funcs; · 6 unmodified lines | |||
| 27 | static inline void ivpu_dbg_bo(struct ivpu_device *vdev, struct ivpu_bo *bo, const char *action) | 28 | static inline void ivpu_dbg_bo(struct ivpu_device *vdev, struct ivpu_bo *bo, const char *action) |
| 28 | { | 29 | { |
| 29 | ivpu_dbg(vdev, BO, | 30 | ivpu_dbg(vdev, BO, |
| 30 | "%6s: bo %8p vpu_addr %9llx size %8zu ctx %d has_pages %d dma_mapped %d mmu_mapped %d wc %d imported %d\n", | 31 | "%6s: bo %8p size %9zu ctx %d vpu_addr %9llx pages %d sgt %d mmu_mapped %d wc %d imported %d\n", |
| 31 | action, bo, bo->vpu_addr, ivpu_bo_size(bo), bo->ctx_id, | 32 | action, bo, ivpu_bo_size(bo), bo->ctx_id, bo->vpu_addr, |
| 32 | (bool)bo->base.pages, (bool)bo->base.sgt, bo->mmu_mapped, bo->base.map_wc, | 33 | (bool)bo->base.pages, (bool)bo->base.sgt, bo->mmu_mapped, bo->base.map_wc, |
| 33 | (bool)drm_gem_is_imported(&bo->base.base)); | 34 | (bool)drm_gem_is_imported(&bo->base.base)); |
| 34 | } | 35 | } |
| @@ -43,22 +44,47 @@ static inline void ivpu_bo_unlock(struct ivpu_bo *bo) · 8 unmodified lines | |||
| 43 | dma_resv_unlock(bo->base.base.resv); | 44 | dma_resv_unlock(bo->base.base.resv); |
| 44 | } | 45 | } |
| 45 | 46 | ||
| 47 | static struct sg_table *ivpu_bo_map_attachment(struct ivpu_device *vdev, struct ivpu_bo *bo) | ||
| 48 | { | ||
| 49 | struct sg_table *sgt; | ||
| 50 | |||
| 51 | drm_WARN_ON(&vdev->drm, !bo->base.base.import_attach); | ||
| 52 | |||
| 53 | ivpu_bo_lock(bo); | ||
| 54 | |||
| 55 | sgt = bo->base.sgt; | ||
| 56 | if (!sgt) { | ||
| 57 | sgt = dma_buf_map_attachment(bo->base.base.import_attach, DMA_BIDIRECTIONAL); | ||
| 58 | if (IS_ERR(sgt)) | ||
| 59 | ivpu_err(vdev, "Failed to map BO in IOMMU: %ld\n", PTR_ERR(sgt)); | ||
| 60 | else | ||
| 61 | bo->base.sgt = sgt; | ||
| 62 | } | ||
| 63 | |||
| 64 | ivpu_bo_unlock(bo); | ||
| 65 | |||
| 66 | return sgt; | ||
| 67 | } | ||
| 68 | |||
| 46 | /* | 69 | /* |
| 47 | * ivpu_bo_pin() - pin the backing physical pages and map them to VPU. | 70 | * ivpu_bo_bind() - pin the backing physical pages and map them to VPU. |
| 48 | * | 71 | * |
| 49 | * This function pins physical memory pages, then maps the physical pages | 72 | * This function pins physical memory pages, then maps the physical pages |
| 50 | * to IOMMU address space and finally updates the VPU MMU page tables | 73 | * to IOMMU address space and finally updates the VPU MMU page tables |
| 51 | * to allow the VPU to translate VPU address to IOMMU address. | 74 | * to allow the VPU to translate VPU address to IOMMU address. |
| 52 | */ | 75 | */ |
| 53 | int __must_check ivpu_bo_pin(struct ivpu_bo *bo) | 76 | int __must_check ivpu_bo_bind(struct ivpu_bo *bo) |
| 54 | { | 77 | { |
| 55 | struct ivpu_device *vdev = ivpu_bo_to_vdev(bo); | 78 | struct ivpu_device *vdev = ivpu_bo_to_vdev(bo); |
| 56 | struct sg_table *sgt; | 79 | struct sg_table *sgt; |
| 57 | int ret = 0; | 80 | int ret = 0; |
| 58 | 81 | ||
| 59 | ivpu_dbg_bo(vdev, bo, "pin"); | 82 | ivpu_dbg_bo(vdev, bo, "bind"); |
| 60 | 83 | ||
| 61 | sgt = drm_gem_shmem_get_pages_sgt(&bo->base); | 84 | if (bo->base.base.import_attach) |
| 85 | sgt = ivpu_bo_map_attachment(vdev, bo); | ||
| 86 | else | ||
| 87 | sgt = drm_gem_shmem_get_pages_sgt(&bo->base); | ||
| 62 | if (IS_ERR(sgt)) { | 88 | if (IS_ERR(sgt)) { |
| 63 | ret = PTR_ERR(sgt); | 89 | ret = PTR_ERR(sgt); |
| 64 | ivpu_err(vdev, "Failed to map BO in IOMMU: %d\n", ret); | 90 | ivpu_err(vdev, "Failed to map BO in IOMMU: %d\n", ret); |
| @@ -70,7 +96,7 @@ int __must_check ivpu_bo_pin(struct ivpu_bo *bo) · 5 unmodified lines | |||
| 70 | if (!bo->mmu_mapped) { | 96 | if (!bo->mmu_mapped) { |
| 71 | drm_WARN_ON(&vdev->drm, !bo->ctx); | 97 | drm_WARN_ON(&vdev->drm, !bo->ctx); |
| 72 | ret = ivpu_mmu_context_map_sgt(vdev, bo->ctx, bo->vpu_addr, sgt, | 98 | ret = ivpu_mmu_context_map_sgt(vdev, bo->ctx, bo->vpu_addr, sgt, |
| 73 | ivpu_bo_is_snooped(bo)); | 99 | ivpu_bo_is_snooped(bo), ivpu_bo_is_read_only(bo)); |
| 74 | if (ret) { | 100 | if (ret) { |
| 75 | ivpu_err(vdev, "Failed to map BO in MMU: %d\n", ret); | 101 | ivpu_err(vdev, "Failed to map BO in MMU: %d\n", ret); |
| 76 | goto unlock; | 102 | goto unlock; |
| @@ -99,9 +125,9 @@ ivpu_bo_alloc_vpu_addr(struct ivpu_bo *bo, struct ivpu_mmu_context *ctx, · 22 unmodified lines | |||
| 99 | ret = ivpu_mmu_context_insert_node(ctx, range, ivpu_bo_size(bo), &bo->mm_node); | 125 | ret = ivpu_mmu_context_insert_node(ctx, range, ivpu_bo_size(bo), &bo->mm_node); |
| 100 | if (!ret) { | 126 | if (!ret) { |
| 101 | bo->ctx = ctx; | 127 | bo->ctx = ctx; |
| 128 | bo->ctx_id = ctx->id; | ||
| 102 | bo->vpu_addr = bo->mm_node.start; | 129 | bo->vpu_addr = bo->mm_node.start; |
| 103 | } else { | 130 | ivpu_dbg_bo(vdev, bo, "vaddr"); |
| 104 | ivpu_err(vdev, "Failed to add BO to context %u: %d\n", ctx->id, ret); | ||
| 105 | } | 131 | } |
| 106 | 132 | ||
| 107 | ivpu_bo_unlock(bo); | 133 | ivpu_bo_unlock(bo); |
| @@ -115,7 +141,7 @@ static void ivpu_bo_unbind_locked(struct ivpu_bo *bo) · 7 unmodified lines | |||
| 115 | { | 141 | { |
| 116 | struct ivpu_device *vdev = ivpu_bo_to_vdev(bo); | 142 | struct ivpu_device *vdev = ivpu_bo_to_vdev(bo); |
| 117 | 143 | ||
| 118 | lockdep_assert(dma_resv_held(bo->base.base.resv) || !kref_read(&bo->base.base.refcount)); | 144 | dma_resv_assert_held(bo->base.base.resv); |
| 119 | 145 | ||
| 120 | if (bo->mmu_mapped) { | 146 | if (bo->mmu_mapped) { |
| 121 | drm_WARN_ON(&vdev->drm, !bo->ctx); | 147 | drm_WARN_ON(&vdev->drm, !bo->ctx); |
| @@ -130,13 +156,15 @@ static void ivpu_bo_unbind_locked(struct ivpu_bo *bo) · 8 unmodified lines | |||
| 130 | bo->ctx = NULL; | 156 | bo->ctx = NULL; |
| 131 | } | 157 | } |
| 132 | 158 | ||
| 133 | if (drm_gem_is_imported(&bo->base.base)) | ||
| 134 | return; | ||
| 135 | |||
| 136 | if (bo->base.sgt) { | 159 | if (bo->base.sgt) { |
| 137 | dma_unmap_sgtable(vdev->drm.dev, bo->base.sgt, DMA_BIDIRECTIONAL, 0); | 160 | if (bo->base.base.import_attach) { |
| 138 | sg_free_table(bo->base.sgt); | 161 | dma_buf_unmap_attachment(bo->base.base.import_attach, |
| 139 | kfree(bo->base.sgt); | 162 | bo->base.sgt, DMA_BIDIRECTIONAL); |
| 163 | } else { | ||
| 164 | dma_unmap_sgtable(vdev->drm.dev, bo->base.sgt, DMA_BIDIRECTIONAL, 0); | ||
| 165 | sg_free_table(bo->base.sgt); | ||
| 166 | kfree(bo->base.sgt); | ||
| 167 | } | ||
| 140 | bo->base.sgt = NULL; | 168 | bo->base.sgt = NULL; |
| 141 | } | 169 | } |
| 142 | } | 170 | } |
| @@ -182,10 +210,11 @@ struct drm_gem_object *ivpu_gem_create_object(struct drm_device *dev, size_t siz · 39 unmodified lines | |||
| 182 | struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev, | 210 | struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev, |
| 183 | struct dma_buf *dma_buf) | 211 | struct dma_buf *dma_buf) |
| 184 | { | 212 | { |
| 213 | struct ivpu_device *vdev = to_ivpu_device(dev); | ||
| 185 | struct device *attach_dev = dev->dev; | 214 | struct device *attach_dev = dev->dev; |
| 186 | struct dma_buf_attachment *attach; | 215 | struct dma_buf_attachment *attach; |
| 187 | struct sg_table *sgt; | ||
| 188 | struct drm_gem_object *obj; | 216 | struct drm_gem_object *obj; |
| 217 | struct ivpu_bo *bo; | ||
| 189 | int ret; | 218 | int ret; |
| 190 | 219 | ||
| 191 | attach = dma_buf_attach(dma_buf, attach_dev); | 220 | attach = dma_buf_attach(dma_buf, attach_dev); |
| @@ -194,25 +223,25 @@ struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev, · 2 unmodified lines | |||
| 194 | 223 | ||
| 195 | get_dma_buf(dma_buf); | 224 | get_dma_buf(dma_buf); |
| 196 | 225 | ||
| 197 | sgt = dma_buf_map_attachment_unlocked(attach, DMA_BIDIRECTIONAL); | 226 | obj = drm_gem_shmem_prime_import_sg_table(dev, attach, NULL); |
| 198 | if (IS_ERR(sgt)) { | ||
| 199 | ret = PTR_ERR(sgt); | ||
| 200 | goto fail_detach; | ||
| 201 | } | ||
| 202 | |||
| 203 | obj = drm_gem_shmem_prime_import_sg_table(dev, attach, sgt); | ||
| 204 | if (IS_ERR(obj)) { | 227 | if (IS_ERR(obj)) { |
| 205 | ret = PTR_ERR(obj); | 228 | ret = PTR_ERR(obj); |
| 206 | goto fail_unmap; | 229 | goto fail_detach; |
| 207 | } | 230 | } |
| 208 | 231 | ||
| 209 | obj->import_attach = attach; | 232 | obj->import_attach = attach; |
| 210 | obj->resv = dma_buf->resv; | 233 | obj->resv = dma_buf->resv; |
| 211 | 234 | ||
| 235 | bo = to_ivpu_bo(obj); | ||
| 236 | |||
| 237 | mutex_lock(&vdev->bo_list_lock); | ||
| 238 | list_add_tail(&bo->bo_list_node, &vdev->bo_list); | ||
| 239 | mutex_unlock(&vdev->bo_list_lock); | ||
| 240 | |||
| 241 | ivpu_dbg(vdev, BO, "import: bo %8p size %9zu\n", bo, ivpu_bo_size(bo)); | ||
| 242 | |||
| 212 | return obj; | 243 | return obj; |
| 213 | 244 | ||
| 214 | fail_unmap: | ||
| 215 | dma_buf_unmap_attachment_unlocked(attach, sgt, DMA_BIDIRECTIONAL); | ||
| 216 | fail_detach: | 245 | fail_detach: |
| 217 | dma_buf_detach(dma_buf, attach); | 246 | dma_buf_detach(dma_buf, attach); |
| 218 | dma_buf_put(dma_buf); | 247 | dma_buf_put(dma_buf); |
| @@ -220,7 +249,7 @@ fail_detach: · 1 unmodified line | |||
| 220 | return ERR_PTR(ret); | 249 | return ERR_PTR(ret); |
| 221 | } | 250 | } |
| 222 | 251 | ||
| 223 | static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 flags, u32 ctx_id) | 252 | static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 flags) |
| 224 | { | 253 | { |
| 225 | struct drm_gem_shmem_object *shmem; | 254 | struct drm_gem_shmem_object *shmem; |
| 226 | struct ivpu_bo *bo; | 255 | struct ivpu_bo *bo; |
| @@ -238,7 +267,6 @@ static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 fla · 11 unmodified lines | |||
| 238 | return ERR_CAST(shmem); | 267 | return ERR_CAST(shmem); |
| 239 | 268 | ||
| 240 | bo = to_ivpu_bo(&shmem->base); | 269 | bo = to_ivpu_bo(&shmem->base); |
| 241 | bo->ctx_id = ctx_id; | ||
| 242 | bo->base.map_wc = flags & DRM_IVPU_BO_WC; | 270 | bo->base.map_wc = flags & DRM_IVPU_BO_WC; |
| 243 | bo->flags = flags; | 271 | bo->flags = flags; |
| 244 | 272 | ||
| @@ -246,7 +274,7 @@ static struct ivpu_bo *ivpu_bo_alloc(struct ivpu_device *vdev, u64 size, u32 fla · 1 unmodified line | |||
| 246 | list_add_tail(&bo->bo_list_node, &vdev->bo_list); | 274 | list_add_tail(&bo->bo_list_node, &vdev->bo_list); |
| 247 | mutex_unlock(&vdev->bo_list_lock); | 275 | mutex_unlock(&vdev->bo_list_lock); |
| 248 | 276 | ||
| 249 | ivpu_dbg_bo(vdev, bo, "alloc"); | 277 | ivpu_dbg(vdev, BO, " alloc: bo %8p size %9llu\n", bo, size); |
| 250 | 278 | ||
| 251 | return bo; | 279 | return bo; |
| 252 | } | 280 | } |
| @@ -259,8 +287,8 @@ static int ivpu_gem_bo_open(struct drm_gem_object *obj, struct drm_file *file) · 6 unmodified lines | |||
| 259 | struct ivpu_addr_range *range; | 287 | struct ivpu_addr_range *range; |
| 260 | 288 | ||
| 261 | if (bo->ctx) { | 289 | if (bo->ctx) { |
| 262 | ivpu_warn(vdev, "Can't add BO to ctx %u: already in ctx %u\n", | 290 | ivpu_dbg(vdev, IOCTL, "Can't add BO %pe to ctx %u: already in ctx %u\n", |
| 263 | file_priv->ctx.id, bo->ctx->id); | 291 | bo, file_priv->ctx.id, bo->ctx->id); |
| 264 | return -EALREADY; | 292 | return -EALREADY; |
| 265 | } | 293 | } |
| 266 | 294 | ||
| @@ -281,23 +309,41 @@ static void ivpu_gem_bo_free(struct drm_gem_object *obj) · 14 unmodified lines | |||
| 281 | 309 | ||
| 282 | ivpu_dbg_bo(vdev, bo, "free"); | 310 | ivpu_dbg_bo(vdev, bo, "free"); |
| 283 | 311 | ||
| 312 | drm_WARN_ON(&vdev->drm, list_empty(&bo->bo_list_node)); | ||
| 313 | |||
| 284 | mutex_lock(&vdev->bo_list_lock); | 314 | mutex_lock(&vdev->bo_list_lock); |
| 285 | list_del(&bo->bo_list_node); | 315 | list_del(&bo->bo_list_node); |
| 286 | mutex_unlock(&vdev->bo_list_lock); | ||
| 287 | 316 | ||
| 288 | drm_WARN_ON(&vdev->drm, !drm_gem_is_imported(&bo->base.base) && | 317 | drm_WARN_ON(&vdev->drm, !drm_gem_is_imported(&bo->base.base) && |
| 289 | !dma_resv_test_signaled(obj->resv, DMA_RESV_USAGE_READ)); | 318 | !dma_resv_test_signaled(obj->resv, DMA_RESV_USAGE_READ)); |
| 290 | drm_WARN_ON(&vdev->drm, ivpu_bo_size(bo) == 0); | 319 | drm_WARN_ON(&vdev->drm, ivpu_bo_size(bo) == 0); |
| 291 | drm_WARN_ON(&vdev->drm, bo->base.vaddr); | 320 | drm_WARN_ON(&vdev->drm, bo->base.vaddr); |
| 292 | 321 | ||
| 322 | ivpu_bo_lock(bo); | ||
| 293 | ivpu_bo_unbind_locked(bo); | 323 | ivpu_bo_unbind_locked(bo); |
| 324 | ivpu_bo_unlock(bo); | ||
| 325 | |||
| 326 | mutex_unlock(&vdev->bo_list_lock); | ||
| 327 | |||
| 294 | drm_WARN_ON(&vdev->drm, bo->mmu_mapped); | 328 | drm_WARN_ON(&vdev->drm, bo->mmu_mapped); |
| 295 | drm_WARN_ON(&vdev->drm, bo->ctx); | 329 | drm_WARN_ON(&vdev->drm, bo->ctx); |
| 296 | 330 | ||
| 297 | drm_WARN_ON(obj->dev, refcount_read(&bo->base.pages_use_count) > 1); | 331 | drm_WARN_ON(obj->dev, refcount_read(&bo->base.pages_use_count) > 1); |
| 332 | drm_WARN_ON(obj->dev, bo->base.base.vma_node.vm_files.rb_node); | ||
| 298 | drm_gem_shmem_free(&bo->base); | 333 | drm_gem_shmem_free(&bo->base); |
| 299 | } | 334 | } |
| 300 | 335 | ||
| 336 | static enum drm_gem_object_status ivpu_gem_status(struct drm_gem_object *obj) | ||
| 337 | { | ||
| 338 | struct ivpu_bo *bo = to_ivpu_bo(obj); | ||
| 339 | enum drm_gem_object_status status = 0; | ||
| 340 | |||
| 341 | if (ivpu_bo_is_resident(bo)) | ||
| 342 | status |= DRM_GEM_OBJECT_RESIDENT; | ||
| 343 | |||
| 344 | return status; | ||
| 345 | } | ||
| 346 | |||
| 301 | static const struct drm_gem_object_funcs ivpu_gem_funcs = { | 347 | static const struct drm_gem_object_funcs ivpu_gem_funcs = { |
| 302 | .free = ivpu_gem_bo_free, | 348 | .free = ivpu_gem_bo_free, |
| 303 | .open = ivpu_gem_bo_open, | 349 | .open = ivpu_gem_bo_open, |
| @@ -308,6 +354,7 @@ static const struct drm_gem_object_funcs ivpu_gem_funcs = { · 4 unmodified lines | |||
| 308 | .vmap = drm_gem_shmem_object_vmap, | 354 | .vmap = drm_gem_shmem_object_vmap, |
| 309 | .vunmap = drm_gem_shmem_object_vunmap, | 355 | .vunmap = drm_gem_shmem_object_vunmap, |
| 310 | .mmap = drm_gem_shmem_object_mmap, | 356 | .mmap = drm_gem_shmem_object_mmap, |
| 357 | .status = ivpu_gem_status, | ||
| 311 | .vm_ops = &drm_gem_shmem_vm_ops, | 358 | .vm_ops = &drm_gem_shmem_vm_ops, |
| 312 | }; | 359 | }; |
| 313 | 360 | ||
| @@ -320,25 +367,33 @@ int ivpu_bo_create_ioctl(struct drm_device *dev, void *data, struct drm_file *fi · 6 unmodified lines | |||
| 320 | struct ivpu_bo *bo; | 367 | struct ivpu_bo *bo; |
| 321 | int ret; | 368 | int ret; |
| 322 | 369 | ||
| 323 | if (args->flags & ~DRM_IVPU_BO_FLAGS) | 370 | if (args->flags & ~DRM_IVPU_BO_FLAGS) { |
| 371 | ivpu_dbg(vdev, IOCTL, "Invalid BO flags 0x%x\n", args->flags); | ||
| 324 | return -EINVAL; | 372 | return -EINVAL; |
| 373 | } | ||
| 325 | 374 | ||
| 326 | if (size == 0) | 375 | if (size == 0) { |
| 376 | ivpu_dbg(vdev, IOCTL, "Invalid BO size %llu\n", args->size); | ||
| 327 | return -EINVAL; | 377 | return -EINVAL; |
| 378 | } | ||
| 328 | 379 | ||
| 329 | bo = ivpu_bo_alloc(vdev, size, args->flags, file_priv->ctx.id); | 380 | bo = ivpu_bo_alloc(vdev, size, args->flags); |
| 330 | if (IS_ERR(bo)) { | 381 | if (IS_ERR(bo)) { |
| 331 | ivpu_err(vdev, "Failed to allocate BO: %pe (ctx %u size %llu flags 0x%x)", | 382 | ivpu_dbg(vdev, IOCTL, "Failed to allocate BO: %pe ctx %u size %llu flags 0x%x\n", |
| 332 | bo, file_priv->ctx.id, args->size, args->flags); | 383 | bo, file_priv->ctx.id, args->size, args->flags); |
| 333 | return PTR_ERR(bo); | 384 | return PTR_ERR(bo); |
| 334 | } | 385 | } |
| 335 | 386 | ||
| 387 | drm_WARN_ON(&vdev->drm, bo->base.base.handle_count != 0); | ||
| 388 | |||
| 336 | ret = drm_gem_handle_create(file, &bo->base.base, &args->handle); | 389 | ret = drm_gem_handle_create(file, &bo->base.base, &args->handle); |
| 337 | if (ret) | 390 | if (ret) { |
| 338 | ivpu_err(vdev, "Failed to create handle for BO: %pe (ctx %u size %llu flags 0x%x)", | 391 | ivpu_dbg(vdev, IOCTL, "Failed to create handle for BO: %pe ctx %u size %llu flags 0x%x\n", |
| 339 | bo, file_priv->ctx.id, args->size, args->flags); | 392 | bo, file_priv->ctx.id, args->size, args->flags); |
| 340 | else | 393 | } else { |
| 341 | args->vpu_addr = bo->vpu_addr; | 394 | args->vpu_addr = bo->vpu_addr; |
| 395 | drm_WARN_ON(&vdev->drm, bo->base.base.handle_count != 1); | ||
| 396 | } | ||
| 342 | 397 | ||
| 343 | drm_gem_object_put(&bo->base.base); | 398 | drm_gem_object_put(&bo->base.base); |
| 344 | 399 | ||
| @@ -360,18 +415,21 @@ ivpu_bo_create(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, · 15 unmodified lines | |||
| 360 | drm_WARN_ON(&vdev->drm, !PAGE_ALIGNED(range->end)); | 415 | drm_WARN_ON(&vdev->drm, !PAGE_ALIGNED(range->end)); |
| 361 | drm_WARN_ON(&vdev->drm, !PAGE_ALIGNED(size)); | 416 | drm_WARN_ON(&vdev->drm, !PAGE_ALIGNED(size)); |
| 362 | 417 | ||
| 363 | bo = ivpu_bo_alloc(vdev, size, flags, IVPU_GLOBAL_CONTEXT_MMU_SSID); | 418 | bo = ivpu_bo_alloc(vdev, size, flags); |
| 364 | if (IS_ERR(bo)) { | 419 | if (IS_ERR(bo)) { |
| 365 | ivpu_err(vdev, "Failed to allocate BO: %pe (vpu_addr 0x%llx size %llu flags 0x%x)", | 420 | ivpu_err(vdev, "Failed to allocate BO: %pe vpu_addr 0x%llx size %llu flags 0x%x\n", |
| 366 | bo, range->start, size, flags); | 421 | bo, range->start, size, flags); |
| 367 | return NULL; | 422 | return NULL; |
| 368 | } | 423 | } |
| 369 | 424 | ||
| 370 | ret = ivpu_bo_alloc_vpu_addr(bo, ctx, range); | 425 | ret = ivpu_bo_alloc_vpu_addr(bo, ctx, range); |
| 371 | if (ret) | 426 | if (ret) { |
| 427 | ivpu_err(vdev, "Failed to allocate NPU address for BO: %pe ctx %u size %llu: %d\n", | ||
| 428 | bo, ctx->id, size, ret); | ||
| 372 | goto err_put; | 429 | goto err_put; |
| 430 | } | ||
| 373 | 431 | ||
| 374 | ret = ivpu_bo_pin(bo); | 432 | ret = ivpu_bo_bind(bo); |
| 375 | if (ret) | 433 | if (ret) |
| 376 | goto err_put; | 434 | goto err_put; |
| 377 | 435 | ||
| @@ -391,6 +449,21 @@ err_put: · 13 unmodified lines | |||
| 391 | return NULL; | 449 | return NULL; |
| 392 | } | 450 | } |
| 393 | 451 | ||
| 452 | struct ivpu_bo *ivpu_bo_create_runtime(struct ivpu_device *vdev, u64 addr, u64 size, u32 flags) | ||
| 453 | { | ||
| 454 | struct ivpu_addr_range range; | ||
| 455 | |||
| 456 | if (!ivpu_is_within_range(addr, size, &vdev->hw->ranges.runtime)) { | ||
| 457 | ivpu_err(vdev, "Invalid runtime BO address 0x%llx size %llu\n", addr, size); | ||
| 458 | return NULL; | ||
| 459 | } | ||
| 460 | |||
| 461 | if (ivpu_hw_range_init(vdev, &range, addr, size)) | ||
| 462 | return NULL; | ||
| 463 | |||
| 464 | return ivpu_bo_create(vdev, &vdev->gctx, &range, size, flags); | ||
| 465 | } | ||
| 466 | |||
| 394 | struct ivpu_bo *ivpu_bo_create_global(struct ivpu_device *vdev, u64 size, u32 flags) | 467 | struct ivpu_bo *ivpu_bo_create_global(struct ivpu_device *vdev, u64 size, u32 flags) |
| 395 | { | 468 | { |
| 396 | return ivpu_bo_create(vdev, &vdev->gctx, &vdev->hw->ranges.global, size, flags); | 469 | return ivpu_bo_create(vdev, &vdev->gctx, &vdev->hw->ranges.global, size, flags); |
| @@ -1,6 +1,6 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only */ | 1 | /* SPDX-License-Identifier: GPL-2.0-only */ |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2023 Intel Corporation | 3 | * Copyright (C) 2020-2025 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | #ifndef __IVPU_GEM_H__ | 5 | #ifndef __IVPU_GEM_H__ |
| 6 | #define __IVPU_GEM_H__ | 6 | #define __IVPU_GEM_H__ |
| @@ -24,19 +24,22 @@ struct ivpu_bo { · 17 unmodified lines | |||
| 24 | bool mmu_mapped; | 24 | bool mmu_mapped; |
| 25 | }; | 25 | }; |
| 26 | 26 | ||
| 27 | int ivpu_bo_pin(struct ivpu_bo *bo); | 27 | int ivpu_bo_bind(struct ivpu_bo *bo); |
| 28 | void ivpu_bo_unbind_all_bos_from_context(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx); | 28 | void ivpu_bo_unbind_all_bos_from_context(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx); |
| 29 | 29 | ||
| 30 | struct drm_gem_object *ivpu_gem_create_object(struct drm_device *dev, size_t size); | 30 | struct drm_gem_object *ivpu_gem_create_object(struct drm_device *dev, size_t size); |
| 31 | struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf); | 31 | struct drm_gem_object *ivpu_gem_prime_import(struct drm_device *dev, struct dma_buf *dma_buf); |
| 32 | struct ivpu_bo *ivpu_bo_create(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, | 32 | struct ivpu_bo *ivpu_bo_create(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, |
| 33 | struct ivpu_addr_range *range, u64 size, u32 flags); | 33 | struct ivpu_addr_range *range, u64 size, u32 flags); |
| 34 | struct ivpu_bo *ivpu_bo_create_runtime(struct ivpu_device *vdev, u64 addr, u64 size, u32 flags); | ||
| 34 | struct ivpu_bo *ivpu_bo_create_global(struct ivpu_device *vdev, u64 size, u32 flags); | 35 | struct ivpu_bo *ivpu_bo_create_global(struct ivpu_device *vdev, u64 size, u32 flags); |
| 35 | void ivpu_bo_free(struct ivpu_bo *bo); | 36 | void ivpu_bo_free(struct ivpu_bo *bo); |
| 36 | 37 | ||
| 37 | int ivpu_bo_create_ioctl(struct drm_device *dev, void *data, struct drm_file *file); | 38 | int ivpu_bo_create_ioctl(struct drm_device *dev, void *data, struct drm_file *file); |
| 38 | int ivpu_bo_info_ioctl(struct drm_device *dev, void *data, struct drm_file *file); | 39 | int ivpu_bo_info_ioctl(struct drm_device *dev, void *data, struct drm_file *file); |
| 39 | int ivpu_bo_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file); | 40 | int ivpu_bo_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file); |
| 41 | int ivpu_bo_create_from_userptr_ioctl(struct drm_device *dev, void *data, | ||
| 42 | struct drm_file *file); | ||
| 40 | 43 | ||
| 41 | void ivpu_bo_list(struct drm_device *dev, struct drm_printer *p); | 44 | void ivpu_bo_list(struct drm_device *dev, struct drm_printer *p); |
| 42 | void ivpu_bo_list_print(struct drm_device *dev); | 45 | void ivpu_bo_list_print(struct drm_device *dev); |
| @@ -74,6 +77,16 @@ static inline bool ivpu_bo_is_snooped(struct ivpu_bo *bo) · 31 unmodified lines | |||
| 74 | return ivpu_bo_cache_mode(bo) == DRM_IVPU_BO_CACHED; | 77 | return ivpu_bo_cache_mode(bo) == DRM_IVPU_BO_CACHED; |
| 75 | } | 78 | } |
| 76 | 79 | ||
| 80 | static inline bool ivpu_bo_is_read_only(struct ivpu_bo *bo) | ||
| 81 | { | ||
| 82 | return bo->flags & DRM_IVPU_BO_READ_ONLY; | ||
| 83 | } | ||
| 84 | |||
| 85 | static inline bool ivpu_bo_is_resident(struct ivpu_bo *bo) | ||
| 86 | { | ||
| 87 | return !!bo->base.pages; | ||
| 88 | } | ||
| 89 | |||
| 77 | static inline void *ivpu_to_cpu_addr(struct ivpu_bo *bo, u32 vpu_addr) | 90 | static inline void *ivpu_to_cpu_addr(struct ivpu_bo *bo, u32 vpu_addr) |
| 78 | { | 91 | { |
| 79 | if (vpu_addr < bo->vpu_addr) | 92 | if (vpu_addr < bo->vpu_addr) |
| @@ -96,4 +109,9 @@ static inline u32 cpu_to_vpu_addr(struct ivpu_bo *bo, void *cpu_addr) · 16 unmodified lines | |||
| 96 | return bo->vpu_addr + (cpu_addr - ivpu_bo_vaddr(bo)); | 109 | return bo->vpu_addr + (cpu_addr - ivpu_bo_vaddr(bo)); |
| 97 | } | 110 | } |
| 98 | 111 | ||
| 112 | static inline bool ivpu_bo_is_mappable(struct ivpu_bo *bo) | ||
| 113 | { | ||
| 114 | return bo->flags & DRM_IVPU_BO_MAPPABLE; | ||
| 115 | } | ||
| 116 | |||
| 99 | #endif /* __IVPU_GEM_H__ */ | 117 | #endif /* __IVPU_GEM_H__ */ |
| @@ -0,0 +1,213 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | /* | ||
| 3 | * Copyright (C) 2020-2025 Intel Corporation | ||
| 4 | */ | ||
| 5 | |||
| 6 | #include <linux/dma-buf.h> | ||
| 7 | #include <linux/err.h> | ||
| 8 | #include <linux/highmem.h> | ||
| 9 | #include <linux/mm.h> | ||
| 10 | #include <linux/mman.h> | ||
| 11 | #include <linux/scatterlist.h> | ||
| 12 | #include <linux/slab.h> | ||
| 13 | #include <linux/capability.h> | ||
| 14 | |||
| 15 | #include <drm/drm_device.h> | ||
| 16 | #include <drm/drm_file.h> | ||
| 17 | #include <drm/drm_gem.h> | ||
| 18 | |||
| 19 | #include "ivpu_drv.h" | ||
| 20 | #include "ivpu_gem.h" | ||
| 21 | |||
| 22 | static struct sg_table * | ||
| 23 | ivpu_gem_userptr_dmabuf_map(struct dma_buf_attachment *attachment, | ||
| 24 | enum dma_data_direction direction) | ||
| 25 | { | ||
| 26 | struct sg_table *sgt = attachment->dmabuf->priv; | ||
| 27 | int ret; | ||
| 28 | |||
| 29 | ret = dma_map_sgtable(attachment->dev, sgt, direction, DMA_ATTR_SKIP_CPU_SYNC); | ||
| 30 | if (ret) | ||
| 31 | return ERR_PTR(ret); | ||
| 32 | |||
| 33 | return sgt; | ||
| 34 | } | ||
| 35 | |||
| 36 | static void ivpu_gem_userptr_dmabuf_unmap(struct dma_buf_attachment *attachment, | ||
| 37 | struct sg_table *sgt, | ||
| 38 | enum dma_data_direction direction) | ||
| 39 | { | ||
| 40 | dma_unmap_sgtable(attachment->dev, sgt, direction, DMA_ATTR_SKIP_CPU_SYNC); | ||
| 41 | } | ||
| 42 | |||
| 43 | static void ivpu_gem_userptr_dmabuf_release(struct dma_buf *dma_buf) | ||
| 44 | { | ||
| 45 | struct sg_table *sgt = dma_buf->priv; | ||
| 46 | struct sg_page_iter page_iter; | ||
| 47 | struct page *page; | ||
| 48 | |||
| 49 | for_each_sgtable_page(sgt, &page_iter, 0) { | ||
| 50 | page = sg_page_iter_page(&page_iter); | ||
| 51 | unpin_user_page(page); | ||
| 52 | } | ||
| 53 | |||
| 54 | sg_free_table(sgt); | ||
| 55 | kfree(sgt); | ||
| 56 | } | ||
| 57 | |||
| 58 | static const struct dma_buf_ops ivpu_gem_userptr_dmabuf_ops = { | ||
| 59 | .map_dma_buf = ivpu_gem_userptr_dmabuf_map, | ||
| 60 | .unmap_dma_buf = ivpu_gem_userptr_dmabuf_unmap, | ||
| 61 | .release = ivpu_gem_userptr_dmabuf_release, | ||
| 62 | }; | ||
| 63 | |||
| 64 | static struct dma_buf * | ||
| 65 | ivpu_create_userptr_dmabuf(struct ivpu_device *vdev, void __user *user_ptr, | ||
| 66 | size_t size, uint32_t flags) | ||
| 67 | { | ||
| 68 | struct dma_buf_export_info exp_info = {}; | ||
| 69 | struct dma_buf *dma_buf; | ||
| 70 | struct sg_table *sgt; | ||
| 71 | struct page **pages; | ||
| 72 | unsigned long nr_pages = size >> PAGE_SHIFT; | ||
| 73 | unsigned int gup_flags = FOLL_LONGTERM; | ||
| 74 | int ret, i, pinned; | ||
| 75 | |||
| 76 | /* Add FOLL_WRITE only if the BO is not read-only */ | ||
| 77 | if (!(flags & DRM_IVPU_BO_READ_ONLY)) | ||
| 78 | gup_flags |= FOLL_WRITE; | ||
| 79 | |||
| 80 | pages = kvmalloc_array(nr_pages, sizeof(*pages), GFP_KERNEL); | ||
| 81 | if (!pages) | ||
| 82 | return ERR_PTR(-ENOMEM); | ||
| 83 | |||
| 84 | pinned = pin_user_pages_fast((unsigned long)user_ptr, nr_pages, gup_flags, pages); | ||
| 85 | if (pinned < 0) { | ||
| 86 | ret = pinned; | ||
| 87 | ivpu_dbg(vdev, IOCTL, "Failed to pin user pages: %d\n", ret); | ||
| 88 | goto free_pages_array; | ||
| 89 | } | ||
| 90 | |||
| 91 | if (pinned != nr_pages) { | ||
| 92 | ivpu_dbg(vdev, IOCTL, "Pinned %d pages, expected %lu\n", pinned, nr_pages); | ||
| 93 | ret = -EFAULT; | ||
| 94 | goto unpin_pages; | ||
| 95 | } | ||
| 96 | |||
| 97 | sgt = kmalloc(sizeof(*sgt), GFP_KERNEL); | ||
| 98 | if (!sgt) { | ||
| 99 | ret = -ENOMEM; | ||
| 100 | goto unpin_pages; | ||
| 101 | } | ||
| 102 | |||
| 103 | ret = sg_alloc_table_from_pages(sgt, pages, nr_pages, 0, size, GFP_KERNEL); | ||
| 104 | if (ret) { | ||
| 105 | ivpu_dbg(vdev, IOCTL, "Failed to create sg table: %d\n", ret); | ||
| 106 | goto free_sgt; | ||
| 107 | } | ||
| 108 | |||
| 109 | exp_info.exp_name = "ivpu_userptr_dmabuf"; | ||
| 110 | exp_info.owner = THIS_MODULE; | ||
| 111 | exp_info.ops = &ivpu_gem_userptr_dmabuf_ops; | ||
| 112 | exp_info.size = size; | ||
| 113 | exp_info.flags = O_RDWR | O_CLOEXEC; | ||
| 114 | exp_info.priv = sgt; | ||
| 115 | |||
| 116 | dma_buf = dma_buf_export(&exp_info); | ||
| 117 | if (IS_ERR(dma_buf)) { | ||
| 118 | ret = PTR_ERR(dma_buf); | ||
| 119 | ivpu_dbg(vdev, IOCTL, "Failed to export userptr dma-buf: %d\n", ret); | ||
| 120 | goto free_sg_table; | ||
| 121 | } | ||
| 122 | |||
| 123 | kvfree(pages); | ||
| 124 | return dma_buf; | ||
| 125 | |||
| 126 | free_sg_table: | ||
| 127 | sg_free_table(sgt); | ||
| 128 | free_sgt: | ||
| 129 | kfree(sgt); | ||
| 130 | unpin_pages: | ||
| 131 | for (i = 0; i < pinned; i++) | ||
| 132 | unpin_user_page(pages[i]); | ||
| 133 | free_pages_array: | ||
| 134 | kvfree(pages); | ||
| 135 | return ERR_PTR(ret); | ||
| 136 | } | ||
| 137 | |||
| 138 | static struct ivpu_bo * | ||
| 139 | ivpu_bo_create_from_userptr(struct ivpu_device *vdev, void __user *user_ptr, | ||
| 140 | size_t size, uint32_t flags) | ||
| 141 | { | ||
| 142 | struct dma_buf *dma_buf; | ||
| 143 | struct drm_gem_object *obj; | ||
| 144 | struct ivpu_bo *bo; | ||
| 145 | |||
| 146 | dma_buf = ivpu_create_userptr_dmabuf(vdev, user_ptr, size, flags); | ||
| 147 | if (IS_ERR(dma_buf)) | ||
| 148 | return ERR_CAST(dma_buf); | ||
| 149 | |||
| 150 | obj = ivpu_gem_prime_import(&vdev->drm, dma_buf); | ||
| 151 | if (IS_ERR(obj)) { | ||
| 152 | dma_buf_put(dma_buf); | ||
| 153 | return ERR_CAST(obj); | ||
| 154 | } | ||
| 155 | |||
| 156 | dma_buf_put(dma_buf); | ||
| 157 | |||
| 158 | bo = to_ivpu_bo(obj); | ||
| 159 | bo->flags = flags; | ||
| 160 | |||
| 161 | return bo; | ||
| 162 | } | ||
| 163 | |||
| 164 | int ivpu_bo_create_from_userptr_ioctl(struct drm_device *dev, void *data, struct drm_file *file) | ||
| 165 | { | ||
| 166 | struct drm_ivpu_bo_create_from_userptr *args = data; | ||
| 167 | struct ivpu_file_priv *file_priv = file->driver_priv; | ||
| 168 | struct ivpu_device *vdev = to_ivpu_device(dev); | ||
| 169 | void __user *user_ptr = u64_to_user_ptr(args->user_ptr); | ||
| 170 | struct ivpu_bo *bo; | ||
| 171 | int ret; | ||
| 172 | |||
| 173 | if (args->flags & ~(DRM_IVPU_BO_HIGH_MEM | DRM_IVPU_BO_DMA_MEM | DRM_IVPU_BO_READ_ONLY)) { | ||
| 174 | ivpu_dbg(vdev, IOCTL, "Invalid BO flags: 0x%x\n", args->flags); | ||
| 175 | return -EINVAL; | ||
| 176 | } | ||
| 177 | |||
| 178 | if (!args->user_ptr || !args->size) { | ||
| 179 | ivpu_dbg(vdev, IOCTL, "Userptr or size are zero: ptr %llx size %llu\n", | ||
| 180 | args->user_ptr, args->size); | ||
| 181 | return -EINVAL; | ||
| 182 | } | ||
| 183 | |||
| 184 | if (!PAGE_ALIGNED(args->user_ptr) || !PAGE_ALIGNED(args->size)) { | ||
| 185 | ivpu_dbg(vdev, IOCTL, "Userptr or size not page aligned: ptr %llx size %llu\n", | ||
| 186 | args->user_ptr, args->size); | ||
| 187 | return -EINVAL; | ||
| 188 | } | ||
| 189 | |||
| 190 | if (!access_ok(user_ptr, args->size)) { | ||
| 191 | ivpu_dbg(vdev, IOCTL, "Userptr is not accessible: ptr %llx size %llu\n", | ||
| 192 | args->user_ptr, args->size); | ||
| 193 | return -EFAULT; | ||
| 194 | } | ||
| 195 | |||
| 196 | bo = ivpu_bo_create_from_userptr(vdev, user_ptr, args->size, args->flags); | ||
| 197 | if (IS_ERR(bo)) | ||
| 198 | return PTR_ERR(bo); | ||
| 199 | |||
| 200 | ret = drm_gem_handle_create(file, &bo->base.base, &args->handle); | ||
| 201 | if (ret) { | ||
| 202 | ivpu_dbg(vdev, IOCTL, "Failed to create handle for BO: %pe ctx %u size %llu flags 0x%x\n", | ||
| 203 | bo, file_priv->ctx.id, args->size, args->flags); | ||
| 204 | } else { | ||
| 205 | ivpu_dbg(vdev, BO, "Created userptr BO: handle=%u vpu_addr=0x%llx size=%llu flags=0x%x\n", | ||
| 206 | args->handle, bo->vpu_addr, args->size, bo->flags); | ||
| 207 | args->vpu_addr = bo->vpu_addr; | ||
| 208 | } | ||
| 209 | |||
| 210 | drm_gem_object_put(&bo->base.base); | ||
| 211 | |||
| 212 | return ret; | ||
| 213 | } | ||
| @@ -8,6 +8,8 @@ · 7 unmodified lines | |||
| 8 | #include "ivpu_hw_btrs.h" | 8 | #include "ivpu_hw_btrs.h" |
| 9 | #include "ivpu_hw_ip.h" | 9 | #include "ivpu_hw_ip.h" |
| 10 | 10 | ||
| 11 | #include <asm/msr-index.h> | ||
| 12 | #include <asm/msr.h> | ||
| 11 | #include <linux/dmi.h> | 13 | #include <linux/dmi.h> |
| 12 | #include <linux/fault-inject.h> | 14 | #include <linux/fault-inject.h> |
| 13 | #include <linux/pm_runtime.h> | 15 | #include <linux/pm_runtime.h> |
| @@ -20,6 +22,10 @@ module_param_named_unsafe(fail_hw, ivpu_fail_hw, charp, 0444); · 6 unmodified lines | |||
| 20 | MODULE_PARM_DESC(fail_hw, "<interval>,<probability>,<space>,<times>"); | 22 | MODULE_PARM_DESC(fail_hw, "<interval>,<probability>,<space>,<times>"); |
| 21 | #endif | 23 | #endif |
| 22 | 24 | ||
| 25 | #define FW_SHARED_MEM_ALIGNMENT SZ_512K /* VPU MTRR limitation */ | ||
| 26 | |||
| 27 | #define ECC_MCA_SIGNAL_ENABLE_MASK 0xff | ||
| 28 | |||
| 23 | static char *platform_to_str(u32 platform) | 29 | static char *platform_to_str(u32 platform) |
| 24 | { | 30 | { |
| 25 | switch (platform) { | 31 | switch (platform) { |
| @@ -147,19 +153,39 @@ static void priority_bands_init(struct ivpu_device *vdev) · 121 unmodified lines | |||
| 147 | vdev->hw->hws.process_quantum[VPU_JOB_SCHEDULING_PRIORITY_BAND_REALTIME] = 200000; | 153 | vdev->hw->hws.process_quantum[VPU_JOB_SCHEDULING_PRIORITY_BAND_REALTIME] = 200000; |
| 148 | } | 154 | } |
| 149 | 155 | ||
| 156 | int ivpu_hw_range_init(struct ivpu_device *vdev, struct ivpu_addr_range *range, u64 start, u64 size) | ||
| 157 | { | ||
| 158 | u64 end; | ||
| 159 | |||
| 160 | if (!range || check_add_overflow(start, size, &end)) { | ||
| 161 | ivpu_err(vdev, "Invalid range: start 0x%llx size %llu\n", start, size); | ||
| 162 | return -EINVAL; | ||
| 163 | } | ||
| 164 | |||
| 165 | range->start = start; | ||
| 166 | range->end = end; | ||
| 167 | |||
| 168 | return 0; | ||
| 169 | } | ||
| 170 | |||
| 150 | static void memory_ranges_init(struct ivpu_device *vdev) | 171 | static void memory_ranges_init(struct ivpu_device *vdev) |
| 151 | { | 172 | { |
| 152 | if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX) { | 173 | if (ivpu_hw_ip_gen(vdev) == IVPU_HW_IP_37XX) { |
| 153 | ivpu_hw_range_init(&vdev->hw->ranges.global, 0x80000000, SZ_512M); | 174 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.runtime, 0x84800000, SZ_64M); |
| 154 | ivpu_hw_range_init(&vdev->hw->ranges.user, 0x88000000, 511 * SZ_1M); | 175 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.global, 0x90000000, SZ_256M); |
| 155 | ivpu_hw_range_init(&vdev->hw->ranges.shave, 0x180000000, SZ_2G); | 176 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.user, 0xa0000000, 511 * SZ_1M); |
| 156 | ivpu_hw_range_init(&vdev->hw->ranges.dma, 0x200000000, SZ_128G); | 177 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.shave, 0x180000000, SZ_2G); |
| 178 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.dma, 0x200000000, SZ_128G); | ||
| 157 | } else { | 179 | } else { |
| 158 | ivpu_hw_range_init(&vdev->hw->ranges.global, 0x80000000, SZ_512M); | 180 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.runtime, 0x80000000, SZ_64M); |
| 159 | ivpu_hw_range_init(&vdev->hw->ranges.shave, 0x80000000, SZ_2G); | 181 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.global, 0x90000000, SZ_256M); |
| 160 | ivpu_hw_range_init(&vdev->hw->ranges.user, 0x100000000, SZ_256G); | 182 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.shave, 0x80000000, SZ_2G); |
| 183 | ivpu_hw_range_init(vdev, &vdev->hw->ranges.user, 0x100000000, SZ_256G); | ||
| 161 | vdev->hw->ranges.dma = vdev->hw->ranges.user; | 184 | vdev->hw->ranges.dma = vdev->hw->ranges.user; |
| 162 | } | 185 | } |
| 186 | |||
| 187 | drm_WARN_ON(&vdev->drm, !IS_ALIGNED(vdev->hw->ranges.global.start, | ||
| 188 | FW_SHARED_MEM_ALIGNMENT)); | ||
| 163 | } | 189 | } |
| 164 | 190 | ||
| 165 | static int wp_enable(struct ivpu_device *vdev) | 191 | static int wp_enable(struct ivpu_device *vdev) |
| @@ -373,3 +399,22 @@ irqreturn_t ivpu_hw_irq_handler(int irq, void *ptr) · 207 unmodified lines | |||
| 373 | pm_runtime_mark_last_busy(vdev->drm.dev); | 399 | pm_runtime_mark_last_busy(vdev->drm.dev); |
| 374 | return IRQ_HANDLED; | 400 | return IRQ_HANDLED; |
| 375 | } | 401 | } |
| 402 | |||
| 403 | bool ivpu_hw_uses_ecc_mca_signal(struct ivpu_device *vdev) | ||
| 404 | { | ||
| 405 | unsigned long long msr_integrity_caps; | ||
| 406 | int ret; | ||
| 407 | |||
| 408 | if (ivpu_hw_ip_gen(vdev) < IVPU_HW_IP_50XX) | ||
| 409 | return false; | ||
| 410 | |||
| 411 | ret = rdmsrq_safe(MSR_INTEGRITY_CAPS, &msr_integrity_caps); | ||
| 412 | if (ret) { | ||
| 413 | ivpu_warn(vdev, "Error reading MSR_INTEGRITY_CAPS: %d", ret); | ||
| 414 | return false; | ||
| 415 | } | ||
| 416 | |||
| 417 | ivpu_dbg(vdev, MISC, "MSR_INTEGRITY_CAPS: 0x%llx\n", msr_integrity_caps); | ||
| 418 | |||
| 419 | return msr_integrity_caps & ECC_MCA_SIGNAL_ENABLE_MASK; | ||
| 420 | } | ||
| @@ -21,6 +21,7 @@ struct ivpu_hw_info { · 20 unmodified lines | |||
| 21 | bool (*ip_irq_handler)(struct ivpu_device *vdev, int irq); | 21 | bool (*ip_irq_handler)(struct ivpu_device *vdev, int irq); |
| 22 | } irq; | 22 | } irq; |
| 23 | struct { | 23 | struct { |
| 24 | struct ivpu_addr_range runtime; | ||
| 24 | struct ivpu_addr_range global; | 25 | struct ivpu_addr_range global; |
| 25 | struct ivpu_addr_range user; | 26 | struct ivpu_addr_range user; |
| 26 | struct ivpu_addr_range shave; | 27 | struct ivpu_addr_range shave; |
| @@ -51,6 +52,8 @@ struct ivpu_hw_info { · 24 unmodified lines | |||
| 51 | }; | 52 | }; |
| 52 | 53 | ||
| 53 | int ivpu_hw_init(struct ivpu_device *vdev); | 54 | int ivpu_hw_init(struct ivpu_device *vdev); |
| 55 | int ivpu_hw_range_init(struct ivpu_device *vdev, struct ivpu_addr_range *range, u64 start, | ||
| 56 | u64 size); | ||
| 54 | int ivpu_hw_power_up(struct ivpu_device *vdev); | 57 | int ivpu_hw_power_up(struct ivpu_device *vdev); |
| 55 | int ivpu_hw_power_down(struct ivpu_device *vdev); | 58 | int ivpu_hw_power_down(struct ivpu_device *vdev); |
| 56 | int ivpu_hw_reset(struct ivpu_device *vdev); | 59 | int ivpu_hw_reset(struct ivpu_device *vdev); |
| @@ -60,6 +63,7 @@ void ivpu_irq_handlers_init(struct ivpu_device *vdev); · 3 unmodified lines | |||
| 60 | void ivpu_hw_irq_enable(struct ivpu_device *vdev); | 63 | void ivpu_hw_irq_enable(struct ivpu_device *vdev); |
| 61 | void ivpu_hw_irq_disable(struct ivpu_device *vdev); | 64 | void ivpu_hw_irq_disable(struct ivpu_device *vdev); |
| 62 | irqreturn_t ivpu_hw_irq_handler(int irq, void *ptr); | 65 | irqreturn_t ivpu_hw_irq_handler(int irq, void *ptr); |
| 66 | bool ivpu_hw_uses_ecc_mca_signal(struct ivpu_device *vdev); | ||
| 63 | 67 | ||
| 64 | static inline u32 ivpu_hw_btrs_irq_handler(struct ivpu_device *vdev, int irq) | 68 | static inline u32 ivpu_hw_btrs_irq_handler(struct ivpu_device *vdev, int irq) |
| 65 | { | 69 | { |
| @@ -71,12 +75,6 @@ static inline u32 ivpu_hw_ip_irq_handler(struct ivpu_device *vdev, int irq) · 5 unmodified lines | |||
| 71 | return vdev->hw->irq.ip_irq_handler(vdev, irq); | 75 | return vdev->hw->irq.ip_irq_handler(vdev, irq); |
| 72 | } | 76 | } |
| 73 | 77 | ||
| 74 | static inline void ivpu_hw_range_init(struct ivpu_addr_range *range, u64 start, u64 size) | ||
| 75 | { | ||
| 76 | range->start = start; | ||
| 77 | range->end = start + size; | ||
| 78 | } | ||
| 79 | |||
| 80 | static inline u64 ivpu_hw_range_size(const struct ivpu_addr_range *range) | 78 | static inline u64 ivpu_hw_range_size(const struct ivpu_addr_range *range) |
| 81 | { | 79 | { |
| 82 | return range->end - range->start; | 80 | return range->end - range->start; |
| @@ -321,6 +321,14 @@ static int wait_for_pll_lock(struct ivpu_device *vdev, bool enable) · 320 unmodified lines | |||
| 321 | return REGB_POLL_FLD(VPU_HW_BTRS_MTL_PLL_STATUS, LOCK, exp_val, PLL_TIMEOUT_US); | 321 | return REGB_POLL_FLD(VPU_HW_BTRS_MTL_PLL_STATUS, LOCK, exp_val, PLL_TIMEOUT_US); |
| 322 | } | 322 | } |
| 323 | 323 | ||
| 324 | static int wait_for_cdyn_deassert(struct ivpu_device *vdev) | ||
| 325 | { | ||
| 326 | if (ivpu_hw_btrs_gen(vdev) == IVPU_HW_BTRS_MTL) | ||
| 327 | return 0; | ||
| 328 | |||
| 329 | return REGB_POLL_FLD(VPU_HW_BTRS_LNL_CDYN, CDYN, 0, PLL_TIMEOUT_US); | ||
| 330 | } | ||
| 331 | |||
| 324 | int ivpu_hw_btrs_wp_drive(struct ivpu_device *vdev, bool enable) | 332 | int ivpu_hw_btrs_wp_drive(struct ivpu_device *vdev, bool enable) |
| 325 | { | 333 | { |
| 326 | struct wp_request wp; | 334 | struct wp_request wp; |
| @@ -354,6 +362,14 @@ int ivpu_hw_btrs_wp_drive(struct ivpu_device *vdev, bool enable) · 27 unmodified lines | |||
| 354 | return ret; | 362 | return ret; |
| 355 | } | 363 | } |
| 356 | 364 | ||
| 365 | if (!enable) { | ||
| 366 | ret = wait_for_cdyn_deassert(vdev); | ||
| 367 | if (ret) { | ||
| 368 | ivpu_err(vdev, "Timed out waiting for CDYN deassert\n"); | ||
| 369 | return ret; | ||
| 370 | } | ||
| 371 | } | ||
| 372 | |||
| 357 | return 0; | 373 | return 0; |
| 358 | } | 374 | } |
| 359 | 375 | ||
| @@ -673,7 +689,7 @@ bool ivpu_hw_btrs_irq_handler_lnl(struct ivpu_device *vdev, int irq) · 313 unmodified lines | |||
| 673 | 689 | ||
| 674 | if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, SURV_ERR, status)) { | 690 | if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, SURV_ERR, status)) { |
| 675 | ivpu_dbg(vdev, IRQ, "Survivability IRQ\n"); | 691 | ivpu_dbg(vdev, IRQ, "Survivability IRQ\n"); |
| 676 | queue_work(system_wq, &vdev->irq_dct_work); | 692 | queue_work(system_percpu_wq, &vdev->irq_dct_work); |
| 677 | } | 693 | } |
| 678 | 694 | ||
| 679 | if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, FREQ_CHANGE, status)) { | 695 | if (REG_TEST_FLD(VPU_HW_BTRS_LNL_INTERRUPT_STAT, FREQ_CHANGE, status)) { |
| @@ -752,7 +768,7 @@ int ivpu_hw_btrs_dct_get_request(struct ivpu_device *vdev, bool *enable) · 72 unmodified lines | |||
| 752 | } | 768 | } |
| 753 | } | 769 | } |
| 754 | 770 | ||
| 755 | void ivpu_hw_btrs_dct_set_status(struct ivpu_device *vdev, bool enable, u32 active_percent) | 771 | void ivpu_hw_btrs_dct_set_status(struct ivpu_device *vdev, bool enable, u8 active_percent) |
| 756 | { | 772 | { |
| 757 | u32 val = 0; | 773 | u32 val = 0; |
| 758 | u32 cmd = enable ? DCT_ENABLE : DCT_DISABLE; | 774 | u32 cmd = enable ? DCT_ENABLE : DCT_DISABLE; |
| @@ -36,7 +36,7 @@ u32 ivpu_hw_btrs_dpu_freq_get(struct ivpu_device *vdev); · 35 unmodified lines | |||
| 36 | bool ivpu_hw_btrs_irq_handler_mtl(struct ivpu_device *vdev, int irq); | 36 | bool ivpu_hw_btrs_irq_handler_mtl(struct ivpu_device *vdev, int irq); |
| 37 | bool ivpu_hw_btrs_irq_handler_lnl(struct ivpu_device *vdev, int irq); | 37 | bool ivpu_hw_btrs_irq_handler_lnl(struct ivpu_device *vdev, int irq); |
| 38 | int ivpu_hw_btrs_dct_get_request(struct ivpu_device *vdev, bool *enable); | 38 | int ivpu_hw_btrs_dct_get_request(struct ivpu_device *vdev, bool *enable); |
| 39 | void ivpu_hw_btrs_dct_set_status(struct ivpu_device *vdev, bool enable, u32 active_percent); | 39 | void ivpu_hw_btrs_dct_set_status(struct ivpu_device *vdev, bool enable, u8 active_percent); |
| 40 | u32 ivpu_hw_btrs_telemetry_offset_get(struct ivpu_device *vdev); | 40 | u32 ivpu_hw_btrs_telemetry_offset_get(struct ivpu_device *vdev); |
| 41 | u32 ivpu_hw_btrs_telemetry_size_get(struct ivpu_device *vdev); | 41 | u32 ivpu_hw_btrs_telemetry_size_get(struct ivpu_device *vdev); |
| 42 | u32 ivpu_hw_btrs_telemetry_enable_get(struct ivpu_device *vdev); | 42 | u32 ivpu_hw_btrs_telemetry_enable_get(struct ivpu_device *vdev); |
| @@ -74,6 +74,9 @@ · 73 unmodified lines | |||
| 74 | #define VPU_HW_BTRS_LNL_PLL_FREQ 0x00000148u | 74 | #define VPU_HW_BTRS_LNL_PLL_FREQ 0x00000148u |
| 75 | #define VPU_HW_BTRS_LNL_PLL_FREQ_RATIO_MASK GENMASK(15, 0) | 75 | #define VPU_HW_BTRS_LNL_PLL_FREQ_RATIO_MASK GENMASK(15, 0) |
| 76 | 76 | ||
| 77 | #define VPU_HW_BTRS_LNL_CDYN 0x0000014cu | ||
| 78 | #define VPU_HW_BTRS_LNL_CDYN_CDYN_MASK GENMASK(15, 0) | ||
| 79 | |||
| 77 | #define VPU_HW_BTRS_LNL_TILE_FUSE 0x00000150u | 80 | #define VPU_HW_BTRS_LNL_TILE_FUSE 0x00000150u |
| 78 | #define VPU_HW_BTRS_LNL_TILE_FUSE_VALID_MASK BIT_MASK(0) | 81 | #define VPU_HW_BTRS_LNL_TILE_FUSE_VALID_MASK BIT_MASK(0) |
| 79 | #define VPU_HW_BTRS_LNL_TILE_FUSE_CONFIG_MASK GENMASK(6, 1) | 82 | #define VPU_HW_BTRS_LNL_TILE_FUSE_CONFIG_MASK GENMASK(6, 1) |
| @@ -691,6 +691,13 @@ static void pwr_island_delay_set(struct ivpu_device *vdev) · 690 unmodified lines | |||
| 691 | status = high ? 46 : 3; | 691 | status = high ? 46 : 3; |
| 692 | break; | 692 | break; |
| 693 | 693 | ||
| 694 | case PCI_DEVICE_ID_NVL: | ||
| 695 | post = high ? 198 : 17; | ||
| 696 | post1 = 0; | ||
| 697 | post2 = high ? 198 : 17; | ||
| 698 | status = 0; | ||
| 699 | break; | ||
| 700 | |||
| 694 | default: | 701 | default: |
| 695 | dump_stack(); | 702 | dump_stack(); |
| 696 | ivpu_err(vdev, "Unknown device ID\n"); | 703 | ivpu_err(vdev, "Unknown device ID\n"); |
| @@ -889,6 +896,9 @@ static int soc_cpu_drive_40xx(struct ivpu_device *vdev, bool enable) · 192 unmodified lines | |||
| 889 | 896 | ||
| 890 | static int soc_cpu_enable(struct ivpu_device *vdev) | 897 | static int soc_cpu_enable(struct ivpu_device *vdev) |
| 891 | { | 898 | { |
| 899 | if (ivpu_hw_ip_gen(vdev) >= IVPU_HW_IP_60XX) | ||
| 900 | return 0; | ||
| 901 | |||
| 892 | return soc_cpu_drive_40xx(vdev, true); | 902 | return soc_cpu_drive_40xx(vdev, true); |
| 893 | } | 903 | } |
| 894 | 904 | ||
| @@ -459,7 +459,7 @@ void ivpu_ipc_irq_handler(struct ivpu_device *vdev) · 458 unmodified lines | |||
| 459 | } | 459 | } |
| 460 | } | 460 | } |
| 461 | 461 | ||
| 462 | queue_work(system_wq, &vdev->irq_ipc_work); | 462 | queue_work(system_percpu_wq, &vdev->irq_ipc_work); |
| 463 | } | 463 | } |
| 464 | 464 | ||
| 465 | void ivpu_ipc_irq_work_fn(struct work_struct *work) | 465 | void ivpu_ipc_irq_work_fn(struct work_struct *work) |
| @@ -34,22 +34,20 @@ static void ivpu_cmdq_ring_db(struct ivpu_device *vdev, struct ivpu_cmdq *cmdq) · 33 unmodified lines | |||
| 34 | static int ivpu_preemption_buffers_create(struct ivpu_device *vdev, | 34 | static int ivpu_preemption_buffers_create(struct ivpu_device *vdev, |
| 35 | struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) | 35 | struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) |
| 36 | { | 36 | { |
| 37 | u64 primary_size = ALIGN(vdev->fw->primary_preempt_buf_size, PAGE_SIZE); | 37 | if (ivpu_fw_preempt_buf_size(vdev) == 0) |
| 38 | u64 secondary_size = ALIGN(vdev->fw->secondary_preempt_buf_size, PAGE_SIZE); | ||
| 39 | |||
| 40 | if (vdev->fw->sched_mode != VPU_SCHEDULING_MODE_HW || | ||
| 41 | ivpu_test_mode & IVPU_TEST_MODE_MIP_DISABLE) | ||
| 42 | return 0; | 38 | return 0; |
| 43 | 39 | ||
| 44 | cmdq->primary_preempt_buf = ivpu_bo_create(vdev, &file_priv->ctx, &vdev->hw->ranges.user, | 40 | cmdq->primary_preempt_buf = ivpu_bo_create(vdev, &file_priv->ctx, &vdev->hw->ranges.user, |
| 45 | primary_size, DRM_IVPU_BO_WC); | 41 | vdev->fw->primary_preempt_buf_size, |
| 42 | DRM_IVPU_BO_WC); | ||
| 46 | if (!cmdq->primary_preempt_buf) { | 43 | if (!cmdq->primary_preempt_buf) { |
| 47 | ivpu_err(vdev, "Failed to create primary preemption buffer\n"); | 44 | ivpu_err(vdev, "Failed to create primary preemption buffer\n"); |
| 48 | return -ENOMEM; | 45 | return -ENOMEM; |
| 49 | } | 46 | } |
| 50 | 47 | ||
| 51 | cmdq->secondary_preempt_buf = ivpu_bo_create(vdev, &file_priv->ctx, &vdev->hw->ranges.dma, | 48 | cmdq->secondary_preempt_buf = ivpu_bo_create(vdev, &file_priv->ctx, &vdev->hw->ranges.dma, |
| 52 | secondary_size, DRM_IVPU_BO_WC); | 49 | vdev->fw->secondary_preempt_buf_size, |
| 50 | DRM_IVPU_BO_WC); | ||
| 53 | if (!cmdq->secondary_preempt_buf) { | 51 | if (!cmdq->secondary_preempt_buf) { |
| 54 | ivpu_err(vdev, "Failed to create secondary preemption buffer\n"); | 52 | ivpu_err(vdev, "Failed to create secondary preemption buffer\n"); |
| 55 | goto err_free_primary; | 53 | goto err_free_primary; |
| @@ -66,20 +64,39 @@ err_free_primary: · 10 unmodified lines | |||
| 66 | static void ivpu_preemption_buffers_free(struct ivpu_device *vdev, | 64 | static void ivpu_preemption_buffers_free(struct ivpu_device *vdev, |
| 67 | struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) | 65 | struct ivpu_file_priv *file_priv, struct ivpu_cmdq *cmdq) |
| 68 | { | 66 | { |
| 69 | if (vdev->fw->sched_mode != VPU_SCHEDULING_MODE_HW) | ||
| 70 | return; | ||
| 71 | |||
| 72 | if (cmdq->primary_preempt_buf) | 67 | if (cmdq->primary_preempt_buf) |
| 73 | ivpu_bo_free(cmdq->primary_preempt_buf); | 68 | ivpu_bo_free(cmdq->primary_preempt_buf); |
| 74 | if (cmdq->secondary_preempt_buf) | 69 | if (cmdq->secondary_preempt_buf) |
| 75 | ivpu_bo_free(cmdq->secondary_preempt_buf); | 70 | ivpu_bo_free(cmdq->secondary_preempt_buf); |
| 76 | } | 71 | } |
| 77 | 72 | ||
| 73 | static int ivpu_preemption_job_init(struct ivpu_device *vdev, struct ivpu_file_priv *file_priv, | ||
| 74 | struct ivpu_cmdq *cmdq, struct ivpu_job *job) | ||
| 75 | { | ||
| 76 | int ret; | ||
| 77 | |||
| 78 | /* Use preemption buffer provided by the user space */ | ||
| 79 | if (job->primary_preempt_buf) | ||
| 80 | return 0; | ||
| 81 | |||
| 82 | if (!cmdq->primary_preempt_buf) { | ||
| 83 | /* Allocate per command queue preemption buffers */ | ||
| 84 | ret = ivpu_preemption_buffers_create(vdev, file_priv, cmdq); | ||
| 85 | if (ret) | ||
| 86 | return ret; | ||
| 87 | } | ||
| 88 | |||
| 89 | /* Use preemption buffers allocated by the kernel */ | ||
| 90 | job->primary_preempt_buf = cmdq->primary_preempt_buf; | ||
| 91 | job->secondary_preempt_buf = cmdq->secondary_preempt_buf; | ||
| 92 | |||
| 93 | return 0; | ||
| 94 | } | ||
| 95 | |||
| 78 | static struct ivpu_cmdq *ivpu_cmdq_alloc(struct ivpu_file_priv *file_priv) | 96 | static struct ivpu_cmdq *ivpu_cmdq_alloc(struct ivpu_file_priv *file_priv) |
| 79 | { | 97 | { |
| 80 | struct ivpu_device *vdev = file_priv->vdev; | 98 | struct ivpu_device *vdev = file_priv->vdev; |
| 81 | struct ivpu_cmdq *cmdq; | 99 | struct ivpu_cmdq *cmdq; |
| 82 | int ret; | ||
| 83 | 100 | ||
| 84 | cmdq = kzalloc(sizeof(*cmdq), GFP_KERNEL); | 101 | cmdq = kzalloc(sizeof(*cmdq), GFP_KERNEL); |
| 85 | if (!cmdq) | 102 | if (!cmdq) |
| @@ -89,10 +106,6 @@ static struct ivpu_cmdq *ivpu_cmdq_alloc(struct ivpu_file_priv *file_priv) · 3 unmodified lines | |||
| 89 | if (!cmdq->mem) | 106 | if (!cmdq->mem) |
| 90 | goto err_free_cmdq; | 107 | goto err_free_cmdq; |
| 91 | 108 | ||
| 92 | ret = ivpu_preemption_buffers_create(vdev, file_priv, cmdq); | ||
| 93 | if (ret) | ||
| 94 | ivpu_warn(vdev, "Failed to allocate preemption buffers, preemption limited\n"); | ||
| 95 | |||
| 96 | return cmdq; | 109 | return cmdq; |
| 97 | 110 | ||
| 98 | err_free_cmdq: | 111 | err_free_cmdq: |
| @@ -219,11 +232,13 @@ static int ivpu_register_db(struct ivpu_file_priv *file_priv, struct ivpu_cmdq * · 120 unmodified lines | |||
| 219 | ret = ivpu_jsm_register_db(vdev, file_priv->ctx.id, cmdq->db_id, | 232 | ret = ivpu_jsm_register_db(vdev, file_priv->ctx.id, cmdq->db_id, |
| 220 | cmdq->mem->vpu_addr, ivpu_bo_size(cmdq->mem)); | 233 | cmdq->mem->vpu_addr, ivpu_bo_size(cmdq->mem)); |
| 221 | 234 | ||
| 222 | if (!ret) | 235 | if (!ret) { |
| 223 | ivpu_dbg(vdev, JOB, "DB %d registered to cmdq %d ctx %d priority %d\n", | 236 | ivpu_dbg(vdev, JOB, "DB %d registered to cmdq %d ctx %d priority %d\n", |
| 224 | cmdq->db_id, cmdq->id, file_priv->ctx.id, cmdq->priority); | 237 | cmdq->db_id, cmdq->id, file_priv->ctx.id, cmdq->priority); |
| 225 | else | 238 | } else { |
| 226 | xa_erase(&vdev->db_xa, cmdq->db_id); | 239 | xa_erase(&vdev->db_xa, cmdq->db_id); |
| 240 | cmdq->db_id = 0; | ||
| 241 | } | ||
| 227 | 242 | ||
| 228 | return ret; | 243 | return ret; |
| 229 | } | 244 | } |
| @@ -333,7 +348,7 @@ static struct ivpu_cmdq *ivpu_cmdq_acquire(struct ivpu_file_priv *file_priv, u32 · 103 unmodified lines | |||
| 333 | 348 | ||
| 334 | cmdq = xa_load(&file_priv->cmdq_xa, cmdq_id); | 349 | cmdq = xa_load(&file_priv->cmdq_xa, cmdq_id); |
| 335 | if (!cmdq) { | 350 | if (!cmdq) { |
| 336 | ivpu_warn_ratelimited(vdev, "Failed to find command queue with ID: %u\n", cmdq_id); | 351 | ivpu_dbg(vdev, IOCTL, "Failed to find command queue with ID: %u\n", cmdq_id); |
| 337 | return NULL; | 352 | return NULL; |
| 338 | } | 353 | } |
| 339 | 354 | ||
| @@ -427,17 +442,14 @@ static int ivpu_cmdq_push_job(struct ivpu_cmdq *cmdq, struct ivpu_job *job) · 87 unmodified lines | |||
| 427 | if (unlikely(ivpu_test_mode & IVPU_TEST_MODE_NULL_SUBMISSION)) | 442 | if (unlikely(ivpu_test_mode & IVPU_TEST_MODE_NULL_SUBMISSION)) |
| 428 | entry->flags = VPU_JOB_FLAGS_NULL_SUBMISSION_MASK; | 443 | entry->flags = VPU_JOB_FLAGS_NULL_SUBMISSION_MASK; |
| 429 | 444 | ||
| 430 | if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) { | 445 | if (job->primary_preempt_buf) { |
| 431 | if (cmdq->primary_preempt_buf) { | 446 | entry->primary_preempt_buf_addr = job->primary_preempt_buf->vpu_addr; |
| 432 | entry->primary_preempt_buf_addr = cmdq->primary_preempt_buf->vpu_addr; | 447 | entry->primary_preempt_buf_size = ivpu_bo_size(job->primary_preempt_buf); |
| 433 | entry->primary_preempt_buf_size = ivpu_bo_size(cmdq->primary_preempt_buf); | 448 | } |
| 434 | } | ||
| 435 | 449 | ||
| 436 | if (cmdq->secondary_preempt_buf) { | 450 | if (job->secondary_preempt_buf) { |
| 437 | entry->secondary_preempt_buf_addr = cmdq->secondary_preempt_buf->vpu_addr; | 451 | entry->secondary_preempt_buf_addr = job->secondary_preempt_buf->vpu_addr; |
| 438 | entry->secondary_preempt_buf_size = | 452 | entry->secondary_preempt_buf_size = ivpu_bo_size(job->secondary_preempt_buf); |
| 439 | ivpu_bo_size(cmdq->secondary_preempt_buf); | ||
| 440 | } | ||
| 441 | } | 453 | } |
| 442 | 454 | ||
| 443 | wmb(); /* Ensure that tail is updated after filling entry */ | 455 | wmb(); /* Ensure that tail is updated after filling entry */ |
| @@ -522,7 +534,7 @@ ivpu_job_create(struct ivpu_file_priv *file_priv, u32 engine_idx, u32 bo_count) · 78 unmodified lines | |||
| 522 | job->bo_count = bo_count; | 534 | job->bo_count = bo_count; |
| 523 | job->done_fence = ivpu_fence_create(vdev); | 535 | job->done_fence = ivpu_fence_create(vdev); |
| 524 | if (!job->done_fence) { | 536 | if (!job->done_fence) { |
| 525 | ivpu_warn_ratelimited(vdev, "Failed to create a fence\n"); | 537 | ivpu_err(vdev, "Failed to create a fence\n"); |
| 526 | goto err_free_job; | 538 | goto err_free_job; |
| 527 | } | 539 | } |
| 528 | 540 | ||
| @@ -552,21 +564,26 @@ static struct ivpu_job *ivpu_job_remove_from_submitted_jobs(struct ivpu_device * · 23 unmodified lines | |||
| 552 | return job; | 564 | return job; |
| 553 | } | 565 | } |
| 554 | 566 | ||
| 555 | static int ivpu_job_signal_and_destroy(struct ivpu_device *vdev, u32 job_id, u32 job_status) | 567 | bool ivpu_job_handle_engine_error(struct ivpu_device *vdev, u32 job_id, u32 job_status) |
| 556 | { | 568 | { |
| 557 | struct ivpu_job *job; | ||
| 558 | |||
| 559 | lockdep_assert_held(&vdev->submitted_jobs_lock); | 569 | lockdep_assert_held(&vdev->submitted_jobs_lock); |
| 560 | 570 | ||
| 561 | job = xa_load(&vdev->submitted_jobs_xa, job_id); | 571 | switch (job_status) { |
| 562 | if (!job) | 572 | case VPU_JSM_STATUS_PROCESSING_ERR: |
| 563 | return -ENOENT; | 573 | case VPU_JSM_STATUS_ENGINE_RESET_REQUIRED_MIN ... VPU_JSM_STATUS_ENGINE_RESET_REQUIRED_MAX: |
| 574 | { | ||
| 575 | struct ivpu_job *job = xa_load(&vdev->submitted_jobs_xa, job_id); | ||
| 564 | 576 | ||
| 565 | if (job_status == VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HW) { | 577 | if (!job) |
| 578 | return false; | ||
| 579 | |||
| 580 | /* Trigger an engine reset */ | ||
| 566 | guard(mutex)(&job->file_priv->lock); | 581 | guard(mutex)(&job->file_priv->lock); |
| 567 | 582 | ||
| 583 | job->job_status = job_status; | ||
| 584 | |||
| 568 | if (job->file_priv->has_mmu_faults) | 585 | if (job->file_priv->has_mmu_faults) |
| 569 | return 0; | 586 | return false; |
| 570 | 587 | ||
| 571 | /* | 588 | /* |
| 572 | * Mark context as faulty and defer destruction of the job to jobs abort thread | 589 | * Mark context as faulty and defer destruction of the job to jobs abort thread |
| @@ -574,23 +591,43 @@ static int ivpu_job_signal_and_destroy(struct ivpu_device *vdev, u32 job_id, u32 · 1 unmodified line | |||
| 574 | * status and ensure both are handled in the same way | 591 | * status and ensure both are handled in the same way |
| 575 | */ | 592 | */ |
| 576 | job->file_priv->has_mmu_faults = true; | 593 | job->file_priv->has_mmu_faults = true; |
| 577 | queue_work(system_wq, &vdev->context_abort_work); | 594 | queue_work(system_percpu_wq, &vdev->context_abort_work); |
| 578 | return 0; | 595 | return true; |
| 579 | } | 596 | } |
| 597 | default: | ||
| 598 | /* Complete job with error status, engine reset not required */ | ||
| 599 | break; | ||
| 600 | } | ||
| 601 | |||
| 602 | return false; | ||
| 603 | } | ||
| 580 | 604 | ||
| 581 | job = ivpu_job_remove_from_submitted_jobs(vdev, job_id); | 605 | static int ivpu_job_signal_and_destroy(struct ivpu_device *vdev, u32 job_id, u32 job_status) |
| 606 | { | ||
| 607 | struct ivpu_job *job; | ||
| 608 | |||
| 609 | lockdep_assert_held(&vdev->submitted_jobs_lock); | ||
| 610 | |||
| 611 | job = xa_load(&vdev->submitted_jobs_xa, job_id); | ||
| 582 | if (!job) | 612 | if (!job) |
| 583 | return -ENOENT; | 613 | return -ENOENT; |
| 584 | 614 | ||
| 585 | if (job->file_priv->has_mmu_faults) | 615 | ivpu_job_remove_from_submitted_jobs(vdev, job_id); |
| 586 | job_status = DRM_IVPU_JOB_STATUS_ABORTED; | ||
| 587 | 616 | ||
| 588 | job->bos[CMD_BUF_IDX]->job_status = job_status; | 617 | if (job->job_status == VPU_JSM_STATUS_SUCCESS) { |
| 618 | if (job->file_priv->has_mmu_faults) | ||
| 619 | job->job_status = DRM_IVPU_JOB_STATUS_ABORTED; | ||
| 620 | else | ||
| 621 | job->job_status = job_status; | ||
| 622 | } | ||
| 623 | |||
| 624 | job->bos[CMD_BUF_IDX]->job_status = job->job_status; | ||
| 589 | dma_fence_signal(job->done_fence); | 625 | dma_fence_signal(job->done_fence); |
| 590 | 626 | ||
| 591 | trace_job("done", job); | 627 | trace_job("done", job); |
| 592 | ivpu_dbg(vdev, JOB, "Job complete: id %3u ctx %2d cmdq_id %u engine %d status 0x%x\n", | 628 | ivpu_dbg(vdev, JOB, "Job complete: id %3u ctx %2d cmdq_id %u engine %d status 0x%x\n", |
| 593 | job->job_id, job->file_priv->ctx.id, job->cmdq_id, job->engine_idx, job_status); | 629 | job->job_id, job->file_priv->ctx.id, job->cmdq_id, job->engine_idx, |
| 630 | job->job_status); | ||
| 594 | 631 | ||
| 595 | ivpu_job_destroy(job); | 632 | ivpu_job_destroy(job); |
| 596 | ivpu_stop_job_timeout_detection(vdev); | 633 | ivpu_stop_job_timeout_detection(vdev); |
| @@ -650,7 +687,6 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) · 53 unmodified lines | |||
| 650 | else | 687 | else |
| 651 | cmdq = ivpu_cmdq_acquire(file_priv, cmdq_id); | 688 | cmdq = ivpu_cmdq_acquire(file_priv, cmdq_id); |
| 652 | if (!cmdq) { | 689 | if (!cmdq) { |
| 653 | ivpu_warn_ratelimited(vdev, "Failed to get job queue, ctx %d\n", file_priv->ctx.id); | ||
| 654 | ret = -EINVAL; | 690 | ret = -EINVAL; |
| 655 | goto err_unlock; | 691 | goto err_unlock; |
| 656 | } | 692 | } |
| @@ -661,6 +697,13 @@ static int ivpu_job_submit(struct ivpu_job *job, u8 priority, u32 cmdq_id) · 4 unmodified lines | |||
| 661 | goto err_unlock; | 697 | goto err_unlock; |
| 662 | } | 698 | } |
| 663 | 699 | ||
| 700 | ret = ivpu_preemption_job_init(vdev, file_priv, cmdq, job); | ||
| 701 | if (ret) { | ||
| 702 | ivpu_err(vdev, "Failed to initialize preemption buffers for job %d: %d\n", | ||
| 703 | job->job_id, ret); | ||
| 704 | goto err_unlock; | ||
| 705 | } | ||
| 706 | |||
| 664 | job->cmdq_id = cmdq->id; | 707 | job->cmdq_id = cmdq->id; |
| 665 | 708 | ||
| 666 | is_first_job = xa_empty(&vdev->submitted_jobs_xa); | 709 | is_first_job = xa_empty(&vdev->submitted_jobs_xa); |
| @@ -714,7 +757,7 @@ err_unlock: · 47 unmodified lines | |||
| 714 | 757 | ||
| 715 | static int | 758 | static int |
| 716 | ivpu_job_prepare_bos_for_submit(struct drm_file *file, struct ivpu_job *job, u32 *buf_handles, | 759 | ivpu_job_prepare_bos_for_submit(struct drm_file *file, struct ivpu_job *job, u32 *buf_handles, |
| 717 | u32 buf_count, u32 commands_offset) | 760 | u32 buf_count, u32 commands_offset, u32 preempt_buffer_index) |
| 718 | { | 761 | { |
| 719 | struct ivpu_file_priv *file_priv = job->file_priv; | 762 | struct ivpu_file_priv *file_priv = job->file_priv; |
| 720 | struct ivpu_device *vdev = file_priv->vdev; | 763 | struct ivpu_device *vdev = file_priv->vdev; |
| @@ -727,40 +770,58 @@ ivpu_job_prepare_bos_for_submit(struct drm_file *file, struct ivpu_job *job, u32 · 6 unmodified lines | |||
| 727 | for (i = 0; i < buf_count; i++) { | 770 | for (i = 0; i < buf_count; i++) { |
| 728 | struct drm_gem_object *obj = drm_gem_object_lookup(file, buf_handles[i]); | 771 | struct drm_gem_object *obj = drm_gem_object_lookup(file, buf_handles[i]); |
| 729 | 772 | ||
| 730 | if (!obj) | 773 | if (!obj) { |
| 774 | ivpu_dbg(vdev, IOCTL, "Failed to lookup GEM object with handle %u\n", | ||
| 775 | buf_handles[i]); | ||
| 731 | return -ENOENT; | 776 | return -ENOENT; |
| 777 | } | ||
| 732 | 778 | ||
| 733 | job->bos[i] = to_ivpu_bo(obj); | 779 | job->bos[i] = to_ivpu_bo(obj); |
| 734 | 780 | ||
| 735 | ret = ivpu_bo_pin(job->bos[i]); | 781 | ret = ivpu_bo_bind(job->bos[i]); |
| 736 | if (ret) | 782 | if (ret) |
| 737 | return ret; | 783 | return ret; |
| 738 | } | 784 | } |
| 739 | 785 | ||
| 740 | bo = job->bos[CMD_BUF_IDX]; | 786 | bo = job->bos[CMD_BUF_IDX]; |
| 741 | if (!dma_resv_test_signaled(bo->base.base.resv, DMA_RESV_USAGE_READ)) { | 787 | if (!dma_resv_test_signaled(bo->base.base.resv, DMA_RESV_USAGE_READ)) { |
| 742 | ivpu_warn(vdev, "Buffer is already in use\n"); | 788 | ivpu_dbg(vdev, IOCTL, "Buffer is already in use by another job\n"); |
| 743 | return -EBUSY; | 789 | return -EBUSY; |
| 744 | } | 790 | } |
| 745 | 791 | ||
| 746 | if (commands_offset >= ivpu_bo_size(bo)) { | 792 | if (commands_offset >= ivpu_bo_size(bo)) { |
| 747 | ivpu_warn(vdev, "Invalid command buffer offset %u\n", commands_offset); | 793 | ivpu_dbg(vdev, IOCTL, "Invalid commands offset %u for buffer size %zu\n", |
| 794 | commands_offset, ivpu_bo_size(bo)); | ||
| 748 | return -EINVAL; | 795 | return -EINVAL; |
| 749 | } | 796 | } |
| 750 | 797 | ||
| 751 | job->cmd_buf_vpu_addr = bo->vpu_addr + commands_offset; | 798 | job->cmd_buf_vpu_addr = bo->vpu_addr + commands_offset; |
| 752 | 799 | ||
| 800 | if (preempt_buffer_index) { | ||
| 801 | struct ivpu_bo *preempt_bo = job->bos[preempt_buffer_index]; | ||
| 802 | |||
| 803 | if (ivpu_bo_size(preempt_bo) < ivpu_fw_preempt_buf_size(vdev)) { | ||
| 804 | ivpu_dbg(vdev, IOCTL, "Preemption buffer is too small\n"); | ||
| 805 | return -EINVAL; | ||
| 806 | } | ||
| 807 | if (ivpu_bo_is_mappable(preempt_bo)) { | ||
| 808 | ivpu_dbg(vdev, IOCTL, "Preemption buffer cannot be mappable\n"); | ||
| 809 | return -EINVAL; | ||
| 810 | } | ||
| 811 | job->primary_preempt_buf = preempt_bo; | ||
| 812 | } | ||
| 813 | |||
| 753 | ret = drm_gem_lock_reservations((struct drm_gem_object **)job->bos, buf_count, | 814 | ret = drm_gem_lock_reservations((struct drm_gem_object **)job->bos, buf_count, |
| 754 | &acquire_ctx); | 815 | &acquire_ctx); |
| 755 | if (ret) { | 816 | if (ret) { |
| 756 | ivpu_warn(vdev, "Failed to lock reservations: %d\n", ret); | 817 | ivpu_warn_ratelimited(vdev, "Failed to lock reservations: %d\n", ret); |
| 757 | return ret; | 818 | return ret; |
| 758 | } | 819 | } |
| 759 | 820 | ||
| 760 | for (i = 0; i < buf_count; i++) { | 821 | for (i = 0; i < buf_count; i++) { |
| 761 | ret = dma_resv_reserve_fences(job->bos[i]->base.base.resv, 1); | 822 | ret = dma_resv_reserve_fences(job->bos[i]->base.base.resv, 1); |
| 762 | if (ret) { | 823 | if (ret) { |
| 763 | ivpu_warn(vdev, "Failed to reserve fences: %d\n", ret); | 824 | ivpu_warn_ratelimited(vdev, "Failed to reserve fences: %d\n", ret); |
| 764 | goto unlock_reservations; | 825 | goto unlock_reservations; |
| 765 | } | 826 | } |
| 766 | } | 827 | } |
| @@ -780,7 +841,7 @@ unlock_reservations: · 13 unmodified lines | |||
| 780 | 841 | ||
| 781 | static int ivpu_submit(struct drm_file *file, struct ivpu_file_priv *file_priv, u32 cmdq_id, | 842 | static int ivpu_submit(struct drm_file *file, struct ivpu_file_priv *file_priv, u32 cmdq_id, |
| 782 | u32 buffer_count, u32 engine, void __user *buffers_ptr, u32 cmds_offset, | 843 | u32 buffer_count, u32 engine, void __user *buffers_ptr, u32 cmds_offset, |
| 783 | u8 priority) | 844 | u32 preempt_buffer_index, u8 priority) |
| 784 | { | 845 | { |
| 785 | struct ivpu_device *vdev = file_priv->vdev; | 846 | struct ivpu_device *vdev = file_priv->vdev; |
| 786 | struct ivpu_job *job; | 847 | struct ivpu_job *job; |
| @@ -807,16 +868,14 @@ static int ivpu_submit(struct drm_file *file, struct ivpu_file_priv *file_priv, · 20 unmodified lines | |||
| 807 | 868 | ||
| 808 | job = ivpu_job_create(file_priv, engine, buffer_count); | 869 | job = ivpu_job_create(file_priv, engine, buffer_count); |
| 809 | if (!job) { | 870 | if (!job) { |
| 810 | ivpu_err(vdev, "Failed to create job\n"); | ||
| 811 | ret = -ENOMEM; | 871 | ret = -ENOMEM; |
| 812 | goto err_exit_dev; | 872 | goto err_exit_dev; |
| 813 | } | 873 | } |
| 814 | 874 | ||
| 815 | ret = ivpu_job_prepare_bos_for_submit(file, job, buf_handles, buffer_count, cmds_offset); | 875 | ret = ivpu_job_prepare_bos_for_submit(file, job, buf_handles, buffer_count, cmds_offset, |
| 816 | if (ret) { | 876 | preempt_buffer_index); |
| 817 | ivpu_err(vdev, "Failed to prepare job: %d\n", ret); | 877 | if (ret) |
| 818 | goto err_destroy_job; | 878 | goto err_destroy_job; |
| 819 | } | ||
| 820 | 879 | ||
| 821 | down_read(&vdev->pm->reset_lock); | 880 | down_read(&vdev->pm->reset_lock); |
| 822 | ret = ivpu_job_submit(job, priority, cmdq_id); | 881 | ret = ivpu_job_submit(job, priority, cmdq_id); |
| @@ -842,58 +901,91 @@ err_free_handles: · 19 unmodified lines | |||
| 842 | int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) | 901 | int ivpu_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) |
| 843 | { | 902 | { |
| 844 | struct ivpu_file_priv *file_priv = file->driver_priv; | 903 | struct ivpu_file_priv *file_priv = file->driver_priv; |
| 904 | struct ivpu_device *vdev = file_priv->vdev; | ||
| 845 | struct drm_ivpu_submit *args = data; | 905 | struct drm_ivpu_submit *args = data; |
| 846 | u8 priority; | 906 | u8 priority; |
| 847 | 907 | ||
| 848 | if (args->engine != DRM_IVPU_ENGINE_COMPUTE) | 908 | if (args->engine != DRM_IVPU_ENGINE_COMPUTE) { |
| 909 | ivpu_dbg(vdev, IOCTL, "Invalid engine %d\n", args->engine); | ||
| 849 | return -EINVAL; | 910 | return -EINVAL; |
| 911 | } | ||
| 850 | 912 | ||
| 851 | if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) | 913 | if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) { |
| 914 | ivpu_dbg(vdev, IOCTL, "Invalid priority %d\n", args->priority); | ||
| 852 | return -EINVAL; | 915 | return -EINVAL; |
| 916 | } | ||
| 853 | 917 | ||
| 854 | if (args->buffer_count == 0 || args->buffer_count > JOB_MAX_BUFFER_COUNT) | 918 | if (args->buffer_count == 0 || args->buffer_count > JOB_MAX_BUFFER_COUNT) { |
| 919 | ivpu_dbg(vdev, IOCTL, "Invalid buffer count %u\n", args->buffer_count); | ||
| 855 | return -EINVAL; | 920 | return -EINVAL; |
| 921 | } | ||
| 856 | 922 | ||
| 857 | if (!IS_ALIGNED(args->commands_offset, 8)) | 923 | if (!IS_ALIGNED(args->commands_offset, 8)) { |
| 924 | ivpu_dbg(vdev, IOCTL, "Invalid commands offset %u\n", args->commands_offset); | ||
| 858 | return -EINVAL; | 925 | return -EINVAL; |
| 926 | } | ||
| 859 | 927 | ||
| 860 | if (!file_priv->ctx.id) | 928 | if (!file_priv->ctx.id) { |
| 929 | ivpu_dbg(vdev, IOCTL, "Context not initialized\n"); | ||
| 861 | return -EINVAL; | 930 | return -EINVAL; |
| 931 | } | ||
| 862 | 932 | ||
| 863 | if (file_priv->has_mmu_faults) | 933 | if (file_priv->has_mmu_faults) { |
| 934 | ivpu_dbg(vdev, IOCTL, "Context %u has MMU faults\n", file_priv->ctx.id); | ||
| 864 | return -EBADFD; | 935 | return -EBADFD; |
| 936 | } | ||
| 865 | 937 | ||
| 866 | priority = ivpu_job_to_jsm_priority(args->priority); | 938 | priority = ivpu_job_to_jsm_priority(args->priority); |
| 867 | 939 | ||
| 868 | return ivpu_submit(file, file_priv, 0, args->buffer_count, args->engine, | 940 | return ivpu_submit(file, file_priv, 0, args->buffer_count, args->engine, |
| 869 | (void __user *)args->buffers_ptr, args->commands_offset, priority); | 941 | (void __user *)args->buffers_ptr, args->commands_offset, 0, priority); |
| 870 | } | 942 | } |
| 871 | 943 | ||
| 872 | int ivpu_cmdq_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) | 944 | int ivpu_cmdq_submit_ioctl(struct drm_device *dev, void *data, struct drm_file *file) |
| 873 | { | 945 | { |
| 874 | struct ivpu_file_priv *file_priv = file->driver_priv; | 946 | struct ivpu_file_priv *file_priv = file->driver_priv; |
| 947 | struct ivpu_device *vdev = file_priv->vdev; | ||
| 875 | struct drm_ivpu_cmdq_submit *args = data; | 948 | struct drm_ivpu_cmdq_submit *args = data; |
| 876 | 949 | ||
| 877 | if (!ivpu_is_capable(file_priv->vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) | 950 | if (!ivpu_is_capable(file_priv->vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) { |
| 951 | ivpu_dbg(vdev, IOCTL, "Command queue management not supported\n"); | ||
| 878 | return -ENODEV; | 952 | return -ENODEV; |
| 953 | } | ||
| 879 | 954 | ||
| 880 | if (args->cmdq_id < IVPU_CMDQ_MIN_ID || args->cmdq_id > IVPU_CMDQ_MAX_ID) | 955 | if (args->cmdq_id < IVPU_CMDQ_MIN_ID || args->cmdq_id > IVPU_CMDQ_MAX_ID) { |
| 956 | ivpu_dbg(vdev, IOCTL, "Invalid command queue ID %u\n", args->cmdq_id); | ||
| 881 | return -EINVAL; | 957 | return -EINVAL; |
| 958 | } | ||
| 882 | 959 | ||
| 883 | if (args->buffer_count == 0 || args->buffer_count > JOB_MAX_BUFFER_COUNT) | 960 | if (args->buffer_count == 0 || args->buffer_count > JOB_MAX_BUFFER_COUNT) { |
| 961 | ivpu_dbg(vdev, IOCTL, "Invalid buffer count %u\n", args->buffer_count); | ||
| 884 | return -EINVAL; | 962 | return -EINVAL; |
| 963 | } | ||
| 885 | 964 | ||
| 886 | if (!IS_ALIGNED(args->commands_offset, 8)) | 965 | if (args->preempt_buffer_index >= args->buffer_count) { |
| 966 | ivpu_dbg(vdev, IOCTL, "Invalid preemption buffer index %u\n", | ||
| 967 | args->preempt_buffer_index); | ||
| 887 | return -EINVAL; | 968 | return -EINVAL; |
| 969 | } | ||
| 888 | 970 | ||
| 889 | if (!file_priv->ctx.id) | 971 | if (!IS_ALIGNED(args->commands_offset, 8)) { |
| 972 | ivpu_dbg(vdev, IOCTL, "Invalid commands offset %u\n", args->commands_offset); | ||
| 890 | return -EINVAL; | 973 | return -EINVAL; |
| 974 | } | ||
| 891 | 975 | ||
| 892 | if (file_priv->has_mmu_faults) | 976 | if (!file_priv->ctx.id) { |
| 977 | ivpu_dbg(vdev, IOCTL, "Context not initialized\n"); | ||
| 978 | return -EINVAL; | ||
| 979 | } | ||
| 980 | |||
| 981 | if (file_priv->has_mmu_faults) { | ||
| 982 | ivpu_dbg(vdev, IOCTL, "Context %u has MMU faults\n", file_priv->ctx.id); | ||
| 893 | return -EBADFD; | 983 | return -EBADFD; |
| 984 | } | ||
| 894 | 985 | ||
| 895 | return ivpu_submit(file, file_priv, args->cmdq_id, args->buffer_count, VPU_ENGINE_COMPUTE, | 986 | return ivpu_submit(file, file_priv, args->cmdq_id, args->buffer_count, VPU_ENGINE_COMPUTE, |
| 896 | (void __user *)args->buffers_ptr, args->commands_offset, 0); | 987 | (void __user *)args->buffers_ptr, args->commands_offset, |
| 988 | args->preempt_buffer_index, 0); | ||
| 897 | } | 989 | } |
| 898 | 990 | ||
| 899 | int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file *file) | 991 | int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file *file) |
| @@ -904,11 +996,15 @@ int ivpu_cmdq_create_ioctl(struct drm_device *dev, void *data, struct drm_file * · 4 unmodified lines | |||
| 904 | struct ivpu_cmdq *cmdq; | 996 | struct ivpu_cmdq *cmdq; |
| 905 | int ret; | 997 | int ret; |
| 906 | 998 | ||
| 907 | if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) | 999 | if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) { |
| 1000 | ivpu_dbg(vdev, IOCTL, "Command queue management not supported\n"); | ||
| 908 | return -ENODEV; | 1001 | return -ENODEV; |
| 1002 | } | ||
| 909 | 1003 | ||
| 910 | if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) | 1004 | if (args->priority > DRM_IVPU_JOB_PRIORITY_REALTIME) { |
| 1005 | ivpu_dbg(vdev, IOCTL, "Invalid priority %d\n", args->priority); | ||
| 911 | return -EINVAL; | 1006 | return -EINVAL; |
| 1007 | } | ||
| 912 | 1008 | ||
| 913 | ret = ivpu_rpm_get(vdev); | 1009 | ret = ivpu_rpm_get(vdev); |
| 914 | if (ret < 0) | 1010 | if (ret < 0) |
| @@ -936,8 +1032,10 @@ int ivpu_cmdq_destroy_ioctl(struct drm_device *dev, void *data, struct drm_file · 21 unmodified lines | |||
| 936 | u32 cmdq_id = 0; | 1032 | u32 cmdq_id = 0; |
| 937 | int ret; | 1033 | int ret; |
| 938 | 1034 | ||
| 939 | if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) | 1035 | if (!ivpu_is_capable(vdev, DRM_IVPU_CAP_MANAGE_CMDQ)) { |
| 1036 | ivpu_dbg(vdev, IOCTL, "Command queue management not supported\n"); | ||
| 940 | return -ENODEV; | 1037 | return -ENODEV; |
| 1038 | } | ||
| 941 | 1039 | ||
| 942 | ret = ivpu_rpm_get(vdev); | 1040 | ret = ivpu_rpm_get(vdev); |
| 943 | if (ret < 0) | 1041 | if (ret < 0) |
| @@ -984,7 +1082,9 @@ ivpu_job_done_callback(struct ivpu_device *vdev, struct ivpu_ipc_hdr *ipc_hdr, · 40 unmodified lines | |||
| 984 | payload = (struct vpu_ipc_msg_payload_job_done *)&jsm_msg->payload; | 1082 | payload = (struct vpu_ipc_msg_payload_job_done *)&jsm_msg->payload; |
| 985 | 1083 | ||
| 986 | mutex_lock(&vdev->submitted_jobs_lock); | 1084 | mutex_lock(&vdev->submitted_jobs_lock); |
| 987 | ivpu_job_signal_and_destroy(vdev, payload->job_id, payload->job_status); | 1085 | if (!ivpu_job_handle_engine_error(vdev, payload->job_id, payload->job_status)) |
| 1086 | /* No engine error, complete the job normally */ | ||
| 1087 | ivpu_job_signal_and_destroy(vdev, payload->job_id, payload->job_status); | ||
| 988 | mutex_unlock(&vdev->submitted_jobs_lock); | 1088 | mutex_unlock(&vdev->submitted_jobs_lock); |
| 989 | } | 1089 | } |
| 990 | 1090 | ||
| @@ -1012,7 +1112,7 @@ void ivpu_context_abort_work_fn(struct work_struct *work) · 21 unmodified lines | |||
| 1012 | 1112 | ||
| 1013 | if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) | 1113 | if (vdev->fw->sched_mode == VPU_SCHEDULING_MODE_HW) |
| 1014 | if (ivpu_jsm_reset_engine(vdev, 0)) | 1114 | if (ivpu_jsm_reset_engine(vdev, 0)) |
| 1015 | return; | 1115 | goto runtime_put; |
| 1016 | 1116 | ||
| 1017 | mutex_lock(&vdev->context_list_lock); | 1117 | mutex_lock(&vdev->context_list_lock); |
| 1018 | xa_for_each(&vdev->context_xa, ctx_id, file_priv) { | 1118 | xa_for_each(&vdev->context_xa, ctx_id, file_priv) { |
| @@ -1036,7 +1136,7 @@ void ivpu_context_abort_work_fn(struct work_struct *work) · 17 unmodified lines | |||
| 1036 | goto runtime_put; | 1136 | goto runtime_put; |
| 1037 | 1137 | ||
| 1038 | if (ivpu_jsm_hws_resume_engine(vdev, 0)) | 1138 | if (ivpu_jsm_hws_resume_engine(vdev, 0)) |
| 1039 | return; | 1139 | goto runtime_put; |
| 1040 | /* | 1140 | /* |
| 1041 | * In hardware scheduling mode NPU already has stopped processing jobs | 1141 | * In hardware scheduling mode NPU already has stopped processing jobs |
| 1042 | * and won't send us any further notifications, thus we have to free job related resources | 1142 | * and won't send us any further notifications, thus we have to free job related resources |
| @@ -1049,6 +1149,5 @@ void ivpu_context_abort_work_fn(struct work_struct *work) · 6 unmodified lines | |||
| 1049 | mutex_unlock(&vdev->submitted_jobs_lock); | 1149 | mutex_unlock(&vdev->submitted_jobs_lock); |
| 1050 | 1150 | ||
| 1051 | runtime_put: | 1151 | runtime_put: |
| 1052 | pm_runtime_mark_last_busy(vdev->drm.dev); | ||
| 1053 | pm_runtime_put_autosuspend(vdev->drm.dev); | 1152 | pm_runtime_put_autosuspend(vdev->drm.dev); |
| 1054 | } | 1153 | } |
| @@ -1,6 +1,6 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only */ | 1 | /* SPDX-License-Identifier: GPL-2.0-only */ |
| 2 | /* | 2 | /* |
| 3 | * Copyright (C) 2020-2024 Intel Corporation | 3 | * Copyright (C) 2020-2025 Intel Corporation |
| 4 | */ | 4 | */ |
| 5 | 5 | ||
| 6 | #ifndef __IVPU_JOB_H__ | 6 | #ifndef __IVPU_JOB_H__ |
| @@ -15,12 +15,17 @@ struct ivpu_device; · 8 unmodified lines | |||
| 15 | struct ivpu_file_priv; | 15 | struct ivpu_file_priv; |
| 16 | 16 | ||
| 17 | /** | 17 | /** |
| 18 | * struct ivpu_cmdq - Object representing device queue used to send jobs. | 18 | * struct ivpu_cmdq - Represents a command queue for submitting jobs to the VPU. |
| 19 | * @jobq: Pointer to job queue memory shared with the device | 19 | * Tracks queue memory, preemption buffers, and metadata for job management. |
| 20 | * @mem: Memory allocated for the job queue, shared with device | 20 | * @jobq: Pointer to job queue memory shared with the device |
| 21 | * @entry_count Number of job entries in the queue | 21 | * @primary_preempt_buf: Primary preemption buffer for this queue (optional) |
| 22 | * @db_id: Doorbell assigned to this job queue | 22 | * @secondary_preempt_buf: Secondary preemption buffer for this queue (optional) |
| 23 | * @db_registered: True if doorbell is registered in device | 23 | * @mem: Memory allocated for the job queue, shared with device |
| 24 | * @entry_count: Number of job entries in the queue | ||
| 25 | * @id: Unique command queue ID | ||
| 26 | * @db_id: Doorbell ID assigned to this job queue | ||
| 27 | * @priority: Priority level of the command queue | ||
| 28 | * @is_legacy: True if this is a legacy command queue | ||
| 24 | */ | 29 | */ |
| 25 | struct ivpu_cmdq { | 30 | struct ivpu_cmdq { |
| 26 | struct vpu_job_queue *jobq; | 31 | struct vpu_job_queue *jobq; |
| @@ -35,16 +40,22 @@ struct ivpu_cmdq { · 8 unmodified lines | |||
| 35 | }; | 40 | }; |
| 36 | 41 | ||
| 37 | /** | 42 | /** |
| 38 | * struct ivpu_job - KMD object that represents batchbuffer / DMA buffer. | 43 | * struct ivpu_job - Representing a batch or DMA buffer submitted to the VPU. |
| 39 | * Each batch / DMA buffer is a job to be submitted and executed by the VPU FW. | 44 | * Each job is a unit of execution, tracked by job_id for status reporting from VPU FW. |
| 40 | * This is a unit of execution, and be tracked by the job_id for | 45 | * The structure holds all resources and metadata needed for job submission, execution, |
| 41 | * any status reporting from VPU FW through IPC JOB RET/DONE message. | 46 | * and completion handling. |
| 42 | * @file_priv: The client that submitted this job | 47 | * @vdev: Pointer to the VPU device |
| 43 | * @job_id: Job ID for KMD tracking and job status reporting from VPU FW | 48 | * @file_priv: The client context that submitted this job |
| 44 | * @status: Status of the Job from IPC JOB RET/DONE message | 49 | * @done_fence: Fence signaled when job completes |
| 45 | * @batch_buffer: CPU vaddr points to the batch buffer memory allocated for the job | 50 | * @cmd_buf_vpu_addr: VPU address of the command buffer for this job |
| 46 | * @submit_status_offset: Offset within batch buffer where job completion handler | 51 | * @cmdq_id: Command queue ID used for submission |
| 47 | will update the job status | 52 | * @job_id: Unique job ID for tracking and status reporting |
| 53 | * @engine_idx: Engine index for job execution | ||
| 54 | * @job_status: Status reported by firmware for this job | ||
| 55 | * @primary_preempt_buf: Primary preemption buffer for job | ||
| 56 | * @secondary_preempt_buf: Secondary preemption buffer for job (optional) | ||
| 57 | * @bo_count: Number of buffer objects associated with this job | ||
| 58 | * @bos: Array of buffer objects used by the job (batch buffer is at index 0) | ||
| 48 | */ | 59 | */ |
| 49 | struct ivpu_job { | 60 | struct ivpu_job { |
| 50 | struct ivpu_device *vdev; | 61 | struct ivpu_device *vdev; |
| @@ -54,6 +65,9 @@ struct ivpu_job { · 3 unmodified lines | |||
| 54 | u32 cmdq_id; | 65 | u32 cmdq_id; |
| 55 | u32 job_id; | 66 | u32 job_id; |
| 56 | u32 engine_idx; | 67 | u32 engine_idx; |
| 68 | u32 job_status; | ||
| 69 | struct ivpu_bo *primary_preempt_buf; | ||
| 70 | struct ivpu_bo *secondary_preempt_buf; | ||
| 57 | size_t bo_count; | 71 | size_t bo_count; |
| 58 | struct ivpu_bo *bos[] __counted_by(bo_count); | 72 | struct ivpu_bo *bos[] __counted_by(bo_count); |
| 59 | }; | 73 | }; |
| @@ -71,6 +85,7 @@ void ivpu_cmdq_abort_all_jobs(struct ivpu_device *vdev, u32 ctx_id, u32 cmdq_id) · 11 unmodified lines | |||
| 71 | 85 | ||
| 72 | void ivpu_job_done_consumer_init(struct ivpu_device *vdev); | 86 | void ivpu_job_done_consumer_init(struct ivpu_device *vdev); |
| 73 | void ivpu_job_done_consumer_fini(struct ivpu_device *vdev); | 87 | void ivpu_job_done_consumer_fini(struct ivpu_device *vdev); |
| 88 | bool ivpu_job_handle_engine_error(struct ivpu_device *vdev, u32 job_id, u32 job_status); | ||
| 74 | void ivpu_context_abort_work_fn(struct work_struct *work); | 89 | void ivpu_context_abort_work_fn(struct work_struct *work); |
| 75 | 90 | ||
| 76 | void ivpu_jobs_abort_all(struct ivpu_device *vdev); | 91 | void ivpu_jobs_abort_all(struct ivpu_device *vdev); |
| @@ -970,7 +970,7 @@ void ivpu_mmu_irq_evtq_handler(struct ivpu_device *vdev) · 969 unmodified lines | |||
| 970 | } | 970 | } |
| 971 | } | 971 | } |
| 972 | 972 | ||
| 973 | queue_work(system_wq, &vdev->context_abort_work); | 973 | queue_work(system_percpu_wq, &vdev->context_abort_work); |
| 974 | } | 974 | } |
| 975 | 975 | ||
| 976 | void ivpu_mmu_evtq_dump(struct ivpu_device *vdev) | 976 | void ivpu_mmu_evtq_dump(struct ivpu_device *vdev) |
| @@ -430,7 +430,7 @@ static void ivpu_mmu_context_unmap_pages(struct ivpu_mmu_context *ctx, u64 vpu_a · 429 unmodified lines | |||
| 430 | 430 | ||
| 431 | int | 431 | int |
| 432 | ivpu_mmu_context_map_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, | 432 | ivpu_mmu_context_map_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, |
| 433 | u64 vpu_addr, struct sg_table *sgt, bool llc_coherent) | 433 | u64 vpu_addr, struct sg_table *sgt, bool llc_coherent, bool read_only) |
| 434 | { | 434 | { |
| 435 | size_t start_vpu_addr = vpu_addr; | 435 | size_t start_vpu_addr = vpu_addr; |
| 436 | struct scatterlist *sg; | 436 | struct scatterlist *sg; |
| @@ -450,6 +450,8 @@ ivpu_mmu_context_map_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, · 13 unmodified lines | |||
| 450 | prot = IVPU_MMU_ENTRY_MAPPED; | 450 | prot = IVPU_MMU_ENTRY_MAPPED; |
| 451 | if (llc_coherent) | 451 | if (llc_coherent) |
| 452 | prot |= IVPU_MMU_ENTRY_FLAG_LLC_COHERENT; | 452 | prot |= IVPU_MMU_ENTRY_FLAG_LLC_COHERENT; |
| 453 | if (read_only) | ||
| 454 | prot |= IVPU_MMU_ENTRY_FLAG_RO; | ||
| 453 | 455 | ||
| 454 | mutex_lock(&ctx->lock); | 456 | mutex_lock(&ctx->lock); |
| 455 | 457 | ||
| @@ -527,7 +529,8 @@ ivpu_mmu_context_unmap_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ct · 71 unmodified lines | |||
| 527 | 529 | ||
| 528 | ret = ivpu_mmu_invalidate_tlb(vdev, ctx->id); | 530 | ret = ivpu_mmu_invalidate_tlb(vdev, ctx->id); |
| 529 | if (ret) | 531 | if (ret) |
| 530 | ivpu_warn(vdev, "Failed to invalidate TLB for ctx %u: %d\n", ctx->id, ret); | 532 | ivpu_warn_ratelimited(vdev, "Failed to invalidate TLB for ctx %u: %d\n", |
| 533 | ctx->id, ret); | ||
| 531 | } | 534 | } |
| 532 | 535 | ||
| 533 | int | 536 | int |
| @@ -568,7 +571,7 @@ void ivpu_mmu_context_init(struct ivpu_device *vdev, struct ivpu_mmu_context *ct · 34 unmodified lines | |||
| 568 | mutex_init(&ctx->lock); | 571 | mutex_init(&ctx->lock); |
| 569 | 572 | ||
| 570 | if (!context_id) { | 573 | if (!context_id) { |
| 571 | start = vdev->hw->ranges.global.start; | 574 | start = vdev->hw->ranges.runtime.start; |
| 572 | end = vdev->hw->ranges.shave.end; | 575 | end = vdev->hw->ranges.shave.end; |
| 573 | } else { | 576 | } else { |
| 574 | start = min_t(u64, vdev->hw->ranges.user.start, vdev->hw->ranges.shave.start); | 577 | start = min_t(u64, vdev->hw->ranges.user.start, vdev->hw->ranges.shave.start); |
| @@ -42,7 +42,7 @@ int ivpu_mmu_context_insert_node(struct ivpu_mmu_context *ctx, const struct ivpu · 41 unmodified lines | |||
| 42 | void ivpu_mmu_context_remove_node(struct ivpu_mmu_context *ctx, struct drm_mm_node *node); | 42 | void ivpu_mmu_context_remove_node(struct ivpu_mmu_context *ctx, struct drm_mm_node *node); |
| 43 | 43 | ||
| 44 | int ivpu_mmu_context_map_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, | 44 | int ivpu_mmu_context_map_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, |
| 45 | u64 vpu_addr, struct sg_table *sgt, bool llc_coherent); | 45 | u64 vpu_addr, struct sg_table *sgt, bool llc_coherent, bool read_only); |
| 46 | void ivpu_mmu_context_unmap_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, | 46 | void ivpu_mmu_context_unmap_sgt(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, |
| 47 | u64 vpu_addr, struct sg_table *sgt); | 47 | u64 vpu_addr, struct sg_table *sgt); |
| 48 | int ivpu_mmu_context_set_pages_ro(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, | 48 | int ivpu_mmu_context_set_pages_ro(struct ivpu_device *vdev, struct ivpu_mmu_context *ctx, |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | 8 | ||
| 9 | #include "ivpu_drv.h" | 9 | #include "ivpu_drv.h" |
| 10 | #include "ivpu_gem.h" | 10 | #include "ivpu_gem.h" |
| 11 | #include "ivpu_hw.h" | ||
| 11 | #include "ivpu_jsm_msg.h" | 12 | #include "ivpu_jsm_msg.h" |
| 12 | #include "ivpu_ms.h" | 13 | #include "ivpu_ms.h" |
| 13 | #include "ivpu_pm.h" | 14 | #include "ivpu_pm.h" |
| @@ -37,8 +38,8 @@ int ivpu_ms_start_ioctl(struct drm_device *dev, void *data, struct drm_file *fil · 23 unmodified lines | |||
| 37 | struct drm_ivpu_metric_streamer_start *args = data; | 38 | struct drm_ivpu_metric_streamer_start *args = data; |
| 38 | struct ivpu_device *vdev = file_priv->vdev; | 39 | struct ivpu_device *vdev = file_priv->vdev; |
| 39 | struct ivpu_ms_instance *ms; | 40 | struct ivpu_ms_instance *ms; |
| 40 | u64 single_buff_size; | ||
| 41 | u32 sample_size; | 41 | u32 sample_size; |
| 42 | u64 buf_size; | ||
| 42 | int ret; | 43 | int ret; |
| 43 | 44 | ||
| 44 | if (!args->metric_group_mask || !args->read_period_samples || | 45 | if (!args->metric_group_mask || !args->read_period_samples || |
| @@ -52,7 +53,8 @@ int ivpu_ms_start_ioctl(struct drm_device *dev, void *data, struct drm_file *fil · 7 unmodified lines | |||
| 52 | mutex_lock(&file_priv->ms_lock); | 53 | mutex_lock(&file_priv->ms_lock); |
| 53 | 54 | ||
| 54 | if (get_instance_by_mask(file_priv, args->metric_group_mask)) { | 55 | if (get_instance_by_mask(file_priv, args->metric_group_mask)) { |
| 55 | ivpu_err(vdev, "Instance already exists (mask %#llx)\n", args->metric_group_mask); | 56 | ivpu_dbg(vdev, IOCTL, "Instance already exists (mask %#llx)\n", |
| 57 | args->metric_group_mask); | ||
| 56 | ret = -EALREADY; | 58 | ret = -EALREADY; |
| 57 | goto unlock; | 59 | goto unlock; |
| 58 | } | 60 | } |
| @@ -69,12 +71,18 @@ int ivpu_ms_start_ioctl(struct drm_device *dev, void *data, struct drm_file *fil · 10 unmodified lines | |||
| 69 | if (ret) | 71 | if (ret) |
| 70 | goto err_free_ms; | 72 | goto err_free_ms; |
| 71 | 73 | ||
| 72 | single_buff_size = sample_size * | 74 | buf_size = PAGE_ALIGN((u64)args->read_period_samples * sample_size * |
| 73 | ((u64)args->read_period_samples * MS_READ_PERIOD_MULTIPLIER); | 75 | MS_READ_PERIOD_MULTIPLIER * MS_NUM_BUFFERS); |
| 74 | ms->bo = ivpu_bo_create_global(vdev, PAGE_ALIGN(single_buff_size * MS_NUM_BUFFERS), | 76 | if (buf_size > ivpu_hw_range_size(&vdev->hw->ranges.global)) { |
| 75 | DRM_IVPU_BO_CACHED | DRM_IVPU_BO_MAPPABLE); | 77 | ivpu_dbg(vdev, IOCTL, "Requested MS buffer size %llu exceeds range size %llu\n", |
| 78 | buf_size, ivpu_hw_range_size(&vdev->hw->ranges.global)); | ||
| 79 | ret = -EINVAL; | ||
| 80 | goto err_free_ms; | ||
| 81 | } | ||
| 82 | |||
| 83 | ms->bo = ivpu_bo_create_global(vdev, buf_size, DRM_IVPU_BO_CACHED | DRM_IVPU_BO_MAPPABLE); | ||
| 76 | if (!ms->bo) { | 84 | if (!ms->bo) { |
| 77 | ivpu_err(vdev, "Failed to allocate MS buffer (size %llu)\n", single_buff_size); | 85 | ivpu_dbg(vdev, IOCTL, "Failed to allocate MS buffer (size %llu)\n", buf_size); |
| 78 | ret = -ENOMEM; | 86 | ret = -ENOMEM; |
| 79 | goto err_free_ms; | 87 | goto err_free_ms; |
| 80 | } | 88 | } |
| @@ -175,7 +183,8 @@ int ivpu_ms_get_data_ioctl(struct drm_device *dev, void *data, struct drm_file * · 94 unmodified lines | |||
| 175 | 183 | ||
| 176 | ms = get_instance_by_mask(file_priv, args->metric_group_mask); | 184 | ms = get_instance_by_mask(file_priv, args->metric_group_mask); |
| 177 | if (!ms) { | 185 | if (!ms) { |
| 178 | ivpu_err(vdev, "Instance doesn't exist for mask: %#llx\n", args->metric_group_mask); | 186 | ivpu_dbg(vdev, IOCTL, "Instance doesn't exist for mask: %#llx\n", |
| 187 | args->metric_group_mask); | ||
| 179 | ret = -EINVAL; | 188 | ret = -EINVAL; |
| 180 | goto unlock; | 189 | goto unlock; |
| 181 | } | 190 | } |
| @@ -54,7 +54,7 @@ static void ivpu_pm_prepare_cold_boot(struct ivpu_device *vdev) · 53 unmodified lines | |||
| 54 | static void ivpu_pm_prepare_warm_boot(struct ivpu_device *vdev) | 54 | static void ivpu_pm_prepare_warm_boot(struct ivpu_device *vdev) |
| 55 | { | 55 | { |
| 56 | struct ivpu_fw_info *fw = vdev->fw; | 56 | struct ivpu_fw_info *fw = vdev->fw; |
| 57 | struct vpu_boot_params *bp = ivpu_bo_vaddr(fw->mem); | 57 | struct vpu_boot_params *bp = ivpu_bo_vaddr(fw->mem_bp); |
| 58 | 58 | ||
| 59 | if (!bp->save_restore_ret_address) { | 59 | if (!bp->save_restore_ret_address) { |
| 60 | ivpu_pm_prepare_cold_boot(vdev); | 60 | ivpu_pm_prepare_cold_boot(vdev); |
| @@ -186,7 +186,7 @@ void ivpu_pm_trigger_recovery(struct ivpu_device *vdev, const char *reason) · 125 unmodified lines | |||
| 186 | if (atomic_cmpxchg(&vdev->pm->reset_pending, 0, 1) == 0) { | 186 | if (atomic_cmpxchg(&vdev->pm->reset_pending, 0, 1) == 0) { |
| 187 | ivpu_hw_diagnose_failure(vdev); | 187 | ivpu_hw_diagnose_failure(vdev); |
| 188 | ivpu_hw_irq_disable(vdev); /* Disable IRQ early to protect from IRQ storm */ | 188 | ivpu_hw_irq_disable(vdev); /* Disable IRQ early to protect from IRQ storm */ |
| 189 | queue_work(system_unbound_wq, &vdev->pm->recovery_work); | 189 | queue_work(system_dfl_wq, &vdev->pm->recovery_work); |
| 190 | } | 190 | } |
| 191 | } | 191 | } |
| 192 | 192 | ||
| @@ -226,7 +226,8 @@ void ivpu_start_job_timeout_detection(struct ivpu_device *vdev) · 33 unmodified lines | |||
| 226 | unsigned long timeout_ms = ivpu_tdr_timeout_ms ? ivpu_tdr_timeout_ms : vdev->timeout.tdr; | 226 | unsigned long timeout_ms = ivpu_tdr_timeout_ms ? ivpu_tdr_timeout_ms : vdev->timeout.tdr; |
| 227 | 227 | ||
| 228 | /* No-op if already queued */ | 228 | /* No-op if already queued */ |
| 229 | queue_delayed_work(system_wq, &vdev->pm->job_timeout_work, msecs_to_jiffies(timeout_ms)); | 229 | queue_delayed_work(system_percpu_wq, &vdev->pm->job_timeout_work, |
| 230 | msecs_to_jiffies(timeout_ms)); | ||
| 230 | } | 231 | } |
| 231 | 232 | ||
| 232 | void ivpu_stop_job_timeout_detection(struct ivpu_device *vdev) | 233 | void ivpu_stop_job_timeout_detection(struct ivpu_device *vdev) |
| @@ -359,7 +360,6 @@ int ivpu_rpm_get(struct ivpu_device *vdev) · 126 unmodified lines | |||
| 359 | 360 | ||
| 360 | void ivpu_rpm_put(struct ivpu_device *vdev) | 361 | void ivpu_rpm_put(struct ivpu_device *vdev) |
| 361 | { | 362 | { |
| 362 | pm_runtime_mark_last_busy(vdev->drm.dev); | ||
| 363 | pm_runtime_put_autosuspend(vdev->drm.dev); | 363 | pm_runtime_put_autosuspend(vdev->drm.dev); |
| 364 | } | 364 | } |
| 365 | 365 | ||
| @@ -428,7 +428,6 @@ void ivpu_pm_enable(struct ivpu_device *vdev) · 62 unmodified lines | |||
| 428 | struct device *dev = vdev->drm.dev; | 428 | struct device *dev = vdev->drm.dev; |
| 429 | 429 | ||
| 430 | pm_runtime_allow(dev); | 430 | pm_runtime_allow(dev); |
| 431 | pm_runtime_mark_last_busy(dev); | ||
| 432 | pm_runtime_put_autosuspend(dev); | 431 | pm_runtime_put_autosuspend(dev); |
| 433 | } | 432 | } |
| 434 | 433 | ||
| @@ -502,6 +501,11 @@ void ivpu_pm_irq_dct_work_fn(struct work_struct *work) · 67 unmodified lines | |||
| 502 | else | 501 | else |
| 503 | ret = ivpu_pm_dct_disable(vdev); | 502 | ret = ivpu_pm_dct_disable(vdev); |
| 504 | 503 | ||
| 505 | if (!ret) | 504 | if (!ret) { |
| 506 | ivpu_hw_btrs_dct_set_status(vdev, enable, vdev->pm->dct_active_percent); | 505 | /* Convert percent to U1.7 format */ |
| 506 | u8 val = DIV_ROUND_CLOSEST(vdev->pm->dct_active_percent * 128, 100); | ||
| 507 | |||
| 508 | ivpu_hw_btrs_dct_set_status(vdev, enable, val); | ||
| 509 | } | ||
| 510 | |||
| 507 | } | 511 | } |
| @@ -63,7 +63,8 @@ npu_memory_utilization_show(struct device *dev, struct device_attribute *attr, c · 62 unmodified lines | |||
| 63 | 63 | ||
| 64 | mutex_lock(&vdev->bo_list_lock); | 64 | mutex_lock(&vdev->bo_list_lock); |
| 65 | list_for_each_entry(bo, &vdev->bo_list, bo_list_node) | 65 | list_for_each_entry(bo, &vdev->bo_list, bo_list_node) |
| 66 | total_npu_memory += bo->base.base.size; | 66 | if (ivpu_bo_is_resident(bo)) |
| 67 | total_npu_memory += ivpu_bo_size(bo); | ||
| 67 | mutex_unlock(&vdev->bo_list_lock); | 68 | mutex_unlock(&vdev->bo_list_lock); |
| 68 | 69 | ||
| 69 | return sysfs_emit(buf, "%lld\n", total_npu_memory); | 70 | return sysfs_emit(buf, "%lld\n", total_npu_memory); |
| @@ -1,15 +1,16 @@ | |||
| 1 | /* SPDX-License-Identifier: MIT */ | 1 | /* SPDX-License-Identifier: MIT */ |
| 2 | /* | 2 | /* |
| 3 | * Copyright (c) 2020-2024, Intel Corporation. | 3 | * Copyright (c) 2020-2025, Intel Corporation. |
| 4 | */ | ||
| 5 | |||
| 6 | /** | ||
| 7 | * @addtogroup Jsm | ||
| 8 | * @{ | ||
| 4 | */ | 9 | */ |
| 5 | 10 | ||
| 6 | /** | 11 | /** |
| 7 | * @file | 12 | * @file |
| 8 | * @brief JSM shared definitions | 13 | * @brief JSM shared definitions |
| 9 | * | ||
| 10 | * @ingroup Jsm | ||
| 11 | * @brief JSM shared definitions | ||
| 12 | * @{ | ||
| 13 | */ | 14 | */ |
| 14 | #ifndef VPU_JSM_API_H | 15 | #ifndef VPU_JSM_API_H |
| 15 | #define VPU_JSM_API_H | 16 | #define VPU_JSM_API_H |
| @@ -22,7 +23,7 @@ · 6 unmodified lines | |||
| 22 | /* | 23 | /* |
| 23 | * Minor version changes when API backward compatibility is preserved. | 24 | * Minor version changes when API backward compatibility is preserved. |
| 24 | */ | 25 | */ |
| 25 | #define VPU_JSM_API_VER_MINOR 29 | 26 | #define VPU_JSM_API_VER_MINOR 33 |
| 26 | 27 | ||
| 27 | /* | 28 | /* |
| 28 | * API header changed (field names, documentation, formatting) but API itself has not been changed | 29 | * API header changed (field names, documentation, formatting) but API itself has not been changed |
| @@ -71,9 +72,15 @@ · 42 unmodified lines | |||
| 71 | #define VPU_JSM_STATUS_MVNCI_OUT_OF_RESOURCES 0xAU | 72 | #define VPU_JSM_STATUS_MVNCI_OUT_OF_RESOURCES 0xAU |
| 72 | #define VPU_JSM_STATUS_MVNCI_NOT_IMPLEMENTED 0xBU | 73 | #define VPU_JSM_STATUS_MVNCI_NOT_IMPLEMENTED 0xBU |
| 73 | #define VPU_JSM_STATUS_MVNCI_INTERNAL_ERROR 0xCU | 74 | #define VPU_JSM_STATUS_MVNCI_INTERNAL_ERROR 0xCU |
| 74 | /* Job status returned when the job was preempted mid-inference */ | 75 | /* @deprecated (use VPU_JSM_STATUS_PREEMPTED_MID_COMMAND instead) */ |
| 75 | #define VPU_JSM_STATUS_PREEMPTED_MID_INFERENCE 0xDU | 76 | #define VPU_JSM_STATUS_PREEMPTED_MID_INFERENCE 0xDU |
| 77 | /* Job status returned when the job was preempted mid-command */ | ||
| 78 | #define VPU_JSM_STATUS_PREEMPTED_MID_COMMAND 0xDU | ||
| 79 | /* Range of status codes that require engine reset */ | ||
| 80 | #define VPU_JSM_STATUS_ENGINE_RESET_REQUIRED_MIN 0xEU | ||
| 76 | #define VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HW 0xEU | 81 | #define VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HW 0xEU |
| 82 | #define VPU_JSM_STATUS_MVNCI_PREEMPTION_TIMED_OUT 0xFU | ||
| 83 | #define VPU_JSM_STATUS_ENGINE_RESET_REQUIRED_MAX 0x1FU | ||
| 77 | 84 | ||
| 78 | /* | 85 | /* |
| 79 | * Host <-> VPU IPC channels. | 86 | * Host <-> VPU IPC channels. |
| @@ -134,11 +141,21 @@ enum { · 54 unmodified lines | |||
| 134 | * 2. Native fence queues are only supported on VPU 40xx onwards. | 141 | * 2. Native fence queues are only supported on VPU 40xx onwards. |
| 135 | */ | 142 | */ |
| 136 | VPU_JOB_QUEUE_FLAGS_USE_NATIVE_FENCE_MASK = (1 << 1U), | 143 | VPU_JOB_QUEUE_FLAGS_USE_NATIVE_FENCE_MASK = (1 << 1U), |
| 137 | |||
| 138 | /* | 144 | /* |
| 139 | * Enable turbo mode for testing NPU performance; not recommended for regular usage. | 145 | * Enable turbo mode for testing NPU performance; not recommended for regular usage. |
| 140 | */ | 146 | */ |
| 141 | VPU_JOB_QUEUE_FLAGS_TURBO_MODE = (1 << 2U) | 147 | VPU_JOB_QUEUE_FLAGS_TURBO_MODE = (1 << 2U), |
| 148 | /* | ||
| 149 | * Queue error detection mode flag | ||
| 150 | * For 'interactive' queues (this bit not set), the FW will identify queues that have not | ||
| 151 | * completed a job inside the TDR timeout as in error as part of engine reset sequence. | ||
| 152 | * For 'non-interactive' queues (this bit set), the FW will identify queues that have not | ||
| 153 | * progressed the heartbeat inside the non-interactive no-progress timeout as in error as | ||
| 154 | * part of engine reset sequence. Additionally, there is an upper limit applied to these | ||
| 155 | * queues: even if they progress the heartbeat, if they run longer than non-interactive | ||
| 156 | * timeout, then the FW will also identify them as in error. | ||
| 157 | */ | ||
| 158 | VPU_JOB_QUEUE_FLAGS_NON_INTERACTIVE = (1 << 3U) | ||
| 142 | }; | 159 | }; |
| 143 | 160 | ||
| 144 | /* | 161 | /* |
| @@ -209,7 +226,7 @@ enum { · 64 unmodified lines | |||
| 209 | */ | 226 | */ |
| 210 | #define VPU_INLINE_CMD_TYPE_FENCE_SIGNAL 0x2 | 227 | #define VPU_INLINE_CMD_TYPE_FENCE_SIGNAL 0x2 |
| 211 | 228 | ||
| 212 | /* | 229 | /** |
| 213 | * Job scheduling priority bands for both hardware scheduling and OS scheduling. | 230 | * Job scheduling priority bands for both hardware scheduling and OS scheduling. |
| 214 | */ | 231 | */ |
| 215 | enum vpu_job_scheduling_priority_band { | 232 | enum vpu_job_scheduling_priority_band { |
| @@ -220,16 +237,16 @@ enum vpu_job_scheduling_priority_band { · 4 unmodified lines | |||
| 220 | VPU_JOB_SCHEDULING_PRIORITY_BAND_COUNT = 4, | 237 | VPU_JOB_SCHEDULING_PRIORITY_BAND_COUNT = 4, |
| 221 | }; | 238 | }; |
| 222 | 239 | ||
| 223 | /* | 240 | /** |
| 224 | * Job format. | 241 | * Job format. |
| 225 | * Jobs defines the actual workloads to be executed by a given engine. | 242 | * Jobs defines the actual workloads to be executed by a given engine. |
| 226 | */ | 243 | */ |
| 227 | struct vpu_job_queue_entry { | 244 | struct vpu_job_queue_entry { |
| 228 | /**< Address of VPU commands batch buffer */ | 245 | /** Address of VPU commands batch buffer */ |
| 229 | u64 batch_buf_addr; | 246 | u64 batch_buf_addr; |
| 230 | /**< Job ID */ | 247 | /** Job ID */ |
| 231 | u32 job_id; | 248 | u32 job_id; |
| 232 | /**< Flags bit field, see VPU_JOB_FLAGS_* above */ | 249 | /** Flags bit field, see VPU_JOB_FLAGS_* above */ |
| 233 | u32 flags; | 250 | u32 flags; |
| 234 | /** | 251 | /** |
| 235 | * Doorbell ring timestamp taken by KMD from SoC's global system clock, in | 252 | * Doorbell ring timestamp taken by KMD from SoC's global system clock, in |
| @@ -237,20 +254,20 @@ struct vpu_job_queue_entry { · 1 unmodified line | |||
| 237 | * to match other profiling timestamps. | 254 | * to match other profiling timestamps. |
| 238 | */ | 255 | */ |
| 239 | u64 doorbell_timestamp; | 256 | u64 doorbell_timestamp; |
| 240 | /**< Extra id for job tracking, used only in the firmware perf traces */ | 257 | /** Extra id for job tracking, used only in the firmware perf traces */ |
| 241 | u64 host_tracking_id; | 258 | u64 host_tracking_id; |
| 242 | /**< Address of the primary preemption buffer to use for this job */ | 259 | /** Address of the primary preemption buffer to use for this job */ |
| 243 | u64 primary_preempt_buf_addr; | 260 | u64 primary_preempt_buf_addr; |
| 244 | /**< Size of the primary preemption buffer to use for this job */ | 261 | /** Size of the primary preemption buffer to use for this job */ |
| 245 | u32 primary_preempt_buf_size; | 262 | u32 primary_preempt_buf_size; |
| 246 | /**< Size of secondary preemption buffer to use for this job */ | 263 | /** Size of secondary preemption buffer to use for this job */ |
| 247 | u32 secondary_preempt_buf_size; | 264 | u32 secondary_preempt_buf_size; |
| 248 | /**< Address of secondary preemption buffer to use for this job */ | 265 | /** Address of secondary preemption buffer to use for this job */ |
| 249 | u64 secondary_preempt_buf_addr; | 266 | u64 secondary_preempt_buf_addr; |
| 250 | u64 reserved_0; | 267 | u64 reserved_0; |
| 251 | }; | 268 | }; |
| 252 | 269 | ||
| 253 | /* | 270 | /** |
| 254 | * Inline command format. | 271 | * Inline command format. |
| 255 | * Inline commands are the commands executed at scheduler level (typically, | 272 | * Inline commands are the commands executed at scheduler level (typically, |
| 256 | * synchronization directives). Inline command and job objects must be of | 273 | * synchronization directives). Inline command and job objects must be of |
| @@ -258,34 +275,36 @@ struct vpu_job_queue_entry { · 1 unmodified line | |||
| 258 | */ | 275 | */ |
| 259 | struct vpu_inline_cmd { | 276 | struct vpu_inline_cmd { |
| 260 | u64 reserved_0; | 277 | u64 reserved_0; |
| 261 | /* Inline command type, see VPU_INLINE_CMD_TYPE_* defines. */ | 278 | /** Inline command type, see VPU_INLINE_CMD_TYPE_* defines. */ |
| 262 | u32 type; | 279 | u32 type; |
| 263 | /* Flags bit field, see VPU_JOB_FLAGS_* above. */ | 280 | /** Flags bit field, see VPU_JOB_FLAGS_* above. */ |
| 264 | u32 flags; | 281 | u32 flags; |
| 265 | /* Inline command payload. Depends on inline command type. */ | 282 | /** Inline command payload. Depends on inline command type. */ |
| 266 | union { | 283 | union payload { |
| 267 | /* Fence (wait and signal) commands' payload. */ | 284 | /** Fence (wait and signal) commands' payload. */ |
| 268 | struct { | 285 | struct fence { |
| 269 | /* Fence object handle. */ | 286 | /** Fence object handle. */ |
| 270 | u64 fence_handle; | 287 | u64 fence_handle; |
| 271 | /* User VA of the current fence value. */ | 288 | /** User VA of the current fence value. */ |
| 272 | u64 current_value_va; | 289 | u64 current_value_va; |
| 273 | /* User VA of the monitored fence value (read-only). */ | 290 | /** User VA of the monitored fence value (read-only). */ |
| 274 | u64 monitored_value_va; | 291 | u64 monitored_value_va; |
| 275 | /* Value to wait for or write in fence location. */ | 292 | /** Value to wait for or write in fence location. */ |
| 276 | u64 value; | 293 | u64 value; |
| 277 | /* User VA of the log buffer in which to add log entry on completion. */ | 294 | /** User VA of the log buffer in which to add log entry on completion. */ |
| 278 | u64 log_buffer_va; | 295 | u64 log_buffer_va; |
| 279 | /* NPU private data. */ | 296 | /** NPU private data. */ |
| 280 | u64 npu_private_data; | 297 | u64 npu_private_data; |
| 281 | } fence; | 298 | } fence; |
| 282 | /* Other commands do not have a payload. */ | 299 | /** |
| 283 | /* Payload definition for future inline commands can be inserted here. */ | 300 | * Other commands do not have a payload: |
| 301 | * Payload definition for future inline commands can be inserted here. | ||
| 302 | */ | ||
| 284 | u64 reserved_1[6]; | 303 | u64 reserved_1[6]; |
| 285 | } payload; | 304 | } payload; |
| 286 | }; | 305 | }; |
| 287 | 306 | ||
| 288 | /* | 307 | /** |
| 289 | * Job queue slots can be populated either with job objects or inline command objects. | 308 | * Job queue slots can be populated either with job objects or inline command objects. |
| 290 | */ | 309 | */ |
| 291 | union vpu_jobq_slot { | 310 | union vpu_jobq_slot { |
| @@ -293,7 +312,7 @@ union vpu_jobq_slot { · 1 unmodified line | |||
| 293 | struct vpu_inline_cmd inline_cmd; | 312 | struct vpu_inline_cmd inline_cmd; |
| 294 | }; | 313 | }; |
| 295 | 314 | ||
| 296 | /* | 315 | /** |
| 297 | * Job queue control registers. | 316 | * Job queue control registers. |
| 298 | */ | 317 | */ |
| 299 | struct vpu_job_queue_header { | 318 | struct vpu_job_queue_header { |
| @@ -301,18 +320,18 @@ struct vpu_job_queue_header { · 1 unmodified line | |||
| 301 | u32 head; | 320 | u32 head; |
| 302 | u32 tail; | 321 | u32 tail; |
| 303 | u32 flags; | 322 | u32 flags; |
| 304 | /* Set to 1 to indicate priority_band field is valid */ | 323 | /** Set to 1 to indicate priority_band field is valid */ |
| 305 | u32 priority_band_valid; | 324 | u32 priority_band_valid; |
| 306 | /* | 325 | /** |
| 307 | * Priority for the work of this job queue, valid only if the HWS is NOT used | 326 | * Priority for the work of this job queue, valid only if the HWS is NOT used |
| 308 | * and the `priority_band_valid` is set to 1. It is applied only during | 327 | * and the @ref priority_band_valid is set to 1. It is applied only during |
| 309 | * the VPU_JSM_MSG_REGISTER_DB message processing. | 328 | * the @ref VPU_JSM_MSG_REGISTER_DB message processing. |
| 310 | * The device firmware might use the `priority_band` to optimize the power | 329 | * The device firmware might use the priority_band to optimize the power |
| 311 | * management logic, but it will not affect the order of jobs. | 330 | * management logic, but it will not affect the order of jobs. |
| 312 | * Available priority bands: @see enum vpu_job_scheduling_priority_band | 331 | * Available priority bands: @see enum vpu_job_scheduling_priority_band |
| 313 | */ | 332 | */ |
| 314 | u32 priority_band; | 333 | u32 priority_band; |
| 315 | /* Inside realtime band assigns a further priority, limited to 0..31 range */ | 334 | /** Inside realtime band assigns a further priority, limited to 0..31 range */ |
| 316 | u32 realtime_priority_level; | 335 | u32 realtime_priority_level; |
| 317 | u32 reserved_0[9]; | 336 | u32 reserved_0[9]; |
| 318 | }; | 337 | }; |
| @@ -337,16 +356,16 @@ enum vpu_trace_entity_type { · 18 unmodified lines | |||
| 337 | VPU_TRACE_ENTITY_TYPE_HW_COMPONENT = 2, | 356 | VPU_TRACE_ENTITY_TYPE_HW_COMPONENT = 2, |
| 338 | }; | 357 | }; |
| 339 | 358 | ||
| 340 | /* | 359 | /** |
| 341 | * HWS specific log buffer header details. | 360 | * HWS specific log buffer header details. |
| 342 | * Total size is 32 bytes. | 361 | * Total size is 32 bytes. |
| 343 | */ | 362 | */ |
| 344 | struct vpu_hws_log_buffer_header { | 363 | struct vpu_hws_log_buffer_header { |
| 345 | /* Written by VPU after adding a log entry. Initialised by host to 0. */ | 364 | /** Written by VPU after adding a log entry. Initialised by host to 0. */ |
| 346 | u32 first_free_entry_index; | 365 | u32 first_free_entry_index; |
| 347 | /* Incremented by VPU every time the VPU writes the 0th entry; initialised by host to 0. */ | 366 | /** Incremented by VPU every time the VPU writes the 0th entry; initialised by host to 0. */ |
| 348 | u32 wraparound_count; | 367 | u32 wraparound_count; |
| 349 | /* | 368 | /** |
| 350 | * This is the number of buffers that can be stored in the log buffer provided by the host. | 369 | * This is the number of buffers that can be stored in the log buffer provided by the host. |
| 351 | * It is written by host before passing buffer to VPU. VPU should consider it read-only. | 370 | * It is written by host before passing buffer to VPU. VPU should consider it read-only. |
| 352 | */ | 371 | */ |
| @@ -354,14 +373,14 @@ struct vpu_hws_log_buffer_header { · 1 unmodified line | |||
| 354 | u64 reserved[2]; | 373 | u64 reserved[2]; |
| 355 | }; | 374 | }; |
| 356 | 375 | ||
| 357 | /* | 376 | /** |
| 358 | * HWS specific log buffer entry details. | 377 | * HWS specific log buffer entry details. |
| 359 | * Total size is 32 bytes. | 378 | * Total size is 32 bytes. |
| 360 | */ | 379 | */ |
| 361 | struct vpu_hws_log_buffer_entry { | 380 | struct vpu_hws_log_buffer_entry { |
| 362 | /* VPU timestamp must be an invariant timer tick (not impacted by DVFS) */ | 381 | /** VPU timestamp must be an invariant timer tick (not impacted by DVFS) */ |
| 363 | u64 vpu_timestamp; | 382 | u64 vpu_timestamp; |
| 364 | /* | 383 | /** |
| 365 | * Operation type: | 384 | * Operation type: |
| 366 | * 0 - context state change | 385 | * 0 - context state change |
| 367 | * 1 - queue new work | 386 | * 1 - queue new work |
| @@ -371,7 +390,7 @@ struct vpu_hws_log_buffer_entry { · 3 unmodified lines | |||
| 371 | */ | 390 | */ |
| 372 | u32 operation_type; | 391 | u32 operation_type; |
| 373 | u32 reserved; | 392 | u32 reserved; |
| 374 | /* Operation data depends on operation type */ | 393 | /** Operation data depends on operation type */ |
| 375 | u64 operation_data[2]; | 394 | u64 operation_data[2]; |
| 376 | }; | 395 | }; |
| 377 | 396 | ||
| @@ -381,51 +400,54 @@ enum vpu_hws_native_fence_log_type { · 3 unmodified lines | |||
| 381 | VPU_HWS_NATIVE_FENCE_LOG_TYPE_SIGNALS = 2 | 400 | VPU_HWS_NATIVE_FENCE_LOG_TYPE_SIGNALS = 2 |
| 382 | }; | 401 | }; |
| 383 | 402 | ||
| 384 | /* HWS native fence log buffer header. */ | 403 | /** HWS native fence log buffer header. */ |
| 385 | struct vpu_hws_native_fence_log_header { | 404 | struct vpu_hws_native_fence_log_header { |
| 386 | union { | 405 | union { |
| 387 | struct { | 406 | struct { |
| 388 | /* Index of the first free entry in buffer. */ | 407 | /** Index of the first free entry in buffer. */ |
| 389 | u32 first_free_entry_idx; | 408 | u32 first_free_entry_idx; |
| 390 | /* Incremented each time NPU wraps around the buffer to write next entry. */ | 409 | /** |
| 410 | * Incremented whenever the NPU wraps around the buffer and writes | ||
| 411 | * to the first entry again. | ||
| 412 | */ | ||
| 391 | u32 wraparound_count; | 413 | u32 wraparound_count; |
| 392 | }; | 414 | }; |
| 393 | /* Field allowing atomic update of both fields above. */ | 415 | /** Field allowing atomic update of both fields above. */ |
| 394 | u64 atomic_wraparound_and_entry_idx; | 416 | u64 atomic_wraparound_and_entry_idx; |
| 395 | }; | 417 | }; |
| 396 | /* Log buffer type, see enum vpu_hws_native_fence_log_type. */ | 418 | /** Log buffer type, see enum vpu_hws_native_fence_log_type. */ |
| 397 | u64 type; | 419 | u64 type; |
| 398 | /* Allocated number of entries in the log buffer. */ | 420 | /** Allocated number of entries in the log buffer. */ |
| 399 | u64 entry_nb; | 421 | u64 entry_nb; |
| 400 | u64 reserved[2]; | 422 | u64 reserved[2]; |
| 401 | }; | 423 | }; |
| 402 | 424 | ||
| 403 | /* Native fence log operation types. */ | 425 | /** Native fence log operation types. */ |
| 404 | enum vpu_hws_native_fence_log_op { | 426 | enum vpu_hws_native_fence_log_op { |
| 405 | VPU_HWS_NATIVE_FENCE_LOG_OP_SIGNAL_EXECUTED = 0, | 427 | VPU_HWS_NATIVE_FENCE_LOG_OP_SIGNAL_EXECUTED = 0, |
| 406 | VPU_HWS_NATIVE_FENCE_LOG_OP_WAIT_UNBLOCKED = 1 | 428 | VPU_HWS_NATIVE_FENCE_LOG_OP_WAIT_UNBLOCKED = 1 |
| 407 | }; | 429 | }; |
| 408 | 430 | ||
| 409 | /* HWS native fence log entry. */ | 431 | /** HWS native fence log entry. */ |
| 410 | struct vpu_hws_native_fence_log_entry { | 432 | struct vpu_hws_native_fence_log_entry { |
| 411 | /* Newly signaled/unblocked fence value. */ | 433 | /** Newly signaled/unblocked fence value. */ |
| 412 | u64 fence_value; | 434 | u64 fence_value; |
| 413 | /* Native fence object handle to which this operation belongs. */ | 435 | /** Native fence object handle to which this operation belongs. */ |
| 414 | u64 fence_handle; | 436 | u64 fence_handle; |
| 415 | /* Operation type, see enum vpu_hws_native_fence_log_op. */ | 437 | /** Operation type, see enum vpu_hws_native_fence_log_op. */ |
| 416 | u64 op_type; | 438 | u64 op_type; |
| 417 | u64 reserved_0; | 439 | u64 reserved_0; |
| 418 | /* | 440 | /** |
| 419 | * VPU_HWS_NATIVE_FENCE_LOG_OP_WAIT_UNBLOCKED only: Timestamp at which fence | 441 | * VPU_HWS_NATIVE_FENCE_LOG_OP_WAIT_UNBLOCKED only: Timestamp at which fence |
| 420 | * wait was started (in NPU SysTime). | 442 | * wait was started (in NPU SysTime). |
| 421 | */ | 443 | */ |
| 422 | u64 fence_wait_start_ts; | 444 | u64 fence_wait_start_ts; |
| 423 | u64 reserved_1; | 445 | u64 reserved_1; |
| 424 | /* Timestamp at which fence operation was completed (in NPU SysTime). */ | 446 | /** Timestamp at which fence operation was completed (in NPU SysTime). */ |
| 425 | u64 fence_end_ts; | 447 | u64 fence_end_ts; |
| 426 | }; | 448 | }; |
| 427 | 449 | ||
| 428 | /* Native fence log buffer. */ | 450 | /** Native fence log buffer. */ |
| 429 | struct vpu_hws_native_fence_log_buffer { | 451 | struct vpu_hws_native_fence_log_buffer { |
| 430 | struct vpu_hws_native_fence_log_header header; | 452 | struct vpu_hws_native_fence_log_header header; |
| 431 | struct vpu_hws_native_fence_log_entry entry[]; | 453 | struct vpu_hws_native_fence_log_entry entry[]; |
| @@ -435,10 +457,17 @@ struct vpu_hws_native_fence_log_buffer { · 3 unmodified lines | |||
| 435 | * Host <-> VPU IPC messages types. | 457 | * Host <-> VPU IPC messages types. |
| 436 | */ | 458 | */ |
| 437 | enum vpu_ipc_msg_type { | 459 | enum vpu_ipc_msg_type { |
| 460 | /** Unsupported command */ | ||
| 438 | VPU_JSM_MSG_UNKNOWN = 0xFFFFFFFF, | 461 | VPU_JSM_MSG_UNKNOWN = 0xFFFFFFFF, |
| 439 | 462 | ||
| 440 | /* IPC Host -> Device, Async commands */ | 463 | /** IPC Host -> Device, base id for async commands */ |
| 441 | VPU_JSM_MSG_ASYNC_CMD = 0x1100, | 464 | VPU_JSM_MSG_ASYNC_CMD = 0x1100, |
| 465 | /** | ||
| 466 | * Reset engine. The NPU cancels all the jobs currently executing on the target | ||
| 467 | * engine making the engine become idle and then does a HW reset, before returning | ||
| 468 | * to the host. | ||
| 469 | * @see struct vpu_ipc_msg_payload_engine_reset | ||
| 470 | */ | ||
| 442 | VPU_JSM_MSG_ENGINE_RESET = VPU_JSM_MSG_ASYNC_CMD, | 471 | VPU_JSM_MSG_ENGINE_RESET = VPU_JSM_MSG_ASYNC_CMD, |
| 443 | /** | 472 | /** |
| 444 | * Preempt engine. The NPU stops (preempts) all the jobs currently | 473 | * Preempt engine. The NPU stops (preempts) all the jobs currently |
| @@ -448,10 +477,24 @@ enum vpu_ipc_msg_type { · 3 unmodified lines | |||
| 448 | * the target engine, but it stops processing them (until the queue doorbell | 477 | * the target engine, but it stops processing them (until the queue doorbell |
| 449 | * is rung again); the host is responsible to reset the job queue, either | 478 | * is rung again); the host is responsible to reset the job queue, either |
| 450 | * after preemption or when resubmitting jobs to the queue. | 479 | * after preemption or when resubmitting jobs to the queue. |
| 480 | * @see vpu_ipc_msg_payload_engine_preempt | ||
| 451 | */ | 481 | */ |
| 452 | VPU_JSM_MSG_ENGINE_PREEMPT = 0x1101, | 482 | VPU_JSM_MSG_ENGINE_PREEMPT = 0x1101, |
| 483 | /** | ||
| 484 | * OS scheduling doorbell register command | ||
| 485 | * @see vpu_ipc_msg_payload_register_db | ||
| 486 | */ | ||
| 453 | VPU_JSM_MSG_REGISTER_DB = 0x1102, | 487 | VPU_JSM_MSG_REGISTER_DB = 0x1102, |
| 488 | /** | ||
| 489 | * OS scheduling doorbell unregister command | ||
| 490 | * @see vpu_ipc_msg_payload_unregister_db | ||
| 491 | */ | ||
| 454 | VPU_JSM_MSG_UNREGISTER_DB = 0x1103, | 492 | VPU_JSM_MSG_UNREGISTER_DB = 0x1103, |
| 493 | /** | ||
| 494 | * Query engine heartbeat. Heartbeat is expected to increase monotonically | ||
| 495 | * and increase while work is being progressed by NPU. | ||
| 496 | * @see vpu_ipc_msg_payload_query_engine_hb | ||
| 497 | */ | ||
| 455 | VPU_JSM_MSG_QUERY_ENGINE_HB = 0x1104, | 498 | VPU_JSM_MSG_QUERY_ENGINE_HB = 0x1104, |
| 456 | VPU_JSM_MSG_GET_POWER_LEVEL_COUNT = 0x1105, | 499 | VPU_JSM_MSG_GET_POWER_LEVEL_COUNT = 0x1105, |
| 457 | VPU_JSM_MSG_GET_POWER_LEVEL = 0x1106, | 500 | VPU_JSM_MSG_GET_POWER_LEVEL = 0x1106, |
| @@ -477,6 +520,7 @@ enum vpu_ipc_msg_type { · 19 unmodified lines | |||
| 477 | * aborted and removed from internal scheduling queues. All doorbells assigned | 520 | * aborted and removed from internal scheduling queues. All doorbells assigned |
| 478 | * to the host_ssid are unregistered and any internal FW resources belonging to | 521 | * to the host_ssid are unregistered and any internal FW resources belonging to |
| 479 | * the host_ssid are released. | 522 | * the host_ssid are released. |
| 523 | * @see vpu_ipc_msg_payload_ssid_release | ||
| 480 | */ | 524 | */ |
| 481 | VPU_JSM_MSG_SSID_RELEASE = 0x110e, | 525 | VPU_JSM_MSG_SSID_RELEASE = 0x110e, |
| 482 | /** | 526 | /** |
| @@ -504,43 +548,78 @@ enum vpu_ipc_msg_type { · 21 unmodified lines | |||
| 504 | * @see vpu_jsm_metric_streamer_start | 548 | * @see vpu_jsm_metric_streamer_start |
| 505 | */ | 549 | */ |
| 506 | VPU_JSM_MSG_METRIC_STREAMER_INFO = 0x1112, | 550 | VPU_JSM_MSG_METRIC_STREAMER_INFO = 0x1112, |
| 507 | /** Control command: Priority band setup */ | 551 | /** |
| 552 | * Control command: Priority band setup | ||
| 553 | * @see vpu_ipc_msg_payload_hws_priority_band_setup | ||
| 554 | */ | ||
| 508 | VPU_JSM_MSG_SET_PRIORITY_BAND_SETUP = 0x1113, | 555 | VPU_JSM_MSG_SET_PRIORITY_BAND_SETUP = 0x1113, |
| 509 | /** Control command: Create command queue */ | 556 | /** |
| 557 | * Control command: Create command queue | ||
| 558 | * @see vpu_ipc_msg_payload_hws_create_cmdq | ||
| 559 | */ | ||
| 510 | VPU_JSM_MSG_CREATE_CMD_QUEUE = 0x1114, | 560 | VPU_JSM_MSG_CREATE_CMD_QUEUE = 0x1114, |
| 511 | /** Control command: Destroy command queue */ | 561 | /** |
| 562 | * Control command: Destroy command queue | ||
| 563 | * @see vpu_ipc_msg_payload_hws_destroy_cmdq | ||
| 564 | */ | ||
| 512 | VPU_JSM_MSG_DESTROY_CMD_QUEUE = 0x1115, | 565 | VPU_JSM_MSG_DESTROY_CMD_QUEUE = 0x1115, |
| 513 | /** Control command: Set context scheduling properties */ | 566 | /** |
| 567 | * Control command: Set context scheduling properties | ||
| 568 | * @see vpu_ipc_msg_payload_hws_set_context_sched_properties | ||
| 569 | */ | ||
| 514 | VPU_JSM_MSG_SET_CONTEXT_SCHED_PROPERTIES = 0x1116, | 570 | VPU_JSM_MSG_SET_CONTEXT_SCHED_PROPERTIES = 0x1116, |
| 515 | /* | 571 | /** |
| 516 | * Register a doorbell to notify VPU of new work. The doorbell may later be | 572 | * Register a doorbell to notify VPU of new work. The doorbell may later be |
| 517 | * deallocated or reassigned to another context. | 573 | * deallocated or reassigned to another context. |
| 574 | * @see vpu_jsm_hws_register_db | ||
| 518 | */ | 575 | */ |
| 519 | VPU_JSM_MSG_HWS_REGISTER_DB = 0x1117, | 576 | VPU_JSM_MSG_HWS_REGISTER_DB = 0x1117, |
| 520 | /** Control command: Log buffer setting */ | 577 | /** |
| 578 | * Control command: Log buffer setting | ||
| 579 | * @see vpu_ipc_msg_payload_hws_set_scheduling_log | ||
| 580 | */ | ||
| 521 | VPU_JSM_MSG_HWS_SET_SCHEDULING_LOG = 0x1118, | 581 | VPU_JSM_MSG_HWS_SET_SCHEDULING_LOG = 0x1118, |
| 522 | /* Control command: Suspend command queue. */ | 582 | /** |
| 583 | * Control command: Suspend command queue. | ||
| 584 | * @see vpu_ipc_msg_payload_hws_suspend_cmdq | ||
| 585 | */ | ||
| 523 | VPU_JSM_MSG_HWS_SUSPEND_CMDQ = 0x1119, | 586 | VPU_JSM_MSG_HWS_SUSPEND_CMDQ = 0x1119, |
| 524 | /* Control command: Resume command queue */ | 587 | /** |
| 588 | * Control command: Resume command queue | ||
| 589 | * @see vpu_ipc_msg_payload_hws_resume_cmdq | ||
| 590 | */ | ||
| 525 | VPU_JSM_MSG_HWS_RESUME_CMDQ = 0x111a, | 591 | VPU_JSM_MSG_HWS_RESUME_CMDQ = 0x111a, |
| 526 | /* Control command: Resume engine after reset */ | 592 | /** |
| 593 | * Control command: Resume engine after reset | ||
| 594 | * @see vpu_ipc_msg_payload_hws_resume_engine | ||
| 595 | */ | ||
| 527 | VPU_JSM_MSG_HWS_ENGINE_RESUME = 0x111b, | 596 | VPU_JSM_MSG_HWS_ENGINE_RESUME = 0x111b, |
| 528 | /* Control command: Enable survivability/DCT mode */ | 597 | /** |
| 598 | * Control command: Enable survivability/DCT mode | ||
| 599 | * @see vpu_ipc_msg_payload_pwr_dct_control | ||
| 600 | */ | ||
| 529 | VPU_JSM_MSG_DCT_ENABLE = 0x111c, | 601 | VPU_JSM_MSG_DCT_ENABLE = 0x111c, |
| 530 | /* Control command: Disable survivability/DCT mode */ | 602 | /** |
| 603 | * Control command: Disable survivability/DCT mode | ||
| 604 | * This command has no payload | ||
| 605 | */ | ||
| 531 | VPU_JSM_MSG_DCT_DISABLE = 0x111d, | 606 | VPU_JSM_MSG_DCT_DISABLE = 0x111d, |
| 532 | /** | 607 | /** |
| 533 | * Dump VPU state. To be used for debug purposes only. | 608 | * Dump VPU state. To be used for debug purposes only. |
| 534 | * NOTE: Please introduce new ASYNC commands before this one. * | 609 | * This command has no payload. |
| 610 | * NOTE: Please introduce new ASYNC commands before this one. | ||
| 535 | */ | 611 | */ |
| 536 | VPU_JSM_MSG_STATE_DUMP = 0x11FF, | 612 | VPU_JSM_MSG_STATE_DUMP = 0x11FF, |
| 537 | 613 | ||
| 538 | /* IPC Host -> Device, General commands */ | 614 | /** IPC Host -> Device, base id for general commands */ |
| 539 | VPU_JSM_MSG_GENERAL_CMD = 0x1200, | 615 | VPU_JSM_MSG_GENERAL_CMD = 0x1200, |
| 616 | /** Unsupported command */ | ||
| 540 | VPU_JSM_MSG_BLOB_DEINIT_DEPRECATED = VPU_JSM_MSG_GENERAL_CMD, | 617 | VPU_JSM_MSG_BLOB_DEINIT_DEPRECATED = VPU_JSM_MSG_GENERAL_CMD, |
| 541 | /** | 618 | /** |
| 542 | * Control dyndbg behavior by executing a dyndbg command; equivalent to | 619 | * Control dyndbg behavior by executing a dyndbg command; equivalent to |
| 543 | * Linux command: `echo '<dyndbg_cmd>' > <debugfs>/dynamic_debug/control`. | 620 | * Linux command: |
| 621 | * @verbatim echo '<dyndbg_cmd>' > <debugfs>/dynamic_debug/control @endverbatim | ||
| 622 | * @see vpu_ipc_msg_payload_dyndbg_control | ||
| 544 | */ | 623 | */ |
| 545 | VPU_JSM_MSG_DYNDBG_CONTROL = 0x1201, | 624 | VPU_JSM_MSG_DYNDBG_CONTROL = 0x1201, |
| 546 | /** | 625 | /** |
| @@ -548,17 +627,35 @@ enum vpu_ipc_msg_type { · 1 unmodified line | |||
| 548 | */ | 627 | */ |
| 549 | VPU_JSM_MSG_PWR_D0I3_ENTER = 0x1202, | 628 | VPU_JSM_MSG_PWR_D0I3_ENTER = 0x1202, |
| 550 | 629 | ||
| 551 | /* IPC Device -> Host, Job completion */ | 630 | /** |
| 631 | * IPC Device -> Host, Job completion | ||
| 632 | * @see struct vpu_ipc_msg_payload_job_done | ||
| 633 | */ | ||
| 552 | VPU_JSM_MSG_JOB_DONE = 0x2100, | 634 | VPU_JSM_MSG_JOB_DONE = 0x2100, |
| 553 | /* IPC Device -> Host, Fence signalled */ | 635 | /** |
| 636 | * IPC Device -> Host, Fence signalled | ||
| 637 | * @see vpu_ipc_msg_payload_native_fence_signalled | ||
| 638 | */ | ||
| 554 | VPU_JSM_MSG_NATIVE_FENCE_SIGNALLED = 0x2101, | 639 | VPU_JSM_MSG_NATIVE_FENCE_SIGNALLED = 0x2101, |
| 555 | 640 | ||
| 556 | /* IPC Device -> Host, Async command completion */ | 641 | /* IPC Device -> Host, Async command completion */ |
| 557 | VPU_JSM_MSG_ASYNC_CMD_DONE = 0x2200, | 642 | VPU_JSM_MSG_ASYNC_CMD_DONE = 0x2200, |
| 643 | /** | ||
| 644 | * IPC Device -> Host, engine reset complete | ||
| 645 | * @see vpu_ipc_msg_payload_engine_reset_done | ||
| 646 | */ | ||
| 558 | VPU_JSM_MSG_ENGINE_RESET_DONE = VPU_JSM_MSG_ASYNC_CMD_DONE, | 647 | VPU_JSM_MSG_ENGINE_RESET_DONE = VPU_JSM_MSG_ASYNC_CMD_DONE, |
| 648 | /** | ||
| 649 | * Preempt complete message | ||
| 650 | * @see vpu_ipc_msg_payload_engine_preempt_done | ||
| 651 | */ | ||
| 559 | VPU_JSM_MSG_ENGINE_PREEMPT_DONE = 0x2201, | 652 | VPU_JSM_MSG_ENGINE_PREEMPT_DONE = 0x2201, |
| 560 | VPU_JSM_MSG_REGISTER_DB_DONE = 0x2202, | 653 | VPU_JSM_MSG_REGISTER_DB_DONE = 0x2202, |
| 561 | VPU_JSM_MSG_UNREGISTER_DB_DONE = 0x2203, | 654 | VPU_JSM_MSG_UNREGISTER_DB_DONE = 0x2203, |
| 655 | /** | ||
| 656 | * Response to query engine heartbeat. | ||
| 657 | * @see vpu_ipc_msg_payload_query_engine_hb_done | ||
| 658 | */ | ||
| 562 | VPU_JSM_MSG_QUERY_ENGINE_HB_DONE = 0x2204, | 659 | VPU_JSM_MSG_QUERY_ENGINE_HB_DONE = 0x2204, |
| 563 | VPU_JSM_MSG_GET_POWER_LEVEL_COUNT_DONE = 0x2205, | 660 | VPU_JSM_MSG_GET_POWER_LEVEL_COUNT_DONE = 0x2205, |
| 564 | VPU_JSM_MSG_GET_POWER_LEVEL_DONE = 0x2206, | 661 | VPU_JSM_MSG_GET_POWER_LEVEL_DONE = 0x2206, |
| @@ -575,7 +672,10 @@ enum vpu_ipc_msg_type { · 10 unmodified lines | |||
| 575 | VPU_JSM_MSG_TRACE_GET_CAPABILITY_RSP = 0x220c, | 672 | VPU_JSM_MSG_TRACE_GET_CAPABILITY_RSP = 0x220c, |
| 576 | /** Response to VPU_JSM_MSG_TRACE_GET_NAME. */ | 673 | /** Response to VPU_JSM_MSG_TRACE_GET_NAME. */ |
| 577 | VPU_JSM_MSG_TRACE_GET_NAME_RSP = 0x220d, | 674 | VPU_JSM_MSG_TRACE_GET_NAME_RSP = 0x220d, |
| 578 | /** Response to VPU_JSM_MSG_SSID_RELEASE. */ | 675 | /** |
| 676 | * Response to VPU_JSM_MSG_SSID_RELEASE. | ||
| 677 | * @see vpu_ipc_msg_payload_ssid_release | ||
| 678 | */ | ||
| 579 | VPU_JSM_MSG_SSID_RELEASE_DONE = 0x220e, | 679 | VPU_JSM_MSG_SSID_RELEASE_DONE = 0x220e, |
| 580 | /** | 680 | /** |
| 581 | * Response to VPU_JSM_MSG_METRIC_STREAMER_START. | 681 | * Response to VPU_JSM_MSG_METRIC_STREAMER_START. |
| @@ -605,37 +705,71 @@ enum vpu_ipc_msg_type { · 23 unmodified lines | |||
| 605 | /** | 705 | /** |
| 606 | * Asynchronous event sent from the VPU to the host either when the current | 706 | * Asynchronous event sent from the VPU to the host either when the current |
| 607 | * metric buffer is full or when the VPU has collected a multiple of | 707 | * metric buffer is full or when the VPU has collected a multiple of |
| 608 | * @notify_sample_count samples as indicated through the start command | 708 | * @ref vpu_jsm_metric_streamer_start::notify_sample_count samples as indicated |
| 609 | * (VPU_JSM_MSG_METRIC_STREAMER_START). Returns information about collected | 709 | * through the start command (VPU_JSM_MSG_METRIC_STREAMER_START). Returns |
| 610 | * metric data. | 710 | * information about collected metric data. |
| 611 | * @see vpu_jsm_metric_streamer_done | 711 | * @see vpu_jsm_metric_streamer_done |
| 612 | */ | 712 | */ |
| 613 | VPU_JSM_MSG_METRIC_STREAMER_NOTIFICATION = 0x2213, | 713 | VPU_JSM_MSG_METRIC_STREAMER_NOTIFICATION = 0x2213, |
| 614 | /** Response to control command: Priority band setup */ | 714 | /** |
| 715 | * Response to control command: Priority band setup | ||
| 716 | * @see vpu_ipc_msg_payload_hws_priority_band_setup | ||
| 717 | */ | ||
| 615 | VPU_JSM_MSG_SET_PRIORITY_BAND_SETUP_RSP = 0x2214, | 718 | VPU_JSM_MSG_SET_PRIORITY_BAND_SETUP_RSP = 0x2214, |
| 616 | /** Response to control command: Create command queue */ | 719 | /** |
| 720 | * Response to control command: Create command queue | ||
| 721 | * @see vpu_ipc_msg_payload_hws_create_cmdq_rsp | ||
| 722 | */ | ||
| 617 | VPU_JSM_MSG_CREATE_CMD_QUEUE_RSP = 0x2215, | 723 | VPU_JSM_MSG_CREATE_CMD_QUEUE_RSP = 0x2215, |
| 618 | /** Response to control command: Destroy command queue */ | 724 | /** |
| 725 | * Response to control command: Destroy command queue | ||
| 726 | * @see vpu_ipc_msg_payload_hws_destroy_cmdq | ||
| 727 | */ | ||
| 619 | VPU_JSM_MSG_DESTROY_CMD_QUEUE_RSP = 0x2216, | 728 | VPU_JSM_MSG_DESTROY_CMD_QUEUE_RSP = 0x2216, |
| 620 | /** Response to control command: Set context scheduling properties */ | 729 | /** |
| 730 | * Response to control command: Set context scheduling properties | ||
| 731 | * @see vpu_ipc_msg_payload_hws_set_context_sched_properties | ||
| 732 | */ | ||
| 621 | VPU_JSM_MSG_SET_CONTEXT_SCHED_PROPERTIES_RSP = 0x2217, | 733 | VPU_JSM_MSG_SET_CONTEXT_SCHED_PROPERTIES_RSP = 0x2217, |
| 622 | /** Response to control command: Log buffer setting */ | 734 | /** |
| 735 | * Response to control command: Log buffer setting | ||
| 736 | * @see vpu_ipc_msg_payload_hws_set_scheduling_log | ||
| 737 | */ | ||
| 623 | VPU_JSM_MSG_HWS_SET_SCHEDULING_LOG_RSP = 0x2218, | 738 | VPU_JSM_MSG_HWS_SET_SCHEDULING_LOG_RSP = 0x2218, |
| 624 | /* IPC Device -> Host, HWS notify index entry of log buffer written */ | 739 | /** |
| 740 | * IPC Device -> Host, HWS notify index entry of log buffer written | ||
| 741 | * @see vpu_ipc_msg_payload_hws_scheduling_log_notification | ||
| 742 | */ | ||
| 625 | VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION = 0x2219, | 743 | VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION = 0x2219, |
| 626 | /* IPC Device -> Host, HWS completion of a context suspend request */ | 744 | /** |
| 745 | * IPC Device -> Host, HWS completion of a context suspend request | ||
| 746 | * @see vpu_ipc_msg_payload_hws_suspend_cmdq | ||
| 747 | */ | ||
| 627 | VPU_JSM_MSG_HWS_SUSPEND_CMDQ_DONE = 0x221a, | 748 | VPU_JSM_MSG_HWS_SUSPEND_CMDQ_DONE = 0x221a, |
| 628 | /* Response to control command: Resume command queue */ | 749 | /** |
| 750 | * Response to control command: Resume command queue | ||
| 751 | * @see vpu_ipc_msg_payload_hws_resume_cmdq | ||
| 752 | */ | ||
| 629 | VPU_JSM_MSG_HWS_RESUME_CMDQ_RSP = 0x221b, | 753 | VPU_JSM_MSG_HWS_RESUME_CMDQ_RSP = 0x221b, |
| 630 | /* Response to control command: Resume engine command response */ | 754 | /** |
| 755 | * Response to control command: Resume engine command response | ||
| 756 | * @see vpu_ipc_msg_payload_hws_resume_engine | ||
| 757 | */ | ||
| 631 | VPU_JSM_MSG_HWS_RESUME_ENGINE_DONE = 0x221c, | 758 | VPU_JSM_MSG_HWS_RESUME_ENGINE_DONE = 0x221c, |
| 632 | /* Response to control command: Enable survivability/DCT mode */ | 759 | /** |
| 760 | * Response to control command: Enable survivability/DCT mode | ||
| 761 | * This command has no payload | ||
| 762 | */ | ||
| 633 | VPU_JSM_MSG_DCT_ENABLE_DONE = 0x221d, | 763 | VPU_JSM_MSG_DCT_ENABLE_DONE = 0x221d, |
| 634 | /* Response to control command: Disable survivability/DCT mode */ | 764 | /** |
| 765 | * Response to control command: Disable survivability/DCT mode | ||
| 766 | * This command has no payload | ||
| 767 | */ | ||
| 635 | VPU_JSM_MSG_DCT_DISABLE_DONE = 0x221e, | 768 | VPU_JSM_MSG_DCT_DISABLE_DONE = 0x221e, |
| 636 | /** | 769 | /** |
| 637 | * Response to state dump control command. | 770 | * Response to state dump control command. |
| 638 | * NOTE: Please introduce new ASYNC responses before this one. * | 771 | * This command has no payload. |
| 772 | * NOTE: Please introduce new ASYNC responses before this one. | ||
| 639 | */ | 773 | */ |
| 640 | VPU_JSM_MSG_STATE_DUMP_RSP = 0x22FF, | 774 | VPU_JSM_MSG_STATE_DUMP_RSP = 0x22FF, |
| 641 | 775 | ||
| @@ -653,57 +787,66 @@ enum vpu_ipc_msg_type { · 11 unmodified lines | |||
| 653 | 787 | ||
| 654 | enum vpu_ipc_msg_status { VPU_JSM_MSG_FREE, VPU_JSM_MSG_ALLOCATED }; | 788 | enum vpu_ipc_msg_status { VPU_JSM_MSG_FREE, VPU_JSM_MSG_ALLOCATED }; |
| 655 | 789 | ||
| 656 | /* | 790 | /** |
| 657 | * Host <-> LRT IPC message payload definitions | 791 | * Engine reset request payload |
| 792 | * @see VPU_JSM_MSG_ENGINE_RESET | ||
| 658 | */ | 793 | */ |
| 659 | struct vpu_ipc_msg_payload_engine_reset { | 794 | struct vpu_ipc_msg_payload_engine_reset { |
| 660 | /* Engine to be reset. */ | 795 | /** Engine to be reset. */ |
| 661 | u32 engine_idx; | 796 | u32 engine_idx; |
| 662 | /* Reserved */ | 797 | /** Reserved */ |
| 663 | u32 reserved_0; | 798 | u32 reserved_0; |
| 664 | }; | 799 | }; |
| 665 | 800 | ||
| 801 | /** | ||
| 802 | * Engine preemption request struct | ||
| 803 | * @see VPU_JSM_MSG_ENGINE_PREEMPT | ||
| 804 | */ | ||
| 666 | struct vpu_ipc_msg_payload_engine_preempt { | 805 | struct vpu_ipc_msg_payload_engine_preempt { |
| 667 | /* Engine to be preempted. */ | 806 | /** Engine to be preempted. */ |
| 668 | u32 engine_idx; | 807 | u32 engine_idx; |
| 669 | /* ID of the preemption request. */ | 808 | /** ID of the preemption request. */ |
| 670 | u32 preempt_id; | 809 | u32 preempt_id; |
| 671 | }; | 810 | }; |
| 672 | 811 | ||
| 673 | /* | 812 | /** |
| 674 | * @brief Register doorbell command structure. | 813 | * Register doorbell command structure. |
| 675 | * This structure supports doorbell registration for only OS scheduling. | 814 | * This structure supports doorbell registration for only OS scheduling. |
| 676 | * @see VPU_JSM_MSG_REGISTER_DB | 815 | * @see VPU_JSM_MSG_REGISTER_DB |
| 677 | */ | 816 | */ |
| 678 | struct vpu_ipc_msg_payload_register_db { | 817 | struct vpu_ipc_msg_payload_register_db { |
| 679 | /* Index of the doorbell to register. */ | 818 | /** Index of the doorbell to register. */ |
| 680 | u32 db_idx; | 819 | u32 db_idx; |
| 681 | /* Reserved */ | 820 | /** Reserved */ |
| 682 | u32 reserved_0; | 821 | u32 reserved_0; |
| 683 | /* Virtual address in Global GTT pointing to the start of job queue. */ | 822 | /** Virtual address in Global GTT pointing to the start of job queue. */ |
| 684 | u64 jobq_base; | 823 | u64 jobq_base; |
| 685 | /* Size of the job queue in bytes. */ | 824 | /** Size of the job queue in bytes. */ |
| 686 | u32 jobq_size; | 825 | u32 jobq_size; |
| 687 | /* Host sub-stream ID for the context assigned to the doorbell. */ | 826 | /** Host sub-stream ID for the context assigned to the doorbell. */ |
| 688 | u32 host_ssid; | 827 | u32 host_ssid; |
| 689 | }; | 828 | }; |
| 690 | 829 | ||
| 691 | /** | 830 | /** |
| 692 | * @brief Unregister doorbell command structure. | 831 | * Unregister doorbell command structure. |
| 693 | * Request structure to unregister a doorbell for both HW and OS scheduling. | 832 | * Request structure to unregister a doorbell for both HW and OS scheduling. |
| 694 | * @see VPU_JSM_MSG_UNREGISTER_DB | 833 | * @see VPU_JSM_MSG_UNREGISTER_DB |
| 695 | */ | 834 | */ |
| 696 | struct vpu_ipc_msg_payload_unregister_db { | 835 | struct vpu_ipc_msg_payload_unregister_db { |
| 697 | /* Index of the doorbell to unregister. */ | 836 | /** Index of the doorbell to unregister. */ |
| 698 | u32 db_idx; | 837 | u32 db_idx; |
| 699 | /* Reserved */ | 838 | /** Reserved */ |
| 700 | u32 reserved_0; | 839 | u32 reserved_0; |
| 701 | }; | 840 | }; |
| 702 | 841 | ||
| 842 | /** | ||
| 843 | * Heartbeat request structure | ||
| 844 | * @see VPU_JSM_MSG_QUERY_ENGINE_HB | ||
| 845 | */ | ||
| 703 | struct vpu_ipc_msg_payload_query_engine_hb { | 846 | struct vpu_ipc_msg_payload_query_engine_hb { |
| 704 | /* Engine to return heartbeat value. */ | 847 | /** Engine to return heartbeat value. */ |
| 705 | u32 engine_idx; | 848 | u32 engine_idx; |
| 706 | /* Reserved */ | 849 | /** Reserved */ |
| 707 | u32 reserved_0; | 850 | u32 reserved_0; |
| 708 | }; | 851 | }; |
| 709 | 852 | ||
| @@ -723,10 +866,14 @@ struct vpu_ipc_msg_payload_power_level { · 13 unmodified lines | |||
| 723 | u32 reserved_0; | 866 | u32 reserved_0; |
| 724 | }; | 867 | }; |
| 725 | 868 | ||
| 869 | /** | ||
| 870 | * Structure for requesting ssid release | ||
| 871 | * @see VPU_JSM_MSG_SSID_RELEASE | ||
| 872 | */ | ||
| 726 | struct vpu_ipc_msg_payload_ssid_release { | 873 | struct vpu_ipc_msg_payload_ssid_release { |
| 727 | /* Host sub-stream ID for the context to be released. */ | 874 | /** Host sub-stream ID for the context to be released. */ |
| 728 | u32 host_ssid; | 875 | u32 host_ssid; |
| 729 | /* Reserved */ | 876 | /** Reserved */ |
| 730 | u32 reserved_0; | 877 | u32 reserved_0; |
| 731 | }; | 878 | }; |
| 732 | 879 | ||
| @@ -752,7 +899,7 @@ struct vpu_jsm_metric_streamer_start { · 19 unmodified lines | |||
| 752 | u64 sampling_rate; | 899 | u64 sampling_rate; |
| 753 | /** | 900 | /** |
| 754 | * If > 0 the VPU will send a VPU_JSM_MSG_METRIC_STREAMER_NOTIFICATION message | 901 | * If > 0 the VPU will send a VPU_JSM_MSG_METRIC_STREAMER_NOTIFICATION message |
| 755 | * after every @notify_sample_count samples is collected or dropped by the VPU. | 902 | * after every @ref notify_sample_count samples is collected or dropped by the VPU. |
| 756 | * If set to UINT_MAX the VPU will only generate a notification when the metric | 903 | * If set to UINT_MAX the VPU will only generate a notification when the metric |
| 757 | * buffer is full. If set to 0 the VPU will never generate a notification. | 904 | * buffer is full. If set to 0 the VPU will never generate a notification. |
| 758 | */ | 905 | */ |
| @@ -762,9 +909,9 @@ struct vpu_jsm_metric_streamer_start { · 3 unmodified lines | |||
| 762 | * Address and size of the buffer where the VPU will write metric data. The | 909 | * Address and size of the buffer where the VPU will write metric data. The |
| 763 | * VPU writes all counters from enabled metric groups one after another. If | 910 | * VPU writes all counters from enabled metric groups one after another. If |
| 764 | * there is no space left to write data at the next sample period the VPU | 911 | * there is no space left to write data at the next sample period the VPU |
| 765 | * will switch to the next buffer (@see next_buffer_addr) and will optionally | 912 | * will switch to the next buffer (@ref next_buffer_addr) and will optionally |
| 766 | * send a notification to the host driver if @notify_sample_count is non-zero. | 913 | * send a notification to the host driver if @ref notify_sample_count is non-zero. |
| 767 | * If @next_buffer_addr is NULL the VPU will stop collecting metric data. | 914 | * If @ref next_buffer_addr is NULL the VPU will stop collecting metric data. |
| 768 | */ | 915 | */ |
| 769 | u64 buffer_addr; | 916 | u64 buffer_addr; |
| 770 | u64 buffer_size; | 917 | u64 buffer_size; |
| @@ -827,63 +974,80 @@ struct vpu_jsm_metric_streamer_update { · 56 unmodified lines | |||
| 827 | u64 next_buffer_size; | 974 | u64 next_buffer_size; |
| 828 | }; | 975 | }; |
| 829 | 976 | ||
| 977 | /** | ||
| 978 | * Device -> host job completion message. | ||
| 979 | * @see VPU_JSM_MSG_JOB_DONE | ||
| 980 | */ | ||
| 830 | struct vpu_ipc_msg_payload_job_done { | 981 | struct vpu_ipc_msg_payload_job_done { |
| 831 | /* Engine to which the job was submitted. */ | 982 | /** Engine to which the job was submitted. */ |
| 832 | u32 engine_idx; | 983 | u32 engine_idx; |
| 833 | /* Index of the doorbell to which the job was submitted */ | 984 | /** Index of the doorbell to which the job was submitted */ |
| 834 | u32 db_idx; | 985 | u32 db_idx; |
| 835 | /* ID of the completed job */ | 986 | /** ID of the completed job */ |
| 836 | u32 job_id; | 987 | u32 job_id; |
| 837 | /* Status of the completed job */ | 988 | /** Status of the completed job */ |
| 838 | u32 job_status; | 989 | u32 job_status; |
| 839 | /* Host SSID */ | 990 | /** Host SSID */ |
| 840 | u32 host_ssid; | 991 | u32 host_ssid; |
| 841 | /* Zero Padding */ | 992 | /** Zero Padding */ |
| 842 | u32 reserved_0; | 993 | u32 reserved_0; |
| 843 | /* Command queue id */ | 994 | /** Command queue id */ |
| 844 | u64 cmdq_id; | 995 | u64 cmdq_id; |
| 845 | }; | 996 | }; |
| 846 | 997 | ||
| 847 | /* | 998 | /** |
| 848 | * Notification message upon native fence signalling. | 999 | * Notification message upon native fence signalling. |
| 849 | * @see VPU_JSM_MSG_NATIVE_FENCE_SIGNALLED | 1000 | * @see VPU_JSM_MSG_NATIVE_FENCE_SIGNALLED |
| 850 | */ | 1001 | */ |
| 851 | struct vpu_ipc_msg_payload_native_fence_signalled { | 1002 | struct vpu_ipc_msg_payload_native_fence_signalled { |
| 852 | /* Engine ID. */ | 1003 | /** Engine ID. */ |
| 853 | u32 engine_idx; | 1004 | u32 engine_idx; |
| 854 | /* Host SSID. */ | 1005 | /** Host SSID. */ |
| 855 | u32 host_ssid; | 1006 | u32 host_ssid; |
| 856 | /* CMDQ ID */ | 1007 | /** CMDQ ID */ |
| 857 | u64 cmdq_id; | 1008 | u64 cmdq_id; |
| 858 | /* Fence object handle. */ | 1009 | /** Fence object handle. */ |
| 859 | u64 fence_handle; | 1010 | u64 fence_handle; |
| 860 | }; | 1011 | }; |
| 861 | 1012 | ||
| 1013 | /** | ||
| 1014 | * vpu_ipc_msg_payload_engine_reset_done will contain an array of this structure | ||
| 1015 | * which contains which queues caused reset if FW was able to detect any error. | ||
| 1016 | * @see vpu_ipc_msg_payload_engine_reset_done | ||
| 1017 | */ | ||
| 862 | struct vpu_jsm_engine_reset_context { | 1018 | struct vpu_jsm_engine_reset_context { |
| 863 | /* Host SSID */ | 1019 | /** Host SSID */ |
| 864 | u32 host_ssid; | 1020 | u32 host_ssid; |
| 865 | /* Zero Padding */ | 1021 | /** Zero Padding */ |
| 866 | u32 reserved_0; | 1022 | u32 reserved_0; |
| 867 | /* Command queue id */ | 1023 | /** Command queue id */ |
| 868 | u64 cmdq_id; | 1024 | u64 cmdq_id; |
| 869 | /* See VPU_ENGINE_RESET_CONTEXT_* defines */ | 1025 | /** See VPU_ENGINE_RESET_CONTEXT_* defines */ |
| 870 | u64 flags; | 1026 | u64 flags; |
| 871 | }; | 1027 | }; |
| 872 | 1028 | ||
| 1029 | /** | ||
| 1030 | * Engine reset response. | ||
| 1031 | * @see VPU_JSM_MSG_ENGINE_RESET_DONE | ||
| 1032 | */ | ||
| 873 | struct vpu_ipc_msg_payload_engine_reset_done { | 1033 | struct vpu_ipc_msg_payload_engine_reset_done { |
| 874 | /* Engine ordinal */ | 1034 | /** Engine ordinal */ |
| 875 | u32 engine_idx; | 1035 | u32 engine_idx; |
| 876 | /* Number of impacted contexts */ | 1036 | /** Number of impacted contexts */ |
| 877 | u32 num_impacted_contexts; | 1037 | u32 num_impacted_contexts; |
| 878 | /* Array of impacted command queue ids and their flags */ | 1038 | /** Array of impacted command queue ids and their flags */ |
| 879 | struct vpu_jsm_engine_reset_context | 1039 | struct vpu_jsm_engine_reset_context |
| 880 | impacted_contexts[VPU_MAX_ENGINE_RESET_IMPACTED_CONTEXTS]; | 1040 | impacted_contexts[VPU_MAX_ENGINE_RESET_IMPACTED_CONTEXTS]; |
| 881 | }; | 1041 | }; |
| 882 | 1042 | ||
| 1043 | /** | ||
| 1044 | * Preemption response struct | ||
| 1045 | * @see VPU_JSM_MSG_ENGINE_PREEMPT_DONE | ||
| 1046 | */ | ||
| 883 | struct vpu_ipc_msg_payload_engine_preempt_done { | 1047 | struct vpu_ipc_msg_payload_engine_preempt_done { |
| 884 | /* Engine preempted. */ | 1048 | /** Engine preempted. */ |
| 885 | u32 engine_idx; | 1049 | u32 engine_idx; |
| 886 | /* ID of the preemption request. */ | 1050 | /** ID of the preemption request. */ |
| 887 | u32 preempt_id; | 1051 | u32 preempt_id; |
| 888 | }; | 1052 | }; |
| 889 | 1053 | ||
| @@ -912,12 +1076,16 @@ struct vpu_ipc_msg_payload_unregister_db_done { · 22 unmodified lines | |||
| 912 | u32 reserved_0; | 1076 | u32 reserved_0; |
| 913 | }; | 1077 | }; |
| 914 | 1078 | ||
| 1079 | /** | ||
| 1080 | * Structure for heartbeat response | ||
| 1081 | * @see VPU_JSM_MSG_QUERY_ENGINE_HB_DONE | ||
| 1082 | */ | ||
| 915 | struct vpu_ipc_msg_payload_query_engine_hb_done { | 1083 | struct vpu_ipc_msg_payload_query_engine_hb_done { |
| 916 | /* Engine returning heartbeat value. */ | 1084 | /** Engine returning heartbeat value. */ |
| 917 | u32 engine_idx; | 1085 | u32 engine_idx; |
| 918 | /* Reserved */ | 1086 | /** Reserved */ |
| 919 | u32 reserved_0; | 1087 | u32 reserved_0; |
| 920 | /* Heartbeat value. */ | 1088 | /** Heartbeat value. */ |
| 921 | u64 heartbeat; | 1089 | u64 heartbeat; |
| 922 | }; | 1090 | }; |
| 923 | 1091 | ||
| @@ -937,7 +1105,10 @@ struct vpu_ipc_msg_payload_get_power_level_count_done { · 13 unmodified lines | |||
| 937 | u8 power_limit[16]; | 1105 | u8 power_limit[16]; |
| 938 | }; | 1106 | }; |
| 939 | 1107 | ||
| 940 | /* HWS priority band setup request / response */ | 1108 | /** |
| 1109 | * HWS priority band setup request / response | ||
| 1110 | * @see VPU_JSM_MSG_SET_PRIORITY_BAND_SETUP | ||
| 1111 | */ | ||
| 941 | struct vpu_ipc_msg_payload_hws_priority_band_setup { | 1112 | struct vpu_ipc_msg_payload_hws_priority_band_setup { |
| 942 | /* | 1113 | /* |
| 943 | * Grace period in 100ns units when preempting another priority band for | 1114 | * Grace period in 100ns units when preempting another priority band for |
| @@ -964,15 +1135,23 @@ struct vpu_ipc_msg_payload_hws_priority_band_setup { · 20 unmodified lines | |||
| 964 | * TDR timeout value in milliseconds. Default value of 0 meaning no timeout. | 1135 | * TDR timeout value in milliseconds. Default value of 0 meaning no timeout. |
| 965 | */ | 1136 | */ |
| 966 | u32 tdr_timeout; | 1137 | u32 tdr_timeout; |
| 1138 | /* Non-interactive queue timeout for no progress of heartbeat in milliseconds. | ||
| 1139 | * Default value of 0 meaning no timeout. | ||
| 1140 | */ | ||
| 1141 | u32 non_interactive_no_progress_timeout; | ||
| 1142 | /* | ||
| 1143 | * Non-interactive queue upper limit timeout value in milliseconds. Default | ||
| 1144 | * value of 0 meaning no timeout. | ||
| 1145 | */ | ||
| 1146 | u32 non_interactive_timeout; | ||
| 967 | }; | 1147 | }; |
| 968 | 1148 | ||
| 969 | /* | 1149 | /** |
| 970 | * @brief HWS create command queue request. | 1150 | * @brief HWS create command queue request. |
| 971 | * Host will create a command queue via this command. | 1151 | * Host will create a command queue via this command. |
| 972 | * Note: Cmdq group is a handle of an object which | 1152 | * Note: Cmdq group is a handle of an object which |
| 973 | * may contain one or more command queues. | 1153 | * may contain one or more command queues. |
| 974 | * @see VPU_JSM_MSG_CREATE_CMD_QUEUE | 1154 | * @see VPU_JSM_MSG_CREATE_CMD_QUEUE |
| 975 | * @see VPU_JSM_MSG_CREATE_CMD_QUEUE_RSP | ||
| 976 | */ | 1155 | */ |
| 977 | struct vpu_ipc_msg_payload_hws_create_cmdq { | 1156 | struct vpu_ipc_msg_payload_hws_create_cmdq { |
| 978 | /* Process id */ | 1157 | /* Process id */ |
| @@ -993,66 +1172,73 @@ struct vpu_ipc_msg_payload_hws_create_cmdq { · 14 unmodified lines | |||
| 993 | u32 reserved_0; | 1172 | u32 reserved_0; |
| 994 | }; | 1173 | }; |
| 995 | 1174 | ||
| 996 | /* | 1175 | /** |
| 997 | * @brief HWS create command queue response. | 1176 | * HWS create command queue response. |
| 998 | * @see VPU_JSM_MSG_CREATE_CMD_QUEUE | ||
| 999 | * @see VPU_JSM_MSG_CREATE_CMD_QUEUE_RSP | 1177 | * @see VPU_JSM_MSG_CREATE_CMD_QUEUE_RSP |
| 1000 | */ | 1178 | */ |
| 1001 | struct vpu_ipc_msg_payload_hws_create_cmdq_rsp { | 1179 | struct vpu_ipc_msg_payload_hws_create_cmdq_rsp { |
| 1002 | /* Process id */ | 1180 | /** Process id */ |
| 1003 | u64 process_id; | 1181 | u64 process_id; |
| 1004 | /* Host SSID */ | 1182 | /** Host SSID */ |
| 1005 | u32 host_ssid; | 1183 | u32 host_ssid; |
| 1006 | /* Engine for which queue is being created */ | 1184 | /** Engine for which queue is being created */ |
| 1007 | u32 engine_idx; | 1185 | u32 engine_idx; |
| 1008 | /* Command queue group */ | 1186 | /** Command queue group */ |
| 1009 | u64 cmdq_group; | 1187 | u64 cmdq_group; |
| 1010 | /* Command queue id */ | 1188 | /** Command queue id */ |
| 1011 | u64 cmdq_id; | 1189 | u64 cmdq_id; |
| 1012 | }; | 1190 | }; |
| 1013 | 1191 | ||
| 1014 | /* HWS destroy command queue request / response */ | 1192 | /** |
| 1193 | * HWS destroy command queue request / response | ||
| 1194 | * @see VPU_JSM_MSG_DESTROY_CMD_QUEUE | ||
| 1195 | * @see VPU_JSM_MSG_DESTROY_CMD_QUEUE_RSP | ||
| 1196 | */ | ||
| 1015 | struct vpu_ipc_msg_payload_hws_destroy_cmdq { | 1197 | struct vpu_ipc_msg_payload_hws_destroy_cmdq { |
| 1016 | /* Host SSID */ | 1198 | /** Host SSID */ |
| 1017 | u32 host_ssid; | 1199 | u32 host_ssid; |
| 1018 | /* Zero Padding */ | 1200 | /** Zero Padding */ |
| 1019 | u32 reserved; | 1201 | u32 reserved; |
| 1020 | /* Command queue id */ | 1202 | /** Command queue id */ |
| 1021 | u64 cmdq_id; | 1203 | u64 cmdq_id; |
| 1022 | }; | 1204 | }; |
| 1023 | 1205 | ||
| 1024 | /* HWS set context scheduling properties request / response */ | 1206 | /** |
| 1207 | * HWS set context scheduling properties request / response | ||
| 1208 | * @see VPU_JSM_MSG_SET_CONTEXT_SCHED_PROPERTIES | ||
| 1209 | * @see VPU_JSM_MSG_SET_CONTEXT_SCHED_PROPERTIES_RSP | ||
| 1210 | */ | ||
| 1025 | struct vpu_ipc_msg_payload_hws_set_context_sched_properties { | 1211 | struct vpu_ipc_msg_payload_hws_set_context_sched_properties { |
| 1026 | /* Host SSID */ | 1212 | /** Host SSID */ |
| 1027 | u32 host_ssid; | 1213 | u32 host_ssid; |
| 1028 | /* Zero Padding */ | 1214 | /** Zero Padding */ |
| 1029 | u32 reserved_0; | 1215 | u32 reserved_0; |
| 1030 | /* Command queue id */ | 1216 | /** Command queue id */ |
| 1031 | u64 cmdq_id; | 1217 | u64 cmdq_id; |
| 1032 | /* | 1218 | /** |
| 1033 | * Priority band to assign to work of this context. | 1219 | * Priority band to assign to work of this context. |
| 1034 | * Available priority bands: @see enum vpu_job_scheduling_priority_band | 1220 | * Available priority bands: @see enum vpu_job_scheduling_priority_band |
| 1035 | */ | 1221 | */ |
| 1036 | u32 priority_band; | 1222 | u32 priority_band; |
| 1037 | /* Inside realtime band assigns a further priority */ | 1223 | /** Inside realtime band assigns a further priority */ |
| 1038 | u32 realtime_priority_level; | 1224 | u32 realtime_priority_level; |
| 1039 | /* Priority relative to other contexts in the same process */ | 1225 | /** Priority relative to other contexts in the same process */ |
| 1040 | s32 in_process_priority; | 1226 | s32 in_process_priority; |
| 1041 | /* Zero padding / Reserved */ | 1227 | /** Zero padding / Reserved */ |
| 1042 | u32 reserved_1; | 1228 | u32 reserved_1; |
| 1043 | /* | 1229 | /** |
| 1044 | * Context quantum relative to other contexts of same priority in the same process | 1230 | * Context quantum relative to other contexts of same priority in the same process |
| 1045 | * Minimum value supported by NPU is 1ms (10000 in 100ns units). | 1231 | * Minimum value supported by NPU is 1ms (10000 in 100ns units). |
| 1046 | */ | 1232 | */ |
| 1047 | u64 context_quantum; | 1233 | u64 context_quantum; |
| 1048 | /* Grace period when preempting context of the same priority within the same process */ | 1234 | /** Grace period when preempting context of the same priority within the same process */ |
| 1049 | u64 grace_period_same_priority; | 1235 | u64 grace_period_same_priority; |
| 1050 | /* Grace period when preempting context of a lower priority within the same process */ | 1236 | /** Grace period when preempting context of a lower priority within the same process */ |
| 1051 | u64 grace_period_lower_priority; | 1237 | u64 grace_period_lower_priority; |
| 1052 | }; | 1238 | }; |
| 1053 | 1239 | ||
| 1054 | /* | 1240 | /** |
| 1055 | * @brief Register doorbell command structure. | 1241 | * Register doorbell command structure. |
| 1056 | * This structure supports doorbell registration for both HW and OS scheduling. | 1242 | * This structure supports doorbell registration for both HW and OS scheduling. |
| 1057 | * Note: Queue base and size are added here so that the same structure can be used for | 1243 | * Note: Queue base and size are added here so that the same structure can be used for |
| 1058 | * OS scheduling and HW scheduling. For OS scheduling, cmdq_id will be ignored | 1244 | * OS scheduling and HW scheduling. For OS scheduling, cmdq_id will be ignored |
| @@ -1061,27 +1247,27 @@ struct vpu_ipc_msg_payload_hws_set_context_sched_properties { · 2 unmodified lines | |||
| 1061 | * @see VPU_JSM_MSG_HWS_REGISTER_DB | 1247 | * @see VPU_JSM_MSG_HWS_REGISTER_DB |
| 1062 | */ | 1248 | */ |
| 1063 | struct vpu_jsm_hws_register_db { | 1249 | struct vpu_jsm_hws_register_db { |
| 1064 | /* Index of the doorbell to register. */ | 1250 | /** Index of the doorbell to register. */ |
| 1065 | u32 db_id; | 1251 | u32 db_id; |
| 1066 | /* Host sub-stream ID for the context assigned to the doorbell. */ | 1252 | /** Host sub-stream ID for the context assigned to the doorbell. */ |
| 1067 | u32 host_ssid; | 1253 | u32 host_ssid; |
| 1068 | /* ID of the command queue associated with the doorbell. */ | 1254 | /** ID of the command queue associated with the doorbell. */ |
| 1069 | u64 cmdq_id; | 1255 | u64 cmdq_id; |
| 1070 | /* Virtual address pointing to the start of command queue. */ | 1256 | /** Virtual address pointing to the start of command queue. */ |
| 1071 | u64 cmdq_base; | 1257 | u64 cmdq_base; |
| 1072 | /* Size of the command queue in bytes. */ | 1258 | /** Size of the command queue in bytes. */ |
| 1073 | u64 cmdq_size; | 1259 | u64 cmdq_size; |
| 1074 | }; | 1260 | }; |
| 1075 | 1261 | ||
| 1076 | /* | 1262 | /** |
| 1077 | * @brief Structure to set another buffer to be used for scheduling-related logging. | 1263 | * Structure to set another buffer to be used for scheduling-related logging. |
| 1078 | * The size of the logging buffer and the number of entries is defined as part of the | 1264 | * The size of the logging buffer and the number of entries is defined as part of the |
| 1079 | * buffer itself as described next. | 1265 | * buffer itself as described next. |
| 1080 | * The log buffer received from the host is made up of; | 1266 | * The log buffer received from the host is made up of; |
| 1081 | * - header: 32 bytes in size, as shown in 'struct vpu_hws_log_buffer_header'. | 1267 | * - header: 32 bytes in size, as shown in @ref vpu_hws_log_buffer_header. |
| 1082 | * The header contains the number of log entries in the buffer. | 1268 | * The header contains the number of log entries in the buffer. |
| 1083 | * - log entry: 0 to n-1, each log entry is 32 bytes in size, as shown in | 1269 | * - log entry: 0 to n-1, each log entry is 32 bytes in size, as shown in |
| 1084 | * 'struct vpu_hws_log_buffer_entry'. | 1270 | * @ref vpu_hws_log_buffer_entry. |
| 1085 | * The entry contains the VPU timestamp, operation type and data. | 1271 | * The entry contains the VPU timestamp, operation type and data. |
| 1086 | * The host should provide the notify index value of log buffer to VPU. This is a | 1272 | * The host should provide the notify index value of log buffer to VPU. This is a |
| 1087 | * value defined within the log buffer and when written to will generate the | 1273 | * value defined within the log buffer and when written to will generate the |
| @@ -1095,30 +1281,30 @@ struct vpu_jsm_hws_register_db { · 7 unmodified lines | |||
| 1095 | * @see VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION | 1281 | * @see VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION |
| 1096 | */ | 1282 | */ |
| 1097 | struct vpu_ipc_msg_payload_hws_set_scheduling_log { | 1283 | struct vpu_ipc_msg_payload_hws_set_scheduling_log { |
| 1098 | /* Engine ordinal */ | 1284 | /** Engine ordinal */ |
| 1099 | u32 engine_idx; | 1285 | u32 engine_idx; |
| 1100 | /* Host SSID */ | 1286 | /** Host SSID */ |
| 1101 | u32 host_ssid; | 1287 | u32 host_ssid; |
| 1102 | /* | 1288 | /** |
| 1103 | * VPU log buffer virtual address. | 1289 | * VPU log buffer virtual address. |
| 1104 | * Set to 0 to disable logging for this engine. | 1290 | * Set to 0 to disable logging for this engine. |
| 1105 | */ | 1291 | */ |
| 1106 | u64 vpu_log_buffer_va; | 1292 | u64 vpu_log_buffer_va; |
| 1107 | /* | 1293 | /** |
| 1108 | * Notify index of log buffer. VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION | 1294 | * Notify index of log buffer. VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION |
| 1109 | * is generated when an event log is written to this index. | 1295 | * is generated when an event log is written to this index. |
| 1110 | */ | 1296 | */ |
| 1111 | u64 notify_index; | 1297 | u64 notify_index; |
| 1112 | /* | 1298 | /** |
| 1113 | * Field is now deprecated, will be removed when KMD is updated to support removal | 1299 | * Field is now deprecated, will be removed when KMD is updated to support removal |
| 1114 | */ | 1300 | */ |
| 1115 | u32 enable_extra_events; | 1301 | u32 enable_extra_events; |
| 1116 | /* Zero Padding */ | 1302 | /** Zero Padding */ |
| 1117 | u32 reserved_0; | 1303 | u32 reserved_0; |
| 1118 | }; | 1304 | }; |
| 1119 | 1305 | ||
| 1120 | /* | 1306 | /** |
| 1121 | * @brief The scheduling log notification is generated by VPU when it writes | 1307 | * The scheduling log notification is generated by VPU when it writes |
| 1122 | * an event into the log buffer at the notify_index. VPU notifies host with | 1308 | * an event into the log buffer at the notify_index. VPU notifies host with |
| 1123 | * VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION. This is an asynchronous | 1309 | * VPU_JSM_MSG_HWS_SCHEDULING_LOG_NOTIFICATION. This is an asynchronous |
| 1124 | * message from VPU to host. | 1310 | * message from VPU to host. |
| @@ -1126,14 +1312,14 @@ struct vpu_ipc_msg_payload_hws_set_scheduling_log { · 1 unmodified line | |||
| 1126 | * @see VPU_JSM_MSG_HWS_SET_SCHEDULING_LOG | 1312 | * @see VPU_JSM_MSG_HWS_SET_SCHEDULING_LOG |
| 1127 | */ | 1313 | */ |
| 1128 | struct vpu_ipc_msg_payload_hws_scheduling_log_notification { | 1314 | struct vpu_ipc_msg_payload_hws_scheduling_log_notification { |
| 1129 | /* Engine ordinal */ | 1315 | /** Engine ordinal */ |
| 1130 | u32 engine_idx; | 1316 | u32 engine_idx; |
| 1131 | /* Zero Padding */ | 1317 | /** Zero Padding */ |
| 1132 | u32 reserved_0; | 1318 | u32 reserved_0; |
| 1133 | }; | 1319 | }; |
| 1134 | 1320 | ||
| 1135 | /* | 1321 | /** |
| 1136 | * @brief HWS suspend command queue request and done structure. | 1322 | * HWS suspend command queue request and done structure. |
| 1137 | * Host will request the suspend of contexts and VPU will; | 1323 | * Host will request the suspend of contexts and VPU will; |
| 1138 | * - Suspend all work on this context | 1324 | * - Suspend all work on this context |
| 1139 | * - Preempt any running work | 1325 | * - Preempt any running work |
| @@ -1152,21 +1338,21 @@ struct vpu_ipc_msg_payload_hws_scheduling_log_notification { · 12 unmodified lines | |||
| 1152 | * @see VPU_JSM_MSG_HWS_SUSPEND_CMDQ_DONE | 1338 | * @see VPU_JSM_MSG_HWS_SUSPEND_CMDQ_DONE |
| 1153 | */ | 1339 | */ |
| 1154 | struct vpu_ipc_msg_payload_hws_suspend_cmdq { | 1340 | struct vpu_ipc_msg_payload_hws_suspend_cmdq { |
| 1155 | /* Host SSID */ | 1341 | /** Host SSID */ |
| 1156 | u32 host_ssid; | 1342 | u32 host_ssid; |
| 1157 | /* Zero Padding */ | 1343 | /** Zero Padding */ |
| 1158 | u32 reserved_0; | 1344 | u32 reserved_0; |
| 1159 | /* Command queue id */ | 1345 | /** Command queue id */ |
| 1160 | u64 cmdq_id; | 1346 | u64 cmdq_id; |
| 1161 | /* | 1347 | /** |
| 1162 | * Suspend fence value - reported by the VPU suspend context | 1348 | * Suspend fence value - reported by the VPU suspend context |
| 1163 | * completed once suspend is complete. | 1349 | * completed once suspend is complete. |
| 1164 | */ | 1350 | */ |
| 1165 | u64 suspend_fence_value; | 1351 | u64 suspend_fence_value; |
| 1166 | }; | 1352 | }; |
| 1167 | 1353 | ||
| 1168 | /* | 1354 | /** |
| 1169 | * @brief HWS Resume command queue request / response structure. | 1355 | * HWS Resume command queue request / response structure. |
| 1170 | * Host will request the resume of a context; | 1356 | * Host will request the resume of a context; |
| 1171 | * - VPU will resume all work on this context | 1357 | * - VPU will resume all work on this context |
| 1172 | * - Scheduler will allow this context to be scheduled | 1358 | * - Scheduler will allow this context to be scheduled |
| @@ -1174,25 +1360,25 @@ struct vpu_ipc_msg_payload_hws_suspend_cmdq { · 1 unmodified line | |||
| 1174 | * @see VPU_JSM_MSG_HWS_RESUME_CMDQ_RSP | 1360 | * @see VPU_JSM_MSG_HWS_RESUME_CMDQ_RSP |
| 1175 | */ | 1361 | */ |
| 1176 | struct vpu_ipc_msg_payload_hws_resume_cmdq { | 1362 | struct vpu_ipc_msg_payload_hws_resume_cmdq { |
| 1177 | /* Host SSID */ | 1363 | /** Host SSID */ |
| 1178 | u32 host_ssid; | 1364 | u32 host_ssid; |
| 1179 | /* Zero Padding */ | 1365 | /** Zero Padding */ |
| 1180 | u32 reserved_0; | 1366 | u32 reserved_0; |
| 1181 | /* Command queue id */ | 1367 | /** Command queue id */ |
| 1182 | u64 cmdq_id; | 1368 | u64 cmdq_id; |
| 1183 | }; | 1369 | }; |
| 1184 | 1370 | ||
| 1185 | /* | 1371 | /** |
| 1186 | * @brief HWS Resume engine request / response structure. | 1372 | * HWS Resume engine request / response structure. |
| 1187 | * After a HWS engine reset, all scheduling is stopped on VPU until a engine resume. | 1373 | * After a HWS engine reset, all scheduling is stopped on VPU until an engine resume. |
| 1188 | * Host shall send this command to resume scheduling of any valid queue. | 1374 | * Host shall send this command to resume scheduling of any valid queue. |
| 1189 | * @see VPU_JSM_MSG_HWS_RESUME_ENGINE | 1375 | * @see VPU_JSM_MSG_HWS_ENGINE_RESUME |
| 1190 | * @see VPU_JSM_MSG_HWS_RESUME_ENGINE_DONE | 1376 | * @see VPU_JSM_MSG_HWS_RESUME_ENGINE_DONE |
| 1191 | */ | 1377 | */ |
| 1192 | struct vpu_ipc_msg_payload_hws_resume_engine { | 1378 | struct vpu_ipc_msg_payload_hws_resume_engine { |
| 1193 | /* Engine to be resumed */ | 1379 | /** Engine to be resumed */ |
| 1194 | u32 engine_idx; | 1380 | u32 engine_idx; |
| 1195 | /* Reserved */ | 1381 | /** Reserved */ |
| 1196 | u32 reserved_0; | 1382 | u32 reserved_0; |
| 1197 | }; | 1383 | }; |
| 1198 | 1384 | ||
| @@ -1326,7 +1512,7 @@ struct vpu_jsm_metric_streamer_done { · 127 unmodified lines | |||
| 1326 | /** | 1512 | /** |
| 1327 | * Metric group description placed in the metric buffer after successful completion | 1513 | * Metric group description placed in the metric buffer after successful completion |
| 1328 | * of the VPU_JSM_MSG_METRIC_STREAMER_INFO command. This is followed by one or more | 1514 | * of the VPU_JSM_MSG_METRIC_STREAMER_INFO command. This is followed by one or more |
| 1329 | * @vpu_jsm_metric_counter_descriptor records. | 1515 | * @ref vpu_jsm_metric_counter_descriptor records. |
| 1330 | * @see VPU_JSM_MSG_METRIC_STREAMER_INFO | 1516 | * @see VPU_JSM_MSG_METRIC_STREAMER_INFO |
| 1331 | */ | 1517 | */ |
| 1332 | struct vpu_jsm_metric_group_descriptor { | 1518 | struct vpu_jsm_metric_group_descriptor { |
| @@ -1413,29 +1599,24 @@ struct vpu_jsm_metric_counter_descriptor { · 80 unmodified lines | |||
| 1413 | }; | 1599 | }; |
| 1414 | 1600 | ||
| 1415 | /** | 1601 | /** |
| 1416 | * Payload for VPU_JSM_MSG_DYNDBG_CONTROL requests. | 1602 | * Payload for @ref VPU_JSM_MSG_DYNDBG_CONTROL requests. |
| 1417 | * | 1603 | * |
| 1418 | * VPU_JSM_MSG_DYNDBG_CONTROL are used to control the VPU FW Dynamic Debug | 1604 | * VPU_JSM_MSG_DYNDBG_CONTROL requests are used to control the VPU FW dynamic debug |
| 1419 | * feature, which allows developers to selectively enable / disable MVLOG_DEBUG | 1605 | * feature, which allows developers to selectively enable/disable code to obtain |
| 1420 | * messages. This is equivalent to the Dynamic Debug functionality provided by | 1606 | * additional FW information. This is equivalent to the dynamic debug functionality |
| 1421 | * Linux | 1607 | * provided by Linux. The host can control dynamic debug behavior by sending dyndbg |
| 1422 | * (https://www.kernel.org/doc/html/latest/admin-guide/dynamic-debug-howto.html) | 1608 | * commands, using the same syntax as for Linux dynamic debug commands. |
| 1423 | * The host can control Dynamic Debug behavior by sending dyndbg commands, which | ||
| 1424 | * have the same syntax as Linux | ||
| 1425 | * dyndbg commands. | ||
| 1426 | * | 1609 | * |
| 1427 | * NOTE: in order for MVLOG_DEBUG messages to be actually printed, the host | 1610 | * @see https://www.kernel.org/doc/html/latest/admin-guide/dynamic-debug-howto.html. |
| 1428 | * still has to set the logging level to MVLOG_DEBUG, using the | ||
| 1429 | * VPU_JSM_MSG_TRACE_SET_CONFIG command. | ||
| 1430 | * | 1611 | * |
| 1431 | * The host can see the current dynamic debug configuration by executing a | 1612 | * NOTE: |
| 1432 | * special 'show' command. The dyndbg configuration will be printed to the | 1613 | * As the dynamic debug feature uses MVLOG messages to provide information, the host |
| 1433 | * configured logging destination using MVLOG_INFO logging level. | 1614 | * must first set the logging level to MVLOG_DEBUG, using the @ref VPU_JSM_MSG_TRACE_SET_CONFIG |
| 1615 | * command. | ||
| 1434 | */ | 1616 | */ |
| 1435 | struct vpu_ipc_msg_payload_dyndbg_control { | 1617 | struct vpu_ipc_msg_payload_dyndbg_control { |
| 1436 | /** | 1618 | /** |
| 1437 | * Dyndbg command (same format as Linux dyndbg); must be a NULL-terminated | 1619 | * Dyndbg command to be executed. |
| 1438 | * string. | ||
| 1439 | */ | 1620 | */ |
| 1440 | char dyndbg_cmd[VPU_DYNDBG_CMD_MAX_LEN]; | 1621 | char dyndbg_cmd[VPU_DYNDBG_CMD_MAX_LEN]; |
| 1441 | }; | 1622 | }; |
| @@ -1456,7 +1637,7 @@ struct vpu_ipc_msg_payload_pwr_d0i3_enter { · 14 unmodified lines | |||
| 1456 | }; | 1637 | }; |
| 1457 | 1638 | ||
| 1458 | /** | 1639 | /** |
| 1459 | * Payload for VPU_JSM_MSG_DCT_ENABLE message. | 1640 | * Payload for @ref VPU_JSM_MSG_DCT_ENABLE message. |
| 1460 | * | 1641 | * |
| 1461 | * Default values for DCT active/inactive times are 5.3ms and 30ms respectively, | 1642 | * Default values for DCT active/inactive times are 5.3ms and 30ms respectively, |
| 1462 | * corresponding to a 85% duty cycle. This payload allows the host to tune these | 1643 | * corresponding to a 85% duty cycle. This payload allows the host to tune these |
| @@ -1513,28 +1694,28 @@ union vpu_ipc_msg_payload { · 50 unmodified lines | |||
| 1513 | struct vpu_ipc_msg_payload_pwr_dct_control pwr_dct_control; | 1694 | struct vpu_ipc_msg_payload_pwr_dct_control pwr_dct_control; |
| 1514 | }; | 1695 | }; |
| 1515 | 1696 | ||
| 1516 | /* | 1697 | /** |
| 1517 | * Host <-> LRT IPC message base structure. | 1698 | * Host <-> NPU IPC message base structure. |
| 1518 | * | 1699 | * |
| 1519 | * NOTE: All instances of this object must be aligned on a 64B boundary | 1700 | * NOTE: All instances of this object must be aligned on a 64B boundary |
| 1520 | * to allow proper handling of VPU cache operations. | 1701 | * to allow proper handling of VPU cache operations. |
| 1521 | */ | 1702 | */ |
| 1522 | struct vpu_jsm_msg { | 1703 | struct vpu_jsm_msg { |
| 1523 | /* Reserved */ | 1704 | /** Reserved */ |
| 1524 | u64 reserved_0; | 1705 | u64 reserved_0; |
| 1525 | /* Message type, see vpu_ipc_msg_type enum. */ | 1706 | /** Message type, see @ref vpu_ipc_msg_type. */ |
| 1526 | u32 type; | 1707 | u32 type; |
| 1527 | /* Buffer status, see vpu_ipc_msg_status enum. */ | 1708 | /** Buffer status, see @ref vpu_ipc_msg_status. */ |
| 1528 | u32 status; | 1709 | u32 status; |
| 1529 | /* | 1710 | /** |
| 1530 | * Request ID, provided by the host in a request message and passed | 1711 | * Request ID, provided by the host in a request message and passed |
| 1531 | * back by VPU in the response message. | 1712 | * back by VPU in the response message. |
| 1532 | */ | 1713 | */ |
| 1533 | u32 request_id; | 1714 | u32 request_id; |
| 1534 | /* Request return code set by the VPU, see VPU_JSM_STATUS_* defines. */ | 1715 | /** Request return code set by the VPU, see VPU_JSM_STATUS_* defines. */ |
| 1535 | u32 result; | 1716 | u32 result; |
| 1536 | u64 reserved_1; | 1717 | u64 reserved_1; |
| 1537 | /* Message payload depending on message type, see vpu_ipc_msg_payload union. */ | 1718 | /** Message payload depending on message type, see vpu_ipc_msg_payload union. */ |
| 1538 | union vpu_ipc_msg_payload payload; | 1719 | union vpu_ipc_msg_payload payload; |
| 1539 | }; | 1720 | }; |
| 1540 | 1721 | ||
| @@ -9,6 +9,7 @@ config DRM_ACCEL_QAIC · 8 unmodified lines | |||
| 9 | depends on PCI && HAS_IOMEM | 9 | depends on PCI && HAS_IOMEM |
| 10 | depends on MHI_BUS | 10 | depends on MHI_BUS |
| 11 | select CRC32 | 11 | select CRC32 |
| 12 | select WANT_DEV_COREDUMP | ||
| 12 | help | 13 | help |
| 13 | Enables driver for Qualcomm's Cloud AI accelerator PCIe cards that are | 14 | Enables driver for Qualcomm's Cloud AI accelerator PCIe cards that are |
| 14 | designed to accelerate Deep Learning inference workloads. | 15 | designed to accelerate Deep Learning inference workloads. |
| @@ -11,6 +11,8 @@ qaic-y := \ · 10 unmodified lines | |||
| 11 | qaic_data.o \ | 11 | qaic_data.o \ |
| 12 | qaic_drv.o \ | 12 | qaic_drv.o \ |
| 13 | qaic_ras.o \ | 13 | qaic_ras.o \ |
| 14 | qaic_ssr.o \ | ||
| 15 | qaic_sysfs.o \ | ||
| 14 | qaic_timesync.o \ | 16 | qaic_timesync.o \ |
| 15 | sahara.o | 17 | sahara.o |
| 16 | 18 | ||
| @@ -21,6 +21,7 @@ · 20 unmodified lines | |||
| 21 | 21 | ||
| 22 | #define QAIC_DBC_BASE SZ_128K | 22 | #define QAIC_DBC_BASE SZ_128K |
| 23 | #define QAIC_DBC_SIZE SZ_4K | 23 | #define QAIC_DBC_SIZE SZ_4K |
| 24 | #define QAIC_SSR_DBC_SENTINEL U32_MAX /* No ongoing SSR sentinel */ | ||
| 24 | 25 | ||
| 25 | #define QAIC_NO_PARTITION -1 | 26 | #define QAIC_NO_PARTITION -1 |
| 26 | 27 | ||
| @@ -47,6 +48,22 @@ enum __packed dev_states { · 20 unmodified lines | |||
| 47 | QAIC_ONLINE, | 48 | QAIC_ONLINE, |
| 48 | }; | 49 | }; |
| 49 | 50 | ||
| 51 | enum dbc_states { | ||
| 52 | /* DBC is free and can be activated */ | ||
| 53 | DBC_STATE_IDLE, | ||
| 54 | /* DBC is activated and a workload is running on device */ | ||
| 55 | DBC_STATE_ASSIGNED, | ||
| 56 | /* Sub-system associated with this workload has crashed and it will shutdown soon */ | ||
| 57 | DBC_STATE_BEFORE_SHUTDOWN, | ||
| 58 | /* Sub-system associated with this workload has crashed and it has shutdown */ | ||
| 59 | DBC_STATE_AFTER_SHUTDOWN, | ||
| 60 | /* Sub-system associated with this workload is shutdown and it will be powered up soon */ | ||
| 61 | DBC_STATE_BEFORE_POWER_UP, | ||
| 62 | /* Sub-system associated with this workload is now powered up */ | ||
| 63 | DBC_STATE_AFTER_POWER_UP, | ||
| 64 | DBC_STATE_MAX, | ||
| 65 | }; | ||
| 66 | |||
| 50 | extern bool datapath_polling; | 67 | extern bool datapath_polling; |
| 51 | 68 | ||
| 52 | struct qaic_user { | 69 | struct qaic_user { |
| @@ -114,6 +131,8 @@ struct dma_bridge_chan { · 61 unmodified lines | |||
| 114 | unsigned int irq; | 131 | unsigned int irq; |
| 115 | /* Polling work item to simulate interrupts */ | 132 | /* Polling work item to simulate interrupts */ |
| 116 | struct work_struct poll_work; | 133 | struct work_struct poll_work; |
| 134 | /* Represents various states of this DBC from enum dbc_states */ | ||
| 135 | unsigned int state; | ||
| 117 | }; | 136 | }; |
| 118 | 137 | ||
| 119 | struct qaic_device { | 138 | struct qaic_device { |
| @@ -161,6 +180,8 @@ struct qaic_device { · 41 unmodified lines | |||
| 161 | struct mhi_device *qts_ch; | 180 | struct mhi_device *qts_ch; |
| 162 | /* Work queue for tasks related to MHI "QAIC_TIMESYNC" channel */ | 181 | /* Work queue for tasks related to MHI "QAIC_TIMESYNC" channel */ |
| 163 | struct workqueue_struct *qts_wq; | 182 | struct workqueue_struct *qts_wq; |
| 183 | /* MHI "QAIC_TIMESYNC_PERIODIC" channel device */ | ||
| 184 | struct mhi_device *mqts_ch; | ||
| 164 | /* Head of list of page allocated by MHI bootlog device */ | 185 | /* Head of list of page allocated by MHI bootlog device */ |
| 165 | struct list_head bootlog; | 186 | struct list_head bootlog; |
| 166 | /* MHI bootlog channel device */ | 187 | /* MHI bootlog channel device */ |
| @@ -177,6 +198,14 @@ struct qaic_device { · 10 unmodified lines | |||
| 177 | unsigned int ue_count; | 198 | unsigned int ue_count; |
| 178 | /* Un-correctable non-fatal error count */ | 199 | /* Un-correctable non-fatal error count */ |
| 179 | unsigned int ue_nf_count; | 200 | unsigned int ue_nf_count; |
| 201 | /* MHI SSR channel device */ | ||
| 202 | struct mhi_device *ssr_ch; | ||
| 203 | /* Work queue for tasks related to MHI SSR device */ | ||
| 204 | struct workqueue_struct *ssr_wq; | ||
| 205 | /* Buffer to collect SSR crashdump via SSR MHI channel */ | ||
| 206 | void *ssr_mhi_buf; | ||
| 207 | /* DBC which is under SSR. Sentinel U32_MAX would mean that no SSR in progress */ | ||
| 208 | u32 ssr_dbc; | ||
| 180 | }; | 209 | }; |
| 181 | 210 | ||
| 182 | struct qaic_drm_device { | 211 | struct qaic_drm_device { |
| @@ -195,6 +224,8 @@ struct qaic_drm_device { · 12 unmodified lines | |||
| 195 | struct list_head users; | 224 | struct list_head users; |
| 196 | /* Synchronizes access to users list */ | 225 | /* Synchronizes access to users list */ |
| 197 | struct mutex users_mutex; | 226 | struct mutex users_mutex; |
| 227 | /* Pointer to array of DBC sysfs attributes */ | ||
| 228 | void *sysfs_attrs; | ||
| 198 | }; | 229 | }; |
| 199 | 230 | ||
| 200 | struct qaic_bo { | 231 | struct qaic_bo { |
| @@ -317,6 +348,13 @@ int qaic_partial_execute_bo_ioctl(struct drm_device *dev, void *data, struct drm · 116 unmodified lines | |||
| 317 | int qaic_wait_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv); | 348 | int qaic_wait_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv); |
| 318 | int qaic_perf_stats_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv); | 349 | int qaic_perf_stats_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv); |
| 319 | int qaic_detach_slice_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv); | 350 | int qaic_detach_slice_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file_priv); |
| 320 | void irq_polling_work(struct work_struct *work); | 351 | void qaic_irq_polling_work(struct work_struct *work); |
| 352 | void qaic_dbc_enter_ssr(struct qaic_device *qdev, u32 dbc_id); | ||
| 353 | void qaic_dbc_exit_ssr(struct qaic_device *qdev); | ||
| 354 | |||
| 355 | /* qaic_sysfs.c */ | ||
| 356 | int qaic_sysfs_init(struct qaic_drm_device *qddev); | ||
| 357 | void qaic_sysfs_remove(struct qaic_drm_device *qddev); | ||
| 358 | void set_dbc_state(struct qaic_device *qdev, u32 dbc_id, unsigned int state); | ||
| 321 | 359 | ||
| 322 | #endif /* _QAIC_H_ */ | 360 | #endif /* _QAIC_H_ */ |
| @@ -17,6 +17,7 @@ · 16 unmodified lines | |||
| 17 | #include <linux/overflow.h> | 17 | #include <linux/overflow.h> |
| 18 | #include <linux/pci.h> | 18 | #include <linux/pci.h> |
| 19 | #include <linux/scatterlist.h> | 19 | #include <linux/scatterlist.h> |
| 20 | #include <linux/sched/signal.h> | ||
| 20 | #include <linux/types.h> | 21 | #include <linux/types.h> |
| 21 | #include <linux/uaccess.h> | 22 | #include <linux/uaccess.h> |
| 22 | #include <linux/workqueue.h> | 23 | #include <linux/workqueue.h> |
| @@ -30,7 +31,7 @@ · 7 unmodified lines | |||
| 30 | #define MANAGE_MAGIC_NUMBER ((__force __le32)0x43494151) /* "QAIC" in little endian */ | 31 | #define MANAGE_MAGIC_NUMBER ((__force __le32)0x43494151) /* "QAIC" in little endian */ |
| 31 | #define QAIC_DBC_Q_GAP SZ_256 | 32 | #define QAIC_DBC_Q_GAP SZ_256 |
| 32 | #define QAIC_DBC_Q_BUF_ALIGN SZ_4K | 33 | #define QAIC_DBC_Q_BUF_ALIGN SZ_4K |
| 33 | #define QAIC_MANAGE_EXT_MSG_LENGTH SZ_64K /* Max DMA message length */ | 34 | #define QAIC_MANAGE_WIRE_MSG_LENGTH SZ_64K /* Max DMA message length */ |
| 34 | #define QAIC_WRAPPER_MAX_SIZE SZ_4K | 35 | #define QAIC_WRAPPER_MAX_SIZE SZ_4K |
| 35 | #define QAIC_MHI_RETRY_WAIT_MS 100 | 36 | #define QAIC_MHI_RETRY_WAIT_MS 100 |
| 36 | #define QAIC_MHI_RETRY_MAX 20 | 37 | #define QAIC_MHI_RETRY_MAX 20 |
| @@ -309,6 +310,7 @@ static void save_dbc_buf(struct qaic_device *qdev, struct ioctl_resources *resou · 272 unmodified lines | |||
| 309 | enable_dbc(qdev, dbc_id, usr); | 310 | enable_dbc(qdev, dbc_id, usr); |
| 310 | qdev->dbc[dbc_id].in_use = true; | 311 | qdev->dbc[dbc_id].in_use = true; |
| 311 | resources->buf = NULL; | 312 | resources->buf = NULL; |
| 313 | set_dbc_state(qdev, dbc_id, DBC_STATE_ASSIGNED); | ||
| 312 | } | 314 | } |
| 313 | } | 315 | } |
| 314 | 316 | ||
| @@ -367,7 +369,7 @@ static int encode_passthrough(struct qaic_device *qdev, void *trans, struct wrap · 52 unmodified lines | |||
| 367 | if (in_trans->hdr.len % 8 != 0) | 369 | if (in_trans->hdr.len % 8 != 0) |
| 368 | return -EINVAL; | 370 | return -EINVAL; |
| 369 | 371 | ||
| 370 | if (size_add(msg_hdr_len, in_trans->hdr.len) > QAIC_MANAGE_EXT_MSG_LENGTH) | 372 | if (size_add(msg_hdr_len, in_trans->hdr.len) > QAIC_MANAGE_WIRE_MSG_LENGTH) |
| 371 | return -ENOSPC; | 373 | return -ENOSPC; |
| 372 | 374 | ||
| 373 | trans_wrapper = add_wrapper(wrappers, | 375 | trans_wrapper = add_wrapper(wrappers, |
| @@ -495,7 +497,7 @@ static int encode_addr_size_pairs(struct dma_xfer *xfer, struct wrapper_list *wr · 121 unmodified lines | |||
| 495 | 497 | ||
| 496 | nents = sgt->nents; | 498 | nents = sgt->nents; |
| 497 | nents_dma = nents; | 499 | nents_dma = nents; |
| 498 | *size = QAIC_MANAGE_EXT_MSG_LENGTH - msg_hdr_len - sizeof(**out_trans); | 500 | *size = QAIC_MANAGE_WIRE_MSG_LENGTH - msg_hdr_len - sizeof(**out_trans); |
| 499 | for_each_sgtable_dma_sg(sgt, sg, i) { | 501 | for_each_sgtable_dma_sg(sgt, sg, i) { |
| 500 | *size -= sizeof(*asp); | 502 | *size -= sizeof(*asp); |
| 501 | /* Save 1K for possible follow-up transactions. */ | 503 | /* Save 1K for possible follow-up transactions. */ |
| @@ -576,7 +578,7 @@ static int encode_dma(struct qaic_device *qdev, void *trans, struct wrapper_list · 74 unmodified lines | |||
| 576 | 578 | ||
| 577 | /* There should be enough space to hold at least one ASP entry. */ | 579 | /* There should be enough space to hold at least one ASP entry. */ |
| 578 | if (size_add(msg_hdr_len, sizeof(*out_trans) + sizeof(struct wire_addr_size_pair)) > | 580 | if (size_add(msg_hdr_len, sizeof(*out_trans) + sizeof(struct wire_addr_size_pair)) > |
| 579 | QAIC_MANAGE_EXT_MSG_LENGTH) | 581 | QAIC_MANAGE_WIRE_MSG_LENGTH) |
| 580 | return -ENOMEM; | 582 | return -ENOMEM; |
| 581 | 583 | ||
| 582 | xfer = kmalloc(sizeof(*xfer), GFP_KERNEL); | 584 | xfer = kmalloc(sizeof(*xfer), GFP_KERNEL); |
| @@ -645,7 +647,7 @@ static int encode_activate(struct qaic_device *qdev, void *trans, struct wrapper · 62 unmodified lines | |||
| 645 | msg = &wrapper->msg; | 647 | msg = &wrapper->msg; |
| 646 | msg_hdr_len = le32_to_cpu(msg->hdr.len); | 648 | msg_hdr_len = le32_to_cpu(msg->hdr.len); |
| 647 | 649 | ||
| 648 | if (size_add(msg_hdr_len, sizeof(*out_trans)) > QAIC_MANAGE_MAX_MSG_LENGTH) | 650 | if (size_add(msg_hdr_len, sizeof(*out_trans)) > QAIC_MANAGE_WIRE_MSG_LENGTH) |
| 649 | return -ENOSPC; | 651 | return -ENOSPC; |
| 650 | 652 | ||
| 651 | if (!in_trans->queue_size) | 653 | if (!in_trans->queue_size) |
| @@ -655,8 +657,9 @@ static int encode_activate(struct qaic_device *qdev, void *trans, struct wrapper · 3 unmodified lines | |||
| 655 | return -EINVAL; | 657 | return -EINVAL; |
| 656 | 658 | ||
| 657 | nelem = in_trans->queue_size; | 659 | nelem = in_trans->queue_size; |
| 658 | size = (get_dbc_req_elem_size() + get_dbc_rsp_elem_size()) * nelem; | 660 | if (check_mul_overflow((u32)(get_dbc_req_elem_size() + get_dbc_rsp_elem_size()), |
| 659 | if (size / nelem != get_dbc_req_elem_size() + get_dbc_rsp_elem_size()) | 661 | nelem, |
| 662 | &size)) | ||
| 660 | return -EINVAL; | 663 | return -EINVAL; |
| 661 | 664 | ||
| 662 | if (size + QAIC_DBC_Q_GAP + QAIC_DBC_Q_BUF_ALIGN < size) | 665 | if (size + QAIC_DBC_Q_GAP + QAIC_DBC_Q_BUF_ALIGN < size) |
| @@ -729,7 +732,7 @@ static int encode_status(struct qaic_device *qdev, void *trans, struct wrapper_l · 66 unmodified lines | |||
| 729 | msg = &wrapper->msg; | 732 | msg = &wrapper->msg; |
| 730 | msg_hdr_len = le32_to_cpu(msg->hdr.len); | 733 | msg_hdr_len = le32_to_cpu(msg->hdr.len); |
| 731 | 734 | ||
| 732 | if (size_add(msg_hdr_len, in_trans->hdr.len) > QAIC_MANAGE_MAX_MSG_LENGTH) | 735 | if (size_add(msg_hdr_len, in_trans->hdr.len) > QAIC_MANAGE_WIRE_MSG_LENGTH) |
| 733 | return -ENOSPC; | 736 | return -ENOSPC; |
| 734 | 737 | ||
| 735 | trans_wrapper = add_wrapper(wrappers, sizeof(*trans_wrapper)); | 738 | trans_wrapper = add_wrapper(wrappers, sizeof(*trans_wrapper)); |
| @@ -810,7 +813,7 @@ static int encode_message(struct qaic_device *qdev, struct manage_msg *user_msg, · 74 unmodified lines | |||
| 810 | } | 813 | } |
| 811 | 814 | ||
| 812 | if (ret) | 815 | if (ret) |
| 813 | break; | 816 | goto out; |
| 814 | } | 817 | } |
| 815 | 818 | ||
| 816 | if (user_len != user_msg->len) | 819 | if (user_len != user_msg->len) |
| @@ -921,6 +924,7 @@ static int decode_deactivate(struct qaic_device *qdev, void *trans, u32 *msg_len · 104 unmodified lines | |||
| 921 | } | 924 | } |
| 922 | 925 | ||
| 923 | release_dbc(qdev, dbc_id); | 926 | release_dbc(qdev, dbc_id); |
| 927 | set_dbc_state(qdev, dbc_id, DBC_STATE_IDLE); | ||
| 924 | *msg_len += sizeof(*in_trans); | 928 | *msg_len += sizeof(*in_trans); |
| 925 | 929 | ||
| 926 | return 0; | 930 | return 0; |
| @@ -1052,7 +1056,7 @@ static void *msg_xfer(struct qaic_device *qdev, struct wrapper_list *wrappers, u · 125 unmodified lines | |||
| 1052 | init_completion(&elem.xfer_done); | 1056 | init_completion(&elem.xfer_done); |
| 1053 | if (likely(!qdev->cntl_lost_buf)) { | 1057 | if (likely(!qdev->cntl_lost_buf)) { |
| 1054 | /* | 1058 | /* |
| 1055 | * The max size of request to device is QAIC_MANAGE_EXT_MSG_LENGTH. | 1059 | * The max size of request to device is QAIC_MANAGE_WIRE_MSG_LENGTH. |
| 1056 | * The max size of response from device is QAIC_MANAGE_MAX_MSG_LENGTH. | 1060 | * The max size of response from device is QAIC_MANAGE_MAX_MSG_LENGTH. |
| 1057 | */ | 1061 | */ |
| 1058 | out_buf = kmalloc(QAIC_MANAGE_MAX_MSG_LENGTH, GFP_KERNEL); | 1062 | out_buf = kmalloc(QAIC_MANAGE_MAX_MSG_LENGTH, GFP_KERNEL); |
| @@ -1079,7 +1083,6 @@ static void *msg_xfer(struct qaic_device *qdev, struct wrapper_list *wrappers, u · 20 unmodified lines | |||
| 1079 | 1083 | ||
| 1080 | list_for_each_entry(w, &wrappers->list, list) { | 1084 | list_for_each_entry(w, &wrappers->list, list) { |
| 1081 | kref_get(&w->ref_count); | 1085 | kref_get(&w->ref_count); |
| 1082 | retry_count = 0; | ||
| 1083 | ret = mhi_queue_buf(qdev->cntl_ch, DMA_TO_DEVICE, &w->msg, w->len, | 1086 | ret = mhi_queue_buf(qdev->cntl_ch, DMA_TO_DEVICE, &w->msg, w->len, |
| 1084 | list_is_last(&w->list, &wrappers->list) ? MHI_EOT : MHI_CHAIN); | 1087 | list_is_last(&w->list, &wrappers->list) ? MHI_EOT : MHI_CHAIN); |
| 1085 | if (ret) { | 1088 | if (ret) { |
| @@ -18,6 +18,7 @@ · 17 unmodified lines | |||
| 18 | #include <linux/scatterlist.h> | 18 | #include <linux/scatterlist.h> |
| 19 | #include <linux/spinlock.h> | 19 | #include <linux/spinlock.h> |
| 20 | #include <linux/srcu.h> | 20 | #include <linux/srcu.h> |
| 21 | #include <linux/string.h> | ||
| 21 | #include <linux/types.h> | 22 | #include <linux/types.h> |
| 22 | #include <linux/uaccess.h> | 23 | #include <linux/uaccess.h> |
| 23 | #include <linux/wait.h> | 24 | #include <linux/wait.h> |
| @@ -165,7 +166,7 @@ static void free_slice(struct kref *kref) · 141 unmodified lines | |||
| 165 | drm_gem_object_put(&slice->bo->base); | 166 | drm_gem_object_put(&slice->bo->base); |
| 166 | sg_free_table(slice->sgt); | 167 | sg_free_table(slice->sgt); |
| 167 | kfree(slice->sgt); | 168 | kfree(slice->sgt); |
| 168 | kfree(slice->reqs); | 169 | kvfree(slice->reqs); |
| 169 | kfree(slice); | 170 | kfree(slice); |
| 170 | } | 171 | } |
| 171 | 172 | ||
| @@ -404,7 +405,7 @@ static int qaic_map_one_slice(struct qaic_device *qdev, struct qaic_bo *bo, · 232 unmodified lines | |||
| 404 | goto free_sgt; | 405 | goto free_sgt; |
| 405 | } | 406 | } |
| 406 | 407 | ||
| 407 | slice->reqs = kcalloc(sgt->nents, sizeof(*slice->reqs), GFP_KERNEL); | 408 | slice->reqs = kvcalloc(sgt->nents, sizeof(*slice->reqs), GFP_KERNEL); |
| 408 | if (!slice->reqs) { | 409 | if (!slice->reqs) { |
| 409 | ret = -ENOMEM; | 410 | ret = -ENOMEM; |
| 410 | goto free_slice; | 411 | goto free_slice; |
| @@ -430,7 +431,7 @@ static int qaic_map_one_slice(struct qaic_device *qdev, struct qaic_bo *bo, · 19 unmodified lines | |||
| 430 | return 0; | 431 | return 0; |
| 431 | 432 | ||
| 432 | free_req: | 433 | free_req: |
| 433 | kfree(slice->reqs); | 434 | kvfree(slice->reqs); |
| 434 | free_slice: | 435 | free_slice: |
| 435 | kfree(slice); | 436 | kfree(slice); |
| 436 | free_sgt: | 437 | free_sgt: |
| @@ -643,8 +644,36 @@ static void qaic_free_object(struct drm_gem_object *obj) · 206 unmodified lines | |||
| 643 | kfree(bo); | 644 | kfree(bo); |
| 644 | } | 645 | } |
| 645 | 646 | ||
| 647 | static struct sg_table *qaic_get_sg_table(struct drm_gem_object *obj) | ||
| 648 | { | ||
| 649 | struct qaic_bo *bo = to_qaic_bo(obj); | ||
| 650 | struct scatterlist *sg, *sg_in; | ||
| 651 | struct sg_table *sgt, *sgt_in; | ||
| 652 | int i; | ||
| 653 | |||
| 654 | sgt_in = bo->sgt; | ||
| 655 | |||
| 656 | sgt = kmalloc(sizeof(*sgt), GFP_KERNEL); | ||
| 657 | if (!sgt) | ||
| 658 | return ERR_PTR(-ENOMEM); | ||
| 659 | |||
| 660 | if (sg_alloc_table(sgt, sgt_in->orig_nents, GFP_KERNEL)) { | ||
| 661 | kfree(sgt); | ||
| 662 | return ERR_PTR(-ENOMEM); | ||
| 663 | } | ||
| 664 | |||
| 665 | sg = sgt->sgl; | ||
| 666 | for_each_sgtable_sg(sgt_in, sg_in, i) { | ||
| 667 | memcpy(sg, sg_in, sizeof(*sg)); | ||
| 668 | sg = sg_next(sg); | ||
| 669 | } | ||
| 670 | |||
| 671 | return sgt; | ||
| 672 | } | ||
| 673 | |||
| 646 | static const struct drm_gem_object_funcs qaic_gem_funcs = { | 674 | static const struct drm_gem_object_funcs qaic_gem_funcs = { |
| 647 | .free = qaic_free_object, | 675 | .free = qaic_free_object, |
| 676 | .get_sg_table = qaic_get_sg_table, | ||
| 648 | .print_info = qaic_gem_print_info, | 677 | .print_info = qaic_gem_print_info, |
| 649 | .mmap = qaic_gem_object_mmap, | 678 | .mmap = qaic_gem_object_mmap, |
| 650 | .vm_ops = &drm_vm_ops, | 679 | .vm_ops = &drm_vm_ops, |
| @@ -953,8 +982,9 @@ int qaic_attach_slice_bo_ioctl(struct drm_device *dev, void *data, struct drm_fi · 302 unmodified lines | |||
| 953 | if (args->hdr.count == 0) | 982 | if (args->hdr.count == 0) |
| 954 | return -EINVAL; | 983 | return -EINVAL; |
| 955 | 984 | ||
| 956 | arg_size = args->hdr.count * sizeof(*slice_ent); | 985 | if (check_mul_overflow((unsigned long)args->hdr.count, |
| 957 | if (arg_size / args->hdr.count != sizeof(*slice_ent)) | 986 | (unsigned long)sizeof(*slice_ent), |
| 987 | &arg_size)) | ||
| 958 | return -EINVAL; | 988 | return -EINVAL; |
| 959 | 989 | ||
| 960 | if (!(args->hdr.dir == DMA_TO_DEVICE || args->hdr.dir == DMA_FROM_DEVICE)) | 990 | if (!(args->hdr.dir == DMA_TO_DEVICE || args->hdr.dir == DMA_FROM_DEVICE)) |
| @@ -984,18 +1014,12 @@ int qaic_attach_slice_bo_ioctl(struct drm_device *dev, void *data, struct drm_fi · 23 unmodified lines | |||
| 984 | 1014 | ||
| 985 | user_data = u64_to_user_ptr(args->data); | 1015 | user_data = u64_to_user_ptr(args->data); |
| 986 | 1016 | ||
| 987 | slice_ent = kzalloc(arg_size, GFP_KERNEL); | 1017 | slice_ent = memdup_user(user_data, arg_size); |
| 988 | if (!slice_ent) { | 1018 | if (IS_ERR(slice_ent)) { |
| 989 | ret = -EINVAL; | 1019 | ret = PTR_ERR(slice_ent); |
| 990 | goto unlock_dev_srcu; | 1020 | goto unlock_dev_srcu; |
| 991 | } | 1021 | } |
| 992 | 1022 | ||
| 993 | ret = copy_from_user(slice_ent, user_data, arg_size); | ||
| 994 | if (ret) { | ||
| 995 | ret = -EFAULT; | ||
| 996 | goto free_slice_ent; | ||
| 997 | } | ||
| 998 | |||
| 999 | obj = drm_gem_object_lookup(file_priv, args->hdr.handle); | 1023 | obj = drm_gem_object_lookup(file_priv, args->hdr.handle); |
| 1000 | if (!obj) { | 1024 | if (!obj) { |
| 1001 | ret = -ENOENT; | 1025 | ret = -ENOENT; |
| @@ -1023,6 +1047,11 @@ int qaic_attach_slice_bo_ioctl(struct drm_device *dev, void *data, struct drm_fi · 21 unmodified lines | |||
| 1023 | goto unlock_ch_srcu; | 1047 | goto unlock_ch_srcu; |
| 1024 | } | 1048 | } |
| 1025 | 1049 | ||
| 1050 | if (dbc->id == qdev->ssr_dbc) { | ||
| 1051 | ret = -EPIPE; | ||
| 1052 | goto unlock_ch_srcu; | ||
| 1053 | } | ||
| 1054 | |||
| 1026 | ret = qaic_prepare_bo(qdev, bo, &args->hdr); | 1055 | ret = qaic_prepare_bo(qdev, bo, &args->hdr); |
| 1027 | if (ret) | 1056 | if (ret) |
| 1028 | goto unlock_ch_srcu; | 1057 | goto unlock_ch_srcu; |
| @@ -1300,8 +1329,6 @@ static int __qaic_execute_bo_ioctl(struct drm_device *dev, void *data, struct dr · 271 unmodified lines | |||
| 1300 | int usr_rcu_id, qdev_rcu_id; | 1329 | int usr_rcu_id, qdev_rcu_id; |
| 1301 | struct qaic_device *qdev; | 1330 | struct qaic_device *qdev; |
| 1302 | struct qaic_user *usr; | 1331 | struct qaic_user *usr; |
| 1303 | u8 __user *user_data; | ||
| 1304 | unsigned long n; | ||
| 1305 | u64 received_ts; | 1332 | u64 received_ts; |
| 1306 | u32 queue_level; | 1333 | u32 queue_level; |
| 1307 | u64 submit_ts; | 1334 | u64 submit_ts; |
| @@ -1314,20 +1341,12 @@ static int __qaic_execute_bo_ioctl(struct drm_device *dev, void *data, struct dr · 6 unmodified lines | |||
| 1314 | received_ts = ktime_get_ns(); | 1341 | received_ts = ktime_get_ns(); |
| 1315 | 1342 | ||
| 1316 | size = is_partial ? sizeof(struct qaic_partial_execute_entry) : sizeof(*exec); | 1343 | size = is_partial ? sizeof(struct qaic_partial_execute_entry) : sizeof(*exec); |
| 1317 | n = (unsigned long)size * args->hdr.count; | 1344 | if (args->hdr.count == 0) |
| 1318 | if (args->hdr.count == 0 || n / args->hdr.count != size) | ||
| 1319 | return -EINVAL; | 1345 | return -EINVAL; |
| 1320 | 1346 | ||
| 1321 | user_data = u64_to_user_ptr(args->data); | 1347 | exec = memdup_array_user(u64_to_user_ptr(args->data), args->hdr.count, size); |
| 1322 | 1348 | if (IS_ERR(exec)) | |
| 1323 | exec = kcalloc(args->hdr.count, size, GFP_KERNEL); | 1349 | return PTR_ERR(exec); |
| 1324 | if (!exec) | ||
| 1325 | return -ENOMEM; | ||
| 1326 | |||
| 1327 | if (copy_from_user(exec, user_data, n)) { | ||
| 1328 | ret = -EFAULT; | ||
| 1329 | goto free_exec; | ||
| 1330 | } | ||
| 1331 | 1350 | ||
| 1332 | usr = file_priv->driver_priv; | 1351 | usr = file_priv->driver_priv; |
| 1333 | usr_rcu_id = srcu_read_lock(&usr->qddev_lock); | 1352 | usr_rcu_id = srcu_read_lock(&usr->qddev_lock); |
| @@ -1356,6 +1375,11 @@ static int __qaic_execute_bo_ioctl(struct drm_device *dev, void *data, struct dr · 22 unmodified lines | |||
| 1356 | goto release_ch_rcu; | 1375 | goto release_ch_rcu; |
| 1357 | } | 1376 | } |
| 1358 | 1377 | ||
| 1378 | if (dbc->id == qdev->ssr_dbc) { | ||
| 1379 | ret = -EPIPE; | ||
| 1380 | goto release_ch_rcu; | ||
| 1381 | } | ||
| 1382 | |||
| 1359 | ret = mutex_lock_interruptible(&dbc->req_lock); | 1383 | ret = mutex_lock_interruptible(&dbc->req_lock); |
| 1360 | if (ret) | 1384 | if (ret) |
| 1361 | goto release_ch_rcu; | 1385 | goto release_ch_rcu; |
| @@ -1396,7 +1420,6 @@ unlock_dev_srcu: · 34 unmodified lines | |||
| 1396 | srcu_read_unlock(&qdev->dev_lock, qdev_rcu_id); | 1420 | srcu_read_unlock(&qdev->dev_lock, qdev_rcu_id); |
| 1397 | unlock_usr_srcu: | 1421 | unlock_usr_srcu: |
| 1398 | srcu_read_unlock(&usr->qddev_lock, usr_rcu_id); | 1422 | srcu_read_unlock(&usr->qddev_lock, usr_rcu_id); |
| 1399 | free_exec: | ||
| 1400 | kfree(exec); | 1423 | kfree(exec); |
| 1401 | return ret; | 1424 | return ret; |
| 1402 | } | 1425 | } |
| @@ -1491,7 +1514,7 @@ irqreturn_t dbc_irq_handler(int irq, void *data) · 88 unmodified lines | |||
| 1491 | return IRQ_WAKE_THREAD; | 1514 | return IRQ_WAKE_THREAD; |
| 1492 | } | 1515 | } |
| 1493 | 1516 | ||
| 1494 | void irq_polling_work(struct work_struct *work) | 1517 | void qaic_irq_polling_work(struct work_struct *work) |
| 1495 | { | 1518 | { |
| 1496 | struct dma_bridge_chan *dbc = container_of(work, struct dma_bridge_chan, poll_work); | 1519 | struct dma_bridge_chan *dbc = container_of(work, struct dma_bridge_chan, poll_work); |
| 1497 | unsigned long flags; | 1520 | unsigned long flags; |
| @@ -1709,6 +1732,11 @@ int qaic_wait_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file · 211 unmodified lines | |||
| 1709 | goto unlock_ch_srcu; | 1732 | goto unlock_ch_srcu; |
| 1710 | } | 1733 | } |
| 1711 | 1734 | ||
| 1735 | if (dbc->id == qdev->ssr_dbc) { | ||
| 1736 | ret = -EPIPE; | ||
| 1737 | goto unlock_ch_srcu; | ||
| 1738 | } | ||
| 1739 | |||
| 1712 | obj = drm_gem_object_lookup(file_priv, args->handle); | 1740 | obj = drm_gem_object_lookup(file_priv, args->handle); |
| 1713 | if (!obj) { | 1741 | if (!obj) { |
| 1714 | ret = -ENOENT; | 1742 | ret = -ENOENT; |
| @@ -1729,6 +1757,9 @@ int qaic_wait_bo_ioctl(struct drm_device *dev, void *data, struct drm_file *file · 14 unmodified lines | |||
| 1729 | if (!dbc->usr) | 1757 | if (!dbc->usr) |
| 1730 | ret = -EPERM; | 1758 | ret = -EPERM; |
| 1731 | 1759 | ||
| 1760 | if (dbc->id == qdev->ssr_dbc) | ||
| 1761 | ret = -EPIPE; | ||
| 1762 | |||
| 1732 | put_obj: | 1763 | put_obj: |
| 1733 | drm_gem_object_put(obj); | 1764 | drm_gem_object_put(obj); |
| 1734 | unlock_ch_srcu: | 1765 | unlock_ch_srcu: |
| @@ -1749,7 +1780,8 @@ int qaic_perf_stats_bo_ioctl(struct drm_device *dev, void *data, struct drm_file · 14 unmodified lines | |||
| 1749 | struct qaic_device *qdev; | 1780 | struct qaic_device *qdev; |
| 1750 | struct qaic_user *usr; | 1781 | struct qaic_user *usr; |
| 1751 | struct qaic_bo *bo; | 1782 | struct qaic_bo *bo; |
| 1752 | int ret, i; | 1783 | int ret = 0; |
| 1784 | int i; | ||
| 1753 | 1785 | ||
| 1754 | usr = file_priv->driver_priv; | 1786 | usr = file_priv->driver_priv; |
| 1755 | usr_rcu_id = srcu_read_lock(&usr->qddev_lock); | 1787 | usr_rcu_id = srcu_read_lock(&usr->qddev_lock); |
| @@ -1770,18 +1802,12 @@ int qaic_perf_stats_bo_ioctl(struct drm_device *dev, void *data, struct drm_file · 14 unmodified lines | |||
| 1770 | goto unlock_dev_srcu; | 1802 | goto unlock_dev_srcu; |
| 1771 | } | 1803 | } |
| 1772 | 1804 | ||
| 1773 | ent = kcalloc(args->hdr.count, sizeof(*ent), GFP_KERNEL); | 1805 | ent = memdup_array_user(u64_to_user_ptr(args->data), args->hdr.count, sizeof(*ent)); |
| 1774 | if (!ent) { | 1806 | if (IS_ERR(ent)) { |
| 1775 | ret = -EINVAL; | 1807 | ret = PTR_ERR(ent); |
| 1776 | goto unlock_dev_srcu; | 1808 | goto unlock_dev_srcu; |
| 1777 | } | 1809 | } |
| 1778 | 1810 | ||
| 1779 | ret = copy_from_user(ent, u64_to_user_ptr(args->data), args->hdr.count * sizeof(*ent)); | ||
| 1780 | if (ret) { | ||
| 1781 | ret = -EFAULT; | ||
| 1782 | goto free_ent; | ||
| 1783 | } | ||
| 1784 | |||
| 1785 | for (i = 0; i < args->hdr.count; i++) { | 1811 | for (i = 0; i < args->hdr.count; i++) { |
| 1786 | obj = drm_gem_object_lookup(file_priv, ent[i].handle); | 1812 | obj = drm_gem_object_lookup(file_priv, ent[i].handle); |
| 1787 | if (!obj) { | 1813 | if (!obj) { |
| @@ -1789,6 +1815,16 @@ int qaic_perf_stats_bo_ioctl(struct drm_device *dev, void *data, struct drm_file · 1 unmodified line | |||
| 1789 | goto free_ent; | 1815 | goto free_ent; |
| 1790 | } | 1816 | } |
| 1791 | bo = to_qaic_bo(obj); | 1817 | bo = to_qaic_bo(obj); |
| 1818 | if (!bo->sliced) { | ||
| 1819 | drm_gem_object_put(obj); | ||
| 1820 | ret = -EINVAL; | ||
| 1821 | goto free_ent; | ||
| 1822 | } | ||
| 1823 | if (bo->dbc->id != args->hdr.dbc_id) { | ||
| 1824 | drm_gem_object_put(obj); | ||
| 1825 | ret = -EINVAL; | ||
| 1826 | goto free_ent; | ||
| 1827 | } | ||
| 1792 | /* | 1828 | /* |
| 1793 | * perf stats ioctl is called before wait ioctl is complete then | 1829 | * perf stats ioctl is called before wait ioctl is complete then |
| 1794 | * the latency information is invalid. | 1830 | * the latency information is invalid. |
| @@ -1927,6 +1963,17 @@ static void empty_xfer_list(struct qaic_device *qdev, struct dma_bridge_chan *db · 132 unmodified lines | |||
| 1927 | spin_unlock_irqrestore(&dbc->xfer_lock, flags); | 1963 | spin_unlock_irqrestore(&dbc->xfer_lock, flags); |
| 1928 | } | 1964 | } |
| 1929 | 1965 | ||
| 1966 | static void sync_empty_xfer_list(struct qaic_device *qdev, struct dma_bridge_chan *dbc) | ||
| 1967 | { | ||
| 1968 | empty_xfer_list(qdev, dbc); | ||
| 1969 | synchronize_srcu(&dbc->ch_lock); | ||
| 1970 | /* | ||
| 1971 | * Threads holding channel lock, may add more elements in the xfer_list. | ||
| 1972 | * Flush out these elements from xfer_list. | ||
| 1973 | */ | ||
| 1974 | empty_xfer_list(qdev, dbc); | ||
| 1975 | } | ||
| 1976 | |||
| 1930 | int disable_dbc(struct qaic_device *qdev, u32 dbc_id, struct qaic_user *usr) | 1977 | int disable_dbc(struct qaic_device *qdev, u32 dbc_id, struct qaic_user *usr) |
| 1931 | { | 1978 | { |
| 1932 | if (!qdev->dbc[dbc_id].usr || qdev->dbc[dbc_id].usr->handle != usr->handle) | 1979 | if (!qdev->dbc[dbc_id].usr || qdev->dbc[dbc_id].usr->handle != usr->handle) |
| @@ -1941,7 +1988,7 @@ int disable_dbc(struct qaic_device *qdev, u32 dbc_id, struct qaic_user *usr) · 8 unmodified lines | |||
| 1941 | * enable_dbc - Enable the DBC. DBCs are disabled by removing the context of | 1988 | * enable_dbc - Enable the DBC. DBCs are disabled by removing the context of |
| 1942 | * user. Add user context back to DBC to enable it. This function trusts the | 1989 | * user. Add user context back to DBC to enable it. This function trusts the |
| 1943 | * DBC ID passed and expects the DBC to be disabled. | 1990 | * DBC ID passed and expects the DBC to be disabled. |
| 1944 | * @qdev: Qranium device handle | 1991 | * @qdev: qaic device handle |
| 1945 | * @dbc_id: ID of the DBC | 1992 | * @dbc_id: ID of the DBC |
| 1946 | * @usr: User context | 1993 | * @usr: User context |
| 1947 | */ | 1994 | */ |
| @@ -1955,13 +2002,7 @@ void wakeup_dbc(struct qaic_device *qdev, u32 dbc_id) · 7 unmodified lines | |||
| 1955 | struct dma_bridge_chan *dbc = &qdev->dbc[dbc_id]; | 2002 | struct dma_bridge_chan *dbc = &qdev->dbc[dbc_id]; |
| 1956 | 2003 | ||
| 1957 | dbc->usr = NULL; | 2004 | dbc->usr = NULL; |
| 1958 | empty_xfer_list(qdev, dbc); | 2005 | sync_empty_xfer_list(qdev, dbc); |
| 1959 | synchronize_srcu(&dbc->ch_lock); | ||
| 1960 | /* | ||
| 1961 | * Threads holding channel lock, may add more elements in the xfer_list. | ||
| 1962 | * Flush out these elements from xfer_list. | ||
| 1963 | */ | ||
| 1964 | empty_xfer_list(qdev, dbc); | ||
| 1965 | } | 2006 | } |
| 1966 | 2007 | ||
| 1967 | void release_dbc(struct qaic_device *qdev, u32 dbc_id) | 2008 | void release_dbc(struct qaic_device *qdev, u32 dbc_id) |
| @@ -2002,3 +2043,30 @@ void qaic_data_get_fifo_info(struct dma_bridge_chan *dbc, u32 *head, u32 *tail) · 34 unmodified lines | |||
| 2002 | *head = readl(dbc->dbc_base + REQHP_OFF); | 2043 | *head = readl(dbc->dbc_base + REQHP_OFF); |
| 2003 | *tail = readl(dbc->dbc_base + REQTP_OFF); | 2044 | *tail = readl(dbc->dbc_base + REQTP_OFF); |
| 2004 | } | 2045 | } |
| 2046 | |||
| 2047 | /* | ||
| 2048 | * qaic_dbc_enter_ssr - Prepare to enter in sub system reset(SSR) for given DBC ID. | ||
| 2049 | * @qdev: qaic device handle | ||
| 2050 | * @dbc_id: ID of the DBC which will enter SSR | ||
| 2051 | * | ||
| 2052 | * The device will automatically deactivate the workload as not | ||
| 2053 | * all errors can be silently recovered. The user will be | ||
| 2054 | * notified and will need to decide the required recovery | ||
| 2055 | * action to take. | ||
| 2056 | */ | ||
| 2057 | void qaic_dbc_enter_ssr(struct qaic_device *qdev, u32 dbc_id) | ||
| 2058 | { | ||
| 2059 | qdev->ssr_dbc = dbc_id; | ||
| 2060 | release_dbc(qdev, dbc_id); | ||
| 2061 | } | ||
| 2062 | |||
| 2063 | /* | ||
| 2064 | * qaic_dbc_exit_ssr - Prepare to exit from sub system reset(SSR) for given DBC ID. | ||
| 2065 | * @qdev: qaic device handle | ||
| 2066 | * | ||
| 2067 | * The DBC returns to an operational state and begins accepting work after exiting SSR. | ||
| 2068 | */ | ||
| 2069 | void qaic_dbc_exit_ssr(struct qaic_device *qdev) | ||
| 2070 | { | ||
| 2071 | qdev->ssr_dbc = QAIC_SSR_DBC_SENTINEL; | ||
| 2072 | } | ||
| @@ -30,6 +30,7 @@ · 29 unmodified lines | |||
| 30 | #include "qaic.h" | 30 | #include "qaic.h" |
| 31 | #include "qaic_debugfs.h" | 31 | #include "qaic_debugfs.h" |
| 32 | #include "qaic_ras.h" | 32 | #include "qaic_ras.h" |
| 33 | #include "qaic_ssr.h" | ||
| 33 | #include "qaic_timesync.h" | 34 | #include "qaic_timesync.h" |
| 34 | #include "sahara.h" | 35 | #include "sahara.h" |
| 35 | 36 | ||
| @@ -270,6 +271,13 @@ static int qaic_create_drm_device(struct qaic_device *qdev, s32 partition_id) · 234 unmodified lines | |||
| 270 | return ret; | 271 | return ret; |
| 271 | } | 272 | } |
| 272 | 273 | ||
| 274 | ret = qaic_sysfs_init(qddev); | ||
| 275 | if (ret) { | ||
| 276 | drm_dev_unregister(drm); | ||
| 277 | pci_dbg(qdev->pdev, "qaic_sysfs_init failed %d\n", ret); | ||
| 278 | return ret; | ||
| 279 | } | ||
| 280 | |||
| 273 | qaic_debugfs_init(qddev); | 281 | qaic_debugfs_init(qddev); |
| 274 | 282 | ||
| 275 | return ret; | 283 | return ret; |
| @@ -281,6 +289,7 @@ static void qaic_destroy_drm_device(struct qaic_device *qdev, s32 partition_id) · 5 unmodified lines | |||
| 281 | struct drm_device *drm = to_drm(qddev); | 289 | struct drm_device *drm = to_drm(qddev); |
| 282 | struct qaic_user *usr; | 290 | struct qaic_user *usr; |
| 283 | 291 | ||
| 292 | qaic_sysfs_remove(qddev); | ||
| 284 | drm_dev_unregister(drm); | 293 | drm_dev_unregister(drm); |
| 285 | qddev->partition_id = 0; | 294 | qddev->partition_id = 0; |
| 286 | /* | 295 | /* |
| @@ -382,6 +391,7 @@ void qaic_dev_reset_clean_local_state(struct qaic_device *qdev) · 95 unmodified lines | |||
| 382 | qaic_notify_reset(qdev); | 391 | qaic_notify_reset(qdev); |
| 383 | 392 | ||
| 384 | /* start tearing things down */ | 393 | /* start tearing things down */ |
| 394 | qaic_clean_up_ssr(qdev); | ||
| 385 | for (i = 0; i < qdev->num_dbc; ++i) | 395 | for (i = 0; i < qdev->num_dbc; ++i) |
| 386 | release_dbc(qdev, i); | 396 | release_dbc(qdev, i); |
| 387 | } | 397 | } |
| @@ -431,11 +441,18 @@ static struct qaic_device *create_qdev(struct pci_dev *pdev, · 43 unmodified lines | |||
| 431 | qdev->qts_wq = qaicm_wq_init(drm, "qaic_ts"); | 441 | qdev->qts_wq = qaicm_wq_init(drm, "qaic_ts"); |
| 432 | if (IS_ERR(qdev->qts_wq)) | 442 | if (IS_ERR(qdev->qts_wq)) |
| 433 | return NULL; | 443 | return NULL; |
| 444 | qdev->ssr_wq = qaicm_wq_init(drm, "qaic_ssr"); | ||
| 445 | if (IS_ERR(qdev->ssr_wq)) | ||
| 446 | return NULL; | ||
| 434 | 447 | ||
| 435 | ret = qaicm_srcu_init(drm, &qdev->dev_lock); | 448 | ret = qaicm_srcu_init(drm, &qdev->dev_lock); |
| 436 | if (ret) | 449 | if (ret) |
| 437 | return NULL; | 450 | return NULL; |
| 438 | 451 | ||
| 452 | ret = qaic_ssr_init(qdev, drm); | ||
| 453 | if (ret) | ||
| 454 | pci_info(pdev, "QAIC SSR crashdump collection not supported.\n"); | ||
| 455 | |||
| 439 | qdev->qddev = qddev; | 456 | qdev->qddev = qddev; |
| 440 | qdev->pdev = pdev; | 457 | qdev->pdev = pdev; |
| 441 | qddev->qdev = qdev; | 458 | qddev->qdev = qdev; |
| @@ -545,7 +562,7 @@ static int init_msi(struct qaic_device *qdev, struct pci_dev *pdev) · 103 unmodified lines | |||
| 545 | qdev->dbc[i].irq = pci_irq_vector(pdev, qdev->single_msi ? 0 : i + 1); | 562 | qdev->dbc[i].irq = pci_irq_vector(pdev, qdev->single_msi ? 0 : i + 1); |
| 546 | if (!qdev->single_msi) | 563 | if (!qdev->single_msi) |
| 547 | disable_irq_nosync(qdev->dbc[i].irq); | 564 | disable_irq_nosync(qdev->dbc[i].irq); |
| 548 | INIT_WORK(&qdev->dbc[i].poll_work, irq_polling_work); | 565 | INIT_WORK(&qdev->dbc[i].poll_work, qaic_irq_polling_work); |
| 549 | } | 566 | } |
| 550 | } | 567 | } |
| 551 | 568 | ||
| @@ -660,6 +677,92 @@ static const struct pci_error_handlers qaic_pci_err_handler = { · 108 unmodified lines | |||
| 660 | .reset_done = qaic_pci_reset_done, | 677 | .reset_done = qaic_pci_reset_done, |
| 661 | }; | 678 | }; |
| 662 | 679 | ||
| 680 | static bool qaic_is_under_reset(struct qaic_device *qdev) | ||
| 681 | { | ||
| 682 | int rcu_id; | ||
| 683 | bool ret; | ||
| 684 | |||
| 685 | rcu_id = srcu_read_lock(&qdev->dev_lock); | ||
| 686 | ret = qdev->dev_state != QAIC_ONLINE; | ||
| 687 | srcu_read_unlock(&qdev->dev_lock, rcu_id); | ||
| 688 | return ret; | ||
| 689 | } | ||
| 690 | |||
| 691 | static bool qaic_data_path_busy(struct qaic_device *qdev) | ||
| 692 | { | ||
| 693 | bool ret = false; | ||
| 694 | int dev_rcu_id; | ||
| 695 | int i; | ||
| 696 | |||
| 697 | dev_rcu_id = srcu_read_lock(&qdev->dev_lock); | ||
| 698 | if (qdev->dev_state != QAIC_ONLINE) { | ||
| 699 | srcu_read_unlock(&qdev->dev_lock, dev_rcu_id); | ||
| 700 | return false; | ||
| 701 | } | ||
| 702 | for (i = 0; i < qdev->num_dbc; i++) { | ||
| 703 | struct dma_bridge_chan *dbc = &qdev->dbc[i]; | ||
| 704 | unsigned long flags; | ||
| 705 | int ch_rcu_id; | ||
| 706 | |||
| 707 | ch_rcu_id = srcu_read_lock(&dbc->ch_lock); | ||
| 708 | if (!dbc->usr || !dbc->in_use) { | ||
| 709 | srcu_read_unlock(&dbc->ch_lock, ch_rcu_id); | ||
| 710 | continue; | ||
| 711 | } | ||
| 712 | spin_lock_irqsave(&dbc->xfer_lock, flags); | ||
| 713 | ret = !list_empty(&dbc->xfer_list); | ||
| 714 | spin_unlock_irqrestore(&dbc->xfer_lock, flags); | ||
| 715 | srcu_read_unlock(&dbc->ch_lock, ch_rcu_id); | ||
| 716 | if (ret) | ||
| 717 | break; | ||
| 718 | } | ||
| 719 | srcu_read_unlock(&qdev->dev_lock, dev_rcu_id); | ||
| 720 | return ret; | ||
| 721 | } | ||
| 722 | |||
| 723 | static int qaic_pm_suspend(struct device *dev) | ||
| 724 | { | ||
| 725 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); | ||
| 726 | |||
| 727 | dev_dbg(dev, "Suspending..\n"); | ||
| 728 | if (qaic_data_path_busy(qdev)) { | ||
| 729 | dev_dbg(dev, "Device's datapath is busy. Aborting suspend..\n"); | ||
| 730 | return -EBUSY; | ||
| 731 | } | ||
| 732 | if (qaic_is_under_reset(qdev)) { | ||
| 733 | dev_dbg(dev, "Device is under reset. Aborting suspend..\n"); | ||
| 734 | return -EBUSY; | ||
| 735 | } | ||
| 736 | qaic_mqts_ch_stop_timer(qdev->mqts_ch); | ||
| 737 | qaic_pci_reset_prepare(qdev->pdev); | ||
| 738 | pci_save_state(qdev->pdev); | ||
| 739 | pci_disable_device(qdev->pdev); | ||
| 740 | pci_set_power_state(qdev->pdev, PCI_D3hot); | ||
| 741 | return 0; | ||
| 742 | } | ||
| 743 | |||
| 744 | static int qaic_pm_resume(struct device *dev) | ||
| 745 | { | ||
| 746 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); | ||
| 747 | int ret; | ||
| 748 | |||
| 749 | dev_dbg(dev, "Resuming..\n"); | ||
| 750 | pci_set_power_state(qdev->pdev, PCI_D0); | ||
| 751 | pci_restore_state(qdev->pdev); | ||
| 752 | ret = pci_enable_device(qdev->pdev); | ||
| 753 | if (ret) { | ||
| 754 | dev_err(dev, "pci_enable_device failed on resume %d\n", ret); | ||
| 755 | return ret; | ||
| 756 | } | ||
| 757 | pci_set_master(qdev->pdev); | ||
| 758 | qaic_pci_reset_done(qdev->pdev); | ||
| 759 | return 0; | ||
| 760 | } | ||
| 761 | |||
| 762 | static const struct dev_pm_ops qaic_pm_ops = { | ||
| 763 | SYSTEM_SLEEP_PM_OPS(qaic_pm_suspend, qaic_pm_resume) | ||
| 764 | }; | ||
| 765 | |||
| 663 | static struct pci_driver qaic_pci_driver = { | 766 | static struct pci_driver qaic_pci_driver = { |
| 664 | .name = QAIC_NAME, | 767 | .name = QAIC_NAME, |
| 665 | .id_table = qaic_ids, | 768 | .id_table = qaic_ids, |
| @@ -667,6 +770,9 @@ static struct pci_driver qaic_pci_driver = { · 1 unmodified line | |||
| 667 | .remove = qaic_pci_remove, | 770 | .remove = qaic_pci_remove, |
| 668 | .shutdown = qaic_pci_shutdown, | 771 | .shutdown = qaic_pci_shutdown, |
| 669 | .err_handler = &qaic_pci_err_handler, | 772 | .err_handler = &qaic_pci_err_handler, |
| 773 | .driver = { | ||
| 774 | .pm = pm_sleep_ptr(&qaic_pm_ops), | ||
| 775 | }, | ||
| 670 | }; | 776 | }; |
| 671 | 777 | ||
| 672 | static int __init qaic_init(void) | 778 | static int __init qaic_init(void) |
| @@ -702,9 +808,16 @@ static int __init qaic_init(void) · 29 unmodified lines | |||
| 702 | ret = qaic_ras_register(); | 808 | ret = qaic_ras_register(); |
| 703 | if (ret) | 809 | if (ret) |
| 704 | pr_debug("qaic: qaic_ras_register failed %d\n", ret); | 810 | pr_debug("qaic: qaic_ras_register failed %d\n", ret); |
| 811 | ret = qaic_ssr_register(); | ||
| 812 | if (ret) { | ||
| 813 | pr_debug("qaic: qaic_ssr_register failed %d\n", ret); | ||
| 814 | goto free_bootlog; | ||
| 815 | } | ||
| 705 | 816 | ||
| 706 | return 0; | 817 | return 0; |
| 707 | 818 | ||
| 819 | free_bootlog: | ||
| 820 | qaic_bootlog_unregister(); | ||
| 708 | free_mhi: | 821 | free_mhi: |
| 709 | mhi_driver_unregister(&qaic_mhi_driver); | 822 | mhi_driver_unregister(&qaic_mhi_driver); |
| 710 | free_pci: | 823 | free_pci: |
| @@ -730,6 +843,7 @@ static void __exit qaic_exit(void) · 19 unmodified lines | |||
| 730 | * reinitializing the link_up state after the cleanup is done. | 843 | * reinitializing the link_up state after the cleanup is done. |
| 731 | */ | 844 | */ |
| 732 | link_up = true; | 845 | link_up = true; |
| 846 | qaic_ssr_unregister(); | ||
| 733 | qaic_ras_unregister(); | 847 | qaic_ras_unregister(); |
| 734 | qaic_bootlog_unregister(); | 848 | qaic_bootlog_unregister(); |
| 735 | qaic_timesync_deinit(); | 849 | qaic_timesync_deinit(); |
| @@ -514,21 +514,21 @@ static ssize_t ce_count_show(struct device *dev, struct device_attribute *attr, · 513 unmodified lines | |||
| 514 | { | 514 | { |
| 515 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); | 515 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); |
| 516 | 516 | ||
| 517 | return snprintf(buf, PAGE_SIZE, "%d\n", qdev->ce_count); | 517 | return sysfs_emit(buf, "%d\n", qdev->ce_count); |
| 518 | } | 518 | } |
| 519 | 519 | ||
| 520 | static ssize_t ue_count_show(struct device *dev, struct device_attribute *attr, char *buf) | 520 | static ssize_t ue_count_show(struct device *dev, struct device_attribute *attr, char *buf) |
| 521 | { | 521 | { |
| 522 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); | 522 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); |
| 523 | 523 | ||
| 524 | return snprintf(buf, PAGE_SIZE, "%d\n", qdev->ue_count); | 524 | return sysfs_emit(buf, "%d\n", qdev->ue_count); |
| 525 | } | 525 | } |
| 526 | 526 | ||
| 527 | static ssize_t ue_nonfatal_count_show(struct device *dev, struct device_attribute *attr, char *buf) | 527 | static ssize_t ue_nonfatal_count_show(struct device *dev, struct device_attribute *attr, char *buf) |
| 528 | { | 528 | { |
| 529 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); | 529 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(dev)); |
| 530 | 530 | ||
| 531 | return snprintf(buf, PAGE_SIZE, "%d\n", qdev->ue_nf_count); | 531 | return sysfs_emit(buf, "%d\n", qdev->ue_nf_count); |
| 532 | } | 532 | } |
| 533 | 533 | ||
| 534 | static DEVICE_ATTR_RO(ce_count); | 534 | static DEVICE_ATTR_RO(ce_count); |
| @@ -0,0 +1,815 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | |||
| 3 | /* Copyright (c) 2020-2021, The Linux Foundation. All rights reserved. */ | ||
| 4 | /* Copyright (c) 2021-2024 Qualcomm Innovation Center, Inc. All rights reserved. */ | ||
| 5 | |||
| 6 | #include <asm/byteorder.h> | ||
| 7 | #include <drm/drm_file.h> | ||
| 8 | #include <drm/drm_managed.h> | ||
| 9 | #include <linux/devcoredump.h> | ||
| 10 | #include <linux/device.h> | ||
| 11 | #include <linux/kernel.h> | ||
| 12 | #include <linux/mhi.h> | ||
| 13 | #include <linux/workqueue.h> | ||
| 14 | |||
| 15 | #include "qaic.h" | ||
| 16 | #include "qaic_ssr.h" | ||
| 17 | |||
| 18 | #define SSR_RESP_MSG_SZ 32 | ||
| 19 | #define SSR_MHI_BUF_SIZE SZ_64K | ||
| 20 | #define SSR_MEM_READ_DATA_SIZE ((u64)SSR_MHI_BUF_SIZE - sizeof(struct ssr_crashdump)) | ||
| 21 | #define SSR_MEM_READ_CHUNK_SIZE ((u64)SSR_MEM_READ_DATA_SIZE - sizeof(struct ssr_memory_read_rsp)) | ||
| 22 | |||
| 23 | #define DEBUG_TRANSFER_INFO BIT(0) | ||
| 24 | #define DEBUG_TRANSFER_INFO_RSP BIT(1) | ||
| 25 | #define MEMORY_READ BIT(2) | ||
| 26 | #define MEMORY_READ_RSP BIT(3) | ||
| 27 | #define DEBUG_TRANSFER_DONE BIT(4) | ||
| 28 | #define DEBUG_TRANSFER_DONE_RSP BIT(5) | ||
| 29 | #define SSR_EVENT BIT(8) | ||
| 30 | #define SSR_EVENT_RSP BIT(9) | ||
| 31 | |||
| 32 | #define SSR_EVENT_NACK BIT(0) | ||
| 33 | #define BEFORE_SHUTDOWN BIT(1) | ||
| 34 | #define AFTER_SHUTDOWN BIT(2) | ||
| 35 | #define BEFORE_POWER_UP BIT(3) | ||
| 36 | #define AFTER_POWER_UP BIT(4) | ||
| 37 | |||
| 38 | struct debug_info_table { | ||
| 39 | /* Save preferences. Default is mandatory */ | ||
| 40 | u64 save_perf; | ||
| 41 | /* Base address of the debug region */ | ||
| 42 | u64 mem_base; | ||
| 43 | /* Size of debug region in bytes */ | ||
| 44 | u64 len; | ||
| 45 | /* Description */ | ||
| 46 | char desc[20]; | ||
| 47 | /* Filename of debug region */ | ||
| 48 | char filename[20]; | ||
| 49 | }; | ||
| 50 | |||
| 51 | struct _ssr_hdr { | ||
| 52 | __le32 cmd; | ||
| 53 | __le32 len; | ||
| 54 | __le32 dbc_id; | ||
| 55 | }; | ||
| 56 | |||
| 57 | struct ssr_hdr { | ||
| 58 | u32 cmd; | ||
| 59 | u32 len; | ||
| 60 | u32 dbc_id; | ||
| 61 | }; | ||
| 62 | |||
| 63 | struct ssr_debug_transfer_info { | ||
| 64 | struct ssr_hdr hdr; | ||
| 65 | u32 resv; | ||
| 66 | u64 tbl_addr; | ||
| 67 | u64 tbl_len; | ||
| 68 | } __packed; | ||
| 69 | |||
| 70 | struct ssr_debug_transfer_info_rsp { | ||
| 71 | struct _ssr_hdr hdr; | ||
| 72 | __le32 ret; | ||
| 73 | } __packed; | ||
| 74 | |||
| 75 | struct ssr_memory_read { | ||
| 76 | struct _ssr_hdr hdr; | ||
| 77 | __le32 resv; | ||
| 78 | __le64 addr; | ||
| 79 | __le64 len; | ||
| 80 | } __packed; | ||
| 81 | |||
| 82 | struct ssr_memory_read_rsp { | ||
| 83 | struct _ssr_hdr hdr; | ||
| 84 | __le32 resv; | ||
| 85 | u8 data[]; | ||
| 86 | } __packed; | ||
| 87 | |||
| 88 | struct ssr_debug_transfer_done { | ||
| 89 | struct _ssr_hdr hdr; | ||
| 90 | __le32 resv; | ||
| 91 | } __packed; | ||
| 92 | |||
| 93 | struct ssr_debug_transfer_done_rsp { | ||
| 94 | struct _ssr_hdr hdr; | ||
| 95 | __le32 ret; | ||
| 96 | } __packed; | ||
| 97 | |||
| 98 | struct ssr_event { | ||
| 99 | struct ssr_hdr hdr; | ||
| 100 | u32 event; | ||
| 101 | } __packed; | ||
| 102 | |||
| 103 | struct ssr_event_rsp { | ||
| 104 | struct _ssr_hdr hdr; | ||
| 105 | __le32 event; | ||
| 106 | } __packed; | ||
| 107 | |||
| 108 | struct ssr_resp { | ||
| 109 | /* Work struct to schedule work coming on QAIC_SSR channel */ | ||
| 110 | struct work_struct work; | ||
| 111 | /* Root struct of device, used to access device resources */ | ||
| 112 | struct qaic_device *qdev; | ||
| 113 | /* Buffer used by MHI for transfer requests */ | ||
| 114 | u8 data[] __aligned(8); | ||
| 115 | }; | ||
| 116 | |||
| 117 | /* SSR crashdump book keeping structure */ | ||
| 118 | struct ssr_dump_info { | ||
| 119 | /* DBC associated with this SSR crashdump */ | ||
| 120 | struct dma_bridge_chan *dbc; | ||
| 121 | /* | ||
| 122 | * It will be used when we complete the crashdump download and switch | ||
| 123 | * to waiting on SSR events | ||
| 124 | */ | ||
| 125 | struct ssr_resp *resp; | ||
| 126 | /* MEMORY READ request MHI buffer.*/ | ||
| 127 | struct ssr_memory_read *read_buf_req; | ||
| 128 | /* TRUE: ->read_buf_req is queued for MHI transaction. FALSE: Otherwise */ | ||
| 129 | bool read_buf_req_queued; | ||
| 130 | /* Address of table in host */ | ||
| 131 | void *tbl_addr; | ||
| 132 | /* Total size of table */ | ||
| 133 | u64 tbl_len; | ||
| 134 | /* Offset of table(->tbl_addr) where the new chunk will be dumped */ | ||
| 135 | u64 tbl_off; | ||
| 136 | /* Address of table in device/target */ | ||
| 137 | u64 tbl_addr_dev; | ||
| 138 | /* Ptr to the entire dump */ | ||
| 139 | void *dump_addr; | ||
| 140 | /* Entire crashdump size */ | ||
| 141 | u64 dump_sz; | ||
| 142 | /* Offset of crashdump(->dump_addr) where the new chunk will be dumped */ | ||
| 143 | u64 dump_off; | ||
| 144 | /* Points to the table entry we are currently downloading */ | ||
| 145 | struct debug_info_table *tbl_ent; | ||
| 146 | /* Offset in the current table entry(->tbl_ent) for next chuck */ | ||
| 147 | u64 tbl_ent_off; | ||
| 148 | }; | ||
| 149 | |||
| 150 | struct ssr_crashdump { | ||
| 151 | /* | ||
| 152 | * Points to a book keeping struct maintained by MHI SSR device while | ||
| 153 | * downloading a SSR crashdump. It is NULL when crashdump downloading | ||
| 154 | * not in progress. | ||
| 155 | */ | ||
| 156 | struct ssr_dump_info *dump_info; | ||
| 157 | /* Work struct to schedule work coming on QAIC_SSR channel */ | ||
| 158 | struct work_struct work; | ||
| 159 | /* Root struct of device, used to access device resources */ | ||
| 160 | struct qaic_device *qdev; | ||
| 161 | /* Buffer used by MHI for transfer requests */ | ||
| 162 | u8 data[]; | ||
| 163 | }; | ||
| 164 | |||
| 165 | #define QAIC_SSR_DUMP_V1_MAGIC 0x1234567890abcdef | ||
| 166 | #define QAIC_SSR_DUMP_V1_VER 1 | ||
| 167 | struct dump_file_meta { | ||
| 168 | u64 magic; | ||
| 169 | u64 version; | ||
| 170 | u64 size; /* Total size of the entire dump */ | ||
| 171 | u64 tbl_len; /* Length of the table in byte */ | ||
| 172 | }; | ||
| 173 | |||
| 174 | /* | ||
| 175 | * Layout of crashdump | ||
| 176 | * +------------------------------------------+ | ||
| 177 | * | Crashdump Meta structure | | ||
| 178 | * | type: struct dump_file_meta | | ||
| 179 | * +------------------------------------------+ | ||
| 180 | * | Crashdump Table | | ||
| 181 | * | type: array of struct debug_info_table | | ||
| 182 | * | | | ||
| 183 | * | | | ||
| 184 | * | | | ||
| 185 | * +------------------------------------------+ | ||
| 186 | * | Crashdump | | ||
| 187 | * | | | ||
| 188 | * | | | ||
| 189 | * | | | ||
| 190 | * | | | ||
| 191 | * | | | ||
| 192 | * +------------------------------------------+ | ||
| 193 | */ | ||
| 194 | |||
| 195 | static void free_ssr_dump_info(struct ssr_crashdump *ssr_crash) | ||
| 196 | { | ||
| 197 | struct ssr_dump_info *dump_info = ssr_crash->dump_info; | ||
| 198 | |||
| 199 | ssr_crash->dump_info = NULL; | ||
| 200 | if (!dump_info) | ||
| 201 | return; | ||
| 202 | if (!dump_info->read_buf_req_queued) | ||
| 203 | kfree(dump_info->read_buf_req); | ||
| 204 | vfree(dump_info->tbl_addr); | ||
| 205 | vfree(dump_info->dump_addr); | ||
| 206 | kfree(dump_info); | ||
| 207 | } | ||
| 208 | |||
| 209 | void qaic_clean_up_ssr(struct qaic_device *qdev) | ||
| 210 | { | ||
| 211 | struct ssr_crashdump *ssr_crash = qdev->ssr_mhi_buf; | ||
| 212 | |||
| 213 | if (!ssr_crash) | ||
| 214 | return; | ||
| 215 | |||
| 216 | qaic_dbc_exit_ssr(qdev); | ||
| 217 | free_ssr_dump_info(ssr_crash); | ||
| 218 | } | ||
| 219 | |||
| 220 | static int alloc_dump(struct ssr_dump_info *dump_info) | ||
| 221 | { | ||
| 222 | struct debug_info_table *tbl_ent = dump_info->tbl_addr; | ||
| 223 | struct dump_file_meta *dump_meta; | ||
| 224 | u64 tbl_sz_lp = 0; | ||
| 225 | u64 dump_size = 0; | ||
| 226 | |||
| 227 | while (tbl_sz_lp < dump_info->tbl_len) { | ||
| 228 | le64_to_cpus(&tbl_ent->save_perf); | ||
| 229 | le64_to_cpus(&tbl_ent->mem_base); | ||
| 230 | le64_to_cpus(&tbl_ent->len); | ||
| 231 | |||
| 232 | if (tbl_ent->len == 0) | ||
| 233 | return -EINVAL; | ||
| 234 | |||
| 235 | dump_size += tbl_ent->len; | ||
| 236 | tbl_ent++; | ||
| 237 | tbl_sz_lp += sizeof(*tbl_ent); | ||
| 238 | } | ||
| 239 | |||
| 240 | dump_info->dump_sz = dump_size + dump_info->tbl_len + sizeof(*dump_meta); | ||
| 241 | dump_info->dump_addr = vzalloc(dump_info->dump_sz); | ||
| 242 | if (!dump_info->dump_addr) | ||
| 243 | return -ENOMEM; | ||
| 244 | |||
| 245 | /* Copy crashdump meta and table */ | ||
| 246 | dump_meta = dump_info->dump_addr; | ||
| 247 | dump_meta->magic = QAIC_SSR_DUMP_V1_MAGIC; | ||
| 248 | dump_meta->version = QAIC_SSR_DUMP_V1_VER; | ||
| 249 | dump_meta->size = dump_info->dump_sz; | ||
| 250 | dump_meta->tbl_len = dump_info->tbl_len; | ||
| 251 | memcpy(dump_info->dump_addr + sizeof(*dump_meta), dump_info->tbl_addr, dump_info->tbl_len); | ||
| 252 | /* Offset by crashdump meta and table (copied above) */ | ||
| 253 | dump_info->dump_off = dump_info->tbl_len + sizeof(*dump_meta); | ||
| 254 | |||
| 255 | return 0; | ||
| 256 | } | ||
| 257 | |||
| 258 | static int send_xfer_done(struct qaic_device *qdev, void *resp, u32 dbc_id) | ||
| 259 | { | ||
| 260 | struct ssr_debug_transfer_done *xfer_done; | ||
| 261 | int ret; | ||
| 262 | |||
| 263 | xfer_done = kmalloc(sizeof(*xfer_done), GFP_KERNEL); | ||
| 264 | if (!xfer_done) { | ||
| 265 | ret = -ENOMEM; | ||
| 266 | goto out; | ||
| 267 | } | ||
| 268 | |||
| 269 | ret = mhi_queue_buf(qdev->ssr_ch, DMA_FROM_DEVICE, resp, SSR_RESP_MSG_SZ, MHI_EOT); | ||
| 270 | if (ret) | ||
| 271 | goto free_xfer_done; | ||
| 272 | |||
| 273 | xfer_done->hdr.cmd = cpu_to_le32(DEBUG_TRANSFER_DONE); | ||
| 274 | xfer_done->hdr.len = cpu_to_le32(sizeof(*xfer_done)); | ||
| 275 | xfer_done->hdr.dbc_id = cpu_to_le32(dbc_id); | ||
| 276 | |||
| 277 | ret = mhi_queue_buf(qdev->ssr_ch, DMA_TO_DEVICE, xfer_done, sizeof(*xfer_done), MHI_EOT); | ||
| 278 | if (ret) | ||
| 279 | goto free_xfer_done; | ||
| 280 | |||
| 281 | return 0; | ||
| 282 | |||
| 283 | free_xfer_done: | ||
| 284 | kfree(xfer_done); | ||
| 285 | out: | ||
| 286 | return ret; | ||
| 287 | } | ||
| 288 | |||
| 289 | static int mem_read_req(struct qaic_device *qdev, u64 dest_addr, u64 dest_len) | ||
| 290 | { | ||
| 291 | struct ssr_crashdump *ssr_crash = qdev->ssr_mhi_buf; | ||
| 292 | struct ssr_memory_read *read_buf_req; | ||
| 293 | struct ssr_dump_info *dump_info; | ||
| 294 | int ret; | ||
| 295 | |||
| 296 | dump_info = ssr_crash->dump_info; | ||
| 297 | ret = mhi_queue_buf(qdev->ssr_ch, DMA_FROM_DEVICE, ssr_crash->data, SSR_MEM_READ_DATA_SIZE, | ||
| 298 | MHI_EOT); | ||
| 299 | if (ret) | ||
| 300 | goto out; | ||
| 301 | |||
| 302 | read_buf_req = dump_info->read_buf_req; | ||
| 303 | read_buf_req->hdr.cmd = cpu_to_le32(MEMORY_READ); | ||
| 304 | read_buf_req->hdr.len = cpu_to_le32(sizeof(*read_buf_req)); | ||
| 305 | read_buf_req->hdr.dbc_id = cpu_to_le32(qdev->ssr_dbc); | ||
| 306 | read_buf_req->addr = cpu_to_le64(dest_addr); | ||
| 307 | read_buf_req->len = cpu_to_le64(dest_len); | ||
| 308 | |||
| 309 | ret = mhi_queue_buf(qdev->ssr_ch, DMA_TO_DEVICE, read_buf_req, sizeof(*read_buf_req), | ||
| 310 | MHI_EOT); | ||
| 311 | if (!ret) | ||
| 312 | dump_info->read_buf_req_queued = true; | ||
| 313 | |||
| 314 | out: | ||
| 315 | return ret; | ||
| 316 | } | ||
| 317 | |||
| 318 | static int ssr_copy_table(struct ssr_dump_info *dump_info, void *data, u64 len) | ||
| 319 | { | ||
| 320 | if (len > dump_info->tbl_len - dump_info->tbl_off) | ||
| 321 | return -EINVAL; | ||
| 322 | |||
| 323 | memcpy(dump_info->tbl_addr + dump_info->tbl_off, data, len); | ||
| 324 | dump_info->tbl_off += len; | ||
| 325 | |||
| 326 | /* Entire table has been downloaded, alloc dump memory */ | ||
| 327 | if (dump_info->tbl_off == dump_info->tbl_len) { | ||
| 328 | dump_info->tbl_ent = dump_info->tbl_addr; | ||
| 329 | return alloc_dump(dump_info); | ||
| 330 | } | ||
| 331 | |||
| 332 | return 0; | ||
| 333 | } | ||
| 334 | |||
| 335 | static int ssr_copy_dump(struct ssr_dump_info *dump_info, void *data, u64 len) | ||
| 336 | { | ||
| 337 | struct debug_info_table *tbl_ent; | ||
| 338 | |||
| 339 | tbl_ent = dump_info->tbl_ent; | ||
| 340 | |||
| 341 | if (len > tbl_ent->len - dump_info->tbl_ent_off) | ||
| 342 | return -EINVAL; | ||
| 343 | |||
| 344 | memcpy(dump_info->dump_addr + dump_info->dump_off, data, len); | ||
| 345 | dump_info->dump_off += len; | ||
| 346 | dump_info->tbl_ent_off += len; | ||
| 347 | |||
| 348 | /* | ||
| 349 | * Current segment (a entry in table) of the crashdump is complete, | ||
| 350 | * move to next one | ||
| 351 | */ | ||
| 352 | if (tbl_ent->len == dump_info->tbl_ent_off) { | ||
| 353 | dump_info->tbl_ent++; | ||
| 354 | dump_info->tbl_ent_off = 0; | ||
| 355 | } | ||
| 356 | |||
| 357 | return 0; | ||
| 358 | } | ||
| 359 | |||
| 360 | static void ssr_dump_worker(struct work_struct *work) | ||
| 361 | { | ||
| 362 | struct ssr_crashdump *ssr_crash = container_of(work, struct ssr_crashdump, work); | ||
| 363 | struct qaic_device *qdev = ssr_crash->qdev; | ||
| 364 | struct ssr_memory_read_rsp *mem_rd_resp; | ||
| 365 | struct debug_info_table *tbl_ent; | ||
| 366 | struct ssr_dump_info *dump_info; | ||
| 367 | u64 dest_addr, dest_len; | ||
| 368 | struct _ssr_hdr *_hdr; | ||
| 369 | struct ssr_hdr hdr; | ||
| 370 | u64 data_len; | ||
| 371 | int ret; | ||
| 372 | |||
| 373 | mem_rd_resp = (struct ssr_memory_read_rsp *)ssr_crash->data; | ||
| 374 | _hdr = &mem_rd_resp->hdr; | ||
| 375 | hdr.cmd = le32_to_cpu(_hdr->cmd); | ||
| 376 | hdr.len = le32_to_cpu(_hdr->len); | ||
| 377 | hdr.dbc_id = le32_to_cpu(_hdr->dbc_id); | ||
| 378 | |||
| 379 | if (hdr.dbc_id != qdev->ssr_dbc) | ||
| 380 | goto reset_device; | ||
| 381 | |||
| 382 | dump_info = ssr_crash->dump_info; | ||
| 383 | if (!dump_info) | ||
| 384 | goto reset_device; | ||
| 385 | |||
| 386 | if (hdr.cmd != MEMORY_READ_RSP) | ||
| 387 | goto free_dump_info; | ||
| 388 | |||
| 389 | if (hdr.len > SSR_MEM_READ_DATA_SIZE) | ||
| 390 | goto free_dump_info; | ||
| 391 | |||
| 392 | data_len = hdr.len - sizeof(*mem_rd_resp); | ||
| 393 | |||
| 394 | if (dump_info->tbl_off < dump_info->tbl_len) /* Chunk belongs to table */ | ||
| 395 | ret = ssr_copy_table(dump_info, mem_rd_resp->data, data_len); | ||
| 396 | else /* Chunk belongs to crashdump */ | ||
| 397 | ret = ssr_copy_dump(dump_info, mem_rd_resp->data, data_len); | ||
| 398 | |||
| 399 | if (ret) | ||
| 400 | goto free_dump_info; | ||
| 401 | |||
| 402 | if (dump_info->tbl_off < dump_info->tbl_len) { | ||
| 403 | /* Continue downloading table */ | ||
| 404 | dest_addr = dump_info->tbl_addr_dev + dump_info->tbl_off; | ||
| 405 | dest_len = min(SSR_MEM_READ_CHUNK_SIZE, dump_info->tbl_len - dump_info->tbl_off); | ||
| 406 | ret = mem_read_req(qdev, dest_addr, dest_len); | ||
| 407 | } else if (dump_info->dump_off < dump_info->dump_sz) { | ||
| 408 | /* Continue downloading crashdump */ | ||
| 409 | tbl_ent = dump_info->tbl_ent; | ||
| 410 | dest_addr = tbl_ent->mem_base + dump_info->tbl_ent_off; | ||
| 411 | dest_len = min(SSR_MEM_READ_CHUNK_SIZE, tbl_ent->len - dump_info->tbl_ent_off); | ||
| 412 | ret = mem_read_req(qdev, dest_addr, dest_len); | ||
| 413 | } else { | ||
| 414 | /* Crashdump download complete */ | ||
| 415 | ret = send_xfer_done(qdev, dump_info->resp->data, hdr.dbc_id); | ||
| 416 | } | ||
| 417 | |||
| 418 | /* Most likely a MHI xfer has failed */ | ||
| 419 | if (ret) | ||
| 420 | goto free_dump_info; | ||
| 421 | |||
| 422 | return; | ||
| 423 | |||
| 424 | free_dump_info: | ||
| 425 | /* Free the allocated memory */ | ||
| 426 | free_ssr_dump_info(ssr_crash); | ||
| 427 | reset_device: | ||
| 428 | /* | ||
| 429 | * After subsystem crashes in device crashdump collection begins but | ||
| 430 | * something went wrong while collecting crashdump, now instead of | ||
| 431 | * handling this error we just reset the device as the best effort has | ||
| 432 | * been made | ||
| 433 | */ | ||
| 434 | mhi_soc_reset(qdev->mhi_cntrl); | ||
| 435 | } | ||
| 436 | |||
| 437 | static struct ssr_dump_info *alloc_dump_info(struct qaic_device *qdev, | ||
| 438 | struct ssr_debug_transfer_info *debug_info) | ||
| 439 | { | ||
| 440 | struct ssr_dump_info *dump_info; | ||
| 441 | int ret; | ||
| 442 | |||
| 443 | le64_to_cpus(&debug_info->tbl_len); | ||
| 444 | le64_to_cpus(&debug_info->tbl_addr); | ||
| 445 | |||
| 446 | if (debug_info->tbl_len == 0 || | ||
| 447 | debug_info->tbl_len % sizeof(struct debug_info_table) != 0) { | ||
| 448 | ret = -EINVAL; | ||
| 449 | goto out; | ||
| 450 | } | ||
| 451 | |||
| 452 | /* Allocate SSR crashdump book keeping structure */ | ||
| 453 | dump_info = kzalloc(sizeof(*dump_info), GFP_KERNEL); | ||
| 454 | if (!dump_info) { | ||
| 455 | ret = -ENOMEM; | ||
| 456 | goto out; | ||
| 457 | } | ||
| 458 | |||
| 459 | /* Buffer used to send MEMORY READ request to device via MHI */ | ||
| 460 | dump_info->read_buf_req = kzalloc(sizeof(*dump_info->read_buf_req), GFP_KERNEL); | ||
| 461 | if (!dump_info->read_buf_req) { | ||
| 462 | ret = -ENOMEM; | ||
| 463 | goto free_dump_info; | ||
| 464 | } | ||
| 465 | |||
| 466 | /* Crashdump meta table buffer */ | ||
| 467 | dump_info->tbl_addr = vzalloc(debug_info->tbl_len); | ||
| 468 | if (!dump_info->tbl_addr) { | ||
| 469 | ret = -ENOMEM; | ||
| 470 | goto free_read_buf_req; | ||
| 471 | } | ||
| 472 | |||
| 473 | dump_info->tbl_addr_dev = debug_info->tbl_addr; | ||
| 474 | dump_info->tbl_len = debug_info->tbl_len; | ||
| 475 | |||
| 476 | return dump_info; | ||
| 477 | |||
| 478 | free_read_buf_req: | ||
| 479 | kfree(dump_info->read_buf_req); | ||
| 480 | free_dump_info: | ||
| 481 | kfree(dump_info); | ||
| 482 | out: | ||
| 483 | return ERR_PTR(ret); | ||
| 484 | } | ||
| 485 | |||
| 486 | static int dbg_xfer_info_rsp(struct qaic_device *qdev, struct dma_bridge_chan *dbc, | ||
| 487 | struct ssr_debug_transfer_info *debug_info) | ||
| 488 | { | ||
| 489 | struct ssr_debug_transfer_info_rsp *debug_rsp; | ||
| 490 | struct ssr_crashdump *ssr_crash = NULL; | ||
| 491 | int ret = 0, ret2; | ||
| 492 | |||
| 493 | debug_rsp = kmalloc(sizeof(*debug_rsp), GFP_KERNEL); | ||
| 494 | if (!debug_rsp) | ||
| 495 | return -ENOMEM; | ||
| 496 | |||
| 497 | if (!qdev->ssr_mhi_buf) { | ||
| 498 | ret = -ENOMEM; | ||
| 499 | goto send_rsp; | ||
| 500 | } | ||
| 501 | |||
| 502 | if (dbc->state != DBC_STATE_BEFORE_POWER_UP) { | ||
| 503 | ret = -EINVAL; | ||
| 504 | goto send_rsp; | ||
| 505 | } | ||
| 506 | |||
| 507 | ssr_crash = qdev->ssr_mhi_buf; | ||
| 508 | ssr_crash->dump_info = alloc_dump_info(qdev, debug_info); | ||
| 509 | if (IS_ERR(ssr_crash->dump_info)) { | ||
| 510 | ret = PTR_ERR(ssr_crash->dump_info); | ||
| 511 | ssr_crash->dump_info = NULL; | ||
| 512 | } | ||
| 513 | |||
| 514 | send_rsp: | ||
| 515 | debug_rsp->hdr.cmd = cpu_to_le32(DEBUG_TRANSFER_INFO_RSP); | ||
| 516 | debug_rsp->hdr.len = cpu_to_le32(sizeof(*debug_rsp)); | ||
| 517 | debug_rsp->hdr.dbc_id = cpu_to_le32(dbc->id); | ||
| 518 | /* | ||
| 519 | * 0 = Return an ACK confirming the host is ready to download crashdump | ||
| 520 | * 1 = Return an NACK confirming the host is not ready to download crashdump | ||
| 521 | */ | ||
| 522 | debug_rsp->ret = cpu_to_le32(ret ? 1 : 0); | ||
| 523 | |||
| 524 | ret2 = mhi_queue_buf(qdev->ssr_ch, DMA_TO_DEVICE, debug_rsp, sizeof(*debug_rsp), MHI_EOT); | ||
| 525 | if (ret2) { | ||
| 526 | free_ssr_dump_info(ssr_crash); | ||
| 527 | kfree(debug_rsp); | ||
| 528 | return ret2; | ||
| 529 | } | ||
| 530 | |||
| 531 | return ret; | ||
| 532 | } | ||
| 533 | |||
| 534 | static void dbg_xfer_done_rsp(struct qaic_device *qdev, struct dma_bridge_chan *dbc, | ||
| 535 | struct ssr_debug_transfer_done_rsp *xfer_rsp) | ||
| 536 | { | ||
| 537 | struct ssr_crashdump *ssr_crash = qdev->ssr_mhi_buf; | ||
| 538 | u32 status = le32_to_cpu(xfer_rsp->ret); | ||
| 539 | struct device *dev = &qdev->pdev->dev; | ||
| 540 | struct ssr_dump_info *dump_info; | ||
| 541 | |||
| 542 | dump_info = ssr_crash->dump_info; | ||
| 543 | if (!dump_info) | ||
| 544 | return; | ||
| 545 | |||
| 546 | if (status) { | ||
| 547 | free_ssr_dump_info(ssr_crash); | ||
| 548 | return; | ||
| 549 | } | ||
| 550 | |||
| 551 | dev_coredumpv(dev, dump_info->dump_addr, dump_info->dump_sz, GFP_KERNEL); | ||
| 552 | /* dev_coredumpv will free dump_info->dump_addr */ | ||
| 553 | dump_info->dump_addr = NULL; | ||
| 554 | free_ssr_dump_info(ssr_crash); | ||
| 555 | } | ||
| 556 | |||
| 557 | static void ssr_worker(struct work_struct *work) | ||
| 558 | { | ||
| 559 | struct ssr_resp *resp = container_of(work, struct ssr_resp, work); | ||
| 560 | struct ssr_hdr *hdr = (struct ssr_hdr *)resp->data; | ||
| 561 | struct ssr_dump_info *dump_info = NULL; | ||
| 562 | struct qaic_device *qdev = resp->qdev; | ||
| 563 | struct ssr_crashdump *ssr_crash; | ||
| 564 | struct ssr_event_rsp *event_rsp; | ||
| 565 | struct dma_bridge_chan *dbc; | ||
| 566 | struct ssr_event *event; | ||
| 567 | u32 ssr_event_ack; | ||
| 568 | int ret; | ||
| 569 | |||
| 570 | le32_to_cpus(&hdr->cmd); | ||
| 571 | le32_to_cpus(&hdr->len); | ||
| 572 | le32_to_cpus(&hdr->dbc_id); | ||
| 573 | |||
| 574 | if (hdr->len > SSR_RESP_MSG_SZ) | ||
| 575 | goto out; | ||
| 576 | |||
| 577 | if (hdr->dbc_id >= qdev->num_dbc) | ||
| 578 | goto out; | ||
| 579 | |||
| 580 | dbc = &qdev->dbc[hdr->dbc_id]; | ||
| 581 | |||
| 582 | switch (hdr->cmd) { | ||
| 583 | case DEBUG_TRANSFER_INFO: | ||
| 584 | ret = dbg_xfer_info_rsp(qdev, dbc, (struct ssr_debug_transfer_info *)resp->data); | ||
| 585 | if (ret) | ||
| 586 | break; | ||
| 587 | |||
| 588 | ssr_crash = qdev->ssr_mhi_buf; | ||
| 589 | dump_info = ssr_crash->dump_info; | ||
| 590 | dump_info->dbc = dbc; | ||
| 591 | dump_info->resp = resp; | ||
| 592 | |||
| 593 | /* Start by downloading debug table */ | ||
| 594 | ret = mem_read_req(qdev, dump_info->tbl_addr_dev, | ||
| 595 | min(dump_info->tbl_len, SSR_MEM_READ_CHUNK_SIZE)); | ||
| 596 | if (ret) { | ||
| 597 | free_ssr_dump_info(ssr_crash); | ||
| 598 | break; | ||
| 599 | } | ||
| 600 | |||
| 601 | /* | ||
| 602 | * Till now everything went fine, which means that we will be | ||
| 603 | * collecting crashdump chunk by chunk. Do not queue a response | ||
| 604 | * buffer for SSR cmds till the crashdump is complete. | ||
| 605 | */ | ||
| 606 | return; | ||
| 607 | case SSR_EVENT: | ||
| 608 | event = (struct ssr_event *)hdr; | ||
| 609 | le32_to_cpus(&event->event); | ||
| 610 | ssr_event_ack = event->event; | ||
| 611 | ssr_crash = qdev->ssr_mhi_buf; | ||
| 612 | |||
| 613 | switch (event->event) { | ||
| 614 | case BEFORE_SHUTDOWN: | ||
| 615 | set_dbc_state(qdev, hdr->dbc_id, DBC_STATE_BEFORE_SHUTDOWN); | ||
| 616 | qaic_dbc_enter_ssr(qdev, hdr->dbc_id); | ||
| 617 | break; | ||
| 618 | case AFTER_SHUTDOWN: | ||
| 619 | set_dbc_state(qdev, hdr->dbc_id, DBC_STATE_AFTER_SHUTDOWN); | ||
| 620 | break; | ||
| 621 | case BEFORE_POWER_UP: | ||
| 622 | set_dbc_state(qdev, hdr->dbc_id, DBC_STATE_BEFORE_POWER_UP); | ||
| 623 | break; | ||
| 624 | case AFTER_POWER_UP: | ||
| 625 | /* | ||
| 626 | * If dump info is a non NULL value it means that we | ||
| 627 | * have received this SSR event while downloading a | ||
| 628 | * crashdump for this DBC is still in progress. NACK | ||
| 629 | * the SSR event | ||
| 630 | */ | ||
| 631 | if (ssr_crash && ssr_crash->dump_info) { | ||
| 632 | free_ssr_dump_info(ssr_crash); | ||
| 633 | ssr_event_ack = SSR_EVENT_NACK; | ||
| 634 | break; | ||
| 635 | } | ||
| 636 | |||
| 637 | set_dbc_state(qdev, hdr->dbc_id, DBC_STATE_AFTER_POWER_UP); | ||
| 638 | break; | ||
| 639 | default: | ||
| 640 | break; | ||
| 641 | } | ||
| 642 | |||
| 643 | event_rsp = kmalloc(sizeof(*event_rsp), GFP_KERNEL); | ||
| 644 | if (!event_rsp) | ||
| 645 | break; | ||
| 646 | |||
| 647 | event_rsp->hdr.cmd = cpu_to_le32(SSR_EVENT_RSP); | ||
| 648 | event_rsp->hdr.len = cpu_to_le32(sizeof(*event_rsp)); | ||
| 649 | event_rsp->hdr.dbc_id = cpu_to_le32(hdr->dbc_id); | ||
| 650 | event_rsp->event = cpu_to_le32(ssr_event_ack); | ||
| 651 | |||
| 652 | ret = mhi_queue_buf(qdev->ssr_ch, DMA_TO_DEVICE, event_rsp, sizeof(*event_rsp), | ||
| 653 | MHI_EOT); | ||
| 654 | if (ret) | ||
| 655 | kfree(event_rsp); | ||
| 656 | |||
| 657 | if (event->event == AFTER_POWER_UP && ssr_event_ack != SSR_EVENT_NACK) { | ||
| 658 | qaic_dbc_exit_ssr(qdev); | ||
| 659 | set_dbc_state(qdev, hdr->dbc_id, DBC_STATE_IDLE); | ||
| 660 | } | ||
| 661 | |||
| 662 | break; | ||
| 663 | case DEBUG_TRANSFER_DONE_RSP: | ||
| 664 | dbg_xfer_done_rsp(qdev, dbc, (struct ssr_debug_transfer_done_rsp *)hdr); | ||
| 665 | break; | ||
| 666 | default: | ||
| 667 | break; | ||
| 668 | } | ||
| 669 | |||
| 670 | out: | ||
| 671 | ret = mhi_queue_buf(qdev->ssr_ch, DMA_FROM_DEVICE, resp->data, SSR_RESP_MSG_SZ, MHI_EOT); | ||
| 672 | if (ret) | ||
| 673 | kfree(resp); | ||
| 674 | } | ||
| 675 | |||
| 676 | static int qaic_ssr_mhi_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id) | ||
| 677 | { | ||
| 678 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(mhi_dev->mhi_cntrl->cntrl_dev)); | ||
| 679 | struct ssr_resp *resp; | ||
| 680 | int ret; | ||
| 681 | |||
| 682 | ret = mhi_prepare_for_transfer(mhi_dev); | ||
| 683 | if (ret) | ||
| 684 | return ret; | ||
| 685 | |||
| 686 | resp = kzalloc(sizeof(*resp) + SSR_RESP_MSG_SZ, GFP_KERNEL); | ||
| 687 | if (!resp) { | ||
| 688 | mhi_unprepare_from_transfer(mhi_dev); | ||
| 689 | return -ENOMEM; | ||
| 690 | } | ||
| 691 | |||
| 692 | resp->qdev = qdev; | ||
| 693 | INIT_WORK(&resp->work, ssr_worker); | ||
| 694 | |||
| 695 | ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, resp->data, SSR_RESP_MSG_SZ, MHI_EOT); | ||
| 696 | if (ret) { | ||
| 697 | kfree(resp); | ||
| 698 | mhi_unprepare_from_transfer(mhi_dev); | ||
| 699 | return ret; | ||
| 700 | } | ||
| 701 | |||
| 702 | dev_set_drvdata(&mhi_dev->dev, qdev); | ||
| 703 | qdev->ssr_ch = mhi_dev; | ||
| 704 | |||
| 705 | return 0; | ||
| 706 | } | ||
| 707 | |||
| 708 | static void qaic_ssr_mhi_remove(struct mhi_device *mhi_dev) | ||
| 709 | { | ||
| 710 | struct qaic_device *qdev; | ||
| 711 | |||
| 712 | qdev = dev_get_drvdata(&mhi_dev->dev); | ||
| 713 | mhi_unprepare_from_transfer(qdev->ssr_ch); | ||
| 714 | qdev->ssr_ch = NULL; | ||
| 715 | } | ||
| 716 | |||
| 717 | static void qaic_ssr_mhi_ul_xfer_cb(struct mhi_device *mhi_dev, struct mhi_result *mhi_result) | ||
| 718 | { | ||
| 719 | struct qaic_device *qdev = dev_get_drvdata(&mhi_dev->dev); | ||
| 720 | struct ssr_crashdump *ssr_crash = qdev->ssr_mhi_buf; | ||
| 721 | struct _ssr_hdr *hdr = mhi_result->buf_addr; | ||
| 722 | struct ssr_dump_info *dump_info; | ||
| 723 | |||
| 724 | if (mhi_result->transaction_status) { | ||
| 725 | kfree(mhi_result->buf_addr); | ||
| 726 | return; | ||
| 727 | } | ||
| 728 | |||
| 729 | /* | ||
| 730 | * MEMORY READ is used to download crashdump. And crashdump is | ||
| 731 | * downloaded chunk by chunk in a series of MEMORY READ SSR commands. | ||
| 732 | * Hence to avoid too many kmalloc() and kfree() of the same MEMORY READ | ||
| 733 | * request buffer, we allocate only one such buffer and free it only | ||
| 734 | * once. | ||
| 735 | */ | ||
| 736 | if (le32_to_cpu(hdr->cmd) == MEMORY_READ) { | ||
| 737 | dump_info = ssr_crash->dump_info; | ||
| 738 | if (dump_info) { | ||
| 739 | dump_info->read_buf_req_queued = false; | ||
| 740 | return; | ||
| 741 | } | ||
| 742 | } | ||
| 743 | |||
| 744 | kfree(mhi_result->buf_addr); | ||
| 745 | } | ||
| 746 | |||
| 747 | static void qaic_ssr_mhi_dl_xfer_cb(struct mhi_device *mhi_dev, struct mhi_result *mhi_result) | ||
| 748 | { | ||
| 749 | struct ssr_resp *resp = container_of(mhi_result->buf_addr, struct ssr_resp, data); | ||
| 750 | struct qaic_device *qdev = dev_get_drvdata(&mhi_dev->dev); | ||
| 751 | struct ssr_crashdump *ssr_crash = qdev->ssr_mhi_buf; | ||
| 752 | bool memory_read_rsp = false; | ||
| 753 | |||
| 754 | if (ssr_crash && ssr_crash->data == mhi_result->buf_addr) | ||
| 755 | memory_read_rsp = true; | ||
| 756 | |||
| 757 | if (mhi_result->transaction_status) { | ||
| 758 | /* Do not free SSR crashdump buffer as it allocated via managed APIs */ | ||
| 759 | if (!memory_read_rsp) | ||
| 760 | kfree(resp); | ||
| 761 | return; | ||
| 762 | } | ||
| 763 | |||
| 764 | if (memory_read_rsp) | ||
| 765 | queue_work(qdev->ssr_wq, &ssr_crash->work); | ||
| 766 | else | ||
| 767 | queue_work(qdev->ssr_wq, &resp->work); | ||
| 768 | } | ||
| 769 | |||
| 770 | static const struct mhi_device_id qaic_ssr_mhi_match_table[] = { | ||
| 771 | { .chan = "QAIC_SSR", }, | ||
| 772 | {}, | ||
| 773 | }; | ||
| 774 | |||
| 775 | static struct mhi_driver qaic_ssr_mhi_driver = { | ||
| 776 | .id_table = qaic_ssr_mhi_match_table, | ||
| 777 | .remove = qaic_ssr_mhi_remove, | ||
| 778 | .probe = qaic_ssr_mhi_probe, | ||
| 779 | .ul_xfer_cb = qaic_ssr_mhi_ul_xfer_cb, | ||
| 780 | .dl_xfer_cb = qaic_ssr_mhi_dl_xfer_cb, | ||
| 781 | .driver = { | ||
| 782 | .name = "qaic_ssr", | ||
| 783 | }, | ||
| 784 | }; | ||
| 785 | |||
| 786 | int qaic_ssr_init(struct qaic_device *qdev, struct drm_device *drm) | ||
| 787 | { | ||
| 788 | struct ssr_crashdump *ssr_crash; | ||
| 789 | |||
| 790 | qdev->ssr_dbc = QAIC_SSR_DBC_SENTINEL; | ||
| 791 | |||
| 792 | /* | ||
| 793 | * Device requests only one SSR at a time. So allocating only one | ||
| 794 | * buffer to download crashdump is good enough. | ||
| 795 | */ | ||
| 796 | ssr_crash = drmm_kzalloc(drm, SSR_MHI_BUF_SIZE, GFP_KERNEL); | ||
| 797 | if (!ssr_crash) | ||
| 798 | return -ENOMEM; | ||
| 799 | |||
| 800 | ssr_crash->qdev = qdev; | ||
| 801 | INIT_WORK(&ssr_crash->work, ssr_dump_worker); | ||
| 802 | qdev->ssr_mhi_buf = ssr_crash; | ||
| 803 | |||
| 804 | return 0; | ||
| 805 | } | ||
| 806 | |||
| 807 | int qaic_ssr_register(void) | ||
| 808 | { | ||
| 809 | return mhi_driver_register(&qaic_ssr_mhi_driver); | ||
| 810 | } | ||
| 811 | |||
| 812 | void qaic_ssr_unregister(void) | ||
| 813 | { | ||
| 814 | mhi_driver_unregister(&qaic_ssr_mhi_driver); | ||
| 815 | } | ||
| @@ -0,0 +1,17 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | * | ||
| 3 | * Copyright (c) 2020, The Linux Foundation. All rights reserved. | ||
| 4 | * Copyright (c) 2021, 2024 Qualcomm Innovation Center, Inc. All rights reserved. | ||
| 5 | */ | ||
| 6 | |||
| 7 | #ifndef __QAIC_SSR_H__ | ||
| 8 | #define __QAIC_SSR_H__ | ||
| 9 | |||
| 10 | struct drm_device; | ||
| 11 | struct qaic_device; | ||
| 12 | |||
| 13 | int qaic_ssr_register(void); | ||
| 14 | void qaic_ssr_unregister(void); | ||
| 15 | void qaic_clean_up_ssr(struct qaic_device *qdev); | ||
| 16 | int qaic_ssr_init(struct qaic_device *qdev, struct drm_device *drm); | ||
| 17 | #endif /* __QAIC_SSR_H__ */ | ||
| @@ -0,0 +1,109 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | |||
| 3 | /* Copyright (c) 2020-2025, The Linux Foundation. All rights reserved. */ | ||
| 4 | |||
| 5 | #include <drm/drm_file.h> | ||
| 6 | #include <drm/drm_managed.h> | ||
| 7 | #include <linux/device.h> | ||
| 8 | #include <linux/kernel.h> | ||
| 9 | #include <linux/kobject.h> | ||
| 10 | #include <linux/mutex.h> | ||
| 11 | #include <linux/sysfs.h> | ||
| 12 | |||
| 13 | #include "qaic.h" | ||
| 14 | |||
| 15 | #define NAME_LEN 14 | ||
| 16 | |||
| 17 | struct dbc_attribute { | ||
| 18 | struct device_attribute dev_attr; | ||
| 19 | u32 dbc_id; | ||
| 20 | char name[NAME_LEN]; | ||
| 21 | }; | ||
| 22 | |||
| 23 | static ssize_t dbc_state_show(struct device *dev, struct device_attribute *a, char *buf) | ||
| 24 | { | ||
| 25 | struct dbc_attribute *dbc_attr = container_of(a, struct dbc_attribute, dev_attr); | ||
| 26 | struct drm_minor *minor = dev_get_drvdata(dev); | ||
| 27 | struct qaic_device *qdev; | ||
| 28 | |||
| 29 | qdev = to_qaic_device(minor->dev); | ||
| 30 | return sysfs_emit(buf, "%d\n", qdev->dbc[dbc_attr->dbc_id].state); | ||
| 31 | } | ||
| 32 | |||
| 33 | void set_dbc_state(struct qaic_device *qdev, u32 dbc_id, unsigned int state) | ||
| 34 | { | ||
| 35 | struct device *kdev = to_accel_kdev(qdev->qddev); | ||
| 36 | char *envp[3] = {}; | ||
| 37 | char state_str[16]; | ||
| 38 | char id_str[12]; | ||
| 39 | |||
| 40 | envp[0] = id_str; | ||
| 41 | envp[1] = state_str; | ||
| 42 | |||
| 43 | if (state >= DBC_STATE_MAX) | ||
| 44 | return; | ||
| 45 | if (dbc_id >= qdev->num_dbc) | ||
| 46 | return; | ||
| 47 | if (state == qdev->dbc[dbc_id].state) | ||
| 48 | return; | ||
| 49 | |||
| 50 | scnprintf(id_str, ARRAY_SIZE(id_str), "DBC_ID=%d", dbc_id); | ||
| 51 | scnprintf(state_str, ARRAY_SIZE(state_str), "DBC_STATE=%d", state); | ||
| 52 | |||
| 53 | qdev->dbc[dbc_id].state = state; | ||
| 54 | kobject_uevent_env(&kdev->kobj, KOBJ_CHANGE, envp); | ||
| 55 | } | ||
| 56 | |||
| 57 | int qaic_sysfs_init(struct qaic_drm_device *qddev) | ||
| 58 | { | ||
| 59 | struct device *kdev = to_accel_kdev(qddev); | ||
| 60 | struct drm_device *drm = to_drm(qddev); | ||
| 61 | u32 num_dbc = qddev->qdev->num_dbc; | ||
| 62 | struct dbc_attribute *dbc_attrs; | ||
| 63 | int i, ret; | ||
| 64 | |||
| 65 | dbc_attrs = drmm_kcalloc(drm, num_dbc, sizeof(*dbc_attrs), GFP_KERNEL); | ||
| 66 | if (!dbc_attrs) | ||
| 67 | return -ENOMEM; | ||
| 68 | |||
| 69 | for (i = 0; i < num_dbc; ++i) { | ||
| 70 | struct dbc_attribute *dbc_attr = &dbc_attrs[i]; | ||
| 71 | |||
| 72 | sysfs_attr_init(&dbc_attr->dev_attr.attr); | ||
| 73 | dbc_attr->dbc_id = i; | ||
| 74 | scnprintf(dbc_attr->name, NAME_LEN, "dbc%d_state", i); | ||
| 75 | dbc_attr->dev_attr.attr.name = dbc_attr->name; | ||
| 76 | dbc_attr->dev_attr.attr.mode = 0444; | ||
| 77 | dbc_attr->dev_attr.show = dbc_state_show; | ||
| 78 | ret = sysfs_create_file(&kdev->kobj, &dbc_attr->dev_attr.attr); | ||
| 79 | if (ret) { | ||
| 80 | int j; | ||
| 81 | |||
| 82 | for (j = 0; j < i; ++j) { | ||
| 83 | dbc_attr = &dbc_attrs[j]; | ||
| 84 | sysfs_remove_file(&kdev->kobj, &dbc_attr->dev_attr.attr); | ||
| 85 | } | ||
| 86 | drmm_kfree(drm, dbc_attrs); | ||
| 87 | return ret; | ||
| 88 | } | ||
| 89 | } | ||
| 90 | |||
| 91 | qddev->sysfs_attrs = dbc_attrs; | ||
| 92 | return 0; | ||
| 93 | } | ||
| 94 | |||
| 95 | void qaic_sysfs_remove(struct qaic_drm_device *qddev) | ||
| 96 | { | ||
| 97 | struct dbc_attribute *dbc_attrs = qddev->sysfs_attrs; | ||
| 98 | struct device *kdev = to_accel_kdev(qddev); | ||
| 99 | u32 num_dbc = qddev->qdev->num_dbc; | ||
| 100 | int i; | ||
| 101 | |||
| 102 | if (!dbc_attrs) | ||
| 103 | return; | ||
| 104 | |||
| 105 | qddev->sysfs_attrs = NULL; | ||
| 106 | for (i = 0; i < num_dbc; ++i) | ||
| 107 | sysfs_remove_file(&kdev->kobj, &dbc_attrs[i].dev_attr.attr); | ||
| 108 | drmm_kfree(to_drm(qddev), dbc_attrs); | ||
| 109 | } | ||
| @@ -171,6 +171,13 @@ mod_timer: · 170 unmodified lines | |||
| 171 | dev_err(mqtsdev->dev, "%s mod_timer error:%d\n", __func__, ret); | 171 | dev_err(mqtsdev->dev, "%s mod_timer error:%d\n", __func__, ret); |
| 172 | } | 172 | } |
| 173 | 173 | ||
| 174 | void qaic_mqts_ch_stop_timer(struct mhi_device *mhi_dev) | ||
| 175 | { | ||
| 176 | struct mqts_dev *mqtsdev = dev_get_drvdata(&mhi_dev->dev); | ||
| 177 | |||
| 178 | timer_delete_sync(&mqtsdev->timer); | ||
| 179 | } | ||
| 180 | |||
| 174 | static int qaic_timesync_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id) | 181 | static int qaic_timesync_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id) |
| 175 | { | 182 | { |
| 176 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(mhi_dev->mhi_cntrl->cntrl_dev)); | 183 | struct qaic_device *qdev = pci_get_drvdata(to_pci_dev(mhi_dev->mhi_cntrl->cntrl_dev)); |
| @@ -206,6 +213,7 @@ static int qaic_timesync_probe(struct mhi_device *mhi_dev, const struct mhi_devi · 29 unmodified lines | |||
| 206 | timer->expires = jiffies + msecs_to_jiffies(timesync_delay_ms); | 213 | timer->expires = jiffies + msecs_to_jiffies(timesync_delay_ms); |
| 207 | add_timer(timer); | 214 | add_timer(timer); |
| 208 | dev_set_drvdata(&mhi_dev->dev, mqtsdev); | 215 | dev_set_drvdata(&mhi_dev->dev, mqtsdev); |
| 216 | qdev->mqts_ch = mhi_dev; | ||
| 209 | 217 | ||
| 210 | return 0; | 218 | return 0; |
| 211 | 219 | ||
| @@ -221,6 +229,7 @@ static void qaic_timesync_remove(struct mhi_device *mhi_dev) · 9 unmodified lines | |||
| 221 | { | 229 | { |
| 222 | struct mqts_dev *mqtsdev = dev_get_drvdata(&mhi_dev->dev); | 230 | struct mqts_dev *mqtsdev = dev_get_drvdata(&mhi_dev->dev); |
| 223 | 231 | ||
| 232 | mqtsdev->qdev->mqts_ch = NULL; | ||
| 224 | timer_delete_sync(&mqtsdev->timer); | 233 | timer_delete_sync(&mqtsdev->timer); |
| 225 | mhi_unprepare_from_transfer(mqtsdev->mhi_dev); | 234 | mhi_unprepare_from_transfer(mqtsdev->mhi_dev); |
| 226 | kfree(mqtsdev->sync_msg); | 235 | kfree(mqtsdev->sync_msg); |
| @@ -6,6 +6,9 @@ · 5 unmodified lines | |||
| 6 | #ifndef __QAIC_TIMESYNC_H__ | 6 | #ifndef __QAIC_TIMESYNC_H__ |
| 7 | #define __QAIC_TIMESYNC_H__ | 7 | #define __QAIC_TIMESYNC_H__ |
| 8 | 8 | ||
| 9 | #include <linux/mhi.h> | ||
| 10 | |||
| 9 | int qaic_timesync_init(void); | 11 | int qaic_timesync_init(void); |
| 10 | void qaic_timesync_deinit(void); | 12 | void qaic_timesync_deinit(void); |
| 13 | void qaic_mqts_ch_stop_timer(struct mhi_device *mhi_dev); | ||
| 11 | #endif /* __QAIC_TIMESYNC_H__ */ | 14 | #endif /* __QAIC_TIMESYNC_H__ */ |
| @@ -159,6 +159,7 @@ struct sahara_context { · 158 unmodified lines | |||
| 159 | struct sahara_packet *rx; | 159 | struct sahara_packet *rx; |
| 160 | struct work_struct fw_work; | 160 | struct work_struct fw_work; |
| 161 | struct work_struct dump_work; | 161 | struct work_struct dump_work; |
| 162 | struct work_struct read_data_work; | ||
| 162 | struct mhi_device *mhi_dev; | 163 | struct mhi_device *mhi_dev; |
| 163 | const char * const *image_table; | 164 | const char * const *image_table; |
| 164 | u32 table_size; | 165 | u32 table_size; |
| @@ -174,7 +175,10 @@ struct sahara_context { · 9 unmodified lines | |||
| 174 | u64 dump_image_offset; | 175 | u64 dump_image_offset; |
| 175 | void *mem_dump_freespace; | 176 | void *mem_dump_freespace; |
| 176 | u64 dump_images_left; | 177 | u64 dump_images_left; |
| 178 | u32 read_data_offset; | ||
| 179 | u32 read_data_length; | ||
| 177 | bool is_mem_dump_mode; | 180 | bool is_mem_dump_mode; |
| 181 | bool non_streaming; | ||
| 178 | }; | 182 | }; |
| 179 | 183 | ||
| 180 | static const char * const aic100_image_table[] = { | 184 | static const char * const aic100_image_table[] = { |
| @@ -194,6 +198,7 @@ static const char * const aic200_image_table[] = { · 13 unmodified lines | |||
| 194 | [23] = "qcom/aic200/aop.mbn", | 198 | [23] = "qcom/aic200/aop.mbn", |
| 195 | [32] = "qcom/aic200/tz.mbn", | 199 | [32] = "qcom/aic200/tz.mbn", |
| 196 | [33] = "qcom/aic200/hypvm.mbn", | 200 | [33] = "qcom/aic200/hypvm.mbn", |
| 201 | [38] = "qcom/aic200/xbl_config.elf", | ||
| 197 | [39] = "qcom/aic200/aic200_abl.elf", | 202 | [39] = "qcom/aic200/aic200_abl.elf", |
| 198 | [40] = "qcom/aic200/apdp.mbn", | 203 | [40] = "qcom/aic200/apdp.mbn", |
| 199 | [41] = "qcom/aic200/devcfg.mbn", | 204 | [41] = "qcom/aic200/devcfg.mbn", |
| @@ -202,6 +207,7 @@ static const char * const aic200_image_table[] = { · 2 unmodified lines | |||
| 202 | [49] = "qcom/aic200/shrm.elf", | 207 | [49] = "qcom/aic200/shrm.elf", |
| 203 | [50] = "qcom/aic200/cpucp.elf", | 208 | [50] = "qcom/aic200/cpucp.elf", |
| 204 | [51] = "qcom/aic200/aop_devcfg.mbn", | 209 | [51] = "qcom/aic200/aop_devcfg.mbn", |
| 210 | [54] = "qcom/aic200/qupv3fw.elf", | ||
| 205 | [57] = "qcom/aic200/cpucp_dtbs.elf", | 211 | [57] = "qcom/aic200/cpucp_dtbs.elf", |
| 206 | [62] = "qcom/aic200/uefi_dtbs.elf", | 212 | [62] = "qcom/aic200/uefi_dtbs.elf", |
| 207 | [63] = "qcom/aic200/xbl_ac_config.mbn", | 213 | [63] = "qcom/aic200/xbl_ac_config.mbn", |
| @@ -213,9 +219,15 @@ static const char * const aic200_image_table[] = { · 5 unmodified lines | |||
| 213 | [69] = "qcom/aic200/dcd.mbn", | 219 | [69] = "qcom/aic200/dcd.mbn", |
| 214 | [73] = "qcom/aic200/gearvm.mbn", | 220 | [73] = "qcom/aic200/gearvm.mbn", |
| 215 | [74] = "qcom/aic200/sti.bin", | 221 | [74] = "qcom/aic200/sti.bin", |
| 216 | [75] = "qcom/aic200/pvs.bin", | 222 | [76] = "qcom/aic200/tz_qti_config.mbn", |
| 223 | [78] = "qcom/aic200/pvs.bin", | ||
| 217 | }; | 224 | }; |
| 218 | 225 | ||
| 226 | static bool is_streaming(struct sahara_context *context) | ||
| 227 | { | ||
| 228 | return !context->non_streaming; | ||
| 229 | } | ||
| 230 | |||
| 219 | static int sahara_find_image(struct sahara_context *context, u32 image_id) | 231 | static int sahara_find_image(struct sahara_context *context, u32 image_id) |
| 220 | { | 232 | { |
| 221 | int ret; | 233 | int ret; |
| @@ -265,6 +277,8 @@ static void sahara_send_reset(struct sahara_context *context) · 43 unmodified lines | |||
| 265 | int ret; | 277 | int ret; |
| 266 | 278 | ||
| 267 | context->is_mem_dump_mode = false; | 279 | context->is_mem_dump_mode = false; |
| 280 | context->read_data_offset = 0; | ||
| 281 | context->read_data_length = 0; | ||
| 268 | 282 | ||
| 269 | context->tx[0]->cmd = cpu_to_le32(SAHARA_RESET_CMD); | 283 | context->tx[0]->cmd = cpu_to_le32(SAHARA_RESET_CMD); |
| 270 | context->tx[0]->length = cpu_to_le32(SAHARA_RESET_LENGTH); | 284 | context->tx[0]->length = cpu_to_le32(SAHARA_RESET_LENGTH); |
| @@ -319,9 +333,39 @@ static void sahara_hello(struct sahara_context *context) · 48 unmodified lines | |||
| 319 | dev_err(&context->mhi_dev->dev, "Unable to send hello response %d\n", ret); | 333 | dev_err(&context->mhi_dev->dev, "Unable to send hello response %d\n", ret); |
| 320 | } | 334 | } |
| 321 | 335 | ||
| 336 | static int read_data_helper(struct sahara_context *context, int buf_index) | ||
| 337 | { | ||
| 338 | enum mhi_flags mhi_flag; | ||
| 339 | u32 pkt_data_len; | ||
| 340 | int ret; | ||
| 341 | |||
| 342 | pkt_data_len = min(context->read_data_length, SAHARA_PACKET_MAX_SIZE); | ||
| 343 | |||
| 344 | memcpy(context->tx[buf_index], | ||
| 345 | &context->firmware->data[context->read_data_offset], | ||
| 346 | pkt_data_len); | ||
| 347 | |||
| 348 | context->read_data_offset += pkt_data_len; | ||
| 349 | context->read_data_length -= pkt_data_len; | ||
| 350 | |||
| 351 | if (is_streaming(context) || !context->read_data_length) | ||
| 352 | mhi_flag = MHI_EOT; | ||
| 353 | else | ||
| 354 | mhi_flag = MHI_CHAIN; | ||
| 355 | |||
| 356 | ret = mhi_queue_buf(context->mhi_dev, DMA_TO_DEVICE, | ||
| 357 | context->tx[buf_index], pkt_data_len, mhi_flag); | ||
| 358 | if (ret) { | ||
| 359 | dev_err(&context->mhi_dev->dev, "Unable to send read_data response %d\n", ret); | ||
| 360 | return ret; | ||
| 361 | } | ||
| 362 | |||
| 363 | return 0; | ||
| 364 | } | ||
| 365 | |||
| 322 | static void sahara_read_data(struct sahara_context *context) | 366 | static void sahara_read_data(struct sahara_context *context) |
| 323 | { | 367 | { |
| 324 | u32 image_id, data_offset, data_len, pkt_data_len; | 368 | u32 image_id, data_offset, data_len; |
| 325 | int ret; | 369 | int ret; |
| 326 | int i; | 370 | int i; |
| 327 | 371 | ||
| @@ -357,7 +401,7 @@ static void sahara_read_data(struct sahara_context *context) · 29 unmodified lines | |||
| 357 | * and is not needed here on error. | 401 | * and is not needed here on error. |
| 358 | */ | 402 | */ |
| 359 | 403 | ||
| 360 | if (data_len > SAHARA_TRANSFER_MAX_SIZE) { | 404 | if (context->non_streaming && data_len > SAHARA_TRANSFER_MAX_SIZE) { |
| 361 | dev_err(&context->mhi_dev->dev, "Malformed read_data packet - data len %d exceeds max xfer size %d\n", | 405 | dev_err(&context->mhi_dev->dev, "Malformed read_data packet - data len %d exceeds max xfer size %d\n", |
| 362 | data_len, SAHARA_TRANSFER_MAX_SIZE); | 406 | data_len, SAHARA_TRANSFER_MAX_SIZE); |
| 363 | sahara_send_reset(context); | 407 | sahara_send_reset(context); |
| @@ -378,22 +422,18 @@ static void sahara_read_data(struct sahara_context *context) · 14 unmodified lines | |||
| 378 | return; | 422 | return; |
| 379 | } | 423 | } |
| 380 | 424 | ||
| 381 | for (i = 0; i < SAHARA_NUM_TX_BUF && data_len; ++i) { | 425 | context->read_data_offset = data_offset; |
| 382 | pkt_data_len = min(data_len, SAHARA_PACKET_MAX_SIZE); | 426 | context->read_data_length = data_len; |
| 383 | |||
| 384 | memcpy(context->tx[i], &context->firmware->data[data_offset], pkt_data_len); | ||
| 385 | 427 | ||
| 386 | data_offset += pkt_data_len; | 428 | if (is_streaming(context)) { |
| 387 | data_len -= pkt_data_len; | 429 | schedule_work(&context->read_data_work); |
| 430 | return; | ||
| 431 | } | ||
| 388 | 432 | ||
| 389 | ret = mhi_queue_buf(context->mhi_dev, DMA_TO_DEVICE, | 433 | for (i = 0; i < SAHARA_NUM_TX_BUF && context->read_data_length; ++i) { |
| 390 | context->tx[i], pkt_data_len, | 434 | ret = read_data_helper(context, i); |
| 391 | !data_len ? MHI_EOT : MHI_CHAIN); | 435 | if (ret) |
| 392 | if (ret) { | 436 | break; |
| 393 | dev_err(&context->mhi_dev->dev, "Unable to send read_data response %d\n", | ||
| 394 | ret); | ||
| 395 | return; | ||
| 396 | } | ||
| 397 | } | 437 | } |
| 398 | } | 438 | } |
| 399 | 439 | ||
| @@ -538,6 +578,7 @@ static void sahara_parse_dump_table(struct sahara_context *context) · 138 unmodified lines | |||
| 538 | struct sahara_memory_dump_meta_v1 *dump_meta; | 578 | struct sahara_memory_dump_meta_v1 *dump_meta; |
| 539 | u64 table_nents; | 579 | u64 table_nents; |
| 540 | u64 dump_length; | 580 | u64 dump_length; |
| 581 | u64 mul_bytes; | ||
| 541 | int ret; | 582 | int ret; |
| 542 | u64 i; | 583 | u64 i; |
| 543 | 584 | ||
| @@ -551,8 +592,9 @@ static void sahara_parse_dump_table(struct sahara_context *context) · 7 unmodified lines | |||
| 551 | dev_table[i].description[SAHARA_TABLE_ENTRY_STR_LEN - 1] = 0; | 592 | dev_table[i].description[SAHARA_TABLE_ENTRY_STR_LEN - 1] = 0; |
| 552 | dev_table[i].filename[SAHARA_TABLE_ENTRY_STR_LEN - 1] = 0; | 593 | dev_table[i].filename[SAHARA_TABLE_ENTRY_STR_LEN - 1] = 0; |
| 553 | 594 | ||
| 554 | dump_length = size_add(dump_length, le64_to_cpu(dev_table[i].length)); | 595 | if (check_add_overflow(dump_length, |
| 555 | if (dump_length == SIZE_MAX) { | 596 | le64_to_cpu(dev_table[i].length), |
| 597 | &dump_length)) { | ||
| 556 | /* Discard the dump */ | 598 | /* Discard the dump */ |
| 557 | sahara_send_reset(context); | 599 | sahara_send_reset(context); |
| 558 | return; | 600 | return; |
| @@ -568,14 +610,17 @@ static void sahara_parse_dump_table(struct sahara_context *context) · 9 unmodified lines | |||
| 568 | dev_table[i].filename); | 610 | dev_table[i].filename); |
| 569 | } | 611 | } |
| 570 | 612 | ||
| 571 | dump_length = size_add(dump_length, sizeof(*dump_meta)); | 613 | if (check_add_overflow(dump_length, (u64)sizeof(*dump_meta), &dump_length)) { |
| 572 | if (dump_length == SIZE_MAX) { | ||
| 573 | /* Discard the dump */ | 614 | /* Discard the dump */ |
| 574 | sahara_send_reset(context); | 615 | sahara_send_reset(context); |
| 575 | return; | 616 | return; |
| 576 | } | 617 | } |
| 577 | dump_length = size_add(dump_length, size_mul(sizeof(*image_out_table), table_nents)); | 618 | if (check_mul_overflow((u64)sizeof(*image_out_table), table_nents, &mul_bytes)) { |
| 578 | if (dump_length == SIZE_MAX) { | 619 | /* Discard the dump */ |
| 620 | sahara_send_reset(context); | ||
| 621 | return; | ||
| 622 | } | ||
| 623 | if (check_add_overflow(dump_length, mul_bytes, &dump_length)) { | ||
| 579 | /* Discard the dump */ | 624 | /* Discard the dump */ |
| 580 | sahara_send_reset(context); | 625 | sahara_send_reset(context); |
| 581 | return; | 626 | return; |
| @@ -615,7 +660,7 @@ static void sahara_parse_dump_table(struct sahara_context *context) · 33 unmodified lines | |||
| 615 | 660 | ||
| 616 | /* Request the first chunk of the first image */ | 661 | /* Request the first chunk of the first image */ |
| 617 | context->dump_image = &image_out_table[0]; | 662 | context->dump_image = &image_out_table[0]; |
| 618 | dump_length = min(context->dump_image->length, SAHARA_READ_MAX_SIZE); | 663 | dump_length = min_t(u64, context->dump_image->length, SAHARA_READ_MAX_SIZE); |
| 619 | /* Avoid requesting EOI sized data so that we can identify errors */ | 664 | /* Avoid requesting EOI sized data so that we can identify errors */ |
| 620 | if (dump_length == SAHARA_END_OF_IMAGE_LENGTH) | 665 | if (dump_length == SAHARA_END_OF_IMAGE_LENGTH) |
| 621 | dump_length = SAHARA_END_OF_IMAGE_LENGTH / 2; | 666 | dump_length = SAHARA_END_OF_IMAGE_LENGTH / 2; |
| @@ -663,7 +708,7 @@ static void sahara_parse_dump_image(struct sahara_context *context) · 41 unmodified lines | |||
| 663 | 708 | ||
| 664 | /* Get next image chunk */ | 709 | /* Get next image chunk */ |
| 665 | dump_length = context->dump_image->length - context->dump_image_offset; | 710 | dump_length = context->dump_image->length - context->dump_image_offset; |
| 666 | dump_length = min(dump_length, SAHARA_READ_MAX_SIZE); | 711 | dump_length = min_t(u64, dump_length, SAHARA_READ_MAX_SIZE); |
| 667 | /* Avoid requesting EOI sized data so that we can identify errors */ | 712 | /* Avoid requesting EOI sized data so that we can identify errors */ |
| 668 | if (dump_length == SAHARA_END_OF_IMAGE_LENGTH) | 713 | if (dump_length == SAHARA_END_OF_IMAGE_LENGTH) |
| 669 | dump_length = SAHARA_END_OF_IMAGE_LENGTH / 2; | 714 | dump_length = SAHARA_END_OF_IMAGE_LENGTH / 2; |
| @@ -742,6 +787,13 @@ error: · 72 unmodified lines | |||
| 742 | sahara_send_reset(context); | 787 | sahara_send_reset(context); |
| 743 | } | 788 | } |
| 744 | 789 | ||
| 790 | static void sahara_read_data_processing(struct work_struct *work) | ||
| 791 | { | ||
| 792 | struct sahara_context *context = container_of(work, struct sahara_context, read_data_work); | ||
| 793 | |||
| 794 | read_data_helper(context, 0); | ||
| 795 | } | ||
| 796 | |||
| 745 | static int sahara_mhi_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id) | 797 | static int sahara_mhi_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id) |
| 746 | { | 798 | { |
| 747 | struct sahara_context *context; | 799 | struct sahara_context *context; |
| @@ -756,34 +808,56 @@ static int sahara_mhi_probe(struct mhi_device *mhi_dev, const struct mhi_device_ · 8 unmodified lines | |||
| 756 | if (!context->rx) | 808 | if (!context->rx) |
| 757 | return -ENOMEM; | 809 | return -ENOMEM; |
| 758 | 810 | ||
| 811 | if (!strcmp(mhi_dev->mhi_cntrl->name, "AIC200")) { | ||
| 812 | context->image_table = aic200_image_table; | ||
| 813 | context->table_size = ARRAY_SIZE(aic200_image_table); | ||
| 814 | } else { | ||
| 815 | context->image_table = aic100_image_table; | ||
| 816 | context->table_size = ARRAY_SIZE(aic100_image_table); | ||
| 817 | context->non_streaming = true; | ||
| 818 | } | ||
| 819 | |||
| 759 | /* | 820 | /* |
| 760 | * AIC100 defines SAHARA_TRANSFER_MAX_SIZE as the largest value it | 821 | * There are two firmware implementations for READ_DATA handling. |
| 761 | * will request for READ_DATA. This is larger than | 822 | * The older "SBL" implementation defines a Sahara transfer size, and |
| 762 | * SAHARA_PACKET_MAX_SIZE, and we need 9x SAHARA_PACKET_MAX_SIZE to | 823 | * expects that the response is a single transport transfer. If the |
| 763 | * cover SAHARA_TRANSFER_MAX_SIZE. When the remote side issues a | 824 | * FW wants to transfer a file that is larger than the transfer size, |
| 764 | * READ_DATA, it requires a transfer of the exact size requested. We | 825 | * the FW will issue multiple READ_DATA commands. For this |
| 765 | * can use MHI_CHAIN to link multiple buffers into a single transfer | 826 | * implementation, we need to allocate enough buffers to contain the |
| 766 | * but the remote side will not consume the buffers until it sees an | 827 | * entire Sahara transfer size. |
| 767 | * EOT, thus we need to allocate enough buffers to put in the tx fifo | 828 | * |
| 768 | * to cover an entire READ_DATA request of the max size. | 829 | * The newer "XBL" implementation does not define a maximum transfer |
| 830 | * size and instead expects the data to be streamed over using the | ||
| 831 | * transport level MTU. The FW will issue a single READ_DATA command | ||
| 832 | * of whatever size, and consume multiple transport level transfers | ||
| 833 | * until the expected amount of data is consumed. For this | ||
| 834 | * implementation we only need a single buffer of the transport MTU | ||
| 835 | * but we'll need to be able to use it multiple times for a single | ||
| 836 | * READ_DATA request. | ||
| 837 | * | ||
| 838 | * AIC100 is the SBL implementation and defines SAHARA_TRANSFER_MAX_SIZE | ||
| 839 | * and we need 9x SAHARA_PACKET_MAX_SIZE to cover that. We can use | ||
| 840 | * MHI_CHAIN to link multiple buffers into a single transfer but the | ||
| 841 | * remote side will not consume the buffers until it sees an EOT, thus | ||
| 842 | * we need to allocate enough buffers to put in the tx fifo to cover an | ||
| 843 | * entire READ_DATA request of the max size. | ||
| 844 | * | ||
| 845 | * AIC200 is the XBL implementation, and so a single buffer will work. | ||
| 769 | */ | 846 | */ |
| 770 | for (i = 0; i < SAHARA_NUM_TX_BUF; ++i) { | 847 | for (i = 0; i < SAHARA_NUM_TX_BUF; ++i) { |
| 771 | context->tx[i] = devm_kzalloc(&mhi_dev->dev, SAHARA_PACKET_MAX_SIZE, GFP_KERNEL); | 848 | context->tx[i] = devm_kzalloc(&mhi_dev->dev, |
| 849 | SAHARA_PACKET_MAX_SIZE, | ||
| 850 | GFP_KERNEL); | ||
| 772 | if (!context->tx[i]) | 851 | if (!context->tx[i]) |
| 773 | return -ENOMEM; | 852 | return -ENOMEM; |
| 853 | if (is_streaming(context)) | ||
| 854 | break; | ||
| 774 | } | 855 | } |
| 775 | 856 | ||
| 776 | context->mhi_dev = mhi_dev; | 857 | context->mhi_dev = mhi_dev; |
| 777 | INIT_WORK(&context->fw_work, sahara_processing); | 858 | INIT_WORK(&context->fw_work, sahara_processing); |
| 778 | INIT_WORK(&context->dump_work, sahara_dump_processing); | 859 | INIT_WORK(&context->dump_work, sahara_dump_processing); |
| 779 | 860 | INIT_WORK(&context->read_data_work, sahara_read_data_processing); | |
| 780 | if (!strcmp(mhi_dev->mhi_cntrl->name, "AIC200")) { | ||
| 781 | context->image_table = aic200_image_table; | ||
| 782 | context->table_size = ARRAY_SIZE(aic200_image_table); | ||
| 783 | } else { | ||
| 784 | context->image_table = aic100_image_table; | ||
| 785 | context->table_size = ARRAY_SIZE(aic100_image_table); | ||
| 786 | } | ||
| 787 | 861 | ||
| 788 | context->active_image_id = SAHARA_IMAGE_ID_NONE; | 862 | context->active_image_id = SAHARA_IMAGE_ID_NONE; |
| 789 | dev_set_drvdata(&mhi_dev->dev, context); | 863 | dev_set_drvdata(&mhi_dev->dev, context); |
| @@ -814,6 +888,10 @@ static void sahara_mhi_remove(struct mhi_device *mhi_dev) · 24 unmodified lines | |||
| 814 | 888 | ||
| 815 | static void sahara_mhi_ul_xfer_cb(struct mhi_device *mhi_dev, struct mhi_result *mhi_result) | 889 | static void sahara_mhi_ul_xfer_cb(struct mhi_device *mhi_dev, struct mhi_result *mhi_result) |
| 816 | { | 890 | { |
| 891 | struct sahara_context *context = dev_get_drvdata(&mhi_dev->dev); | ||
| 892 | |||
| 893 | if (!mhi_result->transaction_status && context->read_data_length && is_streaming(context)) | ||
| 894 | schedule_work(&context->read_data_work); | ||
| 817 | } | 895 | } |
| 818 | 896 | ||
| 819 | static void sahara_mhi_dl_xfer_cb(struct mhi_device *mhi_dev, struct mhi_result *mhi_result) | 897 | static void sahara_mhi_dl_xfer_cb(struct mhi_device *mhi_dev, struct mhi_result *mhi_result) |
| @@ -2,6 +2,7 @@ · 1 unmodified line | |||
| 2 | /* Copyright 2024-2025 Tomeu Vizoso <tomeu@tomeuvizoso.net> */ | 2 | /* Copyright 2024-2025 Tomeu Vizoso <tomeu@tomeuvizoso.net> */ |
| 3 | 3 | ||
| 4 | #include <drm/drm_device.h> | 4 | #include <drm/drm_device.h> |
| 5 | #include <drm/drm_print.h> | ||
| 5 | #include <drm/drm_utils.h> | 6 | #include <drm/drm_utils.h> |
| 6 | #include <drm/rocket_accel.h> | 7 | #include <drm/rocket_accel.h> |
| 7 | #include <linux/dma-mapping.h> | 8 | #include <linux/dma-mapping.h> |
| @@ -121,29 +121,27 @@ static const struct dma_fence_ops dma_fence_stub_ops = { · 120 unmodified lines | |||
| 121 | .get_timeline_name = dma_fence_stub_get_name, | 121 | .get_timeline_name = dma_fence_stub_get_name, |
| 122 | }; | 122 | }; |
| 123 | 123 | ||
| 124 | static int __init dma_fence_init_stub(void) | ||
| 125 | { | ||
| 126 | dma_fence_init(&dma_fence_stub, &dma_fence_stub_ops, | ||
| 127 | &dma_fence_stub_lock, 0, 0); | ||
| 128 | |||
| 129 | set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, | ||
| 130 | &dma_fence_stub.flags); | ||
| 131 | |||
| 132 | dma_fence_signal(&dma_fence_stub); | ||
| 133 | return 0; | ||
| 134 | } | ||
| 135 | subsys_initcall(dma_fence_init_stub); | ||
| 136 | |||
| 124 | /** | 137 | /** |
| 125 | * dma_fence_get_stub - return a signaled fence | 138 | * dma_fence_get_stub - return a signaled fence |
| 126 | * | 139 | * |
| 127 | * Return a stub fence which is already signaled. The fence's | 140 | * Return a stub fence which is already signaled. The fence's timestamp |
| 128 | * timestamp corresponds to the first time after boot this | 141 | * corresponds to the initialisation time of the linux kernel. |
| 129 | * function is called. | ||
| 130 | */ | 142 | */ |
| 131 | struct dma_fence *dma_fence_get_stub(void) | 143 | struct dma_fence *dma_fence_get_stub(void) |
| 132 | { | 144 | { |
| 133 | spin_lock(&dma_fence_stub_lock); | ||
| 134 | if (!dma_fence_stub.ops) { | ||
| 135 | dma_fence_init(&dma_fence_stub, | ||
| 136 | &dma_fence_stub_ops, | ||
| 137 | &dma_fence_stub_lock, | ||
| 138 | 0, 0); | ||
| 139 | |||
| 140 | set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, | ||
| 141 | &dma_fence_stub.flags); | ||
| 142 | |||
| 143 | dma_fence_signal_locked(&dma_fence_stub); | ||
| 144 | } | ||
| 145 | spin_unlock(&dma_fence_stub_lock); | ||
| 146 | |||
| 147 | return dma_fence_get(&dma_fence_stub); | 145 | return dma_fence_get(&dma_fence_stub); |
| 148 | } | 146 | } |
| 149 | EXPORT_SYMBOL(dma_fence_get_stub); | 147 | EXPORT_SYMBOL(dma_fence_get_stub); |
| @@ -999,19 +997,21 @@ EXPORT_SYMBOL(dma_fence_set_deadline); · 849 unmodified lines | |||
| 999 | */ | 997 | */ |
| 1000 | void dma_fence_describe(struct dma_fence *fence, struct seq_file *seq) | 998 | void dma_fence_describe(struct dma_fence *fence, struct seq_file *seq) |
| 1001 | { | 999 | { |
| 1002 | const char __rcu *timeline; | 1000 | const char __rcu *timeline = ""; |
| 1003 | const char __rcu *driver; | 1001 | const char __rcu *driver = ""; |
| 1002 | const char *signaled = ""; | ||
| 1004 | 1003 | ||
| 1005 | rcu_read_lock(); | 1004 | rcu_read_lock(); |
| 1006 | 1005 | ||
| 1007 | timeline = dma_fence_timeline_name(fence); | 1006 | if (!dma_fence_is_signaled(fence)) { |
| 1008 | driver = dma_fence_driver_name(fence); | 1007 | timeline = dma_fence_timeline_name(fence); |
| 1008 | driver = dma_fence_driver_name(fence); | ||
| 1009 | signaled = "un"; | ||
| 1010 | } | ||
| 1009 | 1011 | ||
| 1010 | seq_printf(seq, "%s %s seq %llu %ssignalled\n", | 1012 | seq_printf(seq, "%llu:%llu %s %s %ssignalled\n", |
| 1011 | rcu_dereference(driver), | 1013 | fence->context, fence->seqno, timeline, driver, |
| 1012 | rcu_dereference(timeline), | 1014 | signaled); |
| 1013 | fence->seqno, | ||
| 1014 | dma_fence_is_signaled(fence) ? "" : "un"); | ||
| 1015 | 1015 | ||
| 1016 | rcu_read_unlock(); | 1016 | rcu_read_unlock(); |
| 1017 | } | 1017 | } |
| @@ -12,13 +12,3 @@ config DMABUF_HEAPS_CMA · 11 unmodified lines | |||
| 12 | Choose this option to enable dma-buf CMA heap. This heap is backed | 12 | Choose this option to enable dma-buf CMA heap. This heap is backed |
| 13 | by the Contiguous Memory Allocator (CMA). If your system has these | 13 | by the Contiguous Memory Allocator (CMA). If your system has these |
| 14 | regions, you should say Y here. | 14 | regions, you should say Y here. |
| 15 | |||
| 16 | config DMABUF_HEAPS_CMA_LEGACY | ||
| 17 | bool "Legacy DMA-BUF CMA Heap" | ||
| 18 | default y | ||
| 19 | depends on DMABUF_HEAPS_CMA | ||
| 20 | help | ||
| 21 | Add a duplicate CMA-backed dma-buf heap with legacy naming derived | ||
| 22 | from the CMA area's devicetree node, or "reserved" if the area is not | ||
| 23 | defined in the devicetree. This uses the same underlying allocator as | ||
| 24 | CONFIG_DMABUF_HEAPS_CMA. | ||
| @@ -14,6 +14,7 @@ · 13 unmodified lines | |||
| 14 | 14 | ||
| 15 | #include <linux/cma.h> | 15 | #include <linux/cma.h> |
| 16 | #include <linux/dma-buf.h> | 16 | #include <linux/dma-buf.h> |
| 17 | #include <linux/dma-buf/heaps/cma.h> | ||
| 17 | #include <linux/dma-heap.h> | 18 | #include <linux/dma-heap.h> |
| 18 | #include <linux/dma-map-ops.h> | 19 | #include <linux/dma-map-ops.h> |
| 19 | #include <linux/err.h> | 20 | #include <linux/err.h> |
| @@ -21,12 +22,27 @@ · 1 unmodified line | |||
| 21 | #include <linux/io.h> | 22 | #include <linux/io.h> |
| 22 | #include <linux/mm.h> | 23 | #include <linux/mm.h> |
| 23 | #include <linux/module.h> | 24 | #include <linux/module.h> |
| 25 | #include <linux/of.h> | ||
| 26 | #include <linux/of_reserved_mem.h> | ||
| 24 | #include <linux/scatterlist.h> | 27 | #include <linux/scatterlist.h> |
| 25 | #include <linux/slab.h> | 28 | #include <linux/slab.h> |
| 26 | #include <linux/vmalloc.h> | 29 | #include <linux/vmalloc.h> |
| 27 | 30 | ||
| 28 | #define DEFAULT_CMA_NAME "default_cma_region" | 31 | #define DEFAULT_CMA_NAME "default_cma_region" |
| 29 | 32 | ||
| 33 | static struct cma *dma_areas[MAX_CMA_AREAS] __initdata; | ||
| 34 | static unsigned int dma_areas_num __initdata; | ||
| 35 | |||
| 36 | int __init dma_heap_cma_register_heap(struct cma *cma) | ||
| 37 | { | ||
| 38 | if (dma_areas_num >= ARRAY_SIZE(dma_areas)) | ||
| 39 | return -EINVAL; | ||
| 40 | |||
| 41 | dma_areas[dma_areas_num++] = cma; | ||
| 42 | |||
| 43 | return 0; | ||
| 44 | } | ||
| 45 | |||
| 30 | struct cma_heap { | 46 | struct cma_heap { |
| 31 | struct dma_heap *heap; | 47 | struct dma_heap *heap; |
| 32 | struct cma *cma; | 48 | struct cma *cma; |
| @@ -395,33 +411,30 @@ static int __init __add_cma_heap(struct cma *cma, const char *name) · 362 unmodified lines | |||
| 395 | return 0; | 411 | return 0; |
| 396 | } | 412 | } |
| 397 | 413 | ||
| 398 | static int __init add_default_cma_heap(void) | 414 | static int __init add_cma_heaps(void) |
| 399 | { | 415 | { |
| 400 | struct cma *default_cma = dev_get_cma_area(NULL); | 416 | struct cma *default_cma = dev_get_cma_area(NULL); |
| 401 | const char *legacy_cma_name; | 417 | unsigned int i; |
| 402 | int ret; | 418 | int ret; |
| 403 | 419 | ||
| 404 | if (!default_cma) | 420 | if (default_cma) { |
| 405 | return 0; | 421 | ret = __add_cma_heap(default_cma, DEFAULT_CMA_NAME); |
| 422 | if (ret) | ||
| 423 | return ret; | ||
| 424 | } | ||
| 406 | 425 | ||
| 407 | ret = __add_cma_heap(default_cma, DEFAULT_CMA_NAME); | 426 | for (i = 0; i < dma_areas_num; i++) { |
| 408 | if (ret) | 427 | struct cma *cma = dma_areas[i]; |
| 409 | return ret; | ||
| 410 | 428 | ||
| 411 | if (IS_ENABLED(CONFIG_DMABUF_HEAPS_CMA_LEGACY)) { | 429 | ret = __add_cma_heap(cma, cma_get_name(cma)); |
| 412 | legacy_cma_name = cma_get_name(default_cma); | 430 | if (ret) { |
| 413 | if (!strcmp(legacy_cma_name, DEFAULT_CMA_NAME)) { | 431 | pr_warn("Failed to add CMA heap %s", cma_get_name(cma)); |
| 414 | pr_warn("legacy name and default name are the same, skipping legacy heap\n"); | 432 | continue; |
| 415 | return 0; | ||
| 416 | } | 433 | } |
| 417 | 434 | ||
| 418 | ret = __add_cma_heap(default_cma, legacy_cma_name); | ||
| 419 | if (ret) | ||
| 420 | pr_warn("failed to add legacy heap: %pe\n", | ||
| 421 | ERR_PTR(ret)); | ||
| 422 | } | 435 | } |
| 423 | 436 | ||
| 424 | return 0; | 437 | return 0; |
| 425 | } | 438 | } |
| 426 | module_init(add_default_cma_heap); | 439 | module_init(add_cma_heaps); |
| 427 | MODULE_DESCRIPTION("DMA-BUF CMA Heap"); | 440 | MODULE_DESCRIPTION("DMA-BUF CMA Heap"); |
| @@ -186,20 +186,35 @@ static int system_heap_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) · 185 unmodified lines | |||
| 186 | struct system_heap_buffer *buffer = dmabuf->priv; | 186 | struct system_heap_buffer *buffer = dmabuf->priv; |
| 187 | struct sg_table *table = &buffer->sg_table; | 187 | struct sg_table *table = &buffer->sg_table; |
| 188 | unsigned long addr = vma->vm_start; | 188 | unsigned long addr = vma->vm_start; |
| 189 | struct sg_page_iter piter; | 189 | unsigned long pgoff = vma->vm_pgoff; |
| 190 | int ret; | 190 | struct scatterlist *sg; |
| 191 | int i, ret; | ||
| 192 | |||
| 193 | for_each_sgtable_sg(table, sg, i) { | ||
| 194 | unsigned long n = sg->length >> PAGE_SHIFT; | ||
| 191 | 195 | ||
| 192 | for_each_sgtable_page(table, &piter, vma->vm_pgoff) { | 196 | if (pgoff < n) |
| 193 | struct page *page = sg_page_iter_page(&piter); | 197 | break; |
| 198 | pgoff -= n; | ||
| 199 | } | ||
| 200 | |||
| 201 | for (; sg && addr < vma->vm_end; sg = sg_next(sg)) { | ||
| 202 | unsigned long n = (sg->length >> PAGE_SHIFT) - pgoff; | ||
| 203 | struct page *page = sg_page(sg) + pgoff; | ||
| 204 | unsigned long size = n << PAGE_SHIFT; | ||
| 205 | |||
| 206 | if (addr + size > vma->vm_end) | ||
| 207 | size = vma->vm_end - addr; | ||
| 194 | 208 | ||
| 195 | ret = remap_pfn_range(vma, addr, page_to_pfn(page), PAGE_SIZE, | 209 | ret = remap_pfn_range(vma, addr, page_to_pfn(page), |
| 196 | vma->vm_page_prot); | 210 | size, vma->vm_page_prot); |
| 197 | if (ret) | 211 | if (ret) |
| 198 | return ret; | 212 | return ret; |
| 199 | addr += PAGE_SIZE; | 213 | |
| 200 | if (addr >= vma->vm_end) | 214 | addr += size; |
| 201 | return 0; | 215 | pgoff = 0; |
| 202 | } | 216 | } |
| 217 | |||
| 203 | return 0; | 218 | return 0; |
| 204 | } | 219 | } |
| 205 | 220 | ||
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | #include <linux/file.h> | 8 | #include <linux/file.h> |
| 9 | #include <linux/fs.h> | 9 | #include <linux/fs.h> |
| 10 | #include <linux/uaccess.h> | 10 | #include <linux/uaccess.h> |
| 11 | #include <linux/panic.h> | ||
| 11 | #include <linux/slab.h> | 12 | #include <linux/slab.h> |
| 12 | #include <linux/sync_file.h> | 13 | #include <linux/sync_file.h> |
| 13 | 14 | ||
| @@ -349,6 +350,9 @@ static long sw_sync_ioctl_create_fence(struct sync_timeline *obj, · 335 unmodified lines | |||
| 349 | struct sync_file *sync_file; | 350 | struct sync_file *sync_file; |
| 350 | struct sw_sync_create_fence_data data; | 351 | struct sw_sync_create_fence_data data; |
| 351 | 352 | ||
| 353 | /* SW sync fence are inherently unsafe and can deadlock the kernel */ | ||
| 354 | add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); | ||
| 355 | |||
| 352 | if (fd < 0) | 356 | if (fd < 0) |
| 353 | return fd; | 357 | return fd; |
| 354 | 358 | ||
| @@ -41,6 +41,7 @@ drm-y := \ · 40 unmodified lines | |||
| 41 | drm_bridge.o \ | 41 | drm_bridge.o \ |
| 42 | drm_cache.o \ | 42 | drm_cache.o \ |
| 43 | drm_color_mgmt.o \ | 43 | drm_color_mgmt.o \ |
| 44 | drm_colorop.o \ | ||
| 44 | drm_connector.o \ | 45 | drm_connector.o \ |
| 45 | drm_crtc.o \ | 46 | drm_crtc.o \ |
| 46 | drm_displayid.o \ | 47 | drm_displayid.o \ |
| @@ -76,7 +77,8 @@ drm-y := \ · 29 unmodified lines | |||
| 76 | drm-$(CONFIG_DRM_CLIENT) += \ | 77 | drm-$(CONFIG_DRM_CLIENT) += \ |
| 77 | drm_client.o \ | 78 | drm_client.o \ |
| 78 | drm_client_event.o \ | 79 | drm_client_event.o \ |
| 79 | drm_client_modeset.o | 80 | drm_client_modeset.o \ |
| 81 | drm_client_sysrq.o | ||
| 80 | drm-$(CONFIG_DRM_LIB_RANDOM) += lib/drm_random.o | 82 | drm-$(CONFIG_DRM_LIB_RANDOM) += lib/drm_random.o |
| 81 | drm-$(CONFIG_COMPAT) += drm_ioc32.o | 83 | drm-$(CONFIG_COMPAT) += drm_ioc32.o |
| 82 | drm-$(CONFIG_DRM_PANEL) += drm_panel.o | 84 | drm-$(CONFIG_DRM_PANEL) += drm_panel.o |
| @@ -150,7 +152,8 @@ drm_kms_helper-y := \ · 67 unmodified lines | |||
| 150 | drm_plane_helper.o \ | 152 | drm_plane_helper.o \ |
| 151 | drm_probe_helper.o \ | 153 | drm_probe_helper.o \ |
| 152 | drm_self_refresh_helper.o \ | 154 | drm_self_refresh_helper.o \ |
| 153 | drm_simple_kms_helper.o | 155 | drm_simple_kms_helper.o \ |
| 156 | drm_vblank_helper.o | ||
| 154 | drm_kms_helper-$(CONFIG_DRM_PANEL_BRIDGE) += bridge/panel.o | 157 | drm_kms_helper-$(CONFIG_DRM_PANEL_BRIDGE) += bridge/panel.o |
| 155 | drm_kms_helper-$(CONFIG_DRM_FBDEV_EMULATION) += drm_fb_helper.o | 158 | drm_kms_helper-$(CONFIG_DRM_FBDEV_EMULATION) += drm_fb_helper.o |
| 156 | obj-$(CONFIG_DRM_KMS_HELPER) += drm_kms_helper.o | 159 | obj-$(CONFIG_DRM_KMS_HELPER) += drm_kms_helper.o |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | |||
| 16 | #include <drm/drm_gem_dma_helper.h> | 16 | #include <drm/drm_gem_dma_helper.h> |
| 17 | #include <drm/drm_gem_framebuffer_helper.h> | 17 | #include <drm/drm_gem_framebuffer_helper.h> |
| 18 | #include <drm/drm_of.h> | 18 | #include <drm/drm_of.h> |
| 19 | #include <drm/drm_print.h> | ||
| 19 | #include <drm/drm_probe_helper.h> | 20 | #include <drm/drm_probe_helper.h> |
| 20 | #include <drm/drm_vblank.h> | 21 | #include <drm/drm_vblank.h> |
| 21 | 22 | ||
| @@ -43,14 +43,16 @@ config DRM_AMDGPU_SI · 42 unmodified lines | |||
| 43 | bool "Enable amdgpu support for SI parts" | 43 | bool "Enable amdgpu support for SI parts" |
| 44 | depends on DRM_AMDGPU | 44 | depends on DRM_AMDGPU |
| 45 | help | 45 | help |
| 46 | Choose this option if you want to enable experimental support | 46 | Choose this option if you want to enable support |
| 47 | for SI (Southern Islands) asics. | 47 | for SI (Southern Islands) asics. |
| 48 | 48 | ||
| 49 | SI is already supported in radeon. Experimental support for SI | 49 | SI (Southern Islands) are first generation GCN GPUs, |
| 50 | in amdgpu will be disabled by default and is still provided by | 50 | supported by both drivers: radeon (old) and amdgpu (new). |
| 51 | radeon. Use module options to override this: | 51 | By default, SI dedicated GPUs are supported by amdgpu. |
| 52 | 52 | ||
| 53 | radeon.si_support=0 amdgpu.si_support=1 | 53 | Use module options to override this: |
| 54 | To use radeon for SI, | ||
| 55 | radeon.si_support=1 amdgpu.si_support=0 | ||
| 54 | 56 | ||
| 55 | config DRM_AMDGPU_CIK | 57 | config DRM_AMDGPU_CIK |
| 56 | bool "Enable amdgpu support for CIK parts" | 58 | bool "Enable amdgpu support for CIK parts" |
| @@ -59,11 +61,17 @@ config DRM_AMDGPU_CIK · 2 unmodified lines | |||
| 59 | Choose this option if you want to enable support for CIK (Sea | 61 | Choose this option if you want to enable support for CIK (Sea |
| 60 | Islands) asics. | 62 | Islands) asics. |
| 61 | 63 | ||
| 62 | CIK is already supported in radeon. Support for CIK in amdgpu | 64 | CIK (Sea Islands) are second generation GCN GPUs, |
| 63 | will be disabled by default and is still provided by radeon. | 65 | supported by both drivers: radeon (old) and amdgpu (new). |
| 64 | Use module options to override this: | 66 | By default, |
| 67 | CIK dedicated GPUs are supported by amdgpu | ||
| 68 | CIK APUs are supported by radeon | ||
| 65 | 69 | ||
| 70 | Use module options to override this: | ||
| 71 | To use amdgpu for CIK, | ||
| 66 | radeon.cik_support=0 amdgpu.cik_support=1 | 72 | radeon.cik_support=0 amdgpu.cik_support=1 |
| 73 | To use radeon for CIK, | ||
| 74 | radeon.cik_support=1 amdgpu.cik_support=0 | ||
| 67 | 75 | ||
| 68 | config DRM_AMDGPU_USERPTR | 76 | config DRM_AMDGPU_USERPTR |
| 69 | bool "Always enable userptr write support" | 77 | bool "Always enable userptr write support" |
| @@ -37,7 +37,8 @@ ccflags-y := -I$(FULL_AMD_PATH)/include/asic_reg \ · 36 unmodified lines | |||
| 37 | -I$(FULL_AMD_DISPLAY_PATH)/modules/inc \ | 37 | -I$(FULL_AMD_DISPLAY_PATH)/modules/inc \ |
| 38 | -I$(FULL_AMD_DISPLAY_PATH)/dc \ | 38 | -I$(FULL_AMD_DISPLAY_PATH)/dc \ |
| 39 | -I$(FULL_AMD_DISPLAY_PATH)/amdgpu_dm \ | 39 | -I$(FULL_AMD_DISPLAY_PATH)/amdgpu_dm \ |
| 40 | -I$(FULL_AMD_PATH)/amdkfd | 40 | -I$(FULL_AMD_PATH)/amdkfd \ |
| 41 | -I$(FULL_AMD_PATH)/ras/ras_mgr | ||
| 41 | 42 | ||
| 42 | # Locally disable W=1 warnings enabled in drm subsystem Makefile | 43 | # Locally disable W=1 warnings enabled in drm subsystem Makefile |
| 43 | subdir-ccflags-y += -Wno-override-init | 44 | subdir-ccflags-y += -Wno-override-init |
| @@ -77,7 +78,7 @@ amdgpu-$(CONFIG_DRM_AMDGPU_CIK)+= cik.o cik_ih.o \ · 33 unmodified lines | |||
| 77 | dce_v8_0.o gfx_v7_0.o cik_sdma.o uvd_v4_2.o vce_v2_0.o | 78 | dce_v8_0.o gfx_v7_0.o cik_sdma.o uvd_v4_2.o vce_v2_0.o |
| 78 | 79 | ||
| 79 | amdgpu-$(CONFIG_DRM_AMDGPU_SI)+= si.o gmc_v6_0.o gfx_v6_0.o si_ih.o si_dma.o dce_v6_0.o \ | 80 | amdgpu-$(CONFIG_DRM_AMDGPU_SI)+= si.o gmc_v6_0.o gfx_v6_0.o si_ih.o si_dma.o dce_v6_0.o \ |
| 80 | uvd_v3_1.o | 81 | uvd_v3_1.o vce_v1_0.o |
| 81 | 82 | ||
| 82 | amdgpu-y += \ | 83 | amdgpu-y += \ |
| 83 | vi.o mxgpu_vi.o nbio_v6_1.o soc15.o emu_soc.o mxgpu_ai.o nbio_v7_0.o vega10_reg_init.o \ | 84 | vi.o mxgpu_vi.o nbio_v6_1.o soc15.o emu_soc.o mxgpu_ai.o nbio_v7_0.o vega10_reg_init.o \ |
| @@ -324,4 +325,9 @@ amdgpu-y += \ · 240 unmodified lines | |||
| 324 | isp_v4_1_1.o | 325 | isp_v4_1_1.o |
| 325 | endif | 326 | endif |
| 326 | 327 | ||
| 328 | AMD_GPU_RAS_PATH := ../ras | ||
| 329 | AMD_GPU_RAS_FULL_PATH := $(FULL_AMD_PATH)/ras | ||
| 330 | include $(AMD_GPU_RAS_FULL_PATH)/Makefile | ||
| 331 | amdgpu-y += $(AMD_GPU_RAS_FILES) | ||
| 332 | |||
| 327 | obj-$(CONFIG_DRM_AMDGPU)+= amdgpu.o | 333 | obj-$(CONFIG_DRM_AMDGPU)+= amdgpu.o |
| @@ -88,6 +88,10 @@ static int aldebaran_mode2_suspend_ip(struct amdgpu_device *adev) · 87 unmodified lines | |||
| 88 | uint32_t ip_block; | 88 | uint32_t ip_block; |
| 89 | int r, i; | 89 | int r, i; |
| 90 | 90 | ||
| 91 | /* Skip suspend of SDMA IP versions >= 4.4.2. They are multi-aid */ | ||
| 92 | if (adev->aid_mask) | ||
| 93 | ip_block_mask &= ~BIT(AMD_IP_BLOCK_TYPE_SDMA); | ||
| 94 | |||
| 91 | amdgpu_device_set_pg_state(adev, AMD_PG_STATE_UNGATE); | 95 | amdgpu_device_set_pg_state(adev, AMD_PG_STATE_UNGATE); |
| 92 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE); | 96 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE); |
| 93 | 97 | ||
| @@ -372,13 +372,15 @@ void amdgpu_device_ip_get_clockgating_state(struct amdgpu_device *adev, · 371 unmodified lines | |||
| 372 | u64 *flags); | 372 | u64 *flags); |
| 373 | int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, | 373 | int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, |
| 374 | enum amd_ip_block_type block_type); | 374 | enum amd_ip_block_type block_type); |
| 375 | bool amdgpu_device_ip_is_hw(struct amdgpu_device *adev, | ||
| 376 | enum amd_ip_block_type block_type); | ||
| 375 | bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev, | 377 | bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev, |
| 376 | enum amd_ip_block_type block_type); | 378 | enum amd_ip_block_type block_type); |
| 377 | int amdgpu_ip_block_suspend(struct amdgpu_ip_block *ip_block); | 379 | int amdgpu_ip_block_suspend(struct amdgpu_ip_block *ip_block); |
| 378 | 380 | ||
| 379 | int amdgpu_ip_block_resume(struct amdgpu_ip_block *ip_block); | 381 | int amdgpu_ip_block_resume(struct amdgpu_ip_block *ip_block); |
| 380 | 382 | ||
| 381 | #define AMDGPU_MAX_IP_NUM 16 | 383 | #define AMDGPU_MAX_IP_NUM AMD_IP_BLOCK_TYPE_NUM |
| 382 | 384 | ||
| 383 | struct amdgpu_ip_block_status { | 385 | struct amdgpu_ip_block_status { |
| 384 | bool valid; | 386 | bool valid; |
| @@ -839,8 +841,6 @@ struct amd_powerplay { · 454 unmodified lines | |||
| 839 | const struct amd_pm_funcs *pp_funcs; | 841 | const struct amd_pm_funcs *pp_funcs; |
| 840 | }; | 842 | }; |
| 841 | 843 | ||
| 842 | struct ip_discovery_top; | ||
| 843 | |||
| 844 | /* polaris10 kickers */ | 844 | /* polaris10 kickers */ |
| 845 | #define ASICID_IS_P20(did, rid) (((did == 0x67DF) && \ | 845 | #define ASICID_IS_P20(did, rid) (((did == 0x67DF) && \ |
| 846 | ((rid == 0xE3) || \ | 846 | ((rid == 0xE3) || \ |
| @@ -972,8 +972,7 @@ struct amdgpu_device { · 125 unmodified lines | |||
| 972 | struct notifier_block acpi_nb; | 972 | struct notifier_block acpi_nb; |
| 973 | struct notifier_block pm_nb; | 973 | struct notifier_block pm_nb; |
| 974 | struct amdgpu_i2c_chan *i2c_bus[AMDGPU_MAX_I2C_BUS]; | 974 | struct amdgpu_i2c_chan *i2c_bus[AMDGPU_MAX_I2C_BUS]; |
| 975 | struct debugfs_blob_wrapper debugfs_vbios_blob; | 975 | struct debugfs_blob_wrapper debugfs_vbios_blob; |
| 976 | struct debugfs_blob_wrapper debugfs_discovery_blob; | ||
| 977 | struct mutex srbm_mutex; | 976 | struct mutex srbm_mutex; |
| 978 | /* GRBM index mutex. Protects concurrent access to GRBM index */ | 977 | /* GRBM index mutex. Protects concurrent access to GRBM index */ |
| 979 | struct mutex grbm_idx_mutex; | 978 | struct mutex grbm_idx_mutex; |
| @@ -1063,6 +1062,9 @@ struct amdgpu_device { · 83 unmodified lines | |||
| 1063 | u32 log2_max_MBps; | 1062 | u32 log2_max_MBps; |
| 1064 | } mm_stats; | 1063 | } mm_stats; |
| 1065 | 1064 | ||
| 1065 | /* discovery*/ | ||
| 1066 | struct amdgpu_discovery_info discovery; | ||
| 1067 | |||
| 1066 | /* display */ | 1068 | /* display */ |
| 1067 | bool enable_virtual_display; | 1069 | bool enable_virtual_display; |
| 1068 | struct amdgpu_vkms_output *amdgpu_vkms_output; | 1070 | struct amdgpu_vkms_output *amdgpu_vkms_output; |
| @@ -1174,6 +1176,12 @@ struct amdgpu_device { · 105 unmodified lines | |||
| 1174 | * queue fence. | 1176 | * queue fence. |
| 1175 | */ | 1177 | */ |
| 1176 | struct xarray userq_xa; | 1178 | struct xarray userq_xa; |
| 1179 | /** | ||
| 1180 | * @userq_doorbell_xa: Global user queue map (doorbell index → queue) | ||
| 1181 | * Key: doorbell_index (unique global identifier for the queue) | ||
| 1182 | * Value: struct amdgpu_usermode_queue | ||
| 1183 | */ | ||
| 1184 | struct xarray userq_doorbell_xa; | ||
| 1177 | 1185 | ||
| 1178 | /* df */ | 1186 | /* df */ |
| 1179 | struct amdgpu_df df; | 1187 | struct amdgpu_df df; |
| @@ -1265,8 +1273,6 @@ struct amdgpu_device { · 85 unmodified lines | |||
| 1265 | 1273 | ||
| 1266 | struct list_head ras_list; | 1274 | struct list_head ras_list; |
| 1267 | 1275 | ||
| 1268 | struct ip_discovery_top *ip_top; | ||
| 1269 | |||
| 1270 | struct amdgpu_reset_domain *reset_domain; | 1276 | struct amdgpu_reset_domain *reset_domain; |
| 1271 | 1277 | ||
| 1272 | struct mutex benchmark_mutex; | 1278 | struct mutex benchmark_mutex; |
| @@ -1309,9 +1315,8 @@ struct amdgpu_device { · 36 unmodified lines | |||
| 1309 | */ | 1315 | */ |
| 1310 | bool apu_prefer_gtt; | 1316 | bool apu_prefer_gtt; |
| 1311 | 1317 | ||
| 1312 | struct list_head userq_mgr_list; | ||
| 1313 | struct mutex userq_mutex; | ||
| 1314 | bool userq_halt_for_enforce_isolation; | 1318 | bool userq_halt_for_enforce_isolation; |
| 1319 | struct work_struct userq_reset_work; | ||
| 1315 | struct amdgpu_uid *uid_info; | 1320 | struct amdgpu_uid *uid_info; |
| 1316 | 1321 | ||
| 1317 | /* KFD | 1322 | /* KFD |
| @@ -1535,11 +1540,6 @@ int emu_soc_asic_init(struct amdgpu_device *adev); · 217 unmodified lines | |||
| 1535 | #define amdgpu_asic_read_bios_from_rom(adev, b, l) (adev)->asic_funcs->read_bios_from_rom((adev), (b), (l)) | 1540 | #define amdgpu_asic_read_bios_from_rom(adev, b, l) (adev)->asic_funcs->read_bios_from_rom((adev), (b), (l)) |
| 1536 | #define amdgpu_asic_read_register(adev, se, sh, offset, v)((adev)->asic_funcs->read_register((adev), (se), (sh), (offset), (v))) | 1541 | #define amdgpu_asic_read_register(adev, se, sh, offset, v)((adev)->asic_funcs->read_register((adev), (se), (sh), (offset), (v))) |
| 1537 | #define amdgpu_asic_get_config_memsize(adev) (adev)->asic_funcs->get_config_memsize((adev)) | 1542 | #define amdgpu_asic_get_config_memsize(adev) (adev)->asic_funcs->get_config_memsize((adev)) |
| 1538 | #define amdgpu_asic_flush_hdp(adev, r) \ | ||
| 1539 | ((adev)->asic_funcs->flush_hdp ? (adev)->asic_funcs->flush_hdp((adev), (r)) : (adev)->hdp.funcs->flush_hdp((adev), (r))) | ||
| 1540 | #define amdgpu_asic_invalidate_hdp(adev, r) \ | ||
| 1541 | ((adev)->asic_funcs->invalidate_hdp ? (adev)->asic_funcs->invalidate_hdp((adev), (r)) : \ | ||
| 1542 | ((adev)->hdp.funcs->invalidate_hdp ? (adev)->hdp.funcs->invalidate_hdp((adev), (r)) : (void)0)) | ||
| 1543 | #define amdgpu_asic_need_full_reset(adev) (adev)->asic_funcs->need_full_reset((adev)) | 1543 | #define amdgpu_asic_need_full_reset(adev) (adev)->asic_funcs->need_full_reset((adev)) |
| 1544 | #define amdgpu_asic_init_doorbell_index(adev) (adev)->asic_funcs->init_doorbell_index((adev)) | 1544 | #define amdgpu_asic_init_doorbell_index(adev) (adev)->asic_funcs->init_doorbell_index((adev)) |
| 1545 | #define amdgpu_asic_get_pcie_usage(adev, cnt0, cnt1) ((adev)->asic_funcs->get_pcie_usage((adev), (cnt0), (cnt1))) | 1545 | #define amdgpu_asic_get_pcie_usage(adev, cnt0, cnt1) ((adev)->asic_funcs->get_pcie_usage((adev), (cnt0), (cnt1))) |
| @@ -1638,7 +1638,6 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 92 unmodified lines | |||
| 1638 | struct drm_file *file_priv); | 1638 | struct drm_file *file_priv); |
| 1639 | void amdgpu_driver_release_kms(struct drm_device *dev); | 1639 | void amdgpu_driver_release_kms(struct drm_device *dev); |
| 1640 | 1640 | ||
| 1641 | int amdgpu_device_ip_suspend(struct amdgpu_device *adev); | ||
| 1642 | int amdgpu_device_prepare(struct drm_device *dev); | 1641 | int amdgpu_device_prepare(struct drm_device *dev); |
| 1643 | void amdgpu_device_complete(struct drm_device *dev); | 1642 | void amdgpu_device_complete(struct drm_device *dev); |
| 1644 | int amdgpu_device_suspend(struct drm_device *dev, bool fbcon); | 1643 | int amdgpu_device_suspend(struct drm_device *dev, bool fbcon); |
| @@ -302,17 +302,19 @@ static int acp_hw_init(struct amdgpu_ip_block *ip_block) · 301 unmodified lines | |||
| 302 | adev->acp.acp_res[2].end = adev->acp.acp_res[2].start; | 302 | adev->acp.acp_res[2].end = adev->acp.acp_res[2].start; |
| 303 | 303 | ||
| 304 | adev->acp.acp_cell[0].name = "acp_audio_dma"; | 304 | adev->acp.acp_cell[0].name = "acp_audio_dma"; |
| 305 | adev->acp.acp_cell[0].id = 0; | ||
| 305 | adev->acp.acp_cell[0].num_resources = 3; | 306 | adev->acp.acp_cell[0].num_resources = 3; |
| 306 | adev->acp.acp_cell[0].resources = &adev->acp.acp_res[0]; | 307 | adev->acp.acp_cell[0].resources = &adev->acp.acp_res[0]; |
| 307 | adev->acp.acp_cell[0].platform_data = &adev->asic_type; | 308 | adev->acp.acp_cell[0].platform_data = &adev->asic_type; |
| 308 | adev->acp.acp_cell[0].pdata_size = sizeof(adev->asic_type); | 309 | adev->acp.acp_cell[0].pdata_size = sizeof(adev->asic_type); |
| 309 | 310 | ||
| 310 | adev->acp.acp_cell[1].name = "designware-i2s"; | 311 | adev->acp.acp_cell[1].name = "designware-i2s"; |
| 312 | adev->acp.acp_cell[1].id = 1; | ||
| 311 | adev->acp.acp_cell[1].num_resources = 1; | 313 | adev->acp.acp_cell[1].num_resources = 1; |
| 312 | adev->acp.acp_cell[1].resources = &adev->acp.acp_res[1]; | 314 | adev->acp.acp_cell[1].resources = &adev->acp.acp_res[1]; |
| 313 | adev->acp.acp_cell[1].platform_data = &i2s_pdata[0]; | 315 | adev->acp.acp_cell[1].platform_data = &i2s_pdata[0]; |
| 314 | adev->acp.acp_cell[1].pdata_size = sizeof(struct i2s_platform_data); | 316 | adev->acp.acp_cell[1].pdata_size = sizeof(struct i2s_platform_data); |
| 315 | r = mfd_add_hotplug_devices(adev->acp.parent, adev->acp.acp_cell, 2); | 317 | r = mfd_add_devices(adev->acp.parent, 0, adev->acp.acp_cell, 2, NULL, 0, NULL); |
| 316 | if (r) | 318 | if (r) |
| 317 | goto failure; | 319 | goto failure; |
| 318 | r = device_for_each_child(adev->acp.parent, &adev->acp.acp_genpd->gpd, | 320 | r = device_for_each_child(adev->acp.parent, &adev->acp.acp_genpd->gpd, |
| @@ -410,30 +412,34 @@ static int acp_hw_init(struct amdgpu_ip_block *ip_block) · 91 unmodified lines | |||
| 410 | adev->acp.acp_res[4].end = adev->acp.acp_res[4].start; | 412 | adev->acp.acp_res[4].end = adev->acp.acp_res[4].start; |
| 411 | 413 | ||
| 412 | adev->acp.acp_cell[0].name = "acp_audio_dma"; | 414 | adev->acp.acp_cell[0].name = "acp_audio_dma"; |
| 415 | adev->acp.acp_cell[0].id = 0; | ||
| 413 | adev->acp.acp_cell[0].num_resources = 5; | 416 | adev->acp.acp_cell[0].num_resources = 5; |
| 414 | adev->acp.acp_cell[0].resources = &adev->acp.acp_res[0]; | 417 | adev->acp.acp_cell[0].resources = &adev->acp.acp_res[0]; |
| 415 | adev->acp.acp_cell[0].platform_data = &adev->asic_type; | 418 | adev->acp.acp_cell[0].platform_data = &adev->asic_type; |
| 416 | adev->acp.acp_cell[0].pdata_size = sizeof(adev->asic_type); | 419 | adev->acp.acp_cell[0].pdata_size = sizeof(adev->asic_type); |
| 417 | 420 | ||
| 418 | adev->acp.acp_cell[1].name = "designware-i2s"; | 421 | adev->acp.acp_cell[1].name = "designware-i2s"; |
| 422 | adev->acp.acp_cell[1].id = 1; | ||
| 419 | adev->acp.acp_cell[1].num_resources = 1; | 423 | adev->acp.acp_cell[1].num_resources = 1; |
| 420 | adev->acp.acp_cell[1].resources = &adev->acp.acp_res[1]; | 424 | adev->acp.acp_cell[1].resources = &adev->acp.acp_res[1]; |
| 421 | adev->acp.acp_cell[1].platform_data = &i2s_pdata[0]; | 425 | adev->acp.acp_cell[1].platform_data = &i2s_pdata[0]; |
| 422 | adev->acp.acp_cell[1].pdata_size = sizeof(struct i2s_platform_data); | 426 | adev->acp.acp_cell[1].pdata_size = sizeof(struct i2s_platform_data); |
| 423 | 427 | ||
| 424 | adev->acp.acp_cell[2].name = "designware-i2s"; | 428 | adev->acp.acp_cell[2].name = "designware-i2s"; |
| 429 | adev->acp.acp_cell[2].id = 2; | ||
| 425 | adev->acp.acp_cell[2].num_resources = 1; | 430 | adev->acp.acp_cell[2].num_resources = 1; |
| 426 | adev->acp.acp_cell[2].resources = &adev->acp.acp_res[2]; | 431 | adev->acp.acp_cell[2].resources = &adev->acp.acp_res[2]; |
| 427 | adev->acp.acp_cell[2].platform_data = &i2s_pdata[1]; | 432 | adev->acp.acp_cell[2].platform_data = &i2s_pdata[1]; |
| 428 | adev->acp.acp_cell[2].pdata_size = sizeof(struct i2s_platform_data); | 433 | adev->acp.acp_cell[2].pdata_size = sizeof(struct i2s_platform_data); |
| 429 | 434 | ||
| 430 | adev->acp.acp_cell[3].name = "designware-i2s"; | 435 | adev->acp.acp_cell[3].name = "designware-i2s"; |
| 436 | adev->acp.acp_cell[3].id = 3; | ||
| 431 | adev->acp.acp_cell[3].num_resources = 1; | 437 | adev->acp.acp_cell[3].num_resources = 1; |
| 432 | adev->acp.acp_cell[3].resources = &adev->acp.acp_res[3]; | 438 | adev->acp.acp_cell[3].resources = &adev->acp.acp_res[3]; |
| 433 | adev->acp.acp_cell[3].platform_data = &i2s_pdata[2]; | 439 | adev->acp.acp_cell[3].platform_data = &i2s_pdata[2]; |
| 434 | adev->acp.acp_cell[3].pdata_size = sizeof(struct i2s_platform_data); | 440 | adev->acp.acp_cell[3].pdata_size = sizeof(struct i2s_platform_data); |
| 435 | 441 | ||
| 436 | r = mfd_add_hotplug_devices(adev->acp.parent, adev->acp.acp_cell, ACP_DEVS); | 442 | r = mfd_add_devices(adev->acp.parent, 0, adev->acp.acp_cell, ACP_DEVS, NULL, 0, NULL); |
| 437 | if (r) | 443 | if (r) |
| 438 | goto failure; | 444 | goto failure; |
| 439 | 445 | ||
| @@ -507,7 +507,6 @@ static int amdgpu_atif_handler(struct amdgpu_device *adev, · 506 unmodified lines | |||
| 507 | pm_runtime_get_sync(adev_to_drm(adev)->dev); | 507 | pm_runtime_get_sync(adev_to_drm(adev)->dev); |
| 508 | /* Just fire off a uevent and let userspace tell us what to do */ | 508 | /* Just fire off a uevent and let userspace tell us what to do */ |
| 509 | drm_helper_hpd_irq_event(adev_to_drm(adev)); | 509 | drm_helper_hpd_irq_event(adev_to_drm(adev)); |
| 510 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 511 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 510 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 512 | } | 511 | } |
| 513 | } | 512 | } |
| @@ -71,7 +71,7 @@ struct kgd_mem { · 70 unmodified lines | |||
| 71 | struct mutex lock; | 71 | struct mutex lock; |
| 72 | struct amdgpu_bo *bo; | 72 | struct amdgpu_bo *bo; |
| 73 | struct dma_buf *dmabuf; | 73 | struct dma_buf *dmabuf; |
| 74 | struct hmm_range *range; | 74 | struct amdgpu_hmm_range *range; |
| 75 | struct list_head attachments; | 75 | struct list_head attachments; |
| 76 | /* protected by amdkfd_process_info.lock */ | 76 | /* protected by amdkfd_process_info.lock */ |
| 77 | struct list_head validate_list; | 77 | struct list_head validate_list; |
| @@ -1057,7 +1057,7 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr, · 1056 unmodified lines | |||
| 1057 | struct amdkfd_process_info *process_info = mem->process_info; | 1057 | struct amdkfd_process_info *process_info = mem->process_info; |
| 1058 | struct amdgpu_bo *bo = mem->bo; | 1058 | struct amdgpu_bo *bo = mem->bo; |
| 1059 | struct ttm_operation_ctx ctx = { true, false }; | 1059 | struct ttm_operation_ctx ctx = { true, false }; |
| 1060 | struct hmm_range *range; | 1060 | struct amdgpu_hmm_range *range; |
| 1061 | int ret = 0; | 1061 | int ret = 0; |
| 1062 | 1062 | ||
| 1063 | mutex_lock(&process_info->lock); | 1063 | mutex_lock(&process_info->lock); |
| @@ -1089,8 +1089,15 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr, · 25 unmodified lines | |||
| 1089 | return 0; | 1089 | return 0; |
| 1090 | } | 1090 | } |
| 1091 | 1091 | ||
| 1092 | ret = amdgpu_ttm_tt_get_user_pages(bo, &range); | 1092 | range = amdgpu_hmm_range_alloc(NULL); |
| 1093 | if (unlikely(!range)) { | ||
| 1094 | ret = -ENOMEM; | ||
| 1095 | goto unregister_out; | ||
| 1096 | } | ||
| 1097 | |||
| 1098 | ret = amdgpu_ttm_tt_get_user_pages(bo, range); | ||
| 1093 | if (ret) { | 1099 | if (ret) { |
| 1100 | amdgpu_hmm_range_free(range); | ||
| 1094 | if (ret == -EAGAIN) | 1101 | if (ret == -EAGAIN) |
| 1095 | pr_debug("Failed to get user pages, try again\n"); | 1102 | pr_debug("Failed to get user pages, try again\n"); |
| 1096 | else | 1103 | else |
| @@ -1113,7 +1120,7 @@ static int init_user_pages(struct kgd_mem *mem, uint64_t user_addr, · 16 unmodified lines | |||
| 1113 | amdgpu_bo_unreserve(bo); | 1120 | amdgpu_bo_unreserve(bo); |
| 1114 | 1121 | ||
| 1115 | release_out: | 1122 | release_out: |
| 1116 | amdgpu_ttm_tt_get_user_pages_done(bo->tbo.ttm, range); | 1123 | amdgpu_hmm_range_free(range); |
| 1117 | unregister_out: | 1124 | unregister_out: |
| 1118 | if (ret) | 1125 | if (ret) |
| 1119 | amdgpu_hmm_unregister(bo); | 1126 | amdgpu_hmm_unregister(bo); |
| @@ -1920,7 +1927,7 @@ int amdgpu_amdkfd_gpuvm_free_memory_of_gpu( · 800 unmodified lines | |||
| 1920 | if (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm)) { | 1927 | if (amdgpu_ttm_tt_get_usermm(mem->bo->tbo.ttm)) { |
| 1921 | amdgpu_hmm_unregister(mem->bo); | 1928 | amdgpu_hmm_unregister(mem->bo); |
| 1922 | mutex_lock(&process_info->notifier_lock); | 1929 | mutex_lock(&process_info->notifier_lock); |
| 1923 | amdgpu_ttm_tt_discard_user_pages(mem->bo->tbo.ttm, mem->range); | 1930 | amdgpu_hmm_range_free(mem->range); |
| 1924 | mutex_unlock(&process_info->notifier_lock); | 1931 | mutex_unlock(&process_info->notifier_lock); |
| 1925 | } | 1932 | } |
| 1926 | 1933 | ||
| @@ -1958,9 +1965,7 @@ int amdgpu_amdkfd_gpuvm_free_memory_of_gpu( · 31 unmodified lines | |||
| 1958 | */ | 1965 | */ |
| 1959 | if (size) { | 1966 | if (size) { |
| 1960 | if (!is_imported && | 1967 | if (!is_imported && |
| 1961 | (mem->bo->preferred_domains == AMDGPU_GEM_DOMAIN_VRAM || | 1968 | mem->alloc_flags & KFD_IOC_ALLOC_MEM_FLAGS_VRAM) |
| 1962 | (adev->apu_prefer_gtt && | ||
| 1963 | mem->bo->preferred_domains == AMDGPU_GEM_DOMAIN_GTT))) | ||
| 1964 | *size = bo_size; | 1969 | *size = bo_size; |
| 1965 | else | 1970 | else |
| 1966 | *size = 0; | 1971 | *size = 0; |
| @@ -2546,7 +2551,7 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info, · 579 unmodified lines | |||
| 2546 | 2551 | ||
| 2547 | bo = mem->bo; | 2552 | bo = mem->bo; |
| 2548 | 2553 | ||
| 2549 | amdgpu_ttm_tt_discard_user_pages(bo->tbo.ttm, mem->range); | 2554 | amdgpu_hmm_range_free(mem->range); |
| 2550 | mem->range = NULL; | 2555 | mem->range = NULL; |
| 2551 | 2556 | ||
| 2552 | /* BO reservations and getting user pages (hmm_range_fault) | 2557 | /* BO reservations and getting user pages (hmm_range_fault) |
| @@ -2570,9 +2575,14 @@ static int update_invalid_user_pages(struct amdkfd_process_info *process_info, · 17 unmodified lines | |||
| 2570 | } | 2575 | } |
| 2571 | } | 2576 | } |
| 2572 | 2577 | ||
| 2578 | mem->range = amdgpu_hmm_range_alloc(NULL); | ||
| 2579 | if (unlikely(!mem->range)) | ||
| 2580 | return -ENOMEM; | ||
| 2573 | /* Get updated user pages */ | 2581 | /* Get updated user pages */ |
| 2574 | ret = amdgpu_ttm_tt_get_user_pages(bo, &mem->range); | 2582 | ret = amdgpu_ttm_tt_get_user_pages(bo, mem->range); |
| 2575 | if (ret) { | 2583 | if (ret) { |
| 2584 | amdgpu_hmm_range_free(mem->range); | ||
| 2585 | mem->range = NULL; | ||
| 2576 | pr_debug("Failed %d to get user pages\n", ret); | 2586 | pr_debug("Failed %d to get user pages\n", ret); |
| 2577 | 2587 | ||
| 2578 | /* Return -EFAULT bad address error as success. It will | 2588 | /* Return -EFAULT bad address error as success. It will |
| @@ -2745,8 +2755,8 @@ static int confirm_valid_user_pages_locked(struct amdkfd_process_info *process_i · 166 unmodified lines | |||
| 2745 | continue; | 2755 | continue; |
| 2746 | 2756 | ||
| 2747 | /* Only check mem with hmm range associated */ | 2757 | /* Only check mem with hmm range associated */ |
| 2748 | valid = amdgpu_ttm_tt_get_user_pages_done( | 2758 | valid = amdgpu_hmm_range_valid(mem->range); |
| 2749 | mem->bo->tbo.ttm, mem->range); | 2759 | amdgpu_hmm_range_free(mem->range); |
| 2750 | 2760 | ||
| 2751 | mem->range = NULL; | 2761 | mem->range = NULL; |
| 2752 | if (!valid) { | 2762 | if (!valid) { |
| @@ -181,19 +181,22 @@ int amdgpu_atomfirmware_allocate_fb_scratch(struct amdgpu_device *adev) · 180 unmodified lines | |||
| 181 | u8 frev, crev; | 181 | u8 frev, crev; |
| 182 | int usage_bytes = 0; | 182 | int usage_bytes = 0; |
| 183 | 183 | ||
| 184 | if (amdgpu_atom_parse_data_header(ctx, index, NULL, &frev, &crev, &data_offset)) { | 184 | /* Skip atomfirmware allocation for SRIOV VFs when dynamic crit regn is enabled */ |
| 185 | if (frev == 2 && crev == 1) { | 185 | if (!(amdgpu_sriov_vf(adev) && adev->virt.is_dynamic_crit_regn_enabled)) { |
| 186 | fw_usage_v2_1 = | 186 | if (amdgpu_atom_parse_data_header(ctx, index, NULL, &frev, &crev, &data_offset)) { |
| 187 | (struct vram_usagebyfirmware_v2_1 *)(ctx->bios + data_offset); | 187 | if (frev == 2 && crev == 1) { |
| 188 | amdgpu_atomfirmware_allocate_fb_v2_1(adev, | 188 | fw_usage_v2_1 = |
| 189 | fw_usage_v2_1, | 189 | (struct vram_usagebyfirmware_v2_1 *)(ctx->bios + data_offset); |
| 190 | &usage_bytes); | 190 | amdgpu_atomfirmware_allocate_fb_v2_1(adev, |
| 191 | } else if (frev >= 2 && crev >= 2) { | 191 | fw_usage_v2_1, |
| 192 | fw_usage_v2_2 = | 192 | &usage_bytes); |
| 193 | (struct vram_usagebyfirmware_v2_2 *)(ctx->bios + data_offset); | 193 | } else if (frev >= 2 && crev >= 2) { |
| 194 | amdgpu_atomfirmware_allocate_fb_v2_2(adev, | 194 | fw_usage_v2_2 = |
| 195 | fw_usage_v2_2, | 195 | (struct vram_usagebyfirmware_v2_2 *)(ctx->bios + data_offset); |
| 196 | &usage_bytes); | 196 | amdgpu_atomfirmware_allocate_fb_v2_2(adev, |
| 197 | fw_usage_v2_2, | ||
| 198 | &usage_bytes); | ||
| 199 | } | ||
| 197 | } | 200 | } |
| 198 | } | 201 | } |
| 199 | 202 | ||
| @@ -96,13 +96,14 @@ void amdgpu_bios_release(struct amdgpu_device *adev) · 95 unmodified lines | |||
| 96 | * part of the system bios. On boot, the system bios puts a | 96 | * part of the system bios. On boot, the system bios puts a |
| 97 | * copy of the igp rom at the start of vram if a discrete card is | 97 | * copy of the igp rom at the start of vram if a discrete card is |
| 98 | * present. | 98 | * present. |
| 99 | * For SR-IOV, the vbios image is also put in VRAM in the VF. | 99 | * For SR-IOV, if dynamic critical region is not enabled, |
| 100 | * the vbios image is also put at the start of VRAM in the VF. | ||
| 100 | */ | 101 | */ |
| 101 | static bool amdgpu_read_bios_from_vram(struct amdgpu_device *adev) | 102 | static bool amdgpu_read_bios_from_vram(struct amdgpu_device *adev) |
| 102 | { | 103 | { |
| 103 | uint8_t __iomem *bios; | 104 | uint8_t __iomem *bios = NULL; |
| 104 | resource_size_t vram_base; | 105 | resource_size_t vram_base; |
| 105 | resource_size_t size = 256 * 1024; /* ??? */ | 106 | u32 size = 256U * 1024U; /* ??? */ |
| 106 | 107 | ||
| 107 | if (!(adev->flags & AMD_IS_APU)) | 108 | if (!(adev->flags & AMD_IS_APU)) |
| 108 | if (amdgpu_device_need_post(adev)) | 109 | if (amdgpu_device_need_post(adev)) |
| @@ -114,18 +115,33 @@ static bool amdgpu_read_bios_from_vram(struct amdgpu_device *adev) · 5 unmodified lines | |||
| 114 | 115 | ||
| 115 | adev->bios = NULL; | 116 | adev->bios = NULL; |
| 116 | vram_base = pci_resource_start(adev->pdev, 0); | 117 | vram_base = pci_resource_start(adev->pdev, 0); |
| 117 | bios = ioremap_wc(vram_base, size); | ||
| 118 | if (!bios) | ||
| 119 | return false; | ||
| 120 | 118 | ||
| 121 | adev->bios = kmalloc(size, GFP_KERNEL); | 119 | adev->bios = kmalloc(size, GFP_KERNEL); |
| 122 | if (!adev->bios) { | 120 | if (!adev->bios) |
| 123 | iounmap(bios); | ||
| 124 | return false; | 121 | return false; |
| 122 | |||
| 123 | /* For SRIOV with dynamic critical region is enabled, | ||
| 124 | * the vbios image is put at a dynamic offset of VRAM in the VF. | ||
| 125 | * If dynamic critical region is disabled, follow the existing logic as on baremetal. | ||
| 126 | */ | ||
| 127 | if (amdgpu_sriov_vf(adev) && adev->virt.is_dynamic_crit_regn_enabled) { | ||
| 128 | if (amdgpu_virt_get_dynamic_data_info(adev, | ||
| 129 | AMD_SRIOV_MSG_VBIOS_IMG_TABLE_ID, adev->bios, &size)) { | ||
| 130 | amdgpu_bios_release(adev); | ||
| 131 | return false; | ||
| 132 | } | ||
| 133 | } else { | ||
| 134 | bios = ioremap_wc(vram_base, size); | ||
| 135 | if (!bios) { | ||
| 136 | amdgpu_bios_release(adev); | ||
| 137 | return false; | ||
| 138 | } | ||
| 139 | |||
| 140 | memcpy_fromio(adev->bios, bios, size); | ||
| 141 | iounmap(bios); | ||
| 125 | } | 142 | } |
| 143 | |||
| 126 | adev->bios_size = size; | 144 | adev->bios_size = size; |
| 127 | memcpy_fromio(adev->bios, bios, size); | ||
| 128 | iounmap(bios); | ||
| 129 | 145 | ||
| 130 | if (!check_atom_bios(adev, size)) { | 146 | if (!check_atom_bios(adev, size)) { |
| 131 | amdgpu_bios_release(adev); | 147 | amdgpu_bios_release(adev); |
| @@ -38,7 +38,7 @@ struct amdgpu_bo_list_entry { · 37 unmodified lines | |||
| 38 | struct amdgpu_bo *bo; | 38 | struct amdgpu_bo *bo; |
| 39 | struct amdgpu_bo_va *bo_va; | 39 | struct amdgpu_bo_va *bo_va; |
| 40 | uint32_t priority; | 40 | uint32_t priority; |
| 41 | struct hmm_range *range; | 41 | struct amdgpu_hmm_range *range; |
| 42 | bool user_invalidated; | 42 | bool user_invalidated; |
| 43 | }; | 43 | }; |
| 44 | 44 | ||
| @@ -734,10 +734,8 @@ amdgpu_connector_lvds_detect(struct drm_connector *connector, bool force) · 733 unmodified lines | |||
| 734 | 734 | ||
| 735 | amdgpu_connector_update_scratch_regs(connector, ret); | 735 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 736 | 736 | ||
| 737 | if (!drm_kms_helper_is_poll_worker()) { | 737 | if (!drm_kms_helper_is_poll_worker()) |
| 738 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 739 | pm_runtime_put_autosuspend(connector->dev->dev); | 738 | pm_runtime_put_autosuspend(connector->dev->dev); |
| 740 | } | ||
| 741 | 739 | ||
| 742 | return ret; | 740 | return ret; |
| 743 | } | 741 | } |
| @@ -919,10 +917,8 @@ amdgpu_connector_vga_detect(struct drm_connector *connector, bool force) · 175 unmodified lines | |||
| 919 | amdgpu_connector_update_scratch_regs(connector, ret); | 917 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 920 | 918 | ||
| 921 | out: | 919 | out: |
| 922 | if (!drm_kms_helper_is_poll_worker()) { | 920 | if (!drm_kms_helper_is_poll_worker()) |
| 923 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 924 | pm_runtime_put_autosuspend(connector->dev->dev); | 921 | pm_runtime_put_autosuspend(connector->dev->dev); |
| 925 | } | ||
| 926 | 922 | ||
| 927 | return ret; | 923 | return ret; |
| 928 | } | 924 | } |
| @@ -1146,10 +1142,8 @@ out: · 217 unmodified lines | |||
| 1146 | amdgpu_connector_update_scratch_regs(connector, ret); | 1142 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 1147 | 1143 | ||
| 1148 | exit: | 1144 | exit: |
| 1149 | if (!drm_kms_helper_is_poll_worker()) { | 1145 | if (!drm_kms_helper_is_poll_worker()) |
| 1150 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 1151 | pm_runtime_put_autosuspend(connector->dev->dev); | 1146 | pm_runtime_put_autosuspend(connector->dev->dev); |
| 1152 | } | ||
| 1153 | 1147 | ||
| 1154 | return ret; | 1148 | return ret; |
| 1155 | } | 1149 | } |
| @@ -1486,10 +1480,8 @@ amdgpu_connector_dp_detect(struct drm_connector *connector, bool force) · 330 unmodified lines | |||
| 1486 | 1480 | ||
| 1487 | amdgpu_connector_update_scratch_regs(connector, ret); | 1481 | amdgpu_connector_update_scratch_regs(connector, ret); |
| 1488 | out: | 1482 | out: |
| 1489 | if (!drm_kms_helper_is_poll_worker()) { | 1483 | if (!drm_kms_helper_is_poll_worker()) |
| 1490 | pm_runtime_mark_last_busy(connector->dev->dev); | ||
| 1491 | pm_runtime_put_autosuspend(connector->dev->dev); | 1484 | pm_runtime_put_autosuspend(connector->dev->dev); |
| 1492 | } | ||
| 1493 | 1485 | ||
| 1494 | if (connector->connector_type == DRM_MODE_CONNECTOR_DisplayPort || | 1486 | if (connector->connector_type == DRM_MODE_CONNECTOR_DisplayPort || |
| 1495 | connector->connector_type == DRM_MODE_CONNECTOR_eDP) | 1487 | connector->connector_type == DRM_MODE_CONNECTOR_eDP) |
| @@ -29,7 +29,6 @@ · 28 unmodified lines | |||
| 29 | #include <linux/pagemap.h> | 29 | #include <linux/pagemap.h> |
| 30 | #include <linux/sync_file.h> | 30 | #include <linux/sync_file.h> |
| 31 | #include <linux/dma-buf.h> | 31 | #include <linux/dma-buf.h> |
| 32 | #include <linux/hmm.h> | ||
| 33 | 32 | ||
| 34 | #include <drm/amdgpu_drm.h> | 33 | #include <drm/amdgpu_drm.h> |
| 35 | #include <drm/drm_syncobj.h> | 34 | #include <drm/drm_syncobj.h> |
| @@ -41,6 +40,7 @@ · 5 unmodified lines | |||
| 41 | #include "amdgpu_gmc.h" | 40 | #include "amdgpu_gmc.h" |
| 42 | #include "amdgpu_gem.h" | 41 | #include "amdgpu_gem.h" |
| 43 | #include "amdgpu_ras.h" | 42 | #include "amdgpu_ras.h" |
| 43 | #include "amdgpu_hmm.h" | ||
| 44 | 44 | ||
| 45 | static int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, | 45 | static int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, |
| 46 | struct amdgpu_device *adev, | 46 | struct amdgpu_device *adev, |
| @@ -891,12 +891,17 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 844 unmodified lines | |||
| 891 | bool userpage_invalidated = false; | 891 | bool userpage_invalidated = false; |
| 892 | struct amdgpu_bo *bo = e->bo; | 892 | struct amdgpu_bo *bo = e->bo; |
| 893 | 893 | ||
| 894 | r = amdgpu_ttm_tt_get_user_pages(bo, &e->range); | 894 | e->range = amdgpu_hmm_range_alloc(NULL); |
| 895 | if (unlikely(!e->range)) | ||
| 896 | return -ENOMEM; | ||
| 897 | |||
| 898 | r = amdgpu_ttm_tt_get_user_pages(bo, e->range); | ||
| 895 | if (r) | 899 | if (r) |
| 896 | goto out_free_user_pages; | 900 | goto out_free_user_pages; |
| 897 | 901 | ||
| 898 | for (i = 0; i < bo->tbo.ttm->num_pages; i++) { | 902 | for (i = 0; i < bo->tbo.ttm->num_pages; i++) { |
| 899 | if (bo->tbo.ttm->pages[i] != hmm_pfn_to_page(e->range->hmm_pfns[i])) { | 903 | if (bo->tbo.ttm->pages[i] != |
| 904 | hmm_pfn_to_page(e->range->hmm_range.hmm_pfns[i])) { | ||
| 900 | userpage_invalidated = true; | 905 | userpage_invalidated = true; |
| 901 | break; | 906 | break; |
| 902 | } | 907 | } |
| @@ -990,9 +995,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · 87 unmodified lines | |||
| 990 | 995 | ||
| 991 | out_free_user_pages: | 996 | out_free_user_pages: |
| 992 | amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) { | 997 | amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) { |
| 993 | struct amdgpu_bo *bo = e->bo; | 998 | amdgpu_hmm_range_free(e->range); |
| 994 | |||
| 995 | amdgpu_ttm_tt_get_user_pages_done(bo->tbo.ttm, e->range); | ||
| 996 | e->range = NULL; | 999 | e->range = NULL; |
| 997 | } | 1000 | } |
| 998 | mutex_unlock(&p->bo_list->bo_list_mutex); | 1001 | mutex_unlock(&p->bo_list->bo_list_mutex); |
| @@ -1323,8 +1326,8 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p, · 324 unmodified lines | |||
| 1323 | */ | 1326 | */ |
| 1324 | r = 0; | 1327 | r = 0; |
| 1325 | amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) { | 1328 | amdgpu_bo_list_for_each_userptr_entry(e, p->bo_list) { |
| 1326 | r |= !amdgpu_ttm_tt_get_user_pages_done(e->bo->tbo.ttm, | 1329 | r |= !amdgpu_hmm_range_valid(e->range); |
| 1327 | e->range); | 1330 | amdgpu_hmm_range_free(e->range); |
| 1328 | e->range = NULL; | 1331 | e->range = NULL; |
| 1329 | } | 1332 | } |
| 1330 | if (r) { | 1333 | if (r) { |
| @@ -129,7 +129,6 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f, · 128 unmodified lines | |||
| 129 | if (use_bank) { | 129 | if (use_bank) { |
| 130 | if ((sh_bank != 0xFFFFFFFF && sh_bank >= adev->gfx.config.max_sh_per_se) || | 130 | if ((sh_bank != 0xFFFFFFFF && sh_bank >= adev->gfx.config.max_sh_per_se) || |
| 131 | (se_bank != 0xFFFFFFFF && se_bank >= adev->gfx.config.max_shader_engines)) { | 131 | (se_bank != 0xFFFFFFFF && se_bank >= adev->gfx.config.max_shader_engines)) { |
| 132 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 133 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 132 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 134 | amdgpu_virt_disable_access_debugfs(adev); | 133 | amdgpu_virt_disable_access_debugfs(adev); |
| 135 | return -EINVAL; | 134 | return -EINVAL; |
| @@ -179,7 +178,6 @@ end: · 43 unmodified lines | |||
| 179 | if (pm_pg_lock) | 178 | if (pm_pg_lock) |
| 180 | mutex_unlock(&adev->pm.mutex); | 179 | mutex_unlock(&adev->pm.mutex); |
| 181 | 180 | ||
| 182 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 183 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 181 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 184 | 182 | ||
| 185 | amdgpu_virt_disable_access_debugfs(adev); | 183 | amdgpu_virt_disable_access_debugfs(adev); |
| @@ -255,7 +253,6 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off · 69 unmodified lines | |||
| 255 | if (rd->id.use_grbm) { | 253 | if (rd->id.use_grbm) { |
| 256 | if ((rd->id.grbm.sh != 0xFFFFFFFF && rd->id.grbm.sh >= adev->gfx.config.max_sh_per_se) || | 254 | if ((rd->id.grbm.sh != 0xFFFFFFFF && rd->id.grbm.sh >= adev->gfx.config.max_sh_per_se) || |
| 257 | (rd->id.grbm.se != 0xFFFFFFFF && rd->id.grbm.se >= adev->gfx.config.max_shader_engines)) { | 255 | (rd->id.grbm.se != 0xFFFFFFFF && rd->id.grbm.se >= adev->gfx.config.max_shader_engines)) { |
| 258 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 259 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 256 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 260 | amdgpu_virt_disable_access_debugfs(adev); | 257 | amdgpu_virt_disable_access_debugfs(adev); |
| 261 | mutex_unlock(&rd->lock); | 258 | mutex_unlock(&rd->lock); |
| @@ -310,7 +307,6 @@ end: · 48 unmodified lines | |||
| 310 | 307 | ||
| 311 | mutex_unlock(&rd->lock); | 308 | mutex_unlock(&rd->lock); |
| 312 | 309 | ||
| 313 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 314 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 310 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 315 | 311 | ||
| 316 | amdgpu_virt_disable_access_debugfs(adev); | 312 | amdgpu_virt_disable_access_debugfs(adev); |
| @@ -446,7 +442,6 @@ static ssize_t amdgpu_debugfs_gprwave_read(struct file *f, char __user *buf, siz · 129 unmodified lines | |||
| 446 | amdgpu_gfx_select_se_sh(adev, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, rd->id.xcc_id); | 442 | amdgpu_gfx_select_se_sh(adev, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, rd->id.xcc_id); |
| 447 | mutex_unlock(&adev->grbm_idx_mutex); | 443 | mutex_unlock(&adev->grbm_idx_mutex); |
| 448 | 444 | ||
| 449 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 450 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 445 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 451 | 446 | ||
| 452 | if (!x) { | 447 | if (!x) { |
| @@ -557,7 +552,6 @@ static ssize_t amdgpu_debugfs_regs_pcie_read(struct file *f, char __user *buf, · 104 unmodified lines | |||
| 557 | 552 | ||
| 558 | r = result; | 553 | r = result; |
| 559 | out: | 554 | out: |
| 560 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 561 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 555 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 562 | amdgpu_virt_disable_access_debugfs(adev); | 556 | amdgpu_virt_disable_access_debugfs(adev); |
| 563 | return r; | 557 | return r; |
| @@ -617,7 +611,6 @@ static ssize_t amdgpu_debugfs_regs_pcie_write(struct file *f, const char __user · 53 unmodified lines | |||
| 617 | 611 | ||
| 618 | r = result; | 612 | r = result; |
| 619 | out: | 613 | out: |
| 620 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 621 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 614 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 622 | amdgpu_virt_disable_access_debugfs(adev); | 615 | amdgpu_virt_disable_access_debugfs(adev); |
| 623 | return r; | 616 | return r; |
| @@ -676,7 +669,6 @@ static ssize_t amdgpu_debugfs_regs_didt_read(struct file *f, char __user *buf, · 52 unmodified lines | |||
| 676 | 669 | ||
| 677 | r = result; | 670 | r = result; |
| 678 | out: | 671 | out: |
| 679 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 680 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 672 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 681 | amdgpu_virt_disable_access_debugfs(adev); | 673 | amdgpu_virt_disable_access_debugfs(adev); |
| 682 | return r; | 674 | return r; |
| @@ -736,7 +728,6 @@ static ssize_t amdgpu_debugfs_regs_didt_write(struct file *f, const char __user · 53 unmodified lines | |||
| 736 | 728 | ||
| 737 | r = result; | 729 | r = result; |
| 738 | out: | 730 | out: |
| 739 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 740 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 731 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 741 | amdgpu_virt_disable_access_debugfs(adev); | 732 | amdgpu_virt_disable_access_debugfs(adev); |
| 742 | return r; | 733 | return r; |
| @@ -795,7 +786,6 @@ static ssize_t amdgpu_debugfs_regs_smc_read(struct file *f, char __user *buf, · 52 unmodified lines | |||
| 795 | 786 | ||
| 796 | r = result; | 787 | r = result; |
| 797 | out: | 788 | out: |
| 798 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 799 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 789 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 800 | amdgpu_virt_disable_access_debugfs(adev); | 790 | amdgpu_virt_disable_access_debugfs(adev); |
| 801 | return r; | 791 | return r; |
| @@ -855,7 +845,6 @@ static ssize_t amdgpu_debugfs_regs_smc_write(struct file *f, const char __user * · 53 unmodified lines | |||
| 855 | 845 | ||
| 856 | r = result; | 846 | r = result; |
| 857 | out: | 847 | out: |
| 858 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 859 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 848 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 860 | amdgpu_virt_disable_access_debugfs(adev); | 849 | amdgpu_virt_disable_access_debugfs(adev); |
| 861 | return r; | 850 | return r; |
| @@ -1003,7 +992,6 @@ static ssize_t amdgpu_debugfs_sensor_read(struct file *f, char __user *buf, · 141 unmodified lines | |||
| 1003 | 992 | ||
| 1004 | r = amdgpu_dpm_read_sensor(adev, idx, &values[0], &valuesize); | 993 | r = amdgpu_dpm_read_sensor(adev, idx, &values[0], &valuesize); |
| 1005 | 994 | ||
| 1006 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1007 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 995 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1008 | 996 | ||
| 1009 | if (r) { | 997 | if (r) { |
| @@ -1094,7 +1082,6 @@ static ssize_t amdgpu_debugfs_wave_read(struct file *f, char __user *buf, · 84 unmodified lines | |||
| 1094 | amdgpu_gfx_select_se_sh(adev, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0); | 1082 | amdgpu_gfx_select_se_sh(adev, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0); |
| 1095 | mutex_unlock(&adev->grbm_idx_mutex); | 1083 | mutex_unlock(&adev->grbm_idx_mutex); |
| 1096 | 1084 | ||
| 1097 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1098 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1085 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1099 | 1086 | ||
| 1100 | if (!x) { | 1087 | if (!x) { |
| @@ -1192,7 +1179,6 @@ static ssize_t amdgpu_debugfs_gpr_read(struct file *f, char __user *buf, · 91 unmodified lines | |||
| 1192 | amdgpu_gfx_select_se_sh(adev, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0); | 1179 | amdgpu_gfx_select_se_sh(adev, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0); |
| 1193 | mutex_unlock(&adev->grbm_idx_mutex); | 1180 | mutex_unlock(&adev->grbm_idx_mutex); |
| 1194 | 1181 | ||
| 1195 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1196 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1182 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1197 | 1183 | ||
| 1198 | while (size) { | 1184 | while (size) { |
| @@ -1266,7 +1252,6 @@ static ssize_t amdgpu_debugfs_gfxoff_residency_read(struct file *f, char __user · 67 unmodified lines | |||
| 1266 | 1252 | ||
| 1267 | r = result; | 1253 | r = result; |
| 1268 | out: | 1254 | out: |
| 1269 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1270 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1255 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1271 | 1256 | ||
| 1272 | return r; | 1257 | return r; |
| @@ -1315,7 +1300,6 @@ static ssize_t amdgpu_debugfs_gfxoff_residency_write(struct file *f, const char · 42 unmodified lines | |||
| 1315 | 1300 | ||
| 1316 | r = result; | 1301 | r = result; |
| 1317 | out: | 1302 | out: |
| 1318 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1319 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1303 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1320 | 1304 | ||
| 1321 | return r; | 1305 | return r; |
| @@ -1365,7 +1349,6 @@ static ssize_t amdgpu_debugfs_gfxoff_count_read(struct file *f, char __user *buf · 43 unmodified lines | |||
| 1365 | 1349 | ||
| 1366 | r = result; | 1350 | r = result; |
| 1367 | out: | 1351 | out: |
| 1368 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1369 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1352 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1370 | 1353 | ||
| 1371 | return r; | 1354 | return r; |
| @@ -1414,7 +1397,6 @@ static ssize_t amdgpu_debugfs_gfxoff_write(struct file *f, const char __user *bu · 42 unmodified lines | |||
| 1414 | 1397 | ||
| 1415 | r = result; | 1398 | r = result; |
| 1416 | out: | 1399 | out: |
| 1417 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1418 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1400 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1419 | 1401 | ||
| 1420 | return r; | 1402 | return r; |
| @@ -1460,7 +1442,6 @@ static ssize_t amdgpu_debugfs_gfxoff_read(struct file *f, char __user *buf, · 39 unmodified lines | |||
| 1460 | 1442 | ||
| 1461 | r = result; | 1443 | r = result; |
| 1462 | out: | 1444 | out: |
| 1463 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1464 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1445 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1465 | 1446 | ||
| 1466 | return r; | 1447 | return r; |
| @@ -1501,7 +1482,6 @@ static ssize_t amdgpu_debugfs_gfxoff_status_read(struct file *f, char __user *bu · 34 unmodified lines | |||
| 1501 | 1482 | ||
| 1502 | r = result; | 1483 | r = result; |
| 1503 | out: | 1484 | out: |
| 1504 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 1505 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1485 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 1506 | 1486 | ||
| 1507 | return r; | 1487 | return r; |
| @@ -1701,7 +1681,6 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused) · 193 unmodified lines | |||
| 1701 | 1681 | ||
| 1702 | up_write(&adev->reset_domain->sem); | 1682 | up_write(&adev->reset_domain->sem); |
| 1703 | 1683 | ||
| 1704 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1705 | pm_runtime_put_autosuspend(dev->dev); | 1684 | pm_runtime_put_autosuspend(dev->dev); |
| 1706 | 1685 | ||
| 1707 | return 0; | 1686 | return 0; |
| @@ -1721,7 +1700,6 @@ static int amdgpu_debugfs_evict_vram(void *data, u64 *val) · 13 unmodified lines | |||
| 1721 | 1700 | ||
| 1722 | *val = amdgpu_ttm_evict_resources(adev, TTM_PL_VRAM); | 1701 | *val = amdgpu_ttm_evict_resources(adev, TTM_PL_VRAM); |
| 1723 | 1702 | ||
| 1724 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1725 | pm_runtime_put_autosuspend(dev->dev); | 1703 | pm_runtime_put_autosuspend(dev->dev); |
| 1726 | 1704 | ||
| 1727 | return 0; | 1705 | return 0; |
| @@ -1742,7 +1720,6 @@ static int amdgpu_debugfs_evict_gtt(void *data, u64 *val) · 14 unmodified lines | |||
| 1742 | 1720 | ||
| 1743 | *val = amdgpu_ttm_evict_resources(adev, TTM_PL_TT); | 1721 | *val = amdgpu_ttm_evict_resources(adev, TTM_PL_TT); |
| 1744 | 1722 | ||
| 1745 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1746 | pm_runtime_put_autosuspend(dev->dev); | 1723 | pm_runtime_put_autosuspend(dev->dev); |
| 1747 | 1724 | ||
| 1748 | return 0; | 1725 | return 0; |
| @@ -1762,7 +1739,6 @@ static int amdgpu_debugfs_benchmark(void *data, u64 val) · 13 unmodified lines | |||
| 1762 | 1739 | ||
| 1763 | r = amdgpu_benchmark(adev, val); | 1740 | r = amdgpu_benchmark(adev, val); |
| 1764 | 1741 | ||
| 1765 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1766 | pm_runtime_put_autosuspend(dev->dev); | 1742 | pm_runtime_put_autosuspend(dev->dev); |
| 1767 | 1743 | ||
| 1768 | return r; | 1744 | return r; |
| @@ -1902,7 +1878,7 @@ no_preempt: · 133 unmodified lines | |||
| 1902 | continue; | 1878 | continue; |
| 1903 | } | 1879 | } |
| 1904 | job = to_amdgpu_job(s_job); | 1880 | job = to_amdgpu_job(s_job); |
| 1905 | if (preempted && (&job->hw_fence.base) == fence) | 1881 | if (preempted && (&job->hw_fence->base) == fence) |
| 1906 | /* mark the job as preempted */ | 1882 | /* mark the job as preempted */ |
| 1907 | job->preemption_status |= AMDGPU_IB_PREEMPTED; | 1883 | job->preemption_status |= AMDGPU_IB_PREEMPTED; |
| 1908 | } | 1884 | } |
| @@ -2014,7 +1990,6 @@ static int amdgpu_debugfs_sclk_set(void *data, u64 val) · 105 unmodified lines | |||
| 2014 | ret = -EINVAL; | 1990 | ret = -EINVAL; |
| 2015 | 1991 | ||
| 2016 | out: | 1992 | out: |
| 2017 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 2018 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 1993 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 2019 | 1994 | ||
| 2020 | return ret; | 1995 | return ret; |
| @@ -2123,10 +2098,9 @@ int amdgpu_debugfs_init(struct amdgpu_device *adev) · 102 unmodified lines | |||
| 2123 | debugfs_create_blob("amdgpu_vbios", 0444, root, | 2098 | debugfs_create_blob("amdgpu_vbios", 0444, root, |
| 2124 | &adev->debugfs_vbios_blob); | 2099 | &adev->debugfs_vbios_blob); |
| 2125 | 2100 | ||
| 2126 | adev->debugfs_discovery_blob.data = adev->mman.discovery_bin; | 2101 | if (adev->discovery.debugfs_blob.size) |
| 2127 | adev->debugfs_discovery_blob.size = adev->mman.discovery_tmr_size; | 2102 | debugfs_create_blob("amdgpu_discovery", 0444, root, |
| 2128 | debugfs_create_blob("amdgpu_discovery", 0444, root, | 2103 | &adev->discovery.debugfs_blob); |
| 2129 | &adev->debugfs_discovery_blob); | ||
| 2130 | 2104 | ||
| 2131 | return 0; | 2105 | return 0; |
| 2132 | } | 2106 | } |
| @@ -71,6 +71,7 @@ · 70 unmodified lines | |||
| 71 | 71 | ||
| 72 | #include "amdgpu_xgmi.h" | 72 | #include "amdgpu_xgmi.h" |
| 73 | #include "amdgpu_ras.h" | 73 | #include "amdgpu_ras.h" |
| 74 | #include "amdgpu_ras_mgr.h" | ||
| 74 | #include "amdgpu_pmu.h" | 75 | #include "amdgpu_pmu.h" |
| 75 | #include "amdgpu_fru_eeprom.h" | 76 | #include "amdgpu_fru_eeprom.h" |
| 76 | #include "amdgpu_reset.h" | 77 | #include "amdgpu_reset.h" |
| @@ -179,6 +180,10 @@ struct amdgpu_init_level amdgpu_init_minimal_xgmi = { · 102 unmodified lines | |||
| 179 | BIT(AMD_IP_BLOCK_TYPE_PSP) | 180 | BIT(AMD_IP_BLOCK_TYPE_PSP) |
| 180 | }; | 181 | }; |
| 181 | 182 | ||
| 183 | static int amdgpu_device_ip_resume_phase1(struct amdgpu_device *adev); | ||
| 184 | static int amdgpu_device_ip_resume_phase2(struct amdgpu_device *adev); | ||
| 185 | static int amdgpu_device_ip_resume_phase3(struct amdgpu_device *adev); | ||
| 186 | |||
| 182 | static void amdgpu_device_load_switch_state(struct amdgpu_device *adev); | 187 | static void amdgpu_device_load_switch_state(struct amdgpu_device *adev); |
| 183 | 188 | ||
| 184 | static inline bool amdgpu_ip_member_of_hwini(struct amdgpu_device *adev, | 189 | static inline bool amdgpu_ip_member_of_hwini(struct amdgpu_device *adev, |
| @@ -2387,7 +2392,7 @@ int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, · 2202 unmodified lines | |||
| 2387 | } | 2392 | } |
| 2388 | 2393 | ||
| 2389 | /** | 2394 | /** |
| 2390 | * amdgpu_device_ip_is_valid - is the hardware IP enabled | 2395 | * amdgpu_device_ip_is_hw - is the hardware IP enabled |
| 2391 | * | 2396 | * |
| 2392 | * @adev: amdgpu_device pointer | 2397 | * @adev: amdgpu_device pointer |
| 2393 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | 2398 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) |
| @@ -2395,6 +2400,27 @@ int amdgpu_device_ip_wait_for_idle(struct amdgpu_device *adev, · 1 unmodified line | |||
| 2395 | * Check if the hardware IP is enable or not. | 2400 | * Check if the hardware IP is enable or not. |
| 2396 | * Returns true if it the IP is enable, false if not. | 2401 | * Returns true if it the IP is enable, false if not. |
| 2397 | */ | 2402 | */ |
| 2403 | bool amdgpu_device_ip_is_hw(struct amdgpu_device *adev, | ||
| 2404 | enum amd_ip_block_type block_type) | ||
| 2405 | { | ||
| 2406 | int i; | ||
| 2407 | |||
| 2408 | for (i = 0; i < adev->num_ip_blocks; i++) { | ||
| 2409 | if (adev->ip_blocks[i].version->type == block_type) | ||
| 2410 | return adev->ip_blocks[i].status.hw; | ||
| 2411 | } | ||
| 2412 | return false; | ||
| 2413 | } | ||
| 2414 | |||
| 2415 | /** | ||
| 2416 | * amdgpu_device_ip_is_valid - is the hardware IP valid | ||
| 2417 | * | ||
| 2418 | * @adev: amdgpu_device pointer | ||
| 2419 | * @block_type: Type of hardware IP (SMU, GFX, UVD, etc.) | ||
| 2420 | * | ||
| 2421 | * Check if the hardware IP is valid or not. | ||
| 2422 | * Returns true if it the IP is valid, false if not. | ||
| 2423 | */ | ||
| 2398 | bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev, | 2424 | bool amdgpu_device_ip_is_valid(struct amdgpu_device *adev, |
| 2399 | enum amd_ip_block_type block_type) | 2425 | enum amd_ip_block_type block_type) |
| 2400 | { | 2426 | { |
| @@ -2473,6 +2499,7 @@ static const char *ip_block_names[] = { · 72 unmodified lines | |||
| 2473 | [AMD_IP_BLOCK_TYPE_VPE] = "vpe", | 2499 | [AMD_IP_BLOCK_TYPE_VPE] = "vpe", |
| 2474 | [AMD_IP_BLOCK_TYPE_UMSCH_MM] = "umsch_mm", | 2500 | [AMD_IP_BLOCK_TYPE_UMSCH_MM] = "umsch_mm", |
| 2475 | [AMD_IP_BLOCK_TYPE_ISP] = "isp", | 2501 | [AMD_IP_BLOCK_TYPE_ISP] = "isp", |
| 2502 | [AMD_IP_BLOCK_TYPE_RAS] = "ras", | ||
| 2476 | }; | 2503 | }; |
| 2477 | 2504 | ||
| 2478 | static const char *ip_block_name(struct amdgpu_device *adev, enum amd_ip_block_type type) | 2505 | static const char *ip_block_name(struct amdgpu_device *adev, enum amd_ip_block_type type) |
| @@ -2633,12 +2660,12 @@ static int amdgpu_device_parse_gpu_info_fw(struct amdgpu_device *adev) · 154 unmodified lines | |||
| 2633 | chip_name = "arcturus"; | 2660 | chip_name = "arcturus"; |
| 2634 | break; | 2661 | break; |
| 2635 | case CHIP_NAVI12: | 2662 | case CHIP_NAVI12: |
| 2636 | if (adev->mman.discovery_bin) | 2663 | if (adev->discovery.bin) |
| 2637 | return 0; | 2664 | return 0; |
| 2638 | chip_name = "navi12"; | 2665 | chip_name = "navi12"; |
| 2639 | break; | 2666 | break; |
| 2640 | case CHIP_CYAN_SKILLFISH: | 2667 | case CHIP_CYAN_SKILLFISH: |
| 2641 | if (adev->mman.discovery_bin) | 2668 | if (adev->discovery.bin) |
| 2642 | return 0; | 2669 | return 0; |
| 2643 | chip_name = "cyan_skillfish"; | 2670 | chip_name = "cyan_skillfish"; |
| 2644 | break; | 2671 | break; |
| @@ -2763,6 +2790,10 @@ static int amdgpu_device_ip_early_init(struct amdgpu_device *adev) · 118 unmodified lines | |||
| 2763 | r = amdgpu_virt_request_full_gpu(adev, true); | 2790 | r = amdgpu_virt_request_full_gpu(adev, true); |
| 2764 | if (r) | 2791 | if (r) |
| 2765 | return r; | 2792 | return r; |
| 2793 | |||
| 2794 | r = amdgpu_virt_init_critical_region(adev); | ||
| 2795 | if (r) | ||
| 2796 | return r; | ||
| 2766 | } | 2797 | } |
| 2767 | 2798 | ||
| 2768 | switch (adev->asic_type) { | 2799 | switch (adev->asic_type) { |
| @@ -3652,6 +3683,20 @@ static int amdgpu_device_ip_fini_early(struct amdgpu_device *adev) · 883 unmodified lines | |||
| 3652 | "failed to release exclusive mode on fini\n"); | 3683 | "failed to release exclusive mode on fini\n"); |
| 3653 | } | 3684 | } |
| 3654 | 3685 | ||
| 3686 | /* | ||
| 3687 | * Driver reload on the APU can fail due to firmware validation because | ||
| 3688 | * the PSP is always running, as it is shared across the whole SoC. | ||
| 3689 | * This same issue does not occur on dGPU because it has a mechanism | ||
| 3690 | * that checks whether the PSP is running. A solution for those issues | ||
| 3691 | * in the APU is to trigger a GPU reset, but this should be done during | ||
| 3692 | * the unload phase to avoid adding boot latency and screen flicker. | ||
| 3693 | */ | ||
| 3694 | if ((adev->flags & AMD_IS_APU) && !adev->gmc.is_app_apu) { | ||
| 3695 | r = amdgpu_asic_reset(adev); | ||
| 3696 | if (r) | ||
| 3697 | dev_err(adev->dev, "asic reset on %s failed\n", __func__); | ||
| 3698 | } | ||
| 3699 | |||
| 3655 | return 0; | 3700 | return 0; |
| 3656 | } | 3701 | } |
| 3657 | 3702 | ||
| @@ -3762,7 +3807,7 @@ static void amdgpu_device_delay_enable_gfx_off(struct work_struct *work) · 104 unmodified lines | |||
| 3762 | */ | 3807 | */ |
| 3763 | static int amdgpu_device_ip_suspend_phase1(struct amdgpu_device *adev) | 3808 | static int amdgpu_device_ip_suspend_phase1(struct amdgpu_device *adev) |
| 3764 | { | 3809 | { |
| 3765 | int i, r; | 3810 | int i, r, rec; |
| 3766 | 3811 | ||
| 3767 | amdgpu_device_set_pg_state(adev, AMD_PG_STATE_UNGATE); | 3812 | amdgpu_device_set_pg_state(adev, AMD_PG_STATE_UNGATE); |
| 3768 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE); | 3813 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_UNGATE); |
| @@ -3783,13 +3828,25 @@ static int amdgpu_device_ip_suspend_phase1(struct amdgpu_device *adev) · 14 unmodified lines | |||
| 3783 | if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_DCE) | 3828 | if (adev->ip_blocks[i].version->type != AMD_IP_BLOCK_TYPE_DCE) |
| 3784 | continue; | 3829 | continue; |
| 3785 | 3830 | ||
| 3786 | /* XXX handle errors */ | ||
| 3787 | r = amdgpu_ip_block_suspend(&adev->ip_blocks[i]); | 3831 | r = amdgpu_ip_block_suspend(&adev->ip_blocks[i]); |
| 3788 | if (r) | 3832 | if (r) |
| 3789 | return r; | 3833 | goto unwind; |
| 3790 | } | 3834 | } |
| 3791 | 3835 | ||
| 3792 | return 0; | 3836 | return 0; |
| 3837 | unwind: | ||
| 3838 | rec = amdgpu_device_ip_resume_phase3(adev); | ||
| 3839 | if (rec) | ||
| 3840 | dev_err(adev->dev, | ||
| 3841 | "amdgpu_device_ip_resume_phase3 failed during unwind: %d\n", | ||
| 3842 | rec); | ||
| 3843 | |||
| 3844 | amdgpu_dpm_set_df_cstate(adev, DF_CSTATE_ALLOW); | ||
| 3845 | |||
| 3846 | amdgpu_device_set_pg_state(adev, AMD_PG_STATE_GATE); | ||
| 3847 | amdgpu_device_set_cg_state(adev, AMD_CG_STATE_GATE); | ||
| 3848 | |||
| 3849 | return r; | ||
| 3793 | } | 3850 | } |
| 3794 | 3851 | ||
| 3795 | /** | 3852 | /** |
| @@ -3805,7 +3862,7 @@ static int amdgpu_device_ip_suspend_phase1(struct amdgpu_device *adev) · 9 unmodified lines | |||
| 3805 | */ | 3862 | */ |
| 3806 | static int amdgpu_device_ip_suspend_phase2(struct amdgpu_device *adev) | 3863 | static int amdgpu_device_ip_suspend_phase2(struct amdgpu_device *adev) |
| 3807 | { | 3864 | { |
| 3808 | int i, r; | 3865 | int i, r, rec; |
| 3809 | 3866 | ||
| 3810 | if (adev->in_s0ix) | 3867 | if (adev->in_s0ix) |
| 3811 | amdgpu_dpm_gfx_state_change(adev, sGpuChangeState_D3Entry); | 3868 | amdgpu_dpm_gfx_state_change(adev, sGpuChangeState_D3Entry); |
| @@ -3866,9 +3923,9 @@ static int amdgpu_device_ip_suspend_phase2(struct amdgpu_device *adev) · 54 unmodified lines | |||
| 3866 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_PSP) | 3923 | adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_PSP) |
| 3867 | continue; | 3924 | continue; |
| 3868 | 3925 | ||
| 3869 | /* XXX handle errors */ | ||
| 3870 | r = amdgpu_ip_block_suspend(&adev->ip_blocks[i]); | 3926 | r = amdgpu_ip_block_suspend(&adev->ip_blocks[i]); |
| 3871 | adev->ip_blocks[i].status.hw = false; | 3927 | if (r) |
| 3928 | goto unwind; | ||
| 3872 | 3929 | ||
| 3873 | /* handle putting the SMC in the appropriate state */ | 3930 | /* handle putting the SMC in the appropriate state */ |
| 3874 | if (!amdgpu_sriov_vf(adev)) { | 3931 | if (!amdgpu_sriov_vf(adev)) { |
| @@ -3878,13 +3935,40 @@ static int amdgpu_device_ip_suspend_phase2(struct amdgpu_device *adev) · 3 unmodified lines | |||
| 3878 | dev_err(adev->dev, | 3935 | dev_err(adev->dev, |
| 3879 | "SMC failed to set mp1 state %d, %d\n", | 3936 | "SMC failed to set mp1 state %d, %d\n", |
| 3880 | adev->mp1_state, r); | 3937 | adev->mp1_state, r); |
| 3881 | return r; | 3938 | goto unwind; |
| 3882 | } | 3939 | } |
| 3883 | } | 3940 | } |
| 3884 | } | 3941 | } |
| 3885 | } | 3942 | } |
| 3886 | 3943 | ||
| 3887 | return 0; | 3944 | return 0; |
| 3945 | unwind: | ||
| 3946 | /* suspend phase 2 = resume phase 1 + resume phase 2 */ | ||
| 3947 | rec = amdgpu_device_ip_resume_phase1(adev); | ||
| 3948 | if (rec) { | ||
| 3949 | dev_err(adev->dev, | ||
| 3950 | "amdgpu_device_ip_resume_phase1 failed during unwind: %d\n", | ||
| 3951 | rec); | ||
| 3952 | return r; | ||
| 3953 | } | ||
| 3954 | |||
| 3955 | rec = amdgpu_device_fw_loading(adev); | ||
| 3956 | if (rec) { | ||
| 3957 | dev_err(adev->dev, | ||
| 3958 | "amdgpu_device_fw_loading failed during unwind: %d\n", | ||
| 3959 | rec); | ||
| 3960 | return r; | ||
| 3961 | } | ||
| 3962 | |||
| 3963 | rec = amdgpu_device_ip_resume_phase2(adev); | ||
| 3964 | if (rec) { | ||
| 3965 | dev_err(adev->dev, | ||
| 3966 | "amdgpu_device_ip_resume_phase2 failed during unwind: %d\n", | ||
| 3967 | rec); | ||
| 3968 | return r; | ||
| 3969 | } | ||
| 3970 | |||
| 3971 | return r; | ||
| 3888 | } | 3972 | } |
| 3889 | 3973 | ||
| 3890 | /** | 3974 | /** |
| @@ -3898,7 +3982,7 @@ static int amdgpu_device_ip_suspend_phase2(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 3898 | * in each IP into a state suitable for suspend. | 3982 | * in each IP into a state suitable for suspend. |
| 3899 | * Returns 0 on success, negative error code on failure. | 3983 | * Returns 0 on success, negative error code on failure. |
| 3900 | */ | 3984 | */ |
| 3901 | int amdgpu_device_ip_suspend(struct amdgpu_device *adev) | 3985 | static int amdgpu_device_ip_suspend(struct amdgpu_device *adev) |
| 3902 | { | 3986 | { |
| 3903 | int r; | 3987 | int r; |
| 3904 | 3988 | ||
| @@ -4182,25 +4266,13 @@ bool amdgpu_device_asic_has_dc_support(struct pci_dev *pdev, · 277 unmodified lines | |||
| 4182 | case CHIP_PITCAIRN: | 4266 | case CHIP_PITCAIRN: |
| 4183 | case CHIP_VERDE: | 4267 | case CHIP_VERDE: |
| 4184 | case CHIP_OLAND: | 4268 | case CHIP_OLAND: |
| 4185 | /* | 4269 | return amdgpu_dc != 0 && IS_ENABLED(CONFIG_DRM_AMD_DC_SI); |
| 4186 | * We have systems in the wild with these ASICs that require | ||
| 4187 | * LVDS and VGA support which is not supported with DC. | ||
| 4188 | * | ||
| 4189 | * Fallback to the non-DC driver here by default so as not to | ||
| 4190 | * cause regressions. | ||
| 4191 | */ | ||
| 4192 | #if defined(CONFIG_DRM_AMD_DC_SI) | ||
| 4193 | return amdgpu_dc > 0; | ||
| 4194 | #else | ||
| 4195 | return false; | ||
| 4196 | #endif | ||
| 4197 | case CHIP_BONAIRE: | ||
| 4198 | case CHIP_KAVERI: | 4270 | case CHIP_KAVERI: |
| 4199 | case CHIP_KABINI: | 4271 | case CHIP_KABINI: |
| 4200 | case CHIP_MULLINS: | 4272 | case CHIP_MULLINS: |
| 4201 | /* | 4273 | /* |
| 4202 | * We have systems in the wild with these ASICs that require | 4274 | * We have systems in the wild with these ASICs that require |
| 4203 | * VGA support which is not supported with DC. | 4275 | * TRAVIS and NUTMEG support which is not supported with DC. |
| 4204 | * | 4276 | * |
| 4205 | * Fallback to the non-DC driver here by default so as not to | 4277 | * Fallback to the non-DC driver here by default so as not to |
| 4206 | * cause regressions. | 4278 | * cause regressions. |
| @@ -4288,58 +4360,53 @@ static int amdgpu_device_get_job_timeout_settings(struct amdgpu_device *adev) · 81 unmodified lines | |||
| 4288 | long timeout; | 4360 | long timeout; |
| 4289 | int ret = 0; | 4361 | int ret = 0; |
| 4290 | 4362 | ||
| 4291 | /* | 4363 | /* By default timeout for all queues is 2 sec */ |
| 4292 | * By default timeout for jobs is 10 sec | 4364 | adev->gfx_timeout = adev->compute_timeout = adev->sdma_timeout = |
| 4293 | */ | 4365 | adev->video_timeout = msecs_to_jiffies(2000); |
| 4294 | adev->compute_timeout = adev->gfx_timeout = msecs_to_jiffies(10000); | ||
| 4295 | adev->sdma_timeout = adev->video_timeout = adev->gfx_timeout; | ||
| 4296 | 4366 | ||
| 4297 | if (strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) { | 4367 | if (!strnlen(input, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) |
| 4298 | while ((timeout_setting = strsep(&input, ",")) && | 4368 | return 0; |
| 4299 | strnlen(timeout_setting, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) { | ||
| 4300 | ret = kstrtol(timeout_setting, 0, &timeout); | ||
| 4301 | if (ret) | ||
| 4302 | return ret; | ||
| 4303 | 4369 | ||
| 4304 | if (timeout == 0) { | 4370 | while ((timeout_setting = strsep(&input, ",")) && |
| 4305 | index++; | 4371 | strnlen(timeout_setting, AMDGPU_MAX_TIMEOUT_PARAM_LENGTH)) { |
| 4306 | continue; | 4372 | ret = kstrtol(timeout_setting, 0, &timeout); |
| 4307 | } else if (timeout < 0) { | 4373 | if (ret) |
| 4308 | timeout = MAX_SCHEDULE_TIMEOUT; | 4374 | return ret; |
| 4309 | dev_warn(adev->dev, "lockup timeout disabled"); | ||
| 4310 | add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); | ||
| 4311 | } else { | ||
| 4312 | timeout = msecs_to_jiffies(timeout); | ||
| 4313 | } | ||
| 4314 | 4375 | ||
| 4315 | switch (index++) { | 4376 | if (timeout == 0) { |
| 4316 | case 0: | 4377 | index++; |
| 4317 | adev->gfx_timeout = timeout; | 4378 | continue; |
| 4318 | break; | 4379 | } else if (timeout < 0) { |
| 4319 | case 1: | 4380 | timeout = MAX_SCHEDULE_TIMEOUT; |
| 4320 | adev->compute_timeout = timeout; | 4381 | dev_warn(adev->dev, "lockup timeout disabled"); |
| 4321 | break; | 4382 | add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); |
| 4322 | case 2: | 4383 | } else { |
| 4323 | adev->sdma_timeout = timeout; | 4384 | timeout = msecs_to_jiffies(timeout); |
| 4324 | break; | ||
| 4325 | case 3: | ||
| 4326 | adev->video_timeout = timeout; | ||
| 4327 | break; | ||
| 4328 | default: | ||
| 4329 | break; | ||
| 4330 | } | ||
| 4331 | } | 4385 | } |
| 4332 | /* | 4386 | |
| 4333 | * There is only one value specified and | 4387 | switch (index++) { |
| 4334 | * it should apply to all non-compute jobs. | 4388 | case 0: |
| 4335 | */ | 4389 | adev->gfx_timeout = timeout; |
| 4336 | if (index == 1) { | 4390 | break; |
| 4337 | adev->sdma_timeout = adev->video_timeout = adev->gfx_timeout; | 4391 | case 1: |
| 4338 | if (amdgpu_sriov_vf(adev) || amdgpu_passthrough(adev)) | 4392 | adev->compute_timeout = timeout; |
| 4339 | adev->compute_timeout = adev->gfx_timeout; | 4393 | break; |
| 4394 | case 2: | ||
| 4395 | adev->sdma_timeout = timeout; | ||
| 4396 | break; | ||
| 4397 | case 3: | ||
| 4398 | adev->video_timeout = timeout; | ||
| 4399 | break; | ||
| 4400 | default: | ||
| 4401 | break; | ||
| 4340 | } | 4402 | } |
| 4341 | } | 4403 | } |
| 4342 | 4404 | ||
| 4405 | /* When only one value specified apply it to all queues. */ | ||
| 4406 | if (index == 1) | ||
| 4407 | adev->gfx_timeout = adev->compute_timeout = adev->sdma_timeout = | ||
| 4408 | adev->video_timeout = timeout; | ||
| 4409 | |||
| 4343 | return ret; | 4410 | return ret; |
| 4344 | } | 4411 | } |
| 4345 | 4412 | ||
| @@ -4394,6 +4461,55 @@ static void amdgpu_device_set_mcbp(struct amdgpu_device *adev) · 48 unmodified lines | |||
| 4394 | dev_info(adev->dev, "MCBP is enabled\n"); | 4461 | dev_info(adev->dev, "MCBP is enabled\n"); |
| 4395 | } | 4462 | } |
| 4396 | 4463 | ||
| 4464 | static int amdgpu_device_sys_interface_init(struct amdgpu_device *adev) | ||
| 4465 | { | ||
| 4466 | int r; | ||
| 4467 | |||
| 4468 | r = amdgpu_atombios_sysfs_init(adev); | ||
| 4469 | if (r) | ||
| 4470 | drm_err(&adev->ddev, | ||
| 4471 | "registering atombios sysfs failed (%d).\n", r); | ||
| 4472 | |||
| 4473 | r = amdgpu_pm_sysfs_init(adev); | ||
| 4474 | if (r) | ||
| 4475 | dev_err(adev->dev, "registering pm sysfs failed (%d).\n", r); | ||
| 4476 | |||
| 4477 | r = amdgpu_ucode_sysfs_init(adev); | ||
| 4478 | if (r) { | ||
| 4479 | adev->ucode_sysfs_en = false; | ||
| 4480 | dev_err(adev->dev, "Creating firmware sysfs failed (%d).\n", r); | ||
| 4481 | } else | ||
| 4482 | adev->ucode_sysfs_en = true; | ||
| 4483 | |||
| 4484 | r = amdgpu_device_attr_sysfs_init(adev); | ||
| 4485 | if (r) | ||
| 4486 | dev_err(adev->dev, "Could not create amdgpu device attr\n"); | ||
| 4487 | |||
| 4488 | r = devm_device_add_group(adev->dev, &amdgpu_board_attrs_group); | ||
| 4489 | if (r) | ||
| 4490 | dev_err(adev->dev, | ||
| 4491 | "Could not create amdgpu board attributes\n"); | ||
| 4492 | |||
| 4493 | amdgpu_fru_sysfs_init(adev); | ||
| 4494 | amdgpu_reg_state_sysfs_init(adev); | ||
| 4495 | amdgpu_xcp_sysfs_init(adev); | ||
| 4496 | |||
| 4497 | return r; | ||
| 4498 | } | ||
| 4499 | |||
| 4500 | static void amdgpu_device_sys_interface_fini(struct amdgpu_device *adev) | ||
| 4501 | { | ||
| 4502 | if (adev->pm.sysfs_initialized) | ||
| 4503 | amdgpu_pm_sysfs_fini(adev); | ||
| 4504 | if (adev->ucode_sysfs_en) | ||
| 4505 | amdgpu_ucode_sysfs_fini(adev); | ||
| 4506 | amdgpu_device_attr_sysfs_fini(adev); | ||
| 4507 | amdgpu_fru_sysfs_fini(adev); | ||
| 4508 | |||
| 4509 | amdgpu_reg_state_sysfs_fini(adev); | ||
| 4510 | amdgpu_xcp_sysfs_fini(adev); | ||
| 4511 | } | ||
| 4512 | |||
| 4397 | /** | 4513 | /** |
| 4398 | * amdgpu_device_init - initialize the driver | 4514 | * amdgpu_device_init - initialize the driver |
| 4399 | * | 4515 | * |
| @@ -4493,7 +4609,6 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 93 unmodified lines | |||
| 4493 | mutex_init(&adev->gfx.userq_sch_mutex); | 4609 | mutex_init(&adev->gfx.userq_sch_mutex); |
| 4494 | mutex_init(&adev->gfx.workload_profile_mutex); | 4610 | mutex_init(&adev->gfx.workload_profile_mutex); |
| 4495 | mutex_init(&adev->vcn.workload_profile_mutex); | 4611 | mutex_init(&adev->vcn.workload_profile_mutex); |
| 4496 | mutex_init(&adev->userq_mutex); | ||
| 4497 | 4612 | ||
| 4498 | amdgpu_device_init_apu_flags(adev); | 4613 | amdgpu_device_init_apu_flags(adev); |
| 4499 | 4614 | ||
| @@ -4521,7 +4636,7 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 21 unmodified lines | |||
| 4521 | 4636 | ||
| 4522 | INIT_LIST_HEAD(&adev->pm.od_kobj_list); | 4637 | INIT_LIST_HEAD(&adev->pm.od_kobj_list); |
| 4523 | 4638 | ||
| 4524 | INIT_LIST_HEAD(&adev->userq_mgr_list); | 4639 | xa_init(&adev->userq_doorbell_xa); |
| 4525 | 4640 | ||
| 4526 | INIT_DELAYED_WORK(&adev->delayed_init_work, | 4641 | INIT_DELAYED_WORK(&adev->delayed_init_work, |
| 4527 | amdgpu_device_delayed_init_work_handler); | 4642 | amdgpu_device_delayed_init_work_handler); |
| @@ -4544,6 +4659,7 @@ int amdgpu_device_init(struct amdgpu_device *adev, · 16 unmodified lines | |||
| 4544 | } | 4659 | } |
| 4545 | 4660 | ||
| 4546 | INIT_WORK(&adev->xgmi_reset_work, amdgpu_device_xgmi_reset_func); | 4661 | INIT_WORK(&adev->xgmi_reset_work, amdgpu_device_xgmi_reset_func); |
| 4662 | INIT_WORK(&adev->userq_reset_work, amdgpu_userq_reset_work); | ||
| 4547 | 4663 | ||
| 4548 | adev->gfx.gfx_off_req_count = 1; | 4664 | adev->gfx.gfx_off_req_count = 1; |
| 4549 | adev->gfx.gfx_off_residency = 0; | 4665 | adev->gfx.gfx_off_residency = 0; |
| @@ -4817,39 +4933,14 @@ fence_driver_init: · 267 unmodified lines | |||
| 4817 | flush_delayed_work(&adev->delayed_init_work); | 4933 | flush_delayed_work(&adev->delayed_init_work); |
| 4818 | } | 4934 | } |
| 4819 | 4935 | ||
| 4936 | if (adev->init_lvl->level == AMDGPU_INIT_LEVEL_MINIMAL_XGMI) | ||
| 4937 | amdgpu_xgmi_reset_on_init(adev); | ||
| 4820 | /* | 4938 | /* |
| 4821 | * Place those sysfs registering after `late_init`. As some of those | 4939 | * Place those sysfs registering after `late_init`. As some of those |
| 4822 | * operations performed in `late_init` might affect the sysfs | 4940 | * operations performed in `late_init` might affect the sysfs |
| 4823 | * interfaces creating. | 4941 | * interfaces creating. |
| 4824 | */ | 4942 | */ |
| 4825 | r = amdgpu_atombios_sysfs_init(adev); | 4943 | r = amdgpu_device_sys_interface_init(adev); |
| 4826 | if (r) | ||
| 4827 | drm_err(&adev->ddev, | ||
| 4828 | "registering atombios sysfs failed (%d).\n", r); | ||
| 4829 | |||
| 4830 | r = amdgpu_pm_sysfs_init(adev); | ||
| 4831 | if (r) | ||
| 4832 | dev_err(adev->dev, "registering pm sysfs failed (%d).\n", r); | ||
| 4833 | |||
| 4834 | r = amdgpu_ucode_sysfs_init(adev); | ||
| 4835 | if (r) { | ||
| 4836 | adev->ucode_sysfs_en = false; | ||
| 4837 | dev_err(adev->dev, "Creating firmware sysfs failed (%d).\n", r); | ||
| 4838 | } else | ||
| 4839 | adev->ucode_sysfs_en = true; | ||
| 4840 | |||
| 4841 | r = amdgpu_device_attr_sysfs_init(adev); | ||
| 4842 | if (r) | ||
| 4843 | dev_err(adev->dev, "Could not create amdgpu device attr\n"); | ||
| 4844 | |||
| 4845 | r = devm_device_add_group(adev->dev, &amdgpu_board_attrs_group); | ||
| 4846 | if (r) | ||
| 4847 | dev_err(adev->dev, | ||
| 4848 | "Could not create amdgpu board attributes\n"); | ||
| 4849 | |||
| 4850 | amdgpu_fru_sysfs_init(adev); | ||
| 4851 | amdgpu_reg_state_sysfs_init(adev); | ||
| 4852 | amdgpu_xcp_sysfs_init(adev); | ||
| 4853 | 4944 | ||
| 4854 | if (IS_ENABLED(CONFIG_PERF_EVENTS)) | 4945 | if (IS_ENABLED(CONFIG_PERF_EVENTS)) |
| 4855 | r = amdgpu_pmu_init(adev); | 4946 | r = amdgpu_pmu_init(adev); |
| @@ -4877,9 +4968,6 @@ fence_driver_init: · 21 unmodified lines | |||
| 4877 | if (px) | 4968 | if (px) |
| 4878 | vga_switcheroo_init_domain_pm_ops(adev->dev, &adev->vga_pm_domain); | 4969 | vga_switcheroo_init_domain_pm_ops(adev->dev, &adev->vga_pm_domain); |
| 4879 | 4970 | ||
| 4880 | if (adev->init_lvl->level == AMDGPU_INIT_LEVEL_MINIMAL_XGMI) | ||
| 4881 | amdgpu_xgmi_reset_on_init(adev); | ||
| 4882 | |||
| 4883 | amdgpu_device_check_iommu_direct_map(adev); | 4971 | amdgpu_device_check_iommu_direct_map(adev); |
| 4884 | 4972 | ||
| 4885 | adev->pm_nb.notifier_call = amdgpu_device_pm_notifier; | 4973 | adev->pm_nb.notifier_call = amdgpu_device_pm_notifier; |
| @@ -4971,15 +5059,7 @@ void amdgpu_device_fini_hw(struct amdgpu_device *adev) · 85 unmodified lines | |||
| 4971 | } | 5059 | } |
| 4972 | amdgpu_fence_driver_hw_fini(adev); | 5060 | amdgpu_fence_driver_hw_fini(adev); |
| 4973 | 5061 | ||
| 4974 | if (adev->pm.sysfs_initialized) | 5062 | amdgpu_device_sys_interface_fini(adev); |
| 4975 | amdgpu_pm_sysfs_fini(adev); | ||
| 4976 | if (adev->ucode_sysfs_en) | ||
| 4977 | amdgpu_ucode_sysfs_fini(adev); | ||
| 4978 | amdgpu_device_attr_sysfs_fini(adev); | ||
| 4979 | amdgpu_fru_sysfs_fini(adev); | ||
| 4980 | |||
| 4981 | amdgpu_reg_state_sysfs_fini(adev); | ||
| 4982 | amdgpu_xcp_sysfs_fini(adev); | ||
| 4983 | 5063 | ||
| 4984 | /* disable ras feature must before hw fini */ | 5064 | /* disable ras feature must before hw fini */ |
| 4985 | amdgpu_ras_pre_fini(adev); | 5065 | amdgpu_ras_pre_fini(adev); |
| @@ -5054,7 +5134,7 @@ void amdgpu_device_fini_sw(struct amdgpu_device *adev) · 68 unmodified lines | |||
| 5054 | 5134 | ||
| 5055 | if (IS_ENABLED(CONFIG_PERF_EVENTS)) | 5135 | if (IS_ENABLED(CONFIG_PERF_EVENTS)) |
| 5056 | amdgpu_pmu_fini(adev); | 5136 | amdgpu_pmu_fini(adev); |
| 5057 | if (adev->mman.discovery_bin) | 5137 | if (adev->discovery.bin) |
| 5058 | amdgpu_discovery_fini(adev); | 5138 | amdgpu_discovery_fini(adev); |
| 5059 | 5139 | ||
| 5060 | amdgpu_reset_put_reset_domain(adev->reset_domain); | 5140 | amdgpu_reset_put_reset_domain(adev->reset_domain); |
| @@ -5202,7 +5282,7 @@ void amdgpu_device_complete(struct drm_device *dev) · 141 unmodified lines | |||
| 5202 | int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients) | 5282 | int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients) |
| 5203 | { | 5283 | { |
| 5204 | struct amdgpu_device *adev = drm_to_adev(dev); | 5284 | struct amdgpu_device *adev = drm_to_adev(dev); |
| 5205 | int r = 0; | 5285 | int r, rec; |
| 5206 | 5286 | ||
| 5207 | if (dev->switch_power_state == DRM_SWITCH_POWER_OFF) | 5287 | if (dev->switch_power_state == DRM_SWITCH_POWER_OFF) |
| 5208 | return 0; | 5288 | return 0; |
| @@ -5218,35 +5298,92 @@ int amdgpu_device_suspend(struct drm_device *dev, bool notify_clients) · 9 unmodified lines | |||
| 5218 | return r; | 5298 | return r; |
| 5219 | } | 5299 | } |
| 5220 | 5300 | ||
| 5221 | if (amdgpu_acpi_smart_shift_update(adev, AMDGPU_SS_DEV_D3)) | 5301 | r = amdgpu_acpi_smart_shift_update(adev, AMDGPU_SS_DEV_D3); |
| 5222 | dev_warn(adev->dev, "smart shift update failed\n"); | 5302 | if (r) |
| 5303 | goto unwind_sriov; | ||
| 5223 | 5304 | ||
| 5224 | if (notify_clients) | 5305 | if (notify_clients) |
| 5225 | drm_client_dev_suspend(adev_to_drm(adev), false); | 5306 | drm_client_dev_suspend(adev_to_drm(adev)); |
| 5226 | 5307 | ||
| 5227 | cancel_delayed_work_sync(&adev->delayed_init_work); | 5308 | cancel_delayed_work_sync(&adev->delayed_init_work); |
| 5228 | 5309 | ||
| 5229 | amdgpu_ras_suspend(adev); | 5310 | amdgpu_ras_suspend(adev); |
| 5230 | 5311 | ||
| 5231 | amdgpu_device_ip_suspend_phase1(adev); | 5312 | r = amdgpu_device_ip_suspend_phase1(adev); |
| 5313 | if (r) | ||
| 5314 | goto unwind_smartshift; | ||
| 5232 | 5315 | ||
| 5233 | amdgpu_amdkfd_suspend(adev, !amdgpu_sriov_vf(adev) && !adev->in_runpm); | 5316 | amdgpu_amdkfd_suspend(adev, !amdgpu_sriov_vf(adev) && !adev->in_runpm); |
| 5234 | amdgpu_userq_suspend(adev); | 5317 | r = amdgpu_userq_suspend(adev); |
| 5318 | if (r) | ||
| 5319 | goto unwind_ip_phase1; | ||
| 5235 | 5320 | ||
| 5236 | r = amdgpu_device_evict_resources(adev); | 5321 | r = amdgpu_device_evict_resources(adev); |
| 5237 | if (r) | 5322 | if (r) |
| 5238 | return r; | 5323 | goto unwind_userq; |
| 5239 | 5324 | ||
| 5240 | amdgpu_ttm_set_buffer_funcs_status(adev, false); | 5325 | amdgpu_ttm_set_buffer_funcs_status(adev, false); |
| 5241 | 5326 | ||
| 5242 | amdgpu_fence_driver_hw_fini(adev); | 5327 | amdgpu_fence_driver_hw_fini(adev); |
| 5243 | 5328 | ||
| 5244 | amdgpu_device_ip_suspend_phase2(adev); | 5329 | r = amdgpu_device_ip_suspend_phase2(adev); |
| 5330 | if (r) | ||
| 5331 | goto unwind_evict; | ||
| 5245 | 5332 | ||
| 5246 | if (amdgpu_sriov_vf(adev)) | 5333 | if (amdgpu_sriov_vf(adev)) |
| 5247 | amdgpu_virt_release_full_gpu(adev, false); | 5334 | amdgpu_virt_release_full_gpu(adev, false); |
| 5248 | 5335 | ||
| 5249 | return 0; | 5336 | return 0; |
| 5337 | |||
| 5338 | unwind_evict: | ||
| 5339 | if (adev->mman.buffer_funcs_ring->sched.ready) | ||
| 5340 | amdgpu_ttm_set_buffer_funcs_status(adev, true); | ||
| 5341 | amdgpu_fence_driver_hw_init(adev); | ||
| 5342 | |||
| 5343 | unwind_userq: | ||
| 5344 | rec = amdgpu_userq_resume(adev); | ||
| 5345 | if (rec) { | ||
| 5346 | dev_warn(adev->dev, "failed to re-initialize user queues: %d\n", rec); | ||
| 5347 | return r; | ||
| 5348 | } | ||
| 5349 | rec = amdgpu_amdkfd_resume(adev, !amdgpu_sriov_vf(adev) && !adev->in_runpm); | ||
| 5350 | if (rec) { | ||
| 5351 | dev_warn(adev->dev, "failed to re-initialize kfd: %d\n", rec); | ||
| 5352 | return r; | ||
| 5353 | } | ||
| 5354 | |||
| 5355 | unwind_ip_phase1: | ||
| 5356 | /* suspend phase 1 = resume phase 3 */ | ||
| 5357 | rec = amdgpu_device_ip_resume_phase3(adev); | ||
| 5358 | if (rec) { | ||
| 5359 | dev_warn(adev->dev, "failed to re-initialize IPs phase1: %d\n", rec); | ||
| 5360 | return r; | ||
| 5361 | } | ||
| 5362 | |||
| 5363 | unwind_smartshift: | ||
| 5364 | rec = amdgpu_acpi_smart_shift_update(adev, AMDGPU_SS_DEV_D0); | ||
| 5365 | if (rec) { | ||
| 5366 | dev_warn(adev->dev, "failed to re-update smart shift: %d\n", rec); | ||
| 5367 | return r; | ||
| 5368 | } | ||
| 5369 | |||
| 5370 | if (notify_clients) | ||
| 5371 | drm_client_dev_resume(adev_to_drm(adev)); | ||
| 5372 | |||
| 5373 | amdgpu_ras_resume(adev); | ||
| 5374 | |||
| 5375 | unwind_sriov: | ||
| 5376 | if (amdgpu_sriov_vf(adev)) { | ||
| 5377 | rec = amdgpu_virt_request_full_gpu(adev, true); | ||
| 5378 | if (rec) { | ||
| 5379 | dev_warn(adev->dev, "failed to reinitialize sriov: %d\n", rec); | ||
| 5380 | return r; | ||
| 5381 | } | ||
| 5382 | } | ||
| 5383 | |||
| 5384 | adev->in_suspend = adev->in_s0ix = adev->in_s3 = false; | ||
| 5385 | |||
| 5386 | return r; | ||
| 5250 | } | 5387 | } |
| 5251 | 5388 | ||
| 5252 | static inline int amdgpu_virt_resume(struct amdgpu_device *adev) | 5389 | static inline int amdgpu_virt_resume(struct amdgpu_device *adev) |
| @@ -5352,7 +5489,7 @@ exit: · 99 unmodified lines | |||
| 5352 | flush_delayed_work(&adev->delayed_init_work); | 5489 | flush_delayed_work(&adev->delayed_init_work); |
| 5353 | 5490 | ||
| 5354 | if (notify_clients) | 5491 | if (notify_clients) |
| 5355 | drm_client_dev_resume(adev_to_drm(adev), false); | 5492 | drm_client_dev_resume(adev_to_drm(adev)); |
| 5356 | 5493 | ||
| 5357 | amdgpu_ras_resume(adev); | 5494 | amdgpu_ras_resume(adev); |
| 5358 | 5495 | ||
| @@ -5808,11 +5945,6 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, · 449 unmodified lines | |||
| 5808 | if (!amdgpu_ring_sched_ready(ring)) | 5945 | if (!amdgpu_ring_sched_ready(ring)) |
| 5809 | continue; | 5946 | continue; |
| 5810 | 5947 | ||
| 5811 | /* Clear job fence from fence drv to avoid force_completion | ||
| 5812 | * leave NULL and vm flush fence in fence drv | ||
| 5813 | */ | ||
| 5814 | amdgpu_fence_driver_clear_job_fences(ring); | ||
| 5815 | |||
| 5816 | /* after all hw jobs are reset, hw fence is meaningless, so force_completion */ | 5948 | /* after all hw jobs are reset, hw fence is meaningless, so force_completion */ |
| 5817 | amdgpu_fence_driver_force_completion(ring); | 5949 | amdgpu_fence_driver_force_completion(ring); |
| 5818 | } | 5950 | } |
| @@ -5957,7 +6089,11 @@ int amdgpu_device_reinit_after_reset(struct amdgpu_reset_context *reset_context) · 138 unmodified lines | |||
| 5957 | if (r) | 6089 | if (r) |
| 5958 | goto out; | 6090 | goto out; |
| 5959 | 6091 | ||
| 5960 | drm_client_dev_resume(adev_to_drm(tmp_adev), false); | 6092 | r = amdgpu_userq_post_reset(tmp_adev, vram_lost); |
| 6093 | if (r) | ||
| 6094 | goto out; | ||
| 6095 | |||
| 6096 | drm_client_dev_resume(adev_to_drm(tmp_adev)); | ||
| 5961 | 6097 | ||
| 5962 | /* | 6098 | /* |
| 5963 | * The GPU enters bad state once faulty pages | 6099 | * The GPU enters bad state once faulty pages |
| @@ -6179,6 +6315,7 @@ static inline void amdgpu_device_stop_pending_resets(struct amdgpu_device *adev) · 215 unmodified lines | |||
| 6179 | if (!amdgpu_sriov_vf(adev)) | 6315 | if (!amdgpu_sriov_vf(adev)) |
| 6180 | cancel_work(&adev->reset_work); | 6316 | cancel_work(&adev->reset_work); |
| 6181 | #endif | 6317 | #endif |
| 6318 | cancel_work(&adev->userq_reset_work); | ||
| 6182 | 6319 | ||
| 6183 | if (adev->kfd.dev) | 6320 | if (adev->kfd.dev) |
| 6184 | cancel_work(&adev->kfd.reset_work); | 6321 | cancel_work(&adev->kfd.reset_work); |
| @@ -6292,13 +6429,15 @@ static void amdgpu_device_halt_activities(struct amdgpu_device *adev, · 107 unmodified lines | |||
| 6292 | */ | 6429 | */ |
| 6293 | amdgpu_unregister_gpu_instance(tmp_adev); | 6430 | amdgpu_unregister_gpu_instance(tmp_adev); |
| 6294 | 6431 | ||
| 6295 | drm_client_dev_suspend(adev_to_drm(tmp_adev), false); | 6432 | drm_client_dev_suspend(adev_to_drm(tmp_adev)); |
| 6296 | 6433 | ||
| 6297 | /* disable ras on ALL IPs */ | 6434 | /* disable ras on ALL IPs */ |
| 6298 | if (!need_emergency_restart && !amdgpu_reset_in_dpc(adev) && | 6435 | if (!need_emergency_restart && !amdgpu_reset_in_dpc(adev) && |
| 6299 | amdgpu_device_ip_need_full_reset(tmp_adev)) | 6436 | amdgpu_device_ip_need_full_reset(tmp_adev)) |
| 6300 | amdgpu_ras_suspend(tmp_adev); | 6437 | amdgpu_ras_suspend(tmp_adev); |
| 6301 | 6438 | ||
| 6439 | amdgpu_userq_pre_reset(tmp_adev); | ||
| 6440 | |||
| 6302 | for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { | 6441 | for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { |
| 6303 | struct amdgpu_ring *ring = tmp_adev->rings[i]; | 6442 | struct amdgpu_ring *ring = tmp_adev->rings[i]; |
| 6304 | 6443 | ||
| @@ -6528,6 +6667,9 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 223 unmodified lines | |||
| 6528 | goto end_reset; | 6667 | goto end_reset; |
| 6529 | } | 6668 | } |
| 6530 | 6669 | ||
| 6670 | /* Cannot be called after locking reset domain */ | ||
| 6671 | amdgpu_ras_pre_reset(adev, &device_list); | ||
| 6672 | |||
| 6531 | /* We need to lock reset domain only once both for XGMI and single device */ | 6673 | /* We need to lock reset domain only once both for XGMI and single device */ |
| 6532 | amdgpu_device_recovery_get_reset_lock(adev, &device_list); | 6674 | amdgpu_device_recovery_get_reset_lock(adev, &device_list); |
| 6533 | 6675 | ||
| @@ -6541,7 +6683,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, · 7 unmodified lines | |||
| 6541 | * | 6683 | * |
| 6542 | * job->base holds a reference to parent fence | 6684 | * job->base holds a reference to parent fence |
| 6543 | */ | 6685 | */ |
| 6544 | if (job && dma_fence_is_signaled(&job->hw_fence.base)) { | 6686 | if (job && dma_fence_is_signaled(&job->hw_fence->base)) { |
| 6545 | job_signaled = true; | 6687 | job_signaled = true; |
| 6546 | dev_info(adev->dev, "Guilty job already signaled, skipping HW reset"); | 6688 | dev_info(adev->dev, "Guilty job already signaled, skipping HW reset"); |
| 6547 | goto skip_hw_reset; | 6689 | goto skip_hw_reset; |
| @@ -6558,6 +6700,7 @@ skip_sched_resume: · 10 unmodified lines | |||
| 6558 | amdgpu_device_gpu_resume(adev, &device_list, need_emergency_restart); | 6700 | amdgpu_device_gpu_resume(adev, &device_list, need_emergency_restart); |
| 6559 | reset_unlock: | 6701 | reset_unlock: |
| 6560 | amdgpu_device_recovery_put_reset_lock(adev, &device_list); | 6702 | amdgpu_device_recovery_put_reset_lock(adev, &device_list); |
| 6703 | amdgpu_ras_post_reset(adev, &device_list); | ||
| 6561 | end_reset: | 6704 | end_reset: |
| 6562 | if (hive) { | 6705 | if (hive) { |
| 6563 | mutex_unlock(&hive->hive_lock); | 6706 | mutex_unlock(&hive->hive_lock); |
| @@ -7285,10 +7428,17 @@ void amdgpu_device_flush_hdp(struct amdgpu_device *adev, · 721 unmodified lines | |||
| 7285 | if (adev->gmc.xgmi.connected_to_cpu) | 7428 | if (adev->gmc.xgmi.connected_to_cpu) |
| 7286 | return; | 7429 | return; |
| 7287 | 7430 | ||
| 7288 | if (ring && ring->funcs->emit_hdp_flush) | 7431 | if (ring && ring->funcs->emit_hdp_flush) { |
| 7289 | amdgpu_ring_emit_hdp_flush(ring); | 7432 | amdgpu_ring_emit_hdp_flush(ring); |
| 7290 | else | 7433 | return; |
| 7291 | amdgpu_asic_flush_hdp(adev, ring); | 7434 | } |
| 7435 | |||
| 7436 | if (!ring && amdgpu_sriov_runtime(adev)) { | ||
| 7437 | if (!amdgpu_kiq_hdp_flush(adev)) | ||
| 7438 | return; | ||
| 7439 | } | ||
| 7440 | |||
| 7441 | amdgpu_hdp_flush(adev, ring); | ||
| 7292 | } | 7442 | } |
| 7293 | 7443 | ||
| 7294 | void amdgpu_device_invalidate_hdp(struct amdgpu_device *adev, | 7444 | void amdgpu_device_invalidate_hdp(struct amdgpu_device *adev, |
| @@ -7301,7 +7451,7 @@ void amdgpu_device_invalidate_hdp(struct amdgpu_device *adev, · 6 unmodified lines | |||
| 7301 | if (adev->gmc.xgmi.connected_to_cpu) | 7451 | if (adev->gmc.xgmi.connected_to_cpu) |
| 7302 | return; | 7452 | return; |
| 7303 | 7453 | ||
| 7304 | amdgpu_asic_invalidate_hdp(adev, ring); | 7454 | amdgpu_hdp_invalidate(adev, ring); |
| 7305 | } | 7455 | } |
| 7306 | 7456 | ||
| 7307 | int amdgpu_in_reset(struct amdgpu_device *adev) | 7457 | int amdgpu_in_reset(struct amdgpu_device *adev) |
| @@ -107,6 +107,7 @@ · 106 unmodified lines | |||
| 107 | #include "vcn_v5_0_1.h" | 107 | #include "vcn_v5_0_1.h" |
| 108 | #include "jpeg_v5_0_0.h" | 108 | #include "jpeg_v5_0_0.h" |
| 109 | #include "jpeg_v5_0_1.h" | 109 | #include "jpeg_v5_0_1.h" |
| 110 | #include "amdgpu_ras_mgr.h" | ||
| 110 | 111 | ||
| 111 | #include "amdgpu_vpe.h" | 112 | #include "amdgpu_vpe.h" |
| 112 | #if defined(CONFIG_DRM_AMD_ISP) | 113 | #if defined(CONFIG_DRM_AMD_ISP) |
| @@ -254,9 +255,9 @@ static int amdgpu_discovery_read_binary_from_sysmem(struct amdgpu_device *adev, · 141 unmodified lines | |||
| 254 | pos = tmr_offset + tmr_size - DISCOVERY_TMR_OFFSET; | 255 | pos = tmr_offset + tmr_size - DISCOVERY_TMR_OFFSET; |
| 255 | 256 | ||
| 256 | /* This region is read-only and reserved from system use */ | 257 | /* This region is read-only and reserved from system use */ |
| 257 | discv_regn = memremap(pos, adev->mman.discovery_tmr_size, MEMREMAP_WC); | 258 | discv_regn = memremap(pos, adev->discovery.size, MEMREMAP_WC); |
| 258 | if (discv_regn) { | 259 | if (discv_regn) { |
| 259 | memcpy(binary, discv_regn, adev->mman.discovery_tmr_size); | 260 | memcpy(binary, discv_regn, adev->discovery.size); |
| 260 | memunmap(discv_regn); | 261 | memunmap(discv_regn); |
| 261 | return 0; | 262 | return 0; |
| 262 | } | 263 | } |
| @@ -298,10 +299,31 @@ static int amdgpu_discovery_read_binary_from_mem(struct amdgpu_device *adev, · 35 unmodified lines | |||
| 298 | else | 299 | else |
| 299 | vram_size <<= 20; | 300 | vram_size <<= 20; |
| 300 | 301 | ||
| 302 | /* | ||
| 303 | * If in VRAM, discovery TMR is marked for reservation. If it is in system mem, | ||
| 304 | * then it is not required to be reserved. | ||
| 305 | */ | ||
| 301 | if (sz_valid) { | 306 | if (sz_valid) { |
| 302 | uint64_t pos = vram_size - DISCOVERY_TMR_OFFSET; | 307 | if (amdgpu_sriov_vf(adev) && adev->virt.is_dynamic_crit_regn_enabled) { |
| 303 | amdgpu_device_vram_access(adev, pos, (uint32_t *)binary, | 308 | /* For SRIOV VFs with dynamic critical region enabled, |
| 304 | adev->mman.discovery_tmr_size, false); | 309 | * we will get the IPD binary via below call. |
| 310 | * If dynamic critical is disabled, fall through to normal seq. | ||
| 311 | */ | ||
| 312 | if (amdgpu_virt_get_dynamic_data_info(adev, | ||
| 313 | AMD_SRIOV_MSG_IPD_TABLE_ID, binary, | ||
| 314 | &adev->discovery.size)) { | ||
| 315 | dev_err(adev->dev, | ||
| 316 | "failed to read discovery info from dynamic critical region."); | ||
| 317 | ret = -EINVAL; | ||
| 318 | goto exit; | ||
| 319 | } | ||
| 320 | } else { | ||
| 321 | uint64_t pos = vram_size - DISCOVERY_TMR_OFFSET; | ||
| 322 | |||
| 323 | amdgpu_device_vram_access(adev, pos, (uint32_t *)binary, | ||
| 324 | adev->discovery.size, false); | ||
| 325 | adev->discovery.reserve_tmr = true; | ||
| 326 | } | ||
| 305 | } else { | 327 | } else { |
| 306 | ret = amdgpu_discovery_read_binary_from_sysmem(adev, binary); | 328 | ret = amdgpu_discovery_read_binary_from_sysmem(adev, binary); |
| 307 | } | 329 | } |
| @@ -310,7 +332,7 @@ static int amdgpu_discovery_read_binary_from_mem(struct amdgpu_device *adev, · 2 unmodified lines | |||
| 310 | dev_err(adev->dev, | 332 | dev_err(adev->dev, |
| 311 | "failed to read discovery info from memory, vram size read: %llx", | 333 | "failed to read discovery info from memory, vram size read: %llx", |
| 312 | vram_size); | 334 | vram_size); |
| 313 | 335 | exit: | |
| 314 | return ret; | 336 | return ret; |
| 315 | } | 337 | } |
| 316 | 338 | ||
| @@ -389,6 +411,7 @@ static void amdgpu_discovery_harvest_config_quirk(struct amdgpu_device *adev) · 72 unmodified lines | |||
| 389 | static int amdgpu_discovery_verify_npsinfo(struct amdgpu_device *adev, | 411 | static int amdgpu_discovery_verify_npsinfo(struct amdgpu_device *adev, |
| 390 | struct binary_header *bhdr) | 412 | struct binary_header *bhdr) |
| 391 | { | 413 | { |
| 414 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 392 | struct table_info *info; | 415 | struct table_info *info; |
| 393 | uint16_t checksum; | 416 | uint16_t checksum; |
| 394 | uint16_t offset; | 417 | uint16_t offset; |
| @@ -398,14 +421,14 @@ static int amdgpu_discovery_verify_npsinfo(struct amdgpu_device *adev, · 3 unmodified lines | |||
| 398 | checksum = le16_to_cpu(info->checksum); | 421 | checksum = le16_to_cpu(info->checksum); |
| 399 | 422 | ||
| 400 | struct nps_info_header *nhdr = | 423 | struct nps_info_header *nhdr = |
| 401 | (struct nps_info_header *)(adev->mman.discovery_bin + offset); | 424 | (struct nps_info_header *)(discovery_bin + offset); |
| 402 | 425 | ||
| 403 | if (le32_to_cpu(nhdr->table_id) != NPS_INFO_TABLE_ID) { | 426 | if (le32_to_cpu(nhdr->table_id) != NPS_INFO_TABLE_ID) { |
| 404 | dev_dbg(adev->dev, "invalid ip discovery nps info table id\n"); | 427 | dev_dbg(adev->dev, "invalid ip discovery nps info table id\n"); |
| 405 | return -EINVAL; | 428 | return -EINVAL; |
| 406 | } | 429 | } |
| 407 | 430 | ||
| 408 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 431 | if (!amdgpu_discovery_verify_checksum(discovery_bin + offset, |
| 409 | le32_to_cpu(nhdr->size_bytes), | 432 | le32_to_cpu(nhdr->size_bytes), |
| 410 | checksum)) { | 433 | checksum)) { |
| 411 | dev_dbg(adev->dev, "invalid nps info data table checksum\n"); | 434 | dev_dbg(adev->dev, "invalid nps info data table checksum\n"); |
| @@ -417,8 +440,11 @@ static int amdgpu_discovery_verify_npsinfo(struct amdgpu_device *adev, · 5 unmodified lines | |||
| 417 | 440 | ||
| 418 | static const char *amdgpu_discovery_get_fw_name(struct amdgpu_device *adev) | 441 | static const char *amdgpu_discovery_get_fw_name(struct amdgpu_device *adev) |
| 419 | { | 442 | { |
| 420 | if (amdgpu_discovery == 2) | 443 | if (amdgpu_discovery == 2) { |
| 444 | /* Assume there is valid discovery TMR in VRAM even if binary is sideloaded */ | ||
| 445 | adev->discovery.reserve_tmr = true; | ||
| 421 | return "amdgpu/ip_discovery.bin"; | 446 | return "amdgpu/ip_discovery.bin"; |
| 447 | } | ||
| 422 | 448 | ||
| 423 | switch (adev->asic_type) { | 449 | switch (adev->asic_type) { |
| 424 | case CHIP_VEGA10: | 450 | case CHIP_VEGA10: |
| @@ -447,49 +473,53 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 22 unmodified lines | |||
| 447 | { | 473 | { |
| 448 | struct table_info *info; | 474 | struct table_info *info; |
| 449 | struct binary_header *bhdr; | 475 | struct binary_header *bhdr; |
| 476 | uint8_t *discovery_bin; | ||
| 450 | const char *fw_name; | 477 | const char *fw_name; |
| 451 | uint16_t offset; | 478 | uint16_t offset; |
| 452 | uint16_t size; | 479 | uint16_t size; |
| 453 | uint16_t checksum; | 480 | uint16_t checksum; |
| 454 | int r; | 481 | int r; |
| 455 | 482 | ||
| 456 | adev->mman.discovery_tmr_size = DISCOVERY_TMR_SIZE; | 483 | adev->discovery.bin = kzalloc(DISCOVERY_TMR_SIZE, GFP_KERNEL); |
| 457 | adev->mman.discovery_bin = kzalloc(adev->mman.discovery_tmr_size, GFP_KERNEL); | 484 | if (!adev->discovery.bin) |
| 458 | if (!adev->mman.discovery_bin) | ||
| 459 | return -ENOMEM; | 485 | return -ENOMEM; |
| 486 | adev->discovery.size = DISCOVERY_TMR_SIZE; | ||
| 487 | adev->discovery.debugfs_blob.data = adev->discovery.bin; | ||
| 488 | adev->discovery.debugfs_blob.size = adev->discovery.size; | ||
| 460 | 489 | ||
| 490 | discovery_bin = adev->discovery.bin; | ||
| 461 | /* Read from file if it is the preferred option */ | 491 | /* Read from file if it is the preferred option */ |
| 462 | fw_name = amdgpu_discovery_get_fw_name(adev); | 492 | fw_name = amdgpu_discovery_get_fw_name(adev); |
| 463 | if (fw_name != NULL) { | 493 | if (fw_name != NULL) { |
| 464 | drm_dbg(&adev->ddev, "use ip discovery information from file"); | 494 | drm_dbg(&adev->ddev, "use ip discovery information from file"); |
| 465 | r = amdgpu_discovery_read_binary_from_file(adev, adev->mman.discovery_bin, fw_name); | 495 | r = amdgpu_discovery_read_binary_from_file(adev, discovery_bin, |
| 496 | fw_name); | ||
| 466 | if (r) | 497 | if (r) |
| 467 | goto out; | 498 | goto out; |
| 468 | } else { | 499 | } else { |
| 469 | drm_dbg(&adev->ddev, "use ip discovery information from memory"); | 500 | drm_dbg(&adev->ddev, "use ip discovery information from memory"); |
| 470 | r = amdgpu_discovery_read_binary_from_mem( | 501 | r = amdgpu_discovery_read_binary_from_mem(adev, discovery_bin); |
| 471 | adev, adev->mman.discovery_bin); | ||
| 472 | if (r) | 502 | if (r) |
| 473 | goto out; | 503 | goto out; |
| 474 | } | 504 | } |
| 475 | 505 | ||
| 476 | /* check the ip discovery binary signature */ | 506 | /* check the ip discovery binary signature */ |
| 477 | if (!amdgpu_discovery_verify_binary_signature(adev->mman.discovery_bin)) { | 507 | if (!amdgpu_discovery_verify_binary_signature(discovery_bin)) { |
| 478 | dev_err(adev->dev, | 508 | dev_err(adev->dev, |
| 479 | "get invalid ip discovery binary signature\n"); | 509 | "get invalid ip discovery binary signature\n"); |
| 480 | r = -EINVAL; | 510 | r = -EINVAL; |
| 481 | goto out; | 511 | goto out; |
| 482 | } | 512 | } |
| 483 | 513 | ||
| 484 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 514 | bhdr = (struct binary_header *)discovery_bin; |
| 485 | 515 | ||
| 486 | offset = offsetof(struct binary_header, binary_checksum) + | 516 | offset = offsetof(struct binary_header, binary_checksum) + |
| 487 | sizeof(bhdr->binary_checksum); | 517 | sizeof(bhdr->binary_checksum); |
| 488 | size = le16_to_cpu(bhdr->binary_size) - offset; | 518 | size = le16_to_cpu(bhdr->binary_size) - offset; |
| 489 | checksum = le16_to_cpu(bhdr->binary_checksum); | 519 | checksum = le16_to_cpu(bhdr->binary_checksum); |
| 490 | 520 | ||
| 491 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 521 | if (!amdgpu_discovery_verify_checksum(discovery_bin + offset, size, |
| 492 | size, checksum)) { | 522 | checksum)) { |
| 493 | dev_err(adev->dev, "invalid ip discovery binary checksum\n"); | 523 | dev_err(adev->dev, "invalid ip discovery binary checksum\n"); |
| 494 | r = -EINVAL; | 524 | r = -EINVAL; |
| 495 | goto out; | 525 | goto out; |
| @@ -501,15 +531,16 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 5 unmodified lines | |||
| 501 | 531 | ||
| 502 | if (offset) { | 532 | if (offset) { |
| 503 | struct ip_discovery_header *ihdr = | 533 | struct ip_discovery_header *ihdr = |
| 504 | (struct ip_discovery_header *)(adev->mman.discovery_bin + offset); | 534 | (struct ip_discovery_header *)(discovery_bin + offset); |
| 505 | if (le32_to_cpu(ihdr->signature) != DISCOVERY_TABLE_SIGNATURE) { | 535 | if (le32_to_cpu(ihdr->signature) != DISCOVERY_TABLE_SIGNATURE) { |
| 506 | dev_err(adev->dev, "invalid ip discovery data table signature\n"); | 536 | dev_err(adev->dev, "invalid ip discovery data table signature\n"); |
| 507 | r = -EINVAL; | 537 | r = -EINVAL; |
| 508 | goto out; | 538 | goto out; |
| 509 | } | 539 | } |
| 510 | 540 | ||
| 511 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 541 | if (!amdgpu_discovery_verify_checksum(discovery_bin + offset, |
| 512 | le16_to_cpu(ihdr->size), checksum)) { | 542 | le16_to_cpu(ihdr->size), |
| 543 | checksum)) { | ||
| 513 | dev_err(adev->dev, "invalid ip discovery data table checksum\n"); | 544 | dev_err(adev->dev, "invalid ip discovery data table checksum\n"); |
| 514 | r = -EINVAL; | 545 | r = -EINVAL; |
| 515 | goto out; | 546 | goto out; |
| @@ -522,7 +553,7 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 522 | 553 | ||
| 523 | if (offset) { | 554 | if (offset) { |
| 524 | struct gpu_info_header *ghdr = | 555 | struct gpu_info_header *ghdr = |
| 525 | (struct gpu_info_header *)(adev->mman.discovery_bin + offset); | 556 | (struct gpu_info_header *)(discovery_bin + offset); |
| 526 | 557 | ||
| 527 | if (le32_to_cpu(ghdr->table_id) != GC_TABLE_ID) { | 558 | if (le32_to_cpu(ghdr->table_id) != GC_TABLE_ID) { |
| 528 | dev_err(adev->dev, "invalid ip discovery gc table id\n"); | 559 | dev_err(adev->dev, "invalid ip discovery gc table id\n"); |
| @@ -530,8 +561,9 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 1 unmodified line | |||
| 530 | goto out; | 561 | goto out; |
| 531 | } | 562 | } |
| 532 | 563 | ||
| 533 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 564 | if (!amdgpu_discovery_verify_checksum(discovery_bin + offset, |
| 534 | le32_to_cpu(ghdr->size), checksum)) { | 565 | le32_to_cpu(ghdr->size), |
| 566 | checksum)) { | ||
| 535 | dev_err(adev->dev, "invalid gc data table checksum\n"); | 567 | dev_err(adev->dev, "invalid gc data table checksum\n"); |
| 536 | r = -EINVAL; | 568 | r = -EINVAL; |
| 537 | goto out; | 569 | goto out; |
| @@ -544,7 +576,7 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 544 | 576 | ||
| 545 | if (offset) { | 577 | if (offset) { |
| 546 | struct harvest_info_header *hhdr = | 578 | struct harvest_info_header *hhdr = |
| 547 | (struct harvest_info_header *)(adev->mman.discovery_bin + offset); | 579 | (struct harvest_info_header *)(discovery_bin + offset); |
| 548 | 580 | ||
| 549 | if (le32_to_cpu(hhdr->signature) != HARVEST_TABLE_SIGNATURE) { | 581 | if (le32_to_cpu(hhdr->signature) != HARVEST_TABLE_SIGNATURE) { |
| 550 | dev_err(adev->dev, "invalid ip discovery harvest table signature\n"); | 582 | dev_err(adev->dev, "invalid ip discovery harvest table signature\n"); |
| @@ -552,8 +584,9 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 1 unmodified line | |||
| 552 | goto out; | 584 | goto out; |
| 553 | } | 585 | } |
| 554 | 586 | ||
| 555 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 587 | if (!amdgpu_discovery_verify_checksum( |
| 556 | sizeof(struct harvest_table), checksum)) { | 588 | discovery_bin + offset, |
| 589 | sizeof(struct harvest_table), checksum)) { | ||
| 557 | dev_err(adev->dev, "invalid harvest data table checksum\n"); | 590 | dev_err(adev->dev, "invalid harvest data table checksum\n"); |
| 558 | r = -EINVAL; | 591 | r = -EINVAL; |
| 559 | goto out; | 592 | goto out; |
| @@ -566,7 +599,7 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 566 | 599 | ||
| 567 | if (offset) { | 600 | if (offset) { |
| 568 | struct vcn_info_header *vhdr = | 601 | struct vcn_info_header *vhdr = |
| 569 | (struct vcn_info_header *)(adev->mman.discovery_bin + offset); | 602 | (struct vcn_info_header *)(discovery_bin + offset); |
| 570 | 603 | ||
| 571 | if (le32_to_cpu(vhdr->table_id) != VCN_INFO_TABLE_ID) { | 604 | if (le32_to_cpu(vhdr->table_id) != VCN_INFO_TABLE_ID) { |
| 572 | dev_err(adev->dev, "invalid ip discovery vcn table id\n"); | 605 | dev_err(adev->dev, "invalid ip discovery vcn table id\n"); |
| @@ -574,8 +607,9 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 1 unmodified line | |||
| 574 | goto out; | 607 | goto out; |
| 575 | } | 608 | } |
| 576 | 609 | ||
| 577 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 610 | if (!amdgpu_discovery_verify_checksum( |
| 578 | le32_to_cpu(vhdr->size_bytes), checksum)) { | 611 | discovery_bin + offset, |
| 612 | le32_to_cpu(vhdr->size_bytes), checksum)) { | ||
| 579 | dev_err(adev->dev, "invalid vcn data table checksum\n"); | 613 | dev_err(adev->dev, "invalid vcn data table checksum\n"); |
| 580 | r = -EINVAL; | 614 | r = -EINVAL; |
| 581 | goto out; | 615 | goto out; |
| @@ -588,7 +622,7 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 6 unmodified lines | |||
| 588 | 622 | ||
| 589 | if (0 && offset) { | 623 | if (0 && offset) { |
| 590 | struct mall_info_header *mhdr = | 624 | struct mall_info_header *mhdr = |
| 591 | (struct mall_info_header *)(adev->mman.discovery_bin + offset); | 625 | (struct mall_info_header *)(discovery_bin + offset); |
| 592 | 626 | ||
| 593 | if (le32_to_cpu(mhdr->table_id) != MALL_INFO_TABLE_ID) { | 627 | if (le32_to_cpu(mhdr->table_id) != MALL_INFO_TABLE_ID) { |
| 594 | dev_err(adev->dev, "invalid ip discovery mall table id\n"); | 628 | dev_err(adev->dev, "invalid ip discovery mall table id\n"); |
| @@ -596,8 +630,9 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 1 unmodified line | |||
| 596 | goto out; | 630 | goto out; |
| 597 | } | 631 | } |
| 598 | 632 | ||
| 599 | if (!amdgpu_discovery_verify_checksum(adev->mman.discovery_bin + offset, | 633 | if (!amdgpu_discovery_verify_checksum( |
| 600 | le32_to_cpu(mhdr->size_bytes), checksum)) { | 634 | discovery_bin + offset, |
| 635 | le32_to_cpu(mhdr->size_bytes), checksum)) { | ||
| 601 | dev_err(adev->dev, "invalid mall data table checksum\n"); | 636 | dev_err(adev->dev, "invalid mall data table checksum\n"); |
| 602 | r = -EINVAL; | 637 | r = -EINVAL; |
| 603 | goto out; | 638 | goto out; |
| @@ -607,8 +642,8 @@ static int amdgpu_discovery_init(struct amdgpu_device *adev) · 3 unmodified lines | |||
| 607 | return 0; | 642 | return 0; |
| 608 | 643 | ||
| 609 | out: | 644 | out: |
| 610 | kfree(adev->mman.discovery_bin); | 645 | kfree(adev->discovery.bin); |
| 611 | adev->mman.discovery_bin = NULL; | 646 | adev->discovery.bin = NULL; |
| 612 | if ((amdgpu_discovery != 2) && | 647 | if ((amdgpu_discovery != 2) && |
| 613 | (RREG32(mmIP_DISCOVERY_VERSION) == 4)) | 648 | (RREG32(mmIP_DISCOVERY_VERSION) == 4)) |
| 614 | amdgpu_ras_query_boot_status(adev, 4); | 649 | amdgpu_ras_query_boot_status(adev, 4); |
| @@ -620,8 +655,8 @@ static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev); · 5 unmodified lines | |||
| 620 | void amdgpu_discovery_fini(struct amdgpu_device *adev) | 655 | void amdgpu_discovery_fini(struct amdgpu_device *adev) |
| 621 | { | 656 | { |
| 622 | amdgpu_discovery_sysfs_fini(adev); | 657 | amdgpu_discovery_sysfs_fini(adev); |
| 623 | kfree(adev->mman.discovery_bin); | 658 | kfree(adev->discovery.bin); |
| 624 | adev->mman.discovery_bin = NULL; | 659 | adev->discovery.bin = NULL; |
| 625 | } | 660 | } |
| 626 | 661 | ||
| 627 | static int amdgpu_discovery_validate_ip(struct amdgpu_device *adev, | 662 | static int amdgpu_discovery_validate_ip(struct amdgpu_device *adev, |
| @@ -646,6 +681,7 @@ static int amdgpu_discovery_validate_ip(struct amdgpu_device *adev, · 18 unmodified lines | |||
| 646 | static void amdgpu_discovery_read_harvest_bit_per_ip(struct amdgpu_device *adev, | 681 | static void amdgpu_discovery_read_harvest_bit_per_ip(struct amdgpu_device *adev, |
| 647 | uint32_t *vcn_harvest_count) | 682 | uint32_t *vcn_harvest_count) |
| 648 | { | 683 | { |
| 684 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 649 | struct binary_header *bhdr; | 685 | struct binary_header *bhdr; |
| 650 | struct ip_discovery_header *ihdr; | 686 | struct ip_discovery_header *ihdr; |
| 651 | struct die_header *dhdr; | 687 | struct die_header *dhdr; |
| @@ -655,21 +691,21 @@ static void amdgpu_discovery_read_harvest_bit_per_ip(struct amdgpu_device *adev, · 3 unmodified lines | |||
| 655 | uint8_t inst; | 691 | uint8_t inst; |
| 656 | int i, j; | 692 | int i, j; |
| 657 | 693 | ||
| 658 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 694 | bhdr = (struct binary_header *)discovery_bin; |
| 659 | ihdr = (struct ip_discovery_header *)(adev->mman.discovery_bin + | 695 | ihdr = (struct ip_discovery_header |
| 660 | le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset)); | 696 | *)(discovery_bin + |
| 697 | le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset)); | ||
| 661 | num_dies = le16_to_cpu(ihdr->num_dies); | 698 | num_dies = le16_to_cpu(ihdr->num_dies); |
| 662 | 699 | ||
| 663 | /* scan harvest bit of all IP data structures */ | 700 | /* scan harvest bit of all IP data structures */ |
| 664 | for (i = 0; i < num_dies; i++) { | 701 | for (i = 0; i < num_dies; i++) { |
| 665 | die_offset = le16_to_cpu(ihdr->die_info[i].die_offset); | 702 | die_offset = le16_to_cpu(ihdr->die_info[i].die_offset); |
| 666 | dhdr = (struct die_header *)(adev->mman.discovery_bin + die_offset); | 703 | dhdr = (struct die_header *)(discovery_bin + die_offset); |
| 667 | num_ips = le16_to_cpu(dhdr->num_ips); | 704 | num_ips = le16_to_cpu(dhdr->num_ips); |
| 668 | ip_offset = die_offset + sizeof(*dhdr); | 705 | ip_offset = die_offset + sizeof(*dhdr); |
| 669 | 706 | ||
| 670 | for (j = 0; j < num_ips; j++) { | 707 | for (j = 0; j < num_ips; j++) { |
| 671 | ip = (struct ip *)(adev->mman.discovery_bin + | 708 | ip = (struct ip *)(discovery_bin + ip_offset); |
| 672 | ip_offset); | ||
| 673 | inst = ip->number_instance; | 709 | inst = ip->number_instance; |
| 674 | hw_id = le16_to_cpu(ip->hw_id); | 710 | hw_id = le16_to_cpu(ip->hw_id); |
| 675 | if (amdgpu_discovery_validate_ip(adev, inst, hw_id)) | 711 | if (amdgpu_discovery_validate_ip(adev, inst, hw_id)) |
| @@ -711,13 +747,14 @@ static void amdgpu_discovery_read_from_harvest_table(struct amdgpu_device *adev, · 35 unmodified lines | |||
| 711 | uint32_t *vcn_harvest_count, | 747 | uint32_t *vcn_harvest_count, |
| 712 | uint32_t *umc_harvest_count) | 748 | uint32_t *umc_harvest_count) |
| 713 | { | 749 | { |
| 750 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 714 | struct binary_header *bhdr; | 751 | struct binary_header *bhdr; |
| 715 | struct harvest_table *harvest_info; | 752 | struct harvest_table *harvest_info; |
| 716 | u16 offset; | 753 | u16 offset; |
| 717 | int i; | 754 | int i; |
| 718 | uint32_t umc_harvest_config = 0; | 755 | uint32_t umc_harvest_config = 0; |
| 719 | 756 | ||
| 720 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 757 | bhdr = (struct binary_header *)discovery_bin; |
| 721 | offset = le16_to_cpu(bhdr->table_list[HARVEST_INFO].offset); | 758 | offset = le16_to_cpu(bhdr->table_list[HARVEST_INFO].offset); |
| 722 | 759 | ||
| 723 | if (!offset) { | 760 | if (!offset) { |
| @@ -725,7 +762,7 @@ static void amdgpu_discovery_read_from_harvest_table(struct amdgpu_device *adev, · 1 unmodified line | |||
| 725 | return; | 762 | return; |
| 726 | } | 763 | } |
| 727 | 764 | ||
| 728 | harvest_info = (struct harvest_table *)(adev->mman.discovery_bin + offset); | 765 | harvest_info = (struct harvest_table *)(discovery_bin + offset); |
| 729 | 766 | ||
| 730 | for (i = 0; i < 32; i++) { | 767 | for (i = 0; i < 32; i++) { |
| 731 | if (le16_to_cpu(harvest_info->list[i].hw_id) == 0) | 768 | if (le16_to_cpu(harvest_info->list[i].hw_id) == 0) |
| @@ -1021,8 +1058,8 @@ static void ip_disc_release(struct kobject *kobj) · 289 unmodified lines | |||
| 1021 | kobj); | 1058 | kobj); |
| 1022 | struct amdgpu_device *adev = ip_top->adev; | 1059 | struct amdgpu_device *adev = ip_top->adev; |
| 1023 | 1060 | ||
| 1024 | adev->ip_top = NULL; | ||
| 1025 | kfree(ip_top); | 1061 | kfree(ip_top); |
| 1062 | adev->discovery.ip_top = NULL; | ||
| 1026 | } | 1063 | } |
| 1027 | 1064 | ||
| 1028 | static uint8_t amdgpu_discovery_get_harvest_info(struct amdgpu_device *adev, | 1065 | static uint8_t amdgpu_discovery_get_harvest_info(struct amdgpu_device *adev, |
| @@ -1062,6 +1099,7 @@ static int amdgpu_discovery_sysfs_ips(struct amdgpu_device *adev, · 33 unmodified lines | |||
| 1062 | const size_t _ip_offset, const int num_ips, | 1099 | const size_t _ip_offset, const int num_ips, |
| 1063 | bool reg_base_64) | 1100 | bool reg_base_64) |
| 1064 | { | 1101 | { |
| 1102 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1065 | int ii, jj, kk, res; | 1103 | int ii, jj, kk, res; |
| 1066 | uint16_t hw_id; | 1104 | uint16_t hw_id; |
| 1067 | uint8_t inst; | 1105 | uint8_t inst; |
| @@ -1079,7 +1117,7 @@ static int amdgpu_discovery_sysfs_ips(struct amdgpu_device *adev, · 11 unmodified lines | |||
| 1079 | struct ip_v4 *ip; | 1117 | struct ip_v4 *ip; |
| 1080 | struct ip_hw_instance *ip_hw_instance; | 1118 | struct ip_hw_instance *ip_hw_instance; |
| 1081 | 1119 | ||
| 1082 | ip = (struct ip_v4 *)(adev->mman.discovery_bin + ip_offset); | 1120 | ip = (struct ip_v4 *)(discovery_bin + ip_offset); |
| 1083 | inst = ip->instance_number; | 1121 | inst = ip->instance_number; |
| 1084 | hw_id = le16_to_cpu(ip->hw_id); | 1122 | hw_id = le16_to_cpu(ip->hw_id); |
| 1085 | if (amdgpu_discovery_validate_ip(adev, inst, hw_id) || | 1123 | if (amdgpu_discovery_validate_ip(adev, inst, hw_id) || |
| @@ -1166,17 +1204,20 @@ next_ip: · 80 unmodified lines | |||
| 1166 | 1204 | ||
| 1167 | static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev) | 1205 | static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev) |
| 1168 | { | 1206 | { |
| 1207 | struct ip_discovery_top *ip_top = adev->discovery.ip_top; | ||
| 1208 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1169 | struct binary_header *bhdr; | 1209 | struct binary_header *bhdr; |
| 1170 | struct ip_discovery_header *ihdr; | 1210 | struct ip_discovery_header *ihdr; |
| 1171 | struct die_header *dhdr; | 1211 | struct die_header *dhdr; |
| 1172 | struct kset *die_kset = &adev->ip_top->die_kset; | 1212 | struct kset *die_kset = &ip_top->die_kset; |
| 1173 | u16 num_dies, die_offset, num_ips; | 1213 | u16 num_dies, die_offset, num_ips; |
| 1174 | size_t ip_offset; | 1214 | size_t ip_offset; |
| 1175 | int ii, res; | 1215 | int ii, res; |
| 1176 | 1216 | ||
| 1177 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1217 | bhdr = (struct binary_header *)discovery_bin; |
| 1178 | ihdr = (struct ip_discovery_header *)(adev->mman.discovery_bin + | 1218 | ihdr = (struct ip_discovery_header |
| 1179 | le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset)); | 1219 | *)(discovery_bin + |
| 1220 | le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset)); | ||
| 1180 | num_dies = le16_to_cpu(ihdr->num_dies); | 1221 | num_dies = le16_to_cpu(ihdr->num_dies); |
| 1181 | 1222 | ||
| 1182 | DRM_DEBUG("number of dies: %d\n", num_dies); | 1223 | DRM_DEBUG("number of dies: %d\n", num_dies); |
| @@ -1185,7 +1226,7 @@ static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 1185 | struct ip_die_entry *ip_die_entry; | 1226 | struct ip_die_entry *ip_die_entry; |
| 1186 | 1227 | ||
| 1187 | die_offset = le16_to_cpu(ihdr->die_info[ii].die_offset); | 1228 | die_offset = le16_to_cpu(ihdr->die_info[ii].die_offset); |
| 1188 | dhdr = (struct die_header *)(adev->mman.discovery_bin + die_offset); | 1229 | dhdr = (struct die_header *)(discovery_bin + die_offset); |
| 1189 | num_ips = le16_to_cpu(dhdr->num_ips); | 1230 | num_ips = le16_to_cpu(dhdr->num_ips); |
| 1190 | ip_offset = die_offset + sizeof(*dhdr); | 1231 | ip_offset = die_offset + sizeof(*dhdr); |
| 1191 | 1232 | ||
| @@ -1219,30 +1260,32 @@ static int amdgpu_discovery_sysfs_recurse(struct amdgpu_device *adev) · 27 unmodified lines | |||
| 1219 | 1260 | ||
| 1220 | static int amdgpu_discovery_sysfs_init(struct amdgpu_device *adev) | 1261 | static int amdgpu_discovery_sysfs_init(struct amdgpu_device *adev) |
| 1221 | { | 1262 | { |
| 1263 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1264 | struct ip_discovery_top *ip_top; | ||
| 1222 | struct kset *die_kset; | 1265 | struct kset *die_kset; |
| 1223 | int res, ii; | 1266 | int res, ii; |
| 1224 | 1267 | ||
| 1225 | if (!adev->mman.discovery_bin) | 1268 | if (!discovery_bin) |
| 1226 | return -EINVAL; | 1269 | return -EINVAL; |
| 1227 | 1270 | ||
| 1228 | adev->ip_top = kzalloc(sizeof(*adev->ip_top), GFP_KERNEL); | 1271 | ip_top = kzalloc(sizeof(*ip_top), GFP_KERNEL); |
| 1229 | if (!adev->ip_top) | 1272 | if (!ip_top) |
| 1230 | return -ENOMEM; | 1273 | return -ENOMEM; |
| 1231 | 1274 | ||
| 1232 | adev->ip_top->adev = adev; | 1275 | ip_top->adev = adev; |
| 1233 | 1276 | adev->discovery.ip_top = ip_top; | |
| 1234 | res = kobject_init_and_add(&adev->ip_top->kobj, &ip_discovery_ktype, | 1277 | res = kobject_init_and_add(&ip_top->kobj, &ip_discovery_ktype, |
| 1235 | &adev->dev->kobj, "ip_discovery"); | 1278 | &adev->dev->kobj, "ip_discovery"); |
| 1236 | if (res) { | 1279 | if (res) { |
| 1237 | DRM_ERROR("Couldn't init and add ip_discovery/"); | 1280 | DRM_ERROR("Couldn't init and add ip_discovery/"); |
| 1238 | goto Err; | 1281 | goto Err; |
| 1239 | } | 1282 | } |
| 1240 | 1283 | ||
| 1241 | die_kset = &adev->ip_top->die_kset; | 1284 | die_kset = &ip_top->die_kset; |
| 1242 | kobject_set_name(&die_kset->kobj, "%s", "die"); | 1285 | kobject_set_name(&die_kset->kobj, "%s", "die"); |
| 1243 | die_kset->kobj.parent = &adev->ip_top->kobj; | 1286 | die_kset->kobj.parent = &ip_top->kobj; |
| 1244 | die_kset->kobj.ktype = &die_kobj_ktype; | 1287 | die_kset->kobj.ktype = &die_kobj_ktype; |
| 1245 | res = kset_register(&adev->ip_top->die_kset); | 1288 | res = kset_register(&ip_top->die_kset); |
| 1246 | if (res) { | 1289 | if (res) { |
| 1247 | DRM_ERROR("Couldn't register die_kset"); | 1290 | DRM_ERROR("Couldn't register die_kset"); |
| 1248 | goto Err; | 1291 | goto Err; |
| @@ -1256,7 +1299,7 @@ static int amdgpu_discovery_sysfs_init(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 1256 | 1299 | ||
| 1257 | return res; | 1300 | return res; |
| 1258 | Err: | 1301 | Err: |
| 1259 | kobject_put(&adev->ip_top->kobj); | 1302 | kobject_put(&ip_top->kobj); |
| 1260 | return res; | 1303 | return res; |
| 1261 | } | 1304 | } |
| 1262 | 1305 | ||
| @@ -1301,10 +1344,11 @@ static void amdgpu_discovery_sysfs_die_free(struct ip_die_entry *ip_die_entry) · 38 unmodified lines | |||
| 1301 | 1344 | ||
| 1302 | static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev) | 1345 | static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev) |
| 1303 | { | 1346 | { |
| 1347 | struct ip_discovery_top *ip_top = adev->discovery.ip_top; | ||
| 1304 | struct list_head *el, *tmp; | 1348 | struct list_head *el, *tmp; |
| 1305 | struct kset *die_kset; | 1349 | struct kset *die_kset; |
| 1306 | 1350 | ||
| 1307 | die_kset = &adev->ip_top->die_kset; | 1351 | die_kset = &ip_top->die_kset; |
| 1308 | spin_lock(&die_kset->list_lock); | 1352 | spin_lock(&die_kset->list_lock); |
| 1309 | list_for_each_prev_safe(el, tmp, &die_kset->list) { | 1353 | list_for_each_prev_safe(el, tmp, &die_kset->list) { |
| 1310 | list_del_init(el); | 1354 | list_del_init(el); |
| @@ -1313,8 +1357,8 @@ static void amdgpu_discovery_sysfs_fini(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 1313 | spin_lock(&die_kset->list_lock); | 1357 | spin_lock(&die_kset->list_lock); |
| 1314 | } | 1358 | } |
| 1315 | spin_unlock(&die_kset->list_lock); | 1359 | spin_unlock(&die_kset->list_lock); |
| 1316 | kobject_put(&adev->ip_top->die_kset.kobj); | 1360 | kobject_put(&ip_top->die_kset.kobj); |
| 1317 | kobject_put(&adev->ip_top->kobj); | 1361 | kobject_put(&ip_top->kobj); |
| 1318 | } | 1362 | } |
| 1319 | 1363 | ||
| 1320 | /* ================================================== */ | 1364 | /* ================================================== */ |
| @@ -1325,6 +1369,7 @@ static int amdgpu_discovery_reg_base_init(struct amdgpu_device *adev) · 4 unmodified lines | |||
| 1325 | struct binary_header *bhdr; | 1369 | struct binary_header *bhdr; |
| 1326 | struct ip_discovery_header *ihdr; | 1370 | struct ip_discovery_header *ihdr; |
| 1327 | struct die_header *dhdr; | 1371 | struct die_header *dhdr; |
| 1372 | uint8_t *discovery_bin; | ||
| 1328 | struct ip_v4 *ip; | 1373 | struct ip_v4 *ip; |
| 1329 | uint16_t die_offset; | 1374 | uint16_t die_offset; |
| 1330 | uint16_t ip_offset; | 1375 | uint16_t ip_offset; |
| @@ -1340,22 +1385,23 @@ static int amdgpu_discovery_reg_base_init(struct amdgpu_device *adev) · 9 unmodified lines | |||
| 1340 | r = amdgpu_discovery_init(adev); | 1385 | r = amdgpu_discovery_init(adev); |
| 1341 | if (r) | 1386 | if (r) |
| 1342 | return r; | 1387 | return r; |
| 1343 | 1388 | discovery_bin = adev->discovery.bin; | |
| 1344 | wafl_ver = 0; | 1389 | wafl_ver = 0; |
| 1345 | adev->gfx.xcc_mask = 0; | 1390 | adev->gfx.xcc_mask = 0; |
| 1346 | adev->sdma.sdma_mask = 0; | 1391 | adev->sdma.sdma_mask = 0; |
| 1347 | adev->vcn.inst_mask = 0; | 1392 | adev->vcn.inst_mask = 0; |
| 1348 | adev->jpeg.inst_mask = 0; | 1393 | adev->jpeg.inst_mask = 0; |
| 1349 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1394 | bhdr = (struct binary_header *)discovery_bin; |
| 1350 | ihdr = (struct ip_discovery_header *)(adev->mman.discovery_bin + | 1395 | ihdr = (struct ip_discovery_header |
| 1351 | le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset)); | 1396 | *)(discovery_bin + |
| 1397 | le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset)); | ||
| 1352 | num_dies = le16_to_cpu(ihdr->num_dies); | 1398 | num_dies = le16_to_cpu(ihdr->num_dies); |
| 1353 | 1399 | ||
| 1354 | DRM_DEBUG("number of dies: %d\n", num_dies); | 1400 | DRM_DEBUG("number of dies: %d\n", num_dies); |
| 1355 | 1401 | ||
| 1356 | for (i = 0; i < num_dies; i++) { | 1402 | for (i = 0; i < num_dies; i++) { |
| 1357 | die_offset = le16_to_cpu(ihdr->die_info[i].die_offset); | 1403 | die_offset = le16_to_cpu(ihdr->die_info[i].die_offset); |
| 1358 | dhdr = (struct die_header *)(adev->mman.discovery_bin + die_offset); | 1404 | dhdr = (struct die_header *)(discovery_bin + die_offset); |
| 1359 | num_ips = le16_to_cpu(dhdr->num_ips); | 1405 | num_ips = le16_to_cpu(dhdr->num_ips); |
| 1360 | ip_offset = die_offset + sizeof(*dhdr); | 1406 | ip_offset = die_offset + sizeof(*dhdr); |
| 1361 | 1407 | ||
| @@ -1369,7 +1415,7 @@ static int amdgpu_discovery_reg_base_init(struct amdgpu_device *adev) · 7 unmodified lines | |||
| 1369 | le16_to_cpu(dhdr->die_id), num_ips); | 1415 | le16_to_cpu(dhdr->die_id), num_ips); |
| 1370 | 1416 | ||
| 1371 | for (j = 0; j < num_ips; j++) { | 1417 | for (j = 0; j < num_ips; j++) { |
| 1372 | ip = (struct ip_v4 *)(adev->mman.discovery_bin + ip_offset); | 1418 | ip = (struct ip_v4 *)(discovery_bin + ip_offset); |
| 1373 | 1419 | ||
| 1374 | inst = ip->instance_number; | 1420 | inst = ip->instance_number; |
| 1375 | hw_id = le16_to_cpu(ip->hw_id); | 1421 | hw_id = le16_to_cpu(ip->hw_id); |
| @@ -1519,16 +1565,16 @@ next_ip: · 143 unmodified lines | |||
| 1519 | 1565 | ||
| 1520 | static void amdgpu_discovery_harvest_ip(struct amdgpu_device *adev) | 1566 | static void amdgpu_discovery_harvest_ip(struct amdgpu_device *adev) |
| 1521 | { | 1567 | { |
| 1568 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1522 | struct ip_discovery_header *ihdr; | 1569 | struct ip_discovery_header *ihdr; |
| 1523 | struct binary_header *bhdr; | 1570 | struct binary_header *bhdr; |
| 1524 | int vcn_harvest_count = 0; | 1571 | int vcn_harvest_count = 0; |
| 1525 | int umc_harvest_count = 0; | 1572 | int umc_harvest_count = 0; |
| 1526 | uint16_t offset, ihdr_ver; | 1573 | uint16_t offset, ihdr_ver; |
| 1527 | 1574 | ||
| 1528 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1575 | bhdr = (struct binary_header *)discovery_bin; |
| 1529 | offset = le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset); | 1576 | offset = le16_to_cpu(bhdr->table_list[IP_DISCOVERY].offset); |
| 1530 | ihdr = (struct ip_discovery_header *)(adev->mman.discovery_bin + | 1577 | ihdr = (struct ip_discovery_header *)(discovery_bin + offset); |
| 1531 | offset); | ||
| 1532 | ihdr_ver = le16_to_cpu(ihdr->version); | 1578 | ihdr_ver = le16_to_cpu(ihdr->version); |
| 1533 | /* | 1579 | /* |
| 1534 | * Harvest table does not fit Navi1x and legacy GPUs, | 1580 | * Harvest table does not fit Navi1x and legacy GPUs, |
| @@ -1575,22 +1621,23 @@ union gc_info { · 40 unmodified lines | |||
| 1575 | 1621 | ||
| 1576 | static int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev) | 1622 | static int amdgpu_discovery_get_gfx_info(struct amdgpu_device *adev) |
| 1577 | { | 1623 | { |
| 1624 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1578 | struct binary_header *bhdr; | 1625 | struct binary_header *bhdr; |
| 1579 | union gc_info *gc_info; | 1626 | union gc_info *gc_info; |
| 1580 | u16 offset; | 1627 | u16 offset; |
| 1581 | 1628 | ||
| 1582 | if (!adev->mman.discovery_bin) { | 1629 | if (!discovery_bin) { |
| 1583 | DRM_ERROR("ip discovery uninitialized\n"); | 1630 | DRM_ERROR("ip discovery uninitialized\n"); |
| 1584 | return -EINVAL; | 1631 | return -EINVAL; |
| 1585 | } | 1632 | } |
| 1586 | 1633 | ||
| 1587 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1634 | bhdr = (struct binary_header *)discovery_bin; |
| 1588 | offset = le16_to_cpu(bhdr->table_list[GC].offset); | 1635 | offset = le16_to_cpu(bhdr->table_list[GC].offset); |
| 1589 | 1636 | ||
| 1590 | if (!offset) | 1637 | if (!offset) |
| 1591 | return 0; | 1638 | return 0; |
| 1592 | 1639 | ||
| 1593 | gc_info = (union gc_info *)(adev->mman.discovery_bin + offset); | 1640 | gc_info = (union gc_info *)(discovery_bin + offset); |
| 1594 | 1641 | ||
| 1595 | switch (le16_to_cpu(gc_info->v1.header.version_major)) { | 1642 | switch (le16_to_cpu(gc_info->v1.header.version_major)) { |
| 1596 | case 1: | 1643 | case 1: |
| @@ -1683,24 +1730,25 @@ union mall_info { · 86 unmodified lines | |||
| 1683 | 1730 | ||
| 1684 | static int amdgpu_discovery_get_mall_info(struct amdgpu_device *adev) | 1731 | static int amdgpu_discovery_get_mall_info(struct amdgpu_device *adev) |
| 1685 | { | 1732 | { |
| 1733 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1686 | struct binary_header *bhdr; | 1734 | struct binary_header *bhdr; |
| 1687 | union mall_info *mall_info; | 1735 | union mall_info *mall_info; |
| 1688 | u32 u, mall_size_per_umc, m_s_present, half_use; | 1736 | u32 u, mall_size_per_umc, m_s_present, half_use; |
| 1689 | u64 mall_size; | 1737 | u64 mall_size; |
| 1690 | u16 offset; | 1738 | u16 offset; |
| 1691 | 1739 | ||
| 1692 | if (!adev->mman.discovery_bin) { | 1740 | if (!discovery_bin) { |
| 1693 | DRM_ERROR("ip discovery uninitialized\n"); | 1741 | DRM_ERROR("ip discovery uninitialized\n"); |
| 1694 | return -EINVAL; | 1742 | return -EINVAL; |
| 1695 | } | 1743 | } |
| 1696 | 1744 | ||
| 1697 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1745 | bhdr = (struct binary_header *)discovery_bin; |
| 1698 | offset = le16_to_cpu(bhdr->table_list[MALL_INFO].offset); | 1746 | offset = le16_to_cpu(bhdr->table_list[MALL_INFO].offset); |
| 1699 | 1747 | ||
| 1700 | if (!offset) | 1748 | if (!offset) |
| 1701 | return 0; | 1749 | return 0; |
| 1702 | 1750 | ||
| 1703 | mall_info = (union mall_info *)(adev->mman.discovery_bin + offset); | 1751 | mall_info = (union mall_info *)(discovery_bin + offset); |
| 1704 | 1752 | ||
| 1705 | switch (le16_to_cpu(mall_info->v1.header.version_major)) { | 1753 | switch (le16_to_cpu(mall_info->v1.header.version_major)) { |
| 1706 | case 1: | 1754 | case 1: |
| @@ -1739,12 +1787,13 @@ union vcn_info { · 32 unmodified lines | |||
| 1739 | 1787 | ||
| 1740 | static int amdgpu_discovery_get_vcn_info(struct amdgpu_device *adev) | 1788 | static int amdgpu_discovery_get_vcn_info(struct amdgpu_device *adev) |
| 1741 | { | 1789 | { |
| 1790 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1742 | struct binary_header *bhdr; | 1791 | struct binary_header *bhdr; |
| 1743 | union vcn_info *vcn_info; | 1792 | union vcn_info *vcn_info; |
| 1744 | u16 offset; | 1793 | u16 offset; |
| 1745 | int v; | 1794 | int v; |
| 1746 | 1795 | ||
| 1747 | if (!adev->mman.discovery_bin) { | 1796 | if (!discovery_bin) { |
| 1748 | DRM_ERROR("ip discovery uninitialized\n"); | 1797 | DRM_ERROR("ip discovery uninitialized\n"); |
| 1749 | return -EINVAL; | 1798 | return -EINVAL; |
| 1750 | } | 1799 | } |
| @@ -1759,13 +1808,13 @@ static int amdgpu_discovery_get_vcn_info(struct amdgpu_device *adev) · 8 unmodified lines | |||
| 1759 | return -EINVAL; | 1808 | return -EINVAL; |
| 1760 | } | 1809 | } |
| 1761 | 1810 | ||
| 1762 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1811 | bhdr = (struct binary_header *)discovery_bin; |
| 1763 | offset = le16_to_cpu(bhdr->table_list[VCN_INFO].offset); | 1812 | offset = le16_to_cpu(bhdr->table_list[VCN_INFO].offset); |
| 1764 | 1813 | ||
| 1765 | if (!offset) | 1814 | if (!offset) |
| 1766 | return 0; | 1815 | return 0; |
| 1767 | 1816 | ||
| 1768 | vcn_info = (union vcn_info *)(adev->mman.discovery_bin + offset); | 1817 | vcn_info = (union vcn_info *)(discovery_bin + offset); |
| 1769 | 1818 | ||
| 1770 | switch (le16_to_cpu(vcn_info->v1.header.version_major)) { | 1819 | switch (le16_to_cpu(vcn_info->v1.header.version_major)) { |
| 1771 | case 1: | 1820 | case 1: |
| @@ -1825,6 +1874,7 @@ int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, · 53 unmodified lines | |||
| 1825 | struct amdgpu_gmc_memrange **ranges, | 1874 | struct amdgpu_gmc_memrange **ranges, |
| 1826 | int *range_cnt, bool refresh) | 1875 | int *range_cnt, bool refresh) |
| 1827 | { | 1876 | { |
| 1877 | uint8_t *discovery_bin = adev->discovery.bin; | ||
| 1828 | struct amdgpu_gmc_memrange *mem_ranges; | 1878 | struct amdgpu_gmc_memrange *mem_ranges; |
| 1829 | struct binary_header *bhdr; | 1879 | struct binary_header *bhdr; |
| 1830 | union nps_info *nps_info; | 1880 | union nps_info *nps_info; |
| @@ -1841,13 +1891,13 @@ int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, · 10 unmodified lines | |||
| 1841 | return r; | 1891 | return r; |
| 1842 | nps_info = &nps_data; | 1892 | nps_info = &nps_data; |
| 1843 | } else { | 1893 | } else { |
| 1844 | if (!adev->mman.discovery_bin) { | 1894 | if (!discovery_bin) { |
| 1845 | dev_err(adev->dev, | 1895 | dev_err(adev->dev, |
| 1846 | "fetch mem range failed, ip discovery uninitialized\n"); | 1896 | "fetch mem range failed, ip discovery uninitialized\n"); |
| 1847 | return -EINVAL; | 1897 | return -EINVAL; |
| 1848 | } | 1898 | } |
| 1849 | 1899 | ||
| 1850 | bhdr = (struct binary_header *)adev->mman.discovery_bin; | 1900 | bhdr = (struct binary_header *)discovery_bin; |
| 1851 | offset = le16_to_cpu(bhdr->table_list[NPS_INFO].offset); | 1901 | offset = le16_to_cpu(bhdr->table_list[NPS_INFO].offset); |
| 1852 | 1902 | ||
| 1853 | if (!offset) | 1903 | if (!offset) |
| @@ -1857,8 +1907,7 @@ int amdgpu_discovery_get_nps_info(struct amdgpu_device *adev, · 3 unmodified lines | |||
| 1857 | if (amdgpu_discovery_verify_npsinfo(adev, bhdr)) | 1907 | if (amdgpu_discovery_verify_npsinfo(adev, bhdr)) |
| 1858 | return -ENOENT; | 1908 | return -ENOENT; |
| 1859 | 1909 | ||
| 1860 | nps_info = | 1910 | nps_info = (union nps_info *)(discovery_bin + offset); |
| 1861 | (union nps_info *)(adev->mman.discovery_bin + offset); | ||
| 1862 | } | 1911 | } |
| 1863 | 1912 | ||
| 1864 | switch (le16_to_cpu(nps_info->v1.header.version_major)) { | 1913 | switch (le16_to_cpu(nps_info->v1.header.version_major)) { |
| @@ -2361,6 +2410,21 @@ static int amdgpu_discovery_set_sdma_ip_blocks(struct amdgpu_device *adev) · 496 unmodified lines | |||
| 2361 | amdgpu_ip_version(adev, SDMA0_HWIP, 0)); | 2410 | amdgpu_ip_version(adev, SDMA0_HWIP, 0)); |
| 2362 | return -EINVAL; | 2411 | return -EINVAL; |
| 2363 | } | 2412 | } |
| 2413 | |||
| 2414 | return 0; | ||
| 2415 | } | ||
| 2416 | |||
| 2417 | static int amdgpu_discovery_set_ras_ip_blocks(struct amdgpu_device *adev) | ||
| 2418 | { | ||
| 2419 | switch (amdgpu_ip_version(adev, MP0_HWIP, 0)) { | ||
| 2420 | case IP_VERSION(13, 0, 6): | ||
| 2421 | case IP_VERSION(13, 0, 12): | ||
| 2422 | case IP_VERSION(13, 0, 14): | ||
| 2423 | amdgpu_device_ip_block_add(adev, &ras_v1_0_ip_block); | ||
| 2424 | break; | ||
| 2425 | default: | ||
| 2426 | break; | ||
| 2427 | } | ||
| 2364 | return 0; | 2428 | return 0; |
| 2365 | } | 2429 | } |
| 2366 | 2430 | ||
| @@ -3141,6 +3205,10 @@ int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev) · 774 unmodified lines | |||
| 3141 | if (r) | 3205 | if (r) |
| 3142 | return r; | 3206 | return r; |
| 3143 | 3207 | ||
| 3208 | r = amdgpu_discovery_set_ras_ip_blocks(adev); | ||
| 3209 | if (r) | ||
| 3210 | return r; | ||
| 3211 | |||
| 3144 | if ((adev->firmware.load_type == AMDGPU_FW_LOAD_DIRECT && | 3212 | if ((adev->firmware.load_type == AMDGPU_FW_LOAD_DIRECT && |
| 3145 | !amdgpu_sriov_vf(adev)) || | 3213 | !amdgpu_sriov_vf(adev)) || |
| 3146 | (adev->firmware.load_type == AMDGPU_FW_LOAD_RLC_BACKDOOR_AUTO && amdgpu_dpm == 1)) { | 3214 | (adev->firmware.load_type == AMDGPU_FW_LOAD_RLC_BACKDOOR_AUTO && amdgpu_dpm == 1)) { |
| @@ -24,9 +24,21 @@ · 23 unmodified lines | |||
| 24 | #ifndef __AMDGPU_DISCOVERY__ | 24 | #ifndef __AMDGPU_DISCOVERY__ |
| 25 | #define __AMDGPU_DISCOVERY__ | 25 | #define __AMDGPU_DISCOVERY__ |
| 26 | 26 | ||
| 27 | #include <linux/debugfs.h> | ||
| 28 | |||
| 27 | #define DISCOVERY_TMR_SIZE (10 << 10) | 29 | #define DISCOVERY_TMR_SIZE (10 << 10) |
| 28 | #define DISCOVERY_TMR_OFFSET (64 << 10) | 30 | #define DISCOVERY_TMR_OFFSET (64 << 10) |
| 29 | 31 | ||
| 32 | struct ip_discovery_top; | ||
| 33 | |||
| 34 | struct amdgpu_discovery_info { | ||
| 35 | struct debugfs_blob_wrapper debugfs_blob; | ||
| 36 | struct ip_discovery_top *ip_top; | ||
| 37 | uint32_t size; | ||
| 38 | uint8_t *bin; | ||
| 39 | bool reserve_tmr; | ||
| 40 | }; | ||
| 41 | |||
| 30 | void amdgpu_discovery_fini(struct amdgpu_device *adev); | 42 | void amdgpu_discovery_fini(struct amdgpu_device *adev); |
| 31 | int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev); | 43 | int amdgpu_discovery_set_ip_blocks(struct amdgpu_device *adev); |
| 32 | 44 | ||
| @@ -332,8 +332,6 @@ int amdgpu_display_crtc_set_config(struct drm_mode_set *set, · 331 unmodified lines | |||
| 332 | if (crtc->enabled) | 332 | if (crtc->enabled) |
| 333 | active = true; | 333 | active = true; |
| 334 | 334 | ||
| 335 | pm_runtime_mark_last_busy(dev->dev); | ||
| 336 | |||
| 337 | adev = drm_to_adev(dev); | 335 | adev = drm_to_adev(dev); |
| 338 | /* if we have active crtcs and we don't have a power ref, | 336 | /* if we have active crtcs and we don't have a power ref, |
| 339 | * take the current one | 337 | * take the current one |
| @@ -1365,6 +1363,64 @@ static const struct drm_prop_enum_list amdgpu_dither_enum_list[] = { · 1025 unmodified lines | |||
| 1365 | { AMDGPU_FMT_DITHER_ENABLE, "on" }, | 1363 | { AMDGPU_FMT_DITHER_ENABLE, "on" }, |
| 1366 | }; | 1364 | }; |
| 1367 | 1365 | ||
| 1366 | /** | ||
| 1367 | * DOC: property for adaptive backlight modulation | ||
| 1368 | * | ||
| 1369 | * The 'adaptive backlight modulation' property is used for the compositor to | ||
| 1370 | * directly control the adaptive backlight modulation power savings feature | ||
| 1371 | * that is part of DCN hardware. | ||
| 1372 | * | ||
| 1373 | * The property will be attached specifically to eDP panels that support it. | ||
| 1374 | * | ||
| 1375 | * The property is by default set to 'sysfs' to allow the sysfs file 'panel_power_savings' | ||
| 1376 | * to be able to control it. | ||
| 1377 | * If set to 'off' the compositor will ensure it stays off. | ||
| 1378 | * The other values 'min', 'bias min', 'bias max', and 'max' will control the | ||
| 1379 | * intensity of the power savings. | ||
| 1380 | * | ||
| 1381 | * Modifying this value can have implications on color accuracy, so tread | ||
| 1382 | * carefully. | ||
| 1383 | */ | ||
| 1384 | static int amdgpu_display_setup_abm_prop(struct amdgpu_device *adev) | ||
| 1385 | { | ||
| 1386 | const struct drm_prop_enum_list props[] = { | ||
| 1387 | { ABM_SYSFS_CONTROL, "sysfs" }, | ||
| 1388 | { ABM_LEVEL_OFF, "off" }, | ||
| 1389 | { ABM_LEVEL_MIN, "min" }, | ||
| 1390 | { ABM_LEVEL_BIAS_MIN, "bias min" }, | ||
| 1391 | { ABM_LEVEL_BIAS_MAX, "bias max" }, | ||
| 1392 | { ABM_LEVEL_MAX, "max" }, | ||
| 1393 | }; | ||
| 1394 | struct drm_property *prop; | ||
| 1395 | int i; | ||
| 1396 | |||
| 1397 | if (!adev->dc_enabled) | ||
| 1398 | return 0; | ||
| 1399 | |||
| 1400 | prop = drm_property_create(adev_to_drm(adev), DRM_MODE_PROP_ENUM, | ||
| 1401 | "adaptive backlight modulation", | ||
| 1402 | 6); | ||
| 1403 | if (!prop) | ||
| 1404 | return -ENOMEM; | ||
| 1405 | |||
| 1406 | for (i = 0; i < ARRAY_SIZE(props); i++) { | ||
| 1407 | int ret; | ||
| 1408 | |||
| 1409 | ret = drm_property_add_enum(prop, props[i].type, | ||
| 1410 | props[i].name); | ||
| 1411 | |||
| 1412 | if (ret) { | ||
| 1413 | drm_property_destroy(adev_to_drm(adev), prop); | ||
| 1414 | |||
| 1415 | return ret; | ||
| 1416 | } | ||
| 1417 | } | ||
| 1418 | |||
| 1419 | adev->mode_info.abm_level_property = prop; | ||
| 1420 | |||
| 1421 | return 0; | ||
| 1422 | } | ||
| 1423 | |||
| 1368 | int amdgpu_display_modeset_create_props(struct amdgpu_device *adev) | 1424 | int amdgpu_display_modeset_create_props(struct amdgpu_device *adev) |
| 1369 | { | 1425 | { |
| 1370 | int sz; | 1426 | int sz; |
| @@ -1411,7 +1467,7 @@ int amdgpu_display_modeset_create_props(struct amdgpu_device *adev) · 40 unmodified lines | |||
| 1411 | "dither", | 1467 | "dither", |
| 1412 | amdgpu_dither_enum_list, sz); | 1468 | amdgpu_dither_enum_list, sz); |
| 1413 | 1469 | ||
| 1414 | return 0; | 1470 | return amdgpu_display_setup_abm_prop(adev); |
| 1415 | } | 1471 | } |
| 1416 | 1472 | ||
| 1417 | void amdgpu_display_update_priority(struct amdgpu_device *adev) | 1473 | void amdgpu_display_update_priority(struct amdgpu_device *adev) |
| @@ -55,4 +55,11 @@ int amdgpu_display_resume_helper(struct amdgpu_device *adev); · 54 unmodified lines | |||
| 55 | int amdgpu_display_get_scanout_buffer(struct drm_plane *plane, | 55 | int amdgpu_display_get_scanout_buffer(struct drm_plane *plane, |
| 56 | struct drm_scanout_buffer *sb); | 56 | struct drm_scanout_buffer *sb); |
| 57 | 57 | ||
| 58 | #define ABM_SYSFS_CONTROL -1 | ||
| 59 | #define ABM_LEVEL_OFF 0 | ||
| 60 | #define ABM_LEVEL_MIN 1 | ||
| 61 | #define ABM_LEVEL_BIAS_MIN 2 | ||
| 62 | #define ABM_LEVEL_BIAS_MAX 3 | ||
| 63 | #define ABM_LEVEL_MAX 4 | ||
| 64 | |||
| 58 | #endif | 65 | #endif |
| @@ -81,6 +81,19 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf, · 80 unmodified lines | |||
| 81 | struct drm_gem_object *obj = dmabuf->priv; | 81 | struct drm_gem_object *obj = dmabuf->priv; |
| 82 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); | 82 | struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj); |
| 83 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); | 83 | struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev); |
| 84 | int r; | ||
| 85 | |||
| 86 | /* | ||
| 87 | * Disable peer-to-peer access for DCC-enabled VRAM surfaces on GFX12+. | ||
| 88 | * Such buffers cannot be safely accessed over P2P due to device-local | ||
| 89 | * compression metadata. Fallback to system-memory path instead. | ||
| 90 | * Device supports GFX12 (GC 12.x or newer) | ||
| 91 | * BO was created with the AMDGPU_GEM_CREATE_GFX12_DCC flag | ||
| 92 | * | ||
| 93 | */ | ||
| 94 | if (amdgpu_ip_version(adev, GC_HWIP, 0) >= IP_VERSION(12, 0, 0) && | ||
| 95 | bo->flags & AMDGPU_GEM_CREATE_GFX12_DCC) | ||
| 96 | attach->peer2peer = false; | ||
| 84 | 97 | ||
| 85 | /* | 98 | /* |
| 86 | * Disable peer-to-peer access for DCC-enabled VRAM surfaces on GFX12+. | 99 | * Disable peer-to-peer access for DCC-enabled VRAM surfaces on GFX12+. |
| @@ -98,8 +111,14 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf, · 11 unmodified lines | |||
| 98 | pci_p2pdma_distance(adev->pdev, attach->dev, false) < 0) | 111 | pci_p2pdma_distance(adev->pdev, attach->dev, false) < 0) |
| 99 | attach->peer2peer = false; | 112 | attach->peer2peer = false; |
| 100 | 113 | ||
| 114 | r = dma_resv_lock(bo->tbo.base.resv, NULL); | ||
| 115 | if (r) | ||
| 116 | return r; | ||
| 117 | |||
| 101 | amdgpu_vm_bo_update_shared(bo); | 118 | amdgpu_vm_bo_update_shared(bo); |
| 102 | 119 | ||
| 120 | dma_resv_unlock(bo->tbo.base.resv); | ||
| 121 | |||
| 103 | return 0; | 122 | return 0; |
| 104 | } | 123 | } |
| 105 | 124 | ||
| @@ -312,7 +312,7 @@ module_param_named(moverate, amdgpu_moverate, int, 0600); · 311 unmodified lines | |||
| 312 | * DOC: audio (int) | 312 | * DOC: audio (int) |
| 313 | * Set HDMI/DPAudio. Only affects non-DC display handling. The default is -1 (Enabled), set 0 to disabled it. | 313 | * Set HDMI/DPAudio. Only affects non-DC display handling. The default is -1 (Enabled), set 0 to disabled it. |
| 314 | */ | 314 | */ |
| 315 | MODULE_PARM_DESC(audio, "Audio enable (-1 = auto, 0 = disable, 1 = enable)"); | 315 | MODULE_PARM_DESC(audio, "HDMI/DP Audio enable for non DC displays (-1 = auto, 0 = disable, 1 = enable)"); |
| 316 | module_param_named(audio, amdgpu_audio, int, 0444); | 316 | module_param_named(audio, amdgpu_audio, int, 0444); |
| 317 | 317 | ||
| 318 | /** | 318 | /** |
| @@ -354,22 +354,16 @@ module_param_named(svm_default_granularity, amdgpu_svm_default_granularity, uint · 35 unmodified lines | |||
| 354 | * DOC: lockup_timeout (string) | 354 | * DOC: lockup_timeout (string) |
| 355 | * Set GPU scheduler timeout value in ms. | 355 | * Set GPU scheduler timeout value in ms. |
| 356 | * | 356 | * |
| 357 | * The format can be [Non-Compute] or [GFX,Compute,SDMA,Video]. That is there can be one or | 357 | * The format can be [single value] for setting all timeouts at once or |
| 358 | * multiple values specified. 0 and negative values are invalidated. They will be adjusted | 358 | * [GFX,Compute,SDMA,Video] to set individual timeouts. |
| 359 | * to the default timeout. | 359 | * Negative values mean infinity. |
| 360 | * | 360 | * |
| 361 | * - With one value specified, the setting will apply to all non-compute jobs. | 361 | * By default(with no lockup_timeout settings), the timeout for all queues is 2000. |
| 362 | * - With multiple values specified, the first one will be for GFX. | ||
| 363 | * The second one is for Compute. The third and fourth ones are | ||
| 364 | * for SDMA and Video. | ||
| 365 | * | ||
| 366 | * By default(with no lockup_timeout settings), the timeout for all jobs is 10000. | ||
| 367 | */ | 362 | */ |
| 368 | MODULE_PARM_DESC(lockup_timeout, | 363 | MODULE_PARM_DESC(lockup_timeout, |
| 369 | "GPU lockup timeout in ms (default: 10000 for all jobs. " | 364 | "GPU lockup timeout in ms (default: 2000. 0: keep default value. negative: infinity timeout), format: [single value for all] or [GFX,Compute,SDMA,Video]."); |
| 370 | "0: keep default value. negative: infinity timeout), format: for bare metal [Non-Compute] or [GFX,Compute,SDMA,Video]; " | 365 | module_param_string(lockup_timeout, amdgpu_lockup_timeout, |
| 371 | "for passthrough or sriov [all jobs] or [GFX,Compute,SDMA,Video]."); | 366 | sizeof(amdgpu_lockup_timeout), 0444); |
| 372 | module_param_string(lockup_timeout, amdgpu_lockup_timeout, sizeof(amdgpu_lockup_timeout), 0444); | ||
| 373 | 367 | ||
| 374 | /** | 368 | /** |
| 375 | * DOC: dpm (int) | 369 | * DOC: dpm (int) |
| @@ -624,39 +618,39 @@ module_param_named(timeout_period, amdgpu_watchdog_timer.period, uint, 0644); · 248 unmodified lines | |||
| 624 | 618 | ||
| 625 | /** | 619 | /** |
| 626 | * DOC: si_support (int) | 620 | * DOC: si_support (int) |
| 627 | * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, | 621 | * 1 = enabled, 0 = disabled, -1 = default |
| 628 | * set value 0 to use radeon driver, while set value 1 to use amdgpu driver. The default is using radeon driver when it available, | 622 | * |
| 629 | * otherwise using amdgpu driver. | 623 | * SI (Southern Islands) are first generation GCN GPUs, supported by both |
| 630 | */ | 624 | * drivers: radeon (old) and amdgpu (new). This parameter controls whether |
| 625 | * amdgpu should support SI. | ||
| 626 | * By default, SI dedicated GPUs are supported by amdgpu. | ||
| 627 | * Only relevant when CONFIG_DRM_AMDGPU_SI is enabled to build SI support in amdgpu. | ||
| 628 | * See also radeon.si_support which should be disabled when amdgpu.si_support is | ||
| 629 | * enabled, and vice versa. | ||
| 630 | */ | ||
| 631 | int amdgpu_si_support = -1; | ||
| 631 | #ifdef CONFIG_DRM_AMDGPU_SI | 632 | #ifdef CONFIG_DRM_AMDGPU_SI |
| 632 | 633 | MODULE_PARM_DESC(si_support, "SI support (1 = enabled, 0 = disabled, -1 = default)"); | |
| 633 | #if IS_ENABLED(CONFIG_DRM_RADEON) || IS_ENABLED(CONFIG_DRM_RADEON_MODULE) | ||
| 634 | int amdgpu_si_support; | ||
| 635 | MODULE_PARM_DESC(si_support, "SI support (1 = enabled, 0 = disabled (default))"); | ||
| 636 | #else | ||
| 637 | int amdgpu_si_support = 1; | ||
| 638 | MODULE_PARM_DESC(si_support, "SI support (1 = enabled (default), 0 = disabled)"); | ||
| 639 | #endif | ||
| 640 | |||
| 641 | module_param_named(si_support, amdgpu_si_support, int, 0444); | 634 | module_param_named(si_support, amdgpu_si_support, int, 0444); |
| 642 | #endif | 635 | #endif |
| 643 | 636 | ||
| 644 | /** | 637 | /** |
| 645 | * DOC: cik_support (int) | 638 | * DOC: cik_support (int) |
| 646 | * Set CIK support driver. This parameter works after set config CONFIG_DRM_AMDGPU_CIK. For CIK asic, when radeon driver is enabled, | 639 | * 1 = enabled, 0 = disabled, -1 = default |
| 647 | * set value 0 to use radeon driver, while set value 1 to use amdgpu driver. The default is using radeon driver when it available, | 640 | * |
| 648 | * otherwise using amdgpu driver. | 641 | * CIK (Sea Islands) are second generation GCN GPUs, supported by both |
| 649 | */ | 642 | * drivers: radeon (old) and amdgpu (new). This parameter controls whether |
| 643 | * amdgpu should support CIK. | ||
| 644 | * By default: | ||
| 645 | * - CIK dedicated GPUs are supported by amdgpu. | ||
| 646 | * - CIK APUs are supported by radeon (except when radeon is not built). | ||
| 647 | * Only relevant when CONFIG_DRM_AMDGPU_CIK is enabled to build CIK support in amdgpu. | ||
| 648 | * See also radeon.cik_support which should be disabled when amdgpu.cik_support is | ||
| 649 | * enabled, and vice versa. | ||
| 650 | */ | ||
| 651 | int amdgpu_cik_support = -1; | ||
| 650 | #ifdef CONFIG_DRM_AMDGPU_CIK | 652 | #ifdef CONFIG_DRM_AMDGPU_CIK |
| 651 | 653 | MODULE_PARM_DESC(cik_support, "CIK support (1 = enabled, 0 = disabled, -1 = default)"); | |
| 652 | #if IS_ENABLED(CONFIG_DRM_RADEON) || IS_ENABLED(CONFIG_DRM_RADEON_MODULE) | ||
| 653 | int amdgpu_cik_support; | ||
| 654 | MODULE_PARM_DESC(cik_support, "CIK support (1 = enabled, 0 = disabled (default))"); | ||
| 655 | #else | ||
| 656 | int amdgpu_cik_support = 1; | ||
| 657 | MODULE_PARM_DESC(cik_support, "CIK support (1 = enabled (default), 0 = disabled)"); | ||
| 658 | #endif | ||
| 659 | |||
| 660 | module_param_named(cik_support, amdgpu_cik_support, int, 0444); | 654 | module_param_named(cik_support, amdgpu_cik_support, int, 0444); |
| 661 | #endif | 655 | #endif |
| 662 | 656 | ||
| @@ -2234,7 +2228,6 @@ static void amdgpu_get_secondary_funcs(struct amdgpu_device *adev) · 1571 unmodified lines | |||
| 2234 | adev->pdev->bus->number, i); | 2228 | adev->pdev->bus->number, i); |
| 2235 | if (p) { | 2229 | if (p) { |
| 2236 | pm_runtime_get_sync(&p->dev); | 2230 | pm_runtime_get_sync(&p->dev); |
| 2237 | pm_runtime_mark_last_busy(&p->dev); | ||
| 2238 | pm_runtime_put_autosuspend(&p->dev); | 2231 | pm_runtime_put_autosuspend(&p->dev); |
| 2239 | pci_dev_put(p); | 2232 | pci_dev_put(p); |
| 2240 | } | 2233 | } |
| @@ -2313,6 +2306,72 @@ static unsigned long amdgpu_fix_asic_type(struct pci_dev *pdev, unsigned long fl · 72 unmodified lines | |||
| 2313 | return flags; | 2306 | return flags; |
| 2314 | } | 2307 | } |
| 2315 | 2308 | ||
| 2309 | static bool amdgpu_support_enabled(struct device *dev, | ||
| 2310 | const enum amd_asic_type family) | ||
| 2311 | { | ||
| 2312 | const char *gen; | ||
| 2313 | const char *param; | ||
| 2314 | int module_param = -1; | ||
| 2315 | bool radeon_support_built = IS_ENABLED(CONFIG_DRM_RADEON); | ||
| 2316 | bool amdgpu_support_built = false; | ||
| 2317 | bool support_by_default = false; | ||
| 2318 | |||
| 2319 | switch (family) { | ||
| 2320 | case CHIP_TAHITI: | ||
| 2321 | case CHIP_PITCAIRN: | ||
| 2322 | case CHIP_VERDE: | ||
| 2323 | case CHIP_OLAND: | ||
| 2324 | case CHIP_HAINAN: | ||
| 2325 | gen = "SI"; | ||
| 2326 | param = "si_support"; | ||
| 2327 | module_param = amdgpu_si_support; | ||
| 2328 | amdgpu_support_built = IS_ENABLED(CONFIG_DRM_AMDGPU_SI); | ||
| 2329 | support_by_default = true; | ||
| 2330 | break; | ||
| 2331 | |||
| 2332 | case CHIP_BONAIRE: | ||
| 2333 | case CHIP_HAWAII: | ||
| 2334 | support_by_default = true; | ||
| 2335 | fallthrough; | ||
| 2336 | case CHIP_KAVERI: | ||
| 2337 | case CHIP_KABINI: | ||
| 2338 | case CHIP_MULLINS: | ||
| 2339 | gen = "CIK"; | ||
| 2340 | param = "cik_support"; | ||
| 2341 | module_param = amdgpu_cik_support; | ||
| 2342 | amdgpu_support_built = IS_ENABLED(CONFIG_DRM_AMDGPU_CIK); | ||
| 2343 | break; | ||
| 2344 | |||
| 2345 | default: | ||
| 2346 | /* All other chips are supported by amdgpu only */ | ||
| 2347 | return true; | ||
| 2348 | } | ||
| 2349 | |||
| 2350 | if (!amdgpu_support_built) { | ||
| 2351 | dev_info(dev, "amdgpu built without %s support\n", gen); | ||
| 2352 | return false; | ||
| 2353 | } | ||
| 2354 | |||
| 2355 | if ((module_param == -1 && (support_by_default || !radeon_support_built)) || | ||
| 2356 | module_param == 1) { | ||
| 2357 | if (radeon_support_built) | ||
| 2358 | dev_info(dev, "%s support provided by amdgpu.\n" | ||
| 2359 | "Use radeon.%s=1 amdgpu.%s=0 to override.\n", | ||
| 2360 | gen, param, param); | ||
| 2361 | |||
| 2362 | return true; | ||
| 2363 | } | ||
| 2364 | |||
| 2365 | if (radeon_support_built) | ||
| 2366 | dev_info(dev, "%s support provided by radeon.\n" | ||
| 2367 | "Use radeon.%s=0 amdgpu.%s=1 to override.\n", | ||
| 2368 | gen, param, param); | ||
| 2369 | else if (module_param == 0) | ||
| 2370 | dev_info(dev, "%s support disabled by module param\n", gen); | ||
| 2371 | |||
| 2372 | return false; | ||
| 2373 | } | ||
| 2374 | |||
| 2316 | static int amdgpu_pci_probe(struct pci_dev *pdev, | 2375 | static int amdgpu_pci_probe(struct pci_dev *pdev, |
| 2317 | const struct pci_device_id *ent) | 2376 | const struct pci_device_id *ent) |
| 2318 | { | 2377 | { |
| @@ -2360,48 +2419,8 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, · 41 unmodified lines | |||
| 2360 | return -ENOTSUPP; | 2419 | return -ENOTSUPP; |
| 2361 | } | 2420 | } |
| 2362 | 2421 | ||
| 2363 | switch (flags & AMD_ASIC_MASK) { | 2422 | if (!amdgpu_support_enabled(&pdev->dev, flags & AMD_ASIC_MASK)) |
| 2364 | case CHIP_TAHITI: | ||
| 2365 | case CHIP_PITCAIRN: | ||
| 2366 | case CHIP_VERDE: | ||
| 2367 | case CHIP_OLAND: | ||
| 2368 | case CHIP_HAINAN: | ||
| 2369 | #ifdef CONFIG_DRM_AMDGPU_SI | ||
| 2370 | if (!amdgpu_si_support) { | ||
| 2371 | dev_info(&pdev->dev, | ||
| 2372 | "SI support provided by radeon.\n"); | ||
| 2373 | dev_info(&pdev->dev, | ||
| 2374 | "Use radeon.si_support=0 amdgpu.si_support=1 to override.\n" | ||
| 2375 | ); | ||
| 2376 | return -ENODEV; | ||
| 2377 | } | ||
| 2378 | break; | ||
| 2379 | #else | ||
| 2380 | dev_info(&pdev->dev, "amdgpu is built without SI support.\n"); | ||
| 2381 | return -ENODEV; | ||
| 2382 | #endif | ||
| 2383 | case CHIP_KAVERI: | ||
| 2384 | case CHIP_BONAIRE: | ||
| 2385 | case CHIP_HAWAII: | ||
| 2386 | case CHIP_KABINI: | ||
| 2387 | case CHIP_MULLINS: | ||
| 2388 | #ifdef CONFIG_DRM_AMDGPU_CIK | ||
| 2389 | if (!amdgpu_cik_support) { | ||
| 2390 | dev_info(&pdev->dev, | ||
| 2391 | "CIK support provided by radeon.\n"); | ||
| 2392 | dev_info(&pdev->dev, | ||
| 2393 | "Use radeon.cik_support=0 amdgpu.cik_support=1 to override.\n" | ||
| 2394 | ); | ||
| 2395 | return -ENODEV; | ||
| 2396 | } | ||
| 2397 | break; | ||
| 2398 | #else | ||
| 2399 | dev_info(&pdev->dev, "amdgpu is built without CIK support.\n"); | ||
| 2400 | return -ENODEV; | 2423 | return -ENODEV; |
| 2401 | #endif | ||
| 2402 | default: | ||
| 2403 | break; | ||
| 2404 | } | ||
| 2405 | 2424 | ||
| 2406 | adev = devm_drm_dev_alloc(&pdev->dev, &amdgpu_kms_driver, typeof(*adev), ddev); | 2425 | adev = devm_drm_dev_alloc(&pdev->dev, &amdgpu_kms_driver, typeof(*adev), ddev); |
| 2407 | if (IS_ERR(adev)) | 2426 | if (IS_ERR(adev)) |
| @@ -2480,7 +2499,6 @@ retry_init: · 72 unmodified lines | |||
| 2480 | 2499 | ||
| 2481 | pm_runtime_allow(ddev->dev); | 2500 | pm_runtime_allow(ddev->dev); |
| 2482 | 2501 | ||
| 2483 | pm_runtime_mark_last_busy(ddev->dev); | ||
| 2484 | pm_runtime_put_autosuspend(ddev->dev); | 2502 | pm_runtime_put_autosuspend(ddev->dev); |
| 2485 | 2503 | ||
| 2486 | pci_wake_from_d3(pdev, TRUE); | 2504 | pci_wake_from_d3(pdev, TRUE); |
| @@ -2564,7 +2582,8 @@ amdgpu_pci_shutdown(struct pci_dev *pdev) · 77 unmodified lines | |||
| 2564 | */ | 2582 | */ |
| 2565 | if (!amdgpu_passthrough(adev)) | 2583 | if (!amdgpu_passthrough(adev)) |
| 2566 | adev->mp1_state = PP_MP1_STATE_UNLOAD; | 2584 | adev->mp1_state = PP_MP1_STATE_UNLOAD; |
| 2567 | amdgpu_device_ip_suspend(adev); | 2585 | amdgpu_device_prepare(dev); |
| 2586 | amdgpu_device_suspend(dev, true); | ||
| 2568 | adev->mp1_state = PP_MP1_STATE_NONE; | 2587 | adev->mp1_state = PP_MP1_STATE_NONE; |
| 2569 | } | 2588 | } |
| 2570 | 2589 | ||
| @@ -2782,22 +2801,8 @@ static int amdgpu_runtime_idle_check_userq(struct device *dev) · 211 unmodified lines | |||
| 2782 | struct pci_dev *pdev = to_pci_dev(dev); | 2801 | struct pci_dev *pdev = to_pci_dev(dev); |
| 2783 | struct drm_device *drm_dev = pci_get_drvdata(pdev); | 2802 | struct drm_device *drm_dev = pci_get_drvdata(pdev); |
| 2784 | struct amdgpu_device *adev = drm_to_adev(drm_dev); | 2803 | struct amdgpu_device *adev = drm_to_adev(drm_dev); |
| 2785 | struct amdgpu_usermode_queue *queue; | ||
| 2786 | struct amdgpu_userq_mgr *uqm, *tmp; | ||
| 2787 | int queue_id; | ||
| 2788 | int ret = 0; | ||
| 2789 | |||
| 2790 | mutex_lock(&adev->userq_mutex); | ||
| 2791 | list_for_each_entry_safe(uqm, tmp, &adev->userq_mgr_list, list) { | ||
| 2792 | idr_for_each_entry(&uqm->userq_idr, queue, queue_id) { | ||
| 2793 | ret = -EBUSY; | ||
| 2794 | goto done; | ||
| 2795 | } | ||
| 2796 | } | ||
| 2797 | done: | ||
| 2798 | mutex_unlock(&adev->userq_mutex); | ||
| 2799 | 2804 | ||
| 2800 | return ret; | 2805 | return xa_empty(&adev->userq_doorbell_xa) ? 0 : -EBUSY; |
| 2801 | } | 2806 | } |
| 2802 | 2807 | ||
| 2803 | static int amdgpu_pmops_runtime_suspend(struct device *dev) | 2808 | static int amdgpu_pmops_runtime_suspend(struct device *dev) |
| @@ -2944,7 +2949,6 @@ static int amdgpu_pmops_runtime_idle(struct device *dev) · 140 unmodified lines | |||
| 2944 | 2949 | ||
| 2945 | ret = amdgpu_runtime_idle_check_userq(dev); | 2950 | ret = amdgpu_runtime_idle_check_userq(dev); |
| 2946 | done: | 2951 | done: |
| 2947 | pm_runtime_mark_last_busy(dev); | ||
| 2948 | pm_runtime_autosuspend(dev); | 2952 | pm_runtime_autosuspend(dev); |
| 2949 | return ret; | 2953 | return ret; |
| 2950 | } | 2954 | } |
| @@ -2980,7 +2984,6 @@ long amdgpu_drm_ioctl(struct file *filp, · 29 unmodified lines | |||
| 2980 | 2984 | ||
| 2981 | ret = drm_ioctl(filp, cmd, arg); | 2985 | ret = drm_ioctl(filp, cmd, arg); |
| 2982 | 2986 | ||
| 2983 | pm_runtime_mark_last_busy(dev->dev); | ||
| 2984 | out: | 2987 | out: |
| 2985 | pm_runtime_put_autosuspend(dev->dev); | 2988 | pm_runtime_put_autosuspend(dev->dev); |
| 2986 | return ret; | 2989 | return ret; |
| @@ -45,16 +45,11 @@ · 44 unmodified lines | |||
| 45 | * Cast helper | 45 | * Cast helper |
| 46 | */ | 46 | */ |
| 47 | static const struct dma_fence_ops amdgpu_fence_ops; | 47 | static const struct dma_fence_ops amdgpu_fence_ops; |
| 48 | static const struct dma_fence_ops amdgpu_job_fence_ops; | ||
| 49 | static inline struct amdgpu_fence *to_amdgpu_fence(struct dma_fence *f) | 48 | static inline struct amdgpu_fence *to_amdgpu_fence(struct dma_fence *f) |
| 50 | { | 49 | { |
| 51 | struct amdgpu_fence *__f = container_of(f, struct amdgpu_fence, base); | 50 | struct amdgpu_fence *__f = container_of(f, struct amdgpu_fence, base); |
| 52 | 51 | ||
| 53 | if (__f->base.ops == &amdgpu_fence_ops || | 52 | return __f; |
| 54 | __f->base.ops == &amdgpu_job_fence_ops) | ||
| 55 | return __f; | ||
| 56 | |||
| 57 | return NULL; | ||
| 58 | } | 53 | } |
| 59 | 54 | ||
| 60 | /** | 55 | /** |
| @@ -98,51 +93,32 @@ static u32 amdgpu_fence_read(struct amdgpu_ring *ring) · 37 unmodified lines | |||
| 98 | * amdgpu_fence_emit - emit a fence on the requested ring | 93 | * amdgpu_fence_emit - emit a fence on the requested ring |
| 99 | * | 94 | * |
| 100 | * @ring: ring the fence is associated with | 95 | * @ring: ring the fence is associated with |
| 101 | * @f: resulting fence object | ||
| 102 | * @af: amdgpu fence input | 96 | * @af: amdgpu fence input |
| 103 | * @flags: flags to pass into the subordinate .emit_fence() call | 97 | * @flags: flags to pass into the subordinate .emit_fence() call |
| 104 | * | 98 | * |
| 105 | * Emits a fence command on the requested ring (all asics). | 99 | * Emits a fence command on the requested ring (all asics). |
| 106 | * Returns 0 on success, -ENOMEM on failure. | 100 | * Returns 0 on success, -ENOMEM on failure. |
| 107 | */ | 101 | */ |
| 108 | int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f, | 102 | int amdgpu_fence_emit(struct amdgpu_ring *ring, struct amdgpu_fence *af, |
| 109 | struct amdgpu_fence *af, unsigned int flags) | 103 | unsigned int flags) |
| 110 | { | 104 | { |
| 111 | struct amdgpu_device *adev = ring->adev; | 105 | struct amdgpu_device *adev = ring->adev; |
| 112 | struct dma_fence *fence; | 106 | struct dma_fence *fence; |
| 113 | struct amdgpu_fence *am_fence; | ||
| 114 | struct dma_fence __rcu **ptr; | 107 | struct dma_fence __rcu **ptr; |
| 115 | uint32_t seq; | 108 | uint32_t seq; |
| 116 | int r; | 109 | int r; |
| 117 | 110 | ||
| 118 | if (!af) { | 111 | fence = &af->base; |
| 119 | /* create a separate hw fence */ | 112 | af->ring = ring; |
| 120 | am_fence = kzalloc(sizeof(*am_fence), GFP_KERNEL); | ||
| 121 | if (!am_fence) | ||
| 122 | return -ENOMEM; | ||
| 123 | } else { | ||
| 124 | am_fence = af; | ||
| 125 | } | ||
| 126 | fence = &am_fence->base; | ||
| 127 | am_fence->ring = ring; | ||
| 128 | 113 | ||
| 129 | seq = ++ring->fence_drv.sync_seq; | 114 | seq = ++ring->fence_drv.sync_seq; |
| 130 | am_fence->seq = seq; | 115 | dma_fence_init(fence, &amdgpu_fence_ops, |
| 131 | if (af) { | 116 | &ring->fence_drv.lock, |
| 132 | dma_fence_init(fence, &amdgpu_job_fence_ops, | 117 | adev->fence_context + ring->idx, seq); |
| 133 | &ring->fence_drv.lock, | ||
| 134 | adev->fence_context + ring->idx, seq); | ||
| 135 | /* Against remove in amdgpu_job_{free, free_cb} */ | ||
| 136 | dma_fence_get(fence); | ||
| 137 | } else { | ||
| 138 | dma_fence_init(fence, &amdgpu_fence_ops, | ||
| 139 | &ring->fence_drv.lock, | ||
| 140 | adev->fence_context + ring->idx, seq); | ||
| 141 | } | ||
| 142 | 118 | ||
| 143 | amdgpu_ring_emit_fence(ring, ring->fence_drv.gpu_addr, | 119 | amdgpu_ring_emit_fence(ring, ring->fence_drv.gpu_addr, |
| 144 | seq, flags | AMDGPU_FENCE_FLAG_INT); | 120 | seq, flags | AMDGPU_FENCE_FLAG_INT); |
| 145 | amdgpu_fence_save_wptr(fence); | 121 | amdgpu_fence_save_wptr(af); |
| 146 | pm_runtime_get_noresume(adev_to_drm(adev)->dev); | 122 | pm_runtime_get_noresume(adev_to_drm(adev)->dev); |
| 147 | ptr = &ring->fence_drv.fences[seq & ring->fence_drv.num_fences_mask]; | 123 | ptr = &ring->fence_drv.fences[seq & ring->fence_drv.num_fences_mask]; |
| 148 | if (unlikely(rcu_dereference_protected(*ptr, 1))) { | 124 | if (unlikely(rcu_dereference_protected(*ptr, 1))) { |
| @@ -167,8 +143,6 @@ int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f, · 18 unmodified lines | |||
| 167 | */ | 143 | */ |
| 168 | rcu_assign_pointer(*ptr, dma_fence_get(fence)); | 144 | rcu_assign_pointer(*ptr, dma_fence_get(fence)); |
| 169 | 145 | ||
| 170 | *f = fence; | ||
| 171 | |||
| 172 | return 0; | 146 | return 0; |
| 173 | } | 147 | } |
| 174 | 148 | ||
| @@ -276,7 +250,6 @@ bool amdgpu_fence_process(struct amdgpu_ring *ring) · 101 unmodified lines | |||
| 276 | drv->signalled_wptr = am_fence->wptr; | 250 | drv->signalled_wptr = am_fence->wptr; |
| 277 | dma_fence_signal(fence); | 251 | dma_fence_signal(fence); |
| 278 | dma_fence_put(fence); | 252 | dma_fence_put(fence); |
| 279 | pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); | ||
| 280 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); | 253 | pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); |
| 281 | } while (last_seq != seq); | 254 | } while (last_seq != seq); |
| 282 | 255 | ||
| @@ -669,36 +642,6 @@ void amdgpu_fence_driver_hw_init(struct amdgpu_device *adev) · 386 unmodified lines | |||
| 669 | } | 642 | } |
| 670 | } | 643 | } |
| 671 | 644 | ||
| 672 | /** | ||
| 673 | * amdgpu_fence_driver_clear_job_fences - clear job embedded fences of ring | ||
| 674 | * | ||
| 675 | * @ring: fence of the ring to be cleared | ||
| 676 | * | ||
| 677 | */ | ||
| 678 | void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring) | ||
| 679 | { | ||
| 680 | int i; | ||
| 681 | struct dma_fence *old, **ptr; | ||
| 682 | |||
| 683 | for (i = 0; i <= ring->fence_drv.num_fences_mask; i++) { | ||
| 684 | ptr = &ring->fence_drv.fences[i]; | ||
| 685 | old = rcu_dereference_protected(*ptr, 1); | ||
| 686 | if (old && old->ops == &amdgpu_job_fence_ops) { | ||
| 687 | struct amdgpu_job *job; | ||
| 688 | |||
| 689 | /* For non-scheduler bad job, i.e. failed ib test, we need to signal | ||
| 690 | * it right here or we won't be able to track them in fence_drv | ||
| 691 | * and they will remain unsignaled during sa_bo free. | ||
| 692 | */ | ||
| 693 | job = container_of(old, struct amdgpu_job, hw_fence.base); | ||
| 694 | if (!job->base.s_fence && !dma_fence_is_signaled(old)) | ||
| 695 | dma_fence_signal(old); | ||
| 696 | RCU_INIT_POINTER(*ptr, NULL); | ||
| 697 | dma_fence_put(old); | ||
| 698 | } | ||
| 699 | } | ||
| 700 | } | ||
| 701 | |||
| 702 | /** | 645 | /** |
| 703 | * amdgpu_fence_driver_set_error - set error code on fences | 646 | * amdgpu_fence_driver_set_error - set error code on fences |
| 704 | * @ring: the ring which contains the fences | 647 | * @ring: the ring which contains the fences |
| @@ -755,7 +698,7 @@ void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring) · 50 unmodified lines | |||
| 755 | /** | 698 | /** |
| 756 | * amdgpu_fence_driver_guilty_force_completion - force signal of specified sequence | 699 | * amdgpu_fence_driver_guilty_force_completion - force signal of specified sequence |
| 757 | * | 700 | * |
| 758 | * @fence: fence of the ring to signal | 701 | * @af: fence of the ring to signal |
| 759 | * | 702 | * |
| 760 | */ | 703 | */ |
| 761 | void amdgpu_fence_driver_guilty_force_completion(struct amdgpu_fence *af) | 704 | void amdgpu_fence_driver_guilty_force_completion(struct amdgpu_fence *af) |
| @@ -792,15 +735,13 @@ void amdgpu_fence_driver_guilty_force_completion(struct amdgpu_fence *af) · 30 unmodified lines | |||
| 792 | } while (last_seq != seq); | 735 | } while (last_seq != seq); |
| 793 | spin_unlock_irqrestore(&ring->fence_drv.lock, flags); | 736 | spin_unlock_irqrestore(&ring->fence_drv.lock, flags); |
| 794 | /* signal the guilty fence */ | 737 | /* signal the guilty fence */ |
| 795 | amdgpu_fence_write(ring, af->seq); | 738 | amdgpu_fence_write(ring, (u32)af->base.seqno); |
| 796 | amdgpu_fence_process(ring); | 739 | amdgpu_fence_process(ring); |
| 797 | } | 740 | } |
| 798 | 741 | ||
| 799 | void amdgpu_fence_save_wptr(struct dma_fence *fence) | 742 | void amdgpu_fence_save_wptr(struct amdgpu_fence *af) |
| 800 | { | 743 | { |
| 801 | struct amdgpu_fence *am_fence = container_of(fence, struct amdgpu_fence, base); | 744 | af->wptr = af->ring->wptr; |
| 802 | |||
| 803 | am_fence->wptr = am_fence->ring->wptr; | ||
| 804 | } | 745 | } |
| 805 | 746 | ||
| 806 | static void amdgpu_ring_backup_unprocessed_command(struct amdgpu_ring *ring, | 747 | static void amdgpu_ring_backup_unprocessed_command(struct amdgpu_ring *ring, |
| @@ -866,13 +807,6 @@ static const char *amdgpu_fence_get_timeline_name(struct dma_fence *f) · 59 unmodified lines | |||
| 866 | return (const char *)to_amdgpu_fence(f)->ring->name; | 807 | return (const char *)to_amdgpu_fence(f)->ring->name; |
| 867 | } | 808 | } |
| 868 | 809 | ||
| 869 | static const char *amdgpu_job_fence_get_timeline_name(struct dma_fence *f) | ||
| 870 | { | ||
| 871 | struct amdgpu_job *job = container_of(f, struct amdgpu_job, hw_fence.base); | ||
| 872 | |||
| 873 | return (const char *)to_amdgpu_ring(job->base.sched)->name; | ||
| 874 | } | ||
| 875 | |||
| 876 | /** | 810 | /** |
| 877 | * amdgpu_fence_enable_signaling - enable signalling on fence | 811 | * amdgpu_fence_enable_signaling - enable signalling on fence |
| 878 | * @f: fence | 812 | * @f: fence |
| @@ -889,23 +823,6 @@ static bool amdgpu_fence_enable_signaling(struct dma_fence *f) · 10 unmodified lines | |||
| 889 | return true; | 823 | return true; |
| 890 | } | 824 | } |
| 891 | 825 | ||
| 892 | /** | ||
| 893 | * amdgpu_job_fence_enable_signaling - enable signalling on job fence | ||
| 894 | * @f: fence | ||
| 895 | * | ||
| 896 | * This is the simliar function with amdgpu_fence_enable_signaling above, it | ||
| 897 | * only handles the job embedded fence. | ||
| 898 | */ | ||
| 899 | static bool amdgpu_job_fence_enable_signaling(struct dma_fence *f) | ||
| 900 | { | ||
| 901 | struct amdgpu_job *job = container_of(f, struct amdgpu_job, hw_fence.base); | ||
| 902 | |||
| 903 | if (!timer_pending(&to_amdgpu_ring(job->base.sched)->fence_drv.fallback_timer)) | ||
| 904 | amdgpu_fence_schedule_fallback(to_amdgpu_ring(job->base.sched)); | ||
| 905 | |||
| 906 | return true; | ||
| 907 | } | ||
| 908 | |||
| 909 | /** | 826 | /** |
| 910 | * amdgpu_fence_free - free up the fence memory | 827 | * amdgpu_fence_free - free up the fence memory |
| 911 | * | 828 | * |
| @@ -921,21 +838,6 @@ static void amdgpu_fence_free(struct rcu_head *rcu) · 9 unmodified lines | |||
| 921 | kfree(to_amdgpu_fence(f)); | 838 | kfree(to_amdgpu_fence(f)); |
| 922 | } | 839 | } |
| 923 | 840 | ||
| 924 | /** | ||
| 925 | * amdgpu_job_fence_free - free up the job with embedded fence | ||
| 926 | * | ||
| 927 | * @rcu: RCU callback head | ||
| 928 | * | ||
| 929 | * Free up the job with embedded fence after the RCU grace period. | ||
| 930 | */ | ||
| 931 | static void amdgpu_job_fence_free(struct rcu_head *rcu) | ||
| 932 | { | ||
| 933 | struct dma_fence *f = container_of(rcu, struct dma_fence, rcu); | ||
| 934 | |||
| 935 | /* free job if fence has a parent job */ | ||
| 936 | kfree(container_of(f, struct amdgpu_job, hw_fence.base)); | ||
| 937 | } | ||
| 938 | |||
| 939 | /** | 841 | /** |
| 940 | * amdgpu_fence_release - callback that fence can be freed | 842 | * amdgpu_fence_release - callback that fence can be freed |
| 941 | * | 843 | * |
| @@ -949,19 +851,6 @@ static void amdgpu_fence_release(struct dma_fence *f) · 7 unmodified lines | |||
| 949 | call_rcu(&f->rcu, amdgpu_fence_free); | 851 | call_rcu(&f->rcu, amdgpu_fence_free); |
| 950 | } | 852 | } |
| 951 | 853 | ||
| 952 | /** | ||
| 953 | * amdgpu_job_fence_release - callback that job embedded fence can be freed | ||
| 954 | * | ||
| 955 | * @f: fence | ||
| 956 | * | ||
| 957 | * This is the simliar function with amdgpu_fence_release above, it | ||
| 958 | * only handles the job embedded fence. | ||
| 959 | */ | ||
| 960 | static void amdgpu_job_fence_release(struct dma_fence *f) | ||
| 961 | { | ||
| 962 | call_rcu(&f->rcu, amdgpu_job_fence_free); | ||
| 963 | } | ||
| 964 | |||
| 965 | static const struct dma_fence_ops amdgpu_fence_ops = { | 854 | static const struct dma_fence_ops amdgpu_fence_ops = { |
| 966 | .get_driver_name = amdgpu_fence_get_driver_name, | 855 | .get_driver_name = amdgpu_fence_get_driver_name, |
| 967 | .get_timeline_name = amdgpu_fence_get_timeline_name, | 856 | .get_timeline_name = amdgpu_fence_get_timeline_name, |
| @@ -969,13 +858,6 @@ static const struct dma_fence_ops amdgpu_fence_ops = { · 1 unmodified line | |||
| 969 | .release = amdgpu_fence_release, | 858 | .release = amdgpu_fence_release, |
| 970 | }; | 859 | }; |
| 971 | 860 | ||
| 972 | static const struct dma_fence_ops amdgpu_job_fence_ops = { | ||
| 973 | .get_driver_name = amdgpu_fence_get_driver_name, | ||
| 974 | .get_timeline_name = amdgpu_job_fence_get_timeline_name, | ||
| 975 | .enable_signaling = amdgpu_job_fence_enable_signaling, | ||
| 976 | .release = amdgpu_job_fence_release, | ||
| 977 | }; | ||
| 978 | |||
| 979 | /* | 861 | /* |
| 980 | * Fence debugfs | 862 | * Fence debugfs |
| 981 | */ | 863 | */ |
| @@ -1045,7 +927,6 @@ static int gpu_recover_get(void *data, u64 *val) · 63 unmodified lines | |||
| 1045 | 927 | ||
| 1046 | *val = atomic_read(&adev->reset_domain->reset_res); | 928 | *val = atomic_read(&adev->reset_domain->reset_res); |
| 1047 | 929 | ||
| 1048 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1049 | pm_runtime_put_autosuspend(dev->dev); | 930 | pm_runtime_put_autosuspend(dev->dev); |
| 1050 | 931 | ||
| 1051 | return 0; | 932 | return 0; |
| @@ -302,7 +302,6 @@ void amdgpu_gart_unbind(struct amdgpu_device *adev, uint64_t offset, · 301 unmodified lines | |||
| 302 | int pages) | 302 | int pages) |
| 303 | { | 303 | { |
| 304 | unsigned t; | 304 | unsigned t; |
| 305 | unsigned p; | ||
| 306 | int i, j; | 305 | int i, j; |
| 307 | u64 page_base; | 306 | u64 page_base; |
| 308 | /* Starting from VEGA10, system bit must be 0 to mean invalid. */ | 307 | /* Starting from VEGA10, system bit must be 0 to mean invalid. */ |
| @@ -316,8 +315,7 @@ void amdgpu_gart_unbind(struct amdgpu_device *adev, uint64_t offset, · 7 unmodified lines | |||
| 316 | return; | 315 | return; |
| 317 | 316 | ||
| 318 | t = offset / AMDGPU_GPU_PAGE_SIZE; | 317 | t = offset / AMDGPU_GPU_PAGE_SIZE; |
| 319 | p = t / AMDGPU_GPU_PAGES_IN_CPU_PAGE; | 318 | for (i = 0; i < pages; i++) { |
| 320 | for (i = 0; i < pages; i++, p++) { | ||
| 321 | page_base = adev->dummy_page_addr; | 319 | page_base = adev->dummy_page_addr; |
| 322 | if (!adev->gart.ptr) | 320 | if (!adev->gart.ptr) |
| 323 | continue; | 321 | continue; |
| @@ -369,6 +367,42 @@ void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset, · 45 unmodified lines | |||
| 369 | drm_dev_exit(idx); | 367 | drm_dev_exit(idx); |
| 370 | } | 368 | } |
| 371 | 369 | ||
| 370 | /** | ||
| 371 | * amdgpu_gart_map_vram_range - map VRAM pages into the GART page table | ||
| 372 | * | ||
| 373 | * @adev: amdgpu_device pointer | ||
| 374 | * @pa: physical address of the first page to be mapped | ||
| 375 | * @start_page: first page to map in the GART aperture | ||
| 376 | * @num_pages: number of pages to be mapped | ||
| 377 | * @flags: page table entry flags | ||
| 378 | * @dst: CPU address of the GART table | ||
| 379 | * | ||
| 380 | * Binds a BO that is allocated in VRAM to the GART page table | ||
| 381 | * (all ASICs). | ||
| 382 | * | ||
| 383 | * Useful when a kernel BO is located in VRAM but | ||
| 384 | * needs to be accessed from the GART address space. | ||
| 385 | */ | ||
| 386 | void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, | ||
| 387 | uint64_t start_page, uint64_t num_pages, | ||
| 388 | uint64_t flags, void *dst) | ||
| 389 | { | ||
| 390 | u32 i, idx; | ||
| 391 | |||
| 392 | /* The SYSTEM flag indicates the pages aren't in VRAM. */ | ||
| 393 | WARN_ON_ONCE(flags & AMDGPU_PTE_SYSTEM); | ||
| 394 | |||
| 395 | if (!drm_dev_enter(adev_to_drm(adev), &idx)) | ||
| 396 | return; | ||
| 397 | |||
| 398 | for (i = 0; i < num_pages; ++i) { | ||
| 399 | amdgpu_gmc_set_pte_pde(adev, adev->gart.ptr, | ||
| 400 | start_page + i, pa + AMDGPU_GPU_PAGE_SIZE * i, flags); | ||
| 401 | } | ||
| 402 | |||
| 403 | drm_dev_exit(idx); | ||
| 404 | } | ||
| 405 | |||
| 372 | /** | 406 | /** |
| 373 | * amdgpu_gart_bind - bind pages into the gart page table | 407 | * amdgpu_gart_bind - bind pages into the gart page table |
| 374 | * | 408 | * |
| @@ -64,5 +64,8 @@ void amdgpu_gart_map(struct amdgpu_device *adev, uint64_t offset, · 63 unmodified lines | |||
| 64 | void *dst); | 64 | void *dst); |
| 65 | void amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t offset, | 65 | void amdgpu_gart_bind(struct amdgpu_device *adev, uint64_t offset, |
| 66 | int pages, dma_addr_t *dma_addr, uint64_t flags); | 66 | int pages, dma_addr_t *dma_addr, uint64_t flags); |
| 67 | void amdgpu_gart_map_vram_range(struct amdgpu_device *adev, uint64_t pa, | ||
| 68 | uint64_t start_page, uint64_t num_pages, | ||
| 69 | uint64_t flags, void *dst); | ||
| 67 | void amdgpu_gart_invalidate_tlb(struct amdgpu_device *adev); | 70 | void amdgpu_gart_invalidate_tlb(struct amdgpu_device *adev); |
| 68 | #endif | 71 | #endif |
| @@ -198,7 +198,7 @@ static void amdgpu_gem_object_free(struct drm_gem_object *gobj) · 197 unmodified lines | |||
| 198 | struct amdgpu_bo *aobj = gem_to_amdgpu_bo(gobj); | 198 | struct amdgpu_bo *aobj = gem_to_amdgpu_bo(gobj); |
| 199 | 199 | ||
| 200 | amdgpu_hmm_unregister(aobj); | 200 | amdgpu_hmm_unregister(aobj); |
| 201 | ttm_bo_put(&aobj->tbo); | 201 | ttm_bo_fini(&aobj->tbo); |
| 202 | } | 202 | } |
| 203 | 203 | ||
| 204 | int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size, | 204 | int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size, |
| @@ -531,7 +531,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, · 326 unmodified lines | |||
| 531 | struct drm_amdgpu_gem_userptr *args = data; | 531 | struct drm_amdgpu_gem_userptr *args = data; |
| 532 | struct amdgpu_fpriv *fpriv = filp->driver_priv; | 532 | struct amdgpu_fpriv *fpriv = filp->driver_priv; |
| 533 | struct drm_gem_object *gobj; | 533 | struct drm_gem_object *gobj; |
| 534 | struct hmm_range *range; | 534 | struct amdgpu_hmm_range *range; |
| 535 | struct amdgpu_bo *bo; | 535 | struct amdgpu_bo *bo; |
| 536 | uint32_t handle; | 536 | uint32_t handle; |
| 537 | int r; | 537 | int r; |
| @@ -572,10 +572,14 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, · 34 unmodified lines | |||
| 572 | goto release_object; | 572 | goto release_object; |
| 573 | 573 | ||
| 574 | if (args->flags & AMDGPU_GEM_USERPTR_VALIDATE) { | 574 | if (args->flags & AMDGPU_GEM_USERPTR_VALIDATE) { |
| 575 | r = amdgpu_ttm_tt_get_user_pages(bo, &range); | 575 | range = amdgpu_hmm_range_alloc(NULL); |
| 576 | if (r) | 576 | if (unlikely(!range)) |
| 577 | return -ENOMEM; | ||
| 578 | r = amdgpu_ttm_tt_get_user_pages(bo, range); | ||
| 579 | if (r) { | ||
| 580 | amdgpu_hmm_range_free(range); | ||
| 577 | goto release_object; | 581 | goto release_object; |
| 578 | 582 | } | |
| 579 | r = amdgpu_bo_reserve(bo, true); | 583 | r = amdgpu_bo_reserve(bo, true); |
| 580 | if (r) | 584 | if (r) |
| 581 | goto user_pages_done; | 585 | goto user_pages_done; |
| @@ -597,8 +601,7 @@ int amdgpu_gem_userptr_ioctl(struct drm_device *dev, void *data, · 15 unmodified lines | |||
| 597 | 601 | ||
| 598 | user_pages_done: | 602 | user_pages_done: |
| 599 | if (args->flags & AMDGPU_GEM_USERPTR_VALIDATE) | 603 | if (args->flags & AMDGPU_GEM_USERPTR_VALIDATE) |
| 600 | amdgpu_ttm_tt_get_user_pages_done(bo->tbo.ttm, range); | 604 | amdgpu_hmm_range_free(range); |
| 601 | |||
| 602 | release_object: | 605 | release_object: |
| 603 | drm_gem_object_put(gobj); | 606 | drm_gem_object_put(gobj); |
| 604 | 607 | ||
| @@ -33,6 +33,7 @@ · 32 unmodified lines | |||
| 33 | #include "amdgpu_reset.h" | 33 | #include "amdgpu_reset.h" |
| 34 | #include "amdgpu_xcp.h" | 34 | #include "amdgpu_xcp.h" |
| 35 | #include "amdgpu_xgmi.h" | 35 | #include "amdgpu_xgmi.h" |
| 36 | #include "amdgpu_mes.h" | ||
| 36 | #include "nvd.h" | 37 | #include "nvd.h" |
| 37 | 38 | ||
| 38 | /* delay 0.1 second to enable gfx off feature */ | 39 | /* delay 0.1 second to enable gfx off feature */ |
| @@ -1194,6 +1195,75 @@ failed_kiq_write: · 1155 unmodified lines | |||
| 1194 | dev_err(adev->dev, "failed to write reg:%x\n", reg); | 1195 | dev_err(adev->dev, "failed to write reg:%x\n", reg); |
| 1195 | } | 1196 | } |
| 1196 | 1197 | ||
| 1198 | int amdgpu_kiq_hdp_flush(struct amdgpu_device *adev) | ||
| 1199 | { | ||
| 1200 | signed long r, cnt = 0; | ||
| 1201 | unsigned long flags; | ||
| 1202 | uint32_t seq; | ||
| 1203 | struct amdgpu_kiq *kiq = &adev->gfx.kiq[0]; | ||
| 1204 | struct amdgpu_ring *ring = &kiq->ring; | ||
| 1205 | |||
| 1206 | if (amdgpu_device_skip_hw_access(adev)) | ||
| 1207 | return 0; | ||
| 1208 | |||
| 1209 | if (adev->enable_mes_kiq && adev->mes.ring[0].sched.ready) | ||
| 1210 | return amdgpu_mes_hdp_flush(adev); | ||
| 1211 | |||
| 1212 | if (!ring->funcs->emit_hdp_flush) { | ||
| 1213 | return -EOPNOTSUPP; | ||
| 1214 | } | ||
| 1215 | |||
| 1216 | spin_lock_irqsave(&kiq->ring_lock, flags); | ||
| 1217 | r = amdgpu_ring_alloc(ring, 32); | ||
| 1218 | if (r) | ||
| 1219 | goto failed_unlock; | ||
| 1220 | |||
| 1221 | amdgpu_ring_emit_hdp_flush(ring); | ||
| 1222 | r = amdgpu_fence_emit_polling(ring, &seq, MAX_KIQ_REG_WAIT); | ||
| 1223 | if (r) | ||
| 1224 | goto failed_undo; | ||
| 1225 | |||
| 1226 | amdgpu_ring_commit(ring); | ||
| 1227 | spin_unlock_irqrestore(&kiq->ring_lock, flags); | ||
| 1228 | |||
| 1229 | r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT); | ||
| 1230 | |||
| 1231 | /* don't wait anymore for gpu reset case because this way may | ||
| 1232 | * block gpu_recover() routine forever, e.g. this virt_kiq_rreg | ||
| 1233 | * is triggered in TTM and ttm_bo_lock_delayed_workqueue() will | ||
| 1234 | * never return if we keep waiting in virt_kiq_rreg, which cause | ||
| 1235 | * gpu_recover() hang there. | ||
| 1236 | * | ||
| 1237 | * also don't wait anymore for IRQ context | ||
| 1238 | * */ | ||
| 1239 | if (r < 1 && (amdgpu_in_reset(adev) || in_interrupt())) | ||
| 1240 | goto failed_kiq_hdp_flush; | ||
| 1241 | |||
| 1242 | might_sleep(); | ||
| 1243 | while (r < 1 && cnt++ < MAX_KIQ_REG_TRY) { | ||
| 1244 | if (amdgpu_in_reset(adev)) | ||
| 1245 | goto failed_kiq_hdp_flush; | ||
| 1246 | |||
| 1247 | msleep(MAX_KIQ_REG_BAILOUT_INTERVAL); | ||
| 1248 | r = amdgpu_fence_wait_polling(ring, seq, MAX_KIQ_REG_WAIT); | ||
| 1249 | } | ||
| 1250 | |||
| 1251 | if (cnt > MAX_KIQ_REG_TRY) { | ||
| 1252 | dev_err(adev->dev, "failed to flush HDP via KIQ timeout\n"); | ||
| 1253 | return -ETIMEDOUT; | ||
| 1254 | } | ||
| 1255 | |||
| 1256 | return 0; | ||
| 1257 | |||
| 1258 | failed_undo: | ||
| 1259 | amdgpu_ring_undo(ring); | ||
| 1260 | failed_unlock: | ||
| 1261 | spin_unlock_irqrestore(&kiq->ring_lock, flags); | ||
| 1262 | failed_kiq_hdp_flush: | ||
| 1263 | dev_err(adev->dev, "failed to flush HDP via KIQ\n"); | ||
| 1264 | return r < 0 ? r : -EIO; | ||
| 1265 | } | ||
| 1266 | |||
| 1197 | int amdgpu_gfx_get_num_kcq(struct amdgpu_device *adev) | 1267 | int amdgpu_gfx_get_num_kcq(struct amdgpu_device *adev) |
| 1198 | { | 1268 | { |
| 1199 | if (amdgpu_num_kcq == -1) { | 1269 | if (amdgpu_num_kcq == -1) { |
| @@ -1600,7 +1670,6 @@ static ssize_t amdgpu_gfx_set_run_cleaner_shader(struct device *dev, · 400 unmodified lines | |||
| 1600 | 1670 | ||
| 1601 | ret = amdgpu_gfx_run_cleaner_shader(adev, value); | 1671 | ret = amdgpu_gfx_run_cleaner_shader(adev, value); |
| 1602 | 1672 | ||
| 1603 | pm_runtime_mark_last_busy(ddev->dev); | ||
| 1604 | pm_runtime_put_autosuspend(ddev->dev); | 1673 | pm_runtime_put_autosuspend(ddev->dev); |
| 1605 | 1674 | ||
| 1606 | if (ret) | 1675 | if (ret) |
| @@ -2485,3 +2554,4 @@ void amdgpu_debugfs_compute_sched_mask_init(struct amdgpu_device *adev) · 878 unmodified lines | |||
| 2485 | &amdgpu_debugfs_compute_sched_mask_fops); | 2554 | &amdgpu_debugfs_compute_sched_mask_fops); |
| 2486 | #endif | 2555 | #endif |
| 2487 | } | 2556 | } |
| 2557 | |||
| @@ -615,6 +615,7 @@ int amdgpu_gfx_cp_ecc_error_irq(struct amdgpu_device *adev, · 614 unmodified lines | |||
| 615 | struct amdgpu_iv_entry *entry); | 615 | struct amdgpu_iv_entry *entry); |
| 616 | uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg, uint32_t xcc_id); | 616 | uint32_t amdgpu_kiq_rreg(struct amdgpu_device *adev, uint32_t reg, uint32_t xcc_id); |
| 617 | void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v, uint32_t xcc_id); | 617 | void amdgpu_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v, uint32_t xcc_id); |
| 618 | int amdgpu_kiq_hdp_flush(struct amdgpu_device *adev); | ||
| 618 | int amdgpu_gfx_get_num_kcq(struct amdgpu_device *adev); | 619 | int amdgpu_gfx_get_num_kcq(struct amdgpu_device *adev); |
| 619 | void amdgpu_gfx_cp_init_microcode(struct amdgpu_device *adev, uint32_t ucode_id); | 620 | void amdgpu_gfx_cp_init_microcode(struct amdgpu_device *adev, uint32_t ucode_id); |
| 620 | 621 | ||
| @@ -86,6 +86,11 @@ enum amdgpu_memory_partition { · 85 unmodified lines | |||
| 86 | 86 | ||
| 87 | #define AMDGPU_MAX_MEM_RANGES 8 | 87 | #define AMDGPU_MAX_MEM_RANGES 8 |
| 88 | 88 | ||
| 89 | #define AMDGPU_GMC9_FAULT_SOURCE_DATA_RETRY 0x80 | ||
| 90 | #define AMDGPU_GMC9_FAULT_SOURCE_DATA_READ 0x40 | ||
| 91 | #define AMDGPU_GMC9_FAULT_SOURCE_DATA_WRITE 0x20 | ||
| 92 | #define AMDGPU_GMC9_FAULT_SOURCE_DATA_EXE 0x10 | ||
| 93 | |||
| 89 | /* | 94 | /* |
| 90 | * GMC page fault information | 95 | * GMC page fault information |
| 91 | */ | 96 | */ |
| @@ -284,6 +284,7 @@ int amdgpu_gtt_mgr_init(struct amdgpu_device *adev, uint64_t gtt_size) · 283 unmodified lines | |||
| 284 | ttm_resource_manager_init(man, &adev->mman.bdev, gtt_size); | 284 | ttm_resource_manager_init(man, &adev->mman.bdev, gtt_size); |
| 285 | 285 | ||
| 286 | start = AMDGPU_GTT_MAX_TRANSFER_SIZE * AMDGPU_GTT_NUM_TRANSFER_WINDOWS; | 286 | start = AMDGPU_GTT_MAX_TRANSFER_SIZE * AMDGPU_GTT_NUM_TRANSFER_WINDOWS; |
| 287 | start += amdgpu_vce_required_gart_pages(adev); | ||
| 287 | size = (adev->gmc.gart_size >> PAGE_SHIFT) - start; | 288 | size = (adev->gmc.gart_size >> PAGE_SHIFT) - start; |
| 288 | drm_mm_init(&mgr->mm, start, size); | 289 | drm_mm_init(&mgr->mm, start, size); |
| 289 | spin_lock_init(&mgr->lock); | 290 | spin_lock_init(&mgr->lock); |
| @@ -66,3 +66,19 @@ void amdgpu_hdp_generic_flush(struct amdgpu_device *adev, · 65 unmodified lines | |||
| 66 | 0); | 66 | 0); |
| 67 | } | 67 | } |
| 68 | } | 68 | } |
| 69 | |||
| 70 | void amdgpu_hdp_invalidate(struct amdgpu_device *adev, struct amdgpu_ring *ring) | ||
| 71 | { | ||
| 72 | if (adev->asic_funcs && adev->asic_funcs->invalidate_hdp) | ||
| 73 | adev->asic_funcs->invalidate_hdp(adev, ring); | ||
| 74 | else if (adev->hdp.funcs && adev->hdp.funcs->invalidate_hdp) | ||
| 75 | adev->hdp.funcs->invalidate_hdp(adev, ring); | ||
| 76 | } | ||
| 77 | |||
| 78 | void amdgpu_hdp_flush(struct amdgpu_device *adev, struct amdgpu_ring *ring) | ||
| 79 | { | ||
| 80 | if (adev->asic_funcs && adev->asic_funcs->flush_hdp) | ||
| 81 | adev->asic_funcs->flush_hdp(adev, ring); | ||
| 82 | else if (adev->hdp.funcs && adev->hdp.funcs->flush_hdp) | ||
| 83 | adev->hdp.funcs->flush_hdp(adev, ring); | ||
| 84 | } | ||
| @@ -46,4 +46,8 @@ struct amdgpu_hdp { · 45 unmodified lines | |||
| 46 | int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev); | 46 | int amdgpu_hdp_ras_sw_init(struct amdgpu_device *adev); |
| 47 | void amdgpu_hdp_generic_flush(struct amdgpu_device *adev, | 47 | void amdgpu_hdp_generic_flush(struct amdgpu_device *adev, |
| 48 | struct amdgpu_ring *ring); | 48 | struct amdgpu_ring *ring); |
| 49 | void amdgpu_hdp_invalidate(struct amdgpu_device *adev, | ||
| 50 | struct amdgpu_ring *ring); | ||
| 51 | void amdgpu_hdp_flush(struct amdgpu_device *adev, | ||
| 52 | struct amdgpu_ring *ring); | ||
| 49 | #endif /* __AMDGPU_HDP_H__ */ | 53 | #endif /* __AMDGPU_HDP_H__ */ |
| @@ -168,17 +168,13 @@ void amdgpu_hmm_unregister(struct amdgpu_bo *bo) · 167 unmodified lines | |||
| 168 | int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, | 168 | int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, |
| 169 | uint64_t start, uint64_t npages, bool readonly, | 169 | uint64_t start, uint64_t npages, bool readonly, |
| 170 | void *owner, | 170 | void *owner, |
| 171 | struct hmm_range **phmm_range) | 171 | struct amdgpu_hmm_range *range) |
| 172 | { | 172 | { |
| 173 | struct hmm_range *hmm_range; | ||
| 174 | unsigned long end; | 173 | unsigned long end; |
| 175 | unsigned long timeout; | 174 | unsigned long timeout; |
| 176 | unsigned long *pfns; | 175 | unsigned long *pfns; |
| 177 | int r = 0; | 176 | int r = 0; |
| 178 | 177 | struct hmm_range *hmm_range = &range->hmm_range; | |
| 179 | hmm_range = kzalloc(sizeof(*hmm_range), GFP_KERNEL); | ||
| 180 | if (unlikely(!hmm_range)) | ||
| 181 | return -ENOMEM; | ||
| 182 | 178 | ||
| 183 | pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL); | 179 | pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL); |
| 184 | if (unlikely(!pfns)) { | 180 | if (unlikely(!pfns)) { |
| @@ -221,28 +217,77 @@ retry: · 36 unmodified lines | |||
| 221 | hmm_range->start = start; | 217 | hmm_range->start = start; |
| 222 | hmm_range->hmm_pfns = pfns; | 218 | hmm_range->hmm_pfns = pfns; |
| 223 | 219 | ||
| 224 | *phmm_range = hmm_range; | ||
| 225 | |||
| 226 | return 0; | 220 | return 0; |
| 227 | 221 | ||
| 228 | out_free_pfns: | 222 | out_free_pfns: |
| 229 | kvfree(pfns); | 223 | kvfree(pfns); |
| 224 | hmm_range->hmm_pfns = NULL; | ||
| 230 | out_free_range: | 225 | out_free_range: |
| 231 | kfree(hmm_range); | ||
| 232 | |||
| 233 | if (r == -EBUSY) | 226 | if (r == -EBUSY) |
| 234 | r = -EAGAIN; | 227 | r = -EAGAIN; |
| 235 | return r; | 228 | return r; |
| 236 | } | 229 | } |
| 237 | 230 | ||
| 238 | bool amdgpu_hmm_range_get_pages_done(struct hmm_range *hmm_range) | 231 | /** |
| 232 | * amdgpu_hmm_range_valid - check if an HMM range is still valid | ||
| 233 | * @range: pointer to the &struct amdgpu_hmm_range to validate | ||
| 234 | * | ||
| 235 | * Determines whether the given HMM range @range is still valid by | ||
| 236 | * checking for invalidations via the MMU notifier sequence. This is | ||
| 237 | * typically used to verify that the range has not been invalidated | ||
| 238 | * by concurrent address space updates before it is accessed. | ||
| 239 | * | ||
| 240 | * Return: | ||
| 241 | * * true if @range is valid and can be used safely | ||
| 242 | * * false if @range is NULL or has been invalidated | ||
| 243 | */ | ||
| 244 | bool amdgpu_hmm_range_valid(struct amdgpu_hmm_range *range) | ||
| 239 | { | 245 | { |
| 240 | bool r; | 246 | if (!range) |
| 247 | return false; | ||
| 241 | 248 | ||
| 242 | r = mmu_interval_read_retry(hmm_range->notifier, | 249 | return !mmu_interval_read_retry(range->hmm_range.notifier, |
| 243 | hmm_range->notifier_seq); | 250 | range->hmm_range.notifier_seq); |
| 244 | kvfree(hmm_range->hmm_pfns); | 251 | } |
| 245 | kfree(hmm_range); | ||
| 246 | 252 | ||
| 247 | return r; | 253 | /** |
| 254 | * amdgpu_hmm_range_alloc - allocate and initialize an AMDGPU HMM range | ||
| 255 | * @bo: optional buffer object to associate with this HMM range | ||
| 256 | * | ||
| 257 | * Allocates memory for amdgpu_hmm_range and associates it with the @bo passed. | ||
| 258 | * The reference count of the @bo is incremented. | ||
| 259 | * | ||
| 260 | * Return: | ||
| 261 | * Pointer to a newly allocated struct amdgpu_hmm_range on success, | ||
| 262 | * or NULL if memory allocation fails. | ||
| 263 | */ | ||
| 264 | struct amdgpu_hmm_range *amdgpu_hmm_range_alloc(struct amdgpu_bo *bo) | ||
| 265 | { | ||
| 266 | struct amdgpu_hmm_range *range; | ||
| 267 | |||
| 268 | range = kzalloc(sizeof(*range), GFP_KERNEL); | ||
| 269 | if (!range) | ||
| 270 | return NULL; | ||
| 271 | |||
| 272 | range->bo = amdgpu_bo_ref(bo); | ||
| 273 | return range; | ||
| 274 | } | ||
| 275 | |||
| 276 | /** | ||
| 277 | * amdgpu_hmm_range_free - release an AMDGPU HMM range | ||
| 278 | * @range: pointer to the range object to free | ||
| 279 | * | ||
| 280 | * Releases all resources held by @range, including the associated | ||
| 281 | * hmm_pfns and the dropping reference of associated bo if any. | ||
| 282 | * | ||
| 283 | * Return: void | ||
| 284 | */ | ||
| 285 | void amdgpu_hmm_range_free(struct amdgpu_hmm_range *range) | ||
| 286 | { | ||
| 287 | if (!range) | ||
| 288 | return; | ||
| 289 | |||
| 290 | kvfree(range->hmm_range.hmm_pfns); | ||
| 291 | amdgpu_bo_unref(&range->bo); | ||
| 292 | kfree(range); | ||
| 248 | } | 293 | } |
| @@ -31,13 +31,20 @@ · 30 unmodified lines | |||
| 31 | #include <linux/interval_tree.h> | 31 | #include <linux/interval_tree.h> |
| 32 | #include <linux/mmu_notifier.h> | 32 | #include <linux/mmu_notifier.h> |
| 33 | 33 | ||
| 34 | struct amdgpu_hmm_range { | ||
| 35 | struct hmm_range hmm_range; | ||
| 36 | struct amdgpu_bo *bo; | ||
| 37 | }; | ||
| 38 | |||
| 34 | int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, | 39 | int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, |
| 35 | uint64_t start, uint64_t npages, bool readonly, | 40 | uint64_t start, uint64_t npages, bool readonly, |
| 36 | void *owner, | 41 | void *owner, |
| 37 | struct hmm_range **phmm_range); | 42 | struct amdgpu_hmm_range *range); |
| 38 | bool amdgpu_hmm_range_get_pages_done(struct hmm_range *hmm_range); | ||
| 39 | 43 | ||
| 40 | #if defined(CONFIG_HMM_MIRROR) | 44 | #if defined(CONFIG_HMM_MIRROR) |
| 45 | bool amdgpu_hmm_range_valid(struct amdgpu_hmm_range *range); | ||
| 46 | struct amdgpu_hmm_range *amdgpu_hmm_range_alloc(struct amdgpu_bo *bo); | ||
| 47 | void amdgpu_hmm_range_free(struct amdgpu_hmm_range *range); | ||
| 41 | int amdgpu_hmm_register(struct amdgpu_bo *bo, unsigned long addr); | 48 | int amdgpu_hmm_register(struct amdgpu_bo *bo, unsigned long addr); |
| 42 | void amdgpu_hmm_unregister(struct amdgpu_bo *bo); | 49 | void amdgpu_hmm_unregister(struct amdgpu_bo *bo); |
| 43 | #else | 50 | #else |
| @@ -47,7 +54,20 @@ static inline int amdgpu_hmm_register(struct amdgpu_bo *bo, unsigned long addr) · 3 unmodified lines | |||
| 47 | "add CONFIG_ZONE_DEVICE=y in config file to fix this\n"); | 54 | "add CONFIG_ZONE_DEVICE=y in config file to fix this\n"); |
| 48 | return -ENODEV; | 55 | return -ENODEV; |
| 49 | } | 56 | } |
| 57 | |||
| 50 | static inline void amdgpu_hmm_unregister(struct amdgpu_bo *bo) {} | 58 | static inline void amdgpu_hmm_unregister(struct amdgpu_bo *bo) {} |
| 59 | |||
| 60 | static inline bool amdgpu_hmm_range_valid(struct amdgpu_hmm_range *range) | ||
| 61 | { | ||
| 62 | return false; | ||
| 63 | } | ||
| 64 | |||
| 65 | static inline struct amdgpu_hmm_range *amdgpu_hmm_range_alloc(struct amdgpu_bo *bo) | ||
| 66 | { | ||
| 67 | return NULL; | ||
| 68 | } | ||
| 69 | |||
| 70 | static inline void amdgpu_hmm_range_free(struct amdgpu_hmm_range *range) {} | ||
| 51 | #endif | 71 | #endif |
| 52 | 72 | ||
| 53 | #endif | 73 | #endif |
| @@ -149,17 +149,19 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 148 unmodified lines | |||
| 149 | if (job) { | 149 | if (job) { |
| 150 | vm = job->vm; | 150 | vm = job->vm; |
| 151 | fence_ctx = job->base.s_fence ? | 151 | fence_ctx = job->base.s_fence ? |
| 152 | job->base.s_fence->scheduled.context : 0; | 152 | job->base.s_fence->finished.context : 0; |
| 153 | shadow_va = job->shadow_va; | 153 | shadow_va = job->shadow_va; |
| 154 | csa_va = job->csa_va; | 154 | csa_va = job->csa_va; |
| 155 | gds_va = job->gds_va; | 155 | gds_va = job->gds_va; |
| 156 | init_shadow = job->init_shadow; | 156 | init_shadow = job->init_shadow; |
| 157 | af = &job->hw_fence; | 157 | af = job->hw_fence; |
| 158 | /* Save the context of the job for reset handling. | 158 | /* Save the context of the job for reset handling. |
| 159 | * The driver needs this so it can skip the ring | 159 | * The driver needs this so it can skip the ring |
| 160 | * contents for guilty contexts. | 160 | * contents for guilty contexts. |
| 161 | */ | 161 | */ |
| 162 | af->context = job->base.s_fence ? job->base.s_fence->finished.context : 0; | 162 | af->context = fence_ctx; |
| 163 | /* the vm fence is also part of the job's context */ | ||
| 164 | job->hw_vm_fence->context = fence_ctx; | ||
| 163 | } else { | 165 | } else { |
| 164 | vm = NULL; | 166 | vm = NULL; |
| 165 | fence_ctx = 0; | 167 | fence_ctx = 0; |
| @@ -167,23 +169,28 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 1 unmodified line | |||
| 167 | csa_va = 0; | 169 | csa_va = 0; |
| 168 | gds_va = 0; | 170 | gds_va = 0; |
| 169 | init_shadow = false; | 171 | init_shadow = false; |
| 170 | af = NULL; | 172 | af = kzalloc(sizeof(*af), GFP_ATOMIC); |
| 173 | if (!af) | ||
| 174 | return -ENOMEM; | ||
| 171 | } | 175 | } |
| 172 | 176 | ||
| 173 | if (!ring->sched.ready) { | 177 | if (!ring->sched.ready) { |
| 174 | dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name); | 178 | dev_err(adev->dev, "couldn't schedule ib on ring <%s>\n", ring->name); |
| 175 | return -EINVAL; | 179 | r = -EINVAL; |
| 180 | goto free_fence; | ||
| 176 | } | 181 | } |
| 177 | 182 | ||
| 178 | if (vm && !job->vmid) { | 183 | if (vm && !job->vmid) { |
| 179 | dev_err(adev->dev, "VM IB without ID\n"); | 184 | dev_err(adev->dev, "VM IB without ID\n"); |
| 180 | return -EINVAL; | 185 | r = -EINVAL; |
| 186 | goto free_fence; | ||
| 181 | } | 187 | } |
| 182 | 188 | ||
| 183 | if ((ib->flags & AMDGPU_IB_FLAGS_SECURE) && | 189 | if ((ib->flags & AMDGPU_IB_FLAGS_SECURE) && |
| 184 | (!ring->funcs->secure_submission_supported)) { | 190 | (!ring->funcs->secure_submission_supported)) { |
| 185 | dev_err(adev->dev, "secure submissions not supported on ring <%s>\n", ring->name); | 191 | dev_err(adev->dev, "secure submissions not supported on ring <%s>\n", ring->name); |
| 186 | return -EINVAL; | 192 | r = -EINVAL; |
| 193 | goto free_fence; | ||
| 187 | } | 194 | } |
| 188 | 195 | ||
| 189 | alloc_size = ring->funcs->emit_frame_size + num_ibs * | 196 | alloc_size = ring->funcs->emit_frame_size + num_ibs * |
| @@ -192,7 +199,7 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 2 unmodified lines | |||
| 192 | r = amdgpu_ring_alloc(ring, alloc_size); | 199 | r = amdgpu_ring_alloc(ring, alloc_size); |
| 193 | if (r) { | 200 | if (r) { |
| 194 | dev_err(adev->dev, "scheduling IB failed (%d).\n", r); | 201 | dev_err(adev->dev, "scheduling IB failed (%d).\n", r); |
| 195 | return r; | 202 | goto free_fence; |
| 196 | } | 203 | } |
| 197 | 204 | ||
| 198 | need_ctx_switch = ring->current_ctx != fence_ctx; | 205 | need_ctx_switch = ring->current_ctx != fence_ctx; |
| @@ -289,7 +296,7 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 90 unmodified lines | |||
| 289 | amdgpu_ring_init_cond_exec(ring, ring->cond_exe_gpu_addr); | 296 | amdgpu_ring_init_cond_exec(ring, ring->cond_exe_gpu_addr); |
| 290 | } | 297 | } |
| 291 | 298 | ||
| 292 | r = amdgpu_fence_emit(ring, f, af, fence_flags); | 299 | r = amdgpu_fence_emit(ring, af, fence_flags); |
| 293 | if (r) { | 300 | if (r) { |
| 294 | dev_err(adev->dev, "failed to emit fence (%d)\n", r); | 301 | dev_err(adev->dev, "failed to emit fence (%d)\n", r); |
| 295 | if (job && job->vmid) | 302 | if (job && job->vmid) |
| @@ -297,6 +304,10 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 1 unmodified line | |||
| 297 | amdgpu_ring_undo(ring); | 304 | amdgpu_ring_undo(ring); |
| 298 | return r; | 305 | return r; |
| 299 | } | 306 | } |
| 307 | *f = &af->base; | ||
| 308 | /* get a ref for the job */ | ||
| 309 | if (job) | ||
| 310 | dma_fence_get(*f); | ||
| 300 | 311 | ||
| 301 | if (ring->funcs->insert_end) | 312 | if (ring->funcs->insert_end) |
| 302 | ring->funcs->insert_end(ring); | 313 | ring->funcs->insert_end(ring); |
| @@ -317,12 +328,17 @@ int amdgpu_ib_schedule(struct amdgpu_ring *ring, unsigned int num_ibs, · 14 unmodified lines | |||
| 317 | * fence so we know what rings contents to backup | 328 | * fence so we know what rings contents to backup |
| 318 | * after we reset the queue. | 329 | * after we reset the queue. |
| 319 | */ | 330 | */ |
| 320 | amdgpu_fence_save_wptr(*f); | 331 | amdgpu_fence_save_wptr(af); |
| 321 | 332 | ||
| 322 | amdgpu_ring_ib_end(ring); | 333 | amdgpu_ring_ib_end(ring); |
| 323 | amdgpu_ring_commit(ring); | 334 | amdgpu_ring_commit(ring); |
| 324 | 335 | ||
| 325 | return 0; | 336 | return 0; |
| 337 | |||
| 338 | free_fence: | ||
| 339 | if (!job) | ||
| 340 | kfree(af); | ||
| 341 | return r; | ||
| 326 | } | 342 | } |
| 327 | 343 | ||
| 328 | /** | 344 | /** |
| @@ -201,58 +201,34 @@ static int amdgpu_vmid_grab_idle(struct amdgpu_ring *ring, · 200 unmodified lines | |||
| 201 | struct amdgpu_device *adev = ring->adev; | 201 | struct amdgpu_device *adev = ring->adev; |
| 202 | unsigned vmhub = ring->vm_hub; | 202 | unsigned vmhub = ring->vm_hub; |
| 203 | struct amdgpu_vmid_mgr *id_mgr = &adev->vm_manager.id_mgr[vmhub]; | 203 | struct amdgpu_vmid_mgr *id_mgr = &adev->vm_manager.id_mgr[vmhub]; |
| 204 | struct dma_fence **fences; | ||
| 205 | unsigned i; | ||
| 206 | 204 | ||
| 205 | /* If anybody is waiting for a VMID let everybody wait for fairness */ | ||
| 207 | if (!dma_fence_is_signaled(ring->vmid_wait)) { | 206 | if (!dma_fence_is_signaled(ring->vmid_wait)) { |
| 208 | *fence = dma_fence_get(ring->vmid_wait); | 207 | *fence = dma_fence_get(ring->vmid_wait); |
| 209 | return 0; | 208 | return 0; |
| 210 | } | 209 | } |
| 211 | 210 | ||
| 212 | fences = kmalloc_array(id_mgr->num_ids, sizeof(void *), GFP_NOWAIT); | ||
| 213 | if (!fences) | ||
| 214 | return -ENOMEM; | ||
| 215 | |||
| 216 | /* Check if we have an idle VMID */ | 211 | /* Check if we have an idle VMID */ |
| 217 | i = 0; | 212 | list_for_each_entry_reverse((*idle), &id_mgr->ids_lru, list) { |
| 218 | list_for_each_entry((*idle), &id_mgr->ids_lru, list) { | ||
| 219 | /* Don't use per engine and per process VMID at the same time */ | 213 | /* Don't use per engine and per process VMID at the same time */ |
| 220 | struct amdgpu_ring *r = adev->vm_manager.concurrent_flush ? | 214 | struct amdgpu_ring *r = adev->vm_manager.concurrent_flush ? |
| 221 | NULL : ring; | 215 | NULL : ring; |
| 222 | 216 | ||
| 223 | fences[i] = amdgpu_sync_peek_fence(&(*idle)->active, r); | 217 | *fence = amdgpu_sync_peek_fence(&(*idle)->active, r); |
| 224 | if (!fences[i]) | 218 | if (!(*fence)) |
| 225 | break; | 219 | return 0; |
| 226 | ++i; | ||
| 227 | } | 220 | } |
| 228 | 221 | ||
| 229 | /* If we can't find a idle VMID to use, wait till one becomes available */ | 222 | /* |
| 230 | if (&(*idle)->list == &id_mgr->ids_lru) { | 223 | * If we can't find a idle VMID to use, wait on a fence from the least |
| 231 | u64 fence_context = adev->vm_manager.fence_context + ring->idx; | 224 | * recently used in the hope that it will be available soon. |
| 232 | unsigned seqno = ++adev->vm_manager.seqno[ring->idx]; | 225 | */ |
| 233 | struct dma_fence_array *array; | 226 | *idle = NULL; |
| 234 | unsigned j; | 227 | dma_fence_put(ring->vmid_wait); |
| 235 | 228 | ring->vmid_wait = dma_fence_get(*fence); | |
| 236 | *idle = NULL; | ||
| 237 | for (j = 0; j < i; ++j) | ||
| 238 | dma_fence_get(fences[j]); | ||
| 239 | |||
| 240 | array = dma_fence_array_create(i, fences, fence_context, | ||
| 241 | seqno, true); | ||
| 242 | if (!array) { | ||
| 243 | for (j = 0; j < i; ++j) | ||
| 244 | dma_fence_put(fences[j]); | ||
| 245 | kfree(fences); | ||
| 246 | return -ENOMEM; | ||
| 247 | } | ||
| 248 | |||
| 249 | *fence = dma_fence_get(&array->base); | ||
| 250 | dma_fence_put(ring->vmid_wait); | ||
| 251 | ring->vmid_wait = &array->base; | ||
| 252 | return 0; | ||
| 253 | } | ||
| 254 | kfree(fences); | ||
| 255 | 229 | ||
| 230 | /* This is the reference we return */ | ||
| 231 | dma_fence_get(*fence); | ||
| 256 | return 0; | 232 | return 0; |
| 257 | } | 233 | } |
| 258 | 234 | ||
| @@ -313,7 +289,7 @@ static int amdgpu_vmid_grab_reserved(struct amdgpu_vm *vm, · 54 unmodified lines | |||
| 313 | * user of the VMID. | 289 | * user of the VMID. |
| 314 | */ | 290 | */ |
| 315 | r = amdgpu_sync_fence(&(*id)->active, &job->base.s_fence->finished, | 291 | r = amdgpu_sync_fence(&(*id)->active, &job->base.s_fence->finished, |
| 316 | GFP_NOWAIT); | 292 | GFP_ATOMIC); |
| 317 | if (r) | 293 | if (r) |
| 318 | return r; | 294 | return r; |
| 319 | 295 | ||
| @@ -373,7 +349,7 @@ static int amdgpu_vmid_grab_used(struct amdgpu_vm *vm, · 53 unmodified lines | |||
| 373 | */ | 349 | */ |
| 374 | r = amdgpu_sync_fence(&(*id)->active, | 350 | r = amdgpu_sync_fence(&(*id)->active, |
| 375 | &job->base.s_fence->finished, | 351 | &job->base.s_fence->finished, |
| 376 | GFP_NOWAIT); | 352 | GFP_ATOMIC); |
| 377 | if (r) | 353 | if (r) |
| 378 | return r; | 354 | return r; |
| 379 | 355 | ||
| @@ -426,7 +402,7 @@ int amdgpu_vmid_grab(struct amdgpu_vm *vm, struct amdgpu_ring *ring, · 46 unmodified lines | |||
| 426 | /* Remember this submission as user of the VMID */ | 402 | /* Remember this submission as user of the VMID */ |
| 427 | r = amdgpu_sync_fence(&id->active, | 403 | r = amdgpu_sync_fence(&id->active, |
| 428 | &job->base.s_fence->finished, | 404 | &job->base.s_fence->finished, |
| 429 | GFP_NOWAIT); | 405 | GFP_ATOMIC); |
| 430 | if (r) | 406 | if (r) |
| 431 | goto error; | 407 | goto error; |
| 432 | 408 | ||
| @@ -130,14 +130,12 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) · 129 unmodified lines | |||
| 130 | } | 130 | } |
| 131 | 131 | ||
| 132 | /* attempt a per ring reset */ | 132 | /* attempt a per ring reset */ |
| 133 | if (unlikely(adev->debug_disable_gpu_ring_reset)) { | 133 | if (amdgpu_gpu_recovery && |
| 134 | dev_err(adev->dev, "Ring reset disabled by debug mask\n"); | 134 | amdgpu_ring_is_reset_type_supported(ring, AMDGPU_RESET_TYPE_PER_QUEUE) && |
| 135 | } else if (amdgpu_gpu_recovery && | 135 | ring->funcs->reset) { |
| 136 | amdgpu_ring_is_reset_type_supported(ring, AMDGPU_RESET_TYPE_PER_QUEUE) && | ||
| 137 | ring->funcs->reset) { | ||
| 138 | dev_err(adev->dev, "Starting %s ring reset\n", | 136 | dev_err(adev->dev, "Starting %s ring reset\n", |
| 139 | s_job->sched->name); | 137 | s_job->sched->name); |
| 140 | r = amdgpu_ring_reset(ring, job->vmid, &job->hw_fence); | 138 | r = amdgpu_ring_reset(ring, job->vmid, job->hw_fence); |
| 141 | if (!r) { | 139 | if (!r) { |
| 142 | atomic_inc(&ring->adev->gpu_reset_counter); | 140 | atomic_inc(&ring->adev->gpu_reset_counter); |
| 143 | dev_err(adev->dev, "Ring %s reset succeeded\n", | 141 | dev_err(adev->dev, "Ring %s reset succeeded\n", |
| @@ -186,6 +184,9 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 42 unmodified lines | |||
| 186 | unsigned int num_ibs, struct amdgpu_job **job, | 184 | unsigned int num_ibs, struct amdgpu_job **job, |
| 187 | u64 drm_client_id) | 185 | u64 drm_client_id) |
| 188 | { | 186 | { |
| 187 | struct amdgpu_fence *af; | ||
| 188 | int r; | ||
| 189 | |||
| 189 | if (num_ibs == 0) | 190 | if (num_ibs == 0) |
| 190 | return -EINVAL; | 191 | return -EINVAL; |
| 191 | 192 | ||
| @@ -193,6 +194,20 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 1 unmodified line | |||
| 193 | if (!*job) | 194 | if (!*job) |
| 194 | return -ENOMEM; | 195 | return -ENOMEM; |
| 195 | 196 | ||
| 197 | af = kzalloc(sizeof(struct amdgpu_fence), GFP_KERNEL); | ||
| 198 | if (!af) { | ||
| 199 | r = -ENOMEM; | ||
| 200 | goto err_job; | ||
| 201 | } | ||
| 202 | (*job)->hw_fence = af; | ||
| 203 | |||
| 204 | af = kzalloc(sizeof(struct amdgpu_fence), GFP_KERNEL); | ||
| 205 | if (!af) { | ||
| 206 | r = -ENOMEM; | ||
| 207 | goto err_fence; | ||
| 208 | } | ||
| 209 | (*job)->hw_vm_fence = af; | ||
| 210 | |||
| 196 | (*job)->vm = vm; | 211 | (*job)->vm = vm; |
| 197 | 212 | ||
| 198 | amdgpu_sync_create(&(*job)->explicit_sync); | 213 | amdgpu_sync_create(&(*job)->explicit_sync); |
| @@ -204,6 +219,14 @@ int amdgpu_job_alloc(struct amdgpu_device *adev, struct amdgpu_vm *vm, · 5 unmodified lines | |||
| 204 | 219 | ||
| 205 | return drm_sched_job_init(&(*job)->base, entity, 1, owner, | 220 | return drm_sched_job_init(&(*job)->base, entity, 1, owner, |
| 206 | drm_client_id); | 221 | drm_client_id); |
| 222 | |||
| 223 | err_fence: | ||
| 224 | kfree((*job)->hw_fence); | ||
| 225 | err_job: | ||
| 226 | kfree(*job); | ||
| 227 | *job = NULL; | ||
| 228 | |||
| 229 | return r; | ||
| 207 | } | 230 | } |
| 208 | 231 | ||
| 209 | int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, | 232 | int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, |
| @@ -223,7 +246,10 @@ int amdgpu_job_alloc_with_ib(struct amdgpu_device *adev, · 13 unmodified lines | |||
| 223 | if (r) { | 246 | if (r) { |
| 224 | if (entity) | 247 | if (entity) |
| 225 | drm_sched_job_cleanup(&(*job)->base); | 248 | drm_sched_job_cleanup(&(*job)->base); |
| 249 | kfree((*job)->hw_vm_fence); | ||
| 250 | kfree((*job)->hw_fence); | ||
| 226 | kfree(*job); | 251 | kfree(*job); |
| 252 | *job = NULL; | ||
| 227 | } | 253 | } |
| 228 | 254 | ||
| 229 | return r; | 255 | return r; |
| @@ -251,11 +277,11 @@ void amdgpu_job_free_resources(struct amdgpu_job *job) · 21 unmodified lines | |||
| 251 | struct dma_fence *f; | 277 | struct dma_fence *f; |
| 252 | unsigned i; | 278 | unsigned i; |
| 253 | 279 | ||
| 254 | /* Check if any fences where initialized */ | 280 | /* Check if any fences were initialized */ |
| 255 | if (job->base.s_fence && job->base.s_fence->finished.ops) | 281 | if (job->base.s_fence && job->base.s_fence->finished.ops) |
| 256 | f = &job->base.s_fence->finished; | 282 | f = &job->base.s_fence->finished; |
| 257 | else if (job->hw_fence.base.ops) | 283 | else if (job->hw_fence && job->hw_fence->base.ops) |
| 258 | f = &job->hw_fence.base; | 284 | f = &job->hw_fence->base; |
| 259 | else | 285 | else |
| 260 | f = NULL; | 286 | f = NULL; |
| 261 | 287 | ||
| @@ -271,11 +297,16 @@ static void amdgpu_job_free_cb(struct drm_sched_job *s_job) · 9 unmodified lines | |||
| 271 | 297 | ||
| 272 | amdgpu_sync_free(&job->explicit_sync); | 298 | amdgpu_sync_free(&job->explicit_sync); |
| 273 | 299 | ||
| 274 | /* only put the hw fence if has embedded fence */ | 300 | if (job->hw_fence->base.ops) |
| 275 | if (!job->hw_fence.base.ops) | 301 | dma_fence_put(&job->hw_fence->base); |
| 276 | kfree(job); | 302 | else |
| 303 | kfree(job->hw_fence); | ||
| 304 | if (job->hw_vm_fence->base.ops) | ||
| 305 | dma_fence_put(&job->hw_vm_fence->base); | ||
| 277 | else | 306 | else |
| 278 | dma_fence_put(&job->hw_fence.base); | 307 | kfree(job->hw_vm_fence); |
| 308 | |||
| 309 | kfree(job); | ||
| 279 | } | 310 | } |
| 280 | 311 | ||
| 281 | void amdgpu_job_set_gang_leader(struct amdgpu_job *job, | 312 | void amdgpu_job_set_gang_leader(struct amdgpu_job *job, |
| @@ -304,10 +335,16 @@ void amdgpu_job_free(struct amdgpu_job *job) · 22 unmodified lines | |||
| 304 | if (job->gang_submit != &job->base.s_fence->scheduled) | 335 | if (job->gang_submit != &job->base.s_fence->scheduled) |
| 305 | dma_fence_put(job->gang_submit); | 336 | dma_fence_put(job->gang_submit); |
| 306 | 337 | ||
| 307 | if (!job->hw_fence.base.ops) | 338 | if (job->hw_fence->base.ops) |
| 308 | kfree(job); | 339 | dma_fence_put(&job->hw_fence->base); |
| 340 | else | ||
| 341 | kfree(job->hw_fence); | ||
| 342 | if (job->hw_vm_fence->base.ops) | ||
| 343 | dma_fence_put(&job->hw_vm_fence->base); | ||
| 309 | else | 344 | else |
| 310 | dma_fence_put(&job->hw_fence.base); | 345 | kfree(job->hw_vm_fence); |
| 346 | |||
| 347 | kfree(job); | ||
| 311 | } | 348 | } |
| 312 | 349 | ||
| 313 | struct dma_fence *amdgpu_job_submit(struct amdgpu_job *job) | 350 | struct dma_fence *amdgpu_job_submit(struct amdgpu_job *job) |
| @@ -64,7 +64,8 @@ struct amdgpu_job { · 63 unmodified lines | |||
| 64 | struct drm_sched_job base; | 64 | struct drm_sched_job base; |
| 65 | struct amdgpu_vm *vm; | 65 | struct amdgpu_vm *vm; |
| 66 | struct amdgpu_sync explicit_sync; | 66 | struct amdgpu_sync explicit_sync; |
| 67 | struct amdgpu_fence hw_fence; | 67 | struct amdgpu_fence *hw_fence; |
| 68 | struct amdgpu_fence *hw_vm_fence; | ||
| 68 | struct dma_fence *gang_submit; | 69 | struct dma_fence *gang_submit; |
| 69 | uint32_t preamble_status; | 70 | uint32_t preamble_status; |
| 70 | uint32_t preemption_status; | 71 | uint32_t preemption_status; |
| @@ -1471,7 +1471,6 @@ error_pasid: · 1470 unmodified lines | |||
| 1471 | kfree(fpriv); | 1471 | kfree(fpriv); |
| 1472 | 1472 | ||
| 1473 | out_suspend: | 1473 | out_suspend: |
| 1474 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1475 | pm_put: | 1474 | pm_put: |
| 1476 | pm_runtime_put_autosuspend(dev->dev); | 1475 | pm_runtime_put_autosuspend(dev->dev); |
| 1477 | 1476 | ||
| @@ -1539,7 +1538,6 @@ void amdgpu_driver_postclose_kms(struct drm_device *dev, · 61 unmodified lines | |||
| 1539 | kfree(fpriv); | 1538 | kfree(fpriv); |
| 1540 | file_priv->driver_priv = NULL; | 1539 | file_priv->driver_priv = NULL; |
| 1541 | 1540 | ||
| 1542 | pm_runtime_mark_last_busy(dev->dev); | ||
| 1543 | pm_runtime_put_autosuspend(dev->dev); | 1541 | pm_runtime_put_autosuspend(dev->dev); |
| 1544 | } | 1542 | } |
| 1545 | 1543 | ||
| @@ -105,8 +105,8 @@ int amdgpu_mes_init(struct amdgpu_device *adev) · 104 unmodified lines | |||
| 105 | spin_lock_init(&adev->mes.ring_lock[i]); | 105 | spin_lock_init(&adev->mes.ring_lock[i]); |
| 106 | 106 | ||
| 107 | adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK; | 107 | adev->mes.total_max_queue = AMDGPU_FENCE_MES_QUEUE_ID_MASK; |
| 108 | adev->mes.vmid_mask_mmhub = 0xffffff00; | 108 | adev->mes.vmid_mask_mmhub = 0xFF00; |
| 109 | adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xfffffffe : 0xffffff00; | 109 | adev->mes.vmid_mask_gfxhub = adev->gfx.disable_kq ? 0xFFFE : 0xFF00; |
| 110 | 110 | ||
| 111 | num_pipes = adev->gfx.me.num_pipe_per_me * adev->gfx.me.num_me; | 111 | num_pipes = adev->gfx.me.num_pipe_per_me * adev->gfx.me.num_me; |
| 112 | if (num_pipes > AMDGPU_MES_MAX_GFX_PIPES) | 112 | if (num_pipes > AMDGPU_MES_MAX_GFX_PIPES) |
| @@ -528,6 +528,18 @@ error: · 415 unmodified lines | |||
| 528 | return r; | 528 | return r; |
| 529 | } | 529 | } |
| 530 | 530 | ||
| 531 | int amdgpu_mes_hdp_flush(struct amdgpu_device *adev) | ||
| 532 | { | ||
| 533 | uint32_t hdp_flush_req_offset, hdp_flush_done_offset, ref_and_mask; | ||
| 534 | |||
| 535 | hdp_flush_req_offset = adev->nbio.funcs->get_hdp_flush_req_offset(adev); | ||
| 536 | hdp_flush_done_offset = adev->nbio.funcs->get_hdp_flush_done_offset(adev); | ||
| 537 | ref_and_mask = adev->nbio.hdp_flush_reg->ref_and_mask_cp0; | ||
| 538 | |||
| 539 | return amdgpu_mes_reg_write_reg_wait(adev, hdp_flush_req_offset, hdp_flush_done_offset, | ||
| 540 | ref_and_mask, ref_and_mask); | ||
| 541 | } | ||
| 542 | |||
| 531 | int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, | 543 | int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, |
| 532 | uint64_t process_context_addr, | 544 | uint64_t process_context_addr, |
| 533 | uint32_t spi_gdbg_per_vmid_cntl, | 545 | uint32_t spi_gdbg_per_vmid_cntl, |
| @@ -239,6 +239,7 @@ struct mes_add_queue_input { · 238 unmodified lines | |||
| 239 | struct mes_remove_queue_input { | 239 | struct mes_remove_queue_input { |
| 240 | uint32_t doorbell_offset; | 240 | uint32_t doorbell_offset; |
| 241 | uint64_t gang_context_addr; | 241 | uint64_t gang_context_addr; |
| 242 | bool remove_queue_after_reset; | ||
| 242 | }; | 243 | }; |
| 243 | 244 | ||
| 244 | struct mes_map_legacy_queue_input { | 245 | struct mes_map_legacy_queue_input { |
| @@ -428,6 +429,7 @@ int amdgpu_mes_wreg(struct amdgpu_device *adev, · 183 unmodified lines | |||
| 428 | int amdgpu_mes_reg_write_reg_wait(struct amdgpu_device *adev, | 429 | int amdgpu_mes_reg_write_reg_wait(struct amdgpu_device *adev, |
| 429 | uint32_t reg0, uint32_t reg1, | 430 | uint32_t reg0, uint32_t reg1, |
| 430 | uint32_t ref, uint32_t mask); | 431 | uint32_t ref, uint32_t mask); |
| 432 | int amdgpu_mes_hdp_flush(struct amdgpu_device *adev); | ||
| 431 | int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, | 433 | int amdgpu_mes_set_shader_debugger(struct amdgpu_device *adev, |
| 432 | uint64_t process_context_addr, | 434 | uint64_t process_context_addr, |
| 433 | uint32_t spi_gdbg_per_vmid_cntl, | 435 | uint32_t spi_gdbg_per_vmid_cntl, |
| @@ -326,6 +326,8 @@ struct amdgpu_mode_info { · 325 unmodified lines | |||
| 326 | struct drm_property *audio_property; | 326 | struct drm_property *audio_property; |
| 327 | /* FMT dithering */ | 327 | /* FMT dithering */ |
| 328 | struct drm_property *dither_property; | 328 | struct drm_property *dither_property; |
| 329 | /* Adaptive Backlight Modulation (power feature) */ | ||
| 330 | struct drm_property *abm_level_property; | ||
| 329 | /* hardcoded DFP edid from BIOS */ | 331 | /* hardcoded DFP edid from BIOS */ |
| 330 | const struct drm_edid *bios_hardcoded_edid; | 332 | const struct drm_edid *bios_hardcoded_edid; |
| 331 | 333 | ||
| @@ -96,6 +96,7 @@ struct amdgpu_bo_va { · 95 unmodified lines | |||
| 96 | * if non-zero, cannot unmap from GPU because user queues may still access it | 96 | * if non-zero, cannot unmap from GPU because user queues may still access it |
| 97 | */ | 97 | */ |
| 98 | unsigned int queue_refcount; | 98 | unsigned int queue_refcount; |
| 99 | atomic_t userq_va_mapped; | ||
| 99 | }; | 100 | }; |
| 100 | 101 | ||
| 101 | struct amdgpu_bo { | 102 | struct amdgpu_bo { |
| @@ -1539,6 +1539,7 @@ static void psp_xgmi_reflect_topology_info(struct psp_context *psp, · 1538 unmodified lines | |||
| 1539 | uint64_t src_node_id = psp->adev->gmc.xgmi.node_id; | 1539 | uint64_t src_node_id = psp->adev->gmc.xgmi.node_id; |
| 1540 | uint64_t dst_node_id = node_info.node_id; | 1540 | uint64_t dst_node_id = node_info.node_id; |
| 1541 | uint8_t dst_num_hops = node_info.num_hops; | 1541 | uint8_t dst_num_hops = node_info.num_hops; |
| 1542 | uint8_t dst_is_sharing_enabled = node_info.is_sharing_enabled; | ||
| 1542 | uint8_t dst_num_links = node_info.num_links; | 1543 | uint8_t dst_num_links = node_info.num_links; |
| 1543 | 1544 | ||
| 1544 | hive = amdgpu_get_xgmi_hive(psp->adev); | 1545 | hive = amdgpu_get_xgmi_hive(psp->adev); |
| @@ -1558,13 +1559,20 @@ static void psp_xgmi_reflect_topology_info(struct psp_context *psp, · 13 unmodified lines | |||
| 1558 | continue; | 1559 | continue; |
| 1559 | 1560 | ||
| 1560 | mirror_top_info->nodes[j].num_hops = dst_num_hops; | 1561 | mirror_top_info->nodes[j].num_hops = dst_num_hops; |
| 1561 | /* | 1562 | mirror_top_info->nodes[j].is_sharing_enabled = dst_is_sharing_enabled; |
| 1562 | * prevent 0 num_links value re-reflection since reflection | 1563 | /* prevent 0 num_links value re-reflection since reflection |
| 1563 | * criteria is based on num_hops (direct or indirect). | 1564 | * criteria is based on num_hops (direct or indirect). |
| 1564 | * | ||
| 1565 | */ | 1565 | */ |
| 1566 | if (dst_num_links) | 1566 | if (dst_num_links) { |
| 1567 | mirror_top_info->nodes[j].num_links = dst_num_links; | 1567 | mirror_top_info->nodes[j].num_links = dst_num_links; |
| 1568 | /* swap src and dst due to frame of reference */ | ||
| 1569 | for (int k = 0; k < dst_num_links; k++) { | ||
| 1570 | mirror_top_info->nodes[j].port_num[k].src_xgmi_port_num = | ||
| 1571 | node_info.port_num[k].dst_xgmi_port_num; | ||
| 1572 | mirror_top_info->nodes[j].port_num[k].dst_xgmi_port_num = | ||
| 1573 | node_info.port_num[k].src_xgmi_port_num; | ||
| 1574 | } | ||
| 1575 | } | ||
| 1568 | 1576 | ||
| 1569 | break; | 1577 | break; |
| 1570 | } | 1578 | } |
| @@ -1639,9 +1647,10 @@ int psp_xgmi_get_topology_info(struct psp_context *psp, · 68 unmodified lines | |||
| 1639 | amdgpu_ip_version(psp->adev, MP0_HWIP, 0) == | 1647 | amdgpu_ip_version(psp->adev, MP0_HWIP, 0) == |
| 1640 | IP_VERSION(13, 0, 6) || | 1648 | IP_VERSION(13, 0, 6) || |
| 1641 | amdgpu_ip_version(psp->adev, MP0_HWIP, 0) == | 1649 | amdgpu_ip_version(psp->adev, MP0_HWIP, 0) == |
| 1642 | IP_VERSION(13, 0, 14); | 1650 | IP_VERSION(13, 0, 14) || |
| 1643 | bool ta_port_num_support = amdgpu_sriov_vf(psp->adev) ? 0 : | 1651 | amdgpu_sriov_vf(psp->adev); |
| 1644 | psp->xgmi_context.xgmi_ta_caps & EXTEND_PEER_LINK_INFO_CMD_FLAG; | 1652 | bool ta_port_num_support = psp->xgmi_context.xgmi_ta_caps & EXTEND_PEER_LINK_INFO_CMD_FLAG || |
| 1653 | amdgpu_sriov_xgmi_ta_ext_peer_link_en(psp->adev); | ||
| 1645 | 1654 | ||
| 1646 | /* popluate the shared output buffer rather than the cmd input buffer | 1655 | /* popluate the shared output buffer rather than the cmd input buffer |
| 1647 | * with node_ids as the input for GET_PEER_LINKS command execution. | 1656 | * with node_ids as the input for GET_PEER_LINKS command execution. |
| @@ -101,7 +101,6 @@ static ssize_t amdgpu_rap_debugfs_write(struct file *f, const char __user *buf, · 100 unmodified lines | |||
| 101 | } | 101 | } |
| 102 | 102 | ||
| 103 | amdgpu_gfx_off_ctrl(adev, true); | 103 | amdgpu_gfx_off_ctrl(adev, true); |
| 104 | pm_runtime_mark_last_busy(dev->dev); | ||
| 105 | pm_runtime_put_autosuspend(dev->dev); | 104 | pm_runtime_put_autosuspend(dev->dev); |
| 106 | 105 | ||
| 107 | return size; | 106 | return size; |
| @@ -41,6 +41,7 @@ · 40 unmodified lines | |||
| 41 | #include "atom.h" | 41 | #include "atom.h" |
| 42 | #include "amdgpu_reset.h" | 42 | #include "amdgpu_reset.h" |
| 43 | #include "amdgpu_psp.h" | 43 | #include "amdgpu_psp.h" |
| 44 | #include "amdgpu_ras_mgr.h" | ||
| 44 | 45 | ||
| 45 | #ifdef CONFIG_X86_MCE_AMD | 46 | #ifdef CONFIG_X86_MCE_AMD |
| 46 | #include <asm/mce.h> | 47 | #include <asm/mce.h> |
| @@ -149,6 +150,8 @@ static void amdgpu_ras_critical_region_fini(struct amdgpu_device *adev); · 102 unmodified lines | |||
| 149 | 150 | ||
| 150 | #ifdef CONFIG_X86_MCE_AMD | 151 | #ifdef CONFIG_X86_MCE_AMD |
| 151 | static void amdgpu_register_bad_pages_mca_notifier(struct amdgpu_device *adev); | 152 | static void amdgpu_register_bad_pages_mca_notifier(struct amdgpu_device *adev); |
| 153 | static void | ||
| 154 | amdgpu_unregister_bad_pages_mca_notifier(struct amdgpu_device *adev); | ||
| 152 | struct mce_notifier_adev_list { | 155 | struct mce_notifier_adev_list { |
| 153 | struct amdgpu_device *devs[MAX_GPU_INSTANCE]; | 156 | struct amdgpu_device *devs[MAX_GPU_INSTANCE]; |
| 154 | int num_gpu; | 157 | int num_gpu; |
| @@ -611,6 +614,8 @@ static ssize_t amdgpu_ras_debugfs_ctrl_write(struct file *f, · 456 unmodified lines | |||
| 611 | return size; | 614 | return size; |
| 612 | } | 615 | } |
| 613 | 616 | ||
| 617 | static int amdgpu_uniras_clear_badpages_info(struct amdgpu_device *adev); | ||
| 618 | |||
| 614 | /** | 619 | /** |
| 615 | * DOC: AMDGPU RAS debugfs EEPROM table reset interface | 620 | * DOC: AMDGPU RAS debugfs EEPROM table reset interface |
| 616 | * | 621 | * |
| @@ -635,6 +640,11 @@ static ssize_t amdgpu_ras_debugfs_eeprom_write(struct file *f, · 18 unmodified lines | |||
| 635 | (struct amdgpu_device *)file_inode(f)->i_private; | 640 | (struct amdgpu_device *)file_inode(f)->i_private; |
| 636 | int ret; | 641 | int ret; |
| 637 | 642 | ||
| 643 | if (amdgpu_uniras_enabled(adev)) { | ||
| 644 | ret = amdgpu_uniras_clear_badpages_info(adev); | ||
| 645 | return ret ? ret : size; | ||
| 646 | } | ||
| 647 | |||
| 638 | ret = amdgpu_ras_eeprom_reset_table( | 648 | ret = amdgpu_ras_eeprom_reset_table( |
| 639 | &(amdgpu_ras_get_context(adev)->eeprom_control)); | 649 | &(amdgpu_ras_get_context(adev)->eeprom_control)); |
| 640 | 650 | ||
| @@ -1542,9 +1552,51 @@ out_fini_err_data: · 901 unmodified lines | |||
| 1542 | return ret; | 1552 | return ret; |
| 1543 | } | 1553 | } |
| 1544 | 1554 | ||
| 1555 | static int amdgpu_uniras_clear_badpages_info(struct amdgpu_device *adev) | ||
| 1556 | { | ||
| 1557 | struct ras_cmd_dev_handle req = {0}; | ||
| 1558 | int ret; | ||
| 1559 | |||
| 1560 | ret = amdgpu_ras_mgr_handle_ras_cmd(adev, RAS_CMD__CLEAR_BAD_PAGE_INFO, | ||
| 1561 | &req, sizeof(req), NULL, 0); | ||
| 1562 | if (ret) { | ||
| 1563 | dev_err(adev->dev, "Failed to clear bad pages info, ret: %d\n", ret); | ||
| 1564 | return ret; | ||
| 1565 | } | ||
| 1566 | |||
| 1567 | return 0; | ||
| 1568 | } | ||
| 1569 | |||
| 1570 | static int amdgpu_uniras_query_block_ecc(struct amdgpu_device *adev, | ||
| 1571 | struct ras_query_if *info) | ||
| 1572 | { | ||
| 1573 | struct ras_cmd_block_ecc_info_req req = {0}; | ||
| 1574 | struct ras_cmd_block_ecc_info_rsp rsp = {0}; | ||
| 1575 | int ret; | ||
| 1576 | |||
| 1577 | if (!info) | ||
| 1578 | return -EINVAL; | ||
| 1579 | |||
| 1580 | req.block_id = info->head.block; | ||
| 1581 | req.subblock_id = info->head.sub_block_index; | ||
| 1582 | |||
| 1583 | ret = amdgpu_ras_mgr_handle_ras_cmd(adev, RAS_CMD__GET_BLOCK_ECC_STATUS, | ||
| 1584 | &req, sizeof(req), &rsp, sizeof(rsp)); | ||
| 1585 | if (!ret) { | ||
| 1586 | info->ce_count = rsp.ce_count; | ||
| 1587 | info->ue_count = rsp.ue_count; | ||
| 1588 | info->de_count = rsp.de_count; | ||
| 1589 | } | ||
| 1590 | |||
| 1591 | return ret; | ||
| 1592 | } | ||
| 1593 | |||
| 1545 | int amdgpu_ras_query_error_status(struct amdgpu_device *adev, struct ras_query_if *info) | 1594 | int amdgpu_ras_query_error_status(struct amdgpu_device *adev, struct ras_query_if *info) |
| 1546 | { | 1595 | { |
| 1547 | return amdgpu_ras_query_error_status_with_event(adev, info, RAS_EVENT_TYPE_INVALID); | 1596 | if (amdgpu_uniras_enabled(adev)) |
| 1597 | return amdgpu_uniras_query_block_ecc(adev, info); | ||
| 1598 | else | ||
| 1599 | return amdgpu_ras_query_error_status_with_event(adev, info, RAS_EVENT_TYPE_INVALID); | ||
| 1548 | } | 1600 | } |
| 1549 | 1601 | ||
| 1550 | int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, | 1602 | int amdgpu_ras_reset_error_count(struct amdgpu_device *adev, |
| @@ -1596,6 +1648,27 @@ int amdgpu_ras_reset_error_status(struct amdgpu_device *adev, · 45 unmodified lines | |||
| 1596 | return 0; | 1648 | return 0; |
| 1597 | } | 1649 | } |
| 1598 | 1650 | ||
| 1651 | static int amdgpu_uniras_error_inject(struct amdgpu_device *adev, | ||
| 1652 | struct ras_inject_if *info) | ||
| 1653 | { | ||
| 1654 | struct ras_cmd_inject_error_req inject_req; | ||
| 1655 | struct ras_cmd_inject_error_rsp rsp; | ||
| 1656 | |||
| 1657 | if (!info) | ||
| 1658 | return -EINVAL; | ||
| 1659 | |||
| 1660 | memset(&inject_req, 0, sizeof(inject_req)); | ||
| 1661 | inject_req.block_id = info->head.block; | ||
| 1662 | inject_req.subblock_id = info->head.sub_block_index; | ||
| 1663 | inject_req.address = info->address; | ||
| 1664 | inject_req.error_type = info->head.type; | ||
| 1665 | inject_req.instance_mask = info->instance_mask; | ||
| 1666 | inject_req.method = info->value; | ||
| 1667 | |||
| 1668 | return amdgpu_ras_mgr_handle_ras_cmd(adev, RAS_CMD__INJECT_ERROR, | ||
| 1669 | &inject_req, sizeof(inject_req), &rsp, sizeof(rsp)); | ||
| 1670 | } | ||
| 1671 | |||
| 1599 | /* wrapper of psp_ras_trigger_error */ | 1672 | /* wrapper of psp_ras_trigger_error */ |
| 1600 | int amdgpu_ras_error_inject(struct amdgpu_device *adev, | 1673 | int amdgpu_ras_error_inject(struct amdgpu_device *adev, |
| 1601 | struct ras_inject_if *info) | 1674 | struct ras_inject_if *info) |
| @@ -1613,6 +1686,9 @@ int amdgpu_ras_error_inject(struct amdgpu_device *adev, · 11 unmodified lines | |||
| 1613 | info->head.block, | 1686 | info->head.block, |
| 1614 | info->head.sub_block_index); | 1687 | info->head.sub_block_index); |
| 1615 | 1688 | ||
| 1689 | if (amdgpu_uniras_enabled(adev)) | ||
| 1690 | return amdgpu_uniras_error_inject(adev, info); | ||
| 1691 | |||
| 1616 | /* inject on guest isn't allowed, return success directly */ | 1692 | /* inject on guest isn't allowed, return success directly */ |
| 1617 | if (amdgpu_sriov_vf(adev)) | 1693 | if (amdgpu_sriov_vf(adev)) |
| 1618 | return 0; | 1694 | return 0; |
| @@ -1757,7 +1833,9 @@ int amdgpu_ras_query_error_count(struct amdgpu_device *adev, · 138 unmodified lines | |||
| 1757 | /* sysfs begin */ | 1833 | /* sysfs begin */ |
| 1758 | 1834 | ||
| 1759 | static int amdgpu_ras_badpages_read(struct amdgpu_device *adev, | 1835 | static int amdgpu_ras_badpages_read(struct amdgpu_device *adev, |
| 1760 | struct ras_badpage **bps, unsigned int *count); | 1836 | struct ras_badpage *bps, uint32_t count, uint32_t start); |
| 1837 | static int amdgpu_uniras_badpages_read(struct amdgpu_device *adev, | ||
| 1838 | struct ras_badpage *bps, uint32_t count, uint32_t start); | ||
| 1761 | 1839 | ||
| 1762 | static char *amdgpu_ras_badpage_flags_str(unsigned int flags) | 1840 | static char *amdgpu_ras_badpage_flags_str(unsigned int flags) |
| 1763 | { | 1841 | { |
| @@ -1815,19 +1893,50 @@ static ssize_t amdgpu_ras_sysfs_badpages_read(struct file *f, · 51 unmodified lines | |||
| 1815 | unsigned int end = div64_ul(ppos + count - 1, element_size); | 1893 | unsigned int end = div64_ul(ppos + count - 1, element_size); |
| 1816 | ssize_t s = 0; | 1894 | ssize_t s = 0; |
| 1817 | struct ras_badpage *bps = NULL; | 1895 | struct ras_badpage *bps = NULL; |
| 1818 | unsigned int bps_count = 0; | 1896 | int bps_count = 0, i, status; |
| 1897 | uint64_t address; | ||
| 1819 | 1898 | ||
| 1820 | memset(buf, 0, count); | 1899 | memset(buf, 0, count); |
| 1821 | 1900 | ||
| 1822 | if (amdgpu_ras_badpages_read(adev, &bps, &bps_count)) | 1901 | bps_count = end - start; |
| 1902 | bps = kmalloc_array(bps_count, sizeof(*bps), GFP_KERNEL); | ||
| 1903 | if (!bps) | ||
| 1904 | return 0; | ||
| 1905 | |||
| 1906 | memset(bps, 0, sizeof(*bps) * bps_count); | ||
| 1907 | |||
| 1908 | if (amdgpu_uniras_enabled(adev)) | ||
| 1909 | bps_count = amdgpu_uniras_badpages_read(adev, bps, bps_count, start); | ||
| 1910 | else | ||
| 1911 | bps_count = amdgpu_ras_badpages_read(adev, bps, bps_count, start); | ||
| 1912 | |||
| 1913 | if (bps_count <= 0) { | ||
| 1914 | kfree(bps); | ||
| 1823 | return 0; | 1915 | return 0; |
| 1916 | } | ||
| 1917 | |||
| 1918 | for (i = 0; i < bps_count; i++) { | ||
| 1919 | address = ((uint64_t)bps[i].bp) << AMDGPU_GPU_PAGE_SHIFT; | ||
| 1920 | if (amdgpu_ras_check_critical_address(adev, address)) | ||
| 1921 | continue; | ||
| 1922 | |||
| 1923 | bps[i].size = AMDGPU_GPU_PAGE_SIZE; | ||
| 1924 | |||
| 1925 | status = amdgpu_vram_mgr_query_page_status(&adev->mman.vram_mgr, | ||
| 1926 | address); | ||
| 1927 | if (status == -EBUSY) | ||
| 1928 | bps[i].flags = AMDGPU_RAS_RETIRE_PAGE_PENDING; | ||
| 1929 | else if (status == -ENOENT) | ||
| 1930 | bps[i].flags = AMDGPU_RAS_RETIRE_PAGE_FAULT; | ||
| 1931 | else | ||
| 1932 | bps[i].flags = AMDGPU_RAS_RETIRE_PAGE_RESERVED; | ||
| 1824 | 1933 | ||
| 1825 | for (; start < end && start < bps_count; start++) | ||
| 1826 | s += scnprintf(&buf[s], element_size + 1, | 1934 | s += scnprintf(&buf[s], element_size + 1, |
| 1827 | "0x%08x : 0x%08x : %1s\n", | 1935 | "0x%08x : 0x%08x : %1s\n", |
| 1828 | bps[start].bp, | 1936 | bps[i].bp, |
| 1829 | bps[start].size, | 1937 | bps[i].size, |
| 1830 | amdgpu_ras_badpage_flags_str(bps[start].flags)); | 1938 | amdgpu_ras_badpage_flags_str(bps[i].flags)); |
| 1939 | } | ||
| 1831 | 1940 | ||
| 1832 | kfree(bps); | 1941 | kfree(bps); |
| 1833 | 1942 | ||
| @@ -1843,12 +1952,42 @@ static ssize_t amdgpu_ras_sysfs_features_read(struct device *dev, · 9 unmodified lines | |||
| 1843 | return sysfs_emit(buf, "feature mask: 0x%x\n", con->features); | 1952 | return sysfs_emit(buf, "feature mask: 0x%x\n", con->features); |
| 1844 | } | 1953 | } |
| 1845 | 1954 | ||
| 1955 | static bool amdgpu_ras_get_version_info(struct amdgpu_device *adev, u32 *major, | ||
| 1956 | u32 *minor, u32 *rev) | ||
| 1957 | { | ||
| 1958 | int i; | ||
| 1959 | |||
| 1960 | if (!adev || !major || !minor || !rev || !amdgpu_uniras_enabled(adev)) | ||
| 1961 | return false; | ||
| 1962 | |||
| 1963 | for (i = 0; i < adev->num_ip_blocks; i++) { | ||
| 1964 | if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_RAS) { | ||
| 1965 | *major = adev->ip_blocks[i].version->major; | ||
| 1966 | *minor = adev->ip_blocks[i].version->minor; | ||
| 1967 | *rev = adev->ip_blocks[i].version->rev; | ||
| 1968 | return true; | ||
| 1969 | } | ||
| 1970 | } | ||
| 1971 | |||
| 1972 | return false; | ||
| 1973 | } | ||
| 1974 | |||
| 1846 | static ssize_t amdgpu_ras_sysfs_version_show(struct device *dev, | 1975 | static ssize_t amdgpu_ras_sysfs_version_show(struct device *dev, |
| 1847 | struct device_attribute *attr, char *buf) | 1976 | struct device_attribute *attr, char *buf) |
| 1848 | { | 1977 | { |
| 1849 | struct amdgpu_ras *con = | 1978 | struct amdgpu_ras *con = |
| 1850 | container_of(attr, struct amdgpu_ras, version_attr); | 1979 | container_of(attr, struct amdgpu_ras, version_attr); |
| 1851 | return sysfs_emit(buf, "table version: 0x%x\n", con->eeprom_control.tbl_hdr.version); | 1980 | u32 major, minor, rev; |
| 1981 | ssize_t size = 0; | ||
| 1982 | |||
| 1983 | size += sysfs_emit_at(buf, size, "table version: 0x%x\n", | ||
| 1984 | con->eeprom_control.tbl_hdr.version); | ||
| 1985 | |||
| 1986 | if (amdgpu_ras_get_version_info(con->adev, &major, &minor, &rev)) | ||
| 1987 | size += sysfs_emit_at(buf, size, "ras version: %u.%u.%u\n", | ||
| 1988 | major, minor, rev); | ||
| 1989 | |||
| 1990 | return size; | ||
| 1852 | } | 1991 | } |
| 1853 | 1992 | ||
| 1854 | static ssize_t amdgpu_ras_sysfs_schema_show(struct device *dev, | 1993 | static ssize_t amdgpu_ras_sysfs_schema_show(struct device *dev, |
| @@ -2241,6 +2380,11 @@ void amdgpu_ras_interrupt_fatal_error_handler(struct amdgpu_device *adev) · 386 unmodified lines | |||
| 2241 | amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY)) | 2380 | amdgpu_ras_is_err_state(adev, AMDGPU_RAS_BLOCK__ANY)) |
| 2242 | return; | 2381 | return; |
| 2243 | 2382 | ||
| 2383 | if (amdgpu_uniras_enabled(adev)) { | ||
| 2384 | amdgpu_ras_mgr_handle_fatal_interrupt(adev, NULL); | ||
| 2385 | return; | ||
| 2386 | } | ||
| 2387 | |||
| 2244 | if (adev->nbio.ras && | 2388 | if (adev->nbio.ras && |
| 2245 | adev->nbio.ras->handle_ras_controller_intr_no_bifring) | 2389 | adev->nbio.ras->handle_ras_controller_intr_no_bifring) |
| 2246 | adev->nbio.ras->handle_ras_controller_intr_no_bifring(adev); | 2390 | adev->nbio.ras->handle_ras_controller_intr_no_bifring(adev); |
| @@ -2411,6 +2555,16 @@ int amdgpu_ras_interrupt_dispatch(struct amdgpu_device *adev, · 164 unmodified lines | |||
| 2411 | struct ras_manager *obj; | 2555 | struct ras_manager *obj; |
| 2412 | struct ras_ih_data *data; | 2556 | struct ras_ih_data *data; |
| 2413 | 2557 | ||
| 2558 | if (amdgpu_uniras_enabled(adev)) { | ||
| 2559 | struct ras_ih_info ih_info; | ||
| 2560 | |||
| 2561 | memset(&ih_info, 0, sizeof(ih_info)); | ||
| 2562 | ih_info.block = info->head.block; | ||
| 2563 | memcpy(&ih_info.iv_entry, info->entry, sizeof(struct amdgpu_iv_entry)); | ||
| 2564 | |||
| 2565 | return amdgpu_ras_mgr_handle_controller_interrupt(adev, &ih_info); | ||
| 2566 | } | ||
| 2567 | |||
| 2414 | obj = amdgpu_ras_find_obj(adev, &info->head); | 2568 | obj = amdgpu_ras_find_obj(adev, &info->head); |
| 2415 | if (!obj) | 2569 | if (!obj) |
| 2416 | return -EINVAL; | 2570 | return -EINVAL; |
| @@ -2605,62 +2759,83 @@ static void amdgpu_ras_query_err_status(struct amdgpu_device *adev) · 188 unmodified lines | |||
| 2605 | } | 2759 | } |
| 2606 | } | 2760 | } |
| 2607 | 2761 | ||
| 2608 | /* recovery begin */ | ||
| 2609 | |||
| 2610 | /* return 0 on success. | ||
| 2611 | * caller need free bps. | ||
| 2612 | */ | ||
| 2613 | static int amdgpu_ras_badpages_read(struct amdgpu_device *adev, | 2762 | static int amdgpu_ras_badpages_read(struct amdgpu_device *adev, |
| 2614 | struct ras_badpage **bps, unsigned int *count) | 2763 | struct ras_badpage *bps, uint32_t count, uint32_t start) |
| 2615 | { | 2764 | { |
| 2616 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); | 2765 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); |
| 2617 | struct ras_err_handler_data *data; | 2766 | struct ras_err_handler_data *data; |
| 2618 | int i = 0; | 2767 | int r = 0; |
| 2619 | int ret = 0, status; | 2768 | uint32_t i; |
| 2620 | 2769 | ||
| 2621 | if (!con || !con->eh_data || !bps || !count) | 2770 | if (!con || !con->eh_data || !bps || !count) |
| 2622 | return -EINVAL; | 2771 | return -EINVAL; |
| 2623 | 2772 | ||
| 2624 | mutex_lock(&con->recovery_lock); | 2773 | mutex_lock(&con->recovery_lock); |
| 2625 | data = con->eh_data; | 2774 | data = con->eh_data; |
| 2626 | if (!data || data->count == 0) { | 2775 | if (start < data->count) { |
| 2627 | *bps = NULL; | 2776 | for (i = start; i < data->count; i++) { |
| 2628 | ret = -EINVAL; | 2777 | if (!data->bps[i].ts) |
| 2629 | goto out; | 2778 | continue; |
| 2779 | |||
| 2780 | bps[r].bp = data->bps[i].retired_page; | ||
| 2781 | r++; | ||
| 2782 | if (r >= count) | ||
| 2783 | break; | ||
| 2784 | } | ||
| 2630 | } | 2785 | } |
| 2786 | mutex_unlock(&con->recovery_lock); | ||
| 2631 | 2787 | ||
| 2632 | *bps = kmalloc_array(data->count, sizeof(struct ras_badpage), GFP_KERNEL); | 2788 | return r; |
| 2633 | if (!*bps) { | 2789 | } |
| 2634 | ret = -ENOMEM; | ||
| 2635 | goto out; | ||
| 2636 | } | ||
| 2637 | 2790 | ||
| 2638 | for (; i < data->count; i++) { | 2791 | static int amdgpu_uniras_badpages_read(struct amdgpu_device *adev, |
| 2639 | if (!data->bps[i].ts) | 2792 | struct ras_badpage *bps, uint32_t count, uint32_t start) |
| 2640 | continue; | 2793 | { |
| 2794 | struct ras_cmd_bad_pages_info_req cmd_input; | ||
| 2795 | struct ras_cmd_bad_pages_info_rsp *output; | ||
| 2796 | uint32_t group, start_group, end_group; | ||
| 2797 | uint32_t pos, pos_in_group; | ||
| 2798 | int r = 0, i; | ||
| 2641 | 2799 | ||
| 2642 | (*bps)[i] = (struct ras_badpage){ | 2800 | if (!bps || !count) |
| 2643 | .bp = data->bps[i].retired_page, | 2801 | return -EINVAL; |
| 2644 | .size = AMDGPU_GPU_PAGE_SIZE, | ||
| 2645 | .flags = AMDGPU_RAS_RETIRE_PAGE_RESERVED, | ||
| 2646 | }; | ||
| 2647 | 2802 | ||
| 2648 | if (amdgpu_ras_check_critical_address(adev, | 2803 | output = kmalloc(sizeof(*output), GFP_KERNEL); |
| 2649 | data->bps[i].retired_page << AMDGPU_GPU_PAGE_SHIFT)) | 2804 | if (!output) |
| 2650 | continue; | 2805 | return -ENOMEM; |
| 2651 | 2806 | ||
| 2652 | status = amdgpu_vram_mgr_query_page_status(&adev->mman.vram_mgr, | 2807 | memset(&cmd_input, 0, sizeof(cmd_input)); |
| 2653 | data->bps[i].retired_page << AMDGPU_GPU_PAGE_SHIFT); | 2808 | |
| 2654 | if (status == -EBUSY) | 2809 | start_group = start / RAS_CMD_MAX_BAD_PAGES_PER_GROUP; |
| 2655 | (*bps)[i].flags = AMDGPU_RAS_RETIRE_PAGE_PENDING; | 2810 | end_group = (start + count + RAS_CMD_MAX_BAD_PAGES_PER_GROUP - 1) / |
| 2656 | else if (status == -ENOENT) | 2811 | RAS_CMD_MAX_BAD_PAGES_PER_GROUP; |
| 2657 | (*bps)[i].flags = AMDGPU_RAS_RETIRE_PAGE_FAULT; | 2812 | |
| 2813 | pos = start; | ||
| 2814 | for (group = start_group; group < end_group; group++) { | ||
| 2815 | memset(output, 0, sizeof(*output)); | ||
| 2816 | cmd_input.group_index = group; | ||
| 2817 | if (amdgpu_ras_mgr_handle_ras_cmd(adev, RAS_CMD__GET_BAD_PAGES, | ||
| 2818 | &cmd_input, sizeof(cmd_input), output, sizeof(*output))) | ||
| 2819 | goto out; | ||
| 2820 | |||
| 2821 | if (pos >= output->bp_total_cnt) | ||
| 2822 | goto out; | ||
| 2823 | |||
| 2824 | pos_in_group = pos - group * RAS_CMD_MAX_BAD_PAGES_PER_GROUP; | ||
| 2825 | for (i = pos_in_group; i < output->bp_in_group; i++, pos++) { | ||
| 2826 | if (!output->records[i].ts) | ||
| 2827 | continue; | ||
| 2828 | |||
| 2829 | bps[r].bp = output->records[i].retired_page; | ||
| 2830 | r++; | ||
| 2831 | if (r >= count) | ||
| 2832 | goto out; | ||
| 2833 | } | ||
| 2658 | } | 2834 | } |
| 2659 | 2835 | ||
| 2660 | *count = con->bad_page_num; | ||
| 2661 | out: | 2836 | out: |
| 2662 | mutex_unlock(&con->recovery_lock); | 2837 | kfree(output); |
| 2663 | return ret; | 2838 | return r; |
| 2664 | } | 2839 | } |
| 2665 | 2840 | ||
| 2666 | static void amdgpu_ras_set_fed_all(struct amdgpu_device *adev, | 2841 | static void amdgpu_ras_set_fed_all(struct amdgpu_device *adev, |
| @@ -2748,8 +2923,12 @@ static void amdgpu_ras_do_recovery(struct work_struct *work) · 81 unmodified lines | |||
| 2748 | type = amdgpu_ras_get_fatal_error_event(adev); | 2923 | type = amdgpu_ras_get_fatal_error_event(adev); |
| 2749 | list_for_each_entry(remote_adev, | 2924 | list_for_each_entry(remote_adev, |
| 2750 | device_list_handle, gmc.xgmi.head) { | 2925 | device_list_handle, gmc.xgmi.head) { |
| 2751 | amdgpu_ras_query_err_status(remote_adev); | 2926 | if (amdgpu_uniras_enabled(remote_adev)) { |
| 2752 | amdgpu_ras_log_on_err_counter(remote_adev, type); | 2927 | amdgpu_ras_mgr_update_ras_ecc(remote_adev); |
| 2928 | } else { | ||
| 2929 | amdgpu_ras_query_err_status(remote_adev); | ||
| 2930 | amdgpu_ras_log_on_err_counter(remote_adev, type); | ||
| 2931 | } | ||
| 2753 | } | 2932 | } |
| 2754 | 2933 | ||
| 2755 | } | 2934 | } |
| @@ -2837,8 +3016,13 @@ static int amdgpu_ras_mca2pa_by_idx(struct amdgpu_device *adev, · 81 unmodified lines | |||
| 2837 | addr_in.ma.err_addr = bps->address; | 3016 | addr_in.ma.err_addr = bps->address; |
| 2838 | addr_in.ma.socket_id = socket; | 3017 | addr_in.ma.socket_id = socket; |
| 2839 | addr_in.ma.ch_inst = bps->mem_channel; | 3018 | addr_in.ma.ch_inst = bps->mem_channel; |
| 2840 | /* tell RAS TA the node instance is not used */ | 3019 | if (!amdgpu_ras_smu_eeprom_supported(adev)) { |
| 2841 | addr_in.ma.node_inst = TA_RAS_INV_NODE; | 3020 | /* tell RAS TA the node instance is not used */ |
| 3021 | addr_in.ma.node_inst = TA_RAS_INV_NODE; | ||
| 3022 | } else { | ||
| 3023 | addr_in.ma.umc_inst = bps->mcumc_id; | ||
| 3024 | addr_in.ma.node_inst = bps->cu; | ||
| 3025 | } | ||
| 2842 | 3026 | ||
| 2843 | if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr) | 3027 | if (adev->umc.ras && adev->umc.ras->convert_ras_err_addr) |
| 2844 | ret = adev->umc.ras->convert_ras_err_addr(adev, err_data, | 3028 | ret = adev->umc.ras->convert_ras_err_addr(adev, err_data, |
| @@ -2981,8 +3165,16 @@ static int __amdgpu_ras_convert_rec_from_rom(struct amdgpu_device *adev, · 136 unmodified lines | |||
| 2981 | int i = 0; | 3165 | int i = 0; |
| 2982 | enum amdgpu_memory_partition save_nps; | 3166 | enum amdgpu_memory_partition save_nps; |
| 2983 | 3167 | ||
| 2984 | save_nps = (bps->retired_page >> UMC_NPS_SHIFT) & UMC_NPS_MASK; | 3168 | if (!amdgpu_ras_smu_eeprom_supported(adev)) { |
| 2985 | bps->retired_page &= ~(UMC_NPS_MASK << UMC_NPS_SHIFT); | 3169 | save_nps = (bps->retired_page >> UMC_NPS_SHIFT) & UMC_NPS_MASK; |
| 3170 | bps->retired_page &= ~(UMC_NPS_MASK << UMC_NPS_SHIFT); | ||
| 3171 | } else { | ||
| 3172 | /* if pmfw manages eeprom, save_nps is not stored on eeprom, | ||
| 3173 | * we should always convert mca address into physical address, | ||
| 3174 | * make save_nps different from nps | ||
| 3175 | */ | ||
| 3176 | save_nps = nps + 1; | ||
| 3177 | } | ||
| 2986 | 3178 | ||
| 2987 | if (save_nps == nps) { | 3179 | if (save_nps == nps) { |
| 2988 | if (amdgpu_umc_pages_in_a_row(adev, err_data, | 3180 | if (amdgpu_umc_pages_in_a_row(adev, err_data, |
| @@ -3048,7 +3240,8 @@ int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev, · 59 unmodified lines | |||
| 3048 | 3240 | ||
| 3049 | if (from_rom) { | 3241 | if (from_rom) { |
| 3050 | /* there is no pa recs in V3, so skip pa recs processing */ | 3242 | /* there is no pa recs in V3, so skip pa recs processing */ |
| 3051 | if (control->tbl_hdr.version < RAS_TABLE_VER_V3) { | 3243 | if ((control->tbl_hdr.version < RAS_TABLE_VER_V3) && |
| 3244 | !amdgpu_ras_smu_eeprom_supported(adev)) { | ||
| 3052 | for (i = 0; i < pages; i++) { | 3245 | for (i = 0; i < pages; i++) { |
| 3053 | if (control->ras_num_recs - i >= adev->umc.retire_unit) { | 3246 | if (control->ras_num_recs - i >= adev->umc.retire_unit) { |
| 3054 | if ((bps[i].address == bps[i + 1].address) && | 3247 | if ((bps[i].address == bps[i + 1].address) && |
| @@ -3118,7 +3311,13 @@ int amdgpu_ras_save_bad_pages(struct amdgpu_device *adev, · 63 unmodified lines | |||
| 3118 | mutex_lock(&con->recovery_lock); | 3311 | mutex_lock(&con->recovery_lock); |
| 3119 | control = &con->eeprom_control; | 3312 | control = &con->eeprom_control; |
| 3120 | data = con->eh_data; | 3313 | data = con->eh_data; |
| 3121 | unit_num = data->count / adev->umc.retire_unit - control->ras_num_recs; | 3314 | if (amdgpu_ras_smu_eeprom_supported(adev)) |
| 3315 | unit_num = control->ras_num_recs - | ||
| 3316 | control->ras_num_recs_old; | ||
| 3317 | else | ||
| 3318 | unit_num = data->count / adev->umc.retire_unit - | ||
| 3319 | control->ras_num_recs; | ||
| 3320 | |||
| 3122 | save_count = con->bad_page_num - control->ras_num_bad_pages; | 3321 | save_count = con->bad_page_num - control->ras_num_bad_pages; |
| 3123 | mutex_unlock(&con->recovery_lock); | 3322 | mutex_unlock(&con->recovery_lock); |
| 3124 | 3323 | ||
| @@ -3126,7 +3325,7 @@ int amdgpu_ras_save_bad_pages(struct amdgpu_device *adev, · 1 unmodified line | |||
| 3126 | *new_cnt = unit_num; | 3325 | *new_cnt = unit_num; |
| 3127 | 3326 | ||
| 3128 | /* only new entries are saved */ | 3327 | /* only new entries are saved */ |
| 3129 | if (unit_num > 0) { | 3328 | if (unit_num && save_count) { |
| 3130 | /*old asics only save pa to eeprom like before*/ | 3329 | /*old asics only save pa to eeprom like before*/ |
| 3131 | if (IP_VERSION_MAJ(amdgpu_ip_version(adev, UMC_HWIP, 0)) < 12) { | 3330 | if (IP_VERSION_MAJ(amdgpu_ip_version(adev, UMC_HWIP, 0)) < 12) { |
| 3132 | if (amdgpu_ras_eeprom_append(control, | 3331 | if (amdgpu_ras_eeprom_append(control, |
| @@ -3179,7 +3378,8 @@ static int amdgpu_ras_load_bad_pages(struct amdgpu_device *adev) · 46 unmodified lines | |||
| 3179 | /*In V3, there is no pa recs, and some cases(when address==0) may be parsed | 3378 | /*In V3, there is no pa recs, and some cases(when address==0) may be parsed |
| 3180 | as pa recs, so add verion check to avoid it. | 3379 | as pa recs, so add verion check to avoid it. |
| 3181 | */ | 3380 | */ |
| 3182 | if (control->tbl_hdr.version < RAS_TABLE_VER_V3) { | 3381 | if ((control->tbl_hdr.version < RAS_TABLE_VER_V3) && |
| 3382 | !amdgpu_ras_smu_eeprom_supported(adev)) { | ||
| 3183 | for (i = 0; i < control->ras_num_recs; i++) { | 3383 | for (i = 0; i < control->ras_num_recs; i++) { |
| 3184 | if ((control->ras_num_recs - i) >= adev->umc.retire_unit) { | 3384 | if ((control->ras_num_recs - i) >= adev->umc.retire_unit) { |
| 3185 | if ((bps[i].address == bps[i + 1].address) && | 3385 | if ((bps[i].address == bps[i + 1].address) && |
| @@ -3590,7 +3790,12 @@ int amdgpu_ras_init_badpage_info(struct amdgpu_device *adev) · 404 unmodified lines | |||
| 3590 | if (!con || amdgpu_sriov_vf(adev)) | 3790 | if (!con || amdgpu_sriov_vf(adev)) |
| 3591 | return 0; | 3791 | return 0; |
| 3592 | 3792 | ||
| 3793 | if (amdgpu_uniras_enabled(adev)) | ||
| 3794 | return 0; | ||
| 3795 | |||
| 3593 | control = &con->eeprom_control; | 3796 | control = &con->eeprom_control; |
| 3797 | con->ras_smu_drv = amdgpu_dpm_get_ras_smu_driver(adev); | ||
| 3798 | |||
| 3594 | ret = amdgpu_ras_eeprom_init(control); | 3799 | ret = amdgpu_ras_eeprom_init(control); |
| 3595 | control->is_eeprom_valid = !ret; | 3800 | control->is_eeprom_valid = !ret; |
| 3596 | 3801 | ||
| @@ -3751,7 +3956,9 @@ static int amdgpu_ras_recovery_fini(struct amdgpu_device *adev) · 154 unmodified lines | |||
| 3751 | mutex_unlock(&con->recovery_lock); | 3956 | mutex_unlock(&con->recovery_lock); |
| 3752 | 3957 | ||
| 3753 | amdgpu_ras_critical_region_init(adev); | 3958 | amdgpu_ras_critical_region_init(adev); |
| 3754 | 3959 | #ifdef CONFIG_X86_MCE_AMD | |
| 3960 | amdgpu_unregister_bad_pages_mca_notifier(adev); | ||
| 3961 | #endif | ||
| 3755 | return 0; | 3962 | return 0; |
| 3756 | } | 3963 | } |
| 3757 | /* recovery end */ | 3964 | /* recovery end */ |
| @@ -3975,7 +4182,6 @@ static void amdgpu_ras_counte_dw(struct work_struct *work) · 217 unmodified lines | |||
| 3975 | atomic_set(&con->ras_ue_count, ue_count); | 4182 | atomic_set(&con->ras_ue_count, ue_count); |
| 3976 | } | 4183 | } |
| 3977 | 4184 | ||
| 3978 | pm_runtime_mark_last_busy(dev->dev); | ||
| 3979 | Out: | 4185 | Out: |
| 3980 | pm_runtime_put_autosuspend(dev->dev); | 4186 | pm_runtime_put_autosuspend(dev->dev); |
| 3981 | } | 4187 | } |
| @@ -4584,6 +4790,9 @@ int amdgpu_ras_mark_ras_event_caller(struct amdgpu_device *adev, enum ras_event_ · 602 unmodified lines | |||
| 4584 | struct ras_event_state *event_state; | 4790 | struct ras_event_state *event_state; |
| 4585 | int ret = 0; | 4791 | int ret = 0; |
| 4586 | 4792 | ||
| 4793 | if (amdgpu_uniras_enabled(adev)) | ||
| 4794 | return 0; | ||
| 4795 | |||
| 4587 | if (type >= RAS_EVENT_TYPE_COUNT) { | 4796 | if (type >= RAS_EVENT_TYPE_COUNT) { |
| 4588 | ret = -EINVAL; | 4797 | ret = -EINVAL; |
| 4589 | goto out; | 4798 | goto out; |
| @@ -4634,20 +4843,18 @@ u64 amdgpu_ras_acquire_event_id(struct amdgpu_device *adev, enum ras_event_type · 44 unmodified lines | |||
| 4634 | return id; | 4843 | return id; |
| 4635 | } | 4844 | } |
| 4636 | 4845 | ||
| 4637 | void amdgpu_ras_global_ras_isr(struct amdgpu_device *adev) | 4846 | int amdgpu_ras_global_ras_isr(struct amdgpu_device *adev) |
| 4638 | { | 4847 | { |
| 4639 | if (atomic_cmpxchg(&amdgpu_ras_in_intr, 0, 1) == 0) { | 4848 | if (atomic_cmpxchg(&amdgpu_ras_in_intr, 0, 1) == 0) { |
| 4640 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | 4849 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); |
| 4641 | enum ras_event_type type = RAS_EVENT_TYPE_FATAL; | 4850 | enum ras_event_type type = RAS_EVENT_TYPE_FATAL; |
| 4642 | u64 event_id; | 4851 | u64 event_id = RAS_EVENT_INVALID_ID; |
| 4643 | 4852 | ||
| 4644 | if (amdgpu_ras_mark_ras_event(adev, type)) { | 4853 | if (amdgpu_uniras_enabled(adev)) |
| 4645 | dev_err(adev->dev, | 4854 | return 0; |
| 4646 | "uncorrectable hardware error (ERREVENT_ATHUB_INTERRUPT) detected!\n"); | ||
| 4647 | return; | ||
| 4648 | } | ||
| 4649 | 4855 | ||
| 4650 | event_id = amdgpu_ras_acquire_event_id(adev, type); | 4856 | if (!amdgpu_ras_mark_ras_event(adev, type)) |
| 4857 | event_id = amdgpu_ras_acquire_event_id(adev, type); | ||
| 4651 | 4858 | ||
| 4652 | RAS_EVENT_LOG(adev, event_id, "uncorrectable hardware error" | 4859 | RAS_EVENT_LOG(adev, event_id, "uncorrectable hardware error" |
| 4653 | "(ERREVENT_ATHUB_INTERRUPT) detected!\n"); | 4860 | "(ERREVENT_ATHUB_INTERRUPT) detected!\n"); |
| @@ -4656,6 +4863,8 @@ void amdgpu_ras_global_ras_isr(struct amdgpu_device *adev) · 2 unmodified lines | |||
| 4656 | ras->gpu_reset_flags |= AMDGPU_RAS_GPU_RESET_MODE1_RESET; | 4863 | ras->gpu_reset_flags |= AMDGPU_RAS_GPU_RESET_MODE1_RESET; |
| 4657 | amdgpu_ras_reset_gpu(adev); | 4864 | amdgpu_ras_reset_gpu(adev); |
| 4658 | } | 4865 | } |
| 4866 | |||
| 4867 | return -EBUSY; | ||
| 4659 | } | 4868 | } |
| 4660 | 4869 | ||
| 4661 | bool amdgpu_ras_need_emergency_restart(struct amdgpu_device *adev) | 4870 | bool amdgpu_ras_need_emergency_restart(struct amdgpu_device *adev) |
| @@ -4783,6 +4992,28 @@ static void amdgpu_register_bad_pages_mca_notifier(struct amdgpu_device *adev) · 121 unmodified lines | |||
| 4783 | notifier_registered = true; | 4992 | notifier_registered = true; |
| 4784 | } | 4993 | } |
| 4785 | } | 4994 | } |
| 4995 | static void amdgpu_unregister_bad_pages_mca_notifier(struct amdgpu_device *adev) | ||
| 4996 | { | ||
| 4997 | int i, j; | ||
| 4998 | |||
| 4999 | if (!notifier_registered && !mce_adev_list.num_gpu) | ||
| 5000 | return; | ||
| 5001 | for (i = 0, j = 0; i < mce_adev_list.num_gpu; i++) { | ||
| 5002 | if (mce_adev_list.devs[i] == adev) | ||
| 5003 | mce_adev_list.devs[i] = NULL; | ||
| 5004 | if (!mce_adev_list.devs[i]) | ||
| 5005 | ++j; | ||
| 5006 | } | ||
| 5007 | |||
| 5008 | if (j == mce_adev_list.num_gpu) { | ||
| 5009 | mce_adev_list.num_gpu = 0; | ||
| 5010 | /* Unregister x86 notifier with MCE subsystem. */ | ||
| 5011 | if (notifier_registered) { | ||
| 5012 | mce_unregister_decode_chain(&amdgpu_bad_page_nb); | ||
| 5013 | notifier_registered = false; | ||
| 5014 | } | ||
| 5015 | } | ||
| 5016 | } | ||
| 4786 | #endif | 5017 | #endif |
| 4787 | 5018 | ||
| 4788 | struct amdgpu_ras *amdgpu_ras_get_context(struct amdgpu_device *adev) | 5019 | struct amdgpu_ras *amdgpu_ras_get_context(struct amdgpu_device *adev) |
| @@ -5408,6 +5639,9 @@ bool amdgpu_ras_is_rma(struct amdgpu_device *adev) · 619 unmodified lines | |||
| 5408 | { | 5639 | { |
| 5409 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); | 5640 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); |
| 5410 | 5641 | ||
| 5642 | if (amdgpu_uniras_enabled(adev)) | ||
| 5643 | return amdgpu_ras_mgr_is_rma(adev); | ||
| 5644 | |||
| 5411 | if (!con) | 5645 | if (!con) |
| 5412 | return false; | 5646 | return false; |
| 5413 | 5647 | ||
| @@ -5490,3 +5724,25 @@ bool amdgpu_ras_check_critical_address(struct amdgpu_device *adev, uint64_t addr · 76 unmodified lines | |||
| 5490 | 5724 | ||
| 5491 | return ret; | 5725 | return ret; |
| 5492 | } | 5726 | } |
| 5727 | |||
| 5728 | void amdgpu_ras_pre_reset(struct amdgpu_device *adev, | ||
| 5729 | struct list_head *device_list) | ||
| 5730 | { | ||
| 5731 | struct amdgpu_device *tmp_adev = NULL; | ||
| 5732 | |||
| 5733 | list_for_each_entry(tmp_adev, device_list, reset_list) { | ||
| 5734 | if (amdgpu_uniras_enabled(tmp_adev)) | ||
| 5735 | amdgpu_ras_mgr_pre_reset(tmp_adev); | ||
| 5736 | } | ||
| 5737 | } | ||
| 5738 | |||
| 5739 | void amdgpu_ras_post_reset(struct amdgpu_device *adev, | ||
| 5740 | struct list_head *device_list) | ||
| 5741 | { | ||
| 5742 | struct amdgpu_device *tmp_adev = NULL; | ||
| 5743 | |||
| 5744 | list_for_each_entry(tmp_adev, device_list, reset_list) { | ||
| 5745 | if (amdgpu_uniras_enabled(tmp_adev)) | ||
| 5746 | amdgpu_ras_mgr_post_reset(tmp_adev); | ||
| 5747 | } | ||
| 5748 | } | ||
| @@ -503,7 +503,34 @@ struct ras_critical_region { · 502 unmodified lines | |||
| 503 | uint64_t size; | 503 | uint64_t size; |
| 504 | }; | 504 | }; |
| 505 | 505 | ||
| 506 | struct ras_eeprom_table_version { | ||
| 507 | uint32_t minor : 16; | ||
| 508 | uint32_t major : 16; | ||
| 509 | }; | ||
| 510 | |||
| 511 | struct ras_eeprom_smu_funcs { | ||
| 512 | int (*get_ras_table_version)(struct amdgpu_device *adev, | ||
| 513 | uint32_t *table_version); | ||
| 514 | int (*get_badpage_count)(struct amdgpu_device *adev, uint32_t *count, uint32_t timeout); | ||
| 515 | int (*get_badpage_mca_addr)(struct amdgpu_device *adev, uint16_t index, uint64_t *mca_addr); | ||
| 516 | int (*set_timestamp)(struct amdgpu_device *adev, uint64_t timestamp); | ||
| 517 | int (*get_timestamp)(struct amdgpu_device *adev, | ||
| 518 | uint16_t index, uint64_t *timestamp); | ||
| 519 | int (*get_badpage_ipid)(struct amdgpu_device *adev, uint16_t index, uint64_t *ipid); | ||
| 520 | int (*erase_ras_table)(struct amdgpu_device *adev, uint32_t *result); | ||
| 521 | }; | ||
| 522 | |||
| 523 | enum ras_smu_feature_flags { | ||
| 524 | RAS_SMU_FEATURE_BIT__RAS_EEPROM = BIT_ULL(0), | ||
| 525 | }; | ||
| 526 | |||
| 527 | struct ras_smu_drv { | ||
| 528 | const struct ras_eeprom_smu_funcs *smu_eeprom_funcs; | ||
| 529 | void (*ras_smu_feature_flags)(struct amdgpu_device *adev, uint64_t *flags); | ||
| 530 | }; | ||
| 531 | |||
| 506 | struct amdgpu_ras { | 532 | struct amdgpu_ras { |
| 533 | void *ras_mgr; | ||
| 507 | /* ras infrastructure */ | 534 | /* ras infrastructure */ |
| 508 | /* for ras itself. */ | 535 | /* for ras itself. */ |
| 509 | uint32_t features; | 536 | uint32_t features; |
| @@ -590,6 +617,10 @@ struct amdgpu_ras { · 80 unmodified lines | |||
| 590 | 617 | ||
| 591 | /* Protect poison injection */ | 618 | /* Protect poison injection */ |
| 592 | struct mutex poison_lock; | 619 | struct mutex poison_lock; |
| 620 | |||
| 621 | /* Disable/Enable uniras switch */ | ||
| 622 | bool uniras_enabled; | ||
| 623 | const struct ras_smu_drv *ras_smu_drv; | ||
| 593 | }; | 624 | }; |
| 594 | 625 | ||
| 595 | struct ras_fs_data { | 626 | struct ras_fs_data { |
| @@ -909,7 +940,7 @@ static inline void amdgpu_ras_intr_cleared(void) · 313 unmodified lines | |||
| 909 | atomic_set(&amdgpu_ras_in_intr, 0); | 940 | atomic_set(&amdgpu_ras_in_intr, 0); |
| 910 | } | 941 | } |
| 911 | 942 | ||
| 912 | void amdgpu_ras_global_ras_isr(struct amdgpu_device *adev); | 943 | int amdgpu_ras_global_ras_isr(struct amdgpu_device *adev); |
| 913 | 944 | ||
| 914 | void amdgpu_ras_set_error_query_ready(struct amdgpu_device *adev, bool ready); | 945 | void amdgpu_ras_set_error_query_ready(struct amdgpu_device *adev, bool ready); |
| 915 | 946 | ||
| @@ -1008,4 +1039,9 @@ void amdgpu_ras_event_log_print(struct amdgpu_device *adev, u64 event_id, · 92 unmodified lines | |||
| 1008 | const char *fmt, ...); | 1039 | const char *fmt, ...); |
| 1009 | 1040 | ||
| 1010 | bool amdgpu_ras_is_rma(struct amdgpu_device *adev); | 1041 | bool amdgpu_ras_is_rma(struct amdgpu_device *adev); |
| 1042 | |||
| 1043 | void amdgpu_ras_pre_reset(struct amdgpu_device *adev, | ||
| 1044 | struct list_head *device_list); | ||
| 1045 | void amdgpu_ras_post_reset(struct amdgpu_device *adev, | ||
| 1046 | struct list_head *device_list); | ||
| 1011 | #endif | 1047 | #endif |
| @@ -32,6 +32,7 @@ · 31 unmodified lines | |||
| 32 | #include <linux/uaccess.h> | 32 | #include <linux/uaccess.h> |
| 33 | 33 | ||
| 34 | #include "amdgpu_reset.h" | 34 | #include "amdgpu_reset.h" |
| 35 | #include "amdgpu_ras_mgr.h" | ||
| 35 | 36 | ||
| 36 | /* These are memory addresses as would be seen by one or more EEPROM | 37 | /* These are memory addresses as would be seen by one or more EEPROM |
| 37 | * chips strung on the I2C bus, usually by manipulating pins 1-3 of a | 38 | * chips strung on the I2C bus, usually by manipulating pins 1-3 of a |
| @@ -123,6 +124,8 @@ · 85 unmodified lines | |||
| 123 | RAS_TABLE_V2_1_INFO_SIZE) \ | 124 | RAS_TABLE_V2_1_INFO_SIZE) \ |
| 124 | / RAS_TABLE_RECORD_SIZE) | 125 | / RAS_TABLE_RECORD_SIZE) |
| 125 | 126 | ||
| 127 | #define RAS_SMU_MESSAGE_TIMEOUT_MS 1000 /* 1s */ | ||
| 128 | |||
| 126 | /* Given a zero-based index of an EEPROM RAS record, yields the EEPROM | 129 | /* Given a zero-based index of an EEPROM RAS record, yields the EEPROM |
| 127 | * offset off of RAS_TABLE_START. That is, this is something you can | 130 | * offset off of RAS_TABLE_START. That is, this is something you can |
| 128 | * add to control->i2c_address, and then tell I2C layer to read | 131 | * add to control->i2c_address, and then tell I2C layer to read |
| @@ -443,40 +446,57 @@ int amdgpu_ras_eeprom_reset_table(struct amdgpu_ras_eeprom_control *control) · 314 unmodified lines | |||
| 443 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; | 446 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; |
| 444 | struct amdgpu_ras_eeprom_table_ras_info *rai = &control->tbl_rai; | 447 | struct amdgpu_ras_eeprom_table_ras_info *rai = &control->tbl_rai; |
| 445 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); | 448 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); |
| 449 | u32 erase_res = 0; | ||
| 446 | u8 csum; | 450 | u8 csum; |
| 447 | int res; | 451 | int res; |
| 448 | 452 | ||
| 449 | mutex_lock(&control->ras_tbl_mutex); | 453 | mutex_lock(&control->ras_tbl_mutex); |
| 450 | 454 | ||
| 451 | hdr->header = RAS_TABLE_HDR_VAL; | 455 | if (!amdgpu_ras_smu_eeprom_supported(adev)) { |
| 452 | amdgpu_ras_set_eeprom_table_version(control); | 456 | hdr->header = RAS_TABLE_HDR_VAL; |
| 457 | amdgpu_ras_set_eeprom_table_version(control); | ||
| 453 | 458 | ||
| 454 | if (hdr->version >= RAS_TABLE_VER_V2_1) { | 459 | if (hdr->version >= RAS_TABLE_VER_V2_1) { |
| 455 | hdr->first_rec_offset = RAS_RECORD_START_V2_1; | 460 | hdr->first_rec_offset = RAS_RECORD_START_V2_1; |
| 456 | hdr->tbl_size = RAS_TABLE_HEADER_SIZE + | 461 | hdr->tbl_size = RAS_TABLE_HEADER_SIZE + |
| 457 | RAS_TABLE_V2_1_INFO_SIZE; | 462 | RAS_TABLE_V2_1_INFO_SIZE; |
| 458 | rai->rma_status = GPU_HEALTH_USABLE; | 463 | rai->rma_status = GPU_HEALTH_USABLE; |
| 459 | /** | 464 | |
| 460 | * GPU health represented as a percentage. | 465 | control->ras_record_offset = RAS_RECORD_START_V2_1; |
| 461 | * 0 means worst health, 100 means fully health. | 466 | control->ras_max_record_count = RAS_MAX_RECORD_COUNT_V2_1; |
| 462 | */ | 467 | /** |
| 463 | rai->health_percent = 100; | 468 | * GPU health represented as a percentage. |
| 464 | /* ecc_page_threshold = 0 means disable bad page retirement */ | 469 | * 0 means worst health, 100 means fully health. |
| 465 | rai->ecc_page_threshold = con->bad_page_cnt_threshold; | 470 | */ |
| 471 | rai->health_percent = 100; | ||
| 472 | /* ecc_page_threshold = 0 means disable bad page retirement */ | ||
| 473 | rai->ecc_page_threshold = con->bad_page_cnt_threshold; | ||
| 474 | } else { | ||
| 475 | hdr->first_rec_offset = RAS_RECORD_START; | ||
| 476 | hdr->tbl_size = RAS_TABLE_HEADER_SIZE; | ||
| 477 | |||
| 478 | control->ras_record_offset = RAS_RECORD_START; | ||
| 479 | control->ras_max_record_count = RAS_MAX_RECORD_COUNT; | ||
| 480 | } | ||
| 481 | |||
| 482 | csum = __calc_hdr_byte_sum(control); | ||
| 483 | if (hdr->version >= RAS_TABLE_VER_V2_1) | ||
| 484 | csum += __calc_ras_info_byte_sum(control); | ||
| 485 | csum = -csum; | ||
| 486 | hdr->checksum = csum; | ||
| 487 | res = __write_table_header(control); | ||
| 488 | if (!res && hdr->version > RAS_TABLE_VER_V1) | ||
| 489 | res = __write_table_ras_info(control); | ||
| 466 | } else { | 490 | } else { |
| 467 | hdr->first_rec_offset = RAS_RECORD_START; | 491 | res = amdgpu_ras_smu_erase_ras_table(adev, &erase_res); |
| 468 | hdr->tbl_size = RAS_TABLE_HEADER_SIZE; | 492 | if (res || erase_res) { |
| 493 | dev_warn(adev->dev, "RAS EEPROM reset failed, res:%d result:%d", | ||
| 494 | res, erase_res); | ||
| 495 | if (!res) | ||
| 496 | res = -EIO; | ||
| 497 | } | ||
| 469 | } | 498 | } |
| 470 | 499 | ||
| 471 | csum = __calc_hdr_byte_sum(control); | ||
| 472 | if (hdr->version >= RAS_TABLE_VER_V2_1) | ||
| 473 | csum += __calc_ras_info_byte_sum(control); | ||
| 474 | csum = -csum; | ||
| 475 | hdr->checksum = csum; | ||
| 476 | res = __write_table_header(control); | ||
| 477 | if (!res && hdr->version > RAS_TABLE_VER_V1) | ||
| 478 | res = __write_table_ras_info(control); | ||
| 479 | |||
| 480 | control->ras_num_recs = 0; | 500 | control->ras_num_recs = 0; |
| 481 | control->ras_num_bad_pages = 0; | 501 | control->ras_num_bad_pages = 0; |
| 482 | control->ras_num_mca_recs = 0; | 502 | control->ras_num_mca_recs = 0; |
| @@ -556,6 +576,9 @@ bool amdgpu_ras_eeprom_check_err_threshold(struct amdgpu_device *adev) · 73 unmodified lines | |||
| 556 | { | 576 | { |
| 557 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); | 577 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); |
| 558 | 578 | ||
| 579 | if (amdgpu_uniras_enabled(adev)) | ||
| 580 | return amdgpu_ras_mgr_check_eeprom_safety_watermark(adev); | ||
| 581 | |||
| 559 | if (!__is_ras_eeprom_supported(adev) || | 582 | if (!__is_ras_eeprom_supported(adev) || |
| 560 | !amdgpu_bad_page_threshold) | 583 | !amdgpu_bad_page_threshold) |
| 561 | return false; | 584 | return false; |
| @@ -766,7 +789,8 @@ amdgpu_ras_eeprom_update_header(struct amdgpu_ras_eeprom_control *control) · 204 unmodified lines | |||
| 766 | "Saved bad pages %d reaches threshold value %d\n", | 789 | "Saved bad pages %d reaches threshold value %d\n", |
| 767 | control->ras_num_bad_pages, ras->bad_page_cnt_threshold); | 790 | control->ras_num_bad_pages, ras->bad_page_cnt_threshold); |
| 768 | 791 | ||
| 769 | if (adev->cper.enabled && amdgpu_cper_generate_bp_threshold_record(adev)) | 792 | if (adev->cper.enabled && !amdgpu_uniras_enabled(adev) && |
| 793 | amdgpu_cper_generate_bp_threshold_record(adev)) | ||
| 770 | dev_warn(adev->dev, "fail to generate bad page threshold cper records\n"); | 794 | dev_warn(adev->dev, "fail to generate bad page threshold cper records\n"); |
| 771 | 795 | ||
| 772 | if ((amdgpu_bad_page_threshold != -1) && | 796 | if ((amdgpu_bad_page_threshold != -1) && |
| @@ -849,6 +873,71 @@ Out: · 76 unmodified lines | |||
| 849 | return res; | 873 | return res; |
| 850 | } | 874 | } |
| 851 | 875 | ||
| 876 | int amdgpu_ras_eeprom_update_record_num(struct amdgpu_ras_eeprom_control *control) | ||
| 877 | { | ||
| 878 | struct amdgpu_device *adev = to_amdgpu_device(control); | ||
| 879 | int ret, retry = 20; | ||
| 880 | |||
| 881 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 882 | return 0; | ||
| 883 | |||
| 884 | control->ras_num_recs_old = control->ras_num_recs; | ||
| 885 | |||
| 886 | do { | ||
| 887 | /* 1000ms timeout is long enough, smu_get_badpage_count won't | ||
| 888 | * return -EBUSY before timeout. | ||
| 889 | */ | ||
| 890 | ret = amdgpu_ras_smu_get_badpage_count(adev, | ||
| 891 | &(control->ras_num_recs), RAS_SMU_MESSAGE_TIMEOUT_MS); | ||
| 892 | if (!ret && | ||
| 893 | (control->ras_num_recs_old == control->ras_num_recs)) { | ||
| 894 | /* record number update in PMFW needs some time, | ||
| 895 | * smu_get_badpage_count may return immediately without | ||
| 896 | * count update, sleep for a while and retry again. | ||
| 897 | */ | ||
| 898 | msleep(50); | ||
| 899 | retry--; | ||
| 900 | } else { | ||
| 901 | break; | ||
| 902 | } | ||
| 903 | } while (retry); | ||
| 904 | |||
| 905 | /* no update of record number is not a real failure, | ||
| 906 | * don't print warning here | ||
| 907 | */ | ||
| 908 | if (!ret && (control->ras_num_recs_old == control->ras_num_recs)) | ||
| 909 | ret = -EINVAL; | ||
| 910 | |||
| 911 | return ret; | ||
| 912 | } | ||
| 913 | |||
| 914 | static int amdgpu_ras_smu_eeprom_append(struct amdgpu_ras_eeprom_control *control) | ||
| 915 | { | ||
| 916 | struct amdgpu_device *adev = to_amdgpu_device(control); | ||
| 917 | struct amdgpu_ras *con = amdgpu_ras_get_context(adev); | ||
| 918 | |||
| 919 | if (!amdgpu_ras_smu_eeprom_supported(adev) || !con) | ||
| 920 | return 0; | ||
| 921 | |||
| 922 | control->ras_num_bad_pages = con->bad_page_num; | ||
| 923 | |||
| 924 | if (amdgpu_bad_page_threshold != 0 && | ||
| 925 | control->ras_num_bad_pages > con->bad_page_cnt_threshold) { | ||
| 926 | dev_warn(adev->dev, | ||
| 927 | "Saved bad pages %d reaches threshold value %d\n", | ||
| 928 | control->ras_num_bad_pages, con->bad_page_cnt_threshold); | ||
| 929 | |||
| 930 | if (adev->cper.enabled && amdgpu_cper_generate_bp_threshold_record(adev)) | ||
| 931 | dev_warn(adev->dev, "fail to generate bad page threshold cper records\n"); | ||
| 932 | |||
| 933 | if ((amdgpu_bad_page_threshold != -1) && | ||
| 934 | (amdgpu_bad_page_threshold != -2)) | ||
| 935 | con->is_rma = true; | ||
| 936 | } | ||
| 937 | |||
| 938 | return 0; | ||
| 939 | } | ||
| 940 | |||
| 852 | /** | 941 | /** |
| 853 | * amdgpu_ras_eeprom_append -- append records to the EEPROM RAS table | 942 | * amdgpu_ras_eeprom_append -- append records to the EEPROM RAS table |
| 854 | * @control: pointer to control structure | 943 | * @control: pointer to control structure |
| @@ -873,6 +962,9 @@ int amdgpu_ras_eeprom_append(struct amdgpu_ras_eeprom_control *control, · 18 unmodified lines | |||
| 873 | if (!__is_ras_eeprom_supported(adev)) | 962 | if (!__is_ras_eeprom_supported(adev)) |
| 874 | return 0; | 963 | return 0; |
| 875 | 964 | ||
| 965 | if (amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 966 | return amdgpu_ras_smu_eeprom_append(control); | ||
| 967 | |||
| 876 | if (num == 0) { | 968 | if (num == 0) { |
| 877 | dev_err(adev->dev, "will not append 0 records\n"); | 969 | dev_err(adev->dev, "will not append 0 records\n"); |
| 878 | return -EINVAL; | 970 | return -EINVAL; |
| @@ -948,6 +1040,50 @@ static int __amdgpu_ras_eeprom_read(struct amdgpu_ras_eeprom_control *control, · 69 unmodified lines | |||
| 948 | return res; | 1040 | return res; |
| 949 | } | 1041 | } |
| 950 | 1042 | ||
| 1043 | int amdgpu_ras_eeprom_read_idx(struct amdgpu_ras_eeprom_control *control, | ||
| 1044 | struct eeprom_table_record *record, u32 rec_idx, | ||
| 1045 | const u32 num) | ||
| 1046 | { | ||
| 1047 | struct amdgpu_device *adev = to_amdgpu_device(control); | ||
| 1048 | uint64_t ts, end_idx; | ||
| 1049 | int i, ret; | ||
| 1050 | u64 mca, ipid; | ||
| 1051 | |||
| 1052 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1053 | return 0; | ||
| 1054 | |||
| 1055 | if (!adev->umc.ras || !adev->umc.ras->mca_ipid_parse) | ||
| 1056 | return -EOPNOTSUPP; | ||
| 1057 | |||
| 1058 | end_idx = rec_idx + num; | ||
| 1059 | for (i = rec_idx; i < end_idx; i++) { | ||
| 1060 | ret = amdgpu_ras_smu_get_badpage_mca_addr(adev, i, &mca); | ||
| 1061 | if (ret) | ||
| 1062 | return ret; | ||
| 1063 | |||
| 1064 | ret = amdgpu_ras_smu_get_badpage_ipid(adev, i, &ipid); | ||
| 1065 | if (ret) | ||
| 1066 | return ret; | ||
| 1067 | |||
| 1068 | ret = amdgpu_ras_smu_get_timestamp(adev, i, &ts); | ||
| 1069 | if (ret) | ||
| 1070 | return ret; | ||
| 1071 | |||
| 1072 | record[i - rec_idx].address = mca; | ||
| 1073 | /* retired_page (pa) is unused now */ | ||
| 1074 | record[i - rec_idx].retired_page = 0x1ULL; | ||
| 1075 | record[i - rec_idx].ts = ts; | ||
| 1076 | record[i - rec_idx].err_type = AMDGPU_RAS_EEPROM_ERR_NON_RECOVERABLE; | ||
| 1077 | |||
| 1078 | adev->umc.ras->mca_ipid_parse(adev, ipid, | ||
| 1079 | (uint32_t *)&(record[i - rec_idx].cu), | ||
| 1080 | (uint32_t *)&(record[i - rec_idx].mem_channel), | ||
| 1081 | (uint32_t *)&(record[i - rec_idx].mcumc_id), NULL); | ||
| 1082 | } | ||
| 1083 | |||
| 1084 | return 0; | ||
| 1085 | } | ||
| 1086 | |||
| 951 | /** | 1087 | /** |
| 952 | * amdgpu_ras_eeprom_read -- read EEPROM | 1088 | * amdgpu_ras_eeprom_read -- read EEPROM |
| 953 | * @control: pointer to control structure | 1089 | * @control: pointer to control structure |
| @@ -969,6 +1105,9 @@ int amdgpu_ras_eeprom_read(struct amdgpu_ras_eeprom_control *control, · 15 unmodified lines | |||
| 969 | u8 *buf, *pp; | 1105 | u8 *buf, *pp; |
| 970 | u32 g0, g1; | 1106 | u32 g0, g1; |
| 971 | 1107 | ||
| 1108 | if (amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1109 | return amdgpu_ras_eeprom_read_idx(control, record, 0, num); | ||
| 1110 | |||
| 972 | if (!__is_ras_eeprom_supported(adev)) | 1111 | if (!__is_ras_eeprom_supported(adev)) |
| 973 | return 0; | 1112 | return 0; |
| 974 | 1113 | ||
| @@ -1140,6 +1279,10 @@ static ssize_t amdgpu_ras_debugfs_table_read(struct file *f, char __user *buf, · 165 unmodified lines | |||
| 1140 | int res = -EFAULT; | 1279 | int res = -EFAULT; |
| 1141 | size_t data_len; | 1280 | size_t data_len; |
| 1142 | 1281 | ||
| 1282 | /* pmfw manages eeprom data by itself */ | ||
| 1283 | if (amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1284 | return 0; | ||
| 1285 | |||
| 1143 | mutex_lock(&control->ras_tbl_mutex); | 1286 | mutex_lock(&control->ras_tbl_mutex); |
| 1144 | 1287 | ||
| 1145 | /* We want *pos - data_len > 0, which means there's | 1288 | /* We want *pos - data_len > 0, which means there's |
| @@ -1370,6 +1513,42 @@ Out: · 224 unmodified lines | |||
| 1370 | return res == RAS_TABLE_V2_1_INFO_SIZE ? 0 : res; | 1513 | return res == RAS_TABLE_V2_1_INFO_SIZE ? 0 : res; |
| 1371 | } | 1514 | } |
| 1372 | 1515 | ||
| 1516 | static int amdgpu_ras_smu_eeprom_init(struct amdgpu_ras_eeprom_control *control) | ||
| 1517 | { | ||
| 1518 | struct amdgpu_device *adev = to_amdgpu_device(control); | ||
| 1519 | struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; | ||
| 1520 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | ||
| 1521 | uint64_t local_time; | ||
| 1522 | int res; | ||
| 1523 | |||
| 1524 | ras->is_rma = false; | ||
| 1525 | |||
| 1526 | if (!__is_ras_eeprom_supported(adev)) | ||
| 1527 | return 0; | ||
| 1528 | mutex_init(&control->ras_tbl_mutex); | ||
| 1529 | |||
| 1530 | res = amdgpu_ras_smu_get_table_version(adev, &(hdr->version)); | ||
| 1531 | if (res) | ||
| 1532 | return res; | ||
| 1533 | |||
| 1534 | res = amdgpu_ras_smu_get_badpage_count(adev, | ||
| 1535 | &(control->ras_num_recs), 100); | ||
| 1536 | if (res) | ||
| 1537 | return res; | ||
| 1538 | |||
| 1539 | local_time = (uint64_t)ktime_get_real_seconds(); | ||
| 1540 | res = amdgpu_ras_smu_set_timestamp(adev, local_time); | ||
| 1541 | if (res) | ||
| 1542 | return res; | ||
| 1543 | |||
| 1544 | control->ras_max_record_count = 4000; | ||
| 1545 | |||
| 1546 | control->ras_num_mca_recs = 0; | ||
| 1547 | control->ras_num_pa_recs = 0; | ||
| 1548 | |||
| 1549 | return 0; | ||
| 1550 | } | ||
| 1551 | |||
| 1373 | int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) | 1552 | int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) |
| 1374 | { | 1553 | { |
| 1375 | struct amdgpu_device *adev = to_amdgpu_device(control); | 1554 | struct amdgpu_device *adev = to_amdgpu_device(control); |
| @@ -1378,6 +1557,9 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 2 unmodified lines | |||
| 1378 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | 1557 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); |
| 1379 | int res; | 1558 | int res; |
| 1380 | 1559 | ||
| 1560 | if (amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1561 | return amdgpu_ras_smu_eeprom_init(control); | ||
| 1562 | |||
| 1381 | ras->is_rma = false; | 1563 | ras->is_rma = false; |
| 1382 | 1564 | ||
| 1383 | if (!__is_ras_eeprom_supported(adev)) | 1565 | if (!__is_ras_eeprom_supported(adev)) |
| @@ -1444,6 +1626,47 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) · 60 unmodified lines | |||
| 1444 | return 0; | 1626 | return 0; |
| 1445 | } | 1627 | } |
| 1446 | 1628 | ||
| 1629 | static int amdgpu_ras_smu_eeprom_check(struct amdgpu_ras_eeprom_control *control) | ||
| 1630 | { | ||
| 1631 | struct amdgpu_device *adev = to_amdgpu_device(control); | ||
| 1632 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | ||
| 1633 | |||
| 1634 | if (!__is_ras_eeprom_supported(adev)) | ||
| 1635 | return 0; | ||
| 1636 | |||
| 1637 | control->ras_num_bad_pages = ras->bad_page_num; | ||
| 1638 | |||
| 1639 | if ((ras->bad_page_cnt_threshold < control->ras_num_bad_pages) && | ||
| 1640 | amdgpu_bad_page_threshold != 0) { | ||
| 1641 | dev_warn(adev->dev, | ||
| 1642 | "RAS records:%d exceed threshold:%d\n", | ||
| 1643 | control->ras_num_bad_pages, ras->bad_page_cnt_threshold); | ||
| 1644 | if ((amdgpu_bad_page_threshold == -1) || | ||
| 1645 | (amdgpu_bad_page_threshold == -2)) { | ||
| 1646 | dev_warn(adev->dev, | ||
| 1647 | "Please consult AMD Service Action Guide (SAG) for appropriate service procedures\n"); | ||
| 1648 | } else { | ||
| 1649 | ras->is_rma = true; | ||
| 1650 | dev_warn(adev->dev, | ||
| 1651 | "User defined threshold is set, runtime service will be halt when threshold is reached\n"); | ||
| 1652 | } | ||
| 1653 | |||
| 1654 | return 0; | ||
| 1655 | } | ||
| 1656 | |||
| 1657 | dev_dbg(adev->dev, | ||
| 1658 | "Found existing EEPROM table with %d records", | ||
| 1659 | control->ras_num_bad_pages); | ||
| 1660 | |||
| 1661 | /* Warn if we are at 90% of the threshold or above | ||
| 1662 | */ | ||
| 1663 | if (10 * control->ras_num_bad_pages >= 9 * ras->bad_page_cnt_threshold) | ||
| 1664 | dev_warn(adev->dev, "RAS records:%u exceeds 90%% of threshold:%d", | ||
| 1665 | control->ras_num_bad_pages, | ||
| 1666 | ras->bad_page_cnt_threshold); | ||
| 1667 | return 0; | ||
| 1668 | } | ||
| 1669 | |||
| 1447 | int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) | 1670 | int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) |
| 1448 | { | 1671 | { |
| 1449 | struct amdgpu_device *adev = to_amdgpu_device(control); | 1672 | struct amdgpu_device *adev = to_amdgpu_device(control); |
| @@ -1451,6 +1674,9 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control) · 1 unmodified line | |||
| 1451 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | 1674 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); |
| 1452 | int res = 0; | 1675 | int res = 0; |
| 1453 | 1676 | ||
| 1677 | if (amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1678 | return amdgpu_ras_smu_eeprom_check(control); | ||
| 1679 | |||
| 1454 | if (!__is_ras_eeprom_supported(adev)) | 1680 | if (!__is_ras_eeprom_supported(adev)) |
| 1455 | return 0; | 1681 | return 0; |
| 1456 | 1682 | ||
| @@ -1541,7 +1767,8 @@ void amdgpu_ras_eeprom_check_and_recover(struct amdgpu_device *adev) · 84 unmodified lines | |||
| 1541 | struct amdgpu_ras_eeprom_control *control; | 1767 | struct amdgpu_ras_eeprom_control *control; |
| 1542 | int res; | 1768 | int res; |
| 1543 | 1769 | ||
| 1544 | if (!__is_ras_eeprom_supported(adev) || !ras) | 1770 | if (!__is_ras_eeprom_supported(adev) || !ras || |
| 1771 | amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1545 | return; | 1772 | return; |
| 1546 | control = &ras->eeprom_control; | 1773 | control = &ras->eeprom_control; |
| 1547 | if (!control->is_eeprom_valid) | 1774 | if (!control->is_eeprom_valid) |
| @@ -1561,4 +1788,143 @@ void amdgpu_ras_eeprom_check_and_recover(struct amdgpu_device *adev) · 13 unmodified lines | |||
| 1561 | control->is_eeprom_valid = false; | 1788 | control->is_eeprom_valid = false; |
| 1562 | } | 1789 | } |
| 1563 | return; | 1790 | return; |
| 1564 | } | 1791 | } |
| 1792 | |||
| 1793 | static const struct ras_smu_drv *amdgpu_ras_get_smu_ras_drv(struct amdgpu_device *adev) | ||
| 1794 | { | ||
| 1795 | struct amdgpu_ras *ras = amdgpu_ras_get_context(adev); | ||
| 1796 | |||
| 1797 | if (!ras) | ||
| 1798 | return NULL; | ||
| 1799 | |||
| 1800 | return ras->ras_smu_drv; | ||
| 1801 | } | ||
| 1802 | |||
| 1803 | static uint64_t amdgpu_ras_smu_get_feature_flags(struct amdgpu_device *adev) | ||
| 1804 | { | ||
| 1805 | const struct ras_smu_drv *ras_smu_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1806 | uint64_t flags = 0ULL; | ||
| 1807 | |||
| 1808 | if (!ras_smu_drv) | ||
| 1809 | goto out; | ||
| 1810 | |||
| 1811 | if (ras_smu_drv->ras_smu_feature_flags) | ||
| 1812 | ras_smu_drv->ras_smu_feature_flags(adev, &flags); | ||
| 1813 | |||
| 1814 | out: | ||
| 1815 | return flags; | ||
| 1816 | } | ||
| 1817 | |||
| 1818 | bool amdgpu_ras_smu_eeprom_supported(struct amdgpu_device *adev) | ||
| 1819 | { | ||
| 1820 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1821 | uint64_t flags = 0ULL; | ||
| 1822 | |||
| 1823 | if (!__is_ras_eeprom_supported(adev) || !smu_ras_drv) | ||
| 1824 | return false; | ||
| 1825 | |||
| 1826 | if (!smu_ras_drv->smu_eeprom_funcs) | ||
| 1827 | return false; | ||
| 1828 | |||
| 1829 | flags = amdgpu_ras_smu_get_feature_flags(adev); | ||
| 1830 | |||
| 1831 | return !!(flags & RAS_SMU_FEATURE_BIT__RAS_EEPROM); | ||
| 1832 | } | ||
| 1833 | |||
| 1834 | int amdgpu_ras_smu_get_table_version(struct amdgpu_device *adev, | ||
| 1835 | uint32_t *table_version) | ||
| 1836 | { | ||
| 1837 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1838 | |||
| 1839 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1840 | return -EOPNOTSUPP; | ||
| 1841 | |||
| 1842 | if (smu_ras_drv->smu_eeprom_funcs->get_ras_table_version) | ||
| 1843 | return smu_ras_drv->smu_eeprom_funcs->get_ras_table_version(adev, | ||
| 1844 | table_version); | ||
| 1845 | return -EOPNOTSUPP; | ||
| 1846 | } | ||
| 1847 | |||
| 1848 | int amdgpu_ras_smu_get_badpage_count(struct amdgpu_device *adev, | ||
| 1849 | uint32_t *count, uint32_t timeout) | ||
| 1850 | { | ||
| 1851 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1852 | |||
| 1853 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1854 | return -EOPNOTSUPP; | ||
| 1855 | |||
| 1856 | if (smu_ras_drv->smu_eeprom_funcs->get_badpage_count) | ||
| 1857 | return smu_ras_drv->smu_eeprom_funcs->get_badpage_count(adev, | ||
| 1858 | count, timeout); | ||
| 1859 | return -EOPNOTSUPP; | ||
| 1860 | } | ||
| 1861 | |||
| 1862 | int amdgpu_ras_smu_get_badpage_mca_addr(struct amdgpu_device *adev, | ||
| 1863 | uint16_t index, uint64_t *mca_addr) | ||
| 1864 | { | ||
| 1865 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1866 | |||
| 1867 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1868 | return -EOPNOTSUPP; | ||
| 1869 | |||
| 1870 | if (smu_ras_drv->smu_eeprom_funcs->get_badpage_mca_addr) | ||
| 1871 | return smu_ras_drv->smu_eeprom_funcs->get_badpage_mca_addr(adev, | ||
| 1872 | index, mca_addr); | ||
| 1873 | return -EOPNOTSUPP; | ||
| 1874 | } | ||
| 1875 | |||
| 1876 | int amdgpu_ras_smu_set_timestamp(struct amdgpu_device *adev, | ||
| 1877 | uint64_t timestamp) | ||
| 1878 | { | ||
| 1879 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1880 | |||
| 1881 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1882 | return -EOPNOTSUPP; | ||
| 1883 | |||
| 1884 | if (smu_ras_drv->smu_eeprom_funcs->set_timestamp) | ||
| 1885 | return smu_ras_drv->smu_eeprom_funcs->set_timestamp(adev, | ||
| 1886 | timestamp); | ||
| 1887 | return -EOPNOTSUPP; | ||
| 1888 | } | ||
| 1889 | |||
| 1890 | int amdgpu_ras_smu_get_timestamp(struct amdgpu_device *adev, | ||
| 1891 | uint16_t index, uint64_t *timestamp) | ||
| 1892 | { | ||
| 1893 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1894 | |||
| 1895 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1896 | return -EOPNOTSUPP; | ||
| 1897 | |||
| 1898 | if (smu_ras_drv->smu_eeprom_funcs->get_timestamp) | ||
| 1899 | return smu_ras_drv->smu_eeprom_funcs->get_timestamp(adev, | ||
| 1900 | index, timestamp); | ||
| 1901 | return -EOPNOTSUPP; | ||
| 1902 | } | ||
| 1903 | |||
| 1904 | int amdgpu_ras_smu_get_badpage_ipid(struct amdgpu_device *adev, | ||
| 1905 | uint16_t index, uint64_t *ipid) | ||
| 1906 | { | ||
| 1907 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1908 | |||
| 1909 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1910 | return -EOPNOTSUPP; | ||
| 1911 | |||
| 1912 | if (smu_ras_drv->smu_eeprom_funcs->get_badpage_ipid) | ||
| 1913 | return smu_ras_drv->smu_eeprom_funcs->get_badpage_ipid(adev, | ||
| 1914 | index, ipid); | ||
| 1915 | return -EOPNOTSUPP; | ||
| 1916 | } | ||
| 1917 | |||
| 1918 | int amdgpu_ras_smu_erase_ras_table(struct amdgpu_device *adev, | ||
| 1919 | uint32_t *result) | ||
| 1920 | { | ||
| 1921 | const struct ras_smu_drv *smu_ras_drv = amdgpu_ras_get_smu_ras_drv(adev); | ||
| 1922 | |||
| 1923 | if (!amdgpu_ras_smu_eeprom_supported(adev)) | ||
| 1924 | return -EOPNOTSUPP; | ||
| 1925 | |||
| 1926 | if (smu_ras_drv->smu_eeprom_funcs->erase_ras_table) | ||
| 1927 | return smu_ras_drv->smu_eeprom_funcs->erase_ras_table(adev, | ||
| 1928 | result); | ||
| 1929 | return -EOPNOTSUPP; | ||
| 1930 | } | ||
| @@ -82,6 +82,7 @@ struct amdgpu_ras_eeprom_control { · 81 unmodified lines | |||
| 82 | /* Number of records in the table. | 82 | /* Number of records in the table. |
| 83 | */ | 83 | */ |
| 84 | u32 ras_num_recs; | 84 | u32 ras_num_recs; |
| 85 | u32 ras_num_recs_old; | ||
| 85 | 86 | ||
| 86 | /* the bad page number is ras_num_recs or | 87 | /* the bad page number is ras_num_recs or |
| 87 | * ras_num_recs * umc.retire_unit | 88 | * ras_num_recs * umc.retire_unit |
| @@ -163,6 +164,35 @@ int amdgpu_ras_eeprom_check(struct amdgpu_ras_eeprom_control *control); · 75 unmodified lines | |||
| 163 | 164 | ||
| 164 | void amdgpu_ras_eeprom_check_and_recover(struct amdgpu_device *adev); | 165 | void amdgpu_ras_eeprom_check_and_recover(struct amdgpu_device *adev); |
| 165 | 166 | ||
| 167 | bool amdgpu_ras_smu_eeprom_supported(struct amdgpu_device *adev); | ||
| 168 | |||
| 169 | int amdgpu_ras_smu_get_table_version(struct amdgpu_device *adev, | ||
| 170 | uint32_t *table_version); | ||
| 171 | |||
| 172 | int amdgpu_ras_smu_get_badpage_count(struct amdgpu_device *adev, | ||
| 173 | uint32_t *count, uint32_t timeout); | ||
| 174 | |||
| 175 | int amdgpu_ras_smu_get_badpage_mca_addr(struct amdgpu_device *adev, | ||
| 176 | uint16_t index, uint64_t *mca_addr); | ||
| 177 | |||
| 178 | int amdgpu_ras_smu_set_timestamp(struct amdgpu_device *adev, | ||
| 179 | uint64_t timestamp); | ||
| 180 | |||
| 181 | int amdgpu_ras_smu_get_timestamp(struct amdgpu_device *adev, | ||
| 182 | uint16_t index, uint64_t *timestamp); | ||
| 183 | |||
| 184 | int amdgpu_ras_smu_get_badpage_ipid(struct amdgpu_device *adev, | ||
| 185 | uint16_t index, uint64_t *ipid); | ||
| 186 | |||
| 187 | int amdgpu_ras_smu_erase_ras_table(struct amdgpu_device *adev, | ||
| 188 | uint32_t *result); | ||
| 189 | |||
| 190 | int amdgpu_ras_eeprom_read_idx(struct amdgpu_ras_eeprom_control *control, | ||
| 191 | struct eeprom_table_record *record, u32 rec_idx, | ||
| 192 | const u32 num); | ||
| 193 | |||
| 194 | int amdgpu_ras_eeprom_update_record_num(struct amdgpu_ras_eeprom_control *control); | ||
| 195 | |||
| 166 | extern const struct file_operations amdgpu_ras_debugfs_eeprom_size_ops; | 196 | extern const struct file_operations amdgpu_ras_debugfs_eeprom_size_ops; |
| 167 | extern const struct file_operations amdgpu_ras_debugfs_eeprom_table_ops; | 197 | extern const struct file_operations amdgpu_ras_debugfs_eeprom_table_ops; |
| 168 | 198 | ||
| @@ -33,6 +33,7 @@ · 32 unmodified lines | |||
| 33 | 33 | ||
| 34 | #include <drm/amdgpu_drm.h> | 34 | #include <drm/amdgpu_drm.h> |
| 35 | #include "amdgpu.h" | 35 | #include "amdgpu.h" |
| 36 | #include "amdgpu_ras_mgr.h" | ||
| 36 | #include "atom.h" | 37 | #include "atom.h" |
| 37 | 38 | ||
| 38 | /* | 39 | /* |
| @@ -159,8 +160,16 @@ void amdgpu_ring_insert_nop(struct amdgpu_ring *ring, uint32_t count) · 120 unmodified lines | |||
| 159 | */ | 160 | */ |
| 160 | void amdgpu_ring_generic_pad_ib(struct amdgpu_ring *ring, struct amdgpu_ib *ib) | 161 | void amdgpu_ring_generic_pad_ib(struct amdgpu_ring *ring, struct amdgpu_ib *ib) |
| 161 | { | 162 | { |
| 162 | while (ib->length_dw & ring->funcs->align_mask) | 163 | u32 align_mask = ring->funcs->align_mask; |
| 163 | ib->ptr[ib->length_dw++] = ring->funcs->nop; | 164 | u32 count = ib->length_dw & align_mask; |
| 165 | |||
| 166 | if (count) { | ||
| 167 | count = align_mask + 1 - count; | ||
| 168 | |||
| 169 | memset32(&ib->ptr[ib->length_dw], ring->funcs->nop, count); | ||
| 170 | |||
| 171 | ib->length_dw += count; | ||
| 172 | } | ||
| 164 | } | 173 | } |
| 165 | 174 | ||
| 166 | /** | 175 | /** |
| @@ -460,9 +469,6 @@ bool amdgpu_ring_soft_recovery(struct amdgpu_ring *ring, unsigned int vmid, · 293 unmodified lines | |||
| 460 | ktime_t deadline; | 469 | ktime_t deadline; |
| 461 | bool ret; | 470 | bool ret; |
| 462 | 471 | ||
| 463 | if (unlikely(ring->adev->debug_disable_soft_recovery)) | ||
| 464 | return false; | ||
| 465 | |||
| 466 | deadline = ktime_add_us(ktime_get(), 10000); | 472 | deadline = ktime_add_us(ktime_get(), 10000); |
| 467 | 473 | ||
| 468 | if (amdgpu_sriov_vf(ring->adev) || !ring->funcs->soft_recovery || !fence) | 474 | if (amdgpu_sriov_vf(ring->adev) || !ring->funcs->soft_recovery || !fence) |
| @@ -490,6 +496,66 @@ bool amdgpu_ring_soft_recovery(struct amdgpu_ring *ring, unsigned int vmid, · 21 unmodified lines | |||
| 490 | */ | 496 | */ |
| 491 | #if defined(CONFIG_DEBUG_FS) | 497 | #if defined(CONFIG_DEBUG_FS) |
| 492 | 498 | ||
| 499 | static ssize_t amdgpu_ras_cper_debugfs_read(struct file *f, char __user *buf, | ||
| 500 | size_t size, loff_t *offset) | ||
| 501 | { | ||
| 502 | const uint8_t ring_header_size = 12; | ||
| 503 | struct amdgpu_ring *ring = file_inode(f)->i_private; | ||
| 504 | struct ras_cmd_cper_snapshot_req *snapshot_req __free(kfree) = | ||
| 505 | kzalloc(sizeof(struct ras_cmd_cper_snapshot_req), GFP_KERNEL); | ||
| 506 | struct ras_cmd_cper_snapshot_rsp *snapshot_rsp __free(kfree) = | ||
| 507 | kzalloc(sizeof(struct ras_cmd_cper_snapshot_rsp), GFP_KERNEL); | ||
| 508 | struct ras_cmd_cper_record_req *record_req __free(kfree) = | ||
| 509 | kzalloc(sizeof(struct ras_cmd_cper_record_req), GFP_KERNEL); | ||
| 510 | struct ras_cmd_cper_record_rsp *record_rsp __free(kfree) = | ||
| 511 | kzalloc(sizeof(struct ras_cmd_cper_record_rsp), GFP_KERNEL); | ||
| 512 | uint8_t *ring_header __free(kfree) = | ||
| 513 | kzalloc(ring_header_size, GFP_KERNEL); | ||
| 514 | uint32_t total_cper_num; | ||
| 515 | uint64_t start_cper_id; | ||
| 516 | int r; | ||
| 517 | |||
| 518 | if (!snapshot_req || !snapshot_rsp || !record_req || !record_rsp || | ||
| 519 | !ring_header) | ||
| 520 | return -ENOMEM; | ||
| 521 | |||
| 522 | if (!(*offset)) { | ||
| 523 | /* Need at least 12 bytes for the header on the first read */ | ||
| 524 | if (size < ring_header_size) | ||
| 525 | return -EINVAL; | ||
| 526 | |||
| 527 | if (copy_to_user(buf, ring_header, ring_header_size)) | ||
| 528 | return -EFAULT; | ||
| 529 | buf += ring_header_size; | ||
| 530 | size -= ring_header_size; | ||
| 531 | } | ||
| 532 | |||
| 533 | r = amdgpu_ras_mgr_handle_ras_cmd(ring->adev, | ||
| 534 | RAS_CMD__GET_CPER_SNAPSHOT, | ||
| 535 | snapshot_req, sizeof(struct ras_cmd_cper_snapshot_req), | ||
| 536 | snapshot_rsp, sizeof(struct ras_cmd_cper_snapshot_rsp)); | ||
| 537 | if (r || !snapshot_rsp->total_cper_num) | ||
| 538 | return r; | ||
| 539 | |||
| 540 | start_cper_id = snapshot_rsp->start_cper_id; | ||
| 541 | total_cper_num = snapshot_rsp->total_cper_num; | ||
| 542 | |||
| 543 | record_req->buf_ptr = (uint64_t)(uintptr_t)buf; | ||
| 544 | record_req->buf_size = size; | ||
| 545 | record_req->cper_start_id = start_cper_id + *offset; | ||
| 546 | record_req->cper_num = total_cper_num; | ||
| 547 | r = amdgpu_ras_mgr_handle_ras_cmd(ring->adev, RAS_CMD__GET_CPER_RECORD, | ||
| 548 | record_req, sizeof(struct ras_cmd_cper_record_req), | ||
| 549 | record_rsp, sizeof(struct ras_cmd_cper_record_rsp)); | ||
| 550 | if (r) | ||
| 551 | return r; | ||
| 552 | |||
| 553 | r = *offset ? record_rsp->real_data_size : record_rsp->real_data_size + ring_header_size; | ||
| 554 | (*offset) += record_rsp->real_cper_num; | ||
| 555 | |||
| 556 | return r; | ||
| 557 | } | ||
| 558 | |||
| 493 | /* Layout of file is 12 bytes consisting of | 559 | /* Layout of file is 12 bytes consisting of |
| 494 | * - rptr | 560 | * - rptr |
| 495 | * - wptr | 561 | * - wptr |
| @@ -506,6 +572,9 @@ static ssize_t amdgpu_debugfs_ring_read(struct file *f, char __user *buf, · 10 unmodified lines | |||
| 506 | loff_t i; | 572 | loff_t i; |
| 507 | int r; | 573 | int r; |
| 508 | 574 | ||
| 575 | if (ring->funcs->type == AMDGPU_RING_TYPE_CPER && amdgpu_uniras_enabled(ring->adev)) | ||
| 576 | return amdgpu_ras_cper_debugfs_read(f, buf, size, pos); | ||
| 577 | |||
| 509 | if (*pos & 3 || size & 3) | 578 | if (*pos & 3 || size & 3) |
| 510 | return -EINVAL; | 579 | return -EINVAL; |
| 511 | 580 | ||
| @@ -83,6 +83,7 @@ enum amdgpu_ring_type { · 82 unmodified lines | |||
| 83 | AMDGPU_RING_TYPE_MES, | 83 | AMDGPU_RING_TYPE_MES, |
| 84 | AMDGPU_RING_TYPE_UMSCH_MM, | 84 | AMDGPU_RING_TYPE_UMSCH_MM, |
| 85 | AMDGPU_RING_TYPE_CPER, | 85 | AMDGPU_RING_TYPE_CPER, |
| 86 | AMDGPU_RING_TYPE_MAX, | ||
| 86 | }; | 87 | }; |
| 87 | 88 | ||
| 88 | enum amdgpu_ib_pool_type { | 89 | enum amdgpu_ib_pool_type { |
| @@ -147,16 +148,14 @@ struct amdgpu_fence { · 58 unmodified lines | |||
| 147 | u64 wptr; | 148 | u64 wptr; |
| 148 | /* fence context for resets */ | 149 | /* fence context for resets */ |
| 149 | u64 context; | 150 | u64 context; |
| 150 | uint32_t seq; | ||
| 151 | }; | 151 | }; |
| 152 | 152 | ||
| 153 | extern const struct drm_sched_backend_ops amdgpu_sched_ops; | 153 | extern const struct drm_sched_backend_ops amdgpu_sched_ops; |
| 154 | 154 | ||
| 155 | void amdgpu_fence_driver_clear_job_fences(struct amdgpu_ring *ring); | ||
| 156 | void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error); | 155 | void amdgpu_fence_driver_set_error(struct amdgpu_ring *ring, int error); |
| 157 | void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring); | 156 | void amdgpu_fence_driver_force_completion(struct amdgpu_ring *ring); |
| 158 | void amdgpu_fence_driver_guilty_force_completion(struct amdgpu_fence *af); | 157 | void amdgpu_fence_driver_guilty_force_completion(struct amdgpu_fence *af); |
| 159 | void amdgpu_fence_save_wptr(struct dma_fence *fence); | 158 | void amdgpu_fence_save_wptr(struct amdgpu_fence *af); |
| 160 | 159 | ||
| 161 | int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring); | 160 | int amdgpu_fence_driver_init_ring(struct amdgpu_ring *ring); |
| 162 | int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring, | 161 | int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring, |
| @@ -166,8 +165,8 @@ void amdgpu_fence_driver_hw_init(struct amdgpu_device *adev); · 3 unmodified lines | |||
| 166 | void amdgpu_fence_driver_hw_fini(struct amdgpu_device *adev); | 165 | void amdgpu_fence_driver_hw_fini(struct amdgpu_device *adev); |
| 167 | int amdgpu_fence_driver_sw_init(struct amdgpu_device *adev); | 166 | int amdgpu_fence_driver_sw_init(struct amdgpu_device *adev); |
| 168 | void amdgpu_fence_driver_sw_fini(struct amdgpu_device *adev); | 167 | void amdgpu_fence_driver_sw_fini(struct amdgpu_device *adev); |
| 169 | int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f, | 168 | int amdgpu_fence_emit(struct amdgpu_ring *ring, struct amdgpu_fence *af, |
| 170 | struct amdgpu_fence *af, unsigned int flags); | 169 | unsigned int flags); |
| 171 | int amdgpu_fence_emit_polling(struct amdgpu_ring *ring, uint32_t *s, | 170 | int amdgpu_fence_emit_polling(struct amdgpu_ring *ring, uint32_t *s, |
| 172 | uint32_t timeout); | 171 | uint32_t timeout); |
| 173 | bool amdgpu_fence_process(struct amdgpu_ring *ring); | 172 | bool amdgpu_fence_process(struct amdgpu_ring *ring); |
| @@ -159,7 +159,6 @@ static ssize_t amdgpu_securedisplay_debugfs_write(struct file *f, const char __u · 158 unmodified lines | |||
| 159 | dev_err(adev->dev, "Invalid input: %s\n", str); | 159 | dev_err(adev->dev, "Invalid input: %s\n", str); |
| 160 | } | 160 | } |
| 161 | 161 | ||
| 162 | pm_runtime_mark_last_busy(dev->dev); | ||
| 163 | pm_runtime_put_autosuspend(dev->dev); | 162 | pm_runtime_put_autosuspend(dev->dev); |
| 164 | 163 | ||
| 165 | return size; | 164 | return size; |
| @@ -188,7 +188,6 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo, · 187 unmodified lines | |||
| 188 | struct amdgpu_job *job; | 188 | struct amdgpu_job *job; |
| 189 | void *cpu_addr; | 189 | void *cpu_addr; |
| 190 | uint64_t flags; | 190 | uint64_t flags; |
| 191 | unsigned int i; | ||
| 192 | int r; | 191 | int r; |
| 193 | 192 | ||
| 194 | BUG_ON(adev->mman.buffer_funcs->copy_max_bytes < | 193 | BUG_ON(adev->mman.buffer_funcs->copy_max_bytes < |
| @@ -255,16 +254,9 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo, · 60 unmodified lines | |||
| 255 | dma_addr = &bo->ttm->dma_address[mm_cur->start >> PAGE_SHIFT]; | 254 | dma_addr = &bo->ttm->dma_address[mm_cur->start >> PAGE_SHIFT]; |
| 256 | amdgpu_gart_map(adev, 0, num_pages, dma_addr, flags, cpu_addr); | 255 | amdgpu_gart_map(adev, 0, num_pages, dma_addr, flags, cpu_addr); |
| 257 | } else { | 256 | } else { |
| 258 | dma_addr_t dma_address; | 257 | u64 pa = mm_cur->start + adev->vm_manager.vram_base_offset; |
| 259 | |||
| 260 | dma_address = mm_cur->start; | ||
| 261 | dma_address += adev->vm_manager.vram_base_offset; | ||
| 262 | 258 | ||
| 263 | for (i = 0; i < num_pages; ++i) { | 259 | amdgpu_gart_map_vram_range(adev, pa, 0, num_pages, flags, cpu_addr); |
| 264 | amdgpu_gart_map(adev, i << PAGE_SHIFT, 1, &dma_address, | ||
| 265 | flags, cpu_addr); | ||
| 266 | dma_address += PAGE_SIZE; | ||
| 267 | } | ||
| 268 | } | 260 | } |
| 269 | 261 | ||
| 270 | dma_fence_put(amdgpu_job_submit(job)); | 262 | dma_fence_put(amdgpu_job_submit(job)); |
| @@ -286,12 +278,13 @@ static int amdgpu_ttm_map_buffer(struct ttm_buffer_object *bo, · 15 unmodified lines | |||
| 286 | * move and different for a BO to BO copy. | 278 | * move and different for a BO to BO copy. |
| 287 | * | 279 | * |
| 288 | */ | 280 | */ |
| 289 | int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev, | 281 | __attribute__((nonnull)) |
| 290 | const struct amdgpu_copy_mem *src, | 282 | static int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev, |
| 291 | const struct amdgpu_copy_mem *dst, | 283 | const struct amdgpu_copy_mem *src, |
| 292 | uint64_t size, bool tmz, | 284 | const struct amdgpu_copy_mem *dst, |
| 293 | struct dma_resv *resv, | 285 | uint64_t size, bool tmz, |
| 294 | struct dma_fence **f) | 286 | struct dma_resv *resv, |
| 287 | struct dma_fence **f) | ||
| 295 | { | 288 | { |
| 296 | struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; | 289 | struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; |
| 297 | struct amdgpu_res_cursor src_mm, dst_mm; | 290 | struct amdgpu_res_cursor src_mm, dst_mm; |
| @@ -365,9 +358,7 @@ int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev, · 67 unmodified lines | |||
| 365 | } | 358 | } |
| 366 | error: | 359 | error: |
| 367 | mutex_unlock(&adev->mman.gtt_window_lock); | 360 | mutex_unlock(&adev->mman.gtt_window_lock); |
| 368 | if (f) | 361 | *f = fence; |
| 369 | *f = dma_fence_get(fence); | ||
| 370 | dma_fence_put(fence); | ||
| 371 | return r; | 362 | return r; |
| 372 | } | 363 | } |
| 373 | 364 | ||
| @@ -706,10 +697,11 @@ struct amdgpu_ttm_tt { · 332 unmodified lines | |||
| 706 | * memory and start HMM tracking CPU page table update | 697 | * memory and start HMM tracking CPU page table update |
| 707 | * | 698 | * |
| 708 | * Calling function must call amdgpu_ttm_tt_userptr_range_done() once and only | 699 | * Calling function must call amdgpu_ttm_tt_userptr_range_done() once and only |
| 709 | * once afterwards to stop HMM tracking | 700 | * once afterwards to stop HMM tracking. Its the caller responsibility to ensure |
| 701 | * that range is a valid memory and it is freed too. | ||
| 710 | */ | 702 | */ |
| 711 | int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, | 703 | int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, |
| 712 | struct hmm_range **range) | 704 | struct amdgpu_hmm_range *range) |
| 713 | { | 705 | { |
| 714 | struct ttm_tt *ttm = bo->tbo.ttm; | 706 | struct ttm_tt *ttm = bo->tbo.ttm; |
| 715 | struct amdgpu_ttm_tt *gtt = ttm_to_amdgpu_ttm_tt(ttm); | 707 | struct amdgpu_ttm_tt *gtt = ttm_to_amdgpu_ttm_tt(ttm); |
| @@ -719,9 +711,6 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, · 3 unmodified lines | |||
| 719 | bool readonly; | 711 | bool readonly; |
| 720 | int r = 0; | 712 | int r = 0; |
| 721 | 713 | ||
| 722 | /* Make sure get_user_pages_done() can cleanup gracefully */ | ||
| 723 | *range = NULL; | ||
| 724 | |||
| 725 | mm = bo->notifier.mm; | 714 | mm = bo->notifier.mm; |
| 726 | if (unlikely(!mm)) { | 715 | if (unlikely(!mm)) { |
| 727 | DRM_DEBUG_DRIVER("BO is not registered?\n"); | 716 | DRM_DEBUG_DRIVER("BO is not registered?\n"); |
| @@ -756,38 +745,6 @@ out_unlock: · 28 unmodified lines | |||
| 756 | return r; | 745 | return r; |
| 757 | } | 746 | } |
| 758 | 747 | ||
155 more lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 1004 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 265 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 466 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 840 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 512 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 860 lines aren’t shown here.
This diff is large: its 210 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 159 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 316 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 134 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 1556 lines aren’t shown here.
This diff is large: its 3339 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 263 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 699 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 199 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 142 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
File renamed without changes.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
File renamed without changes.
This diff is large: its 14 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 20 lines aren’t shown here.
File renamed without changes.
This diff is large: its 8 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
File renamed without changes.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
File renamed without changes.
This diff is large: its 375 lines aren’t shown here.
File renamed without changes.
This diff is large: its 9 lines aren’t shown here.
File renamed without changes.
This diff is large: its 212 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 6 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 850 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
This diff is large: its 6 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 159 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 17 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This diff is large: its 5 lines aren’t shown here.
File renamed without changes.
File renamed without changes.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 164 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 168 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 160 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 1501 lines aren’t shown here.
This diff is large: its 138 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 1556 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 142 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 144 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 224 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 198 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 145 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 150 lines aren’t shown here.
This diff is large: its 151 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 879 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 132 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 151 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 105 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 675 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 337 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 581 lines aren’t shown here.
This diff is large: its 194 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 286 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 649 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 126 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 191 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 280 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 371 lines aren’t shown here.
This diff is large: its 673 lines aren’t shown here.
This diff is large: its 165 lines aren’t shown here.
This diff is large: its 380 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 523 lines aren’t shown here.
This diff is large: its 427 lines aren’t shown here.
This diff is large: its 604 lines aren’t shown here.
This diff is large: its 316 lines aren’t shown here.
This diff is large: its 305 lines aren’t shown here.
This diff is large: its 1340 lines aren’t shown here.
This diff is large: its 198 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 427 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 318 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 323 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 751 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 232 lines aren’t shown here.
This diff is large: its 708 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 512 lines aren’t shown here.
This diff is large: its 315 lines aren’t shown here.
This diff is large: its 130 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 118 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 106 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 269 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 65 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 127 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 159 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 194 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 320 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 165 lines aren’t shown here.
This diff is large: its 299 lines aren’t shown here.
This diff is large: its 430 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 230 lines aren’t shown here.
This diff is large: its 189 lines aren’t shown here.
This diff is large: its 669 lines aren’t shown here.
This diff is large: its 325 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 600 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 202 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 262 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 172 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 254 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 199 lines aren’t shown here.
This diff is large: its 177 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 277 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 476 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 291 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 911 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 531 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 296 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 185 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 171 lines aren’t shown here.
This diff is large: its 745 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 264 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 169 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 161 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 398 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 90 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 182 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 434 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 231 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 116 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 2328 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 126 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 699 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 752 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 158 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 319 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 421 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 596 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 165 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 191 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 306 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 216 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 187 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 639 lines aren’t shown here.
This diff is large: its 457 lines aren’t shown here.
This diff is large: its 199 lines aren’t shown here.
This diff is large: its 397 lines aren’t shown here.
This diff is large: its 264 lines aren’t shown here.
This diff is large: its 1522 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 630 lines aren’t shown here.
This diff is large: its 714 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 728 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 123 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 1202 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 820 lines aren’t shown here.
This diff is large: its 437 lines aren’t shown here.
This diff is large: its 958 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 542 lines aren’t shown here.
This diff is large: its 234 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 5 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 652 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 147 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 133 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 3321 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 339 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 300 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 488 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 246 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 321 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 186 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 128 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 1687 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 185 lines aren’t shown here.
This diff is large: its 457 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 386 lines aren’t shown here.
This diff is large: its 246 lines aren’t shown here.
This diff is large: its 226 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 278 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 230 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 467 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 202 lines aren’t shown here.
This diff is large: its 617 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 311 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 151 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 224 lines aren’t shown here.
This diff is large: its 62 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 306 lines aren’t shown here.
This diff is large: its 550 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 627 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 141 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 496 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 328 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 272 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 237 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 194 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 365 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 324 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 418 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 193 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 91 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 331 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 261 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 72 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 257 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 97 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 270 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 103 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 415 lines aren’t shown here.
This diff is large: its 166 lines aren’t shown here.
This diff is large: its 121 lines aren’t shown here.
This diff is large: its 188 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 844 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 148 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 812 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 99 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 111 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 105 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 86 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 124 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 79 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 209 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 252 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 432 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 145 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 59 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 95 lines aren’t shown here.
This diff is large: its 326 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 263 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 214 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 270 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 680 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 546 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 952 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 606 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 1225 lines aren’t shown here.
This diff is large: its 56 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 593 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 429 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 438 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 89 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 568 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 287 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 937 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 190 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 563 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 64 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 446 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 137 lines aren’t shown here.
This diff is large: its 304 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 591 lines aren’t shown here.
This diff is large: its 189 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 131 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 260 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 109 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 521 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 226 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 280 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 396 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 341 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 439 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 648 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 334 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 101 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 73 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 254 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 147 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 320 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 117 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 183 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 331 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 455 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 68 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 152 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 55 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 314 lines aren’t shown here.
This diff is large: its 100 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 180 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 189 lines aren’t shown here.
This diff is large: its 680 lines aren’t shown here.
This diff is large: its 228 lines aren’t shown here.
This diff is large: its 930 lines aren’t shown here.
This diff is large: its 129 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 952 lines aren’t shown here.
This diff is large: its 408 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 218 lines aren’t shown here.
This diff is large: its 155 lines aren’t shown here.
This diff is large: its 439 lines aren’t shown here.
This diff is large: its 228 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 696 lines aren’t shown here.
This diff is large: its 52 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 196 lines aren’t shown here.
This diff is large: its 1183 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 276 lines aren’t shown here.
This diff is large: its 276 lines aren’t shown here.
This diff is large: its 87 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 308 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 358 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 88 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 110 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 48 lines aren’t shown here.
This diff is large: its 465 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 43 lines aren’t shown here.
This diff is large: its 46 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 17 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 83 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 219 lines aren’t shown here.
This diff is large: its 262 lines aren’t shown here.
This diff is large: its 126 lines aren’t shown here.
This diff is large: its 238 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.