History

Merge tag 'drm-xe-next-2024-07-30' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-next

drm-xe-next for 6.12 UAPI Changes: - Rename xe perf layer as xe observation layer, but was also made available via fixes to previous verison (Ashutosh) - Use write-back caching mode for system memory on DGFX, but was also mad available via fixes to previous version (Thomas) - Expose SIMD16 EU mask in topology query for userspace to know the type of EU, as available in PVC, Lunar Lake and Battlemage (Lucas) - Return ENOBUFS instead of ENOMEM in vm_bind if failure is tied to an array of binds (Matthew Brost) Driver Changes: - Log cleanup moving messages to debug priority (Michal Wajdeczko) - Add timeout to fences to adhere to dma_buf rules (Matthew Brost) - Rename old engine nomenclature to exec_queue (Matthew Brost) - Convert multiple bind ops to 1 job (Matthew Brost) - Add error injection for vm bind to help testing error path (Matthew Brost) - Fix error handling in page table to propagate correctly to userspace (Matthew Brost) - Re-organize and cleanup SR-IOV related registers (Michal Wajdeczko) - Make the device write barrier compatible with VF (Michal Wajdeczko) - New display workarounds for Battlemage (Matthew Auld) - New media workarounds for Lunar Lake and Battlemage (Ngai-Mint Kwan) - New graphics workarounds for Lunar Lake (Bommu Krishnaiah) - Tracepoint updates (Matthew Brost, Nirmoy Das) - Cleanup the header generation for OOB workarounds (Lucas De Marchi) - Fix leaking HDCP-related object (Nirmoy Das) - Serialize L2 flushes to avoid races (Tejas Upadhyay) - Log pid and comm on job timeout (José Roberto de Souza) - Simplify boilerplate code for live kunit (Michal Wajdeczko) - Improve kunit skips for live kunit (Michal Wajdeczko) - Fix xe_sync cleanup when handling xe_exec ioctl (Ashutosh Dixit) - Limit fair VF LMEM provisioning (Michal Wajdeczko) - New workaround to fence mmio writes in Lunar Lake (Tejas Upadhyay) - Warn on writes inaccessible register in VF (Michal Wajdeczko) - Fix register lookup in VF (Michal Wajdeczko) - Add GSC support for Battlemage (Alexander Usyskin) - Fix wedging only the GT in which timeout occurred (Matthew Brost) - Block device suspend when wedging (Matthew Brost) - Handle compression and migration changes for Battlemage (Akshata Jahagirdar) - Limit access of stolen memory for Lunar Lake (Uma Shankar) - Fail invalid addresses during user fence creation (Matthew Brost) - Refcount xe_file to safely and accurately store fdinfo stats (Umesh Nerlige Ramappa) - Cleanup and fix PM reference for TLB invalidation code (Matthew Brost) - Fix PM reference handling when communicating with GuC (Matthew Brost) - Add new BO flag for 2 MiB alignement and use in VF (Michal Wajdeczko) - Simplify MMIO setup for multi-tile platforms (Lucas De Marchi) - Add check for uninitialized access to OOB workarounds (Lucas De Marchi) - New GSC and HuC firmware blobs for Lunar Lake and Battlemage (Daniele Ceraolo Spurio) - Unify mmio wait logic (Gustavo Sousa) - Fix off-by-one when processing RTP rules (Lucas De Marchi) - Future-proof migrate logic with compressed PAT flag (Matt Roper) - Add WA kunit tests for Battlemage (Lucas De Marchi) - Test active tracking for workaorunds with kunit (Lucas De Marchi) - Add kunit tests for RTP with no actions (Lucas De Marchi) - Unify parse of OR rules in RTP (Lucas De Marchi) - Add performance tuning for Battlemage (Sai Teja Pottumuttu) - Make bit masks unsigned (Geert Uytterhoeven) Signed-off-by: Dave Airlie <airlied@redhat.com>

Dave Airlie committed 2 years ago

Commit a4172af2 parents de9c2c6 f2881df94 files · +3273 −1739Browse files

Unified · Split

94 files
  1. drivers/gpu/drm/i915/display/intel_display_wa.h +8 −0
  2. drivers/gpu/drm/i915/display/intel_fbc.c +6 −0
  3. drivers/gpu/drm/xe/Makefile +5 −18
  4. drivers/gpu/drm/xe/display/intel_fbdev_fb.c +5 −1
  5. drivers/gpu/drm/xe/display/xe_display_wa.c +16 −0
  6. drivers/gpu/drm/xe/display/xe_dsb_buffer.c +8 −0
  7. drivers/gpu/drm/xe/display/xe_fb_pin.c +3 −0
  8. drivers/gpu/drm/xe/display/xe_plane_initial.c +6 −0
  9. drivers/gpu/drm/xe/regs/xe_gt_regs.h +15 −0
  10. drivers/gpu/drm/xe/regs/xe_regs.h +8 −4
  11. drivers/gpu/drm/xe/regs/xe_sriov_regs.h +0 −23
  12. drivers/gpu/drm/xe/tests/Makefile +1 −5
  13. drivers/gpu/drm/xe/tests/xe_bo.c +33 −12
  14. drivers/gpu/drm/xe/tests/xe_bo_test.c +0 −21
  15. drivers/gpu/drm/xe/tests/xe_bo_test.h +0 −14
  16. drivers/gpu/drm/xe/tests/xe_dma_buf.c +20 −6
  17. drivers/gpu/drm/xe/tests/xe_dma_buf_test.c +0 −20
  18. drivers/gpu/drm/xe/tests/xe_dma_buf_test.h +0 −13
  19. drivers/gpu/drm/xe/tests/xe_kunit_helpers.c +39 −0
  20. drivers/gpu/drm/xe/tests/xe_kunit_helpers.h +2 −0
  21. drivers/gpu/drm/xe/tests/xe_live_test_mod.c +11 −0
  22. drivers/gpu/drm/xe/tests/xe_migrate.c +419 −5
  23. drivers/gpu/drm/xe/tests/xe_migrate_test.c +0 −20
  24. drivers/gpu/drm/xe/tests/xe_migrate_test.h +0 −13
  25. drivers/gpu/drm/xe/tests/xe_mocs.c +33 −11
  26. drivers/gpu/drm/xe/tests/xe_mocs_test.c +0 −21
  27. drivers/gpu/drm/xe/tests/xe_mocs_test.h +0 −14
  28. drivers/gpu/drm/xe/tests/xe_pci.c +30 −0
  29. drivers/gpu/drm/xe/tests/xe_pci_test.c +2 −2
  30. drivers/gpu/drm/xe/tests/xe_pci_test.h +2 −0
  31. drivers/gpu/drm/xe/tests/xe_rtp_test.c +197 −22
  32. drivers/gpu/drm/xe/tests/xe_test.h +3 −7
  33. drivers/gpu/drm/xe/tests/xe_wa_test.c +1 −0
  34. drivers/gpu/drm/xe/xe_bo.c +6 −5
  35. drivers/gpu/drm/xe/xe_bo.h +3 −2
  36. drivers/gpu/drm/xe/xe_bo_types.h +2 −0
  37. drivers/gpu/drm/xe/xe_devcoredump.c +2 −8
  38. drivers/gpu/drm/xe/xe_device.c +101 −10
  39. drivers/gpu/drm/xe/xe_device.h +9 −0
  40. drivers/gpu/drm/xe/xe_device_types.h +30 −0
  41. drivers/gpu/drm/xe/xe_drm_client.c +1 −4
  42. drivers/gpu/drm/xe/xe_exec_queue.c +32 −1
  43. drivers/gpu/drm/xe/xe_exec_queue.h +2 −0
  44. drivers/gpu/drm/xe/xe_exec_queue_types.h +7 −6
  45. drivers/gpu/drm/xe/xe_execlist.c +2 −1
  46. drivers/gpu/drm/xe/xe_gen_wa_oob.c +12 −4
  47. drivers/gpu/drm/xe/xe_gt.c +54 −0
  48. drivers/gpu/drm/xe/xe_gt_sriov_pf.c +1 −1
  49. drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c +1 −0
  50. drivers/gpu/drm/xe/xe_gt_sriov_vf.c +27 −1
  51. drivers/gpu/drm/xe/xe_gt_sriov_vf.h +1 −0
  52. drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +111 −94
  53. drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h +11 −1
  54. drivers/gpu/drm/xe/xe_gt_tlb_invalidation_types.h +4 −0
  55. drivers/gpu/drm/xe/xe_gt_topology.c +22 −5
  56. drivers/gpu/drm/xe/xe_gt_types.h +24 −1
  57. drivers/gpu/drm/xe/xe_guc_ct.c +10 −1
  58. drivers/gpu/drm/xe/xe_guc_id_mgr.c +2 −2
  59. drivers/gpu/drm/xe/xe_guc_submit.c +48 −8
  60. drivers/gpu/drm/xe/xe_heci_gsc.c +25 −3
  61. drivers/gpu/drm/xe/xe_heci_gsc.h +8 −2
  62. drivers/gpu/drm/xe/xe_irq.c +2 −0
  63. drivers/gpu/drm/xe/xe_lmtt.c +2 −2
  64. drivers/gpu/drm/xe/xe_migrate.c +290 −238
  65. drivers/gpu/drm/xe/xe_migrate.h +21 −13
  66. drivers/gpu/drm/xe/xe_mmio.c +144 −87
  67. drivers/gpu/drm/xe/xe_mmio.h +0 −1
  68. drivers/gpu/drm/xe/xe_oa.c +1 −1
  69. drivers/gpu/drm/xe/xe_pat.c +10 −1
  70. drivers/gpu/drm/xe/xe_pci.c +5 −2
  71. drivers/gpu/drm/xe/xe_pm.c +8 −0
  72. drivers/gpu/drm/xe/xe_preempt_fence.c +9 −3
  73. drivers/gpu/drm/xe/xe_pt.c +848 −462
  74. drivers/gpu/drm/xe/xe_pt.h +6 −8
  75. drivers/gpu/drm/xe/xe_pt_types.h +48 −0
  76. drivers/gpu/drm/xe/xe_query.c +3 −1
  77. drivers/gpu/drm/xe/xe_rtp.c +12 −30
  78. drivers/gpu/drm/xe/xe_rtp.h +2 −2
  79. drivers/gpu/drm/xe/xe_rtp_helpers.h +6 −0
  80. drivers/gpu/drm/xe/xe_sa.c +7 −0
  81. drivers/gpu/drm/xe/xe_sriov.c +1 −1
  82. drivers/gpu/drm/xe/xe_sync.c +8 −12
  83. drivers/gpu/drm/xe/xe_sync.h +0 −1
  84. drivers/gpu/drm/xe/xe_trace.h +52 −0
  85. drivers/gpu/drm/xe/xe_trace_bo.h +5 −5
  86. drivers/gpu/drm/xe/xe_tuning.c +8 −0
  87. drivers/gpu/drm/xe/xe_uc_fw.c +3 −0
  88. drivers/gpu/drm/xe/xe_vm.c +273 −426
  89. drivers/gpu/drm/xe/xe_vm.h +2 −0
  90. drivers/gpu/drm/xe/xe_vm_types.h +30 −25
  91. drivers/gpu/drm/xe/xe_wa.c +15 −0
  92. drivers/gpu/drm/xe/xe_wa.h +6 −1
  93. drivers/gpu/drm/xe/xe_wa_oob.rules +2 −0
  94. include/uapi/drm/xe_drm.h +13 −5
drivers/gpu/drm/i915/display/intel_display_wa.h+8−0
@@ -6,8 +6,16 @@ · 5 unmodified lines
66 #ifndef __INTEL_DISPLAY_WA_H__
77 #define __INTEL_DISPLAY_WA_H__
88
9+#include <linux/types.h>
10+
911 struct drm_i915_private;
1012
1113 void intel_display_wa_apply(struct drm_i915_private *i915);
1214
15+#ifdef I915
16+static inline bool intel_display_needs_wa_16023588340(struct drm_i915_private *i915) { return false; }
17+#else
18+bool intel_display_needs_wa_16023588340(struct drm_i915_private *i915);
19+#endif
20+
1321 #endif
drivers/gpu/drm/i915/display/intel_fbc.c+6−0
@@ -56,6 +56,7 @@ · 55 unmodified lines
5656 #include "intel_display_device.h"
5757 #include "intel_display_trace.h"
5858 #include "intel_display_types.h"
59+#include "intel_display_wa.h"
5960 #include "intel_fbc.h"
6061 #include "intel_fbc_regs.h"
6162 #include "intel_frontbuffer.h"
@@ -1237,6 +1238,11 @@ static int intel_fbc_check_plane(struct intel_atomic_state *state, · 1175 unmodified lines
12371238 return 0;
12381239 }
12391240
1241+ if (intel_display_needs_wa_16023588340(i915)) {
1242+ plane_state->no_fbc_reason = "Wa_16023588340";
1243+ return 0;
1244+ }
1245+
12401246 /* WaFbcTurnOffFbcWhenHyperVisorIsUsed:skl,bxt */
12411247 if (i915_vtd_active(i915) && (IS_SKYLAKE(i915) || IS_BROXTON(i915))) {
12421248 plane_state->no_fbc_reason = "VT-d enabled";
drivers/gpu/drm/xe/Makefile+5−18
@@ -12,32 +12,15 @@ subdir-ccflags-$(CONFIG_DRM_XE_WERROR) += -Werror · 11 unmodified lines
1212 subdir-ccflags-y += -I$(obj) -I$(src)
1313
1414 # generated sources
15−hostprogs := xe_gen_wa_oob
1615
16+hostprogs := xe_gen_wa_oob
1717 generated_oob := $(obj)/generated/xe_wa_oob.c $(obj)/generated/xe_wa_oob.h
18−
1918 quiet_cmd_wa_oob = GEN $(notdir $(generated_oob))
2019 cmd_wa_oob = mkdir -p $(@D); $^ $(generated_oob)
21−
2220 $(obj)/generated/%_wa_oob.c $(obj)/generated/%_wa_oob.h: $(obj)/xe_gen_wa_oob \
2321 $(src)/xe_wa_oob.rules
2422 $(call cmd,wa_oob)
2523
26−uses_generated_oob := \
27− $(obj)/xe_ggtt.o \
28− $(obj)/xe_gsc.o \
29− $(obj)/xe_gt.o \
30− $(obj)/xe_guc.o \
31− $(obj)/xe_guc_ads.o \
32− $(obj)/xe_guc_pc.o \
33− $(obj)/xe_migrate.o \
34− $(obj)/xe_ring_ops.o \
35− $(obj)/xe_vm.o \
36− $(obj)/xe_wa.o \
37− $(obj)/xe_ttm_stolen_mgr.o
38−
39−$(uses_generated_oob): $(generated_oob)
40−
4124 # Please keep these build lists sorted!
4225
4326 # core driver code
@@ -192,6 +175,7 @@ xe-$(CONFIG_DRM_XE_DISPLAY) += \ · 148 unmodified lines
192175 display/xe_display.o \
193176 display/xe_display_misc.o \
194177 display/xe_display_rps.o \
178+ display/xe_display_wa.o \
195179 display/xe_dsb_buffer.o \
196180 display/xe_fb_pin.o \
197181 display/xe_hdcp_gsc.o \
@@ -320,3 +304,6 @@ quiet_cmd_hdrtest = HDRTEST $(patsubst %.hdrtest,%.h,$@) · 122 unmodified lines
320304
321305 $(obj)/%.hdrtest: $(src)/%.h FORCE
322306 $(call if_changed_dep,hdrtest)
307+
308+uses_generated_oob := $(addprefix $(obj)/, $(xe-y))
309+$(uses_generated_oob): $(obj)/generated/xe_wa_oob.h
drivers/gpu/drm/xe/display/intel_fbdev_fb.c+5−1
@@ -10,6 +10,9 @@ · 9 unmodified lines
1010 #include "xe_bo.h"
1111 #include "xe_gt.h"
1212 #include "xe_ttm_stolen_mgr.h"
13+#include "xe_wa.h"
14+
15+#include <generated/xe_wa_oob.h>
1316
1417 struct intel_framebuffer *intel_fbdev_fb_alloc(struct drm_fb_helper *helper,
1518 struct drm_fb_helper_surface_size *sizes)
@@ -37,7 +40,7 @@ struct intel_framebuffer *intel_fbdev_fb_alloc(struct drm_fb_helper *helper, · 21 unmodified lines
3740 size = PAGE_ALIGN(size);
3841 obj = ERR_PTR(-ENODEV);
3942
40− if (!IS_DGFX(xe)) {
43+ if (!IS_DGFX(xe) && !XE_WA(xe_root_mmio_gt(xe), 22019338487_display)) {
4144 obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe),
4245 NULL, size,
4346 ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |
@@ -48,6 +51,7 @@ struct intel_framebuffer *intel_fbdev_fb_alloc(struct drm_fb_helper *helper, · 4 unmodified lines
4851 else
4952 drm_info(&xe->drm, "Allocated fbdev into stolen failed: %li\n", PTR_ERR(obj));
5053 }
54+
5155 if (IS_ERR(obj)) {
5256 obj = xe_bo_create_pin_map(xe, xe_device_get_root_tile(xe), NULL, size,
5357 ttm_bo_type_kernel, XE_BO_FLAG_SCANOUT |
drivers/gpu/drm/xe/display/xe_display_wa.c+16−0
@@ -0,0 +1,16 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright © 2024 Intel Corporation
4+ */
5+
6+#include "intel_display_wa.h"
7+
8+#include "xe_device.h"
9+#include "xe_wa.h"
10+
11+#include <generated/xe_wa_oob.h>
12+
13+bool intel_display_needs_wa_16023588340(struct drm_i915_private *i915)
14+{
15+ return XE_WA(xe_root_mmio_gt(i915), 16023588340);
16+}
drivers/gpu/drm/xe/display/xe_dsb_buffer.c+8−0
@@ -7,6 +7,8 @@ · 6 unmodified lines
77 #include "intel_display_types.h"
88 #include "intel_dsb_buffer.h"
99 #include "xe_bo.h"
10+#include "xe_device.h"
11+#include "xe_device_types.h"
1012 #include "xe_gt.h"
1113
1214 u32 intel_dsb_buffer_ggtt_offset(struct intel_dsb_buffer *dsb_buf)
@@ -16,7 +18,10 @@ u32 intel_dsb_buffer_ggtt_offset(struct intel_dsb_buffer *dsb_buf) · 3 unmodified lines
1618
1719 void intel_dsb_buffer_write(struct intel_dsb_buffer *dsb_buf, u32 idx, u32 val)
1820 {
21+ struct xe_device *xe = dsb_buf->vma->bo->tile->xe;
22+
1923 iosys_map_wr(&dsb_buf->vma->bo->vmap, idx * 4, u32, val);
24+ xe_device_l2_flush(xe);
2025 }
2126
2227 u32 intel_dsb_buffer_read(struct intel_dsb_buffer *dsb_buf, u32 idx)
@@ -26,9 +31,12 @@ u32 intel_dsb_buffer_read(struct intel_dsb_buffer *dsb_buf, u32 idx) · 3 unmodified lines
2631
2732 void intel_dsb_buffer_memset(struct intel_dsb_buffer *dsb_buf, u32 idx, u32 val, size_t size)
2833 {
34+ struct xe_device *xe = dsb_buf->vma->bo->tile->xe;
35+
2936 WARN_ON(idx > (dsb_buf->buf_size - size) / sizeof(*dsb_buf->cmd_buf));
3037
3138 iosys_map_memset(&dsb_buf->vma->bo->vmap, idx * 4, val, size);
39+ xe_device_l2_flush(xe);
3240 }
3341
3442 bool intel_dsb_buffer_create(struct intel_crtc *crtc, struct intel_dsb_buffer *dsb_buf, size_t size)
drivers/gpu/drm/xe/display/xe_fb_pin.c+3−0
@@ -10,6 +10,7 @@ · 9 unmodified lines
1010 #include "intel_fb.h"
1111 #include "intel_fb_pin.h"
1212 #include "xe_bo.h"
13+#include "xe_device.h"
1314 #include "xe_ggtt.h"
1415 #include "xe_gt.h"
1516 #include "xe_pm.h"
@@ -304,6 +305,8 @@ static struct i915_vma *__xe_pin_fb_vma(const struct intel_framebuffer *fb, · 288 unmodified lines
304305 if (ret)
305306 goto err_unpin;
306307
308+ /* Ensure DPT writes are flushed */
309+ xe_device_l2_flush(xe);
307310 return vma;
308311
309312 err_unpin:
drivers/gpu/drm/xe/display/xe_plane_initial.c+6−0
@@ -18,6 +18,9 @@ · 17 unmodified lines
1818 #include "intel_frontbuffer.h"
1919 #include "intel_plane_initial.h"
2020 #include "xe_bo.h"
21+#include "xe_wa.h"
22+
23+#include <generated/xe_wa_oob.h>
2124
2225 static bool
2326 intel_reuse_initial_plane_obj(struct intel_crtc *this,
@@ -104,6 +107,9 @@ initial_plane_bo(struct xe_device *xe, · 80 unmodified lines
104107 phys_base = base;
105108 flags |= XE_BO_FLAG_STOLEN;
106109
110+ if (XE_WA(xe_root_mmio_gt(xe), 22019338487_display))
111+ return NULL;
112+
107113 /*
108114 * If the FB is too big, just don't use it since fbdev is not very
109115 * important and we should probably use that space with FBC or other
drivers/gpu/drm/xe/regs/xe_gt_regs.h+15−0
@@ -80,6 +80,9 @@ · 79 unmodified lines
8080 #define LE_CACHEABILITY_MASK REG_GENMASK(1, 0)
8181 #define LE_CACHEABILITY(value) REG_FIELD_PREP(LE_CACHEABILITY_MASK, value)
8282
83+#define XE2_GAMREQSTRM_CTRL XE_REG(0x4194)
84+#define CG_DIS_CNTLBUS REG_BIT(6)
85+
8386 #define CCS_AUX_INV XE_REG(0x4208)
8487
8588 #define VD0_AUX_INV XE_REG(0x4218)
@@ -88,6 +91,8 @@ · 2 unmodified lines
8891 #define VE1_AUX_INV XE_REG(0x42b8)
8992 #define AUX_INV REG_BIT(0)
9093
94+#define XE2_LMEM_CFG XE_REG(0x48b0)
95+
9196 #define XEHP_TILE_ADDR_RANGE(_idx) XE_REG_MCR(0x4900 + (_idx) * 4)
9297 #define XEHP_FLAT_CCS_BASE_ADDR XE_REG_MCR(0x4910)
9398 #define XEHP_FLAT_CCS_PTR REG_GENMASK(31, 8)
@@ -103,6 +108,7 @@ · 9 unmodified lines
103108
104109 #define FF_MODE XE_REG_MCR(0x6210)
105110 #define DIS_TE_AUTOSTRIP REG_BIT(31)
111+#define VS_HIT_MAX_VALUE_MASK REG_GENMASK(25, 20)
106112 #define DIS_MESH_PARTIAL_AUTOSTRIP REG_BIT(16)
107113 #define DIS_MESH_AUTOSTRIP REG_BIT(15)
108114
@@ -372,6 +378,11 @@ · 263 unmodified lines
372378
373379 #define XEHPC_L3CLOS_MASK(i) XE_REG_MCR(0xb194 + (i) * 8)
374380
381+#define XE2_GLOBAL_INVAL XE_REG(0xb404)
382+
383+#define SCRATCH1LPFC XE_REG(0xb474)
384+#define EN_L3_RW_CCS_CACHE_FLUSH REG_BIT(0)
385+
375386 #define XE2LPM_L3SQCREG5 XE_REG_MCR(0xb658)
376387
377388 #define XE2_TDF_CTRL XE_REG(0xb418)
@@ -395,6 +406,10 @@ · 17 unmodified lines
395406 #define INVALIDATION_BROADCAST_MODE_DIS REG_BIT(12)
396407 #define GLOBAL_INVALIDATION_MODE REG_BIT(2)
397408
409+#define LMEM_CFG XE_REG(0xcf58)
410+#define LMEM_EN REG_BIT(31)
411+#define LMTT_DIR_PTR REG_GENMASK(30, 0) /* in multiples of 64KB */
412+
398413 #define HALF_SLICE_CHICKEN5 XE_REG_MCR(0xe188, XE_REG_OPTION_MASKED)
399414 #define DISABLE_SAMPLE_G_PERFORMANCE REG_BIT(0)
400415
drivers/gpu/drm/xe/regs/xe_regs.h+8−4
@@ -15,8 +15,6 @@ · 14 unmodified lines
1515 #define GU_MISC_IRQ_OFFSET 0x444f0
1616 #define GU_MISC_GSE REG_BIT(27)
1717
18−#define SOFTWARE_FLAGS_SPR33 XE_REG(0x4f084)
19−
2018 #define GU_CNTL_PROTECTED XE_REG(0x10100C)
2119 #define DRIVERINT_FLR_DIS REG_BIT(31)
2220
@@ -24,11 +22,14 @@ · 1 unmodified line
2422 #define LMEM_INIT REG_BIT(7)
2523 #define DRIVERFLR REG_BIT(31)
2624
25+#define XEHP_CLOCK_GATE_DIS XE_REG(0x101014)
26+#define SGSI_SIDECLK_DIS REG_BIT(17)
27+
2728 #define GU_DEBUG XE_REG(0x101018)
2829 #define DRIVERFLR_STATUS REG_BIT(31)
2930
30−#define XEHP_CLOCK_GATE_DIS XE_REG(0x101014)
31−#define SGSI_SIDECLK_DIS REG_BIT(17)
31+#define VIRTUAL_CTRL_REG XE_REG(0x10108c)
32+#define GUEST_GTT_UPDATE_EN REG_BIT(8)
3233
3334 #define XEHP_MTCFG_ADDR XE_REG(0x101800)
3435 #define TILE_COUNT REG_GENMASK(15, 8)
@@ -66,6 +67,9 @@ · 31 unmodified lines
6667 #define DISPLAY_IRQ REG_BIT(16)
6768 #define GT_DW_IRQ(x) REG_BIT(x)
6869
70+#define VF_CAP_REG XE_REG(0x1901f8, XE_REG_OPTION_VF)
71+#define VF_CAP REG_BIT(0)
72+
6973 #define PVC_RP_STATE_CAP XE_REG(0x281014)
7074
7175 #endif
drivers/gpu/drm/xe/regs/xe_sriov_regs.h+0−23
@@ -1,23 +0,0 @@
1−/* SPDX-License-Identifier: MIT */
2−/*
3− * Copyright © 2023 Intel Corporation
4− */
5−
6−#ifndef _REGS_XE_SRIOV_REGS_H_
7−#define _REGS_XE_SRIOV_REGS_H_
8−
9−#include "regs/xe_reg_defs.h"
10−
11−#define XE2_LMEM_CFG XE_REG(0x48b0)
12−
13−#define LMEM_CFG XE_REG(0xcf58)
14−#define LMEM_EN REG_BIT(31)
15−#define LMTT_DIR_PTR REG_GENMASK(30, 0) /* in multiples of 64KB */
16−
17−#define VIRTUAL_CTRL_REG XE_REG(0x10108c)
18−#define GUEST_GTT_UPDATE_EN REG_BIT(8)
19−
20−#define VF_CAP_REG XE_REG(0x1901f8, XE_REG_OPTION_VF)
21−#define VF_CAP REG_BIT(0)
22−
23−#endif
drivers/gpu/drm/xe/tests/Makefile+1−5
@@ -2,11 +2,7 @@ · 1 unmodified line
22
33 # "live" kunit tests
44 obj-$(CONFIG_DRM_XE_KUNIT_TEST) += xe_live_test.o
5−xe_live_test-y = xe_live_test_mod.o \
6− xe_bo_test.o \
7− xe_dma_buf_test.o \
8− xe_migrate_test.o \
9− xe_mocs_test.o
5+xe_live_test-y = xe_live_test_mod.o
106
117 # Normal kunit tests
128 obj-$(CONFIG_DRM_XE_KUNIT_TEST) += xe_test.o
drivers/gpu/drm/xe/tests/xe_bo.c+33−12
@@ -6,7 +6,7 @@ · 5 unmodified lines
66 #include <kunit/test.h>
77 #include <kunit/visibility.h>
88
9−#include "tests/xe_bo_test.h"
9+#include "tests/xe_kunit_helpers.h"
1010 #include "tests/xe_pci_test.h"
1111 #include "tests/xe_test.h"
1212
@@ -154,12 +154,18 @@ out_unlock: · 141 unmodified lines
154154
155155 static int ccs_test_run_device(struct xe_device *xe)
156156 {
157− struct kunit *test = xe_cur_kunit();
157+ struct kunit *test = kunit_get_current_test();
158158 struct xe_tile *tile;
159159 int id;
160160
161161 if (!xe_device_has_flat_ccs(xe)) {
162− kunit_info(test, "Skipping non-flat-ccs device.\n");
162+ kunit_skip(test, "non-flat-ccs device\n");
163+ return 0;
164+ }
165+
166+ /* For xe2+ dgfx, we don't handle ccs metadata */
167+ if (GRAPHICS_VER(xe) >= 20 && IS_DGFX(xe)) {
168+ kunit_skip(test, "xe2+ dgfx device\n");
163169 return 0;
164170 }
165171
@@ -177,11 +183,12 @@ static int ccs_test_run_device(struct xe_device *xe) · 11 unmodified lines
177183 return 0;
178184 }
179185
180−void xe_ccs_migrate_kunit(struct kunit *test)
186+static void xe_ccs_migrate_kunit(struct kunit *test)
181187 {
182− xe_call_for_each_device(ccs_test_run_device);
188+ struct xe_device *xe = test->priv;
189+
190+ ccs_test_run_device(xe);
183191 }
184−EXPORT_SYMBOL_IF_KUNIT(xe_ccs_migrate_kunit);
185192
186193 static int evict_test_run_tile(struct xe_device *xe, struct xe_tile *tile, struct kunit *test)
187194 {
@@ -325,13 +332,12 @@ cleanup_bo: · 137 unmodified lines
325332
326333 static int evict_test_run_device(struct xe_device *xe)
327334 {
328− struct kunit *test = xe_cur_kunit();
335+ struct kunit *test = kunit_get_current_test();
329336 struct xe_tile *tile;
330337 int id;
331338
332339 if (!IS_DGFX(xe)) {
333− kunit_info(test, "Skipping non-discrete device %s.\n",
334− dev_name(xe->drm.dev));
340+ kunit_skip(test, "non-discrete device\n");
335341 return 0;
336342 }
337343
@@ -345,8 +351,23 @@ static int evict_test_run_device(struct xe_device *xe) · 7 unmodified lines
345351 return 0;
346352 }
347353
348−void xe_bo_evict_kunit(struct kunit *test)
354+static void xe_bo_evict_kunit(struct kunit *test)
349355 {
350− xe_call_for_each_device(evict_test_run_device);
356+ struct xe_device *xe = test->priv;
357+
358+ evict_test_run_device(xe);
351359 }
352−EXPORT_SYMBOL_IF_KUNIT(xe_bo_evict_kunit);
360+
361+static struct kunit_case xe_bo_tests[] = {
362+ KUNIT_CASE_PARAM(xe_ccs_migrate_kunit, xe_pci_live_device_gen_param),
363+ KUNIT_CASE_PARAM(xe_bo_evict_kunit, xe_pci_live_device_gen_param),
364+ {}
365+};
366+
367+VISIBLE_IF_KUNIT
368+struct kunit_suite xe_bo_test_suite = {
369+ .name = "xe_bo",
370+ .test_cases = xe_bo_tests,
371+ .init = xe_kunit_helper_xe_device_live_test_init,
372+};
373+EXPORT_SYMBOL_IF_KUNIT(xe_bo_test_suite);
drivers/gpu/drm/xe/tests/xe_bo_test.c+0−21
@@ -1,21 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0
2−/*
3− * Copyright © 2022 Intel Corporation
4− */
5−
6−#include "xe_bo_test.h"
7−
8−#include <kunit/test.h>
9−
10−static struct kunit_case xe_bo_tests[] = {
11− KUNIT_CASE(xe_ccs_migrate_kunit),
12− KUNIT_CASE(xe_bo_evict_kunit),
13− {}
14−};
15−
16−static struct kunit_suite xe_bo_test_suite = {
17− .name = "xe_bo",
18− .test_cases = xe_bo_tests,
19−};
20−
21−kunit_test_suite(xe_bo_test_suite);
drivers/gpu/drm/xe/tests/xe_bo_test.h+0−14
@@ -1,14 +0,0 @@
1−/* SPDX-License-Identifier: GPL-2.0 AND MIT */
2−/*
3− * Copyright © 2023 Intel Corporation
4− */
5−
6−#ifndef _XE_BO_TEST_H_
7−#define _XE_BO_TEST_H_
8−
9−struct kunit;
10−
11−void xe_ccs_migrate_kunit(struct kunit *test);
12−void xe_bo_evict_kunit(struct kunit *test);
13−
14−#endif
drivers/gpu/drm/xe/tests/xe_dma_buf.c+20−6
@@ -8,7 +8,7 @@ · 7 unmodified lines
88 #include <kunit/test.h>
99 #include <kunit/visibility.h>
1010
11−#include "tests/xe_dma_buf_test.h"
11+#include "tests/xe_kunit_helpers.h"
1212 #include "tests/xe_pci_test.h"
1313
1414 #include "xe_pci.h"
@@ -107,7 +107,7 @@ static void check_residency(struct kunit *test, struct xe_bo *exported, · 92 unmodified lines
107107
108108 static void xe_test_dmabuf_import_same_driver(struct xe_device *xe)
109109 {
110− struct kunit *test = xe_cur_kunit();
110+ struct kunit *test = kunit_get_current_test();
111111 struct dma_buf_test_params *params = to_dma_buf_test_params(test->priv);
112112 struct drm_gem_object *import;
113113 struct dma_buf *dmabuf;
@@ -258,7 +258,7 @@ static const struct dma_buf_test_params test_params[] = { · 144 unmodified lines
258258 static int dma_buf_run_device(struct xe_device *xe)
259259 {
260260 const struct dma_buf_test_params *params;
261− struct kunit *test = xe_cur_kunit();
261+ struct kunit *test = kunit_get_current_test();
262262
263263 xe_pm_runtime_get(xe);
264264 for (params = test_params; params->mem_mask; ++params) {
@@ -274,8 +274,22 @@ static int dma_buf_run_device(struct xe_device *xe) · 9 unmodified lines
274274 return 0;
275275 }
276276
277−void xe_dma_buf_kunit(struct kunit *test)
277+static void xe_dma_buf_kunit(struct kunit *test)
278278 {
279− xe_call_for_each_device(dma_buf_run_device);
279+ struct xe_device *xe = test->priv;
280+
281+ dma_buf_run_device(xe);
280282 }
281−EXPORT_SYMBOL_IF_KUNIT(xe_dma_buf_kunit);
283+
284+static struct kunit_case xe_dma_buf_tests[] = {
285+ KUNIT_CASE_PARAM(xe_dma_buf_kunit, xe_pci_live_device_gen_param),
286+ {}
287+};
288+
289+VISIBLE_IF_KUNIT
290+struct kunit_suite xe_dma_buf_test_suite = {
291+ .name = "xe_dma_buf",
292+ .test_cases = xe_dma_buf_tests,
293+ .init = xe_kunit_helper_xe_device_live_test_init,
294+};
295+EXPORT_SYMBOL_IF_KUNIT(xe_dma_buf_test_suite);
drivers/gpu/drm/xe/tests/xe_dma_buf_test.c+0−20
@@ -1,20 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0
2−/*
3− * Copyright © 2022 Intel Corporation
4− */
5−
6−#include "xe_dma_buf_test.h"
7−
8−#include <kunit/test.h>
9−
10−static struct kunit_case xe_dma_buf_tests[] = {
11− KUNIT_CASE(xe_dma_buf_kunit),
12− {}
13−};
14−
15−static struct kunit_suite xe_dma_buf_test_suite = {
16− .name = "xe_dma_buf",
17− .test_cases = xe_dma_buf_tests,
18−};
19−
20−kunit_test_suite(xe_dma_buf_test_suite);
drivers/gpu/drm/xe/tests/xe_dma_buf_test.h+0−13
@@ -1,13 +0,0 @@
1−/* SPDX-License-Identifier: GPL-2.0 AND MIT */
2−/*
3− * Copyright © 2023 Intel Corporation
4− */
5−
6−#ifndef _XE_DMA_BUF_TEST_H_
7−#define _XE_DMA_BUF_TEST_H_
8−
9−struct kunit;
10−
11−void xe_dma_buf_kunit(struct kunit *test);
12−
13−#endif
drivers/gpu/drm/xe/tests/xe_kunit_helpers.c+39−0
@@ -12,7 +12,9 @@ · 11 unmodified lines
1212
1313 #include "tests/xe_kunit_helpers.h"
1414 #include "tests/xe_pci_test.h"
15+#include "xe_device.h"
1516 #include "xe_device_types.h"
17+#include "xe_pm.h"
1618
1719 /**
1820 * xe_kunit_helper_alloc_xe_device - Allocate a &xe_device for a KUnit test.
@@ -88,3 +90,40 @@ int xe_kunit_helper_xe_device_test_init(struct kunit *test) · 69 unmodified lines
8890 return 0;
8991 }
9092 EXPORT_SYMBOL_IF_KUNIT(xe_kunit_helper_xe_device_test_init);
93+
94+KUNIT_DEFINE_ACTION_WRAPPER(put_xe_pm_runtime, xe_pm_runtime_put, struct xe_device *);
95+
96+/**
97+ * xe_kunit_helper_xe_device_live_test_init - Prepare a &xe_device for
98+ * use in a live KUnit test.
99+ * @test: the &kunit where live &xe_device will be used
100+ *
101+ * This function expects pointer to the &xe_device in the &test.param_value,
102+ * like it is prepared by the &xe_pci_live_device_gen_param and stores that
103+ * pointer as &kunit.priv to allow the test code to access it.
104+ *
105+ * This function makes sure that device is not wedged and then resumes it
106+ * to avoid waking up the device inside the test. It uses deferred cleanup
107+ * action to release a runtime_pm reference.
108+ *
109+ * This function can be used as custom implementation of &kunit_suite.init.
110+ *
111+ * This function uses KUNIT_ASSERT to detect any failures.
112+ *
113+ * Return: Always 0.
114+ */
115+int xe_kunit_helper_xe_device_live_test_init(struct kunit *test)
116+{
117+ struct xe_device *xe = xe_device_const_cast(test->param_value);
118+
119+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, xe);
120+ kunit_info(test, "running on %s device\n", xe->info.platform_name);
121+
122+ KUNIT_ASSERT_FALSE(test, xe_device_wedged(xe));
123+ xe_pm_runtime_get(xe);
124+ KUNIT_ASSERT_EQ(test, 0, kunit_add_action_or_reset(test, put_xe_pm_runtime, xe));
125+
126+ test->priv = xe;
127+ return 0;
128+}
129+EXPORT_SYMBOL_IF_KUNIT(xe_kunit_helper_xe_device_live_test_init);
drivers/gpu/drm/xe/tests/xe_kunit_helpers.h+2−0
@@ -14,4 +14,6 @@ struct xe_device *xe_kunit_helper_alloc_xe_device(struct kunit *test, · 13 unmodified lines
1414 struct device *dev);
1515 int xe_kunit_helper_xe_device_test_init(struct kunit *test);
1616
17+int xe_kunit_helper_xe_device_live_test_init(struct kunit *test);
18+
1719 #endif
drivers/gpu/drm/xe/tests/xe_live_test_mod.c+11−0
@@ -3,6 +3,17 @@ · 2 unmodified lines
33 * Copyright © 2023 Intel Corporation
44 */
55 #include <linux/module.h>
6+#include <kunit/test.h>
7+
8+extern struct kunit_suite xe_bo_test_suite;
9+extern struct kunit_suite xe_dma_buf_test_suite;
10+extern struct kunit_suite xe_migrate_test_suite;
11+extern struct kunit_suite xe_mocs_test_suite;
12+
13+kunit_test_suite(xe_bo_test_suite);
14+kunit_test_suite(xe_dma_buf_test_suite);
15+kunit_test_suite(xe_migrate_test_suite);
16+kunit_test_suite(xe_mocs_test_suite);
617
718 MODULE_AUTHOR("Intel Corporation");
819 MODULE_LICENSE("GPL");
drivers/gpu/drm/xe/tests/xe_migrate.c+419−5
@@ -6,7 +6,7 @@ · 5 unmodified lines
66 #include <kunit/test.h>
77 #include <kunit/visibility.h>
88
9−#include "tests/xe_migrate_test.h"
9+#include "tests/xe_kunit_helpers.h"
1010 #include "tests/xe_pci_test.h"
1111
1212 #include "xe_pci.h"
@@ -334,7 +334,7 @@ vunmap: · 321 unmodified lines
334334
335335 static int migrate_test_run_device(struct xe_device *xe)
336336 {
337− struct kunit *test = xe_cur_kunit();
337+ struct kunit *test = kunit_get_current_test();
338338 struct xe_tile *tile;
339339 int id;
340340
@@ -354,8 +354,422 @@ static int migrate_test_run_device(struct xe_device *xe) · 13 unmodified lines
354354 return 0;
355355 }
356356
357−void xe_migrate_sanity_kunit(struct kunit *test)
357+static void xe_migrate_sanity_kunit(struct kunit *test)
358358 {
359− xe_call_for_each_device(migrate_test_run_device);
359+ struct xe_device *xe = test->priv;
360+
361+ migrate_test_run_device(xe);
362+}
363+
364+static struct dma_fence *blt_copy(struct xe_tile *tile,
365+ struct xe_bo *src_bo, struct xe_bo *dst_bo,
366+ bool copy_only_ccs, const char *str, struct kunit *test)
367+{
368+ struct xe_gt *gt = tile->primary_gt;
369+ struct xe_migrate *m = tile->migrate;
370+ struct xe_device *xe = gt_to_xe(gt);
371+ struct dma_fence *fence = NULL;
372+ u64 size = src_bo->size;
373+ struct xe_res_cursor src_it, dst_it;
374+ struct ttm_resource *src = src_bo->ttm.resource, *dst = dst_bo->ttm.resource;
375+ u64 src_L0_ofs, dst_L0_ofs;
376+ u32 src_L0_pt, dst_L0_pt;
377+ u64 src_L0, dst_L0;
378+ int err;
379+ bool src_is_vram = mem_type_is_vram(src->mem_type);
380+ bool dst_is_vram = mem_type_is_vram(dst->mem_type);
381+
382+ if (!src_is_vram)
383+ xe_res_first_sg(xe_bo_sg(src_bo), 0, size, &src_it);
384+ else
385+ xe_res_first(src, 0, size, &src_it);
386+
387+ if (!dst_is_vram)
388+ xe_res_first_sg(xe_bo_sg(dst_bo), 0, size, &dst_it);
389+ else
390+ xe_res_first(dst, 0, size, &dst_it);
391+
392+ while (size) {
393+ u32 batch_size = 2; /* arb_clear() + MI_BATCH_BUFFER_END */
394+ struct xe_sched_job *job;
395+ struct xe_bb *bb;
396+ u32 flush_flags = 0;
397+ u32 update_idx;
398+ u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
399+ u32 pte_flags;
400+
401+ src_L0 = xe_migrate_res_sizes(m, &src_it);
402+ dst_L0 = xe_migrate_res_sizes(m, &dst_it);
403+
404+ src_L0 = min(src_L0, dst_L0);
405+
406+ pte_flags = src_is_vram ? (PTE_UPDATE_FLAG_IS_VRAM |
407+ PTE_UPDATE_FLAG_IS_COMP_PTE) : 0;
408+ batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
409+ &src_L0_ofs, &src_L0_pt, 0, 0,
410+ avail_pts);
411+
412+ pte_flags = dst_is_vram ? (PTE_UPDATE_FLAG_IS_VRAM |
413+ PTE_UPDATE_FLAG_IS_COMP_PTE) : 0;
414+ batch_size += pte_update_size(m, pte_flags, dst, &dst_it, &src_L0,
415+ &dst_L0_ofs, &dst_L0_pt, 0,
416+ avail_pts, avail_pts);
417+
418+ /* Add copy commands size here */
419+ batch_size += ((copy_only_ccs) ? 0 : EMIT_COPY_DW) +
420+ ((xe_device_has_flat_ccs(xe) && copy_only_ccs) ? EMIT_COPY_CCS_DW : 0);
421+
422+ bb = xe_bb_new(gt, batch_size, xe->info.has_usm);
423+ if (IS_ERR(bb)) {
424+ err = PTR_ERR(bb);
425+ goto err_sync;
426+ }
427+
428+ if (src_is_vram)
429+ xe_res_next(&src_it, src_L0);
430+ else
431+ emit_pte(m, bb, src_L0_pt, src_is_vram, false,
432+ &src_it, src_L0, src);
433+
434+ if (dst_is_vram)
435+ xe_res_next(&dst_it, src_L0);
436+ else
437+ emit_pte(m, bb, dst_L0_pt, dst_is_vram, false,
438+ &dst_it, src_L0, dst);
439+
440+ bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
441+ update_idx = bb->len;
442+ if (!copy_only_ccs)
443+ emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
444+
445+ if (copy_only_ccs)
446+ flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
447+ src_is_vram, dst_L0_ofs,
448+ dst_is_vram, src_L0, dst_L0_ofs,
449+ copy_only_ccs);
450+
451+ job = xe_bb_create_migration_job(m->q, bb,
452+ xe_migrate_batch_base(m, xe->info.has_usm),
453+ update_idx);
454+ if (IS_ERR(job)) {
455+ err = PTR_ERR(job);
456+ goto err;
457+ }
458+
459+ xe_sched_job_add_migrate_flush(job, flush_flags);
460+
461+ mutex_lock(&m->job_mutex);
462+ xe_sched_job_arm(job);
463+ dma_fence_put(fence);
464+ fence = dma_fence_get(&job->drm.s_fence->finished);
465+ xe_sched_job_push(job);
466+
467+ dma_fence_put(m->fence);
468+ m->fence = dma_fence_get(fence);
469+
470+ mutex_unlock(&m->job_mutex);
471+
472+ xe_bb_free(bb, fence);
473+ size -= src_L0;
474+ continue;
475+
476+err:
477+ xe_bb_free(bb, NULL);
478+
479+err_sync:
480+ if (fence) {
481+ dma_fence_wait(fence, false);
482+ dma_fence_put(fence);
483+ }
484+ return ERR_PTR(err);
485+ }
486+
487+ return fence;
488+}
489+
490+static void test_migrate(struct xe_device *xe, struct xe_tile *tile,
491+ struct xe_bo *sys_bo, struct xe_bo *vram_bo, struct xe_bo *ccs_bo,
492+ struct kunit *test)
493+{
494+ struct dma_fence *fence;
495+ u64 expected, retval;
496+ long timeout;
497+ long ret;
498+
499+ expected = 0xd0d0d0d0d0d0d0d0;
500+ xe_map_memset(xe, &sys_bo->vmap, 0, 0xd0, sys_bo->size);
501+
502+ fence = blt_copy(tile, sys_bo, vram_bo, false, "Blit copy from sysmem to vram", test);
503+ if (!sanity_fence_failed(xe, fence, "Blit copy from sysmem to vram", test)) {
504+ retval = xe_map_rd(xe, &vram_bo->vmap, 0, u64);
505+ if (retval == expected)
506+ KUNIT_FAIL(test, "Sanity check failed: VRAM must have compressed value\n");
507+ }
508+ dma_fence_put(fence);
509+
510+ kunit_info(test, "Evict vram buffer object\n");
511+ ret = xe_bo_evict(vram_bo, true);
512+ if (ret) {
513+ KUNIT_FAIL(test, "Failed to evict bo.\n");
514+ return;
515+ }
516+
517+ ret = xe_bo_vmap(vram_bo);
518+ if (ret) {
519+ KUNIT_FAIL(test, "Failed to vmap vram bo: %li\n", ret);
520+ return;
521+ }
522+
523+ retval = xe_map_rd(xe, &vram_bo->vmap, 0, u64);
524+ check(retval, expected, "Clear evicted vram data first value", test);
525+ retval = xe_map_rd(xe, &vram_bo->vmap, vram_bo->size - 8, u64);
526+ check(retval, expected, "Clear evicted vram data last value", test);
527+
528+ fence = blt_copy(tile, vram_bo, ccs_bo,
529+ true, "Blit surf copy from vram to sysmem", test);
530+ if (!sanity_fence_failed(xe, fence, "Clear ccs buffer data", test)) {
531+ retval = xe_map_rd(xe, &ccs_bo->vmap, 0, u64);
532+ check(retval, 0, "Clear ccs data first value", test);
533+
534+ retval = xe_map_rd(xe, &ccs_bo->vmap, ccs_bo->size - 8, u64);
535+ check(retval, 0, "Clear ccs data last value", test);
536+ }
537+ dma_fence_put(fence);
538+
539+ kunit_info(test, "Restore vram buffer object\n");
540+ ret = xe_bo_validate(vram_bo, NULL, false);
541+ if (ret) {
542+ KUNIT_FAIL(test, "Failed to validate vram bo for: %li\n", ret);
543+ return;
544+ }
545+
546+ /* Sync all migration blits */
547+ timeout = dma_resv_wait_timeout(vram_bo->ttm.base.resv,
548+ DMA_RESV_USAGE_KERNEL,
549+ true,
550+ 5 * HZ);
551+ if (timeout <= 0) {
552+ KUNIT_FAIL(test, "Failed to sync bo eviction.\n");
553+ return;
554+ }
555+
556+ ret = xe_bo_vmap(vram_bo);
557+ if (ret) {
558+ KUNIT_FAIL(test, "Failed to vmap vram bo: %li\n", ret);
559+ return;
560+ }
561+
562+ retval = xe_map_rd(xe, &vram_bo->vmap, 0, u64);
563+ check(retval, expected, "Restored value must be equal to initial value", test);
564+ retval = xe_map_rd(xe, &vram_bo->vmap, vram_bo->size - 8, u64);
565+ check(retval, expected, "Restored value must be equal to initial value", test);
566+
567+ fence = blt_copy(tile, vram_bo, ccs_bo,
568+ true, "Blit surf copy from vram to sysmem", test);
569+ if (!sanity_fence_failed(xe, fence, "Clear ccs buffer data", test)) {
570+ retval = xe_map_rd(xe, &ccs_bo->vmap, 0, u64);
571+ check(retval, 0, "Clear ccs data first value", test);
572+ retval = xe_map_rd(xe, &ccs_bo->vmap, ccs_bo->size - 8, u64);
573+ check(retval, 0, "Clear ccs data last value", test);
574+ }
575+ dma_fence_put(fence);
576+}
577+
578+static void test_clear(struct xe_device *xe, struct xe_tile *tile,
579+ struct xe_bo *sys_bo, struct xe_bo *vram_bo, struct kunit *test)
580+{
581+ struct dma_fence *fence;
582+ u64 expected, retval;
583+
584+ expected = 0xd0d0d0d0d0d0d0d0;
585+ xe_map_memset(xe, &sys_bo->vmap, 0, 0xd0, sys_bo->size);
586+
587+ fence = blt_copy(tile, sys_bo, vram_bo, false, "Blit copy from sysmem to vram", test);
588+ if (!sanity_fence_failed(xe, fence, "Blit copy from sysmem to vram", test)) {
589+ retval = xe_map_rd(xe, &vram_bo->vmap, 0, u64);
590+ if (retval == expected)
591+ KUNIT_FAIL(test, "Sanity check failed: VRAM must have compressed value\n");
592+ }
593+ dma_fence_put(fence);
594+
595+ fence = blt_copy(tile, vram_bo, sys_bo, false, "Blit copy from vram to sysmem", test);
596+ if (!sanity_fence_failed(xe, fence, "Blit copy from vram to sysmem", test)) {
597+ retval = xe_map_rd(xe, &sys_bo->vmap, 0, u64);
598+ check(retval, expected, "Decompressed value must be equal to initial value", test);
599+ retval = xe_map_rd(xe, &sys_bo->vmap, sys_bo->size - 8, u64);
600+ check(retval, expected, "Decompressed value must be equal to initial value", test);
601+ }
602+ dma_fence_put(fence);
603+
604+ kunit_info(test, "Clear vram buffer object\n");
605+ expected = 0x0000000000000000;
606+ fence = xe_migrate_clear(tile->migrate, vram_bo, vram_bo->ttm.resource);
607+ if (sanity_fence_failed(xe, fence, "Clear vram_bo", test))
608+ return;
609+ dma_fence_put(fence);
610+
611+ fence = blt_copy(tile, vram_bo, sys_bo,
612+ false, "Blit copy from vram to sysmem", test);
613+ if (!sanity_fence_failed(xe, fence, "Clear main buffer data", test)) {
614+ retval = xe_map_rd(xe, &sys_bo->vmap, 0, u64);
615+ check(retval, expected, "Clear main buffer first value", test);
616+ retval = xe_map_rd(xe, &sys_bo->vmap, sys_bo->size - 8, u64);
617+ check(retval, expected, "Clear main buffer last value", test);
618+ }
619+ dma_fence_put(fence);
620+
621+ fence = blt_copy(tile, vram_bo, sys_bo,
622+ true, "Blit surf copy from vram to sysmem", test);
623+ if (!sanity_fence_failed(xe, fence, "Clear ccs buffer data", test)) {
624+ retval = xe_map_rd(xe, &sys_bo->vmap, 0, u64);
625+ check(retval, expected, "Clear ccs data first value", test);
626+ retval = xe_map_rd(xe, &sys_bo->vmap, sys_bo->size - 8, u64);
627+ check(retval, expected, "Clear ccs data last value", test);
628+ }
629+ dma_fence_put(fence);
630+}
631+
632+static void validate_ccs_test_run_tile(struct xe_device *xe, struct xe_tile *tile,
633+ struct kunit *test)
634+{
635+ struct xe_bo *sys_bo, *vram_bo = NULL, *ccs_bo = NULL;
636+ unsigned int bo_flags = XE_BO_FLAG_VRAM_IF_DGFX(tile);
637+ long ret;
638+
639+ sys_bo = xe_bo_create_user(xe, NULL, NULL, SZ_4M,
640+ DRM_XE_GEM_CPU_CACHING_WC, ttm_bo_type_device,
641+ XE_BO_FLAG_SYSTEM | XE_BO_FLAG_NEEDS_CPU_ACCESS);
642+
643+ if (IS_ERR(sys_bo)) {
644+ KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
645+ PTR_ERR(sys_bo));
646+ return;
647+ }
648+
649+ xe_bo_lock(sys_bo, false);
650+ ret = xe_bo_validate(sys_bo, NULL, false);
651+ if (ret) {
652+ KUNIT_FAIL(test, "Failed to validate system bo for: %li\n", ret);
653+ goto free_sysbo;
654+ }
655+
656+ ret = xe_bo_vmap(sys_bo);
657+ if (ret) {
658+ KUNIT_FAIL(test, "Failed to vmap system bo: %li\n", ret);
659+ goto free_sysbo;
660+ }
661+ xe_bo_unlock(sys_bo);
662+
663+ ccs_bo = xe_bo_create_user(xe, NULL, NULL, SZ_4M, DRM_XE_GEM_CPU_CACHING_WC,
664+ ttm_bo_type_device, bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS);
665+
666+ if (IS_ERR(ccs_bo)) {
667+ KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
668+ PTR_ERR(ccs_bo));
669+ return;
670+ }
671+
672+ xe_bo_lock(ccs_bo, false);
673+ ret = xe_bo_validate(ccs_bo, NULL, false);
674+ if (ret) {
675+ KUNIT_FAIL(test, "Failed to validate system bo for: %li\n", ret);
676+ goto free_ccsbo;
677+ }
678+
679+ ret = xe_bo_vmap(ccs_bo);
680+ if (ret) {
681+ KUNIT_FAIL(test, "Failed to vmap system bo: %li\n", ret);
682+ goto free_ccsbo;
683+ }
684+ xe_bo_unlock(ccs_bo);
685+
686+ vram_bo = xe_bo_create_user(xe, NULL, NULL, SZ_4M, DRM_XE_GEM_CPU_CACHING_WC,
687+ ttm_bo_type_device, bo_flags | XE_BO_FLAG_NEEDS_CPU_ACCESS);
688+ if (IS_ERR(vram_bo)) {
689+ KUNIT_FAIL(test, "xe_bo_create() failed with err=%ld\n",
690+ PTR_ERR(vram_bo));
691+ return;
692+ }
693+
694+ xe_bo_lock(vram_bo, false);
695+ ret = xe_bo_validate(vram_bo, NULL, false);
696+ if (ret) {
697+ KUNIT_FAIL(test, "Failed to validate vram bo for: %li\n", ret);
698+ goto free_vrambo;
699+ }
700+
701+ ret = xe_bo_vmap(vram_bo);
702+ if (ret) {
703+ KUNIT_FAIL(test, "Failed to vmap vram bo: %li\n", ret);
704+ goto free_vrambo;
705+ }
706+
707+ test_clear(xe, tile, sys_bo, vram_bo, test);
708+ test_migrate(xe, tile, sys_bo, vram_bo, ccs_bo, test);
709+ xe_bo_unlock(vram_bo);
710+
711+ xe_bo_lock(vram_bo, false);
712+ xe_bo_vunmap(vram_bo);
713+ xe_bo_unlock(vram_bo);
714+
715+ xe_bo_lock(ccs_bo, false);
716+ xe_bo_vunmap(ccs_bo);
717+ xe_bo_unlock(ccs_bo);
718+
719+ xe_bo_lock(sys_bo, false);
720+ xe_bo_vunmap(sys_bo);
721+ xe_bo_unlock(sys_bo);
722+free_vrambo:
723+ xe_bo_put(vram_bo);
724+free_ccsbo:
725+ xe_bo_put(ccs_bo);
726+free_sysbo:
727+ xe_bo_put(sys_bo);
728+}
729+
730+static int validate_ccs_test_run_device(struct xe_device *xe)
731+{
732+ struct kunit *test = kunit_get_current_test();
733+ struct xe_tile *tile;
734+ int id;
735+
736+ if (!xe_device_has_flat_ccs(xe)) {
737+ kunit_skip(test, "non-flat-ccs device\n");
738+ return 0;
739+ }
740+
741+ if (!(GRAPHICS_VER(xe) >= 20 && IS_DGFX(xe))) {
742+ kunit_skip(test, "non-xe2 discrete device\n");
743+ return 0;
744+ }
745+
746+ xe_pm_runtime_get(xe);
747+
748+ for_each_tile(tile, xe, id)
749+ validate_ccs_test_run_tile(xe, tile, test);
750+
751+ xe_pm_runtime_put(xe);
752+
753+ return 0;
360754 }
361−EXPORT_SYMBOL_IF_KUNIT(xe_migrate_sanity_kunit);
755+
756+static void xe_validate_ccs_kunit(struct kunit *test)
757+{
758+ struct xe_device *xe = test->priv;
759+
760+ validate_ccs_test_run_device(xe);
761+}
762+
763+static struct kunit_case xe_migrate_tests[] = {
764+ KUNIT_CASE_PARAM(xe_migrate_sanity_kunit, xe_pci_live_device_gen_param),
765+ KUNIT_CASE_PARAM(xe_validate_ccs_kunit, xe_pci_live_device_gen_param),
766+ {}
767+};
768+
769+VISIBLE_IF_KUNIT
770+struct kunit_suite xe_migrate_test_suite = {
771+ .name = "xe_migrate",
772+ .test_cases = xe_migrate_tests,
773+ .init = xe_kunit_helper_xe_device_live_test_init,
774+};
775+EXPORT_SYMBOL_IF_KUNIT(xe_migrate_test_suite);
drivers/gpu/drm/xe/tests/xe_migrate_test.c+0−20
@@ -1,20 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0
2−/*
3− * Copyright © 2022 Intel Corporation
4− */
5−
6−#include "xe_migrate_test.h"
7−
8−#include <kunit/test.h>
9−
10−static struct kunit_case xe_migrate_tests[] = {
11− KUNIT_CASE(xe_migrate_sanity_kunit),
12− {}
13−};
14−
15−static struct kunit_suite xe_migrate_test_suite = {
16− .name = "xe_migrate",
17− .test_cases = xe_migrate_tests,
18−};
19−
20−kunit_test_suite(xe_migrate_test_suite);
drivers/gpu/drm/xe/tests/xe_migrate_test.h+0−13
@@ -1,13 +0,0 @@
1−/* SPDX-License-Identifier: GPL-2.0 AND MIT */
2−/*
3− * Copyright © 2023 Intel Corporation
4− */
5−
6−#ifndef _XE_MIGRATE_TEST_H_
7−#define _XE_MIGRATE_TEST_H_
8−
9−struct kunit;
10−
11−void xe_migrate_sanity_kunit(struct kunit *test);
12−
13−#endif
drivers/gpu/drm/xe/tests/xe_mocs.c+33−11
@@ -6,7 +6,7 @@ · 5 unmodified lines
66 #include <kunit/test.h>
77 #include <kunit/visibility.h>
88
9−#include "tests/xe_mocs_test.h"
9+#include "tests/xe_kunit_helpers.h"
1010 #include "tests/xe_pci_test.h"
1111 #include "tests/xe_test.h"
1212
@@ -23,7 +23,7 @@ struct live_mocs { · 10 unmodified lines
2323 static int live_mocs_init(struct live_mocs *arg, struct xe_gt *gt)
2424 {
2525 unsigned int flags;
26− struct kunit *test = xe_cur_kunit();
26+ struct kunit *test = kunit_get_current_test();
2727
2828 memset(arg, 0, sizeof(*arg));
2929
@@ -41,7 +41,7 @@ static int live_mocs_init(struct live_mocs *arg, struct xe_gt *gt) · 11 unmodified lines
4141 static void read_l3cc_table(struct xe_gt *gt,
4242 const struct xe_mocs_info *info)
4343 {
44− struct kunit *test = xe_cur_kunit();
44+ struct kunit *test = kunit_get_current_test();
4545 u32 l3cc, l3cc_expected;
4646 unsigned int i;
4747 u32 reg_val;
@@ -78,7 +78,7 @@ static void read_l3cc_table(struct xe_gt *gt, · 30 unmodified lines
7878 static void read_mocs_table(struct xe_gt *gt,
7979 const struct xe_mocs_info *info)
8080 {
81− struct kunit *test = xe_cur_kunit();
81+ struct kunit *test = kunit_get_current_test();
8282 u32 mocs, mocs_expected;
8383 unsigned int i;
8484 u32 reg_val;
@@ -134,11 +134,15 @@ static int mocs_kernel_test_run_device(struct xe_device *xe) · 49 unmodified lines
134134 return 0;
135135 }
136136
137−void xe_live_mocs_kernel_kunit(struct kunit *test)
137+static void xe_live_mocs_kernel_kunit(struct kunit *test)
138138 {
139− xe_call_for_each_device(mocs_kernel_test_run_device);
139+ struct xe_device *xe = test->priv;
140+
141+ if (IS_SRIOV_VF(xe))
142+ kunit_skip(test, "this test is N/A for VF");
143+
144+ mocs_kernel_test_run_device(xe);
140145 }
141−EXPORT_SYMBOL_IF_KUNIT(xe_live_mocs_kernel_kunit);
142146
143147 static int mocs_reset_test_run_device(struct xe_device *xe)
144148 {
@@ -148,7 +152,7 @@ static int mocs_reset_test_run_device(struct xe_device *xe) · 3 unmodified lines
148152 struct xe_gt *gt;
149153 unsigned int flags;
150154 int id;
151− struct kunit *test = xe_cur_kunit();
155+ struct kunit *test = kunit_get_current_test();
152156
153157 xe_pm_runtime_get(xe);
154158
@@ -175,8 +179,26 @@ static int mocs_reset_test_run_device(struct xe_device *xe) · 20 unmodified lines
175179 return 0;
176180 }
177181
178−void xe_live_mocs_reset_kunit(struct kunit *test)
182+static void xe_live_mocs_reset_kunit(struct kunit *test)
179183 {
180− xe_call_for_each_device(mocs_reset_test_run_device);
184+ struct xe_device *xe = test->priv;
185+
186+ if (IS_SRIOV_VF(xe))
187+ kunit_skip(test, "this test is N/A for VF");
188+
189+ mocs_reset_test_run_device(xe);
181190 }
182−EXPORT_SYMBOL_IF_KUNIT(xe_live_mocs_reset_kunit);
191+
192+static struct kunit_case xe_mocs_tests[] = {
193+ KUNIT_CASE_PARAM(xe_live_mocs_kernel_kunit, xe_pci_live_device_gen_param),
194+ KUNIT_CASE_PARAM(xe_live_mocs_reset_kunit, xe_pci_live_device_gen_param),
195+ {}
196+};
197+
198+VISIBLE_IF_KUNIT
199+struct kunit_suite xe_mocs_test_suite = {
200+ .name = "xe_mocs",
201+ .test_cases = xe_mocs_tests,
202+ .init = xe_kunit_helper_xe_device_live_test_init,
203+};
204+EXPORT_SYMBOL_IF_KUNIT(xe_mocs_test_suite);
drivers/gpu/drm/xe/tests/xe_mocs_test.c+0−21
@@ -1,21 +0,0 @@
1−// SPDX-License-Identifier: GPL-2.0
2−/*
3− * Copyright © 2022 Intel Corporation
4− */
5−
6−#include "xe_mocs_test.h"
7−
8−#include <kunit/test.h>
9−
10−static struct kunit_case xe_mocs_tests[] = {
11− KUNIT_CASE(xe_live_mocs_kernel_kunit),
12− KUNIT_CASE(xe_live_mocs_reset_kunit),
13− {}
14−};
15−
16−static struct kunit_suite xe_mocs_test_suite = {
17− .name = "xe_mocs",
18− .test_cases = xe_mocs_tests,
19−};
20−
21−kunit_test_suite(xe_mocs_test_suite);
drivers/gpu/drm/xe/tests/xe_mocs_test.h+0−14
@@ -1,14 +0,0 @@
1−/* SPDX-License-Identifier: GPL-2.0 AND MIT */
2−/*
3− * Copyright © 2023 Intel Corporation
4− */
5−
6−#ifndef _XE_MOCS_TEST_H_
7−#define _XE_MOCS_TEST_H_
8−
9−struct kunit;
10−
11−void xe_live_mocs_kernel_kunit(struct kunit *test);
12−void xe_live_mocs_reset_kunit(struct kunit *test);
13−
14−#endif
drivers/gpu/drm/xe/tests/xe_pci.c+30−0
@@ -167,3 +167,33 @@ done: · 166 unmodified lines
167167 return 0;
168168 }
169169 EXPORT_SYMBOL_IF_KUNIT(xe_pci_fake_device_init);
170+
171+/**
172+ * xe_pci_live_device_gen_param - Helper to iterate Xe devices as KUnit parameters
173+ * @prev: the previously returned value, or NULL for the first iteration
174+ * @desc: the buffer for a parameter name
175+ *
176+ * Iterates over the available Xe devices on the system. Uses the device name
177+ * as the parameter name.
178+ *
179+ * To be used only as a parameter generator function in &KUNIT_CASE_PARAM.
180+ *
181+ * Return: pointer to the next &struct xe_device ready to be used as a parameter
182+ * or NULL if there are no more Xe devices on the system.
183+ */
184+const void *xe_pci_live_device_gen_param(const void *prev, char *desc)
185+{
186+ const struct xe_device *xe = prev;
187+ struct device *dev = xe ? xe->drm.dev : NULL;
188+ struct device *next;
189+
190+ next = driver_find_next_device(&xe_pci_driver.driver, dev);
191+ if (dev)
192+ put_device(dev);
193+ if (!next)
194+ return NULL;
195+
196+ snprintf(desc, KUNIT_PARAM_DESC_SIZE, "%s", dev_name(next));
197+ return pdev_to_xe_device(to_pci_dev(next));
198+}
199+EXPORT_SYMBOL_IF_KUNIT(xe_pci_live_device_gen_param);
drivers/gpu/drm/xe/tests/xe_pci_test.c+2−2
@@ -16,7 +16,7 @@ · 15 unmodified lines
1616
1717 static void check_graphics_ip(const struct xe_graphics_desc *graphics)
1818 {
19− struct kunit *test = xe_cur_kunit();
19+ struct kunit *test = kunit_get_current_test();
2020 u64 mask = graphics->hw_engine_mask;
2121
2222 /* RCS, CCS, and BCS engines are allowed on the graphics IP */
@@ -30,7 +30,7 @@ static void check_graphics_ip(const struct xe_graphics_desc *graphics) · 7 unmodified lines
3030
3131 static void check_media_ip(const struct xe_media_desc *media)
3232 {
33− struct kunit *test = xe_cur_kunit();
33+ struct kunit *test = kunit_get_current_test();
3434 u64 mask = media->hw_engine_mask;
3535
3636 /* VCS, VECS and GSCCS engines are allowed on the media IP */
drivers/gpu/drm/xe/tests/xe_pci_test.h+2−0
@@ -35,4 +35,6 @@ struct xe_pci_fake_data { · 34 unmodified lines
3535
3636 int xe_pci_fake_device_init(struct xe_device *xe);
3737
38+const void *xe_pci_live_device_gen_param(const void *prev, char *desc);
39+
3840 #endif
drivers/gpu/drm/xe/tests/xe_rtp_test.c+197−22
@@ -31,16 +31,23 @@ · 30 unmodified lines
3131 #undef XE_REG_MCR
3232 #define XE_REG_MCR(...) XE_REG(__VA_ARGS__, .mcr = 1)
3333
34−struct rtp_test_case {
34+struct rtp_to_sr_test_case {
3535 const char *name;
3636 struct xe_reg expected_reg;
3737 u32 expected_set_bits;
3838 u32 expected_clr_bits;
39− unsigned long expected_count;
39+ unsigned long expected_count_sr_entries;
4040 unsigned int expected_sr_errors;
41+ unsigned long expected_active;
4142 const struct xe_rtp_entry_sr *entries;
4243 };
4344
45+struct rtp_test_case {
46+ const char *name;
47+ unsigned long expected_active;
48+ const struct xe_rtp_entry *entries;
49+};
50+
4451 static bool match_yes(const struct xe_gt *gt, const struct xe_hw_engine *hwe)
4552 {
4653 return true;
@@ -51,13 +58,14 @@ static bool match_no(const struct xe_gt *gt, const struct xe_hw_engine *hwe) · 4 unmodified lines
5158 return false;
5259 }
5360
54−static const struct rtp_test_case cases[] = {
61+static const struct rtp_to_sr_test_case rtp_to_sr_cases[] = {
5562 {
5663 .name = "coalesce-same-reg",
5764 .expected_reg = REGULAR_REG1,
5865 .expected_set_bits = REG_BIT(0) | REG_BIT(1),
5966 .expected_clr_bits = REG_BIT(0) | REG_BIT(1),
60− .expected_count = 1,
67+ .expected_active = BIT(0) | BIT(1),
68+ .expected_count_sr_entries = 1,
6169 /* Different bits on the same register: create a single entry */
6270 .entries = (const struct xe_rtp_entry_sr[]) {
6371 { XE_RTP_NAME("basic-1"),
@@ -76,7 +84,8 @@ static const struct rtp_test_case cases[] = { · 12 unmodified lines
7684 .expected_reg = REGULAR_REG1,
7785 .expected_set_bits = REG_BIT(0),
7886 .expected_clr_bits = REG_BIT(0),
79− .expected_count = 1,
87+ .expected_active = BIT(0),
88+ .expected_count_sr_entries = 1,
8089 /* Don't coalesce second entry since rules don't match */
8190 .entries = (const struct xe_rtp_entry_sr[]) {
8291 { XE_RTP_NAME("basic-1"),
@@ -95,7 +104,8 @@ static const struct rtp_test_case cases[] = { · 12 unmodified lines
95104 .expected_reg = REGULAR_REG1,
96105 .expected_set_bits = REG_BIT(0) | REG_BIT(1) | REG_BIT(2),
97106 .expected_clr_bits = REG_BIT(0) | REG_BIT(1) | REG_BIT(2),
98− .expected_count = 1,
107+ .expected_active = BIT(0) | BIT(1) | BIT(2),
108+ .expected_count_sr_entries = 1,
99109 .entries = (const struct xe_rtp_entry_sr[]) {
100110 { XE_RTP_NAME("first"),
101111 XE_RTP_RULES(FUNC(match_yes), OR, FUNC(match_no)),
@@ -121,7 +131,7 @@ static const struct rtp_test_case cases[] = { · 19 unmodified lines
121131 {
122132 .name = "match-or-xfail",
123133 .expected_reg = REGULAR_REG1,
124− .expected_count = 0,
134+ .expected_count_sr_entries = 0,
125135 .entries = (const struct xe_rtp_entry_sr[]) {
126136 { XE_RTP_NAME("leading-or"),
127137 XE_RTP_RULES(OR, FUNC(match_yes)),
@@ -148,7 +158,8 @@ static const struct rtp_test_case cases[] = { · 20 unmodified lines
148158 .expected_reg = REGULAR_REG1,
149159 .expected_set_bits = REG_BIT(0),
150160 .expected_clr_bits = REG_BIT(0),
151− .expected_count = 1,
161+ .expected_active = BIT(0),
162+ .expected_count_sr_entries = 1,
152163 /* Don't coalesce second entry due to one of the rules */
153164 .entries = (const struct xe_rtp_entry_sr[]) {
154165 { XE_RTP_NAME("basic-1"),
@@ -167,7 +178,8 @@ static const struct rtp_test_case cases[] = { · 12 unmodified lines
167178 .expected_reg = REGULAR_REG1,
168179 .expected_set_bits = REG_BIT(0),
169180 .expected_clr_bits = REG_BIT(0),
170− .expected_count = 2,
181+ .expected_active = BIT(0) | BIT(1),
182+ .expected_count_sr_entries = 2,
171183 /* Same bits on different registers are not coalesced */
172184 .entries = (const struct xe_rtp_entry_sr[]) {
173185 { XE_RTP_NAME("basic-1"),
@@ -186,7 +198,8 @@ static const struct rtp_test_case cases[] = { · 12 unmodified lines
186198 .expected_reg = REGULAR_REG1,
187199 .expected_set_bits = REG_BIT(0),
188200 .expected_clr_bits = REG_BIT(1) | REG_BIT(0),
189− .expected_count = 1,
201+ .expected_active = BIT(0) | BIT(1),
202+ .expected_count_sr_entries = 1,
190203 /* Check clr vs set actions on different bits */
191204 .entries = (const struct xe_rtp_entry_sr[]) {
192205 { XE_RTP_NAME("basic-1"),
@@ -207,7 +220,8 @@ static const struct rtp_test_case cases[] = { · 14 unmodified lines
207220 .expected_reg = REGULAR_REG1,
208221 .expected_set_bits = TEMP_FIELD,
209222 .expected_clr_bits = TEMP_MASK,
210− .expected_count = 1,
223+ .expected_active = BIT(0),
224+ .expected_count_sr_entries = 1,
211225 /* Check FIELD_SET works */
212226 .entries = (const struct xe_rtp_entry_sr[]) {
213227 { XE_RTP_NAME("basic-1"),
@@ -225,7 +239,8 @@ static const struct rtp_test_case cases[] = { · 11 unmodified lines
225239 .expected_reg = REGULAR_REG1,
226240 .expected_set_bits = REG_BIT(0),
227241 .expected_clr_bits = REG_BIT(0),
228− .expected_count = 1,
242+ .expected_active = BIT(0) | BIT(1),
243+ .expected_count_sr_entries = 1,
229244 .expected_sr_errors = 1,
230245 .entries = (const struct xe_rtp_entry_sr[]) {
231246 { XE_RTP_NAME("basic-1"),
@@ -245,7 +260,8 @@ static const struct rtp_test_case cases[] = { · 13 unmodified lines
245260 .expected_reg = REGULAR_REG1,
246261 .expected_set_bits = REG_BIT(0),
247262 .expected_clr_bits = REG_BIT(0),
248− .expected_count = 1,
263+ .expected_active = BIT(0) | BIT(1),
264+ .expected_count_sr_entries = 1,
249265 .expected_sr_errors = 1,
250266 .entries = (const struct xe_rtp_entry_sr[]) {
251267 { XE_RTP_NAME("basic-1"),
@@ -265,7 +281,8 @@ static const struct rtp_test_case cases[] = { · 13 unmodified lines
265281 .expected_reg = REGULAR_REG1,
266282 .expected_set_bits = REG_BIT(0),
267283 .expected_clr_bits = REG_BIT(0),
268− .expected_count = 1,
284+ .expected_active = BIT(0) | BIT(1) | BIT(2),
285+ .expected_count_sr_entries = 1,
269286 .expected_sr_errors = 2,
270287 .entries = (const struct xe_rtp_entry_sr[]) {
271288 { XE_RTP_NAME("basic-1"),
@@ -287,28 +304,35 @@ static const struct rtp_test_case cases[] = { · 15 unmodified lines
287304 },
288305 };
289306
290−static void xe_rtp_process_tests(struct kunit *test)
307+static void xe_rtp_process_to_sr_tests(struct kunit *test)
291308 {
292− const struct rtp_test_case *param = test->param_value;
309+ const struct rtp_to_sr_test_case *param = test->param_value;
293310 struct xe_device *xe = test->priv;
294311 struct xe_gt *gt = xe_device_get_root_tile(xe)->primary_gt;
295312 struct xe_reg_sr *reg_sr = &gt->reg_sr;
296313 const struct xe_reg_sr_entry *sre, *sr_entry = NULL;
297314 struct xe_rtp_process_ctx ctx = XE_RTP_PROCESS_CTX_INITIALIZER(gt);
298− unsigned long idx, count = 0;
315+ unsigned long idx, count_sr_entries = 0, count_rtp_entries = 0, active = 0;
316+
317+ xe_reg_sr_init(reg_sr, "xe_rtp_to_sr_tests", xe);
318+
319+ while (param->entries[count_rtp_entries].rules)
320+ count_rtp_entries++;
299321
300− xe_reg_sr_init(reg_sr, "xe_rtp_tests", xe);
322+ xe_rtp_process_ctx_enable_active_tracking(&ctx, &active, count_rtp_entries);
301323 xe_rtp_process_to_sr(&ctx, param->entries, reg_sr);
302324
303325 xa_for_each(&reg_sr->xa, idx, sre) {
304326 if (idx == param->expected_reg.addr)
305327 sr_entry = sre;
306328
307− count++;
329+ count_sr_entries++;
308330 }
309331
310− KUNIT_EXPECT_EQ(test, count, param->expected_count);
311− if (count) {
332+ KUNIT_EXPECT_EQ(test, active, param->expected_active);
333+
334+ KUNIT_EXPECT_EQ(test, count_sr_entries, param->expected_count_sr_entries);
335+ if (count_sr_entries) {
312336 KUNIT_EXPECT_EQ(test, sr_entry->clr_bits, param->expected_clr_bits);
313337 KUNIT_EXPECT_EQ(test, sr_entry->set_bits, param->expected_set_bits);
314338 KUNIT_EXPECT_EQ(test, sr_entry->reg.raw, param->expected_reg.raw);
@@ -319,12 +343,162 @@ static void xe_rtp_process_tests(struct kunit *test) · 4 unmodified lines
319343 KUNIT_EXPECT_EQ(test, reg_sr->errors, param->expected_sr_errors);
320344 }
321345
346+/*
347+ * Entries below follow the logic used with xe_wa_oob.rules:
348+ * 1) Entries with empty name are OR'ed: all entries marked active since the
349+ * last entry with a name
350+ * 2) There are no action associated with rules
351+ */
352+static const struct rtp_test_case rtp_cases[] = {
353+ {
354+ .name = "active1",
355+ .expected_active = BIT(0),
356+ .entries = (const struct xe_rtp_entry[]) {
357+ { XE_RTP_NAME("r1"),
358+ XE_RTP_RULES(FUNC(match_yes)),
359+ },
360+ {}
361+ },
362+ },
363+ {
364+ .name = "active2",
365+ .expected_active = BIT(0) | BIT(1),
366+ .entries = (const struct xe_rtp_entry[]) {
367+ { XE_RTP_NAME("r1"),
368+ XE_RTP_RULES(FUNC(match_yes)),
369+ },
370+ { XE_RTP_NAME("r2"),
371+ XE_RTP_RULES(FUNC(match_yes)),
372+ },
373+ {}
374+ },
375+ },
376+ {
377+ .name = "active-inactive",
378+ .expected_active = BIT(0),
379+ .entries = (const struct xe_rtp_entry[]) {
380+ { XE_RTP_NAME("r1"),
381+ XE_RTP_RULES(FUNC(match_yes)),
382+ },
383+ { XE_RTP_NAME("r2"),
384+ XE_RTP_RULES(FUNC(match_no)),
385+ },
386+ {}
387+ },
388+ },
389+ {
390+ .name = "inactive-active",
391+ .expected_active = BIT(1),
392+ .entries = (const struct xe_rtp_entry[]) {
393+ { XE_RTP_NAME("r1"),
394+ XE_RTP_RULES(FUNC(match_no)),
395+ },
396+ { XE_RTP_NAME("r2"),
397+ XE_RTP_RULES(FUNC(match_yes)),
398+ },
399+ {}
400+ },
401+ },
402+ {
403+ .name = "inactive-1st_or_active-inactive",
404+ .expected_active = BIT(1),
405+ .entries = (const struct xe_rtp_entry[]) {
406+ { XE_RTP_NAME("r1"),
407+ XE_RTP_RULES(FUNC(match_no)),
408+ },
409+ { XE_RTP_NAME("r2_or_conditions"),
410+ XE_RTP_RULES(FUNC(match_yes), OR,
411+ FUNC(match_no), OR,
412+ FUNC(match_no)) },
413+ { XE_RTP_NAME("r3"),
414+ XE_RTP_RULES(FUNC(match_no)),
415+ },
416+ {}
417+ },
418+ },
419+ {
420+ .name = "inactive-2nd_or_active-inactive",
421+ .expected_active = BIT(1),
422+ .entries = (const struct xe_rtp_entry[]) {
423+ { XE_RTP_NAME("r1"),
424+ XE_RTP_RULES(FUNC(match_no)),
425+ },
426+ { XE_RTP_NAME("r2_or_conditions"),
427+ XE_RTP_RULES(FUNC(match_no), OR,
428+ FUNC(match_yes), OR,
429+ FUNC(match_no)) },
430+ { XE_RTP_NAME("r3"),
431+ XE_RTP_RULES(FUNC(match_no)),
432+ },
433+ {}
434+ },
435+ },
436+ {
437+ .name = "inactive-last_or_active-inactive",
438+ .expected_active = BIT(1),
439+ .entries = (const struct xe_rtp_entry[]) {
440+ { XE_RTP_NAME("r1"),
441+ XE_RTP_RULES(FUNC(match_no)),
442+ },
443+ { XE_RTP_NAME("r2_or_conditions"),
444+ XE_RTP_RULES(FUNC(match_no), OR,
445+ FUNC(match_no), OR,
446+ FUNC(match_yes)) },
447+ { XE_RTP_NAME("r3"),
448+ XE_RTP_RULES(FUNC(match_no)),
449+ },
450+ {}
451+ },
452+ },
453+ {
454+ .name = "inactive-no_or_active-inactive",
455+ .expected_active = 0,
456+ .entries = (const struct xe_rtp_entry[]) {
457+ { XE_RTP_NAME("r1"),
458+ XE_RTP_RULES(FUNC(match_no)),
459+ },
460+ { XE_RTP_NAME("r2_or_conditions"),
461+ XE_RTP_RULES(FUNC(match_no), OR,
462+ FUNC(match_no), OR,
463+ FUNC(match_no)) },
464+ { XE_RTP_NAME("r3"),
465+ XE_RTP_RULES(FUNC(match_no)),
466+ },
467+ {}
468+ },
469+ },
470+};
471+
472+static void xe_rtp_process_tests(struct kunit *test)
473+{
474+ const struct rtp_test_case *param = test->param_value;
475+ struct xe_device *xe = test->priv;
476+ struct xe_gt *gt = xe_device_get_root_tile(xe)->primary_gt;
477+ struct xe_rtp_process_ctx ctx = XE_RTP_PROCESS_CTX_INITIALIZER(gt);
478+ unsigned long count_rtp_entries = 0, active = 0;
479+
480+ while (param->entries[count_rtp_entries].rules)
481+ count_rtp_entries++;
482+
483+ xe_rtp_process_ctx_enable_active_tracking(&ctx, &active, count_rtp_entries);
484+ xe_rtp_process(&ctx, param->entries);
485+
486+ KUNIT_EXPECT_EQ(test, active, param->expected_active);
487+}
488+
489+static void rtp_to_sr_desc(const struct rtp_to_sr_test_case *t, char *desc)
490+{
491+ strscpy(desc, t->name, KUNIT_PARAM_DESC_SIZE);
492+}
493+
494+KUNIT_ARRAY_PARAM(rtp_to_sr, rtp_to_sr_cases, rtp_to_sr_desc);
495+
322496 static void rtp_desc(const struct rtp_test_case *t, char *desc)
323497 {
324498 strscpy(desc, t->name, KUNIT_PARAM_DESC_SIZE);
325499 }
326500
327−KUNIT_ARRAY_PARAM(rtp, cases, rtp_desc);
501+KUNIT_ARRAY_PARAM(rtp, rtp_cases, rtp_desc);
328502
329503 static int xe_rtp_test_init(struct kunit *test)
330504 {
@@ -357,6 +531,7 @@ static void xe_rtp_test_exit(struct kunit *test) · 26 unmodified lines
357531 }
358532
359533 static struct kunit_case xe_rtp_tests[] = {
534+ KUNIT_CASE_PARAM(xe_rtp_process_to_sr_tests, rtp_to_sr_gen_params),
360535 KUNIT_CASE_PARAM(xe_rtp_process_tests, rtp_gen_params),
361536 {}
362537 };
drivers/gpu/drm/xe/tests/xe_test.h+3−7
@@ -9,8 +9,8 @@ · 8 unmodified lines
99 #include <linux/types.h>
1010
1111 #if IS_ENABLED(CONFIG_DRM_XE_KUNIT_TEST)
12−#include <linux/sched.h>
1312 #include <kunit/test.h>
13+#include <kunit/test-bug.h>
1414
1515 /*
1616 * Each test that provides a kunit private test structure, place a test id
@@ -31,8 +31,6 @@ struct xe_test_priv { · 14 unmodified lines
3131
3232 #define XE_TEST_DECLARE(x) x
3333 #define XE_TEST_ONLY(x) unlikely(x)
34−#define XE_TEST_EXPORT
35−#define xe_cur_kunit() current->kunit_test
3634
3735 /**
3836 * xe_cur_kunit_priv - Obtain the struct xe_test_priv pointed to by
@@ -48,10 +46,10 @@ xe_cur_kunit_priv(enum xe_test_priv_id id) · 9 unmodified lines
4846 {
4947 struct xe_test_priv *priv;
5048
51− if (!xe_cur_kunit())
49+ if (!kunit_get_current_test())
5250 return NULL;
5351
54− priv = xe_cur_kunit()->priv;
52+ priv = kunit_get_current_test()->priv;
5553 return priv->id == id ? priv : NULL;
5654 }
5755
@@ -59,8 +57,6 @@ xe_cur_kunit_priv(enum xe_test_priv_id id) · 1 unmodified line
5957
6058 #define XE_TEST_DECLARE(x)
6159 #define XE_TEST_ONLY(x) 0
62−#define XE_TEST_EXPORT static
63−#define xe_cur_kunit() NULL
6460 #define xe_cur_kunit_priv(_id) NULL
6561
6662 #endif
drivers/gpu/drm/xe/tests/xe_wa_test.c+1−0
@@ -74,6 +74,7 @@ static const struct platform_test_case cases[] = { · 73 unmodified lines
7474 GMDID_CASE(METEORLAKE, 1274, A0, 1300, A0),
7575 GMDID_CASE(LUNARLAKE, 2004, A0, 2000, A0),
7676 GMDID_CASE(LUNARLAKE, 2004, B0, 2000, A0),
77+ GMDID_CASE(BATTLEMAGE, 2001, A0, 1301, A1),
7778 };
7879
7980 static void platform_desc(const struct platform_test_case *t, char *desc)
drivers/gpu/drm/xe/xe_bo.c+6−5
@@ -1264,13 +1264,14 @@ struct xe_bo *___xe_bo_create_locked(struct xe_device *xe, struct xe_bo *bo, · 1263 unmodified lines
12641264 if (flags & (XE_BO_FLAG_VRAM_MASK | XE_BO_FLAG_STOLEN) &&
12651265 !(flags & XE_BO_FLAG_IGNORE_MIN_PAGE_SIZE) &&
12661266 ((xe->info.vram_flags & XE_VRAM_FLAGS_NEED64K) ||
1267− (flags & XE_BO_NEEDS_64K))) {
1268− aligned_size = ALIGN(size, SZ_64K);
1267+ (flags & (XE_BO_FLAG_NEEDS_64K | XE_BO_FLAG_NEEDS_2M)))) {
1268+ size_t align = flags & XE_BO_FLAG_NEEDS_2M ? SZ_2M : SZ_64K;
1269+
1270+ aligned_size = ALIGN(size, align);
12691271 if (type != ttm_bo_type_device)
1270− size = ALIGN(size, SZ_64K);
1272+ size = ALIGN(size, align);
12711273 flags |= XE_BO_FLAG_INTERNAL_64K;
1272− alignment = SZ_64K >> PAGE_SHIFT;
1273−
1274+ alignment = align >> PAGE_SHIFT;
12741275 } else {
12751276 aligned_size = ALIGN(size, SZ_4K);
12761277 flags &= ~XE_BO_FLAG_INTERNAL_64K;
drivers/gpu/drm/xe/xe_bo.h+3−2
@@ -36,8 +36,9 @@ · 35 unmodified lines
3636 #define XE_BO_FLAG_PAGETABLE BIT(12)
3737 #define XE_BO_FLAG_NEEDS_CPU_ACCESS BIT(13)
3838 #define XE_BO_FLAG_NEEDS_UC BIT(14)
39−#define XE_BO_NEEDS_64K BIT(15)
40−#define XE_BO_FLAG_GGTT_INVALIDATE BIT(16)
39+#define XE_BO_FLAG_NEEDS_64K BIT(15)
40+#define XE_BO_FLAG_NEEDS_2M BIT(16)
41+#define XE_BO_FLAG_GGTT_INVALIDATE BIT(17)
4142 /* this one is trigger internally only */
4243 #define XE_BO_FLAG_INTERNAL_TEST BIT(30)
4344 #define XE_BO_FLAG_INTERNAL_64K BIT(31)
drivers/gpu/drm/xe/xe_bo_types.h+2−0
@@ -58,6 +58,8 @@ struct xe_bo { · 57 unmodified lines
5858 #endif
5959 /** @freed: List node for delayed put. */
6060 struct llist_node freed;
61+ /** @update_index: Update index if PT BO */
62+ int update_index;
6163 /** @created: Whether the bo has passed initial creation */
6264 bool created;
6365
drivers/gpu/drm/xe/xe_devcoredump.c+2−8
@@ -171,7 +171,6 @@ static void devcoredump_snapshot(struct xe_devcoredump *coredump, · 170 unmodified lines
171171 u32 adj_logical_mask = q->logical_mask;
172172 u32 width_mask = (0x1 << q->width) - 1;
173173 const char *process_name = "no process";
174− struct task_struct *task = NULL;
175174
176175 int i;
177176 bool cookie;
@@ -179,14 +178,9 @@ static void devcoredump_snapshot(struct xe_devcoredump *coredump, · 1 unmodified line
179178 ss->snapshot_time = ktime_get_real();
180179 ss->boot_time = ktime_get_boottime();
181180
182− if (q->vm && q->vm->xef) {
183− task = get_pid_task(q->vm->xef->drm->pid, PIDTYPE_PID);
184− if (task)
185− process_name = task->comm;
186− }
181+ if (q->vm && q->vm->xef)
182+ process_name = q->vm->xef->process_name;
187183 strscpy(ss->process_name, process_name);
188− if (task)
189− put_task_struct(task);
190184
191185 ss->gt = q->gt;
192186 INIT_WORK(&ss->work, xe_devcoredump_deferred_snap_work);
drivers/gpu/drm/xe/xe_device.c+101−10
@@ -54,6 +54,9 @@ · 53 unmodified lines
5454 #include "xe_vm.h"
5555 #include "xe_vram.h"
5656 #include "xe_wait_user_fence.h"
57+#include "xe_wa.h"
58+
59+#include <generated/xe_wa_oob.h>
5760
5861 static int xe_file_open(struct drm_device *dev, struct drm_file *file)
5962 {
@@ -61,6 +64,7 @@ static int xe_file_open(struct drm_device *dev, struct drm_file *file) · 1 unmodified line
6164 struct xe_drm_client *client;
6265 struct xe_file *xef;
6366 int ret = -ENOMEM;
67+ struct task_struct *task = NULL;
6468
6569 xef = kzalloc(sizeof(*xef), GFP_KERNEL);
6670 if (!xef)
@@ -87,9 +91,63 @@ static int xe_file_open(struct drm_device *dev, struct drm_file *file) · 20 unmodified lines
8791 spin_unlock(&xe->clients.lock);
8892
8993 file->driver_priv = xef;
94+ kref_init(&xef->refcount);
95+
96+ task = get_pid_task(rcu_access_pointer(file->pid), PIDTYPE_PID);
97+ if (task) {
98+ xef->process_name = kstrdup(task->comm, GFP_KERNEL);
99+ xef->pid = task->pid;
100+ put_task_struct(task);
101+ }
102+
90103 return 0;
91104 }
92105
106+static void xe_file_destroy(struct kref *ref)
107+{
108+ struct xe_file *xef = container_of(ref, struct xe_file, refcount);
109+ struct xe_device *xe = xef->xe;
110+
111+ xa_destroy(&xef->exec_queue.xa);
112+ mutex_destroy(&xef->exec_queue.lock);
113+ xa_destroy(&xef->vm.xa);
114+ mutex_destroy(&xef->vm.lock);
115+
116+ spin_lock(&xe->clients.lock);
117+ xe->clients.count--;
118+ spin_unlock(&xe->clients.lock);
119+
120+ xe_drm_client_put(xef->client);
121+ kfree(xef->process_name);
122+ kfree(xef);
123+}
124+
125+/**
126+ * xe_file_get() - Take a reference to the xe file object
127+ * @xef: Pointer to the xe file
128+ *
129+ * Anyone with a pointer to xef must take a reference to the xe file
130+ * object using this call.
131+ *
132+ * Return: xe file pointer
133+ */
134+struct xe_file *xe_file_get(struct xe_file *xef)
135+{
136+ kref_get(&xef->refcount);
137+ return xef;
138+}
139+
140+/**
141+ * xe_file_put() - Drop a reference to the xe file object
142+ * @xef: Pointer to the xe file
143+ *
144+ * Used to drop reference to the xef object
145+ */
146+void xe_file_put(struct xe_file *xef)
147+{
148+ kref_put(&xef->refcount, xe_file_destroy);
149+}
150+
93151 static void xe_file_close(struct drm_device *dev, struct drm_file *file)
94152 {
95153 struct xe_device *xe = to_xe_device(dev);
@@ -98,6 +156,8 @@ static void xe_file_close(struct drm_device *dev, struct drm_file *file) · 2 unmodified lines
98156 struct xe_exec_queue *q;
99157 unsigned long idx;
100158
159+ xe_pm_runtime_get(xe);
160+
101161 /*
102162 * No need for exec_queue.lock here as there is no contention for it
103163 * when FD is closing as IOCTLs presumably can't be modifying the
@@ -108,21 +168,14 @@ static void xe_file_close(struct drm_device *dev, struct drm_file *file) · 4 unmodified lines
108168 xe_exec_queue_kill(q);
109169 xe_exec_queue_put(q);
110170 }
111− xa_destroy(&xef->exec_queue.xa);
112− mutex_destroy(&xef->exec_queue.lock);
113171 mutex_lock(&xef->vm.lock);
114172 xa_for_each(&xef->vm.xa, idx, vm)
115173 xe_vm_close_and_put(vm);
116174 mutex_unlock(&xef->vm.lock);
117− xa_destroy(&xef->vm.xa);
118− mutex_destroy(&xef->vm.lock);
119175
120− spin_lock(&xe->clients.lock);
121− xe->clients.count--;
122− spin_unlock(&xe->clients.lock);
176+ xe_file_put(xef);
123177
124− xe_drm_client_put(xef->client);
125− kfree(xef);
178+ xe_pm_runtime_put(xe);
126179 }
127180
128181 static const struct drm_ioctl_desc xe_ioctls[] = {
@@ -744,13 +797,22 @@ void xe_device_shutdown(struct xe_device *xe) · 615 unmodified lines
744797 {
745798 }
746799
800+/**
801+ * xe_device_wmb() - Device specific write memory barrier
802+ * @xe: the &xe_device
803+ *
804+ * While wmb() is sufficient for a barrier if we use system memory, on discrete
805+ * platforms with device memory we additionally need to issue a register write.
806+ * Since it doesn't matter which register we write to, use the read-only VF_CAP
807+ * register that is also marked as accessible by the VFs.
808+ */
747809 void xe_device_wmb(struct xe_device *xe)
748810 {
749811 struct xe_gt *gt = xe_root_mmio_gt(xe);
750812
751813 wmb();
752814 if (IS_DGFX(xe))
753− xe_mmio_write32(gt, SOFTWARE_FLAGS_SPR33, 0);
815+ xe_mmio_write32(gt, VF_CAP_REG, 0);
754816 }
755817
756818 /**
@@ -779,6 +841,11 @@ void xe_device_td_flush(struct xe_device *xe) · 22 unmodified lines
779841 if (!IS_DGFX(xe) || GRAPHICS_VER(xe) < 20)
780842 return;
781843
844+ if (XE_WA(xe_root_mmio_gt(xe), 16023588340)) {
845+ xe_device_l2_flush(xe);
846+ return;
847+ }
848+
782849 for_each_gt(gt, xe, id) {
783850 if (xe_gt_is_media_type(gt))
784851 continue;
@@ -802,6 +869,30 @@ void xe_device_td_flush(struct xe_device *xe) · 17 unmodified lines
802869 }
803870 }
804871
872+void xe_device_l2_flush(struct xe_device *xe)
873+{
874+ struct xe_gt *gt;
875+ int err;
876+
877+ gt = xe_root_mmio_gt(xe);
878+
879+ if (!XE_WA(gt, 16023588340))
880+ return;
881+
882+ err = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
883+ if (err)
884+ return;
885+
886+ spin_lock(&gt->global_invl_lock);
887+ xe_mmio_write32(gt, XE2_GLOBAL_INVAL, 0x1);
888+
889+ if (xe_mmio_wait32(gt, XE2_GLOBAL_INVAL, 0x1, 0x0, 150, NULL, true))
890+ xe_gt_err_once(gt, "Global invalidation timeout\n");
891+ spin_unlock(&gt->global_invl_lock);
892+
893+ xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
894+}
895+
805896 u32 xe_device_ccs_bytes(struct xe_device *xe, u64 size)
806897 {
807898 return xe_device_has_flat_ccs(xe) ?
drivers/gpu/drm/xe/xe_device.h+9−0
@@ -20,6 +20,11 @@ static inline struct xe_device *pdev_to_xe_device(struct pci_dev *pdev) · 19 unmodified lines
2020 return pci_get_drvdata(pdev);
2121 }
2222
23+static inline struct xe_device *xe_device_const_cast(const struct xe_device *xe)
24+{
25+ return (struct xe_device *)xe;
26+}
27+
2328 static inline struct xe_device *ttm_to_xe_device(struct ttm_device *ttm)
2429 {
2530 return container_of(ttm, struct xe_device, ttm);
@@ -162,6 +167,7 @@ u64 xe_device_canonicalize_addr(struct xe_device *xe, u64 address); · 136 unmodified lines
162167 u64 xe_device_uncanonicalize_addr(struct xe_device *xe, u64 address);
163168
164169 void xe_device_td_flush(struct xe_device *xe);
170+void xe_device_l2_flush(struct xe_device *xe);
165171
166172 static inline bool xe_device_wedged(struct xe_device *xe)
167173 {
@@ -170,4 +176,7 @@ static inline bool xe_device_wedged(struct xe_device *xe) · 2 unmodified lines
170176
171177 void xe_device_declare_wedged(struct xe_device *xe);
172178
179+struct xe_file *xe_file_get(struct xe_file *xef);
180+void xe_file_put(struct xe_file *xef);
181+
173182 #endif
drivers/gpu/drm/xe/xe_device_types.h+30−0
@@ -23,6 +23,10 @@ · 22 unmodified lines
2323 #include "xe_sriov_types.h"
2424 #include "xe_step_types.h"
2525
26+#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
27+#define TEST_VM_OPS_ERROR
28+#endif
29+
2630 #if IS_ENABLED(CONFIG_DRM_XE_DISPLAY)
2731 #include "soc/intel_pch.h"
2832 #include "intel_display_core.h"
@@ -40,6 +44,7 @@ struct xe_pat_ops; · 11 unmodified lines
4044 #define MEDIA_VERx100(xe) ((xe)->info.media_verx100)
4145 #define IS_DGFX(xe) ((xe)->info.is_dgfx)
4246 #define HAS_HECI_GSCFI(xe) ((xe)->info.has_heci_gscfi)
47+#define HAS_HECI_CSCFI(xe) ((xe)->info.has_heci_cscfi)
4348
4449 #define XE_VRAM_FLAGS_NEED64K BIT(0)
4550
@@ -285,6 +290,8 @@ struct xe_device { · 239 unmodified lines
285290 u8 skip_pcode:1;
286291 /** @info.has_heci_gscfi: device has heci gscfi */
287292 u8 has_heci_gscfi:1;
293+ /** @info.has_heci_cscfi: device has heci cscfi */
294+ u8 has_heci_cscfi:1;
288295 /** @info.skip_guc_pc: Skip GuC based PM feature init */
289296 u8 skip_guc_pc:1;
290297 /** @info.has_atomic_enable_pte_bit: Device has atomic enable PTE bit */
@@ -477,6 +484,14 @@ struct xe_device { · 186 unmodified lines
477484 int mode;
478485 } wedged;
479486
487+#ifdef TEST_VM_OPS_ERROR
488+ /**
489+ * @vm_inject_error_position: inject errors at different places in VM
490+ * bind IOCTL based on this value
491+ */
492+ u8 vm_inject_error_position;
493+#endif
494+
480495 /* private: */
481496
482497 #if IS_ENABLED(CONFIG_DRM_XE_DISPLAY)
@@ -566,6 +581,21 @@ struct xe_file { · 83 unmodified lines
566581
567582 /** @client: drm client */
568583 struct xe_drm_client *client;
584+
585+ /**
586+ * @process_name: process name for file handle, used to safely output
587+ * during error situations where xe file can outlive process
588+ */
589+ char *process_name;
590+
591+ /**
592+ * @pid: pid for file handle, used to safely output uring error
593+ * situations where xe file can outlive process
594+ */
595+ pid_t pid;
596+
597+ /** @refcount: ref count of this xe file */
598+ struct kref refcount;
569599 };
570600
571601 #endif
drivers/gpu/drm/xe/xe_drm_client.c+1−4
@@ -251,11 +251,8 @@ static void show_run_ticks(struct drm_printer *p, struct drm_file *file) · 250 unmodified lines
251251
252252 /* Accumulate all the exec queues from this client */
253253 mutex_lock(&xef->exec_queue.lock);
254− xa_for_each(&xef->exec_queue.xa, i, q) {
254+ xa_for_each(&xef->exec_queue.xa, i, q)
255255 xe_exec_queue_update_run_ticks(q);
256− xef->run_ticks[q->class] += q->run_ticks - q->old_run_ticks;
257− q->old_run_ticks = q->run_ticks;
258− }
259256 mutex_unlock(&xef->exec_queue.lock);
260257
261258 /* Get the total GPU cycles */
drivers/gpu/drm/xe/xe_exec_queue.c+32−1
@@ -37,6 +37,10 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q) · 36 unmodified lines
3737 {
3838 if (q->vm)
3939 xe_vm_put(q->vm);
40+
41+ if (q->xef)
42+ xe_file_put(q->xef);
43+
4044 kfree(q);
4145 }
4246
@@ -649,6 +653,7 @@ int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data, · 606 unmodified lines
649653 goto kill_exec_queue;
650654
651655 args->exec_queue_id = id;
656+ q->xef = xe_file_get(xef);
652657
653658 return 0;
654659
@@ -762,6 +767,7 @@ bool xe_exec_queue_is_idle(struct xe_exec_queue *q) · 107 unmodified lines
762767 */
763768 void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q)
764769 {
770+ struct xe_file *xef;
765771 struct xe_lrc *lrc;
766772 u32 old_ts, new_ts;
767773
@@ -773,6 +779,8 @@ void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q) · 5 unmodified lines
773779 if (!q->vm || !q->vm->xef)
774780 return;
775781
782+ xef = q->vm->xef;
783+
776784 /*
777785 * Only sample the first LRC. For parallel submission, all of them are
778786 * scheduled together and we compensate that below by multiplying by
@@ -783,7 +791,7 @@ void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q) · 4 unmodified lines
783791 */
784792 lrc = q->lrc[0];
785793 new_ts = xe_lrc_update_timestamp(lrc, &old_ts);
786− q->run_ticks += (new_ts - old_ts) * q->width;
794+ xef->run_ticks[q->class] += (new_ts - old_ts) * q->width;
787795 }
788796
789797 void xe_exec_queue_kill(struct xe_exec_queue *q)
@@ -906,3 +914,26 @@ void xe_exec_queue_last_fence_set(struct xe_exec_queue *q, struct xe_vm *vm, · 116 unmodified lines
906914 xe_exec_queue_last_fence_put(q, vm);
907915 q->last_fence = dma_fence_get(fence);
908916 }
917+
918+/**
919+ * xe_exec_queue_last_fence_test_dep - Test last fence dependency of queue
920+ * @q: The exec queue
921+ * @vm: The VM the engine does a bind or exec for
922+ *
923+ * Returns:
924+ * -ETIME if there exists an unsignalled last fence dependency, zero otherwise.
925+ */
926+int xe_exec_queue_last_fence_test_dep(struct xe_exec_queue *q, struct xe_vm *vm)
927+{
928+ struct dma_fence *fence;
929+ int err = 0;
930+
931+ fence = xe_exec_queue_last_fence_get(q, vm);
932+ if (fence) {
933+ err = test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) ?
934+ 0 : -ETIME;
935+ dma_fence_put(fence);
936+ }
937+
938+ return err;
939+}
drivers/gpu/drm/xe/xe_exec_queue.h+2−0
@@ -75,6 +75,8 @@ struct dma_fence *xe_exec_queue_last_fence_get(struct xe_exec_queue *e, · 74 unmodified lines
7575 struct xe_vm *vm);
7676 void xe_exec_queue_last_fence_set(struct xe_exec_queue *e, struct xe_vm *vm,
7777 struct dma_fence *fence);
78+int xe_exec_queue_last_fence_test_dep(struct xe_exec_queue *q,
79+ struct xe_vm *vm);
7880 void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q);
7981
8082 #endif
drivers/gpu/drm/xe/xe_exec_queue_types.h+7−6
@@ -38,6 +38,9 @@ enum xe_exec_queue_priority { · 37 unmodified lines
3838 * a kernel object.
3939 */
4040 struct xe_exec_queue {
41+ /** @xef: Back pointer to xe file if this is user created exec queue */
42+ struct xe_file *xef;
43+
4144 /** @gt: graphics tile this exec queue can submit to */
4245 struct xe_gt *gt;
4346 /**
@@ -139,10 +142,6 @@ struct xe_exec_queue { · 95 unmodified lines
139142 * Protected by @vm's resv. Unused if @vm == NULL.
140143 */
141144 u64 tlb_flush_seqno;
142− /** @old_run_ticks: prior hw engine class run time in ticks for this exec queue */
143− u64 old_run_ticks;
144− /** @run_ticks: hw engine class run time in ticks for this exec queue */
145− u64 run_ticks;
146145 /** @lrc: logical ring context for this exec queue */
147146 struct xe_lrc *lrc[];
148147 };
@@ -172,9 +171,11 @@ struct xe_exec_queue_ops { · 23 unmodified lines
172171 int (*suspend)(struct xe_exec_queue *q);
173172 /**
174173 * @suspend_wait: Wait for an exec queue to suspend executing, should be
175− * call after suspend.
174+ * call after suspend. In dma-fencing path thus must return within a
175+ * reasonable amount of time. -ETIME return shall indicate an error
176+ * waiting for suspend resulting in associated VM getting killed.
176177 */
177− void (*suspend_wait)(struct xe_exec_queue *q);
178+ int (*suspend_wait)(struct xe_exec_queue *q);
178179 /**
179180 * @resume: Resume exec queue execution, exec queue must be in a suspended
180181 * state and dma fence returned from most recent suspend call must be
drivers/gpu/drm/xe/xe_execlist.c+2−1
@@ -422,10 +422,11 @@ static int execlist_exec_queue_suspend(struct xe_exec_queue *q) · 421 unmodified lines
422422 return 0;
423423 }
424424
425−static void execlist_exec_queue_suspend_wait(struct xe_exec_queue *q)
425+static int execlist_exec_queue_suspend_wait(struct xe_exec_queue *q)
426426
427427 {
428428 /* NIY */
429+ return 0;
429430 }
430431
431432 static void execlist_exec_queue_resume(struct xe_exec_queue *q)
drivers/gpu/drm/xe/xe_gen_wa_oob.c+12−4
@@ -97,19 +97,27 @@ static int parse(FILE *input, FILE *csource, FILE *cheader) · 96 unmodified lines
9797
9898 if (name) {
9999 fprintf(cheader, "\tXE_WA_OOB_%s = %u,\n", name, idx);
100− fprintf(csource, "{ XE_RTP_NAME(\"%s\"), XE_RTP_RULES(%s) },\n",
100+
101+ /* Close previous entry before starting a new one */
102+ if (idx)
103+ fprintf(csource, ") },\n");
104+
105+ fprintf(csource, "{ XE_RTP_NAME(\"%s\"),\n XE_RTP_RULES(%s",
101106 name, rules);
107+ idx++;
102108 } else {
103− fprintf(csource, "{ XE_RTP_NAME(NULL), XE_RTP_RULES(%s) },\n",
104− rules);
109+ fprintf(csource, ", OR,\n\t%s", rules);
105110 }
106111
107− idx++;
108112 lineno++;
109113 if (!is_continuation)
110114 prev_name = name;
111115 }
112116
117+ /* Close last entry */
118+ if (idx)
119+ fprintf(csource, ") },\n");
120+
113121 fprintf(cheader, "\t_XE_WA_OOB_COUNT = %u\n", idx);
114122
115123 return 0;
drivers/gpu/drm/xe/xe_gt.c+54−0
@@ -9,6 +9,7 @@ · 8 unmodified lines
99
1010 #include <drm/drm_managed.h>
1111 #include <drm/xe_drm.h>
12+
1213 #include <generated/xe_wa_oob.h>
1314
1415 #include "instructions/xe_gfxpipe_commands.h"
@@ -95,6 +96,51 @@ void xe_gt_sanitize(struct xe_gt *gt) · 80 unmodified lines
9596 gt->uc.guc.submission_state.enabled = false;
9697 }
9798
99+static void xe_gt_enable_host_l2_vram(struct xe_gt *gt)
100+{
101+ u32 reg;
102+ int err;
103+
104+ if (!XE_WA(gt, 16023588340))
105+ return;
106+
107+ err = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
108+ if (WARN_ON(err))
109+ return;
110+
111+ if (!xe_gt_is_media_type(gt)) {
112+ xe_mmio_write32(gt, SCRATCH1LPFC, EN_L3_RW_CCS_CACHE_FLUSH);
113+ reg = xe_mmio_read32(gt, XE2_GAMREQSTRM_CTRL);
114+ reg |= CG_DIS_CNTLBUS;
115+ xe_mmio_write32(gt, XE2_GAMREQSTRM_CTRL, reg);
116+ }
117+
118+ xe_gt_mcr_multicast_write(gt, XEHPC_L3CLOS_MASK(3), 0x3);
119+ xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
120+}
121+
122+static void xe_gt_disable_host_l2_vram(struct xe_gt *gt)
123+{
124+ u32 reg;
125+ int err;
126+
127+ if (!XE_WA(gt, 16023588340))
128+ return;
129+
130+ if (xe_gt_is_media_type(gt))
131+ return;
132+
133+ err = xe_force_wake_get(gt_to_fw(gt), XE_FW_GT);
134+ if (WARN_ON(err))
135+ return;
136+
137+ reg = xe_mmio_read32(gt, XE2_GAMREQSTRM_CTRL);
138+ reg &= ~CG_DIS_CNTLBUS;
139+ xe_mmio_write32(gt, XE2_GAMREQSTRM_CTRL, reg);
140+
141+ xe_force_wake_put(gt_to_fw(gt), XE_FW_GT);
142+}
143+
98144 /**
99145 * xe_gt_remove() - Clean up the GT structures before driver removal
100146 * @gt: the GT object
@@ -111,6 +157,8 @@ void xe_gt_remove(struct xe_gt *gt) · 10 unmodified lines
111157
112158 for (i = 0; i < XE_ENGINE_CLASS_MAX; ++i)
113159 xe_hw_fence_irq_finish(&gt->fence_irq[i]);
160+
161+ xe_gt_disable_host_l2_vram(gt);
114162 }
115163
116164 static void gt_reset_worker(struct work_struct *w);
@@ -339,6 +387,7 @@ int xe_gt_init_early(struct xe_gt *gt) · 222 unmodified lines
339387
340388 xe_force_wake_init_gt(gt, gt_to_fw(gt));
341389 xe_pcode_init(gt);
390+ spin_lock_init(&gt->global_invl_lock);
342391
343392 return 0;
344393 }
@@ -508,6 +557,7 @@ int xe_gt_init_hwconfig(struct xe_gt *gt) · 163 unmodified lines
508557
509558 xe_gt_mcr_init_early(gt);
510559 xe_pat_init(gt);
560+ xe_gt_enable_host_l2_vram(gt);
511561
512562 err = xe_uc_init(&gt->uc);
513563 if (err)
@@ -643,6 +693,8 @@ static int do_gt_restart(struct xe_gt *gt) · 129 unmodified lines
643693
644694 xe_pat_init(gt);
645695
696+ xe_gt_enable_host_l2_vram(gt);
697+
646698 xe_gt_mcr_set_implicit_defaults(gt);
647699 xe_reg_sr_apply_mmio(&gt->reg_sr, gt);
648700
@@ -796,6 +848,8 @@ int xe_gt_suspend(struct xe_gt *gt) · 147 unmodified lines
796848
797849 xe_gt_idle_disable_pg(gt);
798850
851+ xe_gt_disable_host_l2_vram(gt);
852+
799853 XE_WARN_ON(xe_force_wake_put(gt_to_fw(gt), XE_FORCEWAKE_ALL));
800854 xe_gt_dbg(gt, "suspended\n");
801855
drivers/gpu/drm/xe/xe_gt_sriov_pf.c+1−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
55
66 #include <drm/drm_managed.h>
77
8−#include "regs/xe_sriov_regs.h"
8+#include "regs/xe_regs.h"
99
1010 #include "xe_gt_sriov_pf.h"
1111 #include "xe_gt_sriov_pf_config.h"
drivers/gpu/drm/xe/xe_gt_sriov_pf_config.c+1−0
@@ -1401,6 +1401,7 @@ static int pf_provision_vf_lmem(struct xe_gt *gt, unsigned int vfid, u64 size) · 1400 unmodified lines
14011401 ALIGN(size, PAGE_SIZE),
14021402 ttm_bo_type_kernel,
14031403 XE_BO_FLAG_VRAM_IF_DGFX(tile) |
1404+ XE_BO_FLAG_NEEDS_2M |
14041405 XE_BO_FLAG_PINNED);
14051406 if (IS_ERR(bo))
14061407 return PTR_ERR(bo);
drivers/gpu/drm/xe/xe_gt_sriov_vf.c+27−1
@@ -850,7 +850,7 @@ static struct vf_runtime_reg *vf_lookup_reg(struct xe_gt *gt, u32 addr) · 849 unmodified lines
850850
851851 xe_gt_assert(gt, IS_SRIOV_VF(gt_to_xe(gt)));
852852
853− return bsearch(&key, runtime->regs, runtime->regs_size, sizeof(key),
853+ return bsearch(&key, runtime->regs, runtime->num_regs, sizeof(key),
854854 vf_runtime_reg_cmp);
855855 }
856856
@@ -892,6 +892,32 @@ u32 xe_gt_sriov_vf_read32(struct xe_gt *gt, struct xe_reg reg) · 35 unmodified lines
892892 return rr->value;
893893 }
894894
895+/**
896+ * xe_gt_sriov_vf_write32 - Handle a write to an inaccessible register.
897+ * @gt: the &xe_gt
898+ * @reg: the register to write
899+ * @val: value to write
900+ *
901+ * This function is for VF use only.
902+ * Currently it will trigger a WARN if running on debug build.
903+ */
904+void xe_gt_sriov_vf_write32(struct xe_gt *gt, struct xe_reg reg, u32 val)
905+{
906+ u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
907+
908+ xe_gt_assert(gt, IS_SRIOV_VF(gt_to_xe(gt)));
909+ xe_gt_assert(gt, !reg.vf);
910+
911+ /*
912+ * In the future, we may want to handle selected writes to inaccessible
913+ * registers in some custom way, but for now let's just log a warning
914+ * about such attempt, as likely we might be doing something wrong.
915+ */
916+ xe_gt_WARN(gt, IS_ENABLED(CONFIG_DRM_XE_DEBUG),
917+ "VF is trying to write %#x to an inaccessible register %#x+%#x\n",
918+ val, reg.addr, addr - reg.addr);
919+}
920+
895921 /**
896922 * xe_gt_sriov_vf_print_config - Print VF self config.
897923 * @gt: the &xe_gt
drivers/gpu/drm/xe/xe_gt_sriov_vf.h+1−0
@@ -22,6 +22,7 @@ u32 xe_gt_sriov_vf_gmdid(struct xe_gt *gt); · 21 unmodified lines
2222 u16 xe_gt_sriov_vf_guc_ids(struct xe_gt *gt);
2323 u64 xe_gt_sriov_vf_lmem(struct xe_gt *gt);
2424 u32 xe_gt_sriov_vf_read32(struct xe_gt *gt, struct xe_reg reg);
25+void xe_gt_sriov_vf_write32(struct xe_gt *gt, struct xe_reg reg, u32 val);
2526
2627 void xe_gt_sriov_vf_print_config(struct xe_gt *gt, struct drm_printer *p);
2728 void xe_gt_sriov_vf_print_runtime(struct xe_gt *gt, struct drm_printer *p);
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c+111−94
@@ -13,10 +13,13 @@ · 12 unmodified lines
1313 #include "xe_guc.h"
1414 #include "xe_guc_ct.h"
1515 #include "xe_mmio.h"
16+#include "xe_pm.h"
1617 #include "xe_sriov.h"
1718 #include "xe_trace.h"
1819 #include "regs/xe_guc_regs.h"
1920
21+#define FENCE_STACK_BIT DMA_FENCE_FLAG_USER_BITS
22+
2023 /*
2124 * TLB inval depends on pending commands in the CT queue and then the real
2225 * invalidation time. Double up the time to process full CT queue
@@ -33,6 +36,24 @@ static long tlb_timeout_jiffies(struct xe_gt *gt) · 10 unmodified lines
3336 return hw_tlb_timeout + 2 * delay;
3437 }
3538
39+static void
40+__invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
41+{
42+ bool stack = test_bit(FENCE_STACK_BIT, &fence->base.flags);
43+
44+ trace_xe_gt_tlb_invalidation_fence_signal(xe, fence);
45+ xe_gt_tlb_invalidation_fence_fini(fence);
46+ dma_fence_signal(&fence->base);
47+ if (!stack)
48+ dma_fence_put(&fence->base);
49+}
50+
51+static void
52+invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
53+{
54+ list_del(&fence->link);
55+ __invalidation_fence_signal(xe, fence);
56+}
3657
3758 static void xe_gt_tlb_fence_timeout(struct work_struct *work)
3859 {
@@ -54,10 +75,8 @@ static void xe_gt_tlb_fence_timeout(struct work_struct *work) · 15 unmodified lines
5475 xe_gt_err(gt, "TLB invalidation fence timeout, seqno=%d recv=%d",
5576 fence->seqno, gt->tlb_invalidation.seqno_recv);
5677
57− list_del(&fence->link);
5878 fence->base.error = -ETIME;
59− dma_fence_signal(&fence->base);
60− dma_fence_put(&fence->base);
79+ invalidation_fence_signal(xe, fence);
6180 }
6281 if (!list_empty(&gt->tlb_invalidation.pending_fences))
6382 queue_delayed_work(system_wq,
@@ -87,21 +106,6 @@ int xe_gt_tlb_invalidation_init(struct xe_gt *gt) · 23 unmodified lines
87106 return 0;
88107 }
89108
90−static void
91−__invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
92−{
93− trace_xe_gt_tlb_invalidation_fence_signal(xe, fence);
94− dma_fence_signal(&fence->base);
95− dma_fence_put(&fence->base);
96−}
97−
98−static void
99−invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fence *fence)
100−{
101− list_del(&fence->link);
102− __invalidation_fence_signal(xe, fence);
103−}
104−
105109 /**
106110 * xe_gt_tlb_invalidation_reset - Initialize GT TLB invalidation reset
107111 * @gt: graphics tile
@@ -111,7 +115,6 @@ invalidation_fence_signal(struct xe_device *xe, struct xe_gt_tlb_invalidation_fe · 3 unmodified lines
111115 void xe_gt_tlb_invalidation_reset(struct xe_gt *gt)
112116 {
113117 struct xe_gt_tlb_invalidation_fence *fence, *next;
114− struct xe_guc *guc = &gt->uc.guc;
115118 int pending_seqno;
116119
117120 /*
@@ -134,7 +137,6 @@ void xe_gt_tlb_invalidation_reset(struct xe_gt *gt) · 16 unmodified lines
134137 else
135138 pending_seqno = gt->tlb_invalidation.seqno - 1;
136139 WRITE_ONCE(gt->tlb_invalidation.seqno_recv, pending_seqno);
137− wake_up_all(&guc->ct.wq);
138140
139141 list_for_each_entry_safe(fence, next,
140142 &gt->tlb_invalidation.pending_fences, link)
@@ -165,6 +167,8 @@ static int send_tlb_invalidation(struct xe_guc *guc, · 24 unmodified lines
165167 int seqno;
166168 int ret;
167169
170+ xe_gt_assert(gt, fence);
171+
168172 /*
169173 * XXX: The seqno algorithm relies on TLB invalidation being processed
170174 * in order which they currently are, if that changes the algorithm will
@@ -173,14 +177,12 @@ static int send_tlb_invalidation(struct xe_guc *guc, · 2 unmodified lines
173177
174178 mutex_lock(&guc->ct.lock);
175179 seqno = gt->tlb_invalidation.seqno;
176− if (fence) {
177− fence->seqno = seqno;
178− trace_xe_gt_tlb_invalidation_fence_send(xe, fence);
179− }
180+ fence->seqno = seqno;
181+ trace_xe_gt_tlb_invalidation_fence_send(xe, fence);
180182 action[1] = seqno;
181183 ret = xe_guc_ct_send_locked(&guc->ct, action, len,
182184 G2H_LEN_DW_TLB_INVALIDATE, 1);
183− if (!ret && fence) {
185+ if (!ret) {
184186 spin_lock_irq(&gt->tlb_invalidation.pending_lock);
185187 /*
186188 * We haven't actually published the TLB fence as per
@@ -201,7 +203,7 @@ static int send_tlb_invalidation(struct xe_guc *guc, · 14 unmodified lines
201203 tlb_timeout_jiffies(gt));
202204 }
203205 spin_unlock_irq(&gt->tlb_invalidation.pending_lock);
204− } else if (ret < 0 && fence) {
206+ } else if (ret < 0) {
205207 __invalidation_fence_signal(xe, fence);
206208 }
207209 if (!ret) {
@@ -209,7 +211,6 @@ static int send_tlb_invalidation(struct xe_guc *guc, · 1 unmodified line
209211 TLB_INVALIDATION_SEQNO_MAX;
210212 if (!gt->tlb_invalidation.seqno)
211213 gt->tlb_invalidation.seqno = 1;
212− ret = seqno;
213214 }
214215 mutex_unlock(&guc->ct.lock);
215216
@@ -223,14 +224,16 @@ static int send_tlb_invalidation(struct xe_guc *guc, · 7 unmodified lines
223224 /**
224225 * xe_gt_tlb_invalidation_guc - Issue a TLB invalidation on this GT for the GuC
225226 * @gt: graphics tile
227+ * @fence: invalidation fence which will be signal on TLB invalidation
228+ * completion
226229 *
227230 * Issue a TLB invalidation for the GuC. Completion of TLB is asynchronous and
228− * caller can use seqno + xe_gt_tlb_invalidation_wait to wait for completion.
231+ * caller can use the invalidation fence to wait for completion.
229232 *
230− * Return: Seqno which can be passed to xe_gt_tlb_invalidation_wait on success,
231− * negative error code on error.
233+ * Return: 0 on success, negative error code on error
232234 */
233−static int xe_gt_tlb_invalidation_guc(struct xe_gt *gt)
235+static int xe_gt_tlb_invalidation_guc(struct xe_gt *gt,
236+ struct xe_gt_tlb_invalidation_fence *fence)
234237 {
235238 u32 action[] = {
236239 XE_GUC_ACTION_TLB_INVALIDATION,
@@ -238,7 +241,7 @@ static int xe_gt_tlb_invalidation_guc(struct xe_gt *gt) · 1 unmodified line
238241 MAKE_INVAL_OP(XE_GUC_TLB_INVAL_GUC),
239242 };
240243
241− return send_tlb_invalidation(&gt->uc.guc, NULL, action,
244+ return send_tlb_invalidation(&gt->uc.guc, fence, action,
242245 ARRAY_SIZE(action));
243246 }
244247
@@ -257,13 +260,17 @@ int xe_gt_tlb_invalidation_ggtt(struct xe_gt *gt) · 12 unmodified lines
257260
258261 if (xe_guc_ct_enabled(&gt->uc.guc.ct) &&
259262 gt->uc.guc.submission_state.enabled) {
260− int seqno;
261−
262− seqno = xe_gt_tlb_invalidation_guc(gt);
263− if (seqno <= 0)
264− return seqno;
263+ struct xe_gt_tlb_invalidation_fence fence;
264+ int ret;
265+
266+ xe_gt_tlb_invalidation_fence_init(gt, &fence, true);
267+ ret = xe_gt_tlb_invalidation_guc(gt, &fence);
268+ if (ret < 0) {
269+ xe_gt_tlb_invalidation_fence_fini(&fence);
270+ return ret;
271+ }
265272
266− xe_gt_tlb_invalidation_wait(gt, seqno);
273+ xe_gt_tlb_invalidation_fence_wait(&fence);
267274 } else if (xe_device_uc_enabled(xe) && !xe_device_wedged(xe)) {
268275 if (IS_SRIOV_VF(xe))
269276 return 0;
@@ -290,18 +297,16 @@ int xe_gt_tlb_invalidation_ggtt(struct xe_gt *gt) · 20 unmodified lines
290297 *
291298 * @gt: graphics tile
292299 * @fence: invalidation fence which will be signal on TLB invalidation
293− * completion, can be NULL
300+ * completion
294301 * @start: start address
295302 * @end: end address
296303 * @asid: address space id
297304 *
298305 * Issue a range based TLB invalidation if supported, if not fallback to a full
299− * TLB invalidation. Completion of TLB is asynchronous and caller can either use
300− * the invalidation fence or seqno + xe_gt_tlb_invalidation_wait to wait for
301− * completion.
306+ * TLB invalidation. Completion of TLB is asynchronous and caller can use
307+ * the invalidation fence to wait for completion.
302308 *
303− * Return: Seqno which can be passed to xe_gt_tlb_invalidation_wait on success,
304− * negative error code on error.
309+ * Return: Negative error code on error, 0 on success
305310 */
306311 int xe_gt_tlb_invalidation_range(struct xe_gt *gt,
307312 struct xe_gt_tlb_invalidation_fence *fence,
@@ -312,11 +317,11 @@ int xe_gt_tlb_invalidation_range(struct xe_gt *gt, · 4 unmodified lines
312317 u32 action[MAX_TLB_INVALIDATION_LEN];
313318 int len = 0;
314319
320+ xe_gt_assert(gt, fence);
321+
315322 /* Execlists not supported */
316323 if (gt_to_xe(gt)->info.force_execlist) {
317− if (fence)
318− __invalidation_fence_signal(xe, fence);
319−
324+ __invalidation_fence_signal(xe, fence);
320325 return 0;
321326 }
322327
@@ -382,12 +387,10 @@ int xe_gt_tlb_invalidation_range(struct xe_gt *gt, · 59 unmodified lines
382387 * @vma: VMA to invalidate
383388 *
384389 * Issue a range based TLB invalidation if supported, if not fallback to a full
385− * TLB invalidation. Completion of TLB is asynchronous and caller can either use
386− * the invalidation fence or seqno + xe_gt_tlb_invalidation_wait to wait for
387− * completion.
390+ * TLB invalidation. Completion of TLB is asynchronous and caller can use
391+ * the invalidation fence to wait for completion.
388392 *
389− * Return: Seqno which can be passed to xe_gt_tlb_invalidation_wait on success,
390− * negative error code on error.
393+ * Return: Negative error code on error, 0 on success
391394 */
392395 int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,
393396 struct xe_gt_tlb_invalidation_fence *fence,
@@ -400,43 +403,6 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt, · 6 unmodified lines
400403 xe_vma_vm(vma)->usm.asid);
401404 }
402405
403−/**
404− * xe_gt_tlb_invalidation_wait - Wait for TLB to complete
405− * @gt: graphics tile
406− * @seqno: seqno to wait which was returned from xe_gt_tlb_invalidation
407− *
408− * Wait for tlb_timeout_jiffies() for a TLB invalidation to complete.
409− *
410− * Return: 0 on success, -ETIME on TLB invalidation timeout
411− */
412−int xe_gt_tlb_invalidation_wait(struct xe_gt *gt, int seqno)
413−{
414− struct xe_guc *guc = &gt->uc.guc;
415− int ret;
416−
417− /* Execlists not supported */
418− if (gt_to_xe(gt)->info.force_execlist)
419− return 0;
420−
421− /*
422− * XXX: See above, this algorithm only works if seqno are always in
423− * order
424− */
425− ret = wait_event_timeout(guc->ct.wq,
426− tlb_invalidation_seqno_past(gt, seqno),
427− tlb_timeout_jiffies(gt));
428− if (!ret) {
429− struct drm_printer p = xe_gt_err_printer(gt);
430−
431− xe_gt_err(gt, "TLB invalidation time'd out, seqno=%d, recv=%d\n",
432− seqno, gt->tlb_invalidation.seqno_recv);
433− xe_guc_ct_print(&guc->ct, &p, true);
434− return -ETIME;
435− }
436−
437− return 0;
438−}
439−
440406 /**
441407 * xe_guc_tlb_invalidation_done_handler - TLB invalidation done handler
442408 * @guc: guc
@@ -480,12 +446,7 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len) · 37 unmodified lines
480446 return 0;
481447 }
482448
483− /*
484− * wake_up_all() and wait_event_timeout() already have the correct
485− * barriers.
486− */
487449 WRITE_ONCE(gt->tlb_invalidation.seqno_recv, msg[0]);
488− wake_up_all(&guc->ct.wq);
489450
490451 list_for_each_entry_safe(fence, next,
491452 &gt->tlb_invalidation.pending_fences, link) {
@@ -508,3 +469,59 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len) · 16 unmodified lines
508469
509470 return 0;
510471 }
472+
473+static const char *
474+invalidation_fence_get_driver_name(struct dma_fence *dma_fence)
475+{
476+ return "xe";
477+}
478+
479+static const char *
480+invalidation_fence_get_timeline_name(struct dma_fence *dma_fence)
481+{
482+ return "invalidation_fence";
483+}
484+
485+static const struct dma_fence_ops invalidation_fence_ops = {
486+ .get_driver_name = invalidation_fence_get_driver_name,
487+ .get_timeline_name = invalidation_fence_get_timeline_name,
488+};
489+
490+/**
491+ * xe_gt_tlb_invalidation_fence_init - Initialize TLB invalidation fence
492+ * @gt: GT
493+ * @fence: TLB invalidation fence to initialize
494+ * @stack: fence is stack variable
495+ *
496+ * Initialize TLB invalidation fence for use. xe_gt_tlb_invalidation_fence_fini
497+ * must be called if fence is not signaled.
498+ */
499+void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
500+ struct xe_gt_tlb_invalidation_fence *fence,
501+ bool stack)
502+{
503+ xe_pm_runtime_get_noresume(gt_to_xe(gt));
504+
505+ spin_lock_irq(&gt->tlb_invalidation.lock);
506+ dma_fence_init(&fence->base, &invalidation_fence_ops,
507+ &gt->tlb_invalidation.lock,
508+ dma_fence_context_alloc(1), 1);
509+ spin_unlock_irq(&gt->tlb_invalidation.lock);
510+ INIT_LIST_HEAD(&fence->link);
511+ if (stack)
512+ set_bit(FENCE_STACK_BIT, &fence->base.flags);
513+ else
514+ dma_fence_get(&fence->base);
515+ fence->gt = gt;
516+}
517+
518+/**
519+ * xe_gt_tlb_invalidation_fence_fini - Finalize TLB invalidation fence
520+ * @fence: TLB invalidation fence to finalize
521+ *
522+ * Drop PM ref which fence took durinig init.
523+ */
524+void xe_gt_tlb_invalidation_fence_fini(struct xe_gt_tlb_invalidation_fence *fence)
525+{
526+ xe_pm_runtime_put(gt_to_xe(fence->gt));
527+}
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h+11−1
@@ -23,7 +23,17 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt, · 22 unmodified lines
2323 int xe_gt_tlb_invalidation_range(struct xe_gt *gt,
2424 struct xe_gt_tlb_invalidation_fence *fence,
2525 u64 start, u64 end, u32 asid);
26−int xe_gt_tlb_invalidation_wait(struct xe_gt *gt, int seqno);
2726 int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len);
2827
28+void xe_gt_tlb_invalidation_fence_init(struct xe_gt *gt,
29+ struct xe_gt_tlb_invalidation_fence *fence,
30+ bool stack);
31+void xe_gt_tlb_invalidation_fence_fini(struct xe_gt_tlb_invalidation_fence *fence);
32+
33+static inline void
34+xe_gt_tlb_invalidation_fence_wait(struct xe_gt_tlb_invalidation_fence *fence)
35+{
36+ dma_fence_wait(&fence->base, false);
37+}
38+
2939 #endif /* _XE_GT_TLB_INVALIDATION_ */
drivers/gpu/drm/xe/xe_gt_tlb_invalidation_types.h+4−0
@@ -8,6 +8,8 @@ · 7 unmodified lines
88
99 #include <linux/dma-fence.h>
1010
11+struct xe_gt;
12+
1113 /**
1214 * struct xe_gt_tlb_invalidation_fence - XE GT TLB invalidation fence
1315 *
@@ -17,6 +19,8 @@ · 3 unmodified lines
1719 struct xe_gt_tlb_invalidation_fence {
1820 /** @base: dma fence base */
1921 struct dma_fence base;
22+ /** @gt: GT which fence belong to */
23+ struct xe_gt *gt;
2024 /** @link: link into list of pending tlb fences */
2125 struct list_head link;
2226 /** @seqno: seqno of TLB invalidation to signal fence one */
drivers/gpu/drm/xe/xe_gt_topology.c+22−5
@@ -6,6 +6,7 @@ · 5 unmodified lines
66 #include "xe_gt_topology.h"
77
88 #include <linux/bitmap.h>
9+#include <linux/compiler.h>
910
1011 #include "regs/xe_gt_regs.h"
1112 #include "xe_assert.h"
@@ -31,7 +32,7 @@ load_dss_mask(struct xe_gt *gt, xe_dss_mask_t mask, int numregs, ...) · 19 unmodified lines
3132 }
3233
3334 static void
34−load_eu_mask(struct xe_gt *gt, xe_eu_mask_t mask)
35+load_eu_mask(struct xe_gt *gt, xe_eu_mask_t mask, enum xe_gt_eu_type *eu_type)
3536 {
3637 struct xe_device *xe = gt_to_xe(gt);
3738 u32 reg_val = xe_mmio_read32(gt, XELP_EU_ENABLE);
@@ -47,11 +48,13 @@ load_eu_mask(struct xe_gt *gt, xe_eu_mask_t mask) · 9 unmodified lines
4748 if (GRAPHICS_VERx100(xe) < 1250)
4849 reg_val = ~reg_val & XELP_EU_MASK;
4950
50− /* On PVC, one bit = one EU */
51− if (GRAPHICS_VERx100(xe) == 1260) {
51+ if (GRAPHICS_VERx100(xe) == 1260 || GRAPHICS_VER(xe) >= 20) {
52+ /* SIMD16 EUs, one bit == one EU */
53+ *eu_type = XE_GT_EU_TYPE_SIMD16;
5254 val = reg_val;
5355 } else {
54− /* All other platforms, one bit = 2 EU */
56+ /* SIMD8 EUs, one bit == 2 EU */
57+ *eu_type = XE_GT_EU_TYPE_SIMD8;
5558 for (i = 0; i < fls(reg_val); i++)
5659 if (reg_val & BIT(i))
5760 val |= 0x3 << 2 * i;
@@ -213,7 +216,7 @@ xe_gt_topology_init(struct xe_gt *gt) · 155 unmodified lines
213216 XEHP_GT_COMPUTE_DSS_ENABLE,
214217 XEHPC_GT_COMPUTE_DSS_ENABLE_EXT,
215218 XE2_GT_COMPUTE_DSS_2);
216− load_eu_mask(gt, gt->fuse_topo.eu_mask_per_dss);
219+ load_eu_mask(gt, gt->fuse_topo.eu_mask_per_dss, &gt->fuse_topo.eu_type);
217220 load_l3_bank_mask(gt, gt->fuse_topo.l3_bank_mask);
218221
219222 p = drm_dbg_printer(&gt_to_xe(gt)->drm, DRM_UT_DRIVER, "GT topology");
@@ -221,6 +224,18 @@ xe_gt_topology_init(struct xe_gt *gt) · 1 unmodified line
221224 xe_gt_topology_dump(gt, &p);
222225 }
223226
227+static const char *eu_type_to_str(enum xe_gt_eu_type eu_type)
228+{
229+ switch (eu_type) {
230+ case XE_GT_EU_TYPE_SIMD16:
231+ return "simd16";
232+ case XE_GT_EU_TYPE_SIMD8:
233+ return "simd8";
234+ }
235+
236+ return NULL;
237+}
238+
224239 void
225240 xe_gt_topology_dump(struct xe_gt *gt, struct drm_printer *p)
226241 {
@@ -231,6 +246,8 @@ xe_gt_topology_dump(struct xe_gt *gt, struct drm_printer *p) · 4 unmodified lines
231246
232247 drm_printf(p, "EU mask per DSS: %*pb\n", XE_MAX_EU_FUSE_BITS,
233248 gt->fuse_topo.eu_mask_per_dss);
249+ drm_printf(p, "EU type: %s\n",
250+ eu_type_to_str(gt->fuse_topo.eu_type));
234251
235252 drm_printf(p, "L3 bank mask: %*pb\n", XE_MAX_L3_BANK_MASK_BITS,
236253 gt->fuse_topo.l3_bank_mask);
drivers/gpu/drm/xe/xe_gt_types.h+24−1
@@ -27,6 +27,11 @@ enum xe_gt_type { · 26 unmodified lines
2727 XE_GT_TYPE_MEDIA,
2828 };
2929
30+enum xe_gt_eu_type {
31+ XE_GT_EU_TYPE_SIMD8,
32+ XE_GT_EU_TYPE_SIMD16,
33+};
34+
3035 #define XE_MAX_DSS_FUSE_REGS 3
3136 #define XE_MAX_DSS_FUSE_BITS (32 * XE_MAX_DSS_FUSE_REGS)
3237 #define XE_MAX_EU_FUSE_REGS 1
@@ -343,6 +348,12 @@ struct xe_gt { · 310 unmodified lines
343348
344349 /** @fuse_topo.l3_bank_mask: L3 bank mask */
345350 xe_l3_bank_mask_t l3_bank_mask;
351+
352+ /**
353+ * @fuse_topo.eu_type: type/width of EU stored in
354+ * fuse_topo.eu_mask_per_dss
355+ */
356+ enum xe_gt_eu_type eu_type;
346357 } fuse_topo;
347358
348359 /** @steering: register steering for individual HW units */
@@ -362,6 +373,12 @@ struct xe_gt { · 13 unmodified lines
362373 */
363374 spinlock_t mcr_lock;
364375
376+ /**
377+ * @global_invl_lock: protects the register for the duration
378+ * of a global invalidation of l2 cache
379+ */
380+ spinlock_t global_invl_lock;
381+
365382 /** @wa_active: keep track of active workarounds */
366383 struct {
367384 /** @wa_active.gt: bitmap with active GT workarounds */
@@ -370,8 +387,14 @@ struct xe_gt { · 2 unmodified lines
370387 unsigned long *engine;
371388 /** @wa_active.lrc: bitmap with active LRC workarounds */
372389 unsigned long *lrc;
373− /** @wa_active.oob: bitmap with active OOB workaroudns */
390+ /** @wa_active.oob: bitmap with active OOB workarounds */
374391 unsigned long *oob;
392+ /**
393+ * @wa_active.oob_initialized: mark oob as initialized to help
394+ * detecting misuse of XE_WA() - it can only be called on
395+ * initialization after OOB WAs have being processed
396+ */
397+ bool oob_initialized;
375398 } wa_active;
376399
377400 /** @user_engines: engines present in GT and available to userspace */
drivers/gpu/drm/xe/xe_guc_ct.c+10−1
@@ -327,6 +327,8 @@ static void xe_guc_ct_set_state(struct xe_guc_ct *ct, · 326 unmodified lines
327327 xe_gt_assert(ct_to_gt(ct), ct->g2h_outstanding == 0 ||
328328 state == XE_GUC_CT_STATE_STOPPED);
329329
330+ if (ct->g2h_outstanding)
331+ xe_pm_runtime_put(ct_to_xe(ct));
330332 ct->g2h_outstanding = 0;
331333 ct->state = state;
332334
@@ -495,10 +497,15 @@ static void h2g_reserve_space(struct xe_guc_ct *ct, u32 cmd_len) · 162 unmodified lines
495497 static void __g2h_reserve_space(struct xe_guc_ct *ct, u32 g2h_len, u32 num_g2h)
496498 {
497499 xe_gt_assert(ct_to_gt(ct), g2h_len <= ct->ctbs.g2h.info.space);
500+ xe_gt_assert(ct_to_gt(ct), (!g2h_len && !num_g2h) ||
501+ (g2h_len && num_g2h));
498502
499503 if (g2h_len) {
500504 lockdep_assert_held(&ct->fast_lock);
501505
506+ if (!ct->g2h_outstanding)
507+ xe_pm_runtime_get_noresume(ct_to_xe(ct));
508+
502509 ct->ctbs.g2h.info.space -= g2h_len;
503510 ct->g2h_outstanding += num_g2h;
504511 }
@@ -509,9 +516,11 @@ static void __g2h_release_space(struct xe_guc_ct *ct, u32 g2h_len) · 4 unmodified lines
509516 lockdep_assert_held(&ct->fast_lock);
510517 xe_gt_assert(ct_to_gt(ct), ct->ctbs.g2h.info.space + g2h_len <=
511518 ct->ctbs.g2h.info.size - ct->ctbs.g2h.info.resv_space);
519+ xe_gt_assert(ct_to_gt(ct), ct->g2h_outstanding);
512520
513521 ct->ctbs.g2h.info.space += g2h_len;
514− --ct->g2h_outstanding;
522+ if (!--ct->g2h_outstanding)
523+ xe_pm_runtime_put(ct_to_xe(ct));
515524 }
516525
517526 static void g2h_release_space(struct xe_guc_ct *ct, u32 g2h_len)
drivers/gpu/drm/xe/xe_guc_id_mgr.c+2−2
@@ -97,8 +97,8 @@ int xe_guc_id_mgr_init(struct xe_guc_id_mgr *idm, unsigned int limit) · 96 unmodified lines
9797 if (ret)
9898 return ret;
9999
100− xe_gt_info(idm_to_gt(idm), "using %u GUC ID%s\n",
101− idm->total, str_plural(idm->total));
100+ xe_gt_dbg(idm_to_gt(idm), "using %u GuC ID%s\n",
101+ idm->total, str_plural(idm->total));
102102 return 0;
103103 }
104104
drivers/gpu/drm/xe/xe_guc_submit.c+48−8
@@ -1071,7 +1071,9 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job) · 1070 unmodified lines
10711071 struct xe_exec_queue *q = job->q;
10721072 struct xe_gpu_scheduler *sched = &q->guc->sched;
10731073 struct xe_guc *guc = exec_queue_to_guc(q);
1074+ const char *process_name = "no process";
10741075 int err = -ETIME;
1076+ pid_t pid = -1;
10751077 int i = 0;
10761078 bool wedged, skip_timeout_check;
10771079
@@ -1168,9 +1170,14 @@ trigger_reset: · 90 unmodified lines
11681170 goto sched_enable;
11691171 }
11701172
1171− xe_gt_notice(guc_to_gt(guc), "Timedout job: seqno=%u, lrc_seqno=%u, guc_id=%d, flags=0x%lx",
1173+ if (q->vm && q->vm->xef) {
1174+ process_name = q->vm->xef->process_name;
1175+ pid = q->vm->xef->pid;
1176+ }
1177+ xe_gt_notice(guc_to_gt(guc), "Timedout job: seqno=%u, lrc_seqno=%u, guc_id=%d, flags=0x%lx in %s [%d]",
11721178 xe_sched_job_seqno(job), xe_sched_job_lrc_seqno(job),
1173− q->guc->id, q->flags);
1179+ q->guc->id, q->flags, process_name, pid);
1180+
11741181 trace_xe_sched_job_timedout(job);
11751182
11761183 if (!exec_queue_killed(q))
@@ -1312,6 +1319,15 @@ static void __guc_exec_queue_process_msg_set_sched_props(struct xe_sched_msg *ms · 135 unmodified lines
13121319 kfree(msg);
13131320 }
13141321
1322+static void __suspend_fence_signal(struct xe_exec_queue *q)
1323+{
1324+ if (!q->guc->suspend_pending)
1325+ return;
1326+
1327+ WRITE_ONCE(q->guc->suspend_pending, false);
1328+ wake_up(&q->guc->suspend_wait);
1329+}
1330+
13151331 static void suspend_fence_signal(struct xe_exec_queue *q)
13161332 {
13171333 struct xe_guc *guc = exec_queue_to_guc(q);
@@ -1321,9 +1337,7 @@ static void suspend_fence_signal(struct xe_exec_queue *q) · 3 unmodified lines
13211337 guc_read_stopped(guc));
13221338 xe_assert(xe, q->guc->suspend_pending);
13231339
1324− q->guc->suspend_pending = false;
1325− smp_wmb();
1326− wake_up(&q->guc->suspend_wait);
1340+ __suspend_fence_signal(q);
13271341 }
13281342
13291343 static void __guc_exec_queue_process_msg_suspend(struct xe_sched_msg *msg)
@@ -1375,6 +1389,8 @@ static void __guc_exec_queue_process_msg_resume(struct xe_sched_msg *msg) · 45 unmodified lines
13751389
13761390 static void guc_exec_queue_process_msg(struct xe_sched_msg *msg)
13771391 {
1392+ struct xe_device *xe = guc_to_xe(exec_queue_to_guc(msg->private_data));
1393+
13781394 trace_xe_sched_msg_recv(msg);
13791395
13801396 switch (msg->opcode) {
@@ -1393,6 +1409,8 @@ static void guc_exec_queue_process_msg(struct xe_sched_msg *msg) · 12 unmodified lines
13931409 default:
13941410 XE_WARN_ON("Unknown message type");
13951411 }
1412+
1413+ xe_pm_runtime_put(xe);
13961414 }
13971415
13981416 static const struct drm_sched_backend_ops drm_sched_ops = {
@@ -1476,12 +1494,15 @@ static void guc_exec_queue_kill(struct xe_exec_queue *q) · 77 unmodified lines
14761494 {
14771495 trace_xe_exec_queue_kill(q);
14781496 set_exec_queue_killed(q);
1497+ __suspend_fence_signal(q);
14791498 xe_guc_exec_queue_trigger_cleanup(q);
14801499 }
14811500
14821501 static void guc_exec_queue_add_msg(struct xe_exec_queue *q, struct xe_sched_msg *msg,
14831502 u32 opcode)
14841503 {
1504+ xe_pm_runtime_get_noresume(guc_to_xe(exec_queue_to_guc(q)));
1505+
14851506 INIT_LIST_HEAD(&msg->link);
14861507 msg->opcode = opcode;
14871508 msg->private_data = q;
@@ -1572,12 +1593,31 @@ static int guc_exec_queue_suspend(struct xe_exec_queue *q) · 84 unmodified lines
15721593 return 0;
15731594 }
15741595
1575−static void guc_exec_queue_suspend_wait(struct xe_exec_queue *q)
1596+static int guc_exec_queue_suspend_wait(struct xe_exec_queue *q)
15761597 {
15771598 struct xe_guc *guc = exec_queue_to_guc(q);
1599+ int ret;
1600+
1601+ /*
1602+ * Likely don't need to check exec_queue_killed() as we clear
1603+ * suspend_pending upon kill but to be paranoid but races in which
1604+ * suspend_pending is set after kill also check kill here.
1605+ */
1606+ ret = wait_event_timeout(q->guc->suspend_wait,
1607+ !READ_ONCE(q->guc->suspend_pending) ||
1608+ exec_queue_killed(q) ||
1609+ guc_read_stopped(guc),
1610+ HZ * 5);
15781611
1579− wait_event(q->guc->suspend_wait, !q->guc->suspend_pending ||
1580− guc_read_stopped(guc));
1612+ if (!ret) {
1613+ xe_gt_warn(guc_to_gt(guc),
1614+ "Suspend fence, guc_id=%d, failed to respond",
1615+ q->guc->id);
1616+ /* XXX: Trigger GT reset? */
1617+ return -ETIME;
1618+ }
1619+
1620+ return 0;
15811621 }
15821622
15831623 static void guc_exec_queue_resume(struct xe_exec_queue *q)
drivers/gpu/drm/xe/xe_heci_gsc.c+25−3
@@ -92,7 +92,7 @@ void xe_heci_gsc_fini(struct xe_device *xe) · 91 unmodified lines
9292 {
9393 struct xe_heci_gsc *heci_gsc = &xe->heci_gsc;
9494
95− if (!HAS_HECI_GSCFI(xe))
95+ if (!HAS_HECI_GSCFI(xe) && !HAS_HECI_CSCFI(xe))
9696 return;
9797
9898 if (heci_gsc->adev) {
@@ -177,12 +177,14 @@ void xe_heci_gsc_init(struct xe_device *xe) · 78 unmodified lines
177177 const struct heci_gsc_def *def;
178178 int ret;
179179
180− if (!HAS_HECI_GSCFI(xe))
180+ if (!HAS_HECI_GSCFI(xe) && !HAS_HECI_CSCFI(xe))
181181 return;
182182
183183 heci_gsc->irq = -1;
184184
185− if (xe->info.platform == XE_PVC) {
185+ if (xe->info.platform == XE_BATTLEMAGE) {
186+ def = &heci_gsc_def_dg2;
187+ } else if (xe->info.platform == XE_PVC) {
186188 def = &heci_gsc_def_pvc;
187189 } else if (xe->info.platform == XE_DG2) {
188190 def = &heci_gsc_def_dg2;
@@ -232,3 +234,23 @@ void xe_heci_gsc_irq_handler(struct xe_device *xe, u32 iir) · 43 unmodified lines
232234 if (ret)
233235 drm_err_ratelimited(&xe->drm, "error handling GSC irq: %d\n", ret);
234236 }
237+
238+void xe_heci_csc_irq_handler(struct xe_device *xe, u32 iir)
239+{
240+ int ret;
241+
242+ if ((iir & CSC_IRQ_INTF(1)) == 0)
243+ return;
244+
245+ if (!HAS_HECI_CSCFI(xe)) {
246+ drm_warn_once(&xe->drm, "CSC irq: not supported");
247+ return;
248+ }
249+
250+ if (xe->heci_gsc.irq < 0)
251+ return;
252+
253+ ret = generic_handle_irq(xe->heci_gsc.irq);
254+ if (ret)
255+ drm_err_ratelimited(&xe->drm, "error handling GSC irq: %d\n", ret);
256+}
drivers/gpu/drm/xe/xe_heci_gsc.h+8−2
@@ -11,10 +11,15 @@ struct xe_device; · 10 unmodified lines
1111 struct mei_aux_device;
1212
1313 /*
14− * The HECI1 bit corresponds to bit15 and HECI2 to bit14.
14+ * GSC HECI1 bit corresponds to bit15 and HECI2 to bit14.
1515 * The reason for this is to allow growth for more interfaces in the future.
1616 */
17−#define GSC_IRQ_INTF(_x) BIT(15 - (_x))
17+#define GSC_IRQ_INTF(_x) BIT(15 - (_x))
18+
19+/*
20+ * CSC HECI1 bit corresponds to bit9 and HECI2 to bit10.
21+ */
22+#define CSC_IRQ_INTF(_x) BIT(9 + (_x))
1823
1924 /**
2025 * struct xe_heci_gsc - graphics security controller for xe, HECI interface
@@ -31,5 +36,6 @@ struct xe_heci_gsc { · 10 unmodified lines
3136 void xe_heci_gsc_init(struct xe_device *xe);
3237 void xe_heci_gsc_fini(struct xe_device *xe);
3338 void xe_heci_gsc_irq_handler(struct xe_device *xe, u32 iir);
39+void xe_heci_csc_irq_handler(struct xe_device *xe, u32 iir);
3440
3541 #endif /* __XE_HECI_GSC_DEV_H__ */
drivers/gpu/drm/xe/xe_irq.c+2−0
@@ -459,6 +459,8 @@ static irqreturn_t dg1_irq_handler(int irq, void *arg) · 458 unmodified lines
459459 * the primary tile.
460460 */
461461 if (id == 0) {
462+ if (HAS_HECI_CSCFI(xe))
463+ xe_heci_csc_irq_handler(xe, master_ctl);
462464 xe_display_irq_handler(xe, master_ctl);
463465 gu_misc_iir = gu_misc_irq_ack(xe, master_ctl);
464466 }
drivers/gpu/drm/xe/xe_lmtt.c+2−2
@@ -7,7 +7,7 @@ · 6 unmodified lines
77
88 #include <drm/drm_managed.h>
99
10−#include "regs/xe_sriov_regs.h"
10+#include "regs/xe_gt_regs.h"
1111
1212 #include "xe_assert.h"
1313 #include "xe_bo.h"
@@ -71,7 +71,7 @@ static struct xe_lmtt_pt *lmtt_pt_alloc(struct xe_lmtt *lmtt, unsigned int level · 57 unmodified lines
7171 lmtt->ops->lmtt_pte_num(level)),
7272 ttm_bo_type_kernel,
7373 XE_BO_FLAG_VRAM_IF_DGFX(lmtt_to_tile(lmtt)) |
74− XE_BO_NEEDS_64K | XE_BO_FLAG_PINNED);
74+ XE_BO_FLAG_NEEDS_64K | XE_BO_FLAG_PINNED);
7575 if (IS_ERR(bo)) {
7676 err = PTR_ERR(bo);
7777 goto out_free_pt;
drivers/gpu/drm/xe/xe_migrate.c+290−238
@@ -73,6 +73,7 @@ struct xe_migrate { · 72 unmodified lines
7373 #define NUM_PT_SLOTS 32
7474 #define LEVEL0_PAGE_TABLE_ENCODE_SIZE SZ_2M
7575 #define MAX_NUM_PTE 512
76+#define IDENTITY_OFFSET 256ULL
7677
7778 /*
7879 * Although MI_STORE_DATA_IMM's "length" field is 10-bits, 0x3FE is the largest
@@ -84,15 +85,14 @@ struct xe_migrate { · 5 unmodified lines
8485 #define MAX_PTE_PER_SDI 0x1FE
8586
8687 /**
87− * xe_tile_migrate_engine() - Get this tile's migrate engine.
88+ * xe_tile_migrate_exec_queue() - Get this tile's migrate exec queue.
8889 * @tile: The tile.
8990 *
90− * Returns the default migrate engine of this tile.
91− * TODO: Perhaps this function is slightly misplaced, and even unneeded?
91+ * Returns the default migrate exec queue of this tile.
9292 *
93− * Return: The default migrate engine
93+ * Return: The default migrate exec queue
9494 */
95−struct xe_exec_queue *xe_tile_migrate_engine(struct xe_tile *tile)
95+struct xe_exec_queue *xe_tile_migrate_exec_queue(struct xe_tile *tile)
9696 {
9797 return tile->migrate->q;
9898 }
@@ -121,14 +121,64 @@ static u64 xe_migrate_vm_addr(u64 slot, u32 level) · 22 unmodified lines
121121 return (slot + 1ULL) << xe_pt_shift(level + 1);
122122 }
123123
124−static u64 xe_migrate_vram_ofs(struct xe_device *xe, u64 addr)
124+static u64 xe_migrate_vram_ofs(struct xe_device *xe, u64 addr, bool is_comp_pte)
125125 {
126126 /*
127127 * Remove the DPA to get a correct offset into identity table for the
128128 * migrate offset
129129 */
130+ u64 identity_offset = IDENTITY_OFFSET;
131+
132+ if (GRAPHICS_VER(xe) >= 20 && is_comp_pte)
133+ identity_offset += DIV_ROUND_UP_ULL(xe->mem.vram.actual_physical_size, SZ_1G);
134+
130135 addr -= xe->mem.vram.dpa_base;
131− return addr + (256ULL << xe_pt_shift(2));
136+ return addr + (identity_offset << xe_pt_shift(2));
137+}
138+
139+static void xe_migrate_program_identity(struct xe_device *xe, struct xe_vm *vm, struct xe_bo *bo,
140+ u64 map_ofs, u64 vram_offset, u16 pat_index, u64 pt_2m_ofs)
141+{
142+ u64 pos, ofs, flags;
143+ u64 entry;
144+ /* XXX: Unclear if this should be usable_size? */
145+ u64 vram_limit = xe->mem.vram.actual_physical_size +
146+ xe->mem.vram.dpa_base;
147+ u32 level = 2;
148+
149+ ofs = map_ofs + XE_PAGE_SIZE * level + vram_offset * 8;
150+ flags = vm->pt_ops->pte_encode_addr(xe, 0, pat_index, level,
151+ true, 0);
152+
153+ xe_assert(xe, IS_ALIGNED(xe->mem.vram.usable_size, SZ_2M));
154+
155+ /*
156+ * Use 1GB pages when possible, last chunk always use 2M
157+ * pages as mixing reserved memory (stolen, WOCPM) with a single
158+ * mapping is not allowed on certain platforms.
159+ */
160+ for (pos = xe->mem.vram.dpa_base; pos < vram_limit;
161+ pos += SZ_1G, ofs += 8) {
162+ if (pos + SZ_1G >= vram_limit) {
163+ entry = vm->pt_ops->pde_encode_bo(bo, pt_2m_ofs,
164+ pat_index);
165+ xe_map_wr(xe, &bo->vmap, ofs, u64, entry);
166+
167+ flags = vm->pt_ops->pte_encode_addr(xe, 0,
168+ pat_index,
169+ level - 1,
170+ true, 0);
171+
172+ for (ofs = pt_2m_ofs; pos < vram_limit;
173+ pos += SZ_2M, ofs += 8)
174+ xe_map_wr(xe, &bo->vmap, ofs, u64, pos | flags);
175+ break; /* Ensure pos == vram_limit assert correct */
176+ }
177+
178+ xe_map_wr(xe, &bo->vmap, ofs, u64, pos | flags);
179+ }
180+
181+ xe_assert(xe, pos == vram_limit);
132182 }
133183
134184 static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,
@@ -137,11 +187,13 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, · 2 unmodified lines
137187 struct xe_device *xe = tile_to_xe(tile);
138188 u16 pat_index = xe->pat.idx[XE_CACHE_WB];
139189 u8 id = tile->id;
140− u32 num_entries = NUM_PT_SLOTS, num_level = vm->pt_root[id]->level,
141− num_setup = num_level + 1;
190+ u32 num_entries = NUM_PT_SLOTS, num_level = vm->pt_root[id]->level;
191+#define VRAM_IDENTITY_MAP_COUNT 2
192+ u32 num_setup = num_level + VRAM_IDENTITY_MAP_COUNT;
193+#undef VRAM_IDENTITY_MAP_COUNT
142194 u32 map_ofs, level, i;
143195 struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo;
144− u64 entry, pt30_ofs;
196+ u64 entry, pt29_ofs;
145197
146198 /* Can't bump NUM_PT_SLOTS too high */
147199 BUILD_BUG_ON(NUM_PT_SLOTS > SZ_2M/XE_PAGE_SIZE);
@@ -161,9 +213,9 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, · 13 unmodified lines
161213 if (IS_ERR(bo))
162214 return PTR_ERR(bo);
163215
164− /* PT31 reserved for 2M identity map */
165− pt30_ofs = bo->size - 2 * XE_PAGE_SIZE;
166− entry = vm->pt_ops->pde_encode_bo(bo, pt30_ofs, pat_index);
216+ /* PT30 & PT31 reserved for 2M identity map */
217+ pt29_ofs = bo->size - 3 * XE_PAGE_SIZE;
218+ entry = vm->pt_ops->pde_encode_bo(bo, pt29_ofs, pat_index);
167219 xe_pt_write(xe, &vm->pt_root[id]->bo->vmap, 0, entry);
168220
169221 map_ofs = (num_entries - num_setup) * XE_PAGE_SIZE;
@@ -215,12 +267,12 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, · 45 unmodified lines
215267 } else {
216268 u64 batch_addr = xe_bo_addr(batch, 0, XE_PAGE_SIZE);
217269
218− m->batch_base_ofs = xe_migrate_vram_ofs(xe, batch_addr);
270+ m->batch_base_ofs = xe_migrate_vram_ofs(xe, batch_addr, false);
219271
220272 if (xe->info.has_usm) {
221273 batch = tile->primary_gt->usm.bb_pool->bo;
222274 batch_addr = xe_bo_addr(batch, 0, XE_PAGE_SIZE);
223− m->usm_batch_base_ofs = xe_migrate_vram_ofs(xe, batch_addr);
275+ m->usm_batch_base_ofs = xe_migrate_vram_ofs(xe, batch_addr, false);
224276 }
225277 }
226278
@@ -254,55 +306,36 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, · 27 unmodified lines
254306
255307 /* Identity map the entire vram at 256GiB offset */
256308 if (IS_DGFX(xe)) {
257− u64 pos, ofs, flags;
258− /* XXX: Unclear if this should be usable_size? */
259− u64 vram_limit = xe->mem.vram.actual_physical_size +
260− xe->mem.vram.dpa_base;
309+ u64 pt30_ofs = bo->size - 2 * XE_PAGE_SIZE;
261310
262− level = 2;
263− ofs = map_ofs + XE_PAGE_SIZE * level + 256 * 8;
264− flags = vm->pt_ops->pte_encode_addr(xe, 0, pat_index, level,
265− true, 0);
266−
267− xe_assert(xe, IS_ALIGNED(xe->mem.vram.usable_size, SZ_2M));
311+ xe_migrate_program_identity(xe, vm, bo, map_ofs, IDENTITY_OFFSET,
312+ pat_index, pt30_ofs);
313+ xe_assert(xe, xe->mem.vram.actual_physical_size <=
314+ (MAX_NUM_PTE - IDENTITY_OFFSET) * SZ_1G);
268315
269316 /*
270− * Use 1GB pages when possible, last chunk always use 2M
271− * pages as mixing reserved memory (stolen, WOCPM) with a single
272− * mapping is not allowed on certain platforms.
317+ * Identity map the entire vram for compressed pat_index for xe2+
318+ * if flat ccs is enabled.
273319 */
274− for (pos = xe->mem.vram.dpa_base; pos < vram_limit;
275− pos += SZ_1G, ofs += 8) {
276− if (pos + SZ_1G >= vram_limit) {
277− u64 pt31_ofs = bo->size - XE_PAGE_SIZE;
278−
279− entry = vm->pt_ops->pde_encode_bo(bo, pt31_ofs,
280− pat_index);
281− xe_map_wr(xe, &bo->vmap, ofs, u64, entry);
282−
283− flags = vm->pt_ops->pte_encode_addr(xe, 0,
284− pat_index,
285− level - 1,
286− true, 0);
287−
288− for (ofs = pt31_ofs; pos < vram_limit;
289− pos += SZ_2M, ofs += 8)
290− xe_map_wr(xe, &bo->vmap, ofs, u64, pos | flags);
291− break; /* Ensure pos == vram_limit assert correct */
292− }
293−
294− xe_map_wr(xe, &bo->vmap, ofs, u64, pos | flags);
320+ if (GRAPHICS_VER(xe) >= 20 && xe_device_has_flat_ccs(xe)) {
321+ u16 comp_pat_index = xe->pat.idx[XE_CACHE_NONE_COMPRESSION];
322+ u64 vram_offset = IDENTITY_OFFSET +
323+ DIV_ROUND_UP_ULL(xe->mem.vram.actual_physical_size, SZ_1G);
324+ u64 pt31_ofs = bo->size - XE_PAGE_SIZE;
325+
326+ xe_assert(xe, xe->mem.vram.actual_physical_size <= (MAX_NUM_PTE -
327+ IDENTITY_OFFSET - IDENTITY_OFFSET / 2) * SZ_1G);
328+ xe_migrate_program_identity(xe, vm, bo, map_ofs, vram_offset,
329+ comp_pat_index, pt31_ofs);
295330 }
296−
297− xe_assert(xe, pos == vram_limit);
298331 }
299332
300333 /*
301334 * Example layout created above, with root level = 3:
302335 * [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's
303336 * [PT8]: Kernel PT for VM_BIND, 4 KiB PTE's
304− * [PT9...PT27]: Userspace PT's for VM_BIND, 4 KiB PTE's
305− * [PT28 = PDE 0] [PT29 = PDE 1] [PT30 = PDE 2] [PT31 = 2M vram identity map]
337+ * [PT9...PT26]: Userspace PT's for VM_BIND, 4 KiB PTE's
338+ * [PT27 = PDE 0] [PT28 = PDE 1] [PT29 = PDE 2] [PT30 & PT31 = 2M vram identity map]
306339 *
307340 * This makes the lowest part of the VM point to the pagetables.
308341 * Hence the lowest 2M in the vm should point to itself, with a few writes
@@ -348,6 +381,11 @@ static u32 xe_migrate_usm_logical_mask(struct xe_gt *gt) · 39 unmodified lines
348381 return logical_mask;
349382 }
350383
384+static bool xe_migrate_needs_ccs_emit(struct xe_device *xe)
385+{
386+ return xe_device_has_flat_ccs(xe) && !(GRAPHICS_VER(xe) >= 20 && IS_DGFX(xe));
387+}
388+
351389 /**
352390 * xe_migrate_init() - Initialize a migrate context
353391 * @tile: Back-pointer to the tile we're initializing for.
@@ -421,7 +459,7 @@ struct xe_migrate *xe_migrate_init(struct xe_tile *tile) · 67 unmodified lines
421459 return ERR_PTR(err);
422460
423461 if (IS_DGFX(xe)) {
424− if (xe_device_has_flat_ccs(xe))
462+ if (xe_migrate_needs_ccs_emit(xe))
425463 /* min chunk size corresponds to 4K of CCS Metadata */
426464 m->min_chunk_size = SZ_4K * SZ_64K /
427465 xe_device_ccs_bytes(xe, SZ_64K);
@@ -475,20 +513,26 @@ static bool xe_migrate_allow_identity(u64 size, const struct xe_res_cursor *cur) · 47 unmodified lines
475513 return cur->size >= size;
476514 }
477515
516+#define PTE_UPDATE_FLAG_IS_VRAM BIT(0)
517+#define PTE_UPDATE_FLAG_IS_COMP_PTE BIT(1)
518+
478519 static u32 pte_update_size(struct xe_migrate *m,
479− bool is_vram,
520+ u32 flags,
480521 struct ttm_resource *res,
481522 struct xe_res_cursor *cur,
482523 u64 *L0, u64 *L0_ofs, u32 *L0_pt,
483524 u32 cmd_size, u32 pt_ofs, u32 avail_pts)
484525 {
485526 u32 cmds = 0;
527+ bool is_vram = PTE_UPDATE_FLAG_IS_VRAM & flags;
528+ bool is_comp_pte = PTE_UPDATE_FLAG_IS_COMP_PTE & flags;
486529
487530 *L0_pt = pt_ofs;
488531 if (is_vram && xe_migrate_allow_identity(*L0, cur)) {
489532 /* Offset into identity map. */
490533 *L0_ofs = xe_migrate_vram_ofs(tile_to_xe(m->tile),
491− cur->start + vram_region_gpu_offset(res));
534+ cur->start + vram_region_gpu_offset(res),
535+ is_comp_pte);
492536 cmds += cmd_size;
493537 } else {
494538 /* Clip L0 to available size */
@@ -661,7 +705,7 @@ static u32 xe_migrate_ccs_copy(struct xe_migrate *m, · 166 unmodified lines
661705 struct xe_gt *gt = m->tile->primary_gt;
662706 u32 flush_flags = 0;
663707
664− if (xe_device_has_flat_ccs(gt_to_xe(gt)) && !copy_ccs && dst_is_indirect) {
708+ if (!copy_ccs && dst_is_indirect) {
665709 /*
666710 * If the src is already in vram, then it should already
667711 * have been cleared by us, or has been populated by the
@@ -737,6 +781,8 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · 69 unmodified lines
737781 bool copy_ccs = xe_device_has_flat_ccs(xe) &&
738782 xe_bo_needs_ccs_pages(src_bo) && xe_bo_needs_ccs_pages(dst_bo);
739783 bool copy_system_ccs = copy_ccs && (!src_is_vram || !dst_is_vram);
784+ bool use_comp_pat = xe_device_has_flat_ccs(xe) &&
785+ GRAPHICS_VER(xe) >= 20 && src_is_vram && !dst_is_vram;
740786
741787 /* Copying CCS between two different BOs is not supported yet. */
742788 if (XE_WARN_ON(copy_ccs && src_bo != dst_bo))
@@ -763,10 +809,11 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · 20 unmodified lines
763809 u32 batch_size = 2; /* arb_clear() + MI_BATCH_BUFFER_END */
764810 struct xe_sched_job *job;
765811 struct xe_bb *bb;
766− u32 flush_flags;
812+ u32 flush_flags = 0;
767813 u32 update_idx;
768814 u64 ccs_ofs, ccs_size;
769815 u32 ccs_pt;
816+ u32 pte_flags;
770817
771818 bool usm = xe->info.has_usm;
772819 u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
@@ -779,17 +826,20 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · 6 unmodified lines
779826
780827 src_L0 = min(src_L0, dst_L0);
781828
782− batch_size += pte_update_size(m, src_is_vram, src, &src_it, &src_L0,
829+ pte_flags = src_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
830+ pte_flags |= use_comp_pat ? PTE_UPDATE_FLAG_IS_COMP_PTE : 0;
831+ batch_size += pte_update_size(m, pte_flags, src, &src_it, &src_L0,
783832 &src_L0_ofs, &src_L0_pt, 0, 0,
784833 avail_pts);
785834
786− batch_size += pte_update_size(m, dst_is_vram, dst, &dst_it, &src_L0,
835+ pte_flags = dst_is_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
836+ batch_size += pte_update_size(m, pte_flags, dst, &dst_it, &src_L0,
787837 &dst_L0_ofs, &dst_L0_pt, 0,
788838 avail_pts, avail_pts);
789839
790840 if (copy_system_ccs) {
791841 ccs_size = xe_device_ccs_bytes(xe, src_L0);
792− batch_size += pte_update_size(m, false, NULL, &ccs_it, &ccs_size,
842+ batch_size += pte_update_size(m, 0, NULL, &ccs_it, &ccs_size,
793843 &ccs_ofs, &ccs_pt, 0,
794844 2 * avail_pts,
795845 avail_pts);
@@ -798,7 +848,7 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · 2 unmodified lines
798848
799849 /* Add copy commands size here */
800850 batch_size += ((copy_only_ccs) ? 0 : EMIT_COPY_DW) +
801− ((xe_device_has_flat_ccs(xe) ? EMIT_COPY_CCS_DW : 0));
851+ ((xe_migrate_needs_ccs_emit(xe) ? EMIT_COPY_CCS_DW : 0));
802852
803853 bb = xe_bb_new(gt, batch_size, usm);
804854 if (IS_ERR(bb)) {
@@ -827,11 +877,12 @@ struct dma_fence *xe_migrate_copy(struct xe_migrate *m, · 22 unmodified lines
827877 if (!copy_only_ccs)
828878 emit_copy(gt, bb, src_L0_ofs, dst_L0_ofs, src_L0, XE_PAGE_SIZE);
829879
830− flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
831− IS_DGFX(xe) ? src_is_vram : src_is_pltt,
832− dst_L0_ofs,
833− IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
834− src_L0, ccs_ofs, copy_ccs);
880+ if (xe_migrate_needs_ccs_emit(xe))
881+ flush_flags = xe_migrate_ccs_copy(m, bb, src_L0_ofs,
882+ IS_DGFX(xe) ? src_is_vram : src_is_pltt,
883+ dst_L0_ofs,
884+ IS_DGFX(xe) ? dst_is_vram : dst_is_pltt,
885+ src_L0, ccs_ofs, copy_ccs);
835886
836887 job = xe_bb_create_migration_job(m->q, bb,
837888 xe_migrate_batch_base(m, usm),
@@ -1022,6 +1073,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m, · 184 unmodified lines
10221073 struct xe_sched_job *job;
10231074 struct xe_bb *bb;
10241075 u32 batch_size, update_idx;
1076+ u32 pte_flags;
10251077
10261078 bool usm = xe->info.has_usm;
10271079 u32 avail_pts = max_mem_transfer_per_pass(xe) / LEVEL0_PAGE_TABLE_ENCODE_SIZE;
@@ -1029,13 +1081,14 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m, · 1 unmodified line
10291081 clear_L0 = xe_migrate_res_sizes(m, &src_it);
10301082
10311083 /* Calculate final sizes and batch size.. */
1084+ pte_flags = clear_vram ? PTE_UPDATE_FLAG_IS_VRAM : 0;
10321085 batch_size = 2 +
1033− pte_update_size(m, clear_vram, src, &src_it,
1086+ pte_update_size(m, pte_flags, src, &src_it,
10341087 &clear_L0, &clear_L0_ofs, &clear_L0_pt,
10351088 clear_system_ccs ? 0 : emit_clear_cmd_len(gt), 0,
10361089 avail_pts);
10371090
1038− if (xe_device_has_flat_ccs(xe))
1091+ if (xe_migrate_needs_ccs_emit(xe))
10391092 batch_size += EMIT_COPY_CCS_DW;
10401093
10411094 /* Clear commands */
@@ -1063,7 +1116,7 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m, · 21 unmodified lines
10631116 if (!clear_system_ccs)
10641117 emit_clear(gt, bb, clear_L0_ofs, clear_L0, XE_PAGE_SIZE, clear_vram);
10651118
1066− if (xe_device_has_flat_ccs(xe)) {
1119+ if (xe_migrate_needs_ccs_emit(xe)) {
10671120 emit_copy_ccs(gt, bb, clear_L0_ofs, true,
10681121 m->cleared_mem_ofs, false, clear_L0);
10691122 flush_flags = MI_FLUSH_DW_CCS;
@@ -1126,6 +1179,7 @@ err_sync: · 56 unmodified lines
11261179 }
11271180
11281181 static void write_pgtable(struct xe_tile *tile, struct xe_bb *bb, u64 ppgtt_ofs,
1182+ const struct xe_vm_pgtable_update_op *pt_op,
11291183 const struct xe_vm_pgtable_update *update,
11301184 struct xe_migrate_pt_update *pt_update)
11311185 {
@@ -1146,7 +1200,7 @@ static void write_pgtable(struct xe_tile *tile, struct xe_bb *bb, u64 ppgtt_ofs, · 14 unmodified lines
11461200 if (!ppgtt_ofs)
11471201 ppgtt_ofs = xe_migrate_vram_ofs(tile_to_xe(tile),
11481202 xe_bo_addr(update->pt_bo, 0,
1149− XE_PAGE_SIZE));
1203+ XE_PAGE_SIZE), false);
11501204
11511205 do {
11521206 u64 addr = ppgtt_ofs + ofs * 8;
@@ -1160,8 +1214,12 @@ static void write_pgtable(struct xe_tile *tile, struct xe_bb *bb, u64 ppgtt_ofs, · 7 unmodified lines
11601214 bb->cs[bb->len++] = MI_STORE_DATA_IMM | MI_SDI_NUM_QW(chunk);
11611215 bb->cs[bb->len++] = lower_32_bits(addr);
11621216 bb->cs[bb->len++] = upper_32_bits(addr);
1163− ops->populate(pt_update, tile, NULL, bb->cs + bb->len, ofs, chunk,
1164− update);
1217+ if (pt_op->bind)
1218+ ops->populate(pt_update, tile, NULL, bb->cs + bb->len,
1219+ ofs, chunk, update);
1220+ else
1221+ ops->clear(pt_update, tile, NULL, bb->cs + bb->len,
1222+ ofs, chunk, update);
11651223
11661224 bb->len += chunk * 2;
11671225 ofs += chunk;
@@ -1186,114 +1244,58 @@ struct migrate_test_params { · 18 unmodified lines
11861244
11871245 static struct dma_fence *
11881246 xe_migrate_update_pgtables_cpu(struct xe_migrate *m,
1189− struct xe_vm *vm, struct xe_bo *bo,
1190− const struct xe_vm_pgtable_update *updates,
1191− u32 num_updates, bool wait_vm,
11921247 struct xe_migrate_pt_update *pt_update)
11931248 {
11941249 XE_TEST_DECLARE(struct migrate_test_params *test =
11951250 to_migrate_test_params
11961251 (xe_cur_kunit_priv(XE_TEST_LIVE_MIGRATE));)
11971252 const struct xe_migrate_pt_update_ops *ops = pt_update->ops;
1198− struct dma_fence *fence;
1253+ struct xe_vm *vm = pt_update->vops->vm;
1254+ struct xe_vm_pgtable_update_ops *pt_update_ops =
1255+ &pt_update->vops->pt_update_ops[pt_update->tile_id];
11991256 int err;
1200− u32 i;
1257+ u32 i, j;
12011258
12021259 if (XE_TEST_ONLY(test && test->force_gpu))
12031260 return ERR_PTR(-ETIME);
12041261
1205− if (bo && !dma_resv_test_signaled(bo->ttm.base.resv,
1206− DMA_RESV_USAGE_KERNEL))
1207− return ERR_PTR(-ETIME);
1208−
1209− if (wait_vm && !dma_resv_test_signaled(xe_vm_resv(vm),
1210− DMA_RESV_USAGE_BOOKKEEP))
1211− return ERR_PTR(-ETIME);
1212−
12131262 if (ops->pre_commit) {
12141263 pt_update->job = NULL;
12151264 err = ops->pre_commit(pt_update);
12161265 if (err)
12171266 return ERR_PTR(err);
12181267 }
1219− for (i = 0; i < num_updates; i++) {
1220− const struct xe_vm_pgtable_update *update = &updates[i];
1221−
1222− ops->populate(pt_update, m->tile, &update->pt_bo->vmap, NULL,
1223− update->ofs, update->qwords, update);
1224− }
1225−
1226− if (vm) {
1227− trace_xe_vm_cpu_bind(vm);
1228− xe_device_wmb(vm->xe);
1229− }
1230−
1231− fence = dma_fence_get_stub();
1232−
1233− return fence;
1234−}
1235−
1236−static bool no_in_syncs(struct xe_vm *vm, struct xe_exec_queue *q,
1237− struct xe_sync_entry *syncs, u32 num_syncs)
1238−{
1239− struct dma_fence *fence;
1240− int i;
12411268
1242− for (i = 0; i < num_syncs; i++) {
1243− fence = syncs[i].fence;
1244−
1245− if (fence && !test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
1246− &fence->flags))
1247− return false;
1248− }
1249− if (q) {
1250− fence = xe_exec_queue_last_fence_get(q, vm);
1251− if (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
1252− dma_fence_put(fence);
1253− return false;
1269+ for (i = 0; i < pt_update_ops->num_ops; ++i) {
1270+ const struct xe_vm_pgtable_update_op *pt_op =
1271+ &pt_update_ops->ops[i];
1272+
1273+ for (j = 0; j < pt_op->num_entries; j++) {
1274+ const struct xe_vm_pgtable_update *update =
1275+ &pt_op->entries[j];
1276+
1277+ if (pt_op->bind)
1278+ ops->populate(pt_update, m->tile,
1279+ &update->pt_bo->vmap, NULL,
1280+ update->ofs, update->qwords,
1281+ update);
1282+ else
1283+ ops->clear(pt_update, m->tile,
1284+ &update->pt_bo->vmap, NULL,
1285+ update->ofs, update->qwords, update);
12541286 }
1255− dma_fence_put(fence);
12561287 }
12571288
1258− return true;
1289+ trace_xe_vm_cpu_bind(vm);
1290+ xe_device_wmb(vm->xe);
1291+
1292+ return dma_fence_get_stub();
12591293 }
12601294
1261−/**
1262− * xe_migrate_update_pgtables() - Pipelined page-table update
1263− * @m: The migrate context.
1264− * @vm: The vm we'll be updating.
1265− * @bo: The bo whose dma-resv we will await before updating, or NULL if userptr.
1266− * @q: The exec queue to be used for the update or NULL if the default
1267− * migration engine is to be used.
1268− * @updates: An array of update descriptors.
1269− * @num_updates: Number of descriptors in @updates.
1270− * @syncs: Array of xe_sync_entry to await before updating. Note that waits
1271− * will block the engine timeline.
1272− * @num_syncs: Number of entries in @syncs.
1273− * @pt_update: Pointer to a struct xe_migrate_pt_update, which contains
1274− * pointers to callback functions and, if subclassed, private arguments to
1275− * those.
1276− *
1277− * Perform a pipelined page-table update. The update descriptors are typically
1278− * built under the same lock critical section as a call to this function. If
1279− * using the default engine for the updates, they will be performed in the
1280− * order they grab the job_mutex. If different engines are used, external
1281− * synchronization is needed for overlapping updates to maintain page-table
1282− * consistency. Note that the meaing of "overlapping" is that the updates
1283− * touch the same page-table, which might be a higher-level page-directory.
1284− * If no pipelining is needed, then updates may be performed by the cpu.
1285− *
1286− * Return: A dma_fence that, when signaled, indicates the update completion.
1287− */
1288−struct dma_fence *
1289−xe_migrate_update_pgtables(struct xe_migrate *m,
1290− struct xe_vm *vm,
1291− struct xe_bo *bo,
1292− struct xe_exec_queue *q,
1293− const struct xe_vm_pgtable_update *updates,
1294− u32 num_updates,
1295− struct xe_sync_entry *syncs, u32 num_syncs,
1296− struct xe_migrate_pt_update *pt_update)
1295+static struct dma_fence *
1296+__xe_migrate_update_pgtables(struct xe_migrate *m,
1297+ struct xe_migrate_pt_update *pt_update,
1298+ struct xe_vm_pgtable_update_ops *pt_update_ops)
12971299 {
12981300 const struct xe_migrate_pt_update_ops *ops = pt_update->ops;
12991301 struct xe_tile *tile = m->tile;
@@ -1302,59 +1304,53 @@ xe_migrate_update_pgtables(struct xe_migrate *m, · 2 unmodified lines
13021304 struct xe_sched_job *job;
13031305 struct dma_fence *fence;
13041306 struct drm_suballoc *sa_bo = NULL;
1305− struct xe_vma *vma = pt_update->vma;
13061307 struct xe_bb *bb;
1307− u32 i, batch_size, ppgtt_ofs, update_idx, page_ofs = 0;
1308+ u32 i, j, batch_size = 0, ppgtt_ofs, update_idx, page_ofs = 0;
1309+ u32 num_updates = 0, current_update = 0;
13081310 u64 addr;
13091311 int err = 0;
1310− bool usm = !q && xe->info.has_usm;
1311− bool first_munmap_rebind = vma &&
1312− vma->gpuva.flags & XE_VMA_FIRST_REBIND;
1313− struct xe_exec_queue *q_override = !q ? m->q : q;
1314− u16 pat_index = xe->pat.idx[XE_CACHE_WB];
1312+ bool is_migrate = pt_update_ops->q == m->q;
1313+ bool usm = is_migrate && xe->info.has_usm;
1314+
1315+ for (i = 0; i < pt_update_ops->num_ops; ++i) {
1316+ struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i];
1317+ struct xe_vm_pgtable_update *updates = pt_op->entries;
13151318
1316− /* Use the CPU if no in syncs and engine is idle */
1317− if (no_in_syncs(vm, q, syncs, num_syncs) && xe_exec_queue_is_idle(q_override)) {
1318− fence = xe_migrate_update_pgtables_cpu(m, vm, bo, updates,
1319− num_updates,
1320− first_munmap_rebind,
1321− pt_update);
1322− if (!IS_ERR(fence) || fence == ERR_PTR(-EAGAIN))
1323− return fence;
1319+ num_updates += pt_op->num_entries;
1320+ for (j = 0; j < pt_op->num_entries; ++j) {
1321+ u32 num_cmds = DIV_ROUND_UP(updates[j].qwords,
1322+ MAX_PTE_PER_SDI);
1323+
1324+ /* align noop + MI_STORE_DATA_IMM cmd prefix */
1325+ batch_size += 4 * num_cmds + updates[j].qwords * 2;
1326+ }
13241327 }
13251328
13261329 /* fixed + PTE entries */
13271330 if (IS_DGFX(xe))
1328− batch_size = 2;
1331+ batch_size += 2;
13291332 else
1330− batch_size = 6 + num_updates * 2;
1331−
1332− for (i = 0; i < num_updates; i++) {
1333− u32 num_cmds = DIV_ROUND_UP(updates[i].qwords, MAX_PTE_PER_SDI);
1334−
1335− /* align noop + MI_STORE_DATA_IMM cmd prefix */
1336− batch_size += 4 * num_cmds + updates[i].qwords * 2;
1337− }
1338−
1339− /*
1340− * XXX: Create temp bo to copy from, if batch_size becomes too big?
1341− *
1342− * Worst case: Sum(2 * (each lower level page size) + (top level page size))
1343− * Should be reasonably bound..
1344− */
1345− xe_tile_assert(tile, batch_size < SZ_128K);
1333+ batch_size += 6 * (num_updates / MAX_PTE_PER_SDI + 1) +
1334+ num_updates * 2;
13461335
1347− bb = xe_bb_new(gt, batch_size, !q && xe->info.has_usm);
1336+ bb = xe_bb_new(gt, batch_size, usm);
13481337 if (IS_ERR(bb))
13491338 return ERR_CAST(bb);
13501339
13511340 /* For sysmem PTE's, need to map them in our hole.. */
13521341 if (!IS_DGFX(xe)) {
1342+ u32 ptes, ofs;
1343+
13531344 ppgtt_ofs = NUM_KERNEL_PDE - 1;
1354− if (q) {
1355− xe_tile_assert(tile, num_updates <= NUM_VMUSA_WRITES_PER_UNIT);
1345+ if (!is_migrate) {
1346+ u32 num_units = DIV_ROUND_UP(num_updates,
1347+ NUM_VMUSA_WRITES_PER_UNIT);
13561348
1357− sa_bo = drm_suballoc_new(&m->vm_update_sa, 1,
1349+ if (num_units > m->vm_update_sa.size) {
1350+ err = -ENOBUFS;
1351+ goto err_bb;
1352+ }
1353+ sa_bo = drm_suballoc_new(&m->vm_update_sa, num_units,
13581354 GFP_KERNEL, true, 0);
13591355 if (IS_ERR(sa_bo)) {
13601356 err = PTR_ERR(sa_bo);
@@ -1370,18 +1366,49 @@ xe_migrate_update_pgtables(struct xe_migrate *m, · 9 unmodified lines
13701366 }
13711367
13721368 /* Map our PT's to gtt */
1373− bb->cs[bb->len++] = MI_STORE_DATA_IMM | MI_SDI_NUM_QW(num_updates);
1374− bb->cs[bb->len++] = ppgtt_ofs * XE_PAGE_SIZE + page_ofs;
1375− bb->cs[bb->len++] = 0; /* upper_32_bits */
1376−
1377− for (i = 0; i < num_updates; i++) {
1378− struct xe_bo *pt_bo = updates[i].pt_bo;
1369+ i = 0;
1370+ j = 0;
1371+ ptes = num_updates;
1372+ ofs = ppgtt_ofs * XE_PAGE_SIZE + page_ofs;
1373+ while (ptes) {
1374+ u32 chunk = min(MAX_PTE_PER_SDI, ptes);
1375+ u32 idx = 0;
1376+
1377+ bb->cs[bb->len++] = MI_STORE_DATA_IMM |
1378+ MI_SDI_NUM_QW(chunk);
1379+ bb->cs[bb->len++] = ofs;
1380+ bb->cs[bb->len++] = 0; /* upper_32_bits */
1381+
1382+ for (; i < pt_update_ops->num_ops; ++i) {
1383+ struct xe_vm_pgtable_update_op *pt_op =
1384+ &pt_update_ops->ops[i];
1385+ struct xe_vm_pgtable_update *updates = pt_op->entries;
1386+
1387+ for (; j < pt_op->num_entries; ++j, ++current_update, ++idx) {
1388+ struct xe_vm *vm = pt_update->vops->vm;
1389+ struct xe_bo *pt_bo = updates[j].pt_bo;
1390+
1391+ if (idx == chunk)
1392+ goto next_cmd;
1393+
1394+ xe_tile_assert(tile, pt_bo->size == SZ_4K);
1395+
1396+ /* Map a PT at most once */
1397+ if (pt_bo->update_index < 0)
1398+ pt_bo->update_index = current_update;
1399+
1400+ addr = vm->pt_ops->pte_encode_bo(pt_bo, 0,
1401+ XE_CACHE_WB, 0);
1402+ bb->cs[bb->len++] = lower_32_bits(addr);
1403+ bb->cs[bb->len++] = upper_32_bits(addr);
1404+ }
13791405
1380− xe_tile_assert(tile, pt_bo->size == SZ_4K);
1406+ j = 0;
1407+ }
13811408
1382− addr = vm->pt_ops->pte_encode_bo(pt_bo, 0, pat_index, 0);
1383− bb->cs[bb->len++] = lower_32_bits(addr);
1384− bb->cs[bb->len++] = upper_32_bits(addr);
1409+next_cmd:
1410+ ptes -= chunk;
1411+ ofs += chunk * sizeof(u64);
13851412 }
13861413
13871414 bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
@@ -1389,19 +1416,36 @@ xe_migrate_update_pgtables(struct xe_migrate *m, · 1 unmodified line
13891416
13901417 addr = xe_migrate_vm_addr(ppgtt_ofs, 0) +
13911418 (page_ofs / sizeof(u64)) * XE_PAGE_SIZE;
1392− for (i = 0; i < num_updates; i++)
1393− write_pgtable(tile, bb, addr + i * XE_PAGE_SIZE,
1394− &updates[i], pt_update);
1419+ for (i = 0; i < pt_update_ops->num_ops; ++i) {
1420+ struct xe_vm_pgtable_update_op *pt_op =
1421+ &pt_update_ops->ops[i];
1422+ struct xe_vm_pgtable_update *updates = pt_op->entries;
1423+
1424+ for (j = 0; j < pt_op->num_entries; ++j) {
1425+ struct xe_bo *pt_bo = updates[j].pt_bo;
1426+
1427+ write_pgtable(tile, bb, addr +
1428+ pt_bo->update_index * XE_PAGE_SIZE,
1429+ pt_op, &updates[j], pt_update);
1430+ }
1431+ }
13951432 } else {
13961433 /* phys pages, no preamble required */
13971434 bb->cs[bb->len++] = MI_BATCH_BUFFER_END;
13981435 update_idx = bb->len;
13991436
1400− for (i = 0; i < num_updates; i++)
1401− write_pgtable(tile, bb, 0, &updates[i], pt_update);
1437+ for (i = 0; i < pt_update_ops->num_ops; ++i) {
1438+ struct xe_vm_pgtable_update_op *pt_op =
1439+ &pt_update_ops->ops[i];
1440+ struct xe_vm_pgtable_update *updates = pt_op->entries;
1441+
1442+ for (j = 0; j < pt_op->num_entries; ++j)
1443+ write_pgtable(tile, bb, 0, pt_op, &updates[j],
1444+ pt_update);
1445+ }
14021446 }
14031447
1404− job = xe_bb_create_migration_job(q ?: m->q, bb,
1448+ job = xe_bb_create_migration_job(pt_update_ops->q, bb,
14051449 xe_migrate_batch_base(m, usm),
14061450 update_idx);
14071451 if (IS_ERR(job)) {
@@ -1409,46 +1453,20 @@ xe_migrate_update_pgtables(struct xe_migrate *m, · 1 unmodified line
14091453 goto err_sa;
14101454 }
14111455
1412− /* Wait on BO move */
1413− if (bo) {
1414− err = xe_sched_job_add_deps(job, bo->ttm.base.resv,
1415− DMA_RESV_USAGE_KERNEL);
1416− if (err)
1417− goto err_job;
1418− }
1419−
1420− /*
1421− * Munmap style VM unbind, need to wait for all jobs to be complete /
1422− * trigger preempts before moving forward
1423− */
1424− if (first_munmap_rebind) {
1425− err = xe_sched_job_add_deps(job, xe_vm_resv(vm),
1426− DMA_RESV_USAGE_BOOKKEEP);
1427− if (err)
1428− goto err_job;
1429− }
1430−
1431− err = xe_sched_job_last_fence_add_dep(job, vm);
1432− for (i = 0; !err && i < num_syncs; i++)
1433− err = xe_sync_entry_add_deps(&syncs[i], job);
1434−
1435− if (err)
1436− goto err_job;
1437−
14381456 if (ops->pre_commit) {
14391457 pt_update->job = job;
14401458 err = ops->pre_commit(pt_update);
14411459 if (err)
14421460 goto err_job;
14431461 }
1444− if (!q)
1462+ if (is_migrate)
14451463 mutex_lock(&m->job_mutex);
14461464
14471465 xe_sched_job_arm(job);
14481466 fence = dma_fence_get(&job->drm.s_fence->finished);
14491467 xe_sched_job_push(job);
14501468
1451− if (!q)
1469+ if (is_migrate)
14521470 mutex_unlock(&m->job_mutex);
14531471
14541472 xe_bb_free(bb, fence);
@@ -1465,6 +1483,40 @@ err_bb: · 10 unmodified lines
14651483 return ERR_PTR(err);
14661484 }
14671485
1486+/**
1487+ * xe_migrate_update_pgtables() - Pipelined page-table update
1488+ * @m: The migrate context.
1489+ * @pt_update: PT update arguments
1490+ *
1491+ * Perform a pipelined page-table update. The update descriptors are typically
1492+ * built under the same lock critical section as a call to this function. If
1493+ * using the default engine for the updates, they will be performed in the
1494+ * order they grab the job_mutex. If different engines are used, external
1495+ * synchronization is needed for overlapping updates to maintain page-table
1496+ * consistency. Note that the meaing of "overlapping" is that the updates
1497+ * touch the same page-table, which might be a higher-level page-directory.
1498+ * If no pipelining is needed, then updates may be performed by the cpu.
1499+ *
1500+ * Return: A dma_fence that, when signaled, indicates the update completion.
1501+ */
1502+struct dma_fence *
1503+xe_migrate_update_pgtables(struct xe_migrate *m,
1504+ struct xe_migrate_pt_update *pt_update)
1505+
1506+{
1507+ struct xe_vm_pgtable_update_ops *pt_update_ops =
1508+ &pt_update->vops->pt_update_ops[pt_update->tile_id];
1509+ struct dma_fence *fence;
1510+
1511+ fence = xe_migrate_update_pgtables_cpu(m, pt_update);
1512+
1513+ /* -ETIME indicates a job is needed, anything else is legit error */
1514+ if (!IS_ERR(fence) || PTR_ERR(fence) != -ETIME)
1515+ return fence;
1516+
1517+ return __xe_migrate_update_pgtables(m, pt_update, pt_update_ops);
1518+}
1519+
14681520 /**
14691521 * xe_migrate_wait() - Complete all operations using the xe_migrate context
14701522 * @m: Migrate context to wait for.
drivers/gpu/drm/xe/xe_migrate.h+21−13
@@ -47,6 +47,24 @@ struct xe_migrate_pt_update_ops { · 46 unmodified lines
4747 struct xe_tile *tile, struct iosys_map *map,
4848 void *pos, u32 ofs, u32 num_qwords,
4949 const struct xe_vm_pgtable_update *update);
50+ /**
51+ * @clear: Clear a command buffer or page-table with ptes.
52+ * @pt_update: Embeddable callback argument.
53+ * @tile: The tile for the current operation.
54+ * @map: struct iosys_map into the memory to be populated.
55+ * @pos: If @map is NULL, map into the memory to be populated.
56+ * @ofs: qword offset into @map, unused if @map is NULL.
57+ * @num_qwords: Number of qwords to write.
58+ * @update: Information about the PTEs to be inserted.
59+ *
60+ * This interface is intended to be used as a callback into the
61+ * page-table system to populate command buffers or shared
62+ * page-tables with PTEs.
63+ */
64+ void (*clear)(struct xe_migrate_pt_update *pt_update,
65+ struct xe_tile *tile, struct iosys_map *map,
66+ void *pos, u32 ofs, u32 num_qwords,
67+ const struct xe_vm_pgtable_update *update);
5068
5169 /**
5270 * @pre_commit: Callback to be called just before arming the
@@ -67,14 +85,10 @@ struct xe_migrate_pt_update_ops { · 14 unmodified lines
6785 struct xe_migrate_pt_update {
6886 /** @ops: Pointer to the struct xe_migrate_pt_update_ops callbacks */
6987 const struct xe_migrate_pt_update_ops *ops;
70− /** @vma: The vma we're updating the pagetable for. */
71− struct xe_vma *vma;
88+ /** @vops: VMA operations */
89+ struct xe_vma_ops *vops;
7290 /** @job: The job if a GPU page-table update. NULL otherwise */
7391 struct xe_sched_job *job;
74− /** @start: Start of update for the range fence */
75− u64 start;
76− /** @last: Last of update for the range fence */
77− u64 last;
7892 /** @tile_id: Tile ID of the update */
7993 u8 tile_id;
8094 };
@@ -96,15 +110,9 @@ struct xe_vm *xe_migrate_get_vm(struct xe_migrate *m); · 15 unmodified lines
96110
97111 struct dma_fence *
98112 xe_migrate_update_pgtables(struct xe_migrate *m,
99− struct xe_vm *vm,
100− struct xe_bo *bo,
101− struct xe_exec_queue *q,
102− const struct xe_vm_pgtable_update *updates,
103− u32 num_updates,
104− struct xe_sync_entry *syncs, u32 num_syncs,
105113 struct xe_migrate_pt_update *pt_update);
106114
107115 void xe_migrate_wait(struct xe_migrate *m);
108116
109−struct xe_exec_queue *xe_tile_migrate_engine(struct xe_tile *tile);
117+struct xe_exec_queue *xe_tile_migrate_exec_queue(struct xe_tile *tile);
110118 #endif
drivers/gpu/drm/xe/xe_mmio.c+144−87
@@ -33,29 +33,56 @@ static void tiles_fini(void *arg) · 32 unmodified lines
3333 tile->mmio.regs = NULL;
3434 }
3535
36−int xe_mmio_probe_tiles(struct xe_device *xe)
36+/*
37+ * On multi-tile devices, partition the BAR space for MMIO on each tile,
38+ * possibly accounting for register override on the number of tiles available.
39+ * Resulting memory layout is like below:
40+ *
41+ * .----------------------. <- tile_count * tile_mmio_size
42+ * | .... |
43+ * |----------------------| <- 2 * tile_mmio_size
44+ * | tile1->mmio.regs |
45+ * |----------------------| <- 1 * tile_mmio_size
46+ * | tile0->mmio.regs |
47+ * '----------------------' <- 0MB
48+ */
49+static void mmio_multi_tile_setup(struct xe_device *xe, size_t tile_mmio_size)
3750 {
38− size_t tile_mmio_size = SZ_16M, tile_mmio_ext_size = xe->info.tile_mmio_ext_size;
39− u8 id, tile_count = xe->info.tile_count;
40− struct xe_gt *gt = xe_root_mmio_gt(xe);
4151 struct xe_tile *tile;
4252 void __iomem *regs;
43− u32 mtcfg;
53+ u8 id;
4454
45− if (tile_count == 1)
46− goto add_mmio_ext;
55+ /*
56+ * Nothing to be done as tile 0 has already been setup earlier with the
57+ * entire BAR mapped - see xe_mmio_init()
58+ */
59+ if (xe->info.tile_count == 1)
60+ return;
4761
62+ /* Possibly override number of tile based on configuration register */
4863 if (!xe->info.skip_mtcfg) {
64+ struct xe_gt *gt = xe_root_mmio_gt(xe);
65+ u8 tile_count;
66+ u32 mtcfg;
67+
68+ /*
69+ * Although the per-tile mmio regs are not yet initialized, this
70+ * is fine as it's going to the root gt, that's guaranteed to be
71+ * initialized earlier in xe_mmio_init()
72+ */
4973 mtcfg = xe_mmio_read64_2x32(gt, XEHP_MTCFG_ADDR);
5074 tile_count = REG_FIELD_GET(TILE_COUNT, mtcfg) + 1;
75+
5176 if (tile_count < xe->info.tile_count) {
5277 drm_info(&xe->drm, "tile_count: %d, reduced_tile_count %d\n",
5378 xe->info.tile_count, tile_count);
5479 xe->info.tile_count = tile_count;
5580
5681 /*
57− * FIXME: Needs some work for standalone media, but should be impossible
58− * with multi-tile for now.
82+ * FIXME: Needs some work for standalone media, but
83+ * should be impossible with multi-tile for now:
84+ * multi-tile platform with standalone media doesn't
85+ * exist
5986 */
6087 xe->info.gt_count = xe->info.tile_count;
6188 }
@@ -67,23 +94,51 @@ int xe_mmio_probe_tiles(struct xe_device *xe) · 5 unmodified lines
6794 tile->mmio.regs = regs;
6895 regs += tile_mmio_size;
6996 }
97+}
7098
71−add_mmio_ext:
72− /*
73− * By design, there's a contiguous multi-tile MMIO space (16MB hard coded per tile).
74− * When supported, there could be an additional contiguous multi-tile MMIO extension
75− * space ON TOP of it, and hence the necessity for distinguished MMIO spaces.
76− */
77− if (xe->info.has_mmio_ext) {
78− regs = xe->mmio.regs + tile_mmio_size * tile_count;
99+/*
100+ * On top of all the multi-tile MMIO space there can be a platform-dependent
101+ * extension for each tile, resulting in a layout like below:
102+ *
103+ * .----------------------. <- ext_base + tile_count * tile_mmio_ext_size
104+ * | .... |
105+ * |----------------------| <- ext_base + 2 * tile_mmio_ext_size
106+ * | tile1->mmio_ext.regs |
107+ * |----------------------| <- ext_base + 1 * tile_mmio_ext_size
108+ * | tile0->mmio_ext.regs |
109+ * |======================| <- ext_base = tile_count * tile_mmio_size
110+ * | |
111+ * | mmio.regs |
112+ * | |
113+ * '----------------------' <- 0MB
114+ *
115+ * Set up the tile[]->mmio_ext pointers/sizes.
116+ */
117+static void mmio_extension_setup(struct xe_device *xe, size_t tile_mmio_size,
118+ size_t tile_mmio_ext_size)
119+{
120+ struct xe_tile *tile;
121+ void __iomem *regs;
122+ u8 id;
79123
80− for_each_tile(tile, xe, id) {
81− tile->mmio_ext.size = tile_mmio_ext_size;
82− tile->mmio_ext.regs = regs;
124+ if (!xe->info.has_mmio_ext)
125+ return;
83126
84− regs += tile_mmio_ext_size;
85− }
127+ regs = xe->mmio.regs + tile_mmio_size * xe->info.tile_count;
128+ for_each_tile(tile, xe, id) {
129+ tile->mmio_ext.size = tile_mmio_ext_size;
130+ tile->mmio_ext.regs = regs;
131+ regs += tile_mmio_ext_size;
86132 }
133+}
134+
135+int xe_mmio_probe_tiles(struct xe_device *xe)
136+{
137+ size_t tile_mmio_size = SZ_16M;
138+ size_t tile_mmio_ext_size = xe->info.tile_mmio_ext_size;
139+
140+ mmio_multi_tile_setup(xe, tile_mmio_size);
141+ mmio_extension_setup(xe, tile_mmio_size, tile_mmio_ext_size);
87142
88143 return devm_add_action_or_reset(xe->drm.dev, tiles_fini, xe);
89144 }
@@ -121,12 +176,29 @@ int xe_mmio_init(struct xe_device *xe) · 31 unmodified lines
121176 return devm_add_action_or_reset(xe->drm.dev, mmio_fini, xe);
122177 }
123178
179+static void mmio_flush_pending_writes(struct xe_gt *gt)
180+{
181+#define DUMMY_REG_OFFSET 0x130030
182+ struct xe_tile *tile = gt_to_tile(gt);
183+ int i;
184+
185+ if (tile->xe->info.platform != XE_LUNARLAKE)
186+ return;
187+
188+ /* 4 dummy writes */
189+ for (i = 0; i < 4; i++)
190+ writel(0, tile->mmio.regs + DUMMY_REG_OFFSET);
191+}
192+
124193 u8 xe_mmio_read8(struct xe_gt *gt, struct xe_reg reg)
125194 {
126195 struct xe_tile *tile = gt_to_tile(gt);
127196 u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
128197 u8 val;
129198
199+ /* Wa_15015404425 */
200+ mmio_flush_pending_writes(gt);
201+
130202 val = readb((reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
131203 trace_xe_reg_rw(gt, false, addr, val, sizeof(val));
132204
@@ -139,6 +211,9 @@ u16 xe_mmio_read16(struct xe_gt *gt, struct xe_reg reg) · 6 unmodified lines
139211 u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
140212 u16 val;
141213
214+ /* Wa_15015404425 */
215+ mmio_flush_pending_writes(gt);
216+
142217 val = readw((reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
143218 trace_xe_reg_rw(gt, false, addr, val, sizeof(val));
144219
@@ -151,7 +226,11 @@ void xe_mmio_write32(struct xe_gt *gt, struct xe_reg reg, u32 val) · 6 unmodified lines
151226 u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
152227
153228 trace_xe_reg_rw(gt, true, addr, val, sizeof(val));
154− writel(val, (reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
229+
230+ if (!reg.vf && IS_SRIOV_VF(gt_to_xe(gt)))
231+ xe_gt_sriov_vf_write32(gt, reg, val);
232+ else
233+ writel(val, (reg.ext ? tile->mmio_ext.regs : tile->mmio.regs) + addr);
155234 }
156235
157236 u32 xe_mmio_read32(struct xe_gt *gt, struct xe_reg reg)
@@ -160,6 +239,9 @@ u32 xe_mmio_read32(struct xe_gt *gt, struct xe_reg reg) · 2 unmodified lines
160239 u32 addr = xe_mmio_adjusted_addr(gt, reg.addr);
161240 u32 val;
162241
242+ /* Wa_15015404425 */
243+ mmio_flush_pending_writes(gt);
244+
163245 if (!reg.vf && IS_SRIOV_VF(gt_to_xe(gt)))
164246 val = xe_gt_sriov_vf_read32(gt, reg);
165247 else
@@ -251,37 +333,24 @@ u64 xe_mmio_read64_2x32(struct xe_gt *gt, struct xe_reg reg) · 85 unmodified lines
251333 return (u64)udw << 32 | ldw;
252334 }
253335
254−/**
255− * xe_mmio_wait32() - Wait for a register to match the desired masked value
256− * @gt: MMIO target GT
257− * @reg: register to read value from
258− * @mask: mask to be applied to the value read from the register
259− * @val: desired value after applying the mask
260− * @timeout_us: time out after this period of time. Wait logic tries to be
261− * smart, applying an exponential backoff until @timeout_us is reached.
262− * @out_val: if not NULL, points where to store the last unmasked value
263− * @atomic: needs to be true if calling from an atomic context
264− *
265− * This function polls for the desired masked value and returns zero on success
266− * or -ETIMEDOUT if timed out.
267− *
268− * Note that @timeout_us represents the minimum amount of time to wait before
269− * giving up. The actual time taken by this function can be a little more than
270− * @timeout_us for different reasons, specially in non-atomic contexts. Thus,
271− * it is possible that this function succeeds even after @timeout_us has passed.
272− */
273−int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
274− u32 *out_val, bool atomic)
336+static int __xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
337+ u32 *out_val, bool atomic, bool expect_match)
275338 {
276339 ktime_t cur = ktime_get_raw();
277340 const ktime_t end = ktime_add_us(cur, timeout_us);
278341 int ret = -ETIMEDOUT;
279342 s64 wait = 10;
280343 u32 read;
344+ bool check;
281345
282346 for (;;) {
283347 read = xe_mmio_read32(gt, reg);
284− if ((read & mask) == val) {
348+
349+ check = (read & mask) == val;
350+ if (!expect_match)
351+ check = !check;
352+
353+ if (check) {
285354 ret = 0;
286355 break;
287356 }
@@ -302,7 +371,12 @@ int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 t · 14 unmodified lines
302371
303372 if (ret != 0) {
304373 read = xe_mmio_read32(gt, reg);
305− if ((read & mask) == val)
374+
375+ check = (read & mask) == val;
376+ if (!expect_match)
377+ check = !check;
378+
379+ if (check)
306380 ret = 0;
307381 }
308382
@@ -313,62 +387,45 @@ int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 t · 4 unmodified lines
313387 }
314388
315389 /**
316− * xe_mmio_wait32_not() - Wait for a register to return anything other than the given masked value
390+ * xe_mmio_wait32() - Wait for a register to match the desired masked value
317391 * @gt: MMIO target GT
318392 * @reg: register to read value from
319393 * @mask: mask to be applied to the value read from the register
320− * @val: value to match after applying the mask
394+ * @val: desired value after applying the mask
321395 * @timeout_us: time out after this period of time. Wait logic tries to be
322396 * smart, applying an exponential backoff until @timeout_us is reached.
323397 * @out_val: if not NULL, points where to store the last unmasked value
324398 * @atomic: needs to be true if calling from an atomic context
325399 *
326− * This function polls for a masked value to change from a given value and
327− * returns zero on success or -ETIMEDOUT if timed out.
400+ * This function polls for the desired masked value and returns zero on success
401+ * or -ETIMEDOUT if timed out.
328402 *
329403 * Note that @timeout_us represents the minimum amount of time to wait before
330404 * giving up. The actual time taken by this function can be a little more than
331405 * @timeout_us for different reasons, specially in non-atomic contexts. Thus,
332406 * it is possible that this function succeeds even after @timeout_us has passed.
333407 */
408+int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
409+ u32 *out_val, bool atomic)
410+{
411+ return __xe_mmio_wait32(gt, reg, mask, val, timeout_us, out_val, atomic, true);
412+}
413+
414+/**
415+ * xe_mmio_wait32_not() - Wait for a register to return anything other than the given masked value
416+ * @gt: MMIO target GT
417+ * @reg: register to read value from
418+ * @mask: mask to be applied to the value read from the register
419+ * @val: value not to be matched after applying the mask
420+ * @timeout_us: time out after this period of time
421+ * @out_val: if not NULL, points where to store the last unmasked value
422+ * @atomic: needs to be true if calling from an atomic context
423+ *
424+ * This function works exactly like xe_mmio_wait32() with the exception that
425+ * @val is expected not to be matched.
426+ */
334427 int xe_mmio_wait32_not(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
335428 u32 *out_val, bool atomic)
336429 {
337− ktime_t cur = ktime_get_raw();
338− const ktime_t end = ktime_add_us(cur, timeout_us);
339− int ret = -ETIMEDOUT;
340− s64 wait = 10;
341− u32 read;
342−
343− for (;;) {
344− read = xe_mmio_read32(gt, reg);
345− if ((read & mask) != val) {
346− ret = 0;
347− break;
348− }
349−
350− cur = ktime_get_raw();
351− if (!ktime_before(cur, end))
352− break;
353−
354− if (ktime_after(ktime_add_us(cur, wait), end))
355− wait = ktime_us_delta(end, cur);
356−
357− if (atomic)
358− udelay(wait);
359− else
360− usleep_range(wait, wait << 1);
361− wait <<= 1;
362− }
363−
364− if (ret != 0) {
365− read = xe_mmio_read32(gt, reg);
366− if ((read & mask) != val)
367− ret = 0;
368− }
369−
370− if (out_val)
371− *out_val = read;
372−
373− return ret;
430+ return __xe_mmio_wait32(gt, reg, mask, val, timeout_us, out_val, atomic, false);
374431 }
drivers/gpu/drm/xe/xe_mmio.h+0−1
@@ -22,7 +22,6 @@ u32 xe_mmio_rmw32(struct xe_gt *gt, struct xe_reg reg, u32 clr, u32 set); · 21 unmodified lines
2222 int xe_mmio_write32_and_verify(struct xe_gt *gt, struct xe_reg reg, u32 val, u32 mask, u32 eval);
2323 bool xe_mmio_in_range(const struct xe_gt *gt, const struct xe_mmio_range *range, struct xe_reg reg);
2424
25−int xe_mmio_probe_vram(struct xe_device *xe);
2625 u64 xe_mmio_read64_2x32(struct xe_gt *gt, struct xe_reg reg);
2726 int xe_mmio_wait32(struct xe_gt *gt, struct xe_reg reg, u32 mask, u32 val, u32 timeout_us,
2827 u32 *out_val, bool atomic);
drivers/gpu/drm/xe/xe_oa.c+1−1
@@ -641,7 +641,7 @@ static void xe_oa_store_flex(struct xe_oa_stream *stream, struct xe_lrc *lrc, · 640 unmodified lines
641641 u32 offset = xe_bo_ggtt_addr(lrc->bo);
642642
643643 do {
644− bb->cs[bb->len++] = MI_STORE_DATA_IMM | BIT(22) /* GGTT */ | 2;
644+ bb->cs[bb->len++] = MI_STORE_DATA_IMM | MI_SDI_GGTT | MI_SDI_NUM_DW(1);
645645 bb->cs[bb->len++] = offset + flex->offset * sizeof(u32);
646646 bb->cs[bb->len++] = 0;
647647 bb->cs[bb->len++] = flex->value;
drivers/gpu/drm/xe/xe_pat.c+10−1
@@ -7,6 +7,8 @@ · 6 unmodified lines
77
88 #include <drm/xe_drm.h>
99
10+#include <generated/xe_wa_oob.h>
11+
1012 #include "regs/xe_reg_defs.h"
1113 #include "xe_assert.h"
1214 #include "xe_device.h"
@@ -15,6 +17,7 @@ · 2 unmodified lines
1517 #include "xe_gt_mcr.h"
1618 #include "xe_mmio.h"
1719 #include "xe_sriov.h"
20+#include "xe_wa.h"
1821
1922 #define _PAT_ATS 0x47fc
2023 #define _PAT_INDEX(index) _PICK_EVEN_2RANGES(index, 8, \
@@ -382,7 +385,13 @@ void xe_pat_init_early(struct xe_device *xe) · 361 unmodified lines
382385 if (GRAPHICS_VER(xe) == 20) {
383386 xe->pat.ops = &xe2_pat_ops;
384387 xe->pat.table = xe2_pat_table;
385− xe->pat.n_entries = ARRAY_SIZE(xe2_pat_table);
388+
389+ /* Wa_16023588340. XXX: Should use XE_WA */
390+ if (GRAPHICS_VERx100(xe) == 2001)
391+ xe->pat.n_entries = 28; /* Disable CLOS3 */
392+ else
393+ xe->pat.n_entries = ARRAY_SIZE(xe2_pat_table);
394+
386395 xe->pat.idx[XE_CACHE_NONE] = 3;
387396 xe->pat.idx[XE_CACHE_WT] = 15;
388397 xe->pat.idx[XE_CACHE_WB] = 2;
drivers/gpu/drm/xe/xe_pci.c+5−2
@@ -59,6 +59,7 @@ struct xe_device_desc { · 58 unmodified lines
5959
6060 u8 has_display:1;
6161 u8 has_heci_gscfi:1;
62+ u8 has_heci_cscfi:1;
6263 u8 has_llc:1;
6364 u8 has_mmio_ext:1;
6465 u8 has_sriov:1;
@@ -345,6 +346,7 @@ static const struct xe_device_desc bmg_desc = { · 280 unmodified lines
345346 PLATFORM(BATTLEMAGE),
346347 .has_display = true,
347348 .require_force_probe = true,
349+ .has_heci_cscfi = 1,
348350 };
349351
350352 #undef PLATFORM
@@ -606,6 +608,7 @@ static int xe_info_init_early(struct xe_device *xe, · 255 unmodified lines
606608
607609 xe->info.is_dgfx = desc->is_dgfx;
608610 xe->info.has_heci_gscfi = desc->has_heci_gscfi;
611+ xe->info.has_heci_cscfi = desc->has_heci_cscfi;
609612 xe->info.has_llc = desc->has_llc;
610613 xe->info.has_mmio_ext = desc->has_mmio_ext;
611614 xe->info.has_sriov = desc->has_sriov;
@@ -815,7 +818,7 @@ static int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) · 203 unmodified lines
815818 if (err)
816819 return err;
817820
818− drm_dbg(&xe->drm, "%s %s %04x:%04x dgfx:%d gfx:%s (%d.%02d) media:%s (%d.%02d) display:%s dma_m_s:%d tc:%d gscfi:%d",
821+ drm_dbg(&xe->drm, "%s %s %04x:%04x dgfx:%d gfx:%s (%d.%02d) media:%s (%d.%02d) display:%s dma_m_s:%d tc:%d gscfi:%d cscfi:%d",
819822 desc->platform_name,
820823 subplatform_desc ? subplatform_desc->name : "",
821824 xe->info.devid, xe->info.revid,
@@ -828,7 +831,7 @@ static int xe_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent) · 6 unmodified lines
828831 xe->info.media_verx100 % 100,
829832 str_yes_no(xe->info.enable_display),
830833 xe->info.dma_mask_size, xe->info.tile_count,
831− xe->info.has_heci_gscfi);
834+ xe->info.has_heci_gscfi, xe->info.has_heci_cscfi);
832835
833836 drm_dbg(&xe->drm, "Stepping = (G:%s, M:%s, D:%s, B:%s)\n",
834837 xe_step_name(xe->info.step.graphics),
drivers/gpu/drm/xe/xe_pm.c+8−0
@@ -20,6 +20,7 @@ · 19 unmodified lines
2020 #include "xe_guc.h"
2121 #include "xe_irq.h"
2222 #include "xe_pcode.h"
23+#include "xe_trace.h"
2324 #include "xe_wa.h"
2425
2526 /**
@@ -87,6 +88,7 @@ int xe_pm_suspend(struct xe_device *xe) · 61 unmodified lines
8788 int err;
8889
8990 drm_dbg(&xe->drm, "Suspending device\n");
91+ trace_xe_pm_suspend(xe, __builtin_return_address(0));
9092
9193 for_each_gt(gt, xe, id)
9294 xe_gt_suspend_prepare(gt);
@@ -131,6 +133,7 @@ int xe_pm_resume(struct xe_device *xe) · 38 unmodified lines
131133 int err;
132134
133135 drm_dbg(&xe->drm, "Resuming device\n");
136+ trace_xe_pm_resume(xe, __builtin_return_address(0));
134137
135138 for_each_tile(tile, xe, id)
136139 xe_wa_apply_tile_workarounds(tile);
@@ -326,6 +329,7 @@ int xe_pm_runtime_suspend(struct xe_device *xe) · 189 unmodified lines
326329 u8 id;
327330 int err = 0;
328331
332+ trace_xe_pm_runtime_suspend(xe, __builtin_return_address(0));
329333 /* Disable access_ongoing asserts and prevent recursive pm calls */
330334 xe_pm_write_callback_task(xe, current);
331335
@@ -399,6 +403,7 @@ int xe_pm_runtime_resume(struct xe_device *xe) · 67 unmodified lines
399403 u8 id;
400404 int err = 0;
401405
406+ trace_xe_pm_runtime_resume(xe, __builtin_return_address(0));
402407 /* Disable access_ongoing asserts and prevent recursive pm calls */
403408 xe_pm_write_callback_task(xe, current);
404409
@@ -463,6 +468,7 @@ static void pm_runtime_lockdep_prime(void) · 58 unmodified lines
463468 */
464469 void xe_pm_runtime_get(struct xe_device *xe)
465470 {
471+ trace_xe_pm_runtime_get(xe, __builtin_return_address(0));
466472 pm_runtime_get_noresume(xe->drm.dev);
467473
468474 if (xe_pm_read_callback_task(xe) == current)
@@ -478,6 +484,7 @@ void xe_pm_runtime_get(struct xe_device *xe) · 9 unmodified lines
478484 */
479485 void xe_pm_runtime_put(struct xe_device *xe)
480486 {
487+ trace_xe_pm_runtime_put(xe, __builtin_return_address(0));
481488 if (xe_pm_read_callback_task(xe) == current) {
482489 pm_runtime_put_noidle(xe->drm.dev);
483490 } else {
@@ -495,6 +502,7 @@ void xe_pm_runtime_put(struct xe_device *xe) · 11 unmodified lines
495502 */
496503 int xe_pm_runtime_get_ioctl(struct xe_device *xe)
497504 {
505+ trace_xe_pm_runtime_get_ioctl(xe, __builtin_return_address(0));
498506 if (WARN_ON(xe_pm_read_callback_task(xe) == current))
499507 return -ELOOP;
500508
drivers/gpu/drm/xe/xe_preempt_fence.c+9−3
@@ -17,10 +17,16 @@ static void preempt_fence_work_func(struct work_struct *w) · 16 unmodified lines
1717 container_of(w, typeof(*pfence), preempt_work);
1818 struct xe_exec_queue *q = pfence->q;
1919
20− if (pfence->error)
20+ if (pfence->error) {
2121 dma_fence_set_error(&pfence->base, pfence->error);
22− else
23− q->ops->suspend_wait(q);
22+ } else if (!q->ops->reset_status(q)) {
23+ int err = q->ops->suspend_wait(q);
24+
25+ if (err)
26+ dma_fence_set_error(&pfence->base, err);
27+ } else {
28+ dma_fence_set_error(&pfence->base, -ENOENT);
29+ }
2430
2531 dma_fence_signal(&pfence->base);
2632 /*
drivers/gpu/drm/xe/xe_pt.c+848−462
@@ -9,12 +9,15 @@ · 8 unmodified lines
99 #include "xe_bo.h"
1010 #include "xe_device.h"
1111 #include "xe_drm_client.h"
12+#include "xe_exec_queue.h"
1213 #include "xe_gt.h"
1314 #include "xe_gt_tlb_invalidation.h"
1415 #include "xe_migrate.h"
1516 #include "xe_pt_types.h"
1617 #include "xe_pt_walk.h"
1718 #include "xe_res_cursor.h"
19+#include "xe_sched_job.h"
20+#include "xe_sync.h"
1821 #include "xe_trace.h"
1922 #include "xe_ttm_stolen_mgr.h"
2023 #include "xe_vm.h"
@@ -325,6 +328,7 @@ xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, · 304 unmodified lines
325328 entry->pt = parent;
326329 entry->flags = 0;
327330 entry->qwords = 0;
331+ entry->pt_bo->update_index = -1;
328332
329333 if (alloc_entries) {
330334 entry->pt_entries = kmalloc_array(XE_PDES,
@@ -842,19 +846,27 @@ xe_vm_populate_pgtable(struct xe_migrate_pt_update *pt_update, struct xe_tile *t · 511 unmodified lines
842846 }
843847 }
844848
845−static void xe_pt_abort_bind(struct xe_vma *vma,
846− struct xe_vm_pgtable_update *entries,
847− u32 num_entries)
849+static void xe_pt_cancel_bind(struct xe_vma *vma,
850+ struct xe_vm_pgtable_update *entries,
851+ u32 num_entries)
848852 {
849853 u32 i, j;
850854
851855 for (i = 0; i < num_entries; i++) {
852− if (!entries[i].pt_entries)
856+ struct xe_pt *pt = entries[i].pt;
857+
858+ if (!pt)
853859 continue;
854860
855− for (j = 0; j < entries[i].qwords; j++)
856− xe_pt_destroy(entries[i].pt_entries[j].pt, xe_vma_vm(vma)->flags, NULL);
861+ if (pt->level) {
862+ for (j = 0; j < entries[i].qwords; j++)
863+ xe_pt_destroy(entries[i].pt_entries[j].pt,
864+ xe_vma_vm(vma)->flags, NULL);
865+ }
866+
857867 kfree(entries[i].pt_entries);
868+ entries[i].pt_entries = NULL;
869+ entries[i].qwords = 0;
858870 }
859871 }
860872
@@ -864,18 +876,15 @@ static void xe_pt_commit_locks_assert(struct xe_vma *vma) · 3 unmodified lines
864876
865877 lockdep_assert_held(&vm->lock);
866878
867− if (xe_vma_is_userptr(vma))
868− lockdep_assert_held_read(&vm->userptr.notifier_lock);
869− else if (!xe_vma_is_null(vma))
879+ if (!xe_vma_is_userptr(vma) && !xe_vma_is_null(vma))
870880 dma_resv_assert_held(xe_vma_bo(vma)->ttm.base.resv);
871881
872882 xe_vm_assert_held(vm);
873883 }
874884
875−static void xe_pt_commit_bind(struct xe_vma *vma,
876− struct xe_vm_pgtable_update *entries,
877− u32 num_entries, bool rebind,
878− struct llist_head *deferred)
885+static void xe_pt_commit(struct xe_vma *vma,
886+ struct xe_vm_pgtable_update *entries,
887+ u32 num_entries, struct llist_head *deferred)
879888 {
880889 u32 i, j;
881890
@@ -883,31 +892,90 @@ static void xe_pt_commit_bind(struct xe_vma *vma, · 1 unmodified line
883892
884893 for (i = 0; i < num_entries; i++) {
885894 struct xe_pt *pt = entries[i].pt;
895+
896+ if (!pt->level)
897+ continue;
898+
899+ for (j = 0; j < entries[i].qwords; j++) {
900+ struct xe_pt *oldpte = entries[i].pt_entries[j].pt;
901+
902+ xe_pt_destroy(oldpte, xe_vma_vm(vma)->flags, deferred);
903+ }
904+ }
905+}
906+
907+static void xe_pt_abort_bind(struct xe_vma *vma,
908+ struct xe_vm_pgtable_update *entries,
909+ u32 num_entries, bool rebind)
910+{
911+ int i, j;
912+
913+ xe_pt_commit_locks_assert(vma);
914+
915+ for (i = num_entries - 1; i >= 0; --i) {
916+ struct xe_pt *pt = entries[i].pt;
886917 struct xe_pt_dir *pt_dir;
887918
888919 if (!rebind)
889− pt->num_live += entries[i].qwords;
920+ pt->num_live -= entries[i].qwords;
890921
891− if (!pt->level) {
892− kfree(entries[i].pt_entries);
922+ if (!pt->level)
893923 continue;
924+
925+ pt_dir = as_xe_pt_dir(pt);
926+ for (j = 0; j < entries[i].qwords; j++) {
927+ u32 j_ = j + entries[i].ofs;
928+ struct xe_pt *newpte = xe_pt_entry(pt_dir, j_);
929+ struct xe_pt *oldpte = entries[i].pt_entries[j].pt;
930+
931+ pt_dir->children[j_] = oldpte ? &oldpte->base : 0;
932+ xe_pt_destroy(newpte, xe_vma_vm(vma)->flags, NULL);
894933 }
934+ }
935+}
936+
937+static void xe_pt_commit_prepare_bind(struct xe_vma *vma,
938+ struct xe_vm_pgtable_update *entries,
939+ u32 num_entries, bool rebind)
940+{
941+ u32 i, j;
942+
943+ xe_pt_commit_locks_assert(vma);
944+
945+ for (i = 0; i < num_entries; i++) {
946+ struct xe_pt *pt = entries[i].pt;
947+ struct xe_pt_dir *pt_dir;
948+
949+ if (!rebind)
950+ pt->num_live += entries[i].qwords;
951+
952+ if (!pt->level)
953+ continue;
895954
896955 pt_dir = as_xe_pt_dir(pt);
897956 for (j = 0; j < entries[i].qwords; j++) {
898957 u32 j_ = j + entries[i].ofs;
899958 struct xe_pt *newpte = entries[i].pt_entries[j].pt;
959+ struct xe_pt *oldpte = NULL;
900960
901961 if (xe_pt_entry(pt_dir, j_))
902− xe_pt_destroy(xe_pt_entry(pt_dir, j_),
903− xe_vma_vm(vma)->flags, deferred);
962+ oldpte = xe_pt_entry(pt_dir, j_);
904963
905964 pt_dir->children[j_] = &newpte->base;
965+ entries[i].pt_entries[j].pt = oldpte;
906966 }
907− kfree(entries[i].pt_entries);
908967 }
909968 }
910969
970+static void xe_pt_free_bind(struct xe_vm_pgtable_update *entries,
971+ u32 num_entries)
972+{
973+ u32 i;
974+
975+ for (i = 0; i < num_entries; i++)
976+ kfree(entries[i].pt_entries);
977+}
978+
911979 static int
912980 xe_pt_prepare_bind(struct xe_tile *tile, struct xe_vma *vma,
913981 struct xe_vm_pgtable_update *entries, u32 *num_entries)
@@ -918,20 +986,19 @@ xe_pt_prepare_bind(struct xe_tile *tile, struct xe_vma *vma, · 4 unmodified lines
918986 err = xe_pt_stage_bind(tile, vma, entries, num_entries);
919987 if (!err)
920988 xe_tile_assert(tile, *num_entries);
921− else /* abort! */
922− xe_pt_abort_bind(vma, entries, *num_entries);
923989
924990 return err;
925991 }
926992
927993 static void xe_vm_dbg_print_entries(struct xe_device *xe,
928994 const struct xe_vm_pgtable_update *entries,
929− unsigned int num_entries)
995+ unsigned int num_entries, bool bind)
930996 #if (IS_ENABLED(CONFIG_DRM_XE_DEBUG_VM))
931997 {
932998 unsigned int i;
933999
934− vm_dbg(&xe->drm, "%u entries to update\n", num_entries);
1000+ vm_dbg(&xe->drm, "%s: %u entries to update\n", bind ? "bind" : "unbind",
1001+ num_entries);
9351002 for (i = 0; i < num_entries; i++) {
9361003 const struct xe_vm_pgtable_update *entry = &entries[i];
9371004 struct xe_pt *xe_pt = entry->pt;
@@ -952,66 +1019,108 @@ static void xe_vm_dbg_print_entries(struct xe_device *xe, · 14 unmodified lines
9521019 {}
9531020 #endif
9541021
955−#ifdef CONFIG_DRM_XE_USERPTR_INVAL_INJECT
956−
957−static int xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)
1022+static bool no_in_syncs(struct xe_sync_entry *syncs, u32 num_syncs)
9581023 {
959− u32 divisor = uvma->userptr.divisor ? uvma->userptr.divisor : 2;
960− static u32 count;
1024+ int i;
9611025
962− if (count++ % divisor == divisor - 1) {
963− struct xe_vm *vm = xe_vma_vm(&uvma->vma);
1026+ for (i = 0; i < num_syncs; i++) {
1027+ struct dma_fence *fence = syncs[i].fence;
9641028
965− uvma->userptr.divisor = divisor << 1;
966− spin_lock(&vm->userptr.invalidated_lock);
967− list_move_tail(&uvma->userptr.invalidate_link,
968− &vm->userptr.invalidated);
969− spin_unlock(&vm->userptr.invalidated_lock);
970− return true;
1029+ if (fence && !test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
1030+ &fence->flags))
1031+ return false;
9711032 }
9721033
973− return false;
1034+ return true;
9741035 }
9751036
976−#else
1037+static int job_test_add_deps(struct xe_sched_job *job,
1038+ struct dma_resv *resv,
1039+ enum dma_resv_usage usage)
1040+{
1041+ if (!job) {
1042+ if (!dma_resv_test_signaled(resv, usage))
1043+ return -ETIME;
9771044
978−static bool xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)
1045+ return 0;
1046+ }
1047+
1048+ return xe_sched_job_add_deps(job, resv, usage);
1049+}
1050+
1051+static int vma_add_deps(struct xe_vma *vma, struct xe_sched_job *job)
9791052 {
980− return false;
1053+ struct xe_bo *bo = xe_vma_bo(vma);
1054+
1055+ xe_bo_assert_held(bo);
1056+
1057+ if (bo && !bo->vm)
1058+ return job_test_add_deps(job, bo->ttm.base.resv,
1059+ DMA_RESV_USAGE_KERNEL);
1060+
1061+ return 0;
9811062 }
9821063
983−#endif
1064+static int op_add_deps(struct xe_vm *vm, struct xe_vma_op *op,
1065+ struct xe_sched_job *job)
1066+{
1067+ int err = 0;
9841068
985−/**
986− * struct xe_pt_migrate_pt_update - Callback argument for pre-commit callbacks
987− * @base: Base we derive from.
988− * @bind: Whether this is a bind or an unbind operation. A bind operation
989− * makes the pre-commit callback error with -EAGAIN if it detects a
990− * pending invalidation.
991− * @locked: Whether the pre-commit callback locked the userptr notifier lock
992− * and it needs unlocking.
993− */
994−struct xe_pt_migrate_pt_update {
995− struct xe_migrate_pt_update base;
996− bool bind;
997− bool locked;
998−};
1069+ switch (op->base.op) {
1070+ case DRM_GPUVA_OP_MAP:
1071+ if (!op->map.immediate && xe_vm_in_fault_mode(vm))
1072+ break;
1073+
1074+ err = vma_add_deps(op->map.vma, job);
1075+ break;
1076+ case DRM_GPUVA_OP_REMAP:
1077+ if (op->remap.prev)
1078+ err = vma_add_deps(op->remap.prev, job);
1079+ if (!err && op->remap.next)
1080+ err = vma_add_deps(op->remap.next, job);
1081+ break;
1082+ case DRM_GPUVA_OP_UNMAP:
1083+ break;
1084+ case DRM_GPUVA_OP_PREFETCH:
1085+ err = vma_add_deps(gpuva_to_vma(op->base.prefetch.va), job);
1086+ break;
1087+ default:
1088+ drm_warn(&vm->xe->drm, "NOT POSSIBLE");
1089+ }
1090+
1091+ return err;
1092+}
9991093
1000−/*
1001− * This function adds the needed dependencies to a page-table update job
1002− * to make sure racing jobs for separate bind engines don't race writing
1003− * to the same page-table range, wreaking havoc. Initially use a single
1004− * fence for the entire VM. An optimization would use smaller granularity.
1005− */
10061094 static int xe_pt_vm_dependencies(struct xe_sched_job *job,
1007− struct xe_range_fence_tree *rftree,
1008− u64 start, u64 last)
1095+ struct xe_vm *vm,
1096+ struct xe_vma_ops *vops,
1097+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1098+ struct xe_range_fence_tree *rftree)
10091099 {
10101100 struct xe_range_fence *rtfence;
10111101 struct dma_fence *fence;
1012− int err;
1102+ struct xe_vma_op *op;
1103+ int err = 0, i;
1104+
1105+ xe_vm_assert_held(vm);
10131106
1014− rtfence = xe_range_fence_tree_first(rftree, start, last);
1107+ if (!job && !no_in_syncs(vops->syncs, vops->num_syncs))
1108+ return -ETIME;
1109+
1110+ if (!job && !xe_exec_queue_is_idle(pt_update_ops->q))
1111+ return -ETIME;
1112+
1113+ if (pt_update_ops->wait_vm_bookkeep || pt_update_ops->wait_vm_kernel) {
1114+ err = job_test_add_deps(job, xe_vm_resv(vm),
1115+ pt_update_ops->wait_vm_bookkeep ?
1116+ DMA_RESV_USAGE_BOOKKEEP :
1117+ DMA_RESV_USAGE_KERNEL);
1118+ if (err)
1119+ return err;
1120+ }
1121+
1122+ rtfence = xe_range_fence_tree_first(rftree, pt_update_ops->start,
1123+ pt_update_ops->last);
10151124 while (rtfence) {
10161125 fence = rtfence->fence;
10171126
@@ -1029,80 +1138,173 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job, · 11 unmodified lines
10291138 return err;
10301139 }
10311140
1032− rtfence = xe_range_fence_tree_next(rtfence, start, last);
1141+ rtfence = xe_range_fence_tree_next(rtfence,
1142+ pt_update_ops->start,
1143+ pt_update_ops->last);
10331144 }
10341145
1035− return 0;
1146+ list_for_each_entry(op, &vops->list, link) {
1147+ err = op_add_deps(vm, op, job);
1148+ if (err)
1149+ return err;
1150+ }
1151+
1152+ if (job)
1153+ err = xe_sched_job_last_fence_add_dep(job, vm);
1154+ else
1155+ err = xe_exec_queue_last_fence_test_dep(pt_update_ops->q, vm);
1156+
1157+ for (i = 0; job && !err && i < vops->num_syncs; i++)
1158+ err = xe_sync_entry_add_deps(&vops->syncs[i], job);
1159+
1160+ return err;
10361161 }
10371162
10381163 static int xe_pt_pre_commit(struct xe_migrate_pt_update *pt_update)
10391164 {
1040− struct xe_range_fence_tree *rftree =
1041− &xe_vma_vm(pt_update->vma)->rftree[pt_update->tile_id];
1165+ struct xe_vma_ops *vops = pt_update->vops;
1166+ struct xe_vm *vm = vops->vm;
1167+ struct xe_range_fence_tree *rftree = &vm->rftree[pt_update->tile_id];
1168+ struct xe_vm_pgtable_update_ops *pt_update_ops =
1169+ &vops->pt_update_ops[pt_update->tile_id];
1170+
1171+ return xe_pt_vm_dependencies(pt_update->job, vm, pt_update->vops,
1172+ pt_update_ops, rftree);
1173+}
10421174
1043− return xe_pt_vm_dependencies(pt_update->job, rftree,
1044− pt_update->start, pt_update->last);
1175+#ifdef CONFIG_DRM_XE_USERPTR_INVAL_INJECT
1176+
1177+static bool xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)
1178+{
1179+ u32 divisor = uvma->userptr.divisor ? uvma->userptr.divisor : 2;
1180+ static u32 count;
1181+
1182+ if (count++ % divisor == divisor - 1) {
1183+ uvma->userptr.divisor = divisor << 1;
1184+ return true;
1185+ }
1186+
1187+ return false;
10451188 }
10461189
1047−static int xe_pt_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)
1190+#else
1191+
1192+static bool xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)
10481193 {
1049− struct xe_pt_migrate_pt_update *userptr_update =
1050− container_of(pt_update, typeof(*userptr_update), base);
1051− struct xe_userptr_vma *uvma = to_userptr_vma(pt_update->vma);
1052− unsigned long notifier_seq = uvma->userptr.notifier_seq;
1053− struct xe_vm *vm = xe_vma_vm(&uvma->vma);
1054− int err = xe_pt_vm_dependencies(pt_update->job,
1055− &vm->rftree[pt_update->tile_id],
1056− pt_update->start,
1057− pt_update->last);
1194+ return false;
1195+}
10581196
1059− if (err)
1060− return err;
1197+#endif
10611198
1062− userptr_update->locked = false;
1199+static int vma_check_userptr(struct xe_vm *vm, struct xe_vma *vma,
1200+ struct xe_vm_pgtable_update_ops *pt_update)
1201+{
1202+ struct xe_userptr_vma *uvma;
1203+ unsigned long notifier_seq;
10631204
1064− /*
1065− * Wait until nobody is running the invalidation notifier, and
1066− * since we're exiting the loop holding the notifier lock,
1067− * nobody can proceed invalidating either.
1068− *
1069− * Note that we don't update the vma->userptr.notifier_seq since
1070− * we don't update the userptr pages.
1071− */
1072− do {
1073− down_read(&vm->userptr.notifier_lock);
1074− if (!mmu_interval_read_retry(&uvma->userptr.notifier,
1075− notifier_seq))
1076− break;
1205+ lockdep_assert_held_read(&vm->userptr.notifier_lock);
10771206
1078− up_read(&vm->userptr.notifier_lock);
1207+ if (!xe_vma_is_userptr(vma))
1208+ return 0;
10791209
1080− if (userptr_update->bind)
1081− return -EAGAIN;
1210+ uvma = to_userptr_vma(vma);
1211+ notifier_seq = uvma->userptr.notifier_seq;
10821212
1083− notifier_seq = mmu_interval_read_begin(&uvma->userptr.notifier);
1084− } while (true);
1213+ if (uvma->userptr.initial_bind && !xe_vm_in_fault_mode(vm))
1214+ return 0;
10851215
1086− /* Inject errors to test_whether they are handled correctly */
1087− if (userptr_update->bind && xe_pt_userptr_inject_eagain(uvma)) {
1088− up_read(&vm->userptr.notifier_lock);
1216+ if (!mmu_interval_read_retry(&uvma->userptr.notifier,
1217+ notifier_seq) &&
1218+ !xe_pt_userptr_inject_eagain(uvma))
1219+ return 0;
1220+
1221+ if (xe_vm_in_fault_mode(vm)) {
10891222 return -EAGAIN;
1090− }
1223+ } else {
1224+ spin_lock(&vm->userptr.invalidated_lock);
1225+ list_move_tail(&uvma->userptr.invalidate_link,
1226+ &vm->userptr.invalidated);
1227+ spin_unlock(&vm->userptr.invalidated_lock);
10911228
1092− userptr_update->locked = true;
1229+ if (xe_vm_in_preempt_fence_mode(vm)) {
1230+ struct dma_resv_iter cursor;
1231+ struct dma_fence *fence;
1232+ long err;
1233+
1234+ dma_resv_iter_begin(&cursor, xe_vm_resv(vm),
1235+ DMA_RESV_USAGE_BOOKKEEP);
1236+ dma_resv_for_each_fence_unlocked(&cursor, fence)
1237+ dma_fence_enable_sw_signaling(fence);
1238+ dma_resv_iter_end(&cursor);
1239+
1240+ err = dma_resv_wait_timeout(xe_vm_resv(vm),
1241+ DMA_RESV_USAGE_BOOKKEEP,
1242+ false, MAX_SCHEDULE_TIMEOUT);
1243+ XE_WARN_ON(err <= 0);
1244+ }
1245+ }
10931246
10941247 return 0;
10951248 }
10961249
1097−static const struct xe_migrate_pt_update_ops bind_ops = {
1098− .populate = xe_vm_populate_pgtable,
1099− .pre_commit = xe_pt_pre_commit,
1100−};
1250+static int op_check_userptr(struct xe_vm *vm, struct xe_vma_op *op,
1251+ struct xe_vm_pgtable_update_ops *pt_update)
1252+{
1253+ int err = 0;
11011254
1102−static const struct xe_migrate_pt_update_ops userptr_bind_ops = {
1103− .populate = xe_vm_populate_pgtable,
1104− .pre_commit = xe_pt_userptr_pre_commit,
1105−};
1255+ lockdep_assert_held_read(&vm->userptr.notifier_lock);
1256+
1257+ switch (op->base.op) {
1258+ case DRM_GPUVA_OP_MAP:
1259+ if (!op->map.immediate && xe_vm_in_fault_mode(vm))
1260+ break;
1261+
1262+ err = vma_check_userptr(vm, op->map.vma, pt_update);
1263+ break;
1264+ case DRM_GPUVA_OP_REMAP:
1265+ if (op->remap.prev)
1266+ err = vma_check_userptr(vm, op->remap.prev, pt_update);
1267+ if (!err && op->remap.next)
1268+ err = vma_check_userptr(vm, op->remap.next, pt_update);
1269+ break;
1270+ case DRM_GPUVA_OP_UNMAP:
1271+ break;
1272+ case DRM_GPUVA_OP_PREFETCH:
1273+ err = vma_check_userptr(vm, gpuva_to_vma(op->base.prefetch.va),
1274+ pt_update);
1275+ break;
1276+ default:
1277+ drm_warn(&vm->xe->drm, "NOT POSSIBLE");
1278+ }
1279+
1280+ return err;
1281+}
1282+
1283+static int xe_pt_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)
1284+{
1285+ struct xe_vm *vm = pt_update->vops->vm;
1286+ struct xe_vma_ops *vops = pt_update->vops;
1287+ struct xe_vm_pgtable_update_ops *pt_update_ops =
1288+ &vops->pt_update_ops[pt_update->tile_id];
1289+ struct xe_vma_op *op;
1290+ int err;
1291+
1292+ err = xe_pt_pre_commit(pt_update);
1293+ if (err)
1294+ return err;
1295+
1296+ down_read(&vm->userptr.notifier_lock);
1297+
1298+ list_for_each_entry(op, &vops->list, link) {
1299+ err = op_check_userptr(vm, op, pt_update_ops);
1300+ if (err) {
1301+ up_read(&vm->userptr.notifier_lock);
1302+ break;
1303+ }
1304+ }
1305+
1306+ return err;
1307+}
11061308
11071309 struct invalidation_fence {
11081310 struct xe_gt_tlb_invalidation_fence base;
@@ -1115,23 +1317,6 @@ struct invalidation_fence { · 6 unmodified lines
11151317 u32 asid;
11161318 };
11171319
1118−static const char *
1119−invalidation_fence_get_driver_name(struct dma_fence *dma_fence)
1120−{
1121− return "xe";
1122−}
1123−
1124−static const char *
1125−invalidation_fence_get_timeline_name(struct dma_fence *dma_fence)
1126−{
1127− return "invalidation_fence";
1128−}
1129−
1130−static const struct dma_fence_ops invalidation_fence_ops = {
1131− .get_driver_name = invalidation_fence_get_driver_name,
1132− .get_timeline_name = invalidation_fence_get_timeline_name,
1133−};
1134−
11351320 static void invalidation_fence_cb(struct dma_fence *fence,
11361321 struct dma_fence_cb *cb)
11371322 {
@@ -1161,24 +1346,17 @@ static void invalidation_fence_work_func(struct work_struct *w) · 23 unmodified lines
11611346 ifence->end, ifence->asid);
11621347 }
11631348
1164−static int invalidation_fence_init(struct xe_gt *gt,
1165− struct invalidation_fence *ifence,
1166− struct dma_fence *fence,
1167− u64 start, u64 end, u32 asid)
1349+static void invalidation_fence_init(struct xe_gt *gt,
1350+ struct invalidation_fence *ifence,
1351+ struct dma_fence *fence,
1352+ u64 start, u64 end, u32 asid)
11681353 {
11691354 int ret;
11701355
11711356 trace_xe_gt_tlb_invalidation_fence_create(gt_to_xe(gt), &ifence->base);
11721357
1173− spin_lock_irq(&gt->tlb_invalidation.lock);
1174− dma_fence_init(&ifence->base.base, &invalidation_fence_ops,
1175− &gt->tlb_invalidation.lock,
1176− dma_fence_context_alloc(1), 1);
1177− spin_unlock_irq(&gt->tlb_invalidation.lock);
1178−
1179− INIT_LIST_HEAD(&ifence->base.link);
1358+ xe_gt_tlb_invalidation_fence_init(gt, &ifence->base, false);
11801359
1181− dma_fence_get(&ifence->base.base); /* Ref for caller */
11821360 ifence->fence = fence;
11831361 ifence->gt = gt;
11841362 ifence->start = start;
@@ -1196,192 +1374,6 @@ static int invalidation_fence_init(struct xe_gt *gt, · 11 unmodified lines
11961374 }
11971375
11981376 xe_gt_assert(gt, !ret || ret == -ENOENT);
1199−
1200− return ret && ret != -ENOENT ? ret : 0;
1201−}
1202−
1203−static void xe_pt_calc_rfence_interval(struct xe_vma *vma,
1204− struct xe_pt_migrate_pt_update *update,
1205− struct xe_vm_pgtable_update *entries,
1206− u32 num_entries)
1207−{
1208− int i, level = 0;
1209−
1210− for (i = 0; i < num_entries; i++) {
1211− const struct xe_vm_pgtable_update *entry = &entries[i];
1212−
1213− if (entry->pt->level > level)
1214− level = entry->pt->level;
1215− }
1216−
1217− /* Greedy (non-optimal) calculation but simple */
1218− update->base.start = ALIGN_DOWN(xe_vma_start(vma),
1219− 0x1ull << xe_pt_shift(level));
1220− update->base.last = ALIGN(xe_vma_end(vma),
1221− 0x1ull << xe_pt_shift(level)) - 1;
1222−}
1223−
1224−/**
1225− * __xe_pt_bind_vma() - Build and connect a page-table tree for the vma
1226− * address range.
1227− * @tile: The tile to bind for.
1228− * @vma: The vma to bind.
1229− * @q: The exec_queue with which to do pipelined page-table updates.
1230− * @syncs: Entries to sync on before binding the built tree to the live vm tree.
1231− * @num_syncs: Number of @sync entries.
1232− * @rebind: Whether we're rebinding this vma to the same address range without
1233− * an unbind in-between.
1234− *
1235− * This function builds a page-table tree (see xe_pt_stage_bind() for more
1236− * information on page-table building), and the xe_vm_pgtable_update entries
1237− * abstracting the operations needed to attach it to the main vm tree. It
1238− * then takes the relevant locks and updates the metadata side of the main
1239− * vm tree and submits the operations for pipelined attachment of the
1240− * gpu page-table to the vm main tree, (which can be done either by the
1241− * cpu and the GPU).
1242− *
1243− * Return: A valid dma-fence representing the pipelined attachment operation
1244− * on success, an error pointer on error.
1245− */
1246−struct dma_fence *
1247−__xe_pt_bind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queue *q,
1248− struct xe_sync_entry *syncs, u32 num_syncs,
1249− bool rebind)
1250−{
1251− struct xe_vm_pgtable_update entries[XE_VM_MAX_LEVEL * 2 + 1];
1252− struct xe_pt_migrate_pt_update bind_pt_update = {
1253− .base = {
1254− .ops = xe_vma_is_userptr(vma) ? &userptr_bind_ops : &bind_ops,
1255− .vma = vma,
1256− .tile_id = tile->id,
1257− },
1258− .bind = true,
1259− };
1260− struct xe_vm *vm = xe_vma_vm(vma);
1261− u32 num_entries;
1262− struct dma_fence *fence;
1263− struct invalidation_fence *ifence = NULL;
1264− struct xe_range_fence *rfence;
1265− int err;
1266−
1267− bind_pt_update.locked = false;
1268− xe_bo_assert_held(xe_vma_bo(vma));
1269− xe_vm_assert_held(vm);
1270−
1271− vm_dbg(&xe_vma_vm(vma)->xe->drm,
1272− "Preparing bind, with range [%llx...%llx) engine %p.\n",
1273− xe_vma_start(vma), xe_vma_end(vma), q);
1274−
1275− err = xe_pt_prepare_bind(tile, vma, entries, &num_entries);
1276− if (err)
1277− goto err;
1278−
1279− err = dma_resv_reserve_fences(xe_vm_resv(vm), 1);
1280− if (!err && !xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1281− err = dma_resv_reserve_fences(xe_vma_bo(vma)->ttm.base.resv, 1);
1282− if (err)
1283− goto err;
1284−
1285− xe_tile_assert(tile, num_entries <= ARRAY_SIZE(entries));
1286−
1287− xe_vm_dbg_print_entries(tile_to_xe(tile), entries, num_entries);
1288− xe_pt_calc_rfence_interval(vma, &bind_pt_update, entries,
1289− num_entries);
1290−
1291− /*
1292− * If rebind, we have to invalidate TLB on !LR vms to invalidate
1293− * cached PTEs point to freed memory. on LR vms this is done
1294− * automatically when the context is re-enabled by the rebind worker,
1295− * or in fault mode it was invalidated on PTE zapping.
1296− *
1297− * If !rebind, and scratch enabled VMs, there is a chance the scratch
1298− * PTE is already cached in the TLB so it needs to be invalidated.
1299− * on !LR VMs this is done in the ring ops preceding a batch, but on
1300− * non-faulting LR, in particular on user-space batch buffer chaining,
1301− * it needs to be done here.
1302− */
1303− if ((!rebind && xe_vm_has_scratch(vm) && xe_vm_in_preempt_fence_mode(vm))) {
1304− ifence = kzalloc(sizeof(*ifence), GFP_KERNEL);
1305− if (!ifence)
1306− return ERR_PTR(-ENOMEM);
1307− } else if (rebind && !xe_vm_in_lr_mode(vm)) {
1308− /* We bump also if batch_invalidate_tlb is true */
1309− vm->tlb_flush_seqno++;
1310− }
1311−
1312− rfence = kzalloc(sizeof(*rfence), GFP_KERNEL);
1313− if (!rfence) {
1314− kfree(ifence);
1315− return ERR_PTR(-ENOMEM);
1316− }
1317−
1318− fence = xe_migrate_update_pgtables(tile->migrate,
1319− vm, xe_vma_bo(vma), q,
1320− entries, num_entries,
1321− syncs, num_syncs,
1322− &bind_pt_update.base);
1323− if (!IS_ERR(fence)) {
1324− bool last_munmap_rebind = vma->gpuva.flags & XE_VMA_LAST_REBIND;
1325− LLIST_HEAD(deferred);
1326− int err;
1327−
1328− err = xe_range_fence_insert(&vm->rftree[tile->id], rfence,
1329− &xe_range_fence_kfree_ops,
1330− bind_pt_update.base.start,
1331− bind_pt_update.base.last, fence);
1332− if (err)
1333− dma_fence_wait(fence, false);
1334−
1335− /* TLB invalidation must be done before signaling rebind */
1336− if (ifence) {
1337− int err = invalidation_fence_init(tile->primary_gt,
1338− ifence, fence,
1339− xe_vma_start(vma),
1340− xe_vma_end(vma),
1341− xe_vma_vm(vma)->usm.asid);
1342− if (err) {
1343− dma_fence_put(fence);
1344− kfree(ifence);
1345− return ERR_PTR(err);
1346− }
1347− fence = &ifence->base.base;
1348− }
1349−
1350− /* add shared fence now for pagetable delayed destroy */
1351− dma_resv_add_fence(xe_vm_resv(vm), fence, rebind ||
1352− last_munmap_rebind ?
1353− DMA_RESV_USAGE_KERNEL :
1354− DMA_RESV_USAGE_BOOKKEEP);
1355−
1356− if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1357− dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence,
1358− DMA_RESV_USAGE_BOOKKEEP);
1359− xe_pt_commit_bind(vma, entries, num_entries, rebind,
1360− bind_pt_update.locked ? &deferred : NULL);
1361−
1362− /* This vma is live (again?) now */
1363− vma->tile_present |= BIT(tile->id);
1364−
1365− if (bind_pt_update.locked) {
1366− to_userptr_vma(vma)->userptr.initial_bind = true;
1367− up_read(&vm->userptr.notifier_lock);
1368− xe_bo_put_commit(&deferred);
1369− }
1370− if (!rebind && last_munmap_rebind &&
1371− xe_vm_in_preempt_fence_mode(vm))
1372− xe_vm_queue_rebind_worker(vm);
1373− } else {
1374− kfree(rfence);
1375− kfree(ifence);
1376− if (bind_pt_update.locked)
1377− up_read(&vm->userptr.notifier_lock);
1378− xe_pt_abort_bind(vma, entries, num_entries);
1379− }
1380−
1381− return fence;
1382−
1383−err:
1384− return ERR_PTR(err);
13851377 }
13861378
13871379 struct xe_pt_stage_unbind_walk {
@@ -1466,6 +1458,7 @@ xe_pt_stage_unbind_post_descend(struct xe_ptw *parent, pgoff_t offset, · 78 unmodified lines
14661458 struct xe_pt *xe_child = container_of(*child, typeof(*xe_child), base);
14671459 pgoff_t end_offset;
14681460 u64 size = 1ull << walk->shifts[--level];
1461+ int err;
14691462
14701463 if (!IS_ALIGNED(addr, size))
14711464 addr = xe_walk->modified_start;
@@ -1481,7 +1474,10 @@ xe_pt_stage_unbind_post_descend(struct xe_ptw *parent, pgoff_t offset, · 9 unmodified lines
14811474 &end_offset))
14821475 return 0;
14831476
1484− (void)xe_pt_new_shared(&xe_walk->wupd, xe_child, offset, false);
1477+ err = xe_pt_new_shared(&xe_walk->wupd, xe_child, offset, true);
1478+ if (err)
1479+ return err;
1480+
14851481 xe_walk->wupd.updates[level].update->qwords = end_offset - offset;
14861482
14871483 return 0;
@@ -1534,8 +1530,8 @@ xe_migrate_clear_pgtable_callback(struct xe_migrate_pt_update *pt_update, · 46 unmodified lines
15341530 void *ptr, u32 qword_ofs, u32 num_qwords,
15351531 const struct xe_vm_pgtable_update *update)
15361532 {
1537− struct xe_vma *vma = pt_update->vma;
1538− u64 empty = __xe_pt_empty_pte(tile, xe_vma_vm(vma), update->pt->level);
1533+ struct xe_vm *vm = pt_update->vops->vm;
1534+ u64 empty = __xe_pt_empty_pte(tile, vm, update->pt->level);
15391535 int i;
15401536
15411537 if (map && map->is_iomem)
@@ -1549,181 +1545,571 @@ xe_migrate_clear_pgtable_callback(struct xe_migrate_pt_update *pt_update, · 7 unmodified lines
15491545 memset64(ptr, empty, num_qwords);
15501546 }
15511547
1548+static void xe_pt_abort_unbind(struct xe_vma *vma,
1549+ struct xe_vm_pgtable_update *entries,
1550+ u32 num_entries)
1551+{
1552+ int i, j;
1553+
1554+ xe_pt_commit_locks_assert(vma);
1555+
1556+ for (i = num_entries - 1; i >= 0; --i) {
1557+ struct xe_vm_pgtable_update *entry = &entries[i];
1558+ struct xe_pt *pt = entry->pt;
1559+ struct xe_pt_dir *pt_dir = as_xe_pt_dir(pt);
1560+
1561+ pt->num_live += entry->qwords;
1562+
1563+ if (!pt->level)
1564+ continue;
1565+
1566+ for (j = entry->ofs; j < entry->ofs + entry->qwords; j++)
1567+ pt_dir->children[j] =
1568+ entries[i].pt_entries[j - entry->ofs].pt ?
1569+ &entries[i].pt_entries[j - entry->ofs].pt->base : NULL;
1570+ }
1571+}
1572+
15521573 static void
1553−xe_pt_commit_unbind(struct xe_vma *vma,
1554− struct xe_vm_pgtable_update *entries, u32 num_entries,
1555− struct llist_head *deferred)
1574+xe_pt_commit_prepare_unbind(struct xe_vma *vma,
1575+ struct xe_vm_pgtable_update *entries,
1576+ u32 num_entries)
15561577 {
1557− u32 j;
1578+ int i, j;
15581579
15591580 xe_pt_commit_locks_assert(vma);
15601581
1561− for (j = 0; j < num_entries; ++j) {
1562− struct xe_vm_pgtable_update *entry = &entries[j];
1582+ for (i = 0; i < num_entries; ++i) {
1583+ struct xe_vm_pgtable_update *entry = &entries[i];
15631584 struct xe_pt *pt = entry->pt;
1585+ struct xe_pt_dir *pt_dir;
15641586
15651587 pt->num_live -= entry->qwords;
1566− if (pt->level) {
1567− struct xe_pt_dir *pt_dir = as_xe_pt_dir(pt);
1568− u32 i;
1588+ if (!pt->level)
1589+ continue;
1590+
1591+ pt_dir = as_xe_pt_dir(pt);
1592+ for (j = entry->ofs; j < entry->ofs + entry->qwords; j++) {
1593+ entry->pt_entries[j - entry->ofs].pt =
1594+ xe_pt_entry(pt_dir, j);
1595+ pt_dir->children[j] = NULL;
1596+ }
1597+ }
1598+}
1599+
1600+static void
1601+xe_pt_update_ops_rfence_interval(struct xe_vm_pgtable_update_ops *pt_update_ops,
1602+ struct xe_vma *vma)
1603+{
1604+ u32 current_op = pt_update_ops->current_op;
1605+ struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op];
1606+ int i, level = 0;
1607+ u64 start, last;
15691608
1570− for (i = entry->ofs; i < entry->ofs + entry->qwords;
1571− i++) {
1572− if (xe_pt_entry(pt_dir, i))
1573− xe_pt_destroy(xe_pt_entry(pt_dir, i),
1574− xe_vma_vm(vma)->flags, deferred);
1609+ for (i = 0; i < pt_op->num_entries; i++) {
1610+ const struct xe_vm_pgtable_update *entry = &pt_op->entries[i];
15751611
1576− pt_dir->children[i] = NULL;
1577− }
1612+ if (entry->pt->level > level)
1613+ level = entry->pt->level;
1614+ }
1615+
1616+ /* Greedy (non-optimal) calculation but simple */
1617+ start = ALIGN_DOWN(xe_vma_start(vma), 0x1ull << xe_pt_shift(level));
1618+ last = ALIGN(xe_vma_end(vma), 0x1ull << xe_pt_shift(level)) - 1;
1619+
1620+ if (start < pt_update_ops->start)
1621+ pt_update_ops->start = start;
1622+ if (last > pt_update_ops->last)
1623+ pt_update_ops->last = last;
1624+}
1625+
1626+static int vma_reserve_fences(struct xe_device *xe, struct xe_vma *vma)
1627+{
1628+ if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1629+ return dma_resv_reserve_fences(xe_vma_bo(vma)->ttm.base.resv,
1630+ xe->info.tile_count);
1631+
1632+ return 0;
1633+}
1634+
1635+static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile,
1636+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1637+ struct xe_vma *vma)
1638+{
1639+ u32 current_op = pt_update_ops->current_op;
1640+ struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op];
1641+ int err;
1642+
1643+ xe_bo_assert_held(xe_vma_bo(vma));
1644+
1645+ vm_dbg(&xe_vma_vm(vma)->xe->drm,
1646+ "Preparing bind, with range [%llx...%llx)\n",
1647+ xe_vma_start(vma), xe_vma_end(vma) - 1);
1648+
1649+ pt_op->vma = NULL;
1650+ pt_op->bind = true;
1651+ pt_op->rebind = BIT(tile->id) & vma->tile_present;
1652+
1653+ err = vma_reserve_fences(tile_to_xe(tile), vma);
1654+ if (err)
1655+ return err;
1656+
1657+ err = xe_pt_prepare_bind(tile, vma, pt_op->entries,
1658+ &pt_op->num_entries);
1659+ if (!err) {
1660+ xe_tile_assert(tile, pt_op->num_entries <=
1661+ ARRAY_SIZE(pt_op->entries));
1662+ xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries,
1663+ pt_op->num_entries, true);
1664+
1665+ xe_pt_update_ops_rfence_interval(pt_update_ops, vma);
1666+ ++pt_update_ops->current_op;
1667+ pt_update_ops->needs_userptr_lock |= xe_vma_is_userptr(vma);
1668+
1669+ /*
1670+ * If rebind, we have to invalidate TLB on !LR vms to invalidate
1671+ * cached PTEs point to freed memory. On LR vms this is done
1672+ * automatically when the context is re-enabled by the rebind worker,
1673+ * or in fault mode it was invalidated on PTE zapping.
1674+ *
1675+ * If !rebind, and scratch enabled VMs, there is a chance the scratch
1676+ * PTE is already cached in the TLB so it needs to be invalidated.
1677+ * On !LR VMs this is done in the ring ops preceding a batch, but on
1678+ * non-faulting LR, in particular on user-space batch buffer chaining,
1679+ * it needs to be done here.
1680+ */
1681+ if ((!pt_op->rebind && xe_vm_has_scratch(vm) &&
1682+ xe_vm_in_preempt_fence_mode(vm)))
1683+ pt_update_ops->needs_invalidation = true;
1684+ else if (pt_op->rebind && !xe_vm_in_lr_mode(vm))
1685+ /* We bump also if batch_invalidate_tlb is true */
1686+ vm->tlb_flush_seqno++;
1687+
1688+ vma->tile_staged |= BIT(tile->id);
1689+ pt_op->vma = vma;
1690+ xe_pt_commit_prepare_bind(vma, pt_op->entries,
1691+ pt_op->num_entries, pt_op->rebind);
1692+ } else {
1693+ xe_pt_cancel_bind(vma, pt_op->entries, pt_op->num_entries);
1694+ }
1695+
1696+ return err;
1697+}
1698+
1699+static int unbind_op_prepare(struct xe_tile *tile,
1700+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1701+ struct xe_vma *vma)
1702+{
1703+ u32 current_op = pt_update_ops->current_op;
1704+ struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op];
1705+ int err;
1706+
1707+ if (!((vma->tile_present | vma->tile_staged) & BIT(tile->id)))
1708+ return 0;
1709+
1710+ xe_bo_assert_held(xe_vma_bo(vma));
1711+
1712+ vm_dbg(&xe_vma_vm(vma)->xe->drm,
1713+ "Preparing unbind, with range [%llx...%llx)\n",
1714+ xe_vma_start(vma), xe_vma_end(vma) - 1);
1715+
1716+ /*
1717+ * Wait for invalidation to complete. Can corrupt internal page table
1718+ * state if an invalidation is running while preparing an unbind.
1719+ */
1720+ if (xe_vma_is_userptr(vma) && xe_vm_in_fault_mode(xe_vma_vm(vma)))
1721+ mmu_interval_read_begin(&to_userptr_vma(vma)->userptr.notifier);
1722+
1723+ pt_op->vma = vma;
1724+ pt_op->bind = false;
1725+ pt_op->rebind = false;
1726+
1727+ err = vma_reserve_fences(tile_to_xe(tile), vma);
1728+ if (err)
1729+ return err;
1730+
1731+ pt_op->num_entries = xe_pt_stage_unbind(tile, vma, pt_op->entries);
1732+
1733+ xe_vm_dbg_print_entries(tile_to_xe(tile), pt_op->entries,
1734+ pt_op->num_entries, false);
1735+ xe_pt_update_ops_rfence_interval(pt_update_ops, vma);
1736+ ++pt_update_ops->current_op;
1737+ pt_update_ops->needs_userptr_lock |= xe_vma_is_userptr(vma);
1738+ pt_update_ops->needs_invalidation = true;
1739+
1740+ xe_pt_commit_prepare_unbind(vma, pt_op->entries, pt_op->num_entries);
1741+
1742+ return 0;
1743+}
1744+
1745+static int op_prepare(struct xe_vm *vm,
1746+ struct xe_tile *tile,
1747+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1748+ struct xe_vma_op *op)
1749+{
1750+ int err = 0;
1751+
1752+ xe_vm_assert_held(vm);
1753+
1754+ switch (op->base.op) {
1755+ case DRM_GPUVA_OP_MAP:
1756+ if (!op->map.immediate && xe_vm_in_fault_mode(vm))
1757+ break;
1758+
1759+ err = bind_op_prepare(vm, tile, pt_update_ops, op->map.vma);
1760+ pt_update_ops->wait_vm_kernel = true;
1761+ break;
1762+ case DRM_GPUVA_OP_REMAP:
1763+ err = unbind_op_prepare(tile, pt_update_ops,
1764+ gpuva_to_vma(op->base.remap.unmap->va));
1765+
1766+ if (!err && op->remap.prev) {
1767+ err = bind_op_prepare(vm, tile, pt_update_ops,
1768+ op->remap.prev);
1769+ pt_update_ops->wait_vm_bookkeep = true;
15781770 }
1771+ if (!err && op->remap.next) {
1772+ err = bind_op_prepare(vm, tile, pt_update_ops,
1773+ op->remap.next);
1774+ pt_update_ops->wait_vm_bookkeep = true;
1775+ }
1776+ break;
1777+ case DRM_GPUVA_OP_UNMAP:
1778+ err = unbind_op_prepare(tile, pt_update_ops,
1779+ gpuva_to_vma(op->base.unmap.va));
1780+ break;
1781+ case DRM_GPUVA_OP_PREFETCH:
1782+ err = bind_op_prepare(vm, tile, pt_update_ops,
1783+ gpuva_to_vma(op->base.prefetch.va));
1784+ pt_update_ops->wait_vm_kernel = true;
1785+ break;
1786+ default:
1787+ drm_warn(&vm->xe->drm, "NOT POSSIBLE");
15791788 }
1789+
1790+ return err;
1791+}
1792+
1793+static void
1794+xe_pt_update_ops_init(struct xe_vm_pgtable_update_ops *pt_update_ops)
1795+{
1796+ init_llist_head(&pt_update_ops->deferred);
1797+ pt_update_ops->start = ~0x0ull;
1798+ pt_update_ops->last = 0x0ull;
15801799 }
15811800
1582−static const struct xe_migrate_pt_update_ops unbind_ops = {
1583− .populate = xe_migrate_clear_pgtable_callback,
1801+/**
1802+ * xe_pt_update_ops_prepare() - Prepare PT update operations
1803+ * @tile: Tile of PT update operations
1804+ * @vops: VMA operationa
1805+ *
1806+ * Prepare PT update operations which includes updating internal PT state,
1807+ * allocate memory for page tables, populate page table being pruned in, and
1808+ * create PT update operations for leaf insertion / removal.
1809+ *
1810+ * Return: 0 on success, negative error code on error.
1811+ */
1812+int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)
1813+{
1814+ struct xe_vm_pgtable_update_ops *pt_update_ops =
1815+ &vops->pt_update_ops[tile->id];
1816+ struct xe_vma_op *op;
1817+ int err;
1818+
1819+ lockdep_assert_held(&vops->vm->lock);
1820+ xe_vm_assert_held(vops->vm);
1821+
1822+ xe_pt_update_ops_init(pt_update_ops);
1823+
1824+ err = dma_resv_reserve_fences(xe_vm_resv(vops->vm),
1825+ tile_to_xe(tile)->info.tile_count);
1826+ if (err)
1827+ return err;
1828+
1829+ list_for_each_entry(op, &vops->list, link) {
1830+ err = op_prepare(vops->vm, tile, pt_update_ops, op);
1831+
1832+ if (err)
1833+ return err;
1834+ }
1835+
1836+ xe_tile_assert(tile, pt_update_ops->current_op <=
1837+ pt_update_ops->num_ops);
1838+
1839+#ifdef TEST_VM_OPS_ERROR
1840+ if (vops->inject_error &&
1841+ vops->vm->xe->vm_inject_error_position == FORCE_OP_ERROR_PREPARE)
1842+ return -ENOSPC;
1843+#endif
1844+
1845+ return 0;
1846+}
1847+
1848+static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
1849+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1850+ struct xe_vma *vma, struct dma_fence *fence)
1851+{
1852+ if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1853+ dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence,
1854+ pt_update_ops->wait_vm_bookkeep ?
1855+ DMA_RESV_USAGE_KERNEL :
1856+ DMA_RESV_USAGE_BOOKKEEP);
1857+ vma->tile_present |= BIT(tile->id);
1858+ vma->tile_staged &= ~BIT(tile->id);
1859+ if (xe_vma_is_userptr(vma)) {
1860+ lockdep_assert_held_read(&vm->userptr.notifier_lock);
1861+ to_userptr_vma(vma)->userptr.initial_bind = true;
1862+ }
1863+
1864+ /*
1865+ * Kick rebind worker if this bind triggers preempt fences and not in
1866+ * the rebind worker
1867+ */
1868+ if (pt_update_ops->wait_vm_bookkeep &&
1869+ xe_vm_in_preempt_fence_mode(vm) &&
1870+ !current->mm)
1871+ xe_vm_queue_rebind_worker(vm);
1872+}
1873+
1874+static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile,
1875+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1876+ struct xe_vma *vma, struct dma_fence *fence)
1877+{
1878+ if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1879+ dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence,
1880+ pt_update_ops->wait_vm_bookkeep ?
1881+ DMA_RESV_USAGE_KERNEL :
1882+ DMA_RESV_USAGE_BOOKKEEP);
1883+ vma->tile_present &= ~BIT(tile->id);
1884+ if (!vma->tile_present) {
1885+ list_del_init(&vma->combined_links.rebind);
1886+ if (xe_vma_is_userptr(vma)) {
1887+ lockdep_assert_held_read(&vm->userptr.notifier_lock);
1888+
1889+ spin_lock(&vm->userptr.invalidated_lock);
1890+ list_del_init(&to_userptr_vma(vma)->userptr.invalidate_link);
1891+ spin_unlock(&vm->userptr.invalidated_lock);
1892+ }
1893+ }
1894+}
1895+
1896+static void op_commit(struct xe_vm *vm,
1897+ struct xe_tile *tile,
1898+ struct xe_vm_pgtable_update_ops *pt_update_ops,
1899+ struct xe_vma_op *op, struct dma_fence *fence)
1900+{
1901+ xe_vm_assert_held(vm);
1902+
1903+ switch (op->base.op) {
1904+ case DRM_GPUVA_OP_MAP:
1905+ if (!op->map.immediate && xe_vm_in_fault_mode(vm))
1906+ break;
1907+
1908+ bind_op_commit(vm, tile, pt_update_ops, op->map.vma, fence);
1909+ break;
1910+ case DRM_GPUVA_OP_REMAP:
1911+ unbind_op_commit(vm, tile, pt_update_ops,
1912+ gpuva_to_vma(op->base.remap.unmap->va), fence);
1913+
1914+ if (op->remap.prev)
1915+ bind_op_commit(vm, tile, pt_update_ops, op->remap.prev,
1916+ fence);
1917+ if (op->remap.next)
1918+ bind_op_commit(vm, tile, pt_update_ops, op->remap.next,
1919+ fence);
1920+ break;
1921+ case DRM_GPUVA_OP_UNMAP:
1922+ unbind_op_commit(vm, tile, pt_update_ops,
1923+ gpuva_to_vma(op->base.unmap.va), fence);
1924+ break;
1925+ case DRM_GPUVA_OP_PREFETCH:
1926+ bind_op_commit(vm, tile, pt_update_ops,
1927+ gpuva_to_vma(op->base.prefetch.va), fence);
1928+ break;
1929+ default:
1930+ drm_warn(&vm->xe->drm, "NOT POSSIBLE");
1931+ }
1932+}
1933+
1934+static const struct xe_migrate_pt_update_ops migrate_ops = {
1935+ .populate = xe_vm_populate_pgtable,
1936+ .clear = xe_migrate_clear_pgtable_callback,
15841937 .pre_commit = xe_pt_pre_commit,
15851938 };
15861939
1587−static const struct xe_migrate_pt_update_ops userptr_unbind_ops = {
1588− .populate = xe_migrate_clear_pgtable_callback,
1940+static const struct xe_migrate_pt_update_ops userptr_migrate_ops = {
1941+ .populate = xe_vm_populate_pgtable,
1942+ .clear = xe_migrate_clear_pgtable_callback,
15891943 .pre_commit = xe_pt_userptr_pre_commit,
15901944 };
15911945
15921946 /**
1593− * __xe_pt_unbind_vma() - Disconnect and free a page-table tree for the vma
1594− * address range.
1595− * @tile: The tile to unbind for.
1596− * @vma: The vma to unbind.
1597− * @q: The exec_queue with which to do pipelined page-table updates.
1598− * @syncs: Entries to sync on before disconnecting the tree to be destroyed.
1599− * @num_syncs: Number of @sync entries.
1947+ * xe_pt_update_ops_run() - Run PT update operations
1948+ * @tile: Tile of PT update operations
1949+ * @vops: VMA operationa
16001950 *
1601− * This function builds a the xe_vm_pgtable_update entries abstracting the
1602− * operations needed to detach the page-table tree to be destroyed from the
1603− * man vm tree.
1604− * It then takes the relevant locks and submits the operations for
1605− * pipelined detachment of the gpu page-table from the vm main tree,
1606− * (which can be done either by the cpu and the GPU), Finally it frees the
1607− * detached page-table tree.
1951+ * Run PT update operations which includes committing internal PT state changes,
1952+ * creating job for PT update operations for leaf insertion / removal, and
1953+ * installing job fence in various places.
16081954 *
1609− * Return: A valid dma-fence representing the pipelined detachment operation
1610− * on success, an error pointer on error.
1955+ * Return: fence on success, negative ERR_PTR on error.
16111956 */
16121957 struct dma_fence *
1613−__xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queue *q,
1614− struct xe_sync_entry *syncs, u32 num_syncs)
1958+xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
16151959 {
1616− struct xe_vm_pgtable_update entries[XE_VM_MAX_LEVEL * 2 + 1];
1617− struct xe_pt_migrate_pt_update unbind_pt_update = {
1618− .base = {
1619− .ops = xe_vma_is_userptr(vma) ? &userptr_unbind_ops :
1620− &unbind_ops,
1621− .vma = vma,
1622− .tile_id = tile->id,
1623− },
1624− };
1625− struct xe_vm *vm = xe_vma_vm(vma);
1626− u32 num_entries;
1627− struct dma_fence *fence = NULL;
1628− struct invalidation_fence *ifence;
1960+ struct xe_vm *vm = vops->vm;
1961+ struct xe_vm_pgtable_update_ops *pt_update_ops =
1962+ &vops->pt_update_ops[tile->id];
1963+ struct dma_fence *fence;
1964+ struct invalidation_fence *ifence = NULL;
16291965 struct xe_range_fence *rfence;
1630− int err;
1631−
1632− LLIST_HEAD(deferred);
1966+ struct xe_vma_op *op;
1967+ int err = 0, i;
1968+ struct xe_migrate_pt_update update = {
1969+ .ops = pt_update_ops->needs_userptr_lock ?
1970+ &userptr_migrate_ops :
1971+ &migrate_ops,
1972+ .vops = vops,
1973+ .tile_id = tile->id,
1974+ };
16331975
1634− xe_bo_assert_held(xe_vma_bo(vma));
1976+ lockdep_assert_held(&vm->lock);
16351977 xe_vm_assert_held(vm);
16361978
1637− vm_dbg(&xe_vma_vm(vma)->xe->drm,
1638− "Preparing unbind, with range [%llx...%llx) engine %p.\n",
1639− xe_vma_start(vma), xe_vma_end(vma), q);
1640−
1641− num_entries = xe_pt_stage_unbind(tile, vma, entries);
1642− xe_tile_assert(tile, num_entries <= ARRAY_SIZE(entries));
1979+ if (!pt_update_ops->current_op) {
1980+ xe_tile_assert(tile, xe_vm_in_fault_mode(vm));
16431981
1644− xe_vm_dbg_print_entries(tile_to_xe(tile), entries, num_entries);
1645− xe_pt_calc_rfence_interval(vma, &unbind_pt_update, entries,
1646− num_entries);
1982+ return dma_fence_get_stub();
1983+ }
16471984
1648− err = dma_resv_reserve_fences(xe_vm_resv(vm), 1);
1649− if (!err && !xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1650− err = dma_resv_reserve_fences(xe_vma_bo(vma)->ttm.base.resv, 1);
1651− if (err)
1652− return ERR_PTR(err);
1985+#ifdef TEST_VM_OPS_ERROR
1986+ if (vops->inject_error &&
1987+ vm->xe->vm_inject_error_position == FORCE_OP_ERROR_RUN)
1988+ return ERR_PTR(-ENOSPC);
1989+#endif
16531990
1654− ifence = kzalloc(sizeof(*ifence), GFP_KERNEL);
1655− if (!ifence)
1656− return ERR_PTR(-ENOMEM);
1991+ if (pt_update_ops->needs_invalidation) {
1992+ ifence = kzalloc(sizeof(*ifence), GFP_KERNEL);
1993+ if (!ifence) {
1994+ err = -ENOMEM;
1995+ goto kill_vm_tile1;
1996+ }
1997+ }
16571998
16581999 rfence = kzalloc(sizeof(*rfence), GFP_KERNEL);
16592000 if (!rfence) {
1660− kfree(ifence);
1661− return ERR_PTR(-ENOMEM);
2001+ err = -ENOMEM;
2002+ goto free_ifence;
16622003 }
16632004
1664− /*
1665− * Even if we were already evicted and unbind to destroy, we need to
1666− * clear again here. The eviction may have updated pagetables at a
1667− * lower level, because it needs to be more conservative.
1668− */
1669− fence = xe_migrate_update_pgtables(tile->migrate,
1670− vm, NULL, q ? q :
1671− vm->q[tile->id],
1672− entries, num_entries,
1673− syncs, num_syncs,
1674− &unbind_pt_update.base);
1675− if (!IS_ERR(fence)) {
1676− int err;
1677−
1678− err = xe_range_fence_insert(&vm->rftree[tile->id], rfence,
1679− &xe_range_fence_kfree_ops,
1680− unbind_pt_update.base.start,
1681− unbind_pt_update.base.last, fence);
1682− if (err)
1683− dma_fence_wait(fence, false);
2005+ fence = xe_migrate_update_pgtables(tile->migrate, &update);
2006+ if (IS_ERR(fence)) {
2007+ err = PTR_ERR(fence);
2008+ goto free_rfence;
2009+ }
16842010
1685− /* TLB invalidation must be done before signaling unbind */
1686− err = invalidation_fence_init(tile->primary_gt, ifence, fence,
1687− xe_vma_start(vma),
1688− xe_vma_end(vma),
1689− xe_vma_vm(vma)->usm.asid);
1690− if (err) {
1691− dma_fence_put(fence);
1692− kfree(ifence);
1693− return ERR_PTR(err);
1694− }
1695− fence = &ifence->base.base;
2011+ /* Point of no return - VM killed if failure after this */
2012+ for (i = 0; i < pt_update_ops->current_op; ++i) {
2013+ struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i];
16962014
1697− /* add shared fence now for pagetable delayed destroy */
1698− dma_resv_add_fence(xe_vm_resv(vm), fence,
1699− DMA_RESV_USAGE_BOOKKEEP);
2015+ xe_pt_commit(pt_op->vma, pt_op->entries,
2016+ pt_op->num_entries, &pt_update_ops->deferred);
2017+ pt_op->vma = NULL; /* skip in xe_pt_update_ops_abort */
2018+ }
17002019
1701− /* This fence will be installed by caller when doing eviction */
1702− if (!xe_vma_has_no_bo(vma) && !xe_vma_bo(vma)->vm)
1703− dma_resv_add_fence(xe_vma_bo(vma)->ttm.base.resv, fence,
1704− DMA_RESV_USAGE_BOOKKEEP);
1705− xe_pt_commit_unbind(vma, entries, num_entries,
1706− unbind_pt_update.locked ? &deferred : NULL);
1707− vma->tile_present &= ~BIT(tile->id);
1708− } else {
1709− kfree(rfence);
1710− kfree(ifence);
2020+ if (xe_range_fence_insert(&vm->rftree[tile->id], rfence,
2021+ &xe_range_fence_kfree_ops,
2022+ pt_update_ops->start,
2023+ pt_update_ops->last, fence))
2024+ dma_fence_wait(fence, false);
2025+
2026+ /* tlb invalidation must be done before signaling rebind */
2027+ if (ifence) {
2028+ invalidation_fence_init(tile->primary_gt, ifence, fence,
2029+ pt_update_ops->start,
2030+ pt_update_ops->last, vm->usm.asid);
2031+ fence = &ifence->base.base;
17112032 }
17122033
1713− if (!vma->tile_present)
1714− list_del_init(&vma->combined_links.rebind);
2034+ dma_resv_add_fence(xe_vm_resv(vm), fence,
2035+ pt_update_ops->wait_vm_bookkeep ?
2036+ DMA_RESV_USAGE_KERNEL :
2037+ DMA_RESV_USAGE_BOOKKEEP);
17152038
1716− if (unbind_pt_update.locked) {
1717− xe_tile_assert(tile, xe_vma_is_userptr(vma));
2039+ list_for_each_entry(op, &vops->list, link)
2040+ op_commit(vops->vm, tile, pt_update_ops, op, fence);
17182041
1719− if (!vma->tile_present) {
1720− spin_lock(&vm->userptr.invalidated_lock);
1721− list_del_init(&to_userptr_vma(vma)->userptr.invalidate_link);
1722− spin_unlock(&vm->userptr.invalidated_lock);
1723− }
2042+ if (pt_update_ops->needs_userptr_lock)
17242043 up_read(&vm->userptr.notifier_lock);
1725− xe_bo_put_commit(&deferred);
1726− }
17272044
17282045 return fence;
2046+
2047+free_rfence:
2048+ kfree(rfence);
2049+free_ifence:
2050+ kfree(ifence);
2051+kill_vm_tile1:
2052+ if (err != -EAGAIN && tile->id)
2053+ xe_vm_kill(vops->vm, false);
2054+
2055+ return ERR_PTR(err);
2056+}
2057+
2058+/**
2059+ * xe_pt_update_ops_fini() - Finish PT update operations
2060+ * @tile: Tile of PT update operations
2061+ * @vops: VMA operations
2062+ *
2063+ * Finish PT update operations by committing to destroy page table memory
2064+ */
2065+void xe_pt_update_ops_fini(struct xe_tile *tile, struct xe_vma_ops *vops)
2066+{
2067+ struct xe_vm_pgtable_update_ops *pt_update_ops =
2068+ &vops->pt_update_ops[tile->id];
2069+ int i;
2070+
2071+ lockdep_assert_held(&vops->vm->lock);
2072+ xe_vm_assert_held(vops->vm);
2073+
2074+ for (i = 0; i < pt_update_ops->current_op; ++i) {
2075+ struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i];
2076+
2077+ xe_pt_free_bind(pt_op->entries, pt_op->num_entries);
2078+ }
2079+ xe_bo_put_commit(&vops->pt_update_ops[tile->id].deferred);
2080+}
2081+
2082+/**
2083+ * xe_pt_update_ops_abort() - Abort PT update operations
2084+ * @tile: Tile of PT update operations
2085+ * @vops: VMA operationa
2086+ *
2087+ * Abort PT update operations by unwinding internal PT state
2088+ */
2089+void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops)
2090+{
2091+ struct xe_vm_pgtable_update_ops *pt_update_ops =
2092+ &vops->pt_update_ops[tile->id];
2093+ int i;
2094+
2095+ lockdep_assert_held(&vops->vm->lock);
2096+ xe_vm_assert_held(vops->vm);
2097+
2098+ for (i = pt_update_ops->num_ops - 1; i >= 0; --i) {
2099+ struct xe_vm_pgtable_update_op *pt_op =
2100+ &pt_update_ops->ops[i];
2101+
2102+ if (!pt_op->vma || i >= pt_update_ops->current_op)
2103+ continue;
2104+
2105+ if (pt_op->bind)
2106+ xe_pt_abort_bind(pt_op->vma, pt_op->entries,
2107+ pt_op->num_entries,
2108+ pt_op->rebind);
2109+ else
2110+ xe_pt_abort_unbind(pt_op->vma, pt_op->entries,
2111+ pt_op->num_entries);
2112+ }
2113+
2114+ xe_bo_put_commit(&vops->pt_update_ops[tile->id].deferred);
17292115 }
drivers/gpu/drm/xe/xe_pt.h+6−8
@@ -17,6 +17,7 @@ struct xe_sync_entry; · 16 unmodified lines
1717 struct xe_tile;
1818 struct xe_vm;
1919 struct xe_vma;
20+struct xe_vma_ops;
2021
2122 /* Largest huge pte is currently 1GiB. May become device dependent. */
2223 #define MAX_HUGEPTE_LEVEL 2
@@ -34,14 +35,11 @@ void xe_pt_populate_empty(struct xe_tile *tile, struct xe_vm *vm, · 11 unmodified lines
3435
3536 void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred);
3637
37−struct dma_fence *
38−__xe_pt_bind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queue *q,
39− struct xe_sync_entry *syncs, u32 num_syncs,
40− bool rebind);
41−
42−struct dma_fence *
43−__xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queue *q,
44− struct xe_sync_entry *syncs, u32 num_syncs);
38+int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops);
39+struct dma_fence *xe_pt_update_ops_run(struct xe_tile *tile,
40+ struct xe_vma_ops *vops);
41+void xe_pt_update_ops_fini(struct xe_tile *tile, struct xe_vma_ops *vops);
42+void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops);
4543
4644 bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma);
4745
drivers/gpu/drm/xe/xe_pt_types.h+48−0
@@ -74,4 +74,52 @@ struct xe_vm_pgtable_update { · 73 unmodified lines
7474 u32 flags;
7575 };
7676
77+/** struct xe_vm_pgtable_update_op - Page table update operation */
78+struct xe_vm_pgtable_update_op {
79+ /** @entries: entries to update for this operation */
80+ struct xe_vm_pgtable_update entries[XE_VM_MAX_LEVEL * 2 + 1];
81+ /** @vma: VMA for operation, operation not valid if NULL */
82+ struct xe_vma *vma;
83+ /** @num_entries: number of entries for this update operation */
84+ u32 num_entries;
85+ /** @bind: is a bind */
86+ bool bind;
87+ /** @rebind: is a rebind */
88+ bool rebind;
89+};
90+
91+/** struct xe_vm_pgtable_update_ops: page table update operations */
92+struct xe_vm_pgtable_update_ops {
93+ /** @ops: operations */
94+ struct xe_vm_pgtable_update_op *ops;
95+ /** @deferred: deferred list to destroy PT entries */
96+ struct llist_head deferred;
97+ /** @q: exec queue for PT operations */
98+ struct xe_exec_queue *q;
99+ /** @start: start address of ops */
100+ u64 start;
101+ /** @last: last address of ops */
102+ u64 last;
103+ /** @num_ops: number of operations */
104+ u32 num_ops;
105+ /** @current_op: current operations */
106+ u32 current_op;
107+ /** @needs_userptr_lock: Needs userptr lock */
108+ bool needs_userptr_lock;
109+ /** @needs_invalidation: Needs invalidation */
110+ bool needs_invalidation;
111+ /**
112+ * @wait_vm_bookkeep: PT operations need to wait until VM is idle
113+ * (bookkeep dma-resv slots are idle) and stage all future VM activity
114+ * behind these operations (install PT operations into VM kernel
115+ * dma-resv slot).
116+ */
117+ bool wait_vm_bookkeep;
118+ /**
119+ * @wait_vm_kernel: PT operations need to wait until VM kernel dma-resv
120+ * slots are idle.
121+ */
122+ bool wait_vm_kernel;
123+};
124+
77125 #endif
drivers/gpu/drm/xe/xe_query.c+3−1
@@ -518,7 +518,9 @@ static int query_gt_topology(struct xe_device *xe, · 517 unmodified lines
518518 if (err)
519519 return err;
520520
521− topo.type = DRM_XE_TOPO_EU_PER_DSS;
521+ topo.type = gt->fuse_topo.eu_type == XE_GT_EU_TYPE_SIMD16 ?
522+ DRM_XE_TOPO_SIMD16_EU_PER_DSS :
523+ DRM_XE_TOPO_EU_PER_DSS;
522524 err = copy_mask(&query_ptr, &topo,
523525 gt->fuse_topo.eu_mask_per_dss,
524526 sizeof(gt->fuse_topo.eu_mask_per_dss));
drivers/gpu/drm/xe/xe_rtp.c+12−30
@@ -217,21 +217,19 @@ void xe_rtp_process_ctx_enable_active_tracking(struct xe_rtp_process_ctx *ctx, · 216 unmodified lines
217217 ctx->active_entries = active_entries;
218218 ctx->n_entries = n_entries;
219219 }
220+EXPORT_SYMBOL_IF_KUNIT(xe_rtp_process_ctx_enable_active_tracking);
220221
221222 static void rtp_mark_active(struct xe_device *xe,
222223 struct xe_rtp_process_ctx *ctx,
223− unsigned int first, unsigned int last)
224+ unsigned int idx)
224225 {
225226 if (!ctx->active_entries)
226227 return;
227228
228− if (drm_WARN_ON(&xe->drm, last > ctx->n_entries))
229+ if (drm_WARN_ON(&xe->drm, idx >= ctx->n_entries))
229230 return;
230231
231− if (first == last)
232− bitmap_set(ctx->active_entries, first, 1);
233− else
234− bitmap_set(ctx->active_entries, first, last - first + 2);
232+ bitmap_set(ctx->active_entries, idx, 1);
235233 }
236234
237235 /**
@@ -276,8 +274,7 @@ void xe_rtp_process_to_sr(struct xe_rtp_process_ctx *ctx, · 38 unmodified lines
276274 }
277275
278276 if (match)
279− rtp_mark_active(xe, ctx, entry - entries,
280− entry - entries);
277+ rtp_mark_active(xe, ctx, entry - entries);
281278 }
282279 }
283280 EXPORT_SYMBOL_IF_KUNIT(xe_rtp_process_to_sr);
@@ -288,44 +285,29 @@ EXPORT_SYMBOL_IF_KUNIT(xe_rtp_process_to_sr); · 4 unmodified lines
288285 * @entries: Table with RTP definitions
289286 *
290287 * Walk the table pointed by @entries (with an empty sentinel), executing the
291− * rules. A few differences from xe_rtp_process_to_sr():
292− *
293− * 1. There is no action associated with each entry since this uses
294− * struct xe_rtp_entry. Its main use is for marking active workarounds via
295− * xe_rtp_process_ctx_enable_active_tracking().
296− * 2. There is support for OR operations by having entries with no name.
288+ * rules. One difference from xe_rtp_process_to_sr(): there is no action
289+ * associated with each entry since this uses struct xe_rtp_entry. Its main use
290+ * is for marking active workarounds via
291+ * xe_rtp_process_ctx_enable_active_tracking().
297292 */
298293 void xe_rtp_process(struct xe_rtp_process_ctx *ctx,
299294 const struct xe_rtp_entry *entries)
300295 {
301− const struct xe_rtp_entry *entry, *first_entry;
296+ const struct xe_rtp_entry *entry;
302297 struct xe_hw_engine *hwe;
303298 struct xe_gt *gt;
304299 struct xe_device *xe;
305300
306301 rtp_get_context(ctx, &hwe, &gt, &xe);
307302
308− first_entry = entries;
309− if (drm_WARN_ON(&xe->drm, !first_entry->name))
310− return;
311−
312303 for (entry = entries; entry && entry->rules; entry++) {
313− if (entry->name)
314− first_entry = entry;
315−
316304 if (!rule_matches(xe, gt, hwe, entry->rules, entry->n_rules))
317305 continue;
318306
319− /* Fast-forward entry, eliminating the OR'ed entries */
320− for (entry++; entry && entry->rules; entry++)
321− if (entry->name)
322− break;
323− entry--;
324−
325− rtp_mark_active(xe, ctx, first_entry - entries,
326− entry - entries);
307+ rtp_mark_active(xe, ctx, entry - entries);
327308 }
328309 }
310+EXPORT_SYMBOL_IF_KUNIT(xe_rtp_process);
329311
330312 bool xe_rtp_match_even_instance(const struct xe_gt *gt,
331313 const struct xe_hw_engine *hwe)
drivers/gpu/drm/xe/xe_rtp.h+2−2
@@ -374,7 +374,7 @@ struct xe_reg_sr; · 373 unmodified lines
374374 * XE_RTP_RULES - Helper to set multiple rules to a struct xe_rtp_entry_sr entry
375375 * @...: Rules
376376 *
377− * At least one rule is needed and up to 6 are supported. Multiple rules are
377+ * At least one rule is needed and up to 12 are supported. Multiple rules are
378378 * AND'ed together, i.e. all the rules must evaluate to true for the entry to
379379 * be processed. See XE_RTP_MATCH_* for the possible match rules. Example:
380380 *
@@ -399,7 +399,7 @@ struct xe_reg_sr; · 18 unmodified lines
399399 * XE_RTP_ACTIONS - Helper to set multiple actions to a struct xe_rtp_entry_sr
400400 * @...: Actions to be taken
401401 *
402− * At least one action is needed and up to 6 are supported. See XE_RTP_ACTION_*
402+ * At least one action is needed and up to 12 are supported. See XE_RTP_ACTION_*
403403 * for the possible actions. Example:
404404 *
405405 * .. code-block:: c
drivers/gpu/drm/xe/xe_rtp_helpers.h+6−0
@@ -60,6 +60,12 @@ · 59 unmodified lines
6060 #define XE_RTP_PASTE_4(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_3(prefix_, sep_, _XE_TUPLE_TAIL args_)
6161 #define XE_RTP_PASTE_5(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_4(prefix_, sep_, _XE_TUPLE_TAIL args_)
6262 #define XE_RTP_PASTE_6(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_5(prefix_, sep_, _XE_TUPLE_TAIL args_)
63+#define XE_RTP_PASTE_7(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_6(prefix_, sep_, _XE_TUPLE_TAIL args_)
64+#define XE_RTP_PASTE_8(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_7(prefix_, sep_, _XE_TUPLE_TAIL args_)
65+#define XE_RTP_PASTE_9(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_8(prefix_, sep_, _XE_TUPLE_TAIL args_)
66+#define XE_RTP_PASTE_10(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_9(prefix_, sep_, _XE_TUPLE_TAIL args_)
67+#define XE_RTP_PASTE_11(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_10(prefix_, sep_, _XE_TUPLE_TAIL args_)
68+#define XE_RTP_PASTE_12(prefix_, sep_, args_) _XE_RTP_CONCAT(prefix_, FIRST_ARG args_) __XE_RTP_PASTE_SEP_ ## sep_ XE_RTP_PASTE_11(prefix_, sep_, _XE_TUPLE_TAIL args_)
6369
6470 /*
6571 * XE_RTP_DROP_CAST - Drop cast to convert a compound statement to a initializer
drivers/gpu/drm/xe/xe_sa.c+7−0
@@ -84,6 +84,13 @@ struct xe_sa_manager *xe_sa_bo_manager_init(struct xe_tile *tile, u32 size, u32 · 83 unmodified lines
8484 struct drm_suballoc *xe_sa_bo_new(struct xe_sa_manager *sa_manager,
8585 unsigned int size)
8686 {
87+ /*
88+ * BB to large, return -ENOBUFS indicating user should split
89+ * array of binds into smaller chunks.
90+ */
91+ if (size > sa_manager->base.size)
92+ return ERR_PTR(-ENOBUFS);
93+
8794 return drm_suballoc_new(&sa_manager->base, size, GFP_KERNEL, true, 0);
8895 }
8996
drivers/gpu/drm/xe/xe_sriov.c+1−1
@@ -5,7 +5,7 @@ · 4 unmodified lines
55
66 #include <drm/drm_managed.h>
77
8−#include "regs/xe_sriov_regs.h"
8+#include "regs/xe_regs.h"
99
1010 #include "xe_assert.h"
1111 #include "xe_device.h"
drivers/gpu/drm/xe/xe_sync.c+8−12
@@ -53,14 +53,18 @@ static struct xe_user_fence *user_fence_create(struct xe_device *xe, u64 addr, · 52 unmodified lines
5353 u64 value)
5454 {
5555 struct xe_user_fence *ufence;
56+ u64 __user *ptr = u64_to_user_ptr(addr);
57+
58+ if (!access_ok(ptr, sizeof(ptr)))
59+ return ERR_PTR(-EFAULT);
5660
5761 ufence = kmalloc(sizeof(*ufence), GFP_KERNEL);
5862 if (!ufence)
59− return NULL;
63+ return ERR_PTR(-ENOMEM);
6064
6165 ufence->xe = xe;
6266 kref_init(&ufence->refcount);
63− ufence->addr = u64_to_user_ptr(addr);
67+ ufence->addr = ptr;
6468 ufence->value = value;
6569 ufence->mm = current->mm;
6670 mmgrab(ufence->mm);
@@ -183,8 +187,8 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef, · 116 unmodified lines
183187 } else {
184188 sync->ufence = user_fence_create(xe, sync_in.addr,
185189 sync_in.timeline_value);
186− if (XE_IOCTL_DBG(xe, !sync->ufence))
187− return -ENOMEM;
190+ if (XE_IOCTL_DBG(xe, IS_ERR(sync->ufence)))
191+ return PTR_ERR(sync->ufence);
188192 }
189193
190194 break;
@@ -200,14 +204,6 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef, · 9 unmodified lines
200204 return 0;
201205 }
202206
203−int xe_sync_entry_wait(struct xe_sync_entry *sync)
204−{
205− if (sync->fence)
206− dma_fence_wait(sync->fence, true);
207−
208− return 0;
209−}
210−
211207 int xe_sync_entry_add_deps(struct xe_sync_entry *sync, struct xe_sched_job *job)
212208 {
213209 int err;
drivers/gpu/drm/xe/xe_sync.h+0−1
@@ -22,7 +22,6 @@ int xe_sync_entry_parse(struct xe_device *xe, struct xe_file *xef, · 21 unmodified lines
2222 struct xe_sync_entry *sync,
2323 struct drm_xe_sync __user *sync_user,
2424 unsigned int flags);
25−int xe_sync_entry_wait(struct xe_sync_entry *sync);
2625 int xe_sync_entry_add_deps(struct xe_sync_entry *sync,
2726 struct xe_sched_job *job);
2827 void xe_sync_entry_signal(struct xe_sync_entry *sync,
drivers/gpu/drm/xe/xe_trace.h+52−0
@@ -369,6 +369,58 @@ TRACE_EVENT(xe_reg_rw, · 368 unmodified lines
369369 (u32)(__entry->val >> 32))
370370 );
371371
372+DECLARE_EVENT_CLASS(xe_pm_runtime,
373+ TP_PROTO(struct xe_device *xe, void *caller),
374+ TP_ARGS(xe, caller),
375+
376+ TP_STRUCT__entry(
377+ __string(dev, __dev_name_xe(xe))
378+ __field(void *, caller)
379+ ),
380+
381+ TP_fast_assign(
382+ __assign_str(dev);
383+ __entry->caller = caller;
384+ ),
385+
386+ TP_printk("dev=%s caller_function=%pS", __get_str(dev), __entry->caller)
387+);
388+
389+DEFINE_EVENT(xe_pm_runtime, xe_pm_runtime_get,
390+ TP_PROTO(struct xe_device *xe, void *caller),
391+ TP_ARGS(xe, caller)
392+);
393+
394+DEFINE_EVENT(xe_pm_runtime, xe_pm_runtime_put,
395+ TP_PROTO(struct xe_device *xe, void *caller),
396+ TP_ARGS(xe, caller)
397+);
398+
399+DEFINE_EVENT(xe_pm_runtime, xe_pm_resume,
400+ TP_PROTO(struct xe_device *xe, void *caller),
401+ TP_ARGS(xe, caller)
402+);
403+
404+DEFINE_EVENT(xe_pm_runtime, xe_pm_suspend,
405+ TP_PROTO(struct xe_device *xe, void *caller),
406+ TP_ARGS(xe, caller)
407+);
408+
409+DEFINE_EVENT(xe_pm_runtime, xe_pm_runtime_resume,
410+ TP_PROTO(struct xe_device *xe, void *caller),
411+ TP_ARGS(xe, caller)
412+);
413+
414+DEFINE_EVENT(xe_pm_runtime, xe_pm_runtime_suspend,
415+ TP_PROTO(struct xe_device *xe, void *caller),
416+ TP_ARGS(xe, caller)
417+);
418+
419+DEFINE_EVENT(xe_pm_runtime, xe_pm_runtime_get_ioctl,
420+ TP_PROTO(struct xe_device *xe, void *caller),
421+ TP_ARGS(xe, caller)
422+);
423+
372424 #endif
373425
374426 /* This part must be outside protection */
drivers/gpu/drm/xe/xe_trace_bo.h+5−5
@@ -117,11 +117,6 @@ DEFINE_EVENT(xe_vma, xe_vma_acc, · 116 unmodified lines
117117 TP_ARGS(vma)
118118 );
119119
120−DEFINE_EVENT(xe_vma, xe_vma_fail,
121− TP_PROTO(struct xe_vma *vma),
122− TP_ARGS(vma)
123−);
124−
125120 DEFINE_EVENT(xe_vma, xe_vma_bind,
126121 TP_PROTO(struct xe_vma *vma),
127122 TP_ARGS(vma)
@@ -237,6 +232,11 @@ DEFINE_EVENT(xe_vm, xe_vm_rebind_worker_exit, · 109 unmodified lines
237232 TP_ARGS(vm)
238233 );
239234
235+DEFINE_EVENT(xe_vm, xe_vm_ops_fail,
236+ TP_PROTO(struct xe_vm *vm),
237+ TP_ARGS(vm)
238+);
239+
240240 #endif
241241
242242 /* This part must be outside protection */
drivers/gpu/drm/xe/xe_tuning.c+8−0
@@ -93,6 +93,14 @@ static const struct xe_rtp_entry_sr lrc_tunings[] = { · 92 unmodified lines
9393 REG_FIELD_PREP(L3_PWM_TIMER_INIT_VAL_MASK, 0x7f)))
9494 },
9595
96+ /* Xe2_HPG */
97+
98+ { XE_RTP_NAME("Tuning: vs hit max value"),
99+ XE_RTP_RULES(GRAPHICS_VERSION(2001), ENGINE_CLASS(RENDER)),
100+ XE_RTP_ACTIONS(FIELD_SET(FF_MODE, VS_HIT_MAX_VALUE_MASK,
101+ REG_FIELD_PREP(VS_HIT_MAX_VALUE_MASK, 0x3f)))
102+ },
103+
96104 {}
97105 };
98106
drivers/gpu/drm/xe/xe_uc_fw.c+3−0
@@ -116,6 +116,8 @@ struct fw_blobs_by_type { · 115 unmodified lines
116116 fw_def(TIGERLAKE, major_ver(i915, guc, tgl, 70, 19, 2))
117117
118118 #define XE_HUC_FIRMWARE_DEFS(fw_def, mmp_ver, no_ver) \
119+ fw_def(BATTLEMAGE, no_ver(xe, huc, bmg)) \
120+ fw_def(LUNARLAKE, no_ver(xe, huc, lnl)) \
119121 fw_def(METEORLAKE, no_ver(i915, huc_gsc, mtl)) \
120122 fw_def(DG1, no_ver(i915, huc, dg1)) \
121123 fw_def(ALDERLAKE_P, no_ver(i915, huc, tgl)) \
@@ -125,6 +127,7 @@ struct fw_blobs_by_type { · 3 unmodified lines
125127
126128 /* for the GSC FW we match the compatibility version and not the release one */
127129 #define XE_GSC_FIRMWARE_DEFS(fw_def, major_ver) \
130+ fw_def(LUNARLAKE, major_ver(xe, gsc, lnl, 1, 0, 0)) \
128131 fw_def(METEORLAKE, major_ver(i915, gsc, mtl, 1, 0, 0))
129132
130133 #define MAKE_FW_PATH(dir__, uc__, shortname__, version__) \
drivers/gpu/drm/xe/xe_vm.c+273−426
@@ -133,8 +133,10 @@ static int wait_for_existing_preempt_fences(struct xe_vm *vm) · 132 unmodified lines
133133 if (q->lr.pfence) {
134134 long timeout = dma_fence_wait(q->lr.pfence, false);
135135
136− if (timeout < 0)
136+ /* Only -ETIME on fence indicates VM needs to be killed */
137+ if (timeout < 0 || q->lr.pfence->error == -ETIME)
137138 return -ETIME;
139+
138140 dma_fence_put(q->lr.pfence);
139141 q->lr.pfence = NULL;
140142 }
@@ -311,7 +313,15 @@ int __xe_vm_userptr_needs_repin(struct xe_vm *vm) · 170 unmodified lines
311313
312314 #define XE_VM_REBIND_RETRY_TIMEOUT_MS 1000
313315
314−static void xe_vm_kill(struct xe_vm *vm, bool unlocked)
316+/**
317+ * xe_vm_kill() - VM Kill
318+ * @vm: The VM.
319+ * @unlocked: Flag indicates the VM's dma-resv is not held
320+ *
321+ * Kill the VM by setting banned flag indicated VM is no longer available for
322+ * use. If in preempt fence mode, also kill all exec queue attached to the VM.
323+ */
324+void xe_vm_kill(struct xe_vm *vm, bool unlocked)
315325 {
316326 struct xe_exec_queue *q;
317327
@@ -708,6 +718,42 @@ int xe_vm_userptr_check_repin(struct xe_vm *vm) · 390 unmodified lines
708718 list_empty_careful(&vm->userptr.invalidated)) ? 0 : -EAGAIN;
709719 }
710720
721+static int xe_vma_ops_alloc(struct xe_vma_ops *vops, bool array_of_binds)
722+{
723+ int i;
724+
725+ for (i = 0; i < XE_MAX_TILES_PER_DEVICE; ++i) {
726+ if (!vops->pt_update_ops[i].num_ops)
727+ continue;
728+
729+ vops->pt_update_ops[i].ops =
730+ kmalloc_array(vops->pt_update_ops[i].num_ops,
731+ sizeof(*vops->pt_update_ops[i].ops),
732+ GFP_KERNEL);
733+ if (!vops->pt_update_ops[i].ops)
734+ return array_of_binds ? -ENOBUFS : -ENOMEM;
735+ }
736+
737+ return 0;
738+}
739+
740+static void xe_vma_ops_fini(struct xe_vma_ops *vops)
741+{
742+ int i;
743+
744+ for (i = 0; i < XE_MAX_TILES_PER_DEVICE; ++i)
745+ kfree(vops->pt_update_ops[i].ops);
746+}
747+
748+static void xe_vma_ops_incr_pt_update_ops(struct xe_vma_ops *vops, u8 tile_mask)
749+{
750+ int i;
751+
752+ for (i = 0; i < XE_MAX_TILES_PER_DEVICE; ++i)
753+ if (BIT(i) & tile_mask)
754+ ++vops->pt_update_ops[i].num_ops;
755+}
756+
711757 static void xe_vm_populate_rebind(struct xe_vma_op *op, struct xe_vma *vma,
712758 u8 tile_mask)
713759 {
@@ -735,6 +781,7 @@ static int xe_vm_ops_add_rebind(struct xe_vma_ops *vops, struct xe_vma *vma, · 21 unmodified lines
735781
736782 xe_vm_populate_rebind(op, vma, tile_mask);
737783 list_add_tail(&op->link, &vops->list);
784+ xe_vma_ops_incr_pt_update_ops(vops, tile_mask);
738785
739786 return 0;
740787 }
@@ -751,7 +798,7 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker) · 10 unmodified lines
751798 struct xe_vma *vma, *next;
752799 struct xe_vma_ops vops;
753800 struct xe_vma_op *op, *next_op;
754− int err;
801+ int err, i;
755802
756803 lockdep_assert_held(&vm->lock);
757804 if ((xe_vm_in_lr_mode(vm) && !rebind_worker) ||
@@ -759,6 +806,8 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker) · 1 unmodified line
759806 return 0;
760807
761808 xe_vma_ops_init(&vops, vm, NULL, NULL, 0);
809+ for (i = 0; i < XE_MAX_TILES_PER_DEVICE; ++i)
810+ vops.pt_update_ops[i].wait_vm_bookkeep = true;
762811
763812 xe_vm_assert_held(vm);
764813 list_for_each_entry(vma, &vm->rebind_list, combined_links.rebind) {
@@ -775,6 +824,10 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker) · 10 unmodified lines
775824 goto free_ops;
776825 }
777826
827+ err = xe_vma_ops_alloc(&vops, false);
828+ if (err)
829+ goto free_ops;
830+
778831 fence = ops_execute(vm, &vops);
779832 if (IS_ERR(fence)) {
780833 err = PTR_ERR(fence);
@@ -789,6 +842,7 @@ free_ops: · 8 unmodified lines
789842 list_del(&op->link);
790843 kfree(op);
791844 }
845+ xe_vma_ops_fini(&vops);
792846
793847 return err;
794848 }
@@ -798,6 +852,8 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_ma · 3 unmodified lines
798852 struct dma_fence *fence = NULL;
799853 struct xe_vma_ops vops;
800854 struct xe_vma_op *op, *next_op;
855+ struct xe_tile *tile;
856+ u8 id;
801857 int err;
802858
803859 lockdep_assert_held(&vm->lock);
@@ -805,17 +861,30 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_ma · 1 unmodified line
805861 xe_assert(vm->xe, xe_vm_in_fault_mode(vm));
806862
807863 xe_vma_ops_init(&vops, vm, NULL, NULL, 0);
864+ for_each_tile(tile, vm->xe, id) {
865+ vops.pt_update_ops[id].wait_vm_bookkeep = true;
866+ vops.pt_update_ops[tile->id].q =
867+ xe_tile_migrate_exec_queue(tile);
868+ }
808869
809870 err = xe_vm_ops_add_rebind(&vops, vma, tile_mask);
810871 if (err)
811872 return ERR_PTR(err);
812873
874+ err = xe_vma_ops_alloc(&vops, false);
875+ if (err) {
876+ fence = ERR_PTR(err);
877+ goto free_ops;
878+ }
879+
813880 fence = ops_execute(vm, &vops);
814881
882+free_ops:
815883 list_for_each_entry_safe(op, next_op, &vops.list, link) {
816884 list_del(&op->link);
817885 kfree(op);
818886 }
887+ xe_vma_ops_fini(&vops);
819888
820889 return fence;
821890 }
@@ -1601,6 +1670,10 @@ static void vm_destroy_work_func(struct work_struct *w) · 779 unmodified lines
16011670 XE_WARN_ON(vm->pt_root[id]);
16021671
16031672 trace_xe_vm_free(vm);
1673+
1674+ if (vm->xef)
1675+ xe_file_put(vm->xef);
1676+
16041677 kfree(vm);
16051678 }
16061679
@@ -1637,147 +1710,6 @@ to_wait_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q) · 30 unmodified lines
16371710 return q ? q : vm->q[0];
16381711 }
16391712
1640−static struct dma_fence *
1641−xe_vm_unbind_vma(struct xe_vma *vma, struct xe_exec_queue *q,
1642− struct xe_sync_entry *syncs, u32 num_syncs,
1643− bool first_op, bool last_op)
1644−{
1645− struct xe_vm *vm = xe_vma_vm(vma);
1646− struct xe_exec_queue *wait_exec_queue = to_wait_exec_queue(vm, q);
1647− struct xe_tile *tile;
1648− struct dma_fence *fence = NULL;
1649− struct dma_fence **fences = NULL;
1650− struct dma_fence_array *cf = NULL;
1651− int cur_fence = 0;
1652− int number_tiles = hweight8(vma->tile_present);
1653− int err;
1654− u8 id;
1655−
1656− trace_xe_vma_unbind(vma);
1657−
1658− if (number_tiles > 1) {
1659− fences = kmalloc_array(number_tiles, sizeof(*fences),
1660− GFP_KERNEL);
1661− if (!fences)
1662− return ERR_PTR(-ENOMEM);
1663− }
1664−
1665− for_each_tile(tile, vm->xe, id) {
1666− if (!(vma->tile_present & BIT(id)))
1667− goto next;
1668−
1669− fence = __xe_pt_unbind_vma(tile, vma, q ? q : vm->q[id],
1670− first_op ? syncs : NULL,
1671− first_op ? num_syncs : 0);
1672− if (IS_ERR(fence)) {
1673− err = PTR_ERR(fence);
1674− goto err_fences;
1675− }
1676−
1677− if (fences)
1678− fences[cur_fence++] = fence;
1679−
1680−next:
1681− if (q && vm->pt_root[id] && !list_empty(&q->multi_gt_list))
1682− q = list_next_entry(q, multi_gt_list);
1683− }
1684−
1685− if (fences) {
1686− cf = dma_fence_array_create(number_tiles, fences,
1687− vm->composite_fence_ctx,
1688− vm->composite_fence_seqno++,
1689− false);
1690− if (!cf) {
1691− --vm->composite_fence_seqno;
1692− err = -ENOMEM;
1693− goto err_fences;
1694− }
1695− }
1696−
1697− fence = cf ? &cf->base : !fence ?
1698− xe_exec_queue_last_fence_get(wait_exec_queue, vm) : fence;
1699−
1700− return fence;
1701−
1702−err_fences:
1703− if (fences) {
1704− while (cur_fence)
1705− dma_fence_put(fences[--cur_fence]);
1706− kfree(fences);
1707− }
1708−
1709− return ERR_PTR(err);
1710−}
1711−
1712−static struct dma_fence *
1713−xe_vm_bind_vma(struct xe_vma *vma, struct xe_exec_queue *q,
1714− struct xe_sync_entry *syncs, u32 num_syncs,
1715− u8 tile_mask, bool first_op, bool last_op)
1716−{
1717− struct xe_tile *tile;
1718− struct dma_fence *fence;
1719− struct dma_fence **fences = NULL;
1720− struct dma_fence_array *cf = NULL;
1721− struct xe_vm *vm = xe_vma_vm(vma);
1722− int cur_fence = 0;
1723− int number_tiles = hweight8(tile_mask);
1724− int err;
1725− u8 id;
1726−
1727− trace_xe_vma_bind(vma);
1728−
1729− if (number_tiles > 1) {
1730− fences = kmalloc_array(number_tiles, sizeof(*fences),
1731− GFP_KERNEL);
1732− if (!fences)
1733− return ERR_PTR(-ENOMEM);
1734− }
1735−
1736− for_each_tile(tile, vm->xe, id) {
1737− if (!(tile_mask & BIT(id)))
1738− goto next;
1739−
1740− fence = __xe_pt_bind_vma(tile, vma, q ? q : vm->q[id],
1741− first_op ? syncs : NULL,
1742− first_op ? num_syncs : 0,
1743− vma->tile_present & BIT(id));
1744− if (IS_ERR(fence)) {
1745− err = PTR_ERR(fence);
1746− goto err_fences;
1747− }
1748−
1749− if (fences)
1750− fences[cur_fence++] = fence;
1751−
1752−next:
1753− if (q && vm->pt_root[id] && !list_empty(&q->multi_gt_list))
1754− q = list_next_entry(q, multi_gt_list);
1755− }
1756−
1757− if (fences) {
1758− cf = dma_fence_array_create(number_tiles, fences,
1759− vm->composite_fence_ctx,
1760− vm->composite_fence_seqno++,
1761− false);
1762− if (!cf) {
1763− --vm->composite_fence_seqno;
1764− err = -ENOMEM;
1765− goto err_fences;
1766− }
1767− }
1768−
1769− return cf ? &cf->base : fence;
1770−
1771−err_fences:
1772− if (fences) {
1773− while (cur_fence)
1774− dma_fence_put(fences[--cur_fence]);
1775− kfree(fences);
1776− }
1777−
1778− return ERR_PTR(err);
1779−}
1780−
17811713 static struct xe_user_fence *
17821714 find_ufence_get(struct xe_sync_entry *syncs, u32 num_syncs)
17831715 {
@@ -1793,48 +1725,6 @@ find_ufence_get(struct xe_sync_entry *syncs, u32 num_syncs) · 9 unmodified lines
17931725 return NULL;
17941726 }
17951727
1796−static struct dma_fence *
1797−xe_vm_bind(struct xe_vm *vm, struct xe_vma *vma, struct xe_exec_queue *q,
1798− struct xe_bo *bo, struct xe_sync_entry *syncs, u32 num_syncs,
1799− u8 tile_mask, bool immediate, bool first_op, bool last_op)
1800−{
1801− struct dma_fence *fence;
1802− struct xe_exec_queue *wait_exec_queue = to_wait_exec_queue(vm, q);
1803−
1804− xe_vm_assert_held(vm);
1805− xe_bo_assert_held(bo);
1806−
1807− if (immediate) {
1808− fence = xe_vm_bind_vma(vma, q, syncs, num_syncs, tile_mask,
1809− first_op, last_op);
1810− if (IS_ERR(fence))
1811− return fence;
1812− } else {
1813− xe_assert(vm->xe, xe_vm_in_fault_mode(vm));
1814−
1815− fence = xe_exec_queue_last_fence_get(wait_exec_queue, vm);
1816− }
1817−
1818− return fence;
1819−}
1820−
1821−static struct dma_fence *
1822−xe_vm_unbind(struct xe_vm *vm, struct xe_vma *vma,
1823− struct xe_exec_queue *q, struct xe_sync_entry *syncs,
1824− u32 num_syncs, bool first_op, bool last_op)
1825−{
1826− struct dma_fence *fence;
1827−
1828− xe_vm_assert_held(vm);
1829− xe_bo_assert_held(xe_vma_bo(vma));
1830−
1831− fence = xe_vm_unbind_vma(vma, q, syncs, num_syncs, first_op, last_op);
1832− if (IS_ERR(fence))
1833− return fence;
1834−
1835− return fence;
1836−}
1837−
18381728 #define ALL_DRM_XE_VM_CREATE_FLAGS (DRM_XE_VM_CREATE_FLAG_SCRATCH_PAGE | \
18391729 DRM_XE_VM_CREATE_FLAG_LR_MODE | \
18401730 DRM_XE_VM_CREATE_FLAG_FAULT_MODE)
@@ -1916,7 +1806,7 @@ int xe_vm_create_ioctl(struct drm_device *dev, void *data, · 75 unmodified lines
19161806 }
19171807
19181808 args->vm_id = id;
1919− vm->xef = xef;
1809+ vm->xef = xe_file_get(xef);
19201810
19211811 /* Record BO memory for VM pagetable created against client */
19221812 for_each_tile(tile, xe, id)
@@ -1975,21 +1865,6 @@ static const u32 region_to_mem_type[] = { · 52 unmodified lines
19751865 XE_PL_VRAM1,
19761866 };
19771867
1978−static struct dma_fence *
1979−xe_vm_prefetch(struct xe_vm *vm, struct xe_vma *vma,
1980− struct xe_exec_queue *q, struct xe_sync_entry *syncs,
1981− u32 num_syncs, bool first_op, bool last_op)
1982−{
1983− struct xe_exec_queue *wait_exec_queue = to_wait_exec_queue(vm, q);
1984−
1985− if (vma->tile_mask != (vma->tile_present & ~vma->tile_invalidated)) {
1986− return xe_vm_bind(vm, vma, q, xe_vma_bo(vma), syncs, num_syncs,
1987− vma->tile_mask, true, first_op, last_op);
1988− } else {
1989− return xe_exec_queue_last_fence_get(wait_exec_queue, vm);
1990− }
1991−}
1992−
19931868 static void prep_vma_destroy(struct xe_vm *vm, struct xe_vma *vma,
19941869 bool post_commit)
19951870 {
@@ -2277,14 +2152,10 @@ static int xe_vma_op_commit(struct xe_vm *vm, struct xe_vma_op *op) · 281 unmodified lines
22772152 return err;
22782153 }
22792154
2280−
2281−static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q,
2282− struct drm_gpuva_ops *ops,
2283− struct xe_sync_entry *syncs, u32 num_syncs,
2284− struct xe_vma_ops *vops, bool last)
2155+static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct drm_gpuva_ops *ops,
2156+ struct xe_vma_ops *vops)
22852157 {
22862158 struct xe_device *xe = vm->xe;
2287− struct xe_vma_op *last_op = NULL;
22882159 struct drm_gpuva_op *__op;
22892160 struct xe_tile *tile;
22902161 u8 id, tile_mask = 0;
@@ -2298,19 +2169,10 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, · 7 unmodified lines
22982169 drm_gpuva_for_each_op(__op, ops) {
22992170 struct xe_vma_op *op = gpuva_op_to_vma_op(__op);
23002171 struct xe_vma *vma;
2301− bool first = list_empty(&vops->list);
23022172 unsigned int flags = 0;
23032173
23042174 INIT_LIST_HEAD(&op->link);
23052175 list_add_tail(&op->link, &vops->list);
2306−
2307− if (first) {
2308− op->flags |= XE_VMA_OP_FIRST;
2309− op->num_syncs = num_syncs;
2310− op->syncs = syncs;
2311− }
2312−
2313− op->q = q;
23142176 op->tile_mask = tile_mask;
23152177
23162178 switch (op->base.op) {
@@ -2329,6 +2191,9 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, · 12 unmodified lines
23292191 return PTR_ERR(vma);
23302192
23312193 op->map.vma = vma;
2194+ if (op->map.immediate || !xe_vm_in_fault_mode(vm))
2195+ xe_vma_ops_incr_pt_update_ops(vops,
2196+ op->tile_mask);
23322197 break;
23332198 }
23342199 case DRM_GPUVA_OP_REMAP:
@@ -2373,6 +2238,8 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, · 38 unmodified lines
23732238 vm_dbg(&xe->drm, "REMAP:SKIP_PREV: addr=0x%016llx, range=0x%016llx",
23742239 (ULL)op->remap.start,
23752240 (ULL)op->remap.range);
2241+ } else {
2242+ xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask);
23762243 }
23772244 }
23782245
@@ -2409,203 +2276,30 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, · 30 unmodified lines
24092276 vm_dbg(&xe->drm, "REMAP:SKIP_NEXT: addr=0x%016llx, range=0x%016llx",
24102277 (ULL)op->remap.start,
24112278 (ULL)op->remap.range);
2279+ } else {
2280+ xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask);
24122281 }
24132282 }
2283+ xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask);
24142284 break;
24152285 }
24162286 case DRM_GPUVA_OP_UNMAP:
24172287 case DRM_GPUVA_OP_PREFETCH:
2418− /* Nothing to do */
2288+ /* FIXME: Need to skip some prefetch ops */
2289+ xe_vma_ops_incr_pt_update_ops(vops, op->tile_mask);
24192290 break;
24202291 default:
24212292 drm_warn(&vm->xe->drm, "NOT POSSIBLE");
24222293 }
24232294
2424− last_op = op;
2425−
24262295 err = xe_vma_op_commit(vm, op);
24272296 if (err)
24282297 return err;
24292298 }
24302299
2431− /* FIXME: Unhandled corner case */
2432− XE_WARN_ON(!last_op && last && !list_empty(&vops->list));
2433−
2434− if (!last_op)
2435− return 0;
2436−
2437− if (last) {
2438− last_op->flags |= XE_VMA_OP_LAST;
2439− last_op->num_syncs = num_syncs;
2440− last_op->syncs = syncs;
2441− }
2442−
24432300 return 0;
24442301 }
24452302
2446−static struct dma_fence *op_execute(struct xe_vm *vm, struct xe_vma *vma,
2447− struct xe_vma_op *op)
2448−{
2449− struct dma_fence *fence = NULL;
2450−
2451− lockdep_assert_held(&vm->lock);
2452−
2453− xe_vm_assert_held(vm);
2454− xe_bo_assert_held(xe_vma_bo(vma));
2455−
2456− switch (op->base.op) {
2457− case DRM_GPUVA_OP_MAP:
2458− fence = xe_vm_bind(vm, vma, op->q, xe_vma_bo(vma),
2459− op->syncs, op->num_syncs,
2460− op->tile_mask,
2461− op->map.immediate || !xe_vm_in_fault_mode(vm),
2462− op->flags & XE_VMA_OP_FIRST,
2463− op->flags & XE_VMA_OP_LAST);
2464− break;
2465− case DRM_GPUVA_OP_REMAP:
2466− {
2467− bool prev = !!op->remap.prev;
2468− bool next = !!op->remap.next;
2469−
2470− if (!op->remap.unmap_done) {
2471− if (prev || next)
2472− vma->gpuva.flags |= XE_VMA_FIRST_REBIND;
2473− fence = xe_vm_unbind(vm, vma, op->q, op->syncs,
2474− op->num_syncs,
2475− op->flags & XE_VMA_OP_FIRST,
2476− op->flags & XE_VMA_OP_LAST &&
2477− !prev && !next);
2478− if (IS_ERR(fence))
2479− break;
2480− op->remap.unmap_done = true;
2481− }
2482−
2483− if (prev) {
2484− op->remap.prev->gpuva.flags |= XE_VMA_LAST_REBIND;
2485− dma_fence_put(fence);
2486− fence = xe_vm_bind(vm, op->remap.prev, op->q,
2487− xe_vma_bo(op->remap.prev), op->syncs,
2488− op->num_syncs,
2489− op->remap.prev->tile_mask, true,
2490− false,
2491− op->flags & XE_VMA_OP_LAST && !next);
2492− op->remap.prev->gpuva.flags &= ~XE_VMA_LAST_REBIND;
2493− if (IS_ERR(fence))
2494− break;
2495− op->remap.prev = NULL;
2496− }
2497−
2498− if (next) {
2499− op->remap.next->gpuva.flags |= XE_VMA_LAST_REBIND;
2500− dma_fence_put(fence);
2501− fence = xe_vm_bind(vm, op->remap.next, op->q,
2502− xe_vma_bo(op->remap.next),
2503− op->syncs, op->num_syncs,
2504− op->remap.next->tile_mask, true,
2505− false, op->flags & XE_VMA_OP_LAST);
2506− op->remap.next->gpuva.flags &= ~XE_VMA_LAST_REBIND;
2507− if (IS_ERR(fence))
2508− break;
2509− op->remap.next = NULL;
2510− }
2511−
2512− break;
2513− }
2514− case DRM_GPUVA_OP_UNMAP:
2515− fence = xe_vm_unbind(vm, vma, op->q, op->syncs,
2516− op->num_syncs, op->flags & XE_VMA_OP_FIRST,
2517− op->flags & XE_VMA_OP_LAST);
2518− break;
2519− case DRM_GPUVA_OP_PREFETCH:
2520− fence = xe_vm_prefetch(vm, vma, op->q, op->syncs, op->num_syncs,
2521− op->flags & XE_VMA_OP_FIRST,
2522− op->flags & XE_VMA_OP_LAST);
2523− break;
2524− default:
2525− drm_warn(&vm->xe->drm, "NOT POSSIBLE");
2526− }
2527−
2528− if (IS_ERR(fence))
2529− trace_xe_vma_fail(vma);
2530−
2531− return fence;
2532−}
2533−
2534−static struct dma_fence *
2535−__xe_vma_op_execute(struct xe_vm *vm, struct xe_vma *vma,
2536− struct xe_vma_op *op)
2537−{
2538− struct dma_fence *fence;
2539− int err;
2540−
2541−retry_userptr:
2542− fence = op_execute(vm, vma, op);
2543− if (IS_ERR(fence) && PTR_ERR(fence) == -EAGAIN) {
2544− lockdep_assert_held_write(&vm->lock);
2545−
2546− if (op->base.op == DRM_GPUVA_OP_REMAP) {
2547− if (!op->remap.unmap_done)
2548− vma = gpuva_to_vma(op->base.remap.unmap->va);
2549− else if (op->remap.prev)
2550− vma = op->remap.prev;
2551− else
2552− vma = op->remap.next;
2553− }
2554−
2555− if (xe_vma_is_userptr(vma)) {
2556− err = xe_vma_userptr_pin_pages(to_userptr_vma(vma));
2557− if (!err)
2558− goto retry_userptr;
2559−
2560− fence = ERR_PTR(err);
2561− trace_xe_vma_fail(vma);
2562− }
2563− }
2564−
2565− return fence;
2566−}
2567−
2568−static struct dma_fence *
2569−xe_vma_op_execute(struct xe_vm *vm, struct xe_vma_op *op)
2570−{
2571− struct dma_fence *fence = ERR_PTR(-ENOMEM);
2572−
2573− lockdep_assert_held(&vm->lock);
2574−
2575− switch (op->base.op) {
2576− case DRM_GPUVA_OP_MAP:
2577− fence = __xe_vma_op_execute(vm, op->map.vma, op);
2578− break;
2579− case DRM_GPUVA_OP_REMAP:
2580− {
2581− struct xe_vma *vma;
2582−
2583− if (!op->remap.unmap_done)
2584− vma = gpuva_to_vma(op->base.remap.unmap->va);
2585− else if (op->remap.prev)
2586− vma = op->remap.prev;
2587− else
2588− vma = op->remap.next;
2589−
2590− fence = __xe_vma_op_execute(vm, vma, op);
2591− break;
2592− }
2593− case DRM_GPUVA_OP_UNMAP:
2594− fence = __xe_vma_op_execute(vm, gpuva_to_vma(op->base.unmap.va),
2595− op);
2596− break;
2597− case DRM_GPUVA_OP_PREFETCH:
2598− fence = __xe_vma_op_execute(vm,
2599− gpuva_to_vma(op->base.prefetch.va),
2600− op);
2601− break;
2602− default:
2603− drm_warn(&vm->xe->drm, "NOT POSSIBLE");
2604− }
2605−
2606− return fence;
2607−}
2608−
26092303 static void xe_vma_op_unwind(struct xe_vm *vm, struct xe_vma_op *op,
26102304 bool post_commit, bool prev_post_commit,
26112305 bool next_post_commit)
@@ -2788,26 +2482,157 @@ static int vm_bind_ioctl_ops_lock_and_prep(struct drm_exec *exec, · 176 unmodified lines
27882482 return err;
27892483 }
27902484
2485+#ifdef TEST_VM_OPS_ERROR
2486+ if (vops->inject_error &&
2487+ vm->xe->vm_inject_error_position == FORCE_OP_ERROR_LOCK)
2488+ return -ENOSPC;
2489+#endif
2490+
27912491 return 0;
27922492 }
27932493
2494+static void op_trace(struct xe_vma_op *op)
2495+{
2496+ switch (op->base.op) {
2497+ case DRM_GPUVA_OP_MAP:
2498+ trace_xe_vma_bind(op->map.vma);
2499+ break;
2500+ case DRM_GPUVA_OP_REMAP:
2501+ trace_xe_vma_unbind(gpuva_to_vma(op->base.remap.unmap->va));
2502+ if (op->remap.prev)
2503+ trace_xe_vma_bind(op->remap.prev);
2504+ if (op->remap.next)
2505+ trace_xe_vma_bind(op->remap.next);
2506+ break;
2507+ case DRM_GPUVA_OP_UNMAP:
2508+ trace_xe_vma_unbind(gpuva_to_vma(op->base.unmap.va));
2509+ break;
2510+ case DRM_GPUVA_OP_PREFETCH:
2511+ trace_xe_vma_bind(gpuva_to_vma(op->base.prefetch.va));
2512+ break;
2513+ default:
2514+ XE_WARN_ON("NOT POSSIBLE");
2515+ }
2516+}
2517+
2518+static void trace_xe_vm_ops_execute(struct xe_vma_ops *vops)
2519+{
2520+ struct xe_vma_op *op;
2521+
2522+ list_for_each_entry(op, &vops->list, link)
2523+ op_trace(op);
2524+}
2525+
2526+static int vm_ops_setup_tile_args(struct xe_vm *vm, struct xe_vma_ops *vops)
2527+{
2528+ struct xe_exec_queue *q = vops->q;
2529+ struct xe_tile *tile;
2530+ int number_tiles = 0;
2531+ u8 id;
2532+
2533+ for_each_tile(tile, vm->xe, id) {
2534+ if (vops->pt_update_ops[id].num_ops)
2535+ ++number_tiles;
2536+
2537+ if (vops->pt_update_ops[id].q)
2538+ continue;
2539+
2540+ if (q) {
2541+ vops->pt_update_ops[id].q = q;
2542+ if (vm->pt_root[id] && !list_empty(&q->multi_gt_list))
2543+ q = list_next_entry(q, multi_gt_list);
2544+ } else {
2545+ vops->pt_update_ops[id].q = vm->q[id];
2546+ }
2547+ }
2548+
2549+ return number_tiles;
2550+}
2551+
27942552 static struct dma_fence *ops_execute(struct xe_vm *vm,
27952553 struct xe_vma_ops *vops)
27962554 {
2797− struct xe_vma_op *op, *next;
2555+ struct xe_tile *tile;
27982556 struct dma_fence *fence = NULL;
2557+ struct dma_fence **fences = NULL;
2558+ struct dma_fence_array *cf = NULL;
2559+ int number_tiles = 0, current_fence = 0, err;
2560+ u8 id;
27992561
2800− list_for_each_entry_safe(op, next, &vops->list, link) {
2801− dma_fence_put(fence);
2802− fence = xe_vma_op_execute(vm, op);
2803− if (IS_ERR(fence)) {
2804− drm_warn(&vm->xe->drm, "VM op(%d) failed with %ld",
2805− op->base.op, PTR_ERR(fence));
2806− fence = ERR_PTR(-ENOSPC);
2807− break;
2562+ number_tiles = vm_ops_setup_tile_args(vm, vops);
2563+ if (number_tiles == 0)
2564+ return ERR_PTR(-ENODATA);
2565+
2566+ if (number_tiles > 1) {
2567+ fences = kmalloc_array(number_tiles, sizeof(*fences),
2568+ GFP_KERNEL);
2569+ if (!fences) {
2570+ fence = ERR_PTR(-ENOMEM);
2571+ goto err_trace;
2572+ }
2573+ }
2574+
2575+ for_each_tile(tile, vm->xe, id) {
2576+ if (!vops->pt_update_ops[id].num_ops)
2577+ continue;
2578+
2579+ err = xe_pt_update_ops_prepare(tile, vops);
2580+ if (err) {
2581+ fence = ERR_PTR(err);
2582+ goto err_out;
28082583 }
28092584 }
28102585
2586+ trace_xe_vm_ops_execute(vops);
2587+
2588+ for_each_tile(tile, vm->xe, id) {
2589+ if (!vops->pt_update_ops[id].num_ops)
2590+ continue;
2591+
2592+ fence = xe_pt_update_ops_run(tile, vops);
2593+ if (IS_ERR(fence))
2594+ goto err_out;
2595+
2596+ if (fences)
2597+ fences[current_fence++] = fence;
2598+ }
2599+
2600+ if (fences) {
2601+ cf = dma_fence_array_create(number_tiles, fences,
2602+ vm->composite_fence_ctx,
2603+ vm->composite_fence_seqno++,
2604+ false);
2605+ if (!cf) {
2606+ --vm->composite_fence_seqno;
2607+ fence = ERR_PTR(-ENOMEM);
2608+ goto err_out;
2609+ }
2610+ fence = &cf->base;
2611+ }
2612+
2613+ for_each_tile(tile, vm->xe, id) {
2614+ if (!vops->pt_update_ops[id].num_ops)
2615+ continue;
2616+
2617+ xe_pt_update_ops_fini(tile, vops);
2618+ }
2619+
2620+ return fence;
2621+
2622+err_out:
2623+ for_each_tile(tile, vm->xe, id) {
2624+ if (!vops->pt_update_ops[id].num_ops)
2625+ continue;
2626+
2627+ xe_pt_update_ops_abort(tile, vops);
2628+ }
2629+ while (current_fence)
2630+ dma_fence_put(fences[--current_fence]);
2631+ kfree(fences);
2632+ kfree(cf);
2633+
2634+err_trace:
2635+ trace_xe_vm_ops_fail(vm);
28112636 return fence;
28122637 }
28132638
@@ -2888,12 +2713,10 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm, · 74 unmodified lines
28882713 fence = ops_execute(vm, vops);
28892714 if (IS_ERR(fence)) {
28902715 err = PTR_ERR(fence);
2891− /* FIXME: Killing VM rather than proper error handling */
2892− xe_vm_kill(vm, false);
28932716 goto unlock;
2894− } else {
2895− vm_bind_ioctl_ops_fini(vm, vops, fence);
28962717 }
2718+
2719+ vm_bind_ioctl_ops_fini(vm, vops, fence);
28972720 }
28982721
28992722 unlock:
@@ -2901,11 +2724,18 @@ unlock: · 1 unmodified line
29012724 return err;
29022725 }
29032726
2904−#define SUPPORTED_FLAGS \
2727+#define SUPPORTED_FLAGS_STUB \
29052728 (DRM_XE_VM_BIND_FLAG_READONLY | \
29062729 DRM_XE_VM_BIND_FLAG_IMMEDIATE | \
29072730 DRM_XE_VM_BIND_FLAG_NULL | \
29082731 DRM_XE_VM_BIND_FLAG_DUMPABLE)
2732+
2733+#ifdef TEST_VM_OPS_ERROR
2734+#define SUPPORTED_FLAGS (SUPPORTED_FLAGS_STUB | FORCE_OP_ERROR)
2735+#else
2736+#define SUPPORTED_FLAGS SUPPORTED_FLAGS_STUB
2737+#endif
2738+
29092739 #define XE_64K_PAGE_MASK 0xffffull
29102740 #define ALL_DRM_XE_SYNCS_FLAGS (DRM_XE_SYNCS_FLAG_WAIT_FOR_OP)
29112741
@@ -2931,7 +2761,7 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe, · 19 unmodified lines
29312761 sizeof(struct drm_xe_vm_bind_op),
29322762 GFP_KERNEL | __GFP_ACCOUNT);
29332763 if (!*bind_ops)
2934− return -ENOMEM;
2764+ return args->num_binds > 1 ? -ENOBUFS : -ENOMEM;
29352765
29362766 err = __copy_from_user(*bind_ops, bind_user,
29372767 sizeof(struct drm_xe_vm_bind_op) *
@@ -3250,10 +3080,18 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) · 312 unmodified lines
32503080 goto unwind_ops;
32513081 }
32523082
3253− err = vm_bind_ioctl_ops_parse(vm, q, ops[i], syncs, num_syncs,
3254− &vops, i == args->num_binds - 1);
3083+ err = vm_bind_ioctl_ops_parse(vm, ops[i], &vops);
32553084 if (err)
32563085 goto unwind_ops;
3086+
3087+#ifdef TEST_VM_OPS_ERROR
3088+ if (flags & FORCE_OP_ERROR) {
3089+ vops.inject_error = true;
3090+ vm->xe->vm_inject_error_position =
3091+ (vm->xe->vm_inject_error_position + 1) %
3092+ FORCE_OP_ERROR_COUNT;
3093+ }
3094+#endif
32573095 }
32583096
32593097 /* Nothing to do */
@@ -3262,11 +3100,16 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) · 2 unmodified lines
32623100 goto unwind_ops;
32633101 }
32643102
3103+ err = xe_vma_ops_alloc(&vops, args->num_binds > 1);
3104+ if (err)
3105+ goto unwind_ops;
3106+
32653107 err = vm_bind_ioctl_ops_execute(vm, &vops);
32663108
32673109 unwind_ops:
32683110 if (err && err != -ENODATA)
32693111 vm_bind_ioctl_ops_unwind(vm, ops, args->num_binds);
3112+ xe_vma_ops_fini(&vops);
32703113 for (i = args->num_binds - 1; i >= 0; --i)
32713114 if (ops[i])
32723115 drm_gpuva_ops_free(&vm->gpuvm, ops[i]);
@@ -3337,10 +3180,10 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) · 64 unmodified lines
33373180 {
33383181 struct xe_device *xe = xe_vma_vm(vma)->xe;
33393182 struct xe_tile *tile;
3183+ struct xe_gt_tlb_invalidation_fence fence[XE_MAX_TILES_PER_DEVICE];
33403184 u32 tile_needs_invalidate = 0;
3341− int seqno[XE_MAX_TILES_PER_DEVICE];
33423185 u8 id;
3343− int ret;
3186+ int ret = 0;
33443187
33453188 xe_assert(xe, !xe_vma_is_null(vma));
33463189 trace_xe_vma_invalidate(vma);
@@ -3365,29 +3208,33 @@ int xe_vm_invalidate_vma(struct xe_vma *vma) · 18 unmodified lines
33653208
33663209 for_each_tile(tile, xe, id) {
33673210 if (xe_pt_zap_ptes(tile, vma)) {
3368− tile_needs_invalidate |= BIT(id);
33693211 xe_device_wmb(xe);
3212+ xe_gt_tlb_invalidation_fence_init(tile->primary_gt,
3213+ &fence[id], true);
3214+
33703215 /*
33713216 * FIXME: We potentially need to invalidate multiple
33723217 * GTs within the tile
33733218 */
3374− seqno[id] = xe_gt_tlb_invalidation_vma(tile->primary_gt, NULL, vma);
3375− if (seqno[id] < 0)
3376− return seqno[id];
3377− }
3378− }
3219+ ret = xe_gt_tlb_invalidation_vma(tile->primary_gt,
3220+ &fence[id], vma);
3221+ if (ret < 0) {
3222+ xe_gt_tlb_invalidation_fence_fini(&fence[id]);
3223+ goto wait;
3224+ }
33793225
3380− for_each_tile(tile, xe, id) {
3381− if (tile_needs_invalidate & BIT(id)) {
3382− ret = xe_gt_tlb_invalidation_wait(tile->primary_gt, seqno[id]);
3383− if (ret < 0)
3384− return ret;
3226+ tile_needs_invalidate |= BIT(id);
33853227 }
33863228 }
33873229
3230+wait:
3231+ for_each_tile(tile, xe, id)
3232+ if (tile_needs_invalidate & BIT(id))
3233+ xe_gt_tlb_invalidation_fence_wait(&fence[id]);
3234+
33883235 vma->tile_invalidated = vma->tile_mask;
33893236
3390− return 0;
3237+ return ret;
33913238 }
33923239
33933240 struct xe_vm_snapshot {
drivers/gpu/drm/xe/xe_vm.h+2−0
@@ -259,6 +259,8 @@ static inline struct dma_resv *xe_vm_resv(struct xe_vm *vm) · 258 unmodified lines
259259 return drm_gpuvm_resv(&vm->gpuvm);
260260 }
261261
262+void xe_vm_kill(struct xe_vm *vm, bool unlocked);
263+
262264 /**
263265 * xe_vm_assert_held(vm) - Assert that the vm's reservation object is held.
264266 * @vm: The vm
drivers/gpu/drm/xe/xe_vm_types.h+30−25
@@ -21,18 +21,27 @@ struct xe_bo; · 20 unmodified lines
2121 struct xe_sync_entry;
2222 struct xe_user_fence;
2323 struct xe_vm;
24+struct xe_vm_pgtable_update_op;
25+
26+#if IS_ENABLED(CONFIG_DRM_XE_DEBUG)
27+#define TEST_VM_OPS_ERROR
28+#define FORCE_OP_ERROR BIT(31)
29+
30+#define FORCE_OP_ERROR_LOCK 0
31+#define FORCE_OP_ERROR_PREPARE 1
32+#define FORCE_OP_ERROR_RUN 2
33+#define FORCE_OP_ERROR_COUNT 3
34+#endif
2435
2536 #define XE_VMA_READ_ONLY DRM_GPUVA_USERBITS
2637 #define XE_VMA_DESTROYED (DRM_GPUVA_USERBITS << 1)
2738 #define XE_VMA_ATOMIC_PTE_BIT (DRM_GPUVA_USERBITS << 2)
28−#define XE_VMA_FIRST_REBIND (DRM_GPUVA_USERBITS << 3)
29−#define XE_VMA_LAST_REBIND (DRM_GPUVA_USERBITS << 4)
30−#define XE_VMA_PTE_4K (DRM_GPUVA_USERBITS << 5)
31−#define XE_VMA_PTE_2M (DRM_GPUVA_USERBITS << 6)
32−#define XE_VMA_PTE_1G (DRM_GPUVA_USERBITS << 7)
33−#define XE_VMA_PTE_64K (DRM_GPUVA_USERBITS << 8)
34−#define XE_VMA_PTE_COMPACT (DRM_GPUVA_USERBITS << 9)
35−#define XE_VMA_DUMPABLE (DRM_GPUVA_USERBITS << 10)
39+#define XE_VMA_PTE_4K (DRM_GPUVA_USERBITS << 3)
40+#define XE_VMA_PTE_2M (DRM_GPUVA_USERBITS << 4)
41+#define XE_VMA_PTE_1G (DRM_GPUVA_USERBITS << 5)
42+#define XE_VMA_PTE_64K (DRM_GPUVA_USERBITS << 6)
43+#define XE_VMA_PTE_COMPACT (DRM_GPUVA_USERBITS << 7)
44+#define XE_VMA_DUMPABLE (DRM_GPUVA_USERBITS << 8)
3645
3746 /** struct xe_userptr - User pointer */
3847 struct xe_userptr {
@@ -99,6 +108,9 @@ struct xe_vma { · 60 unmodified lines
99108 */
100109 u8 tile_present;
101110
111+ /** @tile_staged: bind is staged for this VMA */
112+ u8 tile_staged;
113+
102114 /**
103115 * @pat_index: The pat index to use when encoding the PTEs for this vma.
104116 */
@@ -314,31 +326,18 @@ struct xe_vma_op_prefetch { · 209 unmodified lines
314326
315327 /** enum xe_vma_op_flags - flags for VMA operation */
316328 enum xe_vma_op_flags {
317− /** @XE_VMA_OP_FIRST: first VMA operation for a set of syncs */
318− XE_VMA_OP_FIRST = BIT(0),
319− /** @XE_VMA_OP_LAST: last VMA operation for a set of syncs */
320− XE_VMA_OP_LAST = BIT(1),
321329 /** @XE_VMA_OP_COMMITTED: VMA operation committed */
322− XE_VMA_OP_COMMITTED = BIT(2),
330+ XE_VMA_OP_COMMITTED = BIT(0),
323331 /** @XE_VMA_OP_PREV_COMMITTED: Previous VMA operation committed */
324− XE_VMA_OP_PREV_COMMITTED = BIT(3),
332+ XE_VMA_OP_PREV_COMMITTED = BIT(1),
325333 /** @XE_VMA_OP_NEXT_COMMITTED: Next VMA operation committed */
326− XE_VMA_OP_NEXT_COMMITTED = BIT(4),
334+ XE_VMA_OP_NEXT_COMMITTED = BIT(2),
327335 };
328336
329337 /** struct xe_vma_op - VMA operation */
330338 struct xe_vma_op {
331339 /** @base: GPUVA base operation */
332340 struct drm_gpuva_op base;
333− /** @q: exec queue for this operation */
334− struct xe_exec_queue *q;
335− /**
336− * @syncs: syncs for this operation, only used on first and last
337− * operation
338− */
339− struct xe_sync_entry *syncs;
340− /** @num_syncs: number of syncs */
341− u32 num_syncs;
342341 /** @link: async operation link */
343342 struct list_head link;
344343 /** @flags: operation flags */
@@ -362,12 +361,18 @@ struct xe_vma_ops { · 17 unmodified lines
362361 struct list_head list;
363362 /** @vm: VM */
364363 struct xe_vm *vm;
365− /** @q: exec queue these operations */
364+ /** @q: exec queue for VMA operations */
366365 struct xe_exec_queue *q;
367366 /** @syncs: syncs these operation */
368367 struct xe_sync_entry *syncs;
369368 /** @num_syncs: number of syncs */
370369 u32 num_syncs;
370+ /** @pt_update_ops: page table update operations */
371+ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE];
372+#ifdef TEST_VM_OPS_ERROR
373+ /** @inject_error: inject error to test error handling */
374+ bool inject_error;
375+#endif
371376 };
372377
373378 #endif
drivers/gpu/drm/xe/xe_wa.c+15−0
@@ -486,6 +486,10 @@ static const struct xe_rtp_entry_sr engine_was[] = { · 485 unmodified lines
486486 XE_RTP_RULES(GRAPHICS_VERSION(2004), FUNC(xe_rtp_match_first_render_or_compute)),
487487 XE_RTP_ACTIONS(SET(TDL_TSL_CHICKEN, SLM_WMTP_RESTORE))
488488 },
489+ { XE_RTP_NAME("14021402888"),
490+ XE_RTP_RULES(GRAPHICS_VERSION(2004), ENGINE_CLASS(RENDER)),
491+ XE_RTP_ACTIONS(SET(HALF_SLICE_CHICKEN7, CLEAR_OPTIMIZATION_DISABLE))
492+ },
489493
490494 /* Xe2_HPG */
491495
@@ -539,6 +543,16 @@ static const struct xe_rtp_entry_sr engine_was[] = { · 47 unmodified lines
539543 XE_RTP_ACTIONS(SET(HALF_SLICE_CHICKEN7, CLEAR_OPTIMIZATION_DISABLE))
540544 },
541545
546+ /* Xe2_LPM */
547+
548+ { XE_RTP_NAME("16021639441"),
549+ XE_RTP_RULES(MEDIA_VERSION(2000)),
550+ XE_RTP_ACTIONS(SET(CSFE_CHICKEN1(0),
551+ GHWSP_CSB_REPORT_DIS |
552+ PPHWSP_CSB_AND_TIMESTAMP_REPORT_DIS,
553+ XE_RTP_ACTION_FLAG(ENGINE_BASE)))
554+ },
555+
542556 /* Xe2_HPM */
543557
544558 { XE_RTP_NAME("16021639441"),
@@ -741,6 +755,7 @@ void xe_wa_process_oob(struct xe_gt *gt) · 196 unmodified lines
741755
742756 xe_rtp_process_ctx_enable_active_tracking(&ctx, gt->wa_active.oob,
743757 ARRAY_SIZE(oob_was));
758+ gt->wa_active.oob_initialized = true;
744759 xe_rtp_process(&ctx, oob_was);
745760 }
746761
drivers/gpu/drm/xe/xe_wa.h+6−1
@@ -6,6 +6,8 @@ · 5 unmodified lines
66 #ifndef _XE_WA_
77 #define _XE_WA_
88
9+#include "xe_assert.h"
10+
911 struct drm_printer;
1012 struct xe_gt;
1113 struct xe_hw_engine;
@@ -25,6 +27,9 @@ void xe_wa_dump(struct xe_gt *gt, struct drm_printer *p); · 13 unmodified lines
2527 * @gt__: gt instance
2628 * @id__: XE_OOB_<id__>, as generated by build system in generated/xe_wa_oob.h
2729 */
28−#define XE_WA(gt__, id__) test_bit(XE_WA_OOB_ ## id__, (gt__)->wa_active.oob)
30+#define XE_WA(gt__, id__) ({ \
31+ xe_gt_assert(gt__, (gt__)->wa_active.oob_initialized); \
32+ test_bit(XE_WA_OOB_ ## id__, (gt__)->wa_active.oob); \
33+})
2934
3035 #endif
drivers/gpu/drm/xe/xe_wa_oob.rules+2−0
@@ -29,3 +29,5 @@ · 28 unmodified lines
2929 13011645652 GRAPHICS_VERSION(2004)
3030 22019338487 MEDIA_VERSION(2000)
3131 GRAPHICS_VERSION(2001)
32+22019338487_display PLATFORM(LUNARLAKE)
33+16023588340 GRAPHICS_VERSION(2001)
include/uapi/drm/xe_drm.h+13−5
@@ -517,7 +517,14 @@ struct drm_xe_query_gt_list { · 516 unmodified lines
517517 * available per Dual Sub Slices (DSS). For example a query response
518518 * containing the following in mask:
519519 * ``EU_PER_DSS ff ff 00 00 00 00 00 00``
520− * means each DSS has 16 EU.
520+ * means each DSS has 16 SIMD8 EUs. This type may be omitted if device
521+ * doesn't have SIMD8 EUs.
522+ * - %DRM_XE_TOPO_SIMD16_EU_PER_DSS - To query the mask of SIMD16 Execution
523+ * Units (EU) available per Dual Sub Slices (DSS). For example a query
524+ * response containing the following in mask:
525+ * ``SIMD16_EU_PER_DSS ff ff 00 00 00 00 00 00``
526+ * means each DSS has 16 SIMD16 EUs. This type may be omitted if device
527+ * doesn't have SIMD16 EUs.
521528 */
522529 struct drm_xe_query_topology_mask {
523530 /** @gt_id: GT ID the mask is associated with */
@@ -527,6 +534,7 @@ struct drm_xe_query_topology_mask { · 3 unmodified lines
527534 #define DRM_XE_TOPO_DSS_COMPUTE 2
528535 #define DRM_XE_TOPO_L3_BANK 3
529536 #define DRM_XE_TOPO_EU_PER_DSS 4
537+#define DRM_XE_TOPO_SIMD16_EU_PER_DSS 5
530538 /** @type: type of mask */
531539 __u16 type;
532540
@@ -1590,10 +1598,10 @@ enum drm_xe_oa_property_id { · 1057 unmodified lines
15901598 * b. Counter select c. Counter size and d. BC report. Also refer to the
15911599 * oa_formats array in drivers/gpu/drm/xe/xe_oa.c.
15921600 */
1593−#define DRM_XE_OA_FORMAT_MASK_FMT_TYPE (0xff << 0)
1594−#define DRM_XE_OA_FORMAT_MASK_COUNTER_SEL (0xff << 8)
1595−#define DRM_XE_OA_FORMAT_MASK_COUNTER_SIZE (0xff << 16)
1596−#define DRM_XE_OA_FORMAT_MASK_BC_REPORT (0xff << 24)
1601+#define DRM_XE_OA_FORMAT_MASK_FMT_TYPE (0xffu << 0)
1602+#define DRM_XE_OA_FORMAT_MASK_COUNTER_SEL (0xffu << 8)
1603+#define DRM_XE_OA_FORMAT_MASK_COUNTER_SIZE (0xffu << 16)
1604+#define DRM_XE_OA_FORMAT_MASK_BC_REPORT (0xffu << 24)
15971605
15981606 /**
15991607 * @DRM_XE_OA_PROPERTY_OA_PERIOD_EXPONENT: Requests periodic OA unit

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t