History

Merge tag 'drm-intel-gt-next-2021-01-14' of git://anongit.freedesktop.org/drm/drm-intel into drm-next

UAPI Changes: - Deprecate I915_PMU_LAST and optimize state tracking (Tvrtko) Avoid relying on last item ABI marker in i915_drm.h, add a comment to mark as deprecated. Cross-subsystem Changes: Core Changes: Driver Changes: - Restore clear residuals security mitigations for Ivybridge and Baytrail (Chris) - Close #1858: Allow sysadmin to choose applied GPU security mitigations through i915.mitigations=... similar to CPU (Chris) - Fix for #2024: GPU hangs on HSW GT1 (Chris) - Fix for #2707: Driver hang when editing UVs in Blender (Chris, Ville) - Fix for #2797: False positive GuC loading error message (Chris) - Fix for #2859: Missing GuC firmware for older Cometlakes (Chris) - Lessen probability of GPU hang due to DMAR faults [reason 7, next page table ptr is invalid] on Tigerlake (Chris) - Fix REVID macros for TGL to fetch correct stepping (Aditya) - Limit frequency drop to RPe on parking (Chris, Edward) - Limit W/A 1406941453 to TGL, RKL and DG1 (Swathi) - Make W/A 22010271021 permanent on DG1 (Lucas) - Implement W/A 16011163337 to prevent a HS/DS hang on DG1 (Swathi) - Only disable preemption on gen8 render engines (Chris) - Disable arbitration around Braswell's PDP updates (Chris) - Disable arbitration on no-preempt requests (Chris) - Check for arbitration after writing start seqno before busywaiting (Chris) - Retain default context state across shrinking (Venkata, CQ) - Fix mismatch between misplaced vma check and vma insert for 32-bit addressing userspaces (Chris, CQ) - Propagate error for vmap() failure instead kernel NULL deref (Chris) - Propagate error from cancelled submit due to context closure immediately (Chris) - Fix RCU race on HWSP tracking per request (Chris) - Clear CMD parser shadow and GPU reloc batches (Matt A) - Populate logical context during first pin (Maarten) - Optimistically prune dma-resv from the shrinker (Chris) - Fix for virtual engine ownership race (Chris) - Remove timeslice suppression to restore fairness for virtual engines (Chris) - Rearrange IVB/HSW workarounds properly between GT and engine (Chris) - Taint the reset mutex with the shrinker (Chris) - Replace direct submit with direct call to tasklet (Chris) - Multiple corrections to virtual engine dequeue and breadcrumbs code (Chris) - Avoid wakeref from potentially hard IRQ context in PMU (Tvrtko) - Use raw clock for RC6 time estimation in PMU (Tvrtko) - Differentiate OOM failures from invalid map types (Chris) - Fix Gen9 to have 64 MOCS entries similar to Gen11 (Chris) - Ignore repeated attempts to suspend request flow across reset (Chris) - Remove livelock from "do_idle_maps" VT-d W/A (Chris) - Cancel the preemption timeout early in case engine reset fails (Chris) - Code flow optimization in the scheduling code (Chris) - Clear the execlists timers upon reset (Chris) - Drain the breadcrumbs just once (Chris, Matt A) - Track the overall GT awake/busy time (Chris) - Tweak submission tasklet flushing to avoid starvation (Chris) - Track timelines created using the HWSP to restore on resume (Chris) - Use cmpxchg64 for 32b compatilibity for active tracking (Chris) - Prefer recycling an idle GGTT fence to avoid GPU wait (Chris) - Restructure GT code organization for clearer split between GuC and execlists (Chris, Daniele, John, Matt A) - Remove GuC code that will remain unused by new interfaces (Matt B) - Restructure the CS timestamp clocks code to local to GT (Chris) - Fix error return paths in perf code (Zhang) - Replace idr_init() by idr_init_base() in perf (Deepak) - Fix shmem_pin_map error path (Colin) - Drop redundant free_work worker for GEM contexts (Chris, Mika) - Increase readability and understandability of intel_workarounds.c (Lucas) - Defer enabling the breadcrumb interrupt to after submission (Chris) - Deal with buddy alloc block sizes beyond 4G (Venkata, Chris) - Encode fence specific waitqueue behaviour into the wait.flags (Chris) - Don't cancel the breadcrumb interrupt shadow too early (Chris) - Cancel submitted requests upon context reset (Chris) - Use correct locks in GuC code (Tvrtko) - Prevent use of engine->wa_ctx after error (Chris, Matt R) - Fix build warning on 32-bit (Arnd) - Avoid memory leak if platform would have more than 16 W/A (Tvrtko) - Avoid unnecessary #if CONFIG_PM in PMU code (Chris, Tvrtko) - Improve debugging output (Chris, Tvrtko, Matt R) - Make file local variables static (Jani) - Avoid uint*_t types in i915 (Jani) - Selftest improvements (Chris, Matt A, Dan) - Documentation fixes (Chris, Jose) Signed-off-by: Dave Airlie <airlied@redhat.com> # Conflicts: # drivers/gpu/drm/i915/gt/intel_breadcrumbs.c # drivers/gpu/drm/i915/gt/intel_breadcrumbs_types.h # drivers/gpu/drm/i915/gt/intel_lrc.c # drivers/gpu/drm/i915/gvt/mmio_context.h # drivers/gpu/drm/i915/i915_drv.h From: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20210114152232.GA21588@jlahtine-mobl.ger.corp.intel.com

Dave Airlie committed 5 years ago

Commit fb5cfca2 parents dfa7c52 368fd0d130 files · +13692 −12273Browse files

Unified · Split

130 files
  1. Documentation/gpu/i915.rst +1 −1
  2. drivers/gpu/drm/i915/Makefile +5 −1
  3. drivers/gpu/drm/i915/display/intel_overlay.c +1 −0
  4. drivers/gpu/drm/i915/dma_resv_utils.c +17 −0
  5. drivers/gpu/drm/i915/dma_resv_utils.h +13 −0
  6. drivers/gpu/drm/i915/gem/i915_gem_context.c +13 −54
  7. drivers/gpu/drm/i915/gem/i915_gem_context.h +0 −1
  8. drivers/gpu/drm/i915/gem/i915_gem_context_types.h +0 −1
  9. drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c +8 −7
  10. drivers/gpu/drm/i915/gem/i915_gem_object_blt.c +1 −0
  11. drivers/gpu/drm/i915/gem/i915_gem_pages.c +16 −16
  12. drivers/gpu/drm/i915/gem/i915_gem_region.c +20 −16
  13. drivers/gpu/drm/i915/gem/i915_gem_shrinker.c +3 −0
  14. drivers/gpu/drm/i915/gem/i915_gem_stolen.c +2 −3
  15. drivers/gpu/drm/i915/gem/i915_gem_wait.c +3 −5
  16. drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c +1 −1
  17. drivers/gpu/drm/i915/gem/selftests/huge_pages.c +22 −0
  18. drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c +1 −3
  19. drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c +1 −0
  20. drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c +1 −0
  21. drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c +1 −0
  22. drivers/gpu/drm/i915/gt/debugfs_gt_pm.c +15 −12
  23. drivers/gpu/drm/i915/gt/gen7_renderclear.c +94 −63
  24. drivers/gpu/drm/i915/gt/gen8_engine_cs.c +633 −0
  25. drivers/gpu/drm/i915/gt/gen8_engine_cs.h +127 −0
  26. drivers/gpu/drm/i915/gt/intel_breadcrumbs.c +73 −37
  27. drivers/gpu/drm/i915/gt/intel_breadcrumbs.h +16 −1
  28. drivers/gpu/drm/i915/gt/intel_breadcrumbs_types.h +5 −2
  29. drivers/gpu/drm/i915/gt/intel_context.h +7 −4
  30. drivers/gpu/drm/i915/gt/intel_context_sseu.c +0 −1
  31. drivers/gpu/drm/i915/gt/intel_context_types.h +17 −8
  32. drivers/gpu/drm/i915/gt/intel_engine.h +6 −87
  33. drivers/gpu/drm/i915/gt/intel_engine_cs.c +80 −71
  34. drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c +41 −18
  35. drivers/gpu/drm/i915/gt/intel_engine_pm.c +9 −1
  36. drivers/gpu/drm/i915/gt/intel_engine_types.h +5 −11
  37. drivers/gpu/drm/i915/gt/intel_execlists_submission.c +3929 −0
  38. drivers/gpu/drm/i915/gt/intel_execlists_submission.h +47 −0
  39. drivers/gpu/drm/i915/gt/intel_ggtt.c +16 −2
  40. drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c +20 −2
  41. drivers/gpu/drm/i915/gt/intel_gt.c +2 −2
  42. drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c +157 −40
  43. drivers/gpu/drm/i915/gt/intel_gt_clock_utils.h +4 −4
  44. drivers/gpu/drm/i915/gt/intel_gt_irq.c +1 −0
  45. drivers/gpu/drm/i915/gt/intel_gt_pm.c +49 −0
  46. drivers/gpu/drm/i915/gt/intel_gt_pm.h +2 −0
  47. drivers/gpu/drm/i915/gt/intel_gt_requests.c +1 −6
  48. drivers/gpu/drm/i915/gt/intel_gt_types.h +25 −0
  49. drivers/gpu/drm/i915/gt/intel_gtt.c +29 −0
  50. drivers/gpu/drm/i915/gt/intel_gtt.h +3 −0
  51. drivers/gpu/drm/i915/gt/intel_lrc.c +1149 −5731
  52. drivers/gpu/drm/i915/gt/intel_lrc.h +61 −106
  53. drivers/gpu/drm/i915/gt/intel_lrc_reg.h +41 −0
  54. drivers/gpu/drm/i915/gt/intel_mocs.c +1 −1
  55. drivers/gpu/drm/i915/gt/intel_region_lmem.c +1 −1
  56. drivers/gpu/drm/i915/gt/intel_region_lmem.h +0 −2
  57. drivers/gpu/drm/i915/gt/intel_renderstate.c +2 −1
  58. drivers/gpu/drm/i915/gt/intel_reset.c +62 −34
  59. drivers/gpu/drm/i915/gt/intel_reset.h +2 −0
  60. drivers/gpu/drm/i915/gt/intel_ring.c +2 −0
  61. drivers/gpu/drm/i915/gt/intel_ring_submission.c +43 −30
  62. drivers/gpu/drm/i915/gt/intel_rps.c +35 −24
  63. drivers/gpu/drm/i915/gt/intel_rps_types.h +1 −1
  64. drivers/gpu/drm/i915/gt/intel_timeline.c +103 −6
  65. drivers/gpu/drm/i915/gt/intel_timeline.h +18 −7
  66. drivers/gpu/drm/i915/gt/intel_timeline_types.h +2 −0
  67. drivers/gpu/drm/i915/gt/intel_workarounds.c +309 −335
  68. drivers/gpu/drm/i915/gt/mock_engine.c +2 −13
  69. drivers/gpu/drm/i915/gt/selftest_context.c +1 −1
  70. drivers/gpu/drm/i915/gt/selftest_engine_cs.c +1 −0
  71. drivers/gpu/drm/i915/gt/selftest_engine_heartbeat.c +10 −3
  72. drivers/gpu/drm/i915/gt/selftest_engine_pm.c +203 −0
  73. drivers/gpu/drm/i915/gt/selftest_execlists.c +4741 −0
  74. drivers/gpu/drm/i915/gt/selftest_gt_pm.c +4 −4
  75. drivers/gpu/drm/i915/gt/selftest_hangcheck.c +154 −3
  76. drivers/gpu/drm/i915/gt/selftest_lrc.c +70 −4701
  77. drivers/gpu/drm/i915/gt/selftest_mocs.c +28 −52
  78. drivers/gpu/drm/i915/gt/selftest_rc6.c +1 −0
  79. drivers/gpu/drm/i915/gt/selftest_reset.c +13 −6
  80. drivers/gpu/drm/i915/gt/selftest_rps.c +4 −1
  81. drivers/gpu/drm/i915/gt/selftest_timeline.c +7 −7
  82. drivers/gpu/drm/i915/gt/selftest_workarounds.c +93 −103
  83. drivers/gpu/drm/i915/gt/uc/intel_guc.c +2 −14
  84. drivers/gpu/drm/i915/gt/uc/intel_guc.h +0 −7
  85. drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c +1 −0
  86. drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c +6 −7
  87. drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +270 −187
  88. drivers/gpu/drm/i915/gt/uc/intel_guc_submission.h +1 −0
  89. drivers/gpu/drm/i915/gt/uc/intel_uc.c +10 −13
  90. drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c +1 −0
  91. drivers/gpu/drm/i915/gvt/cmd_parser.c +1 −0
  92. drivers/gpu/drm/i915/gvt/handlers.c +1 −1
  93. drivers/gpu/drm/i915/gvt/mmio_context.c +1 −0
  94. drivers/gpu/drm/i915/gvt/mmio_context.h +1 −0
  95. drivers/gpu/drm/i915/gvt/scheduler.c +2 −0
  96. drivers/gpu/drm/i915/i915_active.c +2 −5
  97. drivers/gpu/drm/i915/i915_cmd_parser.c +10 −18
  98. drivers/gpu/drm/i915/i915_debugfs.c +22 −17
  99. drivers/gpu/drm/i915/i915_drv.c +1 −2
  100. drivers/gpu/drm/i915/i915_drv.h +25 −29
  101. drivers/gpu/drm/i915/i915_gem.c +0 −2
  102. drivers/gpu/drm/i915/i915_gem_gtt.c +10 −15
  103. drivers/gpu/drm/i915/i915_getparam.c +1 −1
  104. drivers/gpu/drm/i915/i915_gpu_error.c +1 −1
  105. drivers/gpu/drm/i915/i915_mitigations.c +146 −0
  106. drivers/gpu/drm/i915/i915_mitigations.h +13 −0
  107. drivers/gpu/drm/i915/i915_pci.c +0 −2
  108. drivers/gpu/drm/i915/i915_perf.c +11 −6
  109. drivers/gpu/drm/i915/i915_pmu.c +71 −46
  110. drivers/gpu/drm/i915/i915_pmu.h +24 −11
  111. drivers/gpu/drm/i915/i915_request.c +128 −11
  112. drivers/gpu/drm/i915/i915_request.h +41 −6
  113. drivers/gpu/drm/i915/i915_scheduler.c +28 −4
  114. drivers/gpu/drm/i915/i915_scheduler.h +7 −0
  115. drivers/gpu/drm/i915/i915_scheduler_types.h +10 −0
  116. drivers/gpu/drm/i915/i915_sw_fence.c +15 −10
  117. drivers/gpu/drm/i915/i915_utils.c +1 −1
  118. drivers/gpu/drm/i915/i915_utils.h +6 −1
  119. drivers/gpu/drm/i915/intel_device_info.c +0 −157
  120. drivers/gpu/drm/i915/intel_device_info.h +0 −4
  121. drivers/gpu/drm/i915/intel_memory_region.c +1 −1
  122. drivers/gpu/drm/i915/intel_memory_region.h +0 −5
  123. drivers/gpu/drm/i915/selftests/i915_gem_evict.c +5 −13
  124. drivers/gpu/drm/i915/selftests/i915_gem_gtt.c +1 −0
  125. drivers/gpu/drm/i915/selftests/i915_perf.c +1 −1
  126. drivers/gpu/drm/i915/selftests/i915_request.c +3 −6
  127. drivers/gpu/drm/i915/selftests/igt_spinner.c +4 −0
  128. drivers/gpu/drm/i915/selftests/intel_memory_region.c +98 −12
  129. drivers/gpu/drm/i915/selftests/mock_gem_device.c +0 −2
  130. include/uapi/drm/i915_drm.h +2 −1
Documentation/gpu/i915.rst+1−1
@@ -428,7 +428,7 @@ User Batchbuffer Execution · 427 unmodified lines
428428 Logical Rings, Logical Ring Contexts and Execlists
429429 --------------------------------------------------
430430
431−.. kernel-doc:: drivers/gpu/drm/i915/gt/intel_lrc.c
431+.. kernel-doc:: drivers/gpu/drm/i915/gt/intel_execlists_submission.c
432432 :doc: Logical Rings, Logical Ring Contexts and Execlists
433433
434434 Global GTT views
drivers/gpu/drm/i915/Makefile+5−1
@@ -38,6 +38,7 @@ i915-y += i915_drv.o \ · 37 unmodified lines
3838 i915_config.o \
3939 i915_irq.o \
4040 i915_getparam.o \
41+ i915_mitigations.o \
4142 i915_params.o \
4243 i915_pci.o \
4344 i915_scatterlist.o \
@@ -58,6 +59,7 @@ i915-y += i915_drv.o \ · 14 unmodified lines
5859
5960 # core library code
6061 i915-y += \
62+ dma_resv_utils.o \
6163 i915_memcpy.o \
6264 i915_mm.o \
6365 i915_sw_fence.o \
@@ -82,6 +84,7 @@ gt-y += \ · 18 unmodified lines
8284 gt/gen6_engine_cs.o \
8385 gt/gen6_ppgtt.o \
8486 gt/gen7_renderclear.o \
87+ gt/gen8_engine_cs.o \
8588 gt/gen8_ppgtt.o \
8689 gt/intel_breadcrumbs.o \
8790 gt/intel_context.o \
@@ -91,6 +94,7 @@ gt-y += \ · 3 unmodified lines
9194 gt/intel_engine_heartbeat.o \
9295 gt/intel_engine_pm.o \
9396 gt/intel_engine_user.o \
97+ gt/intel_execlists_submission.o \
9498 gt/intel_ggtt.o \
9599 gt/intel_ggtt_fencing.o \
96100 gt/intel_gt.o \
@@ -106,6 +110,7 @@ gt-y += \ · 9 unmodified lines
106110 gt/intel_mocs.o \
107111 gt/intel_ppgtt.o \
108112 gt/intel_rc6.o \
113+ gt/intel_region_lmem.o \
109114 gt/intel_renderstate.o \
110115 gt/intel_reset.o \
111116 gt/intel_ring.o \
@@ -166,7 +171,6 @@ i915-y += \ · 54 unmodified lines
166171 i915_scheduler.o \
167172 i915_trace_points.o \
168173 i915_vma.o \
169− intel_region_lmem.o \
170174 intel_wopcm.o
171175
172176 # general-purpose microcontroller (GuC) support
drivers/gpu/drm/i915/display/intel_overlay.c+1−0
@@ -29,6 +29,7 @@ · 28 unmodified lines
2929 #include <drm/drm_fourcc.h>
3030
3131 #include "gem/i915_gem_pm.h"
32+#include "gt/intel_gpu_commands.h"
3233 #include "gt/intel_ring.h"
3334
3435 #include "i915_drv.h"
drivers/gpu/drm/i915/dma_resv_utils.c+17−0
@@ -0,0 +1,17 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright © 2020 Intel Corporation
4+ */
5+
6+#include <linux/dma-resv.h>
7+
8+#include "dma_resv_utils.h"
9+
10+void dma_resv_prune(struct dma_resv *resv)
11+{
12+ if (dma_resv_trylock(resv)) {
13+ if (dma_resv_test_signaled_rcu(resv, true))
14+ dma_resv_add_excl_fence(resv, NULL);
15+ dma_resv_unlock(resv);
16+ }
17+}
drivers/gpu/drm/i915/dma_resv_utils.h+13−0
@@ -0,0 +1,13 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright © 2020 Intel Corporation
4+ */
5+
6+#ifndef DMA_RESV_UTILS_H
7+#define DMA_RESV_UTILS_H
8+
9+struct dma_resv;
10+
11+void dma_resv_prune(struct dma_resv *resv);
12+
13+#endif /* DMA_RESV_UTILS_H */
drivers/gpu/drm/i915/gem/i915_gem_context.c+13−54
@@ -72,6 +72,8 @@ · 71 unmodified lines
7272 #include "gt/intel_context_param.h"
7373 #include "gt/intel_engine_heartbeat.h"
7474 #include "gt/intel_engine_user.h"
75+#include "gt/intel_execlists_submission.h" /* virtual_engine */
76+#include "gt/intel_gpu_commands.h"
7577 #include "gt/intel_ring.h"
7678
7779 #include "i915_gem_context.h"
@@ -333,13 +335,12 @@ static struct i915_gem_engines *default_engines(struct i915_gem_context *ctx) · 255 unmodified lines
333335 return e;
334336 }
335337
336−static void i915_gem_context_free(struct i915_gem_context *ctx)
338+void i915_gem_context_release(struct kref *ref)
337339 {
338− GEM_BUG_ON(!i915_gem_context_is_closed(ctx));
340+ struct i915_gem_context *ctx = container_of(ref, typeof(*ctx), ref);
339341
340− spin_lock(&ctx->i915->gem.contexts.lock);
341− list_del(&ctx->link);
342− spin_unlock(&ctx->i915->gem.contexts.lock);
342+ trace_i915_context_free(ctx);
343+ GEM_BUG_ON(!i915_gem_context_is_closed(ctx));
343344
344345 mutex_destroy(&ctx->engines_mutex);
345346 mutex_destroy(&ctx->lut_mutex);
@@ -353,37 +354,6 @@ static void i915_gem_context_free(struct i915_gem_context *ctx) · 7 unmodified lines
353354 kfree_rcu(ctx, rcu);
354355 }
355356
356−static void contexts_free_all(struct llist_node *list)
357−{
358− struct i915_gem_context *ctx, *cn;
359−
360− llist_for_each_entry_safe(ctx, cn, list, free_link)
361− i915_gem_context_free(ctx);
362−}
363−
364−static void contexts_flush_free(struct i915_gem_contexts *gc)
365−{
366− contexts_free_all(llist_del_all(&gc->free_list));
367−}
368−
369−static void contexts_free_worker(struct work_struct *work)
370−{
371− struct i915_gem_contexts *gc =
372− container_of(work, typeof(*gc), free_work);
373−
374− contexts_flush_free(gc);
375−}
376−
377−void i915_gem_context_release(struct kref *ref)
378−{
379− struct i915_gem_context *ctx = container_of(ref, typeof(*ctx), ref);
380− struct i915_gem_contexts *gc = &ctx->i915->gem.contexts;
381−
382− trace_i915_context_free(ctx);
383− if (llist_add(&ctx->free_link, &gc->free_list))
384− schedule_work(&gc->free_work);
385−}
386−
387357 static inline struct i915_gem_engines *
388358 __context_engines_static(const struct i915_gem_context *ctx)
389359 {
@@ -453,6 +423,9 @@ static struct intel_engine_cs *active_engine(struct intel_context *ce) · 63 unmodified lines
453423 struct intel_engine_cs *engine = NULL;
454424 struct i915_request *rq;
455425
426+ if (intel_context_has_inflight(ce))
427+ return intel_context_inflight(ce);
428+
456429 if (!ce->timeline)
457430 return NULL;
458431
@@ -632,6 +605,10 @@ static void context_close(struct i915_gem_context *ctx) · 173 unmodified lines
632605 */
633606 lut_close(ctx);
634607
608+ spin_lock(&ctx->i915->gem.contexts.lock);
609+ list_del(&ctx->link);
610+ spin_unlock(&ctx->i915->gem.contexts.lock);
611+
635612 mutex_unlock(&ctx->mutex);
636613
637614 /*
@@ -849,9 +826,6 @@ i915_gem_create_context(struct drm_i915_private *i915, unsigned int flags) · 211 unmodified lines
849826 !HAS_EXECLISTS(i915))
850827 return ERR_PTR(-EINVAL);
851828
852− /* Reap the stale contexts */
853− contexts_flush_free(&i915->gem.contexts);
854−
855829 ctx = __create_context(i915);
856830 if (IS_ERR(ctx))
857831 return ctx;
@@ -896,23 +870,11 @@ static void init_contexts(struct i915_gem_contexts *gc) · 38 unmodified lines
896870 {
897871 spin_lock_init(&gc->lock);
898872 INIT_LIST_HEAD(&gc->list);
899−
900− INIT_WORK(&gc->free_work, contexts_free_worker);
901− init_llist_head(&gc->free_list);
902873 }
903874
904875 void i915_gem_init__contexts(struct drm_i915_private *i915)
905876 {
906877 init_contexts(&i915->gem.contexts);
907− drm_dbg(&i915->drm, "%s context support initialized\n",
908− DRIVER_CAPS(i915)->has_logical_contexts ?
909− "logical" : "fake");
910−}
911−
912−void i915_gem_driver_release__contexts(struct drm_i915_private *i915)
913−{
914− flush_work(&i915->gem.contexts.free_work);
915− rcu_barrier(); /* and flush the left over RCU frees */
916878 }
917879
918880 static int gem_context_register(struct i915_gem_context *ctx,
@@ -988,7 +950,6 @@ err: · 69 unmodified lines
988950 void i915_gem_context_close(struct drm_file *file)
989951 {
990952 struct drm_i915_file_private *file_priv = file->driver_priv;
991− struct drm_i915_private *i915 = file_priv->dev_priv;
992953 struct i915_address_space *vm;
993954 struct i915_gem_context *ctx;
994955 unsigned long idx;
@@ -1000,8 +961,6 @@ void i915_gem_context_close(struct drm_file *file) · 5 unmodified lines
1000961 xa_for_each(&file_priv->vm_xa, idx, vm)
1001962 i915_vm_put(vm);
1002963 xa_destroy(&file_priv->vm_xa);
1003−
1004− contexts_flush_free(&i915->gem.contexts);
1005964 }
1006965
1007966 int i915_gem_vm_create_ioctl(struct drm_device *dev, void *data,
drivers/gpu/drm/i915/gem/i915_gem_context.h+0−1
@@ -110,7 +110,6 @@ i915_gem_context_clear_user_engines(struct i915_gem_context *ctx) · 109 unmodified lines
110110
111111 /* i915_gem_context.c */
112112 void i915_gem_init__contexts(struct drm_i915_private *i915);
113−void i915_gem_driver_release__contexts(struct drm_i915_private *i915);
114113
115114 int i915_gem_context_open(struct drm_i915_private *i915,
116115 struct drm_file *file);
drivers/gpu/drm/i915/gem/i915_gem_context_types.h+0−1
@@ -108,7 +108,6 @@ struct i915_gem_context { · 107 unmodified lines
108108
109109 /** link: place with &drm_i915_private.context_list */
110110 struct list_head link;
111− struct llist_node free_link;
112111
113112 /**
114113 * @ref: reference count
drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c+8−7
@@ -15,6 +15,7 @@ · 14 unmodified lines
1515
1616 #include "gem/i915_gem_ioctls.h"
1717 #include "gt/intel_context.h"
18+#include "gt/intel_gpu_commands.h"
1819 #include "gt/intel_gt.h"
1920 #include "gt/intel_gt_buffer_pool.h"
2021 #include "gt/intel_gt_pm.h"
@@ -534,8 +535,6 @@ eb_add_vma(struct i915_execbuffer *eb, · 513 unmodified lines
534535 struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];
535536 struct eb_vma *ev = &eb->vma[i];
536537
537− GEM_BUG_ON(i915_vma_is_closed(vma));
538−
539538 ev->vma = vma;
540539 ev->exec = entry;
541540 ev->flags = entry->flags;
@@ -1046,7 +1045,7 @@ static void reloc_gpu_flush(struct i915_execbuffer *eb, struct reloc_cache *cach · 504 unmodified lines
10461045 GEM_BUG_ON(cache->rq_size >= obj->base.size / sizeof(u32));
10471046 cache->rq_cmd[cache->rq_size] = MI_BATCH_BUFFER_END;
10481047
1049− __i915_gem_object_flush_map(obj, 0, sizeof(u32) * (cache->rq_size + 1));
1048+ i915_gem_object_flush_map(obj);
10501049 i915_gem_object_unpin_map(obj);
10511050
10521051 intel_gt_chipset_flush(cache->rq->engine->gt);
@@ -1296,6 +1295,8 @@ static int __reloc_gpu_alloc(struct i915_execbuffer *eb, · 243 unmodified lines
12961295 goto err_pool;
12971296 }
12981297
1298+ memset32(cmd, 0, pool->obj->base.size / sizeof(u32));
1299+
12991300 batch = i915_vma_instance(pool->obj, vma->vm, NULL);
13001301 if (IS_ERR(batch)) {
13011302 err = PTR_ERR(batch);
@@ -2533,6 +2534,9 @@ static int eb_submit(struct i915_execbuffer *eb, struct i915_vma *batch) · 1231 unmodified lines
25332534 {
25342535 int err;
25352536
2537+ if (intel_context_nopreempt(eb->context))
2538+ __set_bit(I915_FENCE_FLAG_NOPREEMPT, &eb->request->fence.flags);
2539+
25362540 err = eb_move_to_gpu(eb);
25372541 if (err)
25382542 return err;
@@ -2573,15 +2577,12 @@ static int eb_submit(struct i915_execbuffer *eb, struct i915_vma *batch) · 34 unmodified lines
25732577 return err;
25742578 }
25752579
2576− if (intel_context_nopreempt(eb->context))
2577− __set_bit(I915_FENCE_FLAG_NOPREEMPT, &eb->request->fence.flags);
2578−
25792580 return 0;
25802581 }
25812582
25822583 static int num_vcs_engines(const struct drm_i915_private *i915)
25832584 {
2584− return hweight64(VDBOX_MASK(&i915->gt));
2585+ return hweight_long(VDBOX_MASK(&i915->gt));
25852586 }
25862587
25872588 /*
drivers/gpu/drm/i915/gem/i915_gem_object_blt.c+1−0
@@ -6,6 +6,7 @@ · 5 unmodified lines
66 #include "i915_drv.h"
77 #include "gt/intel_context.h"
88 #include "gt/intel_engine_pm.h"
9+#include "gt/intel_gpu_commands.h"
910 #include "gt/intel_gt.h"
1011 #include "gt/intel_gt_buffer_pool.h"
1112 #include "gt/intel_ring.h"
drivers/gpu/drm/i915/gem/i915_gem_pages.c+16−16
@@ -238,7 +238,7 @@ unlock: · 237 unmodified lines
238238
239239 /* The 'mapping' part of i915_gem_object_pin_map() below */
240240 static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj,
241− enum i915_map_type type)
241+ enum i915_map_type type)
242242 {
243243 unsigned long n_pages = obj->base.size >> PAGE_SHIFT, i;
244244 struct page *stack[32], **pages = stack, *page;
@@ -281,7 +281,7 @@ static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj, · 36 unmodified lines
281281 /* Too big for stack -- allocate temporary array instead */
282282 pages = kvmalloc_array(n_pages, sizeof(*pages), GFP_KERNEL);
283283 if (!pages)
284− return NULL;
284+ return ERR_PTR(-ENOMEM);
285285 }
286286
287287 i = 0;
@@ -290,11 +290,12 @@ static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj, · 2 unmodified lines
290290 vaddr = vmap(pages, n_pages, 0, pgprot);
291291 if (pages != stack)
292292 kvfree(pages);
293− return vaddr;
293+
294+ return vaddr ?: ERR_PTR(-ENOMEM);
294295 }
295296
296297 static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj,
297− enum i915_map_type type)
298+ enum i915_map_type type)
298299 {
299300 resource_size_t iomap = obj->mm.region->iomap.base -
300301 obj->mm.region->region.start;
@@ -305,13 +306,13 @@ static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj, · 4 unmodified lines
305306 void *vaddr;
306307
307308 if (type != I915_MAP_WC)
308− return NULL;
309+ return ERR_PTR(-ENODEV);
309310
310311 if (n_pfn > ARRAY_SIZE(stack)) {
311312 /* Too big for stack -- allocate temporary array instead */
312313 pfns = kvmalloc_array(n_pfn, sizeof(*pfns), GFP_KERNEL);
313314 if (!pfns)
314− return NULL;
315+ return ERR_PTR(-ENOMEM);
315316 }
316317
317318 i = 0;
@@ -320,7 +321,8 @@ static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj, · 2 unmodified lines
320321 vaddr = vmap_pfn(pfns, n_pfn, pgprot_writecombine(PAGE_KERNEL_IO));
321322 if (pfns != stack)
322323 kvfree(pfns);
323− return vaddr;
324+
325+ return vaddr ?: ERR_PTR(-ENOMEM);
324326 }
325327
326328 /* get, pin, and map the pages of the object into kernel space */
@@ -349,8 +351,10 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj, · 22 unmodified lines
349351 GEM_BUG_ON(i915_gem_object_has_pinned_pages(obj));
350352
351353 err = ____i915_gem_object_get_pages(obj);
352− if (err)
353− goto err_unlock;
354+ if (err) {
355+ ptr = ERR_PTR(err);
356+ goto out_unlock;
357+ }
354358
355359 smp_mb__before_atomic();
356360 }
@@ -362,7 +366,7 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj, · 5 unmodified lines
362366 ptr = page_unpack_bits(obj->mm.mapping, &has_type);
363367 if (ptr && has_type != type) {
364368 if (pinned) {
365− err = -EBUSY;
369+ ptr = ERR_PTR(-EBUSY);
366370 goto err_unpin;
367371 }
368372
@@ -374,15 +378,13 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj, · 5 unmodified lines
374378 if (!ptr) {
375379 if (GEM_WARN_ON(type == I915_MAP_WC &&
376380 !static_cpu_has(X86_FEATURE_PAT)))
377− ptr = NULL;
381+ ptr = ERR_PTR(-ENODEV);
378382 else if (i915_gem_object_has_struct_page(obj))
379383 ptr = i915_gem_object_map_page(obj, type);
380384 else
381385 ptr = i915_gem_object_map_pfn(obj, type);
382− if (!ptr) {
383− err = -ENOMEM;
386+ if (IS_ERR(ptr))
384387 goto err_unpin;
385− }
386388
387389 obj->mm.mapping = page_pack_bits(ptr, type);
388390 }
@@ -393,8 +395,6 @@ out_unlock: · 4 unmodified lines
393395
394396 err_unpin:
395397 atomic_dec(&obj->mm.pages_pin_count);
396−err_unlock:
397− ptr = ERR_PTR(err);
398398 goto out_unlock;
399399 }
400400
drivers/gpu/drm/i915/gem/i915_gem_region.c+20−16
@@ -22,6 +22,7 @@ i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj, · 21 unmodified lines
2222 int
2323 i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)
2424 {
25+ const u64 max_segment = i915_sg_segment_size();
2526 struct intel_memory_region *mem = obj->mm.region;
2627 struct list_head *blocks = &obj->mm.blocks;
2728 resource_size_t size = obj->base.size;
@@ -37,7 +38,7 @@ i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj) · 9 unmodified lines
3738 if (!st)
3839 return -ENOMEM;
3940
40− if (sg_alloc_table(st, size >> ilog2(mem->mm.chunk_size), GFP_KERNEL)) {
41+ if (sg_alloc_table(st, size >> PAGE_SHIFT, GFP_KERNEL)) {
4142 kfree(st);
4243 return -ENOMEM;
4344 }
@@ -64,27 +65,30 @@ i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj) · 20 unmodified lines
6465 i915_buddy_block_size(&mem->mm, block));
6566 offset = i915_buddy_block_offset(block);
6667
67− GEM_BUG_ON(overflows_type(block_size, sg->length));
68+ while (block_size) {
69+ u64 len;
6870
69− if (offset != prev_end ||
70− add_overflows_t(typeof(sg->length), sg->length, block_size)) {
71− if (st->nents) {
72− sg_page_sizes |= sg->length;
73− sg = __sg_next(sg);
71+ if (offset != prev_end || sg->length >= max_segment) {
72+ if (st->nents) {
73+ sg_page_sizes |= sg->length;
74+ sg = __sg_next(sg);
75+ }
76+
77+ sg_dma_address(sg) = mem->region.start + offset;
78+ sg_dma_len(sg) = 0;
79+ sg->length = 0;
80+ st->nents++;
7481 }
7582
76− sg_dma_address(sg) = mem->region.start + offset;
77− sg_dma_len(sg) = block_size;
83+ len = min(block_size, max_segment - sg->length);
84+ sg->length += len;
85+ sg_dma_len(sg) += len;
7886
79− sg->length = block_size;
87+ offset += len;
88+ block_size -= len;
8089
81− st->nents++;
82− } else {
83− sg->length += block_size;
84− sg_dma_len(sg) += block_size;
90+ prev_end = offset;
8591 }
86−
87− prev_end = offset + block_size;
8892 }
8993
9094 sg_page_sizes |= sg->length;
drivers/gpu/drm/i915/gem/i915_gem_shrinker.c+3−0
@@ -15,6 +15,7 @@ · 14 unmodified lines
1515
1616 #include "gt/intel_gt_requests.h"
1717
18+#include "dma_resv_utils.h"
1819 #include "i915_trace.h"
1920
2021 static bool swap_available(void)
@@ -209,6 +210,8 @@ i915_gem_shrink(struct drm_i915_private *i915, · 188 unmodified lines
209210 mutex_unlock(&obj->mm.lock);
210211 }
211212
213+ dma_resv_prune(obj->base.resv);
214+
212215 scanned += obj->base.size >> PAGE_SHIFT;
213216 i915_gem_object_put(obj);
214217
drivers/gpu/drm/i915/gem/i915_gem_stolen.c+2−3
@@ -608,11 +608,10 @@ i915_gem_object_release_stolen(struct drm_i915_gem_object *obj) · 607 unmodified lines
608608 struct drm_mm_node *stolen = fetch_and_zero(&obj->stolen);
609609
610610 GEM_BUG_ON(!stolen);
611−
612− i915_gem_object_release_memory_region(obj);
613−
614611 i915_gem_stolen_remove_node(i915, stolen);
615612 kfree(stolen);
613+
614+ i915_gem_object_release_memory_region(obj);
616615 }
617616
618617 static const struct drm_i915_gem_object_ops i915_gem_object_stolen_ops = {
drivers/gpu/drm/i915/gem/i915_gem_wait.c+3−5
@@ -9,6 +9,7 @@ · 8 unmodified lines
99
1010 #include "gt/intel_engine.h"
1111
12+#include "dma_resv_utils.h"
1213 #include "i915_gem_ioctls.h"
1314 #include "i915_gem_object.h"
1415
@@ -84,11 +85,8 @@ i915_gem_object_wait_reservation(struct dma_resv *resv, · 69 unmodified lines
8485 * Opportunistically prune the fences iff we know they have *all* been
8586 * signaled.
8687 */
87− if (prune_fences && dma_resv_trylock(resv)) {
88− if (dma_resv_test_signaled_rcu(resv, true))
89− dma_resv_add_excl_fence(resv, NULL);
90− dma_resv_unlock(resv);
91− }
88+ if (prune_fences)
89+ dma_resv_prune(resv);
9290
9391 return timeout;
9492 }
drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c+1−1
@@ -27,7 +27,7 @@ static void huge_free_pages(struct drm_i915_gem_object *obj, · 26 unmodified lines
2727
2828 static int huge_get_pages(struct drm_i915_gem_object *obj)
2929 {
30−#define GFP (GFP_KERNEL | __GFP_NOWARN | __GFP_NORETRY)
30+#define GFP (GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL)
3131 const unsigned long nreal = obj->scratch / PAGE_SIZE;
3232 const unsigned long npages = obj->base.size / PAGE_SIZE;
3333 struct scatterlist *sg, *src, *end;
drivers/gpu/drm/i915/gem/selftests/huge_pages.c+22−0
@@ -368,6 +368,27 @@ static int igt_check_page_sizes(struct i915_vma *vma) · 367 unmodified lines
368368 err = -EINVAL;
369369 }
370370
371+ /*
372+ * The dma-api is like a box of chocolates when it comes to the
373+ * alignment of dma addresses, however for LMEM we have total control
374+ * and so can guarantee alignment, likewise when we allocate our blocks
375+ * they should appear in descending order, and if we know that we align
376+ * to the largest page size for the GTT address, we should be able to
377+ * assert that if we see 2M physical pages then we should also get 2M
378+ * GTT pages. If we don't then something might be wrong in our
379+ * construction of the backing pages.
380+ *
381+ * Maintaining alignment is required to utilise huge pages in the ppGGT.
382+ */
383+ if (i915_gem_object_is_lmem(obj) &&
384+ IS_ALIGNED(vma->node.start, SZ_2M) &&
385+ vma->page_sizes.sg & SZ_2M &&
386+ vma->page_sizes.gtt < SZ_2M) {
387+ pr_err("gtt pages mismatch for LMEM, expected 2M GTT pages, sg(%u), gtt(%u)\n",
388+ vma->page_sizes.sg, vma->page_sizes.gtt);
389+ err = -EINVAL;
390+ }
391+
371392 if (obj->mm.page_sizes.gtt) {
372393 pr_err("obj->page_sizes.gtt(%u) should never be set\n",
373394 obj->mm.page_sizes.gtt);
@@ -1333,6 +1354,7 @@ static int igt_ppgtt_sanity_check(void *arg) · 959 unmodified lines
13331354 unsigned int flags;
13341355 } backends[] = {
13351356 { igt_create_system, 0, },
1357+ { igt_create_local, 0, },
13361358 { igt_create_local, I915_BO_ALLOC_CONTIGUOUS, },
13371359 };
13381360 struct {
drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c+1−3
@@ -20,13 +20,11 @@ static int __igt_client_fill(struct intel_engine_cs *engine) · 19 unmodified lines
2020 {
2121 struct intel_context *ce = engine->kernel_context;
2222 struct drm_i915_gem_object *obj;
23− struct rnd_state prng;
23+ I915_RND_STATE(prng);
2424 IGT_TIMEOUT(end);
2525 u32 *vaddr;
2626 int err = 0;
2727
28− prandom_seed_state(&prng, i915_selftest.random_seed);
29−
3028 intel_engine_pm_get(engine);
3129 do {
3230 const u32 max_block_size = S16_MAX * PAGE_SIZE;
drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
77 #include <linux/prime_numbers.h>
88
99 #include "gt/intel_engine_pm.h"
10+#include "gt/intel_gpu_commands.h"
1011 #include "gt/intel_gt.h"
1112 #include "gt/intel_gt_pm.h"
1213 #include "gt/intel_ring.h"
drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
77 #include <linux/prime_numbers.h>
88
99 #include "gt/intel_engine_pm.h"
10+#include "gt/intel_gpu_commands.h"
1011 #include "gt/intel_gt.h"
1112 #include "gt/intel_gt_pm.h"
1213 #include "gem/i915_gem_region.h"
drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c+1−0
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include "gem/i915_gem_context.h"
1010 #include "gem/i915_gem_pm.h"
1111 #include "gt/intel_context.h"
12+#include "gt/intel_gpu_commands.h"
1213 #include "gt/intel_gt.h"
1314 #include "i915_vma.h"
1415 #include "i915_drv.h"
drivers/gpu/drm/i915/gt/debugfs_gt_pm.c+15−12
@@ -11,6 +11,7 @@ · 10 unmodified lines
1111 #include "i915_drv.h"
1212 #include "intel_gt.h"
1313 #include "intel_gt_clock_utils.h"
14+#include "intel_gt_pm.h"
1415 #include "intel_llc.h"
1516 #include "intel_rc6.h"
1617 #include "intel_rps.h"
@@ -403,34 +404,34 @@ static int frequency_show(struct seq_file *m, void *unused) · 386 unmodified lines
403404 seq_printf(m, "RPDECLIMIT: 0x%08x\n", rpdeclimit);
404405 seq_printf(m, "RPNSWREQ: %dMHz\n", reqf);
405406 seq_printf(m, "CAGF: %dMHz\n", cagf);
406− seq_printf(m, "RP CUR UP EI: %d (%dns)\n",
407+ seq_printf(m, "RP CUR UP EI: %d (%lldns)\n",
407408 rpcurupei,
408409 intel_gt_pm_interval_to_ns(gt, rpcurupei));
409− seq_printf(m, "RP CUR UP: %d (%dns)\n",
410+ seq_printf(m, "RP CUR UP: %d (%lldns)\n",
410411 rpcurup, intel_gt_pm_interval_to_ns(gt, rpcurup));
411− seq_printf(m, "RP PREV UP: %d (%dns)\n",
412+ seq_printf(m, "RP PREV UP: %d (%lldns)\n",
412413 rpprevup, intel_gt_pm_interval_to_ns(gt, rpprevup));
413414 seq_printf(m, "Up threshold: %d%%\n",
414415 rps->power.up_threshold);
415− seq_printf(m, "RP UP EI: %d (%dns)\n",
416+ seq_printf(m, "RP UP EI: %d (%lldns)\n",
416417 rpupei, intel_gt_pm_interval_to_ns(gt, rpupei));
417− seq_printf(m, "RP UP THRESHOLD: %d (%dns)\n",
418+ seq_printf(m, "RP UP THRESHOLD: %d (%lldns)\n",
418419 rpupt, intel_gt_pm_interval_to_ns(gt, rpupt));
419420
420− seq_printf(m, "RP CUR DOWN EI: %d (%dns)\n",
421+ seq_printf(m, "RP CUR DOWN EI: %d (%lldns)\n",
421422 rpcurdownei,
422423 intel_gt_pm_interval_to_ns(gt, rpcurdownei));
423− seq_printf(m, "RP CUR DOWN: %d (%dns)\n",
424+ seq_printf(m, "RP CUR DOWN: %d (%lldns)\n",
424425 rpcurdown,
425426 intel_gt_pm_interval_to_ns(gt, rpcurdown));
426− seq_printf(m, "RP PREV DOWN: %d (%dns)\n",
427+ seq_printf(m, "RP PREV DOWN: %d (%lldns)\n",
427428 rpprevdown,
428429 intel_gt_pm_interval_to_ns(gt, rpprevdown));
429430 seq_printf(m, "Down threshold: %d%%\n",
430431 rps->power.down_threshold);
431− seq_printf(m, "RP DOWN EI: %d (%dns)\n",
432+ seq_printf(m, "RP DOWN EI: %d (%lldns)\n",
432433 rpdownei, intel_gt_pm_interval_to_ns(gt, rpdownei));
433− seq_printf(m, "RP DOWN THRESHOLD: %d (%dns)\n",
434+ seq_printf(m, "RP DOWN THRESHOLD: %d (%lldns)\n",
434435 rpdownt, intel_gt_pm_interval_to_ns(gt, rpdownt));
435436
436437 max_freq = (IS_GEN9_LP(i915) ? rp_state_cap >> 0 :
@@ -558,7 +559,9 @@ static int rps_boost_show(struct seq_file *m, void *data) · 121 unmodified lines
558559
559560 seq_printf(m, "RPS enabled? %s\n", yesno(intel_rps_is_enabled(rps)));
560561 seq_printf(m, "RPS active? %s\n", yesno(intel_rps_is_active(rps)));
561− seq_printf(m, "GPU busy? %s\n", yesno(gt->awake));
562+ seq_printf(m, "GPU busy? %s, %llums\n",
563+ yesno(gt->awake),
564+ ktime_to_ms(intel_gt_get_awake_time(gt)));
562565 seq_printf(m, "Boosts outstanding? %d\n",
563566 atomic_read(&rps->num_waiters));
564567 seq_printf(m, "Interactive? %d\n", READ_ONCE(rps->power.interactive));
@@ -575,7 +578,7 @@ static int rps_boost_show(struct seq_file *m, void *data) · 10 unmodified lines
575578 intel_gpu_freq(rps, rps->efficient_freq),
576579 intel_gpu_freq(rps, rps->boost_freq));
577580
578− seq_printf(m, "Wait boosts: %d\n", atomic_read(&rps->boosts));
581+ seq_printf(m, "Wait boosts: %d\n", READ_ONCE(rps->boosts));
579582
580583 if (INTEL_GEN(i915) >= 6 && intel_rps_is_active(rps)) {
581584 struct intel_uncore *uncore = gt->uncore;
drivers/gpu/drm/i915/gt/gen7_renderclear.c+94−63
@@ -7,8 +7,6 @@ · 6 unmodified lines
77 #include "i915_drv.h"
88 #include "intel_gpu_commands.h"
99
10−#define MAX_URB_ENTRIES 64
11−#define STATE_SIZE (4 * 1024)
1210 #define GT3_INLINE_DATA_DELAYS 0x1E00
1311 #define batch_advance(Y, CS) GEM_BUG_ON((Y)->end != (CS))
1412
@@ -34,38 +32,59 @@ struct batch_chunk { · 19 unmodified lines
3432 };
3533
3634 struct batch_vals {
37− u32 max_primitives;
38− u32 max_urb_entries;
39− u32 cmd_size;
40− u32 state_size;
35+ u32 max_threads;
4136 u32 state_start;
42− u32 batch_size;
37+ u32 surface_start;
4338 u32 surface_height;
4439 u32 surface_width;
45− u32 scratch_size;
46− u32 max_size;
40+ u32 size;
4741 };
4842
43+static inline int num_primitives(const struct batch_vals *bv)
44+{
45+ /*
46+ * We need to saturate the GPU with work in order to dispatch
47+ * a shader on every HW thread, and clear the thread-local registers.
48+ * In short, we have to dispatch work faster than the shaders can
49+ * run in order to fill the EU and occupy each HW thread.
50+ */
51+ return bv->max_threads;
52+}
53+
4954 static void
5055 batch_get_defaults(struct drm_i915_private *i915, struct batch_vals *bv)
5156 {
5257 if (IS_HASWELL(i915)) {
53− bv->max_primitives = 280;
54− bv->max_urb_entries = MAX_URB_ENTRIES;
58+ switch (INTEL_INFO(i915)->gt) {
59+ default:
60+ case 1:
61+ bv->max_threads = 70;
62+ break;
63+ case 2:
64+ bv->max_threads = 140;
65+ break;
66+ case 3:
67+ bv->max_threads = 280;
68+ break;
69+ }
5570 bv->surface_height = 16 * 16;
5671 bv->surface_width = 32 * 2 * 16;
5772 } else {
58− bv->max_primitives = 128;
59− bv->max_urb_entries = MAX_URB_ENTRIES / 2;
73+ switch (INTEL_INFO(i915)->gt) {
74+ default:
75+ case 1: /* including vlv */
76+ bv->max_threads = 36;
77+ break;
78+ case 2:
79+ bv->max_threads = 128;
80+ break;
81+ }
6082 bv->surface_height = 16 * 8;
6183 bv->surface_width = 32 * 16;
6284 }
63− bv->cmd_size = bv->max_primitives * 4096;
64− bv->state_size = STATE_SIZE;
65− bv->state_start = bv->cmd_size;
66− bv->batch_size = bv->cmd_size + bv->state_size;
67− bv->scratch_size = bv->surface_height * bv->surface_width;
68− bv->max_size = bv->batch_size + bv->scratch_size;
85+ bv->state_start = round_up(SZ_1K + num_primitives(bv) * 64, SZ_4K);
86+ bv->surface_start = bv->state_start + SZ_4K;
87+ bv->size = bv->surface_start + bv->surface_height * bv->surface_width;
6988 }
7089
7190 static void batch_init(struct batch_chunk *bc,
@@ -155,7 +174,8 @@ static u32 · 83 unmodified lines
155174 gen7_fill_binding_table(struct batch_chunk *state,
156175 const struct batch_vals *bv)
157176 {
158− u32 surface_start = gen7_fill_surface_state(state, bv->batch_size, bv);
177+ u32 surface_start =
178+ gen7_fill_surface_state(state, bv->surface_start, bv);
159179 u32 *cs = batch_alloc_items(state, 32, 8);
160180 u32 offset = batch_offset(state, cs);
161181
@@ -214,9 +234,9 @@ static void · 52 unmodified lines
214234 gen7_emit_state_base_address(struct batch_chunk *batch,
215235 u32 surface_state_base)
216236 {
217− u32 *cs = batch_alloc_items(batch, 0, 12);
237+ u32 *cs = batch_alloc_items(batch, 0, 10);
218238
219− *cs++ = STATE_BASE_ADDRESS | (12 - 2);
239+ *cs++ = STATE_BASE_ADDRESS | (10 - 2);
220240 /* general */
221241 *cs++ = batch_addr(batch) | BASE_ADDRESS_MODIFY;
222242 /* surface */
@@ -233,8 +253,6 @@ gen7_emit_state_base_address(struct batch_chunk *batch, · 10 unmodified lines
233253 *cs++ = BASE_ADDRESS_MODIFY;
234254 *cs++ = 0;
235255 *cs++ = BASE_ADDRESS_MODIFY;
236− *cs++ = 0;
237− *cs++ = 0;
238256 batch_advance(batch, cs);
239257 }
240258
@@ -244,8 +262,7 @@ gen7_emit_vfe_state(struct batch_chunk *batch, · 3 unmodified lines
244262 u32 urb_size, u32 curbe_size,
245263 u32 mode)
246264 {
247− u32 urb_entries = bv->max_urb_entries;
248− u32 threads = bv->max_primitives - 1;
265+ u32 threads = bv->max_threads - 1;
249266 u32 *cs = batch_alloc_items(batch, 32, 8);
250267
251268 *cs++ = MEDIA_VFE_STATE | (8 - 2);
@@ -254,7 +271,7 @@ gen7_emit_vfe_state(struct batch_chunk *batch, · 2 unmodified lines
254271 *cs++ = 0;
255272
256273 /* number of threads & urb entries for GPGPU vs Media Mode */
257− *cs++ = threads << 16 | urb_entries << 8 | mode << 2;
274+ *cs++ = threads << 16 | 1 << 8 | mode << 2;
258275
259276 *cs++ = 0;
260277
@@ -293,17 +310,12 @@ gen7_emit_media_object(struct batch_chunk *batch, · 32 unmodified lines
293310 {
294311 unsigned int x_offset = (media_object_index % 16) * 64;
295312 unsigned int y_offset = (media_object_index / 16) * 16;
296− unsigned int inline_data_size;
297− unsigned int media_batch_size;
298− unsigned int i;
313+ unsigned int pkt = 6 + 3;
299314 u32 *cs;
300315
301− inline_data_size = 112 * 8;
302− media_batch_size = inline_data_size + 6;
303−
304− cs = batch_alloc_items(batch, 8, media_batch_size);
316+ cs = batch_alloc_items(batch, 8, pkt);
305317
306− *cs++ = MEDIA_OBJECT | (media_batch_size - 2);
318+ *cs++ = MEDIA_OBJECT | (pkt - 2);
307319
308320 /* interface descriptor offset */
309321 *cs++ = 0;
@@ -317,25 +329,44 @@ gen7_emit_media_object(struct batch_chunk *batch, · 7 unmodified lines
317329 *cs++ = 0;
318330
319331 /* inline */
320− *cs++ = (y_offset << 16) | (x_offset);
332+ *cs++ = y_offset << 16 | x_offset;
321333 *cs++ = 0;
322334 *cs++ = GT3_INLINE_DATA_DELAYS;
323− for (i = 3; i < inline_data_size; i++)
324− *cs++ = 0;
325335
326336 batch_advance(batch, cs);
327337 }
328338
329339 static void gen7_emit_pipeline_flush(struct batch_chunk *batch)
330340 {
331− u32 *cs = batch_alloc_items(batch, 0, 5);
341+ u32 *cs = batch_alloc_items(batch, 0, 4);
332342
333− *cs++ = GFX_OP_PIPE_CONTROL(5);
334− *cs++ = PIPE_CONTROL_STATE_CACHE_INVALIDATE |
335− PIPE_CONTROL_GLOBAL_GTT_IVB;
343+ *cs++ = GFX_OP_PIPE_CONTROL(4);
344+ *cs++ = PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
345+ PIPE_CONTROL_DEPTH_CACHE_FLUSH |
346+ PIPE_CONTROL_DC_FLUSH_ENABLE |
347+ PIPE_CONTROL_CS_STALL;
336348 *cs++ = 0;
337349 *cs++ = 0;
350+
351+ batch_advance(batch, cs);
352+}
353+
354+static void gen7_emit_pipeline_invalidate(struct batch_chunk *batch)
355+{
356+ u32 *cs = batch_alloc_items(batch, 0, 8);
357+
358+ /* ivb: Stall before STATE_CACHE_INVALIDATE */
359+ *cs++ = GFX_OP_PIPE_CONTROL(4);
360+ *cs++ = PIPE_CONTROL_STALL_AT_SCOREBOARD |
361+ PIPE_CONTROL_CS_STALL;
362+ *cs++ = 0;
363+ *cs++ = 0;
364+
365+ *cs++ = GFX_OP_PIPE_CONTROL(4);
366+ *cs++ = PIPE_CONTROL_STATE_CACHE_INVALIDATE;
338367 *cs++ = 0;
368+ *cs++ = 0;
369+
339370 batch_advance(batch, cs);
340371 }
341372
@@ -344,34 +375,34 @@ static void emit_batch(struct i915_vma * const vma, · 2 unmodified lines
344375 const struct batch_vals *bv)
345376 {
346377 struct drm_i915_private *i915 = vma->vm->i915;
347− unsigned int desc_count = 64;
348− const u32 urb_size = 112;
378+ const unsigned int desc_count = 1;
379+ const unsigned int urb_size = 1;
349380 struct batch_chunk cmds, state;
350− u32 interface_descriptor;
381+ u32 descriptors;
351382 unsigned int i;
352383
353− batch_init(&cmds, vma, start, 0, bv->cmd_size);
354− batch_init(&state, vma, start, bv->state_start, bv->state_size);
384+ batch_init(&cmds, vma, start, 0, bv->state_start);
385+ batch_init(&state, vma, start, bv->state_start, SZ_4K);
355386
356− interface_descriptor =
357− gen7_fill_interface_descriptor(&state, bv,
358− IS_HASWELL(i915) ?
359− &cb_kernel_hsw :
360− &cb_kernel_ivb,
361− desc_count);
362− gen7_emit_pipeline_flush(&cmds);
387+ descriptors = gen7_fill_interface_descriptor(&state, bv,
388+ IS_HASWELL(i915) ?
389+ &cb_kernel_hsw :
390+ &cb_kernel_ivb,
391+ desc_count);
392+
393+ gen7_emit_pipeline_invalidate(&cmds);
363394 batch_add(&cmds, PIPELINE_SELECT | PIPELINE_SELECT_MEDIA);
364395 batch_add(&cmds, MI_NOOP);
365− gen7_emit_state_base_address(&cmds, interface_descriptor);
396+ gen7_emit_pipeline_invalidate(&cmds);
397+
366398 gen7_emit_pipeline_flush(&cmds);
399+ gen7_emit_state_base_address(&cmds, descriptors);
400+ gen7_emit_pipeline_invalidate(&cmds);
367401
368402 gen7_emit_vfe_state(&cmds, bv, urb_size - 1, 0, 0);
403+ gen7_emit_interface_descriptor_load(&cmds, descriptors, desc_count);
369404
370− gen7_emit_interface_descriptor_load(&cmds,
371− interface_descriptor,
372− desc_count);
373−
374− for (i = 0; i < bv->max_primitives; i++)
405+ for (i = 0; i < num_primitives(bv); i++)
375406 gen7_emit_media_object(&cmds, i);
376407
377408 batch_add(&cmds, MI_BATCH_BUFFER_END);
@@ -385,15 +416,15 @@ int gen7_setup_clear_gpr_bb(struct intel_engine_cs * const engine, · 7 unmodified lines
385416
386417 batch_get_defaults(engine->i915, &bv);
387418 if (!vma)
388− return bv.max_size;
419+ return bv.size;
389420
390− GEM_BUG_ON(vma->obj->base.size < bv.max_size);
421+ GEM_BUG_ON(vma->obj->base.size < bv.size);
391422
392423 batch = i915_gem_object_pin_map(vma->obj, I915_MAP_WC);
393424 if (IS_ERR(batch))
394425 return PTR_ERR(batch);
395426
396− emit_batch(vma, memset(batch, 0, bv.max_size), &bv);
427+ emit_batch(vma, memset(batch, 0, bv.size), &bv);
397428
398429 i915_gem_object_flush_map(vma->obj);
399430 __i915_gem_object_release_map(vma->obj);
drivers/gpu/drm/i915/gt/gen8_engine_cs.c+633−0
@@ -0,0 +1,633 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright © 2014 Intel Corporation
4+ */
5+
6+#include "gen8_engine_cs.h"
7+#include "i915_drv.h"
8+#include "intel_lrc.h"
9+#include "intel_gpu_commands.h"
10+#include "intel_ring.h"
11+
12+int gen8_emit_flush_rcs(struct i915_request *rq, u32 mode)
13+{
14+ bool vf_flush_wa = false, dc_flush_wa = false;
15+ u32 *cs, flags = 0;
16+ int len;
17+
18+ flags |= PIPE_CONTROL_CS_STALL;
19+
20+ if (mode & EMIT_FLUSH) {
21+ flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
22+ flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
23+ flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
24+ flags |= PIPE_CONTROL_FLUSH_ENABLE;
25+ }
26+
27+ if (mode & EMIT_INVALIDATE) {
28+ flags |= PIPE_CONTROL_TLB_INVALIDATE;
29+ flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
30+ flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
31+ flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
32+ flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
33+ flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
34+ flags |= PIPE_CONTROL_QW_WRITE;
35+ flags |= PIPE_CONTROL_STORE_DATA_INDEX;
36+
37+ /*
38+ * On GEN9: before VF_CACHE_INVALIDATE we need to emit a NULL
39+ * pipe control.
40+ */
41+ if (IS_GEN(rq->engine->i915, 9))
42+ vf_flush_wa = true;
43+
44+ /* WaForGAMHang:kbl */
45+ if (IS_KBL_GT_REVID(rq->engine->i915, 0, KBL_REVID_B0))
46+ dc_flush_wa = true;
47+ }
48+
49+ len = 6;
50+
51+ if (vf_flush_wa)
52+ len += 6;
53+
54+ if (dc_flush_wa)
55+ len += 12;
56+
57+ cs = intel_ring_begin(rq, len);
58+ if (IS_ERR(cs))
59+ return PTR_ERR(cs);
60+
61+ if (vf_flush_wa)
62+ cs = gen8_emit_pipe_control(cs, 0, 0);
63+
64+ if (dc_flush_wa)
65+ cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_DC_FLUSH_ENABLE,
66+ 0);
67+
68+ cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
69+
70+ if (dc_flush_wa)
71+ cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_CS_STALL, 0);
72+
73+ intel_ring_advance(rq, cs);
74+
75+ return 0;
76+}
77+
78+int gen8_emit_flush_xcs(struct i915_request *rq, u32 mode)
79+{
80+ u32 cmd, *cs;
81+
82+ cs = intel_ring_begin(rq, 4);
83+ if (IS_ERR(cs))
84+ return PTR_ERR(cs);
85+
86+ cmd = MI_FLUSH_DW + 1;
87+
88+ /*
89+ * We always require a command barrier so that subsequent
90+ * commands, such as breadcrumb interrupts, are strictly ordered
91+ * wrt the contents of the write cache being flushed to memory
92+ * (and thus being coherent from the CPU).
93+ */
94+ cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
95+
96+ if (mode & EMIT_INVALIDATE) {
97+ cmd |= MI_INVALIDATE_TLB;
98+ if (rq->engine->class == VIDEO_DECODE_CLASS)
99+ cmd |= MI_INVALIDATE_BSD;
100+ }
101+
102+ *cs++ = cmd;
103+ *cs++ = LRC_PPHWSP_SCRATCH_ADDR;
104+ *cs++ = 0; /* upper addr */
105+ *cs++ = 0; /* value */
106+ intel_ring_advance(rq, cs);
107+
108+ return 0;
109+}
110+
111+int gen11_emit_flush_rcs(struct i915_request *rq, u32 mode)
112+{
113+ if (mode & EMIT_FLUSH) {
114+ u32 *cs;
115+ u32 flags = 0;
116+
117+ flags |= PIPE_CONTROL_CS_STALL;
118+
119+ flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
120+ flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
121+ flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
122+ flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
123+ flags |= PIPE_CONTROL_FLUSH_ENABLE;
124+ flags |= PIPE_CONTROL_QW_WRITE;
125+ flags |= PIPE_CONTROL_STORE_DATA_INDEX;
126+
127+ cs = intel_ring_begin(rq, 6);
128+ if (IS_ERR(cs))
129+ return PTR_ERR(cs);
130+
131+ cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
132+ intel_ring_advance(rq, cs);
133+ }
134+
135+ if (mode & EMIT_INVALIDATE) {
136+ u32 *cs;
137+ u32 flags = 0;
138+
139+ flags |= PIPE_CONTROL_CS_STALL;
140+
141+ flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
142+ flags |= PIPE_CONTROL_TLB_INVALIDATE;
143+ flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
144+ flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
145+ flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
146+ flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
147+ flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
148+ flags |= PIPE_CONTROL_QW_WRITE;
149+ flags |= PIPE_CONTROL_STORE_DATA_INDEX;
150+
151+ cs = intel_ring_begin(rq, 6);
152+ if (IS_ERR(cs))
153+ return PTR_ERR(cs);
154+
155+ cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
156+ intel_ring_advance(rq, cs);
157+ }
158+
159+ return 0;
160+}
161+
162+static u32 preparser_disable(bool state)
163+{
164+ return MI_ARB_CHECK | 1 << 8 | state;
165+}
166+
167+static i915_reg_t aux_inv_reg(const struct intel_engine_cs *engine)
168+{
169+ static const i915_reg_t vd[] = {
170+ GEN12_VD0_AUX_NV,
171+ GEN12_VD1_AUX_NV,
172+ GEN12_VD2_AUX_NV,
173+ GEN12_VD3_AUX_NV,
174+ };
175+
176+ static const i915_reg_t ve[] = {
177+ GEN12_VE0_AUX_NV,
178+ GEN12_VE1_AUX_NV,
179+ };
180+
181+ if (engine->class == VIDEO_DECODE_CLASS)
182+ return vd[engine->instance];
183+
184+ if (engine->class == VIDEO_ENHANCEMENT_CLASS)
185+ return ve[engine->instance];
186+
187+ GEM_BUG_ON("unknown aux_inv reg\n");
188+ return INVALID_MMIO_REG;
189+}
190+
191+static u32 *gen12_emit_aux_table_inv(const i915_reg_t inv_reg, u32 *cs)
192+{
193+ *cs++ = MI_LOAD_REGISTER_IMM(1);
194+ *cs++ = i915_mmio_reg_offset(inv_reg);
195+ *cs++ = AUX_INV;
196+ *cs++ = MI_NOOP;
197+
198+ return cs;
199+}
200+
201+int gen12_emit_flush_rcs(struct i915_request *rq, u32 mode)
202+{
203+ if (mode & EMIT_FLUSH) {
204+ u32 flags = 0;
205+ u32 *cs;
206+
207+ flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
208+ flags |= PIPE_CONTROL_FLUSH_L3;
209+ flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
210+ flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
211+ /* Wa_1409600907:tgl */
212+ flags |= PIPE_CONTROL_DEPTH_STALL;
213+ flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
214+ flags |= PIPE_CONTROL_FLUSH_ENABLE;
215+
216+ flags |= PIPE_CONTROL_STORE_DATA_INDEX;
217+ flags |= PIPE_CONTROL_QW_WRITE;
218+
219+ flags |= PIPE_CONTROL_CS_STALL;
220+
221+ cs = intel_ring_begin(rq, 6);
222+ if (IS_ERR(cs))
223+ return PTR_ERR(cs);
224+
225+ cs = gen12_emit_pipe_control(cs,
226+ PIPE_CONTROL0_HDC_PIPELINE_FLUSH,
227+ flags, LRC_PPHWSP_SCRATCH_ADDR);
228+ intel_ring_advance(rq, cs);
229+ }
230+
231+ if (mode & EMIT_INVALIDATE) {
232+ u32 flags = 0;
233+ u32 *cs;
234+
235+ flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
236+ flags |= PIPE_CONTROL_TLB_INVALIDATE;
237+ flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
238+ flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
239+ flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
240+ flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
241+ flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
242+
243+ flags |= PIPE_CONTROL_STORE_DATA_INDEX;
244+ flags |= PIPE_CONTROL_QW_WRITE;
245+
246+ flags |= PIPE_CONTROL_CS_STALL;
247+
248+ cs = intel_ring_begin(rq, 8 + 4);
249+ if (IS_ERR(cs))
250+ return PTR_ERR(cs);
251+
252+ /*
253+ * Prevent the pre-parser from skipping past the TLB
254+ * invalidate and loading a stale page for the batch
255+ * buffer / request payload.
256+ */
257+ *cs++ = preparser_disable(true);
258+
259+ cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
260+
261+ /* hsdes: 1809175790 */
262+ cs = gen12_emit_aux_table_inv(GEN12_GFX_CCS_AUX_NV, cs);
263+
264+ *cs++ = preparser_disable(false);
265+ intel_ring_advance(rq, cs);
266+ }
267+
268+ return 0;
269+}
270+
271+int gen12_emit_flush_xcs(struct i915_request *rq, u32 mode)
272+{
273+ intel_engine_mask_t aux_inv = 0;
274+ u32 cmd, *cs;
275+
276+ cmd = 4;
277+ if (mode & EMIT_INVALIDATE)
278+ cmd += 2;
279+ if (mode & EMIT_INVALIDATE)
280+ aux_inv = rq->engine->mask & ~BIT(BCS0);
281+ if (aux_inv)
282+ cmd += 2 * hweight8(aux_inv) + 2;
283+
284+ cs = intel_ring_begin(rq, cmd);
285+ if (IS_ERR(cs))
286+ return PTR_ERR(cs);
287+
288+ if (mode & EMIT_INVALIDATE)
289+ *cs++ = preparser_disable(true);
290+
291+ cmd = MI_FLUSH_DW + 1;
292+
293+ /*
294+ * We always require a command barrier so that subsequent
295+ * commands, such as breadcrumb interrupts, are strictly ordered
296+ * wrt the contents of the write cache being flushed to memory
297+ * (and thus being coherent from the CPU).
298+ */
299+ cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
300+
301+ if (mode & EMIT_INVALIDATE) {
302+ cmd |= MI_INVALIDATE_TLB;
303+ if (rq->engine->class == VIDEO_DECODE_CLASS)
304+ cmd |= MI_INVALIDATE_BSD;
305+ }
306+
307+ *cs++ = cmd;
308+ *cs++ = LRC_PPHWSP_SCRATCH_ADDR;
309+ *cs++ = 0; /* upper addr */
310+ *cs++ = 0; /* value */
311+
312+ if (aux_inv) { /* hsdes: 1809175790 */
313+ struct intel_engine_cs *engine;
314+ unsigned int tmp;
315+
316+ *cs++ = MI_LOAD_REGISTER_IMM(hweight8(aux_inv));
317+ for_each_engine_masked(engine, rq->engine->gt,
318+ aux_inv, tmp) {
319+ *cs++ = i915_mmio_reg_offset(aux_inv_reg(engine));
320+ *cs++ = AUX_INV;
321+ }
322+ *cs++ = MI_NOOP;
323+ }
324+
325+ if (mode & EMIT_INVALIDATE)
326+ *cs++ = preparser_disable(false);
327+
328+ intel_ring_advance(rq, cs);
329+
330+ return 0;
331+}
332+
333+static inline u32 preempt_address(struct intel_engine_cs *engine)
334+{
335+ return (i915_ggtt_offset(engine->status_page.vma) +
336+ I915_GEM_HWS_PREEMPT_ADDR);
337+}
338+
339+static u32 hwsp_offset(const struct i915_request *rq)
340+{
341+ const struct intel_timeline_cacheline *cl;
342+
343+ /* Before the request is executed, the timeline/cachline is fixed */
344+
345+ cl = rcu_dereference_protected(rq->hwsp_cacheline, 1);
346+ if (cl)
347+ return cl->ggtt_offset;
348+
349+ return rcu_dereference_protected(rq->timeline, 1)->hwsp_offset;
350+}
351+
352+int gen8_emit_init_breadcrumb(struct i915_request *rq)
353+{
354+ u32 *cs;
355+
356+ GEM_BUG_ON(i915_request_has_initial_breadcrumb(rq));
357+ if (!i915_request_timeline(rq)->has_initial_breadcrumb)
358+ return 0;
359+
360+ cs = intel_ring_begin(rq, 6);
361+ if (IS_ERR(cs))
362+ return PTR_ERR(cs);
363+
364+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
365+ *cs++ = hwsp_offset(rq);
366+ *cs++ = 0;
367+ *cs++ = rq->fence.seqno - 1;
368+
369+ /*
370+ * Check if we have been preempted before we even get started.
371+ *
372+ * After this point i915_request_started() reports true, even if
373+ * we get preempted and so are no longer running.
374+ *
375+ * i915_request_started() is used during preemption processing
376+ * to decide if the request is currently inside the user payload
377+ * or spinning on a kernel semaphore (or earlier). For no-preemption
378+ * requests, we do allow preemption on the semaphore before the user
379+ * payload, but do not allow preemption once the request is started.
380+ *
381+ * i915_request_started() is similarly used during GPU hangs to
382+ * determine if the user's payload was guilty, and if so, the
383+ * request is banned. Before the request is started, it is assumed
384+ * to be unharmed and an innocent victim of another's hang.
385+ */
386+ *cs++ = MI_NOOP;
387+ *cs++ = MI_ARB_CHECK;
388+
389+ intel_ring_advance(rq, cs);
390+
391+ /* Record the updated position of the request's payload */
392+ rq->infix = intel_ring_offset(rq, cs);
393+
394+ __set_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags);
395+
396+ return 0;
397+}
398+
399+int gen8_emit_bb_start_noarb(struct i915_request *rq,
400+ u64 offset, u32 len,
401+ const unsigned int flags)
402+{
403+ u32 *cs;
404+
405+ cs = intel_ring_begin(rq, 4);
406+ if (IS_ERR(cs))
407+ return PTR_ERR(cs);
408+
409+ /*
410+ * WaDisableCtxRestoreArbitration:bdw,chv
411+ *
412+ * We don't need to perform MI_ARB_ENABLE as often as we do (in
413+ * particular all the gen that do not need the w/a at all!), if we
414+ * took care to make sure that on every switch into this context
415+ * (both ordinary and for preemption) that arbitrartion was enabled
416+ * we would be fine. However, for gen8 there is another w/a that
417+ * requires us to not preempt inside GPGPU execution, so we keep
418+ * arbitration disabled for gen8 batches. Arbitration will be
419+ * re-enabled before we close the request
420+ * (engine->emit_fini_breadcrumb).
421+ */
422+ *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
423+
424+ /* FIXME(BDW+): Address space and security selectors. */
425+ *cs++ = MI_BATCH_BUFFER_START_GEN8 |
426+ (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
427+ *cs++ = lower_32_bits(offset);
428+ *cs++ = upper_32_bits(offset);
429+
430+ intel_ring_advance(rq, cs);
431+
432+ return 0;
433+}
434+
435+int gen8_emit_bb_start(struct i915_request *rq,
436+ u64 offset, u32 len,
437+ const unsigned int flags)
438+{
439+ u32 *cs;
440+
441+ if (unlikely(i915_request_has_nopreempt(rq)))
442+ return gen8_emit_bb_start_noarb(rq, offset, len, flags);
443+
444+ cs = intel_ring_begin(rq, 6);
445+ if (IS_ERR(cs))
446+ return PTR_ERR(cs);
447+
448+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
449+
450+ *cs++ = MI_BATCH_BUFFER_START_GEN8 |
451+ (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
452+ *cs++ = lower_32_bits(offset);
453+ *cs++ = upper_32_bits(offset);
454+
455+ *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
456+ *cs++ = MI_NOOP;
457+
458+ intel_ring_advance(rq, cs);
459+
460+ return 0;
461+}
462+
463+static void assert_request_valid(struct i915_request *rq)
464+{
465+ struct intel_ring *ring __maybe_unused = rq->ring;
466+
467+ /* Can we unwind this request without appearing to go forwards? */
468+ GEM_BUG_ON(intel_ring_direction(ring, rq->wa_tail, rq->head) <= 0);
469+}
470+
471+/*
472+ * Reserve space for 2 NOOPs at the end of each request to be
473+ * used as a workaround for not being allowed to do lite
474+ * restore with HEAD==TAIL (WaIdleLiteRestore).
475+ */
476+static u32 *gen8_emit_wa_tail(struct i915_request *rq, u32 *cs)
477+{
478+ /* Ensure there's always at least one preemption point per-request. */
479+ *cs++ = MI_ARB_CHECK;
480+ *cs++ = MI_NOOP;
481+ rq->wa_tail = intel_ring_offset(rq, cs);
482+
483+ /* Check that entire request is less than half the ring */
484+ assert_request_valid(rq);
485+
486+ return cs;
487+}
488+
489+static u32 *emit_preempt_busywait(struct i915_request *rq, u32 *cs)
490+{
491+ *cs++ = MI_SEMAPHORE_WAIT |
492+ MI_SEMAPHORE_GLOBAL_GTT |
493+ MI_SEMAPHORE_POLL |
494+ MI_SEMAPHORE_SAD_EQ_SDD;
495+ *cs++ = 0;
496+ *cs++ = preempt_address(rq->engine);
497+ *cs++ = 0;
498+
499+ return cs;
500+}
501+
502+static __always_inline u32*
503+gen8_emit_fini_breadcrumb_tail(struct i915_request *rq, u32 *cs)
504+{
505+ *cs++ = MI_USER_INTERRUPT;
506+
507+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
508+ if (intel_engine_has_semaphores(rq->engine))
509+ cs = emit_preempt_busywait(rq, cs);
510+
511+ rq->tail = intel_ring_offset(rq, cs);
512+ assert_ring_tail_valid(rq->ring, rq->tail);
513+
514+ return gen8_emit_wa_tail(rq, cs);
515+}
516+
517+static u32 *emit_xcs_breadcrumb(struct i915_request *rq, u32 *cs)
518+{
519+ return gen8_emit_ggtt_write(cs, rq->fence.seqno, hwsp_offset(rq), 0);
520+}
521+
522+u32 *gen8_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs)
523+{
524+ return gen8_emit_fini_breadcrumb_tail(rq, emit_xcs_breadcrumb(rq, cs));
525+}
526+
527+u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs)
528+{
529+ cs = gen8_emit_pipe_control(cs,
530+ PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
531+ PIPE_CONTROL_DEPTH_CACHE_FLUSH |
532+ PIPE_CONTROL_DC_FLUSH_ENABLE,
533+ 0);
534+
535+ /* XXX flush+write+CS_STALL all in one upsets gem_concurrent_blt:kbl */
536+ cs = gen8_emit_ggtt_write_rcs(cs,
537+ rq->fence.seqno,
538+ hwsp_offset(rq),
539+ PIPE_CONTROL_FLUSH_ENABLE |
540+ PIPE_CONTROL_CS_STALL);
541+
542+ return gen8_emit_fini_breadcrumb_tail(rq, cs);
543+}
544+
545+u32 *gen11_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs)
546+{
547+ cs = gen8_emit_ggtt_write_rcs(cs,
548+ rq->fence.seqno,
549+ hwsp_offset(rq),
550+ PIPE_CONTROL_CS_STALL |
551+ PIPE_CONTROL_TILE_CACHE_FLUSH |
552+ PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
553+ PIPE_CONTROL_DEPTH_CACHE_FLUSH |
554+ PIPE_CONTROL_DC_FLUSH_ENABLE |
555+ PIPE_CONTROL_FLUSH_ENABLE);
556+
557+ return gen8_emit_fini_breadcrumb_tail(rq, cs);
558+}
559+
560+/*
561+ * Note that the CS instruction pre-parser will not stall on the breadcrumb
562+ * flush and will continue pre-fetching the instructions after it before the
563+ * memory sync is completed. On pre-gen12 HW, the pre-parser will stop at
564+ * BB_START/END instructions, so, even though we might pre-fetch the pre-amble
565+ * of the next request before the memory has been flushed, we're guaranteed that
566+ * we won't access the batch itself too early.
567+ * However, on gen12+ the parser can pre-fetch across the BB_START/END commands,
568+ * so, if the current request is modifying an instruction in the next request on
569+ * the same intel_context, we might pre-fetch and then execute the pre-update
570+ * instruction. To avoid this, the users of self-modifying code should either
571+ * disable the parser around the code emitting the memory writes, via a new flag
572+ * added to MI_ARB_CHECK, or emit the writes from a different intel_context. For
573+ * the in-kernel use-cases we've opted to use a separate context, see
574+ * reloc_gpu() as an example.
575+ * All the above applies only to the instructions themselves. Non-inline data
576+ * used by the instructions is not pre-fetched.
577+ */
578+
579+static u32 *gen12_emit_preempt_busywait(struct i915_request *rq, u32 *cs)
580+{
581+ *cs++ = MI_ARB_CHECK; /* trigger IDLE->ACTIVE first */
582+ *cs++ = MI_SEMAPHORE_WAIT_TOKEN |
583+ MI_SEMAPHORE_GLOBAL_GTT |
584+ MI_SEMAPHORE_POLL |
585+ MI_SEMAPHORE_SAD_EQ_SDD;
586+ *cs++ = 0;
587+ *cs++ = preempt_address(rq->engine);
588+ *cs++ = 0;
589+ *cs++ = 0;
590+
591+ return cs;
592+}
593+
594+static __always_inline u32*
595+gen12_emit_fini_breadcrumb_tail(struct i915_request *rq, u32 *cs)
596+{
597+ *cs++ = MI_USER_INTERRUPT;
598+
599+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
600+ if (intel_engine_has_semaphores(rq->engine))
601+ cs = gen12_emit_preempt_busywait(rq, cs);
602+
603+ rq->tail = intel_ring_offset(rq, cs);
604+ assert_ring_tail_valid(rq->ring, rq->tail);
605+
606+ return gen8_emit_wa_tail(rq, cs);
607+}
608+
609+u32 *gen12_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs)
610+{
611+ /* XXX Stalling flush before seqno write; post-sync not */
612+ cs = emit_xcs_breadcrumb(rq, __gen8_emit_flush_dw(cs, 0, 0, 0));
613+ return gen12_emit_fini_breadcrumb_tail(rq, cs);
614+}
615+
616+u32 *gen12_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs)
617+{
618+ cs = gen12_emit_ggtt_write_rcs(cs,
619+ rq->fence.seqno,
620+ hwsp_offset(rq),
621+ PIPE_CONTROL0_HDC_PIPELINE_FLUSH,
622+ PIPE_CONTROL_CS_STALL |
623+ PIPE_CONTROL_TILE_CACHE_FLUSH |
624+ PIPE_CONTROL_FLUSH_L3 |
625+ PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
626+ PIPE_CONTROL_DEPTH_CACHE_FLUSH |
627+ /* Wa_1409600907:tgl */
628+ PIPE_CONTROL_DEPTH_STALL |
629+ PIPE_CONTROL_DC_FLUSH_ENABLE |
630+ PIPE_CONTROL_FLUSH_ENABLE);
631+
632+ return gen12_emit_fini_breadcrumb_tail(rq, cs);
633+}
drivers/gpu/drm/i915/gt/gen8_engine_cs.h+127−0
@@ -0,0 +1,127 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright © 2014 Intel Corporation
4+ */
5+
6+#ifndef __GEN8_ENGINE_CS_H__
7+#define __GEN8_ENGINE_CS_H__
8+
9+#include <linux/string.h>
10+#include <linux/types.h>
11+
12+#include "i915_gem.h" /* GEM_BUG_ON */
13+
14+#include "intel_gpu_commands.h"
15+
16+struct i915_request;
17+
18+int gen8_emit_flush_rcs(struct i915_request *rq, u32 mode);
19+int gen11_emit_flush_rcs(struct i915_request *rq, u32 mode);
20+int gen12_emit_flush_rcs(struct i915_request *rq, u32 mode);
21+
22+int gen8_emit_flush_xcs(struct i915_request *rq, u32 mode);
23+int gen12_emit_flush_xcs(struct i915_request *rq, u32 mode);
24+
25+int gen8_emit_init_breadcrumb(struct i915_request *rq);
26+
27+int gen8_emit_bb_start_noarb(struct i915_request *rq,
28+ u64 offset, u32 len,
29+ const unsigned int flags);
30+int gen8_emit_bb_start(struct i915_request *rq,
31+ u64 offset, u32 len,
32+ const unsigned int flags);
33+
34+u32 *gen8_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs);
35+u32 *gen12_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs);
36+
37+u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs);
38+u32 *gen11_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs);
39+u32 *gen12_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs);
40+
41+static inline u32 *
42+__gen8_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
43+{
44+ memset(batch, 0, 6 * sizeof(u32));
45+
46+ batch[0] = GFX_OP_PIPE_CONTROL(6) | flags0;
47+ batch[1] = flags1;
48+ batch[2] = offset;
49+
50+ return batch + 6;
51+}
52+
53+static inline u32 *gen8_emit_pipe_control(u32 *batch, u32 flags, u32 offset)
54+{
55+ return __gen8_emit_pipe_control(batch, 0, flags, offset);
56+}
57+
58+static inline u32 *gen12_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
59+{
60+ return __gen8_emit_pipe_control(batch, flags0, flags1, offset);
61+}
62+
63+static inline u32 *
64+__gen8_emit_write_rcs(u32 *cs, u32 value, u32 offset, u32 flags0, u32 flags1)
65+{
66+ *cs++ = GFX_OP_PIPE_CONTROL(6) | flags0;
67+ *cs++ = flags1 | PIPE_CONTROL_QW_WRITE;
68+ *cs++ = offset;
69+ *cs++ = 0;
70+ *cs++ = value;
71+ *cs++ = 0; /* We're thrashing one extra dword. */
72+
73+ return cs;
74+}
75+
76+static inline u32*
77+gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
78+{
79+ /* We're using qword write, offset should be aligned to 8 bytes. */
80+ GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
81+
82+ return __gen8_emit_write_rcs(cs,
83+ value,
84+ gtt_offset,
85+ 0,
86+ flags | PIPE_CONTROL_GLOBAL_GTT_IVB);
87+}
88+
89+static inline u32*
90+gen12_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags0, u32 flags1)
91+{
92+ /* We're using qword write, offset should be aligned to 8 bytes. */
93+ GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
94+
95+ return __gen8_emit_write_rcs(cs,
96+ value,
97+ gtt_offset,
98+ flags0,
99+ flags1 | PIPE_CONTROL_GLOBAL_GTT_IVB);
100+}
101+
102+static inline u32 *
103+__gen8_emit_flush_dw(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
104+{
105+ *cs++ = (MI_FLUSH_DW + 1) | flags;
106+ *cs++ = gtt_offset;
107+ *cs++ = 0;
108+ *cs++ = value;
109+
110+ return cs;
111+}
112+
113+static inline u32 *
114+gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
115+{
116+ /* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
117+ GEM_BUG_ON(gtt_offset & (1 << 5));
118+ /* Offset should be aligned to 8 bytes for both (QW/DW) write types */
119+ GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
120+
121+ return __gen8_emit_flush_dw(cs,
122+ value,
123+ gtt_offset | MI_FLUSH_DW_USE_GTT,
124+ flags | MI_FLUSH_DW_OP_STOREDW);
125+}
126+
127+#endif /* __GEN8_ENGINE_CS_H__ */
drivers/gpu/drm/i915/gt/intel_breadcrumbs.c+73−37
@@ -134,11 +134,6 @@ static bool remove_signaling_context(struct intel_breadcrumbs *b, · 133 unmodified lines
134134 return true;
135135 }
136136
137−static inline bool __request_completed(const struct i915_request *rq)
138−{
139− return i915_seqno_passed(__hwsp_seqno(rq), rq->fence.seqno);
140−}
141−
142137 __maybe_unused static bool
143138 check_signal_order(struct intel_context *ce, struct i915_request *rq)
144139 {
@@ -192,18 +187,6 @@ static void add_retire(struct intel_breadcrumbs *b, struct intel_timeline *tl) · 47 unmodified lines
192187 intel_engine_add_retire(b->irq_engine, tl);
193188 }
194189
195−static bool __signal_request(struct i915_request *rq)
196−{
197− GEM_BUG_ON(test_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags));
198−
199− if (!__dma_fence_signal(&rq->fence)) {
200− i915_request_put(rq);
201− return false;
202− }
203−
204− return true;
205−}
206−
207190 static struct llist_node *
208191 slist_add(struct llist_node *node, struct llist_node *head)
209192 {
@@ -251,13 +234,14 @@ static void signal_irq_work(struct irq_work *work) · 41 unmodified lines
251234 intel_breadcrumbs_disarm_irq(b);
252235
253236 rcu_read_lock();
237+ atomic_inc(&b->signaler_active);
254238 list_for_each_entry_rcu(ce, &b->signalers, signal_link) {
255239 struct i915_request *rq;
256240
257241 list_for_each_entry_rcu(rq, &ce->signals, signal_link) {
258242 bool release;
259243
260− if (!__request_completed(rq))
244+ if (!__i915_request_is_complete(rq))
261245 break;
262246
263247 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL,
@@ -273,17 +257,20 @@ static void signal_irq_work(struct irq_work *work) · 9 unmodified lines
273257 list_del_rcu(&rq->signal_link);
274258 release = remove_signaling_context(b, ce);
275259 spin_unlock(&ce->signal_lock);
276−
277− if (__signal_request(rq))
278− /* We own signal_node now, xfer to local list */
279− signal = slist_add(&rq->signal_node, signal);
280−
281260 if (release) {
282− add_retire(b, ce->timeline);
261+ if (intel_timeline_is_last(ce->timeline, rq))
262+ add_retire(b, ce->timeline);
283263 intel_context_put(ce);
284264 }
265+
266+ if (__dma_fence_signal(&rq->fence))
267+ /* We own signal_node now, xfer to local list */
268+ signal = slist_add(&rq->signal_node, signal);
269+ else
270+ i915_request_put(rq);
285271 }
286272 }
273+ atomic_dec(&b->signaler_active);
287274 rcu_read_unlock();
288275
289276 llist_for_each_safe(signal, sn, signal) {
@@ -342,17 +329,19 @@ void intel_breadcrumbs_reset(struct intel_breadcrumbs *b) · 52 unmodified lines
342329 spin_unlock_irqrestore(&b->irq_lock, flags);
343330 }
344331
345−void intel_breadcrumbs_park(struct intel_breadcrumbs *b)
332+void __intel_breadcrumbs_park(struct intel_breadcrumbs *b)
346333 {
347− /* Kick the work once more to drain the signalers */
334+ if (!READ_ONCE(b->irq_armed))
335+ return;
336+
337+ /* Kick the work once more to drain the signalers, and disarm the irq */
348338 irq_work_sync(&b->irq_work);
349− while (unlikely(READ_ONCE(b->irq_armed))) {
339+ while (READ_ONCE(b->irq_armed) && !atomic_read(&b->active)) {
350340 local_irq_disable();
351341 signal_irq_work(&b->irq_work);
352342 local_irq_enable();
353343 cond_resched();
354344 }
355− GEM_BUG_ON(!list_empty(&b->signalers));
356345 }
357346
358347 void intel_breadcrumbs_free(struct intel_breadcrumbs *b)
@@ -363,6 +352,17 @@ void intel_breadcrumbs_free(struct intel_breadcrumbs *b) · 4 unmodified lines
363352 kfree(b);
364353 }
365354
355+static void irq_signal_request(struct i915_request *rq,
356+ struct intel_breadcrumbs *b)
357+{
358+ if (!__dma_fence_signal(&rq->fence))
359+ return;
360+
361+ i915_request_get(rq);
362+ if (llist_add(&rq->signal_node, &b->signaled_requests))
363+ irq_work_queue(&b->irq_work);
364+}
365+
366366 static void insert_breadcrumb(struct i915_request *rq)
367367 {
368368 struct intel_breadcrumbs *b = READ_ONCE(rq->engine)->breadcrumbs;
@@ -372,17 +372,13 @@ static void insert_breadcrumb(struct i915_request *rq) · 3 unmodified lines
372372 if (test_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags))
373373 return;
374374
375− i915_request_get(rq);
376−
377375 /*
378376 * If the request is already completed, we can transfer it
379377 * straight onto a signaled list, and queue the irq worker for
380378 * its signal completion.
381379 */
382− if (__request_completed(rq)) {
383− if (__signal_request(rq) &&
384− llist_add(&rq->signal_node, &b->signaled_requests))
385− irq_work_queue(&b->irq_work);
380+ if (__i915_request_is_complete(rq)) {
381+ irq_signal_request(rq, b);
386382 return;
387383 }
388384
@@ -413,6 +409,8 @@ static void insert_breadcrumb(struct i915_request *rq) · 24 unmodified lines
413409 break;
414410 }
415411 }
412+
413+ i915_request_get(rq);
416414 list_add_rcu(&rq->signal_link, pos);
417415 GEM_BUG_ON(!check_signal_order(ce, rq));
418416 GEM_BUG_ON(test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &rq->fence.flags));
@@ -453,22 +451,60 @@ bool i915_request_enable_breadcrumb(struct i915_request *rq) · 34 unmodified lines
453451
454452 void i915_request_cancel_breadcrumb(struct i915_request *rq)
455453 {
454+ struct intel_breadcrumbs *b = READ_ONCE(rq->engine)->breadcrumbs;
456455 struct intel_context *ce = rq->context;
456+ unsigned long flags;
457457 bool release;
458458
459459 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags))
460460 return;
461461
462− spin_lock(&ce->signal_lock);
462+ spin_lock_irqsave(&ce->signal_lock, flags);
463463 list_del_rcu(&rq->signal_link);
464− release = remove_signaling_context(rq->engine->breadcrumbs, ce);
465− spin_unlock(&ce->signal_lock);
464+ release = remove_signaling_context(b, ce);
465+ spin_unlock_irqrestore(&ce->signal_lock, flags);
466466 if (release)
467467 intel_context_put(ce);
468468
469+ if (__i915_request_is_complete(rq))
470+ irq_signal_request(rq, b);
471+
469472 i915_request_put(rq);
470473 }
471474
475+void intel_context_remove_breadcrumbs(struct intel_context *ce,
476+ struct intel_breadcrumbs *b)
477+{
478+ struct i915_request *rq, *rn;
479+ bool release = false;
480+ unsigned long flags;
481+
482+ spin_lock_irqsave(&ce->signal_lock, flags);
483+
484+ if (list_empty(&ce->signals))
485+ goto unlock;
486+
487+ list_for_each_entry_safe(rq, rn, &ce->signals, signal_link) {
488+ GEM_BUG_ON(!__i915_request_is_complete(rq));
489+ if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL,
490+ &rq->fence.flags))
491+ continue;
492+
493+ list_del_rcu(&rq->signal_link);
494+ irq_signal_request(rq, b);
495+ i915_request_put(rq);
496+ }
497+ release = remove_signaling_context(b, ce);
498+
499+unlock:
500+ spin_unlock_irqrestore(&ce->signal_lock, flags);
501+ if (release)
502+ intel_context_put(ce);
503+
504+ while (atomic_read(&b->signaler_active))
505+ cpu_relax();
506+}
507+
472508 static void print_signals(struct intel_breadcrumbs *b, struct drm_printer *p)
473509 {
474510 struct intel_context *ce;
drivers/gpu/drm/i915/gt/intel_breadcrumbs.h+16−1
@@ -6,6 +6,7 @@ · 5 unmodified lines
66 #ifndef __INTEL_BREADCRUMBS__
77 #define __INTEL_BREADCRUMBS__
88
9+#include <linux/atomic.h>
910 #include <linux/irq_work.h>
1011
1112 #include "intel_engine_types.h"
@@ -19,7 +20,18 @@ intel_breadcrumbs_create(struct intel_engine_cs *irq_engine); · 7 unmodified lines
1920 void intel_breadcrumbs_free(struct intel_breadcrumbs *b);
2021
2122 void intel_breadcrumbs_reset(struct intel_breadcrumbs *b);
22−void intel_breadcrumbs_park(struct intel_breadcrumbs *b);
23+void __intel_breadcrumbs_park(struct intel_breadcrumbs *b);
24+
25+static inline void intel_breadcrumbs_unpark(struct intel_breadcrumbs *b)
26+{
27+ atomic_inc(&b->active);
28+}
29+
30+static inline void intel_breadcrumbs_park(struct intel_breadcrumbs *b)
31+{
32+ if (atomic_dec_and_test(&b->active))
33+ __intel_breadcrumbs_park(b);
34+}
2335
2436 static inline void
2537 intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine)
@@ -33,4 +45,7 @@ void intel_engine_print_breadcrumbs(struct intel_engine_cs *engine, · 7 unmodified lines
3345 bool i915_request_enable_breadcrumb(struct i915_request *request);
3446 void i915_request_cancel_breadcrumb(struct i915_request *request);
3547
48+void intel_context_remove_breadcrumbs(struct intel_context *ce,
49+ struct intel_breadcrumbs *b);
50+
3651 #endif /* __INTEL_BREADCRUMBS__ */
drivers/gpu/drm/i915/gt/intel_breadcrumbs_types.h+5−2
@@ -29,17 +29,20 @@ · 28 unmodified lines
2929 * the overhead of waking that client is much preferred.
3030 */
3131 struct intel_breadcrumbs {
32− /* Not all breadcrumbs are attached to physical HW */
33− struct intel_engine_cs *irq_engine;
32+ atomic_t active;
3433
3534 spinlock_t signalers_lock; /* protects the list of signalers */
3635 struct list_head signalers;
3736 struct llist_head signaled_requests;
37+ atomic_t signaler_active;
3838
3939 spinlock_t irq_lock; /* protects the interrupt from hardirq context */
4040 struct irq_work irq_work; /* for use from inside irq_lock */
4141 unsigned int irq_enabled;
4242 bool irq_armed;
43+
44+ /* Not all breadcrumbs are attached to physical HW */
45+ struct intel_engine_cs *irq_engine;
4346 };
4447
4548 #endif /* __INTEL_BREADCRUMBS_TYPES__ */
drivers/gpu/drm/i915/gt/intel_context.h+7−4
@@ -191,6 +191,11 @@ static inline bool intel_context_is_closed(const struct intel_context *ce) · 190 unmodified lines
191191 return test_bit(CONTEXT_CLOSED_BIT, &ce->flags);
192192 }
193193
194+static inline bool intel_context_has_inflight(const struct intel_context *ce)
195+{
196+ return test_bit(COPS_HAS_INFLIGHT_BIT, &ce->ops->flags);
197+}
198+
194199 static inline bool intel_context_use_semaphores(const struct intel_context *ce)
195200 {
196201 return test_bit(CONTEXT_USE_SEMAPHORES, &ce->flags);
@@ -248,16 +253,14 @@ intel_context_clear_nopreempt(struct intel_context *ce) · 51 unmodified lines
248253
249254 static inline u64 intel_context_get_total_runtime_ns(struct intel_context *ce)
250255 {
251− const u32 period =
252− RUNTIME_INFO(ce->engine->i915)->cs_timestamp_period_ns;
256+ const u32 period = ce->engine->gt->clock_period_ns;
253257
254258 return READ_ONCE(ce->runtime.total) * period;
255259 }
256260
257261 static inline u64 intel_context_get_avg_runtime_ns(struct intel_context *ce)
258262 {
259− const u32 period =
260− RUNTIME_INFO(ce->engine->i915)->cs_timestamp_period_ns;
263+ const u32 period = ce->engine->gt->clock_period_ns;
261264
262265 return mul_u32_u32(ewma_runtime_read(&ce->runtime.avg), period);
263266 }
drivers/gpu/drm/i915/gt/intel_context_sseu.c+0−1
@@ -9,7 +9,6 @@ · 8 unmodified lines
99 #include "intel_engine_pm.h"
1010 #include "intel_gpu_commands.h"
1111 #include "intel_lrc.h"
12−#include "intel_lrc_reg.h"
1312 #include "intel_ring.h"
1413 #include "intel_sseu.h"
1514
drivers/gpu/drm/i915/gt/intel_context_types.h+17−8
@@ -30,6 +30,10 @@ struct intel_context; · 29 unmodified lines
3030 struct intel_ring;
3131
3232 struct intel_context_ops {
33+ unsigned long flags;
34+#define COPS_HAS_INFLIGHT_BIT 0
35+#define COPS_HAS_INFLIGHT BIT(COPS_HAS_INFLIGHT_BIT)
36+
3337 int (*alloc)(struct intel_context *ce);
3438
3539 int (*pre_pin)(struct intel_context *ce, struct i915_gem_ww_ctx *ww, void **vaddr);
@@ -58,8 +62,12 @@ struct intel_context { · 22 unmodified lines
5862
5963 struct intel_engine_cs *engine;
6064 struct intel_engine_cs *inflight;
61−#define intel_context_inflight(ce) ptr_mask_bits(READ_ONCE((ce)->inflight), 2)
62−#define intel_context_inflight_count(ce) ptr_unmask_bits(READ_ONCE((ce)->inflight), 2)
65+#define __intel_context_inflight(engine) ptr_mask_bits(engine, 3)
66+#define __intel_context_inflight_count(engine) ptr_unmask_bits(engine, 3)
67+#define intel_context_inflight(ce) \
68+ __intel_context_inflight(READ_ONCE((ce)->inflight))
69+#define intel_context_inflight_count(ce) \
70+ __intel_context_inflight_count(READ_ONCE((ce)->inflight))
6371
6472 struct i915_address_space *vm;
6573 struct i915_gem_context __rcu *gem_context;
@@ -81,12 +89,13 @@ struct intel_context { · 15 unmodified lines
8189 unsigned long flags;
8290 #define CONTEXT_BARRIER_BIT 0
8391 #define CONTEXT_ALLOC_BIT 1
84−#define CONTEXT_VALID_BIT 2
85−#define CONTEXT_CLOSED_BIT 3
86−#define CONTEXT_USE_SEMAPHORES 4
87−#define CONTEXT_BANNED 5
88−#define CONTEXT_FORCE_SINGLE_SUBMISSION 6
89−#define CONTEXT_NOPREEMPT 7
92+#define CONTEXT_INIT_BIT 2
93+#define CONTEXT_VALID_BIT 3
94+#define CONTEXT_CLOSED_BIT 4
95+#define CONTEXT_USE_SEMAPHORES 5
96+#define CONTEXT_BANNED 6
97+#define CONTEXT_FORCE_SINGLE_SUBMISSION 7
98+#define CONTEXT_NOPREEMPT 8
9099
91100 u32 *lrc_reg_state;
92101 union {
drivers/gpu/drm/i915/gt/intel_engine.h+6−87
@@ -15,7 +15,6 @@ · 14 unmodified lines
1515 #include "i915_selftest.h"
1616 #include "gt/intel_timeline.h"
1717 #include "intel_engine_types.h"
18−#include "intel_gpu_commands.h"
1918 #include "intel_workarounds.h"
2019
2120 struct drm_printer;
@@ -223,91 +222,6 @@ void intel_engine_get_instdone(const struct intel_engine_cs *engine, · 201 unmodified lines
223222
224223 void intel_engine_init_execlists(struct intel_engine_cs *engine);
225224
226−static inline u32 *__gen8_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
227−{
228− memset(batch, 0, 6 * sizeof(u32));
229−
230− batch[0] = GFX_OP_PIPE_CONTROL(6) | flags0;
231− batch[1] = flags1;
232− batch[2] = offset;
233−
234− return batch + 6;
235−}
236−
237−static inline u32 *gen8_emit_pipe_control(u32 *batch, u32 flags, u32 offset)
238−{
239− return __gen8_emit_pipe_control(batch, 0, flags, offset);
240−}
241−
242−static inline u32 *gen12_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
243−{
244− return __gen8_emit_pipe_control(batch, flags0, flags1, offset);
245−}
246−
247−static inline u32 *
248−__gen8_emit_write_rcs(u32 *cs, u32 value, u32 offset, u32 flags0, u32 flags1)
249−{
250− *cs++ = GFX_OP_PIPE_CONTROL(6) | flags0;
251− *cs++ = flags1 | PIPE_CONTROL_QW_WRITE;
252− *cs++ = offset;
253− *cs++ = 0;
254− *cs++ = value;
255− *cs++ = 0; /* We're thrashing one extra dword. */
256−
257− return cs;
258−}
259−
260−static inline u32*
261−gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
262−{
263− /* We're using qword write, offset should be aligned to 8 bytes. */
264− GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
265−
266− return __gen8_emit_write_rcs(cs,
267− value,
268− gtt_offset,
269− 0,
270− flags | PIPE_CONTROL_GLOBAL_GTT_IVB);
271−}
272−
273−static inline u32*
274−gen12_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags0, u32 flags1)
275−{
276− /* We're using qword write, offset should be aligned to 8 bytes. */
277− GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
278−
279− return __gen8_emit_write_rcs(cs,
280− value,
281− gtt_offset,
282− flags0,
283− flags1 | PIPE_CONTROL_GLOBAL_GTT_IVB);
284−}
285−
286−static inline u32 *
287−__gen8_emit_flush_dw(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
288−{
289− *cs++ = (MI_FLUSH_DW + 1) | flags;
290− *cs++ = gtt_offset;
291− *cs++ = 0;
292− *cs++ = value;
293−
294− return cs;
295−}
296−
297−static inline u32 *
298−gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
299−{
300− /* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
301− GEM_BUG_ON(gtt_offset & (1 << 5));
302− /* Offset should be aligned to 8 bytes for both (QW/DW) write types */
303− GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
304−
305− return __gen8_emit_flush_dw(cs,
306− value,
307− gtt_offset | MI_FLUSH_DW_USE_GTT,
308− flags | MI_FLUSH_DW_OP_STOREDW);
309−}
310−
311225 static inline void __intel_engine_reset(struct intel_engine_cs *engine,
312226 bool stalled)
313227 {
@@ -318,7 +232,12 @@ static inline void __intel_engine_reset(struct intel_engine_cs *engine, · 4 unmodified lines
318232
319233 bool intel_engines_are_idle(struct intel_gt *gt);
320234 bool intel_engine_is_idle(struct intel_engine_cs *engine);
321−void intel_engine_flush_submission(struct intel_engine_cs *engine);
235+
236+void __intel_engine_flush_submission(struct intel_engine_cs *engine, bool sync);
237+static inline void intel_engine_flush_submission(struct intel_engine_cs *engine)
238+{
239+ __intel_engine_flush_submission(engine, true);
240+}
322241
323242 void intel_engines_reset_default_submission(struct intel_gt *gt);
324243
drivers/gpu/drm/i915/gt/intel_engine_cs.c+80−71
@@ -33,12 +33,14 @@ · 32 unmodified lines
3333 #include "intel_engine.h"
3434 #include "intel_engine_pm.h"
3535 #include "intel_engine_user.h"
36+#include "intel_execlists_submission.h"
3637 #include "intel_gt.h"
3738 #include "intel_gt_requests.h"
3839 #include "intel_gt_pm.h"
39−#include "intel_lrc.h"
40+#include "intel_lrc_reg.h"
4041 #include "intel_reset.h"
4142 #include "intel_ring.h"
43+#include "uc/intel_guc_submission.h"
4244
4345 /* Haswell does have the CXT_SIZE register however it does not appear to be
4446 * valid. Now, docs explain in dwords what is in the context object. The full
@@ -647,6 +649,8 @@ static int init_status_page(struct intel_engine_cs *engine) · 602 unmodified lines
647649 void *vaddr;
648650 int ret;
649651
652+ INIT_LIST_HEAD(&engine->status_page.timelines);
653+
650654 /*
651655 * Though the HWS register does support 36bit addresses, historically
652656 * we have had hangs and corruption reported due to wild writes if
@@ -723,6 +727,9 @@ static int engine_setup_common(struct intel_engine_cs *engine) · 70 unmodified lines
723727 intel_engine_init_whitelist(engine);
724728 intel_engine_init_ctx_wa(engine);
725729
730+ if (INTEL_GEN(engine->i915) >= 12)
731+ engine->flags |= I915_ENGINE_HAS_RELATIVE_MMIO;
732+
726733 return 0;
727734
728735 err_status:
@@ -829,6 +836,21 @@ create_pinned_context(struct intel_engine_cs *engine, · 100 unmodified lines
829836 return ce;
830837 }
831838
839+static void destroy_pinned_context(struct intel_context *ce)
840+{
841+ struct intel_engine_cs *engine = ce->engine;
842+ struct i915_vma *hwsp = engine->status_page.vma;
843+
844+ GEM_BUG_ON(ce->timeline->hwsp_ggtt != hwsp);
845+
846+ mutex_lock(&hwsp->vm->mutex);
847+ list_del(&ce->timeline->engine_link);
848+ mutex_unlock(&hwsp->vm->mutex);
849+
850+ intel_context_unpin(ce);
851+ intel_context_put(ce);
852+}
853+
832854 static struct intel_context *
833855 create_kernel_context(struct intel_engine_cs *engine)
834856 {
@@ -889,7 +911,9 @@ int intel_engines_init(struct intel_gt *gt) · 54 unmodified lines
889911 enum intel_engine_id id;
890912 int err;
891913
892− if (HAS_EXECLISTS(gt->i915))
914+ if (intel_uc_uses_guc_submission(&gt->uc))
915+ setup = intel_guc_submission_setup;
916+ else if (HAS_EXECLISTS(gt->i915))
893917 setup = intel_execlists_submission_setup;
894918 else
895919 setup = intel_ring_submission_setup;
@@ -925,7 +949,6 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine) · 29 unmodified lines
925949 GEM_BUG_ON(!list_empty(&engine->active.requests));
926950 tasklet_kill(&engine->execlists.tasklet); /* flush the callback */
927951
928− cleanup_status_page(engine);
929952 intel_breadcrumbs_free(engine->breadcrumbs);
930953
931954 intel_engine_fini_retire(engine);
@@ -934,11 +957,11 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine) · 2 unmodified lines
934957 if (engine->default_state)
935958 fput(engine->default_state);
936959
937− if (engine->kernel_context) {
938− intel_context_unpin(engine->kernel_context);
939− intel_context_put(engine->kernel_context);
940− }
960+ if (engine->kernel_context)
961+ destroy_pinned_context(engine->kernel_context);
962+
941963 GEM_BUG_ON(!llist_empty(&engine->barrier_tasks));
964+ cleanup_status_page(engine);
942965
943966 intel_wa_list_free(&engine->ctx_wa_list);
944967 intel_wa_list_free(&engine->wa_list);
@@ -1002,32 +1025,50 @@ static unsigned long stop_timeout(const struct intel_engine_cs *engine) · 57 unmodified lines
10021025 return READ_ONCE(engine->props.stop_timeout_ms);
10031026 }
10041027
1005−int intel_engine_stop_cs(struct intel_engine_cs *engine)
1028+static int __intel_engine_stop_cs(struct intel_engine_cs *engine,
1029+ int fast_timeout_us,
1030+ int slow_timeout_ms)
10061031 {
10071032 struct intel_uncore *uncore = engine->uncore;
1008− const u32 base = engine->mmio_base;
1009− const i915_reg_t mode = RING_MI_MODE(base);
1033+ const i915_reg_t mode = RING_MI_MODE(engine->mmio_base);
10101034 int err;
10111035
1036+ intel_uncore_write_fw(uncore, mode, _MASKED_BIT_ENABLE(STOP_RING));
1037+ err = __intel_wait_for_register_fw(engine->uncore, mode,
1038+ MODE_IDLE, MODE_IDLE,
1039+ fast_timeout_us,
1040+ slow_timeout_ms,
1041+ NULL);
1042+
1043+ /* A final mmio read to let GPU writes be hopefully flushed to memory */
1044+ intel_uncore_posting_read_fw(uncore, mode);
1045+ return err;
1046+}
1047+
1048+int intel_engine_stop_cs(struct intel_engine_cs *engine)
1049+{
1050+ int err = 0;
1051+
10121052 if (INTEL_GEN(engine->i915) < 3)
10131053 return -ENODEV;
10141054
10151055 ENGINE_TRACE(engine, "\n");
1056+ if (__intel_engine_stop_cs(engine, 1000, stop_timeout(engine))) {
1057+ ENGINE_TRACE(engine,
1058+ "timed out on STOP_RING -> IDLE; HEAD:%04x, TAIL:%04x\n",
1059+ ENGINE_READ_FW(engine, RING_HEAD) & HEAD_ADDR,
1060+ ENGINE_READ_FW(engine, RING_TAIL) & TAIL_ADDR);
10161061
1017− intel_uncore_write_fw(uncore, mode, _MASKED_BIT_ENABLE(STOP_RING));
1018−
1019− err = 0;
1020− if (__intel_wait_for_register_fw(uncore,
1021− mode, MODE_IDLE, MODE_IDLE,
1022− 1000, stop_timeout(engine),
1023− NULL)) {
1024− ENGINE_TRACE(engine, "timed out on STOP_RING -> IDLE\n");
1025− err = -ETIMEDOUT;
1062+ /*
1063+ * Sometimes we observe that the idle flag is not
1064+ * set even though the ring is empty. So double
1065+ * check before giving up.
1066+ */
1067+ if ((ENGINE_READ_FW(engine, RING_HEAD) & HEAD_ADDR) !=
1068+ (ENGINE_READ_FW(engine, RING_TAIL) & TAIL_ADDR))
1069+ err = -ETIMEDOUT;
10261070 }
10271071
1028− /* A final mmio read to let GPU writes be hopefully flushed to memory */
1029− intel_uncore_posting_read_fw(uncore, mode);
1030−
10311072 return err;
10321073 }
10331074
@@ -1189,17 +1230,13 @@ static bool ring_is_idle(struct intel_engine_cs *engine) · 155 unmodified lines
11891230 return idle;
11901231 }
11911232
1192−void intel_engine_flush_submission(struct intel_engine_cs *engine)
1233+void __intel_engine_flush_submission(struct intel_engine_cs *engine, bool sync)
11931234 {
11941235 struct tasklet_struct *t = &engine->execlists.tasklet;
11951236
11961237 if (!t->func)
11971238 return;
11981239
1199− /* Synchronise and wait for the tasklet on another CPU */
1200− tasklet_kill(t);
1201−
1202− /* Having cancelled the tasklet, ensure that is run */
12031240 local_bh_disable();
12041241 if (tasklet_trylock(t)) {
12051242 /* Must wait for any GPU reset in progress. */
@@ -1208,6 +1245,10 @@ void intel_engine_flush_submission(struct intel_engine_cs *engine) · 2 unmodified lines
12081245 tasklet_unlock(t);
12091246 }
12101247 local_bh_enable();
1248+
1249+ /* Synchronise and wait for the tasklet on another CPU */
1250+ if (sync)
1251+ tasklet_unlock_wait(t);
12111252 }
12121253
12131254 /**
@@ -1273,8 +1314,12 @@ void intel_engines_reset_default_submission(struct intel_gt *gt) · 59 unmodified lines
12731314 struct intel_engine_cs *engine;
12741315 enum intel_engine_id id;
12751316
1276− for_each_engine(engine, gt, id)
1317+ for_each_engine(engine, gt, id) {
1318+ if (engine->sanitize)
1319+ engine->sanitize(engine);
1320+
12771321 engine->set_default_submission(engine);
1322+ }
12781323 }
12791324
12801325 bool intel_engine_can_store_dword(struct intel_engine_cs *engine)
@@ -1294,44 +1339,6 @@ bool intel_engine_can_store_dword(struct intel_engine_cs *engine) · 13 unmodified lines
12941339 }
12951340 }
12961341
1297−static int print_sched_attr(const struct i915_sched_attr *attr,
1298− char *buf, int x, int len)
1299−{
1300− if (attr->priority == I915_PRIORITY_INVALID)
1301− return x;
1302−
1303− x += snprintf(buf + x, len - x,
1304− " prio=%d", attr->priority);
1305−
1306− return x;
1307−}
1308−
1309−static void print_request(struct drm_printer *m,
1310− struct i915_request *rq,
1311− const char *prefix)
1312−{
1313− const char *name = rq->fence.ops->get_timeline_name(&rq->fence);
1314− char buf[80] = "";
1315− int x = 0;
1316−
1317− x = print_sched_attr(&rq->sched.attr, buf, x, sizeof(buf));
1318−
1319− drm_printf(m, "%s %llx:%llx%s%s %s @ %dms: %s\n",
1320− prefix,
1321− rq->fence.context, rq->fence.seqno,
1322− i915_request_completed(rq) ? "!" :
1323− i915_request_started(rq) ? "*" :
1324− "",
1325− test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
1326− &rq->fence.flags) ? "+" :
1327− test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT,
1328− &rq->fence.flags) ? "-" :
1329− "",
1330− buf,
1331− jiffies_to_msecs(jiffies - rq->emitted_jiffies),
1332− name);
1333−}
1334−
13351342 static struct intel_timeline *get_timeline(struct i915_request *rq)
13361343 {
13371344 struct intel_timeline *tl;
@@ -1480,7 +1487,9 @@ static void intel_engine_print_registers(struct intel_engine_cs *engine, · 142 unmodified lines
14801487 drm_printf(m, "\tIPEHR: 0x%08x\n", ENGINE_READ(engine, IPEHR));
14811488 }
14821489
1483− if (HAS_EXECLISTS(dev_priv)) {
1490+ if (intel_engine_in_guc_submission_mode(engine)) {
1491+ /* nothing to print yet */
1492+ } else if (HAS_EXECLISTS(dev_priv)) {
14841493 struct i915_request * const *port, *rq;
14851494 const u32 *hws =
14861495 &engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];
@@ -1529,7 +1538,7 @@ static void intel_engine_print_registers(struct intel_engine_cs *engine, · 42 unmodified lines
15291538 intel_context_is_banned(rq->context) ? "*" : "");
15301539 len += print_ring(hdr + len, sizeof(hdr) - len, rq);
15311540 scnprintf(hdr + len, sizeof(hdr) - len, "rq: ");
1532− print_request(m, rq, hdr);
1541+ i915_request_show(m, rq, hdr, 0);
15331542 }
15341543 for (port = execlists->pending; (rq = *port); port++) {
15351544 char hdr[160];
@@ -1543,7 +1552,7 @@ static void intel_engine_print_registers(struct intel_engine_cs *engine, · 7 unmodified lines
15431552 intel_context_is_banned(rq->context) ? "*" : "");
15441553 len += print_ring(hdr + len, sizeof(hdr) - len, rq);
15451554 scnprintf(hdr + len, sizeof(hdr) - len, "rq: ");
1546− print_request(m, rq, hdr);
1555+ i915_request_show(m, rq, hdr, 0);
15471556 }
15481557 rcu_read_unlock();
15491558 execlists_active_unlock_bh(execlists);
@@ -1687,7 +1696,7 @@ void intel_engine_dump(struct intel_engine_cs *engine, · 137 unmodified lines
16871696 if (rq) {
16881697 struct intel_timeline *tl = get_timeline(rq);
16891698
1690− print_request(m, rq, "\t\tactive ");
1699+ i915_request_show(m, rq, "\t\tactive ", 0);
16911700
16921701 drm_printf(m, "\t\tring->start: 0x%08x\n",
16931702 i915_ggtt_offset(rq->ring->vma));
@@ -1725,7 +1734,7 @@ void intel_engine_dump(struct intel_engine_cs *engine, · 31 unmodified lines
17251734 drm_printf(m, "\tDevice is asleep; skipping register dump\n");
17261735 }
17271736
1728− intel_execlists_show_requests(engine, m, print_request, 8);
1737+ intel_execlists_show_requests(engine, m, i915_request_show, 8);
17291738
17301739 drm_printf(m, "HWSP:\n");
17311740 hexdump(m, engine->status_page.addr, PAGE_SIZE);
drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c+41−18
@@ -37,6 +37,18 @@ static bool next_heartbeat(struct intel_engine_cs *engine) · 36 unmodified lines
3737 return true;
3838 }
3939
40+static struct i915_request *
41+heartbeat_create(struct intel_context *ce, gfp_t gfp)
42+{
43+ struct i915_request *rq;
44+
45+ intel_context_enter(ce);
46+ rq = __i915_request_create(ce, gfp);
47+ intel_context_exit(ce);
48+
49+ return rq;
50+}
51+
4052 static void idle_pulse(struct intel_engine_cs *engine, struct i915_request *rq)
4153 {
4254 engine->wakeref_serial = READ_ONCE(engine->serial) + 1;
@@ -45,6 +57,15 @@ static void idle_pulse(struct intel_engine_cs *engine, struct i915_request *rq) · 2 unmodified lines
4557 engine->heartbeat.systole = i915_request_get(rq);
4658 }
4759
60+static void heartbeat_commit(struct i915_request *rq,
61+ const struct i915_sched_attr *attr)
62+{
63+ idle_pulse(rq->engine, rq);
64+
65+ __i915_request_commit(rq);
66+ __i915_request_queue(rq, attr);
67+}
68+
4869 static void show_heartbeat(const struct i915_request *rq,
4970 struct intel_engine_cs *engine)
5071 {
@@ -139,16 +160,11 @@ static void heartbeat(struct work_struct *wrk) · 88 unmodified lines
139160 goto out;
140161 }
141162
142− intel_context_enter(ce);
143− rq = __i915_request_create(ce, GFP_NOWAIT | __GFP_NOWARN);
144− intel_context_exit(ce);
163+ rq = heartbeat_create(ce, GFP_NOWAIT | __GFP_NOWARN);
145164 if (IS_ERR(rq))
146165 goto unlock;
147166
148− idle_pulse(engine, rq);
149−
150− __i915_request_commit(rq);
151− __i915_request_queue(rq, &attr);
167+ heartbeat_commit(rq, &attr);
152168
153169 unlock:
154170 mutex_unlock(&ce->timeline->mutex);
@@ -187,17 +203,13 @@ static int __intel_engine_pulse(struct intel_engine_cs *engine) · 32 unmodified lines
187203 GEM_BUG_ON(!intel_engine_has_preemption(engine));
188204 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
189205
190− intel_context_enter(ce);
191− rq = __i915_request_create(ce, GFP_NOWAIT | __GFP_NOWARN);
192− intel_context_exit(ce);
206+ rq = heartbeat_create(ce, GFP_NOWAIT | __GFP_NOWARN);
193207 if (IS_ERR(rq))
194208 return PTR_ERR(rq);
195209
196210 __set_bit(I915_FENCE_FLAG_SENTINEL, &rq->fence.flags);
197− idle_pulse(engine, rq);
198211
199− __i915_request_commit(rq);
200− __i915_request_queue(rq, &attr);
212+ heartbeat_commit(rq, &attr);
201213 GEM_BUG_ON(rq->sched.attr.priority < I915_PRIORITY_BARRIER);
202214
203215 return 0;
@@ -273,8 +285,12 @@ int intel_engine_pulse(struct intel_engine_cs *engine) · 69 unmodified lines
273285
274286 int intel_engine_flush_barriers(struct intel_engine_cs *engine)
275287 {
288+ struct i915_sched_attr attr = {
289+ .priority = I915_USER_PRIORITY(I915_PRIORITY_MIN),
290+ };
291+ struct intel_context *ce = engine->kernel_context;
276292 struct i915_request *rq;
277− int err = 0;
293+ int err;
278294
279295 if (llist_empty(&engine->barrier_tasks))
280296 return 0;
@@ -282,15 +298,22 @@ int intel_engine_flush_barriers(struct intel_engine_cs *engine) · 1 unmodified line
282298 if (!intel_engine_pm_get_if_awake(engine))
283299 return 0;
284300
285− rq = i915_request_create(engine->kernel_context);
301+ if (mutex_lock_interruptible(&ce->timeline->mutex)) {
302+ err = -EINTR;
303+ goto out_rpm;
304+ }
305+
306+ rq = heartbeat_create(ce, GFP_KERNEL);
286307 if (IS_ERR(rq)) {
287308 err = PTR_ERR(rq);
288− goto out_rpm;
309+ goto out_unlock;
289310 }
290311
291− idle_pulse(engine, rq);
292− i915_request_add(rq);
312+ heartbeat_commit(rq, &attr);
293313
314+ err = 0;
315+out_unlock:
316+ mutex_unlock(&ce->timeline->mutex);
294317 out_rpm:
295318 intel_engine_pm_put(engine);
296319 return err;
drivers/gpu/drm/i915/gt/intel_engine_pm.c+9−1
@@ -60,11 +60,19 @@ static int __engine_unpark(struct intel_wakeref *wf) · 59 unmodified lines
6060
6161 /* Scrub the context image after our loss of control */
6262 ce->ops->reset(ce);
63+
64+ CE_TRACE(ce, "reset { seqno:%x, *hwsp:%x, ring:%x }\n",
65+ ce->timeline->seqno,
66+ READ_ONCE(*ce->timeline->hwsp_seqno),
67+ ce->ring->emit);
68+ GEM_BUG_ON(ce->timeline->seqno !=
69+ READ_ONCE(*ce->timeline->hwsp_seqno));
6370 }
6471
6572 if (engine->unpark)
6673 engine->unpark(engine);
6774
75+ intel_breadcrumbs_unpark(engine->breadcrumbs);
6876 intel_engine_unpark_heartbeat(engine);
6977 return 0;
7078 }
@@ -136,7 +144,7 @@ __queue_and_release_pm(struct i915_request *rq, · 65 unmodified lines
136144 list_add_tail(&tl->link, &timelines->active_list);
137145
138146 /* Hand the request over to HW and so engine_retire() */
139− __i915_request_queue(rq, NULL);
147+ __i915_request_queue_bh(rq);
140148
141149 /* Let new submissions commence (and maybe retire this timeline) */
142150 __intel_wakeref_defer_park(&engine->wakeref);
drivers/gpu/drm/i915/gt/intel_engine_types.h+5−11
@@ -68,6 +68,7 @@ typedef u8 intel_engine_mask_t; · 67 unmodified lines
6868 #define ALL_ENGINES ((intel_engine_mask_t)~0ul)
6969
7070 struct intel_hw_status_page {
71+ struct list_head timelines;
7172 struct i915_vma *vma;
7273 u32 *addr;
7374 };
@@ -183,7 +184,8 @@ struct intel_engine_execlists { · 109 unmodified lines
183184 * Reserve the upper 16b for tracking internal errors.
184185 */
185186 u32 error_interrupt;
186−#define ERROR_CSB BIT(31)
187+#define ERROR_CSB BIT(31)
188+#define ERROR_PREEMPT BIT(30)
187189
188190 /**
189191 * @reset_ccid: Active CCID [EXECLISTS_STATUS_HI] at the time of reset
@@ -236,16 +238,6 @@ struct intel_engine_execlists { · 46 unmodified lines
236238 */
237239 unsigned int port_mask;
238240
239− /**
240− * @switch_priority_hint: Second context priority.
241− *
242− * We submit multiple contexts to the HW simultaneously and would
243− * like to occasionally switch between them to emulate timeslicing.
244− * To know when timeslicing is suitable, we track the priority of
245− * the context submitted second.
246− */
247− int switch_priority_hint;
248−
249241 /**
250242 * @queue_priority_hint: Highest pending priority.
251243 *
@@ -559,6 +551,8 @@ struct intel_engine_cs { · 307 unmodified lines
559551 unsigned long stop_timeout_ms;
560552 unsigned long timeslice_duration_ms;
561553 } props, defaults;
554+
555+ I915_SELFTEST_DECLARE(struct fault_attr reset_timeout);
562556 };
563557
564558 static inline bool
drivers/gpu/drm/i915/gt/intel_execlists_submission.c+3929−0
@@ -0,0 +1,3929 @@
1+// SPDX-License-Identifier: MIT
2+/*
3+ * Copyright © 2014 Intel Corporation
4+ */
5+
6+/**
7+ * DOC: Logical Rings, Logical Ring Contexts and Execlists
8+ *
9+ * Motivation:
10+ * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts".
11+ * These expanded contexts enable a number of new abilities, especially
12+ * "Execlists" (also implemented in this file).
13+ *
14+ * One of the main differences with the legacy HW contexts is that logical
15+ * ring contexts incorporate many more things to the context's state, like
16+ * PDPs or ringbuffer control registers:
17+ *
18+ * The reason why PDPs are included in the context is straightforward: as
19+ * PPGTTs (per-process GTTs) are actually per-context, having the PDPs
20+ * contained there mean you don't need to do a ppgtt->switch_mm yourself,
21+ * instead, the GPU will do it for you on the context switch.
22+ *
23+ * But, what about the ringbuffer control registers (head, tail, etc..)?
24+ * shouldn't we just need a set of those per engine command streamer? This is
25+ * where the name "Logical Rings" starts to make sense: by virtualizing the
26+ * rings, the engine cs shifts to a new "ring buffer" with every context
27+ * switch. When you want to submit a workload to the GPU you: A) choose your
28+ * context, B) find its appropriate virtualized ring, C) write commands to it
29+ * and then, finally, D) tell the GPU to switch to that context.
30+ *
31+ * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch
32+ * to a contexts is via a context execution list, ergo "Execlists".
33+ *
34+ * LRC implementation:
35+ * Regarding the creation of contexts, we have:
36+ *
37+ * - One global default context.
38+ * - One local default context for each opened fd.
39+ * - One local extra context for each context create ioctl call.
40+ *
41+ * Now that ringbuffers belong per-context (and not per-engine, like before)
42+ * and that contexts are uniquely tied to a given engine (and not reusable,
43+ * like before) we need:
44+ *
45+ * - One ringbuffer per-engine inside each context.
46+ * - One backing object per-engine inside each context.
47+ *
48+ * The global default context starts its life with these new objects fully
49+ * allocated and populated. The local default context for each opened fd is
50+ * more complex, because we don't know at creation time which engine is going
51+ * to use them. To handle this, we have implemented a deferred creation of LR
52+ * contexts:
53+ *
54+ * The local context starts its life as a hollow or blank holder, that only
55+ * gets populated for a given engine once we receive an execbuffer. If later
56+ * on we receive another execbuffer ioctl for the same context but a different
57+ * engine, we allocate/populate a new ringbuffer and context backing object and
58+ * so on.
59+ *
60+ * Finally, regarding local contexts created using the ioctl call: as they are
61+ * only allowed with the render ring, we can allocate & populate them right
62+ * away (no need to defer anything, at least for now).
63+ *
64+ * Execlists implementation:
65+ * Execlists are the new method by which, on gen8+ hardware, workloads are
66+ * submitted for execution (as opposed to the legacy, ringbuffer-based, method).
67+ * This method works as follows:
68+ *
69+ * When a request is committed, its commands (the BB start and any leading or
70+ * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer
71+ * for the appropriate context. The tail pointer in the hardware context is not
72+ * updated at this time, but instead, kept by the driver in the ringbuffer
73+ * structure. A structure representing this request is added to a request queue
74+ * for the appropriate engine: this structure contains a copy of the context's
75+ * tail after the request was written to the ring buffer and a pointer to the
76+ * context itself.
77+ *
78+ * If the engine's request queue was empty before the request was added, the
79+ * queue is processed immediately. Otherwise the queue will be processed during
80+ * a context switch interrupt. In any case, elements on the queue will get sent
81+ * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a
82+ * globally unique 20-bits submission ID.
83+ *
84+ * When execution of a request completes, the GPU updates the context status
85+ * buffer with a context complete event and generates a context switch interrupt.
86+ * During the interrupt handling, the driver examines the events in the buffer:
87+ * for each context complete event, if the announced ID matches that on the head
88+ * of the request queue, then that request is retired and removed from the queue.
89+ *
90+ * After processing, if any requests were retired and the queue is not empty
91+ * then a new execution list can be submitted. The two requests at the front of
92+ * the queue are next to be submitted but since a context may not occur twice in
93+ * an execution list, if subsequent requests have the same ID as the first then
94+ * the two requests must be combined. This is done simply by discarding requests
95+ * at the head of the queue until either only one requests is left (in which case
96+ * we use a NULL second context) or the first two requests have unique IDs.
97+ *
98+ * By always executing the first two requests in the queue the driver ensures
99+ * that the GPU is kept as busy as possible. In the case where a single context
100+ * completes but a second context is still executing, the request for this second
101+ * context will be at the head of the queue when we remove the first one. This
102+ * request will then be resubmitted along with a new request for a different context,
103+ * which will cause the hardware to continue executing the second request and queue
104+ * the new request (the GPU detects the condition of a context getting preempted
105+ * with the same context and optimizes the context switch flow by not doing
106+ * preemption, but just sampling the new tail pointer).
107+ *
108+ */
109+#include <linux/interrupt.h>
110+
111+#include "i915_drv.h"
112+#include "i915_trace.h"
113+#include "i915_vgpu.h"
114+#include "gen8_engine_cs.h"
115+#include "intel_breadcrumbs.h"
116+#include "intel_context.h"
117+#include "intel_engine_pm.h"
118+#include "intel_execlists_submission.h"
119+#include "intel_gt.h"
120+#include "intel_gt_pm.h"
121+#include "intel_gt_requests.h"
122+#include "intel_lrc.h"
123+#include "intel_lrc_reg.h"
124+#include "intel_mocs.h"
125+#include "intel_reset.h"
126+#include "intel_ring.h"
127+#include "intel_workarounds.h"
128+#include "shmem_utils.h"
129+
130+#define RING_EXECLIST_QFULL (1 << 0x2)
131+#define RING_EXECLIST1_VALID (1 << 0x3)
132+#define RING_EXECLIST0_VALID (1 << 0x4)
133+#define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE)
134+#define RING_EXECLIST1_ACTIVE (1 << 0x11)
135+#define RING_EXECLIST0_ACTIVE (1 << 0x12)
136+
137+#define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0)
138+#define GEN8_CTX_STATUS_PREEMPTED (1 << 1)
139+#define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2)
140+#define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3)
141+#define GEN8_CTX_STATUS_COMPLETE (1 << 4)
142+#define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15)
143+
144+#define GEN8_CTX_STATUS_COMPLETED_MASK \
145+ (GEN8_CTX_STATUS_COMPLETE | GEN8_CTX_STATUS_PREEMPTED)
146+
147+#define GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE (0x1) /* lower csb dword */
148+#define GEN12_CTX_SWITCH_DETAIL(csb_dw) ((csb_dw) & 0xF) /* upper csb dword */
149+#define GEN12_CSB_SW_CTX_ID_MASK GENMASK(25, 15)
150+#define GEN12_IDLE_CTX_ID 0x7FF
151+#define GEN12_CSB_CTX_VALID(csb_dw) \
152+ (FIELD_GET(GEN12_CSB_SW_CTX_ID_MASK, csb_dw) != GEN12_IDLE_CTX_ID)
153+
154+/* Typical size of the average request (2 pipecontrols and a MI_BB) */
155+#define EXECLISTS_REQUEST_SIZE 64 /* bytes */
156+
157+struct virtual_engine {
158+ struct intel_engine_cs base;
159+ struct intel_context context;
160+ struct rcu_work rcu;
161+
162+ /*
163+ * We allow only a single request through the virtual engine at a time
164+ * (each request in the timeline waits for the completion fence of
165+ * the previous before being submitted). By restricting ourselves to
166+ * only submitting a single request, each request is placed on to a
167+ * physical to maximise load spreading (by virtue of the late greedy
168+ * scheduling -- each real engine takes the next available request
169+ * upon idling).
170+ */
171+ struct i915_request *request;
172+
173+ /*
174+ * We keep a rbtree of available virtual engines inside each physical
175+ * engine, sorted by priority. Here we preallocate the nodes we need
176+ * for the virtual engine, indexed by physical_engine->id.
177+ */
178+ struct ve_node {
179+ struct rb_node rb;
180+ int prio;
181+ } nodes[I915_NUM_ENGINES];
182+
183+ /*
184+ * Keep track of bonded pairs -- restrictions upon on our selection
185+ * of physical engines any particular request may be submitted to.
186+ * If we receive a submit-fence from a master engine, we will only
187+ * use one of sibling_mask physical engines.
188+ */
189+ struct ve_bond {
190+ const struct intel_engine_cs *master;
191+ intel_engine_mask_t sibling_mask;
192+ } *bonds;
193+ unsigned int num_bonds;
194+
195+ /* And finally, which physical engines this virtual engine maps onto. */
196+ unsigned int num_siblings;
197+ struct intel_engine_cs *siblings[];
198+};
199+
200+static struct virtual_engine *to_virtual_engine(struct intel_engine_cs *engine)
201+{
202+ GEM_BUG_ON(!intel_engine_is_virtual(engine));
203+ return container_of(engine, struct virtual_engine, base);
204+}
205+
206+static struct i915_request *
207+__active_request(const struct intel_timeline * const tl,
208+ struct i915_request *rq,
209+ int error)
210+{
211+ struct i915_request *active = rq;
212+
213+ list_for_each_entry_from_reverse(rq, &tl->requests, link) {
214+ if (__i915_request_is_complete(rq))
215+ break;
216+
217+ if (error) {
218+ i915_request_set_error_once(rq, error);
219+ __i915_request_skip(rq);
220+ }
221+ active = rq;
222+ }
223+
224+ return active;
225+}
226+
227+static struct i915_request *
228+active_request(const struct intel_timeline * const tl, struct i915_request *rq)
229+{
230+ return __active_request(tl, rq, 0);
231+}
232+
233+static inline void
234+ring_set_paused(const struct intel_engine_cs *engine, int state)
235+{
236+ /*
237+ * We inspect HWS_PREEMPT with a semaphore inside
238+ * engine->emit_fini_breadcrumb. If the dword is true,
239+ * the ring is paused as the semaphore will busywait
240+ * until the dword is false.
241+ */
242+ engine->status_page.addr[I915_GEM_HWS_PREEMPT] = state;
243+ if (state)
244+ wmb();
245+}
246+
247+static inline struct i915_priolist *to_priolist(struct rb_node *rb)
248+{
249+ return rb_entry(rb, struct i915_priolist, node);
250+}
251+
252+static inline int rq_prio(const struct i915_request *rq)
253+{
254+ return READ_ONCE(rq->sched.attr.priority);
255+}
256+
257+static int effective_prio(const struct i915_request *rq)
258+{
259+ int prio = rq_prio(rq);
260+
261+ /*
262+ * If this request is special and must not be interrupted at any
263+ * cost, so be it. Note we are only checking the most recent request
264+ * in the context and so may be masking an earlier vip request. It
265+ * is hoped that under the conditions where nopreempt is used, this
266+ * will not matter (i.e. all requests to that context will be
267+ * nopreempt for as long as desired).
268+ */
269+ if (i915_request_has_nopreempt(rq))
270+ prio = I915_PRIORITY_UNPREEMPTABLE;
271+
272+ return prio;
273+}
274+
275+static int queue_prio(const struct intel_engine_execlists *execlists)
276+{
277+ struct i915_priolist *p;
278+ struct rb_node *rb;
279+
280+ rb = rb_first_cached(&execlists->queue);
281+ if (!rb)
282+ return INT_MIN;
283+
284+ /*
285+ * As the priolist[] are inverted, with the highest priority in [0],
286+ * we have to flip the index value to become priority.
287+ */
288+ p = to_priolist(rb);
289+ if (!I915_USER_PRIORITY_SHIFT)
290+ return p->priority;
291+
292+ return ((p->priority + 1) << I915_USER_PRIORITY_SHIFT) - ffs(p->used);
293+}
294+
295+static int virtual_prio(const struct intel_engine_execlists *el)
296+{
297+ struct rb_node *rb = rb_first_cached(&el->virtual);
298+
299+ return rb ? rb_entry(rb, struct ve_node, rb)->prio : INT_MIN;
300+}
301+
302+static inline bool need_preempt(const struct intel_engine_cs *engine,
303+ const struct i915_request *rq)
304+{
305+ int last_prio;
306+
307+ if (!intel_engine_has_semaphores(engine))
308+ return false;
309+
310+ /*
311+ * Check if the current priority hint merits a preemption attempt.
312+ *
313+ * We record the highest value priority we saw during rescheduling
314+ * prior to this dequeue, therefore we know that if it is strictly
315+ * less than the current tail of ESLP[0], we do not need to force
316+ * a preempt-to-idle cycle.
317+ *
318+ * However, the priority hint is a mere hint that we may need to
319+ * preempt. If that hint is stale or we may be trying to preempt
320+ * ourselves, ignore the request.
321+ *
322+ * More naturally we would write
323+ * prio >= max(0, last);
324+ * except that we wish to prevent triggering preemption at the same
325+ * priority level: the task that is running should remain running
326+ * to preserve FIFO ordering of dependencies.
327+ */
328+ last_prio = max(effective_prio(rq), I915_PRIORITY_NORMAL - 1);
329+ if (engine->execlists.queue_priority_hint <= last_prio)
330+ return false;
331+
332+ /*
333+ * Check against the first request in ELSP[1], it will, thanks to the
334+ * power of PI, be the highest priority of that context.
335+ */
336+ if (!list_is_last(&rq->sched.link, &engine->active.requests) &&
337+ rq_prio(list_next_entry(rq, sched.link)) > last_prio)
338+ return true;
339+
340+ /*
341+ * If the inflight context did not trigger the preemption, then maybe
342+ * it was the set of queued requests? Pick the highest priority in
343+ * the queue (the first active priolist) and see if it deserves to be
344+ * running instead of ELSP[0].
345+ *
346+ * The highest priority request in the queue can not be either
347+ * ELSP[0] or ELSP[1] as, thanks again to PI, if it was the same
348+ * context, it's priority would not exceed ELSP[0] aka last_prio.
349+ */
350+ return max(virtual_prio(&engine->execlists),
351+ queue_prio(&engine->execlists)) > last_prio;
352+}
353+
354+__maybe_unused static inline bool
355+assert_priority_queue(const struct i915_request *prev,
356+ const struct i915_request *next)
357+{
358+ /*
359+ * Without preemption, the prev may refer to the still active element
360+ * which we refuse to let go.
361+ *
362+ * Even with preemption, there are times when we think it is better not
363+ * to preempt and leave an ostensibly lower priority request in flight.
364+ */
365+ if (i915_request_is_active(prev))
366+ return true;
367+
368+ return rq_prio(prev) >= rq_prio(next);
369+}
370+
371+static struct i915_request *
372+__unwind_incomplete_requests(struct intel_engine_cs *engine)
373+{
374+ struct i915_request *rq, *rn, *active = NULL;
375+ struct list_head *pl;
376+ int prio = I915_PRIORITY_INVALID;
377+
378+ lockdep_assert_held(&engine->active.lock);
379+
380+ list_for_each_entry_safe_reverse(rq, rn,
381+ &engine->active.requests,
382+ sched.link) {
383+ if (__i915_request_is_complete(rq)) {
384+ list_del_init(&rq->sched.link);
385+ continue;
386+ }
387+
388+ __i915_request_unsubmit(rq);
389+
390+ GEM_BUG_ON(rq_prio(rq) == I915_PRIORITY_INVALID);
391+ if (rq_prio(rq) != prio) {
392+ prio = rq_prio(rq);
393+ pl = i915_sched_lookup_priolist(engine, prio);
394+ }
395+ GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
396+
397+ list_move(&rq->sched.link, pl);
398+ set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
399+
400+ /* Check in case we rollback so far we wrap [size/2] */
401+ if (intel_ring_direction(rq->ring,
402+ rq->tail,
403+ rq->ring->tail + 8) > 0)
404+ rq->context->lrc.desc |= CTX_DESC_FORCE_RESTORE;
405+
406+ active = rq;
407+ }
408+
409+ return active;
410+}
411+
412+struct i915_request *
413+execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists)
414+{
415+ struct intel_engine_cs *engine =
416+ container_of(execlists, typeof(*engine), execlists);
417+
418+ return __unwind_incomplete_requests(engine);
419+}
420+
421+static inline void
422+execlists_context_status_change(struct i915_request *rq, unsigned long status)
423+{
424+ /*
425+ * Only used when GVT-g is enabled now. When GVT-g is disabled,
426+ * The compiler should eliminate this function as dead-code.
427+ */
428+ if (!IS_ENABLED(CONFIG_DRM_I915_GVT))
429+ return;
430+
431+ atomic_notifier_call_chain(&rq->engine->context_status_notifier,
432+ status, rq);
433+}
434+
435+static void intel_engine_context_in(struct intel_engine_cs *engine)
436+{
437+ unsigned long flags;
438+
439+ if (atomic_add_unless(&engine->stats.active, 1, 0))
440+ return;
441+
442+ write_seqlock_irqsave(&engine->stats.lock, flags);
443+ if (!atomic_add_unless(&engine->stats.active, 1, 0)) {
444+ engine->stats.start = ktime_get();
445+ atomic_inc(&engine->stats.active);
446+ }
447+ write_sequnlock_irqrestore(&engine->stats.lock, flags);
448+}
449+
450+static void intel_engine_context_out(struct intel_engine_cs *engine)
451+{
452+ unsigned long flags;
453+
454+ GEM_BUG_ON(!atomic_read(&engine->stats.active));
455+
456+ if (atomic_add_unless(&engine->stats.active, -1, 1))
457+ return;
458+
459+ write_seqlock_irqsave(&engine->stats.lock, flags);
460+ if (atomic_dec_and_test(&engine->stats.active)) {
461+ engine->stats.total =
462+ ktime_add(engine->stats.total,
463+ ktime_sub(ktime_get(), engine->stats.start));
464+ }
465+ write_sequnlock_irqrestore(&engine->stats.lock, flags);
466+}
467+
468+static void reset_active(struct i915_request *rq,
469+ struct intel_engine_cs *engine)
470+{
471+ struct intel_context * const ce = rq->context;
472+ u32 head;
473+
474+ /*
475+ * The executing context has been cancelled. We want to prevent
476+ * further execution along this context and propagate the error on
477+ * to anything depending on its results.
478+ *
479+ * In __i915_request_submit(), we apply the -EIO and remove the
480+ * requests' payloads for any banned requests. But first, we must
481+ * rewind the context back to the start of the incomplete request so
482+ * that we do not jump back into the middle of the batch.
483+ *
484+ * We preserve the breadcrumbs and semaphores of the incomplete
485+ * requests so that inter-timeline dependencies (i.e other timelines)
486+ * remain correctly ordered. And we defer to __i915_request_submit()
487+ * so that all asynchronous waits are correctly handled.
488+ */
489+ ENGINE_TRACE(engine, "{ reset rq=%llx:%lld }\n",
490+ rq->fence.context, rq->fence.seqno);
491+
492+ /* On resubmission of the active request, payload will be scrubbed */
493+ if (__i915_request_is_complete(rq))
494+ head = rq->tail;
495+ else
496+ head = __active_request(ce->timeline, rq, -EIO)->head;
497+ head = intel_ring_wrap(ce->ring, head);
498+
499+ /* Scrub the context image to prevent replaying the previous batch */
500+ lrc_init_regs(ce, engine, true);
501+
502+ /* We've switched away, so this should be a no-op, but intent matters */
503+ ce->lrc.lrca = lrc_update_regs(ce, engine, head);
504+}
505+
506+static inline struct intel_engine_cs *
507+__execlists_schedule_in(struct i915_request *rq)
508+{
509+ struct intel_engine_cs * const engine = rq->engine;
510+ struct intel_context * const ce = rq->context;
511+
512+ intel_context_get(ce);
513+
514+ if (unlikely(intel_context_is_closed(ce) &&
515+ !intel_engine_has_heartbeat(engine)))
516+ intel_context_set_banned(ce);
517+
518+ if (unlikely(intel_context_is_banned(ce)))
519+ reset_active(rq, engine);
520+
521+ if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
522+ lrc_check_regs(ce, engine, "before");
523+
524+ if (ce->tag) {
525+ /* Use a fixed tag for OA and friends */
526+ GEM_BUG_ON(ce->tag <= BITS_PER_LONG);
527+ ce->lrc.ccid = ce->tag;
528+ } else {
529+ /* We don't need a strict matching tag, just different values */
530+ unsigned int tag = __ffs(engine->context_tag);
531+
532+ GEM_BUG_ON(tag >= BITS_PER_LONG);
533+ __clear_bit(tag, &engine->context_tag);
534+ ce->lrc.ccid = (1 + tag) << (GEN11_SW_CTX_ID_SHIFT - 32);
535+
536+ BUILD_BUG_ON(BITS_PER_LONG > GEN12_MAX_CONTEXT_HW_ID);
537+ }
538+
539+ ce->lrc.ccid |= engine->execlists.ccid;
540+
541+ __intel_gt_pm_get(engine->gt);
542+ if (engine->fw_domain && !atomic_fetch_inc(&engine->fw_active))
543+ intel_uncore_forcewake_get(engine->uncore, engine->fw_domain);
544+ execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_IN);
545+ intel_engine_context_in(engine);
546+
547+ CE_TRACE(ce, "schedule-in, ccid:%x\n", ce->lrc.ccid);
548+
549+ return engine;
550+}
551+
552+static inline void execlists_schedule_in(struct i915_request *rq, int idx)
553+{
554+ struct intel_context * const ce = rq->context;
555+ struct intel_engine_cs *old;
556+
557+ GEM_BUG_ON(!intel_engine_pm_is_awake(rq->engine));
558+ trace_i915_request_in(rq, idx);
559+
560+ old = ce->inflight;
561+ if (!old)
562+ old = __execlists_schedule_in(rq);
563+ WRITE_ONCE(ce->inflight, ptr_inc(old));
564+
565+ GEM_BUG_ON(intel_context_inflight(ce) != rq->engine);
566+}
567+
568+static void
569+resubmit_virtual_request(struct i915_request *rq, struct virtual_engine *ve)
570+{
571+ struct intel_engine_cs *engine = rq->engine;
572+
573+ spin_lock_irq(&engine->active.lock);
574+
575+ clear_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
576+ WRITE_ONCE(rq->engine, &ve->base);
577+ ve->base.submit_request(rq);
578+
579+ spin_unlock_irq(&engine->active.lock);
580+}
581+
582+static void kick_siblings(struct i915_request *rq, struct intel_context *ce)
583+{
584+ struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
585+ struct intel_engine_cs *engine = rq->engine;
586+
587+ /*
588+ * After this point, the rq may be transferred to a new sibling, so
589+ * before we clear ce->inflight make sure that the context has been
590+ * removed from the b->signalers and furthermore we need to make sure
591+ * that the concurrent iterator in signal_irq_work is no longer
592+ * following ce->signal_link.
593+ */
594+ if (!list_empty(&ce->signals))
595+ intel_context_remove_breadcrumbs(ce, engine->breadcrumbs);
596+
597+ /*
598+ * This engine is now too busy to run this virtual request, so
599+ * see if we can find an alternative engine for it to execute on.
600+ * Once a request has become bonded to this engine, we treat it the
601+ * same as other native request.
602+ */
603+ if (i915_request_in_priority_queue(rq) &&
604+ rq->execution_mask != engine->mask)
605+ resubmit_virtual_request(rq, ve);
606+
607+ if (READ_ONCE(ve->request))
608+ tasklet_hi_schedule(&ve->base.execlists.tasklet);
609+}
610+
611+static inline void __execlists_schedule_out(struct i915_request *rq)
612+{
613+ struct intel_context * const ce = rq->context;
614+ struct intel_engine_cs * const engine = rq->engine;
615+ unsigned int ccid;
616+
617+ /*
618+ * NB process_csb() is not under the engine->active.lock and hence
619+ * schedule_out can race with schedule_in meaning that we should
620+ * refrain from doing non-trivial work here.
621+ */
622+
623+ CE_TRACE(ce, "schedule-out, ccid:%x\n", ce->lrc.ccid);
624+
625+ if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
626+ lrc_check_regs(ce, engine, "after");
627+
628+ /*
629+ * If we have just completed this context, the engine may now be
630+ * idle and we want to re-enter powersaving.
631+ */
632+ if (intel_timeline_is_last(ce->timeline, rq) &&
633+ __i915_request_is_complete(rq))
634+ intel_engine_add_retire(engine, ce->timeline);
635+
636+ ccid = ce->lrc.ccid;
637+ ccid >>= GEN11_SW_CTX_ID_SHIFT - 32;
638+ ccid &= GEN12_MAX_CONTEXT_HW_ID;
639+ if (ccid < BITS_PER_LONG) {
640+ GEM_BUG_ON(ccid == 0);
641+ GEM_BUG_ON(test_bit(ccid - 1, &engine->context_tag));
642+ __set_bit(ccid - 1, &engine->context_tag);
643+ }
644+
645+ lrc_update_runtime(ce);
646+ intel_engine_context_out(engine);
647+ execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_OUT);
648+ if (engine->fw_domain && !atomic_dec_return(&engine->fw_active))
649+ intel_uncore_forcewake_put(engine->uncore, engine->fw_domain);
650+ intel_gt_pm_put_async(engine->gt);
651+
652+ /*
653+ * If this is part of a virtual engine, its next request may
654+ * have been blocked waiting for access to the active context.
655+ * We have to kick all the siblings again in case we need to
656+ * switch (e.g. the next request is not runnable on this
657+ * engine). Hopefully, we will already have submitted the next
658+ * request before the tasklet runs and do not need to rebuild
659+ * each virtual tree and kick everyone again.
660+ */
661+ if (ce->engine != engine)
662+ kick_siblings(rq, ce);
663+}
664+
665+static inline void
666+execlists_schedule_out(struct i915_request *rq)
667+{
668+ struct intel_context * const ce = rq->context;
669+
670+ trace_i915_request_out(rq);
671+
672+ GEM_BUG_ON(!ce->inflight);
673+ ce->inflight = ptr_dec(ce->inflight);
674+ if (!__intel_context_inflight_count(ce->inflight)) {
675+ GEM_BUG_ON(ce->inflight != rq->engine);
676+ __execlists_schedule_out(rq);
677+ WRITE_ONCE(ce->inflight, NULL);
678+ intel_context_put(ce);
679+ }
680+
681+ i915_request_put(rq);
682+}
683+
684+static u64 execlists_update_context(struct i915_request *rq)
685+{
686+ struct intel_context *ce = rq->context;
687+ u64 desc = ce->lrc.desc;
688+ u32 tail, prev;
689+
690+ /*
691+ * WaIdleLiteRestore:bdw,skl
692+ *
693+ * We should never submit the context with the same RING_TAIL twice
694+ * just in case we submit an empty ring, which confuses the HW.
695+ *
696+ * We append a couple of NOOPs (gen8_emit_wa_tail) after the end of
697+ * the normal request to be able to always advance the RING_TAIL on
698+ * subsequent resubmissions (for lite restore). Should that fail us,
699+ * and we try and submit the same tail again, force the context
700+ * reload.
701+ *
702+ * If we need to return to a preempted context, we need to skip the
703+ * lite-restore and force it to reload the RING_TAIL. Otherwise, the
704+ * HW has a tendency to ignore us rewinding the TAIL to the end of
705+ * an earlier request.
706+ */
707+ GEM_BUG_ON(ce->lrc_reg_state[CTX_RING_TAIL] != rq->ring->tail);
708+ prev = rq->ring->tail;
709+ tail = intel_ring_set_tail(rq->ring, rq->tail);
710+ if (unlikely(intel_ring_direction(rq->ring, tail, prev) <= 0))
711+ desc |= CTX_DESC_FORCE_RESTORE;
712+ ce->lrc_reg_state[CTX_RING_TAIL] = tail;
713+ rq->tail = rq->wa_tail;
714+
715+ /*
716+ * Make sure the context image is complete before we submit it to HW.
717+ *
718+ * Ostensibly, writes (including the WCB) should be flushed prior to
719+ * an uncached write such as our mmio register access, the empirical
720+ * evidence (esp. on Braswell) suggests that the WC write into memory
721+ * may not be visible to the HW prior to the completion of the UC
722+ * register write and that we may begin execution from the context
723+ * before its image is complete leading to invalid PD chasing.
724+ */
725+ wmb();
726+
727+ ce->lrc.desc &= ~CTX_DESC_FORCE_RESTORE;
728+ return desc;
729+}
730+
731+static inline void write_desc(struct intel_engine_execlists *execlists, u64 desc, u32 port)
732+{
733+ if (execlists->ctrl_reg) {
734+ writel(lower_32_bits(desc), execlists->submit_reg + port * 2);
735+ writel(upper_32_bits(desc), execlists->submit_reg + port * 2 + 1);
736+ } else {
737+ writel(upper_32_bits(desc), execlists->submit_reg);
738+ writel(lower_32_bits(desc), execlists->submit_reg);
739+ }
740+}
741+
742+static __maybe_unused char *
743+dump_port(char *buf, int buflen, const char *prefix, struct i915_request *rq)
744+{
745+ if (!rq)
746+ return "";
747+
748+ snprintf(buf, buflen, "%sccid:%x %llx:%lld%s prio %d",
749+ prefix,
750+ rq->context->lrc.ccid,
751+ rq->fence.context, rq->fence.seqno,
752+ __i915_request_is_complete(rq) ? "!" :
753+ __i915_request_has_started(rq) ? "*" :
754+ "",
755+ rq_prio(rq));
756+
757+ return buf;
758+}
759+
760+static __maybe_unused void
761+trace_ports(const struct intel_engine_execlists *execlists,
762+ const char *msg,
763+ struct i915_request * const *ports)
764+{
765+ const struct intel_engine_cs *engine =
766+ container_of(execlists, typeof(*engine), execlists);
767+ char __maybe_unused p0[40], p1[40];
768+
769+ if (!ports[0])
770+ return;
771+
772+ ENGINE_TRACE(engine, "%s { %s%s }\n", msg,
773+ dump_port(p0, sizeof(p0), "", ports[0]),
774+ dump_port(p1, sizeof(p1), ", ", ports[1]));
775+}
776+
777+static inline bool
778+reset_in_progress(const struct intel_engine_execlists *execlists)
779+{
780+ return unlikely(!__tasklet_is_enabled(&execlists->tasklet));
781+}
782+
783+static __maybe_unused bool
784+assert_pending_valid(const struct intel_engine_execlists *execlists,
785+ const char *msg)
786+{
787+ struct intel_engine_cs *engine =
788+ container_of(execlists, typeof(*engine), execlists);
789+ struct i915_request * const *port, *rq;
790+ struct intel_context *ce = NULL;
791+ bool sentinel = false;
792+ u32 ccid = -1;
793+
794+ trace_ports(execlists, msg, execlists->pending);
795+
796+ /* We may be messing around with the lists during reset, lalala */
797+ if (reset_in_progress(execlists))
798+ return true;
799+
800+ if (!execlists->pending[0]) {
801+ GEM_TRACE_ERR("%s: Nothing pending for promotion!\n",
802+ engine->name);
803+ return false;
804+ }
805+
806+ if (execlists->pending[execlists_num_ports(execlists)]) {
807+ GEM_TRACE_ERR("%s: Excess pending[%d] for promotion!\n",
808+ engine->name, execlists_num_ports(execlists));
809+ return false;
810+ }
811+
812+ for (port = execlists->pending; (rq = *port); port++) {
813+ unsigned long flags;
814+ bool ok = true;
815+
816+ GEM_BUG_ON(!kref_read(&rq->fence.refcount));
817+ GEM_BUG_ON(!i915_request_is_active(rq));
818+
819+ if (ce == rq->context) {
820+ GEM_TRACE_ERR("%s: Dup context:%llx in pending[%zd]\n",
821+ engine->name,
822+ ce->timeline->fence_context,
823+ port - execlists->pending);
824+ return false;
825+ }
826+ ce = rq->context;
827+
828+ if (ccid == ce->lrc.ccid) {
829+ GEM_TRACE_ERR("%s: Dup ccid:%x context:%llx in pending[%zd]\n",
830+ engine->name,
831+ ccid, ce->timeline->fence_context,
832+ port - execlists->pending);
833+ return false;
834+ }
835+ ccid = ce->lrc.ccid;
836+
837+ /*
838+ * Sentinels are supposed to be the last request so they flush
839+ * the current execution off the HW. Check that they are the only
840+ * request in the pending submission.
841+ */
842+ if (sentinel) {
843+ GEM_TRACE_ERR("%s: context:%llx after sentinel in pending[%zd]\n",
844+ engine->name,
845+ ce->timeline->fence_context,
846+ port - execlists->pending);
847+ return false;
848+ }
849+ sentinel = i915_request_has_sentinel(rq);
850+
851+ /*
852+ * We want virtual requests to only be in the first slot so
853+ * that they are never stuck behind a hog and can be immediately
854+ * transferred onto the next idle engine.
855+ */
856+ if (rq->execution_mask != engine->mask &&
857+ port != execlists->pending) {
858+ GEM_TRACE_ERR("%s: virtual engine:%llx not in prime position[%zd]\n",
859+ engine->name,
860+ ce->timeline->fence_context,
861+ port - execlists->pending);
862+ return false;
863+ }
864+
865+ /* Hold tightly onto the lock to prevent concurrent retires! */
866+ if (!spin_trylock_irqsave(&rq->lock, flags))
867+ continue;
868+
869+ if (__i915_request_is_complete(rq))
870+ goto unlock;
871+
872+ if (i915_active_is_idle(&ce->active) &&
873+ !intel_context_is_barrier(ce)) {
874+ GEM_TRACE_ERR("%s: Inactive context:%llx in pending[%zd]\n",
875+ engine->name,
876+ ce->timeline->fence_context,
877+ port - execlists->pending);
878+ ok = false;
879+ goto unlock;
880+ }
881+
882+ if (!i915_vma_is_pinned(ce->state)) {
883+ GEM_TRACE_ERR("%s: Unpinned context:%llx in pending[%zd]\n",
884+ engine->name,
885+ ce->timeline->fence_context,
886+ port - execlists->pending);
887+ ok = false;
888+ goto unlock;
889+ }
890+
891+ if (!i915_vma_is_pinned(ce->ring->vma)) {
892+ GEM_TRACE_ERR("%s: Unpinned ring:%llx in pending[%zd]\n",
893+ engine->name,
894+ ce->timeline->fence_context,
895+ port - execlists->pending);
896+ ok = false;
897+ goto unlock;
898+ }
899+
900+unlock:
901+ spin_unlock_irqrestore(&rq->lock, flags);
902+ if (!ok)
903+ return false;
904+ }
905+
906+ return ce;
907+}
908+
909+static void execlists_submit_ports(struct intel_engine_cs *engine)
910+{
911+ struct intel_engine_execlists *execlists = &engine->execlists;
912+ unsigned int n;
913+
914+ GEM_BUG_ON(!assert_pending_valid(execlists, "submit"));
915+
916+ /*
917+ * We can skip acquiring intel_runtime_pm_get() here as it was taken
918+ * on our behalf by the request (see i915_gem_mark_busy()) and it will
919+ * not be relinquished until the device is idle (see
920+ * i915_gem_idle_work_handler()). As a precaution, we make sure
921+ * that all ELSP are drained i.e. we have processed the CSB,
922+ * before allowing ourselves to idle and calling intel_runtime_pm_put().
923+ */
924+ GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
925+
926+ /*
927+ * ELSQ note: the submit queue is not cleared after being submitted
928+ * to the HW so we need to make sure we always clean it up. This is
929+ * currently ensured by the fact that we always write the same number
930+ * of elsq entries, keep this in mind before changing the loop below.
931+ */
932+ for (n = execlists_num_ports(execlists); n--; ) {
933+ struct i915_request *rq = execlists->pending[n];
934+
935+ write_desc(execlists,
936+ rq ? execlists_update_context(rq) : 0,
937+ n);
938+ }
939+
940+ /* we need to manually load the submit queue */
941+ if (execlists->ctrl_reg)
942+ writel(EL_CTRL_LOAD, execlists->ctrl_reg);
943+}
944+
945+static bool ctx_single_port_submission(const struct intel_context *ce)
946+{
947+ return (IS_ENABLED(CONFIG_DRM_I915_GVT) &&
948+ intel_context_force_single_submission(ce));
949+}
950+
951+static bool can_merge_ctx(const struct intel_context *prev,
952+ const struct intel_context *next)
953+{
954+ if (prev != next)
955+ return false;
956+
957+ if (ctx_single_port_submission(prev))
958+ return false;
959+
960+ return true;
961+}
962+
963+static unsigned long i915_request_flags(const struct i915_request *rq)
964+{
965+ return READ_ONCE(rq->fence.flags);
966+}
967+
968+static bool can_merge_rq(const struct i915_request *prev,
969+ const struct i915_request *next)
970+{
971+ GEM_BUG_ON(prev == next);
972+ GEM_BUG_ON(!assert_priority_queue(prev, next));
973+
974+ /*
975+ * We do not submit known completed requests. Therefore if the next
976+ * request is already completed, we can pretend to merge it in
977+ * with the previous context (and we will skip updating the ELSP
978+ * and tracking). Thus hopefully keeping the ELSP full with active
979+ * contexts, despite the best efforts of preempt-to-busy to confuse
980+ * us.
981+ */
982+ if (__i915_request_is_complete(next))
983+ return true;
984+
985+ if (unlikely((i915_request_flags(prev) ^ i915_request_flags(next)) &
986+ (BIT(I915_FENCE_FLAG_NOPREEMPT) |
987+ BIT(I915_FENCE_FLAG_SENTINEL))))
988+ return false;
989+
990+ if (!can_merge_ctx(prev->context, next->context))
991+ return false;
992+
993+ GEM_BUG_ON(i915_seqno_passed(prev->fence.seqno, next->fence.seqno));
994+ return true;
995+}
996+
997+static bool virtual_matches(const struct virtual_engine *ve,
998+ const struct i915_request *rq,
999+ const struct intel_engine_cs *engine)
1000+{
1001+ const struct intel_engine_cs *inflight;
1002+
1003+ if (!rq)
1004+ return false;
1005+
1006+ if (!(rq->execution_mask & engine->mask)) /* We peeked too soon! */
1007+ return false;
1008+
1009+ /*
1010+ * We track when the HW has completed saving the context image
1011+ * (i.e. when we have seen the final CS event switching out of
1012+ * the context) and must not overwrite the context image before
1013+ * then. This restricts us to only using the active engine
1014+ * while the previous virtualized request is inflight (so
1015+ * we reuse the register offsets). This is a very small
1016+ * hystersis on the greedy seelction algorithm.
1017+ */
1018+ inflight = intel_context_inflight(&ve->context);
1019+ if (inflight && inflight != engine)
1020+ return false;
1021+
1022+ return true;
1023+}
1024+
1025+static struct virtual_engine *
1026+first_virtual_engine(struct intel_engine_cs *engine)
1027+{
1028+ struct intel_engine_execlists *el = &engine->execlists;
1029+ struct rb_node *rb = rb_first_cached(&el->virtual);
1030+
1031+ while (rb) {
1032+ struct virtual_engine *ve =
1033+ rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
1034+ struct i915_request *rq = READ_ONCE(ve->request);
1035+
1036+ /* lazily cleanup after another engine handled rq */
1037+ if (!rq || !virtual_matches(ve, rq, engine)) {
1038+ rb_erase_cached(rb, &el->virtual);
1039+ RB_CLEAR_NODE(rb);
1040+ rb = rb_first_cached(&el->virtual);
1041+ continue;
1042+ }
1043+
1044+ return ve;
1045+ }
1046+
1047+ return NULL;
1048+}
1049+
1050+static void virtual_xfer_context(struct virtual_engine *ve,
1051+ struct intel_engine_cs *engine)
1052+{
1053+ unsigned int n;
1054+
1055+ if (likely(engine == ve->siblings[0]))
1056+ return;
1057+
1058+ GEM_BUG_ON(READ_ONCE(ve->context.inflight));
1059+ if (!intel_engine_has_relative_mmio(engine))
1060+ lrc_update_offsets(&ve->context, engine);
1061+
1062+ /*
1063+ * Move the bound engine to the top of the list for
1064+ * future execution. We then kick this tasklet first
1065+ * before checking others, so that we preferentially
1066+ * reuse this set of bound registers.
1067+ */
1068+ for (n = 1; n < ve->num_siblings; n++) {
1069+ if (ve->siblings[n] == engine) {
1070+ swap(ve->siblings[n], ve->siblings[0]);
1071+ break;
1072+ }
1073+ }
1074+}
1075+
1076+static void defer_request(struct i915_request *rq, struct list_head * const pl)
1077+{
1078+ LIST_HEAD(list);
1079+
1080+ /*
1081+ * We want to move the interrupted request to the back of
1082+ * the round-robin list (i.e. its priority level), but
1083+ * in doing so, we must then move all requests that were in
1084+ * flight and were waiting for the interrupted request to
1085+ * be run after it again.
1086+ */
1087+ do {
1088+ struct i915_dependency *p;
1089+
1090+ GEM_BUG_ON(i915_request_is_active(rq));
1091+ list_move_tail(&rq->sched.link, pl);
1092+
1093+ for_each_waiter(p, rq) {
1094+ struct i915_request *w =
1095+ container_of(p->waiter, typeof(*w), sched);
1096+
1097+ if (p->flags & I915_DEPENDENCY_WEAK)
1098+ continue;
1099+
1100+ /* Leave semaphores spinning on the other engines */
1101+ if (w->engine != rq->engine)
1102+ continue;
1103+
1104+ /* No waiter should start before its signaler */
1105+ GEM_BUG_ON(i915_request_has_initial_breadcrumb(w) &&
1106+ __i915_request_has_started(w) &&
1107+ !__i915_request_is_complete(rq));
1108+
1109+ GEM_BUG_ON(i915_request_is_active(w));
1110+ if (!i915_request_is_ready(w))
1111+ continue;
1112+
1113+ if (rq_prio(w) < rq_prio(rq))
1114+ continue;
1115+
1116+ GEM_BUG_ON(rq_prio(w) > rq_prio(rq));
1117+ list_move_tail(&w->sched.link, &list);
1118+ }
1119+
1120+ rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
1121+ } while (rq);
1122+}
1123+
1124+static void defer_active(struct intel_engine_cs *engine)
1125+{
1126+ struct i915_request *rq;
1127+
1128+ rq = __unwind_incomplete_requests(engine);
1129+ if (!rq)
1130+ return;
1131+
1132+ defer_request(rq, i915_sched_lookup_priolist(engine, rq_prio(rq)));
1133+}
1134+
1135+static bool
1136+timeslice_yield(const struct intel_engine_execlists *el,
1137+ const struct i915_request *rq)
1138+{
1139+ /*
1140+ * Once bitten, forever smitten!
1141+ *
1142+ * If the active context ever busy-waited on a semaphore,
1143+ * it will be treated as a hog until the end of its timeslice (i.e.
1144+ * until it is scheduled out and replaced by a new submission,
1145+ * possibly even its own lite-restore). The HW only sends an interrupt
1146+ * on the first miss, and we do know if that semaphore has been
1147+ * signaled, or even if it is now stuck on another semaphore. Play
1148+ * safe, yield if it might be stuck -- it will be given a fresh
1149+ * timeslice in the near future.
1150+ */
1151+ return rq->context->lrc.ccid == READ_ONCE(el->yield);
1152+}
1153+
1154+static bool needs_timeslice(const struct intel_engine_cs *engine,
1155+ const struct i915_request *rq)
1156+{
1157+ if (!intel_engine_has_timeslices(engine))
1158+ return false;
1159+
1160+ /* If not currently active, or about to switch, wait for next event */
1161+ if (!rq || __i915_request_is_complete(rq))
1162+ return false;
1163+
1164+ /* We do not need to start the timeslice until after the ACK */
1165+ if (READ_ONCE(engine->execlists.pending[0]))
1166+ return false;
1167+
1168+ /* If ELSP[1] is occupied, always check to see if worth slicing */
1169+ if (!list_is_last_rcu(&rq->sched.link, &engine->active.requests)) {
1170+ ENGINE_TRACE(engine, "timeslice required for second inflight context\n");
1171+ return true;
1172+ }
1173+
1174+ /* Otherwise, ELSP[0] is by itself, but may be waiting in the queue */
1175+ if (!RB_EMPTY_ROOT(&engine->execlists.queue.rb_root)) {
1176+ ENGINE_TRACE(engine, "timeslice required for queue\n");
1177+ return true;
1178+ }
1179+
1180+ if (!RB_EMPTY_ROOT(&engine->execlists.virtual.rb_root)) {
1181+ ENGINE_TRACE(engine, "timeslice required for virtual\n");
1182+ return true;
1183+ }
1184+
1185+ return false;
1186+}
1187+
1188+static bool
1189+timeslice_expired(struct intel_engine_cs *engine, const struct i915_request *rq)
1190+{
1191+ const struct intel_engine_execlists *el = &engine->execlists;
1192+
1193+ if (i915_request_has_nopreempt(rq) && __i915_request_has_started(rq))
1194+ return false;
1195+
1196+ if (!needs_timeslice(engine, rq))
1197+ return false;
1198+
1199+ return timer_expired(&el->timer) || timeslice_yield(el, rq);
1200+}
1201+
1202+static unsigned long timeslice(const struct intel_engine_cs *engine)
1203+{
1204+ return READ_ONCE(engine->props.timeslice_duration_ms);
1205+}
1206+
1207+static void start_timeslice(struct intel_engine_cs *engine)
1208+{
1209+ struct intel_engine_execlists *el = &engine->execlists;
1210+ unsigned long duration;
1211+
1212+ /* Disable the timer if there is nothing to switch to */
1213+ duration = 0;
1214+ if (needs_timeslice(engine, *el->active)) {
1215+ /* Avoid continually prolonging an active timeslice */
1216+ if (timer_active(&el->timer)) {
1217+ /*
1218+ * If we just submitted a new ELSP after an old
1219+ * context, that context may have already consumed
1220+ * its timeslice, so recheck.
1221+ */
1222+ if (!timer_pending(&el->timer))
1223+ tasklet_hi_schedule(&el->tasklet);
1224+ return;
1225+ }
1226+
1227+ duration = timeslice(engine);
1228+ }
1229+
1230+ set_timer_ms(&el->timer, duration);
1231+}
1232+
1233+static void record_preemption(struct intel_engine_execlists *execlists)
1234+{
1235+ (void)I915_SELFTEST_ONLY(execlists->preempt_hang.count++);
1236+}
1237+
1238+static unsigned long active_preempt_timeout(struct intel_engine_cs *engine,
1239+ const struct i915_request *rq)
1240+{
1241+ if (!rq)
1242+ return 0;
1243+
1244+ /* Force a fast reset for terminated contexts (ignoring sysfs!) */
1245+ if (unlikely(intel_context_is_banned(rq->context)))
1246+ return 1;
1247+
1248+ return READ_ONCE(engine->props.preempt_timeout_ms);
1249+}
1250+
1251+static void set_preempt_timeout(struct intel_engine_cs *engine,
1252+ const struct i915_request *rq)
1253+{
1254+ if (!intel_engine_has_preempt_reset(engine))
1255+ return;
1256+
1257+ set_timer_ms(&engine->execlists.preempt,
1258+ active_preempt_timeout(engine, rq));
1259+}
1260+
1261+static void execlists_dequeue(struct intel_engine_cs *engine)
1262+{
1263+ struct intel_engine_execlists * const execlists = &engine->execlists;
1264+ struct i915_request **port = execlists->pending;
1265+ struct i915_request ** const last_port = port + execlists->port_mask;
1266+ struct i915_request *last = *execlists->active;
1267+ struct virtual_engine *ve;
1268+ struct rb_node *rb;
1269+ bool submit = false;
1270+
1271+ /*
1272+ * Hardware submission is through 2 ports. Conceptually each port
1273+ * has a (RING_START, RING_HEAD, RING_TAIL) tuple. RING_START is
1274+ * static for a context, and unique to each, so we only execute
1275+ * requests belonging to a single context from each ring. RING_HEAD
1276+ * is maintained by the CS in the context image, it marks the place
1277+ * where it got up to last time, and through RING_TAIL we tell the CS
1278+ * where we want to execute up to this time.
1279+ *
1280+ * In this list the requests are in order of execution. Consecutive
1281+ * requests from the same context are adjacent in the ringbuffer. We
1282+ * can combine these requests into a single RING_TAIL update:
1283+ *
1284+ * RING_HEAD...req1...req2
1285+ * ^- RING_TAIL
1286+ * since to execute req2 the CS must first execute req1.
1287+ *
1288+ * Our goal then is to point each port to the end of a consecutive
1289+ * sequence of requests as being the most optimal (fewest wake ups
1290+ * and context switches) submission.
1291+ */
1292+
1293+ spin_lock(&engine->active.lock);
1294+
1295+ /*
1296+ * If the queue is higher priority than the last
1297+ * request in the currently active context, submit afresh.
1298+ * We will resubmit again afterwards in case we need to split
1299+ * the active context to interject the preemption request,
1300+ * i.e. we will retrigger preemption following the ack in case
1301+ * of trouble.
1302+ *
1303+ * In theory we can skip over completed contexts that have not
1304+ * yet been processed by events (as those events are in flight):
1305+ *
1306+ * while ((last = *active) && i915_request_completed(last))
1307+ * active++;
1308+ *
1309+ * However, the GPU cannot handle this as it will ultimately
1310+ * find itself trying to jump back into a context it has just
1311+ * completed and barf.
1312+ */
1313+
1314+ if (last) {
1315+ if (__i915_request_is_complete(last)) {
1316+ goto check_secondary;
1317+ } else if (need_preempt(engine, last)) {
1318+ ENGINE_TRACE(engine,
1319+ "preempting last=%llx:%lld, prio=%d, hint=%d\n",
1320+ last->fence.context,
1321+ last->fence.seqno,
1322+ last->sched.attr.priority,
1323+ execlists->queue_priority_hint);
1324+ record_preemption(execlists);
1325+
1326+ /*
1327+ * Don't let the RING_HEAD advance past the breadcrumb
1328+ * as we unwind (and until we resubmit) so that we do
1329+ * not accidentally tell it to go backwards.
1330+ */
1331+ ring_set_paused(engine, 1);
1332+
1333+ /*
1334+ * Note that we have not stopped the GPU at this point,
1335+ * so we are unwinding the incomplete requests as they
1336+ * remain inflight and so by the time we do complete
1337+ * the preemption, some of the unwound requests may
1338+ * complete!
1339+ */
1340+ __unwind_incomplete_requests(engine);
1341+
1342+ last = NULL;
1343+ } else if (timeslice_expired(engine, last)) {
1344+ ENGINE_TRACE(engine,
1345+ "expired:%s last=%llx:%lld, prio=%d, hint=%d, yield?=%s\n",
1346+ yesno(timer_expired(&execlists->timer)),
1347+ last->fence.context, last->fence.seqno,
1348+ rq_prio(last),
1349+ execlists->queue_priority_hint,
1350+ yesno(timeslice_yield(execlists, last)));
1351+
1352+ /*
1353+ * Consume this timeslice; ensure we start a new one.
1354+ *
1355+ * The timeslice expired, and we will unwind the
1356+ * running contexts and recompute the next ELSP.
1357+ * If that submit will be the same pair of contexts
1358+ * (due to dependency ordering), we will skip the
1359+ * submission. If we don't cancel the timer now,
1360+ * we will see that the timer has expired and
1361+ * reschedule the tasklet; continually until the
1362+ * next context switch or other preeemption event.
1363+ *
1364+ * Since we have decided to reschedule based on
1365+ * consumption of this timeslice, if we submit the
1366+ * same context again, grant it a full timeslice.
1367+ */
1368+ cancel_timer(&execlists->timer);
1369+ ring_set_paused(engine, 1);
1370+ defer_active(engine);
1371+
1372+ /*
1373+ * Unlike for preemption, if we rewind and continue
1374+ * executing the same context as previously active,
1375+ * the order of execution will remain the same and
1376+ * the tail will only advance. We do not need to
1377+ * force a full context restore, as a lite-restore
1378+ * is sufficient to resample the monotonic TAIL.
1379+ *
1380+ * If we switch to any other context, similarly we
1381+ * will not rewind TAIL of current context, and
1382+ * normal save/restore will preserve state and allow
1383+ * us to later continue executing the same request.
1384+ */
1385+ last = NULL;
1386+ } else {
1387+ /*
1388+ * Otherwise if we already have a request pending
1389+ * for execution after the current one, we can
1390+ * just wait until the next CS event before
1391+ * queuing more. In either case we will force a
1392+ * lite-restore preemption event, but if we wait
1393+ * we hopefully coalesce several updates into a single
1394+ * submission.
1395+ */
1396+check_secondary:
1397+ if (!list_is_last(&last->sched.link,
1398+ &engine->active.requests)) {
1399+ /*
1400+ * Even if ELSP[1] is occupied and not worthy
1401+ * of timeslices, our queue might be.
1402+ */
1403+ spin_unlock(&engine->active.lock);
1404+ return;
1405+ }
1406+ }
1407+ }
1408+
1409+ /* XXX virtual is always taking precedence */
1410+ while ((ve = first_virtual_engine(engine))) {
1411+ struct i915_request *rq;
1412+
1413+ spin_lock(&ve->base.active.lock);
1414+
1415+ rq = ve->request;
1416+ if (unlikely(!virtual_matches(ve, rq, engine)))
1417+ goto unlock; /* lost the race to a sibling */
1418+
1419+ GEM_BUG_ON(rq->engine != &ve->base);
1420+ GEM_BUG_ON(rq->context != &ve->context);
1421+
1422+ if (unlikely(rq_prio(rq) < queue_prio(execlists))) {
1423+ spin_unlock(&ve->base.active.lock);
1424+ break;
1425+ }
1426+
1427+ if (last && !can_merge_rq(last, rq)) {
1428+ spin_unlock(&ve->base.active.lock);
1429+ spin_unlock(&engine->active.lock);
1430+ return; /* leave this for another sibling */
1431+ }
1432+
1433+ ENGINE_TRACE(engine,
1434+ "virtual rq=%llx:%lld%s, new engine? %s\n",
1435+ rq->fence.context,
1436+ rq->fence.seqno,
1437+ __i915_request_is_complete(rq) ? "!" :
1438+ __i915_request_has_started(rq) ? "*" :
1439+ "",
1440+ yesno(engine != ve->siblings[0]));
1441+
1442+ WRITE_ONCE(ve->request, NULL);
1443+ WRITE_ONCE(ve->base.execlists.queue_priority_hint, INT_MIN);
1444+
1445+ rb = &ve->nodes[engine->id].rb;
1446+ rb_erase_cached(rb, &execlists->virtual);
1447+ RB_CLEAR_NODE(rb);
1448+
1449+ GEM_BUG_ON(!(rq->execution_mask & engine->mask));
1450+ WRITE_ONCE(rq->engine, engine);
1451+
1452+ if (__i915_request_submit(rq)) {
1453+ /*
1454+ * Only after we confirm that we will submit
1455+ * this request (i.e. it has not already
1456+ * completed), do we want to update the context.
1457+ *
1458+ * This serves two purposes. It avoids
1459+ * unnecessary work if we are resubmitting an
1460+ * already completed request after timeslicing.
1461+ * But more importantly, it prevents us altering
1462+ * ve->siblings[] on an idle context, where
1463+ * we may be using ve->siblings[] in
1464+ * virtual_context_enter / virtual_context_exit.
1465+ */
1466+ virtual_xfer_context(ve, engine);
1467+ GEM_BUG_ON(ve->siblings[0] != engine);
1468+
1469+ submit = true;
1470+ last = rq;
1471+ }
1472+
1473+ i915_request_put(rq);
1474+unlock:
1475+ spin_unlock(&ve->base.active.lock);
1476+
1477+ /*
1478+ * Hmm, we have a bunch of virtual engine requests,
1479+ * but the first one was already completed (thanks
1480+ * preempt-to-busy!). Keep looking at the veng queue
1481+ * until we have no more relevant requests (i.e.
1482+ * the normal submit queue has higher priority).
1483+ */
1484+ if (submit)
1485+ break;
1486+ }
1487+
1488+ while ((rb = rb_first_cached(&execlists->queue))) {
1489+ struct i915_priolist *p = to_priolist(rb);
1490+ struct i915_request *rq, *rn;
1491+ int i;
1492+
1493+ priolist_for_each_request_consume(rq, rn, p, i) {
1494+ bool merge = true;
1495+
1496+ /*
1497+ * Can we combine this request with the current port?
1498+ * It has to be the same context/ringbuffer and not
1499+ * have any exceptions (e.g. GVT saying never to
1500+ * combine contexts).
1501+ *
1502+ * If we can combine the requests, we can execute both
1503+ * by updating the RING_TAIL to point to the end of the
1504+ * second request, and so we never need to tell the
1505+ * hardware about the first.
1506+ */
1507+ if (last && !can_merge_rq(last, rq)) {
1508+ /*
1509+ * If we are on the second port and cannot
1510+ * combine this request with the last, then we
1511+ * are done.
1512+ */
1513+ if (port == last_port)
1514+ goto done;
1515+
1516+ /*
1517+ * We must not populate both ELSP[] with the
1518+ * same LRCA, i.e. we must submit 2 different
1519+ * contexts if we submit 2 ELSP.
1520+ */
1521+ if (last->context == rq->context)
1522+ goto done;
1523+
1524+ if (i915_request_has_sentinel(last))
1525+ goto done;
1526+
1527+ /*
1528+ * We avoid submitting virtual requests into
1529+ * the secondary ports so that we can migrate
1530+ * the request immediately to another engine
1531+ * rather than wait for the primary request.
1532+ */
1533+ if (rq->execution_mask != engine->mask)
1534+ goto done;
1535+
1536+ /*
1537+ * If GVT overrides us we only ever submit
1538+ * port[0], leaving port[1] empty. Note that we
1539+ * also have to be careful that we don't queue
1540+ * the same context (even though a different
1541+ * request) to the second port.
1542+ */
1543+ if (ctx_single_port_submission(last->context) ||
1544+ ctx_single_port_submission(rq->context))
1545+ goto done;
1546+
1547+ merge = false;
1548+ }
1549+
1550+ if (__i915_request_submit(rq)) {
1551+ if (!merge) {
1552+ *port++ = i915_request_get(last);
1553+ last = NULL;
1554+ }
1555+
1556+ GEM_BUG_ON(last &&
1557+ !can_merge_ctx(last->context,
1558+ rq->context));
1559+ GEM_BUG_ON(last &&
1560+ i915_seqno_passed(last->fence.seqno,
1561+ rq->fence.seqno));
1562+
1563+ submit = true;
1564+ last = rq;
1565+ }
1566+ }
1567+
1568+ rb_erase_cached(&p->node, &execlists->queue);
1569+ i915_priolist_free(p);
1570+ }
1571+done:
1572+ *port++ = i915_request_get(last);
1573+
1574+ /*
1575+ * Here be a bit of magic! Or sleight-of-hand, whichever you prefer.
1576+ *
1577+ * We choose the priority hint such that if we add a request of greater
1578+ * priority than this, we kick the submission tasklet to decide on
1579+ * the right order of submitting the requests to hardware. We must
1580+ * also be prepared to reorder requests as they are in-flight on the
1581+ * HW. We derive the priority hint then as the first "hole" in
1582+ * the HW submission ports and if there are no available slots,
1583+ * the priority of the lowest executing request, i.e. last.
1584+ *
1585+ * When we do receive a higher priority request ready to run from the
1586+ * user, see queue_request(), the priority hint is bumped to that
1587+ * request triggering preemption on the next dequeue (or subsequent
1588+ * interrupt for secondary ports).
1589+ */
1590+ execlists->queue_priority_hint = queue_prio(execlists);
1591+ spin_unlock(&engine->active.lock);
1592+
1593+ /*
1594+ * We can skip poking the HW if we ended up with exactly the same set
1595+ * of requests as currently running, e.g. trying to timeslice a pair
1596+ * of ordered contexts.
1597+ */
1598+ if (submit &&
1599+ memcmp(execlists->active,
1600+ execlists->pending,
1601+ (port - execlists->pending) * sizeof(*port))) {
1602+ *port = NULL;
1603+ while (port-- != execlists->pending)
1604+ execlists_schedule_in(*port, port - execlists->pending);
1605+
1606+ WRITE_ONCE(execlists->yield, -1);
1607+ set_preempt_timeout(engine, *execlists->active);
1608+ execlists_submit_ports(engine);
1609+ } else {
1610+ ring_set_paused(engine, 0);
1611+ while (port-- != execlists->pending)
1612+ i915_request_put(*port);
1613+ *execlists->pending = NULL;
1614+ }
1615+}
1616+
1617+static void execlists_dequeue_irq(struct intel_engine_cs *engine)
1618+{
1619+ local_irq_disable(); /* Suspend interrupts across request submission */
1620+ execlists_dequeue(engine);
1621+ local_irq_enable(); /* flush irq_work (e.g. breadcrumb enabling) */
1622+}
1623+
1624+static inline void clear_ports(struct i915_request **ports, int count)
1625+{
1626+ memset_p((void **)ports, NULL, count);
1627+}
1628+
1629+static inline void
1630+copy_ports(struct i915_request **dst, struct i915_request **src, int count)
1631+{
1632+ /* A memcpy_p() would be very useful here! */
1633+ while (count--)
1634+ WRITE_ONCE(*dst++, *src++); /* avoid write tearing */
1635+}
1636+
1637+static struct i915_request **
1638+cancel_port_requests(struct intel_engine_execlists * const execlists,
1639+ struct i915_request **inactive)
1640+{
1641+ struct i915_request * const *port;
1642+
1643+ for (port = execlists->pending; *port; port++)
1644+ *inactive++ = *port;
1645+ clear_ports(execlists->pending, ARRAY_SIZE(execlists->pending));
1646+
1647+ /* Mark the end of active before we overwrite *active */
1648+ for (port = xchg(&execlists->active, execlists->pending); *port; port++)
1649+ *inactive++ = *port;
1650+ clear_ports(execlists->inflight, ARRAY_SIZE(execlists->inflight));
1651+
1652+ smp_wmb(); /* complete the seqlock for execlists_active() */
1653+ WRITE_ONCE(execlists->active, execlists->inflight);
1654+
1655+ /* Having cancelled all outstanding process_csb(), stop their timers */
1656+ GEM_BUG_ON(execlists->pending[0]);
1657+ cancel_timer(&execlists->timer);
1658+ cancel_timer(&execlists->preempt);
1659+
1660+ return inactive;
1661+}
1662+
1663+static inline void
1664+invalidate_csb_entries(const u64 *first, const u64 *last)
1665+{
1666+ clflush((void *)first);
1667+ clflush((void *)last);
1668+}
1669+
1670+/*
1671+ * Starting with Gen12, the status has a new format:
1672+ *
1673+ * bit 0: switched to new queue
1674+ * bit 1: reserved
1675+ * bit 2: semaphore wait mode (poll or signal), only valid when
1676+ * switch detail is set to "wait on semaphore"
1677+ * bits 3-5: engine class
1678+ * bits 6-11: engine instance
1679+ * bits 12-14: reserved
1680+ * bits 15-25: sw context id of the lrc the GT switched to
1681+ * bits 26-31: sw counter of the lrc the GT switched to
1682+ * bits 32-35: context switch detail
1683+ * - 0: ctx complete
1684+ * - 1: wait on sync flip
1685+ * - 2: wait on vblank
1686+ * - 3: wait on scanline
1687+ * - 4: wait on semaphore
1688+ * - 5: context preempted (not on SEMAPHORE_WAIT or
1689+ * WAIT_FOR_EVENT)
1690+ * bit 36: reserved
1691+ * bits 37-43: wait detail (for switch detail 1 to 4)
1692+ * bits 44-46: reserved
1693+ * bits 47-57: sw context id of the lrc the GT switched away from
1694+ * bits 58-63: sw counter of the lrc the GT switched away from
1695+ */
1696+static inline bool gen12_csb_parse(const u64 csb)
1697+{
1698+ bool ctx_away_valid = GEN12_CSB_CTX_VALID(upper_32_bits(csb));
1699+ bool new_queue =
1700+ lower_32_bits(csb) & GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE;
1701+
1702+ /*
1703+ * The context switch detail is not guaranteed to be 5 when a preemption
1704+ * occurs, so we can't just check for that. The check below works for
1705+ * all the cases we care about, including preemptions of WAIT
1706+ * instructions and lite-restore. Preempt-to-idle via the CTRL register
1707+ * would require some extra handling, but we don't support that.
1708+ */
1709+ if (!ctx_away_valid || new_queue) {
1710+ GEM_BUG_ON(!GEN12_CSB_CTX_VALID(lower_32_bits(csb)));
1711+ return true;
1712+ }
1713+
1714+ /*
1715+ * switch detail = 5 is covered by the case above and we do not expect a
1716+ * context switch on an unsuccessful wait instruction since we always
1717+ * use polling mode.
1718+ */
1719+ GEM_BUG_ON(GEN12_CTX_SWITCH_DETAIL(upper_32_bits(csb)));
1720+ return false;
1721+}
1722+
1723+static inline bool gen8_csb_parse(const u64 csb)
1724+{
1725+ return csb & (GEN8_CTX_STATUS_IDLE_ACTIVE | GEN8_CTX_STATUS_PREEMPTED);
1726+}
1727+
1728+static noinline u64
1729+wa_csb_read(const struct intel_engine_cs *engine, u64 * const csb)
1730+{
1731+ u64 entry;
1732+
1733+ /*
1734+ * Reading from the HWSP has one particular advantage: we can detect
1735+ * a stale entry. Since the write into HWSP is broken, we have no reason
1736+ * to trust the HW at all, the mmio entry may equally be unordered, so
1737+ * we prefer the path that is self-checking and as a last resort,
1738+ * return the mmio value.
1739+ *
1740+ * tgl,dg1:HSDES#22011327657
1741+ */
1742+ preempt_disable();
1743+ if (wait_for_atomic_us((entry = READ_ONCE(*csb)) != -1, 10)) {
1744+ int idx = csb - engine->execlists.csb_status;
1745+ int status;
1746+
1747+ status = GEN8_EXECLISTS_STATUS_BUF;
1748+ if (idx >= 6) {
1749+ status = GEN11_EXECLISTS_STATUS_BUF2;
1750+ idx -= 6;
1751+ }
1752+ status += sizeof(u64) * idx;
1753+
1754+ entry = intel_uncore_read64(engine->uncore,
1755+ _MMIO(engine->mmio_base + status));
1756+ }
1757+ preempt_enable();
1758+
1759+ return entry;
1760+}
1761+
1762+static inline u64
1763+csb_read(const struct intel_engine_cs *engine, u64 * const csb)
1764+{
1765+ u64 entry = READ_ONCE(*csb);
1766+
1767+ /*
1768+ * Unfortunately, the GPU does not always serialise its write
1769+ * of the CSB entries before its write of the CSB pointer, at least
1770+ * from the perspective of the CPU, using what is known as a Global
1771+ * Observation Point. We may read a new CSB tail pointer, but then
1772+ * read the stale CSB entries, causing us to misinterpret the
1773+ * context-switch events, and eventually declare the GPU hung.
1774+ *
1775+ * icl:HSDES#1806554093
1776+ * tgl:HSDES#22011248461
1777+ */
1778+ if (unlikely(entry == -1))
1779+ entry = wa_csb_read(engine, csb);
1780+
1781+ /* Consume this entry so that we can spot its future reuse. */
1782+ WRITE_ONCE(*csb, -1);
1783+
1784+ /* ELSP is an implicit wmb() before the GPU wraps and overwrites csb */
1785+ return entry;
1786+}
1787+
1788+static void new_timeslice(struct intel_engine_execlists *el)
1789+{
1790+ /* By cancelling, we will start afresh in start_timeslice() */
1791+ cancel_timer(&el->timer);
1792+}
1793+
1794+static struct i915_request **
1795+process_csb(struct intel_engine_cs *engine, struct i915_request **inactive)
1796+{
1797+ struct intel_engine_execlists * const execlists = &engine->execlists;
1798+ u64 * const buf = execlists->csb_status;
1799+ const u8 num_entries = execlists->csb_size;
1800+ struct i915_request **prev;
1801+ u8 head, tail;
1802+
1803+ /*
1804+ * As we modify our execlists state tracking we require exclusive
1805+ * access. Either we are inside the tasklet, or the tasklet is disabled
1806+ * and we assume that is only inside the reset paths and so serialised.
1807+ */
1808+ GEM_BUG_ON(!tasklet_is_locked(&execlists->tasklet) &&
1809+ !reset_in_progress(execlists));
1810+ GEM_BUG_ON(!intel_engine_in_execlists_submission_mode(engine));
1811+
1812+ /*
1813+ * Note that csb_write, csb_status may be either in HWSP or mmio.
1814+ * When reading from the csb_write mmio register, we have to be
1815+ * careful to only use the GEN8_CSB_WRITE_PTR portion, which is
1816+ * the low 4bits. As it happens we know the next 4bits are always
1817+ * zero and so we can simply masked off the low u8 of the register
1818+ * and treat it identically to reading from the HWSP (without having
1819+ * to use explicit shifting and masking, and probably bifurcating
1820+ * the code to handle the legacy mmio read).
1821+ */
1822+ head = execlists->csb_head;
1823+ tail = READ_ONCE(*execlists->csb_write);
1824+ if (unlikely(head == tail))
1825+ return inactive;
1826+
1827+ /*
1828+ * We will consume all events from HW, or at least pretend to.
1829+ *
1830+ * The sequence of events from the HW is deterministic, and derived
1831+ * from our writes to the ELSP, with a smidgen of variability for
1832+ * the arrival of the asynchronous requests wrt to the inflight
1833+ * execution. If the HW sends an event that does not correspond with
1834+ * the one we are expecting, we have to abandon all hope as we lose
1835+ * all tracking of what the engine is actually executing. We will
1836+ * only detect we are out of sequence with the HW when we get an
1837+ * 'impossible' event because we have already drained our own
1838+ * preemption/promotion queue. If this occurs, we know that we likely
1839+ * lost track of execution earlier and must unwind and restart, the
1840+ * simplest way is by stop processing the event queue and force the
1841+ * engine to reset.
1842+ */
1843+ execlists->csb_head = tail;
1844+ ENGINE_TRACE(engine, "cs-irq head=%d, tail=%d\n", head, tail);
1845+
1846+ /*
1847+ * Hopefully paired with a wmb() in HW!
1848+ *
1849+ * We must complete the read of the write pointer before any reads
1850+ * from the CSB, so that we do not see stale values. Without an rmb
1851+ * (lfence) the HW may speculatively perform the CSB[] reads *before*
1852+ * we perform the READ_ONCE(*csb_write).
1853+ */
1854+ rmb();
1855+
1856+ /* Remember who was last running under the timer */
1857+ prev = inactive;
1858+ *prev = NULL;
1859+
1860+ do {
1861+ bool promote;
1862+ u64 csb;
1863+
1864+ if (++head == num_entries)
1865+ head = 0;
1866+
1867+ /*
1868+ * We are flying near dragons again.
1869+ *
1870+ * We hold a reference to the request in execlist_port[]
1871+ * but no more than that. We are operating in softirq
1872+ * context and so cannot hold any mutex or sleep. That
1873+ * prevents us stopping the requests we are processing
1874+ * in port[] from being retired simultaneously (the
1875+ * breadcrumb will be complete before we see the
1876+ * context-switch). As we only hold the reference to the
1877+ * request, any pointer chasing underneath the request
1878+ * is subject to a potential use-after-free. Thus we
1879+ * store all of the bookkeeping within port[] as
1880+ * required, and avoid using unguarded pointers beneath
1881+ * request itself. The same applies to the atomic
1882+ * status notifier.
1883+ */
1884+
1885+ csb = csb_read(engine, buf + head);
1886+ ENGINE_TRACE(engine, "csb[%d]: status=0x%08x:0x%08x\n",
1887+ head, upper_32_bits(csb), lower_32_bits(csb));
1888+
1889+ if (INTEL_GEN(engine->i915) >= 12)
1890+ promote = gen12_csb_parse(csb);
1891+ else
1892+ promote = gen8_csb_parse(csb);
1893+ if (promote) {
1894+ struct i915_request * const *old = execlists->active;
1895+
1896+ if (GEM_WARN_ON(!*execlists->pending)) {
1897+ execlists->error_interrupt |= ERROR_CSB;
1898+ break;
1899+ }
1900+
1901+ ring_set_paused(engine, 0);
1902+
1903+ /* Point active to the new ELSP; prevent overwriting */
1904+ WRITE_ONCE(execlists->active, execlists->pending);
1905+ smp_wmb(); /* notify execlists_active() */
1906+
1907+ /* cancel old inflight, prepare for switch */
1908+ trace_ports(execlists, "preempted", old);
1909+ while (*old)
1910+ *inactive++ = *old++;
1911+
1912+ /* switch pending to inflight */
1913+ GEM_BUG_ON(!assert_pending_valid(execlists, "promote"));
1914+ copy_ports(execlists->inflight,
1915+ execlists->pending,
1916+ execlists_num_ports(execlists));
1917+ smp_wmb(); /* complete the seqlock */
1918+ WRITE_ONCE(execlists->active, execlists->inflight);
1919+
1920+ /* XXX Magic delay for tgl */
1921+ ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
1922+
1923+ WRITE_ONCE(execlists->pending[0], NULL);
1924+ } else {
1925+ if (GEM_WARN_ON(!*execlists->active)) {
1926+ execlists->error_interrupt |= ERROR_CSB;
1927+ break;
1928+ }
1929+
1930+ /* port0 completed, advanced to port1 */
1931+ trace_ports(execlists, "completed", execlists->active);
1932+
1933+ /*
1934+ * We rely on the hardware being strongly
1935+ * ordered, that the breadcrumb write is
1936+ * coherent (visible from the CPU) before the
1937+ * user interrupt is processed. One might assume
1938+ * that the breadcrumb write being before the
1939+ * user interrupt and the CS event for the context
1940+ * switch would therefore be before the CS event
1941+ * itself...
1942+ */
1943+ if (GEM_SHOW_DEBUG() &&
1944+ !__i915_request_is_complete(*execlists->active)) {
1945+ struct i915_request *rq = *execlists->active;
1946+ const u32 *regs __maybe_unused =
1947+ rq->context->lrc_reg_state;
1948+
1949+ ENGINE_TRACE(engine,
1950+ "context completed before request!\n");
1951+ ENGINE_TRACE(engine,
1952+ "ring:{start:0x%08x, head:%04x, tail:%04x, ctl:%08x, mode:%08x}\n",
1953+ ENGINE_READ(engine, RING_START),
1954+ ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR,
1955+ ENGINE_READ(engine, RING_TAIL) & TAIL_ADDR,
1956+ ENGINE_READ(engine, RING_CTL),
1957+ ENGINE_READ(engine, RING_MI_MODE));
1958+ ENGINE_TRACE(engine,
1959+ "rq:{start:%08x, head:%04x, tail:%04x, seqno:%llx:%d, hwsp:%d}, ",
1960+ i915_ggtt_offset(rq->ring->vma),
1961+ rq->head, rq->tail,
1962+ rq->fence.context,
1963+ lower_32_bits(rq->fence.seqno),
1964+ hwsp_seqno(rq));
1965+ ENGINE_TRACE(engine,
1966+ "ctx:{start:%08x, head:%04x, tail:%04x}, ",
1967+ regs[CTX_RING_START],
1968+ regs[CTX_RING_HEAD],
1969+ regs[CTX_RING_TAIL]);
1970+ }
1971+
1972+ *inactive++ = *execlists->active++;
1973+
1974+ GEM_BUG_ON(execlists->active - execlists->inflight >
1975+ execlists_num_ports(execlists));
1976+ }
1977+ } while (head != tail);
1978+
1979+ /*
1980+ * Gen11 has proven to fail wrt global observation point between
1981+ * entry and tail update, failing on the ordering and thus
1982+ * we see an old entry in the context status buffer.
1983+ *
1984+ * Forcibly evict out entries for the next gpu csb update,
1985+ * to increase the odds that we get a fresh entries with non
1986+ * working hardware. The cost for doing so comes out mostly with
1987+ * the wash as hardware, working or not, will need to do the
1988+ * invalidation before.
1989+ */
1990+ invalidate_csb_entries(&buf[0], &buf[num_entries - 1]);
1991+
1992+ /*
1993+ * We assume that any event reflects a change in context flow
1994+ * and merits a fresh timeslice. We reinstall the timer after
1995+ * inspecting the queue to see if we need to resumbit.
1996+ */
1997+ if (*prev != *execlists->active) /* elide lite-restores */
1998+ new_timeslice(execlists);
1999+
2000+ return inactive;
2001+}
2002+
2003+static void post_process_csb(struct i915_request **port,
2004+ struct i915_request **last)
2005+{
2006+ while (port != last)
2007+ execlists_schedule_out(*port++);
2008+}
2009+
2010+static void __execlists_hold(struct i915_request *rq)
2011+{
2012+ LIST_HEAD(list);
2013+
2014+ do {
2015+ struct i915_dependency *p;
2016+
2017+ if (i915_request_is_active(rq))
2018+ __i915_request_unsubmit(rq);
2019+
2020+ clear_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2021+ list_move_tail(&rq->sched.link, &rq->engine->active.hold);
2022+ i915_request_set_hold(rq);
2023+ RQ_TRACE(rq, "on hold\n");
2024+
2025+ for_each_waiter(p, rq) {
2026+ struct i915_request *w =
2027+ container_of(p->waiter, typeof(*w), sched);
2028+
2029+ /* Leave semaphores spinning on the other engines */
2030+ if (w->engine != rq->engine)
2031+ continue;
2032+
2033+ if (!i915_request_is_ready(w))
2034+ continue;
2035+
2036+ if (__i915_request_is_complete(w))
2037+ continue;
2038+
2039+ if (i915_request_on_hold(w))
2040+ continue;
2041+
2042+ list_move_tail(&w->sched.link, &list);
2043+ }
2044+
2045+ rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2046+ } while (rq);
2047+}
2048+
2049+static bool execlists_hold(struct intel_engine_cs *engine,
2050+ struct i915_request *rq)
2051+{
2052+ if (i915_request_on_hold(rq))
2053+ return false;
2054+
2055+ spin_lock_irq(&engine->active.lock);
2056+
2057+ if (__i915_request_is_complete(rq)) { /* too late! */
2058+ rq = NULL;
2059+ goto unlock;
2060+ }
2061+
2062+ /*
2063+ * Transfer this request onto the hold queue to prevent it
2064+ * being resumbitted to HW (and potentially completed) before we have
2065+ * released it. Since we may have already submitted following
2066+ * requests, we need to remove those as well.
2067+ */
2068+ GEM_BUG_ON(i915_request_on_hold(rq));
2069+ GEM_BUG_ON(rq->engine != engine);
2070+ __execlists_hold(rq);
2071+ GEM_BUG_ON(list_empty(&engine->active.hold));
2072+
2073+unlock:
2074+ spin_unlock_irq(&engine->active.lock);
2075+ return rq;
2076+}
2077+
2078+static bool hold_request(const struct i915_request *rq)
2079+{
2080+ struct i915_dependency *p;
2081+ bool result = false;
2082+
2083+ /*
2084+ * If one of our ancestors is on hold, we must also be on hold,
2085+ * otherwise we will bypass it and execute before it.
2086+ */
2087+ rcu_read_lock();
2088+ for_each_signaler(p, rq) {
2089+ const struct i915_request *s =
2090+ container_of(p->signaler, typeof(*s), sched);
2091+
2092+ if (s->engine != rq->engine)
2093+ continue;
2094+
2095+ result = i915_request_on_hold(s);
2096+ if (result)
2097+ break;
2098+ }
2099+ rcu_read_unlock();
2100+
2101+ return result;
2102+}
2103+
2104+static void __execlists_unhold(struct i915_request *rq)
2105+{
2106+ LIST_HEAD(list);
2107+
2108+ do {
2109+ struct i915_dependency *p;
2110+
2111+ RQ_TRACE(rq, "hold release\n");
2112+
2113+ GEM_BUG_ON(!i915_request_on_hold(rq));
2114+ GEM_BUG_ON(!i915_sw_fence_signaled(&rq->submit));
2115+
2116+ i915_request_clear_hold(rq);
2117+ list_move_tail(&rq->sched.link,
2118+ i915_sched_lookup_priolist(rq->engine,
2119+ rq_prio(rq)));
2120+ set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2121+
2122+ /* Also release any children on this engine that are ready */
2123+ for_each_waiter(p, rq) {
2124+ struct i915_request *w =
2125+ container_of(p->waiter, typeof(*w), sched);
2126+
2127+ /* Propagate any change in error status */
2128+ if (rq->fence.error)
2129+ i915_request_set_error_once(w, rq->fence.error);
2130+
2131+ if (w->engine != rq->engine)
2132+ continue;
2133+
2134+ if (!i915_request_on_hold(w))
2135+ continue;
2136+
2137+ /* Check that no other parents are also on hold */
2138+ if (hold_request(w))
2139+ continue;
2140+
2141+ list_move_tail(&w->sched.link, &list);
2142+ }
2143+
2144+ rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2145+ } while (rq);
2146+}
2147+
2148+static void execlists_unhold(struct intel_engine_cs *engine,
2149+ struct i915_request *rq)
2150+{
2151+ spin_lock_irq(&engine->active.lock);
2152+
2153+ /*
2154+ * Move this request back to the priority queue, and all of its
2155+ * children and grandchildren that were suspended along with it.
2156+ */
2157+ __execlists_unhold(rq);
2158+
2159+ if (rq_prio(rq) > engine->execlists.queue_priority_hint) {
2160+ engine->execlists.queue_priority_hint = rq_prio(rq);
2161+ tasklet_hi_schedule(&engine->execlists.tasklet);
2162+ }
2163+
2164+ spin_unlock_irq(&engine->active.lock);
2165+}
2166+
2167+struct execlists_capture {
2168+ struct work_struct work;
2169+ struct i915_request *rq;
2170+ struct i915_gpu_coredump *error;
2171+};
2172+
2173+static void execlists_capture_work(struct work_struct *work)
2174+{
2175+ struct execlists_capture *cap = container_of(work, typeof(*cap), work);
2176+ const gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
2177+ struct intel_engine_cs *engine = cap->rq->engine;
2178+ struct intel_gt_coredump *gt = cap->error->gt;
2179+ struct intel_engine_capture_vma *vma;
2180+
2181+ /* Compress all the objects attached to the request, slow! */
2182+ vma = intel_engine_coredump_add_request(gt->engine, cap->rq, gfp);
2183+ if (vma) {
2184+ struct i915_vma_compress *compress =
2185+ i915_vma_capture_prepare(gt);
2186+
2187+ intel_engine_coredump_add_vma(gt->engine, vma, compress);
2188+ i915_vma_capture_finish(gt, compress);
2189+ }
2190+
2191+ gt->simulated = gt->engine->simulated;
2192+ cap->error->simulated = gt->simulated;
2193+
2194+ /* Publish the error state, and announce it to the world */
2195+ i915_error_state_store(cap->error);
2196+ i915_gpu_coredump_put(cap->error);
2197+
2198+ /* Return this request and all that depend upon it for signaling */
2199+ execlists_unhold(engine, cap->rq);
2200+ i915_request_put(cap->rq);
2201+
2202+ kfree(cap);
2203+}
2204+
2205+static struct execlists_capture *capture_regs(struct intel_engine_cs *engine)
2206+{
2207+ const gfp_t gfp = GFP_ATOMIC | __GFP_NOWARN;
2208+ struct execlists_capture *cap;
2209+
2210+ cap = kmalloc(sizeof(*cap), gfp);
2211+ if (!cap)
2212+ return NULL;
2213+
2214+ cap->error = i915_gpu_coredump_alloc(engine->i915, gfp);
2215+ if (!cap->error)
2216+ goto err_cap;
2217+
2218+ cap->error->gt = intel_gt_coredump_alloc(engine->gt, gfp);
2219+ if (!cap->error->gt)
2220+ goto err_gpu;
2221+
2222+ cap->error->gt->engine = intel_engine_coredump_alloc(engine, gfp);
2223+ if (!cap->error->gt->engine)
2224+ goto err_gt;
2225+
2226+ cap->error->gt->engine->hung = true;
2227+
2228+ return cap;
2229+
2230+err_gt:
2231+ kfree(cap->error->gt);
2232+err_gpu:
2233+ kfree(cap->error);
2234+err_cap:
2235+ kfree(cap);
2236+ return NULL;
2237+}
2238+
2239+static struct i915_request *
2240+active_context(struct intel_engine_cs *engine, u32 ccid)
2241+{
2242+ const struct intel_engine_execlists * const el = &engine->execlists;
2243+ struct i915_request * const *port, *rq;
2244+
2245+ /*
2246+ * Use the most recent result from process_csb(), but just in case
2247+ * we trigger an error (via interrupt) before the first CS event has
2248+ * been written, peek at the next submission.
2249+ */
2250+
2251+ for (port = el->active; (rq = *port); port++) {
2252+ if (rq->context->lrc.ccid == ccid) {
2253+ ENGINE_TRACE(engine,
2254+ "ccid:%x found at active:%zd\n",
2255+ ccid, port - el->active);
2256+ return rq;
2257+ }
2258+ }
2259+
2260+ for (port = el->pending; (rq = *port); port++) {
2261+ if (rq->context->lrc.ccid == ccid) {
2262+ ENGINE_TRACE(engine,
2263+ "ccid:%x found at pending:%zd\n",
2264+ ccid, port - el->pending);
2265+ return rq;
2266+ }
2267+ }
2268+
2269+ ENGINE_TRACE(engine, "ccid:%x not found\n", ccid);
2270+ return NULL;
2271+}
2272+
2273+static u32 active_ccid(struct intel_engine_cs *engine)
2274+{
2275+ return ENGINE_READ_FW(engine, RING_EXECLIST_STATUS_HI);
2276+}
2277+
2278+static void execlists_capture(struct intel_engine_cs *engine)
2279+{
2280+ struct execlists_capture *cap;
2281+
2282+ if (!IS_ENABLED(CONFIG_DRM_I915_CAPTURE_ERROR))
2283+ return;
2284+
2285+ /*
2286+ * We need to _quickly_ capture the engine state before we reset.
2287+ * We are inside an atomic section (softirq) here and we are delaying
2288+ * the forced preemption event.
2289+ */
2290+ cap = capture_regs(engine);
2291+ if (!cap)
2292+ return;
2293+
2294+ spin_lock_irq(&engine->active.lock);
2295+ cap->rq = active_context(engine, active_ccid(engine));
2296+ if (cap->rq) {
2297+ cap->rq = active_request(cap->rq->context->timeline, cap->rq);
2298+ cap->rq = i915_request_get_rcu(cap->rq);
2299+ }
2300+ spin_unlock_irq(&engine->active.lock);
2301+ if (!cap->rq)
2302+ goto err_free;
2303+
2304+ /*
2305+ * Remove the request from the execlists queue, and take ownership
2306+ * of the request. We pass it to our worker who will _slowly_ compress
2307+ * all the pages the _user_ requested for debugging their batch, after
2308+ * which we return it to the queue for signaling.
2309+ *
2310+ * By removing them from the execlists queue, we also remove the
2311+ * requests from being processed by __unwind_incomplete_requests()
2312+ * during the intel_engine_reset(), and so they will *not* be replayed
2313+ * afterwards.
2314+ *
2315+ * Note that because we have not yet reset the engine at this point,
2316+ * it is possible for the request that we have identified as being
2317+ * guilty, did in fact complete and we will then hit an arbitration
2318+ * point allowing the outstanding preemption to succeed. The likelihood
2319+ * of that is very low (as capturing of the engine registers should be
2320+ * fast enough to run inside an irq-off atomic section!), so we will
2321+ * simply hold that request accountable for being non-preemptible
2322+ * long enough to force the reset.
2323+ */
2324+ if (!execlists_hold(engine, cap->rq))
2325+ goto err_rq;
2326+
2327+ INIT_WORK(&cap->work, execlists_capture_work);
2328+ schedule_work(&cap->work);
2329+ return;
2330+
2331+err_rq:
2332+ i915_request_put(cap->rq);
2333+err_free:
2334+ i915_gpu_coredump_put(cap->error);
2335+ kfree(cap);
2336+}
2337+
2338+static void execlists_reset(struct intel_engine_cs *engine, const char *msg)
2339+{
2340+ const unsigned int bit = I915_RESET_ENGINE + engine->id;
2341+ unsigned long *lock = &engine->gt->reset.flags;
2342+
2343+ if (!intel_has_reset_engine(engine->gt))
2344+ return;
2345+
2346+ if (test_and_set_bit(bit, lock))
2347+ return;
2348+
2349+ ENGINE_TRACE(engine, "reset for %s\n", msg);
2350+
2351+ /* Mark this tasklet as disabled to avoid waiting for it to complete */
2352+ tasklet_disable_nosync(&engine->execlists.tasklet);
2353+
2354+ ring_set_paused(engine, 1); /* Freeze the current request in place */
2355+ execlists_capture(engine);
2356+ intel_engine_reset(engine, msg);
2357+
2358+ tasklet_enable(&engine->execlists.tasklet);
2359+ clear_and_wake_up_bit(bit, lock);
2360+}
2361+
2362+static bool preempt_timeout(const struct intel_engine_cs *const engine)
2363+{
2364+ const struct timer_list *t = &engine->execlists.preempt;
2365+
2366+ if (!CONFIG_DRM_I915_PREEMPT_TIMEOUT)
2367+ return false;
2368+
2369+ if (!timer_expired(t))
2370+ return false;
2371+
2372+ return engine->execlists.pending[0];
2373+}
2374+
2375+/*
2376+ * Check the unread Context Status Buffers and manage the submission of new
2377+ * contexts to the ELSP accordingly.
2378+ */
2379+static void execlists_submission_tasklet(unsigned long data)
2380+{
2381+ struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
2382+ struct i915_request *post[2 * EXECLIST_MAX_PORTS];
2383+ struct i915_request **inactive;
2384+
2385+ rcu_read_lock();
2386+ inactive = process_csb(engine, post);
2387+ GEM_BUG_ON(inactive - post > ARRAY_SIZE(post));
2388+
2389+ if (unlikely(preempt_timeout(engine))) {
2390+ cancel_timer(&engine->execlists.preempt);
2391+ engine->execlists.error_interrupt |= ERROR_PREEMPT;
2392+ }
2393+
2394+ if (unlikely(READ_ONCE(engine->execlists.error_interrupt))) {
2395+ const char *msg;
2396+
2397+ /* Generate the error message in priority wrt to the user! */
2398+ if (engine->execlists.error_interrupt & GENMASK(15, 0))
2399+ msg = "CS error"; /* thrown by a user payload */
2400+ else if (engine->execlists.error_interrupt & ERROR_CSB)
2401+ msg = "invalid CSB event";
2402+ else if (engine->execlists.error_interrupt & ERROR_PREEMPT)
2403+ msg = "preemption time out";
2404+ else
2405+ msg = "internal error";
2406+
2407+ engine->execlists.error_interrupt = 0;
2408+ execlists_reset(engine, msg);
2409+ }
2410+
2411+ if (!engine->execlists.pending[0]) {
2412+ execlists_dequeue_irq(engine);
2413+ start_timeslice(engine);
2414+ }
2415+
2416+ post_process_csb(post, inactive);
2417+ rcu_read_unlock();
2418+}
2419+
2420+static void __execlists_kick(struct intel_engine_execlists *execlists)
2421+{
2422+ /* Kick the tasklet for some interrupt coalescing and reset handling */
2423+ tasklet_hi_schedule(&execlists->tasklet);
2424+}
2425+
2426+#define execlists_kick(t, member) \
2427+ __execlists_kick(container_of(t, struct intel_engine_execlists, member))
2428+
2429+static void execlists_timeslice(struct timer_list *timer)
2430+{
2431+ execlists_kick(timer, timer);
2432+}
2433+
2434+static void execlists_preempt(struct timer_list *timer)
2435+{
2436+ execlists_kick(timer, preempt);
2437+}
2438+
2439+static void queue_request(struct intel_engine_cs *engine,
2440+ struct i915_request *rq)
2441+{
2442+ GEM_BUG_ON(!list_empty(&rq->sched.link));
2443+ list_add_tail(&rq->sched.link,
2444+ i915_sched_lookup_priolist(engine, rq_prio(rq)));
2445+ set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2446+}
2447+
2448+static bool submit_queue(struct intel_engine_cs *engine,
2449+ const struct i915_request *rq)
2450+{
2451+ struct intel_engine_execlists *execlists = &engine->execlists;
2452+
2453+ if (rq_prio(rq) <= execlists->queue_priority_hint)
2454+ return false;
2455+
2456+ execlists->queue_priority_hint = rq_prio(rq);
2457+ return true;
2458+}
2459+
2460+static bool ancestor_on_hold(const struct intel_engine_cs *engine,
2461+ const struct i915_request *rq)
2462+{
2463+ GEM_BUG_ON(i915_request_on_hold(rq));
2464+ return !list_empty(&engine->active.hold) && hold_request(rq);
2465+}
2466+
2467+static void execlists_submit_request(struct i915_request *request)
2468+{
2469+ struct intel_engine_cs *engine = request->engine;
2470+ unsigned long flags;
2471+
2472+ /* Will be called from irq-context when using foreign fences. */
2473+ spin_lock_irqsave(&engine->active.lock, flags);
2474+
2475+ if (unlikely(ancestor_on_hold(engine, request))) {
2476+ RQ_TRACE(request, "ancestor on hold\n");
2477+ list_add_tail(&request->sched.link, &engine->active.hold);
2478+ i915_request_set_hold(request);
2479+ } else {
2480+ queue_request(engine, request);
2481+
2482+ GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
2483+ GEM_BUG_ON(list_empty(&request->sched.link));
2484+
2485+ if (submit_queue(engine, request))
2486+ __execlists_kick(&engine->execlists);
2487+ }
2488+
2489+ spin_unlock_irqrestore(&engine->active.lock, flags);
2490+}
2491+
2492+static int execlists_context_pre_pin(struct intel_context *ce,
2493+ struct i915_gem_ww_ctx *ww,
2494+ void **vaddr)
2495+{
2496+ return lrc_pre_pin(ce, ce->engine, ww, vaddr);
2497+}
2498+
2499+static int execlists_context_pin(struct intel_context *ce, void *vaddr)
2500+{
2501+ return lrc_pin(ce, ce->engine, vaddr);
2502+}
2503+
2504+static int execlists_context_alloc(struct intel_context *ce)
2505+{
2506+ return lrc_alloc(ce, ce->engine);
2507+}
2508+
2509+static const struct intel_context_ops execlists_context_ops = {
2510+ .flags = COPS_HAS_INFLIGHT,
2511+
2512+ .alloc = execlists_context_alloc,
2513+
2514+ .pre_pin = execlists_context_pre_pin,
2515+ .pin = execlists_context_pin,
2516+ .unpin = lrc_unpin,
2517+ .post_unpin = lrc_post_unpin,
2518+
2519+ .enter = intel_context_enter_engine,
2520+ .exit = intel_context_exit_engine,
2521+
2522+ .reset = lrc_reset,
2523+ .destroy = lrc_destroy,
2524+};
2525+
2526+static int emit_pdps(struct i915_request *rq)
2527+{
2528+ const struct intel_engine_cs * const engine = rq->engine;
2529+ struct i915_ppgtt * const ppgtt = i915_vm_to_ppgtt(rq->context->vm);
2530+ int err, i;
2531+ u32 *cs;
2532+
2533+ GEM_BUG_ON(intel_vgpu_active(rq->engine->i915));
2534+
2535+ /*
2536+ * Beware ye of the dragons, this sequence is magic!
2537+ *
2538+ * Small changes to this sequence can cause anything from
2539+ * GPU hangs to forcewake errors and machine lockups!
2540+ */
2541+
2542+ cs = intel_ring_begin(rq, 2);
2543+ if (IS_ERR(cs))
2544+ return PTR_ERR(cs);
2545+
2546+ *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
2547+ *cs++ = MI_NOOP;
2548+ intel_ring_advance(rq, cs);
2549+
2550+ /* Flush any residual operations from the context load */
2551+ err = engine->emit_flush(rq, EMIT_FLUSH);
2552+ if (err)
2553+ return err;
2554+
2555+ /* Magic required to prevent forcewake errors! */
2556+ err = engine->emit_flush(rq, EMIT_INVALIDATE);
2557+ if (err)
2558+ return err;
2559+
2560+ cs = intel_ring_begin(rq, 4 * GEN8_3LVL_PDPES + 2);
2561+ if (IS_ERR(cs))
2562+ return PTR_ERR(cs);
2563+
2564+ /* Ensure the LRI have landed before we invalidate & continue */
2565+ *cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES) | MI_LRI_FORCE_POSTED;
2566+ for (i = GEN8_3LVL_PDPES; i--; ) {
2567+ const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
2568+ u32 base = engine->mmio_base;
2569+
2570+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i));
2571+ *cs++ = upper_32_bits(pd_daddr);
2572+ *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i));
2573+ *cs++ = lower_32_bits(pd_daddr);
2574+ }
2575+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
2576+ intel_ring_advance(rq, cs);
2577+
2578+ intel_ring_advance(rq, cs);
2579+
2580+ return 0;
2581+}
2582+
2583+static int execlists_request_alloc(struct i915_request *request)
2584+{
2585+ int ret;
2586+
2587+ GEM_BUG_ON(!intel_context_is_pinned(request->context));
2588+
2589+ /*
2590+ * Flush enough space to reduce the likelihood of waiting after
2591+ * we start building the request - in which case we will just
2592+ * have to repeat work.
2593+ */
2594+ request->reserved_space += EXECLISTS_REQUEST_SIZE;
2595+
2596+ /*
2597+ * Note that after this point, we have committed to using
2598+ * this request as it is being used to both track the
2599+ * state of engine initialisation and liveness of the
2600+ * golden renderstate above. Think twice before you try
2601+ * to cancel/unwind this request now.
2602+ */
2603+
2604+ if (!i915_vm_is_4lvl(request->context->vm)) {
2605+ ret = emit_pdps(request);
2606+ if (ret)
2607+ return ret;
2608+ }
2609+
2610+ /* Unconditionally invalidate GPU caches and TLBs. */
2611+ ret = request->engine->emit_flush(request, EMIT_INVALIDATE);
2612+ if (ret)
2613+ return ret;
2614+
2615+ request->reserved_space -= EXECLISTS_REQUEST_SIZE;
2616+ return 0;
2617+}
2618+
2619+static void reset_csb_pointers(struct intel_engine_cs *engine)
2620+{
2621+ struct intel_engine_execlists * const execlists = &engine->execlists;
2622+ const unsigned int reset_value = execlists->csb_size - 1;
2623+
2624+ ring_set_paused(engine, 0);
2625+
2626+ /*
2627+ * Sometimes Icelake forgets to reset its pointers on a GPU reset.
2628+ * Bludgeon them with a mmio update to be sure.
2629+ */
2630+ ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
2631+ 0xffff << 16 | reset_value << 8 | reset_value);
2632+ ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
2633+
2634+ /*
2635+ * After a reset, the HW starts writing into CSB entry [0]. We
2636+ * therefore have to set our HEAD pointer back one entry so that
2637+ * the *first* entry we check is entry 0. To complicate this further,
2638+ * as we don't wait for the first interrupt after reset, we have to
2639+ * fake the HW write to point back to the last entry so that our
2640+ * inline comparison of our cached head position against the last HW
2641+ * write works even before the first interrupt.
2642+ */
2643+ execlists->csb_head = reset_value;
2644+ WRITE_ONCE(*execlists->csb_write, reset_value);
2645+ wmb(); /* Make sure this is visible to HW (paranoia?) */
2646+
2647+ /* Check that the GPU does indeed update the CSB entries! */
2648+ memset(execlists->csb_status, -1, (reset_value + 1) * sizeof(u64));
2649+ invalidate_csb_entries(&execlists->csb_status[0],
2650+ &execlists->csb_status[reset_value]);
2651+
2652+ /* Once more for luck and our trusty paranoia */
2653+ ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
2654+ 0xffff << 16 | reset_value << 8 | reset_value);
2655+ ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
2656+
2657+ GEM_BUG_ON(READ_ONCE(*execlists->csb_write) != reset_value);
2658+}
2659+
2660+static void sanitize_hwsp(struct intel_engine_cs *engine)
2661+{
2662+ struct intel_timeline *tl;
2663+
2664+ list_for_each_entry(tl, &engine->status_page.timelines, engine_link)
2665+ intel_timeline_reset_seqno(tl);
2666+}
2667+
2668+static void execlists_sanitize(struct intel_engine_cs *engine)
2669+{
2670+ GEM_BUG_ON(execlists_active(&engine->execlists));
2671+
2672+ /*
2673+ * Poison residual state on resume, in case the suspend didn't!
2674+ *
2675+ * We have to assume that across suspend/resume (or other loss
2676+ * of control) that the contents of our pinned buffers has been
2677+ * lost, replaced by garbage. Since this doesn't always happen,
2678+ * let's poison such state so that we more quickly spot when
2679+ * we falsely assume it has been preserved.
2680+ */
2681+ if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
2682+ memset(engine->status_page.addr, POISON_INUSE, PAGE_SIZE);
2683+
2684+ reset_csb_pointers(engine);
2685+
2686+ /*
2687+ * The kernel_context HWSP is stored in the status_page. As above,
2688+ * that may be lost on resume/initialisation, and so we need to
2689+ * reset the value in the HWSP.
2690+ */
2691+ sanitize_hwsp(engine);
2692+
2693+ /* And scrub the dirty cachelines for the HWSP */
2694+ clflush_cache_range(engine->status_page.addr, PAGE_SIZE);
2695+}
2696+
2697+static void enable_error_interrupt(struct intel_engine_cs *engine)
2698+{
2699+ u32 status;
2700+
2701+ engine->execlists.error_interrupt = 0;
2702+ ENGINE_WRITE(engine, RING_EMR, ~0u);
2703+ ENGINE_WRITE(engine, RING_EIR, ~0u); /* clear all existing errors */
2704+
2705+ status = ENGINE_READ(engine, RING_ESR);
2706+ if (unlikely(status)) {
2707+ drm_err(&engine->i915->drm,
2708+ "engine '%s' resumed still in error: %08x\n",
2709+ engine->name, status);
2710+ __intel_gt_reset(engine->gt, engine->mask);
2711+ }
2712+
2713+ /*
2714+ * On current gen8+, we have 2 signals to play with
2715+ *
2716+ * - I915_ERROR_INSTUCTION (bit 0)
2717+ *
2718+ * Generate an error if the command parser encounters an invalid
2719+ * instruction
2720+ *
2721+ * This is a fatal error.
2722+ *
2723+ * - CP_PRIV (bit 2)
2724+ *
2725+ * Generate an error on privilege violation (where the CP replaces
2726+ * the instruction with a no-op). This also fires for writes into
2727+ * read-only scratch pages.
2728+ *
2729+ * This is a non-fatal error, parsing continues.
2730+ *
2731+ * * there are a few others defined for odd HW that we do not use
2732+ *
2733+ * Since CP_PRIV fires for cases where we have chosen to ignore the
2734+ * error (as the HW is validating and suppressing the mistakes), we
2735+ * only unmask the instruction error bit.
2736+ */
2737+ ENGINE_WRITE(engine, RING_EMR, ~I915_ERROR_INSTRUCTION);
2738+}
2739+
2740+static void enable_execlists(struct intel_engine_cs *engine)
2741+{
2742+ u32 mode;
2743+
2744+ assert_forcewakes_active(engine->uncore, FORCEWAKE_ALL);
2745+
2746+ intel_engine_set_hwsp_writemask(engine, ~0u); /* HWSTAM */
2747+
2748+ if (INTEL_GEN(engine->i915) >= 11)
2749+ mode = _MASKED_BIT_ENABLE(GEN11_GFX_DISABLE_LEGACY_MODE);
2750+ else
2751+ mode = _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE);
2752+ ENGINE_WRITE_FW(engine, RING_MODE_GEN7, mode);
2753+
2754+ ENGINE_WRITE_FW(engine, RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
2755+
2756+ ENGINE_WRITE_FW(engine,
2757+ RING_HWS_PGA,
2758+ i915_ggtt_offset(engine->status_page.vma));
2759+ ENGINE_POSTING_READ(engine, RING_HWS_PGA);
2760+
2761+ enable_error_interrupt(engine);
2762+}
2763+
2764+static bool unexpected_starting_state(struct intel_engine_cs *engine)
2765+{
2766+ bool unexpected = false;
2767+
2768+ if (ENGINE_READ_FW(engine, RING_MI_MODE) & STOP_RING) {
2769+ drm_dbg(&engine->i915->drm,
2770+ "STOP_RING still set in RING_MI_MODE\n");
2771+ unexpected = true;
2772+ }
2773+
2774+ return unexpected;
2775+}
2776+
2777+static int execlists_resume(struct intel_engine_cs *engine)
2778+{
2779+ intel_mocs_init_engine(engine);
2780+
2781+ intel_breadcrumbs_reset(engine->breadcrumbs);
2782+
2783+ if (GEM_SHOW_DEBUG() && unexpected_starting_state(engine)) {
2784+ struct drm_printer p = drm_debug_printer(__func__);
2785+
2786+ intel_engine_dump(engine, &p, NULL);
2787+ }
2788+
2789+ enable_execlists(engine);
2790+
2791+ return 0;
2792+}
2793+
2794+static void execlists_reset_prepare(struct intel_engine_cs *engine)
2795+{
2796+ struct intel_engine_execlists * const execlists = &engine->execlists;
2797+
2798+ ENGINE_TRACE(engine, "depth<-%d\n",
2799+ atomic_read(&execlists->tasklet.count));
2800+
2801+ /*
2802+ * Prevent request submission to the hardware until we have
2803+ * completed the reset in i915_gem_reset_finish(). If a request
2804+ * is completed by one engine, it may then queue a request
2805+ * to a second via its execlists->tasklet *just* as we are
2806+ * calling engine->resume() and also writing the ELSP.
2807+ * Turning off the execlists->tasklet until the reset is over
2808+ * prevents the race.
2809+ */
2810+ __tasklet_disable_sync_once(&execlists->tasklet);
2811+ GEM_BUG_ON(!reset_in_progress(execlists));
2812+
2813+ /*
2814+ * We stop engines, otherwise we might get failed reset and a
2815+ * dead gpu (on elk). Also as modern gpu as kbl can suffer
2816+ * from system hang if batchbuffer is progressing when
2817+ * the reset is issued, regardless of READY_TO_RESET ack.
2818+ * Thus assume it is best to stop engines on all gens
2819+ * where we have a gpu reset.
2820+ *
2821+ * WaKBLVECSSemaphoreWaitPoll:kbl (on ALL_ENGINES)
2822+ *
2823+ * FIXME: Wa for more modern gens needs to be validated
2824+ */
2825+ ring_set_paused(engine, 1);
2826+ intel_engine_stop_cs(engine);
2827+
2828+ engine->execlists.reset_ccid = active_ccid(engine);
2829+}
2830+
2831+static struct i915_request **
2832+reset_csb(struct intel_engine_cs *engine, struct i915_request **inactive)
2833+{
2834+ struct intel_engine_execlists * const execlists = &engine->execlists;
2835+
2836+ mb(); /* paranoia: read the CSB pointers from after the reset */
2837+ clflush(execlists->csb_write);
2838+ mb();
2839+
2840+ inactive = process_csb(engine, inactive); /* drain preemption events */
2841+
2842+ /* Following the reset, we need to reload the CSB read/write pointers */
2843+ reset_csb_pointers(engine);
2844+
2845+ return inactive;
2846+}
2847+
2848+static void
2849+execlists_reset_active(struct intel_engine_cs *engine, bool stalled)
2850+{
2851+ struct intel_context *ce;
2852+ struct i915_request *rq;
2853+ u32 head;
2854+
2855+ /*
2856+ * Save the currently executing context, even if we completed
2857+ * its request, it was still running at the time of the
2858+ * reset and will have been clobbered.
2859+ */
2860+ rq = active_context(engine, engine->execlists.reset_ccid);
2861+ if (!rq)
2862+ return;
2863+
2864+ ce = rq->context;
2865+ GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
2866+
2867+ if (__i915_request_is_complete(rq)) {
2868+ /* Idle context; tidy up the ring so we can restart afresh */
2869+ head = intel_ring_wrap(ce->ring, rq->tail);
2870+ goto out_replay;
2871+ }
2872+
2873+ /* We still have requests in-flight; the engine should be active */
2874+ GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
2875+
2876+ /* Context has requests still in-flight; it should not be idle! */
2877+ GEM_BUG_ON(i915_active_is_idle(&ce->active));
2878+
2879+ rq = active_request(ce->timeline, rq);
2880+ head = intel_ring_wrap(ce->ring, rq->head);
2881+ GEM_BUG_ON(head == ce->ring->tail);
2882+
2883+ /*
2884+ * If this request hasn't started yet, e.g. it is waiting on a
2885+ * semaphore, we need to avoid skipping the request or else we
2886+ * break the signaling chain. However, if the context is corrupt
2887+ * the request will not restart and we will be stuck with a wedged
2888+ * device. It is quite often the case that if we issue a reset
2889+ * while the GPU is loading the context image, that the context
2890+ * image becomes corrupt.
2891+ *
2892+ * Otherwise, if we have not started yet, the request should replay
2893+ * perfectly and we do not need to flag the result as being erroneous.
2894+ */
2895+ if (!__i915_request_has_started(rq))
2896+ goto out_replay;
2897+
2898+ /*
2899+ * If the request was innocent, we leave the request in the ELSP
2900+ * and will try to replay it on restarting. The context image may
2901+ * have been corrupted by the reset, in which case we may have
2902+ * to service a new GPU hang, but more likely we can continue on
2903+ * without impact.
2904+ *
2905+ * If the request was guilty, we presume the context is corrupt
2906+ * and have to at least restore the RING register in the context
2907+ * image back to the expected values to skip over the guilty request.
2908+ */
2909+ __i915_request_reset(rq, stalled);
2910+
2911+ /*
2912+ * We want a simple context + ring to execute the breadcrumb update.
2913+ * We cannot rely on the context being intact across the GPU hang,
2914+ * so clear it and rebuild just what we need for the breadcrumb.
2915+ * All pending requests for this context will be zapped, and any
2916+ * future request will be after userspace has had the opportunity
2917+ * to recreate its own state.
2918+ */
2919+out_replay:
2920+ ENGINE_TRACE(engine, "replay {head:%04x, tail:%04x}\n",
2921+ head, ce->ring->tail);
2922+ lrc_reset_regs(ce, engine);
2923+ ce->lrc.lrca = lrc_update_regs(ce, engine, head);
2924+}
2925+
2926+static void execlists_reset_csb(struct intel_engine_cs *engine, bool stalled)
2927+{
2928+ struct intel_engine_execlists * const execlists = &engine->execlists;
2929+ struct i915_request *post[2 * EXECLIST_MAX_PORTS];
2930+ struct i915_request **inactive;
2931+
2932+ rcu_read_lock();
2933+ inactive = reset_csb(engine, post);
2934+
2935+ execlists_reset_active(engine, true);
2936+
2937+ inactive = cancel_port_requests(execlists, inactive);
2938+ post_process_csb(post, inactive);
2939+ rcu_read_unlock();
2940+}
2941+
2942+static void execlists_reset_rewind(struct intel_engine_cs *engine, bool stalled)
2943+{
2944+ unsigned long flags;
2945+
2946+ ENGINE_TRACE(engine, "\n");
2947+
2948+ /* Process the csb, find the guilty context and throw away */
2949+ execlists_reset_csb(engine, stalled);
2950+
2951+ /* Push back any incomplete requests for replay after the reset. */
2952+ rcu_read_lock();
2953+ spin_lock_irqsave(&engine->active.lock, flags);
2954+ __unwind_incomplete_requests(engine);
2955+ spin_unlock_irqrestore(&engine->active.lock, flags);
2956+ rcu_read_unlock();
2957+}
2958+
2959+static void nop_submission_tasklet(unsigned long data)
2960+{
2961+ struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
2962+
2963+ /* The driver is wedged; don't process any more events. */
2964+ WRITE_ONCE(engine->execlists.queue_priority_hint, INT_MIN);
2965+}
2966+
2967+static void execlists_reset_cancel(struct intel_engine_cs *engine)
2968+{
2969+ struct intel_engine_execlists * const execlists = &engine->execlists;
2970+ struct i915_request *rq, *rn;
2971+ struct rb_node *rb;
2972+ unsigned long flags;
2973+
2974+ ENGINE_TRACE(engine, "\n");
2975+
2976+ /*
2977+ * Before we call engine->cancel_requests(), we should have exclusive
2978+ * access to the submission state. This is arranged for us by the
2979+ * caller disabling the interrupt generation, the tasklet and other
2980+ * threads that may then access the same state, giving us a free hand
2981+ * to reset state. However, we still need to let lockdep be aware that
2982+ * we know this state may be accessed in hardirq context, so we
2983+ * disable the irq around this manipulation and we want to keep
2984+ * the spinlock focused on its duties and not accidentally conflate
2985+ * coverage to the submission's irq state. (Similarly, although we
2986+ * shouldn't need to disable irq around the manipulation of the
2987+ * submission's irq state, we also wish to remind ourselves that
2988+ * it is irq state.)
2989+ */
2990+ execlists_reset_csb(engine, true);
2991+
2992+ rcu_read_lock();
2993+ spin_lock_irqsave(&engine->active.lock, flags);
2994+
2995+ /* Mark all executing requests as skipped. */
2996+ list_for_each_entry(rq, &engine->active.requests, sched.link)
2997+ i915_request_mark_eio(rq);
2998+ intel_engine_signal_breadcrumbs(engine);
2999+
3000+ /* Flush the queued requests to the timeline list (for retiring). */
3001+ while ((rb = rb_first_cached(&execlists->queue))) {
3002+ struct i915_priolist *p = to_priolist(rb);
3003+ int i;
3004+
3005+ priolist_for_each_request_consume(rq, rn, p, i) {
3006+ i915_request_mark_eio(rq);
3007+ __i915_request_submit(rq);
3008+ }
3009+
3010+ rb_erase_cached(&p->node, &execlists->queue);
3011+ i915_priolist_free(p);
3012+ }
3013+
3014+ /* On-hold requests will be flushed to timeline upon their release */
3015+ list_for_each_entry(rq, &engine->active.hold, sched.link)
3016+ i915_request_mark_eio(rq);
3017+
3018+ /* Cancel all attached virtual engines */
3019+ while ((rb = rb_first_cached(&execlists->virtual))) {
3020+ struct virtual_engine *ve =
3021+ rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
3022+
3023+ rb_erase_cached(rb, &execlists->virtual);
3024+ RB_CLEAR_NODE(rb);
3025+
3026+ spin_lock(&ve->base.active.lock);
3027+ rq = fetch_and_zero(&ve->request);
3028+ if (rq) {
3029+ i915_request_mark_eio(rq);
3030+
3031+ rq->engine = engine;
3032+ __i915_request_submit(rq);
3033+ i915_request_put(rq);
3034+
3035+ ve->base.execlists.queue_priority_hint = INT_MIN;
3036+ }
3037+ spin_unlock(&ve->base.active.lock);
3038+ }
3039+
3040+ /* Remaining _unready_ requests will be nop'ed when submitted */
3041+
3042+ execlists->queue_priority_hint = INT_MIN;
3043+ execlists->queue = RB_ROOT_CACHED;
3044+
3045+ GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
3046+ execlists->tasklet.func = nop_submission_tasklet;
3047+
3048+ spin_unlock_irqrestore(&engine->active.lock, flags);
3049+ rcu_read_unlock();
3050+}
3051+
3052+static void execlists_reset_finish(struct intel_engine_cs *engine)
3053+{
3054+ struct intel_engine_execlists * const execlists = &engine->execlists;
3055+
3056+ /*
3057+ * After a GPU reset, we may have requests to replay. Do so now while
3058+ * we still have the forcewake to be sure that the GPU is not allowed
3059+ * to sleep before we restart and reload a context.
3060+ *
3061+ * If the GPU reset fails, the engine may still be alive with requests
3062+ * inflight. We expect those to complete, or for the device to be
3063+ * reset as the next level of recovery, and as a final resort we
3064+ * will declare the device wedged.
3065+ */
3066+ GEM_BUG_ON(!reset_in_progress(execlists));
3067+
3068+ /* And kick in case we missed a new request submission. */
3069+ if (__tasklet_enable(&execlists->tasklet))
3070+ __execlists_kick(execlists);
3071+
3072+ ENGINE_TRACE(engine, "depth->%d\n",
3073+ atomic_read(&execlists->tasklet.count));
3074+}
3075+
3076+static void gen8_logical_ring_enable_irq(struct intel_engine_cs *engine)
3077+{
3078+ ENGINE_WRITE(engine, RING_IMR,
3079+ ~(engine->irq_enable_mask | engine->irq_keep_mask));
3080+ ENGINE_POSTING_READ(engine, RING_IMR);
3081+}
3082+
3083+static void gen8_logical_ring_disable_irq(struct intel_engine_cs *engine)
3084+{
3085+ ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
3086+}
3087+
3088+static void execlists_park(struct intel_engine_cs *engine)
3089+{
3090+ cancel_timer(&engine->execlists.timer);
3091+ cancel_timer(&engine->execlists.preempt);
3092+}
3093+
3094+static bool can_preempt(struct intel_engine_cs *engine)
3095+{
3096+ if (INTEL_GEN(engine->i915) > 8)
3097+ return true;
3098+
3099+ /* GPGPU on bdw requires extra w/a; not implemented */
3100+ return engine->class != RENDER_CLASS;
3101+}
3102+
3103+static void execlists_set_default_submission(struct intel_engine_cs *engine)
3104+{
3105+ engine->submit_request = execlists_submit_request;
3106+ engine->schedule = i915_schedule;
3107+ engine->execlists.tasklet.func = execlists_submission_tasklet;
3108+
3109+ engine->reset.prepare = execlists_reset_prepare;
3110+ engine->reset.rewind = execlists_reset_rewind;
3111+ engine->reset.cancel = execlists_reset_cancel;
3112+ engine->reset.finish = execlists_reset_finish;
3113+
3114+ engine->park = execlists_park;
3115+ engine->unpark = NULL;
3116+
3117+ engine->flags |= I915_ENGINE_SUPPORTS_STATS;
3118+ if (!intel_vgpu_active(engine->i915)) {
3119+ engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
3120+ if (can_preempt(engine)) {
3121+ engine->flags |= I915_ENGINE_HAS_PREEMPTION;
3122+ if (IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
3123+ engine->flags |= I915_ENGINE_HAS_TIMESLICES;
3124+ }
3125+ }
3126+
3127+ if (intel_engine_has_preemption(engine))
3128+ engine->emit_bb_start = gen8_emit_bb_start;
3129+ else
3130+ engine->emit_bb_start = gen8_emit_bb_start_noarb;
3131+}
3132+
3133+static void execlists_shutdown(struct intel_engine_cs *engine)
3134+{
3135+ /* Synchronise with residual timers and any softirq they raise */
3136+ del_timer_sync(&engine->execlists.timer);
3137+ del_timer_sync(&engine->execlists.preempt);
3138+ tasklet_kill(&engine->execlists.tasklet);
3139+}
3140+
3141+static void execlists_release(struct intel_engine_cs *engine)
3142+{
3143+ engine->sanitize = NULL; /* no longer in control, nothing to sanitize */
3144+
3145+ execlists_shutdown(engine);
3146+
3147+ intel_engine_cleanup_common(engine);
3148+ lrc_fini_wa_ctx(engine);
3149+}
3150+
3151+static void
3152+logical_ring_default_vfuncs(struct intel_engine_cs *engine)
3153+{
3154+ /* Default vfuncs which can be overriden by each engine. */
3155+
3156+ engine->resume = execlists_resume;
3157+
3158+ engine->cops = &execlists_context_ops;
3159+ engine->request_alloc = execlists_request_alloc;
3160+
3161+ engine->emit_flush = gen8_emit_flush_xcs;
3162+ engine->emit_init_breadcrumb = gen8_emit_init_breadcrumb;
3163+ engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb_xcs;
3164+ if (INTEL_GEN(engine->i915) >= 12) {
3165+ engine->emit_fini_breadcrumb = gen12_emit_fini_breadcrumb_xcs;
3166+ engine->emit_flush = gen12_emit_flush_xcs;
3167+ }
3168+ engine->set_default_submission = execlists_set_default_submission;
3169+
3170+ if (INTEL_GEN(engine->i915) < 11) {
3171+ engine->irq_enable = gen8_logical_ring_enable_irq;
3172+ engine->irq_disable = gen8_logical_ring_disable_irq;
3173+ } else {
3174+ /*
3175+ * TODO: On Gen11 interrupt masks need to be clear
3176+ * to allow C6 entry. Keep interrupts enabled at
3177+ * and take the hit of generating extra interrupts
3178+ * until a more refined solution exists.
3179+ */
3180+ }
3181+}
3182+
3183+static inline void
3184+logical_ring_default_irqs(struct intel_engine_cs *engine)
3185+{
3186+ unsigned int shift = 0;
3187+
3188+ if (INTEL_GEN(engine->i915) < 11) {
3189+ const u8 irq_shifts[] = {
3190+ [RCS0] = GEN8_RCS_IRQ_SHIFT,
3191+ [BCS0] = GEN8_BCS_IRQ_SHIFT,
3192+ [VCS0] = GEN8_VCS0_IRQ_SHIFT,
3193+ [VCS1] = GEN8_VCS1_IRQ_SHIFT,
3194+ [VECS0] = GEN8_VECS_IRQ_SHIFT,
3195+ };
3196+
3197+ shift = irq_shifts[engine->id];
3198+ }
3199+
3200+ engine->irq_enable_mask = GT_RENDER_USER_INTERRUPT << shift;
3201+ engine->irq_keep_mask = GT_CONTEXT_SWITCH_INTERRUPT << shift;
3202+ engine->irq_keep_mask |= GT_CS_MASTER_ERROR_INTERRUPT << shift;
3203+ engine->irq_keep_mask |= GT_WAIT_SEMAPHORE_INTERRUPT << shift;
3204+}
3205+
3206+static void rcs_submission_override(struct intel_engine_cs *engine)
3207+{
3208+ switch (INTEL_GEN(engine->i915)) {
3209+ case 12:
3210+ engine->emit_flush = gen12_emit_flush_rcs;
3211+ engine->emit_fini_breadcrumb = gen12_emit_fini_breadcrumb_rcs;
3212+ break;
3213+ case 11:
3214+ engine->emit_flush = gen11_emit_flush_rcs;
3215+ engine->emit_fini_breadcrumb = gen11_emit_fini_breadcrumb_rcs;
3216+ break;
3217+ default:
3218+ engine->emit_flush = gen8_emit_flush_rcs;
3219+ engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb_rcs;
3220+ break;
3221+ }
3222+}
3223+
3224+int intel_execlists_submission_setup(struct intel_engine_cs *engine)
3225+{
3226+ struct intel_engine_execlists * const execlists = &engine->execlists;
3227+ struct drm_i915_private *i915 = engine->i915;
3228+ struct intel_uncore *uncore = engine->uncore;
3229+ u32 base = engine->mmio_base;
3230+
3231+ tasklet_init(&engine->execlists.tasklet,
3232+ execlists_submission_tasklet, (unsigned long)engine);
3233+ timer_setup(&engine->execlists.timer, execlists_timeslice, 0);
3234+ timer_setup(&engine->execlists.preempt, execlists_preempt, 0);
3235+
3236+ logical_ring_default_vfuncs(engine);
3237+ logical_ring_default_irqs(engine);
3238+
3239+ if (engine->class == RENDER_CLASS)
3240+ rcs_submission_override(engine);
3241+
3242+ lrc_init_wa_ctx(engine);
3243+
3244+ if (HAS_LOGICAL_RING_ELSQ(i915)) {
3245+ execlists->submit_reg = uncore->regs +
3246+ i915_mmio_reg_offset(RING_EXECLIST_SQ_CONTENTS(base));
3247+ execlists->ctrl_reg = uncore->regs +
3248+ i915_mmio_reg_offset(RING_EXECLIST_CONTROL(base));
3249+ } else {
3250+ execlists->submit_reg = uncore->regs +
3251+ i915_mmio_reg_offset(RING_ELSP(base));
3252+ }
3253+
3254+ execlists->csb_status =
3255+ (u64 *)&engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];
3256+
3257+ execlists->csb_write =
3258+ &engine->status_page.addr[intel_hws_csb_write_index(i915)];
3259+
3260+ if (INTEL_GEN(i915) < 11)
3261+ execlists->csb_size = GEN8_CSB_ENTRIES;
3262+ else
3263+ execlists->csb_size = GEN11_CSB_ENTRIES;
3264+
3265+ engine->context_tag = GENMASK(BITS_PER_LONG - 2, 0);
3266+ if (INTEL_GEN(engine->i915) >= 11) {
3267+ execlists->ccid |= engine->instance << (GEN11_ENGINE_INSTANCE_SHIFT - 32);
3268+ execlists->ccid |= engine->class << (GEN11_ENGINE_CLASS_SHIFT - 32);
3269+ }
3270+
3271+ /* Finally, take ownership and responsibility for cleanup! */
3272+ engine->sanitize = execlists_sanitize;
3273+ engine->release = execlists_release;
3274+
3275+ return 0;
3276+}
3277+
3278+static struct list_head *virtual_queue(struct virtual_engine *ve)
3279+{
3280+ return &ve->base.execlists.default_priolist.requests[0];
3281+}
3282+
3283+static void rcu_virtual_context_destroy(struct work_struct *wrk)
3284+{
3285+ struct virtual_engine *ve =
3286+ container_of(wrk, typeof(*ve), rcu.work);
3287+ unsigned int n;
3288+
3289+ GEM_BUG_ON(ve->context.inflight);
3290+
3291+ /* Preempt-to-busy may leave a stale request behind. */
3292+ if (unlikely(ve->request)) {
3293+ struct i915_request *old;
3294+
3295+ spin_lock_irq(&ve->base.active.lock);
3296+
3297+ old = fetch_and_zero(&ve->request);
3298+ if (old) {
3299+ GEM_BUG_ON(!i915_request_completed(old));
3300+ __i915_request_submit(old);
3301+ i915_request_put(old);
3302+ }
3303+
3304+ spin_unlock_irq(&ve->base.active.lock);
3305+ }
3306+
3307+ /*
3308+ * Flush the tasklet in case it is still running on another core.
3309+ *
3310+ * This needs to be done before we remove ourselves from the siblings'
3311+ * rbtrees as in the case it is running in parallel, it may reinsert
3312+ * the rb_node into a sibling.
3313+ */
3314+ tasklet_kill(&ve->base.execlists.tasklet);
3315+
3316+ /* Decouple ourselves from the siblings, no more access allowed. */
3317+ for (n = 0; n < ve->num_siblings; n++) {
3318+ struct intel_engine_cs *sibling = ve->siblings[n];
3319+ struct rb_node *node = &ve->nodes[sibling->id].rb;
3320+
3321+ if (RB_EMPTY_NODE(node))
3322+ continue;
3323+
3324+ spin_lock_irq(&sibling->active.lock);
3325+
3326+ /* Detachment is lazily performed in the execlists tasklet */
3327+ if (!RB_EMPTY_NODE(node))
3328+ rb_erase_cached(node, &sibling->execlists.virtual);
3329+
3330+ spin_unlock_irq(&sibling->active.lock);
3331+ }
3332+ GEM_BUG_ON(__tasklet_is_scheduled(&ve->base.execlists.tasklet));
3333+ GEM_BUG_ON(!list_empty(virtual_queue(ve)));
3334+
3335+ lrc_fini(&ve->context);
3336+ intel_context_fini(&ve->context);
3337+
3338+ intel_breadcrumbs_free(ve->base.breadcrumbs);
3339+ intel_engine_free_request_pool(&ve->base);
3340+
3341+ kfree(ve->bonds);
3342+ kfree(ve);
3343+}
3344+
3345+static void virtual_context_destroy(struct kref *kref)
3346+{
3347+ struct virtual_engine *ve =
3348+ container_of(kref, typeof(*ve), context.ref);
3349+
3350+ GEM_BUG_ON(!list_empty(&ve->context.signals));
3351+
3352+ /*
3353+ * When destroying the virtual engine, we have to be aware that
3354+ * it may still be in use from an hardirq/softirq context causing
3355+ * the resubmission of a completed request (background completion
3356+ * due to preempt-to-busy). Before we can free the engine, we need
3357+ * to flush the submission code and tasklets that are still potentially
3358+ * accessing the engine. Flushing the tasklets requires process context,
3359+ * and since we can guard the resubmit onto the engine with an RCU read
3360+ * lock, we can delegate the free of the engine to an RCU worker.
3361+ */
3362+ INIT_RCU_WORK(&ve->rcu, rcu_virtual_context_destroy);
3363+ queue_rcu_work(system_wq, &ve->rcu);
3364+}
3365+
3366+static void virtual_engine_initial_hint(struct virtual_engine *ve)
3367+{
3368+ int swp;
3369+
3370+ /*
3371+ * Pick a random sibling on starting to help spread the load around.
3372+ *
3373+ * New contexts are typically created with exactly the same order
3374+ * of siblings, and often started in batches. Due to the way we iterate
3375+ * the array of sibling when submitting requests, sibling[0] is
3376+ * prioritised for dequeuing. If we make sure that sibling[0] is fairly
3377+ * randomised across the system, we also help spread the load by the
3378+ * first engine we inspect being different each time.
3379+ *
3380+ * NB This does not force us to execute on this engine, it will just
3381+ * typically be the first we inspect for submission.
3382+ */
3383+ swp = prandom_u32_max(ve->num_siblings);
3384+ if (swp)
3385+ swap(ve->siblings[swp], ve->siblings[0]);
3386+}
3387+
3388+static int virtual_context_alloc(struct intel_context *ce)
3389+{
3390+ struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3391+
3392+ return lrc_alloc(ce, ve->siblings[0]);
3393+}
3394+
3395+static int virtual_context_pre_pin(struct intel_context *ce,
3396+ struct i915_gem_ww_ctx *ww,
3397+ void **vaddr)
3398+{
3399+ struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3400+
3401+ /* Note: we must use a real engine class for setting up reg state */
3402+ return lrc_pre_pin(ce, ve->siblings[0], ww, vaddr);
3403+}
3404+
3405+static int virtual_context_pin(struct intel_context *ce, void *vaddr)
3406+{
3407+ struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3408+
3409+ return lrc_pin(ce, ve->siblings[0], vaddr);
3410+}
3411+
3412+static void virtual_context_enter(struct intel_context *ce)
3413+{
3414+ struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3415+ unsigned int n;
3416+
3417+ for (n = 0; n < ve->num_siblings; n++)
3418+ intel_engine_pm_get(ve->siblings[n]);
3419+
3420+ intel_timeline_enter(ce->timeline);
3421+}
3422+
3423+static void virtual_context_exit(struct intel_context *ce)
3424+{
3425+ struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3426+ unsigned int n;
3427+
3428+ intel_timeline_exit(ce->timeline);
3429+
3430+ for (n = 0; n < ve->num_siblings; n++)
3431+ intel_engine_pm_put(ve->siblings[n]);
3432+}
3433+
3434+static const struct intel_context_ops virtual_context_ops = {
3435+ .flags = COPS_HAS_INFLIGHT,
3436+
3437+ .alloc = virtual_context_alloc,
3438+
3439+ .pre_pin = virtual_context_pre_pin,
3440+ .pin = virtual_context_pin,
3441+ .unpin = lrc_unpin,
3442+ .post_unpin = lrc_post_unpin,
3443+
3444+ .enter = virtual_context_enter,
3445+ .exit = virtual_context_exit,
3446+
3447+ .destroy = virtual_context_destroy,
3448+};
3449+
3450+static intel_engine_mask_t virtual_submission_mask(struct virtual_engine *ve)
3451+{
3452+ struct i915_request *rq;
3453+ intel_engine_mask_t mask;
3454+
3455+ rq = READ_ONCE(ve->request);
3456+ if (!rq)
3457+ return 0;
3458+
3459+ /* The rq is ready for submission; rq->execution_mask is now stable. */
3460+ mask = rq->execution_mask;
3461+ if (unlikely(!mask)) {
3462+ /* Invalid selection, submit to a random engine in error */
3463+ i915_request_set_error_once(rq, -ENODEV);
3464+ mask = ve->siblings[0]->mask;
3465+ }
3466+
3467+ ENGINE_TRACE(&ve->base, "rq=%llx:%lld, mask=%x, prio=%d\n",
3468+ rq->fence.context, rq->fence.seqno,
3469+ mask, ve->base.execlists.queue_priority_hint);
3470+
3471+ return mask;
3472+}
3473+
3474+static void virtual_submission_tasklet(unsigned long data)
3475+{
3476+ struct virtual_engine * const ve = (struct virtual_engine *)data;
3477+ const int prio = READ_ONCE(ve->base.execlists.queue_priority_hint);
3478+ intel_engine_mask_t mask;
3479+ unsigned int n;
3480+
3481+ rcu_read_lock();
3482+ mask = virtual_submission_mask(ve);
3483+ rcu_read_unlock();
3484+ if (unlikely(!mask))
3485+ return;
3486+
3487+ for (n = 0; n < ve->num_siblings; n++) {
3488+ struct intel_engine_cs *sibling = READ_ONCE(ve->siblings[n]);
3489+ struct ve_node * const node = &ve->nodes[sibling->id];
3490+ struct rb_node **parent, *rb;
3491+ bool first;
3492+
3493+ if (!READ_ONCE(ve->request))
3494+ break; /* already handled by a sibling's tasklet */
3495+
3496+ spin_lock_irq(&sibling->active.lock);
3497+
3498+ if (unlikely(!(mask & sibling->mask))) {
3499+ if (!RB_EMPTY_NODE(&node->rb)) {
3500+ rb_erase_cached(&node->rb,
3501+ &sibling->execlists.virtual);
3502+ RB_CLEAR_NODE(&node->rb);
3503+ }
3504+
3505+ goto unlock_engine;
3506+ }
3507+
3508+ if (unlikely(!RB_EMPTY_NODE(&node->rb))) {
3509+ /*
3510+ * Cheat and avoid rebalancing the tree if we can
3511+ * reuse this node in situ.
3512+ */
3513+ first = rb_first_cached(&sibling->execlists.virtual) ==
3514+ &node->rb;
3515+ if (prio == node->prio || (prio > node->prio && first))
3516+ goto submit_engine;
3517+
3518+ rb_erase_cached(&node->rb, &sibling->execlists.virtual);
3519+ }
3520+
3521+ rb = NULL;
3522+ first = true;
3523+ parent = &sibling->execlists.virtual.rb_root.rb_node;
3524+ while (*parent) {
3525+ struct ve_node *other;
3526+
3527+ rb = *parent;
3528+ other = rb_entry(rb, typeof(*other), rb);
3529+ if (prio > other->prio) {
3530+ parent = &rb->rb_left;
3531+ } else {
3532+ parent = &rb->rb_right;
3533+ first = false;
3534+ }
3535+ }
3536+
3537+ rb_link_node(&node->rb, rb, parent);
3538+ rb_insert_color_cached(&node->rb,
3539+ &sibling->execlists.virtual,
3540+ first);
3541+
3542+submit_engine:
3543+ GEM_BUG_ON(RB_EMPTY_NODE(&node->rb));
3544+ node->prio = prio;
3545+ if (first && prio > sibling->execlists.queue_priority_hint)
3546+ tasklet_hi_schedule(&sibling->execlists.tasklet);
3547+
3548+unlock_engine:
3549+ spin_unlock_irq(&sibling->active.lock);
3550+
3551+ if (intel_context_inflight(&ve->context))
3552+ break;
3553+ }
3554+}
3555+
3556+static void virtual_submit_request(struct i915_request *rq)
3557+{
3558+ struct virtual_engine *ve = to_virtual_engine(rq->engine);
3559+ unsigned long flags;
3560+
3561+ ENGINE_TRACE(&ve->base, "rq=%llx:%lld\n",
3562+ rq->fence.context,
3563+ rq->fence.seqno);
3564+
3565+ GEM_BUG_ON(ve->base.submit_request != virtual_submit_request);
3566+
3567+ spin_lock_irqsave(&ve->base.active.lock, flags);
3568+
3569+ /* By the time we resubmit a request, it may be completed */
3570+ if (__i915_request_is_complete(rq)) {
3571+ __i915_request_submit(rq);
3572+ goto unlock;
3573+ }
3574+
3575+ if (ve->request) { /* background completion from preempt-to-busy */
3576+ GEM_BUG_ON(!i915_request_completed(ve->request));
3577+ __i915_request_submit(ve->request);
3578+ i915_request_put(ve->request);
3579+ }
3580+
3581+ ve->base.execlists.queue_priority_hint = rq_prio(rq);
3582+ ve->request = i915_request_get(rq);
3583+
3584+ GEM_BUG_ON(!list_empty(virtual_queue(ve)));
3585+ list_move_tail(&rq->sched.link, virtual_queue(ve));
3586+
3587+ tasklet_hi_schedule(&ve->base.execlists.tasklet);
3588+
3589+unlock:
3590+ spin_unlock_irqrestore(&ve->base.active.lock, flags);
3591+}
3592+
3593+static struct ve_bond *
3594+virtual_find_bond(struct virtual_engine *ve,
3595+ const struct intel_engine_cs *master)
3596+{
3597+ int i;
3598+
3599+ for (i = 0; i < ve->num_bonds; i++) {
3600+ if (ve->bonds[i].master == master)
3601+ return &ve->bonds[i];
3602+ }
3603+
3604+ return NULL;
3605+}
3606+
3607+static void
3608+virtual_bond_execute(struct i915_request *rq, struct dma_fence *signal)
3609+{
3610+ struct virtual_engine *ve = to_virtual_engine(rq->engine);
3611+ intel_engine_mask_t allowed, exec;
3612+ struct ve_bond *bond;
3613+
3614+ allowed = ~to_request(signal)->engine->mask;
3615+
3616+ bond = virtual_find_bond(ve, to_request(signal)->engine);
3617+ if (bond)
3618+ allowed &= bond->sibling_mask;
3619+
3620+ /* Restrict the bonded request to run on only the available engines */
3621+ exec = READ_ONCE(rq->execution_mask);
3622+ while (!try_cmpxchg(&rq->execution_mask, &exec, exec & allowed))
3623+ ;
3624+
3625+ /* Prevent the master from being re-run on the bonded engines */
3626+ to_request(signal)->execution_mask &= ~allowed;
3627+}
3628+
3629+struct intel_context *
3630+intel_execlists_create_virtual(struct intel_engine_cs **siblings,
3631+ unsigned int count)
3632+{
3633+ struct virtual_engine *ve;
3634+ unsigned int n;
3635+ int err;
3636+
3637+ if (count == 0)
3638+ return ERR_PTR(-EINVAL);
3639+
3640+ if (count == 1)
3641+ return intel_context_create(siblings[0]);
3642+
3643+ ve = kzalloc(struct_size(ve, siblings, count), GFP_KERNEL);
3644+ if (!ve)
3645+ return ERR_PTR(-ENOMEM);
3646+
3647+ ve->base.i915 = siblings[0]->i915;
3648+ ve->base.gt = siblings[0]->gt;
3649+ ve->base.uncore = siblings[0]->uncore;
3650+ ve->base.id = -1;
3651+
3652+ ve->base.class = OTHER_CLASS;
3653+ ve->base.uabi_class = I915_ENGINE_CLASS_INVALID;
3654+ ve->base.instance = I915_ENGINE_CLASS_INVALID_VIRTUAL;
3655+ ve->base.uabi_instance = I915_ENGINE_CLASS_INVALID_VIRTUAL;
3656+
3657+ /*
3658+ * The decision on whether to submit a request using semaphores
3659+ * depends on the saturated state of the engine. We only compute
3660+ * this during HW submission of the request, and we need for this
3661+ * state to be globally applied to all requests being submitted
3662+ * to this engine. Virtual engines encompass more than one physical
3663+ * engine and so we cannot accurately tell in advance if one of those
3664+ * engines is already saturated and so cannot afford to use a semaphore
3665+ * and be pessimized in priority for doing so -- if we are the only
3666+ * context using semaphores after all other clients have stopped, we
3667+ * will be starved on the saturated system. Such a global switch for
3668+ * semaphores is less than ideal, but alas is the current compromise.
3669+ */
3670+ ve->base.saturated = ALL_ENGINES;
3671+
3672+ snprintf(ve->base.name, sizeof(ve->base.name), "virtual");
3673+
3674+ intel_engine_init_active(&ve->base, ENGINE_VIRTUAL);
3675+ intel_engine_init_execlists(&ve->base);
3676+
3677+ ve->base.cops = &virtual_context_ops;
3678+ ve->base.request_alloc = execlists_request_alloc;
3679+
3680+ ve->base.schedule = i915_schedule;
3681+ ve->base.submit_request = virtual_submit_request;
3682+ ve->base.bond_execute = virtual_bond_execute;
3683+
3684+ INIT_LIST_HEAD(virtual_queue(ve));
3685+ ve->base.execlists.queue_priority_hint = INT_MIN;
3686+ tasklet_init(&ve->base.execlists.tasklet,
3687+ virtual_submission_tasklet,
3688+ (unsigned long)ve);
3689+
3690+ intel_context_init(&ve->context, &ve->base);
3691+
3692+ ve->base.breadcrumbs = intel_breadcrumbs_create(NULL);
3693+ if (!ve->base.breadcrumbs) {
3694+ err = -ENOMEM;
3695+ goto err_put;
3696+ }
3697+
3698+ for (n = 0; n < count; n++) {
3699+ struct intel_engine_cs *sibling = siblings[n];
3700+
3701+ GEM_BUG_ON(!is_power_of_2(sibling->mask));
3702+ if (sibling->mask & ve->base.mask) {
3703+ DRM_DEBUG("duplicate %s entry in load balancer\n",
3704+ sibling->name);
3705+ err = -EINVAL;
3706+ goto err_put;
3707+ }
3708+
3709+ /*
3710+ * The virtual engine implementation is tightly coupled to
3711+ * the execlists backend -- we push out request directly
3712+ * into a tree inside each physical engine. We could support
3713+ * layering if we handle cloning of the requests and
3714+ * submitting a copy into each backend.
3715+ */
3716+ if (sibling->execlists.tasklet.func !=
3717+ execlists_submission_tasklet) {
3718+ err = -ENODEV;
3719+ goto err_put;
3720+ }
3721+
3722+ GEM_BUG_ON(RB_EMPTY_NODE(&ve->nodes[sibling->id].rb));
3723+ RB_CLEAR_NODE(&ve->nodes[sibling->id].rb);
3724+
3725+ ve->siblings[ve->num_siblings++] = sibling;
3726+ ve->base.mask |= sibling->mask;
3727+
3728+ /*
3729+ * All physical engines must be compatible for their emission
3730+ * functions (as we build the instructions during request
3731+ * construction and do not alter them before submission
3732+ * on the physical engine). We use the engine class as a guide
3733+ * here, although that could be refined.
3734+ */
3735+ if (ve->base.class != OTHER_CLASS) {
3736+ if (ve->base.class != sibling->class) {
3737+ DRM_DEBUG("invalid mixing of engine class, sibling %d, already %d\n",
3738+ sibling->class, ve->base.class);
3739+ err = -EINVAL;
3740+ goto err_put;
3741+ }
3742+ continue;
3743+ }
3744+
3745+ ve->base.class = sibling->class;
3746+ ve->base.uabi_class = sibling->uabi_class;
3747+ snprintf(ve->base.name, sizeof(ve->base.name),
3748+ "v%dx%d", ve->base.class, count);
3749+ ve->base.context_size = sibling->context_size;
3750+
3751+ ve->base.emit_bb_start = sibling->emit_bb_start;
3752+ ve->base.emit_flush = sibling->emit_flush;
3753+ ve->base.emit_init_breadcrumb = sibling->emit_init_breadcrumb;
3754+ ve->base.emit_fini_breadcrumb = sibling->emit_fini_breadcrumb;
3755+ ve->base.emit_fini_breadcrumb_dw =
3756+ sibling->emit_fini_breadcrumb_dw;
3757+
3758+ ve->base.flags = sibling->flags;
3759+ }
3760+
3761+ ve->base.flags |= I915_ENGINE_IS_VIRTUAL;
3762+
3763+ virtual_engine_initial_hint(ve);
3764+ return &ve->context;
3765+
3766+err_put:
3767+ intel_context_put(&ve->context);
3768+ return ERR_PTR(err);
3769+}
3770+
3771+struct intel_context *
3772+intel_execlists_clone_virtual(struct intel_engine_cs *src)
3773+{
3774+ struct virtual_engine *se = to_virtual_engine(src);
3775+ struct intel_context *dst;
3776+
3777+ dst = intel_execlists_create_virtual(se->siblings,
3778+ se->num_siblings);
3779+ if (IS_ERR(dst))
3780+ return dst;
3781+
3782+ if (se->num_bonds) {
3783+ struct virtual_engine *de = to_virtual_engine(dst->engine);
3784+
3785+ de->bonds = kmemdup(se->bonds,
3786+ sizeof(*se->bonds) * se->num_bonds,
3787+ GFP_KERNEL);
3788+ if (!de->bonds) {
3789+ intel_context_put(dst);
3790+ return ERR_PTR(-ENOMEM);
3791+ }
3792+
3793+ de->num_bonds = se->num_bonds;
3794+ }
3795+
3796+ return dst;
3797+}
3798+
3799+int intel_virtual_engine_attach_bond(struct intel_engine_cs *engine,
3800+ const struct intel_engine_cs *master,
3801+ const struct intel_engine_cs *sibling)
3802+{
3803+ struct virtual_engine *ve = to_virtual_engine(engine);
3804+ struct ve_bond *bond;
3805+ int n;
3806+
3807+ /* Sanity check the sibling is part of the virtual engine */
3808+ for (n = 0; n < ve->num_siblings; n++)
3809+ if (sibling == ve->siblings[n])
3810+ break;
3811+ if (n == ve->num_siblings)
3812+ return -EINVAL;
3813+
3814+ bond = virtual_find_bond(ve, master);
3815+ if (bond) {
3816+ bond->sibling_mask |= sibling->mask;
3817+ return 0;
3818+ }
3819+
3820+ bond = krealloc(ve->bonds,
3821+ sizeof(*bond) * (ve->num_bonds + 1),
3822+ GFP_KERNEL);
3823+ if (!bond)
3824+ return -ENOMEM;
3825+
3826+ bond[ve->num_bonds].master = master;
3827+ bond[ve->num_bonds].sibling_mask = sibling->mask;
3828+
3829+ ve->bonds = bond;
3830+ ve->num_bonds++;
3831+
3832+ return 0;
3833+}
3834+
3835+void intel_execlists_show_requests(struct intel_engine_cs *engine,
3836+ struct drm_printer *m,
3837+ void (*show_request)(struct drm_printer *m,
3838+ const struct i915_request *rq,
3839+ const char *prefix,
3840+ int indent),
3841+ unsigned int max)
3842+{
3843+ const struct intel_engine_execlists *execlists = &engine->execlists;
3844+ struct i915_request *rq, *last;
3845+ unsigned long flags;
3846+ unsigned int count;
3847+ struct rb_node *rb;
3848+
3849+ spin_lock_irqsave(&engine->active.lock, flags);
3850+
3851+ last = NULL;
3852+ count = 0;
3853+ list_for_each_entry(rq, &engine->active.requests, sched.link) {
3854+ if (count++ < max - 1)
3855+ show_request(m, rq, "\t\t", 0);
3856+ else
3857+ last = rq;
3858+ }
3859+ if (last) {
3860+ if (count > max) {
3861+ drm_printf(m,
3862+ "\t\t...skipping %d executing requests...\n",
3863+ count - max);
3864+ }
3865+ show_request(m, last, "\t\t", 0);
3866+ }
3867+
3868+ if (execlists->queue_priority_hint != INT_MIN)
3869+ drm_printf(m, "\t\tQueue priority hint: %d\n",
3870+ READ_ONCE(execlists->queue_priority_hint));
3871+
3872+ last = NULL;
3873+ count = 0;
3874+ for (rb = rb_first_cached(&execlists->queue); rb; rb = rb_next(rb)) {
3875+ struct i915_priolist *p = rb_entry(rb, typeof(*p), node);
3876+ int i;
3877+
3878+ priolist_for_each_request(rq, p, i) {
3879+ if (count++ < max - 1)
3880+ show_request(m, rq, "\t\t", 0);
3881+ else
3882+ last = rq;
3883+ }
3884+ }
3885+ if (last) {
3886+ if (count > max) {
3887+ drm_printf(m,
3888+ "\t\t...skipping %d queued requests...\n",
3889+ count - max);
3890+ }
3891+ show_request(m, last, "\t\t", 0);
3892+ }
3893+
3894+ last = NULL;
3895+ count = 0;
3896+ for (rb = rb_first_cached(&execlists->virtual); rb; rb = rb_next(rb)) {
3897+ struct virtual_engine *ve =
3898+ rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
3899+ struct i915_request *rq = READ_ONCE(ve->request);
3900+
3901+ if (rq) {
3902+ if (count++ < max - 1)
3903+ show_request(m, rq, "\t\t", 0);
3904+ else
3905+ last = rq;
3906+ }
3907+ }
3908+ if (last) {
3909+ if (count > max) {
3910+ drm_printf(m,
3911+ "\t\t...skipping %d virtual requests...\n",
3912+ count - max);
3913+ }
3914+ show_request(m, last, "\t\t", 0);
3915+ }
3916+
3917+ spin_unlock_irqrestore(&engine->active.lock, flags);
3918+}
3919+
3920+bool
3921+intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine)
3922+{
3923+ return engine->set_default_submission ==
3924+ execlists_set_default_submission;
3925+}
3926+
3927+#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
3928+#include "selftest_execlists.c"
3929+#endif
drivers/gpu/drm/i915/gt/intel_execlists_submission.h+47−0
@@ -0,0 +1,47 @@
1+/* SPDX-License-Identifier: MIT */
2+/*
3+ * Copyright © 2014 Intel Corporation
4+ */
5+
6+#ifndef __INTEL_EXECLISTS_SUBMISSION_H__
7+#define __INTEL_EXECLISTS_SUBMISSION_H__
8+
9+#include <linux/types.h>
10+
11+struct drm_printer;
12+
13+struct i915_request;
14+struct intel_context;
15+struct intel_engine_cs;
16+
17+enum {
18+ INTEL_CONTEXT_SCHEDULE_IN = 0,
19+ INTEL_CONTEXT_SCHEDULE_OUT,
20+ INTEL_CONTEXT_SCHEDULE_PREEMPTED,
21+};
22+
23+int intel_execlists_submission_setup(struct intel_engine_cs *engine);
24+
25+void intel_execlists_show_requests(struct intel_engine_cs *engine,
26+ struct drm_printer *m,
27+ void (*show_request)(struct drm_printer *m,
28+ const struct i915_request *rq,
29+ const char *prefix,
30+ int indent),
31+ unsigned int max);
32+
33+struct intel_context *
34+intel_execlists_create_virtual(struct intel_engine_cs **siblings,
35+ unsigned int count);
36+
37+struct intel_context *
38+intel_execlists_clone_virtual(struct intel_engine_cs *src);
39+
40+int intel_virtual_engine_attach_bond(struct intel_engine_cs *engine,
41+ const struct intel_engine_cs *master,
42+ const struct intel_engine_cs *sibling);
43+
44+bool
45+intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine);
46+
47+#endif /* __INTEL_EXECLISTS_SUBMISSION_H__ */
drivers/gpu/drm/i915/gt/intel_ggtt.c+16−2
@@ -101,7 +101,16 @@ static bool needs_idle_maps(struct drm_i915_private *i915) · 100 unmodified lines
101101 * Query intel_iommu to see if we need the workaround. Presumably that
102102 * was loaded first.
103103 */
104− return IS_GEN(i915, 5) && IS_MOBILE(i915) && intel_vtd_active();
104+ if (!intel_vtd_active())
105+ return false;
106+
107+ if (IS_GEN(i915, 5) && IS_MOBILE(i915))
108+ return true;
109+
110+ if (IS_GEN(i915, 12))
111+ return true; /* XXX DMAR fault reason 7 */
112+
113+ return false;
105114 }
106115
107116 void i915_ggtt_suspend(struct i915_ggtt *ggtt)
@@ -1050,7 +1059,12 @@ static int i915_gmch_probe(struct i915_ggtt *ggtt) · 942 unmodified lines
10501059
10511060 ggtt->vm.alloc_pt_dma = alloc_pt_dma;
10521061
1053− ggtt->do_idle_maps = needs_idle_maps(i915);
1062+ if (needs_idle_maps(i915)) {
1063+ drm_notice(&i915->drm,
1064+ "Flushing DMA requests before IOMMU unmaps; performance may be degraded\n");
1065+ ggtt->do_idle_maps = true;
1066+ }
1067+
10541068 ggtt->vm.insert_page = i915_ggtt_insert_page;
10551069 ggtt->vm.insert_entries = i915_ggtt_insert_entries;
10561070 ggtt->vm.clear_range = i915_ggtt_clear_range;
drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c+20−2
@@ -320,13 +320,31 @@ void i915_vma_revoke_fence(struct i915_vma *vma) · 319 unmodified lines
320320 fence_write(fence);
321321 }
322322
323+static bool fence_is_active(const struct i915_fence_reg *fence)
324+{
325+ return fence->vma && i915_vma_is_active(fence->vma);
326+}
327+
323328 static struct i915_fence_reg *fence_find(struct i915_ggtt *ggtt)
324329 {
325− struct i915_fence_reg *fence;
330+ struct i915_fence_reg *active = NULL;
331+ struct i915_fence_reg *fence, *fn;
326332
327− list_for_each_entry(fence, &ggtt->fence_list, link) {
333+ list_for_each_entry_safe(fence, fn, &ggtt->fence_list, link) {
328334 GEM_BUG_ON(fence->vma && fence->vma->fence != fence);
329335
336+ if (fence == active) /* now seen this fence twice */
337+ active = ERR_PTR(-EAGAIN);
338+
339+ /* Prefer idle fences so we do not have to wait on the GPU */
340+ if (active != ERR_PTR(-EAGAIN) && fence_is_active(fence)) {
341+ if (!active)
342+ active = fence;
343+
344+ list_move_tail(&fence->link, &ggtt->fence_list);
345+ continue;
346+ }
347+
330348 if (atomic_read(&fence->pin_count))
331349 continue;
332350
drivers/gpu/drm/i915/gt/intel_gt.c+2−2
@@ -46,6 +46,8 @@ void intel_gt_init_hw_early(struct intel_gt *gt, struct i915_ggtt *ggtt) · 45 unmodified lines
4646
4747 int intel_gt_init_mmio(struct intel_gt *gt)
4848 {
49+ intel_gt_init_clock_frequency(gt);
50+
4951 intel_uc_init_mmio(&gt->uc);
5052 intel_sseu_info_init(gt);
5153
@@ -546,8 +548,6 @@ int intel_gt_init(struct intel_gt *gt) · 494 unmodified lines
546548 */
547549 intel_uncore_forcewake_get(gt->uncore, FORCEWAKE_ALL);
548550
549− intel_gt_init_clock_frequency(gt);
550−
551551 err = intel_gt_init_scratch(gt, IS_GEN(gt->i915, 2) ? SZ_256K : SZ_4K);
552552 if (err)
553553 goto out_fw;
drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c+157−40
@@ -7,34 +7,146 @@ · 6 unmodified lines
77 #include "intel_gt.h"
88 #include "intel_gt_clock_utils.h"
99
10−#define MHZ_12 12000000 /* 12MHz (24MHz/2), 83.333ns */
11−#define MHZ_12_5 12500000 /* 12.5MHz (25MHz/2), 80ns */
12−#define MHZ_19_2 19200000 /* 19.2MHz, 52.083ns */
10+static u32 read_reference_ts_freq(struct intel_uncore *uncore)
11+{
12+ u32 ts_override = intel_uncore_read(uncore, GEN9_TIMESTAMP_OVERRIDE);
13+ u32 base_freq, frac_freq;
14+
15+ base_freq = ((ts_override & GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_MASK) >>
16+ GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_SHIFT) + 1;
17+ base_freq *= 1000000;
18+
19+ frac_freq = ((ts_override &
20+ GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_MASK) >>
21+ GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_SHIFT);
22+ frac_freq = 1000000 / (frac_freq + 1);
23+
24+ return base_freq + frac_freq;
25+}
26+
27+static u32 gen10_get_crystal_clock_freq(struct intel_uncore *uncore,
28+ u32 rpm_config_reg)
29+{
30+ u32 f19_2_mhz = 19200000;
31+ u32 f24_mhz = 24000000;
32+ u32 crystal_clock =
33+ (rpm_config_reg & GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >>
34+ GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT;
1335
14−static u32 read_clock_frequency(const struct intel_gt *gt)
36+ switch (crystal_clock) {
37+ case GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_19_2_MHZ:
38+ return f19_2_mhz;
39+ case GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_24_MHZ:
40+ return f24_mhz;
41+ default:
42+ MISSING_CASE(crystal_clock);
43+ return 0;
44+ }
45+}
46+
47+static u32 gen11_get_crystal_clock_freq(struct intel_uncore *uncore,
48+ u32 rpm_config_reg)
1549 {
16− if (INTEL_GEN(gt->i915) >= 11) {
17− u32 config;
18−
19− config = intel_uncore_read(gt->uncore, RPM_CONFIG0);
20− config &= GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK;
21− config >>= GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT;
22−
23− switch (config) {
24− case 0: return MHZ_12;
25− case 1:
26− case 2: return MHZ_19_2;
27− default:
28− case 3: return MHZ_12_5;
50+ u32 f19_2_mhz = 19200000;
51+ u32 f24_mhz = 24000000;
52+ u32 f25_mhz = 25000000;
53+ u32 f38_4_mhz = 38400000;
54+ u32 crystal_clock =
55+ (rpm_config_reg & GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >>
56+ GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT;
57+
58+ switch (crystal_clock) {
59+ case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_24_MHZ:
60+ return f24_mhz;
61+ case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_19_2_MHZ:
62+ return f19_2_mhz;
63+ case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_38_4_MHZ:
64+ return f38_4_mhz;
65+ case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_25_MHZ:
66+ return f25_mhz;
67+ default:
68+ MISSING_CASE(crystal_clock);
69+ return 0;
70+ }
71+}
72+
73+static u32 read_clock_frequency(struct intel_uncore *uncore)
74+{
75+ u32 f12_5_mhz = 12500000;
76+ u32 f19_2_mhz = 19200000;
77+ u32 f24_mhz = 24000000;
78+
79+ if (INTEL_GEN(uncore->i915) <= 4) {
80+ /*
81+ * PRMs say:
82+ *
83+ * "The value in this register increments once every 16
84+ * hclks." (through the “Clocking Configuration”
85+ * (“CLKCFG”) MCHBAR register)
86+ */
87+ return RUNTIME_INFO(uncore->i915)->rawclk_freq * 1000 / 16;
88+ } else if (INTEL_GEN(uncore->i915) <= 8) {
89+ /*
90+ * PRMs say:
91+ *
92+ * "The PCU TSC counts 10ns increments; this timestamp
93+ * reflects bits 38:3 of the TSC (i.e. 80ns granularity,
94+ * rolling over every 1.5 hours).
95+ */
96+ return f12_5_mhz;
97+ } else if (INTEL_GEN(uncore->i915) <= 9) {
98+ u32 ctc_reg = intel_uncore_read(uncore, CTC_MODE);
99+ u32 freq = 0;
100+
101+ if ((ctc_reg & CTC_SOURCE_PARAMETER_MASK) == CTC_SOURCE_DIVIDE_LOGIC) {
102+ freq = read_reference_ts_freq(uncore);
103+ } else {
104+ freq = IS_GEN9_LP(uncore->i915) ? f19_2_mhz : f24_mhz;
105+
106+ /*
107+ * Now figure out how the command stream's timestamp
108+ * register increments from this frequency (it might
109+ * increment only every few clock cycle).
110+ */
111+ freq >>= 3 - ((ctc_reg & CTC_SHIFT_PARAMETER_MASK) >>
112+ CTC_SHIFT_PARAMETER_SHIFT);
29113 }
30− } else if (INTEL_GEN(gt->i915) >= 9) {
31− if (IS_GEN9_LP(gt->i915))
32− return MHZ_19_2;
33− else
34− return MHZ_12;
35− } else {
36− return MHZ_12_5;
114+
115+ return freq;
116+ } else if (INTEL_GEN(uncore->i915) <= 12) {
117+ u32 ctc_reg = intel_uncore_read(uncore, CTC_MODE);
118+ u32 freq = 0;
119+
120+ /*
121+ * First figure out the reference frequency. There are 2 ways
122+ * we can compute the frequency, either through the
123+ * TIMESTAMP_OVERRIDE register or through RPM_CONFIG. CTC_MODE
124+ * tells us which one we should use.
125+ */
126+ if ((ctc_reg & CTC_SOURCE_PARAMETER_MASK) == CTC_SOURCE_DIVIDE_LOGIC) {
127+ freq = read_reference_ts_freq(uncore);
128+ } else {
129+ u32 c0 = intel_uncore_read(uncore, RPM_CONFIG0);
130+
131+ if (INTEL_GEN(uncore->i915) <= 10)
132+ freq = gen10_get_crystal_clock_freq(uncore, c0);
133+ else
134+ freq = gen11_get_crystal_clock_freq(uncore, c0);
135+
136+ /*
137+ * Now figure out how the command stream's timestamp
138+ * register increments from this frequency (it might
139+ * increment only every few clock cycle).
140+ */
141+ freq >>= 3 - ((c0 & GEN10_RPM_CONFIG0_CTC_SHIFT_PARAMETER_MASK) >>
142+ GEN10_RPM_CONFIG0_CTC_SHIFT_PARAMETER_SHIFT);
143+ }
144+
145+ return freq;
37146 }
147+
148+ MISSING_CASE("Unknown gen, unable to read command streamer timestamp frequency\n");
149+ return 0;
38150 }
39151
40152 void intel_gt_init_clock_frequency(struct intel_gt *gt)
@@ -43,20 +155,27 @@ void intel_gt_init_clock_frequency(struct intel_gt *gt) · 2 unmodified lines
43155 * Note that on gen11+, the clock frequency may be reconfigured.
44156 * We do not, and we assume nobody else does.
45157 */
46− gt->clock_frequency = read_clock_frequency(gt);
158+ gt->clock_frequency = read_clock_frequency(gt->uncore);
159+ if (gt->clock_frequency)
160+ gt->clock_period_ns = intel_gt_clock_interval_to_ns(gt, 1);
161+
47162 GT_TRACE(gt,
48− "Using clock frequency: %dkHz\n",
49− gt->clock_frequency / 1000);
163+ "Using clock frequency: %dkHz, period: %dns, wrap: %lldms\n",
164+ gt->clock_frequency / 1000,
165+ gt->clock_period_ns,
166+ div_u64(mul_u32_u32(gt->clock_period_ns, S32_MAX),
167+ USEC_PER_SEC));
168+
50169 }
51170
52171 #if IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)
53172 void intel_gt_check_clock_frequency(const struct intel_gt *gt)
54173 {
55− if (gt->clock_frequency != read_clock_frequency(gt)) {
174+ if (gt->clock_frequency != read_clock_frequency(gt->uncore)) {
56175 dev_err(gt->i915->drm.dev,
57176 "GT clock frequency changed, was %uHz, now %uHz!\n",
58177 gt->clock_frequency,
59− read_clock_frequency(gt));
178+ read_clock_frequency(gt->uncore));
60179 }
61180 }
62181 #endif
@@ -66,26 +185,24 @@ static u64 div_u64_roundup(u64 nom, u32 den) · 3 unmodified lines
66185 return div_u64(nom + den - 1, den);
67186 }
68187
69−u32 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u32 count)
188+u64 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u64 count)
70189 {
71− return div_u64_roundup(mul_u32_u32(count, 1000 * 1000 * 1000),
72− gt->clock_frequency);
190+ return div_u64_roundup(count * NSEC_PER_SEC, gt->clock_frequency);
73191 }
74192
75−u32 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u32 count)
193+u64 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u64 count)
76194 {
77195 return intel_gt_clock_interval_to_ns(gt, 16 * count);
78196 }
79197
80−u32 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u32 ns)
198+u64 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u64 ns)
81199 {
82− return div_u64_roundup(mul_u32_u32(gt->clock_frequency, ns),
83− 1000 * 1000 * 1000);
200+ return div_u64_roundup(gt->clock_frequency * ns, NSEC_PER_SEC);
84201 }
85202
86−u32 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u32 ns)
203+u64 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u64 ns)
87204 {
88− u32 val;
205+ u64 val;
89206
90207 /*
91208 * Make these a multiple of magic 25 to avoid SNB (eg. Dell XPS
@@ -94,9 +211,9 @@ u32 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u32 ns) · 2 unmodified lines
94211 * EI/thresholds are "bad", leading to a very sluggish or even
95212 * frozen machine.
96213 */
97− val = DIV_ROUND_UP(intel_gt_ns_to_clock_interval(gt, ns), 16);
214+ val = div_u64_roundup(intel_gt_ns_to_clock_interval(gt, ns), 16);
98215 if (IS_GEN(gt->i915, 6))
99− val = roundup(val, 25);
216+ val = div_u64_roundup(val, 25) * 25;
100217
101218 return val;
102219 }
drivers/gpu/drm/i915/gt/intel_gt_clock_utils.h+4−4
@@ -18,10 +18,10 @@ void intel_gt_check_clock_frequency(const struct intel_gt *gt); · 17 unmodified lines
1818 static inline void intel_gt_check_clock_frequency(const struct intel_gt *gt) {}
1919 #endif
2020
21−u32 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u32 count);
22−u32 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u32 count);
21+u64 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u64 count);
22+u64 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u64 count);
2323
24−u32 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u32 ns);
25−u32 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u32 ns);
24+u64 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u64 ns);
25+u64 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u64 ns);
2626
2727 #endif /* __INTEL_GT_CLOCK_UTILS_H__ */
drivers/gpu/drm/i915/gt/intel_gt_irq.c+1−0
@@ -11,6 +11,7 @@ · 10 unmodified lines
1111 #include "intel_breadcrumbs.h"
1212 #include "intel_gt.h"
1313 #include "intel_gt_irq.h"
14+#include "intel_lrc_reg.h"
1415 #include "intel_uncore.h"
1516 #include "intel_rps.h"
1617
drivers/gpu/drm/i915/gt/intel_gt_pm.c+49−0
@@ -39,6 +39,28 @@ static void user_forcewake(struct intel_gt *gt, bool suspend) · 38 unmodified lines
3939 intel_gt_pm_put(gt);
4040 }
4141
42+static void runtime_begin(struct intel_gt *gt)
43+{
44+ local_irq_disable();
45+ write_seqcount_begin(&gt->stats.lock);
46+ gt->stats.start = ktime_get();
47+ gt->stats.active = true;
48+ write_seqcount_end(&gt->stats.lock);
49+ local_irq_enable();
50+}
51+
52+static void runtime_end(struct intel_gt *gt)
53+{
54+ local_irq_disable();
55+ write_seqcount_begin(&gt->stats.lock);
56+ gt->stats.active = false;
57+ gt->stats.total =
58+ ktime_add(gt->stats.total,
59+ ktime_sub(ktime_get(), gt->stats.start));
60+ write_seqcount_end(&gt->stats.lock);
61+ local_irq_enable();
62+}
63+
4264 static int __gt_unpark(struct intel_wakeref *wf)
4365 {
4466 struct intel_gt *gt = container_of(wf, typeof(*gt), wakeref);
@@ -67,6 +89,7 @@ static int __gt_unpark(struct intel_wakeref *wf) · 22 unmodified lines
6789 i915_pmu_gt_unparked(i915);
6890
6991 intel_gt_unpark_requests(gt);
92+ runtime_begin(gt);
7093
7194 return 0;
7295 }
@@ -79,6 +102,7 @@ static int __gt_park(struct intel_wakeref *wf) · 6 unmodified lines
79102
80103 GT_TRACE(gt, "\n");
81104
105+ runtime_end(gt);
82106 intel_gt_park_requests(gt);
83107
84108 i915_vma_parked(gt);
@@ -106,6 +130,7 @@ static const struct intel_wakeref_ops wf_ops = { · 21 unmodified lines
106130 void intel_gt_pm_init_early(struct intel_gt *gt)
107131 {
108132 intel_wakeref_init(&gt->wakeref, gt->uncore->rpm, &wf_ops);
133+ seqcount_mutex_init(&gt->stats.lock, &gt->wakeref.mutex);
109134 }
110135
111136 void intel_gt_pm_init(struct intel_gt *gt)
@@ -339,6 +364,30 @@ int intel_gt_runtime_resume(struct intel_gt *gt) · 227 unmodified lines
339364 return intel_uc_runtime_resume(&gt->uc);
340365 }
341366
367+static ktime_t __intel_gt_get_awake_time(const struct intel_gt *gt)
368+{
369+ ktime_t total = gt->stats.total;
370+
371+ if (gt->stats.active)
372+ total = ktime_add(total,
373+ ktime_sub(ktime_get(), gt->stats.start));
374+
375+ return total;
376+}
377+
378+ktime_t intel_gt_get_awake_time(const struct intel_gt *gt)
379+{
380+ unsigned int seq;
381+ ktime_t total;
382+
383+ do {
384+ seq = read_seqcount_begin(&gt->stats.lock);
385+ total = __intel_gt_get_awake_time(gt);
386+ } while (read_seqcount_retry(&gt->stats.lock, seq));
387+
388+ return total;
389+}
390+
342391 #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
343392 #include "selftest_gt_pm.c"
344393 #endif
drivers/gpu/drm/i915/gt/intel_gt_pm.h+2−0
@@ -58,6 +58,8 @@ int intel_gt_resume(struct intel_gt *gt); · 57 unmodified lines
5858 void intel_gt_runtime_suspend(struct intel_gt *gt);
5959 int intel_gt_runtime_resume(struct intel_gt *gt);
6060
61+ktime_t intel_gt_get_awake_time(const struct intel_gt *gt);
62+
6163 static inline bool is_mock_gt(const struct intel_gt *gt)
6264 {
6365 return I915_SELFTEST_ONLY(gt->awake == -ENODEV);
drivers/gpu/drm/i915/gt/intel_gt_requests.c+1−6
@@ -135,13 +135,8 @@ long intel_gt_retire_requests_timeout(struct intel_gt *gt, long timeout) · 134 unmodified lines
135135 struct intel_gt_timelines *timelines = &gt->timelines;
136136 struct intel_timeline *tl, *tn;
137137 unsigned long active_count = 0;
138− bool interruptible;
139138 LIST_HEAD(free);
140139
141− interruptible = true;
142− if (unlikely(timeout < 0))
143− timeout = -timeout, interruptible = false;
144−
145140 flush_submission(gt, timeout); /* kick the ksoftirqd tasklets */
146141 spin_lock(&timelines->lock);
147142 list_for_each_entry_safe(tl, tn, &timelines->active_list, link) {
@@ -163,7 +158,7 @@ long intel_gt_retire_requests_timeout(struct intel_gt *gt, long timeout) · 15 unmodified lines
163158 mutex_unlock(&tl->mutex);
164159
165160 timeout = dma_fence_wait_timeout(fence,
166− interruptible,
161+ true,
167162 timeout);
168163 dma_fence_put(fence);
169164
drivers/gpu/drm/i915/gt/intel_gt_types.h+25−0
@@ -75,6 +75,7 @@ struct intel_gt { · 74 unmodified lines
7575 intel_wakeref_t awake;
7676
7777 u32 clock_frequency;
78+ u32 clock_period_ns;
7879
7980 struct intel_llc llc;
8081 struct intel_rc6 rc6;
@@ -87,6 +88,30 @@ struct intel_gt { · 6 unmodified lines
8788
8889 u32 pm_guc_events;
8990
91+ struct {
92+ bool active;
93+
94+ /**
95+ * @lock: Lock protecting the below fields.
96+ */
97+ seqcount_mutex_t lock;
98+
99+ /**
100+ * @total: Total time this engine was busy.
101+ *
102+ * Accumulated time not counting the most recent block in cases
103+ * where engine is currently busy (active > 0).
104+ */
105+ ktime_t total;
106+
107+ /**
108+ * @start: Timestamp of the last idle to active transition.
109+ *
110+ * Idle is defined as active == 0, active is active > 0.
111+ */
112+ ktime_t start;
113+ } stats;
114+
90115 struct intel_engine_cs *engine[I915_NUM_ENGINES];
91116 struct intel_engine_cs *engine_class[MAX_ENGINE_CLASS + 1]
92117 [MAX_ENGINE_INSTANCE + 1];
drivers/gpu/drm/i915/gt/intel_gtt.c+29−0
@@ -422,6 +422,35 @@ void setup_private_pat(struct intel_uncore *uncore) · 421 unmodified lines
422422 bdw_setup_private_ppat(uncore);
423423 }
424424
425+struct i915_vma *
426+__vm_create_scratch_for_read(struct i915_address_space *vm, unsigned long size)
427+{
428+ struct drm_i915_gem_object *obj;
429+ struct i915_vma *vma;
430+ int err;
431+
432+ obj = i915_gem_object_create_internal(vm->i915, PAGE_ALIGN(size));
433+ if (IS_ERR(obj))
434+ return ERR_CAST(obj);
435+
436+ i915_gem_object_set_cache_coherency(obj, I915_CACHING_CACHED);
437+
438+ vma = i915_vma_instance(obj, vm, NULL);
439+ if (IS_ERR(vma)) {
440+ i915_gem_object_put(obj);
441+ return vma;
442+ }
443+
444+ err = i915_vma_pin(vma, 0, 0,
445+ i915_vma_is_ggtt(vma) ? PIN_GLOBAL : PIN_USER);
446+ if (err) {
447+ i915_vma_put(vma);
448+ return ERR_PTR(err);
449+ }
450+
451+ return vma;
452+}
453+
425454 #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
426455 #include "selftests/mock_gtt.c"
427456 #endif
drivers/gpu/drm/i915/gt/intel_gtt.h+3−0
@@ -573,6 +573,9 @@ int i915_vm_pin_pt_stash(struct i915_address_space *vm, · 572 unmodified lines
573573 void i915_vm_free_pt_stash(struct i915_address_space *vm,
574574 struct i915_vm_pt_stash *stash);
575575
576+struct i915_vma *
577+__vm_create_scratch_for_read(struct i915_address_space *vm, unsigned long size);
578+
576579 static inline struct sgt_dma {
577580 struct scatterlist *sg;
578581 dma_addr_t dma, max;
drivers/gpu/drm/i915/gt/intel_lrc.c+1149−5731
@@ -1,681 +1,216 @@
1+// SPDX-License-Identifier: MIT
12 /*
23 * Copyright © 2014 Intel Corporation
3− *
4− * Permission is hereby granted, free of charge, to any person obtaining a
5− * copy of this software and associated documentation files (the "Software"),
6− * to deal in the Software without restriction, including without limitation
7− * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8− * and/or sell copies of the Software, and to permit persons to whom the
9− * Software is furnished to do so, subject to the following conditions:
10− *
11− * The above copyright notice and this permission notice (including the next
12− * paragraph) shall be included in all copies or substantial portions of the
13− * Software.
14− *
15− * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16− * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17− * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18− * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19− * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
20− * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
21− * IN THE SOFTWARE.
22− *
23− * Authors:
24− * Ben Widawsky <ben@bwidawsk.net>
25− * Michel Thierry <michel.thierry@intel.com>
26− * Thomas Daniel <thomas.daniel@intel.com>
27− * Oscar Mateo <oscar.mateo@intel.com>
28− *
29− */
30−
31−/**
32− * DOC: Logical Rings, Logical Ring Contexts and Execlists
33− *
34− * Motivation:
35− * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts".
36− * These expanded contexts enable a number of new abilities, especially
37− * "Execlists" (also implemented in this file).
38− *
39− * One of the main differences with the legacy HW contexts is that logical
40− * ring contexts incorporate many more things to the context's state, like
41− * PDPs or ringbuffer control registers:
42− *
43− * The reason why PDPs are included in the context is straightforward: as
44− * PPGTTs (per-process GTTs) are actually per-context, having the PDPs
45− * contained there mean you don't need to do a ppgtt->switch_mm yourself,
46− * instead, the GPU will do it for you on the context switch.
47− *
48− * But, what about the ringbuffer control registers (head, tail, etc..)?
49− * shouldn't we just need a set of those per engine command streamer? This is
50− * where the name "Logical Rings" starts to make sense: by virtualizing the
51− * rings, the engine cs shifts to a new "ring buffer" with every context
52− * switch. When you want to submit a workload to the GPU you: A) choose your
53− * context, B) find its appropriate virtualized ring, C) write commands to it
54− * and then, finally, D) tell the GPU to switch to that context.
55− *
56− * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch
57− * to a contexts is via a context execution list, ergo "Execlists".
58− *
59− * LRC implementation:
60− * Regarding the creation of contexts, we have:
61− *
62− * - One global default context.
63− * - One local default context for each opened fd.
64− * - One local extra context for each context create ioctl call.
65− *
66− * Now that ringbuffers belong per-context (and not per-engine, like before)
67− * and that contexts are uniquely tied to a given engine (and not reusable,
68− * like before) we need:
69− *
70− * - One ringbuffer per-engine inside each context.
71− * - One backing object per-engine inside each context.
72− *
73− * The global default context starts its life with these new objects fully
74− * allocated and populated. The local default context for each opened fd is
75− * more complex, because we don't know at creation time which engine is going
76− * to use them. To handle this, we have implemented a deferred creation of LR
77− * contexts:
78− *
79− * The local context starts its life as a hollow or blank holder, that only
80− * gets populated for a given engine once we receive an execbuffer. If later
81− * on we receive another execbuffer ioctl for the same context but a different
82− * engine, we allocate/populate a new ringbuffer and context backing object and
83− * so on.
84− *
85− * Finally, regarding local contexts created using the ioctl call: as they are
86− * only allowed with the render ring, we can allocate & populate them right
87− * away (no need to defer anything, at least for now).
88− *
89− * Execlists implementation:
90− * Execlists are the new method by which, on gen8+ hardware, workloads are
91− * submitted for execution (as opposed to the legacy, ringbuffer-based, method).
92− * This method works as follows:
93− *
94− * When a request is committed, its commands (the BB start and any leading or
95− * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer
96− * for the appropriate context. The tail pointer in the hardware context is not
97− * updated at this time, but instead, kept by the driver in the ringbuffer
98− * structure. A structure representing this request is added to a request queue
99− * for the appropriate engine: this structure contains a copy of the context's
100− * tail after the request was written to the ring buffer and a pointer to the
101− * context itself.
102− *
103− * If the engine's request queue was empty before the request was added, the
104− * queue is processed immediately. Otherwise the queue will be processed during
105− * a context switch interrupt. In any case, elements on the queue will get sent
106− * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a
107− * globally unique 20-bits submission ID.
108− *
109− * When execution of a request completes, the GPU updates the context status
110− * buffer with a context complete event and generates a context switch interrupt.
111− * During the interrupt handling, the driver examines the events in the buffer:
112− * for each context complete event, if the announced ID matches that on the head
113− * of the request queue, then that request is retired and removed from the queue.
114− *
115− * After processing, if any requests were retired and the queue is not empty
116− * then a new execution list can be submitted. The two requests at the front of
117− * the queue are next to be submitted but since a context may not occur twice in
118− * an execution list, if subsequent requests have the same ID as the first then
119− * the two requests must be combined. This is done simply by discarding requests
120− * at the head of the queue until either only one requests is left (in which case
121− * we use a NULL second context) or the first two requests have unique IDs.
122− *
123− * By always executing the first two requests in the queue the driver ensures
124− * that the GPU is kept as busy as possible. In the case where a single context
125− * completes but a second context is still executing, the request for this second
126− * context will be at the head of the queue when we remove the first one. This
127− * request will then be resubmitted along with a new request for a different context,
128− * which will cause the hardware to continue executing the second request and queue
129− * the new request (the GPU detects the condition of a context getting preempted
130− * with the same context and optimizes the context switch flow by not doing
131− * preemption, but just sampling the new tail pointer).
132− *
1334 */
134−#include <linux/interrupt.h>
1355
6+#include "gen8_engine_cs.h"
1367 #include "i915_drv.h"
1378 #include "i915_perf.h"
138−#include "i915_trace.h"
139−#include "i915_vgpu.h"
140−#include "intel_breadcrumbs.h"
141−#include "intel_context.h"
142−#include "intel_engine_pm.h"
9+#include "intel_engine.h"
10+#include "intel_gpu_commands.h"
14311 #include "intel_gt.h"
144−#include "intel_gt_pm.h"
145−#include "intel_gt_requests.h"
12+#include "intel_lrc.h"
14613 #include "intel_lrc_reg.h"
147−#include "intel_mocs.h"
148−#include "intel_reset.h"
14914 #include "intel_ring.h"
150−#include "intel_workarounds.h"
15115 #include "shmem_utils.h"
15216
153−#define RING_EXECLIST_QFULL (1 << 0x2)
154−#define RING_EXECLIST1_VALID (1 << 0x3)
155−#define RING_EXECLIST0_VALID (1 << 0x4)
156−#define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE)
157−#define RING_EXECLIST1_ACTIVE (1 << 0x11)
158−#define RING_EXECLIST0_ACTIVE (1 << 0x12)
159−
160−#define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0)
161−#define GEN8_CTX_STATUS_PREEMPTED (1 << 1)
162−#define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2)
163−#define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3)
164−#define GEN8_CTX_STATUS_COMPLETE (1 << 4)
165−#define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15)
166−
167−#define GEN8_CTX_STATUS_COMPLETED_MASK \
168− (GEN8_CTX_STATUS_COMPLETE | GEN8_CTX_STATUS_PREEMPTED)
17+static void set_offsets(u32 *regs,
18+ const u8 *data,
19+ const struct intel_engine_cs *engine,
20+ bool close)
21+#define NOP(x) (BIT(7) | (x))
22+#define LRI(count, flags) ((flags) << 6 | (count) | BUILD_BUG_ON_ZERO(count >= BIT(6)))
23+#define POSTED BIT(0)
24+#define REG(x) (((x) >> 2) | BUILD_BUG_ON_ZERO(x >= 0x200))
25+#define REG16(x) \
26+ (((x) >> 9) | BIT(7) | BUILD_BUG_ON_ZERO(x >= 0x10000)), \
27+ (((x) >> 2) & 0x7f)
28+#define END 0
29+{
30+ const u32 base = engine->mmio_base;
16931
170−#define CTX_DESC_FORCE_RESTORE BIT_ULL(2)
32+ while (*data) {
33+ u8 count, flags;
17134
172−#define GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE (0x1) /* lower csb dword */
173−#define GEN12_CTX_SWITCH_DETAIL(csb_dw) ((csb_dw) & 0xF) /* upper csb dword */
174−#define GEN12_CSB_SW_CTX_ID_MASK GENMASK(25, 15)
175−#define GEN12_IDLE_CTX_ID 0x7FF
176−#define GEN12_CSB_CTX_VALID(csb_dw) \
177− (FIELD_GET(GEN12_CSB_SW_CTX_ID_MASK, csb_dw) != GEN12_IDLE_CTX_ID)
35+ if (*data & BIT(7)) { /* skip */
36+ count = *data++ & ~BIT(7);
37+ regs += count;
38+ continue;
39+ }
17840
179−/* Typical size of the average request (2 pipecontrols and a MI_BB) */
180−#define EXECLISTS_REQUEST_SIZE 64 /* bytes */
41+ count = *data & 0x3f;
42+ flags = *data >> 6;
43+ data++;
18144
182−struct virtual_engine {
183− struct intel_engine_cs base;
184− struct intel_context context;
185− struct rcu_work rcu;
45+ *regs = MI_LOAD_REGISTER_IMM(count);
46+ if (flags & POSTED)
47+ *regs |= MI_LRI_FORCE_POSTED;
48+ if (INTEL_GEN(engine->i915) >= 11)
49+ *regs |= MI_LRI_LRM_CS_MMIO;
50+ regs++;
18651
187− /*
188− * We allow only a single request through the virtual engine at a time
189− * (each request in the timeline waits for the completion fence of
190− * the previous before being submitted). By restricting ourselves to
191− * only submitting a single request, each request is placed on to a
192− * physical to maximise load spreading (by virtue of the late greedy
193− * scheduling -- each real engine takes the next available request
194− * upon idling).
195− */
196− struct i915_request *request;
52+ GEM_BUG_ON(!count);
53+ do {
54+ u32 offset = 0;
55+ u8 v;
19756
198− /*
199− * We keep a rbtree of available virtual engines inside each physical
200− * engine, sorted by priority. Here we preallocate the nodes we need
201− * for the virtual engine, indexed by physical_engine->id.
202− */
203− struct ve_node {
204− struct rb_node rb;
205− int prio;
206− } nodes[I915_NUM_ENGINES];
57+ do {
58+ v = *data++;
59+ offset <<= 7;
60+ offset |= v & ~BIT(7);
61+ } while (v & BIT(7));
20762
208− /*
209− * Keep track of bonded pairs -- restrictions upon on our selection
210− * of physical engines any particular request may be submitted to.
211− * If we receive a submit-fence from a master engine, we will only
212− * use one of sibling_mask physical engines.
213− */
214− struct ve_bond {
215− const struct intel_engine_cs *master;
216− intel_engine_mask_t sibling_mask;
217− } *bonds;
218− unsigned int num_bonds;
219−
220− /* And finally, which physical engines this virtual engine maps onto. */
221− unsigned int num_siblings;
222− struct intel_engine_cs *siblings[];
223−};
63+ regs[0] = base + (offset << 2);
64+ regs += 2;
65+ } while (--count);
66+ }
22467
225−static struct virtual_engine *to_virtual_engine(struct intel_engine_cs *engine)
226−{
227− GEM_BUG_ON(!intel_engine_is_virtual(engine));
228− return container_of(engine, struct virtual_engine, base);
68+ if (close) {
69+ /* Close the batch; used mainly by live_lrc_layout() */
70+ *regs = MI_BATCH_BUFFER_END;
71+ if (INTEL_GEN(engine->i915) >= 10)
72+ *regs |= BIT(0);
73+ }
22974 }
23075
231−static int __execlists_context_alloc(struct intel_context *ce,
232− struct intel_engine_cs *engine);
233−
234−static void execlists_init_reg_state(u32 *reg_state,
235− const struct intel_context *ce,
236− const struct intel_engine_cs *engine,
237− const struct intel_ring *ring,
238− bool close);
239−static void
240−__execlists_update_reg_state(const struct intel_context *ce,
241− const struct intel_engine_cs *engine,
242− u32 head);
76+static const u8 gen8_xcs_offsets[] = {
77+ NOP(1),
78+ LRI(11, 0),
79+ REG16(0x244),
80+ REG(0x034),
81+ REG(0x030),
82+ REG(0x038),
83+ REG(0x03c),
84+ REG(0x168),
85+ REG(0x140),
86+ REG(0x110),
87+ REG(0x11c),
88+ REG(0x114),
89+ REG(0x118),
24390
244−static int lrc_ring_mi_mode(const struct intel_engine_cs *engine)
245−{
246− if (INTEL_GEN(engine->i915) >= 12)
247− return 0x60;
248− else if (INTEL_GEN(engine->i915) >= 9)
249− return 0x54;
250− else if (engine->class == RENDER_CLASS)
251− return 0x58;
252− else
253− return -1;
254−}
91+ NOP(9),
92+ LRI(9, 0),
93+ REG16(0x3a8),
94+ REG16(0x28c),
95+ REG16(0x288),
96+ REG16(0x284),
97+ REG16(0x280),
98+ REG16(0x27c),
99+ REG16(0x278),
100+ REG16(0x274),
101+ REG16(0x270),
255102
256−static int lrc_ring_gpr0(const struct intel_engine_cs *engine)
257−{
258− if (INTEL_GEN(engine->i915) >= 12)
259− return 0x74;
260− else if (INTEL_GEN(engine->i915) >= 9)
261− return 0x68;
262− else if (engine->class == RENDER_CLASS)
263− return 0xd8;
264− else
265− return -1;
266−}
103+ NOP(13),
104+ LRI(2, 0),
105+ REG16(0x200),
106+ REG(0x028),
267107
268−static int lrc_ring_wa_bb_per_ctx(const struct intel_engine_cs *engine)
269−{
270− if (INTEL_GEN(engine->i915) >= 12)
271− return 0x12;
272− else if (INTEL_GEN(engine->i915) >= 9 || engine->class == RENDER_CLASS)
273− return 0x18;
274− else
275− return -1;
276−}
108+ END
109+};
277110
278−static int lrc_ring_indirect_ptr(const struct intel_engine_cs *engine)
279−{
280− int x;
111+static const u8 gen9_xcs_offsets[] = {
112+ NOP(1),
113+ LRI(14, POSTED),
114+ REG16(0x244),
115+ REG(0x034),
116+ REG(0x030),
117+ REG(0x038),
118+ REG(0x03c),
119+ REG(0x168),
120+ REG(0x140),
121+ REG(0x110),
122+ REG(0x11c),
123+ REG(0x114),
124+ REG(0x118),
125+ REG(0x1c0),
126+ REG(0x1c4),
127+ REG(0x1c8),
281128
282− x = lrc_ring_wa_bb_per_ctx(engine);
283− if (x < 0)
284− return x;
129+ NOP(3),
130+ LRI(9, POSTED),
131+ REG16(0x3a8),
132+ REG16(0x28c),
133+ REG16(0x288),
134+ REG16(0x284),
135+ REG16(0x280),
136+ REG16(0x27c),
137+ REG16(0x278),
138+ REG16(0x274),
139+ REG16(0x270),
285140
286− return x + 2;
287−}
141+ NOP(13),
142+ LRI(1, POSTED),
143+ REG16(0x200),
288144
289−static int lrc_ring_indirect_offset(const struct intel_engine_cs *engine)
290−{
291− int x;
145+ NOP(13),
146+ LRI(44, POSTED),
147+ REG(0x028),
148+ REG(0x09c),
149+ REG(0x0c0),
150+ REG(0x178),
151+ REG(0x17c),
152+ REG16(0x358),
153+ REG(0x170),
154+ REG(0x150),
155+ REG(0x154),
156+ REG(0x158),
157+ REG16(0x41c),
158+ REG16(0x600),
159+ REG16(0x604),
160+ REG16(0x608),
161+ REG16(0x60c),
162+ REG16(0x610),
163+ REG16(0x614),
164+ REG16(0x618),
165+ REG16(0x61c),
166+ REG16(0x620),
167+ REG16(0x624),
168+ REG16(0x628),
169+ REG16(0x62c),
170+ REG16(0x630),
171+ REG16(0x634),
172+ REG16(0x638),
173+ REG16(0x63c),
174+ REG16(0x640),
175+ REG16(0x644),
176+ REG16(0x648),
177+ REG16(0x64c),
178+ REG16(0x650),
179+ REG16(0x654),
180+ REG16(0x658),
181+ REG16(0x65c),
182+ REG16(0x660),
183+ REG16(0x664),
184+ REG16(0x668),
185+ REG16(0x66c),
186+ REG16(0x670),
187+ REG16(0x674),
188+ REG16(0x678),
189+ REG16(0x67c),
190+ REG(0x068),
292191
293− x = lrc_ring_indirect_ptr(engine);
294− if (x < 0)
295− return x;
192+ END
193+};
296194
297− return x + 2;
298−}
195+static const u8 gen12_xcs_offsets[] = {
196+ NOP(1),
197+ LRI(13, POSTED),
198+ REG16(0x244),
199+ REG(0x034),
200+ REG(0x030),
201+ REG(0x038),
202+ REG(0x03c),
203+ REG(0x168),
204+ REG(0x140),
205+ REG(0x110),
206+ REG(0x1c0),
207+ REG(0x1c4),
208+ REG(0x1c8),
209+ REG(0x180),
210+ REG16(0x2b4),
299211
300−static int lrc_ring_cmd_buf_cctl(const struct intel_engine_cs *engine)
301−{
302− if (engine->class != RENDER_CLASS)
303− return -1;
304−
305− if (INTEL_GEN(engine->i915) >= 12)
306− return 0xb6;
307− else if (INTEL_GEN(engine->i915) >= 11)
308− return 0xaa;
309− else
310− return -1;
311−}
312−
313−static u32
314−lrc_ring_indirect_offset_default(const struct intel_engine_cs *engine)
315−{
316− switch (INTEL_GEN(engine->i915)) {
317− default:
318− MISSING_CASE(INTEL_GEN(engine->i915));
319− fallthrough;
320− case 12:
321− return GEN12_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
322− case 11:
323− return GEN11_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
324− case 10:
325− return GEN10_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
326− case 9:
327− return GEN9_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
328− case 8:
329− return GEN8_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
330− }
331−}
332−
333−static void
334−lrc_ring_setup_indirect_ctx(u32 *regs,
335− const struct intel_engine_cs *engine,
336− u32 ctx_bb_ggtt_addr,
337− u32 size)
338−{
339− GEM_BUG_ON(!size);
340− GEM_BUG_ON(!IS_ALIGNED(size, CACHELINE_BYTES));
341− GEM_BUG_ON(lrc_ring_indirect_ptr(engine) == -1);
342− regs[lrc_ring_indirect_ptr(engine) + 1] =
343− ctx_bb_ggtt_addr | (size / CACHELINE_BYTES);
344−
345− GEM_BUG_ON(lrc_ring_indirect_offset(engine) == -1);
346− regs[lrc_ring_indirect_offset(engine) + 1] =
347− lrc_ring_indirect_offset_default(engine) << 6;
348−}
349−
350−static u32 intel_context_get_runtime(const struct intel_context *ce)
351−{
352− /*
353− * We can use either ppHWSP[16] which is recorded before the context
354− * switch (and so excludes the cost of context switches) or use the
355− * value from the context image itself, which is saved/restored earlier
356− * and so includes the cost of the save.
357− */
358− return READ_ONCE(ce->lrc_reg_state[CTX_TIMESTAMP]);
359−}
360−
361−static void mark_eio(struct i915_request *rq)
362−{
363− if (i915_request_completed(rq))
364− return;
365−
366− GEM_BUG_ON(i915_request_signaled(rq));
367−
368− i915_request_set_error_once(rq, -EIO);
369− i915_request_mark_complete(rq);
370−}
371−
372−static struct i915_request *
373−active_request(const struct intel_timeline * const tl, struct i915_request *rq)
374−{
375− struct i915_request *active = rq;
376−
377− rcu_read_lock();
378− list_for_each_entry_continue_reverse(rq, &tl->requests, link) {
379− if (i915_request_completed(rq))
380− break;
381−
382− active = rq;
383− }
384− rcu_read_unlock();
385−
386− return active;
387−}
388−
389−static inline u32 intel_hws_preempt_address(struct intel_engine_cs *engine)
390−{
391− return (i915_ggtt_offset(engine->status_page.vma) +
392− I915_GEM_HWS_PREEMPT_ADDR);
393−}
394−
395−static inline void
396−ring_set_paused(const struct intel_engine_cs *engine, int state)
397−{
398− /*
399− * We inspect HWS_PREEMPT with a semaphore inside
400− * engine->emit_fini_breadcrumb. If the dword is true,
401− * the ring is paused as the semaphore will busywait
402− * until the dword is false.
403− */
404− engine->status_page.addr[I915_GEM_HWS_PREEMPT] = state;
405− if (state)
406− wmb();
407−}
408−
409−static inline struct i915_priolist *to_priolist(struct rb_node *rb)
410−{
411− return rb_entry(rb, struct i915_priolist, node);
412−}
413−
414−static inline int rq_prio(const struct i915_request *rq)
415−{
416− return READ_ONCE(rq->sched.attr.priority);
417−}
418−
419−static int effective_prio(const struct i915_request *rq)
420−{
421− int prio = rq_prio(rq);
422−
423− /*
424− * If this request is special and must not be interrupted at any
425− * cost, so be it. Note we are only checking the most recent request
426− * in the context and so may be masking an earlier vip request. It
427− * is hoped that under the conditions where nopreempt is used, this
428− * will not matter (i.e. all requests to that context will be
429− * nopreempt for as long as desired).
430− */
431− if (i915_request_has_nopreempt(rq))
432− prio = I915_PRIORITY_UNPREEMPTABLE;
433−
434− return prio;
435−}
436−
437−static int queue_prio(const struct intel_engine_execlists *execlists)
438−{
439− struct i915_priolist *p;
440− struct rb_node *rb;
441−
442− rb = rb_first_cached(&execlists->queue);
443− if (!rb)
444− return INT_MIN;
445−
446− /*
447− * As the priolist[] are inverted, with the highest priority in [0],
448− * we have to flip the index value to become priority.
449− */
450− p = to_priolist(rb);
451− if (!I915_USER_PRIORITY_SHIFT)
452− return p->priority;
453−
454− return ((p->priority + 1) << I915_USER_PRIORITY_SHIFT) - ffs(p->used);
455−}
456−
457−static inline bool need_preempt(const struct intel_engine_cs *engine,
458− const struct i915_request *rq,
459− struct rb_node *rb)
460−{
461− int last_prio;
462−
463− if (!intel_engine_has_semaphores(engine))
464− return false;
465−
466− /*
467− * Check if the current priority hint merits a preemption attempt.
468− *
469− * We record the highest value priority we saw during rescheduling
470− * prior to this dequeue, therefore we know that if it is strictly
471− * less than the current tail of ESLP[0], we do not need to force
472− * a preempt-to-idle cycle.
473− *
474− * However, the priority hint is a mere hint that we may need to
475− * preempt. If that hint is stale or we may be trying to preempt
476− * ourselves, ignore the request.
477− *
478− * More naturally we would write
479− * prio >= max(0, last);
480− * except that we wish to prevent triggering preemption at the same
481− * priority level: the task that is running should remain running
482− * to preserve FIFO ordering of dependencies.
483− */
484− last_prio = max(effective_prio(rq), I915_PRIORITY_NORMAL - 1);
485− if (engine->execlists.queue_priority_hint <= last_prio)
486− return false;
487−
488− /*
489− * Check against the first request in ELSP[1], it will, thanks to the
490− * power of PI, be the highest priority of that context.
491− */
492− if (!list_is_last(&rq->sched.link, &engine->active.requests) &&
493− rq_prio(list_next_entry(rq, sched.link)) > last_prio)
494− return true;
495−
496− if (rb) {
497− struct virtual_engine *ve =
498− rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
499− bool preempt = false;
500−
501− if (engine == ve->siblings[0]) { /* only preempt one sibling */
502− struct i915_request *next;
503−
504− rcu_read_lock();
505− next = READ_ONCE(ve->request);
506− if (next)
507− preempt = rq_prio(next) > last_prio;
508− rcu_read_unlock();
509− }
510−
511− if (preempt)
512− return preempt;
513− }
514−
515− /*
516− * If the inflight context did not trigger the preemption, then maybe
517− * it was the set of queued requests? Pick the highest priority in
518− * the queue (the first active priolist) and see if it deserves to be
519− * running instead of ELSP[0].
520− *
521− * The highest priority request in the queue can not be either
522− * ELSP[0] or ELSP[1] as, thanks again to PI, if it was the same
523− * context, it's priority would not exceed ELSP[0] aka last_prio.
524− */
525− return queue_prio(&engine->execlists) > last_prio;
526−}
527−
528−__maybe_unused static inline bool
529−assert_priority_queue(const struct i915_request *prev,
530− const struct i915_request *next)
531−{
532− /*
533− * Without preemption, the prev may refer to the still active element
534− * which we refuse to let go.
535− *
536− * Even with preemption, there are times when we think it is better not
537− * to preempt and leave an ostensibly lower priority request in flight.
538− */
539− if (i915_request_is_active(prev))
540− return true;
541−
542− return rq_prio(prev) >= rq_prio(next);
543−}
544−
545−/*
546− * The context descriptor encodes various attributes of a context,
547− * including its GTT address and some flags. Because it's fairly
548− * expensive to calculate, we'll just do it once and cache the result,
549− * which remains valid until the context is unpinned.
550− *
551− * This is what a descriptor looks like, from LSB to MSB::
552− *
553− * bits 0-11: flags, GEN8_CTX_* (cached in ctx->desc_template)
554− * bits 12-31: LRCA, GTT address of (the HWSP of) this context
555− * bits 32-52: ctx ID, a globally unique tag (highest bit used by GuC)
556− * bits 53-54: mbz, reserved for use by hardware
557− * bits 55-63: group ID, currently unused and set to 0
558− *
559− * Starting from Gen11, the upper dword of the descriptor has a new format:
560− *
561− * bits 32-36: reserved
562− * bits 37-47: SW context ID
563− * bits 48:53: engine instance
564− * bit 54: mbz, reserved for use by hardware
565− * bits 55-60: SW counter
566− * bits 61-63: engine class
567− *
568− * engine info, SW context ID and SW counter need to form a unique number
569− * (Context ID) per lrc.
570− */
571−static u32
572−lrc_descriptor(struct intel_context *ce, struct intel_engine_cs *engine)
573−{
574− u32 desc;
575−
576− desc = INTEL_LEGACY_32B_CONTEXT;
577− if (i915_vm_is_4lvl(ce->vm))
578− desc = INTEL_LEGACY_64B_CONTEXT;
579− desc <<= GEN8_CTX_ADDRESSING_MODE_SHIFT;
580−
581− desc |= GEN8_CTX_VALID | GEN8_CTX_PRIVILEGE;
582− if (IS_GEN(engine->i915, 8))
583− desc |= GEN8_CTX_L3LLC_COHERENT;
584−
585− return i915_ggtt_offset(ce->state) | desc;
586−}
587−
588−static inline unsigned int dword_in_page(void *addr)
589−{
590− return offset_in_page(addr) / sizeof(u32);
591−}
592−
593−static void set_offsets(u32 *regs,
594− const u8 *data,
595− const struct intel_engine_cs *engine,
596− bool clear)
597−#define NOP(x) (BIT(7) | (x))
598−#define LRI(count, flags) ((flags) << 6 | (count) | BUILD_BUG_ON_ZERO(count >= BIT(6)))
599−#define POSTED BIT(0)
600−#define REG(x) (((x) >> 2) | BUILD_BUG_ON_ZERO(x >= 0x200))
601−#define REG16(x) \
602− (((x) >> 9) | BIT(7) | BUILD_BUG_ON_ZERO(x >= 0x10000)), \
603− (((x) >> 2) & 0x7f)
604−#define END(total_state_size) 0, (total_state_size)
605−{
606− const u32 base = engine->mmio_base;
607−
608− while (*data) {
609− u8 count, flags;
610−
611− if (*data & BIT(7)) { /* skip */
612− count = *data++ & ~BIT(7);
613− if (clear)
614− memset32(regs, MI_NOOP, count);
615− regs += count;
616− continue;
617− }
618−
619− count = *data & 0x3f;
620− flags = *data >> 6;
621− data++;
622−
623− *regs = MI_LOAD_REGISTER_IMM(count);
624− if (flags & POSTED)
625− *regs |= MI_LRI_FORCE_POSTED;
626− if (INTEL_GEN(engine->i915) >= 11)
627− *regs |= MI_LRI_LRM_CS_MMIO;
628− regs++;
629−
630− GEM_BUG_ON(!count);
631− do {
632− u32 offset = 0;
633− u8 v;
634−
635− do {
636− v = *data++;
637− offset <<= 7;
638− offset |= v & ~BIT(7);
639− } while (v & BIT(7));
640−
641− regs[0] = base + (offset << 2);
642− if (clear)
643− regs[1] = 0;
644− regs += 2;
645− } while (--count);
646− }
647−
648− if (clear) {
649− u8 count = *++data;
650−
651− /* Clear past the tail for HW access */
652− GEM_BUG_ON(dword_in_page(regs) > count);
653− memset32(regs, MI_NOOP, count - dword_in_page(regs));
654−
655− /* Close the batch; used mainly by live_lrc_layout() */
656− *regs = MI_BATCH_BUFFER_END;
657− if (INTEL_GEN(engine->i915) >= 10)
658− *regs |= BIT(0);
659− }
660−}
661−
662−static const u8 gen8_xcs_offsets[] = {
663− NOP(1),
664− LRI(11, 0),
665− REG16(0x244),
666− REG(0x034),
667− REG(0x030),
668− REG(0x038),
669− REG(0x03c),
670− REG(0x168),
671− REG(0x140),
672− REG(0x110),
673− REG(0x11c),
674− REG(0x114),
675− REG(0x118),
676−
677− NOP(9),
678− LRI(9, 0),
212+ NOP(5),
213+ LRI(9, POSTED),
679214 REG16(0x3a8),
680215 REG16(0x28c),
681216 REG16(0x288),
@@ -686,15 +221,10 @@ static const u8 gen8_xcs_offsets[] = { · 4 unmodified lines
686221 REG16(0x274),
687222 REG16(0x270),
688223
689− NOP(13),
690− LRI(2, 0),
691− REG16(0x200),
692− REG(0x028),
693−
694− END(80)
224+ END
695225 };
696226
697−static const u8 gen9_xcs_offsets[] = {
227+static const u8 gen8_rcs_offsets[] = {
698228 NOP(1),
699229 LRI(14, POSTED),
700230 REG16(0x244),
@@ -725,126 +255,10 @@ static const u8 gen9_xcs_offsets[] = { · 24 unmodified lines
725255 REG16(0x270),
726256
727257 NOP(13),
728− LRI(1, POSTED),
729− REG16(0x200),
258+ LRI(1, 0),
259+ REG(0x0c8),
730260
731− NOP(13),
732− LRI(44, POSTED),
733− REG(0x028),
734− REG(0x09c),
735− REG(0x0c0),
736− REG(0x178),
737− REG(0x17c),
738− REG16(0x358),
739− REG(0x170),
740− REG(0x150),
741− REG(0x154),
742− REG(0x158),
743− REG16(0x41c),
744− REG16(0x600),
745− REG16(0x604),
746− REG16(0x608),
747− REG16(0x60c),
748− REG16(0x610),
749− REG16(0x614),
750− REG16(0x618),
751− REG16(0x61c),
752− REG16(0x620),
753− REG16(0x624),
754− REG16(0x628),
755− REG16(0x62c),
756− REG16(0x630),
757− REG16(0x634),
758− REG16(0x638),
759− REG16(0x63c),
760− REG16(0x640),
761− REG16(0x644),
762− REG16(0x648),
763− REG16(0x64c),
764− REG16(0x650),
765− REG16(0x654),
766− REG16(0x658),
767− REG16(0x65c),
768− REG16(0x660),
769− REG16(0x664),
770− REG16(0x668),
771− REG16(0x66c),
772− REG16(0x670),
773− REG16(0x674),
774− REG16(0x678),
775− REG16(0x67c),
776− REG(0x068),
777−
778− END(176)
779−};
780−
781−static const u8 gen12_xcs_offsets[] = {
782− NOP(1),
783− LRI(13, POSTED),
784− REG16(0x244),
785− REG(0x034),
786− REG(0x030),
787− REG(0x038),
788− REG(0x03c),
789− REG(0x168),
790− REG(0x140),
791− REG(0x110),
792− REG(0x1c0),
793− REG(0x1c4),
794− REG(0x1c8),
795− REG(0x180),
796− REG16(0x2b4),
797−
798− NOP(5),
799− LRI(9, POSTED),
800− REG16(0x3a8),
801− REG16(0x28c),
802− REG16(0x288),
803− REG16(0x284),
804− REG16(0x280),
805− REG16(0x27c),
806− REG16(0x278),
807− REG16(0x274),
808− REG16(0x270),
809−
810− END(80)
811−};
812−
813−static const u8 gen8_rcs_offsets[] = {
814− NOP(1),
815− LRI(14, POSTED),
816− REG16(0x244),
817− REG(0x034),
818− REG(0x030),
819− REG(0x038),
820− REG(0x03c),
821− REG(0x168),
822− REG(0x140),
823− REG(0x110),
824− REG(0x11c),
825− REG(0x114),
826− REG(0x118),
827− REG(0x1c0),
828− REG(0x1c4),
829− REG(0x1c8),
830−
831− NOP(3),
832− LRI(9, POSTED),
833− REG16(0x3a8),
834− REG16(0x28c),
835− REG16(0x288),
836− REG16(0x284),
837− REG16(0x280),
838− REG16(0x27c),
839− REG16(0x278),
840− REG16(0x274),
841− REG16(0x270),
842−
843− NOP(13),
844− LRI(1, 0),
845− REG(0x0c8),
846−
847− END(80)
261+ END
848262 };
849263
850264 static const u8 gen9_rcs_offsets[] = {
@@ -897,4382 +311,323 @@ static const u8 gen9_rcs_offsets[] = { · 46 unmodified lines
897311 REG16(0x600),
898312 REG16(0x604),
899313 REG16(0x608),
900− REG16(0x60c),
901− REG16(0x610),
902− REG16(0x614),
903− REG16(0x618),
904− REG16(0x61c),
905− REG16(0x620),
906− REG16(0x624),
907− REG16(0x628),
908− REG16(0x62c),
909− REG16(0x630),
910− REG16(0x634),
911− REG16(0x638),
912− REG16(0x63c),
913− REG16(0x640),
914− REG16(0x644),
915− REG16(0x648),
916− REG16(0x64c),
917− REG16(0x650),
918− REG16(0x654),
919− REG16(0x658),
920− REG16(0x65c),
921− REG16(0x660),
922− REG16(0x664),
923− REG16(0x668),
924− REG16(0x66c),
925− REG16(0x670),
926− REG16(0x674),
927− REG16(0x678),
928− REG16(0x67c),
929− REG(0x68),
930−
931− END(176)
932−};
933−
934−static const u8 gen11_rcs_offsets[] = {
935− NOP(1),
936− LRI(15, POSTED),
937− REG16(0x244),
938− REG(0x034),
939− REG(0x030),
940− REG(0x038),
941− REG(0x03c),
942− REG(0x168),
943− REG(0x140),
944− REG(0x110),
945− REG(0x11c),
946− REG(0x114),
947− REG(0x118),
948− REG(0x1c0),
949− REG(0x1c4),
950− REG(0x1c8),
951− REG(0x180),
952−
953− NOP(1),
954− LRI(9, POSTED),
955− REG16(0x3a8),
956− REG16(0x28c),
957− REG16(0x288),
958− REG16(0x284),
959− REG16(0x280),
960− REG16(0x27c),
961− REG16(0x278),
962− REG16(0x274),
963− REG16(0x270),
964−
965− LRI(1, POSTED),
966− REG(0x1b0),
967−
968− NOP(10),
969− LRI(1, 0),
970− REG(0x0c8),
971−
972− END(80)
973−};
974−
975−static const u8 gen12_rcs_offsets[] = {
976− NOP(1),
977− LRI(13, POSTED),
978− REG16(0x244),
979− REG(0x034),
980− REG(0x030),
981− REG(0x038),
982− REG(0x03c),
983− REG(0x168),
984− REG(0x140),
985− REG(0x110),
986− REG(0x1c0),
987− REG(0x1c4),
988− REG(0x1c8),
989− REG(0x180),
990− REG16(0x2b4),
991−
992− NOP(5),
993− LRI(9, POSTED),
994− REG16(0x3a8),
995− REG16(0x28c),
996− REG16(0x288),
997− REG16(0x284),
998− REG16(0x280),
999− REG16(0x27c),
1000− REG16(0x278),
1001− REG16(0x274),
1002− REG16(0x270),
1003−
1004− LRI(3, POSTED),
1005− REG(0x1b0),
1006− REG16(0x5a8),
1007− REG16(0x5ac),
1008−
1009− NOP(6),
1010− LRI(1, 0),
1011− REG(0x0c8),
1012− NOP(3 + 9 + 1),
1013−
1014− LRI(51, POSTED),
1015− REG16(0x588),
1016− REG16(0x588),
1017− REG16(0x588),
1018− REG16(0x588),
1019− REG16(0x588),
1020− REG16(0x588),
1021− REG(0x028),
1022− REG(0x09c),
1023− REG(0x0c0),
1024− REG(0x178),
1025− REG(0x17c),
1026− REG16(0x358),
1027− REG(0x170),
1028− REG(0x150),
1029− REG(0x154),
1030− REG(0x158),
1031− REG16(0x41c),
1032− REG16(0x600),
1033− REG16(0x604),
1034− REG16(0x608),
1035− REG16(0x60c),
1036− REG16(0x610),
1037− REG16(0x614),
1038− REG16(0x618),
1039− REG16(0x61c),
1040− REG16(0x620),
1041− REG16(0x624),
1042− REG16(0x628),
1043− REG16(0x62c),
1044− REG16(0x630),
1045− REG16(0x634),
1046− REG16(0x638),
1047− REG16(0x63c),
1048− REG16(0x640),
1049− REG16(0x644),
1050− REG16(0x648),
1051− REG16(0x64c),
1052− REG16(0x650),
1053− REG16(0x654),
1054− REG16(0x658),
1055− REG16(0x65c),
1056− REG16(0x660),
1057− REG16(0x664),
1058− REG16(0x668),
1059− REG16(0x66c),
1060− REG16(0x670),
1061− REG16(0x674),
1062− REG16(0x678),
1063− REG16(0x67c),
1064− REG(0x068),
1065− REG(0x084),
1066− NOP(1),
1067−
1068− END(192)
1069−};
1070−
1071−#undef END
1072−#undef REG16
1073−#undef REG
1074−#undef LRI
1075−#undef NOP
1076−
1077−static const u8 *reg_offsets(const struct intel_engine_cs *engine)
1078−{
1079− /*
1080− * The gen12+ lists only have the registers we program in the basic
1081− * default state. We rely on the context image using relative
1082− * addressing to automatic fixup the register state between the
1083− * physical engines for virtual engine.
1084− */
1085− GEM_BUG_ON(INTEL_GEN(engine->i915) >= 12 &&
1086− !intel_engine_has_relative_mmio(engine));
1087−
1088− if (engine->class == RENDER_CLASS) {
1089− if (INTEL_GEN(engine->i915) >= 12)
1090− return gen12_rcs_offsets;
1091− else if (INTEL_GEN(engine->i915) >= 11)
1092− return gen11_rcs_offsets;
1093− else if (INTEL_GEN(engine->i915) >= 9)
1094− return gen9_rcs_offsets;
1095− else
1096− return gen8_rcs_offsets;
1097− } else {
1098− if (INTEL_GEN(engine->i915) >= 12)
1099− return gen12_xcs_offsets;
1100− else if (INTEL_GEN(engine->i915) >= 9)
1101− return gen9_xcs_offsets;
1102− else
1103− return gen8_xcs_offsets;
1104− }
1105−}
1106−
1107−static struct i915_request *
1108−__unwind_incomplete_requests(struct intel_engine_cs *engine)
1109−{
1110− struct i915_request *rq, *rn, *active = NULL;
1111− struct list_head *pl;
1112− int prio = I915_PRIORITY_INVALID;
1113−
1114− lockdep_assert_held(&engine->active.lock);
1115−
1116− list_for_each_entry_safe_reverse(rq, rn,
1117− &engine->active.requests,
1118− sched.link) {
1119− if (i915_request_completed(rq))
1120− continue; /* XXX */
1121−
1122− __i915_request_unsubmit(rq);
1123−
1124− /*
1125− * Push the request back into the queue for later resubmission.
1126− * If this request is not native to this physical engine (i.e.
1127− * it came from a virtual source), push it back onto the virtual
1128− * engine so that it can be moved across onto another physical
1129− * engine as load dictates.
1130− */
1131− if (likely(rq->execution_mask == engine->mask)) {
1132− GEM_BUG_ON(rq_prio(rq) == I915_PRIORITY_INVALID);
1133− if (rq_prio(rq) != prio) {
1134− prio = rq_prio(rq);
1135− pl = i915_sched_lookup_priolist(engine, prio);
1136− }
1137− GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
1138−
1139− list_move(&rq->sched.link, pl);
1140− set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
1141−
1142− /* Check in case we rollback so far we wrap [size/2] */
1143− if (intel_ring_direction(rq->ring,
1144− rq->tail,
1145− rq->ring->tail + 8) > 0)
1146− rq->context->lrc.desc |= CTX_DESC_FORCE_RESTORE;
1147−
1148− active = rq;
1149− } else {
1150− struct intel_engine_cs *owner = rq->context->engine;
1151−
1152− WRITE_ONCE(rq->engine, owner);
1153− owner->submit_request(rq);
1154− active = NULL;
1155− }
1156− }
1157−
1158− return active;
1159−}
1160−
1161−struct i915_request *
1162−execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists)
1163−{
1164− struct intel_engine_cs *engine =
1165− container_of(execlists, typeof(*engine), execlists);
1166−
1167− return __unwind_incomplete_requests(engine);
1168−}
1169−
1170−static inline void
1171−execlists_context_status_change(struct i915_request *rq, unsigned long status)
1172−{
1173− /*
1174− * Only used when GVT-g is enabled now. When GVT-g is disabled,
1175− * The compiler should eliminate this function as dead-code.
1176− */
1177− if (!IS_ENABLED(CONFIG_DRM_I915_GVT))
1178− return;
1179−
1180− atomic_notifier_call_chain(&rq->engine->context_status_notifier,
1181− status, rq);
1182−}
1183−
1184−static void intel_engine_context_in(struct intel_engine_cs *engine)
1185−{
1186− unsigned long flags;
1187−
1188− if (atomic_add_unless(&engine->stats.active, 1, 0))
1189− return;
1190−
1191− write_seqlock_irqsave(&engine->stats.lock, flags);
1192− if (!atomic_add_unless(&engine->stats.active, 1, 0)) {
1193− engine->stats.start = ktime_get();
1194− atomic_inc(&engine->stats.active);
1195− }
1196− write_sequnlock_irqrestore(&engine->stats.lock, flags);
1197−}
1198−
1199−static void intel_engine_context_out(struct intel_engine_cs *engine)
1200−{
1201− unsigned long flags;
1202−
1203− GEM_BUG_ON(!atomic_read(&engine->stats.active));
1204−
1205− if (atomic_add_unless(&engine->stats.active, -1, 1))
1206− return;
1207−
1208− write_seqlock_irqsave(&engine->stats.lock, flags);
1209− if (atomic_dec_and_test(&engine->stats.active)) {
1210− engine->stats.total =
1211− ktime_add(engine->stats.total,
1212− ktime_sub(ktime_get(), engine->stats.start));
1213− }
1214− write_sequnlock_irqrestore(&engine->stats.lock, flags);
1215−}
1216−
1217−static void
1218−execlists_check_context(const struct intel_context *ce,
1219− const struct intel_engine_cs *engine,
1220− const char *when)
1221−{
1222− const struct intel_ring *ring = ce->ring;
1223− u32 *regs = ce->lrc_reg_state;
1224− bool valid = true;
1225− int x;
1226−
1227− if (regs[CTX_RING_START] != i915_ggtt_offset(ring->vma)) {
1228− pr_err("%s: context submitted with incorrect RING_START [%08x], expected %08x\n",
1229− engine->name,
1230− regs[CTX_RING_START],
1231− i915_ggtt_offset(ring->vma));
1232− regs[CTX_RING_START] = i915_ggtt_offset(ring->vma);
1233− valid = false;
1234− }
1235−
1236− if ((regs[CTX_RING_CTL] & ~(RING_WAIT | RING_WAIT_SEMAPHORE)) !=
1237− (RING_CTL_SIZE(ring->size) | RING_VALID)) {
1238− pr_err("%s: context submitted with incorrect RING_CTL [%08x], expected %08x\n",
1239− engine->name,
1240− regs[CTX_RING_CTL],
1241− (u32)(RING_CTL_SIZE(ring->size) | RING_VALID));
1242− regs[CTX_RING_CTL] = RING_CTL_SIZE(ring->size) | RING_VALID;
1243− valid = false;
1244− }
1245−
1246− x = lrc_ring_mi_mode(engine);
1247− if (x != -1 && regs[x + 1] & (regs[x + 1] >> 16) & STOP_RING) {
1248− pr_err("%s: context submitted with STOP_RING [%08x] in RING_MI_MODE\n",
1249− engine->name, regs[x + 1]);
1250− regs[x + 1] &= ~STOP_RING;
1251− regs[x + 1] |= STOP_RING << 16;
1252− valid = false;
1253− }
1254−
1255− WARN_ONCE(!valid, "Invalid lrc state found %s submission\n", when);
1256−}
1257−
1258−static void restore_default_state(struct intel_context *ce,
1259− struct intel_engine_cs *engine)
1260−{
1261− u32 *regs;
1262−
1263− regs = memset(ce->lrc_reg_state, 0, engine->context_size - PAGE_SIZE);
1264− execlists_init_reg_state(regs, ce, engine, ce->ring, true);
1265−
1266− ce->runtime.last = intel_context_get_runtime(ce);
1267−}
1268−
1269−static void reset_active(struct i915_request *rq,
1270− struct intel_engine_cs *engine)
1271−{
1272− struct intel_context * const ce = rq->context;
1273− u32 head;
1274−
1275− /*
1276− * The executing context has been cancelled. We want to prevent
1277− * further execution along this context and propagate the error on
1278− * to anything depending on its results.
1279− *
1280− * In __i915_request_submit(), we apply the -EIO and remove the
1281− * requests' payloads for any banned requests. But first, we must
1282− * rewind the context back to the start of the incomplete request so
1283− * that we do not jump back into the middle of the batch.
1284− *
1285− * We preserve the breadcrumbs and semaphores of the incomplete
1286− * requests so that inter-timeline dependencies (i.e other timelines)
1287− * remain correctly ordered. And we defer to __i915_request_submit()
1288− * so that all asynchronous waits are correctly handled.
1289− */
1290− ENGINE_TRACE(engine, "{ rq=%llx:%lld }\n",
1291− rq->fence.context, rq->fence.seqno);
1292−
1293− /* On resubmission of the active request, payload will be scrubbed */
1294− if (i915_request_completed(rq))
1295− head = rq->tail;
1296− else
1297− head = active_request(ce->timeline, rq)->head;
1298− head = intel_ring_wrap(ce->ring, head);
1299−
1300− /* Scrub the context image to prevent replaying the previous batch */
1301− restore_default_state(ce, engine);
1302− __execlists_update_reg_state(ce, engine, head);
1303−
1304− /* We've switched away, so this should be a no-op, but intent matters */
1305− ce->lrc.desc |= CTX_DESC_FORCE_RESTORE;
1306−}
1307−
1308−static void st_update_runtime_underflow(struct intel_context *ce, s32 dt)
1309−{
1310−#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
1311− ce->runtime.num_underflow += dt < 0;
1312− ce->runtime.max_underflow = max_t(u32, ce->runtime.max_underflow, -dt);
1313−#endif
1314−}
1315−
1316−static void intel_context_update_runtime(struct intel_context *ce)
1317−{
1318− u32 old;
1319− s32 dt;
1320−
1321− if (intel_context_is_barrier(ce))
1322− return;
1323−
1324− old = ce->runtime.last;
1325− ce->runtime.last = intel_context_get_runtime(ce);
1326− dt = ce->runtime.last - old;
1327−
1328− if (unlikely(dt <= 0)) {
1329− CE_TRACE(ce, "runtime underflow: last=%u, new=%u, delta=%d\n",
1330− old, ce->runtime.last, dt);
1331− st_update_runtime_underflow(ce, dt);
1332− return;
1333− }
1334−
1335− ewma_runtime_add(&ce->runtime.avg, dt);
1336− ce->runtime.total += dt;
1337−}
1338−
1339−static inline struct intel_engine_cs *
1340−__execlists_schedule_in(struct i915_request *rq)
1341−{
1342− struct intel_engine_cs * const engine = rq->engine;
1343− struct intel_context * const ce = rq->context;
1344−
1345− intel_context_get(ce);
1346−
1347− if (unlikely(intel_context_is_banned(ce)))
1348− reset_active(rq, engine);
1349−
1350− if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
1351− execlists_check_context(ce, engine, "before");
1352−
1353− if (ce->tag) {
1354− /* Use a fixed tag for OA and friends */
1355− GEM_BUG_ON(ce->tag <= BITS_PER_LONG);
1356− ce->lrc.ccid = ce->tag;
1357− } else {
1358− /* We don't need a strict matching tag, just different values */
1359− unsigned int tag = ffs(READ_ONCE(engine->context_tag));
1360−
1361− GEM_BUG_ON(tag == 0 || tag >= BITS_PER_LONG);
1362− clear_bit(tag - 1, &engine->context_tag);
1363− ce->lrc.ccid = tag << (GEN11_SW_CTX_ID_SHIFT - 32);
1364−
1365− BUILD_BUG_ON(BITS_PER_LONG > GEN12_MAX_CONTEXT_HW_ID);
1366− }
1367−
1368− ce->lrc.ccid |= engine->execlists.ccid;
1369−
1370− __intel_gt_pm_get(engine->gt);
1371− if (engine->fw_domain && !atomic_fetch_inc(&engine->fw_active))
1372− intel_uncore_forcewake_get(engine->uncore, engine->fw_domain);
1373− execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_IN);
1374− intel_engine_context_in(engine);
1375−
1376− return engine;
1377−}
1378−
1379−static inline struct i915_request *
1380−execlists_schedule_in(struct i915_request *rq, int idx)
1381−{
1382− struct intel_context * const ce = rq->context;
1383− struct intel_engine_cs *old;
1384−
1385− GEM_BUG_ON(!intel_engine_pm_is_awake(rq->engine));
1386− trace_i915_request_in(rq, idx);
1387−
1388− old = READ_ONCE(ce->inflight);
1389− do {
1390− if (!old) {
1391− WRITE_ONCE(ce->inflight, __execlists_schedule_in(rq));
1392− break;
1393− }
1394− } while (!try_cmpxchg(&ce->inflight, &old, ptr_inc(old)));
1395−
1396− GEM_BUG_ON(intel_context_inflight(ce) != rq->engine);
1397− return i915_request_get(rq);
1398−}
1399−
1400−static void kick_siblings(struct i915_request *rq, struct intel_context *ce)
1401−{
1402− struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
1403− struct i915_request *next = READ_ONCE(ve->request);
1404−
1405− if (next == rq || (next && next->execution_mask & ~rq->execution_mask))
1406− tasklet_hi_schedule(&ve->base.execlists.tasklet);
1407−}
1408−
1409−static inline void
1410−__execlists_schedule_out(struct i915_request *rq,
1411− struct intel_engine_cs * const engine,
1412− unsigned int ccid)
1413−{
1414− struct intel_context * const ce = rq->context;
1415−
1416− /*
1417− * NB process_csb() is not under the engine->active.lock and hence
1418− * schedule_out can race with schedule_in meaning that we should
1419− * refrain from doing non-trivial work here.
1420− */
1421−
1422− if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
1423− execlists_check_context(ce, engine, "after");
1424−
1425− /*
1426− * If we have just completed this context, the engine may now be
1427− * idle and we want to re-enter powersaving.
1428− */
1429− if (list_is_last_rcu(&rq->link, &ce->timeline->requests) &&
1430− i915_request_completed(rq))
1431− intel_engine_add_retire(engine, ce->timeline);
1432−
1433− ccid >>= GEN11_SW_CTX_ID_SHIFT - 32;
1434− ccid &= GEN12_MAX_CONTEXT_HW_ID;
1435− if (ccid < BITS_PER_LONG) {
1436− GEM_BUG_ON(ccid == 0);
1437− GEM_BUG_ON(test_bit(ccid - 1, &engine->context_tag));
1438− set_bit(ccid - 1, &engine->context_tag);
1439− }
1440−
1441− intel_context_update_runtime(ce);
1442− intel_engine_context_out(engine);
1443− execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_OUT);
1444− if (engine->fw_domain && !atomic_dec_return(&engine->fw_active))
1445− intel_uncore_forcewake_put(engine->uncore, engine->fw_domain);
1446− intel_gt_pm_put_async(engine->gt);
1447−
1448− /*
1449− * If this is part of a virtual engine, its next request may
1450− * have been blocked waiting for access to the active context.
1451− * We have to kick all the siblings again in case we need to
1452− * switch (e.g. the next request is not runnable on this
1453− * engine). Hopefully, we will already have submitted the next
1454− * request before the tasklet runs and do not need to rebuild
1455− * each virtual tree and kick everyone again.
1456− */
1457− if (ce->engine != engine)
1458− kick_siblings(rq, ce);
1459−
1460− intel_context_put(ce);
1461−}
1462−
1463−static inline void
1464−execlists_schedule_out(struct i915_request *rq)
1465−{
1466− struct intel_context * const ce = rq->context;
1467− struct intel_engine_cs *cur, *old;
1468− u32 ccid;
1469−
1470− trace_i915_request_out(rq);
1471−
1472− ccid = rq->context->lrc.ccid;
1473− old = READ_ONCE(ce->inflight);
1474− do
1475− cur = ptr_unmask_bits(old, 2) ? ptr_dec(old) : NULL;
1476− while (!try_cmpxchg(&ce->inflight, &old, cur));
1477− if (!cur)
1478− __execlists_schedule_out(rq, old, ccid);
1479−
1480− i915_request_put(rq);
1481−}
1482−
1483−static u64 execlists_update_context(struct i915_request *rq)
1484−{
1485− struct intel_context *ce = rq->context;
1486− u64 desc = ce->lrc.desc;
1487− u32 tail, prev;
1488−
1489− /*
1490− * WaIdleLiteRestore:bdw,skl
1491− *
1492− * We should never submit the context with the same RING_TAIL twice
1493− * just in case we submit an empty ring, which confuses the HW.
1494− *
1495− * We append a couple of NOOPs (gen8_emit_wa_tail) after the end of
1496− * the normal request to be able to always advance the RING_TAIL on
1497− * subsequent resubmissions (for lite restore). Should that fail us,
1498− * and we try and submit the same tail again, force the context
1499− * reload.
1500− *
1501− * If we need to return to a preempted context, we need to skip the
1502− * lite-restore and force it to reload the RING_TAIL. Otherwise, the
1503− * HW has a tendency to ignore us rewinding the TAIL to the end of
1504− * an earlier request.
1505− */
1506− GEM_BUG_ON(ce->lrc_reg_state[CTX_RING_TAIL] != rq->ring->tail);
1507− prev = rq->ring->tail;
1508− tail = intel_ring_set_tail(rq->ring, rq->tail);
1509− if (unlikely(intel_ring_direction(rq->ring, tail, prev) <= 0))
1510− desc |= CTX_DESC_FORCE_RESTORE;
1511− ce->lrc_reg_state[CTX_RING_TAIL] = tail;
1512− rq->tail = rq->wa_tail;
1513−
1514− /*
1515− * Make sure the context image is complete before we submit it to HW.
1516− *
1517− * Ostensibly, writes (including the WCB) should be flushed prior to
1518− * an uncached write such as our mmio register access, the empirical
1519− * evidence (esp. on Braswell) suggests that the WC write into memory
1520− * may not be visible to the HW prior to the completion of the UC
1521− * register write and that we may begin execution from the context
1522− * before its image is complete leading to invalid PD chasing.
1523− */
1524− wmb();
1525−
1526− ce->lrc.desc &= ~CTX_DESC_FORCE_RESTORE;
1527− return desc;
1528−}
1529−
1530−static inline void write_desc(struct intel_engine_execlists *execlists, u64 desc, u32 port)
1531−{
1532− if (execlists->ctrl_reg) {
1533− writel(lower_32_bits(desc), execlists->submit_reg + port * 2);
1534− writel(upper_32_bits(desc), execlists->submit_reg + port * 2 + 1);
1535− } else {
1536− writel(upper_32_bits(desc), execlists->submit_reg);
1537− writel(lower_32_bits(desc), execlists->submit_reg);
1538− }
1539−}
1540−
1541−static __maybe_unused char *
1542−dump_port(char *buf, int buflen, const char *prefix, struct i915_request *rq)
1543−{
1544− if (!rq)
1545− return "";
1546−
1547− snprintf(buf, buflen, "%sccid:%x %llx:%lld%s prio %d",
1548− prefix,
1549− rq->context->lrc.ccid,
1550− rq->fence.context, rq->fence.seqno,
1551− i915_request_completed(rq) ? "!" :
1552− i915_request_started(rq) ? "*" :
1553− "",
1554− rq_prio(rq));
1555−
1556− return buf;
1557−}
1558−
1559−static __maybe_unused void
1560−trace_ports(const struct intel_engine_execlists *execlists,
1561− const char *msg,
1562− struct i915_request * const *ports)
1563−{
1564− const struct intel_engine_cs *engine =
1565− container_of(execlists, typeof(*engine), execlists);
1566− char __maybe_unused p0[40], p1[40];
1567−
1568− if (!ports[0])
1569− return;
1570−
1571− ENGINE_TRACE(engine, "%s { %s%s }\n", msg,
1572− dump_port(p0, sizeof(p0), "", ports[0]),
1573− dump_port(p1, sizeof(p1), ", ", ports[1]));
1574−}
1575−
1576−static inline bool
1577−reset_in_progress(const struct intel_engine_execlists *execlists)
1578−{
1579− return unlikely(!__tasklet_is_enabled(&execlists->tasklet));
1580−}
1581−
1582−static __maybe_unused bool
1583−assert_pending_valid(const struct intel_engine_execlists *execlists,
1584− const char *msg)
1585−{
1586− struct intel_engine_cs *engine =
1587− container_of(execlists, typeof(*engine), execlists);
1588− struct i915_request * const *port, *rq;
1589− struct intel_context *ce = NULL;
1590− bool sentinel = false;
1591− u32 ccid = -1;
1592−
1593− trace_ports(execlists, msg, execlists->pending);
1594−
1595− /* We may be messing around with the lists during reset, lalala */
1596− if (reset_in_progress(execlists))
1597− return true;
1598−
1599− if (!execlists->pending[0]) {
1600− GEM_TRACE_ERR("%s: Nothing pending for promotion!\n",
1601− engine->name);
1602− return false;
1603− }
1604−
1605− if (execlists->pending[execlists_num_ports(execlists)]) {
1606− GEM_TRACE_ERR("%s: Excess pending[%d] for promotion!\n",
1607− engine->name, execlists_num_ports(execlists));
1608− return false;
1609− }
1610−
1611− for (port = execlists->pending; (rq = *port); port++) {
1612− unsigned long flags;
1613− bool ok = true;
1614−
1615− GEM_BUG_ON(!kref_read(&rq->fence.refcount));
1616− GEM_BUG_ON(!i915_request_is_active(rq));
1617−
1618− if (ce == rq->context) {
1619− GEM_TRACE_ERR("%s: Dup context:%llx in pending[%zd]\n",
1620− engine->name,
1621− ce->timeline->fence_context,
1622− port - execlists->pending);
1623− return false;
1624− }
1625− ce = rq->context;
1626−
1627− if (ccid == ce->lrc.ccid) {
1628− GEM_TRACE_ERR("%s: Dup ccid:%x context:%llx in pending[%zd]\n",
1629− engine->name,
1630− ccid, ce->timeline->fence_context,
1631− port - execlists->pending);
1632− return false;
1633− }
1634− ccid = ce->lrc.ccid;
1635−
1636− /*
1637− * Sentinels are supposed to be the last request so they flush
1638− * the current execution off the HW. Check that they are the only
1639− * request in the pending submission.
1640− */
1641− if (sentinel) {
1642− GEM_TRACE_ERR("%s: context:%llx after sentinel in pending[%zd]\n",
1643− engine->name,
1644− ce->timeline->fence_context,
1645− port - execlists->pending);
1646− return false;
1647− }
1648− sentinel = i915_request_has_sentinel(rq);
1649−
1650− /* Hold tightly onto the lock to prevent concurrent retires! */
1651− if (!spin_trylock_irqsave(&rq->lock, flags))
1652− continue;
1653−
1654− if (i915_request_completed(rq))
1655− goto unlock;
1656−
1657− if (i915_active_is_idle(&ce->active) &&
1658− !intel_context_is_barrier(ce)) {
1659− GEM_TRACE_ERR("%s: Inactive context:%llx in pending[%zd]\n",
1660− engine->name,
1661− ce->timeline->fence_context,
1662− port - execlists->pending);
1663− ok = false;
1664− goto unlock;
1665− }
1666−
1667− if (!i915_vma_is_pinned(ce->state)) {
1668− GEM_TRACE_ERR("%s: Unpinned context:%llx in pending[%zd]\n",
1669− engine->name,
1670− ce->timeline->fence_context,
1671− port - execlists->pending);
1672− ok = false;
1673− goto unlock;
1674− }
1675−
1676− if (!i915_vma_is_pinned(ce->ring->vma)) {
1677− GEM_TRACE_ERR("%s: Unpinned ring:%llx in pending[%zd]\n",
1678− engine->name,
1679− ce->timeline->fence_context,
1680− port - execlists->pending);
1681− ok = false;
1682− goto unlock;
1683− }
1684−
1685−unlock:
1686− spin_unlock_irqrestore(&rq->lock, flags);
1687− if (!ok)
1688− return false;
1689− }
1690−
1691− return ce;
1692−}
1693−
1694−static void execlists_submit_ports(struct intel_engine_cs *engine)
1695−{
1696− struct intel_engine_execlists *execlists = &engine->execlists;
1697− unsigned int n;
1698−
1699− GEM_BUG_ON(!assert_pending_valid(execlists, "submit"));
1700−
1701− /*
1702− * We can skip acquiring intel_runtime_pm_get() here as it was taken
1703− * on our behalf by the request (see i915_gem_mark_busy()) and it will
1704− * not be relinquished until the device is idle (see
1705− * i915_gem_idle_work_handler()). As a precaution, we make sure
1706− * that all ELSP are drained i.e. we have processed the CSB,
1707− * before allowing ourselves to idle and calling intel_runtime_pm_put().
1708− */
1709− GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
1710−
1711− /*
1712− * ELSQ note: the submit queue is not cleared after being submitted
1713− * to the HW so we need to make sure we always clean it up. This is
1714− * currently ensured by the fact that we always write the same number
1715− * of elsq entries, keep this in mind before changing the loop below.
1716− */
1717− for (n = execlists_num_ports(execlists); n--; ) {
1718− struct i915_request *rq = execlists->pending[n];
1719−
1720− write_desc(execlists,
1721− rq ? execlists_update_context(rq) : 0,
1722− n);
1723− }
1724−
1725− /* we need to manually load the submit queue */
1726− if (execlists->ctrl_reg)
1727− writel(EL_CTRL_LOAD, execlists->ctrl_reg);
1728−}
1729−
1730−static bool ctx_single_port_submission(const struct intel_context *ce)
1731−{
1732− return (IS_ENABLED(CONFIG_DRM_I915_GVT) &&
1733− intel_context_force_single_submission(ce));
1734−}
1735−
1736−static bool can_merge_ctx(const struct intel_context *prev,
1737− const struct intel_context *next)
1738−{
1739− if (prev != next)
1740− return false;
1741−
1742− if (ctx_single_port_submission(prev))
1743− return false;
1744−
1745− return true;
1746−}
1747−
1748−static unsigned long i915_request_flags(const struct i915_request *rq)
1749−{
1750− return READ_ONCE(rq->fence.flags);
1751−}
1752−
1753−static bool can_merge_rq(const struct i915_request *prev,
1754− const struct i915_request *next)
1755−{
1756− GEM_BUG_ON(prev == next);
1757− GEM_BUG_ON(!assert_priority_queue(prev, next));
1758−
1759− /*
1760− * We do not submit known completed requests. Therefore if the next
1761− * request is already completed, we can pretend to merge it in
1762− * with the previous context (and we will skip updating the ELSP
1763− * and tracking). Thus hopefully keeping the ELSP full with active
1764− * contexts, despite the best efforts of preempt-to-busy to confuse
1765− * us.
1766− */
1767− if (i915_request_completed(next))
1768− return true;
1769−
1770− if (unlikely((i915_request_flags(prev) ^ i915_request_flags(next)) &
1771− (BIT(I915_FENCE_FLAG_NOPREEMPT) |
1772− BIT(I915_FENCE_FLAG_SENTINEL))))
1773− return false;
1774−
1775− if (!can_merge_ctx(prev->context, next->context))
1776− return false;
1777−
1778− GEM_BUG_ON(i915_seqno_passed(prev->fence.seqno, next->fence.seqno));
1779− return true;
1780−}
1781−
1782−static void virtual_update_register_offsets(u32 *regs,
1783− struct intel_engine_cs *engine)
1784−{
1785− set_offsets(regs, reg_offsets(engine), engine, false);
1786−}
1787−
1788−static bool virtual_matches(const struct virtual_engine *ve,
1789− const struct i915_request *rq,
1790− const struct intel_engine_cs *engine)
1791−{
1792− const struct intel_engine_cs *inflight;
1793−
1794− if (!(rq->execution_mask & engine->mask)) /* We peeked too soon! */
1795− return false;
1796−
1797− /*
1798− * We track when the HW has completed saving the context image
1799− * (i.e. when we have seen the final CS event switching out of
1800− * the context) and must not overwrite the context image before
1801− * then. This restricts us to only using the active engine
1802− * while the previous virtualized request is inflight (so
1803− * we reuse the register offsets). This is a very small
1804− * hystersis on the greedy seelction algorithm.
1805− */
1806− inflight = intel_context_inflight(&ve->context);
1807− if (inflight && inflight != engine)
1808− return false;
1809−
1810− return true;
1811−}
1812−
1813−static void virtual_xfer_context(struct virtual_engine *ve,
1814− struct intel_engine_cs *engine)
1815−{
1816− unsigned int n;
1817−
1818− if (likely(engine == ve->siblings[0]))
1819− return;
1820−
1821− GEM_BUG_ON(READ_ONCE(ve->context.inflight));
1822− if (!intel_engine_has_relative_mmio(engine))
1823− virtual_update_register_offsets(ve->context.lrc_reg_state,
1824− engine);
1825−
1826− /*
1827− * Move the bound engine to the top of the list for
1828− * future execution. We then kick this tasklet first
1829− * before checking others, so that we preferentially
1830− * reuse this set of bound registers.
1831− */
1832− for (n = 1; n < ve->num_siblings; n++) {
1833− if (ve->siblings[n] == engine) {
1834− swap(ve->siblings[n], ve->siblings[0]);
1835− break;
1836− }
1837− }
1838−}
1839−
1840−#define for_each_waiter(p__, rq__) \
1841− list_for_each_entry_lockless(p__, \
1842− &(rq__)->sched.waiters_list, \
1843− wait_link)
1844−
1845−#define for_each_signaler(p__, rq__) \
1846− list_for_each_entry_rcu(p__, \
1847− &(rq__)->sched.signalers_list, \
1848− signal_link)
1849−
1850−static void defer_request(struct i915_request *rq, struct list_head * const pl)
1851−{
1852− LIST_HEAD(list);
1853−
1854− /*
1855− * We want to move the interrupted request to the back of
1856− * the round-robin list (i.e. its priority level), but
1857− * in doing so, we must then move all requests that were in
1858− * flight and were waiting for the interrupted request to
1859− * be run after it again.
1860− */
1861− do {
1862− struct i915_dependency *p;
1863−
1864− GEM_BUG_ON(i915_request_is_active(rq));
1865− list_move_tail(&rq->sched.link, pl);
1866−
1867− for_each_waiter(p, rq) {
1868− struct i915_request *w =
1869− container_of(p->waiter, typeof(*w), sched);
1870−
1871− if (p->flags & I915_DEPENDENCY_WEAK)
1872− continue;
1873−
1874− /* Leave semaphores spinning on the other engines */
1875− if (w->engine != rq->engine)
1876− continue;
1877−
1878− /* No waiter should start before its signaler */
1879− GEM_BUG_ON(i915_request_has_initial_breadcrumb(w) &&
1880− i915_request_started(w) &&
1881− !i915_request_completed(rq));
1882−
1883− GEM_BUG_ON(i915_request_is_active(w));
1884− if (!i915_request_is_ready(w))
1885− continue;
1886−
1887− if (rq_prio(w) < rq_prio(rq))
1888− continue;
1889−
1890− GEM_BUG_ON(rq_prio(w) > rq_prio(rq));
1891− list_move_tail(&w->sched.link, &list);
1892− }
1893−
1894− rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
1895− } while (rq);
1896−}
1897−
1898−static void defer_active(struct intel_engine_cs *engine)
1899−{
1900− struct i915_request *rq;
1901−
1902− rq = __unwind_incomplete_requests(engine);
1903− if (!rq)
1904− return;
1905−
1906− defer_request(rq, i915_sched_lookup_priolist(engine, rq_prio(rq)));
1907−}
1908−
1909−static bool
1910−need_timeslice(const struct intel_engine_cs *engine,
1911− const struct i915_request *rq,
1912− const struct rb_node *rb)
1913−{
1914− int hint;
1915−
1916− if (!intel_engine_has_timeslices(engine))
1917− return false;
1918−
1919− hint = engine->execlists.queue_priority_hint;
1920−
1921− if (rb) {
1922− const struct virtual_engine *ve =
1923− rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
1924− const struct intel_engine_cs *inflight =
1925− intel_context_inflight(&ve->context);
1926−
1927− if (!inflight || inflight == engine) {
1928− struct i915_request *next;
1929−
1930− rcu_read_lock();
1931− next = READ_ONCE(ve->request);
1932− if (next)
1933− hint = max(hint, rq_prio(next));
1934− rcu_read_unlock();
1935− }
1936− }
1937−
1938− if (!list_is_last(&rq->sched.link, &engine->active.requests))
1939− hint = max(hint, rq_prio(list_next_entry(rq, sched.link)));
1940−
1941− GEM_BUG_ON(hint >= I915_PRIORITY_UNPREEMPTABLE);
1942− return hint >= effective_prio(rq);
1943−}
1944−
1945−static bool
1946−timeslice_yield(const struct intel_engine_execlists *el,
1947− const struct i915_request *rq)
1948−{
1949− /*
1950− * Once bitten, forever smitten!
1951− *
1952− * If the active context ever busy-waited on a semaphore,
1953− * it will be treated as a hog until the end of its timeslice (i.e.
1954− * until it is scheduled out and replaced by a new submission,
1955− * possibly even its own lite-restore). The HW only sends an interrupt
1956− * on the first miss, and we do know if that semaphore has been
1957− * signaled, or even if it is now stuck on another semaphore. Play
1958− * safe, yield if it might be stuck -- it will be given a fresh
1959− * timeslice in the near future.
1960− */
1961− return rq->context->lrc.ccid == READ_ONCE(el->yield);
1962−}
1963−
1964−static bool
1965−timeslice_expired(const struct intel_engine_execlists *el,
1966− const struct i915_request *rq)
1967−{
1968− return timer_expired(&el->timer) || timeslice_yield(el, rq);
1969−}
1970−
1971−static int
1972−switch_prio(struct intel_engine_cs *engine, const struct i915_request *rq)
1973−{
1974− if (list_is_last(&rq->sched.link, &engine->active.requests))
1975− return engine->execlists.queue_priority_hint;
1976−
1977− return rq_prio(list_next_entry(rq, sched.link));
1978−}
1979−
1980−static inline unsigned long
1981−timeslice(const struct intel_engine_cs *engine)
1982−{
1983− return READ_ONCE(engine->props.timeslice_duration_ms);
1984−}
1985−
1986−static unsigned long active_timeslice(const struct intel_engine_cs *engine)
1987−{
1988− const struct intel_engine_execlists *execlists = &engine->execlists;
1989− const struct i915_request *rq = *execlists->active;
1990−
1991− if (!rq || i915_request_completed(rq))
1992− return 0;
1993−
1994− if (READ_ONCE(execlists->switch_priority_hint) < effective_prio(rq))
1995− return 0;
1996−
1997− return timeslice(engine);
1998−}
1999−
2000−static void set_timeslice(struct intel_engine_cs *engine)
2001−{
2002− unsigned long duration;
2003−
2004− if (!intel_engine_has_timeslices(engine))
2005− return;
2006−
2007− duration = active_timeslice(engine);
2008− ENGINE_TRACE(engine, "bump timeslicing, interval:%lu", duration);
2009−
2010− set_timer_ms(&engine->execlists.timer, duration);
2011−}
2012−
2013−static void start_timeslice(struct intel_engine_cs *engine, int prio)
2014−{
2015− struct intel_engine_execlists *execlists = &engine->execlists;
2016− unsigned long duration;
2017−
2018− if (!intel_engine_has_timeslices(engine))
2019− return;
2020−
2021− WRITE_ONCE(execlists->switch_priority_hint, prio);
2022− if (prio == INT_MIN)
2023− return;
2024−
2025− if (timer_pending(&execlists->timer))
2026− return;
2027−
2028− duration = timeslice(engine);
2029− ENGINE_TRACE(engine,
2030− "start timeslicing, prio:%d, interval:%lu",
2031− prio, duration);
2032−
2033− set_timer_ms(&execlists->timer, duration);
2034−}
2035−
2036−static void record_preemption(struct intel_engine_execlists *execlists)
2037−{
2038− (void)I915_SELFTEST_ONLY(execlists->preempt_hang.count++);
2039−}
2040−
2041−static unsigned long active_preempt_timeout(struct intel_engine_cs *engine,
2042− const struct i915_request *rq)
2043−{
2044− if (!rq)
2045− return 0;
2046−
2047− /* Force a fast reset for terminated contexts (ignoring sysfs!) */
2048− if (unlikely(intel_context_is_banned(rq->context)))
2049− return 1;
2050−
2051− return READ_ONCE(engine->props.preempt_timeout_ms);
2052−}
2053−
2054−static void set_preempt_timeout(struct intel_engine_cs *engine,
2055− const struct i915_request *rq)
2056−{
2057− if (!intel_engine_has_preempt_reset(engine))
2058− return;
2059−
2060− set_timer_ms(&engine->execlists.preempt,
2061− active_preempt_timeout(engine, rq));
2062−}
2063−
2064−static inline void clear_ports(struct i915_request **ports, int count)
2065−{
2066− memset_p((void **)ports, NULL, count);
2067−}
2068−
2069−static inline void
2070−copy_ports(struct i915_request **dst, struct i915_request **src, int count)
2071−{
2072− /* A memcpy_p() would be very useful here! */
2073− while (count--)
2074− WRITE_ONCE(*dst++, *src++); /* avoid write tearing */
2075−}
2076−
2077−static void execlists_dequeue(struct intel_engine_cs *engine)
2078−{
2079− struct intel_engine_execlists * const execlists = &engine->execlists;
2080− struct i915_request **port = execlists->pending;
2081− struct i915_request ** const last_port = port + execlists->port_mask;
2082− struct i915_request * const *active;
2083− struct i915_request *last;
2084− struct rb_node *rb;
2085− bool submit = false;
2086−
2087− /*
2088− * Hardware submission is through 2 ports. Conceptually each port
2089− * has a (RING_START, RING_HEAD, RING_TAIL) tuple. RING_START is
2090− * static for a context, and unique to each, so we only execute
2091− * requests belonging to a single context from each ring. RING_HEAD
2092− * is maintained by the CS in the context image, it marks the place
2093− * where it got up to last time, and through RING_TAIL we tell the CS
2094− * where we want to execute up to this time.
2095− *
2096− * In this list the requests are in order of execution. Consecutive
2097− * requests from the same context are adjacent in the ringbuffer. We
2098− * can combine these requests into a single RING_TAIL update:
2099− *
2100− * RING_HEAD...req1...req2
2101− * ^- RING_TAIL
2102− * since to execute req2 the CS must first execute req1.
2103− *
2104− * Our goal then is to point each port to the end of a consecutive
2105− * sequence of requests as being the most optimal (fewest wake ups
2106− * and context switches) submission.
2107− */
2108−
2109− for (rb = rb_first_cached(&execlists->virtual); rb; ) {
2110− struct virtual_engine *ve =
2111− rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
2112− struct i915_request *rq = READ_ONCE(ve->request);
2113−
2114− if (!rq) { /* lazily cleanup after another engine handled rq */
2115− rb_erase_cached(rb, &execlists->virtual);
2116− RB_CLEAR_NODE(rb);
2117− rb = rb_first_cached(&execlists->virtual);
2118− continue;
2119− }
2120−
2121− if (!virtual_matches(ve, rq, engine)) {
2122− rb = rb_next(rb);
2123− continue;
2124− }
2125−
2126− break;
2127− }
2128−
2129− /*
2130− * If the queue is higher priority than the last
2131− * request in the currently active context, submit afresh.
2132− * We will resubmit again afterwards in case we need to split
2133− * the active context to interject the preemption request,
2134− * i.e. we will retrigger preemption following the ack in case
2135− * of trouble.
2136− */
2137− active = READ_ONCE(execlists->active);
2138−
2139− /*
2140− * In theory we can skip over completed contexts that have not
2141− * yet been processed by events (as those events are in flight):
2142− *
2143− * while ((last = *active) && i915_request_completed(last))
2144− * active++;
2145− *
2146− * However, the GPU cannot handle this as it will ultimately
2147− * find itself trying to jump back into a context it has just
2148− * completed and barf.
2149− */
2150−
2151− if ((last = *active)) {
2152− if (need_preempt(engine, last, rb)) {
2153− if (i915_request_completed(last)) {
2154− tasklet_hi_schedule(&execlists->tasklet);
2155− return;
2156− }
2157−
2158− ENGINE_TRACE(engine,
2159− "preempting last=%llx:%lld, prio=%d, hint=%d\n",
2160− last->fence.context,
2161− last->fence.seqno,
2162− last->sched.attr.priority,
2163− execlists->queue_priority_hint);
2164− record_preemption(execlists);
2165−
2166− /*
2167− * Don't let the RING_HEAD advance past the breadcrumb
2168− * as we unwind (and until we resubmit) so that we do
2169− * not accidentally tell it to go backwards.
2170− */
2171− ring_set_paused(engine, 1);
2172−
2173− /*
2174− * Note that we have not stopped the GPU at this point,
2175− * so we are unwinding the incomplete requests as they
2176− * remain inflight and so by the time we do complete
2177− * the preemption, some of the unwound requests may
2178− * complete!
2179− */
2180− __unwind_incomplete_requests(engine);
2181−
2182− last = NULL;
2183− } else if (need_timeslice(engine, last, rb) &&
2184− timeslice_expired(execlists, last)) {
2185− if (i915_request_completed(last)) {
2186− tasklet_hi_schedule(&execlists->tasklet);
2187− return;
2188− }
2189−
2190− ENGINE_TRACE(engine,
2191− "expired last=%llx:%lld, prio=%d, hint=%d, yield?=%s\n",
2192− last->fence.context,
2193− last->fence.seqno,
2194− last->sched.attr.priority,
2195− execlists->queue_priority_hint,
2196− yesno(timeslice_yield(execlists, last)));
2197−
2198− ring_set_paused(engine, 1);
2199− defer_active(engine);
2200−
2201− /*
2202− * Unlike for preemption, if we rewind and continue
2203− * executing the same context as previously active,
2204− * the order of execution will remain the same and
2205− * the tail will only advance. We do not need to
2206− * force a full context restore, as a lite-restore
2207− * is sufficient to resample the monotonic TAIL.
2208− *
2209− * If we switch to any other context, similarly we
2210− * will not rewind TAIL of current context, and
2211− * normal save/restore will preserve state and allow
2212− * us to later continue executing the same request.
2213− */
2214− last = NULL;
2215− } else {
2216− /*
2217− * Otherwise if we already have a request pending
2218− * for execution after the current one, we can
2219− * just wait until the next CS event before
2220− * queuing more. In either case we will force a
2221− * lite-restore preemption event, but if we wait
2222− * we hopefully coalesce several updates into a single
2223− * submission.
2224− */
2225− if (!list_is_last(&last->sched.link,
2226− &engine->active.requests)) {
2227− /*
2228− * Even if ELSP[1] is occupied and not worthy
2229− * of timeslices, our queue might be.
2230− */
2231− start_timeslice(engine, queue_prio(execlists));
2232− return;
2233− }
2234− }
2235− }
2236−
2237− while (rb) { /* XXX virtual is always taking precedence */
2238− struct virtual_engine *ve =
2239− rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
2240− struct i915_request *rq;
2241−
2242− spin_lock(&ve->base.active.lock);
2243−
2244− rq = ve->request;
2245− if (unlikely(!rq)) { /* lost the race to a sibling */
2246− spin_unlock(&ve->base.active.lock);
2247− rb_erase_cached(rb, &execlists->virtual);
2248− RB_CLEAR_NODE(rb);
2249− rb = rb_first_cached(&execlists->virtual);
2250− continue;
2251− }
2252−
2253− GEM_BUG_ON(rq != ve->request);
2254− GEM_BUG_ON(rq->engine != &ve->base);
2255− GEM_BUG_ON(rq->context != &ve->context);
2256−
2257− if (rq_prio(rq) >= queue_prio(execlists)) {
2258− if (!virtual_matches(ve, rq, engine)) {
2259− spin_unlock(&ve->base.active.lock);
2260− rb = rb_next(rb);
2261− continue;
2262− }
2263−
2264− if (last && !can_merge_rq(last, rq)) {
2265− spin_unlock(&ve->base.active.lock);
2266− start_timeslice(engine, rq_prio(rq));
2267− return; /* leave this for another sibling */
2268− }
2269−
2270− ENGINE_TRACE(engine,
2271− "virtual rq=%llx:%lld%s, new engine? %s\n",
2272− rq->fence.context,
2273− rq->fence.seqno,
2274− i915_request_completed(rq) ? "!" :
2275− i915_request_started(rq) ? "*" :
2276− "",
2277− yesno(engine != ve->siblings[0]));
2278−
2279− WRITE_ONCE(ve->request, NULL);
2280− WRITE_ONCE(ve->base.execlists.queue_priority_hint,
2281− INT_MIN);
2282− rb_erase_cached(rb, &execlists->virtual);
2283− RB_CLEAR_NODE(rb);
2284−
2285− GEM_BUG_ON(!(rq->execution_mask & engine->mask));
2286− WRITE_ONCE(rq->engine, engine);
2287−
2288− if (__i915_request_submit(rq)) {
2289− /*
2290− * Only after we confirm that we will submit
2291− * this request (i.e. it has not already
2292− * completed), do we want to update the context.
2293− *
2294− * This serves two purposes. It avoids
2295− * unnecessary work if we are resubmitting an
2296− * already completed request after timeslicing.
2297− * But more importantly, it prevents us altering
2298− * ve->siblings[] on an idle context, where
2299− * we may be using ve->siblings[] in
2300− * virtual_context_enter / virtual_context_exit.
2301− */
2302− virtual_xfer_context(ve, engine);
2303− GEM_BUG_ON(ve->siblings[0] != engine);
2304−
2305− submit = true;
2306− last = rq;
2307− }
2308− i915_request_put(rq);
2309−
2310− /*
2311− * Hmm, we have a bunch of virtual engine requests,
2312− * but the first one was already completed (thanks
2313− * preempt-to-busy!). Keep looking at the veng queue
2314− * until we have no more relevant requests (i.e.
2315− * the normal submit queue has higher priority).
2316− */
2317− if (!submit) {
2318− spin_unlock(&ve->base.active.lock);
2319− rb = rb_first_cached(&execlists->virtual);
2320− continue;
2321− }
2322− }
2323−
2324− spin_unlock(&ve->base.active.lock);
2325− break;
2326− }
2327−
2328− while ((rb = rb_first_cached(&execlists->queue))) {
2329− struct i915_priolist *p = to_priolist(rb);
2330− struct i915_request *rq, *rn;
2331− int i;
2332−
2333− priolist_for_each_request_consume(rq, rn, p, i) {
2334− bool merge = true;
2335−
2336− /*
2337− * Can we combine this request with the current port?
2338− * It has to be the same context/ringbuffer and not
2339− * have any exceptions (e.g. GVT saying never to
2340− * combine contexts).
2341− *
2342− * If we can combine the requests, we can execute both
2343− * by updating the RING_TAIL to point to the end of the
2344− * second request, and so we never need to tell the
2345− * hardware about the first.
2346− */
2347− if (last && !can_merge_rq(last, rq)) {
2348− /*
2349− * If we are on the second port and cannot
2350− * combine this request with the last, then we
2351− * are done.
2352− */
2353− if (port == last_port)
2354− goto done;
2355−
2356− /*
2357− * We must not populate both ELSP[] with the
2358− * same LRCA, i.e. we must submit 2 different
2359− * contexts if we submit 2 ELSP.
2360− */
2361− if (last->context == rq->context)
2362− goto done;
2363−
2364− if (i915_request_has_sentinel(last))
2365− goto done;
2366−
2367− /*
2368− * If GVT overrides us we only ever submit
2369− * port[0], leaving port[1] empty. Note that we
2370− * also have to be careful that we don't queue
2371− * the same context (even though a different
2372− * request) to the second port.
2373− */
2374− if (ctx_single_port_submission(last->context) ||
2375− ctx_single_port_submission(rq->context))
2376− goto done;
2377−
2378− merge = false;
2379− }
2380−
2381− if (__i915_request_submit(rq)) {
2382− if (!merge) {
2383− *port = execlists_schedule_in(last, port - execlists->pending);
2384− port++;
2385− last = NULL;
2386− }
2387−
2388− GEM_BUG_ON(last &&
2389− !can_merge_ctx(last->context,
2390− rq->context));
2391− GEM_BUG_ON(last &&
2392− i915_seqno_passed(last->fence.seqno,
2393− rq->fence.seqno));
2394−
2395− submit = true;
2396− last = rq;
2397− }
2398− }
2399−
2400− rb_erase_cached(&p->node, &execlists->queue);
2401− i915_priolist_free(p);
2402− }
2403−
2404−done:
2405− /*
2406− * Here be a bit of magic! Or sleight-of-hand, whichever you prefer.
2407− *
2408− * We choose the priority hint such that if we add a request of greater
2409− * priority than this, we kick the submission tasklet to decide on
2410− * the right order of submitting the requests to hardware. We must
2411− * also be prepared to reorder requests as they are in-flight on the
2412− * HW. We derive the priority hint then as the first "hole" in
2413− * the HW submission ports and if there are no available slots,
2414− * the priority of the lowest executing request, i.e. last.
2415− *
2416− * When we do receive a higher priority request ready to run from the
2417− * user, see queue_request(), the priority hint is bumped to that
2418− * request triggering preemption on the next dequeue (or subsequent
2419− * interrupt for secondary ports).
2420− */
2421− execlists->queue_priority_hint = queue_prio(execlists);
2422−
2423− if (submit) {
2424− *port = execlists_schedule_in(last, port - execlists->pending);
2425− execlists->switch_priority_hint =
2426− switch_prio(engine, *execlists->pending);
2427−
2428− /*
2429− * Skip if we ended up with exactly the same set of requests,
2430− * e.g. trying to timeslice a pair of ordered contexts
2431− */
2432− if (!memcmp(active, execlists->pending,
2433− (port - execlists->pending + 1) * sizeof(*port))) {
2434− do
2435− execlists_schedule_out(fetch_and_zero(port));
2436− while (port-- != execlists->pending);
2437−
2438− goto skip_submit;
2439− }
2440− clear_ports(port + 1, last_port - port);
2441−
2442− WRITE_ONCE(execlists->yield, -1);
2443− set_preempt_timeout(engine, *active);
2444− execlists_submit_ports(engine);
2445− } else {
2446− start_timeslice(engine, execlists->queue_priority_hint);
2447−skip_submit:
2448− ring_set_paused(engine, 0);
2449− }
2450−}
2451−
2452−static void
2453−cancel_port_requests(struct intel_engine_execlists * const execlists)
2454−{
2455− struct i915_request * const *port;
2456−
2457− for (port = execlists->pending; *port; port++)
2458− execlists_schedule_out(*port);
2459− clear_ports(execlists->pending, ARRAY_SIZE(execlists->pending));
2460−
2461− /* Mark the end of active before we overwrite *active */
2462− for (port = xchg(&execlists->active, execlists->pending); *port; port++)
2463− execlists_schedule_out(*port);
2464− clear_ports(execlists->inflight, ARRAY_SIZE(execlists->inflight));
2465−
2466− smp_wmb(); /* complete the seqlock for execlists_active() */
2467− WRITE_ONCE(execlists->active, execlists->inflight);
2468−}
2469−
2470−static inline void
2471−invalidate_csb_entries(const u64 *first, const u64 *last)
2472−{
2473− clflush((void *)first);
2474− clflush((void *)last);
2475−}
2476−
2477−/*
2478− * Starting with Gen12, the status has a new format:
2479− *
2480− * bit 0: switched to new queue
2481− * bit 1: reserved
2482− * bit 2: semaphore wait mode (poll or signal), only valid when
2483− * switch detail is set to "wait on semaphore"
2484− * bits 3-5: engine class
2485− * bits 6-11: engine instance
2486− * bits 12-14: reserved
2487− * bits 15-25: sw context id of the lrc the GT switched to
2488− * bits 26-31: sw counter of the lrc the GT switched to
2489− * bits 32-35: context switch detail
2490− * - 0: ctx complete
2491− * - 1: wait on sync flip
2492− * - 2: wait on vblank
2493− * - 3: wait on scanline
2494− * - 4: wait on semaphore
2495− * - 5: context preempted (not on SEMAPHORE_WAIT or
2496− * WAIT_FOR_EVENT)
2497− * bit 36: reserved
2498− * bits 37-43: wait detail (for switch detail 1 to 4)
2499− * bits 44-46: reserved
2500− * bits 47-57: sw context id of the lrc the GT switched away from
2501− * bits 58-63: sw counter of the lrc the GT switched away from
2502− */
2503−static inline bool gen12_csb_parse(const u64 csb)
2504−{
2505− bool ctx_away_valid = GEN12_CSB_CTX_VALID(upper_32_bits(csb));
2506− bool new_queue =
2507− lower_32_bits(csb) & GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE;
2508−
2509− /*
2510− * The context switch detail is not guaranteed to be 5 when a preemption
2511− * occurs, so we can't just check for that. The check below works for
2512− * all the cases we care about, including preemptions of WAIT
2513− * instructions and lite-restore. Preempt-to-idle via the CTRL register
2514− * would require some extra handling, but we don't support that.
2515− */
2516− if (!ctx_away_valid || new_queue) {
2517− GEM_BUG_ON(!GEN12_CSB_CTX_VALID(lower_32_bits(csb)));
2518− return true;
2519− }
2520−
2521− /*
2522− * switch detail = 5 is covered by the case above and we do not expect a
2523− * context switch on an unsuccessful wait instruction since we always
2524− * use polling mode.
2525− */
2526− GEM_BUG_ON(GEN12_CTX_SWITCH_DETAIL(upper_32_bits(csb)));
2527− return false;
2528−}
2529−
2530−static inline bool gen8_csb_parse(const u64 csb)
2531−{
2532− return csb & (GEN8_CTX_STATUS_IDLE_ACTIVE | GEN8_CTX_STATUS_PREEMPTED);
2533−}
2534−
2535−static noinline u64
2536−wa_csb_read(const struct intel_engine_cs *engine, u64 * const csb)
2537−{
2538− u64 entry;
2539−
2540− /*
2541− * Reading from the HWSP has one particular advantage: we can detect
2542− * a stale entry. Since the write into HWSP is broken, we have no reason
2543− * to trust the HW at all, the mmio entry may equally be unordered, so
2544− * we prefer the path that is self-checking and as a last resort,
2545− * return the mmio value.
2546− *
2547− * tgl,dg1:HSDES#22011327657
2548− */
2549− preempt_disable();
2550− if (wait_for_atomic_us((entry = READ_ONCE(*csb)) != -1, 10)) {
2551− int idx = csb - engine->execlists.csb_status;
2552− int status;
2553−
2554− status = GEN8_EXECLISTS_STATUS_BUF;
2555− if (idx >= 6) {
2556− status = GEN11_EXECLISTS_STATUS_BUF2;
2557− idx -= 6;
2558− }
2559− status += sizeof(u64) * idx;
2560−
2561− entry = intel_uncore_read64(engine->uncore,
2562− _MMIO(engine->mmio_base + status));
2563− }
2564− preempt_enable();
2565−
2566− return entry;
2567−}
2568−
2569−static inline u64
2570−csb_read(const struct intel_engine_cs *engine, u64 * const csb)
2571−{
2572− u64 entry = READ_ONCE(*csb);
2573−
2574− /*
2575− * Unfortunately, the GPU does not always serialise its write
2576− * of the CSB entries before its write of the CSB pointer, at least
2577− * from the perspective of the CPU, using what is known as a Global
2578− * Observation Point. We may read a new CSB tail pointer, but then
2579− * read the stale CSB entries, causing us to misinterpret the
2580− * context-switch events, and eventually declare the GPU hung.
2581− *
2582− * icl:HSDES#1806554093
2583− * tgl:HSDES#22011248461
2584− */
2585− if (unlikely(entry == -1))
2586− entry = wa_csb_read(engine, csb);
2587−
2588− /* Consume this entry so that we can spot its future reuse. */
2589− WRITE_ONCE(*csb, -1);
2590−
2591− /* ELSP is an implicit wmb() before the GPU wraps and overwrites csb */
2592− return entry;
2593−}
2594−
2595−static void process_csb(struct intel_engine_cs *engine)
2596−{
2597− struct intel_engine_execlists * const execlists = &engine->execlists;
2598− u64 * const buf = execlists->csb_status;
2599− const u8 num_entries = execlists->csb_size;
2600− u8 head, tail;
2601−
2602− /*
2603− * As we modify our execlists state tracking we require exclusive
2604− * access. Either we are inside the tasklet, or the tasklet is disabled
2605− * and we assume that is only inside the reset paths and so serialised.
2606− */
2607− GEM_BUG_ON(!tasklet_is_locked(&execlists->tasklet) &&
2608− !reset_in_progress(execlists));
2609− GEM_BUG_ON(!intel_engine_in_execlists_submission_mode(engine));
2610−
2611− /*
2612− * Note that csb_write, csb_status may be either in HWSP or mmio.
2613− * When reading from the csb_write mmio register, we have to be
2614− * careful to only use the GEN8_CSB_WRITE_PTR portion, which is
2615− * the low 4bits. As it happens we know the next 4bits are always
2616− * zero and so we can simply masked off the low u8 of the register
2617− * and treat it identically to reading from the HWSP (without having
2618− * to use explicit shifting and masking, and probably bifurcating
2619− * the code to handle the legacy mmio read).
2620− */
2621− head = execlists->csb_head;
2622− tail = READ_ONCE(*execlists->csb_write);
2623− if (unlikely(head == tail))
2624− return;
2625−
2626− /*
2627− * We will consume all events from HW, or at least pretend to.
2628− *
2629− * The sequence of events from the HW is deterministic, and derived
2630− * from our writes to the ELSP, with a smidgen of variability for
2631− * the arrival of the asynchronous requests wrt to the inflight
2632− * execution. If the HW sends an event that does not correspond with
2633− * the one we are expecting, we have to abandon all hope as we lose
2634− * all tracking of what the engine is actually executing. We will
2635− * only detect we are out of sequence with the HW when we get an
2636− * 'impossible' event because we have already drained our own
2637− * preemption/promotion queue. If this occurs, we know that we likely
2638− * lost track of execution earlier and must unwind and restart, the
2639− * simplest way is by stop processing the event queue and force the
2640− * engine to reset.
2641− */
2642− execlists->csb_head = tail;
2643− ENGINE_TRACE(engine, "cs-irq head=%d, tail=%d\n", head, tail);
2644−
2645− /*
2646− * Hopefully paired with a wmb() in HW!
2647− *
2648− * We must complete the read of the write pointer before any reads
2649− * from the CSB, so that we do not see stale values. Without an rmb
2650− * (lfence) the HW may speculatively perform the CSB[] reads *before*
2651− * we perform the READ_ONCE(*csb_write).
2652− */
2653− rmb();
2654− do {
2655− bool promote;
2656− u64 csb;
2657−
2658− if (++head == num_entries)
2659− head = 0;
2660−
2661− /*
2662− * We are flying near dragons again.
2663− *
2664− * We hold a reference to the request in execlist_port[]
2665− * but no more than that. We are operating in softirq
2666− * context and so cannot hold any mutex or sleep. That
2667− * prevents us stopping the requests we are processing
2668− * in port[] from being retired simultaneously (the
2669− * breadcrumb will be complete before we see the
2670− * context-switch). As we only hold the reference to the
2671− * request, any pointer chasing underneath the request
2672− * is subject to a potential use-after-free. Thus we
2673− * store all of the bookkeeping within port[] as
2674− * required, and avoid using unguarded pointers beneath
2675− * request itself. The same applies to the atomic
2676− * status notifier.
2677− */
2678−
2679− csb = csb_read(engine, buf + head);
2680− ENGINE_TRACE(engine, "csb[%d]: status=0x%08x:0x%08x\n",
2681− head, upper_32_bits(csb), lower_32_bits(csb));
2682−
2683− if (INTEL_GEN(engine->i915) >= 12)
2684− promote = gen12_csb_parse(csb);
2685− else
2686− promote = gen8_csb_parse(csb);
2687− if (promote) {
2688− struct i915_request * const *old = execlists->active;
2689−
2690− if (GEM_WARN_ON(!*execlists->pending)) {
2691− execlists->error_interrupt |= ERROR_CSB;
2692− break;
2693− }
2694−
2695− ring_set_paused(engine, 0);
2696−
2697− /* Point active to the new ELSP; prevent overwriting */
2698− WRITE_ONCE(execlists->active, execlists->pending);
2699− smp_wmb(); /* notify execlists_active() */
2700−
2701− /* cancel old inflight, prepare for switch */
2702− trace_ports(execlists, "preempted", old);
2703− while (*old)
2704− execlists_schedule_out(*old++);
2705−
2706− /* switch pending to inflight */
2707− GEM_BUG_ON(!assert_pending_valid(execlists, "promote"));
2708− copy_ports(execlists->inflight,
2709− execlists->pending,
2710− execlists_num_ports(execlists));
2711− smp_wmb(); /* complete the seqlock */
2712− WRITE_ONCE(execlists->active, execlists->inflight);
2713−
2714− /* XXX Magic delay for tgl */
2715− ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
2716−
2717− WRITE_ONCE(execlists->pending[0], NULL);
2718− } else {
2719− if (GEM_WARN_ON(!*execlists->active)) {
2720− execlists->error_interrupt |= ERROR_CSB;
2721− break;
2722− }
2723−
2724− /* port0 completed, advanced to port1 */
2725− trace_ports(execlists, "completed", execlists->active);
2726−
2727− /*
2728− * We rely on the hardware being strongly
2729− * ordered, that the breadcrumb write is
2730− * coherent (visible from the CPU) before the
2731− * user interrupt is processed. One might assume
2732− * that the breadcrumb write being before the
2733− * user interrupt and the CS event for the context
2734− * switch would therefore be before the CS event
2735− * itself...
2736− */
2737− if (GEM_SHOW_DEBUG() &&
2738− !i915_request_completed(*execlists->active)) {
2739− struct i915_request *rq = *execlists->active;
2740− const u32 *regs __maybe_unused =
2741− rq->context->lrc_reg_state;
2742−
2743− ENGINE_TRACE(engine,
2744− "context completed before request!\n");
2745− ENGINE_TRACE(engine,
2746− "ring:{start:0x%08x, head:%04x, tail:%04x, ctl:%08x, mode:%08x}\n",
2747− ENGINE_READ(engine, RING_START),
2748− ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR,
2749− ENGINE_READ(engine, RING_TAIL) & TAIL_ADDR,
2750− ENGINE_READ(engine, RING_CTL),
2751− ENGINE_READ(engine, RING_MI_MODE));
2752− ENGINE_TRACE(engine,
2753− "rq:{start:%08x, head:%04x, tail:%04x, seqno:%llx:%d, hwsp:%d}, ",
2754− i915_ggtt_offset(rq->ring->vma),
2755− rq->head, rq->tail,
2756− rq->fence.context,
2757− lower_32_bits(rq->fence.seqno),
2758− hwsp_seqno(rq));
2759− ENGINE_TRACE(engine,
2760− "ctx:{start:%08x, head:%04x, tail:%04x}, ",
2761− regs[CTX_RING_START],
2762− regs[CTX_RING_HEAD],
2763− regs[CTX_RING_TAIL]);
2764− }
2765−
2766− execlists_schedule_out(*execlists->active++);
2767−
2768− GEM_BUG_ON(execlists->active - execlists->inflight >
2769− execlists_num_ports(execlists));
2770− }
2771− } while (head != tail);
2772−
2773− set_timeslice(engine);
2774−
2775− /*
2776− * Gen11 has proven to fail wrt global observation point between
2777− * entry and tail update, failing on the ordering and thus
2778− * we see an old entry in the context status buffer.
2779− *
2780− * Forcibly evict out entries for the next gpu csb update,
2781− * to increase the odds that we get a fresh entries with non
2782− * working hardware. The cost for doing so comes out mostly with
2783− * the wash as hardware, working or not, will need to do the
2784− * invalidation before.
2785− */
2786− invalidate_csb_entries(&buf[0], &buf[num_entries - 1]);
2787−}
2788−
2789−static void __execlists_submission_tasklet(struct intel_engine_cs *const engine)
2790−{
2791− lockdep_assert_held(&engine->active.lock);
2792− if (!READ_ONCE(engine->execlists.pending[0])) {
2793− rcu_read_lock(); /* protect peeking at execlists->active */
2794− execlists_dequeue(engine);
2795− rcu_read_unlock();
2796− }
2797−}
2798−
2799−static void __execlists_hold(struct i915_request *rq)
2800−{
2801− LIST_HEAD(list);
2802−
2803− do {
2804− struct i915_dependency *p;
2805−
2806− if (i915_request_is_active(rq))
2807− __i915_request_unsubmit(rq);
2808−
2809− clear_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2810− list_move_tail(&rq->sched.link, &rq->engine->active.hold);
2811− i915_request_set_hold(rq);
2812− RQ_TRACE(rq, "on hold\n");
2813−
2814− for_each_waiter(p, rq) {
2815− struct i915_request *w =
2816− container_of(p->waiter, typeof(*w), sched);
2817−
2818− /* Leave semaphores spinning on the other engines */
2819− if (w->engine != rq->engine)
2820− continue;
2821−
2822− if (!i915_request_is_ready(w))
2823− continue;
2824−
2825− if (i915_request_completed(w))
2826− continue;
2827−
2828− if (i915_request_on_hold(w))
2829− continue;
2830−
2831− list_move_tail(&w->sched.link, &list);
2832− }
2833−
2834− rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2835− } while (rq);
2836−}
2837−
2838−static bool execlists_hold(struct intel_engine_cs *engine,
2839− struct i915_request *rq)
2840−{
2841− if (i915_request_on_hold(rq))
2842− return false;
2843−
2844− spin_lock_irq(&engine->active.lock);
2845−
2846− if (i915_request_completed(rq)) { /* too late! */
2847− rq = NULL;
2848− goto unlock;
2849− }
2850−
2851− if (rq->engine != engine) { /* preempted virtual engine */
2852− struct virtual_engine *ve = to_virtual_engine(rq->engine);
2853−
2854− /*
2855− * intel_context_inflight() is only protected by virtue
2856− * of process_csb() being called only by the tasklet (or
2857− * directly from inside reset while the tasklet is suspended).
2858− * Assert that neither of those are allowed to run while we
2859− * poke at the request queues.
2860− */
2861− GEM_BUG_ON(!reset_in_progress(&engine->execlists));
2862−
2863− /*
2864− * An unsubmitted request along a virtual engine will
2865− * remain on the active (this) engine until we are able
2866− * to process the context switch away (and so mark the
2867− * context as no longer in flight). That cannot have happened
2868− * yet, otherwise we would not be hanging!
2869− */
2870− spin_lock(&ve->base.active.lock);
2871− GEM_BUG_ON(intel_context_inflight(rq->context) != engine);
2872− GEM_BUG_ON(ve->request != rq);
2873− ve->request = NULL;
2874− spin_unlock(&ve->base.active.lock);
2875− i915_request_put(rq);
2876−
2877− rq->engine = engine;
2878− }
2879−
2880− /*
2881− * Transfer this request onto the hold queue to prevent it
2882− * being resumbitted to HW (and potentially completed) before we have
2883− * released it. Since we may have already submitted following
2884− * requests, we need to remove those as well.
2885− */
2886− GEM_BUG_ON(i915_request_on_hold(rq));
2887− GEM_BUG_ON(rq->engine != engine);
2888− __execlists_hold(rq);
2889− GEM_BUG_ON(list_empty(&engine->active.hold));
2890−
2891−unlock:
2892− spin_unlock_irq(&engine->active.lock);
2893− return rq;
2894−}
2895−
2896−static bool hold_request(const struct i915_request *rq)
2897−{
2898− struct i915_dependency *p;
2899− bool result = false;
2900−
2901− /*
2902− * If one of our ancestors is on hold, we must also be on hold,
2903− * otherwise we will bypass it and execute before it.
2904− */
2905− rcu_read_lock();
2906− for_each_signaler(p, rq) {
2907− const struct i915_request *s =
2908− container_of(p->signaler, typeof(*s), sched);
2909−
2910− if (s->engine != rq->engine)
2911− continue;
2912−
2913− result = i915_request_on_hold(s);
2914− if (result)
2915− break;
2916− }
2917− rcu_read_unlock();
2918−
2919− return result;
2920−}
2921−
2922−static void __execlists_unhold(struct i915_request *rq)
2923−{
2924− LIST_HEAD(list);
2925−
2926− do {
2927− struct i915_dependency *p;
2928−
2929− RQ_TRACE(rq, "hold release\n");
2930−
2931− GEM_BUG_ON(!i915_request_on_hold(rq));
2932− GEM_BUG_ON(!i915_sw_fence_signaled(&rq->submit));
2933−
2934− i915_request_clear_hold(rq);
2935− list_move_tail(&rq->sched.link,
2936− i915_sched_lookup_priolist(rq->engine,
2937− rq_prio(rq)));
2938− set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2939−
2940− /* Also release any children on this engine that are ready */
2941− for_each_waiter(p, rq) {
2942− struct i915_request *w =
2943− container_of(p->waiter, typeof(*w), sched);
2944−
2945− /* Propagate any change in error status */
2946− if (rq->fence.error)
2947− i915_request_set_error_once(w, rq->fence.error);
2948−
2949− if (w->engine != rq->engine)
2950− continue;
2951−
2952− if (!i915_request_on_hold(w))
2953− continue;
2954−
2955− /* Check that no other parents are also on hold */
2956− if (hold_request(w))
2957− continue;
2958−
2959− list_move_tail(&w->sched.link, &list);
2960− }
2961−
2962− rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2963− } while (rq);
2964−}
2965−
2966−static void execlists_unhold(struct intel_engine_cs *engine,
2967− struct i915_request *rq)
2968−{
2969− spin_lock_irq(&engine->active.lock);
2970−
2971− /*
2972− * Move this request back to the priority queue, and all of its
2973− * children and grandchildren that were suspended along with it.
2974− */
2975− __execlists_unhold(rq);
2976−
2977− if (rq_prio(rq) > engine->execlists.queue_priority_hint) {
2978− engine->execlists.queue_priority_hint = rq_prio(rq);
2979− tasklet_hi_schedule(&engine->execlists.tasklet);
2980− }
2981−
2982− spin_unlock_irq(&engine->active.lock);
2983−}
2984−
2985−struct execlists_capture {
2986− struct work_struct work;
2987− struct i915_request *rq;
2988− struct i915_gpu_coredump *error;
2989−};
2990−
2991−static void execlists_capture_work(struct work_struct *work)
2992−{
2993− struct execlists_capture *cap = container_of(work, typeof(*cap), work);
2994− const gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
2995− struct intel_engine_cs *engine = cap->rq->engine;
2996− struct intel_gt_coredump *gt = cap->error->gt;
2997− struct intel_engine_capture_vma *vma;
2998−
2999− /* Compress all the objects attached to the request, slow! */
3000− vma = intel_engine_coredump_add_request(gt->engine, cap->rq, gfp);
3001− if (vma) {
3002− struct i915_vma_compress *compress =
3003− i915_vma_capture_prepare(gt);
3004−
3005− intel_engine_coredump_add_vma(gt->engine, vma, compress);
3006− i915_vma_capture_finish(gt, compress);
3007− }
3008−
3009− gt->simulated = gt->engine->simulated;
3010− cap->error->simulated = gt->simulated;
3011−
3012− /* Publish the error state, and announce it to the world */
3013− i915_error_state_store(cap->error);
3014− i915_gpu_coredump_put(cap->error);
3015−
3016− /* Return this request and all that depend upon it for signaling */
3017− execlists_unhold(engine, cap->rq);
3018− i915_request_put(cap->rq);
3019−
3020− kfree(cap);
3021−}
3022−
3023−static struct execlists_capture *capture_regs(struct intel_engine_cs *engine)
3024−{
3025− const gfp_t gfp = GFP_ATOMIC | __GFP_NOWARN;
3026− struct execlists_capture *cap;
3027−
3028− cap = kmalloc(sizeof(*cap), gfp);
3029− if (!cap)
3030− return NULL;
3031−
3032− cap->error = i915_gpu_coredump_alloc(engine->i915, gfp);
3033− if (!cap->error)
3034− goto err_cap;
3035−
3036− cap->error->gt = intel_gt_coredump_alloc(engine->gt, gfp);
3037− if (!cap->error->gt)
3038− goto err_gpu;
3039−
3040− cap->error->gt->engine = intel_engine_coredump_alloc(engine, gfp);
3041− if (!cap->error->gt->engine)
3042− goto err_gt;
3043−
3044− cap->error->gt->engine->hung = true;
3045−
3046− return cap;
3047−
3048−err_gt:
3049− kfree(cap->error->gt);
3050−err_gpu:
3051− kfree(cap->error);
3052−err_cap:
3053− kfree(cap);
3054− return NULL;
3055−}
3056−
3057−static struct i915_request *
3058−active_context(struct intel_engine_cs *engine, u32 ccid)
3059−{
3060− const struct intel_engine_execlists * const el = &engine->execlists;
3061− struct i915_request * const *port, *rq;
3062−
3063− /*
3064− * Use the most recent result from process_csb(), but just in case
3065− * we trigger an error (via interrupt) before the first CS event has
3066− * been written, peek at the next submission.
3067− */
3068−
3069− for (port = el->active; (rq = *port); port++) {
3070− if (rq->context->lrc.ccid == ccid) {
3071− ENGINE_TRACE(engine,
3072− "ccid found at active:%zd\n",
3073− port - el->active);
3074− return rq;
3075− }
3076− }
3077−
3078− for (port = el->pending; (rq = *port); port++) {
3079− if (rq->context->lrc.ccid == ccid) {
3080− ENGINE_TRACE(engine,
3081− "ccid found at pending:%zd\n",
3082− port - el->pending);
3083− return rq;
3084− }
3085− }
3086−
3087− ENGINE_TRACE(engine, "ccid:%x not found\n", ccid);
3088− return NULL;
3089−}
3090−
3091−static u32 active_ccid(struct intel_engine_cs *engine)
3092−{
3093− return ENGINE_READ_FW(engine, RING_EXECLIST_STATUS_HI);
3094−}
3095−
3096−static void execlists_capture(struct intel_engine_cs *engine)
3097−{
3098− struct execlists_capture *cap;
3099−
3100− if (!IS_ENABLED(CONFIG_DRM_I915_CAPTURE_ERROR))
3101− return;
3102−
3103− /*
3104− * We need to _quickly_ capture the engine state before we reset.
3105− * We are inside an atomic section (softirq) here and we are delaying
3106− * the forced preemption event.
3107− */
3108− cap = capture_regs(engine);
3109− if (!cap)
3110− return;
3111−
3112− spin_lock_irq(&engine->active.lock);
3113− cap->rq = active_context(engine, active_ccid(engine));
3114− if (cap->rq) {
3115− cap->rq = active_request(cap->rq->context->timeline, cap->rq);
3116− cap->rq = i915_request_get_rcu(cap->rq);
3117− }
3118− spin_unlock_irq(&engine->active.lock);
3119− if (!cap->rq)
3120− goto err_free;
3121−
3122− /*
3123− * Remove the request from the execlists queue, and take ownership
3124− * of the request. We pass it to our worker who will _slowly_ compress
3125− * all the pages the _user_ requested for debugging their batch, after
3126− * which we return it to the queue for signaling.
3127− *
3128− * By removing them from the execlists queue, we also remove the
3129− * requests from being processed by __unwind_incomplete_requests()
3130− * during the intel_engine_reset(), and so they will *not* be replayed
3131− * afterwards.
3132− *
3133− * Note that because we have not yet reset the engine at this point,
3134− * it is possible for the request that we have identified as being
3135− * guilty, did in fact complete and we will then hit an arbitration
3136− * point allowing the outstanding preemption to succeed. The likelihood
3137− * of that is very low (as capturing of the engine registers should be
3138− * fast enough to run inside an irq-off atomic section!), so we will
3139− * simply hold that request accountable for being non-preemptible
3140− * long enough to force the reset.
3141− */
3142− if (!execlists_hold(engine, cap->rq))
3143− goto err_rq;
3144−
3145− INIT_WORK(&cap->work, execlists_capture_work);
3146− schedule_work(&cap->work);
3147− return;
3148−
3149−err_rq:
3150− i915_request_put(cap->rq);
3151−err_free:
3152− i915_gpu_coredump_put(cap->error);
3153− kfree(cap);
3154−}
3155−
3156−static void execlists_reset(struct intel_engine_cs *engine, const char *msg)
3157−{
3158− const unsigned int bit = I915_RESET_ENGINE + engine->id;
3159− unsigned long *lock = &engine->gt->reset.flags;
3160−
3161− if (!intel_has_reset_engine(engine->gt))
3162− return;
3163−
3164− if (test_and_set_bit(bit, lock))
3165− return;
3166−
3167− ENGINE_TRACE(engine, "reset for %s\n", msg);
3168−
3169− /* Mark this tasklet as disabled to avoid waiting for it to complete */
3170− tasklet_disable_nosync(&engine->execlists.tasklet);
3171−
3172− ring_set_paused(engine, 1); /* Freeze the current request in place */
3173− execlists_capture(engine);
3174− intel_engine_reset(engine, msg);
3175−
3176− tasklet_enable(&engine->execlists.tasklet);
3177− clear_and_wake_up_bit(bit, lock);
3178−}
3179−
3180−static bool preempt_timeout(const struct intel_engine_cs *const engine)
3181−{
3182− const struct timer_list *t = &engine->execlists.preempt;
3183−
3184− if (!CONFIG_DRM_I915_PREEMPT_TIMEOUT)
3185− return false;
3186−
3187− if (!timer_expired(t))
3188− return false;
3189−
3190− return READ_ONCE(engine->execlists.pending[0]);
3191−}
3192−
3193−/*
3194− * Check the unread Context Status Buffers and manage the submission of new
3195− * contexts to the ELSP accordingly.
3196− */
3197−static void execlists_submission_tasklet(unsigned long data)
3198−{
3199− struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
3200− bool timeout = preempt_timeout(engine);
3201−
3202− process_csb(engine);
3203−
3204− if (unlikely(READ_ONCE(engine->execlists.error_interrupt))) {
3205− const char *msg;
3206−
3207− /* Generate the error message in priority wrt to the user! */
3208− if (engine->execlists.error_interrupt & GENMASK(15, 0))
3209− msg = "CS error"; /* thrown by a user payload */
3210− else if (engine->execlists.error_interrupt & ERROR_CSB)
3211− msg = "invalid CSB event";
3212− else
3213− msg = "internal error";
3214−
3215− engine->execlists.error_interrupt = 0;
3216− execlists_reset(engine, msg);
3217− }
3218−
3219− if (!READ_ONCE(engine->execlists.pending[0]) || timeout) {
3220− unsigned long flags;
3221−
3222− spin_lock_irqsave(&engine->active.lock, flags);
3223− __execlists_submission_tasklet(engine);
3224− spin_unlock_irqrestore(&engine->active.lock, flags);
3225−
3226− /* Recheck after serialising with direct-submission */
3227− if (unlikely(timeout && preempt_timeout(engine))) {
3228− cancel_timer(&engine->execlists.preempt);
3229− execlists_reset(engine, "preemption time out");
3230− }
3231− }
3232−}
3233−
3234−static void __execlists_kick(struct intel_engine_execlists *execlists)
3235−{
3236− /* Kick the tasklet for some interrupt coalescing and reset handling */
3237− tasklet_hi_schedule(&execlists->tasklet);
3238−}
3239−
3240−#define execlists_kick(t, member) \
3241− __execlists_kick(container_of(t, struct intel_engine_execlists, member))
3242−
3243−static void execlists_timeslice(struct timer_list *timer)
3244−{
3245− execlists_kick(timer, timer);
3246−}
3247−
3248−static void execlists_preempt(struct timer_list *timer)
3249−{
3250− execlists_kick(timer, preempt);
3251−}
3252−
3253−static void queue_request(struct intel_engine_cs *engine,
3254− struct i915_request *rq)
3255−{
3256− GEM_BUG_ON(!list_empty(&rq->sched.link));
3257− list_add_tail(&rq->sched.link,
3258− i915_sched_lookup_priolist(engine, rq_prio(rq)));
3259− set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
3260−}
3261−
3262−static void __submit_queue_imm(struct intel_engine_cs *engine)
3263−{
3264− struct intel_engine_execlists * const execlists = &engine->execlists;
3265−
3266− if (reset_in_progress(execlists))
3267− return; /* defer until we restart the engine following reset */
3268−
3269− __execlists_submission_tasklet(engine);
3270−}
3271−
3272−static void submit_queue(struct intel_engine_cs *engine,
3273− const struct i915_request *rq)
3274−{
3275− struct intel_engine_execlists *execlists = &engine->execlists;
3276−
3277− if (rq_prio(rq) <= execlists->queue_priority_hint)
3278− return;
3279−
3280− execlists->queue_priority_hint = rq_prio(rq);
3281− __submit_queue_imm(engine);
3282−}
3283−
3284−static bool ancestor_on_hold(const struct intel_engine_cs *engine,
3285− const struct i915_request *rq)
3286−{
3287− GEM_BUG_ON(i915_request_on_hold(rq));
3288− return !list_empty(&engine->active.hold) && hold_request(rq);
3289−}
3290−
3291−static void flush_csb(struct intel_engine_cs *engine)
3292−{
3293− struct intel_engine_execlists *el = &engine->execlists;
3294−
3295− if (READ_ONCE(el->pending[0]) && tasklet_trylock(&el->tasklet)) {
3296− if (!reset_in_progress(el))
3297− process_csb(engine);
3298− tasklet_unlock(&el->tasklet);
3299− }
3300−}
3301−
3302−static void execlists_submit_request(struct i915_request *request)
3303−{
3304− struct intel_engine_cs *engine = request->engine;
3305− unsigned long flags;
3306−
3307− /* Hopefully we clear execlists->pending[] to let us through */
3308− flush_csb(engine);
3309−
3310− /* Will be called from irq-context when using foreign fences. */
3311− spin_lock_irqsave(&engine->active.lock, flags);
3312−
3313− if (unlikely(ancestor_on_hold(engine, request))) {
3314− RQ_TRACE(request, "ancestor on hold\n");
3315− list_add_tail(&request->sched.link, &engine->active.hold);
3316− i915_request_set_hold(request);
3317− } else {
3318− queue_request(engine, request);
3319−
3320− GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
3321− GEM_BUG_ON(list_empty(&request->sched.link));
3322−
3323− submit_queue(engine, request);
3324− }
3325−
3326− spin_unlock_irqrestore(&engine->active.lock, flags);
3327−}
3328−
3329−static void __execlists_context_fini(struct intel_context *ce)
3330−{
3331− intel_ring_put(ce->ring);
3332− i915_vma_put(ce->state);
3333−}
3334−
3335−static void execlists_context_destroy(struct kref *kref)
3336−{
3337− struct intel_context *ce = container_of(kref, typeof(*ce), ref);
3338−
3339− GEM_BUG_ON(!i915_active_is_idle(&ce->active));
3340− GEM_BUG_ON(intel_context_is_pinned(ce));
3341−
3342− if (ce->state)
3343− __execlists_context_fini(ce);
3344−
3345− intel_context_fini(ce);
3346− intel_context_free(ce);
3347−}
3348−
3349−static void
3350−set_redzone(void *vaddr, const struct intel_engine_cs *engine)
3351−{
3352− if (!IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
3353− return;
3354−
3355− vaddr += engine->context_size;
3356−
3357− memset(vaddr, CONTEXT_REDZONE, I915_GTT_PAGE_SIZE);
3358−}
3359−
3360−static void
3361−check_redzone(const void *vaddr, const struct intel_engine_cs *engine)
3362−{
3363− if (!IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
3364− return;
3365−
3366− vaddr += engine->context_size;
3367−
3368− if (memchr_inv(vaddr, CONTEXT_REDZONE, I915_GTT_PAGE_SIZE))
3369− drm_err_once(&engine->i915->drm,
3370− "%s context redzone overwritten!\n",
3371− engine->name);
3372−}
3373−
3374−static void execlists_context_unpin(struct intel_context *ce)
3375−{
3376− check_redzone((void *)ce->lrc_reg_state - LRC_STATE_OFFSET,
3377− ce->engine);
3378−}
3379−
3380−static void execlists_context_post_unpin(struct intel_context *ce)
3381−{
3382− i915_gem_object_unpin_map(ce->state->obj);
3383−}
3384−
3385−static u32 *
3386−gen12_emit_timestamp_wa(const struct intel_context *ce, u32 *cs)
3387−{
3388− *cs++ = MI_LOAD_REGISTER_MEM_GEN8 |
3389− MI_SRM_LRM_GLOBAL_GTT |
3390− MI_LRI_LRM_CS_MMIO;
3391− *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3392− *cs++ = i915_ggtt_offset(ce->state) + LRC_STATE_OFFSET +
3393− CTX_TIMESTAMP * sizeof(u32);
3394− *cs++ = 0;
3395−
3396− *cs++ = MI_LOAD_REGISTER_REG |
3397− MI_LRR_SOURCE_CS_MMIO |
3398− MI_LRI_LRM_CS_MMIO;
3399− *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3400− *cs++ = i915_mmio_reg_offset(RING_CTX_TIMESTAMP(0));
3401−
3402− *cs++ = MI_LOAD_REGISTER_REG |
3403− MI_LRR_SOURCE_CS_MMIO |
3404− MI_LRI_LRM_CS_MMIO;
3405− *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3406− *cs++ = i915_mmio_reg_offset(RING_CTX_TIMESTAMP(0));
3407−
3408− return cs;
3409−}
3410−
3411−static u32 *
3412−gen12_emit_restore_scratch(const struct intel_context *ce, u32 *cs)
3413−{
3414− GEM_BUG_ON(lrc_ring_gpr0(ce->engine) == -1);
3415−
3416− *cs++ = MI_LOAD_REGISTER_MEM_GEN8 |
3417− MI_SRM_LRM_GLOBAL_GTT |
3418− MI_LRI_LRM_CS_MMIO;
3419− *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3420− *cs++ = i915_ggtt_offset(ce->state) + LRC_STATE_OFFSET +
3421− (lrc_ring_gpr0(ce->engine) + 1) * sizeof(u32);
3422− *cs++ = 0;
3423−
3424− return cs;
3425−}
3426−
3427−static u32 *
3428−gen12_emit_cmd_buf_wa(const struct intel_context *ce, u32 *cs)
3429−{
3430− GEM_BUG_ON(lrc_ring_cmd_buf_cctl(ce->engine) == -1);
3431−
3432− *cs++ = MI_LOAD_REGISTER_MEM_GEN8 |
3433− MI_SRM_LRM_GLOBAL_GTT |
3434− MI_LRI_LRM_CS_MMIO;
3435− *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3436− *cs++ = i915_ggtt_offset(ce->state) + LRC_STATE_OFFSET +
3437− (lrc_ring_cmd_buf_cctl(ce->engine) + 1) * sizeof(u32);
3438− *cs++ = 0;
3439−
3440− *cs++ = MI_LOAD_REGISTER_REG |
3441− MI_LRR_SOURCE_CS_MMIO |
3442− MI_LRI_LRM_CS_MMIO;
3443− *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3444− *cs++ = i915_mmio_reg_offset(RING_CMD_BUF_CCTL(0));
3445−
3446− return cs;
3447−}
3448−
3449−static u32 *
3450−gen12_emit_indirect_ctx_rcs(const struct intel_context *ce, u32 *cs)
3451−{
3452− cs = gen12_emit_timestamp_wa(ce, cs);
3453− cs = gen12_emit_cmd_buf_wa(ce, cs);
3454− cs = gen12_emit_restore_scratch(ce, cs);
3455−
3456− return cs;
3457−}
3458−
3459−static u32 *
3460−gen12_emit_indirect_ctx_xcs(const struct intel_context *ce, u32 *cs)
3461−{
3462− cs = gen12_emit_timestamp_wa(ce, cs);
3463− cs = gen12_emit_restore_scratch(ce, cs);
3464−
3465− return cs;
3466−}
3467−
3468−static inline u32 context_wa_bb_offset(const struct intel_context *ce)
3469−{
3470− return PAGE_SIZE * ce->wa_bb_page;
3471−}
3472−
3473−static u32 *context_indirect_bb(const struct intel_context *ce)
3474−{
3475− void *ptr;
3476−
3477− GEM_BUG_ON(!ce->wa_bb_page);
3478−
3479− ptr = ce->lrc_reg_state;
3480− ptr -= LRC_STATE_OFFSET; /* back to start of context image */
3481− ptr += context_wa_bb_offset(ce);
3482−
3483− return ptr;
3484−}
3485−
3486−static void
3487−setup_indirect_ctx_bb(const struct intel_context *ce,
3488− const struct intel_engine_cs *engine,
3489− u32 *(*emit)(const struct intel_context *, u32 *))
3490−{
3491− u32 * const start = context_indirect_bb(ce);
3492− u32 *cs;
3493−
3494− cs = emit(ce, start);
3495− GEM_BUG_ON(cs - start > I915_GTT_PAGE_SIZE / sizeof(*cs));
3496− while ((unsigned long)cs % CACHELINE_BYTES)
3497− *cs++ = MI_NOOP;
3498−
3499− lrc_ring_setup_indirect_ctx(ce->lrc_reg_state, engine,
3500− i915_ggtt_offset(ce->state) +
3501− context_wa_bb_offset(ce),
3502− (cs - start) * sizeof(*cs));
3503−}
3504−
3505−static void
3506−__execlists_update_reg_state(const struct intel_context *ce,
3507− const struct intel_engine_cs *engine,
3508− u32 head)
3509−{
3510− struct intel_ring *ring = ce->ring;
3511− u32 *regs = ce->lrc_reg_state;
3512−
3513− GEM_BUG_ON(!intel_ring_offset_valid(ring, head));
3514− GEM_BUG_ON(!intel_ring_offset_valid(ring, ring->tail));
3515−
3516− regs[CTX_RING_START] = i915_ggtt_offset(ring->vma);
3517− regs[CTX_RING_HEAD] = head;
3518− regs[CTX_RING_TAIL] = ring->tail;
3519− regs[CTX_RING_CTL] = RING_CTL_SIZE(ring->size) | RING_VALID;
3520−
3521− /* RPCS */
3522− if (engine->class == RENDER_CLASS) {
3523− regs[CTX_R_PWR_CLK_STATE] =
3524− intel_sseu_make_rpcs(engine->gt, &ce->sseu);
3525−
3526− i915_oa_init_reg_state(ce, engine);
3527− }
3528−
3529− if (ce->wa_bb_page) {
3530− u32 *(*fn)(const struct intel_context *ce, u32 *cs);
3531−
3532− fn = gen12_emit_indirect_ctx_xcs;
3533− if (ce->engine->class == RENDER_CLASS)
3534− fn = gen12_emit_indirect_ctx_rcs;
3535−
3536− /* Mutually exclusive wrt to global indirect bb */
3537− GEM_BUG_ON(engine->wa_ctx.indirect_ctx.size);
3538− setup_indirect_ctx_bb(ce, engine, fn);
3539− }
3540−}
3541−
3542−static int
3543−execlists_context_pre_pin(struct intel_context *ce,
3544− struct i915_gem_ww_ctx *ww, void **vaddr)
3545−{
3546− GEM_BUG_ON(!ce->state);
3547− GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
3548−
3549− *vaddr = i915_gem_object_pin_map(ce->state->obj,
3550− i915_coherent_map_type(ce->engine->i915) |
3551− I915_MAP_OVERRIDE);
3552−
3553− return PTR_ERR_OR_ZERO(*vaddr);
3554−}
3555−
3556−static int
3557−__execlists_context_pin(struct intel_context *ce,
3558− struct intel_engine_cs *engine,
3559− void *vaddr)
3560−{
3561− ce->lrc.lrca = lrc_descriptor(ce, engine) | CTX_DESC_FORCE_RESTORE;
3562− ce->lrc_reg_state = vaddr + LRC_STATE_OFFSET;
3563− __execlists_update_reg_state(ce, engine, ce->ring->tail);
3564−
3565− return 0;
3566−}
3567−
3568−static int execlists_context_pin(struct intel_context *ce, void *vaddr)
3569−{
3570− return __execlists_context_pin(ce, ce->engine, vaddr);
3571−}
3572−
3573−static int execlists_context_alloc(struct intel_context *ce)
3574−{
3575− return __execlists_context_alloc(ce, ce->engine);
3576−}
3577−
3578−static void execlists_context_reset(struct intel_context *ce)
3579−{
3580− CE_TRACE(ce, "reset\n");
3581− GEM_BUG_ON(!intel_context_is_pinned(ce));
3582−
3583− intel_ring_reset(ce->ring, ce->ring->emit);
3584−
3585− /* Scrub away the garbage */
3586− execlists_init_reg_state(ce->lrc_reg_state,
3587− ce, ce->engine, ce->ring, true);
3588− __execlists_update_reg_state(ce, ce->engine, ce->ring->tail);
3589−
3590− ce->lrc.desc |= CTX_DESC_FORCE_RESTORE;
3591−}
3592−
3593−static const struct intel_context_ops execlists_context_ops = {
3594− .alloc = execlists_context_alloc,
3595−
3596− .pre_pin = execlists_context_pre_pin,
3597− .pin = execlists_context_pin,
3598− .unpin = execlists_context_unpin,
3599− .post_unpin = execlists_context_post_unpin,
3600−
3601− .enter = intel_context_enter_engine,
3602− .exit = intel_context_exit_engine,
3603−
3604− .reset = execlists_context_reset,
3605− .destroy = execlists_context_destroy,
3606−};
3607−
3608−static u32 hwsp_offset(const struct i915_request *rq)
3609−{
3610− const struct intel_timeline_cacheline *cl;
3611−
3612− /* Before the request is executed, the timeline/cachline is fixed */
3613−
3614− cl = rcu_dereference_protected(rq->hwsp_cacheline, 1);
3615− if (cl)
3616− return cl->ggtt_offset;
3617−
3618− return rcu_dereference_protected(rq->timeline, 1)->hwsp_offset;
3619−}
3620−
3621−static int gen8_emit_init_breadcrumb(struct i915_request *rq)
3622−{
3623− u32 *cs;
3624−
3625− GEM_BUG_ON(i915_request_has_initial_breadcrumb(rq));
3626− if (!i915_request_timeline(rq)->has_initial_breadcrumb)
3627− return 0;
3628−
3629− cs = intel_ring_begin(rq, 6);
3630− if (IS_ERR(cs))
3631− return PTR_ERR(cs);
3632−
3633− /*
3634− * Check if we have been preempted before we even get started.
3635− *
3636− * After this point i915_request_started() reports true, even if
3637− * we get preempted and so are no longer running.
3638− */
3639− *cs++ = MI_ARB_CHECK;
3640− *cs++ = MI_NOOP;
3641−
3642− *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
3643− *cs++ = hwsp_offset(rq);
3644− *cs++ = 0;
3645− *cs++ = rq->fence.seqno - 1;
3646−
3647− intel_ring_advance(rq, cs);
3648−
3649− /* Record the updated position of the request's payload */
3650− rq->infix = intel_ring_offset(rq, cs);
3651−
3652− __set_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags);
3653−
3654− return 0;
3655−}
3656−
3657−static int emit_pdps(struct i915_request *rq)
3658−{
3659− const struct intel_engine_cs * const engine = rq->engine;
3660− struct i915_ppgtt * const ppgtt = i915_vm_to_ppgtt(rq->context->vm);
3661− int err, i;
3662− u32 *cs;
3663−
3664− GEM_BUG_ON(intel_vgpu_active(rq->engine->i915));
3665−
3666− /*
3667− * Beware ye of the dragons, this sequence is magic!
3668− *
3669− * Small changes to this sequence can cause anything from
3670− * GPU hangs to forcewake errors and machine lockups!
3671− */
3672−
3673− /* Flush any residual operations from the context load */
3674− err = engine->emit_flush(rq, EMIT_FLUSH);
3675− if (err)
3676− return err;
3677−
3678− /* Magic required to prevent forcewake errors! */
3679− err = engine->emit_flush(rq, EMIT_INVALIDATE);
3680− if (err)
3681− return err;
3682−
3683− cs = intel_ring_begin(rq, 4 * GEN8_3LVL_PDPES + 2);
3684− if (IS_ERR(cs))
3685− return PTR_ERR(cs);
3686−
3687− /* Ensure the LRI have landed before we invalidate & continue */
3688− *cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES) | MI_LRI_FORCE_POSTED;
3689− for (i = GEN8_3LVL_PDPES; i--; ) {
3690− const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
3691− u32 base = engine->mmio_base;
3692−
3693− *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i));
3694− *cs++ = upper_32_bits(pd_daddr);
3695− *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i));
3696− *cs++ = lower_32_bits(pd_daddr);
3697− }
3698− *cs++ = MI_NOOP;
3699−
3700− intel_ring_advance(rq, cs);
3701−
3702− return 0;
3703−}
3704−
3705−static int execlists_request_alloc(struct i915_request *request)
3706−{
3707− int ret;
3708−
3709− GEM_BUG_ON(!intel_context_is_pinned(request->context));
3710−
3711− /*
3712− * Flush enough space to reduce the likelihood of waiting after
3713− * we start building the request - in which case we will just
3714− * have to repeat work.
3715− */
3716− request->reserved_space += EXECLISTS_REQUEST_SIZE;
3717−
3718− /*
3719− * Note that after this point, we have committed to using
3720− * this request as it is being used to both track the
3721− * state of engine initialisation and liveness of the
3722− * golden renderstate above. Think twice before you try
3723− * to cancel/unwind this request now.
3724− */
3725−
3726− if (!i915_vm_is_4lvl(request->context->vm)) {
3727− ret = emit_pdps(request);
3728− if (ret)
3729− return ret;
3730− }
3731−
3732− /* Unconditionally invalidate GPU caches and TLBs. */
3733− ret = request->engine->emit_flush(request, EMIT_INVALIDATE);
3734− if (ret)
3735− return ret;
3736−
3737− request->reserved_space -= EXECLISTS_REQUEST_SIZE;
3738− return 0;
3739−}
3740−
3741−/*
3742− * In this WA we need to set GEN8_L3SQCREG4[21:21] and reset it after
3743− * PIPE_CONTROL instruction. This is required for the flush to happen correctly
3744− * but there is a slight complication as this is applied in WA batch where the
3745− * values are only initialized once so we cannot take register value at the
3746− * beginning and reuse it further; hence we save its value to memory, upload a
3747− * constant value with bit21 set and then we restore it back with the saved value.
3748− * To simplify the WA, a constant value is formed by using the default value
3749− * of this register. This shouldn't be a problem because we are only modifying
3750− * it for a short period and this batch in non-premptible. We can ofcourse
3751− * use additional instructions that read the actual value of the register
3752− * at that time and set our bit of interest but it makes the WA complicated.
3753− *
3754− * This WA is also required for Gen9 so extracting as a function avoids
3755− * code duplication.
3756− */
3757−static u32 *
3758−gen8_emit_flush_coherentl3_wa(struct intel_engine_cs *engine, u32 *batch)
3759−{
3760− /* NB no one else is allowed to scribble over scratch + 256! */
3761− *batch++ = MI_STORE_REGISTER_MEM_GEN8 | MI_SRM_LRM_GLOBAL_GTT;
3762− *batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
3763− *batch++ = intel_gt_scratch_offset(engine->gt,
3764− INTEL_GT_SCRATCH_FIELD_COHERENTL3_WA);
3765− *batch++ = 0;
3766−
3767− *batch++ = MI_LOAD_REGISTER_IMM(1);
3768− *batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
3769− *batch++ = 0x40400000 | GEN8_LQSC_FLUSH_COHERENT_LINES;
3770−
3771− batch = gen8_emit_pipe_control(batch,
3772− PIPE_CONTROL_CS_STALL |
3773− PIPE_CONTROL_DC_FLUSH_ENABLE,
3774− 0);
3775−
3776− *batch++ = MI_LOAD_REGISTER_MEM_GEN8 | MI_SRM_LRM_GLOBAL_GTT;
3777− *batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
3778− *batch++ = intel_gt_scratch_offset(engine->gt,
3779− INTEL_GT_SCRATCH_FIELD_COHERENTL3_WA);
3780− *batch++ = 0;
3781−
3782− return batch;
3783−}
3784−
3785−/*
3786− * Typically we only have one indirect_ctx and per_ctx batch buffer which are
3787− * initialized at the beginning and shared across all contexts but this field
3788− * helps us to have multiple batches at different offsets and select them based
3789− * on a criteria. At the moment this batch always start at the beginning of the page
3790− * and at this point we don't have multiple wa_ctx batch buffers.
3791− *
3792− * The number of WA applied are not known at the beginning; we use this field
3793− * to return the no of DWORDS written.
3794− *
3795− * It is to be noted that this batch does not contain MI_BATCH_BUFFER_END
3796− * so it adds NOOPs as padding to make it cacheline aligned.
3797− * MI_BATCH_BUFFER_END will be added to perctx batch and both of them together
3798− * makes a complete batch buffer.
3799− */
3800−static u32 *gen8_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
3801−{
3802− /* WaDisableCtxRestoreArbitration:bdw,chv */
3803− *batch++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
3804−
3805− /* WaFlushCoherentL3CacheLinesAtContextSwitch:bdw */
3806− if (IS_BROADWELL(engine->i915))
3807− batch = gen8_emit_flush_coherentl3_wa(engine, batch);
3808−
3809− /* WaClearSlmSpaceAtContextSwitch:bdw,chv */
3810− /* Actual scratch location is at 128 bytes offset */
3811− batch = gen8_emit_pipe_control(batch,
3812− PIPE_CONTROL_FLUSH_L3 |
3813− PIPE_CONTROL_STORE_DATA_INDEX |
3814− PIPE_CONTROL_CS_STALL |
3815− PIPE_CONTROL_QW_WRITE,
3816− LRC_PPHWSP_SCRATCH_ADDR);
3817−
3818− *batch++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
3819−
3820− /* Pad to end of cacheline */
3821− while ((unsigned long)batch % CACHELINE_BYTES)
3822− *batch++ = MI_NOOP;
3823−
3824− /*
3825− * MI_BATCH_BUFFER_END is not required in Indirect ctx BB because
3826− * execution depends on the length specified in terms of cache lines
3827− * in the register CTX_RCS_INDIRECT_CTX
3828− */
3829−
3830− return batch;
3831−}
3832−
3833−struct lri {
3834− i915_reg_t reg;
3835− u32 value;
3836−};
3837−
3838−static u32 *emit_lri(u32 *batch, const struct lri *lri, unsigned int count)
3839−{
3840− GEM_BUG_ON(!count || count > 63);
3841−
3842− *batch++ = MI_LOAD_REGISTER_IMM(count);
3843− do {
3844− *batch++ = i915_mmio_reg_offset(lri->reg);
3845− *batch++ = lri->value;
3846− } while (lri++, --count);
3847− *batch++ = MI_NOOP;
3848−
3849− return batch;
3850−}
3851−
3852−static u32 *gen9_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
3853−{
3854− static const struct lri lri[] = {
3855− /* WaDisableGatherAtSetShaderCommonSlice:skl,bxt,kbl,glk */
3856− {
3857− COMMON_SLICE_CHICKEN2,
3858− __MASKED_FIELD(GEN9_DISABLE_GATHER_AT_SET_SHADER_COMMON_SLICE,
3859− 0),
3860− },
3861−
3862− /* BSpec: 11391 */
3863− {
3864− FF_SLICE_CHICKEN,
3865− __MASKED_FIELD(FF_SLICE_CHICKEN_CL_PROVOKING_VERTEX_FIX,
3866− FF_SLICE_CHICKEN_CL_PROVOKING_VERTEX_FIX),
3867− },
3868−
3869− /* BSpec: 11299 */
3870− {
3871− _3D_CHICKEN3,
3872− __MASKED_FIELD(_3D_CHICKEN_SF_PROVOKING_VERTEX_FIX,
3873− _3D_CHICKEN_SF_PROVOKING_VERTEX_FIX),
3874− }
3875− };
3876−
3877− *batch++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
3878−
3879− /* WaFlushCoherentL3CacheLinesAtContextSwitch:skl,bxt,glk */
3880− batch = gen8_emit_flush_coherentl3_wa(engine, batch);
3881−
3882− /* WaClearSlmSpaceAtContextSwitch:skl,bxt,kbl,glk,cfl */
3883− batch = gen8_emit_pipe_control(batch,
3884− PIPE_CONTROL_FLUSH_L3 |
3885− PIPE_CONTROL_STORE_DATA_INDEX |
3886− PIPE_CONTROL_CS_STALL |
3887− PIPE_CONTROL_QW_WRITE,
3888− LRC_PPHWSP_SCRATCH_ADDR);
3889−
3890− batch = emit_lri(batch, lri, ARRAY_SIZE(lri));
3891−
3892− /* WaMediaPoolStateCmdInWABB:bxt,glk */
3893− if (HAS_POOLED_EU(engine->i915)) {
3894− /*
3895− * EU pool configuration is setup along with golden context
3896− * during context initialization. This value depends on
3897− * device type (2x6 or 3x6) and needs to be updated based
3898− * on which subslice is disabled especially for 2x6
3899− * devices, however it is safe to load default
3900− * configuration of 3x6 device instead of masking off
3901− * corresponding bits because HW ignores bits of a disabled
3902− * subslice and drops down to appropriate config. Please
3903− * see render_state_setup() in i915_gem_render_state.c for
3904− * possible configurations, to avoid duplication they are
3905− * not shown here again.
3906− */
3907− *batch++ = GEN9_MEDIA_POOL_STATE;
3908− *batch++ = GEN9_MEDIA_POOL_ENABLE;
3909− *batch++ = 0x00777000;
3910− *batch++ = 0;
3911− *batch++ = 0;
3912− *batch++ = 0;
3913− }
3914−
3915− *batch++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
3916−
3917− /* Pad to end of cacheline */
3918− while ((unsigned long)batch % CACHELINE_BYTES)
3919− *batch++ = MI_NOOP;
3920−
3921− return batch;
3922−}
3923−
3924−static u32 *
3925−gen10_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
3926−{
3927− int i;
3928−
3929− /*
3930− * WaPipeControlBefore3DStateSamplePattern: cnl
3931− *
3932− * Ensure the engine is idle prior to programming a
3933− * 3DSTATE_SAMPLE_PATTERN during a context restore.
3934− */
3935− batch = gen8_emit_pipe_control(batch,
3936− PIPE_CONTROL_CS_STALL,
3937− 0);
3938− /*
3939− * WaPipeControlBefore3DStateSamplePattern says we need 4 dwords for
3940− * the PIPE_CONTROL followed by 12 dwords of 0x0, so 16 dwords in
3941− * total. However, a PIPE_CONTROL is 6 dwords long, not 4, which is
3942− * confusing. Since gen8_emit_pipe_control() already advances the
3943− * batch by 6 dwords, we advance the other 10 here, completing a
3944− * cacheline. It's not clear if the workaround requires this padding
3945− * before other commands, or if it's just the regular padding we would
3946− * already have for the workaround bb, so leave it here for now.
3947− */
3948− for (i = 0; i < 10; i++)
3949− *batch++ = MI_NOOP;
3950−
3951− /* Pad to end of cacheline */
3952− while ((unsigned long)batch % CACHELINE_BYTES)
3953− *batch++ = MI_NOOP;
3954−
3955− return batch;
3956−}
3957−
3958−#define CTX_WA_BB_OBJ_SIZE (PAGE_SIZE)
3959−
3960−static int lrc_setup_wa_ctx(struct intel_engine_cs *engine)
3961−{
3962− struct drm_i915_gem_object *obj;
3963− struct i915_vma *vma;
3964− int err;
3965−
3966− obj = i915_gem_object_create_shmem(engine->i915, CTX_WA_BB_OBJ_SIZE);
3967− if (IS_ERR(obj))
3968− return PTR_ERR(obj);
3969−
3970− vma = i915_vma_instance(obj, &engine->gt->ggtt->vm, NULL);
3971− if (IS_ERR(vma)) {
3972− err = PTR_ERR(vma);
3973− goto err;
3974− }
3975−
3976− err = i915_ggtt_pin(vma, NULL, 0, PIN_HIGH);
3977− if (err)
3978− goto err;
3979−
3980− engine->wa_ctx.vma = vma;
3981− return 0;
3982−
3983−err:
3984− i915_gem_object_put(obj);
3985− return err;
3986−}
3987−
3988−static void lrc_destroy_wa_ctx(struct intel_engine_cs *engine)
3989−{
3990− i915_vma_unpin_and_release(&engine->wa_ctx.vma, 0);
3991−}
3992−
3993−typedef u32 *(*wa_bb_func_t)(struct intel_engine_cs *engine, u32 *batch);
3994−
3995−static int intel_init_workaround_bb(struct intel_engine_cs *engine)
3996−{
3997− struct i915_ctx_workarounds *wa_ctx = &engine->wa_ctx;
3998− struct i915_wa_ctx_bb *wa_bb[2] = { &wa_ctx->indirect_ctx,
3999− &wa_ctx->per_ctx };
4000− wa_bb_func_t wa_bb_fn[2];
4001− void *batch, *batch_ptr;
4002− unsigned int i;
4003− int ret;
4004−
4005− if (engine->class != RENDER_CLASS)
4006− return 0;
4007−
4008− switch (INTEL_GEN(engine->i915)) {
4009− case 12:
4010− case 11:
4011− return 0;
4012− case 10:
4013− wa_bb_fn[0] = gen10_init_indirectctx_bb;
4014− wa_bb_fn[1] = NULL;
4015− break;
4016− case 9:
4017− wa_bb_fn[0] = gen9_init_indirectctx_bb;
4018− wa_bb_fn[1] = NULL;
4019− break;
4020− case 8:
4021− wa_bb_fn[0] = gen8_init_indirectctx_bb;
4022− wa_bb_fn[1] = NULL;
4023− break;
4024− default:
4025− MISSING_CASE(INTEL_GEN(engine->i915));
4026− return 0;
4027− }
4028−
4029− ret = lrc_setup_wa_ctx(engine);
4030− if (ret) {
4031− drm_dbg(&engine->i915->drm,
4032− "Failed to setup context WA page: %d\n", ret);
4033− return ret;
4034− }
4035−
4036− batch = i915_gem_object_pin_map(wa_ctx->vma->obj, I915_MAP_WB);
4037−
4038− /*
4039− * Emit the two workaround batch buffers, recording the offset from the
4040− * start of the workaround batch buffer object for each and their
4041− * respective sizes.
4042− */
4043− batch_ptr = batch;
4044− for (i = 0; i < ARRAY_SIZE(wa_bb_fn); i++) {
4045− wa_bb[i]->offset = batch_ptr - batch;
4046− if (GEM_DEBUG_WARN_ON(!IS_ALIGNED(wa_bb[i]->offset,
4047− CACHELINE_BYTES))) {
4048− ret = -EINVAL;
4049− break;
4050− }
4051− if (wa_bb_fn[i])
4052− batch_ptr = wa_bb_fn[i](engine, batch_ptr);
4053− wa_bb[i]->size = batch_ptr - (batch + wa_bb[i]->offset);
4054− }
4055− GEM_BUG_ON(batch_ptr - batch > CTX_WA_BB_OBJ_SIZE);
4056−
4057− __i915_gem_object_flush_map(wa_ctx->vma->obj, 0, batch_ptr - batch);
4058− __i915_gem_object_release_map(wa_ctx->vma->obj);
4059− if (ret)
4060− lrc_destroy_wa_ctx(engine);
4061−
4062− return ret;
4063−}
4064−
4065−static void reset_csb_pointers(struct intel_engine_cs *engine)
4066−{
4067− struct intel_engine_execlists * const execlists = &engine->execlists;
4068− const unsigned int reset_value = execlists->csb_size - 1;
4069−
4070− ring_set_paused(engine, 0);
4071−
4072− /*
4073− * Sometimes Icelake forgets to reset its pointers on a GPU reset.
4074− * Bludgeon them with a mmio update to be sure.
4075− */
4076− ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
4077− 0xffff << 16 | reset_value << 8 | reset_value);
4078− ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
4079−
4080− /*
4081− * After a reset, the HW starts writing into CSB entry [0]. We
4082− * therefore have to set our HEAD pointer back one entry so that
4083− * the *first* entry we check is entry 0. To complicate this further,
4084− * as we don't wait for the first interrupt after reset, we have to
4085− * fake the HW write to point back to the last entry so that our
4086− * inline comparison of our cached head position against the last HW
4087− * write works even before the first interrupt.
4088− */
4089− execlists->csb_head = reset_value;
4090− WRITE_ONCE(*execlists->csb_write, reset_value);
4091− wmb(); /* Make sure this is visible to HW (paranoia?) */
4092−
4093− /* Check that the GPU does indeed update the CSB entries! */
4094− memset(execlists->csb_status, -1, (reset_value + 1) * sizeof(u64));
4095− invalidate_csb_entries(&execlists->csb_status[0],
4096− &execlists->csb_status[reset_value]);
4097−
4098− /* Once more for luck and our trusty paranoia */
4099− ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
4100− 0xffff << 16 | reset_value << 8 | reset_value);
4101− ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
4102−
4103− GEM_BUG_ON(READ_ONCE(*execlists->csb_write) != reset_value);
4104−}
4105−
4106−static void execlists_sanitize(struct intel_engine_cs *engine)
4107−{
4108− GEM_BUG_ON(execlists_active(&engine->execlists));
4109−
4110− /*
4111− * Poison residual state on resume, in case the suspend didn't!
4112− *
4113− * We have to assume that across suspend/resume (or other loss
4114− * of control) that the contents of our pinned buffers has been
4115− * lost, replaced by garbage. Since this doesn't always happen,
4116− * let's poison such state so that we more quickly spot when
4117− * we falsely assume it has been preserved.
4118− */
4119− if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
4120− memset(engine->status_page.addr, POISON_INUSE, PAGE_SIZE);
4121−
4122− reset_csb_pointers(engine);
4123−
4124− /*
4125− * The kernel_context HWSP is stored in the status_page. As above,
4126− * that may be lost on resume/initialisation, and so we need to
4127− * reset the value in the HWSP.
4128− */
4129− intel_timeline_reset_seqno(engine->kernel_context->timeline);
4130−
4131− /* And scrub the dirty cachelines for the HWSP */
4132− clflush_cache_range(engine->status_page.addr, PAGE_SIZE);
4133−}
4134−
4135−static void enable_error_interrupt(struct intel_engine_cs *engine)
4136−{
4137− u32 status;
4138−
4139− engine->execlists.error_interrupt = 0;
4140− ENGINE_WRITE(engine, RING_EMR, ~0u);
4141− ENGINE_WRITE(engine, RING_EIR, ~0u); /* clear all existing errors */
4142−
4143− status = ENGINE_READ(engine, RING_ESR);
4144− if (unlikely(status)) {
4145− drm_err(&engine->i915->drm,
4146− "engine '%s' resumed still in error: %08x\n",
4147− engine->name, status);
4148− __intel_gt_reset(engine->gt, engine->mask);
4149− }
4150−
4151− /*
4152− * On current gen8+, we have 2 signals to play with
4153− *
4154− * - I915_ERROR_INSTUCTION (bit 0)
4155− *
4156− * Generate an error if the command parser encounters an invalid
4157− * instruction
4158− *
4159− * This is a fatal error.
4160− *
4161− * - CP_PRIV (bit 2)
4162− *
4163− * Generate an error on privilege violation (where the CP replaces
4164− * the instruction with a no-op). This also fires for writes into
4165− * read-only scratch pages.
4166− *
4167− * This is a non-fatal error, parsing continues.
4168− *
4169− * * there are a few others defined for odd HW that we do not use
4170− *
4171− * Since CP_PRIV fires for cases where we have chosen to ignore the
4172− * error (as the HW is validating and suppressing the mistakes), we
4173− * only unmask the instruction error bit.
4174− */
4175− ENGINE_WRITE(engine, RING_EMR, ~I915_ERROR_INSTRUCTION);
4176−}
4177−
4178−static void enable_execlists(struct intel_engine_cs *engine)
4179−{
4180− u32 mode;
4181−
4182− assert_forcewakes_active(engine->uncore, FORCEWAKE_ALL);
4183−
4184− intel_engine_set_hwsp_writemask(engine, ~0u); /* HWSTAM */
4185−
4186− if (INTEL_GEN(engine->i915) >= 11)
4187− mode = _MASKED_BIT_ENABLE(GEN11_GFX_DISABLE_LEGACY_MODE);
4188− else
4189− mode = _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE);
4190− ENGINE_WRITE_FW(engine, RING_MODE_GEN7, mode);
4191−
4192− ENGINE_WRITE_FW(engine, RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
4193−
4194− ENGINE_WRITE_FW(engine,
4195− RING_HWS_PGA,
4196− i915_ggtt_offset(engine->status_page.vma));
4197− ENGINE_POSTING_READ(engine, RING_HWS_PGA);
4198−
4199− enable_error_interrupt(engine);
4200−
4201− engine->context_tag = GENMASK(BITS_PER_LONG - 2, 0);
4202−}
4203−
4204−static bool unexpected_starting_state(struct intel_engine_cs *engine)
4205−{
4206− bool unexpected = false;
4207−
4208− if (ENGINE_READ_FW(engine, RING_MI_MODE) & STOP_RING) {
4209− drm_dbg(&engine->i915->drm,
4210− "STOP_RING still set in RING_MI_MODE\n");
4211− unexpected = true;
4212− }
4213−
4214− return unexpected;
4215−}
4216−
4217−static int execlists_resume(struct intel_engine_cs *engine)
4218−{
4219− intel_mocs_init_engine(engine);
4220−
4221− intel_breadcrumbs_reset(engine->breadcrumbs);
4222−
4223− if (GEM_SHOW_DEBUG() && unexpected_starting_state(engine)) {
4224− struct drm_printer p = drm_debug_printer(__func__);
4225−
4226− intel_engine_dump(engine, &p, NULL);
4227− }
4228−
4229− enable_execlists(engine);
4230−
4231− return 0;
4232−}
4233−
4234−static void execlists_reset_prepare(struct intel_engine_cs *engine)
4235−{
4236− struct intel_engine_execlists * const execlists = &engine->execlists;
4237− unsigned long flags;
4238−
4239− ENGINE_TRACE(engine, "depth<-%d\n",
4240− atomic_read(&execlists->tasklet.count));
4241−
4242− /*
4243− * Prevent request submission to the hardware until we have
4244− * completed the reset in i915_gem_reset_finish(). If a request
4245− * is completed by one engine, it may then queue a request
4246− * to a second via its execlists->tasklet *just* as we are
4247− * calling engine->resume() and also writing the ELSP.
4248− * Turning off the execlists->tasklet until the reset is over
4249− * prevents the race.
4250− */
4251− __tasklet_disable_sync_once(&execlists->tasklet);
4252− GEM_BUG_ON(!reset_in_progress(execlists));
4253−
4254− /* And flush any current direct submission. */
4255− spin_lock_irqsave(&engine->active.lock, flags);
4256− spin_unlock_irqrestore(&engine->active.lock, flags);
4257−
4258− /*
4259− * We stop engines, otherwise we might get failed reset and a
4260− * dead gpu (on elk). Also as modern gpu as kbl can suffer
4261− * from system hang if batchbuffer is progressing when
4262− * the reset is issued, regardless of READY_TO_RESET ack.
4263− * Thus assume it is best to stop engines on all gens
4264− * where we have a gpu reset.
4265− *
4266− * WaKBLVECSSemaphoreWaitPoll:kbl (on ALL_ENGINES)
4267− *
4268− * FIXME: Wa for more modern gens needs to be validated
4269− */
4270− ring_set_paused(engine, 1);
4271− intel_engine_stop_cs(engine);
4272−
4273− engine->execlists.reset_ccid = active_ccid(engine);
4274−}
4275−
4276−static void __reset_stop_ring(u32 *regs, const struct intel_engine_cs *engine)
4277−{
4278− int x;
4279−
4280− x = lrc_ring_mi_mode(engine);
4281− if (x != -1) {
4282− regs[x + 1] &= ~STOP_RING;
4283− regs[x + 1] |= STOP_RING << 16;
4284− }
4285−}
4286−
4287−static void __execlists_reset_reg_state(const struct intel_context *ce,
4288− const struct intel_engine_cs *engine)
4289−{
4290− u32 *regs = ce->lrc_reg_state;
4291−
4292− __reset_stop_ring(regs, engine);
4293−}
4294−
4295−static void __execlists_reset(struct intel_engine_cs *engine, bool stalled)
4296−{
4297− struct intel_engine_execlists * const execlists = &engine->execlists;
4298− struct intel_context *ce;
4299− struct i915_request *rq;
4300− u32 head;
4301−
4302− mb(); /* paranoia: read the CSB pointers from after the reset */
4303− clflush(execlists->csb_write);
4304− mb();
4305−
4306− process_csb(engine); /* drain preemption events */
4307−
4308− /* Following the reset, we need to reload the CSB read/write pointers */
4309− reset_csb_pointers(engine);
4310−
4311− /*
4312− * Save the currently executing context, even if we completed
4313− * its request, it was still running at the time of the
4314− * reset and will have been clobbered.
4315− */
4316− rq = active_context(engine, engine->execlists.reset_ccid);
4317− if (!rq)
4318− goto unwind;
4319−
4320− ce = rq->context;
4321− GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
4322−
4323− if (i915_request_completed(rq)) {
4324− /* Idle context; tidy up the ring so we can restart afresh */
4325− head = intel_ring_wrap(ce->ring, rq->tail);
4326− goto out_replay;
4327− }
4328−
4329− /* We still have requests in-flight; the engine should be active */
4330− GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
4331−
4332− /* Context has requests still in-flight; it should not be idle! */
4333− GEM_BUG_ON(i915_active_is_idle(&ce->active));
4334−
4335− rq = active_request(ce->timeline, rq);
4336− head = intel_ring_wrap(ce->ring, rq->head);
4337− GEM_BUG_ON(head == ce->ring->tail);
4338−
4339− /*
4340− * If this request hasn't started yet, e.g. it is waiting on a
4341− * semaphore, we need to avoid skipping the request or else we
4342− * break the signaling chain. However, if the context is corrupt
4343− * the request will not restart and we will be stuck with a wedged
4344− * device. It is quite often the case that if we issue a reset
4345− * while the GPU is loading the context image, that the context
4346− * image becomes corrupt.
4347− *
4348− * Otherwise, if we have not started yet, the request should replay
4349− * perfectly and we do not need to flag the result as being erroneous.
4350− */
4351− if (!i915_request_started(rq))
4352− goto out_replay;
4353−
4354− /*
4355− * If the request was innocent, we leave the request in the ELSP
4356− * and will try to replay it on restarting. The context image may
4357− * have been corrupted by the reset, in which case we may have
4358− * to service a new GPU hang, but more likely we can continue on
4359− * without impact.
4360− *
4361− * If the request was guilty, we presume the context is corrupt
4362− * and have to at least restore the RING register in the context
4363− * image back to the expected values to skip over the guilty request.
4364− */
4365− __i915_request_reset(rq, stalled);
4366−
4367− /*
4368− * We want a simple context + ring to execute the breadcrumb update.
4369− * We cannot rely on the context being intact across the GPU hang,
4370− * so clear it and rebuild just what we need for the breadcrumb.
4371− * All pending requests for this context will be zapped, and any
4372− * future request will be after userspace has had the opportunity
4373− * to recreate its own state.
4374− */
4375−out_replay:
4376− ENGINE_TRACE(engine, "replay {head:%04x, tail:%04x}\n",
4377− head, ce->ring->tail);
4378− __execlists_reset_reg_state(ce, engine);
4379− __execlists_update_reg_state(ce, engine, head);
4380− ce->lrc.desc |= CTX_DESC_FORCE_RESTORE; /* paranoid: GPU was reset! */
4381−
4382−unwind:
4383− /* Push back any incomplete requests for replay after the reset. */
4384− cancel_port_requests(execlists);
4385− __unwind_incomplete_requests(engine);
4386−}
4387−
4388−static void execlists_reset_rewind(struct intel_engine_cs *engine, bool stalled)
4389−{
4390− unsigned long flags;
4391−
4392− ENGINE_TRACE(engine, "\n");
4393−
4394− spin_lock_irqsave(&engine->active.lock, flags);
4395−
4396− __execlists_reset(engine, stalled);
4397−
4398− spin_unlock_irqrestore(&engine->active.lock, flags);
4399−}
4400−
4401−static void nop_submission_tasklet(unsigned long data)
4402−{
4403− struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
4404−
4405− /* The driver is wedged; don't process any more events. */
4406− WRITE_ONCE(engine->execlists.queue_priority_hint, INT_MIN);
4407−}
4408−
4409−static void execlists_reset_cancel(struct intel_engine_cs *engine)
4410−{
4411− struct intel_engine_execlists * const execlists = &engine->execlists;
4412− struct i915_request *rq, *rn;
4413− struct rb_node *rb;
4414− unsigned long flags;
4415−
4416− ENGINE_TRACE(engine, "\n");
4417−
4418− /*
4419− * Before we call engine->cancel_requests(), we should have exclusive
4420− * access to the submission state. This is arranged for us by the
4421− * caller disabling the interrupt generation, the tasklet and other
4422− * threads that may then access the same state, giving us a free hand
4423− * to reset state. However, we still need to let lockdep be aware that
4424− * we know this state may be accessed in hardirq context, so we
4425− * disable the irq around this manipulation and we want to keep
4426− * the spinlock focused on its duties and not accidentally conflate
4427− * coverage to the submission's irq state. (Similarly, although we
4428− * shouldn't need to disable irq around the manipulation of the
4429− * submission's irq state, we also wish to remind ourselves that
4430− * it is irq state.)
4431− */
4432− spin_lock_irqsave(&engine->active.lock, flags);
4433−
4434− __execlists_reset(engine, true);
4435−
4436− /* Mark all executing requests as skipped. */
4437− list_for_each_entry(rq, &engine->active.requests, sched.link)
4438− mark_eio(rq);
4439− intel_engine_signal_breadcrumbs(engine);
4440−
4441− /* Flush the queued requests to the timeline list (for retiring). */
4442− while ((rb = rb_first_cached(&execlists->queue))) {
4443− struct i915_priolist *p = to_priolist(rb);
4444− int i;
4445−
4446− priolist_for_each_request_consume(rq, rn, p, i) {
4447− mark_eio(rq);
4448− __i915_request_submit(rq);
4449− }
4450−
4451− rb_erase_cached(&p->node, &execlists->queue);
4452− i915_priolist_free(p);
4453− }
4454−
4455− /* On-hold requests will be flushed to timeline upon their release */
4456− list_for_each_entry(rq, &engine->active.hold, sched.link)
4457− mark_eio(rq);
4458−
4459− /* Cancel all attached virtual engines */
4460− while ((rb = rb_first_cached(&execlists->virtual))) {
4461− struct virtual_engine *ve =
4462− rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
4463−
4464− rb_erase_cached(rb, &execlists->virtual);
4465− RB_CLEAR_NODE(rb);
4466−
4467− spin_lock(&ve->base.active.lock);
4468− rq = fetch_and_zero(&ve->request);
4469− if (rq) {
4470− mark_eio(rq);
4471−
4472− rq->engine = engine;
4473− __i915_request_submit(rq);
4474− i915_request_put(rq);
4475−
4476− ve->base.execlists.queue_priority_hint = INT_MIN;
4477− }
4478− spin_unlock(&ve->base.active.lock);
4479− }
4480−
4481− /* Remaining _unready_ requests will be nop'ed when submitted */
4482−
4483− execlists->queue_priority_hint = INT_MIN;
4484− execlists->queue = RB_ROOT_CACHED;
4485−
4486− GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
4487− execlists->tasklet.func = nop_submission_tasklet;
4488−
4489− spin_unlock_irqrestore(&engine->active.lock, flags);
4490−}
4491−
4492−static void execlists_reset_finish(struct intel_engine_cs *engine)
4493−{
4494− struct intel_engine_execlists * const execlists = &engine->execlists;
4495−
4496− /*
4497− * After a GPU reset, we may have requests to replay. Do so now while
4498− * we still have the forcewake to be sure that the GPU is not allowed
4499− * to sleep before we restart and reload a context.
4500− */
4501− GEM_BUG_ON(!reset_in_progress(execlists));
4502− if (!RB_EMPTY_ROOT(&execlists->queue.rb_root))
4503− execlists->tasklet.func(execlists->tasklet.data);
4504−
4505− if (__tasklet_enable(&execlists->tasklet))
4506− /* And kick in case we missed a new request submission. */
4507− tasklet_hi_schedule(&execlists->tasklet);
4508− ENGINE_TRACE(engine, "depth->%d\n",
4509− atomic_read(&execlists->tasklet.count));
4510−}
4511−
4512−static int gen8_emit_bb_start_noarb(struct i915_request *rq,
4513− u64 offset, u32 len,
4514− const unsigned int flags)
4515−{
4516− u32 *cs;
4517−
4518− cs = intel_ring_begin(rq, 4);
4519− if (IS_ERR(cs))
4520− return PTR_ERR(cs);
4521−
4522− /*
4523− * WaDisableCtxRestoreArbitration:bdw,chv
4524− *
4525− * We don't need to perform MI_ARB_ENABLE as often as we do (in
4526− * particular all the gen that do not need the w/a at all!), if we
4527− * took care to make sure that on every switch into this context
4528− * (both ordinary and for preemption) that arbitrartion was enabled
4529− * we would be fine. However, for gen8 there is another w/a that
4530− * requires us to not preempt inside GPGPU execution, so we keep
4531− * arbitration disabled for gen8 batches. Arbitration will be
4532− * re-enabled before we close the request
4533− * (engine->emit_fini_breadcrumb).
4534− */
4535− *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
4536−
4537− /* FIXME(BDW+): Address space and security selectors. */
4538− *cs++ = MI_BATCH_BUFFER_START_GEN8 |
4539− (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
4540− *cs++ = lower_32_bits(offset);
4541− *cs++ = upper_32_bits(offset);
4542−
4543− intel_ring_advance(rq, cs);
4544−
4545− return 0;
4546−}
4547−
4548−static int gen8_emit_bb_start(struct i915_request *rq,
4549− u64 offset, u32 len,
4550− const unsigned int flags)
4551−{
4552− u32 *cs;
4553−
4554− cs = intel_ring_begin(rq, 6);
4555− if (IS_ERR(cs))
4556− return PTR_ERR(cs);
4557−
4558− *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
4559−
4560− *cs++ = MI_BATCH_BUFFER_START_GEN8 |
4561− (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
4562− *cs++ = lower_32_bits(offset);
4563− *cs++ = upper_32_bits(offset);
4564−
4565− *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
4566− *cs++ = MI_NOOP;
4567−
4568− intel_ring_advance(rq, cs);
4569−
4570− return 0;
4571−}
4572−
4573−static void gen8_logical_ring_enable_irq(struct intel_engine_cs *engine)
4574−{
4575− ENGINE_WRITE(engine, RING_IMR,
4576− ~(engine->irq_enable_mask | engine->irq_keep_mask));
4577− ENGINE_POSTING_READ(engine, RING_IMR);
4578−}
4579−
4580−static void gen8_logical_ring_disable_irq(struct intel_engine_cs *engine)
4581−{
4582− ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
4583−}
4584−
4585−static int gen8_emit_flush(struct i915_request *request, u32 mode)
4586−{
4587− u32 cmd, *cs;
4588−
4589− cs = intel_ring_begin(request, 4);
4590− if (IS_ERR(cs))
4591− return PTR_ERR(cs);
4592−
4593− cmd = MI_FLUSH_DW + 1;
4594−
4595− /* We always require a command barrier so that subsequent
4596− * commands, such as breadcrumb interrupts, are strictly ordered
4597− * wrt the contents of the write cache being flushed to memory
4598− * (and thus being coherent from the CPU).
4599− */
4600− cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
4601−
4602− if (mode & EMIT_INVALIDATE) {
4603− cmd |= MI_INVALIDATE_TLB;
4604− if (request->engine->class == VIDEO_DECODE_CLASS)
4605− cmd |= MI_INVALIDATE_BSD;
4606− }
4607−
4608− *cs++ = cmd;
4609− *cs++ = LRC_PPHWSP_SCRATCH_ADDR;
4610− *cs++ = 0; /* upper addr */
4611− *cs++ = 0; /* value */
4612− intel_ring_advance(request, cs);
4613−
4614− return 0;
4615−}
4616−
4617−static int gen8_emit_flush_render(struct i915_request *request,
4618− u32 mode)
4619−{
4620− bool vf_flush_wa = false, dc_flush_wa = false;
4621− u32 *cs, flags = 0;
4622− int len;
4623−
4624− flags |= PIPE_CONTROL_CS_STALL;
4625−
4626− if (mode & EMIT_FLUSH) {
4627− flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
4628− flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
4629− flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
4630− flags |= PIPE_CONTROL_FLUSH_ENABLE;
4631− }
4632−
4633− if (mode & EMIT_INVALIDATE) {
4634− flags |= PIPE_CONTROL_TLB_INVALIDATE;
4635− flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
4636− flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
4637− flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
4638− flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
4639− flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
4640− flags |= PIPE_CONTROL_QW_WRITE;
4641− flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4642−
4643− /*
4644− * On GEN9: before VF_CACHE_INVALIDATE we need to emit a NULL
4645− * pipe control.
4646− */
4647− if (IS_GEN(request->engine->i915, 9))
4648− vf_flush_wa = true;
4649−
4650− /* WaForGAMHang:kbl */
4651− if (IS_KBL_GT_REVID(request->engine->i915, 0, KBL_REVID_B0))
4652− dc_flush_wa = true;
4653− }
4654−
4655− len = 6;
4656−
4657− if (vf_flush_wa)
4658− len += 6;
4659−
4660− if (dc_flush_wa)
4661− len += 12;
4662−
4663− cs = intel_ring_begin(request, len);
4664− if (IS_ERR(cs))
4665− return PTR_ERR(cs);
4666−
4667− if (vf_flush_wa)
4668− cs = gen8_emit_pipe_control(cs, 0, 0);
4669−
4670− if (dc_flush_wa)
4671− cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_DC_FLUSH_ENABLE,
4672− 0);
4673−
4674− cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4675−
4676− if (dc_flush_wa)
4677− cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_CS_STALL, 0);
4678−
4679− intel_ring_advance(request, cs);
4680−
4681− return 0;
4682−}
4683−
4684−static int gen11_emit_flush_render(struct i915_request *request,
4685− u32 mode)
4686−{
4687− if (mode & EMIT_FLUSH) {
4688− u32 *cs;
4689− u32 flags = 0;
4690−
4691− flags |= PIPE_CONTROL_CS_STALL;
4692−
4693− flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
4694− flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
4695− flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
4696− flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
4697− flags |= PIPE_CONTROL_FLUSH_ENABLE;
4698− flags |= PIPE_CONTROL_QW_WRITE;
4699− flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4700−
4701− cs = intel_ring_begin(request, 6);
4702− if (IS_ERR(cs))
4703− return PTR_ERR(cs);
4704−
4705− cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4706− intel_ring_advance(request, cs);
4707− }
4708−
4709− if (mode & EMIT_INVALIDATE) {
4710− u32 *cs;
4711− u32 flags = 0;
4712−
4713− flags |= PIPE_CONTROL_CS_STALL;
4714−
4715− flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
4716− flags |= PIPE_CONTROL_TLB_INVALIDATE;
4717− flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
4718− flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
4719− flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
4720− flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
4721− flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
4722− flags |= PIPE_CONTROL_QW_WRITE;
4723− flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4724−
4725− cs = intel_ring_begin(request, 6);
4726− if (IS_ERR(cs))
4727− return PTR_ERR(cs);
4728−
4729− cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4730− intel_ring_advance(request, cs);
4731− }
4732−
4733− return 0;
4734−}
4735−
4736−static u32 preparser_disable(bool state)
4737−{
4738− return MI_ARB_CHECK | 1 << 8 | state;
4739−}
4740−
4741−static i915_reg_t aux_inv_reg(const struct intel_engine_cs *engine)
4742−{
4743− static const i915_reg_t vd[] = {
4744− GEN12_VD0_AUX_NV,
4745− GEN12_VD1_AUX_NV,
4746− GEN12_VD2_AUX_NV,
4747− GEN12_VD3_AUX_NV,
4748− };
4749−
4750− static const i915_reg_t ve[] = {
4751− GEN12_VE0_AUX_NV,
4752− GEN12_VE1_AUX_NV,
4753− };
4754−
4755− if (engine->class == VIDEO_DECODE_CLASS)
4756− return vd[engine->instance];
4757−
4758− if (engine->class == VIDEO_ENHANCEMENT_CLASS)
4759− return ve[engine->instance];
4760−
4761− GEM_BUG_ON("unknown aux_inv_reg\n");
4762−
4763− return INVALID_MMIO_REG;
4764−}
4765−
4766−static u32 *
4767−gen12_emit_aux_table_inv(const i915_reg_t inv_reg, u32 *cs)
4768−{
4769− *cs++ = MI_LOAD_REGISTER_IMM(1);
4770− *cs++ = i915_mmio_reg_offset(inv_reg);
4771− *cs++ = AUX_INV;
4772− *cs++ = MI_NOOP;
4773−
4774− return cs;
4775−}
4776−
4777−static int gen12_emit_flush_render(struct i915_request *request,
4778− u32 mode)
4779−{
4780− if (mode & EMIT_FLUSH) {
4781− u32 flags = 0;
4782− u32 *cs;
4783−
4784− flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
4785− flags |= PIPE_CONTROL_FLUSH_L3;
4786− flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
4787− flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
4788− /* Wa_1409600907:tgl */
4789− flags |= PIPE_CONTROL_DEPTH_STALL;
4790− flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
4791− flags |= PIPE_CONTROL_FLUSH_ENABLE;
4792−
4793− flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4794− flags |= PIPE_CONTROL_QW_WRITE;
4795−
4796− flags |= PIPE_CONTROL_CS_STALL;
4797−
4798− cs = intel_ring_begin(request, 6);
4799− if (IS_ERR(cs))
4800− return PTR_ERR(cs);
4801−
4802− cs = gen12_emit_pipe_control(cs,
4803− PIPE_CONTROL0_HDC_PIPELINE_FLUSH,
4804− flags, LRC_PPHWSP_SCRATCH_ADDR);
4805− intel_ring_advance(request, cs);
4806− }
4807−
4808− if (mode & EMIT_INVALIDATE) {
4809− u32 flags = 0;
4810− u32 *cs;
4811−
4812− flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
4813− flags |= PIPE_CONTROL_TLB_INVALIDATE;
4814− flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
4815− flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
4816− flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
4817− flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
4818− flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
4819−
4820− flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4821− flags |= PIPE_CONTROL_QW_WRITE;
4822−
4823− flags |= PIPE_CONTROL_CS_STALL;
4824−
4825− cs = intel_ring_begin(request, 8 + 4);
4826− if (IS_ERR(cs))
4827− return PTR_ERR(cs);
4828−
4829− /*
4830− * Prevent the pre-parser from skipping past the TLB
4831− * invalidate and loading a stale page for the batch
4832− * buffer / request payload.
4833− */
4834− *cs++ = preparser_disable(true);
4835−
4836− cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4837−
4838− /* hsdes: 1809175790 */
4839− cs = gen12_emit_aux_table_inv(GEN12_GFX_CCS_AUX_NV, cs);
4840−
4841− *cs++ = preparser_disable(false);
4842− intel_ring_advance(request, cs);
4843− }
4844−
4845− return 0;
4846−}
4847−
4848−static int gen12_emit_flush(struct i915_request *request, u32 mode)
4849−{
4850− intel_engine_mask_t aux_inv = 0;
4851− u32 cmd, *cs;
4852−
4853− cmd = 4;
4854− if (mode & EMIT_INVALIDATE)
4855− cmd += 2;
4856− if (mode & EMIT_INVALIDATE)
4857− aux_inv = request->engine->mask & ~BIT(BCS0);
4858− if (aux_inv)
4859− cmd += 2 * hweight8(aux_inv) + 2;
4860−
4861− cs = intel_ring_begin(request, cmd);
4862− if (IS_ERR(cs))
4863− return PTR_ERR(cs);
4864−
4865− if (mode & EMIT_INVALIDATE)
4866− *cs++ = preparser_disable(true);
4867−
4868− cmd = MI_FLUSH_DW + 1;
4869−
4870− /* We always require a command barrier so that subsequent
4871− * commands, such as breadcrumb interrupts, are strictly ordered
4872− * wrt the contents of the write cache being flushed to memory
4873− * (and thus being coherent from the CPU).
4874− */
4875− cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
4876−
4877− if (mode & EMIT_INVALIDATE) {
4878− cmd |= MI_INVALIDATE_TLB;
4879− if (request->engine->class == VIDEO_DECODE_CLASS)
4880− cmd |= MI_INVALIDATE_BSD;
4881− }
4882−
4883− *cs++ = cmd;

2191 more lines aren’t shown here.

drivers/gpu/drm/i915/gt/intel_lrc.h+61−106
@@ -1,90 +1,20 @@
1+/* SPDX-License-Identifier: MIT */
12 /*
23 * Copyright © 2014 Intel Corporation
3− *
4− * Permission is hereby granted, free of charge, to any person obtaining a
5− * copy of this software and associated documentation files (the "Software"),
6− * to deal in the Software without restriction, including without limitation
7− * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8− * and/or sell copies of the Software, and to permit persons to whom the
9− * Software is furnished to do so, subject to the following conditions:
10− *
11− * The above copyright notice and this permission notice (including the next
12− * paragraph) shall be included in all copies or substantial portions of the
13− * Software.
14− *
15− * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16− * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17− * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18− * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19− * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
20− * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
21− * DEALINGS IN THE SOFTWARE.
224 */
235
24−#ifndef _INTEL_LRC_H_
25−#define _INTEL_LRC_H_
6+#ifndef __INTEL_LRC_H__
7+#define __INTEL_LRC_H__
268
279 #include <linux/types.h>
2810
29−struct drm_printer;
11+#include "intel_context.h"
12+#include "intel_lrc_reg.h"
3013
31−struct drm_i915_private;
32−struct i915_gem_context;
33−struct i915_request;
34−struct intel_context;
14+struct drm_i915_gem_object;
3515 struct intel_engine_cs;
16+struct intel_ring;
3617
37−/* Execlists regs */
38−#define RING_ELSP(base) _MMIO((base) + 0x230)
39−#define RING_EXECLIST_STATUS_LO(base) _MMIO((base) + 0x234)
40−#define RING_EXECLIST_STATUS_HI(base) _MMIO((base) + 0x234 + 4)
41−#define RING_CONTEXT_CONTROL(base) _MMIO((base) + 0x244)
42−#define CTX_CTRL_INHIBIT_SYN_CTX_SWITCH (1 << 3)
43−#define CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT (1 << 0)
44−#define CTX_CTRL_RS_CTX_ENABLE (1 << 1)
45−#define CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT (1 << 2)
46−#define GEN12_CTX_CTRL_OAR_CONTEXT_ENABLE (1 << 8)
47−#define RING_CONTEXT_STATUS_PTR(base) _MMIO((base) + 0x3a0)
48−#define RING_EXECLIST_SQ_CONTENTS(base) _MMIO((base) + 0x510)
49−#define RING_EXECLIST_CONTROL(base) _MMIO((base) + 0x550)
50−
51−#define EL_CTRL_LOAD (1 << 0)
52−
53−/* The docs specify that the write pointer wraps around after 5h, "After status
54− * is written out to the last available status QW at offset 5h, this pointer
55− * wraps to 0."
56− *
57− * Therefore, one must infer than even though there are 3 bits available, 6 and
58− * 7 appear to be * reserved.
59− */
60−#define GEN8_CSB_ENTRIES 6
61−#define GEN8_CSB_PTR_MASK 0x7
62−#define GEN8_CSB_READ_PTR_MASK (GEN8_CSB_PTR_MASK << 8)
63−#define GEN8_CSB_WRITE_PTR_MASK (GEN8_CSB_PTR_MASK << 0)
64−
65−#define GEN11_CSB_ENTRIES 12
66−#define GEN11_CSB_PTR_MASK 0xf
67−#define GEN11_CSB_READ_PTR_MASK (GEN11_CSB_PTR_MASK << 8)
68−#define GEN11_CSB_WRITE_PTR_MASK (GEN11_CSB_PTR_MASK << 0)
69−
70−#define MAX_CONTEXT_HW_ID (1<<21) /* exclusive */
71−#define MAX_GUC_CONTEXT_HW_ID (1 << 20) /* exclusive */
72−#define GEN11_MAX_CONTEXT_HW_ID (1<<11) /* exclusive */
73−/* in Gen12 ID 0x7FF is reserved to indicate idle */
74−#define GEN12_MAX_CONTEXT_HW_ID (GEN11_MAX_CONTEXT_HW_ID - 1)
75−
76−enum {
77− INTEL_CONTEXT_SCHEDULE_IN = 0,
78− INTEL_CONTEXT_SCHEDULE_OUT,
79− INTEL_CONTEXT_SCHEDULE_PREEMPTED,
80−};
81−
82−/* Logical Rings */
83−void intel_logical_ring_cleanup(struct intel_engine_cs *engine);
84−
85−int intel_execlists_submission_setup(struct intel_engine_cs *engine);
86−
87−/* Logical Ring Contexts */
8818 /* At the start of the context image is its per-process HWS page */
8919 #define LRC_PPHWSP_PN (0)
9020 #define LRC_PPHWSP_SZ (1)
@@ -96,32 +26,57 @@ int intel_execlists_submission_setup(struct intel_engine_cs *engine); · 5 unmodified lines
9626 #define LRC_PPHWSP_SCRATCH 0x34
9727 #define LRC_PPHWSP_SCRATCH_ADDR (LRC_PPHWSP_SCRATCH * sizeof(u32))
9828
99−void intel_execlists_set_default_submission(struct intel_engine_cs *engine);
100−
101−void intel_lr_context_reset(struct intel_engine_cs *engine,
102− struct intel_context *ce,
103− u32 head,
104− bool scrub);
105−
106−void intel_execlists_show_requests(struct intel_engine_cs *engine,
107− struct drm_printer *m,
108− void (*show_request)(struct drm_printer *m,
109− struct i915_request *rq,
110− const char *prefix),
111− unsigned int max);
112−
113−struct intel_context *
114−intel_execlists_create_virtual(struct intel_engine_cs **siblings,
115− unsigned int count);
116−
117−struct intel_context *
118−intel_execlists_clone_virtual(struct intel_engine_cs *src);
119−
120−int intel_virtual_engine_attach_bond(struct intel_engine_cs *engine,
121− const struct intel_engine_cs *master,
122− const struct intel_engine_cs *sibling);
123−
124−bool
125−intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine);
126−
127−#endif /* _INTEL_LRC_H_ */
29+void lrc_init_wa_ctx(struct intel_engine_cs *engine);
30+void lrc_fini_wa_ctx(struct intel_engine_cs *engine);
31+
32+int lrc_alloc(struct intel_context *ce,
33+ struct intel_engine_cs *engine);
34+void lrc_reset(struct intel_context *ce);
35+void lrc_fini(struct intel_context *ce);
36+void lrc_destroy(struct kref *kref);
37+
38+int
39+lrc_pre_pin(struct intel_context *ce,
40+ struct intel_engine_cs *engine,
41+ struct i915_gem_ww_ctx *ww,
42+ void **vaddr);
43+int
44+lrc_pin(struct intel_context *ce,
45+ struct intel_engine_cs *engine,
46+ void *vaddr);
47+void lrc_unpin(struct intel_context *ce);
48+void lrc_post_unpin(struct intel_context *ce);
49+
50+void lrc_init_state(struct intel_context *ce,
51+ struct intel_engine_cs *engine,
52+ void *state);
53+
54+void lrc_init_regs(const struct intel_context *ce,
55+ const struct intel_engine_cs *engine,
56+ bool clear);
57+void lrc_reset_regs(const struct intel_context *ce,
58+ const struct intel_engine_cs *engine);
59+
60+u32 lrc_update_regs(const struct intel_context *ce,
61+ const struct intel_engine_cs *engine,
62+ u32 head);
63+void lrc_update_offsets(struct intel_context *ce,
64+ struct intel_engine_cs *engine);
65+
66+void lrc_check_regs(const struct intel_context *ce,
67+ const struct intel_engine_cs *engine,
68+ const char *when);
69+
70+void lrc_update_runtime(struct intel_context *ce);
71+static inline u32 lrc_get_runtime(const struct intel_context *ce)
72+{
73+ /*
74+ * We can use either ppHWSP[16] which is recorded before the context
75+ * switch (and so excludes the cost of context switches) or use the
76+ * value from the context image itself, which is saved/restored earlier
77+ * and so includes the cost of the save.
78+ */
79+ return READ_ONCE(ce->lrc_reg_state[CTX_TIMESTAMP]);
80+}
81+
82+#endif /* __INTEL_LRC_H__ */
drivers/gpu/drm/i915/gt/intel_lrc_reg.h+41−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
99
1010 #include <linux/types.h>
1111
12+#define CTX_DESC_FORCE_RESTORE BIT_ULL(2)
13+
1214 /* GEN8 to GEN12 Reg State Context */
1315 #define CTX_CONTEXT_CONTROL (0x02 + 1)
1416 #define CTX_RING_HEAD (0x04 + 1)
@@ -52,4 +54,43 @@ · 37 unmodified lines
5254 #define GEN8_EXECLISTS_STATUS_BUF 0x370
5355 #define GEN11_EXECLISTS_STATUS_BUF2 0x3c0
5456
57+/* Execlists regs */
58+#define RING_ELSP(base) _MMIO((base) + 0x230)
59+#define RING_EXECLIST_STATUS_LO(base) _MMIO((base) + 0x234)
60+#define RING_EXECLIST_STATUS_HI(base) _MMIO((base) + 0x234 + 4)
61+#define RING_CONTEXT_CONTROL(base) _MMIO((base) + 0x244)
62+#define CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT REG_BIT(0)
63+#define CTX_CTRL_RS_CTX_ENABLE REG_BIT(1)
64+#define CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT REG_BIT(2)
65+#define CTX_CTRL_INHIBIT_SYN_CTX_SWITCH REG_BIT(3)
66+#define GEN12_CTX_CTRL_OAR_CONTEXT_ENABLE REG_BIT(8)
67+#define RING_CONTEXT_STATUS_PTR(base) _MMIO((base) + 0x3a0)
68+#define RING_EXECLIST_SQ_CONTENTS(base) _MMIO((base) + 0x510)
69+#define RING_EXECLIST_CONTROL(base) _MMIO((base) + 0x550)
70+#define EL_CTRL_LOAD REG_BIT(0)
71+
72+/*
73+ * The docs specify that the write pointer wraps around after 5h, "After status
74+ * is written out to the last available status QW at offset 5h, this pointer
75+ * wraps to 0."
76+ *
77+ * Therefore, one must infer than even though there are 3 bits available, 6 and
78+ * 7 appear to be * reserved.
79+ */
80+#define GEN8_CSB_ENTRIES 6
81+#define GEN8_CSB_PTR_MASK 0x7
82+#define GEN8_CSB_READ_PTR_MASK (GEN8_CSB_PTR_MASK << 8)
83+#define GEN8_CSB_WRITE_PTR_MASK (GEN8_CSB_PTR_MASK << 0)
84+
85+#define GEN11_CSB_ENTRIES 12
86+#define GEN11_CSB_PTR_MASK 0xf
87+#define GEN11_CSB_READ_PTR_MASK (GEN11_CSB_PTR_MASK << 8)
88+#define GEN11_CSB_WRITE_PTR_MASK (GEN11_CSB_PTR_MASK << 0)
89+
90+#define MAX_CONTEXT_HW_ID (1 << 21) /* exclusive */
91+#define MAX_GUC_CONTEXT_HW_ID (1 << 20) /* exclusive */
92+#define GEN11_MAX_CONTEXT_HW_ID (1 << 11) /* exclusive */
93+/* in Gen12 ID 0x7FF is reserved to indicate idle */
94+#define GEN12_MAX_CONTEXT_HW_ID (GEN11_MAX_CONTEXT_HW_ID - 1)
95+
5596 #endif /* _INTEL_LRC_REG_H_ */
drivers/gpu/drm/i915/gt/intel_mocs.c+1−1
@@ -24,8 +24,8 @@ · 23 unmodified lines
2424
2525 #include "intel_engine.h"
2626 #include "intel_gt.h"
27+#include "intel_lrc_reg.h"
2728 #include "intel_mocs.h"
28−#include "intel_lrc.h"
2929 #include "intel_ring.h"
3030
3131 /* structures required */
drivers/gpu/drm/i915/gt/intel_region_lmem.crenamed from drivers/gpu/drm/i915/intel_region_lmem.c+1−1
@@ -95,7 +95,7 @@ region_lmem_init(struct intel_memory_region *mem) · 94 unmodified lines
9595 return ret;
9696 }
9797
98−const struct intel_memory_region_ops intel_region_lmem_ops = {
98+static const struct intel_memory_region_ops intel_region_lmem_ops = {
9999 .init = region_lmem_init,
100100 .release = region_lmem_release,
101101 .create_object = __i915_gem_lmem_object_create,
drivers/gpu/drm/i915/gt/intel_region_lmem.hrenamed from drivers/gpu/drm/i915/intel_region_lmem.h+0−2
@@ -8,8 +8,6 @@ · 7 unmodified lines
88
99 struct drm_i915_private;
1010
11−extern const struct intel_memory_region_ops intel_region_lmem_ops;
12−
1311 struct intel_memory_region *
1412 intel_setup_fake_lmem(struct drm_i915_private *i915);
1513
drivers/gpu/drm/i915/gt/intel_renderstate.c+2−1
@@ -27,7 +27,8 @@ · 26 unmodified lines
2727
2828 #include "i915_drv.h"
2929 #include "intel_renderstate.h"
30−#include "gt/intel_context.h"
30+#include "intel_context.h"
31+#include "intel_gpu_commands.h"
3132 #include "intel_ring.h"
3233
3334 static const struct intel_renderstate_rodata *
drivers/gpu/drm/i915/gt/intel_reset.c+62−34
@@ -40,20 +40,19 @@ static void rmw_clear_fw(struct intel_uncore *uncore, i915_reg_t reg, u32 clr) · 39 unmodified lines
4040 intel_uncore_rmw_fw(uncore, reg, clr, 0);
4141 }
4242
43−static void engine_skip_context(struct i915_request *rq)
43+static void skip_context(struct i915_request *rq)
4444 {
45− struct intel_engine_cs *engine = rq->engine;
4645 struct intel_context *hung_ctx = rq->context;
4746
48− if (!i915_request_is_active(rq))
49− return;
47+ list_for_each_entry_from_rcu(rq, &hung_ctx->timeline->requests, link) {
48+ if (!i915_request_is_active(rq))
49+ return;
5050
51− lockdep_assert_held(&engine->active.lock);
52− list_for_each_entry_continue(rq, &engine->active.requests, sched.link)
5351 if (rq->context == hung_ctx) {
5452 i915_request_set_error_once(rq, -EIO);
5553 __i915_request_skip(rq);
5654 }
55+ }
5756 }
5857
5958 static void client_mark_guilty(struct i915_gem_context *ctx, bool banned)
@@ -160,7 +159,7 @@ void __i915_request_reset(struct i915_request *rq, bool guilty) · 100 unmodified lines
160159 i915_request_set_error_once(rq, -EIO);
161160 __i915_request_skip(rq);
162161 if (mark_guilty(rq))
163− engine_skip_context(rq);
162+ skip_context(rq);
164163 } else {
165164 i915_request_set_error_once(rq, -EAGAIN);
166165 mark_innocent(rq);
@@ -231,7 +230,7 @@ static int g4x_do_reset(struct intel_gt *gt, · 64 unmodified lines
231230 GRDOM_MEDIA | GRDOM_RESET_ENABLE);
232231 ret = wait_for_atomic(g4x_reset_complete(pdev), 50);
233232 if (ret) {
234− drm_dbg(&gt->i915->drm, "Wait for media reset failed\n");
233+ GT_TRACE(gt, "Wait for media reset failed\n");
235234 goto out;
236235 }
237236
@@ -239,7 +238,7 @@ static int g4x_do_reset(struct intel_gt *gt, · 1 unmodified line
239238 GRDOM_RENDER | GRDOM_RESET_ENABLE);
240239 ret = wait_for_atomic(g4x_reset_complete(pdev), 50);
241240 if (ret) {
242− drm_dbg(&gt->i915->drm, "Wait for render reset failed\n");
241+ GT_TRACE(gt, "Wait for render reset failed\n");
243242 goto out;
244243 }
245244
@@ -265,7 +264,7 @@ static int ilk_do_reset(struct intel_gt *gt, intel_engine_mask_t engine_mask, · 19 unmodified lines
265264 5000, 0,
266265 NULL);
267266 if (ret) {
268− drm_dbg(&gt->i915->drm, "Wait for render reset failed\n");
267+ GT_TRACE(gt, "Wait for render reset failed\n");
269268 goto out;
270269 }
271270
@@ -276,7 +275,7 @@ static int ilk_do_reset(struct intel_gt *gt, intel_engine_mask_t engine_mask, · 4 unmodified lines
276275 5000, 0,
277276 NULL);
278277 if (ret) {
279− drm_dbg(&gt->i915->drm, "Wait for media reset failed\n");
278+ GT_TRACE(gt, "Wait for media reset failed\n");
280279 goto out;
281280 }
282281
@@ -305,9 +304,9 @@ static int gen6_hw_domain_reset(struct intel_gt *gt, u32 hw_domain_mask) · 22 unmodified lines
305304 500, 0,
306305 NULL);
307306 if (err)
308− drm_dbg(&gt->i915->drm,
309− "Wait for 0x%08x engines reset failed\n",
310− hw_domain_mask);
307+ GT_TRACE(gt,
308+ "Wait for 0x%08x engines reset failed\n",
309+ hw_domain_mask);
311310
312311 return err;
313312 }
@@ -407,8 +406,7 @@ static int gen11_lock_sfc(struct intel_engine_cs *engine, u32 *hw_mask) · 93 unmodified lines
407406 return 0;
408407
409408 if (ret) {
410− drm_dbg(&engine->i915->drm,
411− "Wait for SFC forced lock ack failed\n");
409+ ENGINE_TRACE(engine, "Wait for SFC forced lock ack failed\n");
412410 return ret;
413411 }
414412
@@ -499,6 +497,9 @@ static int gen8_engine_reset_prepare(struct intel_engine_cs *engine) · 84 unmodified lines
499497 u32 request, mask, ack;
500498 int ret;
501499
500+ if (I915_SELFTEST_ONLY(should_fail(&engine->reset_timeout, 1)))
501+ return -ETIMEDOUT;
502+
502503 ack = intel_uncore_read_fw(uncore, reg);
503504 if (ack & RESET_CTL_CAT_ERROR) {
504505 /*
@@ -754,8 +755,10 @@ static int gt_reset(struct intel_gt *gt, intel_engine_mask_t stalled_mask) · 249 unmodified lines
754755 if (err)
755756 return err;
756757
758+ local_bh_disable();
757759 for_each_engine(engine, gt, id)
758760 __intel_engine_reset(engine, stalled_mask & engine->mask);
761+ local_bh_enable();
759762
760763 intel_ggtt_restore_fences(gt->ggtt);
761764
@@ -833,9 +836,11 @@ static void __intel_gt_set_wedged(struct intel_gt *gt) · 71 unmodified lines
833836 set_bit(I915_WEDGED, &gt->reset.flags);
834837
835838 /* Mark all executing requests as skipped */
839+ local_bh_disable();
836840 for_each_engine(engine, gt, id)
837841 if (engine->reset.cancel)
838842 engine->reset.cancel(engine);
843+ local_bh_enable();
839844
840845 reset_finish(gt, awake);
841846
@@ -1110,20 +1115,7 @@ static inline int intel_gt_reset_engine(struct intel_engine_cs *engine) · 268 unmodified lines
11101115 return __intel_gt_reset(engine->gt, engine->mask);
11111116 }
11121117
1113−/**
1114− * intel_engine_reset - reset GPU engine to recover from a hang
1115− * @engine: engine to reset
1116− * @msg: reason for GPU reset; or NULL for no drm_notice()
1117− *
1118− * Reset a specific GPU engine. Useful if a hang is detected.
1119− * Returns zero on successful reset or otherwise an error code.
1120− *
1121− * Procedure is:
1122− * - identifies the request that caused the hang and it is dropped
1123− * - reset engine (which will force the engine to idle)
1124− * - re-init/configure engine
1125− */
1126−int intel_engine_reset(struct intel_engine_cs *engine, const char *msg)
1118+int __intel_engine_reset_bh(struct intel_engine_cs *engine, const char *msg)
11271119 {
11281120 struct intel_gt *gt = engine->gt;
11291121 bool uses_guc = intel_engine_in_guc_submission_mode(engine);
@@ -1148,8 +1140,7 @@ int intel_engine_reset(struct intel_engine_cs *engine, const char *msg) · 18 unmodified lines
11481140 ret = intel_guc_reset_engine(&engine->gt->uc.guc, engine);
11491141 if (ret) {
11501142 /* If we fail here, we expect to fallback to a global reset */
1151− drm_dbg(&gt->i915->drm, "%sFailed to reset %s, ret=%d\n",
1152− uses_guc ? "GuC " : "", engine->name, ret);
1143+ ENGINE_TRACE(engine, "Failed to reset, err: %d\n", ret);
11531144 goto out;
11541145 }
11551146
@@ -1174,6 +1165,30 @@ out: · 18 unmodified lines
11741165 return ret;
11751166 }
11761167
1168+/**
1169+ * intel_engine_reset - reset GPU engine to recover from a hang
1170+ * @engine: engine to reset
1171+ * @msg: reason for GPU reset; or NULL for no drm_notice()
1172+ *
1173+ * Reset a specific GPU engine. Useful if a hang is detected.
1174+ * Returns zero on successful reset or otherwise an error code.
1175+ *
1176+ * Procedure is:
1177+ * - identifies the request that caused the hang and it is dropped
1178+ * - reset engine (which will force the engine to idle)
1179+ * - re-init/configure engine
1180+ */
1181+int intel_engine_reset(struct intel_engine_cs *engine, const char *msg)
1182+{
1183+ int err;
1184+
1185+ local_bh_disable();
1186+ err = __intel_engine_reset_bh(engine, msg);
1187+ local_bh_enable();
1188+
1189+ return err;
1190+}
1191+
11771192 static void intel_gt_reset_global(struct intel_gt *gt,
11781193 u32 engine_mask,
11791194 const char *reason)
@@ -1186,7 +1201,7 @@ static void intel_gt_reset_global(struct intel_gt *gt, · 6 unmodified lines
11861201
11871202 kobject_uevent_env(kobj, KOBJ_CHANGE, error_event);
11881203
1189− drm_dbg(&gt->i915->drm, "resetting chip, engines=%x\n", engine_mask);
1204+ GT_TRACE(gt, "resetting chip, engines=%x\n", engine_mask);
11901205 kobject_uevent_env(kobj, KOBJ_CHANGE, reset_event);
11911206
11921207 /* Use a watchdog to ensure that our reset completes */
@@ -1260,18 +1275,20 @@ void intel_gt_handle_error(struct intel_gt *gt, · 67 unmodified lines
12601275 * single reset fails.
12611276 */
12621277 if (intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
1278+ local_bh_disable();
12631279 for_each_engine_masked(engine, gt, engine_mask, tmp) {
12641280 BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
12651281 if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
12661282 &gt->reset.flags))
12671283 continue;
12681284
1269− if (intel_engine_reset(engine, msg) == 0)
1285+ if (__intel_engine_reset_bh(engine, msg) == 0)
12701286 engine_mask &= ~engine->mask;
12711287
12721288 clear_and_wake_up_bit(I915_RESET_ENGINE + engine->id,
12731289 &gt->reset.flags);
12741290 }
1291+ local_bh_enable();
12751292 }
12761293
12771294 if (!engine_mask)
@@ -1380,6 +1397,17 @@ void intel_gt_init_reset(struct intel_gt *gt) · 102 unmodified lines
13801397 mutex_init(&gt->reset.mutex);
13811398 init_srcu_struct(&gt->reset.backoff_srcu);
13821399
1400+ /*
1401+ * While undesirable to wait inside the shrinker, complain anyway.
1402+ *
1403+ * If we have to wait during shrinking, we guarantee forward progress
1404+ * by forcing the reset. Therefore during the reset we must not
1405+ * re-enter the shrinker. By declaring that we take the reset mutex
1406+ * within the shrinker, we forbid ourselves from performing any
1407+ * fs-reclaim or taking related locks during reset.
1408+ */
1409+ i915_gem_shrinker_taints_mutex(gt->i915, &gt->reset.mutex);
1410+
13831411 /* no GPU until we are ready! */
13841412 __set_bit(I915_WEDGED, &gt->reset.flags);
13851413 }
drivers/gpu/drm/i915/gt/intel_reset.h+2−0
@@ -34,6 +34,8 @@ void intel_gt_reset(struct intel_gt *gt, · 33 unmodified lines
3434 const char *reason);
3535 int intel_engine_reset(struct intel_engine_cs *engine,
3636 const char *reason);
37+int __intel_engine_reset_bh(struct intel_engine_cs *engine,
38+ const char *reason);
3739
3840 void __i915_request_reset(struct i915_request *rq, bool guilty);
3941
drivers/gpu/drm/i915/gt/intel_ring.c+2−0
@@ -5,9 +5,11 @@ · 4 unmodified lines
55 */
66
77 #include "gem/i915_gem_object.h"
8+
89 #include "i915_drv.h"
910 #include "i915_vma.h"
1011 #include "intel_engine.h"
12+#include "intel_gpu_commands.h"
1113 #include "intel_ring.h"
1214 #include "intel_timeline.h"
1315
drivers/gpu/drm/i915/gt/intel_ring_submission.c+43−30
@@ -32,6 +32,7 @@ · 31 unmodified lines
3232 #include "gen6_ppgtt.h"
3333 #include "gen7_renderclear.h"
3434 #include "i915_drv.h"
35+#include "i915_mitigations.h"
3536 #include "intel_breadcrumbs.h"
3637 #include "intel_context.h"
3738 #include "intel_gt.h"
@@ -158,30 +159,7 @@ static void ring_setup_status_page(struct intel_engine_cs *engine) · 120 unmodified lines
158159
159160 static bool stop_ring(struct intel_engine_cs *engine)
160161 {
161− struct drm_i915_private *dev_priv = engine->i915;
162−
163− if (INTEL_GEN(dev_priv) > 2) {
164− ENGINE_WRITE(engine,
165− RING_MI_MODE, _MASKED_BIT_ENABLE(STOP_RING));
166− if (intel_wait_for_register(engine->uncore,
167− RING_MI_MODE(engine->mmio_base),
168− MODE_IDLE,
169− MODE_IDLE,
170− 1000)) {
171− drm_err(&dev_priv->drm,
172− "%s : timed out trying to stop ring\n",
173− engine->name);
174−
175− /*
176− * Sometimes we observe that the idle flag is not
177− * set even though the ring is empty. So double
178− * check before giving up.
179− */
180− if (ENGINE_READ(engine, RING_HEAD) !=
181− ENGINE_READ(engine, RING_TAIL))
182− return false;
183− }
184− }
162+ intel_engine_stop_cs(engine);
185163
186164 ENGINE_WRITE(engine, RING_HEAD, ENGINE_READ(engine, RING_TAIL));
187165
@@ -321,6 +299,39 @@ out: · 133 unmodified lines
321299 return ret;
322300 }
323301
302+static void sanitize_hwsp(struct intel_engine_cs *engine)
303+{
304+ struct intel_timeline *tl;
305+
306+ list_for_each_entry(tl, &engine->status_page.timelines, engine_link)
307+ intel_timeline_reset_seqno(tl);
308+}
309+
310+static void xcs_sanitize(struct intel_engine_cs *engine)
311+{
312+ /*
313+ * Poison residual state on resume, in case the suspend didn't!
314+ *
315+ * We have to assume that across suspend/resume (or other loss
316+ * of control) that the contents of our pinned buffers has been
317+ * lost, replaced by garbage. Since this doesn't always happen,
318+ * let's poison such state so that we more quickly spot when
319+ * we falsely assume it has been preserved.
320+ */
321+ if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
322+ memset(engine->status_page.addr, POISON_INUSE, PAGE_SIZE);
323+
324+ /*
325+ * The kernel_context HWSP is stored in the status_page. As above,
326+ * that may be lost on resume/initialisation, and so we need to
327+ * reset the value in the HWSP.
328+ */
329+ sanitize_hwsp(engine);
330+
331+ /* And scrub the dirty cachelines for the HWSP */
332+ clflush_cache_range(engine->status_page.addr, PAGE_SIZE);
333+}
334+
324335 static void reset_prepare(struct intel_engine_cs *engine)
325336 {
326337 struct intel_uncore *uncore = engine->uncore;
@@ -440,10 +451,8 @@ static void reset_cancel(struct intel_engine_cs *engine) · 113 unmodified lines
440451 spin_lock_irqsave(&engine->active.lock, flags);
441452
442453 /* Mark all submitted requests as skipped. */
443− list_for_each_entry(request, &engine->active.requests, sched.link) {
444− i915_request_set_error_once(request, -EIO);
445− i915_request_mark_complete(request);
446− }
454+ list_for_each_entry(request, &engine->active.requests, sched.link)
455+ i915_request_mark_eio(request);
447456 intel_engine_signal_breadcrumbs(engine);
448457
449458 /* Remaining _unready_ requests will be nop'ed when submitted */
@@ -602,6 +611,7 @@ static int ring_context_pin(struct intel_context *ce, void *unused) · 152 unmodified lines
602611 static void ring_context_reset(struct intel_context *ce)
603612 {
604613 intel_ring_reset(ce->ring, ce->ring->emit);
614+ clear_bit(CONTEXT_VALID_BIT, &ce->flags);
605615 }
606616
607617 static const struct intel_context_ops ring_context_ops = {
@@ -886,7 +896,8 @@ static int switch_context(struct i915_request *rq) · 278 unmodified lines
886896 GEM_BUG_ON(HAS_EXECLISTS(engine->i915));
887897
888898 if (engine->wa_ctx.vma && ce != engine->kernel_context) {
889− if (engine->wa_ctx.vma->private != ce) {
899+ if (engine->wa_ctx.vma->private != ce &&
900+ i915_mitigate_clear_residuals()) {
890901 ret = clear_residuals(rq);
891902 if (ret)
892903 return ret;
@@ -1069,6 +1080,8 @@ static void setup_common(struct intel_engine_cs *engine) · 176 unmodified lines
10691080 setup_irq(engine);
10701081
10711082 engine->resume = xcs_resume;
1083+ engine->sanitize = xcs_sanitize;
1084+
10721085 engine->reset.prepare = reset_prepare;
10731086 engine->reset.rewind = reset_rewind;
10741087 engine->reset.cancel = reset_cancel;
@@ -1290,7 +1303,7 @@ int intel_ring_submission_setup(struct intel_engine_cs *engine) · 215 unmodified lines
12901303
12911304 GEM_BUG_ON(timeline->hwsp_ggtt != engine->status_page.vma);
12921305
1293− if (IS_HASWELL(engine->i915) && engine->class == RENDER_CLASS) {
1306+ if (IS_GEN(engine->i915, 7) && engine->class == RENDER_CLASS) {
12941307 err = gen7_ctx_switch_bb_init(engine);
12951308 if (err)
12961309 goto err_ring_unpin;
drivers/gpu/drm/i915/gt/intel_rps.c+35−24
@@ -400,7 +400,7 @@ static unsigned int gen5_invert_freq(struct intel_rps *rps, · 399 unmodified lines
400400 return val;
401401 }
402402
403−static bool gen5_rps_set(struct intel_rps *rps, u8 val)
403+static int __gen5_rps_set(struct intel_rps *rps, u8 val)
404404 {
405405 struct intel_uncore *uncore = rps_to_uncore(rps);
406406 u16 rgvswctl;
@@ -410,7 +410,7 @@ static bool gen5_rps_set(struct intel_rps *rps, u8 val) · 3 unmodified lines
410410 rgvswctl = intel_uncore_read16(uncore, MEMSWCTL);
411411 if (rgvswctl & MEMCTL_CMD_STS) {
412412 DRM_DEBUG("gpu busy, RCS change rejected\n");
413− return false; /* still busy with another command */
413+ return -EBUSY; /* still busy with another command */
414414 }
415415
416416 /* Invert the frequency bin into an ips delay */
@@ -426,7 +426,18 @@ static bool gen5_rps_set(struct intel_rps *rps, u8 val) · 9 unmodified lines
426426 rgvswctl |= MEMCTL_CMD_STS;
427427 intel_uncore_write16(uncore, MEMSWCTL, rgvswctl);
428428
429− return true;
429+ return 0;
430+}
431+
432+static int gen5_rps_set(struct intel_rps *rps, u8 val)
433+{
434+ int err;
435+
436+ spin_lock_irq(&mchdev_lock);
437+ err = __gen5_rps_set(rps, val);
438+ spin_unlock_irq(&mchdev_lock);
439+
440+ return err;
430441 }
431442
432443 static unsigned long intel_pxfreq(u32 vidfreq)
@@ -557,7 +568,7 @@ static bool gen5_rps_enable(struct intel_rps *rps) · 124 unmodified lines
557568 "stuck trying to change perf mode\n");
558569 mdelay(1);
559570
560− gen5_rps_set(rps, rps->cur_freq);
571+ __gen5_rps_set(rps, rps->cur_freq);
561572
562573 rps->ips.last_count1 = intel_uncore_read(uncore, DMIEC);
563574 rps->ips.last_count1 += intel_uncore_read(uncore, DDREC);
@@ -599,7 +610,7 @@ static void gen5_rps_disable(struct intel_rps *rps) · 35 unmodified lines
599610 intel_uncore_write(uncore, MEMINTRSTS, MEMINT_EVAL_CHG);
600611
601612 /* Go back to the starting frequency */
602− gen5_rps_set(rps, rps->idle_freq);
613+ __gen5_rps_set(rps, rps->idle_freq);
603614 mdelay(1);
604615 rgvswctl |= MEMCTL_CMD_STS;
605616 intel_uncore_write(uncore, MEMSWCTL, rgvswctl);
@@ -797,20 +808,19 @@ static int rps_set(struct intel_rps *rps, u8 val, bool update) · 191 unmodified lines
797808 struct drm_i915_private *i915 = rps_to_i915(rps);
798809 int err;
799810
800− if (INTEL_GEN(i915) < 6)
801− return 0;
802−
803811 if (val == rps->last_freq)
804812 return 0;
805813
806814 if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))
807815 err = vlv_rps_set(rps, val);
808− else
816+ else if (INTEL_GEN(i915) >= 6)
809817 err = gen6_rps_set(rps, val);
818+ else
819+ err = gen5_rps_set(rps, val);
810820 if (err)
811821 return err;
812822
813− if (update)
823+ if (update && INTEL_GEN(i915) >= 6)
814824 gen6_rps_set_thresholds(rps, val);
815825 rps->last_freq = val;
816826
@@ -852,6 +862,8 @@ void intel_rps_park(struct intel_rps *rps) · 35 unmodified lines
852862 {
853863 int adj;
854864
865+ GEM_BUG_ON(atomic_read(&rps->num_waiters));
866+
855867 if (!intel_rps_clear_active(rps))
856868 return;
857869
@@ -907,28 +919,27 @@ void intel_rps_park(struct intel_rps *rps) · 49 unmodified lines
907919
908920 void intel_rps_boost(struct i915_request *rq)
909921 {
910− struct intel_rps *rps = &READ_ONCE(rq->engine)->gt->rps;
911− unsigned long flags;
912−
913− if (i915_request_signaled(rq) || !intel_rps_is_active(rps))
922+ if (i915_request_signaled(rq) || i915_request_has_waitboost(rq))
914923 return;
915924
916925 /* Serializes with i915_request_retire() */
917− spin_lock_irqsave(&rq->lock, flags);
918− if (!i915_request_has_waitboost(rq) &&
919− !dma_fence_is_signaled_locked(&rq->fence)) {
920− set_bit(I915_FENCE_FLAG_BOOST, &rq->fence.flags);
926+ if (!test_and_set_bit(I915_FENCE_FLAG_BOOST, &rq->fence.flags)) {
927+ struct intel_rps *rps = &READ_ONCE(rq->engine)->gt->rps;
928+
929+ if (atomic_fetch_inc(&rps->num_waiters))
930+ return;
931+
932+ if (!intel_rps_is_active(rps))
933+ return;
921934
922935 GT_TRACE(rps_to_gt(rps), "boost fence:%llx:%llx\n",
923936 rq->fence.context, rq->fence.seqno);
924937
925− if (!atomic_fetch_inc(&rps->num_waiters) &&
926− READ_ONCE(rps->cur_freq) < rps->boost_freq)
938+ if (READ_ONCE(rps->cur_freq) < rps->boost_freq)
927939 schedule_work(&rps->work);
928940
929− atomic_inc(&rps->boosts);
941+ WRITE_ONCE(rps->boosts, rps->boosts + 1); /* debug only */
930942 }
931− spin_unlock_irqrestore(&rq->lock, flags);
932943 }
933944
934945 int intel_rps_set(struct intel_rps *rps, u8 val)
@@ -1798,7 +1809,7 @@ void gen5_rps_irq_handler(struct intel_rps *rps) · 863 unmodified lines
17981809 rps->min_freq_softlimit,
17991810 rps->max_freq_softlimit);
18001811
1801− if (new_freq != rps->cur_freq && gen5_rps_set(rps, new_freq))
1812+ if (new_freq != rps->cur_freq && !__gen5_rps_set(rps, new_freq))
18021813 rps->cur_freq = new_freq;
18031814
18041815 spin_unlock(&mchdev_lock);
@@ -2109,7 +2120,7 @@ bool i915_gpu_turbo_disable(void) · 304 unmodified lines
21092120
21102121 spin_lock_irq(&mchdev_lock);
21112122 rps->max_freq_softlimit = rps->min_freq;
2112− ret = gen5_rps_set(&i915->gt.rps, rps->min_freq);
2123+ ret = !__gen5_rps_set(&i915->gt.rps, rps->min_freq);
21132124 spin_unlock_irq(&mchdev_lock);
21142125
21152126 drm_dev_put(&i915->drm);
drivers/gpu/drm/i915/gt/intel_rps_types.h+1−1
@@ -93,7 +93,7 @@ struct intel_rps { · 92 unmodified lines
9393 } power;
9494
9595 atomic_t num_waiters;
96− atomic_t boosts;
96+ unsigned int boosts;
9797
9898 /* manual wa residency calculations */
9999 struct intel_rps_ei ei;
drivers/gpu/drm/i915/gt/intel_timeline.c+103−6
@@ -126,6 +126,10 @@ static void __rcu_cacheline_free(struct rcu_head *rcu) · 125 unmodified lines
126126 struct intel_timeline_cacheline *cl =
127127 container_of(rcu, typeof(*cl), rcu);
128128
129+ /* Must wait until after all *rq->hwsp are complete before removing */
130+ i915_gem_object_unpin_map(cl->hwsp->vma->obj);
131+ __idle_hwsp_free(cl->hwsp, ptr_unmask_bits(cl->vaddr, CACHELINE_BITS));
132+
129133 i915_active_fini(&cl->active);
130134 kfree(cl);
131135 }
@@ -133,11 +137,6 @@ static void __rcu_cacheline_free(struct rcu_head *rcu) · 1 unmodified line
133137 static void __idle_cacheline_free(struct intel_timeline_cacheline *cl)
134138 {
135139 GEM_BUG_ON(!i915_active_is_idle(&cl->active));
136−
137− i915_gem_object_unpin_map(cl->hwsp->vma->obj);
138− i915_vma_put(cl->hwsp->vma);
139− __idle_hwsp_free(cl->hwsp, ptr_unmask_bits(cl->vaddr, CACHELINE_BITS));
140−
141140 call_rcu(&cl->rcu, __rcu_cacheline_free);
142141 }
143142
@@ -179,7 +178,6 @@ cacheline_alloc(struct intel_timeline_hwsp *hwsp, unsigned int cacheline) · 35 unmodified lines
179178 return ERR_CAST(vaddr);
180179 }
181180
182− i915_vma_get(hwsp->vma);
183181 cl->hwsp = hwsp;
184182 cl->vaddr = page_pack_bits(vaddr, cacheline);
185183
@@ -321,6 +319,25 @@ __intel_timeline_create(struct intel_gt *gt, · 135 unmodified lines
321319 return timeline;
322320 }
323321
322+struct intel_timeline *
323+intel_timeline_create_from_engine(struct intel_engine_cs *engine,
324+ unsigned int offset)
325+{
326+ struct i915_vma *hwsp = engine->status_page.vma;
327+ struct intel_timeline *tl;
328+
329+ tl = __intel_timeline_create(engine->gt, hwsp, offset);
330+ if (IS_ERR(tl))
331+ return tl;
332+
333+ /* Borrow a nearby lock; we only create these timelines during init */
334+ mutex_lock(&hwsp->vm->mutex);
335+ list_add_tail(&tl->engine_link, &engine->status_page.timelines);
336+ mutex_unlock(&hwsp->vm->mutex);
337+
338+ return tl;
339+}
340+
324341 void __intel_timeline_pin(struct intel_timeline *tl)
325342 {
326343 GEM_BUG_ON(!atomic_read(&tl->pin_count));
@@ -617,6 +634,86 @@ void intel_gt_fini_timelines(struct intel_gt *gt) · 290 unmodified lines
617634 GEM_BUG_ON(!list_empty(&timelines->hwsp_free_list));
618635 }
619636
637+void intel_gt_show_timelines(struct intel_gt *gt,
638+ struct drm_printer *m,
639+ void (*show_request)(struct drm_printer *m,
640+ const struct i915_request *rq,
641+ const char *prefix,
642+ int indent))
643+{
644+ struct intel_gt_timelines *timelines = &gt->timelines;
645+ struct intel_timeline *tl, *tn;
646+ LIST_HEAD(free);
647+
648+ spin_lock(&timelines->lock);
649+ list_for_each_entry_safe(tl, tn, &timelines->active_list, link) {
650+ unsigned long count, ready, inflight;
651+ struct i915_request *rq, *rn;
652+ struct dma_fence *fence;
653+
654+ if (!mutex_trylock(&tl->mutex)) {
655+ drm_printf(m, "Timeline %llx: busy; skipping\n",
656+ tl->fence_context);
657+ continue;
658+ }
659+
660+ intel_timeline_get(tl);
661+ GEM_BUG_ON(!atomic_read(&tl->active_count));
662+ atomic_inc(&tl->active_count); /* pin the list element */
663+ spin_unlock(&timelines->lock);
664+
665+ count = 0;
666+ ready = 0;
667+ inflight = 0;
668+ list_for_each_entry_safe(rq, rn, &tl->requests, link) {
669+ if (i915_request_completed(rq))
670+ continue;
671+
672+ count++;
673+ if (i915_request_is_ready(rq))
674+ ready++;
675+ if (i915_request_is_active(rq))
676+ inflight++;
677+ }
678+
679+ drm_printf(m, "Timeline %llx: { ", tl->fence_context);
680+ drm_printf(m, "count: %lu, ready: %lu, inflight: %lu",
681+ count, ready, inflight);
682+ drm_printf(m, ", seqno: { current: %d, last: %d }",
683+ *tl->hwsp_seqno, tl->seqno);
684+ fence = i915_active_fence_get(&tl->last_request);
685+ if (fence) {
686+ drm_printf(m, ", engine: %s",
687+ to_request(fence)->engine->name);
688+ dma_fence_put(fence);
689+ }
690+ drm_printf(m, " }\n");
691+
692+ if (show_request) {
693+ list_for_each_entry_safe(rq, rn, &tl->requests, link)
694+ show_request(m, rq, "", 2);
695+ }
696+
697+ mutex_unlock(&tl->mutex);
698+ spin_lock(&timelines->lock);
699+
700+ /* Resume list iteration after reacquiring spinlock */
701+ list_safe_reset_next(tl, tn, link);
702+ if (atomic_dec_and_test(&tl->active_count))
703+ list_del(&tl->link);
704+
705+ /* Defer the final release to after the spinlock */
706+ if (refcount_dec_and_test(&tl->kref.refcount)) {
707+ GEM_BUG_ON(atomic_read(&tl->active_count));
708+ list_add(&tl->link, &free);
709+ }
710+ }
711+ spin_unlock(&timelines->lock);
712+
713+ list_for_each_entry_safe(tl, tn, &free, link)
714+ __intel_timeline_free(&tl->kref);
715+}
716+
620717 #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
621718 #include "gt/selftests/mock_timeline.c"
622719 #include "gt/selftest_timeline.c"
drivers/gpu/drm/i915/gt/intel_timeline.h+18−7
@@ -31,6 +31,8 @@ · 30 unmodified lines
3131 #include "i915_syncmap.h"
3232 #include "intel_timeline_types.h"
3333
34+struct drm_printer;
35+
3436 struct intel_timeline *
3537 __intel_timeline_create(struct intel_gt *gt,
3638 struct i915_vma *global_hwsp,
@@ -42,14 +44,9 @@ intel_timeline_create(struct intel_gt *gt) · 5 unmodified lines
4244 return __intel_timeline_create(gt, NULL, 0);
4345 }
4446
45−static inline struct intel_timeline *
47+struct intel_timeline *
4648 intel_timeline_create_from_engine(struct intel_engine_cs *engine,
47− unsigned int offset)
48−{
49− return __intel_timeline_create(engine->gt,
50− engine->status_page.vma,
51− offset);
52−}
49+ unsigned int offset);
5350
5451 static inline struct intel_timeline *
5552 intel_timeline_get(struct intel_timeline *timeline)
@@ -106,4 +103,18 @@ int intel_timeline_read_hwsp(struct i915_request *from, · 50 unmodified lines
106103 void intel_gt_init_timelines(struct intel_gt *gt);
107104 void intel_gt_fini_timelines(struct intel_gt *gt);
108105
106+void intel_gt_show_timelines(struct intel_gt *gt,
107+ struct drm_printer *m,
108+ void (*show_request)(struct drm_printer *m,
109+ const struct i915_request *rq,
110+ const char *prefix,
111+ int indent));
112+
113+static inline bool
114+intel_timeline_is_last(const struct intel_timeline *tl,
115+ const struct i915_request *rq)
116+{
117+ return list_is_last_rcu(&rq->link, &tl->requests);
118+}
119+
109120 #endif
drivers/gpu/drm/i915/gt/intel_timeline_types.h+2−0
@@ -84,6 +84,8 @@ struct intel_timeline { · 83 unmodified lines
8484 struct list_head link;
8585 struct intel_gt *gt;
8686
87+ struct list_head engine_link;
88+
8789 struct kref kref;
8890 struct rcu_head rcu;
8991 };
drivers/gpu/drm/i915/gt/intel_workarounds.c+309−335
@@ -7,6 +7,7 @@ · 6 unmodified lines
77 #include "i915_drv.h"
88 #include "intel_context.h"
99 #include "intel_engine_pm.h"
10+#include "intel_gpu_commands.h"
1011 #include "intel_gt.h"
1112 #include "intel_ring.h"
1213 #include "intel_workarounds.h"
@@ -194,7 +195,7 @@ static void wa_add(struct i915_wa_list *wal, i915_reg_t reg, · 181 unmodified lines
194195 }
195196
196197 static void
197−wa_write_masked_or(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set)
198+wa_write_clr_set(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set)
198199 {
199200 wa_add(wal, reg, clear, set, clear);
200201 }
@@ -202,21 +203,32 @@ wa_write_masked_or(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set) · 1 unmodified line
202203 static void
203204 wa_write(struct i915_wa_list *wal, i915_reg_t reg, u32 set)
204205 {
205− wa_write_masked_or(wal, reg, ~0, set);
206+ wa_write_clr_set(wal, reg, ~0, set);
206207 }
207208
208209 static void
209210 wa_write_or(struct i915_wa_list *wal, i915_reg_t reg, u32 set)
210211 {
211− wa_write_masked_or(wal, reg, set, set);
212+ wa_write_clr_set(wal, reg, set, set);
212213 }
213214
214215 static void
215216 wa_write_clr(struct i915_wa_list *wal, i915_reg_t reg, u32 clr)
216217 {
217− wa_write_masked_or(wal, reg, clr, 0);
218+ wa_write_clr_set(wal, reg, clr, 0);
218219 }
219220
221+/*
222+ * WA operations on "masked register". A masked register has the upper 16 bits
223+ * documented as "masked" in b-spec. Its purpose is to allow writing to just a
224+ * portion of the register without a rmw: you simply write in the upper 16 bits
225+ * the mask of bits you are going to modify.
226+ *
227+ * The wa_masked_* family of functions already does the necessary operations to
228+ * calculate the mask based on the parameters passed, so user only has to
229+ * provide the lower 16 bits of that register.
230+ */
231+
220232 static void
221233 wa_masked_en(struct i915_wa_list *wal, i915_reg_t reg, u32 val)
222234 {
@@ -229,38 +241,36 @@ wa_masked_dis(struct i915_wa_list *wal, i915_reg_t reg, u32 val) · 6 unmodified lines
229241 wa_add(wal, reg, 0, _MASKED_BIT_DISABLE(val), val);
230242 }
231243
232−#define WA_SET_BIT_MASKED(addr, mask) \
233− wa_masked_en(wal, (addr), (mask))
234−
235−#define WA_CLR_BIT_MASKED(addr, mask) \
236− wa_masked_dis(wal, (addr), (mask))
237−
238−#define WA_SET_FIELD_MASKED(addr, mask, value) \
239− wa_write_masked_or(wal, (addr), 0, _MASKED_FIELD((mask), (value)))
244+static void
245+wa_masked_field_set(struct i915_wa_list *wal, i915_reg_t reg,
246+ u32 mask, u32 val)
247+{
248+ wa_add(wal, reg, 0, _MASKED_FIELD(mask, val), mask);
249+}
240250
241251 static void gen6_ctx_workarounds_init(struct intel_engine_cs *engine,
242252 struct i915_wa_list *wal)
243253 {
244− WA_SET_BIT_MASKED(INSTPM, INSTPM_FORCE_ORDERING);
254+ wa_masked_en(wal, INSTPM, INSTPM_FORCE_ORDERING);
245255 }
246256
247257 static void gen7_ctx_workarounds_init(struct intel_engine_cs *engine,
248258 struct i915_wa_list *wal)
249259 {
250− WA_SET_BIT_MASKED(INSTPM, INSTPM_FORCE_ORDERING);
260+ wa_masked_en(wal, INSTPM, INSTPM_FORCE_ORDERING);
251261 }
252262
253263 static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine,
254264 struct i915_wa_list *wal)
255265 {
256− WA_SET_BIT_MASKED(INSTPM, INSTPM_FORCE_ORDERING);
266+ wa_masked_en(wal, INSTPM, INSTPM_FORCE_ORDERING);
257267
258268 /* WaDisableAsyncFlipPerfMode:bdw,chv */
259− WA_SET_BIT_MASKED(MI_MODE, ASYNC_FLIP_PERF_DISABLE);
269+ wa_masked_en(wal, MI_MODE, ASYNC_FLIP_PERF_DISABLE);
260270
261271 /* WaDisablePartialInstShootdown:bdw,chv */
262− WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN,
263− PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);
272+ wa_masked_en(wal, GEN8_ROW_CHICKEN,
273+ PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);
264274
265275 /* Use Force Non-Coherent whenever executing a 3D context. This is a
266276 * workaround for for a possible hang in the unlikely event a TLB
@@ -268,9 +278,9 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, · 1 unmodified line
268278 */
269279 /* WaForceEnableNonCoherent:bdw,chv */
270280 /* WaHdcDisableFetchWhenMasked:bdw,chv */
271− WA_SET_BIT_MASKED(HDC_CHICKEN0,
272− HDC_DONOT_FETCH_MEM_WHEN_MASKED |
273− HDC_FORCE_NON_COHERENT);
281+ wa_masked_en(wal, HDC_CHICKEN0,
282+ HDC_DONOT_FETCH_MEM_WHEN_MASKED |
283+ HDC_FORCE_NON_COHERENT);
274284
275285 /* From the Haswell PRM, Command Reference: Registers, CACHE_MODE_0:
276286 * "The Hierarchical Z RAW Stall Optimization allows non-overlapping
@@ -280,10 +290,10 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, · 3 unmodified lines
280290 *
281291 * This optimization is off by default for BDW and CHV; turn it on.
282292 */
283− WA_CLR_BIT_MASKED(CACHE_MODE_0_GEN7, HIZ_RAW_STALL_OPT_DISABLE);
293+ wa_masked_dis(wal, CACHE_MODE_0_GEN7, HIZ_RAW_STALL_OPT_DISABLE);
284294
285295 /* Wa4x4STCOptimizationDisable:bdw,chv */
286− WA_SET_BIT_MASKED(CACHE_MODE_1, GEN8_4x4_STC_OPTIMIZATION_DISABLE);
296+ wa_masked_en(wal, CACHE_MODE_1, GEN8_4x4_STC_OPTIMIZATION_DISABLE);
287297
288298 /*
289299 * BSpec recommends 8x4 when MSAA is used,
@@ -293,7 +303,7 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, · 3 unmodified lines
293303 * disable bit, which we don't touch here, but it's good
294304 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
295305 */
296− WA_SET_FIELD_MASKED(GEN7_GT_MODE,
306+ wa_masked_field_set(wal, GEN7_GT_MODE,
297307 GEN6_WIZ_HASHING_MASK,
298308 GEN6_WIZ_HASHING_16x4);
299309 }
@@ -306,24 +316,24 @@ static void bdw_ctx_workarounds_init(struct intel_engine_cs *engine, · 6 unmodified lines
306316 gen8_ctx_workarounds_init(engine, wal);
307317
308318 /* WaDisableThreadStallDopClockGating:bdw (pre-production) */
309− WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);
319+ wa_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);
310320
311321 /* WaDisableDopClockGating:bdw
312322 *
313323 * Also see the related UCGTCL1 write in bdw_init_clock_gating()
314324 * to disable EUTC clock gating.
315325 */
316− WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,
317− DOP_CLOCK_GATING_DISABLE);
326+ wa_masked_en(wal, GEN7_ROW_CHICKEN2,
327+ DOP_CLOCK_GATING_DISABLE);
318328
319− WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN3,
320− GEN8_SAMPLER_POWER_BYPASS_DIS);
329+ wa_masked_en(wal, HALF_SLICE_CHICKEN3,
330+ GEN8_SAMPLER_POWER_BYPASS_DIS);
321331
322− WA_SET_BIT_MASKED(HDC_CHICKEN0,
323− /* WaForceContextSaveRestoreNonCoherent:bdw */
324− HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |
325− /* WaDisableFenceDestinationToSLM:bdw (pre-prod) */
326− (IS_BDW_GT3(i915) ? HDC_FENCE_DEST_SLM_DISABLE : 0));
332+ wa_masked_en(wal, HDC_CHICKEN0,
333+ /* WaForceContextSaveRestoreNonCoherent:bdw */
334+ HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |
335+ /* WaDisableFenceDestinationToSLM:bdw (pre-prod) */
336+ (IS_BDW_GT3(i915) ? HDC_FENCE_DEST_SLM_DISABLE : 0));
327337 }
328338
329339 static void chv_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -332,10 +342,10 @@ static void chv_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
332342 gen8_ctx_workarounds_init(engine, wal);
333343
334344 /* WaDisableThreadStallDopClockGating:chv */
335− WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);
345+ wa_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);
336346
337347 /* Improve HiZ throughput on CHV. */
338− WA_SET_BIT_MASKED(HIZ_CHICKEN, CHV_HZ_8X8_MODE_IN_1X);
348+ wa_masked_en(wal, HIZ_CHICKEN, CHV_HZ_8X8_MODE_IN_1X);
339349 }
340350
341351 static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -349,38 +359,38 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, · 7 unmodified lines
349359 * Must match Display Engine. See
350360 * WaCompressedResourceDisplayNewHashMode.
351361 */
352− WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,
353− GEN9_PBE_COMPRESSED_HASH_SELECTION);
354− WA_SET_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN7,
355− GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR);
362+ wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
363+ GEN9_PBE_COMPRESSED_HASH_SELECTION);
364+ wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7,
365+ GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR);
356366 }
357367
358368 /* WaClearFlowControlGpgpuContextSave:skl,bxt,kbl,glk,cfl */
359369 /* WaDisablePartialInstShootdown:skl,bxt,kbl,glk,cfl */
360− WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN,
361− FLOW_CONTROL_ENABLE |
362− PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);
370+ wa_masked_en(wal, GEN8_ROW_CHICKEN,
371+ FLOW_CONTROL_ENABLE |
372+ PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);
363373
364374 /* WaEnableYV12BugFixInHalfSliceChicken7:skl,bxt,kbl,glk,cfl */
365375 /* WaEnableSamplerGPGPUPreemptionSupport:skl,bxt,kbl,cfl */
366− WA_SET_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN7,
367− GEN9_ENABLE_YV12_BUGFIX |
368− GEN9_ENABLE_GPGPU_PREEMPTION);
376+ wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7,
377+ GEN9_ENABLE_YV12_BUGFIX |
378+ GEN9_ENABLE_GPGPU_PREEMPTION);
369379
370380 /* Wa4x4STCOptimizationDisable:skl,bxt,kbl,glk,cfl */
371381 /* WaDisablePartialResolveInVc:skl,bxt,kbl,cfl */
372− WA_SET_BIT_MASKED(CACHE_MODE_1,
373− GEN8_4x4_STC_OPTIMIZATION_DISABLE |
374− GEN9_PARTIAL_RESOLVE_IN_VC_DISABLE);
382+ wa_masked_en(wal, CACHE_MODE_1,
383+ GEN8_4x4_STC_OPTIMIZATION_DISABLE |
384+ GEN9_PARTIAL_RESOLVE_IN_VC_DISABLE);
375385
376386 /* WaCcsTlbPrefetchDisable:skl,bxt,kbl,glk,cfl */
377− WA_CLR_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN5,
378− GEN9_CCS_TLB_PREFETCH_ENABLE);
387+ wa_masked_dis(wal, GEN9_HALF_SLICE_CHICKEN5,
388+ GEN9_CCS_TLB_PREFETCH_ENABLE);
379389
380390 /* WaForceContextSaveRestoreNonCoherent:skl,bxt,kbl,cfl */
381− WA_SET_BIT_MASKED(HDC_CHICKEN0,
382− HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |
383− HDC_FORCE_CSR_NON_COHERENT_OVR_DISABLE);
391+ wa_masked_en(wal, HDC_CHICKEN0,
392+ HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |
393+ HDC_FORCE_CSR_NON_COHERENT_OVR_DISABLE);
384394
385395 /* WaForceEnableNonCoherent and WaDisableHDCInvalidation are
386396 * both tied to WaForceContextSaveRestoreNonCoherent
@@ -396,19 +406,19 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, · 9 unmodified lines
396406 */
397407
398408 /* WaForceEnableNonCoherent:skl,bxt,kbl,cfl */
399− WA_SET_BIT_MASKED(HDC_CHICKEN0,
400− HDC_FORCE_NON_COHERENT);
409+ wa_masked_en(wal, HDC_CHICKEN0,
410+ HDC_FORCE_NON_COHERENT);
401411
402412 /* WaDisableSamplerPowerBypassForSOPingPong:skl,bxt,kbl,cfl */
403413 if (IS_SKYLAKE(i915) ||
404414 IS_KABYLAKE(i915) ||
405415 IS_COFFEELAKE(i915) ||
406416 IS_COMETLAKE(i915))
407− WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN3,
408− GEN8_SAMPLER_POWER_BYPASS_DIS);
417+ wa_masked_en(wal, HALF_SLICE_CHICKEN3,
418+ GEN8_SAMPLER_POWER_BYPASS_DIS);
409419
410420 /* WaDisableSTUnitPowerOptimization:skl,bxt,kbl,glk,cfl */
411− WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN2, GEN8_ST_PO_DISABLE);
421+ wa_masked_en(wal, HALF_SLICE_CHICKEN2, GEN8_ST_PO_DISABLE);
412422
413423 /*
414424 * Supporting preemption with fine-granularity requires changes in the
@@ -422,16 +432,16 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, · 7 unmodified lines
422432 */
423433
424434 /* WaDisable3DMidCmdPreemption:skl,bxt,glk,cfl,[cnl] */
425− WA_CLR_BIT_MASKED(GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);
435+ wa_masked_dis(wal, GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);
426436
427437 /* WaDisableGPGPUMidCmdPreemption:skl,bxt,blk,cfl,[cnl] */
428− WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,
438+ wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
429439 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
430440 GEN9_PREEMPT_GPGPU_COMMAND_LEVEL);
431441
432442 /* WaClearHIZ_WM_CHICKEN3:bxt,glk */
433443 if (IS_GEN9_LP(i915))
434− WA_SET_BIT_MASKED(GEN9_WM_CHICKEN3, GEN9_FACTOR_IN_CLR_VAL_HIZ);
444+ wa_masked_en(wal, GEN9_WM_CHICKEN3, GEN9_FACTOR_IN_CLR_VAL_HIZ);
435445 }
436446
437447 static void skl_tune_iz_hashing(struct intel_engine_cs *engine,
@@ -465,7 +475,7 @@ static void skl_tune_iz_hashing(struct intel_engine_cs *engine, · 27 unmodified lines
465475 return;
466476
467477 /* Tune IZ hashing. See intel_device_info_runtime_init() */
468− WA_SET_FIELD_MASKED(GEN7_GT_MODE,
478+ wa_masked_field_set(wal, GEN7_GT_MODE,
469479 GEN9_IZ_HASHING_MASK(2) |
470480 GEN9_IZ_HASHING_MASK(1) |
471481 GEN9_IZ_HASHING_MASK(0),
@@ -487,12 +497,12 @@ static void bxt_ctx_workarounds_init(struct intel_engine_cs *engine, · 15 unmodified lines
487497 gen9_ctx_workarounds_init(engine, wal);
488498
489499 /* WaDisableThreadStallDopClockGating:bxt */
490− WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN,
491− STALL_DOP_GATING_DISABLE);
500+ wa_masked_en(wal, GEN8_ROW_CHICKEN,
501+ STALL_DOP_GATING_DISABLE);
492502
493503 /* WaToEnableHwFixForPushConstHWBug:bxt */
494− WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,
495− GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
504+ wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
505+ GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
496506 }
497507
498508 static void kbl_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -504,12 +514,12 @@ static void kbl_ctx_workarounds_init(struct intel_engine_cs *engine, · 5 unmodified lines
504514
505515 /* WaToEnableHwFixForPushConstHWBug:kbl */
506516 if (IS_KBL_GT_REVID(i915, KBL_REVID_C0, REVID_FOREVER))
507− WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,
508− GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
517+ wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
518+ GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
509519
510520 /* WaDisableSbeCacheDispatchPortSharing:kbl */
511− WA_SET_BIT_MASKED(GEN7_HALF_SLICE_CHICKEN1,
512− GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);
521+ wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1,
522+ GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);
513523 }
514524
515525 static void glk_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -518,8 +528,8 @@ static void glk_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
518528 gen9_ctx_workarounds_init(engine, wal);
519529
520530 /* WaToEnableHwFixForPushConstHWBug:glk */
521− WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,
522− GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
531+ wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
532+ GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
523533 }
524534
525535 static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -528,41 +538,41 @@ static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
528538 gen9_ctx_workarounds_init(engine, wal);
529539
530540 /* WaToEnableHwFixForPushConstHWBug:cfl */
531− WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,
532− GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
541+ wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
542+ GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
533543
534544 /* WaDisableSbeCacheDispatchPortSharing:cfl */
535− WA_SET_BIT_MASKED(GEN7_HALF_SLICE_CHICKEN1,
536− GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);
545+ wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1,
546+ GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);
537547 }
538548
539549 static void cnl_ctx_workarounds_init(struct intel_engine_cs *engine,
540550 struct i915_wa_list *wal)
541551 {
542552 /* WaForceContextSaveRestoreNonCoherent:cnl */
543− WA_SET_BIT_MASKED(CNL_HDC_CHICKEN0,
544− HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT);
553+ wa_masked_en(wal, CNL_HDC_CHICKEN0,
554+ HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT);
545555
546556 /* WaDisableReplayBufferBankArbitrationOptimization:cnl */
547− WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,
548− GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
557+ wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
558+ GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
549559
550560 /* WaPushConstantDereferenceHoldDisable:cnl */
551− WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2, PUSH_CONSTANT_DEREF_DISABLE);
561+ wa_masked_en(wal, GEN7_ROW_CHICKEN2, PUSH_CONSTANT_DEREF_DISABLE);
552562
553563 /* FtrEnableFastAnisoL1BankingFix:cnl */
554− WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN3, CNL_FAST_ANISO_L1_BANKING_FIX);
564+ wa_masked_en(wal, HALF_SLICE_CHICKEN3, CNL_FAST_ANISO_L1_BANKING_FIX);
555565
556566 /* WaDisable3DMidCmdPreemption:cnl */
557− WA_CLR_BIT_MASKED(GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);
567+ wa_masked_dis(wal, GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);
558568
559569 /* WaDisableGPGPUMidCmdPreemption:cnl */
560− WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,
570+ wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
561571 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
562572 GEN9_PREEMPT_GPGPU_COMMAND_LEVEL);
563573
564574 /* WaDisableEarlyEOT:cnl */
565− WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, DISABLE_EARLY_EOT);
575+ wa_masked_en(wal, GEN8_ROW_CHICKEN, DISABLE_EARLY_EOT);
566576 }
567577
568578 static void icl_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -580,8 +590,8 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, · 11 unmodified lines
580590 * Formerly known as WaPushConstantDereferenceHoldDisable
581591 */
582592 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_B0))
583− WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,
584− PUSH_CONSTANT_DEREF_DISABLE);
593+ wa_masked_en(wal, GEN7_ROW_CHICKEN2,
594+ PUSH_CONSTANT_DEREF_DISABLE);
585595
586596 /* WaForceEnableNonCoherent:icl
587597 * This is not the same workaround as in early Gen9 platforms, where
@@ -590,40 +600,40 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
590600 * (the register is whitelisted in hardware now, so UMDs can opt in
591601 * for coherency if they have a good reason).
592602 */
593− WA_SET_BIT_MASKED(ICL_HDC_MODE, HDC_FORCE_NON_COHERENT);
603+ wa_masked_en(wal, ICL_HDC_MODE, HDC_FORCE_NON_COHERENT);
594604
595605 /* Wa_2006611047:icl (pre-prod)
596606 * Formerly known as WaDisableImprovedTdlClkGating
597607 */
598608 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))
599− WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,
600− GEN11_TDL_CLOCK_GATING_FIX_DISABLE);
609+ wa_masked_en(wal, GEN7_ROW_CHICKEN2,
610+ GEN11_TDL_CLOCK_GATING_FIX_DISABLE);
601611
602612 /* Wa_2006665173:icl (pre-prod) */
603613 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))
604− WA_SET_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,
605− GEN11_BLEND_EMB_FIX_DISABLE_IN_RCC);
614+ wa_masked_en(wal, GEN11_COMMON_SLICE_CHICKEN3,
615+ GEN11_BLEND_EMB_FIX_DISABLE_IN_RCC);
606616
607617 /* WaEnableFloatBlendOptimization:icl */
608− wa_write_masked_or(wal,
609− GEN10_CACHE_MODE_SS,
610− 0, /* write-only, so skip validation */
611− _MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE));
618+ wa_write_clr_set(wal,
619+ GEN10_CACHE_MODE_SS,
620+ 0, /* write-only, so skip validation */
621+ _MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE));
612622
613623 /* WaDisableGPGPUMidThreadPreemption:icl */
614− WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,
624+ wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
615625 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
616626 GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);
617627
618628 /* allow headerless messages for preemptible GPGPU context */
619− WA_SET_BIT_MASKED(GEN10_SAMPLER_MODE,
620− GEN11_SAMPLER_ENABLE_HEADLESS_MSG);
629+ wa_masked_en(wal, GEN10_SAMPLER_MODE,
630+ GEN11_SAMPLER_ENABLE_HEADLESS_MSG);
621631
622632 /* Wa_1604278689:icl,ehl */
623633 wa_write(wal, IVB_FBC_RT_BASE, 0xFFFFFFFF & ~ILK_FBC_RT_VALID);
624− wa_write_masked_or(wal, IVB_FBC_RT_BASE_UPPER,
625− 0, /* write-only register; skip validation */
626− 0xFFFFFFFF);
634+ wa_write_clr_set(wal, IVB_FBC_RT_BASE_UPPER,
635+ 0, /* write-only register; skip validation */
636+ 0xFFFFFFFF);
627637
628638 /* Wa_1406306137:icl,ehl */
629639 wa_masked_en(wal, GEN9_ROW_CHICKEN4, GEN11_DIS_PICK_2ND_EU);
@@ -643,11 +653,11 @@ static void gen12_ctx_workarounds_init(struct intel_engine_cs *engine, · 13 unmodified lines
643653 * Wa_14010443199:rkl
644654 * Wa_14010698770:rkl
645655 */
646− WA_SET_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,
647− GEN12_DISABLE_CPS_AWARE_COLOR_PIPE);
656+ wa_masked_en(wal, GEN11_COMMON_SLICE_CHICKEN3,
657+ GEN12_DISABLE_CPS_AWARE_COLOR_PIPE);
648658
649659 /* WaDisableGPGPUMidThreadPreemption:gen12 */
650− WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,
660+ wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
651661 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
652662 GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);
653663 }
@@ -680,12 +690,22 @@ static void dg1_ctx_workarounds_init(struct intel_engine_cs *engine, · 26 unmodified lines
680690 gen12_ctx_workarounds_init(engine, wal);
681691
682692 /* Wa_1409044764 */
683− WA_CLR_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,
684− DG1_FLOAT_POINT_BLEND_OPT_STRICT_MODE_EN);
693+ wa_masked_dis(wal, GEN11_COMMON_SLICE_CHICKEN3,
694+ DG1_FLOAT_POINT_BLEND_OPT_STRICT_MODE_EN);
685695
686696 /* Wa_22010493298 */
687− WA_SET_BIT_MASKED(HIZ_CHICKEN,
688− DG1_HZ_READ_SUPPRESSION_OPTIMIZATION_DISABLE);
697+ wa_masked_en(wal, HIZ_CHICKEN,
698+ DG1_HZ_READ_SUPPRESSION_OPTIMIZATION_DISABLE);
699+
700+ /*
701+ * Wa_16011163337
702+ *
703+ * Like in tgl_ctx_workarounds_init(), read verification is ignored due
704+ * to Wa_1608008084.
705+ */
706+ wa_add(wal,
707+ FF_MODE2,
708+ FF_MODE2_GS_TIMER_MASK, FF_MODE2_GS_TIMER_224, 0);
689709 }
690710
691711 static void
@@ -804,57 +824,11 @@ ilk_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 112 unmodified lines
804824 static void
805825 snb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)
806826 {
807− /* WaDisableHiZPlanesWhenMSAAEnabled:snb */
808− wa_masked_en(wal,
809− _3D_CHICKEN,
810− _3D_CHICKEN_HIZ_PLANE_DISABLE_MSAA_4X_SNB);
811−
812− /* WaDisable_RenderCache_OperationalFlush:snb */
813− wa_masked_dis(wal, CACHE_MODE_0, RC_OP_FLUSH_ENABLE);
814−
815− /*
816− * BSpec recommends 8x4 when MSAA is used,
817− * however in practice 16x4 seems fastest.
818− *
819− * Note that PS/WM thread counts depend on the WIZ hashing
820− * disable bit, which we don't touch here, but it's good
821− * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
822− */
823− wa_add(wal,
824− GEN6_GT_MODE, 0,
825− _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
826− GEN6_WIZ_HASHING_16x4);
827−
828− wa_masked_dis(wal, CACHE_MODE_0, CM0_STC_EVICT_DISABLE_LRA_SNB);
829−
830− wa_masked_en(wal,
831− _3D_CHICKEN3,
832− /* WaStripsFansDisableFastClipPerformanceFix:snb */
833− _3D_CHICKEN3_SF_DISABLE_FASTCLIP_CULL |
834− /*
835− * Bspec says:
836− * "This bit must be set if 3DSTATE_CLIP clip mode is set
837− * to normal and 3DSTATE_SF number of SF output attributes
838− * is more than 16."
839− */
840− _3D_CHICKEN3_SF_DISABLE_PIPELINED_ATTR_FETCH);
841827 }
842828
843829 static void
844830 ivb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)
845831 {
846− /* WaDisableEarlyCull:ivb */
847− wa_masked_en(wal, _3D_CHICKEN3, _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
848−
849− /* WaDisablePSDDualDispatchEnable:ivb */
850− if (IS_IVB_GT1(i915))
851− wa_masked_en(wal,
852− GEN7_HALF_SLICE_CHICKEN1,
853− GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
854−
855− /* WaDisable_RenderCache_OperationalFlush:ivb */
856− wa_masked_dis(wal, CACHE_MODE_0_GEN7, RC_OP_FLUSH_ENABLE);
857−
858832 /* Apply the WaDisableRHWOOptimizationForRenderHang:ivb workaround. */
859833 wa_masked_dis(wal,
860834 GEN7_COMMON_SLICE_CHICKEN1,
@@ -866,90 +840,14 @@ ivb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 5 unmodified lines
866840
867841 /* WaForceL3Serialization:ivb */
868842 wa_write_clr(wal, GEN7_L3SQCREG4, L3SQ_URB_READ_CAM_MATCH_DISABLE);
869−
870− /*
871− * WaVSThreadDispatchOverride:ivb,vlv
872− *
873− * This actually overrides the dispatch
874− * mode for all thread types.
875− */
876− wa_write_masked_or(wal, GEN7_FF_THREAD_MODE,
877− GEN7_FF_SCHED_MASK,
878− GEN7_FF_TS_SCHED_HW |
879− GEN7_FF_VS_SCHED_HW |
880− GEN7_FF_DS_SCHED_HW);
881−
882− if (0) { /* causes HiZ corruption on ivb:gt1 */
883− /* enable HiZ Raw Stall Optimization */
884− wa_masked_dis(wal, CACHE_MODE_0_GEN7, HIZ_RAW_STALL_OPT_DISABLE);
885− }
886−
887− /* WaDisable4x2SubspanOptimization:ivb */
888− wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
889−
890− /*
891− * BSpec recommends 8x4 when MSAA is used,
892− * however in practice 16x4 seems fastest.
893− *
894− * Note that PS/WM thread counts depend on the WIZ hashing
895− * disable bit, which we don't touch here, but it's good
896− * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
897− */
898− wa_add(wal, GEN7_GT_MODE, 0,
899− _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
900− GEN6_WIZ_HASHING_16x4);
901843 }
902844
903845 static void
904846 vlv_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)
905847 {
906− /* WaDisableEarlyCull:vlv */
907− wa_masked_en(wal, _3D_CHICKEN3, _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
908−
909− /* WaPsdDispatchEnable:vlv */
910− /* WaDisablePSDDualDispatchEnable:vlv */
911− wa_masked_en(wal,
912− GEN7_HALF_SLICE_CHICKEN1,
913− GEN7_MAX_PS_THREAD_DEP |
914− GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
915−
916− /* WaDisable_RenderCache_OperationalFlush:vlv */
917− wa_masked_dis(wal, CACHE_MODE_0_GEN7, RC_OP_FLUSH_ENABLE);
918−
919848 /* WaForceL3Serialization:vlv */
920849 wa_write_clr(wal, GEN7_L3SQCREG4, L3SQ_URB_READ_CAM_MATCH_DISABLE);
921850
922− /*
923− * WaVSThreadDispatchOverride:ivb,vlv
924− *
925− * This actually overrides the dispatch
926− * mode for all thread types.
927− */
928− wa_write_masked_or(wal,
929− GEN7_FF_THREAD_MODE,
930− GEN7_FF_SCHED_MASK,
931− GEN7_FF_TS_SCHED_HW |
932− GEN7_FF_VS_SCHED_HW |
933− GEN7_FF_DS_SCHED_HW);
934−
935− /*
936− * BSpec says this must be set, even though
937− * WaDisable4x2SubspanOptimization isn't listed for VLV.
938− */
939− wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
940−
941− /*
942− * BSpec recommends 8x4 when MSAA is used,
943− * however in practice 16x4 seems fastest.
944− *
945− * Note that PS/WM thread counts depend on the WIZ hashing
946− * disable bit, which we don't touch here, but it's good
947− * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
948− */
949− wa_add(wal, GEN7_GT_MODE, 0,
950− _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
951− GEN6_WIZ_HASHING_16x4);
952−
953851 /*
954852 * WaIncreaseL3CreditsForVLVB0:vlv
955853 * This is the hardware default actually.
@@ -970,31 +868,6 @@ hsw_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 14 unmodified lines
970868
971869 /* WaVSRefCountFullforceMissDisable:hsw */
972870 wa_write_clr(wal, GEN7_FF_THREAD_MODE, GEN7_FF_VS_REF_CNT_FFME);
973−
974− wa_masked_dis(wal,
975− CACHE_MODE_0_GEN7,
976− /* WaDisable_RenderCache_OperationalFlush:hsw */
977− RC_OP_FLUSH_ENABLE |
978− /* enable HiZ Raw Stall Optimization */
979− HIZ_RAW_STALL_OPT_DISABLE);
980−
981− /* WaDisable4x2SubspanOptimization:hsw */
982− wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
983−
984− /*
985− * BSpec recommends 8x4 when MSAA is used,
986− * however in practice 16x4 seems fastest.
987− *
988− * Note that PS/WM thread counts depend on the WIZ hashing
989− * disable bit, which we don't touch here, but it's good
990− * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
991− */
992− wa_add(wal, GEN7_GT_MODE, 0,
993− _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
994− GEN6_WIZ_HASHING_16x4);
995−
996− /* WaSampleCChickenBitEnable:hsw */
997− wa_masked_en(wal, HALF_SLICE_CHICKEN3, HSW_SAMPLE_C_PERFORMANCE);
998871 }
999872
1000873 static void
@@ -1164,7 +1037,7 @@ wa_init_mcr(struct drm_i915_private *i915, struct i915_wa_list *wal) · 163 unmodified lines
11641037
11651038 drm_dbg(&i915->drm, "MCR slice/subslice = %x\n", mcr);
11661039
1167− wa_write_masked_or(wal, GEN8_MCR_SELECTOR, mcr_mask, mcr);
1040+ wa_write_clr_set(wal, GEN8_MCR_SELECTOR, mcr_mask, mcr);
11681041 }
11691042
11701043 static void
@@ -1189,10 +1062,10 @@ icl_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 18 unmodified lines
11891062 GAMT_ECO_ENABLE_IN_PLACE_DECOMPRESS);
11901063
11911064 /* WaModifyGamTlbPartitioning:icl */
1192− wa_write_masked_or(wal,
1193− GEN11_GACB_PERF_CTRL,
1194− GEN11_HASH_CTRL_MASK,
1195− GEN11_HASH_CTRL_BIT0 | GEN11_HASH_CTRL_BIT4);
1065+ wa_write_clr_set(wal,
1066+ GEN11_GACB_PERF_CTRL,
1067+ GEN11_HASH_CTRL_MASK,
1068+ GEN11_HASH_CTRL_BIT0 | GEN11_HASH_CTRL_BIT4);
11961069
11971070 /* Wa_1405766107:icl
11981071 * Formerly known as WaCL2SFHalfMaxAlloc
@@ -1260,6 +1133,11 @@ tgl_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 61 unmodified lines
12601133 wa_write_or(wal,
12611134 SLICE_UNIT_LEVEL_CLKGATE,
12621135 L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS);
1136+
1137+ /* Wa_1408615072:tgl[a0] */
1138+ if (IS_TGL_UY_GT_REVID(i915, TGL_REVID_A0, TGL_REVID_A0))
1139+ wa_write_or(wal, UNSLICE_UNIT_LEVEL_CLKGATE2,
1140+ VSUNIT_CLKGATE_DIS_TGL);
12631141 }
12641142
12651143 static void
@@ -1358,9 +1236,9 @@ static bool · 92 unmodified lines
13581236 wa_verify(const struct i915_wa *wa, u32 cur, const char *name, const char *from)
13591237 {
13601238 if ((cur ^ wa->set) & wa->read) {
1361− DRM_ERROR("%s workaround lost on %s! (%x=%x/%x, expected %x)\n",
1239+ DRM_ERROR("%s workaround lost on %s! (reg[%x]=0x%x, relevant bits were 0x%x vs expected 0x%x)\n",
13621240 name, from, i915_mmio_reg_offset(wa->reg),
1363− cur, cur & wa->read, wa->set);
1241+ cur, cur & wa->read, wa->set & wa->read);
13641242
13651243 return false;
13661244 }
@@ -1425,6 +1303,7 @@ bool intel_gt_verify_workarounds(struct intel_gt *gt, const char *from) · 58 unmodified lines
14251303 return wa_list_verify(gt->uncore, &gt->i915->gt_wa_list, from);
14261304 }
14271305
1306+__maybe_unused
14281307 static inline bool is_nonpriv_flags_valid(u32 flags)
14291308 {
14301309 /* Check only valid flag bits are set */
@@ -1752,10 +1631,6 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 321 unmodified lines
17521631 wa_write_or(wal,
17531632 GEN7_SARCHKMD,
17541633 GEN7_DISABLE_SAMPLER_PREFETCH);
1755−
1756− /* Wa_1408615072:tgl */
1757− wa_write_or(wal, UNSLICE_UNIT_LEVEL_CLKGATE2,
1758− VSUNIT_CLKGATE_DIS_TGL);
17591634 }
17601635
17611636 if (IS_DG1(i915) || IS_ROCKETLAKE(i915) || IS_TIGERLAKE(i915)) {
@@ -1770,6 +1645,19 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 8 unmodified lines
17701645 */
17711646 wa_write_or(wal, GEN7_FF_THREAD_MODE,
17721647 GEN12_FF_TESSELATION_DOP_GATE_DISABLE);
1648+
1649+ /*
1650+ * Wa_1606700617:tgl,dg1
1651+ * Wa_22010271021:tgl,rkl,dg1
1652+ */
1653+ wa_masked_en(wal,
1654+ GEN9_CS_DEBUG_MODE1,
1655+ FF_DOP_CLOCK_GATE_DISABLE);
1656+
1657+ /* Wa_1406941453:tgl,rkl,dg1 */
1658+ wa_masked_en(wal,
1659+ GEN10_SAMPLER_MODE,
1660+ ENABLE_SMALLPL);
17731661 }
17741662
17751663 if (IS_DG1_REVID(i915, DG1_REVID_A0, DG1_REVID_A0) ||
@@ -1798,21 +1686,6 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 22 unmodified lines
17981686 GEN6_RC_SLEEP_PSMI_CONTROL,
17991687 GEN12_WAIT_FOR_EVENT_POWER_DOWN_DISABLE |
18001688 GEN8_RC_SEMA_IDLE_MSG_DISABLE);
1801−
1802− /*
1803− * Wa_1606700617:tgl
1804− * Wa_22010271021:tgl,rkl
1805− */
1806− wa_masked_en(wal,
1807− GEN9_CS_DEBUG_MODE1,
1808− FF_DOP_CLOCK_GATE_DISABLE);
1809− }
1810−
1811− if (IS_GEN(i915, 12)) {
1812− /* Wa_1406941453:gen12 */
1813− wa_masked_en(wal,
1814− GEN10_SAMPLER_MODE,
1815− ENABLE_SMALLPL);
18161689 }
18171690
18181691 if (IS_GEN(i915, 11)) {
@@ -1838,14 +1711,14 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 19 unmodified lines
18381711 * Wa_1604223664:icl
18391712 * Formerly known as WaL3BankAddressHashing
18401713 */
1841− wa_write_masked_or(wal,
1842− GEN8_GARBCNTL,
1843− GEN11_HASH_CTRL_EXCL_MASK,
1844− GEN11_HASH_CTRL_EXCL_BIT0);
1845− wa_write_masked_or(wal,
1846− GEN11_GLBLINVL,
1847− GEN11_BANK_HASH_ADDR_EXCL_MASK,
1848− GEN11_BANK_HASH_ADDR_EXCL_BIT0);
1714+ wa_write_clr_set(wal,
1715+ GEN8_GARBCNTL,
1716+ GEN11_HASH_CTRL_EXCL_MASK,
1717+ GEN11_HASH_CTRL_EXCL_BIT0);
1718+ wa_write_clr_set(wal,
1719+ GEN11_GLBLINVL,
1720+ GEN11_BANK_HASH_ADDR_EXCL_MASK,
1721+ GEN11_BANK_HASH_ADDR_EXCL_BIT0);
18491722
18501723 /*
18511724 * Wa_1405733216:icl
@@ -1874,10 +1747,10 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 22 unmodified lines
18741747 GEN7_DISABLE_SAMPLER_PREFETCH);
18751748
18761749 /* Wa_1409178092:icl */
1877− wa_write_masked_or(wal,
1878− GEN11_SCRATCH2,
1879− GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE,
1880− 0);
1750+ wa_write_clr_set(wal,
1751+ GEN11_SCRATCH2,
1752+ GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE,
1753+ 0);
18811754
18821755 /* WaEnable32PlaneMode:icl */
18831756 wa_masked_en(wal, GEN9_CSFE_CHICKEN1_RCS,
@@ -1951,11 +1824,11 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 67 unmodified lines
19511824
19521825 /* WaProgramL3SqcReg1DefaultForPerf:bxt,glk */
19531826 if (IS_GEN9_LP(i915))
1954− wa_write_masked_or(wal,
1955− GEN8_L3SQCREG1,
1956− L3_PRIO_CREDITS_MASK,
1957− L3_GENERAL_PRIO_CREDITS(62) |
1958− L3_HIGH_PRIO_CREDITS(2));
1827+ wa_write_clr_set(wal,
1828+ GEN8_L3SQCREG1,
1829+ L3_PRIO_CREDITS_MASK,
1830+ L3_GENERAL_PRIO_CREDITS(62) |
1831+ L3_HIGH_PRIO_CREDITS(2));
19591832
19601833 /* WaOCLCoherentLineFlush:skl,bxt,kbl,cfl */
19611834 wa_write_or(wal,
@@ -1963,12 +1836,112 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 1 unmodified line
19631836 GEN8_LQSC_FLUSH_COHERENT_LINES);
19641837 }
19651838
1966− if (IS_GEN(i915, 7))
1839+ if (IS_HASWELL(i915)) {
1840+ /* WaSampleCChickenBitEnable:hsw */
1841+ wa_masked_en(wal,
1842+ HALF_SLICE_CHICKEN3, HSW_SAMPLE_C_PERFORMANCE);
1843+
1844+ wa_masked_dis(wal,
1845+ CACHE_MODE_0_GEN7,
1846+ /* enable HiZ Raw Stall Optimization */
1847+ HIZ_RAW_STALL_OPT_DISABLE);
1848+
1849+ /* WaDisable4x2SubspanOptimization:hsw */
1850+ wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
1851+ }
1852+
1853+ if (IS_VALLEYVIEW(i915)) {
1854+ /* WaDisableEarlyCull:vlv */
1855+ wa_masked_en(wal,
1856+ _3D_CHICKEN3,
1857+ _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
1858+
1859+ /*
1860+ * WaVSThreadDispatchOverride:ivb,vlv
1861+ *
1862+ * This actually overrides the dispatch
1863+ * mode for all thread types.
1864+ */
1865+ wa_write_clr_set(wal,
1866+ GEN7_FF_THREAD_MODE,
1867+ GEN7_FF_SCHED_MASK,
1868+ GEN7_FF_TS_SCHED_HW |
1869+ GEN7_FF_VS_SCHED_HW |
1870+ GEN7_FF_DS_SCHED_HW);
1871+
1872+ /* WaPsdDispatchEnable:vlv */
1873+ /* WaDisablePSDDualDispatchEnable:vlv */
1874+ wa_masked_en(wal,
1875+ GEN7_HALF_SLICE_CHICKEN1,
1876+ GEN7_MAX_PS_THREAD_DEP |
1877+ GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
1878+ }
1879+
1880+ if (IS_IVYBRIDGE(i915)) {
1881+ /* WaDisableEarlyCull:ivb */
1882+ wa_masked_en(wal,
1883+ _3D_CHICKEN3,
1884+ _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
1885+
1886+ if (0) { /* causes HiZ corruption on ivb:gt1 */
1887+ /* enable HiZ Raw Stall Optimization */
1888+ wa_masked_dis(wal,
1889+ CACHE_MODE_0_GEN7,
1890+ HIZ_RAW_STALL_OPT_DISABLE);
1891+ }
1892+
1893+ /*
1894+ * WaVSThreadDispatchOverride:ivb,vlv
1895+ *
1896+ * This actually overrides the dispatch
1897+ * mode for all thread types.
1898+ */
1899+ wa_write_clr_set(wal,
1900+ GEN7_FF_THREAD_MODE,
1901+ GEN7_FF_SCHED_MASK,
1902+ GEN7_FF_TS_SCHED_HW |
1903+ GEN7_FF_VS_SCHED_HW |
1904+ GEN7_FF_DS_SCHED_HW);
1905+
1906+ /* WaDisablePSDDualDispatchEnable:ivb */
1907+ if (IS_IVB_GT1(i915))
1908+ wa_masked_en(wal,
1909+ GEN7_HALF_SLICE_CHICKEN1,
1910+ GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
1911+ }
1912+
1913+ if (IS_GEN(i915, 7)) {
19671914 /* WaBCSVCSTlbInvalidationMode:ivb,vlv,hsw */
19681915 wa_masked_en(wal,
19691916 GFX_MODE_GEN7,
19701917 GFX_TLB_INVALIDATE_EXPLICIT | GFX_REPLAY_MODE);
19711918
1919+ /* WaDisable_RenderCache_OperationalFlush:ivb,vlv,hsw */
1920+ wa_masked_dis(wal, CACHE_MODE_0_GEN7, RC_OP_FLUSH_ENABLE);
1921+
1922+ /*
1923+ * BSpec says this must be set, even though
1924+ * WaDisable4x2SubspanOptimization:ivb,hsw
1925+ * WaDisable4x2SubspanOptimization isn't listed for VLV.
1926+ */
1927+ wa_masked_en(wal,
1928+ CACHE_MODE_1,
1929+ PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
1930+
1931+ /*
1932+ * BSpec recommends 8x4 when MSAA is used,
1933+ * however in practice 16x4 seems fastest.
1934+ *
1935+ * Note that PS/WM thread counts depend on the WIZ hashing
1936+ * disable bit, which we don't touch here, but it's good
1937+ * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
1938+ */
1939+ wa_add(wal, GEN7_GT_MODE, 0,
1940+ _MASKED_FIELD(GEN6_WIZ_HASHING_MASK,
1941+ GEN6_WIZ_HASHING_16x4),
1942+ GEN6_WIZ_HASHING_16x4);
1943+ }
1944+
19721945 if (IS_GEN_RANGE(i915, 6, 7))
19731946 /*
19741947 * We need to disable the AsyncFlip performance optimisations in
@@ -1991,6 +1964,39 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 16 unmodified lines
19911964 GFX_MODE,
19921965 GFX_TLB_INVALIDATE_EXPLICIT);
19931966
1967+ /* WaDisableHiZPlanesWhenMSAAEnabled:snb */
1968+ wa_masked_en(wal,
1969+ _3D_CHICKEN,
1970+ _3D_CHICKEN_HIZ_PLANE_DISABLE_MSAA_4X_SNB);
1971+
1972+ wa_masked_en(wal,
1973+ _3D_CHICKEN3,
1974+ /* WaStripsFansDisableFastClipPerformanceFix:snb */
1975+ _3D_CHICKEN3_SF_DISABLE_FASTCLIP_CULL |
1976+ /*
1977+ * Bspec says:
1978+ * "This bit must be set if 3DSTATE_CLIP clip mode is set
1979+ * to normal and 3DSTATE_SF number of SF output attributes
1980+ * is more than 16."
1981+ */
1982+ _3D_CHICKEN3_SF_DISABLE_PIPELINED_ATTR_FETCH);
1983+
1984+ /*
1985+ * BSpec recommends 8x4 when MSAA is used,
1986+ * however in practice 16x4 seems fastest.
1987+ *
1988+ * Note that PS/WM thread counts depend on the WIZ hashing
1989+ * disable bit, which we don't touch here, but it's good
1990+ * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
1991+ */
1992+ wa_add(wal,
1993+ GEN6_GT_MODE, 0,
1994+ _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
1995+ GEN6_WIZ_HASHING_16x4);
1996+
1997+ /* WaDisable_RenderCache_OperationalFlush:snb */
1998+ wa_masked_dis(wal, CACHE_MODE_0, RC_OP_FLUSH_ENABLE);
1999+
19942000 /*
19952001 * From the Sandybridge PRM, volume 1 part 3, page 24:
19962002 * "If this bit is set, STCunit will have LRA as replacement
@@ -2067,39 +2073,6 @@ void intel_engine_apply_workarounds(struct intel_engine_cs *engine) · 70 unmodified lines
20672073 wa_list_apply(engine->uncore, &engine->wa_list);
20682074 }
20692075
2070−static struct i915_vma *
2071−create_scratch(struct i915_address_space *vm, int count)
2072−{
2073− struct drm_i915_gem_object *obj;
2074− struct i915_vma *vma;
2075− unsigned int size;
2076− int err;
2077−
2078− size = round_up(count * sizeof(u32), PAGE_SIZE);
2079− obj = i915_gem_object_create_internal(vm->i915, size);
2080− if (IS_ERR(obj))
2081− return ERR_CAST(obj);
2082−
2083− i915_gem_object_set_cache_coherency(obj, I915_CACHE_LLC);
2084−
2085− vma = i915_vma_instance(obj, vm, NULL);
2086− if (IS_ERR(vma)) {
2087− err = PTR_ERR(vma);
2088− goto err_obj;
2089− }
2090−
2091− err = i915_vma_pin(vma, 0, 0,
2092− i915_vma_is_ggtt(vma) ? PIN_GLOBAL : PIN_USER);
2093− if (err)
2094− goto err_obj;
2095−
2096− return vma;
2097−
2098−err_obj:
2099− i915_gem_object_put(obj);
2100− return ERR_PTR(err);
2101−}
2102−
21032076 struct mcr_range {
21042077 u32 start;
21052078 u32 end;
@@ -2202,7 +2175,8 @@ static int engine_wa_list_verify(struct intel_context *ce, · 96 unmodified lines
22022175 if (!wal->count)
22032176 return 0;
22042177
2205− vma = create_scratch(&ce->engine->gt->ggtt->vm, wal->count);
2178+ vma = __vm_create_scratch_for_read(&ce->engine->gt->ggtt->vm,
2179+ wal->count * sizeof(u32));
22062180 if (IS_ERR(vma))
22072181 return PTR_ERR(vma);
22082182
drivers/gpu/drm/i915/gt/mock_engine.c+2−13
@@ -245,17 +245,6 @@ static void mock_reset_rewind(struct intel_engine_cs *engine, bool stalled) · 244 unmodified lines
245245 GEM_BUG_ON(stalled);
246246 }
247247
248−static void mark_eio(struct i915_request *rq)
249−{
250− if (i915_request_completed(rq))
251− return;
252−
253− GEM_BUG_ON(i915_request_signaled(rq));
254−
255− i915_request_set_error_once(rq, -EIO);
256− i915_request_mark_complete(rq);
257−}
258−
259248 static void mock_reset_cancel(struct intel_engine_cs *engine)
260249 {
261250 struct mock_engine *mock =
@@ -269,12 +258,12 @@ static void mock_reset_cancel(struct intel_engine_cs *engine) · 7 unmodified lines
269258
270259 /* Mark all submitted requests as skipped. */
271260 list_for_each_entry(rq, &engine->active.requests, sched.link)
272− mark_eio(rq);
261+ i915_request_mark_eio(rq);
273262 intel_engine_signal_breadcrumbs(engine);
274263
275264 /* Cancel and submit all pending requests. */
276265 list_for_each_entry(rq, &mock->hw_queue, mock.link) {
277− mark_eio(rq);
266+ i915_request_mark_eio(rq);
278267 __i915_request_submit(rq);
279268 }
280269 INIT_LIST_HEAD(&mock->hw_queue);
drivers/gpu/drm/i915/gt/selftest_context.c+1−1
@@ -25,7 +25,7 @@ static int request_sync(struct i915_request *rq) · 24 unmodified lines
2525 /* Opencode i915_request_add() so we can keep the timeline locked. */
2626 __i915_request_commit(rq);
2727 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
28− __i915_request_queue(rq, NULL);
28+ __i915_request_queue_bh(rq);
2929
3030 timeout = i915_request_wait(rq, 0, HZ / 10);
3131 if (timeout < 0)
drivers/gpu/drm/i915/gt/selftest_engine_cs.c+1−0
@@ -6,6 +6,7 @@ · 5 unmodified lines
66
77 #include <linux/sort.h>
88
9+#include "intel_gpu_commands.h"
910 #include "intel_gt_pm.h"
1011 #include "intel_rps.h"
1112
drivers/gpu/drm/i915/gt/selftest_engine_heartbeat.c+10−3
@@ -197,6 +197,7 @@ static int cmp_u32(const void *_a, const void *_b) · 196 unmodified lines
197197
198198 static int __live_heartbeat_fast(struct intel_engine_cs *engine)
199199 {
200+ const unsigned int error_threshold = max(20000u, jiffies_to_usecs(6));
200201 struct intel_context *ce;
201202 struct i915_request *rq;
202203 ktime_t t0, t1;
@@ -254,12 +255,18 @@ static int __live_heartbeat_fast(struct intel_engine_cs *engine) · 51 unmodified lines
254255 times[0],
255256 times[ARRAY_SIZE(times) - 1]);
256257
257− /* Min work delay is 2 * 2 (worst), +1 for scheduling, +1 for slack */
258− if (times[ARRAY_SIZE(times) / 2] > jiffies_to_usecs(6)) {
258+ /*
259+ * Ideally, the upper bound on min work delay would be something like
260+ * 2 * 2 (worst), +1 for scheduling, +1 for slack. In practice, we
261+ * are, even with system_wq_highpri, at the mercy of the CPU scheduler
262+ * and may be stuck behind some slow work for many millisecond. Such
263+ * as our very own display workers.
264+ */
265+ if (times[ARRAY_SIZE(times) / 2] > error_threshold) {
259266 pr_err("%s: Heartbeat delay was %uus, expected less than %dus\n",
260267 engine->name,
261268 times[ARRAY_SIZE(times) / 2],
262− jiffies_to_usecs(6));
269+ error_threshold);
263270 err = -EINVAL;
264271 }
265272
drivers/gpu/drm/i915/gt/selftest_engine_pm.c+203−0
@@ -4,13 +4,215 @@ · 3 unmodified lines
44 * Copyright © 2018 Intel Corporation
55 */
66
7+#include <linux/sort.h>
8+
79 #include "i915_selftest.h"
10+#include "intel_gpu_commands.h"
11+#include "intel_gt_clock_utils.h"
812 #include "selftest_engine.h"
913 #include "selftest_engine_heartbeat.h"
1014 #include "selftests/igt_atomic.h"
1115 #include "selftests/igt_flush_test.h"
1216 #include "selftests/igt_spinner.h"
1317
18+#define COUNT 5
19+
20+static int cmp_u64(const void *A, const void *B)
21+{
22+ const u64 *a = A, *b = B;
23+
24+ return *a - *b;
25+}
26+
27+static u64 trifilter(u64 *a)
28+{
29+ sort(a, COUNT, sizeof(*a), cmp_u64, NULL);
30+ return (a[1] + 2 * a[2] + a[3]) >> 2;
31+}
32+
33+static u32 *emit_wait(u32 *cs, u32 offset, int op, u32 value)
34+{
35+ *cs++ = MI_SEMAPHORE_WAIT |
36+ MI_SEMAPHORE_GLOBAL_GTT |
37+ MI_SEMAPHORE_POLL |
38+ op;
39+ *cs++ = value;
40+ *cs++ = offset;
41+ *cs++ = 0;
42+
43+ return cs;
44+}
45+
46+static u32 *emit_store(u32 *cs, u32 offset, u32 value)
47+{
48+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
49+ *cs++ = offset;
50+ *cs++ = 0;
51+ *cs++ = value;
52+
53+ return cs;
54+}
55+
56+static u32 *emit_srm(u32 *cs, i915_reg_t reg, u32 offset)
57+{
58+ *cs++ = MI_STORE_REGISTER_MEM_GEN8 | MI_USE_GGTT;
59+ *cs++ = i915_mmio_reg_offset(reg);
60+ *cs++ = offset;
61+ *cs++ = 0;
62+
63+ return cs;
64+}
65+
66+static void write_semaphore(u32 *x, u32 value)
67+{
68+ WRITE_ONCE(*x, value);
69+ wmb();
70+}
71+
72+static int __measure_timestamps(struct intel_context *ce,
73+ u64 *dt, u64 *d_ring, u64 *d_ctx)
74+{
75+ struct intel_engine_cs *engine = ce->engine;
76+ u32 *sema = memset32(engine->status_page.addr + 1000, 0, 5);
77+ u32 offset = i915_ggtt_offset(engine->status_page.vma);
78+ struct i915_request *rq;
79+ u32 *cs;
80+
81+ rq = intel_context_create_request(ce);
82+ if (IS_ERR(rq))
83+ return PTR_ERR(rq);
84+
85+ cs = intel_ring_begin(rq, 28);
86+ if (IS_ERR(cs)) {
87+ i915_request_add(rq);
88+ return PTR_ERR(cs);
89+ }
90+
91+ /* Signal & wait for start */
92+ cs = emit_store(cs, offset + 4008, 1);
93+ cs = emit_wait(cs, offset + 4008, MI_SEMAPHORE_SAD_NEQ_SDD, 1);
94+
95+ cs = emit_srm(cs, RING_TIMESTAMP(engine->mmio_base), offset + 4000);
96+ cs = emit_srm(cs, RING_CTX_TIMESTAMP(engine->mmio_base), offset + 4004);
97+
98+ /* Busy wait */
99+ cs = emit_wait(cs, offset + 4008, MI_SEMAPHORE_SAD_EQ_SDD, 1);
100+
101+ cs = emit_srm(cs, RING_TIMESTAMP(engine->mmio_base), offset + 4016);
102+ cs = emit_srm(cs, RING_CTX_TIMESTAMP(engine->mmio_base), offset + 4012);
103+
104+ intel_ring_advance(rq, cs);
105+ i915_request_get(rq);
106+ i915_request_add(rq);
107+ intel_engine_flush_submission(engine);
108+
109+ /* Wait for the request to start executing, that then waits for us */
110+ while (READ_ONCE(sema[2]) == 0)
111+ cpu_relax();
112+
113+ /* Run the request for a 100us, sampling timestamps before/after */
114+ preempt_disable();
115+ *dt = local_clock();
116+ write_semaphore(&sema[2], 0);
117+ udelay(100);
118+ *dt = local_clock() - *dt;
119+ write_semaphore(&sema[2], 1);
120+ preempt_enable();
121+
122+ if (i915_request_wait(rq, 0, HZ / 2) < 0) {
123+ i915_request_put(rq);
124+ return -ETIME;
125+ }
126+ i915_request_put(rq);
127+
128+ pr_debug("%s CTX_TIMESTAMP: [%x, %x], RING_TIMESTAMP: [%x, %x]\n",
129+ engine->name, sema[1], sema[3], sema[0], sema[4]);
130+
131+ *d_ctx = sema[3] - sema[1];
132+ *d_ring = sema[4] - sema[0];
133+ return 0;
134+}
135+
136+static int __live_engine_timestamps(struct intel_engine_cs *engine)
137+{
138+ u64 s_ring[COUNT], s_ctx[COUNT], st[COUNT], d_ring, d_ctx, dt;
139+ struct intel_context *ce;
140+ int i, err = 0;
141+
142+ ce = intel_context_create(engine);
143+ if (IS_ERR(ce))
144+ return PTR_ERR(ce);
145+
146+ for (i = 0; i < COUNT; i++) {
147+ err = __measure_timestamps(ce, &st[i], &s_ring[i], &s_ctx[i]);
148+ if (err)
149+ break;
150+ }
151+ intel_context_put(ce);
152+ if (err)
153+ return err;
154+
155+ dt = trifilter(st);
156+ d_ring = trifilter(s_ring);
157+ d_ctx = trifilter(s_ctx);
158+
159+ pr_info("%s elapsed:%lldns, CTX_TIMESTAMP:%lldns, RING_TIMESTAMP:%lldns\n",
160+ engine->name, dt,
161+ intel_gt_clock_interval_to_ns(engine->gt, d_ctx),
162+ intel_gt_clock_interval_to_ns(engine->gt, d_ring));
163+
164+ d_ring = intel_gt_clock_interval_to_ns(engine->gt, d_ring);
165+ if (3 * dt > 4 * d_ring || 4 * dt < 3 * d_ring) {
166+ pr_err("%s Mismatch between ring timestamp and walltime!\n",
167+ engine->name);
168+ return -EINVAL;
169+ }
170+
171+ d_ring = trifilter(s_ring);
172+ d_ctx = trifilter(s_ctx);
173+
174+ d_ctx *= engine->gt->clock_frequency;
175+ if (IS_ICELAKE(engine->i915))
176+ d_ring *= 12500000; /* Fixed 80ns for icl ctx timestamp? */
177+ else
178+ d_ring *= engine->gt->clock_frequency;
179+
180+ if (3 * d_ctx > 4 * d_ring || 4 * d_ctx < 3 * d_ring) {
181+ pr_err("%s Mismatch between ring and context timestamps!\n",
182+ engine->name);
183+ return -EINVAL;
184+ }
185+
186+ return 0;
187+}
188+
189+static int live_engine_timestamps(void *arg)
190+{
191+ struct intel_gt *gt = arg;
192+ struct intel_engine_cs *engine;
193+ enum intel_engine_id id;
194+
195+ /*
196+ * Check that CS_TIMESTAMP / CTX_TIMESTAMP are in sync, i.e. share
197+ * the same CS clock.
198+ */
199+
200+ if (INTEL_GEN(gt->i915) < 8)
201+ return 0;
202+
203+ for_each_engine(engine, gt, id) {
204+ int err;
205+
206+ st_engine_heartbeat_disable(engine);
207+ err = __live_engine_timestamps(engine);
208+ st_engine_heartbeat_enable(engine);
209+ if (err)
210+ return err;
211+ }
212+
213+ return 0;
214+}
215+
14216 static int live_engine_busy_stats(void *arg)
15217 {
16218 struct intel_gt *gt = arg;
@@ -177,6 +379,7 @@ static int live_engine_pm(void *arg) · 160 unmodified lines
177379 int live_engine_pm_selftests(struct intel_gt *gt)
178380 {
179381 static const struct i915_subtest tests[] = {
382+ SUBTEST(live_engine_timestamps),
180383 SUBTEST(live_engine_busy_stats),
181384 SUBTEST(live_engine_pm),
182385 };
drivers/gpu/drm/i915/gt/selftest_execlists.c+4741−0
@@ -0,0 +1,4741 @@
1+/*
2+ * SPDX-License-Identifier: MIT
3+ *
4+ * Copyright © 2018 Intel Corporation
5+ */
6+
7+#include <linux/prime_numbers.h>
8+
9+#include "gem/i915_gem_pm.h"
10+#include "gt/intel_engine_heartbeat.h"
11+#include "gt/intel_reset.h"
12+#include "gt/selftest_engine_heartbeat.h"
13+
14+#include "i915_selftest.h"
15+#include "selftests/i915_random.h"
16+#include "selftests/igt_flush_test.h"
17+#include "selftests/igt_live_test.h"
18+#include "selftests/igt_spinner.h"
19+#include "selftests/lib_sw_fence.h"
20+
21+#include "gem/selftests/igt_gem_utils.h"
22+#include "gem/selftests/mock_context.h"
23+
24+#define CS_GPR(engine, n) ((engine)->mmio_base + 0x600 + (n) * 4)
25+#define NUM_GPR 16
26+#define NUM_GPR_DW (NUM_GPR * 2) /* each GPR is 2 dwords */
27+
28+static bool is_active(struct i915_request *rq)
29+{
30+ if (i915_request_is_active(rq))
31+ return true;
32+
33+ if (i915_request_on_hold(rq))
34+ return true;
35+
36+ if (i915_request_has_initial_breadcrumb(rq) && i915_request_started(rq))
37+ return true;
38+
39+ return false;
40+}
41+
42+static int wait_for_submit(struct intel_engine_cs *engine,
43+ struct i915_request *rq,
44+ unsigned long timeout)
45+{
46+ /* Ignore our own attempts to suppress excess tasklets */
47+ tasklet_hi_schedule(&engine->execlists.tasklet);
48+
49+ timeout += jiffies;
50+ do {
51+ bool done = time_after(jiffies, timeout);
52+
53+ if (i915_request_completed(rq)) /* that was quick! */
54+ return 0;
55+
56+ /* Wait until the HW has acknowleged the submission (or err) */
57+ intel_engine_flush_submission(engine);
58+ if (!READ_ONCE(engine->execlists.pending[0]) && is_active(rq))
59+ return 0;
60+
61+ if (done)
62+ return -ETIME;
63+
64+ cond_resched();
65+ } while (1);
66+}
67+
68+static int wait_for_reset(struct intel_engine_cs *engine,
69+ struct i915_request *rq,
70+ unsigned long timeout)
71+{
72+ timeout += jiffies;
73+
74+ do {
75+ cond_resched();
76+ intel_engine_flush_submission(engine);
77+
78+ if (READ_ONCE(engine->execlists.pending[0]))
79+ continue;
80+
81+ if (i915_request_completed(rq))
82+ break;
83+
84+ if (READ_ONCE(rq->fence.error))
85+ break;
86+ } while (time_before(jiffies, timeout));
87+
88+ flush_scheduled_work();
89+
90+ if (rq->fence.error != -EIO) {
91+ pr_err("%s: hanging request %llx:%lld not reset\n",
92+ engine->name,
93+ rq->fence.context,
94+ rq->fence.seqno);
95+ return -EINVAL;
96+ }
97+
98+ /* Give the request a jiffie to complete after flushing the worker */
99+ if (i915_request_wait(rq, 0,
100+ max(0l, (long)(timeout - jiffies)) + 1) < 0) {
101+ pr_err("%s: hanging request %llx:%lld did not complete\n",
102+ engine->name,
103+ rq->fence.context,
104+ rq->fence.seqno);
105+ return -ETIME;
106+ }
107+
108+ return 0;
109+}
110+
111+static int live_sanitycheck(void *arg)
112+{
113+ struct intel_gt *gt = arg;
114+ struct intel_engine_cs *engine;
115+ enum intel_engine_id id;
116+ struct igt_spinner spin;
117+ int err = 0;
118+
119+ if (!HAS_LOGICAL_RING_CONTEXTS(gt->i915))
120+ return 0;
121+
122+ if (igt_spinner_init(&spin, gt))
123+ return -ENOMEM;
124+
125+ for_each_engine(engine, gt, id) {
126+ struct intel_context *ce;
127+ struct i915_request *rq;
128+
129+ ce = intel_context_create(engine);
130+ if (IS_ERR(ce)) {
131+ err = PTR_ERR(ce);
132+ break;
133+ }
134+
135+ rq = igt_spinner_create_request(&spin, ce, MI_NOOP);
136+ if (IS_ERR(rq)) {
137+ err = PTR_ERR(rq);
138+ goto out_ctx;
139+ }
140+
141+ i915_request_add(rq);
142+ if (!igt_wait_for_spinner(&spin, rq)) {
143+ GEM_TRACE("spinner failed to start\n");
144+ GEM_TRACE_DUMP();
145+ intel_gt_set_wedged(gt);
146+ err = -EIO;
147+ goto out_ctx;
148+ }
149+
150+ igt_spinner_end(&spin);
151+ if (igt_flush_test(gt->i915)) {
152+ err = -EIO;
153+ goto out_ctx;
154+ }
155+
156+out_ctx:
157+ intel_context_put(ce);
158+ if (err)
159+ break;
160+ }
161+
162+ igt_spinner_fini(&spin);
163+ return err;
164+}
165+
166+static int live_unlite_restore(struct intel_gt *gt, int prio)
167+{
168+ struct intel_engine_cs *engine;
169+ enum intel_engine_id id;
170+ struct igt_spinner spin;
171+ int err = -ENOMEM;
172+
173+ /*
174+ * Check that we can correctly context switch between 2 instances
175+ * on the same engine from the same parent context.
176+ */
177+
178+ if (igt_spinner_init(&spin, gt))
179+ return err;
180+
181+ err = 0;
182+ for_each_engine(engine, gt, id) {
183+ struct intel_context *ce[2] = {};
184+ struct i915_request *rq[2];
185+ struct igt_live_test t;
186+ int n;
187+
188+ if (prio && !intel_engine_has_preemption(engine))
189+ continue;
190+
191+ if (!intel_engine_can_store_dword(engine))
192+ continue;
193+
194+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
195+ err = -EIO;
196+ break;
197+ }
198+ st_engine_heartbeat_disable(engine);
199+
200+ for (n = 0; n < ARRAY_SIZE(ce); n++) {
201+ struct intel_context *tmp;
202+
203+ tmp = intel_context_create(engine);
204+ if (IS_ERR(tmp)) {
205+ err = PTR_ERR(tmp);
206+ goto err_ce;
207+ }
208+
209+ err = intel_context_pin(tmp);
210+ if (err) {
211+ intel_context_put(tmp);
212+ goto err_ce;
213+ }
214+
215+ /*
216+ * Setup the pair of contexts such that if we
217+ * lite-restore using the RING_TAIL from ce[1] it
218+ * will execute garbage from ce[0]->ring.
219+ */
220+ memset(tmp->ring->vaddr,
221+ POISON_INUSE, /* IPEHR: 0x5a5a5a5a [hung!] */
222+ tmp->ring->vma->size);
223+
224+ ce[n] = tmp;
225+ }
226+ GEM_BUG_ON(!ce[1]->ring->size);
227+ intel_ring_reset(ce[1]->ring, ce[1]->ring->size / 2);
228+ lrc_update_regs(ce[1], engine, ce[1]->ring->head);
229+
230+ rq[0] = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
231+ if (IS_ERR(rq[0])) {
232+ err = PTR_ERR(rq[0]);
233+ goto err_ce;
234+ }
235+
236+ i915_request_get(rq[0]);
237+ i915_request_add(rq[0]);
238+ GEM_BUG_ON(rq[0]->postfix > ce[1]->ring->emit);
239+
240+ if (!igt_wait_for_spinner(&spin, rq[0])) {
241+ i915_request_put(rq[0]);
242+ goto err_ce;
243+ }
244+
245+ rq[1] = i915_request_create(ce[1]);
246+ if (IS_ERR(rq[1])) {
247+ err = PTR_ERR(rq[1]);
248+ i915_request_put(rq[0]);
249+ goto err_ce;
250+ }
251+
252+ if (!prio) {
253+ /*
254+ * Ensure we do the switch to ce[1] on completion.
255+ *
256+ * rq[0] is already submitted, so this should reduce
257+ * to a no-op (a wait on a request on the same engine
258+ * uses the submit fence, not the completion fence),
259+ * but it will install a dependency on rq[1] for rq[0]
260+ * that will prevent the pair being reordered by
261+ * timeslicing.
262+ */
263+ i915_request_await_dma_fence(rq[1], &rq[0]->fence);
264+ }
265+
266+ i915_request_get(rq[1]);
267+ i915_request_add(rq[1]);
268+ GEM_BUG_ON(rq[1]->postfix <= rq[0]->postfix);
269+ i915_request_put(rq[0]);
270+
271+ if (prio) {
272+ struct i915_sched_attr attr = {
273+ .priority = prio,
274+ };
275+
276+ /* Alternatively preempt the spinner with ce[1] */
277+ engine->schedule(rq[1], &attr);
278+ }
279+
280+ /* And switch back to ce[0] for good measure */
281+ rq[0] = i915_request_create(ce[0]);
282+ if (IS_ERR(rq[0])) {
283+ err = PTR_ERR(rq[0]);
284+ i915_request_put(rq[1]);
285+ goto err_ce;
286+ }
287+
288+ i915_request_await_dma_fence(rq[0], &rq[1]->fence);
289+ i915_request_get(rq[0]);
290+ i915_request_add(rq[0]);
291+ GEM_BUG_ON(rq[0]->postfix > rq[1]->postfix);
292+ i915_request_put(rq[1]);
293+ i915_request_put(rq[0]);
294+
295+err_ce:
296+ intel_engine_flush_submission(engine);
297+ igt_spinner_end(&spin);
298+ for (n = 0; n < ARRAY_SIZE(ce); n++) {
299+ if (IS_ERR_OR_NULL(ce[n]))
300+ break;
301+
302+ intel_context_unpin(ce[n]);
303+ intel_context_put(ce[n]);
304+ }
305+
306+ st_engine_heartbeat_enable(engine);
307+ if (igt_live_test_end(&t))
308+ err = -EIO;
309+ if (err)
310+ break;
311+ }
312+
313+ igt_spinner_fini(&spin);
314+ return err;
315+}
316+
317+static int live_unlite_switch(void *arg)
318+{
319+ return live_unlite_restore(arg, 0);
320+}
321+
322+static int live_unlite_preempt(void *arg)
323+{
324+ return live_unlite_restore(arg, I915_USER_PRIORITY(I915_PRIORITY_MAX));
325+}
326+
327+static int live_unlite_ring(void *arg)
328+{
329+ struct intel_gt *gt = arg;
330+ struct intel_engine_cs *engine;
331+ struct igt_spinner spin;
332+ enum intel_engine_id id;
333+ int err = 0;
334+
335+ /*
336+ * Setup a preemption event that will cause almost the entire ring
337+ * to be unwound, potentially fooling our intel_ring_direction()
338+ * into emitting a forward lite-restore instead of the rollback.
339+ */
340+
341+ if (igt_spinner_init(&spin, gt))
342+ return -ENOMEM;
343+
344+ for_each_engine(engine, gt, id) {
345+ struct intel_context *ce[2] = {};
346+ struct i915_request *rq;
347+ struct igt_live_test t;
348+ int n;
349+
350+ if (!intel_engine_has_preemption(engine))
351+ continue;
352+
353+ if (!intel_engine_can_store_dword(engine))
354+ continue;
355+
356+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
357+ err = -EIO;
358+ break;
359+ }
360+ st_engine_heartbeat_disable(engine);
361+
362+ for (n = 0; n < ARRAY_SIZE(ce); n++) {
363+ struct intel_context *tmp;
364+
365+ tmp = intel_context_create(engine);
366+ if (IS_ERR(tmp)) {
367+ err = PTR_ERR(tmp);
368+ goto err_ce;
369+ }
370+
371+ err = intel_context_pin(tmp);
372+ if (err) {
373+ intel_context_put(tmp);
374+ goto err_ce;
375+ }
376+
377+ memset32(tmp->ring->vaddr,
378+ 0xdeadbeef, /* trigger a hang if executed */
379+ tmp->ring->vma->size / sizeof(u32));
380+
381+ ce[n] = tmp;
382+ }
383+
384+ /* Create max prio spinner, followed by N low prio nops */
385+ rq = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
386+ if (IS_ERR(rq)) {
387+ err = PTR_ERR(rq);
388+ goto err_ce;
389+ }
390+
391+ i915_request_get(rq);
392+ rq->sched.attr.priority = I915_PRIORITY_BARRIER;
393+ i915_request_add(rq);
394+
395+ if (!igt_wait_for_spinner(&spin, rq)) {
396+ intel_gt_set_wedged(gt);
397+ i915_request_put(rq);
398+ err = -ETIME;
399+ goto err_ce;
400+ }
401+
402+ /* Fill the ring, until we will cause a wrap */
403+ n = 0;
404+ while (intel_ring_direction(ce[0]->ring,
405+ rq->wa_tail,
406+ ce[0]->ring->tail) <= 0) {
407+ struct i915_request *tmp;
408+
409+ tmp = intel_context_create_request(ce[0]);
410+ if (IS_ERR(tmp)) {
411+ err = PTR_ERR(tmp);
412+ i915_request_put(rq);
413+ goto err_ce;
414+ }
415+
416+ i915_request_add(tmp);
417+ intel_engine_flush_submission(engine);
418+ n++;
419+ }
420+ intel_engine_flush_submission(engine);
421+ pr_debug("%s: Filled ring with %d nop tails {size:%x, tail:%x, emit:%x, rq.tail:%x}\n",
422+ engine->name, n,
423+ ce[0]->ring->size,
424+ ce[0]->ring->tail,
425+ ce[0]->ring->emit,
426+ rq->tail);
427+ GEM_BUG_ON(intel_ring_direction(ce[0]->ring,
428+ rq->tail,
429+ ce[0]->ring->tail) <= 0);
430+ i915_request_put(rq);
431+
432+ /* Create a second ring to preempt the first ring after rq[0] */
433+ rq = intel_context_create_request(ce[1]);
434+ if (IS_ERR(rq)) {
435+ err = PTR_ERR(rq);
436+ goto err_ce;
437+ }
438+
439+ rq->sched.attr.priority = I915_PRIORITY_BARRIER;
440+ i915_request_get(rq);
441+ i915_request_add(rq);
442+
443+ err = wait_for_submit(engine, rq, HZ / 2);
444+ i915_request_put(rq);
445+ if (err) {
446+ pr_err("%s: preemption request was not submitted\n",
447+ engine->name);
448+ err = -ETIME;
449+ }
450+
451+ pr_debug("%s: ring[0]:{ tail:%x, emit:%x }, ring[1]:{ tail:%x, emit:%x }\n",
452+ engine->name,
453+ ce[0]->ring->tail, ce[0]->ring->emit,
454+ ce[1]->ring->tail, ce[1]->ring->emit);
455+
456+err_ce:
457+ intel_engine_flush_submission(engine);
458+ igt_spinner_end(&spin);
459+ for (n = 0; n < ARRAY_SIZE(ce); n++) {
460+ if (IS_ERR_OR_NULL(ce[n]))
461+ break;
462+
463+ intel_context_unpin(ce[n]);
464+ intel_context_put(ce[n]);
465+ }
466+ st_engine_heartbeat_enable(engine);
467+ if (igt_live_test_end(&t))
468+ err = -EIO;
469+ if (err)
470+ break;
471+ }
472+
473+ igt_spinner_fini(&spin);
474+ return err;
475+}
476+
477+static int live_pin_rewind(void *arg)
478+{
479+ struct intel_gt *gt = arg;
480+ struct intel_engine_cs *engine;
481+ enum intel_engine_id id;
482+ int err = 0;
483+
484+ /*
485+ * We have to be careful not to trust intel_ring too much, for example
486+ * ring->head is updated upon retire which is out of sync with pinning
487+ * the context. Thus we cannot use ring->head to set CTX_RING_HEAD,
488+ * or else we risk writing an older, stale value.
489+ *
490+ * To simulate this, let's apply a bit of deliberate sabotague.
491+ */
492+
493+ for_each_engine(engine, gt, id) {
494+ struct intel_context *ce;
495+ struct i915_request *rq;
496+ struct intel_ring *ring;
497+ struct igt_live_test t;
498+
499+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
500+ err = -EIO;
501+ break;
502+ }
503+
504+ ce = intel_context_create(engine);
505+ if (IS_ERR(ce)) {
506+ err = PTR_ERR(ce);
507+ break;
508+ }
509+
510+ err = intel_context_pin(ce);
511+ if (err) {
512+ intel_context_put(ce);
513+ break;
514+ }
515+
516+ /* Keep the context awake while we play games */
517+ err = i915_active_acquire(&ce->active);
518+ if (err) {
519+ intel_context_unpin(ce);
520+ intel_context_put(ce);
521+ break;
522+ }
523+ ring = ce->ring;
524+
525+ /* Poison the ring, and offset the next request from HEAD */
526+ memset32(ring->vaddr, STACK_MAGIC, ring->size / sizeof(u32));
527+ ring->emit = ring->size / 2;
528+ ring->tail = ring->emit;
529+ GEM_BUG_ON(ring->head);
530+
531+ intel_context_unpin(ce);
532+
533+ /* Submit a simple nop request */
534+ GEM_BUG_ON(intel_context_is_pinned(ce));
535+ rq = intel_context_create_request(ce);
536+ i915_active_release(&ce->active); /* e.g. async retire */
537+ intel_context_put(ce);
538+ if (IS_ERR(rq)) {
539+ err = PTR_ERR(rq);
540+ break;
541+ }
542+ GEM_BUG_ON(!rq->head);
543+ i915_request_add(rq);
544+
545+ /* Expect not to hang! */
546+ if (igt_live_test_end(&t)) {
547+ err = -EIO;
548+ break;
549+ }
550+ }
551+
552+ return err;
553+}
554+
555+static int live_hold_reset(void *arg)
556+{
557+ struct intel_gt *gt = arg;
558+ struct intel_engine_cs *engine;
559+ enum intel_engine_id id;
560+ struct igt_spinner spin;
561+ int err = 0;
562+
563+ /*
564+ * In order to support offline error capture for fast preempt reset,
565+ * we need to decouple the guilty request and ensure that it and its
566+ * descendents are not executed while the capture is in progress.
567+ */
568+
569+ if (!intel_has_reset_engine(gt))
570+ return 0;
571+
572+ if (igt_spinner_init(&spin, gt))
573+ return -ENOMEM;
574+
575+ for_each_engine(engine, gt, id) {
576+ struct intel_context *ce;
577+ struct i915_request *rq;
578+
579+ ce = intel_context_create(engine);
580+ if (IS_ERR(ce)) {
581+ err = PTR_ERR(ce);
582+ break;
583+ }
584+
585+ st_engine_heartbeat_disable(engine);
586+
587+ rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
588+ if (IS_ERR(rq)) {
589+ err = PTR_ERR(rq);
590+ goto out;
591+ }
592+ i915_request_add(rq);
593+
594+ if (!igt_wait_for_spinner(&spin, rq)) {
595+ intel_gt_set_wedged(gt);
596+ err = -ETIME;
597+ goto out;
598+ }
599+
600+ /* We have our request executing, now remove it and reset */
601+
602+ local_bh_disable();
603+ if (test_and_set_bit(I915_RESET_ENGINE + id,
604+ &gt->reset.flags)) {
605+ local_bh_enable();
606+ intel_gt_set_wedged(gt);
607+ err = -EBUSY;
608+ goto out;
609+ }
610+ tasklet_disable(&engine->execlists.tasklet);
611+
612+ engine->execlists.tasklet.func(engine->execlists.tasklet.data);
613+ GEM_BUG_ON(execlists_active(&engine->execlists) != rq);
614+
615+ i915_request_get(rq);
616+ execlists_hold(engine, rq);
617+ GEM_BUG_ON(!i915_request_on_hold(rq));
618+
619+ __intel_engine_reset_bh(engine, NULL);
620+ GEM_BUG_ON(rq->fence.error != -EIO);
621+
622+ tasklet_enable(&engine->execlists.tasklet);
623+ clear_and_wake_up_bit(I915_RESET_ENGINE + id,
624+ &gt->reset.flags);
625+ local_bh_enable();
626+
627+ /* Check that we do not resubmit the held request */
628+ if (!i915_request_wait(rq, 0, HZ / 5)) {
629+ pr_err("%s: on hold request completed!\n",
630+ engine->name);
631+ i915_request_put(rq);
632+ err = -EIO;
633+ goto out;
634+ }
635+ GEM_BUG_ON(!i915_request_on_hold(rq));
636+
637+ /* But is resubmitted on release */
638+ execlists_unhold(engine, rq);
639+ if (i915_request_wait(rq, 0, HZ / 5) < 0) {
640+ pr_err("%s: held request did not complete!\n",
641+ engine->name);
642+ intel_gt_set_wedged(gt);
643+ err = -ETIME;
644+ }
645+ i915_request_put(rq);
646+
647+out:
648+ st_engine_heartbeat_enable(engine);
649+ intel_context_put(ce);
650+ if (err)
651+ break;
652+ }
653+
654+ igt_spinner_fini(&spin);
655+ return err;
656+}
657+
658+static const char *error_repr(int err)
659+{
660+ return err ? "bad" : "good";
661+}
662+
663+static int live_error_interrupt(void *arg)
664+{
665+ static const struct error_phase {
666+ enum { GOOD = 0, BAD = -EIO } error[2];
667+ } phases[] = {
668+ { { BAD, GOOD } },
669+ { { BAD, BAD } },
670+ { { BAD, GOOD } },
671+ { { GOOD, GOOD } }, /* sentinel */
672+ };
673+ struct intel_gt *gt = arg;
674+ struct intel_engine_cs *engine;
675+ enum intel_engine_id id;
676+
677+ /*
678+ * We hook up the CS_MASTER_ERROR_INTERRUPT to have forewarning
679+ * of invalid commands in user batches that will cause a GPU hang.
680+ * This is a faster mechanism than using hangcheck/heartbeats, but
681+ * only detects problems the HW knows about -- it will not warn when
682+ * we kill the HW!
683+ *
684+ * To verify our detection and reset, we throw some invalid commands
685+ * at the HW and wait for the interrupt.
686+ */
687+
688+ if (!intel_has_reset_engine(gt))
689+ return 0;
690+
691+ for_each_engine(engine, gt, id) {
692+ const struct error_phase *p;
693+ int err = 0;
694+
695+ st_engine_heartbeat_disable(engine);
696+
697+ for (p = phases; p->error[0] != GOOD; p++) {
698+ struct i915_request *client[ARRAY_SIZE(phases->error)];
699+ u32 *cs;
700+ int i;
701+
702+ memset(client, 0, sizeof(*client));
703+ for (i = 0; i < ARRAY_SIZE(client); i++) {
704+ struct intel_context *ce;
705+ struct i915_request *rq;
706+
707+ ce = intel_context_create(engine);
708+ if (IS_ERR(ce)) {
709+ err = PTR_ERR(ce);
710+ goto out;
711+ }
712+
713+ rq = intel_context_create_request(ce);
714+ intel_context_put(ce);
715+ if (IS_ERR(rq)) {
716+ err = PTR_ERR(rq);
717+ goto out;
718+ }
719+
720+ if (rq->engine->emit_init_breadcrumb) {
721+ err = rq->engine->emit_init_breadcrumb(rq);
722+ if (err) {
723+ i915_request_add(rq);
724+ goto out;
725+ }
726+ }
727+
728+ cs = intel_ring_begin(rq, 2);
729+ if (IS_ERR(cs)) {
730+ i915_request_add(rq);
731+ err = PTR_ERR(cs);
732+ goto out;
733+ }
734+
735+ if (p->error[i]) {
736+ *cs++ = 0xdeadbeef;
737+ *cs++ = 0xdeadbeef;
738+ } else {
739+ *cs++ = MI_NOOP;
740+ *cs++ = MI_NOOP;
741+ }
742+
743+ client[i] = i915_request_get(rq);
744+ i915_request_add(rq);
745+ }
746+
747+ err = wait_for_submit(engine, client[0], HZ / 2);
748+ if (err) {
749+ pr_err("%s: first request did not start within time!\n",
750+ engine->name);
751+ err = -ETIME;
752+ goto out;
753+ }
754+
755+ for (i = 0; i < ARRAY_SIZE(client); i++) {
756+ if (i915_request_wait(client[i], 0, HZ / 5) < 0)
757+ pr_debug("%s: %s request incomplete!\n",
758+ engine->name,
759+ error_repr(p->error[i]));
760+
761+ if (!i915_request_started(client[i])) {
762+ pr_err("%s: %s request not started!\n",
763+ engine->name,
764+ error_repr(p->error[i]));
765+ err = -ETIME;
766+ goto out;
767+ }
768+
769+ /* Kick the tasklet to process the error */
770+ intel_engine_flush_submission(engine);
771+ if (client[i]->fence.error != p->error[i]) {
772+ pr_err("%s: %s request (%s) with wrong error code: %d\n",
773+ engine->name,
774+ error_repr(p->error[i]),
775+ i915_request_completed(client[i]) ? "completed" : "running",
776+ client[i]->fence.error);
777+ err = -EINVAL;
778+ goto out;
779+ }
780+ }
781+
782+out:
783+ for (i = 0; i < ARRAY_SIZE(client); i++)
784+ if (client[i])
785+ i915_request_put(client[i]);
786+ if (err) {
787+ pr_err("%s: failed at phase[%zd] { %d, %d }\n",
788+ engine->name, p - phases,
789+ p->error[0], p->error[1]);
790+ break;
791+ }
792+ }
793+
794+ st_engine_heartbeat_enable(engine);
795+ if (err) {
796+ intel_gt_set_wedged(gt);
797+ return err;
798+ }
799+ }
800+
801+ return 0;
802+}
803+
804+static int
805+emit_semaphore_chain(struct i915_request *rq, struct i915_vma *vma, int idx)
806+{
807+ u32 *cs;
808+
809+ cs = intel_ring_begin(rq, 10);
810+ if (IS_ERR(cs))
811+ return PTR_ERR(cs);
812+
813+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
814+
815+ *cs++ = MI_SEMAPHORE_WAIT |
816+ MI_SEMAPHORE_GLOBAL_GTT |
817+ MI_SEMAPHORE_POLL |
818+ MI_SEMAPHORE_SAD_NEQ_SDD;
819+ *cs++ = 0;
820+ *cs++ = i915_ggtt_offset(vma) + 4 * idx;
821+ *cs++ = 0;
822+
823+ if (idx > 0) {
824+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
825+ *cs++ = i915_ggtt_offset(vma) + 4 * (idx - 1);
826+ *cs++ = 0;
827+ *cs++ = 1;
828+ } else {
829+ *cs++ = MI_NOOP;
830+ *cs++ = MI_NOOP;
831+ *cs++ = MI_NOOP;
832+ *cs++ = MI_NOOP;
833+ }
834+
835+ *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
836+
837+ intel_ring_advance(rq, cs);
838+ return 0;
839+}
840+
841+static struct i915_request *
842+semaphore_queue(struct intel_engine_cs *engine, struct i915_vma *vma, int idx)
843+{
844+ struct intel_context *ce;
845+ struct i915_request *rq;
846+ int err;
847+
848+ ce = intel_context_create(engine);
849+ if (IS_ERR(ce))
850+ return ERR_CAST(ce);
851+
852+ rq = intel_context_create_request(ce);
853+ if (IS_ERR(rq))
854+ goto out_ce;
855+
856+ err = 0;
857+ if (rq->engine->emit_init_breadcrumb)
858+ err = rq->engine->emit_init_breadcrumb(rq);
859+ if (err == 0)
860+ err = emit_semaphore_chain(rq, vma, idx);
861+ if (err == 0)
862+ i915_request_get(rq);
863+ i915_request_add(rq);
864+ if (err)
865+ rq = ERR_PTR(err);
866+
867+out_ce:
868+ intel_context_put(ce);
869+ return rq;
870+}
871+
872+static int
873+release_queue(struct intel_engine_cs *engine,
874+ struct i915_vma *vma,
875+ int idx, int prio)
876+{
877+ struct i915_sched_attr attr = {
878+ .priority = prio,
879+ };
880+ struct i915_request *rq;
881+ u32 *cs;
882+
883+ rq = intel_engine_create_kernel_request(engine);
884+ if (IS_ERR(rq))
885+ return PTR_ERR(rq);
886+
887+ cs = intel_ring_begin(rq, 4);
888+ if (IS_ERR(cs)) {
889+ i915_request_add(rq);
890+ return PTR_ERR(cs);
891+ }
892+
893+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
894+ *cs++ = i915_ggtt_offset(vma) + 4 * (idx - 1);
895+ *cs++ = 0;
896+ *cs++ = 1;
897+
898+ intel_ring_advance(rq, cs);
899+
900+ i915_request_get(rq);
901+ i915_request_add(rq);
902+
903+ local_bh_disable();
904+ engine->schedule(rq, &attr);
905+ local_bh_enable(); /* kick tasklet */
906+
907+ i915_request_put(rq);
908+
909+ return 0;
910+}
911+
912+static int
913+slice_semaphore_queue(struct intel_engine_cs *outer,
914+ struct i915_vma *vma,
915+ int count)
916+{
917+ struct intel_engine_cs *engine;
918+ struct i915_request *head;
919+ enum intel_engine_id id;
920+ int err, i, n = 0;
921+
922+ head = semaphore_queue(outer, vma, n++);
923+ if (IS_ERR(head))
924+ return PTR_ERR(head);
925+
926+ for_each_engine(engine, outer->gt, id) {
927+ if (!intel_engine_has_preemption(engine))
928+ continue;
929+
930+ for (i = 0; i < count; i++) {
931+ struct i915_request *rq;
932+
933+ rq = semaphore_queue(engine, vma, n++);
934+ if (IS_ERR(rq)) {
935+ err = PTR_ERR(rq);
936+ goto out;
937+ }
938+
939+ i915_request_put(rq);
940+ }
941+ }
942+
943+ err = release_queue(outer, vma, n, I915_PRIORITY_BARRIER);
944+ if (err)
945+ goto out;
946+
947+ if (i915_request_wait(head, 0,
948+ 2 * outer->gt->info.num_engines * (count + 2) * (count + 3)) < 0) {
949+ pr_err("%s: Failed to slice along semaphore chain of length (%d, %d)!\n",
950+ outer->name, count, n);
951+ GEM_TRACE_DUMP();
952+ intel_gt_set_wedged(outer->gt);
953+ err = -EIO;
954+ }
955+
956+out:
957+ i915_request_put(head);
958+ return err;
959+}
960+
961+static int live_timeslice_preempt(void *arg)
962+{
963+ struct intel_gt *gt = arg;
964+ struct drm_i915_gem_object *obj;
965+ struct intel_engine_cs *engine;
966+ enum intel_engine_id id;
967+ struct i915_vma *vma;
968+ void *vaddr;
969+ int err = 0;
970+
971+ /*
972+ * If a request takes too long, we would like to give other users
973+ * a fair go on the GPU. In particular, users may create batches
974+ * that wait upon external input, where that input may even be
975+ * supplied by another GPU job. To avoid blocking forever, we
976+ * need to preempt the current task and replace it with another
977+ * ready task.
978+ */
979+ if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
980+ return 0;
981+
982+ obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
983+ if (IS_ERR(obj))
984+ return PTR_ERR(obj);
985+
986+ vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
987+ if (IS_ERR(vma)) {
988+ err = PTR_ERR(vma);
989+ goto err_obj;
990+ }
991+
992+ vaddr = i915_gem_object_pin_map(obj, I915_MAP_WC);
993+ if (IS_ERR(vaddr)) {
994+ err = PTR_ERR(vaddr);
995+ goto err_obj;
996+ }
997+
998+ err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
999+ if (err)
1000+ goto err_map;
1001+
1002+ err = i915_vma_sync(vma);
1003+ if (err)
1004+ goto err_pin;
1005+
1006+ for_each_engine(engine, gt, id) {
1007+ if (!intel_engine_has_preemption(engine))
1008+ continue;
1009+
1010+ memset(vaddr, 0, PAGE_SIZE);
1011+
1012+ st_engine_heartbeat_disable(engine);
1013+ err = slice_semaphore_queue(engine, vma, 5);
1014+ st_engine_heartbeat_enable(engine);
1015+ if (err)
1016+ goto err_pin;
1017+
1018+ if (igt_flush_test(gt->i915)) {
1019+ err = -EIO;
1020+ goto err_pin;
1021+ }
1022+ }
1023+
1024+err_pin:
1025+ i915_vma_unpin(vma);
1026+err_map:
1027+ i915_gem_object_unpin_map(obj);
1028+err_obj:
1029+ i915_gem_object_put(obj);
1030+ return err;
1031+}
1032+
1033+static struct i915_request *
1034+create_rewinder(struct intel_context *ce,
1035+ struct i915_request *wait,
1036+ void *slot, int idx)
1037+{
1038+ const u32 offset =
1039+ i915_ggtt_offset(ce->engine->status_page.vma) +
1040+ offset_in_page(slot);
1041+ struct i915_request *rq;
1042+ u32 *cs;
1043+ int err;
1044+
1045+ rq = intel_context_create_request(ce);
1046+ if (IS_ERR(rq))
1047+ return rq;
1048+
1049+ if (wait) {
1050+ err = i915_request_await_dma_fence(rq, &wait->fence);
1051+ if (err)
1052+ goto err;
1053+ }
1054+
1055+ cs = intel_ring_begin(rq, 14);
1056+ if (IS_ERR(cs)) {
1057+ err = PTR_ERR(cs);
1058+ goto err;
1059+ }
1060+
1061+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
1062+ *cs++ = MI_NOOP;
1063+
1064+ *cs++ = MI_SEMAPHORE_WAIT |
1065+ MI_SEMAPHORE_GLOBAL_GTT |
1066+ MI_SEMAPHORE_POLL |
1067+ MI_SEMAPHORE_SAD_GTE_SDD;
1068+ *cs++ = idx;
1069+ *cs++ = offset;
1070+ *cs++ = 0;
1071+
1072+ *cs++ = MI_STORE_REGISTER_MEM_GEN8 | MI_USE_GGTT;
1073+ *cs++ = i915_mmio_reg_offset(RING_TIMESTAMP(rq->engine->mmio_base));
1074+ *cs++ = offset + idx * sizeof(u32);
1075+ *cs++ = 0;
1076+
1077+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
1078+ *cs++ = offset;
1079+ *cs++ = 0;
1080+ *cs++ = idx + 1;
1081+
1082+ intel_ring_advance(rq, cs);
1083+
1084+ rq->sched.attr.priority = I915_PRIORITY_MASK;
1085+ err = 0;
1086+err:
1087+ i915_request_get(rq);
1088+ i915_request_add(rq);
1089+ if (err) {
1090+ i915_request_put(rq);
1091+ return ERR_PTR(err);
1092+ }
1093+
1094+ return rq;
1095+}
1096+
1097+static int live_timeslice_rewind(void *arg)
1098+{
1099+ struct intel_gt *gt = arg;
1100+ struct intel_engine_cs *engine;
1101+ enum intel_engine_id id;
1102+
1103+ /*
1104+ * The usual presumption on timeslice expiration is that we replace
1105+ * the active context with another. However, given a chain of
1106+ * dependencies we may end up with replacing the context with itself,
1107+ * but only a few of those requests, forcing us to rewind the
1108+ * RING_TAIL of the original request.
1109+ */
1110+ if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
1111+ return 0;
1112+
1113+ for_each_engine(engine, gt, id) {
1114+ enum { A1, A2, B1 };
1115+ enum { X = 1, Z, Y };
1116+ struct i915_request *rq[3] = {};
1117+ struct intel_context *ce;
1118+ unsigned long timeslice;
1119+ int i, err = 0;
1120+ u32 *slot;
1121+
1122+ if (!intel_engine_has_timeslices(engine))
1123+ continue;
1124+
1125+ /*
1126+ * A:rq1 -- semaphore wait, timestamp X
1127+ * A:rq2 -- write timestamp Y
1128+ *
1129+ * B:rq1 [await A:rq1] -- write timestamp Z
1130+ *
1131+ * Force timeslice, release semaphore.
1132+ *
1133+ * Expect execution/evaluation order XZY
1134+ */
1135+
1136+ st_engine_heartbeat_disable(engine);
1137+ timeslice = xchg(&engine->props.timeslice_duration_ms, 1);
1138+
1139+ slot = memset32(engine->status_page.addr + 1000, 0, 4);
1140+
1141+ ce = intel_context_create(engine);
1142+ if (IS_ERR(ce)) {
1143+ err = PTR_ERR(ce);
1144+ goto err;
1145+ }
1146+
1147+ rq[A1] = create_rewinder(ce, NULL, slot, X);
1148+ if (IS_ERR(rq[A1])) {
1149+ intel_context_put(ce);
1150+ goto err;
1151+ }
1152+
1153+ rq[A2] = create_rewinder(ce, NULL, slot, Y);
1154+ intel_context_put(ce);
1155+ if (IS_ERR(rq[A2]))
1156+ goto err;
1157+
1158+ err = wait_for_submit(engine, rq[A2], HZ / 2);
1159+ if (err) {
1160+ pr_err("%s: failed to submit first context\n",
1161+ engine->name);
1162+ goto err;
1163+ }
1164+
1165+ ce = intel_context_create(engine);
1166+ if (IS_ERR(ce)) {
1167+ err = PTR_ERR(ce);
1168+ goto err;
1169+ }
1170+
1171+ rq[B1] = create_rewinder(ce, rq[A1], slot, Z);
1172+ intel_context_put(ce);
1173+ if (IS_ERR(rq[2]))
1174+ goto err;
1175+
1176+ err = wait_for_submit(engine, rq[B1], HZ / 2);
1177+ if (err) {
1178+ pr_err("%s: failed to submit second context\n",
1179+ engine->name);
1180+ goto err;
1181+ }
1182+
1183+ /* ELSP[] = { { A:rq1, A:rq2 }, { B:rq1 } } */
1184+ ENGINE_TRACE(engine, "forcing tasklet for rewind\n");
1185+ while (i915_request_is_active(rq[A2])) { /* semaphore yield! */
1186+ /* Wait for the timeslice to kick in */
1187+ del_timer(&engine->execlists.timer);
1188+ tasklet_hi_schedule(&engine->execlists.tasklet);
1189+ intel_engine_flush_submission(engine);
1190+ }
1191+ /* -> ELSP[] = { { A:rq1 }, { B:rq1 } } */
1192+ GEM_BUG_ON(!i915_request_is_active(rq[A1]));
1193+ GEM_BUG_ON(!i915_request_is_active(rq[B1]));
1194+ GEM_BUG_ON(i915_request_is_active(rq[A2]));
1195+
1196+ /* Release the hounds! */
1197+ slot[0] = 1;
1198+ wmb(); /* "pairs" with GPU; paranoid kick of internal CPU$ */
1199+
1200+ for (i = 1; i <= 3; i++) {
1201+ unsigned long timeout = jiffies + HZ / 2;
1202+
1203+ while (!READ_ONCE(slot[i]) &&
1204+ time_before(jiffies, timeout))
1205+ ;
1206+
1207+ if (!time_before(jiffies, timeout)) {
1208+ pr_err("%s: rq[%d] timed out\n",
1209+ engine->name, i - 1);
1210+ err = -ETIME;
1211+ goto err;
1212+ }
1213+
1214+ pr_debug("%s: slot[%d]:%x\n", engine->name, i, slot[i]);
1215+ }
1216+
1217+ /* XZY: XZ < XY */
1218+ if (slot[Z] - slot[X] >= slot[Y] - slot[X]) {
1219+ pr_err("%s: timeslicing did not run context B [%u] before A [%u]!\n",
1220+ engine->name,
1221+ slot[Z] - slot[X],
1222+ slot[Y] - slot[X]);
1223+ err = -EINVAL;
1224+ }
1225+
1226+err:
1227+ memset32(&slot[0], -1, 4);
1228+ wmb();
1229+
1230+ engine->props.timeslice_duration_ms = timeslice;
1231+ st_engine_heartbeat_enable(engine);
1232+ for (i = 0; i < 3; i++)
1233+ i915_request_put(rq[i]);
1234+ if (igt_flush_test(gt->i915))
1235+ err = -EIO;
1236+ if (err)
1237+ return err;
1238+ }
1239+
1240+ return 0;
1241+}
1242+
1243+static struct i915_request *nop_request(struct intel_engine_cs *engine)
1244+{
1245+ struct i915_request *rq;
1246+
1247+ rq = intel_engine_create_kernel_request(engine);
1248+ if (IS_ERR(rq))
1249+ return rq;
1250+
1251+ i915_request_get(rq);
1252+ i915_request_add(rq);
1253+
1254+ return rq;
1255+}
1256+
1257+static long slice_timeout(struct intel_engine_cs *engine)
1258+{
1259+ long timeout;
1260+
1261+ /* Enough time for a timeslice to kick in, and kick out */
1262+ timeout = 2 * msecs_to_jiffies_timeout(timeslice(engine));
1263+
1264+ /* Enough time for the nop request to complete */
1265+ timeout += HZ / 5;
1266+
1267+ return timeout + 1;
1268+}
1269+
1270+static int live_timeslice_queue(void *arg)
1271+{
1272+ struct intel_gt *gt = arg;
1273+ struct drm_i915_gem_object *obj;
1274+ struct intel_engine_cs *engine;
1275+ enum intel_engine_id id;
1276+ struct i915_vma *vma;
1277+ void *vaddr;
1278+ int err = 0;
1279+
1280+ /*
1281+ * Make sure that even if ELSP[0] and ELSP[1] are filled with
1282+ * timeslicing between them disabled, we *do* enable timeslicing
1283+ * if the queue demands it. (Normally, we do not submit if
1284+ * ELSP[1] is already occupied, so must rely on timeslicing to
1285+ * eject ELSP[0] in favour of the queue.)
1286+ */
1287+ if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
1288+ return 0;
1289+
1290+ obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
1291+ if (IS_ERR(obj))
1292+ return PTR_ERR(obj);
1293+
1294+ vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
1295+ if (IS_ERR(vma)) {
1296+ err = PTR_ERR(vma);
1297+ goto err_obj;
1298+ }
1299+
1300+ vaddr = i915_gem_object_pin_map(obj, I915_MAP_WC);
1301+ if (IS_ERR(vaddr)) {
1302+ err = PTR_ERR(vaddr);
1303+ goto err_obj;
1304+ }
1305+
1306+ err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
1307+ if (err)
1308+ goto err_map;
1309+
1310+ err = i915_vma_sync(vma);
1311+ if (err)
1312+ goto err_pin;
1313+
1314+ for_each_engine(engine, gt, id) {
1315+ struct i915_sched_attr attr = {
1316+ .priority = I915_USER_PRIORITY(I915_PRIORITY_MAX),
1317+ };
1318+ struct i915_request *rq, *nop;
1319+
1320+ if (!intel_engine_has_preemption(engine))
1321+ continue;
1322+
1323+ st_engine_heartbeat_disable(engine);
1324+ memset(vaddr, 0, PAGE_SIZE);
1325+
1326+ /* ELSP[0]: semaphore wait */
1327+ rq = semaphore_queue(engine, vma, 0);
1328+ if (IS_ERR(rq)) {
1329+ err = PTR_ERR(rq);
1330+ goto err_heartbeat;
1331+ }
1332+ engine->schedule(rq, &attr);
1333+ err = wait_for_submit(engine, rq, HZ / 2);
1334+ if (err) {
1335+ pr_err("%s: Timed out trying to submit semaphores\n",
1336+ engine->name);
1337+ goto err_rq;
1338+ }
1339+
1340+ /* ELSP[1]: nop request */
1341+ nop = nop_request(engine);
1342+ if (IS_ERR(nop)) {
1343+ err = PTR_ERR(nop);
1344+ goto err_rq;
1345+ }
1346+ err = wait_for_submit(engine, nop, HZ / 2);
1347+ i915_request_put(nop);
1348+ if (err) {
1349+ pr_err("%s: Timed out trying to submit nop\n",
1350+ engine->name);
1351+ goto err_rq;
1352+ }
1353+
1354+ GEM_BUG_ON(i915_request_completed(rq));
1355+ GEM_BUG_ON(execlists_active(&engine->execlists) != rq);
1356+
1357+ /* Queue: semaphore signal, matching priority as semaphore */
1358+ err = release_queue(engine, vma, 1, effective_prio(rq));
1359+ if (err)
1360+ goto err_rq;
1361+
1362+ /* Wait until we ack the release_queue and start timeslicing */
1363+ do {
1364+ cond_resched();
1365+ intel_engine_flush_submission(engine);
1366+ } while (READ_ONCE(engine->execlists.pending[0]));
1367+
1368+ /* Timeslice every jiffy, so within 2 we should signal */
1369+ if (i915_request_wait(rq, 0, slice_timeout(engine)) < 0) {
1370+ struct drm_printer p =
1371+ drm_info_printer(gt->i915->drm.dev);
1372+
1373+ pr_err("%s: Failed to timeslice into queue\n",
1374+ engine->name);
1375+ intel_engine_dump(engine, &p,
1376+ "%s\n", engine->name);
1377+
1378+ memset(vaddr, 0xff, PAGE_SIZE);
1379+ err = -EIO;
1380+ }
1381+err_rq:
1382+ i915_request_put(rq);
1383+err_heartbeat:
1384+ st_engine_heartbeat_enable(engine);
1385+ if (err)
1386+ break;
1387+ }
1388+
1389+err_pin:
1390+ i915_vma_unpin(vma);
1391+err_map:
1392+ i915_gem_object_unpin_map(obj);
1393+err_obj:
1394+ i915_gem_object_put(obj);
1395+ return err;
1396+}
1397+
1398+static int live_timeslice_nopreempt(void *arg)
1399+{
1400+ struct intel_gt *gt = arg;
1401+ struct intel_engine_cs *engine;
1402+ enum intel_engine_id id;
1403+ struct igt_spinner spin;
1404+ int err = 0;
1405+
1406+ /*
1407+ * We should not timeslice into a request that is marked with
1408+ * I915_REQUEST_NOPREEMPT.
1409+ */
1410+ if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
1411+ return 0;
1412+
1413+ if (igt_spinner_init(&spin, gt))
1414+ return -ENOMEM;
1415+
1416+ for_each_engine(engine, gt, id) {
1417+ struct intel_context *ce;
1418+ struct i915_request *rq;
1419+ unsigned long timeslice;
1420+
1421+ if (!intel_engine_has_preemption(engine))
1422+ continue;
1423+
1424+ ce = intel_context_create(engine);
1425+ if (IS_ERR(ce)) {
1426+ err = PTR_ERR(ce);
1427+ break;
1428+ }
1429+
1430+ st_engine_heartbeat_disable(engine);
1431+ timeslice = xchg(&engine->props.timeslice_duration_ms, 1);
1432+
1433+ /* Create an unpreemptible spinner */
1434+
1435+ rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
1436+ intel_context_put(ce);
1437+ if (IS_ERR(rq)) {
1438+ err = PTR_ERR(rq);
1439+ goto out_heartbeat;
1440+ }
1441+
1442+ i915_request_get(rq);
1443+ i915_request_add(rq);
1444+
1445+ if (!igt_wait_for_spinner(&spin, rq)) {
1446+ i915_request_put(rq);
1447+ err = -ETIME;
1448+ goto out_spin;
1449+ }
1450+
1451+ set_bit(I915_FENCE_FLAG_NOPREEMPT, &rq->fence.flags);
1452+ i915_request_put(rq);
1453+
1454+ /* Followed by a maximum priority barrier (heartbeat) */
1455+
1456+ ce = intel_context_create(engine);
1457+ if (IS_ERR(ce)) {
1458+ err = PTR_ERR(ce);
1459+ goto out_spin;
1460+ }
1461+
1462+ rq = intel_context_create_request(ce);
1463+ intel_context_put(ce);
1464+ if (IS_ERR(rq)) {
1465+ err = PTR_ERR(rq);
1466+ goto out_spin;
1467+ }
1468+
1469+ rq->sched.attr.priority = I915_PRIORITY_BARRIER;
1470+ i915_request_get(rq);
1471+ i915_request_add(rq);
1472+
1473+ /*
1474+ * Wait until the barrier is in ELSP, and we know timeslicing
1475+ * will have been activated.
1476+ */
1477+ if (wait_for_submit(engine, rq, HZ / 2)) {
1478+ i915_request_put(rq);
1479+ err = -ETIME;
1480+ goto out_spin;
1481+ }
1482+
1483+ /*
1484+ * Since the ELSP[0] request is unpreemptible, it should not
1485+ * allow the maximum priority barrier through. Wait long
1486+ * enough to see if it is timesliced in by mistake.
1487+ */
1488+ if (i915_request_wait(rq, 0, slice_timeout(engine)) >= 0) {
1489+ pr_err("%s: I915_PRIORITY_BARRIER request completed, bypassing no-preempt request\n",
1490+ engine->name);
1491+ err = -EINVAL;
1492+ }
1493+ i915_request_put(rq);
1494+
1495+out_spin:
1496+ igt_spinner_end(&spin);
1497+out_heartbeat:
1498+ xchg(&engine->props.timeslice_duration_ms, timeslice);
1499+ st_engine_heartbeat_enable(engine);
1500+ if (err)
1501+ break;
1502+
1503+ if (igt_flush_test(gt->i915)) {
1504+ err = -EIO;
1505+ break;
1506+ }
1507+ }
1508+
1509+ igt_spinner_fini(&spin);
1510+ return err;
1511+}
1512+
1513+static int live_busywait_preempt(void *arg)
1514+{
1515+ struct intel_gt *gt = arg;
1516+ struct i915_gem_context *ctx_hi, *ctx_lo;
1517+ struct intel_engine_cs *engine;
1518+ struct drm_i915_gem_object *obj;
1519+ struct i915_vma *vma;
1520+ enum intel_engine_id id;
1521+ int err = -ENOMEM;
1522+ u32 *map;
1523+
1524+ /*
1525+ * Verify that even without HAS_LOGICAL_RING_PREEMPTION, we can
1526+ * preempt the busywaits used to synchronise between rings.
1527+ */
1528+
1529+ ctx_hi = kernel_context(gt->i915);
1530+ if (!ctx_hi)
1531+ return -ENOMEM;
1532+ ctx_hi->sched.priority =
1533+ I915_USER_PRIORITY(I915_CONTEXT_MAX_USER_PRIORITY);
1534+
1535+ ctx_lo = kernel_context(gt->i915);
1536+ if (!ctx_lo)
1537+ goto err_ctx_hi;
1538+ ctx_lo->sched.priority =
1539+ I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
1540+
1541+ obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
1542+ if (IS_ERR(obj)) {
1543+ err = PTR_ERR(obj);
1544+ goto err_ctx_lo;
1545+ }
1546+
1547+ map = i915_gem_object_pin_map(obj, I915_MAP_WC);
1548+ if (IS_ERR(map)) {
1549+ err = PTR_ERR(map);
1550+ goto err_obj;
1551+ }
1552+
1553+ vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
1554+ if (IS_ERR(vma)) {
1555+ err = PTR_ERR(vma);
1556+ goto err_map;
1557+ }
1558+
1559+ err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
1560+ if (err)
1561+ goto err_map;
1562+
1563+ err = i915_vma_sync(vma);
1564+ if (err)
1565+ goto err_vma;
1566+
1567+ for_each_engine(engine, gt, id) {
1568+ struct i915_request *lo, *hi;
1569+ struct igt_live_test t;
1570+ u32 *cs;
1571+
1572+ if (!intel_engine_has_preemption(engine))
1573+ continue;
1574+
1575+ if (!intel_engine_can_store_dword(engine))
1576+ continue;
1577+
1578+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
1579+ err = -EIO;
1580+ goto err_vma;
1581+ }
1582+
1583+ /*
1584+ * We create two requests. The low priority request
1585+ * busywaits on a semaphore (inside the ringbuffer where
1586+ * is should be preemptible) and the high priority requests
1587+ * uses a MI_STORE_DWORD_IMM to update the semaphore value
1588+ * allowing the first request to complete. If preemption
1589+ * fails, we hang instead.
1590+ */
1591+
1592+ lo = igt_request_alloc(ctx_lo, engine);
1593+ if (IS_ERR(lo)) {
1594+ err = PTR_ERR(lo);
1595+ goto err_vma;
1596+ }
1597+
1598+ cs = intel_ring_begin(lo, 8);
1599+ if (IS_ERR(cs)) {
1600+ err = PTR_ERR(cs);
1601+ i915_request_add(lo);
1602+ goto err_vma;
1603+ }
1604+
1605+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
1606+ *cs++ = i915_ggtt_offset(vma);
1607+ *cs++ = 0;
1608+ *cs++ = 1;
1609+
1610+ /* XXX Do we need a flush + invalidate here? */
1611+
1612+ *cs++ = MI_SEMAPHORE_WAIT |
1613+ MI_SEMAPHORE_GLOBAL_GTT |
1614+ MI_SEMAPHORE_POLL |
1615+ MI_SEMAPHORE_SAD_EQ_SDD;
1616+ *cs++ = 0;
1617+ *cs++ = i915_ggtt_offset(vma);
1618+ *cs++ = 0;
1619+
1620+ intel_ring_advance(lo, cs);
1621+
1622+ i915_request_get(lo);
1623+ i915_request_add(lo);
1624+
1625+ if (wait_for(READ_ONCE(*map), 10)) {
1626+ i915_request_put(lo);
1627+ err = -ETIMEDOUT;
1628+ goto err_vma;
1629+ }
1630+
1631+ /* Low priority request should be busywaiting now */
1632+ if (i915_request_wait(lo, 0, 1) != -ETIME) {
1633+ i915_request_put(lo);
1634+ pr_err("%s: Busywaiting request did not!\n",
1635+ engine->name);
1636+ err = -EIO;
1637+ goto err_vma;
1638+ }
1639+
1640+ hi = igt_request_alloc(ctx_hi, engine);
1641+ if (IS_ERR(hi)) {
1642+ err = PTR_ERR(hi);
1643+ i915_request_put(lo);
1644+ goto err_vma;
1645+ }
1646+
1647+ cs = intel_ring_begin(hi, 4);
1648+ if (IS_ERR(cs)) {
1649+ err = PTR_ERR(cs);
1650+ i915_request_add(hi);
1651+ i915_request_put(lo);
1652+ goto err_vma;
1653+ }
1654+
1655+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
1656+ *cs++ = i915_ggtt_offset(vma);
1657+ *cs++ = 0;
1658+ *cs++ = 0;
1659+
1660+ intel_ring_advance(hi, cs);
1661+ i915_request_add(hi);
1662+
1663+ if (i915_request_wait(lo, 0, HZ / 5) < 0) {
1664+ struct drm_printer p = drm_info_printer(gt->i915->drm.dev);
1665+
1666+ pr_err("%s: Failed to preempt semaphore busywait!\n",
1667+ engine->name);
1668+
1669+ intel_engine_dump(engine, &p, "%s\n", engine->name);
1670+ GEM_TRACE_DUMP();
1671+
1672+ i915_request_put(lo);
1673+ intel_gt_set_wedged(gt);
1674+ err = -EIO;
1675+ goto err_vma;
1676+ }
1677+ GEM_BUG_ON(READ_ONCE(*map));
1678+ i915_request_put(lo);
1679+
1680+ if (igt_live_test_end(&t)) {
1681+ err = -EIO;
1682+ goto err_vma;
1683+ }
1684+ }
1685+
1686+ err = 0;
1687+err_vma:
1688+ i915_vma_unpin(vma);
1689+err_map:
1690+ i915_gem_object_unpin_map(obj);
1691+err_obj:
1692+ i915_gem_object_put(obj);
1693+err_ctx_lo:
1694+ kernel_context_close(ctx_lo);
1695+err_ctx_hi:
1696+ kernel_context_close(ctx_hi);
1697+ return err;
1698+}
1699+
1700+static struct i915_request *
1701+spinner_create_request(struct igt_spinner *spin,
1702+ struct i915_gem_context *ctx,
1703+ struct intel_engine_cs *engine,
1704+ u32 arb)
1705+{
1706+ struct intel_context *ce;
1707+ struct i915_request *rq;
1708+
1709+ ce = i915_gem_context_get_engine(ctx, engine->legacy_idx);
1710+ if (IS_ERR(ce))
1711+ return ERR_CAST(ce);
1712+
1713+ rq = igt_spinner_create_request(spin, ce, arb);
1714+ intel_context_put(ce);
1715+ return rq;
1716+}
1717+
1718+static int live_preempt(void *arg)
1719+{
1720+ struct intel_gt *gt = arg;
1721+ struct i915_gem_context *ctx_hi, *ctx_lo;
1722+ struct igt_spinner spin_hi, spin_lo;
1723+ struct intel_engine_cs *engine;
1724+ enum intel_engine_id id;
1725+ int err = -ENOMEM;
1726+
1727+ if (igt_spinner_init(&spin_hi, gt))
1728+ return -ENOMEM;
1729+
1730+ if (igt_spinner_init(&spin_lo, gt))
1731+ goto err_spin_hi;
1732+
1733+ ctx_hi = kernel_context(gt->i915);
1734+ if (!ctx_hi)
1735+ goto err_spin_lo;
1736+ ctx_hi->sched.priority =
1737+ I915_USER_PRIORITY(I915_CONTEXT_MAX_USER_PRIORITY);
1738+
1739+ ctx_lo = kernel_context(gt->i915);
1740+ if (!ctx_lo)
1741+ goto err_ctx_hi;
1742+ ctx_lo->sched.priority =
1743+ I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
1744+
1745+ for_each_engine(engine, gt, id) {
1746+ struct igt_live_test t;
1747+ struct i915_request *rq;
1748+
1749+ if (!intel_engine_has_preemption(engine))
1750+ continue;
1751+
1752+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
1753+ err = -EIO;
1754+ goto err_ctx_lo;
1755+ }
1756+
1757+ rq = spinner_create_request(&spin_lo, ctx_lo, engine,
1758+ MI_ARB_CHECK);
1759+ if (IS_ERR(rq)) {
1760+ err = PTR_ERR(rq);
1761+ goto err_ctx_lo;
1762+ }
1763+
1764+ i915_request_add(rq);
1765+ if (!igt_wait_for_spinner(&spin_lo, rq)) {
1766+ GEM_TRACE("lo spinner failed to start\n");
1767+ GEM_TRACE_DUMP();
1768+ intel_gt_set_wedged(gt);
1769+ err = -EIO;
1770+ goto err_ctx_lo;
1771+ }
1772+
1773+ rq = spinner_create_request(&spin_hi, ctx_hi, engine,
1774+ MI_ARB_CHECK);
1775+ if (IS_ERR(rq)) {
1776+ igt_spinner_end(&spin_lo);
1777+ err = PTR_ERR(rq);
1778+ goto err_ctx_lo;
1779+ }
1780+
1781+ i915_request_add(rq);
1782+ if (!igt_wait_for_spinner(&spin_hi, rq)) {
1783+ GEM_TRACE("hi spinner failed to start\n");
1784+ GEM_TRACE_DUMP();
1785+ intel_gt_set_wedged(gt);
1786+ err = -EIO;
1787+ goto err_ctx_lo;
1788+ }
1789+
1790+ igt_spinner_end(&spin_hi);
1791+ igt_spinner_end(&spin_lo);
1792+
1793+ if (igt_live_test_end(&t)) {
1794+ err = -EIO;
1795+ goto err_ctx_lo;
1796+ }
1797+ }
1798+
1799+ err = 0;
1800+err_ctx_lo:
1801+ kernel_context_close(ctx_lo);
1802+err_ctx_hi:
1803+ kernel_context_close(ctx_hi);
1804+err_spin_lo:
1805+ igt_spinner_fini(&spin_lo);
1806+err_spin_hi:
1807+ igt_spinner_fini(&spin_hi);
1808+ return err;
1809+}
1810+
1811+static int live_late_preempt(void *arg)
1812+{
1813+ struct intel_gt *gt = arg;
1814+ struct i915_gem_context *ctx_hi, *ctx_lo;
1815+ struct igt_spinner spin_hi, spin_lo;
1816+ struct intel_engine_cs *engine;
1817+ struct i915_sched_attr attr = {};
1818+ enum intel_engine_id id;
1819+ int err = -ENOMEM;
1820+
1821+ if (igt_spinner_init(&spin_hi, gt))
1822+ return -ENOMEM;
1823+
1824+ if (igt_spinner_init(&spin_lo, gt))
1825+ goto err_spin_hi;
1826+
1827+ ctx_hi = kernel_context(gt->i915);
1828+ if (!ctx_hi)
1829+ goto err_spin_lo;
1830+
1831+ ctx_lo = kernel_context(gt->i915);
1832+ if (!ctx_lo)
1833+ goto err_ctx_hi;
1834+
1835+ /* Make sure ctx_lo stays before ctx_hi until we trigger preemption. */
1836+ ctx_lo->sched.priority = I915_USER_PRIORITY(1);
1837+
1838+ for_each_engine(engine, gt, id) {
1839+ struct igt_live_test t;
1840+ struct i915_request *rq;
1841+
1842+ if (!intel_engine_has_preemption(engine))
1843+ continue;
1844+
1845+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
1846+ err = -EIO;
1847+ goto err_ctx_lo;
1848+ }
1849+
1850+ rq = spinner_create_request(&spin_lo, ctx_lo, engine,
1851+ MI_ARB_CHECK);
1852+ if (IS_ERR(rq)) {
1853+ err = PTR_ERR(rq);
1854+ goto err_ctx_lo;
1855+ }
1856+
1857+ i915_request_add(rq);
1858+ if (!igt_wait_for_spinner(&spin_lo, rq)) {
1859+ pr_err("First context failed to start\n");
1860+ goto err_wedged;
1861+ }
1862+
1863+ rq = spinner_create_request(&spin_hi, ctx_hi, engine,
1864+ MI_NOOP);
1865+ if (IS_ERR(rq)) {
1866+ igt_spinner_end(&spin_lo);
1867+ err = PTR_ERR(rq);
1868+ goto err_ctx_lo;
1869+ }
1870+
1871+ i915_request_add(rq);
1872+ if (igt_wait_for_spinner(&spin_hi, rq)) {
1873+ pr_err("Second context overtook first?\n");
1874+ goto err_wedged;
1875+ }
1876+
1877+ attr.priority = I915_USER_PRIORITY(I915_PRIORITY_MAX);
1878+ engine->schedule(rq, &attr);
1879+
1880+ if (!igt_wait_for_spinner(&spin_hi, rq)) {
1881+ pr_err("High priority context failed to preempt the low priority context\n");
1882+ GEM_TRACE_DUMP();
1883+ goto err_wedged;
1884+ }
1885+
1886+ igt_spinner_end(&spin_hi);
1887+ igt_spinner_end(&spin_lo);
1888+
1889+ if (igt_live_test_end(&t)) {
1890+ err = -EIO;
1891+ goto err_ctx_lo;
1892+ }
1893+ }
1894+
1895+ err = 0;
1896+err_ctx_lo:
1897+ kernel_context_close(ctx_lo);
1898+err_ctx_hi:
1899+ kernel_context_close(ctx_hi);
1900+err_spin_lo:
1901+ igt_spinner_fini(&spin_lo);
1902+err_spin_hi:
1903+ igt_spinner_fini(&spin_hi);
1904+ return err;
1905+
1906+err_wedged:
1907+ igt_spinner_end(&spin_hi);
1908+ igt_spinner_end(&spin_lo);
1909+ intel_gt_set_wedged(gt);
1910+ err = -EIO;
1911+ goto err_ctx_lo;
1912+}
1913+
1914+struct preempt_client {
1915+ struct igt_spinner spin;
1916+ struct i915_gem_context *ctx;
1917+};
1918+
1919+static int preempt_client_init(struct intel_gt *gt, struct preempt_client *c)
1920+{
1921+ c->ctx = kernel_context(gt->i915);
1922+ if (!c->ctx)
1923+ return -ENOMEM;
1924+
1925+ if (igt_spinner_init(&c->spin, gt))
1926+ goto err_ctx;
1927+
1928+ return 0;
1929+
1930+err_ctx:
1931+ kernel_context_close(c->ctx);
1932+ return -ENOMEM;
1933+}
1934+
1935+static void preempt_client_fini(struct preempt_client *c)
1936+{
1937+ igt_spinner_fini(&c->spin);
1938+ kernel_context_close(c->ctx);
1939+}
1940+
1941+static int live_nopreempt(void *arg)
1942+{
1943+ struct intel_gt *gt = arg;
1944+ struct intel_engine_cs *engine;
1945+ struct preempt_client a, b;
1946+ enum intel_engine_id id;
1947+ int err = -ENOMEM;
1948+
1949+ /*
1950+ * Verify that we can disable preemption for an individual request
1951+ * that may be being observed and not want to be interrupted.
1952+ */
1953+
1954+ if (preempt_client_init(gt, &a))
1955+ return -ENOMEM;
1956+ if (preempt_client_init(gt, &b))
1957+ goto err_client_a;
1958+ b.ctx->sched.priority = I915_USER_PRIORITY(I915_PRIORITY_MAX);
1959+
1960+ for_each_engine(engine, gt, id) {
1961+ struct i915_request *rq_a, *rq_b;
1962+
1963+ if (!intel_engine_has_preemption(engine))
1964+ continue;
1965+
1966+ engine->execlists.preempt_hang.count = 0;
1967+
1968+ rq_a = spinner_create_request(&a.spin,
1969+ a.ctx, engine,
1970+ MI_ARB_CHECK);
1971+ if (IS_ERR(rq_a)) {
1972+ err = PTR_ERR(rq_a);
1973+ goto err_client_b;
1974+ }
1975+
1976+ /* Low priority client, but unpreemptable! */
1977+ __set_bit(I915_FENCE_FLAG_NOPREEMPT, &rq_a->fence.flags);
1978+
1979+ i915_request_add(rq_a);
1980+ if (!igt_wait_for_spinner(&a.spin, rq_a)) {
1981+ pr_err("First client failed to start\n");
1982+ goto err_wedged;
1983+ }
1984+
1985+ rq_b = spinner_create_request(&b.spin,
1986+ b.ctx, engine,
1987+ MI_ARB_CHECK);
1988+ if (IS_ERR(rq_b)) {
1989+ err = PTR_ERR(rq_b);
1990+ goto err_client_b;
1991+ }
1992+
1993+ i915_request_add(rq_b);
1994+
1995+ /* B is much more important than A! (But A is unpreemptable.) */
1996+ GEM_BUG_ON(rq_prio(rq_b) <= rq_prio(rq_a));
1997+
1998+ /* Wait long enough for preemption and timeslicing */
1999+ if (igt_wait_for_spinner(&b.spin, rq_b)) {
2000+ pr_err("Second client started too early!\n");
2001+ goto err_wedged;
2002+ }
2003+
2004+ igt_spinner_end(&a.spin);
2005+
2006+ if (!igt_wait_for_spinner(&b.spin, rq_b)) {
2007+ pr_err("Second client failed to start\n");
2008+ goto err_wedged;
2009+ }
2010+
2011+ igt_spinner_end(&b.spin);
2012+
2013+ if (engine->execlists.preempt_hang.count) {
2014+ pr_err("Preemption recorded x%d; should have been suppressed!\n",
2015+ engine->execlists.preempt_hang.count);
2016+ err = -EINVAL;
2017+ goto err_wedged;
2018+ }
2019+
2020+ if (igt_flush_test(gt->i915))
2021+ goto err_wedged;
2022+ }
2023+
2024+ err = 0;
2025+err_client_b:
2026+ preempt_client_fini(&b);
2027+err_client_a:
2028+ preempt_client_fini(&a);
2029+ return err;
2030+
2031+err_wedged:
2032+ igt_spinner_end(&b.spin);
2033+ igt_spinner_end(&a.spin);
2034+ intel_gt_set_wedged(gt);
2035+ err = -EIO;
2036+ goto err_client_b;
2037+}
2038+
2039+struct live_preempt_cancel {
2040+ struct intel_engine_cs *engine;
2041+ struct preempt_client a, b;
2042+};
2043+
2044+static int __cancel_active0(struct live_preempt_cancel *arg)
2045+{
2046+ struct i915_request *rq;
2047+ struct igt_live_test t;
2048+ int err;
2049+
2050+ /* Preempt cancel of ELSP0 */
2051+ GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2052+ if (igt_live_test_begin(&t, arg->engine->i915,
2053+ __func__, arg->engine->name))
2054+ return -EIO;
2055+
2056+ rq = spinner_create_request(&arg->a.spin,
2057+ arg->a.ctx, arg->engine,
2058+ MI_ARB_CHECK);
2059+ if (IS_ERR(rq))
2060+ return PTR_ERR(rq);
2061+
2062+ clear_bit(CONTEXT_BANNED, &rq->context->flags);
2063+ i915_request_get(rq);
2064+ i915_request_add(rq);
2065+ if (!igt_wait_for_spinner(&arg->a.spin, rq)) {
2066+ err = -EIO;
2067+ goto out;
2068+ }
2069+
2070+ intel_context_set_banned(rq->context);
2071+ err = intel_engine_pulse(arg->engine);
2072+ if (err)
2073+ goto out;
2074+
2075+ err = wait_for_reset(arg->engine, rq, HZ / 2);
2076+ if (err) {
2077+ pr_err("Cancelled inflight0 request did not reset\n");
2078+ goto out;
2079+ }
2080+
2081+out:
2082+ i915_request_put(rq);
2083+ if (igt_live_test_end(&t))
2084+ err = -EIO;
2085+ return err;
2086+}
2087+
2088+static int __cancel_active1(struct live_preempt_cancel *arg)
2089+{
2090+ struct i915_request *rq[2] = {};
2091+ struct igt_live_test t;
2092+ int err;
2093+
2094+ /* Preempt cancel of ELSP1 */
2095+ GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2096+ if (igt_live_test_begin(&t, arg->engine->i915,
2097+ __func__, arg->engine->name))
2098+ return -EIO;
2099+
2100+ rq[0] = spinner_create_request(&arg->a.spin,
2101+ arg->a.ctx, arg->engine,
2102+ MI_NOOP); /* no preemption */
2103+ if (IS_ERR(rq[0]))
2104+ return PTR_ERR(rq[0]);
2105+
2106+ clear_bit(CONTEXT_BANNED, &rq[0]->context->flags);
2107+ i915_request_get(rq[0]);
2108+ i915_request_add(rq[0]);
2109+ if (!igt_wait_for_spinner(&arg->a.spin, rq[0])) {
2110+ err = -EIO;
2111+ goto out;
2112+ }
2113+
2114+ rq[1] = spinner_create_request(&arg->b.spin,
2115+ arg->b.ctx, arg->engine,
2116+ MI_ARB_CHECK);
2117+ if (IS_ERR(rq[1])) {
2118+ err = PTR_ERR(rq[1]);
2119+ goto out;
2120+ }
2121+
2122+ clear_bit(CONTEXT_BANNED, &rq[1]->context->flags);
2123+ i915_request_get(rq[1]);
2124+ err = i915_request_await_dma_fence(rq[1], &rq[0]->fence);
2125+ i915_request_add(rq[1]);
2126+ if (err)
2127+ goto out;
2128+
2129+ intel_context_set_banned(rq[1]->context);
2130+ err = intel_engine_pulse(arg->engine);
2131+ if (err)
2132+ goto out;
2133+
2134+ igt_spinner_end(&arg->a.spin);
2135+ err = wait_for_reset(arg->engine, rq[1], HZ / 2);
2136+ if (err)
2137+ goto out;
2138+
2139+ if (rq[0]->fence.error != 0) {
2140+ pr_err("Normal inflight0 request did not complete\n");
2141+ err = -EINVAL;
2142+ goto out;
2143+ }
2144+
2145+ if (rq[1]->fence.error != -EIO) {
2146+ pr_err("Cancelled inflight1 request did not report -EIO\n");
2147+ err = -EINVAL;
2148+ goto out;
2149+ }
2150+
2151+out:
2152+ i915_request_put(rq[1]);
2153+ i915_request_put(rq[0]);
2154+ if (igt_live_test_end(&t))
2155+ err = -EIO;
2156+ return err;
2157+}
2158+
2159+static int __cancel_queued(struct live_preempt_cancel *arg)
2160+{
2161+ struct i915_request *rq[3] = {};
2162+ struct igt_live_test t;
2163+ int err;
2164+
2165+ /* Full ELSP and one in the wings */
2166+ GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2167+ if (igt_live_test_begin(&t, arg->engine->i915,
2168+ __func__, arg->engine->name))
2169+ return -EIO;
2170+
2171+ rq[0] = spinner_create_request(&arg->a.spin,
2172+ arg->a.ctx, arg->engine,
2173+ MI_ARB_CHECK);
2174+ if (IS_ERR(rq[0]))
2175+ return PTR_ERR(rq[0]);
2176+
2177+ clear_bit(CONTEXT_BANNED, &rq[0]->context->flags);
2178+ i915_request_get(rq[0]);
2179+ i915_request_add(rq[0]);
2180+ if (!igt_wait_for_spinner(&arg->a.spin, rq[0])) {
2181+ err = -EIO;
2182+ goto out;
2183+ }
2184+
2185+ rq[1] = igt_request_alloc(arg->b.ctx, arg->engine);
2186+ if (IS_ERR(rq[1])) {
2187+ err = PTR_ERR(rq[1]);
2188+ goto out;
2189+ }
2190+
2191+ clear_bit(CONTEXT_BANNED, &rq[1]->context->flags);
2192+ i915_request_get(rq[1]);
2193+ err = i915_request_await_dma_fence(rq[1], &rq[0]->fence);
2194+ i915_request_add(rq[1]);
2195+ if (err)
2196+ goto out;
2197+
2198+ rq[2] = spinner_create_request(&arg->b.spin,
2199+ arg->a.ctx, arg->engine,
2200+ MI_ARB_CHECK);
2201+ if (IS_ERR(rq[2])) {
2202+ err = PTR_ERR(rq[2]);
2203+ goto out;
2204+ }
2205+
2206+ i915_request_get(rq[2]);
2207+ err = i915_request_await_dma_fence(rq[2], &rq[1]->fence);
2208+ i915_request_add(rq[2]);
2209+ if (err)
2210+ goto out;
2211+
2212+ intel_context_set_banned(rq[2]->context);
2213+ err = intel_engine_pulse(arg->engine);
2214+ if (err)
2215+ goto out;
2216+
2217+ err = wait_for_reset(arg->engine, rq[2], HZ / 2);
2218+ if (err)
2219+ goto out;
2220+
2221+ if (rq[0]->fence.error != -EIO) {
2222+ pr_err("Cancelled inflight0 request did not report -EIO\n");
2223+ err = -EINVAL;
2224+ goto out;
2225+ }
2226+
2227+ if (rq[1]->fence.error != 0) {
2228+ pr_err("Normal inflight1 request did not complete\n");
2229+ err = -EINVAL;
2230+ goto out;
2231+ }
2232+
2233+ if (rq[2]->fence.error != -EIO) {
2234+ pr_err("Cancelled queued request did not report -EIO\n");
2235+ err = -EINVAL;
2236+ goto out;
2237+ }
2238+
2239+out:
2240+ i915_request_put(rq[2]);
2241+ i915_request_put(rq[1]);
2242+ i915_request_put(rq[0]);
2243+ if (igt_live_test_end(&t))
2244+ err = -EIO;
2245+ return err;
2246+}
2247+
2248+static int __cancel_hostile(struct live_preempt_cancel *arg)
2249+{
2250+ struct i915_request *rq;
2251+ int err;
2252+
2253+ /* Preempt cancel non-preemptible spinner in ELSP0 */
2254+ if (!IS_ACTIVE(CONFIG_DRM_I915_PREEMPT_TIMEOUT))
2255+ return 0;
2256+
2257+ if (!intel_has_reset_engine(arg->engine->gt))
2258+ return 0;
2259+
2260+ GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2261+ rq = spinner_create_request(&arg->a.spin,
2262+ arg->a.ctx, arg->engine,
2263+ MI_NOOP); /* preemption disabled */
2264+ if (IS_ERR(rq))
2265+ return PTR_ERR(rq);
2266+
2267+ clear_bit(CONTEXT_BANNED, &rq->context->flags);
2268+ i915_request_get(rq);
2269+ i915_request_add(rq);
2270+ if (!igt_wait_for_spinner(&arg->a.spin, rq)) {
2271+ err = -EIO;
2272+ goto out;
2273+ }
2274+
2275+ intel_context_set_banned(rq->context);
2276+ err = intel_engine_pulse(arg->engine); /* force reset */
2277+ if (err)
2278+ goto out;
2279+
2280+ err = wait_for_reset(arg->engine, rq, HZ / 2);
2281+ if (err) {
2282+ pr_err("Cancelled inflight0 request did not reset\n");
2283+ goto out;
2284+ }
2285+
2286+out:
2287+ i915_request_put(rq);
2288+ if (igt_flush_test(arg->engine->i915))
2289+ err = -EIO;
2290+ return err;
2291+}
2292+
2293+static void force_reset_timeout(struct intel_engine_cs *engine)
2294+{
2295+ engine->reset_timeout.probability = 999;
2296+ atomic_set(&engine->reset_timeout.times, -1);
2297+}
2298+
2299+static void cancel_reset_timeout(struct intel_engine_cs *engine)
2300+{
2301+ memset(&engine->reset_timeout, 0, sizeof(engine->reset_timeout));
2302+}
2303+
2304+static int __cancel_fail(struct live_preempt_cancel *arg)
2305+{
2306+ struct intel_engine_cs *engine = arg->engine;
2307+ struct i915_request *rq;
2308+ int err;
2309+
2310+ if (!IS_ACTIVE(CONFIG_DRM_I915_PREEMPT_TIMEOUT))
2311+ return 0;
2312+
2313+ if (!intel_has_reset_engine(engine->gt))
2314+ return 0;
2315+
2316+ GEM_TRACE("%s(%s)\n", __func__, engine->name);
2317+ rq = spinner_create_request(&arg->a.spin,
2318+ arg->a.ctx, engine,
2319+ MI_NOOP); /* preemption disabled */
2320+ if (IS_ERR(rq))
2321+ return PTR_ERR(rq);
2322+
2323+ clear_bit(CONTEXT_BANNED, &rq->context->flags);
2324+ i915_request_get(rq);
2325+ i915_request_add(rq);
2326+ if (!igt_wait_for_spinner(&arg->a.spin, rq)) {
2327+ err = -EIO;
2328+ goto out;
2329+ }
2330+
2331+ intel_context_set_banned(rq->context);
2332+
2333+ err = intel_engine_pulse(engine);
2334+ if (err)
2335+ goto out;
2336+
2337+ force_reset_timeout(engine);
2338+
2339+ /* force preempt reset [failure] */
2340+ while (!engine->execlists.pending[0])
2341+ intel_engine_flush_submission(engine);
2342+ del_timer_sync(&engine->execlists.preempt);
2343+ intel_engine_flush_submission(engine);
2344+
2345+ cancel_reset_timeout(engine);
2346+
2347+ /* after failure, require heartbeats to reset device */
2348+ intel_engine_set_heartbeat(engine, 1);
2349+ err = wait_for_reset(engine, rq, HZ / 2);
2350+ intel_engine_set_heartbeat(engine,
2351+ engine->defaults.heartbeat_interval_ms);
2352+ if (err) {
2353+ pr_err("Cancelled inflight0 request did not reset\n");
2354+ goto out;
2355+ }
2356+
2357+out:
2358+ i915_request_put(rq);
2359+ if (igt_flush_test(engine->i915))
2360+ err = -EIO;
2361+ return err;
2362+}
2363+
2364+static int live_preempt_cancel(void *arg)
2365+{
2366+ struct intel_gt *gt = arg;
2367+ struct live_preempt_cancel data;
2368+ enum intel_engine_id id;
2369+ int err = -ENOMEM;
2370+
2371+ /*
2372+ * To cancel an inflight context, we need to first remove it from the
2373+ * GPU. That sounds like preemption! Plus a little bit of bookkeeping.
2374+ */
2375+
2376+ if (preempt_client_init(gt, &data.a))
2377+ return -ENOMEM;
2378+ if (preempt_client_init(gt, &data.b))
2379+ goto err_client_a;
2380+
2381+ for_each_engine(data.engine, gt, id) {
2382+ if (!intel_engine_has_preemption(data.engine))
2383+ continue;
2384+
2385+ err = __cancel_active0(&data);
2386+ if (err)
2387+ goto err_wedged;
2388+
2389+ err = __cancel_active1(&data);
2390+ if (err)
2391+ goto err_wedged;
2392+
2393+ err = __cancel_queued(&data);
2394+ if (err)
2395+ goto err_wedged;
2396+
2397+ err = __cancel_hostile(&data);
2398+ if (err)
2399+ goto err_wedged;
2400+
2401+ err = __cancel_fail(&data);
2402+ if (err)
2403+ goto err_wedged;
2404+ }
2405+
2406+ err = 0;
2407+err_client_b:
2408+ preempt_client_fini(&data.b);
2409+err_client_a:
2410+ preempt_client_fini(&data.a);
2411+ return err;
2412+
2413+err_wedged:
2414+ GEM_TRACE_DUMP();
2415+ igt_spinner_end(&data.b.spin);
2416+ igt_spinner_end(&data.a.spin);
2417+ intel_gt_set_wedged(gt);
2418+ goto err_client_b;
2419+}
2420+
2421+static int live_suppress_self_preempt(void *arg)
2422+{
2423+ struct intel_gt *gt = arg;
2424+ struct intel_engine_cs *engine;
2425+ struct i915_sched_attr attr = {
2426+ .priority = I915_USER_PRIORITY(I915_PRIORITY_MAX)
2427+ };
2428+ struct preempt_client a, b;
2429+ enum intel_engine_id id;
2430+ int err = -ENOMEM;
2431+
2432+ /*
2433+ * Verify that if a preemption request does not cause a change in
2434+ * the current execution order, the preempt-to-idle injection is
2435+ * skipped and that we do not accidentally apply it after the CS
2436+ * completion event.
2437+ */
2438+
2439+ if (intel_uc_uses_guc_submission(&gt->uc))
2440+ return 0; /* presume black blox */
2441+
2442+ if (intel_vgpu_active(gt->i915))
2443+ return 0; /* GVT forces single port & request submission */
2444+
2445+ if (preempt_client_init(gt, &a))
2446+ return -ENOMEM;
2447+ if (preempt_client_init(gt, &b))
2448+ goto err_client_a;
2449+
2450+ for_each_engine(engine, gt, id) {
2451+ struct i915_request *rq_a, *rq_b;
2452+ int depth;
2453+
2454+ if (!intel_engine_has_preemption(engine))
2455+ continue;
2456+
2457+ if (igt_flush_test(gt->i915))
2458+ goto err_wedged;
2459+
2460+ st_engine_heartbeat_disable(engine);
2461+ engine->execlists.preempt_hang.count = 0;
2462+
2463+ rq_a = spinner_create_request(&a.spin,
2464+ a.ctx, engine,
2465+ MI_NOOP);
2466+ if (IS_ERR(rq_a)) {
2467+ err = PTR_ERR(rq_a);
2468+ st_engine_heartbeat_enable(engine);
2469+ goto err_client_b;
2470+ }
2471+
2472+ i915_request_add(rq_a);
2473+ if (!igt_wait_for_spinner(&a.spin, rq_a)) {
2474+ pr_err("First client failed to start\n");
2475+ st_engine_heartbeat_enable(engine);
2476+ goto err_wedged;
2477+ }
2478+
2479+ /* Keep postponing the timer to avoid premature slicing */
2480+ mod_timer(&engine->execlists.timer, jiffies + HZ);
2481+ for (depth = 0; depth < 8; depth++) {
2482+ rq_b = spinner_create_request(&b.spin,
2483+ b.ctx, engine,
2484+ MI_NOOP);
2485+ if (IS_ERR(rq_b)) {
2486+ err = PTR_ERR(rq_b);
2487+ st_engine_heartbeat_enable(engine);
2488+ goto err_client_b;
2489+ }
2490+ i915_request_add(rq_b);
2491+
2492+ GEM_BUG_ON(i915_request_completed(rq_a));
2493+ engine->schedule(rq_a, &attr);
2494+ igt_spinner_end(&a.spin);
2495+
2496+ if (!igt_wait_for_spinner(&b.spin, rq_b)) {
2497+ pr_err("Second client failed to start\n");
2498+ st_engine_heartbeat_enable(engine);
2499+ goto err_wedged;
2500+ }
2501+
2502+ swap(a, b);
2503+ rq_a = rq_b;
2504+ }
2505+ igt_spinner_end(&a.spin);
2506+
2507+ if (engine->execlists.preempt_hang.count) {
2508+ pr_err("Preemption on %s recorded x%d, depth %d; should have been suppressed!\n",
2509+ engine->name,
2510+ engine->execlists.preempt_hang.count,
2511+ depth);
2512+ st_engine_heartbeat_enable(engine);
2513+ err = -EINVAL;
2514+ goto err_client_b;
2515+ }
2516+
2517+ st_engine_heartbeat_enable(engine);
2518+ if (igt_flush_test(gt->i915))
2519+ goto err_wedged;
2520+ }
2521+
2522+ err = 0;
2523+err_client_b:
2524+ preempt_client_fini(&b);
2525+err_client_a:
2526+ preempt_client_fini(&a);
2527+ return err;
2528+
2529+err_wedged:
2530+ igt_spinner_end(&b.spin);
2531+ igt_spinner_end(&a.spin);
2532+ intel_gt_set_wedged(gt);
2533+ err = -EIO;
2534+ goto err_client_b;
2535+}
2536+
2537+static int live_chain_preempt(void *arg)
2538+{
2539+ struct intel_gt *gt = arg;
2540+ struct intel_engine_cs *engine;
2541+ struct preempt_client hi, lo;
2542+ enum intel_engine_id id;
2543+ int err = -ENOMEM;
2544+
2545+ /*
2546+ * Build a chain AB...BA between two contexts (A, B) and request
2547+ * preemption of the last request. It should then complete before
2548+ * the previously submitted spinner in B.
2549+ */
2550+
2551+ if (preempt_client_init(gt, &hi))
2552+ return -ENOMEM;
2553+
2554+ if (preempt_client_init(gt, &lo))
2555+ goto err_client_hi;
2556+
2557+ for_each_engine(engine, gt, id) {
2558+ struct i915_sched_attr attr = {
2559+ .priority = I915_USER_PRIORITY(I915_PRIORITY_MAX),
2560+ };
2561+ struct igt_live_test t;
2562+ struct i915_request *rq;
2563+ int ring_size, count, i;
2564+
2565+ if (!intel_engine_has_preemption(engine))
2566+ continue;
2567+
2568+ rq = spinner_create_request(&lo.spin,
2569+ lo.ctx, engine,
2570+ MI_ARB_CHECK);
2571+ if (IS_ERR(rq))
2572+ goto err_wedged;
2573+
2574+ i915_request_get(rq);
2575+ i915_request_add(rq);
2576+
2577+ ring_size = rq->wa_tail - rq->head;
2578+ if (ring_size < 0)
2579+ ring_size += rq->ring->size;
2580+ ring_size = rq->ring->size / ring_size;
2581+ pr_debug("%s(%s): Using maximum of %d requests\n",
2582+ __func__, engine->name, ring_size);
2583+
2584+ igt_spinner_end(&lo.spin);
2585+ if (i915_request_wait(rq, 0, HZ / 2) < 0) {
2586+ pr_err("Timed out waiting to flush %s\n", engine->name);
2587+ i915_request_put(rq);
2588+ goto err_wedged;
2589+ }
2590+ i915_request_put(rq);
2591+
2592+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
2593+ err = -EIO;
2594+ goto err_wedged;
2595+ }
2596+
2597+ for_each_prime_number_from(count, 1, ring_size) {
2598+ rq = spinner_create_request(&hi.spin,
2599+ hi.ctx, engine,
2600+ MI_ARB_CHECK);
2601+ if (IS_ERR(rq))
2602+ goto err_wedged;
2603+ i915_request_add(rq);
2604+ if (!igt_wait_for_spinner(&hi.spin, rq))
2605+ goto err_wedged;
2606+
2607+ rq = spinner_create_request(&lo.spin,
2608+ lo.ctx, engine,
2609+ MI_ARB_CHECK);
2610+ if (IS_ERR(rq))
2611+ goto err_wedged;
2612+ i915_request_add(rq);
2613+
2614+ for (i = 0; i < count; i++) {
2615+ rq = igt_request_alloc(lo.ctx, engine);
2616+ if (IS_ERR(rq))
2617+ goto err_wedged;
2618+ i915_request_add(rq);
2619+ }
2620+
2621+ rq = igt_request_alloc(hi.ctx, engine);
2622+ if (IS_ERR(rq))
2623+ goto err_wedged;
2624+
2625+ i915_request_get(rq);
2626+ i915_request_add(rq);
2627+ engine->schedule(rq, &attr);
2628+
2629+ igt_spinner_end(&hi.spin);
2630+ if (i915_request_wait(rq, 0, HZ / 5) < 0) {
2631+ struct drm_printer p =
2632+ drm_info_printer(gt->i915->drm.dev);
2633+
2634+ pr_err("Failed to preempt over chain of %d\n",
2635+ count);
2636+ intel_engine_dump(engine, &p,
2637+ "%s\n", engine->name);
2638+ i915_request_put(rq);
2639+ goto err_wedged;
2640+ }
2641+ igt_spinner_end(&lo.spin);
2642+ i915_request_put(rq);
2643+
2644+ rq = igt_request_alloc(lo.ctx, engine);
2645+ if (IS_ERR(rq))
2646+ goto err_wedged;
2647+
2648+ i915_request_get(rq);
2649+ i915_request_add(rq);
2650+
2651+ if (i915_request_wait(rq, 0, HZ / 5) < 0) {
2652+ struct drm_printer p =
2653+ drm_info_printer(gt->i915->drm.dev);
2654+
2655+ pr_err("Failed to flush low priority chain of %d requests\n",
2656+ count);
2657+ intel_engine_dump(engine, &p,
2658+ "%s\n", engine->name);
2659+
2660+ i915_request_put(rq);
2661+ goto err_wedged;
2662+ }
2663+ i915_request_put(rq);
2664+ }
2665+
2666+ if (igt_live_test_end(&t)) {
2667+ err = -EIO;
2668+ goto err_wedged;
2669+ }
2670+ }
2671+
2672+ err = 0;
2673+err_client_lo:
2674+ preempt_client_fini(&lo);
2675+err_client_hi:
2676+ preempt_client_fini(&hi);
2677+ return err;
2678+
2679+err_wedged:
2680+ igt_spinner_end(&hi.spin);
2681+ igt_spinner_end(&lo.spin);
2682+ intel_gt_set_wedged(gt);
2683+ err = -EIO;
2684+ goto err_client_lo;
2685+}
2686+
2687+static int create_gang(struct intel_engine_cs *engine,
2688+ struct i915_request **prev)
2689+{
2690+ struct drm_i915_gem_object *obj;
2691+ struct intel_context *ce;
2692+ struct i915_request *rq;
2693+ struct i915_vma *vma;
2694+ u32 *cs;
2695+ int err;
2696+
2697+ ce = intel_context_create(engine);
2698+ if (IS_ERR(ce))
2699+ return PTR_ERR(ce);
2700+
2701+ obj = i915_gem_object_create_internal(engine->i915, 4096);
2702+ if (IS_ERR(obj)) {
2703+ err = PTR_ERR(obj);
2704+ goto err_ce;
2705+ }
2706+
2707+ vma = i915_vma_instance(obj, ce->vm, NULL);
2708+ if (IS_ERR(vma)) {
2709+ err = PTR_ERR(vma);
2710+ goto err_obj;
2711+ }
2712+
2713+ err = i915_vma_pin(vma, 0, 0, PIN_USER);
2714+ if (err)
2715+ goto err_obj;
2716+
2717+ cs = i915_gem_object_pin_map(obj, I915_MAP_WC);
2718+ if (IS_ERR(cs)) {
2719+ err = PTR_ERR(cs);
2720+ goto err_obj;
2721+ }
2722+
2723+ /* Semaphore target: spin until zero */
2724+ *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
2725+
2726+ *cs++ = MI_SEMAPHORE_WAIT |
2727+ MI_SEMAPHORE_POLL |
2728+ MI_SEMAPHORE_SAD_EQ_SDD;
2729+ *cs++ = 0;
2730+ *cs++ = lower_32_bits(vma->node.start);
2731+ *cs++ = upper_32_bits(vma->node.start);
2732+
2733+ if (*prev) {
2734+ u64 offset = (*prev)->batch->node.start;
2735+
2736+ /* Terminate the spinner in the next lower priority batch. */
2737+ *cs++ = MI_STORE_DWORD_IMM_GEN4;
2738+ *cs++ = lower_32_bits(offset);
2739+ *cs++ = upper_32_bits(offset);
2740+ *cs++ = 0;
2741+ }
2742+
2743+ *cs++ = MI_BATCH_BUFFER_END;
2744+ i915_gem_object_flush_map(obj);
2745+ i915_gem_object_unpin_map(obj);
2746+
2747+ rq = intel_context_create_request(ce);
2748+ if (IS_ERR(rq)) {
2749+ err = PTR_ERR(rq);
2750+ goto err_obj;
2751+ }
2752+
2753+ rq->batch = i915_vma_get(vma);
2754+ i915_request_get(rq);
2755+
2756+ i915_vma_lock(vma);
2757+ err = i915_request_await_object(rq, vma->obj, false);
2758+ if (!err)
2759+ err = i915_vma_move_to_active(vma, rq, 0);
2760+ if (!err)
2761+ err = rq->engine->emit_bb_start(rq,
2762+ vma->node.start,
2763+ PAGE_SIZE, 0);
2764+ i915_vma_unlock(vma);
2765+ i915_request_add(rq);
2766+ if (err)
2767+ goto err_rq;
2768+
2769+ i915_gem_object_put(obj);
2770+ intel_context_put(ce);
2771+
2772+ rq->mock.link.next = &(*prev)->mock.link;
2773+ *prev = rq;
2774+ return 0;
2775+
2776+err_rq:
2777+ i915_vma_put(rq->batch);
2778+ i915_request_put(rq);
2779+err_obj:
2780+ i915_gem_object_put(obj);
2781+err_ce:
2782+ intel_context_put(ce);
2783+ return err;
2784+}
2785+
2786+static int __live_preempt_ring(struct intel_engine_cs *engine,
2787+ struct igt_spinner *spin,
2788+ int queue_sz, int ring_sz)
2789+{
2790+ struct intel_context *ce[2] = {};
2791+ struct i915_request *rq;
2792+ struct igt_live_test t;
2793+ int err = 0;
2794+ int n;
2795+
2796+ if (igt_live_test_begin(&t, engine->i915, __func__, engine->name))
2797+ return -EIO;
2798+
2799+ for (n = 0; n < ARRAY_SIZE(ce); n++) {
2800+ struct intel_context *tmp;
2801+
2802+ tmp = intel_context_create(engine);
2803+ if (IS_ERR(tmp)) {
2804+ err = PTR_ERR(tmp);
2805+ goto err_ce;
2806+ }
2807+
2808+ tmp->ring = __intel_context_ring_size(ring_sz);
2809+
2810+ err = intel_context_pin(tmp);
2811+ if (err) {
2812+ intel_context_put(tmp);
2813+ goto err_ce;
2814+ }
2815+
2816+ memset32(tmp->ring->vaddr,
2817+ 0xdeadbeef, /* trigger a hang if executed */
2818+ tmp->ring->vma->size / sizeof(u32));
2819+
2820+ ce[n] = tmp;
2821+ }
2822+
2823+ rq = igt_spinner_create_request(spin, ce[0], MI_ARB_CHECK);
2824+ if (IS_ERR(rq)) {
2825+ err = PTR_ERR(rq);
2826+ goto err_ce;
2827+ }
2828+
2829+ i915_request_get(rq);
2830+ rq->sched.attr.priority = I915_PRIORITY_BARRIER;
2831+ i915_request_add(rq);
2832+
2833+ if (!igt_wait_for_spinner(spin, rq)) {
2834+ intel_gt_set_wedged(engine->gt);
2835+ i915_request_put(rq);
2836+ err = -ETIME;
2837+ goto err_ce;
2838+ }
2839+
2840+ /* Fill the ring, until we will cause a wrap */
2841+ n = 0;
2842+ while (ce[0]->ring->tail - rq->wa_tail <= queue_sz) {
2843+ struct i915_request *tmp;
2844+
2845+ tmp = intel_context_create_request(ce[0]);
2846+ if (IS_ERR(tmp)) {
2847+ err = PTR_ERR(tmp);
2848+ i915_request_put(rq);
2849+ goto err_ce;
2850+ }
2851+
2852+ i915_request_add(tmp);
2853+ intel_engine_flush_submission(engine);
2854+ n++;
2855+ }
2856+ intel_engine_flush_submission(engine);
2857+ pr_debug("%s: Filled %d with %d nop tails {size:%x, tail:%x, emit:%x, rq.tail:%x}\n",
2858+ engine->name, queue_sz, n,
2859+ ce[0]->ring->size,
2860+ ce[0]->ring->tail,
2861+ ce[0]->ring->emit,
2862+ rq->tail);
2863+ i915_request_put(rq);
2864+
2865+ /* Create a second request to preempt the first ring */
2866+ rq = intel_context_create_request(ce[1]);
2867+ if (IS_ERR(rq)) {
2868+ err = PTR_ERR(rq);
2869+ goto err_ce;
2870+ }
2871+
2872+ rq->sched.attr.priority = I915_PRIORITY_BARRIER;
2873+ i915_request_get(rq);
2874+ i915_request_add(rq);
2875+
2876+ err = wait_for_submit(engine, rq, HZ / 2);
2877+ i915_request_put(rq);
2878+ if (err) {
2879+ pr_err("%s: preemption request was not submited\n",
2880+ engine->name);
2881+ err = -ETIME;
2882+ }
2883+
2884+ pr_debug("%s: ring[0]:{ tail:%x, emit:%x }, ring[1]:{ tail:%x, emit:%x }\n",
2885+ engine->name,
2886+ ce[0]->ring->tail, ce[0]->ring->emit,
2887+ ce[1]->ring->tail, ce[1]->ring->emit);
2888+
2889+err_ce:
2890+ intel_engine_flush_submission(engine);
2891+ igt_spinner_end(spin);
2892+ for (n = 0; n < ARRAY_SIZE(ce); n++) {
2893+ if (IS_ERR_OR_NULL(ce[n]))
2894+ break;
2895+
2896+ intel_context_unpin(ce[n]);
2897+ intel_context_put(ce[n]);
2898+ }
2899+ if (igt_live_test_end(&t))
2900+ err = -EIO;
2901+ return err;
2902+}
2903+
2904+static int live_preempt_ring(void *arg)
2905+{
2906+ struct intel_gt *gt = arg;
2907+ struct intel_engine_cs *engine;
2908+ struct igt_spinner spin;
2909+ enum intel_engine_id id;
2910+ int err = 0;
2911+
2912+ /*
2913+ * Check that we rollback large chunks of a ring in order to do a
2914+ * preemption event. Similar to live_unlite_ring, but looking at
2915+ * ring size rather than the impact of intel_ring_direction().
2916+ */
2917+
2918+ if (igt_spinner_init(&spin, gt))
2919+ return -ENOMEM;
2920+
2921+ for_each_engine(engine, gt, id) {
2922+ int n;
2923+
2924+ if (!intel_engine_has_preemption(engine))
2925+ continue;
2926+
2927+ if (!intel_engine_can_store_dword(engine))
2928+ continue;
2929+
2930+ st_engine_heartbeat_disable(engine);
2931+
2932+ for (n = 0; n <= 3; n++) {
2933+ err = __live_preempt_ring(engine, &spin,
2934+ n * SZ_4K / 4, SZ_4K);
2935+ if (err)
2936+ break;
2937+ }
2938+
2939+ st_engine_heartbeat_enable(engine);
2940+ if (err)
2941+ break;
2942+ }
2943+
2944+ igt_spinner_fini(&spin);
2945+ return err;
2946+}
2947+
2948+static int live_preempt_gang(void *arg)
2949+{
2950+ struct intel_gt *gt = arg;
2951+ struct intel_engine_cs *engine;
2952+ enum intel_engine_id id;
2953+
2954+ /*
2955+ * Build as long a chain of preempters as we can, with each
2956+ * request higher priority than the last. Once we are ready, we release
2957+ * the last batch which then precolates down the chain, each releasing
2958+ * the next oldest in turn. The intent is to simply push as hard as we
2959+ * can with the number of preemptions, trying to exceed narrow HW
2960+ * limits. At a minimum, we insist that we can sort all the user
2961+ * high priority levels into execution order.
2962+ */
2963+
2964+ for_each_engine(engine, gt, id) {
2965+ struct i915_request *rq = NULL;
2966+ struct igt_live_test t;
2967+ IGT_TIMEOUT(end_time);
2968+ int prio = 0;
2969+ int err = 0;
2970+ u32 *cs;
2971+
2972+ if (!intel_engine_has_preemption(engine))
2973+ continue;
2974+
2975+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name))
2976+ return -EIO;
2977+
2978+ do {
2979+ struct i915_sched_attr attr = {
2980+ .priority = I915_USER_PRIORITY(prio++),
2981+ };
2982+
2983+ err = create_gang(engine, &rq);
2984+ if (err)
2985+ break;
2986+
2987+ /* Submit each spinner at increasing priority */
2988+ engine->schedule(rq, &attr);
2989+ } while (prio <= I915_PRIORITY_MAX &&
2990+ !__igt_timeout(end_time, NULL));
2991+ pr_debug("%s: Preempt chain of %d requests\n",
2992+ engine->name, prio);
2993+
2994+ /*
2995+ * Such that the last spinner is the highest priority and
2996+ * should execute first. When that spinner completes,
2997+ * it will terminate the next lowest spinner until there
2998+ * are no more spinners and the gang is complete.
2999+ */
3000+ cs = i915_gem_object_pin_map(rq->batch->obj, I915_MAP_WC);
3001+ if (!IS_ERR(cs)) {
3002+ *cs = 0;
3003+ i915_gem_object_unpin_map(rq->batch->obj);
3004+ } else {
3005+ err = PTR_ERR(cs);
3006+ intel_gt_set_wedged(gt);
3007+ }
3008+
3009+ while (rq) { /* wait for each rq from highest to lowest prio */
3010+ struct i915_request *n = list_next_entry(rq, mock.link);
3011+
3012+ if (err == 0 && i915_request_wait(rq, 0, HZ / 5) < 0) {
3013+ struct drm_printer p =
3014+ drm_info_printer(engine->i915->drm.dev);
3015+
3016+ pr_err("Failed to flush chain of %d requests, at %d\n",
3017+ prio, rq_prio(rq) >> I915_USER_PRIORITY_SHIFT);
3018+ intel_engine_dump(engine, &p,
3019+ "%s\n", engine->name);
3020+
3021+ err = -ETIME;
3022+ }
3023+
3024+ i915_vma_put(rq->batch);
3025+ i915_request_put(rq);
3026+ rq = n;
3027+ }
3028+
3029+ if (igt_live_test_end(&t))
3030+ err = -EIO;
3031+ if (err)
3032+ return err;
3033+ }
3034+
3035+ return 0;
3036+}
3037+
3038+static struct i915_vma *
3039+create_gpr_user(struct intel_engine_cs *engine,
3040+ struct i915_vma *result,
3041+ unsigned int offset)
3042+{
3043+ struct drm_i915_gem_object *obj;
3044+ struct i915_vma *vma;
3045+ u32 *cs;
3046+ int err;
3047+ int i;
3048+
3049+ obj = i915_gem_object_create_internal(engine->i915, 4096);
3050+ if (IS_ERR(obj))
3051+ return ERR_CAST(obj);
3052+
3053+ vma = i915_vma_instance(obj, result->vm, NULL);
3054+ if (IS_ERR(vma)) {
3055+ i915_gem_object_put(obj);
3056+ return vma;
3057+ }
3058+
3059+ err = i915_vma_pin(vma, 0, 0, PIN_USER);
3060+ if (err) {
3061+ i915_vma_put(vma);
3062+ return ERR_PTR(err);
3063+ }
3064+
3065+ cs = i915_gem_object_pin_map(obj, I915_MAP_WC);
3066+ if (IS_ERR(cs)) {
3067+ i915_vma_put(vma);
3068+ return ERR_CAST(cs);
3069+ }
3070+
3071+ /* All GPR are clear for new contexts. We use GPR(0) as a constant */
3072+ *cs++ = MI_LOAD_REGISTER_IMM(1);
3073+ *cs++ = CS_GPR(engine, 0);
3074+ *cs++ = 1;
3075+
3076+ for (i = 1; i < NUM_GPR; i++) {
3077+ u64 addr;
3078+
3079+ /*
3080+ * Perform: GPR[i]++
3081+ *
3082+ * As we read and write into the context saved GPR[i], if
3083+ * we restart this batch buffer from an earlier point, we
3084+ * will repeat the increment and store a value > 1.
3085+ */
3086+ *cs++ = MI_MATH(4);
3087+ *cs++ = MI_MATH_LOAD(MI_MATH_REG_SRCA, MI_MATH_REG(i));
3088+ *cs++ = MI_MATH_LOAD(MI_MATH_REG_SRCB, MI_MATH_REG(0));
3089+ *cs++ = MI_MATH_ADD;
3090+ *cs++ = MI_MATH_STORE(MI_MATH_REG(i), MI_MATH_REG_ACCU);
3091+
3092+ addr = result->node.start + offset + i * sizeof(*cs);
3093+ *cs++ = MI_STORE_REGISTER_MEM_GEN8;
3094+ *cs++ = CS_GPR(engine, 2 * i);
3095+ *cs++ = lower_32_bits(addr);
3096+ *cs++ = upper_32_bits(addr);
3097+
3098+ *cs++ = MI_SEMAPHORE_WAIT |
3099+ MI_SEMAPHORE_POLL |
3100+ MI_SEMAPHORE_SAD_GTE_SDD;
3101+ *cs++ = i;
3102+ *cs++ = lower_32_bits(result->node.start);
3103+ *cs++ = upper_32_bits(result->node.start);
3104+ }
3105+
3106+ *cs++ = MI_BATCH_BUFFER_END;
3107+ i915_gem_object_flush_map(obj);
3108+ i915_gem_object_unpin_map(obj);
3109+
3110+ return vma;
3111+}
3112+
3113+static struct i915_vma *create_global(struct intel_gt *gt, size_t sz)
3114+{
3115+ struct drm_i915_gem_object *obj;
3116+ struct i915_vma *vma;
3117+ int err;
3118+
3119+ obj = i915_gem_object_create_internal(gt->i915, sz);
3120+ if (IS_ERR(obj))
3121+ return ERR_CAST(obj);
3122+
3123+ vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
3124+ if (IS_ERR(vma)) {
3125+ i915_gem_object_put(obj);
3126+ return vma;
3127+ }
3128+
3129+ err = i915_ggtt_pin(vma, NULL, 0, 0);
3130+ if (err) {
3131+ i915_vma_put(vma);
3132+ return ERR_PTR(err);
3133+ }
3134+
3135+ return vma;
3136+}
3137+
3138+static struct i915_request *
3139+create_gpr_client(struct intel_engine_cs *engine,
3140+ struct i915_vma *global,
3141+ unsigned int offset)
3142+{
3143+ struct i915_vma *batch, *vma;
3144+ struct intel_context *ce;
3145+ struct i915_request *rq;
3146+ int err;
3147+
3148+ ce = intel_context_create(engine);
3149+ if (IS_ERR(ce))
3150+ return ERR_CAST(ce);
3151+
3152+ vma = i915_vma_instance(global->obj, ce->vm, NULL);
3153+ if (IS_ERR(vma)) {
3154+ err = PTR_ERR(vma);
3155+ goto out_ce;
3156+ }
3157+
3158+ err = i915_vma_pin(vma, 0, 0, PIN_USER);
3159+ if (err)
3160+ goto out_ce;
3161+
3162+ batch = create_gpr_user(engine, vma, offset);
3163+ if (IS_ERR(batch)) {
3164+ err = PTR_ERR(batch);
3165+ goto out_vma;
3166+ }
3167+
3168+ rq = intel_context_create_request(ce);
3169+ if (IS_ERR(rq)) {
3170+ err = PTR_ERR(rq);
3171+ goto out_batch;
3172+ }
3173+
3174+ i915_vma_lock(vma);
3175+ err = i915_request_await_object(rq, vma->obj, false);
3176+ if (!err)
3177+ err = i915_vma_move_to_active(vma, rq, 0);
3178+ i915_vma_unlock(vma);
3179+
3180+ i915_vma_lock(batch);
3181+ if (!err)
3182+ err = i915_request_await_object(rq, batch->obj, false);
3183+ if (!err)
3184+ err = i915_vma_move_to_active(batch, rq, 0);
3185+ if (!err)
3186+ err = rq->engine->emit_bb_start(rq,
3187+ batch->node.start,
3188+ PAGE_SIZE, 0);
3189+ i915_vma_unlock(batch);
3190+ i915_vma_unpin(batch);
3191+
3192+ if (!err)
3193+ i915_request_get(rq);
3194+ i915_request_add(rq);
3195+
3196+out_batch:
3197+ i915_vma_put(batch);
3198+out_vma:
3199+ i915_vma_unpin(vma);
3200+out_ce:
3201+ intel_context_put(ce);
3202+ return err ? ERR_PTR(err) : rq;
3203+}
3204+
3205+static int preempt_user(struct intel_engine_cs *engine,
3206+ struct i915_vma *global,
3207+ int id)
3208+{
3209+ struct i915_sched_attr attr = {
3210+ .priority = I915_PRIORITY_MAX
3211+ };
3212+ struct i915_request *rq;
3213+ int err = 0;
3214+ u32 *cs;
3215+
3216+ rq = intel_engine_create_kernel_request(engine);
3217+ if (IS_ERR(rq))
3218+ return PTR_ERR(rq);
3219+
3220+ cs = intel_ring_begin(rq, 4);
3221+ if (IS_ERR(cs)) {
3222+ i915_request_add(rq);
3223+ return PTR_ERR(cs);
3224+ }
3225+
3226+ *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
3227+ *cs++ = i915_ggtt_offset(global);
3228+ *cs++ = 0;
3229+ *cs++ = id;
3230+
3231+ intel_ring_advance(rq, cs);
3232+
3233+ i915_request_get(rq);
3234+ i915_request_add(rq);
3235+
3236+ engine->schedule(rq, &attr);
3237+
3238+ if (i915_request_wait(rq, 0, HZ / 2) < 0)
3239+ err = -ETIME;
3240+ i915_request_put(rq);
3241+
3242+ return err;
3243+}
3244+
3245+static int live_preempt_user(void *arg)
3246+{
3247+ struct intel_gt *gt = arg;
3248+ struct intel_engine_cs *engine;
3249+ struct i915_vma *global;
3250+ enum intel_engine_id id;
3251+ u32 *result;
3252+ int err = 0;
3253+
3254+ /*
3255+ * In our other tests, we look at preemption in carefully
3256+ * controlled conditions in the ringbuffer. Since most of the
3257+ * time is spent in user batches, most of our preemptions naturally
3258+ * occur there. We want to verify that when we preempt inside a batch
3259+ * we continue on from the current instruction and do not roll back
3260+ * to the start, or another earlier arbitration point.
3261+ *
3262+ * To verify this, we create a batch which is a mixture of
3263+ * MI_MATH (gpr++) MI_SRM (gpr) and preemption points. Then with
3264+ * a few preempting contexts thrown into the mix, we look for any
3265+ * repeated instructions (which show up as incorrect values).
3266+ */
3267+
3268+ global = create_global(gt, 4096);
3269+ if (IS_ERR(global))
3270+ return PTR_ERR(global);
3271+
3272+ result = i915_gem_object_pin_map(global->obj, I915_MAP_WC);
3273+ if (IS_ERR(result)) {
3274+ i915_vma_unpin_and_release(&global, 0);
3275+ return PTR_ERR(result);
3276+ }
3277+
3278+ for_each_engine(engine, gt, id) {
3279+ struct i915_request *client[3] = {};
3280+ struct igt_live_test t;
3281+ int i;
3282+
3283+ if (!intel_engine_has_preemption(engine))
3284+ continue;
3285+
3286+ if (IS_GEN(gt->i915, 8) && engine->class != RENDER_CLASS)
3287+ continue; /* we need per-context GPR */
3288+
3289+ if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
3290+ err = -EIO;
3291+ break;
3292+ }
3293+
3294+ memset(result, 0, 4096);
3295+
3296+ for (i = 0; i < ARRAY_SIZE(client); i++) {
3297+ struct i915_request *rq;
3298+
3299+ rq = create_gpr_client(engine, global,
3300+ NUM_GPR * i * sizeof(u32));
3301+ if (IS_ERR(rq)) {
3302+ err = PTR_ERR(rq);
3303+ goto end_test;
3304+ }
3305+
3306+ client[i] = rq;
3307+ }
3308+
3309+ /* Continuously preempt the set of 3 running contexts */
3310+ for (i = 1; i <= NUM_GPR; i++) {
3311+ err = preempt_user(engine, global, i);
3312+ if (err)
3313+ goto end_test;
3314+ }
3315+
3316+ if (READ_ONCE(result[0]) != NUM_GPR) {
3317+ pr_err("%s: Failed to release semaphore\n",
3318+ engine->name);
3319+ err = -EIO;
3320+ goto end_test;
3321+ }
3322+
3323+ for (i = 0; i < ARRAY_SIZE(client); i++) {
3324+ int gpr;
3325+
3326+ if (i915_request_wait(client[i], 0, HZ / 2) < 0) {
3327+ err = -ETIME;
3328+ goto end_test;
3329+ }
3330+
3331+ for (gpr = 1; gpr < NUM_GPR; gpr++) {
3332+ if (result[NUM_GPR * i + gpr] != 1) {
3333+ pr_err("%s: Invalid result, client %d, gpr %d, result: %d\n",
3334+ engine->name,
3335+ i, gpr, result[NUM_GPR * i + gpr]);
3336+ err = -EINVAL;
3337+ goto end_test;
3338+ }
3339+ }
3340+ }
3341+
3342+end_test:
3343+ for (i = 0; i < ARRAY_SIZE(client); i++) {
3344+ if (!client[i])
3345+ break;
3346+
3347+ i915_request_put(client[i]);
3348+ }
3349+
3350+ /* Flush the semaphores on error */
3351+ smp_store_mb(result[0], -1);
3352+ if (igt_live_test_end(&t))
3353+ err = -EIO;
3354+ if (err)
3355+ break;
3356+ }
3357+
3358+ i915_vma_unpin_and_release(&global, I915_VMA_RELEASE_MAP);
3359+ return err;
3360+}
3361+
3362+static int live_preempt_timeout(void *arg)
3363+{
3364+ struct intel_gt *gt = arg;
3365+ struct i915_gem_context *ctx_hi, *ctx_lo;
3366+ struct igt_spinner spin_lo;
3367+ struct intel_engine_cs *engine;
3368+ enum intel_engine_id id;
3369+ int err = -ENOMEM;
3370+
3371+ /*
3372+ * Check that we force preemption to occur by cancelling the previous
3373+ * context if it refuses to yield the GPU.
3374+ */
3375+ if (!IS_ACTIVE(CONFIG_DRM_I915_PREEMPT_TIMEOUT))
3376+ return 0;
3377+
3378+ if (!intel_has_reset_engine(gt))
3379+ return 0;
3380+
3381+ if (igt_spinner_init(&spin_lo, gt))
3382+ return -ENOMEM;
3383+
3384+ ctx_hi = kernel_context(gt->i915);
3385+ if (!ctx_hi)
3386+ goto err_spin_lo;
3387+ ctx_hi->sched.priority =
3388+ I915_USER_PRIORITY(I915_CONTEXT_MAX_USER_PRIORITY);
3389+
3390+ ctx_lo = kernel_context(gt->i915);
3391+ if (!ctx_lo)
3392+ goto err_ctx_hi;
3393+ ctx_lo->sched.priority =
3394+ I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
3395+
3396+ for_each_engine(engine, gt, id) {
3397+ unsigned long saved_timeout;
3398+ struct i915_request *rq;
3399+
3400+ if (!intel_engine_has_preemption(engine))
3401+ continue;
3402+
3403+ rq = spinner_create_request(&spin_lo, ctx_lo, engine,
3404+ MI_NOOP); /* preemption disabled */
3405+ if (IS_ERR(rq)) {
3406+ err = PTR_ERR(rq);
3407+ goto err_ctx_lo;
3408+ }
3409+
3410+ i915_request_add(rq);
3411+ if (!igt_wait_for_spinner(&spin_lo, rq)) {
3412+ intel_gt_set_wedged(gt);
3413+ err = -EIO;
3414+ goto err_ctx_lo;
3415+ }
3416+
3417+ rq = igt_request_alloc(ctx_hi, engine);
3418+ if (IS_ERR(rq)) {
3419+ igt_spinner_end(&spin_lo);
3420+ err = PTR_ERR(rq);
3421+ goto err_ctx_lo;
3422+ }
3423+
3424+ /* Flush the previous CS ack before changing timeouts */
3425+ while (READ_ONCE(engine->execlists.pending[0]))
3426+ cpu_relax();
3427+
3428+ saved_timeout = engine->props.preempt_timeout_ms;
3429+ engine->props.preempt_timeout_ms = 1; /* in ms, -> 1 jiffie */
3430+
3431+ i915_request_get(rq);
3432+ i915_request_add(rq);
3433+
3434+ intel_engine_flush_submission(engine);
3435+ engine->props.preempt_timeout_ms = saved_timeout;
3436+
3437+ if (i915_request_wait(rq, 0, HZ / 10) < 0) {
3438+ intel_gt_set_wedged(gt);
3439+ i915_request_put(rq);
3440+ err = -ETIME;
3441+ goto err_ctx_lo;
3442+ }
3443+
3444+ igt_spinner_end(&spin_lo);
3445+ i915_request_put(rq);
3446+ }
3447+
3448+ err = 0;
3449+err_ctx_lo:
3450+ kernel_context_close(ctx_lo);
3451+err_ctx_hi:
3452+ kernel_context_close(ctx_hi);
3453+err_spin_lo:
3454+ igt_spinner_fini(&spin_lo);
3455+ return err;
3456+}
3457+
3458+static int random_range(struct rnd_state *rnd, int min, int max)
3459+{
3460+ return i915_prandom_u32_max_state(max - min, rnd) + min;
3461+}
3462+
3463+static int random_priority(struct rnd_state *rnd)
3464+{
3465+ return random_range(rnd, I915_PRIORITY_MIN, I915_PRIORITY_MAX);
3466+}
3467+
3468+struct preempt_smoke {
3469+ struct intel_gt *gt;
3470+ struct i915_gem_context **contexts;
3471+ struct intel_engine_cs *engine;
3472+ struct drm_i915_gem_object *batch;
3473+ unsigned int ncontext;
3474+ struct rnd_state prng;
3475+ unsigned long count;
3476+};
3477+
3478+static struct i915_gem_context *smoke_context(struct preempt_smoke *smoke)
3479+{
3480+ return smoke->contexts[i915_prandom_u32_max_state(smoke->ncontext,
3481+ &smoke->prng)];
3482+}
3483+
3484+static int smoke_submit(struct preempt_smoke *smoke,
3485+ struct i915_gem_context *ctx, int prio,
3486+ struct drm_i915_gem_object *batch)
3487+{
3488+ struct i915_request *rq;
3489+ struct i915_vma *vma = NULL;
3490+ int err = 0;
3491+
3492+ if (batch) {
3493+ struct i915_address_space *vm;
3494+
3495+ vm = i915_gem_context_get_vm_rcu(ctx);
3496+ vma = i915_vma_instance(batch, vm, NULL);
3497+ i915_vm_put(vm);
3498+ if (IS_ERR(vma))
3499+ return PTR_ERR(vma);
3500+
3501+ err = i915_vma_pin(vma, 0, 0, PIN_USER);
3502+ if (err)
3503+ return err;
3504+ }
3505+
3506+ ctx->sched.priority = prio;
3507+
3508+ rq = igt_request_alloc(ctx, smoke->engine);
3509+ if (IS_ERR(rq)) {
3510+ err = PTR_ERR(rq);
3511+ goto unpin;
3512+ }
3513+
3514+ if (vma) {
3515+ i915_vma_lock(vma);
3516+ err = i915_request_await_object(rq, vma->obj, false);
3517+ if (!err)
3518+ err = i915_vma_move_to_active(vma, rq, 0);
3519+ if (!err)
3520+ err = rq->engine->emit_bb_start(rq,
3521+ vma->node.start,
3522+ PAGE_SIZE, 0);
3523+ i915_vma_unlock(vma);
3524+ }
3525+
3526+ i915_request_add(rq);
3527+
3528+unpin:
3529+ if (vma)
3530+ i915_vma_unpin(vma);
3531+
3532+ return err;
3533+}
3534+
3535+static int smoke_crescendo_thread(void *arg)
3536+{
3537+ struct preempt_smoke *smoke = arg;
3538+ IGT_TIMEOUT(end_time);
3539+ unsigned long count;
3540+
3541+ count = 0;
3542+ do {
3543+ struct i915_gem_context *ctx = smoke_context(smoke);
3544+ int err;
3545+
3546+ err = smoke_submit(smoke,
3547+ ctx, count % I915_PRIORITY_MAX,
3548+ smoke->batch);
3549+ if (err)
3550+ return err;
3551+
3552+ count++;
3553+ } while (count < smoke->ncontext && !__igt_timeout(end_time, NULL));
3554+
3555+ smoke->count = count;
3556+ return 0;
3557+}
3558+
3559+static int smoke_crescendo(struct preempt_smoke *smoke, unsigned int flags)
3560+#define BATCH BIT(0)
3561+{
3562+ struct task_struct *tsk[I915_NUM_ENGINES] = {};
3563+ struct preempt_smoke arg[I915_NUM_ENGINES];
3564+ struct intel_engine_cs *engine;
3565+ enum intel_engine_id id;
3566+ unsigned long count;
3567+ int err = 0;
3568+
3569+ for_each_engine(engine, smoke->gt, id) {
3570+ arg[id] = *smoke;
3571+ arg[id].engine = engine;
3572+ if (!(flags & BATCH))
3573+ arg[id].batch = NULL;
3574+ arg[id].count = 0;
3575+
3576+ tsk[id] = kthread_run(smoke_crescendo_thread, &arg,
3577+ "igt/smoke:%d", id);
3578+ if (IS_ERR(tsk[id])) {
3579+ err = PTR_ERR(tsk[id]);
3580+ break;
3581+ }
3582+ get_task_struct(tsk[id]);
3583+ }
3584+
3585+ yield(); /* start all threads before we kthread_stop() */
3586+
3587+ count = 0;
3588+ for_each_engine(engine, smoke->gt, id) {
3589+ int status;
3590+
3591+ if (IS_ERR_OR_NULL(tsk[id]))
3592+ continue;
3593+
3594+ status = kthread_stop(tsk[id]);
3595+ if (status && !err)
3596+ err = status;
3597+
3598+ count += arg[id].count;
3599+
3600+ put_task_struct(tsk[id]);
3601+ }
3602+
3603+ pr_info("Submitted %lu crescendo:%x requests across %d engines and %d contexts\n",
3604+ count, flags, smoke->gt->info.num_engines, smoke->ncontext);
3605+ return 0;
3606+}
3607+
3608+static int smoke_random(struct preempt_smoke *smoke, unsigned int flags)
3609+{
3610+ enum intel_engine_id id;
3611+ IGT_TIMEOUT(end_time);
3612+ unsigned long count;
3613+
3614+ count = 0;
3615+ do {
3616+ for_each_engine(smoke->engine, smoke->gt, id) {
3617+ struct i915_gem_context *ctx = smoke_context(smoke);
3618+ int err;
3619+
3620+ err = smoke_submit(smoke,
3621+ ctx, random_priority(&smoke->prng),
3622+ flags & BATCH ? smoke->batch : NULL);
3623+ if (err)
3624+ return err;
3625+
3626+ count++;
3627+ }
3628+ } while (count < smoke->ncontext && !__igt_timeout(end_time, NULL));
3629+
3630+ pr_info("Submitted %lu random:%x requests across %d engines and %d contexts\n",
3631+ count, flags, smoke->gt->info.num_engines, smoke->ncontext);
3632+ return 0;
3633+}
3634+
3635+static int live_preempt_smoke(void *arg)
3636+{
3637+ struct preempt_smoke smoke = {
3638+ .gt = arg,
3639+ .prng = I915_RND_STATE_INITIALIZER(i915_selftest.random_seed),
3640+ .ncontext = 256,
3641+ };
3642+ const unsigned int phase[] = { 0, BATCH };
3643+ struct igt_live_test t;
3644+ int err = -ENOMEM;
3645+ u32 *cs;
3646+ int n;
3647+
3648+ smoke.contexts = kmalloc_array(smoke.ncontext,
3649+ sizeof(*smoke.contexts),
3650+ GFP_KERNEL);
3651+ if (!smoke.contexts)
3652+ return -ENOMEM;
3653+
3654+ smoke.batch =
3655+ i915_gem_object_create_internal(smoke.gt->i915, PAGE_SIZE);
3656+ if (IS_ERR(smoke.batch)) {
3657+ err = PTR_ERR(smoke.batch);
3658+ goto err_free;
3659+ }
3660+
3661+ cs = i915_gem_object_pin_map(smoke.batch, I915_MAP_WB);
3662+ if (IS_ERR(cs)) {
3663+ err = PTR_ERR(cs);
3664+ goto err_batch;
3665+ }
3666+ for (n = 0; n < PAGE_SIZE / sizeof(*cs) - 1; n++)
3667+ cs[n] = MI_ARB_CHECK;
3668+ cs[n] = MI_BATCH_BUFFER_END;
3669+ i915_gem_object_flush_map(smoke.batch);
3670+ i915_gem_object_unpin_map(smoke.batch);
3671+
3672+ if (igt_live_test_begin(&t, smoke.gt->i915, __func__, "all")) {
3673+ err = -EIO;
3674+ goto err_batch;
3675+ }
3676+
3677+ for (n = 0; n < smoke.ncontext; n++) {
3678+ smoke.contexts[n] = kernel_context(smoke.gt->i915);
3679+ if (!smoke.contexts[n])
3680+ goto err_ctx;
3681+ }
3682+
3683+ for (n = 0; n < ARRAY_SIZE(phase); n++) {
3684+ err = smoke_crescendo(&smoke, phase[n]);
3685+ if (err)
3686+ goto err_ctx;
3687+
3688+ err = smoke_random(&smoke, phase[n]);
3689+ if (err)
3690+ goto err_ctx;
3691+ }
3692+
3693+err_ctx:
3694+ if (igt_live_test_end(&t))
3695+ err = -EIO;
3696+
3697+ for (n = 0; n < smoke.ncontext; n++) {
3698+ if (!smoke.contexts[n])
3699+ break;
3700+ kernel_context_close(smoke.contexts[n]);
3701+ }
3702+
3703+err_batch:
3704+ i915_gem_object_put(smoke.batch);
3705+err_free:
3706+ kfree(smoke.contexts);
3707+
3708+ return err;
3709+}
3710+
3711+static int nop_virtual_engine(struct intel_gt *gt,
3712+ struct intel_engine_cs **siblings,
3713+ unsigned int nsibling,
3714+ unsigned int nctx,
3715+ unsigned int flags)
3716+#define CHAIN BIT(0)
3717+{
3718+ IGT_TIMEOUT(end_time);
3719+ struct i915_request *request[16] = {};
3720+ struct intel_context *ve[16];
3721+ unsigned long n, prime, nc;
3722+ struct igt_live_test t;
3723+ ktime_t times[2] = {};
3724+ int err;
3725+
3726+ GEM_BUG_ON(!nctx || nctx > ARRAY_SIZE(ve));
3727+
3728+ for (n = 0; n < nctx; n++) {
3729+ ve[n] = intel_execlists_create_virtual(siblings, nsibling);
3730+ if (IS_ERR(ve[n])) {
3731+ err = PTR_ERR(ve[n]);
3732+ nctx = n;
3733+ goto out;
3734+ }
3735+
3736+ err = intel_context_pin(ve[n]);
3737+ if (err) {
3738+ intel_context_put(ve[n]);
3739+ nctx = n;
3740+ goto out;
3741+ }
3742+ }
3743+
3744+ err = igt_live_test_begin(&t, gt->i915, __func__, ve[0]->engine->name);
3745+ if (err)
3746+ goto out;
3747+
3748+ for_each_prime_number_from(prime, 1, 8192) {
3749+ times[1] = ktime_get_raw();
3750+
3751+ if (flags & CHAIN) {
3752+ for (nc = 0; nc < nctx; nc++) {
3753+ for (n = 0; n < prime; n++) {
3754+ struct i915_request *rq;
3755+
3756+ rq = i915_request_create(ve[nc]);
3757+ if (IS_ERR(rq)) {
3758+ err = PTR_ERR(rq);
3759+ goto out;
3760+ }
3761+
3762+ if (request[nc])
3763+ i915_request_put(request[nc]);
3764+ request[nc] = i915_request_get(rq);
3765+ i915_request_add(rq);
3766+ }
3767+ }
3768+ } else {
3769+ for (n = 0; n < prime; n++) {
3770+ for (nc = 0; nc < nctx; nc++) {
3771+ struct i915_request *rq;
3772+
3773+ rq = i915_request_create(ve[nc]);
3774+ if (IS_ERR(rq)) {
3775+ err = PTR_ERR(rq);
3776+ goto out;
3777+ }
3778+
3779+ if (request[nc])
3780+ i915_request_put(request[nc]);
3781+ request[nc] = i915_request_get(rq);
3782+ i915_request_add(rq);
3783+ }
3784+ }
3785+ }
3786+
3787+ for (nc = 0; nc < nctx; nc++) {
3788+ if (i915_request_wait(request[nc], 0, HZ / 10) < 0) {
3789+ pr_err("%s(%s): wait for %llx:%lld timed out\n",
3790+ __func__, ve[0]->engine->name,
3791+ request[nc]->fence.context,
3792+ request[nc]->fence.seqno);
3793+
3794+ GEM_TRACE("%s(%s) failed at request %llx:%lld\n",
3795+ __func__, ve[0]->engine->name,
3796+ request[nc]->fence.context,
3797+ request[nc]->fence.seqno);
3798+ GEM_TRACE_DUMP();
3799+ intel_gt_set_wedged(gt);
3800+ break;
3801+ }
3802+ }
3803+
3804+ times[1] = ktime_sub(ktime_get_raw(), times[1]);
3805+ if (prime == 1)
3806+ times[0] = times[1];
3807+
3808+ for (nc = 0; nc < nctx; nc++) {
3809+ i915_request_put(request[nc]);
3810+ request[nc] = NULL;
3811+ }
3812+
3813+ if (__igt_timeout(end_time, NULL))
3814+ break;
3815+ }
3816+
3817+ err = igt_live_test_end(&t);
3818+ if (err)
3819+ goto out;
3820+
3821+ pr_info("Requestx%d latencies on %s: 1 = %lluns, %lu = %lluns\n",
3822+ nctx, ve[0]->engine->name, ktime_to_ns(times[0]),
3823+ prime, div64_u64(ktime_to_ns(times[1]), prime));
3824+
3825+out:
3826+ if (igt_flush_test(gt->i915))
3827+ err = -EIO;
3828+
3829+ for (nc = 0; nc < nctx; nc++) {
3830+ i915_request_put(request[nc]);
3831+ intel_context_unpin(ve[nc]);
3832+ intel_context_put(ve[nc]);
3833+ }
3834+ return err;
3835+}
3836+
3837+static unsigned int
3838+__select_siblings(struct intel_gt *gt,
3839+ unsigned int class,
3840+ struct intel_engine_cs **siblings,
3841+ bool (*filter)(const struct intel_engine_cs *))
3842+{
3843+ unsigned int n = 0;
3844+ unsigned int inst;
3845+
3846+ for (inst = 0; inst <= MAX_ENGINE_INSTANCE; inst++) {
3847+ if (!gt->engine_class[class][inst])
3848+ continue;
3849+
3850+ if (filter && !filter(gt->engine_class[class][inst]))
3851+ continue;
3852+
3853+ siblings[n++] = gt->engine_class[class][inst];
3854+ }
3855+
3856+ return n;
3857+}
3858+
3859+static unsigned int
3860+select_siblings(struct intel_gt *gt,
3861+ unsigned int class,
3862+ struct intel_engine_cs **siblings)
3863+{
3864+ return __select_siblings(gt, class, siblings, NULL);
3865+}
3866+
3867+static int live_virtual_engine(void *arg)
3868+{
3869+ struct intel_gt *gt = arg;
3870+ struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
3871+ struct intel_engine_cs *engine;
3872+ enum intel_engine_id id;
3873+ unsigned int class;
3874+ int err;
3875+
3876+ if (intel_uc_uses_guc_submission(&gt->uc))
3877+ return 0;
3878+
3879+ for_each_engine(engine, gt, id) {
3880+ err = nop_virtual_engine(gt, &engine, 1, 1, 0);
3881+ if (err) {
3882+ pr_err("Failed to wrap engine %s: err=%d\n",
3883+ engine->name, err);
3884+ return err;
3885+ }
3886+ }
3887+
3888+ for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
3889+ int nsibling, n;
3890+
3891+ nsibling = select_siblings(gt, class, siblings);
3892+ if (nsibling < 2)
3893+ continue;
3894+
3895+ for (n = 1; n <= nsibling + 1; n++) {
3896+ err = nop_virtual_engine(gt, siblings, nsibling,
3897+ n, 0);
3898+ if (err)
3899+ return err;
3900+ }
3901+
3902+ err = nop_virtual_engine(gt, siblings, nsibling, n, CHAIN);
3903+ if (err)
3904+ return err;
3905+ }
3906+
3907+ return 0;
3908+}
3909+
3910+static int mask_virtual_engine(struct intel_gt *gt,
3911+ struct intel_engine_cs **siblings,
3912+ unsigned int nsibling)
3913+{
3914+ struct i915_request *request[MAX_ENGINE_INSTANCE + 1];
3915+ struct intel_context *ve;
3916+ struct igt_live_test t;
3917+ unsigned int n;
3918+ int err;
3919+
3920+ /*
3921+ * Check that by setting the execution mask on a request, we can
3922+ * restrict it to our desired engine within the virtual engine.
3923+ */
3924+
3925+ ve = intel_execlists_create_virtual(siblings, nsibling);
3926+ if (IS_ERR(ve)) {
3927+ err = PTR_ERR(ve);
3928+ goto out_close;
3929+ }
3930+
3931+ err = intel_context_pin(ve);
3932+ if (err)
3933+ goto out_put;
3934+
3935+ err = igt_live_test_begin(&t, gt->i915, __func__, ve->engine->name);
3936+ if (err)
3937+ goto out_unpin;
3938+
3939+ for (n = 0; n < nsibling; n++) {
3940+ request[n] = i915_request_create(ve);
3941+ if (IS_ERR(request[n])) {
3942+ err = PTR_ERR(request[n]);
3943+ nsibling = n;
3944+ goto out;
3945+ }
3946+
3947+ /* Reverse order as it's more likely to be unnatural */
3948+ request[n]->execution_mask = siblings[nsibling - n - 1]->mask;
3949+
3950+ i915_request_get(request[n]);
3951+ i915_request_add(request[n]);
3952+ }
3953+
3954+ for (n = 0; n < nsibling; n++) {
3955+ if (i915_request_wait(request[n], 0, HZ / 10) < 0) {
3956+ pr_err("%s(%s): wait for %llx:%lld timed out\n",
3957+ __func__, ve->engine->name,
3958+ request[n]->fence.context,
3959+ request[n]->fence.seqno);
3960+
3961+ GEM_TRACE("%s(%s) failed at request %llx:%lld\n",
3962+ __func__, ve->engine->name,
3963+ request[n]->fence.context,
3964+ request[n]->fence.seqno);
3965+ GEM_TRACE_DUMP();
3966+ intel_gt_set_wedged(gt);
3967+ err = -EIO;
3968+ goto out;
3969+ }
3970+
3971+ if (request[n]->engine != siblings[nsibling - n - 1]) {
3972+ pr_err("Executed on wrong sibling '%s', expected '%s'\n",
3973+ request[n]->engine->name,
3974+ siblings[nsibling - n - 1]->name);
3975+ err = -EINVAL;
3976+ goto out;
3977+ }
3978+ }
3979+
3980+ err = igt_live_test_end(&t);
3981+out:
3982+ if (igt_flush_test(gt->i915))
3983+ err = -EIO;
3984+
3985+ for (n = 0; n < nsibling; n++)
3986+ i915_request_put(request[n]);
3987+
3988+out_unpin:
3989+ intel_context_unpin(ve);
3990+out_put:
3991+ intel_context_put(ve);
3992+out_close:
3993+ return err;
3994+}
3995+
3996+static int live_virtual_mask(void *arg)
3997+{
3998+ struct intel_gt *gt = arg;
3999+ struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4000+ unsigned int class;
4001+ int err;
4002+
4003+ if (intel_uc_uses_guc_submission(&gt->uc))
4004+ return 0;
4005+
4006+ for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4007+ unsigned int nsibling;
4008+
4009+ nsibling = select_siblings(gt, class, siblings);
4010+ if (nsibling < 2)
4011+ continue;
4012+
4013+ err = mask_virtual_engine(gt, siblings, nsibling);
4014+ if (err)
4015+ return err;
4016+ }
4017+
4018+ return 0;
4019+}
4020+
4021+static int slicein_virtual_engine(struct intel_gt *gt,
4022+ struct intel_engine_cs **siblings,
4023+ unsigned int nsibling)
4024+{
4025+ const long timeout = slice_timeout(siblings[0]);
4026+ struct intel_context *ce;
4027+ struct i915_request *rq;
4028+ struct igt_spinner spin;
4029+ unsigned int n;
4030+ int err = 0;
4031+
4032+ /*
4033+ * Virtual requests must take part in timeslicing on the target engines.
4034+ */
4035+
4036+ if (igt_spinner_init(&spin, gt))
4037+ return -ENOMEM;
4038+
4039+ for (n = 0; n < nsibling; n++) {
4040+ ce = intel_context_create(siblings[n]);
4041+ if (IS_ERR(ce)) {
4042+ err = PTR_ERR(ce);
4043+ goto out;
4044+ }
4045+
4046+ rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
4047+ intel_context_put(ce);
4048+ if (IS_ERR(rq)) {
4049+ err = PTR_ERR(rq);
4050+ goto out;
4051+ }
4052+
4053+ i915_request_add(rq);
4054+ }
4055+
4056+ ce = intel_execlists_create_virtual(siblings, nsibling);
4057+ if (IS_ERR(ce)) {
4058+ err = PTR_ERR(ce);
4059+ goto out;
4060+ }
4061+
4062+ rq = intel_context_create_request(ce);
4063+ intel_context_put(ce);
4064+ if (IS_ERR(rq)) {
4065+ err = PTR_ERR(rq);
4066+ goto out;
4067+ }
4068+
4069+ i915_request_get(rq);
4070+ i915_request_add(rq);
4071+ if (i915_request_wait(rq, 0, timeout) < 0) {
4072+ GEM_TRACE_ERR("%s(%s) failed to slice in virtual request\n",
4073+ __func__, rq->engine->name);
4074+ GEM_TRACE_DUMP();
4075+ intel_gt_set_wedged(gt);
4076+ err = -EIO;
4077+ }
4078+ i915_request_put(rq);
4079+
4080+out:
4081+ igt_spinner_end(&spin);
4082+ if (igt_flush_test(gt->i915))
4083+ err = -EIO;
4084+ igt_spinner_fini(&spin);
4085+ return err;
4086+}
4087+
4088+static int sliceout_virtual_engine(struct intel_gt *gt,
4089+ struct intel_engine_cs **siblings,
4090+ unsigned int nsibling)
4091+{
4092+ const long timeout = slice_timeout(siblings[0]);
4093+ struct intel_context *ce;
4094+ struct i915_request *rq;
4095+ struct igt_spinner spin;
4096+ unsigned int n;
4097+ int err = 0;
4098+
4099+ /*
4100+ * Virtual requests must allow others a fair timeslice.
4101+ */
4102+
4103+ if (igt_spinner_init(&spin, gt))
4104+ return -ENOMEM;
4105+
4106+ /* XXX We do not handle oversubscription and fairness with normal rq */
4107+ for (n = 0; n < nsibling; n++) {
4108+ ce = intel_execlists_create_virtual(siblings, nsibling);
4109+ if (IS_ERR(ce)) {
4110+ err = PTR_ERR(ce);
4111+ goto out;
4112+ }
4113+
4114+ rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
4115+ intel_context_put(ce);
4116+ if (IS_ERR(rq)) {
4117+ err = PTR_ERR(rq);
4118+ goto out;
4119+ }
4120+
4121+ i915_request_add(rq);
4122+ }
4123+
4124+ for (n = 0; !err && n < nsibling; n++) {
4125+ ce = intel_context_create(siblings[n]);
4126+ if (IS_ERR(ce)) {
4127+ err = PTR_ERR(ce);
4128+ goto out;
4129+ }
4130+
4131+ rq = intel_context_create_request(ce);
4132+ intel_context_put(ce);
4133+ if (IS_ERR(rq)) {
4134+ err = PTR_ERR(rq);
4135+ goto out;
4136+ }
4137+
4138+ i915_request_get(rq);
4139+ i915_request_add(rq);
4140+ if (i915_request_wait(rq, 0, timeout) < 0) {
4141+ GEM_TRACE_ERR("%s(%s) failed to slice out virtual request\n",
4142+ __func__, siblings[n]->name);
4143+ GEM_TRACE_DUMP();
4144+ intel_gt_set_wedged(gt);
4145+ err = -EIO;
4146+ }
4147+ i915_request_put(rq);
4148+ }
4149+
4150+out:
4151+ igt_spinner_end(&spin);
4152+ if (igt_flush_test(gt->i915))
4153+ err = -EIO;
4154+ igt_spinner_fini(&spin);
4155+ return err;
4156+}
4157+
4158+static int live_virtual_slice(void *arg)
4159+{
4160+ struct intel_gt *gt = arg;
4161+ struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4162+ unsigned int class;
4163+ int err;
4164+
4165+ if (intel_uc_uses_guc_submission(&gt->uc))
4166+ return 0;
4167+
4168+ for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4169+ unsigned int nsibling;
4170+
4171+ nsibling = __select_siblings(gt, class, siblings,
4172+ intel_engine_has_timeslices);
4173+ if (nsibling < 2)
4174+ continue;
4175+
4176+ err = slicein_virtual_engine(gt, siblings, nsibling);
4177+ if (err)
4178+ return err;
4179+
4180+ err = sliceout_virtual_engine(gt, siblings, nsibling);
4181+ if (err)
4182+ return err;
4183+ }
4184+
4185+ return 0;
4186+}
4187+
4188+static int preserved_virtual_engine(struct intel_gt *gt,
4189+ struct intel_engine_cs **siblings,
4190+ unsigned int nsibling)
4191+{
4192+ struct i915_request *last = NULL;
4193+ struct intel_context *ve;
4194+ struct i915_vma *scratch;
4195+ struct igt_live_test t;
4196+ unsigned int n;
4197+ int err = 0;
4198+ u32 *cs;
4199+
4200+ scratch = __vm_create_scratch_for_read(&siblings[0]->gt->ggtt->vm,
4201+ PAGE_SIZE);
4202+ if (IS_ERR(scratch))
4203+ return PTR_ERR(scratch);
4204+
4205+ err = i915_vma_sync(scratch);
4206+ if (err)
4207+ goto out_scratch;
4208+
4209+ ve = intel_execlists_create_virtual(siblings, nsibling);
4210+ if (IS_ERR(ve)) {
4211+ err = PTR_ERR(ve);
4212+ goto out_scratch;
4213+ }
4214+
4215+ err = intel_context_pin(ve);
4216+ if (err)
4217+ goto out_put;
4218+
4219+ err = igt_live_test_begin(&t, gt->i915, __func__, ve->engine->name);
4220+ if (err)
4221+ goto out_unpin;
4222+
4223+ for (n = 0; n < NUM_GPR_DW; n++) {
4224+ struct intel_engine_cs *engine = siblings[n % nsibling];
4225+ struct i915_request *rq;
4226+
4227+ rq = i915_request_create(ve);
4228+ if (IS_ERR(rq)) {
4229+ err = PTR_ERR(rq);
4230+ goto out_end;
4231+ }
4232+
4233+ i915_request_put(last);
4234+ last = i915_request_get(rq);
4235+
4236+ cs = intel_ring_begin(rq, 8);
4237+ if (IS_ERR(cs)) {
4238+ i915_request_add(rq);
4239+ err = PTR_ERR(cs);
4240+ goto out_end;
4241+ }
4242+
4243+ *cs++ = MI_STORE_REGISTER_MEM_GEN8 | MI_USE_GGTT;
4244+ *cs++ = CS_GPR(engine, n);
4245+ *cs++ = i915_ggtt_offset(scratch) + n * sizeof(u32);
4246+ *cs++ = 0;
4247+
4248+ *cs++ = MI_LOAD_REGISTER_IMM(1);
4249+ *cs++ = CS_GPR(engine, (n + 1) % NUM_GPR_DW);
4250+ *cs++ = n + 1;
4251+
4252+ *cs++ = MI_NOOP;
4253+ intel_ring_advance(rq, cs);
4254+
4255+ /* Restrict this request to run on a particular engine */
4256+ rq->execution_mask = engine->mask;
4257+ i915_request_add(rq);
4258+ }
4259+
4260+ if (i915_request_wait(last, 0, HZ / 5) < 0) {
4261+ err = -ETIME;
4262+ goto out_end;
4263+ }
4264+
4265+ cs = i915_gem_object_pin_map(scratch->obj, I915_MAP_WB);
4266+ if (IS_ERR(cs)) {
4267+ err = PTR_ERR(cs);
4268+ goto out_end;
4269+ }
4270+
4271+ for (n = 0; n < NUM_GPR_DW; n++) {
4272+ if (cs[n] != n) {
4273+ pr_err("Incorrect value[%d] found for GPR[%d]\n",
4274+ cs[n], n);
4275+ err = -EINVAL;
4276+ break;
4277+ }
4278+ }
4279+
4280+ i915_gem_object_unpin_map(scratch->obj);
4281+
4282+out_end:
4283+ if (igt_live_test_end(&t))
4284+ err = -EIO;
4285+ i915_request_put(last);
4286+out_unpin:
4287+ intel_context_unpin(ve);
4288+out_put:
4289+ intel_context_put(ve);
4290+out_scratch:
4291+ i915_vma_unpin_and_release(&scratch, 0);
4292+ return err;
4293+}
4294+
4295+static int live_virtual_preserved(void *arg)
4296+{
4297+ struct intel_gt *gt = arg;
4298+ struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4299+ unsigned int class;
4300+
4301+ /*
4302+ * Check that the context image retains non-privileged (user) registers
4303+ * from one engine to the next. For this we check that the CS_GPR
4304+ * are preserved.
4305+ */
4306+
4307+ if (intel_uc_uses_guc_submission(&gt->uc))
4308+ return 0;
4309+
4310+ /* As we use CS_GPR we cannot run before they existed on all engines. */
4311+ if (INTEL_GEN(gt->i915) < 9)
4312+ return 0;
4313+
4314+ for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4315+ int nsibling, err;
4316+
4317+ nsibling = select_siblings(gt, class, siblings);
4318+ if (nsibling < 2)
4319+ continue;
4320+
4321+ err = preserved_virtual_engine(gt, siblings, nsibling);
4322+ if (err)
4323+ return err;
4324+ }
4325+
4326+ return 0;
4327+}
4328+
4329+static int bond_virtual_engine(struct intel_gt *gt,
4330+ unsigned int class,
4331+ struct intel_engine_cs **siblings,
4332+ unsigned int nsibling,
4333+ unsigned int flags)
4334+#define BOND_SCHEDULE BIT(0)
4335+{
4336+ struct intel_engine_cs *master;
4337+ struct i915_request *rq[16];
4338+ enum intel_engine_id id;
4339+ struct igt_spinner spin;
4340+ unsigned long n;
4341+ int err;
4342+
4343+ /*
4344+ * A set of bonded requests is intended to be run concurrently
4345+ * across a number of engines. We use one request per-engine
4346+ * and a magic fence to schedule each of the bonded requests
4347+ * at the same time. A consequence of our current scheduler is that
4348+ * we only move requests to the HW ready queue when the request
4349+ * becomes ready, that is when all of its prerequisite fences have
4350+ * been signaled. As one of those fences is the master submit fence,
4351+ * there is a delay on all secondary fences as the HW may be
4352+ * currently busy. Equally, as all the requests are independent,
4353+ * they may have other fences that delay individual request
4354+ * submission to HW. Ergo, we do not guarantee that all requests are
4355+ * immediately submitted to HW at the same time, just that if the
4356+ * rules are abided by, they are ready at the same time as the
4357+ * first is submitted. Userspace can embed semaphores in its batch
4358+ * to ensure parallel execution of its phases as it requires.
4359+ * Though naturally it gets requested that perhaps the scheduler should
4360+ * take care of parallel execution, even across preemption events on
4361+ * different HW. (The proper answer is of course "lalalala".)
4362+ *
4363+ * With the submit-fence, we have identified three possible phases
4364+ * of synchronisation depending on the master fence: queued (not
4365+ * ready), executing, and signaled. The first two are quite simple
4366+ * and checked below. However, the signaled master fence handling is
4367+ * contentious. Currently we do not distinguish between a signaled
4368+ * fence and an expired fence, as once signaled it does not convey
4369+ * any information about the previous execution. It may even be freed
4370+ * and hence checking later it may not exist at all. Ergo we currently
4371+ * do not apply the bonding constraint for an already signaled fence,
4372+ * as our expectation is that it should not constrain the secondaries
4373+ * and is outside of the scope of the bonded request API (i.e. all
4374+ * userspace requests are meant to be running in parallel). As
4375+ * it imposes no constraint, and is effectively a no-op, we do not
4376+ * check below as normal execution flows are checked extensively above.
4377+ *
4378+ * XXX Is the degenerate handling of signaled submit fences the
4379+ * expected behaviour for userpace?
4380+ */
4381+
4382+ GEM_BUG_ON(nsibling >= ARRAY_SIZE(rq) - 1);
4383+
4384+ if (igt_spinner_init(&spin, gt))
4385+ return -ENOMEM;
4386+
4387+ err = 0;
4388+ rq[0] = ERR_PTR(-ENOMEM);
4389+ for_each_engine(master, gt, id) {
4390+ struct i915_sw_fence fence = {};
4391+ struct intel_context *ce;
4392+
4393+ if (master->class == class)
4394+ continue;
4395+
4396+ ce = intel_context_create(master);
4397+ if (IS_ERR(ce)) {
4398+ err = PTR_ERR(ce);
4399+ goto out;
4400+ }
4401+
4402+ memset_p((void *)rq, ERR_PTR(-EINVAL), ARRAY_SIZE(rq));
4403+
4404+ rq[0] = igt_spinner_create_request(&spin, ce, MI_NOOP);
4405+ intel_context_put(ce);
4406+ if (IS_ERR(rq[0])) {
4407+ err = PTR_ERR(rq[0]);
4408+ goto out;
4409+ }
4410+ i915_request_get(rq[0]);
4411+
4412+ if (flags & BOND_SCHEDULE) {
4413+ onstack_fence_init(&fence);
4414+ err = i915_sw_fence_await_sw_fence_gfp(&rq[0]->submit,
4415+ &fence,
4416+ GFP_KERNEL);
4417+ }
4418+
4419+ i915_request_add(rq[0]);
4420+ if (err < 0)
4421+ goto out;
4422+
4423+ if (!(flags & BOND_SCHEDULE) &&
4424+ !igt_wait_for_spinner(&spin, rq[0])) {
4425+ err = -EIO;
4426+ goto out;
4427+ }
4428+
4429+ for (n = 0; n < nsibling; n++) {
4430+ struct intel_context *ve;
4431+
4432+ ve = intel_execlists_create_virtual(siblings, nsibling);
4433+ if (IS_ERR(ve)) {
4434+ err = PTR_ERR(ve);
4435+ onstack_fence_fini(&fence);
4436+ goto out;
4437+ }
4438+
4439+ err = intel_virtual_engine_attach_bond(ve->engine,
4440+ master,
4441+ siblings[n]);
4442+ if (err) {
4443+ intel_context_put(ve);
4444+ onstack_fence_fini(&fence);
4445+ goto out;
4446+ }
4447+
4448+ err = intel_context_pin(ve);
4449+ intel_context_put(ve);
4450+ if (err) {
4451+ onstack_fence_fini(&fence);
4452+ goto out;
4453+ }
4454+
4455+ rq[n + 1] = i915_request_create(ve);
4456+ intel_context_unpin(ve);
4457+ if (IS_ERR(rq[n + 1])) {
4458+ err = PTR_ERR(rq[n + 1]);
4459+ onstack_fence_fini(&fence);
4460+ goto out;
4461+ }
4462+ i915_request_get(rq[n + 1]);
4463+
4464+ err = i915_request_await_execution(rq[n + 1],
4465+ &rq[0]->fence,
4466+ ve->engine->bond_execute);
4467+ i915_request_add(rq[n + 1]);
4468+ if (err < 0) {
4469+ onstack_fence_fini(&fence);
4470+ goto out;
4471+ }
4472+ }
4473+ onstack_fence_fini(&fence);
4474+ intel_engine_flush_submission(master);
4475+ igt_spinner_end(&spin);
4476+
4477+ if (i915_request_wait(rq[0], 0, HZ / 10) < 0) {
4478+ pr_err("Master request did not execute (on %s)!\n",
4479+ rq[0]->engine->name);
4480+ err = -EIO;
4481+ goto out;
4482+ }
4483+
4484+ for (n = 0; n < nsibling; n++) {
4485+ if (i915_request_wait(rq[n + 1], 0,
4486+ MAX_SCHEDULE_TIMEOUT) < 0) {
4487+ err = -EIO;
4488+ goto out;
4489+ }
4490+
4491+ if (rq[n + 1]->engine != siblings[n]) {
4492+ pr_err("Bonded request did not execute on target engine: expected %s, used %s; master was %s\n",
4493+ siblings[n]->name,
4494+ rq[n + 1]->engine->name,
4495+ rq[0]->engine->name);
4496+ err = -EINVAL;
4497+ goto out;
4498+ }
4499+ }
4500+
4501+ for (n = 0; !IS_ERR(rq[n]); n++)
4502+ i915_request_put(rq[n]);
4503+ rq[0] = ERR_PTR(-ENOMEM);
4504+ }
4505+
4506+out:
4507+ for (n = 0; !IS_ERR(rq[n]); n++)
4508+ i915_request_put(rq[n]);
4509+ if (igt_flush_test(gt->i915))
4510+ err = -EIO;
4511+
4512+ igt_spinner_fini(&spin);
4513+ return err;
4514+}
4515+
4516+static int live_virtual_bond(void *arg)
4517+{
4518+ static const struct phase {
4519+ const char *name;
4520+ unsigned int flags;
4521+ } phases[] = {
4522+ { "", 0 },
4523+ { "schedule", BOND_SCHEDULE },
4524+ { },
4525+ };
4526+ struct intel_gt *gt = arg;
4527+ struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4528+ unsigned int class;
4529+ int err;
4530+
4531+ if (intel_uc_uses_guc_submission(&gt->uc))
4532+ return 0;
4533+
4534+ for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4535+ const struct phase *p;
4536+ int nsibling;
4537+
4538+ nsibling = select_siblings(gt, class, siblings);
4539+ if (nsibling < 2)
4540+ continue;
4541+
4542+ for (p = phases; p->name; p++) {
4543+ err = bond_virtual_engine(gt,
4544+ class, siblings, nsibling,
4545+ p->flags);
4546+ if (err) {
4547+ pr_err("%s(%s): failed class=%d, nsibling=%d, err=%d\n",
4548+ __func__, p->name, class, nsibling, err);
4549+ return err;
4550+ }
4551+ }
4552+ }
4553+
4554+ return 0;
4555+}
4556+
4557+static int reset_virtual_engine(struct intel_gt *gt,
4558+ struct intel_engine_cs **siblings,
4559+ unsigned int nsibling)
4560+{
4561+ struct intel_engine_cs *engine;
4562+ struct intel_context *ve;
4563+ struct igt_spinner spin;
4564+ struct i915_request *rq;
4565+ unsigned int n;
4566+ int err = 0;
4567+
4568+ /*
4569+ * In order to support offline error capture for fast preempt reset,
4570+ * we need to decouple the guilty request and ensure that it and its
4571+ * descendents are not executed while the capture is in progress.
4572+ */
4573+
4574+ if (igt_spinner_init(&spin, gt))
4575+ return -ENOMEM;
4576+
4577+ ve = intel_execlists_create_virtual(siblings, nsibling);
4578+ if (IS_ERR(ve)) {
4579+ err = PTR_ERR(ve);
4580+ goto out_spin;
4581+ }
4582+
4583+ for (n = 0; n < nsibling; n++)
4584+ st_engine_heartbeat_disable(siblings[n]);
4585+
4586+ rq = igt_spinner_create_request(&spin, ve, MI_ARB_CHECK);
4587+ if (IS_ERR(rq)) {
4588+ err = PTR_ERR(rq);
4589+ goto out_heartbeat;
4590+ }
4591+ i915_request_add(rq);
4592+
4593+ if (!igt_wait_for_spinner(&spin, rq)) {
4594+ intel_gt_set_wedged(gt);
4595+ err = -ETIME;
4596+ goto out_heartbeat;
4597+ }
4598+
4599+ engine = rq->engine;
4600+ GEM_BUG_ON(engine == ve->engine);
4601+
4602+ /* Take ownership of the reset and tasklet */
4603+ local_bh_disable();
4604+ if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
4605+ &gt->reset.flags)) {
4606+ local_bh_enable();
4607+ intel_gt_set_wedged(gt);
4608+ err = -EBUSY;
4609+ goto out_heartbeat;
4610+ }
4611+ tasklet_disable(&engine->execlists.tasklet);
4612+
4613+ engine->execlists.tasklet.func(engine->execlists.tasklet.data);
4614+ GEM_BUG_ON(execlists_active(&engine->execlists) != rq);
4615+
4616+ /* Fake a preemption event; failed of course */
4617+ spin_lock_irq(&engine->active.lock);
4618+ __unwind_incomplete_requests(engine);
4619+ spin_unlock_irq(&engine->active.lock);
4620+ GEM_BUG_ON(rq->engine != engine);
4621+
4622+ /* Reset the engine while keeping our active request on hold */
4623+ execlists_hold(engine, rq);
4624+ GEM_BUG_ON(!i915_request_on_hold(rq));
4625+
4626+ __intel_engine_reset_bh(engine, NULL);
4627+ GEM_BUG_ON(rq->fence.error != -EIO);
4628+
4629+ /* Release our grasp on the engine, letting CS flow again */
4630+ tasklet_enable(&engine->execlists.tasklet);
4631+ clear_and_wake_up_bit(I915_RESET_ENGINE + engine->id, &gt->reset.flags);
4632+ local_bh_enable();
4633+
4634+ /* Check that we do not resubmit the held request */
4635+ i915_request_get(rq);
4636+ if (!i915_request_wait(rq, 0, HZ / 5)) {
4637+ pr_err("%s: on hold request completed!\n",
4638+ engine->name);
4639+ intel_gt_set_wedged(gt);
4640+ err = -EIO;
4641+ goto out_rq;
4642+ }
4643+ GEM_BUG_ON(!i915_request_on_hold(rq));
4644+
4645+ /* But is resubmitted on release */
4646+ execlists_unhold(engine, rq);
4647+ if (i915_request_wait(rq, 0, HZ / 5) < 0) {
4648+ pr_err("%s: held request did not complete!\n",
4649+ engine->name);
4650+ intel_gt_set_wedged(gt);
4651+ err = -ETIME;
4652+ }
4653+
4654+out_rq:
4655+ i915_request_put(rq);
4656+out_heartbeat:
4657+ for (n = 0; n < nsibling; n++)
4658+ st_engine_heartbeat_enable(siblings[n]);
4659+
4660+ intel_context_put(ve);
4661+out_spin:
4662+ igt_spinner_fini(&spin);
4663+ return err;
4664+}
4665+
4666+static int live_virtual_reset(void *arg)
4667+{
4668+ struct intel_gt *gt = arg;
4669+ struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4670+ unsigned int class;
4671+
4672+ /*
4673+ * Check that we handle a reset event within a virtual engine.
4674+ * Only the physical engine is reset, but we have to check the flow
4675+ * of the virtual requests around the reset, and make sure it is not
4676+ * forgotten.
4677+ */
4678+
4679+ if (intel_uc_uses_guc_submission(&gt->uc))
4680+ return 0;
4681+
4682+ if (!intel_has_reset_engine(gt))
4683+ return 0;
4684+
4685+ for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4686+ int nsibling, err;
4687+
4688+ nsibling = select_siblings(gt, class, siblings);
4689+ if (nsibling < 2)
4690+ continue;
4691+
4692+ err = reset_virtual_engine(gt, siblings, nsibling);
4693+ if (err)
4694+ return err;
4695+ }
4696+
4697+ return 0;
4698+}
4699+
4700+int intel_execlists_live_selftests(struct drm_i915_private *i915)
4701+{
4702+ static const struct i915_subtest tests[] = {
4703+ SUBTEST(live_sanitycheck),
4704+ SUBTEST(live_unlite_switch),
4705+ SUBTEST(live_unlite_preempt),
4706+ SUBTEST(live_unlite_ring),
4707+ SUBTEST(live_pin_rewind),
4708+ SUBTEST(live_hold_reset),
4709+ SUBTEST(live_error_interrupt),
4710+ SUBTEST(live_timeslice_preempt),
4711+ SUBTEST(live_timeslice_rewind),
4712+ SUBTEST(live_timeslice_queue),
4713+ SUBTEST(live_timeslice_nopreempt),
4714+ SUBTEST(live_busywait_preempt),
4715+ SUBTEST(live_preempt),
4716+ SUBTEST(live_late_preempt),
4717+ SUBTEST(live_nopreempt),
4718+ SUBTEST(live_preempt_cancel),
4719+ SUBTEST(live_suppress_self_preempt),
4720+ SUBTEST(live_chain_preempt),
4721+ SUBTEST(live_preempt_ring),
4722+ SUBTEST(live_preempt_gang),
4723+ SUBTEST(live_preempt_timeout),
4724+ SUBTEST(live_preempt_user),
4725+ SUBTEST(live_preempt_smoke),
4726+ SUBTEST(live_virtual_engine),
4727+ SUBTEST(live_virtual_mask),
4728+ SUBTEST(live_virtual_preserved),
4729+ SUBTEST(live_virtual_slice),
4730+ SUBTEST(live_virtual_bond),
4731+ SUBTEST(live_virtual_reset),
4732+ };
4733+
4734+ if (!HAS_EXECLISTS(i915))
4735+ return 0;
4736+
4737+ if (intel_gt_is_wedged(&i915->gt))
4738+ return 0;
4739+
4740+ return intel_gt_live_subtests(tests, &i915->gt);
4741+}
drivers/gpu/drm/i915/gt/selftest_gt_pm.c+4−4
@@ -71,7 +71,7 @@ static int live_gt_clocks(void *arg) · 70 unmodified lines
7171 enum intel_engine_id id;
7272 int err = 0;
7373
74− if (!RUNTIME_INFO(gt->i915)->cs_timestamp_frequency_hz) { /* unknown */
74+ if (!gt->clock_frequency) { /* unknown */
7575 pr_info("CS_TIMESTAMP frequency unknown\n");
7676 return 0;
7777 }
@@ -112,12 +112,12 @@ static int live_gt_clocks(void *arg) · 34 unmodified lines
112112
113113 measure_clocks(engine, &cycles, &dt);
114114
115− time = i915_cs_timestamp_ticks_to_ns(engine->i915, cycles);
116− expected = i915_cs_timestamp_ns_to_ticks(engine->i915, dt);
115+ time = intel_gt_clock_interval_to_ns(engine->gt, cycles);
116+ expected = intel_gt_ns_to_clock_interval(engine->gt, dt);
117117
118118 pr_info("%s: TIMESTAMP %d cycles [%lldns] in %lldns [%d cycles], using CS clock frequency of %uKHz\n",
119119 engine->name, cycles, time, dt, expected,
120− RUNTIME_INFO(engine->i915)->cs_timestamp_frequency_hz / 1000);
120+ engine->gt->clock_frequency / 1000);
121121
122122 if (9 * time < 8 * dt || 8 * time > 9 * dt) {
123123 pr_err("%s: CS ticks did not match walltime!\n",
drivers/gpu/drm/i915/gt/selftest_hangcheck.c+154−3
@@ -506,7 +506,8 @@ static int igt_reset_nop_engine(void *arg) · 505 unmodified lines
506506 }
507507 err = intel_engine_reset(engine, NULL);
508508 if (err) {
509− pr_err("i915_reset_engine failed\n");
509+ pr_err("intel_engine_reset(%s) failed, err:%d\n",
510+ engine->name, err);
510511 break;
511512 }
512513
@@ -539,6 +540,149 @@ static int igt_reset_nop_engine(void *arg) · 26 unmodified lines
539540 return 0;
540541 }
541542
543+static void force_reset_timeout(struct intel_engine_cs *engine)
544+{
545+ engine->reset_timeout.probability = 999;
546+ atomic_set(&engine->reset_timeout.times, -1);
547+}
548+
549+static void cancel_reset_timeout(struct intel_engine_cs *engine)
550+{
551+ memset(&engine->reset_timeout, 0, sizeof(engine->reset_timeout));
552+}
553+
554+static int igt_reset_fail_engine(void *arg)
555+{
556+ struct intel_gt *gt = arg;
557+ struct intel_engine_cs *engine;
558+ enum intel_engine_id id;
559+
560+ /* Check that we can recover from engine-reset failues */
561+
562+ if (!intel_has_reset_engine(gt))
563+ return 0;
564+
565+ for_each_engine(engine, gt, id) {
566+ unsigned int count;
567+ struct intel_context *ce;
568+ IGT_TIMEOUT(end_time);
569+ int err;
570+
571+ ce = intel_context_create(engine);
572+ if (IS_ERR(ce))
573+ return PTR_ERR(ce);
574+
575+ st_engine_heartbeat_disable(engine);
576+ set_bit(I915_RESET_ENGINE + id, &gt->reset.flags);
577+
578+ force_reset_timeout(engine);
579+ err = intel_engine_reset(engine, NULL);
580+ cancel_reset_timeout(engine);
581+ if (err == 0) /* timeouts only generated on gen8+ */
582+ goto skip;
583+
584+ count = 0;
585+ do {
586+ struct i915_request *last = NULL;
587+ int i;
588+
589+ if (!wait_for_idle(engine)) {
590+ pr_err("%s failed to idle before reset\n",
591+ engine->name);
592+ err = -EIO;
593+ break;
594+ }
595+
596+ for (i = 0; i < count % 15; i++) {
597+ struct i915_request *rq;
598+
599+ rq = intel_context_create_request(ce);
600+ if (IS_ERR(rq)) {
601+ struct drm_printer p =
602+ drm_info_printer(gt->i915->drm.dev);
603+ intel_engine_dump(engine, &p,
604+ "%s(%s): failed to submit request\n",
605+ __func__,
606+ engine->name);
607+
608+ GEM_TRACE("%s(%s): failed to submit request\n",
609+ __func__,
610+ engine->name);
611+ GEM_TRACE_DUMP();
612+
613+ intel_gt_set_wedged(gt);
614+ if (last)
615+ i915_request_put(last);
616+
617+ err = PTR_ERR(rq);
618+ goto out;
619+ }
620+
621+ if (last)
622+ i915_request_put(last);
623+ last = i915_request_get(rq);
624+ i915_request_add(rq);
625+ }
626+
627+ if (count & 1) {
628+ err = intel_engine_reset(engine, NULL);
629+ if (err) {
630+ GEM_TRACE_ERR("intel_engine_reset(%s) failed, err:%d\n",
631+ engine->name, err);
632+ GEM_TRACE_DUMP();
633+ i915_request_put(last);
634+ break;
635+ }
636+ } else {
637+ force_reset_timeout(engine);
638+ err = intel_engine_reset(engine, NULL);
639+ cancel_reset_timeout(engine);
640+ if (err != -ETIMEDOUT) {
641+ pr_err("intel_engine_reset(%s) did not fail, err:%d\n",
642+ engine->name, err);
643+ i915_request_put(last);
644+ break;
645+ }
646+ }
647+
648+ err = 0;
649+ if (last) {
650+ if (i915_request_wait(last, 0, HZ / 2) < 0) {
651+ struct drm_printer p =
652+ drm_info_printer(gt->i915->drm.dev);
653+
654+ intel_engine_dump(engine, &p,
655+ "%s(%s): failed to complete request\n",
656+ __func__,
657+ engine->name);
658+
659+ GEM_TRACE("%s(%s): failed to complete request\n",
660+ __func__,
661+ engine->name);
662+ GEM_TRACE_DUMP();
663+
664+ err = -EIO;
665+ }
666+ i915_request_put(last);
667+ }
668+ count++;
669+ } while (err == 0 && time_before(jiffies, end_time));
670+out:
671+ pr_info("%s(%s): %d resets\n", __func__, engine->name, count);
672+skip:
673+ clear_bit(I915_RESET_ENGINE + id, &gt->reset.flags);
674+ st_engine_heartbeat_enable(engine);
675+ intel_context_put(ce);
676+
677+ if (igt_flush_test(gt->i915))
678+ err = -EIO;
679+ if (err)
680+ return err;
681+ }
682+
683+ return 0;
684+}
685+
542686 static int __igt_reset_engine(struct intel_gt *gt, bool active)
543687 {
544688 struct i915_gpu_error *global = &gt->i915->gpu_error;
@@ -608,7 +752,8 @@ static int __igt_reset_engine(struct intel_gt *gt, bool active) · 63 unmodified lines
608752
609753 err = intel_engine_reset(engine, NULL);
610754 if (err) {
611− pr_err("i915_reset_engine failed\n");
755+ pr_err("intel_engine_reset(%s) failed, err:%d\n",
756+ engine->name, err);
612757 break;
613758 }
614759
@@ -1576,12 +1721,17 @@ static int __igt_atomic_reset_engine(struct intel_engine_cs *engine, · 961 unmodified lines
15761721 engine->name, mode, p->name);
15771722
15781723 tasklet_disable(t);
1724+ if (strcmp(p->name, "softirq"))
1725+ local_bh_disable();
15791726 p->critical_section_begin();
15801727
1581− err = intel_engine_reset(engine, NULL);
1728+ err = __intel_engine_reset_bh(engine, NULL);
15821729
15831730 p->critical_section_end();
1731+ if (strcmp(p->name, "softirq"))
1732+ local_bh_enable();
15841733 tasklet_enable(t);
1734+ tasklet_hi_schedule(t);
15851735
15861736 if (err)
15871737 pr_err("i915_reset_engine(%s:%s) failed under %s\n",
@@ -1687,6 +1837,7 @@ int intel_hangcheck_live_selftests(struct drm_i915_private *i915) · 99 unmodified lines
16871837 SUBTEST(igt_reset_nop_engine),
16881838 SUBTEST(igt_reset_idle_engine),
16891839 SUBTEST(igt_reset_active_engine),
1840+ SUBTEST(igt_reset_fail_engine),
16901841 SUBTEST(igt_reset_engines),
16911842 SUBTEST(igt_reset_engines_atomic),
16921843 SUBTEST(igt_reset_queue),
drivers/gpu/drm/i915/gt/selftest_lrc.c+70−4701
@@ -1,4709 +1,73 @@
1+// SPDX-License-Identifier: MIT
12 /*
2− * SPDX-License-Identifier: MIT
3− *
43 * Copyright © 2018 Intel Corporation
54 */
65
76 #include <linux/prime_numbers.h>
87
9−#include "gem/i915_gem_pm.h"
10−#include "gt/intel_engine_heartbeat.h"
11−#include "gt/intel_reset.h"
12−#include "gt/selftest_engine_heartbeat.h"
13−
14−#include "i915_selftest.h"
15−#include "selftests/i915_random.h"
16−#include "selftests/igt_flush_test.h"
17−#include "selftests/igt_live_test.h"
18−#include "selftests/igt_spinner.h"
19−#include "selftests/lib_sw_fence.h"
20−
21−#include "gem/selftests/igt_gem_utils.h"
22−#include "gem/selftests/mock_context.h"
23−
24−#define CS_GPR(engine, n) ((engine)->mmio_base + 0x600 + (n) * 4)
25−#define NUM_GPR 16
26−#define NUM_GPR_DW (NUM_GPR * 2) /* each GPR is 2 dwords */
27−
28−static struct i915_vma *create_scratch(struct intel_gt *gt)
29−{
30− struct drm_i915_gem_object *obj;
31− struct i915_vma *vma;
32− int err;
33−
34− obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
35− if (IS_ERR(obj))
36− return ERR_CAST(obj);
37−
38− i915_gem_object_set_cache_coherency(obj, I915_CACHING_CACHED);
39−
40− vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
41− if (IS_ERR(vma)) {
42− i915_gem_object_put(obj);
43− return vma;
44− }
45−
46− err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
47− if (err) {
48− i915_gem_object_put(obj);
49− return ERR_PTR(err);
50− }
51−
52− return vma;
53−}
54−
55−static bool is_active(struct i915_request *rq)
56−{
57− if (i915_request_is_active(rq))
58− return true;
59−
60− if (i915_request_on_hold(rq))
61− return true;
62−
63− if (i915_request_has_initial_breadcrumb(rq) && i915_request_started(rq))
64− return true;
65−
66− return false;
67−}
68−
69−static int wait_for_submit(struct intel_engine_cs *engine,
70− struct i915_request *rq,
71− unsigned long timeout)
72−{
73− timeout += jiffies;
74− do {
75− bool done = time_after(jiffies, timeout);
76−
77− if (i915_request_completed(rq)) /* that was quick! */
78− return 0;
79−
80− /* Wait until the HW has acknowleged the submission (or err) */
81− intel_engine_flush_submission(engine);
82− if (!READ_ONCE(engine->execlists.pending[0]) && is_active(rq))
83− return 0;
84−
85− if (done)
86− return -ETIME;
87−
88− cond_resched();
89− } while (1);
90−}
91−
92−static int wait_for_reset(struct intel_engine_cs *engine,
93− struct i915_request *rq,
94− unsigned long timeout)
95−{
96− timeout += jiffies;
97−
98− do {
99− cond_resched();
100− intel_engine_flush_submission(engine);
101−
102− if (READ_ONCE(engine->execlists.pending[0]))
103− continue;
104−
105− if (i915_request_completed(rq))
106− break;
107−
108− if (READ_ONCE(rq->fence.error))
109− break;
110− } while (time_before(jiffies, timeout));
111−
112− flush_scheduled_work();
113−
114− if (rq->fence.error != -EIO) {
115− pr_err("%s: hanging request %llx:%lld not reset\n",
116− engine->name,
117− rq->fence.context,
118− rq->fence.seqno);
119− return -EINVAL;
120− }
121−
122− /* Give the request a jiffie to complete after flushing the worker */
123− if (i915_request_wait(rq, 0,
124− max(0l, (long)(timeout - jiffies)) + 1) < 0) {
125− pr_err("%s: hanging request %llx:%lld did not complete\n",
126− engine->name,
127− rq->fence.context,
128− rq->fence.seqno);
129− return -ETIME;
130− }
131−
132− return 0;
133−}
134−
135−static int live_sanitycheck(void *arg)
136−{
137− struct intel_gt *gt = arg;
138− struct intel_engine_cs *engine;
139− enum intel_engine_id id;
140− struct igt_spinner spin;
141− int err = 0;
142−
143− if (!HAS_LOGICAL_RING_CONTEXTS(gt->i915))
144− return 0;
145−
146− if (igt_spinner_init(&spin, gt))
147− return -ENOMEM;
148−
149− for_each_engine(engine, gt, id) {
150− struct intel_context *ce;
151− struct i915_request *rq;
152−
153− ce = intel_context_create(engine);
154− if (IS_ERR(ce)) {
155− err = PTR_ERR(ce);
156− break;
157− }
158−
159− rq = igt_spinner_create_request(&spin, ce, MI_NOOP);
160− if (IS_ERR(rq)) {
161− err = PTR_ERR(rq);
162− goto out_ctx;
163− }
164−
165− i915_request_add(rq);
166− if (!igt_wait_for_spinner(&spin, rq)) {
167− GEM_TRACE("spinner failed to start\n");
168− GEM_TRACE_DUMP();
169− intel_gt_set_wedged(gt);
170− err = -EIO;
171− goto out_ctx;
172− }
173−
174− igt_spinner_end(&spin);
175− if (igt_flush_test(gt->i915)) {
176− err = -EIO;
177− goto out_ctx;
178− }
179−
180−out_ctx:
181− intel_context_put(ce);
182− if (err)
183− break;
184− }
185−
186− igt_spinner_fini(&spin);
187− return err;
188−}
189−
190−static int live_unlite_restore(struct intel_gt *gt, int prio)
191−{
192− struct intel_engine_cs *engine;
193− enum intel_engine_id id;
194− struct igt_spinner spin;
195− int err = -ENOMEM;
196−
197− /*
198− * Check that we can correctly context switch between 2 instances
199− * on the same engine from the same parent context.
200− */
201−
202− if (igt_spinner_init(&spin, gt))
203− return err;
204−
205− err = 0;
206− for_each_engine(engine, gt, id) {
207− struct intel_context *ce[2] = {};
208− struct i915_request *rq[2];
209− struct igt_live_test t;
210− int n;
211−
212− if (prio && !intel_engine_has_preemption(engine))
213− continue;
214−
215− if (!intel_engine_can_store_dword(engine))
216− continue;
217−
218− if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
219− err = -EIO;
220− break;
221− }
222− st_engine_heartbeat_disable(engine);
223−
224− for (n = 0; n < ARRAY_SIZE(ce); n++) {
225− struct intel_context *tmp;
226−
227− tmp = intel_context_create(engine);
228− if (IS_ERR(tmp)) {
229− err = PTR_ERR(tmp);
230− goto err_ce;
231− }
232−
233− err = intel_context_pin(tmp);
234− if (err) {
235− intel_context_put(tmp);
236− goto err_ce;
237− }
238−
239− /*
240− * Setup the pair of contexts such that if we
241− * lite-restore using the RING_TAIL from ce[1] it
242− * will execute garbage from ce[0]->ring.
243− */
244− memset(tmp->ring->vaddr,
245− POISON_INUSE, /* IPEHR: 0x5a5a5a5a [hung!] */
246− tmp->ring->vma->size);
247−
248− ce[n] = tmp;
249− }
250− GEM_BUG_ON(!ce[1]->ring->size);
251− intel_ring_reset(ce[1]->ring, ce[1]->ring->size / 2);
252− __execlists_update_reg_state(ce[1], engine, ce[1]->ring->head);
253−
254− rq[0] = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
255− if (IS_ERR(rq[0])) {
256− err = PTR_ERR(rq[0]);
257− goto err_ce;
258− }
259−
260− i915_request_get(rq[0]);
261− i915_request_add(rq[0]);
262− GEM_BUG_ON(rq[0]->postfix > ce[1]->ring->emit);
263−
264− if (!igt_wait_for_spinner(&spin, rq[0])) {
265− i915_request_put(rq[0]);
266− goto err_ce;
267− }
268−
269− rq[1] = i915_request_create(ce[1]);
270− if (IS_ERR(rq[1])) {
271− err = PTR_ERR(rq[1]);
272− i915_request_put(rq[0]);
273− goto err_ce;
274− }
275−
276− if (!prio) {
277− /*
278− * Ensure we do the switch to ce[1] on completion.
279− *
280− * rq[0] is already submitted, so this should reduce
281− * to a no-op (a wait on a request on the same engine
282− * uses the submit fence, not the completion fence),
283− * but it will install a dependency on rq[1] for rq[0]
284− * that will prevent the pair being reordered by
285− * timeslicing.
286− */
287− i915_request_await_dma_fence(rq[1], &rq[0]->fence);
288− }
289−
290− i915_request_get(rq[1]);
291− i915_request_add(rq[1]);
292− GEM_BUG_ON(rq[1]->postfix <= rq[0]->postfix);
293− i915_request_put(rq[0]);
294−
295− if (prio) {
296− struct i915_sched_attr attr = {
297− .priority = prio,
298− };
299−
300− /* Alternatively preempt the spinner with ce[1] */
301− engine->schedule(rq[1], &attr);
302− }
303−
304− /* And switch back to ce[0] for good measure */
305− rq[0] = i915_request_create(ce[0]);
306− if (IS_ERR(rq[0])) {
307− err = PTR_ERR(rq[0]);
308− i915_request_put(rq[1]);
309− goto err_ce;
310− }
311−
312− i915_request_await_dma_fence(rq[0], &rq[1]->fence);
313− i915_request_get(rq[0]);
314− i915_request_add(rq[0]);
315− GEM_BUG_ON(rq[0]->postfix > rq[1]->postfix);
316− i915_request_put(rq[1]);
317− i915_request_put(rq[0]);
318−
319−err_ce:
320− intel_engine_flush_submission(engine);
321− igt_spinner_end(&spin);
322− for (n = 0; n < ARRAY_SIZE(ce); n++) {
323− if (IS_ERR_OR_NULL(ce[n]))
324− break;
325−
326− intel_context_unpin(ce[n]);
327− intel_context_put(ce[n]);
328− }
329−
330− st_engine_heartbeat_enable(engine);
331− if (igt_live_test_end(&t))
332− err = -EIO;
333− if (err)
334− break;
335− }
336−
337− igt_spinner_fini(&spin);
338− return err;
339−}
340−
341−static int live_unlite_switch(void *arg)
342−{
343− return live_unlite_restore(arg, 0);
344−}
345−
346−static int live_unlite_preempt(void *arg)
347−{
348− return live_unlite_restore(arg, I915_USER_PRIORITY(I915_PRIORITY_MAX));
349−}
350−
351−static int live_unlite_ring(void *arg)
352−{
353− struct intel_gt *gt = arg;
354− struct intel_engine_cs *engine;
355− struct igt_spinner spin;
356− enum intel_engine_id id;
357− int err = 0;
358−
359− /*
360− * Setup a preemption event that will cause almost the entire ring
361− * to be unwound, potentially fooling our intel_ring_direction()
362− * into emitting a forward lite-restore instead of the rollback.
363− */
364−
365− if (igt_spinner_init(&spin, gt))
366− return -ENOMEM;
367−
368− for_each_engine(engine, gt, id) {
369− struct intel_context *ce[2] = {};
370− struct i915_request *rq;
371− struct igt_live_test t;
372− int n;
373−
374− if (!intel_engine_has_preemption(engine))
375− continue;
376−
377− if (!intel_engine_can_store_dword(engine))
378− continue;
379−
380− if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
381− err = -EIO;
382− break;
383− }
384− st_engine_heartbeat_disable(engine);
385−
386− for (n = 0; n < ARRAY_SIZE(ce); n++) {
387− struct intel_context *tmp;
388−
389− tmp = intel_context_create(engine);
390− if (IS_ERR(tmp)) {
391− err = PTR_ERR(tmp);
392− goto err_ce;
393− }
394−
395− err = intel_context_pin(tmp);
396− if (err) {
397− intel_context_put(tmp);
398− goto err_ce;
399− }
400−
401− memset32(tmp->ring->vaddr,
402− 0xdeadbeef, /* trigger a hang if executed */
403− tmp->ring->vma->size / sizeof(u32));
404−
405− ce[n] = tmp;
406− }
407−
408− /* Create max prio spinner, followed by N low prio nops */
409− rq = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
410− if (IS_ERR(rq)) {
411− err = PTR_ERR(rq);
412− goto err_ce;
413− }
414−
415− i915_request_get(rq);
416− rq->sched.attr.priority = I915_PRIORITY_BARRIER;
417− i915_request_add(rq);
418−
419− if (!igt_wait_for_spinner(&spin, rq)) {
420− intel_gt_set_wedged(gt);
421− i915_request_put(rq);
422− err = -ETIME;
423− goto err_ce;
424− }
425−

4429 more lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_mocs.c+28−52

This diff is large: its 157 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_rc6.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_reset.c+13−6

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_rps.c+4−1

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_timeline.c+7−7

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_workarounds.c+93−103

This diff is large: its 484 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc.c+2−14

This diff is large: its 23 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc.h+0−7

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c+6−7

This diff is large: its 39 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c+270−187

This diff is large: its 604 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_submission.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_uc.c+10−13

This diff is large: its 48 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/cmd_parser.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/handlers.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/mmio_context.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/mmio_context.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/scheduler.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_active.c+2−5

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/i915/i915_cmd_parser.c+10−18

This diff is large: its 83 lines aren’t shown here.

drivers/gpu/drm/i915/i915_debugfs.c+22−17

This diff is large: its 98 lines aren’t shown here.

drivers/gpu/drm/i915/i915_drv.c+1−2

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/i915/i915_drv.h+25−29

This diff is large: its 111 lines aren’t shown here.

drivers/gpu/drm/i915/i915_gem.c+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_gem_gtt.c+10−15

This diff is large: its 33 lines aren’t shown here.

drivers/gpu/drm/i915/i915_getparam.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_gpu_error.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_mitigations.c+146−0

This diff is large: its 147 lines aren’t shown here.

drivers/gpu/drm/i915/i915_mitigations.h+13−0

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pci.c+0−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/i915/i915_perf.c+11−6

This diff is large: its 48 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pmu.c+71−46

This diff is large: its 275 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pmu.h+24−11

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/i915/i915_request.c+128−11

This diff is large: its 210 lines aren’t shown here.

drivers/gpu/drm/i915/i915_request.h+41−6

This diff is large: its 112 lines aren’t shown here.

drivers/gpu/drm/i915/i915_scheduler.c+28−4

This diff is large: its 50 lines aren’t shown here.

drivers/gpu/drm/i915/i915_scheduler.h+7−0

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/i915/i915_scheduler_types.h+10−0

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/i915/i915_sw_fence.c+15−10

This diff is large: its 64 lines aren’t shown here.

drivers/gpu/drm/i915/i915_utils.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_utils.h+6−1

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/i915/intel_device_info.c+0−157

This diff is large: its 171 lines aren’t shown here.

drivers/gpu/drm/i915/intel_device_info.h+0−4

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/i915/intel_memory_region.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_memory_region.h+0−5

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_gem_evict.c+5−13

This diff is large: its 43 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_gem_gtt.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_perf.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_request.c+3−6

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/igt_spinner.c+4−0

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/intel_memory_region.c+98−12

This diff is large: its 161 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/mock_gem_device.c+0−2

This diff is large: its 9 lines aren’t shown here.

include/uapi/drm/i915_drm.h+2−1

This diff is large: its 11 lines aren’t shown here.