History

Merge tag 'drm-intel-gt-next-2021-01-14' of git://anongit.freedesktop.org/drm/drm-intel into drm-next

UAPI Changes: - Deprecate I915_PMU_LAST and optimize state tracking (Tvrtko) Avoid relying on last item ABI marker in i915_drm.h, add a comment to mark as deprecated. Cross-subsystem Changes: Core Changes: Driver Changes: - Restore clear residuals security mitigations for Ivybridge and Baytrail (Chris) - Close #1858: Allow sysadmin to choose applied GPU security mitigations through i915.mitigations=... similar to CPU (Chris) - Fix for #2024: GPU hangs on HSW GT1 (Chris) - Fix for #2707: Driver hang when editing UVs in Blender (Chris, Ville) - Fix for #2797: False positive GuC loading error message (Chris) - Fix for #2859: Missing GuC firmware for older Cometlakes (Chris) - Lessen probability of GPU hang due to DMAR faults [reason 7, next page table ptr is invalid] on Tigerlake (Chris) - Fix REVID macros for TGL to fetch correct stepping (Aditya) - Limit frequency drop to RPe on parking (Chris, Edward) - Limit W/A 1406941453 to TGL, RKL and DG1 (Swathi) - Make W/A 22010271021 permanent on DG1 (Lucas) - Implement W/A 16011163337 to prevent a HS/DS hang on DG1 (Swathi) - Only disable preemption on gen8 render engines (Chris) - Disable arbitration around Braswell's PDP updates (Chris) - Disable arbitration on no-preempt requests (Chris) - Check for arbitration after writing start seqno before busywaiting (Chris) - Retain default context state across shrinking (Venkata, CQ) - Fix mismatch between misplaced vma check and vma insert for 32-bit addressing userspaces (Chris, CQ) - Propagate error for vmap() failure instead kernel NULL deref (Chris) - Propagate error from cancelled submit due to context closure immediately (Chris) - Fix RCU race on HWSP tracking per request (Chris) - Clear CMD parser shadow and GPU reloc batches (Matt A) - Populate logical context during first pin (Maarten) - Optimistically prune dma-resv from the shrinker (Chris) - Fix for virtual engine ownership race (Chris) - Remove timeslice suppression to restore fairness for virtual engines (Chris) - Rearrange IVB/HSW workarounds properly between GT and engine (Chris) - Taint the reset mutex with the shrinker (Chris) - Replace direct submit with direct call to tasklet (Chris) - Multiple corrections to virtual engine dequeue and breadcrumbs code (Chris) - Avoid wakeref from potentially hard IRQ context in PMU (Tvrtko) - Use raw clock for RC6 time estimation in PMU (Tvrtko) - Differentiate OOM failures from invalid map types (Chris) - Fix Gen9 to have 64 MOCS entries similar to Gen11 (Chris) - Ignore repeated attempts to suspend request flow across reset (Chris) - Remove livelock from "do_idle_maps" VT-d W/A (Chris) - Cancel the preemption timeout early in case engine reset fails (Chris) - Code flow optimization in the scheduling code (Chris) - Clear the execlists timers upon reset (Chris) - Drain the breadcrumbs just once (Chris, Matt A) - Track the overall GT awake/busy time (Chris) - Tweak submission tasklet flushing to avoid starvation (Chris) - Track timelines created using the HWSP to restore on resume (Chris) - Use cmpxchg64 for 32b compatilibity for active tracking (Chris) - Prefer recycling an idle GGTT fence to avoid GPU wait (Chris) - Restructure GT code organization for clearer split between GuC and execlists (Chris, Daniele, John, Matt A) - Remove GuC code that will remain unused by new interfaces (Matt B) - Restructure the CS timestamp clocks code to local to GT (Chris) - Fix error return paths in perf code (Zhang) - Replace idr_init() by idr_init_base() in perf (Deepak) - Fix shmem_pin_map error path (Colin) - Drop redundant free_work worker for GEM contexts (Chris, Mika) - Increase readability and understandability of intel_workarounds.c (Lucas) - Defer enabling the breadcrumb interrupt to after submission (Chris) - Deal with buddy alloc block sizes beyond 4G (Venkata, Chris) - Encode fence specific waitqueue behaviour into the wait.flags (Chris) - Don't cancel the breadcrumb interrupt shadow too early (Chris) - Cancel submitted requests upon context reset (Chris) - Use correct locks in GuC code (Tvrtko) - Prevent use of engine->wa_ctx after error (Chris, Matt R) - Fix build warning on 32-bit (Arnd) - Avoid memory leak if platform would have more than 16 W/A (Tvrtko) - Avoid unnecessary #if CONFIG_PM in PMU code (Chris, Tvrtko) - Improve debugging output (Chris, Tvrtko, Matt R) - Make file local variables static (Jani) - Avoid uint*_t types in i915 (Jani) - Selftest improvements (Chris, Matt A, Dan) - Documentation fixes (Chris, Jose) Signed-off-by: Dave Airlie <airlied@redhat.com> # Conflicts: # drivers/gpu/drm/i915/gt/intel_breadcrumbs.c # drivers/gpu/drm/i915/gt/intel_breadcrumbs_types.h # drivers/gpu/drm/i915/gt/intel_lrc.c # drivers/gpu/drm/i915/gvt/mmio_context.h # drivers/gpu/drm/i915/i915_drv.h From: Joonas Lahtinen <joonas.lahtinen@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20210114152232.GA21588@jlahtine-mobl.ger.corp.intel.com

Dave Airlie committed 5 years ago

Commit fb5cfca2 parents dfa7c52 368fd0d130 files · +13692 −12273Browse files

Unified · Split

130 files
  1. Documentation/gpu/i915.rst +1 −1
  2. drivers/gpu/drm/i915/Makefile +5 −1
  3. drivers/gpu/drm/i915/display/intel_overlay.c +1 −0
  4. drivers/gpu/drm/i915/dma_resv_utils.c +17 −0
  5. drivers/gpu/drm/i915/dma_resv_utils.h +13 −0
  6. drivers/gpu/drm/i915/gem/i915_gem_context.c +13 −54
  7. drivers/gpu/drm/i915/gem/i915_gem_context.h +0 −1
  8. drivers/gpu/drm/i915/gem/i915_gem_context_types.h +0 −1
  9. drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c +8 −7
  10. drivers/gpu/drm/i915/gem/i915_gem_object_blt.c +1 −0
  11. drivers/gpu/drm/i915/gem/i915_gem_pages.c +16 −16
  12. drivers/gpu/drm/i915/gem/i915_gem_region.c +20 −16
  13. drivers/gpu/drm/i915/gem/i915_gem_shrinker.c +3 −0
  14. drivers/gpu/drm/i915/gem/i915_gem_stolen.c +2 −3
  15. drivers/gpu/drm/i915/gem/i915_gem_wait.c +3 −5
  16. drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c +1 −1
  17. drivers/gpu/drm/i915/gem/selftests/huge_pages.c +22 −0
  18. drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c +1 −3
  19. drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c +1 −0
  20. drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c +1 −0
  21. drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c +1 −0
  22. drivers/gpu/drm/i915/gt/debugfs_gt_pm.c +15 −12
  23. drivers/gpu/drm/i915/gt/gen7_renderclear.c +94 −63
  24. drivers/gpu/drm/i915/gt/gen8_engine_cs.c +633 −0
  25. drivers/gpu/drm/i915/gt/gen8_engine_cs.h +127 −0
  26. drivers/gpu/drm/i915/gt/intel_breadcrumbs.c +73 −37
  27. drivers/gpu/drm/i915/gt/intel_breadcrumbs.h +16 −1
  28. drivers/gpu/drm/i915/gt/intel_breadcrumbs_types.h +5 −2
  29. drivers/gpu/drm/i915/gt/intel_context.h +7 −4
  30. drivers/gpu/drm/i915/gt/intel_context_sseu.c +0 −1
  31. drivers/gpu/drm/i915/gt/intel_context_types.h +17 −8
  32. drivers/gpu/drm/i915/gt/intel_engine.h +6 −87
  33. drivers/gpu/drm/i915/gt/intel_engine_cs.c +80 −71
  34. drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c +41 −18
  35. drivers/gpu/drm/i915/gt/intel_engine_pm.c +9 −1
  36. drivers/gpu/drm/i915/gt/intel_engine_types.h +5 −11
  37. drivers/gpu/drm/i915/gt/intel_execlists_submission.c +3929 −0
  38. drivers/gpu/drm/i915/gt/intel_execlists_submission.h +47 −0
  39. drivers/gpu/drm/i915/gt/intel_ggtt.c +16 −2
  40. drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c +20 −2
  41. drivers/gpu/drm/i915/gt/intel_gt.c +2 −2
  42. drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c +157 −40
  43. drivers/gpu/drm/i915/gt/intel_gt_clock_utils.h +4 −4
  44. drivers/gpu/drm/i915/gt/intel_gt_irq.c +1 −0
  45. drivers/gpu/drm/i915/gt/intel_gt_pm.c +49 −0
  46. drivers/gpu/drm/i915/gt/intel_gt_pm.h +2 −0
  47. drivers/gpu/drm/i915/gt/intel_gt_requests.c +1 −6
  48. drivers/gpu/drm/i915/gt/intel_gt_types.h +25 −0
  49. drivers/gpu/drm/i915/gt/intel_gtt.c +29 −0
  50. drivers/gpu/drm/i915/gt/intel_gtt.h +3 −0
  51. drivers/gpu/drm/i915/gt/intel_lrc.c +1149 −5731
  52. drivers/gpu/drm/i915/gt/intel_lrc.h +61 −106
  53. drivers/gpu/drm/i915/gt/intel_lrc_reg.h +41 −0
  54. drivers/gpu/drm/i915/gt/intel_mocs.c +1 −1
  55. drivers/gpu/drm/i915/gt/intel_region_lmem.c +1 −1
  56. drivers/gpu/drm/i915/gt/intel_region_lmem.h +0 −2
  57. drivers/gpu/drm/i915/gt/intel_renderstate.c +2 −1
  58. drivers/gpu/drm/i915/gt/intel_reset.c +62 −34
  59. drivers/gpu/drm/i915/gt/intel_reset.h +2 −0
  60. drivers/gpu/drm/i915/gt/intel_ring.c +2 −0
  61. drivers/gpu/drm/i915/gt/intel_ring_submission.c +43 −30
  62. drivers/gpu/drm/i915/gt/intel_rps.c +35 −24
  63. drivers/gpu/drm/i915/gt/intel_rps_types.h +1 −1
  64. drivers/gpu/drm/i915/gt/intel_timeline.c +103 −6
  65. drivers/gpu/drm/i915/gt/intel_timeline.h +18 −7
  66. drivers/gpu/drm/i915/gt/intel_timeline_types.h +2 −0
  67. drivers/gpu/drm/i915/gt/intel_workarounds.c +309 −335
  68. drivers/gpu/drm/i915/gt/mock_engine.c +2 −13
  69. drivers/gpu/drm/i915/gt/selftest_context.c +1 −1
  70. drivers/gpu/drm/i915/gt/selftest_engine_cs.c +1 −0
  71. drivers/gpu/drm/i915/gt/selftest_engine_heartbeat.c +10 −3
  72. drivers/gpu/drm/i915/gt/selftest_engine_pm.c +203 −0
  73. drivers/gpu/drm/i915/gt/selftest_execlists.c +4741 −0
  74. drivers/gpu/drm/i915/gt/selftest_gt_pm.c +4 −4
  75. drivers/gpu/drm/i915/gt/selftest_hangcheck.c +154 −3
  76. drivers/gpu/drm/i915/gt/selftest_lrc.c +70 −4701
  77. drivers/gpu/drm/i915/gt/selftest_mocs.c +28 −52
  78. drivers/gpu/drm/i915/gt/selftest_rc6.c +1 −0
  79. drivers/gpu/drm/i915/gt/selftest_reset.c +13 −6
  80. drivers/gpu/drm/i915/gt/selftest_rps.c +4 −1
  81. drivers/gpu/drm/i915/gt/selftest_timeline.c +7 −7
  82. drivers/gpu/drm/i915/gt/selftest_workarounds.c +93 −103
  83. drivers/gpu/drm/i915/gt/uc/intel_guc.c +2 −14
  84. drivers/gpu/drm/i915/gt/uc/intel_guc.h +0 −7
  85. drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c +1 −0
  86. drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c +6 −7
  87. drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c +270 −187
  88. drivers/gpu/drm/i915/gt/uc/intel_guc_submission.h +1 −0
  89. drivers/gpu/drm/i915/gt/uc/intel_uc.c +10 −13
  90. drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c +1 −0
  91. drivers/gpu/drm/i915/gvt/cmd_parser.c +1 −0
  92. drivers/gpu/drm/i915/gvt/handlers.c +1 −1
  93. drivers/gpu/drm/i915/gvt/mmio_context.c +1 −0
  94. drivers/gpu/drm/i915/gvt/mmio_context.h +1 −0
  95. drivers/gpu/drm/i915/gvt/scheduler.c +2 −0
  96. drivers/gpu/drm/i915/i915_active.c +2 −5
  97. drivers/gpu/drm/i915/i915_cmd_parser.c +10 −18
  98. drivers/gpu/drm/i915/i915_debugfs.c +22 −17
  99. drivers/gpu/drm/i915/i915_drv.c +1 −2
  100. drivers/gpu/drm/i915/i915_drv.h +25 −29
  101. drivers/gpu/drm/i915/i915_gem.c +0 −2
  102. drivers/gpu/drm/i915/i915_gem_gtt.c +10 −15
  103. drivers/gpu/drm/i915/i915_getparam.c +1 −1
  104. drivers/gpu/drm/i915/i915_gpu_error.c +1 −1
  105. drivers/gpu/drm/i915/i915_mitigations.c +146 −0
  106. drivers/gpu/drm/i915/i915_mitigations.h +13 −0
  107. drivers/gpu/drm/i915/i915_pci.c +0 −2
  108. drivers/gpu/drm/i915/i915_perf.c +11 −6
  109. drivers/gpu/drm/i915/i915_pmu.c +71 −46
  110. drivers/gpu/drm/i915/i915_pmu.h +24 −11
  111. drivers/gpu/drm/i915/i915_request.c +128 −11
  112. drivers/gpu/drm/i915/i915_request.h +41 −6
  113. drivers/gpu/drm/i915/i915_scheduler.c +28 −4
  114. drivers/gpu/drm/i915/i915_scheduler.h +7 −0
  115. drivers/gpu/drm/i915/i915_scheduler_types.h +10 −0
  116. drivers/gpu/drm/i915/i915_sw_fence.c +15 −10
  117. drivers/gpu/drm/i915/i915_utils.c +1 −1
  118. drivers/gpu/drm/i915/i915_utils.h +6 −1
  119. drivers/gpu/drm/i915/intel_device_info.c +0 −157
  120. drivers/gpu/drm/i915/intel_device_info.h +0 −4
  121. drivers/gpu/drm/i915/intel_memory_region.c +1 −1
  122. drivers/gpu/drm/i915/intel_memory_region.h +0 −5
  123. drivers/gpu/drm/i915/selftests/i915_gem_evict.c +5 −13
  124. drivers/gpu/drm/i915/selftests/i915_gem_gtt.c +1 −0
  125. drivers/gpu/drm/i915/selftests/i915_perf.c +1 −1
  126. drivers/gpu/drm/i915/selftests/i915_request.c +3 −6
  127. drivers/gpu/drm/i915/selftests/igt_spinner.c +4 −0
  128. drivers/gpu/drm/i915/selftests/intel_memory_region.c +98 −12
  129. drivers/gpu/drm/i915/selftests/mock_gem_device.c +0 −2
  130. include/uapi/drm/i915_drm.h +2 −1
Documentation/gpu/i915.rst+1−1
@@ -428,7 +428,7 @@ User Batchbuffer Execution · 427 unmodified lines
428Logical Rings, Logical Ring Contexts and Execlists428Logical Rings, Logical Ring Contexts and Execlists
429--------------------------------------------------429--------------------------------------------------
430 430
431.. kernel-doc:: drivers/gpu/drm/i915/gt/intel_lrc.c431.. kernel-doc:: drivers/gpu/drm/i915/gt/intel_execlists_submission.c
432 :doc: Logical Rings, Logical Ring Contexts and Execlists432 :doc: Logical Rings, Logical Ring Contexts and Execlists
433 433
434Global GTT views434Global GTT views
drivers/gpu/drm/i915/Makefile+5−1
@@ -38,6 +38,7 @@ i915-y += i915_drv.o \ · 37 unmodified lines
38 i915_config.o \38 i915_config.o \
39 i915_irq.o \39 i915_irq.o \
40 i915_getparam.o \40 i915_getparam.o \
41 i915_mitigations.o \
41 i915_params.o \42 i915_params.o \
42 i915_pci.o \43 i915_pci.o \
43 i915_scatterlist.o \44 i915_scatterlist.o \
@@ -58,6 +59,7 @@ i915-y += i915_drv.o \ · 14 unmodified lines
58 59
59# core library code60# core library code
60i915-y += \61i915-y += \
62 dma_resv_utils.o \
61 i915_memcpy.o \63 i915_memcpy.o \
62 i915_mm.o \64 i915_mm.o \
63 i915_sw_fence.o \65 i915_sw_fence.o \
@@ -82,6 +84,7 @@ gt-y += \ · 18 unmodified lines
82 gt/gen6_engine_cs.o \84 gt/gen6_engine_cs.o \
83 gt/gen6_ppgtt.o \85 gt/gen6_ppgtt.o \
84 gt/gen7_renderclear.o \86 gt/gen7_renderclear.o \
87 gt/gen8_engine_cs.o \
85 gt/gen8_ppgtt.o \88 gt/gen8_ppgtt.o \
86 gt/intel_breadcrumbs.o \89 gt/intel_breadcrumbs.o \
87 gt/intel_context.o \90 gt/intel_context.o \
@@ -91,6 +94,7 @@ gt-y += \ · 3 unmodified lines
91 gt/intel_engine_heartbeat.o \94 gt/intel_engine_heartbeat.o \
92 gt/intel_engine_pm.o \95 gt/intel_engine_pm.o \
93 gt/intel_engine_user.o \96 gt/intel_engine_user.o \
97 gt/intel_execlists_submission.o \
94 gt/intel_ggtt.o \98 gt/intel_ggtt.o \
95 gt/intel_ggtt_fencing.o \99 gt/intel_ggtt_fencing.o \
96 gt/intel_gt.o \100 gt/intel_gt.o \
@@ -106,6 +110,7 @@ gt-y += \ · 9 unmodified lines
106 gt/intel_mocs.o \110 gt/intel_mocs.o \
107 gt/intel_ppgtt.o \111 gt/intel_ppgtt.o \
108 gt/intel_rc6.o \112 gt/intel_rc6.o \
113 gt/intel_region_lmem.o \
109 gt/intel_renderstate.o \114 gt/intel_renderstate.o \
110 gt/intel_reset.o \115 gt/intel_reset.o \
111 gt/intel_ring.o \116 gt/intel_ring.o \
@@ -166,7 +171,6 @@ i915-y += \ · 54 unmodified lines
166 i915_scheduler.o \171 i915_scheduler.o \
167 i915_trace_points.o \172 i915_trace_points.o \
168 i915_vma.o \173 i915_vma.o \
169 intel_region_lmem.o \
170 intel_wopcm.o174 intel_wopcm.o
171 175
172# general-purpose microcontroller (GuC) support176# general-purpose microcontroller (GuC) support
drivers/gpu/drm/i915/display/intel_overlay.c+1−0
@@ -29,6 +29,7 @@ · 28 unmodified lines
29#include <drm/drm_fourcc.h>29#include <drm/drm_fourcc.h>
30 30
31#include "gem/i915_gem_pm.h"31#include "gem/i915_gem_pm.h"
32#include "gt/intel_gpu_commands.h"
32#include "gt/intel_ring.h"33#include "gt/intel_ring.h"
33 34
34#include "i915_drv.h"35#include "i915_drv.h"
drivers/gpu/drm/i915/dma_resv_utils.c+17−0
@@ -0,0 +1,17 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright © 2020 Intel Corporation
4 */
5
6#include <linux/dma-resv.h>
7
8#include "dma_resv_utils.h"
9
10void dma_resv_prune(struct dma_resv *resv)
11{
12 if (dma_resv_trylock(resv)) {
13 if (dma_resv_test_signaled_rcu(resv, true))
14 dma_resv_add_excl_fence(resv, NULL);
15 dma_resv_unlock(resv);
16 }
17}
drivers/gpu/drm/i915/dma_resv_utils.h+13−0
@@ -0,0 +1,13 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright © 2020 Intel Corporation
4 */
5
6#ifndef DMA_RESV_UTILS_H
7#define DMA_RESV_UTILS_H
8
9struct dma_resv;
10
11void dma_resv_prune(struct dma_resv *resv);
12
13#endif /* DMA_RESV_UTILS_H */
drivers/gpu/drm/i915/gem/i915_gem_context.c+13−54
@@ -72,6 +72,8 @@ · 71 unmodified lines
72#include "gt/intel_context_param.h"72#include "gt/intel_context_param.h"
73#include "gt/intel_engine_heartbeat.h"73#include "gt/intel_engine_heartbeat.h"
74#include "gt/intel_engine_user.h"74#include "gt/intel_engine_user.h"
75#include "gt/intel_execlists_submission.h" /* virtual_engine */
76#include "gt/intel_gpu_commands.h"
75#include "gt/intel_ring.h"77#include "gt/intel_ring.h"
76 78
77#include "i915_gem_context.h"79#include "i915_gem_context.h"
@@ -333,13 +335,12 @@ static struct i915_gem_engines *default_engines(struct i915_gem_context *ctx) · 255 unmodified lines
333 return e;335 return e;
334}336}
335 337
336static void i915_gem_context_free(struct i915_gem_context *ctx)338void i915_gem_context_release(struct kref *ref)
337{339{
338 GEM_BUG_ON(!i915_gem_context_is_closed(ctx));340 struct i915_gem_context *ctx = container_of(ref, typeof(*ctx), ref);
339 341
340 spin_lock(&ctx->i915->gem.contexts.lock);342 trace_i915_context_free(ctx);
341 list_del(&ctx->link);343 GEM_BUG_ON(!i915_gem_context_is_closed(ctx));
342 spin_unlock(&ctx->i915->gem.contexts.lock);
343 344
344 mutex_destroy(&ctx->engines_mutex);345 mutex_destroy(&ctx->engines_mutex);
345 mutex_destroy(&ctx->lut_mutex);346 mutex_destroy(&ctx->lut_mutex);
@@ -353,37 +354,6 @@ static void i915_gem_context_free(struct i915_gem_context *ctx) · 7 unmodified lines
353 kfree_rcu(ctx, rcu);354 kfree_rcu(ctx, rcu);
354}355}
355 356
356static void contexts_free_all(struct llist_node *list)
357{
358 struct i915_gem_context *ctx, *cn;
359
360 llist_for_each_entry_safe(ctx, cn, list, free_link)
361 i915_gem_context_free(ctx);
362}
363
364static void contexts_flush_free(struct i915_gem_contexts *gc)
365{
366 contexts_free_all(llist_del_all(&gc->free_list));
367}
368
369static void contexts_free_worker(struct work_struct *work)
370{
371 struct i915_gem_contexts *gc =
372 container_of(work, typeof(*gc), free_work);
373
374 contexts_flush_free(gc);
375}
376
377void i915_gem_context_release(struct kref *ref)
378{
379 struct i915_gem_context *ctx = container_of(ref, typeof(*ctx), ref);
380 struct i915_gem_contexts *gc = &ctx->i915->gem.contexts;
381
382 trace_i915_context_free(ctx);
383 if (llist_add(&ctx->free_link, &gc->free_list))
384 schedule_work(&gc->free_work);
385}
386
387static inline struct i915_gem_engines *357static inline struct i915_gem_engines *
388__context_engines_static(const struct i915_gem_context *ctx)358__context_engines_static(const struct i915_gem_context *ctx)
389{359{
@@ -453,6 +423,9 @@ static struct intel_engine_cs *active_engine(struct intel_context *ce) · 63 unmodified lines
453 struct intel_engine_cs *engine = NULL;423 struct intel_engine_cs *engine = NULL;
454 struct i915_request *rq;424 struct i915_request *rq;
455 425
426 if (intel_context_has_inflight(ce))
427 return intel_context_inflight(ce);
428
456 if (!ce->timeline)429 if (!ce->timeline)
457 return NULL;430 return NULL;
458 431
@@ -632,6 +605,10 @@ static void context_close(struct i915_gem_context *ctx) · 173 unmodified lines
632 */605 */
633 lut_close(ctx);606 lut_close(ctx);
634 607
608 spin_lock(&ctx->i915->gem.contexts.lock);
609 list_del(&ctx->link);
610 spin_unlock(&ctx->i915->gem.contexts.lock);
611
635 mutex_unlock(&ctx->mutex);612 mutex_unlock(&ctx->mutex);
636 613
637 /*614 /*
@@ -849,9 +826,6 @@ i915_gem_create_context(struct drm_i915_private *i915, unsigned int flags) · 211 unmodified lines
849 !HAS_EXECLISTS(i915))826 !HAS_EXECLISTS(i915))
850 return ERR_PTR(-EINVAL);827 return ERR_PTR(-EINVAL);
851 828
852 /* Reap the stale contexts */
853 contexts_flush_free(&i915->gem.contexts);
854
855 ctx = __create_context(i915);829 ctx = __create_context(i915);
856 if (IS_ERR(ctx))830 if (IS_ERR(ctx))
857 return ctx;831 return ctx;
@@ -896,23 +870,11 @@ static void init_contexts(struct i915_gem_contexts *gc) · 38 unmodified lines
896{870{
897 spin_lock_init(&gc->lock);871 spin_lock_init(&gc->lock);
898 INIT_LIST_HEAD(&gc->list);872 INIT_LIST_HEAD(&gc->list);
899
900 INIT_WORK(&gc->free_work, contexts_free_worker);
901 init_llist_head(&gc->free_list);
902}873}
903 874
904void i915_gem_init__contexts(struct drm_i915_private *i915)875void i915_gem_init__contexts(struct drm_i915_private *i915)
905{876{
906 init_contexts(&i915->gem.contexts);877 init_contexts(&i915->gem.contexts);
907 drm_dbg(&i915->drm, "%s context support initialized\n",
908 DRIVER_CAPS(i915)->has_logical_contexts ?
909 "logical" : "fake");
910}
911
912void i915_gem_driver_release__contexts(struct drm_i915_private *i915)
913{
914 flush_work(&i915->gem.contexts.free_work);
915 rcu_barrier(); /* and flush the left over RCU frees */
916}878}
917 879
918static int gem_context_register(struct i915_gem_context *ctx,880static int gem_context_register(struct i915_gem_context *ctx,
@@ -988,7 +950,6 @@ err: · 69 unmodified lines
988void i915_gem_context_close(struct drm_file *file)950void i915_gem_context_close(struct drm_file *file)
989{951{
990 struct drm_i915_file_private *file_priv = file->driver_priv;952 struct drm_i915_file_private *file_priv = file->driver_priv;
991 struct drm_i915_private *i915 = file_priv->dev_priv;
992 struct i915_address_space *vm;953 struct i915_address_space *vm;
993 struct i915_gem_context *ctx;954 struct i915_gem_context *ctx;
994 unsigned long idx;955 unsigned long idx;
@@ -1000,8 +961,6 @@ void i915_gem_context_close(struct drm_file *file) · 5 unmodified lines
1000 xa_for_each(&file_priv->vm_xa, idx, vm)961 xa_for_each(&file_priv->vm_xa, idx, vm)
1001 i915_vm_put(vm);962 i915_vm_put(vm);
1002 xa_destroy(&file_priv->vm_xa);963 xa_destroy(&file_priv->vm_xa);
1003
1004 contexts_flush_free(&i915->gem.contexts);
1005}964}
1006 965
1007int i915_gem_vm_create_ioctl(struct drm_device *dev, void *data,966int i915_gem_vm_create_ioctl(struct drm_device *dev, void *data,
drivers/gpu/drm/i915/gem/i915_gem_context.h+0−1
@@ -110,7 +110,6 @@ i915_gem_context_clear_user_engines(struct i915_gem_context *ctx) · 109 unmodified lines
110 110
111/* i915_gem_context.c */111/* i915_gem_context.c */
112void i915_gem_init__contexts(struct drm_i915_private *i915);112void i915_gem_init__contexts(struct drm_i915_private *i915);
113void i915_gem_driver_release__contexts(struct drm_i915_private *i915);
114 113
115int i915_gem_context_open(struct drm_i915_private *i915,114int i915_gem_context_open(struct drm_i915_private *i915,
116 struct drm_file *file);115 struct drm_file *file);
drivers/gpu/drm/i915/gem/i915_gem_context_types.h+0−1
@@ -108,7 +108,6 @@ struct i915_gem_context { · 107 unmodified lines
108 108
109 /** link: place with &drm_i915_private.context_list */109 /** link: place with &drm_i915_private.context_list */
110 struct list_head link;110 struct list_head link;
111 struct llist_node free_link;
112 111
113 /**112 /**
114 * @ref: reference count113 * @ref: reference count
drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c+8−7
@@ -15,6 +15,7 @@ · 14 unmodified lines
15 15
16#include "gem/i915_gem_ioctls.h"16#include "gem/i915_gem_ioctls.h"
17#include "gt/intel_context.h"17#include "gt/intel_context.h"
18#include "gt/intel_gpu_commands.h"
18#include "gt/intel_gt.h"19#include "gt/intel_gt.h"
19#include "gt/intel_gt_buffer_pool.h"20#include "gt/intel_gt_buffer_pool.h"
20#include "gt/intel_gt_pm.h"21#include "gt/intel_gt_pm.h"
@@ -534,8 +535,6 @@ eb_add_vma(struct i915_execbuffer *eb, · 513 unmodified lines
534 struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];535 struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];
535 struct eb_vma *ev = &eb->vma[i];536 struct eb_vma *ev = &eb->vma[i];
536 537
537 GEM_BUG_ON(i915_vma_is_closed(vma));
538
539 ev->vma = vma;538 ev->vma = vma;
540 ev->exec = entry;539 ev->exec = entry;
541 ev->flags = entry->flags;540 ev->flags = entry->flags;
@@ -1046,7 +1045,7 @@ static void reloc_gpu_flush(struct i915_execbuffer *eb, struct reloc_cache *cach · 504 unmodified lines
1046 GEM_BUG_ON(cache->rq_size >= obj->base.size / sizeof(u32));1045 GEM_BUG_ON(cache->rq_size >= obj->base.size / sizeof(u32));
1047 cache->rq_cmd[cache->rq_size] = MI_BATCH_BUFFER_END;1046 cache->rq_cmd[cache->rq_size] = MI_BATCH_BUFFER_END;
1048 1047
1049 __i915_gem_object_flush_map(obj, 0, sizeof(u32) * (cache->rq_size + 1));1048 i915_gem_object_flush_map(obj);
1050 i915_gem_object_unpin_map(obj);1049 i915_gem_object_unpin_map(obj);
1051 1050
1052 intel_gt_chipset_flush(cache->rq->engine->gt);1051 intel_gt_chipset_flush(cache->rq->engine->gt);
@@ -1296,6 +1295,8 @@ static int __reloc_gpu_alloc(struct i915_execbuffer *eb, · 243 unmodified lines
1296 goto err_pool;1295 goto err_pool;
1297 }1296 }
1298 1297
1298 memset32(cmd, 0, pool->obj->base.size / sizeof(u32));
1299
1299 batch = i915_vma_instance(pool->obj, vma->vm, NULL);1300 batch = i915_vma_instance(pool->obj, vma->vm, NULL);
1300 if (IS_ERR(batch)) {1301 if (IS_ERR(batch)) {
1301 err = PTR_ERR(batch);1302 err = PTR_ERR(batch);
@@ -2533,6 +2534,9 @@ static int eb_submit(struct i915_execbuffer *eb, struct i915_vma *batch) · 1231 unmodified lines
2533{2534{
2534 int err;2535 int err;
2535 2536
2537 if (intel_context_nopreempt(eb->context))
2538 __set_bit(I915_FENCE_FLAG_NOPREEMPT, &eb->request->fence.flags);
2539
2536 err = eb_move_to_gpu(eb);2540 err = eb_move_to_gpu(eb);
2537 if (err)2541 if (err)
2538 return err;2542 return err;
@@ -2573,15 +2577,12 @@ static int eb_submit(struct i915_execbuffer *eb, struct i915_vma *batch) · 34 unmodified lines
2573 return err;2577 return err;
2574 }2578 }
2575 2579
2576 if (intel_context_nopreempt(eb->context))
2577 __set_bit(I915_FENCE_FLAG_NOPREEMPT, &eb->request->fence.flags);
2578
2579 return 0;2580 return 0;
2580}2581}
2581 2582
2582static int num_vcs_engines(const struct drm_i915_private *i915)2583static int num_vcs_engines(const struct drm_i915_private *i915)
2583{2584{
2584 return hweight64(VDBOX_MASK(&i915->gt));2585 return hweight_long(VDBOX_MASK(&i915->gt));
2585}2586}
2586 2587
2587/*2588/*
drivers/gpu/drm/i915/gem/i915_gem_object_blt.c+1−0
@@ -6,6 +6,7 @@ · 5 unmodified lines
6#include "i915_drv.h"6#include "i915_drv.h"
7#include "gt/intel_context.h"7#include "gt/intel_context.h"
8#include "gt/intel_engine_pm.h"8#include "gt/intel_engine_pm.h"
9#include "gt/intel_gpu_commands.h"
9#include "gt/intel_gt.h"10#include "gt/intel_gt.h"
10#include "gt/intel_gt_buffer_pool.h"11#include "gt/intel_gt_buffer_pool.h"
11#include "gt/intel_ring.h"12#include "gt/intel_ring.h"
drivers/gpu/drm/i915/gem/i915_gem_pages.c+16−16
@@ -238,7 +238,7 @@ unlock: · 237 unmodified lines
238 238
239/* The 'mapping' part of i915_gem_object_pin_map() below */239/* The 'mapping' part of i915_gem_object_pin_map() below */
240static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj,240static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj,
241 enum i915_map_type type)241 enum i915_map_type type)
242{242{
243 unsigned long n_pages = obj->base.size >> PAGE_SHIFT, i;243 unsigned long n_pages = obj->base.size >> PAGE_SHIFT, i;
244 struct page *stack[32], **pages = stack, *page;244 struct page *stack[32], **pages = stack, *page;
@@ -281,7 +281,7 @@ static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj, · 36 unmodified lines
281 /* Too big for stack -- allocate temporary array instead */281 /* Too big for stack -- allocate temporary array instead */
282 pages = kvmalloc_array(n_pages, sizeof(*pages), GFP_KERNEL);282 pages = kvmalloc_array(n_pages, sizeof(*pages), GFP_KERNEL);
283 if (!pages)283 if (!pages)
284 return NULL;284 return ERR_PTR(-ENOMEM);
285 }285 }
286 286
287 i = 0;287 i = 0;
@@ -290,11 +290,12 @@ static void *i915_gem_object_map_page(struct drm_i915_gem_object *obj, · 2 unmodified lines
290 vaddr = vmap(pages, n_pages, 0, pgprot);290 vaddr = vmap(pages, n_pages, 0, pgprot);
291 if (pages != stack)291 if (pages != stack)
292 kvfree(pages);292 kvfree(pages);
293 return vaddr;293
294 return vaddr ?: ERR_PTR(-ENOMEM);
294}295}
295 296
296static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj,297static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj,
297 enum i915_map_type type)298 enum i915_map_type type)
298{299{
299 resource_size_t iomap = obj->mm.region->iomap.base -300 resource_size_t iomap = obj->mm.region->iomap.base -
300 obj->mm.region->region.start;301 obj->mm.region->region.start;
@@ -305,13 +306,13 @@ static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj, · 4 unmodified lines
305 void *vaddr;306 void *vaddr;
306 307
307 if (type != I915_MAP_WC)308 if (type != I915_MAP_WC)
308 return NULL;309 return ERR_PTR(-ENODEV);
309 310
310 if (n_pfn > ARRAY_SIZE(stack)) {311 if (n_pfn > ARRAY_SIZE(stack)) {
311 /* Too big for stack -- allocate temporary array instead */312 /* Too big for stack -- allocate temporary array instead */
312 pfns = kvmalloc_array(n_pfn, sizeof(*pfns), GFP_KERNEL);313 pfns = kvmalloc_array(n_pfn, sizeof(*pfns), GFP_KERNEL);
313 if (!pfns)314 if (!pfns)
314 return NULL;315 return ERR_PTR(-ENOMEM);
315 }316 }
316 317
317 i = 0;318 i = 0;
@@ -320,7 +321,8 @@ static void *i915_gem_object_map_pfn(struct drm_i915_gem_object *obj, · 2 unmodified lines
320 vaddr = vmap_pfn(pfns, n_pfn, pgprot_writecombine(PAGE_KERNEL_IO));321 vaddr = vmap_pfn(pfns, n_pfn, pgprot_writecombine(PAGE_KERNEL_IO));
321 if (pfns != stack)322 if (pfns != stack)
322 kvfree(pfns);323 kvfree(pfns);
323 return vaddr;324
325 return vaddr ?: ERR_PTR(-ENOMEM);
324}326}
325 327
326/* get, pin, and map the pages of the object into kernel space */328/* get, pin, and map the pages of the object into kernel space */
@@ -349,8 +351,10 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj, · 22 unmodified lines
349 GEM_BUG_ON(i915_gem_object_has_pinned_pages(obj));351 GEM_BUG_ON(i915_gem_object_has_pinned_pages(obj));
350 352
351 err = ____i915_gem_object_get_pages(obj);353 err = ____i915_gem_object_get_pages(obj);
352 if (err)354 if (err) {
353 goto err_unlock;355 ptr = ERR_PTR(err);
356 goto out_unlock;
357 }
354 358
355 smp_mb__before_atomic();359 smp_mb__before_atomic();
356 }360 }
@@ -362,7 +366,7 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj, · 5 unmodified lines
362 ptr = page_unpack_bits(obj->mm.mapping, &has_type);366 ptr = page_unpack_bits(obj->mm.mapping, &has_type);
363 if (ptr && has_type != type) {367 if (ptr && has_type != type) {
364 if (pinned) {368 if (pinned) {
365 err = -EBUSY;369 ptr = ERR_PTR(-EBUSY);
366 goto err_unpin;370 goto err_unpin;
367 }371 }
368 372
@@ -374,15 +378,13 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj, · 5 unmodified lines
374 if (!ptr) {378 if (!ptr) {
375 if (GEM_WARN_ON(type == I915_MAP_WC &&379 if (GEM_WARN_ON(type == I915_MAP_WC &&
376 !static_cpu_has(X86_FEATURE_PAT)))380 !static_cpu_has(X86_FEATURE_PAT)))
377 ptr = NULL;381 ptr = ERR_PTR(-ENODEV);
378 else if (i915_gem_object_has_struct_page(obj))382 else if (i915_gem_object_has_struct_page(obj))
379 ptr = i915_gem_object_map_page(obj, type);383 ptr = i915_gem_object_map_page(obj, type);
380 else384 else
381 ptr = i915_gem_object_map_pfn(obj, type);385 ptr = i915_gem_object_map_pfn(obj, type);
382 if (!ptr) {386 if (IS_ERR(ptr))
383 err = -ENOMEM;
384 goto err_unpin;387 goto err_unpin;
385 }
386 388
387 obj->mm.mapping = page_pack_bits(ptr, type);389 obj->mm.mapping = page_pack_bits(ptr, type);
388 }390 }
@@ -393,8 +395,6 @@ out_unlock: · 4 unmodified lines
393 395
394err_unpin:396err_unpin:
395 atomic_dec(&obj->mm.pages_pin_count);397 atomic_dec(&obj->mm.pages_pin_count);
396err_unlock:
397 ptr = ERR_PTR(err);
398 goto out_unlock;398 goto out_unlock;
399}399}
400 400
drivers/gpu/drm/i915/gem/i915_gem_region.c+20−16
@@ -22,6 +22,7 @@ i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj, · 21 unmodified lines
22int22int
23i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)23i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)
24{24{
25 const u64 max_segment = i915_sg_segment_size();
25 struct intel_memory_region *mem = obj->mm.region;26 struct intel_memory_region *mem = obj->mm.region;
26 struct list_head *blocks = &obj->mm.blocks;27 struct list_head *blocks = &obj->mm.blocks;
27 resource_size_t size = obj->base.size;28 resource_size_t size = obj->base.size;
@@ -37,7 +38,7 @@ i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj) · 9 unmodified lines
37 if (!st)38 if (!st)
38 return -ENOMEM;39 return -ENOMEM;
39 40
40 if (sg_alloc_table(st, size >> ilog2(mem->mm.chunk_size), GFP_KERNEL)) {41 if (sg_alloc_table(st, size >> PAGE_SHIFT, GFP_KERNEL)) {
41 kfree(st);42 kfree(st);
42 return -ENOMEM;43 return -ENOMEM;
43 }44 }
@@ -64,27 +65,30 @@ i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj) · 20 unmodified lines
64 i915_buddy_block_size(&mem->mm, block));65 i915_buddy_block_size(&mem->mm, block));
65 offset = i915_buddy_block_offset(block);66 offset = i915_buddy_block_offset(block);
66 67
67 GEM_BUG_ON(overflows_type(block_size, sg->length));68 while (block_size) {
69 u64 len;
68 70
69 if (offset != prev_end ||71 if (offset != prev_end || sg->length >= max_segment) {
70 add_overflows_t(typeof(sg->length), sg->length, block_size)) {72 if (st->nents) {
71 if (st->nents) {73 sg_page_sizes |= sg->length;
72 sg_page_sizes |= sg->length;74 sg = __sg_next(sg);
73 sg = __sg_next(sg);75 }
76
77 sg_dma_address(sg) = mem->region.start + offset;
78 sg_dma_len(sg) = 0;
79 sg->length = 0;
80 st->nents++;
74 }81 }
75 82
76 sg_dma_address(sg) = mem->region.start + offset;83 len = min(block_size, max_segment - sg->length);
77 sg_dma_len(sg) = block_size;84 sg->length += len;
85 sg_dma_len(sg) += len;
78 86
79 sg->length = block_size;87 offset += len;
88 block_size -= len;
80 89
81 st->nents++;90 prev_end = offset;
82 } else {
83 sg->length += block_size;
84 sg_dma_len(sg) += block_size;
85 }91 }
86
87 prev_end = offset + block_size;
88 }92 }
89 93
90 sg_page_sizes |= sg->length;94 sg_page_sizes |= sg->length;
drivers/gpu/drm/i915/gem/i915_gem_shrinker.c+3−0
@@ -15,6 +15,7 @@ · 14 unmodified lines
15 15
16#include "gt/intel_gt_requests.h"16#include "gt/intel_gt_requests.h"
17 17
18#include "dma_resv_utils.h"
18#include "i915_trace.h"19#include "i915_trace.h"
19 20
20static bool swap_available(void)21static bool swap_available(void)
@@ -209,6 +210,8 @@ i915_gem_shrink(struct drm_i915_private *i915, · 188 unmodified lines
209 mutex_unlock(&obj->mm.lock);210 mutex_unlock(&obj->mm.lock);
210 }211 }
211 212
213 dma_resv_prune(obj->base.resv);
214
212 scanned += obj->base.size >> PAGE_SHIFT;215 scanned += obj->base.size >> PAGE_SHIFT;
213 i915_gem_object_put(obj);216 i915_gem_object_put(obj);
214 217
drivers/gpu/drm/i915/gem/i915_gem_stolen.c+2−3
@@ -608,11 +608,10 @@ i915_gem_object_release_stolen(struct drm_i915_gem_object *obj) · 607 unmodified lines
608 struct drm_mm_node *stolen = fetch_and_zero(&obj->stolen);608 struct drm_mm_node *stolen = fetch_and_zero(&obj->stolen);
609 609
610 GEM_BUG_ON(!stolen);610 GEM_BUG_ON(!stolen);
611
612 i915_gem_object_release_memory_region(obj);
613
614 i915_gem_stolen_remove_node(i915, stolen);611 i915_gem_stolen_remove_node(i915, stolen);
615 kfree(stolen);612 kfree(stolen);
613
614 i915_gem_object_release_memory_region(obj);
616}615}
617 616
618static const struct drm_i915_gem_object_ops i915_gem_object_stolen_ops = {617static const struct drm_i915_gem_object_ops i915_gem_object_stolen_ops = {
drivers/gpu/drm/i915/gem/i915_gem_wait.c+3−5
@@ -9,6 +9,7 @@ · 8 unmodified lines
9 9
10#include "gt/intel_engine.h"10#include "gt/intel_engine.h"
11 11
12#include "dma_resv_utils.h"
12#include "i915_gem_ioctls.h"13#include "i915_gem_ioctls.h"
13#include "i915_gem_object.h"14#include "i915_gem_object.h"
14 15
@@ -84,11 +85,8 @@ i915_gem_object_wait_reservation(struct dma_resv *resv, · 69 unmodified lines
84 * Opportunistically prune the fences iff we know they have *all* been85 * Opportunistically prune the fences iff we know they have *all* been
85 * signaled.86 * signaled.
86 */87 */
87 if (prune_fences && dma_resv_trylock(resv)) {88 if (prune_fences)
88 if (dma_resv_test_signaled_rcu(resv, true))89 dma_resv_prune(resv);
89 dma_resv_add_excl_fence(resv, NULL);
90 dma_resv_unlock(resv);
91 }
92 90
93 return timeout;91 return timeout;
94}92}
drivers/gpu/drm/i915/gem/selftests/huge_gem_object.c+1−1
@@ -27,7 +27,7 @@ static void huge_free_pages(struct drm_i915_gem_object *obj, · 26 unmodified lines
27 27
28static int huge_get_pages(struct drm_i915_gem_object *obj)28static int huge_get_pages(struct drm_i915_gem_object *obj)
29{29{
30#define GFP (GFP_KERNEL | __GFP_NOWARN | __GFP_NORETRY)30#define GFP (GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL)
31 const unsigned long nreal = obj->scratch / PAGE_SIZE;31 const unsigned long nreal = obj->scratch / PAGE_SIZE;
32 const unsigned long npages = obj->base.size / PAGE_SIZE;32 const unsigned long npages = obj->base.size / PAGE_SIZE;
33 struct scatterlist *sg, *src, *end;33 struct scatterlist *sg, *src, *end;
drivers/gpu/drm/i915/gem/selftests/huge_pages.c+22−0
@@ -368,6 +368,27 @@ static int igt_check_page_sizes(struct i915_vma *vma) · 367 unmodified lines
368 err = -EINVAL;368 err = -EINVAL;
369 }369 }
370 370
371 /*
372 * The dma-api is like a box of chocolates when it comes to the
373 * alignment of dma addresses, however for LMEM we have total control
374 * and so can guarantee alignment, likewise when we allocate our blocks
375 * they should appear in descending order, and if we know that we align
376 * to the largest page size for the GTT address, we should be able to
377 * assert that if we see 2M physical pages then we should also get 2M
378 * GTT pages. If we don't then something might be wrong in our
379 * construction of the backing pages.
380 *
381 * Maintaining alignment is required to utilise huge pages in the ppGGT.
382 */
383 if (i915_gem_object_is_lmem(obj) &&
384 IS_ALIGNED(vma->node.start, SZ_2M) &&
385 vma->page_sizes.sg & SZ_2M &&
386 vma->page_sizes.gtt < SZ_2M) {
387 pr_err("gtt pages mismatch for LMEM, expected 2M GTT pages, sg(%u), gtt(%u)\n",
388 vma->page_sizes.sg, vma->page_sizes.gtt);
389 err = -EINVAL;
390 }
391
371 if (obj->mm.page_sizes.gtt) {392 if (obj->mm.page_sizes.gtt) {
372 pr_err("obj->page_sizes.gtt(%u) should never be set\n",393 pr_err("obj->page_sizes.gtt(%u) should never be set\n",
373 obj->mm.page_sizes.gtt);394 obj->mm.page_sizes.gtt);
@@ -1333,6 +1354,7 @@ static int igt_ppgtt_sanity_check(void *arg) · 959 unmodified lines
1333 unsigned int flags;1354 unsigned int flags;
1334 } backends[] = {1355 } backends[] = {
1335 { igt_create_system, 0, },1356 { igt_create_system, 0, },
1357 { igt_create_local, 0, },
1336 { igt_create_local, I915_BO_ALLOC_CONTIGUOUS, },1358 { igt_create_local, I915_BO_ALLOC_CONTIGUOUS, },
1337 };1359 };
1338 struct {1360 struct {
drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c+1−3
@@ -20,13 +20,11 @@ static int __igt_client_fill(struct intel_engine_cs *engine) · 19 unmodified lines
20{20{
21 struct intel_context *ce = engine->kernel_context;21 struct intel_context *ce = engine->kernel_context;
22 struct drm_i915_gem_object *obj;22 struct drm_i915_gem_object *obj;
23 struct rnd_state prng;23 I915_RND_STATE(prng);
24 IGT_TIMEOUT(end);24 IGT_TIMEOUT(end);
25 u32 *vaddr;25 u32 *vaddr;
26 int err = 0;26 int err = 0;
27 27
28 prandom_seed_state(&prng, i915_selftest.random_seed);
29
30 intel_engine_pm_get(engine);28 intel_engine_pm_get(engine);
31 do {29 do {
32 const u32 max_block_size = S16_MAX * PAGE_SIZE;30 const u32 max_block_size = S16_MAX * PAGE_SIZE;
drivers/gpu/drm/i915/gem/selftests/i915_gem_coherency.c+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
7#include <linux/prime_numbers.h>7#include <linux/prime_numbers.h>
8 8
9#include "gt/intel_engine_pm.h"9#include "gt/intel_engine_pm.h"
10#include "gt/intel_gpu_commands.h"
10#include "gt/intel_gt.h"11#include "gt/intel_gt.h"
11#include "gt/intel_gt_pm.h"12#include "gt/intel_gt_pm.h"
12#include "gt/intel_ring.h"13#include "gt/intel_ring.h"
drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c+1−0
@@ -7,6 +7,7 @@ · 6 unmodified lines
7#include <linux/prime_numbers.h>7#include <linux/prime_numbers.h>
8 8
9#include "gt/intel_engine_pm.h"9#include "gt/intel_engine_pm.h"
10#include "gt/intel_gpu_commands.h"
10#include "gt/intel_gt.h"11#include "gt/intel_gt.h"
11#include "gt/intel_gt_pm.h"12#include "gt/intel_gt_pm.h"
12#include "gem/i915_gem_region.h"13#include "gem/i915_gem_region.h"
drivers/gpu/drm/i915/gem/selftests/igt_gem_utils.c+1−0
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include "gem/i915_gem_context.h"9#include "gem/i915_gem_context.h"
10#include "gem/i915_gem_pm.h"10#include "gem/i915_gem_pm.h"
11#include "gt/intel_context.h"11#include "gt/intel_context.h"
12#include "gt/intel_gpu_commands.h"
12#include "gt/intel_gt.h"13#include "gt/intel_gt.h"
13#include "i915_vma.h"14#include "i915_vma.h"
14#include "i915_drv.h"15#include "i915_drv.h"
drivers/gpu/drm/i915/gt/debugfs_gt_pm.c+15−12
@@ -11,6 +11,7 @@ · 10 unmodified lines
11#include "i915_drv.h"11#include "i915_drv.h"
12#include "intel_gt.h"12#include "intel_gt.h"
13#include "intel_gt_clock_utils.h"13#include "intel_gt_clock_utils.h"
14#include "intel_gt_pm.h"
14#include "intel_llc.h"15#include "intel_llc.h"
15#include "intel_rc6.h"16#include "intel_rc6.h"
16#include "intel_rps.h"17#include "intel_rps.h"
@@ -403,34 +404,34 @@ static int frequency_show(struct seq_file *m, void *unused) · 386 unmodified lines
403 seq_printf(m, "RPDECLIMIT: 0x%08x\n", rpdeclimit);404 seq_printf(m, "RPDECLIMIT: 0x%08x\n", rpdeclimit);
404 seq_printf(m, "RPNSWREQ: %dMHz\n", reqf);405 seq_printf(m, "RPNSWREQ: %dMHz\n", reqf);
405 seq_printf(m, "CAGF: %dMHz\n", cagf);406 seq_printf(m, "CAGF: %dMHz\n", cagf);
406 seq_printf(m, "RP CUR UP EI: %d (%dns)\n",407 seq_printf(m, "RP CUR UP EI: %d (%lldns)\n",
407 rpcurupei,408 rpcurupei,
408 intel_gt_pm_interval_to_ns(gt, rpcurupei));409 intel_gt_pm_interval_to_ns(gt, rpcurupei));
409 seq_printf(m, "RP CUR UP: %d (%dns)\n",410 seq_printf(m, "RP CUR UP: %d (%lldns)\n",
410 rpcurup, intel_gt_pm_interval_to_ns(gt, rpcurup));411 rpcurup, intel_gt_pm_interval_to_ns(gt, rpcurup));
411 seq_printf(m, "RP PREV UP: %d (%dns)\n",412 seq_printf(m, "RP PREV UP: %d (%lldns)\n",
412 rpprevup, intel_gt_pm_interval_to_ns(gt, rpprevup));413 rpprevup, intel_gt_pm_interval_to_ns(gt, rpprevup));
413 seq_printf(m, "Up threshold: %d%%\n",414 seq_printf(m, "Up threshold: %d%%\n",
414 rps->power.up_threshold);415 rps->power.up_threshold);
415 seq_printf(m, "RP UP EI: %d (%dns)\n",416 seq_printf(m, "RP UP EI: %d (%lldns)\n",
416 rpupei, intel_gt_pm_interval_to_ns(gt, rpupei));417 rpupei, intel_gt_pm_interval_to_ns(gt, rpupei));
417 seq_printf(m, "RP UP THRESHOLD: %d (%dns)\n",418 seq_printf(m, "RP UP THRESHOLD: %d (%lldns)\n",
418 rpupt, intel_gt_pm_interval_to_ns(gt, rpupt));419 rpupt, intel_gt_pm_interval_to_ns(gt, rpupt));
419 420
420 seq_printf(m, "RP CUR DOWN EI: %d (%dns)\n",421 seq_printf(m, "RP CUR DOWN EI: %d (%lldns)\n",
421 rpcurdownei,422 rpcurdownei,
422 intel_gt_pm_interval_to_ns(gt, rpcurdownei));423 intel_gt_pm_interval_to_ns(gt, rpcurdownei));
423 seq_printf(m, "RP CUR DOWN: %d (%dns)\n",424 seq_printf(m, "RP CUR DOWN: %d (%lldns)\n",
424 rpcurdown,425 rpcurdown,
425 intel_gt_pm_interval_to_ns(gt, rpcurdown));426 intel_gt_pm_interval_to_ns(gt, rpcurdown));
426 seq_printf(m, "RP PREV DOWN: %d (%dns)\n",427 seq_printf(m, "RP PREV DOWN: %d (%lldns)\n",
427 rpprevdown,428 rpprevdown,
428 intel_gt_pm_interval_to_ns(gt, rpprevdown));429 intel_gt_pm_interval_to_ns(gt, rpprevdown));
429 seq_printf(m, "Down threshold: %d%%\n",430 seq_printf(m, "Down threshold: %d%%\n",
430 rps->power.down_threshold);431 rps->power.down_threshold);
431 seq_printf(m, "RP DOWN EI: %d (%dns)\n",432 seq_printf(m, "RP DOWN EI: %d (%lldns)\n",
432 rpdownei, intel_gt_pm_interval_to_ns(gt, rpdownei));433 rpdownei, intel_gt_pm_interval_to_ns(gt, rpdownei));
433 seq_printf(m, "RP DOWN THRESHOLD: %d (%dns)\n",434 seq_printf(m, "RP DOWN THRESHOLD: %d (%lldns)\n",
434 rpdownt, intel_gt_pm_interval_to_ns(gt, rpdownt));435 rpdownt, intel_gt_pm_interval_to_ns(gt, rpdownt));
435 436
436 max_freq = (IS_GEN9_LP(i915) ? rp_state_cap >> 0 :437 max_freq = (IS_GEN9_LP(i915) ? rp_state_cap >> 0 :
@@ -558,7 +559,9 @@ static int rps_boost_show(struct seq_file *m, void *data) · 121 unmodified lines
558 559
559 seq_printf(m, "RPS enabled? %s\n", yesno(intel_rps_is_enabled(rps)));560 seq_printf(m, "RPS enabled? %s\n", yesno(intel_rps_is_enabled(rps)));
560 seq_printf(m, "RPS active? %s\n", yesno(intel_rps_is_active(rps)));561 seq_printf(m, "RPS active? %s\n", yesno(intel_rps_is_active(rps)));
561 seq_printf(m, "GPU busy? %s\n", yesno(gt->awake));562 seq_printf(m, "GPU busy? %s, %llums\n",
563 yesno(gt->awake),
564 ktime_to_ms(intel_gt_get_awake_time(gt)));
562 seq_printf(m, "Boosts outstanding? %d\n",565 seq_printf(m, "Boosts outstanding? %d\n",
563 atomic_read(&rps->num_waiters));566 atomic_read(&rps->num_waiters));
564 seq_printf(m, "Interactive? %d\n", READ_ONCE(rps->power.interactive));567 seq_printf(m, "Interactive? %d\n", READ_ONCE(rps->power.interactive));
@@ -575,7 +578,7 @@ static int rps_boost_show(struct seq_file *m, void *data) · 10 unmodified lines
575 intel_gpu_freq(rps, rps->efficient_freq),578 intel_gpu_freq(rps, rps->efficient_freq),
576 intel_gpu_freq(rps, rps->boost_freq));579 intel_gpu_freq(rps, rps->boost_freq));
577 580
578 seq_printf(m, "Wait boosts: %d\n", atomic_read(&rps->boosts));581 seq_printf(m, "Wait boosts: %d\n", READ_ONCE(rps->boosts));
579 582
580 if (INTEL_GEN(i915) >= 6 && intel_rps_is_active(rps)) {583 if (INTEL_GEN(i915) >= 6 && intel_rps_is_active(rps)) {
581 struct intel_uncore *uncore = gt->uncore;584 struct intel_uncore *uncore = gt->uncore;
drivers/gpu/drm/i915/gt/gen7_renderclear.c+94−63
@@ -7,8 +7,6 @@ · 6 unmodified lines
7#include "i915_drv.h"7#include "i915_drv.h"
8#include "intel_gpu_commands.h"8#include "intel_gpu_commands.h"
9 9
10#define MAX_URB_ENTRIES 64
11#define STATE_SIZE (4 * 1024)
12#define GT3_INLINE_DATA_DELAYS 0x1E0010#define GT3_INLINE_DATA_DELAYS 0x1E00
13#define batch_advance(Y, CS) GEM_BUG_ON((Y)->end != (CS))11#define batch_advance(Y, CS) GEM_BUG_ON((Y)->end != (CS))
14 12
@@ -34,38 +32,59 @@ struct batch_chunk { · 19 unmodified lines
34};32};
35 33
36struct batch_vals {34struct batch_vals {
37 u32 max_primitives;35 u32 max_threads;
38 u32 max_urb_entries;
39 u32 cmd_size;
40 u32 state_size;
41 u32 state_start;36 u32 state_start;
42 u32 batch_size;37 u32 surface_start;
43 u32 surface_height;38 u32 surface_height;
44 u32 surface_width;39 u32 surface_width;
45 u32 scratch_size;40 u32 size;
46 u32 max_size;
47};41};
48 42
43static inline int num_primitives(const struct batch_vals *bv)
44{
45 /*
46 * We need to saturate the GPU with work in order to dispatch
47 * a shader on every HW thread, and clear the thread-local registers.
48 * In short, we have to dispatch work faster than the shaders can
49 * run in order to fill the EU and occupy each HW thread.
50 */
51 return bv->max_threads;
52}
53
49static void54static void
50batch_get_defaults(struct drm_i915_private *i915, struct batch_vals *bv)55batch_get_defaults(struct drm_i915_private *i915, struct batch_vals *bv)
51{56{
52 if (IS_HASWELL(i915)) {57 if (IS_HASWELL(i915)) {
53 bv->max_primitives = 280;58 switch (INTEL_INFO(i915)->gt) {
54 bv->max_urb_entries = MAX_URB_ENTRIES;59 default:
60 case 1:
61 bv->max_threads = 70;
62 break;
63 case 2:
64 bv->max_threads = 140;
65 break;
66 case 3:
67 bv->max_threads = 280;
68 break;
69 }
55 bv->surface_height = 16 * 16;70 bv->surface_height = 16 * 16;
56 bv->surface_width = 32 * 2 * 16;71 bv->surface_width = 32 * 2 * 16;
57 } else {72 } else {
58 bv->max_primitives = 128;73 switch (INTEL_INFO(i915)->gt) {
59 bv->max_urb_entries = MAX_URB_ENTRIES / 2;74 default:
75 case 1: /* including vlv */
76 bv->max_threads = 36;
77 break;
78 case 2:
79 bv->max_threads = 128;
80 break;
81 }
60 bv->surface_height = 16 * 8;82 bv->surface_height = 16 * 8;
61 bv->surface_width = 32 * 16;83 bv->surface_width = 32 * 16;
62 }84 }
63 bv->cmd_size = bv->max_primitives * 4096;85 bv->state_start = round_up(SZ_1K + num_primitives(bv) * 64, SZ_4K);
64 bv->state_size = STATE_SIZE;86 bv->surface_start = bv->state_start + SZ_4K;
65 bv->state_start = bv->cmd_size;87 bv->size = bv->surface_start + bv->surface_height * bv->surface_width;
66 bv->batch_size = bv->cmd_size + bv->state_size;
67 bv->scratch_size = bv->surface_height * bv->surface_width;
68 bv->max_size = bv->batch_size + bv->scratch_size;
69}88}
70 89
71static void batch_init(struct batch_chunk *bc,90static void batch_init(struct batch_chunk *bc,
@@ -155,7 +174,8 @@ static u32 · 83 unmodified lines
155gen7_fill_binding_table(struct batch_chunk *state,174gen7_fill_binding_table(struct batch_chunk *state,
156 const struct batch_vals *bv)175 const struct batch_vals *bv)
157{176{
158 u32 surface_start = gen7_fill_surface_state(state, bv->batch_size, bv);177 u32 surface_start =
178 gen7_fill_surface_state(state, bv->surface_start, bv);
159 u32 *cs = batch_alloc_items(state, 32, 8);179 u32 *cs = batch_alloc_items(state, 32, 8);
160 u32 offset = batch_offset(state, cs);180 u32 offset = batch_offset(state, cs);
161 181
@@ -214,9 +234,9 @@ static void · 52 unmodified lines
214gen7_emit_state_base_address(struct batch_chunk *batch,234gen7_emit_state_base_address(struct batch_chunk *batch,
215 u32 surface_state_base)235 u32 surface_state_base)
216{236{
217 u32 *cs = batch_alloc_items(batch, 0, 12);237 u32 *cs = batch_alloc_items(batch, 0, 10);
218 238
219 *cs++ = STATE_BASE_ADDRESS | (12 - 2);239 *cs++ = STATE_BASE_ADDRESS | (10 - 2);
220 /* general */240 /* general */
221 *cs++ = batch_addr(batch) | BASE_ADDRESS_MODIFY;241 *cs++ = batch_addr(batch) | BASE_ADDRESS_MODIFY;
222 /* surface */242 /* surface */
@@ -233,8 +253,6 @@ gen7_emit_state_base_address(struct batch_chunk *batch, · 10 unmodified lines
233 *cs++ = BASE_ADDRESS_MODIFY;253 *cs++ = BASE_ADDRESS_MODIFY;
234 *cs++ = 0;254 *cs++ = 0;
235 *cs++ = BASE_ADDRESS_MODIFY;255 *cs++ = BASE_ADDRESS_MODIFY;
236 *cs++ = 0;
237 *cs++ = 0;
238 batch_advance(batch, cs);256 batch_advance(batch, cs);
239}257}
240 258
@@ -244,8 +262,7 @@ gen7_emit_vfe_state(struct batch_chunk *batch, · 3 unmodified lines
244 u32 urb_size, u32 curbe_size,262 u32 urb_size, u32 curbe_size,
245 u32 mode)263 u32 mode)
246{264{
247 u32 urb_entries = bv->max_urb_entries;265 u32 threads = bv->max_threads - 1;
248 u32 threads = bv->max_primitives - 1;
249 u32 *cs = batch_alloc_items(batch, 32, 8);266 u32 *cs = batch_alloc_items(batch, 32, 8);
250 267
251 *cs++ = MEDIA_VFE_STATE | (8 - 2);268 *cs++ = MEDIA_VFE_STATE | (8 - 2);
@@ -254,7 +271,7 @@ gen7_emit_vfe_state(struct batch_chunk *batch, · 2 unmodified lines
254 *cs++ = 0;271 *cs++ = 0;
255 272
256 /* number of threads & urb entries for GPGPU vs Media Mode */273 /* number of threads & urb entries for GPGPU vs Media Mode */
257 *cs++ = threads << 16 | urb_entries << 8 | mode << 2;274 *cs++ = threads << 16 | 1 << 8 | mode << 2;
258 275
259 *cs++ = 0;276 *cs++ = 0;
260 277
@@ -293,17 +310,12 @@ gen7_emit_media_object(struct batch_chunk *batch, · 32 unmodified lines
293{310{
294 unsigned int x_offset = (media_object_index % 16) * 64;311 unsigned int x_offset = (media_object_index % 16) * 64;
295 unsigned int y_offset = (media_object_index / 16) * 16;312 unsigned int y_offset = (media_object_index / 16) * 16;
296 unsigned int inline_data_size;313 unsigned int pkt = 6 + 3;
297 unsigned int media_batch_size;
298 unsigned int i;
299 u32 *cs;314 u32 *cs;
300 315
301 inline_data_size = 112 * 8;316 cs = batch_alloc_items(batch, 8, pkt);
302 media_batch_size = inline_data_size + 6;
303
304 cs = batch_alloc_items(batch, 8, media_batch_size);
305 317
306 *cs++ = MEDIA_OBJECT | (media_batch_size - 2);318 *cs++ = MEDIA_OBJECT | (pkt - 2);
307 319
308 /* interface descriptor offset */320 /* interface descriptor offset */
309 *cs++ = 0;321 *cs++ = 0;
@@ -317,25 +329,44 @@ gen7_emit_media_object(struct batch_chunk *batch, · 7 unmodified lines
317 *cs++ = 0;329 *cs++ = 0;
318 330
319 /* inline */331 /* inline */
320 *cs++ = (y_offset << 16) | (x_offset);332 *cs++ = y_offset << 16 | x_offset;
321 *cs++ = 0;333 *cs++ = 0;
322 *cs++ = GT3_INLINE_DATA_DELAYS;334 *cs++ = GT3_INLINE_DATA_DELAYS;
323 for (i = 3; i < inline_data_size; i++)
324 *cs++ = 0;
325 335
326 batch_advance(batch, cs);336 batch_advance(batch, cs);
327}337}
328 338
329static void gen7_emit_pipeline_flush(struct batch_chunk *batch)339static void gen7_emit_pipeline_flush(struct batch_chunk *batch)
330{340{
331 u32 *cs = batch_alloc_items(batch, 0, 5);341 u32 *cs = batch_alloc_items(batch, 0, 4);
332 342
333 *cs++ = GFX_OP_PIPE_CONTROL(5);343 *cs++ = GFX_OP_PIPE_CONTROL(4);
334 *cs++ = PIPE_CONTROL_STATE_CACHE_INVALIDATE |344 *cs++ = PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
335 PIPE_CONTROL_GLOBAL_GTT_IVB;345 PIPE_CONTROL_DEPTH_CACHE_FLUSH |
346 PIPE_CONTROL_DC_FLUSH_ENABLE |
347 PIPE_CONTROL_CS_STALL;
336 *cs++ = 0;348 *cs++ = 0;
337 *cs++ = 0;349 *cs++ = 0;
350
351 batch_advance(batch, cs);
352}
353
354static void gen7_emit_pipeline_invalidate(struct batch_chunk *batch)
355{
356 u32 *cs = batch_alloc_items(batch, 0, 8);
357
358 /* ivb: Stall before STATE_CACHE_INVALIDATE */
359 *cs++ = GFX_OP_PIPE_CONTROL(4);
360 *cs++ = PIPE_CONTROL_STALL_AT_SCOREBOARD |
361 PIPE_CONTROL_CS_STALL;
362 *cs++ = 0;
363 *cs++ = 0;
364
365 *cs++ = GFX_OP_PIPE_CONTROL(4);
366 *cs++ = PIPE_CONTROL_STATE_CACHE_INVALIDATE;
338 *cs++ = 0;367 *cs++ = 0;
368 *cs++ = 0;
369
339 batch_advance(batch, cs);370 batch_advance(batch, cs);
340}371}
341 372
@@ -344,34 +375,34 @@ static void emit_batch(struct i915_vma * const vma, · 2 unmodified lines
344 const struct batch_vals *bv)375 const struct batch_vals *bv)
345{376{
346 struct drm_i915_private *i915 = vma->vm->i915;377 struct drm_i915_private *i915 = vma->vm->i915;
347 unsigned int desc_count = 64;378 const unsigned int desc_count = 1;
348 const u32 urb_size = 112;379 const unsigned int urb_size = 1;
349 struct batch_chunk cmds, state;380 struct batch_chunk cmds, state;
350 u32 interface_descriptor;381 u32 descriptors;
351 unsigned int i;382 unsigned int i;
352 383
353 batch_init(&cmds, vma, start, 0, bv->cmd_size);384 batch_init(&cmds, vma, start, 0, bv->state_start);
354 batch_init(&state, vma, start, bv->state_start, bv->state_size);385 batch_init(&state, vma, start, bv->state_start, SZ_4K);
355 386
356 interface_descriptor =387 descriptors = gen7_fill_interface_descriptor(&state, bv,
357 gen7_fill_interface_descriptor(&state, bv,388 IS_HASWELL(i915) ?
358 IS_HASWELL(i915) ?389 &cb_kernel_hsw :
359 &cb_kernel_hsw :390 &cb_kernel_ivb,
360 &cb_kernel_ivb,391 desc_count);
361 desc_count);392
362 gen7_emit_pipeline_flush(&cmds);393 gen7_emit_pipeline_invalidate(&cmds);
363 batch_add(&cmds, PIPELINE_SELECT | PIPELINE_SELECT_MEDIA);394 batch_add(&cmds, PIPELINE_SELECT | PIPELINE_SELECT_MEDIA);
364 batch_add(&cmds, MI_NOOP);395 batch_add(&cmds, MI_NOOP);
365 gen7_emit_state_base_address(&cmds, interface_descriptor);396 gen7_emit_pipeline_invalidate(&cmds);
397
366 gen7_emit_pipeline_flush(&cmds);398 gen7_emit_pipeline_flush(&cmds);
399 gen7_emit_state_base_address(&cmds, descriptors);
400 gen7_emit_pipeline_invalidate(&cmds);
367 401
368 gen7_emit_vfe_state(&cmds, bv, urb_size - 1, 0, 0);402 gen7_emit_vfe_state(&cmds, bv, urb_size - 1, 0, 0);
403 gen7_emit_interface_descriptor_load(&cmds, descriptors, desc_count);
369 404
370 gen7_emit_interface_descriptor_load(&cmds,405 for (i = 0; i < num_primitives(bv); i++)
371 interface_descriptor,
372 desc_count);
373
374 for (i = 0; i < bv->max_primitives; i++)
375 gen7_emit_media_object(&cmds, i);406 gen7_emit_media_object(&cmds, i);
376 407
377 batch_add(&cmds, MI_BATCH_BUFFER_END);408 batch_add(&cmds, MI_BATCH_BUFFER_END);
@@ -385,15 +416,15 @@ int gen7_setup_clear_gpr_bb(struct intel_engine_cs * const engine, · 7 unmodified lines
385 416
386 batch_get_defaults(engine->i915, &bv);417 batch_get_defaults(engine->i915, &bv);
387 if (!vma)418 if (!vma)
388 return bv.max_size;419 return bv.size;
389 420
390 GEM_BUG_ON(vma->obj->base.size < bv.max_size);421 GEM_BUG_ON(vma->obj->base.size < bv.size);
391 422
392 batch = i915_gem_object_pin_map(vma->obj, I915_MAP_WC);423 batch = i915_gem_object_pin_map(vma->obj, I915_MAP_WC);
393 if (IS_ERR(batch))424 if (IS_ERR(batch))
394 return PTR_ERR(batch);425 return PTR_ERR(batch);
395 426
396 emit_batch(vma, memset(batch, 0, bv.max_size), &bv);427 emit_batch(vma, memset(batch, 0, bv.size), &bv);
397 428
398 i915_gem_object_flush_map(vma->obj);429 i915_gem_object_flush_map(vma->obj);
399 __i915_gem_object_release_map(vma->obj);430 __i915_gem_object_release_map(vma->obj);
drivers/gpu/drm/i915/gt/gen8_engine_cs.c+633−0
@@ -0,0 +1,633 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright © 2014 Intel Corporation
4 */
5
6#include "gen8_engine_cs.h"
7#include "i915_drv.h"
8#include "intel_lrc.h"
9#include "intel_gpu_commands.h"
10#include "intel_ring.h"
11
12int gen8_emit_flush_rcs(struct i915_request *rq, u32 mode)
13{
14 bool vf_flush_wa = false, dc_flush_wa = false;
15 u32 *cs, flags = 0;
16 int len;
17
18 flags |= PIPE_CONTROL_CS_STALL;
19
20 if (mode & EMIT_FLUSH) {
21 flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
22 flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
23 flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
24 flags |= PIPE_CONTROL_FLUSH_ENABLE;
25 }
26
27 if (mode & EMIT_INVALIDATE) {
28 flags |= PIPE_CONTROL_TLB_INVALIDATE;
29 flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
30 flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
31 flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
32 flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
33 flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
34 flags |= PIPE_CONTROL_QW_WRITE;
35 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
36
37 /*
38 * On GEN9: before VF_CACHE_INVALIDATE we need to emit a NULL
39 * pipe control.
40 */
41 if (IS_GEN(rq->engine->i915, 9))
42 vf_flush_wa = true;
43
44 /* WaForGAMHang:kbl */
45 if (IS_KBL_GT_REVID(rq->engine->i915, 0, KBL_REVID_B0))
46 dc_flush_wa = true;
47 }
48
49 len = 6;
50
51 if (vf_flush_wa)
52 len += 6;
53
54 if (dc_flush_wa)
55 len += 12;
56
57 cs = intel_ring_begin(rq, len);
58 if (IS_ERR(cs))
59 return PTR_ERR(cs);
60
61 if (vf_flush_wa)
62 cs = gen8_emit_pipe_control(cs, 0, 0);
63
64 if (dc_flush_wa)
65 cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_DC_FLUSH_ENABLE,
66 0);
67
68 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
69
70 if (dc_flush_wa)
71 cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_CS_STALL, 0);
72
73 intel_ring_advance(rq, cs);
74
75 return 0;
76}
77
78int gen8_emit_flush_xcs(struct i915_request *rq, u32 mode)
79{
80 u32 cmd, *cs;
81
82 cs = intel_ring_begin(rq, 4);
83 if (IS_ERR(cs))
84 return PTR_ERR(cs);
85
86 cmd = MI_FLUSH_DW + 1;
87
88 /*
89 * We always require a command barrier so that subsequent
90 * commands, such as breadcrumb interrupts, are strictly ordered
91 * wrt the contents of the write cache being flushed to memory
92 * (and thus being coherent from the CPU).
93 */
94 cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
95
96 if (mode & EMIT_INVALIDATE) {
97 cmd |= MI_INVALIDATE_TLB;
98 if (rq->engine->class == VIDEO_DECODE_CLASS)
99 cmd |= MI_INVALIDATE_BSD;
100 }
101
102 *cs++ = cmd;
103 *cs++ = LRC_PPHWSP_SCRATCH_ADDR;
104 *cs++ = 0; /* upper addr */
105 *cs++ = 0; /* value */
106 intel_ring_advance(rq, cs);
107
108 return 0;
109}
110
111int gen11_emit_flush_rcs(struct i915_request *rq, u32 mode)
112{
113 if (mode & EMIT_FLUSH) {
114 u32 *cs;
115 u32 flags = 0;
116
117 flags |= PIPE_CONTROL_CS_STALL;
118
119 flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
120 flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
121 flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
122 flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
123 flags |= PIPE_CONTROL_FLUSH_ENABLE;
124 flags |= PIPE_CONTROL_QW_WRITE;
125 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
126
127 cs = intel_ring_begin(rq, 6);
128 if (IS_ERR(cs))
129 return PTR_ERR(cs);
130
131 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
132 intel_ring_advance(rq, cs);
133 }
134
135 if (mode & EMIT_INVALIDATE) {
136 u32 *cs;
137 u32 flags = 0;
138
139 flags |= PIPE_CONTROL_CS_STALL;
140
141 flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
142 flags |= PIPE_CONTROL_TLB_INVALIDATE;
143 flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
144 flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
145 flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
146 flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
147 flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
148 flags |= PIPE_CONTROL_QW_WRITE;
149 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
150
151 cs = intel_ring_begin(rq, 6);
152 if (IS_ERR(cs))
153 return PTR_ERR(cs);
154
155 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
156 intel_ring_advance(rq, cs);
157 }
158
159 return 0;
160}
161
162static u32 preparser_disable(bool state)
163{
164 return MI_ARB_CHECK | 1 << 8 | state;
165}
166
167static i915_reg_t aux_inv_reg(const struct intel_engine_cs *engine)
168{
169 static const i915_reg_t vd[] = {
170 GEN12_VD0_AUX_NV,
171 GEN12_VD1_AUX_NV,
172 GEN12_VD2_AUX_NV,
173 GEN12_VD3_AUX_NV,
174 };
175
176 static const i915_reg_t ve[] = {
177 GEN12_VE0_AUX_NV,
178 GEN12_VE1_AUX_NV,
179 };
180
181 if (engine->class == VIDEO_DECODE_CLASS)
182 return vd[engine->instance];
183
184 if (engine->class == VIDEO_ENHANCEMENT_CLASS)
185 return ve[engine->instance];
186
187 GEM_BUG_ON("unknown aux_inv reg\n");
188 return INVALID_MMIO_REG;
189}
190
191static u32 *gen12_emit_aux_table_inv(const i915_reg_t inv_reg, u32 *cs)
192{
193 *cs++ = MI_LOAD_REGISTER_IMM(1);
194 *cs++ = i915_mmio_reg_offset(inv_reg);
195 *cs++ = AUX_INV;
196 *cs++ = MI_NOOP;
197
198 return cs;
199}
200
201int gen12_emit_flush_rcs(struct i915_request *rq, u32 mode)
202{
203 if (mode & EMIT_FLUSH) {
204 u32 flags = 0;
205 u32 *cs;
206
207 flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
208 flags |= PIPE_CONTROL_FLUSH_L3;
209 flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
210 flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
211 /* Wa_1409600907:tgl */
212 flags |= PIPE_CONTROL_DEPTH_STALL;
213 flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
214 flags |= PIPE_CONTROL_FLUSH_ENABLE;
215
216 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
217 flags |= PIPE_CONTROL_QW_WRITE;
218
219 flags |= PIPE_CONTROL_CS_STALL;
220
221 cs = intel_ring_begin(rq, 6);
222 if (IS_ERR(cs))
223 return PTR_ERR(cs);
224
225 cs = gen12_emit_pipe_control(cs,
226 PIPE_CONTROL0_HDC_PIPELINE_FLUSH,
227 flags, LRC_PPHWSP_SCRATCH_ADDR);
228 intel_ring_advance(rq, cs);
229 }
230
231 if (mode & EMIT_INVALIDATE) {
232 u32 flags = 0;
233 u32 *cs;
234
235 flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
236 flags |= PIPE_CONTROL_TLB_INVALIDATE;
237 flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
238 flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
239 flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
240 flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
241 flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
242
243 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
244 flags |= PIPE_CONTROL_QW_WRITE;
245
246 flags |= PIPE_CONTROL_CS_STALL;
247
248 cs = intel_ring_begin(rq, 8 + 4);
249 if (IS_ERR(cs))
250 return PTR_ERR(cs);
251
252 /*
253 * Prevent the pre-parser from skipping past the TLB
254 * invalidate and loading a stale page for the batch
255 * buffer / request payload.
256 */
257 *cs++ = preparser_disable(true);
258
259 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
260
261 /* hsdes: 1809175790 */
262 cs = gen12_emit_aux_table_inv(GEN12_GFX_CCS_AUX_NV, cs);
263
264 *cs++ = preparser_disable(false);
265 intel_ring_advance(rq, cs);
266 }
267
268 return 0;
269}
270
271int gen12_emit_flush_xcs(struct i915_request *rq, u32 mode)
272{
273 intel_engine_mask_t aux_inv = 0;
274 u32 cmd, *cs;
275
276 cmd = 4;
277 if (mode & EMIT_INVALIDATE)
278 cmd += 2;
279 if (mode & EMIT_INVALIDATE)
280 aux_inv = rq->engine->mask & ~BIT(BCS0);
281 if (aux_inv)
282 cmd += 2 * hweight8(aux_inv) + 2;
283
284 cs = intel_ring_begin(rq, cmd);
285 if (IS_ERR(cs))
286 return PTR_ERR(cs);
287
288 if (mode & EMIT_INVALIDATE)
289 *cs++ = preparser_disable(true);
290
291 cmd = MI_FLUSH_DW + 1;
292
293 /*
294 * We always require a command barrier so that subsequent
295 * commands, such as breadcrumb interrupts, are strictly ordered
296 * wrt the contents of the write cache being flushed to memory
297 * (and thus being coherent from the CPU).
298 */
299 cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
300
301 if (mode & EMIT_INVALIDATE) {
302 cmd |= MI_INVALIDATE_TLB;
303 if (rq->engine->class == VIDEO_DECODE_CLASS)
304 cmd |= MI_INVALIDATE_BSD;
305 }
306
307 *cs++ = cmd;
308 *cs++ = LRC_PPHWSP_SCRATCH_ADDR;
309 *cs++ = 0; /* upper addr */
310 *cs++ = 0; /* value */
311
312 if (aux_inv) { /* hsdes: 1809175790 */
313 struct intel_engine_cs *engine;
314 unsigned int tmp;
315
316 *cs++ = MI_LOAD_REGISTER_IMM(hweight8(aux_inv));
317 for_each_engine_masked(engine, rq->engine->gt,
318 aux_inv, tmp) {
319 *cs++ = i915_mmio_reg_offset(aux_inv_reg(engine));
320 *cs++ = AUX_INV;
321 }
322 *cs++ = MI_NOOP;
323 }
324
325 if (mode & EMIT_INVALIDATE)
326 *cs++ = preparser_disable(false);
327
328 intel_ring_advance(rq, cs);
329
330 return 0;
331}
332
333static inline u32 preempt_address(struct intel_engine_cs *engine)
334{
335 return (i915_ggtt_offset(engine->status_page.vma) +
336 I915_GEM_HWS_PREEMPT_ADDR);
337}
338
339static u32 hwsp_offset(const struct i915_request *rq)
340{
341 const struct intel_timeline_cacheline *cl;
342
343 /* Before the request is executed, the timeline/cachline is fixed */
344
345 cl = rcu_dereference_protected(rq->hwsp_cacheline, 1);
346 if (cl)
347 return cl->ggtt_offset;
348
349 return rcu_dereference_protected(rq->timeline, 1)->hwsp_offset;
350}
351
352int gen8_emit_init_breadcrumb(struct i915_request *rq)
353{
354 u32 *cs;
355
356 GEM_BUG_ON(i915_request_has_initial_breadcrumb(rq));
357 if (!i915_request_timeline(rq)->has_initial_breadcrumb)
358 return 0;
359
360 cs = intel_ring_begin(rq, 6);
361 if (IS_ERR(cs))
362 return PTR_ERR(cs);
363
364 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
365 *cs++ = hwsp_offset(rq);
366 *cs++ = 0;
367 *cs++ = rq->fence.seqno - 1;
368
369 /*
370 * Check if we have been preempted before we even get started.
371 *
372 * After this point i915_request_started() reports true, even if
373 * we get preempted and so are no longer running.
374 *
375 * i915_request_started() is used during preemption processing
376 * to decide if the request is currently inside the user payload
377 * or spinning on a kernel semaphore (or earlier). For no-preemption
378 * requests, we do allow preemption on the semaphore before the user
379 * payload, but do not allow preemption once the request is started.
380 *
381 * i915_request_started() is similarly used during GPU hangs to
382 * determine if the user's payload was guilty, and if so, the
383 * request is banned. Before the request is started, it is assumed
384 * to be unharmed and an innocent victim of another's hang.
385 */
386 *cs++ = MI_NOOP;
387 *cs++ = MI_ARB_CHECK;
388
389 intel_ring_advance(rq, cs);
390
391 /* Record the updated position of the request's payload */
392 rq->infix = intel_ring_offset(rq, cs);
393
394 __set_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags);
395
396 return 0;
397}
398
399int gen8_emit_bb_start_noarb(struct i915_request *rq,
400 u64 offset, u32 len,
401 const unsigned int flags)
402{
403 u32 *cs;
404
405 cs = intel_ring_begin(rq, 4);
406 if (IS_ERR(cs))
407 return PTR_ERR(cs);
408
409 /*
410 * WaDisableCtxRestoreArbitration:bdw,chv
411 *
412 * We don't need to perform MI_ARB_ENABLE as often as we do (in
413 * particular all the gen that do not need the w/a at all!), if we
414 * took care to make sure that on every switch into this context
415 * (both ordinary and for preemption) that arbitrartion was enabled
416 * we would be fine. However, for gen8 there is another w/a that
417 * requires us to not preempt inside GPGPU execution, so we keep
418 * arbitration disabled for gen8 batches. Arbitration will be
419 * re-enabled before we close the request
420 * (engine->emit_fini_breadcrumb).
421 */
422 *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
423
424 /* FIXME(BDW+): Address space and security selectors. */
425 *cs++ = MI_BATCH_BUFFER_START_GEN8 |
426 (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
427 *cs++ = lower_32_bits(offset);
428 *cs++ = upper_32_bits(offset);
429
430 intel_ring_advance(rq, cs);
431
432 return 0;
433}
434
435int gen8_emit_bb_start(struct i915_request *rq,
436 u64 offset, u32 len,
437 const unsigned int flags)
438{
439 u32 *cs;
440
441 if (unlikely(i915_request_has_nopreempt(rq)))
442 return gen8_emit_bb_start_noarb(rq, offset, len, flags);
443
444 cs = intel_ring_begin(rq, 6);
445 if (IS_ERR(cs))
446 return PTR_ERR(cs);
447
448 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
449
450 *cs++ = MI_BATCH_BUFFER_START_GEN8 |
451 (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
452 *cs++ = lower_32_bits(offset);
453 *cs++ = upper_32_bits(offset);
454
455 *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
456 *cs++ = MI_NOOP;
457
458 intel_ring_advance(rq, cs);
459
460 return 0;
461}
462
463static void assert_request_valid(struct i915_request *rq)
464{
465 struct intel_ring *ring __maybe_unused = rq->ring;
466
467 /* Can we unwind this request without appearing to go forwards? */
468 GEM_BUG_ON(intel_ring_direction(ring, rq->wa_tail, rq->head) <= 0);
469}
470
471/*
472 * Reserve space for 2 NOOPs at the end of each request to be
473 * used as a workaround for not being allowed to do lite
474 * restore with HEAD==TAIL (WaIdleLiteRestore).
475 */
476static u32 *gen8_emit_wa_tail(struct i915_request *rq, u32 *cs)
477{
478 /* Ensure there's always at least one preemption point per-request. */
479 *cs++ = MI_ARB_CHECK;
480 *cs++ = MI_NOOP;
481 rq->wa_tail = intel_ring_offset(rq, cs);
482
483 /* Check that entire request is less than half the ring */
484 assert_request_valid(rq);
485
486 return cs;
487}
488
489static u32 *emit_preempt_busywait(struct i915_request *rq, u32 *cs)
490{
491 *cs++ = MI_SEMAPHORE_WAIT |
492 MI_SEMAPHORE_GLOBAL_GTT |
493 MI_SEMAPHORE_POLL |
494 MI_SEMAPHORE_SAD_EQ_SDD;
495 *cs++ = 0;
496 *cs++ = preempt_address(rq->engine);
497 *cs++ = 0;
498
499 return cs;
500}
501
502static __always_inline u32*
503gen8_emit_fini_breadcrumb_tail(struct i915_request *rq, u32 *cs)
504{
505 *cs++ = MI_USER_INTERRUPT;
506
507 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
508 if (intel_engine_has_semaphores(rq->engine))
509 cs = emit_preempt_busywait(rq, cs);
510
511 rq->tail = intel_ring_offset(rq, cs);
512 assert_ring_tail_valid(rq->ring, rq->tail);
513
514 return gen8_emit_wa_tail(rq, cs);
515}
516
517static u32 *emit_xcs_breadcrumb(struct i915_request *rq, u32 *cs)
518{
519 return gen8_emit_ggtt_write(cs, rq->fence.seqno, hwsp_offset(rq), 0);
520}
521
522u32 *gen8_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs)
523{
524 return gen8_emit_fini_breadcrumb_tail(rq, emit_xcs_breadcrumb(rq, cs));
525}
526
527u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs)
528{
529 cs = gen8_emit_pipe_control(cs,
530 PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
531 PIPE_CONTROL_DEPTH_CACHE_FLUSH |
532 PIPE_CONTROL_DC_FLUSH_ENABLE,
533 0);
534
535 /* XXX flush+write+CS_STALL all in one upsets gem_concurrent_blt:kbl */
536 cs = gen8_emit_ggtt_write_rcs(cs,
537 rq->fence.seqno,
538 hwsp_offset(rq),
539 PIPE_CONTROL_FLUSH_ENABLE |
540 PIPE_CONTROL_CS_STALL);
541
542 return gen8_emit_fini_breadcrumb_tail(rq, cs);
543}
544
545u32 *gen11_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs)
546{
547 cs = gen8_emit_ggtt_write_rcs(cs,
548 rq->fence.seqno,
549 hwsp_offset(rq),
550 PIPE_CONTROL_CS_STALL |
551 PIPE_CONTROL_TILE_CACHE_FLUSH |
552 PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
553 PIPE_CONTROL_DEPTH_CACHE_FLUSH |
554 PIPE_CONTROL_DC_FLUSH_ENABLE |
555 PIPE_CONTROL_FLUSH_ENABLE);
556
557 return gen8_emit_fini_breadcrumb_tail(rq, cs);
558}
559
560/*
561 * Note that the CS instruction pre-parser will not stall on the breadcrumb
562 * flush and will continue pre-fetching the instructions after it before the
563 * memory sync is completed. On pre-gen12 HW, the pre-parser will stop at
564 * BB_START/END instructions, so, even though we might pre-fetch the pre-amble
565 * of the next request before the memory has been flushed, we're guaranteed that
566 * we won't access the batch itself too early.
567 * However, on gen12+ the parser can pre-fetch across the BB_START/END commands,
568 * so, if the current request is modifying an instruction in the next request on
569 * the same intel_context, we might pre-fetch and then execute the pre-update
570 * instruction. To avoid this, the users of self-modifying code should either
571 * disable the parser around the code emitting the memory writes, via a new flag
572 * added to MI_ARB_CHECK, or emit the writes from a different intel_context. For
573 * the in-kernel use-cases we've opted to use a separate context, see
574 * reloc_gpu() as an example.
575 * All the above applies only to the instructions themselves. Non-inline data
576 * used by the instructions is not pre-fetched.
577 */
578
579static u32 *gen12_emit_preempt_busywait(struct i915_request *rq, u32 *cs)
580{
581 *cs++ = MI_ARB_CHECK; /* trigger IDLE->ACTIVE first */
582 *cs++ = MI_SEMAPHORE_WAIT_TOKEN |
583 MI_SEMAPHORE_GLOBAL_GTT |
584 MI_SEMAPHORE_POLL |
585 MI_SEMAPHORE_SAD_EQ_SDD;
586 *cs++ = 0;
587 *cs++ = preempt_address(rq->engine);
588 *cs++ = 0;
589 *cs++ = 0;
590
591 return cs;
592}
593
594static __always_inline u32*
595gen12_emit_fini_breadcrumb_tail(struct i915_request *rq, u32 *cs)
596{
597 *cs++ = MI_USER_INTERRUPT;
598
599 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
600 if (intel_engine_has_semaphores(rq->engine))
601 cs = gen12_emit_preempt_busywait(rq, cs);
602
603 rq->tail = intel_ring_offset(rq, cs);
604 assert_ring_tail_valid(rq->ring, rq->tail);
605
606 return gen8_emit_wa_tail(rq, cs);
607}
608
609u32 *gen12_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs)
610{
611 /* XXX Stalling flush before seqno write; post-sync not */
612 cs = emit_xcs_breadcrumb(rq, __gen8_emit_flush_dw(cs, 0, 0, 0));
613 return gen12_emit_fini_breadcrumb_tail(rq, cs);
614}
615
616u32 *gen12_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs)
617{
618 cs = gen12_emit_ggtt_write_rcs(cs,
619 rq->fence.seqno,
620 hwsp_offset(rq),
621 PIPE_CONTROL0_HDC_PIPELINE_FLUSH,
622 PIPE_CONTROL_CS_STALL |
623 PIPE_CONTROL_TILE_CACHE_FLUSH |
624 PIPE_CONTROL_FLUSH_L3 |
625 PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH |
626 PIPE_CONTROL_DEPTH_CACHE_FLUSH |
627 /* Wa_1409600907:tgl */
628 PIPE_CONTROL_DEPTH_STALL |
629 PIPE_CONTROL_DC_FLUSH_ENABLE |
630 PIPE_CONTROL_FLUSH_ENABLE);
631
632 return gen12_emit_fini_breadcrumb_tail(rq, cs);
633}
drivers/gpu/drm/i915/gt/gen8_engine_cs.h+127−0
@@ -0,0 +1,127 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright © 2014 Intel Corporation
4 */
5
6#ifndef __GEN8_ENGINE_CS_H__
7#define __GEN8_ENGINE_CS_H__
8
9#include <linux/string.h>
10#include <linux/types.h>
11
12#include "i915_gem.h" /* GEM_BUG_ON */
13
14#include "intel_gpu_commands.h"
15
16struct i915_request;
17
18int gen8_emit_flush_rcs(struct i915_request *rq, u32 mode);
19int gen11_emit_flush_rcs(struct i915_request *rq, u32 mode);
20int gen12_emit_flush_rcs(struct i915_request *rq, u32 mode);
21
22int gen8_emit_flush_xcs(struct i915_request *rq, u32 mode);
23int gen12_emit_flush_xcs(struct i915_request *rq, u32 mode);
24
25int gen8_emit_init_breadcrumb(struct i915_request *rq);
26
27int gen8_emit_bb_start_noarb(struct i915_request *rq,
28 u64 offset, u32 len,
29 const unsigned int flags);
30int gen8_emit_bb_start(struct i915_request *rq,
31 u64 offset, u32 len,
32 const unsigned int flags);
33
34u32 *gen8_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs);
35u32 *gen12_emit_fini_breadcrumb_xcs(struct i915_request *rq, u32 *cs);
36
37u32 *gen8_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs);
38u32 *gen11_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs);
39u32 *gen12_emit_fini_breadcrumb_rcs(struct i915_request *rq, u32 *cs);
40
41static inline u32 *
42__gen8_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
43{
44 memset(batch, 0, 6 * sizeof(u32));
45
46 batch[0] = GFX_OP_PIPE_CONTROL(6) | flags0;
47 batch[1] = flags1;
48 batch[2] = offset;
49
50 return batch + 6;
51}
52
53static inline u32 *gen8_emit_pipe_control(u32 *batch, u32 flags, u32 offset)
54{
55 return __gen8_emit_pipe_control(batch, 0, flags, offset);
56}
57
58static inline u32 *gen12_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
59{
60 return __gen8_emit_pipe_control(batch, flags0, flags1, offset);
61}
62
63static inline u32 *
64__gen8_emit_write_rcs(u32 *cs, u32 value, u32 offset, u32 flags0, u32 flags1)
65{
66 *cs++ = GFX_OP_PIPE_CONTROL(6) | flags0;
67 *cs++ = flags1 | PIPE_CONTROL_QW_WRITE;
68 *cs++ = offset;
69 *cs++ = 0;
70 *cs++ = value;
71 *cs++ = 0; /* We're thrashing one extra dword. */
72
73 return cs;
74}
75
76static inline u32*
77gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
78{
79 /* We're using qword write, offset should be aligned to 8 bytes. */
80 GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
81
82 return __gen8_emit_write_rcs(cs,
83 value,
84 gtt_offset,
85 0,
86 flags | PIPE_CONTROL_GLOBAL_GTT_IVB);
87}
88
89static inline u32*
90gen12_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags0, u32 flags1)
91{
92 /* We're using qword write, offset should be aligned to 8 bytes. */
93 GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
94
95 return __gen8_emit_write_rcs(cs,
96 value,
97 gtt_offset,
98 flags0,
99 flags1 | PIPE_CONTROL_GLOBAL_GTT_IVB);
100}
101
102static inline u32 *
103__gen8_emit_flush_dw(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
104{
105 *cs++ = (MI_FLUSH_DW + 1) | flags;
106 *cs++ = gtt_offset;
107 *cs++ = 0;
108 *cs++ = value;
109
110 return cs;
111}
112
113static inline u32 *
114gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
115{
116 /* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
117 GEM_BUG_ON(gtt_offset & (1 << 5));
118 /* Offset should be aligned to 8 bytes for both (QW/DW) write types */
119 GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
120
121 return __gen8_emit_flush_dw(cs,
122 value,
123 gtt_offset | MI_FLUSH_DW_USE_GTT,
124 flags | MI_FLUSH_DW_OP_STOREDW);
125}
126
127#endif /* __GEN8_ENGINE_CS_H__ */
drivers/gpu/drm/i915/gt/intel_breadcrumbs.c+73−37
@@ -134,11 +134,6 @@ static bool remove_signaling_context(struct intel_breadcrumbs *b, · 133 unmodified lines
134 return true;134 return true;
135}135}
136 136
137static inline bool __request_completed(const struct i915_request *rq)
138{
139 return i915_seqno_passed(__hwsp_seqno(rq), rq->fence.seqno);
140}
141
142__maybe_unused static bool137__maybe_unused static bool
143check_signal_order(struct intel_context *ce, struct i915_request *rq)138check_signal_order(struct intel_context *ce, struct i915_request *rq)
144{139{
@@ -192,18 +187,6 @@ static void add_retire(struct intel_breadcrumbs *b, struct intel_timeline *tl) · 47 unmodified lines
192 intel_engine_add_retire(b->irq_engine, tl);187 intel_engine_add_retire(b->irq_engine, tl);
193}188}
194 189
195static bool __signal_request(struct i915_request *rq)
196{
197 GEM_BUG_ON(test_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags));
198
199 if (!__dma_fence_signal(&rq->fence)) {
200 i915_request_put(rq);
201 return false;
202 }
203
204 return true;
205}
206
207static struct llist_node *190static struct llist_node *
208slist_add(struct llist_node *node, struct llist_node *head)191slist_add(struct llist_node *node, struct llist_node *head)
209{192{
@@ -251,13 +234,14 @@ static void signal_irq_work(struct irq_work *work) · 41 unmodified lines
251 intel_breadcrumbs_disarm_irq(b);234 intel_breadcrumbs_disarm_irq(b);
252 235
253 rcu_read_lock();236 rcu_read_lock();
237 atomic_inc(&b->signaler_active);
254 list_for_each_entry_rcu(ce, &b->signalers, signal_link) {238 list_for_each_entry_rcu(ce, &b->signalers, signal_link) {
255 struct i915_request *rq;239 struct i915_request *rq;
256 240
257 list_for_each_entry_rcu(rq, &ce->signals, signal_link) {241 list_for_each_entry_rcu(rq, &ce->signals, signal_link) {
258 bool release;242 bool release;
259 243
260 if (!__request_completed(rq))244 if (!__i915_request_is_complete(rq))
261 break;245 break;
262 246
263 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL,247 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL,
@@ -273,17 +257,20 @@ static void signal_irq_work(struct irq_work *work) · 9 unmodified lines
273 list_del_rcu(&rq->signal_link);257 list_del_rcu(&rq->signal_link);
274 release = remove_signaling_context(b, ce);258 release = remove_signaling_context(b, ce);
275 spin_unlock(&ce->signal_lock);259 spin_unlock(&ce->signal_lock);
276
277 if (__signal_request(rq))
278 /* We own signal_node now, xfer to local list */
279 signal = slist_add(&rq->signal_node, signal);
280
281 if (release) {260 if (release) {
282 add_retire(b, ce->timeline);261 if (intel_timeline_is_last(ce->timeline, rq))
262 add_retire(b, ce->timeline);
283 intel_context_put(ce);263 intel_context_put(ce);
284 }264 }
265
266 if (__dma_fence_signal(&rq->fence))
267 /* We own signal_node now, xfer to local list */
268 signal = slist_add(&rq->signal_node, signal);
269 else
270 i915_request_put(rq);
285 }271 }
286 }272 }
273 atomic_dec(&b->signaler_active);
287 rcu_read_unlock();274 rcu_read_unlock();
288 275
289 llist_for_each_safe(signal, sn, signal) {276 llist_for_each_safe(signal, sn, signal) {
@@ -342,17 +329,19 @@ void intel_breadcrumbs_reset(struct intel_breadcrumbs *b) · 52 unmodified lines
342 spin_unlock_irqrestore(&b->irq_lock, flags);329 spin_unlock_irqrestore(&b->irq_lock, flags);
343}330}
344 331
345void intel_breadcrumbs_park(struct intel_breadcrumbs *b)332void __intel_breadcrumbs_park(struct intel_breadcrumbs *b)
346{333{
347 /* Kick the work once more to drain the signalers */334 if (!READ_ONCE(b->irq_armed))
335 return;
336
337 /* Kick the work once more to drain the signalers, and disarm the irq */
348 irq_work_sync(&b->irq_work);338 irq_work_sync(&b->irq_work);
349 while (unlikely(READ_ONCE(b->irq_armed))) {339 while (READ_ONCE(b->irq_armed) && !atomic_read(&b->active)) {
350 local_irq_disable();340 local_irq_disable();
351 signal_irq_work(&b->irq_work);341 signal_irq_work(&b->irq_work);
352 local_irq_enable();342 local_irq_enable();
353 cond_resched();343 cond_resched();
354 }344 }
355 GEM_BUG_ON(!list_empty(&b->signalers));
356}345}
357 346
358void intel_breadcrumbs_free(struct intel_breadcrumbs *b)347void intel_breadcrumbs_free(struct intel_breadcrumbs *b)
@@ -363,6 +352,17 @@ void intel_breadcrumbs_free(struct intel_breadcrumbs *b) · 4 unmodified lines
363 kfree(b);352 kfree(b);
364}353}
365 354
355static void irq_signal_request(struct i915_request *rq,
356 struct intel_breadcrumbs *b)
357{
358 if (!__dma_fence_signal(&rq->fence))
359 return;
360
361 i915_request_get(rq);
362 if (llist_add(&rq->signal_node, &b->signaled_requests))
363 irq_work_queue(&b->irq_work);
364}
365
366static void insert_breadcrumb(struct i915_request *rq)366static void insert_breadcrumb(struct i915_request *rq)
367{367{
368 struct intel_breadcrumbs *b = READ_ONCE(rq->engine)->breadcrumbs;368 struct intel_breadcrumbs *b = READ_ONCE(rq->engine)->breadcrumbs;
@@ -372,17 +372,13 @@ static void insert_breadcrumb(struct i915_request *rq) · 3 unmodified lines
372 if (test_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags))372 if (test_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags))
373 return;373 return;
374 374
375 i915_request_get(rq);
376
377 /*375 /*
378 * If the request is already completed, we can transfer it376 * If the request is already completed, we can transfer it
379 * straight onto a signaled list, and queue the irq worker for377 * straight onto a signaled list, and queue the irq worker for
380 * its signal completion.378 * its signal completion.
381 */379 */
382 if (__request_completed(rq)) {380 if (__i915_request_is_complete(rq)) {
383 if (__signal_request(rq) &&381 irq_signal_request(rq, b);
384 llist_add(&rq->signal_node, &b->signaled_requests))
385 irq_work_queue(&b->irq_work);
386 return;382 return;
387 }383 }
388 384
@@ -413,6 +409,8 @@ static void insert_breadcrumb(struct i915_request *rq) · 24 unmodified lines
413 break;409 break;
414 }410 }
415 }411 }
412
413 i915_request_get(rq);
416 list_add_rcu(&rq->signal_link, pos);414 list_add_rcu(&rq->signal_link, pos);
417 GEM_BUG_ON(!check_signal_order(ce, rq));415 GEM_BUG_ON(!check_signal_order(ce, rq));
418 GEM_BUG_ON(test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &rq->fence.flags));416 GEM_BUG_ON(test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &rq->fence.flags));
@@ -453,22 +451,60 @@ bool i915_request_enable_breadcrumb(struct i915_request *rq) · 34 unmodified lines
453 451
454void i915_request_cancel_breadcrumb(struct i915_request *rq)452void i915_request_cancel_breadcrumb(struct i915_request *rq)
455{453{
454 struct intel_breadcrumbs *b = READ_ONCE(rq->engine)->breadcrumbs;
456 struct intel_context *ce = rq->context;455 struct intel_context *ce = rq->context;
456 unsigned long flags;
457 bool release;457 bool release;
458 458
459 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags))459 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL, &rq->fence.flags))
460 return;460 return;
461 461
462 spin_lock(&ce->signal_lock);462 spin_lock_irqsave(&ce->signal_lock, flags);
463 list_del_rcu(&rq->signal_link);463 list_del_rcu(&rq->signal_link);
464 release = remove_signaling_context(rq->engine->breadcrumbs, ce);464 release = remove_signaling_context(b, ce);
465 spin_unlock(&ce->signal_lock);465 spin_unlock_irqrestore(&ce->signal_lock, flags);
466 if (release)466 if (release)
467 intel_context_put(ce);467 intel_context_put(ce);
468 468
469 if (__i915_request_is_complete(rq))
470 irq_signal_request(rq, b);
471
469 i915_request_put(rq);472 i915_request_put(rq);
470}473}
471 474
475void intel_context_remove_breadcrumbs(struct intel_context *ce,
476 struct intel_breadcrumbs *b)
477{
478 struct i915_request *rq, *rn;
479 bool release = false;
480 unsigned long flags;
481
482 spin_lock_irqsave(&ce->signal_lock, flags);
483
484 if (list_empty(&ce->signals))
485 goto unlock;
486
487 list_for_each_entry_safe(rq, rn, &ce->signals, signal_link) {
488 GEM_BUG_ON(!__i915_request_is_complete(rq));
489 if (!test_and_clear_bit(I915_FENCE_FLAG_SIGNAL,
490 &rq->fence.flags))
491 continue;
492
493 list_del_rcu(&rq->signal_link);
494 irq_signal_request(rq, b);
495 i915_request_put(rq);
496 }
497 release = remove_signaling_context(b, ce);
498
499unlock:
500 spin_unlock_irqrestore(&ce->signal_lock, flags);
501 if (release)
502 intel_context_put(ce);
503
504 while (atomic_read(&b->signaler_active))
505 cpu_relax();
506}
507
472static void print_signals(struct intel_breadcrumbs *b, struct drm_printer *p)508static void print_signals(struct intel_breadcrumbs *b, struct drm_printer *p)
473{509{
474 struct intel_context *ce;510 struct intel_context *ce;
drivers/gpu/drm/i915/gt/intel_breadcrumbs.h+16−1
@@ -6,6 +6,7 @@ · 5 unmodified lines
6#ifndef __INTEL_BREADCRUMBS__6#ifndef __INTEL_BREADCRUMBS__
7#define __INTEL_BREADCRUMBS__7#define __INTEL_BREADCRUMBS__
8 8
9#include <linux/atomic.h>
9#include <linux/irq_work.h>10#include <linux/irq_work.h>
10 11
11#include "intel_engine_types.h"12#include "intel_engine_types.h"
@@ -19,7 +20,18 @@ intel_breadcrumbs_create(struct intel_engine_cs *irq_engine); · 7 unmodified lines
19void intel_breadcrumbs_free(struct intel_breadcrumbs *b);20void intel_breadcrumbs_free(struct intel_breadcrumbs *b);
20 21
21void intel_breadcrumbs_reset(struct intel_breadcrumbs *b);22void intel_breadcrumbs_reset(struct intel_breadcrumbs *b);
22void intel_breadcrumbs_park(struct intel_breadcrumbs *b);23void __intel_breadcrumbs_park(struct intel_breadcrumbs *b);
24
25static inline void intel_breadcrumbs_unpark(struct intel_breadcrumbs *b)
26{
27 atomic_inc(&b->active);
28}
29
30static inline void intel_breadcrumbs_park(struct intel_breadcrumbs *b)
31{
32 if (atomic_dec_and_test(&b->active))
33 __intel_breadcrumbs_park(b);
34}
23 35
24static inline void36static inline void
25intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine)37intel_engine_signal_breadcrumbs(struct intel_engine_cs *engine)
@@ -33,4 +45,7 @@ void intel_engine_print_breadcrumbs(struct intel_engine_cs *engine, · 7 unmodified lines
33bool i915_request_enable_breadcrumb(struct i915_request *request);45bool i915_request_enable_breadcrumb(struct i915_request *request);
34void i915_request_cancel_breadcrumb(struct i915_request *request);46void i915_request_cancel_breadcrumb(struct i915_request *request);
35 47
48void intel_context_remove_breadcrumbs(struct intel_context *ce,
49 struct intel_breadcrumbs *b);
50
36#endif /* __INTEL_BREADCRUMBS__ */51#endif /* __INTEL_BREADCRUMBS__ */
drivers/gpu/drm/i915/gt/intel_breadcrumbs_types.h+5−2
@@ -29,17 +29,20 @@ · 28 unmodified lines
29 * the overhead of waking that client is much preferred.29 * the overhead of waking that client is much preferred.
30 */30 */
31struct intel_breadcrumbs {31struct intel_breadcrumbs {
32 /* Not all breadcrumbs are attached to physical HW */32 atomic_t active;
33 struct intel_engine_cs *irq_engine;
34 33
35 spinlock_t signalers_lock; /* protects the list of signalers */34 spinlock_t signalers_lock; /* protects the list of signalers */
36 struct list_head signalers;35 struct list_head signalers;
37 struct llist_head signaled_requests;36 struct llist_head signaled_requests;
37 atomic_t signaler_active;
38 38
39 spinlock_t irq_lock; /* protects the interrupt from hardirq context */39 spinlock_t irq_lock; /* protects the interrupt from hardirq context */
40 struct irq_work irq_work; /* for use from inside irq_lock */40 struct irq_work irq_work; /* for use from inside irq_lock */
41 unsigned int irq_enabled;41 unsigned int irq_enabled;
42 bool irq_armed;42 bool irq_armed;
43
44 /* Not all breadcrumbs are attached to physical HW */
45 struct intel_engine_cs *irq_engine;
43};46};
44 47
45#endif /* __INTEL_BREADCRUMBS_TYPES__ */48#endif /* __INTEL_BREADCRUMBS_TYPES__ */
drivers/gpu/drm/i915/gt/intel_context.h+7−4
@@ -191,6 +191,11 @@ static inline bool intel_context_is_closed(const struct intel_context *ce) · 190 unmodified lines
191 return test_bit(CONTEXT_CLOSED_BIT, &ce->flags);191 return test_bit(CONTEXT_CLOSED_BIT, &ce->flags);
192}192}
193 193
194static inline bool intel_context_has_inflight(const struct intel_context *ce)
195{
196 return test_bit(COPS_HAS_INFLIGHT_BIT, &ce->ops->flags);
197}
198
194static inline bool intel_context_use_semaphores(const struct intel_context *ce)199static inline bool intel_context_use_semaphores(const struct intel_context *ce)
195{200{
196 return test_bit(CONTEXT_USE_SEMAPHORES, &ce->flags);201 return test_bit(CONTEXT_USE_SEMAPHORES, &ce->flags);
@@ -248,16 +253,14 @@ intel_context_clear_nopreempt(struct intel_context *ce) · 51 unmodified lines
248 253
249static inline u64 intel_context_get_total_runtime_ns(struct intel_context *ce)254static inline u64 intel_context_get_total_runtime_ns(struct intel_context *ce)
250{255{
251 const u32 period =256 const u32 period = ce->engine->gt->clock_period_ns;
252 RUNTIME_INFO(ce->engine->i915)->cs_timestamp_period_ns;
253 257
254 return READ_ONCE(ce->runtime.total) * period;258 return READ_ONCE(ce->runtime.total) * period;
255}259}
256 260
257static inline u64 intel_context_get_avg_runtime_ns(struct intel_context *ce)261static inline u64 intel_context_get_avg_runtime_ns(struct intel_context *ce)
258{262{
259 const u32 period =263 const u32 period = ce->engine->gt->clock_period_ns;
260 RUNTIME_INFO(ce->engine->i915)->cs_timestamp_period_ns;
261 264
262 return mul_u32_u32(ewma_runtime_read(&ce->runtime.avg), period);265 return mul_u32_u32(ewma_runtime_read(&ce->runtime.avg), period);
263}266}
drivers/gpu/drm/i915/gt/intel_context_sseu.c+0−1
@@ -9,7 +9,6 @@ · 8 unmodified lines
9#include "intel_engine_pm.h"9#include "intel_engine_pm.h"
10#include "intel_gpu_commands.h"10#include "intel_gpu_commands.h"
11#include "intel_lrc.h"11#include "intel_lrc.h"
12#include "intel_lrc_reg.h"
13#include "intel_ring.h"12#include "intel_ring.h"
14#include "intel_sseu.h"13#include "intel_sseu.h"
15 14
drivers/gpu/drm/i915/gt/intel_context_types.h+17−8
@@ -30,6 +30,10 @@ struct intel_context; · 29 unmodified lines
30struct intel_ring;30struct intel_ring;
31 31
32struct intel_context_ops {32struct intel_context_ops {
33 unsigned long flags;
34#define COPS_HAS_INFLIGHT_BIT 0
35#define COPS_HAS_INFLIGHT BIT(COPS_HAS_INFLIGHT_BIT)
36
33 int (*alloc)(struct intel_context *ce);37 int (*alloc)(struct intel_context *ce);
34 38
35 int (*pre_pin)(struct intel_context *ce, struct i915_gem_ww_ctx *ww, void **vaddr);39 int (*pre_pin)(struct intel_context *ce, struct i915_gem_ww_ctx *ww, void **vaddr);
@@ -58,8 +62,12 @@ struct intel_context { · 22 unmodified lines
58 62
59 struct intel_engine_cs *engine;63 struct intel_engine_cs *engine;
60 struct intel_engine_cs *inflight;64 struct intel_engine_cs *inflight;
61#define intel_context_inflight(ce) ptr_mask_bits(READ_ONCE((ce)->inflight), 2)65#define __intel_context_inflight(engine) ptr_mask_bits(engine, 3)
62#define intel_context_inflight_count(ce) ptr_unmask_bits(READ_ONCE((ce)->inflight), 2)66#define __intel_context_inflight_count(engine) ptr_unmask_bits(engine, 3)
67#define intel_context_inflight(ce) \
68 __intel_context_inflight(READ_ONCE((ce)->inflight))
69#define intel_context_inflight_count(ce) \
70 __intel_context_inflight_count(READ_ONCE((ce)->inflight))
63 71
64 struct i915_address_space *vm;72 struct i915_address_space *vm;
65 struct i915_gem_context __rcu *gem_context;73 struct i915_gem_context __rcu *gem_context;
@@ -81,12 +89,13 @@ struct intel_context { · 15 unmodified lines
81 unsigned long flags;89 unsigned long flags;
82#define CONTEXT_BARRIER_BIT 090#define CONTEXT_BARRIER_BIT 0
83#define CONTEXT_ALLOC_BIT 191#define CONTEXT_ALLOC_BIT 1
84#define CONTEXT_VALID_BIT 292#define CONTEXT_INIT_BIT 2
85#define CONTEXT_CLOSED_BIT 393#define CONTEXT_VALID_BIT 3
86#define CONTEXT_USE_SEMAPHORES 494#define CONTEXT_CLOSED_BIT 4
87#define CONTEXT_BANNED 595#define CONTEXT_USE_SEMAPHORES 5
88#define CONTEXT_FORCE_SINGLE_SUBMISSION 696#define CONTEXT_BANNED 6
89#define CONTEXT_NOPREEMPT 797#define CONTEXT_FORCE_SINGLE_SUBMISSION 7
98#define CONTEXT_NOPREEMPT 8
90 99
91 u32 *lrc_reg_state;100 u32 *lrc_reg_state;
92 union {101 union {
drivers/gpu/drm/i915/gt/intel_engine.h+6−87
@@ -15,7 +15,6 @@ · 14 unmodified lines
15#include "i915_selftest.h"15#include "i915_selftest.h"
16#include "gt/intel_timeline.h"16#include "gt/intel_timeline.h"
17#include "intel_engine_types.h"17#include "intel_engine_types.h"
18#include "intel_gpu_commands.h"
19#include "intel_workarounds.h"18#include "intel_workarounds.h"
20 19
21struct drm_printer;20struct drm_printer;
@@ -223,91 +222,6 @@ void intel_engine_get_instdone(const struct intel_engine_cs *engine, · 201 unmodified lines
223 222
224void intel_engine_init_execlists(struct intel_engine_cs *engine);223void intel_engine_init_execlists(struct intel_engine_cs *engine);
225 224
226static inline u32 *__gen8_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
227{
228 memset(batch, 0, 6 * sizeof(u32));
229
230 batch[0] = GFX_OP_PIPE_CONTROL(6) | flags0;
231 batch[1] = flags1;
232 batch[2] = offset;
233
234 return batch + 6;
235}
236
237static inline u32 *gen8_emit_pipe_control(u32 *batch, u32 flags, u32 offset)
238{
239 return __gen8_emit_pipe_control(batch, 0, flags, offset);
240}
241
242static inline u32 *gen12_emit_pipe_control(u32 *batch, u32 flags0, u32 flags1, u32 offset)
243{
244 return __gen8_emit_pipe_control(batch, flags0, flags1, offset);
245}
246
247static inline u32 *
248__gen8_emit_write_rcs(u32 *cs, u32 value, u32 offset, u32 flags0, u32 flags1)
249{
250 *cs++ = GFX_OP_PIPE_CONTROL(6) | flags0;
251 *cs++ = flags1 | PIPE_CONTROL_QW_WRITE;
252 *cs++ = offset;
253 *cs++ = 0;
254 *cs++ = value;
255 *cs++ = 0; /* We're thrashing one extra dword. */
256
257 return cs;
258}
259
260static inline u32*
261gen8_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
262{
263 /* We're using qword write, offset should be aligned to 8 bytes. */
264 GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
265
266 return __gen8_emit_write_rcs(cs,
267 value,
268 gtt_offset,
269 0,
270 flags | PIPE_CONTROL_GLOBAL_GTT_IVB);
271}
272
273static inline u32*
274gen12_emit_ggtt_write_rcs(u32 *cs, u32 value, u32 gtt_offset, u32 flags0, u32 flags1)
275{
276 /* We're using qword write, offset should be aligned to 8 bytes. */
277 GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
278
279 return __gen8_emit_write_rcs(cs,
280 value,
281 gtt_offset,
282 flags0,
283 flags1 | PIPE_CONTROL_GLOBAL_GTT_IVB);
284}
285
286static inline u32 *
287__gen8_emit_flush_dw(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
288{
289 *cs++ = (MI_FLUSH_DW + 1) | flags;
290 *cs++ = gtt_offset;
291 *cs++ = 0;
292 *cs++ = value;
293
294 return cs;
295}
296
297static inline u32 *
298gen8_emit_ggtt_write(u32 *cs, u32 value, u32 gtt_offset, u32 flags)
299{
300 /* w/a: bit 5 needs to be zero for MI_FLUSH_DW address. */
301 GEM_BUG_ON(gtt_offset & (1 << 5));
302 /* Offset should be aligned to 8 bytes for both (QW/DW) write types */
303 GEM_BUG_ON(!IS_ALIGNED(gtt_offset, 8));
304
305 return __gen8_emit_flush_dw(cs,
306 value,
307 gtt_offset | MI_FLUSH_DW_USE_GTT,
308 flags | MI_FLUSH_DW_OP_STOREDW);
309}
310
311static inline void __intel_engine_reset(struct intel_engine_cs *engine,225static inline void __intel_engine_reset(struct intel_engine_cs *engine,
312 bool stalled)226 bool stalled)
313{227{
@@ -318,7 +232,12 @@ static inline void __intel_engine_reset(struct intel_engine_cs *engine, · 4 unmodified lines
318 232
319bool intel_engines_are_idle(struct intel_gt *gt);233bool intel_engines_are_idle(struct intel_gt *gt);
320bool intel_engine_is_idle(struct intel_engine_cs *engine);234bool intel_engine_is_idle(struct intel_engine_cs *engine);
321void intel_engine_flush_submission(struct intel_engine_cs *engine);235
236void __intel_engine_flush_submission(struct intel_engine_cs *engine, bool sync);
237static inline void intel_engine_flush_submission(struct intel_engine_cs *engine)
238{
239 __intel_engine_flush_submission(engine, true);
240}
322 241
323void intel_engines_reset_default_submission(struct intel_gt *gt);242void intel_engines_reset_default_submission(struct intel_gt *gt);
324 243
drivers/gpu/drm/i915/gt/intel_engine_cs.c+80−71
@@ -33,12 +33,14 @@ · 32 unmodified lines
33#include "intel_engine.h"33#include "intel_engine.h"
34#include "intel_engine_pm.h"34#include "intel_engine_pm.h"
35#include "intel_engine_user.h"35#include "intel_engine_user.h"
36#include "intel_execlists_submission.h"
36#include "intel_gt.h"37#include "intel_gt.h"
37#include "intel_gt_requests.h"38#include "intel_gt_requests.h"
38#include "intel_gt_pm.h"39#include "intel_gt_pm.h"
39#include "intel_lrc.h"40#include "intel_lrc_reg.h"
40#include "intel_reset.h"41#include "intel_reset.h"
41#include "intel_ring.h"42#include "intel_ring.h"
43#include "uc/intel_guc_submission.h"
42 44
43/* Haswell does have the CXT_SIZE register however it does not appear to be45/* Haswell does have the CXT_SIZE register however it does not appear to be
44 * valid. Now, docs explain in dwords what is in the context object. The full46 * valid. Now, docs explain in dwords what is in the context object. The full
@@ -647,6 +649,8 @@ static int init_status_page(struct intel_engine_cs *engine) · 602 unmodified lines
647 void *vaddr;649 void *vaddr;
648 int ret;650 int ret;
649 651
652 INIT_LIST_HEAD(&engine->status_page.timelines);
653
650 /*654 /*
651 * Though the HWS register does support 36bit addresses, historically655 * Though the HWS register does support 36bit addresses, historically
652 * we have had hangs and corruption reported due to wild writes if656 * we have had hangs and corruption reported due to wild writes if
@@ -723,6 +727,9 @@ static int engine_setup_common(struct intel_engine_cs *engine) · 70 unmodified lines
723 intel_engine_init_whitelist(engine);727 intel_engine_init_whitelist(engine);
724 intel_engine_init_ctx_wa(engine);728 intel_engine_init_ctx_wa(engine);
725 729
730 if (INTEL_GEN(engine->i915) >= 12)
731 engine->flags |= I915_ENGINE_HAS_RELATIVE_MMIO;
732
726 return 0;733 return 0;
727 734
728err_status:735err_status:
@@ -829,6 +836,21 @@ create_pinned_context(struct intel_engine_cs *engine, · 100 unmodified lines
829 return ce;836 return ce;
830}837}
831 838
839static void destroy_pinned_context(struct intel_context *ce)
840{
841 struct intel_engine_cs *engine = ce->engine;
842 struct i915_vma *hwsp = engine->status_page.vma;
843
844 GEM_BUG_ON(ce->timeline->hwsp_ggtt != hwsp);
845
846 mutex_lock(&hwsp->vm->mutex);
847 list_del(&ce->timeline->engine_link);
848 mutex_unlock(&hwsp->vm->mutex);
849
850 intel_context_unpin(ce);
851 intel_context_put(ce);
852}
853
832static struct intel_context *854static struct intel_context *
833create_kernel_context(struct intel_engine_cs *engine)855create_kernel_context(struct intel_engine_cs *engine)
834{856{
@@ -889,7 +911,9 @@ int intel_engines_init(struct intel_gt *gt) · 54 unmodified lines
889 enum intel_engine_id id;911 enum intel_engine_id id;
890 int err;912 int err;
891 913
892 if (HAS_EXECLISTS(gt->i915))914 if (intel_uc_uses_guc_submission(&gt->uc))
915 setup = intel_guc_submission_setup;
916 else if (HAS_EXECLISTS(gt->i915))
893 setup = intel_execlists_submission_setup;917 setup = intel_execlists_submission_setup;
894 else918 else
895 setup = intel_ring_submission_setup;919 setup = intel_ring_submission_setup;
@@ -925,7 +949,6 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine) · 29 unmodified lines
925 GEM_BUG_ON(!list_empty(&engine->active.requests));949 GEM_BUG_ON(!list_empty(&engine->active.requests));
926 tasklet_kill(&engine->execlists.tasklet); /* flush the callback */950 tasklet_kill(&engine->execlists.tasklet); /* flush the callback */
927 951
928 cleanup_status_page(engine);
929 intel_breadcrumbs_free(engine->breadcrumbs);952 intel_breadcrumbs_free(engine->breadcrumbs);
930 953
931 intel_engine_fini_retire(engine);954 intel_engine_fini_retire(engine);
@@ -934,11 +957,11 @@ void intel_engine_cleanup_common(struct intel_engine_cs *engine) · 2 unmodified lines
934 if (engine->default_state)957 if (engine->default_state)
935 fput(engine->default_state);958 fput(engine->default_state);
936 959
937 if (engine->kernel_context) {960 if (engine->kernel_context)
938 intel_context_unpin(engine->kernel_context);961 destroy_pinned_context(engine->kernel_context);
939 intel_context_put(engine->kernel_context);962
940 }
941 GEM_BUG_ON(!llist_empty(&engine->barrier_tasks));963 GEM_BUG_ON(!llist_empty(&engine->barrier_tasks));
964 cleanup_status_page(engine);
942 965
943 intel_wa_list_free(&engine->ctx_wa_list);966 intel_wa_list_free(&engine->ctx_wa_list);
944 intel_wa_list_free(&engine->wa_list);967 intel_wa_list_free(&engine->wa_list);
@@ -1002,32 +1025,50 @@ static unsigned long stop_timeout(const struct intel_engine_cs *engine) · 57 unmodified lines
1002 return READ_ONCE(engine->props.stop_timeout_ms);1025 return READ_ONCE(engine->props.stop_timeout_ms);
1003}1026}
1004 1027
1005int intel_engine_stop_cs(struct intel_engine_cs *engine)1028static int __intel_engine_stop_cs(struct intel_engine_cs *engine,
1029 int fast_timeout_us,
1030 int slow_timeout_ms)
1006{1031{
1007 struct intel_uncore *uncore = engine->uncore;1032 struct intel_uncore *uncore = engine->uncore;
1008 const u32 base = engine->mmio_base;1033 const i915_reg_t mode = RING_MI_MODE(engine->mmio_base);
1009 const i915_reg_t mode = RING_MI_MODE(base);
1010 int err;1034 int err;
1011 1035
1036 intel_uncore_write_fw(uncore, mode, _MASKED_BIT_ENABLE(STOP_RING));
1037 err = __intel_wait_for_register_fw(engine->uncore, mode,
1038 MODE_IDLE, MODE_IDLE,
1039 fast_timeout_us,
1040 slow_timeout_ms,
1041 NULL);
1042
1043 /* A final mmio read to let GPU writes be hopefully flushed to memory */
1044 intel_uncore_posting_read_fw(uncore, mode);
1045 return err;
1046}
1047
1048int intel_engine_stop_cs(struct intel_engine_cs *engine)
1049{
1050 int err = 0;
1051
1012 if (INTEL_GEN(engine->i915) < 3)1052 if (INTEL_GEN(engine->i915) < 3)
1013 return -ENODEV;1053 return -ENODEV;
1014 1054
1015 ENGINE_TRACE(engine, "\n");1055 ENGINE_TRACE(engine, "\n");
1056 if (__intel_engine_stop_cs(engine, 1000, stop_timeout(engine))) {
1057 ENGINE_TRACE(engine,
1058 "timed out on STOP_RING -> IDLE; HEAD:%04x, TAIL:%04x\n",
1059 ENGINE_READ_FW(engine, RING_HEAD) & HEAD_ADDR,
1060 ENGINE_READ_FW(engine, RING_TAIL) & TAIL_ADDR);
1016 1061
1017 intel_uncore_write_fw(uncore, mode, _MASKED_BIT_ENABLE(STOP_RING));1062 /*
1018 1063 * Sometimes we observe that the idle flag is not
1019 err = 0;1064 * set even though the ring is empty. So double
1020 if (__intel_wait_for_register_fw(uncore,1065 * check before giving up.
1021 mode, MODE_IDLE, MODE_IDLE,1066 */
1022 1000, stop_timeout(engine),1067 if ((ENGINE_READ_FW(engine, RING_HEAD) & HEAD_ADDR) !=
1023 NULL)) {1068 (ENGINE_READ_FW(engine, RING_TAIL) & TAIL_ADDR))
1024 ENGINE_TRACE(engine, "timed out on STOP_RING -> IDLE\n");1069 err = -ETIMEDOUT;
1025 err = -ETIMEDOUT;
1026 }1070 }
1027 1071
1028 /* A final mmio read to let GPU writes be hopefully flushed to memory */
1029 intel_uncore_posting_read_fw(uncore, mode);
1030
1031 return err;1072 return err;
1032}1073}
1033 1074
@@ -1189,17 +1230,13 @@ static bool ring_is_idle(struct intel_engine_cs *engine) · 155 unmodified lines
1189 return idle;1230 return idle;
1190}1231}
1191 1232
1192void intel_engine_flush_submission(struct intel_engine_cs *engine)1233void __intel_engine_flush_submission(struct intel_engine_cs *engine, bool sync)
1193{1234{
1194 struct tasklet_struct *t = &engine->execlists.tasklet;1235 struct tasklet_struct *t = &engine->execlists.tasklet;
1195 1236
1196 if (!t->func)1237 if (!t->func)
1197 return;1238 return;
1198 1239
1199 /* Synchronise and wait for the tasklet on another CPU */
1200 tasklet_kill(t);
1201
1202 /* Having cancelled the tasklet, ensure that is run */
1203 local_bh_disable();1240 local_bh_disable();
1204 if (tasklet_trylock(t)) {1241 if (tasklet_trylock(t)) {
1205 /* Must wait for any GPU reset in progress. */1242 /* Must wait for any GPU reset in progress. */
@@ -1208,6 +1245,10 @@ void intel_engine_flush_submission(struct intel_engine_cs *engine) · 2 unmodified lines
1208 tasklet_unlock(t);1245 tasklet_unlock(t);
1209 }1246 }
1210 local_bh_enable();1247 local_bh_enable();
1248
1249 /* Synchronise and wait for the tasklet on another CPU */
1250 if (sync)
1251 tasklet_unlock_wait(t);
1211}1252}
1212 1253
1213/**1254/**
@@ -1273,8 +1314,12 @@ void intel_engines_reset_default_submission(struct intel_gt *gt) · 59 unmodified lines
1273 struct intel_engine_cs *engine;1314 struct intel_engine_cs *engine;
1274 enum intel_engine_id id;1315 enum intel_engine_id id;
1275 1316
1276 for_each_engine(engine, gt, id)1317 for_each_engine(engine, gt, id) {
1318 if (engine->sanitize)
1319 engine->sanitize(engine);
1320
1277 engine->set_default_submission(engine);1321 engine->set_default_submission(engine);
1322 }
1278}1323}
1279 1324
1280bool intel_engine_can_store_dword(struct intel_engine_cs *engine)1325bool intel_engine_can_store_dword(struct intel_engine_cs *engine)
@@ -1294,44 +1339,6 @@ bool intel_engine_can_store_dword(struct intel_engine_cs *engine) · 13 unmodified lines
1294 }1339 }
1295}1340}
1296 1341
1297static int print_sched_attr(const struct i915_sched_attr *attr,
1298 char *buf, int x, int len)
1299{
1300 if (attr->priority == I915_PRIORITY_INVALID)
1301 return x;
1302
1303 x += snprintf(buf + x, len - x,
1304 " prio=%d", attr->priority);
1305
1306 return x;
1307}
1308
1309static void print_request(struct drm_printer *m,
1310 struct i915_request *rq,
1311 const char *prefix)
1312{
1313 const char *name = rq->fence.ops->get_timeline_name(&rq->fence);
1314 char buf[80] = "";
1315 int x = 0;
1316
1317 x = print_sched_attr(&rq->sched.attr, buf, x, sizeof(buf));
1318
1319 drm_printf(m, "%s %llx:%llx%s%s %s @ %dms: %s\n",
1320 prefix,
1321 rq->fence.context, rq->fence.seqno,
1322 i915_request_completed(rq) ? "!" :
1323 i915_request_started(rq) ? "*" :
1324 "",
1325 test_bit(DMA_FENCE_FLAG_SIGNALED_BIT,
1326 &rq->fence.flags) ? "+" :
1327 test_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT,
1328 &rq->fence.flags) ? "-" :
1329 "",
1330 buf,
1331 jiffies_to_msecs(jiffies - rq->emitted_jiffies),
1332 name);
1333}
1334
1335static struct intel_timeline *get_timeline(struct i915_request *rq)1342static struct intel_timeline *get_timeline(struct i915_request *rq)
1336{1343{
1337 struct intel_timeline *tl;1344 struct intel_timeline *tl;
@@ -1480,7 +1487,9 @@ static void intel_engine_print_registers(struct intel_engine_cs *engine, · 142 unmodified lines
1480 drm_printf(m, "\tIPEHR: 0x%08x\n", ENGINE_READ(engine, IPEHR));1487 drm_printf(m, "\tIPEHR: 0x%08x\n", ENGINE_READ(engine, IPEHR));
1481 }1488 }
1482 1489
1483 if (HAS_EXECLISTS(dev_priv)) {1490 if (intel_engine_in_guc_submission_mode(engine)) {
1491 /* nothing to print yet */
1492 } else if (HAS_EXECLISTS(dev_priv)) {
1484 struct i915_request * const *port, *rq;1493 struct i915_request * const *port, *rq;
1485 const u32 *hws =1494 const u32 *hws =
1486 &engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];1495 &engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];
@@ -1529,7 +1538,7 @@ static void intel_engine_print_registers(struct intel_engine_cs *engine, · 42 unmodified lines
1529 intel_context_is_banned(rq->context) ? "*" : "");1538 intel_context_is_banned(rq->context) ? "*" : "");
1530 len += print_ring(hdr + len, sizeof(hdr) - len, rq);1539 len += print_ring(hdr + len, sizeof(hdr) - len, rq);
1531 scnprintf(hdr + len, sizeof(hdr) - len, "rq: ");1540 scnprintf(hdr + len, sizeof(hdr) - len, "rq: ");
1532 print_request(m, rq, hdr);1541 i915_request_show(m, rq, hdr, 0);
1533 }1542 }
1534 for (port = execlists->pending; (rq = *port); port++) {1543 for (port = execlists->pending; (rq = *port); port++) {
1535 char hdr[160];1544 char hdr[160];
@@ -1543,7 +1552,7 @@ static void intel_engine_print_registers(struct intel_engine_cs *engine, · 7 unmodified lines
1543 intel_context_is_banned(rq->context) ? "*" : "");1552 intel_context_is_banned(rq->context) ? "*" : "");
1544 len += print_ring(hdr + len, sizeof(hdr) - len, rq);1553 len += print_ring(hdr + len, sizeof(hdr) - len, rq);
1545 scnprintf(hdr + len, sizeof(hdr) - len, "rq: ");1554 scnprintf(hdr + len, sizeof(hdr) - len, "rq: ");
1546 print_request(m, rq, hdr);1555 i915_request_show(m, rq, hdr, 0);
1547 }1556 }
1548 rcu_read_unlock();1557 rcu_read_unlock();
1549 execlists_active_unlock_bh(execlists);1558 execlists_active_unlock_bh(execlists);
@@ -1687,7 +1696,7 @@ void intel_engine_dump(struct intel_engine_cs *engine, · 137 unmodified lines
1687 if (rq) {1696 if (rq) {
1688 struct intel_timeline *tl = get_timeline(rq);1697 struct intel_timeline *tl = get_timeline(rq);
1689 1698
1690 print_request(m, rq, "\t\tactive ");1699 i915_request_show(m, rq, "\t\tactive ", 0);
1691 1700
1692 drm_printf(m, "\t\tring->start: 0x%08x\n",1701 drm_printf(m, "\t\tring->start: 0x%08x\n",
1693 i915_ggtt_offset(rq->ring->vma));1702 i915_ggtt_offset(rq->ring->vma));
@@ -1725,7 +1734,7 @@ void intel_engine_dump(struct intel_engine_cs *engine, · 31 unmodified lines
1725 drm_printf(m, "\tDevice is asleep; skipping register dump\n");1734 drm_printf(m, "\tDevice is asleep; skipping register dump\n");
1726 }1735 }
1727 1736
1728 intel_execlists_show_requests(engine, m, print_request, 8);1737 intel_execlists_show_requests(engine, m, i915_request_show, 8);
1729 1738
1730 drm_printf(m, "HWSP:\n");1739 drm_printf(m, "HWSP:\n");
1731 hexdump(m, engine->status_page.addr, PAGE_SIZE);1740 hexdump(m, engine->status_page.addr, PAGE_SIZE);
drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c+41−18
@@ -37,6 +37,18 @@ static bool next_heartbeat(struct intel_engine_cs *engine) · 36 unmodified lines
37 return true;37 return true;
38}38}
39 39
40static struct i915_request *
41heartbeat_create(struct intel_context *ce, gfp_t gfp)
42{
43 struct i915_request *rq;
44
45 intel_context_enter(ce);
46 rq = __i915_request_create(ce, gfp);
47 intel_context_exit(ce);
48
49 return rq;
50}
51
40static void idle_pulse(struct intel_engine_cs *engine, struct i915_request *rq)52static void idle_pulse(struct intel_engine_cs *engine, struct i915_request *rq)
41{53{
42 engine->wakeref_serial = READ_ONCE(engine->serial) + 1;54 engine->wakeref_serial = READ_ONCE(engine->serial) + 1;
@@ -45,6 +57,15 @@ static void idle_pulse(struct intel_engine_cs *engine, struct i915_request *rq) · 2 unmodified lines
45 engine->heartbeat.systole = i915_request_get(rq);57 engine->heartbeat.systole = i915_request_get(rq);
46}58}
47 59
60static void heartbeat_commit(struct i915_request *rq,
61 const struct i915_sched_attr *attr)
62{
63 idle_pulse(rq->engine, rq);
64
65 __i915_request_commit(rq);
66 __i915_request_queue(rq, attr);
67}
68
48static void show_heartbeat(const struct i915_request *rq,69static void show_heartbeat(const struct i915_request *rq,
49 struct intel_engine_cs *engine)70 struct intel_engine_cs *engine)
50{71{
@@ -139,16 +160,11 @@ static void heartbeat(struct work_struct *wrk) · 88 unmodified lines
139 goto out;160 goto out;
140 }161 }
141 162
142 intel_context_enter(ce);163 rq = heartbeat_create(ce, GFP_NOWAIT | __GFP_NOWARN);
143 rq = __i915_request_create(ce, GFP_NOWAIT | __GFP_NOWARN);
144 intel_context_exit(ce);
145 if (IS_ERR(rq))164 if (IS_ERR(rq))
146 goto unlock;165 goto unlock;
147 166
148 idle_pulse(engine, rq);167 heartbeat_commit(rq, &attr);
149
150 __i915_request_commit(rq);
151 __i915_request_queue(rq, &attr);
152 168
153unlock:169unlock:
154 mutex_unlock(&ce->timeline->mutex);170 mutex_unlock(&ce->timeline->mutex);
@@ -187,17 +203,13 @@ static int __intel_engine_pulse(struct intel_engine_cs *engine) · 32 unmodified lines
187 GEM_BUG_ON(!intel_engine_has_preemption(engine));203 GEM_BUG_ON(!intel_engine_has_preemption(engine));
188 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));204 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
189 205
190 intel_context_enter(ce);206 rq = heartbeat_create(ce, GFP_NOWAIT | __GFP_NOWARN);
191 rq = __i915_request_create(ce, GFP_NOWAIT | __GFP_NOWARN);
192 intel_context_exit(ce);
193 if (IS_ERR(rq))207 if (IS_ERR(rq))
194 return PTR_ERR(rq);208 return PTR_ERR(rq);
195 209
196 __set_bit(I915_FENCE_FLAG_SENTINEL, &rq->fence.flags);210 __set_bit(I915_FENCE_FLAG_SENTINEL, &rq->fence.flags);
197 idle_pulse(engine, rq);
198 211
199 __i915_request_commit(rq);212 heartbeat_commit(rq, &attr);
200 __i915_request_queue(rq, &attr);
201 GEM_BUG_ON(rq->sched.attr.priority < I915_PRIORITY_BARRIER);213 GEM_BUG_ON(rq->sched.attr.priority < I915_PRIORITY_BARRIER);
202 214
203 return 0;215 return 0;
@@ -273,8 +285,12 @@ int intel_engine_pulse(struct intel_engine_cs *engine) · 69 unmodified lines
273 285
274int intel_engine_flush_barriers(struct intel_engine_cs *engine)286int intel_engine_flush_barriers(struct intel_engine_cs *engine)
275{287{
288 struct i915_sched_attr attr = {
289 .priority = I915_USER_PRIORITY(I915_PRIORITY_MIN),
290 };
291 struct intel_context *ce = engine->kernel_context;
276 struct i915_request *rq;292 struct i915_request *rq;
277 int err = 0;293 int err;
278 294
279 if (llist_empty(&engine->barrier_tasks))295 if (llist_empty(&engine->barrier_tasks))
280 return 0;296 return 0;
@@ -282,15 +298,22 @@ int intel_engine_flush_barriers(struct intel_engine_cs *engine) · 1 unmodified line
282 if (!intel_engine_pm_get_if_awake(engine))298 if (!intel_engine_pm_get_if_awake(engine))
283 return 0;299 return 0;
284 300
285 rq = i915_request_create(engine->kernel_context);301 if (mutex_lock_interruptible(&ce->timeline->mutex)) {
302 err = -EINTR;
303 goto out_rpm;
304 }
305
306 rq = heartbeat_create(ce, GFP_KERNEL);
286 if (IS_ERR(rq)) {307 if (IS_ERR(rq)) {
287 err = PTR_ERR(rq);308 err = PTR_ERR(rq);
288 goto out_rpm;309 goto out_unlock;
289 }310 }
290 311
291 idle_pulse(engine, rq);312 heartbeat_commit(rq, &attr);
292 i915_request_add(rq);
293 313
314 err = 0;
315out_unlock:
316 mutex_unlock(&ce->timeline->mutex);
294out_rpm:317out_rpm:
295 intel_engine_pm_put(engine);318 intel_engine_pm_put(engine);
296 return err;319 return err;
drivers/gpu/drm/i915/gt/intel_engine_pm.c+9−1
@@ -60,11 +60,19 @@ static int __engine_unpark(struct intel_wakeref *wf) · 59 unmodified lines
60 60
61 /* Scrub the context image after our loss of control */61 /* Scrub the context image after our loss of control */
62 ce->ops->reset(ce);62 ce->ops->reset(ce);
63
64 CE_TRACE(ce, "reset { seqno:%x, *hwsp:%x, ring:%x }\n",
65 ce->timeline->seqno,
66 READ_ONCE(*ce->timeline->hwsp_seqno),
67 ce->ring->emit);
68 GEM_BUG_ON(ce->timeline->seqno !=
69 READ_ONCE(*ce->timeline->hwsp_seqno));
63 }70 }
64 71
65 if (engine->unpark)72 if (engine->unpark)
66 engine->unpark(engine);73 engine->unpark(engine);
67 74
75 intel_breadcrumbs_unpark(engine->breadcrumbs);
68 intel_engine_unpark_heartbeat(engine);76 intel_engine_unpark_heartbeat(engine);
69 return 0;77 return 0;
70}78}
@@ -136,7 +144,7 @@ __queue_and_release_pm(struct i915_request *rq, · 65 unmodified lines
136 list_add_tail(&tl->link, &timelines->active_list);144 list_add_tail(&tl->link, &timelines->active_list);
137 145
138 /* Hand the request over to HW and so engine_retire() */146 /* Hand the request over to HW and so engine_retire() */
139 __i915_request_queue(rq, NULL);147 __i915_request_queue_bh(rq);
140 148
141 /* Let new submissions commence (and maybe retire this timeline) */149 /* Let new submissions commence (and maybe retire this timeline) */
142 __intel_wakeref_defer_park(&engine->wakeref);150 __intel_wakeref_defer_park(&engine->wakeref);
drivers/gpu/drm/i915/gt/intel_engine_types.h+5−11
@@ -68,6 +68,7 @@ typedef u8 intel_engine_mask_t; · 67 unmodified lines
68#define ALL_ENGINES ((intel_engine_mask_t)~0ul)68#define ALL_ENGINES ((intel_engine_mask_t)~0ul)
69 69
70struct intel_hw_status_page {70struct intel_hw_status_page {
71 struct list_head timelines;
71 struct i915_vma *vma;72 struct i915_vma *vma;
72 u32 *addr;73 u32 *addr;
73};74};
@@ -183,7 +184,8 @@ struct intel_engine_execlists { · 109 unmodified lines
183 * Reserve the upper 16b for tracking internal errors.184 * Reserve the upper 16b for tracking internal errors.
184 */185 */
185 u32 error_interrupt;186 u32 error_interrupt;
186#define ERROR_CSB BIT(31)187#define ERROR_CSB BIT(31)
188#define ERROR_PREEMPT BIT(30)
187 189
188 /**190 /**
189 * @reset_ccid: Active CCID [EXECLISTS_STATUS_HI] at the time of reset191 * @reset_ccid: Active CCID [EXECLISTS_STATUS_HI] at the time of reset
@@ -236,16 +238,6 @@ struct intel_engine_execlists { · 46 unmodified lines
236 */238 */
237 unsigned int port_mask;239 unsigned int port_mask;
238 240
239 /**
240 * @switch_priority_hint: Second context priority.
241 *
242 * We submit multiple contexts to the HW simultaneously and would
243 * like to occasionally switch between them to emulate timeslicing.
244 * To know when timeslicing is suitable, we track the priority of
245 * the context submitted second.
246 */
247 int switch_priority_hint;
248
249 /**241 /**
250 * @queue_priority_hint: Highest pending priority.242 * @queue_priority_hint: Highest pending priority.
251 *243 *
@@ -559,6 +551,8 @@ struct intel_engine_cs { · 307 unmodified lines
559 unsigned long stop_timeout_ms;551 unsigned long stop_timeout_ms;
560 unsigned long timeslice_duration_ms;552 unsigned long timeslice_duration_ms;
561 } props, defaults;553 } props, defaults;
554
555 I915_SELFTEST_DECLARE(struct fault_attr reset_timeout);
562};556};
563 557
564static inline bool558static inline bool
drivers/gpu/drm/i915/gt/intel_execlists_submission.c+3929−0
@@ -0,0 +1,3929 @@
1// SPDX-License-Identifier: MIT
2/*
3 * Copyright © 2014 Intel Corporation
4 */
5
6/**
7 * DOC: Logical Rings, Logical Ring Contexts and Execlists
8 *
9 * Motivation:
10 * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts".
11 * These expanded contexts enable a number of new abilities, especially
12 * "Execlists" (also implemented in this file).
13 *
14 * One of the main differences with the legacy HW contexts is that logical
15 * ring contexts incorporate many more things to the context's state, like
16 * PDPs or ringbuffer control registers:
17 *
18 * The reason why PDPs are included in the context is straightforward: as
19 * PPGTTs (per-process GTTs) are actually per-context, having the PDPs
20 * contained there mean you don't need to do a ppgtt->switch_mm yourself,
21 * instead, the GPU will do it for you on the context switch.
22 *
23 * But, what about the ringbuffer control registers (head, tail, etc..)?
24 * shouldn't we just need a set of those per engine command streamer? This is
25 * where the name "Logical Rings" starts to make sense: by virtualizing the
26 * rings, the engine cs shifts to a new "ring buffer" with every context
27 * switch. When you want to submit a workload to the GPU you: A) choose your
28 * context, B) find its appropriate virtualized ring, C) write commands to it
29 * and then, finally, D) tell the GPU to switch to that context.
30 *
31 * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch
32 * to a contexts is via a context execution list, ergo "Execlists".
33 *
34 * LRC implementation:
35 * Regarding the creation of contexts, we have:
36 *
37 * - One global default context.
38 * - One local default context for each opened fd.
39 * - One local extra context for each context create ioctl call.
40 *
41 * Now that ringbuffers belong per-context (and not per-engine, like before)
42 * and that contexts are uniquely tied to a given engine (and not reusable,
43 * like before) we need:
44 *
45 * - One ringbuffer per-engine inside each context.
46 * - One backing object per-engine inside each context.
47 *
48 * The global default context starts its life with these new objects fully
49 * allocated and populated. The local default context for each opened fd is
50 * more complex, because we don't know at creation time which engine is going
51 * to use them. To handle this, we have implemented a deferred creation of LR
52 * contexts:
53 *
54 * The local context starts its life as a hollow or blank holder, that only
55 * gets populated for a given engine once we receive an execbuffer. If later
56 * on we receive another execbuffer ioctl for the same context but a different
57 * engine, we allocate/populate a new ringbuffer and context backing object and
58 * so on.
59 *
60 * Finally, regarding local contexts created using the ioctl call: as they are
61 * only allowed with the render ring, we can allocate & populate them right
62 * away (no need to defer anything, at least for now).
63 *
64 * Execlists implementation:
65 * Execlists are the new method by which, on gen8+ hardware, workloads are
66 * submitted for execution (as opposed to the legacy, ringbuffer-based, method).
67 * This method works as follows:
68 *
69 * When a request is committed, its commands (the BB start and any leading or
70 * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer
71 * for the appropriate context. The tail pointer in the hardware context is not
72 * updated at this time, but instead, kept by the driver in the ringbuffer
73 * structure. A structure representing this request is added to a request queue
74 * for the appropriate engine: this structure contains a copy of the context's
75 * tail after the request was written to the ring buffer and a pointer to the
76 * context itself.
77 *
78 * If the engine's request queue was empty before the request was added, the
79 * queue is processed immediately. Otherwise the queue will be processed during
80 * a context switch interrupt. In any case, elements on the queue will get sent
81 * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a
82 * globally unique 20-bits submission ID.
83 *
84 * When execution of a request completes, the GPU updates the context status
85 * buffer with a context complete event and generates a context switch interrupt.
86 * During the interrupt handling, the driver examines the events in the buffer:
87 * for each context complete event, if the announced ID matches that on the head
88 * of the request queue, then that request is retired and removed from the queue.
89 *
90 * After processing, if any requests were retired and the queue is not empty
91 * then a new execution list can be submitted. The two requests at the front of
92 * the queue are next to be submitted but since a context may not occur twice in
93 * an execution list, if subsequent requests have the same ID as the first then
94 * the two requests must be combined. This is done simply by discarding requests
95 * at the head of the queue until either only one requests is left (in which case
96 * we use a NULL second context) or the first two requests have unique IDs.
97 *
98 * By always executing the first two requests in the queue the driver ensures
99 * that the GPU is kept as busy as possible. In the case where a single context
100 * completes but a second context is still executing, the request for this second
101 * context will be at the head of the queue when we remove the first one. This
102 * request will then be resubmitted along with a new request for a different context,
103 * which will cause the hardware to continue executing the second request and queue
104 * the new request (the GPU detects the condition of a context getting preempted
105 * with the same context and optimizes the context switch flow by not doing
106 * preemption, but just sampling the new tail pointer).
107 *
108 */
109#include <linux/interrupt.h>
110
111#include "i915_drv.h"
112#include "i915_trace.h"
113#include "i915_vgpu.h"
114#include "gen8_engine_cs.h"
115#include "intel_breadcrumbs.h"
116#include "intel_context.h"
117#include "intel_engine_pm.h"
118#include "intel_execlists_submission.h"
119#include "intel_gt.h"
120#include "intel_gt_pm.h"
121#include "intel_gt_requests.h"
122#include "intel_lrc.h"
123#include "intel_lrc_reg.h"
124#include "intel_mocs.h"
125#include "intel_reset.h"
126#include "intel_ring.h"
127#include "intel_workarounds.h"
128#include "shmem_utils.h"
129
130#define RING_EXECLIST_QFULL (1 << 0x2)
131#define RING_EXECLIST1_VALID (1 << 0x3)
132#define RING_EXECLIST0_VALID (1 << 0x4)
133#define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE)
134#define RING_EXECLIST1_ACTIVE (1 << 0x11)
135#define RING_EXECLIST0_ACTIVE (1 << 0x12)
136
137#define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0)
138#define GEN8_CTX_STATUS_PREEMPTED (1 << 1)
139#define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2)
140#define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3)
141#define GEN8_CTX_STATUS_COMPLETE (1 << 4)
142#define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15)
143
144#define GEN8_CTX_STATUS_COMPLETED_MASK \
145 (GEN8_CTX_STATUS_COMPLETE | GEN8_CTX_STATUS_PREEMPTED)
146
147#define GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE (0x1) /* lower csb dword */
148#define GEN12_CTX_SWITCH_DETAIL(csb_dw) ((csb_dw) & 0xF) /* upper csb dword */
149#define GEN12_CSB_SW_CTX_ID_MASK GENMASK(25, 15)
150#define GEN12_IDLE_CTX_ID 0x7FF
151#define GEN12_CSB_CTX_VALID(csb_dw) \
152 (FIELD_GET(GEN12_CSB_SW_CTX_ID_MASK, csb_dw) != GEN12_IDLE_CTX_ID)
153
154/* Typical size of the average request (2 pipecontrols and a MI_BB) */
155#define EXECLISTS_REQUEST_SIZE 64 /* bytes */
156
157struct virtual_engine {
158 struct intel_engine_cs base;
159 struct intel_context context;
160 struct rcu_work rcu;
161
162 /*
163 * We allow only a single request through the virtual engine at a time
164 * (each request in the timeline waits for the completion fence of
165 * the previous before being submitted). By restricting ourselves to
166 * only submitting a single request, each request is placed on to a
167 * physical to maximise load spreading (by virtue of the late greedy
168 * scheduling -- each real engine takes the next available request
169 * upon idling).
170 */
171 struct i915_request *request;
172
173 /*
174 * We keep a rbtree of available virtual engines inside each physical
175 * engine, sorted by priority. Here we preallocate the nodes we need
176 * for the virtual engine, indexed by physical_engine->id.
177 */
178 struct ve_node {
179 struct rb_node rb;
180 int prio;
181 } nodes[I915_NUM_ENGINES];
182
183 /*
184 * Keep track of bonded pairs -- restrictions upon on our selection
185 * of physical engines any particular request may be submitted to.
186 * If we receive a submit-fence from a master engine, we will only
187 * use one of sibling_mask physical engines.
188 */
189 struct ve_bond {
190 const struct intel_engine_cs *master;
191 intel_engine_mask_t sibling_mask;
192 } *bonds;
193 unsigned int num_bonds;
194
195 /* And finally, which physical engines this virtual engine maps onto. */
196 unsigned int num_siblings;
197 struct intel_engine_cs *siblings[];
198};
199
200static struct virtual_engine *to_virtual_engine(struct intel_engine_cs *engine)
201{
202 GEM_BUG_ON(!intel_engine_is_virtual(engine));
203 return container_of(engine, struct virtual_engine, base);
204}
205
206static struct i915_request *
207__active_request(const struct intel_timeline * const tl,
208 struct i915_request *rq,
209 int error)
210{
211 struct i915_request *active = rq;
212
213 list_for_each_entry_from_reverse(rq, &tl->requests, link) {
214 if (__i915_request_is_complete(rq))
215 break;
216
217 if (error) {
218 i915_request_set_error_once(rq, error);
219 __i915_request_skip(rq);
220 }
221 active = rq;
222 }
223
224 return active;
225}
226
227static struct i915_request *
228active_request(const struct intel_timeline * const tl, struct i915_request *rq)
229{
230 return __active_request(tl, rq, 0);
231}
232
233static inline void
234ring_set_paused(const struct intel_engine_cs *engine, int state)
235{
236 /*
237 * We inspect HWS_PREEMPT with a semaphore inside
238 * engine->emit_fini_breadcrumb. If the dword is true,
239 * the ring is paused as the semaphore will busywait
240 * until the dword is false.
241 */
242 engine->status_page.addr[I915_GEM_HWS_PREEMPT] = state;
243 if (state)
244 wmb();
245}
246
247static inline struct i915_priolist *to_priolist(struct rb_node *rb)
248{
249 return rb_entry(rb, struct i915_priolist, node);
250}
251
252static inline int rq_prio(const struct i915_request *rq)
253{
254 return READ_ONCE(rq->sched.attr.priority);
255}
256
257static int effective_prio(const struct i915_request *rq)
258{
259 int prio = rq_prio(rq);
260
261 /*
262 * If this request is special and must not be interrupted at any
263 * cost, so be it. Note we are only checking the most recent request
264 * in the context and so may be masking an earlier vip request. It
265 * is hoped that under the conditions where nopreempt is used, this
266 * will not matter (i.e. all requests to that context will be
267 * nopreempt for as long as desired).
268 */
269 if (i915_request_has_nopreempt(rq))
270 prio = I915_PRIORITY_UNPREEMPTABLE;
271
272 return prio;
273}
274
275static int queue_prio(const struct intel_engine_execlists *execlists)
276{
277 struct i915_priolist *p;
278 struct rb_node *rb;
279
280 rb = rb_first_cached(&execlists->queue);
281 if (!rb)
282 return INT_MIN;
283
284 /*
285 * As the priolist[] are inverted, with the highest priority in [0],
286 * we have to flip the index value to become priority.
287 */
288 p = to_priolist(rb);
289 if (!I915_USER_PRIORITY_SHIFT)
290 return p->priority;
291
292 return ((p->priority + 1) << I915_USER_PRIORITY_SHIFT) - ffs(p->used);
293}
294
295static int virtual_prio(const struct intel_engine_execlists *el)
296{
297 struct rb_node *rb = rb_first_cached(&el->virtual);
298
299 return rb ? rb_entry(rb, struct ve_node, rb)->prio : INT_MIN;
300}
301
302static inline bool need_preempt(const struct intel_engine_cs *engine,
303 const struct i915_request *rq)
304{
305 int last_prio;
306
307 if (!intel_engine_has_semaphores(engine))
308 return false;
309
310 /*
311 * Check if the current priority hint merits a preemption attempt.
312 *
313 * We record the highest value priority we saw during rescheduling
314 * prior to this dequeue, therefore we know that if it is strictly
315 * less than the current tail of ESLP[0], we do not need to force
316 * a preempt-to-idle cycle.
317 *
318 * However, the priority hint is a mere hint that we may need to
319 * preempt. If that hint is stale or we may be trying to preempt
320 * ourselves, ignore the request.
321 *
322 * More naturally we would write
323 * prio >= max(0, last);
324 * except that we wish to prevent triggering preemption at the same
325 * priority level: the task that is running should remain running
326 * to preserve FIFO ordering of dependencies.
327 */
328 last_prio = max(effective_prio(rq), I915_PRIORITY_NORMAL - 1);
329 if (engine->execlists.queue_priority_hint <= last_prio)
330 return false;
331
332 /*
333 * Check against the first request in ELSP[1], it will, thanks to the
334 * power of PI, be the highest priority of that context.
335 */
336 if (!list_is_last(&rq->sched.link, &engine->active.requests) &&
337 rq_prio(list_next_entry(rq, sched.link)) > last_prio)
338 return true;
339
340 /*
341 * If the inflight context did not trigger the preemption, then maybe
342 * it was the set of queued requests? Pick the highest priority in
343 * the queue (the first active priolist) and see if it deserves to be
344 * running instead of ELSP[0].
345 *
346 * The highest priority request in the queue can not be either
347 * ELSP[0] or ELSP[1] as, thanks again to PI, if it was the same
348 * context, it's priority would not exceed ELSP[0] aka last_prio.
349 */
350 return max(virtual_prio(&engine->execlists),
351 queue_prio(&engine->execlists)) > last_prio;
352}
353
354__maybe_unused static inline bool
355assert_priority_queue(const struct i915_request *prev,
356 const struct i915_request *next)
357{
358 /*
359 * Without preemption, the prev may refer to the still active element
360 * which we refuse to let go.
361 *
362 * Even with preemption, there are times when we think it is better not
363 * to preempt and leave an ostensibly lower priority request in flight.
364 */
365 if (i915_request_is_active(prev))
366 return true;
367
368 return rq_prio(prev) >= rq_prio(next);
369}
370
371static struct i915_request *
372__unwind_incomplete_requests(struct intel_engine_cs *engine)
373{
374 struct i915_request *rq, *rn, *active = NULL;
375 struct list_head *pl;
376 int prio = I915_PRIORITY_INVALID;
377
378 lockdep_assert_held(&engine->active.lock);
379
380 list_for_each_entry_safe_reverse(rq, rn,
381 &engine->active.requests,
382 sched.link) {
383 if (__i915_request_is_complete(rq)) {
384 list_del_init(&rq->sched.link);
385 continue;
386 }
387
388 __i915_request_unsubmit(rq);
389
390 GEM_BUG_ON(rq_prio(rq) == I915_PRIORITY_INVALID);
391 if (rq_prio(rq) != prio) {
392 prio = rq_prio(rq);
393 pl = i915_sched_lookup_priolist(engine, prio);
394 }
395 GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
396
397 list_move(&rq->sched.link, pl);
398 set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
399
400 /* Check in case we rollback so far we wrap [size/2] */
401 if (intel_ring_direction(rq->ring,
402 rq->tail,
403 rq->ring->tail + 8) > 0)
404 rq->context->lrc.desc |= CTX_DESC_FORCE_RESTORE;
405
406 active = rq;
407 }
408
409 return active;
410}
411
412struct i915_request *
413execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists)
414{
415 struct intel_engine_cs *engine =
416 container_of(execlists, typeof(*engine), execlists);
417
418 return __unwind_incomplete_requests(engine);
419}
420
421static inline void
422execlists_context_status_change(struct i915_request *rq, unsigned long status)
423{
424 /*
425 * Only used when GVT-g is enabled now. When GVT-g is disabled,
426 * The compiler should eliminate this function as dead-code.
427 */
428 if (!IS_ENABLED(CONFIG_DRM_I915_GVT))
429 return;
430
431 atomic_notifier_call_chain(&rq->engine->context_status_notifier,
432 status, rq);
433}
434
435static void intel_engine_context_in(struct intel_engine_cs *engine)
436{
437 unsigned long flags;
438
439 if (atomic_add_unless(&engine->stats.active, 1, 0))
440 return;
441
442 write_seqlock_irqsave(&engine->stats.lock, flags);
443 if (!atomic_add_unless(&engine->stats.active, 1, 0)) {
444 engine->stats.start = ktime_get();
445 atomic_inc(&engine->stats.active);
446 }
447 write_sequnlock_irqrestore(&engine->stats.lock, flags);
448}
449
450static void intel_engine_context_out(struct intel_engine_cs *engine)
451{
452 unsigned long flags;
453
454 GEM_BUG_ON(!atomic_read(&engine->stats.active));
455
456 if (atomic_add_unless(&engine->stats.active, -1, 1))
457 return;
458
459 write_seqlock_irqsave(&engine->stats.lock, flags);
460 if (atomic_dec_and_test(&engine->stats.active)) {
461 engine->stats.total =
462 ktime_add(engine->stats.total,
463 ktime_sub(ktime_get(), engine->stats.start));
464 }
465 write_sequnlock_irqrestore(&engine->stats.lock, flags);
466}
467
468static void reset_active(struct i915_request *rq,
469 struct intel_engine_cs *engine)
470{
471 struct intel_context * const ce = rq->context;
472 u32 head;
473
474 /*
475 * The executing context has been cancelled. We want to prevent
476 * further execution along this context and propagate the error on
477 * to anything depending on its results.
478 *
479 * In __i915_request_submit(), we apply the -EIO and remove the
480 * requests' payloads for any banned requests. But first, we must
481 * rewind the context back to the start of the incomplete request so
482 * that we do not jump back into the middle of the batch.
483 *
484 * We preserve the breadcrumbs and semaphores of the incomplete
485 * requests so that inter-timeline dependencies (i.e other timelines)
486 * remain correctly ordered. And we defer to __i915_request_submit()
487 * so that all asynchronous waits are correctly handled.
488 */
489 ENGINE_TRACE(engine, "{ reset rq=%llx:%lld }\n",
490 rq->fence.context, rq->fence.seqno);
491
492 /* On resubmission of the active request, payload will be scrubbed */
493 if (__i915_request_is_complete(rq))
494 head = rq->tail;
495 else
496 head = __active_request(ce->timeline, rq, -EIO)->head;
497 head = intel_ring_wrap(ce->ring, head);
498
499 /* Scrub the context image to prevent replaying the previous batch */
500 lrc_init_regs(ce, engine, true);
501
502 /* We've switched away, so this should be a no-op, but intent matters */
503 ce->lrc.lrca = lrc_update_regs(ce, engine, head);
504}
505
506static inline struct intel_engine_cs *
507__execlists_schedule_in(struct i915_request *rq)
508{
509 struct intel_engine_cs * const engine = rq->engine;
510 struct intel_context * const ce = rq->context;
511
512 intel_context_get(ce);
513
514 if (unlikely(intel_context_is_closed(ce) &&
515 !intel_engine_has_heartbeat(engine)))
516 intel_context_set_banned(ce);
517
518 if (unlikely(intel_context_is_banned(ce)))
519 reset_active(rq, engine);
520
521 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
522 lrc_check_regs(ce, engine, "before");
523
524 if (ce->tag) {
525 /* Use a fixed tag for OA and friends */
526 GEM_BUG_ON(ce->tag <= BITS_PER_LONG);
527 ce->lrc.ccid = ce->tag;
528 } else {
529 /* We don't need a strict matching tag, just different values */
530 unsigned int tag = __ffs(engine->context_tag);
531
532 GEM_BUG_ON(tag >= BITS_PER_LONG);
533 __clear_bit(tag, &engine->context_tag);
534 ce->lrc.ccid = (1 + tag) << (GEN11_SW_CTX_ID_SHIFT - 32);
535
536 BUILD_BUG_ON(BITS_PER_LONG > GEN12_MAX_CONTEXT_HW_ID);
537 }
538
539 ce->lrc.ccid |= engine->execlists.ccid;
540
541 __intel_gt_pm_get(engine->gt);
542 if (engine->fw_domain && !atomic_fetch_inc(&engine->fw_active))
543 intel_uncore_forcewake_get(engine->uncore, engine->fw_domain);
544 execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_IN);
545 intel_engine_context_in(engine);
546
547 CE_TRACE(ce, "schedule-in, ccid:%x\n", ce->lrc.ccid);
548
549 return engine;
550}
551
552static inline void execlists_schedule_in(struct i915_request *rq, int idx)
553{
554 struct intel_context * const ce = rq->context;
555 struct intel_engine_cs *old;
556
557 GEM_BUG_ON(!intel_engine_pm_is_awake(rq->engine));
558 trace_i915_request_in(rq, idx);
559
560 old = ce->inflight;
561 if (!old)
562 old = __execlists_schedule_in(rq);
563 WRITE_ONCE(ce->inflight, ptr_inc(old));
564
565 GEM_BUG_ON(intel_context_inflight(ce) != rq->engine);
566}
567
568static void
569resubmit_virtual_request(struct i915_request *rq, struct virtual_engine *ve)
570{
571 struct intel_engine_cs *engine = rq->engine;
572
573 spin_lock_irq(&engine->active.lock);
574
575 clear_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
576 WRITE_ONCE(rq->engine, &ve->base);
577 ve->base.submit_request(rq);
578
579 spin_unlock_irq(&engine->active.lock);
580}
581
582static void kick_siblings(struct i915_request *rq, struct intel_context *ce)
583{
584 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
585 struct intel_engine_cs *engine = rq->engine;
586
587 /*
588 * After this point, the rq may be transferred to a new sibling, so
589 * before we clear ce->inflight make sure that the context has been
590 * removed from the b->signalers and furthermore we need to make sure
591 * that the concurrent iterator in signal_irq_work is no longer
592 * following ce->signal_link.
593 */
594 if (!list_empty(&ce->signals))
595 intel_context_remove_breadcrumbs(ce, engine->breadcrumbs);
596
597 /*
598 * This engine is now too busy to run this virtual request, so
599 * see if we can find an alternative engine for it to execute on.
600 * Once a request has become bonded to this engine, we treat it the
601 * same as other native request.
602 */
603 if (i915_request_in_priority_queue(rq) &&
604 rq->execution_mask != engine->mask)
605 resubmit_virtual_request(rq, ve);
606
607 if (READ_ONCE(ve->request))
608 tasklet_hi_schedule(&ve->base.execlists.tasklet);
609}
610
611static inline void __execlists_schedule_out(struct i915_request *rq)
612{
613 struct intel_context * const ce = rq->context;
614 struct intel_engine_cs * const engine = rq->engine;
615 unsigned int ccid;
616
617 /*
618 * NB process_csb() is not under the engine->active.lock and hence
619 * schedule_out can race with schedule_in meaning that we should
620 * refrain from doing non-trivial work here.
621 */
622
623 CE_TRACE(ce, "schedule-out, ccid:%x\n", ce->lrc.ccid);
624
625 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
626 lrc_check_regs(ce, engine, "after");
627
628 /*
629 * If we have just completed this context, the engine may now be
630 * idle and we want to re-enter powersaving.
631 */
632 if (intel_timeline_is_last(ce->timeline, rq) &&
633 __i915_request_is_complete(rq))
634 intel_engine_add_retire(engine, ce->timeline);
635
636 ccid = ce->lrc.ccid;
637 ccid >>= GEN11_SW_CTX_ID_SHIFT - 32;
638 ccid &= GEN12_MAX_CONTEXT_HW_ID;
639 if (ccid < BITS_PER_LONG) {
640 GEM_BUG_ON(ccid == 0);
641 GEM_BUG_ON(test_bit(ccid - 1, &engine->context_tag));
642 __set_bit(ccid - 1, &engine->context_tag);
643 }
644
645 lrc_update_runtime(ce);
646 intel_engine_context_out(engine);
647 execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_OUT);
648 if (engine->fw_domain && !atomic_dec_return(&engine->fw_active))
649 intel_uncore_forcewake_put(engine->uncore, engine->fw_domain);
650 intel_gt_pm_put_async(engine->gt);
651
652 /*
653 * If this is part of a virtual engine, its next request may
654 * have been blocked waiting for access to the active context.
655 * We have to kick all the siblings again in case we need to
656 * switch (e.g. the next request is not runnable on this
657 * engine). Hopefully, we will already have submitted the next
658 * request before the tasklet runs and do not need to rebuild
659 * each virtual tree and kick everyone again.
660 */
661 if (ce->engine != engine)
662 kick_siblings(rq, ce);
663}
664
665static inline void
666execlists_schedule_out(struct i915_request *rq)
667{
668 struct intel_context * const ce = rq->context;
669
670 trace_i915_request_out(rq);
671
672 GEM_BUG_ON(!ce->inflight);
673 ce->inflight = ptr_dec(ce->inflight);
674 if (!__intel_context_inflight_count(ce->inflight)) {
675 GEM_BUG_ON(ce->inflight != rq->engine);
676 __execlists_schedule_out(rq);
677 WRITE_ONCE(ce->inflight, NULL);
678 intel_context_put(ce);
679 }
680
681 i915_request_put(rq);
682}
683
684static u64 execlists_update_context(struct i915_request *rq)
685{
686 struct intel_context *ce = rq->context;
687 u64 desc = ce->lrc.desc;
688 u32 tail, prev;
689
690 /*
691 * WaIdleLiteRestore:bdw,skl
692 *
693 * We should never submit the context with the same RING_TAIL twice
694 * just in case we submit an empty ring, which confuses the HW.
695 *
696 * We append a couple of NOOPs (gen8_emit_wa_tail) after the end of
697 * the normal request to be able to always advance the RING_TAIL on
698 * subsequent resubmissions (for lite restore). Should that fail us,
699 * and we try and submit the same tail again, force the context
700 * reload.
701 *
702 * If we need to return to a preempted context, we need to skip the
703 * lite-restore and force it to reload the RING_TAIL. Otherwise, the
704 * HW has a tendency to ignore us rewinding the TAIL to the end of
705 * an earlier request.
706 */
707 GEM_BUG_ON(ce->lrc_reg_state[CTX_RING_TAIL] != rq->ring->tail);
708 prev = rq->ring->tail;
709 tail = intel_ring_set_tail(rq->ring, rq->tail);
710 if (unlikely(intel_ring_direction(rq->ring, tail, prev) <= 0))
711 desc |= CTX_DESC_FORCE_RESTORE;
712 ce->lrc_reg_state[CTX_RING_TAIL] = tail;
713 rq->tail = rq->wa_tail;
714
715 /*
716 * Make sure the context image is complete before we submit it to HW.
717 *
718 * Ostensibly, writes (including the WCB) should be flushed prior to
719 * an uncached write such as our mmio register access, the empirical
720 * evidence (esp. on Braswell) suggests that the WC write into memory
721 * may not be visible to the HW prior to the completion of the UC
722 * register write and that we may begin execution from the context
723 * before its image is complete leading to invalid PD chasing.
724 */
725 wmb();
726
727 ce->lrc.desc &= ~CTX_DESC_FORCE_RESTORE;
728 return desc;
729}
730
731static inline void write_desc(struct intel_engine_execlists *execlists, u64 desc, u32 port)
732{
733 if (execlists->ctrl_reg) {
734 writel(lower_32_bits(desc), execlists->submit_reg + port * 2);
735 writel(upper_32_bits(desc), execlists->submit_reg + port * 2 + 1);
736 } else {
737 writel(upper_32_bits(desc), execlists->submit_reg);
738 writel(lower_32_bits(desc), execlists->submit_reg);
739 }
740}
741
742static __maybe_unused char *
743dump_port(char *buf, int buflen, const char *prefix, struct i915_request *rq)
744{
745 if (!rq)
746 return "";
747
748 snprintf(buf, buflen, "%sccid:%x %llx:%lld%s prio %d",
749 prefix,
750 rq->context->lrc.ccid,
751 rq->fence.context, rq->fence.seqno,
752 __i915_request_is_complete(rq) ? "!" :
753 __i915_request_has_started(rq) ? "*" :
754 "",
755 rq_prio(rq));
756
757 return buf;
758}
759
760static __maybe_unused void
761trace_ports(const struct intel_engine_execlists *execlists,
762 const char *msg,
763 struct i915_request * const *ports)
764{
765 const struct intel_engine_cs *engine =
766 container_of(execlists, typeof(*engine), execlists);
767 char __maybe_unused p0[40], p1[40];
768
769 if (!ports[0])
770 return;
771
772 ENGINE_TRACE(engine, "%s { %s%s }\n", msg,
773 dump_port(p0, sizeof(p0), "", ports[0]),
774 dump_port(p1, sizeof(p1), ", ", ports[1]));
775}
776
777static inline bool
778reset_in_progress(const struct intel_engine_execlists *execlists)
779{
780 return unlikely(!__tasklet_is_enabled(&execlists->tasklet));
781}
782
783static __maybe_unused bool
784assert_pending_valid(const struct intel_engine_execlists *execlists,
785 const char *msg)
786{
787 struct intel_engine_cs *engine =
788 container_of(execlists, typeof(*engine), execlists);
789 struct i915_request * const *port, *rq;
790 struct intel_context *ce = NULL;
791 bool sentinel = false;
792 u32 ccid = -1;
793
794 trace_ports(execlists, msg, execlists->pending);
795
796 /* We may be messing around with the lists during reset, lalala */
797 if (reset_in_progress(execlists))
798 return true;
799
800 if (!execlists->pending[0]) {
801 GEM_TRACE_ERR("%s: Nothing pending for promotion!\n",
802 engine->name);
803 return false;
804 }
805
806 if (execlists->pending[execlists_num_ports(execlists)]) {
807 GEM_TRACE_ERR("%s: Excess pending[%d] for promotion!\n",
808 engine->name, execlists_num_ports(execlists));
809 return false;
810 }
811
812 for (port = execlists->pending; (rq = *port); port++) {
813 unsigned long flags;
814 bool ok = true;
815
816 GEM_BUG_ON(!kref_read(&rq->fence.refcount));
817 GEM_BUG_ON(!i915_request_is_active(rq));
818
819 if (ce == rq->context) {
820 GEM_TRACE_ERR("%s: Dup context:%llx in pending[%zd]\n",
821 engine->name,
822 ce->timeline->fence_context,
823 port - execlists->pending);
824 return false;
825 }
826 ce = rq->context;
827
828 if (ccid == ce->lrc.ccid) {
829 GEM_TRACE_ERR("%s: Dup ccid:%x context:%llx in pending[%zd]\n",
830 engine->name,
831 ccid, ce->timeline->fence_context,
832 port - execlists->pending);
833 return false;
834 }
835 ccid = ce->lrc.ccid;
836
837 /*
838 * Sentinels are supposed to be the last request so they flush
839 * the current execution off the HW. Check that they are the only
840 * request in the pending submission.
841 */
842 if (sentinel) {
843 GEM_TRACE_ERR("%s: context:%llx after sentinel in pending[%zd]\n",
844 engine->name,
845 ce->timeline->fence_context,
846 port - execlists->pending);
847 return false;
848 }
849 sentinel = i915_request_has_sentinel(rq);
850
851 /*
852 * We want virtual requests to only be in the first slot so
853 * that they are never stuck behind a hog and can be immediately
854 * transferred onto the next idle engine.
855 */
856 if (rq->execution_mask != engine->mask &&
857 port != execlists->pending) {
858 GEM_TRACE_ERR("%s: virtual engine:%llx not in prime position[%zd]\n",
859 engine->name,
860 ce->timeline->fence_context,
861 port - execlists->pending);
862 return false;
863 }
864
865 /* Hold tightly onto the lock to prevent concurrent retires! */
866 if (!spin_trylock_irqsave(&rq->lock, flags))
867 continue;
868
869 if (__i915_request_is_complete(rq))
870 goto unlock;
871
872 if (i915_active_is_idle(&ce->active) &&
873 !intel_context_is_barrier(ce)) {
874 GEM_TRACE_ERR("%s: Inactive context:%llx in pending[%zd]\n",
875 engine->name,
876 ce->timeline->fence_context,
877 port - execlists->pending);
878 ok = false;
879 goto unlock;
880 }
881
882 if (!i915_vma_is_pinned(ce->state)) {
883 GEM_TRACE_ERR("%s: Unpinned context:%llx in pending[%zd]\n",
884 engine->name,
885 ce->timeline->fence_context,
886 port - execlists->pending);
887 ok = false;
888 goto unlock;
889 }
890
891 if (!i915_vma_is_pinned(ce->ring->vma)) {
892 GEM_TRACE_ERR("%s: Unpinned ring:%llx in pending[%zd]\n",
893 engine->name,
894 ce->timeline->fence_context,
895 port - execlists->pending);
896 ok = false;
897 goto unlock;
898 }
899
900unlock:
901 spin_unlock_irqrestore(&rq->lock, flags);
902 if (!ok)
903 return false;
904 }
905
906 return ce;
907}
908
909static void execlists_submit_ports(struct intel_engine_cs *engine)
910{
911 struct intel_engine_execlists *execlists = &engine->execlists;
912 unsigned int n;
913
914 GEM_BUG_ON(!assert_pending_valid(execlists, "submit"));
915
916 /*
917 * We can skip acquiring intel_runtime_pm_get() here as it was taken
918 * on our behalf by the request (see i915_gem_mark_busy()) and it will
919 * not be relinquished until the device is idle (see
920 * i915_gem_idle_work_handler()). As a precaution, we make sure
921 * that all ELSP are drained i.e. we have processed the CSB,
922 * before allowing ourselves to idle and calling intel_runtime_pm_put().
923 */
924 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
925
926 /*
927 * ELSQ note: the submit queue is not cleared after being submitted
928 * to the HW so we need to make sure we always clean it up. This is
929 * currently ensured by the fact that we always write the same number
930 * of elsq entries, keep this in mind before changing the loop below.
931 */
932 for (n = execlists_num_ports(execlists); n--; ) {
933 struct i915_request *rq = execlists->pending[n];
934
935 write_desc(execlists,
936 rq ? execlists_update_context(rq) : 0,
937 n);
938 }
939
940 /* we need to manually load the submit queue */
941 if (execlists->ctrl_reg)
942 writel(EL_CTRL_LOAD, execlists->ctrl_reg);
943}
944
945static bool ctx_single_port_submission(const struct intel_context *ce)
946{
947 return (IS_ENABLED(CONFIG_DRM_I915_GVT) &&
948 intel_context_force_single_submission(ce));
949}
950
951static bool can_merge_ctx(const struct intel_context *prev,
952 const struct intel_context *next)
953{
954 if (prev != next)
955 return false;
956
957 if (ctx_single_port_submission(prev))
958 return false;
959
960 return true;
961}
962
963static unsigned long i915_request_flags(const struct i915_request *rq)
964{
965 return READ_ONCE(rq->fence.flags);
966}
967
968static bool can_merge_rq(const struct i915_request *prev,
969 const struct i915_request *next)
970{
971 GEM_BUG_ON(prev == next);
972 GEM_BUG_ON(!assert_priority_queue(prev, next));
973
974 /*
975 * We do not submit known completed requests. Therefore if the next
976 * request is already completed, we can pretend to merge it in
977 * with the previous context (and we will skip updating the ELSP
978 * and tracking). Thus hopefully keeping the ELSP full with active
979 * contexts, despite the best efforts of preempt-to-busy to confuse
980 * us.
981 */
982 if (__i915_request_is_complete(next))
983 return true;
984
985 if (unlikely((i915_request_flags(prev) ^ i915_request_flags(next)) &
986 (BIT(I915_FENCE_FLAG_NOPREEMPT) |
987 BIT(I915_FENCE_FLAG_SENTINEL))))
988 return false;
989
990 if (!can_merge_ctx(prev->context, next->context))
991 return false;
992
993 GEM_BUG_ON(i915_seqno_passed(prev->fence.seqno, next->fence.seqno));
994 return true;
995}
996
997static bool virtual_matches(const struct virtual_engine *ve,
998 const struct i915_request *rq,
999 const struct intel_engine_cs *engine)
1000{
1001 const struct intel_engine_cs *inflight;
1002
1003 if (!rq)
1004 return false;
1005
1006 if (!(rq->execution_mask & engine->mask)) /* We peeked too soon! */
1007 return false;
1008
1009 /*
1010 * We track when the HW has completed saving the context image
1011 * (i.e. when we have seen the final CS event switching out of
1012 * the context) and must not overwrite the context image before
1013 * then. This restricts us to only using the active engine
1014 * while the previous virtualized request is inflight (so
1015 * we reuse the register offsets). This is a very small
1016 * hystersis on the greedy seelction algorithm.
1017 */
1018 inflight = intel_context_inflight(&ve->context);
1019 if (inflight && inflight != engine)
1020 return false;
1021
1022 return true;
1023}
1024
1025static struct virtual_engine *
1026first_virtual_engine(struct intel_engine_cs *engine)
1027{
1028 struct intel_engine_execlists *el = &engine->execlists;
1029 struct rb_node *rb = rb_first_cached(&el->virtual);
1030
1031 while (rb) {
1032 struct virtual_engine *ve =
1033 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
1034 struct i915_request *rq = READ_ONCE(ve->request);
1035
1036 /* lazily cleanup after another engine handled rq */
1037 if (!rq || !virtual_matches(ve, rq, engine)) {
1038 rb_erase_cached(rb, &el->virtual);
1039 RB_CLEAR_NODE(rb);
1040 rb = rb_first_cached(&el->virtual);
1041 continue;
1042 }
1043
1044 return ve;
1045 }
1046
1047 return NULL;
1048}
1049
1050static void virtual_xfer_context(struct virtual_engine *ve,
1051 struct intel_engine_cs *engine)
1052{
1053 unsigned int n;
1054
1055 if (likely(engine == ve->siblings[0]))
1056 return;
1057
1058 GEM_BUG_ON(READ_ONCE(ve->context.inflight));
1059 if (!intel_engine_has_relative_mmio(engine))
1060 lrc_update_offsets(&ve->context, engine);
1061
1062 /*
1063 * Move the bound engine to the top of the list for
1064 * future execution. We then kick this tasklet first
1065 * before checking others, so that we preferentially
1066 * reuse this set of bound registers.
1067 */
1068 for (n = 1; n < ve->num_siblings; n++) {
1069 if (ve->siblings[n] == engine) {
1070 swap(ve->siblings[n], ve->siblings[0]);
1071 break;
1072 }
1073 }
1074}
1075
1076static void defer_request(struct i915_request *rq, struct list_head * const pl)
1077{
1078 LIST_HEAD(list);
1079
1080 /*
1081 * We want to move the interrupted request to the back of
1082 * the round-robin list (i.e. its priority level), but
1083 * in doing so, we must then move all requests that were in
1084 * flight and were waiting for the interrupted request to
1085 * be run after it again.
1086 */
1087 do {
1088 struct i915_dependency *p;
1089
1090 GEM_BUG_ON(i915_request_is_active(rq));
1091 list_move_tail(&rq->sched.link, pl);
1092
1093 for_each_waiter(p, rq) {
1094 struct i915_request *w =
1095 container_of(p->waiter, typeof(*w), sched);
1096
1097 if (p->flags & I915_DEPENDENCY_WEAK)
1098 continue;
1099
1100 /* Leave semaphores spinning on the other engines */
1101 if (w->engine != rq->engine)
1102 continue;
1103
1104 /* No waiter should start before its signaler */
1105 GEM_BUG_ON(i915_request_has_initial_breadcrumb(w) &&
1106 __i915_request_has_started(w) &&
1107 !__i915_request_is_complete(rq));
1108
1109 GEM_BUG_ON(i915_request_is_active(w));
1110 if (!i915_request_is_ready(w))
1111 continue;
1112
1113 if (rq_prio(w) < rq_prio(rq))
1114 continue;
1115
1116 GEM_BUG_ON(rq_prio(w) > rq_prio(rq));
1117 list_move_tail(&w->sched.link, &list);
1118 }
1119
1120 rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
1121 } while (rq);
1122}
1123
1124static void defer_active(struct intel_engine_cs *engine)
1125{
1126 struct i915_request *rq;
1127
1128 rq = __unwind_incomplete_requests(engine);
1129 if (!rq)
1130 return;
1131
1132 defer_request(rq, i915_sched_lookup_priolist(engine, rq_prio(rq)));
1133}
1134
1135static bool
1136timeslice_yield(const struct intel_engine_execlists *el,
1137 const struct i915_request *rq)
1138{
1139 /*
1140 * Once bitten, forever smitten!
1141 *
1142 * If the active context ever busy-waited on a semaphore,
1143 * it will be treated as a hog until the end of its timeslice (i.e.
1144 * until it is scheduled out and replaced by a new submission,
1145 * possibly even its own lite-restore). The HW only sends an interrupt
1146 * on the first miss, and we do know if that semaphore has been
1147 * signaled, or even if it is now stuck on another semaphore. Play
1148 * safe, yield if it might be stuck -- it will be given a fresh
1149 * timeslice in the near future.
1150 */
1151 return rq->context->lrc.ccid == READ_ONCE(el->yield);
1152}
1153
1154static bool needs_timeslice(const struct intel_engine_cs *engine,
1155 const struct i915_request *rq)
1156{
1157 if (!intel_engine_has_timeslices(engine))
1158 return false;
1159
1160 /* If not currently active, or about to switch, wait for next event */
1161 if (!rq || __i915_request_is_complete(rq))
1162 return false;
1163
1164 /* We do not need to start the timeslice until after the ACK */
1165 if (READ_ONCE(engine->execlists.pending[0]))
1166 return false;
1167
1168 /* If ELSP[1] is occupied, always check to see if worth slicing */
1169 if (!list_is_last_rcu(&rq->sched.link, &engine->active.requests)) {
1170 ENGINE_TRACE(engine, "timeslice required for second inflight context\n");
1171 return true;
1172 }
1173
1174 /* Otherwise, ELSP[0] is by itself, but may be waiting in the queue */
1175 if (!RB_EMPTY_ROOT(&engine->execlists.queue.rb_root)) {
1176 ENGINE_TRACE(engine, "timeslice required for queue\n");
1177 return true;
1178 }
1179
1180 if (!RB_EMPTY_ROOT(&engine->execlists.virtual.rb_root)) {
1181 ENGINE_TRACE(engine, "timeslice required for virtual\n");
1182 return true;
1183 }
1184
1185 return false;
1186}
1187
1188static bool
1189timeslice_expired(struct intel_engine_cs *engine, const struct i915_request *rq)
1190{
1191 const struct intel_engine_execlists *el = &engine->execlists;
1192
1193 if (i915_request_has_nopreempt(rq) && __i915_request_has_started(rq))
1194 return false;
1195
1196 if (!needs_timeslice(engine, rq))
1197 return false;
1198
1199 return timer_expired(&el->timer) || timeslice_yield(el, rq);
1200}
1201
1202static unsigned long timeslice(const struct intel_engine_cs *engine)
1203{
1204 return READ_ONCE(engine->props.timeslice_duration_ms);
1205}
1206
1207static void start_timeslice(struct intel_engine_cs *engine)
1208{
1209 struct intel_engine_execlists *el = &engine->execlists;
1210 unsigned long duration;
1211
1212 /* Disable the timer if there is nothing to switch to */
1213 duration = 0;
1214 if (needs_timeslice(engine, *el->active)) {
1215 /* Avoid continually prolonging an active timeslice */
1216 if (timer_active(&el->timer)) {
1217 /*
1218 * If we just submitted a new ELSP after an old
1219 * context, that context may have already consumed
1220 * its timeslice, so recheck.
1221 */
1222 if (!timer_pending(&el->timer))
1223 tasklet_hi_schedule(&el->tasklet);
1224 return;
1225 }
1226
1227 duration = timeslice(engine);
1228 }
1229
1230 set_timer_ms(&el->timer, duration);
1231}
1232
1233static void record_preemption(struct intel_engine_execlists *execlists)
1234{
1235 (void)I915_SELFTEST_ONLY(execlists->preempt_hang.count++);
1236}
1237
1238static unsigned long active_preempt_timeout(struct intel_engine_cs *engine,
1239 const struct i915_request *rq)
1240{
1241 if (!rq)
1242 return 0;
1243
1244 /* Force a fast reset for terminated contexts (ignoring sysfs!) */
1245 if (unlikely(intel_context_is_banned(rq->context)))
1246 return 1;
1247
1248 return READ_ONCE(engine->props.preempt_timeout_ms);
1249}
1250
1251static void set_preempt_timeout(struct intel_engine_cs *engine,
1252 const struct i915_request *rq)
1253{
1254 if (!intel_engine_has_preempt_reset(engine))
1255 return;
1256
1257 set_timer_ms(&engine->execlists.preempt,
1258 active_preempt_timeout(engine, rq));
1259}
1260
1261static void execlists_dequeue(struct intel_engine_cs *engine)
1262{
1263 struct intel_engine_execlists * const execlists = &engine->execlists;
1264 struct i915_request **port = execlists->pending;
1265 struct i915_request ** const last_port = port + execlists->port_mask;
1266 struct i915_request *last = *execlists->active;
1267 struct virtual_engine *ve;
1268 struct rb_node *rb;
1269 bool submit = false;
1270
1271 /*
1272 * Hardware submission is through 2 ports. Conceptually each port
1273 * has a (RING_START, RING_HEAD, RING_TAIL) tuple. RING_START is
1274 * static for a context, and unique to each, so we only execute
1275 * requests belonging to a single context from each ring. RING_HEAD
1276 * is maintained by the CS in the context image, it marks the place
1277 * where it got up to last time, and through RING_TAIL we tell the CS
1278 * where we want to execute up to this time.
1279 *
1280 * In this list the requests are in order of execution. Consecutive
1281 * requests from the same context are adjacent in the ringbuffer. We
1282 * can combine these requests into a single RING_TAIL update:
1283 *
1284 * RING_HEAD...req1...req2
1285 * ^- RING_TAIL
1286 * since to execute req2 the CS must first execute req1.
1287 *
1288 * Our goal then is to point each port to the end of a consecutive
1289 * sequence of requests as being the most optimal (fewest wake ups
1290 * and context switches) submission.
1291 */
1292
1293 spin_lock(&engine->active.lock);
1294
1295 /*
1296 * If the queue is higher priority than the last
1297 * request in the currently active context, submit afresh.
1298 * We will resubmit again afterwards in case we need to split
1299 * the active context to interject the preemption request,
1300 * i.e. we will retrigger preemption following the ack in case
1301 * of trouble.
1302 *
1303 * In theory we can skip over completed contexts that have not
1304 * yet been processed by events (as those events are in flight):
1305 *
1306 * while ((last = *active) && i915_request_completed(last))
1307 * active++;
1308 *
1309 * However, the GPU cannot handle this as it will ultimately
1310 * find itself trying to jump back into a context it has just
1311 * completed and barf.
1312 */
1313
1314 if (last) {
1315 if (__i915_request_is_complete(last)) {
1316 goto check_secondary;
1317 } else if (need_preempt(engine, last)) {
1318 ENGINE_TRACE(engine,
1319 "preempting last=%llx:%lld, prio=%d, hint=%d\n",
1320 last->fence.context,
1321 last->fence.seqno,
1322 last->sched.attr.priority,
1323 execlists->queue_priority_hint);
1324 record_preemption(execlists);
1325
1326 /*
1327 * Don't let the RING_HEAD advance past the breadcrumb
1328 * as we unwind (and until we resubmit) so that we do
1329 * not accidentally tell it to go backwards.
1330 */
1331 ring_set_paused(engine, 1);
1332
1333 /*
1334 * Note that we have not stopped the GPU at this point,
1335 * so we are unwinding the incomplete requests as they
1336 * remain inflight and so by the time we do complete
1337 * the preemption, some of the unwound requests may
1338 * complete!
1339 */
1340 __unwind_incomplete_requests(engine);
1341
1342 last = NULL;
1343 } else if (timeslice_expired(engine, last)) {
1344 ENGINE_TRACE(engine,
1345 "expired:%s last=%llx:%lld, prio=%d, hint=%d, yield?=%s\n",
1346 yesno(timer_expired(&execlists->timer)),
1347 last->fence.context, last->fence.seqno,
1348 rq_prio(last),
1349 execlists->queue_priority_hint,
1350 yesno(timeslice_yield(execlists, last)));
1351
1352 /*
1353 * Consume this timeslice; ensure we start a new one.
1354 *
1355 * The timeslice expired, and we will unwind the
1356 * running contexts and recompute the next ELSP.
1357 * If that submit will be the same pair of contexts
1358 * (due to dependency ordering), we will skip the
1359 * submission. If we don't cancel the timer now,
1360 * we will see that the timer has expired and
1361 * reschedule the tasklet; continually until the
1362 * next context switch or other preeemption event.
1363 *
1364 * Since we have decided to reschedule based on
1365 * consumption of this timeslice, if we submit the
1366 * same context again, grant it a full timeslice.
1367 */
1368 cancel_timer(&execlists->timer);
1369 ring_set_paused(engine, 1);
1370 defer_active(engine);
1371
1372 /*
1373 * Unlike for preemption, if we rewind and continue
1374 * executing the same context as previously active,
1375 * the order of execution will remain the same and
1376 * the tail will only advance. We do not need to
1377 * force a full context restore, as a lite-restore
1378 * is sufficient to resample the monotonic TAIL.
1379 *
1380 * If we switch to any other context, similarly we
1381 * will not rewind TAIL of current context, and
1382 * normal save/restore will preserve state and allow
1383 * us to later continue executing the same request.
1384 */
1385 last = NULL;
1386 } else {
1387 /*
1388 * Otherwise if we already have a request pending
1389 * for execution after the current one, we can
1390 * just wait until the next CS event before
1391 * queuing more. In either case we will force a
1392 * lite-restore preemption event, but if we wait
1393 * we hopefully coalesce several updates into a single
1394 * submission.
1395 */
1396check_secondary:
1397 if (!list_is_last(&last->sched.link,
1398 &engine->active.requests)) {
1399 /*
1400 * Even if ELSP[1] is occupied and not worthy
1401 * of timeslices, our queue might be.
1402 */
1403 spin_unlock(&engine->active.lock);
1404 return;
1405 }
1406 }
1407 }
1408
1409 /* XXX virtual is always taking precedence */
1410 while ((ve = first_virtual_engine(engine))) {
1411 struct i915_request *rq;
1412
1413 spin_lock(&ve->base.active.lock);
1414
1415 rq = ve->request;
1416 if (unlikely(!virtual_matches(ve, rq, engine)))
1417 goto unlock; /* lost the race to a sibling */
1418
1419 GEM_BUG_ON(rq->engine != &ve->base);
1420 GEM_BUG_ON(rq->context != &ve->context);
1421
1422 if (unlikely(rq_prio(rq) < queue_prio(execlists))) {
1423 spin_unlock(&ve->base.active.lock);
1424 break;
1425 }
1426
1427 if (last && !can_merge_rq(last, rq)) {
1428 spin_unlock(&ve->base.active.lock);
1429 spin_unlock(&engine->active.lock);
1430 return; /* leave this for another sibling */
1431 }
1432
1433 ENGINE_TRACE(engine,
1434 "virtual rq=%llx:%lld%s, new engine? %s\n",
1435 rq->fence.context,
1436 rq->fence.seqno,
1437 __i915_request_is_complete(rq) ? "!" :
1438 __i915_request_has_started(rq) ? "*" :
1439 "",
1440 yesno(engine != ve->siblings[0]));
1441
1442 WRITE_ONCE(ve->request, NULL);
1443 WRITE_ONCE(ve->base.execlists.queue_priority_hint, INT_MIN);
1444
1445 rb = &ve->nodes[engine->id].rb;
1446 rb_erase_cached(rb, &execlists->virtual);
1447 RB_CLEAR_NODE(rb);
1448
1449 GEM_BUG_ON(!(rq->execution_mask & engine->mask));
1450 WRITE_ONCE(rq->engine, engine);
1451
1452 if (__i915_request_submit(rq)) {
1453 /*
1454 * Only after we confirm that we will submit
1455 * this request (i.e. it has not already
1456 * completed), do we want to update the context.
1457 *
1458 * This serves two purposes. It avoids
1459 * unnecessary work if we are resubmitting an
1460 * already completed request after timeslicing.
1461 * But more importantly, it prevents us altering
1462 * ve->siblings[] on an idle context, where
1463 * we may be using ve->siblings[] in
1464 * virtual_context_enter / virtual_context_exit.
1465 */
1466 virtual_xfer_context(ve, engine);
1467 GEM_BUG_ON(ve->siblings[0] != engine);
1468
1469 submit = true;
1470 last = rq;
1471 }
1472
1473 i915_request_put(rq);
1474unlock:
1475 spin_unlock(&ve->base.active.lock);
1476
1477 /*
1478 * Hmm, we have a bunch of virtual engine requests,
1479 * but the first one was already completed (thanks
1480 * preempt-to-busy!). Keep looking at the veng queue
1481 * until we have no more relevant requests (i.e.
1482 * the normal submit queue has higher priority).
1483 */
1484 if (submit)
1485 break;
1486 }
1487
1488 while ((rb = rb_first_cached(&execlists->queue))) {
1489 struct i915_priolist *p = to_priolist(rb);
1490 struct i915_request *rq, *rn;
1491 int i;
1492
1493 priolist_for_each_request_consume(rq, rn, p, i) {
1494 bool merge = true;
1495
1496 /*
1497 * Can we combine this request with the current port?
1498 * It has to be the same context/ringbuffer and not
1499 * have any exceptions (e.g. GVT saying never to
1500 * combine contexts).
1501 *
1502 * If we can combine the requests, we can execute both
1503 * by updating the RING_TAIL to point to the end of the
1504 * second request, and so we never need to tell the
1505 * hardware about the first.
1506 */
1507 if (last && !can_merge_rq(last, rq)) {
1508 /*
1509 * If we are on the second port and cannot
1510 * combine this request with the last, then we
1511 * are done.
1512 */
1513 if (port == last_port)
1514 goto done;
1515
1516 /*
1517 * We must not populate both ELSP[] with the
1518 * same LRCA, i.e. we must submit 2 different
1519 * contexts if we submit 2 ELSP.
1520 */
1521 if (last->context == rq->context)
1522 goto done;
1523
1524 if (i915_request_has_sentinel(last))
1525 goto done;
1526
1527 /*
1528 * We avoid submitting virtual requests into
1529 * the secondary ports so that we can migrate
1530 * the request immediately to another engine
1531 * rather than wait for the primary request.
1532 */
1533 if (rq->execution_mask != engine->mask)
1534 goto done;
1535
1536 /*
1537 * If GVT overrides us we only ever submit
1538 * port[0], leaving port[1] empty. Note that we
1539 * also have to be careful that we don't queue
1540 * the same context (even though a different
1541 * request) to the second port.
1542 */
1543 if (ctx_single_port_submission(last->context) ||
1544 ctx_single_port_submission(rq->context))
1545 goto done;
1546
1547 merge = false;
1548 }
1549
1550 if (__i915_request_submit(rq)) {
1551 if (!merge) {
1552 *port++ = i915_request_get(last);
1553 last = NULL;
1554 }
1555
1556 GEM_BUG_ON(last &&
1557 !can_merge_ctx(last->context,
1558 rq->context));
1559 GEM_BUG_ON(last &&
1560 i915_seqno_passed(last->fence.seqno,
1561 rq->fence.seqno));
1562
1563 submit = true;
1564 last = rq;
1565 }
1566 }
1567
1568 rb_erase_cached(&p->node, &execlists->queue);
1569 i915_priolist_free(p);
1570 }
1571done:
1572 *port++ = i915_request_get(last);
1573
1574 /*
1575 * Here be a bit of magic! Or sleight-of-hand, whichever you prefer.
1576 *
1577 * We choose the priority hint such that if we add a request of greater
1578 * priority than this, we kick the submission tasklet to decide on
1579 * the right order of submitting the requests to hardware. We must
1580 * also be prepared to reorder requests as they are in-flight on the
1581 * HW. We derive the priority hint then as the first "hole" in
1582 * the HW submission ports and if there are no available slots,
1583 * the priority of the lowest executing request, i.e. last.
1584 *
1585 * When we do receive a higher priority request ready to run from the
1586 * user, see queue_request(), the priority hint is bumped to that
1587 * request triggering preemption on the next dequeue (or subsequent
1588 * interrupt for secondary ports).
1589 */
1590 execlists->queue_priority_hint = queue_prio(execlists);
1591 spin_unlock(&engine->active.lock);
1592
1593 /*
1594 * We can skip poking the HW if we ended up with exactly the same set
1595 * of requests as currently running, e.g. trying to timeslice a pair
1596 * of ordered contexts.
1597 */
1598 if (submit &&
1599 memcmp(execlists->active,
1600 execlists->pending,
1601 (port - execlists->pending) * sizeof(*port))) {
1602 *port = NULL;
1603 while (port-- != execlists->pending)
1604 execlists_schedule_in(*port, port - execlists->pending);
1605
1606 WRITE_ONCE(execlists->yield, -1);
1607 set_preempt_timeout(engine, *execlists->active);
1608 execlists_submit_ports(engine);
1609 } else {
1610 ring_set_paused(engine, 0);
1611 while (port-- != execlists->pending)
1612 i915_request_put(*port);
1613 *execlists->pending = NULL;
1614 }
1615}
1616
1617static void execlists_dequeue_irq(struct intel_engine_cs *engine)
1618{
1619 local_irq_disable(); /* Suspend interrupts across request submission */
1620 execlists_dequeue(engine);
1621 local_irq_enable(); /* flush irq_work (e.g. breadcrumb enabling) */
1622}
1623
1624static inline void clear_ports(struct i915_request **ports, int count)
1625{
1626 memset_p((void **)ports, NULL, count);
1627}
1628
1629static inline void
1630copy_ports(struct i915_request **dst, struct i915_request **src, int count)
1631{
1632 /* A memcpy_p() would be very useful here! */
1633 while (count--)
1634 WRITE_ONCE(*dst++, *src++); /* avoid write tearing */
1635}
1636
1637static struct i915_request **
1638cancel_port_requests(struct intel_engine_execlists * const execlists,
1639 struct i915_request **inactive)
1640{
1641 struct i915_request * const *port;
1642
1643 for (port = execlists->pending; *port; port++)
1644 *inactive++ = *port;
1645 clear_ports(execlists->pending, ARRAY_SIZE(execlists->pending));
1646
1647 /* Mark the end of active before we overwrite *active */
1648 for (port = xchg(&execlists->active, execlists->pending); *port; port++)
1649 *inactive++ = *port;
1650 clear_ports(execlists->inflight, ARRAY_SIZE(execlists->inflight));
1651
1652 smp_wmb(); /* complete the seqlock for execlists_active() */
1653 WRITE_ONCE(execlists->active, execlists->inflight);
1654
1655 /* Having cancelled all outstanding process_csb(), stop their timers */
1656 GEM_BUG_ON(execlists->pending[0]);
1657 cancel_timer(&execlists->timer);
1658 cancel_timer(&execlists->preempt);
1659
1660 return inactive;
1661}
1662
1663static inline void
1664invalidate_csb_entries(const u64 *first, const u64 *last)
1665{
1666 clflush((void *)first);
1667 clflush((void *)last);
1668}
1669
1670/*
1671 * Starting with Gen12, the status has a new format:
1672 *
1673 * bit 0: switched to new queue
1674 * bit 1: reserved
1675 * bit 2: semaphore wait mode (poll or signal), only valid when
1676 * switch detail is set to "wait on semaphore"
1677 * bits 3-5: engine class
1678 * bits 6-11: engine instance
1679 * bits 12-14: reserved
1680 * bits 15-25: sw context id of the lrc the GT switched to
1681 * bits 26-31: sw counter of the lrc the GT switched to
1682 * bits 32-35: context switch detail
1683 * - 0: ctx complete
1684 * - 1: wait on sync flip
1685 * - 2: wait on vblank
1686 * - 3: wait on scanline
1687 * - 4: wait on semaphore
1688 * - 5: context preempted (not on SEMAPHORE_WAIT or
1689 * WAIT_FOR_EVENT)
1690 * bit 36: reserved
1691 * bits 37-43: wait detail (for switch detail 1 to 4)
1692 * bits 44-46: reserved
1693 * bits 47-57: sw context id of the lrc the GT switched away from
1694 * bits 58-63: sw counter of the lrc the GT switched away from
1695 */
1696static inline bool gen12_csb_parse(const u64 csb)
1697{
1698 bool ctx_away_valid = GEN12_CSB_CTX_VALID(upper_32_bits(csb));
1699 bool new_queue =
1700 lower_32_bits(csb) & GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE;
1701
1702 /*
1703 * The context switch detail is not guaranteed to be 5 when a preemption
1704 * occurs, so we can't just check for that. The check below works for
1705 * all the cases we care about, including preemptions of WAIT
1706 * instructions and lite-restore. Preempt-to-idle via the CTRL register
1707 * would require some extra handling, but we don't support that.
1708 */
1709 if (!ctx_away_valid || new_queue) {
1710 GEM_BUG_ON(!GEN12_CSB_CTX_VALID(lower_32_bits(csb)));
1711 return true;
1712 }
1713
1714 /*
1715 * switch detail = 5 is covered by the case above and we do not expect a
1716 * context switch on an unsuccessful wait instruction since we always
1717 * use polling mode.
1718 */
1719 GEM_BUG_ON(GEN12_CTX_SWITCH_DETAIL(upper_32_bits(csb)));
1720 return false;
1721}
1722
1723static inline bool gen8_csb_parse(const u64 csb)
1724{
1725 return csb & (GEN8_CTX_STATUS_IDLE_ACTIVE | GEN8_CTX_STATUS_PREEMPTED);
1726}
1727
1728static noinline u64
1729wa_csb_read(const struct intel_engine_cs *engine, u64 * const csb)
1730{
1731 u64 entry;
1732
1733 /*
1734 * Reading from the HWSP has one particular advantage: we can detect
1735 * a stale entry. Since the write into HWSP is broken, we have no reason
1736 * to trust the HW at all, the mmio entry may equally be unordered, so
1737 * we prefer the path that is self-checking and as a last resort,
1738 * return the mmio value.
1739 *
1740 * tgl,dg1:HSDES#22011327657
1741 */
1742 preempt_disable();
1743 if (wait_for_atomic_us((entry = READ_ONCE(*csb)) != -1, 10)) {
1744 int idx = csb - engine->execlists.csb_status;
1745 int status;
1746
1747 status = GEN8_EXECLISTS_STATUS_BUF;
1748 if (idx >= 6) {
1749 status = GEN11_EXECLISTS_STATUS_BUF2;
1750 idx -= 6;
1751 }
1752 status += sizeof(u64) * idx;
1753
1754 entry = intel_uncore_read64(engine->uncore,
1755 _MMIO(engine->mmio_base + status));
1756 }
1757 preempt_enable();
1758
1759 return entry;
1760}
1761
1762static inline u64
1763csb_read(const struct intel_engine_cs *engine, u64 * const csb)
1764{
1765 u64 entry = READ_ONCE(*csb);
1766
1767 /*
1768 * Unfortunately, the GPU does not always serialise its write
1769 * of the CSB entries before its write of the CSB pointer, at least
1770 * from the perspective of the CPU, using what is known as a Global
1771 * Observation Point. We may read a new CSB tail pointer, but then
1772 * read the stale CSB entries, causing us to misinterpret the
1773 * context-switch events, and eventually declare the GPU hung.
1774 *
1775 * icl:HSDES#1806554093
1776 * tgl:HSDES#22011248461
1777 */
1778 if (unlikely(entry == -1))
1779 entry = wa_csb_read(engine, csb);
1780
1781 /* Consume this entry so that we can spot its future reuse. */
1782 WRITE_ONCE(*csb, -1);
1783
1784 /* ELSP is an implicit wmb() before the GPU wraps and overwrites csb */
1785 return entry;
1786}
1787
1788static void new_timeslice(struct intel_engine_execlists *el)
1789{
1790 /* By cancelling, we will start afresh in start_timeslice() */
1791 cancel_timer(&el->timer);
1792}
1793
1794static struct i915_request **
1795process_csb(struct intel_engine_cs *engine, struct i915_request **inactive)
1796{
1797 struct intel_engine_execlists * const execlists = &engine->execlists;
1798 u64 * const buf = execlists->csb_status;
1799 const u8 num_entries = execlists->csb_size;
1800 struct i915_request **prev;
1801 u8 head, tail;
1802
1803 /*
1804 * As we modify our execlists state tracking we require exclusive
1805 * access. Either we are inside the tasklet, or the tasklet is disabled
1806 * and we assume that is only inside the reset paths and so serialised.
1807 */
1808 GEM_BUG_ON(!tasklet_is_locked(&execlists->tasklet) &&
1809 !reset_in_progress(execlists));
1810 GEM_BUG_ON(!intel_engine_in_execlists_submission_mode(engine));
1811
1812 /*
1813 * Note that csb_write, csb_status may be either in HWSP or mmio.
1814 * When reading from the csb_write mmio register, we have to be
1815 * careful to only use the GEN8_CSB_WRITE_PTR portion, which is
1816 * the low 4bits. As it happens we know the next 4bits are always
1817 * zero and so we can simply masked off the low u8 of the register
1818 * and treat it identically to reading from the HWSP (without having
1819 * to use explicit shifting and masking, and probably bifurcating
1820 * the code to handle the legacy mmio read).
1821 */
1822 head = execlists->csb_head;
1823 tail = READ_ONCE(*execlists->csb_write);
1824 if (unlikely(head == tail))
1825 return inactive;
1826
1827 /*
1828 * We will consume all events from HW, or at least pretend to.
1829 *
1830 * The sequence of events from the HW is deterministic, and derived
1831 * from our writes to the ELSP, with a smidgen of variability for
1832 * the arrival of the asynchronous requests wrt to the inflight
1833 * execution. If the HW sends an event that does not correspond with
1834 * the one we are expecting, we have to abandon all hope as we lose
1835 * all tracking of what the engine is actually executing. We will
1836 * only detect we are out of sequence with the HW when we get an
1837 * 'impossible' event because we have already drained our own
1838 * preemption/promotion queue. If this occurs, we know that we likely
1839 * lost track of execution earlier and must unwind and restart, the
1840 * simplest way is by stop processing the event queue and force the
1841 * engine to reset.
1842 */
1843 execlists->csb_head = tail;
1844 ENGINE_TRACE(engine, "cs-irq head=%d, tail=%d\n", head, tail);
1845
1846 /*
1847 * Hopefully paired with a wmb() in HW!
1848 *
1849 * We must complete the read of the write pointer before any reads
1850 * from the CSB, so that we do not see stale values. Without an rmb
1851 * (lfence) the HW may speculatively perform the CSB[] reads *before*
1852 * we perform the READ_ONCE(*csb_write).
1853 */
1854 rmb();
1855
1856 /* Remember who was last running under the timer */
1857 prev = inactive;
1858 *prev = NULL;
1859
1860 do {
1861 bool promote;
1862 u64 csb;
1863
1864 if (++head == num_entries)
1865 head = 0;
1866
1867 /*
1868 * We are flying near dragons again.
1869 *
1870 * We hold a reference to the request in execlist_port[]
1871 * but no more than that. We are operating in softirq
1872 * context and so cannot hold any mutex or sleep. That
1873 * prevents us stopping the requests we are processing
1874 * in port[] from being retired simultaneously (the
1875 * breadcrumb will be complete before we see the
1876 * context-switch). As we only hold the reference to the
1877 * request, any pointer chasing underneath the request
1878 * is subject to a potential use-after-free. Thus we
1879 * store all of the bookkeeping within port[] as
1880 * required, and avoid using unguarded pointers beneath
1881 * request itself. The same applies to the atomic
1882 * status notifier.
1883 */
1884
1885 csb = csb_read(engine, buf + head);
1886 ENGINE_TRACE(engine, "csb[%d]: status=0x%08x:0x%08x\n",
1887 head, upper_32_bits(csb), lower_32_bits(csb));
1888
1889 if (INTEL_GEN(engine->i915) >= 12)
1890 promote = gen12_csb_parse(csb);
1891 else
1892 promote = gen8_csb_parse(csb);
1893 if (promote) {
1894 struct i915_request * const *old = execlists->active;
1895
1896 if (GEM_WARN_ON(!*execlists->pending)) {
1897 execlists->error_interrupt |= ERROR_CSB;
1898 break;
1899 }
1900
1901 ring_set_paused(engine, 0);
1902
1903 /* Point active to the new ELSP; prevent overwriting */
1904 WRITE_ONCE(execlists->active, execlists->pending);
1905 smp_wmb(); /* notify execlists_active() */
1906
1907 /* cancel old inflight, prepare for switch */
1908 trace_ports(execlists, "preempted", old);
1909 while (*old)
1910 *inactive++ = *old++;
1911
1912 /* switch pending to inflight */
1913 GEM_BUG_ON(!assert_pending_valid(execlists, "promote"));
1914 copy_ports(execlists->inflight,
1915 execlists->pending,
1916 execlists_num_ports(execlists));
1917 smp_wmb(); /* complete the seqlock */
1918 WRITE_ONCE(execlists->active, execlists->inflight);
1919
1920 /* XXX Magic delay for tgl */
1921 ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
1922
1923 WRITE_ONCE(execlists->pending[0], NULL);
1924 } else {
1925 if (GEM_WARN_ON(!*execlists->active)) {
1926 execlists->error_interrupt |= ERROR_CSB;
1927 break;
1928 }
1929
1930 /* port0 completed, advanced to port1 */
1931 trace_ports(execlists, "completed", execlists->active);
1932
1933 /*
1934 * We rely on the hardware being strongly
1935 * ordered, that the breadcrumb write is
1936 * coherent (visible from the CPU) before the
1937 * user interrupt is processed. One might assume
1938 * that the breadcrumb write being before the
1939 * user interrupt and the CS event for the context
1940 * switch would therefore be before the CS event
1941 * itself...
1942 */
1943 if (GEM_SHOW_DEBUG() &&
1944 !__i915_request_is_complete(*execlists->active)) {
1945 struct i915_request *rq = *execlists->active;
1946 const u32 *regs __maybe_unused =
1947 rq->context->lrc_reg_state;
1948
1949 ENGINE_TRACE(engine,
1950 "context completed before request!\n");
1951 ENGINE_TRACE(engine,
1952 "ring:{start:0x%08x, head:%04x, tail:%04x, ctl:%08x, mode:%08x}\n",
1953 ENGINE_READ(engine, RING_START),
1954 ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR,
1955 ENGINE_READ(engine, RING_TAIL) & TAIL_ADDR,
1956 ENGINE_READ(engine, RING_CTL),
1957 ENGINE_READ(engine, RING_MI_MODE));
1958 ENGINE_TRACE(engine,
1959 "rq:{start:%08x, head:%04x, tail:%04x, seqno:%llx:%d, hwsp:%d}, ",
1960 i915_ggtt_offset(rq->ring->vma),
1961 rq->head, rq->tail,
1962 rq->fence.context,
1963 lower_32_bits(rq->fence.seqno),
1964 hwsp_seqno(rq));
1965 ENGINE_TRACE(engine,
1966 "ctx:{start:%08x, head:%04x, tail:%04x}, ",
1967 regs[CTX_RING_START],
1968 regs[CTX_RING_HEAD],
1969 regs[CTX_RING_TAIL]);
1970 }
1971
1972 *inactive++ = *execlists->active++;
1973
1974 GEM_BUG_ON(execlists->active - execlists->inflight >
1975 execlists_num_ports(execlists));
1976 }
1977 } while (head != tail);
1978
1979 /*
1980 * Gen11 has proven to fail wrt global observation point between
1981 * entry and tail update, failing on the ordering and thus
1982 * we see an old entry in the context status buffer.
1983 *
1984 * Forcibly evict out entries for the next gpu csb update,
1985 * to increase the odds that we get a fresh entries with non
1986 * working hardware. The cost for doing so comes out mostly with
1987 * the wash as hardware, working or not, will need to do the
1988 * invalidation before.
1989 */
1990 invalidate_csb_entries(&buf[0], &buf[num_entries - 1]);
1991
1992 /*
1993 * We assume that any event reflects a change in context flow
1994 * and merits a fresh timeslice. We reinstall the timer after
1995 * inspecting the queue to see if we need to resumbit.
1996 */
1997 if (*prev != *execlists->active) /* elide lite-restores */
1998 new_timeslice(execlists);
1999
2000 return inactive;
2001}
2002
2003static void post_process_csb(struct i915_request **port,
2004 struct i915_request **last)
2005{
2006 while (port != last)
2007 execlists_schedule_out(*port++);
2008}
2009
2010static void __execlists_hold(struct i915_request *rq)
2011{
2012 LIST_HEAD(list);
2013
2014 do {
2015 struct i915_dependency *p;
2016
2017 if (i915_request_is_active(rq))
2018 __i915_request_unsubmit(rq);
2019
2020 clear_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2021 list_move_tail(&rq->sched.link, &rq->engine->active.hold);
2022 i915_request_set_hold(rq);
2023 RQ_TRACE(rq, "on hold\n");
2024
2025 for_each_waiter(p, rq) {
2026 struct i915_request *w =
2027 container_of(p->waiter, typeof(*w), sched);
2028
2029 /* Leave semaphores spinning on the other engines */
2030 if (w->engine != rq->engine)
2031 continue;
2032
2033 if (!i915_request_is_ready(w))
2034 continue;
2035
2036 if (__i915_request_is_complete(w))
2037 continue;
2038
2039 if (i915_request_on_hold(w))
2040 continue;
2041
2042 list_move_tail(&w->sched.link, &list);
2043 }
2044
2045 rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2046 } while (rq);
2047}
2048
2049static bool execlists_hold(struct intel_engine_cs *engine,
2050 struct i915_request *rq)
2051{
2052 if (i915_request_on_hold(rq))
2053 return false;
2054
2055 spin_lock_irq(&engine->active.lock);
2056
2057 if (__i915_request_is_complete(rq)) { /* too late! */
2058 rq = NULL;
2059 goto unlock;
2060 }
2061
2062 /*
2063 * Transfer this request onto the hold queue to prevent it
2064 * being resumbitted to HW (and potentially completed) before we have
2065 * released it. Since we may have already submitted following
2066 * requests, we need to remove those as well.
2067 */
2068 GEM_BUG_ON(i915_request_on_hold(rq));
2069 GEM_BUG_ON(rq->engine != engine);
2070 __execlists_hold(rq);
2071 GEM_BUG_ON(list_empty(&engine->active.hold));
2072
2073unlock:
2074 spin_unlock_irq(&engine->active.lock);
2075 return rq;
2076}
2077
2078static bool hold_request(const struct i915_request *rq)
2079{
2080 struct i915_dependency *p;
2081 bool result = false;
2082
2083 /*
2084 * If one of our ancestors is on hold, we must also be on hold,
2085 * otherwise we will bypass it and execute before it.
2086 */
2087 rcu_read_lock();
2088 for_each_signaler(p, rq) {
2089 const struct i915_request *s =
2090 container_of(p->signaler, typeof(*s), sched);
2091
2092 if (s->engine != rq->engine)
2093 continue;
2094
2095 result = i915_request_on_hold(s);
2096 if (result)
2097 break;
2098 }
2099 rcu_read_unlock();
2100
2101 return result;
2102}
2103
2104static void __execlists_unhold(struct i915_request *rq)
2105{
2106 LIST_HEAD(list);
2107
2108 do {
2109 struct i915_dependency *p;
2110
2111 RQ_TRACE(rq, "hold release\n");
2112
2113 GEM_BUG_ON(!i915_request_on_hold(rq));
2114 GEM_BUG_ON(!i915_sw_fence_signaled(&rq->submit));
2115
2116 i915_request_clear_hold(rq);
2117 list_move_tail(&rq->sched.link,
2118 i915_sched_lookup_priolist(rq->engine,
2119 rq_prio(rq)));
2120 set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2121
2122 /* Also release any children on this engine that are ready */
2123 for_each_waiter(p, rq) {
2124 struct i915_request *w =
2125 container_of(p->waiter, typeof(*w), sched);
2126
2127 /* Propagate any change in error status */
2128 if (rq->fence.error)
2129 i915_request_set_error_once(w, rq->fence.error);
2130
2131 if (w->engine != rq->engine)
2132 continue;
2133
2134 if (!i915_request_on_hold(w))
2135 continue;
2136
2137 /* Check that no other parents are also on hold */
2138 if (hold_request(w))
2139 continue;
2140
2141 list_move_tail(&w->sched.link, &list);
2142 }
2143
2144 rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2145 } while (rq);
2146}
2147
2148static void execlists_unhold(struct intel_engine_cs *engine,
2149 struct i915_request *rq)
2150{
2151 spin_lock_irq(&engine->active.lock);
2152
2153 /*
2154 * Move this request back to the priority queue, and all of its
2155 * children and grandchildren that were suspended along with it.
2156 */
2157 __execlists_unhold(rq);
2158
2159 if (rq_prio(rq) > engine->execlists.queue_priority_hint) {
2160 engine->execlists.queue_priority_hint = rq_prio(rq);
2161 tasklet_hi_schedule(&engine->execlists.tasklet);
2162 }
2163
2164 spin_unlock_irq(&engine->active.lock);
2165}
2166
2167struct execlists_capture {
2168 struct work_struct work;
2169 struct i915_request *rq;
2170 struct i915_gpu_coredump *error;
2171};
2172
2173static void execlists_capture_work(struct work_struct *work)
2174{
2175 struct execlists_capture *cap = container_of(work, typeof(*cap), work);
2176 const gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
2177 struct intel_engine_cs *engine = cap->rq->engine;
2178 struct intel_gt_coredump *gt = cap->error->gt;
2179 struct intel_engine_capture_vma *vma;
2180
2181 /* Compress all the objects attached to the request, slow! */
2182 vma = intel_engine_coredump_add_request(gt->engine, cap->rq, gfp);
2183 if (vma) {
2184 struct i915_vma_compress *compress =
2185 i915_vma_capture_prepare(gt);
2186
2187 intel_engine_coredump_add_vma(gt->engine, vma, compress);
2188 i915_vma_capture_finish(gt, compress);
2189 }
2190
2191 gt->simulated = gt->engine->simulated;
2192 cap->error->simulated = gt->simulated;
2193
2194 /* Publish the error state, and announce it to the world */
2195 i915_error_state_store(cap->error);
2196 i915_gpu_coredump_put(cap->error);
2197
2198 /* Return this request and all that depend upon it for signaling */
2199 execlists_unhold(engine, cap->rq);
2200 i915_request_put(cap->rq);
2201
2202 kfree(cap);
2203}
2204
2205static struct execlists_capture *capture_regs(struct intel_engine_cs *engine)
2206{
2207 const gfp_t gfp = GFP_ATOMIC | __GFP_NOWARN;
2208 struct execlists_capture *cap;
2209
2210 cap = kmalloc(sizeof(*cap), gfp);
2211 if (!cap)
2212 return NULL;
2213
2214 cap->error = i915_gpu_coredump_alloc(engine->i915, gfp);
2215 if (!cap->error)
2216 goto err_cap;
2217
2218 cap->error->gt = intel_gt_coredump_alloc(engine->gt, gfp);
2219 if (!cap->error->gt)
2220 goto err_gpu;
2221
2222 cap->error->gt->engine = intel_engine_coredump_alloc(engine, gfp);
2223 if (!cap->error->gt->engine)
2224 goto err_gt;
2225
2226 cap->error->gt->engine->hung = true;
2227
2228 return cap;
2229
2230err_gt:
2231 kfree(cap->error->gt);
2232err_gpu:
2233 kfree(cap->error);
2234err_cap:
2235 kfree(cap);
2236 return NULL;
2237}
2238
2239static struct i915_request *
2240active_context(struct intel_engine_cs *engine, u32 ccid)
2241{
2242 const struct intel_engine_execlists * const el = &engine->execlists;
2243 struct i915_request * const *port, *rq;
2244
2245 /*
2246 * Use the most recent result from process_csb(), but just in case
2247 * we trigger an error (via interrupt) before the first CS event has
2248 * been written, peek at the next submission.
2249 */
2250
2251 for (port = el->active; (rq = *port); port++) {
2252 if (rq->context->lrc.ccid == ccid) {
2253 ENGINE_TRACE(engine,
2254 "ccid:%x found at active:%zd\n",
2255 ccid, port - el->active);
2256 return rq;
2257 }
2258 }
2259
2260 for (port = el->pending; (rq = *port); port++) {
2261 if (rq->context->lrc.ccid == ccid) {
2262 ENGINE_TRACE(engine,
2263 "ccid:%x found at pending:%zd\n",
2264 ccid, port - el->pending);
2265 return rq;
2266 }
2267 }
2268
2269 ENGINE_TRACE(engine, "ccid:%x not found\n", ccid);
2270 return NULL;
2271}
2272
2273static u32 active_ccid(struct intel_engine_cs *engine)
2274{
2275 return ENGINE_READ_FW(engine, RING_EXECLIST_STATUS_HI);
2276}
2277
2278static void execlists_capture(struct intel_engine_cs *engine)
2279{
2280 struct execlists_capture *cap;
2281
2282 if (!IS_ENABLED(CONFIG_DRM_I915_CAPTURE_ERROR))
2283 return;
2284
2285 /*
2286 * We need to _quickly_ capture the engine state before we reset.
2287 * We are inside an atomic section (softirq) here and we are delaying
2288 * the forced preemption event.
2289 */
2290 cap = capture_regs(engine);
2291 if (!cap)
2292 return;
2293
2294 spin_lock_irq(&engine->active.lock);
2295 cap->rq = active_context(engine, active_ccid(engine));
2296 if (cap->rq) {
2297 cap->rq = active_request(cap->rq->context->timeline, cap->rq);
2298 cap->rq = i915_request_get_rcu(cap->rq);
2299 }
2300 spin_unlock_irq(&engine->active.lock);
2301 if (!cap->rq)
2302 goto err_free;
2303
2304 /*
2305 * Remove the request from the execlists queue, and take ownership
2306 * of the request. We pass it to our worker who will _slowly_ compress
2307 * all the pages the _user_ requested for debugging their batch, after
2308 * which we return it to the queue for signaling.
2309 *
2310 * By removing them from the execlists queue, we also remove the
2311 * requests from being processed by __unwind_incomplete_requests()
2312 * during the intel_engine_reset(), and so they will *not* be replayed
2313 * afterwards.
2314 *
2315 * Note that because we have not yet reset the engine at this point,
2316 * it is possible for the request that we have identified as being
2317 * guilty, did in fact complete and we will then hit an arbitration
2318 * point allowing the outstanding preemption to succeed. The likelihood
2319 * of that is very low (as capturing of the engine registers should be
2320 * fast enough to run inside an irq-off atomic section!), so we will
2321 * simply hold that request accountable for being non-preemptible
2322 * long enough to force the reset.
2323 */
2324 if (!execlists_hold(engine, cap->rq))
2325 goto err_rq;
2326
2327 INIT_WORK(&cap->work, execlists_capture_work);
2328 schedule_work(&cap->work);
2329 return;
2330
2331err_rq:
2332 i915_request_put(cap->rq);
2333err_free:
2334 i915_gpu_coredump_put(cap->error);
2335 kfree(cap);
2336}
2337
2338static void execlists_reset(struct intel_engine_cs *engine, const char *msg)
2339{
2340 const unsigned int bit = I915_RESET_ENGINE + engine->id;
2341 unsigned long *lock = &engine->gt->reset.flags;
2342
2343 if (!intel_has_reset_engine(engine->gt))
2344 return;
2345
2346 if (test_and_set_bit(bit, lock))
2347 return;
2348
2349 ENGINE_TRACE(engine, "reset for %s\n", msg);
2350
2351 /* Mark this tasklet as disabled to avoid waiting for it to complete */
2352 tasklet_disable_nosync(&engine->execlists.tasklet);
2353
2354 ring_set_paused(engine, 1); /* Freeze the current request in place */
2355 execlists_capture(engine);
2356 intel_engine_reset(engine, msg);
2357
2358 tasklet_enable(&engine->execlists.tasklet);
2359 clear_and_wake_up_bit(bit, lock);
2360}
2361
2362static bool preempt_timeout(const struct intel_engine_cs *const engine)
2363{
2364 const struct timer_list *t = &engine->execlists.preempt;
2365
2366 if (!CONFIG_DRM_I915_PREEMPT_TIMEOUT)
2367 return false;
2368
2369 if (!timer_expired(t))
2370 return false;
2371
2372 return engine->execlists.pending[0];
2373}
2374
2375/*
2376 * Check the unread Context Status Buffers and manage the submission of new
2377 * contexts to the ELSP accordingly.
2378 */
2379static void execlists_submission_tasklet(unsigned long data)
2380{
2381 struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
2382 struct i915_request *post[2 * EXECLIST_MAX_PORTS];
2383 struct i915_request **inactive;
2384
2385 rcu_read_lock();
2386 inactive = process_csb(engine, post);
2387 GEM_BUG_ON(inactive - post > ARRAY_SIZE(post));
2388
2389 if (unlikely(preempt_timeout(engine))) {
2390 cancel_timer(&engine->execlists.preempt);
2391 engine->execlists.error_interrupt |= ERROR_PREEMPT;
2392 }
2393
2394 if (unlikely(READ_ONCE(engine->execlists.error_interrupt))) {
2395 const char *msg;
2396
2397 /* Generate the error message in priority wrt to the user! */
2398 if (engine->execlists.error_interrupt & GENMASK(15, 0))
2399 msg = "CS error"; /* thrown by a user payload */
2400 else if (engine->execlists.error_interrupt & ERROR_CSB)
2401 msg = "invalid CSB event";
2402 else if (engine->execlists.error_interrupt & ERROR_PREEMPT)
2403 msg = "preemption time out";
2404 else
2405 msg = "internal error";
2406
2407 engine->execlists.error_interrupt = 0;
2408 execlists_reset(engine, msg);
2409 }
2410
2411 if (!engine->execlists.pending[0]) {
2412 execlists_dequeue_irq(engine);
2413 start_timeslice(engine);
2414 }
2415
2416 post_process_csb(post, inactive);
2417 rcu_read_unlock();
2418}
2419
2420static void __execlists_kick(struct intel_engine_execlists *execlists)
2421{
2422 /* Kick the tasklet for some interrupt coalescing and reset handling */
2423 tasklet_hi_schedule(&execlists->tasklet);
2424}
2425
2426#define execlists_kick(t, member) \
2427 __execlists_kick(container_of(t, struct intel_engine_execlists, member))
2428
2429static void execlists_timeslice(struct timer_list *timer)
2430{
2431 execlists_kick(timer, timer);
2432}
2433
2434static void execlists_preempt(struct timer_list *timer)
2435{
2436 execlists_kick(timer, preempt);
2437}
2438
2439static void queue_request(struct intel_engine_cs *engine,
2440 struct i915_request *rq)
2441{
2442 GEM_BUG_ON(!list_empty(&rq->sched.link));
2443 list_add_tail(&rq->sched.link,
2444 i915_sched_lookup_priolist(engine, rq_prio(rq)));
2445 set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2446}
2447
2448static bool submit_queue(struct intel_engine_cs *engine,
2449 const struct i915_request *rq)
2450{
2451 struct intel_engine_execlists *execlists = &engine->execlists;
2452
2453 if (rq_prio(rq) <= execlists->queue_priority_hint)
2454 return false;
2455
2456 execlists->queue_priority_hint = rq_prio(rq);
2457 return true;
2458}
2459
2460static bool ancestor_on_hold(const struct intel_engine_cs *engine,
2461 const struct i915_request *rq)
2462{
2463 GEM_BUG_ON(i915_request_on_hold(rq));
2464 return !list_empty(&engine->active.hold) && hold_request(rq);
2465}
2466
2467static void execlists_submit_request(struct i915_request *request)
2468{
2469 struct intel_engine_cs *engine = request->engine;
2470 unsigned long flags;
2471
2472 /* Will be called from irq-context when using foreign fences. */
2473 spin_lock_irqsave(&engine->active.lock, flags);
2474
2475 if (unlikely(ancestor_on_hold(engine, request))) {
2476 RQ_TRACE(request, "ancestor on hold\n");
2477 list_add_tail(&request->sched.link, &engine->active.hold);
2478 i915_request_set_hold(request);
2479 } else {
2480 queue_request(engine, request);
2481
2482 GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
2483 GEM_BUG_ON(list_empty(&request->sched.link));
2484
2485 if (submit_queue(engine, request))
2486 __execlists_kick(&engine->execlists);
2487 }
2488
2489 spin_unlock_irqrestore(&engine->active.lock, flags);
2490}
2491
2492static int execlists_context_pre_pin(struct intel_context *ce,
2493 struct i915_gem_ww_ctx *ww,
2494 void **vaddr)
2495{
2496 return lrc_pre_pin(ce, ce->engine, ww, vaddr);
2497}
2498
2499static int execlists_context_pin(struct intel_context *ce, void *vaddr)
2500{
2501 return lrc_pin(ce, ce->engine, vaddr);
2502}
2503
2504static int execlists_context_alloc(struct intel_context *ce)
2505{
2506 return lrc_alloc(ce, ce->engine);
2507}
2508
2509static const struct intel_context_ops execlists_context_ops = {
2510 .flags = COPS_HAS_INFLIGHT,
2511
2512 .alloc = execlists_context_alloc,
2513
2514 .pre_pin = execlists_context_pre_pin,
2515 .pin = execlists_context_pin,
2516 .unpin = lrc_unpin,
2517 .post_unpin = lrc_post_unpin,
2518
2519 .enter = intel_context_enter_engine,
2520 .exit = intel_context_exit_engine,
2521
2522 .reset = lrc_reset,
2523 .destroy = lrc_destroy,
2524};
2525
2526static int emit_pdps(struct i915_request *rq)
2527{
2528 const struct intel_engine_cs * const engine = rq->engine;
2529 struct i915_ppgtt * const ppgtt = i915_vm_to_ppgtt(rq->context->vm);
2530 int err, i;
2531 u32 *cs;
2532
2533 GEM_BUG_ON(intel_vgpu_active(rq->engine->i915));
2534
2535 /*
2536 * Beware ye of the dragons, this sequence is magic!
2537 *
2538 * Small changes to this sequence can cause anything from
2539 * GPU hangs to forcewake errors and machine lockups!
2540 */
2541
2542 cs = intel_ring_begin(rq, 2);
2543 if (IS_ERR(cs))
2544 return PTR_ERR(cs);
2545
2546 *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
2547 *cs++ = MI_NOOP;
2548 intel_ring_advance(rq, cs);
2549
2550 /* Flush any residual operations from the context load */
2551 err = engine->emit_flush(rq, EMIT_FLUSH);
2552 if (err)
2553 return err;
2554
2555 /* Magic required to prevent forcewake errors! */
2556 err = engine->emit_flush(rq, EMIT_INVALIDATE);
2557 if (err)
2558 return err;
2559
2560 cs = intel_ring_begin(rq, 4 * GEN8_3LVL_PDPES + 2);
2561 if (IS_ERR(cs))
2562 return PTR_ERR(cs);
2563
2564 /* Ensure the LRI have landed before we invalidate & continue */
2565 *cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES) | MI_LRI_FORCE_POSTED;
2566 for (i = GEN8_3LVL_PDPES; i--; ) {
2567 const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
2568 u32 base = engine->mmio_base;
2569
2570 *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i));
2571 *cs++ = upper_32_bits(pd_daddr);
2572 *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i));
2573 *cs++ = lower_32_bits(pd_daddr);
2574 }
2575 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
2576 intel_ring_advance(rq, cs);
2577
2578 intel_ring_advance(rq, cs);
2579
2580 return 0;
2581}
2582
2583static int execlists_request_alloc(struct i915_request *request)
2584{
2585 int ret;
2586
2587 GEM_BUG_ON(!intel_context_is_pinned(request->context));
2588
2589 /*
2590 * Flush enough space to reduce the likelihood of waiting after
2591 * we start building the request - in which case we will just
2592 * have to repeat work.
2593 */
2594 request->reserved_space += EXECLISTS_REQUEST_SIZE;
2595
2596 /*
2597 * Note that after this point, we have committed to using
2598 * this request as it is being used to both track the
2599 * state of engine initialisation and liveness of the
2600 * golden renderstate above. Think twice before you try
2601 * to cancel/unwind this request now.
2602 */
2603
2604 if (!i915_vm_is_4lvl(request->context->vm)) {
2605 ret = emit_pdps(request);
2606 if (ret)
2607 return ret;
2608 }
2609
2610 /* Unconditionally invalidate GPU caches and TLBs. */
2611 ret = request->engine->emit_flush(request, EMIT_INVALIDATE);
2612 if (ret)
2613 return ret;
2614
2615 request->reserved_space -= EXECLISTS_REQUEST_SIZE;
2616 return 0;
2617}
2618
2619static void reset_csb_pointers(struct intel_engine_cs *engine)
2620{
2621 struct intel_engine_execlists * const execlists = &engine->execlists;
2622 const unsigned int reset_value = execlists->csb_size - 1;
2623
2624 ring_set_paused(engine, 0);
2625
2626 /*
2627 * Sometimes Icelake forgets to reset its pointers on a GPU reset.
2628 * Bludgeon them with a mmio update to be sure.
2629 */
2630 ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
2631 0xffff << 16 | reset_value << 8 | reset_value);
2632 ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
2633
2634 /*
2635 * After a reset, the HW starts writing into CSB entry [0]. We
2636 * therefore have to set our HEAD pointer back one entry so that
2637 * the *first* entry we check is entry 0. To complicate this further,
2638 * as we don't wait for the first interrupt after reset, we have to
2639 * fake the HW write to point back to the last entry so that our
2640 * inline comparison of our cached head position against the last HW
2641 * write works even before the first interrupt.
2642 */
2643 execlists->csb_head = reset_value;
2644 WRITE_ONCE(*execlists->csb_write, reset_value);
2645 wmb(); /* Make sure this is visible to HW (paranoia?) */
2646
2647 /* Check that the GPU does indeed update the CSB entries! */
2648 memset(execlists->csb_status, -1, (reset_value + 1) * sizeof(u64));
2649 invalidate_csb_entries(&execlists->csb_status[0],
2650 &execlists->csb_status[reset_value]);
2651
2652 /* Once more for luck and our trusty paranoia */
2653 ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
2654 0xffff << 16 | reset_value << 8 | reset_value);
2655 ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
2656
2657 GEM_BUG_ON(READ_ONCE(*execlists->csb_write) != reset_value);
2658}
2659
2660static void sanitize_hwsp(struct intel_engine_cs *engine)
2661{
2662 struct intel_timeline *tl;
2663
2664 list_for_each_entry(tl, &engine->status_page.timelines, engine_link)
2665 intel_timeline_reset_seqno(tl);
2666}
2667
2668static void execlists_sanitize(struct intel_engine_cs *engine)
2669{
2670 GEM_BUG_ON(execlists_active(&engine->execlists));
2671
2672 /*
2673 * Poison residual state on resume, in case the suspend didn't!
2674 *
2675 * We have to assume that across suspend/resume (or other loss
2676 * of control) that the contents of our pinned buffers has been
2677 * lost, replaced by garbage. Since this doesn't always happen,
2678 * let's poison such state so that we more quickly spot when
2679 * we falsely assume it has been preserved.
2680 */
2681 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
2682 memset(engine->status_page.addr, POISON_INUSE, PAGE_SIZE);
2683
2684 reset_csb_pointers(engine);
2685
2686 /*
2687 * The kernel_context HWSP is stored in the status_page. As above,
2688 * that may be lost on resume/initialisation, and so we need to
2689 * reset the value in the HWSP.
2690 */
2691 sanitize_hwsp(engine);
2692
2693 /* And scrub the dirty cachelines for the HWSP */
2694 clflush_cache_range(engine->status_page.addr, PAGE_SIZE);
2695}
2696
2697static void enable_error_interrupt(struct intel_engine_cs *engine)
2698{
2699 u32 status;
2700
2701 engine->execlists.error_interrupt = 0;
2702 ENGINE_WRITE(engine, RING_EMR, ~0u);
2703 ENGINE_WRITE(engine, RING_EIR, ~0u); /* clear all existing errors */
2704
2705 status = ENGINE_READ(engine, RING_ESR);
2706 if (unlikely(status)) {
2707 drm_err(&engine->i915->drm,
2708 "engine '%s' resumed still in error: %08x\n",
2709 engine->name, status);
2710 __intel_gt_reset(engine->gt, engine->mask);
2711 }
2712
2713 /*
2714 * On current gen8+, we have 2 signals to play with
2715 *
2716 * - I915_ERROR_INSTUCTION (bit 0)
2717 *
2718 * Generate an error if the command parser encounters an invalid
2719 * instruction
2720 *
2721 * This is a fatal error.
2722 *
2723 * - CP_PRIV (bit 2)
2724 *
2725 * Generate an error on privilege violation (where the CP replaces
2726 * the instruction with a no-op). This also fires for writes into
2727 * read-only scratch pages.
2728 *
2729 * This is a non-fatal error, parsing continues.
2730 *
2731 * * there are a few others defined for odd HW that we do not use
2732 *
2733 * Since CP_PRIV fires for cases where we have chosen to ignore the
2734 * error (as the HW is validating and suppressing the mistakes), we
2735 * only unmask the instruction error bit.
2736 */
2737 ENGINE_WRITE(engine, RING_EMR, ~I915_ERROR_INSTRUCTION);
2738}
2739
2740static void enable_execlists(struct intel_engine_cs *engine)
2741{
2742 u32 mode;
2743
2744 assert_forcewakes_active(engine->uncore, FORCEWAKE_ALL);
2745
2746 intel_engine_set_hwsp_writemask(engine, ~0u); /* HWSTAM */
2747
2748 if (INTEL_GEN(engine->i915) >= 11)
2749 mode = _MASKED_BIT_ENABLE(GEN11_GFX_DISABLE_LEGACY_MODE);
2750 else
2751 mode = _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE);
2752 ENGINE_WRITE_FW(engine, RING_MODE_GEN7, mode);
2753
2754 ENGINE_WRITE_FW(engine, RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
2755
2756 ENGINE_WRITE_FW(engine,
2757 RING_HWS_PGA,
2758 i915_ggtt_offset(engine->status_page.vma));
2759 ENGINE_POSTING_READ(engine, RING_HWS_PGA);
2760
2761 enable_error_interrupt(engine);
2762}
2763
2764static bool unexpected_starting_state(struct intel_engine_cs *engine)
2765{
2766 bool unexpected = false;
2767
2768 if (ENGINE_READ_FW(engine, RING_MI_MODE) & STOP_RING) {
2769 drm_dbg(&engine->i915->drm,
2770 "STOP_RING still set in RING_MI_MODE\n");
2771 unexpected = true;
2772 }
2773
2774 return unexpected;
2775}
2776
2777static int execlists_resume(struct intel_engine_cs *engine)
2778{
2779 intel_mocs_init_engine(engine);
2780
2781 intel_breadcrumbs_reset(engine->breadcrumbs);
2782
2783 if (GEM_SHOW_DEBUG() && unexpected_starting_state(engine)) {
2784 struct drm_printer p = drm_debug_printer(__func__);
2785
2786 intel_engine_dump(engine, &p, NULL);
2787 }
2788
2789 enable_execlists(engine);
2790
2791 return 0;
2792}
2793
2794static void execlists_reset_prepare(struct intel_engine_cs *engine)
2795{
2796 struct intel_engine_execlists * const execlists = &engine->execlists;
2797
2798 ENGINE_TRACE(engine, "depth<-%d\n",
2799 atomic_read(&execlists->tasklet.count));
2800
2801 /*
2802 * Prevent request submission to the hardware until we have
2803 * completed the reset in i915_gem_reset_finish(). If a request
2804 * is completed by one engine, it may then queue a request
2805 * to a second via its execlists->tasklet *just* as we are
2806 * calling engine->resume() and also writing the ELSP.
2807 * Turning off the execlists->tasklet until the reset is over
2808 * prevents the race.
2809 */
2810 __tasklet_disable_sync_once(&execlists->tasklet);
2811 GEM_BUG_ON(!reset_in_progress(execlists));
2812
2813 /*
2814 * We stop engines, otherwise we might get failed reset and a
2815 * dead gpu (on elk). Also as modern gpu as kbl can suffer
2816 * from system hang if batchbuffer is progressing when
2817 * the reset is issued, regardless of READY_TO_RESET ack.
2818 * Thus assume it is best to stop engines on all gens
2819 * where we have a gpu reset.
2820 *
2821 * WaKBLVECSSemaphoreWaitPoll:kbl (on ALL_ENGINES)
2822 *
2823 * FIXME: Wa for more modern gens needs to be validated
2824 */
2825 ring_set_paused(engine, 1);
2826 intel_engine_stop_cs(engine);
2827
2828 engine->execlists.reset_ccid = active_ccid(engine);
2829}
2830
2831static struct i915_request **
2832reset_csb(struct intel_engine_cs *engine, struct i915_request **inactive)
2833{
2834 struct intel_engine_execlists * const execlists = &engine->execlists;
2835
2836 mb(); /* paranoia: read the CSB pointers from after the reset */
2837 clflush(execlists->csb_write);
2838 mb();
2839
2840 inactive = process_csb(engine, inactive); /* drain preemption events */
2841
2842 /* Following the reset, we need to reload the CSB read/write pointers */
2843 reset_csb_pointers(engine);
2844
2845 return inactive;
2846}
2847
2848static void
2849execlists_reset_active(struct intel_engine_cs *engine, bool stalled)
2850{
2851 struct intel_context *ce;
2852 struct i915_request *rq;
2853 u32 head;
2854
2855 /*
2856 * Save the currently executing context, even if we completed
2857 * its request, it was still running at the time of the
2858 * reset and will have been clobbered.
2859 */
2860 rq = active_context(engine, engine->execlists.reset_ccid);
2861 if (!rq)
2862 return;
2863
2864 ce = rq->context;
2865 GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
2866
2867 if (__i915_request_is_complete(rq)) {
2868 /* Idle context; tidy up the ring so we can restart afresh */
2869 head = intel_ring_wrap(ce->ring, rq->tail);
2870 goto out_replay;
2871 }
2872
2873 /* We still have requests in-flight; the engine should be active */
2874 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
2875
2876 /* Context has requests still in-flight; it should not be idle! */
2877 GEM_BUG_ON(i915_active_is_idle(&ce->active));
2878
2879 rq = active_request(ce->timeline, rq);
2880 head = intel_ring_wrap(ce->ring, rq->head);
2881 GEM_BUG_ON(head == ce->ring->tail);
2882
2883 /*
2884 * If this request hasn't started yet, e.g. it is waiting on a
2885 * semaphore, we need to avoid skipping the request or else we
2886 * break the signaling chain. However, if the context is corrupt
2887 * the request will not restart and we will be stuck with a wedged
2888 * device. It is quite often the case that if we issue a reset
2889 * while the GPU is loading the context image, that the context
2890 * image becomes corrupt.
2891 *
2892 * Otherwise, if we have not started yet, the request should replay
2893 * perfectly and we do not need to flag the result as being erroneous.
2894 */
2895 if (!__i915_request_has_started(rq))
2896 goto out_replay;
2897
2898 /*
2899 * If the request was innocent, we leave the request in the ELSP
2900 * and will try to replay it on restarting. The context image may
2901 * have been corrupted by the reset, in which case we may have
2902 * to service a new GPU hang, but more likely we can continue on
2903 * without impact.
2904 *
2905 * If the request was guilty, we presume the context is corrupt
2906 * and have to at least restore the RING register in the context
2907 * image back to the expected values to skip over the guilty request.
2908 */
2909 __i915_request_reset(rq, stalled);
2910
2911 /*
2912 * We want a simple context + ring to execute the breadcrumb update.
2913 * We cannot rely on the context being intact across the GPU hang,
2914 * so clear it and rebuild just what we need for the breadcrumb.
2915 * All pending requests for this context will be zapped, and any
2916 * future request will be after userspace has had the opportunity
2917 * to recreate its own state.
2918 */
2919out_replay:
2920 ENGINE_TRACE(engine, "replay {head:%04x, tail:%04x}\n",
2921 head, ce->ring->tail);
2922 lrc_reset_regs(ce, engine);
2923 ce->lrc.lrca = lrc_update_regs(ce, engine, head);
2924}
2925
2926static void execlists_reset_csb(struct intel_engine_cs *engine, bool stalled)
2927{
2928 struct intel_engine_execlists * const execlists = &engine->execlists;
2929 struct i915_request *post[2 * EXECLIST_MAX_PORTS];
2930 struct i915_request **inactive;
2931
2932 rcu_read_lock();
2933 inactive = reset_csb(engine, post);
2934
2935 execlists_reset_active(engine, true);
2936
2937 inactive = cancel_port_requests(execlists, inactive);
2938 post_process_csb(post, inactive);
2939 rcu_read_unlock();
2940}
2941
2942static void execlists_reset_rewind(struct intel_engine_cs *engine, bool stalled)
2943{
2944 unsigned long flags;
2945
2946 ENGINE_TRACE(engine, "\n");
2947
2948 /* Process the csb, find the guilty context and throw away */
2949 execlists_reset_csb(engine, stalled);
2950
2951 /* Push back any incomplete requests for replay after the reset. */
2952 rcu_read_lock();
2953 spin_lock_irqsave(&engine->active.lock, flags);
2954 __unwind_incomplete_requests(engine);
2955 spin_unlock_irqrestore(&engine->active.lock, flags);
2956 rcu_read_unlock();
2957}
2958
2959static void nop_submission_tasklet(unsigned long data)
2960{
2961 struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
2962
2963 /* The driver is wedged; don't process any more events. */
2964 WRITE_ONCE(engine->execlists.queue_priority_hint, INT_MIN);
2965}
2966
2967static void execlists_reset_cancel(struct intel_engine_cs *engine)
2968{
2969 struct intel_engine_execlists * const execlists = &engine->execlists;
2970 struct i915_request *rq, *rn;
2971 struct rb_node *rb;
2972 unsigned long flags;
2973
2974 ENGINE_TRACE(engine, "\n");
2975
2976 /*
2977 * Before we call engine->cancel_requests(), we should have exclusive
2978 * access to the submission state. This is arranged for us by the
2979 * caller disabling the interrupt generation, the tasklet and other
2980 * threads that may then access the same state, giving us a free hand
2981 * to reset state. However, we still need to let lockdep be aware that
2982 * we know this state may be accessed in hardirq context, so we
2983 * disable the irq around this manipulation and we want to keep
2984 * the spinlock focused on its duties and not accidentally conflate
2985 * coverage to the submission's irq state. (Similarly, although we
2986 * shouldn't need to disable irq around the manipulation of the
2987 * submission's irq state, we also wish to remind ourselves that
2988 * it is irq state.)
2989 */
2990 execlists_reset_csb(engine, true);
2991
2992 rcu_read_lock();
2993 spin_lock_irqsave(&engine->active.lock, flags);
2994
2995 /* Mark all executing requests as skipped. */
2996 list_for_each_entry(rq, &engine->active.requests, sched.link)
2997 i915_request_mark_eio(rq);
2998 intel_engine_signal_breadcrumbs(engine);
2999
3000 /* Flush the queued requests to the timeline list (for retiring). */
3001 while ((rb = rb_first_cached(&execlists->queue))) {
3002 struct i915_priolist *p = to_priolist(rb);
3003 int i;
3004
3005 priolist_for_each_request_consume(rq, rn, p, i) {
3006 i915_request_mark_eio(rq);
3007 __i915_request_submit(rq);
3008 }
3009
3010 rb_erase_cached(&p->node, &execlists->queue);
3011 i915_priolist_free(p);
3012 }
3013
3014 /* On-hold requests will be flushed to timeline upon their release */
3015 list_for_each_entry(rq, &engine->active.hold, sched.link)
3016 i915_request_mark_eio(rq);
3017
3018 /* Cancel all attached virtual engines */
3019 while ((rb = rb_first_cached(&execlists->virtual))) {
3020 struct virtual_engine *ve =
3021 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
3022
3023 rb_erase_cached(rb, &execlists->virtual);
3024 RB_CLEAR_NODE(rb);
3025
3026 spin_lock(&ve->base.active.lock);
3027 rq = fetch_and_zero(&ve->request);
3028 if (rq) {
3029 i915_request_mark_eio(rq);
3030
3031 rq->engine = engine;
3032 __i915_request_submit(rq);
3033 i915_request_put(rq);
3034
3035 ve->base.execlists.queue_priority_hint = INT_MIN;
3036 }
3037 spin_unlock(&ve->base.active.lock);
3038 }
3039
3040 /* Remaining _unready_ requests will be nop'ed when submitted */
3041
3042 execlists->queue_priority_hint = INT_MIN;
3043 execlists->queue = RB_ROOT_CACHED;
3044
3045 GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
3046 execlists->tasklet.func = nop_submission_tasklet;
3047
3048 spin_unlock_irqrestore(&engine->active.lock, flags);
3049 rcu_read_unlock();
3050}
3051
3052static void execlists_reset_finish(struct intel_engine_cs *engine)
3053{
3054 struct intel_engine_execlists * const execlists = &engine->execlists;
3055
3056 /*
3057 * After a GPU reset, we may have requests to replay. Do so now while
3058 * we still have the forcewake to be sure that the GPU is not allowed
3059 * to sleep before we restart and reload a context.
3060 *
3061 * If the GPU reset fails, the engine may still be alive with requests
3062 * inflight. We expect those to complete, or for the device to be
3063 * reset as the next level of recovery, and as a final resort we
3064 * will declare the device wedged.
3065 */
3066 GEM_BUG_ON(!reset_in_progress(execlists));
3067
3068 /* And kick in case we missed a new request submission. */
3069 if (__tasklet_enable(&execlists->tasklet))
3070 __execlists_kick(execlists);
3071
3072 ENGINE_TRACE(engine, "depth->%d\n",
3073 atomic_read(&execlists->tasklet.count));
3074}
3075
3076static void gen8_logical_ring_enable_irq(struct intel_engine_cs *engine)
3077{
3078 ENGINE_WRITE(engine, RING_IMR,
3079 ~(engine->irq_enable_mask | engine->irq_keep_mask));
3080 ENGINE_POSTING_READ(engine, RING_IMR);
3081}
3082
3083static void gen8_logical_ring_disable_irq(struct intel_engine_cs *engine)
3084{
3085 ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
3086}
3087
3088static void execlists_park(struct intel_engine_cs *engine)
3089{
3090 cancel_timer(&engine->execlists.timer);
3091 cancel_timer(&engine->execlists.preempt);
3092}
3093
3094static bool can_preempt(struct intel_engine_cs *engine)
3095{
3096 if (INTEL_GEN(engine->i915) > 8)
3097 return true;
3098
3099 /* GPGPU on bdw requires extra w/a; not implemented */
3100 return engine->class != RENDER_CLASS;
3101}
3102
3103static void execlists_set_default_submission(struct intel_engine_cs *engine)
3104{
3105 engine->submit_request = execlists_submit_request;
3106 engine->schedule = i915_schedule;
3107 engine->execlists.tasklet.func = execlists_submission_tasklet;
3108
3109 engine->reset.prepare = execlists_reset_prepare;
3110 engine->reset.rewind = execlists_reset_rewind;
3111 engine->reset.cancel = execlists_reset_cancel;
3112 engine->reset.finish = execlists_reset_finish;
3113
3114 engine->park = execlists_park;
3115 engine->unpark = NULL;
3116
3117 engine->flags |= I915_ENGINE_SUPPORTS_STATS;
3118 if (!intel_vgpu_active(engine->i915)) {
3119 engine->flags |= I915_ENGINE_HAS_SEMAPHORES;
3120 if (can_preempt(engine)) {
3121 engine->flags |= I915_ENGINE_HAS_PREEMPTION;
3122 if (IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
3123 engine->flags |= I915_ENGINE_HAS_TIMESLICES;
3124 }
3125 }
3126
3127 if (intel_engine_has_preemption(engine))
3128 engine->emit_bb_start = gen8_emit_bb_start;
3129 else
3130 engine->emit_bb_start = gen8_emit_bb_start_noarb;
3131}
3132
3133static void execlists_shutdown(struct intel_engine_cs *engine)
3134{
3135 /* Synchronise with residual timers and any softirq they raise */
3136 del_timer_sync(&engine->execlists.timer);
3137 del_timer_sync(&engine->execlists.preempt);
3138 tasklet_kill(&engine->execlists.tasklet);
3139}
3140
3141static void execlists_release(struct intel_engine_cs *engine)
3142{
3143 engine->sanitize = NULL; /* no longer in control, nothing to sanitize */
3144
3145 execlists_shutdown(engine);
3146
3147 intel_engine_cleanup_common(engine);
3148 lrc_fini_wa_ctx(engine);
3149}
3150
3151static void
3152logical_ring_default_vfuncs(struct intel_engine_cs *engine)
3153{
3154 /* Default vfuncs which can be overriden by each engine. */
3155
3156 engine->resume = execlists_resume;
3157
3158 engine->cops = &execlists_context_ops;
3159 engine->request_alloc = execlists_request_alloc;
3160
3161 engine->emit_flush = gen8_emit_flush_xcs;
3162 engine->emit_init_breadcrumb = gen8_emit_init_breadcrumb;
3163 engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb_xcs;
3164 if (INTEL_GEN(engine->i915) >= 12) {
3165 engine->emit_fini_breadcrumb = gen12_emit_fini_breadcrumb_xcs;
3166 engine->emit_flush = gen12_emit_flush_xcs;
3167 }
3168 engine->set_default_submission = execlists_set_default_submission;
3169
3170 if (INTEL_GEN(engine->i915) < 11) {
3171 engine->irq_enable = gen8_logical_ring_enable_irq;
3172 engine->irq_disable = gen8_logical_ring_disable_irq;
3173 } else {
3174 /*
3175 * TODO: On Gen11 interrupt masks need to be clear
3176 * to allow C6 entry. Keep interrupts enabled at
3177 * and take the hit of generating extra interrupts
3178 * until a more refined solution exists.
3179 */
3180 }
3181}
3182
3183static inline void
3184logical_ring_default_irqs(struct intel_engine_cs *engine)
3185{
3186 unsigned int shift = 0;
3187
3188 if (INTEL_GEN(engine->i915) < 11) {
3189 const u8 irq_shifts[] = {
3190 [RCS0] = GEN8_RCS_IRQ_SHIFT,
3191 [BCS0] = GEN8_BCS_IRQ_SHIFT,
3192 [VCS0] = GEN8_VCS0_IRQ_SHIFT,
3193 [VCS1] = GEN8_VCS1_IRQ_SHIFT,
3194 [VECS0] = GEN8_VECS_IRQ_SHIFT,
3195 };
3196
3197 shift = irq_shifts[engine->id];
3198 }
3199
3200 engine->irq_enable_mask = GT_RENDER_USER_INTERRUPT << shift;
3201 engine->irq_keep_mask = GT_CONTEXT_SWITCH_INTERRUPT << shift;
3202 engine->irq_keep_mask |= GT_CS_MASTER_ERROR_INTERRUPT << shift;
3203 engine->irq_keep_mask |= GT_WAIT_SEMAPHORE_INTERRUPT << shift;
3204}
3205
3206static void rcs_submission_override(struct intel_engine_cs *engine)
3207{
3208 switch (INTEL_GEN(engine->i915)) {
3209 case 12:
3210 engine->emit_flush = gen12_emit_flush_rcs;
3211 engine->emit_fini_breadcrumb = gen12_emit_fini_breadcrumb_rcs;
3212 break;
3213 case 11:
3214 engine->emit_flush = gen11_emit_flush_rcs;
3215 engine->emit_fini_breadcrumb = gen11_emit_fini_breadcrumb_rcs;
3216 break;
3217 default:
3218 engine->emit_flush = gen8_emit_flush_rcs;
3219 engine->emit_fini_breadcrumb = gen8_emit_fini_breadcrumb_rcs;
3220 break;
3221 }
3222}
3223
3224int intel_execlists_submission_setup(struct intel_engine_cs *engine)
3225{
3226 struct intel_engine_execlists * const execlists = &engine->execlists;
3227 struct drm_i915_private *i915 = engine->i915;
3228 struct intel_uncore *uncore = engine->uncore;
3229 u32 base = engine->mmio_base;
3230
3231 tasklet_init(&engine->execlists.tasklet,
3232 execlists_submission_tasklet, (unsigned long)engine);
3233 timer_setup(&engine->execlists.timer, execlists_timeslice, 0);
3234 timer_setup(&engine->execlists.preempt, execlists_preempt, 0);
3235
3236 logical_ring_default_vfuncs(engine);
3237 logical_ring_default_irqs(engine);
3238
3239 if (engine->class == RENDER_CLASS)
3240 rcs_submission_override(engine);
3241
3242 lrc_init_wa_ctx(engine);
3243
3244 if (HAS_LOGICAL_RING_ELSQ(i915)) {
3245 execlists->submit_reg = uncore->regs +
3246 i915_mmio_reg_offset(RING_EXECLIST_SQ_CONTENTS(base));
3247 execlists->ctrl_reg = uncore->regs +
3248 i915_mmio_reg_offset(RING_EXECLIST_CONTROL(base));
3249 } else {
3250 execlists->submit_reg = uncore->regs +
3251 i915_mmio_reg_offset(RING_ELSP(base));
3252 }
3253
3254 execlists->csb_status =
3255 (u64 *)&engine->status_page.addr[I915_HWS_CSB_BUF0_INDEX];
3256
3257 execlists->csb_write =
3258 &engine->status_page.addr[intel_hws_csb_write_index(i915)];
3259
3260 if (INTEL_GEN(i915) < 11)
3261 execlists->csb_size = GEN8_CSB_ENTRIES;
3262 else
3263 execlists->csb_size = GEN11_CSB_ENTRIES;
3264
3265 engine->context_tag = GENMASK(BITS_PER_LONG - 2, 0);
3266 if (INTEL_GEN(engine->i915) >= 11) {
3267 execlists->ccid |= engine->instance << (GEN11_ENGINE_INSTANCE_SHIFT - 32);
3268 execlists->ccid |= engine->class << (GEN11_ENGINE_CLASS_SHIFT - 32);
3269 }
3270
3271 /* Finally, take ownership and responsibility for cleanup! */
3272 engine->sanitize = execlists_sanitize;
3273 engine->release = execlists_release;
3274
3275 return 0;
3276}
3277
3278static struct list_head *virtual_queue(struct virtual_engine *ve)
3279{
3280 return &ve->base.execlists.default_priolist.requests[0];
3281}
3282
3283static void rcu_virtual_context_destroy(struct work_struct *wrk)
3284{
3285 struct virtual_engine *ve =
3286 container_of(wrk, typeof(*ve), rcu.work);
3287 unsigned int n;
3288
3289 GEM_BUG_ON(ve->context.inflight);
3290
3291 /* Preempt-to-busy may leave a stale request behind. */
3292 if (unlikely(ve->request)) {
3293 struct i915_request *old;
3294
3295 spin_lock_irq(&ve->base.active.lock);
3296
3297 old = fetch_and_zero(&ve->request);
3298 if (old) {
3299 GEM_BUG_ON(!i915_request_completed(old));
3300 __i915_request_submit(old);
3301 i915_request_put(old);
3302 }
3303
3304 spin_unlock_irq(&ve->base.active.lock);
3305 }
3306
3307 /*
3308 * Flush the tasklet in case it is still running on another core.
3309 *
3310 * This needs to be done before we remove ourselves from the siblings'
3311 * rbtrees as in the case it is running in parallel, it may reinsert
3312 * the rb_node into a sibling.
3313 */
3314 tasklet_kill(&ve->base.execlists.tasklet);
3315
3316 /* Decouple ourselves from the siblings, no more access allowed. */
3317 for (n = 0; n < ve->num_siblings; n++) {
3318 struct intel_engine_cs *sibling = ve->siblings[n];
3319 struct rb_node *node = &ve->nodes[sibling->id].rb;
3320
3321 if (RB_EMPTY_NODE(node))
3322 continue;
3323
3324 spin_lock_irq(&sibling->active.lock);
3325
3326 /* Detachment is lazily performed in the execlists tasklet */
3327 if (!RB_EMPTY_NODE(node))
3328 rb_erase_cached(node, &sibling->execlists.virtual);
3329
3330 spin_unlock_irq(&sibling->active.lock);
3331 }
3332 GEM_BUG_ON(__tasklet_is_scheduled(&ve->base.execlists.tasklet));
3333 GEM_BUG_ON(!list_empty(virtual_queue(ve)));
3334
3335 lrc_fini(&ve->context);
3336 intel_context_fini(&ve->context);
3337
3338 intel_breadcrumbs_free(ve->base.breadcrumbs);
3339 intel_engine_free_request_pool(&ve->base);
3340
3341 kfree(ve->bonds);
3342 kfree(ve);
3343}
3344
3345static void virtual_context_destroy(struct kref *kref)
3346{
3347 struct virtual_engine *ve =
3348 container_of(kref, typeof(*ve), context.ref);
3349
3350 GEM_BUG_ON(!list_empty(&ve->context.signals));
3351
3352 /*
3353 * When destroying the virtual engine, we have to be aware that
3354 * it may still be in use from an hardirq/softirq context causing
3355 * the resubmission of a completed request (background completion
3356 * due to preempt-to-busy). Before we can free the engine, we need
3357 * to flush the submission code and tasklets that are still potentially
3358 * accessing the engine. Flushing the tasklets requires process context,
3359 * and since we can guard the resubmit onto the engine with an RCU read
3360 * lock, we can delegate the free of the engine to an RCU worker.
3361 */
3362 INIT_RCU_WORK(&ve->rcu, rcu_virtual_context_destroy);
3363 queue_rcu_work(system_wq, &ve->rcu);
3364}
3365
3366static void virtual_engine_initial_hint(struct virtual_engine *ve)
3367{
3368 int swp;
3369
3370 /*
3371 * Pick a random sibling on starting to help spread the load around.
3372 *
3373 * New contexts are typically created with exactly the same order
3374 * of siblings, and often started in batches. Due to the way we iterate
3375 * the array of sibling when submitting requests, sibling[0] is
3376 * prioritised for dequeuing. If we make sure that sibling[0] is fairly
3377 * randomised across the system, we also help spread the load by the
3378 * first engine we inspect being different each time.
3379 *
3380 * NB This does not force us to execute on this engine, it will just
3381 * typically be the first we inspect for submission.
3382 */
3383 swp = prandom_u32_max(ve->num_siblings);
3384 if (swp)
3385 swap(ve->siblings[swp], ve->siblings[0]);
3386}
3387
3388static int virtual_context_alloc(struct intel_context *ce)
3389{
3390 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3391
3392 return lrc_alloc(ce, ve->siblings[0]);
3393}
3394
3395static int virtual_context_pre_pin(struct intel_context *ce,
3396 struct i915_gem_ww_ctx *ww,
3397 void **vaddr)
3398{
3399 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3400
3401 /* Note: we must use a real engine class for setting up reg state */
3402 return lrc_pre_pin(ce, ve->siblings[0], ww, vaddr);
3403}
3404
3405static int virtual_context_pin(struct intel_context *ce, void *vaddr)
3406{
3407 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3408
3409 return lrc_pin(ce, ve->siblings[0], vaddr);
3410}
3411
3412static void virtual_context_enter(struct intel_context *ce)
3413{
3414 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3415 unsigned int n;
3416
3417 for (n = 0; n < ve->num_siblings; n++)
3418 intel_engine_pm_get(ve->siblings[n]);
3419
3420 intel_timeline_enter(ce->timeline);
3421}
3422
3423static void virtual_context_exit(struct intel_context *ce)
3424{
3425 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
3426 unsigned int n;
3427
3428 intel_timeline_exit(ce->timeline);
3429
3430 for (n = 0; n < ve->num_siblings; n++)
3431 intel_engine_pm_put(ve->siblings[n]);
3432}
3433
3434static const struct intel_context_ops virtual_context_ops = {
3435 .flags = COPS_HAS_INFLIGHT,
3436
3437 .alloc = virtual_context_alloc,
3438
3439 .pre_pin = virtual_context_pre_pin,
3440 .pin = virtual_context_pin,
3441 .unpin = lrc_unpin,
3442 .post_unpin = lrc_post_unpin,
3443
3444 .enter = virtual_context_enter,
3445 .exit = virtual_context_exit,
3446
3447 .destroy = virtual_context_destroy,
3448};
3449
3450static intel_engine_mask_t virtual_submission_mask(struct virtual_engine *ve)
3451{
3452 struct i915_request *rq;
3453 intel_engine_mask_t mask;
3454
3455 rq = READ_ONCE(ve->request);
3456 if (!rq)
3457 return 0;
3458
3459 /* The rq is ready for submission; rq->execution_mask is now stable. */
3460 mask = rq->execution_mask;
3461 if (unlikely(!mask)) {
3462 /* Invalid selection, submit to a random engine in error */
3463 i915_request_set_error_once(rq, -ENODEV);
3464 mask = ve->siblings[0]->mask;
3465 }
3466
3467 ENGINE_TRACE(&ve->base, "rq=%llx:%lld, mask=%x, prio=%d\n",
3468 rq->fence.context, rq->fence.seqno,
3469 mask, ve->base.execlists.queue_priority_hint);
3470
3471 return mask;
3472}
3473
3474static void virtual_submission_tasklet(unsigned long data)
3475{
3476 struct virtual_engine * const ve = (struct virtual_engine *)data;
3477 const int prio = READ_ONCE(ve->base.execlists.queue_priority_hint);
3478 intel_engine_mask_t mask;
3479 unsigned int n;
3480
3481 rcu_read_lock();
3482 mask = virtual_submission_mask(ve);
3483 rcu_read_unlock();
3484 if (unlikely(!mask))
3485 return;
3486
3487 for (n = 0; n < ve->num_siblings; n++) {
3488 struct intel_engine_cs *sibling = READ_ONCE(ve->siblings[n]);
3489 struct ve_node * const node = &ve->nodes[sibling->id];
3490 struct rb_node **parent, *rb;
3491 bool first;
3492
3493 if (!READ_ONCE(ve->request))
3494 break; /* already handled by a sibling's tasklet */
3495
3496 spin_lock_irq(&sibling->active.lock);
3497
3498 if (unlikely(!(mask & sibling->mask))) {
3499 if (!RB_EMPTY_NODE(&node->rb)) {
3500 rb_erase_cached(&node->rb,
3501 &sibling->execlists.virtual);
3502 RB_CLEAR_NODE(&node->rb);
3503 }
3504
3505 goto unlock_engine;
3506 }
3507
3508 if (unlikely(!RB_EMPTY_NODE(&node->rb))) {
3509 /*
3510 * Cheat and avoid rebalancing the tree if we can
3511 * reuse this node in situ.
3512 */
3513 first = rb_first_cached(&sibling->execlists.virtual) ==
3514 &node->rb;
3515 if (prio == node->prio || (prio > node->prio && first))
3516 goto submit_engine;
3517
3518 rb_erase_cached(&node->rb, &sibling->execlists.virtual);
3519 }
3520
3521 rb = NULL;
3522 first = true;
3523 parent = &sibling->execlists.virtual.rb_root.rb_node;
3524 while (*parent) {
3525 struct ve_node *other;
3526
3527 rb = *parent;
3528 other = rb_entry(rb, typeof(*other), rb);
3529 if (prio > other->prio) {
3530 parent = &rb->rb_left;
3531 } else {
3532 parent = &rb->rb_right;
3533 first = false;
3534 }
3535 }
3536
3537 rb_link_node(&node->rb, rb, parent);
3538 rb_insert_color_cached(&node->rb,
3539 &sibling->execlists.virtual,
3540 first);
3541
3542submit_engine:
3543 GEM_BUG_ON(RB_EMPTY_NODE(&node->rb));
3544 node->prio = prio;
3545 if (first && prio > sibling->execlists.queue_priority_hint)
3546 tasklet_hi_schedule(&sibling->execlists.tasklet);
3547
3548unlock_engine:
3549 spin_unlock_irq(&sibling->active.lock);
3550
3551 if (intel_context_inflight(&ve->context))
3552 break;
3553 }
3554}
3555
3556static void virtual_submit_request(struct i915_request *rq)
3557{
3558 struct virtual_engine *ve = to_virtual_engine(rq->engine);
3559 unsigned long flags;
3560
3561 ENGINE_TRACE(&ve->base, "rq=%llx:%lld\n",
3562 rq->fence.context,
3563 rq->fence.seqno);
3564
3565 GEM_BUG_ON(ve->base.submit_request != virtual_submit_request);
3566
3567 spin_lock_irqsave(&ve->base.active.lock, flags);
3568
3569 /* By the time we resubmit a request, it may be completed */
3570 if (__i915_request_is_complete(rq)) {
3571 __i915_request_submit(rq);
3572 goto unlock;
3573 }
3574
3575 if (ve->request) { /* background completion from preempt-to-busy */
3576 GEM_BUG_ON(!i915_request_completed(ve->request));
3577 __i915_request_submit(ve->request);
3578 i915_request_put(ve->request);
3579 }
3580
3581 ve->base.execlists.queue_priority_hint = rq_prio(rq);
3582 ve->request = i915_request_get(rq);
3583
3584 GEM_BUG_ON(!list_empty(virtual_queue(ve)));
3585 list_move_tail(&rq->sched.link, virtual_queue(ve));
3586
3587 tasklet_hi_schedule(&ve->base.execlists.tasklet);
3588
3589unlock:
3590 spin_unlock_irqrestore(&ve->base.active.lock, flags);
3591}
3592
3593static struct ve_bond *
3594virtual_find_bond(struct virtual_engine *ve,
3595 const struct intel_engine_cs *master)
3596{
3597 int i;
3598
3599 for (i = 0; i < ve->num_bonds; i++) {
3600 if (ve->bonds[i].master == master)
3601 return &ve->bonds[i];
3602 }
3603
3604 return NULL;
3605}
3606
3607static void
3608virtual_bond_execute(struct i915_request *rq, struct dma_fence *signal)
3609{
3610 struct virtual_engine *ve = to_virtual_engine(rq->engine);
3611 intel_engine_mask_t allowed, exec;
3612 struct ve_bond *bond;
3613
3614 allowed = ~to_request(signal)->engine->mask;
3615
3616 bond = virtual_find_bond(ve, to_request(signal)->engine);
3617 if (bond)
3618 allowed &= bond->sibling_mask;
3619
3620 /* Restrict the bonded request to run on only the available engines */
3621 exec = READ_ONCE(rq->execution_mask);
3622 while (!try_cmpxchg(&rq->execution_mask, &exec, exec & allowed))
3623 ;
3624
3625 /* Prevent the master from being re-run on the bonded engines */
3626 to_request(signal)->execution_mask &= ~allowed;
3627}
3628
3629struct intel_context *
3630intel_execlists_create_virtual(struct intel_engine_cs **siblings,
3631 unsigned int count)
3632{
3633 struct virtual_engine *ve;
3634 unsigned int n;
3635 int err;
3636
3637 if (count == 0)
3638 return ERR_PTR(-EINVAL);
3639
3640 if (count == 1)
3641 return intel_context_create(siblings[0]);
3642
3643 ve = kzalloc(struct_size(ve, siblings, count), GFP_KERNEL);
3644 if (!ve)
3645 return ERR_PTR(-ENOMEM);
3646
3647 ve->base.i915 = siblings[0]->i915;
3648 ve->base.gt = siblings[0]->gt;
3649 ve->base.uncore = siblings[0]->uncore;
3650 ve->base.id = -1;
3651
3652 ve->base.class = OTHER_CLASS;
3653 ve->base.uabi_class = I915_ENGINE_CLASS_INVALID;
3654 ve->base.instance = I915_ENGINE_CLASS_INVALID_VIRTUAL;
3655 ve->base.uabi_instance = I915_ENGINE_CLASS_INVALID_VIRTUAL;
3656
3657 /*
3658 * The decision on whether to submit a request using semaphores
3659 * depends on the saturated state of the engine. We only compute
3660 * this during HW submission of the request, and we need for this
3661 * state to be globally applied to all requests being submitted
3662 * to this engine. Virtual engines encompass more than one physical
3663 * engine and so we cannot accurately tell in advance if one of those
3664 * engines is already saturated and so cannot afford to use a semaphore
3665 * and be pessimized in priority for doing so -- if we are the only
3666 * context using semaphores after all other clients have stopped, we
3667 * will be starved on the saturated system. Such a global switch for
3668 * semaphores is less than ideal, but alas is the current compromise.
3669 */
3670 ve->base.saturated = ALL_ENGINES;
3671
3672 snprintf(ve->base.name, sizeof(ve->base.name), "virtual");
3673
3674 intel_engine_init_active(&ve->base, ENGINE_VIRTUAL);
3675 intel_engine_init_execlists(&ve->base);
3676
3677 ve->base.cops = &virtual_context_ops;
3678 ve->base.request_alloc = execlists_request_alloc;
3679
3680 ve->base.schedule = i915_schedule;
3681 ve->base.submit_request = virtual_submit_request;
3682 ve->base.bond_execute = virtual_bond_execute;
3683
3684 INIT_LIST_HEAD(virtual_queue(ve));
3685 ve->base.execlists.queue_priority_hint = INT_MIN;
3686 tasklet_init(&ve->base.execlists.tasklet,
3687 virtual_submission_tasklet,
3688 (unsigned long)ve);
3689
3690 intel_context_init(&ve->context, &ve->base);
3691
3692 ve->base.breadcrumbs = intel_breadcrumbs_create(NULL);
3693 if (!ve->base.breadcrumbs) {
3694 err = -ENOMEM;
3695 goto err_put;
3696 }
3697
3698 for (n = 0; n < count; n++) {
3699 struct intel_engine_cs *sibling = siblings[n];
3700
3701 GEM_BUG_ON(!is_power_of_2(sibling->mask));
3702 if (sibling->mask & ve->base.mask) {
3703 DRM_DEBUG("duplicate %s entry in load balancer\n",
3704 sibling->name);
3705 err = -EINVAL;
3706 goto err_put;
3707 }
3708
3709 /*
3710 * The virtual engine implementation is tightly coupled to
3711 * the execlists backend -- we push out request directly
3712 * into a tree inside each physical engine. We could support
3713 * layering if we handle cloning of the requests and
3714 * submitting a copy into each backend.
3715 */
3716 if (sibling->execlists.tasklet.func !=
3717 execlists_submission_tasklet) {
3718 err = -ENODEV;
3719 goto err_put;
3720 }
3721
3722 GEM_BUG_ON(RB_EMPTY_NODE(&ve->nodes[sibling->id].rb));
3723 RB_CLEAR_NODE(&ve->nodes[sibling->id].rb);
3724
3725 ve->siblings[ve->num_siblings++] = sibling;
3726 ve->base.mask |= sibling->mask;
3727
3728 /*
3729 * All physical engines must be compatible for their emission
3730 * functions (as we build the instructions during request
3731 * construction and do not alter them before submission
3732 * on the physical engine). We use the engine class as a guide
3733 * here, although that could be refined.
3734 */
3735 if (ve->base.class != OTHER_CLASS) {
3736 if (ve->base.class != sibling->class) {
3737 DRM_DEBUG("invalid mixing of engine class, sibling %d, already %d\n",
3738 sibling->class, ve->base.class);
3739 err = -EINVAL;
3740 goto err_put;
3741 }
3742 continue;
3743 }
3744
3745 ve->base.class = sibling->class;
3746 ve->base.uabi_class = sibling->uabi_class;
3747 snprintf(ve->base.name, sizeof(ve->base.name),
3748 "v%dx%d", ve->base.class, count);
3749 ve->base.context_size = sibling->context_size;
3750
3751 ve->base.emit_bb_start = sibling->emit_bb_start;
3752 ve->base.emit_flush = sibling->emit_flush;
3753 ve->base.emit_init_breadcrumb = sibling->emit_init_breadcrumb;
3754 ve->base.emit_fini_breadcrumb = sibling->emit_fini_breadcrumb;
3755 ve->base.emit_fini_breadcrumb_dw =
3756 sibling->emit_fini_breadcrumb_dw;
3757
3758 ve->base.flags = sibling->flags;
3759 }
3760
3761 ve->base.flags |= I915_ENGINE_IS_VIRTUAL;
3762
3763 virtual_engine_initial_hint(ve);
3764 return &ve->context;
3765
3766err_put:
3767 intel_context_put(&ve->context);
3768 return ERR_PTR(err);
3769}
3770
3771struct intel_context *
3772intel_execlists_clone_virtual(struct intel_engine_cs *src)
3773{
3774 struct virtual_engine *se = to_virtual_engine(src);
3775 struct intel_context *dst;
3776
3777 dst = intel_execlists_create_virtual(se->siblings,
3778 se->num_siblings);
3779 if (IS_ERR(dst))
3780 return dst;
3781
3782 if (se->num_bonds) {
3783 struct virtual_engine *de = to_virtual_engine(dst->engine);
3784
3785 de->bonds = kmemdup(se->bonds,
3786 sizeof(*se->bonds) * se->num_bonds,
3787 GFP_KERNEL);
3788 if (!de->bonds) {
3789 intel_context_put(dst);
3790 return ERR_PTR(-ENOMEM);
3791 }
3792
3793 de->num_bonds = se->num_bonds;
3794 }
3795
3796 return dst;
3797}
3798
3799int intel_virtual_engine_attach_bond(struct intel_engine_cs *engine,
3800 const struct intel_engine_cs *master,
3801 const struct intel_engine_cs *sibling)
3802{
3803 struct virtual_engine *ve = to_virtual_engine(engine);
3804 struct ve_bond *bond;
3805 int n;
3806
3807 /* Sanity check the sibling is part of the virtual engine */
3808 for (n = 0; n < ve->num_siblings; n++)
3809 if (sibling == ve->siblings[n])
3810 break;
3811 if (n == ve->num_siblings)
3812 return -EINVAL;
3813
3814 bond = virtual_find_bond(ve, master);
3815 if (bond) {
3816 bond->sibling_mask |= sibling->mask;
3817 return 0;
3818 }
3819
3820 bond = krealloc(ve->bonds,
3821 sizeof(*bond) * (ve->num_bonds + 1),
3822 GFP_KERNEL);
3823 if (!bond)
3824 return -ENOMEM;
3825
3826 bond[ve->num_bonds].master = master;
3827 bond[ve->num_bonds].sibling_mask = sibling->mask;
3828
3829 ve->bonds = bond;
3830 ve->num_bonds++;
3831
3832 return 0;
3833}
3834
3835void intel_execlists_show_requests(struct intel_engine_cs *engine,
3836 struct drm_printer *m,
3837 void (*show_request)(struct drm_printer *m,
3838 const struct i915_request *rq,
3839 const char *prefix,
3840 int indent),
3841 unsigned int max)
3842{
3843 const struct intel_engine_execlists *execlists = &engine->execlists;
3844 struct i915_request *rq, *last;
3845 unsigned long flags;
3846 unsigned int count;
3847 struct rb_node *rb;
3848
3849 spin_lock_irqsave(&engine->active.lock, flags);
3850
3851 last = NULL;
3852 count = 0;
3853 list_for_each_entry(rq, &engine->active.requests, sched.link) {
3854 if (count++ < max - 1)
3855 show_request(m, rq, "\t\t", 0);
3856 else
3857 last = rq;
3858 }
3859 if (last) {
3860 if (count > max) {
3861 drm_printf(m,
3862 "\t\t...skipping %d executing requests...\n",
3863 count - max);
3864 }
3865 show_request(m, last, "\t\t", 0);
3866 }
3867
3868 if (execlists->queue_priority_hint != INT_MIN)
3869 drm_printf(m, "\t\tQueue priority hint: %d\n",
3870 READ_ONCE(execlists->queue_priority_hint));
3871
3872 last = NULL;
3873 count = 0;
3874 for (rb = rb_first_cached(&execlists->queue); rb; rb = rb_next(rb)) {
3875 struct i915_priolist *p = rb_entry(rb, typeof(*p), node);
3876 int i;
3877
3878 priolist_for_each_request(rq, p, i) {
3879 if (count++ < max - 1)
3880 show_request(m, rq, "\t\t", 0);
3881 else
3882 last = rq;
3883 }
3884 }
3885 if (last) {
3886 if (count > max) {
3887 drm_printf(m,
3888 "\t\t...skipping %d queued requests...\n",
3889 count - max);
3890 }
3891 show_request(m, last, "\t\t", 0);
3892 }
3893
3894 last = NULL;
3895 count = 0;
3896 for (rb = rb_first_cached(&execlists->virtual); rb; rb = rb_next(rb)) {
3897 struct virtual_engine *ve =
3898 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
3899 struct i915_request *rq = READ_ONCE(ve->request);
3900
3901 if (rq) {
3902 if (count++ < max - 1)
3903 show_request(m, rq, "\t\t", 0);
3904 else
3905 last = rq;
3906 }
3907 }
3908 if (last) {
3909 if (count > max) {
3910 drm_printf(m,
3911 "\t\t...skipping %d virtual requests...\n",
3912 count - max);
3913 }
3914 show_request(m, last, "\t\t", 0);
3915 }
3916
3917 spin_unlock_irqrestore(&engine->active.lock, flags);
3918}
3919
3920bool
3921intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine)
3922{
3923 return engine->set_default_submission ==
3924 execlists_set_default_submission;
3925}
3926
3927#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
3928#include "selftest_execlists.c"
3929#endif
drivers/gpu/drm/i915/gt/intel_execlists_submission.h+47−0
@@ -0,0 +1,47 @@
1/* SPDX-License-Identifier: MIT */
2/*
3 * Copyright © 2014 Intel Corporation
4 */
5
6#ifndef __INTEL_EXECLISTS_SUBMISSION_H__
7#define __INTEL_EXECLISTS_SUBMISSION_H__
8
9#include <linux/types.h>
10
11struct drm_printer;
12
13struct i915_request;
14struct intel_context;
15struct intel_engine_cs;
16
17enum {
18 INTEL_CONTEXT_SCHEDULE_IN = 0,
19 INTEL_CONTEXT_SCHEDULE_OUT,
20 INTEL_CONTEXT_SCHEDULE_PREEMPTED,
21};
22
23int intel_execlists_submission_setup(struct intel_engine_cs *engine);
24
25void intel_execlists_show_requests(struct intel_engine_cs *engine,
26 struct drm_printer *m,
27 void (*show_request)(struct drm_printer *m,
28 const struct i915_request *rq,
29 const char *prefix,
30 int indent),
31 unsigned int max);
32
33struct intel_context *
34intel_execlists_create_virtual(struct intel_engine_cs **siblings,
35 unsigned int count);
36
37struct intel_context *
38intel_execlists_clone_virtual(struct intel_engine_cs *src);
39
40int intel_virtual_engine_attach_bond(struct intel_engine_cs *engine,
41 const struct intel_engine_cs *master,
42 const struct intel_engine_cs *sibling);
43
44bool
45intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine);
46
47#endif /* __INTEL_EXECLISTS_SUBMISSION_H__ */
drivers/gpu/drm/i915/gt/intel_ggtt.c+16−2
@@ -101,7 +101,16 @@ static bool needs_idle_maps(struct drm_i915_private *i915) · 100 unmodified lines
101 * Query intel_iommu to see if we need the workaround. Presumably that101 * Query intel_iommu to see if we need the workaround. Presumably that
102 * was loaded first.102 * was loaded first.
103 */103 */
104 return IS_GEN(i915, 5) && IS_MOBILE(i915) && intel_vtd_active();104 if (!intel_vtd_active())
105 return false;
106
107 if (IS_GEN(i915, 5) && IS_MOBILE(i915))
108 return true;
109
110 if (IS_GEN(i915, 12))
111 return true; /* XXX DMAR fault reason 7 */
112
113 return false;
105}114}
106 115
107void i915_ggtt_suspend(struct i915_ggtt *ggtt)116void i915_ggtt_suspend(struct i915_ggtt *ggtt)
@@ -1050,7 +1059,12 @@ static int i915_gmch_probe(struct i915_ggtt *ggtt) · 942 unmodified lines
1050 1059
1051 ggtt->vm.alloc_pt_dma = alloc_pt_dma;1060 ggtt->vm.alloc_pt_dma = alloc_pt_dma;
1052 1061
1053 ggtt->do_idle_maps = needs_idle_maps(i915);1062 if (needs_idle_maps(i915)) {
1063 drm_notice(&i915->drm,
1064 "Flushing DMA requests before IOMMU unmaps; performance may be degraded\n");
1065 ggtt->do_idle_maps = true;
1066 }
1067
1054 ggtt->vm.insert_page = i915_ggtt_insert_page;1068 ggtt->vm.insert_page = i915_ggtt_insert_page;
1055 ggtt->vm.insert_entries = i915_ggtt_insert_entries;1069 ggtt->vm.insert_entries = i915_ggtt_insert_entries;
1056 ggtt->vm.clear_range = i915_ggtt_clear_range;1070 ggtt->vm.clear_range = i915_ggtt_clear_range;
drivers/gpu/drm/i915/gt/intel_ggtt_fencing.c+20−2
@@ -320,13 +320,31 @@ void i915_vma_revoke_fence(struct i915_vma *vma) · 319 unmodified lines
320 fence_write(fence);320 fence_write(fence);
321}321}
322 322
323static bool fence_is_active(const struct i915_fence_reg *fence)
324{
325 return fence->vma && i915_vma_is_active(fence->vma);
326}
327
323static struct i915_fence_reg *fence_find(struct i915_ggtt *ggtt)328static struct i915_fence_reg *fence_find(struct i915_ggtt *ggtt)
324{329{
325 struct i915_fence_reg *fence;330 struct i915_fence_reg *active = NULL;
331 struct i915_fence_reg *fence, *fn;
326 332
327 list_for_each_entry(fence, &ggtt->fence_list, link) {333 list_for_each_entry_safe(fence, fn, &ggtt->fence_list, link) {
328 GEM_BUG_ON(fence->vma && fence->vma->fence != fence);334 GEM_BUG_ON(fence->vma && fence->vma->fence != fence);
329 335
336 if (fence == active) /* now seen this fence twice */
337 active = ERR_PTR(-EAGAIN);
338
339 /* Prefer idle fences so we do not have to wait on the GPU */
340 if (active != ERR_PTR(-EAGAIN) && fence_is_active(fence)) {
341 if (!active)
342 active = fence;
343
344 list_move_tail(&fence->link, &ggtt->fence_list);
345 continue;
346 }
347
330 if (atomic_read(&fence->pin_count))348 if (atomic_read(&fence->pin_count))
331 continue;349 continue;
332 350
drivers/gpu/drm/i915/gt/intel_gt.c+2−2
@@ -46,6 +46,8 @@ void intel_gt_init_hw_early(struct intel_gt *gt, struct i915_ggtt *ggtt) · 45 unmodified lines
46 46
47int intel_gt_init_mmio(struct intel_gt *gt)47int intel_gt_init_mmio(struct intel_gt *gt)
48{48{
49 intel_gt_init_clock_frequency(gt);
50
49 intel_uc_init_mmio(&gt->uc);51 intel_uc_init_mmio(&gt->uc);
50 intel_sseu_info_init(gt);52 intel_sseu_info_init(gt);
51 53
@@ -546,8 +548,6 @@ int intel_gt_init(struct intel_gt *gt) · 494 unmodified lines
546 */548 */
547 intel_uncore_forcewake_get(gt->uncore, FORCEWAKE_ALL);549 intel_uncore_forcewake_get(gt->uncore, FORCEWAKE_ALL);
548 550
549 intel_gt_init_clock_frequency(gt);
550
551 err = intel_gt_init_scratch(gt, IS_GEN(gt->i915, 2) ? SZ_256K : SZ_4K);551 err = intel_gt_init_scratch(gt, IS_GEN(gt->i915, 2) ? SZ_256K : SZ_4K);
552 if (err)552 if (err)
553 goto out_fw;553 goto out_fw;
drivers/gpu/drm/i915/gt/intel_gt_clock_utils.c+157−40
@@ -7,34 +7,146 @@ · 6 unmodified lines
7#include "intel_gt.h"7#include "intel_gt.h"
8#include "intel_gt_clock_utils.h"8#include "intel_gt_clock_utils.h"
9 9
10#define MHZ_12 12000000 /* 12MHz (24MHz/2), 83.333ns */10static u32 read_reference_ts_freq(struct intel_uncore *uncore)
11#define MHZ_12_5 12500000 /* 12.5MHz (25MHz/2), 80ns */11{
12#define MHZ_19_2 19200000 /* 19.2MHz, 52.083ns */12 u32 ts_override = intel_uncore_read(uncore, GEN9_TIMESTAMP_OVERRIDE);
13 u32 base_freq, frac_freq;
14
15 base_freq = ((ts_override & GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_MASK) >>
16 GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DIVIDER_SHIFT) + 1;
17 base_freq *= 1000000;
18
19 frac_freq = ((ts_override &
20 GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_MASK) >>
21 GEN9_TIMESTAMP_OVERRIDE_US_COUNTER_DENOMINATOR_SHIFT);
22 frac_freq = 1000000 / (frac_freq + 1);
23
24 return base_freq + frac_freq;
25}
26
27static u32 gen10_get_crystal_clock_freq(struct intel_uncore *uncore,
28 u32 rpm_config_reg)
29{
30 u32 f19_2_mhz = 19200000;
31 u32 f24_mhz = 24000000;
32 u32 crystal_clock =
33 (rpm_config_reg & GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >>
34 GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT;
13 35
14static u32 read_clock_frequency(const struct intel_gt *gt)36 switch (crystal_clock) {
37 case GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_19_2_MHZ:
38 return f19_2_mhz;
39 case GEN9_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_24_MHZ:
40 return f24_mhz;
41 default:
42 MISSING_CASE(crystal_clock);
43 return 0;
44 }
45}
46
47static u32 gen11_get_crystal_clock_freq(struct intel_uncore *uncore,
48 u32 rpm_config_reg)
15{49{
16 if (INTEL_GEN(gt->i915) >= 11) {50 u32 f19_2_mhz = 19200000;
17 u32 config;51 u32 f24_mhz = 24000000;
18 52 u32 f25_mhz = 25000000;
19 config = intel_uncore_read(gt->uncore, RPM_CONFIG0);53 u32 f38_4_mhz = 38400000;
20 config &= GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK;54 u32 crystal_clock =
21 config >>= GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT;55 (rpm_config_reg & GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_MASK) >>
22 56 GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_SHIFT;
23 switch (config) {57
24 case 0: return MHZ_12;58 switch (crystal_clock) {
25 case 1:59 case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_24_MHZ:
26 case 2: return MHZ_19_2;60 return f24_mhz;
27 default:61 case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_19_2_MHZ:
28 case 3: return MHZ_12_5;62 return f19_2_mhz;
63 case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_38_4_MHZ:
64 return f38_4_mhz;
65 case GEN11_RPM_CONFIG0_CRYSTAL_CLOCK_FREQ_25_MHZ:
66 return f25_mhz;
67 default:
68 MISSING_CASE(crystal_clock);
69 return 0;
70 }
71}
72
73static u32 read_clock_frequency(struct intel_uncore *uncore)
74{
75 u32 f12_5_mhz = 12500000;
76 u32 f19_2_mhz = 19200000;
77 u32 f24_mhz = 24000000;
78
79 if (INTEL_GEN(uncore->i915) <= 4) {
80 /*
81 * PRMs say:
82 *
83 * "The value in this register increments once every 16
84 * hclks." (through the “Clocking Configuration”
85 * (“CLKCFG”) MCHBAR register)
86 */
87 return RUNTIME_INFO(uncore->i915)->rawclk_freq * 1000 / 16;
88 } else if (INTEL_GEN(uncore->i915) <= 8) {
89 /*
90 * PRMs say:
91 *
92 * "The PCU TSC counts 10ns increments; this timestamp
93 * reflects bits 38:3 of the TSC (i.e. 80ns granularity,
94 * rolling over every 1.5 hours).
95 */
96 return f12_5_mhz;
97 } else if (INTEL_GEN(uncore->i915) <= 9) {
98 u32 ctc_reg = intel_uncore_read(uncore, CTC_MODE);
99 u32 freq = 0;
100
101 if ((ctc_reg & CTC_SOURCE_PARAMETER_MASK) == CTC_SOURCE_DIVIDE_LOGIC) {
102 freq = read_reference_ts_freq(uncore);
103 } else {
104 freq = IS_GEN9_LP(uncore->i915) ? f19_2_mhz : f24_mhz;
105
106 /*
107 * Now figure out how the command stream's timestamp
108 * register increments from this frequency (it might
109 * increment only every few clock cycle).
110 */
111 freq >>= 3 - ((ctc_reg & CTC_SHIFT_PARAMETER_MASK) >>
112 CTC_SHIFT_PARAMETER_SHIFT);
29 }113 }
30 } else if (INTEL_GEN(gt->i915) >= 9) {114
31 if (IS_GEN9_LP(gt->i915))115 return freq;
32 return MHZ_19_2;116 } else if (INTEL_GEN(uncore->i915) <= 12) {
33 else117 u32 ctc_reg = intel_uncore_read(uncore, CTC_MODE);
34 return MHZ_12;118 u32 freq = 0;
35 } else {119
36 return MHZ_12_5;120 /*
121 * First figure out the reference frequency. There are 2 ways
122 * we can compute the frequency, either through the
123 * TIMESTAMP_OVERRIDE register or through RPM_CONFIG. CTC_MODE
124 * tells us which one we should use.
125 */
126 if ((ctc_reg & CTC_SOURCE_PARAMETER_MASK) == CTC_SOURCE_DIVIDE_LOGIC) {
127 freq = read_reference_ts_freq(uncore);
128 } else {
129 u32 c0 = intel_uncore_read(uncore, RPM_CONFIG0);
130
131 if (INTEL_GEN(uncore->i915) <= 10)
132 freq = gen10_get_crystal_clock_freq(uncore, c0);
133 else
134 freq = gen11_get_crystal_clock_freq(uncore, c0);
135
136 /*
137 * Now figure out how the command stream's timestamp
138 * register increments from this frequency (it might
139 * increment only every few clock cycle).
140 */
141 freq >>= 3 - ((c0 & GEN10_RPM_CONFIG0_CTC_SHIFT_PARAMETER_MASK) >>
142 GEN10_RPM_CONFIG0_CTC_SHIFT_PARAMETER_SHIFT);
143 }
144
145 return freq;
37 }146 }
147
148 MISSING_CASE("Unknown gen, unable to read command streamer timestamp frequency\n");
149 return 0;
38}150}
39 151
40void intel_gt_init_clock_frequency(struct intel_gt *gt)152void intel_gt_init_clock_frequency(struct intel_gt *gt)
@@ -43,20 +155,27 @@ void intel_gt_init_clock_frequency(struct intel_gt *gt) · 2 unmodified lines
43 * Note that on gen11+, the clock frequency may be reconfigured.155 * Note that on gen11+, the clock frequency may be reconfigured.
44 * We do not, and we assume nobody else does.156 * We do not, and we assume nobody else does.
45 */157 */
46 gt->clock_frequency = read_clock_frequency(gt);158 gt->clock_frequency = read_clock_frequency(gt->uncore);
159 if (gt->clock_frequency)
160 gt->clock_period_ns = intel_gt_clock_interval_to_ns(gt, 1);
161
47 GT_TRACE(gt,162 GT_TRACE(gt,
48 "Using clock frequency: %dkHz\n",163 "Using clock frequency: %dkHz, period: %dns, wrap: %lldms\n",
49 gt->clock_frequency / 1000);164 gt->clock_frequency / 1000,
165 gt->clock_period_ns,
166 div_u64(mul_u32_u32(gt->clock_period_ns, S32_MAX),
167 USEC_PER_SEC));
168
50}169}
51 170
52#if IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)171#if IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM)
53void intel_gt_check_clock_frequency(const struct intel_gt *gt)172void intel_gt_check_clock_frequency(const struct intel_gt *gt)
54{173{
55 if (gt->clock_frequency != read_clock_frequency(gt)) {174 if (gt->clock_frequency != read_clock_frequency(gt->uncore)) {
56 dev_err(gt->i915->drm.dev,175 dev_err(gt->i915->drm.dev,
57 "GT clock frequency changed, was %uHz, now %uHz!\n",176 "GT clock frequency changed, was %uHz, now %uHz!\n",
58 gt->clock_frequency,177 gt->clock_frequency,
59 read_clock_frequency(gt));178 read_clock_frequency(gt->uncore));
60 }179 }
61}180}
62#endif181#endif
@@ -66,26 +185,24 @@ static u64 div_u64_roundup(u64 nom, u32 den) · 3 unmodified lines
66 return div_u64(nom + den - 1, den);185 return div_u64(nom + den - 1, den);
67}186}
68 187
69u32 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u32 count)188u64 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u64 count)
70{189{
71 return div_u64_roundup(mul_u32_u32(count, 1000 * 1000 * 1000),190 return div_u64_roundup(count * NSEC_PER_SEC, gt->clock_frequency);
72 gt->clock_frequency);
73}191}
74 192
75u32 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u32 count)193u64 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u64 count)
76{194{
77 return intel_gt_clock_interval_to_ns(gt, 16 * count);195 return intel_gt_clock_interval_to_ns(gt, 16 * count);
78}196}
79 197
80u32 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u32 ns)198u64 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u64 ns)
81{199{
82 return div_u64_roundup(mul_u32_u32(gt->clock_frequency, ns),200 return div_u64_roundup(gt->clock_frequency * ns, NSEC_PER_SEC);
83 1000 * 1000 * 1000);
84}201}
85 202
86u32 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u32 ns)203u64 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u64 ns)
87{204{
88 u32 val;205 u64 val;
89 206
90 /*207 /*
91 * Make these a multiple of magic 25 to avoid SNB (eg. Dell XPS208 * Make these a multiple of magic 25 to avoid SNB (eg. Dell XPS
@@ -94,9 +211,9 @@ u32 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u32 ns) · 2 unmodified lines
94 * EI/thresholds are "bad", leading to a very sluggish or even211 * EI/thresholds are "bad", leading to a very sluggish or even
95 * frozen machine.212 * frozen machine.
96 */213 */
97 val = DIV_ROUND_UP(intel_gt_ns_to_clock_interval(gt, ns), 16);214 val = div_u64_roundup(intel_gt_ns_to_clock_interval(gt, ns), 16);
98 if (IS_GEN(gt->i915, 6))215 if (IS_GEN(gt->i915, 6))
99 val = roundup(val, 25);216 val = div_u64_roundup(val, 25) * 25;
100 217
101 return val;218 return val;
102}219}
drivers/gpu/drm/i915/gt/intel_gt_clock_utils.h+4−4
@@ -18,10 +18,10 @@ void intel_gt_check_clock_frequency(const struct intel_gt *gt); · 17 unmodified lines
18static inline void intel_gt_check_clock_frequency(const struct intel_gt *gt) {}18static inline void intel_gt_check_clock_frequency(const struct intel_gt *gt) {}
19#endif19#endif
20 20
21u32 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u32 count);21u64 intel_gt_clock_interval_to_ns(const struct intel_gt *gt, u64 count);
22u32 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u32 count);22u64 intel_gt_pm_interval_to_ns(const struct intel_gt *gt, u64 count);
23 23
24u32 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u32 ns);24u64 intel_gt_ns_to_clock_interval(const struct intel_gt *gt, u64 ns);
25u32 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u32 ns);25u64 intel_gt_ns_to_pm_interval(const struct intel_gt *gt, u64 ns);
26 26
27#endif /* __INTEL_GT_CLOCK_UTILS_H__ */27#endif /* __INTEL_GT_CLOCK_UTILS_H__ */
drivers/gpu/drm/i915/gt/intel_gt_irq.c+1−0
@@ -11,6 +11,7 @@ · 10 unmodified lines
11#include "intel_breadcrumbs.h"11#include "intel_breadcrumbs.h"
12#include "intel_gt.h"12#include "intel_gt.h"
13#include "intel_gt_irq.h"13#include "intel_gt_irq.h"
14#include "intel_lrc_reg.h"
14#include "intel_uncore.h"15#include "intel_uncore.h"
15#include "intel_rps.h"16#include "intel_rps.h"
16 17
drivers/gpu/drm/i915/gt/intel_gt_pm.c+49−0
@@ -39,6 +39,28 @@ static void user_forcewake(struct intel_gt *gt, bool suspend) · 38 unmodified lines
39 intel_gt_pm_put(gt);39 intel_gt_pm_put(gt);
40}40}
41 41
42static void runtime_begin(struct intel_gt *gt)
43{
44 local_irq_disable();
45 write_seqcount_begin(&gt->stats.lock);
46 gt->stats.start = ktime_get();
47 gt->stats.active = true;
48 write_seqcount_end(&gt->stats.lock);
49 local_irq_enable();
50}
51
52static void runtime_end(struct intel_gt *gt)
53{
54 local_irq_disable();
55 write_seqcount_begin(&gt->stats.lock);
56 gt->stats.active = false;
57 gt->stats.total =
58 ktime_add(gt->stats.total,
59 ktime_sub(ktime_get(), gt->stats.start));
60 write_seqcount_end(&gt->stats.lock);
61 local_irq_enable();
62}
63
42static int __gt_unpark(struct intel_wakeref *wf)64static int __gt_unpark(struct intel_wakeref *wf)
43{65{
44 struct intel_gt *gt = container_of(wf, typeof(*gt), wakeref);66 struct intel_gt *gt = container_of(wf, typeof(*gt), wakeref);
@@ -67,6 +89,7 @@ static int __gt_unpark(struct intel_wakeref *wf) · 22 unmodified lines
67 i915_pmu_gt_unparked(i915);89 i915_pmu_gt_unparked(i915);
68 90
69 intel_gt_unpark_requests(gt);91 intel_gt_unpark_requests(gt);
92 runtime_begin(gt);
70 93
71 return 0;94 return 0;
72}95}
@@ -79,6 +102,7 @@ static int __gt_park(struct intel_wakeref *wf) · 6 unmodified lines
79 102
80 GT_TRACE(gt, "\n");103 GT_TRACE(gt, "\n");
81 104
105 runtime_end(gt);
82 intel_gt_park_requests(gt);106 intel_gt_park_requests(gt);
83 107
84 i915_vma_parked(gt);108 i915_vma_parked(gt);
@@ -106,6 +130,7 @@ static const struct intel_wakeref_ops wf_ops = { · 21 unmodified lines
106void intel_gt_pm_init_early(struct intel_gt *gt)130void intel_gt_pm_init_early(struct intel_gt *gt)
107{131{
108 intel_wakeref_init(&gt->wakeref, gt->uncore->rpm, &wf_ops);132 intel_wakeref_init(&gt->wakeref, gt->uncore->rpm, &wf_ops);
133 seqcount_mutex_init(&gt->stats.lock, &gt->wakeref.mutex);
109}134}
110 135
111void intel_gt_pm_init(struct intel_gt *gt)136void intel_gt_pm_init(struct intel_gt *gt)
@@ -339,6 +364,30 @@ int intel_gt_runtime_resume(struct intel_gt *gt) · 227 unmodified lines
339 return intel_uc_runtime_resume(&gt->uc);364 return intel_uc_runtime_resume(&gt->uc);
340}365}
341 366
367static ktime_t __intel_gt_get_awake_time(const struct intel_gt *gt)
368{
369 ktime_t total = gt->stats.total;
370
371 if (gt->stats.active)
372 total = ktime_add(total,
373 ktime_sub(ktime_get(), gt->stats.start));
374
375 return total;
376}
377
378ktime_t intel_gt_get_awake_time(const struct intel_gt *gt)
379{
380 unsigned int seq;
381 ktime_t total;
382
383 do {
384 seq = read_seqcount_begin(&gt->stats.lock);
385 total = __intel_gt_get_awake_time(gt);
386 } while (read_seqcount_retry(&gt->stats.lock, seq));
387
388 return total;
389}
390
342#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)391#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
343#include "selftest_gt_pm.c"392#include "selftest_gt_pm.c"
344#endif393#endif
drivers/gpu/drm/i915/gt/intel_gt_pm.h+2−0
@@ -58,6 +58,8 @@ int intel_gt_resume(struct intel_gt *gt); · 57 unmodified lines
58void intel_gt_runtime_suspend(struct intel_gt *gt);58void intel_gt_runtime_suspend(struct intel_gt *gt);
59int intel_gt_runtime_resume(struct intel_gt *gt);59int intel_gt_runtime_resume(struct intel_gt *gt);
60 60
61ktime_t intel_gt_get_awake_time(const struct intel_gt *gt);
62
61static inline bool is_mock_gt(const struct intel_gt *gt)63static inline bool is_mock_gt(const struct intel_gt *gt)
62{64{
63 return I915_SELFTEST_ONLY(gt->awake == -ENODEV);65 return I915_SELFTEST_ONLY(gt->awake == -ENODEV);
drivers/gpu/drm/i915/gt/intel_gt_requests.c+1−6
@@ -135,13 +135,8 @@ long intel_gt_retire_requests_timeout(struct intel_gt *gt, long timeout) · 134 unmodified lines
135 struct intel_gt_timelines *timelines = &gt->timelines;135 struct intel_gt_timelines *timelines = &gt->timelines;
136 struct intel_timeline *tl, *tn;136 struct intel_timeline *tl, *tn;
137 unsigned long active_count = 0;137 unsigned long active_count = 0;
138 bool interruptible;
139 LIST_HEAD(free);138 LIST_HEAD(free);
140 139
141 interruptible = true;
142 if (unlikely(timeout < 0))
143 timeout = -timeout, interruptible = false;
144
145 flush_submission(gt, timeout); /* kick the ksoftirqd tasklets */140 flush_submission(gt, timeout); /* kick the ksoftirqd tasklets */
146 spin_lock(&timelines->lock);141 spin_lock(&timelines->lock);
147 list_for_each_entry_safe(tl, tn, &timelines->active_list, link) {142 list_for_each_entry_safe(tl, tn, &timelines->active_list, link) {
@@ -163,7 +158,7 @@ long intel_gt_retire_requests_timeout(struct intel_gt *gt, long timeout) · 15 unmodified lines
163 mutex_unlock(&tl->mutex);158 mutex_unlock(&tl->mutex);
164 159
165 timeout = dma_fence_wait_timeout(fence,160 timeout = dma_fence_wait_timeout(fence,
166 interruptible,161 true,
167 timeout);162 timeout);
168 dma_fence_put(fence);163 dma_fence_put(fence);
169 164
drivers/gpu/drm/i915/gt/intel_gt_types.h+25−0
@@ -75,6 +75,7 @@ struct intel_gt { · 74 unmodified lines
75 intel_wakeref_t awake;75 intel_wakeref_t awake;
76 76
77 u32 clock_frequency;77 u32 clock_frequency;
78 u32 clock_period_ns;
78 79
79 struct intel_llc llc;80 struct intel_llc llc;
80 struct intel_rc6 rc6;81 struct intel_rc6 rc6;
@@ -87,6 +88,30 @@ struct intel_gt { · 6 unmodified lines
87 88
88 u32 pm_guc_events;89 u32 pm_guc_events;
89 90
91 struct {
92 bool active;
93
94 /**
95 * @lock: Lock protecting the below fields.
96 */
97 seqcount_mutex_t lock;
98
99 /**
100 * @total: Total time this engine was busy.
101 *
102 * Accumulated time not counting the most recent block in cases
103 * where engine is currently busy (active > 0).
104 */
105 ktime_t total;
106
107 /**
108 * @start: Timestamp of the last idle to active transition.
109 *
110 * Idle is defined as active == 0, active is active > 0.
111 */
112 ktime_t start;
113 } stats;
114
90 struct intel_engine_cs *engine[I915_NUM_ENGINES];115 struct intel_engine_cs *engine[I915_NUM_ENGINES];
91 struct intel_engine_cs *engine_class[MAX_ENGINE_CLASS + 1]116 struct intel_engine_cs *engine_class[MAX_ENGINE_CLASS + 1]
92 [MAX_ENGINE_INSTANCE + 1];117 [MAX_ENGINE_INSTANCE + 1];
drivers/gpu/drm/i915/gt/intel_gtt.c+29−0
@@ -422,6 +422,35 @@ void setup_private_pat(struct intel_uncore *uncore) · 421 unmodified lines
422 bdw_setup_private_ppat(uncore);422 bdw_setup_private_ppat(uncore);
423}423}
424 424
425struct i915_vma *
426__vm_create_scratch_for_read(struct i915_address_space *vm, unsigned long size)
427{
428 struct drm_i915_gem_object *obj;
429 struct i915_vma *vma;
430 int err;
431
432 obj = i915_gem_object_create_internal(vm->i915, PAGE_ALIGN(size));
433 if (IS_ERR(obj))
434 return ERR_CAST(obj);
435
436 i915_gem_object_set_cache_coherency(obj, I915_CACHING_CACHED);
437
438 vma = i915_vma_instance(obj, vm, NULL);
439 if (IS_ERR(vma)) {
440 i915_gem_object_put(obj);
441 return vma;
442 }
443
444 err = i915_vma_pin(vma, 0, 0,
445 i915_vma_is_ggtt(vma) ? PIN_GLOBAL : PIN_USER);
446 if (err) {
447 i915_vma_put(vma);
448 return ERR_PTR(err);
449 }
450
451 return vma;
452}
453
425#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)454#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
426#include "selftests/mock_gtt.c"455#include "selftests/mock_gtt.c"
427#endif456#endif
drivers/gpu/drm/i915/gt/intel_gtt.h+3−0
@@ -573,6 +573,9 @@ int i915_vm_pin_pt_stash(struct i915_address_space *vm, · 572 unmodified lines
573void i915_vm_free_pt_stash(struct i915_address_space *vm,573void i915_vm_free_pt_stash(struct i915_address_space *vm,
574 struct i915_vm_pt_stash *stash);574 struct i915_vm_pt_stash *stash);
575 575
576struct i915_vma *
577__vm_create_scratch_for_read(struct i915_address_space *vm, unsigned long size);
578
576static inline struct sgt_dma {579static inline struct sgt_dma {
577 struct scatterlist *sg;580 struct scatterlist *sg;
578 dma_addr_t dma, max;581 dma_addr_t dma, max;
drivers/gpu/drm/i915/gt/intel_lrc.c+1149−5731
@@ -1,681 +1,216 @@
1// SPDX-License-Identifier: MIT
1/*2/*
2 * Copyright © 2014 Intel Corporation3 * Copyright © 2014 Intel Corporation
3 *
4 * Permission is hereby granted, free of charge, to any person obtaining a
5 * copy of this software and associated documentation files (the "Software"),
6 * to deal in the Software without restriction, including without limitation
7 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8 * and/or sell copies of the Software, and to permit persons to whom the
9 * Software is furnished to do so, subject to the following conditions:
10 *
11 * The above copyright notice and this permission notice (including the next
12 * paragraph) shall be included in all copies or substantial portions of the
13 * Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
20 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
21 * IN THE SOFTWARE.
22 *
23 * Authors:
24 * Ben Widawsky <ben@bwidawsk.net>
25 * Michel Thierry <michel.thierry@intel.com>
26 * Thomas Daniel <thomas.daniel@intel.com>
27 * Oscar Mateo <oscar.mateo@intel.com>
28 *
29 */
30
31/**
32 * DOC: Logical Rings, Logical Ring Contexts and Execlists
33 *
34 * Motivation:
35 * GEN8 brings an expansion of the HW contexts: "Logical Ring Contexts".
36 * These expanded contexts enable a number of new abilities, especially
37 * "Execlists" (also implemented in this file).
38 *
39 * One of the main differences with the legacy HW contexts is that logical
40 * ring contexts incorporate many more things to the context's state, like
41 * PDPs or ringbuffer control registers:
42 *
43 * The reason why PDPs are included in the context is straightforward: as
44 * PPGTTs (per-process GTTs) are actually per-context, having the PDPs
45 * contained there mean you don't need to do a ppgtt->switch_mm yourself,
46 * instead, the GPU will do it for you on the context switch.
47 *
48 * But, what about the ringbuffer control registers (head, tail, etc..)?
49 * shouldn't we just need a set of those per engine command streamer? This is
50 * where the name "Logical Rings" starts to make sense: by virtualizing the
51 * rings, the engine cs shifts to a new "ring buffer" with every context
52 * switch. When you want to submit a workload to the GPU you: A) choose your
53 * context, B) find its appropriate virtualized ring, C) write commands to it
54 * and then, finally, D) tell the GPU to switch to that context.
55 *
56 * Instead of the legacy MI_SET_CONTEXT, the way you tell the GPU to switch
57 * to a contexts is via a context execution list, ergo "Execlists".
58 *
59 * LRC implementation:
60 * Regarding the creation of contexts, we have:
61 *
62 * - One global default context.
63 * - One local default context for each opened fd.
64 * - One local extra context for each context create ioctl call.
65 *
66 * Now that ringbuffers belong per-context (and not per-engine, like before)
67 * and that contexts are uniquely tied to a given engine (and not reusable,
68 * like before) we need:
69 *
70 * - One ringbuffer per-engine inside each context.
71 * - One backing object per-engine inside each context.
72 *
73 * The global default context starts its life with these new objects fully
74 * allocated and populated. The local default context for each opened fd is
75 * more complex, because we don't know at creation time which engine is going
76 * to use them. To handle this, we have implemented a deferred creation of LR
77 * contexts:
78 *
79 * The local context starts its life as a hollow or blank holder, that only
80 * gets populated for a given engine once we receive an execbuffer. If later
81 * on we receive another execbuffer ioctl for the same context but a different
82 * engine, we allocate/populate a new ringbuffer and context backing object and
83 * so on.
84 *
85 * Finally, regarding local contexts created using the ioctl call: as they are
86 * only allowed with the render ring, we can allocate & populate them right
87 * away (no need to defer anything, at least for now).
88 *
89 * Execlists implementation:
90 * Execlists are the new method by which, on gen8+ hardware, workloads are
91 * submitted for execution (as opposed to the legacy, ringbuffer-based, method).
92 * This method works as follows:
93 *
94 * When a request is committed, its commands (the BB start and any leading or
95 * trailing commands, like the seqno breadcrumbs) are placed in the ringbuffer
96 * for the appropriate context. The tail pointer in the hardware context is not
97 * updated at this time, but instead, kept by the driver in the ringbuffer
98 * structure. A structure representing this request is added to a request queue
99 * for the appropriate engine: this structure contains a copy of the context's
100 * tail after the request was written to the ring buffer and a pointer to the
101 * context itself.
102 *
103 * If the engine's request queue was empty before the request was added, the
104 * queue is processed immediately. Otherwise the queue will be processed during
105 * a context switch interrupt. In any case, elements on the queue will get sent
106 * (in pairs) to the GPU's ExecLists Submit Port (ELSP, for short) with a
107 * globally unique 20-bits submission ID.
108 *
109 * When execution of a request completes, the GPU updates the context status
110 * buffer with a context complete event and generates a context switch interrupt.
111 * During the interrupt handling, the driver examines the events in the buffer:
112 * for each context complete event, if the announced ID matches that on the head
113 * of the request queue, then that request is retired and removed from the queue.
114 *
115 * After processing, if any requests were retired and the queue is not empty
116 * then a new execution list can be submitted. The two requests at the front of
117 * the queue are next to be submitted but since a context may not occur twice in
118 * an execution list, if subsequent requests have the same ID as the first then
119 * the two requests must be combined. This is done simply by discarding requests
120 * at the head of the queue until either only one requests is left (in which case
121 * we use a NULL second context) or the first two requests have unique IDs.
122 *
123 * By always executing the first two requests in the queue the driver ensures
124 * that the GPU is kept as busy as possible. In the case where a single context
125 * completes but a second context is still executing, the request for this second
126 * context will be at the head of the queue when we remove the first one. This
127 * request will then be resubmitted along with a new request for a different context,
128 * which will cause the hardware to continue executing the second request and queue
129 * the new request (the GPU detects the condition of a context getting preempted
130 * with the same context and optimizes the context switch flow by not doing
131 * preemption, but just sampling the new tail pointer).
132 *
133 */4 */
134#include <linux/interrupt.h>
135 5
6#include "gen8_engine_cs.h"
136#include "i915_drv.h"7#include "i915_drv.h"
137#include "i915_perf.h"8#include "i915_perf.h"
138#include "i915_trace.h"9#include "intel_engine.h"
139#include "i915_vgpu.h"10#include "intel_gpu_commands.h"
140#include "intel_breadcrumbs.h"
141#include "intel_context.h"
142#include "intel_engine_pm.h"
143#include "intel_gt.h"11#include "intel_gt.h"
144#include "intel_gt_pm.h"12#include "intel_lrc.h"
145#include "intel_gt_requests.h"
146#include "intel_lrc_reg.h"13#include "intel_lrc_reg.h"
147#include "intel_mocs.h"
148#include "intel_reset.h"
149#include "intel_ring.h"14#include "intel_ring.h"
150#include "intel_workarounds.h"
151#include "shmem_utils.h"15#include "shmem_utils.h"
152 16
153#define RING_EXECLIST_QFULL (1 << 0x2)17static void set_offsets(u32 *regs,
154#define RING_EXECLIST1_VALID (1 << 0x3)18 const u8 *data,
155#define RING_EXECLIST0_VALID (1 << 0x4)19 const struct intel_engine_cs *engine,
156#define RING_EXECLIST_ACTIVE_STATUS (3 << 0xE)20 bool close)
157#define RING_EXECLIST1_ACTIVE (1 << 0x11)21#define NOP(x) (BIT(7) | (x))
158#define RING_EXECLIST0_ACTIVE (1 << 0x12)22#define LRI(count, flags) ((flags) << 6 | (count) | BUILD_BUG_ON_ZERO(count >= BIT(6)))
159 23#define POSTED BIT(0)
160#define GEN8_CTX_STATUS_IDLE_ACTIVE (1 << 0)24#define REG(x) (((x) >> 2) | BUILD_BUG_ON_ZERO(x >= 0x200))
161#define GEN8_CTX_STATUS_PREEMPTED (1 << 1)25#define REG16(x) \
162#define GEN8_CTX_STATUS_ELEMENT_SWITCH (1 << 2)26 (((x) >> 9) | BIT(7) | BUILD_BUG_ON_ZERO(x >= 0x10000)), \
163#define GEN8_CTX_STATUS_ACTIVE_IDLE (1 << 3)27 (((x) >> 2) & 0x7f)
164#define GEN8_CTX_STATUS_COMPLETE (1 << 4)28#define END 0
165#define GEN8_CTX_STATUS_LITE_RESTORE (1 << 15)29{
166 30 const u32 base = engine->mmio_base;
167#define GEN8_CTX_STATUS_COMPLETED_MASK \
168 (GEN8_CTX_STATUS_COMPLETE | GEN8_CTX_STATUS_PREEMPTED)
169 31
170#define CTX_DESC_FORCE_RESTORE BIT_ULL(2)32 while (*data) {
33 u8 count, flags;
171 34
172#define GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE (0x1) /* lower csb dword */35 if (*data & BIT(7)) { /* skip */
173#define GEN12_CTX_SWITCH_DETAIL(csb_dw) ((csb_dw) & 0xF) /* upper csb dword */36 count = *data++ & ~BIT(7);
174#define GEN12_CSB_SW_CTX_ID_MASK GENMASK(25, 15)37 regs += count;
175#define GEN12_IDLE_CTX_ID 0x7FF38 continue;
176#define GEN12_CSB_CTX_VALID(csb_dw) \39 }
177 (FIELD_GET(GEN12_CSB_SW_CTX_ID_MASK, csb_dw) != GEN12_IDLE_CTX_ID)
178 40
179/* Typical size of the average request (2 pipecontrols and a MI_BB) */41 count = *data & 0x3f;
180#define EXECLISTS_REQUEST_SIZE 64 /* bytes */42 flags = *data >> 6;
43 data++;
181 44
182struct virtual_engine {45 *regs = MI_LOAD_REGISTER_IMM(count);
183 struct intel_engine_cs base;46 if (flags & POSTED)
184 struct intel_context context;47 *regs |= MI_LRI_FORCE_POSTED;
185 struct rcu_work rcu;48 if (INTEL_GEN(engine->i915) >= 11)
49 *regs |= MI_LRI_LRM_CS_MMIO;
50 regs++;
186 51
187 /*52 GEM_BUG_ON(!count);
188 * We allow only a single request through the virtual engine at a time53 do {
189 * (each request in the timeline waits for the completion fence of54 u32 offset = 0;
190 * the previous before being submitted). By restricting ourselves to55 u8 v;
191 * only submitting a single request, each request is placed on to a
192 * physical to maximise load spreading (by virtue of the late greedy
193 * scheduling -- each real engine takes the next available request
194 * upon idling).
195 */
196 struct i915_request *request;
197 56
198 /*57 do {
199 * We keep a rbtree of available virtual engines inside each physical58 v = *data++;
200 * engine, sorted by priority. Here we preallocate the nodes we need59 offset <<= 7;
201 * for the virtual engine, indexed by physical_engine->id.60 offset |= v & ~BIT(7);
202 */61 } while (v & BIT(7));
203 struct ve_node {
204 struct rb_node rb;
205 int prio;
206 } nodes[I915_NUM_ENGINES];
207 62
208 /*63 regs[0] = base + (offset << 2);
209 * Keep track of bonded pairs -- restrictions upon on our selection64 regs += 2;
210 * of physical engines any particular request may be submitted to.65 } while (--count);
211 * If we receive a submit-fence from a master engine, we will only66 }
212 * use one of sibling_mask physical engines.
213 */
214 struct ve_bond {
215 const struct intel_engine_cs *master;
216 intel_engine_mask_t sibling_mask;
217 } *bonds;
218 unsigned int num_bonds;
219
220 /* And finally, which physical engines this virtual engine maps onto. */
221 unsigned int num_siblings;
222 struct intel_engine_cs *siblings[];
223};
224 67
225static struct virtual_engine *to_virtual_engine(struct intel_engine_cs *engine)68 if (close) {
226{69 /* Close the batch; used mainly by live_lrc_layout() */
227 GEM_BUG_ON(!intel_engine_is_virtual(engine));70 *regs = MI_BATCH_BUFFER_END;
228 return container_of(engine, struct virtual_engine, base);71 if (INTEL_GEN(engine->i915) >= 10)
72 *regs |= BIT(0);
73 }
229}74}
230 75
231static int __execlists_context_alloc(struct intel_context *ce,76static const u8 gen8_xcs_offsets[] = {
232 struct intel_engine_cs *engine);77 NOP(1),
233 78 LRI(11, 0),
234static void execlists_init_reg_state(u32 *reg_state,79 REG16(0x244),
235 const struct intel_context *ce,80 REG(0x034),
236 const struct intel_engine_cs *engine,81 REG(0x030),
237 const struct intel_ring *ring,82 REG(0x038),
238 bool close);83 REG(0x03c),
239static void84 REG(0x168),
240__execlists_update_reg_state(const struct intel_context *ce,85 REG(0x140),
241 const struct intel_engine_cs *engine,86 REG(0x110),
242 u32 head);87 REG(0x11c),
88 REG(0x114),
89 REG(0x118),
243 90
244static int lrc_ring_mi_mode(const struct intel_engine_cs *engine)91 NOP(9),
245{92 LRI(9, 0),
246 if (INTEL_GEN(engine->i915) >= 12)93 REG16(0x3a8),
247 return 0x60;94 REG16(0x28c),
248 else if (INTEL_GEN(engine->i915) >= 9)95 REG16(0x288),
249 return 0x54;96 REG16(0x284),
250 else if (engine->class == RENDER_CLASS)97 REG16(0x280),
251 return 0x58;98 REG16(0x27c),
252 else99 REG16(0x278),
253 return -1;100 REG16(0x274),
254}101 REG16(0x270),
255 102
256static int lrc_ring_gpr0(const struct intel_engine_cs *engine)103 NOP(13),
257{104 LRI(2, 0),
258 if (INTEL_GEN(engine->i915) >= 12)105 REG16(0x200),
259 return 0x74;106 REG(0x028),
260 else if (INTEL_GEN(engine->i915) >= 9)
261 return 0x68;
262 else if (engine->class == RENDER_CLASS)
263 return 0xd8;
264 else
265 return -1;
266}
267 107
268static int lrc_ring_wa_bb_per_ctx(const struct intel_engine_cs *engine)108 END
269{109};
270 if (INTEL_GEN(engine->i915) >= 12)
271 return 0x12;
272 else if (INTEL_GEN(engine->i915) >= 9 || engine->class == RENDER_CLASS)
273 return 0x18;
274 else
275 return -1;
276}
277 110
278static int lrc_ring_indirect_ptr(const struct intel_engine_cs *engine)111static const u8 gen9_xcs_offsets[] = {
279{112 NOP(1),
280 int x;113 LRI(14, POSTED),
114 REG16(0x244),
115 REG(0x034),
116 REG(0x030),
117 REG(0x038),
118 REG(0x03c),
119 REG(0x168),
120 REG(0x140),
121 REG(0x110),
122 REG(0x11c),
123 REG(0x114),
124 REG(0x118),
125 REG(0x1c0),
126 REG(0x1c4),
127 REG(0x1c8),
281 128
282 x = lrc_ring_wa_bb_per_ctx(engine);129 NOP(3),
283 if (x < 0)130 LRI(9, POSTED),
284 return x;131 REG16(0x3a8),
132 REG16(0x28c),
133 REG16(0x288),
134 REG16(0x284),
135 REG16(0x280),
136 REG16(0x27c),
137 REG16(0x278),
138 REG16(0x274),
139 REG16(0x270),
285 140
286 return x + 2;141 NOP(13),
287}142 LRI(1, POSTED),
143 REG16(0x200),
288 144
289static int lrc_ring_indirect_offset(const struct intel_engine_cs *engine)145 NOP(13),
290{146 LRI(44, POSTED),
291 int x;147 REG(0x028),
148 REG(0x09c),
149 REG(0x0c0),
150 REG(0x178),
151 REG(0x17c),
152 REG16(0x358),
153 REG(0x170),
154 REG(0x150),
155 REG(0x154),
156 REG(0x158),
157 REG16(0x41c),
158 REG16(0x600),
159 REG16(0x604),
160 REG16(0x608),
161 REG16(0x60c),
162 REG16(0x610),
163 REG16(0x614),
164 REG16(0x618),
165 REG16(0x61c),
166 REG16(0x620),
167 REG16(0x624),
168 REG16(0x628),
169 REG16(0x62c),
170 REG16(0x630),
171 REG16(0x634),
172 REG16(0x638),
173 REG16(0x63c),
174 REG16(0x640),
175 REG16(0x644),
176 REG16(0x648),
177 REG16(0x64c),
178 REG16(0x650),
179 REG16(0x654),
180 REG16(0x658),
181 REG16(0x65c),
182 REG16(0x660),
183 REG16(0x664),
184 REG16(0x668),
185 REG16(0x66c),
186 REG16(0x670),
187 REG16(0x674),
188 REG16(0x678),
189 REG16(0x67c),
190 REG(0x068),
292 191
293 x = lrc_ring_indirect_ptr(engine);192 END
294 if (x < 0)193};
295 return x;
296 194
297 return x + 2;195static const u8 gen12_xcs_offsets[] = {
298}196 NOP(1),
197 LRI(13, POSTED),
198 REG16(0x244),
199 REG(0x034),
200 REG(0x030),
201 REG(0x038),
202 REG(0x03c),
203 REG(0x168),
204 REG(0x140),
205 REG(0x110),
206 REG(0x1c0),
207 REG(0x1c4),
208 REG(0x1c8),
209 REG(0x180),
210 REG16(0x2b4),
299 211
300static int lrc_ring_cmd_buf_cctl(const struct intel_engine_cs *engine)212 NOP(5),
301{213 LRI(9, POSTED),
302 if (engine->class != RENDER_CLASS)
303 return -1;
304
305 if (INTEL_GEN(engine->i915) >= 12)
306 return 0xb6;
307 else if (INTEL_GEN(engine->i915) >= 11)
308 return 0xaa;
309 else
310 return -1;
311}
312
313static u32
314lrc_ring_indirect_offset_default(const struct intel_engine_cs *engine)
315{
316 switch (INTEL_GEN(engine->i915)) {
317 default:
318 MISSING_CASE(INTEL_GEN(engine->i915));
319 fallthrough;
320 case 12:
321 return GEN12_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
322 case 11:
323 return GEN11_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
324 case 10:
325 return GEN10_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
326 case 9:
327 return GEN9_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
328 case 8:
329 return GEN8_CTX_RCS_INDIRECT_CTX_OFFSET_DEFAULT;
330 }
331}
332
333static void
334lrc_ring_setup_indirect_ctx(u32 *regs,
335 const struct intel_engine_cs *engine,
336 u32 ctx_bb_ggtt_addr,
337 u32 size)
338{
339 GEM_BUG_ON(!size);
340 GEM_BUG_ON(!IS_ALIGNED(size, CACHELINE_BYTES));
341 GEM_BUG_ON(lrc_ring_indirect_ptr(engine) == -1);
342 regs[lrc_ring_indirect_ptr(engine) + 1] =
343 ctx_bb_ggtt_addr | (size / CACHELINE_BYTES);
344
345 GEM_BUG_ON(lrc_ring_indirect_offset(engine) == -1);
346 regs[lrc_ring_indirect_offset(engine) + 1] =
347 lrc_ring_indirect_offset_default(engine) << 6;
348}
349
350static u32 intel_context_get_runtime(const struct intel_context *ce)
351{
352 /*
353 * We can use either ppHWSP[16] which is recorded before the context
354 * switch (and so excludes the cost of context switches) or use the
355 * value from the context image itself, which is saved/restored earlier
356 * and so includes the cost of the save.
357 */
358 return READ_ONCE(ce->lrc_reg_state[CTX_TIMESTAMP]);
359}
360
361static void mark_eio(struct i915_request *rq)
362{
363 if (i915_request_completed(rq))
364 return;
365
366 GEM_BUG_ON(i915_request_signaled(rq));
367
368 i915_request_set_error_once(rq, -EIO);
369 i915_request_mark_complete(rq);
370}
371
372static struct i915_request *
373active_request(const struct intel_timeline * const tl, struct i915_request *rq)
374{
375 struct i915_request *active = rq;
376
377 rcu_read_lock();
378 list_for_each_entry_continue_reverse(rq, &tl->requests, link) {
379 if (i915_request_completed(rq))
380 break;
381
382 active = rq;
383 }
384 rcu_read_unlock();
385
386 return active;
387}
388
389static inline u32 intel_hws_preempt_address(struct intel_engine_cs *engine)
390{
391 return (i915_ggtt_offset(engine->status_page.vma) +
392 I915_GEM_HWS_PREEMPT_ADDR);
393}
394
395static inline void
396ring_set_paused(const struct intel_engine_cs *engine, int state)
397{
398 /*
399 * We inspect HWS_PREEMPT with a semaphore inside
400 * engine->emit_fini_breadcrumb. If the dword is true,
401 * the ring is paused as the semaphore will busywait
402 * until the dword is false.
403 */
404 engine->status_page.addr[I915_GEM_HWS_PREEMPT] = state;
405 if (state)
406 wmb();
407}
408
409static inline struct i915_priolist *to_priolist(struct rb_node *rb)
410{
411 return rb_entry(rb, struct i915_priolist, node);
412}
413
414static inline int rq_prio(const struct i915_request *rq)
415{
416 return READ_ONCE(rq->sched.attr.priority);
417}
418
419static int effective_prio(const struct i915_request *rq)
420{
421 int prio = rq_prio(rq);
422
423 /*
424 * If this request is special and must not be interrupted at any
425 * cost, so be it. Note we are only checking the most recent request
426 * in the context and so may be masking an earlier vip request. It
427 * is hoped that under the conditions where nopreempt is used, this
428 * will not matter (i.e. all requests to that context will be
429 * nopreempt for as long as desired).
430 */
431 if (i915_request_has_nopreempt(rq))
432 prio = I915_PRIORITY_UNPREEMPTABLE;
433
434 return prio;
435}
436
437static int queue_prio(const struct intel_engine_execlists *execlists)
438{
439 struct i915_priolist *p;
440 struct rb_node *rb;
441
442 rb = rb_first_cached(&execlists->queue);
443 if (!rb)
444 return INT_MIN;
445
446 /*
447 * As the priolist[] are inverted, with the highest priority in [0],
448 * we have to flip the index value to become priority.
449 */
450 p = to_priolist(rb);
451 if (!I915_USER_PRIORITY_SHIFT)
452 return p->priority;
453
454 return ((p->priority + 1) << I915_USER_PRIORITY_SHIFT) - ffs(p->used);
455}
456
457static inline bool need_preempt(const struct intel_engine_cs *engine,
458 const struct i915_request *rq,
459 struct rb_node *rb)
460{
461 int last_prio;
462
463 if (!intel_engine_has_semaphores(engine))
464 return false;
465
466 /*
467 * Check if the current priority hint merits a preemption attempt.
468 *
469 * We record the highest value priority we saw during rescheduling
470 * prior to this dequeue, therefore we know that if it is strictly
471 * less than the current tail of ESLP[0], we do not need to force
472 * a preempt-to-idle cycle.
473 *
474 * However, the priority hint is a mere hint that we may need to
475 * preempt. If that hint is stale or we may be trying to preempt
476 * ourselves, ignore the request.
477 *
478 * More naturally we would write
479 * prio >= max(0, last);
480 * except that we wish to prevent triggering preemption at the same
481 * priority level: the task that is running should remain running
482 * to preserve FIFO ordering of dependencies.
483 */
484 last_prio = max(effective_prio(rq), I915_PRIORITY_NORMAL - 1);
485 if (engine->execlists.queue_priority_hint <= last_prio)
486 return false;
487
488 /*
489 * Check against the first request in ELSP[1], it will, thanks to the
490 * power of PI, be the highest priority of that context.
491 */
492 if (!list_is_last(&rq->sched.link, &engine->active.requests) &&
493 rq_prio(list_next_entry(rq, sched.link)) > last_prio)
494 return true;
495
496 if (rb) {
497 struct virtual_engine *ve =
498 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
499 bool preempt = false;
500
501 if (engine == ve->siblings[0]) { /* only preempt one sibling */
502 struct i915_request *next;
503
504 rcu_read_lock();
505 next = READ_ONCE(ve->request);
506 if (next)
507 preempt = rq_prio(next) > last_prio;
508 rcu_read_unlock();
509 }
510
511 if (preempt)
512 return preempt;
513 }
514
515 /*
516 * If the inflight context did not trigger the preemption, then maybe
517 * it was the set of queued requests? Pick the highest priority in
518 * the queue (the first active priolist) and see if it deserves to be
519 * running instead of ELSP[0].
520 *
521 * The highest priority request in the queue can not be either
522 * ELSP[0] or ELSP[1] as, thanks again to PI, if it was the same
523 * context, it's priority would not exceed ELSP[0] aka last_prio.
524 */
525 return queue_prio(&engine->execlists) > last_prio;
526}
527
528__maybe_unused static inline bool
529assert_priority_queue(const struct i915_request *prev,
530 const struct i915_request *next)
531{
532 /*
533 * Without preemption, the prev may refer to the still active element
534 * which we refuse to let go.
535 *
536 * Even with preemption, there are times when we think it is better not
537 * to preempt and leave an ostensibly lower priority request in flight.
538 */
539 if (i915_request_is_active(prev))
540 return true;
541
542 return rq_prio(prev) >= rq_prio(next);
543}
544
545/*
546 * The context descriptor encodes various attributes of a context,
547 * including its GTT address and some flags. Because it's fairly
548 * expensive to calculate, we'll just do it once and cache the result,
549 * which remains valid until the context is unpinned.
550 *
551 * This is what a descriptor looks like, from LSB to MSB::
552 *
553 * bits 0-11: flags, GEN8_CTX_* (cached in ctx->desc_template)
554 * bits 12-31: LRCA, GTT address of (the HWSP of) this context
555 * bits 32-52: ctx ID, a globally unique tag (highest bit used by GuC)
556 * bits 53-54: mbz, reserved for use by hardware
557 * bits 55-63: group ID, currently unused and set to 0
558 *
559 * Starting from Gen11, the upper dword of the descriptor has a new format:
560 *
561 * bits 32-36: reserved
562 * bits 37-47: SW context ID
563 * bits 48:53: engine instance
564 * bit 54: mbz, reserved for use by hardware
565 * bits 55-60: SW counter
566 * bits 61-63: engine class
567 *
568 * engine info, SW context ID and SW counter need to form a unique number
569 * (Context ID) per lrc.
570 */
571static u32
572lrc_descriptor(struct intel_context *ce, struct intel_engine_cs *engine)
573{
574 u32 desc;
575
576 desc = INTEL_LEGACY_32B_CONTEXT;
577 if (i915_vm_is_4lvl(ce->vm))
578 desc = INTEL_LEGACY_64B_CONTEXT;
579 desc <<= GEN8_CTX_ADDRESSING_MODE_SHIFT;
580
581 desc |= GEN8_CTX_VALID | GEN8_CTX_PRIVILEGE;
582 if (IS_GEN(engine->i915, 8))
583 desc |= GEN8_CTX_L3LLC_COHERENT;
584
585 return i915_ggtt_offset(ce->state) | desc;
586}
587
588static inline unsigned int dword_in_page(void *addr)
589{
590 return offset_in_page(addr) / sizeof(u32);
591}
592
593static void set_offsets(u32 *regs,
594 const u8 *data,
595 const struct intel_engine_cs *engine,
596 bool clear)
597#define NOP(x) (BIT(7) | (x))
598#define LRI(count, flags) ((flags) << 6 | (count) | BUILD_BUG_ON_ZERO(count >= BIT(6)))
599#define POSTED BIT(0)
600#define REG(x) (((x) >> 2) | BUILD_BUG_ON_ZERO(x >= 0x200))
601#define REG16(x) \
602 (((x) >> 9) | BIT(7) | BUILD_BUG_ON_ZERO(x >= 0x10000)), \
603 (((x) >> 2) & 0x7f)
604#define END(total_state_size) 0, (total_state_size)
605{
606 const u32 base = engine->mmio_base;
607
608 while (*data) {
609 u8 count, flags;
610
611 if (*data & BIT(7)) { /* skip */
612 count = *data++ & ~BIT(7);
613 if (clear)
614 memset32(regs, MI_NOOP, count);
615 regs += count;
616 continue;
617 }
618
619 count = *data & 0x3f;
620 flags = *data >> 6;
621 data++;
622
623 *regs = MI_LOAD_REGISTER_IMM(count);
624 if (flags & POSTED)
625 *regs |= MI_LRI_FORCE_POSTED;
626 if (INTEL_GEN(engine->i915) >= 11)
627 *regs |= MI_LRI_LRM_CS_MMIO;
628 regs++;
629
630 GEM_BUG_ON(!count);
631 do {
632 u32 offset = 0;
633 u8 v;
634
635 do {
636 v = *data++;
637 offset <<= 7;
638 offset |= v & ~BIT(7);
639 } while (v & BIT(7));
640
641 regs[0] = base + (offset << 2);
642 if (clear)
643 regs[1] = 0;
644 regs += 2;
645 } while (--count);
646 }
647
648 if (clear) {
649 u8 count = *++data;
650
651 /* Clear past the tail for HW access */
652 GEM_BUG_ON(dword_in_page(regs) > count);
653 memset32(regs, MI_NOOP, count - dword_in_page(regs));
654
655 /* Close the batch; used mainly by live_lrc_layout() */
656 *regs = MI_BATCH_BUFFER_END;
657 if (INTEL_GEN(engine->i915) >= 10)
658 *regs |= BIT(0);
659 }
660}
661
662static const u8 gen8_xcs_offsets[] = {
663 NOP(1),
664 LRI(11, 0),
665 REG16(0x244),
666 REG(0x034),
667 REG(0x030),
668 REG(0x038),
669 REG(0x03c),
670 REG(0x168),
671 REG(0x140),
672 REG(0x110),
673 REG(0x11c),
674 REG(0x114),
675 REG(0x118),
676
677 NOP(9),
678 LRI(9, 0),
679 REG16(0x3a8),214 REG16(0x3a8),
680 REG16(0x28c),215 REG16(0x28c),
681 REG16(0x288),216 REG16(0x288),
@@ -686,15 +221,10 @@ static const u8 gen8_xcs_offsets[] = { · 4 unmodified lines
686 REG16(0x274),221 REG16(0x274),
687 REG16(0x270),222 REG16(0x270),
688 223
689 NOP(13),224 END
690 LRI(2, 0),
691 REG16(0x200),
692 REG(0x028),
693
694 END(80)
695};225};
696 226
697static const u8 gen9_xcs_offsets[] = {227static const u8 gen8_rcs_offsets[] = {
698 NOP(1),228 NOP(1),
699 LRI(14, POSTED),229 LRI(14, POSTED),
700 REG16(0x244),230 REG16(0x244),
@@ -725,126 +255,10 @@ static const u8 gen9_xcs_offsets[] = { · 24 unmodified lines
725 REG16(0x270),255 REG16(0x270),
726 256
727 NOP(13),257 NOP(13),
728 LRI(1, POSTED),258 LRI(1, 0),
729 REG16(0x200),259 REG(0x0c8),
730 260
731 NOP(13),261 END
732 LRI(44, POSTED),
733 REG(0x028),
734 REG(0x09c),
735 REG(0x0c0),
736 REG(0x178),
737 REG(0x17c),
738 REG16(0x358),
739 REG(0x170),
740 REG(0x150),
741 REG(0x154),
742 REG(0x158),
743 REG16(0x41c),
744 REG16(0x600),
745 REG16(0x604),
746 REG16(0x608),
747 REG16(0x60c),
748 REG16(0x610),
749 REG16(0x614),
750 REG16(0x618),
751 REG16(0x61c),
752 REG16(0x620),
753 REG16(0x624),
754 REG16(0x628),
755 REG16(0x62c),
756 REG16(0x630),
757 REG16(0x634),
758 REG16(0x638),
759 REG16(0x63c),
760 REG16(0x640),
761 REG16(0x644),
762 REG16(0x648),
763 REG16(0x64c),
764 REG16(0x650),
765 REG16(0x654),
766 REG16(0x658),
767 REG16(0x65c),
768 REG16(0x660),
769 REG16(0x664),
770 REG16(0x668),
771 REG16(0x66c),
772 REG16(0x670),
773 REG16(0x674),
774 REG16(0x678),
775 REG16(0x67c),
776 REG(0x068),
777
778 END(176)
779};
780
781static const u8 gen12_xcs_offsets[] = {
782 NOP(1),
783 LRI(13, POSTED),
784 REG16(0x244),
785 REG(0x034),
786 REG(0x030),
787 REG(0x038),
788 REG(0x03c),
789 REG(0x168),
790 REG(0x140),
791 REG(0x110),
792 REG(0x1c0),
793 REG(0x1c4),
794 REG(0x1c8),
795 REG(0x180),
796 REG16(0x2b4),
797
798 NOP(5),
799 LRI(9, POSTED),
800 REG16(0x3a8),
801 REG16(0x28c),
802 REG16(0x288),
803 REG16(0x284),
804 REG16(0x280),
805 REG16(0x27c),
806 REG16(0x278),
807 REG16(0x274),
808 REG16(0x270),
809
810 END(80)
811};
812
813static const u8 gen8_rcs_offsets[] = {
814 NOP(1),
815 LRI(14, POSTED),
816 REG16(0x244),
817 REG(0x034),
818 REG(0x030),
819 REG(0x038),
820 REG(0x03c),
821 REG(0x168),
822 REG(0x140),
823 REG(0x110),
824 REG(0x11c),
825 REG(0x114),
826 REG(0x118),
827 REG(0x1c0),
828 REG(0x1c4),
829 REG(0x1c8),
830
831 NOP(3),
832 LRI(9, POSTED),
833 REG16(0x3a8),
834 REG16(0x28c),
835 REG16(0x288),
836 REG16(0x284),
837 REG16(0x280),
838 REG16(0x27c),
839 REG16(0x278),
840 REG16(0x274),
841 REG16(0x270),
842
843 NOP(13),
844 LRI(1, 0),
845 REG(0x0c8),
846
847 END(80)
848};262};
849 263
850static const u8 gen9_rcs_offsets[] = {264static const u8 gen9_rcs_offsets[] = {
@@ -897,4382 +311,323 @@ static const u8 gen9_rcs_offsets[] = { · 46 unmodified lines
897 REG16(0x600),311 REG16(0x600),
898 REG16(0x604),312 REG16(0x604),
899 REG16(0x608),313 REG16(0x608),
900 REG16(0x60c),
901 REG16(0x610),
902 REG16(0x614),
903 REG16(0x618),
904 REG16(0x61c),
905 REG16(0x620),
906 REG16(0x624),
907 REG16(0x628),
908 REG16(0x62c),
909 REG16(0x630),
910 REG16(0x634),
911 REG16(0x638),
912 REG16(0x63c),
913 REG16(0x640),
914 REG16(0x644),
915 REG16(0x648),
916 REG16(0x64c),
917 REG16(0x650),
918 REG16(0x654),
919 REG16(0x658),
920 REG16(0x65c),
921 REG16(0x660),
922 REG16(0x664),
923 REG16(0x668),
924 REG16(0x66c),
925 REG16(0x670),
926 REG16(0x674),
927 REG16(0x678),
928 REG16(0x67c),
929 REG(0x68),
930
931 END(176)
932};
933
934static const u8 gen11_rcs_offsets[] = {
935 NOP(1),
936 LRI(15, POSTED),
937 REG16(0x244),
938 REG(0x034),
939 REG(0x030),
940 REG(0x038),
941 REG(0x03c),
942 REG(0x168),
943 REG(0x140),
944 REG(0x110),
945 REG(0x11c),
946 REG(0x114),
947 REG(0x118),
948 REG(0x1c0),
949 REG(0x1c4),
950 REG(0x1c8),
951 REG(0x180),
952
953 NOP(1),
954 LRI(9, POSTED),
955 REG16(0x3a8),
956 REG16(0x28c),
957 REG16(0x288),
958 REG16(0x284),
959 REG16(0x280),
960 REG16(0x27c),
961 REG16(0x278),
962 REG16(0x274),
963 REG16(0x270),
964
965 LRI(1, POSTED),
966 REG(0x1b0),
967
968 NOP(10),
969 LRI(1, 0),
970 REG(0x0c8),
971
972 END(80)
973};
974
975static const u8 gen12_rcs_offsets[] = {
976 NOP(1),
977 LRI(13, POSTED),
978 REG16(0x244),
979 REG(0x034),
980 REG(0x030),
981 REG(0x038),
982 REG(0x03c),
983 REG(0x168),
984 REG(0x140),
985 REG(0x110),
986 REG(0x1c0),
987 REG(0x1c4),
988 REG(0x1c8),
989 REG(0x180),
990 REG16(0x2b4),
991
992 NOP(5),
993 LRI(9, POSTED),
994 REG16(0x3a8),
995 REG16(0x28c),
996 REG16(0x288),
997 REG16(0x284),
998 REG16(0x280),
999 REG16(0x27c),
1000 REG16(0x278),
1001 REG16(0x274),
1002 REG16(0x270),
1003
1004 LRI(3, POSTED),
1005 REG(0x1b0),
1006 REG16(0x5a8),
1007 REG16(0x5ac),
1008
1009 NOP(6),
1010 LRI(1, 0),
1011 REG(0x0c8),
1012 NOP(3 + 9 + 1),
1013
1014 LRI(51, POSTED),
1015 REG16(0x588),
1016 REG16(0x588),
1017 REG16(0x588),
1018 REG16(0x588),
1019 REG16(0x588),
1020 REG16(0x588),
1021 REG(0x028),
1022 REG(0x09c),
1023 REG(0x0c0),
1024 REG(0x178),
1025 REG(0x17c),
1026 REG16(0x358),
1027 REG(0x170),
1028 REG(0x150),
1029 REG(0x154),
1030 REG(0x158),
1031 REG16(0x41c),
1032 REG16(0x600),
1033 REG16(0x604),
1034 REG16(0x608),
1035 REG16(0x60c),
1036 REG16(0x610),
1037 REG16(0x614),
1038 REG16(0x618),
1039 REG16(0x61c),
1040 REG16(0x620),
1041 REG16(0x624),
1042 REG16(0x628),
1043 REG16(0x62c),
1044 REG16(0x630),
1045 REG16(0x634),
1046 REG16(0x638),
1047 REG16(0x63c),
1048 REG16(0x640),
1049 REG16(0x644),
1050 REG16(0x648),
1051 REG16(0x64c),
1052 REG16(0x650),
1053 REG16(0x654),
1054 REG16(0x658),
1055 REG16(0x65c),
1056 REG16(0x660),
1057 REG16(0x664),
1058 REG16(0x668),
1059 REG16(0x66c),
1060 REG16(0x670),
1061 REG16(0x674),
1062 REG16(0x678),
1063 REG16(0x67c),
1064 REG(0x068),
1065 REG(0x084),
1066 NOP(1),
1067
1068 END(192)
1069};
1070
1071#undef END
1072#undef REG16
1073#undef REG
1074#undef LRI
1075#undef NOP
1076
1077static const u8 *reg_offsets(const struct intel_engine_cs *engine)
1078{
1079 /*
1080 * The gen12+ lists only have the registers we program in the basic
1081 * default state. We rely on the context image using relative
1082 * addressing to automatic fixup the register state between the
1083 * physical engines for virtual engine.
1084 */
1085 GEM_BUG_ON(INTEL_GEN(engine->i915) >= 12 &&
1086 !intel_engine_has_relative_mmio(engine));
1087
1088 if (engine->class == RENDER_CLASS) {
1089 if (INTEL_GEN(engine->i915) >= 12)
1090 return gen12_rcs_offsets;
1091 else if (INTEL_GEN(engine->i915) >= 11)
1092 return gen11_rcs_offsets;
1093 else if (INTEL_GEN(engine->i915) >= 9)
1094 return gen9_rcs_offsets;
1095 else
1096 return gen8_rcs_offsets;
1097 } else {
1098 if (INTEL_GEN(engine->i915) >= 12)
1099 return gen12_xcs_offsets;
1100 else if (INTEL_GEN(engine->i915) >= 9)
1101 return gen9_xcs_offsets;
1102 else
1103 return gen8_xcs_offsets;
1104 }
1105}
1106
1107static struct i915_request *
1108__unwind_incomplete_requests(struct intel_engine_cs *engine)
1109{
1110 struct i915_request *rq, *rn, *active = NULL;
1111 struct list_head *pl;
1112 int prio = I915_PRIORITY_INVALID;
1113
1114 lockdep_assert_held(&engine->active.lock);
1115
1116 list_for_each_entry_safe_reverse(rq, rn,
1117 &engine->active.requests,
1118 sched.link) {
1119 if (i915_request_completed(rq))
1120 continue; /* XXX */
1121
1122 __i915_request_unsubmit(rq);
1123
1124 /*
1125 * Push the request back into the queue for later resubmission.
1126 * If this request is not native to this physical engine (i.e.
1127 * it came from a virtual source), push it back onto the virtual
1128 * engine so that it can be moved across onto another physical
1129 * engine as load dictates.
1130 */
1131 if (likely(rq->execution_mask == engine->mask)) {
1132 GEM_BUG_ON(rq_prio(rq) == I915_PRIORITY_INVALID);
1133 if (rq_prio(rq) != prio) {
1134 prio = rq_prio(rq);
1135 pl = i915_sched_lookup_priolist(engine, prio);
1136 }
1137 GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
1138
1139 list_move(&rq->sched.link, pl);
1140 set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
1141
1142 /* Check in case we rollback so far we wrap [size/2] */
1143 if (intel_ring_direction(rq->ring,
1144 rq->tail,
1145 rq->ring->tail + 8) > 0)
1146 rq->context->lrc.desc |= CTX_DESC_FORCE_RESTORE;
1147
1148 active = rq;
1149 } else {
1150 struct intel_engine_cs *owner = rq->context->engine;
1151
1152 WRITE_ONCE(rq->engine, owner);
1153 owner->submit_request(rq);
1154 active = NULL;
1155 }
1156 }
1157
1158 return active;
1159}
1160
1161struct i915_request *
1162execlists_unwind_incomplete_requests(struct intel_engine_execlists *execlists)
1163{
1164 struct intel_engine_cs *engine =
1165 container_of(execlists, typeof(*engine), execlists);
1166
1167 return __unwind_incomplete_requests(engine);
1168}
1169
1170static inline void
1171execlists_context_status_change(struct i915_request *rq, unsigned long status)
1172{
1173 /*
1174 * Only used when GVT-g is enabled now. When GVT-g is disabled,
1175 * The compiler should eliminate this function as dead-code.
1176 */
1177 if (!IS_ENABLED(CONFIG_DRM_I915_GVT))
1178 return;
1179
1180 atomic_notifier_call_chain(&rq->engine->context_status_notifier,
1181 status, rq);
1182}
1183
1184static void intel_engine_context_in(struct intel_engine_cs *engine)
1185{
1186 unsigned long flags;
1187
1188 if (atomic_add_unless(&engine->stats.active, 1, 0))
1189 return;
1190
1191 write_seqlock_irqsave(&engine->stats.lock, flags);
1192 if (!atomic_add_unless(&engine->stats.active, 1, 0)) {
1193 engine->stats.start = ktime_get();
1194 atomic_inc(&engine->stats.active);
1195 }
1196 write_sequnlock_irqrestore(&engine->stats.lock, flags);
1197}
1198
1199static void intel_engine_context_out(struct intel_engine_cs *engine)
1200{
1201 unsigned long flags;
1202
1203 GEM_BUG_ON(!atomic_read(&engine->stats.active));
1204
1205 if (atomic_add_unless(&engine->stats.active, -1, 1))
1206 return;
1207
1208 write_seqlock_irqsave(&engine->stats.lock, flags);
1209 if (atomic_dec_and_test(&engine->stats.active)) {
1210 engine->stats.total =
1211 ktime_add(engine->stats.total,
1212 ktime_sub(ktime_get(), engine->stats.start));
1213 }
1214 write_sequnlock_irqrestore(&engine->stats.lock, flags);
1215}
1216
1217static void
1218execlists_check_context(const struct intel_context *ce,
1219 const struct intel_engine_cs *engine,
1220 const char *when)
1221{
1222 const struct intel_ring *ring = ce->ring;
1223 u32 *regs = ce->lrc_reg_state;
1224 bool valid = true;
1225 int x;
1226
1227 if (regs[CTX_RING_START] != i915_ggtt_offset(ring->vma)) {
1228 pr_err("%s: context submitted with incorrect RING_START [%08x], expected %08x\n",
1229 engine->name,
1230 regs[CTX_RING_START],
1231 i915_ggtt_offset(ring->vma));
1232 regs[CTX_RING_START] = i915_ggtt_offset(ring->vma);
1233 valid = false;
1234 }
1235
1236 if ((regs[CTX_RING_CTL] & ~(RING_WAIT | RING_WAIT_SEMAPHORE)) !=
1237 (RING_CTL_SIZE(ring->size) | RING_VALID)) {
1238 pr_err("%s: context submitted with incorrect RING_CTL [%08x], expected %08x\n",
1239 engine->name,
1240 regs[CTX_RING_CTL],
1241 (u32)(RING_CTL_SIZE(ring->size) | RING_VALID));
1242 regs[CTX_RING_CTL] = RING_CTL_SIZE(ring->size) | RING_VALID;
1243 valid = false;
1244 }
1245
1246 x = lrc_ring_mi_mode(engine);
1247 if (x != -1 && regs[x + 1] & (regs[x + 1] >> 16) & STOP_RING) {
1248 pr_err("%s: context submitted with STOP_RING [%08x] in RING_MI_MODE\n",
1249 engine->name, regs[x + 1]);
1250 regs[x + 1] &= ~STOP_RING;
1251 regs[x + 1] |= STOP_RING << 16;
1252 valid = false;
1253 }
1254
1255 WARN_ONCE(!valid, "Invalid lrc state found %s submission\n", when);
1256}
1257
1258static void restore_default_state(struct intel_context *ce,
1259 struct intel_engine_cs *engine)
1260{
1261 u32 *regs;
1262
1263 regs = memset(ce->lrc_reg_state, 0, engine->context_size - PAGE_SIZE);
1264 execlists_init_reg_state(regs, ce, engine, ce->ring, true);
1265
1266 ce->runtime.last = intel_context_get_runtime(ce);
1267}
1268
1269static void reset_active(struct i915_request *rq,
1270 struct intel_engine_cs *engine)
1271{
1272 struct intel_context * const ce = rq->context;
1273 u32 head;
1274
1275 /*
1276 * The executing context has been cancelled. We want to prevent
1277 * further execution along this context and propagate the error on
1278 * to anything depending on its results.
1279 *
1280 * In __i915_request_submit(), we apply the -EIO and remove the
1281 * requests' payloads for any banned requests. But first, we must
1282 * rewind the context back to the start of the incomplete request so
1283 * that we do not jump back into the middle of the batch.
1284 *
1285 * We preserve the breadcrumbs and semaphores of the incomplete
1286 * requests so that inter-timeline dependencies (i.e other timelines)
1287 * remain correctly ordered. And we defer to __i915_request_submit()
1288 * so that all asynchronous waits are correctly handled.
1289 */
1290 ENGINE_TRACE(engine, "{ rq=%llx:%lld }\n",
1291 rq->fence.context, rq->fence.seqno);
1292
1293 /* On resubmission of the active request, payload will be scrubbed */
1294 if (i915_request_completed(rq))
1295 head = rq->tail;
1296 else
1297 head = active_request(ce->timeline, rq)->head;
1298 head = intel_ring_wrap(ce->ring, head);
1299
1300 /* Scrub the context image to prevent replaying the previous batch */
1301 restore_default_state(ce, engine);
1302 __execlists_update_reg_state(ce, engine, head);
1303
1304 /* We've switched away, so this should be a no-op, but intent matters */
1305 ce->lrc.desc |= CTX_DESC_FORCE_RESTORE;
1306}
1307
1308static void st_update_runtime_underflow(struct intel_context *ce, s32 dt)
1309{
1310#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
1311 ce->runtime.num_underflow += dt < 0;
1312 ce->runtime.max_underflow = max_t(u32, ce->runtime.max_underflow, -dt);
1313#endif
1314}
1315
1316static void intel_context_update_runtime(struct intel_context *ce)
1317{
1318 u32 old;
1319 s32 dt;
1320
1321 if (intel_context_is_barrier(ce))
1322 return;
1323
1324 old = ce->runtime.last;
1325 ce->runtime.last = intel_context_get_runtime(ce);
1326 dt = ce->runtime.last - old;
1327
1328 if (unlikely(dt <= 0)) {
1329 CE_TRACE(ce, "runtime underflow: last=%u, new=%u, delta=%d\n",
1330 old, ce->runtime.last, dt);
1331 st_update_runtime_underflow(ce, dt);
1332 return;
1333 }
1334
1335 ewma_runtime_add(&ce->runtime.avg, dt);
1336 ce->runtime.total += dt;
1337}
1338
1339static inline struct intel_engine_cs *
1340__execlists_schedule_in(struct i915_request *rq)
1341{
1342 struct intel_engine_cs * const engine = rq->engine;
1343 struct intel_context * const ce = rq->context;
1344
1345 intel_context_get(ce);
1346
1347 if (unlikely(intel_context_is_banned(ce)))
1348 reset_active(rq, engine);
1349
1350 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
1351 execlists_check_context(ce, engine, "before");
1352
1353 if (ce->tag) {
1354 /* Use a fixed tag for OA and friends */
1355 GEM_BUG_ON(ce->tag <= BITS_PER_LONG);
1356 ce->lrc.ccid = ce->tag;
1357 } else {
1358 /* We don't need a strict matching tag, just different values */
1359 unsigned int tag = ffs(READ_ONCE(engine->context_tag));
1360
1361 GEM_BUG_ON(tag == 0 || tag >= BITS_PER_LONG);
1362 clear_bit(tag - 1, &engine->context_tag);
1363 ce->lrc.ccid = tag << (GEN11_SW_CTX_ID_SHIFT - 32);
1364
1365 BUILD_BUG_ON(BITS_PER_LONG > GEN12_MAX_CONTEXT_HW_ID);
1366 }
1367
1368 ce->lrc.ccid |= engine->execlists.ccid;
1369
1370 __intel_gt_pm_get(engine->gt);
1371 if (engine->fw_domain && !atomic_fetch_inc(&engine->fw_active))
1372 intel_uncore_forcewake_get(engine->uncore, engine->fw_domain);
1373 execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_IN);
1374 intel_engine_context_in(engine);
1375
1376 return engine;
1377}
1378
1379static inline struct i915_request *
1380execlists_schedule_in(struct i915_request *rq, int idx)
1381{
1382 struct intel_context * const ce = rq->context;
1383 struct intel_engine_cs *old;
1384
1385 GEM_BUG_ON(!intel_engine_pm_is_awake(rq->engine));
1386 trace_i915_request_in(rq, idx);
1387
1388 old = READ_ONCE(ce->inflight);
1389 do {
1390 if (!old) {
1391 WRITE_ONCE(ce->inflight, __execlists_schedule_in(rq));
1392 break;
1393 }
1394 } while (!try_cmpxchg(&ce->inflight, &old, ptr_inc(old)));
1395
1396 GEM_BUG_ON(intel_context_inflight(ce) != rq->engine);
1397 return i915_request_get(rq);
1398}
1399
1400static void kick_siblings(struct i915_request *rq, struct intel_context *ce)
1401{
1402 struct virtual_engine *ve = container_of(ce, typeof(*ve), context);
1403 struct i915_request *next = READ_ONCE(ve->request);
1404
1405 if (next == rq || (next && next->execution_mask & ~rq->execution_mask))
1406 tasklet_hi_schedule(&ve->base.execlists.tasklet);
1407}
1408
1409static inline void
1410__execlists_schedule_out(struct i915_request *rq,
1411 struct intel_engine_cs * const engine,
1412 unsigned int ccid)
1413{
1414 struct intel_context * const ce = rq->context;
1415
1416 /*
1417 * NB process_csb() is not under the engine->active.lock and hence
1418 * schedule_out can race with schedule_in meaning that we should
1419 * refrain from doing non-trivial work here.
1420 */
1421
1422 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
1423 execlists_check_context(ce, engine, "after");
1424
1425 /*
1426 * If we have just completed this context, the engine may now be
1427 * idle and we want to re-enter powersaving.
1428 */
1429 if (list_is_last_rcu(&rq->link, &ce->timeline->requests) &&
1430 i915_request_completed(rq))
1431 intel_engine_add_retire(engine, ce->timeline);
1432
1433 ccid >>= GEN11_SW_CTX_ID_SHIFT - 32;
1434 ccid &= GEN12_MAX_CONTEXT_HW_ID;
1435 if (ccid < BITS_PER_LONG) {
1436 GEM_BUG_ON(ccid == 0);
1437 GEM_BUG_ON(test_bit(ccid - 1, &engine->context_tag));
1438 set_bit(ccid - 1, &engine->context_tag);
1439 }
1440
1441 intel_context_update_runtime(ce);
1442 intel_engine_context_out(engine);
1443 execlists_context_status_change(rq, INTEL_CONTEXT_SCHEDULE_OUT);
1444 if (engine->fw_domain && !atomic_dec_return(&engine->fw_active))
1445 intel_uncore_forcewake_put(engine->uncore, engine->fw_domain);
1446 intel_gt_pm_put_async(engine->gt);
1447
1448 /*
1449 * If this is part of a virtual engine, its next request may
1450 * have been blocked waiting for access to the active context.
1451 * We have to kick all the siblings again in case we need to
1452 * switch (e.g. the next request is not runnable on this
1453 * engine). Hopefully, we will already have submitted the next
1454 * request before the tasklet runs and do not need to rebuild
1455 * each virtual tree and kick everyone again.
1456 */
1457 if (ce->engine != engine)
1458 kick_siblings(rq, ce);
1459
1460 intel_context_put(ce);
1461}
1462
1463static inline void
1464execlists_schedule_out(struct i915_request *rq)
1465{
1466 struct intel_context * const ce = rq->context;
1467 struct intel_engine_cs *cur, *old;
1468 u32 ccid;
1469
1470 trace_i915_request_out(rq);
1471
1472 ccid = rq->context->lrc.ccid;
1473 old = READ_ONCE(ce->inflight);
1474 do
1475 cur = ptr_unmask_bits(old, 2) ? ptr_dec(old) : NULL;
1476 while (!try_cmpxchg(&ce->inflight, &old, cur));
1477 if (!cur)
1478 __execlists_schedule_out(rq, old, ccid);
1479
1480 i915_request_put(rq);
1481}
1482
1483static u64 execlists_update_context(struct i915_request *rq)
1484{
1485 struct intel_context *ce = rq->context;
1486 u64 desc = ce->lrc.desc;
1487 u32 tail, prev;
1488
1489 /*
1490 * WaIdleLiteRestore:bdw,skl
1491 *
1492 * We should never submit the context with the same RING_TAIL twice
1493 * just in case we submit an empty ring, which confuses the HW.
1494 *
1495 * We append a couple of NOOPs (gen8_emit_wa_tail) after the end of
1496 * the normal request to be able to always advance the RING_TAIL on
1497 * subsequent resubmissions (for lite restore). Should that fail us,
1498 * and we try and submit the same tail again, force the context
1499 * reload.
1500 *
1501 * If we need to return to a preempted context, we need to skip the
1502 * lite-restore and force it to reload the RING_TAIL. Otherwise, the
1503 * HW has a tendency to ignore us rewinding the TAIL to the end of
1504 * an earlier request.
1505 */
1506 GEM_BUG_ON(ce->lrc_reg_state[CTX_RING_TAIL] != rq->ring->tail);
1507 prev = rq->ring->tail;
1508 tail = intel_ring_set_tail(rq->ring, rq->tail);
1509 if (unlikely(intel_ring_direction(rq->ring, tail, prev) <= 0))
1510 desc |= CTX_DESC_FORCE_RESTORE;
1511 ce->lrc_reg_state[CTX_RING_TAIL] = tail;
1512 rq->tail = rq->wa_tail;
1513
1514 /*
1515 * Make sure the context image is complete before we submit it to HW.
1516 *
1517 * Ostensibly, writes (including the WCB) should be flushed prior to
1518 * an uncached write such as our mmio register access, the empirical
1519 * evidence (esp. on Braswell) suggests that the WC write into memory
1520 * may not be visible to the HW prior to the completion of the UC
1521 * register write and that we may begin execution from the context
1522 * before its image is complete leading to invalid PD chasing.
1523 */
1524 wmb();
1525
1526 ce->lrc.desc &= ~CTX_DESC_FORCE_RESTORE;
1527 return desc;
1528}
1529
1530static inline void write_desc(struct intel_engine_execlists *execlists, u64 desc, u32 port)
1531{
1532 if (execlists->ctrl_reg) {
1533 writel(lower_32_bits(desc), execlists->submit_reg + port * 2);
1534 writel(upper_32_bits(desc), execlists->submit_reg + port * 2 + 1);
1535 } else {
1536 writel(upper_32_bits(desc), execlists->submit_reg);
1537 writel(lower_32_bits(desc), execlists->submit_reg);
1538 }
1539}
1540
1541static __maybe_unused char *
1542dump_port(char *buf, int buflen, const char *prefix, struct i915_request *rq)
1543{
1544 if (!rq)
1545 return "";
1546
1547 snprintf(buf, buflen, "%sccid:%x %llx:%lld%s prio %d",
1548 prefix,
1549 rq->context->lrc.ccid,
1550 rq->fence.context, rq->fence.seqno,
1551 i915_request_completed(rq) ? "!" :
1552 i915_request_started(rq) ? "*" :
1553 "",
1554 rq_prio(rq));
1555
1556 return buf;
1557}
1558
1559static __maybe_unused void
1560trace_ports(const struct intel_engine_execlists *execlists,
1561 const char *msg,
1562 struct i915_request * const *ports)
1563{
1564 const struct intel_engine_cs *engine =
1565 container_of(execlists, typeof(*engine), execlists);
1566 char __maybe_unused p0[40], p1[40];
1567
1568 if (!ports[0])
1569 return;
1570
1571 ENGINE_TRACE(engine, "%s { %s%s }\n", msg,
1572 dump_port(p0, sizeof(p0), "", ports[0]),
1573 dump_port(p1, sizeof(p1), ", ", ports[1]));
1574}
1575
1576static inline bool
1577reset_in_progress(const struct intel_engine_execlists *execlists)
1578{
1579 return unlikely(!__tasklet_is_enabled(&execlists->tasklet));
1580}
1581
1582static __maybe_unused bool
1583assert_pending_valid(const struct intel_engine_execlists *execlists,
1584 const char *msg)
1585{
1586 struct intel_engine_cs *engine =
1587 container_of(execlists, typeof(*engine), execlists);
1588 struct i915_request * const *port, *rq;
1589 struct intel_context *ce = NULL;
1590 bool sentinel = false;
1591 u32 ccid = -1;
1592
1593 trace_ports(execlists, msg, execlists->pending);
1594
1595 /* We may be messing around with the lists during reset, lalala */
1596 if (reset_in_progress(execlists))
1597 return true;
1598
1599 if (!execlists->pending[0]) {
1600 GEM_TRACE_ERR("%s: Nothing pending for promotion!\n",
1601 engine->name);
1602 return false;
1603 }
1604
1605 if (execlists->pending[execlists_num_ports(execlists)]) {
1606 GEM_TRACE_ERR("%s: Excess pending[%d] for promotion!\n",
1607 engine->name, execlists_num_ports(execlists));
1608 return false;
1609 }
1610
1611 for (port = execlists->pending; (rq = *port); port++) {
1612 unsigned long flags;
1613 bool ok = true;
1614
1615 GEM_BUG_ON(!kref_read(&rq->fence.refcount));
1616 GEM_BUG_ON(!i915_request_is_active(rq));
1617
1618 if (ce == rq->context) {
1619 GEM_TRACE_ERR("%s: Dup context:%llx in pending[%zd]\n",
1620 engine->name,
1621 ce->timeline->fence_context,
1622 port - execlists->pending);
1623 return false;
1624 }
1625 ce = rq->context;
1626
1627 if (ccid == ce->lrc.ccid) {
1628 GEM_TRACE_ERR("%s: Dup ccid:%x context:%llx in pending[%zd]\n",
1629 engine->name,
1630 ccid, ce->timeline->fence_context,
1631 port - execlists->pending);
1632 return false;
1633 }
1634 ccid = ce->lrc.ccid;
1635
1636 /*
1637 * Sentinels are supposed to be the last request so they flush
1638 * the current execution off the HW. Check that they are the only
1639 * request in the pending submission.
1640 */
1641 if (sentinel) {
1642 GEM_TRACE_ERR("%s: context:%llx after sentinel in pending[%zd]\n",
1643 engine->name,
1644 ce->timeline->fence_context,
1645 port - execlists->pending);
1646 return false;
1647 }
1648 sentinel = i915_request_has_sentinel(rq);
1649
1650 /* Hold tightly onto the lock to prevent concurrent retires! */
1651 if (!spin_trylock_irqsave(&rq->lock, flags))
1652 continue;
1653
1654 if (i915_request_completed(rq))
1655 goto unlock;
1656
1657 if (i915_active_is_idle(&ce->active) &&
1658 !intel_context_is_barrier(ce)) {
1659 GEM_TRACE_ERR("%s: Inactive context:%llx in pending[%zd]\n",
1660 engine->name,
1661 ce->timeline->fence_context,
1662 port - execlists->pending);
1663 ok = false;
1664 goto unlock;
1665 }
1666
1667 if (!i915_vma_is_pinned(ce->state)) {
1668 GEM_TRACE_ERR("%s: Unpinned context:%llx in pending[%zd]\n",
1669 engine->name,
1670 ce->timeline->fence_context,
1671 port - execlists->pending);
1672 ok = false;
1673 goto unlock;
1674 }
1675
1676 if (!i915_vma_is_pinned(ce->ring->vma)) {
1677 GEM_TRACE_ERR("%s: Unpinned ring:%llx in pending[%zd]\n",
1678 engine->name,
1679 ce->timeline->fence_context,
1680 port - execlists->pending);
1681 ok = false;
1682 goto unlock;
1683 }
1684
1685unlock:
1686 spin_unlock_irqrestore(&rq->lock, flags);
1687 if (!ok)
1688 return false;
1689 }
1690
1691 return ce;
1692}
1693
1694static void execlists_submit_ports(struct intel_engine_cs *engine)
1695{
1696 struct intel_engine_execlists *execlists = &engine->execlists;
1697 unsigned int n;
1698
1699 GEM_BUG_ON(!assert_pending_valid(execlists, "submit"));
1700
1701 /*
1702 * We can skip acquiring intel_runtime_pm_get() here as it was taken
1703 * on our behalf by the request (see i915_gem_mark_busy()) and it will
1704 * not be relinquished until the device is idle (see
1705 * i915_gem_idle_work_handler()). As a precaution, we make sure
1706 * that all ELSP are drained i.e. we have processed the CSB,
1707 * before allowing ourselves to idle and calling intel_runtime_pm_put().
1708 */
1709 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
1710
1711 /*
1712 * ELSQ note: the submit queue is not cleared after being submitted
1713 * to the HW so we need to make sure we always clean it up. This is
1714 * currently ensured by the fact that we always write the same number
1715 * of elsq entries, keep this in mind before changing the loop below.
1716 */
1717 for (n = execlists_num_ports(execlists); n--; ) {
1718 struct i915_request *rq = execlists->pending[n];
1719
1720 write_desc(execlists,
1721 rq ? execlists_update_context(rq) : 0,
1722 n);
1723 }
1724
1725 /* we need to manually load the submit queue */
1726 if (execlists->ctrl_reg)
1727 writel(EL_CTRL_LOAD, execlists->ctrl_reg);
1728}
1729
1730static bool ctx_single_port_submission(const struct intel_context *ce)
1731{
1732 return (IS_ENABLED(CONFIG_DRM_I915_GVT) &&
1733 intel_context_force_single_submission(ce));
1734}
1735
1736static bool can_merge_ctx(const struct intel_context *prev,
1737 const struct intel_context *next)
1738{
1739 if (prev != next)
1740 return false;
1741
1742 if (ctx_single_port_submission(prev))
1743 return false;
1744
1745 return true;
1746}
1747
1748static unsigned long i915_request_flags(const struct i915_request *rq)
1749{
1750 return READ_ONCE(rq->fence.flags);
1751}
1752
1753static bool can_merge_rq(const struct i915_request *prev,
1754 const struct i915_request *next)
1755{
1756 GEM_BUG_ON(prev == next);
1757 GEM_BUG_ON(!assert_priority_queue(prev, next));
1758
1759 /*
1760 * We do not submit known completed requests. Therefore if the next
1761 * request is already completed, we can pretend to merge it in
1762 * with the previous context (and we will skip updating the ELSP
1763 * and tracking). Thus hopefully keeping the ELSP full with active
1764 * contexts, despite the best efforts of preempt-to-busy to confuse
1765 * us.
1766 */
1767 if (i915_request_completed(next))
1768 return true;
1769
1770 if (unlikely((i915_request_flags(prev) ^ i915_request_flags(next)) &
1771 (BIT(I915_FENCE_FLAG_NOPREEMPT) |
1772 BIT(I915_FENCE_FLAG_SENTINEL))))
1773 return false;
1774
1775 if (!can_merge_ctx(prev->context, next->context))
1776 return false;
1777
1778 GEM_BUG_ON(i915_seqno_passed(prev->fence.seqno, next->fence.seqno));
1779 return true;
1780}
1781
1782static void virtual_update_register_offsets(u32 *regs,
1783 struct intel_engine_cs *engine)
1784{
1785 set_offsets(regs, reg_offsets(engine), engine, false);
1786}
1787
1788static bool virtual_matches(const struct virtual_engine *ve,
1789 const struct i915_request *rq,
1790 const struct intel_engine_cs *engine)
1791{
1792 const struct intel_engine_cs *inflight;
1793
1794 if (!(rq->execution_mask & engine->mask)) /* We peeked too soon! */
1795 return false;
1796
1797 /*
1798 * We track when the HW has completed saving the context image
1799 * (i.e. when we have seen the final CS event switching out of
1800 * the context) and must not overwrite the context image before
1801 * then. This restricts us to only using the active engine
1802 * while the previous virtualized request is inflight (so
1803 * we reuse the register offsets). This is a very small
1804 * hystersis on the greedy seelction algorithm.
1805 */
1806 inflight = intel_context_inflight(&ve->context);
1807 if (inflight && inflight != engine)
1808 return false;
1809
1810 return true;
1811}
1812
1813static void virtual_xfer_context(struct virtual_engine *ve,
1814 struct intel_engine_cs *engine)
1815{
1816 unsigned int n;
1817
1818 if (likely(engine == ve->siblings[0]))
1819 return;
1820
1821 GEM_BUG_ON(READ_ONCE(ve->context.inflight));
1822 if (!intel_engine_has_relative_mmio(engine))
1823 virtual_update_register_offsets(ve->context.lrc_reg_state,
1824 engine);
1825
1826 /*
1827 * Move the bound engine to the top of the list for
1828 * future execution. We then kick this tasklet first
1829 * before checking others, so that we preferentially
1830 * reuse this set of bound registers.
1831 */
1832 for (n = 1; n < ve->num_siblings; n++) {
1833 if (ve->siblings[n] == engine) {
1834 swap(ve->siblings[n], ve->siblings[0]);
1835 break;
1836 }
1837 }
1838}
1839
1840#define for_each_waiter(p__, rq__) \
1841 list_for_each_entry_lockless(p__, \
1842 &(rq__)->sched.waiters_list, \
1843 wait_link)
1844
1845#define for_each_signaler(p__, rq__) \
1846 list_for_each_entry_rcu(p__, \
1847 &(rq__)->sched.signalers_list, \
1848 signal_link)
1849
1850static void defer_request(struct i915_request *rq, struct list_head * const pl)
1851{
1852 LIST_HEAD(list);
1853
1854 /*
1855 * We want to move the interrupted request to the back of
1856 * the round-robin list (i.e. its priority level), but
1857 * in doing so, we must then move all requests that were in
1858 * flight and were waiting for the interrupted request to
1859 * be run after it again.
1860 */
1861 do {
1862 struct i915_dependency *p;
1863
1864 GEM_BUG_ON(i915_request_is_active(rq));
1865 list_move_tail(&rq->sched.link, pl);
1866
1867 for_each_waiter(p, rq) {
1868 struct i915_request *w =
1869 container_of(p->waiter, typeof(*w), sched);
1870
1871 if (p->flags & I915_DEPENDENCY_WEAK)
1872 continue;
1873
1874 /* Leave semaphores spinning on the other engines */
1875 if (w->engine != rq->engine)
1876 continue;
1877
1878 /* No waiter should start before its signaler */
1879 GEM_BUG_ON(i915_request_has_initial_breadcrumb(w) &&
1880 i915_request_started(w) &&
1881 !i915_request_completed(rq));
1882
1883 GEM_BUG_ON(i915_request_is_active(w));
1884 if (!i915_request_is_ready(w))
1885 continue;
1886
1887 if (rq_prio(w) < rq_prio(rq))
1888 continue;
1889
1890 GEM_BUG_ON(rq_prio(w) > rq_prio(rq));
1891 list_move_tail(&w->sched.link, &list);
1892 }
1893
1894 rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
1895 } while (rq);
1896}
1897
1898static void defer_active(struct intel_engine_cs *engine)
1899{
1900 struct i915_request *rq;
1901
1902 rq = __unwind_incomplete_requests(engine);
1903 if (!rq)
1904 return;
1905
1906 defer_request(rq, i915_sched_lookup_priolist(engine, rq_prio(rq)));
1907}
1908
1909static bool
1910need_timeslice(const struct intel_engine_cs *engine,
1911 const struct i915_request *rq,
1912 const struct rb_node *rb)
1913{
1914 int hint;
1915
1916 if (!intel_engine_has_timeslices(engine))
1917 return false;
1918
1919 hint = engine->execlists.queue_priority_hint;
1920
1921 if (rb) {
1922 const struct virtual_engine *ve =
1923 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
1924 const struct intel_engine_cs *inflight =
1925 intel_context_inflight(&ve->context);
1926
1927 if (!inflight || inflight == engine) {
1928 struct i915_request *next;
1929
1930 rcu_read_lock();
1931 next = READ_ONCE(ve->request);
1932 if (next)
1933 hint = max(hint, rq_prio(next));
1934 rcu_read_unlock();
1935 }
1936 }
1937
1938 if (!list_is_last(&rq->sched.link, &engine->active.requests))
1939 hint = max(hint, rq_prio(list_next_entry(rq, sched.link)));
1940
1941 GEM_BUG_ON(hint >= I915_PRIORITY_UNPREEMPTABLE);
1942 return hint >= effective_prio(rq);
1943}
1944
1945static bool
1946timeslice_yield(const struct intel_engine_execlists *el,
1947 const struct i915_request *rq)
1948{
1949 /*
1950 * Once bitten, forever smitten!
1951 *
1952 * If the active context ever busy-waited on a semaphore,
1953 * it will be treated as a hog until the end of its timeslice (i.e.
1954 * until it is scheduled out and replaced by a new submission,
1955 * possibly even its own lite-restore). The HW only sends an interrupt
1956 * on the first miss, and we do know if that semaphore has been
1957 * signaled, or even if it is now stuck on another semaphore. Play
1958 * safe, yield if it might be stuck -- it will be given a fresh
1959 * timeslice in the near future.
1960 */
1961 return rq->context->lrc.ccid == READ_ONCE(el->yield);
1962}
1963
1964static bool
1965timeslice_expired(const struct intel_engine_execlists *el,
1966 const struct i915_request *rq)
1967{
1968 return timer_expired(&el->timer) || timeslice_yield(el, rq);
1969}
1970
1971static int
1972switch_prio(struct intel_engine_cs *engine, const struct i915_request *rq)
1973{
1974 if (list_is_last(&rq->sched.link, &engine->active.requests))
1975 return engine->execlists.queue_priority_hint;
1976
1977 return rq_prio(list_next_entry(rq, sched.link));
1978}
1979
1980static inline unsigned long
1981timeslice(const struct intel_engine_cs *engine)
1982{
1983 return READ_ONCE(engine->props.timeslice_duration_ms);
1984}
1985
1986static unsigned long active_timeslice(const struct intel_engine_cs *engine)
1987{
1988 const struct intel_engine_execlists *execlists = &engine->execlists;
1989 const struct i915_request *rq = *execlists->active;
1990
1991 if (!rq || i915_request_completed(rq))
1992 return 0;
1993
1994 if (READ_ONCE(execlists->switch_priority_hint) < effective_prio(rq))
1995 return 0;
1996
1997 return timeslice(engine);
1998}
1999
2000static void set_timeslice(struct intel_engine_cs *engine)
2001{
2002 unsigned long duration;
2003
2004 if (!intel_engine_has_timeslices(engine))
2005 return;
2006
2007 duration = active_timeslice(engine);
2008 ENGINE_TRACE(engine, "bump timeslicing, interval:%lu", duration);
2009
2010 set_timer_ms(&engine->execlists.timer, duration);
2011}
2012
2013static void start_timeslice(struct intel_engine_cs *engine, int prio)
2014{
2015 struct intel_engine_execlists *execlists = &engine->execlists;
2016 unsigned long duration;
2017
2018 if (!intel_engine_has_timeslices(engine))
2019 return;
2020
2021 WRITE_ONCE(execlists->switch_priority_hint, prio);
2022 if (prio == INT_MIN)
2023 return;
2024
2025 if (timer_pending(&execlists->timer))
2026 return;
2027
2028 duration = timeslice(engine);
2029 ENGINE_TRACE(engine,
2030 "start timeslicing, prio:%d, interval:%lu",
2031 prio, duration);
2032
2033 set_timer_ms(&execlists->timer, duration);
2034}
2035
2036static void record_preemption(struct intel_engine_execlists *execlists)
2037{
2038 (void)I915_SELFTEST_ONLY(execlists->preempt_hang.count++);
2039}
2040
2041static unsigned long active_preempt_timeout(struct intel_engine_cs *engine,
2042 const struct i915_request *rq)
2043{
2044 if (!rq)
2045 return 0;
2046
2047 /* Force a fast reset for terminated contexts (ignoring sysfs!) */
2048 if (unlikely(intel_context_is_banned(rq->context)))
2049 return 1;
2050
2051 return READ_ONCE(engine->props.preempt_timeout_ms);
2052}
2053
2054static void set_preempt_timeout(struct intel_engine_cs *engine,
2055 const struct i915_request *rq)
2056{
2057 if (!intel_engine_has_preempt_reset(engine))
2058 return;
2059
2060 set_timer_ms(&engine->execlists.preempt,
2061 active_preempt_timeout(engine, rq));
2062}
2063
2064static inline void clear_ports(struct i915_request **ports, int count)
2065{
2066 memset_p((void **)ports, NULL, count);
2067}
2068
2069static inline void
2070copy_ports(struct i915_request **dst, struct i915_request **src, int count)
2071{
2072 /* A memcpy_p() would be very useful here! */
2073 while (count--)
2074 WRITE_ONCE(*dst++, *src++); /* avoid write tearing */
2075}
2076
2077static void execlists_dequeue(struct intel_engine_cs *engine)
2078{
2079 struct intel_engine_execlists * const execlists = &engine->execlists;
2080 struct i915_request **port = execlists->pending;
2081 struct i915_request ** const last_port = port + execlists->port_mask;
2082 struct i915_request * const *active;
2083 struct i915_request *last;
2084 struct rb_node *rb;
2085 bool submit = false;
2086
2087 /*
2088 * Hardware submission is through 2 ports. Conceptually each port
2089 * has a (RING_START, RING_HEAD, RING_TAIL) tuple. RING_START is
2090 * static for a context, and unique to each, so we only execute
2091 * requests belonging to a single context from each ring. RING_HEAD
2092 * is maintained by the CS in the context image, it marks the place
2093 * where it got up to last time, and through RING_TAIL we tell the CS
2094 * where we want to execute up to this time.
2095 *
2096 * In this list the requests are in order of execution. Consecutive
2097 * requests from the same context are adjacent in the ringbuffer. We
2098 * can combine these requests into a single RING_TAIL update:
2099 *
2100 * RING_HEAD...req1...req2
2101 * ^- RING_TAIL
2102 * since to execute req2 the CS must first execute req1.
2103 *
2104 * Our goal then is to point each port to the end of a consecutive
2105 * sequence of requests as being the most optimal (fewest wake ups
2106 * and context switches) submission.
2107 */
2108
2109 for (rb = rb_first_cached(&execlists->virtual); rb; ) {
2110 struct virtual_engine *ve =
2111 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
2112 struct i915_request *rq = READ_ONCE(ve->request);
2113
2114 if (!rq) { /* lazily cleanup after another engine handled rq */
2115 rb_erase_cached(rb, &execlists->virtual);
2116 RB_CLEAR_NODE(rb);
2117 rb = rb_first_cached(&execlists->virtual);
2118 continue;
2119 }
2120
2121 if (!virtual_matches(ve, rq, engine)) {
2122 rb = rb_next(rb);
2123 continue;
2124 }
2125
2126 break;
2127 }
2128
2129 /*
2130 * If the queue is higher priority than the last
2131 * request in the currently active context, submit afresh.
2132 * We will resubmit again afterwards in case we need to split
2133 * the active context to interject the preemption request,
2134 * i.e. we will retrigger preemption following the ack in case
2135 * of trouble.
2136 */
2137 active = READ_ONCE(execlists->active);
2138
2139 /*
2140 * In theory we can skip over completed contexts that have not
2141 * yet been processed by events (as those events are in flight):
2142 *
2143 * while ((last = *active) && i915_request_completed(last))
2144 * active++;
2145 *
2146 * However, the GPU cannot handle this as it will ultimately
2147 * find itself trying to jump back into a context it has just
2148 * completed and barf.
2149 */
2150
2151 if ((last = *active)) {
2152 if (need_preempt(engine, last, rb)) {
2153 if (i915_request_completed(last)) {
2154 tasklet_hi_schedule(&execlists->tasklet);
2155 return;
2156 }
2157
2158 ENGINE_TRACE(engine,
2159 "preempting last=%llx:%lld, prio=%d, hint=%d\n",
2160 last->fence.context,
2161 last->fence.seqno,
2162 last->sched.attr.priority,
2163 execlists->queue_priority_hint);
2164 record_preemption(execlists);
2165
2166 /*
2167 * Don't let the RING_HEAD advance past the breadcrumb
2168 * as we unwind (and until we resubmit) so that we do
2169 * not accidentally tell it to go backwards.
2170 */
2171 ring_set_paused(engine, 1);
2172
2173 /*
2174 * Note that we have not stopped the GPU at this point,
2175 * so we are unwinding the incomplete requests as they
2176 * remain inflight and so by the time we do complete
2177 * the preemption, some of the unwound requests may
2178 * complete!
2179 */
2180 __unwind_incomplete_requests(engine);
2181
2182 last = NULL;
2183 } else if (need_timeslice(engine, last, rb) &&
2184 timeslice_expired(execlists, last)) {
2185 if (i915_request_completed(last)) {
2186 tasklet_hi_schedule(&execlists->tasklet);
2187 return;
2188 }
2189
2190 ENGINE_TRACE(engine,
2191 "expired last=%llx:%lld, prio=%d, hint=%d, yield?=%s\n",
2192 last->fence.context,
2193 last->fence.seqno,
2194 last->sched.attr.priority,
2195 execlists->queue_priority_hint,
2196 yesno(timeslice_yield(execlists, last)));
2197
2198 ring_set_paused(engine, 1);
2199 defer_active(engine);
2200
2201 /*
2202 * Unlike for preemption, if we rewind and continue
2203 * executing the same context as previously active,
2204 * the order of execution will remain the same and
2205 * the tail will only advance. We do not need to
2206 * force a full context restore, as a lite-restore
2207 * is sufficient to resample the monotonic TAIL.
2208 *
2209 * If we switch to any other context, similarly we
2210 * will not rewind TAIL of current context, and
2211 * normal save/restore will preserve state and allow
2212 * us to later continue executing the same request.
2213 */
2214 last = NULL;
2215 } else {
2216 /*
2217 * Otherwise if we already have a request pending
2218 * for execution after the current one, we can
2219 * just wait until the next CS event before
2220 * queuing more. In either case we will force a
2221 * lite-restore preemption event, but if we wait
2222 * we hopefully coalesce several updates into a single
2223 * submission.
2224 */
2225 if (!list_is_last(&last->sched.link,
2226 &engine->active.requests)) {
2227 /*
2228 * Even if ELSP[1] is occupied and not worthy
2229 * of timeslices, our queue might be.
2230 */
2231 start_timeslice(engine, queue_prio(execlists));
2232 return;
2233 }
2234 }
2235 }
2236
2237 while (rb) { /* XXX virtual is always taking precedence */
2238 struct virtual_engine *ve =
2239 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
2240 struct i915_request *rq;
2241
2242 spin_lock(&ve->base.active.lock);
2243
2244 rq = ve->request;
2245 if (unlikely(!rq)) { /* lost the race to a sibling */
2246 spin_unlock(&ve->base.active.lock);
2247 rb_erase_cached(rb, &execlists->virtual);
2248 RB_CLEAR_NODE(rb);
2249 rb = rb_first_cached(&execlists->virtual);
2250 continue;
2251 }
2252
2253 GEM_BUG_ON(rq != ve->request);
2254 GEM_BUG_ON(rq->engine != &ve->base);
2255 GEM_BUG_ON(rq->context != &ve->context);
2256
2257 if (rq_prio(rq) >= queue_prio(execlists)) {
2258 if (!virtual_matches(ve, rq, engine)) {
2259 spin_unlock(&ve->base.active.lock);
2260 rb = rb_next(rb);
2261 continue;
2262 }
2263
2264 if (last && !can_merge_rq(last, rq)) {
2265 spin_unlock(&ve->base.active.lock);
2266 start_timeslice(engine, rq_prio(rq));
2267 return; /* leave this for another sibling */
2268 }
2269
2270 ENGINE_TRACE(engine,
2271 "virtual rq=%llx:%lld%s, new engine? %s\n",
2272 rq->fence.context,
2273 rq->fence.seqno,
2274 i915_request_completed(rq) ? "!" :
2275 i915_request_started(rq) ? "*" :
2276 "",
2277 yesno(engine != ve->siblings[0]));
2278
2279 WRITE_ONCE(ve->request, NULL);
2280 WRITE_ONCE(ve->base.execlists.queue_priority_hint,
2281 INT_MIN);
2282 rb_erase_cached(rb, &execlists->virtual);
2283 RB_CLEAR_NODE(rb);
2284
2285 GEM_BUG_ON(!(rq->execution_mask & engine->mask));
2286 WRITE_ONCE(rq->engine, engine);
2287
2288 if (__i915_request_submit(rq)) {
2289 /*
2290 * Only after we confirm that we will submit
2291 * this request (i.e. it has not already
2292 * completed), do we want to update the context.
2293 *
2294 * This serves two purposes. It avoids
2295 * unnecessary work if we are resubmitting an
2296 * already completed request after timeslicing.
2297 * But more importantly, it prevents us altering
2298 * ve->siblings[] on an idle context, where
2299 * we may be using ve->siblings[] in
2300 * virtual_context_enter / virtual_context_exit.
2301 */
2302 virtual_xfer_context(ve, engine);
2303 GEM_BUG_ON(ve->siblings[0] != engine);
2304
2305 submit = true;
2306 last = rq;
2307 }
2308 i915_request_put(rq);
2309
2310 /*
2311 * Hmm, we have a bunch of virtual engine requests,
2312 * but the first one was already completed (thanks
2313 * preempt-to-busy!). Keep looking at the veng queue
2314 * until we have no more relevant requests (i.e.
2315 * the normal submit queue has higher priority).
2316 */
2317 if (!submit) {
2318 spin_unlock(&ve->base.active.lock);
2319 rb = rb_first_cached(&execlists->virtual);
2320 continue;
2321 }
2322 }
2323
2324 spin_unlock(&ve->base.active.lock);
2325 break;
2326 }
2327
2328 while ((rb = rb_first_cached(&execlists->queue))) {
2329 struct i915_priolist *p = to_priolist(rb);
2330 struct i915_request *rq, *rn;
2331 int i;
2332
2333 priolist_for_each_request_consume(rq, rn, p, i) {
2334 bool merge = true;
2335
2336 /*
2337 * Can we combine this request with the current port?
2338 * It has to be the same context/ringbuffer and not
2339 * have any exceptions (e.g. GVT saying never to
2340 * combine contexts).
2341 *
2342 * If we can combine the requests, we can execute both
2343 * by updating the RING_TAIL to point to the end of the
2344 * second request, and so we never need to tell the
2345 * hardware about the first.
2346 */
2347 if (last && !can_merge_rq(last, rq)) {
2348 /*
2349 * If we are on the second port and cannot
2350 * combine this request with the last, then we
2351 * are done.
2352 */
2353 if (port == last_port)
2354 goto done;
2355
2356 /*
2357 * We must not populate both ELSP[] with the
2358 * same LRCA, i.e. we must submit 2 different
2359 * contexts if we submit 2 ELSP.
2360 */
2361 if (last->context == rq->context)
2362 goto done;
2363
2364 if (i915_request_has_sentinel(last))
2365 goto done;
2366
2367 /*
2368 * If GVT overrides us we only ever submit
2369 * port[0], leaving port[1] empty. Note that we
2370 * also have to be careful that we don't queue
2371 * the same context (even though a different
2372 * request) to the second port.
2373 */
2374 if (ctx_single_port_submission(last->context) ||
2375 ctx_single_port_submission(rq->context))
2376 goto done;
2377
2378 merge = false;
2379 }
2380
2381 if (__i915_request_submit(rq)) {
2382 if (!merge) {
2383 *port = execlists_schedule_in(last, port - execlists->pending);
2384 port++;
2385 last = NULL;
2386 }
2387
2388 GEM_BUG_ON(last &&
2389 !can_merge_ctx(last->context,
2390 rq->context));
2391 GEM_BUG_ON(last &&
2392 i915_seqno_passed(last->fence.seqno,
2393 rq->fence.seqno));
2394
2395 submit = true;
2396 last = rq;
2397 }
2398 }
2399
2400 rb_erase_cached(&p->node, &execlists->queue);
2401 i915_priolist_free(p);
2402 }
2403
2404done:
2405 /*
2406 * Here be a bit of magic! Or sleight-of-hand, whichever you prefer.
2407 *
2408 * We choose the priority hint such that if we add a request of greater
2409 * priority than this, we kick the submission tasklet to decide on
2410 * the right order of submitting the requests to hardware. We must
2411 * also be prepared to reorder requests as they are in-flight on the
2412 * HW. We derive the priority hint then as the first "hole" in
2413 * the HW submission ports and if there are no available slots,
2414 * the priority of the lowest executing request, i.e. last.
2415 *
2416 * When we do receive a higher priority request ready to run from the
2417 * user, see queue_request(), the priority hint is bumped to that
2418 * request triggering preemption on the next dequeue (or subsequent
2419 * interrupt for secondary ports).
2420 */
2421 execlists->queue_priority_hint = queue_prio(execlists);
2422
2423 if (submit) {
2424 *port = execlists_schedule_in(last, port - execlists->pending);
2425 execlists->switch_priority_hint =
2426 switch_prio(engine, *execlists->pending);
2427
2428 /*
2429 * Skip if we ended up with exactly the same set of requests,
2430 * e.g. trying to timeslice a pair of ordered contexts
2431 */
2432 if (!memcmp(active, execlists->pending,
2433 (port - execlists->pending + 1) * sizeof(*port))) {
2434 do
2435 execlists_schedule_out(fetch_and_zero(port));
2436 while (port-- != execlists->pending);
2437
2438 goto skip_submit;
2439 }
2440 clear_ports(port + 1, last_port - port);
2441
2442 WRITE_ONCE(execlists->yield, -1);
2443 set_preempt_timeout(engine, *active);
2444 execlists_submit_ports(engine);
2445 } else {
2446 start_timeslice(engine, execlists->queue_priority_hint);
2447skip_submit:
2448 ring_set_paused(engine, 0);
2449 }
2450}
2451
2452static void
2453cancel_port_requests(struct intel_engine_execlists * const execlists)
2454{
2455 struct i915_request * const *port;
2456
2457 for (port = execlists->pending; *port; port++)
2458 execlists_schedule_out(*port);
2459 clear_ports(execlists->pending, ARRAY_SIZE(execlists->pending));
2460
2461 /* Mark the end of active before we overwrite *active */
2462 for (port = xchg(&execlists->active, execlists->pending); *port; port++)
2463 execlists_schedule_out(*port);
2464 clear_ports(execlists->inflight, ARRAY_SIZE(execlists->inflight));
2465
2466 smp_wmb(); /* complete the seqlock for execlists_active() */
2467 WRITE_ONCE(execlists->active, execlists->inflight);
2468}
2469
2470static inline void
2471invalidate_csb_entries(const u64 *first, const u64 *last)
2472{
2473 clflush((void *)first);
2474 clflush((void *)last);
2475}
2476
2477/*
2478 * Starting with Gen12, the status has a new format:
2479 *
2480 * bit 0: switched to new queue
2481 * bit 1: reserved
2482 * bit 2: semaphore wait mode (poll or signal), only valid when
2483 * switch detail is set to "wait on semaphore"
2484 * bits 3-5: engine class
2485 * bits 6-11: engine instance
2486 * bits 12-14: reserved
2487 * bits 15-25: sw context id of the lrc the GT switched to
2488 * bits 26-31: sw counter of the lrc the GT switched to
2489 * bits 32-35: context switch detail
2490 * - 0: ctx complete
2491 * - 1: wait on sync flip
2492 * - 2: wait on vblank
2493 * - 3: wait on scanline
2494 * - 4: wait on semaphore
2495 * - 5: context preempted (not on SEMAPHORE_WAIT or
2496 * WAIT_FOR_EVENT)
2497 * bit 36: reserved
2498 * bits 37-43: wait detail (for switch detail 1 to 4)
2499 * bits 44-46: reserved
2500 * bits 47-57: sw context id of the lrc the GT switched away from
2501 * bits 58-63: sw counter of the lrc the GT switched away from
2502 */
2503static inline bool gen12_csb_parse(const u64 csb)
2504{
2505 bool ctx_away_valid = GEN12_CSB_CTX_VALID(upper_32_bits(csb));
2506 bool new_queue =
2507 lower_32_bits(csb) & GEN12_CTX_STATUS_SWITCHED_TO_NEW_QUEUE;
2508
2509 /*
2510 * The context switch detail is not guaranteed to be 5 when a preemption
2511 * occurs, so we can't just check for that. The check below works for
2512 * all the cases we care about, including preemptions of WAIT
2513 * instructions and lite-restore. Preempt-to-idle via the CTRL register
2514 * would require some extra handling, but we don't support that.
2515 */
2516 if (!ctx_away_valid || new_queue) {
2517 GEM_BUG_ON(!GEN12_CSB_CTX_VALID(lower_32_bits(csb)));
2518 return true;
2519 }
2520
2521 /*
2522 * switch detail = 5 is covered by the case above and we do not expect a
2523 * context switch on an unsuccessful wait instruction since we always
2524 * use polling mode.
2525 */
2526 GEM_BUG_ON(GEN12_CTX_SWITCH_DETAIL(upper_32_bits(csb)));
2527 return false;
2528}
2529
2530static inline bool gen8_csb_parse(const u64 csb)
2531{
2532 return csb & (GEN8_CTX_STATUS_IDLE_ACTIVE | GEN8_CTX_STATUS_PREEMPTED);
2533}
2534
2535static noinline u64
2536wa_csb_read(const struct intel_engine_cs *engine, u64 * const csb)
2537{
2538 u64 entry;
2539
2540 /*
2541 * Reading from the HWSP has one particular advantage: we can detect
2542 * a stale entry. Since the write into HWSP is broken, we have no reason
2543 * to trust the HW at all, the mmio entry may equally be unordered, so
2544 * we prefer the path that is self-checking and as a last resort,
2545 * return the mmio value.
2546 *
2547 * tgl,dg1:HSDES#22011327657
2548 */
2549 preempt_disable();
2550 if (wait_for_atomic_us((entry = READ_ONCE(*csb)) != -1, 10)) {
2551 int idx = csb - engine->execlists.csb_status;
2552 int status;
2553
2554 status = GEN8_EXECLISTS_STATUS_BUF;
2555 if (idx >= 6) {
2556 status = GEN11_EXECLISTS_STATUS_BUF2;
2557 idx -= 6;
2558 }
2559 status += sizeof(u64) * idx;
2560
2561 entry = intel_uncore_read64(engine->uncore,
2562 _MMIO(engine->mmio_base + status));
2563 }
2564 preempt_enable();
2565
2566 return entry;
2567}
2568
2569static inline u64
2570csb_read(const struct intel_engine_cs *engine, u64 * const csb)
2571{
2572 u64 entry = READ_ONCE(*csb);
2573
2574 /*
2575 * Unfortunately, the GPU does not always serialise its write
2576 * of the CSB entries before its write of the CSB pointer, at least
2577 * from the perspective of the CPU, using what is known as a Global
2578 * Observation Point. We may read a new CSB tail pointer, but then
2579 * read the stale CSB entries, causing us to misinterpret the
2580 * context-switch events, and eventually declare the GPU hung.
2581 *
2582 * icl:HSDES#1806554093
2583 * tgl:HSDES#22011248461
2584 */
2585 if (unlikely(entry == -1))
2586 entry = wa_csb_read(engine, csb);
2587
2588 /* Consume this entry so that we can spot its future reuse. */
2589 WRITE_ONCE(*csb, -1);
2590
2591 /* ELSP is an implicit wmb() before the GPU wraps and overwrites csb */
2592 return entry;
2593}
2594
2595static void process_csb(struct intel_engine_cs *engine)
2596{
2597 struct intel_engine_execlists * const execlists = &engine->execlists;
2598 u64 * const buf = execlists->csb_status;
2599 const u8 num_entries = execlists->csb_size;
2600 u8 head, tail;
2601
2602 /*
2603 * As we modify our execlists state tracking we require exclusive
2604 * access. Either we are inside the tasklet, or the tasklet is disabled
2605 * and we assume that is only inside the reset paths and so serialised.
2606 */
2607 GEM_BUG_ON(!tasklet_is_locked(&execlists->tasklet) &&
2608 !reset_in_progress(execlists));
2609 GEM_BUG_ON(!intel_engine_in_execlists_submission_mode(engine));
2610
2611 /*
2612 * Note that csb_write, csb_status may be either in HWSP or mmio.
2613 * When reading from the csb_write mmio register, we have to be
2614 * careful to only use the GEN8_CSB_WRITE_PTR portion, which is
2615 * the low 4bits. As it happens we know the next 4bits are always
2616 * zero and so we can simply masked off the low u8 of the register
2617 * and treat it identically to reading from the HWSP (without having
2618 * to use explicit shifting and masking, and probably bifurcating
2619 * the code to handle the legacy mmio read).
2620 */
2621 head = execlists->csb_head;
2622 tail = READ_ONCE(*execlists->csb_write);
2623 if (unlikely(head == tail))
2624 return;
2625
2626 /*
2627 * We will consume all events from HW, or at least pretend to.
2628 *
2629 * The sequence of events from the HW is deterministic, and derived
2630 * from our writes to the ELSP, with a smidgen of variability for
2631 * the arrival of the asynchronous requests wrt to the inflight
2632 * execution. If the HW sends an event that does not correspond with
2633 * the one we are expecting, we have to abandon all hope as we lose
2634 * all tracking of what the engine is actually executing. We will
2635 * only detect we are out of sequence with the HW when we get an
2636 * 'impossible' event because we have already drained our own
2637 * preemption/promotion queue. If this occurs, we know that we likely
2638 * lost track of execution earlier and must unwind and restart, the
2639 * simplest way is by stop processing the event queue and force the
2640 * engine to reset.
2641 */
2642 execlists->csb_head = tail;
2643 ENGINE_TRACE(engine, "cs-irq head=%d, tail=%d\n", head, tail);
2644
2645 /*
2646 * Hopefully paired with a wmb() in HW!
2647 *
2648 * We must complete the read of the write pointer before any reads
2649 * from the CSB, so that we do not see stale values. Without an rmb
2650 * (lfence) the HW may speculatively perform the CSB[] reads *before*
2651 * we perform the READ_ONCE(*csb_write).
2652 */
2653 rmb();
2654 do {
2655 bool promote;
2656 u64 csb;
2657
2658 if (++head == num_entries)
2659 head = 0;
2660
2661 /*
2662 * We are flying near dragons again.
2663 *
2664 * We hold a reference to the request in execlist_port[]
2665 * but no more than that. We are operating in softirq
2666 * context and so cannot hold any mutex or sleep. That
2667 * prevents us stopping the requests we are processing
2668 * in port[] from being retired simultaneously (the
2669 * breadcrumb will be complete before we see the
2670 * context-switch). As we only hold the reference to the
2671 * request, any pointer chasing underneath the request
2672 * is subject to a potential use-after-free. Thus we
2673 * store all of the bookkeeping within port[] as
2674 * required, and avoid using unguarded pointers beneath
2675 * request itself. The same applies to the atomic
2676 * status notifier.
2677 */
2678
2679 csb = csb_read(engine, buf + head);
2680 ENGINE_TRACE(engine, "csb[%d]: status=0x%08x:0x%08x\n",
2681 head, upper_32_bits(csb), lower_32_bits(csb));
2682
2683 if (INTEL_GEN(engine->i915) >= 12)
2684 promote = gen12_csb_parse(csb);
2685 else
2686 promote = gen8_csb_parse(csb);
2687 if (promote) {
2688 struct i915_request * const *old = execlists->active;
2689
2690 if (GEM_WARN_ON(!*execlists->pending)) {
2691 execlists->error_interrupt |= ERROR_CSB;
2692 break;
2693 }
2694
2695 ring_set_paused(engine, 0);
2696
2697 /* Point active to the new ELSP; prevent overwriting */
2698 WRITE_ONCE(execlists->active, execlists->pending);
2699 smp_wmb(); /* notify execlists_active() */
2700
2701 /* cancel old inflight, prepare for switch */
2702 trace_ports(execlists, "preempted", old);
2703 while (*old)
2704 execlists_schedule_out(*old++);
2705
2706 /* switch pending to inflight */
2707 GEM_BUG_ON(!assert_pending_valid(execlists, "promote"));
2708 copy_ports(execlists->inflight,
2709 execlists->pending,
2710 execlists_num_ports(execlists));
2711 smp_wmb(); /* complete the seqlock */
2712 WRITE_ONCE(execlists->active, execlists->inflight);
2713
2714 /* XXX Magic delay for tgl */
2715 ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
2716
2717 WRITE_ONCE(execlists->pending[0], NULL);
2718 } else {
2719 if (GEM_WARN_ON(!*execlists->active)) {
2720 execlists->error_interrupt |= ERROR_CSB;
2721 break;
2722 }
2723
2724 /* port0 completed, advanced to port1 */
2725 trace_ports(execlists, "completed", execlists->active);
2726
2727 /*
2728 * We rely on the hardware being strongly
2729 * ordered, that the breadcrumb write is
2730 * coherent (visible from the CPU) before the
2731 * user interrupt is processed. One might assume
2732 * that the breadcrumb write being before the
2733 * user interrupt and the CS event for the context
2734 * switch would therefore be before the CS event
2735 * itself...
2736 */
2737 if (GEM_SHOW_DEBUG() &&
2738 !i915_request_completed(*execlists->active)) {
2739 struct i915_request *rq = *execlists->active;
2740 const u32 *regs __maybe_unused =
2741 rq->context->lrc_reg_state;
2742
2743 ENGINE_TRACE(engine,
2744 "context completed before request!\n");
2745 ENGINE_TRACE(engine,
2746 "ring:{start:0x%08x, head:%04x, tail:%04x, ctl:%08x, mode:%08x}\n",
2747 ENGINE_READ(engine, RING_START),
2748 ENGINE_READ(engine, RING_HEAD) & HEAD_ADDR,
2749 ENGINE_READ(engine, RING_TAIL) & TAIL_ADDR,
2750 ENGINE_READ(engine, RING_CTL),
2751 ENGINE_READ(engine, RING_MI_MODE));
2752 ENGINE_TRACE(engine,
2753 "rq:{start:%08x, head:%04x, tail:%04x, seqno:%llx:%d, hwsp:%d}, ",
2754 i915_ggtt_offset(rq->ring->vma),
2755 rq->head, rq->tail,
2756 rq->fence.context,
2757 lower_32_bits(rq->fence.seqno),
2758 hwsp_seqno(rq));
2759 ENGINE_TRACE(engine,
2760 "ctx:{start:%08x, head:%04x, tail:%04x}, ",
2761 regs[CTX_RING_START],
2762 regs[CTX_RING_HEAD],
2763 regs[CTX_RING_TAIL]);
2764 }
2765
2766 execlists_schedule_out(*execlists->active++);
2767
2768 GEM_BUG_ON(execlists->active - execlists->inflight >
2769 execlists_num_ports(execlists));
2770 }
2771 } while (head != tail);
2772
2773 set_timeslice(engine);
2774
2775 /*
2776 * Gen11 has proven to fail wrt global observation point between
2777 * entry and tail update, failing on the ordering and thus
2778 * we see an old entry in the context status buffer.
2779 *
2780 * Forcibly evict out entries for the next gpu csb update,
2781 * to increase the odds that we get a fresh entries with non
2782 * working hardware. The cost for doing so comes out mostly with
2783 * the wash as hardware, working or not, will need to do the
2784 * invalidation before.
2785 */
2786 invalidate_csb_entries(&buf[0], &buf[num_entries - 1]);
2787}
2788
2789static void __execlists_submission_tasklet(struct intel_engine_cs *const engine)
2790{
2791 lockdep_assert_held(&engine->active.lock);
2792 if (!READ_ONCE(engine->execlists.pending[0])) {
2793 rcu_read_lock(); /* protect peeking at execlists->active */
2794 execlists_dequeue(engine);
2795 rcu_read_unlock();
2796 }
2797}
2798
2799static void __execlists_hold(struct i915_request *rq)
2800{
2801 LIST_HEAD(list);
2802
2803 do {
2804 struct i915_dependency *p;
2805
2806 if (i915_request_is_active(rq))
2807 __i915_request_unsubmit(rq);
2808
2809 clear_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2810 list_move_tail(&rq->sched.link, &rq->engine->active.hold);
2811 i915_request_set_hold(rq);
2812 RQ_TRACE(rq, "on hold\n");
2813
2814 for_each_waiter(p, rq) {
2815 struct i915_request *w =
2816 container_of(p->waiter, typeof(*w), sched);
2817
2818 /* Leave semaphores spinning on the other engines */
2819 if (w->engine != rq->engine)
2820 continue;
2821
2822 if (!i915_request_is_ready(w))
2823 continue;
2824
2825 if (i915_request_completed(w))
2826 continue;
2827
2828 if (i915_request_on_hold(w))
2829 continue;
2830
2831 list_move_tail(&w->sched.link, &list);
2832 }
2833
2834 rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2835 } while (rq);
2836}
2837
2838static bool execlists_hold(struct intel_engine_cs *engine,
2839 struct i915_request *rq)
2840{
2841 if (i915_request_on_hold(rq))
2842 return false;
2843
2844 spin_lock_irq(&engine->active.lock);
2845
2846 if (i915_request_completed(rq)) { /* too late! */
2847 rq = NULL;
2848 goto unlock;
2849 }
2850
2851 if (rq->engine != engine) { /* preempted virtual engine */
2852 struct virtual_engine *ve = to_virtual_engine(rq->engine);
2853
2854 /*
2855 * intel_context_inflight() is only protected by virtue
2856 * of process_csb() being called only by the tasklet (or
2857 * directly from inside reset while the tasklet is suspended).
2858 * Assert that neither of those are allowed to run while we
2859 * poke at the request queues.
2860 */
2861 GEM_BUG_ON(!reset_in_progress(&engine->execlists));
2862
2863 /*
2864 * An unsubmitted request along a virtual engine will
2865 * remain on the active (this) engine until we are able
2866 * to process the context switch away (and so mark the
2867 * context as no longer in flight). That cannot have happened
2868 * yet, otherwise we would not be hanging!
2869 */
2870 spin_lock(&ve->base.active.lock);
2871 GEM_BUG_ON(intel_context_inflight(rq->context) != engine);
2872 GEM_BUG_ON(ve->request != rq);
2873 ve->request = NULL;
2874 spin_unlock(&ve->base.active.lock);
2875 i915_request_put(rq);
2876
2877 rq->engine = engine;
2878 }
2879
2880 /*
2881 * Transfer this request onto the hold queue to prevent it
2882 * being resumbitted to HW (and potentially completed) before we have
2883 * released it. Since we may have already submitted following
2884 * requests, we need to remove those as well.
2885 */
2886 GEM_BUG_ON(i915_request_on_hold(rq));
2887 GEM_BUG_ON(rq->engine != engine);
2888 __execlists_hold(rq);
2889 GEM_BUG_ON(list_empty(&engine->active.hold));
2890
2891unlock:
2892 spin_unlock_irq(&engine->active.lock);
2893 return rq;
2894}
2895
2896static bool hold_request(const struct i915_request *rq)
2897{
2898 struct i915_dependency *p;
2899 bool result = false;
2900
2901 /*
2902 * If one of our ancestors is on hold, we must also be on hold,
2903 * otherwise we will bypass it and execute before it.
2904 */
2905 rcu_read_lock();
2906 for_each_signaler(p, rq) {
2907 const struct i915_request *s =
2908 container_of(p->signaler, typeof(*s), sched);
2909
2910 if (s->engine != rq->engine)
2911 continue;
2912
2913 result = i915_request_on_hold(s);
2914 if (result)
2915 break;
2916 }
2917 rcu_read_unlock();
2918
2919 return result;
2920}
2921
2922static void __execlists_unhold(struct i915_request *rq)
2923{
2924 LIST_HEAD(list);
2925
2926 do {
2927 struct i915_dependency *p;
2928
2929 RQ_TRACE(rq, "hold release\n");
2930
2931 GEM_BUG_ON(!i915_request_on_hold(rq));
2932 GEM_BUG_ON(!i915_sw_fence_signaled(&rq->submit));
2933
2934 i915_request_clear_hold(rq);
2935 list_move_tail(&rq->sched.link,
2936 i915_sched_lookup_priolist(rq->engine,
2937 rq_prio(rq)));
2938 set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
2939
2940 /* Also release any children on this engine that are ready */
2941 for_each_waiter(p, rq) {
2942 struct i915_request *w =
2943 container_of(p->waiter, typeof(*w), sched);
2944
2945 /* Propagate any change in error status */
2946 if (rq->fence.error)
2947 i915_request_set_error_once(w, rq->fence.error);
2948
2949 if (w->engine != rq->engine)
2950 continue;
2951
2952 if (!i915_request_on_hold(w))
2953 continue;
2954
2955 /* Check that no other parents are also on hold */
2956 if (hold_request(w))
2957 continue;
2958
2959 list_move_tail(&w->sched.link, &list);
2960 }
2961
2962 rq = list_first_entry_or_null(&list, typeof(*rq), sched.link);
2963 } while (rq);
2964}
2965
2966static void execlists_unhold(struct intel_engine_cs *engine,
2967 struct i915_request *rq)
2968{
2969 spin_lock_irq(&engine->active.lock);
2970
2971 /*
2972 * Move this request back to the priority queue, and all of its
2973 * children and grandchildren that were suspended along with it.
2974 */
2975 __execlists_unhold(rq);
2976
2977 if (rq_prio(rq) > engine->execlists.queue_priority_hint) {
2978 engine->execlists.queue_priority_hint = rq_prio(rq);
2979 tasklet_hi_schedule(&engine->execlists.tasklet);
2980 }
2981
2982 spin_unlock_irq(&engine->active.lock);
2983}
2984
2985struct execlists_capture {
2986 struct work_struct work;
2987 struct i915_request *rq;
2988 struct i915_gpu_coredump *error;
2989};
2990
2991static void execlists_capture_work(struct work_struct *work)
2992{
2993 struct execlists_capture *cap = container_of(work, typeof(*cap), work);
2994 const gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN;
2995 struct intel_engine_cs *engine = cap->rq->engine;
2996 struct intel_gt_coredump *gt = cap->error->gt;
2997 struct intel_engine_capture_vma *vma;
2998
2999 /* Compress all the objects attached to the request, slow! */
3000 vma = intel_engine_coredump_add_request(gt->engine, cap->rq, gfp);
3001 if (vma) {
3002 struct i915_vma_compress *compress =
3003 i915_vma_capture_prepare(gt);
3004
3005 intel_engine_coredump_add_vma(gt->engine, vma, compress);
3006 i915_vma_capture_finish(gt, compress);
3007 }
3008
3009 gt->simulated = gt->engine->simulated;
3010 cap->error->simulated = gt->simulated;
3011
3012 /* Publish the error state, and announce it to the world */
3013 i915_error_state_store(cap->error);
3014 i915_gpu_coredump_put(cap->error);
3015
3016 /* Return this request and all that depend upon it for signaling */
3017 execlists_unhold(engine, cap->rq);
3018 i915_request_put(cap->rq);
3019
3020 kfree(cap);
3021}
3022
3023static struct execlists_capture *capture_regs(struct intel_engine_cs *engine)
3024{
3025 const gfp_t gfp = GFP_ATOMIC | __GFP_NOWARN;
3026 struct execlists_capture *cap;
3027
3028 cap = kmalloc(sizeof(*cap), gfp);
3029 if (!cap)
3030 return NULL;
3031
3032 cap->error = i915_gpu_coredump_alloc(engine->i915, gfp);
3033 if (!cap->error)
3034 goto err_cap;
3035
3036 cap->error->gt = intel_gt_coredump_alloc(engine->gt, gfp);
3037 if (!cap->error->gt)
3038 goto err_gpu;
3039
3040 cap->error->gt->engine = intel_engine_coredump_alloc(engine, gfp);
3041 if (!cap->error->gt->engine)
3042 goto err_gt;
3043
3044 cap->error->gt->engine->hung = true;
3045
3046 return cap;
3047
3048err_gt:
3049 kfree(cap->error->gt);
3050err_gpu:
3051 kfree(cap->error);
3052err_cap:
3053 kfree(cap);
3054 return NULL;
3055}
3056
3057static struct i915_request *
3058active_context(struct intel_engine_cs *engine, u32 ccid)
3059{
3060 const struct intel_engine_execlists * const el = &engine->execlists;
3061 struct i915_request * const *port, *rq;
3062
3063 /*
3064 * Use the most recent result from process_csb(), but just in case
3065 * we trigger an error (via interrupt) before the first CS event has
3066 * been written, peek at the next submission.
3067 */
3068
3069 for (port = el->active; (rq = *port); port++) {
3070 if (rq->context->lrc.ccid == ccid) {
3071 ENGINE_TRACE(engine,
3072 "ccid found at active:%zd\n",
3073 port - el->active);
3074 return rq;
3075 }
3076 }
3077
3078 for (port = el->pending; (rq = *port); port++) {
3079 if (rq->context->lrc.ccid == ccid) {
3080 ENGINE_TRACE(engine,
3081 "ccid found at pending:%zd\n",
3082 port - el->pending);
3083 return rq;
3084 }
3085 }
3086
3087 ENGINE_TRACE(engine, "ccid:%x not found\n", ccid);
3088 return NULL;
3089}
3090
3091static u32 active_ccid(struct intel_engine_cs *engine)
3092{
3093 return ENGINE_READ_FW(engine, RING_EXECLIST_STATUS_HI);
3094}
3095
3096static void execlists_capture(struct intel_engine_cs *engine)
3097{
3098 struct execlists_capture *cap;
3099
3100 if (!IS_ENABLED(CONFIG_DRM_I915_CAPTURE_ERROR))
3101 return;
3102
3103 /*
3104 * We need to _quickly_ capture the engine state before we reset.
3105 * We are inside an atomic section (softirq) here and we are delaying
3106 * the forced preemption event.
3107 */
3108 cap = capture_regs(engine);
3109 if (!cap)
3110 return;
3111
3112 spin_lock_irq(&engine->active.lock);
3113 cap->rq = active_context(engine, active_ccid(engine));
3114 if (cap->rq) {
3115 cap->rq = active_request(cap->rq->context->timeline, cap->rq);
3116 cap->rq = i915_request_get_rcu(cap->rq);
3117 }
3118 spin_unlock_irq(&engine->active.lock);
3119 if (!cap->rq)
3120 goto err_free;
3121
3122 /*
3123 * Remove the request from the execlists queue, and take ownership
3124 * of the request. We pass it to our worker who will _slowly_ compress
3125 * all the pages the _user_ requested for debugging their batch, after
3126 * which we return it to the queue for signaling.
3127 *
3128 * By removing them from the execlists queue, we also remove the
3129 * requests from being processed by __unwind_incomplete_requests()
3130 * during the intel_engine_reset(), and so they will *not* be replayed
3131 * afterwards.
3132 *
3133 * Note that because we have not yet reset the engine at this point,
3134 * it is possible for the request that we have identified as being
3135 * guilty, did in fact complete and we will then hit an arbitration
3136 * point allowing the outstanding preemption to succeed. The likelihood
3137 * of that is very low (as capturing of the engine registers should be
3138 * fast enough to run inside an irq-off atomic section!), so we will
3139 * simply hold that request accountable for being non-preemptible
3140 * long enough to force the reset.
3141 */
3142 if (!execlists_hold(engine, cap->rq))
3143 goto err_rq;
3144
3145 INIT_WORK(&cap->work, execlists_capture_work);
3146 schedule_work(&cap->work);
3147 return;
3148
3149err_rq:
3150 i915_request_put(cap->rq);
3151err_free:
3152 i915_gpu_coredump_put(cap->error);
3153 kfree(cap);
3154}
3155
3156static void execlists_reset(struct intel_engine_cs *engine, const char *msg)
3157{
3158 const unsigned int bit = I915_RESET_ENGINE + engine->id;
3159 unsigned long *lock = &engine->gt->reset.flags;
3160
3161 if (!intel_has_reset_engine(engine->gt))
3162 return;
3163
3164 if (test_and_set_bit(bit, lock))
3165 return;
3166
3167 ENGINE_TRACE(engine, "reset for %s\n", msg);
3168
3169 /* Mark this tasklet as disabled to avoid waiting for it to complete */
3170 tasklet_disable_nosync(&engine->execlists.tasklet);
3171
3172 ring_set_paused(engine, 1); /* Freeze the current request in place */
3173 execlists_capture(engine);
3174 intel_engine_reset(engine, msg);
3175
3176 tasklet_enable(&engine->execlists.tasklet);
3177 clear_and_wake_up_bit(bit, lock);
3178}
3179
3180static bool preempt_timeout(const struct intel_engine_cs *const engine)
3181{
3182 const struct timer_list *t = &engine->execlists.preempt;
3183
3184 if (!CONFIG_DRM_I915_PREEMPT_TIMEOUT)
3185 return false;
3186
3187 if (!timer_expired(t))
3188 return false;
3189
3190 return READ_ONCE(engine->execlists.pending[0]);
3191}
3192
3193/*
3194 * Check the unread Context Status Buffers and manage the submission of new
3195 * contexts to the ELSP accordingly.
3196 */
3197static void execlists_submission_tasklet(unsigned long data)
3198{
3199 struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
3200 bool timeout = preempt_timeout(engine);
3201
3202 process_csb(engine);
3203
3204 if (unlikely(READ_ONCE(engine->execlists.error_interrupt))) {
3205 const char *msg;
3206
3207 /* Generate the error message in priority wrt to the user! */
3208 if (engine->execlists.error_interrupt & GENMASK(15, 0))
3209 msg = "CS error"; /* thrown by a user payload */
3210 else if (engine->execlists.error_interrupt & ERROR_CSB)
3211 msg = "invalid CSB event";
3212 else
3213 msg = "internal error";
3214
3215 engine->execlists.error_interrupt = 0;
3216 execlists_reset(engine, msg);
3217 }
3218
3219 if (!READ_ONCE(engine->execlists.pending[0]) || timeout) {
3220 unsigned long flags;
3221
3222 spin_lock_irqsave(&engine->active.lock, flags);
3223 __execlists_submission_tasklet(engine);
3224 spin_unlock_irqrestore(&engine->active.lock, flags);
3225
3226 /* Recheck after serialising with direct-submission */
3227 if (unlikely(timeout && preempt_timeout(engine))) {
3228 cancel_timer(&engine->execlists.preempt);
3229 execlists_reset(engine, "preemption time out");
3230 }
3231 }
3232}
3233
3234static void __execlists_kick(struct intel_engine_execlists *execlists)
3235{
3236 /* Kick the tasklet for some interrupt coalescing and reset handling */
3237 tasklet_hi_schedule(&execlists->tasklet);
3238}
3239
3240#define execlists_kick(t, member) \
3241 __execlists_kick(container_of(t, struct intel_engine_execlists, member))
3242
3243static void execlists_timeslice(struct timer_list *timer)
3244{
3245 execlists_kick(timer, timer);
3246}
3247
3248static void execlists_preempt(struct timer_list *timer)
3249{
3250 execlists_kick(timer, preempt);
3251}
3252
3253static void queue_request(struct intel_engine_cs *engine,
3254 struct i915_request *rq)
3255{
3256 GEM_BUG_ON(!list_empty(&rq->sched.link));
3257 list_add_tail(&rq->sched.link,
3258 i915_sched_lookup_priolist(engine, rq_prio(rq)));
3259 set_bit(I915_FENCE_FLAG_PQUEUE, &rq->fence.flags);
3260}
3261
3262static void __submit_queue_imm(struct intel_engine_cs *engine)
3263{
3264 struct intel_engine_execlists * const execlists = &engine->execlists;
3265
3266 if (reset_in_progress(execlists))
3267 return; /* defer until we restart the engine following reset */
3268
3269 __execlists_submission_tasklet(engine);
3270}
3271
3272static void submit_queue(struct intel_engine_cs *engine,
3273 const struct i915_request *rq)
3274{
3275 struct intel_engine_execlists *execlists = &engine->execlists;
3276
3277 if (rq_prio(rq) <= execlists->queue_priority_hint)
3278 return;
3279
3280 execlists->queue_priority_hint = rq_prio(rq);
3281 __submit_queue_imm(engine);
3282}
3283
3284static bool ancestor_on_hold(const struct intel_engine_cs *engine,
3285 const struct i915_request *rq)
3286{
3287 GEM_BUG_ON(i915_request_on_hold(rq));
3288 return !list_empty(&engine->active.hold) && hold_request(rq);
3289}
3290
3291static void flush_csb(struct intel_engine_cs *engine)
3292{
3293 struct intel_engine_execlists *el = &engine->execlists;
3294
3295 if (READ_ONCE(el->pending[0]) && tasklet_trylock(&el->tasklet)) {
3296 if (!reset_in_progress(el))
3297 process_csb(engine);
3298 tasklet_unlock(&el->tasklet);
3299 }
3300}
3301
3302static void execlists_submit_request(struct i915_request *request)
3303{
3304 struct intel_engine_cs *engine = request->engine;
3305 unsigned long flags;
3306
3307 /* Hopefully we clear execlists->pending[] to let us through */
3308 flush_csb(engine);
3309
3310 /* Will be called from irq-context when using foreign fences. */
3311 spin_lock_irqsave(&engine->active.lock, flags);
3312
3313 if (unlikely(ancestor_on_hold(engine, request))) {
3314 RQ_TRACE(request, "ancestor on hold\n");
3315 list_add_tail(&request->sched.link, &engine->active.hold);
3316 i915_request_set_hold(request);
3317 } else {
3318 queue_request(engine, request);
3319
3320 GEM_BUG_ON(RB_EMPTY_ROOT(&engine->execlists.queue.rb_root));
3321 GEM_BUG_ON(list_empty(&request->sched.link));
3322
3323 submit_queue(engine, request);
3324 }
3325
3326 spin_unlock_irqrestore(&engine->active.lock, flags);
3327}
3328
3329static void __execlists_context_fini(struct intel_context *ce)
3330{
3331 intel_ring_put(ce->ring);
3332 i915_vma_put(ce->state);
3333}
3334
3335static void execlists_context_destroy(struct kref *kref)
3336{
3337 struct intel_context *ce = container_of(kref, typeof(*ce), ref);
3338
3339 GEM_BUG_ON(!i915_active_is_idle(&ce->active));
3340 GEM_BUG_ON(intel_context_is_pinned(ce));
3341
3342 if (ce->state)
3343 __execlists_context_fini(ce);
3344
3345 intel_context_fini(ce);
3346 intel_context_free(ce);
3347}
3348
3349static void
3350set_redzone(void *vaddr, const struct intel_engine_cs *engine)
3351{
3352 if (!IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
3353 return;
3354
3355 vaddr += engine->context_size;
3356
3357 memset(vaddr, CONTEXT_REDZONE, I915_GTT_PAGE_SIZE);
3358}
3359
3360static void
3361check_redzone(const void *vaddr, const struct intel_engine_cs *engine)
3362{
3363 if (!IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
3364 return;
3365
3366 vaddr += engine->context_size;
3367
3368 if (memchr_inv(vaddr, CONTEXT_REDZONE, I915_GTT_PAGE_SIZE))
3369 drm_err_once(&engine->i915->drm,
3370 "%s context redzone overwritten!\n",
3371 engine->name);
3372}
3373
3374static void execlists_context_unpin(struct intel_context *ce)
3375{
3376 check_redzone((void *)ce->lrc_reg_state - LRC_STATE_OFFSET,
3377 ce->engine);
3378}
3379
3380static void execlists_context_post_unpin(struct intel_context *ce)
3381{
3382 i915_gem_object_unpin_map(ce->state->obj);
3383}
3384
3385static u32 *
3386gen12_emit_timestamp_wa(const struct intel_context *ce, u32 *cs)
3387{
3388 *cs++ = MI_LOAD_REGISTER_MEM_GEN8 |
3389 MI_SRM_LRM_GLOBAL_GTT |
3390 MI_LRI_LRM_CS_MMIO;
3391 *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3392 *cs++ = i915_ggtt_offset(ce->state) + LRC_STATE_OFFSET +
3393 CTX_TIMESTAMP * sizeof(u32);
3394 *cs++ = 0;
3395
3396 *cs++ = MI_LOAD_REGISTER_REG |
3397 MI_LRR_SOURCE_CS_MMIO |
3398 MI_LRI_LRM_CS_MMIO;
3399 *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3400 *cs++ = i915_mmio_reg_offset(RING_CTX_TIMESTAMP(0));
3401
3402 *cs++ = MI_LOAD_REGISTER_REG |
3403 MI_LRR_SOURCE_CS_MMIO |
3404 MI_LRI_LRM_CS_MMIO;
3405 *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3406 *cs++ = i915_mmio_reg_offset(RING_CTX_TIMESTAMP(0));
3407
3408 return cs;
3409}
3410
3411static u32 *
3412gen12_emit_restore_scratch(const struct intel_context *ce, u32 *cs)
3413{
3414 GEM_BUG_ON(lrc_ring_gpr0(ce->engine) == -1);
3415
3416 *cs++ = MI_LOAD_REGISTER_MEM_GEN8 |
3417 MI_SRM_LRM_GLOBAL_GTT |
3418 MI_LRI_LRM_CS_MMIO;
3419 *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3420 *cs++ = i915_ggtt_offset(ce->state) + LRC_STATE_OFFSET +
3421 (lrc_ring_gpr0(ce->engine) + 1) * sizeof(u32);
3422 *cs++ = 0;
3423
3424 return cs;
3425}
3426
3427static u32 *
3428gen12_emit_cmd_buf_wa(const struct intel_context *ce, u32 *cs)
3429{
3430 GEM_BUG_ON(lrc_ring_cmd_buf_cctl(ce->engine) == -1);
3431
3432 *cs++ = MI_LOAD_REGISTER_MEM_GEN8 |
3433 MI_SRM_LRM_GLOBAL_GTT |
3434 MI_LRI_LRM_CS_MMIO;
3435 *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3436 *cs++ = i915_ggtt_offset(ce->state) + LRC_STATE_OFFSET +
3437 (lrc_ring_cmd_buf_cctl(ce->engine) + 1) * sizeof(u32);
3438 *cs++ = 0;
3439
3440 *cs++ = MI_LOAD_REGISTER_REG |
3441 MI_LRR_SOURCE_CS_MMIO |
3442 MI_LRI_LRM_CS_MMIO;
3443 *cs++ = i915_mmio_reg_offset(GEN8_RING_CS_GPR(0, 0));
3444 *cs++ = i915_mmio_reg_offset(RING_CMD_BUF_CCTL(0));
3445
3446 return cs;
3447}
3448
3449static u32 *
3450gen12_emit_indirect_ctx_rcs(const struct intel_context *ce, u32 *cs)
3451{
3452 cs = gen12_emit_timestamp_wa(ce, cs);
3453 cs = gen12_emit_cmd_buf_wa(ce, cs);
3454 cs = gen12_emit_restore_scratch(ce, cs);
3455
3456 return cs;
3457}
3458
3459static u32 *
3460gen12_emit_indirect_ctx_xcs(const struct intel_context *ce, u32 *cs)
3461{
3462 cs = gen12_emit_timestamp_wa(ce, cs);
3463 cs = gen12_emit_restore_scratch(ce, cs);
3464
3465 return cs;
3466}
3467
3468static inline u32 context_wa_bb_offset(const struct intel_context *ce)
3469{
3470 return PAGE_SIZE * ce->wa_bb_page;
3471}
3472
3473static u32 *context_indirect_bb(const struct intel_context *ce)
3474{
3475 void *ptr;
3476
3477 GEM_BUG_ON(!ce->wa_bb_page);
3478
3479 ptr = ce->lrc_reg_state;
3480 ptr -= LRC_STATE_OFFSET; /* back to start of context image */
3481 ptr += context_wa_bb_offset(ce);
3482
3483 return ptr;
3484}
3485
3486static void
3487setup_indirect_ctx_bb(const struct intel_context *ce,
3488 const struct intel_engine_cs *engine,
3489 u32 *(*emit)(const struct intel_context *, u32 *))
3490{
3491 u32 * const start = context_indirect_bb(ce);
3492 u32 *cs;
3493
3494 cs = emit(ce, start);
3495 GEM_BUG_ON(cs - start > I915_GTT_PAGE_SIZE / sizeof(*cs));
3496 while ((unsigned long)cs % CACHELINE_BYTES)
3497 *cs++ = MI_NOOP;
3498
3499 lrc_ring_setup_indirect_ctx(ce->lrc_reg_state, engine,
3500 i915_ggtt_offset(ce->state) +
3501 context_wa_bb_offset(ce),
3502 (cs - start) * sizeof(*cs));
3503}
3504
3505static void
3506__execlists_update_reg_state(const struct intel_context *ce,
3507 const struct intel_engine_cs *engine,
3508 u32 head)
3509{
3510 struct intel_ring *ring = ce->ring;
3511 u32 *regs = ce->lrc_reg_state;
3512
3513 GEM_BUG_ON(!intel_ring_offset_valid(ring, head));
3514 GEM_BUG_ON(!intel_ring_offset_valid(ring, ring->tail));
3515
3516 regs[CTX_RING_START] = i915_ggtt_offset(ring->vma);
3517 regs[CTX_RING_HEAD] = head;
3518 regs[CTX_RING_TAIL] = ring->tail;
3519 regs[CTX_RING_CTL] = RING_CTL_SIZE(ring->size) | RING_VALID;
3520
3521 /* RPCS */
3522 if (engine->class == RENDER_CLASS) {
3523 regs[CTX_R_PWR_CLK_STATE] =
3524 intel_sseu_make_rpcs(engine->gt, &ce->sseu);
3525
3526 i915_oa_init_reg_state(ce, engine);
3527 }
3528
3529 if (ce->wa_bb_page) {
3530 u32 *(*fn)(const struct intel_context *ce, u32 *cs);
3531
3532 fn = gen12_emit_indirect_ctx_xcs;
3533 if (ce->engine->class == RENDER_CLASS)
3534 fn = gen12_emit_indirect_ctx_rcs;
3535
3536 /* Mutually exclusive wrt to global indirect bb */
3537 GEM_BUG_ON(engine->wa_ctx.indirect_ctx.size);
3538 setup_indirect_ctx_bb(ce, engine, fn);
3539 }
3540}
3541
3542static int
3543execlists_context_pre_pin(struct intel_context *ce,
3544 struct i915_gem_ww_ctx *ww, void **vaddr)
3545{
3546 GEM_BUG_ON(!ce->state);
3547 GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
3548
3549 *vaddr = i915_gem_object_pin_map(ce->state->obj,
3550 i915_coherent_map_type(ce->engine->i915) |
3551 I915_MAP_OVERRIDE);
3552
3553 return PTR_ERR_OR_ZERO(*vaddr);
3554}
3555
3556static int
3557__execlists_context_pin(struct intel_context *ce,
3558 struct intel_engine_cs *engine,
3559 void *vaddr)
3560{
3561 ce->lrc.lrca = lrc_descriptor(ce, engine) | CTX_DESC_FORCE_RESTORE;
3562 ce->lrc_reg_state = vaddr + LRC_STATE_OFFSET;
3563 __execlists_update_reg_state(ce, engine, ce->ring->tail);
3564
3565 return 0;
3566}
3567
3568static int execlists_context_pin(struct intel_context *ce, void *vaddr)
3569{
3570 return __execlists_context_pin(ce, ce->engine, vaddr);
3571}
3572
3573static int execlists_context_alloc(struct intel_context *ce)
3574{
3575 return __execlists_context_alloc(ce, ce->engine);
3576}
3577
3578static void execlists_context_reset(struct intel_context *ce)
3579{
3580 CE_TRACE(ce, "reset\n");
3581 GEM_BUG_ON(!intel_context_is_pinned(ce));
3582
3583 intel_ring_reset(ce->ring, ce->ring->emit);
3584
3585 /* Scrub away the garbage */
3586 execlists_init_reg_state(ce->lrc_reg_state,
3587 ce, ce->engine, ce->ring, true);
3588 __execlists_update_reg_state(ce, ce->engine, ce->ring->tail);
3589
3590 ce->lrc.desc |= CTX_DESC_FORCE_RESTORE;
3591}
3592
3593static const struct intel_context_ops execlists_context_ops = {
3594 .alloc = execlists_context_alloc,
3595
3596 .pre_pin = execlists_context_pre_pin,
3597 .pin = execlists_context_pin,
3598 .unpin = execlists_context_unpin,
3599 .post_unpin = execlists_context_post_unpin,
3600
3601 .enter = intel_context_enter_engine,
3602 .exit = intel_context_exit_engine,
3603
3604 .reset = execlists_context_reset,
3605 .destroy = execlists_context_destroy,
3606};
3607
3608static u32 hwsp_offset(const struct i915_request *rq)
3609{
3610 const struct intel_timeline_cacheline *cl;
3611
3612 /* Before the request is executed, the timeline/cachline is fixed */
3613
3614 cl = rcu_dereference_protected(rq->hwsp_cacheline, 1);
3615 if (cl)
3616 return cl->ggtt_offset;
3617
3618 return rcu_dereference_protected(rq->timeline, 1)->hwsp_offset;
3619}
3620
3621static int gen8_emit_init_breadcrumb(struct i915_request *rq)
3622{
3623 u32 *cs;
3624
3625 GEM_BUG_ON(i915_request_has_initial_breadcrumb(rq));
3626 if (!i915_request_timeline(rq)->has_initial_breadcrumb)
3627 return 0;
3628
3629 cs = intel_ring_begin(rq, 6);
3630 if (IS_ERR(cs))
3631 return PTR_ERR(cs);
3632
3633 /*
3634 * Check if we have been preempted before we even get started.
3635 *
3636 * After this point i915_request_started() reports true, even if
3637 * we get preempted and so are no longer running.
3638 */
3639 *cs++ = MI_ARB_CHECK;
3640 *cs++ = MI_NOOP;
3641
3642 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
3643 *cs++ = hwsp_offset(rq);
3644 *cs++ = 0;
3645 *cs++ = rq->fence.seqno - 1;
3646
3647 intel_ring_advance(rq, cs);
3648
3649 /* Record the updated position of the request's payload */
3650 rq->infix = intel_ring_offset(rq, cs);
3651
3652 __set_bit(I915_FENCE_FLAG_INITIAL_BREADCRUMB, &rq->fence.flags);
3653
3654 return 0;
3655}
3656
3657static int emit_pdps(struct i915_request *rq)
3658{
3659 const struct intel_engine_cs * const engine = rq->engine;
3660 struct i915_ppgtt * const ppgtt = i915_vm_to_ppgtt(rq->context->vm);
3661 int err, i;
3662 u32 *cs;
3663
3664 GEM_BUG_ON(intel_vgpu_active(rq->engine->i915));
3665
3666 /*
3667 * Beware ye of the dragons, this sequence is magic!
3668 *
3669 * Small changes to this sequence can cause anything from
3670 * GPU hangs to forcewake errors and machine lockups!
3671 */
3672
3673 /* Flush any residual operations from the context load */
3674 err = engine->emit_flush(rq, EMIT_FLUSH);
3675 if (err)
3676 return err;
3677
3678 /* Magic required to prevent forcewake errors! */
3679 err = engine->emit_flush(rq, EMIT_INVALIDATE);
3680 if (err)
3681 return err;
3682
3683 cs = intel_ring_begin(rq, 4 * GEN8_3LVL_PDPES + 2);
3684 if (IS_ERR(cs))
3685 return PTR_ERR(cs);
3686
3687 /* Ensure the LRI have landed before we invalidate & continue */
3688 *cs++ = MI_LOAD_REGISTER_IMM(2 * GEN8_3LVL_PDPES) | MI_LRI_FORCE_POSTED;
3689 for (i = GEN8_3LVL_PDPES; i--; ) {
3690 const dma_addr_t pd_daddr = i915_page_dir_dma_addr(ppgtt, i);
3691 u32 base = engine->mmio_base;
3692
3693 *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_UDW(base, i));
3694 *cs++ = upper_32_bits(pd_daddr);
3695 *cs++ = i915_mmio_reg_offset(GEN8_RING_PDP_LDW(base, i));
3696 *cs++ = lower_32_bits(pd_daddr);
3697 }
3698 *cs++ = MI_NOOP;
3699
3700 intel_ring_advance(rq, cs);
3701
3702 return 0;
3703}
3704
3705static int execlists_request_alloc(struct i915_request *request)
3706{
3707 int ret;
3708
3709 GEM_BUG_ON(!intel_context_is_pinned(request->context));
3710
3711 /*
3712 * Flush enough space to reduce the likelihood of waiting after
3713 * we start building the request - in which case we will just
3714 * have to repeat work.
3715 */
3716 request->reserved_space += EXECLISTS_REQUEST_SIZE;
3717
3718 /*
3719 * Note that after this point, we have committed to using
3720 * this request as it is being used to both track the
3721 * state of engine initialisation and liveness of the
3722 * golden renderstate above. Think twice before you try
3723 * to cancel/unwind this request now.
3724 */
3725
3726 if (!i915_vm_is_4lvl(request->context->vm)) {
3727 ret = emit_pdps(request);
3728 if (ret)
3729 return ret;
3730 }
3731
3732 /* Unconditionally invalidate GPU caches and TLBs. */
3733 ret = request->engine->emit_flush(request, EMIT_INVALIDATE);
3734 if (ret)
3735 return ret;
3736
3737 request->reserved_space -= EXECLISTS_REQUEST_SIZE;
3738 return 0;
3739}
3740
3741/*
3742 * In this WA we need to set GEN8_L3SQCREG4[21:21] and reset it after
3743 * PIPE_CONTROL instruction. This is required for the flush to happen correctly
3744 * but there is a slight complication as this is applied in WA batch where the
3745 * values are only initialized once so we cannot take register value at the
3746 * beginning and reuse it further; hence we save its value to memory, upload a
3747 * constant value with bit21 set and then we restore it back with the saved value.
3748 * To simplify the WA, a constant value is formed by using the default value
3749 * of this register. This shouldn't be a problem because we are only modifying
3750 * it for a short period and this batch in non-premptible. We can ofcourse
3751 * use additional instructions that read the actual value of the register
3752 * at that time and set our bit of interest but it makes the WA complicated.
3753 *
3754 * This WA is also required for Gen9 so extracting as a function avoids
3755 * code duplication.
3756 */
3757static u32 *
3758gen8_emit_flush_coherentl3_wa(struct intel_engine_cs *engine, u32 *batch)
3759{
3760 /* NB no one else is allowed to scribble over scratch + 256! */
3761 *batch++ = MI_STORE_REGISTER_MEM_GEN8 | MI_SRM_LRM_GLOBAL_GTT;
3762 *batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
3763 *batch++ = intel_gt_scratch_offset(engine->gt,
3764 INTEL_GT_SCRATCH_FIELD_COHERENTL3_WA);
3765 *batch++ = 0;
3766
3767 *batch++ = MI_LOAD_REGISTER_IMM(1);
3768 *batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
3769 *batch++ = 0x40400000 | GEN8_LQSC_FLUSH_COHERENT_LINES;
3770
3771 batch = gen8_emit_pipe_control(batch,
3772 PIPE_CONTROL_CS_STALL |
3773 PIPE_CONTROL_DC_FLUSH_ENABLE,
3774 0);
3775
3776 *batch++ = MI_LOAD_REGISTER_MEM_GEN8 | MI_SRM_LRM_GLOBAL_GTT;
3777 *batch++ = i915_mmio_reg_offset(GEN8_L3SQCREG4);
3778 *batch++ = intel_gt_scratch_offset(engine->gt,
3779 INTEL_GT_SCRATCH_FIELD_COHERENTL3_WA);
3780 *batch++ = 0;
3781
3782 return batch;
3783}
3784
3785/*
3786 * Typically we only have one indirect_ctx and per_ctx batch buffer which are
3787 * initialized at the beginning and shared across all contexts but this field
3788 * helps us to have multiple batches at different offsets and select them based
3789 * on a criteria. At the moment this batch always start at the beginning of the page
3790 * and at this point we don't have multiple wa_ctx batch buffers.
3791 *
3792 * The number of WA applied are not known at the beginning; we use this field
3793 * to return the no of DWORDS written.
3794 *
3795 * It is to be noted that this batch does not contain MI_BATCH_BUFFER_END
3796 * so it adds NOOPs as padding to make it cacheline aligned.
3797 * MI_BATCH_BUFFER_END will be added to perctx batch and both of them together
3798 * makes a complete batch buffer.
3799 */
3800static u32 *gen8_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
3801{
3802 /* WaDisableCtxRestoreArbitration:bdw,chv */
3803 *batch++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
3804
3805 /* WaFlushCoherentL3CacheLinesAtContextSwitch:bdw */
3806 if (IS_BROADWELL(engine->i915))
3807 batch = gen8_emit_flush_coherentl3_wa(engine, batch);
3808
3809 /* WaClearSlmSpaceAtContextSwitch:bdw,chv */
3810 /* Actual scratch location is at 128 bytes offset */
3811 batch = gen8_emit_pipe_control(batch,
3812 PIPE_CONTROL_FLUSH_L3 |
3813 PIPE_CONTROL_STORE_DATA_INDEX |
3814 PIPE_CONTROL_CS_STALL |
3815 PIPE_CONTROL_QW_WRITE,
3816 LRC_PPHWSP_SCRATCH_ADDR);
3817
3818 *batch++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
3819
3820 /* Pad to end of cacheline */
3821 while ((unsigned long)batch % CACHELINE_BYTES)
3822 *batch++ = MI_NOOP;
3823
3824 /*
3825 * MI_BATCH_BUFFER_END is not required in Indirect ctx BB because
3826 * execution depends on the length specified in terms of cache lines
3827 * in the register CTX_RCS_INDIRECT_CTX
3828 */
3829
3830 return batch;
3831}
3832
3833struct lri {
3834 i915_reg_t reg;
3835 u32 value;
3836};
3837
3838static u32 *emit_lri(u32 *batch, const struct lri *lri, unsigned int count)
3839{
3840 GEM_BUG_ON(!count || count > 63);
3841
3842 *batch++ = MI_LOAD_REGISTER_IMM(count);
3843 do {
3844 *batch++ = i915_mmio_reg_offset(lri->reg);
3845 *batch++ = lri->value;
3846 } while (lri++, --count);
3847 *batch++ = MI_NOOP;
3848
3849 return batch;
3850}
3851
3852static u32 *gen9_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
3853{
3854 static const struct lri lri[] = {
3855 /* WaDisableGatherAtSetShaderCommonSlice:skl,bxt,kbl,glk */
3856 {
3857 COMMON_SLICE_CHICKEN2,
3858 __MASKED_FIELD(GEN9_DISABLE_GATHER_AT_SET_SHADER_COMMON_SLICE,
3859 0),
3860 },
3861
3862 /* BSpec: 11391 */
3863 {
3864 FF_SLICE_CHICKEN,
3865 __MASKED_FIELD(FF_SLICE_CHICKEN_CL_PROVOKING_VERTEX_FIX,
3866 FF_SLICE_CHICKEN_CL_PROVOKING_VERTEX_FIX),
3867 },
3868
3869 /* BSpec: 11299 */
3870 {
3871 _3D_CHICKEN3,
3872 __MASKED_FIELD(_3D_CHICKEN_SF_PROVOKING_VERTEX_FIX,
3873 _3D_CHICKEN_SF_PROVOKING_VERTEX_FIX),
3874 }
3875 };
3876
3877 *batch++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
3878
3879 /* WaFlushCoherentL3CacheLinesAtContextSwitch:skl,bxt,glk */
3880 batch = gen8_emit_flush_coherentl3_wa(engine, batch);
3881
3882 /* WaClearSlmSpaceAtContextSwitch:skl,bxt,kbl,glk,cfl */
3883 batch = gen8_emit_pipe_control(batch,
3884 PIPE_CONTROL_FLUSH_L3 |
3885 PIPE_CONTROL_STORE_DATA_INDEX |
3886 PIPE_CONTROL_CS_STALL |
3887 PIPE_CONTROL_QW_WRITE,
3888 LRC_PPHWSP_SCRATCH_ADDR);
3889
3890 batch = emit_lri(batch, lri, ARRAY_SIZE(lri));
3891
3892 /* WaMediaPoolStateCmdInWABB:bxt,glk */
3893 if (HAS_POOLED_EU(engine->i915)) {
3894 /*
3895 * EU pool configuration is setup along with golden context
3896 * during context initialization. This value depends on
3897 * device type (2x6 or 3x6) and needs to be updated based
3898 * on which subslice is disabled especially for 2x6
3899 * devices, however it is safe to load default
3900 * configuration of 3x6 device instead of masking off
3901 * corresponding bits because HW ignores bits of a disabled
3902 * subslice and drops down to appropriate config. Please
3903 * see render_state_setup() in i915_gem_render_state.c for
3904 * possible configurations, to avoid duplication they are
3905 * not shown here again.
3906 */
3907 *batch++ = GEN9_MEDIA_POOL_STATE;
3908 *batch++ = GEN9_MEDIA_POOL_ENABLE;
3909 *batch++ = 0x00777000;
3910 *batch++ = 0;
3911 *batch++ = 0;
3912 *batch++ = 0;
3913 }
3914
3915 *batch++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
3916
3917 /* Pad to end of cacheline */
3918 while ((unsigned long)batch % CACHELINE_BYTES)
3919 *batch++ = MI_NOOP;
3920
3921 return batch;
3922}
3923
3924static u32 *
3925gen10_init_indirectctx_bb(struct intel_engine_cs *engine, u32 *batch)
3926{
3927 int i;
3928
3929 /*
3930 * WaPipeControlBefore3DStateSamplePattern: cnl
3931 *
3932 * Ensure the engine is idle prior to programming a
3933 * 3DSTATE_SAMPLE_PATTERN during a context restore.
3934 */
3935 batch = gen8_emit_pipe_control(batch,
3936 PIPE_CONTROL_CS_STALL,
3937 0);
3938 /*
3939 * WaPipeControlBefore3DStateSamplePattern says we need 4 dwords for
3940 * the PIPE_CONTROL followed by 12 dwords of 0x0, so 16 dwords in
3941 * total. However, a PIPE_CONTROL is 6 dwords long, not 4, which is
3942 * confusing. Since gen8_emit_pipe_control() already advances the
3943 * batch by 6 dwords, we advance the other 10 here, completing a
3944 * cacheline. It's not clear if the workaround requires this padding
3945 * before other commands, or if it's just the regular padding we would
3946 * already have for the workaround bb, so leave it here for now.
3947 */
3948 for (i = 0; i < 10; i++)
3949 *batch++ = MI_NOOP;
3950
3951 /* Pad to end of cacheline */
3952 while ((unsigned long)batch % CACHELINE_BYTES)
3953 *batch++ = MI_NOOP;
3954
3955 return batch;
3956}
3957
3958#define CTX_WA_BB_OBJ_SIZE (PAGE_SIZE)
3959
3960static int lrc_setup_wa_ctx(struct intel_engine_cs *engine)
3961{
3962 struct drm_i915_gem_object *obj;
3963 struct i915_vma *vma;
3964 int err;
3965
3966 obj = i915_gem_object_create_shmem(engine->i915, CTX_WA_BB_OBJ_SIZE);
3967 if (IS_ERR(obj))
3968 return PTR_ERR(obj);
3969
3970 vma = i915_vma_instance(obj, &engine->gt->ggtt->vm, NULL);
3971 if (IS_ERR(vma)) {
3972 err = PTR_ERR(vma);
3973 goto err;
3974 }
3975
3976 err = i915_ggtt_pin(vma, NULL, 0, PIN_HIGH);
3977 if (err)
3978 goto err;
3979
3980 engine->wa_ctx.vma = vma;
3981 return 0;
3982
3983err:
3984 i915_gem_object_put(obj);
3985 return err;
3986}
3987
3988static void lrc_destroy_wa_ctx(struct intel_engine_cs *engine)
3989{
3990 i915_vma_unpin_and_release(&engine->wa_ctx.vma, 0);
3991}
3992
3993typedef u32 *(*wa_bb_func_t)(struct intel_engine_cs *engine, u32 *batch);
3994
3995static int intel_init_workaround_bb(struct intel_engine_cs *engine)
3996{
3997 struct i915_ctx_workarounds *wa_ctx = &engine->wa_ctx;
3998 struct i915_wa_ctx_bb *wa_bb[2] = { &wa_ctx->indirect_ctx,
3999 &wa_ctx->per_ctx };
4000 wa_bb_func_t wa_bb_fn[2];
4001 void *batch, *batch_ptr;
4002 unsigned int i;
4003 int ret;
4004
4005 if (engine->class != RENDER_CLASS)
4006 return 0;
4007
4008 switch (INTEL_GEN(engine->i915)) {
4009 case 12:
4010 case 11:
4011 return 0;
4012 case 10:
4013 wa_bb_fn[0] = gen10_init_indirectctx_bb;
4014 wa_bb_fn[1] = NULL;
4015 break;
4016 case 9:
4017 wa_bb_fn[0] = gen9_init_indirectctx_bb;
4018 wa_bb_fn[1] = NULL;
4019 break;
4020 case 8:
4021 wa_bb_fn[0] = gen8_init_indirectctx_bb;
4022 wa_bb_fn[1] = NULL;
4023 break;
4024 default:
4025 MISSING_CASE(INTEL_GEN(engine->i915));
4026 return 0;
4027 }
4028
4029 ret = lrc_setup_wa_ctx(engine);
4030 if (ret) {
4031 drm_dbg(&engine->i915->drm,
4032 "Failed to setup context WA page: %d\n", ret);
4033 return ret;
4034 }
4035
4036 batch = i915_gem_object_pin_map(wa_ctx->vma->obj, I915_MAP_WB);
4037
4038 /*
4039 * Emit the two workaround batch buffers, recording the offset from the
4040 * start of the workaround batch buffer object for each and their
4041 * respective sizes.
4042 */
4043 batch_ptr = batch;
4044 for (i = 0; i < ARRAY_SIZE(wa_bb_fn); i++) {
4045 wa_bb[i]->offset = batch_ptr - batch;
4046 if (GEM_DEBUG_WARN_ON(!IS_ALIGNED(wa_bb[i]->offset,
4047 CACHELINE_BYTES))) {
4048 ret = -EINVAL;
4049 break;
4050 }
4051 if (wa_bb_fn[i])
4052 batch_ptr = wa_bb_fn[i](engine, batch_ptr);
4053 wa_bb[i]->size = batch_ptr - (batch + wa_bb[i]->offset);
4054 }
4055 GEM_BUG_ON(batch_ptr - batch > CTX_WA_BB_OBJ_SIZE);
4056
4057 __i915_gem_object_flush_map(wa_ctx->vma->obj, 0, batch_ptr - batch);
4058 __i915_gem_object_release_map(wa_ctx->vma->obj);
4059 if (ret)
4060 lrc_destroy_wa_ctx(engine);
4061
4062 return ret;
4063}
4064
4065static void reset_csb_pointers(struct intel_engine_cs *engine)
4066{
4067 struct intel_engine_execlists * const execlists = &engine->execlists;
4068 const unsigned int reset_value = execlists->csb_size - 1;
4069
4070 ring_set_paused(engine, 0);
4071
4072 /*
4073 * Sometimes Icelake forgets to reset its pointers on a GPU reset.
4074 * Bludgeon them with a mmio update to be sure.
4075 */
4076 ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
4077 0xffff << 16 | reset_value << 8 | reset_value);
4078 ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
4079
4080 /*
4081 * After a reset, the HW starts writing into CSB entry [0]. We
4082 * therefore have to set our HEAD pointer back one entry so that
4083 * the *first* entry we check is entry 0. To complicate this further,
4084 * as we don't wait for the first interrupt after reset, we have to
4085 * fake the HW write to point back to the last entry so that our
4086 * inline comparison of our cached head position against the last HW
4087 * write works even before the first interrupt.
4088 */
4089 execlists->csb_head = reset_value;
4090 WRITE_ONCE(*execlists->csb_write, reset_value);
4091 wmb(); /* Make sure this is visible to HW (paranoia?) */
4092
4093 /* Check that the GPU does indeed update the CSB entries! */
4094 memset(execlists->csb_status, -1, (reset_value + 1) * sizeof(u64));
4095 invalidate_csb_entries(&execlists->csb_status[0],
4096 &execlists->csb_status[reset_value]);
4097
4098 /* Once more for luck and our trusty paranoia */
4099 ENGINE_WRITE(engine, RING_CONTEXT_STATUS_PTR,
4100 0xffff << 16 | reset_value << 8 | reset_value);
4101 ENGINE_POSTING_READ(engine, RING_CONTEXT_STATUS_PTR);
4102
4103 GEM_BUG_ON(READ_ONCE(*execlists->csb_write) != reset_value);
4104}
4105
4106static void execlists_sanitize(struct intel_engine_cs *engine)
4107{
4108 GEM_BUG_ON(execlists_active(&engine->execlists));
4109
4110 /*
4111 * Poison residual state on resume, in case the suspend didn't!
4112 *
4113 * We have to assume that across suspend/resume (or other loss
4114 * of control) that the contents of our pinned buffers has been
4115 * lost, replaced by garbage. Since this doesn't always happen,
4116 * let's poison such state so that we more quickly spot when
4117 * we falsely assume it has been preserved.
4118 */
4119 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
4120 memset(engine->status_page.addr, POISON_INUSE, PAGE_SIZE);
4121
4122 reset_csb_pointers(engine);
4123
4124 /*
4125 * The kernel_context HWSP is stored in the status_page. As above,
4126 * that may be lost on resume/initialisation, and so we need to
4127 * reset the value in the HWSP.
4128 */
4129 intel_timeline_reset_seqno(engine->kernel_context->timeline);
4130
4131 /* And scrub the dirty cachelines for the HWSP */
4132 clflush_cache_range(engine->status_page.addr, PAGE_SIZE);
4133}
4134
4135static void enable_error_interrupt(struct intel_engine_cs *engine)
4136{
4137 u32 status;
4138
4139 engine->execlists.error_interrupt = 0;
4140 ENGINE_WRITE(engine, RING_EMR, ~0u);
4141 ENGINE_WRITE(engine, RING_EIR, ~0u); /* clear all existing errors */
4142
4143 status = ENGINE_READ(engine, RING_ESR);
4144 if (unlikely(status)) {
4145 drm_err(&engine->i915->drm,
4146 "engine '%s' resumed still in error: %08x\n",
4147 engine->name, status);
4148 __intel_gt_reset(engine->gt, engine->mask);
4149 }
4150
4151 /*
4152 * On current gen8+, we have 2 signals to play with
4153 *
4154 * - I915_ERROR_INSTUCTION (bit 0)
4155 *
4156 * Generate an error if the command parser encounters an invalid
4157 * instruction
4158 *
4159 * This is a fatal error.
4160 *
4161 * - CP_PRIV (bit 2)
4162 *
4163 * Generate an error on privilege violation (where the CP replaces
4164 * the instruction with a no-op). This also fires for writes into
4165 * read-only scratch pages.
4166 *
4167 * This is a non-fatal error, parsing continues.
4168 *
4169 * * there are a few others defined for odd HW that we do not use
4170 *
4171 * Since CP_PRIV fires for cases where we have chosen to ignore the
4172 * error (as the HW is validating and suppressing the mistakes), we
4173 * only unmask the instruction error bit.
4174 */
4175 ENGINE_WRITE(engine, RING_EMR, ~I915_ERROR_INSTRUCTION);
4176}
4177
4178static void enable_execlists(struct intel_engine_cs *engine)
4179{
4180 u32 mode;
4181
4182 assert_forcewakes_active(engine->uncore, FORCEWAKE_ALL);
4183
4184 intel_engine_set_hwsp_writemask(engine, ~0u); /* HWSTAM */
4185
4186 if (INTEL_GEN(engine->i915) >= 11)
4187 mode = _MASKED_BIT_ENABLE(GEN11_GFX_DISABLE_LEGACY_MODE);
4188 else
4189 mode = _MASKED_BIT_ENABLE(GFX_RUN_LIST_ENABLE);
4190 ENGINE_WRITE_FW(engine, RING_MODE_GEN7, mode);
4191
4192 ENGINE_WRITE_FW(engine, RING_MI_MODE, _MASKED_BIT_DISABLE(STOP_RING));
4193
4194 ENGINE_WRITE_FW(engine,
4195 RING_HWS_PGA,
4196 i915_ggtt_offset(engine->status_page.vma));
4197 ENGINE_POSTING_READ(engine, RING_HWS_PGA);
4198
4199 enable_error_interrupt(engine);
4200
4201 engine->context_tag = GENMASK(BITS_PER_LONG - 2, 0);
4202}
4203
4204static bool unexpected_starting_state(struct intel_engine_cs *engine)
4205{
4206 bool unexpected = false;
4207
4208 if (ENGINE_READ_FW(engine, RING_MI_MODE) & STOP_RING) {
4209 drm_dbg(&engine->i915->drm,
4210 "STOP_RING still set in RING_MI_MODE\n");
4211 unexpected = true;
4212 }
4213
4214 return unexpected;
4215}
4216
4217static int execlists_resume(struct intel_engine_cs *engine)
4218{
4219 intel_mocs_init_engine(engine);
4220
4221 intel_breadcrumbs_reset(engine->breadcrumbs);
4222
4223 if (GEM_SHOW_DEBUG() && unexpected_starting_state(engine)) {
4224 struct drm_printer p = drm_debug_printer(__func__);
4225
4226 intel_engine_dump(engine, &p, NULL);
4227 }
4228
4229 enable_execlists(engine);
4230
4231 return 0;
4232}
4233
4234static void execlists_reset_prepare(struct intel_engine_cs *engine)
4235{
4236 struct intel_engine_execlists * const execlists = &engine->execlists;
4237 unsigned long flags;
4238
4239 ENGINE_TRACE(engine, "depth<-%d\n",
4240 atomic_read(&execlists->tasklet.count));
4241
4242 /*
4243 * Prevent request submission to the hardware until we have
4244 * completed the reset in i915_gem_reset_finish(). If a request
4245 * is completed by one engine, it may then queue a request
4246 * to a second via its execlists->tasklet *just* as we are
4247 * calling engine->resume() and also writing the ELSP.
4248 * Turning off the execlists->tasklet until the reset is over
4249 * prevents the race.
4250 */
4251 __tasklet_disable_sync_once(&execlists->tasklet);
4252 GEM_BUG_ON(!reset_in_progress(execlists));
4253
4254 /* And flush any current direct submission. */
4255 spin_lock_irqsave(&engine->active.lock, flags);
4256 spin_unlock_irqrestore(&engine->active.lock, flags);
4257
4258 /*
4259 * We stop engines, otherwise we might get failed reset and a
4260 * dead gpu (on elk). Also as modern gpu as kbl can suffer
4261 * from system hang if batchbuffer is progressing when
4262 * the reset is issued, regardless of READY_TO_RESET ack.
4263 * Thus assume it is best to stop engines on all gens
4264 * where we have a gpu reset.
4265 *
4266 * WaKBLVECSSemaphoreWaitPoll:kbl (on ALL_ENGINES)
4267 *
4268 * FIXME: Wa for more modern gens needs to be validated
4269 */
4270 ring_set_paused(engine, 1);
4271 intel_engine_stop_cs(engine);
4272
4273 engine->execlists.reset_ccid = active_ccid(engine);
4274}
4275
4276static void __reset_stop_ring(u32 *regs, const struct intel_engine_cs *engine)
4277{
4278 int x;
4279
4280 x = lrc_ring_mi_mode(engine);
4281 if (x != -1) {
4282 regs[x + 1] &= ~STOP_RING;
4283 regs[x + 1] |= STOP_RING << 16;
4284 }
4285}
4286
4287static void __execlists_reset_reg_state(const struct intel_context *ce,
4288 const struct intel_engine_cs *engine)
4289{
4290 u32 *regs = ce->lrc_reg_state;
4291
4292 __reset_stop_ring(regs, engine);
4293}
4294
4295static void __execlists_reset(struct intel_engine_cs *engine, bool stalled)
4296{
4297 struct intel_engine_execlists * const execlists = &engine->execlists;
4298 struct intel_context *ce;
4299 struct i915_request *rq;
4300 u32 head;
4301
4302 mb(); /* paranoia: read the CSB pointers from after the reset */
4303 clflush(execlists->csb_write);
4304 mb();
4305
4306 process_csb(engine); /* drain preemption events */
4307
4308 /* Following the reset, we need to reload the CSB read/write pointers */
4309 reset_csb_pointers(engine);
4310
4311 /*
4312 * Save the currently executing context, even if we completed
4313 * its request, it was still running at the time of the
4314 * reset and will have been clobbered.
4315 */
4316 rq = active_context(engine, engine->execlists.reset_ccid);
4317 if (!rq)
4318 goto unwind;
4319
4320 ce = rq->context;
4321 GEM_BUG_ON(!i915_vma_is_pinned(ce->state));
4322
4323 if (i915_request_completed(rq)) {
4324 /* Idle context; tidy up the ring so we can restart afresh */
4325 head = intel_ring_wrap(ce->ring, rq->tail);
4326 goto out_replay;
4327 }
4328
4329 /* We still have requests in-flight; the engine should be active */
4330 GEM_BUG_ON(!intel_engine_pm_is_awake(engine));
4331
4332 /* Context has requests still in-flight; it should not be idle! */
4333 GEM_BUG_ON(i915_active_is_idle(&ce->active));
4334
4335 rq = active_request(ce->timeline, rq);
4336 head = intel_ring_wrap(ce->ring, rq->head);
4337 GEM_BUG_ON(head == ce->ring->tail);
4338
4339 /*
4340 * If this request hasn't started yet, e.g. it is waiting on a
4341 * semaphore, we need to avoid skipping the request or else we
4342 * break the signaling chain. However, if the context is corrupt
4343 * the request will not restart and we will be stuck with a wedged
4344 * device. It is quite often the case that if we issue a reset
4345 * while the GPU is loading the context image, that the context
4346 * image becomes corrupt.
4347 *
4348 * Otherwise, if we have not started yet, the request should replay
4349 * perfectly and we do not need to flag the result as being erroneous.
4350 */
4351 if (!i915_request_started(rq))
4352 goto out_replay;
4353
4354 /*
4355 * If the request was innocent, we leave the request in the ELSP
4356 * and will try to replay it on restarting. The context image may
4357 * have been corrupted by the reset, in which case we may have
4358 * to service a new GPU hang, but more likely we can continue on
4359 * without impact.
4360 *
4361 * If the request was guilty, we presume the context is corrupt
4362 * and have to at least restore the RING register in the context
4363 * image back to the expected values to skip over the guilty request.
4364 */
4365 __i915_request_reset(rq, stalled);
4366
4367 /*
4368 * We want a simple context + ring to execute the breadcrumb update.
4369 * We cannot rely on the context being intact across the GPU hang,
4370 * so clear it and rebuild just what we need for the breadcrumb.
4371 * All pending requests for this context will be zapped, and any
4372 * future request will be after userspace has had the opportunity
4373 * to recreate its own state.
4374 */
4375out_replay:
4376 ENGINE_TRACE(engine, "replay {head:%04x, tail:%04x}\n",
4377 head, ce->ring->tail);
4378 __execlists_reset_reg_state(ce, engine);
4379 __execlists_update_reg_state(ce, engine, head);
4380 ce->lrc.desc |= CTX_DESC_FORCE_RESTORE; /* paranoid: GPU was reset! */
4381
4382unwind:
4383 /* Push back any incomplete requests for replay after the reset. */
4384 cancel_port_requests(execlists);
4385 __unwind_incomplete_requests(engine);
4386}
4387
4388static void execlists_reset_rewind(struct intel_engine_cs *engine, bool stalled)
4389{
4390 unsigned long flags;
4391
4392 ENGINE_TRACE(engine, "\n");
4393
4394 spin_lock_irqsave(&engine->active.lock, flags);
4395
4396 __execlists_reset(engine, stalled);
4397
4398 spin_unlock_irqrestore(&engine->active.lock, flags);
4399}
4400
4401static void nop_submission_tasklet(unsigned long data)
4402{
4403 struct intel_engine_cs * const engine = (struct intel_engine_cs *)data;
4404
4405 /* The driver is wedged; don't process any more events. */
4406 WRITE_ONCE(engine->execlists.queue_priority_hint, INT_MIN);
4407}
4408
4409static void execlists_reset_cancel(struct intel_engine_cs *engine)
4410{
4411 struct intel_engine_execlists * const execlists = &engine->execlists;
4412 struct i915_request *rq, *rn;
4413 struct rb_node *rb;
4414 unsigned long flags;
4415
4416 ENGINE_TRACE(engine, "\n");
4417
4418 /*
4419 * Before we call engine->cancel_requests(), we should have exclusive
4420 * access to the submission state. This is arranged for us by the
4421 * caller disabling the interrupt generation, the tasklet and other
4422 * threads that may then access the same state, giving us a free hand
4423 * to reset state. However, we still need to let lockdep be aware that
4424 * we know this state may be accessed in hardirq context, so we
4425 * disable the irq around this manipulation and we want to keep
4426 * the spinlock focused on its duties and not accidentally conflate
4427 * coverage to the submission's irq state. (Similarly, although we
4428 * shouldn't need to disable irq around the manipulation of the
4429 * submission's irq state, we also wish to remind ourselves that
4430 * it is irq state.)
4431 */
4432 spin_lock_irqsave(&engine->active.lock, flags);
4433
4434 __execlists_reset(engine, true);
4435
4436 /* Mark all executing requests as skipped. */
4437 list_for_each_entry(rq, &engine->active.requests, sched.link)
4438 mark_eio(rq);
4439 intel_engine_signal_breadcrumbs(engine);
4440
4441 /* Flush the queued requests to the timeline list (for retiring). */
4442 while ((rb = rb_first_cached(&execlists->queue))) {
4443 struct i915_priolist *p = to_priolist(rb);
4444 int i;
4445
4446 priolist_for_each_request_consume(rq, rn, p, i) {
4447 mark_eio(rq);
4448 __i915_request_submit(rq);
4449 }
4450
4451 rb_erase_cached(&p->node, &execlists->queue);
4452 i915_priolist_free(p);
4453 }
4454
4455 /* On-hold requests will be flushed to timeline upon their release */
4456 list_for_each_entry(rq, &engine->active.hold, sched.link)
4457 mark_eio(rq);
4458
4459 /* Cancel all attached virtual engines */
4460 while ((rb = rb_first_cached(&execlists->virtual))) {
4461 struct virtual_engine *ve =
4462 rb_entry(rb, typeof(*ve), nodes[engine->id].rb);
4463
4464 rb_erase_cached(rb, &execlists->virtual);
4465 RB_CLEAR_NODE(rb);
4466
4467 spin_lock(&ve->base.active.lock);
4468 rq = fetch_and_zero(&ve->request);
4469 if (rq) {
4470 mark_eio(rq);
4471
4472 rq->engine = engine;
4473 __i915_request_submit(rq);
4474 i915_request_put(rq);
4475
4476 ve->base.execlists.queue_priority_hint = INT_MIN;
4477 }
4478 spin_unlock(&ve->base.active.lock);
4479 }
4480
4481 /* Remaining _unready_ requests will be nop'ed when submitted */
4482
4483 execlists->queue_priority_hint = INT_MIN;
4484 execlists->queue = RB_ROOT_CACHED;
4485
4486 GEM_BUG_ON(__tasklet_is_enabled(&execlists->tasklet));
4487 execlists->tasklet.func = nop_submission_tasklet;
4488
4489 spin_unlock_irqrestore(&engine->active.lock, flags);
4490}
4491
4492static void execlists_reset_finish(struct intel_engine_cs *engine)
4493{
4494 struct intel_engine_execlists * const execlists = &engine->execlists;
4495
4496 /*
4497 * After a GPU reset, we may have requests to replay. Do so now while
4498 * we still have the forcewake to be sure that the GPU is not allowed
4499 * to sleep before we restart and reload a context.
4500 */
4501 GEM_BUG_ON(!reset_in_progress(execlists));
4502 if (!RB_EMPTY_ROOT(&execlists->queue.rb_root))
4503 execlists->tasklet.func(execlists->tasklet.data);
4504
4505 if (__tasklet_enable(&execlists->tasklet))
4506 /* And kick in case we missed a new request submission. */
4507 tasklet_hi_schedule(&execlists->tasklet);
4508 ENGINE_TRACE(engine, "depth->%d\n",
4509 atomic_read(&execlists->tasklet.count));
4510}
4511
4512static int gen8_emit_bb_start_noarb(struct i915_request *rq,
4513 u64 offset, u32 len,
4514 const unsigned int flags)
4515{
4516 u32 *cs;
4517
4518 cs = intel_ring_begin(rq, 4);
4519 if (IS_ERR(cs))
4520 return PTR_ERR(cs);
4521
4522 /*
4523 * WaDisableCtxRestoreArbitration:bdw,chv
4524 *
4525 * We don't need to perform MI_ARB_ENABLE as often as we do (in
4526 * particular all the gen that do not need the w/a at all!), if we
4527 * took care to make sure that on every switch into this context
4528 * (both ordinary and for preemption) that arbitrartion was enabled
4529 * we would be fine. However, for gen8 there is another w/a that
4530 * requires us to not preempt inside GPGPU execution, so we keep
4531 * arbitration disabled for gen8 batches. Arbitration will be
4532 * re-enabled before we close the request
4533 * (engine->emit_fini_breadcrumb).
4534 */
4535 *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
4536
4537 /* FIXME(BDW+): Address space and security selectors. */
4538 *cs++ = MI_BATCH_BUFFER_START_GEN8 |
4539 (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
4540 *cs++ = lower_32_bits(offset);
4541 *cs++ = upper_32_bits(offset);
4542
4543 intel_ring_advance(rq, cs);
4544
4545 return 0;
4546}
4547
4548static int gen8_emit_bb_start(struct i915_request *rq,
4549 u64 offset, u32 len,
4550 const unsigned int flags)
4551{
4552 u32 *cs;
4553
4554 cs = intel_ring_begin(rq, 6);
4555 if (IS_ERR(cs))
4556 return PTR_ERR(cs);
4557
4558 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
4559
4560 *cs++ = MI_BATCH_BUFFER_START_GEN8 |
4561 (flags & I915_DISPATCH_SECURE ? 0 : BIT(8));
4562 *cs++ = lower_32_bits(offset);
4563 *cs++ = upper_32_bits(offset);
4564
4565 *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
4566 *cs++ = MI_NOOP;
4567
4568 intel_ring_advance(rq, cs);
4569
4570 return 0;
4571}
4572
4573static void gen8_logical_ring_enable_irq(struct intel_engine_cs *engine)
4574{
4575 ENGINE_WRITE(engine, RING_IMR,
4576 ~(engine->irq_enable_mask | engine->irq_keep_mask));
4577 ENGINE_POSTING_READ(engine, RING_IMR);
4578}
4579
4580static void gen8_logical_ring_disable_irq(struct intel_engine_cs *engine)
4581{
4582 ENGINE_WRITE(engine, RING_IMR, ~engine->irq_keep_mask);
4583}
4584
4585static int gen8_emit_flush(struct i915_request *request, u32 mode)
4586{
4587 u32 cmd, *cs;
4588
4589 cs = intel_ring_begin(request, 4);
4590 if (IS_ERR(cs))
4591 return PTR_ERR(cs);
4592
4593 cmd = MI_FLUSH_DW + 1;
4594
4595 /* We always require a command barrier so that subsequent
4596 * commands, such as breadcrumb interrupts, are strictly ordered
4597 * wrt the contents of the write cache being flushed to memory
4598 * (and thus being coherent from the CPU).
4599 */
4600 cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
4601
4602 if (mode & EMIT_INVALIDATE) {
4603 cmd |= MI_INVALIDATE_TLB;
4604 if (request->engine->class == VIDEO_DECODE_CLASS)
4605 cmd |= MI_INVALIDATE_BSD;
4606 }
4607
4608 *cs++ = cmd;
4609 *cs++ = LRC_PPHWSP_SCRATCH_ADDR;
4610 *cs++ = 0; /* upper addr */
4611 *cs++ = 0; /* value */
4612 intel_ring_advance(request, cs);
4613
4614 return 0;
4615}
4616
4617static int gen8_emit_flush_render(struct i915_request *request,
4618 u32 mode)
4619{
4620 bool vf_flush_wa = false, dc_flush_wa = false;
4621 u32 *cs, flags = 0;
4622 int len;
4623
4624 flags |= PIPE_CONTROL_CS_STALL;
4625
4626 if (mode & EMIT_FLUSH) {
4627 flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
4628 flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
4629 flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
4630 flags |= PIPE_CONTROL_FLUSH_ENABLE;
4631 }
4632
4633 if (mode & EMIT_INVALIDATE) {
4634 flags |= PIPE_CONTROL_TLB_INVALIDATE;
4635 flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
4636 flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
4637 flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
4638 flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
4639 flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
4640 flags |= PIPE_CONTROL_QW_WRITE;
4641 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4642
4643 /*
4644 * On GEN9: before VF_CACHE_INVALIDATE we need to emit a NULL
4645 * pipe control.
4646 */
4647 if (IS_GEN(request->engine->i915, 9))
4648 vf_flush_wa = true;
4649
4650 /* WaForGAMHang:kbl */
4651 if (IS_KBL_GT_REVID(request->engine->i915, 0, KBL_REVID_B0))
4652 dc_flush_wa = true;
4653 }
4654
4655 len = 6;
4656
4657 if (vf_flush_wa)
4658 len += 6;
4659
4660 if (dc_flush_wa)
4661 len += 12;
4662
4663 cs = intel_ring_begin(request, len);
4664 if (IS_ERR(cs))
4665 return PTR_ERR(cs);
4666
4667 if (vf_flush_wa)
4668 cs = gen8_emit_pipe_control(cs, 0, 0);
4669
4670 if (dc_flush_wa)
4671 cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_DC_FLUSH_ENABLE,
4672 0);
4673
4674 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4675
4676 if (dc_flush_wa)
4677 cs = gen8_emit_pipe_control(cs, PIPE_CONTROL_CS_STALL, 0);
4678
4679 intel_ring_advance(request, cs);
4680
4681 return 0;
4682}
4683
4684static int gen11_emit_flush_render(struct i915_request *request,
4685 u32 mode)
4686{
4687 if (mode & EMIT_FLUSH) {
4688 u32 *cs;
4689 u32 flags = 0;
4690
4691 flags |= PIPE_CONTROL_CS_STALL;
4692
4693 flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
4694 flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
4695 flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
4696 flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
4697 flags |= PIPE_CONTROL_FLUSH_ENABLE;
4698 flags |= PIPE_CONTROL_QW_WRITE;
4699 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4700
4701 cs = intel_ring_begin(request, 6);
4702 if (IS_ERR(cs))
4703 return PTR_ERR(cs);
4704
4705 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4706 intel_ring_advance(request, cs);
4707 }
4708
4709 if (mode & EMIT_INVALIDATE) {
4710 u32 *cs;
4711 u32 flags = 0;
4712
4713 flags |= PIPE_CONTROL_CS_STALL;
4714
4715 flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
4716 flags |= PIPE_CONTROL_TLB_INVALIDATE;
4717 flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
4718 flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
4719 flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
4720 flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
4721 flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
4722 flags |= PIPE_CONTROL_QW_WRITE;
4723 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4724
4725 cs = intel_ring_begin(request, 6);
4726 if (IS_ERR(cs))
4727 return PTR_ERR(cs);
4728
4729 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4730 intel_ring_advance(request, cs);
4731 }
4732
4733 return 0;
4734}
4735
4736static u32 preparser_disable(bool state)
4737{
4738 return MI_ARB_CHECK | 1 << 8 | state;
4739}
4740
4741static i915_reg_t aux_inv_reg(const struct intel_engine_cs *engine)
4742{
4743 static const i915_reg_t vd[] = {
4744 GEN12_VD0_AUX_NV,
4745 GEN12_VD1_AUX_NV,
4746 GEN12_VD2_AUX_NV,
4747 GEN12_VD3_AUX_NV,
4748 };
4749
4750 static const i915_reg_t ve[] = {
4751 GEN12_VE0_AUX_NV,
4752 GEN12_VE1_AUX_NV,
4753 };
4754
4755 if (engine->class == VIDEO_DECODE_CLASS)
4756 return vd[engine->instance];
4757
4758 if (engine->class == VIDEO_ENHANCEMENT_CLASS)
4759 return ve[engine->instance];
4760
4761 GEM_BUG_ON("unknown aux_inv_reg\n");
4762
4763 return INVALID_MMIO_REG;
4764}
4765
4766static u32 *
4767gen12_emit_aux_table_inv(const i915_reg_t inv_reg, u32 *cs)
4768{
4769 *cs++ = MI_LOAD_REGISTER_IMM(1);
4770 *cs++ = i915_mmio_reg_offset(inv_reg);
4771 *cs++ = AUX_INV;
4772 *cs++ = MI_NOOP;
4773
4774 return cs;
4775}
4776
4777static int gen12_emit_flush_render(struct i915_request *request,
4778 u32 mode)
4779{
4780 if (mode & EMIT_FLUSH) {
4781 u32 flags = 0;
4782 u32 *cs;
4783
4784 flags |= PIPE_CONTROL_TILE_CACHE_FLUSH;
4785 flags |= PIPE_CONTROL_FLUSH_L3;
4786 flags |= PIPE_CONTROL_RENDER_TARGET_CACHE_FLUSH;
4787 flags |= PIPE_CONTROL_DEPTH_CACHE_FLUSH;
4788 /* Wa_1409600907:tgl */
4789 flags |= PIPE_CONTROL_DEPTH_STALL;
4790 flags |= PIPE_CONTROL_DC_FLUSH_ENABLE;
4791 flags |= PIPE_CONTROL_FLUSH_ENABLE;
4792
4793 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4794 flags |= PIPE_CONTROL_QW_WRITE;
4795
4796 flags |= PIPE_CONTROL_CS_STALL;
4797
4798 cs = intel_ring_begin(request, 6);
4799 if (IS_ERR(cs))
4800 return PTR_ERR(cs);
4801
4802 cs = gen12_emit_pipe_control(cs,
4803 PIPE_CONTROL0_HDC_PIPELINE_FLUSH,
4804 flags, LRC_PPHWSP_SCRATCH_ADDR);
4805 intel_ring_advance(request, cs);
4806 }
4807
4808 if (mode & EMIT_INVALIDATE) {
4809 u32 flags = 0;
4810 u32 *cs;
4811
4812 flags |= PIPE_CONTROL_COMMAND_CACHE_INVALIDATE;
4813 flags |= PIPE_CONTROL_TLB_INVALIDATE;
4814 flags |= PIPE_CONTROL_INSTRUCTION_CACHE_INVALIDATE;
4815 flags |= PIPE_CONTROL_TEXTURE_CACHE_INVALIDATE;
4816 flags |= PIPE_CONTROL_VF_CACHE_INVALIDATE;
4817 flags |= PIPE_CONTROL_CONST_CACHE_INVALIDATE;
4818 flags |= PIPE_CONTROL_STATE_CACHE_INVALIDATE;
4819
4820 flags |= PIPE_CONTROL_STORE_DATA_INDEX;
4821 flags |= PIPE_CONTROL_QW_WRITE;
4822
4823 flags |= PIPE_CONTROL_CS_STALL;
4824
4825 cs = intel_ring_begin(request, 8 + 4);
4826 if (IS_ERR(cs))
4827 return PTR_ERR(cs);
4828
4829 /*
4830 * Prevent the pre-parser from skipping past the TLB
4831 * invalidate and loading a stale page for the batch
4832 * buffer / request payload.
4833 */
4834 *cs++ = preparser_disable(true);
4835
4836 cs = gen8_emit_pipe_control(cs, flags, LRC_PPHWSP_SCRATCH_ADDR);
4837
4838 /* hsdes: 1809175790 */
4839 cs = gen12_emit_aux_table_inv(GEN12_GFX_CCS_AUX_NV, cs);
4840
4841 *cs++ = preparser_disable(false);
4842 intel_ring_advance(request, cs);
4843 }
4844
4845 return 0;
4846}
4847
4848static int gen12_emit_flush(struct i915_request *request, u32 mode)
4849{
4850 intel_engine_mask_t aux_inv = 0;
4851 u32 cmd, *cs;
4852
4853 cmd = 4;
4854 if (mode & EMIT_INVALIDATE)
4855 cmd += 2;
4856 if (mode & EMIT_INVALIDATE)
4857 aux_inv = request->engine->mask & ~BIT(BCS0);
4858 if (aux_inv)
4859 cmd += 2 * hweight8(aux_inv) + 2;
4860
4861 cs = intel_ring_begin(request, cmd);
4862 if (IS_ERR(cs))
4863 return PTR_ERR(cs);
4864
4865 if (mode & EMIT_INVALIDATE)
4866 *cs++ = preparser_disable(true);
4867
4868 cmd = MI_FLUSH_DW + 1;
4869
4870 /* We always require a command barrier so that subsequent
4871 * commands, such as breadcrumb interrupts, are strictly ordered
4872 * wrt the contents of the write cache being flushed to memory
4873 * (and thus being coherent from the CPU).
4874 */
4875 cmd |= MI_FLUSH_DW_STORE_INDEX | MI_FLUSH_DW_OP_STOREDW;
4876
4877 if (mode & EMIT_INVALIDATE) {
4878 cmd |= MI_INVALIDATE_TLB;
4879 if (request->engine->class == VIDEO_DECODE_CLASS)
4880 cmd |= MI_INVALIDATE_BSD;
4881 }
4882
4883 *cs++ = cmd;

2191 more lines aren’t shown here.

drivers/gpu/drm/i915/gt/intel_lrc.h+61−106
@@ -1,90 +1,20 @@
1/* SPDX-License-Identifier: MIT */
1/*2/*
2 * Copyright © 2014 Intel Corporation3 * Copyright © 2014 Intel Corporation
3 *
4 * Permission is hereby granted, free of charge, to any person obtaining a
5 * copy of this software and associated documentation files (the "Software"),
6 * to deal in the Software without restriction, including without limitation
7 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
8 * and/or sell copies of the Software, and to permit persons to whom the
9 * Software is furnished to do so, subject to the following conditions:
10 *
11 * The above copyright notice and this permission notice (including the next
12 * paragraph) shall be included in all copies or substantial portions of the
13 * Software.
14 *
15 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
18 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
20 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
21 * DEALINGS IN THE SOFTWARE.
22 */4 */
23 5
24#ifndef _INTEL_LRC_H_6#ifndef __INTEL_LRC_H__
25#define _INTEL_LRC_H_7#define __INTEL_LRC_H__
26 8
27#include <linux/types.h>9#include <linux/types.h>
28 10
29struct drm_printer;11#include "intel_context.h"
12#include "intel_lrc_reg.h"
30 13
31struct drm_i915_private;14struct drm_i915_gem_object;
32struct i915_gem_context;
33struct i915_request;
34struct intel_context;
35struct intel_engine_cs;15struct intel_engine_cs;
16struct intel_ring;
36 17
37/* Execlists regs */
38#define RING_ELSP(base) _MMIO((base) + 0x230)
39#define RING_EXECLIST_STATUS_LO(base) _MMIO((base) + 0x234)
40#define RING_EXECLIST_STATUS_HI(base) _MMIO((base) + 0x234 + 4)
41#define RING_CONTEXT_CONTROL(base) _MMIO((base) + 0x244)
42#define CTX_CTRL_INHIBIT_SYN_CTX_SWITCH (1 << 3)
43#define CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT (1 << 0)
44#define CTX_CTRL_RS_CTX_ENABLE (1 << 1)
45#define CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT (1 << 2)
46#define GEN12_CTX_CTRL_OAR_CONTEXT_ENABLE (1 << 8)
47#define RING_CONTEXT_STATUS_PTR(base) _MMIO((base) + 0x3a0)
48#define RING_EXECLIST_SQ_CONTENTS(base) _MMIO((base) + 0x510)
49#define RING_EXECLIST_CONTROL(base) _MMIO((base) + 0x550)
50
51#define EL_CTRL_LOAD (1 << 0)
52
53/* The docs specify that the write pointer wraps around after 5h, "After status
54 * is written out to the last available status QW at offset 5h, this pointer
55 * wraps to 0."
56 *
57 * Therefore, one must infer than even though there are 3 bits available, 6 and
58 * 7 appear to be * reserved.
59 */
60#define GEN8_CSB_ENTRIES 6
61#define GEN8_CSB_PTR_MASK 0x7
62#define GEN8_CSB_READ_PTR_MASK (GEN8_CSB_PTR_MASK << 8)
63#define GEN8_CSB_WRITE_PTR_MASK (GEN8_CSB_PTR_MASK << 0)
64
65#define GEN11_CSB_ENTRIES 12
66#define GEN11_CSB_PTR_MASK 0xf
67#define GEN11_CSB_READ_PTR_MASK (GEN11_CSB_PTR_MASK << 8)
68#define GEN11_CSB_WRITE_PTR_MASK (GEN11_CSB_PTR_MASK << 0)
69
70#define MAX_CONTEXT_HW_ID (1<<21) /* exclusive */
71#define MAX_GUC_CONTEXT_HW_ID (1 << 20) /* exclusive */
72#define GEN11_MAX_CONTEXT_HW_ID (1<<11) /* exclusive */
73/* in Gen12 ID 0x7FF is reserved to indicate idle */
74#define GEN12_MAX_CONTEXT_HW_ID (GEN11_MAX_CONTEXT_HW_ID - 1)
75
76enum {
77 INTEL_CONTEXT_SCHEDULE_IN = 0,
78 INTEL_CONTEXT_SCHEDULE_OUT,
79 INTEL_CONTEXT_SCHEDULE_PREEMPTED,
80};
81
82/* Logical Rings */
83void intel_logical_ring_cleanup(struct intel_engine_cs *engine);
84
85int intel_execlists_submission_setup(struct intel_engine_cs *engine);
86
87/* Logical Ring Contexts */
88/* At the start of the context image is its per-process HWS page */18/* At the start of the context image is its per-process HWS page */
89#define LRC_PPHWSP_PN (0)19#define LRC_PPHWSP_PN (0)
90#define LRC_PPHWSP_SZ (1)20#define LRC_PPHWSP_SZ (1)
@@ -96,32 +26,57 @@ int intel_execlists_submission_setup(struct intel_engine_cs *engine); · 5 unmodified lines
96#define LRC_PPHWSP_SCRATCH 0x3426#define LRC_PPHWSP_SCRATCH 0x34
97#define LRC_PPHWSP_SCRATCH_ADDR (LRC_PPHWSP_SCRATCH * sizeof(u32))27#define LRC_PPHWSP_SCRATCH_ADDR (LRC_PPHWSP_SCRATCH * sizeof(u32))
98 28
99void intel_execlists_set_default_submission(struct intel_engine_cs *engine);29void lrc_init_wa_ctx(struct intel_engine_cs *engine);
100 30void lrc_fini_wa_ctx(struct intel_engine_cs *engine);
101void intel_lr_context_reset(struct intel_engine_cs *engine,31
102 struct intel_context *ce,32int lrc_alloc(struct intel_context *ce,
103 u32 head,33 struct intel_engine_cs *engine);
104 bool scrub);34void lrc_reset(struct intel_context *ce);
105 35void lrc_fini(struct intel_context *ce);
106void intel_execlists_show_requests(struct intel_engine_cs *engine,36void lrc_destroy(struct kref *kref);
107 struct drm_printer *m,37
108 void (*show_request)(struct drm_printer *m,38int
109 struct i915_request *rq,39lrc_pre_pin(struct intel_context *ce,
110 const char *prefix),40 struct intel_engine_cs *engine,
111 unsigned int max);41 struct i915_gem_ww_ctx *ww,
112 42 void **vaddr);
113struct intel_context *43int
114intel_execlists_create_virtual(struct intel_engine_cs **siblings,44lrc_pin(struct intel_context *ce,
115 unsigned int count);45 struct intel_engine_cs *engine,
116 46 void *vaddr);
117struct intel_context *47void lrc_unpin(struct intel_context *ce);
118intel_execlists_clone_virtual(struct intel_engine_cs *src);48void lrc_post_unpin(struct intel_context *ce);
119 49
120int intel_virtual_engine_attach_bond(struct intel_engine_cs *engine,50void lrc_init_state(struct intel_context *ce,
121 const struct intel_engine_cs *master,51 struct intel_engine_cs *engine,
122 const struct intel_engine_cs *sibling);52 void *state);
123 53
124bool54void lrc_init_regs(const struct intel_context *ce,
125intel_engine_in_execlists_submission_mode(const struct intel_engine_cs *engine);55 const struct intel_engine_cs *engine,
126 56 bool clear);
127#endif /* _INTEL_LRC_H_ */57void lrc_reset_regs(const struct intel_context *ce,
58 const struct intel_engine_cs *engine);
59
60u32 lrc_update_regs(const struct intel_context *ce,
61 const struct intel_engine_cs *engine,
62 u32 head);
63void lrc_update_offsets(struct intel_context *ce,
64 struct intel_engine_cs *engine);
65
66void lrc_check_regs(const struct intel_context *ce,
67 const struct intel_engine_cs *engine,
68 const char *when);
69
70void lrc_update_runtime(struct intel_context *ce);
71static inline u32 lrc_get_runtime(const struct intel_context *ce)
72{
73 /*
74 * We can use either ppHWSP[16] which is recorded before the context
75 * switch (and so excludes the cost of context switches) or use the
76 * value from the context image itself, which is saved/restored earlier
77 * and so includes the cost of the save.
78 */
79 return READ_ONCE(ce->lrc_reg_state[CTX_TIMESTAMP]);
80}
81
82#endif /* __INTEL_LRC_H__ */
drivers/gpu/drm/i915/gt/intel_lrc_reg.h+41−0
@@ -9,6 +9,8 @@ · 8 unmodified lines
9 9
10#include <linux/types.h>10#include <linux/types.h>
11 11
12#define CTX_DESC_FORCE_RESTORE BIT_ULL(2)
13
12/* GEN8 to GEN12 Reg State Context */14/* GEN8 to GEN12 Reg State Context */
13#define CTX_CONTEXT_CONTROL (0x02 + 1)15#define CTX_CONTEXT_CONTROL (0x02 + 1)
14#define CTX_RING_HEAD (0x04 + 1)16#define CTX_RING_HEAD (0x04 + 1)
@@ -52,4 +54,43 @@ · 37 unmodified lines
52#define GEN8_EXECLISTS_STATUS_BUF 0x37054#define GEN8_EXECLISTS_STATUS_BUF 0x370
53#define GEN11_EXECLISTS_STATUS_BUF2 0x3c055#define GEN11_EXECLISTS_STATUS_BUF2 0x3c0
54 56
57/* Execlists regs */
58#define RING_ELSP(base) _MMIO((base) + 0x230)
59#define RING_EXECLIST_STATUS_LO(base) _MMIO((base) + 0x234)
60#define RING_EXECLIST_STATUS_HI(base) _MMIO((base) + 0x234 + 4)
61#define RING_CONTEXT_CONTROL(base) _MMIO((base) + 0x244)
62#define CTX_CTRL_ENGINE_CTX_RESTORE_INHIBIT REG_BIT(0)
63#define CTX_CTRL_RS_CTX_ENABLE REG_BIT(1)
64#define CTX_CTRL_ENGINE_CTX_SAVE_INHIBIT REG_BIT(2)
65#define CTX_CTRL_INHIBIT_SYN_CTX_SWITCH REG_BIT(3)
66#define GEN12_CTX_CTRL_OAR_CONTEXT_ENABLE REG_BIT(8)
67#define RING_CONTEXT_STATUS_PTR(base) _MMIO((base) + 0x3a0)
68#define RING_EXECLIST_SQ_CONTENTS(base) _MMIO((base) + 0x510)
69#define RING_EXECLIST_CONTROL(base) _MMIO((base) + 0x550)
70#define EL_CTRL_LOAD REG_BIT(0)
71
72/*
73 * The docs specify that the write pointer wraps around after 5h, "After status
74 * is written out to the last available status QW at offset 5h, this pointer
75 * wraps to 0."
76 *
77 * Therefore, one must infer than even though there are 3 bits available, 6 and
78 * 7 appear to be * reserved.
79 */
80#define GEN8_CSB_ENTRIES 6
81#define GEN8_CSB_PTR_MASK 0x7
82#define GEN8_CSB_READ_PTR_MASK (GEN8_CSB_PTR_MASK << 8)
83#define GEN8_CSB_WRITE_PTR_MASK (GEN8_CSB_PTR_MASK << 0)
84
85#define GEN11_CSB_ENTRIES 12
86#define GEN11_CSB_PTR_MASK 0xf
87#define GEN11_CSB_READ_PTR_MASK (GEN11_CSB_PTR_MASK << 8)
88#define GEN11_CSB_WRITE_PTR_MASK (GEN11_CSB_PTR_MASK << 0)
89
90#define MAX_CONTEXT_HW_ID (1 << 21) /* exclusive */
91#define MAX_GUC_CONTEXT_HW_ID (1 << 20) /* exclusive */
92#define GEN11_MAX_CONTEXT_HW_ID (1 << 11) /* exclusive */
93/* in Gen12 ID 0x7FF is reserved to indicate idle */
94#define GEN12_MAX_CONTEXT_HW_ID (GEN11_MAX_CONTEXT_HW_ID - 1)
95
55#endif /* _INTEL_LRC_REG_H_ */96#endif /* _INTEL_LRC_REG_H_ */
drivers/gpu/drm/i915/gt/intel_mocs.c+1−1
@@ -24,8 +24,8 @@ · 23 unmodified lines
24 24
25#include "intel_engine.h"25#include "intel_engine.h"
26#include "intel_gt.h"26#include "intel_gt.h"
27#include "intel_lrc_reg.h"
27#include "intel_mocs.h"28#include "intel_mocs.h"
28#include "intel_lrc.h"
29#include "intel_ring.h"29#include "intel_ring.h"
30 30
31/* structures required */31/* structures required */
drivers/gpu/drm/i915/gt/intel_region_lmem.crenamed from drivers/gpu/drm/i915/intel_region_lmem.c+1−1
@@ -95,7 +95,7 @@ region_lmem_init(struct intel_memory_region *mem) · 94 unmodified lines
95 return ret;95 return ret;
96}96}
97 97
98const struct intel_memory_region_ops intel_region_lmem_ops = {98static const struct intel_memory_region_ops intel_region_lmem_ops = {
99 .init = region_lmem_init,99 .init = region_lmem_init,
100 .release = region_lmem_release,100 .release = region_lmem_release,
101 .create_object = __i915_gem_lmem_object_create,101 .create_object = __i915_gem_lmem_object_create,
drivers/gpu/drm/i915/gt/intel_region_lmem.hrenamed from drivers/gpu/drm/i915/intel_region_lmem.h+0−2
@@ -8,8 +8,6 @@ · 7 unmodified lines
8 8
9struct drm_i915_private;9struct drm_i915_private;
10 10
11extern const struct intel_memory_region_ops intel_region_lmem_ops;
12
13struct intel_memory_region *11struct intel_memory_region *
14intel_setup_fake_lmem(struct drm_i915_private *i915);12intel_setup_fake_lmem(struct drm_i915_private *i915);
15 13
drivers/gpu/drm/i915/gt/intel_renderstate.c+2−1
@@ -27,7 +27,8 @@ · 26 unmodified lines
27 27
28#include "i915_drv.h"28#include "i915_drv.h"
29#include "intel_renderstate.h"29#include "intel_renderstate.h"
30#include "gt/intel_context.h"30#include "intel_context.h"
31#include "intel_gpu_commands.h"
31#include "intel_ring.h"32#include "intel_ring.h"
32 33
33static const struct intel_renderstate_rodata *34static const struct intel_renderstate_rodata *
drivers/gpu/drm/i915/gt/intel_reset.c+62−34
@@ -40,20 +40,19 @@ static void rmw_clear_fw(struct intel_uncore *uncore, i915_reg_t reg, u32 clr) · 39 unmodified lines
40 intel_uncore_rmw_fw(uncore, reg, clr, 0);40 intel_uncore_rmw_fw(uncore, reg, clr, 0);
41}41}
42 42
43static void engine_skip_context(struct i915_request *rq)43static void skip_context(struct i915_request *rq)
44{44{
45 struct intel_engine_cs *engine = rq->engine;
46 struct intel_context *hung_ctx = rq->context;45 struct intel_context *hung_ctx = rq->context;
47 46
48 if (!i915_request_is_active(rq))47 list_for_each_entry_from_rcu(rq, &hung_ctx->timeline->requests, link) {
49 return;48 if (!i915_request_is_active(rq))
49 return;
50 50
51 lockdep_assert_held(&engine->active.lock);
52 list_for_each_entry_continue(rq, &engine->active.requests, sched.link)
53 if (rq->context == hung_ctx) {51 if (rq->context == hung_ctx) {
54 i915_request_set_error_once(rq, -EIO);52 i915_request_set_error_once(rq, -EIO);
55 __i915_request_skip(rq);53 __i915_request_skip(rq);
56 }54 }
55 }
57}56}
58 57
59static void client_mark_guilty(struct i915_gem_context *ctx, bool banned)58static void client_mark_guilty(struct i915_gem_context *ctx, bool banned)
@@ -160,7 +159,7 @@ void __i915_request_reset(struct i915_request *rq, bool guilty) · 100 unmodified lines
160 i915_request_set_error_once(rq, -EIO);159 i915_request_set_error_once(rq, -EIO);
161 __i915_request_skip(rq);160 __i915_request_skip(rq);
162 if (mark_guilty(rq))161 if (mark_guilty(rq))
163 engine_skip_context(rq);162 skip_context(rq);
164 } else {163 } else {
165 i915_request_set_error_once(rq, -EAGAIN);164 i915_request_set_error_once(rq, -EAGAIN);
166 mark_innocent(rq);165 mark_innocent(rq);
@@ -231,7 +230,7 @@ static int g4x_do_reset(struct intel_gt *gt, · 64 unmodified lines
231 GRDOM_MEDIA | GRDOM_RESET_ENABLE);230 GRDOM_MEDIA | GRDOM_RESET_ENABLE);
232 ret = wait_for_atomic(g4x_reset_complete(pdev), 50);231 ret = wait_for_atomic(g4x_reset_complete(pdev), 50);
233 if (ret) {232 if (ret) {
234 drm_dbg(&gt->i915->drm, "Wait for media reset failed\n");233 GT_TRACE(gt, "Wait for media reset failed\n");
235 goto out;234 goto out;
236 }235 }
237 236
@@ -239,7 +238,7 @@ static int g4x_do_reset(struct intel_gt *gt, · 1 unmodified line
239 GRDOM_RENDER | GRDOM_RESET_ENABLE);238 GRDOM_RENDER | GRDOM_RESET_ENABLE);
240 ret = wait_for_atomic(g4x_reset_complete(pdev), 50);239 ret = wait_for_atomic(g4x_reset_complete(pdev), 50);
241 if (ret) {240 if (ret) {
242 drm_dbg(&gt->i915->drm, "Wait for render reset failed\n");241 GT_TRACE(gt, "Wait for render reset failed\n");
243 goto out;242 goto out;
244 }243 }
245 244
@@ -265,7 +264,7 @@ static int ilk_do_reset(struct intel_gt *gt, intel_engine_mask_t engine_mask, · 19 unmodified lines
265 5000, 0,264 5000, 0,
266 NULL);265 NULL);
267 if (ret) {266 if (ret) {
268 drm_dbg(&gt->i915->drm, "Wait for render reset failed\n");267 GT_TRACE(gt, "Wait for render reset failed\n");
269 goto out;268 goto out;
270 }269 }
271 270
@@ -276,7 +275,7 @@ static int ilk_do_reset(struct intel_gt *gt, intel_engine_mask_t engine_mask, · 4 unmodified lines
276 5000, 0,275 5000, 0,
277 NULL);276 NULL);
278 if (ret) {277 if (ret) {
279 drm_dbg(&gt->i915->drm, "Wait for media reset failed\n");278 GT_TRACE(gt, "Wait for media reset failed\n");
280 goto out;279 goto out;
281 }280 }
282 281
@@ -305,9 +304,9 @@ static int gen6_hw_domain_reset(struct intel_gt *gt, u32 hw_domain_mask) · 22 unmodified lines
305 500, 0,304 500, 0,
306 NULL);305 NULL);
307 if (err)306 if (err)
308 drm_dbg(&gt->i915->drm,307 GT_TRACE(gt,
309 "Wait for 0x%08x engines reset failed\n",308 "Wait for 0x%08x engines reset failed\n",
310 hw_domain_mask);309 hw_domain_mask);
311 310
312 return err;311 return err;
313}312}
@@ -407,8 +406,7 @@ static int gen11_lock_sfc(struct intel_engine_cs *engine, u32 *hw_mask) · 93 unmodified lines
407 return 0;406 return 0;
408 407
409 if (ret) {408 if (ret) {
410 drm_dbg(&engine->i915->drm,409 ENGINE_TRACE(engine, "Wait for SFC forced lock ack failed\n");
411 "Wait for SFC forced lock ack failed\n");
412 return ret;410 return ret;
413 }411 }
414 412
@@ -499,6 +497,9 @@ static int gen8_engine_reset_prepare(struct intel_engine_cs *engine) · 84 unmodified lines
499 u32 request, mask, ack;497 u32 request, mask, ack;
500 int ret;498 int ret;
501 499
500 if (I915_SELFTEST_ONLY(should_fail(&engine->reset_timeout, 1)))
501 return -ETIMEDOUT;
502
502 ack = intel_uncore_read_fw(uncore, reg);503 ack = intel_uncore_read_fw(uncore, reg);
503 if (ack & RESET_CTL_CAT_ERROR) {504 if (ack & RESET_CTL_CAT_ERROR) {
504 /*505 /*
@@ -754,8 +755,10 @@ static int gt_reset(struct intel_gt *gt, intel_engine_mask_t stalled_mask) · 249 unmodified lines
754 if (err)755 if (err)
755 return err;756 return err;
756 757
758 local_bh_disable();
757 for_each_engine(engine, gt, id)759 for_each_engine(engine, gt, id)
758 __intel_engine_reset(engine, stalled_mask & engine->mask);760 __intel_engine_reset(engine, stalled_mask & engine->mask);
761 local_bh_enable();
759 762
760 intel_ggtt_restore_fences(gt->ggtt);763 intel_ggtt_restore_fences(gt->ggtt);
761 764
@@ -833,9 +836,11 @@ static void __intel_gt_set_wedged(struct intel_gt *gt) · 71 unmodified lines
833 set_bit(I915_WEDGED, &gt->reset.flags);836 set_bit(I915_WEDGED, &gt->reset.flags);
834 837
835 /* Mark all executing requests as skipped */838 /* Mark all executing requests as skipped */
839 local_bh_disable();
836 for_each_engine(engine, gt, id)840 for_each_engine(engine, gt, id)
837 if (engine->reset.cancel)841 if (engine->reset.cancel)
838 engine->reset.cancel(engine);842 engine->reset.cancel(engine);
843 local_bh_enable();
839 844
840 reset_finish(gt, awake);845 reset_finish(gt, awake);
841 846
@@ -1110,20 +1115,7 @@ static inline int intel_gt_reset_engine(struct intel_engine_cs *engine) · 268 unmodified lines
1110 return __intel_gt_reset(engine->gt, engine->mask);1115 return __intel_gt_reset(engine->gt, engine->mask);
1111}1116}
1112 1117
1113/**1118int __intel_engine_reset_bh(struct intel_engine_cs *engine, const char *msg)
1114 * intel_engine_reset - reset GPU engine to recover from a hang
1115 * @engine: engine to reset
1116 * @msg: reason for GPU reset; or NULL for no drm_notice()
1117 *
1118 * Reset a specific GPU engine. Useful if a hang is detected.
1119 * Returns zero on successful reset or otherwise an error code.
1120 *
1121 * Procedure is:
1122 * - identifies the request that caused the hang and it is dropped
1123 * - reset engine (which will force the engine to idle)
1124 * - re-init/configure engine
1125 */
1126int intel_engine_reset(struct intel_engine_cs *engine, const char *msg)
1127{1119{
1128 struct intel_gt *gt = engine->gt;1120 struct intel_gt *gt = engine->gt;
1129 bool uses_guc = intel_engine_in_guc_submission_mode(engine);1121 bool uses_guc = intel_engine_in_guc_submission_mode(engine);
@@ -1148,8 +1140,7 @@ int intel_engine_reset(struct intel_engine_cs *engine, const char *msg) · 18 unmodified lines
1148 ret = intel_guc_reset_engine(&engine->gt->uc.guc, engine);1140 ret = intel_guc_reset_engine(&engine->gt->uc.guc, engine);
1149 if (ret) {1141 if (ret) {
1150 /* If we fail here, we expect to fallback to a global reset */1142 /* If we fail here, we expect to fallback to a global reset */
1151 drm_dbg(&gt->i915->drm, "%sFailed to reset %s, ret=%d\n",1143 ENGINE_TRACE(engine, "Failed to reset, err: %d\n", ret);
1152 uses_guc ? "GuC " : "", engine->name, ret);
1153 goto out;1144 goto out;
1154 }1145 }
1155 1146
@@ -1174,6 +1165,30 @@ out: · 18 unmodified lines
1174 return ret;1165 return ret;
1175}1166}
1176 1167
1168/**
1169 * intel_engine_reset - reset GPU engine to recover from a hang
1170 * @engine: engine to reset
1171 * @msg: reason for GPU reset; or NULL for no drm_notice()
1172 *
1173 * Reset a specific GPU engine. Useful if a hang is detected.
1174 * Returns zero on successful reset or otherwise an error code.
1175 *
1176 * Procedure is:
1177 * - identifies the request that caused the hang and it is dropped
1178 * - reset engine (which will force the engine to idle)
1179 * - re-init/configure engine
1180 */
1181int intel_engine_reset(struct intel_engine_cs *engine, const char *msg)
1182{
1183 int err;
1184
1185 local_bh_disable();
1186 err = __intel_engine_reset_bh(engine, msg);
1187 local_bh_enable();
1188
1189 return err;
1190}
1191
1177static void intel_gt_reset_global(struct intel_gt *gt,1192static void intel_gt_reset_global(struct intel_gt *gt,
1178 u32 engine_mask,1193 u32 engine_mask,
1179 const char *reason)1194 const char *reason)
@@ -1186,7 +1201,7 @@ static void intel_gt_reset_global(struct intel_gt *gt, · 6 unmodified lines
1186 1201
1187 kobject_uevent_env(kobj, KOBJ_CHANGE, error_event);1202 kobject_uevent_env(kobj, KOBJ_CHANGE, error_event);
1188 1203
1189 drm_dbg(&gt->i915->drm, "resetting chip, engines=%x\n", engine_mask);1204 GT_TRACE(gt, "resetting chip, engines=%x\n", engine_mask);
1190 kobject_uevent_env(kobj, KOBJ_CHANGE, reset_event);1205 kobject_uevent_env(kobj, KOBJ_CHANGE, reset_event);
1191 1206
1192 /* Use a watchdog to ensure that our reset completes */1207 /* Use a watchdog to ensure that our reset completes */
@@ -1260,18 +1275,20 @@ void intel_gt_handle_error(struct intel_gt *gt, · 67 unmodified lines
1260 * single reset fails.1275 * single reset fails.
1261 */1276 */
1262 if (intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {1277 if (intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
1278 local_bh_disable();
1263 for_each_engine_masked(engine, gt, engine_mask, tmp) {1279 for_each_engine_masked(engine, gt, engine_mask, tmp) {
1264 BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);1280 BUILD_BUG_ON(I915_RESET_MODESET >= I915_RESET_ENGINE);
1265 if (test_and_set_bit(I915_RESET_ENGINE + engine->id,1281 if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
1266 &gt->reset.flags))1282 &gt->reset.flags))
1267 continue;1283 continue;
1268 1284
1269 if (intel_engine_reset(engine, msg) == 0)1285 if (__intel_engine_reset_bh(engine, msg) == 0)
1270 engine_mask &= ~engine->mask;1286 engine_mask &= ~engine->mask;
1271 1287
1272 clear_and_wake_up_bit(I915_RESET_ENGINE + engine->id,1288 clear_and_wake_up_bit(I915_RESET_ENGINE + engine->id,
1273 &gt->reset.flags);1289 &gt->reset.flags);
1274 }1290 }
1291 local_bh_enable();
1275 }1292 }
1276 1293
1277 if (!engine_mask)1294 if (!engine_mask)
@@ -1380,6 +1397,17 @@ void intel_gt_init_reset(struct intel_gt *gt) · 102 unmodified lines
1380 mutex_init(&gt->reset.mutex);1397 mutex_init(&gt->reset.mutex);
1381 init_srcu_struct(&gt->reset.backoff_srcu);1398 init_srcu_struct(&gt->reset.backoff_srcu);
1382 1399
1400 /*
1401 * While undesirable to wait inside the shrinker, complain anyway.
1402 *
1403 * If we have to wait during shrinking, we guarantee forward progress
1404 * by forcing the reset. Therefore during the reset we must not
1405 * re-enter the shrinker. By declaring that we take the reset mutex
1406 * within the shrinker, we forbid ourselves from performing any
1407 * fs-reclaim or taking related locks during reset.
1408 */
1409 i915_gem_shrinker_taints_mutex(gt->i915, &gt->reset.mutex);
1410
1383 /* no GPU until we are ready! */1411 /* no GPU until we are ready! */
1384 __set_bit(I915_WEDGED, &gt->reset.flags);1412 __set_bit(I915_WEDGED, &gt->reset.flags);
1385}1413}
drivers/gpu/drm/i915/gt/intel_reset.h+2−0
@@ -34,6 +34,8 @@ void intel_gt_reset(struct intel_gt *gt, · 33 unmodified lines
34 const char *reason);34 const char *reason);
35int intel_engine_reset(struct intel_engine_cs *engine,35int intel_engine_reset(struct intel_engine_cs *engine,
36 const char *reason);36 const char *reason);
37int __intel_engine_reset_bh(struct intel_engine_cs *engine,
38 const char *reason);
37 39
38void __i915_request_reset(struct i915_request *rq, bool guilty);40void __i915_request_reset(struct i915_request *rq, bool guilty);
39 41
drivers/gpu/drm/i915/gt/intel_ring.c+2−0
@@ -5,9 +5,11 @@ · 4 unmodified lines
5 */5 */
6 6
7#include "gem/i915_gem_object.h"7#include "gem/i915_gem_object.h"
8
8#include "i915_drv.h"9#include "i915_drv.h"
9#include "i915_vma.h"10#include "i915_vma.h"
10#include "intel_engine.h"11#include "intel_engine.h"
12#include "intel_gpu_commands.h"
11#include "intel_ring.h"13#include "intel_ring.h"
12#include "intel_timeline.h"14#include "intel_timeline.h"
13 15
drivers/gpu/drm/i915/gt/intel_ring_submission.c+43−30
@@ -32,6 +32,7 @@ · 31 unmodified lines
32#include "gen6_ppgtt.h"32#include "gen6_ppgtt.h"
33#include "gen7_renderclear.h"33#include "gen7_renderclear.h"
34#include "i915_drv.h"34#include "i915_drv.h"
35#include "i915_mitigations.h"
35#include "intel_breadcrumbs.h"36#include "intel_breadcrumbs.h"
36#include "intel_context.h"37#include "intel_context.h"
37#include "intel_gt.h"38#include "intel_gt.h"
@@ -158,30 +159,7 @@ static void ring_setup_status_page(struct intel_engine_cs *engine) · 120 unmodified lines
158 159
159static bool stop_ring(struct intel_engine_cs *engine)160static bool stop_ring(struct intel_engine_cs *engine)
160{161{
161 struct drm_i915_private *dev_priv = engine->i915;162 intel_engine_stop_cs(engine);
162
163 if (INTEL_GEN(dev_priv) > 2) {
164 ENGINE_WRITE(engine,
165 RING_MI_MODE, _MASKED_BIT_ENABLE(STOP_RING));
166 if (intel_wait_for_register(engine->uncore,
167 RING_MI_MODE(engine->mmio_base),
168 MODE_IDLE,
169 MODE_IDLE,
170 1000)) {
171 drm_err(&dev_priv->drm,
172 "%s : timed out trying to stop ring\n",
173 engine->name);
174
175 /*
176 * Sometimes we observe that the idle flag is not
177 * set even though the ring is empty. So double
178 * check before giving up.
179 */
180 if (ENGINE_READ(engine, RING_HEAD) !=
181 ENGINE_READ(engine, RING_TAIL))
182 return false;
183 }
184 }
185 163
186 ENGINE_WRITE(engine, RING_HEAD, ENGINE_READ(engine, RING_TAIL));164 ENGINE_WRITE(engine, RING_HEAD, ENGINE_READ(engine, RING_TAIL));
187 165
@@ -321,6 +299,39 @@ out: · 133 unmodified lines
321 return ret;299 return ret;
322}300}
323 301
302static void sanitize_hwsp(struct intel_engine_cs *engine)
303{
304 struct intel_timeline *tl;
305
306 list_for_each_entry(tl, &engine->status_page.timelines, engine_link)
307 intel_timeline_reset_seqno(tl);
308}
309
310static void xcs_sanitize(struct intel_engine_cs *engine)
311{
312 /*
313 * Poison residual state on resume, in case the suspend didn't!
314 *
315 * We have to assume that across suspend/resume (or other loss
316 * of control) that the contents of our pinned buffers has been
317 * lost, replaced by garbage. Since this doesn't always happen,
318 * let's poison such state so that we more quickly spot when
319 * we falsely assume it has been preserved.
320 */
321 if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
322 memset(engine->status_page.addr, POISON_INUSE, PAGE_SIZE);
323
324 /*
325 * The kernel_context HWSP is stored in the status_page. As above,
326 * that may be lost on resume/initialisation, and so we need to
327 * reset the value in the HWSP.
328 */
329 sanitize_hwsp(engine);
330
331 /* And scrub the dirty cachelines for the HWSP */
332 clflush_cache_range(engine->status_page.addr, PAGE_SIZE);
333}
334
324static void reset_prepare(struct intel_engine_cs *engine)335static void reset_prepare(struct intel_engine_cs *engine)
325{336{
326 struct intel_uncore *uncore = engine->uncore;337 struct intel_uncore *uncore = engine->uncore;
@@ -440,10 +451,8 @@ static void reset_cancel(struct intel_engine_cs *engine) · 113 unmodified lines
440 spin_lock_irqsave(&engine->active.lock, flags);451 spin_lock_irqsave(&engine->active.lock, flags);
441 452
442 /* Mark all submitted requests as skipped. */453 /* Mark all submitted requests as skipped. */
443 list_for_each_entry(request, &engine->active.requests, sched.link) {454 list_for_each_entry(request, &engine->active.requests, sched.link)
444 i915_request_set_error_once(request, -EIO);455 i915_request_mark_eio(request);
445 i915_request_mark_complete(request);
446 }
447 intel_engine_signal_breadcrumbs(engine);456 intel_engine_signal_breadcrumbs(engine);
448 457
449 /* Remaining _unready_ requests will be nop'ed when submitted */458 /* Remaining _unready_ requests will be nop'ed when submitted */
@@ -602,6 +611,7 @@ static int ring_context_pin(struct intel_context *ce, void *unused) · 152 unmodified lines
602static void ring_context_reset(struct intel_context *ce)611static void ring_context_reset(struct intel_context *ce)
603{612{
604 intel_ring_reset(ce->ring, ce->ring->emit);613 intel_ring_reset(ce->ring, ce->ring->emit);
614 clear_bit(CONTEXT_VALID_BIT, &ce->flags);
605}615}
606 616
607static const struct intel_context_ops ring_context_ops = {617static const struct intel_context_ops ring_context_ops = {
@@ -886,7 +896,8 @@ static int switch_context(struct i915_request *rq) · 278 unmodified lines
886 GEM_BUG_ON(HAS_EXECLISTS(engine->i915));896 GEM_BUG_ON(HAS_EXECLISTS(engine->i915));
887 897
888 if (engine->wa_ctx.vma && ce != engine->kernel_context) {898 if (engine->wa_ctx.vma && ce != engine->kernel_context) {
889 if (engine->wa_ctx.vma->private != ce) {899 if (engine->wa_ctx.vma->private != ce &&
900 i915_mitigate_clear_residuals()) {
890 ret = clear_residuals(rq);901 ret = clear_residuals(rq);
891 if (ret)902 if (ret)
892 return ret;903 return ret;
@@ -1069,6 +1080,8 @@ static void setup_common(struct intel_engine_cs *engine) · 176 unmodified lines
1069 setup_irq(engine);1080 setup_irq(engine);
1070 1081
1071 engine->resume = xcs_resume;1082 engine->resume = xcs_resume;
1083 engine->sanitize = xcs_sanitize;
1084
1072 engine->reset.prepare = reset_prepare;1085 engine->reset.prepare = reset_prepare;
1073 engine->reset.rewind = reset_rewind;1086 engine->reset.rewind = reset_rewind;
1074 engine->reset.cancel = reset_cancel;1087 engine->reset.cancel = reset_cancel;
@@ -1290,7 +1303,7 @@ int intel_ring_submission_setup(struct intel_engine_cs *engine) · 215 unmodified lines
1290 1303
1291 GEM_BUG_ON(timeline->hwsp_ggtt != engine->status_page.vma);1304 GEM_BUG_ON(timeline->hwsp_ggtt != engine->status_page.vma);
1292 1305
1293 if (IS_HASWELL(engine->i915) && engine->class == RENDER_CLASS) {1306 if (IS_GEN(engine->i915, 7) && engine->class == RENDER_CLASS) {
1294 err = gen7_ctx_switch_bb_init(engine);1307 err = gen7_ctx_switch_bb_init(engine);
1295 if (err)1308 if (err)
1296 goto err_ring_unpin;1309 goto err_ring_unpin;
drivers/gpu/drm/i915/gt/intel_rps.c+35−24
@@ -400,7 +400,7 @@ static unsigned int gen5_invert_freq(struct intel_rps *rps, · 399 unmodified lines
400 return val;400 return val;
401}401}
402 402
403static bool gen5_rps_set(struct intel_rps *rps, u8 val)403static int __gen5_rps_set(struct intel_rps *rps, u8 val)
404{404{
405 struct intel_uncore *uncore = rps_to_uncore(rps);405 struct intel_uncore *uncore = rps_to_uncore(rps);
406 u16 rgvswctl;406 u16 rgvswctl;
@@ -410,7 +410,7 @@ static bool gen5_rps_set(struct intel_rps *rps, u8 val) · 3 unmodified lines
410 rgvswctl = intel_uncore_read16(uncore, MEMSWCTL);410 rgvswctl = intel_uncore_read16(uncore, MEMSWCTL);
411 if (rgvswctl & MEMCTL_CMD_STS) {411 if (rgvswctl & MEMCTL_CMD_STS) {
412 DRM_DEBUG("gpu busy, RCS change rejected\n");412 DRM_DEBUG("gpu busy, RCS change rejected\n");
413 return false; /* still busy with another command */413 return -EBUSY; /* still busy with another command */
414 }414 }
415 415
416 /* Invert the frequency bin into an ips delay */416 /* Invert the frequency bin into an ips delay */
@@ -426,7 +426,18 @@ static bool gen5_rps_set(struct intel_rps *rps, u8 val) · 9 unmodified lines
426 rgvswctl |= MEMCTL_CMD_STS;426 rgvswctl |= MEMCTL_CMD_STS;
427 intel_uncore_write16(uncore, MEMSWCTL, rgvswctl);427 intel_uncore_write16(uncore, MEMSWCTL, rgvswctl);
428 428
429 return true;429 return 0;
430}
431
432static int gen5_rps_set(struct intel_rps *rps, u8 val)
433{
434 int err;
435
436 spin_lock_irq(&mchdev_lock);
437 err = __gen5_rps_set(rps, val);
438 spin_unlock_irq(&mchdev_lock);
439
440 return err;
430}441}
431 442
432static unsigned long intel_pxfreq(u32 vidfreq)443static unsigned long intel_pxfreq(u32 vidfreq)
@@ -557,7 +568,7 @@ static bool gen5_rps_enable(struct intel_rps *rps) · 124 unmodified lines
557 "stuck trying to change perf mode\n");568 "stuck trying to change perf mode\n");
558 mdelay(1);569 mdelay(1);
559 570
560 gen5_rps_set(rps, rps->cur_freq);571 __gen5_rps_set(rps, rps->cur_freq);
561 572
562 rps->ips.last_count1 = intel_uncore_read(uncore, DMIEC);573 rps->ips.last_count1 = intel_uncore_read(uncore, DMIEC);
563 rps->ips.last_count1 += intel_uncore_read(uncore, DDREC);574 rps->ips.last_count1 += intel_uncore_read(uncore, DDREC);
@@ -599,7 +610,7 @@ static void gen5_rps_disable(struct intel_rps *rps) · 35 unmodified lines
599 intel_uncore_write(uncore, MEMINTRSTS, MEMINT_EVAL_CHG);610 intel_uncore_write(uncore, MEMINTRSTS, MEMINT_EVAL_CHG);
600 611
601 /* Go back to the starting frequency */612 /* Go back to the starting frequency */
602 gen5_rps_set(rps, rps->idle_freq);613 __gen5_rps_set(rps, rps->idle_freq);
603 mdelay(1);614 mdelay(1);
604 rgvswctl |= MEMCTL_CMD_STS;615 rgvswctl |= MEMCTL_CMD_STS;
605 intel_uncore_write(uncore, MEMSWCTL, rgvswctl);616 intel_uncore_write(uncore, MEMSWCTL, rgvswctl);
@@ -797,20 +808,19 @@ static int rps_set(struct intel_rps *rps, u8 val, bool update) · 191 unmodified lines
797 struct drm_i915_private *i915 = rps_to_i915(rps);808 struct drm_i915_private *i915 = rps_to_i915(rps);
798 int err;809 int err;
799 810
800 if (INTEL_GEN(i915) < 6)
801 return 0;
802
803 if (val == rps->last_freq)811 if (val == rps->last_freq)
804 return 0;812 return 0;
805 813
806 if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))814 if (IS_VALLEYVIEW(i915) || IS_CHERRYVIEW(i915))
807 err = vlv_rps_set(rps, val);815 err = vlv_rps_set(rps, val);
808 else816 else if (INTEL_GEN(i915) >= 6)
809 err = gen6_rps_set(rps, val);817 err = gen6_rps_set(rps, val);
818 else
819 err = gen5_rps_set(rps, val);
810 if (err)820 if (err)
811 return err;821 return err;
812 822
813 if (update)823 if (update && INTEL_GEN(i915) >= 6)
814 gen6_rps_set_thresholds(rps, val);824 gen6_rps_set_thresholds(rps, val);
815 rps->last_freq = val;825 rps->last_freq = val;
816 826
@@ -852,6 +862,8 @@ void intel_rps_park(struct intel_rps *rps) · 35 unmodified lines
852{862{
853 int adj;863 int adj;
854 864
865 GEM_BUG_ON(atomic_read(&rps->num_waiters));
866
855 if (!intel_rps_clear_active(rps))867 if (!intel_rps_clear_active(rps))
856 return;868 return;
857 869
@@ -907,28 +919,27 @@ void intel_rps_park(struct intel_rps *rps) · 49 unmodified lines
907 919
908void intel_rps_boost(struct i915_request *rq)920void intel_rps_boost(struct i915_request *rq)
909{921{
910 struct intel_rps *rps = &READ_ONCE(rq->engine)->gt->rps;922 if (i915_request_signaled(rq) || i915_request_has_waitboost(rq))
911 unsigned long flags;
912
913 if (i915_request_signaled(rq) || !intel_rps_is_active(rps))
914 return;923 return;
915 924
916 /* Serializes with i915_request_retire() */925 /* Serializes with i915_request_retire() */
917 spin_lock_irqsave(&rq->lock, flags);926 if (!test_and_set_bit(I915_FENCE_FLAG_BOOST, &rq->fence.flags)) {
918 if (!i915_request_has_waitboost(rq) &&927 struct intel_rps *rps = &READ_ONCE(rq->engine)->gt->rps;
919 !dma_fence_is_signaled_locked(&rq->fence)) {928
920 set_bit(I915_FENCE_FLAG_BOOST, &rq->fence.flags);929 if (atomic_fetch_inc(&rps->num_waiters))
930 return;
931
932 if (!intel_rps_is_active(rps))
933 return;
921 934
922 GT_TRACE(rps_to_gt(rps), "boost fence:%llx:%llx\n",935 GT_TRACE(rps_to_gt(rps), "boost fence:%llx:%llx\n",
923 rq->fence.context, rq->fence.seqno);936 rq->fence.context, rq->fence.seqno);
924 937
925 if (!atomic_fetch_inc(&rps->num_waiters) &&938 if (READ_ONCE(rps->cur_freq) < rps->boost_freq)
926 READ_ONCE(rps->cur_freq) < rps->boost_freq)
927 schedule_work(&rps->work);939 schedule_work(&rps->work);
928 940
929 atomic_inc(&rps->boosts);941 WRITE_ONCE(rps->boosts, rps->boosts + 1); /* debug only */
930 }942 }
931 spin_unlock_irqrestore(&rq->lock, flags);
932}943}
933 944
934int intel_rps_set(struct intel_rps *rps, u8 val)945int intel_rps_set(struct intel_rps *rps, u8 val)
@@ -1798,7 +1809,7 @@ void gen5_rps_irq_handler(struct intel_rps *rps) · 863 unmodified lines
1798 rps->min_freq_softlimit,1809 rps->min_freq_softlimit,
1799 rps->max_freq_softlimit);1810 rps->max_freq_softlimit);
1800 1811
1801 if (new_freq != rps->cur_freq && gen5_rps_set(rps, new_freq))1812 if (new_freq != rps->cur_freq && !__gen5_rps_set(rps, new_freq))
1802 rps->cur_freq = new_freq;1813 rps->cur_freq = new_freq;
1803 1814
1804 spin_unlock(&mchdev_lock);1815 spin_unlock(&mchdev_lock);
@@ -2109,7 +2120,7 @@ bool i915_gpu_turbo_disable(void) · 304 unmodified lines
2109 2120
2110 spin_lock_irq(&mchdev_lock);2121 spin_lock_irq(&mchdev_lock);
2111 rps->max_freq_softlimit = rps->min_freq;2122 rps->max_freq_softlimit = rps->min_freq;
2112 ret = gen5_rps_set(&i915->gt.rps, rps->min_freq);2123 ret = !__gen5_rps_set(&i915->gt.rps, rps->min_freq);
2113 spin_unlock_irq(&mchdev_lock);2124 spin_unlock_irq(&mchdev_lock);
2114 2125
2115 drm_dev_put(&i915->drm);2126 drm_dev_put(&i915->drm);
drivers/gpu/drm/i915/gt/intel_rps_types.h+1−1
@@ -93,7 +93,7 @@ struct intel_rps { · 92 unmodified lines
93 } power;93 } power;
94 94
95 atomic_t num_waiters;95 atomic_t num_waiters;
96 atomic_t boosts;96 unsigned int boosts;
97 97
98 /* manual wa residency calculations */98 /* manual wa residency calculations */
99 struct intel_rps_ei ei;99 struct intel_rps_ei ei;
drivers/gpu/drm/i915/gt/intel_timeline.c+103−6
@@ -126,6 +126,10 @@ static void __rcu_cacheline_free(struct rcu_head *rcu) · 125 unmodified lines
126 struct intel_timeline_cacheline *cl =126 struct intel_timeline_cacheline *cl =
127 container_of(rcu, typeof(*cl), rcu);127 container_of(rcu, typeof(*cl), rcu);
128 128
129 /* Must wait until after all *rq->hwsp are complete before removing */
130 i915_gem_object_unpin_map(cl->hwsp->vma->obj);
131 __idle_hwsp_free(cl->hwsp, ptr_unmask_bits(cl->vaddr, CACHELINE_BITS));
132
129 i915_active_fini(&cl->active);133 i915_active_fini(&cl->active);
130 kfree(cl);134 kfree(cl);
131}135}
@@ -133,11 +137,6 @@ static void __rcu_cacheline_free(struct rcu_head *rcu) · 1 unmodified line
133static void __idle_cacheline_free(struct intel_timeline_cacheline *cl)137static void __idle_cacheline_free(struct intel_timeline_cacheline *cl)
134{138{
135 GEM_BUG_ON(!i915_active_is_idle(&cl->active));139 GEM_BUG_ON(!i915_active_is_idle(&cl->active));
136
137 i915_gem_object_unpin_map(cl->hwsp->vma->obj);
138 i915_vma_put(cl->hwsp->vma);
139 __idle_hwsp_free(cl->hwsp, ptr_unmask_bits(cl->vaddr, CACHELINE_BITS));
140
141 call_rcu(&cl->rcu, __rcu_cacheline_free);140 call_rcu(&cl->rcu, __rcu_cacheline_free);
142}141}
143 142
@@ -179,7 +178,6 @@ cacheline_alloc(struct intel_timeline_hwsp *hwsp, unsigned int cacheline) · 35 unmodified lines
179 return ERR_CAST(vaddr);178 return ERR_CAST(vaddr);
180 }179 }
181 180
182 i915_vma_get(hwsp->vma);
183 cl->hwsp = hwsp;181 cl->hwsp = hwsp;
184 cl->vaddr = page_pack_bits(vaddr, cacheline);182 cl->vaddr = page_pack_bits(vaddr, cacheline);
185 183
@@ -321,6 +319,25 @@ __intel_timeline_create(struct intel_gt *gt, · 135 unmodified lines
321 return timeline;319 return timeline;
322}320}
323 321
322struct intel_timeline *
323intel_timeline_create_from_engine(struct intel_engine_cs *engine,
324 unsigned int offset)
325{
326 struct i915_vma *hwsp = engine->status_page.vma;
327 struct intel_timeline *tl;
328
329 tl = __intel_timeline_create(engine->gt, hwsp, offset);
330 if (IS_ERR(tl))
331 return tl;
332
333 /* Borrow a nearby lock; we only create these timelines during init */
334 mutex_lock(&hwsp->vm->mutex);
335 list_add_tail(&tl->engine_link, &engine->status_page.timelines);
336 mutex_unlock(&hwsp->vm->mutex);
337
338 return tl;
339}
340
324void __intel_timeline_pin(struct intel_timeline *tl)341void __intel_timeline_pin(struct intel_timeline *tl)
325{342{
326 GEM_BUG_ON(!atomic_read(&tl->pin_count));343 GEM_BUG_ON(!atomic_read(&tl->pin_count));
@@ -617,6 +634,86 @@ void intel_gt_fini_timelines(struct intel_gt *gt) · 290 unmodified lines
617 GEM_BUG_ON(!list_empty(&timelines->hwsp_free_list));634 GEM_BUG_ON(!list_empty(&timelines->hwsp_free_list));
618}635}
619 636
637void intel_gt_show_timelines(struct intel_gt *gt,
638 struct drm_printer *m,
639 void (*show_request)(struct drm_printer *m,
640 const struct i915_request *rq,
641 const char *prefix,
642 int indent))
643{
644 struct intel_gt_timelines *timelines = &gt->timelines;
645 struct intel_timeline *tl, *tn;
646 LIST_HEAD(free);
647
648 spin_lock(&timelines->lock);
649 list_for_each_entry_safe(tl, tn, &timelines->active_list, link) {
650 unsigned long count, ready, inflight;
651 struct i915_request *rq, *rn;
652 struct dma_fence *fence;
653
654 if (!mutex_trylock(&tl->mutex)) {
655 drm_printf(m, "Timeline %llx: busy; skipping\n",
656 tl->fence_context);
657 continue;
658 }
659
660 intel_timeline_get(tl);
661 GEM_BUG_ON(!atomic_read(&tl->active_count));
662 atomic_inc(&tl->active_count); /* pin the list element */
663 spin_unlock(&timelines->lock);
664
665 count = 0;
666 ready = 0;
667 inflight = 0;
668 list_for_each_entry_safe(rq, rn, &tl->requests, link) {
669 if (i915_request_completed(rq))
670 continue;
671
672 count++;
673 if (i915_request_is_ready(rq))
674 ready++;
675 if (i915_request_is_active(rq))
676 inflight++;
677 }
678
679 drm_printf(m, "Timeline %llx: { ", tl->fence_context);
680 drm_printf(m, "count: %lu, ready: %lu, inflight: %lu",
681 count, ready, inflight);
682 drm_printf(m, ", seqno: { current: %d, last: %d }",
683 *tl->hwsp_seqno, tl->seqno);
684 fence = i915_active_fence_get(&tl->last_request);
685 if (fence) {
686 drm_printf(m, ", engine: %s",
687 to_request(fence)->engine->name);
688 dma_fence_put(fence);
689 }
690 drm_printf(m, " }\n");
691
692 if (show_request) {
693 list_for_each_entry_safe(rq, rn, &tl->requests, link)
694 show_request(m, rq, "", 2);
695 }
696
697 mutex_unlock(&tl->mutex);
698 spin_lock(&timelines->lock);
699
700 /* Resume list iteration after reacquiring spinlock */
701 list_safe_reset_next(tl, tn, link);
702 if (atomic_dec_and_test(&tl->active_count))
703 list_del(&tl->link);
704
705 /* Defer the final release to after the spinlock */
706 if (refcount_dec_and_test(&tl->kref.refcount)) {
707 GEM_BUG_ON(atomic_read(&tl->active_count));
708 list_add(&tl->link, &free);
709 }
710 }
711 spin_unlock(&timelines->lock);
712
713 list_for_each_entry_safe(tl, tn, &free, link)
714 __intel_timeline_free(&tl->kref);
715}
716
620#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)717#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
621#include "gt/selftests/mock_timeline.c"718#include "gt/selftests/mock_timeline.c"
622#include "gt/selftest_timeline.c"719#include "gt/selftest_timeline.c"
drivers/gpu/drm/i915/gt/intel_timeline.h+18−7
@@ -31,6 +31,8 @@ · 30 unmodified lines
31#include "i915_syncmap.h"31#include "i915_syncmap.h"
32#include "intel_timeline_types.h"32#include "intel_timeline_types.h"
33 33
34struct drm_printer;
35
34struct intel_timeline *36struct intel_timeline *
35__intel_timeline_create(struct intel_gt *gt,37__intel_timeline_create(struct intel_gt *gt,
36 struct i915_vma *global_hwsp,38 struct i915_vma *global_hwsp,
@@ -42,14 +44,9 @@ intel_timeline_create(struct intel_gt *gt) · 5 unmodified lines
42 return __intel_timeline_create(gt, NULL, 0);44 return __intel_timeline_create(gt, NULL, 0);
43}45}
44 46
45static inline struct intel_timeline *47struct intel_timeline *
46intel_timeline_create_from_engine(struct intel_engine_cs *engine,48intel_timeline_create_from_engine(struct intel_engine_cs *engine,
47 unsigned int offset)49 unsigned int offset);
48{
49 return __intel_timeline_create(engine->gt,
50 engine->status_page.vma,
51 offset);
52}
53 50
54static inline struct intel_timeline *51static inline struct intel_timeline *
55intel_timeline_get(struct intel_timeline *timeline)52intel_timeline_get(struct intel_timeline *timeline)
@@ -106,4 +103,18 @@ int intel_timeline_read_hwsp(struct i915_request *from, · 50 unmodified lines
106void intel_gt_init_timelines(struct intel_gt *gt);103void intel_gt_init_timelines(struct intel_gt *gt);
107void intel_gt_fini_timelines(struct intel_gt *gt);104void intel_gt_fini_timelines(struct intel_gt *gt);
108 105
106void intel_gt_show_timelines(struct intel_gt *gt,
107 struct drm_printer *m,
108 void (*show_request)(struct drm_printer *m,
109 const struct i915_request *rq,
110 const char *prefix,
111 int indent));
112
113static inline bool
114intel_timeline_is_last(const struct intel_timeline *tl,
115 const struct i915_request *rq)
116{
117 return list_is_last_rcu(&rq->link, &tl->requests);
118}
119
109#endif120#endif
drivers/gpu/drm/i915/gt/intel_timeline_types.h+2−0
@@ -84,6 +84,8 @@ struct intel_timeline { · 83 unmodified lines
84 struct list_head link;84 struct list_head link;
85 struct intel_gt *gt;85 struct intel_gt *gt;
86 86
87 struct list_head engine_link;
88
87 struct kref kref;89 struct kref kref;
88 struct rcu_head rcu;90 struct rcu_head rcu;
89};91};
drivers/gpu/drm/i915/gt/intel_workarounds.c+309−335
@@ -7,6 +7,7 @@ · 6 unmodified lines
7#include "i915_drv.h"7#include "i915_drv.h"
8#include "intel_context.h"8#include "intel_context.h"
9#include "intel_engine_pm.h"9#include "intel_engine_pm.h"
10#include "intel_gpu_commands.h"
10#include "intel_gt.h"11#include "intel_gt.h"
11#include "intel_ring.h"12#include "intel_ring.h"
12#include "intel_workarounds.h"13#include "intel_workarounds.h"
@@ -194,7 +195,7 @@ static void wa_add(struct i915_wa_list *wal, i915_reg_t reg, · 181 unmodified lines
194}195}
195 196
196static void197static void
197wa_write_masked_or(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set)198wa_write_clr_set(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set)
198{199{
199 wa_add(wal, reg, clear, set, clear);200 wa_add(wal, reg, clear, set, clear);
200}201}
@@ -202,21 +203,32 @@ wa_write_masked_or(struct i915_wa_list *wal, i915_reg_t reg, u32 clear, u32 set) · 1 unmodified line
202static void203static void
203wa_write(struct i915_wa_list *wal, i915_reg_t reg, u32 set)204wa_write(struct i915_wa_list *wal, i915_reg_t reg, u32 set)
204{205{
205 wa_write_masked_or(wal, reg, ~0, set);206 wa_write_clr_set(wal, reg, ~0, set);
206}207}
207 208
208static void209static void
209wa_write_or(struct i915_wa_list *wal, i915_reg_t reg, u32 set)210wa_write_or(struct i915_wa_list *wal, i915_reg_t reg, u32 set)
210{211{
211 wa_write_masked_or(wal, reg, set, set);212 wa_write_clr_set(wal, reg, set, set);
212}213}
213 214
214static void215static void
215wa_write_clr(struct i915_wa_list *wal, i915_reg_t reg, u32 clr)216wa_write_clr(struct i915_wa_list *wal, i915_reg_t reg, u32 clr)
216{217{
217 wa_write_masked_or(wal, reg, clr, 0);218 wa_write_clr_set(wal, reg, clr, 0);
218}219}
219 220
221/*
222 * WA operations on "masked register". A masked register has the upper 16 bits
223 * documented as "masked" in b-spec. Its purpose is to allow writing to just a
224 * portion of the register without a rmw: you simply write in the upper 16 bits
225 * the mask of bits you are going to modify.
226 *
227 * The wa_masked_* family of functions already does the necessary operations to
228 * calculate the mask based on the parameters passed, so user only has to
229 * provide the lower 16 bits of that register.
230 */
231
220static void232static void
221wa_masked_en(struct i915_wa_list *wal, i915_reg_t reg, u32 val)233wa_masked_en(struct i915_wa_list *wal, i915_reg_t reg, u32 val)
222{234{
@@ -229,38 +241,36 @@ wa_masked_dis(struct i915_wa_list *wal, i915_reg_t reg, u32 val) · 6 unmodified lines
229 wa_add(wal, reg, 0, _MASKED_BIT_DISABLE(val), val);241 wa_add(wal, reg, 0, _MASKED_BIT_DISABLE(val), val);
230}242}
231 243
232#define WA_SET_BIT_MASKED(addr, mask) \244static void
233 wa_masked_en(wal, (addr), (mask))245wa_masked_field_set(struct i915_wa_list *wal, i915_reg_t reg,
234 246 u32 mask, u32 val)
235#define WA_CLR_BIT_MASKED(addr, mask) \247{
236 wa_masked_dis(wal, (addr), (mask))248 wa_add(wal, reg, 0, _MASKED_FIELD(mask, val), mask);
237 249}
238#define WA_SET_FIELD_MASKED(addr, mask, value) \
239 wa_write_masked_or(wal, (addr), 0, _MASKED_FIELD((mask), (value)))
240 250
241static void gen6_ctx_workarounds_init(struct intel_engine_cs *engine,251static void gen6_ctx_workarounds_init(struct intel_engine_cs *engine,
242 struct i915_wa_list *wal)252 struct i915_wa_list *wal)
243{253{
244 WA_SET_BIT_MASKED(INSTPM, INSTPM_FORCE_ORDERING);254 wa_masked_en(wal, INSTPM, INSTPM_FORCE_ORDERING);
245}255}
246 256
247static void gen7_ctx_workarounds_init(struct intel_engine_cs *engine,257static void gen7_ctx_workarounds_init(struct intel_engine_cs *engine,
248 struct i915_wa_list *wal)258 struct i915_wa_list *wal)
249{259{
250 WA_SET_BIT_MASKED(INSTPM, INSTPM_FORCE_ORDERING);260 wa_masked_en(wal, INSTPM, INSTPM_FORCE_ORDERING);
251}261}
252 262
253static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine,263static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine,
254 struct i915_wa_list *wal)264 struct i915_wa_list *wal)
255{265{
256 WA_SET_BIT_MASKED(INSTPM, INSTPM_FORCE_ORDERING);266 wa_masked_en(wal, INSTPM, INSTPM_FORCE_ORDERING);
257 267
258 /* WaDisableAsyncFlipPerfMode:bdw,chv */268 /* WaDisableAsyncFlipPerfMode:bdw,chv */
259 WA_SET_BIT_MASKED(MI_MODE, ASYNC_FLIP_PERF_DISABLE);269 wa_masked_en(wal, MI_MODE, ASYNC_FLIP_PERF_DISABLE);
260 270
261 /* WaDisablePartialInstShootdown:bdw,chv */271 /* WaDisablePartialInstShootdown:bdw,chv */
262 WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN,272 wa_masked_en(wal, GEN8_ROW_CHICKEN,
263 PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);273 PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);
264 274
265 /* Use Force Non-Coherent whenever executing a 3D context. This is a275 /* Use Force Non-Coherent whenever executing a 3D context. This is a
266 * workaround for for a possible hang in the unlikely event a TLB276 * workaround for for a possible hang in the unlikely event a TLB
@@ -268,9 +278,9 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, · 1 unmodified line
268 */278 */
269 /* WaForceEnableNonCoherent:bdw,chv */279 /* WaForceEnableNonCoherent:bdw,chv */
270 /* WaHdcDisableFetchWhenMasked:bdw,chv */280 /* WaHdcDisableFetchWhenMasked:bdw,chv */
271 WA_SET_BIT_MASKED(HDC_CHICKEN0,281 wa_masked_en(wal, HDC_CHICKEN0,
272 HDC_DONOT_FETCH_MEM_WHEN_MASKED |282 HDC_DONOT_FETCH_MEM_WHEN_MASKED |
273 HDC_FORCE_NON_COHERENT);283 HDC_FORCE_NON_COHERENT);
274 284
275 /* From the Haswell PRM, Command Reference: Registers, CACHE_MODE_0:285 /* From the Haswell PRM, Command Reference: Registers, CACHE_MODE_0:
276 * "The Hierarchical Z RAW Stall Optimization allows non-overlapping286 * "The Hierarchical Z RAW Stall Optimization allows non-overlapping
@@ -280,10 +290,10 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, · 3 unmodified lines
280 *290 *
281 * This optimization is off by default for BDW and CHV; turn it on.291 * This optimization is off by default for BDW and CHV; turn it on.
282 */292 */
283 WA_CLR_BIT_MASKED(CACHE_MODE_0_GEN7, HIZ_RAW_STALL_OPT_DISABLE);293 wa_masked_dis(wal, CACHE_MODE_0_GEN7, HIZ_RAW_STALL_OPT_DISABLE);
284 294
285 /* Wa4x4STCOptimizationDisable:bdw,chv */295 /* Wa4x4STCOptimizationDisable:bdw,chv */
286 WA_SET_BIT_MASKED(CACHE_MODE_1, GEN8_4x4_STC_OPTIMIZATION_DISABLE);296 wa_masked_en(wal, CACHE_MODE_1, GEN8_4x4_STC_OPTIMIZATION_DISABLE);
287 297
288 /*298 /*
289 * BSpec recommends 8x4 when MSAA is used,299 * BSpec recommends 8x4 when MSAA is used,
@@ -293,7 +303,7 @@ static void gen8_ctx_workarounds_init(struct intel_engine_cs *engine, · 3 unmodified lines
293 * disable bit, which we don't touch here, but it's good303 * disable bit, which we don't touch here, but it's good
294 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).304 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
295 */305 */
296 WA_SET_FIELD_MASKED(GEN7_GT_MODE,306 wa_masked_field_set(wal, GEN7_GT_MODE,
297 GEN6_WIZ_HASHING_MASK,307 GEN6_WIZ_HASHING_MASK,
298 GEN6_WIZ_HASHING_16x4);308 GEN6_WIZ_HASHING_16x4);
299}309}
@@ -306,24 +316,24 @@ static void bdw_ctx_workarounds_init(struct intel_engine_cs *engine, · 6 unmodified lines
306 gen8_ctx_workarounds_init(engine, wal);316 gen8_ctx_workarounds_init(engine, wal);
307 317
308 /* WaDisableThreadStallDopClockGating:bdw (pre-production) */318 /* WaDisableThreadStallDopClockGating:bdw (pre-production) */
309 WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);319 wa_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);
310 320
311 /* WaDisableDopClockGating:bdw321 /* WaDisableDopClockGating:bdw
312 *322 *
313 * Also see the related UCGTCL1 write in bdw_init_clock_gating()323 * Also see the related UCGTCL1 write in bdw_init_clock_gating()
314 * to disable EUTC clock gating.324 * to disable EUTC clock gating.
315 */325 */
316 WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,326 wa_masked_en(wal, GEN7_ROW_CHICKEN2,
317 DOP_CLOCK_GATING_DISABLE);327 DOP_CLOCK_GATING_DISABLE);
318 328
319 WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN3,329 wa_masked_en(wal, HALF_SLICE_CHICKEN3,
320 GEN8_SAMPLER_POWER_BYPASS_DIS);330 GEN8_SAMPLER_POWER_BYPASS_DIS);
321 331
322 WA_SET_BIT_MASKED(HDC_CHICKEN0,332 wa_masked_en(wal, HDC_CHICKEN0,
323 /* WaForceContextSaveRestoreNonCoherent:bdw */333 /* WaForceContextSaveRestoreNonCoherent:bdw */
324 HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |334 HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |
325 /* WaDisableFenceDestinationToSLM:bdw (pre-prod) */335 /* WaDisableFenceDestinationToSLM:bdw (pre-prod) */
326 (IS_BDW_GT3(i915) ? HDC_FENCE_DEST_SLM_DISABLE : 0));336 (IS_BDW_GT3(i915) ? HDC_FENCE_DEST_SLM_DISABLE : 0));
327}337}
328 338
329static void chv_ctx_workarounds_init(struct intel_engine_cs *engine,339static void chv_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -332,10 +342,10 @@ static void chv_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
332 gen8_ctx_workarounds_init(engine, wal);342 gen8_ctx_workarounds_init(engine, wal);
333 343
334 /* WaDisableThreadStallDopClockGating:chv */344 /* WaDisableThreadStallDopClockGating:chv */
335 WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);345 wa_masked_en(wal, GEN8_ROW_CHICKEN, STALL_DOP_GATING_DISABLE);
336 346
337 /* Improve HiZ throughput on CHV. */347 /* Improve HiZ throughput on CHV. */
338 WA_SET_BIT_MASKED(HIZ_CHICKEN, CHV_HZ_8X8_MODE_IN_1X);348 wa_masked_en(wal, HIZ_CHICKEN, CHV_HZ_8X8_MODE_IN_1X);
339}349}
340 350
341static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine,351static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -349,38 +359,38 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, · 7 unmodified lines
349 * Must match Display Engine. See359 * Must match Display Engine. See
350 * WaCompressedResourceDisplayNewHashMode.360 * WaCompressedResourceDisplayNewHashMode.
351 */361 */
352 WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,362 wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
353 GEN9_PBE_COMPRESSED_HASH_SELECTION);363 GEN9_PBE_COMPRESSED_HASH_SELECTION);
354 WA_SET_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN7,364 wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7,
355 GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR);365 GEN9_SAMPLER_HASH_COMPRESSED_READ_ADDR);
356 }366 }
357 367
358 /* WaClearFlowControlGpgpuContextSave:skl,bxt,kbl,glk,cfl */368 /* WaClearFlowControlGpgpuContextSave:skl,bxt,kbl,glk,cfl */
359 /* WaDisablePartialInstShootdown:skl,bxt,kbl,glk,cfl */369 /* WaDisablePartialInstShootdown:skl,bxt,kbl,glk,cfl */
360 WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN,370 wa_masked_en(wal, GEN8_ROW_CHICKEN,
361 FLOW_CONTROL_ENABLE |371 FLOW_CONTROL_ENABLE |
362 PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);372 PARTIAL_INSTRUCTION_SHOOTDOWN_DISABLE);
363 373
364 /* WaEnableYV12BugFixInHalfSliceChicken7:skl,bxt,kbl,glk,cfl */374 /* WaEnableYV12BugFixInHalfSliceChicken7:skl,bxt,kbl,glk,cfl */
365 /* WaEnableSamplerGPGPUPreemptionSupport:skl,bxt,kbl,cfl */375 /* WaEnableSamplerGPGPUPreemptionSupport:skl,bxt,kbl,cfl */
366 WA_SET_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN7,376 wa_masked_en(wal, GEN9_HALF_SLICE_CHICKEN7,
367 GEN9_ENABLE_YV12_BUGFIX |377 GEN9_ENABLE_YV12_BUGFIX |
368 GEN9_ENABLE_GPGPU_PREEMPTION);378 GEN9_ENABLE_GPGPU_PREEMPTION);
369 379
370 /* Wa4x4STCOptimizationDisable:skl,bxt,kbl,glk,cfl */380 /* Wa4x4STCOptimizationDisable:skl,bxt,kbl,glk,cfl */
371 /* WaDisablePartialResolveInVc:skl,bxt,kbl,cfl */381 /* WaDisablePartialResolveInVc:skl,bxt,kbl,cfl */
372 WA_SET_BIT_MASKED(CACHE_MODE_1,382 wa_masked_en(wal, CACHE_MODE_1,
373 GEN8_4x4_STC_OPTIMIZATION_DISABLE |383 GEN8_4x4_STC_OPTIMIZATION_DISABLE |
374 GEN9_PARTIAL_RESOLVE_IN_VC_DISABLE);384 GEN9_PARTIAL_RESOLVE_IN_VC_DISABLE);
375 385
376 /* WaCcsTlbPrefetchDisable:skl,bxt,kbl,glk,cfl */386 /* WaCcsTlbPrefetchDisable:skl,bxt,kbl,glk,cfl */
377 WA_CLR_BIT_MASKED(GEN9_HALF_SLICE_CHICKEN5,387 wa_masked_dis(wal, GEN9_HALF_SLICE_CHICKEN5,
378 GEN9_CCS_TLB_PREFETCH_ENABLE);388 GEN9_CCS_TLB_PREFETCH_ENABLE);
379 389
380 /* WaForceContextSaveRestoreNonCoherent:skl,bxt,kbl,cfl */390 /* WaForceContextSaveRestoreNonCoherent:skl,bxt,kbl,cfl */
381 WA_SET_BIT_MASKED(HDC_CHICKEN0,391 wa_masked_en(wal, HDC_CHICKEN0,
382 HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |392 HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT |
383 HDC_FORCE_CSR_NON_COHERENT_OVR_DISABLE);393 HDC_FORCE_CSR_NON_COHERENT_OVR_DISABLE);
384 394
385 /* WaForceEnableNonCoherent and WaDisableHDCInvalidation are395 /* WaForceEnableNonCoherent and WaDisableHDCInvalidation are
386 * both tied to WaForceContextSaveRestoreNonCoherent396 * both tied to WaForceContextSaveRestoreNonCoherent
@@ -396,19 +406,19 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, · 9 unmodified lines
396 */406 */
397 407
398 /* WaForceEnableNonCoherent:skl,bxt,kbl,cfl */408 /* WaForceEnableNonCoherent:skl,bxt,kbl,cfl */
399 WA_SET_BIT_MASKED(HDC_CHICKEN0,409 wa_masked_en(wal, HDC_CHICKEN0,
400 HDC_FORCE_NON_COHERENT);410 HDC_FORCE_NON_COHERENT);
401 411
402 /* WaDisableSamplerPowerBypassForSOPingPong:skl,bxt,kbl,cfl */412 /* WaDisableSamplerPowerBypassForSOPingPong:skl,bxt,kbl,cfl */
403 if (IS_SKYLAKE(i915) ||413 if (IS_SKYLAKE(i915) ||
404 IS_KABYLAKE(i915) ||414 IS_KABYLAKE(i915) ||
405 IS_COFFEELAKE(i915) ||415 IS_COFFEELAKE(i915) ||
406 IS_COMETLAKE(i915))416 IS_COMETLAKE(i915))
407 WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN3,417 wa_masked_en(wal, HALF_SLICE_CHICKEN3,
408 GEN8_SAMPLER_POWER_BYPASS_DIS);418 GEN8_SAMPLER_POWER_BYPASS_DIS);
409 419
410 /* WaDisableSTUnitPowerOptimization:skl,bxt,kbl,glk,cfl */420 /* WaDisableSTUnitPowerOptimization:skl,bxt,kbl,glk,cfl */
411 WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN2, GEN8_ST_PO_DISABLE);421 wa_masked_en(wal, HALF_SLICE_CHICKEN2, GEN8_ST_PO_DISABLE);
412 422
413 /*423 /*
414 * Supporting preemption with fine-granularity requires changes in the424 * Supporting preemption with fine-granularity requires changes in the
@@ -422,16 +432,16 @@ static void gen9_ctx_workarounds_init(struct intel_engine_cs *engine, · 7 unmodified lines
422 */432 */
423 433
424 /* WaDisable3DMidCmdPreemption:skl,bxt,glk,cfl,[cnl] */434 /* WaDisable3DMidCmdPreemption:skl,bxt,glk,cfl,[cnl] */
425 WA_CLR_BIT_MASKED(GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);435 wa_masked_dis(wal, GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);
426 436
427 /* WaDisableGPGPUMidCmdPreemption:skl,bxt,blk,cfl,[cnl] */437 /* WaDisableGPGPUMidCmdPreemption:skl,bxt,blk,cfl,[cnl] */
428 WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,438 wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
429 GEN9_PREEMPT_GPGPU_LEVEL_MASK,439 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
430 GEN9_PREEMPT_GPGPU_COMMAND_LEVEL);440 GEN9_PREEMPT_GPGPU_COMMAND_LEVEL);
431 441
432 /* WaClearHIZ_WM_CHICKEN3:bxt,glk */442 /* WaClearHIZ_WM_CHICKEN3:bxt,glk */
433 if (IS_GEN9_LP(i915))443 if (IS_GEN9_LP(i915))
434 WA_SET_BIT_MASKED(GEN9_WM_CHICKEN3, GEN9_FACTOR_IN_CLR_VAL_HIZ);444 wa_masked_en(wal, GEN9_WM_CHICKEN3, GEN9_FACTOR_IN_CLR_VAL_HIZ);
435}445}
436 446
437static void skl_tune_iz_hashing(struct intel_engine_cs *engine,447static void skl_tune_iz_hashing(struct intel_engine_cs *engine,
@@ -465,7 +475,7 @@ static void skl_tune_iz_hashing(struct intel_engine_cs *engine, · 27 unmodified lines
465 return;475 return;
466 476
467 /* Tune IZ hashing. See intel_device_info_runtime_init() */477 /* Tune IZ hashing. See intel_device_info_runtime_init() */
468 WA_SET_FIELD_MASKED(GEN7_GT_MODE,478 wa_masked_field_set(wal, GEN7_GT_MODE,
469 GEN9_IZ_HASHING_MASK(2) |479 GEN9_IZ_HASHING_MASK(2) |
470 GEN9_IZ_HASHING_MASK(1) |480 GEN9_IZ_HASHING_MASK(1) |
471 GEN9_IZ_HASHING_MASK(0),481 GEN9_IZ_HASHING_MASK(0),
@@ -487,12 +497,12 @@ static void bxt_ctx_workarounds_init(struct intel_engine_cs *engine, · 15 unmodified lines
487 gen9_ctx_workarounds_init(engine, wal);497 gen9_ctx_workarounds_init(engine, wal);
488 498
489 /* WaDisableThreadStallDopClockGating:bxt */499 /* WaDisableThreadStallDopClockGating:bxt */
490 WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN,500 wa_masked_en(wal, GEN8_ROW_CHICKEN,
491 STALL_DOP_GATING_DISABLE);501 STALL_DOP_GATING_DISABLE);
492 502
493 /* WaToEnableHwFixForPushConstHWBug:bxt */503 /* WaToEnableHwFixForPushConstHWBug:bxt */
494 WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,504 wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
495 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);505 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
496}506}
497 507
498static void kbl_ctx_workarounds_init(struct intel_engine_cs *engine,508static void kbl_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -504,12 +514,12 @@ static void kbl_ctx_workarounds_init(struct intel_engine_cs *engine, · 5 unmodified lines
504 514
505 /* WaToEnableHwFixForPushConstHWBug:kbl */515 /* WaToEnableHwFixForPushConstHWBug:kbl */
506 if (IS_KBL_GT_REVID(i915, KBL_REVID_C0, REVID_FOREVER))516 if (IS_KBL_GT_REVID(i915, KBL_REVID_C0, REVID_FOREVER))
507 WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,517 wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
508 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);518 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
509 519
510 /* WaDisableSbeCacheDispatchPortSharing:kbl */520 /* WaDisableSbeCacheDispatchPortSharing:kbl */
511 WA_SET_BIT_MASKED(GEN7_HALF_SLICE_CHICKEN1,521 wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1,
512 GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);522 GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);
513}523}
514 524
515static void glk_ctx_workarounds_init(struct intel_engine_cs *engine,525static void glk_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -518,8 +528,8 @@ static void glk_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
518 gen9_ctx_workarounds_init(engine, wal);528 gen9_ctx_workarounds_init(engine, wal);
519 529
520 /* WaToEnableHwFixForPushConstHWBug:glk */530 /* WaToEnableHwFixForPushConstHWBug:glk */
521 WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,531 wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
522 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);532 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
523}533}
524 534
525static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine,535static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -528,41 +538,41 @@ static void cfl_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
528 gen9_ctx_workarounds_init(engine, wal);538 gen9_ctx_workarounds_init(engine, wal);
529 539
530 /* WaToEnableHwFixForPushConstHWBug:cfl */540 /* WaToEnableHwFixForPushConstHWBug:cfl */
531 WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,541 wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
532 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);542 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
533 543
534 /* WaDisableSbeCacheDispatchPortSharing:cfl */544 /* WaDisableSbeCacheDispatchPortSharing:cfl */
535 WA_SET_BIT_MASKED(GEN7_HALF_SLICE_CHICKEN1,545 wa_masked_en(wal, GEN7_HALF_SLICE_CHICKEN1,
536 GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);546 GEN7_SBE_SS_CACHE_DISPATCH_PORT_SHARING_DISABLE);
537}547}
538 548
539static void cnl_ctx_workarounds_init(struct intel_engine_cs *engine,549static void cnl_ctx_workarounds_init(struct intel_engine_cs *engine,
540 struct i915_wa_list *wal)550 struct i915_wa_list *wal)
541{551{
542 /* WaForceContextSaveRestoreNonCoherent:cnl */552 /* WaForceContextSaveRestoreNonCoherent:cnl */
543 WA_SET_BIT_MASKED(CNL_HDC_CHICKEN0,553 wa_masked_en(wal, CNL_HDC_CHICKEN0,
544 HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT);554 HDC_FORCE_CONTEXT_SAVE_RESTORE_NON_COHERENT);
545 555
546 /* WaDisableReplayBufferBankArbitrationOptimization:cnl */556 /* WaDisableReplayBufferBankArbitrationOptimization:cnl */
547 WA_SET_BIT_MASKED(COMMON_SLICE_CHICKEN2,557 wa_masked_en(wal, COMMON_SLICE_CHICKEN2,
548 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);558 GEN8_SBE_DISABLE_REPLAY_BUF_OPTIMIZATION);
549 559
550 /* WaPushConstantDereferenceHoldDisable:cnl */560 /* WaPushConstantDereferenceHoldDisable:cnl */
551 WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2, PUSH_CONSTANT_DEREF_DISABLE);561 wa_masked_en(wal, GEN7_ROW_CHICKEN2, PUSH_CONSTANT_DEREF_DISABLE);
552 562
553 /* FtrEnableFastAnisoL1BankingFix:cnl */563 /* FtrEnableFastAnisoL1BankingFix:cnl */
554 WA_SET_BIT_MASKED(HALF_SLICE_CHICKEN3, CNL_FAST_ANISO_L1_BANKING_FIX);564 wa_masked_en(wal, HALF_SLICE_CHICKEN3, CNL_FAST_ANISO_L1_BANKING_FIX);
555 565
556 /* WaDisable3DMidCmdPreemption:cnl */566 /* WaDisable3DMidCmdPreemption:cnl */
557 WA_CLR_BIT_MASKED(GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);567 wa_masked_dis(wal, GEN8_CS_CHICKEN1, GEN9_PREEMPT_3D_OBJECT_LEVEL);
558 568
559 /* WaDisableGPGPUMidCmdPreemption:cnl */569 /* WaDisableGPGPUMidCmdPreemption:cnl */
560 WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,570 wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
561 GEN9_PREEMPT_GPGPU_LEVEL_MASK,571 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
562 GEN9_PREEMPT_GPGPU_COMMAND_LEVEL);572 GEN9_PREEMPT_GPGPU_COMMAND_LEVEL);
563 573
564 /* WaDisableEarlyEOT:cnl */574 /* WaDisableEarlyEOT:cnl */
565 WA_SET_BIT_MASKED(GEN8_ROW_CHICKEN, DISABLE_EARLY_EOT);575 wa_masked_en(wal, GEN8_ROW_CHICKEN, DISABLE_EARLY_EOT);
566}576}
567 577
568static void icl_ctx_workarounds_init(struct intel_engine_cs *engine,578static void icl_ctx_workarounds_init(struct intel_engine_cs *engine,
@@ -580,8 +590,8 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, · 11 unmodified lines
580 * Formerly known as WaPushConstantDereferenceHoldDisable590 * Formerly known as WaPushConstantDereferenceHoldDisable
581 */591 */
582 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_B0))592 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_B0))
583 WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,593 wa_masked_en(wal, GEN7_ROW_CHICKEN2,
584 PUSH_CONSTANT_DEREF_DISABLE);594 PUSH_CONSTANT_DEREF_DISABLE);
585 595
586 /* WaForceEnableNonCoherent:icl596 /* WaForceEnableNonCoherent:icl
587 * This is not the same workaround as in early Gen9 platforms, where597 * This is not the same workaround as in early Gen9 platforms, where
@@ -590,40 +600,40 @@ static void icl_ctx_workarounds_init(struct intel_engine_cs *engine, · 2 unmodified lines
590 * (the register is whitelisted in hardware now, so UMDs can opt in600 * (the register is whitelisted in hardware now, so UMDs can opt in
591 * for coherency if they have a good reason).601 * for coherency if they have a good reason).
592 */602 */
593 WA_SET_BIT_MASKED(ICL_HDC_MODE, HDC_FORCE_NON_COHERENT);603 wa_masked_en(wal, ICL_HDC_MODE, HDC_FORCE_NON_COHERENT);
594 604
595 /* Wa_2006611047:icl (pre-prod)605 /* Wa_2006611047:icl (pre-prod)
596 * Formerly known as WaDisableImprovedTdlClkGating606 * Formerly known as WaDisableImprovedTdlClkGating
597 */607 */
598 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))608 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))
599 WA_SET_BIT_MASKED(GEN7_ROW_CHICKEN2,609 wa_masked_en(wal, GEN7_ROW_CHICKEN2,
600 GEN11_TDL_CLOCK_GATING_FIX_DISABLE);610 GEN11_TDL_CLOCK_GATING_FIX_DISABLE);
601 611
602 /* Wa_2006665173:icl (pre-prod) */612 /* Wa_2006665173:icl (pre-prod) */
603 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))613 if (IS_ICL_REVID(i915, ICL_REVID_A0, ICL_REVID_A0))
604 WA_SET_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,614 wa_masked_en(wal, GEN11_COMMON_SLICE_CHICKEN3,
605 GEN11_BLEND_EMB_FIX_DISABLE_IN_RCC);615 GEN11_BLEND_EMB_FIX_DISABLE_IN_RCC);
606 616
607 /* WaEnableFloatBlendOptimization:icl */617 /* WaEnableFloatBlendOptimization:icl */
608 wa_write_masked_or(wal,618 wa_write_clr_set(wal,
609 GEN10_CACHE_MODE_SS,619 GEN10_CACHE_MODE_SS,
610 0, /* write-only, so skip validation */620 0, /* write-only, so skip validation */
611 _MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE));621 _MASKED_BIT_ENABLE(FLOAT_BLEND_OPTIMIZATION_ENABLE));
612 622
613 /* WaDisableGPGPUMidThreadPreemption:icl */623 /* WaDisableGPGPUMidThreadPreemption:icl */
614 WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,624 wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
615 GEN9_PREEMPT_GPGPU_LEVEL_MASK,625 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
616 GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);626 GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);
617 627
618 /* allow headerless messages for preemptible GPGPU context */628 /* allow headerless messages for preemptible GPGPU context */
619 WA_SET_BIT_MASKED(GEN10_SAMPLER_MODE,629 wa_masked_en(wal, GEN10_SAMPLER_MODE,
620 GEN11_SAMPLER_ENABLE_HEADLESS_MSG);630 GEN11_SAMPLER_ENABLE_HEADLESS_MSG);
621 631
622 /* Wa_1604278689:icl,ehl */632 /* Wa_1604278689:icl,ehl */
623 wa_write(wal, IVB_FBC_RT_BASE, 0xFFFFFFFF & ~ILK_FBC_RT_VALID);633 wa_write(wal, IVB_FBC_RT_BASE, 0xFFFFFFFF & ~ILK_FBC_RT_VALID);
624 wa_write_masked_or(wal, IVB_FBC_RT_BASE_UPPER,634 wa_write_clr_set(wal, IVB_FBC_RT_BASE_UPPER,
625 0, /* write-only register; skip validation */635 0, /* write-only register; skip validation */
626 0xFFFFFFFF);636 0xFFFFFFFF);
627 637
628 /* Wa_1406306137:icl,ehl */638 /* Wa_1406306137:icl,ehl */
629 wa_masked_en(wal, GEN9_ROW_CHICKEN4, GEN11_DIS_PICK_2ND_EU);639 wa_masked_en(wal, GEN9_ROW_CHICKEN4, GEN11_DIS_PICK_2ND_EU);
@@ -643,11 +653,11 @@ static void gen12_ctx_workarounds_init(struct intel_engine_cs *engine, · 13 unmodified lines
643 * Wa_14010443199:rkl653 * Wa_14010443199:rkl
644 * Wa_14010698770:rkl654 * Wa_14010698770:rkl
645 */655 */
646 WA_SET_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,656 wa_masked_en(wal, GEN11_COMMON_SLICE_CHICKEN3,
647 GEN12_DISABLE_CPS_AWARE_COLOR_PIPE);657 GEN12_DISABLE_CPS_AWARE_COLOR_PIPE);
648 658
649 /* WaDisableGPGPUMidThreadPreemption:gen12 */659 /* WaDisableGPGPUMidThreadPreemption:gen12 */
650 WA_SET_FIELD_MASKED(GEN8_CS_CHICKEN1,660 wa_masked_field_set(wal, GEN8_CS_CHICKEN1,
651 GEN9_PREEMPT_GPGPU_LEVEL_MASK,661 GEN9_PREEMPT_GPGPU_LEVEL_MASK,
652 GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);662 GEN9_PREEMPT_GPGPU_THREAD_GROUP_LEVEL);
653}663}
@@ -680,12 +690,22 @@ static void dg1_ctx_workarounds_init(struct intel_engine_cs *engine, · 26 unmodified lines
680 gen12_ctx_workarounds_init(engine, wal);690 gen12_ctx_workarounds_init(engine, wal);
681 691
682 /* Wa_1409044764 */692 /* Wa_1409044764 */
683 WA_CLR_BIT_MASKED(GEN11_COMMON_SLICE_CHICKEN3,693 wa_masked_dis(wal, GEN11_COMMON_SLICE_CHICKEN3,
684 DG1_FLOAT_POINT_BLEND_OPT_STRICT_MODE_EN);694 DG1_FLOAT_POINT_BLEND_OPT_STRICT_MODE_EN);
685 695
686 /* Wa_22010493298 */696 /* Wa_22010493298 */
687 WA_SET_BIT_MASKED(HIZ_CHICKEN,697 wa_masked_en(wal, HIZ_CHICKEN,
688 DG1_HZ_READ_SUPPRESSION_OPTIMIZATION_DISABLE);698 DG1_HZ_READ_SUPPRESSION_OPTIMIZATION_DISABLE);
699
700 /*
701 * Wa_16011163337
702 *
703 * Like in tgl_ctx_workarounds_init(), read verification is ignored due
704 * to Wa_1608008084.
705 */
706 wa_add(wal,
707 FF_MODE2,
708 FF_MODE2_GS_TIMER_MASK, FF_MODE2_GS_TIMER_224, 0);
689}709}
690 710
691static void711static void
@@ -804,57 +824,11 @@ ilk_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 112 unmodified lines
804static void824static void
805snb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)825snb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)
806{826{
807 /* WaDisableHiZPlanesWhenMSAAEnabled:snb */
808 wa_masked_en(wal,
809 _3D_CHICKEN,
810 _3D_CHICKEN_HIZ_PLANE_DISABLE_MSAA_4X_SNB);
811
812 /* WaDisable_RenderCache_OperationalFlush:snb */
813 wa_masked_dis(wal, CACHE_MODE_0, RC_OP_FLUSH_ENABLE);
814
815 /*
816 * BSpec recommends 8x4 when MSAA is used,
817 * however in practice 16x4 seems fastest.
818 *
819 * Note that PS/WM thread counts depend on the WIZ hashing
820 * disable bit, which we don't touch here, but it's good
821 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
822 */
823 wa_add(wal,
824 GEN6_GT_MODE, 0,
825 _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
826 GEN6_WIZ_HASHING_16x4);
827
828 wa_masked_dis(wal, CACHE_MODE_0, CM0_STC_EVICT_DISABLE_LRA_SNB);
829
830 wa_masked_en(wal,
831 _3D_CHICKEN3,
832 /* WaStripsFansDisableFastClipPerformanceFix:snb */
833 _3D_CHICKEN3_SF_DISABLE_FASTCLIP_CULL |
834 /*
835 * Bspec says:
836 * "This bit must be set if 3DSTATE_CLIP clip mode is set
837 * to normal and 3DSTATE_SF number of SF output attributes
838 * is more than 16."
839 */
840 _3D_CHICKEN3_SF_DISABLE_PIPELINED_ATTR_FETCH);
841}827}
842 828
843static void829static void
844ivb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)830ivb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)
845{831{
846 /* WaDisableEarlyCull:ivb */
847 wa_masked_en(wal, _3D_CHICKEN3, _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
848
849 /* WaDisablePSDDualDispatchEnable:ivb */
850 if (IS_IVB_GT1(i915))
851 wa_masked_en(wal,
852 GEN7_HALF_SLICE_CHICKEN1,
853 GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
854
855 /* WaDisable_RenderCache_OperationalFlush:ivb */
856 wa_masked_dis(wal, CACHE_MODE_0_GEN7, RC_OP_FLUSH_ENABLE);
857
858 /* Apply the WaDisableRHWOOptimizationForRenderHang:ivb workaround. */832 /* Apply the WaDisableRHWOOptimizationForRenderHang:ivb workaround. */
859 wa_masked_dis(wal,833 wa_masked_dis(wal,
860 GEN7_COMMON_SLICE_CHICKEN1,834 GEN7_COMMON_SLICE_CHICKEN1,
@@ -866,90 +840,14 @@ ivb_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 5 unmodified lines
866 840
867 /* WaForceL3Serialization:ivb */841 /* WaForceL3Serialization:ivb */
868 wa_write_clr(wal, GEN7_L3SQCREG4, L3SQ_URB_READ_CAM_MATCH_DISABLE);842 wa_write_clr(wal, GEN7_L3SQCREG4, L3SQ_URB_READ_CAM_MATCH_DISABLE);
869
870 /*
871 * WaVSThreadDispatchOverride:ivb,vlv
872 *
873 * This actually overrides the dispatch
874 * mode for all thread types.
875 */
876 wa_write_masked_or(wal, GEN7_FF_THREAD_MODE,
877 GEN7_FF_SCHED_MASK,
878 GEN7_FF_TS_SCHED_HW |
879 GEN7_FF_VS_SCHED_HW |
880 GEN7_FF_DS_SCHED_HW);
881
882 if (0) { /* causes HiZ corruption on ivb:gt1 */
883 /* enable HiZ Raw Stall Optimization */
884 wa_masked_dis(wal, CACHE_MODE_0_GEN7, HIZ_RAW_STALL_OPT_DISABLE);
885 }
886
887 /* WaDisable4x2SubspanOptimization:ivb */
888 wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
889
890 /*
891 * BSpec recommends 8x4 when MSAA is used,
892 * however in practice 16x4 seems fastest.
893 *
894 * Note that PS/WM thread counts depend on the WIZ hashing
895 * disable bit, which we don't touch here, but it's good
896 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
897 */
898 wa_add(wal, GEN7_GT_MODE, 0,
899 _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
900 GEN6_WIZ_HASHING_16x4);
901}843}
902 844
903static void845static void
904vlv_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)846vlv_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal)
905{847{
906 /* WaDisableEarlyCull:vlv */
907 wa_masked_en(wal, _3D_CHICKEN3, _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
908
909 /* WaPsdDispatchEnable:vlv */
910 /* WaDisablePSDDualDispatchEnable:vlv */
911 wa_masked_en(wal,
912 GEN7_HALF_SLICE_CHICKEN1,
913 GEN7_MAX_PS_THREAD_DEP |
914 GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
915
916 /* WaDisable_RenderCache_OperationalFlush:vlv */
917 wa_masked_dis(wal, CACHE_MODE_0_GEN7, RC_OP_FLUSH_ENABLE);
918
919 /* WaForceL3Serialization:vlv */848 /* WaForceL3Serialization:vlv */
920 wa_write_clr(wal, GEN7_L3SQCREG4, L3SQ_URB_READ_CAM_MATCH_DISABLE);849 wa_write_clr(wal, GEN7_L3SQCREG4, L3SQ_URB_READ_CAM_MATCH_DISABLE);
921 850
922 /*
923 * WaVSThreadDispatchOverride:ivb,vlv
924 *
925 * This actually overrides the dispatch
926 * mode for all thread types.
927 */
928 wa_write_masked_or(wal,
929 GEN7_FF_THREAD_MODE,
930 GEN7_FF_SCHED_MASK,
931 GEN7_FF_TS_SCHED_HW |
932 GEN7_FF_VS_SCHED_HW |
933 GEN7_FF_DS_SCHED_HW);
934
935 /*
936 * BSpec says this must be set, even though
937 * WaDisable4x2SubspanOptimization isn't listed for VLV.
938 */
939 wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
940
941 /*
942 * BSpec recommends 8x4 when MSAA is used,
943 * however in practice 16x4 seems fastest.
944 *
945 * Note that PS/WM thread counts depend on the WIZ hashing
946 * disable bit, which we don't touch here, but it's good
947 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
948 */
949 wa_add(wal, GEN7_GT_MODE, 0,
950 _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
951 GEN6_WIZ_HASHING_16x4);
952
953 /*851 /*
954 * WaIncreaseL3CreditsForVLVB0:vlv852 * WaIncreaseL3CreditsForVLVB0:vlv
955 * This is the hardware default actually.853 * This is the hardware default actually.
@@ -970,31 +868,6 @@ hsw_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 14 unmodified lines
970 868
971 /* WaVSRefCountFullforceMissDisable:hsw */869 /* WaVSRefCountFullforceMissDisable:hsw */
972 wa_write_clr(wal, GEN7_FF_THREAD_MODE, GEN7_FF_VS_REF_CNT_FFME);870 wa_write_clr(wal, GEN7_FF_THREAD_MODE, GEN7_FF_VS_REF_CNT_FFME);
973
974 wa_masked_dis(wal,
975 CACHE_MODE_0_GEN7,
976 /* WaDisable_RenderCache_OperationalFlush:hsw */
977 RC_OP_FLUSH_ENABLE |
978 /* enable HiZ Raw Stall Optimization */
979 HIZ_RAW_STALL_OPT_DISABLE);
980
981 /* WaDisable4x2SubspanOptimization:hsw */
982 wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
983
984 /*
985 * BSpec recommends 8x4 when MSAA is used,
986 * however in practice 16x4 seems fastest.
987 *
988 * Note that PS/WM thread counts depend on the WIZ hashing
989 * disable bit, which we don't touch here, but it's good
990 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
991 */
992 wa_add(wal, GEN7_GT_MODE, 0,
993 _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
994 GEN6_WIZ_HASHING_16x4);
995
996 /* WaSampleCChickenBitEnable:hsw */
997 wa_masked_en(wal, HALF_SLICE_CHICKEN3, HSW_SAMPLE_C_PERFORMANCE);
998}871}
999 872
1000static void873static void
@@ -1164,7 +1037,7 @@ wa_init_mcr(struct drm_i915_private *i915, struct i915_wa_list *wal) · 163 unmodified lines
1164 1037
1165 drm_dbg(&i915->drm, "MCR slice/subslice = %x\n", mcr);1038 drm_dbg(&i915->drm, "MCR slice/subslice = %x\n", mcr);
1166 1039
1167 wa_write_masked_or(wal, GEN8_MCR_SELECTOR, mcr_mask, mcr);1040 wa_write_clr_set(wal, GEN8_MCR_SELECTOR, mcr_mask, mcr);
1168}1041}
1169 1042
1170static void1043static void
@@ -1189,10 +1062,10 @@ icl_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 18 unmodified lines
1189 GAMT_ECO_ENABLE_IN_PLACE_DECOMPRESS);1062 GAMT_ECO_ENABLE_IN_PLACE_DECOMPRESS);
1190 1063
1191 /* WaModifyGamTlbPartitioning:icl */1064 /* WaModifyGamTlbPartitioning:icl */
1192 wa_write_masked_or(wal,1065 wa_write_clr_set(wal,
1193 GEN11_GACB_PERF_CTRL,1066 GEN11_GACB_PERF_CTRL,
1194 GEN11_HASH_CTRL_MASK,1067 GEN11_HASH_CTRL_MASK,
1195 GEN11_HASH_CTRL_BIT0 | GEN11_HASH_CTRL_BIT4);1068 GEN11_HASH_CTRL_BIT0 | GEN11_HASH_CTRL_BIT4);
1196 1069
1197 /* Wa_1405766107:icl1070 /* Wa_1405766107:icl
1198 * Formerly known as WaCL2SFHalfMaxAlloc1071 * Formerly known as WaCL2SFHalfMaxAlloc
@@ -1260,6 +1133,11 @@ tgl_gt_workarounds_init(struct drm_i915_private *i915, struct i915_wa_list *wal) · 61 unmodified lines
1260 wa_write_or(wal,1133 wa_write_or(wal,
1261 SLICE_UNIT_LEVEL_CLKGATE,1134 SLICE_UNIT_LEVEL_CLKGATE,
1262 L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS);1135 L3_CLKGATE_DIS | L3_CR2X_CLKGATE_DIS);
1136
1137 /* Wa_1408615072:tgl[a0] */
1138 if (IS_TGL_UY_GT_REVID(i915, TGL_REVID_A0, TGL_REVID_A0))
1139 wa_write_or(wal, UNSLICE_UNIT_LEVEL_CLKGATE2,
1140 VSUNIT_CLKGATE_DIS_TGL);
1263}1141}
1264 1142
1265static void1143static void
@@ -1358,9 +1236,9 @@ static bool · 92 unmodified lines
1358wa_verify(const struct i915_wa *wa, u32 cur, const char *name, const char *from)1236wa_verify(const struct i915_wa *wa, u32 cur, const char *name, const char *from)
1359{1237{
1360 if ((cur ^ wa->set) & wa->read) {1238 if ((cur ^ wa->set) & wa->read) {
1361 DRM_ERROR("%s workaround lost on %s! (%x=%x/%x, expected %x)\n",1239 DRM_ERROR("%s workaround lost on %s! (reg[%x]=0x%x, relevant bits were 0x%x vs expected 0x%x)\n",
1362 name, from, i915_mmio_reg_offset(wa->reg),1240 name, from, i915_mmio_reg_offset(wa->reg),
1363 cur, cur & wa->read, wa->set);1241 cur, cur & wa->read, wa->set & wa->read);
1364 1242
1365 return false;1243 return false;
1366 }1244 }
@@ -1425,6 +1303,7 @@ bool intel_gt_verify_workarounds(struct intel_gt *gt, const char *from) · 58 unmodified lines
1425 return wa_list_verify(gt->uncore, &gt->i915->gt_wa_list, from);1303 return wa_list_verify(gt->uncore, &gt->i915->gt_wa_list, from);
1426}1304}
1427 1305
1306__maybe_unused
1428static inline bool is_nonpriv_flags_valid(u32 flags)1307static inline bool is_nonpriv_flags_valid(u32 flags)
1429{1308{
1430 /* Check only valid flag bits are set */1309 /* Check only valid flag bits are set */
@@ -1752,10 +1631,6 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 321 unmodified lines
1752 wa_write_or(wal,1631 wa_write_or(wal,
1753 GEN7_SARCHKMD,1632 GEN7_SARCHKMD,
1754 GEN7_DISABLE_SAMPLER_PREFETCH);1633 GEN7_DISABLE_SAMPLER_PREFETCH);
1755
1756 /* Wa_1408615072:tgl */
1757 wa_write_or(wal, UNSLICE_UNIT_LEVEL_CLKGATE2,
1758 VSUNIT_CLKGATE_DIS_TGL);
1759 }1634 }
1760 1635
1761 if (IS_DG1(i915) || IS_ROCKETLAKE(i915) || IS_TIGERLAKE(i915)) {1636 if (IS_DG1(i915) || IS_ROCKETLAKE(i915) || IS_TIGERLAKE(i915)) {
@@ -1770,6 +1645,19 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 8 unmodified lines
1770 */1645 */
1771 wa_write_or(wal, GEN7_FF_THREAD_MODE,1646 wa_write_or(wal, GEN7_FF_THREAD_MODE,
1772 GEN12_FF_TESSELATION_DOP_GATE_DISABLE);1647 GEN12_FF_TESSELATION_DOP_GATE_DISABLE);
1648
1649 /*
1650 * Wa_1606700617:tgl,dg1
1651 * Wa_22010271021:tgl,rkl,dg1
1652 */
1653 wa_masked_en(wal,
1654 GEN9_CS_DEBUG_MODE1,
1655 FF_DOP_CLOCK_GATE_DISABLE);
1656
1657 /* Wa_1406941453:tgl,rkl,dg1 */
1658 wa_masked_en(wal,
1659 GEN10_SAMPLER_MODE,
1660 ENABLE_SMALLPL);
1773 }1661 }
1774 1662
1775 if (IS_DG1_REVID(i915, DG1_REVID_A0, DG1_REVID_A0) ||1663 if (IS_DG1_REVID(i915, DG1_REVID_A0, DG1_REVID_A0) ||
@@ -1798,21 +1686,6 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 22 unmodified lines
1798 GEN6_RC_SLEEP_PSMI_CONTROL,1686 GEN6_RC_SLEEP_PSMI_CONTROL,
1799 GEN12_WAIT_FOR_EVENT_POWER_DOWN_DISABLE |1687 GEN12_WAIT_FOR_EVENT_POWER_DOWN_DISABLE |
1800 GEN8_RC_SEMA_IDLE_MSG_DISABLE);1688 GEN8_RC_SEMA_IDLE_MSG_DISABLE);
1801
1802 /*
1803 * Wa_1606700617:tgl
1804 * Wa_22010271021:tgl,rkl
1805 */
1806 wa_masked_en(wal,
1807 GEN9_CS_DEBUG_MODE1,
1808 FF_DOP_CLOCK_GATE_DISABLE);
1809 }
1810
1811 if (IS_GEN(i915, 12)) {
1812 /* Wa_1406941453:gen12 */
1813 wa_masked_en(wal,
1814 GEN10_SAMPLER_MODE,
1815 ENABLE_SMALLPL);
1816 }1689 }
1817 1690
1818 if (IS_GEN(i915, 11)) {1691 if (IS_GEN(i915, 11)) {
@@ -1838,14 +1711,14 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 19 unmodified lines
1838 * Wa_1604223664:icl1711 * Wa_1604223664:icl
1839 * Formerly known as WaL3BankAddressHashing1712 * Formerly known as WaL3BankAddressHashing
1840 */1713 */
1841 wa_write_masked_or(wal,1714 wa_write_clr_set(wal,
1842 GEN8_GARBCNTL,1715 GEN8_GARBCNTL,
1843 GEN11_HASH_CTRL_EXCL_MASK,1716 GEN11_HASH_CTRL_EXCL_MASK,
1844 GEN11_HASH_CTRL_EXCL_BIT0);1717 GEN11_HASH_CTRL_EXCL_BIT0);
1845 wa_write_masked_or(wal,1718 wa_write_clr_set(wal,
1846 GEN11_GLBLINVL,1719 GEN11_GLBLINVL,
1847 GEN11_BANK_HASH_ADDR_EXCL_MASK,1720 GEN11_BANK_HASH_ADDR_EXCL_MASK,
1848 GEN11_BANK_HASH_ADDR_EXCL_BIT0);1721 GEN11_BANK_HASH_ADDR_EXCL_BIT0);
1849 1722
1850 /*1723 /*
1851 * Wa_1405733216:icl1724 * Wa_1405733216:icl
@@ -1874,10 +1747,10 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 22 unmodified lines
1874 GEN7_DISABLE_SAMPLER_PREFETCH);1747 GEN7_DISABLE_SAMPLER_PREFETCH);
1875 1748
1876 /* Wa_1409178092:icl */1749 /* Wa_1409178092:icl */
1877 wa_write_masked_or(wal,1750 wa_write_clr_set(wal,
1878 GEN11_SCRATCH2,1751 GEN11_SCRATCH2,
1879 GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE,1752 GEN11_COHERENT_PARTIAL_WRITE_MERGE_ENABLE,
1880 0);1753 0);
1881 1754
1882 /* WaEnable32PlaneMode:icl */1755 /* WaEnable32PlaneMode:icl */
1883 wa_masked_en(wal, GEN9_CSFE_CHICKEN1_RCS,1756 wa_masked_en(wal, GEN9_CSFE_CHICKEN1_RCS,
@@ -1951,11 +1824,11 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 67 unmodified lines
1951 1824
1952 /* WaProgramL3SqcReg1DefaultForPerf:bxt,glk */1825 /* WaProgramL3SqcReg1DefaultForPerf:bxt,glk */
1953 if (IS_GEN9_LP(i915))1826 if (IS_GEN9_LP(i915))
1954 wa_write_masked_or(wal,1827 wa_write_clr_set(wal,
1955 GEN8_L3SQCREG1,1828 GEN8_L3SQCREG1,
1956 L3_PRIO_CREDITS_MASK,1829 L3_PRIO_CREDITS_MASK,
1957 L3_GENERAL_PRIO_CREDITS(62) |1830 L3_GENERAL_PRIO_CREDITS(62) |
1958 L3_HIGH_PRIO_CREDITS(2));1831 L3_HIGH_PRIO_CREDITS(2));
1959 1832
1960 /* WaOCLCoherentLineFlush:skl,bxt,kbl,cfl */1833 /* WaOCLCoherentLineFlush:skl,bxt,kbl,cfl */
1961 wa_write_or(wal,1834 wa_write_or(wal,
@@ -1963,12 +1836,112 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 1 unmodified line
1963 GEN8_LQSC_FLUSH_COHERENT_LINES);1836 GEN8_LQSC_FLUSH_COHERENT_LINES);
1964 }1837 }
1965 1838
1966 if (IS_GEN(i915, 7))1839 if (IS_HASWELL(i915)) {
1840 /* WaSampleCChickenBitEnable:hsw */
1841 wa_masked_en(wal,
1842 HALF_SLICE_CHICKEN3, HSW_SAMPLE_C_PERFORMANCE);
1843
1844 wa_masked_dis(wal,
1845 CACHE_MODE_0_GEN7,
1846 /* enable HiZ Raw Stall Optimization */
1847 HIZ_RAW_STALL_OPT_DISABLE);
1848
1849 /* WaDisable4x2SubspanOptimization:hsw */
1850 wa_masked_en(wal, CACHE_MODE_1, PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
1851 }
1852
1853 if (IS_VALLEYVIEW(i915)) {
1854 /* WaDisableEarlyCull:vlv */
1855 wa_masked_en(wal,
1856 _3D_CHICKEN3,
1857 _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
1858
1859 /*
1860 * WaVSThreadDispatchOverride:ivb,vlv
1861 *
1862 * This actually overrides the dispatch
1863 * mode for all thread types.
1864 */
1865 wa_write_clr_set(wal,
1866 GEN7_FF_THREAD_MODE,
1867 GEN7_FF_SCHED_MASK,
1868 GEN7_FF_TS_SCHED_HW |
1869 GEN7_FF_VS_SCHED_HW |
1870 GEN7_FF_DS_SCHED_HW);
1871
1872 /* WaPsdDispatchEnable:vlv */
1873 /* WaDisablePSDDualDispatchEnable:vlv */
1874 wa_masked_en(wal,
1875 GEN7_HALF_SLICE_CHICKEN1,
1876 GEN7_MAX_PS_THREAD_DEP |
1877 GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
1878 }
1879
1880 if (IS_IVYBRIDGE(i915)) {
1881 /* WaDisableEarlyCull:ivb */
1882 wa_masked_en(wal,
1883 _3D_CHICKEN3,
1884 _3D_CHICKEN_SF_DISABLE_OBJEND_CULL);
1885
1886 if (0) { /* causes HiZ corruption on ivb:gt1 */
1887 /* enable HiZ Raw Stall Optimization */
1888 wa_masked_dis(wal,
1889 CACHE_MODE_0_GEN7,
1890 HIZ_RAW_STALL_OPT_DISABLE);
1891 }
1892
1893 /*
1894 * WaVSThreadDispatchOverride:ivb,vlv
1895 *
1896 * This actually overrides the dispatch
1897 * mode for all thread types.
1898 */
1899 wa_write_clr_set(wal,
1900 GEN7_FF_THREAD_MODE,
1901 GEN7_FF_SCHED_MASK,
1902 GEN7_FF_TS_SCHED_HW |
1903 GEN7_FF_VS_SCHED_HW |
1904 GEN7_FF_DS_SCHED_HW);
1905
1906 /* WaDisablePSDDualDispatchEnable:ivb */
1907 if (IS_IVB_GT1(i915))
1908 wa_masked_en(wal,
1909 GEN7_HALF_SLICE_CHICKEN1,
1910 GEN7_PSD_SINGLE_PORT_DISPATCH_ENABLE);
1911 }
1912
1913 if (IS_GEN(i915, 7)) {
1967 /* WaBCSVCSTlbInvalidationMode:ivb,vlv,hsw */1914 /* WaBCSVCSTlbInvalidationMode:ivb,vlv,hsw */
1968 wa_masked_en(wal,1915 wa_masked_en(wal,
1969 GFX_MODE_GEN7,1916 GFX_MODE_GEN7,
1970 GFX_TLB_INVALIDATE_EXPLICIT | GFX_REPLAY_MODE);1917 GFX_TLB_INVALIDATE_EXPLICIT | GFX_REPLAY_MODE);
1971 1918
1919 /* WaDisable_RenderCache_OperationalFlush:ivb,vlv,hsw */
1920 wa_masked_dis(wal, CACHE_MODE_0_GEN7, RC_OP_FLUSH_ENABLE);
1921
1922 /*
1923 * BSpec says this must be set, even though
1924 * WaDisable4x2SubspanOptimization:ivb,hsw
1925 * WaDisable4x2SubspanOptimization isn't listed for VLV.
1926 */
1927 wa_masked_en(wal,
1928 CACHE_MODE_1,
1929 PIXEL_SUBSPAN_COLLECT_OPT_DISABLE);
1930
1931 /*
1932 * BSpec recommends 8x4 when MSAA is used,
1933 * however in practice 16x4 seems fastest.
1934 *
1935 * Note that PS/WM thread counts depend on the WIZ hashing
1936 * disable bit, which we don't touch here, but it's good
1937 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
1938 */
1939 wa_add(wal, GEN7_GT_MODE, 0,
1940 _MASKED_FIELD(GEN6_WIZ_HASHING_MASK,
1941 GEN6_WIZ_HASHING_16x4),
1942 GEN6_WIZ_HASHING_16x4);
1943 }
1944
1972 if (IS_GEN_RANGE(i915, 6, 7))1945 if (IS_GEN_RANGE(i915, 6, 7))
1973 /*1946 /*
1974 * We need to disable the AsyncFlip performance optimisations in1947 * We need to disable the AsyncFlip performance optimisations in
@@ -1991,6 +1964,39 @@ rcs_engine_wa_init(struct intel_engine_cs *engine, struct i915_wa_list *wal) · 16 unmodified lines
1991 GFX_MODE,1964 GFX_MODE,
1992 GFX_TLB_INVALIDATE_EXPLICIT);1965 GFX_TLB_INVALIDATE_EXPLICIT);
1993 1966
1967 /* WaDisableHiZPlanesWhenMSAAEnabled:snb */
1968 wa_masked_en(wal,
1969 _3D_CHICKEN,
1970 _3D_CHICKEN_HIZ_PLANE_DISABLE_MSAA_4X_SNB);
1971
1972 wa_masked_en(wal,
1973 _3D_CHICKEN3,
1974 /* WaStripsFansDisableFastClipPerformanceFix:snb */
1975 _3D_CHICKEN3_SF_DISABLE_FASTCLIP_CULL |
1976 /*
1977 * Bspec says:
1978 * "This bit must be set if 3DSTATE_CLIP clip mode is set
1979 * to normal and 3DSTATE_SF number of SF output attributes
1980 * is more than 16."
1981 */
1982 _3D_CHICKEN3_SF_DISABLE_PIPELINED_ATTR_FETCH);
1983
1984 /*
1985 * BSpec recommends 8x4 when MSAA is used,
1986 * however in practice 16x4 seems fastest.
1987 *
1988 * Note that PS/WM thread counts depend on the WIZ hashing
1989 * disable bit, which we don't touch here, but it's good
1990 * to keep in mind (see 3DSTATE_PS and 3DSTATE_WM).
1991 */
1992 wa_add(wal,
1993 GEN6_GT_MODE, 0,
1994 _MASKED_FIELD(GEN6_WIZ_HASHING_MASK, GEN6_WIZ_HASHING_16x4),
1995 GEN6_WIZ_HASHING_16x4);
1996
1997 /* WaDisable_RenderCache_OperationalFlush:snb */
1998 wa_masked_dis(wal, CACHE_MODE_0, RC_OP_FLUSH_ENABLE);
1999
1994 /*2000 /*
1995 * From the Sandybridge PRM, volume 1 part 3, page 24:2001 * From the Sandybridge PRM, volume 1 part 3, page 24:
1996 * "If this bit is set, STCunit will have LRA as replacement2002 * "If this bit is set, STCunit will have LRA as replacement
@@ -2067,39 +2073,6 @@ void intel_engine_apply_workarounds(struct intel_engine_cs *engine) · 70 unmodified lines
2067 wa_list_apply(engine->uncore, &engine->wa_list);2073 wa_list_apply(engine->uncore, &engine->wa_list);
2068}2074}
2069 2075
2070static struct i915_vma *
2071create_scratch(struct i915_address_space *vm, int count)
2072{
2073 struct drm_i915_gem_object *obj;
2074 struct i915_vma *vma;
2075 unsigned int size;
2076 int err;
2077
2078 size = round_up(count * sizeof(u32), PAGE_SIZE);
2079 obj = i915_gem_object_create_internal(vm->i915, size);
2080 if (IS_ERR(obj))
2081 return ERR_CAST(obj);
2082
2083 i915_gem_object_set_cache_coherency(obj, I915_CACHE_LLC);
2084
2085 vma = i915_vma_instance(obj, vm, NULL);
2086 if (IS_ERR(vma)) {
2087 err = PTR_ERR(vma);
2088 goto err_obj;
2089 }
2090
2091 err = i915_vma_pin(vma, 0, 0,
2092 i915_vma_is_ggtt(vma) ? PIN_GLOBAL : PIN_USER);
2093 if (err)
2094 goto err_obj;
2095
2096 return vma;
2097
2098err_obj:
2099 i915_gem_object_put(obj);
2100 return ERR_PTR(err);
2101}
2102
2103struct mcr_range {2076struct mcr_range {
2104 u32 start;2077 u32 start;
2105 u32 end;2078 u32 end;
@@ -2202,7 +2175,8 @@ static int engine_wa_list_verify(struct intel_context *ce, · 96 unmodified lines
2202 if (!wal->count)2175 if (!wal->count)
2203 return 0;2176 return 0;
2204 2177
2205 vma = create_scratch(&ce->engine->gt->ggtt->vm, wal->count);2178 vma = __vm_create_scratch_for_read(&ce->engine->gt->ggtt->vm,
2179 wal->count * sizeof(u32));
2206 if (IS_ERR(vma))2180 if (IS_ERR(vma))
2207 return PTR_ERR(vma);2181 return PTR_ERR(vma);
2208 2182
drivers/gpu/drm/i915/gt/mock_engine.c+2−13
@@ -245,17 +245,6 @@ static void mock_reset_rewind(struct intel_engine_cs *engine, bool stalled) · 244 unmodified lines
245 GEM_BUG_ON(stalled);245 GEM_BUG_ON(stalled);
246}246}
247 247
248static void mark_eio(struct i915_request *rq)
249{
250 if (i915_request_completed(rq))
251 return;
252
253 GEM_BUG_ON(i915_request_signaled(rq));
254
255 i915_request_set_error_once(rq, -EIO);
256 i915_request_mark_complete(rq);
257}
258
259static void mock_reset_cancel(struct intel_engine_cs *engine)248static void mock_reset_cancel(struct intel_engine_cs *engine)
260{249{
261 struct mock_engine *mock =250 struct mock_engine *mock =
@@ -269,12 +258,12 @@ static void mock_reset_cancel(struct intel_engine_cs *engine) · 7 unmodified lines
269 258
270 /* Mark all submitted requests as skipped. */259 /* Mark all submitted requests as skipped. */
271 list_for_each_entry(rq, &engine->active.requests, sched.link)260 list_for_each_entry(rq, &engine->active.requests, sched.link)
272 mark_eio(rq);261 i915_request_mark_eio(rq);
273 intel_engine_signal_breadcrumbs(engine);262 intel_engine_signal_breadcrumbs(engine);
274 263
275 /* Cancel and submit all pending requests. */264 /* Cancel and submit all pending requests. */
276 list_for_each_entry(rq, &mock->hw_queue, mock.link) {265 list_for_each_entry(rq, &mock->hw_queue, mock.link) {
277 mark_eio(rq);266 i915_request_mark_eio(rq);
278 __i915_request_submit(rq);267 __i915_request_submit(rq);
279 }268 }
280 INIT_LIST_HEAD(&mock->hw_queue);269 INIT_LIST_HEAD(&mock->hw_queue);
drivers/gpu/drm/i915/gt/selftest_context.c+1−1
@@ -25,7 +25,7 @@ static int request_sync(struct i915_request *rq) · 24 unmodified lines
25 /* Opencode i915_request_add() so we can keep the timeline locked. */25 /* Opencode i915_request_add() so we can keep the timeline locked. */
26 __i915_request_commit(rq);26 __i915_request_commit(rq);
27 rq->sched.attr.priority = I915_PRIORITY_BARRIER;27 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
28 __i915_request_queue(rq, NULL);28 __i915_request_queue_bh(rq);
29 29
30 timeout = i915_request_wait(rq, 0, HZ / 10);30 timeout = i915_request_wait(rq, 0, HZ / 10);
31 if (timeout < 0)31 if (timeout < 0)
drivers/gpu/drm/i915/gt/selftest_engine_cs.c+1−0
@@ -6,6 +6,7 @@ · 5 unmodified lines
6 6
7#include <linux/sort.h>7#include <linux/sort.h>
8 8
9#include "intel_gpu_commands.h"
9#include "intel_gt_pm.h"10#include "intel_gt_pm.h"
10#include "intel_rps.h"11#include "intel_rps.h"
11 12
drivers/gpu/drm/i915/gt/selftest_engine_heartbeat.c+10−3
@@ -197,6 +197,7 @@ static int cmp_u32(const void *_a, const void *_b) · 196 unmodified lines
197 197
198static int __live_heartbeat_fast(struct intel_engine_cs *engine)198static int __live_heartbeat_fast(struct intel_engine_cs *engine)
199{199{
200 const unsigned int error_threshold = max(20000u, jiffies_to_usecs(6));
200 struct intel_context *ce;201 struct intel_context *ce;
201 struct i915_request *rq;202 struct i915_request *rq;
202 ktime_t t0, t1;203 ktime_t t0, t1;
@@ -254,12 +255,18 @@ static int __live_heartbeat_fast(struct intel_engine_cs *engine) · 51 unmodified lines
254 times[0],255 times[0],
255 times[ARRAY_SIZE(times) - 1]);256 times[ARRAY_SIZE(times) - 1]);
256 257
257 /* Min work delay is 2 * 2 (worst), +1 for scheduling, +1 for slack */258 /*
258 if (times[ARRAY_SIZE(times) / 2] > jiffies_to_usecs(6)) {259 * Ideally, the upper bound on min work delay would be something like
260 * 2 * 2 (worst), +1 for scheduling, +1 for slack. In practice, we
261 * are, even with system_wq_highpri, at the mercy of the CPU scheduler
262 * and may be stuck behind some slow work for many millisecond. Such
263 * as our very own display workers.
264 */
265 if (times[ARRAY_SIZE(times) / 2] > error_threshold) {
259 pr_err("%s: Heartbeat delay was %uus, expected less than %dus\n",266 pr_err("%s: Heartbeat delay was %uus, expected less than %dus\n",
260 engine->name,267 engine->name,
261 times[ARRAY_SIZE(times) / 2],268 times[ARRAY_SIZE(times) / 2],
262 jiffies_to_usecs(6));269 error_threshold);
263 err = -EINVAL;270 err = -EINVAL;
264 }271 }
265 272
drivers/gpu/drm/i915/gt/selftest_engine_pm.c+203−0
@@ -4,13 +4,215 @@ · 3 unmodified lines
4 * Copyright © 2018 Intel Corporation4 * Copyright © 2018 Intel Corporation
5 */5 */
6 6
7#include <linux/sort.h>
8
7#include "i915_selftest.h"9#include "i915_selftest.h"
10#include "intel_gpu_commands.h"
11#include "intel_gt_clock_utils.h"
8#include "selftest_engine.h"12#include "selftest_engine.h"
9#include "selftest_engine_heartbeat.h"13#include "selftest_engine_heartbeat.h"
10#include "selftests/igt_atomic.h"14#include "selftests/igt_atomic.h"
11#include "selftests/igt_flush_test.h"15#include "selftests/igt_flush_test.h"
12#include "selftests/igt_spinner.h"16#include "selftests/igt_spinner.h"
13 17
18#define COUNT 5
19
20static int cmp_u64(const void *A, const void *B)
21{
22 const u64 *a = A, *b = B;
23
24 return *a - *b;
25}
26
27static u64 trifilter(u64 *a)
28{
29 sort(a, COUNT, sizeof(*a), cmp_u64, NULL);
30 return (a[1] + 2 * a[2] + a[3]) >> 2;
31}
32
33static u32 *emit_wait(u32 *cs, u32 offset, int op, u32 value)
34{
35 *cs++ = MI_SEMAPHORE_WAIT |
36 MI_SEMAPHORE_GLOBAL_GTT |
37 MI_SEMAPHORE_POLL |
38 op;
39 *cs++ = value;
40 *cs++ = offset;
41 *cs++ = 0;
42
43 return cs;
44}
45
46static u32 *emit_store(u32 *cs, u32 offset, u32 value)
47{
48 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
49 *cs++ = offset;
50 *cs++ = 0;
51 *cs++ = value;
52
53 return cs;
54}
55
56static u32 *emit_srm(u32 *cs, i915_reg_t reg, u32 offset)
57{
58 *cs++ = MI_STORE_REGISTER_MEM_GEN8 | MI_USE_GGTT;
59 *cs++ = i915_mmio_reg_offset(reg);
60 *cs++ = offset;
61 *cs++ = 0;
62
63 return cs;
64}
65
66static void write_semaphore(u32 *x, u32 value)
67{
68 WRITE_ONCE(*x, value);
69 wmb();
70}
71
72static int __measure_timestamps(struct intel_context *ce,
73 u64 *dt, u64 *d_ring, u64 *d_ctx)
74{
75 struct intel_engine_cs *engine = ce->engine;
76 u32 *sema = memset32(engine->status_page.addr + 1000, 0, 5);
77 u32 offset = i915_ggtt_offset(engine->status_page.vma);
78 struct i915_request *rq;
79 u32 *cs;
80
81 rq = intel_context_create_request(ce);
82 if (IS_ERR(rq))
83 return PTR_ERR(rq);
84
85 cs = intel_ring_begin(rq, 28);
86 if (IS_ERR(cs)) {
87 i915_request_add(rq);
88 return PTR_ERR(cs);
89 }
90
91 /* Signal & wait for start */
92 cs = emit_store(cs, offset + 4008, 1);
93 cs = emit_wait(cs, offset + 4008, MI_SEMAPHORE_SAD_NEQ_SDD, 1);
94
95 cs = emit_srm(cs, RING_TIMESTAMP(engine->mmio_base), offset + 4000);
96 cs = emit_srm(cs, RING_CTX_TIMESTAMP(engine->mmio_base), offset + 4004);
97
98 /* Busy wait */
99 cs = emit_wait(cs, offset + 4008, MI_SEMAPHORE_SAD_EQ_SDD, 1);
100
101 cs = emit_srm(cs, RING_TIMESTAMP(engine->mmio_base), offset + 4016);
102 cs = emit_srm(cs, RING_CTX_TIMESTAMP(engine->mmio_base), offset + 4012);
103
104 intel_ring_advance(rq, cs);
105 i915_request_get(rq);
106 i915_request_add(rq);
107 intel_engine_flush_submission(engine);
108
109 /* Wait for the request to start executing, that then waits for us */
110 while (READ_ONCE(sema[2]) == 0)
111 cpu_relax();
112
113 /* Run the request for a 100us, sampling timestamps before/after */
114 preempt_disable();
115 *dt = local_clock();
116 write_semaphore(&sema[2], 0);
117 udelay(100);
118 *dt = local_clock() - *dt;
119 write_semaphore(&sema[2], 1);
120 preempt_enable();
121
122 if (i915_request_wait(rq, 0, HZ / 2) < 0) {
123 i915_request_put(rq);
124 return -ETIME;
125 }
126 i915_request_put(rq);
127
128 pr_debug("%s CTX_TIMESTAMP: [%x, %x], RING_TIMESTAMP: [%x, %x]\n",
129 engine->name, sema[1], sema[3], sema[0], sema[4]);
130
131 *d_ctx = sema[3] - sema[1];
132 *d_ring = sema[4] - sema[0];
133 return 0;
134}
135
136static int __live_engine_timestamps(struct intel_engine_cs *engine)
137{
138 u64 s_ring[COUNT], s_ctx[COUNT], st[COUNT], d_ring, d_ctx, dt;
139 struct intel_context *ce;
140 int i, err = 0;
141
142 ce = intel_context_create(engine);
143 if (IS_ERR(ce))
144 return PTR_ERR(ce);
145
146 for (i = 0; i < COUNT; i++) {
147 err = __measure_timestamps(ce, &st[i], &s_ring[i], &s_ctx[i]);
148 if (err)
149 break;
150 }
151 intel_context_put(ce);
152 if (err)
153 return err;
154
155 dt = trifilter(st);
156 d_ring = trifilter(s_ring);
157 d_ctx = trifilter(s_ctx);
158
159 pr_info("%s elapsed:%lldns, CTX_TIMESTAMP:%lldns, RING_TIMESTAMP:%lldns\n",
160 engine->name, dt,
161 intel_gt_clock_interval_to_ns(engine->gt, d_ctx),
162 intel_gt_clock_interval_to_ns(engine->gt, d_ring));
163
164 d_ring = intel_gt_clock_interval_to_ns(engine->gt, d_ring);
165 if (3 * dt > 4 * d_ring || 4 * dt < 3 * d_ring) {
166 pr_err("%s Mismatch between ring timestamp and walltime!\n",
167 engine->name);
168 return -EINVAL;
169 }
170
171 d_ring = trifilter(s_ring);
172 d_ctx = trifilter(s_ctx);
173
174 d_ctx *= engine->gt->clock_frequency;
175 if (IS_ICELAKE(engine->i915))
176 d_ring *= 12500000; /* Fixed 80ns for icl ctx timestamp? */
177 else
178 d_ring *= engine->gt->clock_frequency;
179
180 if (3 * d_ctx > 4 * d_ring || 4 * d_ctx < 3 * d_ring) {
181 pr_err("%s Mismatch between ring and context timestamps!\n",
182 engine->name);
183 return -EINVAL;
184 }
185
186 return 0;
187}
188
189static int live_engine_timestamps(void *arg)
190{
191 struct intel_gt *gt = arg;
192 struct intel_engine_cs *engine;
193 enum intel_engine_id id;
194
195 /*
196 * Check that CS_TIMESTAMP / CTX_TIMESTAMP are in sync, i.e. share
197 * the same CS clock.
198 */
199
200 if (INTEL_GEN(gt->i915) < 8)
201 return 0;
202
203 for_each_engine(engine, gt, id) {
204 int err;
205
206 st_engine_heartbeat_disable(engine);
207 err = __live_engine_timestamps(engine);
208 st_engine_heartbeat_enable(engine);
209 if (err)
210 return err;
211 }
212
213 return 0;
214}
215
14static int live_engine_busy_stats(void *arg)216static int live_engine_busy_stats(void *arg)
15{217{
16 struct intel_gt *gt = arg;218 struct intel_gt *gt = arg;
@@ -177,6 +379,7 @@ static int live_engine_pm(void *arg) · 160 unmodified lines
177int live_engine_pm_selftests(struct intel_gt *gt)379int live_engine_pm_selftests(struct intel_gt *gt)
178{380{
179 static const struct i915_subtest tests[] = {381 static const struct i915_subtest tests[] = {
382 SUBTEST(live_engine_timestamps),
180 SUBTEST(live_engine_busy_stats),383 SUBTEST(live_engine_busy_stats),
181 SUBTEST(live_engine_pm),384 SUBTEST(live_engine_pm),
182 };385 };
drivers/gpu/drm/i915/gt/selftest_execlists.c+4741−0
@@ -0,0 +1,4741 @@
1/*
2 * SPDX-License-Identifier: MIT
3 *
4 * Copyright © 2018 Intel Corporation
5 */
6
7#include <linux/prime_numbers.h>
8
9#include "gem/i915_gem_pm.h"
10#include "gt/intel_engine_heartbeat.h"
11#include "gt/intel_reset.h"
12#include "gt/selftest_engine_heartbeat.h"
13
14#include "i915_selftest.h"
15#include "selftests/i915_random.h"
16#include "selftests/igt_flush_test.h"
17#include "selftests/igt_live_test.h"
18#include "selftests/igt_spinner.h"
19#include "selftests/lib_sw_fence.h"
20
21#include "gem/selftests/igt_gem_utils.h"
22#include "gem/selftests/mock_context.h"
23
24#define CS_GPR(engine, n) ((engine)->mmio_base + 0x600 + (n) * 4)
25#define NUM_GPR 16
26#define NUM_GPR_DW (NUM_GPR * 2) /* each GPR is 2 dwords */
27
28static bool is_active(struct i915_request *rq)
29{
30 if (i915_request_is_active(rq))
31 return true;
32
33 if (i915_request_on_hold(rq))
34 return true;
35
36 if (i915_request_has_initial_breadcrumb(rq) && i915_request_started(rq))
37 return true;
38
39 return false;
40}
41
42static int wait_for_submit(struct intel_engine_cs *engine,
43 struct i915_request *rq,
44 unsigned long timeout)
45{
46 /* Ignore our own attempts to suppress excess tasklets */
47 tasklet_hi_schedule(&engine->execlists.tasklet);
48
49 timeout += jiffies;
50 do {
51 bool done = time_after(jiffies, timeout);
52
53 if (i915_request_completed(rq)) /* that was quick! */
54 return 0;
55
56 /* Wait until the HW has acknowleged the submission (or err) */
57 intel_engine_flush_submission(engine);
58 if (!READ_ONCE(engine->execlists.pending[0]) && is_active(rq))
59 return 0;
60
61 if (done)
62 return -ETIME;
63
64 cond_resched();
65 } while (1);
66}
67
68static int wait_for_reset(struct intel_engine_cs *engine,
69 struct i915_request *rq,
70 unsigned long timeout)
71{
72 timeout += jiffies;
73
74 do {
75 cond_resched();
76 intel_engine_flush_submission(engine);
77
78 if (READ_ONCE(engine->execlists.pending[0]))
79 continue;
80
81 if (i915_request_completed(rq))
82 break;
83
84 if (READ_ONCE(rq->fence.error))
85 break;
86 } while (time_before(jiffies, timeout));
87
88 flush_scheduled_work();
89
90 if (rq->fence.error != -EIO) {
91 pr_err("%s: hanging request %llx:%lld not reset\n",
92 engine->name,
93 rq->fence.context,
94 rq->fence.seqno);
95 return -EINVAL;
96 }
97
98 /* Give the request a jiffie to complete after flushing the worker */
99 if (i915_request_wait(rq, 0,
100 max(0l, (long)(timeout - jiffies)) + 1) < 0) {
101 pr_err("%s: hanging request %llx:%lld did not complete\n",
102 engine->name,
103 rq->fence.context,
104 rq->fence.seqno);
105 return -ETIME;
106 }
107
108 return 0;
109}
110
111static int live_sanitycheck(void *arg)
112{
113 struct intel_gt *gt = arg;
114 struct intel_engine_cs *engine;
115 enum intel_engine_id id;
116 struct igt_spinner spin;
117 int err = 0;
118
119 if (!HAS_LOGICAL_RING_CONTEXTS(gt->i915))
120 return 0;
121
122 if (igt_spinner_init(&spin, gt))
123 return -ENOMEM;
124
125 for_each_engine(engine, gt, id) {
126 struct intel_context *ce;
127 struct i915_request *rq;
128
129 ce = intel_context_create(engine);
130 if (IS_ERR(ce)) {
131 err = PTR_ERR(ce);
132 break;
133 }
134
135 rq = igt_spinner_create_request(&spin, ce, MI_NOOP);
136 if (IS_ERR(rq)) {
137 err = PTR_ERR(rq);
138 goto out_ctx;
139 }
140
141 i915_request_add(rq);
142 if (!igt_wait_for_spinner(&spin, rq)) {
143 GEM_TRACE("spinner failed to start\n");
144 GEM_TRACE_DUMP();
145 intel_gt_set_wedged(gt);
146 err = -EIO;
147 goto out_ctx;
148 }
149
150 igt_spinner_end(&spin);
151 if (igt_flush_test(gt->i915)) {
152 err = -EIO;
153 goto out_ctx;
154 }
155
156out_ctx:
157 intel_context_put(ce);
158 if (err)
159 break;
160 }
161
162 igt_spinner_fini(&spin);
163 return err;
164}
165
166static int live_unlite_restore(struct intel_gt *gt, int prio)
167{
168 struct intel_engine_cs *engine;
169 enum intel_engine_id id;
170 struct igt_spinner spin;
171 int err = -ENOMEM;
172
173 /*
174 * Check that we can correctly context switch between 2 instances
175 * on the same engine from the same parent context.
176 */
177
178 if (igt_spinner_init(&spin, gt))
179 return err;
180
181 err = 0;
182 for_each_engine(engine, gt, id) {
183 struct intel_context *ce[2] = {};
184 struct i915_request *rq[2];
185 struct igt_live_test t;
186 int n;
187
188 if (prio && !intel_engine_has_preemption(engine))
189 continue;
190
191 if (!intel_engine_can_store_dword(engine))
192 continue;
193
194 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
195 err = -EIO;
196 break;
197 }
198 st_engine_heartbeat_disable(engine);
199
200 for (n = 0; n < ARRAY_SIZE(ce); n++) {
201 struct intel_context *tmp;
202
203 tmp = intel_context_create(engine);
204 if (IS_ERR(tmp)) {
205 err = PTR_ERR(tmp);
206 goto err_ce;
207 }
208
209 err = intel_context_pin(tmp);
210 if (err) {
211 intel_context_put(tmp);
212 goto err_ce;
213 }
214
215 /*
216 * Setup the pair of contexts such that if we
217 * lite-restore using the RING_TAIL from ce[1] it
218 * will execute garbage from ce[0]->ring.
219 */
220 memset(tmp->ring->vaddr,
221 POISON_INUSE, /* IPEHR: 0x5a5a5a5a [hung!] */
222 tmp->ring->vma->size);
223
224 ce[n] = tmp;
225 }
226 GEM_BUG_ON(!ce[1]->ring->size);
227 intel_ring_reset(ce[1]->ring, ce[1]->ring->size / 2);
228 lrc_update_regs(ce[1], engine, ce[1]->ring->head);
229
230 rq[0] = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
231 if (IS_ERR(rq[0])) {
232 err = PTR_ERR(rq[0]);
233 goto err_ce;
234 }
235
236 i915_request_get(rq[0]);
237 i915_request_add(rq[0]);
238 GEM_BUG_ON(rq[0]->postfix > ce[1]->ring->emit);
239
240 if (!igt_wait_for_spinner(&spin, rq[0])) {
241 i915_request_put(rq[0]);
242 goto err_ce;
243 }
244
245 rq[1] = i915_request_create(ce[1]);
246 if (IS_ERR(rq[1])) {
247 err = PTR_ERR(rq[1]);
248 i915_request_put(rq[0]);
249 goto err_ce;
250 }
251
252 if (!prio) {
253 /*
254 * Ensure we do the switch to ce[1] on completion.
255 *
256 * rq[0] is already submitted, so this should reduce
257 * to a no-op (a wait on a request on the same engine
258 * uses the submit fence, not the completion fence),
259 * but it will install a dependency on rq[1] for rq[0]
260 * that will prevent the pair being reordered by
261 * timeslicing.
262 */
263 i915_request_await_dma_fence(rq[1], &rq[0]->fence);
264 }
265
266 i915_request_get(rq[1]);
267 i915_request_add(rq[1]);
268 GEM_BUG_ON(rq[1]->postfix <= rq[0]->postfix);
269 i915_request_put(rq[0]);
270
271 if (prio) {
272 struct i915_sched_attr attr = {
273 .priority = prio,
274 };
275
276 /* Alternatively preempt the spinner with ce[1] */
277 engine->schedule(rq[1], &attr);
278 }
279
280 /* And switch back to ce[0] for good measure */
281 rq[0] = i915_request_create(ce[0]);
282 if (IS_ERR(rq[0])) {
283 err = PTR_ERR(rq[0]);
284 i915_request_put(rq[1]);
285 goto err_ce;
286 }
287
288 i915_request_await_dma_fence(rq[0], &rq[1]->fence);
289 i915_request_get(rq[0]);
290 i915_request_add(rq[0]);
291 GEM_BUG_ON(rq[0]->postfix > rq[1]->postfix);
292 i915_request_put(rq[1]);
293 i915_request_put(rq[0]);
294
295err_ce:
296 intel_engine_flush_submission(engine);
297 igt_spinner_end(&spin);
298 for (n = 0; n < ARRAY_SIZE(ce); n++) {
299 if (IS_ERR_OR_NULL(ce[n]))
300 break;
301
302 intel_context_unpin(ce[n]);
303 intel_context_put(ce[n]);
304 }
305
306 st_engine_heartbeat_enable(engine);
307 if (igt_live_test_end(&t))
308 err = -EIO;
309 if (err)
310 break;
311 }
312
313 igt_spinner_fini(&spin);
314 return err;
315}
316
317static int live_unlite_switch(void *arg)
318{
319 return live_unlite_restore(arg, 0);
320}
321
322static int live_unlite_preempt(void *arg)
323{
324 return live_unlite_restore(arg, I915_USER_PRIORITY(I915_PRIORITY_MAX));
325}
326
327static int live_unlite_ring(void *arg)
328{
329 struct intel_gt *gt = arg;
330 struct intel_engine_cs *engine;
331 struct igt_spinner spin;
332 enum intel_engine_id id;
333 int err = 0;
334
335 /*
336 * Setup a preemption event that will cause almost the entire ring
337 * to be unwound, potentially fooling our intel_ring_direction()
338 * into emitting a forward lite-restore instead of the rollback.
339 */
340
341 if (igt_spinner_init(&spin, gt))
342 return -ENOMEM;
343
344 for_each_engine(engine, gt, id) {
345 struct intel_context *ce[2] = {};
346 struct i915_request *rq;
347 struct igt_live_test t;
348 int n;
349
350 if (!intel_engine_has_preemption(engine))
351 continue;
352
353 if (!intel_engine_can_store_dword(engine))
354 continue;
355
356 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
357 err = -EIO;
358 break;
359 }
360 st_engine_heartbeat_disable(engine);
361
362 for (n = 0; n < ARRAY_SIZE(ce); n++) {
363 struct intel_context *tmp;
364
365 tmp = intel_context_create(engine);
366 if (IS_ERR(tmp)) {
367 err = PTR_ERR(tmp);
368 goto err_ce;
369 }
370
371 err = intel_context_pin(tmp);
372 if (err) {
373 intel_context_put(tmp);
374 goto err_ce;
375 }
376
377 memset32(tmp->ring->vaddr,
378 0xdeadbeef, /* trigger a hang if executed */
379 tmp->ring->vma->size / sizeof(u32));
380
381 ce[n] = tmp;
382 }
383
384 /* Create max prio spinner, followed by N low prio nops */
385 rq = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
386 if (IS_ERR(rq)) {
387 err = PTR_ERR(rq);
388 goto err_ce;
389 }
390
391 i915_request_get(rq);
392 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
393 i915_request_add(rq);
394
395 if (!igt_wait_for_spinner(&spin, rq)) {
396 intel_gt_set_wedged(gt);
397 i915_request_put(rq);
398 err = -ETIME;
399 goto err_ce;
400 }
401
402 /* Fill the ring, until we will cause a wrap */
403 n = 0;
404 while (intel_ring_direction(ce[0]->ring,
405 rq->wa_tail,
406 ce[0]->ring->tail) <= 0) {
407 struct i915_request *tmp;
408
409 tmp = intel_context_create_request(ce[0]);
410 if (IS_ERR(tmp)) {
411 err = PTR_ERR(tmp);
412 i915_request_put(rq);
413 goto err_ce;
414 }
415
416 i915_request_add(tmp);
417 intel_engine_flush_submission(engine);
418 n++;
419 }
420 intel_engine_flush_submission(engine);
421 pr_debug("%s: Filled ring with %d nop tails {size:%x, tail:%x, emit:%x, rq.tail:%x}\n",
422 engine->name, n,
423 ce[0]->ring->size,
424 ce[0]->ring->tail,
425 ce[0]->ring->emit,
426 rq->tail);
427 GEM_BUG_ON(intel_ring_direction(ce[0]->ring,
428 rq->tail,
429 ce[0]->ring->tail) <= 0);
430 i915_request_put(rq);
431
432 /* Create a second ring to preempt the first ring after rq[0] */
433 rq = intel_context_create_request(ce[1]);
434 if (IS_ERR(rq)) {
435 err = PTR_ERR(rq);
436 goto err_ce;
437 }
438
439 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
440 i915_request_get(rq);
441 i915_request_add(rq);
442
443 err = wait_for_submit(engine, rq, HZ / 2);
444 i915_request_put(rq);
445 if (err) {
446 pr_err("%s: preemption request was not submitted\n",
447 engine->name);
448 err = -ETIME;
449 }
450
451 pr_debug("%s: ring[0]:{ tail:%x, emit:%x }, ring[1]:{ tail:%x, emit:%x }\n",
452 engine->name,
453 ce[0]->ring->tail, ce[0]->ring->emit,
454 ce[1]->ring->tail, ce[1]->ring->emit);
455
456err_ce:
457 intel_engine_flush_submission(engine);
458 igt_spinner_end(&spin);
459 for (n = 0; n < ARRAY_SIZE(ce); n++) {
460 if (IS_ERR_OR_NULL(ce[n]))
461 break;
462
463 intel_context_unpin(ce[n]);
464 intel_context_put(ce[n]);
465 }
466 st_engine_heartbeat_enable(engine);
467 if (igt_live_test_end(&t))
468 err = -EIO;
469 if (err)
470 break;
471 }
472
473 igt_spinner_fini(&spin);
474 return err;
475}
476
477static int live_pin_rewind(void *arg)
478{
479 struct intel_gt *gt = arg;
480 struct intel_engine_cs *engine;
481 enum intel_engine_id id;
482 int err = 0;
483
484 /*
485 * We have to be careful not to trust intel_ring too much, for example
486 * ring->head is updated upon retire which is out of sync with pinning
487 * the context. Thus we cannot use ring->head to set CTX_RING_HEAD,
488 * or else we risk writing an older, stale value.
489 *
490 * To simulate this, let's apply a bit of deliberate sabotague.
491 */
492
493 for_each_engine(engine, gt, id) {
494 struct intel_context *ce;
495 struct i915_request *rq;
496 struct intel_ring *ring;
497 struct igt_live_test t;
498
499 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
500 err = -EIO;
501 break;
502 }
503
504 ce = intel_context_create(engine);
505 if (IS_ERR(ce)) {
506 err = PTR_ERR(ce);
507 break;
508 }
509
510 err = intel_context_pin(ce);
511 if (err) {
512 intel_context_put(ce);
513 break;
514 }
515
516 /* Keep the context awake while we play games */
517 err = i915_active_acquire(&ce->active);
518 if (err) {
519 intel_context_unpin(ce);
520 intel_context_put(ce);
521 break;
522 }
523 ring = ce->ring;
524
525 /* Poison the ring, and offset the next request from HEAD */
526 memset32(ring->vaddr, STACK_MAGIC, ring->size / sizeof(u32));
527 ring->emit = ring->size / 2;
528 ring->tail = ring->emit;
529 GEM_BUG_ON(ring->head);
530
531 intel_context_unpin(ce);
532
533 /* Submit a simple nop request */
534 GEM_BUG_ON(intel_context_is_pinned(ce));
535 rq = intel_context_create_request(ce);
536 i915_active_release(&ce->active); /* e.g. async retire */
537 intel_context_put(ce);
538 if (IS_ERR(rq)) {
539 err = PTR_ERR(rq);
540 break;
541 }
542 GEM_BUG_ON(!rq->head);
543 i915_request_add(rq);
544
545 /* Expect not to hang! */
546 if (igt_live_test_end(&t)) {
547 err = -EIO;
548 break;
549 }
550 }
551
552 return err;
553}
554
555static int live_hold_reset(void *arg)
556{
557 struct intel_gt *gt = arg;
558 struct intel_engine_cs *engine;
559 enum intel_engine_id id;
560 struct igt_spinner spin;
561 int err = 0;
562
563 /*
564 * In order to support offline error capture for fast preempt reset,
565 * we need to decouple the guilty request and ensure that it and its
566 * descendents are not executed while the capture is in progress.
567 */
568
569 if (!intel_has_reset_engine(gt))
570 return 0;
571
572 if (igt_spinner_init(&spin, gt))
573 return -ENOMEM;
574
575 for_each_engine(engine, gt, id) {
576 struct intel_context *ce;
577 struct i915_request *rq;
578
579 ce = intel_context_create(engine);
580 if (IS_ERR(ce)) {
581 err = PTR_ERR(ce);
582 break;
583 }
584
585 st_engine_heartbeat_disable(engine);
586
587 rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
588 if (IS_ERR(rq)) {
589 err = PTR_ERR(rq);
590 goto out;
591 }
592 i915_request_add(rq);
593
594 if (!igt_wait_for_spinner(&spin, rq)) {
595 intel_gt_set_wedged(gt);
596 err = -ETIME;
597 goto out;
598 }
599
600 /* We have our request executing, now remove it and reset */
601
602 local_bh_disable();
603 if (test_and_set_bit(I915_RESET_ENGINE + id,
604 &gt->reset.flags)) {
605 local_bh_enable();
606 intel_gt_set_wedged(gt);
607 err = -EBUSY;
608 goto out;
609 }
610 tasklet_disable(&engine->execlists.tasklet);
611
612 engine->execlists.tasklet.func(engine->execlists.tasklet.data);
613 GEM_BUG_ON(execlists_active(&engine->execlists) != rq);
614
615 i915_request_get(rq);
616 execlists_hold(engine, rq);
617 GEM_BUG_ON(!i915_request_on_hold(rq));
618
619 __intel_engine_reset_bh(engine, NULL);
620 GEM_BUG_ON(rq->fence.error != -EIO);
621
622 tasklet_enable(&engine->execlists.tasklet);
623 clear_and_wake_up_bit(I915_RESET_ENGINE + id,
624 &gt->reset.flags);
625 local_bh_enable();
626
627 /* Check that we do not resubmit the held request */
628 if (!i915_request_wait(rq, 0, HZ / 5)) {
629 pr_err("%s: on hold request completed!\n",
630 engine->name);
631 i915_request_put(rq);
632 err = -EIO;
633 goto out;
634 }
635 GEM_BUG_ON(!i915_request_on_hold(rq));
636
637 /* But is resubmitted on release */
638 execlists_unhold(engine, rq);
639 if (i915_request_wait(rq, 0, HZ / 5) < 0) {
640 pr_err("%s: held request did not complete!\n",
641 engine->name);
642 intel_gt_set_wedged(gt);
643 err = -ETIME;
644 }
645 i915_request_put(rq);
646
647out:
648 st_engine_heartbeat_enable(engine);
649 intel_context_put(ce);
650 if (err)
651 break;
652 }
653
654 igt_spinner_fini(&spin);
655 return err;
656}
657
658static const char *error_repr(int err)
659{
660 return err ? "bad" : "good";
661}
662
663static int live_error_interrupt(void *arg)
664{
665 static const struct error_phase {
666 enum { GOOD = 0, BAD = -EIO } error[2];
667 } phases[] = {
668 { { BAD, GOOD } },
669 { { BAD, BAD } },
670 { { BAD, GOOD } },
671 { { GOOD, GOOD } }, /* sentinel */
672 };
673 struct intel_gt *gt = arg;
674 struct intel_engine_cs *engine;
675 enum intel_engine_id id;
676
677 /*
678 * We hook up the CS_MASTER_ERROR_INTERRUPT to have forewarning
679 * of invalid commands in user batches that will cause a GPU hang.
680 * This is a faster mechanism than using hangcheck/heartbeats, but
681 * only detects problems the HW knows about -- it will not warn when
682 * we kill the HW!
683 *
684 * To verify our detection and reset, we throw some invalid commands
685 * at the HW and wait for the interrupt.
686 */
687
688 if (!intel_has_reset_engine(gt))
689 return 0;
690
691 for_each_engine(engine, gt, id) {
692 const struct error_phase *p;
693 int err = 0;
694
695 st_engine_heartbeat_disable(engine);
696
697 for (p = phases; p->error[0] != GOOD; p++) {
698 struct i915_request *client[ARRAY_SIZE(phases->error)];
699 u32 *cs;
700 int i;
701
702 memset(client, 0, sizeof(*client));
703 for (i = 0; i < ARRAY_SIZE(client); i++) {
704 struct intel_context *ce;
705 struct i915_request *rq;
706
707 ce = intel_context_create(engine);
708 if (IS_ERR(ce)) {
709 err = PTR_ERR(ce);
710 goto out;
711 }
712
713 rq = intel_context_create_request(ce);
714 intel_context_put(ce);
715 if (IS_ERR(rq)) {
716 err = PTR_ERR(rq);
717 goto out;
718 }
719
720 if (rq->engine->emit_init_breadcrumb) {
721 err = rq->engine->emit_init_breadcrumb(rq);
722 if (err) {
723 i915_request_add(rq);
724 goto out;
725 }
726 }
727
728 cs = intel_ring_begin(rq, 2);
729 if (IS_ERR(cs)) {
730 i915_request_add(rq);
731 err = PTR_ERR(cs);
732 goto out;
733 }
734
735 if (p->error[i]) {
736 *cs++ = 0xdeadbeef;
737 *cs++ = 0xdeadbeef;
738 } else {
739 *cs++ = MI_NOOP;
740 *cs++ = MI_NOOP;
741 }
742
743 client[i] = i915_request_get(rq);
744 i915_request_add(rq);
745 }
746
747 err = wait_for_submit(engine, client[0], HZ / 2);
748 if (err) {
749 pr_err("%s: first request did not start within time!\n",
750 engine->name);
751 err = -ETIME;
752 goto out;
753 }
754
755 for (i = 0; i < ARRAY_SIZE(client); i++) {
756 if (i915_request_wait(client[i], 0, HZ / 5) < 0)
757 pr_debug("%s: %s request incomplete!\n",
758 engine->name,
759 error_repr(p->error[i]));
760
761 if (!i915_request_started(client[i])) {
762 pr_err("%s: %s request not started!\n",
763 engine->name,
764 error_repr(p->error[i]));
765 err = -ETIME;
766 goto out;
767 }
768
769 /* Kick the tasklet to process the error */
770 intel_engine_flush_submission(engine);
771 if (client[i]->fence.error != p->error[i]) {
772 pr_err("%s: %s request (%s) with wrong error code: %d\n",
773 engine->name,
774 error_repr(p->error[i]),
775 i915_request_completed(client[i]) ? "completed" : "running",
776 client[i]->fence.error);
777 err = -EINVAL;
778 goto out;
779 }
780 }
781
782out:
783 for (i = 0; i < ARRAY_SIZE(client); i++)
784 if (client[i])
785 i915_request_put(client[i]);
786 if (err) {
787 pr_err("%s: failed at phase[%zd] { %d, %d }\n",
788 engine->name, p - phases,
789 p->error[0], p->error[1]);
790 break;
791 }
792 }
793
794 st_engine_heartbeat_enable(engine);
795 if (err) {
796 intel_gt_set_wedged(gt);
797 return err;
798 }
799 }
800
801 return 0;
802}
803
804static int
805emit_semaphore_chain(struct i915_request *rq, struct i915_vma *vma, int idx)
806{
807 u32 *cs;
808
809 cs = intel_ring_begin(rq, 10);
810 if (IS_ERR(cs))
811 return PTR_ERR(cs);
812
813 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
814
815 *cs++ = MI_SEMAPHORE_WAIT |
816 MI_SEMAPHORE_GLOBAL_GTT |
817 MI_SEMAPHORE_POLL |
818 MI_SEMAPHORE_SAD_NEQ_SDD;
819 *cs++ = 0;
820 *cs++ = i915_ggtt_offset(vma) + 4 * idx;
821 *cs++ = 0;
822
823 if (idx > 0) {
824 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
825 *cs++ = i915_ggtt_offset(vma) + 4 * (idx - 1);
826 *cs++ = 0;
827 *cs++ = 1;
828 } else {
829 *cs++ = MI_NOOP;
830 *cs++ = MI_NOOP;
831 *cs++ = MI_NOOP;
832 *cs++ = MI_NOOP;
833 }
834
835 *cs++ = MI_ARB_ON_OFF | MI_ARB_DISABLE;
836
837 intel_ring_advance(rq, cs);
838 return 0;
839}
840
841static struct i915_request *
842semaphore_queue(struct intel_engine_cs *engine, struct i915_vma *vma, int idx)
843{
844 struct intel_context *ce;
845 struct i915_request *rq;
846 int err;
847
848 ce = intel_context_create(engine);
849 if (IS_ERR(ce))
850 return ERR_CAST(ce);
851
852 rq = intel_context_create_request(ce);
853 if (IS_ERR(rq))
854 goto out_ce;
855
856 err = 0;
857 if (rq->engine->emit_init_breadcrumb)
858 err = rq->engine->emit_init_breadcrumb(rq);
859 if (err == 0)
860 err = emit_semaphore_chain(rq, vma, idx);
861 if (err == 0)
862 i915_request_get(rq);
863 i915_request_add(rq);
864 if (err)
865 rq = ERR_PTR(err);
866
867out_ce:
868 intel_context_put(ce);
869 return rq;
870}
871
872static int
873release_queue(struct intel_engine_cs *engine,
874 struct i915_vma *vma,
875 int idx, int prio)
876{
877 struct i915_sched_attr attr = {
878 .priority = prio,
879 };
880 struct i915_request *rq;
881 u32 *cs;
882
883 rq = intel_engine_create_kernel_request(engine);
884 if (IS_ERR(rq))
885 return PTR_ERR(rq);
886
887 cs = intel_ring_begin(rq, 4);
888 if (IS_ERR(cs)) {
889 i915_request_add(rq);
890 return PTR_ERR(cs);
891 }
892
893 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
894 *cs++ = i915_ggtt_offset(vma) + 4 * (idx - 1);
895 *cs++ = 0;
896 *cs++ = 1;
897
898 intel_ring_advance(rq, cs);
899
900 i915_request_get(rq);
901 i915_request_add(rq);
902
903 local_bh_disable();
904 engine->schedule(rq, &attr);
905 local_bh_enable(); /* kick tasklet */
906
907 i915_request_put(rq);
908
909 return 0;
910}
911
912static int
913slice_semaphore_queue(struct intel_engine_cs *outer,
914 struct i915_vma *vma,
915 int count)
916{
917 struct intel_engine_cs *engine;
918 struct i915_request *head;
919 enum intel_engine_id id;
920 int err, i, n = 0;
921
922 head = semaphore_queue(outer, vma, n++);
923 if (IS_ERR(head))
924 return PTR_ERR(head);
925
926 for_each_engine(engine, outer->gt, id) {
927 if (!intel_engine_has_preemption(engine))
928 continue;
929
930 for (i = 0; i < count; i++) {
931 struct i915_request *rq;
932
933 rq = semaphore_queue(engine, vma, n++);
934 if (IS_ERR(rq)) {
935 err = PTR_ERR(rq);
936 goto out;
937 }
938
939 i915_request_put(rq);
940 }
941 }
942
943 err = release_queue(outer, vma, n, I915_PRIORITY_BARRIER);
944 if (err)
945 goto out;
946
947 if (i915_request_wait(head, 0,
948 2 * outer->gt->info.num_engines * (count + 2) * (count + 3)) < 0) {
949 pr_err("%s: Failed to slice along semaphore chain of length (%d, %d)!\n",
950 outer->name, count, n);
951 GEM_TRACE_DUMP();
952 intel_gt_set_wedged(outer->gt);
953 err = -EIO;
954 }
955
956out:
957 i915_request_put(head);
958 return err;
959}
960
961static int live_timeslice_preempt(void *arg)
962{
963 struct intel_gt *gt = arg;
964 struct drm_i915_gem_object *obj;
965 struct intel_engine_cs *engine;
966 enum intel_engine_id id;
967 struct i915_vma *vma;
968 void *vaddr;
969 int err = 0;
970
971 /*
972 * If a request takes too long, we would like to give other users
973 * a fair go on the GPU. In particular, users may create batches
974 * that wait upon external input, where that input may even be
975 * supplied by another GPU job. To avoid blocking forever, we
976 * need to preempt the current task and replace it with another
977 * ready task.
978 */
979 if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
980 return 0;
981
982 obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
983 if (IS_ERR(obj))
984 return PTR_ERR(obj);
985
986 vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
987 if (IS_ERR(vma)) {
988 err = PTR_ERR(vma);
989 goto err_obj;
990 }
991
992 vaddr = i915_gem_object_pin_map(obj, I915_MAP_WC);
993 if (IS_ERR(vaddr)) {
994 err = PTR_ERR(vaddr);
995 goto err_obj;
996 }
997
998 err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
999 if (err)
1000 goto err_map;
1001
1002 err = i915_vma_sync(vma);
1003 if (err)
1004 goto err_pin;
1005
1006 for_each_engine(engine, gt, id) {
1007 if (!intel_engine_has_preemption(engine))
1008 continue;
1009
1010 memset(vaddr, 0, PAGE_SIZE);
1011
1012 st_engine_heartbeat_disable(engine);
1013 err = slice_semaphore_queue(engine, vma, 5);
1014 st_engine_heartbeat_enable(engine);
1015 if (err)
1016 goto err_pin;
1017
1018 if (igt_flush_test(gt->i915)) {
1019 err = -EIO;
1020 goto err_pin;
1021 }
1022 }
1023
1024err_pin:
1025 i915_vma_unpin(vma);
1026err_map:
1027 i915_gem_object_unpin_map(obj);
1028err_obj:
1029 i915_gem_object_put(obj);
1030 return err;
1031}
1032
1033static struct i915_request *
1034create_rewinder(struct intel_context *ce,
1035 struct i915_request *wait,
1036 void *slot, int idx)
1037{
1038 const u32 offset =
1039 i915_ggtt_offset(ce->engine->status_page.vma) +
1040 offset_in_page(slot);
1041 struct i915_request *rq;
1042 u32 *cs;
1043 int err;
1044
1045 rq = intel_context_create_request(ce);
1046 if (IS_ERR(rq))
1047 return rq;
1048
1049 if (wait) {
1050 err = i915_request_await_dma_fence(rq, &wait->fence);
1051 if (err)
1052 goto err;
1053 }
1054
1055 cs = intel_ring_begin(rq, 14);
1056 if (IS_ERR(cs)) {
1057 err = PTR_ERR(cs);
1058 goto err;
1059 }
1060
1061 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
1062 *cs++ = MI_NOOP;
1063
1064 *cs++ = MI_SEMAPHORE_WAIT |
1065 MI_SEMAPHORE_GLOBAL_GTT |
1066 MI_SEMAPHORE_POLL |
1067 MI_SEMAPHORE_SAD_GTE_SDD;
1068 *cs++ = idx;
1069 *cs++ = offset;
1070 *cs++ = 0;
1071
1072 *cs++ = MI_STORE_REGISTER_MEM_GEN8 | MI_USE_GGTT;
1073 *cs++ = i915_mmio_reg_offset(RING_TIMESTAMP(rq->engine->mmio_base));
1074 *cs++ = offset + idx * sizeof(u32);
1075 *cs++ = 0;
1076
1077 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
1078 *cs++ = offset;
1079 *cs++ = 0;
1080 *cs++ = idx + 1;
1081
1082 intel_ring_advance(rq, cs);
1083
1084 rq->sched.attr.priority = I915_PRIORITY_MASK;
1085 err = 0;
1086err:
1087 i915_request_get(rq);
1088 i915_request_add(rq);
1089 if (err) {
1090 i915_request_put(rq);
1091 return ERR_PTR(err);
1092 }
1093
1094 return rq;
1095}
1096
1097static int live_timeslice_rewind(void *arg)
1098{
1099 struct intel_gt *gt = arg;
1100 struct intel_engine_cs *engine;
1101 enum intel_engine_id id;
1102
1103 /*
1104 * The usual presumption on timeslice expiration is that we replace
1105 * the active context with another. However, given a chain of
1106 * dependencies we may end up with replacing the context with itself,
1107 * but only a few of those requests, forcing us to rewind the
1108 * RING_TAIL of the original request.
1109 */
1110 if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
1111 return 0;
1112
1113 for_each_engine(engine, gt, id) {
1114 enum { A1, A2, B1 };
1115 enum { X = 1, Z, Y };
1116 struct i915_request *rq[3] = {};
1117 struct intel_context *ce;
1118 unsigned long timeslice;
1119 int i, err = 0;
1120 u32 *slot;
1121
1122 if (!intel_engine_has_timeslices(engine))
1123 continue;
1124
1125 /*
1126 * A:rq1 -- semaphore wait, timestamp X
1127 * A:rq2 -- write timestamp Y
1128 *
1129 * B:rq1 [await A:rq1] -- write timestamp Z
1130 *
1131 * Force timeslice, release semaphore.
1132 *
1133 * Expect execution/evaluation order XZY
1134 */
1135
1136 st_engine_heartbeat_disable(engine);
1137 timeslice = xchg(&engine->props.timeslice_duration_ms, 1);
1138
1139 slot = memset32(engine->status_page.addr + 1000, 0, 4);
1140
1141 ce = intel_context_create(engine);
1142 if (IS_ERR(ce)) {
1143 err = PTR_ERR(ce);
1144 goto err;
1145 }
1146
1147 rq[A1] = create_rewinder(ce, NULL, slot, X);
1148 if (IS_ERR(rq[A1])) {
1149 intel_context_put(ce);
1150 goto err;
1151 }
1152
1153 rq[A2] = create_rewinder(ce, NULL, slot, Y);
1154 intel_context_put(ce);
1155 if (IS_ERR(rq[A2]))
1156 goto err;
1157
1158 err = wait_for_submit(engine, rq[A2], HZ / 2);
1159 if (err) {
1160 pr_err("%s: failed to submit first context\n",
1161 engine->name);
1162 goto err;
1163 }
1164
1165 ce = intel_context_create(engine);
1166 if (IS_ERR(ce)) {
1167 err = PTR_ERR(ce);
1168 goto err;
1169 }
1170
1171 rq[B1] = create_rewinder(ce, rq[A1], slot, Z);
1172 intel_context_put(ce);
1173 if (IS_ERR(rq[2]))
1174 goto err;
1175
1176 err = wait_for_submit(engine, rq[B1], HZ / 2);
1177 if (err) {
1178 pr_err("%s: failed to submit second context\n",
1179 engine->name);
1180 goto err;
1181 }
1182
1183 /* ELSP[] = { { A:rq1, A:rq2 }, { B:rq1 } } */
1184 ENGINE_TRACE(engine, "forcing tasklet for rewind\n");
1185 while (i915_request_is_active(rq[A2])) { /* semaphore yield! */
1186 /* Wait for the timeslice to kick in */
1187 del_timer(&engine->execlists.timer);
1188 tasklet_hi_schedule(&engine->execlists.tasklet);
1189 intel_engine_flush_submission(engine);
1190 }
1191 /* -> ELSP[] = { { A:rq1 }, { B:rq1 } } */
1192 GEM_BUG_ON(!i915_request_is_active(rq[A1]));
1193 GEM_BUG_ON(!i915_request_is_active(rq[B1]));
1194 GEM_BUG_ON(i915_request_is_active(rq[A2]));
1195
1196 /* Release the hounds! */
1197 slot[0] = 1;
1198 wmb(); /* "pairs" with GPU; paranoid kick of internal CPU$ */
1199
1200 for (i = 1; i <= 3; i++) {
1201 unsigned long timeout = jiffies + HZ / 2;
1202
1203 while (!READ_ONCE(slot[i]) &&
1204 time_before(jiffies, timeout))
1205 ;
1206
1207 if (!time_before(jiffies, timeout)) {
1208 pr_err("%s: rq[%d] timed out\n",
1209 engine->name, i - 1);
1210 err = -ETIME;
1211 goto err;
1212 }
1213
1214 pr_debug("%s: slot[%d]:%x\n", engine->name, i, slot[i]);
1215 }
1216
1217 /* XZY: XZ < XY */
1218 if (slot[Z] - slot[X] >= slot[Y] - slot[X]) {
1219 pr_err("%s: timeslicing did not run context B [%u] before A [%u]!\n",
1220 engine->name,
1221 slot[Z] - slot[X],
1222 slot[Y] - slot[X]);
1223 err = -EINVAL;
1224 }
1225
1226err:
1227 memset32(&slot[0], -1, 4);
1228 wmb();
1229
1230 engine->props.timeslice_duration_ms = timeslice;
1231 st_engine_heartbeat_enable(engine);
1232 for (i = 0; i < 3; i++)
1233 i915_request_put(rq[i]);
1234 if (igt_flush_test(gt->i915))
1235 err = -EIO;
1236 if (err)
1237 return err;
1238 }
1239
1240 return 0;
1241}
1242
1243static struct i915_request *nop_request(struct intel_engine_cs *engine)
1244{
1245 struct i915_request *rq;
1246
1247 rq = intel_engine_create_kernel_request(engine);
1248 if (IS_ERR(rq))
1249 return rq;
1250
1251 i915_request_get(rq);
1252 i915_request_add(rq);
1253
1254 return rq;
1255}
1256
1257static long slice_timeout(struct intel_engine_cs *engine)
1258{
1259 long timeout;
1260
1261 /* Enough time for a timeslice to kick in, and kick out */
1262 timeout = 2 * msecs_to_jiffies_timeout(timeslice(engine));
1263
1264 /* Enough time for the nop request to complete */
1265 timeout += HZ / 5;
1266
1267 return timeout + 1;
1268}
1269
1270static int live_timeslice_queue(void *arg)
1271{
1272 struct intel_gt *gt = arg;
1273 struct drm_i915_gem_object *obj;
1274 struct intel_engine_cs *engine;
1275 enum intel_engine_id id;
1276 struct i915_vma *vma;
1277 void *vaddr;
1278 int err = 0;
1279
1280 /*
1281 * Make sure that even if ELSP[0] and ELSP[1] are filled with
1282 * timeslicing between them disabled, we *do* enable timeslicing
1283 * if the queue demands it. (Normally, we do not submit if
1284 * ELSP[1] is already occupied, so must rely on timeslicing to
1285 * eject ELSP[0] in favour of the queue.)
1286 */
1287 if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
1288 return 0;
1289
1290 obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
1291 if (IS_ERR(obj))
1292 return PTR_ERR(obj);
1293
1294 vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
1295 if (IS_ERR(vma)) {
1296 err = PTR_ERR(vma);
1297 goto err_obj;
1298 }
1299
1300 vaddr = i915_gem_object_pin_map(obj, I915_MAP_WC);
1301 if (IS_ERR(vaddr)) {
1302 err = PTR_ERR(vaddr);
1303 goto err_obj;
1304 }
1305
1306 err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
1307 if (err)
1308 goto err_map;
1309
1310 err = i915_vma_sync(vma);
1311 if (err)
1312 goto err_pin;
1313
1314 for_each_engine(engine, gt, id) {
1315 struct i915_sched_attr attr = {
1316 .priority = I915_USER_PRIORITY(I915_PRIORITY_MAX),
1317 };
1318 struct i915_request *rq, *nop;
1319
1320 if (!intel_engine_has_preemption(engine))
1321 continue;
1322
1323 st_engine_heartbeat_disable(engine);
1324 memset(vaddr, 0, PAGE_SIZE);
1325
1326 /* ELSP[0]: semaphore wait */
1327 rq = semaphore_queue(engine, vma, 0);
1328 if (IS_ERR(rq)) {
1329 err = PTR_ERR(rq);
1330 goto err_heartbeat;
1331 }
1332 engine->schedule(rq, &attr);
1333 err = wait_for_submit(engine, rq, HZ / 2);
1334 if (err) {
1335 pr_err("%s: Timed out trying to submit semaphores\n",
1336 engine->name);
1337 goto err_rq;
1338 }
1339
1340 /* ELSP[1]: nop request */
1341 nop = nop_request(engine);
1342 if (IS_ERR(nop)) {
1343 err = PTR_ERR(nop);
1344 goto err_rq;
1345 }
1346 err = wait_for_submit(engine, nop, HZ / 2);
1347 i915_request_put(nop);
1348 if (err) {
1349 pr_err("%s: Timed out trying to submit nop\n",
1350 engine->name);
1351 goto err_rq;
1352 }
1353
1354 GEM_BUG_ON(i915_request_completed(rq));
1355 GEM_BUG_ON(execlists_active(&engine->execlists) != rq);
1356
1357 /* Queue: semaphore signal, matching priority as semaphore */
1358 err = release_queue(engine, vma, 1, effective_prio(rq));
1359 if (err)
1360 goto err_rq;
1361
1362 /* Wait until we ack the release_queue and start timeslicing */
1363 do {
1364 cond_resched();
1365 intel_engine_flush_submission(engine);
1366 } while (READ_ONCE(engine->execlists.pending[0]));
1367
1368 /* Timeslice every jiffy, so within 2 we should signal */
1369 if (i915_request_wait(rq, 0, slice_timeout(engine)) < 0) {
1370 struct drm_printer p =
1371 drm_info_printer(gt->i915->drm.dev);
1372
1373 pr_err("%s: Failed to timeslice into queue\n",
1374 engine->name);
1375 intel_engine_dump(engine, &p,
1376 "%s\n", engine->name);
1377
1378 memset(vaddr, 0xff, PAGE_SIZE);
1379 err = -EIO;
1380 }
1381err_rq:
1382 i915_request_put(rq);
1383err_heartbeat:
1384 st_engine_heartbeat_enable(engine);
1385 if (err)
1386 break;
1387 }
1388
1389err_pin:
1390 i915_vma_unpin(vma);
1391err_map:
1392 i915_gem_object_unpin_map(obj);
1393err_obj:
1394 i915_gem_object_put(obj);
1395 return err;
1396}
1397
1398static int live_timeslice_nopreempt(void *arg)
1399{
1400 struct intel_gt *gt = arg;
1401 struct intel_engine_cs *engine;
1402 enum intel_engine_id id;
1403 struct igt_spinner spin;
1404 int err = 0;
1405
1406 /*
1407 * We should not timeslice into a request that is marked with
1408 * I915_REQUEST_NOPREEMPT.
1409 */
1410 if (!IS_ACTIVE(CONFIG_DRM_I915_TIMESLICE_DURATION))
1411 return 0;
1412
1413 if (igt_spinner_init(&spin, gt))
1414 return -ENOMEM;
1415
1416 for_each_engine(engine, gt, id) {
1417 struct intel_context *ce;
1418 struct i915_request *rq;
1419 unsigned long timeslice;
1420
1421 if (!intel_engine_has_preemption(engine))
1422 continue;
1423
1424 ce = intel_context_create(engine);
1425 if (IS_ERR(ce)) {
1426 err = PTR_ERR(ce);
1427 break;
1428 }
1429
1430 st_engine_heartbeat_disable(engine);
1431 timeslice = xchg(&engine->props.timeslice_duration_ms, 1);
1432
1433 /* Create an unpreemptible spinner */
1434
1435 rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
1436 intel_context_put(ce);
1437 if (IS_ERR(rq)) {
1438 err = PTR_ERR(rq);
1439 goto out_heartbeat;
1440 }
1441
1442 i915_request_get(rq);
1443 i915_request_add(rq);
1444
1445 if (!igt_wait_for_spinner(&spin, rq)) {
1446 i915_request_put(rq);
1447 err = -ETIME;
1448 goto out_spin;
1449 }
1450
1451 set_bit(I915_FENCE_FLAG_NOPREEMPT, &rq->fence.flags);
1452 i915_request_put(rq);
1453
1454 /* Followed by a maximum priority barrier (heartbeat) */
1455
1456 ce = intel_context_create(engine);
1457 if (IS_ERR(ce)) {
1458 err = PTR_ERR(ce);
1459 goto out_spin;
1460 }
1461
1462 rq = intel_context_create_request(ce);
1463 intel_context_put(ce);
1464 if (IS_ERR(rq)) {
1465 err = PTR_ERR(rq);
1466 goto out_spin;
1467 }
1468
1469 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
1470 i915_request_get(rq);
1471 i915_request_add(rq);
1472
1473 /*
1474 * Wait until the barrier is in ELSP, and we know timeslicing
1475 * will have been activated.
1476 */
1477 if (wait_for_submit(engine, rq, HZ / 2)) {
1478 i915_request_put(rq);
1479 err = -ETIME;
1480 goto out_spin;
1481 }
1482
1483 /*
1484 * Since the ELSP[0] request is unpreemptible, it should not
1485 * allow the maximum priority barrier through. Wait long
1486 * enough to see if it is timesliced in by mistake.
1487 */
1488 if (i915_request_wait(rq, 0, slice_timeout(engine)) >= 0) {
1489 pr_err("%s: I915_PRIORITY_BARRIER request completed, bypassing no-preempt request\n",
1490 engine->name);
1491 err = -EINVAL;
1492 }
1493 i915_request_put(rq);
1494
1495out_spin:
1496 igt_spinner_end(&spin);
1497out_heartbeat:
1498 xchg(&engine->props.timeslice_duration_ms, timeslice);
1499 st_engine_heartbeat_enable(engine);
1500 if (err)
1501 break;
1502
1503 if (igt_flush_test(gt->i915)) {
1504 err = -EIO;
1505 break;
1506 }
1507 }
1508
1509 igt_spinner_fini(&spin);
1510 return err;
1511}
1512
1513static int live_busywait_preempt(void *arg)
1514{
1515 struct intel_gt *gt = arg;
1516 struct i915_gem_context *ctx_hi, *ctx_lo;
1517 struct intel_engine_cs *engine;
1518 struct drm_i915_gem_object *obj;
1519 struct i915_vma *vma;
1520 enum intel_engine_id id;
1521 int err = -ENOMEM;
1522 u32 *map;
1523
1524 /*
1525 * Verify that even without HAS_LOGICAL_RING_PREEMPTION, we can
1526 * preempt the busywaits used to synchronise between rings.
1527 */
1528
1529 ctx_hi = kernel_context(gt->i915);
1530 if (!ctx_hi)
1531 return -ENOMEM;
1532 ctx_hi->sched.priority =
1533 I915_USER_PRIORITY(I915_CONTEXT_MAX_USER_PRIORITY);
1534
1535 ctx_lo = kernel_context(gt->i915);
1536 if (!ctx_lo)
1537 goto err_ctx_hi;
1538 ctx_lo->sched.priority =
1539 I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
1540
1541 obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
1542 if (IS_ERR(obj)) {
1543 err = PTR_ERR(obj);
1544 goto err_ctx_lo;
1545 }
1546
1547 map = i915_gem_object_pin_map(obj, I915_MAP_WC);
1548 if (IS_ERR(map)) {
1549 err = PTR_ERR(map);
1550 goto err_obj;
1551 }
1552
1553 vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
1554 if (IS_ERR(vma)) {
1555 err = PTR_ERR(vma);
1556 goto err_map;
1557 }
1558
1559 err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
1560 if (err)
1561 goto err_map;
1562
1563 err = i915_vma_sync(vma);
1564 if (err)
1565 goto err_vma;
1566
1567 for_each_engine(engine, gt, id) {
1568 struct i915_request *lo, *hi;
1569 struct igt_live_test t;
1570 u32 *cs;
1571
1572 if (!intel_engine_has_preemption(engine))
1573 continue;
1574
1575 if (!intel_engine_can_store_dword(engine))
1576 continue;
1577
1578 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
1579 err = -EIO;
1580 goto err_vma;
1581 }
1582
1583 /*
1584 * We create two requests. The low priority request
1585 * busywaits on a semaphore (inside the ringbuffer where
1586 * is should be preemptible) and the high priority requests
1587 * uses a MI_STORE_DWORD_IMM to update the semaphore value
1588 * allowing the first request to complete. If preemption
1589 * fails, we hang instead.
1590 */
1591
1592 lo = igt_request_alloc(ctx_lo, engine);
1593 if (IS_ERR(lo)) {
1594 err = PTR_ERR(lo);
1595 goto err_vma;
1596 }
1597
1598 cs = intel_ring_begin(lo, 8);
1599 if (IS_ERR(cs)) {
1600 err = PTR_ERR(cs);
1601 i915_request_add(lo);
1602 goto err_vma;
1603 }
1604
1605 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
1606 *cs++ = i915_ggtt_offset(vma);
1607 *cs++ = 0;
1608 *cs++ = 1;
1609
1610 /* XXX Do we need a flush + invalidate here? */
1611
1612 *cs++ = MI_SEMAPHORE_WAIT |
1613 MI_SEMAPHORE_GLOBAL_GTT |
1614 MI_SEMAPHORE_POLL |
1615 MI_SEMAPHORE_SAD_EQ_SDD;
1616 *cs++ = 0;
1617 *cs++ = i915_ggtt_offset(vma);
1618 *cs++ = 0;
1619
1620 intel_ring_advance(lo, cs);
1621
1622 i915_request_get(lo);
1623 i915_request_add(lo);
1624
1625 if (wait_for(READ_ONCE(*map), 10)) {
1626 i915_request_put(lo);
1627 err = -ETIMEDOUT;
1628 goto err_vma;
1629 }
1630
1631 /* Low priority request should be busywaiting now */
1632 if (i915_request_wait(lo, 0, 1) != -ETIME) {
1633 i915_request_put(lo);
1634 pr_err("%s: Busywaiting request did not!\n",
1635 engine->name);
1636 err = -EIO;
1637 goto err_vma;
1638 }
1639
1640 hi = igt_request_alloc(ctx_hi, engine);
1641 if (IS_ERR(hi)) {
1642 err = PTR_ERR(hi);
1643 i915_request_put(lo);
1644 goto err_vma;
1645 }
1646
1647 cs = intel_ring_begin(hi, 4);
1648 if (IS_ERR(cs)) {
1649 err = PTR_ERR(cs);
1650 i915_request_add(hi);
1651 i915_request_put(lo);
1652 goto err_vma;
1653 }
1654
1655 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
1656 *cs++ = i915_ggtt_offset(vma);
1657 *cs++ = 0;
1658 *cs++ = 0;
1659
1660 intel_ring_advance(hi, cs);
1661 i915_request_add(hi);
1662
1663 if (i915_request_wait(lo, 0, HZ / 5) < 0) {
1664 struct drm_printer p = drm_info_printer(gt->i915->drm.dev);
1665
1666 pr_err("%s: Failed to preempt semaphore busywait!\n",
1667 engine->name);
1668
1669 intel_engine_dump(engine, &p, "%s\n", engine->name);
1670 GEM_TRACE_DUMP();
1671
1672 i915_request_put(lo);
1673 intel_gt_set_wedged(gt);
1674 err = -EIO;
1675 goto err_vma;
1676 }
1677 GEM_BUG_ON(READ_ONCE(*map));
1678 i915_request_put(lo);
1679
1680 if (igt_live_test_end(&t)) {
1681 err = -EIO;
1682 goto err_vma;
1683 }
1684 }
1685
1686 err = 0;
1687err_vma:
1688 i915_vma_unpin(vma);
1689err_map:
1690 i915_gem_object_unpin_map(obj);
1691err_obj:
1692 i915_gem_object_put(obj);
1693err_ctx_lo:
1694 kernel_context_close(ctx_lo);
1695err_ctx_hi:
1696 kernel_context_close(ctx_hi);
1697 return err;
1698}
1699
1700static struct i915_request *
1701spinner_create_request(struct igt_spinner *spin,
1702 struct i915_gem_context *ctx,
1703 struct intel_engine_cs *engine,
1704 u32 arb)
1705{
1706 struct intel_context *ce;
1707 struct i915_request *rq;
1708
1709 ce = i915_gem_context_get_engine(ctx, engine->legacy_idx);
1710 if (IS_ERR(ce))
1711 return ERR_CAST(ce);
1712
1713 rq = igt_spinner_create_request(spin, ce, arb);
1714 intel_context_put(ce);
1715 return rq;
1716}
1717
1718static int live_preempt(void *arg)
1719{
1720 struct intel_gt *gt = arg;
1721 struct i915_gem_context *ctx_hi, *ctx_lo;
1722 struct igt_spinner spin_hi, spin_lo;
1723 struct intel_engine_cs *engine;
1724 enum intel_engine_id id;
1725 int err = -ENOMEM;
1726
1727 if (igt_spinner_init(&spin_hi, gt))
1728 return -ENOMEM;
1729
1730 if (igt_spinner_init(&spin_lo, gt))
1731 goto err_spin_hi;
1732
1733 ctx_hi = kernel_context(gt->i915);
1734 if (!ctx_hi)
1735 goto err_spin_lo;
1736 ctx_hi->sched.priority =
1737 I915_USER_PRIORITY(I915_CONTEXT_MAX_USER_PRIORITY);
1738
1739 ctx_lo = kernel_context(gt->i915);
1740 if (!ctx_lo)
1741 goto err_ctx_hi;
1742 ctx_lo->sched.priority =
1743 I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
1744
1745 for_each_engine(engine, gt, id) {
1746 struct igt_live_test t;
1747 struct i915_request *rq;
1748
1749 if (!intel_engine_has_preemption(engine))
1750 continue;
1751
1752 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
1753 err = -EIO;
1754 goto err_ctx_lo;
1755 }
1756
1757 rq = spinner_create_request(&spin_lo, ctx_lo, engine,
1758 MI_ARB_CHECK);
1759 if (IS_ERR(rq)) {
1760 err = PTR_ERR(rq);
1761 goto err_ctx_lo;
1762 }
1763
1764 i915_request_add(rq);
1765 if (!igt_wait_for_spinner(&spin_lo, rq)) {
1766 GEM_TRACE("lo spinner failed to start\n");
1767 GEM_TRACE_DUMP();
1768 intel_gt_set_wedged(gt);
1769 err = -EIO;
1770 goto err_ctx_lo;
1771 }
1772
1773 rq = spinner_create_request(&spin_hi, ctx_hi, engine,
1774 MI_ARB_CHECK);
1775 if (IS_ERR(rq)) {
1776 igt_spinner_end(&spin_lo);
1777 err = PTR_ERR(rq);
1778 goto err_ctx_lo;
1779 }
1780
1781 i915_request_add(rq);
1782 if (!igt_wait_for_spinner(&spin_hi, rq)) {
1783 GEM_TRACE("hi spinner failed to start\n");
1784 GEM_TRACE_DUMP();
1785 intel_gt_set_wedged(gt);
1786 err = -EIO;
1787 goto err_ctx_lo;
1788 }
1789
1790 igt_spinner_end(&spin_hi);
1791 igt_spinner_end(&spin_lo);
1792
1793 if (igt_live_test_end(&t)) {
1794 err = -EIO;
1795 goto err_ctx_lo;
1796 }
1797 }
1798
1799 err = 0;
1800err_ctx_lo:
1801 kernel_context_close(ctx_lo);
1802err_ctx_hi:
1803 kernel_context_close(ctx_hi);
1804err_spin_lo:
1805 igt_spinner_fini(&spin_lo);
1806err_spin_hi:
1807 igt_spinner_fini(&spin_hi);
1808 return err;
1809}
1810
1811static int live_late_preempt(void *arg)
1812{
1813 struct intel_gt *gt = arg;
1814 struct i915_gem_context *ctx_hi, *ctx_lo;
1815 struct igt_spinner spin_hi, spin_lo;
1816 struct intel_engine_cs *engine;
1817 struct i915_sched_attr attr = {};
1818 enum intel_engine_id id;
1819 int err = -ENOMEM;
1820
1821 if (igt_spinner_init(&spin_hi, gt))
1822 return -ENOMEM;
1823
1824 if (igt_spinner_init(&spin_lo, gt))
1825 goto err_spin_hi;
1826
1827 ctx_hi = kernel_context(gt->i915);
1828 if (!ctx_hi)
1829 goto err_spin_lo;
1830
1831 ctx_lo = kernel_context(gt->i915);
1832 if (!ctx_lo)
1833 goto err_ctx_hi;
1834
1835 /* Make sure ctx_lo stays before ctx_hi until we trigger preemption. */
1836 ctx_lo->sched.priority = I915_USER_PRIORITY(1);
1837
1838 for_each_engine(engine, gt, id) {
1839 struct igt_live_test t;
1840 struct i915_request *rq;
1841
1842 if (!intel_engine_has_preemption(engine))
1843 continue;
1844
1845 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
1846 err = -EIO;
1847 goto err_ctx_lo;
1848 }
1849
1850 rq = spinner_create_request(&spin_lo, ctx_lo, engine,
1851 MI_ARB_CHECK);
1852 if (IS_ERR(rq)) {
1853 err = PTR_ERR(rq);
1854 goto err_ctx_lo;
1855 }
1856
1857 i915_request_add(rq);
1858 if (!igt_wait_for_spinner(&spin_lo, rq)) {
1859 pr_err("First context failed to start\n");
1860 goto err_wedged;
1861 }
1862
1863 rq = spinner_create_request(&spin_hi, ctx_hi, engine,
1864 MI_NOOP);
1865 if (IS_ERR(rq)) {
1866 igt_spinner_end(&spin_lo);
1867 err = PTR_ERR(rq);
1868 goto err_ctx_lo;
1869 }
1870
1871 i915_request_add(rq);
1872 if (igt_wait_for_spinner(&spin_hi, rq)) {
1873 pr_err("Second context overtook first?\n");
1874 goto err_wedged;
1875 }
1876
1877 attr.priority = I915_USER_PRIORITY(I915_PRIORITY_MAX);
1878 engine->schedule(rq, &attr);
1879
1880 if (!igt_wait_for_spinner(&spin_hi, rq)) {
1881 pr_err("High priority context failed to preempt the low priority context\n");
1882 GEM_TRACE_DUMP();
1883 goto err_wedged;
1884 }
1885
1886 igt_spinner_end(&spin_hi);
1887 igt_spinner_end(&spin_lo);
1888
1889 if (igt_live_test_end(&t)) {
1890 err = -EIO;
1891 goto err_ctx_lo;
1892 }
1893 }
1894
1895 err = 0;
1896err_ctx_lo:
1897 kernel_context_close(ctx_lo);
1898err_ctx_hi:
1899 kernel_context_close(ctx_hi);
1900err_spin_lo:
1901 igt_spinner_fini(&spin_lo);
1902err_spin_hi:
1903 igt_spinner_fini(&spin_hi);
1904 return err;
1905
1906err_wedged:
1907 igt_spinner_end(&spin_hi);
1908 igt_spinner_end(&spin_lo);
1909 intel_gt_set_wedged(gt);
1910 err = -EIO;
1911 goto err_ctx_lo;
1912}
1913
1914struct preempt_client {
1915 struct igt_spinner spin;
1916 struct i915_gem_context *ctx;
1917};
1918
1919static int preempt_client_init(struct intel_gt *gt, struct preempt_client *c)
1920{
1921 c->ctx = kernel_context(gt->i915);
1922 if (!c->ctx)
1923 return -ENOMEM;
1924
1925 if (igt_spinner_init(&c->spin, gt))
1926 goto err_ctx;
1927
1928 return 0;
1929
1930err_ctx:
1931 kernel_context_close(c->ctx);
1932 return -ENOMEM;
1933}
1934
1935static void preempt_client_fini(struct preempt_client *c)
1936{
1937 igt_spinner_fini(&c->spin);
1938 kernel_context_close(c->ctx);
1939}
1940
1941static int live_nopreempt(void *arg)
1942{
1943 struct intel_gt *gt = arg;
1944 struct intel_engine_cs *engine;
1945 struct preempt_client a, b;
1946 enum intel_engine_id id;
1947 int err = -ENOMEM;
1948
1949 /*
1950 * Verify that we can disable preemption for an individual request
1951 * that may be being observed and not want to be interrupted.
1952 */
1953
1954 if (preempt_client_init(gt, &a))
1955 return -ENOMEM;
1956 if (preempt_client_init(gt, &b))
1957 goto err_client_a;
1958 b.ctx->sched.priority = I915_USER_PRIORITY(I915_PRIORITY_MAX);
1959
1960 for_each_engine(engine, gt, id) {
1961 struct i915_request *rq_a, *rq_b;
1962
1963 if (!intel_engine_has_preemption(engine))
1964 continue;
1965
1966 engine->execlists.preempt_hang.count = 0;
1967
1968 rq_a = spinner_create_request(&a.spin,
1969 a.ctx, engine,
1970 MI_ARB_CHECK);
1971 if (IS_ERR(rq_a)) {
1972 err = PTR_ERR(rq_a);
1973 goto err_client_b;
1974 }
1975
1976 /* Low priority client, but unpreemptable! */
1977 __set_bit(I915_FENCE_FLAG_NOPREEMPT, &rq_a->fence.flags);
1978
1979 i915_request_add(rq_a);
1980 if (!igt_wait_for_spinner(&a.spin, rq_a)) {
1981 pr_err("First client failed to start\n");
1982 goto err_wedged;
1983 }
1984
1985 rq_b = spinner_create_request(&b.spin,
1986 b.ctx, engine,
1987 MI_ARB_CHECK);
1988 if (IS_ERR(rq_b)) {
1989 err = PTR_ERR(rq_b);
1990 goto err_client_b;
1991 }
1992
1993 i915_request_add(rq_b);
1994
1995 /* B is much more important than A! (But A is unpreemptable.) */
1996 GEM_BUG_ON(rq_prio(rq_b) <= rq_prio(rq_a));
1997
1998 /* Wait long enough for preemption and timeslicing */
1999 if (igt_wait_for_spinner(&b.spin, rq_b)) {
2000 pr_err("Second client started too early!\n");
2001 goto err_wedged;
2002 }
2003
2004 igt_spinner_end(&a.spin);
2005
2006 if (!igt_wait_for_spinner(&b.spin, rq_b)) {
2007 pr_err("Second client failed to start\n");
2008 goto err_wedged;
2009 }
2010
2011 igt_spinner_end(&b.spin);
2012
2013 if (engine->execlists.preempt_hang.count) {
2014 pr_err("Preemption recorded x%d; should have been suppressed!\n",
2015 engine->execlists.preempt_hang.count);
2016 err = -EINVAL;
2017 goto err_wedged;
2018 }
2019
2020 if (igt_flush_test(gt->i915))
2021 goto err_wedged;
2022 }
2023
2024 err = 0;
2025err_client_b:
2026 preempt_client_fini(&b);
2027err_client_a:
2028 preempt_client_fini(&a);
2029 return err;
2030
2031err_wedged:
2032 igt_spinner_end(&b.spin);
2033 igt_spinner_end(&a.spin);
2034 intel_gt_set_wedged(gt);
2035 err = -EIO;
2036 goto err_client_b;
2037}
2038
2039struct live_preempt_cancel {
2040 struct intel_engine_cs *engine;
2041 struct preempt_client a, b;
2042};
2043
2044static int __cancel_active0(struct live_preempt_cancel *arg)
2045{
2046 struct i915_request *rq;
2047 struct igt_live_test t;
2048 int err;
2049
2050 /* Preempt cancel of ELSP0 */
2051 GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2052 if (igt_live_test_begin(&t, arg->engine->i915,
2053 __func__, arg->engine->name))
2054 return -EIO;
2055
2056 rq = spinner_create_request(&arg->a.spin,
2057 arg->a.ctx, arg->engine,
2058 MI_ARB_CHECK);
2059 if (IS_ERR(rq))
2060 return PTR_ERR(rq);
2061
2062 clear_bit(CONTEXT_BANNED, &rq->context->flags);
2063 i915_request_get(rq);
2064 i915_request_add(rq);
2065 if (!igt_wait_for_spinner(&arg->a.spin, rq)) {
2066 err = -EIO;
2067 goto out;
2068 }
2069
2070 intel_context_set_banned(rq->context);
2071 err = intel_engine_pulse(arg->engine);
2072 if (err)
2073 goto out;
2074
2075 err = wait_for_reset(arg->engine, rq, HZ / 2);
2076 if (err) {
2077 pr_err("Cancelled inflight0 request did not reset\n");
2078 goto out;
2079 }
2080
2081out:
2082 i915_request_put(rq);
2083 if (igt_live_test_end(&t))
2084 err = -EIO;
2085 return err;
2086}
2087
2088static int __cancel_active1(struct live_preempt_cancel *arg)
2089{
2090 struct i915_request *rq[2] = {};
2091 struct igt_live_test t;
2092 int err;
2093
2094 /* Preempt cancel of ELSP1 */
2095 GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2096 if (igt_live_test_begin(&t, arg->engine->i915,
2097 __func__, arg->engine->name))
2098 return -EIO;
2099
2100 rq[0] = spinner_create_request(&arg->a.spin,
2101 arg->a.ctx, arg->engine,
2102 MI_NOOP); /* no preemption */
2103 if (IS_ERR(rq[0]))
2104 return PTR_ERR(rq[0]);
2105
2106 clear_bit(CONTEXT_BANNED, &rq[0]->context->flags);
2107 i915_request_get(rq[0]);
2108 i915_request_add(rq[0]);
2109 if (!igt_wait_for_spinner(&arg->a.spin, rq[0])) {
2110 err = -EIO;
2111 goto out;
2112 }
2113
2114 rq[1] = spinner_create_request(&arg->b.spin,
2115 arg->b.ctx, arg->engine,
2116 MI_ARB_CHECK);
2117 if (IS_ERR(rq[1])) {
2118 err = PTR_ERR(rq[1]);
2119 goto out;
2120 }
2121
2122 clear_bit(CONTEXT_BANNED, &rq[1]->context->flags);
2123 i915_request_get(rq[1]);
2124 err = i915_request_await_dma_fence(rq[1], &rq[0]->fence);
2125 i915_request_add(rq[1]);
2126 if (err)
2127 goto out;
2128
2129 intel_context_set_banned(rq[1]->context);
2130 err = intel_engine_pulse(arg->engine);
2131 if (err)
2132 goto out;
2133
2134 igt_spinner_end(&arg->a.spin);
2135 err = wait_for_reset(arg->engine, rq[1], HZ / 2);
2136 if (err)
2137 goto out;
2138
2139 if (rq[0]->fence.error != 0) {
2140 pr_err("Normal inflight0 request did not complete\n");
2141 err = -EINVAL;
2142 goto out;
2143 }
2144
2145 if (rq[1]->fence.error != -EIO) {
2146 pr_err("Cancelled inflight1 request did not report -EIO\n");
2147 err = -EINVAL;
2148 goto out;
2149 }
2150
2151out:
2152 i915_request_put(rq[1]);
2153 i915_request_put(rq[0]);
2154 if (igt_live_test_end(&t))
2155 err = -EIO;
2156 return err;
2157}
2158
2159static int __cancel_queued(struct live_preempt_cancel *arg)
2160{
2161 struct i915_request *rq[3] = {};
2162 struct igt_live_test t;
2163 int err;
2164
2165 /* Full ELSP and one in the wings */
2166 GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2167 if (igt_live_test_begin(&t, arg->engine->i915,
2168 __func__, arg->engine->name))
2169 return -EIO;
2170
2171 rq[0] = spinner_create_request(&arg->a.spin,
2172 arg->a.ctx, arg->engine,
2173 MI_ARB_CHECK);
2174 if (IS_ERR(rq[0]))
2175 return PTR_ERR(rq[0]);
2176
2177 clear_bit(CONTEXT_BANNED, &rq[0]->context->flags);
2178 i915_request_get(rq[0]);
2179 i915_request_add(rq[0]);
2180 if (!igt_wait_for_spinner(&arg->a.spin, rq[0])) {
2181 err = -EIO;
2182 goto out;
2183 }
2184
2185 rq[1] = igt_request_alloc(arg->b.ctx, arg->engine);
2186 if (IS_ERR(rq[1])) {
2187 err = PTR_ERR(rq[1]);
2188 goto out;
2189 }
2190
2191 clear_bit(CONTEXT_BANNED, &rq[1]->context->flags);
2192 i915_request_get(rq[1]);
2193 err = i915_request_await_dma_fence(rq[1], &rq[0]->fence);
2194 i915_request_add(rq[1]);
2195 if (err)
2196 goto out;
2197
2198 rq[2] = spinner_create_request(&arg->b.spin,
2199 arg->a.ctx, arg->engine,
2200 MI_ARB_CHECK);
2201 if (IS_ERR(rq[2])) {
2202 err = PTR_ERR(rq[2]);
2203 goto out;
2204 }
2205
2206 i915_request_get(rq[2]);
2207 err = i915_request_await_dma_fence(rq[2], &rq[1]->fence);
2208 i915_request_add(rq[2]);
2209 if (err)
2210 goto out;
2211
2212 intel_context_set_banned(rq[2]->context);
2213 err = intel_engine_pulse(arg->engine);
2214 if (err)
2215 goto out;
2216
2217 err = wait_for_reset(arg->engine, rq[2], HZ / 2);
2218 if (err)
2219 goto out;
2220
2221 if (rq[0]->fence.error != -EIO) {
2222 pr_err("Cancelled inflight0 request did not report -EIO\n");
2223 err = -EINVAL;
2224 goto out;
2225 }
2226
2227 if (rq[1]->fence.error != 0) {
2228 pr_err("Normal inflight1 request did not complete\n");
2229 err = -EINVAL;
2230 goto out;
2231 }
2232
2233 if (rq[2]->fence.error != -EIO) {
2234 pr_err("Cancelled queued request did not report -EIO\n");
2235 err = -EINVAL;
2236 goto out;
2237 }
2238
2239out:
2240 i915_request_put(rq[2]);
2241 i915_request_put(rq[1]);
2242 i915_request_put(rq[0]);
2243 if (igt_live_test_end(&t))
2244 err = -EIO;
2245 return err;
2246}
2247
2248static int __cancel_hostile(struct live_preempt_cancel *arg)
2249{
2250 struct i915_request *rq;
2251 int err;
2252
2253 /* Preempt cancel non-preemptible spinner in ELSP0 */
2254 if (!IS_ACTIVE(CONFIG_DRM_I915_PREEMPT_TIMEOUT))
2255 return 0;
2256
2257 if (!intel_has_reset_engine(arg->engine->gt))
2258 return 0;
2259
2260 GEM_TRACE("%s(%s)\n", __func__, arg->engine->name);
2261 rq = spinner_create_request(&arg->a.spin,
2262 arg->a.ctx, arg->engine,
2263 MI_NOOP); /* preemption disabled */
2264 if (IS_ERR(rq))
2265 return PTR_ERR(rq);
2266
2267 clear_bit(CONTEXT_BANNED, &rq->context->flags);
2268 i915_request_get(rq);
2269 i915_request_add(rq);
2270 if (!igt_wait_for_spinner(&arg->a.spin, rq)) {
2271 err = -EIO;
2272 goto out;
2273 }
2274
2275 intel_context_set_banned(rq->context);
2276 err = intel_engine_pulse(arg->engine); /* force reset */
2277 if (err)
2278 goto out;
2279
2280 err = wait_for_reset(arg->engine, rq, HZ / 2);
2281 if (err) {
2282 pr_err("Cancelled inflight0 request did not reset\n");
2283 goto out;
2284 }
2285
2286out:
2287 i915_request_put(rq);
2288 if (igt_flush_test(arg->engine->i915))
2289 err = -EIO;
2290 return err;
2291}
2292
2293static void force_reset_timeout(struct intel_engine_cs *engine)
2294{
2295 engine->reset_timeout.probability = 999;
2296 atomic_set(&engine->reset_timeout.times, -1);
2297}
2298
2299static void cancel_reset_timeout(struct intel_engine_cs *engine)
2300{
2301 memset(&engine->reset_timeout, 0, sizeof(engine->reset_timeout));
2302}
2303
2304static int __cancel_fail(struct live_preempt_cancel *arg)
2305{
2306 struct intel_engine_cs *engine = arg->engine;
2307 struct i915_request *rq;
2308 int err;
2309
2310 if (!IS_ACTIVE(CONFIG_DRM_I915_PREEMPT_TIMEOUT))
2311 return 0;
2312
2313 if (!intel_has_reset_engine(engine->gt))
2314 return 0;
2315
2316 GEM_TRACE("%s(%s)\n", __func__, engine->name);
2317 rq = spinner_create_request(&arg->a.spin,
2318 arg->a.ctx, engine,
2319 MI_NOOP); /* preemption disabled */
2320 if (IS_ERR(rq))
2321 return PTR_ERR(rq);
2322
2323 clear_bit(CONTEXT_BANNED, &rq->context->flags);
2324 i915_request_get(rq);
2325 i915_request_add(rq);
2326 if (!igt_wait_for_spinner(&arg->a.spin, rq)) {
2327 err = -EIO;
2328 goto out;
2329 }
2330
2331 intel_context_set_banned(rq->context);
2332
2333 err = intel_engine_pulse(engine);
2334 if (err)
2335 goto out;
2336
2337 force_reset_timeout(engine);
2338
2339 /* force preempt reset [failure] */
2340 while (!engine->execlists.pending[0])
2341 intel_engine_flush_submission(engine);
2342 del_timer_sync(&engine->execlists.preempt);
2343 intel_engine_flush_submission(engine);
2344
2345 cancel_reset_timeout(engine);
2346
2347 /* after failure, require heartbeats to reset device */
2348 intel_engine_set_heartbeat(engine, 1);
2349 err = wait_for_reset(engine, rq, HZ / 2);
2350 intel_engine_set_heartbeat(engine,
2351 engine->defaults.heartbeat_interval_ms);
2352 if (err) {
2353 pr_err("Cancelled inflight0 request did not reset\n");
2354 goto out;
2355 }
2356
2357out:
2358 i915_request_put(rq);
2359 if (igt_flush_test(engine->i915))
2360 err = -EIO;
2361 return err;
2362}
2363
2364static int live_preempt_cancel(void *arg)
2365{
2366 struct intel_gt *gt = arg;
2367 struct live_preempt_cancel data;
2368 enum intel_engine_id id;
2369 int err = -ENOMEM;
2370
2371 /*
2372 * To cancel an inflight context, we need to first remove it from the
2373 * GPU. That sounds like preemption! Plus a little bit of bookkeeping.
2374 */
2375
2376 if (preempt_client_init(gt, &data.a))
2377 return -ENOMEM;
2378 if (preempt_client_init(gt, &data.b))
2379 goto err_client_a;
2380
2381 for_each_engine(data.engine, gt, id) {
2382 if (!intel_engine_has_preemption(data.engine))
2383 continue;
2384
2385 err = __cancel_active0(&data);
2386 if (err)
2387 goto err_wedged;
2388
2389 err = __cancel_active1(&data);
2390 if (err)
2391 goto err_wedged;
2392
2393 err = __cancel_queued(&data);
2394 if (err)
2395 goto err_wedged;
2396
2397 err = __cancel_hostile(&data);
2398 if (err)
2399 goto err_wedged;
2400
2401 err = __cancel_fail(&data);
2402 if (err)
2403 goto err_wedged;
2404 }
2405
2406 err = 0;
2407err_client_b:
2408 preempt_client_fini(&data.b);
2409err_client_a:
2410 preempt_client_fini(&data.a);
2411 return err;
2412
2413err_wedged:
2414 GEM_TRACE_DUMP();
2415 igt_spinner_end(&data.b.spin);
2416 igt_spinner_end(&data.a.spin);
2417 intel_gt_set_wedged(gt);
2418 goto err_client_b;
2419}
2420
2421static int live_suppress_self_preempt(void *arg)
2422{
2423 struct intel_gt *gt = arg;
2424 struct intel_engine_cs *engine;
2425 struct i915_sched_attr attr = {
2426 .priority = I915_USER_PRIORITY(I915_PRIORITY_MAX)
2427 };
2428 struct preempt_client a, b;
2429 enum intel_engine_id id;
2430 int err = -ENOMEM;
2431
2432 /*
2433 * Verify that if a preemption request does not cause a change in
2434 * the current execution order, the preempt-to-idle injection is
2435 * skipped and that we do not accidentally apply it after the CS
2436 * completion event.
2437 */
2438
2439 if (intel_uc_uses_guc_submission(&gt->uc))
2440 return 0; /* presume black blox */
2441
2442 if (intel_vgpu_active(gt->i915))
2443 return 0; /* GVT forces single port & request submission */
2444
2445 if (preempt_client_init(gt, &a))
2446 return -ENOMEM;
2447 if (preempt_client_init(gt, &b))
2448 goto err_client_a;
2449
2450 for_each_engine(engine, gt, id) {
2451 struct i915_request *rq_a, *rq_b;
2452 int depth;
2453
2454 if (!intel_engine_has_preemption(engine))
2455 continue;
2456
2457 if (igt_flush_test(gt->i915))
2458 goto err_wedged;
2459
2460 st_engine_heartbeat_disable(engine);
2461 engine->execlists.preempt_hang.count = 0;
2462
2463 rq_a = spinner_create_request(&a.spin,
2464 a.ctx, engine,
2465 MI_NOOP);
2466 if (IS_ERR(rq_a)) {
2467 err = PTR_ERR(rq_a);
2468 st_engine_heartbeat_enable(engine);
2469 goto err_client_b;
2470 }
2471
2472 i915_request_add(rq_a);
2473 if (!igt_wait_for_spinner(&a.spin, rq_a)) {
2474 pr_err("First client failed to start\n");
2475 st_engine_heartbeat_enable(engine);
2476 goto err_wedged;
2477 }
2478
2479 /* Keep postponing the timer to avoid premature slicing */
2480 mod_timer(&engine->execlists.timer, jiffies + HZ);
2481 for (depth = 0; depth < 8; depth++) {
2482 rq_b = spinner_create_request(&b.spin,
2483 b.ctx, engine,
2484 MI_NOOP);
2485 if (IS_ERR(rq_b)) {
2486 err = PTR_ERR(rq_b);
2487 st_engine_heartbeat_enable(engine);
2488 goto err_client_b;
2489 }
2490 i915_request_add(rq_b);
2491
2492 GEM_BUG_ON(i915_request_completed(rq_a));
2493 engine->schedule(rq_a, &attr);
2494 igt_spinner_end(&a.spin);
2495
2496 if (!igt_wait_for_spinner(&b.spin, rq_b)) {
2497 pr_err("Second client failed to start\n");
2498 st_engine_heartbeat_enable(engine);
2499 goto err_wedged;
2500 }
2501
2502 swap(a, b);
2503 rq_a = rq_b;
2504 }
2505 igt_spinner_end(&a.spin);
2506
2507 if (engine->execlists.preempt_hang.count) {
2508 pr_err("Preemption on %s recorded x%d, depth %d; should have been suppressed!\n",
2509 engine->name,
2510 engine->execlists.preempt_hang.count,
2511 depth);
2512 st_engine_heartbeat_enable(engine);
2513 err = -EINVAL;
2514 goto err_client_b;
2515 }
2516
2517 st_engine_heartbeat_enable(engine);
2518 if (igt_flush_test(gt->i915))
2519 goto err_wedged;
2520 }
2521
2522 err = 0;
2523err_client_b:
2524 preempt_client_fini(&b);
2525err_client_a:
2526 preempt_client_fini(&a);
2527 return err;
2528
2529err_wedged:
2530 igt_spinner_end(&b.spin);
2531 igt_spinner_end(&a.spin);
2532 intel_gt_set_wedged(gt);
2533 err = -EIO;
2534 goto err_client_b;
2535}
2536
2537static int live_chain_preempt(void *arg)
2538{
2539 struct intel_gt *gt = arg;
2540 struct intel_engine_cs *engine;
2541 struct preempt_client hi, lo;
2542 enum intel_engine_id id;
2543 int err = -ENOMEM;
2544
2545 /*
2546 * Build a chain AB...BA between two contexts (A, B) and request
2547 * preemption of the last request. It should then complete before
2548 * the previously submitted spinner in B.
2549 */
2550
2551 if (preempt_client_init(gt, &hi))
2552 return -ENOMEM;
2553
2554 if (preempt_client_init(gt, &lo))
2555 goto err_client_hi;
2556
2557 for_each_engine(engine, gt, id) {
2558 struct i915_sched_attr attr = {
2559 .priority = I915_USER_PRIORITY(I915_PRIORITY_MAX),
2560 };
2561 struct igt_live_test t;
2562 struct i915_request *rq;
2563 int ring_size, count, i;
2564
2565 if (!intel_engine_has_preemption(engine))
2566 continue;
2567
2568 rq = spinner_create_request(&lo.spin,
2569 lo.ctx, engine,
2570 MI_ARB_CHECK);
2571 if (IS_ERR(rq))
2572 goto err_wedged;
2573
2574 i915_request_get(rq);
2575 i915_request_add(rq);
2576
2577 ring_size = rq->wa_tail - rq->head;
2578 if (ring_size < 0)
2579 ring_size += rq->ring->size;
2580 ring_size = rq->ring->size / ring_size;
2581 pr_debug("%s(%s): Using maximum of %d requests\n",
2582 __func__, engine->name, ring_size);
2583
2584 igt_spinner_end(&lo.spin);
2585 if (i915_request_wait(rq, 0, HZ / 2) < 0) {
2586 pr_err("Timed out waiting to flush %s\n", engine->name);
2587 i915_request_put(rq);
2588 goto err_wedged;
2589 }
2590 i915_request_put(rq);
2591
2592 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
2593 err = -EIO;
2594 goto err_wedged;
2595 }
2596
2597 for_each_prime_number_from(count, 1, ring_size) {
2598 rq = spinner_create_request(&hi.spin,
2599 hi.ctx, engine,
2600 MI_ARB_CHECK);
2601 if (IS_ERR(rq))
2602 goto err_wedged;
2603 i915_request_add(rq);
2604 if (!igt_wait_for_spinner(&hi.spin, rq))
2605 goto err_wedged;
2606
2607 rq = spinner_create_request(&lo.spin,
2608 lo.ctx, engine,
2609 MI_ARB_CHECK);
2610 if (IS_ERR(rq))
2611 goto err_wedged;
2612 i915_request_add(rq);
2613
2614 for (i = 0; i < count; i++) {
2615 rq = igt_request_alloc(lo.ctx, engine);
2616 if (IS_ERR(rq))
2617 goto err_wedged;
2618 i915_request_add(rq);
2619 }
2620
2621 rq = igt_request_alloc(hi.ctx, engine);
2622 if (IS_ERR(rq))
2623 goto err_wedged;
2624
2625 i915_request_get(rq);
2626 i915_request_add(rq);
2627 engine->schedule(rq, &attr);
2628
2629 igt_spinner_end(&hi.spin);
2630 if (i915_request_wait(rq, 0, HZ / 5) < 0) {
2631 struct drm_printer p =
2632 drm_info_printer(gt->i915->drm.dev);
2633
2634 pr_err("Failed to preempt over chain of %d\n",
2635 count);
2636 intel_engine_dump(engine, &p,
2637 "%s\n", engine->name);
2638 i915_request_put(rq);
2639 goto err_wedged;
2640 }
2641 igt_spinner_end(&lo.spin);
2642 i915_request_put(rq);
2643
2644 rq = igt_request_alloc(lo.ctx, engine);
2645 if (IS_ERR(rq))
2646 goto err_wedged;
2647
2648 i915_request_get(rq);
2649 i915_request_add(rq);
2650
2651 if (i915_request_wait(rq, 0, HZ / 5) < 0) {
2652 struct drm_printer p =
2653 drm_info_printer(gt->i915->drm.dev);
2654
2655 pr_err("Failed to flush low priority chain of %d requests\n",
2656 count);
2657 intel_engine_dump(engine, &p,
2658 "%s\n", engine->name);
2659
2660 i915_request_put(rq);
2661 goto err_wedged;
2662 }
2663 i915_request_put(rq);
2664 }
2665
2666 if (igt_live_test_end(&t)) {
2667 err = -EIO;
2668 goto err_wedged;
2669 }
2670 }
2671
2672 err = 0;
2673err_client_lo:
2674 preempt_client_fini(&lo);
2675err_client_hi:
2676 preempt_client_fini(&hi);
2677 return err;
2678
2679err_wedged:
2680 igt_spinner_end(&hi.spin);
2681 igt_spinner_end(&lo.spin);
2682 intel_gt_set_wedged(gt);
2683 err = -EIO;
2684 goto err_client_lo;
2685}
2686
2687static int create_gang(struct intel_engine_cs *engine,
2688 struct i915_request **prev)
2689{
2690 struct drm_i915_gem_object *obj;
2691 struct intel_context *ce;
2692 struct i915_request *rq;
2693 struct i915_vma *vma;
2694 u32 *cs;
2695 int err;
2696
2697 ce = intel_context_create(engine);
2698 if (IS_ERR(ce))
2699 return PTR_ERR(ce);
2700
2701 obj = i915_gem_object_create_internal(engine->i915, 4096);
2702 if (IS_ERR(obj)) {
2703 err = PTR_ERR(obj);
2704 goto err_ce;
2705 }
2706
2707 vma = i915_vma_instance(obj, ce->vm, NULL);
2708 if (IS_ERR(vma)) {
2709 err = PTR_ERR(vma);
2710 goto err_obj;
2711 }
2712
2713 err = i915_vma_pin(vma, 0, 0, PIN_USER);
2714 if (err)
2715 goto err_obj;
2716
2717 cs = i915_gem_object_pin_map(obj, I915_MAP_WC);
2718 if (IS_ERR(cs)) {
2719 err = PTR_ERR(cs);
2720 goto err_obj;
2721 }
2722
2723 /* Semaphore target: spin until zero */
2724 *cs++ = MI_ARB_ON_OFF | MI_ARB_ENABLE;
2725
2726 *cs++ = MI_SEMAPHORE_WAIT |
2727 MI_SEMAPHORE_POLL |
2728 MI_SEMAPHORE_SAD_EQ_SDD;
2729 *cs++ = 0;
2730 *cs++ = lower_32_bits(vma->node.start);
2731 *cs++ = upper_32_bits(vma->node.start);
2732
2733 if (*prev) {
2734 u64 offset = (*prev)->batch->node.start;
2735
2736 /* Terminate the spinner in the next lower priority batch. */
2737 *cs++ = MI_STORE_DWORD_IMM_GEN4;
2738 *cs++ = lower_32_bits(offset);
2739 *cs++ = upper_32_bits(offset);
2740 *cs++ = 0;
2741 }
2742
2743 *cs++ = MI_BATCH_BUFFER_END;
2744 i915_gem_object_flush_map(obj);
2745 i915_gem_object_unpin_map(obj);
2746
2747 rq = intel_context_create_request(ce);
2748 if (IS_ERR(rq)) {
2749 err = PTR_ERR(rq);
2750 goto err_obj;
2751 }
2752
2753 rq->batch = i915_vma_get(vma);
2754 i915_request_get(rq);
2755
2756 i915_vma_lock(vma);
2757 err = i915_request_await_object(rq, vma->obj, false);
2758 if (!err)
2759 err = i915_vma_move_to_active(vma, rq, 0);
2760 if (!err)
2761 err = rq->engine->emit_bb_start(rq,
2762 vma->node.start,
2763 PAGE_SIZE, 0);
2764 i915_vma_unlock(vma);
2765 i915_request_add(rq);
2766 if (err)
2767 goto err_rq;
2768
2769 i915_gem_object_put(obj);
2770 intel_context_put(ce);
2771
2772 rq->mock.link.next = &(*prev)->mock.link;
2773 *prev = rq;
2774 return 0;
2775
2776err_rq:
2777 i915_vma_put(rq->batch);
2778 i915_request_put(rq);
2779err_obj:
2780 i915_gem_object_put(obj);
2781err_ce:
2782 intel_context_put(ce);
2783 return err;
2784}
2785
2786static int __live_preempt_ring(struct intel_engine_cs *engine,
2787 struct igt_spinner *spin,
2788 int queue_sz, int ring_sz)
2789{
2790 struct intel_context *ce[2] = {};
2791 struct i915_request *rq;
2792 struct igt_live_test t;
2793 int err = 0;
2794 int n;
2795
2796 if (igt_live_test_begin(&t, engine->i915, __func__, engine->name))
2797 return -EIO;
2798
2799 for (n = 0; n < ARRAY_SIZE(ce); n++) {
2800 struct intel_context *tmp;
2801
2802 tmp = intel_context_create(engine);
2803 if (IS_ERR(tmp)) {
2804 err = PTR_ERR(tmp);
2805 goto err_ce;
2806 }
2807
2808 tmp->ring = __intel_context_ring_size(ring_sz);
2809
2810 err = intel_context_pin(tmp);
2811 if (err) {
2812 intel_context_put(tmp);
2813 goto err_ce;
2814 }
2815
2816 memset32(tmp->ring->vaddr,
2817 0xdeadbeef, /* trigger a hang if executed */
2818 tmp->ring->vma->size / sizeof(u32));
2819
2820 ce[n] = tmp;
2821 }
2822
2823 rq = igt_spinner_create_request(spin, ce[0], MI_ARB_CHECK);
2824 if (IS_ERR(rq)) {
2825 err = PTR_ERR(rq);
2826 goto err_ce;
2827 }
2828
2829 i915_request_get(rq);
2830 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
2831 i915_request_add(rq);
2832
2833 if (!igt_wait_for_spinner(spin, rq)) {
2834 intel_gt_set_wedged(engine->gt);
2835 i915_request_put(rq);
2836 err = -ETIME;
2837 goto err_ce;
2838 }
2839
2840 /* Fill the ring, until we will cause a wrap */
2841 n = 0;
2842 while (ce[0]->ring->tail - rq->wa_tail <= queue_sz) {
2843 struct i915_request *tmp;
2844
2845 tmp = intel_context_create_request(ce[0]);
2846 if (IS_ERR(tmp)) {
2847 err = PTR_ERR(tmp);
2848 i915_request_put(rq);
2849 goto err_ce;
2850 }
2851
2852 i915_request_add(tmp);
2853 intel_engine_flush_submission(engine);
2854 n++;
2855 }
2856 intel_engine_flush_submission(engine);
2857 pr_debug("%s: Filled %d with %d nop tails {size:%x, tail:%x, emit:%x, rq.tail:%x}\n",
2858 engine->name, queue_sz, n,
2859 ce[0]->ring->size,
2860 ce[0]->ring->tail,
2861 ce[0]->ring->emit,
2862 rq->tail);
2863 i915_request_put(rq);
2864
2865 /* Create a second request to preempt the first ring */
2866 rq = intel_context_create_request(ce[1]);
2867 if (IS_ERR(rq)) {
2868 err = PTR_ERR(rq);
2869 goto err_ce;
2870 }
2871
2872 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
2873 i915_request_get(rq);
2874 i915_request_add(rq);
2875
2876 err = wait_for_submit(engine, rq, HZ / 2);
2877 i915_request_put(rq);
2878 if (err) {
2879 pr_err("%s: preemption request was not submited\n",
2880 engine->name);
2881 err = -ETIME;
2882 }
2883
2884 pr_debug("%s: ring[0]:{ tail:%x, emit:%x }, ring[1]:{ tail:%x, emit:%x }\n",
2885 engine->name,
2886 ce[0]->ring->tail, ce[0]->ring->emit,
2887 ce[1]->ring->tail, ce[1]->ring->emit);
2888
2889err_ce:
2890 intel_engine_flush_submission(engine);
2891 igt_spinner_end(spin);
2892 for (n = 0; n < ARRAY_SIZE(ce); n++) {
2893 if (IS_ERR_OR_NULL(ce[n]))
2894 break;
2895
2896 intel_context_unpin(ce[n]);
2897 intel_context_put(ce[n]);
2898 }
2899 if (igt_live_test_end(&t))
2900 err = -EIO;
2901 return err;
2902}
2903
2904static int live_preempt_ring(void *arg)
2905{
2906 struct intel_gt *gt = arg;
2907 struct intel_engine_cs *engine;
2908 struct igt_spinner spin;
2909 enum intel_engine_id id;
2910 int err = 0;
2911
2912 /*
2913 * Check that we rollback large chunks of a ring in order to do a
2914 * preemption event. Similar to live_unlite_ring, but looking at
2915 * ring size rather than the impact of intel_ring_direction().
2916 */
2917
2918 if (igt_spinner_init(&spin, gt))
2919 return -ENOMEM;
2920
2921 for_each_engine(engine, gt, id) {
2922 int n;
2923
2924 if (!intel_engine_has_preemption(engine))
2925 continue;
2926
2927 if (!intel_engine_can_store_dword(engine))
2928 continue;
2929
2930 st_engine_heartbeat_disable(engine);
2931
2932 for (n = 0; n <= 3; n++) {
2933 err = __live_preempt_ring(engine, &spin,
2934 n * SZ_4K / 4, SZ_4K);
2935 if (err)
2936 break;
2937 }
2938
2939 st_engine_heartbeat_enable(engine);
2940 if (err)
2941 break;
2942 }
2943
2944 igt_spinner_fini(&spin);
2945 return err;
2946}
2947
2948static int live_preempt_gang(void *arg)
2949{
2950 struct intel_gt *gt = arg;
2951 struct intel_engine_cs *engine;
2952 enum intel_engine_id id;
2953
2954 /*
2955 * Build as long a chain of preempters as we can, with each
2956 * request higher priority than the last. Once we are ready, we release
2957 * the last batch which then precolates down the chain, each releasing
2958 * the next oldest in turn. The intent is to simply push as hard as we
2959 * can with the number of preemptions, trying to exceed narrow HW
2960 * limits. At a minimum, we insist that we can sort all the user
2961 * high priority levels into execution order.
2962 */
2963
2964 for_each_engine(engine, gt, id) {
2965 struct i915_request *rq = NULL;
2966 struct igt_live_test t;
2967 IGT_TIMEOUT(end_time);
2968 int prio = 0;
2969 int err = 0;
2970 u32 *cs;
2971
2972 if (!intel_engine_has_preemption(engine))
2973 continue;
2974
2975 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name))
2976 return -EIO;
2977
2978 do {
2979 struct i915_sched_attr attr = {
2980 .priority = I915_USER_PRIORITY(prio++),
2981 };
2982
2983 err = create_gang(engine, &rq);
2984 if (err)
2985 break;
2986
2987 /* Submit each spinner at increasing priority */
2988 engine->schedule(rq, &attr);
2989 } while (prio <= I915_PRIORITY_MAX &&
2990 !__igt_timeout(end_time, NULL));
2991 pr_debug("%s: Preempt chain of %d requests\n",
2992 engine->name, prio);
2993
2994 /*
2995 * Such that the last spinner is the highest priority and
2996 * should execute first. When that spinner completes,
2997 * it will terminate the next lowest spinner until there
2998 * are no more spinners and the gang is complete.
2999 */
3000 cs = i915_gem_object_pin_map(rq->batch->obj, I915_MAP_WC);
3001 if (!IS_ERR(cs)) {
3002 *cs = 0;
3003 i915_gem_object_unpin_map(rq->batch->obj);
3004 } else {
3005 err = PTR_ERR(cs);
3006 intel_gt_set_wedged(gt);
3007 }
3008
3009 while (rq) { /* wait for each rq from highest to lowest prio */
3010 struct i915_request *n = list_next_entry(rq, mock.link);
3011
3012 if (err == 0 && i915_request_wait(rq, 0, HZ / 5) < 0) {
3013 struct drm_printer p =
3014 drm_info_printer(engine->i915->drm.dev);
3015
3016 pr_err("Failed to flush chain of %d requests, at %d\n",
3017 prio, rq_prio(rq) >> I915_USER_PRIORITY_SHIFT);
3018 intel_engine_dump(engine, &p,
3019 "%s\n", engine->name);
3020
3021 err = -ETIME;
3022 }
3023
3024 i915_vma_put(rq->batch);
3025 i915_request_put(rq);
3026 rq = n;
3027 }
3028
3029 if (igt_live_test_end(&t))
3030 err = -EIO;
3031 if (err)
3032 return err;
3033 }
3034
3035 return 0;
3036}
3037
3038static struct i915_vma *
3039create_gpr_user(struct intel_engine_cs *engine,
3040 struct i915_vma *result,
3041 unsigned int offset)
3042{
3043 struct drm_i915_gem_object *obj;
3044 struct i915_vma *vma;
3045 u32 *cs;
3046 int err;
3047 int i;
3048
3049 obj = i915_gem_object_create_internal(engine->i915, 4096);
3050 if (IS_ERR(obj))
3051 return ERR_CAST(obj);
3052
3053 vma = i915_vma_instance(obj, result->vm, NULL);
3054 if (IS_ERR(vma)) {
3055 i915_gem_object_put(obj);
3056 return vma;
3057 }
3058
3059 err = i915_vma_pin(vma, 0, 0, PIN_USER);
3060 if (err) {
3061 i915_vma_put(vma);
3062 return ERR_PTR(err);
3063 }
3064
3065 cs = i915_gem_object_pin_map(obj, I915_MAP_WC);
3066 if (IS_ERR(cs)) {
3067 i915_vma_put(vma);
3068 return ERR_CAST(cs);
3069 }
3070
3071 /* All GPR are clear for new contexts. We use GPR(0) as a constant */
3072 *cs++ = MI_LOAD_REGISTER_IMM(1);
3073 *cs++ = CS_GPR(engine, 0);
3074 *cs++ = 1;
3075
3076 for (i = 1; i < NUM_GPR; i++) {
3077 u64 addr;
3078
3079 /*
3080 * Perform: GPR[i]++
3081 *
3082 * As we read and write into the context saved GPR[i], if
3083 * we restart this batch buffer from an earlier point, we
3084 * will repeat the increment and store a value > 1.
3085 */
3086 *cs++ = MI_MATH(4);
3087 *cs++ = MI_MATH_LOAD(MI_MATH_REG_SRCA, MI_MATH_REG(i));
3088 *cs++ = MI_MATH_LOAD(MI_MATH_REG_SRCB, MI_MATH_REG(0));
3089 *cs++ = MI_MATH_ADD;
3090 *cs++ = MI_MATH_STORE(MI_MATH_REG(i), MI_MATH_REG_ACCU);
3091
3092 addr = result->node.start + offset + i * sizeof(*cs);
3093 *cs++ = MI_STORE_REGISTER_MEM_GEN8;
3094 *cs++ = CS_GPR(engine, 2 * i);
3095 *cs++ = lower_32_bits(addr);
3096 *cs++ = upper_32_bits(addr);
3097
3098 *cs++ = MI_SEMAPHORE_WAIT |
3099 MI_SEMAPHORE_POLL |
3100 MI_SEMAPHORE_SAD_GTE_SDD;
3101 *cs++ = i;
3102 *cs++ = lower_32_bits(result->node.start);
3103 *cs++ = upper_32_bits(result->node.start);
3104 }
3105
3106 *cs++ = MI_BATCH_BUFFER_END;
3107 i915_gem_object_flush_map(obj);
3108 i915_gem_object_unpin_map(obj);
3109
3110 return vma;
3111}
3112
3113static struct i915_vma *create_global(struct intel_gt *gt, size_t sz)
3114{
3115 struct drm_i915_gem_object *obj;
3116 struct i915_vma *vma;
3117 int err;
3118
3119 obj = i915_gem_object_create_internal(gt->i915, sz);
3120 if (IS_ERR(obj))
3121 return ERR_CAST(obj);
3122
3123 vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
3124 if (IS_ERR(vma)) {
3125 i915_gem_object_put(obj);
3126 return vma;
3127 }
3128
3129 err = i915_ggtt_pin(vma, NULL, 0, 0);
3130 if (err) {
3131 i915_vma_put(vma);
3132 return ERR_PTR(err);
3133 }
3134
3135 return vma;
3136}
3137
3138static struct i915_request *
3139create_gpr_client(struct intel_engine_cs *engine,
3140 struct i915_vma *global,
3141 unsigned int offset)
3142{
3143 struct i915_vma *batch, *vma;
3144 struct intel_context *ce;
3145 struct i915_request *rq;
3146 int err;
3147
3148 ce = intel_context_create(engine);
3149 if (IS_ERR(ce))
3150 return ERR_CAST(ce);
3151
3152 vma = i915_vma_instance(global->obj, ce->vm, NULL);
3153 if (IS_ERR(vma)) {
3154 err = PTR_ERR(vma);
3155 goto out_ce;
3156 }
3157
3158 err = i915_vma_pin(vma, 0, 0, PIN_USER);
3159 if (err)
3160 goto out_ce;
3161
3162 batch = create_gpr_user(engine, vma, offset);
3163 if (IS_ERR(batch)) {
3164 err = PTR_ERR(batch);
3165 goto out_vma;
3166 }
3167
3168 rq = intel_context_create_request(ce);
3169 if (IS_ERR(rq)) {
3170 err = PTR_ERR(rq);
3171 goto out_batch;
3172 }
3173
3174 i915_vma_lock(vma);
3175 err = i915_request_await_object(rq, vma->obj, false);
3176 if (!err)
3177 err = i915_vma_move_to_active(vma, rq, 0);
3178 i915_vma_unlock(vma);
3179
3180 i915_vma_lock(batch);
3181 if (!err)
3182 err = i915_request_await_object(rq, batch->obj, false);
3183 if (!err)
3184 err = i915_vma_move_to_active(batch, rq, 0);
3185 if (!err)
3186 err = rq->engine->emit_bb_start(rq,
3187 batch->node.start,
3188 PAGE_SIZE, 0);
3189 i915_vma_unlock(batch);
3190 i915_vma_unpin(batch);
3191
3192 if (!err)
3193 i915_request_get(rq);
3194 i915_request_add(rq);
3195
3196out_batch:
3197 i915_vma_put(batch);
3198out_vma:
3199 i915_vma_unpin(vma);
3200out_ce:
3201 intel_context_put(ce);
3202 return err ? ERR_PTR(err) : rq;
3203}
3204
3205static int preempt_user(struct intel_engine_cs *engine,
3206 struct i915_vma *global,
3207 int id)
3208{
3209 struct i915_sched_attr attr = {
3210 .priority = I915_PRIORITY_MAX
3211 };
3212 struct i915_request *rq;
3213 int err = 0;
3214 u32 *cs;
3215
3216 rq = intel_engine_create_kernel_request(engine);
3217 if (IS_ERR(rq))
3218 return PTR_ERR(rq);
3219
3220 cs = intel_ring_begin(rq, 4);
3221 if (IS_ERR(cs)) {
3222 i915_request_add(rq);
3223 return PTR_ERR(cs);
3224 }
3225
3226 *cs++ = MI_STORE_DWORD_IMM_GEN4 | MI_USE_GGTT;
3227 *cs++ = i915_ggtt_offset(global);
3228 *cs++ = 0;
3229 *cs++ = id;
3230
3231 intel_ring_advance(rq, cs);
3232
3233 i915_request_get(rq);
3234 i915_request_add(rq);
3235
3236 engine->schedule(rq, &attr);
3237
3238 if (i915_request_wait(rq, 0, HZ / 2) < 0)
3239 err = -ETIME;
3240 i915_request_put(rq);
3241
3242 return err;
3243}
3244
3245static int live_preempt_user(void *arg)
3246{
3247 struct intel_gt *gt = arg;
3248 struct intel_engine_cs *engine;
3249 struct i915_vma *global;
3250 enum intel_engine_id id;
3251 u32 *result;
3252 int err = 0;
3253
3254 /*
3255 * In our other tests, we look at preemption in carefully
3256 * controlled conditions in the ringbuffer. Since most of the
3257 * time is spent in user batches, most of our preemptions naturally
3258 * occur there. We want to verify that when we preempt inside a batch
3259 * we continue on from the current instruction and do not roll back
3260 * to the start, or another earlier arbitration point.
3261 *
3262 * To verify this, we create a batch which is a mixture of
3263 * MI_MATH (gpr++) MI_SRM (gpr) and preemption points. Then with
3264 * a few preempting contexts thrown into the mix, we look for any
3265 * repeated instructions (which show up as incorrect values).
3266 */
3267
3268 global = create_global(gt, 4096);
3269 if (IS_ERR(global))
3270 return PTR_ERR(global);
3271
3272 result = i915_gem_object_pin_map(global->obj, I915_MAP_WC);
3273 if (IS_ERR(result)) {
3274 i915_vma_unpin_and_release(&global, 0);
3275 return PTR_ERR(result);
3276 }
3277
3278 for_each_engine(engine, gt, id) {
3279 struct i915_request *client[3] = {};
3280 struct igt_live_test t;
3281 int i;
3282
3283 if (!intel_engine_has_preemption(engine))
3284 continue;
3285
3286 if (IS_GEN(gt->i915, 8) && engine->class != RENDER_CLASS)
3287 continue; /* we need per-context GPR */
3288
3289 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
3290 err = -EIO;
3291 break;
3292 }
3293
3294 memset(result, 0, 4096);
3295
3296 for (i = 0; i < ARRAY_SIZE(client); i++) {
3297 struct i915_request *rq;
3298
3299 rq = create_gpr_client(engine, global,
3300 NUM_GPR * i * sizeof(u32));
3301 if (IS_ERR(rq)) {
3302 err = PTR_ERR(rq);
3303 goto end_test;
3304 }
3305
3306 client[i] = rq;
3307 }
3308
3309 /* Continuously preempt the set of 3 running contexts */
3310 for (i = 1; i <= NUM_GPR; i++) {
3311 err = preempt_user(engine, global, i);
3312 if (err)
3313 goto end_test;
3314 }
3315
3316 if (READ_ONCE(result[0]) != NUM_GPR) {
3317 pr_err("%s: Failed to release semaphore\n",
3318 engine->name);
3319 err = -EIO;
3320 goto end_test;
3321 }
3322
3323 for (i = 0; i < ARRAY_SIZE(client); i++) {
3324 int gpr;
3325
3326 if (i915_request_wait(client[i], 0, HZ / 2) < 0) {
3327 err = -ETIME;
3328 goto end_test;
3329 }
3330
3331 for (gpr = 1; gpr < NUM_GPR; gpr++) {
3332 if (result[NUM_GPR * i + gpr] != 1) {
3333 pr_err("%s: Invalid result, client %d, gpr %d, result: %d\n",
3334 engine->name,
3335 i, gpr, result[NUM_GPR * i + gpr]);
3336 err = -EINVAL;
3337 goto end_test;
3338 }
3339 }
3340 }
3341
3342end_test:
3343 for (i = 0; i < ARRAY_SIZE(client); i++) {
3344 if (!client[i])
3345 break;
3346
3347 i915_request_put(client[i]);
3348 }
3349
3350 /* Flush the semaphores on error */
3351 smp_store_mb(result[0], -1);
3352 if (igt_live_test_end(&t))
3353 err = -EIO;
3354 if (err)
3355 break;
3356 }
3357
3358 i915_vma_unpin_and_release(&global, I915_VMA_RELEASE_MAP);
3359 return err;
3360}
3361
3362static int live_preempt_timeout(void *arg)
3363{
3364 struct intel_gt *gt = arg;
3365 struct i915_gem_context *ctx_hi, *ctx_lo;
3366 struct igt_spinner spin_lo;
3367 struct intel_engine_cs *engine;
3368 enum intel_engine_id id;
3369 int err = -ENOMEM;
3370
3371 /*
3372 * Check that we force preemption to occur by cancelling the previous
3373 * context if it refuses to yield the GPU.
3374 */
3375 if (!IS_ACTIVE(CONFIG_DRM_I915_PREEMPT_TIMEOUT))
3376 return 0;
3377
3378 if (!intel_has_reset_engine(gt))
3379 return 0;
3380
3381 if (igt_spinner_init(&spin_lo, gt))
3382 return -ENOMEM;
3383
3384 ctx_hi = kernel_context(gt->i915);
3385 if (!ctx_hi)
3386 goto err_spin_lo;
3387 ctx_hi->sched.priority =
3388 I915_USER_PRIORITY(I915_CONTEXT_MAX_USER_PRIORITY);
3389
3390 ctx_lo = kernel_context(gt->i915);
3391 if (!ctx_lo)
3392 goto err_ctx_hi;
3393 ctx_lo->sched.priority =
3394 I915_USER_PRIORITY(I915_CONTEXT_MIN_USER_PRIORITY);
3395
3396 for_each_engine(engine, gt, id) {
3397 unsigned long saved_timeout;
3398 struct i915_request *rq;
3399
3400 if (!intel_engine_has_preemption(engine))
3401 continue;
3402
3403 rq = spinner_create_request(&spin_lo, ctx_lo, engine,
3404 MI_NOOP); /* preemption disabled */
3405 if (IS_ERR(rq)) {
3406 err = PTR_ERR(rq);
3407 goto err_ctx_lo;
3408 }
3409
3410 i915_request_add(rq);
3411 if (!igt_wait_for_spinner(&spin_lo, rq)) {
3412 intel_gt_set_wedged(gt);
3413 err = -EIO;
3414 goto err_ctx_lo;
3415 }
3416
3417 rq = igt_request_alloc(ctx_hi, engine);
3418 if (IS_ERR(rq)) {
3419 igt_spinner_end(&spin_lo);
3420 err = PTR_ERR(rq);
3421 goto err_ctx_lo;
3422 }
3423
3424 /* Flush the previous CS ack before changing timeouts */
3425 while (READ_ONCE(engine->execlists.pending[0]))
3426 cpu_relax();
3427
3428 saved_timeout = engine->props.preempt_timeout_ms;
3429 engine->props.preempt_timeout_ms = 1; /* in ms, -> 1 jiffie */
3430
3431 i915_request_get(rq);
3432 i915_request_add(rq);
3433
3434 intel_engine_flush_submission(engine);
3435 engine->props.preempt_timeout_ms = saved_timeout;
3436
3437 if (i915_request_wait(rq, 0, HZ / 10) < 0) {
3438 intel_gt_set_wedged(gt);
3439 i915_request_put(rq);
3440 err = -ETIME;
3441 goto err_ctx_lo;
3442 }
3443
3444 igt_spinner_end(&spin_lo);
3445 i915_request_put(rq);
3446 }
3447
3448 err = 0;
3449err_ctx_lo:
3450 kernel_context_close(ctx_lo);
3451err_ctx_hi:
3452 kernel_context_close(ctx_hi);
3453err_spin_lo:
3454 igt_spinner_fini(&spin_lo);
3455 return err;
3456}
3457
3458static int random_range(struct rnd_state *rnd, int min, int max)
3459{
3460 return i915_prandom_u32_max_state(max - min, rnd) + min;
3461}
3462
3463static int random_priority(struct rnd_state *rnd)
3464{
3465 return random_range(rnd, I915_PRIORITY_MIN, I915_PRIORITY_MAX);
3466}
3467
3468struct preempt_smoke {
3469 struct intel_gt *gt;
3470 struct i915_gem_context **contexts;
3471 struct intel_engine_cs *engine;
3472 struct drm_i915_gem_object *batch;
3473 unsigned int ncontext;
3474 struct rnd_state prng;
3475 unsigned long count;
3476};
3477
3478static struct i915_gem_context *smoke_context(struct preempt_smoke *smoke)
3479{
3480 return smoke->contexts[i915_prandom_u32_max_state(smoke->ncontext,
3481 &smoke->prng)];
3482}
3483
3484static int smoke_submit(struct preempt_smoke *smoke,
3485 struct i915_gem_context *ctx, int prio,
3486 struct drm_i915_gem_object *batch)
3487{
3488 struct i915_request *rq;
3489 struct i915_vma *vma = NULL;
3490 int err = 0;
3491
3492 if (batch) {
3493 struct i915_address_space *vm;
3494
3495 vm = i915_gem_context_get_vm_rcu(ctx);
3496 vma = i915_vma_instance(batch, vm, NULL);
3497 i915_vm_put(vm);
3498 if (IS_ERR(vma))
3499 return PTR_ERR(vma);
3500
3501 err = i915_vma_pin(vma, 0, 0, PIN_USER);
3502 if (err)
3503 return err;
3504 }
3505
3506 ctx->sched.priority = prio;
3507
3508 rq = igt_request_alloc(ctx, smoke->engine);
3509 if (IS_ERR(rq)) {
3510 err = PTR_ERR(rq);
3511 goto unpin;
3512 }
3513
3514 if (vma) {
3515 i915_vma_lock(vma);
3516 err = i915_request_await_object(rq, vma->obj, false);
3517 if (!err)
3518 err = i915_vma_move_to_active(vma, rq, 0);
3519 if (!err)
3520 err = rq->engine->emit_bb_start(rq,
3521 vma->node.start,
3522 PAGE_SIZE, 0);
3523 i915_vma_unlock(vma);
3524 }
3525
3526 i915_request_add(rq);
3527
3528unpin:
3529 if (vma)
3530 i915_vma_unpin(vma);
3531
3532 return err;
3533}
3534
3535static int smoke_crescendo_thread(void *arg)
3536{
3537 struct preempt_smoke *smoke = arg;
3538 IGT_TIMEOUT(end_time);
3539 unsigned long count;
3540
3541 count = 0;
3542 do {
3543 struct i915_gem_context *ctx = smoke_context(smoke);
3544 int err;
3545
3546 err = smoke_submit(smoke,
3547 ctx, count % I915_PRIORITY_MAX,
3548 smoke->batch);
3549 if (err)
3550 return err;
3551
3552 count++;
3553 } while (count < smoke->ncontext && !__igt_timeout(end_time, NULL));
3554
3555 smoke->count = count;
3556 return 0;
3557}
3558
3559static int smoke_crescendo(struct preempt_smoke *smoke, unsigned int flags)
3560#define BATCH BIT(0)
3561{
3562 struct task_struct *tsk[I915_NUM_ENGINES] = {};
3563 struct preempt_smoke arg[I915_NUM_ENGINES];
3564 struct intel_engine_cs *engine;
3565 enum intel_engine_id id;
3566 unsigned long count;
3567 int err = 0;
3568
3569 for_each_engine(engine, smoke->gt, id) {
3570 arg[id] = *smoke;
3571 arg[id].engine = engine;
3572 if (!(flags & BATCH))
3573 arg[id].batch = NULL;
3574 arg[id].count = 0;
3575
3576 tsk[id] = kthread_run(smoke_crescendo_thread, &arg,
3577 "igt/smoke:%d", id);
3578 if (IS_ERR(tsk[id])) {
3579 err = PTR_ERR(tsk[id]);
3580 break;
3581 }
3582 get_task_struct(tsk[id]);
3583 }
3584
3585 yield(); /* start all threads before we kthread_stop() */
3586
3587 count = 0;
3588 for_each_engine(engine, smoke->gt, id) {
3589 int status;
3590
3591 if (IS_ERR_OR_NULL(tsk[id]))
3592 continue;
3593
3594 status = kthread_stop(tsk[id]);
3595 if (status && !err)
3596 err = status;
3597
3598 count += arg[id].count;
3599
3600 put_task_struct(tsk[id]);
3601 }
3602
3603 pr_info("Submitted %lu crescendo:%x requests across %d engines and %d contexts\n",
3604 count, flags, smoke->gt->info.num_engines, smoke->ncontext);
3605 return 0;
3606}
3607
3608static int smoke_random(struct preempt_smoke *smoke, unsigned int flags)
3609{
3610 enum intel_engine_id id;
3611 IGT_TIMEOUT(end_time);
3612 unsigned long count;
3613
3614 count = 0;
3615 do {
3616 for_each_engine(smoke->engine, smoke->gt, id) {
3617 struct i915_gem_context *ctx = smoke_context(smoke);
3618 int err;
3619
3620 err = smoke_submit(smoke,
3621 ctx, random_priority(&smoke->prng),
3622 flags & BATCH ? smoke->batch : NULL);
3623 if (err)
3624 return err;
3625
3626 count++;
3627 }
3628 } while (count < smoke->ncontext && !__igt_timeout(end_time, NULL));
3629
3630 pr_info("Submitted %lu random:%x requests across %d engines and %d contexts\n",
3631 count, flags, smoke->gt->info.num_engines, smoke->ncontext);
3632 return 0;
3633}
3634
3635static int live_preempt_smoke(void *arg)
3636{
3637 struct preempt_smoke smoke = {
3638 .gt = arg,
3639 .prng = I915_RND_STATE_INITIALIZER(i915_selftest.random_seed),
3640 .ncontext = 256,
3641 };
3642 const unsigned int phase[] = { 0, BATCH };
3643 struct igt_live_test t;
3644 int err = -ENOMEM;
3645 u32 *cs;
3646 int n;
3647
3648 smoke.contexts = kmalloc_array(smoke.ncontext,
3649 sizeof(*smoke.contexts),
3650 GFP_KERNEL);
3651 if (!smoke.contexts)
3652 return -ENOMEM;
3653
3654 smoke.batch =
3655 i915_gem_object_create_internal(smoke.gt->i915, PAGE_SIZE);
3656 if (IS_ERR(smoke.batch)) {
3657 err = PTR_ERR(smoke.batch);
3658 goto err_free;
3659 }
3660
3661 cs = i915_gem_object_pin_map(smoke.batch, I915_MAP_WB);
3662 if (IS_ERR(cs)) {
3663 err = PTR_ERR(cs);
3664 goto err_batch;
3665 }
3666 for (n = 0; n < PAGE_SIZE / sizeof(*cs) - 1; n++)
3667 cs[n] = MI_ARB_CHECK;
3668 cs[n] = MI_BATCH_BUFFER_END;
3669 i915_gem_object_flush_map(smoke.batch);
3670 i915_gem_object_unpin_map(smoke.batch);
3671
3672 if (igt_live_test_begin(&t, smoke.gt->i915, __func__, "all")) {
3673 err = -EIO;
3674 goto err_batch;
3675 }
3676
3677 for (n = 0; n < smoke.ncontext; n++) {
3678 smoke.contexts[n] = kernel_context(smoke.gt->i915);
3679 if (!smoke.contexts[n])
3680 goto err_ctx;
3681 }
3682
3683 for (n = 0; n < ARRAY_SIZE(phase); n++) {
3684 err = smoke_crescendo(&smoke, phase[n]);
3685 if (err)
3686 goto err_ctx;
3687
3688 err = smoke_random(&smoke, phase[n]);
3689 if (err)
3690 goto err_ctx;
3691 }
3692
3693err_ctx:
3694 if (igt_live_test_end(&t))
3695 err = -EIO;
3696
3697 for (n = 0; n < smoke.ncontext; n++) {
3698 if (!smoke.contexts[n])
3699 break;
3700 kernel_context_close(smoke.contexts[n]);
3701 }
3702
3703err_batch:
3704 i915_gem_object_put(smoke.batch);
3705err_free:
3706 kfree(smoke.contexts);
3707
3708 return err;
3709}
3710
3711static int nop_virtual_engine(struct intel_gt *gt,
3712 struct intel_engine_cs **siblings,
3713 unsigned int nsibling,
3714 unsigned int nctx,
3715 unsigned int flags)
3716#define CHAIN BIT(0)
3717{
3718 IGT_TIMEOUT(end_time);
3719 struct i915_request *request[16] = {};
3720 struct intel_context *ve[16];
3721 unsigned long n, prime, nc;
3722 struct igt_live_test t;
3723 ktime_t times[2] = {};
3724 int err;
3725
3726 GEM_BUG_ON(!nctx || nctx > ARRAY_SIZE(ve));
3727
3728 for (n = 0; n < nctx; n++) {
3729 ve[n] = intel_execlists_create_virtual(siblings, nsibling);
3730 if (IS_ERR(ve[n])) {
3731 err = PTR_ERR(ve[n]);
3732 nctx = n;
3733 goto out;
3734 }
3735
3736 err = intel_context_pin(ve[n]);
3737 if (err) {
3738 intel_context_put(ve[n]);
3739 nctx = n;
3740 goto out;
3741 }
3742 }
3743
3744 err = igt_live_test_begin(&t, gt->i915, __func__, ve[0]->engine->name);
3745 if (err)
3746 goto out;
3747
3748 for_each_prime_number_from(prime, 1, 8192) {
3749 times[1] = ktime_get_raw();
3750
3751 if (flags & CHAIN) {
3752 for (nc = 0; nc < nctx; nc++) {
3753 for (n = 0; n < prime; n++) {
3754 struct i915_request *rq;
3755
3756 rq = i915_request_create(ve[nc]);
3757 if (IS_ERR(rq)) {
3758 err = PTR_ERR(rq);
3759 goto out;
3760 }
3761
3762 if (request[nc])
3763 i915_request_put(request[nc]);
3764 request[nc] = i915_request_get(rq);
3765 i915_request_add(rq);
3766 }
3767 }
3768 } else {
3769 for (n = 0; n < prime; n++) {
3770 for (nc = 0; nc < nctx; nc++) {
3771 struct i915_request *rq;
3772
3773 rq = i915_request_create(ve[nc]);
3774 if (IS_ERR(rq)) {
3775 err = PTR_ERR(rq);
3776 goto out;
3777 }
3778
3779 if (request[nc])
3780 i915_request_put(request[nc]);
3781 request[nc] = i915_request_get(rq);
3782 i915_request_add(rq);
3783 }
3784 }
3785 }
3786
3787 for (nc = 0; nc < nctx; nc++) {
3788 if (i915_request_wait(request[nc], 0, HZ / 10) < 0) {
3789 pr_err("%s(%s): wait for %llx:%lld timed out\n",
3790 __func__, ve[0]->engine->name,
3791 request[nc]->fence.context,
3792 request[nc]->fence.seqno);
3793
3794 GEM_TRACE("%s(%s) failed at request %llx:%lld\n",
3795 __func__, ve[0]->engine->name,
3796 request[nc]->fence.context,
3797 request[nc]->fence.seqno);
3798 GEM_TRACE_DUMP();
3799 intel_gt_set_wedged(gt);
3800 break;
3801 }
3802 }
3803
3804 times[1] = ktime_sub(ktime_get_raw(), times[1]);
3805 if (prime == 1)
3806 times[0] = times[1];
3807
3808 for (nc = 0; nc < nctx; nc++) {
3809 i915_request_put(request[nc]);
3810 request[nc] = NULL;
3811 }
3812
3813 if (__igt_timeout(end_time, NULL))
3814 break;
3815 }
3816
3817 err = igt_live_test_end(&t);
3818 if (err)
3819 goto out;
3820
3821 pr_info("Requestx%d latencies on %s: 1 = %lluns, %lu = %lluns\n",
3822 nctx, ve[0]->engine->name, ktime_to_ns(times[0]),
3823 prime, div64_u64(ktime_to_ns(times[1]), prime));
3824
3825out:
3826 if (igt_flush_test(gt->i915))
3827 err = -EIO;
3828
3829 for (nc = 0; nc < nctx; nc++) {
3830 i915_request_put(request[nc]);
3831 intel_context_unpin(ve[nc]);
3832 intel_context_put(ve[nc]);
3833 }
3834 return err;
3835}
3836
3837static unsigned int
3838__select_siblings(struct intel_gt *gt,
3839 unsigned int class,
3840 struct intel_engine_cs **siblings,
3841 bool (*filter)(const struct intel_engine_cs *))
3842{
3843 unsigned int n = 0;
3844 unsigned int inst;
3845
3846 for (inst = 0; inst <= MAX_ENGINE_INSTANCE; inst++) {
3847 if (!gt->engine_class[class][inst])
3848 continue;
3849
3850 if (filter && !filter(gt->engine_class[class][inst]))
3851 continue;
3852
3853 siblings[n++] = gt->engine_class[class][inst];
3854 }
3855
3856 return n;
3857}
3858
3859static unsigned int
3860select_siblings(struct intel_gt *gt,
3861 unsigned int class,
3862 struct intel_engine_cs **siblings)
3863{
3864 return __select_siblings(gt, class, siblings, NULL);
3865}
3866
3867static int live_virtual_engine(void *arg)
3868{
3869 struct intel_gt *gt = arg;
3870 struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
3871 struct intel_engine_cs *engine;
3872 enum intel_engine_id id;
3873 unsigned int class;
3874 int err;
3875
3876 if (intel_uc_uses_guc_submission(&gt->uc))
3877 return 0;
3878
3879 for_each_engine(engine, gt, id) {
3880 err = nop_virtual_engine(gt, &engine, 1, 1, 0);
3881 if (err) {
3882 pr_err("Failed to wrap engine %s: err=%d\n",
3883 engine->name, err);
3884 return err;
3885 }
3886 }
3887
3888 for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
3889 int nsibling, n;
3890
3891 nsibling = select_siblings(gt, class, siblings);
3892 if (nsibling < 2)
3893 continue;
3894
3895 for (n = 1; n <= nsibling + 1; n++) {
3896 err = nop_virtual_engine(gt, siblings, nsibling,
3897 n, 0);
3898 if (err)
3899 return err;
3900 }
3901
3902 err = nop_virtual_engine(gt, siblings, nsibling, n, CHAIN);
3903 if (err)
3904 return err;
3905 }
3906
3907 return 0;
3908}
3909
3910static int mask_virtual_engine(struct intel_gt *gt,
3911 struct intel_engine_cs **siblings,
3912 unsigned int nsibling)
3913{
3914 struct i915_request *request[MAX_ENGINE_INSTANCE + 1];
3915 struct intel_context *ve;
3916 struct igt_live_test t;
3917 unsigned int n;
3918 int err;
3919
3920 /*
3921 * Check that by setting the execution mask on a request, we can
3922 * restrict it to our desired engine within the virtual engine.
3923 */
3924
3925 ve = intel_execlists_create_virtual(siblings, nsibling);
3926 if (IS_ERR(ve)) {
3927 err = PTR_ERR(ve);
3928 goto out_close;
3929 }
3930
3931 err = intel_context_pin(ve);
3932 if (err)
3933 goto out_put;
3934
3935 err = igt_live_test_begin(&t, gt->i915, __func__, ve->engine->name);
3936 if (err)
3937 goto out_unpin;
3938
3939 for (n = 0; n < nsibling; n++) {
3940 request[n] = i915_request_create(ve);
3941 if (IS_ERR(request[n])) {
3942 err = PTR_ERR(request[n]);
3943 nsibling = n;
3944 goto out;
3945 }
3946
3947 /* Reverse order as it's more likely to be unnatural */
3948 request[n]->execution_mask = siblings[nsibling - n - 1]->mask;
3949
3950 i915_request_get(request[n]);
3951 i915_request_add(request[n]);
3952 }
3953
3954 for (n = 0; n < nsibling; n++) {
3955 if (i915_request_wait(request[n], 0, HZ / 10) < 0) {
3956 pr_err("%s(%s): wait for %llx:%lld timed out\n",
3957 __func__, ve->engine->name,
3958 request[n]->fence.context,
3959 request[n]->fence.seqno);
3960
3961 GEM_TRACE("%s(%s) failed at request %llx:%lld\n",
3962 __func__, ve->engine->name,
3963 request[n]->fence.context,
3964 request[n]->fence.seqno);
3965 GEM_TRACE_DUMP();
3966 intel_gt_set_wedged(gt);
3967 err = -EIO;
3968 goto out;
3969 }
3970
3971 if (request[n]->engine != siblings[nsibling - n - 1]) {
3972 pr_err("Executed on wrong sibling '%s', expected '%s'\n",
3973 request[n]->engine->name,
3974 siblings[nsibling - n - 1]->name);
3975 err = -EINVAL;
3976 goto out;
3977 }
3978 }
3979
3980 err = igt_live_test_end(&t);
3981out:
3982 if (igt_flush_test(gt->i915))
3983 err = -EIO;
3984
3985 for (n = 0; n < nsibling; n++)
3986 i915_request_put(request[n]);
3987
3988out_unpin:
3989 intel_context_unpin(ve);
3990out_put:
3991 intel_context_put(ve);
3992out_close:
3993 return err;
3994}
3995
3996static int live_virtual_mask(void *arg)
3997{
3998 struct intel_gt *gt = arg;
3999 struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4000 unsigned int class;
4001 int err;
4002
4003 if (intel_uc_uses_guc_submission(&gt->uc))
4004 return 0;
4005
4006 for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4007 unsigned int nsibling;
4008
4009 nsibling = select_siblings(gt, class, siblings);
4010 if (nsibling < 2)
4011 continue;
4012
4013 err = mask_virtual_engine(gt, siblings, nsibling);
4014 if (err)
4015 return err;
4016 }
4017
4018 return 0;
4019}
4020
4021static int slicein_virtual_engine(struct intel_gt *gt,
4022 struct intel_engine_cs **siblings,
4023 unsigned int nsibling)
4024{
4025 const long timeout = slice_timeout(siblings[0]);
4026 struct intel_context *ce;
4027 struct i915_request *rq;
4028 struct igt_spinner spin;
4029 unsigned int n;
4030 int err = 0;
4031
4032 /*
4033 * Virtual requests must take part in timeslicing on the target engines.
4034 */
4035
4036 if (igt_spinner_init(&spin, gt))
4037 return -ENOMEM;
4038
4039 for (n = 0; n < nsibling; n++) {
4040 ce = intel_context_create(siblings[n]);
4041 if (IS_ERR(ce)) {
4042 err = PTR_ERR(ce);
4043 goto out;
4044 }
4045
4046 rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
4047 intel_context_put(ce);
4048 if (IS_ERR(rq)) {
4049 err = PTR_ERR(rq);
4050 goto out;
4051 }
4052
4053 i915_request_add(rq);
4054 }
4055
4056 ce = intel_execlists_create_virtual(siblings, nsibling);
4057 if (IS_ERR(ce)) {
4058 err = PTR_ERR(ce);
4059 goto out;
4060 }
4061
4062 rq = intel_context_create_request(ce);
4063 intel_context_put(ce);
4064 if (IS_ERR(rq)) {
4065 err = PTR_ERR(rq);
4066 goto out;
4067 }
4068
4069 i915_request_get(rq);
4070 i915_request_add(rq);
4071 if (i915_request_wait(rq, 0, timeout) < 0) {
4072 GEM_TRACE_ERR("%s(%s) failed to slice in virtual request\n",
4073 __func__, rq->engine->name);
4074 GEM_TRACE_DUMP();
4075 intel_gt_set_wedged(gt);
4076 err = -EIO;
4077 }
4078 i915_request_put(rq);
4079
4080out:
4081 igt_spinner_end(&spin);
4082 if (igt_flush_test(gt->i915))
4083 err = -EIO;
4084 igt_spinner_fini(&spin);
4085 return err;
4086}
4087
4088static int sliceout_virtual_engine(struct intel_gt *gt,
4089 struct intel_engine_cs **siblings,
4090 unsigned int nsibling)
4091{
4092 const long timeout = slice_timeout(siblings[0]);
4093 struct intel_context *ce;
4094 struct i915_request *rq;
4095 struct igt_spinner spin;
4096 unsigned int n;
4097 int err = 0;
4098
4099 /*
4100 * Virtual requests must allow others a fair timeslice.
4101 */
4102
4103 if (igt_spinner_init(&spin, gt))
4104 return -ENOMEM;
4105
4106 /* XXX We do not handle oversubscription and fairness with normal rq */
4107 for (n = 0; n < nsibling; n++) {
4108 ce = intel_execlists_create_virtual(siblings, nsibling);
4109 if (IS_ERR(ce)) {
4110 err = PTR_ERR(ce);
4111 goto out;
4112 }
4113
4114 rq = igt_spinner_create_request(&spin, ce, MI_ARB_CHECK);
4115 intel_context_put(ce);
4116 if (IS_ERR(rq)) {
4117 err = PTR_ERR(rq);
4118 goto out;
4119 }
4120
4121 i915_request_add(rq);
4122 }
4123
4124 for (n = 0; !err && n < nsibling; n++) {
4125 ce = intel_context_create(siblings[n]);
4126 if (IS_ERR(ce)) {
4127 err = PTR_ERR(ce);
4128 goto out;
4129 }
4130
4131 rq = intel_context_create_request(ce);
4132 intel_context_put(ce);
4133 if (IS_ERR(rq)) {
4134 err = PTR_ERR(rq);
4135 goto out;
4136 }
4137
4138 i915_request_get(rq);
4139 i915_request_add(rq);
4140 if (i915_request_wait(rq, 0, timeout) < 0) {
4141 GEM_TRACE_ERR("%s(%s) failed to slice out virtual request\n",
4142 __func__, siblings[n]->name);
4143 GEM_TRACE_DUMP();
4144 intel_gt_set_wedged(gt);
4145 err = -EIO;
4146 }
4147 i915_request_put(rq);
4148 }
4149
4150out:
4151 igt_spinner_end(&spin);
4152 if (igt_flush_test(gt->i915))
4153 err = -EIO;
4154 igt_spinner_fini(&spin);
4155 return err;
4156}
4157
4158static int live_virtual_slice(void *arg)
4159{
4160 struct intel_gt *gt = arg;
4161 struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4162 unsigned int class;
4163 int err;
4164
4165 if (intel_uc_uses_guc_submission(&gt->uc))
4166 return 0;
4167
4168 for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4169 unsigned int nsibling;
4170
4171 nsibling = __select_siblings(gt, class, siblings,
4172 intel_engine_has_timeslices);
4173 if (nsibling < 2)
4174 continue;
4175
4176 err = slicein_virtual_engine(gt, siblings, nsibling);
4177 if (err)
4178 return err;
4179
4180 err = sliceout_virtual_engine(gt, siblings, nsibling);
4181 if (err)
4182 return err;
4183 }
4184
4185 return 0;
4186}
4187
4188static int preserved_virtual_engine(struct intel_gt *gt,
4189 struct intel_engine_cs **siblings,
4190 unsigned int nsibling)
4191{
4192 struct i915_request *last = NULL;
4193 struct intel_context *ve;
4194 struct i915_vma *scratch;
4195 struct igt_live_test t;
4196 unsigned int n;
4197 int err = 0;
4198 u32 *cs;
4199
4200 scratch = __vm_create_scratch_for_read(&siblings[0]->gt->ggtt->vm,
4201 PAGE_SIZE);
4202 if (IS_ERR(scratch))
4203 return PTR_ERR(scratch);
4204
4205 err = i915_vma_sync(scratch);
4206 if (err)
4207 goto out_scratch;
4208
4209 ve = intel_execlists_create_virtual(siblings, nsibling);
4210 if (IS_ERR(ve)) {
4211 err = PTR_ERR(ve);
4212 goto out_scratch;
4213 }
4214
4215 err = intel_context_pin(ve);
4216 if (err)
4217 goto out_put;
4218
4219 err = igt_live_test_begin(&t, gt->i915, __func__, ve->engine->name);
4220 if (err)
4221 goto out_unpin;
4222
4223 for (n = 0; n < NUM_GPR_DW; n++) {
4224 struct intel_engine_cs *engine = siblings[n % nsibling];
4225 struct i915_request *rq;
4226
4227 rq = i915_request_create(ve);
4228 if (IS_ERR(rq)) {
4229 err = PTR_ERR(rq);
4230 goto out_end;
4231 }
4232
4233 i915_request_put(last);
4234 last = i915_request_get(rq);
4235
4236 cs = intel_ring_begin(rq, 8);
4237 if (IS_ERR(cs)) {
4238 i915_request_add(rq);
4239 err = PTR_ERR(cs);
4240 goto out_end;
4241 }
4242
4243 *cs++ = MI_STORE_REGISTER_MEM_GEN8 | MI_USE_GGTT;
4244 *cs++ = CS_GPR(engine, n);
4245 *cs++ = i915_ggtt_offset(scratch) + n * sizeof(u32);
4246 *cs++ = 0;
4247
4248 *cs++ = MI_LOAD_REGISTER_IMM(1);
4249 *cs++ = CS_GPR(engine, (n + 1) % NUM_GPR_DW);
4250 *cs++ = n + 1;
4251
4252 *cs++ = MI_NOOP;
4253 intel_ring_advance(rq, cs);
4254
4255 /* Restrict this request to run on a particular engine */
4256 rq->execution_mask = engine->mask;
4257 i915_request_add(rq);
4258 }
4259
4260 if (i915_request_wait(last, 0, HZ / 5) < 0) {
4261 err = -ETIME;
4262 goto out_end;
4263 }
4264
4265 cs = i915_gem_object_pin_map(scratch->obj, I915_MAP_WB);
4266 if (IS_ERR(cs)) {
4267 err = PTR_ERR(cs);
4268 goto out_end;
4269 }
4270
4271 for (n = 0; n < NUM_GPR_DW; n++) {
4272 if (cs[n] != n) {
4273 pr_err("Incorrect value[%d] found for GPR[%d]\n",
4274 cs[n], n);
4275 err = -EINVAL;
4276 break;
4277 }
4278 }
4279
4280 i915_gem_object_unpin_map(scratch->obj);
4281
4282out_end:
4283 if (igt_live_test_end(&t))
4284 err = -EIO;
4285 i915_request_put(last);
4286out_unpin:
4287 intel_context_unpin(ve);
4288out_put:
4289 intel_context_put(ve);
4290out_scratch:
4291 i915_vma_unpin_and_release(&scratch, 0);
4292 return err;
4293}
4294
4295static int live_virtual_preserved(void *arg)
4296{
4297 struct intel_gt *gt = arg;
4298 struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4299 unsigned int class;
4300
4301 /*
4302 * Check that the context image retains non-privileged (user) registers
4303 * from one engine to the next. For this we check that the CS_GPR
4304 * are preserved.
4305 */
4306
4307 if (intel_uc_uses_guc_submission(&gt->uc))
4308 return 0;
4309
4310 /* As we use CS_GPR we cannot run before they existed on all engines. */
4311 if (INTEL_GEN(gt->i915) < 9)
4312 return 0;
4313
4314 for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4315 int nsibling, err;
4316
4317 nsibling = select_siblings(gt, class, siblings);
4318 if (nsibling < 2)
4319 continue;
4320
4321 err = preserved_virtual_engine(gt, siblings, nsibling);
4322 if (err)
4323 return err;
4324 }
4325
4326 return 0;
4327}
4328
4329static int bond_virtual_engine(struct intel_gt *gt,
4330 unsigned int class,
4331 struct intel_engine_cs **siblings,
4332 unsigned int nsibling,
4333 unsigned int flags)
4334#define BOND_SCHEDULE BIT(0)
4335{
4336 struct intel_engine_cs *master;
4337 struct i915_request *rq[16];
4338 enum intel_engine_id id;
4339 struct igt_spinner spin;
4340 unsigned long n;
4341 int err;
4342
4343 /*
4344 * A set of bonded requests is intended to be run concurrently
4345 * across a number of engines. We use one request per-engine
4346 * and a magic fence to schedule each of the bonded requests
4347 * at the same time. A consequence of our current scheduler is that
4348 * we only move requests to the HW ready queue when the request
4349 * becomes ready, that is when all of its prerequisite fences have
4350 * been signaled. As one of those fences is the master submit fence,
4351 * there is a delay on all secondary fences as the HW may be
4352 * currently busy. Equally, as all the requests are independent,
4353 * they may have other fences that delay individual request
4354 * submission to HW. Ergo, we do not guarantee that all requests are
4355 * immediately submitted to HW at the same time, just that if the
4356 * rules are abided by, they are ready at the same time as the
4357 * first is submitted. Userspace can embed semaphores in its batch
4358 * to ensure parallel execution of its phases as it requires.
4359 * Though naturally it gets requested that perhaps the scheduler should
4360 * take care of parallel execution, even across preemption events on
4361 * different HW. (The proper answer is of course "lalalala".)
4362 *
4363 * With the submit-fence, we have identified three possible phases
4364 * of synchronisation depending on the master fence: queued (not
4365 * ready), executing, and signaled. The first two are quite simple
4366 * and checked below. However, the signaled master fence handling is
4367 * contentious. Currently we do not distinguish between a signaled
4368 * fence and an expired fence, as once signaled it does not convey
4369 * any information about the previous execution. It may even be freed
4370 * and hence checking later it may not exist at all. Ergo we currently
4371 * do not apply the bonding constraint for an already signaled fence,
4372 * as our expectation is that it should not constrain the secondaries
4373 * and is outside of the scope of the bonded request API (i.e. all
4374 * userspace requests are meant to be running in parallel). As
4375 * it imposes no constraint, and is effectively a no-op, we do not
4376 * check below as normal execution flows are checked extensively above.
4377 *
4378 * XXX Is the degenerate handling of signaled submit fences the
4379 * expected behaviour for userpace?
4380 */
4381
4382 GEM_BUG_ON(nsibling >= ARRAY_SIZE(rq) - 1);
4383
4384 if (igt_spinner_init(&spin, gt))
4385 return -ENOMEM;
4386
4387 err = 0;
4388 rq[0] = ERR_PTR(-ENOMEM);
4389 for_each_engine(master, gt, id) {
4390 struct i915_sw_fence fence = {};
4391 struct intel_context *ce;
4392
4393 if (master->class == class)
4394 continue;
4395
4396 ce = intel_context_create(master);
4397 if (IS_ERR(ce)) {
4398 err = PTR_ERR(ce);
4399 goto out;
4400 }
4401
4402 memset_p((void *)rq, ERR_PTR(-EINVAL), ARRAY_SIZE(rq));
4403
4404 rq[0] = igt_spinner_create_request(&spin, ce, MI_NOOP);
4405 intel_context_put(ce);
4406 if (IS_ERR(rq[0])) {
4407 err = PTR_ERR(rq[0]);
4408 goto out;
4409 }
4410 i915_request_get(rq[0]);
4411
4412 if (flags & BOND_SCHEDULE) {
4413 onstack_fence_init(&fence);
4414 err = i915_sw_fence_await_sw_fence_gfp(&rq[0]->submit,
4415 &fence,
4416 GFP_KERNEL);
4417 }
4418
4419 i915_request_add(rq[0]);
4420 if (err < 0)
4421 goto out;
4422
4423 if (!(flags & BOND_SCHEDULE) &&
4424 !igt_wait_for_spinner(&spin, rq[0])) {
4425 err = -EIO;
4426 goto out;
4427 }
4428
4429 for (n = 0; n < nsibling; n++) {
4430 struct intel_context *ve;
4431
4432 ve = intel_execlists_create_virtual(siblings, nsibling);
4433 if (IS_ERR(ve)) {
4434 err = PTR_ERR(ve);
4435 onstack_fence_fini(&fence);
4436 goto out;
4437 }
4438
4439 err = intel_virtual_engine_attach_bond(ve->engine,
4440 master,
4441 siblings[n]);
4442 if (err) {
4443 intel_context_put(ve);
4444 onstack_fence_fini(&fence);
4445 goto out;
4446 }
4447
4448 err = intel_context_pin(ve);
4449 intel_context_put(ve);
4450 if (err) {
4451 onstack_fence_fini(&fence);
4452 goto out;
4453 }
4454
4455 rq[n + 1] = i915_request_create(ve);
4456 intel_context_unpin(ve);
4457 if (IS_ERR(rq[n + 1])) {
4458 err = PTR_ERR(rq[n + 1]);
4459 onstack_fence_fini(&fence);
4460 goto out;
4461 }
4462 i915_request_get(rq[n + 1]);
4463
4464 err = i915_request_await_execution(rq[n + 1],
4465 &rq[0]->fence,
4466 ve->engine->bond_execute);
4467 i915_request_add(rq[n + 1]);
4468 if (err < 0) {
4469 onstack_fence_fini(&fence);
4470 goto out;
4471 }
4472 }
4473 onstack_fence_fini(&fence);
4474 intel_engine_flush_submission(master);
4475 igt_spinner_end(&spin);
4476
4477 if (i915_request_wait(rq[0], 0, HZ / 10) < 0) {
4478 pr_err("Master request did not execute (on %s)!\n",
4479 rq[0]->engine->name);
4480 err = -EIO;
4481 goto out;
4482 }
4483
4484 for (n = 0; n < nsibling; n++) {
4485 if (i915_request_wait(rq[n + 1], 0,
4486 MAX_SCHEDULE_TIMEOUT) < 0) {
4487 err = -EIO;
4488 goto out;
4489 }
4490
4491 if (rq[n + 1]->engine != siblings[n]) {
4492 pr_err("Bonded request did not execute on target engine: expected %s, used %s; master was %s\n",
4493 siblings[n]->name,
4494 rq[n + 1]->engine->name,
4495 rq[0]->engine->name);
4496 err = -EINVAL;
4497 goto out;
4498 }
4499 }
4500
4501 for (n = 0; !IS_ERR(rq[n]); n++)
4502 i915_request_put(rq[n]);
4503 rq[0] = ERR_PTR(-ENOMEM);
4504 }
4505
4506out:
4507 for (n = 0; !IS_ERR(rq[n]); n++)
4508 i915_request_put(rq[n]);
4509 if (igt_flush_test(gt->i915))
4510 err = -EIO;
4511
4512 igt_spinner_fini(&spin);
4513 return err;
4514}
4515
4516static int live_virtual_bond(void *arg)
4517{
4518 static const struct phase {
4519 const char *name;
4520 unsigned int flags;
4521 } phases[] = {
4522 { "", 0 },
4523 { "schedule", BOND_SCHEDULE },
4524 { },
4525 };
4526 struct intel_gt *gt = arg;
4527 struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4528 unsigned int class;
4529 int err;
4530
4531 if (intel_uc_uses_guc_submission(&gt->uc))
4532 return 0;
4533
4534 for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4535 const struct phase *p;
4536 int nsibling;
4537
4538 nsibling = select_siblings(gt, class, siblings);
4539 if (nsibling < 2)
4540 continue;
4541
4542 for (p = phases; p->name; p++) {
4543 err = bond_virtual_engine(gt,
4544 class, siblings, nsibling,
4545 p->flags);
4546 if (err) {
4547 pr_err("%s(%s): failed class=%d, nsibling=%d, err=%d\n",
4548 __func__, p->name, class, nsibling, err);
4549 return err;
4550 }
4551 }
4552 }
4553
4554 return 0;
4555}
4556
4557static int reset_virtual_engine(struct intel_gt *gt,
4558 struct intel_engine_cs **siblings,
4559 unsigned int nsibling)
4560{
4561 struct intel_engine_cs *engine;
4562 struct intel_context *ve;
4563 struct igt_spinner spin;
4564 struct i915_request *rq;
4565 unsigned int n;
4566 int err = 0;
4567
4568 /*
4569 * In order to support offline error capture for fast preempt reset,
4570 * we need to decouple the guilty request and ensure that it and its
4571 * descendents are not executed while the capture is in progress.
4572 */
4573
4574 if (igt_spinner_init(&spin, gt))
4575 return -ENOMEM;
4576
4577 ve = intel_execlists_create_virtual(siblings, nsibling);
4578 if (IS_ERR(ve)) {
4579 err = PTR_ERR(ve);
4580 goto out_spin;
4581 }
4582
4583 for (n = 0; n < nsibling; n++)
4584 st_engine_heartbeat_disable(siblings[n]);
4585
4586 rq = igt_spinner_create_request(&spin, ve, MI_ARB_CHECK);
4587 if (IS_ERR(rq)) {
4588 err = PTR_ERR(rq);
4589 goto out_heartbeat;
4590 }
4591 i915_request_add(rq);
4592
4593 if (!igt_wait_for_spinner(&spin, rq)) {
4594 intel_gt_set_wedged(gt);
4595 err = -ETIME;
4596 goto out_heartbeat;
4597 }
4598
4599 engine = rq->engine;
4600 GEM_BUG_ON(engine == ve->engine);
4601
4602 /* Take ownership of the reset and tasklet */
4603 local_bh_disable();
4604 if (test_and_set_bit(I915_RESET_ENGINE + engine->id,
4605 &gt->reset.flags)) {
4606 local_bh_enable();
4607 intel_gt_set_wedged(gt);
4608 err = -EBUSY;
4609 goto out_heartbeat;
4610 }
4611 tasklet_disable(&engine->execlists.tasklet);
4612
4613 engine->execlists.tasklet.func(engine->execlists.tasklet.data);
4614 GEM_BUG_ON(execlists_active(&engine->execlists) != rq);
4615
4616 /* Fake a preemption event; failed of course */
4617 spin_lock_irq(&engine->active.lock);
4618 __unwind_incomplete_requests(engine);
4619 spin_unlock_irq(&engine->active.lock);
4620 GEM_BUG_ON(rq->engine != engine);
4621
4622 /* Reset the engine while keeping our active request on hold */
4623 execlists_hold(engine, rq);
4624 GEM_BUG_ON(!i915_request_on_hold(rq));
4625
4626 __intel_engine_reset_bh(engine, NULL);
4627 GEM_BUG_ON(rq->fence.error != -EIO);
4628
4629 /* Release our grasp on the engine, letting CS flow again */
4630 tasklet_enable(&engine->execlists.tasklet);
4631 clear_and_wake_up_bit(I915_RESET_ENGINE + engine->id, &gt->reset.flags);
4632 local_bh_enable();
4633
4634 /* Check that we do not resubmit the held request */
4635 i915_request_get(rq);
4636 if (!i915_request_wait(rq, 0, HZ / 5)) {
4637 pr_err("%s: on hold request completed!\n",
4638 engine->name);
4639 intel_gt_set_wedged(gt);
4640 err = -EIO;
4641 goto out_rq;
4642 }
4643 GEM_BUG_ON(!i915_request_on_hold(rq));
4644
4645 /* But is resubmitted on release */
4646 execlists_unhold(engine, rq);
4647 if (i915_request_wait(rq, 0, HZ / 5) < 0) {
4648 pr_err("%s: held request did not complete!\n",
4649 engine->name);
4650 intel_gt_set_wedged(gt);
4651 err = -ETIME;
4652 }
4653
4654out_rq:
4655 i915_request_put(rq);
4656out_heartbeat:
4657 for (n = 0; n < nsibling; n++)
4658 st_engine_heartbeat_enable(siblings[n]);
4659
4660 intel_context_put(ve);
4661out_spin:
4662 igt_spinner_fini(&spin);
4663 return err;
4664}
4665
4666static int live_virtual_reset(void *arg)
4667{
4668 struct intel_gt *gt = arg;
4669 struct intel_engine_cs *siblings[MAX_ENGINE_INSTANCE + 1];
4670 unsigned int class;
4671
4672 /*
4673 * Check that we handle a reset event within a virtual engine.
4674 * Only the physical engine is reset, but we have to check the flow
4675 * of the virtual requests around the reset, and make sure it is not
4676 * forgotten.
4677 */
4678
4679 if (intel_uc_uses_guc_submission(&gt->uc))
4680 return 0;
4681
4682 if (!intel_has_reset_engine(gt))
4683 return 0;
4684
4685 for (class = 0; class <= MAX_ENGINE_CLASS; class++) {
4686 int nsibling, err;
4687
4688 nsibling = select_siblings(gt, class, siblings);
4689 if (nsibling < 2)
4690 continue;
4691
4692 err = reset_virtual_engine(gt, siblings, nsibling);
4693 if (err)
4694 return err;
4695 }
4696
4697 return 0;
4698}
4699
4700int intel_execlists_live_selftests(struct drm_i915_private *i915)
4701{
4702 static const struct i915_subtest tests[] = {
4703 SUBTEST(live_sanitycheck),
4704 SUBTEST(live_unlite_switch),
4705 SUBTEST(live_unlite_preempt),
4706 SUBTEST(live_unlite_ring),
4707 SUBTEST(live_pin_rewind),
4708 SUBTEST(live_hold_reset),
4709 SUBTEST(live_error_interrupt),
4710 SUBTEST(live_timeslice_preempt),
4711 SUBTEST(live_timeslice_rewind),
4712 SUBTEST(live_timeslice_queue),
4713 SUBTEST(live_timeslice_nopreempt),
4714 SUBTEST(live_busywait_preempt),
4715 SUBTEST(live_preempt),
4716 SUBTEST(live_late_preempt),
4717 SUBTEST(live_nopreempt),
4718 SUBTEST(live_preempt_cancel),
4719 SUBTEST(live_suppress_self_preempt),
4720 SUBTEST(live_chain_preempt),
4721 SUBTEST(live_preempt_ring),
4722 SUBTEST(live_preempt_gang),
4723 SUBTEST(live_preempt_timeout),
4724 SUBTEST(live_preempt_user),
4725 SUBTEST(live_preempt_smoke),
4726 SUBTEST(live_virtual_engine),
4727 SUBTEST(live_virtual_mask),
4728 SUBTEST(live_virtual_preserved),
4729 SUBTEST(live_virtual_slice),
4730 SUBTEST(live_virtual_bond),
4731 SUBTEST(live_virtual_reset),
4732 };
4733
4734 if (!HAS_EXECLISTS(i915))
4735 return 0;
4736
4737 if (intel_gt_is_wedged(&i915->gt))
4738 return 0;
4739
4740 return intel_gt_live_subtests(tests, &i915->gt);
4741}
drivers/gpu/drm/i915/gt/selftest_gt_pm.c+4−4
@@ -71,7 +71,7 @@ static int live_gt_clocks(void *arg) · 70 unmodified lines
71 enum intel_engine_id id;71 enum intel_engine_id id;
72 int err = 0;72 int err = 0;
73 73
74 if (!RUNTIME_INFO(gt->i915)->cs_timestamp_frequency_hz) { /* unknown */74 if (!gt->clock_frequency) { /* unknown */
75 pr_info("CS_TIMESTAMP frequency unknown\n");75 pr_info("CS_TIMESTAMP frequency unknown\n");
76 return 0;76 return 0;
77 }77 }
@@ -112,12 +112,12 @@ static int live_gt_clocks(void *arg) · 34 unmodified lines
112 112
113 measure_clocks(engine, &cycles, &dt);113 measure_clocks(engine, &cycles, &dt);
114 114
115 time = i915_cs_timestamp_ticks_to_ns(engine->i915, cycles);115 time = intel_gt_clock_interval_to_ns(engine->gt, cycles);
116 expected = i915_cs_timestamp_ns_to_ticks(engine->i915, dt);116 expected = intel_gt_ns_to_clock_interval(engine->gt, dt);
117 117
118 pr_info("%s: TIMESTAMP %d cycles [%lldns] in %lldns [%d cycles], using CS clock frequency of %uKHz\n",118 pr_info("%s: TIMESTAMP %d cycles [%lldns] in %lldns [%d cycles], using CS clock frequency of %uKHz\n",
119 engine->name, cycles, time, dt, expected,119 engine->name, cycles, time, dt, expected,
120 RUNTIME_INFO(engine->i915)->cs_timestamp_frequency_hz / 1000);120 engine->gt->clock_frequency / 1000);
121 121
122 if (9 * time < 8 * dt || 8 * time > 9 * dt) {122 if (9 * time < 8 * dt || 8 * time > 9 * dt) {
123 pr_err("%s: CS ticks did not match walltime!\n",123 pr_err("%s: CS ticks did not match walltime!\n",
drivers/gpu/drm/i915/gt/selftest_hangcheck.c+154−3
@@ -506,7 +506,8 @@ static int igt_reset_nop_engine(void *arg) · 505 unmodified lines
506 }506 }
507 err = intel_engine_reset(engine, NULL);507 err = intel_engine_reset(engine, NULL);
508 if (err) {508 if (err) {
509 pr_err("i915_reset_engine failed\n");509 pr_err("intel_engine_reset(%s) failed, err:%d\n",
510 engine->name, err);
510 break;511 break;
511 }512 }
512 513
@@ -539,6 +540,149 @@ static int igt_reset_nop_engine(void *arg) · 26 unmodified lines
539 return 0;540 return 0;
540}541}
541 542
543static void force_reset_timeout(struct intel_engine_cs *engine)
544{
545 engine->reset_timeout.probability = 999;
546 atomic_set(&engine->reset_timeout.times, -1);
547}
548
549static void cancel_reset_timeout(struct intel_engine_cs *engine)
550{
551 memset(&engine->reset_timeout, 0, sizeof(engine->reset_timeout));
552}
553
554static int igt_reset_fail_engine(void *arg)
555{
556 struct intel_gt *gt = arg;
557 struct intel_engine_cs *engine;
558 enum intel_engine_id id;
559
560 /* Check that we can recover from engine-reset failues */
561
562 if (!intel_has_reset_engine(gt))
563 return 0;
564
565 for_each_engine(engine, gt, id) {
566 unsigned int count;
567 struct intel_context *ce;
568 IGT_TIMEOUT(end_time);
569 int err;
570
571 ce = intel_context_create(engine);
572 if (IS_ERR(ce))
573 return PTR_ERR(ce);
574
575 st_engine_heartbeat_disable(engine);
576 set_bit(I915_RESET_ENGINE + id, &gt->reset.flags);
577
578 force_reset_timeout(engine);
579 err = intel_engine_reset(engine, NULL);
580 cancel_reset_timeout(engine);
581 if (err == 0) /* timeouts only generated on gen8+ */
582 goto skip;
583
584 count = 0;
585 do {
586 struct i915_request *last = NULL;
587 int i;
588
589 if (!wait_for_idle(engine)) {
590 pr_err("%s failed to idle before reset\n",
591 engine->name);
592 err = -EIO;
593 break;
594 }
595
596 for (i = 0; i < count % 15; i++) {
597 struct i915_request *rq;
598
599 rq = intel_context_create_request(ce);
600 if (IS_ERR(rq)) {
601 struct drm_printer p =
602 drm_info_printer(gt->i915->drm.dev);
603 intel_engine_dump(engine, &p,
604 "%s(%s): failed to submit request\n",
605 __func__,
606 engine->name);
607
608 GEM_TRACE("%s(%s): failed to submit request\n",
609 __func__,
610 engine->name);
611 GEM_TRACE_DUMP();
612
613 intel_gt_set_wedged(gt);
614 if (last)
615 i915_request_put(last);
616
617 err = PTR_ERR(rq);
618 goto out;
619 }
620
621 if (last)
622 i915_request_put(last);
623 last = i915_request_get(rq);
624 i915_request_add(rq);
625 }
626
627 if (count & 1) {
628 err = intel_engine_reset(engine, NULL);
629 if (err) {
630 GEM_TRACE_ERR("intel_engine_reset(%s) failed, err:%d\n",
631 engine->name, err);
632 GEM_TRACE_DUMP();
633 i915_request_put(last);
634 break;
635 }
636 } else {
637 force_reset_timeout(engine);
638 err = intel_engine_reset(engine, NULL);
639 cancel_reset_timeout(engine);
640 if (err != -ETIMEDOUT) {
641 pr_err("intel_engine_reset(%s) did not fail, err:%d\n",
642 engine->name, err);
643 i915_request_put(last);
644 break;
645 }
646 }
647
648 err = 0;
649 if (last) {
650 if (i915_request_wait(last, 0, HZ / 2) < 0) {
651 struct drm_printer p =
652 drm_info_printer(gt->i915->drm.dev);
653
654 intel_engine_dump(engine, &p,
655 "%s(%s): failed to complete request\n",
656 __func__,
657 engine->name);
658
659 GEM_TRACE("%s(%s): failed to complete request\n",
660 __func__,
661 engine->name);
662 GEM_TRACE_DUMP();
663
664 err = -EIO;
665 }
666 i915_request_put(last);
667 }
668 count++;
669 } while (err == 0 && time_before(jiffies, end_time));
670out:
671 pr_info("%s(%s): %d resets\n", __func__, engine->name, count);
672skip:
673 clear_bit(I915_RESET_ENGINE + id, &gt->reset.flags);
674 st_engine_heartbeat_enable(engine);
675 intel_context_put(ce);
676
677 if (igt_flush_test(gt->i915))
678 err = -EIO;
679 if (err)
680 return err;
681 }
682
683 return 0;
684}
685
542static int __igt_reset_engine(struct intel_gt *gt, bool active)686static int __igt_reset_engine(struct intel_gt *gt, bool active)
543{687{
544 struct i915_gpu_error *global = &gt->i915->gpu_error;688 struct i915_gpu_error *global = &gt->i915->gpu_error;
@@ -608,7 +752,8 @@ static int __igt_reset_engine(struct intel_gt *gt, bool active) · 63 unmodified lines
608 752
609 err = intel_engine_reset(engine, NULL);753 err = intel_engine_reset(engine, NULL);
610 if (err) {754 if (err) {
611 pr_err("i915_reset_engine failed\n");755 pr_err("intel_engine_reset(%s) failed, err:%d\n",
756 engine->name, err);
612 break;757 break;
613 }758 }
614 759
@@ -1576,12 +1721,17 @@ static int __igt_atomic_reset_engine(struct intel_engine_cs *engine, · 961 unmodified lines
1576 engine->name, mode, p->name);1721 engine->name, mode, p->name);
1577 1722
1578 tasklet_disable(t);1723 tasklet_disable(t);
1724 if (strcmp(p->name, "softirq"))
1725 local_bh_disable();
1579 p->critical_section_begin();1726 p->critical_section_begin();
1580 1727
1581 err = intel_engine_reset(engine, NULL);1728 err = __intel_engine_reset_bh(engine, NULL);
1582 1729
1583 p->critical_section_end();1730 p->critical_section_end();
1731 if (strcmp(p->name, "softirq"))
1732 local_bh_enable();
1584 tasklet_enable(t);1733 tasklet_enable(t);
1734 tasklet_hi_schedule(t);
1585 1735
1586 if (err)1736 if (err)
1587 pr_err("i915_reset_engine(%s:%s) failed under %s\n",1737 pr_err("i915_reset_engine(%s:%s) failed under %s\n",
@@ -1687,6 +1837,7 @@ int intel_hangcheck_live_selftests(struct drm_i915_private *i915) · 99 unmodified lines
1687 SUBTEST(igt_reset_nop_engine),1837 SUBTEST(igt_reset_nop_engine),
1688 SUBTEST(igt_reset_idle_engine),1838 SUBTEST(igt_reset_idle_engine),
1689 SUBTEST(igt_reset_active_engine),1839 SUBTEST(igt_reset_active_engine),
1840 SUBTEST(igt_reset_fail_engine),
1690 SUBTEST(igt_reset_engines),1841 SUBTEST(igt_reset_engines),
1691 SUBTEST(igt_reset_engines_atomic),1842 SUBTEST(igt_reset_engines_atomic),
1692 SUBTEST(igt_reset_queue),1843 SUBTEST(igt_reset_queue),
drivers/gpu/drm/i915/gt/selftest_lrc.c+70−4701
@@ -1,4709 +1,73 @@
1// SPDX-License-Identifier: MIT
1/*2/*
2 * SPDX-License-Identifier: MIT
3 *
4 * Copyright © 2018 Intel Corporation3 * Copyright © 2018 Intel Corporation
5 */4 */
6 5
7#include <linux/prime_numbers.h>6#include <linux/prime_numbers.h>
8 7
9#include "gem/i915_gem_pm.h"
10#include "gt/intel_engine_heartbeat.h"
11#include "gt/intel_reset.h"
12#include "gt/selftest_engine_heartbeat.h"
13
14#include "i915_selftest.h"
15#include "selftests/i915_random.h"
16#include "selftests/igt_flush_test.h"
17#include "selftests/igt_live_test.h"
18#include "selftests/igt_spinner.h"
19#include "selftests/lib_sw_fence.h"
20
21#include "gem/selftests/igt_gem_utils.h"
22#include "gem/selftests/mock_context.h"
23
24#define CS_GPR(engine, n) ((engine)->mmio_base + 0x600 + (n) * 4)
25#define NUM_GPR 16
26#define NUM_GPR_DW (NUM_GPR * 2) /* each GPR is 2 dwords */
27
28static struct i915_vma *create_scratch(struct intel_gt *gt)
29{
30 struct drm_i915_gem_object *obj;
31 struct i915_vma *vma;
32 int err;
33
34 obj = i915_gem_object_create_internal(gt->i915, PAGE_SIZE);
35 if (IS_ERR(obj))
36 return ERR_CAST(obj);
37
38 i915_gem_object_set_cache_coherency(obj, I915_CACHING_CACHED);
39
40 vma = i915_vma_instance(obj, &gt->ggtt->vm, NULL);
41 if (IS_ERR(vma)) {
42 i915_gem_object_put(obj);
43 return vma;
44 }
45
46 err = i915_vma_pin(vma, 0, 0, PIN_GLOBAL);
47 if (err) {
48 i915_gem_object_put(obj);
49 return ERR_PTR(err);
50 }
51
52 return vma;
53}
54
55static bool is_active(struct i915_request *rq)
56{
57 if (i915_request_is_active(rq))
58 return true;
59
60 if (i915_request_on_hold(rq))
61 return true;
62
63 if (i915_request_has_initial_breadcrumb(rq) && i915_request_started(rq))
64 return true;
65
66 return false;
67}
68
69static int wait_for_submit(struct intel_engine_cs *engine,
70 struct i915_request *rq,
71 unsigned long timeout)
72{
73 timeout += jiffies;
74 do {
75 bool done = time_after(jiffies, timeout);
76
77 if (i915_request_completed(rq)) /* that was quick! */
78 return 0;
79
80 /* Wait until the HW has acknowleged the submission (or err) */
81 intel_engine_flush_submission(engine);
82 if (!READ_ONCE(engine->execlists.pending[0]) && is_active(rq))
83 return 0;
84
85 if (done)
86 return -ETIME;
87
88 cond_resched();
89 } while (1);
90}
91
92static int wait_for_reset(struct intel_engine_cs *engine,
93 struct i915_request *rq,
94 unsigned long timeout)
95{
96 timeout += jiffies;
97
98 do {
99 cond_resched();
100 intel_engine_flush_submission(engine);
101
102 if (READ_ONCE(engine->execlists.pending[0]))
103 continue;
104
105 if (i915_request_completed(rq))
106 break;
107
108 if (READ_ONCE(rq->fence.error))
109 break;
110 } while (time_before(jiffies, timeout));
111
112 flush_scheduled_work();
113
114 if (rq->fence.error != -EIO) {
115 pr_err("%s: hanging request %llx:%lld not reset\n",
116 engine->name,
117 rq->fence.context,
118 rq->fence.seqno);
119 return -EINVAL;
120 }
121
122 /* Give the request a jiffie to complete after flushing the worker */
123 if (i915_request_wait(rq, 0,
124 max(0l, (long)(timeout - jiffies)) + 1) < 0) {
125 pr_err("%s: hanging request %llx:%lld did not complete\n",
126 engine->name,
127 rq->fence.context,
128 rq->fence.seqno);
129 return -ETIME;
130 }
131
132 return 0;
133}
134
135static int live_sanitycheck(void *arg)
136{
137 struct intel_gt *gt = arg;
138 struct intel_engine_cs *engine;
139 enum intel_engine_id id;
140 struct igt_spinner spin;
141 int err = 0;
142
143 if (!HAS_LOGICAL_RING_CONTEXTS(gt->i915))
144 return 0;
145
146 if (igt_spinner_init(&spin, gt))
147 return -ENOMEM;
148
149 for_each_engine(engine, gt, id) {
150 struct intel_context *ce;
151 struct i915_request *rq;
152
153 ce = intel_context_create(engine);
154 if (IS_ERR(ce)) {
155 err = PTR_ERR(ce);
156 break;
157 }
158
159 rq = igt_spinner_create_request(&spin, ce, MI_NOOP);
160 if (IS_ERR(rq)) {
161 err = PTR_ERR(rq);
162 goto out_ctx;
163 }
164
165 i915_request_add(rq);
166 if (!igt_wait_for_spinner(&spin, rq)) {
167 GEM_TRACE("spinner failed to start\n");
168 GEM_TRACE_DUMP();
169 intel_gt_set_wedged(gt);
170 err = -EIO;
171 goto out_ctx;
172 }
173
174 igt_spinner_end(&spin);
175 if (igt_flush_test(gt->i915)) {
176 err = -EIO;
177 goto out_ctx;
178 }
179
180out_ctx:
181 intel_context_put(ce);
182 if (err)
183 break;
184 }
185
186 igt_spinner_fini(&spin);
187 return err;
188}
189
190static int live_unlite_restore(struct intel_gt *gt, int prio)
191{
192 struct intel_engine_cs *engine;
193 enum intel_engine_id id;
194 struct igt_spinner spin;
195 int err = -ENOMEM;
196
197 /*
198 * Check that we can correctly context switch between 2 instances
199 * on the same engine from the same parent context.
200 */
201
202 if (igt_spinner_init(&spin, gt))
203 return err;
204
205 err = 0;
206 for_each_engine(engine, gt, id) {
207 struct intel_context *ce[2] = {};
208 struct i915_request *rq[2];
209 struct igt_live_test t;
210 int n;
211
212 if (prio && !intel_engine_has_preemption(engine))
213 continue;
214
215 if (!intel_engine_can_store_dword(engine))
216 continue;
217
218 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
219 err = -EIO;
220 break;
221 }
222 st_engine_heartbeat_disable(engine);
223
224 for (n = 0; n < ARRAY_SIZE(ce); n++) {
225 struct intel_context *tmp;
226
227 tmp = intel_context_create(engine);
228 if (IS_ERR(tmp)) {
229 err = PTR_ERR(tmp);
230 goto err_ce;
231 }
232
233 err = intel_context_pin(tmp);
234 if (err) {
235 intel_context_put(tmp);
236 goto err_ce;
237 }
238
239 /*
240 * Setup the pair of contexts such that if we
241 * lite-restore using the RING_TAIL from ce[1] it
242 * will execute garbage from ce[0]->ring.
243 */
244 memset(tmp->ring->vaddr,
245 POISON_INUSE, /* IPEHR: 0x5a5a5a5a [hung!] */
246 tmp->ring->vma->size);
247
248 ce[n] = tmp;
249 }
250 GEM_BUG_ON(!ce[1]->ring->size);
251 intel_ring_reset(ce[1]->ring, ce[1]->ring->size / 2);
252 __execlists_update_reg_state(ce[1], engine, ce[1]->ring->head);
253
254 rq[0] = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
255 if (IS_ERR(rq[0])) {
256 err = PTR_ERR(rq[0]);
257 goto err_ce;
258 }
259
260 i915_request_get(rq[0]);
261 i915_request_add(rq[0]);
262 GEM_BUG_ON(rq[0]->postfix > ce[1]->ring->emit);
263
264 if (!igt_wait_for_spinner(&spin, rq[0])) {
265 i915_request_put(rq[0]);
266 goto err_ce;
267 }
268
269 rq[1] = i915_request_create(ce[1]);
270 if (IS_ERR(rq[1])) {
271 err = PTR_ERR(rq[1]);
272 i915_request_put(rq[0]);
273 goto err_ce;
274 }
275
276 if (!prio) {
277 /*
278 * Ensure we do the switch to ce[1] on completion.
279 *
280 * rq[0] is already submitted, so this should reduce
281 * to a no-op (a wait on a request on the same engine
282 * uses the submit fence, not the completion fence),
283 * but it will install a dependency on rq[1] for rq[0]
284 * that will prevent the pair being reordered by
285 * timeslicing.
286 */
287 i915_request_await_dma_fence(rq[1], &rq[0]->fence);
288 }
289
290 i915_request_get(rq[1]);
291 i915_request_add(rq[1]);
292 GEM_BUG_ON(rq[1]->postfix <= rq[0]->postfix);
293 i915_request_put(rq[0]);
294
295 if (prio) {
296 struct i915_sched_attr attr = {
297 .priority = prio,
298 };
299
300 /* Alternatively preempt the spinner with ce[1] */
301 engine->schedule(rq[1], &attr);
302 }
303
304 /* And switch back to ce[0] for good measure */
305 rq[0] = i915_request_create(ce[0]);
306 if (IS_ERR(rq[0])) {
307 err = PTR_ERR(rq[0]);
308 i915_request_put(rq[1]);
309 goto err_ce;
310 }
311
312 i915_request_await_dma_fence(rq[0], &rq[1]->fence);
313 i915_request_get(rq[0]);
314 i915_request_add(rq[0]);
315 GEM_BUG_ON(rq[0]->postfix > rq[1]->postfix);
316 i915_request_put(rq[1]);
317 i915_request_put(rq[0]);
318
319err_ce:
320 intel_engine_flush_submission(engine);
321 igt_spinner_end(&spin);
322 for (n = 0; n < ARRAY_SIZE(ce); n++) {
323 if (IS_ERR_OR_NULL(ce[n]))
324 break;
325
326 intel_context_unpin(ce[n]);
327 intel_context_put(ce[n]);
328 }
329
330 st_engine_heartbeat_enable(engine);
331 if (igt_live_test_end(&t))
332 err = -EIO;
333 if (err)
334 break;
335 }
336
337 igt_spinner_fini(&spin);
338 return err;
339}
340
341static int live_unlite_switch(void *arg)
342{
343 return live_unlite_restore(arg, 0);
344}
345
346static int live_unlite_preempt(void *arg)
347{
348 return live_unlite_restore(arg, I915_USER_PRIORITY(I915_PRIORITY_MAX));
349}
350
351static int live_unlite_ring(void *arg)
352{
353 struct intel_gt *gt = arg;
354 struct intel_engine_cs *engine;
355 struct igt_spinner spin;
356 enum intel_engine_id id;
357 int err = 0;
358
359 /*
360 * Setup a preemption event that will cause almost the entire ring
361 * to be unwound, potentially fooling our intel_ring_direction()
362 * into emitting a forward lite-restore instead of the rollback.
363 */
364
365 if (igt_spinner_init(&spin, gt))
366 return -ENOMEM;
367
368 for_each_engine(engine, gt, id) {
369 struct intel_context *ce[2] = {};
370 struct i915_request *rq;
371 struct igt_live_test t;
372 int n;
373
374 if (!intel_engine_has_preemption(engine))
375 continue;
376
377 if (!intel_engine_can_store_dword(engine))
378 continue;
379
380 if (igt_live_test_begin(&t, gt->i915, __func__, engine->name)) {
381 err = -EIO;
382 break;
383 }
384 st_engine_heartbeat_disable(engine);
385
386 for (n = 0; n < ARRAY_SIZE(ce); n++) {
387 struct intel_context *tmp;
388
389 tmp = intel_context_create(engine);
390 if (IS_ERR(tmp)) {
391 err = PTR_ERR(tmp);
392 goto err_ce;
393 }
394
395 err = intel_context_pin(tmp);
396 if (err) {
397 intel_context_put(tmp);
398 goto err_ce;
399 }
400
401 memset32(tmp->ring->vaddr,
402 0xdeadbeef, /* trigger a hang if executed */
403 tmp->ring->vma->size / sizeof(u32));
404
405 ce[n] = tmp;
406 }
407
408 /* Create max prio spinner, followed by N low prio nops */
409 rq = igt_spinner_create_request(&spin, ce[0], MI_ARB_CHECK);
410 if (IS_ERR(rq)) {
411 err = PTR_ERR(rq);
412 goto err_ce;
413 }
414
415 i915_request_get(rq);
416 rq->sched.attr.priority = I915_PRIORITY_BARRIER;
417 i915_request_add(rq);
418
419 if (!igt_wait_for_spinner(&spin, rq)) {
420 intel_gt_set_wedged(gt);
421 i915_request_put(rq);
422 err = -ETIME;
423 goto err_ce;
424 }
425

4429 more lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_mocs.c+28−52

This diff is large: its 157 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_rc6.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_reset.c+13−6

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_rps.c+4−1

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_timeline.c+7−7

This diff is large: its 37 lines aren’t shown here.

drivers/gpu/drm/i915/gt/selftest_workarounds.c+93−103

This diff is large: its 484 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc.c+2−14

This diff is large: its 23 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc.h+0−7

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_ads.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_fw.c+6−7

This diff is large: its 39 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c+270−187

This diff is large: its 604 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_guc_submission.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_uc.c+10−13

This diff is large: its 48 lines aren’t shown here.

drivers/gpu/drm/i915/gt/uc/intel_uc_fw.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/cmd_parser.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/handlers.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/mmio_context.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/mmio_context.h+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/gvt/scheduler.c+2−0

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_active.c+2−5

This diff is large: its 29 lines aren’t shown here.

drivers/gpu/drm/i915/i915_cmd_parser.c+10−18

This diff is large: its 83 lines aren’t shown here.

drivers/gpu/drm/i915/i915_debugfs.c+22−17

This diff is large: its 98 lines aren’t shown here.

drivers/gpu/drm/i915/i915_drv.c+1−2

This diff is large: its 17 lines aren’t shown here.

drivers/gpu/drm/i915/i915_drv.h+25−29

This diff is large: its 111 lines aren’t shown here.

drivers/gpu/drm/i915/i915_gem.c+0−2

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_gem_gtt.c+10−15

This diff is large: its 33 lines aren’t shown here.

drivers/gpu/drm/i915/i915_getparam.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_gpu_error.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_mitigations.c+146−0

This diff is large: its 147 lines aren’t shown here.

drivers/gpu/drm/i915/i915_mitigations.h+13−0

This diff is large: its 14 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pci.c+0−2

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/i915/i915_perf.c+11−6

This diff is large: its 48 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pmu.c+71−46

This diff is large: its 275 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pmu.h+24−11

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/i915/i915_request.c+128−11

This diff is large: its 210 lines aren’t shown here.

drivers/gpu/drm/i915/i915_request.h+41−6

This diff is large: its 112 lines aren’t shown here.

drivers/gpu/drm/i915/i915_scheduler.c+28−4

This diff is large: its 50 lines aren’t shown here.

drivers/gpu/drm/i915/i915_scheduler.h+7−0

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/i915/i915_scheduler_types.h+10−0

This diff is large: its 15 lines aren’t shown here.

drivers/gpu/drm/i915/i915_sw_fence.c+15−10

This diff is large: its 64 lines aren’t shown here.

drivers/gpu/drm/i915/i915_utils.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_utils.h+6−1

This diff is large: its 16 lines aren’t shown here.

drivers/gpu/drm/i915/intel_device_info.c+0−157

This diff is large: its 171 lines aren’t shown here.

drivers/gpu/drm/i915/intel_device_info.h+0−4

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/i915/intel_memory_region.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_memory_region.h+0−5

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_gem_evict.c+5−13

This diff is large: its 43 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_gem_gtt.c+1−0

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_perf.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_request.c+3−6

This diff is large: its 40 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/igt_spinner.c+4−0

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/intel_memory_region.c+98−12

This diff is large: its 161 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/mock_gem_device.c+0−2

This diff is large: its 9 lines aren’t shown here.

include/uapi/drm/i915_drm.h+2−1

This diff is large: its 11 lines aren’t shown here.