Merge tag 'drm-intel-next-2017-06-19' of git://anongit.freedesktop.org/git/drm-intel into drm-next

Final pile of features for 4.13 New uabi: - batch bo in first slot, for faster execbuf assembly in userspace (Chris Wilson) - (sub)slice getparam, needed for mesa perf support (Robert Bragg) First pile of patches for cnl/cfl support, maintained by Rodrigo but with lots of contributions from others. Still incomplete since public review still ongoing. Features/refactoring: - Make execbuf faster (Chris Wilson), a pile of series to make execbuf buffer handling have fewer passes, use less list walking, postpone more work to async workers and shuffle buffers less, all to make the common case much faster (in some cases at least). - cold boot support for glk dsi (Madhav Chauhan) - Clean up pipe A quirk and related old platform hacks (Ville) - perf sampling support for kbl/glk (Lionel) - perf cleanups (Robert Bragg) - wire atomic state to backlight code, to avoid pipe lookup hacks (Maarten) - reduce request waiting latency/overhead to remove the spinning and associated cpu cycle wasting (Chris) - fix 90/270 rotation wm computation (Ville) - new ddb allocation algo for skl (Kumar Mahesh) - fix regression due to system suspend optimiazatino (Imre) - the usual pile of small cleanups and refactors all over GVT updates contained in this tag: - optimization for per-VM mmio save/restore (Changbin) - optimization for mmio hash table (Changbin) - scheduler optimization with event (Ping) - vGPU reset refinement (Fred) - other misc refactor and cleanups, etc. * tag 'drm-intel-next-2017-06-19' of git://anongit.freedesktop.org/git/drm-intel: (170 commits) drm/i915: Update DRIVER_DATE to 20170619 drm/i915/cfl: Introduce Coffee Lake workarounds. drm/i915: Store 9 bits of PCI Device ID for platforms with a LP PCH drm/i915: Stash a pointer to the obj's resv in the vma drm/i915: Async GPU relocation processing drm/i915: Allow execbuffer to use the first object as the batch drm/i915: Wait upon userptr get-user-pages within execbuffer drm/i915: First try the previous execbuffer location drm/i915: Store a persistent reference for an object in the execbuffer cache drm/i915: Eliminate lots of iterations over the execobjects array drm/i915: Disable EXEC_OBJECT_ASYNC when doing relocations drm/i915: Pass vma to relocate entry drm/i915: Store a direct lookup from object handle to vma drm/i915: Fix retrieval of hangcheck stats drm/i915: Store i915_gem_object_is_coherent() as a bit next to cache-dirty drm/i915: Mark CPU cache as dirty on every transition for CPU writes drm/i915: Make i915_vma_destroy() static drm/i915: Actually attach the tv_format property to the SDVO connector Revert "drm/i915/skl: New ddb allocation algorithm" drm/i915/glk: Add cold boot sequence for GLK DSI ...

Dave Airlie committed 9 years ago

Commit 305b9ed2 parents eafae13 9ddb8e1108 files · +36839 −3301Browse files

Unified · Split

108 files

drivers/gpu/drm/i915/Makefile +10 −1
drivers/gpu/drm/i915/dvo_ch7xxx.c +4 −6
drivers/gpu/drm/i915/gvt/Makefile +1 −1
drivers/gpu/drm/i915/gvt/cmd_parser.c +3 −53
drivers/gpu/drm/i915/gvt/execlist.c +23 −33
drivers/gpu/drm/i915/gvt/firmware.c +2 −7
drivers/gpu/drm/i915/gvt/gtt.c +9 −6
drivers/gpu/drm/i915/gvt/gvt.c +4 −2
drivers/gpu/drm/i915/gvt/gvt.h +97 −3
drivers/gpu/drm/i915/gvt/handlers.c +177 −126
drivers/gpu/drm/i915/gvt/interrupt.c +10 −10
drivers/gpu/drm/i915/gvt/mmio.c +25 −96
drivers/gpu/drm/i915/gvt/mmio.h +16 −28
drivers/gpu/drm/i915/gvt/mpt.h +1 −2
drivers/gpu/drm/i915/gvt/render.c +40 −8
drivers/gpu/drm/i915/gvt/render.h +2 −2
drivers/gpu/drm/i915/gvt/sched_policy.c +22 −5
drivers/gpu/drm/i915/gvt/scheduler.c +32 −7
drivers/gpu/drm/i915/gvt/scheduler.h +4 −0
drivers/gpu/drm/i915/gvt/trace.h +129 −49
drivers/gpu/drm/i915/gvt/vgpu.c +7 −2
drivers/gpu/drm/i915/i915_debugfs.c +41 −27
drivers/gpu/drm/i915/i915_drv.c +41 −5
drivers/gpu/drm/i915/i915_drv.h +76 −30
drivers/gpu/drm/i915/i915_gem.c +102 −62
drivers/gpu/drm/i915/i915_gem_batch_pool.c +17 −2
drivers/gpu/drm/i915/i915_gem_clflush.c +8 −9
drivers/gpu/drm/i915/i915_gem_context.c +82 −4
drivers/gpu/drm/i915/i915_gem_context.h +26 −0
drivers/gpu/drm/i915/i915_gem_evict.c +65 −54
drivers/gpu/drm/i915/i915_gem_execbuffer.c +1631 −1123
drivers/gpu/drm/i915/i915_gem_gtt.c +13 −3
drivers/gpu/drm/i915/i915_gem_gtt.h +2 −0
drivers/gpu/drm/i915/i915_gem_internal.c +3 −1
drivers/gpu/drm/i915/i915_gem_object.h +18 −4
drivers/gpu/drm/i915/i915_gem_request.c +7 −13
drivers/gpu/drm/i915/i915_gem_shrinker.c +21 −7
drivers/gpu/drm/i915/i915_gem_stolen.c +1 −0
drivers/gpu/drm/i915/i915_gem_userptr.c +19 −5
drivers/gpu/drm/i915/i915_guc_submission.c +1 −1
drivers/gpu/drm/i915/i915_irq.c +4 −2
drivers/gpu/drm/i915/i915_oa_bdw.c +5376 −0
drivers/gpu/drm/i915/i915_oa_bdw.h +40 −0
drivers/gpu/drm/i915/i915_oa_bxt.c +2690 −0
drivers/gpu/drm/i915/i915_oa_bxt.h +40 −0
drivers/gpu/drm/i915/i915_oa_chv.c +2873 −0
drivers/gpu/drm/i915/i915_oa_chv.h +40 −0
drivers/gpu/drm/i915/i915_oa_glk.c +2602 −0
drivers/gpu/drm/i915/i915_oa_glk.h +40 −0
drivers/gpu/drm/i915/i915_oa_hsw.c +187 −76
drivers/gpu/drm/i915/i915_oa_hsw.h +3 −1
drivers/gpu/drm/i915/i915_oa_kblgt2.c +2991 −0
drivers/gpu/drm/i915/i915_oa_kblgt2.h +40 −0
drivers/gpu/drm/i915/i915_oa_kblgt3.c +3040 −0
drivers/gpu/drm/i915/i915_oa_kblgt3.h +40 −0
drivers/gpu/drm/i915/i915_oa_sklgt2.c +3479 −0
drivers/gpu/drm/i915/i915_oa_sklgt2.h +40 −0
drivers/gpu/drm/i915/i915_oa_sklgt3.c +3039 −0
drivers/gpu/drm/i915/i915_oa_sklgt3.h +40 −0
drivers/gpu/drm/i915/i915_oa_sklgt4.c +3093 −0
drivers/gpu/drm/i915/i915_oa_sklgt4.h +40 −0
drivers/gpu/drm/i915/i915_pci.c +57 −30
drivers/gpu/drm/i915/i915_perf.c +1032 −94
drivers/gpu/drm/i915/i915_pvinfo.h +2 −6
drivers/gpu/drm/i915/i915_reg.h +270 −17
drivers/gpu/drm/i915/i915_utils.h +5 −0
drivers/gpu/drm/i915/i915_vgpu.c +35 −19
drivers/gpu/drm/i915/i915_vma.c +25 −3
drivers/gpu/drm/i915/i915_vma.h +15 −5
drivers/gpu/drm/i915/intel_atomic.c +107 −30
drivers/gpu/drm/i915/intel_breadcrumbs.c +43 −30
drivers/gpu/drm/i915/intel_cdclk.c +350 −5
drivers/gpu/drm/i915/intel_csr.c +12 −5
drivers/gpu/drm/i915/intel_ddi.c +377 −15
drivers/gpu/drm/i915/intel_device_info.c +6 −5
drivers/gpu/drm/i915/intel_display.c +162 −158
drivers/gpu/drm/i915/intel_dp.c +48 −142
drivers/gpu/drm/i915/intel_dp_aux_backlight.c +9 −8
drivers/gpu/drm/i915/intel_dpll_mgr.c +435 −2
drivers/gpu/drm/i915/intel_dpll_mgr.h +4 −0
drivers/gpu/drm/i915/intel_drv.h +39 −20
drivers/gpu/drm/i915/intel_dsi.c +83 −84
drivers/gpu/drm/i915/intel_dsi_dcs_backlight.c +10 −12
drivers/gpu/drm/i915/intel_engine_cs.c +64 −22
drivers/gpu/drm/i915/intel_fifo_underrun.c +1 −1
drivers/gpu/drm/i915/intel_guc_loader.c +1 −1
drivers/gpu/drm/i915/intel_gvt.c +32 −11
drivers/gpu/drm/i915/intel_gvt.h +5 −0
drivers/gpu/drm/i915/intel_hdmi.c +38 −119
drivers/gpu/drm/i915/intel_huc.c +1 −1
drivers/gpu/drm/i915/intel_i2c.c +13 −2
drivers/gpu/drm/i915/intel_lrc.c +7 −0
drivers/gpu/drm/i915/intel_lvds.c +16 −57
drivers/gpu/drm/i915/intel_mocs.c +1 −1
drivers/gpu/drm/i915/intel_opregion.c +1 −1
drivers/gpu/drm/i915/intel_overlay.c +0 −1
drivers/gpu/drm/i915/intel_panel.c +186 −79
drivers/gpu/drm/i915/intel_pm.c +126 −22
drivers/gpu/drm/i915/intel_runtime_pm.c +308 −8
drivers/gpu/drm/i915/intel_sdvo.c +284 −256
drivers/gpu/drm/i915/intel_uc.c +5 −2
drivers/gpu/drm/i915/intel_uncore.c +0 −126
drivers/gpu/drm/i915/selftests/huge_gem_object.c +3 −1
drivers/gpu/drm/i915/selftests/i915_gem_evict.c +2 −2
drivers/gpu/drm/i915/selftests/i915_vma.c +3 −13
drivers/gpu/drm/i915/selftests/mock_context.c +11 −1
include/drm/i915_pciids.h +40 −0
include/uapi/drm/i915_drm.h +36 −8

drivers/gpu/drm/i915/Makefile+10−1

@@ -129,7 +129,16 @@ i915-y += i915_vgpu.o · 128 unmodified lines
129		129
130	# perf code	130	# perf code
131	i915-y += i915_perf.o \	131	i915-y += i915_perf.o \
132	i915_oa_hsw.o	132	i915_oa_hsw.o \
		133	i915_oa_bdw.o \
		134	i915_oa_chv.o \
		135	i915_oa_sklgt2.o \
		136	i915_oa_sklgt3.o \
		137	i915_oa_sklgt4.o \
		138	i915_oa_bxt.o \
		139	i915_oa_kblgt2.o \
		140	i915_oa_kblgt3.o \
		141	i915_oa_glk.o
133		142
134	ifeq ($(CONFIG_DRM_I915_GVT),y)	143	ifeq ($(CONFIG_DRM_I915_GVT),y)
135	i915-y += intel_gvt.o	144	i915-y += intel_gvt.o

drivers/gpu/drm/i915/dvo_ch7xxx.c+4−6

 	name = ch7xxx_get_id(vendor);
 	if (!name) {
-		DRM_DEBUG_KMS("ch7xxx not detected; got 0x%02x from %s "
+		DRM_DEBUG_KMS("ch7xxx not detected; got VID 0x%02x from %s slave %d.\n",
-				"slave %d.\n",
+			      vendor, adapter->name, dvo->slave_addr);
-			  vendor, adapter->name, dvo->slave_addr);
 		goto out;
 	}
 	devid = ch7xxx_get_did(device);
 	if (!devid) {
-		DRM_DEBUG_KMS("ch7xxx not detected; got 0x%02x from %s "
+		DRM_DEBUG_KMS("ch7xxx not detected; got DID 0x%02x from %s slave %d.\n",
-				"slave %d.\n",
+			      device, adapter->name, dvo->slave_addr);
-			  vendor, adapter->name, dvo->slave_addr);
 		goto out;
 	}

drivers/gpu/drm/i915/gvt/Makefile+1−1

@@ -3,6 +3,6 @@ GVT_SOURCE := gvt.o aperture_gm.o handlers.o vgpu.o trace_points.o firmware.o \ · 2 unmodified lines
3	interrupt.o gtt.o cfg_space.o opregion.o mmio.o display.o edid.o \	3	interrupt.o gtt.o cfg_space.o opregion.o mmio.o display.o edid.o \
4	execlist.o scheduler.o sched_policy.o render.o cmd_parser.o	4	execlist.o scheduler.o sched_policy.o render.o cmd_parser.o
5		5
6	ccflags-y += -I$(src) -I$(src)/$(GVT_DIR) -Wall	6	ccflags-y += -I$(src) -I$(src)/$(GVT_DIR)
7	i915-y += $(addprefix $(GVT_DIR)/, $(GVT_SOURCE))	7	i915-y += $(addprefix $(GVT_DIR)/, $(GVT_SOURCE))
8	obj-$(CONFIG_DRM_I915_GVT_KVMGT) += $(GVT_DIR)/kvmgt.o	8	obj-$(CONFIG_DRM_I915_GVT_KVMGT) += $(GVT_DIR)/kvmgt.o

drivers/gpu/drm/i915/gvt/cmd_parser.c+3−53

@@ -2414,53 +2414,13 @@ static void add_cmd_entry(struct intel_gvt gvt, struct cmd_entry e) · 2413 unmodified lines
2414	hash_add(gvt->cmd_table, &e->hlist, e->info->opcode);	2414	hash_add(gvt->cmd_table, &e->hlist, e->info->opcode);
2415	}	2415	}
2416		2416
2417	#define GVT_MAX_CMD_LENGTH 20 /* In Dword */
2418
2419	static void trace_cs_command(struct parser_exec_state *s,
2420	cycles_t cost_pre_cmd_handler, cycles_t cost_cmd_handler)
2421	{
2422	/* This buffer is used by ftrace to store all commands copied from
2423	* guest gma space. Sometimes commands can cross pages, this should
2424	* not be handled in ftrace logic. So this is just used as a
2425	* 'bounce buffer'
2426	*/
2427	u32 cmd_trace_buf[GVT_MAX_CMD_LENGTH];
2428	int i;
2429	u32 cmd_len = cmd_length(s);
2430	/* The chosen value of GVT_MAX_CMD_LENGTH are just based on
2431	* following two considerations:
2432	* 1) From observation, most common ring commands is not that long.
2433	* But there are execeptions. So it indeed makes sence to observe
2434	* longer commands.
2435	* 2) From the performance and debugging point of view, dumping all
2436	* contents of very commands is not necessary.
2437	* We mgith shrink GVT_MAX_CMD_LENGTH or remove this trace event in
2438	* future for performance considerations.
2439	*/
2440	if (unlikely(cmd_len > GVT_MAX_CMD_LENGTH)) {
2441	gvt_dbg_cmd("cmd length exceed tracing limitation!\n");
2442	cmd_len = GVT_MAX_CMD_LENGTH;
2443	}
2444
2445	for (i = 0; i < cmd_len; i++)
2446	cmd_trace_buf[i] = cmd_val(s, i);
2447
2448	trace_gvt_command(s->vgpu->id, s->ring_id, s->ip_gma, cmd_trace_buf,
2449	cmd_len, s->buf_type == RING_BUFFER_INSTRUCTION,
2450	cost_pre_cmd_handler, cost_cmd_handler);
2451	}
2452
2453	/* call the cmd handler, and advance ip */	2417	/* call the cmd handler, and advance ip */
2454	static int cmd_parser_exec(struct parser_exec_state *s)	2418	static int cmd_parser_exec(struct parser_exec_state *s)
2455	{	2419	{
		2420	struct intel_vgpu *vgpu = s->vgpu;
2456	struct cmd_info *info;	2421	struct cmd_info *info;
2457	u32 cmd;	2422	u32 cmd;
2458	int ret = 0;	2423	int ret = 0;
2459	cycles_t t0, t1, t2;
2460	struct parser_exec_state s_before_advance_custom;
2461	struct intel_vgpu *vgpu = s->vgpu;
2462
2463	t0 = get_cycles();
2464		2424
2465	cmd = cmd_val(s, 0);	2425	cmd = cmd_val(s, 0);
2466		2426
@@ -2471,13 +2431,10 @@ static int cmd_parser_exec(struct parser_exec_state *s) · 4 unmodified lines
2471	return -EINVAL;	2431	return -EINVAL;
2472	}	2432	}
2473		2433
2474	gvt_dbg_cmd("%s\n", info->name);
2475
2476	s->info = info;	2434	s->info = info;
2477		2435
2478	t1 = get_cycles();	2436	trace_gvt_command(vgpu->id, s->ring_id, s->ip_gma, s->ip_va,
2479		2437	cmd_length(s), s->buf_type);
2480	s_before_advance_custom = *s;
2481		2438
2482	if (info->handler) {	2439	if (info->handler) {
2483	ret = info->handler(s);	2440	ret = info->handler(s);
@@ -2486,9 +2443,6 @@ static int cmd_parser_exec(struct parser_exec_state *s) · 2 unmodified lines
2486	return ret;	2443	return ret;
2487	}	2444	}
2488	}	2445	}
2489	t2 = get_cycles();
2490
2491	trace_cs_command(&s_before_advance_custom, t1 - t0, t2 - t1);
2492		2446
2493	if (!(info->flag & F_IP_ADVANCE_CUSTOM)) {	2447	if (!(info->flag & F_IP_ADVANCE_CUSTOM)) {
2494	ret = cmd_advance_default(s);	2448	ret = cmd_advance_default(s);
@@ -2522,8 +2476,6 @@ static int command_scan(struct parser_exec_state *s, · 27 unmodified lines
2522	gma_tail = rb_start + rb_tail;	2476	gma_tail = rb_start + rb_tail;
2523	gma_bottom = rb_start + rb_len;	2477	gma_bottom = rb_start + rb_len;
2524		2478
2525	gvt_dbg_cmd("scan_start: start=%lx end=%lx\n", gma_head, gma_tail);
2526
2527	while (s->ip_gma != gma_tail) {	2479	while (s->ip_gma != gma_tail) {
2528	if (s->buf_type == RING_BUFFER_INSTRUCTION) {	2480	if (s->buf_type == RING_BUFFER_INSTRUCTION) {
2529	if (!(s->ip_gma >= rb_start) \|\|	2481	if (!(s->ip_gma >= rb_start) \|\|
@@ -2552,8 +2504,6 @@ static int command_scan(struct parser_exec_state *s, · 22 unmodified lines
2552	}	2504	}
2553	}	2505	}
2554		2506
2555	gvt_dbg_cmd("scan_end\n");
2556
2557	return ret;	2507	return ret;
2558	}	2508	}
2559		2509

drivers/gpu/drm/i915/gvt/execlist.c+23−33

@@ -708,53 +708,43 @@ static int submit_context(struct intel_vgpu *vgpu, int ring_id, · 707 unmodified lines
708	int intel_vgpu_submit_execlist(struct intel_vgpu *vgpu, int ring_id)	708	int intel_vgpu_submit_execlist(struct intel_vgpu *vgpu, int ring_id)
709	{	709	{
710	struct intel_vgpu_execlist *execlist = &vgpu->execlist[ring_id];	710	struct intel_vgpu_execlist *execlist = &vgpu->execlist[ring_id];
711	struct execlist_ctx_descriptor_format *desc[2], valid_desc[2];	711	struct execlist_ctx_descriptor_format desc[2];
712	unsigned long valid_desc_bitmap = 0;	712	int i, ret;
713	bool emulate_schedule_in = true;
714	int ret;
715	int i;
716		713
717	memset(valid_desc, 0, sizeof(valid_desc));	714	desc[0] = *get_desc_from_elsp_dwords(&execlist->elsp_dwords, 1);
		715	desc[1] = *get_desc_from_elsp_dwords(&execlist->elsp_dwords, 0);
718		716
719	desc[0] = get_desc_from_elsp_dwords(&execlist->elsp_dwords, 1);	717	if (!desc[0].valid) {
720	desc[1] = get_desc_from_elsp_dwords(&execlist->elsp_dwords, 0);	718	gvt_vgpu_err("invalid elsp submission, desc0 is invalid\n");
		719	goto inv_desc;
		720	}
721		721
722	for (i = 0; i < 2; i++) {	722	for (i = 0; i < ARRAY_SIZE(desc); i++) {
723	if (!desc[i]->valid)	723	if (!desc[i].valid)
724	continue;	724	continue;
725		725	if (!desc[i].privilege_access) {
726	if (!desc[i]->privilege_access) {
727	gvt_vgpu_err("unexpected GGTT elsp submission\n");	726	gvt_vgpu_err("unexpected GGTT elsp submission\n");
728	return -EINVAL;	727	goto inv_desc;
729	}	728	}
730
731	/* TODO: add another guest context checks here. */
732	set_bit(i, &valid_desc_bitmap);
733	valid_desc[i] = *desc[i];
734	}
735
736	if (!valid_desc_bitmap) {
737	gvt_vgpu_err("no valid desc in a elsp submission\n");
738	return -EINVAL;
739	}
740
741	if (!test_bit(0, (void *)&valid_desc_bitmap) &&
742	test_bit(1, (void *)&valid_desc_bitmap)) {
743	gvt_vgpu_err("weird elsp submission, desc 0 is not valid\n");
744	return -EINVAL;
745	}	729	}
746		730
747	/* submit workload */	731	/* submit workload */
748	for_each_set_bit(i, (void *)&valid_desc_bitmap, 2) {	732	for (i = 0; i < ARRAY_SIZE(desc); i++) {
749	ret = submit_context(vgpu, ring_id, &valid_desc[i],	733	if (!desc[i].valid)
750	emulate_schedule_in);	734	continue;
		735	ret = submit_context(vgpu, ring_id, &desc[i], i == 0);
751	if (ret) {	736	if (ret) {
752	gvt_vgpu_err("fail to schedule workload\n");	737	gvt_vgpu_err("failed to submit desc %d\n", i);
753	return ret;	738	return ret;
754	}	739	}
755	emulate_schedule_in = false;
756	}	740	}
		741
757	return 0;	742	return 0;
		743
		744	inv_desc:
		745	gvt_vgpu_err("descriptors content: desc0 %08x %08x desc1 %08x %08x\n",
		746	desc[0].udw, desc[0].ldw, desc[1].udw, desc[1].ldw);
		747	return -EINVAL;
758	}	748	}
759		749
760	static void init_vgpu_execlist(struct intel_vgpu *vgpu, int ring_id)	750	static void init_vgpu_execlist(struct intel_vgpu *vgpu, int ring_id)

drivers/gpu/drm/i915/gvt/firmware.c+2−7

 	p = firmware + h->mmio_offset;
-	hash_for_each(gvt->mmio.mmio_info_table, i, e, node) {
+	hash_for_each(gvt->mmio.mmio_info_table, i, e, node)
-		int j;
+		*(u32 *)(p + e->offset) = I915_READ_NOTRACE(_MMIO(e->offset));
-		for (j = 0; j < e->length; j += 4)
-			*(u32 *)(p + e->offset + j) =
-				I915_READ_NOTRACE(_MMIO(e->offset + j));
-	}
 	memcpy(gvt->firmware.mmio, p, info->mmio_size);

drivers/gpu/drm/i915/gvt/gtt.c+9−6

 	return readq(addr);
 }
+static void gtt_invalidate(struct drm_i915_private *dev_priv)
+{
+	mmio_hw_access_pre(dev_priv);
+	I915_WRITE(GFX_FLSH_CNTL_GEN6, GFX_FLSH_CNTL_EN);
+	mmio_hw_access_post(dev_priv);
+}
 static void write_pte64(struct drm_i915_private *dev_priv,
 		unsigned long index, u64 pte)
 {
 	void __iomem *addr = (gen8_pte_t __iomem *)dev_priv->ggtt.gsm + index;
 	writeq(pte, addr);
-	I915_WRITE(GFX_FLSH_CNTL_GEN6, GFX_FLSH_CNTL_EN);
-	POSTING_READ(GFX_FLSH_CNTL_GEN6);
 }
 static inline struct intel_gvt_gtt_entry *gtt_get_entry64(void *pt,
 	}
 	ggtt_set_shadow_entry(ggtt_mm, &m, g_gtt_index);
+	gtt_invalidate(gvt->dev_priv);
 	ggtt_set_guest_entry(ggtt_mm, &e, g_gtt_index);
 	return 0;
 }
 	u32 num_entries;
 	struct intel_gvt_gtt_entry e;
-	intel_runtime_pm_get(dev_priv);
 	memset(&e, 0, sizeof(struct intel_gvt_gtt_entry));
 	e.type = GTT_TYPE_GGTT_PTE;
 	ops->set_pfn(&e, gvt->gtt.scratch_ggtt_mfn);
 	for (offset = 0; offset < num_entries; offset++)
 		ops->set_entry(NULL, &e, index + offset, false, 0, vgpu);
-	intel_runtime_pm_put(dev_priv);
+	gtt_invalidate(dev_priv);
 }
 /**

drivers/gpu/drm/i915/gvt/gvt.c+4−2

@@ -147,7 +147,9 @@ static int gvt_service_thread(void *data) · 146 unmodified lines
147	mutex_unlock(&gvt->lock);	147	mutex_unlock(&gvt->lock);
148	}	148	}
149		149
150	if (test_and_clear_bit(INTEL_GVT_REQUEST_SCHED,	150	if (test_bit(INTEL_GVT_REQUEST_SCHED,
		151	(void *)&gvt->service_request) \|\|
		152	test_bit(INTEL_GVT_REQUEST_EVENT_SCHED,
151	(void *)&gvt->service_request)) {	153	(void *)&gvt->service_request)) {
152	intel_gvt_schedule(gvt);	154	intel_gvt_schedule(gvt);
153	}	155	}
@@ -244,7 +246,7 @@ int intel_gvt_init_device(struct drm_i915_private *dev_priv) · 90 unmodified lines
244	gvt_dbg_core("init gvt device\n");	246	gvt_dbg_core("init gvt device\n");
245		247
246	idr_init(&gvt->vgpu_idr);	248	idr_init(&gvt->vgpu_idr);
247		249	spin_lock_init(&gvt->scheduler.mmio_context_lock);
248	mutex_init(&gvt->lock);	250	mutex_init(&gvt->lock);
249	gvt->dev_priv = dev_priv;	251	gvt->dev_priv = dev_priv;
250		252

drivers/gpu/drm/i915/gvt/gvt.h+97−3

@@ -165,7 +165,6 @@ struct intel_vgpu { · 164 unmodified lines
165	struct list_head workload_q_head[I915_NUM_ENGINES];	165	struct list_head workload_q_head[I915_NUM_ENGINES];
166	struct kmem_cache *workloads;	166	struct kmem_cache *workloads;
167	atomic_t running_workload_num;	167	atomic_t running_workload_num;
168	ktime_t last_ctx_submit_time;
169	DECLARE_BITMAP(tlb_handle_pending, I915_NUM_ENGINES);	168	DECLARE_BITMAP(tlb_handle_pending, I915_NUM_ENGINES);
170	struct i915_gem_context *shadow_ctx;	169	struct i915_gem_context *shadow_ctx;
171		170
@@ -196,11 +195,27 @@ struct intel_gvt_fence { · 24 unmodified lines
196	unsigned long vgpu_allocated_fence_num;	195	unsigned long vgpu_allocated_fence_num;
197	};	196	};
198		197
199	#define INTEL_GVT_MMIO_HASH_BITS 9	198	#define INTEL_GVT_MMIO_HASH_BITS 11
200		199
201	struct intel_gvt_mmio {	200	struct intel_gvt_mmio {
202	u32 *mmio_attribute;	201	u8 *mmio_attribute;
		202	/* Register contains RO bits */
		203	#define F_RO (1 << 0)
		204	/* Register contains graphics address */
		205	#define F_GMADR (1 << 1)
		206	/* Mode mask registers with high 16 bits as the mask bits */
		207	#define F_MODE_MASK (1 << 2)
		208	/* This reg can be accessed by GPU commands */
		209	#define F_CMD_ACCESS (1 << 3)
		210	/* This reg has been accessed by a VM */
		211	#define F_ACCESSED (1 << 4)
		212	/* This reg has been accessed through GPU commands */
		213	#define F_CMD_ACCESSED (1 << 5)
		214	/* This reg could be accessed by unaligned address */
		215	#define F_UNALIGN (1 << 6)
		216
203	DECLARE_HASHTABLE(mmio_info_table, INTEL_GVT_MMIO_HASH_BITS);	217	DECLARE_HASHTABLE(mmio_info_table, INTEL_GVT_MMIO_HASH_BITS);
		218	unsigned int num_tracked_mmio;
204	};	219	};
205		220
206	struct intel_gvt_firmware {	221	struct intel_gvt_firmware {
@@ -257,7 +272,12 @@ static inline struct intel_gvt to_gvt(struct drm_i915_private i915) · 50 unmodified lines
257		272
258	enum {	273	enum {
259	INTEL_GVT_REQUEST_EMULATE_VBLANK = 0,	274	INTEL_GVT_REQUEST_EMULATE_VBLANK = 0,
		275
		276	/* Scheduling trigger by timer */
260	INTEL_GVT_REQUEST_SCHED = 1,	277	INTEL_GVT_REQUEST_SCHED = 1,
		278
		279	/* Scheduling trigger by event */
		280	INTEL_GVT_REQUEST_EVENT_SCHED = 2,
261	};	281	};
262		282
263	static inline void intel_gvt_request_service(struct intel_gvt *gvt,	283	static inline void intel_gvt_request_service(struct intel_gvt *gvt,
@@ -473,6 +493,80 @@ enum { · 209 unmodified lines
473	GVT_FAILSAFE_INSUFFICIENT_RESOURCE,	493	GVT_FAILSAFE_INSUFFICIENT_RESOURCE,
474	};	494	};
475		495
		496	static inline void mmio_hw_access_pre(struct drm_i915_private *dev_priv)
		497	{
		498	intel_runtime_pm_get(dev_priv);
		499	}
		500
		501	static inline void mmio_hw_access_post(struct drm_i915_private *dev_priv)
		502	{
		503	intel_runtime_pm_put(dev_priv);
		504	}
		505
		506	/**
		507	* intel_gvt_mmio_set_accessed - mark a MMIO has been accessed
		508	* @gvt: a GVT device
		509	* @offset: register offset
		510	*
		511	*/
		512	static inline void intel_gvt_mmio_set_accessed(
		513	struct intel_gvt *gvt, unsigned int offset)
		514	{
		515	gvt->mmio.mmio_attribute[offset >> 2] \|= F_ACCESSED;
		516	}
		517
		518	/**
		519	* intel_gvt_mmio_is_cmd_accessed - mark a MMIO could be accessed by command
		520	* @gvt: a GVT device
		521	* @offset: register offset
		522	*
		523	*/
		524	static inline bool intel_gvt_mmio_is_cmd_access(
		525	struct intel_gvt *gvt, unsigned int offset)
		526	{
		527	return gvt->mmio.mmio_attribute[offset >> 2] & F_CMD_ACCESS;
		528	}
		529
		530	/**
		531	* intel_gvt_mmio_is_unalign - mark a MMIO could be accessed unaligned
		532	* @gvt: a GVT device
		533	* @offset: register offset
		534	*
		535	*/
		536	static inline bool intel_gvt_mmio_is_unalign(
		537	struct intel_gvt *gvt, unsigned int offset)
		538	{
		539	return gvt->mmio.mmio_attribute[offset >> 2] & F_UNALIGN;
		540	}
		541
		542	/**
		543	* intel_gvt_mmio_set_cmd_accessed - mark a MMIO has been accessed by command
		544	* @gvt: a GVT device
		545	* @offset: register offset
		546	*
		547	*/
		548	static inline void intel_gvt_mmio_set_cmd_accessed(
		549	struct intel_gvt *gvt, unsigned int offset)
		550	{
		551	gvt->mmio.mmio_attribute[offset >> 2] \|= F_CMD_ACCESSED;
		552	}
		553
		554	/**
		555	* intel_gvt_mmio_has_mode_mask - if a MMIO has a mode mask
		556	* @gvt: a GVT device
		557	* @offset: register offset
		558	*
		559	* Returns:
		560	* True if a MMIO has a mode mask in its higher 16 bits, false if it isn't.
		561	*
		562	*/
		563	static inline bool intel_gvt_mmio_has_mode_mask(
		564	struct intel_gvt *gvt, unsigned int offset)
		565	{
		566	return gvt->mmio.mmio_attribute[offset >> 2] & F_MODE_MASK;
		567	}
		568
		569	#include "trace.h"
476	#include "mpt.h"	570	#include "mpt.h"
477		571
478	#endif	572	#endif

drivers/gpu/drm/i915/gvt/handlers.c+177−126

@@ -47,21 +47,6 @@ · 46 unmodified lines
47	#define PCH_PP_OFF_DELAYS _MMIO(0xc720c)	47	#define PCH_PP_OFF_DELAYS _MMIO(0xc720c)
48	#define PCH_PP_DIVISOR _MMIO(0xc7210)	48	#define PCH_PP_DIVISOR _MMIO(0xc7210)
49		49
50	/* Register contains RO bits */
51	#define F_RO (1 << 0)
52	/* Register contains graphics address */
53	#define F_GMADR (1 << 1)
54	/* Mode mask registers with high 16 bits as the mask bits */
55	#define F_MODE_MASK (1 << 2)
56	/* This reg can be accessed by GPU commands */
57	#define F_CMD_ACCESS (1 << 3)
58	/* This reg has been accessed by a VM */
59	#define F_ACCESSED (1 << 4)
60	/* This reg has been accessed through GPU commands */
61	#define F_CMD_ACCESSED (1 << 5)
62	/* This reg could be accessed by unaligned address */
63	#define F_UNALIGN (1 << 6)
64
65	unsigned long intel_gvt_get_device_type(struct intel_gvt *gvt)	50	unsigned long intel_gvt_get_device_type(struct intel_gvt *gvt)
66	{	51	{
67	if (IS_BROADWELL(gvt->dev_priv))	52	if (IS_BROADWELL(gvt->dev_priv))
@@ -92,11 +77,22 @@ static void write_vreg(struct intel_vgpu *vgpu, unsigned int offset, · 24 unmodified lines
92	memcpy(&vgpu_vreg(vgpu, offset), p_data, bytes);	77	memcpy(&vgpu_vreg(vgpu, offset), p_data, bytes);
93	}	78	}
94		79
		80	static struct intel_gvt_mmio_info find_mmio_info(struct intel_gvt gvt,
		81	unsigned int offset)
		82	{
		83	struct intel_gvt_mmio_info *e;
		84
		85	hash_for_each_possible(gvt->mmio.mmio_info_table, e, node, offset) {
		86	if (e->offset == offset)
		87	return e;
		88	}
		89	return NULL;
		90	}
		91
95	static int new_mmio_info(struct intel_gvt *gvt,	92	static int new_mmio_info(struct intel_gvt *gvt,
96	u32 offset, u32 flags, u32 size,	93	u32 offset, u8 flags, u32 size,
97	u32 addr_mask, u32 ro_mask, u32 device,	94	u32 addr_mask, u32 ro_mask, u32 device,
98	int (read)(struct intel_vgpu , unsigned int, void *, unsigned int),	95	gvt_mmio_func read, gvt_mmio_func write)
99	int (write)(struct intel_vgpu , unsigned int, void *, unsigned int))
100	{	96	{
101	struct intel_gvt_mmio_info info, p;	97	struct intel_gvt_mmio_info info, p;
102	u32 start, end, i;	98	u32 start, end, i;
@@ -116,13 +112,11 @@ static int new_mmio_info(struct intel_gvt *gvt, · 13 unmodified lines
116	return -ENOMEM;	112	return -ENOMEM;
117		113
118	info->offset = i;	114	info->offset = i;
119	p = intel_gvt_find_mmio_info(gvt, info->offset);	115	p = find_mmio_info(gvt, info->offset);
120	if (p)	116	if (p)
121	gvt_err("dup mmio definition offset %x\n",	117	gvt_err("dup mmio definition offset %x\n",
122	info->offset);	118	info->offset);
123	info->size = size;	119
124	info->length = (i + 4) < end ? 4 : (end - i);
125	info->addr_mask = addr_mask;
126	info->ro_mask = ro_mask;	120	info->ro_mask = ro_mask;
127	info->device = device;	121	info->device = device;
128	info->read = read ? read : intel_vgpu_default_mmio_read;	122	info->read = read ? read : intel_vgpu_default_mmio_read;
@@ -130,6 +124,7 @@ static int new_mmio_info(struct intel_gvt *gvt, · 1 unmodified line
130	gvt->mmio.mmio_attribute[info->offset / 4] = flags;	124	gvt->mmio.mmio_attribute[info->offset / 4] = flags;
131	INIT_HLIST_NODE(&info->node);	125	INIT_HLIST_NODE(&info->node);
132	hash_add(gvt->mmio.mmio_info_table, &info->node, info->offset);	126	hash_add(gvt->mmio.mmio_info_table, &info->node, info->offset);
		127	gvt->mmio.num_tracked_mmio++;
133	}	128	}
134	return 0;	129	return 0;
135	}	130	}
@@ -209,6 +204,7 @@ static int fence_mmio_read(struct intel_vgpu *vgpu, unsigned int off, · 73 unmodified lines
209	static int fence_mmio_write(struct intel_vgpu *vgpu, unsigned int off,	204	static int fence_mmio_write(struct intel_vgpu *vgpu, unsigned int off,
210	void *p_data, unsigned int bytes)	205	void *p_data, unsigned int bytes)
211	{	206	{
		207	struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
212	unsigned int fence_num = offset_to_fence_num(off);	208	unsigned int fence_num = offset_to_fence_num(off);
213	int ret;	209	int ret;
214		210
@@ -217,8 +213,10 @@ static int fence_mmio_write(struct intel_vgpu *vgpu, unsigned int off, · 2 unmodified lines
217	return ret;	213	return ret;
218	write_vreg(vgpu, off, p_data, bytes);	214	write_vreg(vgpu, off, p_data, bytes);
219		215
		216	mmio_hw_access_pre(dev_priv);
220	intel_vgpu_write_fence(vgpu, fence_num,	217	intel_vgpu_write_fence(vgpu, fence_num,
221	vgpu_vreg64(vgpu, fence_num_to_offset(fence_num)));	218	vgpu_vreg64(vgpu, fence_num_to_offset(fence_num)));
		219	mmio_hw_access_post(dev_priv);
222	return 0;	220	return 0;
223	}	221	}
224		222
@@ -300,6 +298,9 @@ static int gdrst_mmio_write(struct intel_vgpu *vgpu, unsigned int offset, · 75 unmodified lines
300		298
301	intel_gvt_reset_vgpu_locked(vgpu, false, engine_mask);	299	intel_gvt_reset_vgpu_locked(vgpu, false, engine_mask);
302		300
		301	/* sw will wait for the device to ack the reset request */
		302	vgpu_vreg(vgpu, offset) = 0;
		303
303	return 0;	304	return 0;
304	}	305	}
305		306
@@ -1265,7 +1266,10 @@ static int gen9_trtte_write(struct intel_vgpu *vgpu, unsigned int offset, · 959 unmodified lines
1265	}	1266	}
1266	write_vreg(vgpu, offset, p_data, bytes);	1267	write_vreg(vgpu, offset, p_data, bytes);
1267	/* TRTTE is not per-context */	1268	/* TRTTE is not per-context */
		1269
		1270	mmio_hw_access_pre(dev_priv);
1268	I915_WRITE(_MMIO(offset), vgpu_vreg(vgpu, offset));	1271	I915_WRITE(_MMIO(offset), vgpu_vreg(vgpu, offset));
		1272	mmio_hw_access_post(dev_priv);
1269		1273
1270	return 0;	1274	return 0;
1271	}	1275	}
@@ -1278,7 +1282,9 @@ static int gen9_trtt_chicken_write(struct intel_vgpu *vgpu, unsigned int offset, · 6 unmodified lines
1278		1282
1279	if (val & 1) {	1283	if (val & 1) {
1280	/* unblock hw logic */	1284	/* unblock hw logic */
		1285	mmio_hw_access_pre(dev_priv);
1281	I915_WRITE(_MMIO(offset), val);	1286	I915_WRITE(_MMIO(offset), val);
		1287	mmio_hw_access_post(dev_priv);
1282	}	1288	}
1283	write_vreg(vgpu, offset, p_data, bytes);	1289	write_vreg(vgpu, offset, p_data, bytes);
1284	return 0;	1290	return 0;
@@ -1415,7 +1421,20 @@ static int ring_timestamp_mmio_read(struct intel_vgpu *vgpu, · 130 unmodified lines
1415	{	1421	{
1416	struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;	1422	struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
1417		1423
		1424	mmio_hw_access_pre(dev_priv);
1418	vgpu_vreg(vgpu, offset) = I915_READ(_MMIO(offset));	1425	vgpu_vreg(vgpu, offset) = I915_READ(_MMIO(offset));
		1426	mmio_hw_access_post(dev_priv);
		1427	return intel_vgpu_default_mmio_read(vgpu, offset, p_data, bytes);
		1428	}
		1429
		1430	static int instdone_mmio_read(struct intel_vgpu *vgpu,
		1431	unsigned int offset, void *p_data, unsigned int bytes)
		1432	{
		1433	struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
		1434
		1435	mmio_hw_access_pre(dev_priv);
		1436	vgpu_vreg(vgpu, offset) = I915_READ(_MMIO(offset));
		1437	mmio_hw_access_post(dev_priv);
1419	return intel_vgpu_default_mmio_read(vgpu, offset, p_data, bytes);	1438	return intel_vgpu_default_mmio_read(vgpu, offset, p_data, bytes);
1420	}	1439	}
1421		1440
@@ -1434,7 +1453,6 @@ static int elsp_mmio_write(struct intel_vgpu *vgpu, unsigned int offset, · 12 unmodified lines
1434		1453
1435	execlist->elsp_dwords.data[execlist->elsp_dwords.index] = data;	1454	execlist->elsp_dwords.data[execlist->elsp_dwords.index] = data;
1436	if (execlist->elsp_dwords.index == 3) {	1455	if (execlist->elsp_dwords.index == 3) {
1437	vgpu->last_ctx_submit_time = ktime_get();
1438	ret = intel_vgpu_submit_execlist(vgpu, ring_id);	1456	ret = intel_vgpu_submit_execlist(vgpu, ring_id);
1439	if(ret)	1457	if(ret)
1440	gvt_vgpu_err("fail submit workload on ring %d\n",	1458	gvt_vgpu_err("fail submit workload on ring %d\n",
@@ -1603,6 +1621,12 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) · 162 unmodified lines
1603	MMIO_RING_DFH(RING_REG, D_ALL, F_CMD_ACCESS, NULL, NULL);	1621	MMIO_RING_DFH(RING_REG, D_ALL, F_CMD_ACCESS, NULL, NULL);
1604	#undef RING_REG	1622	#undef RING_REG
1605		1623
		1624	#define RING_REG(base) (base + 0x6c)
		1625	MMIO_RING_DFH(RING_REG, D_ALL, 0, instdone_mmio_read, NULL);
		1626	MMIO_DH(RING_REG(GEN8_BSD2_RING_BASE), D_ALL, instdone_mmio_read, NULL);
		1627	#undef RING_REG
		1628	MMIO_DH(GEN7_SC_INSTDONE, D_BDW_PLUS, instdone_mmio_read, NULL);
		1629
1606	MMIO_GM_RDR(0x2148, D_ALL, NULL, NULL);	1630	MMIO_GM_RDR(0x2148, D_ALL, NULL, NULL);
1607	MMIO_GM_RDR(CCID, D_ALL, NULL, NULL);	1631	MMIO_GM_RDR(CCID, D_ALL, NULL, NULL);
1608	MMIO_GM_RDR(0x12198, D_ALL, NULL, NULL);	1632	MMIO_GM_RDR(0x12198, D_ALL, NULL, NULL);
@@ -1779,10 +1803,6 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) · 170 unmodified lines
1779	MMIO_D(SPRSCALE(PIPE_C), D_ALL);	1803	MMIO_D(SPRSCALE(PIPE_C), D_ALL);
1780	MMIO_D(SPRSURFLIVE(PIPE_C), D_ALL);	1804	MMIO_D(SPRSURFLIVE(PIPE_C), D_ALL);
1781		1805
1782	MMIO_F(LGC_PALETTE(PIPE_A, 0), 4 * 256, 0, 0, 0, D_ALL, NULL, NULL);
1783	MMIO_F(LGC_PALETTE(PIPE_B, 0), 4 * 256, 0, 0, 0, D_ALL, NULL, NULL);
1784	MMIO_F(LGC_PALETTE(PIPE_C, 0), 4 * 256, 0, 0, 0, D_ALL, NULL, NULL);
1785
1786	MMIO_D(HTOTAL(TRANSCODER_A), D_ALL);	1806	MMIO_D(HTOTAL(TRANSCODER_A), D_ALL);
1787	MMIO_D(HBLANK(TRANSCODER_A), D_ALL);	1807	MMIO_D(HBLANK(TRANSCODER_A), D_ALL);
1788	MMIO_D(HSYNC(TRANSCODER_A), D_ALL);	1808	MMIO_D(HSYNC(TRANSCODER_A), D_ALL);
@@ -2187,7 +2207,7 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) · 398 unmodified lines
2187	MMIO_DFH(GTFIFODBG, D_ALL, F_CMD_ACCESS, NULL, NULL);	2207	MMIO_DFH(GTFIFODBG, D_ALL, F_CMD_ACCESS, NULL, NULL);
2188	MMIO_DFH(GTFIFOCTL, D_ALL, F_CMD_ACCESS, NULL, NULL);	2208	MMIO_DFH(GTFIFOCTL, D_ALL, F_CMD_ACCESS, NULL, NULL);
2189	MMIO_DH(FORCEWAKE_MT, D_PRE_SKL, NULL, mul_force_wake_write);	2209	MMIO_DH(FORCEWAKE_MT, D_PRE_SKL, NULL, mul_force_wake_write);
2190	MMIO_DH(FORCEWAKE_ACK_HSW, D_HSW \| D_BDW, NULL, NULL);	2210	MMIO_DH(FORCEWAKE_ACK_HSW, D_BDW, NULL, NULL);
2191	MMIO_D(ECOBUS, D_ALL);	2211	MMIO_D(ECOBUS, D_ALL);
2192	MMIO_DH(GEN6_RC_CONTROL, D_ALL, NULL, NULL);	2212	MMIO_DH(GEN6_RC_CONTROL, D_ALL, NULL, NULL);
2193	MMIO_DH(GEN6_RC_STATE, D_ALL, NULL, NULL);	2213	MMIO_DH(GEN6_RC_STATE, D_ALL, NULL, NULL);
@@ -2219,22 +2239,19 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) · 25 unmodified lines
2219	MMIO_D(GEN6_RC6p_THRESHOLD, D_ALL);	2239	MMIO_D(GEN6_RC6p_THRESHOLD, D_ALL);
2220	MMIO_D(GEN6_RC6pp_THRESHOLD, D_ALL);	2240	MMIO_D(GEN6_RC6pp_THRESHOLD, D_ALL);
2221	MMIO_D(GEN6_PMINTRMSK, D_ALL);	2241	MMIO_D(GEN6_PMINTRMSK, D_ALL);
2222	MMIO_DH(HSW_PWR_WELL_BIOS, D_HSW \| D_BDW, NULL, power_well_ctl_mmio_write);	2242	MMIO_DH(HSW_PWR_WELL_BIOS, D_BDW, NULL, power_well_ctl_mmio_write);
2223	MMIO_DH(HSW_PWR_WELL_DRIVER, D_HSW \| D_BDW, NULL, power_well_ctl_mmio_write);	2243	MMIO_DH(HSW_PWR_WELL_DRIVER, D_BDW, NULL, power_well_ctl_mmio_write);
2224	MMIO_DH(HSW_PWR_WELL_KVMR, D_HSW \| D_BDW, NULL, power_well_ctl_mmio_write);	2244	MMIO_DH(HSW_PWR_WELL_KVMR, D_BDW, NULL, power_well_ctl_mmio_write);
2225	MMIO_DH(HSW_PWR_WELL_DEBUG, D_HSW \| D_BDW, NULL, power_well_ctl_mmio_write);	2245	MMIO_DH(HSW_PWR_WELL_DEBUG, D_BDW, NULL, power_well_ctl_mmio_write);
2226	MMIO_DH(HSW_PWR_WELL_CTL5, D_HSW \| D_BDW, NULL, power_well_ctl_mmio_write);	2246	MMIO_DH(HSW_PWR_WELL_CTL5, D_BDW, NULL, power_well_ctl_mmio_write);
2227	MMIO_DH(HSW_PWR_WELL_CTL6, D_HSW \| D_BDW, NULL, power_well_ctl_mmio_write);	2247	MMIO_DH(HSW_PWR_WELL_CTL6, D_BDW, NULL, power_well_ctl_mmio_write);
2228		2248
2229	MMIO_D(RSTDBYCTL, D_ALL);	2249	MMIO_D(RSTDBYCTL, D_ALL);
2230		2250
2231	MMIO_DH(GEN6_GDRST, D_ALL, NULL, gdrst_mmio_write);	2251	MMIO_DH(GEN6_GDRST, D_ALL, NULL, gdrst_mmio_write);
2232	MMIO_F(FENCE_REG_GEN6_LO(0), 0x80, 0, 0, 0, D_ALL, fence_mmio_read, fence_mmio_write);	2252	MMIO_F(FENCE_REG_GEN6_LO(0), 0x80, 0, 0, 0, D_ALL, fence_mmio_read, fence_mmio_write);
2233	MMIO_F(VGT_PVINFO_PAGE, VGT_PVINFO_SIZE, F_UNALIGN, 0, 0, D_ALL, pvinfo_mmio_read, pvinfo_mmio_write);
2234	MMIO_DH(CPU_VGACNTRL, D_ALL, NULL, vga_control_mmio_write);	2253	MMIO_DH(CPU_VGACNTRL, D_ALL, NULL, vga_control_mmio_write);
2235		2254
2236	MMIO_F(MCHBAR_MIRROR_BASE_SNB, 0x40000, 0, 0, 0, D_ALL, NULL, NULL);
2237
2238	MMIO_D(TILECTL, D_ALL);	2255	MMIO_D(TILECTL, D_ALL);
2239		2256
2240	MMIO_D(GEN6_UCGCTL1, D_ALL);	2257	MMIO_D(GEN6_UCGCTL1, D_ALL);
@@ -2242,7 +2259,6 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) · 1 unmodified line
2242		2259
2243	MMIO_F(0x4f000, 0x90, 0, 0, 0, D_ALL, NULL, NULL);	2260	MMIO_F(0x4f000, 0x90, 0, 0, 0, D_ALL, NULL, NULL);
2244		2261
2245	MMIO_D(GEN6_PCODE_MAILBOX, D_PRE_BDW);
2246	MMIO_D(GEN6_PCODE_DATA, D_ALL);	2262	MMIO_D(GEN6_PCODE_DATA, D_ALL);
2247	MMIO_D(0x13812c, D_ALL);	2263	MMIO_D(0x13812c, D_ALL);
2248	MMIO_DH(GEN7_ERR_INT, D_ALL, NULL, NULL);	2264	MMIO_DH(GEN7_ERR_INT, D_ALL, NULL, NULL);
@@ -2321,14 +2337,13 @@ static int init_generic_mmio_info(struct intel_gvt *gvt) · 72 unmodified lines
2321	MMIO_D(0x1a054, D_ALL);	2337	MMIO_D(0x1a054, D_ALL);
2322		2338
2323	MMIO_D(0x44070, D_ALL);	2339	MMIO_D(0x44070, D_ALL);
2324	MMIO_DFH(0x215c, D_HSW_PLUS, F_CMD_ACCESS, NULL, NULL);	2340	MMIO_DFH(0x215c, D_BDW_PLUS, F_CMD_ACCESS, NULL, NULL);
2325	MMIO_DFH(0x2178, D_ALL, F_CMD_ACCESS, NULL, NULL);	2341	MMIO_DFH(0x2178, D_ALL, F_CMD_ACCESS, NULL, NULL);
2326	MMIO_DFH(0x217c, D_ALL, F_CMD_ACCESS, NULL, NULL);	2342	MMIO_DFH(0x217c, D_ALL, F_CMD_ACCESS, NULL, NULL);
2327	MMIO_DFH(0x12178, D_ALL, F_CMD_ACCESS, NULL, NULL);	2343	MMIO_DFH(0x12178, D_ALL, F_CMD_ACCESS, NULL, NULL);
2328	MMIO_DFH(0x1217c, D_ALL, F_CMD_ACCESS, NULL, NULL);	2344	MMIO_DFH(0x1217c, D_ALL, F_CMD_ACCESS, NULL, NULL);
2329		2345
2330	MMIO_F(0x2290, 8, F_CMD_ACCESS, 0, 0, D_HSW_PLUS, NULL, NULL);	2346	MMIO_F(0x2290, 8, F_CMD_ACCESS, 0, 0, D_BDW_PLUS, NULL, NULL);
2331	MMIO_DFH(GEN7_OACONTROL, D_HSW, F_CMD_ACCESS, NULL, NULL);
2332	MMIO_D(0x2b00, D_BDW_PLUS);	2347	MMIO_D(0x2b00, D_BDW_PLUS);
2333	MMIO_D(0x2360, D_BDW_PLUS);	2348	MMIO_D(0x2360, D_BDW_PLUS);
2334	MMIO_F(0x5200, 32, F_CMD_ACCESS, 0, 0, D_ALL, NULL, NULL);	2349	MMIO_F(0x5200, 32, F_CMD_ACCESS, 0, 0, D_ALL, NULL, NULL);
@@ -2766,7 +2781,6 @@ static int init_skl_mmio_info(struct intel_gvt *gvt) · 431 unmodified lines
2766	MMIO_D(0x72380, D_SKL_PLUS);	2781	MMIO_D(0x72380, D_SKL_PLUS);
2767	MMIO_D(0x7039c, D_SKL_PLUS);	2782	MMIO_D(0x7039c, D_SKL_PLUS);
2768		2783
2769	MMIO_F(0x80000, 0x3000, 0, 0, 0, D_SKL_PLUS, NULL, NULL);
2770	MMIO_D(0x8f074, D_SKL \| D_KBL);	2784	MMIO_D(0x8f074, D_SKL \| D_KBL);
2771	MMIO_D(0x8f004, D_SKL \| D_KBL);	2785	MMIO_D(0x8f004, D_SKL \| D_KBL);
2772	MMIO_D(0x8f034, D_SKL \| D_KBL);	2786	MMIO_D(0x8f034, D_SKL \| D_KBL);
@@ -2840,26 +2854,36 @@ static int init_skl_mmio_info(struct intel_gvt *gvt) · 67 unmodified lines
2840	return 0;	2854	return 0;
2841	}	2855	}
2842		2856
2843	/**	2857	/* Special MMIO blocks. */
2844	* intel_gvt_find_mmio_info - find MMIO information entry by aligned offset	2858	static struct gvt_mmio_block {
2845	* @gvt: GVT device	2859	unsigned int device;
2846	* @offset: register offset	2860	i915_reg_t offset;
2847	*	2861	unsigned int size;
2848	* This function is used to find the MMIO information entry from hash table	2862	gvt_mmio_func read;
2849	*	2863	gvt_mmio_func write;
2850	* Returns:	2864	} gvt_mmio_blocks[] = {
2851	* pointer to MMIO information entry, NULL if not exists	2865	{D_SKL_PLUS, _MMIO(CSR_MMIO_START_RANGE), 0x3000, NULL, NULL},
2852	*/	2866	{D_ALL, _MMIO(MCHBAR_MIRROR_BASE_SNB), 0x40000, NULL, NULL},
2853	struct intel_gvt_mmio_info intel_gvt_find_mmio_info(struct intel_gvt gvt,	2867	{D_ALL, _MMIO(VGT_PVINFO_PAGE), VGT_PVINFO_SIZE,
2854	unsigned int offset)	2868	pvinfo_mmio_read, pvinfo_mmio_write},
2855	{	2869	{D_ALL, LGC_PALETTE(PIPE_A, 0), 1024, NULL, NULL},
2856	struct intel_gvt_mmio_info *e;	2870	{D_ALL, LGC_PALETTE(PIPE_B, 0), 1024, NULL, NULL},
		2871	{D_ALL, LGC_PALETTE(PIPE_C, 0), 1024, NULL, NULL},
		2872	};
2857		2873
2858	WARN_ON(!IS_ALIGNED(offset, 4));	2874	static struct gvt_mmio_block find_mmio_block(struct intel_gvt gvt,
		2875	unsigned int offset)
		2876	{
		2877	unsigned long device = intel_gvt_get_device_type(gvt);
		2878	struct gvt_mmio_block *block = gvt_mmio_blocks;
		2879	int i;
2859		2880
2860	hash_for_each_possible(gvt->mmio.mmio_info_table, e, node, offset) {	2881	for (i = 0; i < ARRAY_SIZE(gvt_mmio_blocks); i++, block++) {
2861	if (e->offset == offset)	2882	if (!(device & block->device))
2862	return e;	2883	continue;
		2884	if (offset >= INTEL_GVT_MMIO_OFFSET(block->offset) &&
		2885	offset < INTEL_GVT_MMIO_OFFSET(block->offset) + block->size)
		2886	return block;
2863	}	2887	}
2864	return NULL;	2888	return NULL;
2865	}	2889	}
@@ -2899,9 +2923,10 @@ int intel_gvt_setup_mmio_info(struct intel_gvt *gvt) · 33 unmodified lines
2899	{	2923	{
2900	struct intel_gvt_device_info *info = &gvt->device_info;	2924	struct intel_gvt_device_info *info = &gvt->device_info;
2901	struct drm_i915_private *dev_priv = gvt->dev_priv;	2925	struct drm_i915_private *dev_priv = gvt->dev_priv;
		2926	int size = info->mmio_size / 4 * sizeof(*gvt->mmio.mmio_attribute);
2902	int ret;	2927	int ret;
2903		2928
2904	gvt->mmio.mmio_attribute = vzalloc(info->mmio_size);	2929	gvt->mmio.mmio_attribute = vzalloc(size);
2905	if (!gvt->mmio.mmio_attribute)	2930	if (!gvt->mmio.mmio_attribute)
2906	return -ENOMEM;	2931	return -ENOMEM;
2907		2932
@@ -2922,77 +2947,15 @@ int intel_gvt_setup_mmio_info(struct intel_gvt *gvt) · 14 unmodified lines
2922	if (ret)	2947	if (ret)
2923	goto err;	2948	goto err;
2924	}	2949	}
		2950
		2951	gvt_dbg_mmio("traced %u virtual mmio registers\n",
		2952	gvt->mmio.num_tracked_mmio);
2925	return 0;	2953	return 0;
2926	err:	2954	err:
2927	intel_gvt_clean_mmio_info(gvt);	2955	intel_gvt_clean_mmio_info(gvt);
2928	return ret;	2956	return ret;
2929	}	2957	}
2930		2958
2931	/**
2932	* intel_gvt_mmio_set_accessed - mark a MMIO has been accessed
2933	* @gvt: a GVT device
2934	* @offset: register offset
2935	*
2936	*/
2937	void intel_gvt_mmio_set_accessed(struct intel_gvt *gvt, unsigned int offset)
2938	{
2939	gvt->mmio.mmio_attribute[offset >> 2] \|=
2940	F_ACCESSED;
2941	}
2942
2943	/**
2944	* intel_gvt_mmio_is_cmd_accessed - mark a MMIO could be accessed by command
2945	* @gvt: a GVT device
2946	* @offset: register offset
2947	*
2948	*/
2949	bool intel_gvt_mmio_is_cmd_access(struct intel_gvt *gvt,
2950	unsigned int offset)
2951	{
2952	return gvt->mmio.mmio_attribute[offset >> 2] &
2953	F_CMD_ACCESS;
2954	}
2955
2956	/**
2957	* intel_gvt_mmio_is_unalign - mark a MMIO could be accessed unaligned
2958	* @gvt: a GVT device
2959	* @offset: register offset
2960	*
2961	*/
2962	bool intel_gvt_mmio_is_unalign(struct intel_gvt *gvt,
2963	unsigned int offset)
2964	{
2965	return gvt->mmio.mmio_attribute[offset >> 2] &
2966	F_UNALIGN;
2967	}
2968
2969	/**
2970	* intel_gvt_mmio_set_cmd_accessed - mark a MMIO has been accessed by command
2971	* @gvt: a GVT device
2972	* @offset: register offset
2973	*
2974	*/
2975	void intel_gvt_mmio_set_cmd_accessed(struct intel_gvt *gvt,
2976	unsigned int offset)
2977	{
2978	gvt->mmio.mmio_attribute[offset >> 2] \|=
2979	F_CMD_ACCESSED;
2980	}
2981
2982	/**
2983	* intel_gvt_mmio_has_mode_mask - if a MMIO has a mode mask
2984	* @gvt: a GVT device
2985	* @offset: register offset
2986	*
2987	* Returns:
2988	* True if a MMIO has a mode mask in its higher 16 bits, false if it isn't.
2989	*
2990	*/
2991	bool intel_gvt_mmio_has_mode_mask(struct intel_gvt *gvt, unsigned int offset)
2992	{
2993	return gvt->mmio.mmio_attribute[offset >> 2] &
2994	F_MODE_MASK;
2995	}
2996		2959
2997	/**	2960	/**
2998	* intel_vgpu_default_mmio_read - default MMIO read handler	2961	* intel_vgpu_default_mmio_read - default MMIO read handler
@@ -3044,3 +3007,91 @@ bool intel_gvt_in_force_nonpriv_whitelist(struct intel_gvt *gvt, · 45 unmodified lines
3044	{	3007	{
3045	return in_whitelist(offset);	3008	return in_whitelist(offset);
3046	}	3009	}
		3010
		3011	/**
		3012	* intel_vgpu_mmio_reg_rw - emulate tracked mmio registers
		3013	* @vgpu: a vGPU
		3014	* @offset: register offset
		3015	* @pdata: data buffer
		3016	* @bytes: data length
		3017	*
		3018	* Returns:
		3019	* Zero on success, negative error code if failed.
		3020	*/
		3021	int intel_vgpu_mmio_reg_rw(struct intel_vgpu *vgpu, unsigned int offset,
		3022	void *pdata, unsigned int bytes, bool is_read)
		3023	{
		3024	struct intel_gvt *gvt = vgpu->gvt;
		3025	struct intel_gvt_mmio_info *mmio_info;
		3026	struct gvt_mmio_block *mmio_block;
		3027	gvt_mmio_func func;
		3028	int ret;
		3029
		3030	if (WARN_ON(bytes > 4))
		3031	return -EINVAL;
		3032
		3033	/*
		3034	* Handle special MMIO blocks.
		3035	*/
		3036	mmio_block = find_mmio_block(gvt, offset);
		3037	if (mmio_block) {
		3038	func = is_read ? mmio_block->read : mmio_block->write;
		3039	if (func)
		3040	return func(vgpu, offset, pdata, bytes);
		3041	goto default_rw;
		3042	}
		3043
		3044	/*
		3045	* Normal tracked MMIOs.
		3046	*/
		3047	mmio_info = find_mmio_info(gvt, offset);
		3048	if (!mmio_info) {
		3049	if (!vgpu->mmio.disable_warn_untrack)
		3050	gvt_vgpu_err("untracked MMIO %08x len %d\n",
		3051	offset, bytes);
		3052	goto default_rw;
		3053	}
		3054
		3055	if (is_read)
		3056	return mmio_info->read(vgpu, offset, pdata, bytes);
		3057	else {
		3058	u64 ro_mask = mmio_info->ro_mask;
		3059	u32 old_vreg = 0, old_sreg = 0;
		3060	u64 data = 0;
		3061
		3062	if (intel_gvt_mmio_has_mode_mask(gvt, mmio_info->offset)) {
		3063	old_vreg = vgpu_vreg(vgpu, offset);
		3064	old_sreg = vgpu_sreg(vgpu, offset);
		3065	}
		3066
		3067	if (likely(!ro_mask))
		3068	ret = mmio_info->write(vgpu, offset, pdata, bytes);
		3069	else if (!~ro_mask) {
		3070	gvt_vgpu_err("try to write RO reg %x\n", offset);
		3071	return 0;
		3072	} else {
		3073	/* keep the RO bits in the virtual register */
		3074	memcpy(&data, pdata, bytes);
		3075	data &= ~ro_mask;
		3076	data \|= vgpu_vreg(vgpu, offset) & ro_mask;
		3077	ret = mmio_info->write(vgpu, offset, &data, bytes);
		3078	}
		3079
		3080	/* higher 16bits of mode ctl regs are mask bits for change */
		3081	if (intel_gvt_mmio_has_mode_mask(gvt, mmio_info->offset)) {
		3082	u32 mask = vgpu_vreg(vgpu, offset) >> 16;
		3083
		3084	vgpu_vreg(vgpu, offset) = (old_vreg & ~mask)
		3085	\| (vgpu_vreg(vgpu, offset) & mask);
		3086	vgpu_sreg(vgpu, offset) = (old_sreg & ~mask)
		3087	\| (vgpu_sreg(vgpu, offset) & mask);
		3088	}
		3089	}
		3090
		3091	return ret;
		3092
		3093	default_rw:
		3094	return is_read ?
		3095	intel_vgpu_default_mmio_read(vgpu, offset, pdata, bytes) :
		3096	intel_vgpu_default_mmio_write(vgpu, offset, pdata, bytes);
		3097	}

drivers/gpu/drm/i915/gvt/interrupt.c+10−10

 #include "i915_drv.h"
 #include "gvt.h"
+#include "trace.h"
 /* common offset among interrupt control registers */
 #define regbase_to_isr(base)	(base)
 	struct intel_gvt_irq_ops *ops = gvt->irq.ops;
 	u32 imr = *(u32 *)p_data;
-	gvt_dbg_irq("write IMR %x, new %08x, old %08x, changed %08x\n",
+	trace_write_ir(vgpu->id, "IMR", reg, imr, vgpu_vreg(vgpu, reg),
-		    reg, imr, vgpu_vreg(vgpu, reg), vgpu_vreg(vgpu, reg) ^ imr);
+		       (vgpu_vreg(vgpu, reg) ^ imr));
 	vgpu_vreg(vgpu, reg) = imr;
 	u32 ier = *(u32 *)p_data;
 	u32 virtual_ier = vgpu_vreg(vgpu, reg);
-	gvt_dbg_irq("write MASTER_IRQ %x, new %08x, old %08x, changed %08x\n",
+	trace_write_ir(vgpu->id, "MASTER_IRQ", reg, ier, virtual_ier,
-		    reg, ier, virtual_ier, virtual_ier ^ ier);
+		       (virtual_ier ^ ier));
 	/*
 	 * GEN8_MASTER_IRQ is a special irq register,
 	struct intel_gvt_irq_info *info;
 	u32 ier = *(u32 *)p_data;
-	gvt_dbg_irq("write IER %x, new %08x, old %08x, changed %08x\n",
+	trace_write_ir(vgpu->id, "IER", reg, ier, vgpu_vreg(vgpu, reg),
-		    reg, ier, vgpu_vreg(vgpu, reg), vgpu_vreg(vgpu, reg) ^ ier);
+		       (vgpu_vreg(vgpu, reg) ^ ier));
 	vgpu_vreg(vgpu, reg) = ier;
 		iir_to_regbase(reg));
 	u32 iir = *(u32 *)p_data;
-	gvt_dbg_irq("write IIR %x, new %08x, old %08x, changed %08x\n",
+	trace_write_ir(vgpu->id, "IIR", reg, iir, vgpu_vreg(vgpu, reg),
-		    reg, iir, vgpu_vreg(vgpu, reg), vgpu_vreg(vgpu, reg) ^ iir);
+		       (vgpu_vreg(vgpu, reg) ^ iir));
 	if (WARN_ON(!info))
 		return -EINVAL;
 	if (!test_bit(bit, (void *)&vgpu_vreg(vgpu,
 					regbase_to_imr(reg_base)))) {
-		gvt_dbg_irq("set bit (%d) for (%s) for vgpu (%d)\n",
+		trace_propagate_event(vgpu->id, irq_name[event], bit);
-				bit, irq_name[event], vgpu->id);
 		set_bit(bit, (void *)&vgpu_vreg(vgpu,
 					regbase_to_iir(reg_base)));
 	}

drivers/gpu/drm/i915/gvt/mmio.c+25−96

@@ -123,7 +123,6 @@ int intel_vgpu_emulate_mmio_read(struct intel_vgpu *vgpu, uint64_t pa, · 122 unmodified lines
123	void *p_data, unsigned int bytes)	123	void *p_data, unsigned int bytes)
124	{	124	{
125	struct intel_gvt *gvt = vgpu->gvt;	125	struct intel_gvt *gvt = vgpu->gvt;
126	struct intel_gvt_mmio_info *mmio;
127	unsigned int offset = 0;	126	unsigned int offset = 0;
128	int ret = -EINVAL;	127	int ret = -EINVAL;
129		128
@@ -187,32 +186,8 @@ int intel_vgpu_emulate_mmio_read(struct intel_vgpu *vgpu, uint64_t pa, · 57 unmodified lines
187	goto err;	186	goto err;
188	}	187	}
189		188
190	mmio = intel_gvt_find_mmio_info(gvt, rounddown(offset, 4));	189	ret = intel_vgpu_mmio_reg_rw(vgpu, offset, p_data, bytes, true);
191	if (mmio) {	190	if (ret < 0)
192	if (!intel_gvt_mmio_is_unalign(gvt, mmio->offset)) {
193	if (WARN_ON(offset + bytes > mmio->offset + mmio->size))
194	goto err;
195	if (WARN_ON(mmio->offset != offset))
196	goto err;
197	}
198	ret = mmio->read(vgpu, offset, p_data, bytes);
199	} else {
200	ret = intel_vgpu_default_mmio_read(vgpu, offset, p_data, bytes);
201
202	if (!vgpu->mmio.disable_warn_untrack) {
203	gvt_vgpu_err("read untracked MMIO %x(%dB) val %x\n",
204	offset, bytes, (u32 )p_data);
205
206	if (offset == 0x206c) {
207	gvt_vgpu_err("------------------------------------------\n");
208	gvt_vgpu_err("likely triggers a gfx reset\n");
209	gvt_vgpu_err("------------------------------------------\n");
210	vgpu->mmio.disable_warn_untrack = true;
211	}
212	}
213	}
214
215	if (ret)
216	goto err;	191	goto err;
217		192
218	intel_gvt_mmio_set_accessed(gvt, offset);	193	intel_gvt_mmio_set_accessed(gvt, offset);
@@ -239,9 +214,7 @@ int intel_vgpu_emulate_mmio_write(struct intel_vgpu *vgpu, uint64_t pa, · 20 unmodified lines
239	void *p_data, unsigned int bytes)	214	void *p_data, unsigned int bytes)
240	{	215	{
241	struct intel_gvt *gvt = vgpu->gvt;	216	struct intel_gvt *gvt = vgpu->gvt;
242	struct intel_gvt_mmio_info *mmio;
243	unsigned int offset = 0;	217	unsigned int offset = 0;
244	u32 old_vreg = 0, old_sreg = 0;
245	int ret = -EINVAL;	218	int ret = -EINVAL;
246		219
247	if (vgpu->failsafe) {	220	if (vgpu->failsafe) {
@@ -296,66 +269,10 @@ int intel_vgpu_emulate_mmio_write(struct intel_vgpu *vgpu, uint64_t pa, · 48 unmodified lines
296	return ret;	269	return ret;
297	}	270	}
298		271
299	mmio = intel_gvt_find_mmio_info(gvt, rounddown(offset, 4));	272	ret = intel_vgpu_mmio_reg_rw(vgpu, offset, p_data, bytes, false);
300	if (!mmio && !vgpu->mmio.disable_warn_untrack)	273	if (ret < 0)
301	gvt_dbg_mmio("vgpu%d: write untracked MMIO %x len %d val %x\n",
302	vgpu->id, offset, bytes, (u32 )p_data);
303
304	if (!intel_gvt_mmio_is_unalign(gvt, offset)) {
305	if (WARN_ON(!IS_ALIGNED(offset, bytes)))
306	goto err;
307	}
308
309	if (mmio) {
310	u64 ro_mask = mmio->ro_mask;
311
312	if (!intel_gvt_mmio_is_unalign(gvt, mmio->offset)) {
313	if (WARN_ON(offset + bytes > mmio->offset + mmio->size))
314	goto err;
315	if (WARN_ON(mmio->offset != offset))
316	goto err;
317	}
318
319	if (intel_gvt_mmio_has_mode_mask(gvt, mmio->offset)) {
320	old_vreg = vgpu_vreg(vgpu, offset);
321	old_sreg = vgpu_sreg(vgpu, offset);
322	}
323
324	if (!ro_mask) {
325	ret = mmio->write(vgpu, offset, p_data, bytes);
326	} else {
327	/* Protect RO bits like HW */
328	u64 data = 0;
329
330	/* all register bits are RO. */
331	if (ro_mask == ~(u64)0) {
332	gvt_vgpu_err("try to write RO reg %x\n",
333	offset);
334	ret = 0;
335	goto out;
336	}
337	/* keep the RO bits in the virtual register */
338	memcpy(&data, p_data, bytes);
339	data &= ~mmio->ro_mask;
340	data \|= vgpu_vreg(vgpu, offset) & mmio->ro_mask;
341	ret = mmio->write(vgpu, offset, &data, bytes);
342	}
343
344	/* higher 16bits of mode ctl regs are mask bits for change */
345	if (intel_gvt_mmio_has_mode_mask(gvt, mmio->offset)) {
346	u32 mask = vgpu_vreg(vgpu, offset) >> 16;
347
348	vgpu_vreg(vgpu, offset) = (old_vreg & ~mask)
349	\| (vgpu_vreg(vgpu, offset) & mask);
350	vgpu_sreg(vgpu, offset) = (old_sreg & ~mask)
351	\| (vgpu_sreg(vgpu, offset) & mask);
352	}
353	} else
354	ret = intel_vgpu_default_mmio_write(vgpu, offset, p_data,
355	bytes);
356	if (ret)
357	goto err;	274	goto err;
358	out:	275
359	intel_gvt_mmio_set_accessed(gvt, offset);	276	intel_gvt_mmio_set_accessed(gvt, offset);
360	mutex_unlock(&gvt->lock);	277	mutex_unlock(&gvt->lock);
361	return 0;	278	return 0;
@@ -372,20 +289,32 @@ err: · 10 unmodified lines
372	* @vgpu: a vGPU	289	* @vgpu: a vGPU
373	*	290	*
374	*/	291	*/
375	void intel_vgpu_reset_mmio(struct intel_vgpu *vgpu)	292	void intel_vgpu_reset_mmio(struct intel_vgpu *vgpu, bool dmlr)
376	{	293	{
377	struct intel_gvt *gvt = vgpu->gvt;	294	struct intel_gvt *gvt = vgpu->gvt;
378	const struct intel_gvt_device_info *info = &gvt->device_info;	295	const struct intel_gvt_device_info *info = &gvt->device_info;
		296	void *mmio = gvt->firmware.mmio;
		297
		298	if (dmlr) {
		299	memcpy(vgpu->mmio.vreg, mmio, info->mmio_size);
		300	memcpy(vgpu->mmio.sreg, mmio, info->mmio_size);
379		301
380	memcpy(vgpu->mmio.vreg, gvt->firmware.mmio, info->mmio_size);	302	vgpu_vreg(vgpu, GEN6_GT_THREAD_STATUS_REG) = 0;
381	memcpy(vgpu->mmio.sreg, gvt->firmware.mmio, info->mmio_size);
382		303
383	vgpu_vreg(vgpu, GEN6_GT_THREAD_STATUS_REG) = 0;	304	/* set the bit 0:2(Core C-State ) to C0 */
		305	vgpu_vreg(vgpu, GEN6_GT_CORE_STATUS) = 0;
384		306
385	/* set the bit 0:2(Core C-State ) to C0 */	307	vgpu->mmio.disable_warn_untrack = false;
386	vgpu_vreg(vgpu, GEN6_GT_CORE_STATUS) = 0;	308	} else {
		309	#define GVT_GEN8_MMIO_RESET_OFFSET (0x44200)
		310	/* only reset the engine related, so starting with 0x44200
		311	* interrupt include DE,display mmio related will not be
		312	* touched
		313	*/
		314	memcpy(vgpu->mmio.vreg, mmio, GVT_GEN8_MMIO_RESET_OFFSET);
		315	memcpy(vgpu->mmio.sreg, mmio, GVT_GEN8_MMIO_RESET_OFFSET);
		316	}
387		317
388	vgpu->mmio.disable_warn_untrack = false;
389	}	318	}
390		319
391	/**	320	/**
@@ -405,7 +334,7 @@ int intel_vgpu_init_mmio(struct intel_vgpu *vgpu) · 13 unmodified lines
405		334
406	vgpu->mmio.sreg = vgpu->mmio.vreg + info->mmio_size;	335	vgpu->mmio.sreg = vgpu->mmio.vreg + info->mmio_size;
407		336
408	intel_vgpu_reset_mmio(vgpu);	337	intel_vgpu_reset_mmio(vgpu, true);
409		338
410	return 0;	339	return 0;
411	}	340	}

drivers/gpu/drm/i915/gvt/mmio.h+16−28

@@ -39,36 +39,28 @@ · 38 unmodified lines
39	struct intel_gvt;	39	struct intel_gvt;
40	struct intel_vgpu;	40	struct intel_vgpu;
41		41
42	#define D_SNB (1 << 0)	42	#define D_BDW (1 << 0)
43	#define D_IVB (1 << 1)	43	#define D_SKL (1 << 1)
44	#define D_HSW (1 << 2)	44	#define D_KBL (1 << 2)
45	#define D_BDW (1 << 3)
46	#define D_SKL (1 << 4)
47	#define D_KBL (1 << 5)
48		45
49	#define D_GEN9PLUS (D_SKL \| D_KBL)	46	#define D_GEN9PLUS (D_SKL \| D_KBL)
50	#define D_GEN8PLUS (D_BDW \| D_SKL \| D_KBL)	47	#define D_GEN8PLUS (D_BDW \| D_SKL \| D_KBL)
51	#define D_GEN75PLUS (D_HSW \| D_BDW \| D_SKL \| D_KBL)
52	#define D_GEN7PLUS (D_IVB \| D_HSW \| D_BDW \| D_SKL \| D_KBL)
53		48
54	#define D_SKL_PLUS (D_SKL \| D_KBL)	49	#define D_SKL_PLUS (D_SKL \| D_KBL)
55	#define D_BDW_PLUS (D_BDW \| D_SKL \| D_KBL)	50	#define D_BDW_PLUS (D_BDW \| D_SKL \| D_KBL)
56	#define D_HSW_PLUS (D_HSW \| D_BDW \| D_SKL \| D_KBL)
57	#define D_IVB_PLUS (D_IVB \| D_HSW \| D_BDW \| D_SKL \| D_KBL)
58		51
59	#define D_PRE_BDW (D_SNB \| D_IVB \| D_HSW)	52	#define D_PRE_SKL (D_BDW)
60	#define D_PRE_SKL (D_SNB \| D_IVB \| D_HSW \| D_BDW)	53	#define D_ALL (D_BDW \| D_SKL \| D_KBL)
61	#define D_ALL (D_SNB \| D_IVB \| D_HSW \| D_BDW \| D_SKL \| D_KBL)	54
		55	typedef int (gvt_mmio_func)(struct intel_vgpu , unsigned int, void *,
		56	unsigned int);
62		57
63	struct intel_gvt_mmio_info {	58	struct intel_gvt_mmio_info {
64	u32 offset;	59	u32 offset;
65	u32 size;
66	u32 length;
67	u32 addr_mask;
68	u64 ro_mask;	60	u64 ro_mask;
69	u32 device;	61	u32 device;
70	int (read)(struct intel_vgpu , unsigned int, void *, unsigned int);	62	gvt_mmio_func read;
71	int (write)(struct intel_vgpu , unsigned int, void *, unsigned int);	63	gvt_mmio_func write;
72	u32 addr_range;	64	u32 addr_range;
73	struct hlist_node node;	65	struct hlist_node node;
74	};	66	};
@@ -79,8 +71,6 @@ bool intel_gvt_match_device(struct intel_gvt *gvt, unsigned long device); · 4 unmodified lines
79	int intel_gvt_setup_mmio_info(struct intel_gvt *gvt);	71	int intel_gvt_setup_mmio_info(struct intel_gvt *gvt);
80	void intel_gvt_clean_mmio_info(struct intel_gvt *gvt);	72	void intel_gvt_clean_mmio_info(struct intel_gvt *gvt);
81		73
82	struct intel_gvt_mmio_info intel_gvt_find_mmio_info(struct intel_gvt gvt,
83	unsigned int offset);
84	#define INTEL_GVT_MMIO_OFFSET(reg) ({ \	74	#define INTEL_GVT_MMIO_OFFSET(reg) ({ \
85	typeof(reg) __reg = reg; \	75	typeof(reg) __reg = reg; \
86	u32 offset = (u32 )&__reg; \	76	u32 offset = (u32 )&__reg; \
@@ -88,7 +78,7 @@ struct intel_gvt_mmio_info intel_gvt_find_mmio_info(struct intel_gvt gvt, · 1 unmodified line
88	})	78	})
89		79
90	int intel_vgpu_init_mmio(struct intel_vgpu *vgpu);	80	int intel_vgpu_init_mmio(struct intel_vgpu *vgpu);
91	void intel_vgpu_reset_mmio(struct intel_vgpu *vgpu);	81	void intel_vgpu_reset_mmio(struct intel_vgpu *vgpu, bool dmlr);
92	void intel_vgpu_clean_mmio(struct intel_vgpu *vgpu);	82	void intel_vgpu_clean_mmio(struct intel_vgpu *vgpu);
93		83
94	int intel_vgpu_gpa_to_mmio_offset(struct intel_vgpu *vgpu, u64 gpa);	84	int intel_vgpu_gpa_to_mmio_offset(struct intel_vgpu *vgpu, u64 gpa);
@@ -97,13 +87,7 @@ int intel_vgpu_emulate_mmio_read(struct intel_vgpu *vgpu, u64 pa, · 2 unmodified lines
97	void *p_data, unsigned int bytes);	87	void *p_data, unsigned int bytes);
98	int intel_vgpu_emulate_mmio_write(struct intel_vgpu *vgpu, u64 pa,	88	int intel_vgpu_emulate_mmio_write(struct intel_vgpu *vgpu, u64 pa,
99	void *p_data, unsigned int bytes);	89	void *p_data, unsigned int bytes);
100	bool intel_gvt_mmio_is_cmd_access(struct intel_gvt *gvt,	90
101	unsigned int offset);
102	bool intel_gvt_mmio_is_unalign(struct intel_gvt *gvt, unsigned int offset);
103	void intel_gvt_mmio_set_accessed(struct intel_gvt *gvt, unsigned int offset);
104	void intel_gvt_mmio_set_cmd_accessed(struct intel_gvt *gvt,
105	unsigned int offset);
106	bool intel_gvt_mmio_has_mode_mask(struct intel_gvt *gvt, unsigned int offset);
107	int intel_vgpu_default_mmio_read(struct intel_vgpu *vgpu, unsigned int offset,	91	int intel_vgpu_default_mmio_read(struct intel_vgpu *vgpu, unsigned int offset,
108	void *p_data, unsigned int bytes);	92	void *p_data, unsigned int bytes);
109	int intel_vgpu_default_mmio_write(struct intel_vgpu *vgpu, unsigned int offset,	93	int intel_vgpu_default_mmio_write(struct intel_vgpu *vgpu, unsigned int offset,
@@ -111,4 +95,8 @@ int intel_vgpu_default_mmio_write(struct intel_vgpu *vgpu, unsigned int offset, · 1 unmodified line
111		95
112	bool intel_gvt_in_force_nonpriv_whitelist(struct intel_gvt *gvt,	96	bool intel_gvt_in_force_nonpriv_whitelist(struct intel_gvt *gvt,
113	unsigned int offset);	97	unsigned int offset);
		98
		99	int intel_vgpu_mmio_reg_rw(struct intel_vgpu *vgpu, unsigned int offset,
		100	void *pdata, unsigned int bytes, bool is_read);
		101
114	#endif	102	#endif

drivers/gpu/drm/i915/gvt/mpt.h+1−2

@@ -133,8 +133,7 @@ static inline int intel_gvt_hypervisor_inject_msi(struct intel_vgpu *vgpu) · 132 unmodified lines
133	if (WARN(control & GENMASK(15, 1), "only support one MSI format\n"))	133	if (WARN(control & GENMASK(15, 1), "only support one MSI format\n"))
134	return -EINVAL;	134	return -EINVAL;
135		135
136	gvt_dbg_irq("vgpu%d: inject msi address %x data%x\n", vgpu->id, addr,	136	trace_inject_msi(vgpu->id, addr, data);
137	data);
138		137
139	ret = intel_gvt_host.mpt->inject_msi(vgpu->handle, addr, data);	138	ret = intel_gvt_host.mpt->inject_msi(vgpu->handle, addr, data);
140	if (ret)	139	if (ret)

drivers/gpu/drm/i915/gvt/render.c+40−8

 #include "i915_drv.h"
 #include "gvt.h"
+#include "trace.h"
 struct render_mmio {
 	int ring_id;
 #define CTX_CONTEXT_CONTROL_VAL	0x03
-void intel_gvt_load_render_mmio(struct intel_vgpu *vgpu, int ring_id)
+/* Switch ring mmio values (context) from host to a vgpu. */
+static void switch_mmio_to_vgpu(struct intel_vgpu *vgpu, int ring_id)
 {
 	struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
 	struct render_mmio *mmio;
 		I915_WRITE(mmio->reg, v);
 		POSTING_READ(mmio->reg);
-		gvt_dbg_render("load reg %x old %x new %x\n",
+		trace_render_mmio(vgpu->id, "load",
-				i915_mmio_reg_offset(mmio->reg),
+				  i915_mmio_reg_offset(mmio->reg),
-				mmio->value, v);
+				  mmio->value, v);
 	}
 	handle_tlb_pending_event(vgpu, ring_id);
 }
-void intel_gvt_restore_render_mmio(struct intel_vgpu *vgpu, int ring_id)
+/* Switch ring mmio values (context) from vgpu to host. */
+static void switch_mmio_to_host(struct intel_vgpu *vgpu, int ring_id)
 {
 	struct drm_i915_private *dev_priv = vgpu->gvt->dev_priv;
 	struct render_mmio *mmio;
 		I915_WRITE(mmio->reg, v);
 		POSTING_READ(mmio->reg);
-		gvt_dbg_render("restore reg %x old %x new %x\n",
+		trace_render_mmio(vgpu->id, "restore",
-				i915_mmio_reg_offset(mmio->reg),
+				  i915_mmio_reg_offset(mmio->reg),
-				mmio->value, v);
+				  mmio->value, v);
 	}
 }
+/**
+ * intel_gvt_switch_render_mmio - switch mmio context of specific engine
+ * @pre: the last vGPU that own the engine
+ * @next: the vGPU to switch to
+ * @ring_id: specify the engine
+ *
+ * If pre is null indicates that host own the engine. If next is null
+ * indicates that we are switching to host workload.
+ */
+void intel_gvt_switch_mmio(struct intel_vgpu *pre,
+			   struct intel_vgpu *next, int ring_id)
+{
+	if (WARN_ON(!pre && !next))
+		return;
+	gvt_dbg_render("switch ring %d from %s to %s\n", ring_id,
+		       pre ? "vGPU" : "host", next ? "vGPU" : "HOST");
+	/**
+	 * TODO: Optimize for vGPU to vGPU switch by merging
+	 * switch_mmio_to_host() and switch_mmio_to_vgpu().
+	 */
+	if (pre)
+		switch_mmio_to_host(pre, ring_id);
+	if (next)
+		switch_mmio_to_vgpu(next, ring_id);
+}

drivers/gpu/drm/i915/gvt/render.h+2−2

@@ -36,8 +36,8 @@ · 35 unmodified lines
36	#ifndef __GVT_RENDER_H__	36	#ifndef __GVT_RENDER_H__
37	#define __GVT_RENDER_H__	37	#define __GVT_RENDER_H__
38		38
39	void intel_gvt_load_render_mmio(struct intel_vgpu *vgpu, int ring_id);	39	void intel_gvt_switch_mmio(struct intel_vgpu *pre,
		40	struct intel_vgpu *next, int ring_id);
40		41
41	void intel_gvt_restore_render_mmio(struct intel_vgpu *vgpu, int ring_id);
42		42
43	#endif	43	#endif

drivers/gpu/drm/i915/gvt/sched_policy.c+22−5

 	struct intel_gvt_workload_scheduler *scheduler = &gvt->scheduler;
 	struct vgpu_sched_data *vgpu_data;
 	struct intel_vgpu *vgpu = NULL;
-	static uint64_t timer_check;
-	if (!(timer_check++ % GVT_TS_BALANCE_PERIOD_MS))
-		gvt_balance_timeslice(sched_data);
 	/* no active vgpu or has already had a target */
 	if (list_empty(&sched_data->lru_runq_head) || scheduler->next_vgpu)
 		goto out;
 void intel_gvt_schedule(struct intel_gvt *gvt)
 {
 	struct gvt_sched_data *sched_data = gvt->scheduler.sched_data;
+	static uint64_t timer_check;
 	mutex_lock(&gvt->lock);
+	if (test_and_clear_bit(INTEL_GVT_REQUEST_SCHED,
+				(void *)&gvt->service_request)) {
+		if (!(timer_check++ % GVT_TS_BALANCE_PERIOD_MS))
+			gvt_balance_timeslice(sched_data);
+	}
+	clear_bit(INTEL_GVT_REQUEST_EVENT_SCHED, (void *)&gvt->service_request);
 	tbs_sched_func(sched_data);
 	mutex_unlock(&gvt->lock);
 }
 static void tbs_sched_clean_vgpu(struct intel_vgpu *vgpu)
 {
+	struct intel_gvt_workload_scheduler *scheduler = &vgpu->gvt->scheduler;
+	int ring_id;
 	kfree(vgpu->sched_data);
 	vgpu->sched_data = NULL;
+	spin_lock_bh(&scheduler->mmio_context_lock);
+	for (ring_id = 0; ring_id < I915_NUM_ENGINES; ring_id++) {
+		if (scheduler->engine_owner[ring_id] == vgpu) {
+			intel_gvt_switch_mmio(vgpu, NULL, ring_id);
+			scheduler->engine_owner[ring_id] = NULL;
+		}
+	}
+	spin_unlock_bh(&scheduler->mmio_context_lock);
 }
 static void tbs_sched_start_schedule(struct intel_vgpu *vgpu)

drivers/gpu/drm/i915/gvt/scheduler.c+32−7

 	struct intel_gvt *gvt = container_of(nb, struct intel_gvt,
 				shadow_ctx_notifier_block[req->engine->id]);
 	struct intel_gvt_workload_scheduler *scheduler = &gvt->scheduler;
-	struct intel_vgpu_workload *workload =
+	enum intel_engine_id ring_id = req->engine->id;
-		scheduler->current_workload[req->engine->id];
+	struct intel_vgpu_workload *workload;
+	if (!is_gvt_request(req)) {
+		spin_lock_bh(&scheduler->mmio_context_lock);
+		if (action == INTEL_CONTEXT_SCHEDULE_IN &&
+		    scheduler->engine_owner[ring_id]) {
+			/* Switch ring from vGPU to host. */
+			intel_gvt_switch_mmio(scheduler->engine_owner[ring_id],
+					      NULL, ring_id);
+			scheduler->engine_owner[ring_id] = NULL;
+		}
+		spin_unlock_bh(&scheduler->mmio_context_lock);
-	if (!is_gvt_request(req) || unlikely(!workload))
+		return NOTIFY_OK;
+	}
+	workload = scheduler->current_workload[ring_id];
+	if (unlikely(!workload))
 		return NOTIFY_OK;
 	switch (action) {
 	case INTEL_CONTEXT_SCHEDULE_IN:
-		intel_gvt_load_render_mmio(workload->vgpu,
+		spin_lock_bh(&scheduler->mmio_context_lock);
-					   workload->ring_id);
+		if (workload->vgpu != scheduler->engine_owner[ring_id]) {
+			/* Switch ring from host to vGPU or vGPU to vGPU. */
+			intel_gvt_switch_mmio(scheduler->engine_owner[ring_id],
+					      workload->vgpu, ring_id);
+			scheduler->engine_owner[ring_id] = workload->vgpu;
+		} else
+			gvt_dbg_sched("skip ring %d mmio switch for vgpu%d\n",
+				      ring_id, workload->vgpu->id);
+		spin_unlock_bh(&scheduler->mmio_context_lock);
 		atomic_set(&workload->shadow_ctx_active, 1);
 		break;
 	case INTEL_CONTEXT_SCHEDULE_OUT:
-		intel_gvt_restore_render_mmio(workload->vgpu,
-					      workload->ring_id);
 		/* If the status is -EINPROGRESS means this workload
 		 * doesn't meet any issue during dispatching so when
 		 * get the SCHEDULE_OUT set the status to be zero for
 	atomic_dec(&vgpu->running_workload_num);
 	wake_up(&scheduler->workload_complete_wq);
+	if (gvt->scheduler.need_reschedule)
+		intel_gvt_request_service(gvt, INTEL_GVT_REQUEST_EVENT_SCHED);
 	mutex_unlock(&gvt->lock);
 }

drivers/gpu/drm/i915/gvt/scheduler.h+4−0

 	struct intel_vgpu_workload *current_workload[I915_NUM_ENGINES];
 	bool need_reschedule;
+	spinlock_t mmio_context_lock;
+	/* can be null when owner is host */
+	struct intel_vgpu *engine_owner[I915_NUM_ENGINES];
 	wait_queue_head_t workload_complete_wq;
 	struct task_struct *thread[I915_NUM_ENGINES];
 	wait_queue_head_t waitq[I915_NUM_ENGINES];

drivers/gpu/drm/i915/gvt/trace.h+129−49

@@ -224,58 +224,138 @@ TRACE_EVENT(oos_sync, · 223 unmodified lines
224	TP_printk("%s", __entry->buf)	224	TP_printk("%s", __entry->buf)
225	);	225	);
226		226
227	#define MAX_CMD_STR_LEN 256
228	TRACE_EVENT(gvt_command,	227	TRACE_EVENT(gvt_command,
229	TP_PROTO(u8 vm_id, u8 ring_id, u32 ip_gma, u32 *cmd_va, u32 cmd_len, bool ring_buffer_cmd, cycles_t cost_pre_cmd_handler, cycles_t cost_cmd_handler),	228	TP_PROTO(u8 vgpu_id, u8 ring_id, u32 ip_gma, u32 *cmd_va, u32 cmd_len,
230		229	u32 buf_type),
231	TP_ARGS(vm_id, ring_id, ip_gma, cmd_va, cmd_len, ring_buffer_cmd, cost_pre_cmd_handler, cost_cmd_handler),	230
232		231	TP_ARGS(vgpu_id, ring_id, ip_gma, cmd_va, cmd_len, buf_type),
233	TP_STRUCT__entry(	232
234	__field(u8, vm_id)	233	TP_STRUCT__entry(
235	__field(u8, ring_id)	234	__field(u8, vgpu_id)
236	__field(int, i)	235	__field(u8, ring_id)
237	__array(char, tmp_buf, MAX_CMD_STR_LEN)	236	__field(u32, ip_gma)
238	__array(char, cmd_str, MAX_CMD_STR_LEN)	237	__field(u32, buf_type)
239	),	238	__field(u32, cmd_len)
240		239	__dynamic_array(u32, raw_cmd, cmd_len)
241	TP_fast_assign(	240	),
242	__entry->vm_id = vm_id;	241
243	__entry->ring_id = ring_id;	242	TP_fast_assign(
244	__entry->cmd_str[0] = '\0';	243	__entry->vgpu_id = vgpu_id;
245	snprintf(__entry->tmp_buf, MAX_CMD_STR_LEN, "VM(%d) Ring(%d): %s ip(%08x) pre handler cost (%llu), handler cost (%llu) ", vm_id, ring_id, ring_buffer_cmd ? "RB":"BB", ip_gma, cost_pre_cmd_handler, cost_cmd_handler);	244	__entry->ring_id = ring_id;
246	strcat(__entry->cmd_str, __entry->tmp_buf);	245	__entry->ip_gma = ip_gma;
247	entry->i = 0;	246	__entry->buf_type = buf_type;
248	while (cmd_len > 0) {	247	__entry->cmd_len = cmd_len;
249	if (cmd_len >= 8) {	248	memcpy(__get_dynamic_array(raw_cmd), cmd_va, cmd_len * sizeof(*cmd_va));
250	snprintf(__entry->tmp_buf, MAX_CMD_STR_LEN, "%08x %08x %08x %08x %08x %08x %08x %08x ",	249	),
251	cmd_va[__entry->i], cmd_va[__entry->i+1], cmd_va[__entry->i+2], cmd_va[__entry->i+3],	250
252	cmd_va[__entry->i+4], cmd_va[__entry->i+5], cmd_va[__entry->i+6], cmd_va[__entry->i+7]);	251
253	__entry->i += 8;	252	TP_printk("vgpu%d ring %d: buf_type %u, ip_gma %08x, raw cmd %s",
254	cmd_len -= 8;	253	__entry->vgpu_id,
255	strcat(__entry->cmd_str, __entry->tmp_buf);	254	__entry->ring_id,
256	} else if (cmd_len >= 4) {	255	__entry->buf_type,
257	snprintf(__entry->tmp_buf, MAX_CMD_STR_LEN, "%08x %08x %08x %08x ",	256	__entry->ip_gma,
258	cmd_va[__entry->i], cmd_va[__entry->i+1], cmd_va[__entry->i+2], cmd_va[__entry->i+3]);	257	__print_array(__get_dynamic_array(raw_cmd), __entry->cmd_len, 4))
259	__entry->i += 4;	258	);
260	cmd_len -= 4;	259
261	strcat(__entry->cmd_str, __entry->tmp_buf);	260	#define GVT_TEMP_STR_LEN 10
262	} else if (cmd_len >= 2) {	261	TRACE_EVENT(write_ir,
263	snprintf(__entry->tmp_buf, MAX_CMD_STR_LEN, "%08x %08x ", cmd_va[__entry->i], cmd_va[__entry->i+1]);	262	TP_PROTO(int id, char *reg_name, unsigned int reg, unsigned int new_val,
264	__entry->i += 2;	263	unsigned int old_val, bool changed),
265	cmd_len -= 2;	264
266	strcat(__entry->cmd_str, __entry->tmp_buf);	265	TP_ARGS(id, reg_name, reg, new_val, old_val, changed),
267	} else if (cmd_len == 1) {	266
268	snprintf(__entry->tmp_buf, MAX_CMD_STR_LEN, "%08x ", cmd_va[__entry->i]);	267	TP_STRUCT__entry(
269	__entry->i += 1;	268	__field(int, id)
270	cmd_len -= 1;	269	__array(char, buf, GVT_TEMP_STR_LEN)
271	strcat(__entry->cmd_str, __entry->tmp_buf);	270	__field(unsigned int, reg)
272	}	271	__field(unsigned int, new_val)
273	}	272	__field(unsigned int, old_val)
274	strcat(__entry->cmd_str, "\n");	273	__field(bool, changed)
275	),	274	),
		275
		276	TP_fast_assign(
		277	__entry->id = id;
		278	snprintf(__entry->buf, GVT_TEMP_STR_LEN, "%s", reg_name);
		279	__entry->reg = reg;
		280	__entry->new_val = new_val;
		281	__entry->old_val = old_val;
		282	__entry->changed = changed;
		283	),
		284
		285	TP_printk("VM%u write [%s] %x, new %08x, old %08x, changed %08x\n",
		286	__entry->id, __entry->buf, __entry->reg, __entry->new_val,
		287	__entry->old_val, __entry->changed)
		288	);
		289
		290	TRACE_EVENT(propagate_event,
		291	TP_PROTO(int id, const char *irq_name, int bit),
		292
		293	TP_ARGS(id, irq_name, bit),
		294
		295	TP_STRUCT__entry(
		296	__field(int, id)
		297	__array(char, buf, GVT_TEMP_STR_LEN)
		298	__field(int, bit)
		299	),
276		300
277	TP_printk("%s", __entry->cmd_str)	301	TP_fast_assign(
		302	__entry->id = id;
		303	snprintf(__entry->buf, GVT_TEMP_STR_LEN, "%s", irq_name);
		304	__entry->bit = bit;
		305	),
		306
		307	TP_printk("Set bit (%d) for (%s) for vgpu (%d)\n",
		308	__entry->bit, __entry->buf, __entry->id)
278	);	309	);
		310
		311	TRACE_EVENT(inject_msi,
		312	TP_PROTO(int id, unsigned int address, unsigned int data),
		313
		314	TP_ARGS(id, address, data),
		315
		316	TP_STRUCT__entry(
		317	__field(int, id)
		318	__field(unsigned int, address)
		319	__field(unsigned int, data)
		320	),
		321
		322	TP_fast_assign(
		323	__entry->id = id;
		324	__entry->address = address;
		325	__entry->data = data;
		326	),
		327
		328	TP_printk("vgpu%d:inject msi address %x data %x\n",
		329	__entry->id, __entry->address, __entry->data)
		330	);
		331
		332	TRACE_EVENT(render_mmio,
		333	TP_PROTO(int id, char *action, unsigned int reg,
		334	unsigned int old_val, unsigned int new_val),
		335
		336	TP_ARGS(id, action, reg, new_val, old_val),
		337
		338	TP_STRUCT__entry(
		339	__field(int, id)
		340	__array(char, buf, GVT_TEMP_STR_LEN)
		341	__field(unsigned int, reg)
		342	__field(unsigned int, old_val)
		343	__field(unsigned int, new_val)
		344	),
		345
		346	TP_fast_assign(
		347	__entry->id = id;
		348	snprintf(__entry->buf, GVT_TEMP_STR_LEN, "%s", action);
		349	__entry->reg = reg;
		350	__entry->old_val = old_val;
		351	__entry->new_val = new_val;
		352	),
		353
		354	TP_printk("VM%u %s reg %x, old %08x new %08x\n",
		355	__entry->id, __entry->buf, __entry->reg,
		356	__entry->old_val, __entry->new_val)
		357	);
		358
279	#endif /* _GVT_TRACE_H_ */	359	#endif /* _GVT_TRACE_H_ */
280		360
281	/* This part must be out of protection */	361	/* This part must be out of protection */

drivers/gpu/drm/i915/gvt/vgpu.c+7−2

@@ -501,9 +501,14 @@ void intel_gvt_reset_vgpu_locked(struct intel_vgpu *vgpu, bool dmlr, · 500 unmodified lines
501		501
502	/* full GPU reset or device model level reset */	502	/* full GPU reset or device model level reset */
503	if (engine_mask == ALL_ENGINES \|\| dmlr) {	503	if (engine_mask == ALL_ENGINES \|\| dmlr) {
		504
504	intel_vgpu_reset_gtt(vgpu, dmlr);	505	intel_vgpu_reset_gtt(vgpu, dmlr);
505	intel_vgpu_reset_resource(vgpu);	506
506	intel_vgpu_reset_mmio(vgpu);	507	/fence will not be reset during virtual reset /
		508	if (dmlr)
		509	intel_vgpu_reset_resource(vgpu);
		510
		511	intel_vgpu_reset_mmio(vgpu, dmlr);
507	populate_pvinfo_page(vgpu);	512	populate_pvinfo_page(vgpu);
508	intel_vgpu_reset_display(vgpu);	513	intel_vgpu_reset_display(vgpu);
509		514

drivers/gpu/drm/i915/i915_debugfs.c+41−27

@@ -1670,12 +1670,22 @@ static int i915_fbc_status(struct seq_file m, void unused) · 1669 unmodified lines
1670	seq_printf(m, "FBC disabled: %s\n",	1670	seq_printf(m, "FBC disabled: %s\n",
1671	dev_priv->fbc.no_fbc_reason);	1671	dev_priv->fbc.no_fbc_reason);
1672		1672
1673	if (intel_fbc_is_active(dev_priv) && INTEL_GEN(dev_priv) >= 7) {	1673	if (intel_fbc_is_active(dev_priv)) {
1674	uint32_t mask = INTEL_GEN(dev_priv) >= 8 ?	1674	u32 mask;
1675	BDW_FBC_COMPRESSION_MASK :	1675
1676	IVB_FBC_COMPRESSION_MASK;	1676	if (INTEL_GEN(dev_priv) >= 8)
1677	seq_printf(m, "Compressing: %s\n",	1677	mask = I915_READ(IVB_FBC_STATUS2) & BDW_FBC_COMP_SEG_MASK;
1678	yesno(I915_READ(FBC_STATUS2) & mask));	1678	else if (INTEL_GEN(dev_priv) >= 7)
		1679	mask = I915_READ(IVB_FBC_STATUS2) & IVB_FBC_COMP_SEG_MASK;
		1680	else if (INTEL_GEN(dev_priv) >= 5)
		1681	mask = I915_READ(ILK_DPFC_STATUS) & ILK_DPFC_COMP_SEG_MASK;
		1682	else if (IS_G4X(dev_priv))
		1683	mask = I915_READ(DPFC_STATUS) & DPFC_COMP_SEG_MASK;
		1684	else
		1685	mask = I915_READ(FBC_STATUS) & (FBC_STAT_COMPRESSING \|
		1686	FBC_STAT_COMPRESSED);
		1687
		1688	seq_printf(m, "Compressing: %s\n", yesno(mask));
1679	}	1689	}
1680		1690
1681	mutex_unlock(&dev_priv->fbc.lock);	1691	mutex_unlock(&dev_priv->fbc.lock);
@@ -1684,7 +1694,7 @@ static int i915_fbc_status(struct seq_file m, void unused) · 2 unmodified lines
1684	return 0;	1694	return 0;
1685	}	1695	}
1686		1696
1687	static int i915_fbc_fc_get(void data, u64 val)	1697	static int i915_fbc_false_color_get(void data, u64 val)
1688	{	1698	{
1689	struct drm_i915_private *dev_priv = data;	1699	struct drm_i915_private *dev_priv = data;
1690		1700
@@ -1696,7 +1706,7 @@ static int i915_fbc_fc_get(void data, u64 val) · 5 unmodified lines
1696	return 0;	1706	return 0;
1697	}	1707	}
1698		1708
1699	static int i915_fbc_fc_set(void *data, u64 val)	1709	static int i915_fbc_false_color_set(void *data, u64 val)
1700	{	1710	{
1701	struct drm_i915_private *dev_priv = data;	1711	struct drm_i915_private *dev_priv = data;
1702	u32 reg;	1712	u32 reg;
@@ -1717,8 +1727,8 @@ static int i915_fbc_fc_set(void *data, u64 val) · 14 unmodified lines
1717	return 0;	1727	return 0;
1718	}	1728	}
1719		1729
1720	DEFINE_SIMPLE_ATTRIBUTE(i915_fbc_fc_fops,	1730	DEFINE_SIMPLE_ATTRIBUTE(i915_fbc_false_color_fops,
1721	i915_fbc_fc_get, i915_fbc_fc_set,	1731	i915_fbc_false_color_get, i915_fbc_false_color_set,
1722	"%llu\n");	1732	"%llu\n");
1723		1733
1724	static int i915_ips_status(struct seq_file m, void unused)	1734	static int i915_ips_status(struct seq_file m, void unused)
@@ -1988,6 +1998,12 @@ static int i915_context_status(struct seq_file m, void unused) · 263 unmodified lines
1988	seq_putc(m, '\n');	1998	seq_putc(m, '\n');
1989	}	1999	}
1990		2000
		2001	seq_printf(m,
		2002	"\tvma hashtable size=%u (actual %lu), count=%u\n",
		2003	ctx->vma_lut.ht_size,
		2004	BIT(ctx->vma_lut.ht_bits),
		2005	ctx->vma_lut.ht_count);
		2006
1991	seq_putc(m, '\n');	2007	seq_putc(m, '\n');
1992	}	2008	}
1993		2009
@@ -4289,26 +4305,27 @@ i915_drop_caches_set(void *data, u64 val) · 2295 unmodified lines
4289	{	4305	{
4290	struct drm_i915_private *dev_priv = data;	4306	struct drm_i915_private *dev_priv = data;
4291	struct drm_device *dev = &dev_priv->drm;	4307	struct drm_device *dev = &dev_priv->drm;
4292	int ret;	4308	int ret = 0;
4293		4309
4294	DRM_DEBUG("Dropping caches: 0x%08llx\n", val);	4310	DRM_DEBUG("Dropping caches: 0x%08llx\n", val);
4295		4311
4296	/* No need to check and wait for gpu resets, only libdrm auto-restarts	4312	/* No need to check and wait for gpu resets, only libdrm auto-restarts
4297	* on ioctls on -EAGAIN. */	4313	* on ioctls on -EAGAIN. */
4298	ret = mutex_lock_interruptible(&dev->struct_mutex);	4314	if (val & (DROP_ACTIVE \| DROP_RETIRE)) {
4299	if (ret)	4315	ret = mutex_lock_interruptible(&dev->struct_mutex);
4300	return ret;
4301
4302	if (val & DROP_ACTIVE) {
4303	ret = i915_gem_wait_for_idle(dev_priv,
4304	I915_WAIT_INTERRUPTIBLE \|
4305	I915_WAIT_LOCKED);
4306	if (ret)	4316	if (ret)
4307	goto unlock;	4317	return ret;
4308	}
4309		4318
4310	if (val & DROP_RETIRE)	4319	if (val & DROP_ACTIVE)
4311	i915_gem_retire_requests(dev_priv);	4320	ret = i915_gem_wait_for_idle(dev_priv,
		4321	I915_WAIT_INTERRUPTIBLE \|
		4322	I915_WAIT_LOCKED);
		4323
		4324	if (val & DROP_RETIRE)
		4325	i915_gem_retire_requests(dev_priv);
		4326
		4327	mutex_unlock(&dev->struct_mutex);
		4328	}
4312		4329
4313	lockdep_set_current_reclaim_state(GFP_KERNEL);	4330	lockdep_set_current_reclaim_state(GFP_KERNEL);
4314	if (val & DROP_BOUND)	4331	if (val & DROP_BOUND)
@@ -4321,9 +4338,6 @@ i915_drop_caches_set(void *data, u64 val) · 6 unmodified lines
4321	i915_gem_shrink_all(dev_priv);	4338	i915_gem_shrink_all(dev_priv);
4322	lockdep_clear_current_reclaim_state();	4339	lockdep_clear_current_reclaim_state();
4323		4340
4324	unlock:
4325	mutex_unlock(&dev->struct_mutex);
4326
4327	if (val & DROP_FREED) {	4341	if (val & DROP_FREED) {
4328	synchronize_rcu();	4342	synchronize_rcu();
4329	i915_gem_drain_freed_objects(dev_priv);	4343	i915_gem_drain_freed_objects(dev_priv);
@@ -4861,7 +4875,7 @@ static const struct i915_debugfs_files { · 531 unmodified lines
4861	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},	4875	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
4862	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},	4876	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
4863	{"i915_cur_wm_latency", &i915_cur_wm_latency_fops},	4877	{"i915_cur_wm_latency", &i915_cur_wm_latency_fops},
4864	{"i915_fbc_false_color", &i915_fbc_fc_fops},	4878	{"i915_fbc_false_color", &i915_fbc_false_color_fops},
4865	{"i915_dp_test_data", &i915_displayport_test_data_fops},	4879	{"i915_dp_test_data", &i915_displayport_test_data_fops},
4866	{"i915_dp_test_type", &i915_displayport_test_type_fops},	4880	{"i915_dp_test_type", &i915_displayport_test_type_fops},
4867	{"i915_dp_test_active", &i915_displayport_test_active_fops},	4881	{"i915_dp_test_active", &i915_displayport_test_active_fops},

drivers/gpu/drm/i915/i915_drv.c+41−5

 	} else if (IS_SKYLAKE(dev_priv) || IS_KABYLAKE(dev_priv)) {
 		ret = PCH_SPT;
 		DRM_DEBUG_KMS("Assuming SunrisePoint PCH\n");
+	} else if (IS_COFFEELAKE(dev_priv) || IS_CANNONLAKE(dev_priv)) {
+		ret = PCH_CNP;
+		DRM_DEBUG_KMS("Assuming CannonPoint PCH\n");
 	}
 	return ret;
 	while ((pch = pci_get_class(PCI_CLASS_BRIDGE_ISA << 8, pch))) {
 		if (pch->vendor == PCI_VENDOR_ID_INTEL) {
 			unsigned short id = pch->device & INTEL_PCH_DEVICE_ID_MASK;
-			dev_priv->pch_id = id;
+			unsigned short id_ext = pch->device &
+				INTEL_PCH_DEVICE_ID_MASK_EXT;
 			if (id == INTEL_PCH_IBX_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_IBX;
 				DRM_DEBUG_KMS("Found Ibex Peak PCH\n");
 				WARN_ON(!IS_GEN5(dev_priv));
 			} else if (id == INTEL_PCH_CPT_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_CPT;
 				DRM_DEBUG_KMS("Found CougarPoint PCH\n");
 				WARN_ON(!(IS_GEN6(dev_priv) ||
 					IS_IVYBRIDGE(dev_priv)));
 			} else if (id == INTEL_PCH_PPT_DEVICE_ID_TYPE) {
 				/* PantherPoint is CPT compatible */
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_CPT;
 				DRM_DEBUG_KMS("Found PantherPoint PCH\n");
 				WARN_ON(!(IS_GEN6(dev_priv) ||
 					IS_IVYBRIDGE(dev_priv)));
 			} else if (id == INTEL_PCH_LPT_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_LPT;
 				DRM_DEBUG_KMS("Found LynxPoint PCH\n");
 				WARN_ON(!IS_HASWELL(dev_priv) &&
 				WARN_ON(IS_HSW_ULT(dev_priv) ||
 					IS_BDW_ULT(dev_priv));
 			} else if (id == INTEL_PCH_LPT_LP_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_LPT;
 				DRM_DEBUG_KMS("Found LynxPoint LP PCH\n");
 				WARN_ON(!IS_HASWELL(dev_priv) &&
 				WARN_ON(!IS_HSW_ULT(dev_priv) &&
 					!IS_BDW_ULT(dev_priv));
 			} else if (id == INTEL_PCH_SPT_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_SPT;
 				DRM_DEBUG_KMS("Found SunrisePoint PCH\n");
 				WARN_ON(!IS_SKYLAKE(dev_priv) &&
 					!IS_KABYLAKE(dev_priv));
-			} else if (id == INTEL_PCH_SPT_LP_DEVICE_ID_TYPE) {
+			} else if (id_ext == INTEL_PCH_SPT_LP_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id_ext;
 				dev_priv->pch_type = PCH_SPT;
 				DRM_DEBUG_KMS("Found SunrisePoint LP PCH\n");
 				WARN_ON(!IS_SKYLAKE(dev_priv) &&
 					!IS_KABYLAKE(dev_priv));
 			} else if (id == INTEL_PCH_KBP_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type = PCH_KBP;
 				DRM_DEBUG_KMS("Found KabyPoint PCH\n");
 				WARN_ON(!IS_SKYLAKE(dev_priv) &&
 					!IS_KABYLAKE(dev_priv));
+			} else if (id == INTEL_PCH_CNP_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id;
+				dev_priv->pch_type = PCH_CNP;
+				DRM_DEBUG_KMS("Found CannonPoint PCH\n");
+				WARN_ON(!IS_CANNONLAKE(dev_priv) &&
+					!IS_COFFEELAKE(dev_priv));
+			} else if (id_ext == INTEL_PCH_CNP_LP_DEVICE_ID_TYPE) {
+				dev_priv->pch_id = id_ext;
+				dev_priv->pch_type = PCH_CNP;
+				DRM_DEBUG_KMS("Found CannonPoint LP PCH\n");
+				WARN_ON(!IS_CANNONLAKE(dev_priv) &&
+					!IS_COFFEELAKE(dev_priv));
 			} else if ((id == INTEL_PCH_P2X_DEVICE_ID_TYPE) ||
 				   (id == INTEL_PCH_P3X_DEVICE_ID_TYPE) ||
 				   ((id == INTEL_PCH_QEMU_DEVICE_ID_TYPE) &&
 					    PCI_SUBVENDOR_ID_REDHAT_QUMRANET &&
 				    pch->subsystem_device ==
 					    PCI_SUBDEVICE_ID_QEMU)) {
+				dev_priv->pch_id = id;
 				dev_priv->pch_type =
 					intel_virt_detect_pch(dev_priv);
 			} else
 	case I915_PARAM_HAS_EXEC_ASYNC:
 	case I915_PARAM_HAS_EXEC_FENCE:
 	case I915_PARAM_HAS_EXEC_CAPTURE:
+	case I915_PARAM_HAS_EXEC_BATCH_FIRST:
 		/* For the time being all of these are always true;
 		 * if some supported hardware does not have one of these
 		 * features this value needs to be provided from
 		 */
 		value = 1;
 		break;
+	case I915_PARAM_SLICE_MASK:
+		value = INTEL_INFO(dev_priv)->sseu.slice_mask;
+		if (!value)
+			return -ENODEV;
+		break;
+	case I915_PARAM_SUBSLICE_MASK:
+		value = INTEL_INFO(dev_priv)->sseu.subslice_mask;
+		if (!value)
+			return -ENODEV;
+		break;
 	default:
 		DRM_DEBUG("Unknown parameter %d\n", param->param);
 		return -EINVAL;
 	intel_uc_fini_hw(dev_priv);
 	i915_gem_cleanup_engines(dev_priv);
 	i915_gem_context_fini(dev_priv);
+	i915_gem_cleanup_userptr(dev_priv);
 	mutex_unlock(&dev_priv->drm.struct_mutex);
 	i915_gem_drain_freed_objects(dev_priv);
 	DRM_DEBUG_DRIVER("use GPU semaphores? %s\n", yesno(i915.semaphores));
 	intel_uc_sanitize_options(dev_priv);
+	intel_gvt_sanitize_options(dev_priv);
 }
 /**
 	intel_guc_resume(dev_priv);
-	if (IS_GEN6(dev_priv))
-		intel_init_pch_refclk(dev_priv);
 	if (IS_GEN9_LP(dev_priv)) {
 		bxt_disable_dc9(dev_priv);
 		bxt_display_core_init(dev_priv, true);

drivers/gpu/drm/i915/i915_drv.h+76−30

 #include <linux/i2c.h>
 #include <linux/i2c-algo-bit.h>
 #include <linux/backlight.h>
-#include <linux/hashtable.h>
+#include <linux/hash.h>
 #include <linux/intel-iommu.h>
 #include <linux/kref.h>
 #include <linux/pm_qos.h>
 #define DRIVER_NAME		"i915"
 #define DRIVER_DESC		"Intel Graphics"
-#define DRIVER_DATE		"20170529"
+#define DRIVER_DATE		"20170619"
-#define DRIVER_TIMESTAMP	1496041258
+#define DRIVER_TIMESTAMP	1497857498
 /* Use I915_STATE_WARN(x) and I915_STATE_WARN_ON() (rather than WARN() and
  * WARN_ON()) for hw state sanity checks to check for unexpected conditions
 	func(has_aliasing_ppgtt); \
 	func(has_csr); \
 	func(has_ddi); \
-	func(has_decoupled_mmio); \
 	func(has_dp_mst); \
 	func(has_fbc); \
 	func(has_fpga_dbg); \
 	INTEL_BROXTON,
 	INTEL_KABYLAKE,
 	INTEL_GEMINILAKE,
+	INTEL_COFFEELAKE,
+	INTEL_CANNONLAKE,
 	INTEL_MAX_PLATFORMS
 };
 	PCH_LPT,	/* Lynxpoint PCH */
 	PCH_SPT,        /* Sunrisepoint PCH */
 	PCH_KBP,        /* Kabypoint PCH */
+	PCH_CNP,        /* Cannonpoint PCH */
 	PCH_NOP,
 };
 	SBI_MPHY,
 };
-#define QUIRK_PIPEA_FORCE (1<<0)
 #define QUIRK_LVDS_SSC_DISABLE (1<<1)
 #define QUIRK_INVERT_BRIGHTNESS (1<<2)
 #define QUIRK_BACKLIGHT_PRESENT (1<<3)
-#define QUIRK_PIPEB_FORCE (1<<4)
 #define QUIRK_PIN_SWIZZLED_PAGES (1<<5)
 struct intel_fbdev;
 	/** LRU list of objects with fence regs on them. */
 	struct list_head fence_list;
+	/**
+	 * Workqueue to fault in userptr pages, flushed by the execbuf
+	 * when required but otherwise left to userspace to try again
+	 * on EAGAIN.
+	 */
+	struct workqueue_struct *userptr_wq;
 	u64 unordered_timeline;
 	/* the indicator for dispatch video commands on two BSD rings */
 	void (*init_oa_buffer)(struct drm_i915_private *dev_priv);
 	/**
-	 * @enable_metric_set: Applies any MUX configuration to set up the
+	 * @select_metric_set: The auto generated code that checks whether a
-	 * Boolean and Custom (B/C) counters that are part of the counter
+	 * requested OA config is applicable to the system and if so sets up
-	 * reports being sampled. May apply system constraints such as
+	 * the mux, oa and flex eu register config pointers according to the
+	 * current dev_priv->perf.oa.metrics_set.
+	 */
+	int (*select_metric_set)(struct drm_i915_private *dev_priv);
+	/**
+	 * @enable_metric_set: Selects and applies any MUX configuration to set
+	 * up the Boolean and Custom (B/C) counters that are part of the
+	 * counter reports being sampled. May apply system constraints such as
 	 * disabling EU clock gating as required.
 	 */
 	int (*enable_metric_set)(struct drm_i915_private *dev_priv);
 		    size_t *offset);
 	/**
-	 * @oa_buffer_check: Check for OA buffer data + update tail
+	 * @oa_hw_tail_read: read the OA tail pointer register
-	 *
-	 * This is either called via fops or the poll check hrtimer (atomic
-	 * ctx) without any locks taken.
 	 *
-	 * It's safe to read OA config state here unlocked, assuming that this
+	 * In particular this enables us to share all the fiddly code for
-	 * is only called while the stream is enabled, while the global OA
+	 * handling the OA unit tail pointer race that affects multiple
-	 * configuration can't be modified.
+	 * generations.
-	 *
-	 * Efficiency is more important than avoiding some false positives
-	 * here, which will be handled gracefully - likely resulting in an
-	 * %EAGAIN error for userspace.
 	 */
-	bool (*oa_buffer_check)(struct drm_i915_private *dev_priv);
+	u32 (*oa_hw_tail_read)(struct drm_i915_private *dev_priv);
 };
 struct intel_cdclk_state {
 		struct mutex lock;
 		struct list_head streams;
-		spinlock_t hook_lock;
 		struct {
 			struct i915_perf_stream *exclusive_stream;
 			bool periodic;
 			int period_exponent;
+			int timestamp_frequency;
 			int metrics_set;
-			const struct i915_oa_reg *mux_regs;
+			const struct i915_oa_reg *mux_regs[6];
-			int mux_regs_len;
+			int mux_regs_lens[6];
+			int n_mux_configs;
 			const struct i915_oa_reg *b_counter_regs;
 			int b_counter_regs_len;
+			const struct i915_oa_reg *flex_regs;
+			int flex_regs_len;
 			struct {
 				struct i915_vma *vma;
 				u8 *vaddr;
+				u32 last_ctx_id;
 				int format;
 				int format_size;
 			} oa_buffer;
 			u32 gen7_latched_oastatus1;
+			u32 ctx_oactxctrl_offset;
+			u32 ctx_flexeu0_offset;
+			/**
+			 * The RPT_ID/reason field for Gen8+ includes a bit
+			 * to determine if the CTX ID in the report is valid
+			 * but the specific bit differs between Gen 8 and 9
+			 */
+			u32 gen8_valid_ctx_bit;
 			struct i915_oa_ops ops;
 			const struct i915_oa_format *oa_formats;
 #define IS_BROXTON(dev_priv)	((dev_priv)->info.platform == INTEL_BROXTON)
 #define IS_KABYLAKE(dev_priv)	((dev_priv)->info.platform == INTEL_KABYLAKE)
 #define IS_GEMINILAKE(dev_priv)	((dev_priv)->info.platform == INTEL_GEMINILAKE)
+#define IS_COFFEELAKE(dev_priv)	((dev_priv)->info.platform == INTEL_COFFEELAKE)
+#define IS_CANNONLAKE(dev_priv)	((dev_priv)->info.platform == INTEL_CANNONLAKE)
 #define IS_MOBILE(dev_priv)	((dev_priv)->info.is_mobile)
 #define IS_HSW_EARLY_SDV(dev_priv) (IS_HASWELL(dev_priv) && \
 				    (INTEL_DEVID(dev_priv) & 0xFF00) == 0x0C00)
 #define IS_KBL_ULX(dev_priv)	(INTEL_DEVID(dev_priv) == 0x590E || \
 				 INTEL_DEVID(dev_priv) == 0x5915 || \
 				 INTEL_DEVID(dev_priv) == 0x591E)
+#define IS_SKL_GT2(dev_priv)	(IS_SKYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
 #define IS_SKL_GT3(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
 #define IS_SKL_GT4(dev_priv)	(IS_SKYLAKE(dev_priv) && \
 				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0030)
+#define IS_KBL_GT2(dev_priv)	(IS_KABYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0010)
+#define IS_KBL_GT3(dev_priv)	(IS_KABYLAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x0020)
+#define IS_CFL_ULT(dev_priv)	(IS_COFFEELAKE(dev_priv) && \
+				 (INTEL_DEVID(dev_priv) & 0x00F0) == 0x00A0)
 #define IS_ALPHA_SUPPORT(intel_info) ((intel_info)->is_alpha_support)
 #define IS_GLK_REVID(dev_priv, since, until) \
 	(IS_GEMINILAKE(dev_priv) && IS_REVID(dev_priv, since, until))
+#define CNL_REVID_A0		0x0
+#define CNL_REVID_B0		0x1
+#define IS_CNL_REVID(p, since, until) \
+	(IS_CANNONLAKE(p) && IS_REVID(p, since, until))
 /*
  * The genX designation typically refers to the render engine, so render
  * capability related checks should use IS_GEN, while display and other checks
 #define IS_GEN7(dev_priv)	(!!((dev_priv)->info.gen_mask & BIT(6)))
 #define IS_GEN8(dev_priv)	(!!((dev_priv)->info.gen_mask & BIT(7)))
 #define IS_GEN9(dev_priv)	(!!((dev_priv)->info.gen_mask & BIT(8)))
+#define IS_GEN10(dev_priv)	(!!((dev_priv)->info.gen_mask & BIT(9)))
 #define IS_LP(dev_priv)	(INTEL_INFO(dev_priv)->is_lp)
 #define IS_GEN9_LP(dev_priv)	(IS_GEN9(dev_priv) && IS_LP(dev_priv))
 #define HAS_POOLED_EU(dev_priv)	((dev_priv)->info.has_pooled_eu)
 #define INTEL_PCH_DEVICE_ID_MASK		0xff00
+#define INTEL_PCH_DEVICE_ID_MASK_EXT		0xff80
 #define INTEL_PCH_IBX_DEVICE_ID_TYPE		0x3b00
 #define INTEL_PCH_CPT_DEVICE_ID_TYPE		0x1c00
 #define INTEL_PCH_PPT_DEVICE_ID_TYPE		0x1e00
 #define INTEL_PCH_SPT_DEVICE_ID_TYPE		0xA100
 #define INTEL_PCH_SPT_LP_DEVICE_ID_TYPE		0x9D00
 #define INTEL_PCH_KBP_DEVICE_ID_TYPE		0xA200
+#define INTEL_PCH_CNP_DEVICE_ID_TYPE		0xA300
+#define INTEL_PCH_CNP_LP_DEVICE_ID_TYPE		0x9D80
 #define INTEL_PCH_P2X_DEVICE_ID_TYPE		0x7100
 #define INTEL_PCH_P3X_DEVICE_ID_TYPE		0x7000
 #define INTEL_PCH_QEMU_DEVICE_ID_TYPE		0x2900 /* qemu q35 has 2918 */
 #define INTEL_PCH_TYPE(dev_priv) ((dev_priv)->pch_type)
+#define HAS_PCH_CNP(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_CNP)
+#define HAS_PCH_CNP_LP(dev_priv) \
+	((dev_priv)->pch_id == INTEL_PCH_CNP_LP_DEVICE_ID_TYPE)
 #define HAS_PCH_KBP(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_KBP)
 #define HAS_PCH_SPT(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_SPT)
 #define HAS_PCH_LPT(dev_priv) (INTEL_PCH_TYPE(dev_priv) == PCH_LPT)
 #define HAS_GMCH_DISPLAY(dev_priv) ((dev_priv)->info.has_gmch_display)
-#define HAS_LSPCON(dev_priv) (IS_GEN9(dev_priv))
+#define HAS_LSPCON(dev_priv) (INTEL_GEN(dev_priv) >= 9)
 /* DPF == dynamic parity feature */
 #define HAS_L3_DPF(dev_priv) ((dev_priv)->info.has_l3_dpf)
 #define GT_FREQUENCY_MULTIPLIER 50
 #define GEN9_FREQ_SCALER 3
-#define HAS_DECOUPLED_MMIO(dev_priv) (INTEL_INFO(dev_priv)->has_decoupled_mmio)
 #include "i915_trace.h"
 static inline bool intel_vtd_active(void)
 			      struct drm_file *file_priv);
 int i915_gem_get_tiling_ioctl(struct drm_device *dev, void *data,
 			      struct drm_file *file_priv);
-void i915_gem_init_userptr(struct drm_i915_private *dev_priv);
+int i915_gem_init_userptr(struct drm_i915_private *dev_priv);
+void i915_gem_cleanup_userptr(struct drm_i915_private *dev_priv);
 int i915_gem_userptr_ioctl(struct drm_device *dev, void *data,
 			   struct drm_file *file);
 int i915_gem_get_aperture_ioctl(struct drm_device *dev, void *data,
 int i915_perf_open_ioctl(struct drm_device *dev, void *data,
 			 struct drm_file *file);
+void i915_oa_init_reg_state(struct intel_engine_cs *engine,
+			    struct i915_gem_context *ctx,
+			    uint32_t *reg_state);
 /* i915_gem_evict.c */
 int __must_check i915_gem_evict_something(struct i915_address_space *vm,
 int __must_check i915_gem_evict_for_node(struct i915_address_space *vm,
 					 struct drm_mm_node *node,
 					 unsigned int flags);
-int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle);
+int i915_gem_evict_vm(struct i915_address_space *vm);
 /* belongs in i915_gem_gtt.h */
 static inline void i915_gem_chipset_flush(struct drm_i915_private *dev_priv)

drivers/gpu/drm/i915/i915_gem.c+102−62

@@ -49,10 +49,10 @@ static void i915_gem_flush_free_objects(struct drm_i915_private *i915); · 48 unmodified lines
49		49
50	static bool cpu_write_needs_clflush(struct drm_i915_gem_object *obj)	50	static bool cpu_write_needs_clflush(struct drm_i915_gem_object *obj)
51	{	51	{
52	if (obj->base.write_domain == I915_GEM_DOMAIN_CPU)	52	if (obj->cache_dirty)
53	return false;	53	return false;
54		54
55	if (!i915_gem_object_is_coherent(obj))	55	if (!obj->cache_coherent)
56	return true;	56	return true;
57		57
58	return obj->pin_display;	58	return obj->pin_display;
@@ -143,9 +143,9 @@ i915_gem_get_aperture_ioctl(struct drm_device dev, void data, · 84 unmodified lines
143	struct i915_ggtt *ggtt = &dev_priv->ggtt;	143	struct i915_ggtt *ggtt = &dev_priv->ggtt;
144	struct drm_i915_gem_get_aperture *args = data;	144	struct drm_i915_gem_get_aperture *args = data;
145	struct i915_vma *vma;	145	struct i915_vma *vma;
146	size_t pinned;	146	u64 pinned;
147		147
148	pinned = 0;	148	pinned = ggtt->base.reserved;
149	mutex_lock(&dev->struct_mutex);	149	mutex_lock(&dev->struct_mutex);
150	list_for_each_entry(vma, &ggtt->base.active_list, vm_link)	150	list_for_each_entry(vma, &ggtt->base.active_list, vm_link)
151	if (i915_vma_is_pinned(vma))	151	if (i915_vma_is_pinned(vma))
@@ -233,6 +233,14 @@ err_phys: · 81 unmodified lines
233	return st;	233	return st;
234	}	234	}
235		235
		236	static void __start_cpu_write(struct drm_i915_gem_object *obj)
		237	{
		238	obj->base.read_domains = I915_GEM_DOMAIN_CPU;
		239	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
		240	if (cpu_write_needs_clflush(obj))
		241	obj->cache_dirty = true;
		242	}
		243
236	static void	244	static void
237	__i915_gem_object_release_shmem(struct drm_i915_gem_object *obj,	245	__i915_gem_object_release_shmem(struct drm_i915_gem_object *obj,
238	struct sg_table *pages,	246	struct sg_table *pages,
@@ -245,11 +253,10 @@ __i915_gem_object_release_shmem(struct drm_i915_gem_object *obj, · 6 unmodified lines
245		253
246	if (needs_clflush &&	254	if (needs_clflush &&
247	(obj->base.read_domains & I915_GEM_DOMAIN_CPU) == 0 &&	255	(obj->base.read_domains & I915_GEM_DOMAIN_CPU) == 0 &&
248	!i915_gem_object_is_coherent(obj))	256	!obj->cache_coherent)
249	drm_clflush_sg(pages);	257	drm_clflush_sg(pages);
250		258
251	obj->base.read_domains = I915_GEM_DOMAIN_CPU;	259	__start_cpu_write(obj);
252	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
253	}	260	}
254		261
255	static void	262	static void
@@ -684,6 +691,12 @@ i915_gem_dumb_create(struct drm_file *file, · 428 unmodified lines
684	args->size, &args->handle);	691	args->size, &args->handle);
685	}	692	}
686		693
		694	static bool gpu_write_needs_clflush(struct drm_i915_gem_object *obj)
		695	{
		696	return !(obj->cache_level == I915_CACHE_NONE \|\|
		697	obj->cache_level == I915_CACHE_WT);
		698	}
		699
687	/**	700	/**
688	* Creates a new mm object and returns a handle to it.	701	* Creates a new mm object and returns a handle to it.
689	* @dev: drm device pointer	702	* @dev: drm device pointer
@@ -753,6 +766,11 @@ flush_write_domain(struct drm_i915_gem_object *obj, unsigned int flush_domains) · 63 unmodified lines
753	case I915_GEM_DOMAIN_CPU:	766	case I915_GEM_DOMAIN_CPU:
754	i915_gem_clflush_object(obj, I915_CLFLUSH_SYNC);	767	i915_gem_clflush_object(obj, I915_CLFLUSH_SYNC);
755	break;	768	break;
		769
		770	case I915_GEM_DOMAIN_RENDER:
		771	if (gpu_write_needs_clflush(obj))
		772	obj->cache_dirty = true;
		773	break;
756	}	774	}
757		775
758	obj->base.write_domain = 0;	776	obj->base.write_domain = 0;
@@ -838,8 +856,7 @@ int i915_gem_obj_prepare_shmem_read(struct drm_i915_gem_object *obj, · 79 unmodified lines
838	if (ret)	856	if (ret)
839	return ret;	857	return ret;
840		858
841	if (i915_gem_object_is_coherent(obj) \|\|	859	if (obj->cache_coherent \|\| !static_cpu_has(X86_FEATURE_CLFLUSH)) {
842	!static_cpu_has(X86_FEATURE_CLFLUSH)) {
843	ret = i915_gem_object_set_to_cpu_domain(obj, false);	860	ret = i915_gem_object_set_to_cpu_domain(obj, false);
844	if (ret)	861	if (ret)
845	goto err_unpin;	862	goto err_unpin;
@@ -854,7 +871,8 @@ int i915_gem_obj_prepare_shmem_read(struct drm_i915_gem_object *obj, · 8 unmodified lines
854	* optimizes for the case when the gpu will dirty the data	871	* optimizes for the case when the gpu will dirty the data
855	* anyway again before the next pread happens.	872	* anyway again before the next pread happens.
856	*/	873	*/
857	if (!(obj->base.read_domains & I915_GEM_DOMAIN_CPU))	874	if (!obj->cache_dirty &&
		875	!(obj->base.read_domains & I915_GEM_DOMAIN_CPU))
858	*needs_clflush = CLFLUSH_BEFORE;	876	*needs_clflush = CLFLUSH_BEFORE;
859		877
860	out:	878	out:
@@ -890,8 +908,7 @@ int i915_gem_obj_prepare_shmem_write(struct drm_i915_gem_object *obj, · 29 unmodified lines
890	if (ret)	908	if (ret)
891	return ret;	909	return ret;
892		910
893	if (i915_gem_object_is_coherent(obj) \|\|	911	if (obj->cache_coherent \|\| !static_cpu_has(X86_FEATURE_CLFLUSH)) {
894	!static_cpu_has(X86_FEATURE_CLFLUSH)) {
895	ret = i915_gem_object_set_to_cpu_domain(obj, true);	912	ret = i915_gem_object_set_to_cpu_domain(obj, true);
896	if (ret)	913	if (ret)
897	goto err_unpin;	914	goto err_unpin;
@@ -906,14 +923,16 @@ int i915_gem_obj_prepare_shmem_write(struct drm_i915_gem_object *obj, · 8 unmodified lines
906	* This optimizes for the case when the gpu will use the data	923	* This optimizes for the case when the gpu will use the data
907	* right away and we therefore have to clflush anyway.	924	* right away and we therefore have to clflush anyway.
908	*/	925	*/
909	if (obj->base.write_domain != I915_GEM_DOMAIN_CPU)	926	if (!obj->cache_dirty) {
910	*needs_clflush \|= CLFLUSH_AFTER;	927	*needs_clflush \|= CLFLUSH_AFTER;
911		928
912	/* Same trick applies to invalidate partially written cachelines read	929	/*
913	* before writing.	930	* Same trick applies to invalidate partially written
914	*/	931	* cachelines read before writing.
915	if (!(obj->base.read_domains & I915_GEM_DOMAIN_CPU))	932	*/
916	*needs_clflush \|= CLFLUSH_BEFORE;	933	if (!(obj->base.read_domains & I915_GEM_DOMAIN_CPU))
		934	*needs_clflush \|= CLFLUSH_BEFORE;
		935	}
917		936
918	out:	937	out:
919	intel_fb_obj_invalidate(obj, ORIGIN_CPU);	938	intel_fb_obj_invalidate(obj, ORIGIN_CPU);
@@ -2337,8 +2356,8 @@ i915_gem_object_get_pages_gtt(struct drm_i915_gem_object *obj) · 1417 unmodified lines
2337	struct page *page;	2356	struct page *page;
2338	unsigned long last_pfn = 0; /* suppress gcc warning */	2357	unsigned long last_pfn = 0; /* suppress gcc warning */
2339	unsigned int max_segment;	2358	unsigned int max_segment;
		2359	gfp_t noreclaim;
2340	int ret;	2360	int ret;
2341	gfp_t gfp;
2342		2361
2343	/* Assert that the object is not currently in any GPU domain. As it	2362	/* Assert that the object is not currently in any GPU domain. As it
2344	* wasn't in the GTT, there shouldn't be any way it could have been in	2363	* wasn't in the GTT, there shouldn't be any way it could have been in
@@ -2367,22 +2386,30 @@ rebuild_st: · 22 unmodified lines
2367	* Fail silently without starting the shrinker	2386	* Fail silently without starting the shrinker
2368	*/	2387	*/
2369	mapping = obj->base.filp->f_mapping;	2388	mapping = obj->base.filp->f_mapping;
2370	gfp = mapping_gfp_constraint(mapping, ~(__GFP_IO \| __GFP_RECLAIM));	2389	noreclaim = mapping_gfp_constraint(mapping, ~__GFP_RECLAIM);
2371	gfp \|= __GFP_NORETRY \| __GFP_NOWARN;	2390	noreclaim \|= __GFP_NORETRY \| __GFP_NOWARN;
		2391
2372	sg = st->sgl;	2392	sg = st->sgl;
2373	st->nents = 0;	2393	st->nents = 0;
2374	for (i = 0; i < page_count; i++) {	2394	for (i = 0; i < page_count; i++) {
2375	page = shmem_read_mapping_page_gfp(mapping, i, gfp);	2395	const unsigned int shrink[] = {
2376	if (unlikely(IS_ERR(page))) {	2396	I915_SHRINK_BOUND \| I915_SHRINK_UNBOUND \| I915_SHRINK_PURGEABLE,
2377	i915_gem_shrink(dev_priv,	2397	0,
2378	page_count,	2398	}, *s = shrink;
2379	I915_SHRINK_BOUND \|	2399	gfp_t gfp = noreclaim;
2380	I915_SHRINK_UNBOUND \|	2400
2381	I915_SHRINK_PURGEABLE);	2401	do {
2382	page = shmem_read_mapping_page_gfp(mapping, i, gfp);	2402	page = shmem_read_mapping_page_gfp(mapping, i, gfp);
2383	}	2403	if (likely(!IS_ERR(page)))
2384	if (unlikely(IS_ERR(page))) {	2404	break;
2385	gfp_t reclaim;	2405
		2406	if (!*s) {
		2407	ret = PTR_ERR(page);
		2408	goto err_sg;
		2409	}
		2410
		2411	i915_gem_shrink(dev_priv, 2 * page_count, *s++);
		2412	cond_resched();
2386		2413
2387	/* We've tried hard to allocate the memory by reaping	2414	/* We've tried hard to allocate the memory by reaping
2388	* our own buffer, now let the real VM do its job and	2415	* our own buffer, now let the real VM do its job and
@@ -2392,15 +2419,26 @@ rebuild_st: · 3 unmodified lines
2392	* defer the oom here by reporting the ENOMEM back	2419	* defer the oom here by reporting the ENOMEM back
2393	* to userspace.	2420	* to userspace.
2394	*/	2421	*/
2395	reclaim = mapping_gfp_mask(mapping);	2422	if (!*s) {
2396	reclaim \|= __GFP_NORETRY; /* reclaim, but no oom */	2423	/* reclaim and warn, but no oom */
2397		2424	gfp = mapping_gfp_mask(mapping);
2398	page = shmem_read_mapping_page_gfp(mapping, i, reclaim);	2425
2399	if (IS_ERR(page)) {	2426	/* Our bo are always dirty and so we require
2400	ret = PTR_ERR(page);	2427	* kswapd to reclaim our pages (direct reclaim
2401	goto err_sg;	2428	* does not effectively begin pageout of our
		2429	* buffers on its own). However, direct reclaim
		2430	* only waits for kswapd when under allocation
		2431	* congestion. So as a result __GFP_RECLAIM is
		2432	* unreliable and fails to actually reclaim our
		2433	* dirty pages -- unless you try over and over
		2434	* again with !__GFP_NORETRY. However, we still
		2435	* want to fail this allocation rather than
		2436	* trigger the out-of-memory killer and for
		2437	* this we want the future __GFP_MAYFAIL.
		2438	*/
2402	}	2439	}
2403	}	2440	} while (1);
		2441
2404	if (!i \|\|	2442	if (!i \|\|
2405	sg->length >= max_segment \|\|	2443	sg->length >= max_segment \|\|
2406	page_to_pfn(page) != last_pfn + 1) {	2444	page_to_pfn(page) != last_pfn + 1) {
@@ -3223,6 +3261,10 @@ void i915_gem_close_object(struct drm_gem_object gem, struct drm_file file) · 816 unmodified lines
3223	if (vma->vm->file == fpriv)	3261	if (vma->vm->file == fpriv)
3224	i915_vma_close(vma);	3262	i915_vma_close(vma);
3225		3263
		3264	vma = obj->vma_hashed;
		3265	if (vma && vma->ctx->file_priv == fpriv)
		3266	i915_vma_unlink_ctx(vma);
		3267
3226	if (i915_gem_object_is_active(obj) &&	3268	if (i915_gem_object_is_active(obj) &&
3227	!i915_gem_object_has_active_reference(obj)) {	3269	!i915_gem_object_has_active_reference(obj)) {
3228	i915_gem_object_set_active_reference(obj);	3270	i915_gem_object_set_active_reference(obj);
@@ -3376,10 +3418,13 @@ int i915_gem_wait_for_idle(struct drm_i915_private *i915, unsigned int flags) · 147 unmodified lines
3376		3418
3377	static void __i915_gem_object_flush_for_display(struct drm_i915_gem_object *obj)	3419	static void __i915_gem_object_flush_for_display(struct drm_i915_gem_object *obj)
3378	{	3420	{
3379	if (obj->base.write_domain != I915_GEM_DOMAIN_CPU && !obj->cache_dirty)	3421	/*
3380	return;	3422	* We manually flush the CPU domain so that we can override and
3381		3423	* force the flush for the display, and perform it asyncrhonously.
3382	i915_gem_clflush_object(obj, I915_CLFLUSH_FORCE);	3424	*/
		3425	flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);
		3426	if (obj->cache_dirty)
		3427	i915_gem_clflush_object(obj, I915_CLFLUSH_FORCE);
3383	obj->base.write_domain = 0;	3428	obj->base.write_domain = 0;
3384	}	3429	}
3385		3430
@@ -3638,13 +3683,11 @@ restart: · 252 unmodified lines
3638	}	3683	}
3639	}	3684	}
3640		3685
3641	if (obj->base.write_domain == I915_GEM_DOMAIN_CPU &&
3642	i915_gem_object_is_coherent(obj))
3643	obj->cache_dirty = true;
3644
3645	list_for_each_entry(vma, &obj->vma_list, obj_link)	3686	list_for_each_entry(vma, &obj->vma_list, obj_link)
3646	vma->node.color = cache_level;	3687	vma->node.color = cache_level;
3647	obj->cache_level = cache_level;	3688	obj->cache_level = cache_level;
		3689	obj->cache_coherent = i915_gem_object_is_coherent(obj);
		3690	obj->cache_dirty = true; /* Always invalidate stale cachelines */
3648		3691
3649	return 0;	3692	return 0;
3650	}	3693	}
@@ -3866,9 +3909,6 @@ i915_gem_object_set_to_cpu_domain(struct drm_i915_gem_object *obj, bool write) · 215 unmodified lines
3866	if (ret)	3909	if (ret)
3867	return ret;	3910	return ret;
3868		3911
3869	if (obj->base.write_domain == I915_GEM_DOMAIN_CPU)
3870	return 0;
3871
3872	flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);	3912	flush_write_domain(obj, ~I915_GEM_DOMAIN_CPU);
3873		3913
3874	/* Flush the CPU cache if it's still invalid. */	3914	/* Flush the CPU cache if it's still invalid. */
@@ -3880,15 +3920,13 @@ i915_gem_object_set_to_cpu_domain(struct drm_i915_gem_object *obj, bool write) · 5 unmodified lines
3880	/* It should now be out of any other write domains, and we can update	3920	/* It should now be out of any other write domains, and we can update
3881	* the domain values for our changes.	3921	* the domain values for our changes.
3882	*/	3922	*/
3883	GEM_BUG_ON((obj->base.write_domain & ~I915_GEM_DOMAIN_CPU) != 0);	3923	GEM_BUG_ON(obj->base.write_domain & ~I915_GEM_DOMAIN_CPU);
3884		3924
3885	/* If we're writing through the CPU, then the GPU read domains will	3925	/* If we're writing through the CPU, then the GPU read domains will
3886	* need to be invalidated at next use.	3926	* need to be invalidated at next use.
3887	*/	3927	*/
3888	if (write) {	3928	if (write)
3889	obj->base.read_domains = I915_GEM_DOMAIN_CPU;	3929	__start_cpu_write(obj);
3890	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
3891	}
3892		3930
3893	return 0;	3931	return 0;
3894	}	3932	}
@@ -4220,7 +4258,6 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj, · 325 unmodified lines
4220		4258
4221	INIT_LIST_HEAD(&obj->global_link);	4259	INIT_LIST_HEAD(&obj->global_link);
4222	INIT_LIST_HEAD(&obj->userfault_link);	4260	INIT_LIST_HEAD(&obj->userfault_link);
4223	INIT_LIST_HEAD(&obj->obj_exec_link);
4224	INIT_LIST_HEAD(&obj->vma_list);	4261	INIT_LIST_HEAD(&obj->vma_list);
4225	INIT_LIST_HEAD(&obj->batch_pool_link);	4262	INIT_LIST_HEAD(&obj->batch_pool_link);
4226		4263
@@ -4285,6 +4322,7 @@ i915_gem_object_create(struct drm_i915_private *dev_priv, u64 size) · 58 unmodified lines
4285		4322
4286	mapping = obj->base.filp->f_mapping;	4323	mapping = obj->base.filp->f_mapping;
4287	mapping_set_gfp_mask(mapping, mask);	4324	mapping_set_gfp_mask(mapping, mask);
		4325	GEM_BUG_ON(!(mapping_gfp_mask(mapping) & __GFP_RECLAIM));
4288		4326
4289	i915_gem_object_init(obj, &i915_gem_object_ops);	4327	i915_gem_object_init(obj, &i915_gem_object_ops);
4290		4328
@@ -4308,6 +4346,9 @@ i915_gem_object_create(struct drm_i915_private *dev_priv, u64 size) · 17 unmodified lines
4308	} else	4346	} else
4309	obj->cache_level = I915_CACHE_NONE;	4347	obj->cache_level = I915_CACHE_NONE;
4310		4348
		4349	obj->cache_coherent = i915_gem_object_is_coherent(obj);
		4350	obj->cache_dirty = !obj->cache_coherent;
		4351
4311	trace_i915_gem_object_create(obj);	4352	trace_i915_gem_object_create(obj);
4312		4353
4313	return obj;	4354	return obj;
@@ -4356,7 +4397,6 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915, · 42 unmodified lines
4356	GEM_BUG_ON(i915_gem_object_is_active(obj));	4397	GEM_BUG_ON(i915_gem_object_is_active(obj));
4357	list_for_each_entry_safe(vma, vn,	4398	list_for_each_entry_safe(vma, vn,
4358	&obj->vma_list, obj_link) {	4399	&obj->vma_list, obj_link) {
4359	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
4360	GEM_BUG_ON(i915_vma_is_active(vma));	4400	GEM_BUG_ON(i915_vma_is_active(vma));
4361	vma->flags &= ~I915_VMA_PIN_MASK;	4401	vma->flags &= ~I915_VMA_PIN_MASK;
4362	i915_vma_close(vma);	4402	i915_vma_close(vma);
@@ -4763,7 +4803,9 @@ int i915_gem_init(struct drm_i915_private *dev_priv) · 400 unmodified lines
4763	*/	4803	*/
4764	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);	4804	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
4765		4805
4766	i915_gem_init_userptr(dev_priv);	4806	ret = i915_gem_init_userptr(dev_priv);
		4807	if (ret)
		4808	goto out_unlock;
4767		4809
4768	ret = i915_gem_init_ggtt(dev_priv);	4810	ret = i915_gem_init_ggtt(dev_priv);
4769	if (ret)	4811	if (ret)
@@ -4974,10 +5016,8 @@ int i915_gem_freeze_late(struct drm_i915_private *dev_priv) · 204 unmodified lines
4974		5016
4975	mutex_lock(&dev_priv->drm.struct_mutex);	5017	mutex_lock(&dev_priv->drm.struct_mutex);
4976	for (p = phases; *p; p++) {	5018	for (p = phases; *p; p++) {
4977	list_for_each_entry(obj, *p, global_link) {	5019	list_for_each_entry(obj, *p, global_link)
4978	obj->base.read_domains = I915_GEM_DOMAIN_CPU;	5020	__start_cpu_write(obj);
4979	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
4980	}
4981	}	5021	}
4982	mutex_unlock(&dev_priv->drm.struct_mutex);	5022	mutex_unlock(&dev_priv->drm.struct_mutex);
4983		5023

drivers/gpu/drm/i915/i915_gem_batch_pool.c+17−2

@@ -114,12 +114,27 @@ i915_gem_batch_pool_get(struct i915_gem_batch_pool *pool, · 113 unmodified lines
114	list_for_each_entry(obj, list, batch_pool_link) {	114	list_for_each_entry(obj, list, batch_pool_link) {
115	/* The batches are strictly LRU ordered */	115	/* The batches are strictly LRU ordered */
116	if (i915_gem_object_is_active(obj)) {	116	if (i915_gem_object_is_active(obj)) {
117	if (!reservation_object_test_signaled_rcu(obj->resv,	117	struct reservation_object *resv = obj->resv;
118	true))	118
		119	if (!reservation_object_test_signaled_rcu(resv, true))
119	break;	120	break;
120		121
121	i915_gem_retire_requests(pool->engine->i915);	122	i915_gem_retire_requests(pool->engine->i915);
122	GEM_BUG_ON(i915_gem_object_is_active(obj));	123	GEM_BUG_ON(i915_gem_object_is_active(obj));
		124
		125	/*
		126	* The object is now idle, clear the array of shared
		127	* fences before we add a new request. Although, we
		128	* remain on the same engine, we may be on a different
		129	* timeline and so may continually grow the array,
		130	* trapping a reference to all the old fences, rather
		131	* than replace the existing fence.
		132	*/
		133	if (rcu_access_pointer(resv->fence)) {
		134	reservation_object_lock(resv, NULL);
		135	reservation_object_add_excl_fence(resv, NULL);
		136	reservation_object_unlock(resv);
		137	}
123	}	138	}
124		139
125	GEM_BUG_ON(!reservation_object_test_signaled_rcu(obj->resv,	140	GEM_BUG_ON(!reservation_object_test_signaled_rcu(obj->resv,

drivers/gpu/drm/i915/i915_gem_clflush.c+8−9

@@ -71,8 +71,6 @@ static const struct dma_fence_ops i915_clflush_ops = { · 70 unmodified lines
71	static void __i915_do_clflush(struct drm_i915_gem_object *obj)	71	static void __i915_do_clflush(struct drm_i915_gem_object *obj)
72	{	72	{
73	drm_clflush_sg(obj->mm.pages);	73	drm_clflush_sg(obj->mm.pages);
74	obj->cache_dirty = false;
75
76	intel_fb_obj_flush(obj, ORIGIN_CPU);	74	intel_fb_obj_flush(obj, ORIGIN_CPU);
77	}	75	}
78		76
@@ -81,9 +79,6 @@ static void i915_clflush_work(struct work_struct *work) · 2 unmodified lines
81	struct clflush clflush = container_of(work, typeof(clflush), work);	79	struct clflush clflush = container_of(work, typeof(clflush), work);
82	struct drm_i915_gem_object *obj = clflush->obj;	80	struct drm_i915_gem_object *obj = clflush->obj;
83		81
84	if (!obj->cache_dirty)
85	goto out;
86
87	if (i915_gem_object_pin_pages(obj)) {	82	if (i915_gem_object_pin_pages(obj)) {
88	DRM_ERROR("Failed to acquire obj->pages for clflushing\n");	83	DRM_ERROR("Failed to acquire obj->pages for clflushing\n");
89	goto out;	84	goto out;
@@ -131,10 +126,10 @@ void i915_gem_clflush_object(struct drm_i915_gem_object *obj, · 41 unmodified lines
131	* anything not backed by physical memory we consider to be always	126	* anything not backed by physical memory we consider to be always
132	* coherent and not need clflushing.	127	* coherent and not need clflushing.
133	*/	128	*/
134	if (!i915_gem_object_has_struct_page(obj))	129	if (!i915_gem_object_has_struct_page(obj)) {
		130	obj->cache_dirty = false;
135	return;	131	return;
136		132	}
137	obj->cache_dirty = true;
138		133
139	/* If the GPU is snooping the contents of the CPU cache,	134	/* If the GPU is snooping the contents of the CPU cache,
140	* we do not need to manually clear the CPU cache lines. However,	135	* we do not need to manually clear the CPU cache lines. However,
@@ -144,7 +139,7 @@ void i915_gem_clflush_object(struct drm_i915_gem_object *obj, · 3 unmodified lines
144	* snooping behaviour occurs naturally as the result of our domain	139	* snooping behaviour occurs naturally as the result of our domain
145	* tracking.	140	* tracking.
146	*/	141	*/
147	if (!(flags & I915_CLFLUSH_FORCE) && i915_gem_object_is_coherent(obj))	142	if (!(flags & I915_CLFLUSH_FORCE) && obj->cache_coherent)
148	return;	143	return;
149		144
150	trace_i915_gem_object_clflush(obj);	145	trace_i915_gem_object_clflush(obj);
@@ -153,6 +148,8 @@ void i915_gem_clflush_object(struct drm_i915_gem_object *obj, · 2 unmodified lines
153	if (!(flags & I915_CLFLUSH_SYNC))	148	if (!(flags & I915_CLFLUSH_SYNC))
154	clflush = kmalloc(sizeof(*clflush), GFP_KERNEL);	149	clflush = kmalloc(sizeof(*clflush), GFP_KERNEL);
155	if (clflush) {	150	if (clflush) {
		151	GEM_BUG_ON(!obj->cache_dirty);
		152
156	dma_fence_init(&clflush->dma,	153	dma_fence_init(&clflush->dma,
157	&i915_clflush_ops,	154	&i915_clflush_ops,
158	&clflush_lock,	155	&clflush_lock,
@@ -180,4 +177,6 @@ void i915_gem_clflush_object(struct drm_i915_gem_object *obj, · 21 unmodified lines
180	} else {	177	} else {
181	GEM_BUG_ON(obj->base.write_domain != I915_GEM_DOMAIN_CPU);	178	GEM_BUG_ON(obj->base.write_domain != I915_GEM_DOMAIN_CPU);
182	}	179	}
		180
		181	obj->cache_dirty = false;
183	}	182	}

drivers/gpu/drm/i915/i915_gem_context.c+82−4

@@ -85,6 +85,7 @@ · 84 unmodified lines
85	*	85	*
86	*/	86	*/
87		87
		88	#include <linux/log2.h>
88	#include <drm/drmP.h>	89	#include <drm/drmP.h>
89	#include <drm/i915_drm.h>	90	#include <drm/i915_drm.h>
90	#include "i915_drv.h"	91	#include "i915_drv.h"
@@ -92,6 +93,71 @@ · 1 unmodified line
92		93
93	#define ALL_L3_SLICES(dev) (1 << NUM_L3_SLICES(dev)) - 1	94	#define ALL_L3_SLICES(dev) (1 << NUM_L3_SLICES(dev)) - 1
94		95
		96	/* Initial size (as log2) to preallocate the handle->object hashtable */
		97	#define VMA_HT_BITS 2u /* 4 x 2 pointers, 64 bytes minimum */
		98
		99	static void resize_vma_ht(struct work_struct *work)
		100	{
		101	struct i915_gem_context_vma_lut *lut =
		102	container_of(work, typeof(*lut), resize);
		103	unsigned int bits, new_bits, size, i;
		104	struct hlist_head *new_ht;
		105
		106	GEM_BUG_ON(!(lut->ht_size & I915_CTX_RESIZE_IN_PROGRESS));
		107
		108	bits = 1 + ilog2(4*lut->ht_count/3 + 1);
		109	new_bits = min_t(unsigned int,
		110	max(bits, VMA_HT_BITS),
		111	sizeof(unsigned int) * BITS_PER_BYTE - 1);
		112	if (new_bits == lut->ht_bits)
		113	goto out;
		114
		115	new_ht = kzalloc(sizeof(*new_ht)<<new_bits, GFP_KERNEL \| __GFP_NOWARN);
		116	if (!new_ht)
		117	new_ht = vzalloc(sizeof(*new_ht)<<new_bits);
		118	if (!new_ht)
		119	/* Pretend resize succeeded and stop calling us for a bit! */
		120	goto out;
		121
		122	size = BIT(lut->ht_bits);
		123	for (i = 0; i < size; i++) {
		124	struct i915_vma *vma;
		125	struct hlist_node *tmp;
		126
		127	hlist_for_each_entry_safe(vma, tmp, &lut->ht[i], ctx_node)
		128	hlist_add_head(&vma->ctx_node,
		129	&new_ht[hash_32(vma->ctx_handle,
		130	new_bits)]);
		131	}
		132	kvfree(lut->ht);
		133	lut->ht = new_ht;
		134	lut->ht_bits = new_bits;
		135	out:
		136	smp_store_release(&lut->ht_size, BIT(bits));
		137	GEM_BUG_ON(lut->ht_size & I915_CTX_RESIZE_IN_PROGRESS);
		138	}
		139
		140	static void vma_lut_free(struct i915_gem_context *ctx)
		141	{
		142	struct i915_gem_context_vma_lut *lut = &ctx->vma_lut;
		143	unsigned int i, size;
		144
		145	if (lut->ht_size & I915_CTX_RESIZE_IN_PROGRESS)
		146	cancel_work_sync(&lut->resize);
		147
		148	size = BIT(lut->ht_bits);
		149	for (i = 0; i < size; i++) {
		150	struct i915_vma *vma;
		151
		152	hlist_for_each_entry(vma, &lut->ht[i], ctx_node) {
		153	vma->obj->vma_hashed = NULL;
		154	vma->ctx = NULL;
		155	i915_vma_put(vma);
		156	}
		157	}
		158	kvfree(lut->ht);
		159	}
		160
95	void i915_gem_context_free(struct kref *ctx_ref)	161	void i915_gem_context_free(struct kref *ctx_ref)
96	{	162	{
97	struct i915_gem_context ctx = container_of(ctx_ref, typeof(ctx), ref);	163	struct i915_gem_context ctx = container_of(ctx_ref, typeof(ctx), ref);
@@ -101,6 +167,7 @@ void i915_gem_context_free(struct kref *ctx_ref) · 3 unmodified lines
101	trace_i915_context_free(ctx);	167	trace_i915_context_free(ctx);
102	GEM_BUG_ON(!i915_gem_context_is_closed(ctx));	168	GEM_BUG_ON(!i915_gem_context_is_closed(ctx));
103		169
		170	vma_lut_free(ctx);
104	i915_ppgtt_put(ctx->ppgtt);	171	i915_ppgtt_put(ctx->ppgtt);
105		172
106	for (i = 0; i < I915_NUM_ENGINES; i++) {	173	for (i = 0; i < I915_NUM_ENGINES; i++) {
@@ -118,6 +185,7 @@ void i915_gem_context_free(struct kref *ctx_ref) · 11 unmodified lines
118		185
119	kfree(ctx->name);	186	kfree(ctx->name);
120	put_pid(ctx->pid);	187	put_pid(ctx->pid);
		188
121	list_del(&ctx->link);	189	list_del(&ctx->link);
122		190
123	ida_simple_remove(&ctx->i915->context_hw_ida, ctx->hw_id);	191	ida_simple_remove(&ctx->i915->context_hw_ida, ctx->hw_id);
@@ -201,13 +269,24 @@ __create_hw_context(struct drm_i915_private *dev_priv, · 77 unmodified lines
201	ctx->i915 = dev_priv;	269	ctx->i915 = dev_priv;
202	ctx->priority = I915_PRIORITY_NORMAL;	270	ctx->priority = I915_PRIORITY_NORMAL;
203		271
		272	ctx->vma_lut.ht_bits = VMA_HT_BITS;
		273	ctx->vma_lut.ht_size = BIT(VMA_HT_BITS);
		274	BUILD_BUG_ON(BIT(VMA_HT_BITS) == I915_CTX_RESIZE_IN_PROGRESS);
		275	ctx->vma_lut.ht = kcalloc(ctx->vma_lut.ht_size,
		276	sizeof(*ctx->vma_lut.ht),
		277	GFP_KERNEL);
		278	if (!ctx->vma_lut.ht)
		279	goto err_out;
		280
		281	INIT_WORK(&ctx->vma_lut.resize, resize_vma_ht);
		282
204	/* Default context will never have a file_priv */	283	/* Default context will never have a file_priv */
205	ret = DEFAULT_CONTEXT_HANDLE;	284	ret = DEFAULT_CONTEXT_HANDLE;
206	if (file_priv) {	285	if (file_priv) {
207	ret = idr_alloc(&file_priv->context_idr, ctx,	286	ret = idr_alloc(&file_priv->context_idr, ctx,
208	DEFAULT_CONTEXT_HANDLE, 0, GFP_KERNEL);	287	DEFAULT_CONTEXT_HANDLE, 0, GFP_KERNEL);
209	if (ret < 0)	288	if (ret < 0)
210	goto err_out;	289	goto err_lut;
211	}	290	}
212	ctx->user_handle = ret;	291	ctx->user_handle = ret;
213		292
@@ -248,6 +327,8 @@ __create_hw_context(struct drm_i915_private *dev_priv, · 34 unmodified lines
248	err_pid:	327	err_pid:
249	put_pid(ctx->pid);	328	put_pid(ctx->pid);
250	idr_remove(&file_priv->context_idr, ctx->user_handle);	329	idr_remove(&file_priv->context_idr, ctx->user_handle);
		330	err_lut:
		331	kvfree(ctx->vma_lut.ht);
251	err_out:	332	err_out:
252	context_close(ctx);	333	context_close(ctx);
253	return ERR_PTR(ret);	334	return ERR_PTR(ret);
@@ -1034,9 +1115,6 @@ int i915_gem_context_reset_stats_ioctl(struct drm_device *dev, · 780 unmodified lines
1034	if (args->flags \|\| args->pad)	1115	if (args->flags \|\| args->pad)
1035	return -EINVAL;	1116	return -EINVAL;
1036		1117
1037	if (args->ctx_id == DEFAULT_CONTEXT_HANDLE && !capable(CAP_SYS_ADMIN))
1038	return -EPERM;
1039
1040	ret = i915_mutex_lock_interruptible(dev);	1118	ret = i915_mutex_lock_interruptible(dev);
1041	if (ret)	1119	if (ret)
1042	return ret;	1120	return ret;

drivers/gpu/drm/i915/i915_gem_context.h+26−0

@@ -143,6 +143,32 @@ struct i915_gem_context { · 142 unmodified lines
143	/** ggtt_offset_bias: placement restriction for context objects */	143	/** ggtt_offset_bias: placement restriction for context objects */
144	u32 ggtt_offset_bias;	144	u32 ggtt_offset_bias;
145		145
		146	struct i915_gem_context_vma_lut {
		147	/** ht_size: last request size to allocate the hashtable for. */
		148	unsigned int ht_size;
		149	#define I915_CTX_RESIZE_IN_PROGRESS BIT(0)
		150	/** ht_bits: real log2(size) of hashtable. */
		151	unsigned int ht_bits;
		152	/** ht_count: current number of entries inside the hashtable */
		153	unsigned int ht_count;
		154
		155	/** ht: the array of buckets comprising the simple hashtable */
		156	struct hlist_head *ht;
		157
		158	/**
		159	* resize: After an execbuf completes, we check the load factor
		160	* of the hashtable. If the hashtable is too full, or too empty,
		161	* we schedule a task to resize the hashtable. During the
		162	* resize, the entries are moved between different buckets and
		163	* so we cannot simultaneously read the hashtable as it is
		164	* being resized (unlike rhashtable). Therefore we treat the
		165	* active work as a strong barrier, pausing a subsequent
		166	* execbuf to wait for the resize worker to complete, if
		167	* required.
		168	*/
		169	struct work_struct resize;
		170	} vma_lut;
		171
146	/** engine: per-engine logical HW state */	172	/** engine: per-engine logical HW state */
147	struct intel_context {	173	struct intel_context {
148	struct i915_vma *state;	174	struct i915_vma *state;

drivers/gpu/drm/i915/i915_gem_evict.c+65−54

@@ -50,6 +50,29 @@ static bool ggtt_is_idle(struct drm_i915_private *dev_priv) · 49 unmodified lines
50	return true;	50	return true;
51	}	51	}
52		52
		53	static int ggtt_flush(struct drm_i915_private *i915)
		54	{
		55	int err;
		56
		57	/* Not everything in the GGTT is tracked via vma (otherwise we
		58	* could evict as required with minimal stalling) so we are forced
		59	* to idle the GPU and explicitly retire outstanding requests in
		60	* the hopes that we can then remove contexts and the like only
		61	* bound by their active reference.
		62	*/
		63	err = i915_gem_switch_to_kernel_context(i915);
		64	if (err)
		65	return err;
		66
		67	err = i915_gem_wait_for_idle(i915,
		68	I915_WAIT_INTERRUPTIBLE \|
		69	I915_WAIT_LOCKED);
		70	if (err)
		71	return err;
		72
		73	return 0;
		74	}
		75
53	static bool	76	static bool
54	mark_free(struct drm_mm_scan *scan,	77	mark_free(struct drm_mm_scan *scan,
55	struct i915_vma *vma,	78	struct i915_vma *vma,
@@ -59,13 +82,10 @@ mark_free(struct drm_mm_scan *scan, · 3 unmodified lines
59	if (i915_vma_is_pinned(vma))	82	if (i915_vma_is_pinned(vma))
60	return false;	83	return false;
61		84
62	if (WARN_ON(!list_empty(&vma->exec_list)))
63	return false;
64
65	if (flags & PIN_NONFAULT && !list_empty(&vma->obj->userfault_link))	85	if (flags & PIN_NONFAULT && !list_empty(&vma->obj->userfault_link))
66	return false;	86	return false;
67		87
68	list_add(&vma->exec_list, unwind);	88	list_add(&vma->evict_link, unwind);
69	return drm_mm_scan_add_block(scan, &vma->node);	89	return drm_mm_scan_add_block(scan, &vma->node);
70	}	90	}
71		91
@@ -157,11 +177,9 @@ search_again: · 85 unmodified lines
157	} while (*++phase);	177	} while (*++phase);
158		178
159	/* Nothing found, clean up and bail out! */	179	/* Nothing found, clean up and bail out! */
160	list_for_each_entry_safe(vma, next, &eviction_list, exec_list) {	180	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
161	ret = drm_mm_scan_remove_block(&scan, &vma->node);	181	ret = drm_mm_scan_remove_block(&scan, &vma->node);
162	BUG_ON(ret);	182	BUG_ON(ret);
163
164	INIT_LIST_HEAD(&vma->exec_list);
165	}	183	}
166		184
167	/* Can we unpin some objects such as idle hw contents,	185	/* Can we unpin some objects such as idle hw contents,
@@ -180,19 +198,7 @@ search_again: · 12 unmodified lines
180	return intel_has_pending_fb_unpin(dev_priv) ? -EAGAIN : -ENOSPC;	198	return intel_has_pending_fb_unpin(dev_priv) ? -EAGAIN : -ENOSPC;
181	}	199	}
182		200
183	/* Not everything in the GGTT is tracked via vma (otherwise we	201	ret = ggtt_flush(dev_priv);
184	* could evict as required with minimal stalling) so we are forced
185	* to idle the GPU and explicitly retire outstanding requests in
186	* the hopes that we can then remove contexts and the like only
187	* bound by their active reference.
188	*/
189	ret = i915_gem_switch_to_kernel_context(dev_priv);
190	if (ret)
191	return ret;
192
193	ret = i915_gem_wait_for_idle(dev_priv,
194	I915_WAIT_INTERRUPTIBLE \|
195	I915_WAIT_LOCKED);
196	if (ret)	202	if (ret)
197	return ret;	203	return ret;
198		204
@@ -205,21 +211,16 @@ found: · 6 unmodified lines
205	* calling unbind (which may remove the active reference	211	* calling unbind (which may remove the active reference
206	* of any of our objects, thus corrupting the list).	212	* of any of our objects, thus corrupting the list).
207	*/	213	*/
208	list_for_each_entry_safe(vma, next, &eviction_list, exec_list) {	214	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
209	if (drm_mm_scan_remove_block(&scan, &vma->node))	215	if (drm_mm_scan_remove_block(&scan, &vma->node))
210	__i915_vma_pin(vma);	216	__i915_vma_pin(vma);
211	else	217	else
212	list_del_init(&vma->exec_list);	218	list_del(&vma->evict_link);
213	}	219	}
214		220
215	/* Unbinding will emit any required flushes */	221	/* Unbinding will emit any required flushes */
216	ret = 0;	222	ret = 0;
217	while (!list_empty(&eviction_list)) {	223	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
218	vma = list_first_entry(&eviction_list,
219	struct i915_vma,
220	exec_list);
221
222	list_del_init(&vma->exec_list);
223	__i915_vma_unpin(vma);	224	__i915_vma_unpin(vma);
224	if (ret == 0)	225	if (ret == 0)
225	ret = i915_vma_unbind(vma);	226	ret = i915_vma_unbind(vma);
@@ -315,7 +316,7 @@ int i915_gem_evict_for_node(struct i915_address_space *vm, · 89 unmodified lines
315	}	316	}
316		317
317	/* Overlap of objects in the same batch? */	318	/* Overlap of objects in the same batch? */
318	if (i915_vma_is_pinned(vma) \|\| !list_empty(&vma->exec_list)) {	319	if (i915_vma_is_pinned(vma)) {
319	ret = -ENOSPC;	320	ret = -ENOSPC;
320	if (vma->exec_entry &&	321	if (vma->exec_entry &&
321	vma->exec_entry->flags & EXEC_OBJECT_PINNED)	322	vma->exec_entry->flags & EXEC_OBJECT_PINNED)
@@ -332,11 +333,10 @@ int i915_gem_evict_for_node(struct i915_address_space *vm, · 10 unmodified lines
332	* reference) another in our eviction list.	333	* reference) another in our eviction list.
333	*/	334	*/
334	__i915_vma_pin(vma);	335	__i915_vma_pin(vma);
335	list_add(&vma->exec_list, &eviction_list);	336	list_add(&vma->evict_link, &eviction_list);
336	}	337	}
337		338
338	list_for_each_entry_safe(vma, next, &eviction_list, exec_list) {	339	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
339	list_del_init(&vma->exec_list);
340	__i915_vma_unpin(vma);	340	__i915_vma_unpin(vma);
341	if (ret == 0)	341	if (ret == 0)
342	ret = i915_vma_unbind(vma);	342	ret = i915_vma_unbind(vma);
@@ -348,10 +348,8 @@ int i915_gem_evict_for_node(struct i915_address_space *vm, · 5 unmodified lines
348	/**	348	/**
349	* i915_gem_evict_vm - Evict all idle vmas from a vm	349	* i915_gem_evict_vm - Evict all idle vmas from a vm
350	* @vm: Address space to cleanse	350	* @vm: Address space to cleanse
351	* @do_idle: Boolean directing whether to idle first.
352	*	351	*
353	* This function evicts all idles vmas from a vm. If all unpinned vmas should be	352	* This function evicts all vmas from a vm.
354	* evicted the @do_idle needs to be set to true.
355	*	353	*
356	* This is used by the execbuf code as a last-ditch effort to defragment the	354	* This is used by the execbuf code as a last-ditch effort to defragment the
357	* address space.	355	* address space.
@@ -359,37 +357,50 @@ int i915_gem_evict_for_node(struct i915_address_space *vm, · 1 unmodified line
359	* To clarify: This is for freeing up virtual address space, not for freeing	357	* To clarify: This is for freeing up virtual address space, not for freeing
360	* memory in e.g. the shrinker.	358	* memory in e.g. the shrinker.
361	*/	359	*/
362	int i915_gem_evict_vm(struct i915_address_space *vm, bool do_idle)	360	int i915_gem_evict_vm(struct i915_address_space *vm)
363	{	361	{
		362	struct list_head *phases[] = {
		363	&vm->inactive_list,
		364	&vm->active_list,
		365	NULL
		366	}, **phase;
		367	struct list_head eviction_list;
364	struct i915_vma vma, next;	368	struct i915_vma vma, next;
365	int ret;	369	int ret;
366		370
367	lockdep_assert_held(&vm->i915->drm.struct_mutex);	371	lockdep_assert_held(&vm->i915->drm.struct_mutex);
368	trace_i915_gem_evict_vm(vm);	372	trace_i915_gem_evict_vm(vm);
369		373
370	if (do_idle) {	374	/* Switch back to the default context in order to unpin
371	struct drm_i915_private *dev_priv = vm->i915;	375	* the existing context objects. However, such objects only
372		376	* pin themselves inside the global GTT and performing the
373	if (i915_is_ggtt(vm)) {	377	* switch otherwise is ineffective.
374	ret = i915_gem_switch_to_kernel_context(dev_priv);	378	*/
375	if (ret)	379	if (i915_is_ggtt(vm)) {
376	return ret;	380	ret = ggtt_flush(vm->i915);
377	}
378
379	ret = i915_gem_wait_for_idle(dev_priv,
380	I915_WAIT_INTERRUPTIBLE \|
381	I915_WAIT_LOCKED);
382	if (ret)	381	if (ret)
383	return ret;	382	return ret;
384
385	WARN_ON(!list_empty(&vm->active_list));
386	}	383	}
387		384
388	list_for_each_entry_safe(vma, next, &vm->inactive_list, vm_link)	385	INIT_LIST_HEAD(&eviction_list);
389	if (!i915_vma_is_pinned(vma))	386	phase = phases;
390	WARN_ON(i915_vma_unbind(vma));	387	do {
		388	list_for_each_entry(vma, *phase, vm_link) {
		389	if (i915_vma_is_pinned(vma))
		390	continue;
391		391
392	return 0;	392	__i915_vma_pin(vma);
		393	list_add(&vma->evict_link, &eviction_list);
		394	}
		395	} while (*++phase);
		396
		397	ret = 0;
		398	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
		399	__i915_vma_unpin(vma);
		400	if (ret == 0)
		401	ret = i915_vma_unbind(vma);
		402	}
		403	return ret;
393	}	404	}
394		405
395	#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)	406	#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)

drivers/gpu/drm/i915/i915_gem_execbuffer.c+1631−1123

@@ -40,143 +40,726 @@ · 39 unmodified lines
40	#include "intel_drv.h"	40	#include "intel_drv.h"
41	#include "intel_frontbuffer.h"	41	#include "intel_frontbuffer.h"
42		42
43	#define DBG_USE_CPU_RELOC 0 /* -1 force GTT relocs; 1 force CPU relocs */	43	enum {
		44	FORCE_CPU_RELOC = 1,
		45	FORCE_GTT_RELOC,
		46	FORCE_GPU_RELOC,
		47	#define DBG_FORCE_RELOC 0 /* choose one of the above! */
		48	};
		49
		50	#define __EXEC_OBJECT_HAS_REF BIT(31)
		51	#define __EXEC_OBJECT_HAS_PIN BIT(30)
		52	#define __EXEC_OBJECT_HAS_FENCE BIT(29)
		53	#define __EXEC_OBJECT_NEEDS_MAP BIT(28)
		54	#define __EXEC_OBJECT_NEEDS_BIAS BIT(27)
		55	#define __EXEC_OBJECT_INTERNAL_FLAGS (~0u << 27) /* all of the above */
		56	#define __EXEC_OBJECT_RESERVED (__EXEC_OBJECT_HAS_PIN \| __EXEC_OBJECT_HAS_FENCE)
44		57
45	#define __EXEC_OBJECT_HAS_PIN (1<<31)	58	#define __EXEC_HAS_RELOC BIT(31)
46	#define __EXEC_OBJECT_HAS_FENCE (1<<30)	59	#define __EXEC_VALIDATED BIT(30)
47	#define __EXEC_OBJECT_NEEDS_MAP (1<<29)	60	#define UPDATE PIN_OFFSET_FIXED
48	#define __EXEC_OBJECT_NEEDS_BIAS (1<<28)
49	#define __EXEC_OBJECT_INTERNAL_FLAGS (0xf<<28) /* all of the above */
50		61
51	#define BATCH_OFFSET_BIAS (256*1024)	62	#define BATCH_OFFSET_BIAS (256*1024)
52		63
53	struct i915_execbuffer_params {	64	#define __I915_EXEC_ILLEGAL_FLAGS \
54	struct drm_device *dev;	65	(__I915_EXEC_UNKNOWN_FLAGS \| I915_EXEC_CONSTANTS_MASK)
55	struct drm_file *file;	66
56	struct i915_vma *batch;	67	/**
57	u32 dispatch_flags;	68	* DOC: User command execution
58	u32 args_batch_start_offset;	69	*
59	struct intel_engine_cs *engine;	70	* Userspace submits commands to be executed on the GPU as an instruction
60	struct i915_gem_context *ctx;	71	* stream within a GEM object we call a batchbuffer. This instructions may
61	struct drm_i915_gem_request *request;	72	* refer to other GEM objects containing auxiliary state such as kernels,
62	};	73	* samplers, render targets and even secondary batchbuffers. Userspace does
		74	* not know where in the GPU memory these objects reside and so before the
		75	* batchbuffer is passed to the GPU for execution, those addresses in the
		76	* batchbuffer and auxiliary objects are updated. This is known as relocation,
		77	* or patching. To try and avoid having to relocate each object on the next
		78	* execution, userspace is told the location of those objects in this pass,
		79	* but this remains just a hint as the kernel may choose a new location for
		80	* any object in the future.
		81	*
		82	* Processing an execbuf ioctl is conceptually split up into a few phases.
		83	*
		84	* 1. Validation - Ensure all the pointers, handles and flags are valid.
		85	* 2. Reservation - Assign GPU address space for every object
		86	* 3. Relocation - Update any addresses to point to the final locations
		87	* 4. Serialisation - Order the request with respect to its dependencies
		88	* 5. Construction - Construct a request to execute the batchbuffer
		89	* 6. Submission (at some point in the future execution)
		90	*
		91	* Reserving resources for the execbuf is the most complicated phase. We
		92	* neither want to have to migrate the object in the address space, nor do
		93	* we want to have to update any relocations pointing to this object. Ideally,
		94	* we want to leave the object where it is and for all the existing relocations
		95	* to match. If the object is given a new address, or if userspace thinks the
		96	* object is elsewhere, we have to parse all the relocation entries and update
		97	* the addresses. Userspace can set the I915_EXEC_NORELOC flag to hint that
		98	* all the target addresses in all of its objects match the value in the
		99	* relocation entries and that they all match the presumed offsets given by the
		100	* list of execbuffer objects. Using this knowledge, we know that if we haven't
		101	* moved any buffers, all the relocation entries are valid and we can skip
		102	* the update. (If userspace is wrong, the likely outcome is an impromptu GPU
		103	* hang.) The requirement for using I915_EXEC_NO_RELOC are:
		104	*
		105	* The addresses written in the objects must match the corresponding
		106	* reloc.presumed_offset which in turn must match the corresponding
		107	* execobject.offset.
		108	*
		109	* Any render targets written to in the batch must be flagged with
		110	* EXEC_OBJECT_WRITE.
		111	*
		112	* To avoid stalling, execobject.offset should match the current
		113	* address of that object within the active context.
		114	*
		115	* The reservation is done is multiple phases. First we try and keep any
		116	* object already bound in its current location - so as long as meets the
		117	* constraints imposed by the new execbuffer. Any object left unbound after the
		118	* first pass is then fitted into any available idle space. If an object does
		119	* not fit, all objects are removed from the reservation and the process rerun
		120	* after sorting the objects into a priority order (more difficult to fit
		121	* objects are tried first). Failing that, the entire VM is cleared and we try
		122	* to fit the execbuf once last time before concluding that it simply will not
		123	* fit.
		124	*
		125	* A small complication to all of this is that we allow userspace not only to
		126	* specify an alignment and a size for the object in the address space, but
		127	* we also allow userspace to specify the exact offset. This objects are
		128	* simpler to place (the location is known a priori) all we have to do is make
		129	* sure the space is available.
		130	*
		131	* Once all the objects are in place, patching up the buried pointers to point
		132	* to the final locations is a fairly simple job of walking over the relocation
		133	* entry arrays, looking up the right address and rewriting the value into
		134	* the object. Simple! ... The relocation entries are stored in user memory
		135	* and so to access them we have to copy them into a local buffer. That copy
		136	* has to avoid taking any pagefaults as they may lead back to a GEM object
		137	* requiring the struct_mutex (i.e. recursive deadlock). So once again we split
		138	* the relocation into multiple passes. First we try to do everything within an
		139	* atomic context (avoid the pagefaults) which requires that we never wait. If
		140	* we detect that we may wait, or if we need to fault, then we have to fallback
		141	* to a slower path. The slowpath has to drop the mutex. (Can you hear alarm
		142	* bells yet?) Dropping the mutex means that we lose all the state we have
		143	* built up so far for the execbuf and we must reset any global data. However,
		144	* we do leave the objects pinned in their final locations - which is a
		145	* potential issue for concurrent execbufs. Once we have left the mutex, we can
		146	* allocate and copy all the relocation entries into a large array at our
		147	* leisure, reacquire the mutex, reclaim all the objects and other state and
		148	* then proceed to update any incorrect addresses with the objects.
		149	*
		150	* As we process the relocation entries, we maintain a record of whether the
		151	* object is being written to. Using NORELOC, we expect userspace to provide
		152	* this information instead. We also check whether we can skip the relocation
		153	* by comparing the expected value inside the relocation entry with the target's
		154	* final address. If they differ, we have to map the current object and rewrite
		155	* the 4 or 8 byte pointer within.
		156	*
		157	* Serialising an execbuf is quite simple according to the rules of the GEM
		158	* ABI. Execution within each context is ordered by the order of submission.
		159	* Writes to any GEM object are in order of submission and are exclusive. Reads
		160	* from a GEM object are unordered with respect to other reads, but ordered by
		161	* writes. A write submitted after a read cannot occur before the read, and
		162	* similarly any read submitted after a write cannot occur before the write.
		163	* Writes are ordered between engines such that only one write occurs at any
		164	* time (completing any reads beforehand) - using semaphores where available
		165	* and CPU serialisation otherwise. Other GEM access obey the same rules, any
		166	* write (either via mmaps using set-domain, or via pwrite) must flush all GPU
		167	* reads before starting, and any read (either using set-domain or pread) must
		168	* flush all GPU writes before starting. (Note we only employ a barrier before,
		169	* we currently rely on userspace not concurrently starting a new execution
		170	* whilst reading or writing to an object. This may be an advantage or not
		171	* depending on how much you trust userspace not to shoot themselves in the
		172	* foot.) Serialisation may just result in the request being inserted into
		173	* a DAG awaiting its turn, but most simple is to wait on the CPU until
		174	* all dependencies are resolved.
		175	*
		176	* After all of that, is just a matter of closing the request and handing it to
		177	* the hardware (well, leaving it in a queue to be executed). However, we also
		178	* offer the ability for batchbuffers to be run with elevated privileges so
		179	* that they access otherwise hidden registers. (Used to adjust L3 cache etc.)
		180	* Before any batch is given extra privileges we first must check that it
		181	* contains no nefarious instructions, we check that each instruction is from
		182	* our whitelist and all registers are also from an allowed list. We first
		183	* copy the user's batchbuffer to a shadow (so that the user doesn't have
		184	* access to it, either by the CPU or GPU as we scan it) and then parse each
		185	* instruction. If everything is ok, we set a flag telling the hardware to run
		186	* the batchbuffer in trusted mode, otherwise the ioctl is rejected.
		187	*/
		188
		189	struct i915_execbuffer {
		190	struct drm_i915_private i915; /* i915 backpointer */
		191	struct drm_file file; /* per-file lookup tables and limits */
		192	struct drm_i915_gem_execbuffer2 args; /* ioctl parameters */
		193	struct drm_i915_gem_exec_object2 exec; /* ioctl execobj[] */
		194
		195	struct intel_engine_cs engine; /* engine to queue the request to */
		196	struct i915_gem_context ctx; /* context for building the request */
		197	struct i915_address_space vm; /* GTT and vma for the request */
		198
		199	struct drm_i915_gem_request request; /* our request to build */
		200	struct i915_vma batch; /* identity of the batch obj/vma */
		201
		202	/** actual size of execobj[] as we may extend it for the cmdparser */
		203	unsigned int buffer_count;
63		204
64	struct eb_vmas {	205	/** list of vma not yet bound during reservation phase */
65	struct drm_i915_private *i915;	206	struct list_head unbound;
66	struct list_head vmas;	207
67	int and;	208	/** list of vma that have execobj.relocation_count */
68	union {	209	struct list_head relocs;
69	struct i915_vma *lut[0];	210
70	struct hlist_head buckets[0];	211	/**
71	};	212	* Track the most recently used object for relocations, as we
		213	* frequently have to perform multiple relocations within the same
		214	* obj/page
		215	*/
		216	struct reloc_cache {
		217	struct drm_mm_node node; /** temporary GTT binding */
		218	unsigned long vaddr; /** Current kmap address */
		219	unsigned long page; /** Currently mapped page index */
		220	unsigned int gen; /** Cached value of INTEL_GEN */
		221	bool use_64bit_reloc : 1;
		222	bool has_llc : 1;
		223	bool has_fence : 1;
		224	bool needs_unfenced : 1;
		225
		226	struct drm_i915_gem_request *rq;
		227	u32 *rq_cmd;
		228	unsigned int rq_size;
		229	} reloc_cache;
		230
		231	u64 invalid_flags; /** Set of execobj.flags that are invalid */
		232	u32 context_flags; /** Set of execobj.flags to insert from the ctx */
		233
		234	u32 batch_start_offset; /** Location within object of batch */
		235	u32 batch_len; /** Length of batch within object */
		236	u32 batch_flags; /** Flags composed for emit_bb_start() */
		237
		238	/**
		239	* Indicate either the size of the hastable used to resolve
		240	* relocation handles, or if negative that we are using a direct
		241	* index into the execobj[].
		242	*/
		243	int lut_size;
		244	struct hlist_head buckets; /* ht for relocation handles */
72	};	245	};
73		246
74	static struct eb_vmas *	247	/*
75	eb_create(struct drm_i915_private *i915,	248	* As an alternative to creating a hashtable of handle-to-vma for a batch,
76	struct drm_i915_gem_execbuffer2 *args)	249	* we used the last available reserved field in the execobject[] and stash
		250	* a link from the execobj to its vma.
		251	*/
		252	#define __exec_to_vma(ee) (ee)->rsvd2
		253	#define exec_to_vma(ee) u64_to_ptr(struct i915_vma, __exec_to_vma(ee))
		254
		255	/*
		256	* Used to convert any address to canonical form.
		257	* Starting from gen8, some commands (e.g. STATE_BASE_ADDRESS,
		258	* MI_LOAD_REGISTER_MEM and others, see Broadwell PRM Vol2a) require the
		259	* addresses to be in a canonical form:
		260	* "GraphicsAddress[63:48] are ignored by the HW and assumed to be in correct
		261	* canonical form [63:48] == [47]."
		262	*/
		263	#define GEN8_HIGH_ADDRESS_BIT 47
		264	static inline u64 gen8_canonical_addr(u64 address)
77	{	265	{
78	struct eb_vmas *eb = NULL;	266	return sign_extend64(address, GEN8_HIGH_ADDRESS_BIT);
79		267	}
80	if (args->flags & I915_EXEC_HANDLE_LUT) {
81	unsigned size = args->buffer_count;
82	size = sizeof(struct i915_vma );
83	size += sizeof(struct eb_vmas);
84	eb = kmalloc(size, GFP_TEMPORARY \| __GFP_NOWARN \| __GFP_NORETRY);
85	}
86
87	if (eb == NULL) {
88	unsigned size = args->buffer_count;
89	unsigned count = PAGE_SIZE / sizeof(struct hlist_head) / 2;
90	BUILD_BUG_ON_NOT_POWER_OF_2(PAGE_SIZE / sizeof(struct hlist_head));
91	while (count > 2*size)
92	count >>= 1;
93	eb = kzalloc(count*sizeof(struct hlist_head) +
94	sizeof(struct eb_vmas),
95	GFP_TEMPORARY);
96	if (eb == NULL)
97	return eb;
98
99	eb->and = count - 1;
100	} else
101	eb->and = -args->buffer_count;
102		268
103	eb->i915 = i915;	269	static inline u64 gen8_noncanonical_addr(u64 address)
104	INIT_LIST_HEAD(&eb->vmas);	270	{
105	return eb;	271	return address & GENMASK_ULL(GEN8_HIGH_ADDRESS_BIT, 0);
106	}	272	}
107		273
108	static void	274	static int eb_create(struct i915_execbuffer *eb)
109	eb_reset(struct eb_vmas *eb)
110	{	275	{
111	if (eb->and >= 0)	276	if (!(eb->args->flags & I915_EXEC_HANDLE_LUT)) {
112	memset(eb->buckets, 0, (eb->and+1)*sizeof(struct hlist_head));	277	unsigned int size = 1 + ilog2(eb->buffer_count);
		278
		279	/*
		280	* Without a 1:1 association between relocation handles and
		281	* the execobject[] index, we instead create a hashtable.
		282	* We size it dynamically based on available memory, starting
		283	* first with 1:1 assocative hash and scaling back until
		284	* the allocation succeeds.
		285	*
		286	* Later on we use a positive lut_size to indicate we are
		287	* using this hashtable, and a negative value to indicate a
		288	* direct lookup.
		289	*/
		290	do {
		291	eb->buckets = kzalloc(sizeof(struct hlist_head) << size,
		292	GFP_TEMPORARY \|
		293	__GFP_NORETRY \|
		294	__GFP_NOWARN);
		295	if (eb->buckets)
		296	break;
		297	} while (--size);
		298
		299	if (unlikely(!eb->buckets)) {
		300	eb->buckets = kzalloc(sizeof(struct hlist_head),
		301	GFP_TEMPORARY);
		302	if (unlikely(!eb->buckets))
		303	return -ENOMEM;
		304	}
		305
		306	eb->lut_size = size;
		307	} else {
		308	eb->lut_size = -eb->buffer_count;
		309	}
		310
		311	return 0;
113	}	312	}
114		313
115	static struct i915_vma *	314	static bool
116	eb_get_batch(struct eb_vmas *eb)	315	eb_vma_misplaced(const struct drm_i915_gem_exec_object2 *entry,
		316	const struct i915_vma *vma)
		317	{
		318	if (!(entry->flags & __EXEC_OBJECT_HAS_PIN))
		319	return true;
		320
		321	if (vma->node.size < entry->pad_to_size)
		322	return true;
		323
		324	if (entry->alignment && !IS_ALIGNED(vma->node.start, entry->alignment))
		325	return true;
		326
		327	if (entry->flags & EXEC_OBJECT_PINNED &&
		328	vma->node.start != entry->offset)
		329	return true;
		330
		331	if (entry->flags & __EXEC_OBJECT_NEEDS_BIAS &&
		332	vma->node.start < BATCH_OFFSET_BIAS)
		333	return true;
		334
		335	if (!(entry->flags & EXEC_OBJECT_SUPPORTS_48B_ADDRESS) &&
		336	(vma->node.start + vma->node.size - 1) >> 32)
		337	return true;
		338
		339	return false;
		340	}
		341
		342	static inline void
		343	eb_pin_vma(struct i915_execbuffer *eb,
		344	struct drm_i915_gem_exec_object2 *entry,
		345	struct i915_vma *vma)
		346	{
		347	u64 flags;
		348
		349	if (vma->node.size)
		350	flags = vma->node.start;
		351	else
		352	flags = entry->offset & PIN_OFFSET_MASK;
		353
		354	flags \|= PIN_USER \| PIN_NOEVICT \| PIN_OFFSET_FIXED;
		355	if (unlikely(entry->flags & EXEC_OBJECT_NEEDS_GTT))
		356	flags \|= PIN_GLOBAL;
		357
		358	if (unlikely(i915_vma_pin(vma, 0, 0, flags)))
		359	return;
		360
		361	if (unlikely(entry->flags & EXEC_OBJECT_NEEDS_FENCE)) {
		362	if (unlikely(i915_vma_get_fence(vma))) {
		363	i915_vma_unpin(vma);
		364	return;
		365	}
		366
		367	if (i915_vma_pin_fence(vma))
		368	entry->flags \|= __EXEC_OBJECT_HAS_FENCE;
		369	}
		370
		371	entry->flags \|= __EXEC_OBJECT_HAS_PIN;
		372	}
		373
		374	static inline void
		375	__eb_unreserve_vma(struct i915_vma *vma,
		376	const struct drm_i915_gem_exec_object2 *entry)
		377	{
		378	GEM_BUG_ON(!(entry->flags & __EXEC_OBJECT_HAS_PIN));
		379
		380	if (unlikely(entry->flags & __EXEC_OBJECT_HAS_FENCE))
		381	i915_vma_unpin_fence(vma);
		382
		383	__i915_vma_unpin(vma);
		384	}
		385
		386	static inline void
		387	eb_unreserve_vma(struct i915_vma *vma,
		388	struct drm_i915_gem_exec_object2 *entry)
		389	{
		390	if (!(entry->flags & __EXEC_OBJECT_HAS_PIN))
		391	return;
		392
		393	__eb_unreserve_vma(vma, entry);
		394	entry->flags &= ~__EXEC_OBJECT_RESERVED;
		395	}
		396
		397	static int
		398	eb_validate_vma(struct i915_execbuffer *eb,
		399	struct drm_i915_gem_exec_object2 *entry,
		400	struct i915_vma *vma)
117	{	401	{
118	struct i915_vma vma = list_entry(eb->vmas.prev, typeof(vma), exec_list);	402	if (unlikely(entry->flags & eb->invalid_flags))
		403	return -EINVAL;
		404
		405	if (unlikely(entry->alignment && !is_power_of_2(entry->alignment)))
		406	return -EINVAL;
119		407
120	/*	408	/*
121	* SNA is doing fancy tricks with compressing batch buffers, which leads	409	* Offset can be used as input (EXEC_OBJECT_PINNED), reject
122	* to negative relocation deltas. Usually that works out ok since the	410	* any non-page-aligned or non-canonical addresses.
123	* relocate address is still positive, except when the batch is placed
124	* very low in the GTT. Ensure this doesn't happen.
125	*
126	* Note that actual hangs have only been observed on gen7, but for
127	* paranoia do it everywhere.
128	*/	411	*/
129	if ((vma->exec_entry->flags & EXEC_OBJECT_PINNED) == 0)	412	if (unlikely(entry->flags & EXEC_OBJECT_PINNED &&
130	vma->exec_entry->flags \|= __EXEC_OBJECT_NEEDS_BIAS;	413	entry->offset != gen8_canonical_addr(entry->offset & PAGE_MASK)))
		414	return -EINVAL;
131		415
132	return vma;	416	/* pad_to_size was once a reserved field, so sanitize it */
		417	if (entry->flags & EXEC_OBJECT_PAD_TO_SIZE) {
		418	if (unlikely(offset_in_page(entry->pad_to_size)))
		419	return -EINVAL;
		420	} else {
		421	entry->pad_to_size = 0;
		422	}
		423
		424	if (unlikely(vma->exec_entry)) {
		425	DRM_DEBUG("Object [handle %d, index %d] appears more than once in object list\n",
		426	entry->handle, (int)(entry - eb->exec));
		427	return -EINVAL;
		428	}
		429
		430	/*
		431	* From drm_mm perspective address space is continuous,
		432	* so from this point we're always using non-canonical
		433	* form internally.
		434	*/
		435	entry->offset = gen8_noncanonical_addr(entry->offset);
		436
		437	return 0;
133	}	438	}
134		439
135	static int	440	static int
136	eb_lookup_vmas(struct eb_vmas *eb,	441	eb_add_vma(struct i915_execbuffer *eb,
137	struct drm_i915_gem_exec_object2 *exec,	442	struct drm_i915_gem_exec_object2 *entry,
138	const struct drm_i915_gem_execbuffer2 *args,	443	struct i915_vma *vma)
139	struct i915_address_space *vm,
140	struct drm_file *file)
141	{	444	{
142	struct drm_i915_gem_object *obj;	445	int err;
143	struct list_head objects;
144	int i, ret;
145		446
146	INIT_LIST_HEAD(&objects);	447	GEM_BUG_ON(i915_vma_is_closed(vma));
147	spin_lock(&file->table_lock);	448
148	/* Grab a reference to the object and release the lock so we can lookup	449	if (!(eb->args->flags & __EXEC_VALIDATED)) {
149	* or create the VMA without using GFP_ATOMIC */	450	err = eb_validate_vma(eb, entry, vma);
150	for (i = 0; i < args->buffer_count; i++) {	451	if (unlikely(err))
151	obj = to_intel_bo(idr_find(&file->object_idr, exec[i].handle));	452	return err;
152	if (obj == NULL) {	453	}
153	spin_unlock(&file->table_lock);	454
154	DRM_DEBUG("Invalid object handle %d at index %d\n",	455	if (eb->lut_size >= 0) {
155	exec[i].handle, i);	456	vma->exec_handle = entry->handle;
156	ret = -ENOENT;	457	hlist_add_head(&vma->exec_node,
157	goto err;	458	&eb->buckets[hash_32(entry->handle,
		459	eb->lut_size)]);
		460	}
		461
		462	if (entry->relocation_count)
		463	list_add_tail(&vma->reloc_link, &eb->relocs);
		464
		465	if (!eb->reloc_cache.has_fence) {
		466	entry->flags &= ~EXEC_OBJECT_NEEDS_FENCE;
		467	} else {
		468	if ((entry->flags & EXEC_OBJECT_NEEDS_FENCE \|\|
		469	eb->reloc_cache.needs_unfenced) &&
		470	i915_gem_object_is_tiled(vma->obj))
		471	entry->flags \|= EXEC_OBJECT_NEEDS_GTT \| __EXEC_OBJECT_NEEDS_MAP;
		472	}
		473
		474	if (!(entry->flags & EXEC_OBJECT_PINNED))
		475	entry->flags \|= eb->context_flags;
		476
		477	/*
		478	* Stash a pointer from the vma to execobj, so we can query its flags,
		479	* size, alignment etc as provided by the user. Also we stash a pointer
		480	* to the vma inside the execobj so that we can use a direct lookup
		481	* to find the right target VMA when doing relocations.
		482	*/
		483	vma->exec_entry = entry;
		484	__exec_to_vma(entry) = (uintptr_t)vma;
		485
		486	err = 0;
		487	eb_pin_vma(eb, entry, vma);
		488	if (eb_vma_misplaced(entry, vma)) {
		489	eb_unreserve_vma(vma, entry);
		490
		491	list_add_tail(&vma->exec_link, &eb->unbound);
		492	if (drm_mm_node_allocated(&vma->node))
		493	err = i915_vma_unbind(vma);
		494	} else {
		495	if (entry->offset != vma->node.start) {
		496	entry->offset = vma->node.start \| UPDATE;
		497	eb->args->flags \|= __EXEC_HAS_RELOC;
		498	}
		499	}
		500	return err;
		501	}
		502
		503	static inline int use_cpu_reloc(const struct reloc_cache *cache,
		504	const struct drm_i915_gem_object *obj)
		505	{
		506	if (!i915_gem_object_has_struct_page(obj))
		507	return false;
		508
		509	if (DBG_FORCE_RELOC == FORCE_CPU_RELOC)
		510	return true;
		511
		512	if (DBG_FORCE_RELOC == FORCE_GTT_RELOC)
		513	return false;
		514
		515	return (cache->has_llc \|\|
		516	obj->cache_dirty \|\|
		517	obj->cache_level != I915_CACHE_NONE);
		518	}
		519
		520	static int eb_reserve_vma(const struct i915_execbuffer *eb,
		521	struct i915_vma *vma)
		522	{
		523	struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;
		524	u64 flags;
		525	int err;
		526
		527	flags = PIN_USER \| PIN_NONBLOCK;
		528	if (entry->flags & EXEC_OBJECT_NEEDS_GTT)
		529	flags \|= PIN_GLOBAL;
		530
		531	/*
		532	* Wa32bitGeneralStateOffset & Wa32bitInstructionBaseOffset,
		533	* limit address to the first 4GBs for unflagged objects.
		534	*/
		535	if (!(entry->flags & EXEC_OBJECT_SUPPORTS_48B_ADDRESS))
		536	flags \|= PIN_ZONE_4G;
		537
		538	if (entry->flags & __EXEC_OBJECT_NEEDS_MAP)
		539	flags \|= PIN_MAPPABLE;
		540
		541	if (entry->flags & EXEC_OBJECT_PINNED) {
		542	flags \|= entry->offset \| PIN_OFFSET_FIXED;
		543	flags &= ~PIN_NONBLOCK; /* force overlapping PINNED checks */
		544	} else if (entry->flags & __EXEC_OBJECT_NEEDS_BIAS) {
		545	flags \|= BATCH_OFFSET_BIAS \| PIN_OFFSET_BIAS;
		546	}
		547
		548	err = i915_vma_pin(vma, entry->pad_to_size, entry->alignment, flags);
		549	if (err)
		550	return err;
		551
		552	if (entry->offset != vma->node.start) {
		553	entry->offset = vma->node.start \| UPDATE;
		554	eb->args->flags \|= __EXEC_HAS_RELOC;
		555	}
		556
		557	entry->flags \|= __EXEC_OBJECT_HAS_PIN;
		558	GEM_BUG_ON(eb_vma_misplaced(entry, vma));
		559
		560	if (unlikely(entry->flags & EXEC_OBJECT_NEEDS_FENCE)) {
		561	err = i915_vma_get_fence(vma);
		562	if (unlikely(err)) {
		563	i915_vma_unpin(vma);
		564	return err;
		565	}
		566
		567	if (i915_vma_pin_fence(vma))
		568	entry->flags \|= __EXEC_OBJECT_HAS_FENCE;
		569	}
		570
		571	return 0;
		572	}
		573
		574	static int eb_reserve(struct i915_execbuffer *eb)
		575	{
		576	const unsigned int count = eb->buffer_count;
		577	struct list_head last;
		578	struct i915_vma *vma;
		579	unsigned int i, pass;
		580	int err;
		581
		582	/*
		583	* Attempt to pin all of the buffers into the GTT.
		584	* This is done in 3 phases:
		585	*
		586	* 1a. Unbind all objects that do not match the GTT constraints for
		587	* the execbuffer (fenceable, mappable, alignment etc).
		588	* 1b. Increment pin count for already bound objects.
		589	* 2. Bind new objects.
		590	* 3. Decrement pin count.
		591	*
		592	* This avoid unnecessary unbinding of later objects in order to make
		593	* room for the earlier objects unless we need to defragment.
		594	*/
		595
		596	pass = 0;
		597	err = 0;
		598	do {
		599	list_for_each_entry(vma, &eb->unbound, exec_link) {
		600	err = eb_reserve_vma(eb, vma);
		601	if (err)
		602	break;
		603	}
		604	if (err != -ENOSPC)
		605	return err;
		606
		607	/* Resort all the objects into priority order */
		608	INIT_LIST_HEAD(&eb->unbound);
		609	INIT_LIST_HEAD(&last);
		610	for (i = 0; i < count; i++) {
		611	struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];
		612
		613	if (entry->flags & EXEC_OBJECT_PINNED &&
		614	entry->flags & __EXEC_OBJECT_HAS_PIN)
		615	continue;
		616
		617	vma = exec_to_vma(entry);
		618	eb_unreserve_vma(vma, entry);
		619
		620	if (entry->flags & EXEC_OBJECT_PINNED)
		621	list_add(&vma->exec_link, &eb->unbound);
		622	else if (entry->flags & __EXEC_OBJECT_NEEDS_MAP)
		623	list_add_tail(&vma->exec_link, &eb->unbound);
		624	else
		625	list_add_tail(&vma->exec_link, &last);
		626	}
		627	list_splice_tail(&last, &eb->unbound);
		628
		629	switch (pass++) {
		630	case 0:
		631	break;
		632
		633	case 1:
		634	/* Too fragmented, unbind everything and retry */
		635	err = i915_gem_evict_vm(eb->vm);
		636	if (err)
		637	return err;
		638	break;
		639
		640	default:
		641	return -ENOSPC;
		642	}
		643	} while (1);
		644	}
		645
		646	static inline struct hlist_head *
		647	ht_head(const struct i915_gem_context_vma_lut *lut, u32 handle)
		648	{
		649	return &lut->ht[hash_32(handle, lut->ht_bits)];
		650	}
		651
		652	static inline bool
		653	ht_needs_resize(const struct i915_gem_context_vma_lut *lut)
		654	{
		655	return (4lut->ht_count > 3lut->ht_size \|\|
		656	4*lut->ht_count + 1 < lut->ht_size);
		657	}
		658
		659	static unsigned int eb_batch_index(const struct i915_execbuffer *eb)
		660	{
		661	if (eb->args->flags & I915_EXEC_BATCH_FIRST)
		662	return 0;
		663	else
		664	return eb->buffer_count - 1;
		665	}
		666
		667	static int eb_select_context(struct i915_execbuffer *eb)
		668	{
		669	struct i915_gem_context *ctx;
		670
		671	ctx = i915_gem_context_lookup(eb->file->driver_priv, eb->args->rsvd1);
		672	if (unlikely(IS_ERR(ctx)))
		673	return PTR_ERR(ctx);
		674
		675	if (unlikely(i915_gem_context_is_banned(ctx))) {
		676	DRM_DEBUG("Context %u tried to submit while banned\n",
		677	ctx->user_handle);
		678	return -EIO;
		679	}
		680
		681	eb->ctx = i915_gem_context_get(ctx);
		682	eb->vm = ctx->ppgtt ? &ctx->ppgtt->base : &eb->i915->ggtt.base;
		683
		684	eb->context_flags = 0;
		685	if (ctx->flags & CONTEXT_NO_ZEROMAP)
		686	eb->context_flags \|= __EXEC_OBJECT_NEEDS_BIAS;
		687
		688	return 0;
		689	}
		690
		691	static int eb_lookup_vmas(struct i915_execbuffer *eb)
		692	{
		693	#define INTERMEDIATE BIT(0)
		694	const unsigned int count = eb->buffer_count;
		695	struct i915_gem_context_vma_lut *lut = &eb->ctx->vma_lut;
		696	struct i915_vma *vma;
		697	struct idr *idr;
		698	unsigned int i;
		699	int slow_pass = -1;
		700	int err;
		701
		702	INIT_LIST_HEAD(&eb->relocs);
		703	INIT_LIST_HEAD(&eb->unbound);
		704
		705	if (unlikely(lut->ht_size & I915_CTX_RESIZE_IN_PROGRESS))
		706	flush_work(&lut->resize);
		707	GEM_BUG_ON(lut->ht_size & I915_CTX_RESIZE_IN_PROGRESS);
		708
		709	for (i = 0; i < count; i++) {
		710	__exec_to_vma(&eb->exec[i]) = 0;
		711
		712	hlist_for_each_entry(vma,
		713	ht_head(lut, eb->exec[i].handle),
		714	ctx_node) {
		715	if (vma->ctx_handle != eb->exec[i].handle)
		716	continue;
		717
		718	err = eb_add_vma(eb, &eb->exec[i], vma);
		719	if (unlikely(err))
		720	return err;
		721
		722	goto next_vma;
158	}	723	}
159		724
160	if (!list_empty(&obj->obj_exec_link)) {	725	if (slow_pass < 0)
161	spin_unlock(&file->table_lock);	726	slow_pass = i;
162	DRM_DEBUG("Object %p [handle %d, index %d] appears more than once in object list\n",	727	next_vma: ;
163	obj, exec[i].handle, i);	728	}
164	ret = -EINVAL;	729
		730	if (slow_pass < 0)
		731	goto out;
		732
		733	spin_lock(&eb->file->table_lock);
		734	/*
		735	* Grab a reference to the object and release the lock so we can lookup
		736	* or create the VMA without using GFP_ATOMIC
		737	*/
		738	idr = &eb->file->object_idr;
		739	for (i = slow_pass; i < count; i++) {
		740	struct drm_i915_gem_object *obj;
		741
		742	if (__exec_to_vma(&eb->exec[i]))
		743	continue;
		744
		745	obj = to_intel_bo(idr_find(idr, eb->exec[i].handle));
		746	if (unlikely(!obj)) {
		747	spin_unlock(&eb->file->table_lock);
		748	DRM_DEBUG("Invalid object handle %d at index %d\n",
		749	eb->exec[i].handle, i);
		750	err = -ENOENT;
165	goto err;	751	goto err;
166	}	752	}
167		753
168	i915_gem_object_get(obj);	754	__exec_to_vma(&eb->exec[i]) = INTERMEDIATE \| (uintptr_t)obj;
169	list_add_tail(&obj->obj_exec_link, &objects);
170	}	755	}
171	spin_unlock(&file->table_lock);	756	spin_unlock(&eb->file->table_lock);
172		757
173	i = 0;	758	for (i = slow_pass; i < count; i++) {
174	while (!list_empty(&objects)) {	759	struct drm_i915_gem_object *obj;
175	struct i915_vma *vma;
176		760
177	obj = list_first_entry(&objects,	761	if (!(__exec_to_vma(&eb->exec[i]) & INTERMEDIATE))
178	struct drm_i915_gem_object,	762	continue;
179	obj_exec_link);
180		763
181	/*	764	/*
182	* NOTE: We can leak any vmas created here when something fails	765	* NOTE: We can leak any vmas created here when something fails
@@ -186,59 +769,93 @@ eb_lookup_vmas(struct eb_vmas *eb, · 3 unmodified lines
186	* from the (obj, vm) we don't run the risk of creating	769	* from the (obj, vm) we don't run the risk of creating
187	* duplicated vmas for the same vm.	770	* duplicated vmas for the same vm.
188	*/	771	*/
189	vma = i915_vma_instance(obj, vm, NULL);	772	obj = u64_to_ptr(typeof(*obj),
		773	__exec_to_vma(&eb->exec[i]) & ~INTERMEDIATE);
		774	vma = i915_vma_instance(obj, eb->vm, NULL);
190	if (unlikely(IS_ERR(vma))) {	775	if (unlikely(IS_ERR(vma))) {
191	DRM_DEBUG("Failed to lookup VMA\n");	776	DRM_DEBUG("Failed to lookup VMA\n");
192	ret = PTR_ERR(vma);	777	err = PTR_ERR(vma);
193	goto err;	778	goto err;
194	}	779	}
195		780
196	/* Transfer ownership from the objects list to the vmas list. */	781	/* First come, first served */
197	list_add_tail(&vma->exec_list, &eb->vmas);	782	if (!vma->ctx) {
198	list_del_init(&obj->obj_exec_link);	783	vma->ctx = eb->ctx;
		784	vma->ctx_handle = eb->exec[i].handle;
		785	hlist_add_head(&vma->ctx_node,
		786	ht_head(lut, eb->exec[i].handle));
		787	lut->ht_count++;
		788	lut->ht_size \|= I915_CTX_RESIZE_IN_PROGRESS;
		789	if (i915_vma_is_ggtt(vma)) {
		790	GEM_BUG_ON(obj->vma_hashed);
		791	obj->vma_hashed = vma;
		792	}
199		793
200	vma->exec_entry = &exec[i];	794	i915_vma_get(vma);
201	if (eb->and < 0) {	795	}
202	eb->lut[i] = vma;	796
203	} else {	797	err = eb_add_vma(eb, &eb->exec[i], vma);
204	uint32_t handle = args->flags & I915_EXEC_HANDLE_LUT ? i : exec[i].handle;	798	if (unlikely(err))
205	vma->exec_handle = handle;	799	goto err;
206	hlist_add_head(&vma->exec_node,	800
207	&eb->buckets[handle & eb->and]);	801	/* Only after we validated the user didn't use our bits */
		802	if (vma->ctx != eb->ctx) {
		803	i915_vma_get(vma);
		804	eb->exec[i].flags \|= __EXEC_OBJECT_HAS_REF;
208	}	805	}
209	++i;
210	}	806	}
211		807
212	return 0;	808	if (lut->ht_size & I915_CTX_RESIZE_IN_PROGRESS) {
		809	if (ht_needs_resize(lut))
		810	queue_work(system_highpri_wq, &lut->resize);
		811	else
		812	lut->ht_size &= ~I915_CTX_RESIZE_IN_PROGRESS;
		813	}
		814
		815	out:
		816	/* take note of the batch buffer before we might reorder the lists */
		817	i = eb_batch_index(eb);
		818	eb->batch = exec_to_vma(&eb->exec[i]);
		819
		820	/*
		821	* SNA is doing fancy tricks with compressing batch buffers, which leads
		822	* to negative relocation deltas. Usually that works out ok since the
		823	* relocate address is still positive, except when the batch is placed
		824	* very low in the GTT. Ensure this doesn't happen.
		825	*
		826	* Note that actual hangs have only been observed on gen7, but for
		827	* paranoia do it everywhere.
		828	*/
		829	if (!(eb->exec[i].flags & EXEC_OBJECT_PINNED))
		830	eb->exec[i].flags \|= __EXEC_OBJECT_NEEDS_BIAS;
		831	if (eb->reloc_cache.has_fence)
		832	eb->exec[i].flags \|= EXEC_OBJECT_NEEDS_FENCE;
213		833
		834	eb->args->flags \|= __EXEC_VALIDATED;
		835	return eb_reserve(eb);
214		836
215	err:	837	err:
216	while (!list_empty(&objects)) {	838	for (i = slow_pass; i < count; i++) {
217	obj = list_first_entry(&objects,	839	if (__exec_to_vma(&eb->exec[i]) & INTERMEDIATE)
218	struct drm_i915_gem_object,	840	__exec_to_vma(&eb->exec[i]) = 0;
219	obj_exec_link);
220	list_del_init(&obj->obj_exec_link);
221	i915_gem_object_put(obj);
222	}	841	}
223	/*	842	lut->ht_size &= ~I915_CTX_RESIZE_IN_PROGRESS;
224	* Objects already transfered to the vmas list will be unreferenced by	843	return err;
225	* eb_destroy.	844	#undef INTERMEDIATE
226	*/
227
228	return ret;
229	}	845	}
230		846
231	static struct i915_vma eb_get_vma(struct eb_vmas eb, unsigned long handle)	847	static struct i915_vma *
		848	eb_get_vma(const struct i915_execbuffer *eb, unsigned long handle)
232	{	849	{
233	if (eb->and < 0) {	850	if (eb->lut_size < 0) {
234	if (handle >= -eb->and)	851	if (handle >= -eb->lut_size)
235	return NULL;	852	return NULL;
236	return eb->lut[handle];	853	return exec_to_vma(&eb->exec[handle]);
237	} else {	854	} else {
238	struct hlist_head *head;	855	struct hlist_head *head;
239	struct i915_vma *vma;	856	struct i915_vma *vma;
240		857
241	head = &eb->buckets[handle & eb->and];	858	head = &eb->buckets[hash_32(handle, eb->lut_size)];
242	hlist_for_each_entry(vma, head, exec_node) {	859	hlist_for_each_entry(vma, head, exec_node) {
243	if (vma->exec_handle == handle)	860	if (vma->exec_handle == handle)
244	return vma;	861	return vma;
@@ -247,96 +864,69 @@ static struct i915_vma eb_get_vma(struct eb_vmas eb, unsigned long handle) · 2 unmodified lines
247	}	864	}
248	}	865	}
249		866
250	static void	867	static void eb_release_vmas(const struct i915_execbuffer *eb)
251	i915_gem_execbuffer_unreserve_vma(struct i915_vma *vma)
252	{	868	{
253	struct drm_i915_gem_exec_object2 *entry;	869	const unsigned int count = eb->buffer_count;
		870	unsigned int i;
254		871
255	if (!drm_mm_node_allocated(&vma->node))	872	for (i = 0; i < count; i++) {
256	return;	873	struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];
257		874	struct i915_vma *vma = exec_to_vma(entry);
258	entry = vma->exec_entry;
259		875
260	if (entry->flags & __EXEC_OBJECT_HAS_FENCE)	876	if (!vma)
261	i915_vma_unpin_fence(vma);	877	continue;
262		878
263	if (entry->flags & __EXEC_OBJECT_HAS_PIN)	879	GEM_BUG_ON(vma->exec_entry != entry);
264	__i915_vma_unpin(vma);	880	vma->exec_entry = NULL;
265		881
266	entry->flags &= ~(__EXEC_OBJECT_HAS_FENCE \| __EXEC_OBJECT_HAS_PIN);	882	if (entry->flags & __EXEC_OBJECT_HAS_PIN)
267	}	883	__eb_unreserve_vma(vma, entry);
268		884
269	static void eb_destroy(struct eb_vmas *eb)	885	if (entry->flags & __EXEC_OBJECT_HAS_REF)
270	{	886	i915_vma_put(vma);
271	while (!list_empty(&eb->vmas)) {
272	struct i915_vma *vma;
273		887
274	vma = list_first_entry(&eb->vmas,	888	entry->flags &=
275	struct i915_vma,	889	~(__EXEC_OBJECT_RESERVED \| __EXEC_OBJECT_HAS_REF);
276	exec_list);
277	list_del_init(&vma->exec_list);
278	i915_gem_execbuffer_unreserve_vma(vma);
279	vma->exec_entry = NULL;
280	i915_vma_put(vma);
281	}	890	}
282	kfree(eb);
283	}	891	}
284		892
285	static inline int use_cpu_reloc(struct drm_i915_gem_object *obj)	893	static void eb_reset_vmas(const struct i915_execbuffer *eb)
286	{	894	{
287	if (!i915_gem_object_has_struct_page(obj))	895	eb_release_vmas(eb);
288	return false;	896	if (eb->lut_size >= 0)
289		897	memset(eb->buckets, 0,
290	if (DBG_USE_CPU_RELOC)	898	sizeof(struct hlist_head) << eb->lut_size);
291	return DBG_USE_CPU_RELOC > 0;
292
293	return (HAS_LLC(to_i915(obj->base.dev)) \|\|
294	obj->base.write_domain == I915_GEM_DOMAIN_CPU \|\|
295	obj->cache_level != I915_CACHE_NONE);
296	}	899	}
297		900
298	/* Used to convert any address to canonical form.	901	static void eb_destroy(const struct i915_execbuffer *eb)
299	* Starting from gen8, some commands (e.g. STATE_BASE_ADDRESS,
300	* MI_LOAD_REGISTER_MEM and others, see Broadwell PRM Vol2a) require the
301	* addresses to be in a canonical form:
302	* "GraphicsAddress[63:48] are ignored by the HW and assumed to be in correct
303	* canonical form [63:48] == [47]."
304	*/
305	#define GEN8_HIGH_ADDRESS_BIT 47
306	static inline uint64_t gen8_canonical_addr(uint64_t address)
307	{	902	{
308	return sign_extend64(address, GEN8_HIGH_ADDRESS_BIT);	903	GEM_BUG_ON(eb->reloc_cache.rq);
309	}
310		904
311	static inline uint64_t gen8_noncanonical_addr(uint64_t address)	905	if (eb->lut_size >= 0)
312	{	906	kfree(eb->buckets);
313	return address & ((1ULL << (GEN8_HIGH_ADDRESS_BIT + 1)) - 1);
314	}	907	}
315		908
316	static inline uint64_t	909	static inline u64
317	relocation_target(const struct drm_i915_gem_relocation_entry *reloc,	910	relocation_target(const struct drm_i915_gem_relocation_entry *reloc,
318	uint64_t target_offset)	911	const struct i915_vma *target)
319	{	912	{
320	return gen8_canonical_addr((int)reloc->delta + target_offset);	913	return gen8_canonical_addr((int)reloc->delta + target->node.start);
321	}	914	}
322		915
323	struct reloc_cache {
324	struct drm_i915_private *i915;
325	struct drm_mm_node node;
326	unsigned long vaddr;
327	unsigned int page;
328	bool use_64bit_reloc;
329	};
330
331	static void reloc_cache_init(struct reloc_cache *cache,	916	static void reloc_cache_init(struct reloc_cache *cache,
332	struct drm_i915_private *i915)	917	struct drm_i915_private *i915)
333	{	918	{
334	cache->page = -1;	919	cache->page = -1;
335	cache->vaddr = 0;	920	cache->vaddr = 0;
336	cache->i915 = i915;
337	/* Must be a variable in the struct to allow GCC to unroll. */	921	/* Must be a variable in the struct to allow GCC to unroll. */
		922	cache->gen = INTEL_GEN(i915);
		923	cache->has_llc = HAS_LLC(i915);
338	cache->use_64bit_reloc = HAS_64BIT_RELOC(i915);	924	cache->use_64bit_reloc = HAS_64BIT_RELOC(i915);
		925	cache->has_fence = cache->gen < 4;
		926	cache->needs_unfenced = INTEL_INFO(i915)->unfenced_needs_alignment;
339	cache->node.allocated = false;	927	cache->node.allocated = false;
		928	cache->rq = NULL;
		929	cache->rq_size = 0;
340	}	930	}
341		931
342	static inline void *unmask_page(unsigned long p)	932	static inline void *unmask_page(unsigned long p)
@@ -351,10 +941,31 @@ static inline unsigned int unmask_flags(unsigned long p) · 8 unmodified lines
351		941
352	#define KMAP 0x4 /* after CLFLUSH_FLAGS */	942	#define KMAP 0x4 /* after CLFLUSH_FLAGS */
353		943
354	static void reloc_cache_fini(struct reloc_cache *cache)	944	static inline struct i915_ggtt cache_to_ggtt(struct reloc_cache cache)
		945	{
		946	struct drm_i915_private *i915 =
		947	container_of(cache, struct i915_execbuffer, reloc_cache)->i915;
		948	return &i915->ggtt;
		949	}
		950
		951	static void reloc_gpu_flush(struct reloc_cache *cache)
		952	{
		953	GEM_BUG_ON(cache->rq_size >= cache->rq->batch->obj->base.size / sizeof(u32));
		954	cache->rq_cmd[cache->rq_size] = MI_BATCH_BUFFER_END;
		955	i915_gem_object_unpin_map(cache->rq->batch->obj);
		956	i915_gem_chipset_flush(cache->rq->i915);
		957
		958	__i915_add_request(cache->rq, true);
		959	cache->rq = NULL;
		960	}
		961
		962	static void reloc_cache_reset(struct reloc_cache *cache)
355	{	963	{
356	void *vaddr;	964	void *vaddr;
357		965
		966	if (cache->rq)
		967	reloc_gpu_flush(cache);
		968
358	if (!cache->vaddr)	969	if (!cache->vaddr)
359	return;	970	return;
360		971
@@ -369,7 +980,7 @@ static void reloc_cache_fini(struct reloc_cache *cache) · 8 unmodified lines
369	wmb();	980	wmb();
370	io_mapping_unmap_atomic((void __iomem *)vaddr);	981	io_mapping_unmap_atomic((void __iomem *)vaddr);
371	if (cache->node.allocated) {	982	if (cache->node.allocated) {
372	struct i915_ggtt *ggtt = &cache->i915->ggtt;	983	struct i915_ggtt *ggtt = cache_to_ggtt(cache);
373		984
374	ggtt->base.clear_range(&ggtt->base,	985	ggtt->base.clear_range(&ggtt->base,
375	cache->node.start,	986	cache->node.start,
@@ -379,11 +990,14 @@ static void reloc_cache_fini(struct reloc_cache *cache) · 3 unmodified lines
379	i915_vma_unpin((struct i915_vma *)cache->node.mm);	990	i915_vma_unpin((struct i915_vma *)cache->node.mm);
380	}	991	}
381	}	992	}
		993
		994	cache->vaddr = 0;
		995	cache->page = -1;
382	}	996	}
383		997
384	static void reloc_kmap(struct drm_i915_gem_object obj,	998	static void reloc_kmap(struct drm_i915_gem_object obj,
385	struct reloc_cache *cache,	999	struct reloc_cache *cache,
386	int page)	1000	unsigned long page)
387	{	1001	{
388	void *vaddr;	1002	void *vaddr;
389		1003
@@ -391,11 +1005,11 @@ static void reloc_kmap(struct drm_i915_gem_object obj, · 1 unmodified line
391	kunmap_atomic(unmask_page(cache->vaddr));	1005	kunmap_atomic(unmask_page(cache->vaddr));
392	} else {	1006	} else {
393	unsigned int flushes;	1007	unsigned int flushes;
394	int ret;	1008	int err;
395		1009
396	ret = i915_gem_obj_prepare_shmem_write(obj, &flushes);	1010	err = i915_gem_obj_prepare_shmem_write(obj, &flushes);
397	if (ret)	1011	if (err)
398	return ERR_PTR(ret);	1012	return ERR_PTR(err);
399		1013
400	BUILD_BUG_ON(KMAP & CLFLUSH_FLAGS);	1014	BUILD_BUG_ON(KMAP & CLFLUSH_FLAGS);
401	BUILD_BUG_ON((KMAP \| CLFLUSH_FLAGS) & PAGE_MASK);	1015	BUILD_BUG_ON((KMAP \| CLFLUSH_FLAGS) & PAGE_MASK);
@@ -415,9 +1029,9 @@ static void reloc_kmap(struct drm_i915_gem_object obj, · 13 unmodified lines
415		1029
416	static void reloc_iomap(struct drm_i915_gem_object obj,	1030	static void reloc_iomap(struct drm_i915_gem_object obj,
417	struct reloc_cache *cache,	1031	struct reloc_cache *cache,
418	int page)	1032	unsigned long page)
419	{	1033	{
420	struct i915_ggtt *ggtt = &cache->i915->ggtt;	1034	struct i915_ggtt *ggtt = cache_to_ggtt(cache);
421	unsigned long offset;	1035	unsigned long offset;
422	void *vaddr;	1036	void *vaddr;
423		1037
@@ -425,31 +1039,31 @@ static void reloc_iomap(struct drm_i915_gem_object obj, · 1 unmodified line
425	io_mapping_unmap_atomic((void __force __iomem *) unmask_page(cache->vaddr));	1039	io_mapping_unmap_atomic((void __force __iomem *) unmask_page(cache->vaddr));
426	} else {	1040	} else {
427	struct i915_vma *vma;	1041	struct i915_vma *vma;
428	int ret;	1042	int err;
429		1043
430	if (use_cpu_reloc(obj))	1044	if (use_cpu_reloc(cache, obj))
431	return NULL;	1045	return NULL;
432		1046
433	ret = i915_gem_object_set_to_gtt_domain(obj, true);	1047	err = i915_gem_object_set_to_gtt_domain(obj, true);
434	if (ret)	1048	if (err)
435	return ERR_PTR(ret);	1049	return ERR_PTR(err);
436		1050
437	vma = i915_gem_object_ggtt_pin(obj, NULL, 0, 0,	1051	vma = i915_gem_object_ggtt_pin(obj, NULL, 0, 0,
438	PIN_MAPPABLE \| PIN_NONBLOCK);	1052	PIN_MAPPABLE \| PIN_NONBLOCK);
439	if (IS_ERR(vma)) {	1053	if (IS_ERR(vma)) {
440	memset(&cache->node, 0, sizeof(cache->node));	1054	memset(&cache->node, 0, sizeof(cache->node));
441	ret = drm_mm_insert_node_in_range	1055	err = drm_mm_insert_node_in_range
442	(&ggtt->base.mm, &cache->node,	1056	(&ggtt->base.mm, &cache->node,
443	PAGE_SIZE, 0, I915_COLOR_UNEVICTABLE,	1057	PAGE_SIZE, 0, I915_COLOR_UNEVICTABLE,
444	0, ggtt->mappable_end,	1058	0, ggtt->mappable_end,
445	DRM_MM_INSERT_LOW);	1059	DRM_MM_INSERT_LOW);
446	if (ret) /* no inactive aperture space, use cpu reloc */	1060	if (err) /* no inactive aperture space, use cpu reloc */
447	return NULL;	1061	return NULL;
448	} else {	1062	} else {
449	ret = i915_vma_put_fence(vma);	1063	err = i915_vma_put_fence(vma);
450	if (ret) {	1064	if (err) {
451	i915_vma_unpin(vma);	1065	i915_vma_unpin(vma);
452	return ERR_PTR(ret);	1066	return ERR_PTR(err);
453	}	1067	}
454		1068
455	cache->node.start = vma->node.start;	1069	cache->node.start = vma->node.start;
@@ -467,7 +1081,8 @@ static void reloc_iomap(struct drm_i915_gem_object obj, · 11 unmodified lines
467	offset += page << PAGE_SHIFT;	1081	offset += page << PAGE_SHIFT;
468	}	1082	}
469		1083
470	vaddr = (void __force *) io_mapping_map_atomic_wc(&cache->i915->ggtt.mappable, offset);	1084	vaddr = (void __force *)io_mapping_map_atomic_wc(&ggtt->mappable,
		1085	offset);
471	cache->page = page;	1086	cache->page = page;
472	cache->vaddr = (unsigned long)vaddr;	1087	cache->vaddr = (unsigned long)vaddr;
473		1088
@@ -476,7 +1091,7 @@ static void reloc_iomap(struct drm_i915_gem_object obj, · 2 unmodified lines
476		1091
477	static void reloc_vaddr(struct drm_i915_gem_object obj,	1092	static void reloc_vaddr(struct drm_i915_gem_object obj,
478	struct reloc_cache *cache,	1093	struct reloc_cache *cache,
479	int page)	1094	unsigned long page)
480	{	1095	{
481	void *vaddr;	1096	void *vaddr;
482		1097
@@ -503,7 +1118,8 @@ static void clflush_write32(u32 *addr, u32 value, unsigned int flushes) · 20 unmodified lines
503		1118
504	*addr = value;	1119	*addr = value;
505		1120
506	/* Writes to the same cacheline are serialised by the CPU	1121	/*
		1122	* Writes to the same cacheline are serialised by the CPU
507	* (including clflush). On the write path, we only require	1123	* (including clflush). On the write path, we only require
508	* that it hits memory in an orderly fashion and place	1124	* that it hits memory in an orderly fashion and place
509	* mb barriers at the start and end of the relocation phase	1125	* mb barriers at the start and end of the relocation phase
@@ -515,25 +1131,201 @@ static void clflush_write32(u32 *addr, u32 value, unsigned int flushes) · 5 unmodified lines
515	*addr = value;	1131	*addr = value;
516	}	1132	}
517		1133
518	static int	1134	static int __reloc_gpu_alloc(struct i915_execbuffer *eb,
519	relocate_entry(struct drm_i915_gem_object *obj,	1135	struct i915_vma *vma,
		1136	unsigned int len)
		1137	{
		1138	struct reloc_cache *cache = &eb->reloc_cache;
		1139	struct drm_i915_gem_object *obj;
		1140	struct drm_i915_gem_request *rq;
		1141	struct i915_vma *batch;
		1142	u32 *cmd;
		1143	int err;
		1144
		1145	GEM_BUG_ON(vma->obj->base.write_domain & I915_GEM_DOMAIN_CPU);
		1146
		1147	obj = i915_gem_batch_pool_get(&eb->engine->batch_pool, PAGE_SIZE);
		1148	if (IS_ERR(obj))
		1149	return PTR_ERR(obj);
		1150
		1151	cmd = i915_gem_object_pin_map(obj,
		1152	cache->has_llc ? I915_MAP_WB : I915_MAP_WC);
		1153	i915_gem_object_unpin_pages(obj);
		1154	if (IS_ERR(cmd))
		1155	return PTR_ERR(cmd);
		1156
		1157	err = i915_gem_object_set_to_wc_domain(obj, false);
		1158	if (err)
		1159	goto err_unmap;
		1160
		1161	batch = i915_vma_instance(obj, vma->vm, NULL);
		1162	if (IS_ERR(batch)) {
		1163	err = PTR_ERR(batch);
		1164	goto err_unmap;
		1165	}
		1166
		1167	err = i915_vma_pin(batch, 0, 0, PIN_USER \| PIN_NONBLOCK);
		1168	if (err)
		1169	goto err_unmap;
		1170
		1171	rq = i915_gem_request_alloc(eb->engine, eb->ctx);
		1172	if (IS_ERR(rq)) {
		1173	err = PTR_ERR(rq);
		1174	goto err_unpin;
		1175	}
		1176
		1177	err = i915_gem_request_await_object(rq, vma->obj, true);
		1178	if (err)
		1179	goto err_request;
		1180
		1181	err = eb->engine->emit_flush(rq, EMIT_INVALIDATE);
		1182	if (err)
		1183	goto err_request;
		1184
		1185	err = i915_switch_context(rq);
		1186	if (err)
		1187	goto err_request;
		1188
		1189	err = eb->engine->emit_bb_start(rq,
		1190	batch->node.start, PAGE_SIZE,
		1191	cache->gen > 5 ? 0 : I915_DISPATCH_SECURE);
		1192	if (err)
		1193	goto err_request;
		1194
		1195	GEM_BUG_ON(!reservation_object_test_signaled_rcu(batch->resv, true));
		1196	i915_vma_move_to_active(batch, rq, 0);
		1197	reservation_object_lock(batch->resv, NULL);
		1198	reservation_object_add_excl_fence(batch->resv, &rq->fence);
		1199	reservation_object_unlock(batch->resv);
		1200	i915_vma_unpin(batch);
		1201
		1202	i915_vma_move_to_active(vma, rq, true);
		1203	reservation_object_lock(vma->resv, NULL);
		1204	reservation_object_add_excl_fence(vma->resv, &rq->fence);
		1205	reservation_object_unlock(vma->resv);
		1206
		1207	rq->batch = batch;
		1208
		1209	cache->rq = rq;
		1210	cache->rq_cmd = cmd;
		1211	cache->rq_size = 0;
		1212
		1213	/* Return with batch mapping (cmd) still pinned */
		1214	return 0;
		1215
		1216	err_request:
		1217	i915_add_request(rq);
		1218	err_unpin:
		1219	i915_vma_unpin(batch);
		1220	err_unmap:
		1221	i915_gem_object_unpin_map(obj);
		1222	return err;
		1223	}
		1224
		1225	static u32 reloc_gpu(struct i915_execbuffer eb,
		1226	struct i915_vma *vma,
		1227	unsigned int len)
		1228	{
		1229	struct reloc_cache *cache = &eb->reloc_cache;
		1230	u32 *cmd;
		1231
		1232	if (cache->rq_size > PAGE_SIZE/sizeof(u32) - (len + 1))
		1233	reloc_gpu_flush(cache);
		1234
		1235	if (unlikely(!cache->rq)) {
		1236	int err;
		1237
		1238	err = __reloc_gpu_alloc(eb, vma, len);
		1239	if (unlikely(err))
		1240	return ERR_PTR(err);
		1241	}
		1242
		1243	cmd = cache->rq_cmd + cache->rq_size;
		1244	cache->rq_size += len;
		1245
		1246	return cmd;
		1247	}
		1248
		1249	static u64
		1250	relocate_entry(struct i915_vma *vma,
520	const struct drm_i915_gem_relocation_entry *reloc,	1251	const struct drm_i915_gem_relocation_entry *reloc,
521	struct reloc_cache *cache,	1252	struct i915_execbuffer *eb,
522	u64 target_offset)	1253	const struct i915_vma *target)
523	{	1254	{
524	u64 offset = reloc->offset;	1255	u64 offset = reloc->offset;
525	bool wide = cache->use_64bit_reloc;	1256	u64 target_offset = relocation_target(reloc, target);
		1257	bool wide = eb->reloc_cache.use_64bit_reloc;
526	void *vaddr;	1258	void *vaddr;
527		1259
528	target_offset = relocation_target(reloc, target_offset);	1260	if (!eb->reloc_cache.vaddr &&
		1261	(DBG_FORCE_RELOC == FORCE_GPU_RELOC \|\|
		1262	!reservation_object_test_signaled_rcu(vma->resv, true))) {
		1263	const unsigned int gen = eb->reloc_cache.gen;
		1264	unsigned int len;
		1265	u32 *batch;
		1266	u64 addr;
		1267
		1268	if (wide)
		1269	len = offset & 7 ? 8 : 5;
		1270	else if (gen >= 4)
		1271	len = 4;
		1272	else if (gen >= 3)
		1273	len = 3;
		1274	else /* On gen2 MI_STORE_DWORD_IMM uses a physical address */
		1275	goto repeat;
		1276
		1277	batch = reloc_gpu(eb, vma, len);
		1278	if (IS_ERR(batch))
		1279	goto repeat;
		1280
		1281	addr = gen8_canonical_addr(vma->node.start + offset);
		1282	if (wide) {
		1283	if (offset & 7) {
		1284	*batch++ = MI_STORE_DWORD_IMM_GEN4;
		1285	*batch++ = lower_32_bits(addr);
		1286	*batch++ = upper_32_bits(addr);
		1287	*batch++ = lower_32_bits(target_offset);
		1288
		1289	addr = gen8_canonical_addr(addr + 4);
		1290
		1291	*batch++ = MI_STORE_DWORD_IMM_GEN4;
		1292	*batch++ = lower_32_bits(addr);
		1293	*batch++ = upper_32_bits(addr);
		1294	*batch++ = upper_32_bits(target_offset);
		1295	} else {
		1296	*batch++ = (MI_STORE_DWORD_IMM_GEN4 \| (1 << 21)) + 1;
		1297	*batch++ = lower_32_bits(addr);
		1298	*batch++ = upper_32_bits(addr);
		1299	*batch++ = lower_32_bits(target_offset);
		1300	*batch++ = upper_32_bits(target_offset);
		1301	}
		1302	} else if (gen >= 6) {
		1303	*batch++ = MI_STORE_DWORD_IMM_GEN4;
		1304	*batch++ = 0;
		1305	*batch++ = addr;
		1306	*batch++ = target_offset;
		1307	} else if (gen >= 4) {
		1308	*batch++ = MI_STORE_DWORD_IMM_GEN4 \| MI_USE_GGTT;
		1309	*batch++ = 0;
		1310	*batch++ = addr;
		1311	*batch++ = target_offset;
		1312	} else {
		1313	*batch++ = MI_STORE_DWORD_IMM \| MI_MEM_VIRTUAL;
		1314	*batch++ = addr;
		1315	*batch++ = target_offset;
		1316	}
		1317
		1318	goto out;
		1319	}
		1320
529	repeat:	1321	repeat:
530	vaddr = reloc_vaddr(obj, cache, offset >> PAGE_SHIFT);	1322	vaddr = reloc_vaddr(vma->obj, &eb->reloc_cache, offset >> PAGE_SHIFT);
531	if (IS_ERR(vaddr))	1323	if (IS_ERR(vaddr))
532	return PTR_ERR(vaddr);	1324	return PTR_ERR(vaddr);
533		1325
534	clflush_write32(vaddr + offset_in_page(offset),	1326	clflush_write32(vaddr + offset_in_page(offset),
535	lower_32_bits(target_offset),	1327	lower_32_bits(target_offset),
536	cache->vaddr);	1328	eb->reloc_cache.vaddr);
537		1329
538	if (wide) {	1330	if (wide) {
539	offset += sizeof(u32);	1331	offset += sizeof(u32);
@@ -542,48 +1334,29 @@ repeat: · 2 unmodified lines
542	goto repeat;	1334	goto repeat;
543	}	1335	}
544		1336
545	return 0;	1337	out:
		1338	return target->node.start \| UPDATE;
546	}	1339	}
547		1340
548	static int	1341	static u64
549	i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj,	1342	eb_relocate_entry(struct i915_execbuffer *eb,
550	struct eb_vmas *eb,	1343	struct i915_vma *vma,
551	struct drm_i915_gem_relocation_entry *reloc,	1344	const struct drm_i915_gem_relocation_entry *reloc)
552	struct reloc_cache *cache)
553	{	1345	{
554	struct drm_i915_private *dev_priv = to_i915(obj->base.dev);	1346	struct i915_vma *target;
555	struct drm_gem_object *target_obj;	1347	int err;
556	struct drm_i915_gem_object *target_i915_obj;
557	struct i915_vma *target_vma;
558	uint64_t target_offset;
559	int ret;
560		1348
561	/* we've already hold a reference to all valid objects */	1349	/* we've already hold a reference to all valid objects */
562	target_vma = eb_get_vma(eb, reloc->target_handle);	1350	target = eb_get_vma(eb, reloc->target_handle);
563	if (unlikely(target_vma == NULL))	1351	if (unlikely(!target))
564	return -ENOENT;	1352	return -ENOENT;
565	target_i915_obj = target_vma->obj;
566	target_obj = &target_vma->obj->base;
567
568	target_offset = gen8_canonical_addr(target_vma->node.start);
569
570	/* Sandybridge PPGTT errata: We need a global gtt mapping for MI and
571	* pipe_control writes because the gpu doesn't properly redirect them
572	* through the ppgtt for non_secure batchbuffers. */
573	if (unlikely(IS_GEN6(dev_priv) &&
574	reloc->write_domain == I915_GEM_DOMAIN_INSTRUCTION)) {
575	ret = i915_vma_bind(target_vma, target_i915_obj->cache_level,
576	PIN_GLOBAL);
577	if (WARN_ONCE(ret, "Unexpected failure to bind target VMA!"))
578	return ret;
579	}
580		1353
581	/* Validate that the target is in a valid r/w GPU domain */	1354	/* Validate that the target is in a valid r/w GPU domain */
582	if (unlikely(reloc->write_domain & (reloc->write_domain - 1))) {	1355	if (unlikely(reloc->write_domain & (reloc->write_domain - 1))) {
583	DRM_DEBUG("reloc with multiple write domains: "	1356	DRM_DEBUG("reloc with multiple write domains: "
584	"obj %p target %d offset %d "	1357	"target %d offset %d "
585	"read %08x write %08x",	1358	"read %08x write %08x",
586	obj, reloc->target_handle,	1359	reloc->target_handle,
587	(int) reloc->offset,	1360	(int) reloc->offset,
588	reloc->read_domains,	1361	reloc->read_domains,
589	reloc->write_domain);	1362	reloc->write_domain);
@@ -592,75 +1365,103 @@ i915_gem_execbuffer_relocate_entry(struct drm_i915_gem_object *obj, · 2 unmodified lines
592	if (unlikely((reloc->write_domain \| reloc->read_domains)	1365	if (unlikely((reloc->write_domain \| reloc->read_domains)
593	& ~I915_GEM_GPU_DOMAINS)) {	1366	& ~I915_GEM_GPU_DOMAINS)) {
594	DRM_DEBUG("reloc with read/write non-GPU domains: "	1367	DRM_DEBUG("reloc with read/write non-GPU domains: "
595	"obj %p target %d offset %d "	1368	"target %d offset %d "
596	"read %08x write %08x",	1369	"read %08x write %08x",
597	obj, reloc->target_handle,	1370	reloc->target_handle,
598	(int) reloc->offset,	1371	(int) reloc->offset,
599	reloc->read_domains,	1372	reloc->read_domains,
600	reloc->write_domain);	1373	reloc->write_domain);
601	return -EINVAL;	1374	return -EINVAL;
602	}	1375	}
603		1376
604	target_obj->pending_read_domains \|= reloc->read_domains;	1377	if (reloc->write_domain) {
605	target_obj->pending_write_domain \|= reloc->write_domain;	1378	target->exec_entry->flags \|= EXEC_OBJECT_WRITE;
		1379
		1380	/*
		1381	* Sandybridge PPGTT errata: We need a global gtt mapping
		1382	* for MI and pipe_control writes because the gpu doesn't
		1383	* properly redirect them through the ppgtt for non_secure
		1384	* batchbuffers.
		1385	*/
		1386	if (reloc->write_domain == I915_GEM_DOMAIN_INSTRUCTION &&
		1387	IS_GEN6(eb->i915)) {
		1388	err = i915_vma_bind(target, target->obj->cache_level,
		1389	PIN_GLOBAL);
		1390	if (WARN_ONCE(err,
		1391	"Unexpected failure to bind target VMA!"))
		1392	return err;
		1393	}
		1394	}
606		1395
607	/* If the relocation already has the right value in it, no	1396	/*
		1397	* If the relocation already has the right value in it, no
608	* more work needs to be done.	1398	* more work needs to be done.
609	*/	1399	*/
610	if (target_offset == reloc->presumed_offset)	1400	if (!DBG_FORCE_RELOC &&
		1401	gen8_canonical_addr(target->node.start) == reloc->presumed_offset)
611	return 0;	1402	return 0;
612		1403
613	/* Check that the relocation address is valid... */	1404	/* Check that the relocation address is valid... */
614	if (unlikely(reloc->offset >	1405	if (unlikely(reloc->offset >
615	obj->base.size - (cache->use_64bit_reloc ? 8 : 4))) {	1406	vma->size - (eb->reloc_cache.use_64bit_reloc ? 8 : 4))) {
616	DRM_DEBUG("Relocation beyond object bounds: "	1407	DRM_DEBUG("Relocation beyond object bounds: "
617	"obj %p target %d offset %d size %d.\n",	1408	"target %d offset %d size %d.\n",
618	obj, reloc->target_handle,	1409	reloc->target_handle,
619	(int) reloc->offset,	1410	(int)reloc->offset,
620	(int) obj->base.size);	1411	(int)vma->size);
621	return -EINVAL;	1412	return -EINVAL;
622	}	1413	}
623	if (unlikely(reloc->offset & 3)) {	1414	if (unlikely(reloc->offset & 3)) {
624	DRM_DEBUG("Relocation not 4-byte aligned: "	1415	DRM_DEBUG("Relocation not 4-byte aligned: "
625	"obj %p target %d offset %d.\n",	1416	"target %d offset %d.\n",
626	obj, reloc->target_handle,	1417	reloc->target_handle,
627	(int) reloc->offset);	1418	(int)reloc->offset);
628	return -EINVAL;	1419	return -EINVAL;
629	}	1420	}
630		1421
631	ret = relocate_entry(obj, reloc, cache, target_offset);	1422	/*
632	if (ret)	1423	* If we write into the object, we need to force the synchronisation
633	return ret;	1424	* barrier, either with an asynchronous clflush or if we executed the
		1425	* patching using the GPU (though that should be serialised by the
		1426	* timeline). To be completely sure, and since we are required to
		1427	* do relocations we are already stalling, disable the user's opt
		1428	* of our synchronisation.
		1429	*/
		1430	vma->exec_entry->flags &= ~EXEC_OBJECT_ASYNC;
634		1431
635	/* and update the user's relocation entry */	1432	/* and update the user's relocation entry */
636	reloc->presumed_offset = target_offset;	1433	return relocate_entry(vma, reloc, eb, target);
637	return 0;
638	}	1434	}
639		1435
640	static int	1436	static int eb_relocate_vma(struct i915_execbuffer eb, struct i915_vma vma)
641	i915_gem_execbuffer_relocate_vma(struct i915_vma *vma,
642	struct eb_vmas *eb)
643	{	1437	{
644	#define N_RELOC(x) ((x) / sizeof(struct drm_i915_gem_relocation_entry))	1438	#define N_RELOC(x) ((x) / sizeof(struct drm_i915_gem_relocation_entry))
645	struct drm_i915_gem_relocation_entry stack_reloc[N_RELOC(512)];	1439	struct drm_i915_gem_relocation_entry stack[N_RELOC(512)];
646	struct drm_i915_gem_relocation_entry __user *user_relocs;	1440	struct drm_i915_gem_relocation_entry __user *urelocs;
647	struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;	1441	const struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;
648	struct reloc_cache cache;	1442	unsigned int remain;
649	int remain, ret = 0;
650
651	user_relocs = u64_to_user_ptr(entry->relocs_ptr);
652	reloc_cache_init(&cache, eb->i915);
653		1443
		1444	urelocs = u64_to_user_ptr(entry->relocs_ptr);
654	remain = entry->relocation_count;	1445	remain = entry->relocation_count;
655	while (remain) {	1446	if (unlikely(remain > N_RELOC(ULONG_MAX)))
656	struct drm_i915_gem_relocation_entry *r = stack_reloc;	1447	return -EINVAL;
657	unsigned long unwritten;
658	unsigned int count;
659		1448
660	count = min_t(unsigned int, remain, ARRAY_SIZE(stack_reloc));	1449	/*
661	remain -= count;	1450	* We must check that the entire relocation array is safe
		1451	* to read. However, if the array is not writable the user loses
		1452	* the updated relocation values.
		1453	*/
		1454	if (unlikely(!access_ok(VERIFY_READ, urelocs, remain*sizeof(urelocs))))
		1455	return -EFAULT;
		1456
		1457	do {
		1458	struct drm_i915_gem_relocation_entry *r = stack;
		1459	unsigned int count =
		1460	min_t(unsigned int, remain, ARRAY_SIZE(stack));
		1461	unsigned int copied;
662		1462
663	/* This is the fast path and we cannot handle a pagefault	1463	/*
		1464	* This is the fast path and we cannot handle a pagefault
664	* whilst holding the struct mutex lest the user pass in the	1465	* whilst holding the struct mutex lest the user pass in the
665	* relocations contained within a mmaped bo. For in such a case	1466	* relocations contained within a mmaped bo. For in such a case
666	* we, the page fault handler would call i915_gem_fault() and	1467	* we, the page fault handler would call i915_gem_fault() and
@@ -668,489 +1469,408 @@ i915_gem_execbuffer_relocate_vma(struct i915_vma *vma, · 1 unmodified line
668	* this is bad and so lockdep complains vehemently.	1469	* this is bad and so lockdep complains vehemently.
669	*/	1470	*/
670	pagefault_disable();	1471	pagefault_disable();
671	unwritten = __copy_from_user_inatomic(r, user_relocs, count*sizeof(r[0]));	1472	copied = __copy_from_user_inatomic(r, urelocs, count * sizeof(r[0]));
672	pagefault_enable();	1473	pagefault_enable();
673	if (unlikely(unwritten)) {	1474	if (unlikely(copied)) {
674	ret = -EFAULT;	1475	remain = -EFAULT;
675	goto out;	1476	goto out;
676	}	1477	}
677		1478
		1479	remain -= count;
678	do {	1480	do {
679	u64 offset = r->presumed_offset;	1481	u64 offset = eb_relocate_entry(eb, vma, r);
680		1482
681	ret = i915_gem_execbuffer_relocate_entry(vma->obj, eb, r, &cache);	1483	if (likely(offset == 0)) {
682	if (ret)	1484	} else if ((s64)offset < 0) {
		1485	remain = (int)offset;
683	goto out;	1486	goto out;
684		1487	} else {
685	if (r->presumed_offset != offset) {	1488	/*
686	pagefault_disable();	1489	* Note that reporting an error now
687	unwritten = __put_user(r->presumed_offset,	1490	* leaves everything in an inconsistent
688	&user_relocs->presumed_offset);	1491	* state as we have already changed
689	pagefault_enable();	1492	* the relocation value inside the
690	if (unlikely(unwritten)) {	1493	* object. As we have not changed the
691	/* Note that reporting an error now	1494	* reloc.presumed_offset or will not
692	* leaves everything in an inconsistent	1495	* change the execobject.offset, on the
693	* state as we have already changed	1496	* call we may not rewrite the value
694	* the relocation value inside the	1497	* inside the object, leaving it
695	* object. As we have not changed the	1498	* dangling and causing a GPU hang. Unless
696	* reloc.presumed_offset or will not	1499	* userspace dynamically rebuilds the
697	* change the execobject.offset, on the	1500	* relocations on each execbuf rather than
698	* call we may not rewrite the value	1501	* presume a static tree.
699	* inside the object, leaving it	1502	*
700	* dangling and causing a GPU hang.	1503	* We did previously check if the relocations
701	*/	1504	* were writable (access_ok), an error now
702	ret = -EFAULT;	1505	* would be a strange race with mprotect,
703	goto out;	1506	* having already demonstrated that we
704	}	1507	* can read from this userspace address.
		1508	*/
		1509	offset = gen8_canonical_addr(offset & ~UPDATE);
		1510	__put_user(offset,
		1511	&urelocs[r-stack].presumed_offset);
705	}	1512	}
706		1513	} while (r++, --count);
707	user_relocs++;	1514	urelocs += ARRAY_SIZE(stack);
708	r++;	1515	} while (remain);
709	} while (--count);
710	}
711
712	out:	1516	out:
713	reloc_cache_fini(&cache);	1517	reloc_cache_reset(&eb->reloc_cache);
714	return ret;	1518	return remain;
715	#undef N_RELOC
716	}	1519	}
717		1520
718	static int	1521	static int
719	i915_gem_execbuffer_relocate_vma_slow(struct i915_vma *vma,	1522	eb_relocate_vma_slow(struct i915_execbuffer eb, struct i915_vma vma)
720	struct eb_vmas *eb,
721	struct drm_i915_gem_relocation_entry *relocs)
722	{	1523	{
723	const struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;	1524	const struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;
724	struct reloc_cache cache;	1525	struct drm_i915_gem_relocation_entry *relocs =
725	int i, ret = 0;	1526	u64_to_ptr(typeof(*relocs), entry->relocs_ptr);
		1527	unsigned int i;
		1528	int err;
726		1529
727	reloc_cache_init(&cache, eb->i915);
728	for (i = 0; i < entry->relocation_count; i++) {	1530	for (i = 0; i < entry->relocation_count; i++) {
729	ret = i915_gem_execbuffer_relocate_entry(vma->obj, eb, &relocs[i], &cache);	1531	u64 offset = eb_relocate_entry(eb, vma, &relocs[i]);
730	if (ret)
731	break;
732	}
733	reloc_cache_fini(&cache);
734		1532
735	return ret;	1533	if ((s64)offset < 0) {
		1534	err = (int)offset;
		1535	goto err;
		1536	}
		1537	}
		1538	err = 0;
		1539	err:
		1540	reloc_cache_reset(&eb->reloc_cache);
		1541	return err;
736	}	1542	}
737		1543
738	static int	1544	static int check_relocations(const struct drm_i915_gem_exec_object2 *entry)
739	i915_gem_execbuffer_relocate(struct eb_vmas *eb)
740	{	1545	{
741	struct i915_vma *vma;	1546	const char __user addr, end;
742	int ret = 0;	1547	unsigned long size;
		1548	char __maybe_unused c;
743		1549
744	list_for_each_entry(vma, &eb->vmas, exec_list) {	1550	size = entry->relocation_count;
745	ret = i915_gem_execbuffer_relocate_vma(vma, eb);	1551	if (size == 0)
746	if (ret)	1552	return 0;
747	break;
748	}
749		1553
750	return ret;	1554	if (size > N_RELOC(ULONG_MAX))
751	}	1555	return -EINVAL;
752		1556
753	static bool only_mappable_for_reloc(unsigned int flags)	1557	addr = u64_to_user_ptr(entry->relocs_ptr);
754	{	1558	size *= sizeof(struct drm_i915_gem_relocation_entry);
755	return (flags & (EXEC_OBJECT_NEEDS_FENCE \| __EXEC_OBJECT_NEEDS_MAP)) ==	1559	if (!access_ok(VERIFY_READ, addr, size))
756	__EXEC_OBJECT_NEEDS_MAP;	1560	return -EFAULT;
		1561
		1562	end = addr + size;
		1563	for (; addr < end; addr += PAGE_SIZE) {
		1564	int err = __get_user(c, addr);
		1565	if (err)
		1566	return err;
		1567	}
		1568	return __get_user(c, end - 1);
757	}	1569	}
758		1570
759	static int	1571	static int eb_copy_relocations(const struct i915_execbuffer *eb)
760	i915_gem_execbuffer_reserve_vma(struct i915_vma *vma,
761	struct intel_engine_cs *engine,
762	bool *need_reloc)
763	{	1572	{
764	struct drm_i915_gem_object *obj = vma->obj;	1573	const unsigned int count = eb->buffer_count;
765	struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;	1574	unsigned int i;
766	uint64_t flags;	1575	int err;
767	int ret;
768		1576
769	flags = PIN_USER;	1577	for (i = 0; i < count; i++) {
770	if (entry->flags & EXEC_OBJECT_NEEDS_GTT)	1578	const unsigned int nreloc = eb->exec[i].relocation_count;
771	flags \|= PIN_GLOBAL;	1579	struct drm_i915_gem_relocation_entry __user *urelocs;
		1580	struct drm_i915_gem_relocation_entry *relocs;
		1581	unsigned long size;
		1582	unsigned long copied;
772		1583
773	if (!drm_mm_node_allocated(&vma->node)) {	1584	if (nreloc == 0)
774	/* Wa32bitGeneralStateOffset & Wa32bitInstructionBaseOffset,	1585	continue;
775	* limit address to the first 4GBs for unflagged objects.
776	*/
777	if ((entry->flags & EXEC_OBJECT_SUPPORTS_48B_ADDRESS) == 0)
778	flags \|= PIN_ZONE_4G;
779	if (entry->flags & __EXEC_OBJECT_NEEDS_MAP)
780	flags \|= PIN_GLOBAL \| PIN_MAPPABLE;
781	if (entry->flags & __EXEC_OBJECT_NEEDS_BIAS)
782	flags \|= BATCH_OFFSET_BIAS \| PIN_OFFSET_BIAS;
783	if (entry->flags & EXEC_OBJECT_PINNED)
784	flags \|= entry->offset \| PIN_OFFSET_FIXED;
785	if ((flags & PIN_MAPPABLE) == 0)
786	flags \|= PIN_HIGH;
787	}
788
789	ret = i915_vma_pin(vma,
790	entry->pad_to_size,
791	entry->alignment,
792	flags);
793	if ((ret == -ENOSPC \|\| ret == -E2BIG) &&
794	only_mappable_for_reloc(entry->flags))
795	ret = i915_vma_pin(vma,
796	entry->pad_to_size,
797	entry->alignment,
798	flags & ~PIN_MAPPABLE);
799	if (ret)
800	return ret;
801		1586
802	entry->flags \|= __EXEC_OBJECT_HAS_PIN;	1587	err = check_relocations(&eb->exec[i]);
		1588	if (err)
		1589	goto err;
803		1590
804	if (entry->flags & EXEC_OBJECT_NEEDS_FENCE) {	1591	urelocs = u64_to_user_ptr(eb->exec[i].relocs_ptr);
805	ret = i915_vma_get_fence(vma);	1592	size = nreloc * sizeof(*relocs);
806	if (ret)
807	return ret;
808		1593
809	if (i915_vma_pin_fence(vma))	1594	relocs = kvmalloc_array(size, 1, GFP_TEMPORARY);
810	entry->flags \|= __EXEC_OBJECT_HAS_FENCE;	1595	if (!relocs) {
811	}	1596	kvfree(relocs);
		1597	err = -ENOMEM;
		1598	goto err;
		1599	}
812		1600
813	if (entry->offset != vma->node.start) {	1601	/* copy_from_user is limited to < 4GiB */
814	entry->offset = vma->node.start;	1602	copied = 0;
815	*need_reloc = true;	1603	do {
816	}	1604	unsigned int len =
		1605	min_t(u64, BIT_ULL(31), size - copied);
		1606
		1607	if (__copy_from_user((char *)relocs + copied,
		1608	(char *)urelocs + copied,
		1609	len)) {
		1610	kvfree(relocs);
		1611	err = -EFAULT;
		1612	goto err;
		1613	}
		1614
		1615	copied += len;
		1616	} while (copied < size);
		1617
		1618	/*
		1619	* As we do not update the known relocation offsets after
		1620	* relocating (due to the complexities in lock handling),
		1621	* we need to mark them as invalid now so that we force the
		1622	* relocation processing next time. Just in case the target
		1623	* object is evicted and then rebound into its old
		1624	* presumed_offset before the next execbuffer - if that
		1625	* happened we would make the mistake of assuming that the
		1626	* relocations were valid.
		1627	*/
		1628	user_access_begin();
		1629	for (copied = 0; copied < nreloc; copied++)
		1630	unsafe_put_user(-1,
		1631	&urelocs[copied].presumed_offset,
		1632	end_user);
		1633	end_user:
		1634	user_access_end();
817		1635
818	if (entry->flags & EXEC_OBJECT_WRITE) {	1636	eb->exec[i].relocs_ptr = (uintptr_t)relocs;
819	obj->base.pending_read_domains = I915_GEM_DOMAIN_RENDER;
820	obj->base.pending_write_domain = I915_GEM_DOMAIN_RENDER;
821	}	1637	}
822		1638
823	return 0;	1639	return 0;
		1640
		1641	err:
		1642	while (i--) {
		1643	struct drm_i915_gem_relocation_entry *relocs =
		1644	u64_to_ptr(typeof(*relocs), eb->exec[i].relocs_ptr);
		1645	if (eb->exec[i].relocation_count)
		1646	kvfree(relocs);
		1647	}
		1648	return err;
824	}	1649	}
825		1650
826	static bool	1651	static int eb_prefault_relocations(const struct i915_execbuffer *eb)
827	need_reloc_mappable(struct i915_vma *vma)
828	{	1652	{
829	struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;	1653	const unsigned int count = eb->buffer_count;
830		1654	unsigned int i;
831	if (entry->relocation_count == 0)
832	return false;
833		1655
834	if (!i915_vma_is_ggtt(vma))	1656	if (unlikely(i915.prefault_disable))
835	return false;	1657	return 0;
836		1658
837	/* See also use_cpu_reloc() */	1659	for (i = 0; i < count; i++) {
838	if (HAS_LLC(to_i915(vma->obj->base.dev)))	1660	int err;
839	return false;
840		1661
841	if (vma->obj->base.write_domain == I915_GEM_DOMAIN_CPU)	1662	err = check_relocations(&eb->exec[i]);
842	return false;	1663	if (err)
		1664	return err;
		1665	}
843		1666
844	return true;	1667	return 0;
845	}	1668	}
846		1669
847	static bool	1670	static noinline int eb_relocate_slow(struct i915_execbuffer *eb)
848	eb_vma_misplaced(struct i915_vma *vma)
849	{	1671	{
850	struct drm_i915_gem_exec_object2 *entry = vma->exec_entry;	1672	struct drm_device *dev = &eb->i915->drm;
851		1673	bool have_copy = false;
852	WARN_ON(entry->flags & __EXEC_OBJECT_NEEDS_MAP &&	1674	struct i915_vma *vma;
853	!i915_vma_is_ggtt(vma));	1675	int err = 0;
854
855	if (entry->alignment && !IS_ALIGNED(vma->node.start, entry->alignment))
856	return true;
857
858	if (vma->node.size < entry->pad_to_size)
859	return true;
860
861	if (entry->flags & EXEC_OBJECT_PINNED &&
862	vma->node.start != entry->offset)
863	return true;
864
865	if (entry->flags & __EXEC_OBJECT_NEEDS_BIAS &&
866	vma->node.start < BATCH_OFFSET_BIAS)
867	return true;
868
869	/* avoid costly ping-pong once a batch bo ended up non-mappable */
870	if (entry->flags & __EXEC_OBJECT_NEEDS_MAP &&
871	!i915_vma_is_map_and_fenceable(vma))
872	return !only_mappable_for_reloc(entry->flags);
873		1676
874	if ((entry->flags & EXEC_OBJECT_SUPPORTS_48B_ADDRESS) == 0 &&	1677	repeat:
875	(vma->node.start + vma->node.size - 1) >> 32)	1678	if (signal_pending(current)) {
876	return true;	1679	err = -ERESTARTSYS;
		1680	goto out;
		1681	}
877		1682
878	return false;	1683	/* We may process another execbuffer during the unlock... */
879	}	1684	eb_reset_vmas(eb);
		1685	mutex_unlock(&dev->struct_mutex);
880		1686
881	static int	1687	/*
882	i915_gem_execbuffer_reserve(struct intel_engine_cs *engine,	1688	* We take 3 passes through the slowpatch.
883	struct list_head *vmas,
884	struct i915_gem_context *ctx,
885	bool *need_relocs)
886	{
887	struct drm_i915_gem_object *obj;
888	struct i915_vma *vma;
889	struct i915_address_space *vm;
890	struct list_head ordered_vmas;
891	struct list_head pinned_vmas;
892	bool has_fenced_gpu_access = INTEL_GEN(engine->i915) < 4;
893	bool needs_unfenced_map = INTEL_INFO(engine->i915)->unfenced_needs_alignment;
894	int retry;
895
896	vm = list_first_entry(vmas, struct i915_vma, exec_list)->vm;
897
898	INIT_LIST_HEAD(&ordered_vmas);
899	INIT_LIST_HEAD(&pinned_vmas);
900	while (!list_empty(vmas)) {
901	struct drm_i915_gem_exec_object2 *entry;
902	bool need_fence, need_mappable;
903
904	vma = list_first_entry(vmas, struct i915_vma, exec_list);
905	obj = vma->obj;
906	entry = vma->exec_entry;
907
908	if (ctx->flags & CONTEXT_NO_ZEROMAP)
909	entry->flags \|= __EXEC_OBJECT_NEEDS_BIAS;
910
911	if (!has_fenced_gpu_access)
912	entry->flags &= ~EXEC_OBJECT_NEEDS_FENCE;
913	need_fence =
914	(entry->flags & EXEC_OBJECT_NEEDS_FENCE \|\|
915	needs_unfenced_map) &&
916	i915_gem_object_is_tiled(obj);
917	need_mappable = need_fence \|\| need_reloc_mappable(vma);
918
919	if (entry->flags & EXEC_OBJECT_PINNED)
920	list_move_tail(&vma->exec_list, &pinned_vmas);
921	else if (need_mappable) {
922	entry->flags \|= __EXEC_OBJECT_NEEDS_MAP;
923	list_move(&vma->exec_list, &ordered_vmas);
924	} else
925	list_move_tail(&vma->exec_list, &ordered_vmas);
926
927	obj->base.pending_read_domains = I915_GEM_GPU_DOMAINS & ~I915_GEM_DOMAIN_COMMAND;
928	obj->base.pending_write_domain = 0;
929	}
930	list_splice(&ordered_vmas, vmas);
931	list_splice(&pinned_vmas, vmas);
932
933	/* Attempt to pin all of the buffers into the GTT.
934	* This is done in 3 phases:
935	*	1689	*
936	* 1a. Unbind all objects that do not match the GTT constraints for	1690	* 1 - we try to just prefault all the user relocation entries and
937	* the execbuffer (fenceable, mappable, alignment etc).	1691	* then attempt to reuse the atomic pagefault disabled fast path again.
938	* 1b. Increment pin count for already bound objects.
939	* 2. Bind new objects.
940	* 3. Decrement pin count.
941	*	1692	*
942	* This avoid unnecessary unbinding of later objects in order to make	1693	* 2 - we copy the user entries to a local buffer here outside of the
943	* room for the earlier objects unless we need to defragment.	1694	* local and allow ourselves to wait upon any rendering before
		1695	* relocations
		1696	*
		1697	* 3 - we already have a local copy of the relocation entries, but
		1698	* were interrupted (EAGAIN) whilst waiting for the objects, try again.
944	*/	1699	*/
945	retry = 0;	1700	if (!err) {
946	do {	1701	err = eb_prefault_relocations(eb);
947	int ret = 0;	1702	} else if (!have_copy) {
948		1703	err = eb_copy_relocations(eb);
949	/* Unbind any ill-fitting objects or pin. */	1704	have_copy = err == 0;
950	list_for_each_entry(vma, vmas, exec_list) {	1705	} else {
951	if (!drm_mm_node_allocated(&vma->node))	1706	cond_resched();
952	continue;	1707	err = 0;
953		1708	}
954	if (eb_vma_misplaced(vma))	1709	if (err) {
955	ret = i915_vma_unbind(vma);	1710	mutex_lock(&dev->struct_mutex);
956	else	1711	goto out;
957	ret = i915_gem_execbuffer_reserve_vma(vma,	1712	}
958	engine,
959	need_relocs);
960	if (ret)
961	goto err;
962	}
963
964	/* Bind fresh objects */
965	list_for_each_entry(vma, vmas, exec_list) {
966	if (drm_mm_node_allocated(&vma->node))
967	continue;
968
969	ret = i915_gem_execbuffer_reserve_vma(vma, engine,
970	need_relocs);
971	if (ret)
972	goto err;
973	}
974
975	err:
976	if (ret != -ENOSPC \|\| retry++)
977	return ret;
978
979	/* Decrement pin count for bound objects */
980	list_for_each_entry(vma, vmas, exec_list)
981	i915_gem_execbuffer_unreserve_vma(vma);
982		1713
983	ret = i915_gem_evict_vm(vm, true);	1714	/* A frequent cause for EAGAIN are currently unavailable client pages */
984	if (ret)	1715	flush_workqueue(eb->i915->mm.userptr_wq);
985	return ret;
986	} while (1);
987	}
988		1716
989	static int	1717	err = i915_mutex_lock_interruptible(dev);
990	i915_gem_execbuffer_relocate_slow(struct drm_device *dev,	1718	if (err) {
991	struct drm_i915_gem_execbuffer2 *args,	1719	mutex_lock(&dev->struct_mutex);
992	struct drm_file *file,	1720	goto out;
993	struct intel_engine_cs *engine,	1721	}
994	struct eb_vmas *eb,
995	struct drm_i915_gem_exec_object2 *exec,
996	struct i915_gem_context *ctx)
997	{
998	struct drm_i915_gem_relocation_entry *reloc;
999	struct i915_address_space *vm;
1000	struct i915_vma *vma;
1001	bool need_relocs;
1002	int *reloc_offset;
1003	int i, total, ret;
1004	unsigned count = args->buffer_count;
1005		1722
1006	vm = list_first_entry(&eb->vmas, struct i915_vma, exec_list)->vm;	1723	/* reacquire the objects */
		1724	err = eb_lookup_vmas(eb);
		1725	if (err)
		1726	goto err;
1007		1727
1008	/* We may process another execbuffer during the unlock... */	1728	list_for_each_entry(vma, &eb->relocs, reloc_link) {
1009	while (!list_empty(&eb->vmas)) {	1729	if (!have_copy) {
1010	vma = list_first_entry(&eb->vmas, struct i915_vma, exec_list);	1730	pagefault_disable();
1011	list_del_init(&vma->exec_list);	1731	err = eb_relocate_vma(eb, vma);
1012	i915_gem_execbuffer_unreserve_vma(vma);	1732	pagefault_enable();
1013	i915_vma_put(vma);	1733	if (err)
		1734	goto repeat;
		1735	} else {
		1736	err = eb_relocate_vma_slow(eb, vma);
		1737	if (err)
		1738	goto err;
		1739	}
1014	}	1740	}
1015		1741
1016	mutex_unlock(&dev->struct_mutex);	1742	/*
		1743	* Leave the user relocations as are, this is the painfully slow path,
		1744	* and we want to avoid the complication of dropping the lock whilst
		1745	* having buffers reserved in the aperture and so causing spurious
		1746	* ENOSPC for random operations.
		1747	*/
1017		1748
1018	total = 0;	1749	err:
1019	for (i = 0; i < count; i++)	1750	if (err == -EAGAIN)
1020	total += exec[i].relocation_count;	1751	goto repeat;
1021		1752
1022	reloc_offset = kvmalloc_array(count, sizeof(*reloc_offset), GFP_KERNEL);	1753	out:
1023	reloc = kvmalloc_array(total, sizeof(*reloc), GFP_KERNEL);	1754	if (have_copy) {
1024	if (reloc == NULL \|\| reloc_offset == NULL) {	1755	const unsigned int count = eb->buffer_count;
1025	kvfree(reloc);	1756	unsigned int i;
1026	kvfree(reloc_offset);
1027	mutex_lock(&dev->struct_mutex);
1028	return -ENOMEM;
1029	}
1030		1757
1031	total = 0;	1758	for (i = 0; i < count; i++) {
1032	for (i = 0; i < count; i++) {	1759	const struct drm_i915_gem_exec_object2 *entry =
1033	struct drm_i915_gem_relocation_entry __user *user_relocs;	1760	&eb->exec[i];
1034	u64 invalid_offset = (u64)-1;	1761	struct drm_i915_gem_relocation_entry *relocs;
1035	int j;
1036		1762
1037	user_relocs = u64_to_user_ptr(exec[i].relocs_ptr);	1763	if (!entry->relocation_count)
		1764	continue;
1038		1765
1039	if (copy_from_user(reloc+total, user_relocs,	1766	relocs = u64_to_ptr(typeof(*relocs), entry->relocs_ptr);
1040	exec[i].relocation_count * sizeof(*reloc))) {	1767	kvfree(relocs);
1041	ret = -EFAULT;
1042	mutex_lock(&dev->struct_mutex);
1043	goto err;
1044	}	1768	}
		1769	}
1045		1770
1046	/* As we do not update the known relocation offsets after	1771	return err ?: have_copy;
1047	* relocating (due to the complexities in lock handling),	1772	}
1048	* we need to mark them as invalid now so that we force the
1049	* relocation processing next time. Just in case the target
1050	* object is evicted and then rebound into its old
1051	* presumed_offset before the next execbuffer - if that
1052	* happened we would make the mistake of assuming that the
1053	* relocations were valid.
1054	*/
1055	for (j = 0; j < exec[i].relocation_count; j++) {
1056	if (__copy_to_user(&user_relocs[j].presumed_offset,
1057	&invalid_offset,
1058	sizeof(invalid_offset))) {
1059	ret = -EFAULT;
1060	mutex_lock(&dev->struct_mutex);
1061	goto err;
1062	}
1063	}
1064		1773
1065	reloc_offset[i] = total;	1774	static int eb_relocate(struct i915_execbuffer *eb)
1066	total += exec[i].relocation_count;	1775	{
1067	}	1776	if (eb_lookup_vmas(eb))
		1777	goto slow;
1068		1778
1069	ret = i915_mutex_lock_interruptible(dev);	1779	/* The objects are in their final locations, apply the relocations. */
1070	if (ret) {	1780	if (eb->args->flags & __EXEC_HAS_RELOC) {
1071	mutex_lock(&dev->struct_mutex);	1781	struct i915_vma *vma;
1072	goto err;	1782
		1783	list_for_each_entry(vma, &eb->relocs, reloc_link) {
		1784	if (eb_relocate_vma(eb, vma))
		1785	goto slow;
		1786	}
1073	}	1787	}
1074		1788
1075	/* reacquire the objects */	1789	return 0;
1076	eb_reset(eb);
1077	ret = eb_lookup_vmas(eb, exec, args, vm, file);
1078	if (ret)
1079	goto err;
1080		1790
1081	need_relocs = (args->flags & I915_EXEC_NO_RELOC) == 0;	1791	slow:
1082	ret = i915_gem_execbuffer_reserve(engine, &eb->vmas, ctx,	1792	return eb_relocate_slow(eb);
1083	&need_relocs);	1793	}
1084	if (ret)
1085	goto err;
1086		1794
1087	list_for_each_entry(vma, &eb->vmas, exec_list) {	1795	static void eb_export_fence(struct i915_vma *vma,
1088	int offset = vma->exec_entry - exec;	1796	struct drm_i915_gem_request *req,
1089	ret = i915_gem_execbuffer_relocate_vma_slow(vma, eb,	1797	unsigned int flags)
1090	reloc + reloc_offset[offset]);	1798	{
1091	if (ret)	1799	struct reservation_object *resv = vma->resv;
1092	goto err;
1093	}
1094		1800
1095	/* Leave the user relocations as are, this is the painfully slow path,	1801	/*
1096	* and we want to avoid the complication of dropping the lock whilst	1802	* Ignore errors from failing to allocate the new fence, we can't
1097	* having buffers reserved in the aperture and so causing spurious	1803	* handle an error right now. Worst case should be missed
1098	* ENOSPC for random operations.	1804	* synchronisation leading to rendering corruption.
1099	*/	1805	*/
1100		1806	reservation_object_lock(resv, NULL);
1101	err:	1807	if (flags & EXEC_OBJECT_WRITE)
1102	kvfree(reloc);	1808	reservation_object_add_excl_fence(resv, &req->fence);
1103	kvfree(reloc_offset);	1809	else if (reservation_object_reserve_shared(resv) == 0)
1104	return ret;	1810	reservation_object_add_shared_fence(resv, &req->fence);
		1811	reservation_object_unlock(resv);
1105	}	1812	}
1106		1813
1107	static int	1814	static int eb_move_to_gpu(struct i915_execbuffer *eb)
1108	i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
1109	struct list_head *vmas)
1110	{	1815	{
1111	struct i915_vma *vma;	1816	const unsigned int count = eb->buffer_count;
1112	int ret;	1817	unsigned int i;
		1818	int err;
1113		1819
1114	list_for_each_entry(vma, vmas, exec_list) {	1820	for (i = 0; i < count; i++) {
		1821	const struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];
		1822	struct i915_vma *vma = exec_to_vma(entry);
1115	struct drm_i915_gem_object *obj = vma->obj;	1823	struct drm_i915_gem_object *obj = vma->obj;
1116		1824
1117	if (vma->exec_entry->flags & EXEC_OBJECT_CAPTURE) {	1825	if (entry->flags & EXEC_OBJECT_CAPTURE) {
1118	struct i915_gem_capture_list *capture;	1826	struct i915_gem_capture_list *capture;
1119		1827
1120	capture = kmalloc(sizeof(*capture), GFP_KERNEL);	1828	capture = kmalloc(sizeof(*capture), GFP_KERNEL);
1121	if (unlikely(!capture))	1829	if (unlikely(!capture))
1122	return -ENOMEM;	1830	return -ENOMEM;
1123		1831
1124	capture->next = req->capture_list;	1832	capture->next = eb->request->capture_list;
1125	capture->vma = vma;	1833	capture->vma = vma;
1126	req->capture_list = capture;	1834	eb->request->capture_list = capture;
1127	}	1835	}
1128		1836
1129	if (vma->exec_entry->flags & EXEC_OBJECT_ASYNC)	1837	if (entry->flags & EXEC_OBJECT_ASYNC)
1130	continue;	1838	goto skip_flushes;
1131		1839
1132	if (obj->base.write_domain & I915_GEM_DOMAIN_CPU) {	1840	if (unlikely(obj->cache_dirty && !obj->cache_coherent))
1133	i915_gem_clflush_object(obj, 0);	1841	i915_gem_clflush_object(obj, 0);
1134	obj->base.write_domain = 0;
1135	}
1136		1842
1137	ret = i915_gem_request_await_object	1843	err = i915_gem_request_await_object
1138	(req, obj, obj->base.pending_write_domain);	1844	(eb->request, obj, entry->flags & EXEC_OBJECT_WRITE);
1139	if (ret)	1845	if (err)
1140	return ret;	1846	return err;
		1847
		1848	skip_flushes:
		1849	i915_vma_move_to_active(vma, eb->request, entry->flags);
		1850	__eb_unreserve_vma(vma, entry);
		1851	vma->exec_entry = NULL;
		1852	}
		1853
		1854	for (i = 0; i < count; i++) {
		1855	const struct drm_i915_gem_exec_object2 *entry = &eb->exec[i];
		1856	struct i915_vma *vma = exec_to_vma(entry);
		1857
		1858	eb_export_fence(vma, eb->request, entry->flags);
		1859	if (unlikely(entry->flags & __EXEC_OBJECT_HAS_REF))
		1860	i915_vma_put(vma);
1141	}	1861	}
		1862	eb->exec = NULL;
1142		1863
1143	/* Unconditionally flush any chipset caches (for streaming writes). */	1864	/* Unconditionally flush any chipset caches (for streaming writes). */
1144	i915_gem_chipset_flush(req->engine->i915);	1865	i915_gem_chipset_flush(eb->i915);
1145		1866
1146	/* Unconditionally invalidate GPU caches and TLBs. */	1867	/* Unconditionally invalidate GPU caches and TLBs. */
1147	return req->engine->emit_flush(req, EMIT_INVALIDATE);	1868	return eb->engine->emit_flush(eb->request, EMIT_INVALIDATE);
1148	}	1869	}
1149		1870
1150	static bool	1871	static bool i915_gem_check_execbuffer(struct drm_i915_gem_execbuffer2 *exec)
1151	i915_gem_check_execbuffer(struct drm_i915_gem_execbuffer2 *exec)
1152	{	1872	{
1153	if (exec->flags & __I915_EXEC_UNKNOWN_FLAGS)	1873	if (exec->flags & __I915_EXEC_ILLEGAL_FLAGS)
1154	return false;	1874	return false;
1155		1875
1156	/* Kernel clipping was a DRI1 misfeature */	1876	/* Kernel clipping was a DRI1 misfeature */
@@ -1170,107 +1890,6 @@ i915_gem_check_execbuffer(struct drm_i915_gem_execbuffer2 *exec) · 13 unmodified lines
1170	return true;	1890	return true;
1171	}	1891	}
1172		1892
1173	static int
1174	validate_exec_list(struct drm_device *dev,
1175	struct drm_i915_gem_exec_object2 *exec,
1176	int count)
1177	{
1178	unsigned relocs_total = 0;
1179	unsigned relocs_max = UINT_MAX / sizeof(struct drm_i915_gem_relocation_entry);
1180	unsigned invalid_flags;
1181	int i;
1182
1183	/* INTERNAL flags must not overlap with external ones */
1184	BUILD_BUG_ON(__EXEC_OBJECT_INTERNAL_FLAGS & ~__EXEC_OBJECT_UNKNOWN_FLAGS);
1185
1186	invalid_flags = __EXEC_OBJECT_UNKNOWN_FLAGS;
1187	if (USES_FULL_PPGTT(dev))
1188	invalid_flags \|= EXEC_OBJECT_NEEDS_GTT;
1189
1190	for (i = 0; i < count; i++) {
1191	char __user *ptr = u64_to_user_ptr(exec[i].relocs_ptr);
1192	int length; /* limited by fault_in_pages_readable() */
1193
1194	if (exec[i].flags & invalid_flags)
1195	return -EINVAL;
1196
1197	/* Offset can be used as input (EXEC_OBJECT_PINNED), reject
1198	* any non-page-aligned or non-canonical addresses.
1199	*/
1200	if (exec[i].flags & EXEC_OBJECT_PINNED) {
1201	if (exec[i].offset !=
1202	gen8_canonical_addr(exec[i].offset & PAGE_MASK))
1203	return -EINVAL;
1204	}
1205
1206	/* From drm_mm perspective address space is continuous,
1207	* so from this point we're always using non-canonical
1208	* form internally.
1209	*/
1210	exec[i].offset = gen8_noncanonical_addr(exec[i].offset);
1211
1212	if (exec[i].alignment && !is_power_of_2(exec[i].alignment))
1213	return -EINVAL;
1214
1215	/* pad_to_size was once a reserved field, so sanitize it */
1216	if (exec[i].flags & EXEC_OBJECT_PAD_TO_SIZE) {
1217	if (offset_in_page(exec[i].pad_to_size))
1218	return -EINVAL;
1219	} else {
1220	exec[i].pad_to_size = 0;
1221	}
1222
1223	/* First check for malicious input causing overflow in
1224	* the worst case where we need to allocate the entire
1225	* relocation tree as a single array.
1226	*/
1227	if (exec[i].relocation_count > relocs_max - relocs_total)
1228	return -EINVAL;
1229	relocs_total += exec[i].relocation_count;
1230
1231	length = exec[i].relocation_count *
1232	sizeof(struct drm_i915_gem_relocation_entry);
1233	/*
1234	* We must check that the entire relocation array is safe
1235	* to read, but since we may need to update the presumed
1236	* offsets during execution, check for full write access.
1237	*/
1238	if (!access_ok(VERIFY_WRITE, ptr, length))
1239	return -EFAULT;
1240
1241	if (likely(!i915.prefault_disable)) {
1242	if (fault_in_pages_readable(ptr, length))
1243	return -EFAULT;
1244	}
1245	}
1246
1247	return 0;
1248	}
1249
1250	static struct i915_gem_context *
1251	i915_gem_validate_context(struct drm_device dev, struct drm_file file,
1252	struct intel_engine_cs *engine, const u32 ctx_id)
1253	{
1254	struct i915_gem_context *ctx;
1255
1256	ctx = i915_gem_context_lookup(file->driver_priv, ctx_id);
1257	if (IS_ERR(ctx))
1258	return ctx;
1259
1260	if (i915_gem_context_is_banned(ctx)) {
1261	DRM_DEBUG("Context %u tried to submit while banned\n", ctx_id);
1262	return ERR_PTR(-EIO);
1263	}
1264
1265	return ctx;
1266	}
1267
1268	static bool gpu_write_needs_clflush(struct drm_i915_gem_object *obj)
1269	{
1270	return !(obj->cache_level == I915_CACHE_NONE \|\|
1271	obj->cache_level == I915_CACHE_WT);
1272	}
1273
1274	void i915_vma_move_to_active(struct i915_vma *vma,	1893	void i915_vma_move_to_active(struct i915_vma *vma,
1275	struct drm_i915_gem_request *req,	1894	struct drm_i915_gem_request *req,
1276	unsigned int flags)	1895	unsigned int flags)
@@ -1281,7 +1900,8 @@ void i915_vma_move_to_active(struct i915_vma *vma, · 4 unmodified lines
1281	lockdep_assert_held(&req->i915->drm.struct_mutex);	1900	lockdep_assert_held(&req->i915->drm.struct_mutex);
1282	GEM_BUG_ON(!drm_mm_node_allocated(&vma->node));	1901	GEM_BUG_ON(!drm_mm_node_allocated(&vma->node));
1283		1902
1284	/* Add a reference if we're newly entering the active list.	1903	/*
		1904	* Add a reference if we're newly entering the active list.
1285	* The order in which we add operations to the retirement queue is	1905	* The order in which we add operations to the retirement queue is
1286	* vital here: mark_active adds to the start of the callback list,	1906	* vital here: mark_active adds to the start of the callback list,
1287	* such that subsequent callbacks are called first. Therefore we	1907	* such that subsequent callbacks are called first. Therefore we
@@ -1294,61 +1914,22 @@ void i915_vma_move_to_active(struct i915_vma *vma, · 6 unmodified lines
1294	i915_gem_active_set(&vma->last_read[idx], req);	1914	i915_gem_active_set(&vma->last_read[idx], req);
1295	list_move_tail(&vma->vm_link, &vma->vm->active_list);	1915	list_move_tail(&vma->vm_link, &vma->vm->active_list);
1296		1916
		1917	obj->base.write_domain = 0;
1297	if (flags & EXEC_OBJECT_WRITE) {	1918	if (flags & EXEC_OBJECT_WRITE) {
		1919	obj->base.write_domain = I915_GEM_DOMAIN_RENDER;
		1920
1298	if (intel_fb_obj_invalidate(obj, ORIGIN_CS))	1921	if (intel_fb_obj_invalidate(obj, ORIGIN_CS))
1299	i915_gem_active_set(&obj->frontbuffer_write, req);	1922	i915_gem_active_set(&obj->frontbuffer_write, req);
1300		1923
1301	/* update for the implicit flush after a batch */	1924	obj->base.read_domains = 0;
1302	obj->base.write_domain &= ~I915_GEM_GPU_DOMAINS;
1303	if (!obj->cache_dirty && gpu_write_needs_clflush(obj))
1304	obj->cache_dirty = true;
1305	}	1925	}
		1926	obj->base.read_domains \|= I915_GEM_GPU_DOMAINS;
1306		1927
1307	if (flags & EXEC_OBJECT_NEEDS_FENCE)	1928	if (flags & EXEC_OBJECT_NEEDS_FENCE)
1308	i915_gem_active_set(&vma->last_fence, req);	1929	i915_gem_active_set(&vma->last_fence, req);
1309	}	1930	}
1310		1931
1311	static void eb_export_fence(struct drm_i915_gem_object *obj,	1932	static int i915_reset_gen7_sol_offsets(struct drm_i915_gem_request *req)
1312	struct drm_i915_gem_request *req,
1313	unsigned int flags)
1314	{
1315	struct reservation_object *resv = obj->resv;
1316
1317	/* Ignore errors from failing to allocate the new fence, we can't
1318	* handle an error right now. Worst case should be missed
1319	* synchronisation leading to rendering corruption.
1320	*/
1321	reservation_object_lock(resv, NULL);
1322	if (flags & EXEC_OBJECT_WRITE)
1323	reservation_object_add_excl_fence(resv, &req->fence);
1324	else if (reservation_object_reserve_shared(resv) == 0)
1325	reservation_object_add_shared_fence(resv, &req->fence);
1326	reservation_object_unlock(resv);
1327	}
1328
1329	static void
1330	i915_gem_execbuffer_move_to_active(struct list_head *vmas,
1331	struct drm_i915_gem_request *req)
1332	{
1333	struct i915_vma *vma;
1334
1335	list_for_each_entry(vma, vmas, exec_list) {
1336	struct drm_i915_gem_object *obj = vma->obj;
1337
1338	obj->base.write_domain = obj->base.pending_write_domain;
1339	if (obj->base.write_domain)
1340	vma->exec_entry->flags \|= EXEC_OBJECT_WRITE;
1341	else
1342	obj->base.pending_read_domains \|= obj->base.read_domains;
1343	obj->base.read_domains = obj->base.pending_read_domains;
1344
1345	i915_vma_move_to_active(vma, req, vma->exec_entry->flags);
1346	eb_export_fence(obj, req, vma->exec_entry->flags);
1347	}
1348	}
1349
1350	static int
1351	i915_reset_gen7_sol_offsets(struct drm_i915_gem_request *req)
1352	{	1933	{
1353	u32 *cs;	1934	u32 *cs;
1354	int i;	1935	int i;
@@ -1358,50 +1939,43 @@ i915_reset_gen7_sol_offsets(struct drm_i915_gem_request *req) · 3 unmodified lines
1358	return -EINVAL;	1939	return -EINVAL;
1359	}	1940	}
1360		1941
1361	cs = intel_ring_begin(req, 4 * 3);	1942	cs = intel_ring_begin(req, 4 * 2 + 2);
1362	if (IS_ERR(cs))	1943	if (IS_ERR(cs))
1363	return PTR_ERR(cs);	1944	return PTR_ERR(cs);
1364		1945
		1946	*cs++ = MI_LOAD_REGISTER_IMM(4);
1365	for (i = 0; i < 4; i++) {	1947	for (i = 0; i < 4; i++) {
1366	*cs++ = MI_LOAD_REGISTER_IMM(1);
1367	*cs++ = i915_mmio_reg_offset(GEN7_SO_WRITE_OFFSET(i));	1948	*cs++ = i915_mmio_reg_offset(GEN7_SO_WRITE_OFFSET(i));
1368	*cs++ = 0;	1949	*cs++ = 0;
1369	}	1950	}
1370		1951	*cs++ = MI_NOOP;
1371	intel_ring_advance(req, cs);	1952	intel_ring_advance(req, cs);
1372		1953
1373	return 0;	1954	return 0;
1374	}	1955	}
1375		1956
1376	static struct i915_vma *	1957	static struct i915_vma eb_parse(struct i915_execbuffer eb, bool is_master)
1377	i915_gem_execbuffer_parse(struct intel_engine_cs *engine,
1378	struct drm_i915_gem_exec_object2 *shadow_exec_entry,
1379	struct drm_i915_gem_object *batch_obj,
1380	struct eb_vmas *eb,
1381	u32 batch_start_offset,
1382	u32 batch_len,
1383	bool is_master)
1384	{	1958	{
1385	struct drm_i915_gem_object *shadow_batch_obj;	1959	struct drm_i915_gem_object *shadow_batch_obj;
1386	struct i915_vma *vma;	1960	struct i915_vma *vma;
1387	int ret;	1961	int err;
1388		1962
1389	shadow_batch_obj = i915_gem_batch_pool_get(&engine->batch_pool,	1963	shadow_batch_obj = i915_gem_batch_pool_get(&eb->engine->batch_pool,
1390	PAGE_ALIGN(batch_len));	1964	PAGE_ALIGN(eb->batch_len));
1391	if (IS_ERR(shadow_batch_obj))	1965	if (IS_ERR(shadow_batch_obj))
1392	return ERR_CAST(shadow_batch_obj);	1966	return ERR_CAST(shadow_batch_obj);
1393		1967
1394	ret = intel_engine_cmd_parser(engine,	1968	err = intel_engine_cmd_parser(eb->engine,
1395	batch_obj,	1969	eb->batch->obj,
1396	shadow_batch_obj,	1970	shadow_batch_obj,
1397	batch_start_offset,	1971	eb->batch_start_offset,
1398	batch_len,	1972	eb->batch_len,
1399	is_master);	1973	is_master);
1400	if (ret) {	1974	if (err) {
1401	if (ret == -EACCES) /* unhandled chained batch */	1975	if (err == -EACCES) /* unhandled chained batch */
1402	vma = NULL;	1976	vma = NULL;
1403	else	1977	else
1404	vma = ERR_PTR(ret);	1978	vma = ERR_PTR(err);
1405	goto out;	1979	goto out;
1406	}	1980	}
1407		1981
@@ -1409,12 +1983,11 @@ i915_gem_execbuffer_parse(struct intel_engine_cs *engine, · 1 unmodified line
1409	if (IS_ERR(vma))	1983	if (IS_ERR(vma))
1410	goto out;	1984	goto out;
1411		1985
1412	memset(shadow_exec_entry, 0, sizeof(*shadow_exec_entry));	1986	vma->exec_entry =
1413		1987	memset(&eb->exec[eb->buffer_count++],
1414	vma->exec_entry = shadow_exec_entry;	1988	0, sizeof(*vma->exec_entry));
1415	vma->exec_entry->flags = __EXEC_OBJECT_HAS_PIN;	1989	vma->exec_entry->flags = __EXEC_OBJECT_HAS_PIN \| __EXEC_OBJECT_HAS_REF;
1416	i915_gem_object_get(shadow_batch_obj);	1990	__exec_to_vma(vma->exec_entry) = (uintptr_t)i915_vma_get(vma);
1417	list_add_tail(&vma->exec_list, &eb->vmas);
1418		1991
1419	out:	1992	out:
1420	i915_gem_object_unpin_pages(shadow_batch_obj);	1993	i915_gem_object_unpin_pages(shadow_batch_obj);
@@ -1422,54 +1995,37 @@ out: · 1 unmodified line
1422	}	1995	}
1423		1996
1424	static void	1997	static void
1425	add_to_client(struct drm_i915_gem_request *req,	1998	add_to_client(struct drm_i915_gem_request req, struct drm_file file)
1426	struct drm_file *file)
1427	{	1999	{
1428	req->file_priv = file->driver_priv;	2000	req->file_priv = file->driver_priv;
1429	list_add_tail(&req->client_link, &req->file_priv->mm.request_list);	2001	list_add_tail(&req->client_link, &req->file_priv->mm.request_list);
1430	}	2002	}
1431		2003
1432	static int	2004	static int eb_submit(struct i915_execbuffer *eb)
1433	execbuf_submit(struct i915_execbuffer_params *params,
1434	struct drm_i915_gem_execbuffer2 *args,
1435	struct list_head *vmas)
1436	{	2005	{
1437	u64 exec_start, exec_len;	2006	int err;
1438	int ret;
1439
1440	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
1441	if (ret)
1442	return ret;
1443		2007
1444	ret = i915_switch_context(params->request);	2008	err = eb_move_to_gpu(eb);
1445	if (ret)	2009	if (err)
1446	return ret;	2010	return err;
1447		2011
1448	if (args->flags & I915_EXEC_CONSTANTS_MASK) {	2012	err = i915_switch_context(eb->request);
1449	DRM_DEBUG("I915_EXEC_CONSTANTS_* unsupported\n");	2013	if (err)
1450	return -EINVAL;	2014	return err;
1451	}
1452		2015
1453	if (args->flags & I915_EXEC_GEN7_SOL_RESET) {	2016	if (eb->args->flags & I915_EXEC_GEN7_SOL_RESET) {
1454	ret = i915_reset_gen7_sol_offsets(params->request);	2017	err = i915_reset_gen7_sol_offsets(eb->request);
1455	if (ret)	2018	if (err)
1456	return ret;	2019	return err;
1457	}	2020	}
1458		2021
1459	exec_len = args->batch_len;	2022	err = eb->engine->emit_bb_start(eb->request,
1460	exec_start = params->batch->node.start +	2023	eb->batch->node.start +
1461	params->args_batch_start_offset;	2024	eb->batch_start_offset,
1462		2025	eb->batch_len,
1463	if (exec_len == 0)	2026	eb->batch_flags);
1464	exec_len = params->batch->size - params->args_batch_start_offset;	2027	if (err)
1465		2028	return err;
1466	ret = params->engine->emit_bb_start(params->request,
1467	exec_start, exec_len,
1468	params->dispatch_flags);
1469	if (ret)
1470	return ret;
1471
1472	i915_gem_execbuffer_move_to_active(vmas, params->request);
1473		2029
1474	return 0;	2030	return 0;
1475	}	2031	}
@@ -1551,66 +2107,62 @@ eb_select_engine(struct drm_i915_private *dev_priv, · 75 unmodified lines
1551	}	2107	}
1552		2108
1553	static int	2109	static int
1554	i915_gem_do_execbuffer(struct drm_device dev, void data,	2110	i915_gem_do_execbuffer(struct drm_device *dev,
1555	struct drm_file *file,	2111	struct drm_file *file,
1556	struct drm_i915_gem_execbuffer2 *args,	2112	struct drm_i915_gem_execbuffer2 *args,
1557	struct drm_i915_gem_exec_object2 *exec)	2113	struct drm_i915_gem_exec_object2 *exec)
1558	{	2114	{
1559	struct drm_i915_private *dev_priv = to_i915(dev);	2115	struct i915_execbuffer eb;
1560	struct i915_ggtt *ggtt = &dev_priv->ggtt;
1561	struct eb_vmas *eb;
1562	struct drm_i915_gem_exec_object2 shadow_exec_entry;
1563	struct intel_engine_cs *engine;
1564	struct i915_gem_context *ctx;
1565	struct i915_address_space *vm;
1566	struct i915_execbuffer_params params_master; /* XXX: will be removed later */
1567	struct i915_execbuffer_params *params = &params_master;
1568	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
1569	u32 dispatch_flags;
1570	struct dma_fence *in_fence = NULL;	2116	struct dma_fence *in_fence = NULL;
1571	struct sync_file *out_fence = NULL;	2117	struct sync_file *out_fence = NULL;
1572	int out_fence_fd = -1;	2118	int out_fence_fd = -1;
1573	int ret;	2119	int err;
1574	bool need_relocs;	2120
1575		2121	BUILD_BUG_ON(__EXEC_OBJECT_INTERNAL_FLAGS &
1576	if (!i915_gem_check_execbuffer(args))	2122	~__EXEC_OBJECT_UNKNOWN_FLAGS);
1577	return -EINVAL;	2123
1578		2124	eb.i915 = to_i915(dev);
1579	ret = validate_exec_list(dev, exec, args->buffer_count);	2125	eb.file = file;
1580	if (ret)	2126	eb.args = args;
1581	return ret;	2127	if (DBG_FORCE_RELOC \|\| !(args->flags & I915_EXEC_NO_RELOC))
1582		2128	args->flags \|= __EXEC_HAS_RELOC;
1583	dispatch_flags = 0;	2129	eb.exec = exec;
		2130	eb.ctx = NULL;
		2131	eb.invalid_flags = __EXEC_OBJECT_UNKNOWN_FLAGS;
		2132	if (USES_FULL_PPGTT(eb.i915))
		2133	eb.invalid_flags \|= EXEC_OBJECT_NEEDS_GTT;
		2134	reloc_cache_init(&eb.reloc_cache, eb.i915);
		2135
		2136	eb.buffer_count = args->buffer_count;
		2137	eb.batch_start_offset = args->batch_start_offset;
		2138	eb.batch_len = args->batch_len;
		2139
		2140	eb.batch_flags = 0;
1584	if (args->flags & I915_EXEC_SECURE) {	2141	if (args->flags & I915_EXEC_SECURE) {
1585	if (!drm_is_current_master(file) \|\| !capable(CAP_SYS_ADMIN))	2142	if (!drm_is_current_master(file) \|\| !capable(CAP_SYS_ADMIN))
1586	return -EPERM;	2143	return -EPERM;
1587		2144
1588	dispatch_flags \|= I915_DISPATCH_SECURE;	2145	eb.batch_flags \|= I915_DISPATCH_SECURE;
1589	}	2146	}
1590	if (args->flags & I915_EXEC_IS_PINNED)	2147	if (args->flags & I915_EXEC_IS_PINNED)
1591	dispatch_flags \|= I915_DISPATCH_PINNED;	2148	eb.batch_flags \|= I915_DISPATCH_PINNED;
1592
1593	engine = eb_select_engine(dev_priv, file, args);
1594	if (!engine)
1595	return -EINVAL;
1596		2149
1597	if (args->buffer_count < 1) {	2150	eb.engine = eb_select_engine(eb.i915, file, args);
1598	DRM_DEBUG("execbuf with %d buffers\n", args->buffer_count);	2151	if (!eb.engine)
1599	return -EINVAL;	2152	return -EINVAL;
1600	}
1601		2153
1602	if (args->flags & I915_EXEC_RESOURCE_STREAMER) {	2154	if (args->flags & I915_EXEC_RESOURCE_STREAMER) {
1603	if (!HAS_RESOURCE_STREAMER(dev_priv)) {	2155	if (!HAS_RESOURCE_STREAMER(eb.i915)) {
1604	DRM_DEBUG("RS is only allowed for Haswell, Gen8 and above\n");	2156	DRM_DEBUG("RS is only allowed for Haswell, Gen8 and above\n");
1605	return -EINVAL;	2157	return -EINVAL;
1606	}	2158	}
1607	if (engine->id != RCS) {	2159	if (eb.engine->id != RCS) {
1608	DRM_DEBUG("RS is not available on %s\n",	2160	DRM_DEBUG("RS is not available on %s\n",
1609	engine->name);	2161	eb.engine->name);
1610	return -EINVAL;	2162	return -EINVAL;
1611	}	2163	}
1612		2164
1613	dispatch_flags \|= I915_DISPATCH_RS;	2165	eb.batch_flags \|= I915_DISPATCH_RS;
1614	}	2166	}
1615		2167
1616	if (args->flags & I915_EXEC_FENCE_IN) {	2168	if (args->flags & I915_EXEC_FENCE_IN) {
@@ -1622,102 +2174,62 @@ i915_gem_do_execbuffer(struct drm_device dev, void data, · 5 unmodified lines
1622	if (args->flags & I915_EXEC_FENCE_OUT) {	2174	if (args->flags & I915_EXEC_FENCE_OUT) {
1623	out_fence_fd = get_unused_fd_flags(O_CLOEXEC);	2175	out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
1624	if (out_fence_fd < 0) {	2176	if (out_fence_fd < 0) {
1625	ret = out_fence_fd;	2177	err = out_fence_fd;
1626	goto err_in_fence;	2178	goto err_in_fence;
1627	}	2179	}
1628	}	2180	}
1629		2181
1630	/* Take a local wakeref for preparing to dispatch the execbuf as	2182	if (eb_create(&eb))
		2183	return -ENOMEM;
		2184
		2185	/*
		2186	* Take a local wakeref for preparing to dispatch the execbuf as
1631	* we expect to access the hardware fairly frequently in the	2187	* we expect to access the hardware fairly frequently in the
1632	* process. Upon first dispatch, we acquire another prolonged	2188	* process. Upon first dispatch, we acquire another prolonged
1633	* wakeref that we hold until the GPU has been idle for at least	2189	* wakeref that we hold until the GPU has been idle for at least
1634	* 100ms.	2190	* 100ms.
1635	*/	2191	*/
1636	intel_runtime_pm_get(dev_priv);	2192	intel_runtime_pm_get(eb.i915);
		2193	err = i915_mutex_lock_interruptible(dev);
		2194	if (err)
		2195	goto err_rpm;
1637		2196
1638	ret = i915_mutex_lock_interruptible(dev);	2197	err = eb_select_context(&eb);
1639	if (ret)	2198	if (unlikely(err))
1640	goto pre_mutex_err;	2199	goto err_unlock;
1641		2200
1642	ctx = i915_gem_validate_context(dev, file, engine, ctx_id);	2201	err = eb_relocate(&eb);
1643	if (IS_ERR(ctx)) {	2202	if (err)
1644	mutex_unlock(&dev->struct_mutex);	2203	/*
1645	ret = PTR_ERR(ctx);	2204	* If the user expects the execobject.offset and
1646	goto pre_mutex_err;	2205	* reloc.presumed_offset to be an exact match,
1647	}	2206	* as for using NO_RELOC, then we cannot update
1648		2207	* the execobject.offset until we have completed
1649	i915_gem_context_get(ctx);	2208	* relocation.
1650		2209	*/
1651	if (ctx->ppgtt)	2210	args->flags &= ~__EXEC_HAS_RELOC;
1652	vm = &ctx->ppgtt->base;	2211	if (err < 0)
1653	else	2212	goto err_vma;
1654	vm = &ggtt->base;
1655
1656	memset(&params_master, 0x00, sizeof(params_master));
1657
1658	eb = eb_create(dev_priv, args);
1659	if (eb == NULL) {
1660	i915_gem_context_put(ctx);
1661	mutex_unlock(&dev->struct_mutex);
1662	ret = -ENOMEM;
1663	goto pre_mutex_err;
1664	}
1665
1666	/* Look up object handles */
1667	ret = eb_lookup_vmas(eb, exec, args, vm, file);
1668	if (ret)
1669	goto err;
1670
1671	/* take note of the batch buffer before we might reorder the lists */
1672	params->batch = eb_get_batch(eb);
1673
1674	/* Move the objects en-masse into the GTT, evicting if necessary. */
1675	need_relocs = (args->flags & I915_EXEC_NO_RELOC) == 0;
1676	ret = i915_gem_execbuffer_reserve(engine, &eb->vmas, ctx,
1677	&need_relocs);
1678	if (ret)
1679	goto err;
1680
1681	/* The objects are in their final locations, apply the relocations. */
1682	if (need_relocs)
1683	ret = i915_gem_execbuffer_relocate(eb);
1684	if (ret) {
1685	if (ret == -EFAULT) {
1686	ret = i915_gem_execbuffer_relocate_slow(dev, args, file,
1687	engine,
1688	eb, exec, ctx);
1689	BUG_ON(!mutex_is_locked(&dev->struct_mutex));
1690	}
1691	if (ret)
1692	goto err;
1693	}
1694		2213
1695	/* Set the pending read domains for the batch buffer to COMMAND */	2214	if (unlikely(eb.batch->exec_entry->flags & EXEC_OBJECT_WRITE)) {
1696	if (params->batch->obj->base.pending_write_domain) {
1697	DRM_DEBUG("Attempting to use self-modifying batch buffer\n");	2215	DRM_DEBUG("Attempting to use self-modifying batch buffer\n");
1698	ret = -EINVAL;	2216	err = -EINVAL;
1699	goto err;	2217	goto err_vma;
1700	}	2218	}
1701	if (args->batch_start_offset > params->batch->size \|\|	2219	if (eb.batch_start_offset > eb.batch->size \|\|
1702	args->batch_len > params->batch->size - args->batch_start_offset) {	2220	eb.batch_len > eb.batch->size - eb.batch_start_offset) {
1703	DRM_DEBUG("Attempting to use out-of-bounds batch\n");	2221	DRM_DEBUG("Attempting to use out-of-bounds batch\n");
1704	ret = -EINVAL;	2222	err = -EINVAL;
1705	goto err;	2223	goto err_vma;
1706	}	2224	}
1707		2225
1708	params->args_batch_start_offset = args->batch_start_offset;	2226	if (eb.engine->needs_cmd_parser && eb.batch_len) {
1709	if (engine->needs_cmd_parser && args->batch_len) {
1710	struct i915_vma *vma;	2227	struct i915_vma *vma;
1711		2228
1712	vma = i915_gem_execbuffer_parse(engine, &shadow_exec_entry,	2229	vma = eb_parse(&eb, drm_is_current_master(file));
1713	params->batch->obj,
1714	eb,
1715	args->batch_start_offset,
1716	args->batch_len,
1717	drm_is_current_master(file));
1718	if (IS_ERR(vma)) {	2230	if (IS_ERR(vma)) {
1719	ret = PTR_ERR(vma);	2231	err = PTR_ERR(vma);
1720	goto err;	2232	goto err_vma;
1721	}	2233	}
1722		2234
1723	if (vma) {	2235	if (vma) {
@@ -1730,19 +2242,20 @@ i915_gem_do_execbuffer(struct drm_device dev, void data, · 6 unmodified lines
1730	* specifically don't want that set on batches the	2242	* specifically don't want that set on batches the
1731	* command parser has accepted.	2243	* command parser has accepted.
1732	*/	2244	*/
1733	dispatch_flags \|= I915_DISPATCH_SECURE;	2245	eb.batch_flags \|= I915_DISPATCH_SECURE;
1734	params->args_batch_start_offset = 0;	2246	eb.batch_start_offset = 0;
1735	params->batch = vma;	2247	eb.batch = vma;
1736	}	2248	}
1737	}	2249	}
1738		2250
1739	params->batch->obj->base.pending_read_domains \|= I915_GEM_DOMAIN_COMMAND;	2251	if (eb.batch_len == 0)
		2252	eb.batch_len = eb.batch->size - eb.batch_start_offset;
1740		2253
1741	/* snb/ivb/vlv conflate the "batch in ppgtt" bit with the "non-secure	2254	/*
		2255	* snb/ivb/vlv conflate the "batch in ppgtt" bit with the "non-secure
1742	* batch" bit. Hence we need to pin secure batches into the global gtt.	2256	* batch" bit. Hence we need to pin secure batches into the global gtt.
1743	* hsw should have this fixed, but bdw mucks it up again. */	2257	* hsw should have this fixed, but bdw mucks it up again. */
1744	if (dispatch_flags & I915_DISPATCH_SECURE) {	2258	if (eb.batch_flags & I915_DISPATCH_SECURE) {
1745	struct drm_i915_gem_object *obj = params->batch->obj;
1746	struct i915_vma *vma;	2259	struct i915_vma *vma;
1747		2260
1748	/*	2261	/*
@@ -1755,66 +2268,56 @@ i915_gem_do_execbuffer(struct drm_device dev, void data, · 6 unmodified lines
1755	* fitting due to fragmentation.	2268	* fitting due to fragmentation.
1756	* So this is actually safe.	2269	* So this is actually safe.
1757	*/	2270	*/
1758	vma = i915_gem_object_ggtt_pin(obj, NULL, 0, 0, 0);	2271	vma = i915_gem_object_ggtt_pin(eb.batch->obj, NULL, 0, 0, 0);
1759	if (IS_ERR(vma)) {	2272	if (IS_ERR(vma)) {
1760	ret = PTR_ERR(vma);	2273	err = PTR_ERR(vma);
1761	goto err;	2274	goto err_vma;
1762	}	2275	}
1763		2276
1764	params->batch = vma;	2277	eb.batch = vma;
1765	}	2278	}
1766		2279
		2280	/* All GPU relocation batches must be submitted prior to the user rq */
		2281	GEM_BUG_ON(eb.reloc_cache.rq);
		2282
1767	/* Allocate a request for this batch buffer nice and early. */	2283	/* Allocate a request for this batch buffer nice and early. */
1768	params->request = i915_gem_request_alloc(engine, ctx);	2284	eb.request = i915_gem_request_alloc(eb.engine, eb.ctx);
1769	if (IS_ERR(params->request)) {	2285	if (IS_ERR(eb.request)) {
1770	ret = PTR_ERR(params->request);	2286	err = PTR_ERR(eb.request);
1771	goto err_batch_unpin;	2287	goto err_batch_unpin;
1772	}	2288	}
1773		2289
1774	if (in_fence) {	2290	if (in_fence) {
1775	ret = i915_gem_request_await_dma_fence(params->request,	2291	err = i915_gem_request_await_dma_fence(eb.request, in_fence);
1776	in_fence);	2292	if (err < 0)
1777	if (ret < 0)
1778	goto err_request;	2293	goto err_request;
1779	}	2294	}
1780		2295
1781	if (out_fence_fd != -1) {	2296	if (out_fence_fd != -1) {
1782	out_fence = sync_file_create(&params->request->fence);	2297	out_fence = sync_file_create(&eb.request->fence);
1783	if (!out_fence) {	2298	if (!out_fence) {
1784	ret = -ENOMEM;	2299	err = -ENOMEM;
1785	goto err_request;	2300	goto err_request;
1786	}	2301	}
1787	}	2302	}
1788		2303
1789	/* Whilst this request exists, batch_obj will be on the	2304	/*
		2305	* Whilst this request exists, batch_obj will be on the
1790	* active_list, and so will hold the active reference. Only when this	2306	* active_list, and so will hold the active reference. Only when this
1791	* request is retired will the the batch_obj be moved onto the	2307	* request is retired will the the batch_obj be moved onto the
1792	* inactive_list and lose its active reference. Hence we do not need	2308	* inactive_list and lose its active reference. Hence we do not need
1793	* to explicitly hold another reference here.	2309	* to explicitly hold another reference here.
1794	*/	2310	*/
1795	params->request->batch = params->batch;	2311	eb.request->batch = eb.batch;
1796
1797	/*
1798	* Save assorted stuff away to pass through to *_submission().
1799	* NB: This data should be 'persistent' and not local as it will
1800	* kept around beyond the duration of the IOCTL once the GPU
1801	* scheduler arrives.
1802	*/
1803	params->dev = dev;
1804	params->file = file;
1805	params->engine = engine;
1806	params->dispatch_flags = dispatch_flags;
1807	params->ctx = ctx;
1808		2312
1809	trace_i915_gem_request_queue(params->request, dispatch_flags);	2313	trace_i915_gem_request_queue(eb.request, eb.batch_flags);
1810		2314	err = eb_submit(&eb);
1811	ret = execbuf_submit(params, args, &eb->vmas);
1812	err_request:	2315	err_request:
1813	__i915_add_request(params->request, ret == 0);	2316	__i915_add_request(eb.request, err == 0);
1814	add_to_client(params->request, file);	2317	add_to_client(eb.request, file);
1815		2318
1816	if (out_fence) {	2319	if (out_fence) {
1817	if (ret == 0) {	2320	if (err == 0) {
1818	fd_install(out_fence_fd, out_fence->file);	2321	fd_install(out_fence_fd, out_fence->file);
1819	args->rsvd2 &= GENMASK_ULL(0, 31); /* keep in-fence */	2322	args->rsvd2 &= GENMASK_ULL(0, 31); /* keep in-fence */
1820	args->rsvd2 \|= (u64)out_fence_fd << 32;	2323	args->rsvd2 \|= (u64)out_fence_fd << 32;
@@ -1825,30 +2328,22 @@ err_request: · 4 unmodified lines
1825	}	2328	}
1826		2329
1827	err_batch_unpin:	2330	err_batch_unpin:
1828	/*	2331	if (eb.batch_flags & I915_DISPATCH_SECURE)
1829	* FIXME: We crucially rely upon the active tracking for the (ppgtt)	2332	i915_vma_unpin(eb.batch);
1830	* batch vma for correctness. For less ugly and less fragility this	2333	err_vma:
1831	* needs to be adjusted to also track the ggtt batch vma properly as	2334	if (eb.exec)
1832	* active.	2335	eb_release_vmas(&eb);
1833	*/	2336	i915_gem_context_put(eb.ctx);
1834	if (dispatch_flags & I915_DISPATCH_SECURE)	2337	err_unlock:
1835	i915_vma_unpin(params->batch);
1836	err:
1837	/* the request owns the ref now */
1838	i915_gem_context_put(ctx);
1839	eb_destroy(eb);
1840
1841	mutex_unlock(&dev->struct_mutex);	2338	mutex_unlock(&dev->struct_mutex);
1842		2339	err_rpm:
1843	pre_mutex_err:	2340	intel_runtime_pm_put(eb.i915);
1844	/* intel_gpu_busy should also get a ref, so it will free when the device	2341	eb_destroy(&eb);
1845	* is really idle. */
1846	intel_runtime_pm_put(dev_priv);
1847	if (out_fence_fd != -1)	2342	if (out_fence_fd != -1)
1848	put_unused_fd(out_fence_fd);	2343	put_unused_fd(out_fence_fd);
1849	err_in_fence:	2344	err_in_fence:
1850	dma_fence_put(in_fence);	2345	dma_fence_put(in_fence);
1851	return ret;	2346	return err;
1852	}	2347	}
1853		2348
1854	/*	2349	/*
@@ -1859,20 +2354,38 @@ int · 4 unmodified lines
1859	i915_gem_execbuffer(struct drm_device dev, void data,	2354	i915_gem_execbuffer(struct drm_device dev, void data,
1860	struct drm_file *file)	2355	struct drm_file *file)
1861	{	2356	{
		2357	const size_t sz = sizeof(struct drm_i915_gem_exec_object2);
1862	struct drm_i915_gem_execbuffer *args = data;	2358	struct drm_i915_gem_execbuffer *args = data;
1863	struct drm_i915_gem_execbuffer2 exec2;	2359	struct drm_i915_gem_execbuffer2 exec2;
1864	struct drm_i915_gem_exec_object *exec_list = NULL;	2360	struct drm_i915_gem_exec_object *exec_list = NULL;
1865	struct drm_i915_gem_exec_object2 *exec2_list = NULL;	2361	struct drm_i915_gem_exec_object2 *exec2_list = NULL;
1866	int ret, i;	2362	unsigned int i;
		2363	int err;
1867		2364
1868	if (args->buffer_count < 1) {	2365	if (args->buffer_count < 1 \|\| args->buffer_count > SIZE_MAX / sz - 1) {
1869	DRM_DEBUG("execbuf with %d buffers\n", args->buffer_count);	2366	DRM_DEBUG("execbuf2 with %d buffers\n", args->buffer_count);
1870	return -EINVAL;	2367	return -EINVAL;
1871	}	2368	}
1872		2369
		2370	exec2.buffers_ptr = args->buffers_ptr;
		2371	exec2.buffer_count = args->buffer_count;
		2372	exec2.batch_start_offset = args->batch_start_offset;
		2373	exec2.batch_len = args->batch_len;
		2374	exec2.DR1 = args->DR1;
		2375	exec2.DR4 = args->DR4;
		2376	exec2.num_cliprects = args->num_cliprects;
		2377	exec2.cliprects_ptr = args->cliprects_ptr;
		2378	exec2.flags = I915_EXEC_RENDER;
		2379	i915_execbuffer2_set_context_id(exec2, 0);
		2380
		2381	if (!i915_gem_check_execbuffer(&exec2))
		2382	return -EINVAL;
		2383
1873	/* Copy in the exec list from userland */	2384	/* Copy in the exec list from userland */
1874	exec_list = kvmalloc_array(sizeof(*exec_list), args->buffer_count, GFP_KERNEL);	2385	exec_list = kvmalloc_array(args->buffer_count, sizeof(*exec_list),
1875	exec2_list = kvmalloc_array(sizeof(*exec2_list), args->buffer_count, GFP_KERNEL);	2386	__GFP_NOWARN \| GFP_TEMPORARY);
		2387	exec2_list = kvmalloc_array(args->buffer_count + 1, sz,
		2388	__GFP_NOWARN \| GFP_TEMPORARY);
1876	if (exec_list == NULL \|\| exec2_list == NULL) {	2389	if (exec_list == NULL \|\| exec2_list == NULL) {
1877	DRM_DEBUG("Failed to allocate exec list for %d buffers\n",	2390	DRM_DEBUG("Failed to allocate exec list for %d buffers\n",
1878	args->buffer_count);	2391	args->buffer_count);
@@ -1880,12 +2393,12 @@ i915_gem_execbuffer(struct drm_device dev, void data, · 1 unmodified line
1880	kvfree(exec2_list);	2393	kvfree(exec2_list);
1881	return -ENOMEM;	2394	return -ENOMEM;
1882	}	2395	}
1883	ret = copy_from_user(exec_list,	2396	err = copy_from_user(exec_list,
1884	u64_to_user_ptr(args->buffers_ptr),	2397	u64_to_user_ptr(args->buffers_ptr),
1885	sizeof(exec_list) args->buffer_count);	2398	sizeof(exec_list) args->buffer_count);
1886	if (ret != 0) {	2399	if (err) {
1887	DRM_DEBUG("copy %d exec entries failed %d\n",	2400	DRM_DEBUG("copy %d exec entries failed %d\n",
1888	args->buffer_count, ret);	2401	args->buffer_count, err);
1889	kvfree(exec_list);	2402	kvfree(exec_list);
1890	kvfree(exec2_list);	2403	kvfree(exec2_list);
1891	return -EFAULT;	2404	return -EFAULT;
@@ -1903,99 +2416,94 @@ i915_gem_execbuffer(struct drm_device dev, void data, · 11 unmodified lines
1903	exec2_list[i].flags = 0;	2416	exec2_list[i].flags = 0;
1904	}	2417	}
1905		2418
1906	exec2.buffers_ptr = args->buffers_ptr;	2419	err = i915_gem_do_execbuffer(dev, file, &exec2, exec2_list);
1907	exec2.buffer_count = args->buffer_count;	2420	if (exec2.flags & __EXEC_HAS_RELOC) {
1908	exec2.batch_start_offset = args->batch_start_offset;
1909	exec2.batch_len = args->batch_len;
1910	exec2.DR1 = args->DR1;
1911	exec2.DR4 = args->DR4;
1912	exec2.num_cliprects = args->num_cliprects;
1913	exec2.cliprects_ptr = args->cliprects_ptr;
1914	exec2.flags = I915_EXEC_RENDER;
1915	i915_execbuffer2_set_context_id(exec2, 0);
1916
1917	ret = i915_gem_do_execbuffer(dev, data, file, &exec2, exec2_list);
1918	if (!ret) {
1919	struct drm_i915_gem_exec_object __user *user_exec_list =	2421	struct drm_i915_gem_exec_object __user *user_exec_list =
1920	u64_to_user_ptr(args->buffers_ptr);	2422	u64_to_user_ptr(args->buffers_ptr);
1921		2423
1922	/* Copy the new buffer offsets back to the user's exec list. */	2424	/* Copy the new buffer offsets back to the user's exec list. */
1923	for (i = 0; i < args->buffer_count; i++) {	2425	for (i = 0; i < args->buffer_count; i++) {
		2426	if (!(exec2_list[i].offset & UPDATE))
		2427	continue;
		2428
1924	exec2_list[i].offset =	2429	exec2_list[i].offset =
1925	gen8_canonical_addr(exec2_list[i].offset);	2430	gen8_canonical_addr(exec2_list[i].offset & PIN_OFFSET_MASK);
1926	ret = __copy_to_user(&user_exec_list[i].offset,	2431	exec2_list[i].offset &= PIN_OFFSET_MASK;
1927	&exec2_list[i].offset,	2432	if (__copy_to_user(&user_exec_list[i].offset,
1928	sizeof(user_exec_list[i].offset));	2433	&exec2_list[i].offset,
1929	if (ret) {	2434	sizeof(user_exec_list[i].offset)))
1930	ret = -EFAULT;
1931	DRM_DEBUG("failed to copy %d exec entries "
1932	"back to user (%d)\n",
1933	args->buffer_count, ret);
1934	break;	2435	break;
1935	}
1936	}	2436	}
1937	}	2437	}
1938		2438
1939	kvfree(exec_list);	2439	kvfree(exec_list);
1940	kvfree(exec2_list);	2440	kvfree(exec2_list);
1941	return ret;	2441	return err;
1942	}	2442	}
1943		2443
1944	int	2444	int
1945	i915_gem_execbuffer2(struct drm_device dev, void data,	2445	i915_gem_execbuffer2(struct drm_device dev, void data,
1946	struct drm_file *file)	2446	struct drm_file *file)
1947	{	2447	{
		2448	const size_t sz = sizeof(struct drm_i915_gem_exec_object2);
1948	struct drm_i915_gem_execbuffer2 *args = data;	2449	struct drm_i915_gem_execbuffer2 *args = data;
1949	struct drm_i915_gem_exec_object2 *exec2_list = NULL;	2450	struct drm_i915_gem_exec_object2 *exec2_list;
1950	int ret;	2451	int err;
1951		2452
1952	if (args->buffer_count < 1 \|\|	2453	if (args->buffer_count < 1 \|\| args->buffer_count > SIZE_MAX / sz - 1) {
1953	args->buffer_count > UINT_MAX / sizeof(*exec2_list)) {
1954	DRM_DEBUG("execbuf2 with %d buffers\n", args->buffer_count);	2454	DRM_DEBUG("execbuf2 with %d buffers\n", args->buffer_count);
1955	return -EINVAL;	2455	return -EINVAL;
1956	}	2456	}
1957		2457
1958	exec2_list = kvmalloc_array(args->buffer_count,	2458	if (!i915_gem_check_execbuffer(args))
1959	sizeof(*exec2_list),	2459	return -EINVAL;
1960	GFP_TEMPORARY);	2460
		2461	/* Allocate an extra slot for use by the command parser */
		2462	exec2_list = kvmalloc_array(args->buffer_count + 1, sz,
		2463	__GFP_NOWARN \| GFP_TEMPORARY);
1961	if (exec2_list == NULL) {	2464	if (exec2_list == NULL) {
1962	DRM_DEBUG("Failed to allocate exec list for %d buffers\n",	2465	DRM_DEBUG("Failed to allocate exec list for %d buffers\n",
1963	args->buffer_count);	2466	args->buffer_count);
1964	return -ENOMEM;	2467	return -ENOMEM;
1965	}	2468	}
1966	ret = copy_from_user(exec2_list,	2469	if (copy_from_user(exec2_list,
1967	u64_to_user_ptr(args->buffers_ptr),	2470	u64_to_user_ptr(args->buffers_ptr),
1968	sizeof(exec2_list) args->buffer_count);	2471	sizeof(exec2_list) args->buffer_count)) {
1969	if (ret != 0) {	2472	DRM_DEBUG("copy %d exec entries failed\n", args->buffer_count);
1970	DRM_DEBUG("copy %d exec entries failed %d\n",
1971	args->buffer_count, ret);
1972	kvfree(exec2_list);	2473	kvfree(exec2_list);
1973	return -EFAULT;	2474	return -EFAULT;
1974	}	2475	}
1975		2476
1976	ret = i915_gem_do_execbuffer(dev, data, file, args, exec2_list);	2477	err = i915_gem_do_execbuffer(dev, file, args, exec2_list);
1977	if (!ret) {	2478
1978	/* Copy the new buffer offsets back to the user's exec list. */	2479	/*
		2480	* Now that we have begun execution of the batchbuffer, we ignore
		2481	* any new error after this point. Also given that we have already
		2482	* updated the associated relocations, we try to write out the current
		2483	* object locations irrespective of any error.
		2484	*/
		2485	if (args->flags & __EXEC_HAS_RELOC) {
1979	struct drm_i915_gem_exec_object2 __user *user_exec_list =	2486	struct drm_i915_gem_exec_object2 __user *user_exec_list =
1980	u64_to_user_ptr(args->buffers_ptr);	2487	u64_to_user_ptr(args->buffers_ptr);
1981	int i;	2488	unsigned int i;
1982		2489
		2490	/* Copy the new buffer offsets back to the user's exec list. */
		2491	user_access_begin();
1983	for (i = 0; i < args->buffer_count; i++) {	2492	for (i = 0; i < args->buffer_count; i++) {
		2493	if (!(exec2_list[i].offset & UPDATE))
		2494	continue;
		2495
1984	exec2_list[i].offset =	2496	exec2_list[i].offset =
1985	gen8_canonical_addr(exec2_list[i].offset);	2497	gen8_canonical_addr(exec2_list[i].offset & PIN_OFFSET_MASK);
1986	ret = __copy_to_user(&user_exec_list[i].offset,	2498	unsafe_put_user(exec2_list[i].offset,
1987	&exec2_list[i].offset,	2499	&user_exec_list[i].offset,
1988	sizeof(user_exec_list[i].offset));	2500	end_user);
1989	if (ret) {
1990	ret = -EFAULT;
1991	DRM_DEBUG("failed to copy %d exec entries "
1992	"back to user\n",
1993	args->buffer_count);
1994	break;
1995	}
1996	}	2501	}
		2502	end_user:
		2503	user_access_end();
1997	}	2504	}
1998		2505
		2506	args->flags &= ~__I915_EXEC_UNKNOWN_FLAGS;
1999	kvfree(exec2_list);	2507	kvfree(exec2_list);
2000	return ret;	2508	return err;
2001	}	2509	}

drivers/gpu/drm/i915/i915_gem_gtt.c+13−3

 	 * called on driver load and after a GPU reset, so you can place
 	 * workarounds here even if they get overwritten by GPU reset.
 	 */
-	/* WaIncreaseDefaultTLBEntries:chv,bdw,skl,bxt,kbl,glk */
+	/* WaIncreaseDefaultTLBEntries:chv,bdw,skl,bxt,kbl,glk,cfl */
 	if (IS_BROADWELL(dev_priv))
 		I915_WRITE(GEN8_L3_LRA_1_GPGPU, GEN8_L3_LRA_1_GPGPU_DEFAULT_VALUE_BDW);
 	else if (IS_CHERRYVIEW(dev_priv))
 void i915_ggtt_enable_guc(struct drm_i915_private *i915)
 {
+	GEM_BUG_ON(i915->ggtt.invalidate != gen6_ggtt_invalidate);
 	i915->ggtt.invalidate = guc_ggtt_invalidate;
 }
 void i915_ggtt_disable_guc(struct drm_i915_private *i915)
 {
-	if (i915->ggtt.invalidate == guc_ggtt_invalidate)
+	/* We should only be called after i915_ggtt_enable_guc() */
-		i915->ggtt.invalidate = gen6_ggtt_invalidate;
+	GEM_BUG_ON(i915->ggtt.invalidate != guc_ggtt_invalidate);
+	i915->ggtt.invalidate = gen6_ggtt_invalidate;
 }
 void i915_gem_restore_gtt_mappings(struct drm_i915_private *dev_priv)
 	if (err != -ENOSPC)
 		return err;
+	if (flags & PIN_NOEVICT)
+		return -ENOSPC;
 	err = i915_gem_evict_for_node(vm, node, flags);
 	if (err == 0)
 		err = drm_mm_reserve_node(&vm->mm, node);
 	if (err != -ENOSPC)
 		return err;
+	if (flags & PIN_NOEVICT)
+		return -ENOSPC;
 	/* No free space, pick a slot at random.
 	 *
 	 * There is a pathological case here using a GTT shared between

drivers/gpu/drm/i915/i915_gem_gtt.h+2−0

 	struct drm_i915_file_private *file;
 	struct list_head global_link;
 	u64 total;		/* size addr space maps (ex. 2GB for ggtt) */
+	u64 reserved;		/* size addr space reserved */
 	bool closed;
 #define PIN_MAPPABLE		BIT(1)
 #define PIN_ZONE_4G		BIT(2)
 #define PIN_NONFAULT		BIT(3)
+#define PIN_NOEVICT		BIT(4)
 #define PIN_MBZ			BIT(5) /* I915_VMA_PIN_OVERFLOW */
 #define PIN_GLOBAL		BIT(6) /* I915_VMA_GLOBAL_BIND */

drivers/gpu/drm/i915/i915_gem_internal.c+3−1

 	drm_gem_private_object_init(&i915->drm, &obj->base, size);
 	i915_gem_object_init(obj, &i915_gem_object_internal_ops);
-	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
 	obj->base.read_domains = I915_GEM_DOMAIN_CPU;
+	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
 	obj->cache_level = HAS_LLC(i915) ? I915_CACHE_LLC : I915_CACHE_NONE;
+	obj->cache_coherent = i915_gem_object_is_coherent(obj);
+	obj->cache_dirty = !obj->cache_coherent;
 	return obj;
 }

drivers/gpu/drm/i915/i915_gem_object.h+18−4

 	const struct drm_i915_gem_object_ops *ops;
-	/** List of VMAs backed by this object */
+	/**
+	 * @vma_list: List of VMAs backed by this object
+	 *
+	 * The VMA on this list are ordered by type, all GGTT vma are placed
+	 * at the head and all ppGTT vma are placed at the tail. The different
+	 * types of GGTT vma are unordered between themselves, use the
+	 * @vma_tree (which has a defined order between all VMA) to find an
+	 * exact match.
+	 */
 	struct list_head vma_list;
+	/**
+	 * @vma_tree: Ordered tree of VMAs backed by this object
+	 *
+	 * All VMA created for this object are placed in the @vma_tree for
+	 * fast retrieval via a binary search in i915_vma_instance().
+	 * They are also added to @vma_list for easy iteration.
+	 */
 	struct rb_root vma_tree;
+	struct i915_vma *vma_hashed;
 	/** Stolen memory for this object, instead of being backed by shmem. */
 	struct drm_mm_node *stolen;
 	 */
 	struct list_head userfault_link;
-	/** Used in execbuf to temporarily hold a ref */
-	struct list_head obj_exec_link;
 	struct list_head batch_pool_link;
 	I915_SELFTEST_DECLARE(struct list_head st_link);
 	unsigned long gt_ro:1;
 	unsigned int cache_level:3;
 	unsigned int cache_dirty:1;
+	unsigned int cache_coherent:1;
 	atomic_t frontbuffer_bits;
 	unsigned int frontbuffer_ggtt_origin; /* write once */

drivers/gpu/drm/i915/i915_gem_request.c+7−13

 		return false;
 	intel_engine_enable_signaling(to_request(fence), true);
-	return true;
+	return !i915_fence_signaled(fence);
 }
 static signed long i915_fence_wait(struct dma_fence *fence,
 i915_gem_request_await_request(struct drm_i915_gem_request *to,
 			       struct drm_i915_gem_request *from)
 {
-	u32 seqno;
 	int ret;
 	GEM_BUG_ON(to == from);
 		return ret < 0 ? ret : 0;
 	}
-	seqno = i915_gem_request_global_seqno(from);
+	if (to->engine->semaphore.sync_to) {
-	if (!seqno)
+		u32 seqno;
-		goto await_dma_fence;
-	if (!to->engine->semaphore.sync_to) {
+		GEM_BUG_ON(!from->engine->semaphore.signal);
-		if (!__i915_gem_request_started(from, seqno))
-			goto await_dma_fence;
-		if (!__i915_spin_request(from, seqno, TASK_INTERRUPTIBLE, 2))
+		seqno = i915_gem_request_global_seqno(from);
+		if (!seqno)
 			goto await_dma_fence;
-	} else {
-		GEM_BUG_ON(!from->engine->semaphore.signal);
 		if (seqno <= to->timeline->global_sync[from->engine->id])
 			return 0;
 			return ret;
 		to->timeline->global_sync[from->engine->id] = seqno;
+		return 0;
 	}
-	return 0;
 await_dma_fence:
 	ret = i915_sw_fence_await_dma_fence(&to->submit,
 					    &from->fence, 0,

drivers/gpu/drm/i915/i915_gem_shrinker.c+21−7

 static bool shrinker_lock(struct drm_i915_private *dev_priv, bool *unlock)
 {
 	switch (mutex_trylock_recursive(&dev_priv->drm.struct_mutex)) {
-	case MUTEX_TRYLOCK_FAILED:
-		return false;
-	case MUTEX_TRYLOCK_SUCCESS:
-		*unlock = true;
-		return true;
 	case MUTEX_TRYLOCK_RECURSIVE:
 		*unlock = false;
 		return true;
+	case MUTEX_TRYLOCK_FAILED:
+		do {
+			cpu_relax();
+			if (mutex_trylock(&dev_priv->drm.struct_mutex)) {
+	case MUTEX_TRYLOCK_SUCCESS:
+				*unlock = true;
+				return true;
+			}
+		} while (!need_resched());
+		return false;
 	}
 	BUG();
 					 sc->nr_to_scan - freed,
 					 I915_SHRINK_BOUND |
 					 I915_SHRINK_UNBOUND);
+	if (freed < sc->nr_to_scan && current_is_kswapd()) {
+		intel_runtime_pm_get(dev_priv);
+		freed += i915_gem_shrink(dev_priv,
+					 sc->nr_to_scan - freed,
+					 I915_SHRINK_ACTIVE |
+					 I915_SHRINK_BOUND |
+					 I915_SHRINK_UNBOUND);
+		intel_runtime_pm_put(dev_priv);
+	}
 	shrinker_unlock(dev_priv, unlock);

drivers/gpu/drm/i915/i915_gem_stolen.c+1−0

@@ -590,6 +590,7 @@ _i915_gem_object_create_stolen(struct drm_i915_private *dev_priv, · 589 unmodified lines
590	obj->stolen = stolen;	590	obj->stolen = stolen;
591	obj->base.read_domains = I915_GEM_DOMAIN_CPU \| I915_GEM_DOMAIN_GTT;	591	obj->base.read_domains = I915_GEM_DOMAIN_CPU \| I915_GEM_DOMAIN_GTT;
592	obj->cache_level = HAS_LLC(dev_priv) ? I915_CACHE_LLC : I915_CACHE_NONE;	592	obj->cache_level = HAS_LLC(dev_priv) ? I915_CACHE_LLC : I915_CACHE_NONE;
		593	obj->cache_coherent = true; /* assumptions! more like cache_oblivious */
593		594
594	if (i915_gem_object_pin_pages(obj))	595	if (i915_gem_object_pin_pages(obj))
595	goto cleanup;	596	goto cleanup;

drivers/gpu/drm/i915/i915_gem_userptr.c+19−5

 	mutex_unlock(&mm->i915->mm_lock);
 	INIT_WORK(&mm->work, __i915_mm_struct_free__worker);
-	schedule_work(&mm->work);
+	queue_work(mm->i915->mm.userptr_wq, &mm->work);
 }
 static void
 	get_task_struct(work->task);
 	INIT_WORK(&work->work, __i915_gem_userptr_get_pages_worker);
-	schedule_work(&work->work);
+	queue_work(to_i915(obj->base.dev)->mm.userptr_wq, &work->work);
 	return ERR_PTR(-EAGAIN);
 }
 	drm_gem_private_object_init(dev, &obj->base, args->user_size);
 	i915_gem_object_init(obj, &i915_gem_userptr_ops);
-	obj->cache_level = I915_CACHE_LLC;
-	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
 	obj->base.read_domains = I915_GEM_DOMAIN_CPU;
+	obj->base.write_domain = I915_GEM_DOMAIN_CPU;
+	obj->cache_level = I915_CACHE_LLC;
+	obj->cache_coherent = i915_gem_object_is_coherent(obj);
+	obj->cache_dirty = !obj->cache_coherent;
 	obj->userptr.ptr = args->user_ptr;
 	obj->userptr.read_only = !!(args->flags & I915_USERPTR_READ_ONLY);
 	return 0;
 }
-void i915_gem_init_userptr(struct drm_i915_private *dev_priv)
+int i915_gem_init_userptr(struct drm_i915_private *dev_priv)
 {
 	mutex_init(&dev_priv->mm_lock);
 	hash_init(dev_priv->mm_structs);
+	dev_priv->mm.userptr_wq =
+		alloc_workqueue("i915-userptr-acquire", WQ_HIGHPRI, 0);
+	if (!dev_priv->mm.userptr_wq)
+		return -ENOMEM;
+	return 0;
+}
+void i915_gem_cleanup_userptr(struct drm_i915_private *dev_priv)
+{
+	destroy_workqueue(dev_priv->mm.userptr_wq);
 }

drivers/gpu/drm/i915/i915_guc_submission.c+1−1

@@ -105,7 +105,7 @@ static int __reserve_doorbell(struct i915_guc_client *client) · 104 unmodified lines
105	end += offset;	105	end += offset;
106	}	106	}
107		107
108	id = find_next_zero_bit(client->guc->doorbell_bitmap, offset, end);	108	id = find_next_zero_bit(client->guc->doorbell_bitmap, end, offset);
109	if (id == end)	109	if (id == end)
110	return -ENOSPC;	110	return -ENOSPC;
111		111

drivers/gpu/drm/i915/i915_irq.c+4−2

 			I915_WRITE(SDEIIR, iir);
 			ret = IRQ_HANDLED;
-			if (HAS_PCH_SPT(dev_priv) || HAS_PCH_KBP(dev_priv))
+			if (HAS_PCH_SPT(dev_priv) || HAS_PCH_KBP(dev_priv) ||
+			    HAS_PCH_CNP(dev_priv))
 				spt_irq_handler(dev_priv, iir);
 			else
 				cpt_irq_handler(dev_priv, iir);
 		dev->driver->disable_vblank = gen8_disable_vblank;
 		if (IS_GEN9_LP(dev_priv))
 			dev_priv->display.hpd_irq_setup = bxt_hpd_irq_setup;
-		else if (HAS_PCH_SPT(dev_priv) || HAS_PCH_KBP(dev_priv))
+		else if (HAS_PCH_SPT(dev_priv) || HAS_PCH_KBP(dev_priv) ||
+			 HAS_PCH_CNP(dev_priv))
 			dev_priv->display.hpd_irq_setup = spt_hpd_irq_setup;
 		else
 			dev_priv->display.hpd_irq_setup = ilk_hpd_irq_setup;

drivers/gpu/drm/i915/i915_oa_bdw.c+5376−0

@@ -0,0 +1,5376 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#include <linux/sysfs.h>
	30
	31	#include "i915_drv.h"
	32	#include "i915_oa_bdw.h"
	33
	34	enum metric_set_id {
	35	METRIC_SET_ID_RENDER_BASIC = 1,
	36	METRIC_SET_ID_COMPUTE_BASIC,
	37	METRIC_SET_ID_RENDER_PIPE_PROFILE,
	38	METRIC_SET_ID_MEMORY_READS,
	39	METRIC_SET_ID_MEMORY_WRITES,
	40	METRIC_SET_ID_COMPUTE_EXTENDED,
	41	METRIC_SET_ID_COMPUTE_L3_CACHE,
	42	METRIC_SET_ID_DATA_PORT_READS_COALESCING,
	43	METRIC_SET_ID_DATA_PORT_WRITES_COALESCING,
	44	METRIC_SET_ID_HDC_AND_SF,
	45	METRIC_SET_ID_L3_1,
	46	METRIC_SET_ID_L3_2,
	47	METRIC_SET_ID_L3_3,
	48	METRIC_SET_ID_L3_4,
	49	METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND,
	50	METRIC_SET_ID_SAMPLER_1,
	51	METRIC_SET_ID_SAMPLER_2,
	52	METRIC_SET_ID_TDL_1,
	53	METRIC_SET_ID_TDL_2,
	54	METRIC_SET_ID_COMPUTE_EXTRA,
	55	METRIC_SET_ID_VME_PIPE,
	56	METRIC_SET_ID_TEST_OA,
	57	};
	58
	59	int i915_oa_n_builtin_metric_sets_bdw = 22;
	60
	61	static const struct i915_oa_reg b_counter_config_render_basic[] = {
	62	{ _MMIO(0x2710), 0x00000000 },
	63	{ _MMIO(0x2714), 0x00800000 },
	64	{ _MMIO(0x2720), 0x00000000 },
	65	{ _MMIO(0x2724), 0x00800000 },
	66	{ _MMIO(0x2740), 0x00000000 },
	67	};
	68
	69	static const struct i915_oa_reg flex_eu_config_render_basic[] = {
	70	{ _MMIO(0xe458), 0x00005004 },
	71	{ _MMIO(0xe558), 0x00010003 },
	72	{ _MMIO(0xe658), 0x00012011 },
	73	{ _MMIO(0xe758), 0x00015014 },
	74	{ _MMIO(0xe45c), 0x00051050 },
	75	{ _MMIO(0xe55c), 0x00053052 },
	76	{ _MMIO(0xe65c), 0x00055054 },
	77	};
	78
	79	static const struct i915_oa_reg mux_config_render_basic_0_slices_0x01[] = {
	80	{ _MMIO(0x9888), 0x143f000f },
	81	{ _MMIO(0x9888), 0x14110014 },
	82	{ _MMIO(0x9888), 0x14310014 },
	83	{ _MMIO(0x9888), 0x14bf000f },
	84	{ _MMIO(0x9888), 0x118a0317 },
	85	{ _MMIO(0x9888), 0x13837be0 },
	86	{ _MMIO(0x9888), 0x3b800060 },
	87	{ _MMIO(0x9888), 0x3d800005 },
	88	{ _MMIO(0x9888), 0x005c4000 },
	89	{ _MMIO(0x9888), 0x065c8000 },
	90	{ _MMIO(0x9888), 0x085cc000 },
	91	{ _MMIO(0x9888), 0x003d8000 },
	92	{ _MMIO(0x9888), 0x183d0800 },
	93	{ _MMIO(0x9888), 0x0a3f0023 },
	94	{ _MMIO(0x9888), 0x103f0000 },
	95	{ _MMIO(0x9888), 0x00584000 },
	96	{ _MMIO(0x9888), 0x08584000 },
	97	{ _MMIO(0x9888), 0x0a5a4000 },
	98	{ _MMIO(0x9888), 0x005b4000 },
	99	{ _MMIO(0x9888), 0x0e5b8000 },
	100	{ _MMIO(0x9888), 0x185b2400 },
	101	{ _MMIO(0x9888), 0x0a1d4000 },
	102	{ _MMIO(0x9888), 0x0c1f0800 },
	103	{ _MMIO(0x9888), 0x0e1faa00 },
	104	{ _MMIO(0x9888), 0x00384000 },
	105	{ _MMIO(0x9888), 0x0e384000 },
	106	{ _MMIO(0x9888), 0x16384000 },
	107	{ _MMIO(0x9888), 0x18380001 },
	108	{ _MMIO(0x9888), 0x00392000 },
	109	{ _MMIO(0x9888), 0x06398000 },
	110	{ _MMIO(0x9888), 0x0839a000 },
	111	{ _MMIO(0x9888), 0x0a391000 },
	112	{ _MMIO(0x9888), 0x00104000 },
	113	{ _MMIO(0x9888), 0x08104000 },
	114	{ _MMIO(0x9888), 0x00110030 },
	115	{ _MMIO(0x9888), 0x08110031 },
	116	{ _MMIO(0x9888), 0x10110000 },
	117	{ _MMIO(0x9888), 0x00134000 },
	118	{ _MMIO(0x9888), 0x16130020 },
	119	{ _MMIO(0x9888), 0x06308000 },
	120	{ _MMIO(0x9888), 0x08308000 },
	121	{ _MMIO(0x9888), 0x06311800 },
	122	{ _MMIO(0x9888), 0x08311880 },
	123	{ _MMIO(0x9888), 0x10310000 },
	124	{ _MMIO(0x9888), 0x0e334000 },
	125	{ _MMIO(0x9888), 0x16330080 },
	126	{ _MMIO(0x9888), 0x0abf1180 },
	127	{ _MMIO(0x9888), 0x10bf0000 },
	128	{ _MMIO(0x9888), 0x0ada8000 },
	129	{ _MMIO(0x9888), 0x0a9d8000 },
	130	{ _MMIO(0x9888), 0x109f0002 },
	131	{ _MMIO(0x9888), 0x0ab94000 },
	132	{ _MMIO(0x9888), 0x0d888000 },
	133	{ _MMIO(0x9888), 0x038a0380 },
	134	{ _MMIO(0x9888), 0x058a000e },
	135	{ _MMIO(0x9888), 0x018a8000 },
	136	{ _MMIO(0x9888), 0x0f8a8000 },
	137	{ _MMIO(0x9888), 0x198a8000 },
	138	{ _MMIO(0x9888), 0x1b8a00a0 },
	139	{ _MMIO(0x9888), 0x078a0000 },
	140	{ _MMIO(0x9888), 0x098a0000 },
	141	{ _MMIO(0x9888), 0x238b2820 },
	142	{ _MMIO(0x9888), 0x258b2550 },
	143	{ _MMIO(0x9888), 0x198c1000 },
	144	{ _MMIO(0x9888), 0x0b8d8000 },
	145	{ _MMIO(0x9888), 0x1f85aa80 },
	146	{ _MMIO(0x9888), 0x2185aaa0 },
	147	{ _MMIO(0x9888), 0x2385002a },
	148	{ _MMIO(0x9888), 0x0d831021 },
	149	{ _MMIO(0x9888), 0x0f83572f },
	150	{ _MMIO(0x9888), 0x01835680 },
	151	{ _MMIO(0x9888), 0x0383002c },
	152	{ _MMIO(0x9888), 0x11830000 },
	153	{ _MMIO(0x9888), 0x19835400 },
	154	{ _MMIO(0x9888), 0x1b830001 },
	155	{ _MMIO(0x9888), 0x05830000 },
	156	{ _MMIO(0x9888), 0x07834000 },
	157	{ _MMIO(0x9888), 0x09834000 },
	158	{ _MMIO(0x9888), 0x0184c000 },
	159	{ _MMIO(0x9888), 0x07848000 },
	160	{ _MMIO(0x9888), 0x0984c000 },
	161	{ _MMIO(0x9888), 0x0b84c000 },
	162	{ _MMIO(0x9888), 0x0d84c000 },
	163	{ _MMIO(0x9888), 0x0f84c000 },
	164	{ _MMIO(0x9888), 0x0384c000 },
	165	{ _MMIO(0x9888), 0x05844000 },
	166	{ _MMIO(0x9888), 0x1b80c137 },
	167	{ _MMIO(0x9888), 0x1d80c147 },
	168	{ _MMIO(0x9888), 0x21800000 },
	169	{ _MMIO(0x9888), 0x1180c000 },
	170	{ _MMIO(0x9888), 0x17808000 },
	171	{ _MMIO(0x9888), 0x1980c000 },
	172	{ _MMIO(0x9888), 0x1f80c000 },
	173	{ _MMIO(0x9888), 0x1380c000 },
	174	{ _MMIO(0x9888), 0x15804000 },
	175	{ _MMIO(0x9888), 0x4d801110 },
	176	{ _MMIO(0x9888), 0x4f800331 },
	177	{ _MMIO(0x9888), 0x43800802 },
	178	{ _MMIO(0x9888), 0x51800000 },
	179	{ _MMIO(0x9888), 0x45801465 },
	180	{ _MMIO(0x9888), 0x53801111 },
	181	{ _MMIO(0x9888), 0x478014a5 },
	182	{ _MMIO(0x9888), 0x31800000 },
	183	{ _MMIO(0x9888), 0x3f800ca5 },
	184	{ _MMIO(0x9888), 0x41800003 },
	185	};
	186
	187	static const struct i915_oa_reg mux_config_render_basic_1_slices_0x02[] = {
	188	{ _MMIO(0x9888), 0x143f000f },
	189	{ _MMIO(0x9888), 0x14bf000f },
	190	{ _MMIO(0x9888), 0x14910014 },
	191	{ _MMIO(0x9888), 0x14b10014 },
	192	{ _MMIO(0x9888), 0x118a0317 },
	193	{ _MMIO(0x9888), 0x13837be0 },
	194	{ _MMIO(0x9888), 0x3b800060 },
	195	{ _MMIO(0x9888), 0x3d800005 },
	196	{ _MMIO(0x9888), 0x0a3f0023 },
	197	{ _MMIO(0x9888), 0x103f0000 },
	198	{ _MMIO(0x9888), 0x0a5a4000 },
	199	{ _MMIO(0x9888), 0x0a1d4000 },
	200	{ _MMIO(0x9888), 0x0e1f8000 },
	201	{ _MMIO(0x9888), 0x0a391000 },
	202	{ _MMIO(0x9888), 0x00dc4000 },
	203	{ _MMIO(0x9888), 0x06dc8000 },
	204	{ _MMIO(0x9888), 0x08dcc000 },
	205	{ _MMIO(0x9888), 0x00bd8000 },
	206	{ _MMIO(0x9888), 0x18bd0800 },
	207	{ _MMIO(0x9888), 0x0abf1180 },
	208	{ _MMIO(0x9888), 0x10bf0000 },
	209	{ _MMIO(0x9888), 0x00d84000 },
	210	{ _MMIO(0x9888), 0x08d84000 },
	211	{ _MMIO(0x9888), 0x0ada8000 },
	212	{ _MMIO(0x9888), 0x00db4000 },
	213	{ _MMIO(0x9888), 0x0edb8000 },
	214	{ _MMIO(0x9888), 0x18db2400 },
	215	{ _MMIO(0x9888), 0x0a9d8000 },
	216	{ _MMIO(0x9888), 0x0c9f0800 },
	217	{ _MMIO(0x9888), 0x0e9f2a00 },
	218	{ _MMIO(0x9888), 0x109f0002 },
	219	{ _MMIO(0x9888), 0x00b84000 },
	220	{ _MMIO(0x9888), 0x0eb84000 },
	221	{ _MMIO(0x9888), 0x16b84000 },
	222	{ _MMIO(0x9888), 0x18b80001 },
	223	{ _MMIO(0x9888), 0x00b92000 },
	224	{ _MMIO(0x9888), 0x06b98000 },
	225	{ _MMIO(0x9888), 0x08b9a000 },
	226	{ _MMIO(0x9888), 0x0ab94000 },
	227	{ _MMIO(0x9888), 0x00904000 },
	228	{ _MMIO(0x9888), 0x08904000 },
	229	{ _MMIO(0x9888), 0x00910030 },
	230	{ _MMIO(0x9888), 0x08910031 },
	231	{ _MMIO(0x9888), 0x10910000 },
	232	{ _MMIO(0x9888), 0x00934000 },
	233	{ _MMIO(0x9888), 0x16930020 },
	234	{ _MMIO(0x9888), 0x06b08000 },
	235	{ _MMIO(0x9888), 0x08b08000 },
	236	{ _MMIO(0x9888), 0x06b11800 },
	237	{ _MMIO(0x9888), 0x08b11880 },
	238	{ _MMIO(0x9888), 0x10b10000 },
	239	{ _MMIO(0x9888), 0x0eb34000 },
	240	{ _MMIO(0x9888), 0x16b30080 },
	241	{ _MMIO(0x9888), 0x01888000 },
	242	{ _MMIO(0x9888), 0x0d88b800 },
	243	{ _MMIO(0x9888), 0x038a0380 },
	244	{ _MMIO(0x9888), 0x058a000e },
	245	{ _MMIO(0x9888), 0x1b8a0080 },
	246	{ _MMIO(0x9888), 0x078a0000 },
	247	{ _MMIO(0x9888), 0x098a0000 },
	248	{ _MMIO(0x9888), 0x238b2840 },
	249	{ _MMIO(0x9888), 0x258b26a0 },
	250	{ _MMIO(0x9888), 0x018c4000 },
	251	{ _MMIO(0x9888), 0x0f8c4000 },
	252	{ _MMIO(0x9888), 0x178c2000 },
	253	{ _MMIO(0x9888), 0x198c1100 },
	254	{ _MMIO(0x9888), 0x018d2000 },
	255	{ _MMIO(0x9888), 0x078d8000 },
	256	{ _MMIO(0x9888), 0x098da000 },
	257	{ _MMIO(0x9888), 0x0b8d8000 },
	258	{ _MMIO(0x9888), 0x1f85aa80 },
	259	{ _MMIO(0x9888), 0x2185aaa0 },
	260	{ _MMIO(0x9888), 0x2385002a },
	261	{ _MMIO(0x9888), 0x0d831021 },
	262	{ _MMIO(0x9888), 0x0f83572f },
	263	{ _MMIO(0x9888), 0x01835680 },
	264	{ _MMIO(0x9888), 0x0383002c },
	265	{ _MMIO(0x9888), 0x11830000 },
	266	{ _MMIO(0x9888), 0x19835400 },
	267	{ _MMIO(0x9888), 0x1b830001 },
	268	{ _MMIO(0x9888), 0x05830000 },
	269	{ _MMIO(0x9888), 0x07834000 },
	270	{ _MMIO(0x9888), 0x09834000 },
	271	{ _MMIO(0x9888), 0x0184c000 },
	272	{ _MMIO(0x9888), 0x07848000 },
	273	{ _MMIO(0x9888), 0x0984c000 },
	274	{ _MMIO(0x9888), 0x0b84c000 },
	275	{ _MMIO(0x9888), 0x0d84c000 },
	276	{ _MMIO(0x9888), 0x0f84c000 },
	277	{ _MMIO(0x9888), 0x0384c000 },
	278	{ _MMIO(0x9888), 0x05844000 },
	279	{ _MMIO(0x9888), 0x1b80c137 },
	280	{ _MMIO(0x9888), 0x1d80c147 },
	281	{ _MMIO(0x9888), 0x21800000 },
	282	{ _MMIO(0x9888), 0x1180c000 },
	283	{ _MMIO(0x9888), 0x17808000 },
	284	{ _MMIO(0x9888), 0x1980c000 },
	285	{ _MMIO(0x9888), 0x1f80c000 },
	286	{ _MMIO(0x9888), 0x1380c000 },
	287	{ _MMIO(0x9888), 0x15804000 },
	288	{ _MMIO(0x9888), 0x4d801550 },
	289	{ _MMIO(0x9888), 0x4f800331 },
	290	{ _MMIO(0x9888), 0x43800802 },
	291	{ _MMIO(0x9888), 0x51800400 },
	292	{ _MMIO(0x9888), 0x458004a1 },
	293	{ _MMIO(0x9888), 0x53805555 },
	294	{ _MMIO(0x9888), 0x47800421 },
	295	{ _MMIO(0x9888), 0x31800000 },
	296	{ _MMIO(0x9888), 0x3f801421 },
	297	{ _MMIO(0x9888), 0x41800845 },
	298	};
	299
	300	static int
	301	get_render_basic_mux_config(struct drm_i915_private *dev_priv,
	302	const struct i915_oa_reg **regs,
	303	int *lens)
	304	{
	305	int n = 0;
	306
	307	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 2);
	308	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 2);
	309
	310	if (INTEL_INFO(dev_priv)->sseu.slice_mask & 0x01) {
	311	regs[n] = mux_config_render_basic_0_slices_0x01;
	312	lens[n] = ARRAY_SIZE(mux_config_render_basic_0_slices_0x01);
	313	n++;
	314	}
	315	if (INTEL_INFO(dev_priv)->sseu.slice_mask & 0x02) {
	316	regs[n] = mux_config_render_basic_1_slices_0x02;
	317	lens[n] = ARRAY_SIZE(mux_config_render_basic_1_slices_0x02);
	318	n++;
	319	}
	320
	321	return n;
	322	}
	323
	324	static const struct i915_oa_reg b_counter_config_compute_basic[] = {
	325	{ _MMIO(0x2710), 0x00000000 },
	326	{ _MMIO(0x2714), 0x00800000 },
	327	{ _MMIO(0x2720), 0x00000000 },
	328	{ _MMIO(0x2724), 0x00800000 },
	329	{ _MMIO(0x2740), 0x00000000 },
	330	};
	331
	332	static const struct i915_oa_reg flex_eu_config_compute_basic[] = {
	333	{ _MMIO(0xe458), 0x00005004 },
	334	{ _MMIO(0xe558), 0x00000003 },
	335	{ _MMIO(0xe658), 0x00002001 },
	336	{ _MMIO(0xe758), 0x00778008 },
	337	{ _MMIO(0xe45c), 0x00088078 },
	338	{ _MMIO(0xe55c), 0x00808708 },
	339	{ _MMIO(0xe65c), 0x00a08908 },
	340	};
	341
	342	static const struct i915_oa_reg mux_config_compute_basic_0_slices_0x01[] = {
	343	{ _MMIO(0x9888), 0x105c00e0 },
	344	{ _MMIO(0x9888), 0x105800e0 },
	345	{ _MMIO(0x9888), 0x103800e0 },
	346	{ _MMIO(0x9888), 0x3580001a },
	347	{ _MMIO(0x9888), 0x3b800060 },
	348	{ _MMIO(0x9888), 0x3d800005 },
	349	{ _MMIO(0x9888), 0x065c2100 },
	350	{ _MMIO(0x9888), 0x0a5c0041 },
	351	{ _MMIO(0x9888), 0x0c5c6600 },
	352	{ _MMIO(0x9888), 0x005c6580 },
	353	{ _MMIO(0x9888), 0x085c8000 },
	354	{ _MMIO(0x9888), 0x0e5c8000 },
	355	{ _MMIO(0x9888), 0x00580042 },
	356	{ _MMIO(0x9888), 0x08582080 },
	357	{ _MMIO(0x9888), 0x0c58004c },
	358	{ _MMIO(0x9888), 0x0e582580 },
	359	{ _MMIO(0x9888), 0x005b4000 },
	360	{ _MMIO(0x9888), 0x185b1000 },
	361	{ _MMIO(0x9888), 0x1a5b0104 },
	362	{ _MMIO(0x9888), 0x0c1fa800 },
	363	{ _MMIO(0x9888), 0x0e1faa00 },
	364	{ _MMIO(0x9888), 0x101f02aa },
	365	{ _MMIO(0x9888), 0x08380042 },
	366	{ _MMIO(0x9888), 0x0a382080 },
	367	{ _MMIO(0x9888), 0x0e38404c },
	368	{ _MMIO(0x9888), 0x0238404b },
	369	{ _MMIO(0x9888), 0x00384000 },
	370	{ _MMIO(0x9888), 0x16380000 },
	371	{ _MMIO(0x9888), 0x18381145 },
	372	{ _MMIO(0x9888), 0x04380000 },
	373	{ _MMIO(0x9888), 0x0039a000 },
	374	{ _MMIO(0x9888), 0x06398000 },
	375	{ _MMIO(0x9888), 0x0839a000 },
	376	{ _MMIO(0x9888), 0x0a39a000 },
	377	{ _MMIO(0x9888), 0x0c39a000 },
	378	{ _MMIO(0x9888), 0x0e39a000 },
	379	{ _MMIO(0x9888), 0x02392000 },
	380	{ _MMIO(0x9888), 0x018a8000 },
	381	{ _MMIO(0x9888), 0x0f8a8000 },
	382	{ _MMIO(0x9888), 0x198a8000 },
	383	{ _MMIO(0x9888), 0x1b8aaaa0 },
	384	{ _MMIO(0x9888), 0x1d8a0002 },
	385	{ _MMIO(0x9888), 0x038a8000 },
	386	{ _MMIO(0x9888), 0x058a8000 },
	387	{ _MMIO(0x9888), 0x238b02a0 },
	388	{ _MMIO(0x9888), 0x258b5550 },
	389	{ _MMIO(0x9888), 0x278b0015 },
	390	{ _MMIO(0x9888), 0x1f850a80 },
	391	{ _MMIO(0x9888), 0x2185aaa0 },
	392	{ _MMIO(0x9888), 0x2385002a },
	393	{ _MMIO(0x9888), 0x01834000 },
	394	{ _MMIO(0x9888), 0x0f834000 },
	395	{ _MMIO(0x9888), 0x19835400 },
	396	{ _MMIO(0x9888), 0x1b830155 },
	397	{ _MMIO(0x9888), 0x03834000 },
	398	{ _MMIO(0x9888), 0x05834000 },
	399	{ _MMIO(0x9888), 0x0184c000 },
	400	{ _MMIO(0x9888), 0x07848000 },
	401	{ _MMIO(0x9888), 0x0984c000 },
	402	{ _MMIO(0x9888), 0x0b84c000 },
	403	{ _MMIO(0x9888), 0x0d84c000 },
	404	{ _MMIO(0x9888), 0x0f84c000 },
	405	{ _MMIO(0x9888), 0x03844000 },
	406	{ _MMIO(0x9888), 0x17808137 },
	407	{ _MMIO(0x9888), 0x1980c147 },
	408	{ _MMIO(0x9888), 0x1b80c0e5 },
	409	{ _MMIO(0x9888), 0x1d80c0e3 },
	410	{ _MMIO(0x9888), 0x21800000 },
	411	{ _MMIO(0x9888), 0x1180c000 },
	412	{ _MMIO(0x9888), 0x1f80c000 },
	413	{ _MMIO(0x9888), 0x13804000 },
	414	{ _MMIO(0x9888), 0x15800000 },
	415	{ _MMIO(0xd24), 0x00000000 },
	416	{ _MMIO(0x9888), 0x4d801000 },
	417	{ _MMIO(0x9888), 0x4f800111 },
	418	{ _MMIO(0x9888), 0x43800062 },
	419	{ _MMIO(0x9888), 0x51800000 },
	420	{ _MMIO(0x9888), 0x45800062 },
	421	{ _MMIO(0x9888), 0x53800000 },
	422	{ _MMIO(0x9888), 0x47800062 },
	423	{ _MMIO(0x9888), 0x31800000 },
	424	{ _MMIO(0x9888), 0x3f801062 },
	425	{ _MMIO(0x9888), 0x41801084 },
	426	};
	427
	428	static const struct i915_oa_reg mux_config_compute_basic_2_slices_0x02[] = {
	429	{ _MMIO(0x9888), 0x10dc00e0 },
	430	{ _MMIO(0x9888), 0x10d800e0 },
	431	{ _MMIO(0x9888), 0x10b800e0 },
	432	{ _MMIO(0x9888), 0x3580001a },
	433	{ _MMIO(0x9888), 0x3b800060 },
	434	{ _MMIO(0x9888), 0x3d800005 },
	435	{ _MMIO(0x9888), 0x06dc2100 },
	436	{ _MMIO(0x9888), 0x0adc0041 },
	437	{ _MMIO(0x9888), 0x0cdc6600 },
	438	{ _MMIO(0x9888), 0x00dc6580 },
	439	{ _MMIO(0x9888), 0x08dc8000 },
	440	{ _MMIO(0x9888), 0x0edc8000 },
	441	{ _MMIO(0x9888), 0x00d80042 },
	442	{ _MMIO(0x9888), 0x08d82080 },
	443	{ _MMIO(0x9888), 0x0cd8004c },
	444	{ _MMIO(0x9888), 0x0ed82580 },
	445	{ _MMIO(0x9888), 0x00db4000 },
	446	{ _MMIO(0x9888), 0x18db1000 },
	447	{ _MMIO(0x9888), 0x1adb0104 },
	448	{ _MMIO(0x9888), 0x0c9fa800 },
	449	{ _MMIO(0x9888), 0x0e9faa00 },
	450	{ _MMIO(0x9888), 0x109f02aa },
	451	{ _MMIO(0x9888), 0x08b80042 },
	452	{ _MMIO(0x9888), 0x0ab82080 },
	453	{ _MMIO(0x9888), 0x0eb8404c },
	454	{ _MMIO(0x9888), 0x02b8404b },
	455	{ _MMIO(0x9888), 0x00b84000 },
	456	{ _MMIO(0x9888), 0x16b80000 },
	457	{ _MMIO(0x9888), 0x18b81145 },
	458	{ _MMIO(0x9888), 0x04b80000 },
	459	{ _MMIO(0x9888), 0x00b9a000 },
	460	{ _MMIO(0x9888), 0x06b98000 },
	461	{ _MMIO(0x9888), 0x08b9a000 },
	462	{ _MMIO(0x9888), 0x0ab9a000 },
	463	{ _MMIO(0x9888), 0x0cb9a000 },
	464	{ _MMIO(0x9888), 0x0eb9a000 },
	465	{ _MMIO(0x9888), 0x02b92000 },
	466	{ _MMIO(0x9888), 0x01888000 },
	467	{ _MMIO(0x9888), 0x0d88f800 },
	468	{ _MMIO(0x9888), 0x0f88000f },
	469	{ _MMIO(0x9888), 0x03888000 },
	470	{ _MMIO(0x9888), 0x05888000 },
	471	{ _MMIO(0x9888), 0x238b0540 },
	472	{ _MMIO(0x9888), 0x258baaa0 },
	473	{ _MMIO(0x9888), 0x278b002a },
	474	{ _MMIO(0x9888), 0x018c4000 },
	475	{ _MMIO(0x9888), 0x0f8c4000 },
	476	{ _MMIO(0x9888), 0x178c2000 },
	477	{ _MMIO(0x9888), 0x198c5500 },
	478	{ _MMIO(0x9888), 0x1b8c0015 },
	479	{ _MMIO(0x9888), 0x038c4000 },
	480	{ _MMIO(0x9888), 0x058c4000 },
	481	{ _MMIO(0x9888), 0x018da000 },
	482	{ _MMIO(0x9888), 0x078d8000 },
	483	{ _MMIO(0x9888), 0x098da000 },
	484	{ _MMIO(0x9888), 0x0b8da000 },
	485	{ _MMIO(0x9888), 0x0d8da000 },
	486	{ _MMIO(0x9888), 0x0f8da000 },
	487	{ _MMIO(0x9888), 0x038d2000 },
	488	{ _MMIO(0x9888), 0x1f850a80 },
	489	{ _MMIO(0x9888), 0x2185aaa0 },
	490	{ _MMIO(0x9888), 0x2385002a },
	491	{ _MMIO(0x9888), 0x01834000 },
	492	{ _MMIO(0x9888), 0x0f834000 },
	493	{ _MMIO(0x9888), 0x19835400 },
	494	{ _MMIO(0x9888), 0x1b830155 },
	495	{ _MMIO(0x9888), 0x03834000 },
	496	{ _MMIO(0x9888), 0x05834000 },
	497	{ _MMIO(0x9888), 0x0184c000 },
	498	{ _MMIO(0x9888), 0x07848000 },
	499	{ _MMIO(0x9888), 0x0984c000 },
	500	{ _MMIO(0x9888), 0x0b84c000 },
	501	{ _MMIO(0x9888), 0x0d84c000 },
	502	{ _MMIO(0x9888), 0x0f84c000 },
	503	{ _MMIO(0x9888), 0x03844000 },
	504	{ _MMIO(0x9888), 0x17808137 },
	505	{ _MMIO(0x9888), 0x1980c147 },
	506	{ _MMIO(0x9888), 0x1b80c0e5 },
	507	{ _MMIO(0x9888), 0x1d80c0e3 },
	508	{ _MMIO(0x9888), 0x21800000 },
	509	{ _MMIO(0x9888), 0x1180c000 },
	510	{ _MMIO(0x9888), 0x1f80c000 },
	511	{ _MMIO(0x9888), 0x13804000 },
	512	{ _MMIO(0x9888), 0x15800000 },
	513	{ _MMIO(0xd24), 0x00000000 },
	514	{ _MMIO(0x9888), 0x4d805000 },
	515	{ _MMIO(0x9888), 0x4f800555 },
	516	{ _MMIO(0x9888), 0x43800062 },
	517	{ _MMIO(0x9888), 0x51800000 },
	518	{ _MMIO(0x9888), 0x45800062 },
	519	{ _MMIO(0x9888), 0x53800000 },
	520	{ _MMIO(0x9888), 0x47800062 },
	521	{ _MMIO(0x9888), 0x31800000 },
	522	{ _MMIO(0x9888), 0x3f800062 },
	523	{ _MMIO(0x9888), 0x41800000 },
	524	};
	525
	526	static int
	527	get_compute_basic_mux_config(struct drm_i915_private *dev_priv,
	528	const struct i915_oa_reg **regs,
	529	int *lens)
	530	{
	531	int n = 0;
	532
	533	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 2);
	534	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 2);
	535
	536	if (INTEL_INFO(dev_priv)->sseu.slice_mask & 0x01) {
	537	regs[n] = mux_config_compute_basic_0_slices_0x01;
	538	lens[n] = ARRAY_SIZE(mux_config_compute_basic_0_slices_0x01);
	539	n++;
	540	}
	541	if (INTEL_INFO(dev_priv)->sseu.slice_mask & 0x02) {
	542	regs[n] = mux_config_compute_basic_2_slices_0x02;
	543	lens[n] = ARRAY_SIZE(mux_config_compute_basic_2_slices_0x02);
	544	n++;
	545	}
	546
	547	return n;
	548	}
	549
	550	static const struct i915_oa_reg b_counter_config_render_pipe_profile[] = {
	551	{ _MMIO(0x2724), 0xf0800000 },
	552	{ _MMIO(0x2720), 0x00000000 },
	553	{ _MMIO(0x2714), 0xf0800000 },
	554	{ _MMIO(0x2710), 0x00000000 },
	555	{ _MMIO(0x2740), 0x00000000 },
	556	{ _MMIO(0x2770), 0x0007ffea },
	557	{ _MMIO(0x2774), 0x00007ffc },
	558	{ _MMIO(0x2778), 0x0007affa },
	559	{ _MMIO(0x277c), 0x0000f5fd },
	560	{ _MMIO(0x2780), 0x00079ffa },
	561	{ _MMIO(0x2784), 0x0000f3fb },
	562	{ _MMIO(0x2788), 0x0007bf7a },
	563	{ _MMIO(0x278c), 0x0000f7e7 },
	564	{ _MMIO(0x2790), 0x0007fefa },
	565	{ _MMIO(0x2794), 0x0000f7cf },
	566	{ _MMIO(0x2798), 0x00077ffa },
	567	{ _MMIO(0x279c), 0x0000efdf },
	568	{ _MMIO(0x27a0), 0x0006fffa },
	569	{ _MMIO(0x27a4), 0x0000cfbf },
	570	{ _MMIO(0x27a8), 0x0003fffa },
	571	{ _MMIO(0x27ac), 0x00005f7f },
	572	};
	573
	574	static const struct i915_oa_reg flex_eu_config_render_pipe_profile[] = {
	575	{ _MMIO(0xe458), 0x00005004 },
	576	{ _MMIO(0xe558), 0x00015014 },
	577	{ _MMIO(0xe658), 0x00025024 },
	578	{ _MMIO(0xe758), 0x00035034 },
	579	{ _MMIO(0xe45c), 0x00045044 },
	580	{ _MMIO(0xe55c), 0x00055054 },
	581	{ _MMIO(0xe65c), 0x00065064 },
	582	};
	583
	584	static const struct i915_oa_reg mux_config_render_pipe_profile[] = {
	585	{ _MMIO(0x9888), 0x0a1e0000 },
	586	{ _MMIO(0x9888), 0x0c1f000f },
	587	{ _MMIO(0x9888), 0x10176800 },
	588	{ _MMIO(0x9888), 0x1191001f },
	589	{ _MMIO(0x9888), 0x0b880320 },
	590	{ _MMIO(0x9888), 0x01890c40 },
	591	{ _MMIO(0x9888), 0x118a1c00 },
	592	{ _MMIO(0x9888), 0x118d7c00 },
	593	{ _MMIO(0x9888), 0x118e0020 },
	594	{ _MMIO(0x9888), 0x118f4c00 },
	595	{ _MMIO(0x9888), 0x11900000 },
	596	{ _MMIO(0x9888), 0x13900001 },
	597	{ _MMIO(0x9888), 0x065c4000 },
	598	{ _MMIO(0x9888), 0x0c3d8000 },
	599	{ _MMIO(0x9888), 0x06584000 },
	600	{ _MMIO(0x9888), 0x0c5b4000 },
	601	{ _MMIO(0x9888), 0x081e0040 },
	602	{ _MMIO(0x9888), 0x0e1e0000 },
	603	{ _MMIO(0x9888), 0x021f5400 },
	604	{ _MMIO(0x9888), 0x001f0000 },
	605	{ _MMIO(0x9888), 0x101f0010 },
	606	{ _MMIO(0x9888), 0x0e1f0080 },
	607	{ _MMIO(0x9888), 0x0c384000 },
	608	{ _MMIO(0x9888), 0x06392000 },
	609	{ _MMIO(0x9888), 0x0c13c000 },
	610	{ _MMIO(0x9888), 0x06164000 },
	611	{ _MMIO(0x9888), 0x06170012 },
	612	{ _MMIO(0x9888), 0x00170000 },
	613	{ _MMIO(0x9888), 0x01910005 },
	614	{ _MMIO(0x9888), 0x07880002 },
	615	{ _MMIO(0x9888), 0x01880c00 },
	616	{ _MMIO(0x9888), 0x0f880000 },
	617	{ _MMIO(0x9888), 0x0d880000 },
	618	{ _MMIO(0x9888), 0x05880000 },
	619	{ _MMIO(0x9888), 0x09890032 },
	620	{ _MMIO(0x9888), 0x078a0800 },
	621	{ _MMIO(0x9888), 0x0f8a0a00 },
	622	{ _MMIO(0x9888), 0x198a4000 },
	623	{ _MMIO(0x9888), 0x1b8a2000 },
	624	{ _MMIO(0x9888), 0x1d8a0000 },
	625	{ _MMIO(0x9888), 0x038a4000 },
	626	{ _MMIO(0x9888), 0x0b8a8000 },
	627	{ _MMIO(0x9888), 0x0d8a8000 },
	628	{ _MMIO(0x9888), 0x238b54c0 },
	629	{ _MMIO(0x9888), 0x258baa55 },
	630	{ _MMIO(0x9888), 0x278b0019 },
	631	{ _MMIO(0x9888), 0x198c0100 },
	632	{ _MMIO(0x9888), 0x058c4000 },
	633	{ _MMIO(0x9888), 0x0f8d0015 },
	634	{ _MMIO(0x9888), 0x018d1000 },
	635	{ _MMIO(0x9888), 0x098d8000 },
	636	{ _MMIO(0x9888), 0x0b8df000 },
	637	{ _MMIO(0x9888), 0x0d8d3000 },
	638	{ _MMIO(0x9888), 0x038de000 },
	639	{ _MMIO(0x9888), 0x058d3000 },
	640	{ _MMIO(0x9888), 0x0d8e0004 },
	641	{ _MMIO(0x9888), 0x058e000c },
	642	{ _MMIO(0x9888), 0x098e0000 },
	643	{ _MMIO(0x9888), 0x078e0000 },
	644	{ _MMIO(0x9888), 0x038e0000 },
	645	{ _MMIO(0x9888), 0x0b8f0020 },
	646	{ _MMIO(0x9888), 0x198f0c00 },
	647	{ _MMIO(0x9888), 0x078f8000 },
	648	{ _MMIO(0x9888), 0x098f4000 },
	649	{ _MMIO(0x9888), 0x0b900980 },
	650	{ _MMIO(0x9888), 0x03900d80 },
	651	{ _MMIO(0x9888), 0x01900000 },
	652	{ _MMIO(0x9888), 0x1f85aa80 },
	653	{ _MMIO(0x9888), 0x2185aaaa },
	654	{ _MMIO(0x9888), 0x2385002a },
	655	{ _MMIO(0x9888), 0x01834000 },
	656	{ _MMIO(0x9888), 0x0f834000 },
	657	{ _MMIO(0x9888), 0x19835400 },
	658	{ _MMIO(0x9888), 0x1b830155 },
	659	{ _MMIO(0x9888), 0x03834000 },
	660	{ _MMIO(0x9888), 0x05834000 },
	661	{ _MMIO(0x9888), 0x07834000 },
	662	{ _MMIO(0x9888), 0x09834000 },
	663	{ _MMIO(0x9888), 0x0b834000 },
	664	{ _MMIO(0x9888), 0x0d834000 },
	665	{ _MMIO(0x9888), 0x0184c000 },
	666	{ _MMIO(0x9888), 0x0784c000 },
	667	{ _MMIO(0x9888), 0x0984c000 },
	668	{ _MMIO(0x9888), 0x0b84c000 },
	669	{ _MMIO(0x9888), 0x0d84c000 },
	670	{ _MMIO(0x9888), 0x0f84c000 },
	671	{ _MMIO(0x9888), 0x0384c000 },
	672	{ _MMIO(0x9888), 0x0584c000 },
	673	{ _MMIO(0x9888), 0x1180c000 },
	674	{ _MMIO(0x9888), 0x1780c000 },
	675	{ _MMIO(0x9888), 0x1980c000 },
	676	{ _MMIO(0x9888), 0x1b80c000 },
	677	{ _MMIO(0x9888), 0x1d80c000 },
	678	{ _MMIO(0x9888), 0x1f80c000 },
	679	{ _MMIO(0x9888), 0x1380c000 },
	680	{ _MMIO(0x9888), 0x1580c000 },
	681	{ _MMIO(0xd24), 0x00000000 },
	682	{ _MMIO(0x9888), 0x4d801111 },
	683	{ _MMIO(0x9888), 0x3d800800 },
	684	{ _MMIO(0x9888), 0x4f801011 },
	685	{ _MMIO(0x9888), 0x43800443 },
	686	{ _MMIO(0x9888), 0x51801111 },
	687	{ _MMIO(0x9888), 0x45800422 },
	688	{ _MMIO(0x9888), 0x53801111 },
	689	{ _MMIO(0x9888), 0x47800c60 },
	690	{ _MMIO(0x9888), 0x21800000 },
	691	{ _MMIO(0x9888), 0x31800000 },
	692	{ _MMIO(0x9888), 0x3f800422 },
	693	{ _MMIO(0x9888), 0x41800021 },
	694	};
	695
	696	static int
	697	get_render_pipe_profile_mux_config(struct drm_i915_private *dev_priv,
	698	const struct i915_oa_reg **regs,
	699	int *lens)
	700	{
	701	int n = 0;
	702
	703	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	704	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	705
	706	regs[n] = mux_config_render_pipe_profile;
	707	lens[n] = ARRAY_SIZE(mux_config_render_pipe_profile);
	708	n++;
	709
	710	return n;
	711	}
	712
	713	static const struct i915_oa_reg b_counter_config_memory_reads[] = {
	714	{ _MMIO(0x2724), 0xf0800000 },
	715	{ _MMIO(0x2720), 0x00000000 },
	716	{ _MMIO(0x2714), 0xf0800000 },
	717	{ _MMIO(0x2710), 0x00000000 },
	718	{ _MMIO(0x274c), 0x86543210 },
	719	{ _MMIO(0x2748), 0x86543210 },
	720	{ _MMIO(0x2744), 0x00006667 },
	721	{ _MMIO(0x2740), 0x00000000 },
	722	{ _MMIO(0x275c), 0x86543210 },
	723	{ _MMIO(0x2758), 0x86543210 },
	724	{ _MMIO(0x2754), 0x00006465 },
	725	{ _MMIO(0x2750), 0x00000000 },
	726	{ _MMIO(0x2770), 0x0007f81a },
	727	{ _MMIO(0x2774), 0x0000fe00 },
	728	{ _MMIO(0x2778), 0x0007f82a },
	729	{ _MMIO(0x277c), 0x0000fe00 },
	730	{ _MMIO(0x2780), 0x0007f872 },
	731	{ _MMIO(0x2784), 0x0000fe00 },
	732	{ _MMIO(0x2788), 0x0007f8ba },
	733	{ _MMIO(0x278c), 0x0000fe00 },
	734	{ _MMIO(0x2790), 0x0007f87a },
	735	{ _MMIO(0x2794), 0x0000fe00 },
	736	{ _MMIO(0x2798), 0x0007f8ea },
	737	{ _MMIO(0x279c), 0x0000fe00 },
	738	{ _MMIO(0x27a0), 0x0007f8e2 },
	739	{ _MMIO(0x27a4), 0x0000fe00 },
	740	{ _MMIO(0x27a8), 0x0007f8f2 },
	741	{ _MMIO(0x27ac), 0x0000fe00 },
	742	};
	743
	744	static const struct i915_oa_reg flex_eu_config_memory_reads[] = {
	745	{ _MMIO(0xe458), 0x00005004 },
	746	{ _MMIO(0xe558), 0x00015014 },
	747	{ _MMIO(0xe658), 0x00025024 },
	748	{ _MMIO(0xe758), 0x00035034 },
	749	{ _MMIO(0xe45c), 0x00045044 },
	750	{ _MMIO(0xe55c), 0x00055054 },
	751	{ _MMIO(0xe65c), 0x00065064 },
	752	};
	753
	754	static const struct i915_oa_reg mux_config_memory_reads[] = {
	755	{ _MMIO(0x9888), 0x198b0343 },
	756	{ _MMIO(0x9888), 0x13845800 },
	757	{ _MMIO(0x9888), 0x15840018 },
	758	{ _MMIO(0x9888), 0x3580001a },
	759	{ _MMIO(0x9888), 0x038b6300 },
	760	{ _MMIO(0x9888), 0x058b6b62 },
	761	{ _MMIO(0x9888), 0x078b006a },
	762	{ _MMIO(0x9888), 0x118b0000 },
	763	{ _MMIO(0x9888), 0x238b0000 },
	764	{ _MMIO(0x9888), 0x258b0000 },
	765	{ _MMIO(0x9888), 0x1f85a080 },
	766	{ _MMIO(0x9888), 0x2185aaaa },
	767	{ _MMIO(0x9888), 0x2385000a },
	768	{ _MMIO(0x9888), 0x07834000 },
	769	{ _MMIO(0x9888), 0x09834000 },
	770	{ _MMIO(0x9888), 0x0b834000 },
	771	{ _MMIO(0x9888), 0x0d834000 },
	772	{ _MMIO(0x9888), 0x01840018 },
	773	{ _MMIO(0x9888), 0x07844c80 },
	774	{ _MMIO(0x9888), 0x09840d9a },
	775	{ _MMIO(0x9888), 0x0b840e9c },
	776	{ _MMIO(0x9888), 0x0d840f9e },
	777	{ _MMIO(0x9888), 0x0f840010 },
	778	{ _MMIO(0x9888), 0x11840000 },
	779	{ _MMIO(0x9888), 0x03848000 },
	780	{ _MMIO(0x9888), 0x0584c000 },
	781	{ _MMIO(0x9888), 0x2f8000e5 },
	782	{ _MMIO(0x9888), 0x138080e3 },
	783	{ _MMIO(0x9888), 0x1580c0e1 },
	784	{ _MMIO(0x9888), 0x21800000 },
	785	{ _MMIO(0x9888), 0x11804000 },
	786	{ _MMIO(0x9888), 0x1780c000 },
	787	{ _MMIO(0x9888), 0x1980c000 },
	788	{ _MMIO(0x9888), 0x1b80c000 },
	789	{ _MMIO(0x9888), 0x1d80c000 },
	790	{ _MMIO(0x9888), 0x1f804000 },
	791	{ _MMIO(0xd24), 0x00000000 },
	792	{ _MMIO(0x9888), 0x4d800000 },
	793	{ _MMIO(0x9888), 0x3d800800 },
	794	{ _MMIO(0x9888), 0x4f800000 },
	795	{ _MMIO(0x9888), 0x43800842 },
	796	{ _MMIO(0x9888), 0x51800000 },
	797	{ _MMIO(0x9888), 0x45800842 },
	798	{ _MMIO(0x9888), 0x53800000 },
	799	{ _MMIO(0x9888), 0x47801042 },
	800	{ _MMIO(0x9888), 0x31800000 },
	801	{ _MMIO(0x9888), 0x3f800084 },
	802	{ _MMIO(0x9888), 0x41800000 },
	803	};
	804
	805	static int
	806	get_memory_reads_mux_config(struct drm_i915_private *dev_priv,
	807	const struct i915_oa_reg **regs,
	808	int *lens)
	809	{
	810	int n = 0;
	811
	812	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	813	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	814
	815	regs[n] = mux_config_memory_reads;
	816	lens[n] = ARRAY_SIZE(mux_config_memory_reads);
	817	n++;
	818
	819	return n;
	820	}
	821
	822	static const struct i915_oa_reg b_counter_config_memory_writes[] = {
	823	{ _MMIO(0x2724), 0xf0800000 },
	824	{ _MMIO(0x2720), 0x00000000 },
	825	{ _MMIO(0x2714), 0xf0800000 },
	826	{ _MMIO(0x2710), 0x00000000 },
	827	{ _MMIO(0x274c), 0x86543210 },
	828	{ _MMIO(0x2748), 0x86543210 },
	829	{ _MMIO(0x2744), 0x00006667 },
	830	{ _MMIO(0x2740), 0x00000000 },
	831	{ _MMIO(0x275c), 0x86543210 },
	832	{ _MMIO(0x2758), 0x86543210 },
	833	{ _MMIO(0x2754), 0x00006465 },
	834	{ _MMIO(0x2750), 0x00000000 },
	835	{ _MMIO(0x2770), 0x0007f81a },
	836	{ _MMIO(0x2774), 0x0000fe00 },
	837	{ _MMIO(0x2778), 0x0007f82a },
	838	{ _MMIO(0x277c), 0x0000fe00 },
	839	{ _MMIO(0x2780), 0x0007f822 },
	840	{ _MMIO(0x2784), 0x0000fe00 },
	841	{ _MMIO(0x2788), 0x0007f8ba },
	842	{ _MMIO(0x278c), 0x0000fe00 },
	843	{ _MMIO(0x2790), 0x0007f87a },
	844	{ _MMIO(0x2794), 0x0000fe00 },
	845	{ _MMIO(0x2798), 0x0007f8ea },
	846	{ _MMIO(0x279c), 0x0000fe00 },
	847	{ _MMIO(0x27a0), 0x0007f8e2 },
	848	{ _MMIO(0x27a4), 0x0000fe00 },
	849	{ _MMIO(0x27a8), 0x0007f8f2 },
	850	{ _MMIO(0x27ac), 0x0000fe00 },
	851	};
	852
	853	static const struct i915_oa_reg flex_eu_config_memory_writes[] = {
	854	{ _MMIO(0xe458), 0x00005004 },
	855	{ _MMIO(0xe558), 0x00015014 },
	856	{ _MMIO(0xe658), 0x00025024 },
	857	{ _MMIO(0xe758), 0x00035034 },
	858	{ _MMIO(0xe45c), 0x00045044 },
	859	{ _MMIO(0xe55c), 0x00055054 },
	860	{ _MMIO(0xe65c), 0x00065064 },
	861	};
	862
	863	static const struct i915_oa_reg mux_config_memory_writes[] = {
	864	{ _MMIO(0x9888), 0x198b0343 },
	865	{ _MMIO(0x9888), 0x13845400 },
	866	{ _MMIO(0x9888), 0x3580001a },
	867	{ _MMIO(0x9888), 0x3d800805 },
	868	{ _MMIO(0x9888), 0x038b6300 },
	869	{ _MMIO(0x9888), 0x058b6b62 },
	870	{ _MMIO(0x9888), 0x078b006a },
	871	{ _MMIO(0x9888), 0x118b0000 },
	872	{ _MMIO(0x9888), 0x238b0000 },
	873	{ _MMIO(0x9888), 0x258b0000 },
	874	{ _MMIO(0x9888), 0x1f85a080 },
	875	{ _MMIO(0x9888), 0x2185aaaa },
	876	{ _MMIO(0x9888), 0x23850002 },
	877	{ _MMIO(0x9888), 0x07834000 },
	878	{ _MMIO(0x9888), 0x09834000 },
	879	{ _MMIO(0x9888), 0x0b834000 },
	880	{ _MMIO(0x9888), 0x0d834000 },
	881	{ _MMIO(0x9888), 0x01840010 },
	882	{ _MMIO(0x9888), 0x07844880 },
	883	{ _MMIO(0x9888), 0x09840992 },
	884	{ _MMIO(0x9888), 0x0b840a94 },
	885	{ _MMIO(0x9888), 0x0d840b96 },
	886	{ _MMIO(0x9888), 0x11840000 },
	887	{ _MMIO(0x9888), 0x03848000 },
	888	{ _MMIO(0x9888), 0x0584c000 },
	889	{ _MMIO(0x9888), 0x2d800147 },
	890	{ _MMIO(0x9888), 0x2f8000e5 },
	891	{ _MMIO(0x9888), 0x138080e3 },
	892	{ _MMIO(0x9888), 0x1580c0e1 },
	893	{ _MMIO(0x9888), 0x21800000 },
	894	{ _MMIO(0x9888), 0x11804000 },
	895	{ _MMIO(0x9888), 0x1780c000 },
	896	{ _MMIO(0x9888), 0x1980c000 },
	897	{ _MMIO(0x9888), 0x1b80c000 },
	898	{ _MMIO(0x9888), 0x1d80c000 },
	899	{ _MMIO(0x9888), 0x1f800000 },
	900	{ _MMIO(0xd24), 0x00000000 },
	901	{ _MMIO(0x9888), 0x4d800000 },
	902	{ _MMIO(0x9888), 0x4f800000 },
	903	{ _MMIO(0x9888), 0x43800842 },
	904	{ _MMIO(0x9888), 0x51800000 },
	905	{ _MMIO(0x9888), 0x45800842 },
	906	{ _MMIO(0x9888), 0x53800000 },
	907	{ _MMIO(0x9888), 0x47801082 },
	908	{ _MMIO(0x9888), 0x31800000 },
	909	{ _MMIO(0x9888), 0x3f800084 },
	910	{ _MMIO(0x9888), 0x41800000 },
	911	};
	912
	913	static int
	914	get_memory_writes_mux_config(struct drm_i915_private *dev_priv,
	915	const struct i915_oa_reg **regs,
	916	int *lens)
	917	{
	918	int n = 0;
	919
	920	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	921	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	922
	923	regs[n] = mux_config_memory_writes;
	924	lens[n] = ARRAY_SIZE(mux_config_memory_writes);
	925	n++;
	926
	927	return n;
	928	}
	929
	930	static const struct i915_oa_reg b_counter_config_compute_extended[] = {
	931	{ _MMIO(0x2724), 0xf0800000 },
	932	{ _MMIO(0x2720), 0x00000000 },
	933	{ _MMIO(0x2714), 0xf0800000 },
	934	{ _MMIO(0x2710), 0x00000000 },
	935	{ _MMIO(0x2740), 0x00000000 },
	936	{ _MMIO(0x2770), 0x0007fc2a },
	937	{ _MMIO(0x2774), 0x0000bf00 },
	938	{ _MMIO(0x2778), 0x0007fc6a },
	939	{ _MMIO(0x277c), 0x0000bf00 },
	940	{ _MMIO(0x2780), 0x0007fc92 },
	941	{ _MMIO(0x2784), 0x0000bf00 },
	942	{ _MMIO(0x2788), 0x0007fca2 },
	943	{ _MMIO(0x278c), 0x0000bf00 },
	944	{ _MMIO(0x2790), 0x0007fc32 },
	945	{ _MMIO(0x2794), 0x0000bf00 },
	946	{ _MMIO(0x2798), 0x0007fc9a },
	947	{ _MMIO(0x279c), 0x0000bf00 },
	948	{ _MMIO(0x27a0), 0x0007fe6a },
	949	{ _MMIO(0x27a4), 0x0000bf00 },
	950	{ _MMIO(0x27a8), 0x0007fe7a },
	951	{ _MMIO(0x27ac), 0x0000bf00 },
	952	};
	953
	954	static const struct i915_oa_reg flex_eu_config_compute_extended[] = {
	955	{ _MMIO(0xe458), 0x00005004 },
	956	{ _MMIO(0xe558), 0x00000003 },
	957	{ _MMIO(0xe658), 0x00002001 },
	958	{ _MMIO(0xe758), 0x00778008 },
	959	{ _MMIO(0xe45c), 0x00088078 },
	960	{ _MMIO(0xe55c), 0x00808708 },
	961	{ _MMIO(0xe65c), 0x00a08908 },
	962	};
	963
	964	static const struct i915_oa_reg mux_config_compute_extended_0_subslices_0x01[] = {
	965	{ _MMIO(0x9888), 0x143d0160 },
	966	{ _MMIO(0x9888), 0x163d2800 },
	967	{ _MMIO(0x9888), 0x183d0120 },
	968	{ _MMIO(0x9888), 0x105800e0 },
	969	{ _MMIO(0x9888), 0x005cc000 },
	970	{ _MMIO(0x9888), 0x065c8000 },
	971	{ _MMIO(0x9888), 0x085cc000 },
	972	{ _MMIO(0x9888), 0x0a5cc000 },
	973	{ _MMIO(0x9888), 0x0c5cc000 },
	974	{ _MMIO(0x9888), 0x0e5cc000 },
	975	{ _MMIO(0x9888), 0x025cc000 },
	976	{ _MMIO(0x9888), 0x045cc000 },
	977	{ _MMIO(0x9888), 0x003d0011 },
	978	{ _MMIO(0x9888), 0x063d0900 },
	979	{ _MMIO(0x9888), 0x083d0a13 },
	980	{ _MMIO(0x9888), 0x0a3d0b15 },
	981	{ _MMIO(0x9888), 0x0c3d2317 },
	982	{ _MMIO(0x9888), 0x043d21b7 },
	983	{ _MMIO(0x9888), 0x103d0000 },
	984	{ _MMIO(0x9888), 0x0e3d0000 },
	985	{ _MMIO(0x9888), 0x1a3d0000 },
	986	{ _MMIO(0x9888), 0x0e5825c1 },
	987	{ _MMIO(0x9888), 0x00586100 },
	988	{ _MMIO(0x9888), 0x0258204c },
	989	{ _MMIO(0x9888), 0x06588000 },
	990	{ _MMIO(0x9888), 0x0858c000 },
	991	{ _MMIO(0x9888), 0x0a58c000 },
	992	{ _MMIO(0x9888), 0x0c58c000 },
	993	{ _MMIO(0x9888), 0x0458c000 },
	994	{ _MMIO(0x9888), 0x005b4000 },
	995	{ _MMIO(0x9888), 0x0e5b4000 },
	996	{ _MMIO(0x9888), 0x185b5400 },
	997	{ _MMIO(0x9888), 0x1a5b0155 },
	998	{ _MMIO(0x9888), 0x025b4000 },
	999	{ _MMIO(0x9888), 0x045b4000 },
	1000	{ _MMIO(0x9888), 0x065b4000 },
	1001	{ _MMIO(0x9888), 0x085b4000 },
	1002	{ _MMIO(0x9888), 0x0a5b4000 },
	1003	{ _MMIO(0x9888), 0x0c1fa800 },
	1004	{ _MMIO(0x9888), 0x0e1faa2a },
	1005	{ _MMIO(0x9888), 0x101f02aa },
	1006	{ _MMIO(0x9888), 0x00384000 },
	1007	{ _MMIO(0x9888), 0x0e384000 },
	1008	{ _MMIO(0x9888), 0x16384000 },
	1009	{ _MMIO(0x9888), 0x18381555 },
	1010	{ _MMIO(0x9888), 0x02384000 },
	1011	{ _MMIO(0x9888), 0x04384000 },
	1012	{ _MMIO(0x9888), 0x06384000 },
	1013	{ _MMIO(0x9888), 0x08384000 },
	1014	{ _MMIO(0x9888), 0x0a384000 },
	1015	{ _MMIO(0x9888), 0x0039a000 },
	1016	{ _MMIO(0x9888), 0x06398000 },
	1017	{ _MMIO(0x9888), 0x0839a000 },
	1018	{ _MMIO(0x9888), 0x0a39a000 },
	1019	{ _MMIO(0x9888), 0x0c39a000 },
	1020	{ _MMIO(0x9888), 0x0e39a000 },
	1021	{ _MMIO(0x9888), 0x0239a000 },
	1022	{ _MMIO(0x9888), 0x0439a000 },
	1023	{ _MMIO(0x9888), 0x018a8000 },
	1024	{ _MMIO(0x9888), 0x0f8a8000 },
	1025	{ _MMIO(0x9888), 0x198a8000 },
	1026	{ _MMIO(0x9888), 0x1b8aaaa0 },
	1027	{ _MMIO(0x9888), 0x1d8a0002 },
	1028	{ _MMIO(0x9888), 0x038a8000 },
	1029	{ _MMIO(0x9888), 0x058a8000 },
	1030	{ _MMIO(0x9888), 0x078a8000 },
	1031	{ _MMIO(0x9888), 0x098a8000 },
	1032	{ _MMIO(0x9888), 0x0b8a8000 },
	1033	{ _MMIO(0x9888), 0x238b2aa0 },
	1034	{ _MMIO(0x9888), 0x258b5551 },
	1035	{ _MMIO(0x9888), 0x278b0015 },
	1036	{ _MMIO(0x9888), 0x1f85aa80 },
	1037	{ _MMIO(0x9888), 0x2185aaa2 },
	1038	{ _MMIO(0x9888), 0x2385002a },
	1039	{ _MMIO(0x9888), 0x01834000 },
	1040	{ _MMIO(0x9888), 0x0f834000 },
	1041	{ _MMIO(0x9888), 0x19835400 },
	1042	{ _MMIO(0x9888), 0x1b830155 },
	1043	{ _MMIO(0x9888), 0x03834000 },
	1044	{ _MMIO(0x9888), 0x05834000 },
	1045	{ _MMIO(0x9888), 0x07834000 },
	1046	{ _MMIO(0x9888), 0x09834000 },
	1047	{ _MMIO(0x9888), 0x0b834000 },
	1048	{ _MMIO(0x9888), 0x0184c000 },
	1049	{ _MMIO(0x9888), 0x07848000 },
	1050	{ _MMIO(0x9888), 0x0984c000 },
	1051	{ _MMIO(0x9888), 0x0b84c000 },
	1052	{ _MMIO(0x9888), 0x0d84c000 },
	1053	{ _MMIO(0x9888), 0x0f84c000 },
	1054	{ _MMIO(0x9888), 0x0384c000 },
	1055	{ _MMIO(0x9888), 0x0584c000 },
	1056	{ _MMIO(0x9888), 0x1180c000 },
	1057	{ _MMIO(0x9888), 0x17808000 },
	1058	{ _MMIO(0x9888), 0x1980c000 },
	1059	{ _MMIO(0x9888), 0x1b80c000 },
	1060	{ _MMIO(0x9888), 0x1d80c000 },
	1061	{ _MMIO(0x9888), 0x1f80c000 },
	1062	{ _MMIO(0x9888), 0x1380c000 },
	1063	{ _MMIO(0x9888), 0x1580c000 },
	1064	{ _MMIO(0xd24), 0x00000000 },
	1065	{ _MMIO(0x9888), 0x4d800000 },
	1066	{ _MMIO(0x9888), 0x3d800000 },
	1067	{ _MMIO(0x9888), 0x4f800000 },
	1068	{ _MMIO(0x9888), 0x43800000 },
	1069	{ _MMIO(0x9888), 0x51800000 },
	1070	{ _MMIO(0x9888), 0x45800000 },
	1071	{ _MMIO(0x9888), 0x53800000 },
	1072	{ _MMIO(0x9888), 0x47800420 },
	1073	{ _MMIO(0x9888), 0x21800000 },
	1074	{ _MMIO(0x9888), 0x31800000 },
	1075	{ _MMIO(0x9888), 0x3f800421 },
	1076	{ _MMIO(0x9888), 0x41800000 },
	1077	};
	1078
	1079	static const struct i915_oa_reg mux_config_compute_extended_2_subslices_0x02[] = {
	1080	{ _MMIO(0x9888), 0x105c00e0 },
	1081	{ _MMIO(0x9888), 0x145b0160 },
	1082	{ _MMIO(0x9888), 0x165b2800 },
	1083	{ _MMIO(0x9888), 0x185b0120 },
	1084	{ _MMIO(0x9888), 0x0e5c25c1 },
	1085	{ _MMIO(0x9888), 0x005c6100 },
	1086	{ _MMIO(0x9888), 0x025c204c },
	1087	{ _MMIO(0x9888), 0x065c8000 },
	1088	{ _MMIO(0x9888), 0x085cc000 },
	1089	{ _MMIO(0x9888), 0x0a5cc000 },
	1090	{ _MMIO(0x9888), 0x0c5cc000 },
	1091	{ _MMIO(0x9888), 0x045cc000 },
	1092	{ _MMIO(0x9888), 0x005b0011 },
	1093	{ _MMIO(0x9888), 0x065b0900 },
	1094	{ _MMIO(0x9888), 0x085b0a13 },
	1095	{ _MMIO(0x9888), 0x0a5b0b15 },
	1096	{ _MMIO(0x9888), 0x0c5b2317 },
	1097	{ _MMIO(0x9888), 0x045b21b7 },
	1098	{ _MMIO(0x9888), 0x105b0000 },
	1099	{ _MMIO(0x9888), 0x0e5b0000 },
	1100	{ _MMIO(0x9888), 0x1a5b0000 },
	1101	{ _MMIO(0x9888), 0x0c1fa800 },
	1102	{ _MMIO(0x9888), 0x0e1faa2a },
	1103	{ _MMIO(0x9888), 0x101f02aa },
	1104	{ _MMIO(0x9888), 0x00384000 },
	1105	{ _MMIO(0x9888), 0x0e384000 },
	1106	{ _MMIO(0x9888), 0x16384000 },
	1107	{ _MMIO(0x9888), 0x18381555 },
	1108	{ _MMIO(0x9888), 0x02384000 },
	1109	{ _MMIO(0x9888), 0x04384000 },
	1110	{ _MMIO(0x9888), 0x06384000 },
	1111	{ _MMIO(0x9888), 0x08384000 },
	1112	{ _MMIO(0x9888), 0x0a384000 },
	1113	{ _MMIO(0x9888), 0x0039a000 },
	1114	{ _MMIO(0x9888), 0x06398000 },
	1115	{ _MMIO(0x9888), 0x0839a000 },
	1116	{ _MMIO(0x9888), 0x0a39a000 },
	1117	{ _MMIO(0x9888), 0x0c39a000 },
	1118	{ _MMIO(0x9888), 0x0e39a000 },
	1119	{ _MMIO(0x9888), 0x0239a000 },
	1120	{ _MMIO(0x9888), 0x0439a000 },
	1121	{ _MMIO(0x9888), 0x018a8000 },
	1122	{ _MMIO(0x9888), 0x0f8a8000 },
	1123	{ _MMIO(0x9888), 0x198a8000 },
	1124	{ _MMIO(0x9888), 0x1b8aaaa0 },
	1125	{ _MMIO(0x9888), 0x1d8a0002 },
	1126	{ _MMIO(0x9888), 0x038a8000 },
	1127	{ _MMIO(0x9888), 0x058a8000 },
	1128	{ _MMIO(0x9888), 0x078a8000 },
	1129	{ _MMIO(0x9888), 0x098a8000 },
	1130	{ _MMIO(0x9888), 0x0b8a8000 },
	1131	{ _MMIO(0x9888), 0x238b2aa0 },
	1132	{ _MMIO(0x9888), 0x258b5551 },
	1133	{ _MMIO(0x9888), 0x278b0015 },
	1134	{ _MMIO(0x9888), 0x1f85aa80 },
	1135	{ _MMIO(0x9888), 0x2185aaa2 },
	1136	{ _MMIO(0x9888), 0x2385002a },
	1137	{ _MMIO(0x9888), 0x01834000 },
	1138	{ _MMIO(0x9888), 0x0f834000 },
	1139	{ _MMIO(0x9888), 0x19835400 },
	1140	{ _MMIO(0x9888), 0x1b830155 },
	1141	{ _MMIO(0x9888), 0x03834000 },
	1142	{ _MMIO(0x9888), 0x05834000 },
	1143	{ _MMIO(0x9888), 0x07834000 },
	1144	{ _MMIO(0x9888), 0x09834000 },
	1145	{ _MMIO(0x9888), 0x0b834000 },
	1146	{ _MMIO(0x9888), 0x0184c000 },
	1147	{ _MMIO(0x9888), 0x07848000 },
	1148	{ _MMIO(0x9888), 0x0984c000 },
	1149	{ _MMIO(0x9888), 0x0b84c000 },
	1150	{ _MMIO(0x9888), 0x0d84c000 },
	1151	{ _MMIO(0x9888), 0x0f84c000 },
	1152	{ _MMIO(0x9888), 0x0384c000 },
	1153	{ _MMIO(0x9888), 0x0584c000 },
	1154	{ _MMIO(0x9888), 0x1180c000 },
	1155	{ _MMIO(0x9888), 0x17808000 },
	1156	{ _MMIO(0x9888), 0x1980c000 },
	1157	{ _MMIO(0x9888), 0x1b80c000 },
	1158	{ _MMIO(0x9888), 0x1d80c000 },
	1159	{ _MMIO(0x9888), 0x1f80c000 },
	1160	{ _MMIO(0x9888), 0x1380c000 },
	1161	{ _MMIO(0x9888), 0x1580c000 },
	1162	{ _MMIO(0xd24), 0x00000000 },
	1163	{ _MMIO(0x9888), 0x4d800000 },
	1164	{ _MMIO(0x9888), 0x3d800000 },
	1165	{ _MMIO(0x9888), 0x4f800000 },
	1166	{ _MMIO(0x9888), 0x43800000 },
	1167	{ _MMIO(0x9888), 0x51800000 },
	1168	{ _MMIO(0x9888), 0x45800000 },
	1169	{ _MMIO(0x9888), 0x53800000 },
	1170	{ _MMIO(0x9888), 0x47800420 },
	1171	{ _MMIO(0x9888), 0x21800000 },
	1172	{ _MMIO(0x9888), 0x31800000 },
	1173	{ _MMIO(0x9888), 0x3f800421 },
	1174	{ _MMIO(0x9888), 0x41800000 },
	1175	};
	1176
	1177	static const struct i915_oa_reg mux_config_compute_extended_4_subslices_0x04[] = {
	1178	{ _MMIO(0x9888), 0x103800e0 },
	1179	{ _MMIO(0x9888), 0x143a0160 },
	1180	{ _MMIO(0x9888), 0x163a2800 },
	1181	{ _MMIO(0x9888), 0x183a0120 },
	1182	{ _MMIO(0x9888), 0x0c1fa800 },
	1183	{ _MMIO(0x9888), 0x0e1faa2a },
	1184	{ _MMIO(0x9888), 0x101f02aa },
	1185	{ _MMIO(0x9888), 0x0e38a5c1 },
	1186	{ _MMIO(0x9888), 0x0038a100 },
	1187	{ _MMIO(0x9888), 0x0238204c },
	1188	{ _MMIO(0x9888), 0x16388000 },
	1189	{ _MMIO(0x9888), 0x183802aa },
	1190	{ _MMIO(0x9888), 0x04380000 },
	1191	{ _MMIO(0x9888), 0x06380000 },
	1192	{ _MMIO(0x9888), 0x08388000 },
	1193	{ _MMIO(0x9888), 0x0a388000 },
	1194	{ _MMIO(0x9888), 0x0039a000 },
	1195	{ _MMIO(0x9888), 0x06398000 },
	1196	{ _MMIO(0x9888), 0x0839a000 },
	1197	{ _MMIO(0x9888), 0x0a39a000 },
	1198	{ _MMIO(0x9888), 0x0c39a000 },
	1199	{ _MMIO(0x9888), 0x0e39a000 },
	1200	{ _MMIO(0x9888), 0x0239a000 },
	1201	{ _MMIO(0x9888), 0x0439a000 },
	1202	{ _MMIO(0x9888), 0x003a0011 },
	1203	{ _MMIO(0x9888), 0x063a0900 },
	1204	{ _MMIO(0x9888), 0x083a0a13 },
	1205	{ _MMIO(0x9888), 0x0a3a0b15 },
	1206	{ _MMIO(0x9888), 0x0c3a2317 },
	1207	{ _MMIO(0x9888), 0x043a21b7 },
	1208	{ _MMIO(0x9888), 0x103a0000 },
	1209	{ _MMIO(0x9888), 0x0e3a0000 },
	1210	{ _MMIO(0x9888), 0x1a3a0000 },
	1211	{ _MMIO(0x9888), 0x018a8000 },
	1212	{ _MMIO(0x9888), 0x0f8a8000 },
	1213	{ _MMIO(0x9888), 0x198a8000 },
	1214	{ _MMIO(0x9888), 0x1b8aaaa0 },
	1215	{ _MMIO(0x9888), 0x1d8a0002 },
	1216	{ _MMIO(0x9888), 0x038a8000 },
	1217	{ _MMIO(0x9888), 0x058a8000 },
	1218	{ _MMIO(0x9888), 0x078a8000 },
	1219	{ _MMIO(0x9888), 0x098a8000 },
	1220	{ _MMIO(0x9888), 0x0b8a8000 },
	1221	{ _MMIO(0x9888), 0x238b2aa0 },
	1222	{ _MMIO(0x9888), 0x258b5551 },
	1223	{ _MMIO(0x9888), 0x278b0015 },
	1224	{ _MMIO(0x9888), 0x1f85aa80 },
	1225	{ _MMIO(0x9888), 0x2185aaa2 },
	1226	{ _MMIO(0x9888), 0x2385002a },
	1227	{ _MMIO(0x9888), 0x01834000 },
	1228	{ _MMIO(0x9888), 0x0f834000 },
	1229	{ _MMIO(0x9888), 0x19835400 },
	1230	{ _MMIO(0x9888), 0x1b830155 },
	1231	{ _MMIO(0x9888), 0x03834000 },
	1232	{ _MMIO(0x9888), 0x05834000 },
	1233	{ _MMIO(0x9888), 0x07834000 },
	1234	{ _MMIO(0x9888), 0x09834000 },
	1235	{ _MMIO(0x9888), 0x0b834000 },
	1236	{ _MMIO(0x9888), 0x0184c000 },
	1237	{ _MMIO(0x9888), 0x07848000 },
	1238	{ _MMIO(0x9888), 0x0984c000 },
	1239	{ _MMIO(0x9888), 0x0b84c000 },
	1240	{ _MMIO(0x9888), 0x0d84c000 },
	1241	{ _MMIO(0x9888), 0x0f84c000 },
	1242	{ _MMIO(0x9888), 0x0384c000 },
	1243	{ _MMIO(0x9888), 0x0584c000 },
	1244	{ _MMIO(0x9888), 0x1180c000 },
	1245	{ _MMIO(0x9888), 0x17808000 },
	1246	{ _MMIO(0x9888), 0x1980c000 },
	1247	{ _MMIO(0x9888), 0x1b80c000 },
	1248	{ _MMIO(0x9888), 0x1d80c000 },
	1249	{ _MMIO(0x9888), 0x1f80c000 },
	1250	{ _MMIO(0x9888), 0x1380c000 },
	1251	{ _MMIO(0x9888), 0x1580c000 },
	1252	{ _MMIO(0xd24), 0x00000000 },
	1253	{ _MMIO(0x9888), 0x4d800000 },
	1254	{ _MMIO(0x9888), 0x3d800000 },
	1255	{ _MMIO(0x9888), 0x4f800000 },
	1256	{ _MMIO(0x9888), 0x43800000 },
	1257	{ _MMIO(0x9888), 0x51800000 },
	1258	{ _MMIO(0x9888), 0x45800000 },
	1259	{ _MMIO(0x9888), 0x53800000 },
	1260	{ _MMIO(0x9888), 0x47800420 },
	1261	{ _MMIO(0x9888), 0x21800000 },
	1262	{ _MMIO(0x9888), 0x31800000 },
	1263	{ _MMIO(0x9888), 0x3f800421 },
	1264	{ _MMIO(0x9888), 0x41800000 },
	1265	};
	1266
	1267	static const struct i915_oa_reg mux_config_compute_extended_1_subslices_0x08[] = {
	1268	{ _MMIO(0x9888), 0x14bd0160 },
	1269	{ _MMIO(0x9888), 0x16bd2800 },
	1270	{ _MMIO(0x9888), 0x18bd0120 },
	1271	{ _MMIO(0x9888), 0x10d800e0 },
	1272	{ _MMIO(0x9888), 0x00dcc000 },
	1273	{ _MMIO(0x9888), 0x06dc8000 },
	1274	{ _MMIO(0x9888), 0x08dcc000 },
	1275	{ _MMIO(0x9888), 0x0adcc000 },
	1276	{ _MMIO(0x9888), 0x0cdcc000 },
	1277	{ _MMIO(0x9888), 0x0edcc000 },
	1278	{ _MMIO(0x9888), 0x02dcc000 },
	1279	{ _MMIO(0x9888), 0x04dcc000 },
	1280	{ _MMIO(0x9888), 0x00bd0011 },
	1281	{ _MMIO(0x9888), 0x06bd0900 },
	1282	{ _MMIO(0x9888), 0x08bd0a13 },
	1283	{ _MMIO(0x9888), 0x0abd0b15 },
	1284	{ _MMIO(0x9888), 0x0cbd2317 },
	1285	{ _MMIO(0x9888), 0x04bd21b7 },
	1286	{ _MMIO(0x9888), 0x10bd0000 },
	1287	{ _MMIO(0x9888), 0x0ebd0000 },
	1288	{ _MMIO(0x9888), 0x1abd0000 },
	1289	{ _MMIO(0x9888), 0x0ed825c1 },
	1290	{ _MMIO(0x9888), 0x00d86100 },
	1291	{ _MMIO(0x9888), 0x02d8204c },
	1292	{ _MMIO(0x9888), 0x06d88000 },
	1293	{ _MMIO(0x9888), 0x08d8c000 },
	1294	{ _MMIO(0x9888), 0x0ad8c000 },
	1295	{ _MMIO(0x9888), 0x0cd8c000 },
	1296	{ _MMIO(0x9888), 0x04d8c000 },
	1297	{ _MMIO(0x9888), 0x00db4000 },
	1298	{ _MMIO(0x9888), 0x0edb4000 },
	1299	{ _MMIO(0x9888), 0x18db5400 },
	1300	{ _MMIO(0x9888), 0x1adb0155 },
	1301	{ _MMIO(0x9888), 0x02db4000 },
	1302	{ _MMIO(0x9888), 0x04db4000 },
	1303	{ _MMIO(0x9888), 0x06db4000 },
	1304	{ _MMIO(0x9888), 0x08db4000 },
	1305	{ _MMIO(0x9888), 0x0adb4000 },
	1306	{ _MMIO(0x9888), 0x0c9fa800 },
	1307	{ _MMIO(0x9888), 0x0e9faa2a },
	1308	{ _MMIO(0x9888), 0x109f02aa },
	1309	{ _MMIO(0x9888), 0x00b84000 },
	1310	{ _MMIO(0x9888), 0x0eb84000 },
	1311	{ _MMIO(0x9888), 0x16b84000 },
	1312	{ _MMIO(0x9888), 0x18b81555 },
	1313	{ _MMIO(0x9888), 0x02b84000 },
	1314	{ _MMIO(0x9888), 0x04b84000 },
	1315	{ _MMIO(0x9888), 0x06b84000 },
	1316	{ _MMIO(0x9888), 0x08b84000 },
	1317	{ _MMIO(0x9888), 0x0ab84000 },
	1318	{ _MMIO(0x9888), 0x00b9a000 },
	1319	{ _MMIO(0x9888), 0x06b98000 },
	1320	{ _MMIO(0x9888), 0x08b9a000 },
	1321	{ _MMIO(0x9888), 0x0ab9a000 },
	1322	{ _MMIO(0x9888), 0x0cb9a000 },
	1323	{ _MMIO(0x9888), 0x0eb9a000 },
	1324	{ _MMIO(0x9888), 0x02b9a000 },
	1325	{ _MMIO(0x9888), 0x04b9a000 },
	1326	{ _MMIO(0x9888), 0x01888000 },
	1327	{ _MMIO(0x9888), 0x0d88f800 },
	1328	{ _MMIO(0x9888), 0x0f88000f },
	1329	{ _MMIO(0x9888), 0x03888000 },
	1330	{ _MMIO(0x9888), 0x05888000 },
	1331	{ _MMIO(0x9888), 0x07888000 },
	1332	{ _MMIO(0x9888), 0x09888000 },
	1333	{ _MMIO(0x9888), 0x0b888000 },
	1334	{ _MMIO(0x9888), 0x238b5540 },
	1335	{ _MMIO(0x9888), 0x258baaa2 },
	1336	{ _MMIO(0x9888), 0x278b002a },
	1337	{ _MMIO(0x9888), 0x018c4000 },
	1338	{ _MMIO(0x9888), 0x0f8c4000 },
	1339	{ _MMIO(0x9888), 0x178c2000 },
	1340	{ _MMIO(0x9888), 0x198c5500 },
	1341	{ _MMIO(0x9888), 0x1b8c0015 },
	1342	{ _MMIO(0x9888), 0x038c4000 },
	1343	{ _MMIO(0x9888), 0x058c4000 },
	1344	{ _MMIO(0x9888), 0x078c4000 },
	1345	{ _MMIO(0x9888), 0x098c4000 },
	1346	{ _MMIO(0x9888), 0x0b8c4000 },
	1347	{ _MMIO(0x9888), 0x018da000 },
	1348	{ _MMIO(0x9888), 0x078d8000 },
	1349	{ _MMIO(0x9888), 0x098da000 },
	1350	{ _MMIO(0x9888), 0x0b8da000 },
	1351	{ _MMIO(0x9888), 0x0d8da000 },
	1352	{ _MMIO(0x9888), 0x0f8da000 },
	1353	{ _MMIO(0x9888), 0x038da000 },
	1354	{ _MMIO(0x9888), 0x058da000 },
	1355	{ _MMIO(0x9888), 0x1f85aa80 },
	1356	{ _MMIO(0x9888), 0x2185aaa2 },
	1357	{ _MMIO(0x9888), 0x2385002a },
	1358	{ _MMIO(0x9888), 0x01834000 },
	1359	{ _MMIO(0x9888), 0x0f834000 },
	1360	{ _MMIO(0x9888), 0x19835400 },
	1361	{ _MMIO(0x9888), 0x1b830155 },
	1362	{ _MMIO(0x9888), 0x03834000 },
	1363	{ _MMIO(0x9888), 0x05834000 },
	1364	{ _MMIO(0x9888), 0x07834000 },
	1365	{ _MMIO(0x9888), 0x09834000 },
	1366	{ _MMIO(0x9888), 0x0b834000 },
	1367	{ _MMIO(0x9888), 0x0184c000 },
	1368	{ _MMIO(0x9888), 0x07848000 },
	1369	{ _MMIO(0x9888), 0x0984c000 },
	1370	{ _MMIO(0x9888), 0x0b84c000 },
	1371	{ _MMIO(0x9888), 0x0d84c000 },
	1372	{ _MMIO(0x9888), 0x0f84c000 },
	1373	{ _MMIO(0x9888), 0x0384c000 },
	1374	{ _MMIO(0x9888), 0x0584c000 },
	1375	{ _MMIO(0x9888), 0x1180c000 },
	1376	{ _MMIO(0x9888), 0x17808000 },
	1377	{ _MMIO(0x9888), 0x1980c000 },
	1378	{ _MMIO(0x9888), 0x1b80c000 },
	1379	{ _MMIO(0x9888), 0x1d80c000 },
	1380	{ _MMIO(0x9888), 0x1f80c000 },
	1381	{ _MMIO(0x9888), 0x1380c000 },
	1382	{ _MMIO(0x9888), 0x1580c000 },
	1383	{ _MMIO(0xd24), 0x00000000 },
	1384	{ _MMIO(0x9888), 0x4d800000 },
	1385	{ _MMIO(0x9888), 0x3d800000 },
	1386	{ _MMIO(0x9888), 0x4f800000 },
	1387	{ _MMIO(0x9888), 0x43800000 },
	1388	{ _MMIO(0x9888), 0x51800000 },
	1389	{ _MMIO(0x9888), 0x45800000 },
	1390	{ _MMIO(0x9888), 0x53800000 },
	1391	{ _MMIO(0x9888), 0x47800420 },
	1392	{ _MMIO(0x9888), 0x21800000 },
	1393	{ _MMIO(0x9888), 0x31800000 },
	1394	{ _MMIO(0x9888), 0x3f800421 },
	1395	{ _MMIO(0x9888), 0x41800000 },
	1396	};
	1397
	1398	static const struct i915_oa_reg mux_config_compute_extended_3_subslices_0x10[] = {
	1399	{ _MMIO(0x9888), 0x10dc00e0 },
	1400	{ _MMIO(0x9888), 0x14db0160 },
	1401	{ _MMIO(0x9888), 0x16db2800 },
	1402	{ _MMIO(0x9888), 0x18db0120 },
	1403	{ _MMIO(0x9888), 0x0edc25c1 },
	1404	{ _MMIO(0x9888), 0x00dc6100 },
	1405	{ _MMIO(0x9888), 0x02dc204c },
	1406	{ _MMIO(0x9888), 0x06dc8000 },
	1407	{ _MMIO(0x9888), 0x08dcc000 },
	1408	{ _MMIO(0x9888), 0x0adcc000 },
	1409	{ _MMIO(0x9888), 0x0cdcc000 },
	1410	{ _MMIO(0x9888), 0x04dcc000 },
	1411	{ _MMIO(0x9888), 0x00db0011 },
	1412	{ _MMIO(0x9888), 0x06db0900 },
	1413	{ _MMIO(0x9888), 0x08db0a13 },
	1414	{ _MMIO(0x9888), 0x0adb0b15 },
	1415	{ _MMIO(0x9888), 0x0cdb2317 },
	1416	{ _MMIO(0x9888), 0x04db21b7 },
	1417	{ _MMIO(0x9888), 0x10db0000 },
	1418	{ _MMIO(0x9888), 0x0edb0000 },
	1419	{ _MMIO(0x9888), 0x1adb0000 },
	1420	{ _MMIO(0x9888), 0x0c9fa800 },
	1421	{ _MMIO(0x9888), 0x0e9faa2a },
	1422	{ _MMIO(0x9888), 0x109f02aa },
	1423	{ _MMIO(0x9888), 0x00b84000 },
	1424	{ _MMIO(0x9888), 0x0eb84000 },
	1425	{ _MMIO(0x9888), 0x16b84000 },
	1426	{ _MMIO(0x9888), 0x18b81555 },
	1427	{ _MMIO(0x9888), 0x02b84000 },
	1428	{ _MMIO(0x9888), 0x04b84000 },
	1429	{ _MMIO(0x9888), 0x06b84000 },
	1430	{ _MMIO(0x9888), 0x08b84000 },
	1431	{ _MMIO(0x9888), 0x0ab84000 },
	1432	{ _MMIO(0x9888), 0x00b9a000 },
	1433	{ _MMIO(0x9888), 0x06b98000 },
	1434	{ _MMIO(0x9888), 0x08b9a000 },
	1435	{ _MMIO(0x9888), 0x0ab9a000 },
	1436	{ _MMIO(0x9888), 0x0cb9a000 },
	1437	{ _MMIO(0x9888), 0x0eb9a000 },
	1438	{ _MMIO(0x9888), 0x02b9a000 },
	1439	{ _MMIO(0x9888), 0x04b9a000 },
	1440	{ _MMIO(0x9888), 0x01888000 },
	1441	{ _MMIO(0x9888), 0x0d88f800 },
	1442	{ _MMIO(0x9888), 0x0f88000f },
	1443	{ _MMIO(0x9888), 0x03888000 },
	1444	{ _MMIO(0x9888), 0x05888000 },
	1445	{ _MMIO(0x9888), 0x07888000 },
	1446	{ _MMIO(0x9888), 0x09888000 },
	1447	{ _MMIO(0x9888), 0x0b888000 },
	1448	{ _MMIO(0x9888), 0x238b5540 },
	1449	{ _MMIO(0x9888), 0x258baaa2 },
	1450	{ _MMIO(0x9888), 0x278b002a },
	1451	{ _MMIO(0x9888), 0x018c4000 },
	1452	{ _MMIO(0x9888), 0x0f8c4000 },
	1453	{ _MMIO(0x9888), 0x178c2000 },
	1454	{ _MMIO(0x9888), 0x198c5500 },
	1455	{ _MMIO(0x9888), 0x1b8c0015 },
	1456	{ _MMIO(0x9888), 0x038c4000 },
	1457	{ _MMIO(0x9888), 0x058c4000 },
	1458	{ _MMIO(0x9888), 0x078c4000 },
	1459	{ _MMIO(0x9888), 0x098c4000 },
	1460	{ _MMIO(0x9888), 0x0b8c4000 },
	1461	{ _MMIO(0x9888), 0x018da000 },
	1462	{ _MMIO(0x9888), 0x078d8000 },
	1463	{ _MMIO(0x9888), 0x098da000 },
	1464	{ _MMIO(0x9888), 0x0b8da000 },
	1465	{ _MMIO(0x9888), 0x0d8da000 },
	1466	{ _MMIO(0x9888), 0x0f8da000 },
	1467	{ _MMIO(0x9888), 0x038da000 },
	1468	{ _MMIO(0x9888), 0x058da000 },
	1469	{ _MMIO(0x9888), 0x1f85aa80 },
	1470	{ _MMIO(0x9888), 0x2185aaa2 },
	1471	{ _MMIO(0x9888), 0x2385002a },
	1472	{ _MMIO(0x9888), 0x01834000 },
	1473	{ _MMIO(0x9888), 0x0f834000 },
	1474	{ _MMIO(0x9888), 0x19835400 },
	1475	{ _MMIO(0x9888), 0x1b830155 },
	1476	{ _MMIO(0x9888), 0x03834000 },
	1477	{ _MMIO(0x9888), 0x05834000 },
	1478	{ _MMIO(0x9888), 0x07834000 },
	1479	{ _MMIO(0x9888), 0x09834000 },
	1480	{ _MMIO(0x9888), 0x0b834000 },
	1481	{ _MMIO(0x9888), 0x0184c000 },
	1482	{ _MMIO(0x9888), 0x07848000 },
	1483	{ _MMIO(0x9888), 0x0984c000 },
	1484	{ _MMIO(0x9888), 0x0b84c000 },
	1485	{ _MMIO(0x9888), 0x0d84c000 },
	1486	{ _MMIO(0x9888), 0x0f84c000 },
	1487	{ _MMIO(0x9888), 0x0384c000 },
	1488	{ _MMIO(0x9888), 0x0584c000 },
	1489	{ _MMIO(0x9888), 0x1180c000 },
	1490	{ _MMIO(0x9888), 0x17808000 },
	1491	{ _MMIO(0x9888), 0x1980c000 },
	1492	{ _MMIO(0x9888), 0x1b80c000 },
	1493	{ _MMIO(0x9888), 0x1d80c000 },
	1494	{ _MMIO(0x9888), 0x1f80c000 },
	1495	{ _MMIO(0x9888), 0x1380c000 },
	1496	{ _MMIO(0x9888), 0x1580c000 },
	1497	{ _MMIO(0xd24), 0x00000000 },
	1498	{ _MMIO(0x9888), 0x4d800000 },
	1499	{ _MMIO(0x9888), 0x3d800000 },
	1500	{ _MMIO(0x9888), 0x4f800000 },
	1501	{ _MMIO(0x9888), 0x43800000 },
	1502	{ _MMIO(0x9888), 0x51800000 },
	1503	{ _MMIO(0x9888), 0x45800000 },
	1504	{ _MMIO(0x9888), 0x53800000 },
	1505	{ _MMIO(0x9888), 0x47800420 },
	1506	{ _MMIO(0x9888), 0x21800000 },
	1507	{ _MMIO(0x9888), 0x31800000 },
	1508	{ _MMIO(0x9888), 0x3f800421 },
	1509	{ _MMIO(0x9888), 0x41800000 },
	1510	};
	1511
	1512	static const struct i915_oa_reg mux_config_compute_extended_5_subslices_0x20[] = {
	1513	{ _MMIO(0x9888), 0x10b800e0 },
	1514	{ _MMIO(0x9888), 0x14ba0160 },
	1515	{ _MMIO(0x9888), 0x16ba2800 },
	1516	{ _MMIO(0x9888), 0x18ba0120 },
	1517	{ _MMIO(0x9888), 0x0c9fa800 },
	1518	{ _MMIO(0x9888), 0x0e9faa2a },
	1519	{ _MMIO(0x9888), 0x109f02aa },
	1520	{ _MMIO(0x9888), 0x0eb8a5c1 },
	1521	{ _MMIO(0x9888), 0x00b8a100 },
	1522	{ _MMIO(0x9888), 0x02b8204c },
	1523	{ _MMIO(0x9888), 0x16b88000 },
	1524	{ _MMIO(0x9888), 0x18b802aa },
	1525	{ _MMIO(0x9888), 0x04b80000 },
	1526	{ _MMIO(0x9888), 0x06b80000 },
	1527	{ _MMIO(0x9888), 0x08b88000 },
	1528	{ _MMIO(0x9888), 0x0ab88000 },
	1529	{ _MMIO(0x9888), 0x00b9a000 },
	1530	{ _MMIO(0x9888), 0x06b98000 },
	1531	{ _MMIO(0x9888), 0x08b9a000 },
	1532	{ _MMIO(0x9888), 0x0ab9a000 },
	1533	{ _MMIO(0x9888), 0x0cb9a000 },
	1534	{ _MMIO(0x9888), 0x0eb9a000 },
	1535	{ _MMIO(0x9888), 0x02b9a000 },
	1536	{ _MMIO(0x9888), 0x04b9a000 },
	1537	{ _MMIO(0x9888), 0x00ba0011 },
	1538	{ _MMIO(0x9888), 0x06ba0900 },
	1539	{ _MMIO(0x9888), 0x08ba0a13 },
	1540	{ _MMIO(0x9888), 0x0aba0b15 },
	1541	{ _MMIO(0x9888), 0x0cba2317 },
	1542	{ _MMIO(0x9888), 0x04ba21b7 },
	1543	{ _MMIO(0x9888), 0x10ba0000 },
	1544	{ _MMIO(0x9888), 0x0eba0000 },
	1545	{ _MMIO(0x9888), 0x1aba0000 },
	1546	{ _MMIO(0x9888), 0x01888000 },
	1547	{ _MMIO(0x9888), 0x0d88f800 },
	1548	{ _MMIO(0x9888), 0x0f88000f },
	1549	{ _MMIO(0x9888), 0x03888000 },
	1550	{ _MMIO(0x9888), 0x05888000 },
	1551	{ _MMIO(0x9888), 0x07888000 },
	1552	{ _MMIO(0x9888), 0x09888000 },
	1553	{ _MMIO(0x9888), 0x0b888000 },
	1554	{ _MMIO(0x9888), 0x238b5540 },
	1555	{ _MMIO(0x9888), 0x258baaa2 },
	1556	{ _MMIO(0x9888), 0x278b002a },
	1557	{ _MMIO(0x9888), 0x018c4000 },
	1558	{ _MMIO(0x9888), 0x0f8c4000 },
	1559	{ _MMIO(0x9888), 0x178c2000 },
	1560	{ _MMIO(0x9888), 0x198c5500 },
	1561	{ _MMIO(0x9888), 0x1b8c0015 },
	1562	{ _MMIO(0x9888), 0x038c4000 },
	1563	{ _MMIO(0x9888), 0x058c4000 },
	1564	{ _MMIO(0x9888), 0x078c4000 },
	1565	{ _MMIO(0x9888), 0x098c4000 },
	1566	{ _MMIO(0x9888), 0x0b8c4000 },
	1567	{ _MMIO(0x9888), 0x018da000 },
	1568	{ _MMIO(0x9888), 0x078d8000 },
	1569	{ _MMIO(0x9888), 0x098da000 },
	1570	{ _MMIO(0x9888), 0x0b8da000 },
	1571	{ _MMIO(0x9888), 0x0d8da000 },
	1572	{ _MMIO(0x9888), 0x0f8da000 },
	1573	{ _MMIO(0x9888), 0x038da000 },
	1574	{ _MMIO(0x9888), 0x058da000 },
	1575	{ _MMIO(0x9888), 0x1f85aa80 },
	1576	{ _MMIO(0x9888), 0x2185aaa2 },
	1577	{ _MMIO(0x9888), 0x2385002a },
	1578	{ _MMIO(0x9888), 0x01834000 },
	1579	{ _MMIO(0x9888), 0x0f834000 },
	1580	{ _MMIO(0x9888), 0x19835400 },
	1581	{ _MMIO(0x9888), 0x1b830155 },
	1582	{ _MMIO(0x9888), 0x03834000 },
	1583	{ _MMIO(0x9888), 0x05834000 },
	1584	{ _MMIO(0x9888), 0x07834000 },
	1585	{ _MMIO(0x9888), 0x09834000 },
	1586	{ _MMIO(0x9888), 0x0b834000 },
	1587	{ _MMIO(0x9888), 0x0184c000 },
	1588	{ _MMIO(0x9888), 0x07848000 },
	1589	{ _MMIO(0x9888), 0x0984c000 },
	1590	{ _MMIO(0x9888), 0x0b84c000 },
	1591	{ _MMIO(0x9888), 0x0d84c000 },
	1592	{ _MMIO(0x9888), 0x0f84c000 },
	1593	{ _MMIO(0x9888), 0x0384c000 },
	1594	{ _MMIO(0x9888), 0x0584c000 },
	1595	{ _MMIO(0x9888), 0x1180c000 },
	1596	{ _MMIO(0x9888), 0x17808000 },
	1597	{ _MMIO(0x9888), 0x1980c000 },
	1598	{ _MMIO(0x9888), 0x1b80c000 },
	1599	{ _MMIO(0x9888), 0x1d80c000 },
	1600	{ _MMIO(0x9888), 0x1f80c000 },
	1601	{ _MMIO(0x9888), 0x1380c000 },
	1602	{ _MMIO(0x9888), 0x1580c000 },
	1603	{ _MMIO(0xd24), 0x00000000 },
	1604	{ _MMIO(0x9888), 0x4d800000 },
	1605	{ _MMIO(0x9888), 0x3d800000 },
	1606	{ _MMIO(0x9888), 0x4f800000 },
	1607	{ _MMIO(0x9888), 0x43800000 },
	1608	{ _MMIO(0x9888), 0x51800000 },
	1609	{ _MMIO(0x9888), 0x45800000 },
	1610	{ _MMIO(0x9888), 0x53800000 },
	1611	{ _MMIO(0x9888), 0x47800420 },
	1612	{ _MMIO(0x9888), 0x21800000 },
	1613	{ _MMIO(0x9888), 0x31800000 },
	1614	{ _MMIO(0x9888), 0x3f800421 },
	1615	{ _MMIO(0x9888), 0x41800000 },
	1616	};
	1617
	1618	static int
	1619	get_compute_extended_mux_config(struct drm_i915_private *dev_priv,
	1620	const struct i915_oa_reg **regs,
	1621	int *lens)
	1622	{
	1623	int n = 0;
	1624
	1625	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 6);
	1626	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 6);
	1627
	1628	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x01) {
	1629	regs[n] = mux_config_compute_extended_0_subslices_0x01;
	1630	lens[n] = ARRAY_SIZE(mux_config_compute_extended_0_subslices_0x01);
	1631	n++;
	1632	}
	1633	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x08) {
	1634	regs[n] = mux_config_compute_extended_1_subslices_0x08;
	1635	lens[n] = ARRAY_SIZE(mux_config_compute_extended_1_subslices_0x08);
	1636	n++;
	1637	}
	1638	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x02) {
	1639	regs[n] = mux_config_compute_extended_2_subslices_0x02;
	1640	lens[n] = ARRAY_SIZE(mux_config_compute_extended_2_subslices_0x02);
	1641	n++;
	1642	}
	1643	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x10) {
	1644	regs[n] = mux_config_compute_extended_3_subslices_0x10;
	1645	lens[n] = ARRAY_SIZE(mux_config_compute_extended_3_subslices_0x10);
	1646	n++;
	1647	}
	1648	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x04) {
	1649	regs[n] = mux_config_compute_extended_4_subslices_0x04;
	1650	lens[n] = ARRAY_SIZE(mux_config_compute_extended_4_subslices_0x04);
	1651	n++;
	1652	}
	1653	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x20) {
	1654	regs[n] = mux_config_compute_extended_5_subslices_0x20;
	1655	lens[n] = ARRAY_SIZE(mux_config_compute_extended_5_subslices_0x20);
	1656	n++;
	1657	}
	1658
	1659	return n;
	1660	}
	1661
	1662	static const struct i915_oa_reg b_counter_config_compute_l3_cache[] = {
	1663	{ _MMIO(0x2710), 0x00000000 },
	1664	{ _MMIO(0x2714), 0x30800000 },
	1665	{ _MMIO(0x2720), 0x00000000 },
	1666	{ _MMIO(0x2724), 0x30800000 },
	1667	{ _MMIO(0x2740), 0x00000000 },
	1668	{ _MMIO(0x2770), 0x0007fffa },
	1669	{ _MMIO(0x2774), 0x0000fefe },
	1670	{ _MMIO(0x2778), 0x0007fffa },
	1671	{ _MMIO(0x277c), 0x0000fefd },
	1672	{ _MMIO(0x2790), 0x0007fffa },
	1673	{ _MMIO(0x2794), 0x0000fbef },
	1674	{ _MMIO(0x2798), 0x0007fffa },
	1675	{ _MMIO(0x279c), 0x0000fbdf },
	1676	};
	1677
	1678	static const struct i915_oa_reg flex_eu_config_compute_l3_cache[] = {
	1679	{ _MMIO(0xe458), 0x00005004 },
	1680	{ _MMIO(0xe558), 0x00000003 },
	1681	{ _MMIO(0xe658), 0x00002001 },
	1682	{ _MMIO(0xe758), 0x00101100 },
	1683	{ _MMIO(0xe45c), 0x00201200 },
	1684	{ _MMIO(0xe55c), 0x00301300 },
	1685	{ _MMIO(0xe65c), 0x00401400 },
	1686	};
	1687
	1688	static const struct i915_oa_reg mux_config_compute_l3_cache[] = {
	1689	{ _MMIO(0x9888), 0x143f00b3 },
	1690	{ _MMIO(0x9888), 0x14bf00b3 },
	1691	{ _MMIO(0x9888), 0x138303c0 },
	1692	{ _MMIO(0x9888), 0x3b800060 },
	1693	{ _MMIO(0x9888), 0x3d800805 },
	1694	{ _MMIO(0x9888), 0x003f0029 },
	1695	{ _MMIO(0x9888), 0x063f1400 },
	1696	{ _MMIO(0x9888), 0x083f1225 },
	1697	{ _MMIO(0x9888), 0x0e3f1327 },
	1698	{ _MMIO(0x9888), 0x103f0000 },
	1699	{ _MMIO(0x9888), 0x005a4000 },
	1700	{ _MMIO(0x9888), 0x065a8000 },
	1701	{ _MMIO(0x9888), 0x085ac000 },
	1702	{ _MMIO(0x9888), 0x0e5ac000 },
	1703	{ _MMIO(0x9888), 0x001d4000 },
	1704	{ _MMIO(0x9888), 0x061d8000 },
	1705	{ _MMIO(0x9888), 0x081dc000 },
	1706	{ _MMIO(0x9888), 0x0e1dc000 },
	1707	{ _MMIO(0x9888), 0x0c1f0800 },
	1708	{ _MMIO(0x9888), 0x0e1f2a00 },
	1709	{ _MMIO(0x9888), 0x101f0280 },
	1710	{ _MMIO(0x9888), 0x00391000 },
	1711	{ _MMIO(0x9888), 0x06394000 },
	1712	{ _MMIO(0x9888), 0x08395000 },
	1713	{ _MMIO(0x9888), 0x0e395000 },
	1714	{ _MMIO(0x9888), 0x0abf1429 },
	1715	{ _MMIO(0x9888), 0x0cbf1225 },
	1716	{ _MMIO(0x9888), 0x00bf1380 },
	1717	{ _MMIO(0x9888), 0x02bf0026 },
	1718	{ _MMIO(0x9888), 0x10bf0000 },
	1719	{ _MMIO(0x9888), 0x0adac000 },
	1720	{ _MMIO(0x9888), 0x0cdac000 },
	1721	{ _MMIO(0x9888), 0x00da8000 },
	1722	{ _MMIO(0x9888), 0x02da4000 },
	1723	{ _MMIO(0x9888), 0x0a9dc000 },
	1724	{ _MMIO(0x9888), 0x0c9dc000 },
	1725	{ _MMIO(0x9888), 0x009d8000 },
	1726	{ _MMIO(0x9888), 0x029d4000 },
	1727	{ _MMIO(0x9888), 0x0e9f8000 },
	1728	{ _MMIO(0x9888), 0x109f002a },
	1729	{ _MMIO(0x9888), 0x0c9fa000 },
	1730	{ _MMIO(0x9888), 0x0ab95000 },
	1731	{ _MMIO(0x9888), 0x0cb95000 },
	1732	{ _MMIO(0x9888), 0x00b94000 },
	1733	{ _MMIO(0x9888), 0x02b91000 },
	1734	{ _MMIO(0x9888), 0x0d88c000 },
	1735	{ _MMIO(0x9888), 0x0f880003 },
	1736	{ _MMIO(0x9888), 0x03888000 },
	1737	{ _MMIO(0x9888), 0x05888000 },
	1738	{ _MMIO(0x9888), 0x018a8000 },
	1739	{ _MMIO(0x9888), 0x0f8a8000 },
	1740	{ _MMIO(0x9888), 0x198a8000 },
	1741	{ _MMIO(0x9888), 0x1b8a8020 },
	1742	{ _MMIO(0x9888), 0x1d8a0002 },
	1743	{ _MMIO(0x9888), 0x238b0520 },
	1744	{ _MMIO(0x9888), 0x258ba950 },
	1745	{ _MMIO(0x9888), 0x278b0016 },
	1746	{ _MMIO(0x9888), 0x198c5400 },
	1747	{ _MMIO(0x9888), 0x1b8c0001 },
	1748	{ _MMIO(0x9888), 0x038c4000 },
	1749	{ _MMIO(0x9888), 0x058c4000 },
	1750	{ _MMIO(0x9888), 0x0b8da000 },
	1751	{ _MMIO(0x9888), 0x0d8da000 },
	1752	{ _MMIO(0x9888), 0x018d8000 },
	1753	{ _MMIO(0x9888), 0x038d2000 },
	1754	{ _MMIO(0x9888), 0x1f85aa80 },
	1755	{ _MMIO(0x9888), 0x2185aaa0 },
	1756	{ _MMIO(0x9888), 0x2385002a },
	1757	{ _MMIO(0x9888), 0x03835180 },
	1758	{ _MMIO(0x9888), 0x05834022 },
	1759	{ _MMIO(0x9888), 0x11830000 },
	1760	{ _MMIO(0x9888), 0x01834000 },
	1761	{ _MMIO(0x9888), 0x0f834000 },
	1762	{ _MMIO(0x9888), 0x19835400 },
	1763	{ _MMIO(0x9888), 0x1b830155 },
	1764	{ _MMIO(0x9888), 0x07830000 },
	1765	{ _MMIO(0x9888), 0x09830000 },
	1766	{ _MMIO(0x9888), 0x0184c000 },
	1767	{ _MMIO(0x9888), 0x07848000 },
	1768	{ _MMIO(0x9888), 0x0984c000 },
	1769	{ _MMIO(0x9888), 0x0b84c000 },
	1770	{ _MMIO(0x9888), 0x0d84c000 },
	1771	{ _MMIO(0x9888), 0x0f84c000 },
	1772	{ _MMIO(0x9888), 0x0384c000 },
	1773	{ _MMIO(0x9888), 0x05844000 },
	1774	{ _MMIO(0x9888), 0x1b80c137 },
	1775	{ _MMIO(0x9888), 0x1d80c147 },
	1776	{ _MMIO(0x9888), 0x21800000 },
	1777	{ _MMIO(0x9888), 0x1180c000 },
	1778	{ _MMIO(0x9888), 0x17808000 },
	1779	{ _MMIO(0x9888), 0x1980c000 },
	1780	{ _MMIO(0x9888), 0x1f80c000 },
	1781	{ _MMIO(0x9888), 0x1380c000 },
	1782	{ _MMIO(0x9888), 0x15804000 },
	1783	{ _MMIO(0xd24), 0x00000000 },
	1784	{ _MMIO(0x9888), 0x4d801000 },
	1785	{ _MMIO(0x9888), 0x4f800111 },
	1786	{ _MMIO(0x9888), 0x43800842 },
	1787	{ _MMIO(0x9888), 0x51800000 },
	1788	{ _MMIO(0x9888), 0x45800000 },
	1789	{ _MMIO(0x9888), 0x53800000 },
	1790	{ _MMIO(0x9888), 0x47800840 },
	1791	{ _MMIO(0x9888), 0x31800000 },
	1792	{ _MMIO(0x9888), 0x3f800800 },
	1793	{ _MMIO(0x9888), 0x418014a2 },
	1794	};
	1795
	1796	static int
	1797	get_compute_l3_cache_mux_config(struct drm_i915_private *dev_priv,
	1798	const struct i915_oa_reg **regs,
	1799	int *lens)
	1800	{
	1801	int n = 0;
	1802
	1803	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1804	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1805
	1806	regs[n] = mux_config_compute_l3_cache;
	1807	lens[n] = ARRAY_SIZE(mux_config_compute_l3_cache);
	1808	n++;
	1809
	1810	return n;
	1811	}
	1812
	1813	static const struct i915_oa_reg b_counter_config_data_port_reads_coalescing[] = {
	1814	{ _MMIO(0x2724), 0xf0800000 },
	1815	{ _MMIO(0x2720), 0x00000000 },
	1816	{ _MMIO(0x2714), 0xf0800000 },
	1817	{ _MMIO(0x2710), 0x00000000 },
	1818	{ _MMIO(0x274c), 0xba98ba98 },
	1819	{ _MMIO(0x2748), 0xba98ba98 },
	1820	{ _MMIO(0x2744), 0x00003377 },
	1821	{ _MMIO(0x2740), 0x00000000 },
	1822	{ _MMIO(0x2770), 0x0007fff2 },
	1823	{ _MMIO(0x2774), 0x00007ff0 },
	1824	{ _MMIO(0x2778), 0x0007ffe2 },
	1825	{ _MMIO(0x277c), 0x00007ff0 },
	1826	{ _MMIO(0x2780), 0x0007ffc2 },
	1827	{ _MMIO(0x2784), 0x00007ff0 },
	1828	{ _MMIO(0x2788), 0x0007ff82 },
	1829	{ _MMIO(0x278c), 0x00007ff0 },
	1830	{ _MMIO(0x2790), 0x0007fffa },
	1831	{ _MMIO(0x2794), 0x0000bfef },
	1832	{ _MMIO(0x2798), 0x0007fffa },
	1833	{ _MMIO(0x279c), 0x0000bfdf },
	1834	{ _MMIO(0x27a0), 0x0007fffa },
	1835	{ _MMIO(0x27a4), 0x0000bfbf },
	1836	{ _MMIO(0x27a8), 0x0007fffa },
	1837	{ _MMIO(0x27ac), 0x0000bf7f },
	1838	};
	1839
	1840	static const struct i915_oa_reg flex_eu_config_data_port_reads_coalescing[] = {
	1841	{ _MMIO(0xe458), 0x00005004 },
	1842	{ _MMIO(0xe558), 0x00000003 },
	1843	{ _MMIO(0xe658), 0x00002001 },
	1844	{ _MMIO(0xe758), 0x00778008 },
	1845	{ _MMIO(0xe45c), 0x00088078 },
	1846	{ _MMIO(0xe55c), 0x00808708 },
	1847	{ _MMIO(0xe65c), 0x00a08908 },
	1848	};
	1849
	1850	static const struct i915_oa_reg mux_config_data_port_reads_coalescing_0_subslices_0x01[] = {
	1851	{ _MMIO(0x9888), 0x103d0005 },
	1852	{ _MMIO(0x9888), 0x163d240b },
	1853	{ _MMIO(0x9888), 0x1058022f },
	1854	{ _MMIO(0x9888), 0x185b5520 },
	1855	{ _MMIO(0x9888), 0x198b0003 },
	1856	{ _MMIO(0x9888), 0x005cc000 },
	1857	{ _MMIO(0x9888), 0x065cc000 },
	1858	{ _MMIO(0x9888), 0x085cc000 },
	1859	{ _MMIO(0x9888), 0x0a5cc000 },
	1860	{ _MMIO(0x9888), 0x0c5cc000 },
	1861	{ _MMIO(0x9888), 0x0e5cc000 },
	1862	{ _MMIO(0x9888), 0x025c4000 },
	1863	{ _MMIO(0x9888), 0x045c8000 },
	1864	{ _MMIO(0x9888), 0x003d0000 },
	1865	{ _MMIO(0x9888), 0x063d00b0 },
	1866	{ _MMIO(0x9888), 0x083d0182 },
	1867	{ _MMIO(0x9888), 0x0a3d10a0 },
	1868	{ _MMIO(0x9888), 0x0c3d11a2 },
	1869	{ _MMIO(0x9888), 0x0e3d0000 },
	1870	{ _MMIO(0x9888), 0x183d0000 },
	1871	{ _MMIO(0x9888), 0x1a3d0000 },
	1872	{ _MMIO(0x9888), 0x0e582242 },
	1873	{ _MMIO(0x9888), 0x00586700 },
	1874	{ _MMIO(0x9888), 0x0258004f },
	1875	{ _MMIO(0x9888), 0x0658c000 },
	1876	{ _MMIO(0x9888), 0x0858c000 },
	1877	{ _MMIO(0x9888), 0x0a58c000 },
	1878	{ _MMIO(0x9888), 0x0c58c000 },
	1879	{ _MMIO(0x9888), 0x045b6300 },
	1880	{ _MMIO(0x9888), 0x105b0000 },
	1881	{ _MMIO(0x9888), 0x005b4000 },
	1882	{ _MMIO(0x9888), 0x0e5b4000 },
	1883	{ _MMIO(0x9888), 0x1a5b0155 },
	1884	{ _MMIO(0x9888), 0x025b4000 },
	1885	{ _MMIO(0x9888), 0x0a5b0000 },
	1886	{ _MMIO(0x9888), 0x0c5b4000 },
	1887	{ _MMIO(0x9888), 0x0c1fa800 },
	1888	{ _MMIO(0x9888), 0x0e1faaa0 },
	1889	{ _MMIO(0x9888), 0x101f02aa },
	1890	{ _MMIO(0x9888), 0x00384000 },
	1891	{ _MMIO(0x9888), 0x0e384000 },
	1892	{ _MMIO(0x9888), 0x16384000 },
	1893	{ _MMIO(0x9888), 0x18381555 },
	1894	{ _MMIO(0x9888), 0x02384000 },
	1895	{ _MMIO(0x9888), 0x04384000 },
	1896	{ _MMIO(0x9888), 0x0a384000 },
	1897	{ _MMIO(0x9888), 0x0c384000 },
	1898	{ _MMIO(0x9888), 0x0039a000 },
	1899	{ _MMIO(0x9888), 0x0639a000 },
	1900	{ _MMIO(0x9888), 0x0839a000 },
	1901	{ _MMIO(0x9888), 0x0a39a000 },
	1902	{ _MMIO(0x9888), 0x0c39a000 },
	1903	{ _MMIO(0x9888), 0x0e39a000 },
	1904	{ _MMIO(0x9888), 0x02392000 },
	1905	{ _MMIO(0x9888), 0x04398000 },
	1906	{ _MMIO(0x9888), 0x018a8000 },
	1907	{ _MMIO(0x9888), 0x0f8a8000 },
	1908	{ _MMIO(0x9888), 0x198a8000 },
	1909	{ _MMIO(0x9888), 0x1b8aaaa0 },
	1910	{ _MMIO(0x9888), 0x1d8a0002 },
	1911	{ _MMIO(0x9888), 0x038a8000 },
	1912	{ _MMIO(0x9888), 0x058a8000 },
	1913	{ _MMIO(0x9888), 0x0b8a8000 },
	1914	{ _MMIO(0x9888), 0x0d8a8000 },
	1915	{ _MMIO(0x9888), 0x038b6300 },
	1916	{ _MMIO(0x9888), 0x058b0062 },
	1917	{ _MMIO(0x9888), 0x118b0000 },
	1918	{ _MMIO(0x9888), 0x238b02a0 },
	1919	{ _MMIO(0x9888), 0x258b5555 },
	1920	{ _MMIO(0x9888), 0x278b0015 },
	1921	{ _MMIO(0x9888), 0x1f85aa80 },
	1922	{ _MMIO(0x9888), 0x2185aaaa },
	1923	{ _MMIO(0x9888), 0x2385002a },
	1924	{ _MMIO(0x9888), 0x01834000 },
	1925	{ _MMIO(0x9888), 0x0f834000 },
	1926	{ _MMIO(0x9888), 0x19835400 },
	1927	{ _MMIO(0x9888), 0x1b830155 },
	1928	{ _MMIO(0x9888), 0x03834000 },
	1929	{ _MMIO(0x9888), 0x05834000 },
	1930	{ _MMIO(0x9888), 0x07834000 },
	1931	{ _MMIO(0x9888), 0x09834000 },
	1932	{ _MMIO(0x9888), 0x0b834000 },
	1933	{ _MMIO(0x9888), 0x0d834000 },
	1934	{ _MMIO(0x9888), 0x0184c000 },
	1935	{ _MMIO(0x9888), 0x0784c000 },
	1936	{ _MMIO(0x9888), 0x0984c000 },
	1937	{ _MMIO(0x9888), 0x0b84c000 },
	1938	{ _MMIO(0x9888), 0x0d84c000 },
	1939	{ _MMIO(0x9888), 0x0f84c000 },
	1940	{ _MMIO(0x9888), 0x0384c000 },
	1941	{ _MMIO(0x9888), 0x0584c000 },
	1942	{ _MMIO(0x9888), 0x1180c000 },
	1943	{ _MMIO(0x9888), 0x1780c000 },
	1944	{ _MMIO(0x9888), 0x1980c000 },
	1945	{ _MMIO(0x9888), 0x1b80c000 },
	1946	{ _MMIO(0x9888), 0x1d80c000 },
	1947	{ _MMIO(0x9888), 0x1f80c000 },
	1948	{ _MMIO(0x9888), 0x1380c000 },
	1949	{ _MMIO(0x9888), 0x1580c000 },
	1950	{ _MMIO(0xd24), 0x00000000 },
	1951	{ _MMIO(0x9888), 0x4d801000 },
	1952	{ _MMIO(0x9888), 0x3d800000 },
	1953	{ _MMIO(0x9888), 0x4f800001 },
	1954	{ _MMIO(0x9888), 0x43800000 },
	1955	{ _MMIO(0x9888), 0x51800000 },
	1956	{ _MMIO(0x9888), 0x45800000 },
	1957	{ _MMIO(0x9888), 0x53800000 },
	1958	{ _MMIO(0x9888), 0x47800420 },
	1959	{ _MMIO(0x9888), 0x21800000 },
	1960	{ _MMIO(0x9888), 0x31800000 },
	1961	{ _MMIO(0x9888), 0x3f800421 },
	1962	{ _MMIO(0x9888), 0x41800041 },
	1963	};
	1964
	1965	static int
	1966	get_data_port_reads_coalescing_mux_config(struct drm_i915_private *dev_priv,
	1967	const struct i915_oa_reg **regs,
	1968	int *lens)
	1969	{
	1970	int n = 0;
	1971
	1972	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1973	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1974
	1975	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x01) {
	1976	regs[n] = mux_config_data_port_reads_coalescing_0_subslices_0x01;
	1977	lens[n] = ARRAY_SIZE(mux_config_data_port_reads_coalescing_0_subslices_0x01);
	1978	n++;
	1979	}
	1980
	1981	return n;
	1982	}
	1983
	1984	static const struct i915_oa_reg b_counter_config_data_port_writes_coalescing[] = {
	1985	{ _MMIO(0x2724), 0xf0800000 },
	1986	{ _MMIO(0x2720), 0x00000000 },
	1987	{ _MMIO(0x2714), 0xf0800000 },
	1988	{ _MMIO(0x2710), 0x00000000 },
	1989	{ _MMIO(0x274c), 0xba98ba98 },
	1990	{ _MMIO(0x2748), 0xba98ba98 },
	1991	{ _MMIO(0x2744), 0x00003377 },
	1992	{ _MMIO(0x2740), 0x00000000 },
	1993	{ _MMIO(0x2770), 0x0007ff72 },
	1994	{ _MMIO(0x2774), 0x0000bfd0 },
	1995	{ _MMIO(0x2778), 0x0007ff62 },
	1996	{ _MMIO(0x277c), 0x0000bfd0 },
	1997	{ _MMIO(0x2780), 0x0007ff42 },
	1998	{ _MMIO(0x2784), 0x0000bfd0 },
	1999	{ _MMIO(0x2788), 0x0007ff02 },
	2000	{ _MMIO(0x278c), 0x0000bfd0 },
	2001	{ _MMIO(0x2790), 0x0005fff2 },
	2002	{ _MMIO(0x2794), 0x0000bfd0 },
	2003	{ _MMIO(0x2798), 0x0005ffe2 },
	2004	{ _MMIO(0x279c), 0x0000bfd0 },
	2005	{ _MMIO(0x27a0), 0x0005ffc2 },
	2006	{ _MMIO(0x27a4), 0x0000bfd0 },
	2007	{ _MMIO(0x27a8), 0x0005ff82 },
	2008	{ _MMIO(0x27ac), 0x0000bfd0 },
	2009	};
	2010
	2011	static const struct i915_oa_reg flex_eu_config_data_port_writes_coalescing[] = {
	2012	{ _MMIO(0xe458), 0x00005004 },
	2013	{ _MMIO(0xe558), 0x00000003 },
	2014	{ _MMIO(0xe658), 0x00002001 },
	2015	{ _MMIO(0xe758), 0x00778008 },
	2016	{ _MMIO(0xe45c), 0x00088078 },
	2017	{ _MMIO(0xe55c), 0x00808708 },
	2018	{ _MMIO(0xe65c), 0x00a08908 },
	2019	};
	2020
	2021	static const struct i915_oa_reg mux_config_data_port_writes_coalescing_0_subslices_0x01[] = {
	2022	{ _MMIO(0x9888), 0x103d0005 },
	2023	{ _MMIO(0x9888), 0x143d0120 },
	2024	{ _MMIO(0x9888), 0x163d2400 },
	2025	{ _MMIO(0x9888), 0x1058022f },
	2026	{ _MMIO(0x9888), 0x105b0000 },
	2027	{ _MMIO(0x9888), 0x198b0003 },
	2028	{ _MMIO(0x9888), 0x005cc000 },
	2029	{ _MMIO(0x9888), 0x065cc000 },
	2030	{ _MMIO(0x9888), 0x085cc000 },
	2031	{ _MMIO(0x9888), 0x0a5cc000 },
	2032	{ _MMIO(0x9888), 0x0e5cc000 },
	2033	{ _MMIO(0x9888), 0x025c4000 },
	2034	{ _MMIO(0x9888), 0x045c8000 },
	2035	{ _MMIO(0x9888), 0x003d0000 },
	2036	{ _MMIO(0x9888), 0x063d0094 },
	2037	{ _MMIO(0x9888), 0x083d0182 },
	2038	{ _MMIO(0x9888), 0x0a3d1814 },
	2039	{ _MMIO(0x9888), 0x0e3d0000 },
	2040	{ _MMIO(0x9888), 0x183d0000 },
	2041	{ _MMIO(0x9888), 0x1a3d0000 },
	2042	{ _MMIO(0x9888), 0x0c3d0000 },
	2043	{ _MMIO(0x9888), 0x0e582242 },
	2044	{ _MMIO(0x9888), 0x00586700 },
	2045	{ _MMIO(0x9888), 0x0258004f },
	2046	{ _MMIO(0x9888), 0x0658c000 },
	2047	{ _MMIO(0x9888), 0x0858c000 },
	2048	{ _MMIO(0x9888), 0x0a58c000 },
	2049	{ _MMIO(0x9888), 0x045b6a80 },
	2050	{ _MMIO(0x9888), 0x005b4000 },
	2051	{ _MMIO(0x9888), 0x0e5b4000 },
	2052	{ _MMIO(0x9888), 0x185b5400 },
	2053	{ _MMIO(0x9888), 0x1a5b0141 },
	2054	{ _MMIO(0x9888), 0x025b4000 },
	2055	{ _MMIO(0x9888), 0x0a5b0000 },
	2056	{ _MMIO(0x9888), 0x0c5b4000 },
	2057	{ _MMIO(0x9888), 0x0c1fa800 },
	2058	{ _MMIO(0x9888), 0x0e1faaa0 },
	2059	{ _MMIO(0x9888), 0x101f0282 },
	2060	{ _MMIO(0x9888), 0x00384000 },
	2061	{ _MMIO(0x9888), 0x0e384000 },
	2062	{ _MMIO(0x9888), 0x16384000 },
	2063	{ _MMIO(0x9888), 0x18381415 },
	2064	{ _MMIO(0x9888), 0x02384000 },
	2065	{ _MMIO(0x9888), 0x04384000 },
	2066	{ _MMIO(0x9888), 0x0a384000 },
	2067	{ _MMIO(0x9888), 0x0c384000 },
	2068	{ _MMIO(0x9888), 0x0039a000 },
	2069	{ _MMIO(0x9888), 0x0639a000 },
	2070	{ _MMIO(0x9888), 0x0839a000 },
	2071	{ _MMIO(0x9888), 0x0a39a000 },
	2072	{ _MMIO(0x9888), 0x0e39a000 },
	2073	{ _MMIO(0x9888), 0x02392000 },
	2074	{ _MMIO(0x9888), 0x04398000 },
	2075	{ _MMIO(0x9888), 0x018a8000 },
	2076	{ _MMIO(0x9888), 0x0f8a8000 },
	2077	{ _MMIO(0x9888), 0x198a8000 },
	2078	{ _MMIO(0x9888), 0x1b8a82a0 },
	2079	{ _MMIO(0x9888), 0x1d8a0002 },
	2080	{ _MMIO(0x9888), 0x038a8000 },
	2081	{ _MMIO(0x9888), 0x058a8000 },
	2082	{ _MMIO(0x9888), 0x0b8a8000 },
	2083	{ _MMIO(0x9888), 0x0d8a8000 },
	2084	{ _MMIO(0x9888), 0x038b6300 },
	2085	{ _MMIO(0x9888), 0x058b0062 },
	2086	{ _MMIO(0x9888), 0x118b0000 },
	2087	{ _MMIO(0x9888), 0x238b02a0 },
	2088	{ _MMIO(0x9888), 0x258b1555 },
	2089	{ _MMIO(0x9888), 0x278b0014 },
	2090	{ _MMIO(0x9888), 0x1f85aa80 },
	2091	{ _MMIO(0x9888), 0x21852aaa },
	2092	{ _MMIO(0x9888), 0x23850028 },
	2093	{ _MMIO(0x9888), 0x01834000 },
	2094	{ _MMIO(0x9888), 0x0f834000 },
	2095	{ _MMIO(0x9888), 0x19835400 },
	2096	{ _MMIO(0x9888), 0x1b830141 },
	2097	{ _MMIO(0x9888), 0x03834000 },
	2098	{ _MMIO(0x9888), 0x05834000 },
	2099	{ _MMIO(0x9888), 0x07834000 },
	2100	{ _MMIO(0x9888), 0x09834000 },
	2101	{ _MMIO(0x9888), 0x0b834000 },
	2102	{ _MMIO(0x9888), 0x0d834000 },
	2103	{ _MMIO(0x9888), 0x0184c000 },
	2104	{ _MMIO(0x9888), 0x0784c000 },
	2105	{ _MMIO(0x9888), 0x0984c000 },
	2106	{ _MMIO(0x9888), 0x0b84c000 },
	2107	{ _MMIO(0x9888), 0x0f84c000 },
	2108	{ _MMIO(0x9888), 0x0384c000 },
	2109	{ _MMIO(0x9888), 0x0584c000 },
	2110	{ _MMIO(0x9888), 0x1180c000 },
	2111	{ _MMIO(0x9888), 0x1780c000 },
	2112	{ _MMIO(0x9888), 0x1980c000 },
	2113	{ _MMIO(0x9888), 0x1b80c000 },
	2114	{ _MMIO(0x9888), 0x1f80c000 },
	2115	{ _MMIO(0x9888), 0x1380c000 },
	2116	{ _MMIO(0x9888), 0x1580c000 },
	2117	{ _MMIO(0xd24), 0x00000000 },
	2118	{ _MMIO(0x9888), 0x4d801000 },
	2119	{ _MMIO(0x9888), 0x3d800000 },
	2120	{ _MMIO(0x9888), 0x4f800001 },
	2121	{ _MMIO(0x9888), 0x43800000 },
	2122	{ _MMIO(0x9888), 0x51800000 },
	2123	{ _MMIO(0x9888), 0x45800000 },
	2124	{ _MMIO(0x9888), 0x21800000 },
	2125	{ _MMIO(0x9888), 0x31800000 },
	2126	{ _MMIO(0x9888), 0x53800000 },
	2127	{ _MMIO(0x9888), 0x47800420 },
	2128	{ _MMIO(0x9888), 0x3f800421 },
	2129	{ _MMIO(0x9888), 0x41800041 },
	2130	};
	2131
	2132	static int
	2133	get_data_port_writes_coalescing_mux_config(struct drm_i915_private *dev_priv,
	2134	const struct i915_oa_reg **regs,
	2135	int *lens)
	2136	{
	2137	int n = 0;
	2138
	2139	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2140	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2141
	2142	if (INTEL_INFO(dev_priv)->sseu.subslice_mask & 0x01) {
	2143	regs[n] = mux_config_data_port_writes_coalescing_0_subslices_0x01;
	2144	lens[n] = ARRAY_SIZE(mux_config_data_port_writes_coalescing_0_subslices_0x01);
	2145	n++;
	2146	}
	2147
	2148	return n;
	2149	}
	2150
	2151	static const struct i915_oa_reg b_counter_config_hdc_and_sf[] = {
	2152	{ _MMIO(0x2740), 0x00000000 },
	2153	{ _MMIO(0x2744), 0x00800000 },
	2154	{ _MMIO(0x2710), 0x00000000 },
	2155	{ _MMIO(0x2714), 0x10800000 },
	2156	{ _MMIO(0x2720), 0x00000000 },
	2157	{ _MMIO(0x2724), 0x00800000 },
	2158	{ _MMIO(0x2770), 0x00000002 },
	2159	{ _MMIO(0x2774), 0x0000fff7 },
	2160	};
	2161
	2162	static const struct i915_oa_reg flex_eu_config_hdc_and_sf[] = {
	2163	{ _MMIO(0xe458), 0x00005004 },
	2164	{ _MMIO(0xe558), 0x00010003 },
	2165	{ _MMIO(0xe658), 0x00012011 },
	2166	{ _MMIO(0xe758), 0x00015014 },
	2167	{ _MMIO(0xe45c), 0x00051050 },
	2168	{ _MMIO(0xe55c), 0x00053052 },
	2169	{ _MMIO(0xe65c), 0x00055054 },
	2170	};
	2171
	2172	static const struct i915_oa_reg mux_config_hdc_and_sf[] = {
	2173	{ _MMIO(0x9888), 0x105c0232 },
	2174	{ _MMIO(0x9888), 0x10580232 },
	2175	{ _MMIO(0x9888), 0x10380232 },
	2176	{ _MMIO(0x9888), 0x10dc0232 },
	2177	{ _MMIO(0x9888), 0x10d80232 },
	2178	{ _MMIO(0x9888), 0x10b80232 },
	2179	{ _MMIO(0x9888), 0x118e4400 },
	2180	{ _MMIO(0x9888), 0x025c6080 },
	2181	{ _MMIO(0x9888), 0x045c004b },
	2182	{ _MMIO(0x9888), 0x005c8000 },
	2183	{ _MMIO(0x9888), 0x00582080 },
	2184	{ _MMIO(0x9888), 0x0258004b },
	2185	{ _MMIO(0x9888), 0x025b4000 },
	2186	{ _MMIO(0x9888), 0x045b4000 },
	2187	{ _MMIO(0x9888), 0x0c1fa000 },
	2188	{ _MMIO(0x9888), 0x0e1f00aa },
	2189	{ _MMIO(0x9888), 0x04386080 },
	2190	{ _MMIO(0x9888), 0x0638404b },
	2191	{ _MMIO(0x9888), 0x02384000 },
	2192	{ _MMIO(0x9888), 0x08384000 },
	2193	{ _MMIO(0x9888), 0x0a380000 },
	2194	{ _MMIO(0x9888), 0x0c380000 },
	2195	{ _MMIO(0x9888), 0x00398000 },
	2196	{ _MMIO(0x9888), 0x0239a000 },
	2197	{ _MMIO(0x9888), 0x0439a000 },
	2198	{ _MMIO(0x9888), 0x06392000 },
	2199	{ _MMIO(0x9888), 0x0cdc25c1 },
	2200	{ _MMIO(0x9888), 0x0adcc000 },
	2201	{ _MMIO(0x9888), 0x0ad825c1 },
	2202	{ _MMIO(0x9888), 0x18db4000 },
	2203	{ _MMIO(0x9888), 0x1adb0001 },
	2204	{ _MMIO(0x9888), 0x0e9f8000 },
	2205	{ _MMIO(0x9888), 0x109f02aa },
	2206	{ _MMIO(0x9888), 0x0eb825c1 },
	2207	{ _MMIO(0x9888), 0x18b80154 },
	2208	{ _MMIO(0x9888), 0x0ab9a000 },
	2209	{ _MMIO(0x9888), 0x0cb9a000 },
	2210	{ _MMIO(0x9888), 0x0eb9a000 },
	2211	{ _MMIO(0x9888), 0x0d88c000 },
	2212	{ _MMIO(0x9888), 0x0f88000f },
	2213	{ _MMIO(0x9888), 0x038a8000 },
	2214	{ _MMIO(0x9888), 0x058a8000 },
	2215	{ _MMIO(0x9888), 0x078a8000 },
	2216	{ _MMIO(0x9888), 0x098a8000 },
	2217	{ _MMIO(0x9888), 0x0b8a8000 },
	2218	{ _MMIO(0x9888), 0x0d8a8000 },
	2219	{ _MMIO(0x9888), 0x258baa05 },
	2220	{ _MMIO(0x9888), 0x278b002a },
	2221	{ _MMIO(0x9888), 0x238b2a80 },
	2222	{ _MMIO(0x9888), 0x198c5400 },
	2223	{ _MMIO(0x9888), 0x1b8c0015 },
	2224	{ _MMIO(0x9888), 0x098dc000 },
	2225	{ _MMIO(0x9888), 0x0b8da000 },
	2226	{ _MMIO(0x9888), 0x0d8da000 },
	2227	{ _MMIO(0x9888), 0x0f8da000 },
	2228	{ _MMIO(0x9888), 0x098e05c0 },
	2229	{ _MMIO(0x9888), 0x058e0000 },
	2230	{ _MMIO(0x9888), 0x198f0020 },
	2231	{ _MMIO(0x9888), 0x2185aa0a },
	2232	{ _MMIO(0x9888), 0x2385002a },
	2233	{ _MMIO(0x9888), 0x1f85aa00 },
	2234	{ _MMIO(0x9888), 0x19835000 },
	2235	{ _MMIO(0x9888), 0x1b830155 },
	2236	{ _MMIO(0x9888), 0x03834000 },
	2237	{ _MMIO(0x9888), 0x05834000 },
	2238	{ _MMIO(0x9888), 0x07834000 },
	2239	{ _MMIO(0x9888), 0x09834000 },
	2240	{ _MMIO(0x9888), 0x0b834000 },
	2241	{ _MMIO(0x9888), 0x0d834000 },
	2242	{ _MMIO(0x9888), 0x09848000 },
	2243	{ _MMIO(0x9888), 0x0b84c000 },
	2244	{ _MMIO(0x9888), 0x0d84c000 },
	2245	{ _MMIO(0x9888), 0x0f84c000 },
	2246	{ _MMIO(0x9888), 0x01848000 },
	2247	{ _MMIO(0x9888), 0x0384c000 },
	2248	{ _MMIO(0x9888), 0x0584c000 },
	2249	{ _MMIO(0x9888), 0x07844000 },
	2250	{ _MMIO(0x9888), 0x19808000 },
	2251	{ _MMIO(0x9888), 0x1b80c000 },
	2252	{ _MMIO(0x9888), 0x1d80c000 },
	2253	{ _MMIO(0x9888), 0x1f80c000 },
	2254	{ _MMIO(0x9888), 0x11808000 },
	2255	{ _MMIO(0x9888), 0x1380c000 },
	2256	{ _MMIO(0x9888), 0x1580c000 },
	2257	{ _MMIO(0x9888), 0x17804000 },
	2258	{ _MMIO(0x9888), 0x51800040 },
	2259	{ _MMIO(0x9888), 0x43800400 },
	2260	{ _MMIO(0x9888), 0x45800800 },
	2261	{ _MMIO(0x9888), 0x53800000 },
	2262	{ _MMIO(0x9888), 0x47800c62 },
	2263	{ _MMIO(0x9888), 0x21800000 },
	2264	{ _MMIO(0x9888), 0x31800000 },
	2265	{ _MMIO(0x9888), 0x4d800000 },
	2266	{ _MMIO(0x9888), 0x3f801042 },
	2267	{ _MMIO(0x9888), 0x4f800000 },
	2268	{ _MMIO(0x9888), 0x418014a4 },
	2269	};
	2270
	2271	static int
	2272	get_hdc_and_sf_mux_config(struct drm_i915_private *dev_priv,
	2273	const struct i915_oa_reg **regs,
	2274	int *lens)
	2275	{
	2276	int n = 0;
	2277
	2278	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2279	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2280
	2281	regs[n] = mux_config_hdc_and_sf;
	2282	lens[n] = ARRAY_SIZE(mux_config_hdc_and_sf);
	2283	n++;
	2284
	2285	return n;
	2286	}
	2287
	2288	static const struct i915_oa_reg b_counter_config_l3_1[] = {
	2289	{ _MMIO(0x2740), 0x00000000 },
	2290	{ _MMIO(0x2744), 0x00800000 },
	2291	{ _MMIO(0x2710), 0x00000000 },
	2292	{ _MMIO(0x2714), 0xf0800000 },
	2293	{ _MMIO(0x2720), 0x00000000 },
	2294	{ _MMIO(0x2724), 0xf0800000 },
	2295	{ _MMIO(0x2770), 0x00100070 },
	2296	{ _MMIO(0x2774), 0x0000fff1 },
	2297	{ _MMIO(0x2778), 0x00014002 },
	2298	{ _MMIO(0x277c), 0x0000c3ff },
	2299	{ _MMIO(0x2780), 0x00010002 },
	2300	{ _MMIO(0x2784), 0x0000c7ff },
	2301	{ _MMIO(0x2788), 0x00004002 },
	2302	{ _MMIO(0x278c), 0x0000d3ff },
	2303	{ _MMIO(0x2790), 0x00100700 },
	2304	{ _MMIO(0x2794), 0x0000ff1f },
	2305	{ _MMIO(0x2798), 0x00001402 },
	2306	{ _MMIO(0x279c), 0x0000fc3f },
	2307	{ _MMIO(0x27a0), 0x00001002 },
	2308	{ _MMIO(0x27a4), 0x0000fc7f },
	2309	{ _MMIO(0x27a8), 0x00000402 },
	2310	{ _MMIO(0x27ac), 0x0000fd3f },
	2311	};
	2312
	2313	static const struct i915_oa_reg flex_eu_config_l3_1[] = {
	2314	{ _MMIO(0xe458), 0x00005004 },
	2315	{ _MMIO(0xe558), 0x00010003 },
	2316	{ _MMIO(0xe658), 0x00012011 },
	2317	{ _MMIO(0xe758), 0x00015014 },
	2318	{ _MMIO(0xe45c), 0x00051050 },
	2319	{ _MMIO(0xe55c), 0x00053052 },
	2320	{ _MMIO(0xe65c), 0x00055054 },
	2321	};
	2322
	2323	static const struct i915_oa_reg mux_config_l3_1[] = {
	2324	{ _MMIO(0x9888), 0x10bf03da },
	2325	{ _MMIO(0x9888), 0x14bf0001 },
	2326	{ _MMIO(0x9888), 0x12980340 },
	2327	{ _MMIO(0x9888), 0x12990340 },
	2328	{ _MMIO(0x9888), 0x0cbf1187 },
	2329	{ _MMIO(0x9888), 0x0ebf1205 },
	2330	{ _MMIO(0x9888), 0x00bf0500 },
	2331	{ _MMIO(0x9888), 0x02bf042b },
	2332	{ _MMIO(0x9888), 0x04bf002c },
	2333	{ _MMIO(0x9888), 0x0cdac000 },
	2334	{ _MMIO(0x9888), 0x0edac000 },
	2335	{ _MMIO(0x9888), 0x00da8000 },
	2336	{ _MMIO(0x9888), 0x02dac000 },
	2337	{ _MMIO(0x9888), 0x04da4000 },
	2338	{ _MMIO(0x9888), 0x04983400 },
	2339	{ _MMIO(0x9888), 0x10980000 },
	2340	{ _MMIO(0x9888), 0x06990034 },
	2341	{ _MMIO(0x9888), 0x10990000 },
	2342	{ _MMIO(0x9888), 0x0c9dc000 },
	2343	{ _MMIO(0x9888), 0x0e9dc000 },
	2344	{ _MMIO(0x9888), 0x009d8000 },
	2345	{ _MMIO(0x9888), 0x029dc000 },
	2346	{ _MMIO(0x9888), 0x049d4000 },
	2347	{ _MMIO(0x9888), 0x109f02a8 },
	2348	{ _MMIO(0x9888), 0x0c9fa000 },
	2349	{ _MMIO(0x9888), 0x0e9f00ba },
	2350	{ _MMIO(0x9888), 0x0cb88000 },
	2351	{ _MMIO(0x9888), 0x0cb95000 },
	2352	{ _MMIO(0x9888), 0x0eb95000 },
	2353	{ _MMIO(0x9888), 0x00b94000 },
	2354	{ _MMIO(0x9888), 0x02b95000 },
	2355	{ _MMIO(0x9888), 0x04b91000 },
	2356	{ _MMIO(0x9888), 0x06b92000 },
	2357	{ _MMIO(0x9888), 0x0cba4000 },
	2358	{ _MMIO(0x9888), 0x0f88000f },
	2359	{ _MMIO(0x9888), 0x03888000 },
	2360	{ _MMIO(0x9888), 0x05888000 },
	2361	{ _MMIO(0x9888), 0x07888000 },
	2362	{ _MMIO(0x9888), 0x09888000 },
	2363	{ _MMIO(0x9888), 0x0b888000 },
	2364	{ _MMIO(0x9888), 0x0d880400 },
	2365	{ _MMIO(0x9888), 0x258b800a },
	2366	{ _MMIO(0x9888), 0x278b002a },
	2367	{ _MMIO(0x9888), 0x238b5500 },
	2368	{ _MMIO(0x9888), 0x198c4000 },
	2369	{ _MMIO(0x9888), 0x1b8c0015 },
	2370	{ _MMIO(0x9888), 0x038c4000 },
	2371	{ _MMIO(0x9888), 0x058c4000 },
	2372	{ _MMIO(0x9888), 0x078c4000 },
	2373	{ _MMIO(0x9888), 0x098c4000 },
	2374	{ _MMIO(0x9888), 0x0b8c4000 },
	2375	{ _MMIO(0x9888), 0x0d8c4000 },
	2376	{ _MMIO(0x9888), 0x0d8da000 },
	2377	{ _MMIO(0x9888), 0x0f8da000 },
	2378	{ _MMIO(0x9888), 0x018d8000 },
	2379	{ _MMIO(0x9888), 0x038da000 },
	2380	{ _MMIO(0x9888), 0x058da000 },
	2381	{ _MMIO(0x9888), 0x078d2000 },
	2382	{ _MMIO(0x9888), 0x2185800a },
	2383	{ _MMIO(0x9888), 0x2385002a },
	2384	{ _MMIO(0x9888), 0x1f85aa00 },
	2385	{ _MMIO(0x9888), 0x1b830154 },
	2386	{ _MMIO(0x9888), 0x03834000 },
	2387	{ _MMIO(0x9888), 0x05834000 },
	2388	{ _MMIO(0x9888), 0x07834000 },
	2389	{ _MMIO(0x9888), 0x09834000 },
	2390	{ _MMIO(0x9888), 0x0b834000 },
	2391	{ _MMIO(0x9888), 0x0d834000 },
	2392	{ _MMIO(0x9888), 0x0d84c000 },
	2393	{ _MMIO(0x9888), 0x0f84c000 },
	2394	{ _MMIO(0x9888), 0x01848000 },
	2395	{ _MMIO(0x9888), 0x0384c000 },
	2396	{ _MMIO(0x9888), 0x0584c000 },
	2397	{ _MMIO(0x9888), 0x07844000 },
	2398	{ _MMIO(0x9888), 0x1d80c000 },
	2399	{ _MMIO(0x9888), 0x1f80c000 },
	2400	{ _MMIO(0x9888), 0x11808000 },
	2401	{ _MMIO(0x9888), 0x1380c000 },
	2402	{ _MMIO(0x9888), 0x1580c000 },
	2403	{ _MMIO(0x9888), 0x17804000 },
	2404	{ _MMIO(0x9888), 0x53800000 },
	2405	{ _MMIO(0x9888), 0x45800000 },
	2406	{ _MMIO(0x9888), 0x47800000 },
	2407	{ _MMIO(0x9888), 0x21800000 },
	2408	{ _MMIO(0x9888), 0x31800000 },
	2409	{ _MMIO(0x9888), 0x4d800000 },
	2410	{ _MMIO(0x9888), 0x3f800000 },
	2411	{ _MMIO(0x9888), 0x4f800000 },
	2412	{ _MMIO(0x9888), 0x41800060 },
	2413	};
	2414
	2415	static int
	2416	get_l3_1_mux_config(struct drm_i915_private *dev_priv,
	2417	const struct i915_oa_reg **regs,
	2418	int *lens)
	2419	{
	2420	int n = 0;
	2421
	2422	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2423	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2424
	2425	regs[n] = mux_config_l3_1;
	2426	lens[n] = ARRAY_SIZE(mux_config_l3_1);
	2427	n++;
	2428
	2429	return n;
	2430	}
	2431
	2432	static const struct i915_oa_reg b_counter_config_l3_2[] = {
	2433	{ _MMIO(0x2740), 0x00000000 },
	2434	{ _MMIO(0x2744), 0x00800000 },
	2435	{ _MMIO(0x2710), 0x00000000 },
	2436	{ _MMIO(0x2714), 0xf0800000 },
	2437	{ _MMIO(0x2720), 0x00000000 },
	2438	{ _MMIO(0x2724), 0xf0800000 },
	2439	{ _MMIO(0x2770), 0x00100070 },
	2440	{ _MMIO(0x2774), 0x0000fff1 },
	2441	{ _MMIO(0x2778), 0x00014002 },
	2442	{ _MMIO(0x277c), 0x0000c3ff },
	2443	{ _MMIO(0x2780), 0x00010002 },
	2444	{ _MMIO(0x2784), 0x0000c7ff },
	2445	{ _MMIO(0x2788), 0x00004002 },
	2446	{ _MMIO(0x278c), 0x0000d3ff },
	2447	{ _MMIO(0x2790), 0x00100700 },
	2448	{ _MMIO(0x2794), 0x0000ff1f },
	2449	{ _MMIO(0x2798), 0x00001402 },
	2450	{ _MMIO(0x279c), 0x0000fc3f },
	2451	{ _MMIO(0x27a0), 0x00001002 },
	2452	{ _MMIO(0x27a4), 0x0000fc7f },
	2453	{ _MMIO(0x27a8), 0x00000402 },
	2454	{ _MMIO(0x27ac), 0x0000fd3f },
	2455	};
	2456
	2457	static const struct i915_oa_reg flex_eu_config_l3_2[] = {
	2458	{ _MMIO(0xe458), 0x00005004 },
	2459	{ _MMIO(0xe558), 0x00010003 },
	2460	{ _MMIO(0xe658), 0x00012011 },
	2461	{ _MMIO(0xe758), 0x00015014 },
	2462	{ _MMIO(0xe45c), 0x00051050 },
	2463	{ _MMIO(0xe55c), 0x00053052 },
	2464	{ _MMIO(0xe65c), 0x00055054 },
	2465	};
	2466
	2467	static const struct i915_oa_reg mux_config_l3_2[] = {
	2468	{ _MMIO(0x9888), 0x103f03da },
	2469	{ _MMIO(0x9888), 0x143f0001 },
	2470	{ _MMIO(0x9888), 0x12180340 },
	2471	{ _MMIO(0x9888), 0x12190340 },
	2472	{ _MMIO(0x9888), 0x0c3f1187 },
	2473	{ _MMIO(0x9888), 0x0e3f1205 },
	2474	{ _MMIO(0x9888), 0x003f0500 },
	2475	{ _MMIO(0x9888), 0x023f042b },
	2476	{ _MMIO(0x9888), 0x043f002c },
	2477	{ _MMIO(0x9888), 0x0c5ac000 },
	2478	{ _MMIO(0x9888), 0x0e5ac000 },
	2479	{ _MMIO(0x9888), 0x005a8000 },
	2480	{ _MMIO(0x9888), 0x025ac000 },
	2481	{ _MMIO(0x9888), 0x045a4000 },
	2482	{ _MMIO(0x9888), 0x04183400 },
	2483	{ _MMIO(0x9888), 0x10180000 },
	2484	{ _MMIO(0x9888), 0x06190034 },
	2485	{ _MMIO(0x9888), 0x10190000 },
	2486	{ _MMIO(0x9888), 0x0c1dc000 },
	2487	{ _MMIO(0x9888), 0x0e1dc000 },
	2488	{ _MMIO(0x9888), 0x001d8000 },
	2489	{ _MMIO(0x9888), 0x021dc000 },
	2490	{ _MMIO(0x9888), 0x041d4000 },
	2491	{ _MMIO(0x9888), 0x101f02a8 },
	2492	{ _MMIO(0x9888), 0x0c1fa000 },
	2493	{ _MMIO(0x9888), 0x0e1f00ba },
	2494	{ _MMIO(0x9888), 0x0c388000 },
	2495	{ _MMIO(0x9888), 0x0c395000 },
	2496	{ _MMIO(0x9888), 0x0e395000 },
	2497	{ _MMIO(0x9888), 0x00394000 },
	2498	{ _MMIO(0x9888), 0x02395000 },
	2499	{ _MMIO(0x9888), 0x04391000 },
	2500	{ _MMIO(0x9888), 0x06392000 },
	2501	{ _MMIO(0x9888), 0x0c3a4000 },
	2502	{ _MMIO(0x9888), 0x1b8aa800 },
	2503	{ _MMIO(0x9888), 0x1d8a0002 },
	2504	{ _MMIO(0x9888), 0x038a8000 },
	2505	{ _MMIO(0x9888), 0x058a8000 },
	2506	{ _MMIO(0x9888), 0x078a8000 },
	2507	{ _MMIO(0x9888), 0x098a8000 },
	2508	{ _MMIO(0x9888), 0x0b8a8000 },
	2509	{ _MMIO(0x9888), 0x0d8a8000 },
	2510	{ _MMIO(0x9888), 0x258b4005 },
	2511	{ _MMIO(0x9888), 0x278b0015 },
	2512	{ _MMIO(0x9888), 0x238b2a80 },
	2513	{ _MMIO(0x9888), 0x2185800a },
	2514	{ _MMIO(0x9888), 0x2385002a },
	2515	{ _MMIO(0x9888), 0x1f85aa00 },
	2516	{ _MMIO(0x9888), 0x1b830154 },
	2517	{ _MMIO(0x9888), 0x03834000 },
	2518	{ _MMIO(0x9888), 0x05834000 },
	2519	{ _MMIO(0x9888), 0x07834000 },
	2520	{ _MMIO(0x9888), 0x09834000 },
	2521	{ _MMIO(0x9888), 0x0b834000 },
	2522	{ _MMIO(0x9888), 0x0d834000 },
	2523	{ _MMIO(0x9888), 0x0d84c000 },
	2524	{ _MMIO(0x9888), 0x0f84c000 },
	2525	{ _MMIO(0x9888), 0x01848000 },
	2526	{ _MMIO(0x9888), 0x0384c000 },
	2527	{ _MMIO(0x9888), 0x0584c000 },
	2528	{ _MMIO(0x9888), 0x07844000 },
	2529	{ _MMIO(0x9888), 0x1d80c000 },
	2530	{ _MMIO(0x9888), 0x1f80c000 },
	2531	{ _MMIO(0x9888), 0x11808000 },
	2532	{ _MMIO(0x9888), 0x1380c000 },
	2533	{ _MMIO(0x9888), 0x1580c000 },
	2534	{ _MMIO(0x9888), 0x17804000 },
	2535	{ _MMIO(0x9888), 0x53800000 },
	2536	{ _MMIO(0x9888), 0x45800000 },
	2537	{ _MMIO(0x9888), 0x47800000 },
	2538	{ _MMIO(0x9888), 0x21800000 },
	2539	{ _MMIO(0x9888), 0x31800000 },
	2540	{ _MMIO(0x9888), 0x4d800000 },
	2541	{ _MMIO(0x9888), 0x3f800000 },
	2542	{ _MMIO(0x9888), 0x4f800000 },
	2543	{ _MMIO(0x9888), 0x41800060 },
	2544	};
	2545
	2546	static int
	2547	get_l3_2_mux_config(struct drm_i915_private *dev_priv,
	2548	const struct i915_oa_reg **regs,
	2549	int *lens)
	2550	{
	2551	int n = 0;
	2552
	2553	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2554	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2555
	2556	regs[n] = mux_config_l3_2;
	2557	lens[n] = ARRAY_SIZE(mux_config_l3_2);
	2558	n++;
	2559
	2560	return n;
	2561	}
	2562
	2563	static const struct i915_oa_reg b_counter_config_l3_3[] = {
	2564	{ _MMIO(0x2740), 0x00000000 },
	2565	{ _MMIO(0x2744), 0x00800000 },
	2566	{ _MMIO(0x2710), 0x00000000 },
	2567	{ _MMIO(0x2714), 0xf0800000 },
	2568	{ _MMIO(0x2720), 0x00000000 },
	2569	{ _MMIO(0x2724), 0xf0800000 },
	2570	{ _MMIO(0x2770), 0x00100070 },
	2571	{ _MMIO(0x2774), 0x0000fff1 },
	2572	{ _MMIO(0x2778), 0x00014002 },
	2573	{ _MMIO(0x277c), 0x0000c3ff },
	2574	{ _MMIO(0x2780), 0x00010002 },
	2575	{ _MMIO(0x2784), 0x0000c7ff },
	2576	{ _MMIO(0x2788), 0x00004002 },
	2577	{ _MMIO(0x278c), 0x0000d3ff },
	2578	{ _MMIO(0x2790), 0x00100700 },
	2579	{ _MMIO(0x2794), 0x0000ff1f },
	2580	{ _MMIO(0x2798), 0x00001402 },
	2581	{ _MMIO(0x279c), 0x0000fc3f },
	2582	{ _MMIO(0x27a0), 0x00001002 },
	2583	{ _MMIO(0x27a4), 0x0000fc7f },
	2584	{ _MMIO(0x27a8), 0x00000402 },
	2585	{ _MMIO(0x27ac), 0x0000fd3f },
	2586	};
	2587
	2588	static const struct i915_oa_reg flex_eu_config_l3_3[] = {
	2589	{ _MMIO(0xe458), 0x00005004 },
	2590	{ _MMIO(0xe558), 0x00010003 },
	2591	{ _MMIO(0xe658), 0x00012011 },
	2592	{ _MMIO(0xe758), 0x00015014 },
	2593	{ _MMIO(0xe45c), 0x00051050 },
	2594	{ _MMIO(0xe55c), 0x00053052 },
	2595	{ _MMIO(0xe65c), 0x00055054 },
	2596	};
	2597
	2598	static const struct i915_oa_reg mux_config_l3_3[] = {
	2599	{ _MMIO(0x9888), 0x121b0340 },
	2600	{ _MMIO(0x9888), 0x103f0274 },
	2601	{ _MMIO(0x9888), 0x123f0000 },
	2602	{ _MMIO(0x9888), 0x129b0340 },
	2603	{ _MMIO(0x9888), 0x10bf0274 },
	2604	{ _MMIO(0x9888), 0x12bf0000 },
	2605	{ _MMIO(0x9888), 0x041b3400 },
	2606	{ _MMIO(0x9888), 0x101b0000 },
	2607	{ _MMIO(0x9888), 0x045c8000 },
	2608	{ _MMIO(0x9888), 0x0a3d4000 },
	2609	{ _MMIO(0x9888), 0x003f0080 },
	2610	{ _MMIO(0x9888), 0x023f0793 },
	2611	{ _MMIO(0x9888), 0x043f0014 },
	2612	{ _MMIO(0x9888), 0x04588000 },
	2613	{ _MMIO(0x9888), 0x005a8000 },
	2614	{ _MMIO(0x9888), 0x025ac000 },
	2615	{ _MMIO(0x9888), 0x045a4000 },
	2616	{ _MMIO(0x9888), 0x0a5b4000 },
	2617	{ _MMIO(0x9888), 0x001d8000 },
	2618	{ _MMIO(0x9888), 0x021dc000 },
	2619	{ _MMIO(0x9888), 0x041d4000 },
	2620	{ _MMIO(0x9888), 0x0c1fa000 },
	2621	{ _MMIO(0x9888), 0x0e1f002a },
	2622	{ _MMIO(0x9888), 0x0a384000 },
	2623	{ _MMIO(0x9888), 0x00394000 },
	2624	{ _MMIO(0x9888), 0x02395000 },
	2625	{ _MMIO(0x9888), 0x04399000 },
	2626	{ _MMIO(0x9888), 0x069b0034 },
	2627	{ _MMIO(0x9888), 0x109b0000 },
	2628	{ _MMIO(0x9888), 0x06dc4000 },
	2629	{ _MMIO(0x9888), 0x0cbd4000 },
	2630	{ _MMIO(0x9888), 0x0cbf0981 },
	2631	{ _MMIO(0x9888), 0x0ebf0a0f },
	2632	{ _MMIO(0x9888), 0x06d84000 },
	2633	{ _MMIO(0x9888), 0x0cdac000 },
	2634	{ _MMIO(0x9888), 0x0edac000 },
	2635	{ _MMIO(0x9888), 0x0cdb4000 },
	2636	{ _MMIO(0x9888), 0x0c9dc000 },
	2637	{ _MMIO(0x9888), 0x0e9dc000 },
	2638	{ _MMIO(0x9888), 0x109f02a8 },
	2639	{ _MMIO(0x9888), 0x0e9f0080 },
	2640	{ _MMIO(0x9888), 0x0cb84000 },
	2641	{ _MMIO(0x9888), 0x0cb95000 },
	2642	{ _MMIO(0x9888), 0x0eb95000 },
	2643	{ _MMIO(0x9888), 0x06b92000 },
	2644	{ _MMIO(0x9888), 0x0f88000f },
	2645	{ _MMIO(0x9888), 0x0d880400 },
	2646	{ _MMIO(0x9888), 0x038a8000 },
	2647	{ _MMIO(0x9888), 0x058a8000 },
	2648	{ _MMIO(0x9888), 0x078a8000 },
	2649	{ _MMIO(0x9888), 0x098a8000 },
	2650	{ _MMIO(0x9888), 0x0b8a8000 },
	2651	{ _MMIO(0x9888), 0x258b8009 },
	2652	{ _MMIO(0x9888), 0x278b002a },
	2653	{ _MMIO(0x9888), 0x238b2a80 },
	2654	{ _MMIO(0x9888), 0x198c4000 },
	2655	{ _MMIO(0x9888), 0x1b8c0015 },
	2656	{ _MMIO(0x9888), 0x0d8c4000 },
	2657	{ _MMIO(0x9888), 0x0d8da000 },
	2658	{ _MMIO(0x9888), 0x0f8da000 },
	2659	{ _MMIO(0x9888), 0x078d2000 },
	2660	{ _MMIO(0x9888), 0x2185800a },
	2661	{ _MMIO(0x9888), 0x2385002a },
	2662	{ _MMIO(0x9888), 0x1f85aa00 },
	2663	{ _MMIO(0x9888), 0x1b830154 },
	2664	{ _MMIO(0x9888), 0x03834000 },
	2665	{ _MMIO(0x9888), 0x05834000 },
	2666	{ _MMIO(0x9888), 0x07834000 },
	2667	{ _MMIO(0x9888), 0x09834000 },
	2668	{ _MMIO(0x9888), 0x0b834000 },
	2669	{ _MMIO(0x9888), 0x0d834000 },
	2670	{ _MMIO(0x9888), 0x0d84c000 },
	2671	{ _MMIO(0x9888), 0x0f84c000 },
	2672	{ _MMIO(0x9888), 0x01848000 },
	2673	{ _MMIO(0x9888), 0x0384c000 },
	2674	{ _MMIO(0x9888), 0x0584c000 },
	2675	{ _MMIO(0x9888), 0x07844000 },
	2676	{ _MMIO(0x9888), 0x1d80c000 },
	2677	{ _MMIO(0x9888), 0x1f80c000 },
	2678	{ _MMIO(0x9888), 0x11808000 },
	2679	{ _MMIO(0x9888), 0x1380c000 },
	2680	{ _MMIO(0x9888), 0x1580c000 },
	2681	{ _MMIO(0x9888), 0x17804000 },
	2682	{ _MMIO(0x9888), 0x53800000 },
	2683	{ _MMIO(0x9888), 0x45800c00 },
	2684	{ _MMIO(0x9888), 0x47800c63 },
	2685	{ _MMIO(0x9888), 0x21800000 },
	2686	{ _MMIO(0x9888), 0x31800000 },
	2687	{ _MMIO(0x9888), 0x4d800000 },
	2688	{ _MMIO(0x9888), 0x3f8014a5 },
	2689	{ _MMIO(0x9888), 0x4f800000 },
	2690	{ _MMIO(0x9888), 0x41800045 },
	2691	};
	2692
	2693	static int
	2694	get_l3_3_mux_config(struct drm_i915_private *dev_priv,
	2695	const struct i915_oa_reg **regs,
	2696	int *lens)
	2697	{
	2698	int n = 0;
	2699
	2700	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2701	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2702
	2703	regs[n] = mux_config_l3_3;
	2704	lens[n] = ARRAY_SIZE(mux_config_l3_3);
	2705	n++;
	2706
	2707	return n;
	2708	}
	2709
	2710	static const struct i915_oa_reg b_counter_config_l3_4[] = {
	2711	{ _MMIO(0x2740), 0x00000000 },
	2712	{ _MMIO(0x2744), 0x00800000 },
	2713	{ _MMIO(0x2710), 0x00000000 },
	2714	{ _MMIO(0x2714), 0xf0800000 },
	2715	{ _MMIO(0x2720), 0x00000000 },
	2716	{ _MMIO(0x2724), 0xf0800000 },
	2717	{ _MMIO(0x2770), 0x00100070 },
	2718	{ _MMIO(0x2774), 0x0000fff1 },
	2719	{ _MMIO(0x2778), 0x00014002 },
	2720	{ _MMIO(0x277c), 0x0000c3ff },
	2721	{ _MMIO(0x2780), 0x00010002 },
	2722	{ _MMIO(0x2784), 0x0000c7ff },
	2723	{ _MMIO(0x2788), 0x00004002 },
	2724	{ _MMIO(0x278c), 0x0000d3ff },
	2725	{ _MMIO(0x2790), 0x00100700 },
	2726	{ _MMIO(0x2794), 0x0000ff1f },
	2727	{ _MMIO(0x2798), 0x00001402 },
	2728	{ _MMIO(0x279c), 0x0000fc3f },
	2729	{ _MMIO(0x27a0), 0x00001002 },
	2730	{ _MMIO(0x27a4), 0x0000fc7f },
	2731	{ _MMIO(0x27a8), 0x00000402 },
	2732	{ _MMIO(0x27ac), 0x0000fd3f },
	2733	};
	2734
	2735	static const struct i915_oa_reg flex_eu_config_l3_4[] = {
	2736	{ _MMIO(0xe458), 0x00005004 },
	2737	{ _MMIO(0xe558), 0x00010003 },
	2738	{ _MMIO(0xe658), 0x00012011 },
	2739	{ _MMIO(0xe758), 0x00015014 },
	2740	{ _MMIO(0xe45c), 0x00051050 },
	2741	{ _MMIO(0xe55c), 0x00053052 },
	2742	{ _MMIO(0xe65c), 0x00055054 },
	2743	};
	2744
	2745	static const struct i915_oa_reg mux_config_l3_4[] = {
	2746	{ _MMIO(0x9888), 0x121a0340 },
	2747	{ _MMIO(0x9888), 0x103f0017 },
	2748	{ _MMIO(0x9888), 0x123f0020 },
	2749	{ _MMIO(0x9888), 0x129a0340 },
	2750	{ _MMIO(0x9888), 0x10bf0017 },
	2751	{ _MMIO(0x9888), 0x12bf0020 },
	2752	{ _MMIO(0x9888), 0x041a3400 },
	2753	{ _MMIO(0x9888), 0x101a0000 },
	2754	{ _MMIO(0x9888), 0x043b8000 },
	2755	{ _MMIO(0x9888), 0x0a3e0010 },
	2756	{ _MMIO(0x9888), 0x003f0200 },
	2757	{ _MMIO(0x9888), 0x023f0113 },
	2758	{ _MMIO(0x9888), 0x043f0014 },
	2759	{ _MMIO(0x9888), 0x02592000 },
	2760	{ _MMIO(0x9888), 0x005a8000 },
	2761	{ _MMIO(0x9888), 0x025ac000 },
	2762	{ _MMIO(0x9888), 0x045a4000 },
	2763	{ _MMIO(0x9888), 0x0a1c8000 },
	2764	{ _MMIO(0x9888), 0x001d8000 },
	2765	{ _MMIO(0x9888), 0x021dc000 },
	2766	{ _MMIO(0x9888), 0x041d4000 },
	2767	{ _MMIO(0x9888), 0x0a1e8000 },
	2768	{ _MMIO(0x9888), 0x0c1fa000 },
	2769	{ _MMIO(0x9888), 0x0e1f001a },
	2770	{ _MMIO(0x9888), 0x00394000 },
	2771	{ _MMIO(0x9888), 0x02395000 },
	2772	{ _MMIO(0x9888), 0x04391000 },
	2773	{ _MMIO(0x9888), 0x069a0034 },
	2774	{ _MMIO(0x9888), 0x109a0000 },
	2775	{ _MMIO(0x9888), 0x06bb4000 },
	2776	{ _MMIO(0x9888), 0x0abe0040 },
	2777	{ _MMIO(0x9888), 0x0cbf0984 },
	2778	{ _MMIO(0x9888), 0x0ebf0a02 },
	2779	{ _MMIO(0x9888), 0x02d94000 },
	2780	{ _MMIO(0x9888), 0x0cdac000 },
	2781	{ _MMIO(0x9888), 0x0edac000 },
	2782	{ _MMIO(0x9888), 0x0c9c0400 },
	2783	{ _MMIO(0x9888), 0x0c9dc000 },
	2784	{ _MMIO(0x9888), 0x0e9dc000 },
	2785	{ _MMIO(0x9888), 0x0c9e0400 },
	2786	{ _MMIO(0x9888), 0x109f02a8 },
	2787	{ _MMIO(0x9888), 0x0e9f0040 },
	2788	{ _MMIO(0x9888), 0x0cb95000 },
	2789	{ _MMIO(0x9888), 0x0eb95000 },
	2790	{ _MMIO(0x9888), 0x0f88000f },
	2791	{ _MMIO(0x9888), 0x0d880400 },
	2792	{ _MMIO(0x9888), 0x038a8000 },
	2793	{ _MMIO(0x9888), 0x058a8000 },
	2794	{ _MMIO(0x9888), 0x078a8000 },
	2795	{ _MMIO(0x9888), 0x098a8000 },
	2796	{ _MMIO(0x9888), 0x0b8a8000 },
	2797	{ _MMIO(0x9888), 0x258b8009 },
	2798	{ _MMIO(0x9888), 0x278b002a },
	2799	{ _MMIO(0x9888), 0x238b2a80 },
	2800	{ _MMIO(0x9888), 0x198c4000 },
	2801	{ _MMIO(0x9888), 0x1b8c0015 },
	2802	{ _MMIO(0x9888), 0x0d8c4000 },
	2803	{ _MMIO(0x9888), 0x0d8da000 },
	2804	{ _MMIO(0x9888), 0x0f8da000 },
	2805	{ _MMIO(0x9888), 0x078d2000 },
	2806	{ _MMIO(0x9888), 0x2185800a },
	2807	{ _MMIO(0x9888), 0x2385002a },
	2808	{ _MMIO(0x9888), 0x1f85aa00 },
	2809	{ _MMIO(0x9888), 0x1b830154 },
	2810	{ _MMIO(0x9888), 0x03834000 },
	2811	{ _MMIO(0x9888), 0x05834000 },
	2812	{ _MMIO(0x9888), 0x07834000 },
	2813	{ _MMIO(0x9888), 0x09834000 },
	2814	{ _MMIO(0x9888), 0x0b834000 },
	2815	{ _MMIO(0x9888), 0x0d834000 },
	2816	{ _MMIO(0x9888), 0x0d84c000 },
	2817	{ _MMIO(0x9888), 0x0f84c000 },
	2818	{ _MMIO(0x9888), 0x01848000 },
	2819	{ _MMIO(0x9888), 0x0384c000 },
	2820	{ _MMIO(0x9888), 0x0584c000 },
	2821	{ _MMIO(0x9888), 0x07844000 },
	2822	{ _MMIO(0x9888), 0x1d80c000 },
	2823	{ _MMIO(0x9888), 0x1f80c000 },
	2824	{ _MMIO(0x9888), 0x11808000 },
	2825	{ _MMIO(0x9888), 0x1380c000 },
	2826	{ _MMIO(0x9888), 0x1580c000 },
	2827	{ _MMIO(0x9888), 0x17804000 },
	2828	{ _MMIO(0x9888), 0x53800000 },
	2829	{ _MMIO(0x9888), 0x45800800 },
	2830	{ _MMIO(0x9888), 0x47800842 },
	2831	{ _MMIO(0x9888), 0x21800000 },
	2832	{ _MMIO(0x9888), 0x31800000 },
	2833	{ _MMIO(0x9888), 0x4d800000 },
	2834	{ _MMIO(0x9888), 0x3f801084 },
	2835	{ _MMIO(0x9888), 0x4f800000 },
	2836	{ _MMIO(0x9888), 0x41800044 },
	2837	};
	2838
	2839	static int
	2840	get_l3_4_mux_config(struct drm_i915_private *dev_priv,
	2841	const struct i915_oa_reg **regs,
	2842	int *lens)
	2843	{
	2844	int n = 0;
	2845
	2846	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2847	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2848
	2849	regs[n] = mux_config_l3_4;
	2850	lens[n] = ARRAY_SIZE(mux_config_l3_4);
	2851	n++;
	2852
	2853	return n;
	2854	}
	2855
	2856	static const struct i915_oa_reg b_counter_config_rasterizer_and_pixel_backend[] = {
	2857	{ _MMIO(0x2740), 0x00000000 },
	2858	{ _MMIO(0x2744), 0x00800000 },
	2859	{ _MMIO(0x2710), 0x00000000 },
	2860	{ _MMIO(0x2714), 0xf0800000 },
	2861	{ _MMIO(0x2720), 0x00000000 },
	2862	{ _MMIO(0x2724), 0x30800000 },
	2863	{ _MMIO(0x2770), 0x00006000 },
	2864	{ _MMIO(0x2774), 0x0000f3ff },
	2865	{ _MMIO(0x2778), 0x00001800 },
	2866	{ _MMIO(0x277c), 0x0000fcff },
	2867	{ _MMIO(0x2780), 0x00000600 },
	2868	{ _MMIO(0x2784), 0x0000ff3f },
	2869	{ _MMIO(0x2788), 0x00000180 },
	2870	{ _MMIO(0x278c), 0x0000ffcf },
	2871	{ _MMIO(0x2790), 0x00000060 },
	2872	{ _MMIO(0x2794), 0x0000fff3 },
	2873	{ _MMIO(0x2798), 0x00000018 },
	2874	{ _MMIO(0x279c), 0x0000fffc },
	2875	};
	2876
	2877	static const struct i915_oa_reg flex_eu_config_rasterizer_and_pixel_backend[] = {
	2878	{ _MMIO(0xe458), 0x00005004 },
	2879	{ _MMIO(0xe558), 0x00010003 },
	2880	{ _MMIO(0xe658), 0x00012011 },
	2881	{ _MMIO(0xe758), 0x00015014 },
	2882	{ _MMIO(0xe45c), 0x00051050 },
	2883	{ _MMIO(0xe55c), 0x00053052 },
	2884	{ _MMIO(0xe65c), 0x00055054 },
	2885	};
	2886
	2887	static const struct i915_oa_reg mux_config_rasterizer_and_pixel_backend[] = {
	2888	{ _MMIO(0x9888), 0x143b000e },
	2889	{ _MMIO(0x9888), 0x043c55c0 },
	2890	{ _MMIO(0x9888), 0x0a1e0280 },
	2891	{ _MMIO(0x9888), 0x0c1e0408 },
	2892	{ _MMIO(0x9888), 0x10390000 },
	2893	{ _MMIO(0x9888), 0x12397a1f },
	2894	{ _MMIO(0x9888), 0x14bb000e },
	2895	{ _MMIO(0x9888), 0x04bc5000 },
	2896	{ _MMIO(0x9888), 0x0a9e0296 },
	2897	{ _MMIO(0x9888), 0x0c9e0008 },
	2898	{ _MMIO(0x9888), 0x10b90000 },
	2899	{ _MMIO(0x9888), 0x12b97a1f },
	2900	{ _MMIO(0x9888), 0x063b0042 },
	2901	{ _MMIO(0x9888), 0x103b0000 },
	2902	{ _MMIO(0x9888), 0x083c0000 },
	2903	{ _MMIO(0x9888), 0x0a3e0040 },
	2904	{ _MMIO(0x9888), 0x043f8000 },
	2905	{ _MMIO(0x9888), 0x02594000 },
	2906	{ _MMIO(0x9888), 0x045a8000 },
	2907	{ _MMIO(0x9888), 0x0c1c0400 },
	2908	{ _MMIO(0x9888), 0x041d8000 },
	2909	{ _MMIO(0x9888), 0x081e02c0 },
	2910	{ _MMIO(0x9888), 0x0e1e0000 },
	2911	{ _MMIO(0x9888), 0x0c1fa800 },
	2912	{ _MMIO(0x9888), 0x0e1f0260 },
	2913	{ _MMIO(0x9888), 0x101f0014 },
	2914	{ _MMIO(0x9888), 0x003905e0 },
	2915	{ _MMIO(0x9888), 0x06390bc0 },
	2916	{ _MMIO(0x9888), 0x02390018 },
	2917	{ _MMIO(0x9888), 0x04394000 },
	2918	{ _MMIO(0x9888), 0x04bb0042 },
	2919	{ _MMIO(0x9888), 0x10bb0000 },
	2920	{ _MMIO(0x9888), 0x02bc05c0 },
	2921	{ _MMIO(0x9888), 0x08bc0000 },
	2922	{ _MMIO(0x9888), 0x0abe0004 },
	2923	{ _MMIO(0x9888), 0x02bf8000 },
	2924	{ _MMIO(0x9888), 0x02d91000 },
	2925	{ _MMIO(0x9888), 0x02da8000 },
	2926	{ _MMIO(0x9888), 0x089c8000 },
	2927	{ _MMIO(0x9888), 0x029d8000 },
	2928	{ _MMIO(0x9888), 0x089e8000 },
	2929	{ _MMIO(0x9888), 0x0e9e0000 },
	2930	{ _MMIO(0x9888), 0x0e9fa806 },
	2931	{ _MMIO(0x9888), 0x109f0142 },
	2932	{ _MMIO(0x9888), 0x08b90617 },
	2933	{ _MMIO(0x9888), 0x0ab90be0 },
	2934	{ _MMIO(0x9888), 0x02b94000 },
	2935	{ _MMIO(0x9888), 0x0d88f000 },
	2936	{ _MMIO(0x9888), 0x0f88000c },
	2937	{ _MMIO(0x9888), 0x07888000 },
	2938	{ _MMIO(0x9888), 0x09888000 },
	2939	{ _MMIO(0x9888), 0x018a8000 },
	2940	{ _MMIO(0x9888), 0x0f8a8000 },
	2941	{ _MMIO(0x9888), 0x1b8a2800 },
	2942	{ _MMIO(0x9888), 0x038a8000 },
	2943	{ _MMIO(0x9888), 0x058a8000 },
	2944	{ _MMIO(0x9888), 0x0b8a8000 },
	2945	{ _MMIO(0x9888), 0x0d8a8000 },
	2946	{ _MMIO(0x9888), 0x238b52a0 },
	2947	{ _MMIO(0x9888), 0x258b6a95 },
	2948	{ _MMIO(0x9888), 0x278b0029 },
	2949	{ _MMIO(0x9888), 0x178c2000 },
	2950	{ _MMIO(0x9888), 0x198c1500 },
	2951	{ _MMIO(0x9888), 0x1b8c0014 },
	2952	{ _MMIO(0x9888), 0x078c4000 },
	2953	{ _MMIO(0x9888), 0x098c4000 },
	2954	{ _MMIO(0x9888), 0x098da000 },
	2955	{ _MMIO(0x9888), 0x0b8da000 },
	2956	{ _MMIO(0x9888), 0x0f8da000 },
	2957	{ _MMIO(0x9888), 0x038d8000 },
	2958	{ _MMIO(0x9888), 0x058d2000 },
	2959	{ _MMIO(0x9888), 0x1f85aa80 },
	2960	{ _MMIO(0x9888), 0x2185aaaa },
	2961	{ _MMIO(0x9888), 0x2385002a },
	2962	{ _MMIO(0x9888), 0x01834000 },
	2963	{ _MMIO(0x9888), 0x0f834000 },
	2964	{ _MMIO(0x9888), 0x19835400 },
	2965	{ _MMIO(0x9888), 0x1b830155 },
	2966	{ _MMIO(0x9888), 0x03834000 },
	2967	{ _MMIO(0x9888), 0x05834000 },
	2968	{ _MMIO(0x9888), 0x07834000 },
	2969	{ _MMIO(0x9888), 0x09834000 },
	2970	{ _MMIO(0x9888), 0x0b834000 },
	2971	{ _MMIO(0x9888), 0x0d834000 },
	2972	{ _MMIO(0x9888), 0x0184c000 },
	2973	{ _MMIO(0x9888), 0x0784c000 },
	2974	{ _MMIO(0x9888), 0x0984c000 },
	2975	{ _MMIO(0x9888), 0x0b84c000 },
	2976	{ _MMIO(0x9888), 0x0d84c000 },
	2977	{ _MMIO(0x9888), 0x0f84c000 },
	2978	{ _MMIO(0x9888), 0x0384c000 },
	2979	{ _MMIO(0x9888), 0x0584c000 },
	2980	{ _MMIO(0x9888), 0x1180c000 },
	2981	{ _MMIO(0x9888), 0x1780c000 },
	2982	{ _MMIO(0x9888), 0x1980c000 },
	2983	{ _MMIO(0x9888), 0x1b80c000 },
	2984	{ _MMIO(0x9888), 0x1d80c000 },
	2985	{ _MMIO(0x9888), 0x1f80c000 },
	2986	{ _MMIO(0x9888), 0x1380c000 },
	2987	{ _MMIO(0x9888), 0x1580c000 },
	2988	{ _MMIO(0x9888), 0x4d800444 },
	2989	{ _MMIO(0x9888), 0x3d800000 },
	2990	{ _MMIO(0x9888), 0x4f804000 },
	2991	{ _MMIO(0x9888), 0x43801080 },
	2992	{ _MMIO(0x9888), 0x51800000 },
	2993	{ _MMIO(0x9888), 0x45800084 },
	2994	{ _MMIO(0x9888), 0x53800044 },
	2995	{ _MMIO(0x9888), 0x47801080 },
	2996	{ _MMIO(0x9888), 0x21800000 },
	2997	{ _MMIO(0x9888), 0x31800000 },
	2998	{ _MMIO(0x9888), 0x3f800000 },
	2999	{ _MMIO(0x9888), 0x41800840 },
	3000	};
	3001
	3002	static int
	3003	get_rasterizer_and_pixel_backend_mux_config(struct drm_i915_private *dev_priv,
	3004	const struct i915_oa_reg **regs,
	3005	int *lens)
	3006	{
	3007	int n = 0;
	3008
	3009	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	3010	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	3011
	3012	regs[n] = mux_config_rasterizer_and_pixel_backend;
	3013	lens[n] = ARRAY_SIZE(mux_config_rasterizer_and_pixel_backend);
	3014	n++;
	3015
	3016	return n;
	3017	}
	3018
	3019	static const struct i915_oa_reg b_counter_config_sampler_1[] = {
	3020	{ _MMIO(0x2740), 0x00000000 },
	3021	{ _MMIO(0x2744), 0x00800000 },
	3022	{ _MMIO(0x2710), 0x00000000 },
	3023	{ _MMIO(0x2714), 0x70800000 },
	3024	{ _MMIO(0x2720), 0x00000000 },
	3025	{ _MMIO(0x2724), 0x00800000 },
	3026	{ _MMIO(0x2770), 0x0000c000 },
	3027	{ _MMIO(0x2774), 0x0000e7ff },
	3028	{ _MMIO(0x2778), 0x00003000 },
	3029	{ _MMIO(0x277c), 0x0000f9ff },
	3030	{ _MMIO(0x2780), 0x00000c00 },
	3031	{ _MMIO(0x2784), 0x0000fe7f },
	3032	};
	3033
	3034	static const struct i915_oa_reg flex_eu_config_sampler_1[] = {
	3035	{ _MMIO(0xe458), 0x00005004 },
	3036	{ _MMIO(0xe558), 0x00010003 },
	3037	{ _MMIO(0xe658), 0x00012011 },
	3038	{ _MMIO(0xe758), 0x00015014 },
	3039	{ _MMIO(0xe45c), 0x00051050 },
	3040	{ _MMIO(0xe55c), 0x00053052 },
	3041	{ _MMIO(0xe65c), 0x00055054 },
	3042	};
	3043
	3044	static const struct i915_oa_reg mux_config_sampler_1[] = {
	3045	{ _MMIO(0x9888), 0x18921400 },
	3046	{ _MMIO(0x9888), 0x149500ab },
	3047	{ _MMIO(0x9888), 0x18b21400 },
	3048	{ _MMIO(0x9888), 0x14b500ab },
	3049	{ _MMIO(0x9888), 0x18d21400 },
	3050	{ _MMIO(0x9888), 0x14d500ab },
	3051	{ _MMIO(0x9888), 0x0cdc8000 },
	3052	{ _MMIO(0x9888), 0x0edc4000 },
	3053	{ _MMIO(0x9888), 0x02dcc000 },
	3054	{ _MMIO(0x9888), 0x04dcc000 },
	3055	{ _MMIO(0x9888), 0x1abd00a0 },
	3056	{ _MMIO(0x9888), 0x0abd8000 },
	3057	{ _MMIO(0x9888), 0x0cd88000 },
	3058	{ _MMIO(0x9888), 0x0ed84000 },
	3059	{ _MMIO(0x9888), 0x04d88000 },
	3060	{ _MMIO(0x9888), 0x1adb0050 },
	3061	{ _MMIO(0x9888), 0x04db8000 },
	3062	{ _MMIO(0x9888), 0x06db8000 },
	3063	{ _MMIO(0x9888), 0x08db8000 },
	3064	{ _MMIO(0x9888), 0x0adb4000 },
	3065	{ _MMIO(0x9888), 0x109f02a0 },
	3066	{ _MMIO(0x9888), 0x0c9fa000 },
	3067	{ _MMIO(0x9888), 0x0e9f00aa },
	3068	{ _MMIO(0x9888), 0x18b82500 },
	3069	{ _MMIO(0x9888), 0x02b88000 },
	3070	{ _MMIO(0x9888), 0x04b84000 },
	3071	{ _MMIO(0x9888), 0x06b84000 },
	3072	{ _MMIO(0x9888), 0x08b84000 },
	3073	{ _MMIO(0x9888), 0x0ab84000 },
	3074	{ _MMIO(0x9888), 0x0cb88000 },
	3075	{ _MMIO(0x9888), 0x0cb98000 },
	3076	{ _MMIO(0x9888), 0x0eb9a000 },
	3077	{ _MMIO(0x9888), 0x00b98000 },
	3078	{ _MMIO(0x9888), 0x02b9a000 },
	3079	{ _MMIO(0x9888), 0x04b9a000 },
	3080	{ _MMIO(0x9888), 0x06b92000 },
	3081	{ _MMIO(0x9888), 0x1aba0200 },
	3082	{ _MMIO(0x9888), 0x02ba8000 },
	3083	{ _MMIO(0x9888), 0x0cba8000 },
	3084	{ _MMIO(0x9888), 0x04908000 },
	3085	{ _MMIO(0x9888), 0x04918000 },
	3086	{ _MMIO(0x9888), 0x04927300 },
	3087	{ _MMIO(0x9888), 0x10920000 },
	3088	{ _MMIO(0x9888), 0x1893000a },
	3089	{ _MMIO(0x9888), 0x0a934000 },
	3090	{ _MMIO(0x9888), 0x0a946000 },
	3091	{ _MMIO(0x9888), 0x0c959000 },
	3092	{ _MMIO(0x9888), 0x0e950098 },
	3093	{ _MMIO(0x9888), 0x10950000 },
	3094	{ _MMIO(0x9888), 0x04b04000 },
	3095	{ _MMIO(0x9888), 0x04b14000 },
	3096	{ _MMIO(0x9888), 0x04b20073 },
	3097	{ _MMIO(0x9888), 0x10b20000 },
	3098	{ _MMIO(0x9888), 0x04b38000 },
	3099	{ _MMIO(0x9888), 0x06b38000 },
	3100	{ _MMIO(0x9888), 0x08b34000 },
	3101	{ _MMIO(0x9888), 0x04b4c000 },
	3102	{ _MMIO(0x9888), 0x02b59890 },
	3103	{ _MMIO(0x9888), 0x10b50000 },
	3104	{ _MMIO(0x9888), 0x06d04000 },
	3105	{ _MMIO(0x9888), 0x06d14000 },
	3106	{ _MMIO(0x9888), 0x06d20073 },
	3107	{ _MMIO(0x9888), 0x10d20000 },
	3108	{ _MMIO(0x9888), 0x18d30020 },
	3109	{ _MMIO(0x9888), 0x02d38000 },
	3110	{ _MMIO(0x9888), 0x0cd34000 },
	3111	{ _MMIO(0x9888), 0x0ad48000 },
	3112	{ _MMIO(0x9888), 0x04d42000 },
	3113	{ _MMIO(0x9888), 0x0ed59000 },
	3114	{ _MMIO(0x9888), 0x00d59800 },
	3115	{ _MMIO(0x9888), 0x10d50000 },
	3116	{ _MMIO(0x9888), 0x0f88000e },
	3117	{ _MMIO(0x9888), 0x03888000 },
	3118	{ _MMIO(0x9888), 0x05888000 },
	3119	{ _MMIO(0x9888), 0x07888000 },
	3120	{ _MMIO(0x9888), 0x09888000 },
	3121	{ _MMIO(0x9888), 0x0b888000 },
	3122	{ _MMIO(0x9888), 0x0d880400 },
	3123	{ _MMIO(0x9888), 0x278b002a },
	3124	{ _MMIO(0x9888), 0x238b5500 },
	3125	{ _MMIO(0x9888), 0x258b000a },
	3126	{ _MMIO(0x9888), 0x1b8c0015 },
	3127	{ _MMIO(0x9888), 0x038c4000 },
	3128	{ _MMIO(0x9888), 0x058c4000 },
	3129	{ _MMIO(0x9888), 0x078c4000 },
	3130	{ _MMIO(0x9888), 0x098c4000 },
	3131	{ _MMIO(0x9888), 0x0b8c4000 },
	3132	{ _MMIO(0x9888), 0x0d8c4000 },
	3133	{ _MMIO(0x9888), 0x0d8d8000 },
	3134	{ _MMIO(0x9888), 0x0f8da000 },
	3135	{ _MMIO(0x9888), 0x018d8000 },
	3136	{ _MMIO(0x9888), 0x038da000 },
	3137	{ _MMIO(0x9888), 0x058da000 },
	3138	{ _MMIO(0x9888), 0x078d2000 },
	3139	{ _MMIO(0x9888), 0x2385002a },
	3140	{ _MMIO(0x9888), 0x1f85aa00 },
	3141	{ _MMIO(0x9888), 0x2185000a },
	3142	{ _MMIO(0x9888), 0x1b830150 },
	3143	{ _MMIO(0x9888), 0x03834000 },
	3144	{ _MMIO(0x9888), 0x05834000 },
	3145	{ _MMIO(0x9888), 0x07834000 },
	3146	{ _MMIO(0x9888), 0x09834000 },
	3147	{ _MMIO(0x9888), 0x0b834000 },
	3148	{ _MMIO(0x9888), 0x0d834000 },
	3149	{ _MMIO(0x9888), 0x0d848000 },
	3150	{ _MMIO(0x9888), 0x0f84c000 },
	3151	{ _MMIO(0x9888), 0x01848000 },
	3152	{ _MMIO(0x9888), 0x0384c000 },
	3153	{ _MMIO(0x9888), 0x0584c000 },
	3154	{ _MMIO(0x9888), 0x07844000 },
	3155	{ _MMIO(0x9888), 0x1d808000 },
	3156	{ _MMIO(0x9888), 0x1f80c000 },
	3157	{ _MMIO(0x9888), 0x11808000 },
	3158	{ _MMIO(0x9888), 0x1380c000 },
	3159	{ _MMIO(0x9888), 0x1580c000 },
	3160	{ _MMIO(0x9888), 0x17804000 },
	3161	{ _MMIO(0x9888), 0x53800000 },
	3162	{ _MMIO(0x9888), 0x47801021 },
	3163	{ _MMIO(0x9888), 0x21800000 },
	3164	{ _MMIO(0x9888), 0x31800000 },
	3165	{ _MMIO(0x9888), 0x4d800000 },
	3166	{ _MMIO(0x9888), 0x3f800c64 },
	3167	{ _MMIO(0x9888), 0x4f800000 },
	3168	{ _MMIO(0x9888), 0x41800c02 },
	3169	};
	3170
	3171	static int
	3172	get_sampler_1_mux_config(struct drm_i915_private *dev_priv,
	3173	const struct i915_oa_reg **regs,
	3174	int *lens)
	3175	{
	3176	int n = 0;
	3177
	3178	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	3179	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	3180
	3181	regs[n] = mux_config_sampler_1;
	3182	lens[n] = ARRAY_SIZE(mux_config_sampler_1);
	3183	n++;
	3184
	3185	return n;
	3186	}
	3187
	3188	static const struct i915_oa_reg b_counter_config_sampler_2[] = {
	3189	{ _MMIO(0x2740), 0x00000000 },
	3190	{ _MMIO(0x2744), 0x00800000 },
	3191	{ _MMIO(0x2710), 0x00000000 },
	3192	{ _MMIO(0x2714), 0x70800000 },
	3193	{ _MMIO(0x2720), 0x00000000 },
	3194	{ _MMIO(0x2724), 0x00800000 },
	3195	{ _MMIO(0x2770), 0x0000c000 },
	3196	{ _MMIO(0x2774), 0x0000e7ff },
	3197	{ _MMIO(0x2778), 0x00003000 },
	3198	{ _MMIO(0x277c), 0x0000f9ff },
	3199	{ _MMIO(0x2780), 0x00000c00 },
	3200	{ _MMIO(0x2784), 0x0000fe7f },
	3201	};
	3202
	3203	static const struct i915_oa_reg flex_eu_config_sampler_2[] = {
	3204	{ _MMIO(0xe458), 0x00005004 },
	3205	{ _MMIO(0xe558), 0x00010003 },
	3206	{ _MMIO(0xe658), 0x00012011 },
	3207	{ _MMIO(0xe758), 0x00015014 },
	3208	{ _MMIO(0xe45c), 0x00051050 },
	3209	{ _MMIO(0xe55c), 0x00053052 },
	3210	{ _MMIO(0xe65c), 0x00055054 },
	3211	};
	3212
	3213	static const struct i915_oa_reg mux_config_sampler_2[] = {
	3214	{ _MMIO(0x9888), 0x18121400 },
	3215	{ _MMIO(0x9888), 0x141500ab },
	3216	{ _MMIO(0x9888), 0x18321400 },
	3217	{ _MMIO(0x9888), 0x143500ab },
	3218	{ _MMIO(0x9888), 0x18521400 },
	3219	{ _MMIO(0x9888), 0x145500ab },
	3220	{ _MMIO(0x9888), 0x0c5c8000 },
	3221	{ _MMIO(0x9888), 0x0e5c4000 },
	3222	{ _MMIO(0x9888), 0x025cc000 },
	3223	{ _MMIO(0x9888), 0x045cc000 },
	3224	{ _MMIO(0x9888), 0x1a3d00a0 },
	3225	{ _MMIO(0x9888), 0x0a3d8000 },
	3226	{ _MMIO(0x9888), 0x0c588000 },
	3227	{ _MMIO(0x9888), 0x0e584000 },
	3228	{ _MMIO(0x9888), 0x04588000 },
	3229	{ _MMIO(0x9888), 0x1a5b0050 },
	3230	{ _MMIO(0x9888), 0x045b8000 },
	3231	{ _MMIO(0x9888), 0x065b8000 },
	3232	{ _MMIO(0x9888), 0x085b8000 },
	3233	{ _MMIO(0x9888), 0x0a5b4000 },
	3234	{ _MMIO(0x9888), 0x101f02a0 },
	3235	{ _MMIO(0x9888), 0x0c1fa000 },
	3236	{ _MMIO(0x9888), 0x0e1f00aa },
	3237	{ _MMIO(0x9888), 0x18382500 },
	3238	{ _MMIO(0x9888), 0x02388000 },
	3239	{ _MMIO(0x9888), 0x04384000 },
	3240	{ _MMIO(0x9888), 0x06384000 },
	3241	{ _MMIO(0x9888), 0x08384000 },
	3242	{ _MMIO(0x9888), 0x0a384000 },
	3243	{ _MMIO(0x9888), 0x0c388000 },
	3244	{ _MMIO(0x9888), 0x0c398000 },
	3245	{ _MMIO(0x9888), 0x0e39a000 },
	3246	{ _MMIO(0x9888), 0x00398000 },
	3247	{ _MMIO(0x9888), 0x0239a000 },
	3248	{ _MMIO(0x9888), 0x0439a000 },
	3249	{ _MMIO(0x9888), 0x06392000 },
	3250	{ _MMIO(0x9888), 0x1a3a0200 },
	3251	{ _MMIO(0x9888), 0x023a8000 },
	3252	{ _MMIO(0x9888), 0x0c3a8000 },
	3253	{ _MMIO(0x9888), 0x04108000 },
	3254	{ _MMIO(0x9888), 0x04118000 },
	3255	{ _MMIO(0x9888), 0x04127300 },
	3256	{ _MMIO(0x9888), 0x10120000 },
	3257	{ _MMIO(0x9888), 0x1813000a },
	3258	{ _MMIO(0x9888), 0x0a134000 },
	3259	{ _MMIO(0x9888), 0x0a146000 },
	3260	{ _MMIO(0x9888), 0x0c159000 },
	3261	{ _MMIO(0x9888), 0x0e150098 },
	3262	{ _MMIO(0x9888), 0x10150000 },
	3263	{ _MMIO(0x9888), 0x04304000 },
	3264	{ _MMIO(0x9888), 0x04314000 },
	3265	{ _MMIO(0x9888), 0x04320073 },
	3266	{ _MMIO(0x9888), 0x10320000 },
	3267	{ _MMIO(0x9888), 0x04338000 },
	3268	{ _MMIO(0x9888), 0x06338000 },
	3269	{ _MMIO(0x9888), 0x08334000 },
	3270	{ _MMIO(0x9888), 0x0434c000 },
	3271	{ _MMIO(0x9888), 0x02359890 },
	3272	{ _MMIO(0x9888), 0x10350000 },
	3273	{ _MMIO(0x9888), 0x06504000 },
	3274	{ _MMIO(0x9888), 0x06514000 },
	3275	{ _MMIO(0x9888), 0x06520073 },
	3276	{ _MMIO(0x9888), 0x10520000 },
	3277	{ _MMIO(0x9888), 0x18530020 },
	3278	{ _MMIO(0x9888), 0x02538000 },
	3279	{ _MMIO(0x9888), 0x0c534000 },
	3280	{ _MMIO(0x9888), 0x0a548000 },
	3281	{ _MMIO(0x9888), 0x04542000 },
	3282	{ _MMIO(0x9888), 0x0e559000 },
	3283	{ _MMIO(0x9888), 0x00559800 },
	3284	{ _MMIO(0x9888), 0x10550000 },
	3285	{ _MMIO(0x9888), 0x1b8aa000 },
	3286	{ _MMIO(0x9888), 0x1d8a0002 },
	3287	{ _MMIO(0x9888), 0x038a8000 },
	3288	{ _MMIO(0x9888), 0x058a8000 },
	3289	{ _MMIO(0x9888), 0x078a8000 },
	3290	{ _MMIO(0x9888), 0x098a8000 },
	3291	{ _MMIO(0x9888), 0x0b8a8000 },
	3292	{ _MMIO(0x9888), 0x0d8a8000 },
	3293	{ _MMIO(0x9888), 0x278b0015 },
	3294	{ _MMIO(0x9888), 0x238b2a80 },
	3295	{ _MMIO(0x9888), 0x258b0005 },
	3296	{ _MMIO(0x9888), 0x2385002a },
	3297	{ _MMIO(0x9888), 0x1f85aa00 },
	3298	{ _MMIO(0x9888), 0x2185000a },
	3299	{ _MMIO(0x9888), 0x1b830150 },
	3300	{ _MMIO(0x9888), 0x03834000 },
	3301	{ _MMIO(0x9888), 0x05834000 },
	3302	{ _MMIO(0x9888), 0x07834000 },
	3303	{ _MMIO(0x9888), 0x09834000 },
	3304	{ _MMIO(0x9888), 0x0b834000 },
	3305	{ _MMIO(0x9888), 0x0d834000 },
	3306	{ _MMIO(0x9888), 0x0d848000 },
	3307	{ _MMIO(0x9888), 0x0f84c000 },
	3308	{ _MMIO(0x9888), 0x01848000 },
	3309	{ _MMIO(0x9888), 0x0384c000 },
	3310	{ _MMIO(0x9888), 0x0584c000 },
	3311	{ _MMIO(0x9888), 0x07844000 },
	3312	{ _MMIO(0x9888), 0x1d808000 },
	3313	{ _MMIO(0x9888), 0x1f80c000 },
	3314	{ _MMIO(0x9888), 0x11808000 },
	3315	{ _MMIO(0x9888), 0x1380c000 },
	3316	{ _MMIO(0x9888), 0x1580c000 },
	3317	{ _MMIO(0x9888), 0x17804000 },
	3318	{ _MMIO(0x9888), 0x53800000 },
	3319	{ _MMIO(0x9888), 0x47801021 },
	3320	{ _MMIO(0x9888), 0x21800000 },
	3321	{ _MMIO(0x9888), 0x31800000 },
	3322	{ _MMIO(0x9888), 0x4d800000 },
	3323	{ _MMIO(0x9888), 0x3f800c64 },
	3324	{ _MMIO(0x9888), 0x4f800000 },
	3325	{ _MMIO(0x9888), 0x41800c02 },
	3326	};
	3327
	3328	static int
	3329	get_sampler_2_mux_config(struct drm_i915_private *dev_priv,
	3330	const struct i915_oa_reg **regs,
	3331	int *lens)
	3332	{
	3333	int n = 0;
	3334
	3335	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	3336	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	3337
	3338	regs[n] = mux_config_sampler_2;
	3339	lens[n] = ARRAY_SIZE(mux_config_sampler_2);
	3340	n++;
	3341
	3342	return n;
	3343	}
	3344
	3345	static const struct i915_oa_reg b_counter_config_tdl_1[] = {
	3346	{ _MMIO(0x2740), 0x00000000 },
	3347	{ _MMIO(0x2744), 0x00800000 },
	3348	{ _MMIO(0x2710), 0x00000000 },
	3349	{ _MMIO(0x2714), 0xf0800000 },
	3350	{ _MMIO(0x2720), 0x00000000 },
	3351	{ _MMIO(0x2724), 0x30800000 },
	3352	{ _MMIO(0x2770), 0x00000002 },
	3353	{ _MMIO(0x2774), 0x0000fdff },
	3354	{ _MMIO(0x2778), 0x00000000 },
	3355	{ _MMIO(0x277c), 0x0000fe7f },
	3356	{ _MMIO(0x2780), 0x00000002 },
	3357	{ _MMIO(0x2784), 0x0000ffbf },
	3358	{ _MMIO(0x2788), 0x00000000 },
	3359	{ _MMIO(0x278c), 0x0000ffcf },
	3360	{ _MMIO(0x2790), 0x00000002 },
	3361	{ _MMIO(0x2794), 0x0000fff7 },
	3362	{ _MMIO(0x2798), 0x00000000 },
	3363	{ _MMIO(0x279c), 0x0000fff9 },
	3364	};
	3365
	3366	static const struct i915_oa_reg flex_eu_config_tdl_1[] = {
	3367	{ _MMIO(0xe458), 0x00005004 },
	3368	{ _MMIO(0xe558), 0x00010003 },
	3369	{ _MMIO(0xe658), 0x00012011 },
	3370	{ _MMIO(0xe758), 0x00015014 },
	3371	{ _MMIO(0xe45c), 0x00051050 },
	3372	{ _MMIO(0xe55c), 0x00053052 },
	3373	{ _MMIO(0xe65c), 0x00055054 },
	3374	};
	3375
	3376	static const struct i915_oa_reg mux_config_tdl_1[] = {
	3377	{ _MMIO(0x9888), 0x16154d60 },
	3378	{ _MMIO(0x9888), 0x16352e60 },
	3379	{ _MMIO(0x9888), 0x16554d60 },
	3380	{ _MMIO(0x9888), 0x16950000 },
	3381	{ _MMIO(0x9888), 0x16b50000 },
	3382	{ _MMIO(0x9888), 0x16d50000 },
	3383	{ _MMIO(0x9888), 0x005c8000 },
	3384	{ _MMIO(0x9888), 0x045cc000 },
	3385	{ _MMIO(0x9888), 0x065c4000 },
	3386	{ _MMIO(0x9888), 0x083d8000 },
	3387	{ _MMIO(0x9888), 0x0a3d8000 },
	3388	{ _MMIO(0x9888), 0x0458c000 },
	3389	{ _MMIO(0x9888), 0x025b8000 },
	3390	{ _MMIO(0x9888), 0x085b4000 },
	3391	{ _MMIO(0x9888), 0x0a5b4000 },
	3392	{ _MMIO(0x9888), 0x0c5b8000 },
	3393	{ _MMIO(0x9888), 0x0c1fa000 },
	3394	{ _MMIO(0x9888), 0x0e1f00aa },
	3395	{ _MMIO(0x9888), 0x02384000 },
	3396	{ _MMIO(0x9888), 0x04388000 },
	3397	{ _MMIO(0x9888), 0x06388000 },
	3398	{ _MMIO(0x9888), 0x08384000 },
	3399	{ _MMIO(0x9888), 0x0a384000 },
	3400	{ _MMIO(0x9888), 0x0c384000 },
	3401	{ _MMIO(0x9888), 0x00398000 },
	3402	{ _MMIO(0x9888), 0x0239a000 },
	3403	{ _MMIO(0x9888), 0x0439a000 },
	3404	{ _MMIO(0x9888), 0x06392000 },
	3405	{ _MMIO(0x9888), 0x043a8000 },
	3406	{ _MMIO(0x9888), 0x063a8000 },
	3407	{ _MMIO(0x9888), 0x08138000 },
	3408	{ _MMIO(0x9888), 0x0a138000 },
	3409	{ _MMIO(0x9888), 0x06143000 },
	3410	{ _MMIO(0x9888), 0x0415cfc7 },
	3411	{ _MMIO(0x9888), 0x10150000 },
	3412	{ _MMIO(0x9888), 0x02338000 },
	3413	{ _MMIO(0x9888), 0x0c338000 },
	3414	{ _MMIO(0x9888), 0x04342000 },
	3415	{ _MMIO(0x9888), 0x06344000 },
	3416	{ _MMIO(0x9888), 0x0035c700 },
	3417	{ _MMIO(0x9888), 0x063500cf },
	3418	{ _MMIO(0x9888), 0x10350000 },
	3419	{ _MMIO(0x9888), 0x04538000 },
	3420	{ _MMIO(0x9888), 0x06538000 },
	3421	{ _MMIO(0x9888), 0x0454c000 },
	3422	{ _MMIO(0x9888), 0x0255cfc7 },
	3423	{ _MMIO(0x9888), 0x10550000 },
	3424	{ _MMIO(0x9888), 0x06dc8000 },
	3425	{ _MMIO(0x9888), 0x08dc4000 },
	3426	{ _MMIO(0x9888), 0x0cdcc000 },
	3427	{ _MMIO(0x9888), 0x0edcc000 },
	3428	{ _MMIO(0x9888), 0x1abd00a8 },
	3429	{ _MMIO(0x9888), 0x0cd8c000 },
	3430	{ _MMIO(0x9888), 0x0ed84000 },
	3431	{ _MMIO(0x9888), 0x0edb8000 },
	3432	{ _MMIO(0x9888), 0x18db0800 },
	3433	{ _MMIO(0x9888), 0x1adb0254 },
	3434	{ _MMIO(0x9888), 0x0e9faa00 },
	3435	{ _MMIO(0x9888), 0x109f02aa },
	3436	{ _MMIO(0x9888), 0x0eb84000 },
	3437	{ _MMIO(0x9888), 0x16b84000 },
	3438	{ _MMIO(0x9888), 0x18b8156a },
	3439	{ _MMIO(0x9888), 0x06b98000 },
	3440	{ _MMIO(0x9888), 0x08b9a000 },
	3441	{ _MMIO(0x9888), 0x0ab9a000 },
	3442	{ _MMIO(0x9888), 0x0cb9a000 },
	3443	{ _MMIO(0x9888), 0x0eb9a000 },
	3444	{ _MMIO(0x9888), 0x18baa000 },
	3445	{ _MMIO(0x9888), 0x1aba0002 },
	3446	{ _MMIO(0x9888), 0x16934000 },
	3447	{ _MMIO(0x9888), 0x1893000a },
	3448	{ _MMIO(0x9888), 0x0a947000 },
	3449	{ _MMIO(0x9888), 0x0c95c5c1 },
	3450	{ _MMIO(0x9888), 0x0e9500c3 },
	3451	{ _MMIO(0x9888), 0x10950000 },
	3452	{ _MMIO(0x9888), 0x0eb38000 },
	3453	{ _MMIO(0x9888), 0x16b30040 },
	3454	{ _MMIO(0x9888), 0x18b30020 },
	3455	{ _MMIO(0x9888), 0x06b48000 },
	3456	{ _MMIO(0x9888), 0x08b41000 },
	3457	{ _MMIO(0x9888), 0x0ab48000 },
	3458	{ _MMIO(0x9888), 0x06b5c500 },
	3459	{ _MMIO(0x9888), 0x08b500c3 },
	3460	{ _MMIO(0x9888), 0x0eb5c100 },
	3461	{ _MMIO(0x9888), 0x10b50000 },
	3462	{ _MMIO(0x9888), 0x16d31500 },
	3463	{ _MMIO(0x9888), 0x08d4e000 },
	3464	{ _MMIO(0x9888), 0x08d5c100 },
	3465	{ _MMIO(0x9888), 0x0ad5c3c5 },
	3466	{ _MMIO(0x9888), 0x10d50000 },
	3467	{ _MMIO(0x9888), 0x0d88f800 },
	3468	{ _MMIO(0x9888), 0x0f88000f },
	3469	{ _MMIO(0x9888), 0x038a8000 },
	3470	{ _MMIO(0x9888), 0x058a8000 },
	3471	{ _MMIO(0x9888), 0x078a8000 },
	3472	{ _MMIO(0x9888), 0x098a8000 },
	3473	{ _MMIO(0x9888), 0x0b8a8000 },
	3474	{ _MMIO(0x9888), 0x0d8a8000 },
	3475	{ _MMIO(0x9888), 0x258baaa5 },
	3476	{ _MMIO(0x9888), 0x278b002a },
	3477	{ _MMIO(0x9888), 0x238b2a80 },
	3478	{ _MMIO(0x9888), 0x0f8c4000 },
	3479	{ _MMIO(0x9888), 0x178c2000 },
	3480	{ _MMIO(0x9888), 0x198c5500 },
	3481	{ _MMIO(0x9888), 0x1b8c0015 },
	3482	{ _MMIO(0x9888), 0x078d8000 },
	3483	{ _MMIO(0x9888), 0x098da000 },
	3484	{ _MMIO(0x9888), 0x0b8da000 },
	3485	{ _MMIO(0x9888), 0x0d8da000 },
	3486	{ _MMIO(0x9888), 0x0f8da000 },
	3487	{ _MMIO(0x9888), 0x2185aaaa },
	3488	{ _MMIO(0x9888), 0x2385002a },
	3489	{ _MMIO(0x9888), 0x1f85aa00 },
	3490	{ _MMIO(0x9888), 0x0f834000 },
	3491	{ _MMIO(0x9888), 0x19835400 },
	3492	{ _MMIO(0x9888), 0x1b830155 },
	3493	{ _MMIO(0x9888), 0x03834000 },
	3494	{ _MMIO(0x9888), 0x05834000 },
	3495	{ _MMIO(0x9888), 0x07834000 },
	3496	{ _MMIO(0x9888), 0x09834000 },
	3497	{ _MMIO(0x9888), 0x0b834000 },
	3498	{ _MMIO(0x9888), 0x0d834000 },
	3499	{ _MMIO(0x9888), 0x0784c000 },
	3500	{ _MMIO(0x9888), 0x0984c000 },
	3501	{ _MMIO(0x9888), 0x0b84c000 },
	3502	{ _MMIO(0x9888), 0x0d84c000 },
	3503	{ _MMIO(0x9888), 0x0f84c000 },
	3504	{ _MMIO(0x9888), 0x01848000 },
	3505	{ _MMIO(0x9888), 0x0384c000 },
	3506	{ _MMIO(0x9888), 0x0584c000 },
	3507	{ _MMIO(0x9888), 0x1780c000 },
	3508	{ _MMIO(0x9888), 0x1980c000 },
	3509	{ _MMIO(0x9888), 0x1b80c000 },
	3510	{ _MMIO(0x9888), 0x1d80c000 },
	3511	{ _MMIO(0x9888), 0x1f80c000 },
	3512	{ _MMIO(0x9888), 0x11808000 },
	3513	{ _MMIO(0x9888), 0x1380c000 },
	3514	{ _MMIO(0x9888), 0x1580c000 },
	3515	{ _MMIO(0x9888), 0x4f800000 },
	3516	{ _MMIO(0x9888), 0x43800c42 },
	3517	{ _MMIO(0x9888), 0x51800000 },
	3518	{ _MMIO(0x9888), 0x45800063 },
	3519	{ _MMIO(0x9888), 0x53800000 },
	3520	{ _MMIO(0x9888), 0x47800800 },
	3521	{ _MMIO(0x9888), 0x21800000 },
	3522	{ _MMIO(0x9888), 0x31800000 },
	3523	{ _MMIO(0x9888), 0x4d800000 },
	3524	{ _MMIO(0x9888), 0x3f8014a4 },
	3525	{ _MMIO(0x9888), 0x41801042 },
	3526	};
	3527
	3528	static int
	3529	get_tdl_1_mux_config(struct drm_i915_private *dev_priv,
	3530	const struct i915_oa_reg **regs,
	3531	int *lens)
	3532	{
	3533	int n = 0;
	3534
	3535	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	3536	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	3537
	3538	regs[n] = mux_config_tdl_1;
	3539	lens[n] = ARRAY_SIZE(mux_config_tdl_1);
	3540	n++;
	3541
	3542	return n;
	3543	}
	3544
	3545	static const struct i915_oa_reg b_counter_config_tdl_2[] = {
	3546	{ _MMIO(0x2740), 0x00000000 },
	3547	{ _MMIO(0x2744), 0x00800000 },
	3548	{ _MMIO(0x2710), 0x00000000 },
	3549	{ _MMIO(0x2714), 0xf0800000 },
	3550	{ _MMIO(0x2720), 0x00000000 },
	3551	{ _MMIO(0x2724), 0x30800000 },
	3552	{ _MMIO(0x2770), 0x00000002 },
	3553	{ _MMIO(0x2774), 0x0000fdff },
	3554	{ _MMIO(0x2778), 0x00000000 },
	3555	{ _MMIO(0x277c), 0x0000fe7f },
	3556	{ _MMIO(0x2780), 0x00000000 },
	3557	{ _MMIO(0x2784), 0x0000ff9f },
	3558	{ _MMIO(0x2788), 0x00000000 },
	3559	{ _MMIO(0x278c), 0x0000ffe7 },
	3560	{ _MMIO(0x2790), 0x00000002 },
	3561	{ _MMIO(0x2794), 0x0000fffb },
	3562	{ _MMIO(0x2798), 0x00000002 },
	3563	{ _MMIO(0x279c), 0x0000fffd },
	3564	};
	3565
	3566	static const struct i915_oa_reg flex_eu_config_tdl_2[] = {
	3567	{ _MMIO(0xe458), 0x00005004 },
	3568	{ _MMIO(0xe558), 0x00010003 },
	3569	{ _MMIO(0xe658), 0x00012011 },
	3570	{ _MMIO(0xe758), 0x00015014 },
	3571	{ _MMIO(0xe45c), 0x00051050 },
	3572	{ _MMIO(0xe55c), 0x00053052 },
	3573	{ _MMIO(0xe65c), 0x00055054 },
	3574	};
	3575
	3576	static const struct i915_oa_reg mux_config_tdl_2[] = {
	3577	{ _MMIO(0x9888), 0x16150000 },
	3578	{ _MMIO(0x9888), 0x16350000 },
	3579	{ _MMIO(0x9888), 0x16550000 },
	3580	{ _MMIO(0x9888), 0x16952e60 },
	3581	{ _MMIO(0x9888), 0x16b54d60 },
	3582	{ _MMIO(0x9888), 0x16d52e60 },
	3583	{ _MMIO(0x9888), 0x065c8000 },
	3584	{ _MMIO(0x9888), 0x085cc000 },
	3585	{ _MMIO(0x9888), 0x0a5cc000 },
	3586	{ _MMIO(0x9888), 0x0c5c4000 },
	3587	{ _MMIO(0x9888), 0x0e3d8000 },
	3588	{ _MMIO(0x9888), 0x183da000 },
	3589	{ _MMIO(0x9888), 0x06588000 },
	3590	{ _MMIO(0x9888), 0x08588000 },
	3591	{ _MMIO(0x9888), 0x0a584000 },
	3592	{ _MMIO(0x9888), 0x0e5b4000 },
	3593	{ _MMIO(0x9888), 0x185b5800 },
	3594	{ _MMIO(0x9888), 0x1a5b000a },
	3595	{ _MMIO(0x9888), 0x0e1faa00 },
	3596	{ _MMIO(0x9888), 0x101f02aa },
	3597	{ _MMIO(0x9888), 0x0e384000 },
	3598	{ _MMIO(0x9888), 0x16384000 },
	3599	{ _MMIO(0x9888), 0x18382a55 },
	3600	{ _MMIO(0x9888), 0x06398000 },
	3601	{ _MMIO(0x9888), 0x0839a000 },
	3602	{ _MMIO(0x9888), 0x0a39a000 },
	3603	{ _MMIO(0x9888), 0x0c39a000 },
	3604	{ _MMIO(0x9888), 0x0e39a000 },
	3605	{ _MMIO(0x9888), 0x1a3a02a0 },
	3606	{ _MMIO(0x9888), 0x0e138000 },
	3607	{ _MMIO(0x9888), 0x16130500 },
	3608	{ _MMIO(0x9888), 0x06148000 },
	3609	{ _MMIO(0x9888), 0x08146000 },
	3610	{ _MMIO(0x9888), 0x0615c100 },
	3611	{ _MMIO(0x9888), 0x0815c500 },
	3612	{ _MMIO(0x9888), 0x0a1500c3 },
	3613	{ _MMIO(0x9888), 0x10150000 },
	3614	{ _MMIO(0x9888), 0x16335040 },
	3615	{ _MMIO(0x9888), 0x08349000 },
	3616	{ _MMIO(0x9888), 0x0a341000 },
	3617	{ _MMIO(0x9888), 0x083500c1 },
	3618	{ _MMIO(0x9888), 0x0a35c500 },
	3619	{ _MMIO(0x9888), 0x0c3500c3 },
	3620	{ _MMIO(0x9888), 0x10350000 },
	3621	{ _MMIO(0x9888), 0x1853002a },
	3622	{ _MMIO(0x9888), 0x0a54e000 },
	3623	{ _MMIO(0x9888), 0x0c55c500 },
	3624	{ _MMIO(0x9888), 0x0e55c1c3 },
	3625	{ _MMIO(0x9888), 0x10550000 },
	3626	{ _MMIO(0x9888), 0x00dc8000 },
	3627	{ _MMIO(0x9888), 0x02dcc000 },
	3628	{ _MMIO(0x9888), 0x04dc4000 },
	3629	{ _MMIO(0x9888), 0x04bd8000 },
	3630	{ _MMIO(0x9888), 0x06bd8000 },
	3631	{ _MMIO(0x9888), 0x02d8c000 },
	3632	{ _MMIO(0x9888), 0x02db8000 },
	3633	{ _MMIO(0x9888), 0x04db4000 },
	3634	{ _MMIO(0x9888), 0x06db4000 },
	3635	{ _MMIO(0x9888), 0x08db8000 },
	3636	{ _MMIO(0x9888), 0x0c9fa000 },
	3637	{ _MMIO(0x9888), 0x0e9f00aa },
	3638	{ _MMIO(0x9888), 0x02b84000 },
	3639	{ _MMIO(0x9888), 0x04b84000 },
	3640	{ _MMIO(0x9888), 0x06b84000 },
	3641	{ _MMIO(0x9888), 0x08b84000 },
	3642	{ _MMIO(0x9888), 0x0ab88000 },
	3643	{ _MMIO(0x9888), 0x0cb88000 },
	3644	{ _MMIO(0x9888), 0x00b98000 },
	3645	{ _MMIO(0x9888), 0x02b9a000 },
	3646	{ _MMIO(0x9888), 0x04b9a000 },
	3647	{ _MMIO(0x9888), 0x06b92000 },
	3648	{ _MMIO(0x9888), 0x0aba8000 },
	3649	{ _MMIO(0x9888), 0x0cba8000 },
	3650	{ _MMIO(0x9888), 0x04938000 },
	3651	{ _MMIO(0x9888), 0x06938000 },
	3652	{ _MMIO(0x9888), 0x0494c000 },
	3653	{ _MMIO(0x9888), 0x0295cfc7 },
	3654	{ _MMIO(0x9888), 0x10950000 },
	3655	{ _MMIO(0x9888), 0x02b38000 },
	3656	{ _MMIO(0x9888), 0x08b38000 },
	3657	{ _MMIO(0x9888), 0x04b42000 },
	3658	{ _MMIO(0x9888), 0x06b41000 },
	3659	{ _MMIO(0x9888), 0x00b5c700 },
	3660	{ _MMIO(0x9888), 0x04b500cf },
	3661	{ _MMIO(0x9888), 0x10b50000 },
	3662	{ _MMIO(0x9888), 0x0ad38000 },
	3663	{ _MMIO(0x9888), 0x0cd38000 },
	3664	{ _MMIO(0x9888), 0x06d46000 },
	3665	{ _MMIO(0x9888), 0x04d5c700 },
	3666	{ _MMIO(0x9888), 0x06d500cf },
	3667	{ _MMIO(0x9888), 0x10d50000 },
	3668	{ _MMIO(0x9888), 0x03888000 },
	3669	{ _MMIO(0x9888), 0x05888000 },
	3670	{ _MMIO(0x9888), 0x07888000 },
	3671	{ _MMIO(0x9888), 0x09888000 },
	3672	{ _MMIO(0x9888), 0x0b888000 },
	3673	{ _MMIO(0x9888), 0x0d880400 },
	3674	{ _MMIO(0x9888), 0x0f8a8000 },
	3675	{ _MMIO(0x9888), 0x198a8000 },
	3676	{ _MMIO(0x9888), 0x1b8aaaa0 },
	3677	{ _MMIO(0x9888), 0x1d8a0002 },
	3678	{ _MMIO(0x9888), 0x258b555a },
	3679	{ _MMIO(0x9888), 0x278b0015 },
	3680	{ _MMIO(0x9888), 0x238b5500 },
	3681	{ _MMIO(0x9888), 0x038c4000 },
	3682	{ _MMIO(0x9888), 0x058c4000 },
	3683	{ _MMIO(0x9888), 0x078c4000 },
	3684	{ _MMIO(0x9888), 0x098c4000 },
	3685	{ _MMIO(0x9888), 0x0b8c4000 },
	3686	{ _MMIO(0x9888), 0x0d8c4000 },
	3687	{ _MMIO(0x9888), 0x018d8000 },
	3688	{ _MMIO(0x9888), 0x038da000 },
	3689	{ _MMIO(0x9888), 0x058da000 },
	3690	{ _MMIO(0x9888), 0x078d2000 },
	3691	{ _MMIO(0x9888), 0x2185aaaa },
	3692	{ _MMIO(0x9888), 0x2385002a },
	3693	{ _MMIO(0x9888), 0x1f85aa00 },
	3694	{ _MMIO(0x9888), 0x0f834000 },
	3695	{ _MMIO(0x9888), 0x19835400 },
	3696	{ _MMIO(0x9888), 0x1b830155 },
	3697	{ _MMIO(0x9888), 0x03834000 },
	3698	{ _MMIO(0x9888), 0x05834000 },
	3699	{ _MMIO(0x9888), 0x07834000 },
	3700	{ _MMIO(0x9888), 0x09834000 },
	3701	{ _MMIO(0x9888), 0x0b834000 },
	3702	{ _MMIO(0x9888), 0x0d834000 },
	3703	{ _MMIO(0x9888), 0x0784c000 },
	3704	{ _MMIO(0x9888), 0x0984c000 },
	3705	{ _MMIO(0x9888), 0x0b84c000 },
	3706	{ _MMIO(0x9888), 0x0d84c000 },
	3707	{ _MMIO(0x9888), 0x0f84c000 },
	3708	{ _MMIO(0x9888), 0x01848000 },
	3709	{ _MMIO(0x9888), 0x0384c000 },
	3710	{ _MMIO(0x9888), 0x0584c000 },
	3711	{ _MMIO(0x9888), 0x1780c000 },
	3712	{ _MMIO(0x9888), 0x1980c000 },
	3713	{ _MMIO(0x9888), 0x1b80c000 },
	3714	{ _MMIO(0x9888), 0x1d80c000 },
	3715	{ _MMIO(0x9888), 0x1f80c000 },
	3716	{ _MMIO(0x9888), 0x11808000 },
	3717	{ _MMIO(0x9888), 0x1380c000 },
	3718	{ _MMIO(0x9888), 0x1580c000 },
	3719	{ _MMIO(0x9888), 0x4f800000 },
	3720	{ _MMIO(0x9888), 0x43800882 },
	3721	{ _MMIO(0x9888), 0x51800000 },
	3722	{ _MMIO(0x9888), 0x45801082 },
	3723	{ _MMIO(0x9888), 0x53800000 },
	3724	{ _MMIO(0x9888), 0x478014a5 },
	3725	{ _MMIO(0x9888), 0x21800000 },
	3726	{ _MMIO(0x9888), 0x31800000 },
	3727	{ _MMIO(0x9888), 0x4d800000 },
	3728	{ _MMIO(0x9888), 0x3f800002 },
	3729	{ _MMIO(0x9888), 0x41800c62 },
	3730	};
	3731
	3732	static int
	3733	get_tdl_2_mux_config(struct drm_i915_private *dev_priv,
	3734	const struct i915_oa_reg **regs,
	3735	int *lens)
	3736	{
	3737	int n = 0;
	3738
	3739	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	3740	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	3741
	3742	regs[n] = mux_config_tdl_2;
	3743	lens[n] = ARRAY_SIZE(mux_config_tdl_2);
	3744	n++;
	3745
	3746	return n;
	3747	}
	3748
	3749	static const struct i915_oa_reg b_counter_config_compute_extra[] = {
	3750	{ _MMIO(0x2740), 0x00000000 },
	3751	{ _MMIO(0x2744), 0x00800000 },
	3752	{ _MMIO(0x2710), 0x00000000 },
	3753	{ _MMIO(0x2714), 0x00800000 },
	3754	{ _MMIO(0x2720), 0x00000000 },
	3755	{ _MMIO(0x2724), 0x00800000 },
	3756	};
	3757
	3758	static const struct i915_oa_reg flex_eu_config_compute_extra[] = {
	3759	{ _MMIO(0xe458), 0x00001000 },
	3760	{ _MMIO(0xe558), 0x00003002 },
	3761	{ _MMIO(0xe658), 0x00005004 },
	3762	{ _MMIO(0xe758), 0x00011010 },
	3763	{ _MMIO(0xe45c), 0x00050012 },
	3764	{ _MMIO(0xe55c), 0x00052051 },
	3765	{ _MMIO(0xe65c), 0x00000008 },
	3766	};
	3767
	3768	static const struct i915_oa_reg mux_config_compute_extra[] = {
	3769	{ _MMIO(0x9888), 0x161503e0 },
	3770	{ _MMIO(0x9888), 0x163503e0 },
	3771	{ _MMIO(0x9888), 0x165503e0 },
	3772	{ _MMIO(0x9888), 0x169503e0 },
	3773	{ _MMIO(0x9888), 0x16b503e0 },
	3774	{ _MMIO(0x9888), 0x16d503e0 },
	3775	{ _MMIO(0x9888), 0x045cc000 },
	3776	{ _MMIO(0x9888), 0x083d8000 },
	3777	{ _MMIO(0x9888), 0x04584000 },
	3778	{ _MMIO(0x9888), 0x085b4000 },
	3779	{ _MMIO(0x9888), 0x0a5b8000 },
	3780	{ _MMIO(0x9888), 0x0e1f00a8 },
	3781	{ _MMIO(0x9888), 0x08384000 },
	3782	{ _MMIO(0x9888), 0x0a384000 },
	3783	{ _MMIO(0x9888), 0x0c388000 },
	3784	{ _MMIO(0x9888), 0x0439a000 },
	3785	{ _MMIO(0x9888), 0x06392000 },
	3786	{ _MMIO(0x9888), 0x0c3a8000 },
	3787	{ _MMIO(0x9888), 0x08138000 },
	3788	{ _MMIO(0x9888), 0x06141000 },
	3789	{ _MMIO(0x9888), 0x041500c3 },
	3790	{ _MMIO(0x9888), 0x10150000 },
	3791	{ _MMIO(0x9888), 0x0a338000 },
	3792	{ _MMIO(0x9888), 0x06342000 },
	3793	{ _MMIO(0x9888), 0x0435c300 },
	3794	{ _MMIO(0x9888), 0x10350000 },
	3795	{ _MMIO(0x9888), 0x0c538000 },
	3796	{ _MMIO(0x9888), 0x06544000 },
	3797	{ _MMIO(0x9888), 0x065500c3 },
	3798	{ _MMIO(0x9888), 0x10550000 },
	3799	{ _MMIO(0x9888), 0x00dc8000 },
	3800	{ _MMIO(0x9888), 0x02dc4000 },
	3801	{ _MMIO(0x9888), 0x02bd8000 },
	3802	{ _MMIO(0x9888), 0x00d88000 },
	3803	{ _MMIO(0x9888), 0x02db4000 },
	3804	{ _MMIO(0x9888), 0x04db8000 },
	3805	{ _MMIO(0x9888), 0x0c9fa000 },
	3806	{ _MMIO(0x9888), 0x0e9f0002 },
	3807	{ _MMIO(0x9888), 0x02b84000 },
	3808	{ _MMIO(0x9888), 0x04b84000 },
	3809	{ _MMIO(0x9888), 0x06b88000 },
	3810	{ _MMIO(0x9888), 0x00b98000 },
	3811	{ _MMIO(0x9888), 0x02b9a000 },
	3812	{ _MMIO(0x9888), 0x06ba8000 },
	3813	{ _MMIO(0x9888), 0x02938000 },
	3814	{ _MMIO(0x9888), 0x04942000 },
	3815	{ _MMIO(0x9888), 0x0095c300 },
	3816	{ _MMIO(0x9888), 0x10950000 },
	3817	{ _MMIO(0x9888), 0x04b38000 },
	3818	{ _MMIO(0x9888), 0x04b44000 },
	3819	{ _MMIO(0x9888), 0x02b500c3 },
	3820	{ _MMIO(0x9888), 0x10b50000 },
	3821	{ _MMIO(0x9888), 0x06d38000 },
	3822	{ _MMIO(0x9888), 0x04d48000 },
	3823	{ _MMIO(0x9888), 0x02d5c300 },
	3824	{ _MMIO(0x9888), 0x10d50000 },
	3825	{ _MMIO(0x9888), 0x03888000 },
	3826	{ _MMIO(0x9888), 0x05888000 },
	3827	{ _MMIO(0x9888), 0x07888000 },
	3828	{ _MMIO(0x9888), 0x098a8000 },
	3829	{ _MMIO(0x9888), 0x0b8a8000 },
	3830	{ _MMIO(0x9888), 0x0d8a8000 },
	3831	{ _MMIO(0x9888), 0x238b3500 },
	3832	{ _MMIO(0x9888), 0x258b0005 },
	3833	{ _MMIO(0x9888), 0x038c4000 },
	3834	{ _MMIO(0x9888), 0x058c4000 },
	3835	{ _MMIO(0x9888), 0x078c4000 },
	3836	{ _MMIO(0x9888), 0x018d8000 },
	3837	{ _MMIO(0x9888), 0x038da000 },
	3838	{ _MMIO(0x9888), 0x1f85aa00 },
	3839	{ _MMIO(0x9888), 0x2185000a },
	3840	{ _MMIO(0x9888), 0x03834000 },
	3841	{ _MMIO(0x9888), 0x05834000 },
	3842	{ _MMIO(0x9888), 0x07834000 },
	3843	{ _MMIO(0x9888), 0x09834000 },
	3844	{ _MMIO(0x9888), 0x0b834000 },
	3845	{ _MMIO(0x9888), 0x0d834000 },
	3846	{ _MMIO(0x9888), 0x01848000 },
	3847	{ _MMIO(0x9888), 0x0384c000 },
	3848	{ _MMIO(0x9888), 0x0584c000 },
	3849	{ _MMIO(0x9888), 0x07844000 },
	3850	{ _MMIO(0x9888), 0x11808000 },
	3851	{ _MMIO(0x9888), 0x1380c000 },
	3852	{ _MMIO(0x9888), 0x1580c000 },
	3853	{ _MMIO(0x9888), 0x17804000 },
	3854	{ _MMIO(0x9888), 0x21800000 },
	3855	{ _MMIO(0x9888), 0x4d800000 },
	3856	{ _MMIO(0x9888), 0x3f800c40 },
	3857	{ _MMIO(0x9888), 0x4f800000 },
	3858	{ _MMIO(0x9888), 0x41801482 },
	3859	{ _MMIO(0x9888), 0x31800000 },
	3860	};
	3861
	3862	static int
	3863	get_compute_extra_mux_config(struct drm_i915_private *dev_priv,
	3864	const struct i915_oa_reg **regs,
	3865	int *lens)
	3866	{
	3867	int n = 0;
	3868
	3869	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	3870	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	3871
	3872	regs[n] = mux_config_compute_extra;
	3873	lens[n] = ARRAY_SIZE(mux_config_compute_extra);
	3874	n++;
	3875
	3876	return n;
	3877	}
	3878
	3879	static const struct i915_oa_reg b_counter_config_vme_pipe[] = {
	3880	{ _MMIO(0x2740), 0x00000000 },
	3881	{ _MMIO(0x2710), 0x00000000 },
	3882	{ _MMIO(0x2714), 0xf0800000 },
	3883	{ _MMIO(0x2720), 0x00000000 },
	3884	{ _MMIO(0x2724), 0x30800000 },
	3885	{ _MMIO(0x2770), 0x00100030 },
	3886	{ _MMIO(0x2774), 0x0000fff9 },
	3887	{ _MMIO(0x2778), 0x00000002 },
	3888	{ _MMIO(0x277c), 0x0000fffc },
	3889	{ _MMIO(0x2780), 0x00000002 },
	3890	{ _MMIO(0x2784), 0x0000fff3 },
	3891	{ _MMIO(0x2788), 0x00100180 },
	3892	{ _MMIO(0x278c), 0x0000ffcf },
	3893	{ _MMIO(0x2790), 0x00000002 },
	3894	{ _MMIO(0x2794), 0x0000ffcf },
	3895	{ _MMIO(0x2798), 0x00000002 },
	3896	{ _MMIO(0x279c), 0x0000ff3f },
	3897	};
	3898
	3899	static const struct i915_oa_reg flex_eu_config_vme_pipe[] = {
	3900	{ _MMIO(0xe458), 0x00005004 },
	3901	{ _MMIO(0xe558), 0x00008003 },
	3902	};
	3903
	3904	static const struct i915_oa_reg mux_config_vme_pipe[] = {
	3905	{ _MMIO(0x9888), 0x14100812 },
	3906	{ _MMIO(0x9888), 0x14125800 },
	3907	{ _MMIO(0x9888), 0x161200c0 },
	3908	{ _MMIO(0x9888), 0x14300812 },
	3909	{ _MMIO(0x9888), 0x14325800 },
	3910	{ _MMIO(0x9888), 0x163200c0 },
	3911	{ _MMIO(0x9888), 0x005c4000 },
	3912	{ _MMIO(0x9888), 0x065c8000 },
	3913	{ _MMIO(0x9888), 0x085cc000 },
	3914	{ _MMIO(0x9888), 0x0a5cc000 },
	3915	{ _MMIO(0x9888), 0x0c5cc000 },
	3916	{ _MMIO(0x9888), 0x003d8000 },
	3917	{ _MMIO(0x9888), 0x0e3d8000 },
	3918	{ _MMIO(0x9888), 0x183d2800 },
	3919	{ _MMIO(0x9888), 0x00584000 },
	3920	{ _MMIO(0x9888), 0x06588000 },
	3921	{ _MMIO(0x9888), 0x0858c000 },
	3922	{ _MMIO(0x9888), 0x005b4000 },
	3923	{ _MMIO(0x9888), 0x0e5b4000 },
	3924	{ _MMIO(0x9888), 0x185b9400 },
	3925	{ _MMIO(0x9888), 0x1a5b002a },
	3926	{ _MMIO(0x9888), 0x0c1f0800 },
	3927	{ _MMIO(0x9888), 0x0e1faa00 },
	3928	{ _MMIO(0x9888), 0x101f002a },
	3929	{ _MMIO(0x9888), 0x00384000 },
	3930	{ _MMIO(0x9888), 0x0e384000 },
	3931	{ _MMIO(0x9888), 0x16384000 },
	3932	{ _MMIO(0x9888), 0x18380155 },
	3933	{ _MMIO(0x9888), 0x00392000 },
	3934	{ _MMIO(0x9888), 0x06398000 },
	3935	{ _MMIO(0x9888), 0x0839a000 },
	3936	{ _MMIO(0x9888), 0x0a39a000 },
	3937	{ _MMIO(0x9888), 0x0c39a000 },
	3938	{ _MMIO(0x9888), 0x00100047 },
	3939	{ _MMIO(0x9888), 0x06101a80 },
	3940	{ _MMIO(0x9888), 0x10100000 },
	3941	{ _MMIO(0x9888), 0x0810c000 },
	3942	{ _MMIO(0x9888), 0x0811c000 },
	3943	{ _MMIO(0x9888), 0x08126151 },
	3944	{ _MMIO(0x9888), 0x10120000 },
	3945	{ _MMIO(0x9888), 0x00134000 },
	3946	{ _MMIO(0x9888), 0x0e134000 },
	3947	{ _MMIO(0x9888), 0x161300a0 },
	3948	{ _MMIO(0x9888), 0x0a301ac7 },
	3949	{ _MMIO(0x9888), 0x10300000 },
	3950	{ _MMIO(0x9888), 0x0c30c000 },
	3951	{ _MMIO(0x9888), 0x0c31c000 },
	3952	{ _MMIO(0x9888), 0x0c326151 },
	3953	{ _MMIO(0x9888), 0x10320000 },
	3954	{ _MMIO(0x9888), 0x16332a00 },
	3955	{ _MMIO(0x9888), 0x18330001 },
	3956	{ _MMIO(0x9888), 0x018a8000 },
	3957	{ _MMIO(0x9888), 0x0f8a8000 },
	3958	{ _MMIO(0x9888), 0x198a8000 },
	3959	{ _MMIO(0x9888), 0x1b8a2aa0 },
	3960	{ _MMIO(0x9888), 0x238b0020 },
	3961	{ _MMIO(0x9888), 0x258b5550 },
	3962	{ _MMIO(0x9888), 0x278b0001 },
	3963	{ _MMIO(0x9888), 0x1f850080 },
	3964	{ _MMIO(0x9888), 0x2185aaa0 },
	3965	{ _MMIO(0x9888), 0x23850002 },
	3966	{ _MMIO(0x9888), 0x01834000 },
	3967	{ _MMIO(0x9888), 0x0f834000 },
	3968	{ _MMIO(0x9888), 0x19835400 },
	3969	{ _MMIO(0x9888), 0x1b830015 },
	3970	{ _MMIO(0x9888), 0x01844000 },
	3971	{ _MMIO(0x9888), 0x07848000 },
	3972	{ _MMIO(0x9888), 0x0984c000 },
	3973	{ _MMIO(0x9888), 0x0b84c000 },
	3974	{ _MMIO(0x9888), 0x0d84c000 },
	3975	{ _MMIO(0x9888), 0x11804000 },
	3976	{ _MMIO(0x9888), 0x17808000 },
	3977	{ _MMIO(0x9888), 0x1980c000 },
	3978	{ _MMIO(0x9888), 0x1b80c000 },
	3979	{ _MMIO(0x9888), 0x1d80c000 },
	3980	{ _MMIO(0x9888), 0x4d800000 },
	3981	{ _MMIO(0x9888), 0x3d800800 },
	3982	{ _MMIO(0x9888), 0x4f800000 },
	3983	{ _MMIO(0x9888), 0x43800002 },
	3984	{ _MMIO(0x9888), 0x51800000 },
	3985	{ _MMIO(0x9888), 0x45800884 },
	3986	{ _MMIO(0x9888), 0x53800000 },
	3987	{ _MMIO(0x9888), 0x47800002 },
	3988	{ _MMIO(0x9888), 0x21800000 },
	3989	{ _MMIO(0x9888), 0x31800000 },
	3990	};
	3991
	3992	static int
	3993	get_vme_pipe_mux_config(struct drm_i915_private *dev_priv,
	3994	const struct i915_oa_reg **regs,
	3995	int *lens)
	3996	{
	3997	int n = 0;
	3998
	3999	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	4000	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	4001
	4002	regs[n] = mux_config_vme_pipe;
	4003	lens[n] = ARRAY_SIZE(mux_config_vme_pipe);
	4004	n++;
	4005
	4006	return n;
	4007	}
	4008
	4009	static const struct i915_oa_reg b_counter_config_test_oa[] = {
	4010	{ _MMIO(0x2740), 0x00000000 },
	4011	{ _MMIO(0x2744), 0x00800000 },
	4012	{ _MMIO(0x2714), 0xf0800000 },
	4013	{ _MMIO(0x2710), 0x00000000 },
	4014	{ _MMIO(0x2724), 0xf0800000 },
	4015	{ _MMIO(0x2720), 0x00000000 },
	4016	{ _MMIO(0x2770), 0x00000004 },
	4017	{ _MMIO(0x2774), 0x00000000 },
	4018	{ _MMIO(0x2778), 0x00000003 },
	4019	{ _MMIO(0x277c), 0x00000000 },
	4020	{ _MMIO(0x2780), 0x00000007 },
	4021	{ _MMIO(0x2784), 0x00000000 },
	4022	{ _MMIO(0x2788), 0x00100002 },
	4023	{ _MMIO(0x278c), 0x0000fff7 },
	4024	{ _MMIO(0x2790), 0x00100002 },
	4025	{ _MMIO(0x2794), 0x0000ffcf },
	4026	{ _MMIO(0x2798), 0x00100082 },
	4027	{ _MMIO(0x279c), 0x0000ffef },
	4028	{ _MMIO(0x27a0), 0x001000c2 },
	4029	{ _MMIO(0x27a4), 0x0000ffe7 },
	4030	{ _MMIO(0x27a8), 0x00100001 },
	4031	{ _MMIO(0x27ac), 0x0000ffe7 },
	4032	};
	4033
	4034	static const struct i915_oa_reg flex_eu_config_test_oa[] = {
	4035	};
	4036
	4037	static const struct i915_oa_reg mux_config_test_oa[] = {
	4038	{ _MMIO(0x9888), 0x198b0000 },
	4039	{ _MMIO(0x9888), 0x078b0066 },
	4040	{ _MMIO(0x9888), 0x118b0000 },
	4041	{ _MMIO(0x9888), 0x258b0000 },
	4042	{ _MMIO(0x9888), 0x21850008 },
	4043	{ _MMIO(0x9888), 0x0d834000 },
	4044	{ _MMIO(0x9888), 0x07844000 },
	4045	{ _MMIO(0x9888), 0x17804000 },
	4046	{ _MMIO(0x9888), 0x21800000 },
	4047	{ _MMIO(0x9888), 0x4f800000 },
	4048	{ _MMIO(0x9888), 0x41800000 },
	4049	{ _MMIO(0x9888), 0x31800000 },
	4050	};
	4051
	4052	static int
	4053	get_test_oa_mux_config(struct drm_i915_private *dev_priv,
	4054	const struct i915_oa_reg **regs,
	4055	int *lens)
	4056	{
	4057	int n = 0;
	4058
	4059	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	4060	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	4061
	4062	regs[n] = mux_config_test_oa;
	4063	lens[n] = ARRAY_SIZE(mux_config_test_oa);
	4064	n++;
	4065
	4066	return n;
	4067	}
	4068
	4069	int i915_oa_select_metric_set_bdw(struct drm_i915_private *dev_priv)
	4070	{
	4071	dev_priv->perf.oa.n_mux_configs = 0;
	4072	dev_priv->perf.oa.b_counter_regs = NULL;
	4073	dev_priv->perf.oa.b_counter_regs_len = 0;
	4074	dev_priv->perf.oa.flex_regs = NULL;
	4075	dev_priv->perf.oa.flex_regs_len = 0;
	4076
	4077	switch (dev_priv->perf.oa.metrics_set) {
	4078	case METRIC_SET_ID_RENDER_BASIC:
	4079	dev_priv->perf.oa.n_mux_configs =
	4080	get_render_basic_mux_config(dev_priv,
	4081	dev_priv->perf.oa.mux_regs,
	4082	dev_priv->perf.oa.mux_regs_lens);
	4083	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4084	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_BASIC\" metric set\n");
	4085
	4086	/* EINVAL because *_register_sysfs already checked this
	4087	* and so it wouldn't have been advertised to userspace and
	4088	* so shouldn't have been requested
	4089	*/
	4090	return -EINVAL;
	4091	}
	4092
	4093	dev_priv->perf.oa.b_counter_regs =
	4094	b_counter_config_render_basic;
	4095	dev_priv->perf.oa.b_counter_regs_len =
	4096	ARRAY_SIZE(b_counter_config_render_basic);
	4097
	4098	dev_priv->perf.oa.flex_regs =
	4099	flex_eu_config_render_basic;
	4100	dev_priv->perf.oa.flex_regs_len =
	4101	ARRAY_SIZE(flex_eu_config_render_basic);
	4102
	4103	return 0;
	4104	case METRIC_SET_ID_COMPUTE_BASIC:
	4105	dev_priv->perf.oa.n_mux_configs =
	4106	get_compute_basic_mux_config(dev_priv,
	4107	dev_priv->perf.oa.mux_regs,
	4108	dev_priv->perf.oa.mux_regs_lens);
	4109	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4110	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_BASIC\" metric set\n");
	4111
	4112	/* EINVAL because *_register_sysfs already checked this
	4113	* and so it wouldn't have been advertised to userspace and
	4114	* so shouldn't have been requested
	4115	*/
	4116	return -EINVAL;
	4117	}
	4118
	4119	dev_priv->perf.oa.b_counter_regs =
	4120	b_counter_config_compute_basic;
	4121	dev_priv->perf.oa.b_counter_regs_len =
	4122	ARRAY_SIZE(b_counter_config_compute_basic);
	4123
	4124	dev_priv->perf.oa.flex_regs =
	4125	flex_eu_config_compute_basic;
	4126	dev_priv->perf.oa.flex_regs_len =
	4127	ARRAY_SIZE(flex_eu_config_compute_basic);
	4128
	4129	return 0;
	4130	case METRIC_SET_ID_RENDER_PIPE_PROFILE:
	4131	dev_priv->perf.oa.n_mux_configs =
	4132	get_render_pipe_profile_mux_config(dev_priv,
	4133	dev_priv->perf.oa.mux_regs,
	4134	dev_priv->perf.oa.mux_regs_lens);
	4135	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4136	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_PIPE_PROFILE\" metric set\n");
	4137
	4138	/* EINVAL because *_register_sysfs already checked this
	4139	* and so it wouldn't have been advertised to userspace and
	4140	* so shouldn't have been requested
	4141	*/
	4142	return -EINVAL;
	4143	}
	4144
	4145	dev_priv->perf.oa.b_counter_regs =
	4146	b_counter_config_render_pipe_profile;
	4147	dev_priv->perf.oa.b_counter_regs_len =
	4148	ARRAY_SIZE(b_counter_config_render_pipe_profile);
	4149
	4150	dev_priv->perf.oa.flex_regs =
	4151	flex_eu_config_render_pipe_profile;
	4152	dev_priv->perf.oa.flex_regs_len =
	4153	ARRAY_SIZE(flex_eu_config_render_pipe_profile);
	4154
	4155	return 0;
	4156	case METRIC_SET_ID_MEMORY_READS:
	4157	dev_priv->perf.oa.n_mux_configs =
	4158	get_memory_reads_mux_config(dev_priv,
	4159	dev_priv->perf.oa.mux_regs,
	4160	dev_priv->perf.oa.mux_regs_lens);
	4161	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4162	DRM_DEBUG_DRIVER("No suitable MUX config for \"MEMORY_READS\" metric set\n");
	4163
	4164	/* EINVAL because *_register_sysfs already checked this
	4165	* and so it wouldn't have been advertised to userspace and
	4166	* so shouldn't have been requested
	4167	*/
	4168	return -EINVAL;
	4169	}
	4170
	4171	dev_priv->perf.oa.b_counter_regs =
	4172	b_counter_config_memory_reads;
	4173	dev_priv->perf.oa.b_counter_regs_len =
	4174	ARRAY_SIZE(b_counter_config_memory_reads);
	4175
	4176	dev_priv->perf.oa.flex_regs =
	4177	flex_eu_config_memory_reads;
	4178	dev_priv->perf.oa.flex_regs_len =
	4179	ARRAY_SIZE(flex_eu_config_memory_reads);
	4180
	4181	return 0;
	4182	case METRIC_SET_ID_MEMORY_WRITES:
	4183	dev_priv->perf.oa.n_mux_configs =
	4184	get_memory_writes_mux_config(dev_priv,
	4185	dev_priv->perf.oa.mux_regs,
	4186	dev_priv->perf.oa.mux_regs_lens);
	4187	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4188	DRM_DEBUG_DRIVER("No suitable MUX config for \"MEMORY_WRITES\" metric set\n");
	4189
	4190	/* EINVAL because *_register_sysfs already checked this
	4191	* and so it wouldn't have been advertised to userspace and
	4192	* so shouldn't have been requested
	4193	*/
	4194	return -EINVAL;
	4195	}
	4196
	4197	dev_priv->perf.oa.b_counter_regs =
	4198	b_counter_config_memory_writes;
	4199	dev_priv->perf.oa.b_counter_regs_len =
	4200	ARRAY_SIZE(b_counter_config_memory_writes);
	4201
	4202	dev_priv->perf.oa.flex_regs =
	4203	flex_eu_config_memory_writes;
	4204	dev_priv->perf.oa.flex_regs_len =
	4205	ARRAY_SIZE(flex_eu_config_memory_writes);
	4206
	4207	return 0;
	4208	case METRIC_SET_ID_COMPUTE_EXTENDED:
	4209	dev_priv->perf.oa.n_mux_configs =
	4210	get_compute_extended_mux_config(dev_priv,
	4211	dev_priv->perf.oa.mux_regs,
	4212	dev_priv->perf.oa.mux_regs_lens);
	4213	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4214	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_EXTENDED\" metric set\n");
	4215
	4216	/* EINVAL because *_register_sysfs already checked this
	4217	* and so it wouldn't have been advertised to userspace and
	4218	* so shouldn't have been requested
	4219	*/
	4220	return -EINVAL;
	4221	}
	4222
	4223	dev_priv->perf.oa.b_counter_regs =
	4224	b_counter_config_compute_extended;
	4225	dev_priv->perf.oa.b_counter_regs_len =
	4226	ARRAY_SIZE(b_counter_config_compute_extended);
	4227
	4228	dev_priv->perf.oa.flex_regs =
	4229	flex_eu_config_compute_extended;
	4230	dev_priv->perf.oa.flex_regs_len =
	4231	ARRAY_SIZE(flex_eu_config_compute_extended);
	4232
	4233	return 0;
	4234	case METRIC_SET_ID_COMPUTE_L3_CACHE:
	4235	dev_priv->perf.oa.n_mux_configs =
	4236	get_compute_l3_cache_mux_config(dev_priv,
	4237	dev_priv->perf.oa.mux_regs,
	4238	dev_priv->perf.oa.mux_regs_lens);
	4239	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4240	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_L3_CACHE\" metric set\n");
	4241
	4242	/* EINVAL because *_register_sysfs already checked this
	4243	* and so it wouldn't have been advertised to userspace and
	4244	* so shouldn't have been requested
	4245	*/
	4246	return -EINVAL;
	4247	}
	4248
	4249	dev_priv->perf.oa.b_counter_regs =
	4250	b_counter_config_compute_l3_cache;
	4251	dev_priv->perf.oa.b_counter_regs_len =
	4252	ARRAY_SIZE(b_counter_config_compute_l3_cache);
	4253
	4254	dev_priv->perf.oa.flex_regs =
	4255	flex_eu_config_compute_l3_cache;
	4256	dev_priv->perf.oa.flex_regs_len =
	4257	ARRAY_SIZE(flex_eu_config_compute_l3_cache);
	4258
	4259	return 0;
	4260	case METRIC_SET_ID_DATA_PORT_READS_COALESCING:
	4261	dev_priv->perf.oa.n_mux_configs =
	4262	get_data_port_reads_coalescing_mux_config(dev_priv,
	4263	dev_priv->perf.oa.mux_regs,
	4264	dev_priv->perf.oa.mux_regs_lens);
	4265	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4266	DRM_DEBUG_DRIVER("No suitable MUX config for \"DATA_PORT_READS_COALESCING\" metric set\n");
	4267
	4268	/* EINVAL because *_register_sysfs already checked this
	4269	* and so it wouldn't have been advertised to userspace and
	4270	* so shouldn't have been requested
	4271	*/
	4272	return -EINVAL;
	4273	}
	4274
	4275	dev_priv->perf.oa.b_counter_regs =
	4276	b_counter_config_data_port_reads_coalescing;
	4277	dev_priv->perf.oa.b_counter_regs_len =
	4278	ARRAY_SIZE(b_counter_config_data_port_reads_coalescing);
	4279
	4280	dev_priv->perf.oa.flex_regs =
	4281	flex_eu_config_data_port_reads_coalescing;
	4282	dev_priv->perf.oa.flex_regs_len =
	4283	ARRAY_SIZE(flex_eu_config_data_port_reads_coalescing);
	4284
	4285	return 0;
	4286	case METRIC_SET_ID_DATA_PORT_WRITES_COALESCING:
	4287	dev_priv->perf.oa.n_mux_configs =
	4288	get_data_port_writes_coalescing_mux_config(dev_priv,
	4289	dev_priv->perf.oa.mux_regs,
	4290	dev_priv->perf.oa.mux_regs_lens);
	4291	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4292	DRM_DEBUG_DRIVER("No suitable MUX config for \"DATA_PORT_WRITES_COALESCING\" metric set\n");
	4293
	4294	/* EINVAL because *_register_sysfs already checked this
	4295	* and so it wouldn't have been advertised to userspace and
	4296	* so shouldn't have been requested
	4297	*/
	4298	return -EINVAL;
	4299	}
	4300
	4301	dev_priv->perf.oa.b_counter_regs =
	4302	b_counter_config_data_port_writes_coalescing;
	4303	dev_priv->perf.oa.b_counter_regs_len =
	4304	ARRAY_SIZE(b_counter_config_data_port_writes_coalescing);
	4305
	4306	dev_priv->perf.oa.flex_regs =
	4307	flex_eu_config_data_port_writes_coalescing;
	4308	dev_priv->perf.oa.flex_regs_len =
	4309	ARRAY_SIZE(flex_eu_config_data_port_writes_coalescing);
	4310
	4311	return 0;
	4312	case METRIC_SET_ID_HDC_AND_SF:
	4313	dev_priv->perf.oa.n_mux_configs =
	4314	get_hdc_and_sf_mux_config(dev_priv,
	4315	dev_priv->perf.oa.mux_regs,
	4316	dev_priv->perf.oa.mux_regs_lens);
	4317	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4318	DRM_DEBUG_DRIVER("No suitable MUX config for \"HDC_AND_SF\" metric set\n");
	4319
	4320	/* EINVAL because *_register_sysfs already checked this
	4321	* and so it wouldn't have been advertised to userspace and
	4322	* so shouldn't have been requested
	4323	*/
	4324	return -EINVAL;
	4325	}
	4326
	4327	dev_priv->perf.oa.b_counter_regs =
	4328	b_counter_config_hdc_and_sf;
	4329	dev_priv->perf.oa.b_counter_regs_len =
	4330	ARRAY_SIZE(b_counter_config_hdc_and_sf);
	4331
	4332	dev_priv->perf.oa.flex_regs =
	4333	flex_eu_config_hdc_and_sf;
	4334	dev_priv->perf.oa.flex_regs_len =
	4335	ARRAY_SIZE(flex_eu_config_hdc_and_sf);
	4336
	4337	return 0;
	4338	case METRIC_SET_ID_L3_1:
	4339	dev_priv->perf.oa.n_mux_configs =
	4340	get_l3_1_mux_config(dev_priv,
	4341	dev_priv->perf.oa.mux_regs,
	4342	dev_priv->perf.oa.mux_regs_lens);
	4343	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4344	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_1\" metric set\n");
	4345
	4346	/* EINVAL because *_register_sysfs already checked this
	4347	* and so it wouldn't have been advertised to userspace and
	4348	* so shouldn't have been requested
	4349	*/
	4350	return -EINVAL;
	4351	}
	4352
	4353	dev_priv->perf.oa.b_counter_regs =
	4354	b_counter_config_l3_1;
	4355	dev_priv->perf.oa.b_counter_regs_len =
	4356	ARRAY_SIZE(b_counter_config_l3_1);
	4357
	4358	dev_priv->perf.oa.flex_regs =
	4359	flex_eu_config_l3_1;
	4360	dev_priv->perf.oa.flex_regs_len =
	4361	ARRAY_SIZE(flex_eu_config_l3_1);
	4362
	4363	return 0;
	4364	case METRIC_SET_ID_L3_2:
	4365	dev_priv->perf.oa.n_mux_configs =
	4366	get_l3_2_mux_config(dev_priv,
	4367	dev_priv->perf.oa.mux_regs,
	4368	dev_priv->perf.oa.mux_regs_lens);
	4369	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4370	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_2\" metric set\n");
	4371
	4372	/* EINVAL because *_register_sysfs already checked this
	4373	* and so it wouldn't have been advertised to userspace and
	4374	* so shouldn't have been requested
	4375	*/
	4376	return -EINVAL;
	4377	}
	4378
	4379	dev_priv->perf.oa.b_counter_regs =
	4380	b_counter_config_l3_2;
	4381	dev_priv->perf.oa.b_counter_regs_len =
	4382	ARRAY_SIZE(b_counter_config_l3_2);
	4383
	4384	dev_priv->perf.oa.flex_regs =
	4385	flex_eu_config_l3_2;
	4386	dev_priv->perf.oa.flex_regs_len =
	4387	ARRAY_SIZE(flex_eu_config_l3_2);
	4388
	4389	return 0;
	4390	case METRIC_SET_ID_L3_3:
	4391	dev_priv->perf.oa.n_mux_configs =
	4392	get_l3_3_mux_config(dev_priv,
	4393	dev_priv->perf.oa.mux_regs,
	4394	dev_priv->perf.oa.mux_regs_lens);
	4395	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4396	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_3\" metric set\n");
	4397
	4398	/* EINVAL because *_register_sysfs already checked this
	4399	* and so it wouldn't have been advertised to userspace and
	4400	* so shouldn't have been requested
	4401	*/
	4402	return -EINVAL;
	4403	}
	4404
	4405	dev_priv->perf.oa.b_counter_regs =
	4406	b_counter_config_l3_3;
	4407	dev_priv->perf.oa.b_counter_regs_len =
	4408	ARRAY_SIZE(b_counter_config_l3_3);
	4409
	4410	dev_priv->perf.oa.flex_regs =
	4411	flex_eu_config_l3_3;
	4412	dev_priv->perf.oa.flex_regs_len =
	4413	ARRAY_SIZE(flex_eu_config_l3_3);
	4414
	4415	return 0;
	4416	case METRIC_SET_ID_L3_4:
	4417	dev_priv->perf.oa.n_mux_configs =
	4418	get_l3_4_mux_config(dev_priv,
	4419	dev_priv->perf.oa.mux_regs,
	4420	dev_priv->perf.oa.mux_regs_lens);
	4421	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4422	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_4\" metric set\n");
	4423
	4424	/* EINVAL because *_register_sysfs already checked this
	4425	* and so it wouldn't have been advertised to userspace and
	4426	* so shouldn't have been requested
	4427	*/
	4428	return -EINVAL;
	4429	}
	4430
	4431	dev_priv->perf.oa.b_counter_regs =
	4432	b_counter_config_l3_4;
	4433	dev_priv->perf.oa.b_counter_regs_len =
	4434	ARRAY_SIZE(b_counter_config_l3_4);
	4435
	4436	dev_priv->perf.oa.flex_regs =
	4437	flex_eu_config_l3_4;
	4438	dev_priv->perf.oa.flex_regs_len =
	4439	ARRAY_SIZE(flex_eu_config_l3_4);
	4440
	4441	return 0;
	4442	case METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND:
	4443	dev_priv->perf.oa.n_mux_configs =
	4444	get_rasterizer_and_pixel_backend_mux_config(dev_priv,
	4445	dev_priv->perf.oa.mux_regs,
	4446	dev_priv->perf.oa.mux_regs_lens);
	4447	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4448	DRM_DEBUG_DRIVER("No suitable MUX config for \"RASTERIZER_AND_PIXEL_BACKEND\" metric set\n");
	4449
	4450	/* EINVAL because *_register_sysfs already checked this
	4451	* and so it wouldn't have been advertised to userspace and
	4452	* so shouldn't have been requested
	4453	*/
	4454	return -EINVAL;
	4455	}
	4456
	4457	dev_priv->perf.oa.b_counter_regs =
	4458	b_counter_config_rasterizer_and_pixel_backend;
	4459	dev_priv->perf.oa.b_counter_regs_len =
	4460	ARRAY_SIZE(b_counter_config_rasterizer_and_pixel_backend);
	4461
	4462	dev_priv->perf.oa.flex_regs =
	4463	flex_eu_config_rasterizer_and_pixel_backend;
	4464	dev_priv->perf.oa.flex_regs_len =
	4465	ARRAY_SIZE(flex_eu_config_rasterizer_and_pixel_backend);
	4466
	4467	return 0;
	4468	case METRIC_SET_ID_SAMPLER_1:
	4469	dev_priv->perf.oa.n_mux_configs =
	4470	get_sampler_1_mux_config(dev_priv,
	4471	dev_priv->perf.oa.mux_regs,
	4472	dev_priv->perf.oa.mux_regs_lens);
	4473	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4474	DRM_DEBUG_DRIVER("No suitable MUX config for \"SAMPLER_1\" metric set\n");
	4475
	4476	/* EINVAL because *_register_sysfs already checked this
	4477	* and so it wouldn't have been advertised to userspace and
	4478	* so shouldn't have been requested
	4479	*/
	4480	return -EINVAL;
	4481	}
	4482
	4483	dev_priv->perf.oa.b_counter_regs =
	4484	b_counter_config_sampler_1;
	4485	dev_priv->perf.oa.b_counter_regs_len =
	4486	ARRAY_SIZE(b_counter_config_sampler_1);
	4487
	4488	dev_priv->perf.oa.flex_regs =
	4489	flex_eu_config_sampler_1;
	4490	dev_priv->perf.oa.flex_regs_len =
	4491	ARRAY_SIZE(flex_eu_config_sampler_1);
	4492
	4493	return 0;
	4494	case METRIC_SET_ID_SAMPLER_2:
	4495	dev_priv->perf.oa.n_mux_configs =
	4496	get_sampler_2_mux_config(dev_priv,
	4497	dev_priv->perf.oa.mux_regs,
	4498	dev_priv->perf.oa.mux_regs_lens);
	4499	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4500	DRM_DEBUG_DRIVER("No suitable MUX config for \"SAMPLER_2\" metric set\n");
	4501
	4502	/* EINVAL because *_register_sysfs already checked this
	4503	* and so it wouldn't have been advertised to userspace and
	4504	* so shouldn't have been requested
	4505	*/
	4506	return -EINVAL;
	4507	}
	4508
	4509	dev_priv->perf.oa.b_counter_regs =
	4510	b_counter_config_sampler_2;
	4511	dev_priv->perf.oa.b_counter_regs_len =
	4512	ARRAY_SIZE(b_counter_config_sampler_2);
	4513
	4514	dev_priv->perf.oa.flex_regs =
	4515	flex_eu_config_sampler_2;
	4516	dev_priv->perf.oa.flex_regs_len =
	4517	ARRAY_SIZE(flex_eu_config_sampler_2);
	4518
	4519	return 0;
	4520	case METRIC_SET_ID_TDL_1:
	4521	dev_priv->perf.oa.n_mux_configs =
	4522	get_tdl_1_mux_config(dev_priv,
	4523	dev_priv->perf.oa.mux_regs,
	4524	dev_priv->perf.oa.mux_regs_lens);
	4525	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4526	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_1\" metric set\n");
	4527
	4528	/* EINVAL because *_register_sysfs already checked this
	4529	* and so it wouldn't have been advertised to userspace and
	4530	* so shouldn't have been requested
	4531	*/
	4532	return -EINVAL;
	4533	}
	4534
	4535	dev_priv->perf.oa.b_counter_regs =
	4536	b_counter_config_tdl_1;
	4537	dev_priv->perf.oa.b_counter_regs_len =
	4538	ARRAY_SIZE(b_counter_config_tdl_1);
	4539
	4540	dev_priv->perf.oa.flex_regs =
	4541	flex_eu_config_tdl_1;
	4542	dev_priv->perf.oa.flex_regs_len =
	4543	ARRAY_SIZE(flex_eu_config_tdl_1);
	4544
	4545	return 0;
	4546	case METRIC_SET_ID_TDL_2:
	4547	dev_priv->perf.oa.n_mux_configs =
	4548	get_tdl_2_mux_config(dev_priv,
	4549	dev_priv->perf.oa.mux_regs,
	4550	dev_priv->perf.oa.mux_regs_lens);
	4551	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4552	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_2\" metric set\n");
	4553
	4554	/* EINVAL because *_register_sysfs already checked this
	4555	* and so it wouldn't have been advertised to userspace and
	4556	* so shouldn't have been requested
	4557	*/
	4558	return -EINVAL;
	4559	}
	4560
	4561	dev_priv->perf.oa.b_counter_regs =
	4562	b_counter_config_tdl_2;
	4563	dev_priv->perf.oa.b_counter_regs_len =
	4564	ARRAY_SIZE(b_counter_config_tdl_2);
	4565
	4566	dev_priv->perf.oa.flex_regs =
	4567	flex_eu_config_tdl_2;
	4568	dev_priv->perf.oa.flex_regs_len =
	4569	ARRAY_SIZE(flex_eu_config_tdl_2);
	4570
	4571	return 0;
	4572	case METRIC_SET_ID_COMPUTE_EXTRA:
	4573	dev_priv->perf.oa.n_mux_configs =
	4574	get_compute_extra_mux_config(dev_priv,
	4575	dev_priv->perf.oa.mux_regs,
	4576	dev_priv->perf.oa.mux_regs_lens);
	4577	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4578	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_EXTRA\" metric set\n");
	4579
	4580	/* EINVAL because *_register_sysfs already checked this
	4581	* and so it wouldn't have been advertised to userspace and
	4582	* so shouldn't have been requested
	4583	*/
	4584	return -EINVAL;
	4585	}
	4586
	4587	dev_priv->perf.oa.b_counter_regs =
	4588	b_counter_config_compute_extra;
	4589	dev_priv->perf.oa.b_counter_regs_len =
	4590	ARRAY_SIZE(b_counter_config_compute_extra);
	4591
	4592	dev_priv->perf.oa.flex_regs =
	4593	flex_eu_config_compute_extra;
	4594	dev_priv->perf.oa.flex_regs_len =
	4595	ARRAY_SIZE(flex_eu_config_compute_extra);
	4596
	4597	return 0;
	4598	case METRIC_SET_ID_VME_PIPE:
	4599	dev_priv->perf.oa.n_mux_configs =
	4600	get_vme_pipe_mux_config(dev_priv,
	4601	dev_priv->perf.oa.mux_regs,
	4602	dev_priv->perf.oa.mux_regs_lens);
	4603	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4604	DRM_DEBUG_DRIVER("No suitable MUX config for \"VME_PIPE\" metric set\n");
	4605
	4606	/* EINVAL because *_register_sysfs already checked this
	4607	* and so it wouldn't have been advertised to userspace and
	4608	* so shouldn't have been requested
	4609	*/
	4610	return -EINVAL;
	4611	}
	4612
	4613	dev_priv->perf.oa.b_counter_regs =
	4614	b_counter_config_vme_pipe;
	4615	dev_priv->perf.oa.b_counter_regs_len =
	4616	ARRAY_SIZE(b_counter_config_vme_pipe);
	4617
	4618	dev_priv->perf.oa.flex_regs =
	4619	flex_eu_config_vme_pipe;
	4620	dev_priv->perf.oa.flex_regs_len =
	4621	ARRAY_SIZE(flex_eu_config_vme_pipe);
	4622
	4623	return 0;
	4624	case METRIC_SET_ID_TEST_OA:
	4625	dev_priv->perf.oa.n_mux_configs =
	4626	get_test_oa_mux_config(dev_priv,
	4627	dev_priv->perf.oa.mux_regs,
	4628	dev_priv->perf.oa.mux_regs_lens);
	4629	if (dev_priv->perf.oa.n_mux_configs == 0) {
	4630	DRM_DEBUG_DRIVER("No suitable MUX config for \"TEST_OA\" metric set\n");
	4631
	4632	/* EINVAL because *_register_sysfs already checked this
	4633	* and so it wouldn't have been advertised to userspace and
	4634	* so shouldn't have been requested
	4635	*/
	4636	return -EINVAL;
	4637	}
	4638
	4639	dev_priv->perf.oa.b_counter_regs =
	4640	b_counter_config_test_oa;
	4641	dev_priv->perf.oa.b_counter_regs_len =
	4642	ARRAY_SIZE(b_counter_config_test_oa);
	4643
	4644	dev_priv->perf.oa.flex_regs =
	4645	flex_eu_config_test_oa;
	4646	dev_priv->perf.oa.flex_regs_len =
	4647	ARRAY_SIZE(flex_eu_config_test_oa);
	4648
	4649	return 0;
	4650	default:
	4651	return -ENODEV;
	4652	}
	4653	}
	4654
	4655	static ssize_t
	4656	show_render_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	4657	{
	4658	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_BASIC);
	4659	}
	4660
	4661	static struct device_attribute dev_attr_render_basic_id = {
	4662	.attr = { .name = "id", .mode = 0444 },
	4663	.show = show_render_basic_id,
	4664	.store = NULL,
	4665	};
	4666
	4667	static struct attribute *attrs_render_basic[] = {
	4668	&dev_attr_render_basic_id.attr,
	4669	NULL,
	4670	};
	4671
	4672	static struct attribute_group group_render_basic = {
	4673	.name = "b541bd57-0e0f-4154-b4c0-5858010a2bf7",
	4674	.attrs = attrs_render_basic,
	4675	};
	4676
	4677	static ssize_t
	4678	show_compute_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	4679	{
	4680	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_BASIC);
	4681	}
	4682
	4683	static struct device_attribute dev_attr_compute_basic_id = {
	4684	.attr = { .name = "id", .mode = 0444 },
	4685	.show = show_compute_basic_id,
	4686	.store = NULL,
	4687	};
	4688
	4689	static struct attribute *attrs_compute_basic[] = {
	4690	&dev_attr_compute_basic_id.attr,
	4691	NULL,
	4692	};
	4693
	4694	static struct attribute_group group_compute_basic = {
	4695	.name = "35fbc9b2-a891-40a6-a38d-022bb7057552",
	4696	.attrs = attrs_compute_basic,
	4697	};
	4698
	4699	static ssize_t
	4700	show_render_pipe_profile_id(struct device kdev, struct device_attribute attr, char *buf)
	4701	{
	4702	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_PIPE_PROFILE);
	4703	}
	4704
	4705	static struct device_attribute dev_attr_render_pipe_profile_id = {
	4706	.attr = { .name = "id", .mode = 0444 },
	4707	.show = show_render_pipe_profile_id,
	4708	.store = NULL,
	4709	};
	4710
	4711	static struct attribute *attrs_render_pipe_profile[] = {
	4712	&dev_attr_render_pipe_profile_id.attr,
	4713	NULL,
	4714	};
	4715
	4716	static struct attribute_group group_render_pipe_profile = {
	4717	.name = "233d0544-fff7-4281-8291-e02f222aff72",
	4718	.attrs = attrs_render_pipe_profile,
	4719	};
	4720
	4721	static ssize_t
	4722	show_memory_reads_id(struct device kdev, struct device_attribute attr, char *buf)
	4723	{
	4724	return sprintf(buf, "%d\n", METRIC_SET_ID_MEMORY_READS);
	4725	}
	4726
	4727	static struct device_attribute dev_attr_memory_reads_id = {
	4728	.attr = { .name = "id", .mode = 0444 },
	4729	.show = show_memory_reads_id,
	4730	.store = NULL,
	4731	};
	4732
	4733	static struct attribute *attrs_memory_reads[] = {
	4734	&dev_attr_memory_reads_id.attr,
	4735	NULL,
	4736	};
	4737
	4738	static struct attribute_group group_memory_reads = {
	4739	.name = "2b255d48-2117-4fef-a8f7-f151e1d25a2c",
	4740	.attrs = attrs_memory_reads,
	4741	};
	4742
	4743	static ssize_t
	4744	show_memory_writes_id(struct device kdev, struct device_attribute attr, char *buf)
	4745	{
	4746	return sprintf(buf, "%d\n", METRIC_SET_ID_MEMORY_WRITES);
	4747	}
	4748
	4749	static struct device_attribute dev_attr_memory_writes_id = {
	4750	.attr = { .name = "id", .mode = 0444 },
	4751	.show = show_memory_writes_id,
	4752	.store = NULL,
	4753	};
	4754
	4755	static struct attribute *attrs_memory_writes[] = {
	4756	&dev_attr_memory_writes_id.attr,
	4757	NULL,
	4758	};
	4759
	4760	static struct attribute_group group_memory_writes = {
	4761	.name = "f7fd3220-b466-4a4d-9f98-b0caf3f2394c",
	4762	.attrs = attrs_memory_writes,
	4763	};
	4764
	4765	static ssize_t
	4766	show_compute_extended_id(struct device kdev, struct device_attribute attr, char *buf)
	4767	{
	4768	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_EXTENDED);
	4769	}
	4770
	4771	static struct device_attribute dev_attr_compute_extended_id = {
	4772	.attr = { .name = "id", .mode = 0444 },
	4773	.show = show_compute_extended_id,
	4774	.store = NULL,
	4775	};
	4776
	4777	static struct attribute *attrs_compute_extended[] = {
	4778	&dev_attr_compute_extended_id.attr,
	4779	NULL,
	4780	};
	4781
	4782	static struct attribute_group group_compute_extended = {
	4783	.name = "e99ccaca-821c-4df9-97a7-96bdb7204e43",
	4784	.attrs = attrs_compute_extended,
	4785	};
	4786
	4787	static ssize_t
	4788	show_compute_l3_cache_id(struct device kdev, struct device_attribute attr, char *buf)
	4789	{
	4790	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_L3_CACHE);
	4791	}
	4792
	4793	static struct device_attribute dev_attr_compute_l3_cache_id = {
	4794	.attr = { .name = "id", .mode = 0444 },
	4795	.show = show_compute_l3_cache_id,
	4796	.store = NULL,
	4797	};
	4798
	4799	static struct attribute *attrs_compute_l3_cache[] = {
	4800	&dev_attr_compute_l3_cache_id.attr,
	4801	NULL,
	4802	};
	4803
	4804	static struct attribute_group group_compute_l3_cache = {
	4805	.name = "27a364dc-8225-4ecb-b607-d6f1925598d9",
	4806	.attrs = attrs_compute_l3_cache,
	4807	};
	4808
	4809	static ssize_t
	4810	show_data_port_reads_coalescing_id(struct device kdev, struct device_attribute attr, char *buf)
	4811	{
	4812	return sprintf(buf, "%d\n", METRIC_SET_ID_DATA_PORT_READS_COALESCING);
	4813	}
	4814
	4815	static struct device_attribute dev_attr_data_port_reads_coalescing_id = {
	4816	.attr = { .name = "id", .mode = 0444 },
	4817	.show = show_data_port_reads_coalescing_id,
	4818	.store = NULL,
	4819	};
	4820
	4821	static struct attribute *attrs_data_port_reads_coalescing[] = {
	4822	&dev_attr_data_port_reads_coalescing_id.attr,
	4823	NULL,
	4824	};
	4825
	4826	static struct attribute_group group_data_port_reads_coalescing = {
	4827	.name = "857fc630-2f09-4804-85f1-084adfadd5ab",
	4828	.attrs = attrs_data_port_reads_coalescing,
	4829	};
	4830
	4831	static ssize_t
	4832	show_data_port_writes_coalescing_id(struct device kdev, struct device_attribute attr, char *buf)
	4833	{
	4834	return sprintf(buf, "%d\n", METRIC_SET_ID_DATA_PORT_WRITES_COALESCING);
	4835	}
	4836
	4837	static struct device_attribute dev_attr_data_port_writes_coalescing_id = {
	4838	.attr = { .name = "id", .mode = 0444 },
	4839	.show = show_data_port_writes_coalescing_id,
	4840	.store = NULL,
	4841	};
	4842
	4843	static struct attribute *attrs_data_port_writes_coalescing[] = {
	4844	&dev_attr_data_port_writes_coalescing_id.attr,
	4845	NULL,
	4846	};
	4847
	4848	static struct attribute_group group_data_port_writes_coalescing = {
	4849	.name = "343ebc99-4a55-414c-8c17-d8e259cf5e20",
	4850	.attrs = attrs_data_port_writes_coalescing,
	4851	};
	4852
	4853	static ssize_t
	4854	show_hdc_and_sf_id(struct device kdev, struct device_attribute attr, char *buf)
	4855	{
	4856	return sprintf(buf, "%d\n", METRIC_SET_ID_HDC_AND_SF);
	4857	}
	4858
	4859	static struct device_attribute dev_attr_hdc_and_sf_id = {
	4860	.attr = { .name = "id", .mode = 0444 },
	4861	.show = show_hdc_and_sf_id,
	4862	.store = NULL,
	4863	};
	4864
	4865	static struct attribute *attrs_hdc_and_sf[] = {
	4866	&dev_attr_hdc_and_sf_id.attr,
	4867	NULL,
	4868	};
	4869
	4870	static struct attribute_group group_hdc_and_sf = {
	4871	.name = "7bdafd88-a4fa-4ed5-bc09-1a977aa5be3e",
	4872	.attrs = attrs_hdc_and_sf,
	4873	};
	4874
	4875	static ssize_t
	4876	show_l3_1_id(struct device kdev, struct device_attribute attr, char *buf)
	4877	{
	4878	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_1);
	4879	}
	4880
	4881	static struct device_attribute dev_attr_l3_1_id = {
	4882	.attr = { .name = "id", .mode = 0444 },
	4883	.show = show_l3_1_id,
	4884	.store = NULL,
	4885	};
	4886
	4887	static struct attribute *attrs_l3_1[] = {
	4888	&dev_attr_l3_1_id.attr,
	4889	NULL,
	4890	};
	4891
	4892	static struct attribute_group group_l3_1 = {
	4893	.name = "9385ebb2-f34f-4aa5-aec5-7e9cbbea0f0b",
	4894	.attrs = attrs_l3_1,
	4895	};
	4896
	4897	static ssize_t
	4898	show_l3_2_id(struct device kdev, struct device_attribute attr, char *buf)
	4899	{
	4900	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_2);
	4901	}
	4902
	4903	static struct device_attribute dev_attr_l3_2_id = {
	4904	.attr = { .name = "id", .mode = 0444 },
	4905	.show = show_l3_2_id,
	4906	.store = NULL,
	4907	};
	4908
	4909	static struct attribute *attrs_l3_2[] = {
	4910	&dev_attr_l3_2_id.attr,
	4911	NULL,
	4912	};
	4913
	4914	static struct attribute_group group_l3_2 = {
	4915	.name = "446ae59b-ff2e-41c9-b49e-0184a54bf00a",
	4916	.attrs = attrs_l3_2,
	4917	};
	4918
	4919	static ssize_t
	4920	show_l3_3_id(struct device kdev, struct device_attribute attr, char *buf)
	4921	{
	4922	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_3);
	4923	}
	4924
	4925	static struct device_attribute dev_attr_l3_3_id = {
	4926	.attr = { .name = "id", .mode = 0444 },
	4927	.show = show_l3_3_id,
	4928	.store = NULL,
	4929	};
	4930
	4931	static struct attribute *attrs_l3_3[] = {
	4932	&dev_attr_l3_3_id.attr,
	4933	NULL,
	4934	};
	4935
	4936	static struct attribute_group group_l3_3 = {
	4937	.name = "84a7956f-1ea4-4d0d-837f-e39a0376e38c",
	4938	.attrs = attrs_l3_3,
	4939	};
	4940
	4941	static ssize_t
	4942	show_l3_4_id(struct device kdev, struct device_attribute attr, char *buf)
	4943	{
	4944	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_4);
	4945	}
	4946
	4947	static struct device_attribute dev_attr_l3_4_id = {
	4948	.attr = { .name = "id", .mode = 0444 },
	4949	.show = show_l3_4_id,
	4950	.store = NULL,
	4951	};
	4952
	4953	static struct attribute *attrs_l3_4[] = {
	4954	&dev_attr_l3_4_id.attr,
	4955	NULL,
	4956	};
	4957
	4958	static struct attribute_group group_l3_4 = {
	4959	.name = "92b493d9-df18-4bed-be06-5cac6f2a6f5f",
	4960	.attrs = attrs_l3_4,
	4961	};
	4962
	4963	static ssize_t
	4964	show_rasterizer_and_pixel_backend_id(struct device kdev, struct device_attribute attr, char *buf)
	4965	{
	4966	return sprintf(buf, "%d\n", METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND);
	4967	}
	4968
	4969	static struct device_attribute dev_attr_rasterizer_and_pixel_backend_id = {
	4970	.attr = { .name = "id", .mode = 0444 },
	4971	.show = show_rasterizer_and_pixel_backend_id,
	4972	.store = NULL,
	4973	};
	4974
	4975	static struct attribute *attrs_rasterizer_and_pixel_backend[] = {
	4976	&dev_attr_rasterizer_and_pixel_backend_id.attr,
	4977	NULL,
	4978	};
	4979
	4980	static struct attribute_group group_rasterizer_and_pixel_backend = {
	4981	.name = "14345c35-cc46-40d0-bb04-6ed1fbb43679",
	4982	.attrs = attrs_rasterizer_and_pixel_backend,
	4983	};
	4984
	4985	static ssize_t
	4986	show_sampler_1_id(struct device kdev, struct device_attribute attr, char *buf)
	4987	{
	4988	return sprintf(buf, "%d\n", METRIC_SET_ID_SAMPLER_1);
	4989	}
	4990
	4991	static struct device_attribute dev_attr_sampler_1_id = {
	4992	.attr = { .name = "id", .mode = 0444 },
	4993	.show = show_sampler_1_id,
	4994	.store = NULL,
	4995	};
	4996
	4997	static struct attribute *attrs_sampler_1[] = {
	4998	&dev_attr_sampler_1_id.attr,
	4999	NULL,

377 more lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_bdw.h+40−0

@@ -0,0 +1,40 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#ifndef __I915_OA_BDW_H__
	30	#define __I915_OA_BDW_H__
	31
	32	extern int i915_oa_n_builtin_metric_sets_bdw;
	33
	34	extern int i915_oa_select_metric_set_bdw(struct drm_i915_private *dev_priv);
	35
	36	extern int i915_perf_register_sysfs_bdw(struct drm_i915_private *dev_priv);
	37
	38	extern void i915_perf_unregister_sysfs_bdw(struct drm_i915_private *dev_priv);
	39
	40	#endif

drivers/gpu/drm/i915/i915_oa_bxt.c+2690−0

@@ -0,0 +1,2690 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#include <linux/sysfs.h>
	30
	31	#include "i915_drv.h"
	32	#include "i915_oa_bxt.h"
	33
	34	enum metric_set_id {
	35	METRIC_SET_ID_RENDER_BASIC = 1,
	36	METRIC_SET_ID_COMPUTE_BASIC,
	37	METRIC_SET_ID_RENDER_PIPE_PROFILE,
	38	METRIC_SET_ID_MEMORY_READS,
	39	METRIC_SET_ID_MEMORY_WRITES,
	40	METRIC_SET_ID_COMPUTE_EXTENDED,
	41	METRIC_SET_ID_COMPUTE_L3_CACHE,
	42	METRIC_SET_ID_HDC_AND_SF,
	43	METRIC_SET_ID_L3_1,
	44	METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND,
	45	METRIC_SET_ID_SAMPLER,
	46	METRIC_SET_ID_TDL_1,
	47	METRIC_SET_ID_TDL_2,
	48	METRIC_SET_ID_COMPUTE_EXTRA,
	49	METRIC_SET_ID_TEST_OA,
	50	};
	51
	52	int i915_oa_n_builtin_metric_sets_bxt = 15;
	53
	54	static const struct i915_oa_reg b_counter_config_render_basic[] = {
	55	{ _MMIO(0x2710), 0x00000000 },
	56	{ _MMIO(0x2714), 0x00800000 },
	57	{ _MMIO(0x2720), 0x00000000 },
	58	{ _MMIO(0x2724), 0x00800000 },
	59	{ _MMIO(0x2740), 0x00000000 },
	60	};
	61
	62	static const struct i915_oa_reg flex_eu_config_render_basic[] = {
	63	{ _MMIO(0xe458), 0x00005004 },
	64	{ _MMIO(0xe558), 0x00010003 },
	65	{ _MMIO(0xe658), 0x00012011 },
	66	{ _MMIO(0xe758), 0x00015014 },
	67	{ _MMIO(0xe45c), 0x00051050 },
	68	{ _MMIO(0xe55c), 0x00053052 },
	69	{ _MMIO(0xe65c), 0x00055054 },
	70	};
	71
	72	static const struct i915_oa_reg mux_config_render_basic_0_sku_gte_0x03[] = {
	73	{ _MMIO(0x9888), 0x166c00f0 },
	74	{ _MMIO(0x9888), 0x12120280 },
	75	{ _MMIO(0x9888), 0x12320280 },
	76	{ _MMIO(0x9888), 0x11930317 },
	77	{ _MMIO(0x9888), 0x159303df },
	78	{ _MMIO(0x9888), 0x3f900c00 },
	79	{ _MMIO(0x9888), 0x419000a0 },
	80	{ _MMIO(0x9888), 0x002d1000 },
	81	{ _MMIO(0x9888), 0x062d4000 },
	82	{ _MMIO(0x9888), 0x082d5000 },
	83	{ _MMIO(0x9888), 0x0a2d1000 },
	84	{ _MMIO(0x9888), 0x0c2e0800 },
	85	{ _MMIO(0x9888), 0x0e2e5900 },
	86	{ _MMIO(0x9888), 0x0a4c8000 },
	87	{ _MMIO(0x9888), 0x0c4c8000 },
	88	{ _MMIO(0x9888), 0x0e4c4000 },
	89	{ _MMIO(0x9888), 0x064e8000 },
	90	{ _MMIO(0x9888), 0x084e8000 },
	91	{ _MMIO(0x9888), 0x0a4e2000 },
	92	{ _MMIO(0x9888), 0x1c4f0010 },
	93	{ _MMIO(0x9888), 0x0a6c0053 },
	94	{ _MMIO(0x9888), 0x106c0000 },
	95	{ _MMIO(0x9888), 0x1c6c0000 },
	96	{ _MMIO(0x9888), 0x1a0fcc00 },
	97	{ _MMIO(0x9888), 0x1c0f0002 },
	98	{ _MMIO(0x9888), 0x1c2c0040 },
	99	{ _MMIO(0x9888), 0x00101000 },
	100	{ _MMIO(0x9888), 0x04101000 },
	101	{ _MMIO(0x9888), 0x00114000 },
	102	{ _MMIO(0x9888), 0x08114000 },
	103	{ _MMIO(0x9888), 0x00120020 },
	104	{ _MMIO(0x9888), 0x08120021 },
	105	{ _MMIO(0x9888), 0x00141000 },
	106	{ _MMIO(0x9888), 0x08141000 },
	107	{ _MMIO(0x9888), 0x02308000 },
	108	{ _MMIO(0x9888), 0x04302000 },
	109	{ _MMIO(0x9888), 0x06318000 },
	110	{ _MMIO(0x9888), 0x08318000 },
	111	{ _MMIO(0x9888), 0x06320800 },
	112	{ _MMIO(0x9888), 0x08320840 },
	113	{ _MMIO(0x9888), 0x00320000 },
	114	{ _MMIO(0x9888), 0x06344000 },
	115	{ _MMIO(0x9888), 0x08344000 },
	116	{ _MMIO(0x9888), 0x0d931831 },
	117	{ _MMIO(0x9888), 0x0f939f3f },
	118	{ _MMIO(0x9888), 0x01939e80 },
	119	{ _MMIO(0x9888), 0x039303bc },
	120	{ _MMIO(0x9888), 0x0593000e },
	121	{ _MMIO(0x9888), 0x1993002a },
	122	{ _MMIO(0x9888), 0x07930000 },
	123	{ _MMIO(0x9888), 0x09930000 },
	124	{ _MMIO(0x9888), 0x1d900177 },
	125	{ _MMIO(0x9888), 0x1f900187 },
	126	{ _MMIO(0x9888), 0x35900000 },
	127	{ _MMIO(0x9888), 0x13904000 },
	128	{ _MMIO(0x9888), 0x21904000 },
	129	{ _MMIO(0x9888), 0x23904000 },
	130	{ _MMIO(0x9888), 0x25904000 },
	131	{ _MMIO(0x9888), 0x27904000 },
	132	{ _MMIO(0x9888), 0x2b904000 },
	133	{ _MMIO(0x9888), 0x2d904000 },
	134	{ _MMIO(0x9888), 0x2f904000 },
	135	{ _MMIO(0x9888), 0x31904000 },
	136	{ _MMIO(0x9888), 0x15904000 },
	137	{ _MMIO(0x9888), 0x17904000 },
	138	{ _MMIO(0x9888), 0x19904000 },
	139	{ _MMIO(0x9888), 0x1b904000 },
	140	{ _MMIO(0x9888), 0x53901110 },
	141	{ _MMIO(0x9888), 0x43900423 },
	142	{ _MMIO(0x9888), 0x55900111 },
	143	{ _MMIO(0x9888), 0x47900c02 },
	144	{ _MMIO(0x9888), 0x57900000 },
	145	{ _MMIO(0x9888), 0x49900020 },
	146	{ _MMIO(0x9888), 0x59901111 },
	147	{ _MMIO(0x9888), 0x4b900421 },
	148	{ _MMIO(0x9888), 0x37900000 },
	149	{ _MMIO(0x9888), 0x33900000 },
	150	{ _MMIO(0x9888), 0x4d900001 },
	151	{ _MMIO(0x9888), 0x45900821 },
	152	};
	153
	154	static int
	155	get_render_basic_mux_config(struct drm_i915_private *dev_priv,
	156	const struct i915_oa_reg **regs,
	157	int *lens)
	158	{
	159	int n = 0;
	160
	161	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	162	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	163
	164	if (dev_priv->drm.pdev->revision >= 0x03) {
	165	regs[n] = mux_config_render_basic_0_sku_gte_0x03;
	166	lens[n] = ARRAY_SIZE(mux_config_render_basic_0_sku_gte_0x03);
	167	n++;
	168	}
	169
	170	return n;
	171	}
	172
	173	static const struct i915_oa_reg b_counter_config_compute_basic[] = {
	174	{ _MMIO(0x2710), 0x00000000 },
	175	{ _MMIO(0x2714), 0x00800000 },
	176	{ _MMIO(0x2720), 0x00000000 },
	177	{ _MMIO(0x2724), 0x00800000 },
	178	{ _MMIO(0x2740), 0x00000000 },
	179	};
	180
	181	static const struct i915_oa_reg flex_eu_config_compute_basic[] = {
	182	{ _MMIO(0xe458), 0x00005004 },
	183	{ _MMIO(0xe558), 0x00000003 },
	184	{ _MMIO(0xe658), 0x00002001 },
	185	{ _MMIO(0xe758), 0x00778008 },
	186	{ _MMIO(0xe45c), 0x00088078 },
	187	{ _MMIO(0xe55c), 0x00808708 },
	188	{ _MMIO(0xe65c), 0x00a08908 },
	189	};
	190
	191	static const struct i915_oa_reg mux_config_compute_basic[] = {
	192	{ _MMIO(0x9888), 0x104f00e0 },
	193	{ _MMIO(0x9888), 0x124f1c00 },
	194	{ _MMIO(0x9888), 0x39900340 },
	195	{ _MMIO(0x9888), 0x3f900c00 },
	196	{ _MMIO(0x9888), 0x41900000 },
	197	{ _MMIO(0x9888), 0x002d5000 },
	198	{ _MMIO(0x9888), 0x062d4000 },
	199	{ _MMIO(0x9888), 0x082d4000 },
	200	{ _MMIO(0x9888), 0x0a2d1000 },
	201	{ _MMIO(0x9888), 0x0c2d5000 },
	202	{ _MMIO(0x9888), 0x0e2d4000 },
	203	{ _MMIO(0x9888), 0x0c2e1400 },
	204	{ _MMIO(0x9888), 0x0e2e5100 },
	205	{ _MMIO(0x9888), 0x102e0114 },
	206	{ _MMIO(0x9888), 0x044cc000 },
	207	{ _MMIO(0x9888), 0x0a4c8000 },
	208	{ _MMIO(0x9888), 0x0c4c8000 },
	209	{ _MMIO(0x9888), 0x0e4c4000 },
	210	{ _MMIO(0x9888), 0x104c8000 },
	211	{ _MMIO(0x9888), 0x124c8000 },
	212	{ _MMIO(0x9888), 0x164c2000 },
	213	{ _MMIO(0x9888), 0x004ea000 },
	214	{ _MMIO(0x9888), 0x064e8000 },
	215	{ _MMIO(0x9888), 0x084e8000 },
	216	{ _MMIO(0x9888), 0x0a4e2000 },
	217	{ _MMIO(0x9888), 0x0c4ea000 },
	218	{ _MMIO(0x9888), 0x0e4e8000 },
	219	{ _MMIO(0x9888), 0x004f6b42 },
	220	{ _MMIO(0x9888), 0x064f6200 },
	221	{ _MMIO(0x9888), 0x084f4100 },
	222	{ _MMIO(0x9888), 0x0a4f0061 },
	223	{ _MMIO(0x9888), 0x0c4f6c4c },
	224	{ _MMIO(0x9888), 0x0e4f4b00 },
	225	{ _MMIO(0x9888), 0x1a4f0000 },
	226	{ _MMIO(0x9888), 0x1c4f0000 },
	227	{ _MMIO(0x9888), 0x180f5000 },
	228	{ _MMIO(0x9888), 0x1a0f8800 },
	229	{ _MMIO(0x9888), 0x1c0f08a2 },
	230	{ _MMIO(0x9888), 0x182c4000 },
	231	{ _MMIO(0x9888), 0x1c2c1451 },
	232	{ _MMIO(0x9888), 0x1e2c0001 },
	233	{ _MMIO(0x9888), 0x1a2c0010 },
	234	{ _MMIO(0x9888), 0x01938000 },
	235	{ _MMIO(0x9888), 0x0f938000 },
	236	{ _MMIO(0x9888), 0x19938a28 },
	237	{ _MMIO(0x9888), 0x03938000 },
	238	{ _MMIO(0x9888), 0x19900177 },
	239	{ _MMIO(0x9888), 0x1b900178 },
	240	{ _MMIO(0x9888), 0x1d900125 },
	241	{ _MMIO(0x9888), 0x1f900123 },
	242	{ _MMIO(0x9888), 0x35900000 },
	243	{ _MMIO(0x9888), 0x13904000 },
	244	{ _MMIO(0x9888), 0x21904000 },
	245	{ _MMIO(0x9888), 0x25904000 },
	246	{ _MMIO(0x9888), 0x27904000 },
	247	{ _MMIO(0x9888), 0x2b904000 },
	248	{ _MMIO(0x9888), 0x2d904000 },
	249	{ _MMIO(0x9888), 0x31904000 },
	250	{ _MMIO(0x9888), 0x15904000 },
	251	{ _MMIO(0x9888), 0x53901000 },
	252	{ _MMIO(0x9888), 0x43900000 },
	253	{ _MMIO(0x9888), 0x55900111 },
	254	{ _MMIO(0x9888), 0x47900000 },
	255	{ _MMIO(0x9888), 0x57900000 },
	256	{ _MMIO(0x9888), 0x49900000 },
	257	{ _MMIO(0x9888), 0x59900000 },
	258	{ _MMIO(0x9888), 0x4b900000 },
	259	{ _MMIO(0x9888), 0x37900000 },
	260	{ _MMIO(0x9888), 0x33900000 },
	261	{ _MMIO(0x9888), 0x4d900000 },
	262	{ _MMIO(0x9888), 0x45900000 },
	263	};
	264
	265	static int
	266	get_compute_basic_mux_config(struct drm_i915_private *dev_priv,
	267	const struct i915_oa_reg **regs,
	268	int *lens)
	269	{
	270	int n = 0;
	271
	272	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	273	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	274
	275	regs[n] = mux_config_compute_basic;
	276	lens[n] = ARRAY_SIZE(mux_config_compute_basic);
	277	n++;
	278
	279	return n;
	280	}
	281
	282	static const struct i915_oa_reg b_counter_config_render_pipe_profile[] = {
	283	{ _MMIO(0x2724), 0xf0800000 },
	284	{ _MMIO(0x2720), 0x00000000 },
	285	{ _MMIO(0x2714), 0xf0800000 },
	286	{ _MMIO(0x2710), 0x00000000 },
	287	{ _MMIO(0x2740), 0x00000000 },
	288	{ _MMIO(0x2770), 0x0007ffea },
	289	{ _MMIO(0x2774), 0x00007ffc },
	290	{ _MMIO(0x2778), 0x0007affa },
	291	{ _MMIO(0x277c), 0x0000f5fd },
	292	{ _MMIO(0x2780), 0x00079ffa },
	293	{ _MMIO(0x2784), 0x0000f3fb },
	294	{ _MMIO(0x2788), 0x0007bf7a },
	295	{ _MMIO(0x278c), 0x0000f7e7 },
	296	{ _MMIO(0x2790), 0x0007fefa },
	297	{ _MMIO(0x2794), 0x0000f7cf },
	298	{ _MMIO(0x2798), 0x00077ffa },
	299	{ _MMIO(0x279c), 0x0000efdf },
	300	{ _MMIO(0x27a0), 0x0006fffa },
	301	{ _MMIO(0x27a4), 0x0000cfbf },
	302	{ _MMIO(0x27a8), 0x0003fffa },
	303	{ _MMIO(0x27ac), 0x00005f7f },
	304	};
	305
	306	static const struct i915_oa_reg flex_eu_config_render_pipe_profile[] = {
	307	{ _MMIO(0xe458), 0x00005004 },
	308	{ _MMIO(0xe558), 0x00015014 },
	309	{ _MMIO(0xe658), 0x00025024 },
	310	{ _MMIO(0xe758), 0x00035034 },
	311	{ _MMIO(0xe45c), 0x00045044 },
	312	{ _MMIO(0xe55c), 0x00055054 },
	313	{ _MMIO(0xe65c), 0x00065064 },
	314	};
	315
	316	static const struct i915_oa_reg mux_config_render_pipe_profile[] = {
	317	{ _MMIO(0x9888), 0x0c2e001f },
	318	{ _MMIO(0x9888), 0x0a2f0000 },
	319	{ _MMIO(0x9888), 0x10186800 },
	320	{ _MMIO(0x9888), 0x11810019 },
	321	{ _MMIO(0x9888), 0x15810013 },
	322	{ _MMIO(0x9888), 0x13820020 },
	323	{ _MMIO(0x9888), 0x11830020 },
	324	{ _MMIO(0x9888), 0x17840000 },
	325	{ _MMIO(0x9888), 0x11860007 },
	326	{ _MMIO(0x9888), 0x21860000 },
	327	{ _MMIO(0x9888), 0x178703e0 },
	328	{ _MMIO(0x9888), 0x0c2d8000 },
	329	{ _MMIO(0x9888), 0x042d4000 },
	330	{ _MMIO(0x9888), 0x062d1000 },
	331	{ _MMIO(0x9888), 0x022e5400 },
	332	{ _MMIO(0x9888), 0x002e0000 },
	333	{ _MMIO(0x9888), 0x0e2e0080 },
	334	{ _MMIO(0x9888), 0x082f0040 },
	335	{ _MMIO(0x9888), 0x002f0000 },
	336	{ _MMIO(0x9888), 0x06143000 },
	337	{ _MMIO(0x9888), 0x06174000 },
	338	{ _MMIO(0x9888), 0x06180012 },
	339	{ _MMIO(0x9888), 0x00180000 },
	340	{ _MMIO(0x9888), 0x0d804000 },
	341	{ _MMIO(0x9888), 0x0f804000 },
	342	{ _MMIO(0x9888), 0x05804000 },
	343	{ _MMIO(0x9888), 0x09810200 },
	344	{ _MMIO(0x9888), 0x0b810030 },
	345	{ _MMIO(0x9888), 0x03810003 },
	346	{ _MMIO(0x9888), 0x21819140 },
	347	{ _MMIO(0x9888), 0x23819050 },
	348	{ _MMIO(0x9888), 0x25810018 },
	349	{ _MMIO(0x9888), 0x0b820980 },
	350	{ _MMIO(0x9888), 0x03820d80 },
	351	{ _MMIO(0x9888), 0x11820000 },
	352	{ _MMIO(0x9888), 0x0182c000 },
	353	{ _MMIO(0x9888), 0x07828000 },
	354	{ _MMIO(0x9888), 0x09824000 },
	355	{ _MMIO(0x9888), 0x0f828000 },
	356	{ _MMIO(0x9888), 0x0d830004 },
	357	{ _MMIO(0x9888), 0x0583000c },
	358	{ _MMIO(0x9888), 0x0f831000 },
	359	{ _MMIO(0x9888), 0x01848072 },
	360	{ _MMIO(0x9888), 0x11840000 },
	361	{ _MMIO(0x9888), 0x07848000 },
	362	{ _MMIO(0x9888), 0x09844000 },
	363	{ _MMIO(0x9888), 0x0f848000 },
	364	{ _MMIO(0x9888), 0x07860000 },
	365	{ _MMIO(0x9888), 0x09860092 },
	366	{ _MMIO(0x9888), 0x0f860400 },
	367	{ _MMIO(0x9888), 0x01869100 },
	368	{ _MMIO(0x9888), 0x0f870065 },
	369	{ _MMIO(0x9888), 0x01870000 },
	370	{ _MMIO(0x9888), 0x19930800 },
	371	{ _MMIO(0x9888), 0x0b938000 },
	372	{ _MMIO(0x9888), 0x0d938000 },
	373	{ _MMIO(0x9888), 0x1b952000 },
	374	{ _MMIO(0x9888), 0x1d955055 },
	375	{ _MMIO(0x9888), 0x1f951455 },
	376	{ _MMIO(0x9888), 0x0992a000 },
	377	{ _MMIO(0x9888), 0x0f928000 },
	378	{ _MMIO(0x9888), 0x1192a800 },
	379	{ _MMIO(0x9888), 0x1392028a },
	380	{ _MMIO(0x9888), 0x0b92a000 },
	381	{ _MMIO(0x9888), 0x0d922000 },
	382	{ _MMIO(0x9888), 0x13908000 },
	383	{ _MMIO(0x9888), 0x21908000 },
	384	{ _MMIO(0x9888), 0x23908000 },
	385	{ _MMIO(0x9888), 0x25908000 },
	386	{ _MMIO(0x9888), 0x27908000 },
	387	{ _MMIO(0x9888), 0x29908000 },
	388	{ _MMIO(0x9888), 0x2b908000 },
	389	{ _MMIO(0x9888), 0x2d904000 },
	390	{ _MMIO(0x9888), 0x2f908000 },
	391	{ _MMIO(0x9888), 0x31908000 },
	392	{ _MMIO(0x9888), 0x15908000 },
	393	{ _MMIO(0x9888), 0x17908000 },
	394	{ _MMIO(0x9888), 0x19908000 },
	395	{ _MMIO(0x9888), 0x1b908000 },
	396	{ _MMIO(0x9888), 0x1d904000 },
	397	{ _MMIO(0x9888), 0x1f904000 },
	398	{ _MMIO(0x9888), 0x53900000 },
	399	{ _MMIO(0x9888), 0x43900c01 },
	400	{ _MMIO(0x9888), 0x55900000 },
	401	{ _MMIO(0x9888), 0x47900000 },
	402	{ _MMIO(0x9888), 0x57900000 },
	403	{ _MMIO(0x9888), 0x49900863 },
	404	{ _MMIO(0x9888), 0x59900000 },
	405	{ _MMIO(0x9888), 0x4b900061 },
	406	{ _MMIO(0x9888), 0x37900000 },
	407	{ _MMIO(0x9888), 0x33900000 },
	408	{ _MMIO(0x9888), 0x4d900000 },
	409	{ _MMIO(0x9888), 0x45900c22 },
	410	};
	411
	412	static int
	413	get_render_pipe_profile_mux_config(struct drm_i915_private *dev_priv,
	414	const struct i915_oa_reg **regs,
	415	int *lens)
	416	{
	417	int n = 0;
	418
	419	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	420	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	421
	422	regs[n] = mux_config_render_pipe_profile;
	423	lens[n] = ARRAY_SIZE(mux_config_render_pipe_profile);
	424	n++;
	425
	426	return n;
	427	}
	428
	429	static const struct i915_oa_reg b_counter_config_memory_reads[] = {
	430	{ _MMIO(0x272c), 0xffffffff },
	431	{ _MMIO(0x2728), 0xffffffff },
	432	{ _MMIO(0x2724), 0xf0800000 },
	433	{ _MMIO(0x2720), 0x00000000 },
	434	{ _MMIO(0x271c), 0xffffffff },
	435	{ _MMIO(0x2718), 0xffffffff },
	436	{ _MMIO(0x2714), 0xf0800000 },
	437	{ _MMIO(0x2710), 0x00000000 },
	438	{ _MMIO(0x274c), 0x86543210 },
	439	{ _MMIO(0x2748), 0x86543210 },
	440	{ _MMIO(0x2744), 0x00006667 },
	441	{ _MMIO(0x2740), 0x00000000 },
	442	{ _MMIO(0x275c), 0x86543210 },
	443	{ _MMIO(0x2758), 0x86543210 },
	444	{ _MMIO(0x2754), 0x00006465 },
	445	{ _MMIO(0x2750), 0x00000000 },
	446	{ _MMIO(0x2770), 0x0007f81a },
	447	{ _MMIO(0x2774), 0x0000fe00 },
	448	{ _MMIO(0x2778), 0x0007f82a },
	449	{ _MMIO(0x277c), 0x0000fe00 },
	450	{ _MMIO(0x2780), 0x0007f872 },
	451	{ _MMIO(0x2784), 0x0000fe00 },
	452	{ _MMIO(0x2788), 0x0007f8ba },
	453	{ _MMIO(0x278c), 0x0000fe00 },
	454	{ _MMIO(0x2790), 0x0007f87a },
	455	{ _MMIO(0x2794), 0x0000fe00 },
	456	{ _MMIO(0x2798), 0x0007f8ea },
	457	{ _MMIO(0x279c), 0x0000fe00 },
	458	{ _MMIO(0x27a0), 0x0007f8e2 },
	459	{ _MMIO(0x27a4), 0x0000fe00 },
	460	{ _MMIO(0x27a8), 0x0007f8f2 },
	461	{ _MMIO(0x27ac), 0x0000fe00 },
	462	};
	463
	464	static const struct i915_oa_reg flex_eu_config_memory_reads[] = {
	465	{ _MMIO(0xe458), 0x00005004 },
	466	{ _MMIO(0xe558), 0x00015014 },
	467	{ _MMIO(0xe658), 0x00025024 },
	468	{ _MMIO(0xe758), 0x00035034 },
	469	{ _MMIO(0xe45c), 0x00045044 },
	470	{ _MMIO(0xe55c), 0x00055054 },
	471	{ _MMIO(0xe65c), 0x00065064 },
	472	};
	473
	474	static const struct i915_oa_reg mux_config_memory_reads[] = {
	475	{ _MMIO(0x9888), 0x19800343 },
	476	{ _MMIO(0x9888), 0x39900340 },
	477	{ _MMIO(0x9888), 0x3f901000 },
	478	{ _MMIO(0x9888), 0x41900003 },
	479	{ _MMIO(0x9888), 0x03803180 },
	480	{ _MMIO(0x9888), 0x058035e2 },
	481	{ _MMIO(0x9888), 0x0780006a },
	482	{ _MMIO(0x9888), 0x11800000 },
	483	{ _MMIO(0x9888), 0x2181a000 },
	484	{ _MMIO(0x9888), 0x2381000a },
	485	{ _MMIO(0x9888), 0x1d950550 },
	486	{ _MMIO(0x9888), 0x0b928000 },
	487	{ _MMIO(0x9888), 0x0d92a000 },
	488	{ _MMIO(0x9888), 0x0f922000 },
	489	{ _MMIO(0x9888), 0x13900170 },
	490	{ _MMIO(0x9888), 0x21900171 },
	491	{ _MMIO(0x9888), 0x23900172 },
	492	{ _MMIO(0x9888), 0x25900173 },
	493	{ _MMIO(0x9888), 0x27900174 },
	494	{ _MMIO(0x9888), 0x29900175 },
	495	{ _MMIO(0x9888), 0x2b900176 },
	496	{ _MMIO(0x9888), 0x2d900177 },
	497	{ _MMIO(0x9888), 0x2f90017f },
	498	{ _MMIO(0x9888), 0x31900125 },
	499	{ _MMIO(0x9888), 0x15900123 },
	500	{ _MMIO(0x9888), 0x17900121 },
	501	{ _MMIO(0x9888), 0x35900000 },
	502	{ _MMIO(0x9888), 0x19908000 },
	503	{ _MMIO(0x9888), 0x1b908000 },
	504	{ _MMIO(0x9888), 0x1d908000 },
	505	{ _MMIO(0x9888), 0x1f908000 },
	506	{ _MMIO(0x9888), 0x53900000 },
	507	{ _MMIO(0x9888), 0x43901084 },
	508	{ _MMIO(0x9888), 0x55900000 },
	509	{ _MMIO(0x9888), 0x47901080 },
	510	{ _MMIO(0x9888), 0x57900000 },
	511	{ _MMIO(0x9888), 0x49901084 },
	512	{ _MMIO(0x9888), 0x59900000 },
	513	{ _MMIO(0x9888), 0x4b901084 },
	514	{ _MMIO(0x9888), 0x37900000 },
	515	{ _MMIO(0x9888), 0x33900000 },
	516	{ _MMIO(0x9888), 0x4d900004 },
	517	{ _MMIO(0x9888), 0x45900000 },
	518	};
	519
	520	static int
	521	get_memory_reads_mux_config(struct drm_i915_private *dev_priv,
	522	const struct i915_oa_reg **regs,
	523	int *lens)
	524	{
	525	int n = 0;
	526
	527	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	528	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	529
	530	regs[n] = mux_config_memory_reads;
	531	lens[n] = ARRAY_SIZE(mux_config_memory_reads);
	532	n++;
	533
	534	return n;
	535	}
	536
	537	static const struct i915_oa_reg b_counter_config_memory_writes[] = {
	538	{ _MMIO(0x272c), 0xffffffff },
	539	{ _MMIO(0x2728), 0xffffffff },
	540	{ _MMIO(0x2724), 0xf0800000 },
	541	{ _MMIO(0x2720), 0x00000000 },
	542	{ _MMIO(0x271c), 0xffffffff },
	543	{ _MMIO(0x2718), 0xffffffff },
	544	{ _MMIO(0x2714), 0xf0800000 },
	545	{ _MMIO(0x2710), 0x00000000 },
	546	{ _MMIO(0x274c), 0x86543210 },
	547	{ _MMIO(0x2748), 0x86543210 },
	548	{ _MMIO(0x2744), 0x00006667 },
	549	{ _MMIO(0x2740), 0x00000000 },
	550	{ _MMIO(0x275c), 0x86543210 },
	551	{ _MMIO(0x2758), 0x86543210 },
	552	{ _MMIO(0x2754), 0x00006465 },
	553	{ _MMIO(0x2750), 0x00000000 },
	554	{ _MMIO(0x2770), 0x0007f81a },
	555	{ _MMIO(0x2774), 0x0000fe00 },
	556	{ _MMIO(0x2778), 0x0007f82a },
	557	{ _MMIO(0x277c), 0x0000fe00 },
	558	{ _MMIO(0x2780), 0x0007f822 },
	559	{ _MMIO(0x2784), 0x0000fe00 },
	560	{ _MMIO(0x2788), 0x0007f8ba },
	561	{ _MMIO(0x278c), 0x0000fe00 },
	562	{ _MMIO(0x2790), 0x0007f87a },
	563	{ _MMIO(0x2794), 0x0000fe00 },
	564	{ _MMIO(0x2798), 0x0007f8ea },
	565	{ _MMIO(0x279c), 0x0000fe00 },
	566	{ _MMIO(0x27a0), 0x0007f8e2 },
	567	{ _MMIO(0x27a4), 0x0000fe00 },
	568	{ _MMIO(0x27a8), 0x0007f8f2 },
	569	{ _MMIO(0x27ac), 0x0000fe00 },
	570	};
	571
	572	static const struct i915_oa_reg flex_eu_config_memory_writes[] = {
	573	{ _MMIO(0xe458), 0x00005004 },
	574	{ _MMIO(0xe558), 0x00015014 },
	575	{ _MMIO(0xe658), 0x00025024 },
	576	{ _MMIO(0xe758), 0x00035034 },
	577	{ _MMIO(0xe45c), 0x00045044 },
	578	{ _MMIO(0xe55c), 0x00055054 },
	579	{ _MMIO(0xe65c), 0x00065064 },
	580	};
	581
	582	static const struct i915_oa_reg mux_config_memory_writes[] = {
	583	{ _MMIO(0x9888), 0x19800343 },
	584	{ _MMIO(0x9888), 0x39900340 },
	585	{ _MMIO(0x9888), 0x3f900000 },
	586	{ _MMIO(0x9888), 0x41900080 },
	587	{ _MMIO(0x9888), 0x03803180 },
	588	{ _MMIO(0x9888), 0x058035e2 },
	589	{ _MMIO(0x9888), 0x0780006a },
	590	{ _MMIO(0x9888), 0x11800000 },
	591	{ _MMIO(0x9888), 0x2181a000 },
	592	{ _MMIO(0x9888), 0x2381000a },
	593	{ _MMIO(0x9888), 0x1d950550 },
	594	{ _MMIO(0x9888), 0x0b928000 },
	595	{ _MMIO(0x9888), 0x0d92a000 },
	596	{ _MMIO(0x9888), 0x0f922000 },
	597	{ _MMIO(0x9888), 0x13900180 },
	598	{ _MMIO(0x9888), 0x21900181 },
	599	{ _MMIO(0x9888), 0x23900182 },
	600	{ _MMIO(0x9888), 0x25900183 },
	601	{ _MMIO(0x9888), 0x27900184 },
	602	{ _MMIO(0x9888), 0x29900185 },
	603	{ _MMIO(0x9888), 0x2b900186 },
	604	{ _MMIO(0x9888), 0x2d900187 },
	605	{ _MMIO(0x9888), 0x2f900170 },
	606	{ _MMIO(0x9888), 0x31900125 },
	607	{ _MMIO(0x9888), 0x15900123 },
	608	{ _MMIO(0x9888), 0x17900121 },
	609	{ _MMIO(0x9888), 0x35900000 },
	610	{ _MMIO(0x9888), 0x19908000 },
	611	{ _MMIO(0x9888), 0x1b908000 },
	612	{ _MMIO(0x9888), 0x1d908000 },
	613	{ _MMIO(0x9888), 0x1f908000 },
	614	{ _MMIO(0x9888), 0x53900000 },
	615	{ _MMIO(0x9888), 0x43901084 },
	616	{ _MMIO(0x9888), 0x55900000 },
	617	{ _MMIO(0x9888), 0x47901080 },
	618	{ _MMIO(0x9888), 0x57900000 },
	619	{ _MMIO(0x9888), 0x49901084 },
	620	{ _MMIO(0x9888), 0x59900000 },
	621	{ _MMIO(0x9888), 0x4b901084 },
	622	{ _MMIO(0x9888), 0x37900000 },
	623	{ _MMIO(0x9888), 0x33900000 },
	624	{ _MMIO(0x9888), 0x4d900004 },
	625	{ _MMIO(0x9888), 0x45900000 },
	626	};
	627
	628	static int
	629	get_memory_writes_mux_config(struct drm_i915_private *dev_priv,
	630	const struct i915_oa_reg **regs,
	631	int *lens)
	632	{
	633	int n = 0;
	634
	635	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	636	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	637
	638	regs[n] = mux_config_memory_writes;
	639	lens[n] = ARRAY_SIZE(mux_config_memory_writes);
	640	n++;
	641
	642	return n;
	643	}
	644
	645	static const struct i915_oa_reg b_counter_config_compute_extended[] = {
	646	{ _MMIO(0x2724), 0xf0800000 },
	647	{ _MMIO(0x2720), 0x00000000 },
	648	{ _MMIO(0x2714), 0xf0800000 },
	649	{ _MMIO(0x2710), 0x00000000 },
	650	{ _MMIO(0x2740), 0x00000000 },
	651	{ _MMIO(0x2770), 0x0007fc2a },
	652	{ _MMIO(0x2774), 0x0000bf00 },
	653	{ _MMIO(0x2778), 0x0007fc6a },
	654	{ _MMIO(0x277c), 0x0000bf00 },
	655	{ _MMIO(0x2780), 0x0007fc92 },
	656	{ _MMIO(0x2784), 0x0000bf00 },
	657	{ _MMIO(0x2788), 0x0007fca2 },
	658	{ _MMIO(0x278c), 0x0000bf00 },
	659	{ _MMIO(0x2790), 0x0007fc32 },
	660	{ _MMIO(0x2794), 0x0000bf00 },
	661	{ _MMIO(0x2798), 0x0007fc9a },
	662	{ _MMIO(0x279c), 0x0000bf00 },
	663	{ _MMIO(0x27a0), 0x0007fe6a },
	664	{ _MMIO(0x27a4), 0x0000bf00 },
	665	{ _MMIO(0x27a8), 0x0007fe7a },
	666	{ _MMIO(0x27ac), 0x0000bf00 },
	667	};
	668
	669	static const struct i915_oa_reg flex_eu_config_compute_extended[] = {
	670	{ _MMIO(0xe458), 0x00005004 },
	671	{ _MMIO(0xe558), 0x00000003 },
	672	{ _MMIO(0xe658), 0x00002001 },
	673	{ _MMIO(0xe758), 0x00778008 },
	674	{ _MMIO(0xe45c), 0x00088078 },
	675	{ _MMIO(0xe55c), 0x00808708 },
	676	{ _MMIO(0xe65c), 0x00a08908 },
	677	};
	678
	679	static const struct i915_oa_reg mux_config_compute_extended[] = {
	680	{ _MMIO(0x9888), 0x104f00e0 },
	681	{ _MMIO(0x9888), 0x141c0160 },
	682	{ _MMIO(0x9888), 0x161c0015 },
	683	{ _MMIO(0x9888), 0x181c0120 },
	684	{ _MMIO(0x9888), 0x002d5000 },
	685	{ _MMIO(0x9888), 0x062d4000 },
	686	{ _MMIO(0x9888), 0x082d5000 },
	687	{ _MMIO(0x9888), 0x0a2d5000 },
	688	{ _MMIO(0x9888), 0x0c2d5000 },
	689	{ _MMIO(0x9888), 0x0e2d5000 },
	690	{ _MMIO(0x9888), 0x022d5000 },
	691	{ _MMIO(0x9888), 0x042d5000 },
	692	{ _MMIO(0x9888), 0x0c2e5400 },
	693	{ _MMIO(0x9888), 0x0e2e5515 },
	694	{ _MMIO(0x9888), 0x102e0155 },
	695	{ _MMIO(0x9888), 0x044cc000 },
	696	{ _MMIO(0x9888), 0x0a4c8000 },
	697	{ _MMIO(0x9888), 0x0c4cc000 },
	698	{ _MMIO(0x9888), 0x0e4cc000 },
	699	{ _MMIO(0x9888), 0x104c8000 },
	700	{ _MMIO(0x9888), 0x124c8000 },
	701	{ _MMIO(0x9888), 0x144c8000 },
	702	{ _MMIO(0x9888), 0x164c2000 },
	703	{ _MMIO(0x9888), 0x064cc000 },
	704	{ _MMIO(0x9888), 0x084cc000 },
	705	{ _MMIO(0x9888), 0x004ea000 },
	706	{ _MMIO(0x9888), 0x064e8000 },
	707	{ _MMIO(0x9888), 0x084ea000 },
	708	{ _MMIO(0x9888), 0x0a4ea000 },
	709	{ _MMIO(0x9888), 0x0c4ea000 },
	710	{ _MMIO(0x9888), 0x0e4ea000 },
	711	{ _MMIO(0x9888), 0x024ea000 },
	712	{ _MMIO(0x9888), 0x044ea000 },
	713	{ _MMIO(0x9888), 0x0e4f4b41 },
	714	{ _MMIO(0x9888), 0x004f4200 },
	715	{ _MMIO(0x9888), 0x024f404c },
	716	{ _MMIO(0x9888), 0x1c4f0000 },
	717	{ _MMIO(0x9888), 0x1a4f0000 },
	718	{ _MMIO(0x9888), 0x001b4000 },
	719	{ _MMIO(0x9888), 0x061b8000 },
	720	{ _MMIO(0x9888), 0x081bc000 },
	721	{ _MMIO(0x9888), 0x0a1bc000 },
	722	{ _MMIO(0x9888), 0x0c1bc000 },
	723	{ _MMIO(0x9888), 0x041bc000 },
	724	{ _MMIO(0x9888), 0x001c0031 },
	725	{ _MMIO(0x9888), 0x061c1900 },
	726	{ _MMIO(0x9888), 0x081c1a33 },
	727	{ _MMIO(0x9888), 0x0a1c1b35 },
	728	{ _MMIO(0x9888), 0x0c1c3337 },
	729	{ _MMIO(0x9888), 0x041c31c7 },
	730	{ _MMIO(0x9888), 0x180f5000 },
	731	{ _MMIO(0x9888), 0x1a0fa8aa },
	732	{ _MMIO(0x9888), 0x1c0f0aaa },
	733	{ _MMIO(0x9888), 0x182c8000 },
	734	{ _MMIO(0x9888), 0x1c2c6aaa },
	735	{ _MMIO(0x9888), 0x1e2c0001 },
	736	{ _MMIO(0x9888), 0x1a2c2950 },
	737	{ _MMIO(0x9888), 0x01938000 },
	738	{ _MMIO(0x9888), 0x0f938000 },
	739	{ _MMIO(0x9888), 0x1993aaaa },
	740	{ _MMIO(0x9888), 0x03938000 },
	741	{ _MMIO(0x9888), 0x05938000 },
	742	{ _MMIO(0x9888), 0x07938000 },
	743	{ _MMIO(0x9888), 0x09938000 },
	744	{ _MMIO(0x9888), 0x0b938000 },
	745	{ _MMIO(0x9888), 0x13904000 },
	746	{ _MMIO(0x9888), 0x21904000 },
	747	{ _MMIO(0x9888), 0x23904000 },
	748	{ _MMIO(0x9888), 0x25904000 },
	749	{ _MMIO(0x9888), 0x27904000 },
	750	{ _MMIO(0x9888), 0x29904000 },
	751	{ _MMIO(0x9888), 0x2b904000 },
	752	{ _MMIO(0x9888), 0x2d904000 },
	753	{ _MMIO(0x9888), 0x2f904000 },
	754	{ _MMIO(0x9888), 0x31904000 },
	755	{ _MMIO(0x9888), 0x15904000 },
	756	{ _MMIO(0x9888), 0x17904000 },
	757	{ _MMIO(0x9888), 0x19904000 },
	758	{ _MMIO(0x9888), 0x1b904000 },
	759	{ _MMIO(0x9888), 0x1d904000 },
	760	{ _MMIO(0x9888), 0x53900000 },
	761	{ _MMIO(0x9888), 0x43900420 },
	762	{ _MMIO(0x9888), 0x55900000 },
	763	{ _MMIO(0x9888), 0x47900000 },
	764	{ _MMIO(0x9888), 0x57900000 },
	765	{ _MMIO(0x9888), 0x49900000 },
	766	{ _MMIO(0x9888), 0x59900000 },
	767	{ _MMIO(0x9888), 0x4b900400 },
	768	{ _MMIO(0x9888), 0x37900000 },
	769	{ _MMIO(0x9888), 0x33900000 },
	770	{ _MMIO(0x9888), 0x4d900001 },
	771	{ _MMIO(0x9888), 0x45900001 },
	772	};
	773
	774	static int
	775	get_compute_extended_mux_config(struct drm_i915_private *dev_priv,
	776	const struct i915_oa_reg **regs,
	777	int *lens)
	778	{
	779	int n = 0;
	780
	781	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	782	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	783
	784	regs[n] = mux_config_compute_extended;
	785	lens[n] = ARRAY_SIZE(mux_config_compute_extended);
	786	n++;
	787
	788	return n;
	789	}
	790
	791	static const struct i915_oa_reg b_counter_config_compute_l3_cache[] = {
	792	{ _MMIO(0x2710), 0x00000000 },
	793	{ _MMIO(0x2714), 0x30800000 },
	794	{ _MMIO(0x2720), 0x00000000 },
	795	{ _MMIO(0x2724), 0x30800000 },
	796	{ _MMIO(0x2740), 0x00000000 },
	797	{ _MMIO(0x2770), 0x0007fffa },
	798	{ _MMIO(0x2774), 0x0000fefe },
	799	{ _MMIO(0x2778), 0x0007fffa },
	800	{ _MMIO(0x277c), 0x0000fefd },
	801	{ _MMIO(0x2790), 0x0007fffa },
	802	{ _MMIO(0x2794), 0x0000fbef },
	803	{ _MMIO(0x2798), 0x0007fffa },
	804	{ _MMIO(0x279c), 0x0000fbdf },
	805	};
	806
	807	static const struct i915_oa_reg flex_eu_config_compute_l3_cache[] = {
	808	{ _MMIO(0xe458), 0x00005004 },
	809	{ _MMIO(0xe558), 0x00000003 },
	810	{ _MMIO(0xe658), 0x00002001 },
	811	{ _MMIO(0xe758), 0x00101100 },
	812	{ _MMIO(0xe45c), 0x00201200 },
	813	{ _MMIO(0xe55c), 0x00301300 },
	814	{ _MMIO(0xe65c), 0x00401400 },
	815	};
	816
	817	static const struct i915_oa_reg mux_config_compute_l3_cache[] = {
	818	{ _MMIO(0x9888), 0x166c03b0 },
	819	{ _MMIO(0x9888), 0x1593001e },
	820	{ _MMIO(0x9888), 0x3f900c00 },
	821	{ _MMIO(0x9888), 0x41900000 },
	822	{ _MMIO(0x9888), 0x002d1000 },
	823	{ _MMIO(0x9888), 0x062d4000 },
	824	{ _MMIO(0x9888), 0x082d5000 },
	825	{ _MMIO(0x9888), 0x0e2d5000 },
	826	{ _MMIO(0x9888), 0x0c2e0400 },
	827	{ _MMIO(0x9888), 0x0e2e1500 },
	828	{ _MMIO(0x9888), 0x102e0140 },
	829	{ _MMIO(0x9888), 0x044c4000 },
	830	{ _MMIO(0x9888), 0x0a4c8000 },
	831	{ _MMIO(0x9888), 0x0c4cc000 },
	832	{ _MMIO(0x9888), 0x144c8000 },
	833	{ _MMIO(0x9888), 0x164c2000 },
	834	{ _MMIO(0x9888), 0x004e2000 },
	835	{ _MMIO(0x9888), 0x064e8000 },
	836	{ _MMIO(0x9888), 0x084ea000 },
	837	{ _MMIO(0x9888), 0x0e4ea000 },
	838	{ _MMIO(0x9888), 0x1a4f4001 },
	839	{ _MMIO(0x9888), 0x1c4f5005 },
	840	{ _MMIO(0x9888), 0x006c0051 },
	841	{ _MMIO(0x9888), 0x066c5000 },
	842	{ _MMIO(0x9888), 0x086c5c5d },
	843	{ _MMIO(0x9888), 0x0e6c5e5f },
	844	{ _MMIO(0x9888), 0x106c0000 },
	845	{ _MMIO(0x9888), 0x146c0000 },
	846	{ _MMIO(0x9888), 0x1a6c0000 },
	847	{ _MMIO(0x9888), 0x1c6c0000 },
	848	{ _MMIO(0x9888), 0x180f1000 },
	849	{ _MMIO(0x9888), 0x1a0fa800 },
	850	{ _MMIO(0x9888), 0x1c0f0a00 },
	851	{ _MMIO(0x9888), 0x182c4000 },
	852	{ _MMIO(0x9888), 0x1c2c4015 },
	853	{ _MMIO(0x9888), 0x1e2c0001 },
	854	{ _MMIO(0x9888), 0x03931980 },
	855	{ _MMIO(0x9888), 0x05930032 },
	856	{ _MMIO(0x9888), 0x11930000 },
	857	{ _MMIO(0x9888), 0x01938000 },
	858	{ _MMIO(0x9888), 0x0f938000 },
	859	{ _MMIO(0x9888), 0x1993a00a },
	860	{ _MMIO(0x9888), 0x07930000 },
	861	{ _MMIO(0x9888), 0x09930000 },
	862	{ _MMIO(0x9888), 0x1d900177 },
	863	{ _MMIO(0x9888), 0x1f900178 },
	864	{ _MMIO(0x9888), 0x35900000 },
	865	{ _MMIO(0x9888), 0x13904000 },
	866	{ _MMIO(0x9888), 0x21904000 },
	867	{ _MMIO(0x9888), 0x23904000 },
	868	{ _MMIO(0x9888), 0x25904000 },
	869	{ _MMIO(0x9888), 0x2f904000 },
	870	{ _MMIO(0x9888), 0x31904000 },
	871	{ _MMIO(0x9888), 0x19904000 },
	872	{ _MMIO(0x9888), 0x1b904000 },
	873	{ _MMIO(0x9888), 0x53901000 },
	874	{ _MMIO(0x9888), 0x43900000 },
	875	{ _MMIO(0x9888), 0x55900111 },
	876	{ _MMIO(0x9888), 0x47900001 },
	877	{ _MMIO(0x9888), 0x57900000 },
	878	{ _MMIO(0x9888), 0x49900000 },
	879	{ _MMIO(0x9888), 0x37900000 },
	880	{ _MMIO(0x9888), 0x33900000 },
	881	{ _MMIO(0x9888), 0x59900000 },
	882	{ _MMIO(0x9888), 0x4b900000 },
	883	{ _MMIO(0x9888), 0x4d900000 },
	884	{ _MMIO(0x9888), 0x45900400 },
	885	};
	886
	887	static int
	888	get_compute_l3_cache_mux_config(struct drm_i915_private *dev_priv,
	889	const struct i915_oa_reg **regs,
	890	int *lens)
	891	{
	892	int n = 0;
	893
	894	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	895	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	896
	897	regs[n] = mux_config_compute_l3_cache;
	898	lens[n] = ARRAY_SIZE(mux_config_compute_l3_cache);
	899	n++;
	900
	901	return n;
	902	}
	903
	904	static const struct i915_oa_reg b_counter_config_hdc_and_sf[] = {
	905	{ _MMIO(0x2740), 0x00000000 },
	906	{ _MMIO(0x2744), 0x00800000 },
	907	{ _MMIO(0x2710), 0x00000000 },
	908	{ _MMIO(0x2714), 0x10800000 },
	909	{ _MMIO(0x2720), 0x00000000 },
	910	{ _MMIO(0x2724), 0x00800000 },
	911	{ _MMIO(0x2770), 0x00000002 },
	912	{ _MMIO(0x2774), 0x0000fdff },
	913	};
	914
	915	static const struct i915_oa_reg flex_eu_config_hdc_and_sf[] = {
	916	{ _MMIO(0xe458), 0x00005004 },
	917	{ _MMIO(0xe558), 0x00010003 },
	918	{ _MMIO(0xe658), 0x00012011 },
	919	{ _MMIO(0xe758), 0x00015014 },
	920	{ _MMIO(0xe45c), 0x00051050 },
	921	{ _MMIO(0xe55c), 0x00053052 },
	922	{ _MMIO(0xe65c), 0x00055054 },
	923	};
	924
	925	static const struct i915_oa_reg mux_config_hdc_and_sf[] = {
	926	{ _MMIO(0x9888), 0x104f0232 },
	927	{ _MMIO(0x9888), 0x124f4640 },
	928	{ _MMIO(0x9888), 0x11834400 },
	929	{ _MMIO(0x9888), 0x022d4000 },
	930	{ _MMIO(0x9888), 0x042d5000 },
	931	{ _MMIO(0x9888), 0x062d1000 },
	932	{ _MMIO(0x9888), 0x0e2e0055 },
	933	{ _MMIO(0x9888), 0x064c8000 },
	934	{ _MMIO(0x9888), 0x084cc000 },
	935	{ _MMIO(0x9888), 0x0a4c4000 },
	936	{ _MMIO(0x9888), 0x024e8000 },
	937	{ _MMIO(0x9888), 0x044ea000 },
	938	{ _MMIO(0x9888), 0x064e2000 },
	939	{ _MMIO(0x9888), 0x024f6100 },
	940	{ _MMIO(0x9888), 0x044f416b },
	941	{ _MMIO(0x9888), 0x064f004b },
	942	{ _MMIO(0x9888), 0x1a4f0000 },
	943	{ _MMIO(0x9888), 0x1a0f02a8 },
	944	{ _MMIO(0x9888), 0x1a2c5500 },
	945	{ _MMIO(0x9888), 0x0f808000 },
	946	{ _MMIO(0x9888), 0x25810020 },
	947	{ _MMIO(0x9888), 0x0f8305c0 },
	948	{ _MMIO(0x9888), 0x07938000 },
	949	{ _MMIO(0x9888), 0x09938000 },
	950	{ _MMIO(0x9888), 0x0b938000 },
	951	{ _MMIO(0x9888), 0x0d938000 },
	952	{ _MMIO(0x9888), 0x1f951000 },
	953	{ _MMIO(0x9888), 0x13920200 },
	954	{ _MMIO(0x9888), 0x31908000 },
	955	{ _MMIO(0x9888), 0x19904000 },
	956	{ _MMIO(0x9888), 0x1b904000 },
	957	{ _MMIO(0x9888), 0x1d904000 },
	958	{ _MMIO(0x9888), 0x1f904000 },
	959	{ _MMIO(0x9888), 0x37900000 },
	960	{ _MMIO(0x9888), 0x59900000 },
	961	{ _MMIO(0x9888), 0x4d900003 },
	962	{ _MMIO(0x9888), 0x53900000 },
	963	{ _MMIO(0x9888), 0x45900000 },
	964	{ _MMIO(0x9888), 0x55900000 },
	965	{ _MMIO(0x9888), 0x47900000 },
	966	{ _MMIO(0x9888), 0x33900000 },
	967	};
	968
	969	static int
	970	get_hdc_and_sf_mux_config(struct drm_i915_private *dev_priv,
	971	const struct i915_oa_reg **regs,
	972	int *lens)
	973	{
	974	int n = 0;
	975
	976	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	977	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	978
	979	regs[n] = mux_config_hdc_and_sf;
	980	lens[n] = ARRAY_SIZE(mux_config_hdc_and_sf);
	981	n++;
	982
	983	return n;
	984	}
	985
	986	static const struct i915_oa_reg b_counter_config_l3_1[] = {
	987	{ _MMIO(0x2740), 0x00000000 },
	988	{ _MMIO(0x2744), 0x00800000 },
	989	{ _MMIO(0x2710), 0x00000000 },
	990	{ _MMIO(0x2714), 0xf0800000 },
	991	{ _MMIO(0x2720), 0x00000000 },
	992	{ _MMIO(0x2724), 0xf0800000 },
	993	{ _MMIO(0x2770), 0x00100070 },
	994	{ _MMIO(0x2774), 0x0000fff1 },
	995	{ _MMIO(0x2778), 0x00014002 },
	996	{ _MMIO(0x277c), 0x0000c3ff },
	997	{ _MMIO(0x2780), 0x00010002 },
	998	{ _MMIO(0x2784), 0x0000c7ff },
	999	{ _MMIO(0x2788), 0x00004002 },
	1000	{ _MMIO(0x278c), 0x0000d3ff },
	1001	{ _MMIO(0x2790), 0x00100700 },
	1002	{ _MMIO(0x2794), 0x0000ff1f },
	1003	{ _MMIO(0x2798), 0x00001402 },
	1004	{ _MMIO(0x279c), 0x0000fc3f },
	1005	{ _MMIO(0x27a0), 0x00001002 },
	1006	{ _MMIO(0x27a4), 0x0000fc7f },
	1007	{ _MMIO(0x27a8), 0x00000402 },
	1008	{ _MMIO(0x27ac), 0x0000fd3f },
	1009	};
	1010
	1011	static const struct i915_oa_reg flex_eu_config_l3_1[] = {
	1012	{ _MMIO(0xe458), 0x00005004 },
	1013	{ _MMIO(0xe558), 0x00010003 },
	1014	{ _MMIO(0xe658), 0x00012011 },
	1015	{ _MMIO(0xe758), 0x00015014 },
	1016	{ _MMIO(0xe45c), 0x00051050 },
	1017	{ _MMIO(0xe55c), 0x00053052 },
	1018	{ _MMIO(0xe65c), 0x00055054 },
	1019	};
	1020
	1021	static const struct i915_oa_reg mux_config_l3_1_0_sku_gte_0x03[] = {
	1022	{ _MMIO(0x9888), 0x12643400 },
	1023	{ _MMIO(0x9888), 0x12653400 },
	1024	{ _MMIO(0x9888), 0x106c6800 },
	1025	{ _MMIO(0x9888), 0x126c001e },
	1026	{ _MMIO(0x9888), 0x166c0010 },
	1027	{ _MMIO(0x9888), 0x0c2d5000 },
	1028	{ _MMIO(0x9888), 0x0e2d5000 },
	1029	{ _MMIO(0x9888), 0x002d4000 },
	1030	{ _MMIO(0x9888), 0x022d5000 },
	1031	{ _MMIO(0x9888), 0x042d5000 },
	1032	{ _MMIO(0x9888), 0x062d1000 },
	1033	{ _MMIO(0x9888), 0x102e0154 },
	1034	{ _MMIO(0x9888), 0x0c2e5000 },
	1035	{ _MMIO(0x9888), 0x0e2e0055 },
	1036	{ _MMIO(0x9888), 0x104c8000 },
	1037	{ _MMIO(0x9888), 0x124c8000 },
	1038	{ _MMIO(0x9888), 0x144c8000 },
	1039	{ _MMIO(0x9888), 0x164c2000 },
	1040	{ _MMIO(0x9888), 0x044c8000 },
	1041	{ _MMIO(0x9888), 0x064cc000 },
	1042	{ _MMIO(0x9888), 0x084cc000 },
	1043	{ _MMIO(0x9888), 0x0a4c4000 },
	1044	{ _MMIO(0x9888), 0x0c4ea000 },
	1045	{ _MMIO(0x9888), 0x0e4ea000 },
	1046	{ _MMIO(0x9888), 0x004e8000 },
	1047	{ _MMIO(0x9888), 0x024ea000 },
	1048	{ _MMIO(0x9888), 0x044ea000 },
	1049	{ _MMIO(0x9888), 0x064e2000 },
	1050	{ _MMIO(0x9888), 0x1c4f5500 },
	1051	{ _MMIO(0x9888), 0x1a4f1554 },
	1052	{ _MMIO(0x9888), 0x0a640024 },
	1053	{ _MMIO(0x9888), 0x10640000 },
	1054	{ _MMIO(0x9888), 0x04640000 },
	1055	{ _MMIO(0x9888), 0x0c650024 },
	1056	{ _MMIO(0x9888), 0x10650000 },
	1057	{ _MMIO(0x9888), 0x06650000 },
	1058	{ _MMIO(0x9888), 0x0c6c5327 },
	1059	{ _MMIO(0x9888), 0x0e6c5425 },
	1060	{ _MMIO(0x9888), 0x006c2a00 },
	1061	{ _MMIO(0x9888), 0x026c285b },
	1062	{ _MMIO(0x9888), 0x046c005c },
	1063	{ _MMIO(0x9888), 0x1c6c0000 },
	1064	{ _MMIO(0x9888), 0x1a6c0900 },
	1065	{ _MMIO(0x9888), 0x1c0f0aa0 },
	1066	{ _MMIO(0x9888), 0x180f4000 },
	1067	{ _MMIO(0x9888), 0x1a0f02aa },
	1068	{ _MMIO(0x9888), 0x1c2c5400 },
	1069	{ _MMIO(0x9888), 0x1e2c0001 },
	1070	{ _MMIO(0x9888), 0x1a2c5550 },
	1071	{ _MMIO(0x9888), 0x1993aa00 },
	1072	{ _MMIO(0x9888), 0x03938000 },
	1073	{ _MMIO(0x9888), 0x05938000 },
	1074	{ _MMIO(0x9888), 0x07938000 },
	1075	{ _MMIO(0x9888), 0x09938000 },
	1076	{ _MMIO(0x9888), 0x0b938000 },
	1077	{ _MMIO(0x9888), 0x0d938000 },
	1078	{ _MMIO(0x9888), 0x2b904000 },
	1079	{ _MMIO(0x9888), 0x2d904000 },
	1080	{ _MMIO(0x9888), 0x2f904000 },
	1081	{ _MMIO(0x9888), 0x31904000 },
	1082	{ _MMIO(0x9888), 0x15904000 },
	1083	{ _MMIO(0x9888), 0x17904000 },
	1084	{ _MMIO(0x9888), 0x19904000 },
	1085	{ _MMIO(0x9888), 0x1b904000 },
	1086	{ _MMIO(0x9888), 0x1d904000 },
	1087	{ _MMIO(0x9888), 0x1f904000 },
	1088	{ _MMIO(0x9888), 0x59900000 },
	1089	{ _MMIO(0x9888), 0x4b900421 },
	1090	{ _MMIO(0x9888), 0x37900000 },
	1091	{ _MMIO(0x9888), 0x33900000 },
	1092	{ _MMIO(0x9888), 0x4d900001 },
	1093	{ _MMIO(0x9888), 0x53900000 },
	1094	{ _MMIO(0x9888), 0x43900420 },
	1095	{ _MMIO(0x9888), 0x45900021 },
	1096	{ _MMIO(0x9888), 0x55900000 },
	1097	{ _MMIO(0x9888), 0x47900000 },
	1098	};
	1099
	1100	static const struct i915_oa_reg mux_config_l3_1_0_sku_lt_0x03[] = {
	1101	{ _MMIO(0x9888), 0x14640340 },
	1102	{ _MMIO(0x9888), 0x14650340 },
	1103	{ _MMIO(0x9888), 0x106c6800 },
	1104	{ _MMIO(0x9888), 0x126c001e },
	1105	{ _MMIO(0x9888), 0x166c0010 },
	1106	{ _MMIO(0x9888), 0x0c2d5000 },
	1107	{ _MMIO(0x9888), 0x0e2d5000 },
	1108	{ _MMIO(0x9888), 0x002d4000 },
	1109	{ _MMIO(0x9888), 0x022d5000 },
	1110	{ _MMIO(0x9888), 0x042d5000 },
	1111	{ _MMIO(0x9888), 0x062d1000 },
	1112	{ _MMIO(0x9888), 0x102e0154 },
	1113	{ _MMIO(0x9888), 0x0c2e5000 },
	1114	{ _MMIO(0x9888), 0x0e2e0055 },
	1115	{ _MMIO(0x9888), 0x104c8000 },
	1116	{ _MMIO(0x9888), 0x124c8000 },
	1117	{ _MMIO(0x9888), 0x144c8000 },
	1118	{ _MMIO(0x9888), 0x164c2000 },
	1119	{ _MMIO(0x9888), 0x044c8000 },
	1120	{ _MMIO(0x9888), 0x064cc000 },
	1121	{ _MMIO(0x9888), 0x084cc000 },
	1122	{ _MMIO(0x9888), 0x0a4c4000 },
	1123	{ _MMIO(0x9888), 0x0c4ea000 },
	1124	{ _MMIO(0x9888), 0x0e4ea000 },
	1125	{ _MMIO(0x9888), 0x004e8000 },
	1126	{ _MMIO(0x9888), 0x024ea000 },
	1127	{ _MMIO(0x9888), 0x044ea000 },
	1128	{ _MMIO(0x9888), 0x064e2000 },
	1129	{ _MMIO(0x9888), 0x1c4f5500 },
	1130	{ _MMIO(0x9888), 0x1a4f1554 },
	1131	{ _MMIO(0x9888), 0x04642400 },
	1132	{ _MMIO(0x9888), 0x22640000 },
	1133	{ _MMIO(0x9888), 0x1a640000 },
	1134	{ _MMIO(0x9888), 0x06650024 },
	1135	{ _MMIO(0x9888), 0x22650000 },
	1136	{ _MMIO(0x9888), 0x1c650000 },
	1137	{ _MMIO(0x9888), 0x0c6c5327 },
	1138	{ _MMIO(0x9888), 0x0e6c5425 },
	1139	{ _MMIO(0x9888), 0x006c2a00 },
	1140	{ _MMIO(0x9888), 0x026c285b },
	1141	{ _MMIO(0x9888), 0x046c005c },
	1142	{ _MMIO(0x9888), 0x1c6c0000 },
	1143	{ _MMIO(0x9888), 0x1a6c0900 },
	1144	{ _MMIO(0x9888), 0x1c0f0aa0 },
	1145	{ _MMIO(0x9888), 0x180f4000 },
	1146	{ _MMIO(0x9888), 0x1a0f02aa },
	1147	{ _MMIO(0x9888), 0x1c2c5400 },
	1148	{ _MMIO(0x9888), 0x1e2c0001 },
	1149	{ _MMIO(0x9888), 0x1a2c5550 },
	1150	{ _MMIO(0x9888), 0x1993aa00 },
	1151	{ _MMIO(0x9888), 0x03938000 },
	1152	{ _MMIO(0x9888), 0x05938000 },
	1153	{ _MMIO(0x9888), 0x07938000 },
	1154	{ _MMIO(0x9888), 0x09938000 },
	1155	{ _MMIO(0x9888), 0x0b938000 },
	1156	{ _MMIO(0x9888), 0x0d938000 },
	1157	{ _MMIO(0x9888), 0x2b904000 },
	1158	{ _MMIO(0x9888), 0x2d904000 },
	1159	{ _MMIO(0x9888), 0x2f904000 },
	1160	{ _MMIO(0x9888), 0x31904000 },
	1161	{ _MMIO(0x9888), 0x15904000 },
	1162	{ _MMIO(0x9888), 0x17904000 },
	1163	{ _MMIO(0x9888), 0x19904000 },
	1164	{ _MMIO(0x9888), 0x1b904000 },
	1165	{ _MMIO(0x9888), 0x1d904000 },
	1166	{ _MMIO(0x9888), 0x1f904000 },
	1167	{ _MMIO(0x9888), 0x59900000 },
	1168	{ _MMIO(0x9888), 0x4b900421 },
	1169	{ _MMIO(0x9888), 0x37900000 },
	1170	{ _MMIO(0x9888), 0x33900000 },
	1171	{ _MMIO(0x9888), 0x4d900001 },
	1172	{ _MMIO(0x9888), 0x53900000 },
	1173	{ _MMIO(0x9888), 0x43900420 },
	1174	{ _MMIO(0x9888), 0x45900021 },
	1175	{ _MMIO(0x9888), 0x55900000 },
	1176	{ _MMIO(0x9888), 0x47900000 },
	1177	};
	1178
	1179	static int
	1180	get_l3_1_mux_config(struct drm_i915_private *dev_priv,
	1181	const struct i915_oa_reg **regs,
	1182	int *lens)
	1183	{
	1184	int n = 0;
	1185
	1186	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 2);
	1187	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 2);
	1188
	1189	if (dev_priv->drm.pdev->revision >= 0x03) {
	1190	regs[n] = mux_config_l3_1_0_sku_gte_0x03;
	1191	lens[n] = ARRAY_SIZE(mux_config_l3_1_0_sku_gte_0x03);
	1192	n++;
	1193	}
	1194	if (dev_priv->drm.pdev->revision < 0x03) {
	1195	regs[n] = mux_config_l3_1_0_sku_lt_0x03;
	1196	lens[n] = ARRAY_SIZE(mux_config_l3_1_0_sku_lt_0x03);
	1197	n++;
	1198	}
	1199
	1200	return n;
	1201	}
	1202
	1203	static const struct i915_oa_reg b_counter_config_rasterizer_and_pixel_backend[] = {
	1204	{ _MMIO(0x2740), 0x00000000 },
	1205	{ _MMIO(0x2744), 0x00800000 },
	1206	{ _MMIO(0x2710), 0x00000000 },
	1207	{ _MMIO(0x2714), 0x30800000 },
	1208	{ _MMIO(0x2720), 0x00000000 },
	1209	{ _MMIO(0x2724), 0x00800000 },
	1210	{ _MMIO(0x2770), 0x00000002 },
	1211	{ _MMIO(0x2774), 0x0000efff },
	1212	{ _MMIO(0x2778), 0x00006000 },
	1213	{ _MMIO(0x277c), 0x0000f3ff },
	1214	};
	1215
	1216	static const struct i915_oa_reg flex_eu_config_rasterizer_and_pixel_backend[] = {
	1217	{ _MMIO(0xe458), 0x00005004 },
	1218	{ _MMIO(0xe558), 0x00010003 },
	1219	{ _MMIO(0xe658), 0x00012011 },
	1220	{ _MMIO(0xe758), 0x00015014 },
	1221	{ _MMIO(0xe45c), 0x00051050 },
	1222	{ _MMIO(0xe55c), 0x00053052 },
	1223	{ _MMIO(0xe65c), 0x00055054 },
	1224	};
	1225
	1226	static const struct i915_oa_reg mux_config_rasterizer_and_pixel_backend[] = {
	1227	{ _MMIO(0x9888), 0x102d7800 },
	1228	{ _MMIO(0x9888), 0x122d79e0 },
	1229	{ _MMIO(0x9888), 0x0c2f0004 },
	1230	{ _MMIO(0x9888), 0x100e3800 },
	1231	{ _MMIO(0x9888), 0x180f0005 },
	1232	{ _MMIO(0x9888), 0x002d0940 },
	1233	{ _MMIO(0x9888), 0x022d802f },
	1234	{ _MMIO(0x9888), 0x042d4013 },
	1235	{ _MMIO(0x9888), 0x062d1000 },
	1236	{ _MMIO(0x9888), 0x0e2e0050 },
	1237	{ _MMIO(0x9888), 0x022f0010 },
	1238	{ _MMIO(0x9888), 0x002f0000 },
	1239	{ _MMIO(0x9888), 0x084c8000 },
	1240	{ _MMIO(0x9888), 0x0a4c4000 },
	1241	{ _MMIO(0x9888), 0x044e8000 },
	1242	{ _MMIO(0x9888), 0x064e2000 },
	1243	{ _MMIO(0x9888), 0x040e0480 },
	1244	{ _MMIO(0x9888), 0x000e0000 },
	1245	{ _MMIO(0x9888), 0x060f0027 },
	1246	{ _MMIO(0x9888), 0x100f0000 },
	1247	{ _MMIO(0x9888), 0x1a0f0040 },
	1248	{ _MMIO(0x9888), 0x03938000 },
	1249	{ _MMIO(0x9888), 0x05938000 },
	1250	{ _MMIO(0x9888), 0x07938000 },
	1251	{ _MMIO(0x9888), 0x09938000 },
	1252	{ _MMIO(0x9888), 0x0b938000 },
	1253	{ _MMIO(0x9888), 0x0d938000 },
	1254	{ _MMIO(0x9888), 0x15904000 },
	1255	{ _MMIO(0x9888), 0x17904000 },
	1256	{ _MMIO(0x9888), 0x19904000 },
	1257	{ _MMIO(0x9888), 0x1b904000 },
	1258	{ _MMIO(0x9888), 0x1d904000 },
	1259	{ _MMIO(0x9888), 0x1f904000 },
	1260	{ _MMIO(0x9888), 0x37900000 },
	1261	{ _MMIO(0x9888), 0x53900000 },
	1262	{ _MMIO(0x9888), 0x439014a0 },
	1263	{ _MMIO(0x9888), 0x459000a4 },
	1264	{ _MMIO(0x9888), 0x55900000 },
	1265	{ _MMIO(0x9888), 0x47900001 },
	1266	{ _MMIO(0x9888), 0x33900000 },
	1267	};
	1268
	1269	static int
	1270	get_rasterizer_and_pixel_backend_mux_config(struct drm_i915_private *dev_priv,
	1271	const struct i915_oa_reg **regs,
	1272	int *lens)
	1273	{
	1274	int n = 0;
	1275
	1276	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1277	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1278
	1279	regs[n] = mux_config_rasterizer_and_pixel_backend;
	1280	lens[n] = ARRAY_SIZE(mux_config_rasterizer_and_pixel_backend);
	1281	n++;
	1282
	1283	return n;
	1284	}
	1285
	1286	static const struct i915_oa_reg b_counter_config_sampler[] = {
	1287	{ _MMIO(0x2740), 0x00000000 },
	1288	{ _MMIO(0x2744), 0x00800000 },
	1289	{ _MMIO(0x2710), 0x00000000 },
	1290	{ _MMIO(0x2714), 0x70800000 },
	1291	{ _MMIO(0x2720), 0x00000000 },
	1292	{ _MMIO(0x2724), 0x00800000 },
	1293	{ _MMIO(0x2770), 0x0000c000 },
	1294	{ _MMIO(0x2774), 0x0000e7ff },
	1295	{ _MMIO(0x2778), 0x00003000 },
	1296	{ _MMIO(0x277c), 0x0000f9ff },
	1297	{ _MMIO(0x2780), 0x00000c00 },
	1298	{ _MMIO(0x2784), 0x0000fe7f },
	1299	};
	1300
	1301	static const struct i915_oa_reg flex_eu_config_sampler[] = {
	1302	{ _MMIO(0xe458), 0x00005004 },
	1303	{ _MMIO(0xe558), 0x00010003 },
	1304	{ _MMIO(0xe658), 0x00012011 },
	1305	{ _MMIO(0xe758), 0x00015014 },
	1306	{ _MMIO(0xe45c), 0x00051050 },
	1307	{ _MMIO(0xe55c), 0x00053052 },
	1308	{ _MMIO(0xe65c), 0x00055054 },
	1309	};
	1310
	1311	static const struct i915_oa_reg mux_config_sampler[] = {
	1312	{ _MMIO(0x9888), 0x121300a0 },
	1313	{ _MMIO(0x9888), 0x141600ab },
	1314	{ _MMIO(0x9888), 0x123300a0 },
	1315	{ _MMIO(0x9888), 0x143600ab },
	1316	{ _MMIO(0x9888), 0x125300a0 },
	1317	{ _MMIO(0x9888), 0x145600ab },
	1318	{ _MMIO(0x9888), 0x0c2d4000 },
	1319	{ _MMIO(0x9888), 0x0e2d5000 },
	1320	{ _MMIO(0x9888), 0x002d4000 },
	1321	{ _MMIO(0x9888), 0x022d5000 },
	1322	{ _MMIO(0x9888), 0x042d5000 },
	1323	{ _MMIO(0x9888), 0x062d1000 },
	1324	{ _MMIO(0x9888), 0x102e01a0 },
	1325	{ _MMIO(0x9888), 0x0c2e5000 },
	1326	{ _MMIO(0x9888), 0x0e2e0065 },
	1327	{ _MMIO(0x9888), 0x164c2000 },
	1328	{ _MMIO(0x9888), 0x044c8000 },
	1329	{ _MMIO(0x9888), 0x064cc000 },
	1330	{ _MMIO(0x9888), 0x084c4000 },
	1331	{ _MMIO(0x9888), 0x0a4c4000 },
	1332	{ _MMIO(0x9888), 0x0e4e8000 },
	1333	{ _MMIO(0x9888), 0x004e8000 },
	1334	{ _MMIO(0x9888), 0x024ea000 },
	1335	{ _MMIO(0x9888), 0x044e2000 },
	1336	{ _MMIO(0x9888), 0x064e2000 },
	1337	{ _MMIO(0x9888), 0x1c0f0800 },
	1338	{ _MMIO(0x9888), 0x180f4000 },
	1339	{ _MMIO(0x9888), 0x1a0f023f },
	1340	{ _MMIO(0x9888), 0x1e2c0003 },
	1341	{ _MMIO(0x9888), 0x1a2cc030 },
	1342	{ _MMIO(0x9888), 0x04132180 },
	1343	{ _MMIO(0x9888), 0x02130000 },
	1344	{ _MMIO(0x9888), 0x0c148000 },
	1345	{ _MMIO(0x9888), 0x0e142000 },
	1346	{ _MMIO(0x9888), 0x04148000 },
	1347	{ _MMIO(0x9888), 0x1e150140 },
	1348	{ _MMIO(0x9888), 0x1c150040 },
	1349	{ _MMIO(0x9888), 0x0c163000 },
	1350	{ _MMIO(0x9888), 0x0e160068 },
	1351	{ _MMIO(0x9888), 0x10160000 },
	1352	{ _MMIO(0x9888), 0x18160000 },
	1353	{ _MMIO(0x9888), 0x0a164000 },
	1354	{ _MMIO(0x9888), 0x04330043 },
	1355	{ _MMIO(0x9888), 0x02330000 },
	1356	{ _MMIO(0x9888), 0x0234a000 },
	1357	{ _MMIO(0x9888), 0x04342000 },
	1358	{ _MMIO(0x9888), 0x1c350015 },
	1359	{ _MMIO(0x9888), 0x02363460 },
	1360	{ _MMIO(0x9888), 0x10360000 },
	1361	{ _MMIO(0x9888), 0x04360000 },
	1362	{ _MMIO(0x9888), 0x06360000 },
	1363	{ _MMIO(0x9888), 0x08364000 },
	1364	{ _MMIO(0x9888), 0x06530043 },
	1365	{ _MMIO(0x9888), 0x02530000 },
	1366	{ _MMIO(0x9888), 0x0e548000 },
	1367	{ _MMIO(0x9888), 0x00548000 },
	1368	{ _MMIO(0x9888), 0x06542000 },
	1369	{ _MMIO(0x9888), 0x1e550400 },
	1370	{ _MMIO(0x9888), 0x1a552000 },
	1371	{ _MMIO(0x9888), 0x1c550100 },
	1372	{ _MMIO(0x9888), 0x0e563000 },
	1373	{ _MMIO(0x9888), 0x00563400 },
	1374	{ _MMIO(0x9888), 0x10560000 },
	1375	{ _MMIO(0x9888), 0x18560000 },
	1376	{ _MMIO(0x9888), 0x02560000 },
	1377	{ _MMIO(0x9888), 0x0c564000 },
	1378	{ _MMIO(0x9888), 0x1993a800 },
	1379	{ _MMIO(0x9888), 0x03938000 },
	1380	{ _MMIO(0x9888), 0x05938000 },
	1381	{ _MMIO(0x9888), 0x07938000 },
	1382	{ _MMIO(0x9888), 0x09938000 },
	1383	{ _MMIO(0x9888), 0x0b938000 },
	1384	{ _MMIO(0x9888), 0x0d938000 },
	1385	{ _MMIO(0x9888), 0x2d904000 },
	1386	{ _MMIO(0x9888), 0x2f904000 },
	1387	{ _MMIO(0x9888), 0x31904000 },
	1388	{ _MMIO(0x9888), 0x15904000 },
	1389	{ _MMIO(0x9888), 0x17904000 },
	1390	{ _MMIO(0x9888), 0x19904000 },
	1391	{ _MMIO(0x9888), 0x1b904000 },
	1392	{ _MMIO(0x9888), 0x1d904000 },
	1393	{ _MMIO(0x9888), 0x1f904000 },
	1394	{ _MMIO(0x9888), 0x59900000 },
	1395	{ _MMIO(0x9888), 0x4b9014a0 },
	1396	{ _MMIO(0x9888), 0x37900000 },
	1397	{ _MMIO(0x9888), 0x33900000 },
	1398	{ _MMIO(0x9888), 0x4d900001 },
	1399	{ _MMIO(0x9888), 0x53900000 },
	1400	{ _MMIO(0x9888), 0x43900820 },
	1401	{ _MMIO(0x9888), 0x45901022 },
	1402	{ _MMIO(0x9888), 0x55900000 },
	1403	{ _MMIO(0x9888), 0x47900000 },
	1404	};
	1405
	1406	static int
	1407	get_sampler_mux_config(struct drm_i915_private *dev_priv,
	1408	const struct i915_oa_reg **regs,
	1409	int *lens)
	1410	{
	1411	int n = 0;
	1412
	1413	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1414	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1415
	1416	regs[n] = mux_config_sampler;
	1417	lens[n] = ARRAY_SIZE(mux_config_sampler);
	1418	n++;
	1419
	1420	return n;
	1421	}
	1422
	1423	static const struct i915_oa_reg b_counter_config_tdl_1[] = {
	1424	{ _MMIO(0x2740), 0x00000000 },
	1425	{ _MMIO(0x2744), 0x00800000 },
	1426	{ _MMIO(0x2710), 0x00000000 },
	1427	{ _MMIO(0x2714), 0xf0800000 },
	1428	{ _MMIO(0x2720), 0x00000000 },
	1429	{ _MMIO(0x2724), 0x30800000 },
	1430	{ _MMIO(0x2770), 0x00000002 },
	1431	{ _MMIO(0x2774), 0x00007fff },
	1432	{ _MMIO(0x2778), 0x00000000 },
	1433	{ _MMIO(0x277c), 0x00009fff },
	1434	{ _MMIO(0x2780), 0x00000002 },
	1435	{ _MMIO(0x2784), 0x0000efff },
	1436	{ _MMIO(0x2788), 0x00000000 },
	1437	{ _MMIO(0x278c), 0x0000f3ff },
	1438	{ _MMIO(0x2790), 0x00000002 },
	1439	{ _MMIO(0x2794), 0x0000fdff },
	1440	{ _MMIO(0x2798), 0x00000000 },
	1441	{ _MMIO(0x279c), 0x0000fe7f },
	1442	};
	1443
	1444	static const struct i915_oa_reg flex_eu_config_tdl_1[] = {
	1445	{ _MMIO(0xe458), 0x00005004 },
	1446	{ _MMIO(0xe558), 0x00010003 },
	1447	{ _MMIO(0xe658), 0x00012011 },
	1448	{ _MMIO(0xe758), 0x00015014 },
	1449	{ _MMIO(0xe45c), 0x00051050 },
	1450	{ _MMIO(0xe55c), 0x00053052 },
	1451	{ _MMIO(0xe65c), 0x00055054 },
	1452	};
	1453
	1454	static const struct i915_oa_reg mux_config_tdl_1[] = {
	1455	{ _MMIO(0x9888), 0x141a0000 },
	1456	{ _MMIO(0x9888), 0x143a0000 },
	1457	{ _MMIO(0x9888), 0x145a0000 },
	1458	{ _MMIO(0x9888), 0x0c2d4000 },
	1459	{ _MMIO(0x9888), 0x0e2d5000 },
	1460	{ _MMIO(0x9888), 0x002d4000 },
	1461	{ _MMIO(0x9888), 0x022d5000 },
	1462	{ _MMIO(0x9888), 0x042d5000 },
	1463	{ _MMIO(0x9888), 0x062d1000 },
	1464	{ _MMIO(0x9888), 0x102e0150 },
	1465	{ _MMIO(0x9888), 0x0c2e5000 },
	1466	{ _MMIO(0x9888), 0x0e2e006a },
	1467	{ _MMIO(0x9888), 0x124c8000 },
	1468	{ _MMIO(0x9888), 0x144c8000 },
	1469	{ _MMIO(0x9888), 0x164c2000 },
	1470	{ _MMIO(0x9888), 0x044c8000 },
	1471	{ _MMIO(0x9888), 0x064c4000 },
	1472	{ _MMIO(0x9888), 0x0a4c4000 },
	1473	{ _MMIO(0x9888), 0x0c4e8000 },
	1474	{ _MMIO(0x9888), 0x0e4ea000 },
	1475	{ _MMIO(0x9888), 0x004e8000 },
	1476	{ _MMIO(0x9888), 0x024e2000 },
	1477	{ _MMIO(0x9888), 0x064e2000 },
	1478	{ _MMIO(0x9888), 0x1c0f0bc0 },
	1479	{ _MMIO(0x9888), 0x180f4000 },
	1480	{ _MMIO(0x9888), 0x1a0f0302 },
	1481	{ _MMIO(0x9888), 0x1e2c0003 },
	1482	{ _MMIO(0x9888), 0x1a2c00f0 },
	1483	{ _MMIO(0x9888), 0x021a3080 },
	1484	{ _MMIO(0x9888), 0x041a31e5 },
	1485	{ _MMIO(0x9888), 0x02148000 },
	1486	{ _MMIO(0x9888), 0x0414a000 },
	1487	{ _MMIO(0x9888), 0x1c150054 },
	1488	{ _MMIO(0x9888), 0x06168000 },
	1489	{ _MMIO(0x9888), 0x08168000 },
	1490	{ _MMIO(0x9888), 0x0a168000 },
	1491	{ _MMIO(0x9888), 0x0c3a3280 },
	1492	{ _MMIO(0x9888), 0x0e3a0063 },
	1493	{ _MMIO(0x9888), 0x063a0061 },
	1494	{ _MMIO(0x9888), 0x023a0000 },
	1495	{ _MMIO(0x9888), 0x0c348000 },
	1496	{ _MMIO(0x9888), 0x0e342000 },
	1497	{ _MMIO(0x9888), 0x06342000 },
	1498	{ _MMIO(0x9888), 0x1e350140 },
	1499	{ _MMIO(0x9888), 0x1c350100 },
	1500	{ _MMIO(0x9888), 0x18360028 },
	1501	{ _MMIO(0x9888), 0x0c368000 },
	1502	{ _MMIO(0x9888), 0x0e5a3080 },
	1503	{ _MMIO(0x9888), 0x005a3280 },
	1504	{ _MMIO(0x9888), 0x025a0063 },
	1505	{ _MMIO(0x9888), 0x0e548000 },
	1506	{ _MMIO(0x9888), 0x00548000 },
	1507	{ _MMIO(0x9888), 0x02542000 },
	1508	{ _MMIO(0x9888), 0x1e550400 },
	1509	{ _MMIO(0x9888), 0x1a552000 },
	1510	{ _MMIO(0x9888), 0x1c550001 },
	1511	{ _MMIO(0x9888), 0x18560080 },
	1512	{ _MMIO(0x9888), 0x02568000 },
	1513	{ _MMIO(0x9888), 0x04568000 },
	1514	{ _MMIO(0x9888), 0x1993a800 },
	1515	{ _MMIO(0x9888), 0x03938000 },
	1516	{ _MMIO(0x9888), 0x05938000 },
	1517	{ _MMIO(0x9888), 0x07938000 },
	1518	{ _MMIO(0x9888), 0x09938000 },
	1519	{ _MMIO(0x9888), 0x0b938000 },
	1520	{ _MMIO(0x9888), 0x0d938000 },
	1521	{ _MMIO(0x9888), 0x2d904000 },
	1522	{ _MMIO(0x9888), 0x2f904000 },
	1523	{ _MMIO(0x9888), 0x31904000 },
	1524	{ _MMIO(0x9888), 0x15904000 },
	1525	{ _MMIO(0x9888), 0x17904000 },
	1526	{ _MMIO(0x9888), 0x19904000 },
	1527	{ _MMIO(0x9888), 0x1b904000 },
	1528	{ _MMIO(0x9888), 0x1d904000 },
	1529	{ _MMIO(0x9888), 0x1f904000 },
	1530	{ _MMIO(0x9888), 0x59900000 },
	1531	{ _MMIO(0x9888), 0x4b900420 },
	1532	{ _MMIO(0x9888), 0x37900000 },
	1533	{ _MMIO(0x9888), 0x33900000 },
	1534	{ _MMIO(0x9888), 0x4d900000 },
	1535	{ _MMIO(0x9888), 0x53900000 },
	1536	{ _MMIO(0x9888), 0x43900000 },
	1537	{ _MMIO(0x9888), 0x45901084 },
	1538	{ _MMIO(0x9888), 0x55900000 },
	1539	{ _MMIO(0x9888), 0x47900001 },
	1540	};
	1541
	1542	static int
	1543	get_tdl_1_mux_config(struct drm_i915_private *dev_priv,
	1544	const struct i915_oa_reg **regs,
	1545	int *lens)
	1546	{
	1547	int n = 0;
	1548
	1549	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1550	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1551
	1552	regs[n] = mux_config_tdl_1;
	1553	lens[n] = ARRAY_SIZE(mux_config_tdl_1);
	1554	n++;
	1555
	1556	return n;
	1557	}
	1558
	1559	static const struct i915_oa_reg b_counter_config_tdl_2[] = {
	1560	{ _MMIO(0x2740), 0x00000000 },
	1561	{ _MMIO(0x2744), 0x00800000 },
	1562	{ _MMIO(0x2710), 0x00000000 },
	1563	{ _MMIO(0x2714), 0x00800000 },
	1564	{ _MMIO(0x2720), 0x00000000 },
	1565	{ _MMIO(0x2724), 0x00800000 },
	1566	};
	1567
	1568	static const struct i915_oa_reg flex_eu_config_tdl_2[] = {
	1569	{ _MMIO(0xe458), 0x00005004 },
	1570	{ _MMIO(0xe558), 0x00010003 },
	1571	{ _MMIO(0xe658), 0x00012011 },
	1572	{ _MMIO(0xe758), 0x00015014 },
	1573	{ _MMIO(0xe45c), 0x00051050 },
	1574	{ _MMIO(0xe55c), 0x00053052 },
	1575	{ _MMIO(0xe65c), 0x00055054 },
	1576	};
	1577
	1578	static const struct i915_oa_reg mux_config_tdl_2[] = {
	1579	{ _MMIO(0x9888), 0x141a026b },
	1580	{ _MMIO(0x9888), 0x143a0173 },
	1581	{ _MMIO(0x9888), 0x145a026b },
	1582	{ _MMIO(0x9888), 0x002d4000 },
	1583	{ _MMIO(0x9888), 0x022d5000 },
	1584	{ _MMIO(0x9888), 0x042d5000 },
	1585	{ _MMIO(0x9888), 0x062d1000 },
	1586	{ _MMIO(0x9888), 0x0c2e5000 },
	1587	{ _MMIO(0x9888), 0x0e2e0069 },
	1588	{ _MMIO(0x9888), 0x044c8000 },
	1589	{ _MMIO(0x9888), 0x064cc000 },
	1590	{ _MMIO(0x9888), 0x0a4c4000 },
	1591	{ _MMIO(0x9888), 0x004e8000 },
	1592	{ _MMIO(0x9888), 0x024ea000 },
	1593	{ _MMIO(0x9888), 0x064e2000 },
	1594	{ _MMIO(0x9888), 0x180f6000 },
	1595	{ _MMIO(0x9888), 0x1a0f030a },
	1596	{ _MMIO(0x9888), 0x1a2c03c0 },
	1597	{ _MMIO(0x9888), 0x041a37e7 },
	1598	{ _MMIO(0x9888), 0x021a0000 },
	1599	{ _MMIO(0x9888), 0x0414a000 },
	1600	{ _MMIO(0x9888), 0x1c150050 },
	1601	{ _MMIO(0x9888), 0x08168000 },
	1602	{ _MMIO(0x9888), 0x0a168000 },
	1603	{ _MMIO(0x9888), 0x003a3380 },
	1604	{ _MMIO(0x9888), 0x063a006f },
	1605	{ _MMIO(0x9888), 0x023a0000 },
	1606	{ _MMIO(0x9888), 0x00348000 },
	1607	{ _MMIO(0x9888), 0x06342000 },
	1608	{ _MMIO(0x9888), 0x1a352000 },
	1609	{ _MMIO(0x9888), 0x1c350100 },
	1610	{ _MMIO(0x9888), 0x02368000 },
	1611	{ _MMIO(0x9888), 0x0c368000 },
	1612	{ _MMIO(0x9888), 0x025a37e7 },
	1613	{ _MMIO(0x9888), 0x0254a000 },
	1614	{ _MMIO(0x9888), 0x1c550005 },
	1615	{ _MMIO(0x9888), 0x04568000 },
	1616	{ _MMIO(0x9888), 0x06568000 },
	1617	{ _MMIO(0x9888), 0x03938000 },
	1618	{ _MMIO(0x9888), 0x05938000 },
	1619	{ _MMIO(0x9888), 0x07938000 },
	1620	{ _MMIO(0x9888), 0x09938000 },
	1621	{ _MMIO(0x9888), 0x0b938000 },
	1622	{ _MMIO(0x9888), 0x0d938000 },
	1623	{ _MMIO(0x9888), 0x15904000 },
	1624	{ _MMIO(0x9888), 0x17904000 },
	1625	{ _MMIO(0x9888), 0x19904000 },
	1626	{ _MMIO(0x9888), 0x1b904000 },
	1627	{ _MMIO(0x9888), 0x1d904000 },
	1628	{ _MMIO(0x9888), 0x1f904000 },
	1629	{ _MMIO(0x9888), 0x37900000 },
	1630	{ _MMIO(0x9888), 0x53900000 },
	1631	{ _MMIO(0x9888), 0x43900020 },
	1632	{ _MMIO(0x9888), 0x45901080 },
	1633	{ _MMIO(0x9888), 0x55900000 },
	1634	{ _MMIO(0x9888), 0x47900001 },
	1635	{ _MMIO(0x9888), 0x33900000 },
	1636	};
	1637
	1638	static int
	1639	get_tdl_2_mux_config(struct drm_i915_private *dev_priv,
	1640	const struct i915_oa_reg **regs,
	1641	int *lens)
	1642	{
	1643	int n = 0;
	1644
	1645	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1646	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1647
	1648	regs[n] = mux_config_tdl_2;
	1649	lens[n] = ARRAY_SIZE(mux_config_tdl_2);
	1650	n++;
	1651
	1652	return n;
	1653	}
	1654
	1655	static const struct i915_oa_reg b_counter_config_compute_extra[] = {
	1656	{ _MMIO(0x2740), 0x00000000 },
	1657	{ _MMIO(0x2744), 0x00800000 },
	1658	{ _MMIO(0x2710), 0x00000000 },
	1659	{ _MMIO(0x2714), 0x00800000 },
	1660	{ _MMIO(0x2720), 0x00000000 },
	1661	{ _MMIO(0x2724), 0x00800000 },
	1662	};
	1663
	1664	static const struct i915_oa_reg flex_eu_config_compute_extra[] = {
	1665	{ _MMIO(0xe458), 0x00001000 },
	1666	{ _MMIO(0xe558), 0x00003002 },
	1667	{ _MMIO(0xe658), 0x00005004 },
	1668	{ _MMIO(0xe758), 0x00011010 },
	1669	{ _MMIO(0xe45c), 0x00050012 },
	1670	{ _MMIO(0xe55c), 0x00052051 },
	1671	{ _MMIO(0xe65c), 0x00000008 },
	1672	};
	1673
	1674	static const struct i915_oa_reg mux_config_compute_extra[] = {
	1675	{ _MMIO(0x9888), 0x141a001f },
	1676	{ _MMIO(0x9888), 0x143a001f },
	1677	{ _MMIO(0x9888), 0x145a001f },
	1678	{ _MMIO(0x9888), 0x042d5000 },
	1679	{ _MMIO(0x9888), 0x062d1000 },
	1680	{ _MMIO(0x9888), 0x0e2e0094 },
	1681	{ _MMIO(0x9888), 0x084cc000 },
	1682	{ _MMIO(0x9888), 0x044ea000 },
	1683	{ _MMIO(0x9888), 0x1a0f00e0 },
	1684	{ _MMIO(0x9888), 0x1a2c0c00 },
	1685	{ _MMIO(0x9888), 0x061a0063 },
	1686	{ _MMIO(0x9888), 0x021a0000 },
	1687	{ _MMIO(0x9888), 0x06142000 },
	1688	{ _MMIO(0x9888), 0x1c150100 },
	1689	{ _MMIO(0x9888), 0x0c168000 },
	1690	{ _MMIO(0x9888), 0x043a3180 },
	1691	{ _MMIO(0x9888), 0x023a0000 },
	1692	{ _MMIO(0x9888), 0x04348000 },
	1693	{ _MMIO(0x9888), 0x1c350040 },
	1694	{ _MMIO(0x9888), 0x0a368000 },
	1695	{ _MMIO(0x9888), 0x045a0063 },
	1696	{ _MMIO(0x9888), 0x025a0000 },
	1697	{ _MMIO(0x9888), 0x04542000 },
	1698	{ _MMIO(0x9888), 0x1c550010 },
	1699	{ _MMIO(0x9888), 0x08568000 },
	1700	{ _MMIO(0x9888), 0x09938000 },
	1701	{ _MMIO(0x9888), 0x0b938000 },
	1702	{ _MMIO(0x9888), 0x0d938000 },
	1703	{ _MMIO(0x9888), 0x1b904000 },
	1704	{ _MMIO(0x9888), 0x1d904000 },
	1705	{ _MMIO(0x9888), 0x1f904000 },
	1706	{ _MMIO(0x9888), 0x37900000 },
	1707	{ _MMIO(0x9888), 0x55900000 },
	1708	{ _MMIO(0x9888), 0x45900400 },
	1709	{ _MMIO(0x9888), 0x47900004 },
	1710	{ _MMIO(0x9888), 0x33900000 },
	1711	};
	1712
	1713	static int
	1714	get_compute_extra_mux_config(struct drm_i915_private *dev_priv,
	1715	const struct i915_oa_reg **regs,
	1716	int *lens)
	1717	{
	1718	int n = 0;
	1719
	1720	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1721	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1722
	1723	regs[n] = mux_config_compute_extra;
	1724	lens[n] = ARRAY_SIZE(mux_config_compute_extra);
	1725	n++;
	1726
	1727	return n;
	1728	}
	1729
	1730	static const struct i915_oa_reg b_counter_config_test_oa[] = {
	1731	{ _MMIO(0x2740), 0x00000000 },
	1732	{ _MMIO(0x2744), 0x00800000 },
	1733	{ _MMIO(0x2714), 0xf0800000 },
	1734	{ _MMIO(0x2710), 0x00000000 },
	1735	{ _MMIO(0x2724), 0xf0800000 },
	1736	{ _MMIO(0x2720), 0x00000000 },
	1737	{ _MMIO(0x2770), 0x00000004 },
	1738	{ _MMIO(0x2774), 0x00000000 },
	1739	{ _MMIO(0x2778), 0x00000003 },
	1740	{ _MMIO(0x277c), 0x00000000 },
	1741	{ _MMIO(0x2780), 0x00000007 },
	1742	{ _MMIO(0x2784), 0x00000000 },
	1743	{ _MMIO(0x2788), 0x00100002 },
	1744	{ _MMIO(0x278c), 0x0000fff7 },
	1745	{ _MMIO(0x2790), 0x00100002 },
	1746	{ _MMIO(0x2794), 0x0000ffcf },
	1747	{ _MMIO(0x2798), 0x00100082 },
	1748	{ _MMIO(0x279c), 0x0000ffef },
	1749	{ _MMIO(0x27a0), 0x001000c2 },
	1750	{ _MMIO(0x27a4), 0x0000ffe7 },
	1751	{ _MMIO(0x27a8), 0x00100001 },
	1752	{ _MMIO(0x27ac), 0x0000ffe7 },
	1753	};
	1754
	1755	static const struct i915_oa_reg flex_eu_config_test_oa[] = {
	1756	};
	1757
	1758	static const struct i915_oa_reg mux_config_test_oa[] = {
	1759	{ _MMIO(0x9888), 0x19800000 },
	1760	{ _MMIO(0x9888), 0x07800063 },
	1761	{ _MMIO(0x9888), 0x11800000 },
	1762	{ _MMIO(0x9888), 0x23810008 },
	1763	{ _MMIO(0x9888), 0x1d950400 },
	1764	{ _MMIO(0x9888), 0x0f922000 },
	1765	{ _MMIO(0x9888), 0x1f908000 },
	1766	{ _MMIO(0x9888), 0x37900000 },
	1767	{ _MMIO(0x9888), 0x55900000 },
	1768	{ _MMIO(0x9888), 0x47900000 },
	1769	{ _MMIO(0x9888), 0x33900000 },
	1770	};
	1771
	1772	static int
	1773	get_test_oa_mux_config(struct drm_i915_private *dev_priv,
	1774	const struct i915_oa_reg **regs,
	1775	int *lens)
	1776	{
	1777	int n = 0;
	1778
	1779	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1780	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1781
	1782	regs[n] = mux_config_test_oa;
	1783	lens[n] = ARRAY_SIZE(mux_config_test_oa);
	1784	n++;
	1785
	1786	return n;
	1787	}
	1788
	1789	int i915_oa_select_metric_set_bxt(struct drm_i915_private *dev_priv)
	1790	{
	1791	dev_priv->perf.oa.n_mux_configs = 0;
	1792	dev_priv->perf.oa.b_counter_regs = NULL;
	1793	dev_priv->perf.oa.b_counter_regs_len = 0;
	1794	dev_priv->perf.oa.flex_regs = NULL;
	1795	dev_priv->perf.oa.flex_regs_len = 0;
	1796
	1797	switch (dev_priv->perf.oa.metrics_set) {
	1798	case METRIC_SET_ID_RENDER_BASIC:
	1799	dev_priv->perf.oa.n_mux_configs =
	1800	get_render_basic_mux_config(dev_priv,
	1801	dev_priv->perf.oa.mux_regs,
	1802	dev_priv->perf.oa.mux_regs_lens);
	1803	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1804	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_BASIC\" metric set\n");
	1805
	1806	/* EINVAL because *_register_sysfs already checked this
	1807	* and so it wouldn't have been advertised to userspace and
	1808	* so shouldn't have been requested
	1809	*/
	1810	return -EINVAL;
	1811	}
	1812
	1813	dev_priv->perf.oa.b_counter_regs =
	1814	b_counter_config_render_basic;
	1815	dev_priv->perf.oa.b_counter_regs_len =
	1816	ARRAY_SIZE(b_counter_config_render_basic);
	1817
	1818	dev_priv->perf.oa.flex_regs =
	1819	flex_eu_config_render_basic;
	1820	dev_priv->perf.oa.flex_regs_len =
	1821	ARRAY_SIZE(flex_eu_config_render_basic);
	1822
	1823	return 0;
	1824	case METRIC_SET_ID_COMPUTE_BASIC:
	1825	dev_priv->perf.oa.n_mux_configs =
	1826	get_compute_basic_mux_config(dev_priv,
	1827	dev_priv->perf.oa.mux_regs,
	1828	dev_priv->perf.oa.mux_regs_lens);
	1829	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1830	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_BASIC\" metric set\n");
	1831
	1832	/* EINVAL because *_register_sysfs already checked this
	1833	* and so it wouldn't have been advertised to userspace and
	1834	* so shouldn't have been requested
	1835	*/
	1836	return -EINVAL;
	1837	}
	1838
	1839	dev_priv->perf.oa.b_counter_regs =
	1840	b_counter_config_compute_basic;
	1841	dev_priv->perf.oa.b_counter_regs_len =
	1842	ARRAY_SIZE(b_counter_config_compute_basic);
	1843
	1844	dev_priv->perf.oa.flex_regs =
	1845	flex_eu_config_compute_basic;
	1846	dev_priv->perf.oa.flex_regs_len =
	1847	ARRAY_SIZE(flex_eu_config_compute_basic);
	1848
	1849	return 0;
	1850	case METRIC_SET_ID_RENDER_PIPE_PROFILE:
	1851	dev_priv->perf.oa.n_mux_configs =
	1852	get_render_pipe_profile_mux_config(dev_priv,
	1853	dev_priv->perf.oa.mux_regs,
	1854	dev_priv->perf.oa.mux_regs_lens);
	1855	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1856	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_PIPE_PROFILE\" metric set\n");
	1857
	1858	/* EINVAL because *_register_sysfs already checked this
	1859	* and so it wouldn't have been advertised to userspace and
	1860	* so shouldn't have been requested
	1861	*/
	1862	return -EINVAL;
	1863	}
	1864
	1865	dev_priv->perf.oa.b_counter_regs =
	1866	b_counter_config_render_pipe_profile;
	1867	dev_priv->perf.oa.b_counter_regs_len =
	1868	ARRAY_SIZE(b_counter_config_render_pipe_profile);
	1869
	1870	dev_priv->perf.oa.flex_regs =
	1871	flex_eu_config_render_pipe_profile;
	1872	dev_priv->perf.oa.flex_regs_len =
	1873	ARRAY_SIZE(flex_eu_config_render_pipe_profile);
	1874
	1875	return 0;
	1876	case METRIC_SET_ID_MEMORY_READS:
	1877	dev_priv->perf.oa.n_mux_configs =
	1878	get_memory_reads_mux_config(dev_priv,
	1879	dev_priv->perf.oa.mux_regs,
	1880	dev_priv->perf.oa.mux_regs_lens);
	1881	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1882	DRM_DEBUG_DRIVER("No suitable MUX config for \"MEMORY_READS\" metric set\n");
	1883
	1884	/* EINVAL because *_register_sysfs already checked this
	1885	* and so it wouldn't have been advertised to userspace and
	1886	* so shouldn't have been requested
	1887	*/
	1888	return -EINVAL;
	1889	}
	1890
	1891	dev_priv->perf.oa.b_counter_regs =
	1892	b_counter_config_memory_reads;
	1893	dev_priv->perf.oa.b_counter_regs_len =
	1894	ARRAY_SIZE(b_counter_config_memory_reads);
	1895
	1896	dev_priv->perf.oa.flex_regs =
	1897	flex_eu_config_memory_reads;
	1898	dev_priv->perf.oa.flex_regs_len =
	1899	ARRAY_SIZE(flex_eu_config_memory_reads);
	1900
	1901	return 0;
	1902	case METRIC_SET_ID_MEMORY_WRITES:
	1903	dev_priv->perf.oa.n_mux_configs =
	1904	get_memory_writes_mux_config(dev_priv,
	1905	dev_priv->perf.oa.mux_regs,
	1906	dev_priv->perf.oa.mux_regs_lens);
	1907	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1908	DRM_DEBUG_DRIVER("No suitable MUX config for \"MEMORY_WRITES\" metric set\n");
	1909
	1910	/* EINVAL because *_register_sysfs already checked this
	1911	* and so it wouldn't have been advertised to userspace and
	1912	* so shouldn't have been requested
	1913	*/
	1914	return -EINVAL;
	1915	}
	1916
	1917	dev_priv->perf.oa.b_counter_regs =
	1918	b_counter_config_memory_writes;
	1919	dev_priv->perf.oa.b_counter_regs_len =
	1920	ARRAY_SIZE(b_counter_config_memory_writes);
	1921
	1922	dev_priv->perf.oa.flex_regs =
	1923	flex_eu_config_memory_writes;
	1924	dev_priv->perf.oa.flex_regs_len =
	1925	ARRAY_SIZE(flex_eu_config_memory_writes);
	1926
	1927	return 0;
	1928	case METRIC_SET_ID_COMPUTE_EXTENDED:
	1929	dev_priv->perf.oa.n_mux_configs =
	1930	get_compute_extended_mux_config(dev_priv,
	1931	dev_priv->perf.oa.mux_regs,
	1932	dev_priv->perf.oa.mux_regs_lens);
	1933	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1934	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_EXTENDED\" metric set\n");
	1935
	1936	/* EINVAL because *_register_sysfs already checked this
	1937	* and so it wouldn't have been advertised to userspace and
	1938	* so shouldn't have been requested
	1939	*/
	1940	return -EINVAL;
	1941	}
	1942
	1943	dev_priv->perf.oa.b_counter_regs =
	1944	b_counter_config_compute_extended;
	1945	dev_priv->perf.oa.b_counter_regs_len =
	1946	ARRAY_SIZE(b_counter_config_compute_extended);
	1947
	1948	dev_priv->perf.oa.flex_regs =
	1949	flex_eu_config_compute_extended;
	1950	dev_priv->perf.oa.flex_regs_len =
	1951	ARRAY_SIZE(flex_eu_config_compute_extended);
	1952
	1953	return 0;
	1954	case METRIC_SET_ID_COMPUTE_L3_CACHE:
	1955	dev_priv->perf.oa.n_mux_configs =
	1956	get_compute_l3_cache_mux_config(dev_priv,
	1957	dev_priv->perf.oa.mux_regs,
	1958	dev_priv->perf.oa.mux_regs_lens);
	1959	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1960	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_L3_CACHE\" metric set\n");
	1961
	1962	/* EINVAL because *_register_sysfs already checked this
	1963	* and so it wouldn't have been advertised to userspace and
	1964	* so shouldn't have been requested
	1965	*/
	1966	return -EINVAL;
	1967	}
	1968
	1969	dev_priv->perf.oa.b_counter_regs =
	1970	b_counter_config_compute_l3_cache;
	1971	dev_priv->perf.oa.b_counter_regs_len =
	1972	ARRAY_SIZE(b_counter_config_compute_l3_cache);
	1973
	1974	dev_priv->perf.oa.flex_regs =
	1975	flex_eu_config_compute_l3_cache;
	1976	dev_priv->perf.oa.flex_regs_len =
	1977	ARRAY_SIZE(flex_eu_config_compute_l3_cache);
	1978
	1979	return 0;
	1980	case METRIC_SET_ID_HDC_AND_SF:
	1981	dev_priv->perf.oa.n_mux_configs =
	1982	get_hdc_and_sf_mux_config(dev_priv,
	1983	dev_priv->perf.oa.mux_regs,
	1984	dev_priv->perf.oa.mux_regs_lens);
	1985	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1986	DRM_DEBUG_DRIVER("No suitable MUX config for \"HDC_AND_SF\" metric set\n");
	1987
	1988	/* EINVAL because *_register_sysfs already checked this
	1989	* and so it wouldn't have been advertised to userspace and
	1990	* so shouldn't have been requested
	1991	*/
	1992	return -EINVAL;
	1993	}
	1994
	1995	dev_priv->perf.oa.b_counter_regs =
	1996	b_counter_config_hdc_and_sf;
	1997	dev_priv->perf.oa.b_counter_regs_len =
	1998	ARRAY_SIZE(b_counter_config_hdc_and_sf);
	1999
	2000	dev_priv->perf.oa.flex_regs =
	2001	flex_eu_config_hdc_and_sf;
	2002	dev_priv->perf.oa.flex_regs_len =
	2003	ARRAY_SIZE(flex_eu_config_hdc_and_sf);
	2004
	2005	return 0;
	2006	case METRIC_SET_ID_L3_1:
	2007	dev_priv->perf.oa.n_mux_configs =
	2008	get_l3_1_mux_config(dev_priv,
	2009	dev_priv->perf.oa.mux_regs,
	2010	dev_priv->perf.oa.mux_regs_lens);
	2011	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2012	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_1\" metric set\n");
	2013
	2014	/* EINVAL because *_register_sysfs already checked this
	2015	* and so it wouldn't have been advertised to userspace and
	2016	* so shouldn't have been requested
	2017	*/
	2018	return -EINVAL;
	2019	}
	2020
	2021	dev_priv->perf.oa.b_counter_regs =
	2022	b_counter_config_l3_1;
	2023	dev_priv->perf.oa.b_counter_regs_len =
	2024	ARRAY_SIZE(b_counter_config_l3_1);
	2025
	2026	dev_priv->perf.oa.flex_regs =
	2027	flex_eu_config_l3_1;
	2028	dev_priv->perf.oa.flex_regs_len =
	2029	ARRAY_SIZE(flex_eu_config_l3_1);
	2030
	2031	return 0;
	2032	case METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND:
	2033	dev_priv->perf.oa.n_mux_configs =
	2034	get_rasterizer_and_pixel_backend_mux_config(dev_priv,
	2035	dev_priv->perf.oa.mux_regs,
	2036	dev_priv->perf.oa.mux_regs_lens);
	2037	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2038	DRM_DEBUG_DRIVER("No suitable MUX config for \"RASTERIZER_AND_PIXEL_BACKEND\" metric set\n");
	2039
	2040	/* EINVAL because *_register_sysfs already checked this
	2041	* and so it wouldn't have been advertised to userspace and
	2042	* so shouldn't have been requested
	2043	*/
	2044	return -EINVAL;
	2045	}
	2046
	2047	dev_priv->perf.oa.b_counter_regs =
	2048	b_counter_config_rasterizer_and_pixel_backend;
	2049	dev_priv->perf.oa.b_counter_regs_len =
	2050	ARRAY_SIZE(b_counter_config_rasterizer_and_pixel_backend);
	2051
	2052	dev_priv->perf.oa.flex_regs =
	2053	flex_eu_config_rasterizer_and_pixel_backend;
	2054	dev_priv->perf.oa.flex_regs_len =
	2055	ARRAY_SIZE(flex_eu_config_rasterizer_and_pixel_backend);
	2056
	2057	return 0;
	2058	case METRIC_SET_ID_SAMPLER:
	2059	dev_priv->perf.oa.n_mux_configs =
	2060	get_sampler_mux_config(dev_priv,
	2061	dev_priv->perf.oa.mux_regs,
	2062	dev_priv->perf.oa.mux_regs_lens);
	2063	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2064	DRM_DEBUG_DRIVER("No suitable MUX config for \"SAMPLER\" metric set\n");
	2065
	2066	/* EINVAL because *_register_sysfs already checked this
	2067	* and so it wouldn't have been advertised to userspace and
	2068	* so shouldn't have been requested
	2069	*/
	2070	return -EINVAL;
	2071	}
	2072
	2073	dev_priv->perf.oa.b_counter_regs =
	2074	b_counter_config_sampler;
	2075	dev_priv->perf.oa.b_counter_regs_len =
	2076	ARRAY_SIZE(b_counter_config_sampler);
	2077
	2078	dev_priv->perf.oa.flex_regs =
	2079	flex_eu_config_sampler;
	2080	dev_priv->perf.oa.flex_regs_len =
	2081	ARRAY_SIZE(flex_eu_config_sampler);
	2082
	2083	return 0;
	2084	case METRIC_SET_ID_TDL_1:
	2085	dev_priv->perf.oa.n_mux_configs =
	2086	get_tdl_1_mux_config(dev_priv,
	2087	dev_priv->perf.oa.mux_regs,
	2088	dev_priv->perf.oa.mux_regs_lens);
	2089	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2090	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_1\" metric set\n");
	2091
	2092	/* EINVAL because *_register_sysfs already checked this
	2093	* and so it wouldn't have been advertised to userspace and
	2094	* so shouldn't have been requested
	2095	*/
	2096	return -EINVAL;
	2097	}
	2098
	2099	dev_priv->perf.oa.b_counter_regs =
	2100	b_counter_config_tdl_1;
	2101	dev_priv->perf.oa.b_counter_regs_len =
	2102	ARRAY_SIZE(b_counter_config_tdl_1);
	2103
	2104	dev_priv->perf.oa.flex_regs =
	2105	flex_eu_config_tdl_1;
	2106	dev_priv->perf.oa.flex_regs_len =
	2107	ARRAY_SIZE(flex_eu_config_tdl_1);
	2108
	2109	return 0;
	2110	case METRIC_SET_ID_TDL_2:
	2111	dev_priv->perf.oa.n_mux_configs =
	2112	get_tdl_2_mux_config(dev_priv,
	2113	dev_priv->perf.oa.mux_regs,
	2114	dev_priv->perf.oa.mux_regs_lens);
	2115	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2116	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_2\" metric set\n");
	2117
	2118	/* EINVAL because *_register_sysfs already checked this
	2119	* and so it wouldn't have been advertised to userspace and
	2120	* so shouldn't have been requested
	2121	*/
	2122	return -EINVAL;
	2123	}
	2124
	2125	dev_priv->perf.oa.b_counter_regs =
	2126	b_counter_config_tdl_2;
	2127	dev_priv->perf.oa.b_counter_regs_len =
	2128	ARRAY_SIZE(b_counter_config_tdl_2);
	2129
	2130	dev_priv->perf.oa.flex_regs =
	2131	flex_eu_config_tdl_2;
	2132	dev_priv->perf.oa.flex_regs_len =
	2133	ARRAY_SIZE(flex_eu_config_tdl_2);
	2134
	2135	return 0;
	2136	case METRIC_SET_ID_COMPUTE_EXTRA:
	2137	dev_priv->perf.oa.n_mux_configs =
	2138	get_compute_extra_mux_config(dev_priv,
	2139	dev_priv->perf.oa.mux_regs,
	2140	dev_priv->perf.oa.mux_regs_lens);
	2141	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2142	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_EXTRA\" metric set\n");
	2143
	2144	/* EINVAL because *_register_sysfs already checked this
	2145	* and so it wouldn't have been advertised to userspace and
	2146	* so shouldn't have been requested
	2147	*/
	2148	return -EINVAL;
	2149	}
	2150
	2151	dev_priv->perf.oa.b_counter_regs =
	2152	b_counter_config_compute_extra;
	2153	dev_priv->perf.oa.b_counter_regs_len =
	2154	ARRAY_SIZE(b_counter_config_compute_extra);
	2155
	2156	dev_priv->perf.oa.flex_regs =
	2157	flex_eu_config_compute_extra;
	2158	dev_priv->perf.oa.flex_regs_len =
	2159	ARRAY_SIZE(flex_eu_config_compute_extra);
	2160
	2161	return 0;
	2162	case METRIC_SET_ID_TEST_OA:
	2163	dev_priv->perf.oa.n_mux_configs =
	2164	get_test_oa_mux_config(dev_priv,
	2165	dev_priv->perf.oa.mux_regs,
	2166	dev_priv->perf.oa.mux_regs_lens);
	2167	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2168	DRM_DEBUG_DRIVER("No suitable MUX config for \"TEST_OA\" metric set\n");
	2169
	2170	/* EINVAL because *_register_sysfs already checked this
	2171	* and so it wouldn't have been advertised to userspace and
	2172	* so shouldn't have been requested
	2173	*/
	2174	return -EINVAL;
	2175	}
	2176
	2177	dev_priv->perf.oa.b_counter_regs =
	2178	b_counter_config_test_oa;
	2179	dev_priv->perf.oa.b_counter_regs_len =
	2180	ARRAY_SIZE(b_counter_config_test_oa);
	2181
	2182	dev_priv->perf.oa.flex_regs =
	2183	flex_eu_config_test_oa;
	2184	dev_priv->perf.oa.flex_regs_len =
	2185	ARRAY_SIZE(flex_eu_config_test_oa);
	2186
	2187	return 0;
	2188	default:
	2189	return -ENODEV;
	2190	}
	2191	}
	2192
	2193	static ssize_t
	2194	show_render_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	2195	{
	2196	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_BASIC);
	2197	}
	2198
	2199	static struct device_attribute dev_attr_render_basic_id = {
	2200	.attr = { .name = "id", .mode = 0444 },
	2201	.show = show_render_basic_id,
	2202	.store = NULL,
	2203	};
	2204
	2205	static struct attribute *attrs_render_basic[] = {
	2206	&dev_attr_render_basic_id.attr,
	2207	NULL,
	2208	};
	2209
	2210	static struct attribute_group group_render_basic = {
	2211	.name = "22b9519a-e9ba-4c41-8b54-f4f8ca14fa0a",
	2212	.attrs = attrs_render_basic,
	2213	};
	2214
	2215	static ssize_t
	2216	show_compute_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	2217	{
	2218	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_BASIC);
	2219	}
	2220
	2221	static struct device_attribute dev_attr_compute_basic_id = {
	2222	.attr = { .name = "id", .mode = 0444 },
	2223	.show = show_compute_basic_id,
	2224	.store = NULL,
	2225	};
	2226
	2227	static struct attribute *attrs_compute_basic[] = {
	2228	&dev_attr_compute_basic_id.attr,
	2229	NULL,
	2230	};
	2231
	2232	static struct attribute_group group_compute_basic = {
	2233	.name = "012d72cf-82a9-4d25-8ddf-74076fd30797",
	2234	.attrs = attrs_compute_basic,
	2235	};
	2236
	2237	static ssize_t
	2238	show_render_pipe_profile_id(struct device kdev, struct device_attribute attr, char *buf)
	2239	{
	2240	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_PIPE_PROFILE);
	2241	}
	2242
	2243	static struct device_attribute dev_attr_render_pipe_profile_id = {
	2244	.attr = { .name = "id", .mode = 0444 },
	2245	.show = show_render_pipe_profile_id,
	2246	.store = NULL,
	2247	};
	2248
	2249	static struct attribute *attrs_render_pipe_profile[] = {
	2250	&dev_attr_render_pipe_profile_id.attr,
	2251	NULL,
	2252	};
	2253
	2254	static struct attribute_group group_render_pipe_profile = {
	2255	.name = "ce416533-e49e-4211-80af-ec513590a914",
	2256	.attrs = attrs_render_pipe_profile,
	2257	};
	2258
	2259	static ssize_t
	2260	show_memory_reads_id(struct device kdev, struct device_attribute attr, char *buf)
	2261	{
	2262	return sprintf(buf, "%d\n", METRIC_SET_ID_MEMORY_READS);
	2263	}
	2264
	2265	static struct device_attribute dev_attr_memory_reads_id = {
	2266	.attr = { .name = "id", .mode = 0444 },
	2267	.show = show_memory_reads_id,
	2268	.store = NULL,
	2269	};
	2270
	2271	static struct attribute *attrs_memory_reads[] = {
	2272	&dev_attr_memory_reads_id.attr,
	2273	NULL,
	2274	};
	2275
	2276	static struct attribute_group group_memory_reads = {
	2277	.name = "398e2452-18d7-42d0-b241-e4d0a9148ada",
	2278	.attrs = attrs_memory_reads,
	2279	};
	2280
	2281	static ssize_t
	2282	show_memory_writes_id(struct device kdev, struct device_attribute attr, char *buf)
	2283	{
	2284	return sprintf(buf, "%d\n", METRIC_SET_ID_MEMORY_WRITES);
	2285	}
	2286
	2287	static struct device_attribute dev_attr_memory_writes_id = {
	2288	.attr = { .name = "id", .mode = 0444 },
	2289	.show = show_memory_writes_id,
	2290	.store = NULL,
	2291	};
	2292
	2293	static struct attribute *attrs_memory_writes[] = {
	2294	&dev_attr_memory_writes_id.attr,
	2295	NULL,
	2296	};
	2297
	2298	static struct attribute_group group_memory_writes = {
	2299	.name = "d324a0d6-7269-4847-a5c2-6f71ddc7fed5",
	2300	.attrs = attrs_memory_writes,
	2301	};
	2302
	2303	static ssize_t
	2304	show_compute_extended_id(struct device kdev, struct device_attribute attr, char *buf)
	2305	{
	2306	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_EXTENDED);
	2307	}
	2308
	2309	static struct device_attribute dev_attr_compute_extended_id = {
	2310	.attr = { .name = "id", .mode = 0444 },
	2311	.show = show_compute_extended_id,
	2312	.store = NULL,
	2313	};
	2314
	2315	static struct attribute *attrs_compute_extended[] = {
	2316	&dev_attr_compute_extended_id.attr,
	2317	NULL,
	2318	};
	2319
	2320	static struct attribute_group group_compute_extended = {
	2321	.name = "caf3596a-7bb1-4dec-b3b3-2a080d283b49",
	2322	.attrs = attrs_compute_extended,
	2323	};
	2324
	2325	static ssize_t
	2326	show_compute_l3_cache_id(struct device kdev, struct device_attribute attr, char *buf)
	2327	{
	2328	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_L3_CACHE);
	2329	}
	2330
	2331	static struct device_attribute dev_attr_compute_l3_cache_id = {
	2332	.attr = { .name = "id", .mode = 0444 },
	2333	.show = show_compute_l3_cache_id,
	2334	.store = NULL,
	2335	};
	2336
	2337	static struct attribute *attrs_compute_l3_cache[] = {
	2338	&dev_attr_compute_l3_cache_id.attr,
	2339	NULL,
	2340	};
	2341
	2342	static struct attribute_group group_compute_l3_cache = {
	2343	.name = "49b956e2-d5b9-47e0-9d8a-cee5e8cec527",
	2344	.attrs = attrs_compute_l3_cache,
	2345	};
	2346
	2347	static ssize_t
	2348	show_hdc_and_sf_id(struct device kdev, struct device_attribute attr, char *buf)
	2349	{
	2350	return sprintf(buf, "%d\n", METRIC_SET_ID_HDC_AND_SF);
	2351	}
	2352
	2353	static struct device_attribute dev_attr_hdc_and_sf_id = {
	2354	.attr = { .name = "id", .mode = 0444 },
	2355	.show = show_hdc_and_sf_id,
	2356	.store = NULL,
	2357	};
	2358
	2359	static struct attribute *attrs_hdc_and_sf[] = {
	2360	&dev_attr_hdc_and_sf_id.attr,
	2361	NULL,
	2362	};
	2363
	2364	static struct attribute_group group_hdc_and_sf = {
	2365	.name = "f64ef50a-bdba-4b35-8f09-203c13d8ee5a",
	2366	.attrs = attrs_hdc_and_sf,
	2367	};
	2368
	2369	static ssize_t
	2370	show_l3_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2371	{
	2372	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_1);
	2373	}
	2374
	2375	static struct device_attribute dev_attr_l3_1_id = {
	2376	.attr = { .name = "id", .mode = 0444 },
	2377	.show = show_l3_1_id,
	2378	.store = NULL,
	2379	};
	2380
	2381	static struct attribute *attrs_l3_1[] = {
	2382	&dev_attr_l3_1_id.attr,
	2383	NULL,
	2384	};
	2385
	2386	static struct attribute_group group_l3_1 = {
	2387	.name = "00ad5a41-7eab-4f7a-9103-49d411c67219",
	2388	.attrs = attrs_l3_1,
	2389	};
	2390
	2391	static ssize_t
	2392	show_rasterizer_and_pixel_backend_id(struct device kdev, struct device_attribute attr, char *buf)
	2393	{
	2394	return sprintf(buf, "%d\n", METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND);
	2395	}
	2396
	2397	static struct device_attribute dev_attr_rasterizer_and_pixel_backend_id = {
	2398	.attr = { .name = "id", .mode = 0444 },
	2399	.show = show_rasterizer_and_pixel_backend_id,
	2400	.store = NULL,
	2401	};
	2402
	2403	static struct attribute *attrs_rasterizer_and_pixel_backend[] = {
	2404	&dev_attr_rasterizer_and_pixel_backend_id.attr,
	2405	NULL,
	2406	};
	2407
	2408	static struct attribute_group group_rasterizer_and_pixel_backend = {
	2409	.name = "46dc44ca-491c-4cc1-a951-e7b3e62bf02b",
	2410	.attrs = attrs_rasterizer_and_pixel_backend,
	2411	};
	2412
	2413	static ssize_t
	2414	show_sampler_id(struct device kdev, struct device_attribute attr, char *buf)
	2415	{
	2416	return sprintf(buf, "%d\n", METRIC_SET_ID_SAMPLER);
	2417	}
	2418
	2419	static struct device_attribute dev_attr_sampler_id = {
	2420	.attr = { .name = "id", .mode = 0444 },
	2421	.show = show_sampler_id,
	2422	.store = NULL,
	2423	};
	2424
	2425	static struct attribute *attrs_sampler[] = {
	2426	&dev_attr_sampler_id.attr,
	2427	NULL,
	2428	};
	2429
	2430	static struct attribute_group group_sampler = {
	2431	.name = "8364e2a8-af63-40af-b0d5-42969a255654",
	2432	.attrs = attrs_sampler,
	2433	};
	2434
	2435	static ssize_t
	2436	show_tdl_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2437	{
	2438	return sprintf(buf, "%d\n", METRIC_SET_ID_TDL_1);
	2439	}
	2440
	2441	static struct device_attribute dev_attr_tdl_1_id = {
	2442	.attr = { .name = "id", .mode = 0444 },
	2443	.show = show_tdl_1_id,
	2444	.store = NULL,
	2445	};
	2446
	2447	static struct attribute *attrs_tdl_1[] = {
	2448	&dev_attr_tdl_1_id.attr,
	2449	NULL,
	2450	};
	2451
	2452	static struct attribute_group group_tdl_1 = {
	2453	.name = "175c8092-cb25-4d1e-8dc7-b4fdd39e2d92",
	2454	.attrs = attrs_tdl_1,
	2455	};
	2456
	2457	static ssize_t
	2458	show_tdl_2_id(struct device kdev, struct device_attribute attr, char *buf)
	2459	{
	2460	return sprintf(buf, "%d\n", METRIC_SET_ID_TDL_2);
	2461	}
	2462
	2463	static struct device_attribute dev_attr_tdl_2_id = {
	2464	.attr = { .name = "id", .mode = 0444 },
	2465	.show = show_tdl_2_id,
	2466	.store = NULL,
	2467	};
	2468
	2469	static struct attribute *attrs_tdl_2[] = {
	2470	&dev_attr_tdl_2_id.attr,
	2471	NULL,
	2472	};
	2473
	2474	static struct attribute_group group_tdl_2 = {
	2475	.name = "d260f03f-b34d-4b49-a44e-436819117332",
	2476	.attrs = attrs_tdl_2,
	2477	};
	2478
	2479	static ssize_t
	2480	show_compute_extra_id(struct device kdev, struct device_attribute attr, char *buf)
	2481	{
	2482	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_EXTRA);
	2483	}
	2484
	2485	static struct device_attribute dev_attr_compute_extra_id = {
	2486	.attr = { .name = "id", .mode = 0444 },
	2487	.show = show_compute_extra_id,
	2488	.store = NULL,
	2489	};
	2490
	2491	static struct attribute *attrs_compute_extra[] = {
	2492	&dev_attr_compute_extra_id.attr,
	2493	NULL,
	2494	};
	2495
	2496	static struct attribute_group group_compute_extra = {
	2497	.name = "fa6ecf21-2cb8-4d0b-9308-6e4a7b4ca87a",
	2498	.attrs = attrs_compute_extra,
	2499	};
	2500
	2501	static ssize_t
	2502	show_test_oa_id(struct device kdev, struct device_attribute attr, char *buf)
	2503	{
	2504	return sprintf(buf, "%d\n", METRIC_SET_ID_TEST_OA);
	2505	}
	2506
	2507	static struct device_attribute dev_attr_test_oa_id = {
	2508	.attr = { .name = "id", .mode = 0444 },
	2509	.show = show_test_oa_id,
	2510	.store = NULL,
	2511	};
	2512
	2513	static struct attribute *attrs_test_oa[] = {
	2514	&dev_attr_test_oa_id.attr,
	2515	NULL,
	2516	};
	2517
	2518	static struct attribute_group group_test_oa = {
	2519	.name = "5ee72f5c-092f-421e-8b70-225f7c3e9612",
	2520	.attrs = attrs_test_oa,
	2521	};
	2522
	2523	int
	2524	i915_perf_register_sysfs_bxt(struct drm_i915_private *dev_priv)
	2525	{
	2526	const struct i915_oa_reg *mux_regs[ARRAY_SIZE(dev_priv->perf.oa.mux_regs)];
	2527	int mux_lens[ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens)];
	2528	int ret = 0;
	2529
	2530	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens)) {
	2531	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2532	if (ret)
	2533	goto error_render_basic;
	2534	}
	2535	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens)) {
	2536	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2537	if (ret)
	2538	goto error_compute_basic;
	2539	}
	2540	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens)) {
	2541	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2542	if (ret)
	2543	goto error_render_pipe_profile;
	2544	}
	2545	if (get_memory_reads_mux_config(dev_priv, mux_regs, mux_lens)) {
	2546	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_memory_reads);
	2547	if (ret)
	2548	goto error_memory_reads;
	2549	}
	2550	if (get_memory_writes_mux_config(dev_priv, mux_regs, mux_lens)) {
	2551	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_memory_writes);
	2552	if (ret)
	2553	goto error_memory_writes;
	2554	}
	2555	if (get_compute_extended_mux_config(dev_priv, mux_regs, mux_lens)) {
	2556	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_extended);
	2557	if (ret)
	2558	goto error_compute_extended;
	2559	}
	2560	if (get_compute_l3_cache_mux_config(dev_priv, mux_regs, mux_lens)) {
	2561	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_l3_cache);
	2562	if (ret)
	2563	goto error_compute_l3_cache;
	2564	}
	2565	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens)) {
	2566	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2567	if (ret)
	2568	goto error_hdc_and_sf;
	2569	}
	2570	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2571	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2572	if (ret)
	2573	goto error_l3_1;
	2574	}
	2575	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens)) {
	2576	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2577	if (ret)
	2578	goto error_rasterizer_and_pixel_backend;
	2579	}
	2580	if (get_sampler_mux_config(dev_priv, mux_regs, mux_lens)) {
	2581	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_sampler);
	2582	if (ret)
	2583	goto error_sampler;
	2584	}
	2585	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2586	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2587	if (ret)
	2588	goto error_tdl_1;
	2589	}
	2590	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens)) {
	2591	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2592	if (ret)
	2593	goto error_tdl_2;
	2594	}
	2595	if (get_compute_extra_mux_config(dev_priv, mux_regs, mux_lens)) {
	2596	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_extra);
	2597	if (ret)
	2598	goto error_compute_extra;
	2599	}
	2600	if (get_test_oa_mux_config(dev_priv, mux_regs, mux_lens)) {
	2601	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_test_oa);
	2602	if (ret)
	2603	goto error_test_oa;
	2604	}
	2605
	2606	return 0;
	2607
	2608	error_test_oa:
	2609	if (get_compute_extra_mux_config(dev_priv, mux_regs, mux_lens))
	2610	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_extra);
	2611	error_compute_extra:
	2612	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens))
	2613	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2614	error_tdl_2:
	2615	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens))
	2616	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2617	error_tdl_1:
	2618	if (get_sampler_mux_config(dev_priv, mux_regs, mux_lens))
	2619	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_sampler);
	2620	error_sampler:
	2621	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens))
	2622	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2623	error_rasterizer_and_pixel_backend:
	2624	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens))
	2625	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2626	error_l3_1:
	2627	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens))
	2628	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2629	error_hdc_and_sf:
	2630	if (get_compute_l3_cache_mux_config(dev_priv, mux_regs, mux_lens))
	2631	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_l3_cache);
	2632	error_compute_l3_cache:
	2633	if (get_compute_extended_mux_config(dev_priv, mux_regs, mux_lens))
	2634	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_extended);
	2635	error_compute_extended:
	2636	if (get_memory_writes_mux_config(dev_priv, mux_regs, mux_lens))
	2637	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_memory_writes);
	2638	error_memory_writes:
	2639	if (get_memory_reads_mux_config(dev_priv, mux_regs, mux_lens))
	2640	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_memory_reads);
	2641	error_memory_reads:
	2642	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens))
	2643	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2644	error_render_pipe_profile:
	2645	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2646	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2647	error_compute_basic:
	2648	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2649	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2650	error_render_basic:
	2651	return ret;
	2652	}
	2653
	2654	void
	2655	i915_perf_unregister_sysfs_bxt(struct drm_i915_private *dev_priv)
	2656	{
	2657	const struct i915_oa_reg *mux_regs[ARRAY_SIZE(dev_priv->perf.oa.mux_regs)];
	2658	int mux_lens[ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens)];
	2659
	2660	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2661	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2662	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2663	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2664	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens))
	2665	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2666	if (get_memory_reads_mux_config(dev_priv, mux_regs, mux_lens))
	2667	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_memory_reads);
	2668	if (get_memory_writes_mux_config(dev_priv, mux_regs, mux_lens))
	2669	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_memory_writes);
	2670	if (get_compute_extended_mux_config(dev_priv, mux_regs, mux_lens))
	2671	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_extended);
	2672	if (get_compute_l3_cache_mux_config(dev_priv, mux_regs, mux_lens))
	2673	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_l3_cache);
	2674	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens))
	2675	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2676	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens))
	2677	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2678	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens))
	2679	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2680	if (get_sampler_mux_config(dev_priv, mux_regs, mux_lens))
	2681	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_sampler);
	2682	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens))
	2683	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2684	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens))
	2685	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2686	if (get_compute_extra_mux_config(dev_priv, mux_regs, mux_lens))
	2687	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_extra);
	2688	if (get_test_oa_mux_config(dev_priv, mux_regs, mux_lens))
	2689	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_test_oa);
	2690	}

drivers/gpu/drm/i915/i915_oa_bxt.h+40−0

@@ -0,0 +1,40 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#ifndef __I915_OA_BXT_H__
	30	#define __I915_OA_BXT_H__
	31
	32	extern int i915_oa_n_builtin_metric_sets_bxt;
	33
	34	extern int i915_oa_select_metric_set_bxt(struct drm_i915_private *dev_priv);
	35
	36	extern int i915_perf_register_sysfs_bxt(struct drm_i915_private *dev_priv);
	37
	38	extern void i915_perf_unregister_sysfs_bxt(struct drm_i915_private *dev_priv);
	39
	40	#endif

drivers/gpu/drm/i915/i915_oa_chv.c+2873−0

@@ -0,0 +1,2873 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#include <linux/sysfs.h>
	30
	31	#include "i915_drv.h"
	32	#include "i915_oa_chv.h"
	33
	34	enum metric_set_id {
	35	METRIC_SET_ID_RENDER_BASIC = 1,
	36	METRIC_SET_ID_COMPUTE_BASIC,
	37	METRIC_SET_ID_RENDER_PIPE_PROFILE,
	38	METRIC_SET_ID_HDC_AND_SF,
	39	METRIC_SET_ID_L3_1,
	40	METRIC_SET_ID_L3_2,
	41	METRIC_SET_ID_L3_3,
	42	METRIC_SET_ID_L3_4,
	43	METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND,
	44	METRIC_SET_ID_SAMPLER_1,
	45	METRIC_SET_ID_SAMPLER_2,
	46	METRIC_SET_ID_TDL_1,
	47	METRIC_SET_ID_TDL_2,
	48	METRIC_SET_ID_TEST_OA,
	49	};
	50
	51	int i915_oa_n_builtin_metric_sets_chv = 14;
	52
	53	static const struct i915_oa_reg b_counter_config_render_basic[] = {
	54	{ _MMIO(0x2740), 0x00000000 },
	55	{ _MMIO(0x2710), 0x00000000 },
	56	{ _MMIO(0x2714), 0x00800000 },
	57	{ _MMIO(0x2720), 0x00000000 },
	58	{ _MMIO(0x2724), 0x00800000 },
	59	};
	60
	61	static const struct i915_oa_reg flex_eu_config_render_basic[] = {
	62	{ _MMIO(0xe458), 0x00005004 },
	63	{ _MMIO(0xe558), 0x00010003 },
	64	{ _MMIO(0xe658), 0x00012011 },
	65	{ _MMIO(0xe758), 0x00015014 },
	66	{ _MMIO(0xe45c), 0x00051050 },
	67	{ _MMIO(0xe55c), 0x00053052 },
	68	{ _MMIO(0xe65c), 0x00055054 },
	69	};
	70
	71	static const struct i915_oa_reg mux_config_render_basic[] = {
	72	{ _MMIO(0x9888), 0x59800000 },
	73	{ _MMIO(0x9888), 0x59800001 },
	74	{ _MMIO(0x9888), 0x285a0006 },
	75	{ _MMIO(0x9888), 0x2c110014 },
	76	{ _MMIO(0x9888), 0x2e110000 },
	77	{ _MMIO(0x9888), 0x2c310014 },
	78	{ _MMIO(0x9888), 0x2e310000 },
	79	{ _MMIO(0x9888), 0x2b8303df },
	80	{ _MMIO(0x9888), 0x3580024f },
	81	{ _MMIO(0x9888), 0x00580888 },
	82	{ _MMIO(0x9888), 0x1e5a0015 },
	83	{ _MMIO(0x9888), 0x205a0014 },
	84	{ _MMIO(0x9888), 0x045a0000 },
	85	{ _MMIO(0x9888), 0x025a0000 },
	86	{ _MMIO(0x9888), 0x02180500 },
	87	{ _MMIO(0x9888), 0x00190555 },
	88	{ _MMIO(0x9888), 0x021d0500 },
	89	{ _MMIO(0x9888), 0x021f0a00 },
	90	{ _MMIO(0x9888), 0x00380444 },
	91	{ _MMIO(0x9888), 0x02390500 },
	92	{ _MMIO(0x9888), 0x003a0666 },
	93	{ _MMIO(0x9888), 0x00100111 },
	94	{ _MMIO(0x9888), 0x06110030 },
	95	{ _MMIO(0x9888), 0x0a110031 },
	96	{ _MMIO(0x9888), 0x0e110046 },
	97	{ _MMIO(0x9888), 0x04110000 },
	98	{ _MMIO(0x9888), 0x00110000 },
	99	{ _MMIO(0x9888), 0x00130111 },
	100	{ _MMIO(0x9888), 0x00300444 },
	101	{ _MMIO(0x9888), 0x08310030 },
	102	{ _MMIO(0x9888), 0x0c310031 },
	103	{ _MMIO(0x9888), 0x10310046 },
	104	{ _MMIO(0x9888), 0x04310000 },
	105	{ _MMIO(0x9888), 0x00310000 },
	106	{ _MMIO(0x9888), 0x00330444 },
	107	{ _MMIO(0x9888), 0x038a0a00 },
	108	{ _MMIO(0x9888), 0x018b0fff },
	109	{ _MMIO(0x9888), 0x038b0a00 },
	110	{ _MMIO(0x9888), 0x01855000 },
	111	{ _MMIO(0x9888), 0x03850055 },
	112	{ _MMIO(0x9888), 0x13830021 },
	113	{ _MMIO(0x9888), 0x15830020 },
	114	{ _MMIO(0x9888), 0x1783002f },
	115	{ _MMIO(0x9888), 0x1983002e },
	116	{ _MMIO(0x9888), 0x1b83002d },
	117	{ _MMIO(0x9888), 0x1d83002c },
	118	{ _MMIO(0x9888), 0x05830000 },
	119	{ _MMIO(0x9888), 0x01840555 },
	120	{ _MMIO(0x9888), 0x03840500 },
	121	{ _MMIO(0x9888), 0x23800074 },
	122	{ _MMIO(0x9888), 0x2580007d },
	123	{ _MMIO(0x9888), 0x05800000 },
	124	{ _MMIO(0x9888), 0x01805000 },
	125	{ _MMIO(0x9888), 0x03800055 },
	126	{ _MMIO(0x9888), 0x01865000 },
	127	{ _MMIO(0x9888), 0x03860055 },
	128	{ _MMIO(0x9888), 0x01875000 },
	129	{ _MMIO(0x9888), 0x03870055 },
	130	{ _MMIO(0x9888), 0x418000aa },
	131	{ _MMIO(0x9888), 0x4380000a },
	132	{ _MMIO(0x9888), 0x45800000 },
	133	{ _MMIO(0x9888), 0x4780000a },
	134	{ _MMIO(0x9888), 0x49800000 },
	135	{ _MMIO(0x9888), 0x4b800000 },
	136	{ _MMIO(0x9888), 0x4d800000 },
	137	{ _MMIO(0x9888), 0x4f800000 },
	138	{ _MMIO(0x9888), 0x51800000 },
	139	{ _MMIO(0x9888), 0x53800000 },
	140	{ _MMIO(0x9888), 0x55800000 },
	141	{ _MMIO(0x9888), 0x57800000 },
	142	{ _MMIO(0x9888), 0x59800000 },
	143	};
	144
	145	static int
	146	get_render_basic_mux_config(struct drm_i915_private *dev_priv,
	147	const struct i915_oa_reg **regs,
	148	int *lens)
	149	{
	150	int n = 0;
	151
	152	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	153	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	154
	155	regs[n] = mux_config_render_basic;
	156	lens[n] = ARRAY_SIZE(mux_config_render_basic);
	157	n++;
	158
	159	return n;
	160	}
	161
	162	static const struct i915_oa_reg b_counter_config_compute_basic[] = {
	163	{ _MMIO(0x2710), 0x00000000 },
	164	{ _MMIO(0x2714), 0x00800000 },
	165	{ _MMIO(0x2720), 0x00000000 },
	166	{ _MMIO(0x2724), 0x00800000 },
	167	};
	168
	169	static const struct i915_oa_reg flex_eu_config_compute_basic[] = {
	170	{ _MMIO(0xe458), 0x00005004 },
	171	{ _MMIO(0xe558), 0x00000003 },
	172	{ _MMIO(0xe658), 0x00002001 },
	173	{ _MMIO(0xe758), 0x00778008 },
	174	{ _MMIO(0xe45c), 0x00088078 },
	175	{ _MMIO(0xe55c), 0x00808708 },
	176	{ _MMIO(0xe65c), 0x00a08908 },
	177	};
	178
	179	static const struct i915_oa_reg mux_config_compute_basic[] = {
	180	{ _MMIO(0x9888), 0x59800000 },
	181	{ _MMIO(0x9888), 0x59800001 },
	182	{ _MMIO(0x9888), 0x2e5800e0 },
	183	{ _MMIO(0x9888), 0x2e3800e0 },
	184	{ _MMIO(0x9888), 0x3580024f },
	185	{ _MMIO(0x9888), 0x3d800140 },
	186	{ _MMIO(0x9888), 0x08580042 },
	187	{ _MMIO(0x9888), 0x0c580040 },
	188	{ _MMIO(0x9888), 0x1058004c },
	189	{ _MMIO(0x9888), 0x1458004b },
	190	{ _MMIO(0x9888), 0x04580000 },
	191	{ _MMIO(0x9888), 0x00580000 },
	192	{ _MMIO(0x9888), 0x00195555 },
	193	{ _MMIO(0x9888), 0x06380042 },
	194	{ _MMIO(0x9888), 0x0a380040 },
	195	{ _MMIO(0x9888), 0x0e38004c },
	196	{ _MMIO(0x9888), 0x1238004b },
	197	{ _MMIO(0x9888), 0x04380000 },
	198	{ _MMIO(0x9888), 0x00384444 },
	199	{ _MMIO(0x9888), 0x003a5555 },
	200	{ _MMIO(0x9888), 0x018bffff },
	201	{ _MMIO(0x9888), 0x01845555 },
	202	{ _MMIO(0x9888), 0x17800074 },
	203	{ _MMIO(0x9888), 0x1980007d },
	204	{ _MMIO(0x9888), 0x1b80007c },
	205	{ _MMIO(0x9888), 0x1d8000b6 },
	206	{ _MMIO(0x9888), 0x1f8000b7 },
	207	{ _MMIO(0x9888), 0x05800000 },
	208	{ _MMIO(0x9888), 0x03800000 },
	209	{ _MMIO(0x9888), 0x418000aa },
	210	{ _MMIO(0x9888), 0x438000aa },
	211	{ _MMIO(0x9888), 0x45800000 },
	212	{ _MMIO(0x9888), 0x47800000 },
	213	{ _MMIO(0x9888), 0x4980012a },
	214	{ _MMIO(0x9888), 0x4b80012a },
	215	{ _MMIO(0x9888), 0x4d80012a },
	216	{ _MMIO(0x9888), 0x4f80012a },
	217	{ _MMIO(0x9888), 0x518001ce },
	218	{ _MMIO(0x9888), 0x538001ce },
	219	{ _MMIO(0x9888), 0x5580000e },
	220	{ _MMIO(0x9888), 0x59800000 },
	221	};
	222
	223	static int
	224	get_compute_basic_mux_config(struct drm_i915_private *dev_priv,
	225	const struct i915_oa_reg **regs,
	226	int *lens)
	227	{
	228	int n = 0;
	229
	230	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	231	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	232
	233	regs[n] = mux_config_compute_basic;
	234	lens[n] = ARRAY_SIZE(mux_config_compute_basic);
	235	n++;
	236
	237	return n;
	238	}
	239
	240	static const struct i915_oa_reg b_counter_config_render_pipe_profile[] = {
	241	{ _MMIO(0x2724), 0xf0800000 },
	242	{ _MMIO(0x2720), 0x00000000 },
	243	{ _MMIO(0x2714), 0xf0800000 },
	244	{ _MMIO(0x2710), 0x00000000 },
	245	{ _MMIO(0x2770), 0x0007ffea },
	246	{ _MMIO(0x2774), 0x00007ffc },
	247	{ _MMIO(0x2778), 0x0007affa },
	248	{ _MMIO(0x277c), 0x0000f5fd },
	249	{ _MMIO(0x2780), 0x00079ffa },
	250	{ _MMIO(0x2784), 0x0000f3fb },
	251	{ _MMIO(0x2788), 0x0007bf7a },
	252	{ _MMIO(0x278c), 0x0000f7e7 },
	253	{ _MMIO(0x2790), 0x0007fefa },
	254	{ _MMIO(0x2794), 0x0000f7cf },
	255	{ _MMIO(0x2798), 0x00077ffa },
	256	{ _MMIO(0x279c), 0x0000efdf },
	257	{ _MMIO(0x27a0), 0x0006fffa },
	258	{ _MMIO(0x27a4), 0x0000cfbf },
	259	{ _MMIO(0x27a8), 0x0003fffa },
	260	{ _MMIO(0x27ac), 0x00005f7f },
	261	};
	262
	263	static const struct i915_oa_reg flex_eu_config_render_pipe_profile[] = {
	264	{ _MMIO(0xe458), 0x00005004 },
	265	{ _MMIO(0xe558), 0x00015014 },
	266	{ _MMIO(0xe658), 0x00025024 },
	267	{ _MMIO(0xe758), 0x00035034 },
	268	{ _MMIO(0xe45c), 0x00045044 },
	269	{ _MMIO(0xe55c), 0x00055054 },
	270	{ _MMIO(0xe65c), 0x00065064 },
	271	};
	272
	273	static const struct i915_oa_reg mux_config_render_pipe_profile[] = {
	274	{ _MMIO(0x9888), 0x59800000 },
	275	{ _MMIO(0x9888), 0x59800001 },
	276	{ _MMIO(0x9888), 0x261e0000 },
	277	{ _MMIO(0x9888), 0x281f000f },
	278	{ _MMIO(0x9888), 0x2817001a },
	279	{ _MMIO(0x9888), 0x2791001f },
	280	{ _MMIO(0x9888), 0x27880019 },
	281	{ _MMIO(0x9888), 0x2d890000 },
	282	{ _MMIO(0x9888), 0x278a0007 },
	283	{ _MMIO(0x9888), 0x298d001f },
	284	{ _MMIO(0x9888), 0x278e0020 },
	285	{ _MMIO(0x9888), 0x2b8f0012 },
	286	{ _MMIO(0x9888), 0x29900000 },
	287	{ _MMIO(0x9888), 0x00184000 },
	288	{ _MMIO(0x9888), 0x02181000 },
	289	{ _MMIO(0x9888), 0x02194000 },
	290	{ _MMIO(0x9888), 0x141e0002 },
	291	{ _MMIO(0x9888), 0x041e0000 },
	292	{ _MMIO(0x9888), 0x001e0000 },
	293	{ _MMIO(0x9888), 0x221f0015 },
	294	{ _MMIO(0x9888), 0x041f0000 },
	295	{ _MMIO(0x9888), 0x001f4000 },
	296	{ _MMIO(0x9888), 0x021f0000 },
	297	{ _MMIO(0x9888), 0x023a8000 },
	298	{ _MMIO(0x9888), 0x0213c000 },
	299	{ _MMIO(0x9888), 0x02164000 },
	300	{ _MMIO(0x9888), 0x24170012 },
	301	{ _MMIO(0x9888), 0x04170000 },
	302	{ _MMIO(0x9888), 0x07910005 },
	303	{ _MMIO(0x9888), 0x05910000 },
	304	{ _MMIO(0x9888), 0x01911500 },
	305	{ _MMIO(0x9888), 0x03910501 },
	306	{ _MMIO(0x9888), 0x0d880002 },
	307	{ _MMIO(0x9888), 0x1d880003 },
	308	{ _MMIO(0x9888), 0x05880000 },
	309	{ _MMIO(0x9888), 0x0b890032 },
	310	{ _MMIO(0x9888), 0x1b890031 },
	311	{ _MMIO(0x9888), 0x05890000 },
	312	{ _MMIO(0x9888), 0x01890040 },
	313	{ _MMIO(0x9888), 0x03890040 },
	314	{ _MMIO(0x9888), 0x098a0000 },
	315	{ _MMIO(0x9888), 0x198a0004 },
	316	{ _MMIO(0x9888), 0x058a0000 },
	317	{ _MMIO(0x9888), 0x018a8050 },
	318	{ _MMIO(0x9888), 0x038a2050 },
	319	{ _MMIO(0x9888), 0x018b95a9 },
	320	{ _MMIO(0x9888), 0x038be5a9 },
	321	{ _MMIO(0x9888), 0x018c1500 },
	322	{ _MMIO(0x9888), 0x038c0501 },
	323	{ _MMIO(0x9888), 0x178d0015 },
	324	{ _MMIO(0x9888), 0x058d0000 },
	325	{ _MMIO(0x9888), 0x138e0004 },
	326	{ _MMIO(0x9888), 0x218e000c },
	327	{ _MMIO(0x9888), 0x058e0000 },
	328	{ _MMIO(0x9888), 0x018e0500 },
	329	{ _MMIO(0x9888), 0x038e0101 },
	330	{ _MMIO(0x9888), 0x0f8f0027 },
	331	{ _MMIO(0x9888), 0x058f0000 },
	332	{ _MMIO(0x9888), 0x018f0000 },
	333	{ _MMIO(0x9888), 0x038f0001 },
	334	{ _MMIO(0x9888), 0x11900013 },
	335	{ _MMIO(0x9888), 0x1f900017 },
	336	{ _MMIO(0x9888), 0x05900000 },
	337	{ _MMIO(0x9888), 0x01900100 },
	338	{ _MMIO(0x9888), 0x03900001 },
	339	{ _MMIO(0x9888), 0x01845555 },
	340	{ _MMIO(0x9888), 0x03845555 },
	341	{ _MMIO(0x9888), 0x418000aa },
	342	{ _MMIO(0x9888), 0x438000aa },
	343	{ _MMIO(0x9888), 0x458000aa },
	344	{ _MMIO(0x9888), 0x478000aa },
	345	{ _MMIO(0x9888), 0x4980018c },
	346	{ _MMIO(0x9888), 0x4b80014b },
	347	{ _MMIO(0x9888), 0x4d800128 },
	348	{ _MMIO(0x9888), 0x4f80012a },
	349	{ _MMIO(0x9888), 0x51800187 },
	350	{ _MMIO(0x9888), 0x5380014b },
	351	{ _MMIO(0x9888), 0x55800149 },
	352	{ _MMIO(0x9888), 0x5780010a },
	353	{ _MMIO(0x9888), 0x59800000 },
	354	};
	355
	356	static int
	357	get_render_pipe_profile_mux_config(struct drm_i915_private *dev_priv,
	358	const struct i915_oa_reg **regs,
	359	int *lens)
	360	{
	361	int n = 0;
	362
	363	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	364	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	365
	366	regs[n] = mux_config_render_pipe_profile;
	367	lens[n] = ARRAY_SIZE(mux_config_render_pipe_profile);
	368	n++;
	369
	370	return n;
	371	}
	372
	373	static const struct i915_oa_reg b_counter_config_hdc_and_sf[] = {
	374	{ _MMIO(0x2740), 0x00000000 },
	375	{ _MMIO(0x2744), 0x00800000 },
	376	{ _MMIO(0x2710), 0x00000000 },
	377	{ _MMIO(0x2714), 0x10800000 },
	378	{ _MMIO(0x2720), 0x00000000 },
	379	{ _MMIO(0x2724), 0x00800000 },
	380	{ _MMIO(0x2770), 0x00000002 },
	381	{ _MMIO(0x2774), 0x0000fff7 },
	382	};
	383
	384	static const struct i915_oa_reg flex_eu_config_hdc_and_sf[] = {
	385	{ _MMIO(0xe458), 0x00005004 },
	386	{ _MMIO(0xe558), 0x00010003 },
	387	{ _MMIO(0xe658), 0x00012011 },
	388	{ _MMIO(0xe758), 0x00015014 },
	389	{ _MMIO(0xe45c), 0x00051050 },
	390	{ _MMIO(0xe55c), 0x00053052 },
	391	{ _MMIO(0xe65c), 0x00055054 },
	392	};
	393
	394	static const struct i915_oa_reg mux_config_hdc_and_sf[] = {
	395	{ _MMIO(0x9888), 0x105c0232 },
	396	{ _MMIO(0x9888), 0x10580232 },
	397	{ _MMIO(0x9888), 0x10380232 },
	398	{ _MMIO(0x9888), 0x10dc0232 },
	399	{ _MMIO(0x9888), 0x10d80232 },
	400	{ _MMIO(0x9888), 0x10b80232 },
	401	{ _MMIO(0x9888), 0x118e4400 },
	402	{ _MMIO(0x9888), 0x025c6080 },
	403	{ _MMIO(0x9888), 0x045c004b },
	404	{ _MMIO(0x9888), 0x005c8000 },
	405	{ _MMIO(0x9888), 0x00582080 },
	406	{ _MMIO(0x9888), 0x0258004b },
	407	{ _MMIO(0x9888), 0x025b4000 },
	408	{ _MMIO(0x9888), 0x045b4000 },
	409	{ _MMIO(0x9888), 0x0c1fa000 },
	410	{ _MMIO(0x9888), 0x0e1f00aa },
	411	{ _MMIO(0x9888), 0x04386080 },
	412	{ _MMIO(0x9888), 0x0638404b },
	413	{ _MMIO(0x9888), 0x02384000 },
	414	{ _MMIO(0x9888), 0x08384000 },
	415	{ _MMIO(0x9888), 0x0a380000 },
	416	{ _MMIO(0x9888), 0x0c380000 },
	417	{ _MMIO(0x9888), 0x00398000 },
	418	{ _MMIO(0x9888), 0x0239a000 },
	419	{ _MMIO(0x9888), 0x0439a000 },
	420	{ _MMIO(0x9888), 0x06392000 },
	421	{ _MMIO(0x9888), 0x0cdc25c1 },
	422	{ _MMIO(0x9888), 0x0adcc000 },
	423	{ _MMIO(0x9888), 0x0ad825c1 },
	424	{ _MMIO(0x9888), 0x18db4000 },
	425	{ _MMIO(0x9888), 0x1adb0001 },
	426	{ _MMIO(0x9888), 0x0e9f8000 },
	427	{ _MMIO(0x9888), 0x109f02aa },
	428	{ _MMIO(0x9888), 0x0eb825c1 },
	429	{ _MMIO(0x9888), 0x18b80154 },
	430	{ _MMIO(0x9888), 0x0ab9a000 },
	431	{ _MMIO(0x9888), 0x0cb9a000 },
	432	{ _MMIO(0x9888), 0x0eb9a000 },
	433	{ _MMIO(0x9888), 0x0d88c000 },
	434	{ _MMIO(0x9888), 0x0f88000f },
	435	{ _MMIO(0x9888), 0x038a8000 },
	436	{ _MMIO(0x9888), 0x058a8000 },
	437	{ _MMIO(0x9888), 0x078a8000 },
	438	{ _MMIO(0x9888), 0x098a8000 },
	439	{ _MMIO(0x9888), 0x0b8a8000 },
	440	{ _MMIO(0x9888), 0x0d8a8000 },
	441	{ _MMIO(0x9888), 0x258baa05 },
	442	{ _MMIO(0x9888), 0x278b002a },
	443	{ _MMIO(0x9888), 0x238b2a80 },
	444	{ _MMIO(0x9888), 0x198c5400 },
	445	{ _MMIO(0x9888), 0x1b8c0015 },
	446	{ _MMIO(0x9888), 0x098dc000 },
	447	{ _MMIO(0x9888), 0x0b8da000 },
	448	{ _MMIO(0x9888), 0x0d8da000 },
	449	{ _MMIO(0x9888), 0x0f8da000 },
	450	{ _MMIO(0x9888), 0x098e05c0 },
	451	{ _MMIO(0x9888), 0x058e0000 },
	452	{ _MMIO(0x9888), 0x198f0020 },
	453	{ _MMIO(0x9888), 0x2185aa0a },
	454	{ _MMIO(0x9888), 0x2385002a },
	455	{ _MMIO(0x9888), 0x1f85aa00 },
	456	{ _MMIO(0x9888), 0x19835000 },
	457	{ _MMIO(0x9888), 0x1b830155 },
	458	{ _MMIO(0x9888), 0x03834000 },
	459	{ _MMIO(0x9888), 0x05834000 },
	460	{ _MMIO(0x9888), 0x07834000 },
	461	{ _MMIO(0x9888), 0x09834000 },
	462	{ _MMIO(0x9888), 0x0b834000 },
	463	{ _MMIO(0x9888), 0x0d834000 },
	464	{ _MMIO(0x9888), 0x09848000 },
	465	{ _MMIO(0x9888), 0x0b84c000 },
	466	{ _MMIO(0x9888), 0x0d84c000 },
	467	{ _MMIO(0x9888), 0x0f84c000 },
	468	{ _MMIO(0x9888), 0x01848000 },
	469	{ _MMIO(0x9888), 0x0384c000 },
	470	{ _MMIO(0x9888), 0x0584c000 },
	471	{ _MMIO(0x9888), 0x07844000 },
	472	{ _MMIO(0x9888), 0x19808000 },
	473	{ _MMIO(0x9888), 0x1b80c000 },
	474	{ _MMIO(0x9888), 0x1d80c000 },
	475	{ _MMIO(0x9888), 0x1f80c000 },
	476	{ _MMIO(0x9888), 0x11808000 },
	477	{ _MMIO(0x9888), 0x1380c000 },
	478	{ _MMIO(0x9888), 0x1580c000 },
	479	{ _MMIO(0x9888), 0x17804000 },
	480	{ _MMIO(0x9888), 0x51800040 },
	481	{ _MMIO(0x9888), 0x43800400 },
	482	{ _MMIO(0x9888), 0x45800800 },
	483	{ _MMIO(0x9888), 0x53800000 },
	484	{ _MMIO(0x9888), 0x47800c62 },
	485	{ _MMIO(0x9888), 0x21800000 },
	486	{ _MMIO(0x9888), 0x31800000 },
	487	{ _MMIO(0x9888), 0x4d800000 },
	488	{ _MMIO(0x9888), 0x3f801042 },
	489	{ _MMIO(0x9888), 0x4f800000 },
	490	{ _MMIO(0x9888), 0x418014a4 },
	491	};
	492
	493	static int
	494	get_hdc_and_sf_mux_config(struct drm_i915_private *dev_priv,
	495	const struct i915_oa_reg **regs,
	496	int *lens)
	497	{
	498	int n = 0;
	499
	500	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	501	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	502
	503	regs[n] = mux_config_hdc_and_sf;
	504	lens[n] = ARRAY_SIZE(mux_config_hdc_and_sf);
	505	n++;
	506
	507	return n;
	508	}
	509
	510	static const struct i915_oa_reg b_counter_config_l3_1[] = {
	511	{ _MMIO(0x2740), 0x00000000 },
	512	{ _MMIO(0x2744), 0x00800000 },
	513	{ _MMIO(0x2710), 0x00000000 },
	514	{ _MMIO(0x2714), 0xf0800000 },
	515	{ _MMIO(0x2720), 0x00000000 },
	516	{ _MMIO(0x2724), 0xf0800000 },
	517	{ _MMIO(0x2770), 0x00100070 },
	518	{ _MMIO(0x2774), 0x0000fff1 },
	519	{ _MMIO(0x2778), 0x00014002 },
	520	{ _MMIO(0x277c), 0x0000c3ff },
	521	{ _MMIO(0x2780), 0x00010002 },
	522	{ _MMIO(0x2784), 0x0000c7ff },
	523	{ _MMIO(0x2788), 0x00004002 },
	524	{ _MMIO(0x278c), 0x0000d3ff },
	525	{ _MMIO(0x2790), 0x00100700 },
	526	{ _MMIO(0x2794), 0x0000ff1f },
	527	{ _MMIO(0x2798), 0x00001402 },
	528	{ _MMIO(0x279c), 0x0000fc3f },
	529	{ _MMIO(0x27a0), 0x00001002 },
	530	{ _MMIO(0x27a4), 0x0000fc7f },
	531	{ _MMIO(0x27a8), 0x00000402 },
	532	{ _MMIO(0x27ac), 0x0000fd3f },
	533	};
	534
	535	static const struct i915_oa_reg flex_eu_config_l3_1[] = {
	536	{ _MMIO(0xe458), 0x00005004 },
	537	{ _MMIO(0xe558), 0x00010003 },
	538	{ _MMIO(0xe658), 0x00012011 },
	539	{ _MMIO(0xe758), 0x00015014 },
	540	{ _MMIO(0xe45c), 0x00051050 },
	541	{ _MMIO(0xe55c), 0x00053052 },
	542	{ _MMIO(0xe65c), 0x00055054 },
	543	};
	544
	545	static const struct i915_oa_reg mux_config_l3_1[] = {
	546	{ _MMIO(0x9888), 0x10bf03da },
	547	{ _MMIO(0x9888), 0x14bf0001 },
	548	{ _MMIO(0x9888), 0x12980340 },
	549	{ _MMIO(0x9888), 0x12990340 },
	550	{ _MMIO(0x9888), 0x0cbf1187 },
	551	{ _MMIO(0x9888), 0x0ebf1205 },
	552	{ _MMIO(0x9888), 0x00bf0500 },
	553	{ _MMIO(0x9888), 0x02bf042b },
	554	{ _MMIO(0x9888), 0x04bf002c },
	555	{ _MMIO(0x9888), 0x0cdac000 },
	556	{ _MMIO(0x9888), 0x0edac000 },
	557	{ _MMIO(0x9888), 0x00da8000 },
	558	{ _MMIO(0x9888), 0x02dac000 },
	559	{ _MMIO(0x9888), 0x04da4000 },
	560	{ _MMIO(0x9888), 0x04983400 },
	561	{ _MMIO(0x9888), 0x10980000 },
	562	{ _MMIO(0x9888), 0x06990034 },
	563	{ _MMIO(0x9888), 0x10990000 },
	564	{ _MMIO(0x9888), 0x0c9dc000 },
	565	{ _MMIO(0x9888), 0x0e9dc000 },
	566	{ _MMIO(0x9888), 0x009d8000 },
	567	{ _MMIO(0x9888), 0x029dc000 },
	568	{ _MMIO(0x9888), 0x049d4000 },
	569	{ _MMIO(0x9888), 0x109f02a8 },
	570	{ _MMIO(0x9888), 0x0c9fa000 },
	571	{ _MMIO(0x9888), 0x0e9f00ba },
	572	{ _MMIO(0x9888), 0x0cb88000 },
	573	{ _MMIO(0x9888), 0x0cb95000 },
	574	{ _MMIO(0x9888), 0x0eb95000 },
	575	{ _MMIO(0x9888), 0x00b94000 },
	576	{ _MMIO(0x9888), 0x02b95000 },
	577	{ _MMIO(0x9888), 0x04b91000 },
	578	{ _MMIO(0x9888), 0x06b92000 },
	579	{ _MMIO(0x9888), 0x0cba4000 },
	580	{ _MMIO(0x9888), 0x0f88000f },
	581	{ _MMIO(0x9888), 0x03888000 },
	582	{ _MMIO(0x9888), 0x05888000 },
	583	{ _MMIO(0x9888), 0x07888000 },
	584	{ _MMIO(0x9888), 0x09888000 },
	585	{ _MMIO(0x9888), 0x0b888000 },
	586	{ _MMIO(0x9888), 0x0d880400 },
	587	{ _MMIO(0x9888), 0x258b800a },
	588	{ _MMIO(0x9888), 0x278b002a },
	589	{ _MMIO(0x9888), 0x238b5500 },
	590	{ _MMIO(0x9888), 0x198c4000 },
	591	{ _MMIO(0x9888), 0x1b8c0015 },
	592	{ _MMIO(0x9888), 0x038c4000 },
	593	{ _MMIO(0x9888), 0x058c4000 },
	594	{ _MMIO(0x9888), 0x078c4000 },
	595	{ _MMIO(0x9888), 0x098c4000 },
	596	{ _MMIO(0x9888), 0x0b8c4000 },
	597	{ _MMIO(0x9888), 0x0d8c4000 },
	598	{ _MMIO(0x9888), 0x0d8da000 },
	599	{ _MMIO(0x9888), 0x0f8da000 },
	600	{ _MMIO(0x9888), 0x018d8000 },
	601	{ _MMIO(0x9888), 0x038da000 },
	602	{ _MMIO(0x9888), 0x058da000 },
	603	{ _MMIO(0x9888), 0x078d2000 },
	604	{ _MMIO(0x9888), 0x2185800a },
	605	{ _MMIO(0x9888), 0x2385002a },
	606	{ _MMIO(0x9888), 0x1f85aa00 },
	607	{ _MMIO(0x9888), 0x1b830154 },
	608	{ _MMIO(0x9888), 0x03834000 },
	609	{ _MMIO(0x9888), 0x05834000 },
	610	{ _MMIO(0x9888), 0x07834000 },
	611	{ _MMIO(0x9888), 0x09834000 },
	612	{ _MMIO(0x9888), 0x0b834000 },
	613	{ _MMIO(0x9888), 0x0d834000 },
	614	{ _MMIO(0x9888), 0x0d84c000 },
	615	{ _MMIO(0x9888), 0x0f84c000 },
	616	{ _MMIO(0x9888), 0x01848000 },
	617	{ _MMIO(0x9888), 0x0384c000 },
	618	{ _MMIO(0x9888), 0x0584c000 },
	619	{ _MMIO(0x9888), 0x07844000 },
	620	{ _MMIO(0x9888), 0x1d80c000 },
	621	{ _MMIO(0x9888), 0x1f80c000 },
	622	{ _MMIO(0x9888), 0x11808000 },
	623	{ _MMIO(0x9888), 0x1380c000 },
	624	{ _MMIO(0x9888), 0x1580c000 },
	625	{ _MMIO(0x9888), 0x17804000 },
	626	{ _MMIO(0x9888), 0x53800000 },
	627	{ _MMIO(0x9888), 0x45800000 },
	628	{ _MMIO(0x9888), 0x47800000 },
	629	{ _MMIO(0x9888), 0x21800000 },
	630	{ _MMIO(0x9888), 0x31800000 },
	631	{ _MMIO(0x9888), 0x4d800000 },
	632	{ _MMIO(0x9888), 0x3f800000 },
	633	{ _MMIO(0x9888), 0x4f800000 },
	634	{ _MMIO(0x9888), 0x41800060 },
	635	};
	636
	637	static int
	638	get_l3_1_mux_config(struct drm_i915_private *dev_priv,
	639	const struct i915_oa_reg **regs,
	640	int *lens)
	641	{
	642	int n = 0;
	643
	644	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	645	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	646
	647	regs[n] = mux_config_l3_1;
	648	lens[n] = ARRAY_SIZE(mux_config_l3_1);
	649	n++;
	650
	651	return n;
	652	}
	653
	654	static const struct i915_oa_reg b_counter_config_l3_2[] = {
	655	{ _MMIO(0x2740), 0x00000000 },
	656	{ _MMIO(0x2744), 0x00800000 },
	657	{ _MMIO(0x2710), 0x00000000 },
	658	{ _MMIO(0x2714), 0xf0800000 },
	659	{ _MMIO(0x2720), 0x00000000 },
	660	{ _MMIO(0x2724), 0xf0800000 },
	661	{ _MMIO(0x2770), 0x00100070 },
	662	{ _MMIO(0x2774), 0x0000fff1 },
	663	{ _MMIO(0x2778), 0x00014002 },
	664	{ _MMIO(0x277c), 0x0000c3ff },
	665	{ _MMIO(0x2780), 0x00010002 },
	666	{ _MMIO(0x2784), 0x0000c7ff },
	667	{ _MMIO(0x2788), 0x00004002 },
	668	{ _MMIO(0x278c), 0x0000d3ff },
	669	{ _MMIO(0x2790), 0x00100700 },
	670	{ _MMIO(0x2794), 0x0000ff1f },
	671	{ _MMIO(0x2798), 0x00001402 },
	672	{ _MMIO(0x279c), 0x0000fc3f },
	673	{ _MMIO(0x27a0), 0x00001002 },
	674	{ _MMIO(0x27a4), 0x0000fc7f },
	675	{ _MMIO(0x27a8), 0x00000402 },
	676	{ _MMIO(0x27ac), 0x0000fd3f },
	677	};
	678
	679	static const struct i915_oa_reg flex_eu_config_l3_2[] = {
	680	{ _MMIO(0xe458), 0x00005004 },
	681	{ _MMIO(0xe558), 0x00010003 },
	682	{ _MMIO(0xe658), 0x00012011 },
	683	{ _MMIO(0xe758), 0x00015014 },
	684	{ _MMIO(0xe45c), 0x00051050 },
	685	{ _MMIO(0xe55c), 0x00053052 },
	686	{ _MMIO(0xe65c), 0x00055054 },
	687	};
	688
	689	static const struct i915_oa_reg mux_config_l3_2[] = {
	690	{ _MMIO(0x9888), 0x103f03da },
	691	{ _MMIO(0x9888), 0x143f0001 },
	692	{ _MMIO(0x9888), 0x12180340 },
	693	{ _MMIO(0x9888), 0x12190340 },
	694	{ _MMIO(0x9888), 0x0c3f1187 },
	695	{ _MMIO(0x9888), 0x0e3f1205 },
	696	{ _MMIO(0x9888), 0x003f0500 },
	697	{ _MMIO(0x9888), 0x023f042b },
	698	{ _MMIO(0x9888), 0x043f002c },
	699	{ _MMIO(0x9888), 0x0c5ac000 },
	700	{ _MMIO(0x9888), 0x0e5ac000 },
	701	{ _MMIO(0x9888), 0x005a8000 },
	702	{ _MMIO(0x9888), 0x025ac000 },
	703	{ _MMIO(0x9888), 0x045a4000 },
	704	{ _MMIO(0x9888), 0x04183400 },
	705	{ _MMIO(0x9888), 0x10180000 },
	706	{ _MMIO(0x9888), 0x06190034 },
	707	{ _MMIO(0x9888), 0x10190000 },
	708	{ _MMIO(0x9888), 0x0c1dc000 },
	709	{ _MMIO(0x9888), 0x0e1dc000 },
	710	{ _MMIO(0x9888), 0x001d8000 },
	711	{ _MMIO(0x9888), 0x021dc000 },
	712	{ _MMIO(0x9888), 0x041d4000 },
	713	{ _MMIO(0x9888), 0x101f02a8 },
	714	{ _MMIO(0x9888), 0x0c1fa000 },
	715	{ _MMIO(0x9888), 0x0e1f00ba },
	716	{ _MMIO(0x9888), 0x0c388000 },
	717	{ _MMIO(0x9888), 0x0c395000 },
	718	{ _MMIO(0x9888), 0x0e395000 },
	719	{ _MMIO(0x9888), 0x00394000 },
	720	{ _MMIO(0x9888), 0x02395000 },
	721	{ _MMIO(0x9888), 0x04391000 },
	722	{ _MMIO(0x9888), 0x06392000 },
	723	{ _MMIO(0x9888), 0x0c3a4000 },
	724	{ _MMIO(0x9888), 0x1b8aa800 },
	725	{ _MMIO(0x9888), 0x1d8a0002 },
	726	{ _MMIO(0x9888), 0x038a8000 },
	727	{ _MMIO(0x9888), 0x058a8000 },
	728	{ _MMIO(0x9888), 0x078a8000 },
	729	{ _MMIO(0x9888), 0x098a8000 },
	730	{ _MMIO(0x9888), 0x0b8a8000 },
	731	{ _MMIO(0x9888), 0x0d8a8000 },
	732	{ _MMIO(0x9888), 0x258b4005 },
	733	{ _MMIO(0x9888), 0x278b0015 },
	734	{ _MMIO(0x9888), 0x238b2a80 },
	735	{ _MMIO(0x9888), 0x2185800a },
	736	{ _MMIO(0x9888), 0x2385002a },
	737	{ _MMIO(0x9888), 0x1f85aa00 },
	738	{ _MMIO(0x9888), 0x1b830154 },
	739	{ _MMIO(0x9888), 0x03834000 },
	740	{ _MMIO(0x9888), 0x05834000 },
	741	{ _MMIO(0x9888), 0x07834000 },
	742	{ _MMIO(0x9888), 0x09834000 },
	743	{ _MMIO(0x9888), 0x0b834000 },
	744	{ _MMIO(0x9888), 0x0d834000 },
	745	{ _MMIO(0x9888), 0x0d84c000 },
	746	{ _MMIO(0x9888), 0x0f84c000 },
	747	{ _MMIO(0x9888), 0x01848000 },
	748	{ _MMIO(0x9888), 0x0384c000 },
	749	{ _MMIO(0x9888), 0x0584c000 },
	750	{ _MMIO(0x9888), 0x07844000 },
	751	{ _MMIO(0x9888), 0x1d80c000 },
	752	{ _MMIO(0x9888), 0x1f80c000 },
	753	{ _MMIO(0x9888), 0x11808000 },
	754	{ _MMIO(0x9888), 0x1380c000 },
	755	{ _MMIO(0x9888), 0x1580c000 },
	756	{ _MMIO(0x9888), 0x17804000 },
	757	{ _MMIO(0x9888), 0x53800000 },
	758	{ _MMIO(0x9888), 0x45800000 },
	759	{ _MMIO(0x9888), 0x47800000 },
	760	{ _MMIO(0x9888), 0x21800000 },
	761	{ _MMIO(0x9888), 0x31800000 },
	762	{ _MMIO(0x9888), 0x4d800000 },
	763	{ _MMIO(0x9888), 0x3f800000 },
	764	{ _MMIO(0x9888), 0x4f800000 },
	765	{ _MMIO(0x9888), 0x41800060 },
	766	};
	767
	768	static int
	769	get_l3_2_mux_config(struct drm_i915_private *dev_priv,
	770	const struct i915_oa_reg **regs,
	771	int *lens)
	772	{
	773	int n = 0;
	774
	775	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	776	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	777
	778	regs[n] = mux_config_l3_2;
	779	lens[n] = ARRAY_SIZE(mux_config_l3_2);
	780	n++;
	781
	782	return n;
	783	}
	784
	785	static const struct i915_oa_reg b_counter_config_l3_3[] = {
	786	{ _MMIO(0x2740), 0x00000000 },
	787	{ _MMIO(0x2744), 0x00800000 },
	788	{ _MMIO(0x2710), 0x00000000 },
	789	{ _MMIO(0x2714), 0xf0800000 },
	790	{ _MMIO(0x2720), 0x00000000 },
	791	{ _MMIO(0x2724), 0xf0800000 },
	792	{ _MMIO(0x2770), 0x00100070 },
	793	{ _MMIO(0x2774), 0x0000fff1 },
	794	{ _MMIO(0x2778), 0x00014002 },
	795	{ _MMIO(0x277c), 0x0000c3ff },
	796	{ _MMIO(0x2780), 0x00010002 },
	797	{ _MMIO(0x2784), 0x0000c7ff },
	798	{ _MMIO(0x2788), 0x00004002 },
	799	{ _MMIO(0x278c), 0x0000d3ff },
	800	{ _MMIO(0x2790), 0x00100700 },
	801	{ _MMIO(0x2794), 0x0000ff1f },
	802	{ _MMIO(0x2798), 0x00001402 },
	803	{ _MMIO(0x279c), 0x0000fc3f },
	804	{ _MMIO(0x27a0), 0x00001002 },
	805	{ _MMIO(0x27a4), 0x0000fc7f },
	806	{ _MMIO(0x27a8), 0x00000402 },
	807	{ _MMIO(0x27ac), 0x0000fd3f },
	808	};
	809
	810	static const struct i915_oa_reg flex_eu_config_l3_3[] = {
	811	{ _MMIO(0xe458), 0x00005004 },
	812	{ _MMIO(0xe558), 0x00010003 },
	813	{ _MMIO(0xe658), 0x00012011 },
	814	{ _MMIO(0xe758), 0x00015014 },
	815	{ _MMIO(0xe45c), 0x00051050 },
	816	{ _MMIO(0xe55c), 0x00053052 },
	817	{ _MMIO(0xe65c), 0x00055054 },
	818	};
	819
	820	static const struct i915_oa_reg mux_config_l3_3[] = {
	821	{ _MMIO(0x9888), 0x121b0340 },
	822	{ _MMIO(0x9888), 0x103f0274 },
	823	{ _MMIO(0x9888), 0x123f0000 },
	824	{ _MMIO(0x9888), 0x129b0340 },
	825	{ _MMIO(0x9888), 0x10bf0274 },
	826	{ _MMIO(0x9888), 0x12bf0000 },
	827	{ _MMIO(0x9888), 0x041b3400 },
	828	{ _MMIO(0x9888), 0x101b0000 },
	829	{ _MMIO(0x9888), 0x045c8000 },
	830	{ _MMIO(0x9888), 0x0a3d4000 },
	831	{ _MMIO(0x9888), 0x003f0080 },
	832	{ _MMIO(0x9888), 0x023f0793 },
	833	{ _MMIO(0x9888), 0x043f0014 },
	834	{ _MMIO(0x9888), 0x04588000 },
	835	{ _MMIO(0x9888), 0x005a8000 },
	836	{ _MMIO(0x9888), 0x025ac000 },
	837	{ _MMIO(0x9888), 0x045a4000 },
	838	{ _MMIO(0x9888), 0x0a5b4000 },
	839	{ _MMIO(0x9888), 0x001d8000 },
	840	{ _MMIO(0x9888), 0x021dc000 },
	841	{ _MMIO(0x9888), 0x041d4000 },
	842	{ _MMIO(0x9888), 0x0c1fa000 },
	843	{ _MMIO(0x9888), 0x0e1f002a },
	844	{ _MMIO(0x9888), 0x0a384000 },
	845	{ _MMIO(0x9888), 0x00394000 },
	846	{ _MMIO(0x9888), 0x02395000 },
	847	{ _MMIO(0x9888), 0x04399000 },
	848	{ _MMIO(0x9888), 0x069b0034 },
	849	{ _MMIO(0x9888), 0x109b0000 },
	850	{ _MMIO(0x9888), 0x06dc4000 },
	851	{ _MMIO(0x9888), 0x0cbd4000 },
	852	{ _MMIO(0x9888), 0x0cbf0981 },
	853	{ _MMIO(0x9888), 0x0ebf0a0f },
	854	{ _MMIO(0x9888), 0x06d84000 },
	855	{ _MMIO(0x9888), 0x0cdac000 },
	856	{ _MMIO(0x9888), 0x0edac000 },
	857	{ _MMIO(0x9888), 0x0cdb4000 },
	858	{ _MMIO(0x9888), 0x0c9dc000 },
	859	{ _MMIO(0x9888), 0x0e9dc000 },
	860	{ _MMIO(0x9888), 0x109f02a8 },
	861	{ _MMIO(0x9888), 0x0e9f0080 },
	862	{ _MMIO(0x9888), 0x0cb84000 },
	863	{ _MMIO(0x9888), 0x0cb95000 },
	864	{ _MMIO(0x9888), 0x0eb95000 },
	865	{ _MMIO(0x9888), 0x06b92000 },
	866	{ _MMIO(0x9888), 0x0f88000f },
	867	{ _MMIO(0x9888), 0x0d880400 },
	868	{ _MMIO(0x9888), 0x038a8000 },
	869	{ _MMIO(0x9888), 0x058a8000 },
	870	{ _MMIO(0x9888), 0x078a8000 },
	871	{ _MMIO(0x9888), 0x098a8000 },
	872	{ _MMIO(0x9888), 0x0b8a8000 },
	873	{ _MMIO(0x9888), 0x258b8009 },
	874	{ _MMIO(0x9888), 0x278b002a },
	875	{ _MMIO(0x9888), 0x238b2a80 },
	876	{ _MMIO(0x9888), 0x198c4000 },
	877	{ _MMIO(0x9888), 0x1b8c0015 },
	878	{ _MMIO(0x9888), 0x0d8c4000 },
	879	{ _MMIO(0x9888), 0x0d8da000 },
	880	{ _MMIO(0x9888), 0x0f8da000 },
	881	{ _MMIO(0x9888), 0x078d2000 },
	882	{ _MMIO(0x9888), 0x2185800a },
	883	{ _MMIO(0x9888), 0x2385002a },
	884	{ _MMIO(0x9888), 0x1f85aa00 },
	885	{ _MMIO(0x9888), 0x1b830154 },
	886	{ _MMIO(0x9888), 0x03834000 },
	887	{ _MMIO(0x9888), 0x05834000 },
	888	{ _MMIO(0x9888), 0x07834000 },
	889	{ _MMIO(0x9888), 0x09834000 },
	890	{ _MMIO(0x9888), 0x0b834000 },
	891	{ _MMIO(0x9888), 0x0d834000 },
	892	{ _MMIO(0x9888), 0x0d84c000 },
	893	{ _MMIO(0x9888), 0x0f84c000 },
	894	{ _MMIO(0x9888), 0x01848000 },
	895	{ _MMIO(0x9888), 0x0384c000 },
	896	{ _MMIO(0x9888), 0x0584c000 },
	897	{ _MMIO(0x9888), 0x07844000 },
	898	{ _MMIO(0x9888), 0x1d80c000 },
	899	{ _MMIO(0x9888), 0x1f80c000 },
	900	{ _MMIO(0x9888), 0x11808000 },
	901	{ _MMIO(0x9888), 0x1380c000 },
	902	{ _MMIO(0x9888), 0x1580c000 },
	903	{ _MMIO(0x9888), 0x17804000 },
	904	{ _MMIO(0x9888), 0x53800000 },
	905	{ _MMIO(0x9888), 0x45800c00 },
	906	{ _MMIO(0x9888), 0x47800c63 },
	907	{ _MMIO(0x9888), 0x21800000 },
	908	{ _MMIO(0x9888), 0x31800000 },
	909	{ _MMIO(0x9888), 0x4d800000 },
	910	{ _MMIO(0x9888), 0x3f8014a5 },
	911	{ _MMIO(0x9888), 0x4f800000 },
	912	{ _MMIO(0x9888), 0x41800045 },
	913	};
	914
	915	static int
	916	get_l3_3_mux_config(struct drm_i915_private *dev_priv,
	917	const struct i915_oa_reg **regs,
	918	int *lens)
	919	{
	920	int n = 0;
	921
	922	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	923	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	924
	925	regs[n] = mux_config_l3_3;
	926	lens[n] = ARRAY_SIZE(mux_config_l3_3);
	927	n++;
	928
	929	return n;
	930	}
	931
	932	static const struct i915_oa_reg b_counter_config_l3_4[] = {
	933	{ _MMIO(0x2740), 0x00000000 },
	934	{ _MMIO(0x2744), 0x00800000 },
	935	{ _MMIO(0x2710), 0x00000000 },
	936	{ _MMIO(0x2714), 0xf0800000 },
	937	{ _MMIO(0x2720), 0x00000000 },
	938	{ _MMIO(0x2724), 0xf0800000 },
	939	{ _MMIO(0x2770), 0x00100070 },
	940	{ _MMIO(0x2774), 0x0000fff1 },
	941	{ _MMIO(0x2778), 0x00014002 },
	942	{ _MMIO(0x277c), 0x0000c3ff },
	943	{ _MMIO(0x2780), 0x00010002 },
	944	{ _MMIO(0x2784), 0x0000c7ff },
	945	{ _MMIO(0x2788), 0x00004002 },
	946	{ _MMIO(0x278c), 0x0000d3ff },
	947	{ _MMIO(0x2790), 0x00100700 },
	948	{ _MMIO(0x2794), 0x0000ff1f },
	949	{ _MMIO(0x2798), 0x00001402 },
	950	{ _MMIO(0x279c), 0x0000fc3f },
	951	{ _MMIO(0x27a0), 0x00001002 },
	952	{ _MMIO(0x27a4), 0x0000fc7f },
	953	{ _MMIO(0x27a8), 0x00000402 },
	954	{ _MMIO(0x27ac), 0x0000fd3f },
	955	};
	956
	957	static const struct i915_oa_reg flex_eu_config_l3_4[] = {
	958	{ _MMIO(0xe458), 0x00005004 },
	959	{ _MMIO(0xe558), 0x00010003 },
	960	{ _MMIO(0xe658), 0x00012011 },
	961	{ _MMIO(0xe758), 0x00015014 },
	962	{ _MMIO(0xe45c), 0x00051050 },
	963	{ _MMIO(0xe55c), 0x00053052 },
	964	{ _MMIO(0xe65c), 0x00055054 },
	965	};
	966
	967	static const struct i915_oa_reg mux_config_l3_4[] = {
	968	{ _MMIO(0x9888), 0x121a0340 },
	969	{ _MMIO(0x9888), 0x103f0017 },
	970	{ _MMIO(0x9888), 0x123f0020 },
	971	{ _MMIO(0x9888), 0x129a0340 },
	972	{ _MMIO(0x9888), 0x10bf0017 },
	973	{ _MMIO(0x9888), 0x12bf0020 },
	974	{ _MMIO(0x9888), 0x041a3400 },
	975	{ _MMIO(0x9888), 0x101a0000 },
	976	{ _MMIO(0x9888), 0x043b8000 },
	977	{ _MMIO(0x9888), 0x0a3e0010 },
	978	{ _MMIO(0x9888), 0x003f0200 },
	979	{ _MMIO(0x9888), 0x023f0113 },
	980	{ _MMIO(0x9888), 0x043f0014 },
	981	{ _MMIO(0x9888), 0x02592000 },
	982	{ _MMIO(0x9888), 0x005a8000 },
	983	{ _MMIO(0x9888), 0x025ac000 },
	984	{ _MMIO(0x9888), 0x045a4000 },
	985	{ _MMIO(0x9888), 0x0a1c8000 },
	986	{ _MMIO(0x9888), 0x001d8000 },
	987	{ _MMIO(0x9888), 0x021dc000 },
	988	{ _MMIO(0x9888), 0x041d4000 },
	989	{ _MMIO(0x9888), 0x0a1e8000 },
	990	{ _MMIO(0x9888), 0x0c1fa000 },
	991	{ _MMIO(0x9888), 0x0e1f001a },
	992	{ _MMIO(0x9888), 0x00394000 },
	993	{ _MMIO(0x9888), 0x02395000 },
	994	{ _MMIO(0x9888), 0x04391000 },
	995	{ _MMIO(0x9888), 0x069a0034 },
	996	{ _MMIO(0x9888), 0x109a0000 },
	997	{ _MMIO(0x9888), 0x06bb4000 },
	998	{ _MMIO(0x9888), 0x0abe0040 },
	999	{ _MMIO(0x9888), 0x0cbf0984 },
	1000	{ _MMIO(0x9888), 0x0ebf0a02 },
	1001	{ _MMIO(0x9888), 0x02d94000 },
	1002	{ _MMIO(0x9888), 0x0cdac000 },
	1003	{ _MMIO(0x9888), 0x0edac000 },
	1004	{ _MMIO(0x9888), 0x0c9c0400 },
	1005	{ _MMIO(0x9888), 0x0c9dc000 },
	1006	{ _MMIO(0x9888), 0x0e9dc000 },
	1007	{ _MMIO(0x9888), 0x0c9e0400 },
	1008	{ _MMIO(0x9888), 0x109f02a8 },
	1009	{ _MMIO(0x9888), 0x0e9f0040 },
	1010	{ _MMIO(0x9888), 0x0cb95000 },
	1011	{ _MMIO(0x9888), 0x0eb95000 },
	1012	{ _MMIO(0x9888), 0x0f88000f },
	1013	{ _MMIO(0x9888), 0x0d880400 },
	1014	{ _MMIO(0x9888), 0x038a8000 },
	1015	{ _MMIO(0x9888), 0x058a8000 },
	1016	{ _MMIO(0x9888), 0x078a8000 },
	1017	{ _MMIO(0x9888), 0x098a8000 },
	1018	{ _MMIO(0x9888), 0x0b8a8000 },
	1019	{ _MMIO(0x9888), 0x258b8009 },
	1020	{ _MMIO(0x9888), 0x278b002a },
	1021	{ _MMIO(0x9888), 0x238b2a80 },
	1022	{ _MMIO(0x9888), 0x198c4000 },
	1023	{ _MMIO(0x9888), 0x1b8c0015 },
	1024	{ _MMIO(0x9888), 0x0d8c4000 },
	1025	{ _MMIO(0x9888), 0x0d8da000 },
	1026	{ _MMIO(0x9888), 0x0f8da000 },
	1027	{ _MMIO(0x9888), 0x078d2000 },
	1028	{ _MMIO(0x9888), 0x2185800a },
	1029	{ _MMIO(0x9888), 0x2385002a },
	1030	{ _MMIO(0x9888), 0x1f85aa00 },
	1031	{ _MMIO(0x9888), 0x1b830154 },
	1032	{ _MMIO(0x9888), 0x03834000 },
	1033	{ _MMIO(0x9888), 0x05834000 },
	1034	{ _MMIO(0x9888), 0x07834000 },
	1035	{ _MMIO(0x9888), 0x09834000 },
	1036	{ _MMIO(0x9888), 0x0b834000 },
	1037	{ _MMIO(0x9888), 0x0d834000 },
	1038	{ _MMIO(0x9888), 0x0d84c000 },
	1039	{ _MMIO(0x9888), 0x0f84c000 },
	1040	{ _MMIO(0x9888), 0x01848000 },
	1041	{ _MMIO(0x9888), 0x0384c000 },
	1042	{ _MMIO(0x9888), 0x0584c000 },
	1043	{ _MMIO(0x9888), 0x07844000 },
	1044	{ _MMIO(0x9888), 0x1d80c000 },
	1045	{ _MMIO(0x9888), 0x1f80c000 },
	1046	{ _MMIO(0x9888), 0x11808000 },
	1047	{ _MMIO(0x9888), 0x1380c000 },
	1048	{ _MMIO(0x9888), 0x1580c000 },
	1049	{ _MMIO(0x9888), 0x17804000 },
	1050	{ _MMIO(0x9888), 0x53800000 },
	1051	{ _MMIO(0x9888), 0x45800800 },
	1052	{ _MMIO(0x9888), 0x47800842 },
	1053	{ _MMIO(0x9888), 0x21800000 },
	1054	{ _MMIO(0x9888), 0x31800000 },
	1055	{ _MMIO(0x9888), 0x4d800000 },
	1056	{ _MMIO(0x9888), 0x3f801084 },
	1057	{ _MMIO(0x9888), 0x4f800000 },
	1058	{ _MMIO(0x9888), 0x41800044 },
	1059	};
	1060
	1061	static int
	1062	get_l3_4_mux_config(struct drm_i915_private *dev_priv,
	1063	const struct i915_oa_reg **regs,
	1064	int *lens)
	1065	{
	1066	int n = 0;
	1067
	1068	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1069	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1070
	1071	regs[n] = mux_config_l3_4;
	1072	lens[n] = ARRAY_SIZE(mux_config_l3_4);
	1073	n++;
	1074
	1075	return n;
	1076	}
	1077
	1078	static const struct i915_oa_reg b_counter_config_rasterizer_and_pixel_backend[] = {
	1079	{ _MMIO(0x2740), 0x00000000 },
	1080	{ _MMIO(0x2744), 0x00800000 },
	1081	{ _MMIO(0x2710), 0x00000000 },
	1082	{ _MMIO(0x2714), 0xf0800000 },
	1083	{ _MMIO(0x2720), 0x00000000 },
	1084	{ _MMIO(0x2724), 0x30800000 },
	1085	{ _MMIO(0x2770), 0x00006000 },
	1086	{ _MMIO(0x2774), 0x0000f3ff },
	1087	{ _MMIO(0x2778), 0x00001800 },
	1088	{ _MMIO(0x277c), 0x0000fcff },
	1089	{ _MMIO(0x2780), 0x00000600 },
	1090	{ _MMIO(0x2784), 0x0000ff3f },
	1091	{ _MMIO(0x2788), 0x00000180 },
	1092	{ _MMIO(0x278c), 0x0000ffcf },
	1093	{ _MMIO(0x2790), 0x00000060 },
	1094	{ _MMIO(0x2794), 0x0000fff3 },
	1095	{ _MMIO(0x2798), 0x00000018 },
	1096	{ _MMIO(0x279c), 0x0000fffc },
	1097	};
	1098
	1099	static const struct i915_oa_reg flex_eu_config_rasterizer_and_pixel_backend[] = {
	1100	{ _MMIO(0xe458), 0x00005004 },
	1101	{ _MMIO(0xe558), 0x00010003 },
	1102	{ _MMIO(0xe658), 0x00012011 },
	1103	{ _MMIO(0xe758), 0x00015014 },
	1104	{ _MMIO(0xe45c), 0x00051050 },
	1105	{ _MMIO(0xe55c), 0x00053052 },
	1106	{ _MMIO(0xe65c), 0x00055054 },
	1107	};
	1108
	1109	static const struct i915_oa_reg mux_config_rasterizer_and_pixel_backend[] = {
	1110	{ _MMIO(0x9888), 0x143b000e },
	1111	{ _MMIO(0x9888), 0x043c55c0 },
	1112	{ _MMIO(0x9888), 0x0a1e0280 },
	1113	{ _MMIO(0x9888), 0x0c1e0408 },
	1114	{ _MMIO(0x9888), 0x10390000 },
	1115	{ _MMIO(0x9888), 0x12397a1f },
	1116	{ _MMIO(0x9888), 0x14bb000e },
	1117	{ _MMIO(0x9888), 0x04bc5000 },
	1118	{ _MMIO(0x9888), 0x0a9e0296 },
	1119	{ _MMIO(0x9888), 0x0c9e0008 },
	1120	{ _MMIO(0x9888), 0x10b90000 },
	1121	{ _MMIO(0x9888), 0x12b97a1f },
	1122	{ _MMIO(0x9888), 0x063b0042 },
	1123	{ _MMIO(0x9888), 0x103b0000 },
	1124	{ _MMIO(0x9888), 0x083c0000 },
	1125	{ _MMIO(0x9888), 0x0a3e0040 },
	1126	{ _MMIO(0x9888), 0x043f8000 },
	1127	{ _MMIO(0x9888), 0x02594000 },
	1128	{ _MMIO(0x9888), 0x045a8000 },
	1129	{ _MMIO(0x9888), 0x0c1c0400 },
	1130	{ _MMIO(0x9888), 0x041d8000 },
	1131	{ _MMIO(0x9888), 0x081e02c0 },
	1132	{ _MMIO(0x9888), 0x0e1e0000 },
	1133	{ _MMIO(0x9888), 0x0c1fa800 },
	1134	{ _MMIO(0x9888), 0x0e1f0260 },
	1135	{ _MMIO(0x9888), 0x101f0014 },
	1136	{ _MMIO(0x9888), 0x003905e0 },
	1137	{ _MMIO(0x9888), 0x06390bc0 },
	1138	{ _MMIO(0x9888), 0x02390018 },
	1139	{ _MMIO(0x9888), 0x04394000 },
	1140	{ _MMIO(0x9888), 0x04bb0042 },
	1141	{ _MMIO(0x9888), 0x10bb0000 },
	1142	{ _MMIO(0x9888), 0x02bc05c0 },
	1143	{ _MMIO(0x9888), 0x08bc0000 },
	1144	{ _MMIO(0x9888), 0x0abe0004 },
	1145	{ _MMIO(0x9888), 0x02bf8000 },
	1146	{ _MMIO(0x9888), 0x02d91000 },
	1147	{ _MMIO(0x9888), 0x02da8000 },
	1148	{ _MMIO(0x9888), 0x089c8000 },
	1149	{ _MMIO(0x9888), 0x029d8000 },
	1150	{ _MMIO(0x9888), 0x089e8000 },
	1151	{ _MMIO(0x9888), 0x0e9e0000 },
	1152	{ _MMIO(0x9888), 0x0e9fa806 },
	1153	{ _MMIO(0x9888), 0x109f0142 },
	1154	{ _MMIO(0x9888), 0x08b90617 },
	1155	{ _MMIO(0x9888), 0x0ab90be0 },
	1156	{ _MMIO(0x9888), 0x02b94000 },
	1157	{ _MMIO(0x9888), 0x0d88f000 },
	1158	{ _MMIO(0x9888), 0x0f88000c },
	1159	{ _MMIO(0x9888), 0x07888000 },
	1160	{ _MMIO(0x9888), 0x09888000 },
	1161	{ _MMIO(0x9888), 0x018a8000 },
	1162	{ _MMIO(0x9888), 0x0f8a8000 },
	1163	{ _MMIO(0x9888), 0x1b8a2800 },
	1164	{ _MMIO(0x9888), 0x038a8000 },
	1165	{ _MMIO(0x9888), 0x058a8000 },
	1166	{ _MMIO(0x9888), 0x0b8a8000 },
	1167	{ _MMIO(0x9888), 0x0d8a8000 },
	1168	{ _MMIO(0x9888), 0x238b52a0 },
	1169	{ _MMIO(0x9888), 0x258b6a95 },
	1170	{ _MMIO(0x9888), 0x278b0029 },
	1171	{ _MMIO(0x9888), 0x178c2000 },
	1172	{ _MMIO(0x9888), 0x198c1500 },
	1173	{ _MMIO(0x9888), 0x1b8c0014 },
	1174	{ _MMIO(0x9888), 0x078c4000 },
	1175	{ _MMIO(0x9888), 0x098c4000 },
	1176	{ _MMIO(0x9888), 0x098da000 },
	1177	{ _MMIO(0x9888), 0x0b8da000 },
	1178	{ _MMIO(0x9888), 0x0f8da000 },
	1179	{ _MMIO(0x9888), 0x038d8000 },
	1180	{ _MMIO(0x9888), 0x058d2000 },
	1181	{ _MMIO(0x9888), 0x1f85aa80 },
	1182	{ _MMIO(0x9888), 0x2185aaaa },
	1183	{ _MMIO(0x9888), 0x2385002a },
	1184	{ _MMIO(0x9888), 0x01834000 },
	1185	{ _MMIO(0x9888), 0x0f834000 },
	1186	{ _MMIO(0x9888), 0x19835400 },
	1187	{ _MMIO(0x9888), 0x1b830155 },
	1188	{ _MMIO(0x9888), 0x03834000 },
	1189	{ _MMIO(0x9888), 0x05834000 },
	1190	{ _MMIO(0x9888), 0x07834000 },
	1191	{ _MMIO(0x9888), 0x09834000 },
	1192	{ _MMIO(0x9888), 0x0b834000 },
	1193	{ _MMIO(0x9888), 0x0d834000 },
	1194	{ _MMIO(0x9888), 0x0184c000 },
	1195	{ _MMIO(0x9888), 0x0784c000 },
	1196	{ _MMIO(0x9888), 0x0984c000 },
	1197	{ _MMIO(0x9888), 0x0b84c000 },
	1198	{ _MMIO(0x9888), 0x0d84c000 },
	1199	{ _MMIO(0x9888), 0x0f84c000 },
	1200	{ _MMIO(0x9888), 0x0384c000 },
	1201	{ _MMIO(0x9888), 0x0584c000 },
	1202	{ _MMIO(0x9888), 0x1180c000 },
	1203	{ _MMIO(0x9888), 0x1780c000 },
	1204	{ _MMIO(0x9888), 0x1980c000 },
	1205	{ _MMIO(0x9888), 0x1b80c000 },
	1206	{ _MMIO(0x9888), 0x1d80c000 },
	1207	{ _MMIO(0x9888), 0x1f80c000 },
	1208	{ _MMIO(0x9888), 0x1380c000 },
	1209	{ _MMIO(0x9888), 0x1580c000 },
	1210	{ _MMIO(0x9888), 0x4d800444 },
	1211	{ _MMIO(0x9888), 0x3d800000 },
	1212	{ _MMIO(0x9888), 0x4f804000 },
	1213	{ _MMIO(0x9888), 0x43801080 },
	1214	{ _MMIO(0x9888), 0x51800000 },
	1215	{ _MMIO(0x9888), 0x45800084 },
	1216	{ _MMIO(0x9888), 0x53800044 },
	1217	{ _MMIO(0x9888), 0x47801080 },
	1218	{ _MMIO(0x9888), 0x21800000 },
	1219	{ _MMIO(0x9888), 0x31800000 },
	1220	{ _MMIO(0x9888), 0x3f800000 },
	1221	{ _MMIO(0x9888), 0x41800840 },
	1222	};
	1223
	1224	static int
	1225	get_rasterizer_and_pixel_backend_mux_config(struct drm_i915_private *dev_priv,
	1226	const struct i915_oa_reg **regs,
	1227	int *lens)
	1228	{
	1229	int n = 0;
	1230
	1231	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1232	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1233
	1234	regs[n] = mux_config_rasterizer_and_pixel_backend;
	1235	lens[n] = ARRAY_SIZE(mux_config_rasterizer_and_pixel_backend);
	1236	n++;
	1237
	1238	return n;
	1239	}
	1240
	1241	static const struct i915_oa_reg b_counter_config_sampler_1[] = {
	1242	{ _MMIO(0x2740), 0x00000000 },
	1243	{ _MMIO(0x2744), 0x00800000 },
	1244	{ _MMIO(0x2710), 0x00000000 },
	1245	{ _MMIO(0x2714), 0x70800000 },
	1246	{ _MMIO(0x2720), 0x00000000 },
	1247	{ _MMIO(0x2724), 0x00800000 },
	1248	{ _MMIO(0x2770), 0x0000c000 },
	1249	{ _MMIO(0x2774), 0x0000e7ff },
	1250	{ _MMIO(0x2778), 0x00003000 },
	1251	{ _MMIO(0x277c), 0x0000f9ff },
	1252	{ _MMIO(0x2780), 0x00000c00 },
	1253	{ _MMIO(0x2784), 0x0000fe7f },
	1254	};
	1255
	1256	static const struct i915_oa_reg flex_eu_config_sampler_1[] = {
	1257	{ _MMIO(0xe458), 0x00005004 },
	1258	{ _MMIO(0xe558), 0x00010003 },
	1259	{ _MMIO(0xe658), 0x00012011 },
	1260	{ _MMIO(0xe758), 0x00015014 },
	1261	{ _MMIO(0xe45c), 0x00051050 },
	1262	{ _MMIO(0xe55c), 0x00053052 },
	1263	{ _MMIO(0xe65c), 0x00055054 },
	1264	};
	1265
	1266	static const struct i915_oa_reg mux_config_sampler_1[] = {
	1267	{ _MMIO(0x9888), 0x18921400 },
	1268	{ _MMIO(0x9888), 0x149500ab },
	1269	{ _MMIO(0x9888), 0x18b21400 },
	1270	{ _MMIO(0x9888), 0x14b500ab },
	1271	{ _MMIO(0x9888), 0x18d21400 },
	1272	{ _MMIO(0x9888), 0x14d500ab },
	1273	{ _MMIO(0x9888), 0x0cdc8000 },
	1274	{ _MMIO(0x9888), 0x0edc4000 },
	1275	{ _MMIO(0x9888), 0x02dcc000 },
	1276	{ _MMIO(0x9888), 0x04dcc000 },
	1277	{ _MMIO(0x9888), 0x1abd00a0 },
	1278	{ _MMIO(0x9888), 0x0abd8000 },
	1279	{ _MMIO(0x9888), 0x0cd88000 },
	1280	{ _MMIO(0x9888), 0x0ed84000 },
	1281	{ _MMIO(0x9888), 0x04d88000 },
	1282	{ _MMIO(0x9888), 0x1adb0050 },
	1283	{ _MMIO(0x9888), 0x04db8000 },
	1284	{ _MMIO(0x9888), 0x06db8000 },
	1285	{ _MMIO(0x9888), 0x08db8000 },
	1286	{ _MMIO(0x9888), 0x0adb4000 },
	1287	{ _MMIO(0x9888), 0x109f02a0 },
	1288	{ _MMIO(0x9888), 0x0c9fa000 },
	1289	{ _MMIO(0x9888), 0x0e9f00aa },
	1290	{ _MMIO(0x9888), 0x18b82500 },
	1291	{ _MMIO(0x9888), 0x02b88000 },
	1292	{ _MMIO(0x9888), 0x04b84000 },
	1293	{ _MMIO(0x9888), 0x06b84000 },
	1294	{ _MMIO(0x9888), 0x08b84000 },
	1295	{ _MMIO(0x9888), 0x0ab84000 },
	1296	{ _MMIO(0x9888), 0x0cb88000 },
	1297	{ _MMIO(0x9888), 0x0cb98000 },
	1298	{ _MMIO(0x9888), 0x0eb9a000 },
	1299	{ _MMIO(0x9888), 0x00b98000 },
	1300	{ _MMIO(0x9888), 0x02b9a000 },
	1301	{ _MMIO(0x9888), 0x04b9a000 },
	1302	{ _MMIO(0x9888), 0x06b92000 },
	1303	{ _MMIO(0x9888), 0x1aba0200 },
	1304	{ _MMIO(0x9888), 0x02ba8000 },
	1305	{ _MMIO(0x9888), 0x0cba8000 },
	1306	{ _MMIO(0x9888), 0x04908000 },
	1307	{ _MMIO(0x9888), 0x04918000 },
	1308	{ _MMIO(0x9888), 0x04927300 },
	1309	{ _MMIO(0x9888), 0x10920000 },
	1310	{ _MMIO(0x9888), 0x1893000a },
	1311	{ _MMIO(0x9888), 0x0a934000 },
	1312	{ _MMIO(0x9888), 0x0a946000 },
	1313	{ _MMIO(0x9888), 0x0c959000 },
	1314	{ _MMIO(0x9888), 0x0e950098 },
	1315	{ _MMIO(0x9888), 0x10950000 },
	1316	{ _MMIO(0x9888), 0x04b04000 },
	1317	{ _MMIO(0x9888), 0x04b14000 },
	1318	{ _MMIO(0x9888), 0x04b20073 },
	1319	{ _MMIO(0x9888), 0x10b20000 },
	1320	{ _MMIO(0x9888), 0x04b38000 },
	1321	{ _MMIO(0x9888), 0x06b38000 },
	1322	{ _MMIO(0x9888), 0x08b34000 },
	1323	{ _MMIO(0x9888), 0x04b4c000 },
	1324	{ _MMIO(0x9888), 0x02b59890 },
	1325	{ _MMIO(0x9888), 0x10b50000 },
	1326	{ _MMIO(0x9888), 0x06d04000 },
	1327	{ _MMIO(0x9888), 0x06d14000 },
	1328	{ _MMIO(0x9888), 0x06d20073 },
	1329	{ _MMIO(0x9888), 0x10d20000 },
	1330	{ _MMIO(0x9888), 0x18d30020 },
	1331	{ _MMIO(0x9888), 0x02d38000 },
	1332	{ _MMIO(0x9888), 0x0cd34000 },
	1333	{ _MMIO(0x9888), 0x0ad48000 },
	1334	{ _MMIO(0x9888), 0x04d42000 },
	1335	{ _MMIO(0x9888), 0x0ed59000 },
	1336	{ _MMIO(0x9888), 0x00d59800 },
	1337	{ _MMIO(0x9888), 0x10d50000 },
	1338	{ _MMIO(0x9888), 0x0f88000e },
	1339	{ _MMIO(0x9888), 0x03888000 },
	1340	{ _MMIO(0x9888), 0x05888000 },
	1341	{ _MMIO(0x9888), 0x07888000 },
	1342	{ _MMIO(0x9888), 0x09888000 },
	1343	{ _MMIO(0x9888), 0x0b888000 },
	1344	{ _MMIO(0x9888), 0x0d880400 },
	1345	{ _MMIO(0x9888), 0x278b002a },
	1346	{ _MMIO(0x9888), 0x238b5500 },
	1347	{ _MMIO(0x9888), 0x258b000a },
	1348	{ _MMIO(0x9888), 0x1b8c0015 },
	1349	{ _MMIO(0x9888), 0x038c4000 },
	1350	{ _MMIO(0x9888), 0x058c4000 },
	1351	{ _MMIO(0x9888), 0x078c4000 },
	1352	{ _MMIO(0x9888), 0x098c4000 },
	1353	{ _MMIO(0x9888), 0x0b8c4000 },
	1354	{ _MMIO(0x9888), 0x0d8c4000 },
	1355	{ _MMIO(0x9888), 0x0d8d8000 },
	1356	{ _MMIO(0x9888), 0x0f8da000 },
	1357	{ _MMIO(0x9888), 0x018d8000 },
	1358	{ _MMIO(0x9888), 0x038da000 },
	1359	{ _MMIO(0x9888), 0x058da000 },
	1360	{ _MMIO(0x9888), 0x078d2000 },
	1361	{ _MMIO(0x9888), 0x2385002a },
	1362	{ _MMIO(0x9888), 0x1f85aa00 },
	1363	{ _MMIO(0x9888), 0x2185000a },
	1364	{ _MMIO(0x9888), 0x1b830150 },
	1365	{ _MMIO(0x9888), 0x03834000 },
	1366	{ _MMIO(0x9888), 0x05834000 },
	1367	{ _MMIO(0x9888), 0x07834000 },
	1368	{ _MMIO(0x9888), 0x09834000 },
	1369	{ _MMIO(0x9888), 0x0b834000 },
	1370	{ _MMIO(0x9888), 0x0d834000 },
	1371	{ _MMIO(0x9888), 0x0d848000 },
	1372	{ _MMIO(0x9888), 0x0f84c000 },
	1373	{ _MMIO(0x9888), 0x01848000 },
	1374	{ _MMIO(0x9888), 0x0384c000 },
	1375	{ _MMIO(0x9888), 0x0584c000 },
	1376	{ _MMIO(0x9888), 0x07844000 },
	1377	{ _MMIO(0x9888), 0x1d808000 },
	1378	{ _MMIO(0x9888), 0x1f80c000 },
	1379	{ _MMIO(0x9888), 0x11808000 },
	1380	{ _MMIO(0x9888), 0x1380c000 },
	1381	{ _MMIO(0x9888), 0x1580c000 },
	1382	{ _MMIO(0x9888), 0x17804000 },
	1383	{ _MMIO(0x9888), 0x53800000 },
	1384	{ _MMIO(0x9888), 0x47801021 },
	1385	{ _MMIO(0x9888), 0x21800000 },
	1386	{ _MMIO(0x9888), 0x31800000 },
	1387	{ _MMIO(0x9888), 0x4d800000 },
	1388	{ _MMIO(0x9888), 0x3f800c64 },
	1389	{ _MMIO(0x9888), 0x4f800000 },
	1390	{ _MMIO(0x9888), 0x41800c02 },
	1391	};
	1392
	1393	static int
	1394	get_sampler_1_mux_config(struct drm_i915_private *dev_priv,
	1395	const struct i915_oa_reg **regs,
	1396	int *lens)
	1397	{
	1398	int n = 0;
	1399
	1400	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1401	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1402
	1403	regs[n] = mux_config_sampler_1;
	1404	lens[n] = ARRAY_SIZE(mux_config_sampler_1);
	1405	n++;
	1406
	1407	return n;
	1408	}
	1409
	1410	static const struct i915_oa_reg b_counter_config_sampler_2[] = {
	1411	{ _MMIO(0x2740), 0x00000000 },
	1412	{ _MMIO(0x2744), 0x00800000 },
	1413	{ _MMIO(0x2710), 0x00000000 },
	1414	{ _MMIO(0x2714), 0x70800000 },
	1415	{ _MMIO(0x2720), 0x00000000 },
	1416	{ _MMIO(0x2724), 0x00800000 },
	1417	{ _MMIO(0x2770), 0x0000c000 },
	1418	{ _MMIO(0x2774), 0x0000e7ff },
	1419	{ _MMIO(0x2778), 0x00003000 },
	1420	{ _MMIO(0x277c), 0x0000f9ff },
	1421	{ _MMIO(0x2780), 0x00000c00 },
	1422	{ _MMIO(0x2784), 0x0000fe7f },
	1423	};
	1424
	1425	static const struct i915_oa_reg flex_eu_config_sampler_2[] = {
	1426	{ _MMIO(0xe458), 0x00005004 },
	1427	{ _MMIO(0xe558), 0x00010003 },
	1428	{ _MMIO(0xe658), 0x00012011 },
	1429	{ _MMIO(0xe758), 0x00015014 },
	1430	{ _MMIO(0xe45c), 0x00051050 },
	1431	{ _MMIO(0xe55c), 0x00053052 },
	1432	{ _MMIO(0xe65c), 0x00055054 },
	1433	};
	1434
	1435	static const struct i915_oa_reg mux_config_sampler_2[] = {
	1436	{ _MMIO(0x9888), 0x18121400 },
	1437	{ _MMIO(0x9888), 0x141500ab },
	1438	{ _MMIO(0x9888), 0x18321400 },
	1439	{ _MMIO(0x9888), 0x143500ab },
	1440	{ _MMIO(0x9888), 0x18521400 },
	1441	{ _MMIO(0x9888), 0x145500ab },
	1442	{ _MMIO(0x9888), 0x0c5c8000 },
	1443	{ _MMIO(0x9888), 0x0e5c4000 },
	1444	{ _MMIO(0x9888), 0x025cc000 },
	1445	{ _MMIO(0x9888), 0x045cc000 },
	1446	{ _MMIO(0x9888), 0x1a3d00a0 },
	1447	{ _MMIO(0x9888), 0x0a3d8000 },
	1448	{ _MMIO(0x9888), 0x0c588000 },
	1449	{ _MMIO(0x9888), 0x0e584000 },
	1450	{ _MMIO(0x9888), 0x04588000 },
	1451	{ _MMIO(0x9888), 0x1a5b0050 },
	1452	{ _MMIO(0x9888), 0x045b8000 },
	1453	{ _MMIO(0x9888), 0x065b8000 },
	1454	{ _MMIO(0x9888), 0x085b8000 },
	1455	{ _MMIO(0x9888), 0x0a5b4000 },
	1456	{ _MMIO(0x9888), 0x101f02a0 },
	1457	{ _MMIO(0x9888), 0x0c1fa000 },
	1458	{ _MMIO(0x9888), 0x0e1f00aa },
	1459	{ _MMIO(0x9888), 0x18382500 },
	1460	{ _MMIO(0x9888), 0x02388000 },
	1461	{ _MMIO(0x9888), 0x04384000 },
	1462	{ _MMIO(0x9888), 0x06384000 },
	1463	{ _MMIO(0x9888), 0x08384000 },
	1464	{ _MMIO(0x9888), 0x0a384000 },
	1465	{ _MMIO(0x9888), 0x0c388000 },
	1466	{ _MMIO(0x9888), 0x0c398000 },
	1467	{ _MMIO(0x9888), 0x0e39a000 },
	1468	{ _MMIO(0x9888), 0x00398000 },
	1469	{ _MMIO(0x9888), 0x0239a000 },
	1470	{ _MMIO(0x9888), 0x0439a000 },
	1471	{ _MMIO(0x9888), 0x06392000 },
	1472	{ _MMIO(0x9888), 0x1a3a0200 },
	1473	{ _MMIO(0x9888), 0x023a8000 },
	1474	{ _MMIO(0x9888), 0x0c3a8000 },
	1475	{ _MMIO(0x9888), 0x04108000 },
	1476	{ _MMIO(0x9888), 0x04118000 },
	1477	{ _MMIO(0x9888), 0x04127300 },
	1478	{ _MMIO(0x9888), 0x10120000 },
	1479	{ _MMIO(0x9888), 0x1813000a },
	1480	{ _MMIO(0x9888), 0x0a134000 },
	1481	{ _MMIO(0x9888), 0x0a146000 },
	1482	{ _MMIO(0x9888), 0x0c159000 },
	1483	{ _MMIO(0x9888), 0x0e150098 },
	1484	{ _MMIO(0x9888), 0x10150000 },
	1485	{ _MMIO(0x9888), 0x04304000 },
	1486	{ _MMIO(0x9888), 0x04314000 },
	1487	{ _MMIO(0x9888), 0x04320073 },
	1488	{ _MMIO(0x9888), 0x10320000 },
	1489	{ _MMIO(0x9888), 0x04338000 },
	1490	{ _MMIO(0x9888), 0x06338000 },
	1491	{ _MMIO(0x9888), 0x08334000 },
	1492	{ _MMIO(0x9888), 0x0434c000 },
	1493	{ _MMIO(0x9888), 0x02359890 },
	1494	{ _MMIO(0x9888), 0x10350000 },
	1495	{ _MMIO(0x9888), 0x06504000 },
	1496	{ _MMIO(0x9888), 0x06514000 },
	1497	{ _MMIO(0x9888), 0x06520073 },
	1498	{ _MMIO(0x9888), 0x10520000 },
	1499	{ _MMIO(0x9888), 0x18530020 },
	1500	{ _MMIO(0x9888), 0x02538000 },
	1501	{ _MMIO(0x9888), 0x0c534000 },
	1502	{ _MMIO(0x9888), 0x0a548000 },
	1503	{ _MMIO(0x9888), 0x04542000 },
	1504	{ _MMIO(0x9888), 0x0e559000 },
	1505	{ _MMIO(0x9888), 0x00559800 },
	1506	{ _MMIO(0x9888), 0x10550000 },
	1507	{ _MMIO(0x9888), 0x1b8aa000 },
	1508	{ _MMIO(0x9888), 0x1d8a0002 },
	1509	{ _MMIO(0x9888), 0x038a8000 },
	1510	{ _MMIO(0x9888), 0x058a8000 },
	1511	{ _MMIO(0x9888), 0x078a8000 },
	1512	{ _MMIO(0x9888), 0x098a8000 },
	1513	{ _MMIO(0x9888), 0x0b8a8000 },
	1514	{ _MMIO(0x9888), 0x0d8a8000 },
	1515	{ _MMIO(0x9888), 0x278b0015 },
	1516	{ _MMIO(0x9888), 0x238b2a80 },
	1517	{ _MMIO(0x9888), 0x258b0005 },
	1518	{ _MMIO(0x9888), 0x2385002a },
	1519	{ _MMIO(0x9888), 0x1f85aa00 },
	1520	{ _MMIO(0x9888), 0x2185000a },
	1521	{ _MMIO(0x9888), 0x1b830150 },
	1522	{ _MMIO(0x9888), 0x03834000 },
	1523	{ _MMIO(0x9888), 0x05834000 },
	1524	{ _MMIO(0x9888), 0x07834000 },
	1525	{ _MMIO(0x9888), 0x09834000 },
	1526	{ _MMIO(0x9888), 0x0b834000 },
	1527	{ _MMIO(0x9888), 0x0d834000 },
	1528	{ _MMIO(0x9888), 0x0d848000 },
	1529	{ _MMIO(0x9888), 0x0f84c000 },
	1530	{ _MMIO(0x9888), 0x01848000 },
	1531	{ _MMIO(0x9888), 0x0384c000 },
	1532	{ _MMIO(0x9888), 0x0584c000 },
	1533	{ _MMIO(0x9888), 0x07844000 },
	1534	{ _MMIO(0x9888), 0x1d808000 },
	1535	{ _MMIO(0x9888), 0x1f80c000 },
	1536	{ _MMIO(0x9888), 0x11808000 },
	1537	{ _MMIO(0x9888), 0x1380c000 },
	1538	{ _MMIO(0x9888), 0x1580c000 },
	1539	{ _MMIO(0x9888), 0x17804000 },
	1540	{ _MMIO(0x9888), 0x53800000 },
	1541	{ _MMIO(0x9888), 0x47801021 },
	1542	{ _MMIO(0x9888), 0x21800000 },
	1543	{ _MMIO(0x9888), 0x31800000 },
	1544	{ _MMIO(0x9888), 0x4d800000 },
	1545	{ _MMIO(0x9888), 0x3f800c64 },
	1546	{ _MMIO(0x9888), 0x4f800000 },
	1547	{ _MMIO(0x9888), 0x41800c02 },
	1548	};
	1549
	1550	static int
	1551	get_sampler_2_mux_config(struct drm_i915_private *dev_priv,
	1552	const struct i915_oa_reg **regs,
	1553	int *lens)
	1554	{
	1555	int n = 0;
	1556
	1557	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1558	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1559
	1560	regs[n] = mux_config_sampler_2;
	1561	lens[n] = ARRAY_SIZE(mux_config_sampler_2);
	1562	n++;
	1563
	1564	return n;
	1565	}
	1566
	1567	static const struct i915_oa_reg b_counter_config_tdl_1[] = {
	1568	{ _MMIO(0x2740), 0x00000000 },
	1569	{ _MMIO(0x2744), 0x00800000 },
	1570	{ _MMIO(0x2710), 0x00000000 },
	1571	{ _MMIO(0x2714), 0xf0800000 },
	1572	{ _MMIO(0x2720), 0x00000000 },
	1573	{ _MMIO(0x2724), 0x30800000 },
	1574	{ _MMIO(0x2770), 0x00000002 },
	1575	{ _MMIO(0x2774), 0x0000fdff },
	1576	{ _MMIO(0x2778), 0x00000000 },
	1577	{ _MMIO(0x277c), 0x0000fe7f },
	1578	{ _MMIO(0x2780), 0x00000002 },
	1579	{ _MMIO(0x2784), 0x0000ffbf },
	1580	{ _MMIO(0x2788), 0x00000000 },
	1581	{ _MMIO(0x278c), 0x0000ffcf },
	1582	{ _MMIO(0x2790), 0x00000002 },
	1583	{ _MMIO(0x2794), 0x0000fff7 },
	1584	{ _MMIO(0x2798), 0x00000000 },
	1585	{ _MMIO(0x279c), 0x0000fff9 },
	1586	};
	1587
	1588	static const struct i915_oa_reg flex_eu_config_tdl_1[] = {
	1589	{ _MMIO(0xe458), 0x00005004 },
	1590	{ _MMIO(0xe558), 0x00010003 },
	1591	{ _MMIO(0xe658), 0x00012011 },
	1592	{ _MMIO(0xe758), 0x00015014 },
	1593	{ _MMIO(0xe45c), 0x00051050 },
	1594	{ _MMIO(0xe55c), 0x00053052 },
	1595	{ _MMIO(0xe65c), 0x00055054 },
	1596	};
	1597
	1598	static const struct i915_oa_reg mux_config_tdl_1[] = {
	1599	{ _MMIO(0x9888), 0x16154d60 },
	1600	{ _MMIO(0x9888), 0x16352e60 },
	1601	{ _MMIO(0x9888), 0x16554d60 },
	1602	{ _MMIO(0x9888), 0x16950000 },
	1603	{ _MMIO(0x9888), 0x16b50000 },
	1604	{ _MMIO(0x9888), 0x16d50000 },
	1605	{ _MMIO(0x9888), 0x005c8000 },
	1606	{ _MMIO(0x9888), 0x045cc000 },
	1607	{ _MMIO(0x9888), 0x065c4000 },
	1608	{ _MMIO(0x9888), 0x083d8000 },
	1609	{ _MMIO(0x9888), 0x0a3d8000 },
	1610	{ _MMIO(0x9888), 0x0458c000 },
	1611	{ _MMIO(0x9888), 0x025b8000 },
	1612	{ _MMIO(0x9888), 0x085b4000 },
	1613	{ _MMIO(0x9888), 0x0a5b4000 },
	1614	{ _MMIO(0x9888), 0x0c5b8000 },
	1615	{ _MMIO(0x9888), 0x0c1fa000 },
	1616	{ _MMIO(0x9888), 0x0e1f00aa },
	1617	{ _MMIO(0x9888), 0x02384000 },
	1618	{ _MMIO(0x9888), 0x04388000 },
	1619	{ _MMIO(0x9888), 0x06388000 },
	1620	{ _MMIO(0x9888), 0x08384000 },
	1621	{ _MMIO(0x9888), 0x0a384000 },
	1622	{ _MMIO(0x9888), 0x0c384000 },
	1623	{ _MMIO(0x9888), 0x00398000 },
	1624	{ _MMIO(0x9888), 0x0239a000 },
	1625	{ _MMIO(0x9888), 0x0439a000 },
	1626	{ _MMIO(0x9888), 0x06392000 },
	1627	{ _MMIO(0x9888), 0x043a8000 },
	1628	{ _MMIO(0x9888), 0x063a8000 },
	1629	{ _MMIO(0x9888), 0x08138000 },
	1630	{ _MMIO(0x9888), 0x0a138000 },
	1631	{ _MMIO(0x9888), 0x06143000 },
	1632	{ _MMIO(0x9888), 0x0415cfc7 },
	1633	{ _MMIO(0x9888), 0x10150000 },
	1634	{ _MMIO(0x9888), 0x02338000 },
	1635	{ _MMIO(0x9888), 0x0c338000 },
	1636	{ _MMIO(0x9888), 0x04342000 },
	1637	{ _MMIO(0x9888), 0x06344000 },
	1638	{ _MMIO(0x9888), 0x0035c700 },
	1639	{ _MMIO(0x9888), 0x063500cf },
	1640	{ _MMIO(0x9888), 0x10350000 },
	1641	{ _MMIO(0x9888), 0x04538000 },
	1642	{ _MMIO(0x9888), 0x06538000 },
	1643	{ _MMIO(0x9888), 0x0454c000 },
	1644	{ _MMIO(0x9888), 0x0255cfc7 },
	1645	{ _MMIO(0x9888), 0x10550000 },
	1646	{ _MMIO(0x9888), 0x06dc8000 },
	1647	{ _MMIO(0x9888), 0x08dc4000 },
	1648	{ _MMIO(0x9888), 0x0cdcc000 },
	1649	{ _MMIO(0x9888), 0x0edcc000 },
	1650	{ _MMIO(0x9888), 0x1abd00a8 },
	1651	{ _MMIO(0x9888), 0x0cd8c000 },
	1652	{ _MMIO(0x9888), 0x0ed84000 },
	1653	{ _MMIO(0x9888), 0x0edb8000 },
	1654	{ _MMIO(0x9888), 0x18db0800 },
	1655	{ _MMIO(0x9888), 0x1adb0254 },
	1656	{ _MMIO(0x9888), 0x0e9faa00 },
	1657	{ _MMIO(0x9888), 0x109f02aa },
	1658	{ _MMIO(0x9888), 0x0eb84000 },
	1659	{ _MMIO(0x9888), 0x16b84000 },
	1660	{ _MMIO(0x9888), 0x18b8156a },
	1661	{ _MMIO(0x9888), 0x06b98000 },
	1662	{ _MMIO(0x9888), 0x08b9a000 },
	1663	{ _MMIO(0x9888), 0x0ab9a000 },
	1664	{ _MMIO(0x9888), 0x0cb9a000 },
	1665	{ _MMIO(0x9888), 0x0eb9a000 },
	1666	{ _MMIO(0x9888), 0x18baa000 },
	1667	{ _MMIO(0x9888), 0x1aba0002 },
	1668	{ _MMIO(0x9888), 0x16934000 },
	1669	{ _MMIO(0x9888), 0x1893000a },
	1670	{ _MMIO(0x9888), 0x0a947000 },
	1671	{ _MMIO(0x9888), 0x0c95c5c1 },
	1672	{ _MMIO(0x9888), 0x0e9500c3 },
	1673	{ _MMIO(0x9888), 0x10950000 },
	1674	{ _MMIO(0x9888), 0x0eb38000 },
	1675	{ _MMIO(0x9888), 0x16b30040 },
	1676	{ _MMIO(0x9888), 0x18b30020 },
	1677	{ _MMIO(0x9888), 0x06b48000 },
	1678	{ _MMIO(0x9888), 0x08b41000 },
	1679	{ _MMIO(0x9888), 0x0ab48000 },
	1680	{ _MMIO(0x9888), 0x06b5c500 },
	1681	{ _MMIO(0x9888), 0x08b500c3 },
	1682	{ _MMIO(0x9888), 0x0eb5c100 },
	1683	{ _MMIO(0x9888), 0x10b50000 },
	1684	{ _MMIO(0x9888), 0x16d31500 },
	1685	{ _MMIO(0x9888), 0x08d4e000 },
	1686	{ _MMIO(0x9888), 0x08d5c100 },
	1687	{ _MMIO(0x9888), 0x0ad5c3c5 },
	1688	{ _MMIO(0x9888), 0x10d50000 },
	1689	{ _MMIO(0x9888), 0x0d88f800 },
	1690	{ _MMIO(0x9888), 0x0f88000f },
	1691	{ _MMIO(0x9888), 0x038a8000 },
	1692	{ _MMIO(0x9888), 0x058a8000 },
	1693	{ _MMIO(0x9888), 0x078a8000 },
	1694	{ _MMIO(0x9888), 0x098a8000 },
	1695	{ _MMIO(0x9888), 0x0b8a8000 },
	1696	{ _MMIO(0x9888), 0x0d8a8000 },
	1697	{ _MMIO(0x9888), 0x258baaa5 },
	1698	{ _MMIO(0x9888), 0x278b002a },
	1699	{ _MMIO(0x9888), 0x238b2a80 },
	1700	{ _MMIO(0x9888), 0x0f8c4000 },
	1701	{ _MMIO(0x9888), 0x178c2000 },
	1702	{ _MMIO(0x9888), 0x198c5500 },
	1703	{ _MMIO(0x9888), 0x1b8c0015 },
	1704	{ _MMIO(0x9888), 0x078d8000 },
	1705	{ _MMIO(0x9888), 0x098da000 },
	1706	{ _MMIO(0x9888), 0x0b8da000 },
	1707	{ _MMIO(0x9888), 0x0d8da000 },
	1708	{ _MMIO(0x9888), 0x0f8da000 },
	1709	{ _MMIO(0x9888), 0x2185aaaa },
	1710	{ _MMIO(0x9888), 0x2385002a },
	1711	{ _MMIO(0x9888), 0x1f85aa00 },
	1712	{ _MMIO(0x9888), 0x0f834000 },
	1713	{ _MMIO(0x9888), 0x19835400 },
	1714	{ _MMIO(0x9888), 0x1b830155 },
	1715	{ _MMIO(0x9888), 0x03834000 },
	1716	{ _MMIO(0x9888), 0x05834000 },
	1717	{ _MMIO(0x9888), 0x07834000 },
	1718	{ _MMIO(0x9888), 0x09834000 },
	1719	{ _MMIO(0x9888), 0x0b834000 },
	1720	{ _MMIO(0x9888), 0x0d834000 },
	1721	{ _MMIO(0x9888), 0x0784c000 },
	1722	{ _MMIO(0x9888), 0x0984c000 },
	1723	{ _MMIO(0x9888), 0x0b84c000 },
	1724	{ _MMIO(0x9888), 0x0d84c000 },
	1725	{ _MMIO(0x9888), 0x0f84c000 },
	1726	{ _MMIO(0x9888), 0x01848000 },
	1727	{ _MMIO(0x9888), 0x0384c000 },
	1728	{ _MMIO(0x9888), 0x0584c000 },
	1729	{ _MMIO(0x9888), 0x1780c000 },
	1730	{ _MMIO(0x9888), 0x1980c000 },
	1731	{ _MMIO(0x9888), 0x1b80c000 },
	1732	{ _MMIO(0x9888), 0x1d80c000 },
	1733	{ _MMIO(0x9888), 0x1f80c000 },
	1734	{ _MMIO(0x9888), 0x11808000 },
	1735	{ _MMIO(0x9888), 0x1380c000 },
	1736	{ _MMIO(0x9888), 0x1580c000 },
	1737	{ _MMIO(0x9888), 0x4f800000 },
	1738	{ _MMIO(0x9888), 0x43800c42 },
	1739	{ _MMIO(0x9888), 0x51800000 },
	1740	{ _MMIO(0x9888), 0x45800063 },
	1741	{ _MMIO(0x9888), 0x53800000 },
	1742	{ _MMIO(0x9888), 0x47800800 },
	1743	{ _MMIO(0x9888), 0x21800000 },
	1744	{ _MMIO(0x9888), 0x31800000 },
	1745	{ _MMIO(0x9888), 0x4d800000 },
	1746	{ _MMIO(0x9888), 0x3f8014a4 },
	1747	{ _MMIO(0x9888), 0x41801042 },
	1748	};
	1749
	1750	static int
	1751	get_tdl_1_mux_config(struct drm_i915_private *dev_priv,
	1752	const struct i915_oa_reg **regs,
	1753	int *lens)
	1754	{
	1755	int n = 0;
	1756
	1757	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1758	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1759
	1760	regs[n] = mux_config_tdl_1;
	1761	lens[n] = ARRAY_SIZE(mux_config_tdl_1);
	1762	n++;
	1763
	1764	return n;
	1765	}
	1766
	1767	static const struct i915_oa_reg b_counter_config_tdl_2[] = {
	1768	{ _MMIO(0x2740), 0x00000000 },
	1769	{ _MMIO(0x2744), 0x00800000 },
	1770	{ _MMIO(0x2710), 0x00000000 },
	1771	{ _MMIO(0x2714), 0xf0800000 },
	1772	{ _MMIO(0x2720), 0x00000000 },
	1773	{ _MMIO(0x2724), 0x30800000 },
	1774	{ _MMIO(0x2770), 0x00000002 },
	1775	{ _MMIO(0x2774), 0x0000fdff },
	1776	{ _MMIO(0x2778), 0x00000000 },
	1777	{ _MMIO(0x277c), 0x0000fe7f },
	1778	{ _MMIO(0x2780), 0x00000000 },
	1779	{ _MMIO(0x2784), 0x0000ff9f },
	1780	{ _MMIO(0x2788), 0x00000000 },
	1781	{ _MMIO(0x278c), 0x0000ffe7 },
	1782	{ _MMIO(0x2790), 0x00000002 },
	1783	{ _MMIO(0x2794), 0x0000fffb },
	1784	{ _MMIO(0x2798), 0x00000002 },
	1785	{ _MMIO(0x279c), 0x0000fffd },
	1786	};
	1787
	1788	static const struct i915_oa_reg flex_eu_config_tdl_2[] = {
	1789	{ _MMIO(0xe458), 0x00005004 },
	1790	{ _MMIO(0xe558), 0x00010003 },
	1791	{ _MMIO(0xe658), 0x00012011 },
	1792	{ _MMIO(0xe758), 0x00015014 },
	1793	{ _MMIO(0xe45c), 0x00051050 },
	1794	{ _MMIO(0xe55c), 0x00053052 },
	1795	{ _MMIO(0xe65c), 0x00055054 },
	1796	};
	1797
	1798	static const struct i915_oa_reg mux_config_tdl_2[] = {
	1799	{ _MMIO(0x9888), 0x16150000 },
	1800	{ _MMIO(0x9888), 0x16350000 },
	1801	{ _MMIO(0x9888), 0x16550000 },
	1802	{ _MMIO(0x9888), 0x16952e60 },
	1803	{ _MMIO(0x9888), 0x16b54d60 },
	1804	{ _MMIO(0x9888), 0x16d52e60 },
	1805	{ _MMIO(0x9888), 0x065c8000 },
	1806	{ _MMIO(0x9888), 0x085cc000 },
	1807	{ _MMIO(0x9888), 0x0a5cc000 },
	1808	{ _MMIO(0x9888), 0x0c5c4000 },
	1809	{ _MMIO(0x9888), 0x0e3d8000 },
	1810	{ _MMIO(0x9888), 0x183da000 },
	1811	{ _MMIO(0x9888), 0x06588000 },
	1812	{ _MMIO(0x9888), 0x08588000 },
	1813	{ _MMIO(0x9888), 0x0a584000 },
	1814	{ _MMIO(0x9888), 0x0e5b4000 },
	1815	{ _MMIO(0x9888), 0x185b5800 },
	1816	{ _MMIO(0x9888), 0x1a5b000a },
	1817	{ _MMIO(0x9888), 0x0e1faa00 },
	1818	{ _MMIO(0x9888), 0x101f02aa },
	1819	{ _MMIO(0x9888), 0x0e384000 },
	1820	{ _MMIO(0x9888), 0x16384000 },
	1821	{ _MMIO(0x9888), 0x18382a55 },
	1822	{ _MMIO(0x9888), 0x06398000 },
	1823	{ _MMIO(0x9888), 0x0839a000 },
	1824	{ _MMIO(0x9888), 0x0a39a000 },
	1825	{ _MMIO(0x9888), 0x0c39a000 },
	1826	{ _MMIO(0x9888), 0x0e39a000 },
	1827	{ _MMIO(0x9888), 0x1a3a02a0 },
	1828	{ _MMIO(0x9888), 0x0e138000 },
	1829	{ _MMIO(0x9888), 0x16130500 },
	1830	{ _MMIO(0x9888), 0x06148000 },
	1831	{ _MMIO(0x9888), 0x08146000 },
	1832	{ _MMIO(0x9888), 0x0615c100 },
	1833	{ _MMIO(0x9888), 0x0815c500 },
	1834	{ _MMIO(0x9888), 0x0a1500c3 },
	1835	{ _MMIO(0x9888), 0x10150000 },
	1836	{ _MMIO(0x9888), 0x16335040 },
	1837	{ _MMIO(0x9888), 0x08349000 },
	1838	{ _MMIO(0x9888), 0x0a341000 },
	1839	{ _MMIO(0x9888), 0x083500c1 },
	1840	{ _MMIO(0x9888), 0x0a35c500 },
	1841	{ _MMIO(0x9888), 0x0c3500c3 },
	1842	{ _MMIO(0x9888), 0x10350000 },
	1843	{ _MMIO(0x9888), 0x1853002a },
	1844	{ _MMIO(0x9888), 0x0a54e000 },
	1845	{ _MMIO(0x9888), 0x0c55c500 },
	1846	{ _MMIO(0x9888), 0x0e55c1c3 },
	1847	{ _MMIO(0x9888), 0x10550000 },
	1848	{ _MMIO(0x9888), 0x00dc8000 },
	1849	{ _MMIO(0x9888), 0x02dcc000 },
	1850	{ _MMIO(0x9888), 0x04dc4000 },
	1851	{ _MMIO(0x9888), 0x04bd8000 },
	1852	{ _MMIO(0x9888), 0x06bd8000 },
	1853	{ _MMIO(0x9888), 0x02d8c000 },
	1854	{ _MMIO(0x9888), 0x02db8000 },
	1855	{ _MMIO(0x9888), 0x04db4000 },
	1856	{ _MMIO(0x9888), 0x06db4000 },
	1857	{ _MMIO(0x9888), 0x08db8000 },
	1858	{ _MMIO(0x9888), 0x0c9fa000 },
	1859	{ _MMIO(0x9888), 0x0e9f00aa },
	1860	{ _MMIO(0x9888), 0x02b84000 },
	1861	{ _MMIO(0x9888), 0x04b84000 },
	1862	{ _MMIO(0x9888), 0x06b84000 },
	1863	{ _MMIO(0x9888), 0x08b84000 },
	1864	{ _MMIO(0x9888), 0x0ab88000 },
	1865	{ _MMIO(0x9888), 0x0cb88000 },
	1866	{ _MMIO(0x9888), 0x00b98000 },
	1867	{ _MMIO(0x9888), 0x02b9a000 },
	1868	{ _MMIO(0x9888), 0x04b9a000 },
	1869	{ _MMIO(0x9888), 0x06b92000 },
	1870	{ _MMIO(0x9888), 0x0aba8000 },
	1871	{ _MMIO(0x9888), 0x0cba8000 },
	1872	{ _MMIO(0x9888), 0x04938000 },
	1873	{ _MMIO(0x9888), 0x06938000 },
	1874	{ _MMIO(0x9888), 0x0494c000 },
	1875	{ _MMIO(0x9888), 0x0295cfc7 },
	1876	{ _MMIO(0x9888), 0x10950000 },
	1877	{ _MMIO(0x9888), 0x02b38000 },
	1878	{ _MMIO(0x9888), 0x08b38000 },
	1879	{ _MMIO(0x9888), 0x04b42000 },
	1880	{ _MMIO(0x9888), 0x06b41000 },
	1881	{ _MMIO(0x9888), 0x00b5c700 },
	1882	{ _MMIO(0x9888), 0x04b500cf },
	1883	{ _MMIO(0x9888), 0x10b50000 },
	1884	{ _MMIO(0x9888), 0x0ad38000 },
	1885	{ _MMIO(0x9888), 0x0cd38000 },
	1886	{ _MMIO(0x9888), 0x06d46000 },
	1887	{ _MMIO(0x9888), 0x04d5c700 },
	1888	{ _MMIO(0x9888), 0x06d500cf },
	1889	{ _MMIO(0x9888), 0x10d50000 },
	1890	{ _MMIO(0x9888), 0x03888000 },
	1891	{ _MMIO(0x9888), 0x05888000 },
	1892	{ _MMIO(0x9888), 0x07888000 },
	1893	{ _MMIO(0x9888), 0x09888000 },
	1894	{ _MMIO(0x9888), 0x0b888000 },
	1895	{ _MMIO(0x9888), 0x0d880400 },
	1896	{ _MMIO(0x9888), 0x0f8a8000 },
	1897	{ _MMIO(0x9888), 0x198a8000 },
	1898	{ _MMIO(0x9888), 0x1b8aaaa0 },
	1899	{ _MMIO(0x9888), 0x1d8a0002 },
	1900	{ _MMIO(0x9888), 0x258b555a },
	1901	{ _MMIO(0x9888), 0x278b0015 },
	1902	{ _MMIO(0x9888), 0x238b5500 },
	1903	{ _MMIO(0x9888), 0x038c4000 },
	1904	{ _MMIO(0x9888), 0x058c4000 },
	1905	{ _MMIO(0x9888), 0x078c4000 },
	1906	{ _MMIO(0x9888), 0x098c4000 },
	1907	{ _MMIO(0x9888), 0x0b8c4000 },
	1908	{ _MMIO(0x9888), 0x0d8c4000 },
	1909	{ _MMIO(0x9888), 0x018d8000 },
	1910	{ _MMIO(0x9888), 0x038da000 },
	1911	{ _MMIO(0x9888), 0x058da000 },
	1912	{ _MMIO(0x9888), 0x078d2000 },
	1913	{ _MMIO(0x9888), 0x2185aaaa },
	1914	{ _MMIO(0x9888), 0x2385002a },
	1915	{ _MMIO(0x9888), 0x1f85aa00 },
	1916	{ _MMIO(0x9888), 0x0f834000 },
	1917	{ _MMIO(0x9888), 0x19835400 },
	1918	{ _MMIO(0x9888), 0x1b830155 },
	1919	{ _MMIO(0x9888), 0x03834000 },
	1920	{ _MMIO(0x9888), 0x05834000 },
	1921	{ _MMIO(0x9888), 0x07834000 },
	1922	{ _MMIO(0x9888), 0x09834000 },
	1923	{ _MMIO(0x9888), 0x0b834000 },
	1924	{ _MMIO(0x9888), 0x0d834000 },
	1925	{ _MMIO(0x9888), 0x0784c000 },
	1926	{ _MMIO(0x9888), 0x0984c000 },
	1927	{ _MMIO(0x9888), 0x0b84c000 },
	1928	{ _MMIO(0x9888), 0x0d84c000 },
	1929	{ _MMIO(0x9888), 0x0f84c000 },
	1930	{ _MMIO(0x9888), 0x01848000 },
	1931	{ _MMIO(0x9888), 0x0384c000 },
	1932	{ _MMIO(0x9888), 0x0584c000 },
	1933	{ _MMIO(0x9888), 0x1780c000 },
	1934	{ _MMIO(0x9888), 0x1980c000 },
	1935	{ _MMIO(0x9888), 0x1b80c000 },
	1936	{ _MMIO(0x9888), 0x1d80c000 },
	1937	{ _MMIO(0x9888), 0x1f80c000 },
	1938	{ _MMIO(0x9888), 0x11808000 },
	1939	{ _MMIO(0x9888), 0x1380c000 },
	1940	{ _MMIO(0x9888), 0x1580c000 },
	1941	{ _MMIO(0x9888), 0x4f800000 },
	1942	{ _MMIO(0x9888), 0x43800882 },
	1943	{ _MMIO(0x9888), 0x51800000 },
	1944	{ _MMIO(0x9888), 0x45801082 },
	1945	{ _MMIO(0x9888), 0x53800000 },
	1946	{ _MMIO(0x9888), 0x478014a5 },
	1947	{ _MMIO(0x9888), 0x21800000 },
	1948	{ _MMIO(0x9888), 0x31800000 },
	1949	{ _MMIO(0x9888), 0x4d800000 },
	1950	{ _MMIO(0x9888), 0x3f800002 },
	1951	{ _MMIO(0x9888), 0x41800c62 },
	1952	};
	1953
	1954	static int
	1955	get_tdl_2_mux_config(struct drm_i915_private *dev_priv,
	1956	const struct i915_oa_reg **regs,
	1957	int *lens)
	1958	{
	1959	int n = 0;
	1960
	1961	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1962	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1963
	1964	regs[n] = mux_config_tdl_2;
	1965	lens[n] = ARRAY_SIZE(mux_config_tdl_2);
	1966	n++;
	1967
	1968	return n;
	1969	}
	1970
	1971	static const struct i915_oa_reg b_counter_config_test_oa[] = {
	1972	{ _MMIO(0x2740), 0x00000000 },
	1973	{ _MMIO(0x2744), 0x00800000 },
	1974	{ _MMIO(0x2714), 0xf0800000 },
	1975	{ _MMIO(0x2710), 0x00000000 },
	1976	{ _MMIO(0x2724), 0xf0800000 },
	1977	{ _MMIO(0x2720), 0x00000000 },
	1978	{ _MMIO(0x2770), 0x00000004 },
	1979	{ _MMIO(0x2774), 0x00000000 },
	1980	{ _MMIO(0x2778), 0x00000003 },
	1981	{ _MMIO(0x277c), 0x00000000 },
	1982	{ _MMIO(0x2780), 0x00000007 },
	1983	{ _MMIO(0x2784), 0x00000000 },
	1984	{ _MMIO(0x2788), 0x00100002 },
	1985	{ _MMIO(0x278c), 0x0000fff7 },
	1986	{ _MMIO(0x2790), 0x00100002 },
	1987	{ _MMIO(0x2794), 0x0000ffcf },
	1988	{ _MMIO(0x2798), 0x00100082 },
	1989	{ _MMIO(0x279c), 0x0000ffef },
	1990	{ _MMIO(0x27a0), 0x001000c2 },
	1991	{ _MMIO(0x27a4), 0x0000ffe7 },
	1992	{ _MMIO(0x27a8), 0x00100001 },
	1993	{ _MMIO(0x27ac), 0x0000ffe7 },
	1994	};
	1995
	1996	static const struct i915_oa_reg flex_eu_config_test_oa[] = {
	1997	};
	1998
	1999	static const struct i915_oa_reg mux_config_test_oa[] = {
	2000	{ _MMIO(0x9888), 0x59800000 },
	2001	{ _MMIO(0x9888), 0x59800001 },
	2002	{ _MMIO(0x9888), 0x338b0000 },
	2003	{ _MMIO(0x9888), 0x258b0066 },
	2004	{ _MMIO(0x9888), 0x058b0000 },
	2005	{ _MMIO(0x9888), 0x038b0000 },
	2006	{ _MMIO(0x9888), 0x03844000 },
	2007	{ _MMIO(0x9888), 0x47800080 },
	2008	{ _MMIO(0x9888), 0x57800000 },
	2009	{ _MMIO(0x1823a4), 0x00000000 },
	2010	{ _MMIO(0x9888), 0x59800000 },
	2011	};
	2012
	2013	static int
	2014	get_test_oa_mux_config(struct drm_i915_private *dev_priv,
	2015	const struct i915_oa_reg **regs,
	2016	int *lens)
	2017	{
	2018	int n = 0;
	2019
	2020	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	2021	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	2022
	2023	regs[n] = mux_config_test_oa;
	2024	lens[n] = ARRAY_SIZE(mux_config_test_oa);
	2025	n++;
	2026
	2027	return n;
	2028	}
	2029
	2030	int i915_oa_select_metric_set_chv(struct drm_i915_private *dev_priv)
	2031	{
	2032	dev_priv->perf.oa.n_mux_configs = 0;
	2033	dev_priv->perf.oa.b_counter_regs = NULL;
	2034	dev_priv->perf.oa.b_counter_regs_len = 0;
	2035	dev_priv->perf.oa.flex_regs = NULL;
	2036	dev_priv->perf.oa.flex_regs_len = 0;
	2037
	2038	switch (dev_priv->perf.oa.metrics_set) {
	2039	case METRIC_SET_ID_RENDER_BASIC:
	2040	dev_priv->perf.oa.n_mux_configs =
	2041	get_render_basic_mux_config(dev_priv,
	2042	dev_priv->perf.oa.mux_regs,
	2043	dev_priv->perf.oa.mux_regs_lens);
	2044	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2045	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_BASIC\" metric set\n");
	2046
	2047	/* EINVAL because *_register_sysfs already checked this
	2048	* and so it wouldn't have been advertised to userspace and
	2049	* so shouldn't have been requested
	2050	*/
	2051	return -EINVAL;
	2052	}
	2053
	2054	dev_priv->perf.oa.b_counter_regs =
	2055	b_counter_config_render_basic;
	2056	dev_priv->perf.oa.b_counter_regs_len =
	2057	ARRAY_SIZE(b_counter_config_render_basic);
	2058
	2059	dev_priv->perf.oa.flex_regs =
	2060	flex_eu_config_render_basic;
	2061	dev_priv->perf.oa.flex_regs_len =
	2062	ARRAY_SIZE(flex_eu_config_render_basic);
	2063
	2064	return 0;
	2065	case METRIC_SET_ID_COMPUTE_BASIC:
	2066	dev_priv->perf.oa.n_mux_configs =
	2067	get_compute_basic_mux_config(dev_priv,
	2068	dev_priv->perf.oa.mux_regs,
	2069	dev_priv->perf.oa.mux_regs_lens);
	2070	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2071	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_BASIC\" metric set\n");
	2072
	2073	/* EINVAL because *_register_sysfs already checked this
	2074	* and so it wouldn't have been advertised to userspace and
	2075	* so shouldn't have been requested
	2076	*/
	2077	return -EINVAL;
	2078	}
	2079
	2080	dev_priv->perf.oa.b_counter_regs =
	2081	b_counter_config_compute_basic;
	2082	dev_priv->perf.oa.b_counter_regs_len =
	2083	ARRAY_SIZE(b_counter_config_compute_basic);
	2084
	2085	dev_priv->perf.oa.flex_regs =
	2086	flex_eu_config_compute_basic;
	2087	dev_priv->perf.oa.flex_regs_len =
	2088	ARRAY_SIZE(flex_eu_config_compute_basic);
	2089
	2090	return 0;
	2091	case METRIC_SET_ID_RENDER_PIPE_PROFILE:
	2092	dev_priv->perf.oa.n_mux_configs =
	2093	get_render_pipe_profile_mux_config(dev_priv,
	2094	dev_priv->perf.oa.mux_regs,
	2095	dev_priv->perf.oa.mux_regs_lens);
	2096	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2097	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_PIPE_PROFILE\" metric set\n");
	2098
	2099	/* EINVAL because *_register_sysfs already checked this
	2100	* and so it wouldn't have been advertised to userspace and
	2101	* so shouldn't have been requested
	2102	*/
	2103	return -EINVAL;
	2104	}
	2105
	2106	dev_priv->perf.oa.b_counter_regs =
	2107	b_counter_config_render_pipe_profile;
	2108	dev_priv->perf.oa.b_counter_regs_len =
	2109	ARRAY_SIZE(b_counter_config_render_pipe_profile);
	2110
	2111	dev_priv->perf.oa.flex_regs =
	2112	flex_eu_config_render_pipe_profile;
	2113	dev_priv->perf.oa.flex_regs_len =
	2114	ARRAY_SIZE(flex_eu_config_render_pipe_profile);
	2115
	2116	return 0;
	2117	case METRIC_SET_ID_HDC_AND_SF:
	2118	dev_priv->perf.oa.n_mux_configs =
	2119	get_hdc_and_sf_mux_config(dev_priv,
	2120	dev_priv->perf.oa.mux_regs,
	2121	dev_priv->perf.oa.mux_regs_lens);
	2122	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2123	DRM_DEBUG_DRIVER("No suitable MUX config for \"HDC_AND_SF\" metric set\n");
	2124
	2125	/* EINVAL because *_register_sysfs already checked this
	2126	* and so it wouldn't have been advertised to userspace and
	2127	* so shouldn't have been requested
	2128	*/
	2129	return -EINVAL;
	2130	}
	2131
	2132	dev_priv->perf.oa.b_counter_regs =
	2133	b_counter_config_hdc_and_sf;
	2134	dev_priv->perf.oa.b_counter_regs_len =
	2135	ARRAY_SIZE(b_counter_config_hdc_and_sf);
	2136
	2137	dev_priv->perf.oa.flex_regs =
	2138	flex_eu_config_hdc_and_sf;
	2139	dev_priv->perf.oa.flex_regs_len =
	2140	ARRAY_SIZE(flex_eu_config_hdc_and_sf);
	2141
	2142	return 0;
	2143	case METRIC_SET_ID_L3_1:
	2144	dev_priv->perf.oa.n_mux_configs =
	2145	get_l3_1_mux_config(dev_priv,
	2146	dev_priv->perf.oa.mux_regs,
	2147	dev_priv->perf.oa.mux_regs_lens);
	2148	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2149	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_1\" metric set\n");
	2150
	2151	/* EINVAL because *_register_sysfs already checked this
	2152	* and so it wouldn't have been advertised to userspace and
	2153	* so shouldn't have been requested
	2154	*/
	2155	return -EINVAL;
	2156	}
	2157
	2158	dev_priv->perf.oa.b_counter_regs =
	2159	b_counter_config_l3_1;
	2160	dev_priv->perf.oa.b_counter_regs_len =
	2161	ARRAY_SIZE(b_counter_config_l3_1);
	2162
	2163	dev_priv->perf.oa.flex_regs =
	2164	flex_eu_config_l3_1;
	2165	dev_priv->perf.oa.flex_regs_len =
	2166	ARRAY_SIZE(flex_eu_config_l3_1);
	2167
	2168	return 0;
	2169	case METRIC_SET_ID_L3_2:
	2170	dev_priv->perf.oa.n_mux_configs =
	2171	get_l3_2_mux_config(dev_priv,
	2172	dev_priv->perf.oa.mux_regs,
	2173	dev_priv->perf.oa.mux_regs_lens);
	2174	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2175	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_2\" metric set\n");
	2176
	2177	/* EINVAL because *_register_sysfs already checked this
	2178	* and so it wouldn't have been advertised to userspace and
	2179	* so shouldn't have been requested
	2180	*/
	2181	return -EINVAL;
	2182	}
	2183
	2184	dev_priv->perf.oa.b_counter_regs =
	2185	b_counter_config_l3_2;
	2186	dev_priv->perf.oa.b_counter_regs_len =
	2187	ARRAY_SIZE(b_counter_config_l3_2);
	2188
	2189	dev_priv->perf.oa.flex_regs =
	2190	flex_eu_config_l3_2;
	2191	dev_priv->perf.oa.flex_regs_len =
	2192	ARRAY_SIZE(flex_eu_config_l3_2);
	2193
	2194	return 0;
	2195	case METRIC_SET_ID_L3_3:
	2196	dev_priv->perf.oa.n_mux_configs =
	2197	get_l3_3_mux_config(dev_priv,
	2198	dev_priv->perf.oa.mux_regs,
	2199	dev_priv->perf.oa.mux_regs_lens);
	2200	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2201	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_3\" metric set\n");
	2202
	2203	/* EINVAL because *_register_sysfs already checked this
	2204	* and so it wouldn't have been advertised to userspace and
	2205	* so shouldn't have been requested
	2206	*/
	2207	return -EINVAL;
	2208	}
	2209
	2210	dev_priv->perf.oa.b_counter_regs =
	2211	b_counter_config_l3_3;
	2212	dev_priv->perf.oa.b_counter_regs_len =
	2213	ARRAY_SIZE(b_counter_config_l3_3);
	2214
	2215	dev_priv->perf.oa.flex_regs =
	2216	flex_eu_config_l3_3;
	2217	dev_priv->perf.oa.flex_regs_len =
	2218	ARRAY_SIZE(flex_eu_config_l3_3);
	2219
	2220	return 0;
	2221	case METRIC_SET_ID_L3_4:
	2222	dev_priv->perf.oa.n_mux_configs =
	2223	get_l3_4_mux_config(dev_priv,
	2224	dev_priv->perf.oa.mux_regs,
	2225	dev_priv->perf.oa.mux_regs_lens);
	2226	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2227	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_4\" metric set\n");
	2228
	2229	/* EINVAL because *_register_sysfs already checked this
	2230	* and so it wouldn't have been advertised to userspace and
	2231	* so shouldn't have been requested
	2232	*/
	2233	return -EINVAL;
	2234	}
	2235
	2236	dev_priv->perf.oa.b_counter_regs =
	2237	b_counter_config_l3_4;
	2238	dev_priv->perf.oa.b_counter_regs_len =
	2239	ARRAY_SIZE(b_counter_config_l3_4);
	2240
	2241	dev_priv->perf.oa.flex_regs =
	2242	flex_eu_config_l3_4;
	2243	dev_priv->perf.oa.flex_regs_len =
	2244	ARRAY_SIZE(flex_eu_config_l3_4);
	2245
	2246	return 0;
	2247	case METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND:
	2248	dev_priv->perf.oa.n_mux_configs =
	2249	get_rasterizer_and_pixel_backend_mux_config(dev_priv,
	2250	dev_priv->perf.oa.mux_regs,
	2251	dev_priv->perf.oa.mux_regs_lens);
	2252	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2253	DRM_DEBUG_DRIVER("No suitable MUX config for \"RASTERIZER_AND_PIXEL_BACKEND\" metric set\n");
	2254
	2255	/* EINVAL because *_register_sysfs already checked this
	2256	* and so it wouldn't have been advertised to userspace and
	2257	* so shouldn't have been requested
	2258	*/
	2259	return -EINVAL;
	2260	}
	2261
	2262	dev_priv->perf.oa.b_counter_regs =
	2263	b_counter_config_rasterizer_and_pixel_backend;
	2264	dev_priv->perf.oa.b_counter_regs_len =
	2265	ARRAY_SIZE(b_counter_config_rasterizer_and_pixel_backend);
	2266
	2267	dev_priv->perf.oa.flex_regs =
	2268	flex_eu_config_rasterizer_and_pixel_backend;
	2269	dev_priv->perf.oa.flex_regs_len =
	2270	ARRAY_SIZE(flex_eu_config_rasterizer_and_pixel_backend);
	2271
	2272	return 0;
	2273	case METRIC_SET_ID_SAMPLER_1:
	2274	dev_priv->perf.oa.n_mux_configs =
	2275	get_sampler_1_mux_config(dev_priv,
	2276	dev_priv->perf.oa.mux_regs,
	2277	dev_priv->perf.oa.mux_regs_lens);
	2278	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2279	DRM_DEBUG_DRIVER("No suitable MUX config for \"SAMPLER_1\" metric set\n");
	2280
	2281	/* EINVAL because *_register_sysfs already checked this
	2282	* and so it wouldn't have been advertised to userspace and
	2283	* so shouldn't have been requested
	2284	*/
	2285	return -EINVAL;
	2286	}
	2287
	2288	dev_priv->perf.oa.b_counter_regs =
	2289	b_counter_config_sampler_1;
	2290	dev_priv->perf.oa.b_counter_regs_len =
	2291	ARRAY_SIZE(b_counter_config_sampler_1);
	2292
	2293	dev_priv->perf.oa.flex_regs =
	2294	flex_eu_config_sampler_1;
	2295	dev_priv->perf.oa.flex_regs_len =
	2296	ARRAY_SIZE(flex_eu_config_sampler_1);
	2297
	2298	return 0;
	2299	case METRIC_SET_ID_SAMPLER_2:
	2300	dev_priv->perf.oa.n_mux_configs =
	2301	get_sampler_2_mux_config(dev_priv,
	2302	dev_priv->perf.oa.mux_regs,
	2303	dev_priv->perf.oa.mux_regs_lens);
	2304	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2305	DRM_DEBUG_DRIVER("No suitable MUX config for \"SAMPLER_2\" metric set\n");
	2306
	2307	/* EINVAL because *_register_sysfs already checked this
	2308	* and so it wouldn't have been advertised to userspace and
	2309	* so shouldn't have been requested
	2310	*/
	2311	return -EINVAL;
	2312	}
	2313
	2314	dev_priv->perf.oa.b_counter_regs =
	2315	b_counter_config_sampler_2;
	2316	dev_priv->perf.oa.b_counter_regs_len =
	2317	ARRAY_SIZE(b_counter_config_sampler_2);
	2318
	2319	dev_priv->perf.oa.flex_regs =
	2320	flex_eu_config_sampler_2;
	2321	dev_priv->perf.oa.flex_regs_len =
	2322	ARRAY_SIZE(flex_eu_config_sampler_2);
	2323
	2324	return 0;
	2325	case METRIC_SET_ID_TDL_1:
	2326	dev_priv->perf.oa.n_mux_configs =
	2327	get_tdl_1_mux_config(dev_priv,
	2328	dev_priv->perf.oa.mux_regs,
	2329	dev_priv->perf.oa.mux_regs_lens);
	2330	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2331	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_1\" metric set\n");
	2332
	2333	/* EINVAL because *_register_sysfs already checked this
	2334	* and so it wouldn't have been advertised to userspace and
	2335	* so shouldn't have been requested
	2336	*/
	2337	return -EINVAL;
	2338	}
	2339
	2340	dev_priv->perf.oa.b_counter_regs =
	2341	b_counter_config_tdl_1;
	2342	dev_priv->perf.oa.b_counter_regs_len =
	2343	ARRAY_SIZE(b_counter_config_tdl_1);
	2344
	2345	dev_priv->perf.oa.flex_regs =
	2346	flex_eu_config_tdl_1;
	2347	dev_priv->perf.oa.flex_regs_len =
	2348	ARRAY_SIZE(flex_eu_config_tdl_1);
	2349
	2350	return 0;
	2351	case METRIC_SET_ID_TDL_2:
	2352	dev_priv->perf.oa.n_mux_configs =
	2353	get_tdl_2_mux_config(dev_priv,
	2354	dev_priv->perf.oa.mux_regs,
	2355	dev_priv->perf.oa.mux_regs_lens);
	2356	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2357	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_2\" metric set\n");
	2358
	2359	/* EINVAL because *_register_sysfs already checked this
	2360	* and so it wouldn't have been advertised to userspace and
	2361	* so shouldn't have been requested
	2362	*/
	2363	return -EINVAL;
	2364	}
	2365
	2366	dev_priv->perf.oa.b_counter_regs =
	2367	b_counter_config_tdl_2;
	2368	dev_priv->perf.oa.b_counter_regs_len =
	2369	ARRAY_SIZE(b_counter_config_tdl_2);
	2370
	2371	dev_priv->perf.oa.flex_regs =
	2372	flex_eu_config_tdl_2;
	2373	dev_priv->perf.oa.flex_regs_len =
	2374	ARRAY_SIZE(flex_eu_config_tdl_2);
	2375
	2376	return 0;
	2377	case METRIC_SET_ID_TEST_OA:
	2378	dev_priv->perf.oa.n_mux_configs =
	2379	get_test_oa_mux_config(dev_priv,
	2380	dev_priv->perf.oa.mux_regs,
	2381	dev_priv->perf.oa.mux_regs_lens);
	2382	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2383	DRM_DEBUG_DRIVER("No suitable MUX config for \"TEST_OA\" metric set\n");
	2384
	2385	/* EINVAL because *_register_sysfs already checked this
	2386	* and so it wouldn't have been advertised to userspace and
	2387	* so shouldn't have been requested
	2388	*/
	2389	return -EINVAL;
	2390	}
	2391
	2392	dev_priv->perf.oa.b_counter_regs =
	2393	b_counter_config_test_oa;
	2394	dev_priv->perf.oa.b_counter_regs_len =
	2395	ARRAY_SIZE(b_counter_config_test_oa);
	2396
	2397	dev_priv->perf.oa.flex_regs =
	2398	flex_eu_config_test_oa;
	2399	dev_priv->perf.oa.flex_regs_len =
	2400	ARRAY_SIZE(flex_eu_config_test_oa);
	2401
	2402	return 0;
	2403	default:
	2404	return -ENODEV;
	2405	}
	2406	}
	2407
	2408	static ssize_t
	2409	show_render_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	2410	{
	2411	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_BASIC);
	2412	}
	2413
	2414	static struct device_attribute dev_attr_render_basic_id = {
	2415	.attr = { .name = "id", .mode = 0444 },
	2416	.show = show_render_basic_id,
	2417	.store = NULL,
	2418	};
	2419
	2420	static struct attribute *attrs_render_basic[] = {
	2421	&dev_attr_render_basic_id.attr,
	2422	NULL,
	2423	};
	2424
	2425	static struct attribute_group group_render_basic = {
	2426	.name = "9d8a3af5-c02c-4a4a-b947-f1672469e0fb",
	2427	.attrs = attrs_render_basic,
	2428	};
	2429
	2430	static ssize_t
	2431	show_compute_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	2432	{
	2433	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_BASIC);
	2434	}
	2435
	2436	static struct device_attribute dev_attr_compute_basic_id = {
	2437	.attr = { .name = "id", .mode = 0444 },
	2438	.show = show_compute_basic_id,
	2439	.store = NULL,
	2440	};
	2441
	2442	static struct attribute *attrs_compute_basic[] = {
	2443	&dev_attr_compute_basic_id.attr,
	2444	NULL,
	2445	};
	2446
	2447	static struct attribute_group group_compute_basic = {
	2448	.name = "f522a89c-ecd1-4522-8331-3383c54af5f5",
	2449	.attrs = attrs_compute_basic,
	2450	};
	2451
	2452	static ssize_t
	2453	show_render_pipe_profile_id(struct device kdev, struct device_attribute attr, char *buf)
	2454	{
	2455	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_PIPE_PROFILE);
	2456	}
	2457
	2458	static struct device_attribute dev_attr_render_pipe_profile_id = {
	2459	.attr = { .name = "id", .mode = 0444 },
	2460	.show = show_render_pipe_profile_id,
	2461	.store = NULL,
	2462	};
	2463
	2464	static struct attribute *attrs_render_pipe_profile[] = {
	2465	&dev_attr_render_pipe_profile_id.attr,
	2466	NULL,
	2467	};
	2468
	2469	static struct attribute_group group_render_pipe_profile = {
	2470	.name = "a9ccc03d-a943-4e6b-9cd6-13e063075927",
	2471	.attrs = attrs_render_pipe_profile,
	2472	};
	2473
	2474	static ssize_t
	2475	show_hdc_and_sf_id(struct device kdev, struct device_attribute attr, char *buf)
	2476	{
	2477	return sprintf(buf, "%d\n", METRIC_SET_ID_HDC_AND_SF);
	2478	}
	2479
	2480	static struct device_attribute dev_attr_hdc_and_sf_id = {
	2481	.attr = { .name = "id", .mode = 0444 },
	2482	.show = show_hdc_and_sf_id,
	2483	.store = NULL,
	2484	};
	2485
	2486	static struct attribute *attrs_hdc_and_sf[] = {
	2487	&dev_attr_hdc_and_sf_id.attr,
	2488	NULL,
	2489	};
	2490
	2491	static struct attribute_group group_hdc_and_sf = {
	2492	.name = "2cf0c064-68df-4fac-9b3f-57f51ca8a069",
	2493	.attrs = attrs_hdc_and_sf,
	2494	};
	2495
	2496	static ssize_t
	2497	show_l3_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2498	{
	2499	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_1);
	2500	}
	2501
	2502	static struct device_attribute dev_attr_l3_1_id = {
	2503	.attr = { .name = "id", .mode = 0444 },
	2504	.show = show_l3_1_id,
	2505	.store = NULL,
	2506	};
	2507
	2508	static struct attribute *attrs_l3_1[] = {
	2509	&dev_attr_l3_1_id.attr,
	2510	NULL,
	2511	};
	2512
	2513	static struct attribute_group group_l3_1 = {
	2514	.name = "78a87ff9-543a-49ce-95ea-26d86071ea93",
	2515	.attrs = attrs_l3_1,
	2516	};
	2517
	2518	static ssize_t
	2519	show_l3_2_id(struct device kdev, struct device_attribute attr, char *buf)
	2520	{
	2521	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_2);
	2522	}
	2523
	2524	static struct device_attribute dev_attr_l3_2_id = {
	2525	.attr = { .name = "id", .mode = 0444 },
	2526	.show = show_l3_2_id,
	2527	.store = NULL,
	2528	};
	2529
	2530	static struct attribute *attrs_l3_2[] = {
	2531	&dev_attr_l3_2_id.attr,
	2532	NULL,
	2533	};
	2534
	2535	static struct attribute_group group_l3_2 = {
	2536	.name = "9f2cece5-7bfe-4320-ad66-8c7cc526bec5",
	2537	.attrs = attrs_l3_2,
	2538	};
	2539
	2540	static ssize_t
	2541	show_l3_3_id(struct device kdev, struct device_attribute attr, char *buf)
	2542	{
	2543	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_3);
	2544	}
	2545
	2546	static struct device_attribute dev_attr_l3_3_id = {
	2547	.attr = { .name = "id", .mode = 0444 },
	2548	.show = show_l3_3_id,
	2549	.store = NULL,
	2550	};
	2551
	2552	static struct attribute *attrs_l3_3[] = {
	2553	&dev_attr_l3_3_id.attr,
	2554	NULL,
	2555	};
	2556
	2557	static struct attribute_group group_l3_3 = {
	2558	.name = "d890ef38-d309-47e4-b8b5-aa779bb19ab0",
	2559	.attrs = attrs_l3_3,
	2560	};
	2561
	2562	static ssize_t
	2563	show_l3_4_id(struct device kdev, struct device_attribute attr, char *buf)
	2564	{
	2565	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_4);
	2566	}
	2567
	2568	static struct device_attribute dev_attr_l3_4_id = {
	2569	.attr = { .name = "id", .mode = 0444 },
	2570	.show = show_l3_4_id,
	2571	.store = NULL,
	2572	};
	2573
	2574	static struct attribute *attrs_l3_4[] = {
	2575	&dev_attr_l3_4_id.attr,
	2576	NULL,
	2577	};
	2578
	2579	static struct attribute_group group_l3_4 = {
	2580	.name = "5fdff4a6-9dc8-45e1-bfda-ef54869fbdd4",
	2581	.attrs = attrs_l3_4,
	2582	};
	2583
	2584	static ssize_t
	2585	show_rasterizer_and_pixel_backend_id(struct device kdev, struct device_attribute attr, char *buf)
	2586	{
	2587	return sprintf(buf, "%d\n", METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND);
	2588	}
	2589
	2590	static struct device_attribute dev_attr_rasterizer_and_pixel_backend_id = {
	2591	.attr = { .name = "id", .mode = 0444 },
	2592	.show = show_rasterizer_and_pixel_backend_id,
	2593	.store = NULL,
	2594	};
	2595
	2596	static struct attribute *attrs_rasterizer_and_pixel_backend[] = {
	2597	&dev_attr_rasterizer_and_pixel_backend_id.attr,
	2598	NULL,
	2599	};
	2600
	2601	static struct attribute_group group_rasterizer_and_pixel_backend = {
	2602	.name = "2c0e45e1-7e2c-4a14-ae00-0b7ec868b8aa",
	2603	.attrs = attrs_rasterizer_and_pixel_backend,
	2604	};
	2605
	2606	static ssize_t
	2607	show_sampler_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2608	{
	2609	return sprintf(buf, "%d\n", METRIC_SET_ID_SAMPLER_1);
	2610	}
	2611
	2612	static struct device_attribute dev_attr_sampler_1_id = {
	2613	.attr = { .name = "id", .mode = 0444 },
	2614	.show = show_sampler_1_id,
	2615	.store = NULL,
	2616	};
	2617
	2618	static struct attribute *attrs_sampler_1[] = {
	2619	&dev_attr_sampler_1_id.attr,
	2620	NULL,
	2621	};
	2622
	2623	static struct attribute_group group_sampler_1 = {
	2624	.name = "71148d78-baf5-474f-878a-e23158d0265d",
	2625	.attrs = attrs_sampler_1,
	2626	};
	2627
	2628	static ssize_t
	2629	show_sampler_2_id(struct device kdev, struct device_attribute attr, char *buf)
	2630	{
	2631	return sprintf(buf, "%d\n", METRIC_SET_ID_SAMPLER_2);
	2632	}
	2633
	2634	static struct device_attribute dev_attr_sampler_2_id = {
	2635	.attr = { .name = "id", .mode = 0444 },
	2636	.show = show_sampler_2_id,
	2637	.store = NULL,
	2638	};
	2639
	2640	static struct attribute *attrs_sampler_2[] = {
	2641	&dev_attr_sampler_2_id.attr,
	2642	NULL,
	2643	};
	2644
	2645	static struct attribute_group group_sampler_2 = {
	2646	.name = "b996a2b7-c59c-492d-877a-8cd54fd6df84",
	2647	.attrs = attrs_sampler_2,
	2648	};
	2649
	2650	static ssize_t
	2651	show_tdl_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2652	{
	2653	return sprintf(buf, "%d\n", METRIC_SET_ID_TDL_1);
	2654	}
	2655
	2656	static struct device_attribute dev_attr_tdl_1_id = {
	2657	.attr = { .name = "id", .mode = 0444 },
	2658	.show = show_tdl_1_id,
	2659	.store = NULL,
	2660	};
	2661
	2662	static struct attribute *attrs_tdl_1[] = {
	2663	&dev_attr_tdl_1_id.attr,
	2664	NULL,
	2665	};
	2666
	2667	static struct attribute_group group_tdl_1 = {
	2668	.name = "eb2fecba-b431-42e7-8261-fe9429a6e67a",
	2669	.attrs = attrs_tdl_1,
	2670	};
	2671
	2672	static ssize_t
	2673	show_tdl_2_id(struct device kdev, struct device_attribute attr, char *buf)
	2674	{
	2675	return sprintf(buf, "%d\n", METRIC_SET_ID_TDL_2);
	2676	}
	2677
	2678	static struct device_attribute dev_attr_tdl_2_id = {
	2679	.attr = { .name = "id", .mode = 0444 },
	2680	.show = show_tdl_2_id,
	2681	.store = NULL,
	2682	};
	2683
	2684	static struct attribute *attrs_tdl_2[] = {
	2685	&dev_attr_tdl_2_id.attr,
	2686	NULL,
	2687	};
	2688
	2689	static struct attribute_group group_tdl_2 = {
	2690	.name = "60749470-a648-4a4b-9f10-dbfe1e36e44d",
	2691	.attrs = attrs_tdl_2,
	2692	};
	2693
	2694	static ssize_t
	2695	show_test_oa_id(struct device kdev, struct device_attribute attr, char *buf)
	2696	{
	2697	return sprintf(buf, "%d\n", METRIC_SET_ID_TEST_OA);
	2698	}
	2699
	2700	static struct device_attribute dev_attr_test_oa_id = {
	2701	.attr = { .name = "id", .mode = 0444 },
	2702	.show = show_test_oa_id,
	2703	.store = NULL,
	2704	};
	2705
	2706	static struct attribute *attrs_test_oa[] = {
	2707	&dev_attr_test_oa_id.attr,
	2708	NULL,
	2709	};
	2710
	2711	static struct attribute_group group_test_oa = {
	2712	.name = "4a534b07-cba3-414d-8d60-874830e883aa",
	2713	.attrs = attrs_test_oa,
	2714	};
	2715
	2716	int
	2717	i915_perf_register_sysfs_chv(struct drm_i915_private *dev_priv)
	2718	{
	2719	const struct i915_oa_reg *mux_regs[ARRAY_SIZE(dev_priv->perf.oa.mux_regs)];
	2720	int mux_lens[ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens)];
	2721	int ret = 0;
	2722
	2723	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens)) {
	2724	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2725	if (ret)
	2726	goto error_render_basic;
	2727	}
	2728	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens)) {
	2729	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2730	if (ret)
	2731	goto error_compute_basic;
	2732	}
	2733	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens)) {
	2734	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2735	if (ret)
	2736	goto error_render_pipe_profile;
	2737	}
	2738	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens)) {
	2739	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2740	if (ret)
	2741	goto error_hdc_and_sf;
	2742	}
	2743	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2744	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2745	if (ret)
	2746	goto error_l3_1;
	2747	}
	2748	if (get_l3_2_mux_config(dev_priv, mux_regs, mux_lens)) {
	2749	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_l3_2);
	2750	if (ret)
	2751	goto error_l3_2;
	2752	}
	2753	if (get_l3_3_mux_config(dev_priv, mux_regs, mux_lens)) {
	2754	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_l3_3);
	2755	if (ret)
	2756	goto error_l3_3;
	2757	}
	2758	if (get_l3_4_mux_config(dev_priv, mux_regs, mux_lens)) {
	2759	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_l3_4);
	2760	if (ret)
	2761	goto error_l3_4;
	2762	}
	2763	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens)) {
	2764	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2765	if (ret)
	2766	goto error_rasterizer_and_pixel_backend;
	2767	}
	2768	if (get_sampler_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2769	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_sampler_1);
	2770	if (ret)
	2771	goto error_sampler_1;
	2772	}
	2773	if (get_sampler_2_mux_config(dev_priv, mux_regs, mux_lens)) {
	2774	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_sampler_2);
	2775	if (ret)
	2776	goto error_sampler_2;
	2777	}
	2778	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2779	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2780	if (ret)
	2781	goto error_tdl_1;
	2782	}
	2783	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens)) {
	2784	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2785	if (ret)
	2786	goto error_tdl_2;
	2787	}
	2788	if (get_test_oa_mux_config(dev_priv, mux_regs, mux_lens)) {
	2789	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_test_oa);
	2790	if (ret)
	2791	goto error_test_oa;
	2792	}
	2793
	2794	return 0;
	2795
	2796	error_test_oa:
	2797	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens))
	2798	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2799	error_tdl_2:
	2800	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens))
	2801	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2802	error_tdl_1:
	2803	if (get_sampler_2_mux_config(dev_priv, mux_regs, mux_lens))
	2804	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_sampler_2);
	2805	error_sampler_2:
	2806	if (get_sampler_1_mux_config(dev_priv, mux_regs, mux_lens))
	2807	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_sampler_1);
	2808	error_sampler_1:
	2809	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens))
	2810	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2811	error_rasterizer_and_pixel_backend:
	2812	if (get_l3_4_mux_config(dev_priv, mux_regs, mux_lens))
	2813	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_4);
	2814	error_l3_4:
	2815	if (get_l3_3_mux_config(dev_priv, mux_regs, mux_lens))
	2816	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_3);
	2817	error_l3_3:
	2818	if (get_l3_2_mux_config(dev_priv, mux_regs, mux_lens))
	2819	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_2);
	2820	error_l3_2:
	2821	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens))
	2822	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2823	error_l3_1:
	2824	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens))
	2825	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2826	error_hdc_and_sf:
	2827	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens))
	2828	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2829	error_render_pipe_profile:
	2830	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2831	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2832	error_compute_basic:
	2833	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2834	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2835	error_render_basic:
	2836	return ret;
	2837	}
	2838
	2839	void
	2840	i915_perf_unregister_sysfs_chv(struct drm_i915_private *dev_priv)
	2841	{
	2842	const struct i915_oa_reg *mux_regs[ARRAY_SIZE(dev_priv->perf.oa.mux_regs)];
	2843	int mux_lens[ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens)];
	2844
	2845	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2846	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2847	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens))
	2848	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2849	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens))
	2850	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2851	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens))
	2852	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2853	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens))
	2854	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2855	if (get_l3_2_mux_config(dev_priv, mux_regs, mux_lens))
	2856	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_2);
	2857	if (get_l3_3_mux_config(dev_priv, mux_regs, mux_lens))
	2858	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_3);
	2859	if (get_l3_4_mux_config(dev_priv, mux_regs, mux_lens))
	2860	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_l3_4);
	2861	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens))
	2862	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2863	if (get_sampler_1_mux_config(dev_priv, mux_regs, mux_lens))
	2864	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_sampler_1);
	2865	if (get_sampler_2_mux_config(dev_priv, mux_regs, mux_lens))
	2866	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_sampler_2);
	2867	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens))
	2868	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2869	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens))
	2870	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2871	if (get_test_oa_mux_config(dev_priv, mux_regs, mux_lens))
	2872	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_test_oa);
	2873	}

drivers/gpu/drm/i915/i915_oa_chv.h+40−0

@@ -0,0 +1,40 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#ifndef __I915_OA_CHV_H__
	30	#define __I915_OA_CHV_H__
	31
	32	extern int i915_oa_n_builtin_metric_sets_chv;
	33
	34	extern int i915_oa_select_metric_set_chv(struct drm_i915_private *dev_priv);
	35
	36	extern int i915_perf_register_sysfs_chv(struct drm_i915_private *dev_priv);
	37
	38	extern void i915_perf_unregister_sysfs_chv(struct drm_i915_private *dev_priv);
	39
	40	#endif

drivers/gpu/drm/i915/i915_oa_glk.c+2602−0

@@ -0,0 +1,2602 @@
	1	/*
	2	* Autogenerated file by GPU Top : https://github.com/rib/gputop
	3	* DO NOT EDIT manually!
	4	*
	5	*
	6	* Copyright (c) 2015 Intel Corporation
	7	*
	8	* Permission is hereby granted, free of charge, to any person obtaining a
	9	* copy of this software and associated documentation files (the "Software"),
	10	* to deal in the Software without restriction, including without limitation
	11	* the rights to use, copy, modify, merge, publish, distribute, sublicense,
	12	* and/or sell copies of the Software, and to permit persons to whom the
	13	* Software is furnished to do so, subject to the following conditions:
	14	*
	15	* The above copyright notice and this permission notice (including the next
	16	* paragraph) shall be included in all copies or substantial portions of the
	17	* Software.
	18	*
	19	* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	20	* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	21	* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
	22	* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	23	* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
	24	* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
	25	* IN THE SOFTWARE.
	26	*
	27	*/
	28
	29	#include <linux/sysfs.h>
	30
	31	#include "i915_drv.h"
	32	#include "i915_oa_glk.h"
	33
	34	enum metric_set_id {
	35	METRIC_SET_ID_RENDER_BASIC = 1,
	36	METRIC_SET_ID_COMPUTE_BASIC,
	37	METRIC_SET_ID_RENDER_PIPE_PROFILE,
	38	METRIC_SET_ID_MEMORY_READS,
	39	METRIC_SET_ID_MEMORY_WRITES,
	40	METRIC_SET_ID_COMPUTE_EXTENDED,
	41	METRIC_SET_ID_COMPUTE_L3_CACHE,
	42	METRIC_SET_ID_HDC_AND_SF,
	43	METRIC_SET_ID_L3_1,
	44	METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND,
	45	METRIC_SET_ID_SAMPLER,
	46	METRIC_SET_ID_TDL_1,
	47	METRIC_SET_ID_TDL_2,
	48	METRIC_SET_ID_COMPUTE_EXTRA,
	49	METRIC_SET_ID_TEST_OA,
	50	};
	51
	52	int i915_oa_n_builtin_metric_sets_glk = 15;
	53
	54	static const struct i915_oa_reg b_counter_config_render_basic[] = {
	55	{ _MMIO(0x2710), 0x00000000 },
	56	{ _MMIO(0x2714), 0x00800000 },
	57	{ _MMIO(0x2720), 0x00000000 },
	58	{ _MMIO(0x2724), 0x00800000 },
	59	{ _MMIO(0x2740), 0x00000000 },
	60	};
	61
	62	static const struct i915_oa_reg flex_eu_config_render_basic[] = {
	63	{ _MMIO(0xe458), 0x00005004 },
	64	{ _MMIO(0xe558), 0x00010003 },
	65	{ _MMIO(0xe658), 0x00012011 },
	66	{ _MMIO(0xe758), 0x00015014 },
	67	{ _MMIO(0xe45c), 0x00051050 },
	68	{ _MMIO(0xe55c), 0x00053052 },
	69	{ _MMIO(0xe65c), 0x00055054 },
	70	};
	71
	72	static const struct i915_oa_reg mux_config_render_basic[] = {
	73	{ _MMIO(0x9888), 0x166c00f0 },
	74	{ _MMIO(0x9888), 0x12120280 },
	75	{ _MMIO(0x9888), 0x12320280 },
	76	{ _MMIO(0x9888), 0x11930317 },
	77	{ _MMIO(0x9888), 0x159303df },
	78	{ _MMIO(0x9888), 0x3f900c00 },
	79	{ _MMIO(0x9888), 0x419000a0 },
	80	{ _MMIO(0x9888), 0x002d1000 },
	81	{ _MMIO(0x9888), 0x062d4000 },
	82	{ _MMIO(0x9888), 0x082d5000 },
	83	{ _MMIO(0x9888), 0x0a2d1000 },
	84	{ _MMIO(0x9888), 0x0c2e0800 },
	85	{ _MMIO(0x9888), 0x0e2e5900 },
	86	{ _MMIO(0x9888), 0x0a4c8000 },
	87	{ _MMIO(0x9888), 0x0c4c8000 },
	88	{ _MMIO(0x9888), 0x0e4c4000 },
	89	{ _MMIO(0x9888), 0x064e8000 },
	90	{ _MMIO(0x9888), 0x084e8000 },
	91	{ _MMIO(0x9888), 0x0a4e2000 },
	92	{ _MMIO(0x9888), 0x1c4f0010 },
	93	{ _MMIO(0x9888), 0x0a6c0053 },
	94	{ _MMIO(0x9888), 0x106c0000 },
	95	{ _MMIO(0x9888), 0x1c6c0000 },
	96	{ _MMIO(0x9888), 0x1a0fcc00 },
	97	{ _MMIO(0x9888), 0x1c0f0002 },
	98	{ _MMIO(0x9888), 0x1c2c0040 },
	99	{ _MMIO(0x9888), 0x00101000 },
	100	{ _MMIO(0x9888), 0x04101000 },
	101	{ _MMIO(0x9888), 0x00114000 },
	102	{ _MMIO(0x9888), 0x08114000 },
	103	{ _MMIO(0x9888), 0x00120020 },
	104	{ _MMIO(0x9888), 0x08120021 },
	105	{ _MMIO(0x9888), 0x00141000 },
	106	{ _MMIO(0x9888), 0x08141000 },
	107	{ _MMIO(0x9888), 0x02308000 },
	108	{ _MMIO(0x9888), 0x04302000 },
	109	{ _MMIO(0x9888), 0x06318000 },
	110	{ _MMIO(0x9888), 0x08318000 },
	111	{ _MMIO(0x9888), 0x06320800 },
	112	{ _MMIO(0x9888), 0x08320840 },
	113	{ _MMIO(0x9888), 0x00320000 },
	114	{ _MMIO(0x9888), 0x06344000 },
	115	{ _MMIO(0x9888), 0x08344000 },
	116	{ _MMIO(0x9888), 0x0d931831 },
	117	{ _MMIO(0x9888), 0x0f939f3f },
	118	{ _MMIO(0x9888), 0x01939e80 },
	119	{ _MMIO(0x9888), 0x039303bc },
	120	{ _MMIO(0x9888), 0x0593000e },
	121	{ _MMIO(0x9888), 0x1993002a },
	122	{ _MMIO(0x9888), 0x07930000 },
	123	{ _MMIO(0x9888), 0x09930000 },
	124	{ _MMIO(0x9888), 0x1d900177 },
	125	{ _MMIO(0x9888), 0x1f900187 },
	126	{ _MMIO(0x9888), 0x35900000 },
	127	{ _MMIO(0x9888), 0x13904000 },
	128	{ _MMIO(0x9888), 0x21904000 },
	129	{ _MMIO(0x9888), 0x23904000 },
	130	{ _MMIO(0x9888), 0x25904000 },
	131	{ _MMIO(0x9888), 0x27904000 },
	132	{ _MMIO(0x9888), 0x2b904000 },
	133	{ _MMIO(0x9888), 0x2d904000 },
	134	{ _MMIO(0x9888), 0x2f904000 },
	135	{ _MMIO(0x9888), 0x31904000 },
	136	{ _MMIO(0x9888), 0x15904000 },
	137	{ _MMIO(0x9888), 0x17904000 },
	138	{ _MMIO(0x9888), 0x19904000 },
	139	{ _MMIO(0x9888), 0x1b904000 },
	140	{ _MMIO(0x9888), 0x53901110 },
	141	{ _MMIO(0x9888), 0x43900423 },
	142	{ _MMIO(0x9888), 0x55900111 },
	143	{ _MMIO(0x9888), 0x47900c02 },
	144	{ _MMIO(0x9888), 0x57900000 },
	145	{ _MMIO(0x9888), 0x49900020 },
	146	{ _MMIO(0x9888), 0x59901111 },
	147	{ _MMIO(0x9888), 0x4b900421 },
	148	{ _MMIO(0x9888), 0x37900000 },
	149	{ _MMIO(0x9888), 0x33900000 },
	150	{ _MMIO(0x9888), 0x4d900001 },
	151	{ _MMIO(0x9888), 0x45900821 },
	152	};
	153
	154	static int
	155	get_render_basic_mux_config(struct drm_i915_private *dev_priv,
	156	const struct i915_oa_reg **regs,
	157	int *lens)
	158	{
	159	int n = 0;
	160
	161	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	162	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	163
	164	regs[n] = mux_config_render_basic;
	165	lens[n] = ARRAY_SIZE(mux_config_render_basic);
	166	n++;
	167
	168	return n;
	169	}
	170
	171	static const struct i915_oa_reg b_counter_config_compute_basic[] = {
	172	{ _MMIO(0x2710), 0x00000000 },
	173	{ _MMIO(0x2714), 0x00800000 },
	174	{ _MMIO(0x2720), 0x00000000 },
	175	{ _MMIO(0x2724), 0x00800000 },
	176	{ _MMIO(0x2740), 0x00000000 },
	177	};
	178
	179	static const struct i915_oa_reg flex_eu_config_compute_basic[] = {
	180	{ _MMIO(0xe458), 0x00005004 },
	181	{ _MMIO(0xe558), 0x00000003 },
	182	{ _MMIO(0xe658), 0x00002001 },
	183	{ _MMIO(0xe758), 0x00778008 },
	184	{ _MMIO(0xe45c), 0x00088078 },
	185	{ _MMIO(0xe55c), 0x00808708 },
	186	{ _MMIO(0xe65c), 0x00a08908 },
	187	};
	188
	189	static const struct i915_oa_reg mux_config_compute_basic[] = {
	190	{ _MMIO(0x9888), 0x104f00e0 },
	191	{ _MMIO(0x9888), 0x124f1c00 },
	192	{ _MMIO(0x9888), 0x39900340 },
	193	{ _MMIO(0x9888), 0x3f900c00 },
	194	{ _MMIO(0x9888), 0x41900000 },
	195	{ _MMIO(0x9888), 0x002d5000 },
	196	{ _MMIO(0x9888), 0x062d4000 },
	197	{ _MMIO(0x9888), 0x082d4000 },
	198	{ _MMIO(0x9888), 0x0a2d1000 },
	199	{ _MMIO(0x9888), 0x0c2d5000 },
	200	{ _MMIO(0x9888), 0x0e2d4000 },
	201	{ _MMIO(0x9888), 0x0c2e1400 },
	202	{ _MMIO(0x9888), 0x0e2e5100 },
	203	{ _MMIO(0x9888), 0x102e0114 },
	204	{ _MMIO(0x9888), 0x044cc000 },
	205	{ _MMIO(0x9888), 0x0a4c8000 },
	206	{ _MMIO(0x9888), 0x0c4c8000 },
	207	{ _MMIO(0x9888), 0x0e4c4000 },
	208	{ _MMIO(0x9888), 0x104c8000 },
	209	{ _MMIO(0x9888), 0x124c8000 },
	210	{ _MMIO(0x9888), 0x164c2000 },
	211	{ _MMIO(0x9888), 0x004ea000 },
	212	{ _MMIO(0x9888), 0x064e8000 },
	213	{ _MMIO(0x9888), 0x084e8000 },
	214	{ _MMIO(0x9888), 0x0a4e2000 },
	215	{ _MMIO(0x9888), 0x0c4ea000 },
	216	{ _MMIO(0x9888), 0x0e4e8000 },
	217	{ _MMIO(0x9888), 0x004f6b42 },
	218	{ _MMIO(0x9888), 0x064f6200 },
	219	{ _MMIO(0x9888), 0x084f4100 },
	220	{ _MMIO(0x9888), 0x0a4f0061 },
	221	{ _MMIO(0x9888), 0x0c4f6c4c },
	222	{ _MMIO(0x9888), 0x0e4f4b00 },
	223	{ _MMIO(0x9888), 0x1a4f0000 },
	224	{ _MMIO(0x9888), 0x1c4f0000 },
	225	{ _MMIO(0x9888), 0x180f5000 },
	226	{ _MMIO(0x9888), 0x1a0f8800 },
	227	{ _MMIO(0x9888), 0x1c0f08a2 },
	228	{ _MMIO(0x9888), 0x182c4000 },
	229	{ _MMIO(0x9888), 0x1c2c1451 },
	230	{ _MMIO(0x9888), 0x1e2c0001 },
	231	{ _MMIO(0x9888), 0x1a2c0010 },
	232	{ _MMIO(0x9888), 0x01938000 },
	233	{ _MMIO(0x9888), 0x0f938000 },
	234	{ _MMIO(0x9888), 0x19938a28 },
	235	{ _MMIO(0x9888), 0x03938000 },
	236	{ _MMIO(0x9888), 0x19900177 },
	237	{ _MMIO(0x9888), 0x1b900178 },
	238	{ _MMIO(0x9888), 0x1d900125 },
	239	{ _MMIO(0x9888), 0x1f900123 },
	240	{ _MMIO(0x9888), 0x35900000 },
	241	{ _MMIO(0x9888), 0x13904000 },
	242	{ _MMIO(0x9888), 0x21904000 },
	243	{ _MMIO(0x9888), 0x25904000 },
	244	{ _MMIO(0x9888), 0x27904000 },
	245	{ _MMIO(0x9888), 0x2b904000 },
	246	{ _MMIO(0x9888), 0x2d904000 },
	247	{ _MMIO(0x9888), 0x31904000 },
	248	{ _MMIO(0x9888), 0x15904000 },
	249	{ _MMIO(0x9888), 0x53901000 },
	250	{ _MMIO(0x9888), 0x43900000 },
	251	{ _MMIO(0x9888), 0x55900111 },
	252	{ _MMIO(0x9888), 0x47900000 },
	253	{ _MMIO(0x9888), 0x57900000 },
	254	{ _MMIO(0x9888), 0x49900000 },
	255	{ _MMIO(0x9888), 0x59900000 },
	256	{ _MMIO(0x9888), 0x4b900000 },
	257	{ _MMIO(0x9888), 0x37900000 },
	258	{ _MMIO(0x9888), 0x33900000 },
	259	{ _MMIO(0x9888), 0x4d900000 },
	260	{ _MMIO(0x9888), 0x45900000 },
	261	};
	262
	263	static int
	264	get_compute_basic_mux_config(struct drm_i915_private *dev_priv,
	265	const struct i915_oa_reg **regs,
	266	int *lens)
	267	{
	268	int n = 0;
	269
	270	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	271	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	272
	273	regs[n] = mux_config_compute_basic;
	274	lens[n] = ARRAY_SIZE(mux_config_compute_basic);
	275	n++;
	276
	277	return n;
	278	}
	279
	280	static const struct i915_oa_reg b_counter_config_render_pipe_profile[] = {
	281	{ _MMIO(0x2724), 0xf0800000 },
	282	{ _MMIO(0x2720), 0x00000000 },
	283	{ _MMIO(0x2714), 0xf0800000 },
	284	{ _MMIO(0x2710), 0x00000000 },
	285	{ _MMIO(0x2740), 0x00000000 },
	286	{ _MMIO(0x2770), 0x0007ffea },
	287	{ _MMIO(0x2774), 0x00007ffc },
	288	{ _MMIO(0x2778), 0x0007affa },
	289	{ _MMIO(0x277c), 0x0000f5fd },
	290	{ _MMIO(0x2780), 0x00079ffa },
	291	{ _MMIO(0x2784), 0x0000f3fb },
	292	{ _MMIO(0x2788), 0x0007bf7a },
	293	{ _MMIO(0x278c), 0x0000f7e7 },
	294	{ _MMIO(0x2790), 0x0007fefa },
	295	{ _MMIO(0x2794), 0x0000f7cf },
	296	{ _MMIO(0x2798), 0x00077ffa },
	297	{ _MMIO(0x279c), 0x0000efdf },
	298	{ _MMIO(0x27a0), 0x0006fffa },
	299	{ _MMIO(0x27a4), 0x0000cfbf },
	300	{ _MMIO(0x27a8), 0x0003fffa },
	301	{ _MMIO(0x27ac), 0x00005f7f },
	302	};
	303
	304	static const struct i915_oa_reg flex_eu_config_render_pipe_profile[] = {
	305	{ _MMIO(0xe458), 0x00005004 },
	306	{ _MMIO(0xe558), 0x00015014 },
	307	{ _MMIO(0xe658), 0x00025024 },
	308	{ _MMIO(0xe758), 0x00035034 },
	309	{ _MMIO(0xe45c), 0x00045044 },
	310	{ _MMIO(0xe55c), 0x00055054 },
	311	{ _MMIO(0xe65c), 0x00065064 },
	312	};
	313
	314	static const struct i915_oa_reg mux_config_render_pipe_profile[] = {
	315	{ _MMIO(0x9888), 0x0c2e001f },
	316	{ _MMIO(0x9888), 0x0a2f0000 },
	317	{ _MMIO(0x9888), 0x10186800 },
	318	{ _MMIO(0x9888), 0x11810019 },
	319	{ _MMIO(0x9888), 0x15810013 },
	320	{ _MMIO(0x9888), 0x13820020 },
	321	{ _MMIO(0x9888), 0x11830020 },
	322	{ _MMIO(0x9888), 0x17840000 },
	323	{ _MMIO(0x9888), 0x11860007 },
	324	{ _MMIO(0x9888), 0x21860000 },
	325	{ _MMIO(0x9888), 0x178703e0 },
	326	{ _MMIO(0x9888), 0x0c2d8000 },
	327	{ _MMIO(0x9888), 0x042d4000 },
	328	{ _MMIO(0x9888), 0x062d1000 },
	329	{ _MMIO(0x9888), 0x022e5400 },
	330	{ _MMIO(0x9888), 0x002e0000 },
	331	{ _MMIO(0x9888), 0x0e2e0080 },
	332	{ _MMIO(0x9888), 0x082f0040 },
	333	{ _MMIO(0x9888), 0x002f0000 },
	334	{ _MMIO(0x9888), 0x06143000 },
	335	{ _MMIO(0x9888), 0x06174000 },
	336	{ _MMIO(0x9888), 0x06180012 },
	337	{ _MMIO(0x9888), 0x00180000 },
	338	{ _MMIO(0x9888), 0x0d804000 },
	339	{ _MMIO(0x9888), 0x0f804000 },
	340	{ _MMIO(0x9888), 0x05804000 },
	341	{ _MMIO(0x9888), 0x09810200 },
	342	{ _MMIO(0x9888), 0x0b810030 },
	343	{ _MMIO(0x9888), 0x03810003 },
	344	{ _MMIO(0x9888), 0x21819140 },
	345	{ _MMIO(0x9888), 0x23819050 },
	346	{ _MMIO(0x9888), 0x25810018 },
	347	{ _MMIO(0x9888), 0x0b820980 },
	348	{ _MMIO(0x9888), 0x03820d80 },
	349	{ _MMIO(0x9888), 0x11820000 },
	350	{ _MMIO(0x9888), 0x0182c000 },
	351	{ _MMIO(0x9888), 0x07828000 },
	352	{ _MMIO(0x9888), 0x09824000 },
	353	{ _MMIO(0x9888), 0x0f828000 },
	354	{ _MMIO(0x9888), 0x0d830004 },
	355	{ _MMIO(0x9888), 0x0583000c },
	356	{ _MMIO(0x9888), 0x0f831000 },
	357	{ _MMIO(0x9888), 0x01848072 },
	358	{ _MMIO(0x9888), 0x11840000 },
	359	{ _MMIO(0x9888), 0x07848000 },
	360	{ _MMIO(0x9888), 0x09844000 },
	361	{ _MMIO(0x9888), 0x0f848000 },
	362	{ _MMIO(0x9888), 0x07860000 },
	363	{ _MMIO(0x9888), 0x09860092 },
	364	{ _MMIO(0x9888), 0x0f860400 },
	365	{ _MMIO(0x9888), 0x01869100 },
	366	{ _MMIO(0x9888), 0x0f870065 },
	367	{ _MMIO(0x9888), 0x01870000 },
	368	{ _MMIO(0x9888), 0x19930800 },
	369	{ _MMIO(0x9888), 0x0b938000 },
	370	{ _MMIO(0x9888), 0x0d938000 },
	371	{ _MMIO(0x9888), 0x1b952000 },
	372	{ _MMIO(0x9888), 0x1d955055 },
	373	{ _MMIO(0x9888), 0x1f951455 },
	374	{ _MMIO(0x9888), 0x0992a000 },
	375	{ _MMIO(0x9888), 0x0f928000 },
	376	{ _MMIO(0x9888), 0x1192a800 },
	377	{ _MMIO(0x9888), 0x1392028a },
	378	{ _MMIO(0x9888), 0x0b92a000 },
	379	{ _MMIO(0x9888), 0x0d922000 },
	380	{ _MMIO(0x9888), 0x13908000 },
	381	{ _MMIO(0x9888), 0x21908000 },
	382	{ _MMIO(0x9888), 0x23908000 },
	383	{ _MMIO(0x9888), 0x25908000 },
	384	{ _MMIO(0x9888), 0x27908000 },
	385	{ _MMIO(0x9888), 0x29908000 },
	386	{ _MMIO(0x9888), 0x2b908000 },
	387	{ _MMIO(0x9888), 0x2d904000 },
	388	{ _MMIO(0x9888), 0x2f908000 },
	389	{ _MMIO(0x9888), 0x31908000 },
	390	{ _MMIO(0x9888), 0x15908000 },
	391	{ _MMIO(0x9888), 0x17908000 },
	392	{ _MMIO(0x9888), 0x19908000 },
	393	{ _MMIO(0x9888), 0x1b908000 },
	394	{ _MMIO(0x9888), 0x1d904000 },
	395	{ _MMIO(0x9888), 0x1f904000 },
	396	{ _MMIO(0x9888), 0x53900000 },
	397	{ _MMIO(0x9888), 0x43900c01 },
	398	{ _MMIO(0x9888), 0x55900000 },
	399	{ _MMIO(0x9888), 0x47900000 },
	400	{ _MMIO(0x9888), 0x57900000 },
	401	{ _MMIO(0x9888), 0x49900863 },
	402	{ _MMIO(0x9888), 0x59900000 },
	403	{ _MMIO(0x9888), 0x4b900061 },
	404	{ _MMIO(0x9888), 0x37900000 },
	405	{ _MMIO(0x9888), 0x33900000 },
	406	{ _MMIO(0x9888), 0x4d900000 },
	407	{ _MMIO(0x9888), 0x45900c22 },
	408	};
	409
	410	static int
	411	get_render_pipe_profile_mux_config(struct drm_i915_private *dev_priv,
	412	const struct i915_oa_reg **regs,
	413	int *lens)
	414	{
	415	int n = 0;
	416
	417	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	418	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	419
	420	regs[n] = mux_config_render_pipe_profile;
	421	lens[n] = ARRAY_SIZE(mux_config_render_pipe_profile);
	422	n++;
	423
	424	return n;
	425	}
	426
	427	static const struct i915_oa_reg b_counter_config_memory_reads[] = {
	428	{ _MMIO(0x272c), 0xffffffff },
	429	{ _MMIO(0x2728), 0xffffffff },
	430	{ _MMIO(0x2724), 0xf0800000 },
	431	{ _MMIO(0x2720), 0x00000000 },
	432	{ _MMIO(0x271c), 0xffffffff },
	433	{ _MMIO(0x2718), 0xffffffff },
	434	{ _MMIO(0x2714), 0xf0800000 },
	435	{ _MMIO(0x2710), 0x00000000 },
	436	{ _MMIO(0x274c), 0x86543210 },
	437	{ _MMIO(0x2748), 0x86543210 },
	438	{ _MMIO(0x2744), 0x00006667 },
	439	{ _MMIO(0x2740), 0x00000000 },
	440	{ _MMIO(0x275c), 0x86543210 },
	441	{ _MMIO(0x2758), 0x86543210 },
	442	{ _MMIO(0x2754), 0x00006465 },
	443	{ _MMIO(0x2750), 0x00000000 },
	444	{ _MMIO(0x2770), 0x0007f81a },
	445	{ _MMIO(0x2774), 0x0000fe00 },
	446	{ _MMIO(0x2778), 0x0007f82a },
	447	{ _MMIO(0x277c), 0x0000fe00 },
	448	{ _MMIO(0x2780), 0x0007f872 },
	449	{ _MMIO(0x2784), 0x0000fe00 },
	450	{ _MMIO(0x2788), 0x0007f8ba },
	451	{ _MMIO(0x278c), 0x0000fe00 },
	452	{ _MMIO(0x2790), 0x0007f87a },
	453	{ _MMIO(0x2794), 0x0000fe00 },
	454	{ _MMIO(0x2798), 0x0007f8ea },
	455	{ _MMIO(0x279c), 0x0000fe00 },
	456	{ _MMIO(0x27a0), 0x0007f8e2 },
	457	{ _MMIO(0x27a4), 0x0000fe00 },
	458	{ _MMIO(0x27a8), 0x0007f8f2 },
	459	{ _MMIO(0x27ac), 0x0000fe00 },
	460	};
	461
	462	static const struct i915_oa_reg flex_eu_config_memory_reads[] = {
	463	{ _MMIO(0xe458), 0x00005004 },
	464	{ _MMIO(0xe558), 0x00015014 },
	465	{ _MMIO(0xe658), 0x00025024 },
	466	{ _MMIO(0xe758), 0x00035034 },
	467	{ _MMIO(0xe45c), 0x00045044 },
	468	{ _MMIO(0xe55c), 0x00055054 },
	469	{ _MMIO(0xe65c), 0x00065064 },
	470	};
	471
	472	static const struct i915_oa_reg mux_config_memory_reads[] = {
	473	{ _MMIO(0x9888), 0x19800343 },
	474	{ _MMIO(0x9888), 0x39900340 },
	475	{ _MMIO(0x9888), 0x3f901000 },
	476	{ _MMIO(0x9888), 0x41900003 },
	477	{ _MMIO(0x9888), 0x03803180 },
	478	{ _MMIO(0x9888), 0x058035e2 },
	479	{ _MMIO(0x9888), 0x0780006a },
	480	{ _MMIO(0x9888), 0x11800000 },
	481	{ _MMIO(0x9888), 0x2181a000 },
	482	{ _MMIO(0x9888), 0x2381000a },
	483	{ _MMIO(0x9888), 0x1d950550 },
	484	{ _MMIO(0x9888), 0x0b928000 },
	485	{ _MMIO(0x9888), 0x0d92a000 },
	486	{ _MMIO(0x9888), 0x0f922000 },
	487	{ _MMIO(0x9888), 0x13900170 },
	488	{ _MMIO(0x9888), 0x21900171 },
	489	{ _MMIO(0x9888), 0x23900172 },
	490	{ _MMIO(0x9888), 0x25900173 },
	491	{ _MMIO(0x9888), 0x27900174 },
	492	{ _MMIO(0x9888), 0x29900175 },
	493	{ _MMIO(0x9888), 0x2b900176 },
	494	{ _MMIO(0x9888), 0x2d900177 },
	495	{ _MMIO(0x9888), 0x2f90017f },
	496	{ _MMIO(0x9888), 0x31900125 },
	497	{ _MMIO(0x9888), 0x15900123 },
	498	{ _MMIO(0x9888), 0x17900121 },
	499	{ _MMIO(0x9888), 0x35900000 },
	500	{ _MMIO(0x9888), 0x19908000 },
	501	{ _MMIO(0x9888), 0x1b908000 },
	502	{ _MMIO(0x9888), 0x1d908000 },
	503	{ _MMIO(0x9888), 0x1f908000 },
	504	{ _MMIO(0x9888), 0x53900000 },
	505	{ _MMIO(0x9888), 0x43901084 },
	506	{ _MMIO(0x9888), 0x55900000 },
	507	{ _MMIO(0x9888), 0x47901080 },
	508	{ _MMIO(0x9888), 0x57900000 },
	509	{ _MMIO(0x9888), 0x49901084 },
	510	{ _MMIO(0x9888), 0x59900000 },
	511	{ _MMIO(0x9888), 0x4b901084 },
	512	{ _MMIO(0x9888), 0x37900000 },
	513	{ _MMIO(0x9888), 0x33900000 },
	514	{ _MMIO(0x9888), 0x4d900004 },
	515	{ _MMIO(0x9888), 0x45900000 },
	516	};
	517
	518	static int
	519	get_memory_reads_mux_config(struct drm_i915_private *dev_priv,
	520	const struct i915_oa_reg **regs,
	521	int *lens)
	522	{
	523	int n = 0;
	524
	525	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	526	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	527
	528	regs[n] = mux_config_memory_reads;
	529	lens[n] = ARRAY_SIZE(mux_config_memory_reads);
	530	n++;
	531
	532	return n;
	533	}
	534
	535	static const struct i915_oa_reg b_counter_config_memory_writes[] = {
	536	{ _MMIO(0x272c), 0xffffffff },
	537	{ _MMIO(0x2728), 0xffffffff },
	538	{ _MMIO(0x2724), 0xf0800000 },
	539	{ _MMIO(0x2720), 0x00000000 },
	540	{ _MMIO(0x271c), 0xffffffff },
	541	{ _MMIO(0x2718), 0xffffffff },
	542	{ _MMIO(0x2714), 0xf0800000 },
	543	{ _MMIO(0x2710), 0x00000000 },
	544	{ _MMIO(0x274c), 0x86543210 },
	545	{ _MMIO(0x2748), 0x86543210 },
	546	{ _MMIO(0x2744), 0x00006667 },
	547	{ _MMIO(0x2740), 0x00000000 },
	548	{ _MMIO(0x275c), 0x86543210 },
	549	{ _MMIO(0x2758), 0x86543210 },
	550	{ _MMIO(0x2754), 0x00006465 },
	551	{ _MMIO(0x2750), 0x00000000 },
	552	{ _MMIO(0x2770), 0x0007f81a },
	553	{ _MMIO(0x2774), 0x0000fe00 },
	554	{ _MMIO(0x2778), 0x0007f82a },
	555	{ _MMIO(0x277c), 0x0000fe00 },
	556	{ _MMIO(0x2780), 0x0007f822 },
	557	{ _MMIO(0x2784), 0x0000fe00 },
	558	{ _MMIO(0x2788), 0x0007f8ba },
	559	{ _MMIO(0x278c), 0x0000fe00 },
	560	{ _MMIO(0x2790), 0x0007f87a },
	561	{ _MMIO(0x2794), 0x0000fe00 },
	562	{ _MMIO(0x2798), 0x0007f8ea },
	563	{ _MMIO(0x279c), 0x0000fe00 },
	564	{ _MMIO(0x27a0), 0x0007f8e2 },
	565	{ _MMIO(0x27a4), 0x0000fe00 },
	566	{ _MMIO(0x27a8), 0x0007f8f2 },
	567	{ _MMIO(0x27ac), 0x0000fe00 },
	568	};
	569
	570	static const struct i915_oa_reg flex_eu_config_memory_writes[] = {
	571	{ _MMIO(0xe458), 0x00005004 },
	572	{ _MMIO(0xe558), 0x00015014 },
	573	{ _MMIO(0xe658), 0x00025024 },
	574	{ _MMIO(0xe758), 0x00035034 },
	575	{ _MMIO(0xe45c), 0x00045044 },
	576	{ _MMIO(0xe55c), 0x00055054 },
	577	{ _MMIO(0xe65c), 0x00065064 },
	578	};
	579
	580	static const struct i915_oa_reg mux_config_memory_writes[] = {
	581	{ _MMIO(0x9888), 0x19800343 },
	582	{ _MMIO(0x9888), 0x39900340 },
	583	{ _MMIO(0x9888), 0x3f900000 },
	584	{ _MMIO(0x9888), 0x41900080 },
	585	{ _MMIO(0x9888), 0x03803180 },
	586	{ _MMIO(0x9888), 0x058035e2 },
	587	{ _MMIO(0x9888), 0x0780006a },
	588	{ _MMIO(0x9888), 0x11800000 },
	589	{ _MMIO(0x9888), 0x2181a000 },
	590	{ _MMIO(0x9888), 0x2381000a },
	591	{ _MMIO(0x9888), 0x1d950550 },
	592	{ _MMIO(0x9888), 0x0b928000 },
	593	{ _MMIO(0x9888), 0x0d92a000 },
	594	{ _MMIO(0x9888), 0x0f922000 },
	595	{ _MMIO(0x9888), 0x13900180 },
	596	{ _MMIO(0x9888), 0x21900181 },
	597	{ _MMIO(0x9888), 0x23900182 },
	598	{ _MMIO(0x9888), 0x25900183 },
	599	{ _MMIO(0x9888), 0x27900184 },
	600	{ _MMIO(0x9888), 0x29900185 },
	601	{ _MMIO(0x9888), 0x2b900186 },
	602	{ _MMIO(0x9888), 0x2d900187 },
	603	{ _MMIO(0x9888), 0x2f900170 },
	604	{ _MMIO(0x9888), 0x31900125 },
	605	{ _MMIO(0x9888), 0x15900123 },
	606	{ _MMIO(0x9888), 0x17900121 },
	607	{ _MMIO(0x9888), 0x35900000 },
	608	{ _MMIO(0x9888), 0x19908000 },
	609	{ _MMIO(0x9888), 0x1b908000 },
	610	{ _MMIO(0x9888), 0x1d908000 },
	611	{ _MMIO(0x9888), 0x1f908000 },
	612	{ _MMIO(0x9888), 0x53900000 },
	613	{ _MMIO(0x9888), 0x43901084 },
	614	{ _MMIO(0x9888), 0x55900000 },
	615	{ _MMIO(0x9888), 0x47901080 },
	616	{ _MMIO(0x9888), 0x57900000 },
	617	{ _MMIO(0x9888), 0x49901084 },
	618	{ _MMIO(0x9888), 0x59900000 },
	619	{ _MMIO(0x9888), 0x4b901084 },
	620	{ _MMIO(0x9888), 0x37900000 },
	621	{ _MMIO(0x9888), 0x33900000 },
	622	{ _MMIO(0x9888), 0x4d900004 },
	623	{ _MMIO(0x9888), 0x45900000 },
	624	};
	625
	626	static int
	627	get_memory_writes_mux_config(struct drm_i915_private *dev_priv,
	628	const struct i915_oa_reg **regs,
	629	int *lens)
	630	{
	631	int n = 0;
	632
	633	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	634	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	635
	636	regs[n] = mux_config_memory_writes;
	637	lens[n] = ARRAY_SIZE(mux_config_memory_writes);
	638	n++;
	639
	640	return n;
	641	}
	642
	643	static const struct i915_oa_reg b_counter_config_compute_extended[] = {
	644	{ _MMIO(0x2724), 0xf0800000 },
	645	{ _MMIO(0x2720), 0x00000000 },
	646	{ _MMIO(0x2714), 0xf0800000 },
	647	{ _MMIO(0x2710), 0x00000000 },
	648	{ _MMIO(0x2740), 0x00000000 },
	649	{ _MMIO(0x2770), 0x0007fc2a },
	650	{ _MMIO(0x2774), 0x0000bf00 },
	651	{ _MMIO(0x2778), 0x0007fc6a },
	652	{ _MMIO(0x277c), 0x0000bf00 },
	653	{ _MMIO(0x2780), 0x0007fc92 },
	654	{ _MMIO(0x2784), 0x0000bf00 },
	655	{ _MMIO(0x2788), 0x0007fca2 },
	656	{ _MMIO(0x278c), 0x0000bf00 },
	657	{ _MMIO(0x2790), 0x0007fc32 },
	658	{ _MMIO(0x2794), 0x0000bf00 },
	659	{ _MMIO(0x2798), 0x0007fc9a },
	660	{ _MMIO(0x279c), 0x0000bf00 },
	661	{ _MMIO(0x27a0), 0x0007fe6a },
	662	{ _MMIO(0x27a4), 0x0000bf00 },
	663	{ _MMIO(0x27a8), 0x0007fe7a },
	664	{ _MMIO(0x27ac), 0x0000bf00 },
	665	};
	666
	667	static const struct i915_oa_reg flex_eu_config_compute_extended[] = {
	668	{ _MMIO(0xe458), 0x00005004 },
	669	{ _MMIO(0xe558), 0x00000003 },
	670	{ _MMIO(0xe658), 0x00002001 },
	671	{ _MMIO(0xe758), 0x00778008 },
	672	{ _MMIO(0xe45c), 0x00088078 },
	673	{ _MMIO(0xe55c), 0x00808708 },
	674	{ _MMIO(0xe65c), 0x00a08908 },
	675	};
	676
	677	static const struct i915_oa_reg mux_config_compute_extended[] = {
	678	{ _MMIO(0x9888), 0x104f00e0 },
	679	{ _MMIO(0x9888), 0x141c0160 },
	680	{ _MMIO(0x9888), 0x161c0015 },
	681	{ _MMIO(0x9888), 0x181c0120 },
	682	{ _MMIO(0x9888), 0x002d5000 },
	683	{ _MMIO(0x9888), 0x062d4000 },
	684	{ _MMIO(0x9888), 0x082d5000 },
	685	{ _MMIO(0x9888), 0x0a2d5000 },
	686	{ _MMIO(0x9888), 0x0c2d5000 },
	687	{ _MMIO(0x9888), 0x0e2d5000 },
	688	{ _MMIO(0x9888), 0x022d5000 },
	689	{ _MMIO(0x9888), 0x042d5000 },
	690	{ _MMIO(0x9888), 0x0c2e5400 },
	691	{ _MMIO(0x9888), 0x0e2e5515 },
	692	{ _MMIO(0x9888), 0x102e0155 },
	693	{ _MMIO(0x9888), 0x044cc000 },
	694	{ _MMIO(0x9888), 0x0a4c8000 },
	695	{ _MMIO(0x9888), 0x0c4cc000 },
	696	{ _MMIO(0x9888), 0x0e4cc000 },
	697	{ _MMIO(0x9888), 0x104c8000 },
	698	{ _MMIO(0x9888), 0x124c8000 },
	699	{ _MMIO(0x9888), 0x144c8000 },
	700	{ _MMIO(0x9888), 0x164c2000 },
	701	{ _MMIO(0x9888), 0x064cc000 },
	702	{ _MMIO(0x9888), 0x084cc000 },
	703	{ _MMIO(0x9888), 0x004ea000 },
	704	{ _MMIO(0x9888), 0x064e8000 },
	705	{ _MMIO(0x9888), 0x084ea000 },
	706	{ _MMIO(0x9888), 0x0a4ea000 },
	707	{ _MMIO(0x9888), 0x0c4ea000 },
	708	{ _MMIO(0x9888), 0x0e4ea000 },
	709	{ _MMIO(0x9888), 0x024ea000 },
	710	{ _MMIO(0x9888), 0x044ea000 },
	711	{ _MMIO(0x9888), 0x0e4f4b41 },
	712	{ _MMIO(0x9888), 0x004f4200 },
	713	{ _MMIO(0x9888), 0x024f404c },
	714	{ _MMIO(0x9888), 0x1c4f0000 },
	715	{ _MMIO(0x9888), 0x1a4f0000 },
	716	{ _MMIO(0x9888), 0x001b4000 },
	717	{ _MMIO(0x9888), 0x061b8000 },
	718	{ _MMIO(0x9888), 0x081bc000 },
	719	{ _MMIO(0x9888), 0x0a1bc000 },
	720	{ _MMIO(0x9888), 0x0c1bc000 },
	721	{ _MMIO(0x9888), 0x041bc000 },
	722	{ _MMIO(0x9888), 0x001c0031 },
	723	{ _MMIO(0x9888), 0x061c1900 },
	724	{ _MMIO(0x9888), 0x081c1a33 },
	725	{ _MMIO(0x9888), 0x0a1c1b35 },
	726	{ _MMIO(0x9888), 0x0c1c3337 },
	727	{ _MMIO(0x9888), 0x041c31c7 },
	728	{ _MMIO(0x9888), 0x180f5000 },
	729	{ _MMIO(0x9888), 0x1a0fa8aa },
	730	{ _MMIO(0x9888), 0x1c0f0aaa },
	731	{ _MMIO(0x9888), 0x182c8000 },
	732	{ _MMIO(0x9888), 0x1c2c6aaa },
	733	{ _MMIO(0x9888), 0x1e2c0001 },
	734	{ _MMIO(0x9888), 0x1a2c2950 },
	735	{ _MMIO(0x9888), 0x01938000 },
	736	{ _MMIO(0x9888), 0x0f938000 },
	737	{ _MMIO(0x9888), 0x1993aaaa },
	738	{ _MMIO(0x9888), 0x03938000 },
	739	{ _MMIO(0x9888), 0x05938000 },
	740	{ _MMIO(0x9888), 0x07938000 },
	741	{ _MMIO(0x9888), 0x09938000 },
	742	{ _MMIO(0x9888), 0x0b938000 },
	743	{ _MMIO(0x9888), 0x13904000 },
	744	{ _MMIO(0x9888), 0x21904000 },
	745	{ _MMIO(0x9888), 0x23904000 },
	746	{ _MMIO(0x9888), 0x25904000 },
	747	{ _MMIO(0x9888), 0x27904000 },
	748	{ _MMIO(0x9888), 0x29904000 },
	749	{ _MMIO(0x9888), 0x2b904000 },
	750	{ _MMIO(0x9888), 0x2d904000 },
	751	{ _MMIO(0x9888), 0x2f904000 },
	752	{ _MMIO(0x9888), 0x31904000 },
	753	{ _MMIO(0x9888), 0x15904000 },
	754	{ _MMIO(0x9888), 0x17904000 },
	755	{ _MMIO(0x9888), 0x19904000 },
	756	{ _MMIO(0x9888), 0x1b904000 },
	757	{ _MMIO(0x9888), 0x1d904000 },
	758	{ _MMIO(0x9888), 0x53900000 },
	759	{ _MMIO(0x9888), 0x43900420 },
	760	{ _MMIO(0x9888), 0x55900000 },
	761	{ _MMIO(0x9888), 0x47900000 },
	762	{ _MMIO(0x9888), 0x57900000 },
	763	{ _MMIO(0x9888), 0x49900000 },
	764	{ _MMIO(0x9888), 0x59900000 },
	765	{ _MMIO(0x9888), 0x4b900400 },
	766	{ _MMIO(0x9888), 0x37900000 },
	767	{ _MMIO(0x9888), 0x33900000 },
	768	{ _MMIO(0x9888), 0x4d900001 },
	769	{ _MMIO(0x9888), 0x45900001 },
	770	};
	771
	772	static int
	773	get_compute_extended_mux_config(struct drm_i915_private *dev_priv,
	774	const struct i915_oa_reg **regs,
	775	int *lens)
	776	{
	777	int n = 0;
	778
	779	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	780	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	781
	782	regs[n] = mux_config_compute_extended;
	783	lens[n] = ARRAY_SIZE(mux_config_compute_extended);
	784	n++;
	785
	786	return n;
	787	}
	788
	789	static const struct i915_oa_reg b_counter_config_compute_l3_cache[] = {
	790	{ _MMIO(0x2710), 0x00000000 },
	791	{ _MMIO(0x2714), 0x30800000 },
	792	{ _MMIO(0x2720), 0x00000000 },
	793	{ _MMIO(0x2724), 0x30800000 },
	794	{ _MMIO(0x2740), 0x00000000 },
	795	{ _MMIO(0x2770), 0x0007fffa },
	796	{ _MMIO(0x2774), 0x0000fefe },
	797	{ _MMIO(0x2778), 0x0007fffa },
	798	{ _MMIO(0x277c), 0x0000fefd },
	799	{ _MMIO(0x2790), 0x0007fffa },
	800	{ _MMIO(0x2794), 0x0000fbef },
	801	{ _MMIO(0x2798), 0x0007fffa },
	802	{ _MMIO(0x279c), 0x0000fbdf },
	803	};
	804
	805	static const struct i915_oa_reg flex_eu_config_compute_l3_cache[] = {
	806	{ _MMIO(0xe458), 0x00005004 },
	807	{ _MMIO(0xe558), 0x00000003 },
	808	{ _MMIO(0xe658), 0x00002001 },
	809	{ _MMIO(0xe758), 0x00101100 },
	810	{ _MMIO(0xe45c), 0x00201200 },
	811	{ _MMIO(0xe55c), 0x00301300 },
	812	{ _MMIO(0xe65c), 0x00401400 },
	813	};
	814
	815	static const struct i915_oa_reg mux_config_compute_l3_cache[] = {
	816	{ _MMIO(0x9888), 0x166c03b0 },
	817	{ _MMIO(0x9888), 0x1593001e },
	818	{ _MMIO(0x9888), 0x3f900c00 },
	819	{ _MMIO(0x9888), 0x41900000 },
	820	{ _MMIO(0x9888), 0x002d1000 },
	821	{ _MMIO(0x9888), 0x062d4000 },
	822	{ _MMIO(0x9888), 0x082d5000 },
	823	{ _MMIO(0x9888), 0x0e2d5000 },
	824	{ _MMIO(0x9888), 0x0c2e0400 },
	825	{ _MMIO(0x9888), 0x0e2e1500 },
	826	{ _MMIO(0x9888), 0x102e0140 },
	827	{ _MMIO(0x9888), 0x044c4000 },
	828	{ _MMIO(0x9888), 0x0a4c8000 },
	829	{ _MMIO(0x9888), 0x0c4cc000 },
	830	{ _MMIO(0x9888), 0x144c8000 },
	831	{ _MMIO(0x9888), 0x164c2000 },
	832	{ _MMIO(0x9888), 0x004e2000 },
	833	{ _MMIO(0x9888), 0x064e8000 },
	834	{ _MMIO(0x9888), 0x084ea000 },
	835	{ _MMIO(0x9888), 0x0e4ea000 },
	836	{ _MMIO(0x9888), 0x1a4f4001 },
	837	{ _MMIO(0x9888), 0x1c4f5005 },
	838	{ _MMIO(0x9888), 0x006c0051 },
	839	{ _MMIO(0x9888), 0x066c5000 },
	840	{ _MMIO(0x9888), 0x086c5c5d },
	841	{ _MMIO(0x9888), 0x0e6c5e5f },
	842	{ _MMIO(0x9888), 0x106c0000 },
	843	{ _MMIO(0x9888), 0x146c0000 },
	844	{ _MMIO(0x9888), 0x1a6c0000 },
	845	{ _MMIO(0x9888), 0x1c6c0000 },
	846	{ _MMIO(0x9888), 0x180f1000 },
	847	{ _MMIO(0x9888), 0x1a0fa800 },
	848	{ _MMIO(0x9888), 0x1c0f0a00 },
	849	{ _MMIO(0x9888), 0x182c4000 },
	850	{ _MMIO(0x9888), 0x1c2c4015 },
	851	{ _MMIO(0x9888), 0x1e2c0001 },
	852	{ _MMIO(0x9888), 0x03931980 },
	853	{ _MMIO(0x9888), 0x05930032 },
	854	{ _MMIO(0x9888), 0x11930000 },
	855	{ _MMIO(0x9888), 0x01938000 },
	856	{ _MMIO(0x9888), 0x0f938000 },
	857	{ _MMIO(0x9888), 0x1993a00a },
	858	{ _MMIO(0x9888), 0x07930000 },
	859	{ _MMIO(0x9888), 0x09930000 },
	860	{ _MMIO(0x9888), 0x1d900177 },
	861	{ _MMIO(0x9888), 0x1f900178 },
	862	{ _MMIO(0x9888), 0x35900000 },
	863	{ _MMIO(0x9888), 0x13904000 },
	864	{ _MMIO(0x9888), 0x21904000 },
	865	{ _MMIO(0x9888), 0x23904000 },
	866	{ _MMIO(0x9888), 0x25904000 },
	867	{ _MMIO(0x9888), 0x2f904000 },
	868	{ _MMIO(0x9888), 0x31904000 },
	869	{ _MMIO(0x9888), 0x19904000 },
	870	{ _MMIO(0x9888), 0x1b904000 },
	871	{ _MMIO(0x9888), 0x53901000 },
	872	{ _MMIO(0x9888), 0x43900000 },
	873	{ _MMIO(0x9888), 0x55900111 },
	874	{ _MMIO(0x9888), 0x47900001 },
	875	{ _MMIO(0x9888), 0x57900000 },
	876	{ _MMIO(0x9888), 0x49900000 },
	877	{ _MMIO(0x9888), 0x37900000 },
	878	{ _MMIO(0x9888), 0x33900000 },
	879	{ _MMIO(0x9888), 0x59900000 },
	880	{ _MMIO(0x9888), 0x4b900000 },
	881	{ _MMIO(0x9888), 0x4d900000 },
	882	{ _MMIO(0x9888), 0x45900400 },
	883	};
	884
	885	static int
	886	get_compute_l3_cache_mux_config(struct drm_i915_private *dev_priv,
	887	const struct i915_oa_reg **regs,
	888	int *lens)
	889	{
	890	int n = 0;
	891
	892	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	893	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	894
	895	regs[n] = mux_config_compute_l3_cache;
	896	lens[n] = ARRAY_SIZE(mux_config_compute_l3_cache);
	897	n++;
	898
	899	return n;
	900	}
	901
	902	static const struct i915_oa_reg b_counter_config_hdc_and_sf[] = {
	903	{ _MMIO(0x2740), 0x00000000 },
	904	{ _MMIO(0x2744), 0x00800000 },
	905	{ _MMIO(0x2710), 0x00000000 },
	906	{ _MMIO(0x2714), 0x10800000 },
	907	{ _MMIO(0x2720), 0x00000000 },
	908	{ _MMIO(0x2724), 0x00800000 },
	909	{ _MMIO(0x2770), 0x00000002 },
	910	{ _MMIO(0x2774), 0x0000fdff },
	911	};
	912
	913	static const struct i915_oa_reg flex_eu_config_hdc_and_sf[] = {
	914	{ _MMIO(0xe458), 0x00005004 },
	915	{ _MMIO(0xe558), 0x00010003 },
	916	{ _MMIO(0xe658), 0x00012011 },
	917	{ _MMIO(0xe758), 0x00015014 },
	918	{ _MMIO(0xe45c), 0x00051050 },
	919	{ _MMIO(0xe55c), 0x00053052 },
	920	{ _MMIO(0xe65c), 0x00055054 },
	921	};
	922
	923	static const struct i915_oa_reg mux_config_hdc_and_sf[] = {
	924	{ _MMIO(0x9888), 0x104f0232 },
	925	{ _MMIO(0x9888), 0x124f4640 },
	926	{ _MMIO(0x9888), 0x11834400 },
	927	{ _MMIO(0x9888), 0x022d4000 },
	928	{ _MMIO(0x9888), 0x042d5000 },
	929	{ _MMIO(0x9888), 0x062d1000 },
	930	{ _MMIO(0x9888), 0x0e2e0055 },
	931	{ _MMIO(0x9888), 0x064c8000 },
	932	{ _MMIO(0x9888), 0x084cc000 },
	933	{ _MMIO(0x9888), 0x0a4c4000 },
	934	{ _MMIO(0x9888), 0x024e8000 },
	935	{ _MMIO(0x9888), 0x044ea000 },
	936	{ _MMIO(0x9888), 0x064e2000 },
	937	{ _MMIO(0x9888), 0x024f6100 },
	938	{ _MMIO(0x9888), 0x044f416b },
	939	{ _MMIO(0x9888), 0x064f004b },
	940	{ _MMIO(0x9888), 0x1a4f0000 },
	941	{ _MMIO(0x9888), 0x1a0f02a8 },
	942	{ _MMIO(0x9888), 0x1a2c5500 },
	943	{ _MMIO(0x9888), 0x0f808000 },
	944	{ _MMIO(0x9888), 0x25810020 },
	945	{ _MMIO(0x9888), 0x0f8305c0 },
	946	{ _MMIO(0x9888), 0x07938000 },
	947	{ _MMIO(0x9888), 0x09938000 },
	948	{ _MMIO(0x9888), 0x0b938000 },
	949	{ _MMIO(0x9888), 0x0d938000 },
	950	{ _MMIO(0x9888), 0x1f951000 },
	951	{ _MMIO(0x9888), 0x13920200 },
	952	{ _MMIO(0x9888), 0x31908000 },
	953	{ _MMIO(0x9888), 0x19904000 },
	954	{ _MMIO(0x9888), 0x1b904000 },
	955	{ _MMIO(0x9888), 0x1d904000 },
	956	{ _MMIO(0x9888), 0x1f904000 },
	957	{ _MMIO(0x9888), 0x37900000 },
	958	{ _MMIO(0x9888), 0x59900000 },
	959	{ _MMIO(0x9888), 0x4d900003 },
	960	{ _MMIO(0x9888), 0x53900000 },
	961	{ _MMIO(0x9888), 0x45900000 },
	962	{ _MMIO(0x9888), 0x55900000 },
	963	{ _MMIO(0x9888), 0x47900000 },
	964	{ _MMIO(0x9888), 0x33900000 },
	965	};
	966
	967	static int
	968	get_hdc_and_sf_mux_config(struct drm_i915_private *dev_priv,
	969	const struct i915_oa_reg **regs,
	970	int *lens)
	971	{
	972	int n = 0;
	973
	974	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	975	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	976
	977	regs[n] = mux_config_hdc_and_sf;
	978	lens[n] = ARRAY_SIZE(mux_config_hdc_and_sf);
	979	n++;
	980
	981	return n;
	982	}
	983
	984	static const struct i915_oa_reg b_counter_config_l3_1[] = {
	985	{ _MMIO(0x2740), 0x00000000 },
	986	{ _MMIO(0x2744), 0x00800000 },
	987	{ _MMIO(0x2710), 0x00000000 },
	988	{ _MMIO(0x2714), 0xf0800000 },
	989	{ _MMIO(0x2720), 0x00000000 },
	990	{ _MMIO(0x2724), 0xf0800000 },
	991	{ _MMIO(0x2770), 0x00100070 },
	992	{ _MMIO(0x2774), 0x0000fff1 },
	993	{ _MMIO(0x2778), 0x00014002 },
	994	{ _MMIO(0x277c), 0x0000c3ff },
	995	{ _MMIO(0x2780), 0x00010002 },
	996	{ _MMIO(0x2784), 0x0000c7ff },
	997	{ _MMIO(0x2788), 0x00004002 },
	998	{ _MMIO(0x278c), 0x0000d3ff },
	999	{ _MMIO(0x2790), 0x00100700 },
	1000	{ _MMIO(0x2794), 0x0000ff1f },
	1001	{ _MMIO(0x2798), 0x00001402 },
	1002	{ _MMIO(0x279c), 0x0000fc3f },
	1003	{ _MMIO(0x27a0), 0x00001002 },
	1004	{ _MMIO(0x27a4), 0x0000fc7f },
	1005	{ _MMIO(0x27a8), 0x00000402 },
	1006	{ _MMIO(0x27ac), 0x0000fd3f },
	1007	};
	1008
	1009	static const struct i915_oa_reg flex_eu_config_l3_1[] = {
	1010	{ _MMIO(0xe458), 0x00005004 },
	1011	{ _MMIO(0xe558), 0x00010003 },
	1012	{ _MMIO(0xe658), 0x00012011 },
	1013	{ _MMIO(0xe758), 0x00015014 },
	1014	{ _MMIO(0xe45c), 0x00051050 },
	1015	{ _MMIO(0xe55c), 0x00053052 },
	1016	{ _MMIO(0xe65c), 0x00055054 },
	1017	};
	1018
	1019	static const struct i915_oa_reg mux_config_l3_1[] = {
	1020	{ _MMIO(0x9888), 0x12643400 },
	1021	{ _MMIO(0x9888), 0x12653400 },
	1022	{ _MMIO(0x9888), 0x106c6800 },
	1023	{ _MMIO(0x9888), 0x126c001e },
	1024	{ _MMIO(0x9888), 0x166c0010 },
	1025	{ _MMIO(0x9888), 0x0c2d5000 },
	1026	{ _MMIO(0x9888), 0x0e2d5000 },
	1027	{ _MMIO(0x9888), 0x002d4000 },
	1028	{ _MMIO(0x9888), 0x022d5000 },
	1029	{ _MMIO(0x9888), 0x042d5000 },
	1030	{ _MMIO(0x9888), 0x062d1000 },
	1031	{ _MMIO(0x9888), 0x102e0154 },
	1032	{ _MMIO(0x9888), 0x0c2e5000 },
	1033	{ _MMIO(0x9888), 0x0e2e0055 },
	1034	{ _MMIO(0x9888), 0x104c8000 },
	1035	{ _MMIO(0x9888), 0x124c8000 },
	1036	{ _MMIO(0x9888), 0x144c8000 },
	1037	{ _MMIO(0x9888), 0x164c2000 },
	1038	{ _MMIO(0x9888), 0x044c8000 },
	1039	{ _MMIO(0x9888), 0x064cc000 },
	1040	{ _MMIO(0x9888), 0x084cc000 },
	1041	{ _MMIO(0x9888), 0x0a4c4000 },
	1042	{ _MMIO(0x9888), 0x0c4ea000 },
	1043	{ _MMIO(0x9888), 0x0e4ea000 },
	1044	{ _MMIO(0x9888), 0x004e8000 },
	1045	{ _MMIO(0x9888), 0x024ea000 },
	1046	{ _MMIO(0x9888), 0x044ea000 },
	1047	{ _MMIO(0x9888), 0x064e2000 },
	1048	{ _MMIO(0x9888), 0x1c4f5500 },
	1049	{ _MMIO(0x9888), 0x1a4f1554 },
	1050	{ _MMIO(0x9888), 0x0a640024 },
	1051	{ _MMIO(0x9888), 0x10640000 },
	1052	{ _MMIO(0x9888), 0x04640000 },
	1053	{ _MMIO(0x9888), 0x0c650024 },
	1054	{ _MMIO(0x9888), 0x10650000 },
	1055	{ _MMIO(0x9888), 0x06650000 },
	1056	{ _MMIO(0x9888), 0x0c6c5327 },
	1057	{ _MMIO(0x9888), 0x0e6c5425 },
	1058	{ _MMIO(0x9888), 0x006c2a00 },
	1059	{ _MMIO(0x9888), 0x026c285b },
	1060	{ _MMIO(0x9888), 0x046c005c },
	1061	{ _MMIO(0x9888), 0x1c6c0000 },
	1062	{ _MMIO(0x9888), 0x1a6c0900 },
	1063	{ _MMIO(0x9888), 0x1c0f0aa0 },
	1064	{ _MMIO(0x9888), 0x180f4000 },
	1065	{ _MMIO(0x9888), 0x1a0f02aa },
	1066	{ _MMIO(0x9888), 0x1c2c5400 },
	1067	{ _MMIO(0x9888), 0x1e2c0001 },
	1068	{ _MMIO(0x9888), 0x1a2c5550 },
	1069	{ _MMIO(0x9888), 0x1993aa00 },
	1070	{ _MMIO(0x9888), 0x03938000 },
	1071	{ _MMIO(0x9888), 0x05938000 },
	1072	{ _MMIO(0x9888), 0x07938000 },
	1073	{ _MMIO(0x9888), 0x09938000 },
	1074	{ _MMIO(0x9888), 0x0b938000 },
	1075	{ _MMIO(0x9888), 0x0d938000 },
	1076	{ _MMIO(0x9888), 0x2b904000 },
	1077	{ _MMIO(0x9888), 0x2d904000 },
	1078	{ _MMIO(0x9888), 0x2f904000 },
	1079	{ _MMIO(0x9888), 0x31904000 },
	1080	{ _MMIO(0x9888), 0x15904000 },
	1081	{ _MMIO(0x9888), 0x17904000 },
	1082	{ _MMIO(0x9888), 0x19904000 },
	1083	{ _MMIO(0x9888), 0x1b904000 },
	1084	{ _MMIO(0x9888), 0x1d904000 },
	1085	{ _MMIO(0x9888), 0x1f904000 },
	1086	{ _MMIO(0x9888), 0x59900000 },
	1087	{ _MMIO(0x9888), 0x4b900421 },
	1088	{ _MMIO(0x9888), 0x37900000 },
	1089	{ _MMIO(0x9888), 0x33900000 },
	1090	{ _MMIO(0x9888), 0x4d900001 },
	1091	{ _MMIO(0x9888), 0x53900000 },
	1092	{ _MMIO(0x9888), 0x43900420 },
	1093	{ _MMIO(0x9888), 0x45900021 },
	1094	{ _MMIO(0x9888), 0x55900000 },
	1095	{ _MMIO(0x9888), 0x47900000 },
	1096	};
	1097
	1098	static int
	1099	get_l3_1_mux_config(struct drm_i915_private *dev_priv,
	1100	const struct i915_oa_reg **regs,
	1101	int *lens)
	1102	{
	1103	int n = 0;
	1104
	1105	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1106	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1107
	1108	regs[n] = mux_config_l3_1;
	1109	lens[n] = ARRAY_SIZE(mux_config_l3_1);
	1110	n++;
	1111
	1112	return n;
	1113	}
	1114
	1115	static const struct i915_oa_reg b_counter_config_rasterizer_and_pixel_backend[] = {
	1116	{ _MMIO(0x2740), 0x00000000 },
	1117	{ _MMIO(0x2744), 0x00800000 },
	1118	{ _MMIO(0x2710), 0x00000000 },
	1119	{ _MMIO(0x2714), 0x30800000 },
	1120	{ _MMIO(0x2720), 0x00000000 },
	1121	{ _MMIO(0x2724), 0x00800000 },
	1122	{ _MMIO(0x2770), 0x00000002 },
	1123	{ _MMIO(0x2774), 0x0000efff },
	1124	{ _MMIO(0x2778), 0x00006000 },
	1125	{ _MMIO(0x277c), 0x0000f3ff },
	1126	};
	1127
	1128	static const struct i915_oa_reg flex_eu_config_rasterizer_and_pixel_backend[] = {
	1129	{ _MMIO(0xe458), 0x00005004 },
	1130	{ _MMIO(0xe558), 0x00010003 },
	1131	{ _MMIO(0xe658), 0x00012011 },
	1132	{ _MMIO(0xe758), 0x00015014 },
	1133	{ _MMIO(0xe45c), 0x00051050 },
	1134	{ _MMIO(0xe55c), 0x00053052 },
	1135	{ _MMIO(0xe65c), 0x00055054 },
	1136	};
	1137
	1138	static const struct i915_oa_reg mux_config_rasterizer_and_pixel_backend[] = {
	1139	{ _MMIO(0x9888), 0x102d7800 },
	1140	{ _MMIO(0x9888), 0x122d79e0 },
	1141	{ _MMIO(0x9888), 0x0c2f0004 },
	1142	{ _MMIO(0x9888), 0x100e3800 },
	1143	{ _MMIO(0x9888), 0x180f0005 },
	1144	{ _MMIO(0x9888), 0x002d0940 },
	1145	{ _MMIO(0x9888), 0x022d802f },
	1146	{ _MMIO(0x9888), 0x042d4013 },
	1147	{ _MMIO(0x9888), 0x062d1000 },
	1148	{ _MMIO(0x9888), 0x0e2e0050 },
	1149	{ _MMIO(0x9888), 0x022f0010 },
	1150	{ _MMIO(0x9888), 0x002f0000 },
	1151	{ _MMIO(0x9888), 0x084c8000 },
	1152	{ _MMIO(0x9888), 0x0a4c4000 },
	1153	{ _MMIO(0x9888), 0x044e8000 },
	1154	{ _MMIO(0x9888), 0x064e2000 },
	1155	{ _MMIO(0x9888), 0x040e0480 },
	1156	{ _MMIO(0x9888), 0x000e0000 },
	1157	{ _MMIO(0x9888), 0x060f0027 },
	1158	{ _MMIO(0x9888), 0x100f0000 },
	1159	{ _MMIO(0x9888), 0x1a0f0040 },
	1160	{ _MMIO(0x9888), 0x03938000 },
	1161	{ _MMIO(0x9888), 0x05938000 },
	1162	{ _MMIO(0x9888), 0x07938000 },
	1163	{ _MMIO(0x9888), 0x09938000 },
	1164	{ _MMIO(0x9888), 0x0b938000 },
	1165	{ _MMIO(0x9888), 0x0d938000 },
	1166	{ _MMIO(0x9888), 0x15904000 },
	1167	{ _MMIO(0x9888), 0x17904000 },
	1168	{ _MMIO(0x9888), 0x19904000 },
	1169	{ _MMIO(0x9888), 0x1b904000 },
	1170	{ _MMIO(0x9888), 0x1d904000 },
	1171	{ _MMIO(0x9888), 0x1f904000 },
	1172	{ _MMIO(0x9888), 0x37900000 },
	1173	{ _MMIO(0x9888), 0x53900000 },
	1174	{ _MMIO(0x9888), 0x439014a0 },
	1175	{ _MMIO(0x9888), 0x459000a4 },
	1176	{ _MMIO(0x9888), 0x55900000 },
	1177	{ _MMIO(0x9888), 0x47900001 },
	1178	{ _MMIO(0x9888), 0x33900000 },
	1179	};
	1180
	1181	static int
	1182	get_rasterizer_and_pixel_backend_mux_config(struct drm_i915_private *dev_priv,
	1183	const struct i915_oa_reg **regs,
	1184	int *lens)
	1185	{
	1186	int n = 0;
	1187
	1188	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1189	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1190
	1191	regs[n] = mux_config_rasterizer_and_pixel_backend;
	1192	lens[n] = ARRAY_SIZE(mux_config_rasterizer_and_pixel_backend);
	1193	n++;
	1194
	1195	return n;
	1196	}
	1197
	1198	static const struct i915_oa_reg b_counter_config_sampler[] = {
	1199	{ _MMIO(0x2740), 0x00000000 },
	1200	{ _MMIO(0x2744), 0x00800000 },
	1201	{ _MMIO(0x2710), 0x00000000 },
	1202	{ _MMIO(0x2714), 0x70800000 },
	1203	{ _MMIO(0x2720), 0x00000000 },
	1204	{ _MMIO(0x2724), 0x00800000 },
	1205	{ _MMIO(0x2770), 0x0000c000 },
	1206	{ _MMIO(0x2774), 0x0000e7ff },
	1207	{ _MMIO(0x2778), 0x00003000 },
	1208	{ _MMIO(0x277c), 0x0000f9ff },
	1209	{ _MMIO(0x2780), 0x00000c00 },
	1210	{ _MMIO(0x2784), 0x0000fe7f },
	1211	};
	1212
	1213	static const struct i915_oa_reg flex_eu_config_sampler[] = {
	1214	{ _MMIO(0xe458), 0x00005004 },
	1215	{ _MMIO(0xe558), 0x00010003 },
	1216	{ _MMIO(0xe658), 0x00012011 },
	1217	{ _MMIO(0xe758), 0x00015014 },
	1218	{ _MMIO(0xe45c), 0x00051050 },
	1219	{ _MMIO(0xe55c), 0x00053052 },
	1220	{ _MMIO(0xe65c), 0x00055054 },
	1221	};
	1222
	1223	static const struct i915_oa_reg mux_config_sampler[] = {
	1224	{ _MMIO(0x9888), 0x121300a0 },
	1225	{ _MMIO(0x9888), 0x141600ab },
	1226	{ _MMIO(0x9888), 0x123300a0 },
	1227	{ _MMIO(0x9888), 0x143600ab },
	1228	{ _MMIO(0x9888), 0x125300a0 },
	1229	{ _MMIO(0x9888), 0x145600ab },
	1230	{ _MMIO(0x9888), 0x0c2d4000 },
	1231	{ _MMIO(0x9888), 0x0e2d5000 },
	1232	{ _MMIO(0x9888), 0x002d4000 },
	1233	{ _MMIO(0x9888), 0x022d5000 },
	1234	{ _MMIO(0x9888), 0x042d5000 },
	1235	{ _MMIO(0x9888), 0x062d1000 },
	1236	{ _MMIO(0x9888), 0x102e01a0 },
	1237	{ _MMIO(0x9888), 0x0c2e5000 },
	1238	{ _MMIO(0x9888), 0x0e2e0065 },
	1239	{ _MMIO(0x9888), 0x164c2000 },
	1240	{ _MMIO(0x9888), 0x044c8000 },
	1241	{ _MMIO(0x9888), 0x064cc000 },
	1242	{ _MMIO(0x9888), 0x084c4000 },
	1243	{ _MMIO(0x9888), 0x0a4c4000 },
	1244	{ _MMIO(0x9888), 0x0e4e8000 },
	1245	{ _MMIO(0x9888), 0x004e8000 },
	1246	{ _MMIO(0x9888), 0x024ea000 },
	1247	{ _MMIO(0x9888), 0x044e2000 },
	1248	{ _MMIO(0x9888), 0x064e2000 },
	1249	{ _MMIO(0x9888), 0x1c0f0800 },
	1250	{ _MMIO(0x9888), 0x180f4000 },
	1251	{ _MMIO(0x9888), 0x1a0f023f },
	1252	{ _MMIO(0x9888), 0x1e2c0003 },
	1253	{ _MMIO(0x9888), 0x1a2cc030 },
	1254	{ _MMIO(0x9888), 0x04132180 },
	1255	{ _MMIO(0x9888), 0x02130000 },
	1256	{ _MMIO(0x9888), 0x0c148000 },
	1257	{ _MMIO(0x9888), 0x0e142000 },
	1258	{ _MMIO(0x9888), 0x04148000 },
	1259	{ _MMIO(0x9888), 0x1e150140 },
	1260	{ _MMIO(0x9888), 0x1c150040 },
	1261	{ _MMIO(0x9888), 0x0c163000 },
	1262	{ _MMIO(0x9888), 0x0e160068 },
	1263	{ _MMIO(0x9888), 0x10160000 },
	1264	{ _MMIO(0x9888), 0x18160000 },
	1265	{ _MMIO(0x9888), 0x0a164000 },
	1266	{ _MMIO(0x9888), 0x04330043 },
	1267	{ _MMIO(0x9888), 0x02330000 },
	1268	{ _MMIO(0x9888), 0x0234a000 },
	1269	{ _MMIO(0x9888), 0x04342000 },
	1270	{ _MMIO(0x9888), 0x1c350015 },
	1271	{ _MMIO(0x9888), 0x02363460 },
	1272	{ _MMIO(0x9888), 0x10360000 },
	1273	{ _MMIO(0x9888), 0x04360000 },
	1274	{ _MMIO(0x9888), 0x06360000 },
	1275	{ _MMIO(0x9888), 0x08364000 },
	1276	{ _MMIO(0x9888), 0x06530043 },
	1277	{ _MMIO(0x9888), 0x02530000 },
	1278	{ _MMIO(0x9888), 0x0e548000 },
	1279	{ _MMIO(0x9888), 0x00548000 },
	1280	{ _MMIO(0x9888), 0x06542000 },
	1281	{ _MMIO(0x9888), 0x1e550400 },
	1282	{ _MMIO(0x9888), 0x1a552000 },
	1283	{ _MMIO(0x9888), 0x1c550100 },
	1284	{ _MMIO(0x9888), 0x0e563000 },
	1285	{ _MMIO(0x9888), 0x00563400 },
	1286	{ _MMIO(0x9888), 0x10560000 },
	1287	{ _MMIO(0x9888), 0x18560000 },
	1288	{ _MMIO(0x9888), 0x02560000 },
	1289	{ _MMIO(0x9888), 0x0c564000 },
	1290	{ _MMIO(0x9888), 0x1993a800 },
	1291	{ _MMIO(0x9888), 0x03938000 },
	1292	{ _MMIO(0x9888), 0x05938000 },
	1293	{ _MMIO(0x9888), 0x07938000 },
	1294	{ _MMIO(0x9888), 0x09938000 },
	1295	{ _MMIO(0x9888), 0x0b938000 },
	1296	{ _MMIO(0x9888), 0x0d938000 },
	1297	{ _MMIO(0x9888), 0x2d904000 },
	1298	{ _MMIO(0x9888), 0x2f904000 },
	1299	{ _MMIO(0x9888), 0x31904000 },
	1300	{ _MMIO(0x9888), 0x15904000 },
	1301	{ _MMIO(0x9888), 0x17904000 },
	1302	{ _MMIO(0x9888), 0x19904000 },
	1303	{ _MMIO(0x9888), 0x1b904000 },
	1304	{ _MMIO(0x9888), 0x1d904000 },
	1305	{ _MMIO(0x9888), 0x1f904000 },
	1306	{ _MMIO(0x9888), 0x59900000 },
	1307	{ _MMIO(0x9888), 0x4b9014a0 },
	1308	{ _MMIO(0x9888), 0x37900000 },
	1309	{ _MMIO(0x9888), 0x33900000 },
	1310	{ _MMIO(0x9888), 0x4d900001 },
	1311	{ _MMIO(0x9888), 0x53900000 },
	1312	{ _MMIO(0x9888), 0x43900820 },
	1313	{ _MMIO(0x9888), 0x45901022 },
	1314	{ _MMIO(0x9888), 0x55900000 },
	1315	{ _MMIO(0x9888), 0x47900000 },
	1316	};
	1317
	1318	static int
	1319	get_sampler_mux_config(struct drm_i915_private *dev_priv,
	1320	const struct i915_oa_reg **regs,
	1321	int *lens)
	1322	{
	1323	int n = 0;
	1324
	1325	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1326	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1327
	1328	regs[n] = mux_config_sampler;
	1329	lens[n] = ARRAY_SIZE(mux_config_sampler);
	1330	n++;
	1331
	1332	return n;
	1333	}
	1334
	1335	static const struct i915_oa_reg b_counter_config_tdl_1[] = {
	1336	{ _MMIO(0x2740), 0x00000000 },
	1337	{ _MMIO(0x2744), 0x00800000 },
	1338	{ _MMIO(0x2710), 0x00000000 },
	1339	{ _MMIO(0x2714), 0xf0800000 },
	1340	{ _MMIO(0x2720), 0x00000000 },
	1341	{ _MMIO(0x2724), 0x30800000 },
	1342	{ _MMIO(0x2770), 0x00000002 },
	1343	{ _MMIO(0x2774), 0x00007fff },
	1344	{ _MMIO(0x2778), 0x00000000 },
	1345	{ _MMIO(0x277c), 0x00009fff },
	1346	{ _MMIO(0x2780), 0x00000002 },
	1347	{ _MMIO(0x2784), 0x0000efff },
	1348	{ _MMIO(0x2788), 0x00000000 },
	1349	{ _MMIO(0x278c), 0x0000f3ff },
	1350	{ _MMIO(0x2790), 0x00000002 },
	1351	{ _MMIO(0x2794), 0x0000fdff },
	1352	{ _MMIO(0x2798), 0x00000000 },
	1353	{ _MMIO(0x279c), 0x0000fe7f },
	1354	};
	1355
	1356	static const struct i915_oa_reg flex_eu_config_tdl_1[] = {
	1357	{ _MMIO(0xe458), 0x00005004 },
	1358	{ _MMIO(0xe558), 0x00010003 },
	1359	{ _MMIO(0xe658), 0x00012011 },
	1360	{ _MMIO(0xe758), 0x00015014 },
	1361	{ _MMIO(0xe45c), 0x00051050 },
	1362	{ _MMIO(0xe55c), 0x00053052 },
	1363	{ _MMIO(0xe65c), 0x00055054 },
	1364	};
	1365
	1366	static const struct i915_oa_reg mux_config_tdl_1[] = {
	1367	{ _MMIO(0x9888), 0x141a0000 },
	1368	{ _MMIO(0x9888), 0x143a0000 },
	1369	{ _MMIO(0x9888), 0x145a0000 },
	1370	{ _MMIO(0x9888), 0x0c2d4000 },
	1371	{ _MMIO(0x9888), 0x0e2d5000 },
	1372	{ _MMIO(0x9888), 0x002d4000 },
	1373	{ _MMIO(0x9888), 0x022d5000 },
	1374	{ _MMIO(0x9888), 0x042d5000 },
	1375	{ _MMIO(0x9888), 0x062d1000 },
	1376	{ _MMIO(0x9888), 0x102e0150 },
	1377	{ _MMIO(0x9888), 0x0c2e5000 },
	1378	{ _MMIO(0x9888), 0x0e2e006a },
	1379	{ _MMIO(0x9888), 0x124c8000 },
	1380	{ _MMIO(0x9888), 0x144c8000 },
	1381	{ _MMIO(0x9888), 0x164c2000 },
	1382	{ _MMIO(0x9888), 0x044c8000 },
	1383	{ _MMIO(0x9888), 0x064c4000 },
	1384	{ _MMIO(0x9888), 0x0a4c4000 },
	1385	{ _MMIO(0x9888), 0x0c4e8000 },
	1386	{ _MMIO(0x9888), 0x0e4ea000 },
	1387	{ _MMIO(0x9888), 0x004e8000 },
	1388	{ _MMIO(0x9888), 0x024e2000 },
	1389	{ _MMIO(0x9888), 0x064e2000 },
	1390	{ _MMIO(0x9888), 0x1c0f0bc0 },
	1391	{ _MMIO(0x9888), 0x180f4000 },
	1392	{ _MMIO(0x9888), 0x1a0f0302 },
	1393	{ _MMIO(0x9888), 0x1e2c0003 },
	1394	{ _MMIO(0x9888), 0x1a2c00f0 },
	1395	{ _MMIO(0x9888), 0x021a3080 },
	1396	{ _MMIO(0x9888), 0x041a31e5 },
	1397	{ _MMIO(0x9888), 0x02148000 },
	1398	{ _MMIO(0x9888), 0x0414a000 },
	1399	{ _MMIO(0x9888), 0x1c150054 },
	1400	{ _MMIO(0x9888), 0x06168000 },
	1401	{ _MMIO(0x9888), 0x08168000 },
	1402	{ _MMIO(0x9888), 0x0a168000 },
	1403	{ _MMIO(0x9888), 0x0c3a3280 },
	1404	{ _MMIO(0x9888), 0x0e3a0063 },
	1405	{ _MMIO(0x9888), 0x063a0061 },
	1406	{ _MMIO(0x9888), 0x023a0000 },
	1407	{ _MMIO(0x9888), 0x0c348000 },
	1408	{ _MMIO(0x9888), 0x0e342000 },
	1409	{ _MMIO(0x9888), 0x06342000 },
	1410	{ _MMIO(0x9888), 0x1e350140 },
	1411	{ _MMIO(0x9888), 0x1c350100 },
	1412	{ _MMIO(0x9888), 0x18360028 },
	1413	{ _MMIO(0x9888), 0x0c368000 },
	1414	{ _MMIO(0x9888), 0x0e5a3080 },
	1415	{ _MMIO(0x9888), 0x005a3280 },
	1416	{ _MMIO(0x9888), 0x025a0063 },
	1417	{ _MMIO(0x9888), 0x0e548000 },
	1418	{ _MMIO(0x9888), 0x00548000 },
	1419	{ _MMIO(0x9888), 0x02542000 },
	1420	{ _MMIO(0x9888), 0x1e550400 },
	1421	{ _MMIO(0x9888), 0x1a552000 },
	1422	{ _MMIO(0x9888), 0x1c550001 },
	1423	{ _MMIO(0x9888), 0x18560080 },
	1424	{ _MMIO(0x9888), 0x02568000 },
	1425	{ _MMIO(0x9888), 0x04568000 },
	1426	{ _MMIO(0x9888), 0x1993a800 },
	1427	{ _MMIO(0x9888), 0x03938000 },
	1428	{ _MMIO(0x9888), 0x05938000 },
	1429	{ _MMIO(0x9888), 0x07938000 },
	1430	{ _MMIO(0x9888), 0x09938000 },
	1431	{ _MMIO(0x9888), 0x0b938000 },
	1432	{ _MMIO(0x9888), 0x0d938000 },
	1433	{ _MMIO(0x9888), 0x2d904000 },
	1434	{ _MMIO(0x9888), 0x2f904000 },
	1435	{ _MMIO(0x9888), 0x31904000 },
	1436	{ _MMIO(0x9888), 0x15904000 },
	1437	{ _MMIO(0x9888), 0x17904000 },
	1438	{ _MMIO(0x9888), 0x19904000 },
	1439	{ _MMIO(0x9888), 0x1b904000 },
	1440	{ _MMIO(0x9888), 0x1d904000 },
	1441	{ _MMIO(0x9888), 0x1f904000 },
	1442	{ _MMIO(0x9888), 0x59900000 },
	1443	{ _MMIO(0x9888), 0x4b900420 },
	1444	{ _MMIO(0x9888), 0x37900000 },
	1445	{ _MMIO(0x9888), 0x33900000 },
	1446	{ _MMIO(0x9888), 0x4d900000 },
	1447	{ _MMIO(0x9888), 0x53900000 },
	1448	{ _MMIO(0x9888), 0x43900000 },
	1449	{ _MMIO(0x9888), 0x45901084 },
	1450	{ _MMIO(0x9888), 0x55900000 },
	1451	{ _MMIO(0x9888), 0x47900001 },
	1452	};
	1453
	1454	static int
	1455	get_tdl_1_mux_config(struct drm_i915_private *dev_priv,
	1456	const struct i915_oa_reg **regs,
	1457	int *lens)
	1458	{
	1459	int n = 0;
	1460
	1461	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1462	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1463
	1464	regs[n] = mux_config_tdl_1;
	1465	lens[n] = ARRAY_SIZE(mux_config_tdl_1);
	1466	n++;
	1467
	1468	return n;
	1469	}
	1470
	1471	static const struct i915_oa_reg b_counter_config_tdl_2[] = {
	1472	{ _MMIO(0x2740), 0x00000000 },
	1473	{ _MMIO(0x2744), 0x00800000 },
	1474	{ _MMIO(0x2710), 0x00000000 },
	1475	{ _MMIO(0x2714), 0x00800000 },
	1476	{ _MMIO(0x2720), 0x00000000 },
	1477	{ _MMIO(0x2724), 0x00800000 },
	1478	};
	1479
	1480	static const struct i915_oa_reg flex_eu_config_tdl_2[] = {
	1481	{ _MMIO(0xe458), 0x00005004 },
	1482	{ _MMIO(0xe558), 0x00010003 },
	1483	{ _MMIO(0xe658), 0x00012011 },
	1484	{ _MMIO(0xe758), 0x00015014 },
	1485	{ _MMIO(0xe45c), 0x00051050 },
	1486	{ _MMIO(0xe55c), 0x00053052 },
	1487	{ _MMIO(0xe65c), 0x00055054 },
	1488	};
	1489
	1490	static const struct i915_oa_reg mux_config_tdl_2[] = {
	1491	{ _MMIO(0x9888), 0x141a026b },
	1492	{ _MMIO(0x9888), 0x143a0173 },
	1493	{ _MMIO(0x9888), 0x145a026b },
	1494	{ _MMIO(0x9888), 0x002d4000 },
	1495	{ _MMIO(0x9888), 0x022d5000 },
	1496	{ _MMIO(0x9888), 0x042d5000 },
	1497	{ _MMIO(0x9888), 0x062d1000 },
	1498	{ _MMIO(0x9888), 0x0c2e5000 },
	1499	{ _MMIO(0x9888), 0x0e2e0069 },
	1500	{ _MMIO(0x9888), 0x044c8000 },
	1501	{ _MMIO(0x9888), 0x064cc000 },
	1502	{ _MMIO(0x9888), 0x0a4c4000 },
	1503	{ _MMIO(0x9888), 0x004e8000 },
	1504	{ _MMIO(0x9888), 0x024ea000 },
	1505	{ _MMIO(0x9888), 0x064e2000 },
	1506	{ _MMIO(0x9888), 0x180f6000 },
	1507	{ _MMIO(0x9888), 0x1a0f030a },
	1508	{ _MMIO(0x9888), 0x1a2c03c0 },
	1509	{ _MMIO(0x9888), 0x041a37e7 },
	1510	{ _MMIO(0x9888), 0x021a0000 },
	1511	{ _MMIO(0x9888), 0x0414a000 },
	1512	{ _MMIO(0x9888), 0x1c150050 },
	1513	{ _MMIO(0x9888), 0x08168000 },
	1514	{ _MMIO(0x9888), 0x0a168000 },
	1515	{ _MMIO(0x9888), 0x003a3380 },
	1516	{ _MMIO(0x9888), 0x063a006f },
	1517	{ _MMIO(0x9888), 0x023a0000 },
	1518	{ _MMIO(0x9888), 0x00348000 },
	1519	{ _MMIO(0x9888), 0x06342000 },
	1520	{ _MMIO(0x9888), 0x1a352000 },
	1521	{ _MMIO(0x9888), 0x1c350100 },
	1522	{ _MMIO(0x9888), 0x02368000 },
	1523	{ _MMIO(0x9888), 0x0c368000 },
	1524	{ _MMIO(0x9888), 0x025a37e7 },
	1525	{ _MMIO(0x9888), 0x0254a000 },
	1526	{ _MMIO(0x9888), 0x1c550005 },
	1527	{ _MMIO(0x9888), 0x04568000 },
	1528	{ _MMIO(0x9888), 0x06568000 },
	1529	{ _MMIO(0x9888), 0x03938000 },
	1530	{ _MMIO(0x9888), 0x05938000 },
	1531	{ _MMIO(0x9888), 0x07938000 },
	1532	{ _MMIO(0x9888), 0x09938000 },
	1533	{ _MMIO(0x9888), 0x0b938000 },
	1534	{ _MMIO(0x9888), 0x0d938000 },
	1535	{ _MMIO(0x9888), 0x15904000 },
	1536	{ _MMIO(0x9888), 0x17904000 },
	1537	{ _MMIO(0x9888), 0x19904000 },
	1538	{ _MMIO(0x9888), 0x1b904000 },
	1539	{ _MMIO(0x9888), 0x1d904000 },
	1540	{ _MMIO(0x9888), 0x1f904000 },
	1541	{ _MMIO(0x9888), 0x37900000 },
	1542	{ _MMIO(0x9888), 0x53900000 },
	1543	{ _MMIO(0x9888), 0x43900020 },
	1544	{ _MMIO(0x9888), 0x45901080 },
	1545	{ _MMIO(0x9888), 0x55900000 },
	1546	{ _MMIO(0x9888), 0x47900001 },
	1547	{ _MMIO(0x9888), 0x33900000 },
	1548	};
	1549
	1550	static int
	1551	get_tdl_2_mux_config(struct drm_i915_private *dev_priv,
	1552	const struct i915_oa_reg **regs,
	1553	int *lens)
	1554	{
	1555	int n = 0;
	1556
	1557	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1558	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1559
	1560	regs[n] = mux_config_tdl_2;
	1561	lens[n] = ARRAY_SIZE(mux_config_tdl_2);
	1562	n++;
	1563
	1564	return n;
	1565	}
	1566
	1567	static const struct i915_oa_reg b_counter_config_compute_extra[] = {
	1568	{ _MMIO(0x2740), 0x00000000 },
	1569	{ _MMIO(0x2744), 0x00800000 },
	1570	{ _MMIO(0x2710), 0x00000000 },
	1571	{ _MMIO(0x2714), 0x00800000 },
	1572	{ _MMIO(0x2720), 0x00000000 },
	1573	{ _MMIO(0x2724), 0x00800000 },
	1574	};
	1575
	1576	static const struct i915_oa_reg flex_eu_config_compute_extra[] = {
	1577	{ _MMIO(0xe458), 0x00001000 },
	1578	{ _MMIO(0xe558), 0x00003002 },
	1579	{ _MMIO(0xe658), 0x00005004 },
	1580	{ _MMIO(0xe758), 0x00011010 },
	1581	{ _MMIO(0xe45c), 0x00050012 },
	1582	{ _MMIO(0xe55c), 0x00052051 },
	1583	{ _MMIO(0xe65c), 0x00000008 },
	1584	};
	1585
	1586	static const struct i915_oa_reg mux_config_compute_extra[] = {
	1587	{ _MMIO(0x9888), 0x141a001f },
	1588	{ _MMIO(0x9888), 0x143a001f },
	1589	{ _MMIO(0x9888), 0x145a001f },
	1590	{ _MMIO(0x9888), 0x042d5000 },
	1591	{ _MMIO(0x9888), 0x062d1000 },
	1592	{ _MMIO(0x9888), 0x0e2e0094 },
	1593	{ _MMIO(0x9888), 0x084cc000 },
	1594	{ _MMIO(0x9888), 0x044ea000 },
	1595	{ _MMIO(0x9888), 0x1a0f00e0 },
	1596	{ _MMIO(0x9888), 0x1a2c0c00 },
	1597	{ _MMIO(0x9888), 0x061a0063 },
	1598	{ _MMIO(0x9888), 0x021a0000 },
	1599	{ _MMIO(0x9888), 0x06142000 },
	1600	{ _MMIO(0x9888), 0x1c150100 },
	1601	{ _MMIO(0x9888), 0x0c168000 },
	1602	{ _MMIO(0x9888), 0x043a3180 },
	1603	{ _MMIO(0x9888), 0x023a0000 },
	1604	{ _MMIO(0x9888), 0x04348000 },
	1605	{ _MMIO(0x9888), 0x1c350040 },
	1606	{ _MMIO(0x9888), 0x0a368000 },
	1607	{ _MMIO(0x9888), 0x045a0063 },
	1608	{ _MMIO(0x9888), 0x025a0000 },
	1609	{ _MMIO(0x9888), 0x04542000 },
	1610	{ _MMIO(0x9888), 0x1c550010 },
	1611	{ _MMIO(0x9888), 0x08568000 },
	1612	{ _MMIO(0x9888), 0x09938000 },
	1613	{ _MMIO(0x9888), 0x0b938000 },
	1614	{ _MMIO(0x9888), 0x0d938000 },
	1615	{ _MMIO(0x9888), 0x1b904000 },
	1616	{ _MMIO(0x9888), 0x1d904000 },
	1617	{ _MMIO(0x9888), 0x1f904000 },
	1618	{ _MMIO(0x9888), 0x37900000 },
	1619	{ _MMIO(0x9888), 0x55900000 },
	1620	{ _MMIO(0x9888), 0x45900400 },
	1621	{ _MMIO(0x9888), 0x47900004 },
	1622	{ _MMIO(0x9888), 0x33900000 },
	1623	};
	1624
	1625	static int
	1626	get_compute_extra_mux_config(struct drm_i915_private *dev_priv,
	1627	const struct i915_oa_reg **regs,
	1628	int *lens)
	1629	{
	1630	int n = 0;
	1631
	1632	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1633	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1634
	1635	regs[n] = mux_config_compute_extra;
	1636	lens[n] = ARRAY_SIZE(mux_config_compute_extra);
	1637	n++;
	1638
	1639	return n;
	1640	}
	1641
	1642	static const struct i915_oa_reg b_counter_config_test_oa[] = {
	1643	{ _MMIO(0x2740), 0x00000000 },
	1644	{ _MMIO(0x2744), 0x00800000 },
	1645	{ _MMIO(0x2714), 0xf0800000 },
	1646	{ _MMIO(0x2710), 0x00000000 },
	1647	{ _MMIO(0x2724), 0xf0800000 },
	1648	{ _MMIO(0x2720), 0x00000000 },
	1649	{ _MMIO(0x2770), 0x00000004 },
	1650	{ _MMIO(0x2774), 0x00000000 },
	1651	{ _MMIO(0x2778), 0x00000003 },
	1652	{ _MMIO(0x277c), 0x00000000 },
	1653	{ _MMIO(0x2780), 0x00000007 },
	1654	{ _MMIO(0x2784), 0x00000000 },
	1655	{ _MMIO(0x2788), 0x00100002 },
	1656	{ _MMIO(0x278c), 0x0000fff7 },
	1657	{ _MMIO(0x2790), 0x00100002 },
	1658	{ _MMIO(0x2794), 0x0000ffcf },
	1659	{ _MMIO(0x2798), 0x00100082 },
	1660	{ _MMIO(0x279c), 0x0000ffef },
	1661	{ _MMIO(0x27a0), 0x001000c2 },
	1662	{ _MMIO(0x27a4), 0x0000ffe7 },
	1663	{ _MMIO(0x27a8), 0x00100001 },
	1664	{ _MMIO(0x27ac), 0x0000ffe7 },
	1665	};
	1666
	1667	static const struct i915_oa_reg flex_eu_config_test_oa[] = {
	1668	};
	1669
	1670	static const struct i915_oa_reg mux_config_test_oa[] = {
	1671	{ _MMIO(0x9888), 0x19800000 },
	1672	{ _MMIO(0x9888), 0x07800063 },
	1673	{ _MMIO(0x9888), 0x11800000 },
	1674	{ _MMIO(0x9888), 0x23810008 },
	1675	{ _MMIO(0x9888), 0x1d950400 },
	1676	{ _MMIO(0x9888), 0x0f922000 },
	1677	{ _MMIO(0x9888), 0x1f908000 },
	1678	{ _MMIO(0x9888), 0x37900000 },
	1679	{ _MMIO(0x9888), 0x55900000 },
	1680	{ _MMIO(0x9888), 0x47900000 },
	1681	{ _MMIO(0x9888), 0x33900000 },
	1682	};
	1683
	1684	static int
	1685	get_test_oa_mux_config(struct drm_i915_private *dev_priv,
	1686	const struct i915_oa_reg **regs,
	1687	int *lens)
	1688	{
	1689	int n = 0;
	1690
	1691	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs) < 1);
	1692	BUILD_BUG_ON(ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens) < 1);
	1693
	1694	regs[n] = mux_config_test_oa;
	1695	lens[n] = ARRAY_SIZE(mux_config_test_oa);
	1696	n++;
	1697
	1698	return n;
	1699	}
	1700
	1701	int i915_oa_select_metric_set_glk(struct drm_i915_private *dev_priv)
	1702	{
	1703	dev_priv->perf.oa.n_mux_configs = 0;
	1704	dev_priv->perf.oa.b_counter_regs = NULL;
	1705	dev_priv->perf.oa.b_counter_regs_len = 0;
	1706	dev_priv->perf.oa.flex_regs = NULL;
	1707	dev_priv->perf.oa.flex_regs_len = 0;
	1708
	1709	switch (dev_priv->perf.oa.metrics_set) {
	1710	case METRIC_SET_ID_RENDER_BASIC:
	1711	dev_priv->perf.oa.n_mux_configs =
	1712	get_render_basic_mux_config(dev_priv,
	1713	dev_priv->perf.oa.mux_regs,
	1714	dev_priv->perf.oa.mux_regs_lens);
	1715	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1716	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_BASIC\" metric set\n");
	1717
	1718	/* EINVAL because *_register_sysfs already checked this
	1719	* and so it wouldn't have been advertised to userspace and
	1720	* so shouldn't have been requested
	1721	*/
	1722	return -EINVAL;
	1723	}
	1724
	1725	dev_priv->perf.oa.b_counter_regs =
	1726	b_counter_config_render_basic;
	1727	dev_priv->perf.oa.b_counter_regs_len =
	1728	ARRAY_SIZE(b_counter_config_render_basic);
	1729
	1730	dev_priv->perf.oa.flex_regs =
	1731	flex_eu_config_render_basic;
	1732	dev_priv->perf.oa.flex_regs_len =
	1733	ARRAY_SIZE(flex_eu_config_render_basic);
	1734
	1735	return 0;
	1736	case METRIC_SET_ID_COMPUTE_BASIC:
	1737	dev_priv->perf.oa.n_mux_configs =
	1738	get_compute_basic_mux_config(dev_priv,
	1739	dev_priv->perf.oa.mux_regs,
	1740	dev_priv->perf.oa.mux_regs_lens);
	1741	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1742	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_BASIC\" metric set\n");
	1743
	1744	/* EINVAL because *_register_sysfs already checked this
	1745	* and so it wouldn't have been advertised to userspace and
	1746	* so shouldn't have been requested
	1747	*/
	1748	return -EINVAL;
	1749	}
	1750
	1751	dev_priv->perf.oa.b_counter_regs =
	1752	b_counter_config_compute_basic;
	1753	dev_priv->perf.oa.b_counter_regs_len =
	1754	ARRAY_SIZE(b_counter_config_compute_basic);
	1755
	1756	dev_priv->perf.oa.flex_regs =
	1757	flex_eu_config_compute_basic;
	1758	dev_priv->perf.oa.flex_regs_len =
	1759	ARRAY_SIZE(flex_eu_config_compute_basic);
	1760
	1761	return 0;
	1762	case METRIC_SET_ID_RENDER_PIPE_PROFILE:
	1763	dev_priv->perf.oa.n_mux_configs =
	1764	get_render_pipe_profile_mux_config(dev_priv,
	1765	dev_priv->perf.oa.mux_regs,
	1766	dev_priv->perf.oa.mux_regs_lens);
	1767	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1768	DRM_DEBUG_DRIVER("No suitable MUX config for \"RENDER_PIPE_PROFILE\" metric set\n");
	1769
	1770	/* EINVAL because *_register_sysfs already checked this
	1771	* and so it wouldn't have been advertised to userspace and
	1772	* so shouldn't have been requested
	1773	*/
	1774	return -EINVAL;
	1775	}
	1776
	1777	dev_priv->perf.oa.b_counter_regs =
	1778	b_counter_config_render_pipe_profile;
	1779	dev_priv->perf.oa.b_counter_regs_len =
	1780	ARRAY_SIZE(b_counter_config_render_pipe_profile);
	1781
	1782	dev_priv->perf.oa.flex_regs =
	1783	flex_eu_config_render_pipe_profile;
	1784	dev_priv->perf.oa.flex_regs_len =
	1785	ARRAY_SIZE(flex_eu_config_render_pipe_profile);
	1786
	1787	return 0;
	1788	case METRIC_SET_ID_MEMORY_READS:
	1789	dev_priv->perf.oa.n_mux_configs =
	1790	get_memory_reads_mux_config(dev_priv,
	1791	dev_priv->perf.oa.mux_regs,
	1792	dev_priv->perf.oa.mux_regs_lens);
	1793	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1794	DRM_DEBUG_DRIVER("No suitable MUX config for \"MEMORY_READS\" metric set\n");
	1795
	1796	/* EINVAL because *_register_sysfs already checked this
	1797	* and so it wouldn't have been advertised to userspace and
	1798	* so shouldn't have been requested
	1799	*/
	1800	return -EINVAL;
	1801	}
	1802
	1803	dev_priv->perf.oa.b_counter_regs =
	1804	b_counter_config_memory_reads;
	1805	dev_priv->perf.oa.b_counter_regs_len =
	1806	ARRAY_SIZE(b_counter_config_memory_reads);
	1807
	1808	dev_priv->perf.oa.flex_regs =
	1809	flex_eu_config_memory_reads;
	1810	dev_priv->perf.oa.flex_regs_len =
	1811	ARRAY_SIZE(flex_eu_config_memory_reads);
	1812
	1813	return 0;
	1814	case METRIC_SET_ID_MEMORY_WRITES:
	1815	dev_priv->perf.oa.n_mux_configs =
	1816	get_memory_writes_mux_config(dev_priv,
	1817	dev_priv->perf.oa.mux_regs,
	1818	dev_priv->perf.oa.mux_regs_lens);
	1819	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1820	DRM_DEBUG_DRIVER("No suitable MUX config for \"MEMORY_WRITES\" metric set\n");
	1821
	1822	/* EINVAL because *_register_sysfs already checked this
	1823	* and so it wouldn't have been advertised to userspace and
	1824	* so shouldn't have been requested
	1825	*/
	1826	return -EINVAL;
	1827	}
	1828
	1829	dev_priv->perf.oa.b_counter_regs =
	1830	b_counter_config_memory_writes;
	1831	dev_priv->perf.oa.b_counter_regs_len =
	1832	ARRAY_SIZE(b_counter_config_memory_writes);
	1833
	1834	dev_priv->perf.oa.flex_regs =
	1835	flex_eu_config_memory_writes;
	1836	dev_priv->perf.oa.flex_regs_len =
	1837	ARRAY_SIZE(flex_eu_config_memory_writes);
	1838
	1839	return 0;
	1840	case METRIC_SET_ID_COMPUTE_EXTENDED:
	1841	dev_priv->perf.oa.n_mux_configs =
	1842	get_compute_extended_mux_config(dev_priv,
	1843	dev_priv->perf.oa.mux_regs,
	1844	dev_priv->perf.oa.mux_regs_lens);
	1845	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1846	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_EXTENDED\" metric set\n");
	1847
	1848	/* EINVAL because *_register_sysfs already checked this
	1849	* and so it wouldn't have been advertised to userspace and
	1850	* so shouldn't have been requested
	1851	*/
	1852	return -EINVAL;
	1853	}
	1854
	1855	dev_priv->perf.oa.b_counter_regs =
	1856	b_counter_config_compute_extended;
	1857	dev_priv->perf.oa.b_counter_regs_len =
	1858	ARRAY_SIZE(b_counter_config_compute_extended);
	1859
	1860	dev_priv->perf.oa.flex_regs =
	1861	flex_eu_config_compute_extended;
	1862	dev_priv->perf.oa.flex_regs_len =
	1863	ARRAY_SIZE(flex_eu_config_compute_extended);
	1864
	1865	return 0;
	1866	case METRIC_SET_ID_COMPUTE_L3_CACHE:
	1867	dev_priv->perf.oa.n_mux_configs =
	1868	get_compute_l3_cache_mux_config(dev_priv,
	1869	dev_priv->perf.oa.mux_regs,
	1870	dev_priv->perf.oa.mux_regs_lens);
	1871	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1872	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_L3_CACHE\" metric set\n");
	1873
	1874	/* EINVAL because *_register_sysfs already checked this
	1875	* and so it wouldn't have been advertised to userspace and
	1876	* so shouldn't have been requested
	1877	*/
	1878	return -EINVAL;
	1879	}
	1880
	1881	dev_priv->perf.oa.b_counter_regs =
	1882	b_counter_config_compute_l3_cache;
	1883	dev_priv->perf.oa.b_counter_regs_len =
	1884	ARRAY_SIZE(b_counter_config_compute_l3_cache);
	1885
	1886	dev_priv->perf.oa.flex_regs =
	1887	flex_eu_config_compute_l3_cache;
	1888	dev_priv->perf.oa.flex_regs_len =
	1889	ARRAY_SIZE(flex_eu_config_compute_l3_cache);
	1890
	1891	return 0;
	1892	case METRIC_SET_ID_HDC_AND_SF:
	1893	dev_priv->perf.oa.n_mux_configs =
	1894	get_hdc_and_sf_mux_config(dev_priv,
	1895	dev_priv->perf.oa.mux_regs,
	1896	dev_priv->perf.oa.mux_regs_lens);
	1897	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1898	DRM_DEBUG_DRIVER("No suitable MUX config for \"HDC_AND_SF\" metric set\n");
	1899
	1900	/* EINVAL because *_register_sysfs already checked this
	1901	* and so it wouldn't have been advertised to userspace and
	1902	* so shouldn't have been requested
	1903	*/
	1904	return -EINVAL;
	1905	}
	1906
	1907	dev_priv->perf.oa.b_counter_regs =
	1908	b_counter_config_hdc_and_sf;
	1909	dev_priv->perf.oa.b_counter_regs_len =
	1910	ARRAY_SIZE(b_counter_config_hdc_and_sf);
	1911
	1912	dev_priv->perf.oa.flex_regs =
	1913	flex_eu_config_hdc_and_sf;
	1914	dev_priv->perf.oa.flex_regs_len =
	1915	ARRAY_SIZE(flex_eu_config_hdc_and_sf);
	1916
	1917	return 0;
	1918	case METRIC_SET_ID_L3_1:
	1919	dev_priv->perf.oa.n_mux_configs =
	1920	get_l3_1_mux_config(dev_priv,
	1921	dev_priv->perf.oa.mux_regs,
	1922	dev_priv->perf.oa.mux_regs_lens);
	1923	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1924	DRM_DEBUG_DRIVER("No suitable MUX config for \"L3_1\" metric set\n");
	1925
	1926	/* EINVAL because *_register_sysfs already checked this
	1927	* and so it wouldn't have been advertised to userspace and
	1928	* so shouldn't have been requested
	1929	*/
	1930	return -EINVAL;
	1931	}
	1932
	1933	dev_priv->perf.oa.b_counter_regs =
	1934	b_counter_config_l3_1;
	1935	dev_priv->perf.oa.b_counter_regs_len =
	1936	ARRAY_SIZE(b_counter_config_l3_1);
	1937
	1938	dev_priv->perf.oa.flex_regs =
	1939	flex_eu_config_l3_1;
	1940	dev_priv->perf.oa.flex_regs_len =
	1941	ARRAY_SIZE(flex_eu_config_l3_1);
	1942
	1943	return 0;
	1944	case METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND:
	1945	dev_priv->perf.oa.n_mux_configs =
	1946	get_rasterizer_and_pixel_backend_mux_config(dev_priv,
	1947	dev_priv->perf.oa.mux_regs,
	1948	dev_priv->perf.oa.mux_regs_lens);
	1949	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1950	DRM_DEBUG_DRIVER("No suitable MUX config for \"RASTERIZER_AND_PIXEL_BACKEND\" metric set\n");
	1951
	1952	/* EINVAL because *_register_sysfs already checked this
	1953	* and so it wouldn't have been advertised to userspace and
	1954	* so shouldn't have been requested
	1955	*/
	1956	return -EINVAL;
	1957	}
	1958
	1959	dev_priv->perf.oa.b_counter_regs =
	1960	b_counter_config_rasterizer_and_pixel_backend;
	1961	dev_priv->perf.oa.b_counter_regs_len =
	1962	ARRAY_SIZE(b_counter_config_rasterizer_and_pixel_backend);
	1963
	1964	dev_priv->perf.oa.flex_regs =
	1965	flex_eu_config_rasterizer_and_pixel_backend;
	1966	dev_priv->perf.oa.flex_regs_len =
	1967	ARRAY_SIZE(flex_eu_config_rasterizer_and_pixel_backend);
	1968
	1969	return 0;
	1970	case METRIC_SET_ID_SAMPLER:
	1971	dev_priv->perf.oa.n_mux_configs =
	1972	get_sampler_mux_config(dev_priv,
	1973	dev_priv->perf.oa.mux_regs,
	1974	dev_priv->perf.oa.mux_regs_lens);
	1975	if (dev_priv->perf.oa.n_mux_configs == 0) {
	1976	DRM_DEBUG_DRIVER("No suitable MUX config for \"SAMPLER\" metric set\n");
	1977
	1978	/* EINVAL because *_register_sysfs already checked this
	1979	* and so it wouldn't have been advertised to userspace and
	1980	* so shouldn't have been requested
	1981	*/
	1982	return -EINVAL;
	1983	}
	1984
	1985	dev_priv->perf.oa.b_counter_regs =
	1986	b_counter_config_sampler;
	1987	dev_priv->perf.oa.b_counter_regs_len =
	1988	ARRAY_SIZE(b_counter_config_sampler);
	1989
	1990	dev_priv->perf.oa.flex_regs =
	1991	flex_eu_config_sampler;
	1992	dev_priv->perf.oa.flex_regs_len =
	1993	ARRAY_SIZE(flex_eu_config_sampler);
	1994
	1995	return 0;
	1996	case METRIC_SET_ID_TDL_1:
	1997	dev_priv->perf.oa.n_mux_configs =
	1998	get_tdl_1_mux_config(dev_priv,
	1999	dev_priv->perf.oa.mux_regs,
	2000	dev_priv->perf.oa.mux_regs_lens);
	2001	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2002	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_1\" metric set\n");
	2003
	2004	/* EINVAL because *_register_sysfs already checked this
	2005	* and so it wouldn't have been advertised to userspace and
	2006	* so shouldn't have been requested
	2007	*/
	2008	return -EINVAL;
	2009	}
	2010
	2011	dev_priv->perf.oa.b_counter_regs =
	2012	b_counter_config_tdl_1;
	2013	dev_priv->perf.oa.b_counter_regs_len =
	2014	ARRAY_SIZE(b_counter_config_tdl_1);
	2015
	2016	dev_priv->perf.oa.flex_regs =
	2017	flex_eu_config_tdl_1;
	2018	dev_priv->perf.oa.flex_regs_len =
	2019	ARRAY_SIZE(flex_eu_config_tdl_1);
	2020
	2021	return 0;
	2022	case METRIC_SET_ID_TDL_2:
	2023	dev_priv->perf.oa.n_mux_configs =
	2024	get_tdl_2_mux_config(dev_priv,
	2025	dev_priv->perf.oa.mux_regs,
	2026	dev_priv->perf.oa.mux_regs_lens);
	2027	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2028	DRM_DEBUG_DRIVER("No suitable MUX config for \"TDL_2\" metric set\n");
	2029
	2030	/* EINVAL because *_register_sysfs already checked this
	2031	* and so it wouldn't have been advertised to userspace and
	2032	* so shouldn't have been requested
	2033	*/
	2034	return -EINVAL;
	2035	}
	2036
	2037	dev_priv->perf.oa.b_counter_regs =
	2038	b_counter_config_tdl_2;
	2039	dev_priv->perf.oa.b_counter_regs_len =
	2040	ARRAY_SIZE(b_counter_config_tdl_2);
	2041
	2042	dev_priv->perf.oa.flex_regs =
	2043	flex_eu_config_tdl_2;
	2044	dev_priv->perf.oa.flex_regs_len =
	2045	ARRAY_SIZE(flex_eu_config_tdl_2);
	2046
	2047	return 0;
	2048	case METRIC_SET_ID_COMPUTE_EXTRA:
	2049	dev_priv->perf.oa.n_mux_configs =
	2050	get_compute_extra_mux_config(dev_priv,
	2051	dev_priv->perf.oa.mux_regs,
	2052	dev_priv->perf.oa.mux_regs_lens);
	2053	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2054	DRM_DEBUG_DRIVER("No suitable MUX config for \"COMPUTE_EXTRA\" metric set\n");
	2055
	2056	/* EINVAL because *_register_sysfs already checked this
	2057	* and so it wouldn't have been advertised to userspace and
	2058	* so shouldn't have been requested
	2059	*/
	2060	return -EINVAL;
	2061	}
	2062
	2063	dev_priv->perf.oa.b_counter_regs =
	2064	b_counter_config_compute_extra;
	2065	dev_priv->perf.oa.b_counter_regs_len =
	2066	ARRAY_SIZE(b_counter_config_compute_extra);
	2067
	2068	dev_priv->perf.oa.flex_regs =
	2069	flex_eu_config_compute_extra;
	2070	dev_priv->perf.oa.flex_regs_len =
	2071	ARRAY_SIZE(flex_eu_config_compute_extra);
	2072
	2073	return 0;
	2074	case METRIC_SET_ID_TEST_OA:
	2075	dev_priv->perf.oa.n_mux_configs =
	2076	get_test_oa_mux_config(dev_priv,
	2077	dev_priv->perf.oa.mux_regs,
	2078	dev_priv->perf.oa.mux_regs_lens);
	2079	if (dev_priv->perf.oa.n_mux_configs == 0) {
	2080	DRM_DEBUG_DRIVER("No suitable MUX config for \"TEST_OA\" metric set\n");
	2081
	2082	/* EINVAL because *_register_sysfs already checked this
	2083	* and so it wouldn't have been advertised to userspace and
	2084	* so shouldn't have been requested
	2085	*/
	2086	return -EINVAL;
	2087	}
	2088
	2089	dev_priv->perf.oa.b_counter_regs =
	2090	b_counter_config_test_oa;
	2091	dev_priv->perf.oa.b_counter_regs_len =
	2092	ARRAY_SIZE(b_counter_config_test_oa);
	2093
	2094	dev_priv->perf.oa.flex_regs =
	2095	flex_eu_config_test_oa;
	2096	dev_priv->perf.oa.flex_regs_len =
	2097	ARRAY_SIZE(flex_eu_config_test_oa);
	2098
	2099	return 0;
	2100	default:
	2101	return -ENODEV;
	2102	}
	2103	}
	2104
	2105	static ssize_t
	2106	show_render_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	2107	{
	2108	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_BASIC);
	2109	}
	2110
	2111	static struct device_attribute dev_attr_render_basic_id = {
	2112	.attr = { .name = "id", .mode = 0444 },
	2113	.show = show_render_basic_id,
	2114	.store = NULL,
	2115	};
	2116
	2117	static struct attribute *attrs_render_basic[] = {
	2118	&dev_attr_render_basic_id.attr,
	2119	NULL,
	2120	};
	2121
	2122	static struct attribute_group group_render_basic = {
	2123	.name = "d72df5c7-5b4a-4274-a43f-00b0fd51fc68",
	2124	.attrs = attrs_render_basic,
	2125	};
	2126
	2127	static ssize_t
	2128	show_compute_basic_id(struct device kdev, struct device_attribute attr, char *buf)
	2129	{
	2130	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_BASIC);
	2131	}
	2132
	2133	static struct device_attribute dev_attr_compute_basic_id = {
	2134	.attr = { .name = "id", .mode = 0444 },
	2135	.show = show_compute_basic_id,
	2136	.store = NULL,
	2137	};
	2138
	2139	static struct attribute *attrs_compute_basic[] = {
	2140	&dev_attr_compute_basic_id.attr,
	2141	NULL,
	2142	};
	2143
	2144	static struct attribute_group group_compute_basic = {
	2145	.name = "814285f6-354d-41d2-ba49-e24e622714a0",
	2146	.attrs = attrs_compute_basic,
	2147	};
	2148
	2149	static ssize_t
	2150	show_render_pipe_profile_id(struct device kdev, struct device_attribute attr, char *buf)
	2151	{
	2152	return sprintf(buf, "%d\n", METRIC_SET_ID_RENDER_PIPE_PROFILE);
	2153	}
	2154
	2155	static struct device_attribute dev_attr_render_pipe_profile_id = {
	2156	.attr = { .name = "id", .mode = 0444 },
	2157	.show = show_render_pipe_profile_id,
	2158	.store = NULL,
	2159	};
	2160
	2161	static struct attribute *attrs_render_pipe_profile[] = {
	2162	&dev_attr_render_pipe_profile_id.attr,
	2163	NULL,
	2164	};
	2165
	2166	static struct attribute_group group_render_pipe_profile = {
	2167	.name = "07d397a6-b3e6-49f6-9433-a4f293d55978",
	2168	.attrs = attrs_render_pipe_profile,
	2169	};
	2170
	2171	static ssize_t
	2172	show_memory_reads_id(struct device kdev, struct device_attribute attr, char *buf)
	2173	{
	2174	return sprintf(buf, "%d\n", METRIC_SET_ID_MEMORY_READS);
	2175	}
	2176
	2177	static struct device_attribute dev_attr_memory_reads_id = {
	2178	.attr = { .name = "id", .mode = 0444 },
	2179	.show = show_memory_reads_id,
	2180	.store = NULL,
	2181	};
	2182
	2183	static struct attribute *attrs_memory_reads[] = {
	2184	&dev_attr_memory_reads_id.attr,
	2185	NULL,
	2186	};
	2187
	2188	static struct attribute_group group_memory_reads = {
	2189	.name = "1a356946-5428-450b-a2f0-89f8783a302d",
	2190	.attrs = attrs_memory_reads,
	2191	};
	2192
	2193	static ssize_t
	2194	show_memory_writes_id(struct device kdev, struct device_attribute attr, char *buf)
	2195	{
	2196	return sprintf(buf, "%d\n", METRIC_SET_ID_MEMORY_WRITES);
	2197	}
	2198
	2199	static struct device_attribute dev_attr_memory_writes_id = {
	2200	.attr = { .name = "id", .mode = 0444 },
	2201	.show = show_memory_writes_id,
	2202	.store = NULL,
	2203	};
	2204
	2205	static struct attribute *attrs_memory_writes[] = {
	2206	&dev_attr_memory_writes_id.attr,
	2207	NULL,
	2208	};
	2209
	2210	static struct attribute_group group_memory_writes = {
	2211	.name = "5299be9d-7a61-4c99-9f81-f87e6c5aaca9",
	2212	.attrs = attrs_memory_writes,
	2213	};
	2214
	2215	static ssize_t
	2216	show_compute_extended_id(struct device kdev, struct device_attribute attr, char *buf)
	2217	{
	2218	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_EXTENDED);
	2219	}
	2220
	2221	static struct device_attribute dev_attr_compute_extended_id = {
	2222	.attr = { .name = "id", .mode = 0444 },
	2223	.show = show_compute_extended_id,
	2224	.store = NULL,
	2225	};
	2226
	2227	static struct attribute *attrs_compute_extended[] = {
	2228	&dev_attr_compute_extended_id.attr,
	2229	NULL,
	2230	};
	2231
	2232	static struct attribute_group group_compute_extended = {
	2233	.name = "bc9bcff2-459a-4cbc-986d-a84b077153f3",
	2234	.attrs = attrs_compute_extended,
	2235	};
	2236
	2237	static ssize_t
	2238	show_compute_l3_cache_id(struct device kdev, struct device_attribute attr, char *buf)
	2239	{
	2240	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_L3_CACHE);
	2241	}
	2242
	2243	static struct device_attribute dev_attr_compute_l3_cache_id = {
	2244	.attr = { .name = "id", .mode = 0444 },
	2245	.show = show_compute_l3_cache_id,
	2246	.store = NULL,
	2247	};
	2248
	2249	static struct attribute *attrs_compute_l3_cache[] = {
	2250	&dev_attr_compute_l3_cache_id.attr,
	2251	NULL,
	2252	};
	2253
	2254	static struct attribute_group group_compute_l3_cache = {
	2255	.name = "88ec931f-5b4a-453a-9db6-a61232b6143d",
	2256	.attrs = attrs_compute_l3_cache,
	2257	};
	2258
	2259	static ssize_t
	2260	show_hdc_and_sf_id(struct device kdev, struct device_attribute attr, char *buf)
	2261	{
	2262	return sprintf(buf, "%d\n", METRIC_SET_ID_HDC_AND_SF);
	2263	}
	2264
	2265	static struct device_attribute dev_attr_hdc_and_sf_id = {
	2266	.attr = { .name = "id", .mode = 0444 },
	2267	.show = show_hdc_and_sf_id,
	2268	.store = NULL,
	2269	};
	2270
	2271	static struct attribute *attrs_hdc_and_sf[] = {
	2272	&dev_attr_hdc_and_sf_id.attr,
	2273	NULL,
	2274	};
	2275
	2276	static struct attribute_group group_hdc_and_sf = {
	2277	.name = "530d176d-2a18-4014-adf8-1500c6c60835",
	2278	.attrs = attrs_hdc_and_sf,
	2279	};
	2280
	2281	static ssize_t
	2282	show_l3_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2283	{
	2284	return sprintf(buf, "%d\n", METRIC_SET_ID_L3_1);
	2285	}
	2286
	2287	static struct device_attribute dev_attr_l3_1_id = {
	2288	.attr = { .name = "id", .mode = 0444 },
	2289	.show = show_l3_1_id,
	2290	.store = NULL,
	2291	};
	2292
	2293	static struct attribute *attrs_l3_1[] = {
	2294	&dev_attr_l3_1_id.attr,
	2295	NULL,
	2296	};
	2297
	2298	static struct attribute_group group_l3_1 = {
	2299	.name = "fdee5a5a-f23c-43d1-aa73-f6257c71671d",
	2300	.attrs = attrs_l3_1,
	2301	};
	2302
	2303	static ssize_t
	2304	show_rasterizer_and_pixel_backend_id(struct device kdev, struct device_attribute attr, char *buf)
	2305	{
	2306	return sprintf(buf, "%d\n", METRIC_SET_ID_RASTERIZER_AND_PIXEL_BACKEND);
	2307	}
	2308
	2309	static struct device_attribute dev_attr_rasterizer_and_pixel_backend_id = {
	2310	.attr = { .name = "id", .mode = 0444 },
	2311	.show = show_rasterizer_and_pixel_backend_id,
	2312	.store = NULL,
	2313	};
	2314
	2315	static struct attribute *attrs_rasterizer_and_pixel_backend[] = {
	2316	&dev_attr_rasterizer_and_pixel_backend_id.attr,
	2317	NULL,
	2318	};
	2319
	2320	static struct attribute_group group_rasterizer_and_pixel_backend = {
	2321	.name = "6617623e-ca73-4791-b2b7-ddedd0846a0c",
	2322	.attrs = attrs_rasterizer_and_pixel_backend,
	2323	};
	2324
	2325	static ssize_t
	2326	show_sampler_id(struct device kdev, struct device_attribute attr, char *buf)
	2327	{
	2328	return sprintf(buf, "%d\n", METRIC_SET_ID_SAMPLER);
	2329	}
	2330
	2331	static struct device_attribute dev_attr_sampler_id = {
	2332	.attr = { .name = "id", .mode = 0444 },
	2333	.show = show_sampler_id,
	2334	.store = NULL,
	2335	};
	2336
	2337	static struct attribute *attrs_sampler[] = {
	2338	&dev_attr_sampler_id.attr,
	2339	NULL,
	2340	};
	2341
	2342	static struct attribute_group group_sampler = {
	2343	.name = "f3b2ea63-e82e-4234-b418-44dd20dd34d0",
	2344	.attrs = attrs_sampler,
	2345	};
	2346
	2347	static ssize_t
	2348	show_tdl_1_id(struct device kdev, struct device_attribute attr, char *buf)
	2349	{
	2350	return sprintf(buf, "%d\n", METRIC_SET_ID_TDL_1);
	2351	}
	2352
	2353	static struct device_attribute dev_attr_tdl_1_id = {
	2354	.attr = { .name = "id", .mode = 0444 },
	2355	.show = show_tdl_1_id,
	2356	.store = NULL,
	2357	};
	2358
	2359	static struct attribute *attrs_tdl_1[] = {
	2360	&dev_attr_tdl_1_id.attr,
	2361	NULL,
	2362	};
	2363
	2364	static struct attribute_group group_tdl_1 = {
	2365	.name = "14411d35-cbf6-4f5e-b68b-190faf9a1a83",
	2366	.attrs = attrs_tdl_1,
	2367	};
	2368
	2369	static ssize_t
	2370	show_tdl_2_id(struct device kdev, struct device_attribute attr, char *buf)
	2371	{
	2372	return sprintf(buf, "%d\n", METRIC_SET_ID_TDL_2);
	2373	}
	2374
	2375	static struct device_attribute dev_attr_tdl_2_id = {
	2376	.attr = { .name = "id", .mode = 0444 },
	2377	.show = show_tdl_2_id,
	2378	.store = NULL,
	2379	};
	2380
	2381	static struct attribute *attrs_tdl_2[] = {
	2382	&dev_attr_tdl_2_id.attr,
	2383	NULL,
	2384	};
	2385
	2386	static struct attribute_group group_tdl_2 = {
	2387	.name = "ffa3f263-0478-4724-8c9f-c911c5ec0f1d",
	2388	.attrs = attrs_tdl_2,
	2389	};
	2390
	2391	static ssize_t
	2392	show_compute_extra_id(struct device kdev, struct device_attribute attr, char *buf)
	2393	{
	2394	return sprintf(buf, "%d\n", METRIC_SET_ID_COMPUTE_EXTRA);
	2395	}
	2396
	2397	static struct device_attribute dev_attr_compute_extra_id = {
	2398	.attr = { .name = "id", .mode = 0444 },
	2399	.show = show_compute_extra_id,
	2400	.store = NULL,
	2401	};
	2402
	2403	static struct attribute *attrs_compute_extra[] = {
	2404	&dev_attr_compute_extra_id.attr,
	2405	NULL,
	2406	};
	2407
	2408	static struct attribute_group group_compute_extra = {
	2409	.name = "15274c82-27d2-4819-876a-7cb1a2c59ba4",
	2410	.attrs = attrs_compute_extra,
	2411	};
	2412
	2413	static ssize_t
	2414	show_test_oa_id(struct device kdev, struct device_attribute attr, char *buf)
	2415	{
	2416	return sprintf(buf, "%d\n", METRIC_SET_ID_TEST_OA);
	2417	}
	2418
	2419	static struct device_attribute dev_attr_test_oa_id = {
	2420	.attr = { .name = "id", .mode = 0444 },
	2421	.show = show_test_oa_id,
	2422	.store = NULL,
	2423	};
	2424
	2425	static struct attribute *attrs_test_oa[] = {
	2426	&dev_attr_test_oa_id.attr,
	2427	NULL,
	2428	};
	2429
	2430	static struct attribute_group group_test_oa = {
	2431	.name = "dd3fd789-e783-4204-8cd0-b671bbccb0cf",
	2432	.attrs = attrs_test_oa,
	2433	};
	2434
	2435	int
	2436	i915_perf_register_sysfs_glk(struct drm_i915_private *dev_priv)
	2437	{
	2438	const struct i915_oa_reg *mux_regs[ARRAY_SIZE(dev_priv->perf.oa.mux_regs)];
	2439	int mux_lens[ARRAY_SIZE(dev_priv->perf.oa.mux_regs_lens)];
	2440	int ret = 0;
	2441
	2442	if (get_render_basic_mux_config(dev_priv, mux_regs, mux_lens)) {
	2443	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_render_basic);
	2444	if (ret)
	2445	goto error_render_basic;
	2446	}
	2447	if (get_compute_basic_mux_config(dev_priv, mux_regs, mux_lens)) {
	2448	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_basic);
	2449	if (ret)
	2450	goto error_compute_basic;
	2451	}
	2452	if (get_render_pipe_profile_mux_config(dev_priv, mux_regs, mux_lens)) {
	2453	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_render_pipe_profile);
	2454	if (ret)
	2455	goto error_render_pipe_profile;
	2456	}
	2457	if (get_memory_reads_mux_config(dev_priv, mux_regs, mux_lens)) {
	2458	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_memory_reads);
	2459	if (ret)
	2460	goto error_memory_reads;
	2461	}
	2462	if (get_memory_writes_mux_config(dev_priv, mux_regs, mux_lens)) {
	2463	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_memory_writes);
	2464	if (ret)
	2465	goto error_memory_writes;
	2466	}
	2467	if (get_compute_extended_mux_config(dev_priv, mux_regs, mux_lens)) {
	2468	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_extended);
	2469	if (ret)
	2470	goto error_compute_extended;
	2471	}
	2472	if (get_compute_l3_cache_mux_config(dev_priv, mux_regs, mux_lens)) {
	2473	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_l3_cache);
	2474	if (ret)
	2475	goto error_compute_l3_cache;
	2476	}
	2477	if (get_hdc_and_sf_mux_config(dev_priv, mux_regs, mux_lens)) {
	2478	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_hdc_and_sf);
	2479	if (ret)
	2480	goto error_hdc_and_sf;
	2481	}
	2482	if (get_l3_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2483	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_l3_1);
	2484	if (ret)
	2485	goto error_l3_1;
	2486	}
	2487	if (get_rasterizer_and_pixel_backend_mux_config(dev_priv, mux_regs, mux_lens)) {
	2488	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_rasterizer_and_pixel_backend);
	2489	if (ret)
	2490	goto error_rasterizer_and_pixel_backend;
	2491	}
	2492	if (get_sampler_mux_config(dev_priv, mux_regs, mux_lens)) {
	2493	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_sampler);
	2494	if (ret)
	2495	goto error_sampler;
	2496	}
	2497	if (get_tdl_1_mux_config(dev_priv, mux_regs, mux_lens)) {
	2498	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_tdl_1);
	2499	if (ret)
	2500	goto error_tdl_1;
	2501	}
	2502	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens)) {
	2503	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2504	if (ret)
	2505	goto error_tdl_2;
	2506	}
	2507	if (get_compute_extra_mux_config(dev_priv, mux_regs, mux_lens)) {
	2508	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_compute_extra);
	2509	if (ret)
	2510	goto error_compute_extra;
	2511	}
	2512	if (get_test_oa_mux_config(dev_priv, mux_regs, mux_lens)) {
	2513	ret = sysfs_create_group(dev_priv->perf.metrics_kobj, &group_test_oa);
	2514	if (ret)
	2515	goto error_test_oa;
	2516	}
	2517
	2518	return 0;
	2519
	2520	error_test_oa:
	2521	if (get_compute_extra_mux_config(dev_priv, mux_regs, mux_lens))
	2522	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_compute_extra);
	2523	error_compute_extra:
	2524	if (get_tdl_2_mux_config(dev_priv, mux_regs, mux_lens))
	2525	sysfs_remove_group(dev_priv->perf.metrics_kobj, &group_tdl_2);
	2526	error_tdl_2:

76 more lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_glk.h+40−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_hsw.c+187−76

This diff is large: its 499 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_hsw.h+3−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_kblgt2.c+2991−0

This diff is large: its 2992 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_kblgt2.h+40−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_kblgt3.c+3040−0

This diff is large: its 3041 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_kblgt3.h+40−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_sklgt2.c+3479−0

This diff is large: its 3480 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_sklgt2.h+40−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_sklgt3.c+3039−0

This diff is large: its 3040 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_sklgt3.h+40−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_sklgt4.c+3093−0

This diff is large: its 3094 lines aren’t shown here.

drivers/gpu/drm/i915/i915_oa_sklgt4.h+40−0

This diff is large: its 41 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pci.c+57−30

This diff is large: its 133 lines aren’t shown here.

drivers/gpu/drm/i915/i915_perf.c+1032−94

This diff is large: its 1401 lines aren’t shown here.

drivers/gpu/drm/i915/i915_pvinfo.h+2−6

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/i915/i915_reg.h+270−17

This diff is large: its 445 lines aren’t shown here.

drivers/gpu/drm/i915/i915_utils.h+5−0

This diff is large: its 12 lines aren’t shown here.

drivers/gpu/drm/i915/i915_vgpu.c+35−19

This diff is large: its 122 lines aren’t shown here.

drivers/gpu/drm/i915/i915_vma.c+25−3

This diff is large: its 66 lines aren’t shown here.

drivers/gpu/drm/i915/i915_vma.h+15−5

This diff is large: its 48 lines aren’t shown here.

drivers/gpu/drm/i915/intel_atomic.c+107−30

This diff is large: its 160 lines aren’t shown here.

drivers/gpu/drm/i915/intel_breadcrumbs.c+43−30

This diff is large: its 155 lines aren’t shown here.

drivers/gpu/drm/i915/intel_cdclk.c+350−5

This diff is large: its 413 lines aren’t shown here.

drivers/gpu/drm/i915/intel_csr.c+12−5

This diff is large: its 47 lines aren’t shown here.

drivers/gpu/drm/i915/intel_ddi.c+377−15

This diff is large: its 515 lines aren’t shown here.

drivers/gpu/drm/i915/intel_device_info.c+6−5

This diff is large: its 38 lines aren’t shown here.

drivers/gpu/drm/i915/intel_display.c+162−158

This diff is large: its 555 lines aren’t shown here.

drivers/gpu/drm/i915/intel_dp.c+48−142

This diff is large: its 368 lines aren’t shown here.

drivers/gpu/drm/i915/intel_dp_aux_backlight.c+9−8

This diff is large: its 57 lines aren’t shown here.

drivers/gpu/drm/i915/intel_dpll_mgr.c+435−2

This diff is large: its 458 lines aren’t shown here.

drivers/gpu/drm/i915/intel_dpll_mgr.h+4−0

This diff is large: its 11 lines aren’t shown here.

drivers/gpu/drm/i915/intel_drv.h+39−20

This diff is large: its 148 lines aren’t shown here.

drivers/gpu/drm/i915/intel_dsi.c+83−84

This diff is large: its 286 lines aren’t shown here.

drivers/gpu/drm/i915/intel_dsi_dcs_backlight.c+10−12

This diff is large: its 56 lines aren’t shown here.

drivers/gpu/drm/i915/intel_engine_cs.c+64−22

This diff is large: its 179 lines aren’t shown here.

drivers/gpu/drm/i915/intel_fifo_underrun.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_guc_loader.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_gvt.c+32−11

This diff is large: its 59 lines aren’t shown here.

drivers/gpu/drm/i915/intel_gvt.h+5−0

This diff is large: its 19 lines aren’t shown here.

drivers/gpu/drm/i915/intel_hdmi.c+38−119

This diff is large: its 296 lines aren’t shown here.

drivers/gpu/drm/i915/intel_huc.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_i2c.c+13−2

This diff is large: its 33 lines aren’t shown here.

drivers/gpu/drm/i915/intel_lrc.c+7−0

This diff is large: its 28 lines aren’t shown here.

drivers/gpu/drm/i915/intel_lvds.c+16−57

This diff is large: its 144 lines aren’t shown here.

drivers/gpu/drm/i915/intel_mocs.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_opregion.c+1−1

This diff is large: its 9 lines aren’t shown here.

drivers/gpu/drm/i915/intel_overlay.c+0−1

This diff is large: its 8 lines aren’t shown here.

drivers/gpu/drm/i915/intel_panel.c+186−79

This diff is large: its 581 lines aren’t shown here.

drivers/gpu/drm/i915/intel_pm.c+126−22

This diff is large: its 252 lines aren’t shown here.

drivers/gpu/drm/i915/intel_runtime_pm.c+308−8

This diff is large: its 403 lines aren’t shown here.

drivers/gpu/drm/i915/intel_sdvo.c+284−256

This diff is large: its 834 lines aren’t shown here.

drivers/gpu/drm/i915/intel_uc.c+5−2

This diff is large: its 22 lines aren’t shown here.

drivers/gpu/drm/i915/intel_uncore.c+0−126

This diff is large: its 168 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/huge_gem_object.c+3−1

This diff is large: its 13 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_gem_evict.c+2−2

This diff is large: its 18 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/i915_vma.c+3−13

This diff is large: its 46 lines aren’t shown here.

drivers/gpu/drm/i915/selftests/mock_context.c+11−1

This diff is large: its 29 lines aren’t shown here.

include/drm/i915_pciids.h+40−0

This diff is large: its 45 lines aren’t shown here.

include/uapi/drm/i915_drm.h+36−8

This diff is large: its 65 lines aren’t shown here.

Merge tag 'drm-intel-next-2017-06-19' of git://anongit.freedesktop.org/git/drm-intel into drm-next

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code