Merge tag 'trace-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace

Pull tracing updates from Steven Rostedt: - Added option for per CPU threads to the hwlat tracer - Have hwlat tracer handle hotplug CPUs - New tracer: osnoise, that detects latency caused by interrupts, softirqs and scheduling of other tasks. - Added timerlat tracer that creates a thread and measures in detail what sources of latency it has for wake ups. - Removed the "success" field of the sched_wakeup trace event. This has been hardcoded as "1" since 2015, no tooling should be looking at it now. If one exists, we can revert this commit, fix that tool and try to remove it again in the future. - tgid mapping fixed to handle more than PID_MAX_DEFAULT pids/tgids. - New boot command line option "tp_printk_stop", as tp_printk causes trace events to write to console. When user space starts, this can easily live lock the system. Having a boot option to stop just after boot up is useful to prevent that from happening. - Have ftrace_dump_on_oops boot command line option take numbers that match the numbers shown in /proc/sys/kernel/ftrace_dump_on_oops. - Bootconfig clean ups, fixes and enhancements. - New ktest script that tests bootconfig options. - Add tracepoint_probe_register_may_exist() to register a tracepoint without triggering a WARN*() if it already exists. BPF has a path from user space that can do this. All other paths are considered a bug. - Small clean ups and fixes * tag 'trace-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace: (49 commits) tracing: Resize tgid_map to pid_max, not PID_MAX_DEFAULT tracing: Simplify & fix saved_tgids logic treewide: Add missing semicolons to __assign_str uses tracing: Change variable type as bool for clean-up trace/timerlat: Fix indentation on timerlat_main() trace/osnoise: Make 'noise' variable s64 in run_osnoise() tracepoint: Add tracepoint_probe_register_may_exist() for BPF tracing tracing: Fix spelling in osnoise tracer "interferences" -> "interference" Documentation: Fix a typo on trace/osnoise-tracer trace/osnoise: Fix return value on osnoise_init_hotplug_support trace/osnoise: Make interval u64 on osnoise_main trace/osnoise: Fix 'no previous prototype' warnings tracing: Have osnoise_main() add a quiescent state for task rcu seq_buf: Make trace_seq_putmem_hex() support data longer than 8 seq_buf: Fix overflow in seq_buf_putmem_hex() trace/osnoise: Support hotplug operations trace/hwlat: Support hotplug operations trace/hwlat: Protect kdata->kthread with get/put_online_cpus trace: Add timerlat tracer trace: Add osnoise tracer ...

Linus Torvalds committed 5 years ago

Commit 757fa802 parents a70bb58 4030a6e74 files · +4439 −369Browse files

Showing the changes to kernel/trace (14 of 74 files). Show all

Unified · Split

14 files

kernel/trace/Kconfig +62 −0
kernel/trace/Makefile +1 −0
kernel/trace/bpf_trace.c +2 −1
kernel/trace/ring_buffer.c +1 −1
kernel/trace/trace.c +211 −54
kernel/trace/trace.h +29 −6
kernel/trace/trace_boot.c +25 −2
kernel/trace/trace_entries.h +41 −0
kernel/trace/trace_events_hist.c +2 −1
kernel/trace/trace_events_trigger.c +2 −1
kernel/trace/trace_hwlat.c +392 −142
kernel/trace/trace_osnoise.c +2059 −0
kernel/trace/trace_output.c +118 −1
kernel/trace/trace_sched_wakeup.c +12 −12

kernel/trace/Kconfig+62−0

@@ -356,6 +356,68 @@ config HWLAT_TRACER · 355 unmodified lines
356	356	file. Every time a latency is greater than tracing_thresh, it will
357	357	be recorded into the ring buffer.
358	358
	359	+config OSNOISE_TRACER
	360	+ bool "OS Noise tracer"
	361	+ select GENERIC_TRACER
	362	+ help
	363	+ In the context of high-performance computing (HPC), the Operating
	364	+ System Noise (osnoise) refers to the interference experienced by an
	365	+ application due to activities inside the operating system. In the
	366	+ context of Linux, NMIs, IRQs, SoftIRQs, and any other system thread
	367	+ can cause noise to the system. Moreover, hardware-related jobs can
	368	+ also cause noise, for example, via SMIs.
	369	+
	370	+ The osnoise tracer leverages the hwlat_detector by running a similar
	371	+ loop with preemption, SoftIRQs and IRQs enabled, thus allowing all
	372	+ the sources of osnoise during its execution. The osnoise tracer takes
	373	+ note of the entry and exit point of any source of interferences,
	374	+ increasing a per-cpu interference counter. It saves an interference
	375	+ counter for each source of interference. The interference counter for
	376	+ NMI, IRQs, SoftIRQs, and threads is increased anytime the tool
	377	+ observes these interferences' entry events. When a noise happens
	378	+ without any interference from the operating system level, the
	379	+ hardware noise counter increases, pointing to a hardware-related
	380	+ noise. In this way, osnoise can account for any source of
	381	+ interference. At the end of the period, the osnoise tracer prints
	382	+ the sum of all noise, the max single noise, the percentage of CPU
	383	+ available for the thread, and the counters for the noise sources.
	384	+
	385	+ In addition to the tracer, a set of tracepoints were added to
	386	+ facilitate the identification of the osnoise source.
	387	+
	388	+ The output will appear in the trace and trace_pipe files.
	389	+
	390	+ To enable this tracer, echo in "osnoise" into the current_tracer
	391	+ file.
	392	+
	393	+config TIMERLAT_TRACER
	394	+ bool "Timerlat tracer"
	395	+ select OSNOISE_TRACER
	396	+ select GENERIC_TRACER
	397	+ help
	398	+ The timerlat tracer aims to help the preemptive kernel developers
	399	+ to find sources of wakeup latencies of real-time threads.
	400	+
	401	+ The tracer creates a per-cpu kernel thread with real-time priority.
	402	+ The tracer thread sets a periodic timer to wakeup itself, and goes
	403	+ to sleep waiting for the timer to fire. At the wakeup, the thread
	404	+ then computes a wakeup latency value as the difference between
	405	+ the current time and the absolute time that the timer was set
	406	+ to expire.
	407	+
	408	+ The tracer prints two lines at every activation. The first is the
	409	+ timer latency observed at the hardirq context before the
	410	+ activation of the thread. The second is the timer latency observed
	411	+ by the thread, which is the same level that cyclictest reports. The
	412	+ ACTIVATION ID field serves to relate the irq execution to its
	413	+ respective thread execution.
	414	+
	415	+ The tracer is build on top of osnoise tracer, and the osnoise:
	416	+ events can be used to trace the source of interference from NMI,
	417	+ IRQs and other threads. It also enables the capture of the
	418	+ stacktrace at the IRQ context, which helps to identify the code
	419	+ path that can cause thread delay.
	420	+
359	421	config MMIOTRACE
360	422	bool "Memory mapped IO tracing"
361	423	depends on HAVE_MMIOTRACE_SUPPORT && PCI

kernel/trace/Makefile+1−0

@@ -58,6 +58,7 @@ obj-$(CONFIG_IRQSOFF_TRACER) += trace_irqsoff.o · 57 unmodified lines
58	58	obj-$(CONFIG_PREEMPT_TRACER) += trace_irqsoff.o
59	59	obj-$(CONFIG_SCHED_TRACER) += trace_sched_wakeup.o
60	60	obj-$(CONFIG_HWLAT_TRACER) += trace_hwlat.o
	61	+obj-$(CONFIG_OSNOISE_TRACER) += trace_osnoise.o
61	62	obj-$(CONFIG_NOP_TRACER) += trace_nop.o
62	63	obj-$(CONFIG_STACK_TRACER) += trace_stack.o
63	64	obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o

kernel/trace/bpf_trace.c+2−1

@@ -1842,7 +1842,8 @@ static int __bpf_probe_register(struct bpf_raw_event_map btp, struct bpf_prog · 1841 unmodified lines
1842	1842	if (prog->aux->max_tp_access > btp->writable_size)
1843	1843	return -EINVAL;
1844	1844
1845		− return tracepoint_probe_register(tp, (void *)btp->bpf_func, prog);
	1845	+ return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func,
	1846	+ prog);
1846	1847	}
1847	1848
1848	1849	int bpf_probe_register(struct bpf_raw_event_map btp, struct bpf_prog prog)

kernel/trace/ring_buffer.c+1−1

@@ -3391,7 +3391,7 @@ static void check_buffer(struct ring_buffer_per_cpu *cpu_buffer, · 3390 unmodified lines
3391	3391	case RINGBUF_TYPE_PADDING:
3392	3392	if (event->time_delta == 1)
3393	3393	break;
3394		− /* fall through */
	3394	+ fallthrough;
3395	3395	case RINGBUF_TYPE_DATA:
3396	3396	ts += event->time_delta;
3397	3397	break;

kernel/trace/trace.c+211−54

@@ -87,6 +87,7 @@ void __init disable_tracing_selftest(const char *reason) · 86 unmodified lines
87	87	/* Pipe tracepoints to printk */
88	88	struct trace_iterator *tracepoint_print_iter;
89	89	int tracepoint_printk;
	90	+static bool tracepoint_printk_stop_on_boot __initdata;
90	91	static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);
91	92
92	93	/* For tracers that don't implement custom flags */
@@ -197,12 +198,12 @@ __setup("ftrace=", set_cmdline_ftrace); · 104 unmodified lines
197	198
198	199	static int __init set_ftrace_dump_on_oops(char *str)
199	200	{
200		− if (str++ != '=' \|\| !str) {
	201	+ if (str++ != '=' \|\| !str \|\| !strcmp("1", str)) {
201	202	ftrace_dump_on_oops = DUMP_ALL;
202	203	return 1;
203	204	}
204	205
205		− if (!strcmp("orig_cpu", str)) {
	206	+ if (!strcmp("orig_cpu", str) \|\| !strcmp("2", str)) {
206	207	ftrace_dump_on_oops = DUMP_ORIG;
207	208	return 1;
208	209	}
@@ -257,6 +258,13 @@ static int __init set_tracepoint_printk(char *str) · 48 unmodified lines
257	258	}
258	259	__setup("tp_printk", set_tracepoint_printk);
259	260
	261	+static int __init set_tracepoint_printk_stop(char *str)
	262	+{
	263	+ tracepoint_printk_stop_on_boot = true;
	264	+ return 1;
	265	+}
	266	+__setup("tp_printk_stop_on_boot", set_tracepoint_printk_stop);
	267	+
260	268	unsigned long long ns2usecs(u64 nsec)
261	269	{
262	270	nsec += 500;
@@ -1683,8 +1691,7 @@ static ssize_t trace_seq_to_buffer(struct trace_seq s, void buf, size_t cnt) · 1420 unmodified lines
1683	1691	unsigned long __read_mostly tracing_thresh;
1684	1692	static const struct file_operations tracing_max_lat_fops;
1685	1693
1686		−#if (defined(CONFIG_TRACER_MAX_TRACE) \|\| defined(CONFIG_HWLAT_TRACER)) && \
1687		− defined(CONFIG_FSNOTIFY)
	1694	+#ifdef LATENCY_FS_NOTIFY
1688	1695
1689	1696	static struct workqueue_struct *fsnotify_wq;
1690	1697
@@ -2185,8 +2192,15 @@ void tracing_reset_all_online_cpus(void) · 494 unmodified lines
2185	2192	}
2186	2193	}
2187	2194
	2195	+/*
	2196	+ * The tgid_map array maps from pid to tgid; i.e. the value stored at index i
	2197	+ * is the tgid last observed corresponding to pid=i.
	2198	+ */
2188	2199	static int *tgid_map;
2189	2200
	2201	+/* The maximum valid index into tgid_map. */
	2202	+static size_t tgid_map_max;
	2203	+
2190	2204	#define SAVED_CMDLINES_DEFAULT 128
2191	2205	#define NO_CMDLINE_MAP UINT_MAX
2192	2206	static arch_spinlock_t trace_cmdline_lock = __ARCH_SPIN_LOCK_UNLOCKED;
@@ -2459,24 +2473,41 @@ void trace_find_cmdline(int pid, char comm[]) · 266 unmodified lines
2459	2473	preempt_enable();
2460	2474	}
2461	2475
	2476	+static int *trace_find_tgid_ptr(int pid)
	2477	+{
	2478	+ /*
	2479	+ * Pairs with the smp_store_release in set_tracer_flag() to ensure that
	2480	+ * if we observe a non-NULL tgid_map then we also observe the correct
	2481	+ * tgid_map_max.
	2482	+ */
	2483	+ int *map = smp_load_acquire(&tgid_map);
	2484	+
	2485	+ if (unlikely(!map \|\| pid > tgid_map_max))
	2486	+ return NULL;
	2487	+
	2488	+ return &map[pid];
	2489	+}
	2490	+
2462	2491	int trace_find_tgid(int pid)
2463	2492	{
2464		− if (unlikely(!tgid_map \|\| !pid \|\| pid > PID_MAX_DEFAULT))
2465		− return 0;
	2493	+ int *ptr = trace_find_tgid_ptr(pid);
2466	2494
2467		− return tgid_map[pid];
	2495	+ return ptr ? *ptr : 0;
2468	2496	}
2469	2497
2470	2498	static int trace_save_tgid(struct task_struct *tsk)
2471	2499	{
	2500	+ int *ptr;
	2501	+
2472	2502	/* treat recording of idle task as a success */
2473	2503	if (!tsk->pid)
2474	2504	return 1;
2475	2505
2476		− if (unlikely(!tgid_map \|\| tsk->pid > PID_MAX_DEFAULT))
	2506	+ ptr = trace_find_tgid_ptr(tsk->pid);
	2507	+ if (!ptr)
2477	2508	return 0;
2478	2509
2479		− tgid_map[tsk->pid] = tsk->tgid;
	2510	+ *ptr = tsk->tgid;
2480	2511	return 1;
2481	2512	}
2482	2513
@@ -2730,9 +2761,45 @@ trace_event_buffer_lock_reserve(struct trace_buffer **current_rb, · 247 unmodified lines
2730	2761	if (!tr->no_filter_buffering_ref &&
2731	2762	(trace_file->flags & (EVENT_FILE_FL_SOFT_DISABLED \| EVENT_FILE_FL_FILTERED)) &&
2732	2763	(entry = this_cpu_read(trace_buffered_event))) {
2733		− /* Try to use the per cpu buffer first */
	2764	+ /*
	2765	+ * Filtering is on, so try to use the per cpu buffer first.
	2766	+ * This buffer will simulate a ring_buffer_event,
	2767	+ * where the type_len is zero and the array[0] will
	2768	+ * hold the full length.
	2769	+ * (see include/linux/ring-buffer.h for details on
	2770	+ * how the ring_buffer_event is structured).
	2771	+ *
	2772	+ * Using a temp buffer during filtering and copying it
	2773	+ * on a matched filter is quicker than writing directly
	2774	+ * into the ring buffer and then discarding it when
	2775	+ * it doesn't match. That is because the discard
	2776	+ * requires several atomic operations to get right.
	2777	+ * Copying on match and doing nothing on a failed match
	2778	+ * is still quicker than no copy on match, but having
	2779	+ * to discard out of the ring buffer on a failed match.
	2780	+ */
	2781	+ int max_len = PAGE_SIZE - struct_size(entry, array, 1);
	2782	+
2734	2783	val = this_cpu_inc_return(trace_buffered_event_cnt);
2735		− if ((len < (PAGE_SIZE - sizeof(*entry) - sizeof(entry->array[0]))) && val == 1) {
	2784	+
	2785	+ /*
	2786	+ * Preemption is disabled, but interrupts and NMIs
	2787	+ * can still come in now. If that happens after
	2788	+ * the above increment, then it will have to go
	2789	+ * back to the old method of allocating the event
	2790	+ * on the ring buffer, and if the filter fails, it
	2791	+ * will have to call ring_buffer_discard_commit()
	2792	+ * to remove it.
	2793	+ *
	2794	+ * Need to also check the unlikely case that the
	2795	+ * length is bigger than the temp buffer size.
	2796	+ * If that happens, then the reserve is pretty much
	2797	+ * guaranteed to fail, as the ring buffer currently
	2798	+ * only allows events less than a page. But that may
	2799	+ * change in the future, so let the ring buffer reserve
	2800	+ * handle the failure in that case.
	2801	+ */
	2802	+ if (val == 1 && likely(len <= max_len)) {
2736	2803	trace_event_setup(entry, type, trace_ctx);
2737	2804	entry->array[0] = len;
2738	2805	return entry;
@@ -5172,6 +5239,8 @@ int trace_keep_overwrite(struct tracer *tracer, u32 mask, int set) · 2433 unmodified lines
5172	5239
5173	5240	int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)
5174	5241	{
	5242	+ int *map;
	5243	+
5175	5244	if ((mask == TRACE_ITER_RECORD_TGID) \|\|
5176	5245	(mask == TRACE_ITER_RECORD_CMD))
5177	5246	lockdep_assert_held(&event_mutex);
@@ -5194,10 +5263,19 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled) · 16 unmodified lines
5194	5263	trace_event_enable_cmd_record(enabled);
5195	5264
5196	5265	if (mask == TRACE_ITER_RECORD_TGID) {
5197		− if (!tgid_map)
5198		− tgid_map = kvcalloc(PID_MAX_DEFAULT + 1,
5199		− sizeof(*tgid_map),
5200		− GFP_KERNEL);
	5266	+ if (!tgid_map) {
	5267	+ tgid_map_max = pid_max;
	5268	+ map = kvcalloc(tgid_map_max + 1, sizeof(*tgid_map),
	5269	+ GFP_KERNEL);
	5270	+
	5271	+ /*
	5272	+ * Pairs with smp_load_acquire() in
	5273	+ * trace_find_tgid_ptr() to ensure that if it observes
	5274	+ * the tgid_map we just allocated then it also observes
	5275	+ * the corresponding tgid_map_max value.
	5276	+ */
	5277	+ smp_store_release(&tgid_map, map);
	5278	+ }
5201	5279	if (!tgid_map) {
5202	5280	tr->trace_flags &= ~TRACE_ITER_RECORD_TGID;
5203	5281	return -ENOMEM;
@@ -5609,37 +5687,16 @@ static const struct file_operations tracing_readme_fops = { · 405 unmodified lines
5609	5687
5610	5688	static void saved_tgids_next(struct seq_file m, void v, loff_t pos)
5611	5689	{
5612		− int *ptr = v;
5613		−
5614		− if (*pos \|\| m->count)
5615		− ptr++;
5616		−
5617		− (*pos)++;
5618		−
5619		− for (; ptr <= &tgid_map[PID_MAX_DEFAULT]; ptr++) {
5620		− if (trace_find_tgid(*ptr))
5621		− return ptr;
5622		− }
	5690	+ int pid = ++(*pos);
5623	5691
5624		− return NULL;
	5692	+ return trace_find_tgid_ptr(pid);
5625	5693	}
5626	5694
5627	5695	static void saved_tgids_start(struct seq_file m, loff_t *pos)
5628	5696	{
5629		− void *v;
5630		− loff_t l = 0;
5631		−
5632		− if (!tgid_map)
5633		− return NULL;
	5697	+ int pid = *pos;
5634	5698
5635		− v = &tgid_map[0];
5636		− while (l <= *pos) {
5637		− v = saved_tgids_next(m, v, &l);
5638		− if (!v)
5639		− return NULL;
5640		− }
5641		−
5642		− return v;
	5699	+ return trace_find_tgid_ptr(pid);
5643	5700	}
5644	5701
5645	5702	static void saved_tgids_stop(struct seq_file m, void v)
@@ -5648,9 +5705,14 @@ static void saved_tgids_stop(struct seq_file m, void v) · 2 unmodified lines
5648	5705
5649	5706	static int saved_tgids_show(struct seq_file m, void v)
5650	5707	{
5651		− int pid = (int *)v - tgid_map;
	5708	+ int entry = (int )v;
	5709	+ int pid = entry - tgid_map;
	5710	+ int tgid = *entry;
5652	5711
5653		− seq_printf(m, "%d %d\n", pid, trace_find_tgid(pid));
	5712	+ if (tgid == 0)
	5713	+ return SEQ_SKIP;
	5714	+
	5715	+ seq_printf(m, "%d %d\n", pid, tgid);
5654	5716	return 0;
5655	5717	}
5656	5718
@@ -6135,7 +6197,7 @@ static int __tracing_resize_ring_buffer(struct trace_array *tr, · 478 unmodified lines
6135	6197	ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
6136	6198	unsigned long size, int cpu_id)
6137	6199	{
6138		− int ret = size;
	6200	+ int ret;
6139	6201
6140	6202	mutex_lock(&trace_types_lock);
6141	6203
@@ -7529,6 +7591,91 @@ static const struct file_operations snapshot_raw_fops = { · 1387 unmodified lines
7529	7591
7530	7592	#endif /* CONFIG_TRACER_SNAPSHOT */
7531	7593
	7594	+/*
	7595	+ * trace_min_max_write - Write a u64 value to a trace_min_max_param struct
	7596	+ * @filp: The active open file structure
	7597	+ * @ubuf: The userspace provided buffer to read value into
	7598	+ * @cnt: The maximum number of bytes to read
	7599	+ * @ppos: The current "file" position
	7600	+ *
	7601	+ * This function implements the write interface for a struct trace_min_max_param.
	7602	+ * The filp->private_data must point to a trace_min_max_param structure that
	7603	+ * defines where to write the value, the min and the max acceptable values,
	7604	+ * and a lock to protect the write.
	7605	+ */
	7606	+static ssize_t
	7607	+trace_min_max_write(struct file filp, const char __user ubuf, size_t cnt, loff_t *ppos)
	7608	+{
	7609	+ struct trace_min_max_param *param = filp->private_data;
	7610	+ u64 val;
	7611	+ int err;
	7612	+
	7613	+ if (!param)
	7614	+ return -EFAULT;
	7615	+
	7616	+ err = kstrtoull_from_user(ubuf, cnt, 10, &val);
	7617	+ if (err)
	7618	+ return err;
	7619	+
	7620	+ if (param->lock)
	7621	+ mutex_lock(param->lock);
	7622	+
	7623	+ if (param->min && val < *param->min)
	7624	+ err = -EINVAL;
	7625	+
	7626	+ if (param->max && val > *param->max)
	7627	+ err = -EINVAL;
	7628	+
	7629	+ if (!err)
	7630	+ *param->val = val;
	7631	+
	7632	+ if (param->lock)
	7633	+ mutex_unlock(param->lock);
	7634	+
	7635	+ if (err)
	7636	+ return err;
	7637	+
	7638	+ return cnt;
	7639	+}
	7640	+
	7641	+/*
	7642	+ * trace_min_max_read - Read a u64 value from a trace_min_max_param struct
	7643	+ * @filp: The active open file structure
	7644	+ * @ubuf: The userspace provided buffer to read value into
	7645	+ * @cnt: The maximum number of bytes to read
	7646	+ * @ppos: The current "file" position
	7647	+ *
	7648	+ * This function implements the read interface for a struct trace_min_max_param.
	7649	+ * The filp->private_data must point to a trace_min_max_param struct with valid
	7650	+ * data.
	7651	+ */
	7652	+static ssize_t
	7653	+trace_min_max_read(struct file filp, char __user ubuf, size_t cnt, loff_t *ppos)
	7654	+{
	7655	+ struct trace_min_max_param *param = filp->private_data;
	7656	+ char buf[U64_STR_SIZE];
	7657	+ int len;
	7658	+ u64 val;
	7659	+
	7660	+ if (!param)
	7661	+ return -EFAULT;
	7662	+
	7663	+ val = *param->val;
	7664	+
	7665	+ if (cnt > sizeof(buf))
	7666	+ cnt = sizeof(buf);
	7667	+
	7668	+ len = snprintf(buf, sizeof(buf), "%llu\n", val);
	7669	+
	7670	+ return simple_read_from_buffer(ubuf, cnt, ppos, buf, len);
	7671	+}
	7672	+
	7673	+const struct file_operations trace_min_max_fops = {
	7674	+ .open = tracing_open_generic,
	7675	+ .read = trace_min_max_read,
	7676	+ .write = trace_min_max_write,
	7677	+};
	7678	+
7532	7679	#define TRACING_LOG_ERRS_MAX 8
7533	7680	#define TRACING_LOG_LOC_MAX 128
7534	7681
@@ -9532,6 +9679,8 @@ static __init int tracer_init_tracefs(void) · 1997 unmodified lines
9532	9679	return 0;
9533	9680	}
9534	9681
	9682	+fs_initcall(tracer_init_tracefs);
	9683	+
9535	9684	static int trace_panic_handler(struct notifier_block *this,
9536	9685	unsigned long event, void *unused)
9537	9686	{
@@ -9952,7 +10101,7 @@ void __init trace_init(void) · 414 unmodified lines
9952	10101	trace_event_init();
9953	10102	}
9954	10103
9955		−__init static int clear_boot_tracer(void)
	10104	+__init static void clear_boot_tracer(void)
9956	10105	{
9957	10106	/*
9958	10107	* The default tracer at boot buffer is an init section.
@@ -9962,26 +10111,21 @@ __init static int clear_boot_tracer(void) · 3 unmodified lines
9962	10111	* about to be freed.
9963	10112	*/
9964	10113	if (!default_bootup_tracer)
9965		− return 0;
	10114	+ return;
9966	10115
9967	10116	printk(KERN_INFO "ftrace bootup tracer '%s' not registered.\n",
9968	10117	default_bootup_tracer);
9969	10118	default_bootup_tracer = NULL;
9970		−
9971		− return 0;
9972	10119	}
9973	10120
9974		−fs_initcall(tracer_init_tracefs);
9975		−late_initcall_sync(clear_boot_tracer);
9976		−
9977	10121	#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK
9978		−__init static int tracing_set_default_clock(void)
	10122	+__init static void tracing_set_default_clock(void)
9979	10123	{
9980	10124	/* sched_clock_stable() is determined in late_initcall */
9981	10125	if (!trace_boot_clock && !sched_clock_stable()) {
9982	10126	if (security_locked_down(LOCKDOWN_TRACEFS)) {
9983	10127	pr_warn("Can not set tracing clock due to lockdown\n");
9984		− return -EPERM;
	10128	+ return;
9985	10129	}
9986	10130
9987	10131	printk(KERN_WARNING
@@ -9991,8 +10135,21 @@ __init static int tracing_set_default_clock(void) · 3 unmodified lines
9991	10135	"on the kernel command line\n");
9992	10136	tracing_set_clock(&global_trace, "global");
9993	10137	}
	10138	+}
	10139	+#else
	10140	+static inline void tracing_set_default_clock(void) { }
	10141	+#endif
	10142	+
	10143	+__init static int late_trace_init(void)
	10144	+{
	10145	+ if (tracepoint_printk && tracepoint_printk_stop_on_boot) {
	10146	+ static_key_disable(&tracepoint_printk_key.key);
	10147	+ tracepoint_printk = 0;
	10148	+ }
9994	10149
	10150	+ tracing_set_default_clock();
	10151	+ clear_boot_tracer();
9995	10152	return 0;
9996	10153	}
9997		−late_initcall_sync(tracing_set_default_clock);
9998		−#endif
	10154	+
	10155	+late_initcall_sync(late_trace_init);

kernel/trace/trace.h+29−6

@@ -45,6 +45,8 @@ enum trace_type { · 44 unmodified lines
45	45	TRACE_BLK,
46	46	TRACE_BPUTS,
47	47	TRACE_HWLAT,
	48	+ TRACE_OSNOISE,
	49	+ TRACE_TIMERLAT,
48	50	TRACE_RAW_DATA,
49	51	TRACE_FUNC_REPEATS,
50	52
@@ -290,7 +292,8 @@ struct trace_array { · 239 unmodified lines
290	292	struct array_buffer max_buffer;
291	293	bool allocated_snapshot;
292	294	#endif
293		−#if defined(CONFIG_TRACER_MAX_TRACE) \|\| defined(CONFIG_HWLAT_TRACER)
	295	+#if defined(CONFIG_TRACER_MAX_TRACE) \|\| defined(CONFIG_HWLAT_TRACER) \
	296	+ \|\| defined(CONFIG_OSNOISE_TRACER)
294	297	unsigned long max_latency;
295	298	#ifdef CONFIG_FSNOTIFY
296	299	struct dentry *d_max_latency;
@@ -438,6 +441,8 @@ extern void __ftrace_bad_type(void); · 141 unmodified lines
438	441	IF_ASSIGN(var, ent, struct bprint_entry, TRACE_BPRINT); \
439	442	IF_ASSIGN(var, ent, struct bputs_entry, TRACE_BPUTS); \
440	443	IF_ASSIGN(var, ent, struct hwlat_entry, TRACE_HWLAT); \
	444	+ IF_ASSIGN(var, ent, struct osnoise_entry, TRACE_OSNOISE);\
	445	+ IF_ASSIGN(var, ent, struct timerlat_entry, TRACE_TIMERLAT);\
441	446	IF_ASSIGN(var, ent, struct raw_data_entry, TRACE_RAW_DATA);\
442	447	IF_ASSIGN(var, ent, struct trace_mmiotrace_rw, \
443	448	TRACE_MMIO_RW); \
@@ -668,15 +673,15 @@ void update_max_tr_single(struct trace_array *tr, · 224 unmodified lines
668	673	struct task_struct *tsk, int cpu);
669	674	#endif /* CONFIG_TRACER_MAX_TRACE */
670	675
671		−#if (defined(CONFIG_TRACER_MAX_TRACE) \|\| defined(CONFIG_HWLAT_TRACER)) && \
672		− defined(CONFIG_FSNOTIFY)
	676	+#if (defined(CONFIG_TRACER_MAX_TRACE) \|\| defined(CONFIG_HWLAT_TRACER) \
	677	+ \|\| defined(CONFIG_OSNOISE_TRACER)) && defined(CONFIG_FSNOTIFY)
	678	+#define LATENCY_FS_NOTIFY
	679	+#endif
673	680
	681	+#ifdef LATENCY_FS_NOTIFY
674	682	void latency_fsnotify(struct trace_array *tr);
675		−
676	683	#else
677		−
678	684	static inline void latency_fsnotify(struct trace_array *tr) { }
679		−
680	685	#endif
681	686
682	687	#ifdef CONFIG_STACKTRACE
@@ -1945,4 +1950,22 @@ static inline bool is_good_name(const char *name) · 1262 unmodified lines
1945	1950	return true;
1946	1951	}
1947	1952
	1953	+/*
	1954	+ * This is a generic way to read and write a u64 value from a file in tracefs.
	1955	+ *
	1956	+ * The value is stored on the variable pointed by *val. The value needs
	1957	+ * to be at least min and at most max. The write is protected by an
	1958	+ * existing *lock.
	1959	+ */
	1960	+struct trace_min_max_param {
	1961	+ struct mutex *lock;
	1962	+ u64 *val;
	1963	+ u64 *min;
	1964	+ u64 *max;
	1965	+};
	1966	+
	1967	+#define U64_STR_SIZE 24 /* 20 digits max */
	1968	+
	1969	+extern const struct file_operations trace_min_max_fops;
	1970	+
1948	1971	#endif /* _LINUX_KERNEL_TRACE_H */

kernel/trace/trace_boot.c+25−2

@@ -225,14 +225,37 @@ static void __init · 224 unmodified lines
225	225	trace_boot_init_events(struct trace_array tr, struct xbc_node node)
226	226	{
227	227	struct xbc_node gnode, enode;
	228	+ bool enable, enable_all = false;
	229	+ const char *data;
228	230
229	231	node = xbc_node_find_child(node, "event");
230	232	if (!node)
231	233	return;
232	234	/* per-event key starts with "event.GROUP.EVENT" */
233		− xbc_node_for_each_child(node, gnode)
234		− xbc_node_for_each_child(gnode, enode)
	235	+ xbc_node_for_each_child(node, gnode) {
	236	+ data = xbc_node_get_data(gnode);
	237	+ if (!strcmp(data, "enable")) {
	238	+ enable_all = true;
	239	+ continue;
	240	+ }
	241	+ enable = false;
	242	+ xbc_node_for_each_child(gnode, enode) {
	243	+ data = xbc_node_get_data(enode);
	244	+ if (!strcmp(data, "enable")) {
	245	+ enable = true;
	246	+ continue;
	247	+ }
235	248	trace_boot_init_one_event(tr, gnode, enode);
	249	+ }
	250	+ /* Event enablement must be done after event settings */
	251	+ if (enable) {
	252	+ data = xbc_node_get_data(gnode);
	253	+ trace_array_set_clr_event(tr, data, NULL, true);
	254	+ }
	255	+ }
	256	+ /* Ditto */
	257	+ if (enable_all)
	258	+ trace_array_set_clr_event(tr, NULL, NULL, true);
236	259	}
237	260	#else
238	261	#define trace_boot_enable_events(tr, node) do {} while (0)

kernel/trace/trace_entries.h+41−0

@@ -360,3 +360,44 @@ FTRACE_ENTRY(func_repeats, func_repeats_entry, · 359 unmodified lines
360	360	__entry->count,
361	361	FUNC_REPEATS_GET_DELTA_TS(__entry))
362	362	);
	363	+
	364	+FTRACE_ENTRY(osnoise, osnoise_entry,
	365	+
	366	+ TRACE_OSNOISE,
	367	+
	368	+ F_STRUCT(
	369	+ __field( u64, noise )
	370	+ __field( u64, runtime )
	371	+ __field( u64, max_sample )
	372	+ __field( unsigned int, hw_count )
	373	+ __field( unsigned int, nmi_count )
	374	+ __field( unsigned int, irq_count )
	375	+ __field( unsigned int, softirq_count )
	376	+ __field( unsigned int, thread_count )
	377	+ ),
	378	+
	379	+ F_printk("noise:%llu\tmax_sample:%llu\thw:%u\tnmi:%u\tirq:%u\tsoftirq:%u\tthread:%u\n",
	380	+ __entry->noise,
	381	+ __entry->max_sample,
	382	+ __entry->hw_count,
	383	+ __entry->nmi_count,
	384	+ __entry->irq_count,
	385	+ __entry->softirq_count,
	386	+ __entry->thread_count)
	387	+);
	388	+
	389	+FTRACE_ENTRY(timerlat, timerlat_entry,
	390	+
	391	+ TRACE_TIMERLAT,
	392	+
	393	+ F_STRUCT(
	394	+ __field( unsigned int, seqnum )
	395	+ __field( int, context )
	396	+ __field( u64, timer_latency )
	397	+ ),
	398	+
	399	+ F_printk("seq:%u\tcontext:%d\ttimer_latency:%llu\n",
	400	+ __entry->seqnum,
	401	+ __entry->context,
	402	+ __entry->timer_latency)
	403	+);

kernel/trace/trace_events_hist.c+2−1

@@ -2434,12 +2434,12 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data, · 2433 unmodified lines
2434	2434	char subsys_name, char event_name, char *field_name)
2435	2435	{
2436	2436	struct trace_array *tr = target_hist_data->event_file->tr;
2437		− struct hist_field *event_var = ERR_PTR(-EINVAL);
2438	2437	struct hist_trigger_data *hist_data;
2439	2438	unsigned int i, n, first = true;
2440	2439	struct field_var_hist *var_hist;
2441	2440	struct trace_event_file *file;
2442	2441	struct hist_field *key_field;
	2442	+ struct hist_field *event_var;
2443	2443	char *saved_filter;
2444	2444	char *cmd;
2445	2445	int ret;
@@ -5232,6 +5232,7 @@ static void unregister_field_var_hists(struct hist_trigger_data *hist_data) · 2786 unmodified lines
5232	5232	cmd = hist_data->field_var_hists[i]->cmd;
5233	5233	ret = event_hist_trigger_func(&trigger_hist_cmd, file,
5234	5234	"!hist", "hist", cmd);
	5235	+ WARN_ON_ONCE(ret < 0);
5235	5236	}
5236	5237	}
5237	5238

kernel/trace/trace_events_trigger.c+2−1

@@ -916,7 +916,8 @@ void unpause_named_trigger(struct event_trigger_data *data) · 915 unmodified lines
916	916
917	917	/**
918	918	* set_named_trigger_data - Associate common named trigger data
919		− * @data: The trigger data of a named trigger to unpause
	919	+ * @data: The trigger data to associate
	920	+ * @named_data: The common named trigger to be associated
920	921	*
921	922	* Named triggers are sets of triggers that share a common set of
922	923	* trigger data. The first named trigger registered with a given name

kernel/trace/trace_hwlat.c+392−142

@@ -34,7 +34,7 @@ · 33 unmodified lines
34	34	* Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com>
35	35	* Copyright (C) 2013-2016 Steven Rostedt, Red Hat, Inc. <srostedt@redhat.com>
36	36	*
37		− * Includes useful feedback from Clark Williams <clark@redhat.com>
	37	+ * Includes useful feedback from Clark Williams <williams@redhat.com>
38	38	*
39	39	*/
40	40	#include <linux/kthread.h>
@@ -54,20 +54,33 @@ static struct trace_array *hwlat_trace; · 13 unmodified lines
54	54	#define DEFAULT_SAMPLE_WIDTH 500000 /* 0.5s */
55	55	#define DEFAULT_LAT_THRESHOLD 10 /* 10us */
56	56
57		−/* sampling thread*/
58		−static struct task_struct *hwlat_kthread;
59		−
60	57	static struct dentry hwlat_sample_width; / sample width us */
61	58	static struct dentry hwlat_sample_window; / sample window us */
	59	+static struct dentry hwlat_thread_mode; / hwlat thread mode */
	60	+
	61	+enum {
	62	+ MODE_NONE = 0,
	63	+ MODE_ROUND_ROBIN,
	64	+ MODE_PER_CPU,
	65	+ MODE_MAX
	66	+};
	67	+static char *thread_mode_str[] = { "none", "round-robin", "per-cpu" };
62	68
63	69	/* Save the previous tracing_thresh value */
64	70	static unsigned long save_tracing_thresh;
65	71
66		−/* NMI timestamp counters */
67		−static u64 nmi_ts_start;
68		−static u64 nmi_total_ts;
69		−static int nmi_count;
70		−static int nmi_cpu;
	72	+/* runtime kthread data */
	73	+struct hwlat_kthread_data {
	74	+ struct task_struct *kthread;
	75	+ /* NMI timestamp counters */
	76	+ u64 nmi_ts_start;
	77	+ u64 nmi_total_ts;
	78	+ int nmi_count;
	79	+ int nmi_cpu;
	80	+};
	81	+
	82	+struct hwlat_kthread_data hwlat_single_cpu_data;
	83	+DEFINE_PER_CPU(struct hwlat_kthread_data, hwlat_per_cpu_data);
71	84
72	85	/* Tells NMIs to call back to the hwlat tracer to record timestamps */
73	86	bool trace_hwlat_callback_enabled;
@@ -96,11 +109,24 @@ static struct hwlat_data { · 22 unmodified lines
96	109	u64 sample_window; /* total sampling window (on+off) */
97	110	u64 sample_width; /* active sampling portion of window */
98	111
	112	+ int thread_mode; /* thread mode */
	113	+
99	114	} hwlat_data = {
100	115	.sample_window = DEFAULT_SAMPLE_WINDOW,
101	116	.sample_width = DEFAULT_SAMPLE_WIDTH,
	117	+ .thread_mode = MODE_ROUND_ROBIN
102	118	};
103	119
	120	+static struct hwlat_kthread_data *get_cpu_data(void)
	121	+{
	122	+ if (hwlat_data.thread_mode == MODE_PER_CPU)
	123	+ return this_cpu_ptr(&hwlat_per_cpu_data);
	124	+ else
	125	+ return &hwlat_single_cpu_data;
	126	+}
	127	+
	128	+static bool hwlat_busy;
	129	+
104	130	static void trace_hwlat_sample(struct hwlat_sample *sample)
105	131	{
106	132	struct trace_array *tr = hwlat_trace;
@@ -136,7 +162,9 @@ static void trace_hwlat_sample(struct hwlat_sample *sample) · 29 unmodified lines
136	162
137	163	void trace_hwlat_callback(bool enter)
138	164	{
139		− if (smp_processor_id() != nmi_cpu)
	165	+ struct hwlat_kthread_data *kdata = get_cpu_data();
	166	+
	167	+ if (!kdata->kthread)
140	168	return;
141	169
142	170	/*
@@ -145,15 +173,24 @@ void trace_hwlat_callback(bool enter) · 2 unmodified lines
145	173	*/
146	174	if (!IS_ENABLED(CONFIG_GENERIC_SCHED_CLOCK)) {
147	175	if (enter)
148		− nmi_ts_start = time_get();
	176	+ kdata->nmi_ts_start = time_get();
149	177	else
150		− nmi_total_ts += time_get() - nmi_ts_start;
	178	+ kdata->nmi_total_ts += time_get() - kdata->nmi_ts_start;
151	179	}
152	180
153	181	if (enter)
154		− nmi_count++;
	182	+ kdata->nmi_count++;
155	183	}
156	184
	185	+/*
	186	+ * hwlat_err - report a hwlat error.
	187	+ */
	188	+#define hwlat_err(msg) ({ \
	189	+ struct trace_array *tr = hwlat_trace; \
	190	+ \
	191	+ trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \
	192	+})
	193	+
157	194	/**
158	195	* get_sample - sample the CPU TSC and look for likely hardware latencies
159	196	*
@@ -163,6 +200,7 @@ void trace_hwlat_callback(bool enter) · 3 unmodified lines
163	200	*/
164	201	static int get_sample(void)
165	202	{
	203	+ struct hwlat_kthread_data *kdata = get_cpu_data();
166	204	struct trace_array *tr = hwlat_trace;
167	205	struct hwlat_sample s;
168	206	time_type start, t1, t2, last_t2;
@@ -175,9 +213,8 @@ static int get_sample(void) · 6 unmodified lines
175	213
176	214	do_div(thresh, NSEC_PER_USEC); /* modifies interval value */
177	215
178		− nmi_cpu = smp_processor_id();
179		− nmi_total_ts = 0;
180		− nmi_count = 0;
	216	+ kdata->nmi_total_ts = 0;
	217	+ kdata->nmi_count = 0;
181	218	/* Make sure NMIs see this first */
182	219	barrier();
183	220
@@ -197,7 +234,7 @@ static int get_sample(void) · 13 unmodified lines
197	234	outer_diff = time_to_us(time_sub(t1, last_t2));
198	235	/* This shouldn't happen */
199	236	if (outer_diff < 0) {
200		− pr_err(BANNER "time running backwards\n");
	237	+ hwlat_err(BANNER "time running backwards\n");
201	238	goto out;
202	239	}
203	240	if (outer_diff > outer_sample)
@@ -209,7 +246,7 @@ static int get_sample(void) · 5 unmodified lines
209	246
210	247	/* Check for possible overflows */
211	248	if (total < last_total) {
212		− pr_err("Time total overflowed\n");
	249	+ hwlat_err("Time total overflowed\n");
213	250	break;
214	251	}
215	252	last_total = total;
@@ -225,7 +262,7 @@ static int get_sample(void) · 9 unmodified lines
225	262
226	263	/* This shouldn't happen */
227	264	if (diff < 0) {
228		− pr_err(BANNER "time running backwards\n");
	265	+ hwlat_err(BANNER "time running backwards\n");
229	266	goto out;
230	267	}
231	268
@@ -247,15 +284,15 @@ static int get_sample(void) · 15 unmodified lines
247	284	ret = 1;
248	285
249	286	/* We read in microseconds */
250		− if (nmi_total_ts)
251		− do_div(nmi_total_ts, NSEC_PER_USEC);
	287	+ if (kdata->nmi_total_ts)
	288	+ do_div(kdata->nmi_total_ts, NSEC_PER_USEC);
252	289
253	290	hwlat_data.count++;
254	291	s.seqnum = hwlat_data.count;
255	292	s.duration = sample;
256	293	s.outer_duration = outer_sample;
257		− s.nmi_total_ts = nmi_total_ts;
258		− s.nmi_count = nmi_count;
	294	+ s.nmi_total_ts = kdata->nmi_total_ts;
	295	+ s.nmi_count = kdata->nmi_count;
259	296	s.count = count;
260	297	trace_hwlat_sample(&s);
261	298
@@ -273,7 +310,6 @@ out: · 11 unmodified lines
273	310	}
274	311
275	312	static struct cpumask save_cpumask;
276		−static bool disable_migrate;
277	313
278	314	static void move_to_next_cpu(void)
279	315	{
@@ -281,15 +317,13 @@ static void move_to_next_cpu(void) · 1 unmodified line
281	317	struct trace_array *tr = hwlat_trace;
282	318	int next_cpu;
283	319
284		− if (disable_migrate)
285		− return;
286	320	/*
287	321	* If for some reason the user modifies the CPU affinity
288	322	* of this thread, then stop migrating for the duration
289	323	* of the current test.
290	324	*/
291	325	if (!cpumask_equal(current_mask, current->cpus_ptr))
292		− goto disable;
	326	+ goto change_mode;
293	327
294	328	get_online_cpus();
295	329	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
@@ -300,7 +334,7 @@ static void move_to_next_cpu(void) · 4 unmodified lines
300	334	next_cpu = cpumask_first(current_mask);
301	335
302	336	if (next_cpu >= nr_cpu_ids) /* Shouldn't happen! */
303		− goto disable;
	337	+ goto change_mode;
304	338
305	339	cpumask_clear(current_mask);
306	340	cpumask_set_cpu(next_cpu, current_mask);
@@ -308,8 +342,9 @@ static void move_to_next_cpu(void) · 1 unmodified line
308	342	sched_setaffinity(0, current_mask);
309	343	return;
310	344
311		− disable:
312		− disable_migrate = true;
	345	+ change_mode:
	346	+ hwlat_data.thread_mode = MODE_NONE;
	347	+ pr_info(BANNER "cpumask changed while in round-robin mode, switching to mode none\n");
313	348	}
314	349
315	350	/*
@@ -328,7 +363,8 @@ static int kthread_fn(void *data) · 12 unmodified lines
328	363
329	364	while (!kthread_should_stop()) {
330	365
331		− move_to_next_cpu();
	366	+ if (hwlat_data.thread_mode == MODE_ROUND_ROBIN)
	367	+ move_to_next_cpu();
332	368
333	369	local_irq_disable();
334	370	get_sample();
@@ -351,178 +387,380 @@ static int kthread_fn(void *data) · 16 unmodified lines
351	387	return 0;
352	388	}
353	389
354		−/**
355		− * start_kthread - Kick off the hardware latency sampling/detector kthread
	390	+/*
	391	+ * stop_stop_kthread - Inform the hardware latency sampling/detector kthread to stop
	392	+ *
	393	+ * This kicks the running hardware latency sampling/detector kernel thread and
	394	+ * tells it to stop sampling now. Use this on unload and at system shutdown.
	395	+ */
	396	+static void stop_single_kthread(void)
	397	+{
	398	+ struct hwlat_kthread_data *kdata = get_cpu_data();
	399	+ struct task_struct *kthread;
	400	+
	401	+ get_online_cpus();
	402	+ kthread = kdata->kthread;
	403	+
	404	+ if (!kthread)
	405	+ goto out_put_cpus;
	406	+
	407	+ kthread_stop(kthread);
	408	+ kdata->kthread = NULL;
	409	+
	410	+out_put_cpus:
	411	+ put_online_cpus();
	412	+}
	413	+
	414	+
	415	+/*
	416	+ * start_single_kthread - Kick off the hardware latency sampling/detector kthread
356	417	*
357	418	* This starts the kernel thread that will sit and sample the CPU timestamp
358	419	* counter (TSC or similar) and look for potential hardware latencies.
359	420	*/
360		−static int start_kthread(struct trace_array *tr)
	421	+static int start_single_kthread(struct trace_array *tr)
361	422	{
	423	+ struct hwlat_kthread_data *kdata = get_cpu_data();
362	424	struct cpumask *current_mask = &save_cpumask;
363	425	struct task_struct *kthread;
364	426	int next_cpu;
365	427
366		− if (hwlat_kthread)
367		− return 0;
368		−
369		− /* Just pick the first CPU on first iteration */
370	428	get_online_cpus();
371		− cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
372		− put_online_cpus();
373		− next_cpu = cpumask_first(current_mask);
	429	+ if (kdata->kthread)
	430	+ goto out_put_cpus;
374	431
375	432	kthread = kthread_create(kthread_fn, NULL, "hwlatd");
376	433	if (IS_ERR(kthread)) {
377	434	pr_err(BANNER "could not start sampling thread\n");
	435	+ put_online_cpus();
378	436	return -ENOMEM;
379	437	}
380	438
381		− cpumask_clear(current_mask);
382		− cpumask_set_cpu(next_cpu, current_mask);
	439	+ /* Just pick the first CPU on first iteration */
	440	+ cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
	441	+
	442	+ if (hwlat_data.thread_mode == MODE_ROUND_ROBIN) {
	443	+ next_cpu = cpumask_first(current_mask);
	444	+ cpumask_clear(current_mask);
	445	+ cpumask_set_cpu(next_cpu, current_mask);
	446	+
	447	+ }
	448	+
383	449	sched_setaffinity(kthread->pid, current_mask);
384	450
385		− hwlat_kthread = kthread;
	451	+ kdata->kthread = kthread;
386	452	wake_up_process(kthread);
387	453
	454	+out_put_cpus:
	455	+ put_online_cpus();
388	456	return 0;
389	457	}
390	458
391		−/**
392		− * stop_kthread - Inform the hardware latency sampling/detector kthread to stop
	459	+/*
	460	+ * stop_cpu_kthread - Stop a hwlat cpu kthread
	461	+ */
	462	+static void stop_cpu_kthread(unsigned int cpu)
	463	+{
	464	+ struct task_struct *kthread;
	465	+
	466	+ kthread = per_cpu(hwlat_per_cpu_data, cpu).kthread;
	467	+ if (kthread)
	468	+ kthread_stop(kthread);
	469	+ per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
	470	+}
	471	+
	472	+/*
	473	+ * stop_per_cpu_kthreads - Inform the hardware latency sampling/detector kthread to stop
393	474	*
394		− * This kicks the running hardware latency sampling/detector kernel thread and
	475	+ * This kicks the running hardware latency sampling/detector kernel threads and
395	476	* tells it to stop sampling now. Use this on unload and at system shutdown.
396	477	*/
397		−static void stop_kthread(void)
	478	+static void stop_per_cpu_kthreads(void)
398	479	{
399		− if (!hwlat_kthread)
400		− return;
401		− kthread_stop(hwlat_kthread);
402		− hwlat_kthread = NULL;
	480	+ unsigned int cpu;
	481	+
	482	+ get_online_cpus();
	483	+ for_each_online_cpu(cpu)
	484	+ stop_cpu_kthread(cpu);
	485	+ put_online_cpus();
403	486	}
404	487
405	488	/*
406		− * hwlat_read - Wrapper read function for reading both window and width
407		− * @filp: The active open file structure
408		− * @ubuf: The userspace provided buffer to read value into
409		− * @cnt: The maximum number of bytes to read
410		− * @ppos: The current "file" position
411		− *
412		− * This function provides a generic read implementation for the global state
413		− * "hwlat_data" structure filesystem entries.
	489	+ * start_cpu_kthread - Start a hwlat cpu kthread
414	490	*/
415		−static ssize_t hwlat_read(struct file filp, char __user ubuf,
416		− size_t cnt, loff_t *ppos)
	491	+static int start_cpu_kthread(unsigned int cpu)
417	492	{
418		− char buf[U64STR_SIZE];
419		− u64 *entry = filp->private_data;
420		− u64 val;
421		− int len;
	493	+ struct task_struct *kthread;
	494	+ char comm[24];
422	495
423		− if (!entry)
424		− return -EFAULT;
	496	+ snprintf(comm, 24, "hwlatd/%d", cpu);
	497	+
	498	+ kthread = kthread_create_on_cpu(kthread_fn, NULL, cpu, comm);
	499	+ if (IS_ERR(kthread)) {
	500	+ pr_err(BANNER "could not start sampling thread\n");
	501	+ return -ENOMEM;
	502	+ }
425	503
426		− if (cnt > sizeof(buf))
427		− cnt = sizeof(buf);
	504	+ per_cpu(hwlat_per_cpu_data, cpu).kthread = kthread;
	505	+ wake_up_process(kthread);
	506	+
	507	+ return 0;
	508	+}
	509	+
	510	+#ifdef CONFIG_HOTPLUG_CPU
	511	+static void hwlat_hotplug_workfn(struct work_struct *dummy)
	512	+{
	513	+ struct trace_array *tr = hwlat_trace;
	514	+ unsigned int cpu = smp_processor_id();
	515	+
	516	+ mutex_lock(&trace_types_lock);
	517	+ mutex_lock(&hwlat_data.lock);
	518	+ get_online_cpus();
	519	+
	520	+ if (!hwlat_busy \|\| hwlat_data.thread_mode != MODE_PER_CPU)
	521	+ goto out_unlock;
428	522
429		− val = *entry;
	523	+ if (!cpumask_test_cpu(cpu, tr->tracing_cpumask))
	524	+ goto out_unlock;
430	525
431		− len = snprintf(buf, sizeof(buf), "%llu\n", val);
	526	+ start_cpu_kthread(cpu);
432	527
433		− return simple_read_from_buffer(ubuf, cnt, ppos, buf, len);
	528	+out_unlock:
	529	+ put_online_cpus();
	530	+ mutex_unlock(&hwlat_data.lock);
	531	+ mutex_unlock(&trace_types_lock);
434	532	}
435	533
436		−/**
437		− * hwlat_width_write - Write function for "width" entry
438		− * @filp: The active open file structure
439		− * @ubuf: The user buffer that contains the value to write
440		− * @cnt: The maximum number of bytes to write to "file"
441		− * @ppos: The current position in @file
	534	+static DECLARE_WORK(hwlat_hotplug_work, hwlat_hotplug_workfn);
	535	+
	536	+/*
	537	+ * hwlat_cpu_init - CPU hotplug online callback function
	538	+ */
	539	+static int hwlat_cpu_init(unsigned int cpu)
	540	+{
	541	+ schedule_work_on(cpu, &hwlat_hotplug_work);
	542	+ return 0;
	543	+}
	544	+
	545	+/*
	546	+ * hwlat_cpu_die - CPU hotplug offline callback function
	547	+ */
	548	+static int hwlat_cpu_die(unsigned int cpu)
	549	+{
	550	+ stop_cpu_kthread(cpu);
	551	+ return 0;
	552	+}
	553	+
	554	+static void hwlat_init_hotplug_support(void)
	555	+{
	556	+ int ret;
	557	+
	558	+ ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "trace/hwlat:online",
	559	+ hwlat_cpu_init, hwlat_cpu_die);
	560	+ if (ret < 0)
	561	+ pr_warn(BANNER "Error to init cpu hotplug support\n");
	562	+
	563	+ return;
	564	+}
	565	+#else /* CONFIG_HOTPLUG_CPU */
	566	+static void hwlat_init_hotplug_support(void)
	567	+{
	568	+ return;
	569	+}
	570	+#endif /* CONFIG_HOTPLUG_CPU */
	571	+
	572	+/*
	573	+ * start_per_cpu_kthreads - Kick off the hardware latency sampling/detector kthreads
442	574	*
443		− * This function provides a write implementation for the "width" interface
444		− * to the hardware latency detector. It can be used to configure
445		− * for how many us of the total window us we will actively sample for any
446		− * hardware-induced latency periods. Obviously, it is not possible to
447		− * sample constantly and have the system respond to a sample reader, or,
448		− * worse, without having the system appear to have gone out to lunch. It
449		− * is enforced that width is less that the total window size.
	575	+ * This starts the kernel threads that will sit on potentially all cpus and
	576	+ * sample the CPU timestamp counter (TSC or similar) and look for potential
	577	+ * hardware latencies.
450	578	*/
451		−static ssize_t
452		−hwlat_width_write(struct file filp, const char __user ubuf,
453		− size_t cnt, loff_t *ppos)
	579	+static int start_per_cpu_kthreads(struct trace_array *tr)
454	580	{
455		− u64 val;
456		− int err;
	581	+ struct cpumask *current_mask = &save_cpumask;
	582	+ unsigned int cpu;
	583	+ int retval;
457	584
458		− err = kstrtoull_from_user(ubuf, cnt, 10, &val);
459		− if (err)
460		− return err;
	585	+ get_online_cpus();
	586	+ /*
	587	+ * Run only on CPUs in which hwlat is allowed to run.
	588	+ */
	589	+ cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
	590	+
	591	+ for_each_online_cpu(cpu)
	592	+ per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
	593	+
	594	+ for_each_cpu(cpu, current_mask) {
	595	+ retval = start_cpu_kthread(cpu);
	596	+ if (retval)
	597	+ goto out_error;
	598	+ }
	599	+ put_online_cpus();
	600	+
	601	+ return 0;
	602	+
	603	+out_error:
	604	+ put_online_cpus();
	605	+ stop_per_cpu_kthreads();
	606	+ return retval;
	607	+}
	608	+
	609	+static void s_mode_start(struct seq_file s, loff_t *pos)
	610	+{
	611	+ int mode = *pos;
461	612
462	613	mutex_lock(&hwlat_data.lock);
463		− if (val < hwlat_data.sample_window)
464		− hwlat_data.sample_width = val;
	614	+
	615	+ if (mode >= MODE_MAX)
	616	+ return NULL;
	617	+
	618	+ return pos;
	619	+}
	620	+
	621	+static void s_mode_next(struct seq_file s, void v, loff_t pos)
	622	+{
	623	+ int mode = ++(*pos);
	624	+
	625	+ if (mode >= MODE_MAX)
	626	+ return NULL;
	627	+
	628	+ return pos;
	629	+}
	630	+
	631	+static int s_mode_show(struct seq_file s, void v)
	632	+{
	633	+ loff_t *pos = v;
	634	+ int mode = *pos;
	635	+
	636	+ if (mode == hwlat_data.thread_mode)
	637	+ seq_printf(s, "[%s]", thread_mode_str[mode]);
465	638	else
466		− err = -EINVAL;
467		− mutex_unlock(&hwlat_data.lock);
	639	+ seq_printf(s, "%s", thread_mode_str[mode]);
468	640
469		− if (err)
470		− return err;
	641	+ if (mode != MODE_MAX)
	642	+ seq_puts(s, " ");
	643	+
	644	+ return 0;
	645	+}
471	646
472		− return cnt;
	647	+static void s_mode_stop(struct seq_file s, void v)
	648	+{
	649	+ seq_puts(s, "\n");
	650	+ mutex_unlock(&hwlat_data.lock);
473	651	}
474	652
	653	+static const struct seq_operations thread_mode_seq_ops = {
	654	+ .start = s_mode_start,
	655	+ .next = s_mode_next,
	656	+ .show = s_mode_show,
	657	+ .stop = s_mode_stop
	658	+};
	659	+
	660	+static int hwlat_mode_open(struct inode inode, struct file file)
	661	+{
	662	+ return seq_open(file, &thread_mode_seq_ops);
	663	+};
	664	+
	665	+static void hwlat_tracer_start(struct trace_array *tr);
	666	+static void hwlat_tracer_stop(struct trace_array *tr);
	667	+
475	668	/**
476		− * hwlat_window_write - Write function for "window" entry
	669	+ * hwlat_mode_write - Write function for "mode" entry
477	670	* @filp: The active open file structure
478	671	* @ubuf: The user buffer that contains the value to write
479	672	* @cnt: The maximum number of bytes to write to "file"
480	673	* @ppos: The current position in @file
481	674	*
482		− * This function provides a write implementation for the "window" interface
483		− * to the hardware latency detector. The window is the total time
484		− * in us that will be considered one sample period. Conceptually, windows
485		− * occur back-to-back and contain a sample width period during which
486		− * actual sampling occurs. Can be used to write a new total window size. It
487		− * is enforced that any value written must be greater than the sample width
488		− * size, or an error results.
	675	+ * This function provides a write implementation for the "mode" interface
	676	+ * to the hardware latency detector. hwlatd has different operation modes.
	677	+ * The "none" sets the allowed cpumask for a single hwlatd thread at the
	678	+ * startup and lets the scheduler handle the migration. The default mode is
	679	+ * the "round-robin" one, in which a single hwlatd thread runs, migrating
	680	+ * among the allowed CPUs in a round-robin fashion. The "per-cpu" mode
	681	+ * creates one hwlatd thread per allowed CPU.
489	682	*/
490		−static ssize_t
491		−hwlat_window_write(struct file filp, const char __user ubuf,
492		− size_t cnt, loff_t *ppos)
	683	+static ssize_t hwlat_mode_write(struct file filp, const char __user ubuf,
	684	+ size_t cnt, loff_t *ppos)
493	685	{
494		− u64 val;
495		− int err;
	686	+ struct trace_array *tr = hwlat_trace;
	687	+ const char *mode;
	688	+ char buf[64];
	689	+ int ret, i;
496	690
497		− err = kstrtoull_from_user(ubuf, cnt, 10, &val);
498		− if (err)
499		− return err;
	691	+ if (cnt >= sizeof(buf))
	692	+ return -EINVAL;
	693	+
	694	+ if (copy_from_user(buf, ubuf, cnt))
	695	+ return -EFAULT;
	696	+
	697	+ buf[cnt] = 0;
	698	+
	699	+ mode = strstrip(buf);
	700	+
	701	+ ret = -EINVAL;
	702	+
	703	+ /*
	704	+ * trace_types_lock is taken to avoid concurrency on start/stop
	705	+ * and hwlat_busy.
	706	+ */
	707	+ mutex_lock(&trace_types_lock);
	708	+ if (hwlat_busy)
	709	+ hwlat_tracer_stop(tr);
500	710
501	711	mutex_lock(&hwlat_data.lock);
502		− if (hwlat_data.sample_width < val)
503		− hwlat_data.sample_window = val;
504		− else
505		− err = -EINVAL;
	712	+
	713	+ for (i = 0; i < MODE_MAX; i++) {
	714	+ if (strcmp(mode, thread_mode_str[i]) == 0) {
	715	+ hwlat_data.thread_mode = i;
	716	+ ret = cnt;
	717	+ }
	718	+ }
	719	+
506	720	mutex_unlock(&hwlat_data.lock);
507	721
508		− if (err)
509		− return err;
	722	+ if (hwlat_busy)
	723	+ hwlat_tracer_start(tr);
	724	+ mutex_unlock(&trace_types_lock);
	725	+
	726	+ *ppos += cnt;
510	727
511		− return cnt;
	728	+
	729	+
	730	+ return ret;
512	731	}
513	732
514		−static const struct file_operations width_fops = {
515		− .open = tracing_open_generic,
516		− .read = hwlat_read,
517		− .write = hwlat_width_write,
	733	+/*
	734	+ * The width parameter is read/write using the generic trace_min_max_param
	735	+ * method. The *val is protected by the hwlat_data lock and is upper
	736	+ * bounded by the window parameter.
	737	+ */
	738	+static struct trace_min_max_param hwlat_width = {
	739	+ .lock = &hwlat_data.lock,
	740	+ .val = &hwlat_data.sample_width,
	741	+ .max = &hwlat_data.sample_window,
	742	+ .min = NULL,
518	743	};
519	744
520		−static const struct file_operations window_fops = {
521		− .open = tracing_open_generic,
522		− .read = hwlat_read,
523		− .write = hwlat_window_write,
	745	+/*
	746	+ * The window parameter is read/write using the generic trace_min_max_param
	747	+ * method. The *val is protected by the hwlat_data lock and is lower
	748	+ * bounded by the width parameter.
	749	+ */
	750	+static struct trace_min_max_param hwlat_window = {
	751	+ .lock = &hwlat_data.lock,
	752	+ .val = &hwlat_data.sample_window,
	753	+ .max = NULL,
	754	+ .min = &hwlat_data.sample_width,
524	755	};
525	756
	757	+static const struct file_operations thread_mode_fops = {
	758	+ .open = hwlat_mode_open,
	759	+ .read = seq_read,
	760	+ .llseek = seq_lseek,
	761	+ .release = seq_release,
	762	+ .write = hwlat_mode_write
	763	+};
526	764	/**
527	765	* init_tracefs - A function to initialize the tracefs interface files
528	766	*
@@ -546,18 +784,25 @@ static int init_tracefs(void) · 17 unmodified lines
546	784
547	785	hwlat_sample_window = tracefs_create_file("window", 0640,
548	786	top_dir,
549		− &hwlat_data.sample_window,
550		− &window_fops);
	787	+ &hwlat_window,
	788	+ &trace_min_max_fops);
551	789	if (!hwlat_sample_window)
552	790	goto err;
553	791
554	792	hwlat_sample_width = tracefs_create_file("width", 0644,
555	793	top_dir,
556		− &hwlat_data.sample_width,
557		− &width_fops);
	794	+ &hwlat_width,
	795	+ &trace_min_max_fops);
558	796	if (!hwlat_sample_width)
559	797	goto err;
560	798
	799	+ hwlat_thread_mode = trace_create_file("mode", 0644,
	800	+ top_dir,
	801	+ NULL,
	802	+ &thread_mode_fops);
	803	+ if (!hwlat_thread_mode)
	804	+ goto err;
	805	+
561	806	return 0;
562	807
563	808	err:
@@ -569,18 +814,22 @@ static void hwlat_tracer_start(struct trace_array *tr) · 5 unmodified lines
569	814	{
570	815	int err;
571	816
572		− err = start_kthread(tr);
	817	+ if (hwlat_data.thread_mode == MODE_PER_CPU)
	818	+ err = start_per_cpu_kthreads(tr);
	819	+ else
	820	+ err = start_single_kthread(tr);
573	821	if (err)
574	822	pr_err(BANNER "Cannot start hwlat kthread\n");
575	823	}
576	824
577	825	static void hwlat_tracer_stop(struct trace_array *tr)
578	826	{
579		− stop_kthread();
	827	+ if (hwlat_data.thread_mode == MODE_PER_CPU)
	828	+ stop_per_cpu_kthreads();
	829	+ else
	830	+ stop_single_kthread();
580	831	}
581	832
582		−static bool hwlat_busy;
583		−
584	833	static int hwlat_tracer_init(struct trace_array *tr)
585	834	{
586	835	/* Only allow one instance to enable this */
@@ -589,7 +838,6 @@ static int hwlat_tracer_init(struct trace_array *tr) · 2 unmodified lines
589	838
590	839	hwlat_trace = tr;
591	840
592		− disable_migrate = false;
593	841	hwlat_data.count = 0;
594	842	tr->max_latency = 0;
595	843	save_tracing_thresh = tracing_thresh;
@@ -608,7 +856,7 @@ static int hwlat_tracer_init(struct trace_array *tr) · 12 unmodified lines
608	856
609	857	static void hwlat_tracer_reset(struct trace_array *tr)
610	858	{
611		− stop_kthread();
	859	+ hwlat_tracer_stop(tr);
612	860
613	861	/* the tracing threshold is static between runs */
614	862	last_tracing_thresh = tracing_thresh;
@@ -637,6 +885,8 @@ __init static int init_hwlat_tracer(void) · 22 unmodified lines
637	885	if (ret)
638	886	return ret;
639	887
	888	+ hwlat_init_hotplug_support();
	889	+
640	890	init_tracefs();
641	891
642	892	return 0;

kernel/trace/trace_osnoise.c+2059−0

@@ -0,0 +1,2059 @@
	1	+// SPDX-License-Identifier: GPL-2.0
	2	+/*
	3	+ * OS Noise Tracer: computes the OS Noise suffered by a running thread.
	4	+ * Timerlat Tracer: measures the wakeup latency of a timer triggered IRQ and thread.
	5	+ *
	6	+ * Based on "hwlat_detector" tracer by:
	7	+ * Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com>
	8	+ * Copyright (C) 2013-2016 Steven Rostedt, Red Hat, Inc. <srostedt@redhat.com>
	9	+ * With feedback from Clark Williams <williams@redhat.com>
	10	+ *
	11	+ * And also based on the rtsl tracer presented on:
	12	+ * DE OLIVEIRA, Daniel Bristot, et al. Demystifying the real-time linux
	13	+ * scheduling latency. In: 32nd Euromicro Conference on Real-Time Systems
	14	+ * (ECRTS 2020). Schloss Dagstuhl-Leibniz-Zentrum fur Informatik, 2020.
	15	+ *
	16	+ * Copyright (C) 2021 Daniel Bristot de Oliveira, Red Hat, Inc. <bristot@redhat.com>
	17	+ */
	18	+
	19	+#include <linux/kthread.h>
	20	+#include <linux/tracefs.h>
	21	+#include <linux/uaccess.h>
	22	+#include <linux/cpumask.h>
	23	+#include <linux/delay.h>
	24	+#include <linux/sched/clock.h>
	25	+#include <uapi/linux/sched/types.h>
	26	+#include <linux/sched.h>
	27	+#include "trace.h"
	28	+
	29	+#ifdef CONFIG_X86_LOCAL_APIC
	30	+#include <asm/trace/irq_vectors.h>
	31	+#undef TRACE_INCLUDE_PATH
	32	+#undef TRACE_INCLUDE_FILE
	33	+#endif /* CONFIG_X86_LOCAL_APIC */
	34	+
	35	+#include <trace/events/irq.h>
	36	+#include <trace/events/sched.h>
	37	+
	38	+#define CREATE_TRACE_POINTS
	39	+#include <trace/events/osnoise.h>
	40	+
	41	+static struct trace_array *osnoise_trace;
	42	+
	43	+/*
	44	+ * Default values.
	45	+ */
	46	+#define BANNER "osnoise: "
	47	+#define DEFAULT_SAMPLE_PERIOD 1000000 /* 1s */
	48	+#define DEFAULT_SAMPLE_RUNTIME 1000000 /* 1s */
	49	+
	50	+#define DEFAULT_TIMERLAT_PERIOD 1000 /* 1ms */
	51	+#define DEFAULT_TIMERLAT_PRIO 95 /* FIFO 95 */
	52	+
	53	+/*
	54	+ * NMI runtime info.
	55	+ */
	56	+struct osn_nmi {
	57	+ u64 count;
	58	+ u64 delta_start;
	59	+};
	60	+
	61	+/*
	62	+ * IRQ runtime info.
	63	+ */
	64	+struct osn_irq {
	65	+ u64 count;
	66	+ u64 arrival_time;
	67	+ u64 delta_start;
	68	+};
	69	+
	70	+#define IRQ_CONTEXT 0
	71	+#define THREAD_CONTEXT 1
	72	+/*
	73	+ * sofirq runtime info.
	74	+ */
	75	+struct osn_softirq {
	76	+ u64 count;
	77	+ u64 arrival_time;
	78	+ u64 delta_start;
	79	+};
	80	+
	81	+/*
	82	+ * thread runtime info.
	83	+ */
	84	+struct osn_thread {
	85	+ u64 count;
	86	+ u64 arrival_time;
	87	+ u64 delta_start;
	88	+};
	89	+
	90	+/*
	91	+ * Runtime information: this structure saves the runtime information used by
	92	+ * one sampling thread.
	93	+ */
	94	+struct osnoise_variables {
	95	+ struct task_struct *kthread;
	96	+ bool sampling;
	97	+ pid_t pid;
	98	+ struct osn_nmi nmi;
	99	+ struct osn_irq irq;
	100	+ struct osn_softirq softirq;
	101	+ struct osn_thread thread;
	102	+ local_t int_counter;
	103	+};
	104	+
	105	+/*
	106	+ * Per-cpu runtime information.
	107	+ */
	108	+DEFINE_PER_CPU(struct osnoise_variables, per_cpu_osnoise_var);
	109	+
	110	+/*
	111	+ * this_cpu_osn_var - Return the per-cpu osnoise_variables on its relative CPU
	112	+ */
	113	+static inline struct osnoise_variables *this_cpu_osn_var(void)
	114	+{
	115	+ return this_cpu_ptr(&per_cpu_osnoise_var);
	116	+}
	117	+
	118	+#ifdef CONFIG_TIMERLAT_TRACER
	119	+/*
	120	+ * Runtime information for the timer mode.
	121	+ */
	122	+struct timerlat_variables {
	123	+ struct task_struct *kthread;
	124	+ struct hrtimer timer;
	125	+ u64 rel_period;
	126	+ u64 abs_period;
	127	+ bool tracing_thread;
	128	+ u64 count;
	129	+};
	130	+
	131	+DEFINE_PER_CPU(struct timerlat_variables, per_cpu_timerlat_var);
	132	+
	133	+/*
	134	+ * this_cpu_tmr_var - Return the per-cpu timerlat_variables on its relative CPU
	135	+ */
	136	+static inline struct timerlat_variables *this_cpu_tmr_var(void)
	137	+{
	138	+ return this_cpu_ptr(&per_cpu_timerlat_var);
	139	+}
	140	+
	141	+/*
	142	+ * tlat_var_reset - Reset the values of the given timerlat_variables
	143	+ */
	144	+static inline void tlat_var_reset(void)
	145	+{
	146	+ struct timerlat_variables *tlat_var;
	147	+ int cpu;
	148	+ /*
	149	+ * So far, all the values are initialized as 0, so
	150	+ * zeroing the structure is perfect.
	151	+ */
	152	+ for_each_cpu(cpu, cpu_online_mask) {
	153	+ tlat_var = per_cpu_ptr(&per_cpu_timerlat_var, cpu);
	154	+ memset(tlat_var, 0, sizeof(*tlat_var));
	155	+ }
	156	+}
	157	+#else /* CONFIG_TIMERLAT_TRACER */
	158	+#define tlat_var_reset() do {} while (0)
	159	+#endif /* CONFIG_TIMERLAT_TRACER */
	160	+
	161	+/*
	162	+ * osn_var_reset - Reset the values of the given osnoise_variables
	163	+ */
	164	+static inline void osn_var_reset(void)
	165	+{
	166	+ struct osnoise_variables *osn_var;
	167	+ int cpu;
	168	+
	169	+ /*
	170	+ * So far, all the values are initialized as 0, so
	171	+ * zeroing the structure is perfect.
	172	+ */
	173	+ for_each_cpu(cpu, cpu_online_mask) {
	174	+ osn_var = per_cpu_ptr(&per_cpu_osnoise_var, cpu);
	175	+ memset(osn_var, 0, sizeof(*osn_var));
	176	+ }
	177	+}
	178	+
	179	+/*
	180	+ * osn_var_reset_all - Reset the value of all per-cpu osnoise_variables
	181	+ */
	182	+static inline void osn_var_reset_all(void)
	183	+{
	184	+ osn_var_reset();
	185	+ tlat_var_reset();
	186	+}
	187	+
	188	+/*
	189	+ * Tells NMIs to call back to the osnoise tracer to record timestamps.
	190	+ */
	191	+bool trace_osnoise_callback_enabled;
	192	+
	193	+/*
	194	+ * osnoise sample structure definition. Used to store the statistics of a
	195	+ * sample run.
	196	+ */
	197	+struct osnoise_sample {
	198	+ u64 runtime; /* runtime */
	199	+ u64 noise; /* noise */
	200	+ u64 max_sample; /* max single noise sample */
	201	+ int hw_count; /* # HW (incl. hypervisor) interference */
	202	+ int nmi_count; /* # NMIs during this sample */
	203	+ int irq_count; /* # IRQs during this sample */
	204	+ int softirq_count; /* # softirqs during this sample */
	205	+ int thread_count; /* # threads during this sample */
	206	+};
	207	+
	208	+#ifdef CONFIG_TIMERLAT_TRACER
	209	+/*
	210	+ * timerlat sample structure definition. Used to store the statistics of
	211	+ * a sample run.
	212	+ */
	213	+struct timerlat_sample {
	214	+ u64 timer_latency; /* timer_latency */
	215	+ unsigned int seqnum; /* unique sequence */
	216	+ int context; /* timer context */
	217	+};
	218	+#endif
	219	+
	220	+/*
	221	+ * Protect the interface.
	222	+ */
	223	+struct mutex interface_lock;
	224	+
	225	+/*
	226	+ * Tracer data.
	227	+ */
	228	+static struct osnoise_data {
	229	+ u64 sample_period; /* total sampling period */
	230	+ u64 sample_runtime; /* active sampling portion of period */
	231	+ u64 stop_tracing; /* stop trace in the internal operation (loop/irq) */
	232	+ u64 stop_tracing_total; /* stop trace in the final operation (report/thread) */
	233	+#ifdef CONFIG_TIMERLAT_TRACER
	234	+ u64 timerlat_period; /* timerlat period */
	235	+ u64 print_stack; /* print IRQ stack if total > */
	236	+ int timerlat_tracer; /* timerlat tracer */
	237	+#endif
	238	+ bool tainted; /* infor users and developers about a problem */
	239	+} osnoise_data = {
	240	+ .sample_period = DEFAULT_SAMPLE_PERIOD,
	241	+ .sample_runtime = DEFAULT_SAMPLE_RUNTIME,
	242	+ .stop_tracing = 0,
	243	+ .stop_tracing_total = 0,
	244	+#ifdef CONFIG_TIMERLAT_TRACER
	245	+ .print_stack = 0,
	246	+ .timerlat_period = DEFAULT_TIMERLAT_PERIOD,
	247	+ .timerlat_tracer = 0,
	248	+#endif
	249	+};
	250	+
	251	+/*
	252	+ * Boolean variable used to inform that the tracer is currently sampling.
	253	+ */
	254	+static bool osnoise_busy;
	255	+
	256	+/*
	257	+ * Print the osnoise header info.
	258	+ */
	259	+static void print_osnoise_headers(struct seq_file *s)
	260	+{
	261	+ if (osnoise_data.tainted)
	262	+ seq_puts(s, "# osnoise is tainted!\n");
	263	+
	264	+ seq_puts(s, "# _-----=> irqs-off\n");
	265	+ seq_puts(s, "# / _----=> need-resched\n");
	266	+ seq_puts(s, "# \| / _---=> hardirq/softirq\n");
	267	+ seq_puts(s, "# \|\| / _--=> preempt-depth ");
	268	+ seq_puts(s, " MAX\n");
	269	+
	270	+ seq_puts(s, "# \|\| / ");
	271	+ seq_puts(s, " SINGLE Interference counters:\n");
	272	+
	273	+ seq_puts(s, "# \|\|\|\| RUNTIME ");
	274	+ seq_puts(s, " NOISE %% OF CPU NOISE +-----------------------------+\n");
	275	+
	276	+ seq_puts(s, "# TASK-PID CPU# \|\|\|\| TIMESTAMP IN US ");
	277	+ seq_puts(s, " IN US AVAILABLE IN US HW NMI IRQ SIRQ THREAD\n");
	278	+
	279	+ seq_puts(s, "# \| \| \| \|\|\|\| \| \| ");
	280	+ seq_puts(s, " \| \| \| \| \| \| \| \|\n");
	281	+}
	282	+
	283	+/*
	284	+ * osnoise_taint - report an osnoise error.
	285	+ */
	286	+#define osnoise_taint(msg) ({ \
	287	+ struct trace_array *tr = osnoise_trace; \
	288	+ \
	289	+ trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \
	290	+ osnoise_data.tainted = true; \
	291	+})
	292	+
	293	+/*
	294	+ * Record an osnoise_sample into the tracer buffer.
	295	+ */
	296	+static void trace_osnoise_sample(struct osnoise_sample *sample)
	297	+{
	298	+ struct trace_array *tr = osnoise_trace;
	299	+ struct trace_buffer *buffer = tr->array_buffer.buffer;
	300	+ struct trace_event_call *call = &event_osnoise;
	301	+ struct ring_buffer_event *event;
	302	+ struct osnoise_entry *entry;
	303	+
	304	+ event = trace_buffer_lock_reserve(buffer, TRACE_OSNOISE, sizeof(*entry),
	305	+ tracing_gen_ctx());
	306	+ if (!event)
	307	+ return;
	308	+ entry = ring_buffer_event_data(event);
	309	+ entry->runtime = sample->runtime;
	310	+ entry->noise = sample->noise;
	311	+ entry->max_sample = sample->max_sample;
	312	+ entry->hw_count = sample->hw_count;
	313	+ entry->nmi_count = sample->nmi_count;
	314	+ entry->irq_count = sample->irq_count;
	315	+ entry->softirq_count = sample->softirq_count;
	316	+ entry->thread_count = sample->thread_count;
	317	+
	318	+ if (!call_filter_check_discard(call, entry, buffer, event))
	319	+ trace_buffer_unlock_commit_nostack(buffer, event);
	320	+}
	321	+
	322	+#ifdef CONFIG_TIMERLAT_TRACER
	323	+/*
	324	+ * Print the timerlat header info.
	325	+ */
	326	+static void print_timerlat_headers(struct seq_file *s)
	327	+{
	328	+ seq_puts(s, "# _-----=> irqs-off\n");
	329	+ seq_puts(s, "# / _----=> need-resched\n");
	330	+ seq_puts(s, "# \| / _---=> hardirq/softirq\n");
	331	+ seq_puts(s, "# \|\| / _--=> preempt-depth\n");
	332	+ seq_puts(s, "# \|\| /\n");
	333	+ seq_puts(s, "# \|\|\|\| ACTIVATION\n");
	334	+ seq_puts(s, "# TASK-PID CPU# \|\|\|\| TIMESTAMP ID ");
	335	+ seq_puts(s, " CONTEXT LATENCY\n");
	336	+ seq_puts(s, "# \| \| \| \|\|\|\| \| \| ");
	337	+ seq_puts(s, " \| \|\n");
	338	+}
	339	+
	340	+/*
	341	+ * Record an timerlat_sample into the tracer buffer.
	342	+ */
	343	+static void trace_timerlat_sample(struct timerlat_sample *sample)
	344	+{
	345	+ struct trace_array *tr = osnoise_trace;
	346	+ struct trace_event_call *call = &event_osnoise;
	347	+ struct trace_buffer *buffer = tr->array_buffer.buffer;
	348	+ struct ring_buffer_event *event;
	349	+ struct timerlat_entry *entry;
	350	+
	351	+ event = trace_buffer_lock_reserve(buffer, TRACE_TIMERLAT, sizeof(*entry),
	352	+ tracing_gen_ctx());
	353	+ if (!event)
	354	+ return;
	355	+ entry = ring_buffer_event_data(event);
	356	+ entry->seqnum = sample->seqnum;
	357	+ entry->context = sample->context;
	358	+ entry->timer_latency = sample->timer_latency;
	359	+
	360	+ if (!call_filter_check_discard(call, entry, buffer, event))
	361	+ trace_buffer_unlock_commit_nostack(buffer, event);
	362	+}
	363	+
	364	+#ifdef CONFIG_STACKTRACE
	365	+
	366	+#define MAX_CALLS 256
	367	+
	368	+/*
	369	+ * Stack trace will take place only at IRQ level, so, no need
	370	+ * to control nesting here.
	371	+ */
	372	+struct trace_stack {
	373	+ int stack_size;
	374	+ int nr_entries;
	375	+ unsigned long calls[MAX_CALLS];
	376	+};
	377	+
	378	+static DEFINE_PER_CPU(struct trace_stack, trace_stack);
	379	+
	380	+/*
	381	+ * timerlat_save_stack - save a stack trace without printing
	382	+ *
	383	+ * Save the current stack trace without printing. The
	384	+ * stack will be printed later, after the end of the measurement.
	385	+ */
	386	+static void timerlat_save_stack(int skip)
	387	+{
	388	+ unsigned int size, nr_entries;
	389	+ struct trace_stack *fstack;
	390	+
	391	+ fstack = this_cpu_ptr(&trace_stack);
	392	+
	393	+ size = ARRAY_SIZE(fstack->calls);
	394	+
	395	+ nr_entries = stack_trace_save(fstack->calls, size, skip);
	396	+
	397	+ fstack->stack_size = nr_entries * sizeof(unsigned long);
	398	+ fstack->nr_entries = nr_entries;
	399	+
	400	+ return;
	401	+
	402	+}
	403	+/*
	404	+ * timerlat_dump_stack - dump a stack trace previously saved
	405	+ *
	406	+ * Dump a saved stack trace into the trace buffer.
	407	+ */
	408	+static void timerlat_dump_stack(void)
	409	+{
	410	+ struct trace_event_call *call = &event_osnoise;
	411	+ struct trace_array *tr = osnoise_trace;
	412	+ struct trace_buffer *buffer = tr->array_buffer.buffer;
	413	+ struct ring_buffer_event *event;
	414	+ struct trace_stack *fstack;
	415	+ struct stack_entry *entry;
	416	+ unsigned int size;
	417	+
	418	+ preempt_disable_notrace();
	419	+ fstack = this_cpu_ptr(&trace_stack);
	420	+ size = fstack->stack_size;
	421	+
	422	+ event = trace_buffer_lock_reserve(buffer, TRACE_STACK, sizeof(*entry) + size,
	423	+ tracing_gen_ctx());
	424	+ if (!event)
	425	+ goto out;
	426	+
	427	+ entry = ring_buffer_event_data(event);
	428	+
	429	+ memcpy(&entry->caller, fstack->calls, size);
	430	+ entry->size = fstack->nr_entries;
	431	+
	432	+ if (!call_filter_check_discard(call, entry, buffer, event))
	433	+ trace_buffer_unlock_commit_nostack(buffer, event);
	434	+
	435	+out:
	436	+ preempt_enable_notrace();
	437	+}
	438	+#else
	439	+#define timerlat_dump_stack() do {} while (0)
	440	+#define timerlat_save_stack(a) do {} while (0)
	441	+#endif /* CONFIG_STACKTRACE */
	442	+#endif /* CONFIG_TIMERLAT_TRACER */
	443	+
	444	+/*
	445	+ * Macros to encapsulate the time capturing infrastructure.
	446	+ */
	447	+#define time_get() trace_clock_local()
	448	+#define time_to_us(x) div_u64(x, 1000)
	449	+#define time_sub(a, b) ((a) - (b))
	450	+
	451	+/*
	452	+ * cond_move_irq_delta_start - Forward the delta_start of a running IRQ
	453	+ *
	454	+ * If an IRQ is preempted by an NMI, its delta_start is pushed forward
	455	+ * to discount the NMI interference.
	456	+ *
	457	+ * See get_int_safe_duration().
	458	+ */
	459	+static inline void
	460	+cond_move_irq_delta_start(struct osnoise_variables *osn_var, u64 duration)
	461	+{
	462	+ if (osn_var->irq.delta_start)
	463	+ osn_var->irq.delta_start += duration;
	464	+}
	465	+
	466	+#ifndef CONFIG_PREEMPT_RT
	467	+/*
	468	+ * cond_move_softirq_delta_start - Forward the delta_start of a running softirq.
	469	+ *
	470	+ * If a softirq is preempted by an IRQ or NMI, its delta_start is pushed
	471	+ * forward to discount the interference.
	472	+ *
	473	+ * See get_int_safe_duration().
	474	+ */
	475	+static inline void
	476	+cond_move_softirq_delta_start(struct osnoise_variables *osn_var, u64 duration)
	477	+{
	478	+ if (osn_var->softirq.delta_start)
	479	+ osn_var->softirq.delta_start += duration;
	480	+}
	481	+#else /* CONFIG_PREEMPT_RT */
	482	+#define cond_move_softirq_delta_start(osn_var, duration) do {} while (0)
	483	+#endif
	484	+
	485	+/*
	486	+ * cond_move_thread_delta_start - Forward the delta_start of a running thread
	487	+ *
	488	+ * If a noisy thread is preempted by an softirq, IRQ or NMI, its delta_start
	489	+ * is pushed forward to discount the interference.
	490	+ *
	491	+ * See get_int_safe_duration().
	492	+ */
	493	+static inline void
	494	+cond_move_thread_delta_start(struct osnoise_variables *osn_var, u64 duration)
	495	+{
	496	+ if (osn_var->thread.delta_start)
	497	+ osn_var->thread.delta_start += duration;
	498	+}
	499	+
	500	+/*
	501	+ * get_int_safe_duration - Get the duration of a window
	502	+ *
	503	+ * The irq, softirq and thread varaibles need to have its duration without
	504	+ * the interference from higher priority interrupts. Instead of keeping a
	505	+ * variable to discount the interrupt interference from these variables, the
	506	+ * starting time of these variables are pushed forward with the interrupt's
	507	+ * duration. In this way, a single variable is used to:
	508	+ *
	509	+ * - Know if a given window is being measured.
	510	+ * - Account its duration.
	511	+ * - Discount the interference.
	512	+ *
	513	+ * To avoid getting inconsistent values, e.g.,:
	514	+ *
	515	+ * now = time_get()
	516	+ * ---> interrupt!
	517	+ * delta_start -= int duration;
	518	+ * <---
	519	+ * duration = now - delta_start;
	520	+ *
	521	+ * result: negative duration if the variable duration before the
	522	+ * interrupt was smaller than the interrupt execution.
	523	+ *
	524	+ * A counter of interrupts is used. If the counter increased, try
	525	+ * to capture an interference safe duration.
	526	+ */
	527	+static inline s64
	528	+get_int_safe_duration(struct osnoise_variables osn_var, u64 delta_start)
	529	+{
	530	+ u64 int_counter, now;
	531	+ s64 duration;
	532	+
	533	+ do {
	534	+ int_counter = local_read(&osn_var->int_counter);
	535	+ /* synchronize with interrupts */
	536	+ barrier();
	537	+
	538	+ now = time_get();
	539	+ duration = (now - *delta_start);
	540	+
	541	+ /* synchronize with interrupts */
	542	+ barrier();
	543	+ } while (int_counter != local_read(&osn_var->int_counter));
	544	+
	545	+ /*
	546	+ * This is an evidence of race conditions that cause
	547	+ * a value to be "discounted" too much.
	548	+ */
	549	+ if (duration < 0)
	550	+ osnoise_taint("Negative duration!\n");
	551	+
	552	+ *delta_start = 0;
	553	+
	554	+ return duration;
	555	+}
	556	+
	557	+/*
	558	+ *
	559	+ * set_int_safe_time - Save the current time on *time, aware of interference
	560	+ *
	561	+ * Get the time, taking into consideration a possible interference from
	562	+ * higher priority interrupts.
	563	+ *
	564	+ * See get_int_safe_duration() for an explanation.
	565	+ */
	566	+static u64
	567	+set_int_safe_time(struct osnoise_variables osn_var, u64 time)
	568	+{
	569	+ u64 int_counter;
	570	+
	571	+ do {
	572	+ int_counter = local_read(&osn_var->int_counter);
	573	+ /* synchronize with interrupts */
	574	+ barrier();
	575	+
	576	+ *time = time_get();
	577	+
	578	+ /* synchronize with interrupts */
	579	+ barrier();
	580	+ } while (int_counter != local_read(&osn_var->int_counter));
	581	+
	582	+ return int_counter;
	583	+}
	584	+
	585	+#ifdef CONFIG_TIMERLAT_TRACER
	586	+/*
	587	+ * copy_int_safe_time - Copy src into desc aware of interference
	588	+ */
	589	+static u64
	590	+copy_int_safe_time(struct osnoise_variables osn_var, u64 dst, u64 *src)
	591	+{
	592	+ u64 int_counter;
	593	+
	594	+ do {
	595	+ int_counter = local_read(&osn_var->int_counter);
	596	+ /* synchronize with interrupts */
	597	+ barrier();
	598	+
	599	+ dst = src;
	600	+
	601	+ /* synchronize with interrupts */
	602	+ barrier();
	603	+ } while (int_counter != local_read(&osn_var->int_counter));
	604	+
	605	+ return int_counter;
	606	+}
	607	+#endif /* CONFIG_TIMERLAT_TRACER */
	608	+
	609	+/*
	610	+ * trace_osnoise_callback - NMI entry/exit callback
	611	+ *
	612	+ * This function is called at the entry and exit NMI code. The bool enter
	613	+ * distinguishes between either case. This function is used to note a NMI
	614	+ * occurrence, compute the noise caused by the NMI, and to remove the noise
	615	+ * it is potentially causing on other interference variables.
	616	+ */
	617	+void trace_osnoise_callback(bool enter)
	618	+{
	619	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	620	+ u64 duration;
	621	+
	622	+ if (!osn_var->sampling)
	623	+ return;
	624	+
	625	+ /*
	626	+ * Currently trace_clock_local() calls sched_clock() and the
	627	+ * generic version is not NMI safe.
	628	+ */
	629	+ if (!IS_ENABLED(CONFIG_GENERIC_SCHED_CLOCK)) {
	630	+ if (enter) {
	631	+ osn_var->nmi.delta_start = time_get();
	632	+ local_inc(&osn_var->int_counter);
	633	+ } else {
	634	+ duration = time_get() - osn_var->nmi.delta_start;
	635	+
	636	+ trace_nmi_noise(osn_var->nmi.delta_start, duration);
	637	+
	638	+ cond_move_irq_delta_start(osn_var, duration);
	639	+ cond_move_softirq_delta_start(osn_var, duration);
	640	+ cond_move_thread_delta_start(osn_var, duration);
	641	+ }
	642	+ }
	643	+
	644	+ if (enter)
	645	+ osn_var->nmi.count++;
	646	+}
	647	+
	648	+/*
	649	+ * osnoise_trace_irq_entry - Note the starting of an IRQ
	650	+ *
	651	+ * Save the starting time of an IRQ. As IRQs are non-preemptive to other IRQs,
	652	+ * it is safe to use a single variable (ons_var->irq) to save the statistics.
	653	+ * The arrival_time is used to report... the arrival time. The delta_start
	654	+ * is used to compute the duration at the IRQ exit handler. See
	655	+ * cond_move_irq_delta_start().
	656	+ */
	657	+void osnoise_trace_irq_entry(int id)
	658	+{
	659	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	660	+
	661	+ if (!osn_var->sampling)
	662	+ return;
	663	+ /*
	664	+ * This value will be used in the report, but not to compute
	665	+ * the execution time, so it is safe to get it unsafe.
	666	+ */
	667	+ osn_var->irq.arrival_time = time_get();
	668	+ set_int_safe_time(osn_var, &osn_var->irq.delta_start);
	669	+ osn_var->irq.count++;
	670	+
	671	+ local_inc(&osn_var->int_counter);
	672	+}
	673	+
	674	+/*
	675	+ * osnoise_irq_exit - Note the end of an IRQ, sava data and trace
	676	+ *
	677	+ * Computes the duration of the IRQ noise, and trace it. Also discounts the
	678	+ * interference from other sources of noise could be currently being accounted.
	679	+ */
	680	+void osnoise_trace_irq_exit(int id, const char *desc)
	681	+{
	682	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	683	+ int duration;
	684	+
	685	+ if (!osn_var->sampling)
	686	+ return;
	687	+
	688	+ duration = get_int_safe_duration(osn_var, &osn_var->irq.delta_start);
	689	+ trace_irq_noise(id, desc, osn_var->irq.arrival_time, duration);
	690	+ osn_var->irq.arrival_time = 0;
	691	+ cond_move_softirq_delta_start(osn_var, duration);
	692	+ cond_move_thread_delta_start(osn_var, duration);
	693	+}
	694	+
	695	+/*
	696	+ * trace_irqentry_callback - Callback to the irq:irq_entry traceevent
	697	+ *
	698	+ * Used to note the starting of an IRQ occurece.
	699	+ */
	700	+static void trace_irqentry_callback(void *data, int irq,
	701	+ struct irqaction *action)
	702	+{
	703	+ osnoise_trace_irq_entry(irq);
	704	+}
	705	+
	706	+/*
	707	+ * trace_irqexit_callback - Callback to the irq:irq_exit traceevent
	708	+ *
	709	+ * Used to note the end of an IRQ occurece.
	710	+ */
	711	+static void trace_irqexit_callback(void *data, int irq,
	712	+ struct irqaction *action, int ret)
	713	+{
	714	+ osnoise_trace_irq_exit(irq, action->name);
	715	+}
	716	+
	717	+/*
	718	+ * arch specific register function.
	719	+ */
	720	+int __weak osnoise_arch_register(void)
	721	+{
	722	+ return 0;
	723	+}
	724	+
	725	+/*
	726	+ * arch specific unregister function.
	727	+ */
	728	+void __weak osnoise_arch_unregister(void)
	729	+{
	730	+ return;
	731	+}
	732	+
	733	+/*
	734	+ * hook_irq_events - Hook IRQ handling events
	735	+ *
	736	+ * This function hooks the IRQ related callbacks to the respective trace
	737	+ * events.
	738	+ */
	739	+static int hook_irq_events(void)
	740	+{
	741	+ int ret;
	742	+
	743	+ ret = register_trace_irq_handler_entry(trace_irqentry_callback, NULL);
	744	+ if (ret)
	745	+ goto out_err;
	746	+
	747	+ ret = register_trace_irq_handler_exit(trace_irqexit_callback, NULL);
	748	+ if (ret)
	749	+ goto out_unregister_entry;
	750	+
	751	+ ret = osnoise_arch_register();
	752	+ if (ret)
	753	+ goto out_irq_exit;
	754	+
	755	+ return 0;
	756	+
	757	+out_irq_exit:
	758	+ unregister_trace_irq_handler_exit(trace_irqexit_callback, NULL);
	759	+out_unregister_entry:
	760	+ unregister_trace_irq_handler_entry(trace_irqentry_callback, NULL);
	761	+out_err:
	762	+ return -EINVAL;
	763	+}
	764	+
	765	+/*
	766	+ * unhook_irq_events - Unhook IRQ handling events
	767	+ *
	768	+ * This function unhooks the IRQ related callbacks to the respective trace
	769	+ * events.
	770	+ */
	771	+static void unhook_irq_events(void)
	772	+{
	773	+ osnoise_arch_unregister();
	774	+ unregister_trace_irq_handler_exit(trace_irqexit_callback, NULL);
	775	+ unregister_trace_irq_handler_entry(trace_irqentry_callback, NULL);
	776	+}
	777	+
	778	+#ifndef CONFIG_PREEMPT_RT
	779	+/*
	780	+ * trace_softirq_entry_callback - Note the starting of a softirq
	781	+ *
	782	+ * Save the starting time of a softirq. As softirqs are non-preemptive to
	783	+ * other softirqs, it is safe to use a single variable (ons_var->softirq)
	784	+ * to save the statistics. The arrival_time is used to report... the
	785	+ * arrival time. The delta_start is used to compute the duration at the
	786	+ * softirq exit handler. See cond_move_softirq_delta_start().
	787	+ */
	788	+static void trace_softirq_entry_callback(void *data, unsigned int vec_nr)
	789	+{
	790	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	791	+
	792	+ if (!osn_var->sampling)
	793	+ return;
	794	+ /*
	795	+ * This value will be used in the report, but not to compute
	796	+ * the execution time, so it is safe to get it unsafe.
	797	+ */
	798	+ osn_var->softirq.arrival_time = time_get();
	799	+ set_int_safe_time(osn_var, &osn_var->softirq.delta_start);
	800	+ osn_var->softirq.count++;
	801	+
	802	+ local_inc(&osn_var->int_counter);
	803	+}
	804	+
	805	+/*
	806	+ * trace_softirq_exit_callback - Note the end of an softirq
	807	+ *
	808	+ * Computes the duration of the softirq noise, and trace it. Also discounts the
	809	+ * interference from other sources of noise could be currently being accounted.
	810	+ */
	811	+static void trace_softirq_exit_callback(void *data, unsigned int vec_nr)
	812	+{
	813	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	814	+ int duration;
	815	+
	816	+ if (!osn_var->sampling)
	817	+ return;
	818	+
	819	+#ifdef CONFIG_TIMERLAT_TRACER
	820	+ /*
	821	+ * If the timerlat is enabled, but the irq handler did
	822	+ * not run yet enabling timerlat_tracer, do not trace.
	823	+ */
	824	+ if (unlikely(osnoise_data.timerlat_tracer)) {
	825	+ struct timerlat_variables *tlat_var;
	826	+ tlat_var = this_cpu_tmr_var();
	827	+ if (!tlat_var->tracing_thread) {
	828	+ osn_var->softirq.arrival_time = 0;
	829	+ osn_var->softirq.delta_start = 0;
	830	+ return;
	831	+ }
	832	+ }
	833	+#endif
	834	+
	835	+ duration = get_int_safe_duration(osn_var, &osn_var->softirq.delta_start);
	836	+ trace_softirq_noise(vec_nr, osn_var->softirq.arrival_time, duration);
	837	+ cond_move_thread_delta_start(osn_var, duration);
	838	+ osn_var->softirq.arrival_time = 0;
	839	+}
	840	+
	841	+/*
	842	+ * hook_softirq_events - Hook softirq handling events
	843	+ *
	844	+ * This function hooks the softirq related callbacks to the respective trace
	845	+ * events.
	846	+ */
	847	+static int hook_softirq_events(void)
	848	+{
	849	+ int ret;
	850	+
	851	+ ret = register_trace_softirq_entry(trace_softirq_entry_callback, NULL);
	852	+ if (ret)
	853	+ goto out_err;
	854	+
	855	+ ret = register_trace_softirq_exit(trace_softirq_exit_callback, NULL);
	856	+ if (ret)
	857	+ goto out_unreg_entry;
	858	+
	859	+ return 0;
	860	+
	861	+out_unreg_entry:
	862	+ unregister_trace_softirq_entry(trace_softirq_entry_callback, NULL);
	863	+out_err:
	864	+ return -EINVAL;
	865	+}
	866	+
	867	+/*
	868	+ * unhook_softirq_events - Unhook softirq handling events
	869	+ *
	870	+ * This function hooks the softirq related callbacks to the respective trace
	871	+ * events.
	872	+ */
	873	+static void unhook_softirq_events(void)
	874	+{
	875	+ unregister_trace_softirq_entry(trace_softirq_entry_callback, NULL);
	876	+ unregister_trace_softirq_exit(trace_softirq_exit_callback, NULL);
	877	+}
	878	+#else /* CONFIG_PREEMPT_RT */
	879	+/*
	880	+ * softirq are threads on the PREEMPT_RT mode.
	881	+ */
	882	+static int hook_softirq_events(void)
	883	+{
	884	+ return 0;
	885	+}
	886	+static void unhook_softirq_events(void)
	887	+{
	888	+}
	889	+#endif
	890	+
	891	+/*
	892	+ * thread_entry - Record the starting of a thread noise window
	893	+ *
	894	+ * It saves the context switch time for a noisy thread, and increments
	895	+ * the interference counters.
	896	+ */
	897	+static void
	898	+thread_entry(struct osnoise_variables osn_var, struct task_struct t)
	899	+{
	900	+ if (!osn_var->sampling)
	901	+ return;
	902	+ /*
	903	+ * The arrival time will be used in the report, but not to compute
	904	+ * the execution time, so it is safe to get it unsafe.
	905	+ */
	906	+ osn_var->thread.arrival_time = time_get();
	907	+
	908	+ set_int_safe_time(osn_var, &osn_var->thread.delta_start);
	909	+
	910	+ osn_var->thread.count++;
	911	+ local_inc(&osn_var->int_counter);
	912	+}
	913	+
	914	+/*
	915	+ * thread_exit - Report the end of a thread noise window
	916	+ *
	917	+ * It computes the total noise from a thread, tracing if needed.
	918	+ */
	919	+static void
	920	+thread_exit(struct osnoise_variables osn_var, struct task_struct t)
	921	+{
	922	+ int duration;
	923	+
	924	+ if (!osn_var->sampling)
	925	+ return;
	926	+
	927	+#ifdef CONFIG_TIMERLAT_TRACER
	928	+ if (osnoise_data.timerlat_tracer) {
	929	+ struct timerlat_variables *tlat_var;
	930	+ tlat_var = this_cpu_tmr_var();
	931	+ if (!tlat_var->tracing_thread) {
	932	+ osn_var->thread.delta_start = 0;
	933	+ osn_var->thread.arrival_time = 0;
	934	+ return;
	935	+ }
	936	+ }
	937	+#endif
	938	+
	939	+ duration = get_int_safe_duration(osn_var, &osn_var->thread.delta_start);
	940	+
	941	+ trace_thread_noise(t, osn_var->thread.arrival_time, duration);
	942	+
	943	+ osn_var->thread.arrival_time = 0;
	944	+}
	945	+
	946	+/*
	947	+ * trace_sched_switch - sched:sched_switch trace event handler
	948	+ *
	949	+ * This function is hooked to the sched:sched_switch trace event, and it is
	950	+ * used to record the beginning and to report the end of a thread noise window.
	951	+ */
	952	+static void
	953	+trace_sched_switch_callback(void data, bool preempt, struct task_struct p,
	954	+ struct task_struct *n)
	955	+{
	956	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	957	+
	958	+ if (p->pid != osn_var->pid)
	959	+ thread_exit(osn_var, p);
	960	+
	961	+ if (n->pid != osn_var->pid)
	962	+ thread_entry(osn_var, n);
	963	+}
	964	+
	965	+/*
	966	+ * hook_thread_events - Hook the insturmentation for thread noise
	967	+ *
	968	+ * Hook the osnoise tracer callbacks to handle the noise from other
	969	+ * threads on the necessary kernel events.
	970	+ */
	971	+static int hook_thread_events(void)
	972	+{
	973	+ int ret;
	974	+
	975	+ ret = register_trace_sched_switch(trace_sched_switch_callback, NULL);
	976	+ if (ret)
	977	+ return -EINVAL;
	978	+
	979	+ return 0;
	980	+}
	981	+
	982	+/*
	983	+ * unhook_thread_events - *nhook the insturmentation for thread noise
	984	+ *
	985	+ * Unook the osnoise tracer callbacks to handle the noise from other
	986	+ * threads on the necessary kernel events.
	987	+ */
	988	+static void unhook_thread_events(void)
	989	+{
	990	+ unregister_trace_sched_switch(trace_sched_switch_callback, NULL);
	991	+}
	992	+
	993	+/*
	994	+ * save_osn_sample_stats - Save the osnoise_sample statistics
	995	+ *
	996	+ * Save the osnoise_sample statistics before the sampling phase. These
	997	+ * values will be used later to compute the diff betwneen the statistics
	998	+ * before and after the osnoise sampling.
	999	+ */
	1000	+static void
	1001	+save_osn_sample_stats(struct osnoise_variables osn_var, struct osnoise_sample s)
	1002	+{
	1003	+ s->nmi_count = osn_var->nmi.count;
	1004	+ s->irq_count = osn_var->irq.count;
	1005	+ s->softirq_count = osn_var->softirq.count;
	1006	+ s->thread_count = osn_var->thread.count;
	1007	+}
	1008	+
	1009	+/*
	1010	+ * diff_osn_sample_stats - Compute the osnoise_sample statistics
	1011	+ *
	1012	+ * After a sample period, compute the difference on the osnoise_sample
	1013	+ * statistics. The struct osnoise_sample *s contains the statistics saved via
	1014	+ * save_osn_sample_stats() before the osnoise sampling.
	1015	+ */
	1016	+static void
	1017	+diff_osn_sample_stats(struct osnoise_variables osn_var, struct osnoise_sample s)
	1018	+{
	1019	+ s->nmi_count = osn_var->nmi.count - s->nmi_count;
	1020	+ s->irq_count = osn_var->irq.count - s->irq_count;
	1021	+ s->softirq_count = osn_var->softirq.count - s->softirq_count;
	1022	+ s->thread_count = osn_var->thread.count - s->thread_count;
	1023	+}
	1024	+
	1025	+/*
	1026	+ * osnoise_stop_tracing - Stop tracing and the tracer.
	1027	+ */
	1028	+static void osnoise_stop_tracing(void)
	1029	+{
	1030	+ struct trace_array *tr = osnoise_trace;
	1031	+ tracer_tracing_off(tr);
	1032	+}
	1033	+
	1034	+/*
	1035	+ * run_osnoise - Sample the time and look for osnoise
	1036	+ *
	1037	+ * Used to capture the time, looking for potential osnoise latency repeatedly.
	1038	+ * Different from hwlat_detector, it is called with preemption and interrupts
	1039	+ * enabled. This allows irqs, softirqs and threads to run, interfering on the
	1040	+ * osnoise sampling thread, as they would do with a regular thread.
	1041	+ */
	1042	+static int run_osnoise(void)
	1043	+{
	1044	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	1045	+ struct trace_array *tr = osnoise_trace;
	1046	+ u64 start, sample, last_sample;
	1047	+ u64 last_int_count, int_count;
	1048	+ s64 noise = 0, max_noise = 0;
	1049	+ s64 total, last_total = 0;
	1050	+ struct osnoise_sample s;
	1051	+ unsigned int threshold;
	1052	+ u64 runtime, stop_in;
	1053	+ u64 sum_noise = 0;
	1054	+ int hw_count = 0;
	1055	+ int ret = -1;
	1056	+
	1057	+ /*
	1058	+ * Considers the current thread as the workload.
	1059	+ */
	1060	+ osn_var->pid = current->pid;
	1061	+
	1062	+ /*
	1063	+ * Save the current stats for the diff
	1064	+ */
	1065	+ save_osn_sample_stats(osn_var, &s);
	1066	+
	1067	+ /*
	1068	+ * if threshold is 0, use the default value of 5 us.
	1069	+ */
	1070	+ threshold = tracing_thresh ? : 5000;
	1071	+
	1072	+ /*
	1073	+ * Make sure NMIs see sampling first
	1074	+ */
	1075	+ osn_var->sampling = true;
	1076	+ barrier();
	1077	+
	1078	+ /*
	1079	+ * Transform the *_us config to nanoseconds to avoid the
	1080	+ * division on the main loop.
	1081	+ */
	1082	+ runtime = osnoise_data.sample_runtime * NSEC_PER_USEC;
	1083	+ stop_in = osnoise_data.stop_tracing * NSEC_PER_USEC;
	1084	+
	1085	+ /*
	1086	+ * Start timestemp
	1087	+ */
	1088	+ start = time_get();
	1089	+
	1090	+ /*
	1091	+ * "previous" loop.
	1092	+ */
	1093	+ last_int_count = set_int_safe_time(osn_var, &last_sample);
	1094	+
	1095	+ do {
	1096	+ /*
	1097	+ * Get sample!
	1098	+ */
	1099	+ int_count = set_int_safe_time(osn_var, &sample);
	1100	+
	1101	+ noise = time_sub(sample, last_sample);
	1102	+
	1103	+ /*
	1104	+ * This shouldn't happen.
	1105	+ */
	1106	+ if (noise < 0) {
	1107	+ osnoise_taint("negative noise!");
	1108	+ goto out;
	1109	+ }
	1110	+
	1111	+ /*
	1112	+ * Sample runtime.
	1113	+ */
	1114	+ total = time_sub(sample, start);
	1115	+
	1116	+ /*
	1117	+ * Check for possible overflows.
	1118	+ */
	1119	+ if (total < last_total) {
	1120	+ osnoise_taint("total overflow!");
	1121	+ break;
	1122	+ }
	1123	+
	1124	+ last_total = total;
	1125	+
	1126	+ if (noise >= threshold) {
	1127	+ int interference = int_count - last_int_count;
	1128	+
	1129	+ if (noise > max_noise)
	1130	+ max_noise = noise;
	1131	+
	1132	+ if (!interference)
	1133	+ hw_count++;
	1134	+
	1135	+ sum_noise += noise;
	1136	+
	1137	+ trace_sample_threshold(last_sample, noise, interference);
	1138	+
	1139	+ if (osnoise_data.stop_tracing)
	1140	+ if (noise > stop_in)
	1141	+ osnoise_stop_tracing();
	1142	+ }
	1143	+
	1144	+ /*
	1145	+ * For the non-preemptive kernel config: let threads runs, if
	1146	+ * they so wish.
	1147	+ */
	1148	+ cond_resched();
	1149	+
	1150	+ last_sample = sample;
	1151	+ last_int_count = int_count;
	1152	+
	1153	+ } while (total < runtime && !kthread_should_stop());
	1154	+
	1155	+ /*
	1156	+ * Finish the above in the view for interrupts.
	1157	+ */
	1158	+ barrier();
	1159	+
	1160	+ osn_var->sampling = false;
	1161	+
	1162	+ /*
	1163	+ * Make sure sampling data is no longer updated.
	1164	+ */
	1165	+ barrier();
	1166	+
	1167	+ /*
	1168	+ * Save noise info.
	1169	+ */
	1170	+ s.noise = time_to_us(sum_noise);
	1171	+ s.runtime = time_to_us(total);
	1172	+ s.max_sample = time_to_us(max_noise);
	1173	+ s.hw_count = hw_count;
	1174	+
	1175	+ /* Save interference stats info */
	1176	+ diff_osn_sample_stats(osn_var, &s);
	1177	+
	1178	+ trace_osnoise_sample(&s);
	1179	+
	1180	+ /* Keep a running maximum ever recorded osnoise "latency" */
	1181	+ if (max_noise > tr->max_latency) {
	1182	+ tr->max_latency = max_noise;
	1183	+ latency_fsnotify(tr);
	1184	+ }
	1185	+
	1186	+ if (osnoise_data.stop_tracing_total)
	1187	+ if (s.noise > osnoise_data.stop_tracing_total)
	1188	+ osnoise_stop_tracing();
	1189	+
	1190	+ return 0;
	1191	+out:
	1192	+ return ret;
	1193	+}
	1194	+
	1195	+static struct cpumask osnoise_cpumask;
	1196	+static struct cpumask save_cpumask;
	1197	+
	1198	+/*
	1199	+ * osnoise_main - The osnoise detection kernel thread
	1200	+ *
	1201	+ * Calls run_osnoise() function to measure the osnoise for the configured runtime,
	1202	+ * every period.
	1203	+ */
	1204	+static int osnoise_main(void *data)
	1205	+{
	1206	+ u64 interval;
	1207	+
	1208	+ while (!kthread_should_stop()) {
	1209	+
	1210	+ run_osnoise();
	1211	+
	1212	+ mutex_lock(&interface_lock);
	1213	+ interval = osnoise_data.sample_period - osnoise_data.sample_runtime;
	1214	+ mutex_unlock(&interface_lock);
	1215	+
	1216	+ do_div(interval, USEC_PER_MSEC);
	1217	+
	1218	+ /*
	1219	+ * differently from hwlat_detector, the osnoise tracer can run
	1220	+ * without a pause because preemption is on.
	1221	+ */
	1222	+ if (interval < 1) {
	1223	+ /* Let synchronize_rcu_tasks() make progress */
	1224	+ cond_resched_tasks_rcu_qs();
	1225	+ continue;
	1226	+ }
	1227	+
	1228	+ if (msleep_interruptible(interval))
	1229	+ break;
	1230	+ }
	1231	+
	1232	+ return 0;
	1233	+}
	1234	+
	1235	+#ifdef CONFIG_TIMERLAT_TRACER
	1236	+/*
	1237	+ * timerlat_irq - hrtimer handler for timerlat.
	1238	+ */
	1239	+static enum hrtimer_restart timerlat_irq(struct hrtimer *timer)
	1240	+{
	1241	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	1242	+ struct trace_array *tr = osnoise_trace;
	1243	+ struct timerlat_variables *tlat;
	1244	+ struct timerlat_sample s;
	1245	+ u64 now;
	1246	+ u64 diff;
	1247	+
	1248	+ /*
	1249	+ * I am not sure if the timer was armed for this CPU. So, get
	1250	+ * the timerlat struct from the timer itself, not from this
	1251	+ * CPU.
	1252	+ */
	1253	+ tlat = container_of(timer, struct timerlat_variables, timer);
	1254	+
	1255	+ now = ktime_to_ns(hrtimer_cb_get_time(&tlat->timer));
	1256	+
	1257	+ /*
	1258	+ * Enable the osnoise: events for thread an softirq.
	1259	+ */
	1260	+ tlat->tracing_thread = true;
	1261	+
	1262	+ osn_var->thread.arrival_time = time_get();
	1263	+
	1264	+ /*
	1265	+ * A hardirq is running: the timer IRQ. It is for sure preempting
	1266	+ * a thread, and potentially preempting a softirq.
	1267	+ *
	1268	+ * At this point, it is not interesting to know the duration of the
	1269	+ * preempted thread (and maybe softirq), but how much time they will
	1270	+ * delay the beginning of the execution of the timer thread.
	1271	+ *
	1272	+ * To get the correct (net) delay added by the softirq, its delta_start
	1273	+ * is set as the IRQ one. In this way, at the return of the IRQ, the delta
	1274	+ * start of the sofitrq will be zeroed, accounting then only the time
	1275	+ * after that.
	1276	+ *
	1277	+ * The thread follows the same principle. However, if a softirq is
	1278	+ * running, the thread needs to receive the softirq delta_start. The
	1279	+ * reason being is that the softirq will be the last to be unfolded,
	1280	+ * resseting the thread delay to zero.
	1281	+ */
	1282	+#ifndef CONFIG_PREEMPT_RT
	1283	+ if (osn_var->softirq.delta_start) {
	1284	+ copy_int_safe_time(osn_var, &osn_var->thread.delta_start,
	1285	+ &osn_var->softirq.delta_start);
	1286	+
	1287	+ copy_int_safe_time(osn_var, &osn_var->softirq.delta_start,
	1288	+ &osn_var->irq.delta_start);
	1289	+ } else {
	1290	+ copy_int_safe_time(osn_var, &osn_var->thread.delta_start,
	1291	+ &osn_var->irq.delta_start);
	1292	+ }
	1293	+#else /* CONFIG_PREEMPT_RT */
	1294	+ /*
	1295	+ * The sofirqs run as threads on RT, so there is not need
	1296	+ * to keep track of it.
	1297	+ */
	1298	+ copy_int_safe_time(osn_var, &osn_var->thread.delta_start, &osn_var->irq.delta_start);
	1299	+#endif /* CONFIG_PREEMPT_RT */
	1300	+
	1301	+ /*
	1302	+ * Compute the current time with the expected time.
	1303	+ */
	1304	+ diff = now - tlat->abs_period;
	1305	+
	1306	+ tlat->count++;
	1307	+ s.seqnum = tlat->count;
	1308	+ s.timer_latency = diff;
	1309	+ s.context = IRQ_CONTEXT;
	1310	+
	1311	+ trace_timerlat_sample(&s);
	1312	+
	1313	+ /* Keep a running maximum ever recorded os noise "latency" */
	1314	+ if (diff > tr->max_latency) {
	1315	+ tr->max_latency = diff;
	1316	+ latency_fsnotify(tr);
	1317	+ }
	1318	+
	1319	+ if (osnoise_data.stop_tracing)
	1320	+ if (time_to_us(diff) >= osnoise_data.stop_tracing)
	1321	+ osnoise_stop_tracing();
	1322	+
	1323	+ wake_up_process(tlat->kthread);
	1324	+
	1325	+ if (osnoise_data.print_stack)
	1326	+ timerlat_save_stack(0);
	1327	+
	1328	+ return HRTIMER_NORESTART;
	1329	+}
	1330	+
	1331	+/*
	1332	+ * wait_next_period - Wait for the next period for timerlat
	1333	+ */
	1334	+static int wait_next_period(struct timerlat_variables *tlat)
	1335	+{
	1336	+ ktime_t next_abs_period, now;
	1337	+ u64 rel_period = osnoise_data.timerlat_period * 1000;
	1338	+
	1339	+ now = hrtimer_cb_get_time(&tlat->timer);
	1340	+ next_abs_period = ns_to_ktime(tlat->abs_period + rel_period);
	1341	+
	1342	+ /*
	1343	+ * Save the next abs_period.
	1344	+ */
	1345	+ tlat->abs_period = (u64) ktime_to_ns(next_abs_period);
	1346	+
	1347	+ /*
	1348	+ * If the new abs_period is in the past, skip the activation.
	1349	+ */
	1350	+ while (ktime_compare(now, next_abs_period) > 0) {
	1351	+ next_abs_period = ns_to_ktime(tlat->abs_period + rel_period);
	1352	+ tlat->abs_period = (u64) ktime_to_ns(next_abs_period);
	1353	+ }
	1354	+
	1355	+ set_current_state(TASK_INTERRUPTIBLE);
	1356	+
	1357	+ hrtimer_start(&tlat->timer, next_abs_period, HRTIMER_MODE_ABS_PINNED_HARD);
	1358	+ schedule();
	1359	+ return 1;
	1360	+}
	1361	+
	1362	+/*
	1363	+ * timerlat_main- Timerlat main
	1364	+ */
	1365	+static int timerlat_main(void *data)
	1366	+{
	1367	+ struct osnoise_variables *osn_var = this_cpu_osn_var();
	1368	+ struct timerlat_variables *tlat = this_cpu_tmr_var();
	1369	+ struct timerlat_sample s;
	1370	+ struct sched_param sp;
	1371	+ u64 now, diff;
	1372	+
	1373	+ /*
	1374	+ * Make the thread RT, that is how cyclictest is usually used.
	1375	+ */
	1376	+ sp.sched_priority = DEFAULT_TIMERLAT_PRIO;
	1377	+ sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
	1378	+
	1379	+ tlat->count = 0;
	1380	+ tlat->tracing_thread = false;
	1381	+
	1382	+ hrtimer_init(&tlat->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
	1383	+ tlat->timer.function = timerlat_irq;
	1384	+ tlat->kthread = current;
	1385	+ osn_var->pid = current->pid;
	1386	+ /*
	1387	+ * Anotate the arrival time.
	1388	+ */
	1389	+ tlat->abs_period = hrtimer_cb_get_time(&tlat->timer);
	1390	+
	1391	+ wait_next_period(tlat);
	1392	+
	1393	+ osn_var->sampling = 1;
	1394	+
	1395	+ while (!kthread_should_stop()) {
	1396	+ now = ktime_to_ns(hrtimer_cb_get_time(&tlat->timer));
	1397	+ diff = now - tlat->abs_period;
	1398	+
	1399	+ s.seqnum = tlat->count;
	1400	+ s.timer_latency = diff;
	1401	+ s.context = THREAD_CONTEXT;
	1402	+
	1403	+ trace_timerlat_sample(&s);
	1404	+
	1405	+#ifdef CONFIG_STACKTRACE
	1406	+ if (osnoise_data.print_stack)
	1407	+ if (osnoise_data.print_stack <= time_to_us(diff))
	1408	+ timerlat_dump_stack();
	1409	+#endif /* CONFIG_STACKTRACE */
	1410	+
	1411	+ tlat->tracing_thread = false;
	1412	+ if (osnoise_data.stop_tracing_total)
	1413	+ if (time_to_us(diff) >= osnoise_data.stop_tracing_total)
	1414	+ osnoise_stop_tracing();
	1415	+
	1416	+ wait_next_period(tlat);
	1417	+ }
	1418	+
	1419	+ hrtimer_cancel(&tlat->timer);
	1420	+ return 0;
	1421	+}
	1422	+#endif /* CONFIG_TIMERLAT_TRACER */
	1423	+
	1424	+/*
	1425	+ * stop_kthread - stop a workload thread
	1426	+ */
	1427	+static void stop_kthread(unsigned int cpu)
	1428	+{
	1429	+ struct task_struct *kthread;
	1430	+
	1431	+ kthread = per_cpu(per_cpu_osnoise_var, cpu).kthread;
	1432	+ if (kthread)
	1433	+ kthread_stop(kthread);
	1434	+ per_cpu(per_cpu_osnoise_var, cpu).kthread = NULL;
	1435	+}
	1436	+
	1437	+/*
	1438	+ * stop_per_cpu_kthread - Stop per-cpu threads
	1439	+ *
	1440	+ * Stop the osnoise sampling htread. Use this on unload and at system
	1441	+ * shutdown.
	1442	+ */
	1443	+static void stop_per_cpu_kthreads(void)
	1444	+{
	1445	+ int cpu;
	1446	+
	1447	+ get_online_cpus();
	1448	+
	1449	+ for_each_online_cpu(cpu)
	1450	+ stop_kthread(cpu);
	1451	+
	1452	+ put_online_cpus();
	1453	+}
	1454	+
	1455	+/*
	1456	+ * start_kthread - Start a workload tread
	1457	+ */
	1458	+static int start_kthread(unsigned int cpu)
	1459	+{
	1460	+ struct task_struct *kthread;
	1461	+ void *main = osnoise_main;
	1462	+ char comm[24];
	1463	+
	1464	+#ifdef CONFIG_TIMERLAT_TRACER
	1465	+ if (osnoise_data.timerlat_tracer) {
	1466	+ snprintf(comm, 24, "timerlat/%d", cpu);
	1467	+ main = timerlat_main;
	1468	+ } else {
	1469	+ snprintf(comm, 24, "osnoise/%d", cpu);
	1470	+ }
	1471	+#else
	1472	+ snprintf(comm, 24, "osnoise/%d", cpu);
	1473	+#endif
	1474	+ kthread = kthread_create_on_cpu(main, NULL, cpu, comm);
	1475	+
	1476	+ if (IS_ERR(kthread)) {
	1477	+ pr_err(BANNER "could not start sampling thread\n");
	1478	+ stop_per_cpu_kthreads();
	1479	+ return -ENOMEM;
	1480	+ }
	1481	+
	1482	+ per_cpu(per_cpu_osnoise_var, cpu).kthread = kthread;
	1483	+ wake_up_process(kthread);
	1484	+
	1485	+ return 0;
	1486	+}
	1487	+
	1488	+/*
	1489	+ * start_per_cpu_kthread - Kick off per-cpu osnoise sampling kthreads
	1490	+ *
	1491	+ * This starts the kernel thread that will look for osnoise on many
	1492	+ * cpus.
	1493	+ */
	1494	+static int start_per_cpu_kthreads(struct trace_array *tr)
	1495	+{
	1496	+ struct cpumask *current_mask = &save_cpumask;
	1497	+ int retval;
	1498	+ int cpu;
	1499	+
	1500	+ get_online_cpus();
	1501	+ /*
	1502	+ * Run only on CPUs in which trace and osnoise are allowed to run.
	1503	+ */
	1504	+ cpumask_and(current_mask, tr->tracing_cpumask, &osnoise_cpumask);
	1505	+ /*
	1506	+ * And the CPU is online.
	1507	+ */
	1508	+ cpumask_and(current_mask, cpu_online_mask, current_mask);
	1509	+
	1510	+ for_each_possible_cpu(cpu)
	1511	+ per_cpu(per_cpu_osnoise_var, cpu).kthread = NULL;
	1512	+
	1513	+ for_each_cpu(cpu, current_mask) {
	1514	+ retval = start_kthread(cpu);
	1515	+ if (retval) {
	1516	+ stop_per_cpu_kthreads();
	1517	+ return retval;
	1518	+ }
	1519	+ }
	1520	+
	1521	+ put_online_cpus();
	1522	+
	1523	+ return 0;
	1524	+}
	1525	+
	1526	+#ifdef CONFIG_HOTPLUG_CPU
	1527	+static void osnoise_hotplug_workfn(struct work_struct *dummy)
	1528	+{
	1529	+ struct trace_array *tr = osnoise_trace;
	1530	+ unsigned int cpu = smp_processor_id();
	1531	+
	1532	+
	1533	+ mutex_lock(&trace_types_lock);
	1534	+
	1535	+ if (!osnoise_busy)
	1536	+ goto out_unlock_trace;
	1537	+
	1538	+ mutex_lock(&interface_lock);
	1539	+ get_online_cpus();
	1540	+
	1541	+ if (!cpumask_test_cpu(cpu, &osnoise_cpumask))
	1542	+ goto out_unlock;
	1543	+
	1544	+ if (!cpumask_test_cpu(cpu, tr->tracing_cpumask))
	1545	+ goto out_unlock;
	1546	+
	1547	+ start_kthread(cpu);
	1548	+
	1549	+out_unlock:
	1550	+ put_online_cpus();
	1551	+ mutex_unlock(&interface_lock);
	1552	+out_unlock_trace:
	1553	+ mutex_unlock(&trace_types_lock);
	1554	+}
	1555	+
	1556	+static DECLARE_WORK(osnoise_hotplug_work, osnoise_hotplug_workfn);
	1557	+
	1558	+/*
	1559	+ * osnoise_cpu_init - CPU hotplug online callback function
	1560	+ */
	1561	+static int osnoise_cpu_init(unsigned int cpu)
	1562	+{
	1563	+ schedule_work_on(cpu, &osnoise_hotplug_work);
	1564	+ return 0;
	1565	+}
	1566	+
	1567	+/*
	1568	+ * osnoise_cpu_die - CPU hotplug offline callback function
	1569	+ */
	1570	+static int osnoise_cpu_die(unsigned int cpu)
	1571	+{
	1572	+ stop_kthread(cpu);
	1573	+ return 0;
	1574	+}
	1575	+
	1576	+static void osnoise_init_hotplug_support(void)
	1577	+{
	1578	+ int ret;
	1579	+
	1580	+ ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "trace/osnoise:online",
	1581	+ osnoise_cpu_init, osnoise_cpu_die);
	1582	+ if (ret < 0)
	1583	+ pr_warn(BANNER "Error to init cpu hotplug support\n");
	1584	+
	1585	+ return;
	1586	+}
	1587	+#else /* CONFIG_HOTPLUG_CPU */
	1588	+static void osnoise_init_hotplug_support(void)
	1589	+{
	1590	+ return;
	1591	+}
	1592	+#endif /* CONFIG_HOTPLUG_CPU */
	1593	+
	1594	+/*
	1595	+ * osnoise_cpus_read - Read function for reading the "cpus" file
	1596	+ * @filp: The active open file structure
	1597	+ * @ubuf: The userspace provided buffer to read value into
	1598	+ * @cnt: The maximum number of bytes to read
	1599	+ * @ppos: The current "file" position
	1600	+ *
	1601	+ * Prints the "cpus" output into the user-provided buffer.
	1602	+ */
	1603	+static ssize_t
	1604	+osnoise_cpus_read(struct file filp, char __user ubuf, size_t count,
	1605	+ loff_t *ppos)
	1606	+{
	1607	+ char *mask_str;
	1608	+ int len;
	1609	+
	1610	+ mutex_lock(&interface_lock);
	1611	+
	1612	+ len = snprintf(NULL, 0, "%*pbl\n", cpumask_pr_args(&osnoise_cpumask)) + 1;
	1613	+ mask_str = kmalloc(len, GFP_KERNEL);
	1614	+ if (!mask_str) {
	1615	+ count = -ENOMEM;
	1616	+ goto out_unlock;
	1617	+ }
	1618	+
	1619	+ len = snprintf(mask_str, len, "%*pbl\n", cpumask_pr_args(&osnoise_cpumask));
	1620	+ if (len >= count) {
	1621	+ count = -EINVAL;
	1622	+ goto out_free;
	1623	+ }
	1624	+
	1625	+ count = simple_read_from_buffer(ubuf, count, ppos, mask_str, len);
	1626	+
	1627	+out_free:
	1628	+ kfree(mask_str);
	1629	+out_unlock:
	1630	+ mutex_unlock(&interface_lock);
	1631	+
	1632	+ return count;
	1633	+}
	1634	+
	1635	+static void osnoise_tracer_start(struct trace_array *tr);
	1636	+static void osnoise_tracer_stop(struct trace_array *tr);
	1637	+
	1638	+/*
	1639	+ * osnoise_cpus_write - Write function for "cpus" entry
	1640	+ * @filp: The active open file structure
	1641	+ * @ubuf: The user buffer that contains the value to write
	1642	+ * @cnt: The maximum number of bytes to write to "file"
	1643	+ * @ppos: The current position in @file
	1644	+ *
	1645	+ * This function provides a write implementation for the "cpus"
	1646	+ * interface to the osnoise trace. By default, it lists all CPUs,
	1647	+ * in this way, allowing osnoise threads to run on any online CPU
	1648	+ * of the system. It serves to restrict the execution of osnoise to the
	1649	+ * set of CPUs writing via this interface. Note that osnoise also
	1650	+ * respects the "tracing_cpumask." Hence, osnoise threads will run only
	1651	+ * on the set of CPUs allowed here AND on "tracing_cpumask." Why not
	1652	+ * have just "tracing_cpumask?" Because the user might be interested
	1653	+ * in tracing what is running on other CPUs. For instance, one might
	1654	+ * run osnoise in one HT CPU while observing what is running on the
	1655	+ * sibling HT CPU.
	1656	+ */
	1657	+static ssize_t
	1658	+osnoise_cpus_write(struct file filp, const char __user ubuf, size_t count,
	1659	+ loff_t *ppos)
	1660	+{
	1661	+ struct trace_array *tr = osnoise_trace;
	1662	+ cpumask_var_t osnoise_cpumask_new;
	1663	+ int running, err;
	1664	+ char buf[256];
	1665	+
	1666	+ if (count >= 256)
	1667	+ return -EINVAL;
	1668	+
	1669	+ if (copy_from_user(buf, ubuf, count))
	1670	+ return -EFAULT;
	1671	+
	1672	+ if (!zalloc_cpumask_var(&osnoise_cpumask_new, GFP_KERNEL))
	1673	+ return -ENOMEM;
	1674	+
	1675	+ err = cpulist_parse(buf, osnoise_cpumask_new);
	1676	+ if (err)
	1677	+ goto err_free;
	1678	+
	1679	+ /*
	1680	+ * trace_types_lock is taken to avoid concurrency on start/stop
	1681	+ * and osnoise_busy.
	1682	+ */
	1683	+ mutex_lock(&trace_types_lock);
	1684	+ running = osnoise_busy;
	1685	+ if (running)
	1686	+ osnoise_tracer_stop(tr);
	1687	+
	1688	+ mutex_lock(&interface_lock);
	1689	+ /*
	1690	+ * osnoise_cpumask is read by CPU hotplug operations.
	1691	+ */
	1692	+ get_online_cpus();
	1693	+
	1694	+ cpumask_copy(&osnoise_cpumask, osnoise_cpumask_new);
	1695	+
	1696	+ put_online_cpus();
	1697	+ mutex_unlock(&interface_lock);
	1698	+
	1699	+ if (running)
	1700	+ osnoise_tracer_start(tr);
	1701	+ mutex_unlock(&trace_types_lock);
	1702	+
	1703	+ free_cpumask_var(osnoise_cpumask_new);
	1704	+ return count;
	1705	+
	1706	+err_free:
	1707	+ free_cpumask_var(osnoise_cpumask_new);
	1708	+
	1709	+ return err;
	1710	+}
	1711	+
	1712	+/*
	1713	+ * osnoise/runtime_us: cannot be greater than the period.
	1714	+ */
	1715	+static struct trace_min_max_param osnoise_runtime = {
	1716	+ .lock = &interface_lock,
	1717	+ .val = &osnoise_data.sample_runtime,
	1718	+ .max = &osnoise_data.sample_period,
	1719	+ .min = NULL,
	1720	+};
	1721	+
	1722	+/*
	1723	+ * osnoise/period_us: cannot be smaller than the runtime.
	1724	+ */
	1725	+static struct trace_min_max_param osnoise_period = {
	1726	+ .lock = &interface_lock,
	1727	+ .val = &osnoise_data.sample_period,
	1728	+ .max = NULL,
	1729	+ .min = &osnoise_data.sample_runtime,
	1730	+};
	1731	+
	1732	+/*
	1733	+ * osnoise/stop_tracing_us: no limit.
	1734	+ */
	1735	+static struct trace_min_max_param osnoise_stop_tracing_in = {
	1736	+ .lock = &interface_lock,
	1737	+ .val = &osnoise_data.stop_tracing,
	1738	+ .max = NULL,
	1739	+ .min = NULL,
	1740	+};
	1741	+
	1742	+/*
	1743	+ * osnoise/stop_tracing_total_us: no limit.
	1744	+ */
	1745	+static struct trace_min_max_param osnoise_stop_tracing_total = {
	1746	+ .lock = &interface_lock,
	1747	+ .val = &osnoise_data.stop_tracing_total,
	1748	+ .max = NULL,
	1749	+ .min = NULL,
	1750	+};
	1751	+
	1752	+#ifdef CONFIG_TIMERLAT_TRACER
	1753	+/*
	1754	+ * osnoise/print_stack: print the stacktrace of the IRQ handler if the total
	1755	+ * latency is higher than val.
	1756	+ */
	1757	+static struct trace_min_max_param osnoise_print_stack = {
	1758	+ .lock = &interface_lock,
	1759	+ .val = &osnoise_data.print_stack,
	1760	+ .max = NULL,
	1761	+ .min = NULL,
	1762	+};
	1763	+
	1764	+/*
	1765	+ * osnoise/timerlat_period: min 100 us, max 1 s
	1766	+ */
	1767	+u64 timerlat_min_period = 100;
	1768	+u64 timerlat_max_period = 1000000;
	1769	+static struct trace_min_max_param timerlat_period = {
	1770	+ .lock = &interface_lock,
	1771	+ .val = &osnoise_data.timerlat_period,
	1772	+ .max = &timerlat_max_period,
	1773	+ .min = &timerlat_min_period,
	1774	+};
	1775	+#endif
	1776	+
	1777	+static const struct file_operations cpus_fops = {
	1778	+ .open = tracing_open_generic,
	1779	+ .read = osnoise_cpus_read,
	1780	+ .write = osnoise_cpus_write,
	1781	+ .llseek = generic_file_llseek,
	1782	+};
	1783	+
	1784	+/*
	1785	+ * init_tracefs - A function to initialize the tracefs interface files
	1786	+ *
	1787	+ * This function creates entries in tracefs for "osnoise" and "timerlat".
	1788	+ * It creates these directories in the tracing directory, and within that
	1789	+ * directory the use can change and view the configs.
	1790	+ */
	1791	+static int init_tracefs(void)
	1792	+{
	1793	+ struct dentry *top_dir;
	1794	+ struct dentry *tmp;
	1795	+ int ret;
	1796	+
	1797	+ ret = tracing_init_dentry();
	1798	+ if (ret)
	1799	+ return -ENOMEM;
	1800	+
	1801	+ top_dir = tracefs_create_dir("osnoise", NULL);
	1802	+ if (!top_dir)
	1803	+ return 0;
	1804	+
	1805	+ tmp = tracefs_create_file("period_us", 0640, top_dir,
	1806	+ &osnoise_period, &trace_min_max_fops);
	1807	+ if (!tmp)
	1808	+ goto err;
	1809	+
	1810	+ tmp = tracefs_create_file("runtime_us", 0644, top_dir,
	1811	+ &osnoise_runtime, &trace_min_max_fops);
	1812	+ if (!tmp)
	1813	+ goto err;
	1814	+
	1815	+ tmp = tracefs_create_file("stop_tracing_us", 0640, top_dir,
	1816	+ &osnoise_stop_tracing_in, &trace_min_max_fops);
	1817	+ if (!tmp)
	1818	+ goto err;
	1819	+
	1820	+ tmp = tracefs_create_file("stop_tracing_total_us", 0640, top_dir,
	1821	+ &osnoise_stop_tracing_total, &trace_min_max_fops);
	1822	+ if (!tmp)
	1823	+ goto err;
	1824	+
	1825	+ tmp = trace_create_file("cpus", 0644, top_dir, NULL, &cpus_fops);
	1826	+ if (!tmp)
	1827	+ goto err;
	1828	+#ifdef CONFIG_TIMERLAT_TRACER
	1829	+#ifdef CONFIG_STACKTRACE
	1830	+ tmp = tracefs_create_file("print_stack", 0640, top_dir,
	1831	+ &osnoise_print_stack, &trace_min_max_fops);
	1832	+ if (!tmp)
	1833	+ goto err;
	1834	+#endif
	1835	+
	1836	+ tmp = tracefs_create_file("timerlat_period_us", 0640, top_dir,
	1837	+ &timerlat_period, &trace_min_max_fops);
	1838	+ if (!tmp)
	1839	+ goto err;
	1840	+#endif
	1841	+
	1842	+ return 0;
	1843	+
	1844	+err:
	1845	+ tracefs_remove(top_dir);
	1846	+ return -ENOMEM;
	1847	+}
	1848	+
	1849	+static int osnoise_hook_events(void)
	1850	+{
	1851	+ int retval;
	1852	+
	1853	+ /*
	1854	+ * Trace is already hooked, we are re-enabling from
	1855	+ * a stop_tracing_*.
	1856	+ */
	1857	+ if (trace_osnoise_callback_enabled)
	1858	+ return 0;
	1859	+
	1860	+ retval = hook_irq_events();
	1861	+ if (retval)
	1862	+ return -EINVAL;
	1863	+
	1864	+ retval = hook_softirq_events();
	1865	+ if (retval)
	1866	+ goto out_unhook_irq;
	1867	+
	1868	+ retval = hook_thread_events();
	1869	+ /*
	1870	+ * All fine!
	1871	+ */
	1872	+ if (!retval)
	1873	+ return 0;
	1874	+
	1875	+ unhook_softirq_events();
	1876	+out_unhook_irq:
	1877	+ unhook_irq_events();
	1878	+ return -EINVAL;
	1879	+}
	1880	+
	1881	+static int __osnoise_tracer_start(struct trace_array *tr)
	1882	+{
	1883	+ int retval;
	1884	+
	1885	+ osn_var_reset_all();
	1886	+
	1887	+ retval = osnoise_hook_events();
	1888	+ if (retval)
	1889	+ return retval;
	1890	+ /*
	1891	+ * Make sure NMIs see reseted values.
	1892	+ */
	1893	+ barrier();
	1894	+ trace_osnoise_callback_enabled = true;
	1895	+
	1896	+ retval = start_per_cpu_kthreads(tr);
	1897	+ if (retval) {
	1898	+ unhook_irq_events();
	1899	+ return retval;
	1900	+ }
	1901	+
	1902	+ osnoise_busy = true;
	1903	+
	1904	+ return 0;
	1905	+}
	1906	+
	1907	+static void osnoise_tracer_start(struct trace_array *tr)
	1908	+{
	1909	+ int retval;
	1910	+
	1911	+ if (osnoise_busy)
	1912	+ return;
	1913	+
	1914	+ retval = __osnoise_tracer_start(tr);
	1915	+ if (retval)
	1916	+ pr_err(BANNER "Error starting osnoise tracer\n");
	1917	+
	1918	+}
	1919	+
	1920	+static void osnoise_tracer_stop(struct trace_array *tr)
	1921	+{
	1922	+ if (!osnoise_busy)
	1923	+ return;
	1924	+
	1925	+ trace_osnoise_callback_enabled = false;
	1926	+ barrier();
	1927	+
	1928	+ stop_per_cpu_kthreads();
	1929	+
	1930	+ unhook_irq_events();
	1931	+ unhook_softirq_events();
	1932	+ unhook_thread_events();
	1933	+
	1934	+ osnoise_busy = false;
	1935	+}
	1936	+
	1937	+static int osnoise_tracer_init(struct trace_array *tr)
	1938	+{
	1939	+
	1940	+ /* Only allow one instance to enable this */
	1941	+ if (osnoise_busy)
	1942	+ return -EBUSY;
	1943	+
	1944	+ osnoise_trace = tr;
	1945	+ tr->max_latency = 0;
	1946	+
	1947	+ osnoise_tracer_start(tr);
	1948	+
	1949	+ return 0;
	1950	+}
	1951	+
	1952	+static void osnoise_tracer_reset(struct trace_array *tr)
	1953	+{
	1954	+ osnoise_tracer_stop(tr);
	1955	+}
	1956	+
	1957	+static struct tracer osnoise_tracer __read_mostly = {
	1958	+ .name = "osnoise",
	1959	+ .init = osnoise_tracer_init,
	1960	+ .reset = osnoise_tracer_reset,
	1961	+ .start = osnoise_tracer_start,
	1962	+ .stop = osnoise_tracer_stop,
	1963	+ .print_header = print_osnoise_headers,
	1964	+ .allow_instances = true,
	1965	+};
	1966	+
	1967	+#ifdef CONFIG_TIMERLAT_TRACER
	1968	+static void timerlat_tracer_start(struct trace_array *tr)
	1969	+{
	1970	+ int retval;
	1971	+
	1972	+ if (osnoise_busy)
	1973	+ return;
	1974	+
	1975	+ osnoise_data.timerlat_tracer = 1;
	1976	+
	1977	+ retval = __osnoise_tracer_start(tr);
	1978	+ if (retval)
	1979	+ goto out_err;
	1980	+
	1981	+ return;
	1982	+out_err:
	1983	+ pr_err(BANNER "Error starting timerlat tracer\n");
	1984	+}
	1985	+
	1986	+static void timerlat_tracer_stop(struct trace_array *tr)
	1987	+{
	1988	+ int cpu;
	1989	+
	1990	+ if (!osnoise_busy)
	1991	+ return;
	1992	+
	1993	+ for_each_online_cpu(cpu)
	1994	+ per_cpu(per_cpu_osnoise_var, cpu).sampling = 0;
	1995	+
	1996	+ osnoise_tracer_stop(tr);
	1997	+
	1998	+ osnoise_data.timerlat_tracer = 0;
	1999	+}
	2000	+
	2001	+static int timerlat_tracer_init(struct trace_array *tr)
	2002	+{
	2003	+ /* Only allow one instance to enable this */
	2004	+ if (osnoise_busy)
	2005	+ return -EBUSY;
	2006	+
	2007	+ osnoise_trace = tr;
	2008	+
	2009	+ tr->max_latency = 0;
	2010	+
	2011	+ timerlat_tracer_start(tr);
	2012	+
	2013	+ return 0;
	2014	+}
	2015	+
	2016	+static void timerlat_tracer_reset(struct trace_array *tr)
	2017	+{
	2018	+ timerlat_tracer_stop(tr);
	2019	+}
	2020	+
	2021	+static struct tracer timerlat_tracer __read_mostly = {
	2022	+ .name = "timerlat",
	2023	+ .init = timerlat_tracer_init,
	2024	+ .reset = timerlat_tracer_reset,
	2025	+ .start = timerlat_tracer_start,
	2026	+ .stop = timerlat_tracer_stop,
	2027	+ .print_header = print_timerlat_headers,
	2028	+ .allow_instances = true,
	2029	+};
	2030	+#endif /* CONFIG_TIMERLAT_TRACER */
	2031	+
	2032	+__init static int init_osnoise_tracer(void)
	2033	+{
	2034	+ int ret;
	2035	+
	2036	+ mutex_init(&interface_lock);
	2037	+
	2038	+ cpumask_copy(&osnoise_cpumask, cpu_all_mask);
	2039	+
	2040	+ ret = register_tracer(&osnoise_tracer);
	2041	+ if (ret) {
	2042	+ pr_err(BANNER "Error registering osnoise!\n");
	2043	+ return ret;
	2044	+ }
	2045	+
	2046	+#ifdef CONFIG_TIMERLAT_TRACER
	2047	+ ret = register_tracer(&timerlat_tracer);
	2048	+ if (ret) {
	2049	+ pr_err(BANNER "Error registering timerlat\n");
	2050	+ return ret;
	2051	+ }
	2052	+#endif
	2053	+ osnoise_init_hotplug_support();
	2054	+
	2055	+ init_tracefs();
	2056	+
	2057	+ return 0;
	2058	+}
	2059	+late_initcall(init_osnoise_tracer);

kernel/trace/trace_output.c+118−1

@@ -1202,7 +1202,6 @@ trace_hwlat_print(struct trace_iterator *iter, int flags, · 1201 unmodified lines
1202	1202	return trace_handle_return(s);
1203	1203	}
1204	1204
1205		−
1206	1205	static enum print_line_t
1207	1206	trace_hwlat_raw(struct trace_iterator *iter, int flags,
1208	1207	struct trace_event *event)
@@ -1232,6 +1231,122 @@ static struct trace_event trace_hwlat_event = { · 23 unmodified lines
1232	1231	.funcs = &trace_hwlat_funcs,
1233	1232	};
1234	1233
	1234	+/* TRACE_OSNOISE */
	1235	+static enum print_line_t
	1236	+trace_osnoise_print(struct trace_iterator *iter, int flags,
	1237	+ struct trace_event *event)
	1238	+{
	1239	+ struct trace_entry *entry = iter->ent;
	1240	+ struct trace_seq *s = &iter->seq;
	1241	+ struct osnoise_entry *field;
	1242	+ u64 ratio, ratio_dec;
	1243	+ u64 net_runtime;
	1244	+
	1245	+ trace_assign_type(field, entry);
	1246	+
	1247	+ /*
	1248	+ * compute the available % of cpu time.
	1249	+ */
	1250	+ net_runtime = field->runtime - field->noise;
	1251	+ ratio = net_runtime * 10000000;
	1252	+ do_div(ratio, field->runtime);
	1253	+ ratio_dec = do_div(ratio, 100000);
	1254	+
	1255	+ trace_seq_printf(s, "%llu %10llu %3llu.%05llu %7llu",
	1256	+ field->runtime,
	1257	+ field->noise,
	1258	+ ratio, ratio_dec,
	1259	+ field->max_sample);
	1260	+
	1261	+ trace_seq_printf(s, " %6u", field->hw_count);
	1262	+ trace_seq_printf(s, " %6u", field->nmi_count);
	1263	+ trace_seq_printf(s, " %6u", field->irq_count);
	1264	+ trace_seq_printf(s, " %6u", field->softirq_count);
	1265	+ trace_seq_printf(s, " %6u", field->thread_count);
	1266	+
	1267	+ trace_seq_putc(s, '\n');
	1268	+
	1269	+ return trace_handle_return(s);
	1270	+}
	1271	+
	1272	+static enum print_line_t
	1273	+trace_osnoise_raw(struct trace_iterator *iter, int flags,
	1274	+ struct trace_event *event)
	1275	+{
	1276	+ struct osnoise_entry *field;
	1277	+ struct trace_seq *s = &iter->seq;
	1278	+
	1279	+ trace_assign_type(field, iter->ent);
	1280	+
	1281	+ trace_seq_printf(s, "%lld %llu %llu %u %u %u %u %u\n",
	1282	+ field->runtime,
	1283	+ field->noise,
	1284	+ field->max_sample,
	1285	+ field->hw_count,
	1286	+ field->nmi_count,
	1287	+ field->irq_count,
	1288	+ field->softirq_count,
	1289	+ field->thread_count);
	1290	+
	1291	+ return trace_handle_return(s);
	1292	+}
	1293	+
	1294	+static struct trace_event_functions trace_osnoise_funcs = {
	1295	+ .trace = trace_osnoise_print,
	1296	+ .raw = trace_osnoise_raw,
	1297	+};
	1298	+
	1299	+static struct trace_event trace_osnoise_event = {
	1300	+ .type = TRACE_OSNOISE,
	1301	+ .funcs = &trace_osnoise_funcs,
	1302	+};
	1303	+
	1304	+/* TRACE_TIMERLAT */
	1305	+static enum print_line_t
	1306	+trace_timerlat_print(struct trace_iterator *iter, int flags,
	1307	+ struct trace_event *event)
	1308	+{
	1309	+ struct trace_entry *entry = iter->ent;
	1310	+ struct trace_seq *s = &iter->seq;
	1311	+ struct timerlat_entry *field;
	1312	+
	1313	+ trace_assign_type(field, entry);
	1314	+
	1315	+ trace_seq_printf(s, "#%-5u context %6s timer_latency %9llu ns\n",
	1316	+ field->seqnum,
	1317	+ field->context ? "thread" : "irq",
	1318	+ field->timer_latency);
	1319	+
	1320	+ return trace_handle_return(s);
	1321	+}
	1322	+
	1323	+static enum print_line_t
	1324	+trace_timerlat_raw(struct trace_iterator *iter, int flags,
	1325	+ struct trace_event *event)
	1326	+{
	1327	+ struct timerlat_entry *field;
	1328	+ struct trace_seq *s = &iter->seq;
	1329	+
	1330	+ trace_assign_type(field, iter->ent);
	1331	+
	1332	+ trace_seq_printf(s, "%u %d %llu\n",
	1333	+ field->seqnum,
	1334	+ field->context,
	1335	+ field->timer_latency);
	1336	+
	1337	+ return trace_handle_return(s);
	1338	+}
	1339	+
	1340	+static struct trace_event_functions trace_timerlat_funcs = {
	1341	+ .trace = trace_timerlat_print,
	1342	+ .raw = trace_timerlat_raw,
	1343	+};
	1344	+
	1345	+static struct trace_event trace_timerlat_event = {
	1346	+ .type = TRACE_TIMERLAT,
	1347	+ .funcs = &trace_timerlat_funcs,
	1348	+};
	1349	+
1235	1350	/* TRACE_BPUTS */
1236	1351	static enum print_line_t
1237	1352	trace_bputs_print(struct trace_iterator *iter, int flags,
@@ -1442,6 +1557,8 @@ static struct trace_event *events[] __initdata = { · 204 unmodified lines
1442	1557	&trace_bprint_event,
1443	1558	&trace_print_event,
1444	1559	&trace_hwlat_event,
	1560	+ &trace_osnoise_event,
	1561	+ &trace_timerlat_event,
1445	1562	&trace_raw_data_event,
1446	1563	&trace_func_repeats_event,
1447	1564	NULL

kernel/trace/trace_sched_wakeup.c+12−12

@@ -26,9 +26,9 @@ static struct task_struct *wakeup_task; · 25 unmodified lines
26	26	static int wakeup_cpu;
27	27	static int wakeup_current_cpu;
28	28	static unsigned wakeup_prio = -1;
29		−static int wakeup_rt;
30		−static int wakeup_dl;
31		−static int tracing_dl = 0;
	29	+static bool wakeup_rt;
	30	+static bool wakeup_dl;
	31	+static bool tracing_dl;
32	32
33	33	static arch_spinlock_t wakeup_lock =
34	34	(arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
@@ -498,7 +498,7 @@ static void __wakeup_reset(struct trace_array *tr) · 463 unmodified lines
498	498	{
499	499	wakeup_cpu = -1;
500	500	wakeup_prio = -1;
501		− tracing_dl = 0;
	501	+ tracing_dl = false;
502	502
503	503	if (wakeup_task)
504	504	put_task_struct(wakeup_task);
@@ -572,9 +572,9 @@ probe_wakeup(void ignore, struct task_struct p) · 67 unmodified lines
572	572	* another task until the first one wakes up.
573	573	*/
574	574	if (dl_task(p))
575		− tracing_dl = 1;
	575	+ tracing_dl = true;
576	576	else
577		− tracing_dl = 0;
	577	+ tracing_dl = false;
578	578
579	579	wakeup_task = get_task_struct(p);
580	580
@@ -685,8 +685,8 @@ static int wakeup_tracer_init(struct trace_array *tr) · 104 unmodified lines
685	685	if (wakeup_busy)
686	686	return -EBUSY;
687	687
688		− wakeup_dl = 0;
689		− wakeup_rt = 0;
	688	+ wakeup_dl = false;
	689	+ wakeup_rt = false;
690	690	return __wakeup_tracer_init(tr);
691	691	}
692	692
@@ -695,8 +695,8 @@ static int wakeup_rt_tracer_init(struct trace_array *tr) · 2 unmodified lines
695	695	if (wakeup_busy)
696	696	return -EBUSY;
697	697
698		− wakeup_dl = 0;
699		− wakeup_rt = 1;
	698	+ wakeup_dl = false;
	699	+ wakeup_rt = true;
700	700	return __wakeup_tracer_init(tr);
701	701	}
702	702
@@ -705,8 +705,8 @@ static int wakeup_dl_tracer_init(struct trace_array *tr) · 2 unmodified lines
705	705	if (wakeup_busy)
706	706	return -EBUSY;
707	707
708		− wakeup_dl = 1;
709		− wakeup_rt = 0;
	708	+ wakeup_dl = true;
	709	+ wakeup_rt = false;
710	710	return __wakeup_tracer_init(tr);
711	711	}
712	712