Merge tag 'trace-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace

Pull tracing updates from Steven Rostedt: - Added option for per CPU threads to the hwlat tracer - Have hwlat tracer handle hotplug CPUs - New tracer: osnoise, that detects latency caused by interrupts, softirqs and scheduling of other tasks. - Added timerlat tracer that creates a thread and measures in detail what sources of latency it has for wake ups. - Removed the "success" field of the sched_wakeup trace event. This has been hardcoded as "1" since 2015, no tooling should be looking at it now. If one exists, we can revert this commit, fix that tool and try to remove it again in the future. - tgid mapping fixed to handle more than PID_MAX_DEFAULT pids/tgids. - New boot command line option "tp_printk_stop", as tp_printk causes trace events to write to console. When user space starts, this can easily live lock the system. Having a boot option to stop just after boot up is useful to prevent that from happening. - Have ftrace_dump_on_oops boot command line option take numbers that match the numbers shown in /proc/sys/kernel/ftrace_dump_on_oops. - Bootconfig clean ups, fixes and enhancements. - New ktest script that tests bootconfig options. - Add tracepoint_probe_register_may_exist() to register a tracepoint without triggering a WARN*() if it already exists. BPF has a path from user space that can do this. All other paths are considered a bug. - Small clean ups and fixes * tag 'trace-v5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace: (49 commits) tracing: Resize tgid_map to pid_max, not PID_MAX_DEFAULT tracing: Simplify & fix saved_tgids logic treewide: Add missing semicolons to __assign_str uses tracing: Change variable type as bool for clean-up trace/timerlat: Fix indentation on timerlat_main() trace/osnoise: Make 'noise' variable s64 in run_osnoise() tracepoint: Add tracepoint_probe_register_may_exist() for BPF tracing tracing: Fix spelling in osnoise tracer "interferences" -> "interference" Documentation: Fix a typo on trace/osnoise-tracer trace/osnoise: Fix return value on osnoise_init_hotplug_support trace/osnoise: Make interval u64 on osnoise_main trace/osnoise: Fix 'no previous prototype' warnings tracing: Have osnoise_main() add a quiescent state for task rcu seq_buf: Make trace_seq_putmem_hex() support data longer than 8 seq_buf: Fix overflow in seq_buf_putmem_hex() trace/osnoise: Support hotplug operations trace/hwlat: Support hotplug operations trace/hwlat: Protect kdata->kthread with get/put_online_cpus trace: Add timerlat tracer trace: Add osnoise tracer ...

Linus Torvalds committed 5 years ago

Commit 757fa802 parents a70bb58 4030a6e74 files · +4439 −369Browse files

Showing the changes to kernel/trace (14 of 74 files). Show all

Unified · Split

14 files

kernel/trace/Kconfig +62 −0
kernel/trace/Makefile +1 −0
kernel/trace/bpf_trace.c +2 −1
kernel/trace/ring_buffer.c +1 −1
kernel/trace/trace.c +211 −54
kernel/trace/trace.h +29 −6
kernel/trace/trace_boot.c +25 −2
kernel/trace/trace_entries.h +41 −0
kernel/trace/trace_events_hist.c +2 −1
kernel/trace/trace_events_trigger.c +2 −1
kernel/trace/trace_hwlat.c +392 −142
kernel/trace/trace_osnoise.c +2059 −0
kernel/trace/trace_output.c +118 −1
kernel/trace/trace_sched_wakeup.c +12 −12

kernel/trace/Kconfig+62−0

@@ -356,6 +356,68 @@ config HWLAT_TRACER · 355 unmodified lines
356	file. Every time a latency is greater than tracing_thresh, it will	356	file. Every time a latency is greater than tracing_thresh, it will
357	be recorded into the ring buffer.	357	be recorded into the ring buffer.
358		358
		359	config OSNOISE_TRACER
		360	bool "OS Noise tracer"
		361	select GENERIC_TRACER
		362	help
		363	In the context of high-performance computing (HPC), the Operating
		364	System Noise (osnoise) refers to the interference experienced by an
		365	application due to activities inside the operating system. In the
		366	context of Linux, NMIs, IRQs, SoftIRQs, and any other system thread
		367	can cause noise to the system. Moreover, hardware-related jobs can
		368	also cause noise, for example, via SMIs.
		369
		370	The osnoise tracer leverages the hwlat_detector by running a similar
		371	loop with preemption, SoftIRQs and IRQs enabled, thus allowing all
		372	the sources of osnoise during its execution. The osnoise tracer takes
		373	note of the entry and exit point of any source of interferences,
		374	increasing a per-cpu interference counter. It saves an interference
		375	counter for each source of interference. The interference counter for
		376	NMI, IRQs, SoftIRQs, and threads is increased anytime the tool
		377	observes these interferences' entry events. When a noise happens
		378	without any interference from the operating system level, the
		379	hardware noise counter increases, pointing to a hardware-related
		380	noise. In this way, osnoise can account for any source of
		381	interference. At the end of the period, the osnoise tracer prints
		382	the sum of all noise, the max single noise, the percentage of CPU
		383	available for the thread, and the counters for the noise sources.
		384
		385	In addition to the tracer, a set of tracepoints were added to
		386	facilitate the identification of the osnoise source.
		387
		388	The output will appear in the trace and trace_pipe files.
		389
		390	To enable this tracer, echo in "osnoise" into the current_tracer
		391	file.
		392
		393	config TIMERLAT_TRACER
		394	bool "Timerlat tracer"
		395	select OSNOISE_TRACER
		396	select GENERIC_TRACER
		397	help
		398	The timerlat tracer aims to help the preemptive kernel developers
		399	to find sources of wakeup latencies of real-time threads.
		400
		401	The tracer creates a per-cpu kernel thread with real-time priority.
		402	The tracer thread sets a periodic timer to wakeup itself, and goes
		403	to sleep waiting for the timer to fire. At the wakeup, the thread
		404	then computes a wakeup latency value as the difference between
		405	the current time and the absolute time that the timer was set
		406	to expire.
		407
		408	The tracer prints two lines at every activation. The first is the
		409	timer latency observed at the hardirq context before the
		410	activation of the thread. The second is the timer latency observed
		411	by the thread, which is the same level that cyclictest reports. The
		412	ACTIVATION ID field serves to relate the irq execution to its
		413	respective thread execution.
		414
		415	The tracer is build on top of osnoise tracer, and the osnoise:
		416	events can be used to trace the source of interference from NMI,
		417	IRQs and other threads. It also enables the capture of the
		418	stacktrace at the IRQ context, which helps to identify the code
		419	path that can cause thread delay.
		420
359	config MMIOTRACE	421	config MMIOTRACE
360	bool "Memory mapped IO tracing"	422	bool "Memory mapped IO tracing"
361	depends on HAVE_MMIOTRACE_SUPPORT && PCI	423	depends on HAVE_MMIOTRACE_SUPPORT && PCI

kernel/trace/Makefile+1−0

@@ -58,6 +58,7 @@ obj-$(CONFIG_IRQSOFF_TRACER) += trace_irqsoff.o · 57 unmodified lines
58	obj-$(CONFIG_PREEMPT_TRACER) += trace_irqsoff.o	58	obj-$(CONFIG_PREEMPT_TRACER) += trace_irqsoff.o
59	obj-$(CONFIG_SCHED_TRACER) += trace_sched_wakeup.o	59	obj-$(CONFIG_SCHED_TRACER) += trace_sched_wakeup.o
60	obj-$(CONFIG_HWLAT_TRACER) += trace_hwlat.o	60	obj-$(CONFIG_HWLAT_TRACER) += trace_hwlat.o
		61	obj-$(CONFIG_OSNOISE_TRACER) += trace_osnoise.o
61	obj-$(CONFIG_NOP_TRACER) += trace_nop.o	62	obj-$(CONFIG_NOP_TRACER) += trace_nop.o
62	obj-$(CONFIG_STACK_TRACER) += trace_stack.o	63	obj-$(CONFIG_STACK_TRACER) += trace_stack.o
63	obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o	64	obj-$(CONFIG_MMIOTRACE) += trace_mmiotrace.o

kernel/trace/bpf_trace.c+2−1

@@ -1842,7 +1842,8 @@ static int __bpf_probe_register(struct bpf_raw_event_map btp, struct bpf_prog · 1841 unmodified lines
1842	if (prog->aux->max_tp_access > btp->writable_size)	1842	if (prog->aux->max_tp_access > btp->writable_size)
1843	return -EINVAL;	1843	return -EINVAL;
1844		1844
1845	return tracepoint_probe_register(tp, (void *)btp->bpf_func, prog);	1845	return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func,
		1846	prog);
1846	}	1847	}
1847		1848
1848	int bpf_probe_register(struct bpf_raw_event_map btp, struct bpf_prog prog)	1849	int bpf_probe_register(struct bpf_raw_event_map btp, struct bpf_prog prog)

kernel/trace/ring_buffer.c+1−1

@@ -3391,7 +3391,7 @@ static void check_buffer(struct ring_buffer_per_cpu *cpu_buffer, · 3390 unmodified lines
3391	case RINGBUF_TYPE_PADDING:	3391	case RINGBUF_TYPE_PADDING:
3392	if (event->time_delta == 1)	3392	if (event->time_delta == 1)
3393	break;	3393	break;
3394	/* fall through */	3394	fallthrough;
3395	case RINGBUF_TYPE_DATA:	3395	case RINGBUF_TYPE_DATA:
3396	ts += event->time_delta;	3396	ts += event->time_delta;
3397	break;	3397	break;

kernel/trace/trace.c+211−54

@@ -87,6 +87,7 @@ void __init disable_tracing_selftest(const char *reason) · 86 unmodified lines
87	/* Pipe tracepoints to printk */	87	/* Pipe tracepoints to printk */
88	struct trace_iterator *tracepoint_print_iter;	88	struct trace_iterator *tracepoint_print_iter;
89	int tracepoint_printk;	89	int tracepoint_printk;
		90	static bool tracepoint_printk_stop_on_boot __initdata;
90	static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);	91	static DEFINE_STATIC_KEY_FALSE(tracepoint_printk_key);
91		92
92	/* For tracers that don't implement custom flags */	93	/* For tracers that don't implement custom flags */
@@ -197,12 +198,12 @@ __setup("ftrace=", set_cmdline_ftrace); · 104 unmodified lines
197		198
198	static int __init set_ftrace_dump_on_oops(char *str)	199	static int __init set_ftrace_dump_on_oops(char *str)
199	{	200	{
200	if (str++ != '=' \|\| !str) {	201	if (str++ != '=' \|\| !str \|\| !strcmp("1", str)) {
201	ftrace_dump_on_oops = DUMP_ALL;	202	ftrace_dump_on_oops = DUMP_ALL;
202	return 1;	203	return 1;
203	}	204	}
204		205
205	if (!strcmp("orig_cpu", str)) {	206	if (!strcmp("orig_cpu", str) \|\| !strcmp("2", str)) {
206	ftrace_dump_on_oops = DUMP_ORIG;	207	ftrace_dump_on_oops = DUMP_ORIG;
207	return 1;	208	return 1;
208	}	209	}
@@ -257,6 +258,13 @@ static int __init set_tracepoint_printk(char *str) · 48 unmodified lines
257	}	258	}
258	__setup("tp_printk", set_tracepoint_printk);	259	__setup("tp_printk", set_tracepoint_printk);
259		260
		261	static int __init set_tracepoint_printk_stop(char *str)
		262	{
		263	tracepoint_printk_stop_on_boot = true;
		264	return 1;
		265	}
		266	__setup("tp_printk_stop_on_boot", set_tracepoint_printk_stop);
		267
260	unsigned long long ns2usecs(u64 nsec)	268	unsigned long long ns2usecs(u64 nsec)
261	{	269	{
262	nsec += 500;	270	nsec += 500;
@@ -1683,8 +1691,7 @@ static ssize_t trace_seq_to_buffer(struct trace_seq s, void buf, size_t cnt) · 1420 unmodified lines
1683	unsigned long __read_mostly tracing_thresh;	1691	unsigned long __read_mostly tracing_thresh;
1684	static const struct file_operations tracing_max_lat_fops;	1692	static const struct file_operations tracing_max_lat_fops;
1685		1693
1686	#if (defined(CONFIG_TRACER_MAX_TRACE) \|\| defined(CONFIG_HWLAT_TRACER)) && \	1694	#ifdef LATENCY_FS_NOTIFY
1687	defined(CONFIG_FSNOTIFY)
1688		1695
1689	static struct workqueue_struct *fsnotify_wq;	1696	static struct workqueue_struct *fsnotify_wq;
1690		1697
@@ -2185,8 +2192,15 @@ void tracing_reset_all_online_cpus(void) · 494 unmodified lines
2185	}	2192	}
2186	}	2193	}
2187		2194
		2195	/*
		2196	* The tgid_map array maps from pid to tgid; i.e. the value stored at index i
		2197	* is the tgid last observed corresponding to pid=i.
		2198	*/
2188	static int *tgid_map;	2199	static int *tgid_map;
2189		2200
		2201	/* The maximum valid index into tgid_map. */
		2202	static size_t tgid_map_max;
		2203
2190	#define SAVED_CMDLINES_DEFAULT 128	2204	#define SAVED_CMDLINES_DEFAULT 128
2191	#define NO_CMDLINE_MAP UINT_MAX	2205	#define NO_CMDLINE_MAP UINT_MAX
2192	static arch_spinlock_t trace_cmdline_lock = __ARCH_SPIN_LOCK_UNLOCKED;	2206	static arch_spinlock_t trace_cmdline_lock = __ARCH_SPIN_LOCK_UNLOCKED;
@@ -2459,24 +2473,41 @@ void trace_find_cmdline(int pid, char comm[]) · 266 unmodified lines
2459	preempt_enable();	2473	preempt_enable();
2460	}	2474	}
2461		2475
		2476	static int *trace_find_tgid_ptr(int pid)
		2477	{
		2478	/*
		2479	* Pairs with the smp_store_release in set_tracer_flag() to ensure that
		2480	* if we observe a non-NULL tgid_map then we also observe the correct
		2481	* tgid_map_max.
		2482	*/
		2483	int *map = smp_load_acquire(&tgid_map);
		2484
		2485	if (unlikely(!map \|\| pid > tgid_map_max))
		2486	return NULL;
		2487
		2488	return &map[pid];
		2489	}
		2490
2462	int trace_find_tgid(int pid)	2491	int trace_find_tgid(int pid)
2463	{	2492	{
2464	if (unlikely(!tgid_map \|\| !pid \|\| pid > PID_MAX_DEFAULT))	2493	int *ptr = trace_find_tgid_ptr(pid);
2465	return 0;
2466		2494
2467	return tgid_map[pid];	2495	return ptr ? *ptr : 0;
2468	}	2496	}
2469		2497
2470	static int trace_save_tgid(struct task_struct *tsk)	2498	static int trace_save_tgid(struct task_struct *tsk)
2471	{	2499	{
		2500	int *ptr;
		2501
2472	/* treat recording of idle task as a success */	2502	/* treat recording of idle task as a success */
2473	if (!tsk->pid)	2503	if (!tsk->pid)
2474	return 1;	2504	return 1;
2475		2505
2476	if (unlikely(!tgid_map \|\| tsk->pid > PID_MAX_DEFAULT))	2506	ptr = trace_find_tgid_ptr(tsk->pid);
		2507	if (!ptr)
2477	return 0;	2508	return 0;
2478		2509
2479	tgid_map[tsk->pid] = tsk->tgid;	2510	*ptr = tsk->tgid;
2480	return 1;	2511	return 1;
2481	}	2512	}
2482		2513
@@ -2730,9 +2761,45 @@ trace_event_buffer_lock_reserve(struct trace_buffer **current_rb, · 247 unmodified lines
2730	if (!tr->no_filter_buffering_ref &&	2761	if (!tr->no_filter_buffering_ref &&
2731	(trace_file->flags & (EVENT_FILE_FL_SOFT_DISABLED \| EVENT_FILE_FL_FILTERED)) &&	2762	(trace_file->flags & (EVENT_FILE_FL_SOFT_DISABLED \| EVENT_FILE_FL_FILTERED)) &&
2732	(entry = this_cpu_read(trace_buffered_event))) {	2763	(entry = this_cpu_read(trace_buffered_event))) {
2733	/* Try to use the per cpu buffer first */	2764	/*
		2765	* Filtering is on, so try to use the per cpu buffer first.
		2766	* This buffer will simulate a ring_buffer_event,
		2767	* where the type_len is zero and the array[0] will
		2768	* hold the full length.
		2769	* (see include/linux/ring-buffer.h for details on
		2770	* how the ring_buffer_event is structured).
		2771	*
		2772	* Using a temp buffer during filtering and copying it
		2773	* on a matched filter is quicker than writing directly
		2774	* into the ring buffer and then discarding it when
		2775	* it doesn't match. That is because the discard
		2776	* requires several atomic operations to get right.
		2777	* Copying on match and doing nothing on a failed match
		2778	* is still quicker than no copy on match, but having
		2779	* to discard out of the ring buffer on a failed match.
		2780	*/
		2781	int max_len = PAGE_SIZE - struct_size(entry, array, 1);
		2782
2734	val = this_cpu_inc_return(trace_buffered_event_cnt);	2783	val = this_cpu_inc_return(trace_buffered_event_cnt);
2735	if ((len < (PAGE_SIZE - sizeof(*entry) - sizeof(entry->array[0]))) && val == 1) {	2784
		2785	/*
		2786	* Preemption is disabled, but interrupts and NMIs
		2787	* can still come in now. If that happens after
		2788	* the above increment, then it will have to go
		2789	* back to the old method of allocating the event
		2790	* on the ring buffer, and if the filter fails, it
		2791	* will have to call ring_buffer_discard_commit()
		2792	* to remove it.
		2793	*
		2794	* Need to also check the unlikely case that the
		2795	* length is bigger than the temp buffer size.
		2796	* If that happens, then the reserve is pretty much
		2797	* guaranteed to fail, as the ring buffer currently
		2798	* only allows events less than a page. But that may
		2799	* change in the future, so let the ring buffer reserve
		2800	* handle the failure in that case.
		2801	*/
		2802	if (val == 1 && likely(len <= max_len)) {
2736	trace_event_setup(entry, type, trace_ctx);	2803	trace_event_setup(entry, type, trace_ctx);
2737	entry->array[0] = len;	2804	entry->array[0] = len;
2738	return entry;	2805	return entry;
@@ -5172,6 +5239,8 @@ int trace_keep_overwrite(struct tracer *tracer, u32 mask, int set) · 2433 unmodified lines
5172		5239
5173	int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)	5240	int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)
5174	{	5241	{
		5242	int *map;
		5243
5175	if ((mask == TRACE_ITER_RECORD_TGID) \|\|	5244	if ((mask == TRACE_ITER_RECORD_TGID) \|\|
5176	(mask == TRACE_ITER_RECORD_CMD))	5245	(mask == TRACE_ITER_RECORD_CMD))
5177	lockdep_assert_held(&event_mutex);	5246	lockdep_assert_held(&event_mutex);
@@ -5194,10 +5263,19 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled) · 16 unmodified lines
5194	trace_event_enable_cmd_record(enabled);	5263	trace_event_enable_cmd_record(enabled);
5195		5264
5196	if (mask == TRACE_ITER_RECORD_TGID) {	5265	if (mask == TRACE_ITER_RECORD_TGID) {
5197	if (!tgid_map)	5266	if (!tgid_map) {
5198	tgid_map = kvcalloc(PID_MAX_DEFAULT + 1,	5267	tgid_map_max = pid_max;
5199	sizeof(*tgid_map),	5268	map = kvcalloc(tgid_map_max + 1, sizeof(*tgid_map),
5200	GFP_KERNEL);	5269	GFP_KERNEL);
		5270
		5271	/*
		5272	* Pairs with smp_load_acquire() in
		5273	* trace_find_tgid_ptr() to ensure that if it observes
		5274	* the tgid_map we just allocated then it also observes
		5275	* the corresponding tgid_map_max value.
		5276	*/
		5277	smp_store_release(&tgid_map, map);
		5278	}
5201	if (!tgid_map) {	5279	if (!tgid_map) {
5202	tr->trace_flags &= ~TRACE_ITER_RECORD_TGID;	5280	tr->trace_flags &= ~TRACE_ITER_RECORD_TGID;
5203	return -ENOMEM;	5281	return -ENOMEM;
@@ -5609,37 +5687,16 @@ static const struct file_operations tracing_readme_fops = { · 405 unmodified lines
5609		5687
5610	static void saved_tgids_next(struct seq_file m, void v, loff_t pos)	5688	static void saved_tgids_next(struct seq_file m, void v, loff_t pos)
5611	{	5689	{
5612	int *ptr = v;	5690	int pid = ++(*pos);
5613
5614	if (*pos \|\| m->count)
5615	ptr++;
5616
5617	(*pos)++;
5618
5619	for (; ptr <= &tgid_map[PID_MAX_DEFAULT]; ptr++) {
5620	if (trace_find_tgid(*ptr))
5621	return ptr;
5622	}
5623		5691
5624	return NULL;	5692	return trace_find_tgid_ptr(pid);
5625	}	5693	}
5626		5694
5627	static void saved_tgids_start(struct seq_file m, loff_t *pos)	5695	static void saved_tgids_start(struct seq_file m, loff_t *pos)
5628	{	5696	{
5629	void *v;	5697	int pid = *pos;
5630	loff_t l = 0;
5631
5632	if (!tgid_map)
5633	return NULL;
5634		5698
5635	v = &tgid_map[0];	5699	return trace_find_tgid_ptr(pid);
5636	while (l <= *pos) {
5637	v = saved_tgids_next(m, v, &l);
5638	if (!v)
5639	return NULL;
5640	}
5641
5642	return v;
5643	}	5700	}
5644		5701
5645	static void saved_tgids_stop(struct seq_file m, void v)	5702	static void saved_tgids_stop(struct seq_file m, void v)
@@ -5648,9 +5705,14 @@ static void saved_tgids_stop(struct seq_file m, void v) · 2 unmodified lines
5648		5705
5649	static int saved_tgids_show(struct seq_file m, void v)	5706	static int saved_tgids_show(struct seq_file m, void v)
5650	{	5707	{
5651	int pid = (int *)v - tgid_map;	5708	int entry = (int )v;
		5709	int pid = entry - tgid_map;
		5710	int tgid = *entry;
5652		5711
5653	seq_printf(m, "%d %d\n", pid, trace_find_tgid(pid));	5712	if (tgid == 0)
		5713	return SEQ_SKIP;
		5714
		5715	seq_printf(m, "%d %d\n", pid, tgid);
5654	return 0;	5716	return 0;
5655	}	5717	}
5656		5718
@@ -6135,7 +6197,7 @@ static int __tracing_resize_ring_buffer(struct trace_array *tr, · 478 unmodified lines
6135	ssize_t tracing_resize_ring_buffer(struct trace_array *tr,	6197	ssize_t tracing_resize_ring_buffer(struct trace_array *tr,
6136	unsigned long size, int cpu_id)	6198	unsigned long size, int cpu_id)
6137	{	6199	{
6138	int ret = size;	6200	int ret;
6139		6201
6140	mutex_lock(&trace_types_lock);	6202	mutex_lock(&trace_types_lock);
6141		6203
@@ -7529,6 +7591,91 @@ static const struct file_operations snapshot_raw_fops = { · 1387 unmodified lines
7529		7591
7530	#endif /* CONFIG_TRACER_SNAPSHOT */	7592	#endif /* CONFIG_TRACER_SNAPSHOT */
7531		7593
		7594	/*
		7595	* trace_min_max_write - Write a u64 value to a trace_min_max_param struct
		7596	* @filp: The active open file structure
		7597	* @ubuf: The userspace provided buffer to read value into
		7598	* @cnt: The maximum number of bytes to read
		7599	* @ppos: The current "file" position
		7600	*
		7601	* This function implements the write interface for a struct trace_min_max_param.
		7602	* The filp->private_data must point to a trace_min_max_param structure that
		7603	* defines where to write the value, the min and the max acceptable values,
		7604	* and a lock to protect the write.
		7605	*/
		7606	static ssize_t
		7607	trace_min_max_write(struct file filp, const char __user ubuf, size_t cnt, loff_t *ppos)
		7608	{
		7609	struct trace_min_max_param *param = filp->private_data;
		7610	u64 val;
		7611	int err;
		7612
		7613	if (!param)
		7614	return -EFAULT;
		7615
		7616	err = kstrtoull_from_user(ubuf, cnt, 10, &val);
		7617	if (err)
		7618	return err;
		7619
		7620	if (param->lock)
		7621	mutex_lock(param->lock);
		7622
		7623	if (param->min && val < *param->min)
		7624	err = -EINVAL;
		7625
		7626	if (param->max && val > *param->max)
		7627	err = -EINVAL;
		7628
		7629	if (!err)
		7630	*param->val = val;
		7631
		7632	if (param->lock)
		7633	mutex_unlock(param->lock);
		7634
		7635	if (err)
		7636	return err;
		7637
		7638	return cnt;
		7639	}
		7640
		7641	/*
		7642	* trace_min_max_read - Read a u64 value from a trace_min_max_param struct
		7643	* @filp: The active open file structure
		7644	* @ubuf: The userspace provided buffer to read value into
		7645	* @cnt: The maximum number of bytes to read
		7646	* @ppos: The current "file" position
		7647	*
		7648	* This function implements the read interface for a struct trace_min_max_param.
		7649	* The filp->private_data must point to a trace_min_max_param struct with valid
		7650	* data.
		7651	*/
		7652	static ssize_t
		7653	trace_min_max_read(struct file filp, char __user ubuf, size_t cnt, loff_t *ppos)
		7654	{
		7655	struct trace_min_max_param *param = filp->private_data;
		7656	char buf[U64_STR_SIZE];
		7657	int len;
		7658	u64 val;
		7659
		7660	if (!param)
		7661	return -EFAULT;
		7662
		7663	val = *param->val;
		7664
		7665	if (cnt > sizeof(buf))
		7666	cnt = sizeof(buf);
		7667
		7668	len = snprintf(buf, sizeof(buf), "%llu\n", val);
		7669
		7670	return simple_read_from_buffer(ubuf, cnt, ppos, buf, len);
		7671	}
		7672
		7673	const struct file_operations trace_min_max_fops = {
		7674	.open = tracing_open_generic,
		7675	.read = trace_min_max_read,
		7676	.write = trace_min_max_write,
		7677	};
		7678
7532	#define TRACING_LOG_ERRS_MAX 8	7679	#define TRACING_LOG_ERRS_MAX 8
7533	#define TRACING_LOG_LOC_MAX 128	7680	#define TRACING_LOG_LOC_MAX 128
7534		7681
@@ -9532,6 +9679,8 @@ static __init int tracer_init_tracefs(void) · 1997 unmodified lines
9532	return 0;	9679	return 0;
9533	}	9680	}
9534		9681
		9682	fs_initcall(tracer_init_tracefs);
		9683
9535	static int trace_panic_handler(struct notifier_block *this,	9684	static int trace_panic_handler(struct notifier_block *this,
9536	unsigned long event, void *unused)	9685	unsigned long event, void *unused)
9537	{	9686	{
@@ -9952,7 +10101,7 @@ void __init trace_init(void) · 414 unmodified lines
9952	trace_event_init();	10101	trace_event_init();
9953	}	10102	}
9954		10103
9955	__init static int clear_boot_tracer(void)	10104	__init static void clear_boot_tracer(void)
9956	{	10105	{
9957	/*	10106	/*
9958	* The default tracer at boot buffer is an init section.	10107	* The default tracer at boot buffer is an init section.
@@ -9962,26 +10111,21 @@ __init static int clear_boot_tracer(void) · 3 unmodified lines
9962	* about to be freed.	10111	* about to be freed.
9963	*/	10112	*/
9964	if (!default_bootup_tracer)	10113	if (!default_bootup_tracer)
9965	return 0;	10114	return;
9966		10115
9967	printk(KERN_INFO "ftrace bootup tracer '%s' not registered.\n",	10116	printk(KERN_INFO "ftrace bootup tracer '%s' not registered.\n",
9968	default_bootup_tracer);	10117	default_bootup_tracer);
9969	default_bootup_tracer = NULL;	10118	default_bootup_tracer = NULL;
9970
9971	return 0;
9972	}	10119	}
9973		10120
9974	fs_initcall(tracer_init_tracefs);
9975	late_initcall_sync(clear_boot_tracer);
9976
9977	#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK	10121	#ifdef CONFIG_HAVE_UNSTABLE_SCHED_CLOCK
9978	__init static int tracing_set_default_clock(void)	10122	__init static void tracing_set_default_clock(void)
9979	{	10123	{
9980	/* sched_clock_stable() is determined in late_initcall */	10124	/* sched_clock_stable() is determined in late_initcall */
9981	if (!trace_boot_clock && !sched_clock_stable()) {	10125	if (!trace_boot_clock && !sched_clock_stable()) {
9982	if (security_locked_down(LOCKDOWN_TRACEFS)) {	10126	if (security_locked_down(LOCKDOWN_TRACEFS)) {
9983	pr_warn("Can not set tracing clock due to lockdown\n");	10127	pr_warn("Can not set tracing clock due to lockdown\n");
9984	return -EPERM;	10128	return;
9985	}	10129	}
9986		10130
9987	printk(KERN_WARNING	10131	printk(KERN_WARNING
@@ -9991,8 +10135,21 @@ __init static int tracing_set_default_clock(void) · 3 unmodified lines
9991	"on the kernel command line\n");	10135	"on the kernel command line\n");
9992	tracing_set_clock(&global_trace, "global");	10136	tracing_set_clock(&global_trace, "global");
9993	}	10137	}
		10138	}
		10139	#else
		10140	static inline void tracing_set_default_clock(void) { }
		10141	#endif
		10142
		10143	__init static int late_trace_init(void)
		10144	{
		10145	if (tracepoint_printk && tracepoint_printk_stop_on_boot) {
		10146	static_key_disable(&tracepoint_printk_key.key);
		10147	tracepoint_printk = 0;
		10148	}
9994		10149
		10150	tracing_set_default_clock();
		10151	clear_boot_tracer();
9995	return 0;	10152	return 0;
9996	}	10153	}
9997	late_initcall_sync(tracing_set_default_clock);	10154
9998	#endif	10155	late_initcall_sync(late_trace_init);

kernel/trace/trace.h+29−6

 	TRACE_BLK,
 	TRACE_BPUTS,
 	TRACE_HWLAT,
+	TRACE_OSNOISE,
+	TRACE_TIMERLAT,
 	TRACE_RAW_DATA,
 	TRACE_FUNC_REPEATS,
 	struct array_buffer	max_buffer;
 	bool			allocated_snapshot;
 #endif
-#if defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER)
+#if defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER) \
+	|| defined(CONFIG_OSNOISE_TRACER)
 	unsigned long		max_latency;
 #ifdef CONFIG_FSNOTIFY
 	struct dentry		*d_max_latency;
 		IF_ASSIGN(var, ent, struct bprint_entry, TRACE_BPRINT);	\
 		IF_ASSIGN(var, ent, struct bputs_entry, TRACE_BPUTS);	\
 		IF_ASSIGN(var, ent, struct hwlat_entry, TRACE_HWLAT);	\
+		IF_ASSIGN(var, ent, struct osnoise_entry, TRACE_OSNOISE);\
+		IF_ASSIGN(var, ent, struct timerlat_entry, TRACE_TIMERLAT);\
 		IF_ASSIGN(var, ent, struct raw_data_entry, TRACE_RAW_DATA);\
 		IF_ASSIGN(var, ent, struct trace_mmiotrace_rw,		\
 			  TRACE_MMIO_RW);				\
 			  struct task_struct *tsk, int cpu);
 #endif /* CONFIG_TRACER_MAX_TRACE */
-#if (defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER)) && \
+#if (defined(CONFIG_TRACER_MAX_TRACE) || defined(CONFIG_HWLAT_TRACER) \
-	defined(CONFIG_FSNOTIFY)
+	|| defined(CONFIG_OSNOISE_TRACER)) && defined(CONFIG_FSNOTIFY)
+#define LATENCY_FS_NOTIFY
+#endif
+#ifdef LATENCY_FS_NOTIFY
 void latency_fsnotify(struct trace_array *tr);
 #else
 static inline void latency_fsnotify(struct trace_array *tr) { }
 #endif
 #ifdef CONFIG_STACKTRACE
 	return true;
 }
+/*
+ * This is a generic way to read and write a u64 value from a file in tracefs.
+ *
+ * The value is stored on the variable pointed by *val. The value needs
+ * to be at least *min and at most *max. The write is protected by an
+ * existing *lock.
+ */
+struct trace_min_max_param {
+	struct mutex	*lock;
+	u64		*val;
+	u64		*min;
+	u64		*max;
+};
+#define U64_STR_SIZE		24	/* 20 digits max */
+extern const struct file_operations trace_min_max_fops;
 #endif /* _LINUX_KERNEL_TRACE_H */

kernel/trace/trace_boot.c+25−2

@@ -225,14 +225,37 @@ static void __init · 224 unmodified lines
225	trace_boot_init_events(struct trace_array tr, struct xbc_node node)	225	trace_boot_init_events(struct trace_array tr, struct xbc_node node)
226	{	226	{
227	struct xbc_node gnode, enode;	227	struct xbc_node gnode, enode;
		228	bool enable, enable_all = false;
		229	const char *data;
228		230
229	node = xbc_node_find_child(node, "event");	231	node = xbc_node_find_child(node, "event");
230	if (!node)	232	if (!node)
231	return;	233	return;
232	/* per-event key starts with "event.GROUP.EVENT" */	234	/* per-event key starts with "event.GROUP.EVENT" */
233	xbc_node_for_each_child(node, gnode)	235	xbc_node_for_each_child(node, gnode) {
234	xbc_node_for_each_child(gnode, enode)	236	data = xbc_node_get_data(gnode);
		237	if (!strcmp(data, "enable")) {
		238	enable_all = true;
		239	continue;
		240	}
		241	enable = false;
		242	xbc_node_for_each_child(gnode, enode) {
		243	data = xbc_node_get_data(enode);
		244	if (!strcmp(data, "enable")) {
		245	enable = true;
		246	continue;
		247	}
235	trace_boot_init_one_event(tr, gnode, enode);	248	trace_boot_init_one_event(tr, gnode, enode);
		249	}
		250	/* Event enablement must be done after event settings */
		251	if (enable) {
		252	data = xbc_node_get_data(gnode);
		253	trace_array_set_clr_event(tr, data, NULL, true);
		254	}
		255	}
		256	/* Ditto */
		257	if (enable_all)
		258	trace_array_set_clr_event(tr, NULL, NULL, true);
236	}	259	}
237	#else	260	#else
238	#define trace_boot_enable_events(tr, node) do {} while (0)	261	#define trace_boot_enable_events(tr, node) do {} while (0)

kernel/trace/trace_entries.h+41−0

@@ -360,3 +360,44 @@ FTRACE_ENTRY(func_repeats, func_repeats_entry, · 359 unmodified lines
360	__entry->count,	360	__entry->count,
361	FUNC_REPEATS_GET_DELTA_TS(__entry))	361	FUNC_REPEATS_GET_DELTA_TS(__entry))
362	);	362	);
		363
		364	FTRACE_ENTRY(osnoise, osnoise_entry,
		365
		366	TRACE_OSNOISE,
		367
		368	F_STRUCT(
		369	__field( u64, noise )
		370	__field( u64, runtime )
		371	__field( u64, max_sample )
		372	__field( unsigned int, hw_count )
		373	__field( unsigned int, nmi_count )
		374	__field( unsigned int, irq_count )
		375	__field( unsigned int, softirq_count )
		376	__field( unsigned int, thread_count )
		377	),
		378
		379	F_printk("noise:%llu\tmax_sample:%llu\thw:%u\tnmi:%u\tirq:%u\tsoftirq:%u\tthread:%u\n",
		380	__entry->noise,
		381	__entry->max_sample,
		382	__entry->hw_count,
		383	__entry->nmi_count,
		384	__entry->irq_count,
		385	__entry->softirq_count,
		386	__entry->thread_count)
		387	);
		388
		389	FTRACE_ENTRY(timerlat, timerlat_entry,
		390
		391	TRACE_TIMERLAT,
		392
		393	F_STRUCT(
		394	__field( unsigned int, seqnum )
		395	__field( int, context )
		396	__field( u64, timer_latency )
		397	),
		398
		399	F_printk("seq:%u\tcontext:%d\ttimer_latency:%llu\n",
		400	__entry->seqnum,
		401	__entry->context,
		402	__entry->timer_latency)
		403	);

kernel/trace/trace_events_hist.c+2−1

 		      char *subsys_name, char *event_name, char *field_name)
 {
 	struct trace_array *tr = target_hist_data->event_file->tr;
-	struct hist_field *event_var = ERR_PTR(-EINVAL);
 	struct hist_trigger_data *hist_data;
 	unsigned int i, n, first = true;
 	struct field_var_hist *var_hist;
 	struct trace_event_file *file;
 	struct hist_field *key_field;
+	struct hist_field *event_var;
 	char *saved_filter;
 	char *cmd;
 	int ret;
 		cmd = hist_data->field_var_hists[i]->cmd;
 		ret = event_hist_trigger_func(&trigger_hist_cmd, file,
 					      "!hist", "hist", cmd);
+		WARN_ON_ONCE(ret < 0);
 	}
 }

kernel/trace/trace_events_trigger.c+2−1

@@ -916,7 +916,8 @@ void unpause_named_trigger(struct event_trigger_data *data) · 915 unmodified lines
916		916
917	/**	917	/**
918	* set_named_trigger_data - Associate common named trigger data	918	* set_named_trigger_data - Associate common named trigger data
919	* @data: The trigger data of a named trigger to unpause	919	* @data: The trigger data to associate
		920	* @named_data: The common named trigger to be associated
920	*	921	*
921	* Named triggers are sets of triggers that share a common set of	922	* Named triggers are sets of triggers that share a common set of
922	* trigger data. The first named trigger registered with a given name	923	* trigger data. The first named trigger registered with a given name

kernel/trace/trace_hwlat.c+392−142

@@ -34,7 +34,7 @@ · 33 unmodified lines
34	* Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com>	34	* Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com>
35	* Copyright (C) 2013-2016 Steven Rostedt, Red Hat, Inc. <srostedt@redhat.com>	35	* Copyright (C) 2013-2016 Steven Rostedt, Red Hat, Inc. <srostedt@redhat.com>
36	*	36	*
37	* Includes useful feedback from Clark Williams <clark@redhat.com>	37	* Includes useful feedback from Clark Williams <williams@redhat.com>
38	*	38	*
39	*/	39	*/
40	#include <linux/kthread.h>	40	#include <linux/kthread.h>
@@ -54,20 +54,33 @@ static struct trace_array *hwlat_trace; · 13 unmodified lines
54	#define DEFAULT_SAMPLE_WIDTH 500000 /* 0.5s */	54	#define DEFAULT_SAMPLE_WIDTH 500000 /* 0.5s */
55	#define DEFAULT_LAT_THRESHOLD 10 /* 10us */	55	#define DEFAULT_LAT_THRESHOLD 10 /* 10us */
56		56
57	/* sampling thread*/
58	static struct task_struct *hwlat_kthread;
59
60	static struct dentry hwlat_sample_width; / sample width us */	57	static struct dentry hwlat_sample_width; / sample width us */
61	static struct dentry hwlat_sample_window; / sample window us */	58	static struct dentry hwlat_sample_window; / sample window us */
		59	static struct dentry hwlat_thread_mode; / hwlat thread mode */
		60
		61	enum {
		62	MODE_NONE = 0,
		63	MODE_ROUND_ROBIN,
		64	MODE_PER_CPU,
		65	MODE_MAX
		66	};
		67	static char *thread_mode_str[] = { "none", "round-robin", "per-cpu" };
62		68
63	/* Save the previous tracing_thresh value */	69	/* Save the previous tracing_thresh value */
64	static unsigned long save_tracing_thresh;	70	static unsigned long save_tracing_thresh;
65		71
66	/* NMI timestamp counters */	72	/* runtime kthread data */
67	static u64 nmi_ts_start;	73	struct hwlat_kthread_data {
68	static u64 nmi_total_ts;	74	struct task_struct *kthread;
69	static int nmi_count;	75	/* NMI timestamp counters */
70	static int nmi_cpu;	76	u64 nmi_ts_start;
		77	u64 nmi_total_ts;
		78	int nmi_count;
		79	int nmi_cpu;
		80	};
		81
		82	struct hwlat_kthread_data hwlat_single_cpu_data;
		83	DEFINE_PER_CPU(struct hwlat_kthread_data, hwlat_per_cpu_data);
71		84
72	/* Tells NMIs to call back to the hwlat tracer to record timestamps */	85	/* Tells NMIs to call back to the hwlat tracer to record timestamps */
73	bool trace_hwlat_callback_enabled;	86	bool trace_hwlat_callback_enabled;
@@ -96,11 +109,24 @@ static struct hwlat_data { · 22 unmodified lines
96	u64 sample_window; /* total sampling window (on+off) */	109	u64 sample_window; /* total sampling window (on+off) */
97	u64 sample_width; /* active sampling portion of window */	110	u64 sample_width; /* active sampling portion of window */
98		111
		112	int thread_mode; /* thread mode */
		113
99	} hwlat_data = {	114	} hwlat_data = {
100	.sample_window = DEFAULT_SAMPLE_WINDOW,	115	.sample_window = DEFAULT_SAMPLE_WINDOW,
101	.sample_width = DEFAULT_SAMPLE_WIDTH,	116	.sample_width = DEFAULT_SAMPLE_WIDTH,
		117	.thread_mode = MODE_ROUND_ROBIN
102	};	118	};
103		119
		120	static struct hwlat_kthread_data *get_cpu_data(void)
		121	{
		122	if (hwlat_data.thread_mode == MODE_PER_CPU)
		123	return this_cpu_ptr(&hwlat_per_cpu_data);
		124	else
		125	return &hwlat_single_cpu_data;
		126	}
		127
		128	static bool hwlat_busy;
		129
104	static void trace_hwlat_sample(struct hwlat_sample *sample)	130	static void trace_hwlat_sample(struct hwlat_sample *sample)
105	{	131	{
106	struct trace_array *tr = hwlat_trace;	132	struct trace_array *tr = hwlat_trace;
@@ -136,7 +162,9 @@ static void trace_hwlat_sample(struct hwlat_sample *sample) · 29 unmodified lines
136		162
137	void trace_hwlat_callback(bool enter)	163	void trace_hwlat_callback(bool enter)
138	{	164	{
139	if (smp_processor_id() != nmi_cpu)	165	struct hwlat_kthread_data *kdata = get_cpu_data();
		166
		167	if (!kdata->kthread)
140	return;	168	return;
141		169
142	/*	170	/*
@@ -145,15 +173,24 @@ void trace_hwlat_callback(bool enter) · 2 unmodified lines
145	*/	173	*/
146	if (!IS_ENABLED(CONFIG_GENERIC_SCHED_CLOCK)) {	174	if (!IS_ENABLED(CONFIG_GENERIC_SCHED_CLOCK)) {
147	if (enter)	175	if (enter)
148	nmi_ts_start = time_get();	176	kdata->nmi_ts_start = time_get();
149	else	177	else
150	nmi_total_ts += time_get() - nmi_ts_start;	178	kdata->nmi_total_ts += time_get() - kdata->nmi_ts_start;
151	}	179	}
152		180
153	if (enter)	181	if (enter)
154	nmi_count++;	182	kdata->nmi_count++;
155	}	183	}
156		184
		185	/*
		186	* hwlat_err - report a hwlat error.
		187	*/
		188	#define hwlat_err(msg) ({ \
		189	struct trace_array *tr = hwlat_trace; \
		190	\
		191	trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \
		192	})
		193
157	/**	194	/**
158	* get_sample - sample the CPU TSC and look for likely hardware latencies	195	* get_sample - sample the CPU TSC and look for likely hardware latencies
159	*	196	*
@@ -163,6 +200,7 @@ void trace_hwlat_callback(bool enter) · 3 unmodified lines
163	*/	200	*/
164	static int get_sample(void)	201	static int get_sample(void)
165	{	202	{
		203	struct hwlat_kthread_data *kdata = get_cpu_data();
166	struct trace_array *tr = hwlat_trace;	204	struct trace_array *tr = hwlat_trace;
167	struct hwlat_sample s;	205	struct hwlat_sample s;
168	time_type start, t1, t2, last_t2;	206	time_type start, t1, t2, last_t2;
@@ -175,9 +213,8 @@ static int get_sample(void) · 6 unmodified lines
175		213
176	do_div(thresh, NSEC_PER_USEC); /* modifies interval value */	214	do_div(thresh, NSEC_PER_USEC); /* modifies interval value */
177		215
178	nmi_cpu = smp_processor_id();	216	kdata->nmi_total_ts = 0;
179	nmi_total_ts = 0;	217	kdata->nmi_count = 0;
180	nmi_count = 0;
181	/* Make sure NMIs see this first */	218	/* Make sure NMIs see this first */
182	barrier();	219	barrier();
183		220
@@ -197,7 +234,7 @@ static int get_sample(void) · 13 unmodified lines
197	outer_diff = time_to_us(time_sub(t1, last_t2));	234	outer_diff = time_to_us(time_sub(t1, last_t2));
198	/* This shouldn't happen */	235	/* This shouldn't happen */
199	if (outer_diff < 0) {	236	if (outer_diff < 0) {
200	pr_err(BANNER "time running backwards\n");	237	hwlat_err(BANNER "time running backwards\n");
201	goto out;	238	goto out;
202	}	239	}
203	if (outer_diff > outer_sample)	240	if (outer_diff > outer_sample)
@@ -209,7 +246,7 @@ static int get_sample(void) · 5 unmodified lines
209		246
210	/* Check for possible overflows */	247	/* Check for possible overflows */
211	if (total < last_total) {	248	if (total < last_total) {
212	pr_err("Time total overflowed\n");	249	hwlat_err("Time total overflowed\n");
213	break;	250	break;
214	}	251	}
215	last_total = total;	252	last_total = total;
@@ -225,7 +262,7 @@ static int get_sample(void) · 9 unmodified lines
225		262
226	/* This shouldn't happen */	263	/* This shouldn't happen */
227	if (diff < 0) {	264	if (diff < 0) {
228	pr_err(BANNER "time running backwards\n");	265	hwlat_err(BANNER "time running backwards\n");
229	goto out;	266	goto out;
230	}	267	}
231		268
@@ -247,15 +284,15 @@ static int get_sample(void) · 15 unmodified lines
247	ret = 1;	284	ret = 1;
248		285
249	/* We read in microseconds */	286	/* We read in microseconds */
250	if (nmi_total_ts)	287	if (kdata->nmi_total_ts)
251	do_div(nmi_total_ts, NSEC_PER_USEC);	288	do_div(kdata->nmi_total_ts, NSEC_PER_USEC);
252		289
253	hwlat_data.count++;	290	hwlat_data.count++;
254	s.seqnum = hwlat_data.count;	291	s.seqnum = hwlat_data.count;
255	s.duration = sample;	292	s.duration = sample;
256	s.outer_duration = outer_sample;	293	s.outer_duration = outer_sample;
257	s.nmi_total_ts = nmi_total_ts;	294	s.nmi_total_ts = kdata->nmi_total_ts;
258	s.nmi_count = nmi_count;	295	s.nmi_count = kdata->nmi_count;
259	s.count = count;	296	s.count = count;
260	trace_hwlat_sample(&s);	297	trace_hwlat_sample(&s);
261		298
@@ -273,7 +310,6 @@ out: · 11 unmodified lines
273	}	310	}
274		311
275	static struct cpumask save_cpumask;	312	static struct cpumask save_cpumask;
276	static bool disable_migrate;
277		313
278	static void move_to_next_cpu(void)	314	static void move_to_next_cpu(void)
279	{	315	{
@@ -281,15 +317,13 @@ static void move_to_next_cpu(void) · 1 unmodified line
281	struct trace_array *tr = hwlat_trace;	317	struct trace_array *tr = hwlat_trace;
282	int next_cpu;	318	int next_cpu;
283		319
284	if (disable_migrate)
285	return;
286	/*	320	/*
287	* If for some reason the user modifies the CPU affinity	321	* If for some reason the user modifies the CPU affinity
288	* of this thread, then stop migrating for the duration	322	* of this thread, then stop migrating for the duration
289	* of the current test.	323	* of the current test.
290	*/	324	*/
291	if (!cpumask_equal(current_mask, current->cpus_ptr))	325	if (!cpumask_equal(current_mask, current->cpus_ptr))
292	goto disable;	326	goto change_mode;
293		327
294	get_online_cpus();	328	get_online_cpus();
295	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);	329	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
@@ -300,7 +334,7 @@ static void move_to_next_cpu(void) · 4 unmodified lines
300	next_cpu = cpumask_first(current_mask);	334	next_cpu = cpumask_first(current_mask);
301		335
302	if (next_cpu >= nr_cpu_ids) /* Shouldn't happen! */	336	if (next_cpu >= nr_cpu_ids) /* Shouldn't happen! */
303	goto disable;	337	goto change_mode;
304		338
305	cpumask_clear(current_mask);	339	cpumask_clear(current_mask);
306	cpumask_set_cpu(next_cpu, current_mask);	340	cpumask_set_cpu(next_cpu, current_mask);
@@ -308,8 +342,9 @@ static void move_to_next_cpu(void) · 1 unmodified line
308	sched_setaffinity(0, current_mask);	342	sched_setaffinity(0, current_mask);
309	return;	343	return;
310		344
311	disable:	345	change_mode:
312	disable_migrate = true;	346	hwlat_data.thread_mode = MODE_NONE;
		347	pr_info(BANNER "cpumask changed while in round-robin mode, switching to mode none\n");
313	}	348	}
314		349
315	/*	350	/*
@@ -328,7 +363,8 @@ static int kthread_fn(void *data) · 12 unmodified lines
328		363
329	while (!kthread_should_stop()) {	364	while (!kthread_should_stop()) {
330		365
331	move_to_next_cpu();	366	if (hwlat_data.thread_mode == MODE_ROUND_ROBIN)
		367	move_to_next_cpu();
332		368
333	local_irq_disable();	369	local_irq_disable();
334	get_sample();	370	get_sample();
@@ -351,178 +387,380 @@ static int kthread_fn(void *data) · 16 unmodified lines
351	return 0;	387	return 0;
352	}	388	}
353		389
354	/**	390	/*
355	* start_kthread - Kick off the hardware latency sampling/detector kthread	391	* stop_stop_kthread - Inform the hardware latency sampling/detector kthread to stop
		392	*
		393	* This kicks the running hardware latency sampling/detector kernel thread and
		394	* tells it to stop sampling now. Use this on unload and at system shutdown.
		395	*/
		396	static void stop_single_kthread(void)
		397	{
		398	struct hwlat_kthread_data *kdata = get_cpu_data();
		399	struct task_struct *kthread;
		400
		401	get_online_cpus();
		402	kthread = kdata->kthread;
		403
		404	if (!kthread)
		405	goto out_put_cpus;
		406
		407	kthread_stop(kthread);
		408	kdata->kthread = NULL;
		409
		410	out_put_cpus:
		411	put_online_cpus();
		412	}
		413
		414
		415	/*
		416	* start_single_kthread - Kick off the hardware latency sampling/detector kthread
356	*	417	*
357	* This starts the kernel thread that will sit and sample the CPU timestamp	418	* This starts the kernel thread that will sit and sample the CPU timestamp
358	* counter (TSC or similar) and look for potential hardware latencies.	419	* counter (TSC or similar) and look for potential hardware latencies.
359	*/	420	*/
360	static int start_kthread(struct trace_array *tr)	421	static int start_single_kthread(struct trace_array *tr)
361	{	422	{
		423	struct hwlat_kthread_data *kdata = get_cpu_data();
362	struct cpumask *current_mask = &save_cpumask;	424	struct cpumask *current_mask = &save_cpumask;
363	struct task_struct *kthread;	425	struct task_struct *kthread;
364	int next_cpu;	426	int next_cpu;
365		427
366	if (hwlat_kthread)
367	return 0;
368
369	/* Just pick the first CPU on first iteration */
370	get_online_cpus();	428	get_online_cpus();
371	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);	429	if (kdata->kthread)
372	put_online_cpus();	430	goto out_put_cpus;
373	next_cpu = cpumask_first(current_mask);
374		431
375	kthread = kthread_create(kthread_fn, NULL, "hwlatd");	432	kthread = kthread_create(kthread_fn, NULL, "hwlatd");
376	if (IS_ERR(kthread)) {	433	if (IS_ERR(kthread)) {
377	pr_err(BANNER "could not start sampling thread\n");	434	pr_err(BANNER "could not start sampling thread\n");
		435	put_online_cpus();
378	return -ENOMEM;	436	return -ENOMEM;
379	}	437	}
380		438
381	cpumask_clear(current_mask);	439	/* Just pick the first CPU on first iteration */
382	cpumask_set_cpu(next_cpu, current_mask);	440	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
		441
		442	if (hwlat_data.thread_mode == MODE_ROUND_ROBIN) {
		443	next_cpu = cpumask_first(current_mask);
		444	cpumask_clear(current_mask);
		445	cpumask_set_cpu(next_cpu, current_mask);
		446
		447	}
		448
383	sched_setaffinity(kthread->pid, current_mask);	449	sched_setaffinity(kthread->pid, current_mask);
384		450
385	hwlat_kthread = kthread;	451	kdata->kthread = kthread;
386	wake_up_process(kthread);	452	wake_up_process(kthread);
387		453
		454	out_put_cpus:
		455	put_online_cpus();
388	return 0;	456	return 0;
389	}	457	}
390		458
391	/**	459	/*
392	* stop_kthread - Inform the hardware latency sampling/detector kthread to stop	460	* stop_cpu_kthread - Stop a hwlat cpu kthread
		461	*/
		462	static void stop_cpu_kthread(unsigned int cpu)
		463	{
		464	struct task_struct *kthread;
		465
		466	kthread = per_cpu(hwlat_per_cpu_data, cpu).kthread;
		467	if (kthread)
		468	kthread_stop(kthread);
		469	per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
		470	}
		471
		472	/*
		473	* stop_per_cpu_kthreads - Inform the hardware latency sampling/detector kthread to stop
393	*	474	*
394	* This kicks the running hardware latency sampling/detector kernel thread and	475	* This kicks the running hardware latency sampling/detector kernel threads and
395	* tells it to stop sampling now. Use this on unload and at system shutdown.	476	* tells it to stop sampling now. Use this on unload and at system shutdown.
396	*/	477	*/
397	static void stop_kthread(void)	478	static void stop_per_cpu_kthreads(void)
398	{	479	{
399	if (!hwlat_kthread)	480	unsigned int cpu;
400	return;	481
401	kthread_stop(hwlat_kthread);	482	get_online_cpus();
402	hwlat_kthread = NULL;	483	for_each_online_cpu(cpu)
		484	stop_cpu_kthread(cpu);
		485	put_online_cpus();
403	}	486	}
404		487
405	/*	488	/*
406	* hwlat_read - Wrapper read function for reading both window and width	489	* start_cpu_kthread - Start a hwlat cpu kthread
407	* @filp: The active open file structure
408	* @ubuf: The userspace provided buffer to read value into
409	* @cnt: The maximum number of bytes to read
410	* @ppos: The current "file" position
411	*
412	* This function provides a generic read implementation for the global state
413	* "hwlat_data" structure filesystem entries.
414	*/	490	*/
415	static ssize_t hwlat_read(struct file filp, char __user ubuf,	491	static int start_cpu_kthread(unsigned int cpu)
416	size_t cnt, loff_t *ppos)
417	{	492	{
418	char buf[U64STR_SIZE];	493	struct task_struct *kthread;
419	u64 *entry = filp->private_data;	494	char comm[24];
420	u64 val;
421	int len;
422		495
423	if (!entry)	496	snprintf(comm, 24, "hwlatd/%d", cpu);
424	return -EFAULT;	497
		498	kthread = kthread_create_on_cpu(kthread_fn, NULL, cpu, comm);
		499	if (IS_ERR(kthread)) {
		500	pr_err(BANNER "could not start sampling thread\n");
		501	return -ENOMEM;
		502	}
425		503
426	if (cnt > sizeof(buf))	504	per_cpu(hwlat_per_cpu_data, cpu).kthread = kthread;
427	cnt = sizeof(buf);	505	wake_up_process(kthread);
		506
		507	return 0;
		508	}
		509
		510	#ifdef CONFIG_HOTPLUG_CPU
		511	static void hwlat_hotplug_workfn(struct work_struct *dummy)
		512	{
		513	struct trace_array *tr = hwlat_trace;
		514	unsigned int cpu = smp_processor_id();
		515
		516	mutex_lock(&trace_types_lock);
		517	mutex_lock(&hwlat_data.lock);
		518	get_online_cpus();
		519
		520	if (!hwlat_busy \|\| hwlat_data.thread_mode != MODE_PER_CPU)
		521	goto out_unlock;
428		522
429	val = *entry;	523	if (!cpumask_test_cpu(cpu, tr->tracing_cpumask))
		524	goto out_unlock;
430		525
431	len = snprintf(buf, sizeof(buf), "%llu\n", val);	526	start_cpu_kthread(cpu);
432		527
433	return simple_read_from_buffer(ubuf, cnt, ppos, buf, len);	528	out_unlock:
		529	put_online_cpus();
		530	mutex_unlock(&hwlat_data.lock);
		531	mutex_unlock(&trace_types_lock);
434	}	532	}
435		533
436	/**	534	static DECLARE_WORK(hwlat_hotplug_work, hwlat_hotplug_workfn);
437	* hwlat_width_write - Write function for "width" entry	535
438	* @filp: The active open file structure	536	/*
439	* @ubuf: The user buffer that contains the value to write	537	* hwlat_cpu_init - CPU hotplug online callback function
440	* @cnt: The maximum number of bytes to write to "file"	538	*/
441	* @ppos: The current position in @file	539	static int hwlat_cpu_init(unsigned int cpu)
		540	{
		541	schedule_work_on(cpu, &hwlat_hotplug_work);
		542	return 0;
		543	}
		544
		545	/*
		546	* hwlat_cpu_die - CPU hotplug offline callback function
		547	*/
		548	static int hwlat_cpu_die(unsigned int cpu)
		549	{
		550	stop_cpu_kthread(cpu);
		551	return 0;
		552	}
		553
		554	static void hwlat_init_hotplug_support(void)
		555	{
		556	int ret;
		557
		558	ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "trace/hwlat:online",
		559	hwlat_cpu_init, hwlat_cpu_die);
		560	if (ret < 0)
		561	pr_warn(BANNER "Error to init cpu hotplug support\n");
		562
		563	return;
		564	}
		565	#else /* CONFIG_HOTPLUG_CPU */
		566	static void hwlat_init_hotplug_support(void)
		567	{
		568	return;
		569	}
		570	#endif /* CONFIG_HOTPLUG_CPU */
		571
		572	/*
		573	* start_per_cpu_kthreads - Kick off the hardware latency sampling/detector kthreads
442	*	574	*
443	* This function provides a write implementation for the "width" interface	575	* This starts the kernel threads that will sit on potentially all cpus and
444	* to the hardware latency detector. It can be used to configure	576	* sample the CPU timestamp counter (TSC or similar) and look for potential
445	* for how many us of the total window us we will actively sample for any	577	* hardware latencies.
446	* hardware-induced latency periods. Obviously, it is not possible to
447	* sample constantly and have the system respond to a sample reader, or,
448	* worse, without having the system appear to have gone out to lunch. It
449	* is enforced that width is less that the total window size.
450	*/	578	*/
451	static ssize_t	579	static int start_per_cpu_kthreads(struct trace_array *tr)
452	hwlat_width_write(struct file filp, const char __user ubuf,
453	size_t cnt, loff_t *ppos)
454	{	580	{
455	u64 val;	581	struct cpumask *current_mask = &save_cpumask;
456	int err;	582	unsigned int cpu;
		583	int retval;
457		584
458	err = kstrtoull_from_user(ubuf, cnt, 10, &val);	585	get_online_cpus();
459	if (err)	586	/*
460	return err;	587	* Run only on CPUs in which hwlat is allowed to run.
		588	*/
		589	cpumask_and(current_mask, cpu_online_mask, tr->tracing_cpumask);
		590
		591	for_each_online_cpu(cpu)
		592	per_cpu(hwlat_per_cpu_data, cpu).kthread = NULL;
		593
		594	for_each_cpu(cpu, current_mask) {
		595	retval = start_cpu_kthread(cpu);
		596	if (retval)
		597	goto out_error;
		598	}
		599	put_online_cpus();
		600
		601	return 0;
		602
		603	out_error:
		604	put_online_cpus();
		605	stop_per_cpu_kthreads();
		606	return retval;
		607	}
		608
		609	static void s_mode_start(struct seq_file s, loff_t *pos)
		610	{
		611	int mode = *pos;
461		612
462	mutex_lock(&hwlat_data.lock);	613	mutex_lock(&hwlat_data.lock);
463	if (val < hwlat_data.sample_window)	614
464	hwlat_data.sample_width = val;	615	if (mode >= MODE_MAX)
		616	return NULL;
		617
		618	return pos;
		619	}
		620
		621	static void s_mode_next(struct seq_file s, void v, loff_t pos)
		622	{
		623	int mode = ++(*pos);
		624
		625	if (mode >= MODE_MAX)
		626	return NULL;
		627
		628	return pos;
		629	}
		630
		631	static int s_mode_show(struct seq_file s, void v)
		632	{
		633	loff_t *pos = v;
		634	int mode = *pos;
		635
		636	if (mode == hwlat_data.thread_mode)
		637	seq_printf(s, "[%s]", thread_mode_str[mode]);
465	else	638	else
466	err = -EINVAL;	639	seq_printf(s, "%s", thread_mode_str[mode]);
467	mutex_unlock(&hwlat_data.lock);
468		640
469	if (err)	641	if (mode != MODE_MAX)
470	return err;	642	seq_puts(s, " ");
		643
		644	return 0;
		645	}
471		646
472	return cnt;	647	static void s_mode_stop(struct seq_file s, void v)
		648	{
		649	seq_puts(s, "\n");
		650	mutex_unlock(&hwlat_data.lock);
473	}	651	}
474		652
		653	static const struct seq_operations thread_mode_seq_ops = {
		654	.start = s_mode_start,
		655	.next = s_mode_next,
		656	.show = s_mode_show,
		657	.stop = s_mode_stop
		658	};
		659
		660	static int hwlat_mode_open(struct inode inode, struct file file)
		661	{
		662	return seq_open(file, &thread_mode_seq_ops);
		663	};
		664
		665	static void hwlat_tracer_start(struct trace_array *tr);
		666	static void hwlat_tracer_stop(struct trace_array *tr);
		667
475	/**	668	/**
476	* hwlat_window_write - Write function for "window" entry	669	* hwlat_mode_write - Write function for "mode" entry
477	* @filp: The active open file structure	670	* @filp: The active open file structure
478	* @ubuf: The user buffer that contains the value to write	671	* @ubuf: The user buffer that contains the value to write
479	* @cnt: The maximum number of bytes to write to "file"	672	* @cnt: The maximum number of bytes to write to "file"
480	* @ppos: The current position in @file	673	* @ppos: The current position in @file
481	*	674	*
482	* This function provides a write implementation for the "window" interface	675	* This function provides a write implementation for the "mode" interface
483	* to the hardware latency detector. The window is the total time	676	* to the hardware latency detector. hwlatd has different operation modes.
484	* in us that will be considered one sample period. Conceptually, windows	677	* The "none" sets the allowed cpumask for a single hwlatd thread at the
485	* occur back-to-back and contain a sample width period during which	678	* startup and lets the scheduler handle the migration. The default mode is
486	* actual sampling occurs. Can be used to write a new total window size. It	679	* the "round-robin" one, in which a single hwlatd thread runs, migrating
487	* is enforced that any value written must be greater than the sample width	680	* among the allowed CPUs in a round-robin fashion. The "per-cpu" mode
488	* size, or an error results.	681	* creates one hwlatd thread per allowed CPU.
489	*/	682	*/
490	static ssize_t	683	static ssize_t hwlat_mode_write(struct file filp, const char __user ubuf,
491	hwlat_window_write(struct file filp, const char __user ubuf,	684	size_t cnt, loff_t *ppos)
492	size_t cnt, loff_t *ppos)
493	{	685	{
494	u64 val;	686	struct trace_array *tr = hwlat_trace;
495	int err;	687	const char *mode;
		688	char buf[64];
		689	int ret, i;
496		690
497	err = kstrtoull_from_user(ubuf, cnt, 10, &val);	691	if (cnt >= sizeof(buf))
498	if (err)	692	return -EINVAL;
499	return err;	693
		694	if (copy_from_user(buf, ubuf, cnt))
		695	return -EFAULT;
		696
		697	buf[cnt] = 0;
		698
		699	mode = strstrip(buf);
		700
		701	ret = -EINVAL;
		702
		703	/*
		704	* trace_types_lock is taken to avoid concurrency on start/stop
		705	* and hwlat_busy.
		706	*/
		707	mutex_lock(&trace_types_lock);
		708	if (hwlat_busy)
		709	hwlat_tracer_stop(tr);
500		710
501	mutex_lock(&hwlat_data.lock);	711	mutex_lock(&hwlat_data.lock);
502	if (hwlat_data.sample_width < val)	712
503	hwlat_data.sample_window = val;	713	for (i = 0; i < MODE_MAX; i++) {
504	else	714	if (strcmp(mode, thread_mode_str[i]) == 0) {
505	err = -EINVAL;	715	hwlat_data.thread_mode = i;
		716	ret = cnt;
		717	}
		718	}
		719
506	mutex_unlock(&hwlat_data.lock);	720	mutex_unlock(&hwlat_data.lock);
507		721
508	if (err)	722	if (hwlat_busy)
509	return err;	723	hwlat_tracer_start(tr);
		724	mutex_unlock(&trace_types_lock);
		725
		726	*ppos += cnt;
510		727
511	return cnt;	728
		729
		730	return ret;
512	}	731	}
513		732
514	static const struct file_operations width_fops = {	733	/*
515	.open = tracing_open_generic,	734	* The width parameter is read/write using the generic trace_min_max_param
516	.read = hwlat_read,	735	* method. The *val is protected by the hwlat_data lock and is upper
517	.write = hwlat_width_write,	736	* bounded by the window parameter.
		737	*/
		738	static struct trace_min_max_param hwlat_width = {
		739	.lock = &hwlat_data.lock,
		740	.val = &hwlat_data.sample_width,
		741	.max = &hwlat_data.sample_window,
		742	.min = NULL,
518	};	743	};
519		744
520	static const struct file_operations window_fops = {	745	/*
521	.open = tracing_open_generic,	746	* The window parameter is read/write using the generic trace_min_max_param
522	.read = hwlat_read,	747	* method. The *val is protected by the hwlat_data lock and is lower
523	.write = hwlat_window_write,	748	* bounded by the width parameter.
		749	*/
		750	static struct trace_min_max_param hwlat_window = {
		751	.lock = &hwlat_data.lock,
		752	.val = &hwlat_data.sample_window,
		753	.max = NULL,
		754	.min = &hwlat_data.sample_width,
524	};	755	};
525		756
		757	static const struct file_operations thread_mode_fops = {
		758	.open = hwlat_mode_open,
		759	.read = seq_read,
		760	.llseek = seq_lseek,
		761	.release = seq_release,
		762	.write = hwlat_mode_write
		763	};
526	/**	764	/**
527	* init_tracefs - A function to initialize the tracefs interface files	765	* init_tracefs - A function to initialize the tracefs interface files
528	*	766	*
@@ -546,18 +784,25 @@ static int init_tracefs(void) · 17 unmodified lines
546		784
547	hwlat_sample_window = tracefs_create_file("window", 0640,	785	hwlat_sample_window = tracefs_create_file("window", 0640,
548	top_dir,	786	top_dir,
549	&hwlat_data.sample_window,	787	&hwlat_window,
550	&window_fops);	788	&trace_min_max_fops);
551	if (!hwlat_sample_window)	789	if (!hwlat_sample_window)
552	goto err;	790	goto err;
553		791
554	hwlat_sample_width = tracefs_create_file("width", 0644,	792	hwlat_sample_width = tracefs_create_file("width", 0644,
555	top_dir,	793	top_dir,
556	&hwlat_data.sample_width,	794	&hwlat_width,
557	&width_fops);	795	&trace_min_max_fops);
558	if (!hwlat_sample_width)	796	if (!hwlat_sample_width)
559	goto err;	797	goto err;
560		798
		799	hwlat_thread_mode = trace_create_file("mode", 0644,
		800	top_dir,
		801	NULL,
		802	&thread_mode_fops);
		803	if (!hwlat_thread_mode)
		804	goto err;
		805
561	return 0;	806	return 0;
562		807
563	err:	808	err:
@@ -569,18 +814,22 @@ static void hwlat_tracer_start(struct trace_array *tr) · 5 unmodified lines
569	{	814	{
570	int err;	815	int err;
571		816
572	err = start_kthread(tr);	817	if (hwlat_data.thread_mode == MODE_PER_CPU)
		818	err = start_per_cpu_kthreads(tr);
		819	else
		820	err = start_single_kthread(tr);
573	if (err)	821	if (err)
574	pr_err(BANNER "Cannot start hwlat kthread\n");	822	pr_err(BANNER "Cannot start hwlat kthread\n");
575	}	823	}
576		824
577	static void hwlat_tracer_stop(struct trace_array *tr)	825	static void hwlat_tracer_stop(struct trace_array *tr)
578	{	826	{
579	stop_kthread();	827	if (hwlat_data.thread_mode == MODE_PER_CPU)
		828	stop_per_cpu_kthreads();
		829	else
		830	stop_single_kthread();
580	}	831	}
581		832
582	static bool hwlat_busy;
583
584	static int hwlat_tracer_init(struct trace_array *tr)	833	static int hwlat_tracer_init(struct trace_array *tr)
585	{	834	{
586	/* Only allow one instance to enable this */	835	/* Only allow one instance to enable this */
@@ -589,7 +838,6 @@ static int hwlat_tracer_init(struct trace_array *tr) · 2 unmodified lines
589		838
590	hwlat_trace = tr;	839	hwlat_trace = tr;
591		840
592	disable_migrate = false;
593	hwlat_data.count = 0;	841	hwlat_data.count = 0;
594	tr->max_latency = 0;	842	tr->max_latency = 0;
595	save_tracing_thresh = tracing_thresh;	843	save_tracing_thresh = tracing_thresh;
@@ -608,7 +856,7 @@ static int hwlat_tracer_init(struct trace_array *tr) · 12 unmodified lines
608		856
609	static void hwlat_tracer_reset(struct trace_array *tr)	857	static void hwlat_tracer_reset(struct trace_array *tr)
610	{	858	{
611	stop_kthread();	859	hwlat_tracer_stop(tr);
612		860
613	/* the tracing threshold is static between runs */	861	/* the tracing threshold is static between runs */
614	last_tracing_thresh = tracing_thresh;	862	last_tracing_thresh = tracing_thresh;
@@ -637,6 +885,8 @@ __init static int init_hwlat_tracer(void) · 22 unmodified lines
637	if (ret)	885	if (ret)
638	return ret;	886	return ret;
639		887
		888	hwlat_init_hotplug_support();
		889
640	init_tracefs();	890	init_tracefs();
641		891
642	return 0;	892	return 0;

kernel/trace/trace_osnoise.c+2059−0

@@ -0,0 +1,2059 @@
	1	// SPDX-License-Identifier: GPL-2.0
	2	/*
	3	* OS Noise Tracer: computes the OS Noise suffered by a running thread.
	4	* Timerlat Tracer: measures the wakeup latency of a timer triggered IRQ and thread.
	5	*
	6	* Based on "hwlat_detector" tracer by:
	7	* Copyright (C) 2008-2009 Jon Masters, Red Hat, Inc. <jcm@redhat.com>
	8	* Copyright (C) 2013-2016 Steven Rostedt, Red Hat, Inc. <srostedt@redhat.com>
	9	* With feedback from Clark Williams <williams@redhat.com>
	10	*
	11	* And also based on the rtsl tracer presented on:
	12	* DE OLIVEIRA, Daniel Bristot, et al. Demystifying the real-time linux
	13	* scheduling latency. In: 32nd Euromicro Conference on Real-Time Systems
	14	* (ECRTS 2020). Schloss Dagstuhl-Leibniz-Zentrum fur Informatik, 2020.
	15	*
	16	* Copyright (C) 2021 Daniel Bristot de Oliveira, Red Hat, Inc. <bristot@redhat.com>
	17	*/
	18
	19	#include <linux/kthread.h>
	20	#include <linux/tracefs.h>
	21	#include <linux/uaccess.h>
	22	#include <linux/cpumask.h>
	23	#include <linux/delay.h>
	24	#include <linux/sched/clock.h>
	25	#include <uapi/linux/sched/types.h>
	26	#include <linux/sched.h>
	27	#include "trace.h"
	28
	29	#ifdef CONFIG_X86_LOCAL_APIC
	30	#include <asm/trace/irq_vectors.h>
	31	#undef TRACE_INCLUDE_PATH
	32	#undef TRACE_INCLUDE_FILE
	33	#endif /* CONFIG_X86_LOCAL_APIC */
	34
	35	#include <trace/events/irq.h>
	36	#include <trace/events/sched.h>
	37
	38	#define CREATE_TRACE_POINTS
	39	#include <trace/events/osnoise.h>
	40
	41	static struct trace_array *osnoise_trace;
	42
	43	/*
	44	* Default values.
	45	*/
	46	#define BANNER "osnoise: "
	47	#define DEFAULT_SAMPLE_PERIOD 1000000 /* 1s */
	48	#define DEFAULT_SAMPLE_RUNTIME 1000000 /* 1s */
	49
	50	#define DEFAULT_TIMERLAT_PERIOD 1000 /* 1ms */
	51	#define DEFAULT_TIMERLAT_PRIO 95 /* FIFO 95 */
	52
	53	/*
	54	* NMI runtime info.
	55	*/
	56	struct osn_nmi {
	57	u64 count;
	58	u64 delta_start;
	59	};
	60
	61	/*
	62	* IRQ runtime info.
	63	*/
	64	struct osn_irq {
	65	u64 count;
	66	u64 arrival_time;
	67	u64 delta_start;
	68	};
	69
	70	#define IRQ_CONTEXT 0
	71	#define THREAD_CONTEXT 1
	72	/*
	73	* sofirq runtime info.
	74	*/
	75	struct osn_softirq {
	76	u64 count;
	77	u64 arrival_time;
	78	u64 delta_start;
	79	};
	80
	81	/*
	82	* thread runtime info.
	83	*/
	84	struct osn_thread {
	85	u64 count;
	86	u64 arrival_time;
	87	u64 delta_start;
	88	};
	89
	90	/*
	91	* Runtime information: this structure saves the runtime information used by
	92	* one sampling thread.
	93	*/
	94	struct osnoise_variables {
	95	struct task_struct *kthread;
	96	bool sampling;
	97	pid_t pid;
	98	struct osn_nmi nmi;
	99	struct osn_irq irq;
	100	struct osn_softirq softirq;
	101	struct osn_thread thread;
	102	local_t int_counter;
	103	};
	104
	105	/*
	106	* Per-cpu runtime information.
	107	*/
	108	DEFINE_PER_CPU(struct osnoise_variables, per_cpu_osnoise_var);
	109
	110	/*
	111	* this_cpu_osn_var - Return the per-cpu osnoise_variables on its relative CPU
	112	*/
	113	static inline struct osnoise_variables *this_cpu_osn_var(void)
	114	{
	115	return this_cpu_ptr(&per_cpu_osnoise_var);
	116	}
	117
	118	#ifdef CONFIG_TIMERLAT_TRACER
	119	/*
	120	* Runtime information for the timer mode.
	121	*/
	122	struct timerlat_variables {
	123	struct task_struct *kthread;
	124	struct hrtimer timer;
	125	u64 rel_period;
	126	u64 abs_period;
	127	bool tracing_thread;
	128	u64 count;
	129	};
	130
	131	DEFINE_PER_CPU(struct timerlat_variables, per_cpu_timerlat_var);
	132
	133	/*
	134	* this_cpu_tmr_var - Return the per-cpu timerlat_variables on its relative CPU
	135	*/
	136	static inline struct timerlat_variables *this_cpu_tmr_var(void)
	137	{
	138	return this_cpu_ptr(&per_cpu_timerlat_var);
	139	}
	140
	141	/*
	142	* tlat_var_reset - Reset the values of the given timerlat_variables
	143	*/
	144	static inline void tlat_var_reset(void)
	145	{
	146	struct timerlat_variables *tlat_var;
	147	int cpu;
	148	/*
	149	* So far, all the values are initialized as 0, so
	150	* zeroing the structure is perfect.
	151	*/
	152	for_each_cpu(cpu, cpu_online_mask) {
	153	tlat_var = per_cpu_ptr(&per_cpu_timerlat_var, cpu);
	154	memset(tlat_var, 0, sizeof(*tlat_var));
	155	}
	156	}
	157	#else /* CONFIG_TIMERLAT_TRACER */
	158	#define tlat_var_reset() do {} while (0)
	159	#endif /* CONFIG_TIMERLAT_TRACER */
	160
	161	/*
	162	* osn_var_reset - Reset the values of the given osnoise_variables
	163	*/
	164	static inline void osn_var_reset(void)
	165	{
	166	struct osnoise_variables *osn_var;
	167	int cpu;
	168
	169	/*
	170	* So far, all the values are initialized as 0, so
	171	* zeroing the structure is perfect.
	172	*/
	173	for_each_cpu(cpu, cpu_online_mask) {
	174	osn_var = per_cpu_ptr(&per_cpu_osnoise_var, cpu);
	175	memset(osn_var, 0, sizeof(*osn_var));
	176	}
	177	}
	178
	179	/*
	180	* osn_var_reset_all - Reset the value of all per-cpu osnoise_variables
	181	*/
	182	static inline void osn_var_reset_all(void)
	183	{
	184	osn_var_reset();
	185	tlat_var_reset();
	186	}
	187
	188	/*
	189	* Tells NMIs to call back to the osnoise tracer to record timestamps.
	190	*/
	191	bool trace_osnoise_callback_enabled;
	192
	193	/*
	194	* osnoise sample structure definition. Used to store the statistics of a
	195	* sample run.
	196	*/
	197	struct osnoise_sample {
	198	u64 runtime; /* runtime */
	199	u64 noise; /* noise */
	200	u64 max_sample; /* max single noise sample */
	201	int hw_count; /* # HW (incl. hypervisor) interference */
	202	int nmi_count; /* # NMIs during this sample */
	203	int irq_count; /* # IRQs during this sample */
	204	int softirq_count; /* # softirqs during this sample */
	205	int thread_count; /* # threads during this sample */
	206	};
	207
	208	#ifdef CONFIG_TIMERLAT_TRACER
	209	/*
	210	* timerlat sample structure definition. Used to store the statistics of
	211	* a sample run.
	212	*/
	213	struct timerlat_sample {
	214	u64 timer_latency; /* timer_latency */
	215	unsigned int seqnum; /* unique sequence */
	216	int context; /* timer context */
	217	};
	218	#endif
	219
	220	/*
	221	* Protect the interface.
	222	*/
	223	struct mutex interface_lock;
	224
	225	/*
	226	* Tracer data.
	227	*/
	228	static struct osnoise_data {
	229	u64 sample_period; /* total sampling period */
	230	u64 sample_runtime; /* active sampling portion of period */
	231	u64 stop_tracing; /* stop trace in the internal operation (loop/irq) */
	232	u64 stop_tracing_total; /* stop trace in the final operation (report/thread) */
	233	#ifdef CONFIG_TIMERLAT_TRACER
	234	u64 timerlat_period; /* timerlat period */
	235	u64 print_stack; /* print IRQ stack if total > */
	236	int timerlat_tracer; /* timerlat tracer */
	237	#endif
	238	bool tainted; /* infor users and developers about a problem */
	239	} osnoise_data = {
	240	.sample_period = DEFAULT_SAMPLE_PERIOD,
	241	.sample_runtime = DEFAULT_SAMPLE_RUNTIME,
	242	.stop_tracing = 0,
	243	.stop_tracing_total = 0,
	244	#ifdef CONFIG_TIMERLAT_TRACER
	245	.print_stack = 0,
	246	.timerlat_period = DEFAULT_TIMERLAT_PERIOD,
	247	.timerlat_tracer = 0,
	248	#endif
	249	};
	250
	251	/*
	252	* Boolean variable used to inform that the tracer is currently sampling.
	253	*/
	254	static bool osnoise_busy;
	255
	256	/*
	257	* Print the osnoise header info.
	258	*/
	259	static void print_osnoise_headers(struct seq_file *s)
	260	{
	261	if (osnoise_data.tainted)
	262	seq_puts(s, "# osnoise is tainted!\n");
	263
	264	seq_puts(s, "# _-----=> irqs-off\n");
	265	seq_puts(s, "# / _----=> need-resched\n");
	266	seq_puts(s, "# \| / _---=> hardirq/softirq\n");
	267	seq_puts(s, "# \|\| / _--=> preempt-depth ");
	268	seq_puts(s, " MAX\n");
	269
	270	seq_puts(s, "# \|\| / ");
	271	seq_puts(s, " SINGLE Interference counters:\n");
	272
	273	seq_puts(s, "# \|\|\|\| RUNTIME ");
	274	seq_puts(s, " NOISE %% OF CPU NOISE +-----------------------------+\n");
	275
	276	seq_puts(s, "# TASK-PID CPU# \|\|\|\| TIMESTAMP IN US ");
	277	seq_puts(s, " IN US AVAILABLE IN US HW NMI IRQ SIRQ THREAD\n");
	278
	279	seq_puts(s, "# \| \| \| \|\|\|\| \| \| ");
	280	seq_puts(s, " \| \| \| \| \| \| \| \|\n");
	281	}
	282
	283	/*
	284	* osnoise_taint - report an osnoise error.
	285	*/
	286	#define osnoise_taint(msg) ({ \
	287	struct trace_array *tr = osnoise_trace; \
	288	\
	289	trace_array_printk_buf(tr->array_buffer.buffer, _THIS_IP_, msg); \
	290	osnoise_data.tainted = true; \
	291	})
	292
	293	/*
	294	* Record an osnoise_sample into the tracer buffer.
	295	*/
	296	static void trace_osnoise_sample(struct osnoise_sample *sample)
	297	{
	298	struct trace_array *tr = osnoise_trace;
	299	struct trace_buffer *buffer = tr->array_buffer.buffer;
	300	struct trace_event_call *call = &event_osnoise;
	301	struct ring_buffer_event *event;
	302	struct osnoise_entry *entry;
	303
	304	event = trace_buffer_lock_reserve(buffer, TRACE_OSNOISE, sizeof(*entry),
	305	tracing_gen_ctx());
	306	if (!event)
	307	return;
	308	entry = ring_buffer_event_data(event);
	309	entry->runtime = sample->runtime;
	310	entry->noise = sample->noise;
	311	entry->max_sample = sample->max_sample;
	312	entry->hw_count = sample->hw_count;
	313	entry->nmi_count = sample->nmi_count;
	314	entry->irq_count = sample->irq_count;
	315	entry->softirq_count = sample->softirq_count;
	316	entry->thread_count = sample->thread_count;
	317
	318	if (!call_filter_check_discard(call, entry, buffer, event))
	319	trace_buffer_unlock_commit_nostack(buffer, event);
	320	}
	321
	322	#ifdef CONFIG_TIMERLAT_TRACER
	323	/*
	324	* Print the timerlat header info.
	325	*/
	326	static void print_timerlat_headers(struct seq_file *s)
	327	{
	328	seq_puts(s, "# _-----=> irqs-off\n");
	329	seq_puts(s, "# / _----=> need-resched\n");
	330	seq_puts(s, "# \| / _---=> hardirq/softirq\n");
	331	seq_puts(s, "# \|\| / _--=> preempt-depth\n");
	332	seq_puts(s, "# \|\| /\n");
	333	seq_puts(s, "# \|\|\|\| ACTIVATION\n");
	334	seq_puts(s, "# TASK-PID CPU# \|\|\|\| TIMESTAMP ID ");
	335	seq_puts(s, " CONTEXT LATENCY\n");
	336	seq_puts(s, "# \| \| \| \|\|\|\| \| \| ");
	337	seq_puts(s, " \| \|\n");
	338	}
	339
	340	/*
	341	* Record an timerlat_sample into the tracer buffer.
	342	*/
	343	static void trace_timerlat_sample(struct timerlat_sample *sample)
	344	{
	345	struct trace_array *tr = osnoise_trace;
	346	struct trace_event_call *call = &event_osnoise;
	347	struct trace_buffer *buffer = tr->array_buffer.buffer;
	348	struct ring_buffer_event *event;
	349	struct timerlat_entry *entry;
	350
	351	event = trace_buffer_lock_reserve(buffer, TRACE_TIMERLAT, sizeof(*entry),
	352	tracing_gen_ctx());
	353	if (!event)
	354	return;
	355	entry = ring_buffer_event_data(event);
	356	entry->seqnum = sample->seqnum;
	357	entry->context = sample->context;
	358	entry->timer_latency = sample->timer_latency;
	359
	360	if (!call_filter_check_discard(call, entry, buffer, event))
	361	trace_buffer_unlock_commit_nostack(buffer, event);
	362	}
	363
	364	#ifdef CONFIG_STACKTRACE
	365
	366	#define MAX_CALLS 256
	367
	368	/*
	369	* Stack trace will take place only at IRQ level, so, no need
	370	* to control nesting here.
	371	*/
	372	struct trace_stack {
	373	int stack_size;
	374	int nr_entries;
	375	unsigned long calls[MAX_CALLS];
	376	};
	377
	378	static DEFINE_PER_CPU(struct trace_stack, trace_stack);
	379
	380	/*
	381	* timerlat_save_stack - save a stack trace without printing
	382	*
	383	* Save the current stack trace without printing. The
	384	* stack will be printed later, after the end of the measurement.
	385	*/
	386	static void timerlat_save_stack(int skip)
	387	{
	388	unsigned int size, nr_entries;
	389	struct trace_stack *fstack;
	390
	391	fstack = this_cpu_ptr(&trace_stack);
	392
	393	size = ARRAY_SIZE(fstack->calls);
	394
	395	nr_entries = stack_trace_save(fstack->calls, size, skip);
	396
	397	fstack->stack_size = nr_entries * sizeof(unsigned long);
	398	fstack->nr_entries = nr_entries;
	399
	400	return;
	401
	402	}
	403	/*
	404	* timerlat_dump_stack - dump a stack trace previously saved
	405	*
	406	* Dump a saved stack trace into the trace buffer.
	407	*/
	408	static void timerlat_dump_stack(void)
	409	{
	410	struct trace_event_call *call = &event_osnoise;
	411	struct trace_array *tr = osnoise_trace;
	412	struct trace_buffer *buffer = tr->array_buffer.buffer;
	413	struct ring_buffer_event *event;
	414	struct trace_stack *fstack;
	415	struct stack_entry *entry;
	416	unsigned int size;
	417
	418	preempt_disable_notrace();
	419	fstack = this_cpu_ptr(&trace_stack);
	420	size = fstack->stack_size;
	421
	422	event = trace_buffer_lock_reserve(buffer, TRACE_STACK, sizeof(*entry) + size,
	423	tracing_gen_ctx());
	424	if (!event)
	425	goto out;
	426
	427	entry = ring_buffer_event_data(event);
	428
	429	memcpy(&entry->caller, fstack->calls, size);
	430	entry->size = fstack->nr_entries;
	431
	432	if (!call_filter_check_discard(call, entry, buffer, event))
	433	trace_buffer_unlock_commit_nostack(buffer, event);
	434
	435	out:
	436	preempt_enable_notrace();
	437	}
	438	#else
	439	#define timerlat_dump_stack() do {} while (0)
	440	#define timerlat_save_stack(a) do {} while (0)
	441	#endif /* CONFIG_STACKTRACE */
	442	#endif /* CONFIG_TIMERLAT_TRACER */
	443
	444	/*
	445	* Macros to encapsulate the time capturing infrastructure.
	446	*/
	447	#define time_get() trace_clock_local()
	448	#define time_to_us(x) div_u64(x, 1000)
	449	#define time_sub(a, b) ((a) - (b))
	450
	451	/*
	452	* cond_move_irq_delta_start - Forward the delta_start of a running IRQ
	453	*
	454	* If an IRQ is preempted by an NMI, its delta_start is pushed forward
	455	* to discount the NMI interference.
	456	*
	457	* See get_int_safe_duration().
	458	*/
	459	static inline void
	460	cond_move_irq_delta_start(struct osnoise_variables *osn_var, u64 duration)
	461	{
	462	if (osn_var->irq.delta_start)
	463	osn_var->irq.delta_start += duration;
	464	}
	465
	466	#ifndef CONFIG_PREEMPT_RT
	467	/*
	468	* cond_move_softirq_delta_start - Forward the delta_start of a running softirq.
	469	*
	470	* If a softirq is preempted by an IRQ or NMI, its delta_start is pushed
	471	* forward to discount the interference.
	472	*
	473	* See get_int_safe_duration().
	474	*/
	475	static inline void
	476	cond_move_softirq_delta_start(struct osnoise_variables *osn_var, u64 duration)
	477	{
	478	if (osn_var->softirq.delta_start)
	479	osn_var->softirq.delta_start += duration;
	480	}
	481	#else /* CONFIG_PREEMPT_RT */
	482	#define cond_move_softirq_delta_start(osn_var, duration) do {} while (0)
	483	#endif
	484
	485	/*
	486	* cond_move_thread_delta_start - Forward the delta_start of a running thread
	487	*
	488	* If a noisy thread is preempted by an softirq, IRQ or NMI, its delta_start
	489	* is pushed forward to discount the interference.
	490	*
	491	* See get_int_safe_duration().
	492	*/
	493	static inline void
	494	cond_move_thread_delta_start(struct osnoise_variables *osn_var, u64 duration)
	495	{
	496	if (osn_var->thread.delta_start)
	497	osn_var->thread.delta_start += duration;
	498	}
	499
	500	/*
	501	* get_int_safe_duration - Get the duration of a window
	502	*
	503	* The irq, softirq and thread varaibles need to have its duration without
	504	* the interference from higher priority interrupts. Instead of keeping a
	505	* variable to discount the interrupt interference from these variables, the
	506	* starting time of these variables are pushed forward with the interrupt's
	507	* duration. In this way, a single variable is used to:
	508	*
	509	* - Know if a given window is being measured.
	510	* - Account its duration.
	511	* - Discount the interference.
	512	*
	513	* To avoid getting inconsistent values, e.g.,:
	514	*
	515	* now = time_get()
	516	* ---> interrupt!
	517	* delta_start -= int duration;
	518	* <---
	519	* duration = now - delta_start;
	520	*
	521	* result: negative duration if the variable duration before the
	522	* interrupt was smaller than the interrupt execution.
	523	*
	524	* A counter of interrupts is used. If the counter increased, try
	525	* to capture an interference safe duration.
	526	*/
	527	static inline s64
	528	get_int_safe_duration(struct osnoise_variables osn_var, u64 delta_start)
	529	{
	530	u64 int_counter, now;
	531	s64 duration;
	532
	533	do {
	534	int_counter = local_read(&osn_var->int_counter);
	535	/* synchronize with interrupts */
	536	barrier();
	537
	538	now = time_get();
	539	duration = (now - *delta_start);
	540
	541	/* synchronize with interrupts */
	542	barrier();
	543	} while (int_counter != local_read(&osn_var->int_counter));
	544
	545	/*
	546	* This is an evidence of race conditions that cause
	547	* a value to be "discounted" too much.
	548	*/
	549	if (duration < 0)
	550	osnoise_taint("Negative duration!\n");
	551
	552	*delta_start = 0;
	553
	554	return duration;
	555	}
	556
	557	/*
	558	*
	559	* set_int_safe_time - Save the current time on *time, aware of interference
	560	*
	561	* Get the time, taking into consideration a possible interference from
	562	* higher priority interrupts.
	563	*
	564	* See get_int_safe_duration() for an explanation.
	565	*/
	566	static u64
	567	set_int_safe_time(struct osnoise_variables osn_var, u64 time)
	568	{
	569	u64 int_counter;
	570
	571	do {
	572	int_counter = local_read(&osn_var->int_counter);
	573	/* synchronize with interrupts */
	574	barrier();
	575
	576	*time = time_get();
	577
	578	/* synchronize with interrupts */
	579	barrier();
	580	} while (int_counter != local_read(&osn_var->int_counter));
	581
	582	return int_counter;
	583	}
	584
	585	#ifdef CONFIG_TIMERLAT_TRACER
	586	/*
	587	* copy_int_safe_time - Copy src into desc aware of interference
	588	*/
	589	static u64
	590	copy_int_safe_time(struct osnoise_variables osn_var, u64 dst, u64 *src)
	591	{
	592	u64 int_counter;
	593
	594	do {
	595	int_counter = local_read(&osn_var->int_counter);
	596	/* synchronize with interrupts */
	597	barrier();
	598
	599	dst = src;
	600
	601	/* synchronize with interrupts */
	602	barrier();
	603	} while (int_counter != local_read(&osn_var->int_counter));
	604
	605	return int_counter;
	606	}
	607	#endif /* CONFIG_TIMERLAT_TRACER */
	608
	609	/*
	610	* trace_osnoise_callback - NMI entry/exit callback
	611	*
	612	* This function is called at the entry and exit NMI code. The bool enter
	613	* distinguishes between either case. This function is used to note a NMI
	614	* occurrence, compute the noise caused by the NMI, and to remove the noise
	615	* it is potentially causing on other interference variables.
	616	*/
	617	void trace_osnoise_callback(bool enter)
	618	{
	619	struct osnoise_variables *osn_var = this_cpu_osn_var();
	620	u64 duration;
	621
	622	if (!osn_var->sampling)
	623	return;
	624
	625	/*
	626	* Currently trace_clock_local() calls sched_clock() and the
	627	* generic version is not NMI safe.
	628	*/
	629	if (!IS_ENABLED(CONFIG_GENERIC_SCHED_CLOCK)) {
	630	if (enter) {
	631	osn_var->nmi.delta_start = time_get();
	632	local_inc(&osn_var->int_counter);
	633	} else {
	634	duration = time_get() - osn_var->nmi.delta_start;
	635
	636	trace_nmi_noise(osn_var->nmi.delta_start, duration);
	637
	638	cond_move_irq_delta_start(osn_var, duration);
	639	cond_move_softirq_delta_start(osn_var, duration);
	640	cond_move_thread_delta_start(osn_var, duration);
	641	}
	642	}
	643
	644	if (enter)
	645	osn_var->nmi.count++;
	646	}
	647
	648	/*
	649	* osnoise_trace_irq_entry - Note the starting of an IRQ
	650	*
	651	* Save the starting time of an IRQ. As IRQs are non-preemptive to other IRQs,
	652	* it is safe to use a single variable (ons_var->irq) to save the statistics.
	653	* The arrival_time is used to report... the arrival time. The delta_start
	654	* is used to compute the duration at the IRQ exit handler. See
	655	* cond_move_irq_delta_start().
	656	*/
	657	void osnoise_trace_irq_entry(int id)
	658	{
	659	struct osnoise_variables *osn_var = this_cpu_osn_var();
	660
	661	if (!osn_var->sampling)
	662	return;
	663	/*
	664	* This value will be used in the report, but not to compute
	665	* the execution time, so it is safe to get it unsafe.
	666	*/
	667	osn_var->irq.arrival_time = time_get();
	668	set_int_safe_time(osn_var, &osn_var->irq.delta_start);
	669	osn_var->irq.count++;
	670
	671	local_inc(&osn_var->int_counter);
	672	}
	673
	674	/*
	675	* osnoise_irq_exit - Note the end of an IRQ, sava data and trace
	676	*
	677	* Computes the duration of the IRQ noise, and trace it. Also discounts the
	678	* interference from other sources of noise could be currently being accounted.
	679	*/
	680	void osnoise_trace_irq_exit(int id, const char *desc)
	681	{
	682	struct osnoise_variables *osn_var = this_cpu_osn_var();
	683	int duration;
	684
	685	if (!osn_var->sampling)
	686	return;
	687
	688	duration = get_int_safe_duration(osn_var, &osn_var->irq.delta_start);
	689	trace_irq_noise(id, desc, osn_var->irq.arrival_time, duration);
	690	osn_var->irq.arrival_time = 0;
	691	cond_move_softirq_delta_start(osn_var, duration);
	692	cond_move_thread_delta_start(osn_var, duration);
	693	}
	694
	695	/*
	696	* trace_irqentry_callback - Callback to the irq:irq_entry traceevent
	697	*
	698	* Used to note the starting of an IRQ occurece.
	699	*/
	700	static void trace_irqentry_callback(void *data, int irq,
	701	struct irqaction *action)
	702	{
	703	osnoise_trace_irq_entry(irq);
	704	}
	705
	706	/*
	707	* trace_irqexit_callback - Callback to the irq:irq_exit traceevent
	708	*
	709	* Used to note the end of an IRQ occurece.
	710	*/
	711	static void trace_irqexit_callback(void *data, int irq,
	712	struct irqaction *action, int ret)
	713	{
	714	osnoise_trace_irq_exit(irq, action->name);
	715	}
	716
	717	/*
	718	* arch specific register function.
	719	*/
	720	int __weak osnoise_arch_register(void)
	721	{
	722	return 0;
	723	}
	724
	725	/*
	726	* arch specific unregister function.
	727	*/
	728	void __weak osnoise_arch_unregister(void)
	729	{
	730	return;
	731	}
	732
	733	/*
	734	* hook_irq_events - Hook IRQ handling events
	735	*
	736	* This function hooks the IRQ related callbacks to the respective trace
	737	* events.
	738	*/
	739	static int hook_irq_events(void)
	740	{
	741	int ret;
	742
	743	ret = register_trace_irq_handler_entry(trace_irqentry_callback, NULL);
	744	if (ret)
	745	goto out_err;
	746
	747	ret = register_trace_irq_handler_exit(trace_irqexit_callback, NULL);
	748	if (ret)
	749	goto out_unregister_entry;
	750
	751	ret = osnoise_arch_register();
	752	if (ret)
	753	goto out_irq_exit;
	754
	755	return 0;
	756
	757	out_irq_exit:
	758	unregister_trace_irq_handler_exit(trace_irqexit_callback, NULL);
	759	out_unregister_entry:
	760	unregister_trace_irq_handler_entry(trace_irqentry_callback, NULL);
	761	out_err:
	762	return -EINVAL;
	763	}
	764
	765	/*
	766	* unhook_irq_events - Unhook IRQ handling events
	767	*
	768	* This function unhooks the IRQ related callbacks to the respective trace
	769	* events.
	770	*/
	771	static void unhook_irq_events(void)
	772	{
	773	osnoise_arch_unregister();
	774	unregister_trace_irq_handler_exit(trace_irqexit_callback, NULL);
	775	unregister_trace_irq_handler_entry(trace_irqentry_callback, NULL);
	776	}
	777
	778	#ifndef CONFIG_PREEMPT_RT
	779	/*
	780	* trace_softirq_entry_callback - Note the starting of a softirq
	781	*
	782	* Save the starting time of a softirq. As softirqs are non-preemptive to
	783	* other softirqs, it is safe to use a single variable (ons_var->softirq)
	784	* to save the statistics. The arrival_time is used to report... the
	785	* arrival time. The delta_start is used to compute the duration at the
	786	* softirq exit handler. See cond_move_softirq_delta_start().
	787	*/
	788	static void trace_softirq_entry_callback(void *data, unsigned int vec_nr)
	789	{
	790	struct osnoise_variables *osn_var = this_cpu_osn_var();
	791
	792	if (!osn_var->sampling)
	793	return;
	794	/*
	795	* This value will be used in the report, but not to compute
	796	* the execution time, so it is safe to get it unsafe.
	797	*/
	798	osn_var->softirq.arrival_time = time_get();
	799	set_int_safe_time(osn_var, &osn_var->softirq.delta_start);
	800	osn_var->softirq.count++;
	801
	802	local_inc(&osn_var->int_counter);
	803	}
	804
	805	/*
	806	* trace_softirq_exit_callback - Note the end of an softirq
	807	*
	808	* Computes the duration of the softirq noise, and trace it. Also discounts the
	809	* interference from other sources of noise could be currently being accounted.
	810	*/
	811	static void trace_softirq_exit_callback(void *data, unsigned int vec_nr)
	812	{
	813	struct osnoise_variables *osn_var = this_cpu_osn_var();
	814	int duration;
	815
	816	if (!osn_var->sampling)
	817	return;
	818
	819	#ifdef CONFIG_TIMERLAT_TRACER
	820	/*
	821	* If the timerlat is enabled, but the irq handler did
	822	* not run yet enabling timerlat_tracer, do not trace.
	823	*/
	824	if (unlikely(osnoise_data.timerlat_tracer)) {
	825	struct timerlat_variables *tlat_var;
	826	tlat_var = this_cpu_tmr_var();
	827	if (!tlat_var->tracing_thread) {
	828	osn_var->softirq.arrival_time = 0;
	829	osn_var->softirq.delta_start = 0;
	830	return;
	831	}
	832	}
	833	#endif
	834
	835	duration = get_int_safe_duration(osn_var, &osn_var->softirq.delta_start);
	836	trace_softirq_noise(vec_nr, osn_var->softirq.arrival_time, duration);
	837	cond_move_thread_delta_start(osn_var, duration);
	838	osn_var->softirq.arrival_time = 0;
	839	}
	840
	841	/*
	842	* hook_softirq_events - Hook softirq handling events
	843	*
	844	* This function hooks the softirq related callbacks to the respective trace
	845	* events.
	846	*/
	847	static int hook_softirq_events(void)
	848	{
	849	int ret;
	850
	851	ret = register_trace_softirq_entry(trace_softirq_entry_callback, NULL);
	852	if (ret)
	853	goto out_err;
	854
	855	ret = register_trace_softirq_exit(trace_softirq_exit_callback, NULL);
	856	if (ret)
	857	goto out_unreg_entry;
	858
	859	return 0;
	860
	861	out_unreg_entry:
	862	unregister_trace_softirq_entry(trace_softirq_entry_callback, NULL);
	863	out_err:
	864	return -EINVAL;
	865	}
	866
	867	/*
	868	* unhook_softirq_events - Unhook softirq handling events
	869	*
	870	* This function hooks the softirq related callbacks to the respective trace
	871	* events.
	872	*/
	873	static void unhook_softirq_events(void)
	874	{
	875	unregister_trace_softirq_entry(trace_softirq_entry_callback, NULL);
	876	unregister_trace_softirq_exit(trace_softirq_exit_callback, NULL);
	877	}
	878	#else /* CONFIG_PREEMPT_RT */
	879	/*
	880	* softirq are threads on the PREEMPT_RT mode.
	881	*/
	882	static int hook_softirq_events(void)
	883	{
	884	return 0;
	885	}
	886	static void unhook_softirq_events(void)
	887	{
	888	}
	889	#endif
	890
	891	/*
	892	* thread_entry - Record the starting of a thread noise window
	893	*
	894	* It saves the context switch time for a noisy thread, and increments
	895	* the interference counters.
	896	*/
	897	static void
	898	thread_entry(struct osnoise_variables osn_var, struct task_struct t)
	899	{
	900	if (!osn_var->sampling)
	901	return;
	902	/*
	903	* The arrival time will be used in the report, but not to compute
	904	* the execution time, so it is safe to get it unsafe.
	905	*/
	906	osn_var->thread.arrival_time = time_get();
	907
	908	set_int_safe_time(osn_var, &osn_var->thread.delta_start);
	909
	910	osn_var->thread.count++;
	911	local_inc(&osn_var->int_counter);
	912	}
	913
	914	/*
	915	* thread_exit - Report the end of a thread noise window
	916	*
	917	* It computes the total noise from a thread, tracing if needed.
	918	*/
	919	static void
	920	thread_exit(struct osnoise_variables osn_var, struct task_struct t)
	921	{
	922	int duration;
	923
	924	if (!osn_var->sampling)
	925	return;
	926
	927	#ifdef CONFIG_TIMERLAT_TRACER
	928	if (osnoise_data.timerlat_tracer) {
	929	struct timerlat_variables *tlat_var;
	930	tlat_var = this_cpu_tmr_var();
	931	if (!tlat_var->tracing_thread) {
	932	osn_var->thread.delta_start = 0;
	933	osn_var->thread.arrival_time = 0;
	934	return;
	935	}
	936	}
	937	#endif
	938
	939	duration = get_int_safe_duration(osn_var, &osn_var->thread.delta_start);
	940
	941	trace_thread_noise(t, osn_var->thread.arrival_time, duration);
	942
	943	osn_var->thread.arrival_time = 0;
	944	}
	945
	946	/*
	947	* trace_sched_switch - sched:sched_switch trace event handler
	948	*
	949	* This function is hooked to the sched:sched_switch trace event, and it is
	950	* used to record the beginning and to report the end of a thread noise window.
	951	*/
	952	static void
	953	trace_sched_switch_callback(void data, bool preempt, struct task_struct p,
	954	struct task_struct *n)
	955	{
	956	struct osnoise_variables *osn_var = this_cpu_osn_var();
	957
	958	if (p->pid != osn_var->pid)
	959	thread_exit(osn_var, p);
	960
	961	if (n->pid != osn_var->pid)
	962	thread_entry(osn_var, n);
	963	}
	964
	965	/*
	966	* hook_thread_events - Hook the insturmentation for thread noise
	967	*
	968	* Hook the osnoise tracer callbacks to handle the noise from other
	969	* threads on the necessary kernel events.
	970	*/
	971	static int hook_thread_events(void)
	972	{
	973	int ret;
	974
	975	ret = register_trace_sched_switch(trace_sched_switch_callback, NULL);
	976	if (ret)
	977	return -EINVAL;
	978
	979	return 0;
	980	}
	981
	982	/*
	983	* unhook_thread_events - *nhook the insturmentation for thread noise
	984	*
	985	* Unook the osnoise tracer callbacks to handle the noise from other
	986	* threads on the necessary kernel events.
	987	*/
	988	static void unhook_thread_events(void)
	989	{
	990	unregister_trace_sched_switch(trace_sched_switch_callback, NULL);
	991	}
	992
	993	/*
	994	* save_osn_sample_stats - Save the osnoise_sample statistics
	995	*
	996	* Save the osnoise_sample statistics before the sampling phase. These
	997	* values will be used later to compute the diff betwneen the statistics
	998	* before and after the osnoise sampling.
	999	*/
	1000	static void
	1001	save_osn_sample_stats(struct osnoise_variables osn_var, struct osnoise_sample s)
	1002	{
	1003	s->nmi_count = osn_var->nmi.count;
	1004	s->irq_count = osn_var->irq.count;
	1005	s->softirq_count = osn_var->softirq.count;
	1006	s->thread_count = osn_var->thread.count;
	1007	}
	1008
	1009	/*
	1010	* diff_osn_sample_stats - Compute the osnoise_sample statistics
	1011	*
	1012	* After a sample period, compute the difference on the osnoise_sample
	1013	* statistics. The struct osnoise_sample *s contains the statistics saved via
	1014	* save_osn_sample_stats() before the osnoise sampling.
	1015	*/
	1016	static void
	1017	diff_osn_sample_stats(struct osnoise_variables osn_var, struct osnoise_sample s)
	1018	{
	1019	s->nmi_count = osn_var->nmi.count - s->nmi_count;
	1020	s->irq_count = osn_var->irq.count - s->irq_count;
	1021	s->softirq_count = osn_var->softirq.count - s->softirq_count;
	1022	s->thread_count = osn_var->thread.count - s->thread_count;
	1023	}
	1024
	1025	/*
	1026	* osnoise_stop_tracing - Stop tracing and the tracer.
	1027	*/
	1028	static void osnoise_stop_tracing(void)
	1029	{
	1030	struct trace_array *tr = osnoise_trace;
	1031	tracer_tracing_off(tr);
	1032	}
	1033
	1034	/*
	1035	* run_osnoise - Sample the time and look for osnoise
	1036	*
	1037	* Used to capture the time, looking for potential osnoise latency repeatedly.
	1038	* Different from hwlat_detector, it is called with preemption and interrupts
	1039	* enabled. This allows irqs, softirqs and threads to run, interfering on the
	1040	* osnoise sampling thread, as they would do with a regular thread.
	1041	*/
	1042	static int run_osnoise(void)
	1043	{
	1044	struct osnoise_variables *osn_var = this_cpu_osn_var();
	1045	struct trace_array *tr = osnoise_trace;
	1046	u64 start, sample, last_sample;
	1047	u64 last_int_count, int_count;
	1048	s64 noise = 0, max_noise = 0;
	1049	s64 total, last_total = 0;
	1050	struct osnoise_sample s;
	1051	unsigned int threshold;
	1052	u64 runtime, stop_in;
	1053	u64 sum_noise = 0;
	1054	int hw_count = 0;
	1055	int ret = -1;
	1056
	1057	/*
	1058	* Considers the current thread as the workload.
	1059	*/
	1060	osn_var->pid = current->pid;
	1061
	1062	/*
	1063	* Save the current stats for the diff
	1064	*/
	1065	save_osn_sample_stats(osn_var, &s);
	1066
	1067	/*
	1068	* if threshold is 0, use the default value of 5 us.
	1069	*/
	1070	threshold = tracing_thresh ? : 5000;
	1071
	1072	/*
	1073	* Make sure NMIs see sampling first
	1074	*/
	1075	osn_var->sampling = true;
	1076	barrier();
	1077
	1078	/*
	1079	* Transform the *_us config to nanoseconds to avoid the
	1080	* division on the main loop.
	1081	*/
	1082	runtime = osnoise_data.sample_runtime * NSEC_PER_USEC;
	1083	stop_in = osnoise_data.stop_tracing * NSEC_PER_USEC;
	1084
	1085	/*
	1086	* Start timestemp
	1087	*/
	1088	start = time_get();
	1089
	1090	/*
	1091	* "previous" loop.
	1092	*/
	1093	last_int_count = set_int_safe_time(osn_var, &last_sample);
	1094
	1095	do {
	1096	/*
	1097	* Get sample!
	1098	*/
	1099	int_count = set_int_safe_time(osn_var, &sample);
	1100
	1101	noise = time_sub(sample, last_sample);
	1102
	1103	/*
	1104	* This shouldn't happen.
	1105	*/
	1106	if (noise < 0) {
	1107	osnoise_taint("negative noise!");
	1108	goto out;
	1109	}
	1110
	1111	/*
	1112	* Sample runtime.
	1113	*/
	1114	total = time_sub(sample, start);
	1115
	1116	/*
	1117	* Check for possible overflows.
	1118	*/
	1119	if (total < last_total) {
	1120	osnoise_taint("total overflow!");
	1121	break;
	1122	}
	1123
	1124	last_total = total;
	1125
	1126	if (noise >= threshold) {
	1127	int interference = int_count - last_int_count;
	1128
	1129	if (noise > max_noise)
	1130	max_noise = noise;
	1131
	1132	if (!interference)
	1133	hw_count++;
	1134
	1135	sum_noise += noise;
	1136
	1137	trace_sample_threshold(last_sample, noise, interference);
	1138
	1139	if (osnoise_data.stop_tracing)
	1140	if (noise > stop_in)
	1141	osnoise_stop_tracing();
	1142	}
	1143
	1144	/*
	1145	* For the non-preemptive kernel config: let threads runs, if
	1146	* they so wish.
	1147	*/
	1148	cond_resched();
	1149
	1150	last_sample = sample;
	1151	last_int_count = int_count;
	1152
	1153	} while (total < runtime && !kthread_should_stop());
	1154
	1155	/*
	1156	* Finish the above in the view for interrupts.
	1157	*/
	1158	barrier();
	1159
	1160	osn_var->sampling = false;
	1161
	1162	/*
	1163	* Make sure sampling data is no longer updated.
	1164	*/
	1165	barrier();
	1166
	1167	/*
	1168	* Save noise info.
	1169	*/
	1170	s.noise = time_to_us(sum_noise);
	1171	s.runtime = time_to_us(total);
	1172	s.max_sample = time_to_us(max_noise);
	1173	s.hw_count = hw_count;
	1174
	1175	/* Save interference stats info */
	1176	diff_osn_sample_stats(osn_var, &s);
	1177
	1178	trace_osnoise_sample(&s);
	1179
	1180	/* Keep a running maximum ever recorded osnoise "latency" */
	1181	if (max_noise > tr->max_latency) {
	1182	tr->max_latency = max_noise;
	1183	latency_fsnotify(tr);
	1184	}
	1185
	1186	if (osnoise_data.stop_tracing_total)
	1187	if (s.noise > osnoise_data.stop_tracing_total)
	1188	osnoise_stop_tracing();
	1189
	1190	return 0;
	1191	out:
	1192	return ret;
	1193	}
	1194
	1195	static struct cpumask osnoise_cpumask;
	1196	static struct cpumask save_cpumask;
	1197
	1198	/*
	1199	* osnoise_main - The osnoise detection kernel thread
	1200	*
	1201	* Calls run_osnoise() function to measure the osnoise for the configured runtime,
	1202	* every period.
	1203	*/
	1204	static int osnoise_main(void *data)
	1205	{
	1206	u64 interval;
	1207
	1208	while (!kthread_should_stop()) {
	1209
	1210	run_osnoise();
	1211
	1212	mutex_lock(&interface_lock);
	1213	interval = osnoise_data.sample_period - osnoise_data.sample_runtime;
	1214	mutex_unlock(&interface_lock);
	1215
	1216	do_div(interval, USEC_PER_MSEC);
	1217
	1218	/*
	1219	* differently from hwlat_detector, the osnoise tracer can run
	1220	* without a pause because preemption is on.
	1221	*/
	1222	if (interval < 1) {
	1223	/* Let synchronize_rcu_tasks() make progress */
	1224	cond_resched_tasks_rcu_qs();
	1225	continue;
	1226	}
	1227
	1228	if (msleep_interruptible(interval))
	1229	break;
	1230	}
	1231
	1232	return 0;
	1233	}
	1234
	1235	#ifdef CONFIG_TIMERLAT_TRACER
	1236	/*
	1237	* timerlat_irq - hrtimer handler for timerlat.
	1238	*/
	1239	static enum hrtimer_restart timerlat_irq(struct hrtimer *timer)
	1240	{
	1241	struct osnoise_variables *osn_var = this_cpu_osn_var();
	1242	struct trace_array *tr = osnoise_trace;
	1243	struct timerlat_variables *tlat;
	1244	struct timerlat_sample s;
	1245	u64 now;
	1246	u64 diff;
	1247
	1248	/*
	1249	* I am not sure if the timer was armed for this CPU. So, get
	1250	* the timerlat struct from the timer itself, not from this
	1251	* CPU.
	1252	*/
	1253	tlat = container_of(timer, struct timerlat_variables, timer);
	1254
	1255	now = ktime_to_ns(hrtimer_cb_get_time(&tlat->timer));
	1256
	1257	/*
	1258	* Enable the osnoise: events for thread an softirq.
	1259	*/
	1260	tlat->tracing_thread = true;
	1261
	1262	osn_var->thread.arrival_time = time_get();
	1263
	1264	/*
	1265	* A hardirq is running: the timer IRQ. It is for sure preempting
	1266	* a thread, and potentially preempting a softirq.
	1267	*
	1268	* At this point, it is not interesting to know the duration of the
	1269	* preempted thread (and maybe softirq), but how much time they will
	1270	* delay the beginning of the execution of the timer thread.
	1271	*
	1272	* To get the correct (net) delay added by the softirq, its delta_start
	1273	* is set as the IRQ one. In this way, at the return of the IRQ, the delta
	1274	* start of the sofitrq will be zeroed, accounting then only the time
	1275	* after that.
	1276	*
	1277	* The thread follows the same principle. However, if a softirq is
	1278	* running, the thread needs to receive the softirq delta_start. The
	1279	* reason being is that the softirq will be the last to be unfolded,
	1280	* resseting the thread delay to zero.
	1281	*/
	1282	#ifndef CONFIG_PREEMPT_RT
	1283	if (osn_var->softirq.delta_start) {
	1284	copy_int_safe_time(osn_var, &osn_var->thread.delta_start,
	1285	&osn_var->softirq.delta_start);
	1286
	1287	copy_int_safe_time(osn_var, &osn_var->softirq.delta_start,
	1288	&osn_var->irq.delta_start);
	1289	} else {
	1290	copy_int_safe_time(osn_var, &osn_var->thread.delta_start,
	1291	&osn_var->irq.delta_start);
	1292	}
	1293	#else /* CONFIG_PREEMPT_RT */
	1294	/*
	1295	* The sofirqs run as threads on RT, so there is not need
	1296	* to keep track of it.
	1297	*/
	1298	copy_int_safe_time(osn_var, &osn_var->thread.delta_start, &osn_var->irq.delta_start);
	1299	#endif /* CONFIG_PREEMPT_RT */
	1300
	1301	/*
	1302	* Compute the current time with the expected time.
	1303	*/
	1304	diff = now - tlat->abs_period;
	1305
	1306	tlat->count++;
	1307	s.seqnum = tlat->count;
	1308	s.timer_latency = diff;
	1309	s.context = IRQ_CONTEXT;
	1310
	1311	trace_timerlat_sample(&s);
	1312
	1313	/* Keep a running maximum ever recorded os noise "latency" */
	1314	if (diff > tr->max_latency) {
	1315	tr->max_latency = diff;
	1316	latency_fsnotify(tr);
	1317	}
	1318
	1319	if (osnoise_data.stop_tracing)
	1320	if (time_to_us(diff) >= osnoise_data.stop_tracing)
	1321	osnoise_stop_tracing();
	1322
	1323	wake_up_process(tlat->kthread);
	1324
	1325	if (osnoise_data.print_stack)
	1326	timerlat_save_stack(0);
	1327
	1328	return HRTIMER_NORESTART;
	1329	}
	1330
	1331	/*
	1332	* wait_next_period - Wait for the next period for timerlat
	1333	*/
	1334	static int wait_next_period(struct timerlat_variables *tlat)
	1335	{
	1336	ktime_t next_abs_period, now;
	1337	u64 rel_period = osnoise_data.timerlat_period * 1000;
	1338
	1339	now = hrtimer_cb_get_time(&tlat->timer);
	1340	next_abs_period = ns_to_ktime(tlat->abs_period + rel_period);
	1341
	1342	/*
	1343	* Save the next abs_period.
	1344	*/
	1345	tlat->abs_period = (u64) ktime_to_ns(next_abs_period);
	1346
	1347	/*
	1348	* If the new abs_period is in the past, skip the activation.
	1349	*/
	1350	while (ktime_compare(now, next_abs_period) > 0) {
	1351	next_abs_period = ns_to_ktime(tlat->abs_period + rel_period);
	1352	tlat->abs_period = (u64) ktime_to_ns(next_abs_period);
	1353	}
	1354
	1355	set_current_state(TASK_INTERRUPTIBLE);
	1356
	1357	hrtimer_start(&tlat->timer, next_abs_period, HRTIMER_MODE_ABS_PINNED_HARD);
	1358	schedule();
	1359	return 1;
	1360	}
	1361
	1362	/*
	1363	* timerlat_main- Timerlat main
	1364	*/
	1365	static int timerlat_main(void *data)
	1366	{
	1367	struct osnoise_variables *osn_var = this_cpu_osn_var();
	1368	struct timerlat_variables *tlat = this_cpu_tmr_var();
	1369	struct timerlat_sample s;
	1370	struct sched_param sp;
	1371	u64 now, diff;
	1372
	1373	/*
	1374	* Make the thread RT, that is how cyclictest is usually used.
	1375	*/
	1376	sp.sched_priority = DEFAULT_TIMERLAT_PRIO;
	1377	sched_setscheduler_nocheck(current, SCHED_FIFO, &sp);
	1378
	1379	tlat->count = 0;
	1380	tlat->tracing_thread = false;
	1381
	1382	hrtimer_init(&tlat->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_HARD);
	1383	tlat->timer.function = timerlat_irq;
	1384	tlat->kthread = current;
	1385	osn_var->pid = current->pid;
	1386	/*
	1387	* Anotate the arrival time.
	1388	*/
	1389	tlat->abs_period = hrtimer_cb_get_time(&tlat->timer);
	1390
	1391	wait_next_period(tlat);
	1392
	1393	osn_var->sampling = 1;
	1394
	1395	while (!kthread_should_stop()) {
	1396	now = ktime_to_ns(hrtimer_cb_get_time(&tlat->timer));
	1397	diff = now - tlat->abs_period;
	1398
	1399	s.seqnum = tlat->count;
	1400	s.timer_latency = diff;
	1401	s.context = THREAD_CONTEXT;
	1402
	1403	trace_timerlat_sample(&s);
	1404
	1405	#ifdef CONFIG_STACKTRACE
	1406	if (osnoise_data.print_stack)
	1407	if (osnoise_data.print_stack <= time_to_us(diff))
	1408	timerlat_dump_stack();
	1409	#endif /* CONFIG_STACKTRACE */
	1410
	1411	tlat->tracing_thread = false;
	1412	if (osnoise_data.stop_tracing_total)
	1413	if (time_to_us(diff) >= osnoise_data.stop_tracing_total)
	1414	osnoise_stop_tracing();
	1415
	1416	wait_next_period(tlat);
	1417	}
	1418
	1419	hrtimer_cancel(&tlat->timer);
	1420	return 0;
	1421	}
	1422	#endif /* CONFIG_TIMERLAT_TRACER */
	1423
	1424	/*
	1425	* stop_kthread - stop a workload thread
	1426	*/
	1427	static void stop_kthread(unsigned int cpu)
	1428	{
	1429	struct task_struct *kthread;
	1430
	1431	kthread = per_cpu(per_cpu_osnoise_var, cpu).kthread;
	1432	if (kthread)
	1433	kthread_stop(kthread);
	1434	per_cpu(per_cpu_osnoise_var, cpu).kthread = NULL;
	1435	}
	1436
	1437	/*
	1438	* stop_per_cpu_kthread - Stop per-cpu threads
	1439	*
	1440	* Stop the osnoise sampling htread. Use this on unload and at system
	1441	* shutdown.
	1442	*/
	1443	static void stop_per_cpu_kthreads(void)
	1444	{
	1445	int cpu;
	1446
	1447	get_online_cpus();
	1448
	1449	for_each_online_cpu(cpu)
	1450	stop_kthread(cpu);
	1451
	1452	put_online_cpus();
	1453	}
	1454
	1455	/*
	1456	* start_kthread - Start a workload tread
	1457	*/
	1458	static int start_kthread(unsigned int cpu)
	1459	{
	1460	struct task_struct *kthread;
	1461	void *main = osnoise_main;
	1462	char comm[24];
	1463
	1464	#ifdef CONFIG_TIMERLAT_TRACER
	1465	if (osnoise_data.timerlat_tracer) {
	1466	snprintf(comm, 24, "timerlat/%d", cpu);
	1467	main = timerlat_main;
	1468	} else {
	1469	snprintf(comm, 24, "osnoise/%d", cpu);
	1470	}
	1471	#else
	1472	snprintf(comm, 24, "osnoise/%d", cpu);
	1473	#endif
	1474	kthread = kthread_create_on_cpu(main, NULL, cpu, comm);
	1475
	1476	if (IS_ERR(kthread)) {
	1477	pr_err(BANNER "could not start sampling thread\n");
	1478	stop_per_cpu_kthreads();
	1479	return -ENOMEM;
	1480	}
	1481
	1482	per_cpu(per_cpu_osnoise_var, cpu).kthread = kthread;
	1483	wake_up_process(kthread);
	1484
	1485	return 0;
	1486	}
	1487
	1488	/*
	1489	* start_per_cpu_kthread - Kick off per-cpu osnoise sampling kthreads
	1490	*
	1491	* This starts the kernel thread that will look for osnoise on many
	1492	* cpus.
	1493	*/
	1494	static int start_per_cpu_kthreads(struct trace_array *tr)
	1495	{
	1496	struct cpumask *current_mask = &save_cpumask;
	1497	int retval;
	1498	int cpu;
	1499
	1500	get_online_cpus();
	1501	/*
	1502	* Run only on CPUs in which trace and osnoise are allowed to run.
	1503	*/
	1504	cpumask_and(current_mask, tr->tracing_cpumask, &osnoise_cpumask);
	1505	/*
	1506	* And the CPU is online.
	1507	*/
	1508	cpumask_and(current_mask, cpu_online_mask, current_mask);
	1509
	1510	for_each_possible_cpu(cpu)
	1511	per_cpu(per_cpu_osnoise_var, cpu).kthread = NULL;
	1512
	1513	for_each_cpu(cpu, current_mask) {
	1514	retval = start_kthread(cpu);
	1515	if (retval) {
	1516	stop_per_cpu_kthreads();
	1517	return retval;
	1518	}
	1519	}
	1520
	1521	put_online_cpus();
	1522
	1523	return 0;
	1524	}
	1525
	1526	#ifdef CONFIG_HOTPLUG_CPU
	1527	static void osnoise_hotplug_workfn(struct work_struct *dummy)
	1528	{
	1529	struct trace_array *tr = osnoise_trace;
	1530	unsigned int cpu = smp_processor_id();
	1531
	1532
	1533	mutex_lock(&trace_types_lock);
	1534
	1535	if (!osnoise_busy)
	1536	goto out_unlock_trace;
	1537
	1538	mutex_lock(&interface_lock);
	1539	get_online_cpus();
	1540
	1541	if (!cpumask_test_cpu(cpu, &osnoise_cpumask))
	1542	goto out_unlock;
	1543
	1544	if (!cpumask_test_cpu(cpu, tr->tracing_cpumask))
	1545	goto out_unlock;
	1546
	1547	start_kthread(cpu);
	1548
	1549	out_unlock:
	1550	put_online_cpus();
	1551	mutex_unlock(&interface_lock);
	1552	out_unlock_trace:
	1553	mutex_unlock(&trace_types_lock);
	1554	}
	1555
	1556	static DECLARE_WORK(osnoise_hotplug_work, osnoise_hotplug_workfn);
	1557
	1558	/*
	1559	* osnoise_cpu_init - CPU hotplug online callback function
	1560	*/
	1561	static int osnoise_cpu_init(unsigned int cpu)
	1562	{
	1563	schedule_work_on(cpu, &osnoise_hotplug_work);
	1564	return 0;
	1565	}
	1566
	1567	/*
	1568	* osnoise_cpu_die - CPU hotplug offline callback function
	1569	*/
	1570	static int osnoise_cpu_die(unsigned int cpu)
	1571	{
	1572	stop_kthread(cpu);
	1573	return 0;
	1574	}
	1575
	1576	static void osnoise_init_hotplug_support(void)
	1577	{
	1578	int ret;
	1579
	1580	ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "trace/osnoise:online",
	1581	osnoise_cpu_init, osnoise_cpu_die);
	1582	if (ret < 0)
	1583	pr_warn(BANNER "Error to init cpu hotplug support\n");
	1584
	1585	return;
	1586	}
	1587	#else /* CONFIG_HOTPLUG_CPU */
	1588	static void osnoise_init_hotplug_support(void)
	1589	{
	1590	return;
	1591	}
	1592	#endif /* CONFIG_HOTPLUG_CPU */
	1593
	1594	/*
	1595	* osnoise_cpus_read - Read function for reading the "cpus" file
	1596	* @filp: The active open file structure
	1597	* @ubuf: The userspace provided buffer to read value into
	1598	* @cnt: The maximum number of bytes to read
	1599	* @ppos: The current "file" position
	1600	*
	1601	* Prints the "cpus" output into the user-provided buffer.
	1602	*/
	1603	static ssize_t
	1604	osnoise_cpus_read(struct file filp, char __user ubuf, size_t count,
	1605	loff_t *ppos)
	1606	{
	1607	char *mask_str;
	1608	int len;
	1609
	1610	mutex_lock(&interface_lock);
	1611
	1612	len = snprintf(NULL, 0, "%*pbl\n", cpumask_pr_args(&osnoise_cpumask)) + 1;
	1613	mask_str = kmalloc(len, GFP_KERNEL);
	1614	if (!mask_str) {
	1615	count = -ENOMEM;
	1616	goto out_unlock;
	1617	}
	1618
	1619	len = snprintf(mask_str, len, "%*pbl\n", cpumask_pr_args(&osnoise_cpumask));
	1620	if (len >= count) {
	1621	count = -EINVAL;
	1622	goto out_free;
	1623	}
	1624
	1625	count = simple_read_from_buffer(ubuf, count, ppos, mask_str, len);
	1626
	1627	out_free:
	1628	kfree(mask_str);
	1629	out_unlock:
	1630	mutex_unlock(&interface_lock);
	1631
	1632	return count;
	1633	}
	1634
	1635	static void osnoise_tracer_start(struct trace_array *tr);
	1636	static void osnoise_tracer_stop(struct trace_array *tr);
	1637
	1638	/*
	1639	* osnoise_cpus_write - Write function for "cpus" entry
	1640	* @filp: The active open file structure
	1641	* @ubuf: The user buffer that contains the value to write
	1642	* @cnt: The maximum number of bytes to write to "file"
	1643	* @ppos: The current position in @file
	1644	*
	1645	* This function provides a write implementation for the "cpus"
	1646	* interface to the osnoise trace. By default, it lists all CPUs,
	1647	* in this way, allowing osnoise threads to run on any online CPU
	1648	* of the system. It serves to restrict the execution of osnoise to the
	1649	* set of CPUs writing via this interface. Note that osnoise also
	1650	* respects the "tracing_cpumask." Hence, osnoise threads will run only
	1651	* on the set of CPUs allowed here AND on "tracing_cpumask." Why not
	1652	* have just "tracing_cpumask?" Because the user might be interested
	1653	* in tracing what is running on other CPUs. For instance, one might
	1654	* run osnoise in one HT CPU while observing what is running on the
	1655	* sibling HT CPU.
	1656	*/
	1657	static ssize_t
	1658	osnoise_cpus_write(struct file filp, const char __user ubuf, size_t count,
	1659	loff_t *ppos)
	1660	{
	1661	struct trace_array *tr = osnoise_trace;
	1662	cpumask_var_t osnoise_cpumask_new;
	1663	int running, err;
	1664	char buf[256];
	1665
	1666	if (count >= 256)
	1667	return -EINVAL;
	1668
	1669	if (copy_from_user(buf, ubuf, count))
	1670	return -EFAULT;
	1671
	1672	if (!zalloc_cpumask_var(&osnoise_cpumask_new, GFP_KERNEL))
	1673	return -ENOMEM;
	1674
	1675	err = cpulist_parse(buf, osnoise_cpumask_new);
	1676	if (err)
	1677	goto err_free;
	1678
	1679	/*
	1680	* trace_types_lock is taken to avoid concurrency on start/stop
	1681	* and osnoise_busy.
	1682	*/
	1683	mutex_lock(&trace_types_lock);
	1684	running = osnoise_busy;
	1685	if (running)
	1686	osnoise_tracer_stop(tr);
	1687
	1688	mutex_lock(&interface_lock);
	1689	/*
	1690	* osnoise_cpumask is read by CPU hotplug operations.
	1691	*/
	1692	get_online_cpus();
	1693
	1694	cpumask_copy(&osnoise_cpumask, osnoise_cpumask_new);
	1695
	1696	put_online_cpus();
	1697	mutex_unlock(&interface_lock);
	1698
	1699	if (running)
	1700	osnoise_tracer_start(tr);
	1701	mutex_unlock(&trace_types_lock);
	1702
	1703	free_cpumask_var(osnoise_cpumask_new);
	1704	return count;
	1705
	1706	err_free:
	1707	free_cpumask_var(osnoise_cpumask_new);
	1708
	1709	return err;
	1710	}
	1711
	1712	/*
	1713	* osnoise/runtime_us: cannot be greater than the period.
	1714	*/
	1715	static struct trace_min_max_param osnoise_runtime = {
	1716	.lock = &interface_lock,
	1717	.val = &osnoise_data.sample_runtime,
	1718	.max = &osnoise_data.sample_period,
	1719	.min = NULL,
	1720	};
	1721
	1722	/*
	1723	* osnoise/period_us: cannot be smaller than the runtime.
	1724	*/
	1725	static struct trace_min_max_param osnoise_period = {
	1726	.lock = &interface_lock,
	1727	.val = &osnoise_data.sample_period,
	1728	.max = NULL,
	1729	.min = &osnoise_data.sample_runtime,
	1730	};
	1731
	1732	/*
	1733	* osnoise/stop_tracing_us: no limit.
	1734	*/
	1735	static struct trace_min_max_param osnoise_stop_tracing_in = {
	1736	.lock = &interface_lock,
	1737	.val = &osnoise_data.stop_tracing,
	1738	.max = NULL,
	1739	.min = NULL,
	1740	};
	1741
	1742	/*
	1743	* osnoise/stop_tracing_total_us: no limit.
	1744	*/
	1745	static struct trace_min_max_param osnoise_stop_tracing_total = {
	1746	.lock = &interface_lock,
	1747	.val = &osnoise_data.stop_tracing_total,
	1748	.max = NULL,
	1749	.min = NULL,
	1750	};
	1751
	1752	#ifdef CONFIG_TIMERLAT_TRACER
	1753	/*
	1754	* osnoise/print_stack: print the stacktrace of the IRQ handler if the total
	1755	* latency is higher than val.
	1756	*/
	1757	static struct trace_min_max_param osnoise_print_stack = {
	1758	.lock = &interface_lock,
	1759	.val = &osnoise_data.print_stack,
	1760	.max = NULL,
	1761	.min = NULL,
	1762	};
	1763
	1764	/*
	1765	* osnoise/timerlat_period: min 100 us, max 1 s
	1766	*/
	1767	u64 timerlat_min_period = 100;
	1768	u64 timerlat_max_period = 1000000;
	1769	static struct trace_min_max_param timerlat_period = {
	1770	.lock = &interface_lock,
	1771	.val = &osnoise_data.timerlat_period,
	1772	.max = &timerlat_max_period,
	1773	.min = &timerlat_min_period,
	1774	};
	1775	#endif
	1776
	1777	static const struct file_operations cpus_fops = {
	1778	.open = tracing_open_generic,
	1779	.read = osnoise_cpus_read,
	1780	.write = osnoise_cpus_write,
	1781	.llseek = generic_file_llseek,
	1782	};
	1783
	1784	/*
	1785	* init_tracefs - A function to initialize the tracefs interface files
	1786	*
	1787	* This function creates entries in tracefs for "osnoise" and "timerlat".
	1788	* It creates these directories in the tracing directory, and within that
	1789	* directory the use can change and view the configs.
	1790	*/
	1791	static int init_tracefs(void)
	1792	{
	1793	struct dentry *top_dir;
	1794	struct dentry *tmp;
	1795	int ret;
	1796
	1797	ret = tracing_init_dentry();
	1798	if (ret)
	1799	return -ENOMEM;
	1800
	1801	top_dir = tracefs_create_dir("osnoise", NULL);
	1802	if (!top_dir)
	1803	return 0;
	1804
	1805	tmp = tracefs_create_file("period_us", 0640, top_dir,
	1806	&osnoise_period, &trace_min_max_fops);
	1807	if (!tmp)
	1808	goto err;
	1809
	1810	tmp = tracefs_create_file("runtime_us", 0644, top_dir,
	1811	&osnoise_runtime, &trace_min_max_fops);
	1812	if (!tmp)
	1813	goto err;
	1814
	1815	tmp = tracefs_create_file("stop_tracing_us", 0640, top_dir,
	1816	&osnoise_stop_tracing_in, &trace_min_max_fops);
	1817	if (!tmp)
	1818	goto err;
	1819
	1820	tmp = tracefs_create_file("stop_tracing_total_us", 0640, top_dir,
	1821	&osnoise_stop_tracing_total, &trace_min_max_fops);
	1822	if (!tmp)
	1823	goto err;
	1824
	1825	tmp = trace_create_file("cpus", 0644, top_dir, NULL, &cpus_fops);
	1826	if (!tmp)
	1827	goto err;
	1828	#ifdef CONFIG_TIMERLAT_TRACER
	1829	#ifdef CONFIG_STACKTRACE
	1830	tmp = tracefs_create_file("print_stack", 0640, top_dir,
	1831	&osnoise_print_stack, &trace_min_max_fops);
	1832	if (!tmp)
	1833	goto err;
	1834	#endif
	1835
	1836	tmp = tracefs_create_file("timerlat_period_us", 0640, top_dir,
	1837	&timerlat_period, &trace_min_max_fops);
	1838	if (!tmp)
	1839	goto err;
	1840	#endif
	1841
	1842	return 0;
	1843
	1844	err:
	1845	tracefs_remove(top_dir);
	1846	return -ENOMEM;
	1847	}
	1848
	1849	static int osnoise_hook_events(void)
	1850	{
	1851	int retval;
	1852
	1853	/*
	1854	* Trace is already hooked, we are re-enabling from
	1855	* a stop_tracing_*.
	1856	*/
	1857	if (trace_osnoise_callback_enabled)
	1858	return 0;
	1859
	1860	retval = hook_irq_events();
	1861	if (retval)
	1862	return -EINVAL;
	1863
	1864	retval = hook_softirq_events();
	1865	if (retval)
	1866	goto out_unhook_irq;
	1867
	1868	retval = hook_thread_events();
	1869	/*
	1870	* All fine!
	1871	*/
	1872	if (!retval)
	1873	return 0;
	1874
	1875	unhook_softirq_events();
	1876	out_unhook_irq:
	1877	unhook_irq_events();
	1878	return -EINVAL;
	1879	}
	1880
	1881	static int __osnoise_tracer_start(struct trace_array *tr)
	1882	{
	1883	int retval;
	1884
	1885	osn_var_reset_all();
	1886
	1887	retval = osnoise_hook_events();
	1888	if (retval)
	1889	return retval;
	1890	/*
	1891	* Make sure NMIs see reseted values.
	1892	*/
	1893	barrier();
	1894	trace_osnoise_callback_enabled = true;
	1895
	1896	retval = start_per_cpu_kthreads(tr);
	1897	if (retval) {
	1898	unhook_irq_events();
	1899	return retval;
	1900	}
	1901
	1902	osnoise_busy = true;
	1903
	1904	return 0;
	1905	}
	1906
	1907	static void osnoise_tracer_start(struct trace_array *tr)
	1908	{
	1909	int retval;
	1910
	1911	if (osnoise_busy)
	1912	return;
	1913
	1914	retval = __osnoise_tracer_start(tr);
	1915	if (retval)
	1916	pr_err(BANNER "Error starting osnoise tracer\n");
	1917
	1918	}
	1919
	1920	static void osnoise_tracer_stop(struct trace_array *tr)
	1921	{
	1922	if (!osnoise_busy)
	1923	return;
	1924
	1925	trace_osnoise_callback_enabled = false;
	1926	barrier();
	1927
	1928	stop_per_cpu_kthreads();
	1929
	1930	unhook_irq_events();
	1931	unhook_softirq_events();
	1932	unhook_thread_events();
	1933
	1934	osnoise_busy = false;
	1935	}
	1936
	1937	static int osnoise_tracer_init(struct trace_array *tr)
	1938	{
	1939
	1940	/* Only allow one instance to enable this */
	1941	if (osnoise_busy)
	1942	return -EBUSY;
	1943
	1944	osnoise_trace = tr;
	1945	tr->max_latency = 0;
	1946
	1947	osnoise_tracer_start(tr);
	1948
	1949	return 0;
	1950	}
	1951
	1952	static void osnoise_tracer_reset(struct trace_array *tr)
	1953	{
	1954	osnoise_tracer_stop(tr);
	1955	}
	1956
	1957	static struct tracer osnoise_tracer __read_mostly = {
	1958	.name = "osnoise",
	1959	.init = osnoise_tracer_init,
	1960	.reset = osnoise_tracer_reset,
	1961	.start = osnoise_tracer_start,
	1962	.stop = osnoise_tracer_stop,
	1963	.print_header = print_osnoise_headers,
	1964	.allow_instances = true,
	1965	};
	1966
	1967	#ifdef CONFIG_TIMERLAT_TRACER
	1968	static void timerlat_tracer_start(struct trace_array *tr)
	1969	{
	1970	int retval;
	1971
	1972	if (osnoise_busy)
	1973	return;
	1974
	1975	osnoise_data.timerlat_tracer = 1;
	1976
	1977	retval = __osnoise_tracer_start(tr);
	1978	if (retval)
	1979	goto out_err;
	1980
	1981	return;
	1982	out_err:
	1983	pr_err(BANNER "Error starting timerlat tracer\n");
	1984	}
	1985
	1986	static void timerlat_tracer_stop(struct trace_array *tr)
	1987	{
	1988	int cpu;
	1989
	1990	if (!osnoise_busy)
	1991	return;
	1992
	1993	for_each_online_cpu(cpu)
	1994	per_cpu(per_cpu_osnoise_var, cpu).sampling = 0;
	1995
	1996	osnoise_tracer_stop(tr);
	1997
	1998	osnoise_data.timerlat_tracer = 0;
	1999	}
	2000
	2001	static int timerlat_tracer_init(struct trace_array *tr)
	2002	{
	2003	/* Only allow one instance to enable this */
	2004	if (osnoise_busy)
	2005	return -EBUSY;
	2006
	2007	osnoise_trace = tr;
	2008
	2009	tr->max_latency = 0;
	2010
	2011	timerlat_tracer_start(tr);
	2012
	2013	return 0;
	2014	}
	2015
	2016	static void timerlat_tracer_reset(struct trace_array *tr)
	2017	{
	2018	timerlat_tracer_stop(tr);
	2019	}
	2020
	2021	static struct tracer timerlat_tracer __read_mostly = {
	2022	.name = "timerlat",
	2023	.init = timerlat_tracer_init,
	2024	.reset = timerlat_tracer_reset,
	2025	.start = timerlat_tracer_start,
	2026	.stop = timerlat_tracer_stop,
	2027	.print_header = print_timerlat_headers,
	2028	.allow_instances = true,
	2029	};
	2030	#endif /* CONFIG_TIMERLAT_TRACER */
	2031
	2032	__init static int init_osnoise_tracer(void)
	2033	{
	2034	int ret;
	2035
	2036	mutex_init(&interface_lock);
	2037
	2038	cpumask_copy(&osnoise_cpumask, cpu_all_mask);
	2039
	2040	ret = register_tracer(&osnoise_tracer);
	2041	if (ret) {
	2042	pr_err(BANNER "Error registering osnoise!\n");
	2043	return ret;
	2044	}
	2045
	2046	#ifdef CONFIG_TIMERLAT_TRACER
	2047	ret = register_tracer(&timerlat_tracer);
	2048	if (ret) {
	2049	pr_err(BANNER "Error registering timerlat\n");
	2050	return ret;
	2051	}
	2052	#endif
	2053	osnoise_init_hotplug_support();
	2054
	2055	init_tracefs();
	2056
	2057	return 0;
	2058	}
	2059	late_initcall(init_osnoise_tracer);

kernel/trace/trace_output.c+118−1

@@ -1202,7 +1202,6 @@ trace_hwlat_print(struct trace_iterator *iter, int flags, · 1201 unmodified lines
1202	return trace_handle_return(s);	1202	return trace_handle_return(s);
1203	}	1203	}
1204		1204
1205
1206	static enum print_line_t	1205	static enum print_line_t
1207	trace_hwlat_raw(struct trace_iterator *iter, int flags,	1206	trace_hwlat_raw(struct trace_iterator *iter, int flags,
1208	struct trace_event *event)	1207	struct trace_event *event)
@@ -1232,6 +1231,122 @@ static struct trace_event trace_hwlat_event = { · 23 unmodified lines
1232	.funcs = &trace_hwlat_funcs,	1231	.funcs = &trace_hwlat_funcs,
1233	};	1232	};
1234		1233
		1234	/* TRACE_OSNOISE */
		1235	static enum print_line_t
		1236	trace_osnoise_print(struct trace_iterator *iter, int flags,
		1237	struct trace_event *event)
		1238	{
		1239	struct trace_entry *entry = iter->ent;
		1240	struct trace_seq *s = &iter->seq;
		1241	struct osnoise_entry *field;
		1242	u64 ratio, ratio_dec;
		1243	u64 net_runtime;
		1244
		1245	trace_assign_type(field, entry);
		1246
		1247	/*
		1248	* compute the available % of cpu time.
		1249	*/
		1250	net_runtime = field->runtime - field->noise;
		1251	ratio = net_runtime * 10000000;
		1252	do_div(ratio, field->runtime);
		1253	ratio_dec = do_div(ratio, 100000);
		1254
		1255	trace_seq_printf(s, "%llu %10llu %3llu.%05llu %7llu",
		1256	field->runtime,
		1257	field->noise,
		1258	ratio, ratio_dec,
		1259	field->max_sample);
		1260
		1261	trace_seq_printf(s, " %6u", field->hw_count);
		1262	trace_seq_printf(s, " %6u", field->nmi_count);
		1263	trace_seq_printf(s, " %6u", field->irq_count);
		1264	trace_seq_printf(s, " %6u", field->softirq_count);
		1265	trace_seq_printf(s, " %6u", field->thread_count);
		1266
		1267	trace_seq_putc(s, '\n');
		1268
		1269	return trace_handle_return(s);
		1270	}
		1271
		1272	static enum print_line_t
		1273	trace_osnoise_raw(struct trace_iterator *iter, int flags,
		1274	struct trace_event *event)
		1275	{
		1276	struct osnoise_entry *field;
		1277	struct trace_seq *s = &iter->seq;
		1278
		1279	trace_assign_type(field, iter->ent);
		1280
		1281	trace_seq_printf(s, "%lld %llu %llu %u %u %u %u %u\n",
		1282	field->runtime,
		1283	field->noise,
		1284	field->max_sample,
		1285	field->hw_count,
		1286	field->nmi_count,
		1287	field->irq_count,
		1288	field->softirq_count,
		1289	field->thread_count);
		1290
		1291	return trace_handle_return(s);
		1292	}
		1293
		1294	static struct trace_event_functions trace_osnoise_funcs = {
		1295	.trace = trace_osnoise_print,
		1296	.raw = trace_osnoise_raw,
		1297	};
		1298
		1299	static struct trace_event trace_osnoise_event = {
		1300	.type = TRACE_OSNOISE,
		1301	.funcs = &trace_osnoise_funcs,
		1302	};
		1303
		1304	/* TRACE_TIMERLAT */
		1305	static enum print_line_t
		1306	trace_timerlat_print(struct trace_iterator *iter, int flags,
		1307	struct trace_event *event)
		1308	{
		1309	struct trace_entry *entry = iter->ent;
		1310	struct trace_seq *s = &iter->seq;
		1311	struct timerlat_entry *field;
		1312
		1313	trace_assign_type(field, entry);
		1314
		1315	trace_seq_printf(s, "#%-5u context %6s timer_latency %9llu ns\n",
		1316	field->seqnum,
		1317	field->context ? "thread" : "irq",
		1318	field->timer_latency);
		1319
		1320	return trace_handle_return(s);
		1321	}
		1322
		1323	static enum print_line_t
		1324	trace_timerlat_raw(struct trace_iterator *iter, int flags,
		1325	struct trace_event *event)
		1326	{
		1327	struct timerlat_entry *field;
		1328	struct trace_seq *s = &iter->seq;
		1329
		1330	trace_assign_type(field, iter->ent);
		1331
		1332	trace_seq_printf(s, "%u %d %llu\n",
		1333	field->seqnum,
		1334	field->context,
		1335	field->timer_latency);
		1336
		1337	return trace_handle_return(s);
		1338	}
		1339
		1340	static struct trace_event_functions trace_timerlat_funcs = {
		1341	.trace = trace_timerlat_print,
		1342	.raw = trace_timerlat_raw,
		1343	};
		1344
		1345	static struct trace_event trace_timerlat_event = {
		1346	.type = TRACE_TIMERLAT,
		1347	.funcs = &trace_timerlat_funcs,
		1348	};
		1349
1235	/* TRACE_BPUTS */	1350	/* TRACE_BPUTS */
1236	static enum print_line_t	1351	static enum print_line_t
1237	trace_bputs_print(struct trace_iterator *iter, int flags,	1352	trace_bputs_print(struct trace_iterator *iter, int flags,
@@ -1442,6 +1557,8 @@ static struct trace_event *events[] __initdata = { · 204 unmodified lines
1442	&trace_bprint_event,	1557	&trace_bprint_event,
1443	&trace_print_event,	1558	&trace_print_event,
1444	&trace_hwlat_event,	1559	&trace_hwlat_event,
		1560	&trace_osnoise_event,
		1561	&trace_timerlat_event,
1445	&trace_raw_data_event,	1562	&trace_raw_data_event,
1446	&trace_func_repeats_event,	1563	&trace_func_repeats_event,
1447	NULL	1564	NULL

kernel/trace/trace_sched_wakeup.c+12−12

@@ -26,9 +26,9 @@ static struct task_struct *wakeup_task; · 25 unmodified lines
26	static int wakeup_cpu;	26	static int wakeup_cpu;
27	static int wakeup_current_cpu;	27	static int wakeup_current_cpu;
28	static unsigned wakeup_prio = -1;	28	static unsigned wakeup_prio = -1;
29	static int wakeup_rt;	29	static bool wakeup_rt;
30	static int wakeup_dl;	30	static bool wakeup_dl;
31	static int tracing_dl = 0;	31	static bool tracing_dl;
32		32
33	static arch_spinlock_t wakeup_lock =	33	static arch_spinlock_t wakeup_lock =
34	(arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;	34	(arch_spinlock_t)__ARCH_SPIN_LOCK_UNLOCKED;
@@ -498,7 +498,7 @@ static void __wakeup_reset(struct trace_array *tr) · 463 unmodified lines
498	{	498	{
499	wakeup_cpu = -1;	499	wakeup_cpu = -1;
500	wakeup_prio = -1;	500	wakeup_prio = -1;
501	tracing_dl = 0;	501	tracing_dl = false;
502		502
503	if (wakeup_task)	503	if (wakeup_task)
504	put_task_struct(wakeup_task);	504	put_task_struct(wakeup_task);
@@ -572,9 +572,9 @@ probe_wakeup(void ignore, struct task_struct p) · 67 unmodified lines
572	* another task until the first one wakes up.	572	* another task until the first one wakes up.
573	*/	573	*/
574	if (dl_task(p))	574	if (dl_task(p))
575	tracing_dl = 1;	575	tracing_dl = true;
576	else	576	else
577	tracing_dl = 0;	577	tracing_dl = false;
578		578
579	wakeup_task = get_task_struct(p);	579	wakeup_task = get_task_struct(p);
580		580
@@ -685,8 +685,8 @@ static int wakeup_tracer_init(struct trace_array *tr) · 104 unmodified lines
685	if (wakeup_busy)	685	if (wakeup_busy)
686	return -EBUSY;	686	return -EBUSY;
687		687
688	wakeup_dl = 0;	688	wakeup_dl = false;
689	wakeup_rt = 0;	689	wakeup_rt = false;
690	return __wakeup_tracer_init(tr);	690	return __wakeup_tracer_init(tr);
691	}	691	}
692		692
@@ -695,8 +695,8 @@ static int wakeup_rt_tracer_init(struct trace_array *tr) · 2 unmodified lines
695	if (wakeup_busy)	695	if (wakeup_busy)
696	return -EBUSY;	696	return -EBUSY;
697		697
698	wakeup_dl = 0;	698	wakeup_dl = false;
699	wakeup_rt = 1;	699	wakeup_rt = true;
700	return __wakeup_tracer_init(tr);	700	return __wakeup_tracer_init(tr);
701	}	701	}
702		702
@@ -705,8 +705,8 @@ static int wakeup_dl_tracer_init(struct trace_array *tr) · 2 unmodified lines
705	if (wakeup_busy)	705	if (wakeup_busy)
706	return -EBUSY;	706	return -EBUSY;
707		707
708	wakeup_dl = 1;	708	wakeup_dl = true;
709	wakeup_rt = 0;	709	wakeup_rt = false;
710	return __wakeup_tracer_init(tr);	710	return __wakeup_tracer_init(tr);
711	}	711	}
712		712