History

printk: add kthread console printers

Create a kthread for each console to perform console printing. During normal operation (@system_state == SYSTEM_RUNNING), the kthread printers are responsible for all printing on their respective consoles. During non-normal operation, console printing is done as it has been: within the context of the printk caller or within irqwork triggered by the printk caller, referred to as direct printing. Since threaded console printers are responsible for all printing during normal operation, this also includes messages generated via deferred printk calls. If direct printing is in effect during a deferred printk call, the queued irqwork will perform the direct printing. To make it clear that this is the only time that the irqwork will perform direct printing, rename the flag PRINTK_PENDING_OUTPUT to PRINTK_PENDING_DIRECT_OUTPUT. Threaded console printers synchronize against each other and against console lockers by taking the console lock for each message that is printed. Note that the kthread printers do not care about direct printing. They will always try to print if new records are available. They can be blocked by direct printing, but will be woken again once direct printing is finished. Console unregistration is a bit tricky because the associated kthread printer cannot be stopped while the console lock is held. A policy is implemented that states: whichever task clears con->thread (under the console lock) is responsible for stopping the kthread. unregister_console() will clear con->thread while the console lock is held and then stop the kthread after releasing the console lock. For consoles that have implemented the exit() callback, the kthread is stopped before exit() is called.

John Ogness committed 4 years ago

Commit 09c5ba01 parent 2bb2b7b2 files · +309 −22Browse files

Showing the changes to kernel (1 of 2 files). Show all

Unified · Split

kernel/printk/printk.c+307−22
@@ -361,6 +361,13 @@ static int console_msg_format = MSG_FORMAT_DEFAULT; · 360 unmodified lines
361361 /* syslog_lock protects syslog_* variables and write access to clear_seq. */
362362 static DEFINE_MUTEX(syslog_lock);
363363
364+/*
365+ * A flag to signify if printk_activate_kthreads() has already started the
366+ * kthread printers. If true, any later registered consoles must start their
367+ * own kthread directly. The flag is write protected by the console_lock.
368+ */
369+static bool printk_kthreads_available;
370+
364371 #ifdef CONFIG_PRINTK
365372 static atomic_t printk_prefer_direct = ATOMIC_INIT(0);
366373
@@ -390,6 +397,39 @@ void printk_prefer_direct_exit(void) · 23 unmodified lines
390397 WARN_ON(atomic_dec_if_positive(&printk_prefer_direct) < 0);
391398 }
392399
400+/*
401+ * Calling printk() always wakes kthread printers so that they can
402+ * flush the new message to their respective consoles. Also, if direct
403+ * printing is allowed, printk() tries to flush the messages directly.
404+ *
405+ * Direct printing is allowed in situations when the kthreads
406+ * are not available or the system is in a problematic state.
407+ *
408+ * See the implementation about possible races.
409+ */
410+static inline bool allow_direct_printing(void)
411+{
412+ /*
413+ * Checking kthread availability is a possible race because the
414+ * kthread printers can become permanently disabled during runtime.
415+ * However, doing that requires holding the console_lock, so any
416+ * pending messages will be direct printed by console_unlock().
417+ */
418+ if (!printk_kthreads_available)
419+ return true;
420+
421+ /*
422+ * Prefer direct printing when the system is in a problematic state.
423+ * The context that sets this state will always see the updated value.
424+ * The other contexts do not care. Anyway, direct printing is just a
425+ * best effort. The direct output is only possible when console_lock
426+ * is not already taken and no kthread printers are actively printing.
427+ */
428+ return (system_state > SYSTEM_RUNNING ||
429+ oops_in_progress ||
430+ atomic_read(&printk_prefer_direct));
431+}
432+
393433 DECLARE_WAIT_QUEUE_HEAD(log_wait);
394434 /* All 3 protected by @syslog_lock. */
395435 /* the next printk record to read by syslog(READ) or /proc/kmsg */
@@ -2280,10 +2320,10 @@ asmlinkage int vprintk_emit(int facility, int level, · 1884 unmodified lines
22802320 printed_len = vprintk_store(facility, level, dev_info, fmt, args);
22812321
22822322 /* If called from the scheduler, we can not call up(). */
2283− if (!in_sched) {
2323+ if (!in_sched && allow_direct_printing()) {
22842324 /*
22852325 * The caller may be holding system-critical or
2286− * timing-sensitive locks. Disable preemption during
2326+ * timing-sensitive locks. Disable preemption during direct
22872327 * printing of all remaining records to all consoles so that
22882328 * this context can return as soon as possible. Hopefully
22892329 * another printk() caller will take over the printing.
@@ -2326,6 +2366,8 @@ EXPORT_SYMBOL(_printk); · 36 unmodified lines
23262366
23272367 static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress);
23282368
2369+static void printk_start_kthread(struct console *con);
2370+
23292371 #else /* CONFIG_PRINTK */
23302372
23312373 #define CONSOLE_LOG_MAX 0
@@ -2359,6 +2401,8 @@ static void call_console_driver(struct console *con, const char *text, size_t le · 27 unmodified lines
23592401 }
23602402 static bool suppress_message_printing(int level) { return false; }
23612403 static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress) { return true; }
2404+static void printk_start_kthread(struct console *con) { }
2405+static bool allow_direct_printing(void) { return true; }
23622406
23632407 #endif /* CONFIG_PRINTK */
23642408
@@ -2559,6 +2603,13 @@ void resume_console(void) · 194 unmodified lines
25592603 down_console_sem();
25602604 console_suspended = 0;
25612605 console_unlock();
2606+
2607+ /*
2608+ * While suspended, new records may have been added to the
2609+ * ringbuffer. Wake up the kthread printers to print them.
2610+ */
2611+ wake_up_klogd();
2612+
25622613 pr_flush(1000, true);
25632614 }
25642615
@@ -2577,6 +2628,9 @@ static int console_cpu_notify(unsigned int cpu) · 12 unmodified lines
25772628 /* If trylock fails, someone else is doing the printing */
25782629 if (console_trylock())
25792630 console_unlock();
2631+
2632+ /* Wake kthread printers. Some may have become usable. */
2633+ wake_up_klogd();
25802634 }
25812635 return 0;
25822636 }
@@ -2648,18 +2702,9 @@ static bool abandon_console_lock_in_panic(void) · 65 unmodified lines
26482702 return atomic_read(&panic_cpu) != raw_smp_processor_id();
26492703 }
26502704
2651−/*
2652− * Check if the given console is currently capable and allowed to print
2653− * records.
2654− *
2655− * Requires the console_lock.
2656− */
2657−static inline bool console_is_usable(struct console *con)
2705+static inline bool __console_is_usable(short flags)
26582706 {
2659− if (!(con->flags & CON_ENABLED))
2660− return false;
2661−
2662− if (!con->write)
2707+ if (!(flags & CON_ENABLED))
26632708 return false;
26642709
26652710 /*
@@ -2668,12 +2713,26 @@ static inline bool console_is_usable(struct console *con) · 2 unmodified lines
26682713 * cope (CON_ANYTIME) don't call them until this CPU is officially up.
26692714 */
26702715 if (!cpu_online(raw_smp_processor_id()) &&
2671− !(con->flags & CON_ANYTIME))
2716+ !(flags & CON_ANYTIME))
26722717 return false;
26732718
26742719 return true;
26752720 }
26762721
2722+/*
2723+ * Check if the given console is currently capable and allowed to print
2724+ * records.
2725+ *
2726+ * Requires the console_lock.
2727+ */
2728+static inline bool console_is_usable(struct console *con)
2729+{
2730+ if (!con->write)
2731+ return false;
2732+
2733+ return __console_is_usable(con->flags);
2734+}
2735+
26772736 static void __console_unlock(void)
26782737 {
26792738 console_locked = 0;
@@ -2786,8 +2845,8 @@ skip: · 106 unmodified lines
27862845 * were flushed to all usable consoles. A returned false informs the caller
27872846 * that everything was not flushed (either there were no usable consoles or
27882847 * another context has taken over printing or it is a panic situation and this
2789− * is not the panic CPU). Regardless the reason, the caller should assume it
2790− * is not useful to immediately try again.
2848+ * is not the panic CPU or direct printing is not preferred). Regardless the
2849+ * reason, the caller should assume it is not useful to immediately try again.
27912850 *
27922851 * Requires the console_lock.
27932852 */
@@ -2804,6 +2863,10 @@ static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handove · 10 unmodified lines
28042863 *handover = false;
28052864
28062865 do {
2866+ /* Let the kthread printers do the work if they can. */
2867+ if (!allow_direct_printing())
2868+ return false;
2869+
28072870 any_progress = false;
28082871
28092872 for_each_console(con) {
@@ -3018,6 +3081,10 @@ void console_start(struct console *console) · 208 unmodified lines
30183081 console_lock();
30193082 console->flags |= CON_ENABLED;
30203083 console_unlock();
3084+
3085+ /* Wake the newly enabled kthread printer. */
3086+ wake_up_klogd();
3087+
30213088 __pr_flush(console, 1000, true);
30223089 }
30233090 EXPORT_SYMBOL(console_start);
@@ -3218,6 +3285,8 @@ void register_console(struct console *newcon) · 194 unmodified lines
32183285 nr_ext_console_drivers++;
32193286
32203287 newcon->dropped = 0;
3288+ newcon->thread = NULL;
3289+
32213290 if (newcon->flags & CON_PRINTBUFFER) {
32223291 /* Get a consistent copy of @syslog_seq. */
32233292 mutex_lock(&syslog_lock);
@@ -3227,6 +3296,10 @@ void register_console(struct console *newcon) · 3 unmodified lines
32273296 /* Begin with next message. */
32283297 newcon->seq = prb_next_seq(prb);
32293298 }
3299+
3300+ if (printk_kthreads_available)
3301+ printk_start_kthread(newcon);
3302+
32303303 console_unlock();
32313304 console_sysfs_notify();
32323305
@@ -3253,6 +3326,7 @@ EXPORT_SYMBOL(register_console); · 20 unmodified lines
32533326
32543327 int unregister_console(struct console *console)
32553328 {
3329+ struct task_struct *thd;
32563330 struct console *con;
32573331 int res;
32583332
@@ -3293,7 +3367,20 @@ int unregister_console(struct console *console) · 34 unmodified lines
32933367 console_drivers->flags |= CON_CONSDEV;
32943368
32953369 console->flags &= ~CON_ENABLED;
3370+
3371+ /*
3372+ * console->thread can only be cleared under the console lock. But
3373+ * stopping the thread must be done without the console lock. The
3374+ * task that clears @thread is the task that stops the kthread.
3375+ */
3376+ thd = console->thread;
3377+ console->thread = NULL;
3378+
32963379 console_unlock();
3380+
3381+ if (thd)
3382+ kthread_stop(thd);
3383+
32973384 console_sysfs_notify();
32983385
32993386 if (console->exit)
@@ -3389,6 +3476,20 @@ static int __init printk_late_init(void) · 89 unmodified lines
33893476 }
33903477 late_initcall(printk_late_init);
33913478
3479+static int __init printk_activate_kthreads(void)
3480+{
3481+ struct console *con;
3482+
3483+ console_lock();
3484+ printk_kthreads_available = true;
3485+ for_each_console(con)
3486+ printk_start_kthread(con);
3487+ console_unlock();
3488+
3489+ return 0;
3490+}
3491+early_initcall(printk_activate_kthreads);
3492+
33923493 #if defined CONFIG_PRINTK
33933494 /* If @con is specified, only wait for that console. Otherwise wait for all. */
33943495 static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress)
@@ -3463,11 +3564,180 @@ bool pr_flush(int timeout_ms, bool reset_on_progress) · 68 unmodified lines
34633564 }
34643565 EXPORT_SYMBOL(pr_flush);
34653566
3567+static void __printk_fallback_preferred_direct(void)
3568+{
3569+ printk_prefer_direct_enter();
3570+ pr_err("falling back to preferred direct printing\n");
3571+ printk_kthreads_available = false;
3572+}
3573+
3574+/*
3575+ * Enter preferred direct printing, but never exit. Mark console threads as
3576+ * unavailable. The system is then forever in preferred direct printing and
3577+ * any printing threads will exit.
3578+ *
3579+ * Must *not* be called under console_lock. Use
3580+ * __printk_fallback_preferred_direct() if already holding console_lock.
3581+ */
3582+static void printk_fallback_preferred_direct(void)
3583+{
3584+ console_lock();
3585+ __printk_fallback_preferred_direct();
3586+ console_unlock();
3587+}
3588+
3589+static bool printer_should_wake(struct console *con, u64 seq)
3590+{
3591+ short flags;
3592+
3593+ if (kthread_should_stop() || !printk_kthreads_available)
3594+ return true;
3595+
3596+ if (console_suspended)
3597+ return false;
3598+
3599+ /*
3600+ * This is an unsafe read from con->flags, but a false positive is
3601+ * not a problem. Worst case it would allow the printer to wake up
3602+ * although it is disabled. But the printer will notice that when
3603+ * attempting to print and instead go back to sleep.
3604+ */
3605+ flags = data_race(READ_ONCE(con->flags));
3606+
3607+ if (!__console_is_usable(flags))
3608+ return false;
3609+
3610+ return prb_read_valid(prb, seq, NULL);
3611+}
3612+
3613+static int printk_kthread_func(void *data)
3614+{
3615+ struct console *con = data;
3616+ char *dropped_text = NULL;
3617+ char *ext_text = NULL;
3618+ bool handover;
3619+ u64 seq = 0;
3620+ char *text;
3621+ int error;
3622+
3623+ text = kmalloc(CONSOLE_LOG_MAX, GFP_KERNEL);
3624+ if (!text) {
3625+ con_printk(KERN_ERR, con, "failed to allocate text buffer\n");
3626+ printk_fallback_preferred_direct();
3627+ goto out;
3628+ }
3629+
3630+ if (con->flags & CON_EXTENDED) {
3631+ ext_text = kmalloc(CONSOLE_EXT_LOG_MAX, GFP_KERNEL);
3632+ if (!ext_text) {
3633+ con_printk(KERN_ERR, con, "failed to allocate ext_text buffer\n");
3634+ printk_fallback_preferred_direct();
3635+ goto out;
3636+ }
3637+ } else {
3638+ dropped_text = kmalloc(DROPPED_TEXT_MAX, GFP_KERNEL);
3639+ if (!dropped_text) {
3640+ con_printk(KERN_ERR, con, "failed to allocate dropped_text buffer\n");
3641+ printk_fallback_preferred_direct();
3642+ goto out;
3643+ }
3644+ }
3645+
3646+ con_printk(KERN_INFO, con, "printing thread started\n");
3647+
3648+ for (;;) {
3649+ /*
3650+ * Guarantee this task is visible on the waitqueue before
3651+ * checking the wake condition.
3652+ *
3653+ * The full memory barrier within set_current_state() of
3654+ * prepare_to_wait_event() pairs with the full memory barrier
3655+ * within wq_has_sleeper().
3656+ *
3657+ * This pairs with __wake_up_klogd:A.
3658+ */
3659+ error = wait_event_interruptible(log_wait,
3660+ printer_should_wake(con, seq)); /* LMM(printk_kthread_func:A) */
3661+
3662+ if (kthread_should_stop() || !printk_kthreads_available)
3663+ break;
3664+
3665+ if (error)
3666+ continue;
3667+
3668+ console_lock();
3669+
3670+ if (console_suspended) {
3671+ up_console_sem();
3672+ continue;
3673+ }
3674+
3675+ if (!console_is_usable(con)) {
3676+ __console_unlock();
3677+ continue;
3678+ }
3679+
3680+ /*
3681+ * Even though the printk kthread is always preemptible, it is
3682+ * still not allowed to call cond_resched() from within
3683+ * console drivers. The task may become non-preemptible in the
3684+ * console driver call chain. For example, vt_console_print()
3685+ * takes a spinlock and then can call into fbcon_redraw(),
3686+ * which can conditionally invoke cond_resched().
3687+ */
3688+ console_may_schedule = 0;
3689+ console_emit_next_record(con, text, ext_text, dropped_text, &handover);
3690+ if (handover)
3691+ continue;
3692+
3693+ seq = con->seq;
3694+
3695+ __console_unlock();
3696+ }
3697+
3698+ con_printk(KERN_INFO, con, "printing thread stopped\n");
3699+out:
3700+ kfree(dropped_text);
3701+ kfree(ext_text);
3702+ kfree(text);
3703+
3704+ console_lock();
3705+ /*
3706+ * If this kthread is being stopped by another task, con->thread will
3707+ * already be NULL. That is fine. The important thing is that it is
3708+ * NULL after the kthread exits.
3709+ */
3710+ con->thread = NULL;
3711+ console_unlock();
3712+
3713+ return 0;
3714+}
3715+
3716+/* Must be called under console_lock. */
3717+static void printk_start_kthread(struct console *con)
3718+{
3719+ /*
3720+ * Do not start a kthread if there is no write() callback. The
3721+ * kthreads assume the write() callback exists.
3722+ */
3723+ if (!con->write)
3724+ return;
3725+
3726+ con->thread = kthread_run(printk_kthread_func, con,
3727+ "pr/%s%d", con->name, con->index);
3728+ if (IS_ERR(con->thread)) {
3729+ con->thread = NULL;
3730+ con_printk(KERN_ERR, con, "unable to start printing thread\n");
3731+ __printk_fallback_preferred_direct();
3732+ return;
3733+ }
3734+}
3735+
34663736 /*
34673737 * Delayed printk version, for scheduler-internal messages:
34683738 */
3469−#define PRINTK_PENDING_WAKEUP 0x01
3470−#define PRINTK_PENDING_OUTPUT 0x02
3739+#define PRINTK_PENDING_WAKEUP 0x01
3740+#define PRINTK_PENDING_DIRECT_OUTPUT 0x02
34713741
34723742 static DEFINE_PER_CPU(int, printk_pending);
34733743
@@ -3475,10 +3745,14 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work) · 1 unmodified line
34753745 {
34763746 int pending = this_cpu_xchg(printk_pending, 0);
34773747
3478− if (pending & PRINTK_PENDING_OUTPUT) {
3748+ if (pending & PRINTK_PENDING_DIRECT_OUTPUT) {
3749+ printk_prefer_direct_enter();
3750+
34793751 /* If trylock fails, someone else is doing the printing */
34803752 if (console_trylock())
34813753 console_unlock();
3754+
3755+ printk_prefer_direct_exit();
34823756 }
34833757
34843758 if (pending & PRINTK_PENDING_WAKEUP)
@@ -3503,10 +3777,11 @@ static void __wake_up_klogd(int val) · 18 unmodified lines
35033777 * prepare_to_wait_event(), which is called after ___wait_event() adds
35043778 * the waiter but before it has checked the wait condition.
35053779 *
3506− * This pairs with devkmsg_read:A and syslog_print:A.
3780+ * This pairs with devkmsg_read:A, syslog_print:A, and
3781+ * printk_kthread_func:A.
35073782 */
35083783 if (wq_has_sleeper(&log_wait) || /* LMM(__wake_up_klogd:A) */
3509− (val & PRINTK_PENDING_OUTPUT)) {
3784+ (val & PRINTK_PENDING_DIRECT_OUTPUT)) {
35103785 this_cpu_or(printk_pending, val);
35113786 irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
35123787 }
@@ -3524,7 +3799,17 @@ void defer_console_output(void) · 11 unmodified lines
35243799 * New messages may have been added directly to the ringbuffer
35253800 * using vprintk_store(), so wake any waiters as well.
35263801 */
3527− __wake_up_klogd(PRINTK_PENDING_WAKEUP | PRINTK_PENDING_OUTPUT);
3802+ int val = PRINTK_PENDING_WAKEUP;
3803+
3804+ /*
3805+ * Make sure that some context will print the messages when direct
3806+ * printing is allowed. This happens in situations when the kthreads
3807+ * may not be as reliable or perhaps unusable.
3808+ */
3809+ if (allow_direct_printing())
3810+ val |= PRINTK_PENDING_DIRECT_OUTPUT;
3811+
3812+ __wake_up_klogd(val);
35283813 }
35293814
35303815 void printk_trigger_flush(void)

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t