History

printk: add kthread console printers

Create a kthread for each console to perform console printing. During normal operation (@system_state == SYSTEM_RUNNING), the kthread printers are responsible for all printing on their respective consoles. During non-normal operation, console printing is done as it has been: within the context of the printk caller or within irqwork triggered by the printk caller, referred to as direct printing. Since threaded console printers are responsible for all printing during normal operation, this also includes messages generated via deferred printk calls. If direct printing is in effect during a deferred printk call, the queued irqwork will perform the direct printing. To make it clear that this is the only time that the irqwork will perform direct printing, rename the flag PRINTK_PENDING_OUTPUT to PRINTK_PENDING_DIRECT_OUTPUT. Threaded console printers synchronize against each other and against console lockers by taking the console lock for each message that is printed. Note that the kthread printers do not care about direct printing. They will always try to print if new records are available. They can be blocked by direct printing, but will be woken again once direct printing is finished. Console unregistration is a bit tricky because the associated kthread printer cannot be stopped while the console lock is held. A policy is implemented that states: whichever task clears con->thread (under the console lock) is responsible for stopping the kthread. unregister_console() will clear con->thread while the console lock is held and then stop the kthread after releasing the console lock. For consoles that have implemented the exit() callback, the kthread is stopped before exit() is called.

John Ogness committed 4 years ago

Commit 09c5ba01 parent 2bb2b7b2 files · +309 −22Browse files

Showing the changes to kernel (1 of 2 files). Show all

Unified · Split

kernel/printk/printk.c+307−22
@@ -361,6 +361,13 @@ static int console_msg_format = MSG_FORMAT_DEFAULT; · 360 unmodified lines
361/* syslog_lock protects syslog_* variables and write access to clear_seq. */361/* syslog_lock protects syslog_* variables and write access to clear_seq. */
362static DEFINE_MUTEX(syslog_lock);362static DEFINE_MUTEX(syslog_lock);
363 363
364/*
365 * A flag to signify if printk_activate_kthreads() has already started the
366 * kthread printers. If true, any later registered consoles must start their
367 * own kthread directly. The flag is write protected by the console_lock.
368 */
369static bool printk_kthreads_available;
370
364#ifdef CONFIG_PRINTK371#ifdef CONFIG_PRINTK
365static atomic_t printk_prefer_direct = ATOMIC_INIT(0);372static atomic_t printk_prefer_direct = ATOMIC_INIT(0);
366 373
@@ -390,6 +397,39 @@ void printk_prefer_direct_exit(void) · 23 unmodified lines
390 WARN_ON(atomic_dec_if_positive(&printk_prefer_direct) < 0);397 WARN_ON(atomic_dec_if_positive(&printk_prefer_direct) < 0);
391}398}
392 399
400/*
401 * Calling printk() always wakes kthread printers so that they can
402 * flush the new message to their respective consoles. Also, if direct
403 * printing is allowed, printk() tries to flush the messages directly.
404 *
405 * Direct printing is allowed in situations when the kthreads
406 * are not available or the system is in a problematic state.
407 *
408 * See the implementation about possible races.
409 */
410static inline bool allow_direct_printing(void)
411{
412 /*
413 * Checking kthread availability is a possible race because the
414 * kthread printers can become permanently disabled during runtime.
415 * However, doing that requires holding the console_lock, so any
416 * pending messages will be direct printed by console_unlock().
417 */
418 if (!printk_kthreads_available)
419 return true;
420
421 /*
422 * Prefer direct printing when the system is in a problematic state.
423 * The context that sets this state will always see the updated value.
424 * The other contexts do not care. Anyway, direct printing is just a
425 * best effort. The direct output is only possible when console_lock
426 * is not already taken and no kthread printers are actively printing.
427 */
428 return (system_state > SYSTEM_RUNNING ||
429 oops_in_progress ||
430 atomic_read(&printk_prefer_direct));
431}
432
393DECLARE_WAIT_QUEUE_HEAD(log_wait);433DECLARE_WAIT_QUEUE_HEAD(log_wait);
394/* All 3 protected by @syslog_lock. */434/* All 3 protected by @syslog_lock. */
395/* the next printk record to read by syslog(READ) or /proc/kmsg */435/* the next printk record to read by syslog(READ) or /proc/kmsg */
@@ -2280,10 +2320,10 @@ asmlinkage int vprintk_emit(int facility, int level, · 1884 unmodified lines
2280 printed_len = vprintk_store(facility, level, dev_info, fmt, args);2320 printed_len = vprintk_store(facility, level, dev_info, fmt, args);
2281 2321
2282 /* If called from the scheduler, we can not call up(). */2322 /* If called from the scheduler, we can not call up(). */
2283 if (!in_sched) {2323 if (!in_sched && allow_direct_printing()) {
2284 /*2324 /*
2285 * The caller may be holding system-critical or2325 * The caller may be holding system-critical or
2286 * timing-sensitive locks. Disable preemption during2326 * timing-sensitive locks. Disable preemption during direct
2287 * printing of all remaining records to all consoles so that2327 * printing of all remaining records to all consoles so that
2288 * this context can return as soon as possible. Hopefully2328 * this context can return as soon as possible. Hopefully
2289 * another printk() caller will take over the printing.2329 * another printk() caller will take over the printing.
@@ -2326,6 +2366,8 @@ EXPORT_SYMBOL(_printk); · 36 unmodified lines
2326 2366
2327static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress);2367static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress);
2328 2368
2369static void printk_start_kthread(struct console *con);
2370
2329#else /* CONFIG_PRINTK */2371#else /* CONFIG_PRINTK */
2330 2372
2331#define CONSOLE_LOG_MAX 02373#define CONSOLE_LOG_MAX 0
@@ -2359,6 +2401,8 @@ static void call_console_driver(struct console *con, const char *text, size_t le · 27 unmodified lines
2359}2401}
2360static bool suppress_message_printing(int level) { return false; }2402static bool suppress_message_printing(int level) { return false; }
2361static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress) { return true; }2403static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress) { return true; }
2404static void printk_start_kthread(struct console *con) { }
2405static bool allow_direct_printing(void) { return true; }
2362 2406
2363#endif /* CONFIG_PRINTK */2407#endif /* CONFIG_PRINTK */
2364 2408
@@ -2559,6 +2603,13 @@ void resume_console(void) · 194 unmodified lines
2559 down_console_sem();2603 down_console_sem();
2560 console_suspended = 0;2604 console_suspended = 0;
2561 console_unlock();2605 console_unlock();
2606
2607 /*
2608 * While suspended, new records may have been added to the
2609 * ringbuffer. Wake up the kthread printers to print them.
2610 */
2611 wake_up_klogd();
2612
2562 pr_flush(1000, true);2613 pr_flush(1000, true);
2563}2614}
2564 2615
@@ -2577,6 +2628,9 @@ static int console_cpu_notify(unsigned int cpu) · 12 unmodified lines
2577 /* If trylock fails, someone else is doing the printing */2628 /* If trylock fails, someone else is doing the printing */
2578 if (console_trylock())2629 if (console_trylock())
2579 console_unlock();2630 console_unlock();
2631
2632 /* Wake kthread printers. Some may have become usable. */
2633 wake_up_klogd();
2580 }2634 }
2581 return 0;2635 return 0;
2582}2636}
@@ -2648,18 +2702,9 @@ static bool abandon_console_lock_in_panic(void) · 65 unmodified lines
2648 return atomic_read(&panic_cpu) != raw_smp_processor_id();2702 return atomic_read(&panic_cpu) != raw_smp_processor_id();
2649}2703}
2650 2704
2651/*2705static inline bool __console_is_usable(short flags)
2652 * Check if the given console is currently capable and allowed to print
2653 * records.
2654 *
2655 * Requires the console_lock.
2656 */
2657static inline bool console_is_usable(struct console *con)
2658{2706{
2659 if (!(con->flags & CON_ENABLED))2707 if (!(flags & CON_ENABLED))
2660 return false;
2661
2662 if (!con->write)
2663 return false;2708 return false;
2664 2709
2665 /*2710 /*
@@ -2668,12 +2713,26 @@ static inline bool console_is_usable(struct console *con) · 2 unmodified lines
2668 * cope (CON_ANYTIME) don't call them until this CPU is officially up.2713 * cope (CON_ANYTIME) don't call them until this CPU is officially up.
2669 */2714 */
2670 if (!cpu_online(raw_smp_processor_id()) &&2715 if (!cpu_online(raw_smp_processor_id()) &&
2671 !(con->flags & CON_ANYTIME))2716 !(flags & CON_ANYTIME))
2672 return false;2717 return false;
2673 2718
2674 return true;2719 return true;
2675}2720}
2676 2721
2722/*
2723 * Check if the given console is currently capable and allowed to print
2724 * records.
2725 *
2726 * Requires the console_lock.
2727 */
2728static inline bool console_is_usable(struct console *con)
2729{
2730 if (!con->write)
2731 return false;
2732
2733 return __console_is_usable(con->flags);
2734}
2735
2677static void __console_unlock(void)2736static void __console_unlock(void)
2678{2737{
2679 console_locked = 0;2738 console_locked = 0;
@@ -2786,8 +2845,8 @@ skip: · 106 unmodified lines
2786 * were flushed to all usable consoles. A returned false informs the caller2845 * were flushed to all usable consoles. A returned false informs the caller
2787 * that everything was not flushed (either there were no usable consoles or2846 * that everything was not flushed (either there were no usable consoles or
2788 * another context has taken over printing or it is a panic situation and this2847 * another context has taken over printing or it is a panic situation and this
2789 * is not the panic CPU). Regardless the reason, the caller should assume it2848 * is not the panic CPU or direct printing is not preferred). Regardless the
2790 * is not useful to immediately try again.2849 * reason, the caller should assume it is not useful to immediately try again.
2791 *2850 *
2792 * Requires the console_lock.2851 * Requires the console_lock.
2793 */2852 */
@@ -2804,6 +2863,10 @@ static bool console_flush_all(bool do_cond_resched, u64 *next_seq, bool *handove · 10 unmodified lines
2804 *handover = false;2863 *handover = false;
2805 2864
2806 do {2865 do {
2866 /* Let the kthread printers do the work if they can. */
2867 if (!allow_direct_printing())
2868 return false;
2869
2807 any_progress = false;2870 any_progress = false;
2808 2871
2809 for_each_console(con) {2872 for_each_console(con) {
@@ -3018,6 +3081,10 @@ void console_start(struct console *console) · 208 unmodified lines
3018 console_lock();3081 console_lock();
3019 console->flags |= CON_ENABLED;3082 console->flags |= CON_ENABLED;
3020 console_unlock();3083 console_unlock();
3084
3085 /* Wake the newly enabled kthread printer. */
3086 wake_up_klogd();
3087
3021 __pr_flush(console, 1000, true);3088 __pr_flush(console, 1000, true);
3022}3089}
3023EXPORT_SYMBOL(console_start);3090EXPORT_SYMBOL(console_start);
@@ -3218,6 +3285,8 @@ void register_console(struct console *newcon) · 194 unmodified lines
3218 nr_ext_console_drivers++;3285 nr_ext_console_drivers++;
3219 3286
3220 newcon->dropped = 0;3287 newcon->dropped = 0;
3288 newcon->thread = NULL;
3289
3221 if (newcon->flags & CON_PRINTBUFFER) {3290 if (newcon->flags & CON_PRINTBUFFER) {
3222 /* Get a consistent copy of @syslog_seq. */3291 /* Get a consistent copy of @syslog_seq. */
3223 mutex_lock(&syslog_lock);3292 mutex_lock(&syslog_lock);
@@ -3227,6 +3296,10 @@ void register_console(struct console *newcon) · 3 unmodified lines
3227 /* Begin with next message. */3296 /* Begin with next message. */
3228 newcon->seq = prb_next_seq(prb);3297 newcon->seq = prb_next_seq(prb);
3229 }3298 }
3299
3300 if (printk_kthreads_available)
3301 printk_start_kthread(newcon);
3302
3230 console_unlock();3303 console_unlock();
3231 console_sysfs_notify();3304 console_sysfs_notify();
3232 3305
@@ -3253,6 +3326,7 @@ EXPORT_SYMBOL(register_console); · 20 unmodified lines
3253 3326
3254int unregister_console(struct console *console)3327int unregister_console(struct console *console)
3255{3328{
3329 struct task_struct *thd;
3256 struct console *con;3330 struct console *con;
3257 int res;3331 int res;
3258 3332
@@ -3293,7 +3367,20 @@ int unregister_console(struct console *console) · 34 unmodified lines
3293 console_drivers->flags |= CON_CONSDEV;3367 console_drivers->flags |= CON_CONSDEV;
3294 3368
3295 console->flags &= ~CON_ENABLED;3369 console->flags &= ~CON_ENABLED;
3370
3371 /*
3372 * console->thread can only be cleared under the console lock. But
3373 * stopping the thread must be done without the console lock. The
3374 * task that clears @thread is the task that stops the kthread.
3375 */
3376 thd = console->thread;
3377 console->thread = NULL;
3378
3296 console_unlock();3379 console_unlock();
3380
3381 if (thd)
3382 kthread_stop(thd);
3383
3297 console_sysfs_notify();3384 console_sysfs_notify();
3298 3385
3299 if (console->exit)3386 if (console->exit)
@@ -3389,6 +3476,20 @@ static int __init printk_late_init(void) · 89 unmodified lines
3389}3476}
3390late_initcall(printk_late_init);3477late_initcall(printk_late_init);
3391 3478
3479static int __init printk_activate_kthreads(void)
3480{
3481 struct console *con;
3482
3483 console_lock();
3484 printk_kthreads_available = true;
3485 for_each_console(con)
3486 printk_start_kthread(con);
3487 console_unlock();
3488
3489 return 0;
3490}
3491early_initcall(printk_activate_kthreads);
3492
3392#if defined CONFIG_PRINTK3493#if defined CONFIG_PRINTK
3393/* If @con is specified, only wait for that console. Otherwise wait for all. */3494/* If @con is specified, only wait for that console. Otherwise wait for all. */
3394static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress)3495static bool __pr_flush(struct console *con, int timeout_ms, bool reset_on_progress)
@@ -3463,11 +3564,180 @@ bool pr_flush(int timeout_ms, bool reset_on_progress) · 68 unmodified lines
3463}3564}
3464EXPORT_SYMBOL(pr_flush);3565EXPORT_SYMBOL(pr_flush);
3465 3566
3567static void __printk_fallback_preferred_direct(void)
3568{
3569 printk_prefer_direct_enter();
3570 pr_err("falling back to preferred direct printing\n");
3571 printk_kthreads_available = false;
3572}
3573
3574/*
3575 * Enter preferred direct printing, but never exit. Mark console threads as
3576 * unavailable. The system is then forever in preferred direct printing and
3577 * any printing threads will exit.
3578 *
3579 * Must *not* be called under console_lock. Use
3580 * __printk_fallback_preferred_direct() if already holding console_lock.
3581 */
3582static void printk_fallback_preferred_direct(void)
3583{
3584 console_lock();
3585 __printk_fallback_preferred_direct();
3586 console_unlock();
3587}
3588
3589static bool printer_should_wake(struct console *con, u64 seq)
3590{
3591 short flags;
3592
3593 if (kthread_should_stop() || !printk_kthreads_available)
3594 return true;
3595
3596 if (console_suspended)
3597 return false;
3598
3599 /*
3600 * This is an unsafe read from con->flags, but a false positive is
3601 * not a problem. Worst case it would allow the printer to wake up
3602 * although it is disabled. But the printer will notice that when
3603 * attempting to print and instead go back to sleep.
3604 */
3605 flags = data_race(READ_ONCE(con->flags));
3606
3607 if (!__console_is_usable(flags))
3608 return false;
3609
3610 return prb_read_valid(prb, seq, NULL);
3611}
3612
3613static int printk_kthread_func(void *data)
3614{
3615 struct console *con = data;
3616 char *dropped_text = NULL;
3617 char *ext_text = NULL;
3618 bool handover;
3619 u64 seq = 0;
3620 char *text;
3621 int error;
3622
3623 text = kmalloc(CONSOLE_LOG_MAX, GFP_KERNEL);
3624 if (!text) {
3625 con_printk(KERN_ERR, con, "failed to allocate text buffer\n");
3626 printk_fallback_preferred_direct();
3627 goto out;
3628 }
3629
3630 if (con->flags & CON_EXTENDED) {
3631 ext_text = kmalloc(CONSOLE_EXT_LOG_MAX, GFP_KERNEL);
3632 if (!ext_text) {
3633 con_printk(KERN_ERR, con, "failed to allocate ext_text buffer\n");
3634 printk_fallback_preferred_direct();
3635 goto out;
3636 }
3637 } else {
3638 dropped_text = kmalloc(DROPPED_TEXT_MAX, GFP_KERNEL);
3639 if (!dropped_text) {
3640 con_printk(KERN_ERR, con, "failed to allocate dropped_text buffer\n");
3641 printk_fallback_preferred_direct();
3642 goto out;
3643 }
3644 }
3645
3646 con_printk(KERN_INFO, con, "printing thread started\n");
3647
3648 for (;;) {
3649 /*
3650 * Guarantee this task is visible on the waitqueue before
3651 * checking the wake condition.
3652 *
3653 * The full memory barrier within set_current_state() of
3654 * prepare_to_wait_event() pairs with the full memory barrier
3655 * within wq_has_sleeper().
3656 *
3657 * This pairs with __wake_up_klogd:A.
3658 */
3659 error = wait_event_interruptible(log_wait,
3660 printer_should_wake(con, seq)); /* LMM(printk_kthread_func:A) */
3661
3662 if (kthread_should_stop() || !printk_kthreads_available)
3663 break;
3664
3665 if (error)
3666 continue;
3667
3668 console_lock();
3669
3670 if (console_suspended) {
3671 up_console_sem();
3672 continue;
3673 }
3674
3675 if (!console_is_usable(con)) {
3676 __console_unlock();
3677 continue;
3678 }
3679
3680 /*
3681 * Even though the printk kthread is always preemptible, it is
3682 * still not allowed to call cond_resched() from within
3683 * console drivers. The task may become non-preemptible in the
3684 * console driver call chain. For example, vt_console_print()
3685 * takes a spinlock and then can call into fbcon_redraw(),
3686 * which can conditionally invoke cond_resched().
3687 */
3688 console_may_schedule = 0;
3689 console_emit_next_record(con, text, ext_text, dropped_text, &handover);
3690 if (handover)
3691 continue;
3692
3693 seq = con->seq;
3694
3695 __console_unlock();
3696 }
3697
3698 con_printk(KERN_INFO, con, "printing thread stopped\n");
3699out:
3700 kfree(dropped_text);
3701 kfree(ext_text);
3702 kfree(text);
3703
3704 console_lock();
3705 /*
3706 * If this kthread is being stopped by another task, con->thread will
3707 * already be NULL. That is fine. The important thing is that it is
3708 * NULL after the kthread exits.
3709 */
3710 con->thread = NULL;
3711 console_unlock();
3712
3713 return 0;
3714}
3715
3716/* Must be called under console_lock. */
3717static void printk_start_kthread(struct console *con)
3718{
3719 /*
3720 * Do not start a kthread if there is no write() callback. The
3721 * kthreads assume the write() callback exists.
3722 */
3723 if (!con->write)
3724 return;
3725
3726 con->thread = kthread_run(printk_kthread_func, con,
3727 "pr/%s%d", con->name, con->index);
3728 if (IS_ERR(con->thread)) {
3729 con->thread = NULL;
3730 con_printk(KERN_ERR, con, "unable to start printing thread\n");
3731 __printk_fallback_preferred_direct();
3732 return;
3733 }
3734}
3735
3466/*3736/*
3467 * Delayed printk version, for scheduler-internal messages:3737 * Delayed printk version, for scheduler-internal messages:
3468 */3738 */
3469#define PRINTK_PENDING_WAKEUP 0x013739#define PRINTK_PENDING_WAKEUP 0x01
3470#define PRINTK_PENDING_OUTPUT 0x023740#define PRINTK_PENDING_DIRECT_OUTPUT 0x02
3471 3741
3472static DEFINE_PER_CPU(int, printk_pending);3742static DEFINE_PER_CPU(int, printk_pending);
3473 3743
@@ -3475,10 +3745,14 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work) · 1 unmodified line
3475{3745{
3476 int pending = this_cpu_xchg(printk_pending, 0);3746 int pending = this_cpu_xchg(printk_pending, 0);
3477 3747
3478 if (pending & PRINTK_PENDING_OUTPUT) {3748 if (pending & PRINTK_PENDING_DIRECT_OUTPUT) {
3749 printk_prefer_direct_enter();
3750
3479 /* If trylock fails, someone else is doing the printing */3751 /* If trylock fails, someone else is doing the printing */
3480 if (console_trylock())3752 if (console_trylock())
3481 console_unlock();3753 console_unlock();
3754
3755 printk_prefer_direct_exit();
3482 }3756 }
3483 3757
3484 if (pending & PRINTK_PENDING_WAKEUP)3758 if (pending & PRINTK_PENDING_WAKEUP)
@@ -3503,10 +3777,11 @@ static void __wake_up_klogd(int val) · 18 unmodified lines
3503 * prepare_to_wait_event(), which is called after ___wait_event() adds3777 * prepare_to_wait_event(), which is called after ___wait_event() adds
3504 * the waiter but before it has checked the wait condition.3778 * the waiter but before it has checked the wait condition.
3505 *3779 *
3506 * This pairs with devkmsg_read:A and syslog_print:A.3780 * This pairs with devkmsg_read:A, syslog_print:A, and
3781 * printk_kthread_func:A.
3507 */3782 */
3508 if (wq_has_sleeper(&log_wait) || /* LMM(__wake_up_klogd:A) */3783 if (wq_has_sleeper(&log_wait) || /* LMM(__wake_up_klogd:A) */
3509 (val & PRINTK_PENDING_OUTPUT)) {3784 (val & PRINTK_PENDING_DIRECT_OUTPUT)) {
3510 this_cpu_or(printk_pending, val);3785 this_cpu_or(printk_pending, val);
3511 irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));3786 irq_work_queue(this_cpu_ptr(&wake_up_klogd_work));
3512 }3787 }
@@ -3524,7 +3799,17 @@ void defer_console_output(void) · 11 unmodified lines
3524 * New messages may have been added directly to the ringbuffer3799 * New messages may have been added directly to the ringbuffer
3525 * using vprintk_store(), so wake any waiters as well.3800 * using vprintk_store(), so wake any waiters as well.
3526 */3801 */
3527 __wake_up_klogd(PRINTK_PENDING_WAKEUP | PRINTK_PENDING_OUTPUT);3802 int val = PRINTK_PENDING_WAKEUP;
3803
3804 /*
3805 * Make sure that some context will print the messages when direct
3806 * printing is allowed. This happens in situations when the kthreads
3807 * may not be as reliable or perhaps unusable.
3808 */
3809 if (allow_direct_printing())
3810 val |= PRINTK_PENDING_DIRECT_OUTPUT;
3811
3812 __wake_up_klogd(val);
3528}3813}
3529 3814
3530void printk_trigger_flush(void)3815void printk_trigger_flush(void)

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t