History

Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull RCU changes for v3.4 from Ingo Molnar. The major features of this series are: - making RCU more aggressive about entering dyntick-idle mode in order to improve energy efficiency - converting a few more call_rcu()s to kfree_rcu()s - applying a number of rcutree fixes and cleanups to rcutiny - removing CONFIG_SMP #ifdefs from treercu - allowing RCU CPU stall times to be set via sysfs - adding CPU-stall capability to rcutorture - adding more RCU-abuse diagnostics - updating documentation - fixing yet more issues located by the still-ongoing top-to-bottom inspection of RCU, this time with a special focus on the CPU-hotplug code path. * 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (48 commits) rcu: Stop spurious warnings from synchronize_sched_expedited rcu: Hold off RCU_FAST_NO_HZ after timer posted rcu: Eliminate softirq-mediated RCU_FAST_NO_HZ idle-entry loop rcu: Add RCU_NONIDLE() for idle-loop RCU read-side critical sections rcu: Allow nesting of rcu_idle_enter() and rcu_idle_exit() rcu: Remove redundant check for rcu_head misalignment PTR_ERR should be called before its argument is cleared. rcu: Convert WARN_ON_ONCE() in rcu_lock_acquire() to lockdep rcu: Trace only after NULL-pointer check rcu: Call out dangers of expedited RCU primitives rcu: Rework detection of use of RCU by offline CPUs lockdep: Add CPU-idle/offline warning to lockdep-RCU splat rcu: No interrupt disabling for rcu_prepare_for_idle() rcu: Move synchronize_sched_expedited() to rcutree.c rcu: Check for illegal use of RCU from offlined CPUs rcu: Update stall-warning documentation rcu: Add CPU-stall capability to rcutorture rcu: Make documentation give more realistic rcutorture duration rcutorture: Permit holding off CPU-hotplug operations during boot rcu: Print scheduling-clock information on RCU CPU stall-warning messages ...

Linus Torvalds committed 14 years ago

Commit 5928a2b2 parents 5ed59af bdd443129 files · +2965 −642Browse files

Unified · Split

29 files
  1. Documentation/RCU/RTFP.txt +1745 −157
  2. Documentation/RCU/checklist.txt +14 −0
  3. Documentation/RCU/stallwarn.txt +80 −7
  4. Documentation/RCU/torture.txt +30 −3
  5. Documentation/RCU/trace.txt +16 −20
  6. arch/s390/kernel/irq.c +1 −8
  7. drivers/target/tcm_fc/tfc_sess.c +1 −11
  8. include/linux/rcupdate.h +80 −3
  9. include/linux/rcutiny.h +6 −4
  10. include/linux/rcutree.h +19 −0
  11. include/linux/sched.h +1 −2
  12. include/linux/srcu.h +11 −4
  13. include/trace/events/rcu.h +39 −24
  14. init/Kconfig +0 −9
  15. kernel/lockdep.c +7 −1
  16. kernel/rcu.h +23 −3
  17. kernel/rcupdate.c +5 −0
  18. kernel/rcutiny.c +20 −6
  19. kernel/rcutiny_plugin.h +60 −17
  20. kernel/rcutorture.c +86 −5
  21. kernel/rcutree.c +371 −136
  22. kernel/rcutree.h +16 −11
  23. kernel/rcutree_plugin.h +276 −174
  24. kernel/rcutree_trace.c +6 −6
  25. kernel/srcu.c +23 −10
  26. lib/Kconfig.debug +24 −0
  27. net/ipv4/cipso_ipv4.c +3 −8
  28. net/ipv4/ip_sockglue.c +1 −6
  29. net/mac80211/mesh_pathtbl.c +1 −7
Documentation/RCU/RTFP.txt+1745−157
@@ -1,9 +1,10 @@
1−Read the F-ing Papers!
1+Read the Fscking Papers!
22
33
44 This document describes RCU-related publications, and is followed by
55 the corresponding bibtex entries. A number of the publications may
6−be found at http://www.rdrop.com/users/paulmck/RCU/.
6+be found at http://www.rdrop.com/users/paulmck/RCU/. For others, browsers
7+and search engines will usually find what you are looking for.
78
89 The first thing resembling RCU was published in 1980, when Kung and Lehman
910 [Kung80] recommended use of a garbage collector to defer destruction
@@ -160,7 +161,26 @@ which Mathieu Desnoyers is now maintaining [MathieuDesnoyers2009URCU] · 150 unmodified lines
160161 [MathieuDesnoyersPhD]. TINY_RCU [PaulEMcKenney2009BloatWatchRCU] made
161162 its appearance, as did expedited RCU [PaulEMcKenney2009expeditedRCU].
162163 The problem of resizeable RCU-protected hash tables may now be on a path
163−to a solution [JoshTriplett2009RPHash].
164+to a solution [JoshTriplett2009RPHash]. A few academic researchers are now
165+using RCU to solve their parallel problems [HariKannan2009DynamicAnalysisRCU].
166+
167+2010 produced a simpler preemptible-RCU implementation
168+based on TREE_RCU [PaulEMcKenney2010SimpleOptRCU], lockdep-RCU
169+[PaulEMcKenney2010LockdepRCU], another resizeable RCU-protected hash
170+table [HerbertXu2010RCUResizeHash] (this one consuming more memory,
171+but allowing arbitrary changes in hash function, as required for DoS
172+avoidance in the networking code), realization of the 2009 RCU-protected
173+hash table with atomic node move [JoshTriplett2010RPHash], an update on
174+the RCU API [PaulEMcKenney2010RCUAPI].
175+
176+2011 marked the inclusion of Nick Piggin's fully lockless dentry search
177+[LinusTorvalds2011Linux2:6:38:rc1:NPigginVFS], an RCU-protected red-black
178+tree using software transactional memory to protect concurrent updates
179+(strange, but true!) [PhilHoward2011RCUTMRBTree], yet another variant of
180+RCU-protected resizeable hash tables [Triplett:2011:RPHash], the 3.0 RCU
181+trainwreck [PaulEMcKenney2011RCU3.0trainwreck], and Neil Brown's "Meet the
182+Lockers" LWN article [NeilBrown2011MeetTheLockers].
183+
164184
165185 Bibtex Entries
166186
@@ -173,6 +193,14 @@ Bibtex Entries · 6 unmodified lines
173193 ,volume="5"
174194 ,number="3"
175195 ,pages="354-382"
196+,note="Available:
197+\url{http://portal.acm.org/citation.cfm?id=320619&dl=GUIDE,}
198+[Viewed December 3, 2007]"
199+,annotation={
200+ Use garbage collector to clean up data after everyone is done with it.
201+ .
202+ Oldest use of something vaguely resembling RCU that I have found.
203+}
176204 }
177205
178206 @techreport{Manber82
@@ -184,6 +212,31 @@ Bibtex Entries · 5 unmodified lines
184212 ,number="82-01-01"
185213 ,month="January"
186214 ,pages="28"
215+,annotation={
216+ .
217+ Superseded by Manber84.
218+ .
219+ Describes concurrent AVL tree implementation. Uses a
220+ garbage-collection mechanism to handle concurrent use and deletion
221+ of nodes in the tree, but lacks the summary-of-execution-history
222+ concept of read-copy locking.
223+ .
224+ Keeps full list of processes that were active when a given
225+ node was to be deleted, and waits until all such processes have
226+ -terminated- before allowing this node to be reused. This is
227+ not described in great detail -- one could imagine using process
228+ IDs for this if the ID space was large enough that overlapping
229+ never occurred.
230+ .
231+ This restriction makes this algorithm unsuitable for use in
232+ systems comprised of long-lived processes. It also produces
233+ completely unacceptable overhead in systems with large numbers
234+ of processes. Finally, it is specific to AVL trees.
235+ .
236+ Cites Kung80, so not an independent invention, but the first
237+ RCU-like usage that does not rely on an automatic garbage
238+ collector.
239+}
187240 }
188241
189242 @article{Manber84
@@ -195,6 +248,74 @@ Bibtex Entries · 5 unmodified lines
195248 ,volume="9"
196249 ,number="3"
197250 ,pages="439-455"
251+,annotation={
252+ Describes concurrent AVL tree implementation. Uses a
253+ garbage-collection mechanism to handle concurrent use and deletion
254+ of nodes in the tree, but lacks the summary-of-execution-history
255+ concept of read-copy locking.
256+ .
257+ Keeps full list of processes that were active when a given
258+ node was to be deleted, and waits until all such processes have
259+ -terminated- before allowing this node to be reused. This is
260+ not described in great detail -- one could imagine using process
261+ IDs for this if the ID space was large enough that overlapping
262+ never occurred.
263+ .
264+ This restriction makes this algorithm unsuitable for use in
265+ systems comprised of long-lived processes. It also produces
266+ completely unacceptable overhead in systems with large numbers
267+ of processes. Finally, it is specific to AVL trees.
268+}
269+}
270+
271+@Conference{RichardRashid87a
272+,Author="Richard Rashid and Avadis Tevanian and Michael Young and
273+David Golub and Robert Baron and David Black and William Bolosky and
274+Jonathan Chew"
275+,Title="Machine-Independent Virtual Memory Management for Paged
276+Uniprocessor and Multiprocessor Architectures"
277+,Booktitle="{2\textsuperscript{nd} Symposium on Architectural Support
278+for Programming Languages and Operating Systems}"
279+,Publisher="Association for Computing Machinery"
280+,Month="October"
281+,Year="1987"
282+,pages="31-39"
283+,Address="Palo Alto, CA"
284+,note="Available:
285+\url{http://www.cse.ucsc.edu/~randal/221/rashid-machvm.pdf}
286+[Viewed February 17, 2005]"
287+,annotation={
288+ Describes lazy TLB flush, where one waits for each CPU to pass
289+ through a scheduling-clock interrupt before reusing a given range
290+ of virtual address. Does not describe how one determines that
291+ all CPUs have in fact taken such an interrupt, though there are
292+ no shortage of straightforward methods for accomplishing this.
293+ .
294+ Note that it does not make sense to just wait a fixed amount of
295+ time, since a given CPU might have interrupts disabled for an
296+ extended amount of time.
297+}
298+}
299+
300+@article{BarbaraLiskov1988ArgusCACM
301+,author = {Barbara Liskov}
302+,title = {Distributed programming in {Argus}}
303+,journal = {Commun. ACM}
304+,volume = {31}
305+,number = {3}
306+,year = {1988}
307+,issn = {0001-0782}
308+,pages = {300--312}
309+,doi = {http://doi.acm.org/10.1145/42392.42399}
310+,publisher = {ACM}
311+,address = {New York, NY, USA}
312+,annotation= {
313+ At the top of page 307: "Conflicts with deposits and withdrawals
314+ are necessary if the reported total is to be up to date. They
315+ could be avoided by having total return a sum that is slightly
316+ out of date." Relies on semantics -- approximate numerical
317+ values sometimes OK.
318+}
198319 }
199320
200321 @techreport{Hennessy89
@@ -216,6 +337,13 @@ Bibtex Entries · 15 unmodified lines
216337 ,year="1990"
217338 ,number="CS-TR-2222.1"
218339 ,month="June"
340+,annotation={
341+ Concurrent access to skip lists. Has both weak and strong search.
342+ Uses concept of ``garbage queue'', but has no real way of cleaning
343+ the garbage efficiently.
344+ .
345+ Appears to be an independent invention of an RCU-like mechanism.
346+}
219347 }
220348
221349 @Book{Adams91
@@ -223,20 +351,15 @@ Bibtex Entries · 1 unmodified line
223351 ,title="Concurrent Programming, Principles, and Practices"
224352 ,Publisher="Benjamin Cummins"
225353 ,Year="1991"
354+,annotation={
355+ Has a few paragraphs describing ``chaotic relaxation'', a
356+ numerical analysis technique that allows multiprocessors to
357+ avoid synchronization overhead by using possibly-stale data.
358+ .
359+ Seems like this is descended from yet another independent
360+ invention of RCU-like function -- but this is restricted
361+ in that reclamation is not necessary.
226362 }
227−
228−@phdthesis{HMassalinPhD
229−,author="H. Massalin"
230−,title="Synthesis: An Efficient Implementation of Fundamental Operating
231−System Services"
232−,school="Columbia University"
233−,address="New York, NY"
234−,year="1992"
235−,annotation="
236− Mondo optimizing compiler.
237− Wait-free stuff.
238− Good advice: defer work to avoid synchronization.
239−"
240363 }
241364
242365 @unpublished{Jacobson93
@@ -244,7 +367,13 @@ System Services" · 1 unmodified line
244367 ,title="Avoid Read-Side Locking Via Delayed Free"
245368 ,year="1993"
246369 ,month="September"
247−,note="Verbal discussion"
370+,note="private communication"
371+,annotation={
372+ Use fixed time delay to approximate grace period. Very simple,
373+ but subject to random memory corruption under heavy load.
374+ .
375+ Independent invention of RCU-like mechanism.
376+}
248377 }
249378
250379 @Conference{AjuJohn95
@@ -256,6 +385,17 @@ System Services" · 5 unmodified lines
256385 ,Year="1995"
257386 ,pages="11-23"
258387 ,Address="New Orleans, LA"
388+,note="Available:
389+\url{https://www.usenix.org/publications/library/proceedings/neworl/full_papers/john.a}
390+[Viewed October 1, 2010]"
391+,annotation={
392+ Age vnodes out of the cache, and have a fixed time set by a kernel
393+ parameter. Not clear that all races were in fact correctly handled.
394+ Used a 20-minute time by default, which would most definitely not
395+ be suitable during DoS attacks or virus scans.
396+ .
397+ Apparently independent invention of RCU-like mechanism.
398+}
259399 }
260400
261401 @conference{Pu95a,
@@ -301,31 +441,47 @@ Utilizing Execution History and Thread Monitoring" · 39 unmodified lines
301441 ,institution="US Patent and Trademark Office"
302442 ,address="Washington, DC"
303443 ,year="1995"
304−,number="US Patent 5,442,758 (contributed under GPL)"
444+,number="US Patent 5,442,758"
305445 ,month="August"
446+,annotation={
447+ Describes the parallel RCU infrastructure. Includes NUMA aspect
448+ (structure of bitmap can reflect bus structure of computer system).
449+ .
450+ Another independent invention of an RCU-like mechanism, but the
451+ "real" RCU this time!
452+}
306453 }
307454
308455 @techreport{Slingwine97
309456 ,author="John D. Slingwine and Paul E. McKenney"
310−,title="Method for maintaining data coherency using thread
311−activity summaries in a multicomputer system"
457+,title="Method for Maintaining Data Coherency Using Thread Activity
458+Summaries in a Multicomputer System"
312459 ,institution="US Patent and Trademark Office"
313460 ,address="Washington, DC"
314461 ,year="1997"
315−,number="US Patent 5,608,893 (contributed under GPL)"
462+,number="US Patent 5,608,893"
316463 ,month="March"
464+,pages="19"
465+,annotation={
466+ Describes use of RCU to synchronize data between a pair of
467+ SMP/NUMA computer systems.
468+}
317469 }
318470
319471 @techreport{Slingwine98
320472 ,author="John D. Slingwine and Paul E. McKenney"
321−,title="Apparatus and method for achieving reduced overhead
322−mutual exclusion and maintaining coherency in a multiprocessor
323−system utilizing execution history and thread monitoring"
473+,title="Apparatus and Method for Achieving Reduced Overhead Mutual
474+Exclusion and Maintaining Coherency in a Multiprocessor System
475+Utilizing Execution History and Thread Monitoring"
324476 ,institution="US Patent and Trademark Office"
325477 ,address="Washington, DC"
326478 ,year="1998"
327−,number="US Patent 5,727,209 (contributed under GPL)"
479+,number="US Patent 5,727,209"
328480 ,month="March"
481+,annotation={
482+ Describes doing an atomic update by copying the data item and
483+ then substituting it into the data structure.
484+}
329485 }
330486
331487 @Conference{McKenney98
@@ -337,6 +493,15 @@ Problems" · 5 unmodified lines
337493 ,Year="1998"
338494 ,pages="509-518"
339495 ,Address="Las Vegas, NV"
496+,note="Available:
497+\url{http://www.rdrop.com/users/paulmck/RCU/rclockpdcsproof.pdf}
498+[Viewed December 3, 2007]"
499+,annotation={
500+ Describes and analyzes RCU mechanism in DYNIX/ptx. Describes
501+ application to linked list update and log-buffer flushing.
502+ Defines 'quiescent state'. Includes both measured and analytic
503+ evaluation.
504+}
340505 }
341506
342507 @Conference{Gamsa99
@@ -349,18 +514,76 @@ Operating System Design and Implementation}" · 6 unmodified lines
349514 ,Year="1999"
350515 ,pages="87-100"
351516 ,Address="New Orleans, LA"
517+,note="Available:
518+\url{http://www.usenix.org/events/osdi99/full_papers/gamsa/gamsa.pdf}
519+[Viewed August 30, 2006]"
520+,annotation={
521+ Use of RCU-like facility in K42/Tornado. Another independent
522+ invention of RCU.
523+ See especially pages 7-9 (Section 5).
524+}
525+}
526+
527+@unpublished{RustyRussell2000a
528+,Author="Rusty Russell"
529+,Title="Re: modular net drivers"
530+,month="June"
531+,year="2000"
532+,day="23"
533+,note="Available:
534+\url{http://oss.sgi.com/projects/netdev/archive/2000-06/msg00250.html}
535+[Viewed April 10, 2006]"
536+,annotation={
537+ Proto-RCU proposal from Phil Rumpf and Rusty Russell.
538+ Yet another independent invention of RCU.
539+ Outline of algorithm to unload modules...
540+ .
541+ Appeared on net-dev mailing list.
542+}
543+}
544+
545+@unpublished{RustyRussell2000b
546+,Author="Rusty Russell"
547+,Title="Re: modular net drivers"
548+,month="June"
549+,year="2000"
550+,day="24"
551+,note="Available:
552+\url{http://oss.sgi.com/projects/netdev/archive/2000-06/msg00254.html}
553+[Viewed April 10, 2006]"
554+,annotation={
555+ Proto-RCU proposal from Phil Rumpf and Rusty Russell.
556+ .
557+ Appeared on net-dev mailing list.
558+}
559+}
560+
561+@unpublished{McKenney01b
562+,Author="Paul E. McKenney and Dipankar Sarma"
563+,Title="Read-Copy Update Mutual Exclusion in {Linux}"
564+,month="February"
565+,year="2001"
566+,note="Available:
567+\url{http://lse.sourceforge.net/locking/rcu/rcupdate_doc.html}
568+[Viewed October 18, 2004]"
569+,annotation={
570+ Prototypical Linux documentation for RCU.
571+}
352572 }
353573
354574 @techreport{Slingwine01
355575 ,author="John D. Slingwine and Paul E. McKenney"
356−,title="Apparatus and method for achieving reduced overhead
357−mutual exclusion and maintaining coherency in a multiprocessor
358−system utilizing execution history and thread monitoring"
576+,title="Apparatus and Method for Achieving Reduced Overhead Mutual
577+Exclusion and Maintaining Coherency in a Multiprocessor System
578+Utilizing Execution History and Thread Monitoring"
359579 ,institution="US Patent and Trademark Office"
360580 ,address="Washington, DC"
361581 ,year="2001"
362−,number="US Patent 5,219,690 (contributed under GPL)"
582+,number="US Patent 6,219,690"
363583 ,month="April"
584+,annotation={
585+ 'Change in mode' aspect of RCU. Can be thought of as a lazy barrier.
586+}
364587 }
365588
366589 @Conference{McKenney01a
@@ -372,14 +595,61 @@ Orran Krieger and Rusty Russell and Dipankar Sarma and Maneesh Soni" · 5 unmodified lines
372595 ,Year="2001"
373596 ,note="Available:
374597 \url{http://www.linuxsymposium.org/2001/abstracts/readcopy.php}
375−\url{http://www.rdrop.com/users/paulmck/rclock/rclock_OLS.2001.05.01c.pdf}
598+\url{http://www.rdrop.com/users/paulmck/RCU/rclock_OLS.2001.05.01c.pdf}
376599 [Viewed June 23, 2004]"
377−annotation="
378−Described RCU, and presented some patches implementing and using it in
379−the Linux kernel.
600+,annotation={
601+ Described RCU, and presented some patches implementing and using
602+ it in the Linux kernel.
603+}
604+}
605+
606+@unpublished{McKenney01f
607+,Author="Paul E. McKenney"
608+,Title="{RFC:} patch to allow lock-free traversal of lists with insertion"
609+,month="October"
610+,year="2001"
611+,note="Available:
612+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=100259266316456&w=2}
613+[Viewed June 23, 2004]"
614+,annotation="
615+ Memory-barrier and Alpha thread. 100 messages, not too bad...
616+"
617+}
618+
619+@unpublished{Spraul01
620+,Author="Manfred Spraul"
621+,Title="Re: {RFC:} patch to allow lock-free traversal of lists with insertion"
622+,month="October"
623+,year="2001"
624+,note="Available:
625+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=100264675012867&w=2}
626+[Viewed June 23, 2004]"
627+,annotation="
628+ Suggested burying memory barriers in Linux's list-manipulation
629+ primitives.
380630 "
381631 }
382632
633+@unpublished{LinusTorvalds2001a
634+,Author="Linus Torvalds"
635+,Title="{Re:} {[Lse-tech]} {Re:} {RFC:} patch to allow lock-free traversal of lists with insertion"
636+,month="October"
637+,year="2001"
638+,note="Available:
639+\url{http://lkml.org/lkml/2001/10/13/105}
640+[Viewed August 21, 2004]"
641+}
642+
643+@unpublished{Blanchard02a
644+,Author="Anton Blanchard"
645+,Title="some RCU dcache and ratcache results"
646+,month="March"
647+,year="2002"
648+,note="Available:
649+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=101637107412972&w=2}
650+[Viewed October 18, 2004]"
651+}
652+
383653 @Conference{Linder02a
384654 ,Author="Hanna Linder and Dipankar Sarma and Maneesh Soni"
385655 ,Title="Scalability of the Directory Entry Cache"
@@ -387,6 +657,10 @@ the Linux kernel. · 1 unmodified line
387657 ,Month="June"
388658 ,Year="2002"
389659 ,pages="289-300"
660+,annotation="
661+ Measured scalability of Linux 2.4 kernel's directory-entry cache
662+ (dcache), and measured some scalability enhancements.
663+"
390664 }
391665
392666 @Conference{McKenney02a
@@ -400,49 +674,76 @@ Andrea Arcangeli and Andi Kleen and Orran Krieger and Rusty Russell" · 7 unmodified lines
400674 ,note="Available:
401675 \url{http://www.linux.org.uk/~ajh/ols2002_proceedings.pdf.gz}
402676 [Viewed June 23, 2004]"
677+,annotation="
678+ Presented and compared a number of RCU implementations for the
679+ Linux kernel.
680+"
403681 }
404682
405−@conference{Michael02a
406−,author="Maged M. Michael"
407−,title="Safe Memory Reclamation for Dynamic Lock-Free Objects Using Atomic
408−Reads and Writes"
409−,Year="2002"
410−,Month="August"
411−,booktitle="{Proceedings of the 21\textsuperscript{st} Annual ACM
412−Symposium on Principles of Distributed Computing}"
413−,pages="21-30"
683+@unpublished{Sarma02a
684+,Author="Dipankar Sarma"
685+,Title="specweb99: dcache scalability results"
686+,month="July"
687+,year="2002"
688+,note="Available:
689+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=102645767914212&w=2}
690+[Viewed June 23, 2004]"
414691 ,annotation="
415− Each thread keeps an array of pointers to items that it is
416− currently referencing. Sort of an inside-out garbage collection
417− mechanism, but one that requires the accessing code to explicitly
418− state its needs. Also requires read-side memory barriers on
419− most architectures.
692+ Compare fastwalk and RCU for dcache. RCU won.
420693 "
421694 }
422695
423−@conference{Michael02b
424−,author="Maged M. Michael"
425−,title="High Performance Dynamic Lock-Free Hash Tables and List-Based Sets"
426−,Year="2002"
427−,Month="August"
428−,booktitle="{Proceedings of the 14\textsuperscript{th} Annual ACM
429−Symposium on Parallel
430−Algorithms and Architecture}"
431−,pages="73-82"
696+@unpublished{Barbieri02
697+,Author="Luca Barbieri"
698+,Title="Re: {[PATCH]} Initial support for struct {vfs\_cred}"
699+,month="August"
700+,year="2002"
701+,note="Available:
702+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=103082050621241&w=2}
703+[Viewed: June 23, 2004]"
432704 ,annotation="
433− Like the title says...
705+ Suggested RCU for vfs\_shared\_cred.
434706 "
435707 }
436708
437−@InProceedings{HerlihyLM02
438−,author={Maurice Herlihy and Victor Luchangco and Mark Moir}
439−,title="The Repeat Offender Problem: A Mechanism for Supporting Dynamic-Sized,
440−Lock-Free Data Structures"
441−,booktitle={Proceedings of 16\textsuperscript{th} International
442−Symposium on Distributed Computing}
443−,year=2002
709+@unpublished{Dickins02a
710+,author="Hugh Dickins"
711+,title="Use RCU for System-V IPC"
712+,year="2002"
713+,month="October"
714+,note="private communication"
715+}
716+
717+@unpublished{Sarma02b
718+,Author="Dipankar Sarma"
719+,Title="Some dcache\_rcu benchmark numbers"
444720 ,month="October"
445−,pages="339-353"
721+,year="2002"
722+,note="Available:
723+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=103462075416638&w=2}
724+[Viewed June 23, 2004]"
725+,annotation="
726+ Performance of dcache RCU on kernbench for 16x NUMA-Q and 1x,
727+ 2x, and 4x systems. RCU does no harm, and helps on 16x.
728+"
729+}
730+
731+@unpublished{LinusTorvalds2003a
732+,Author="Linus Torvalds"
733+,Title="Re: {[PATCH]} small fixes in brlock.h"
734+,month="March"
735+,year="2003"
736+,note="Available:
737+\url{http://lkml.org/lkml/2003/3/9/205}
738+[Viewed March 13, 2006]"
739+,annotation="
740+ Linus suggests replacing brlock with RCU and/or seqlocks:
741+ .
742+ 'It's entirely possible that the current user could be replaced
743+ by RCU and/or seqlocks, and we could get rid of brlocks entirely.'
744+ .
745+ Steve Hemminger responds by replacing them with RCU.
746+"
446747 }
447748
448749 @article{Appavoo03a
@@ -457,6 +758,20 @@ B. Rosenburg and M. Stumm and J. Xenidis" · 8 unmodified lines
457758 ,volume="42"
458759 ,number="1"
459760 ,pages="60-76"
761+,annotation="
762+ Use of RCU to enable hot-swapping for autonomic behavior in K42.
763+"
764+}
765+
766+@unpublished{Seigh03
767+,author="Joseph W. {Seigh II}"
768+,title="Read Copy Update"
769+,Year="2003"
770+,Month="March"
771+,note="email correspondence"
772+,annotation="
773+ Described the relationship of the VM/XA passive serialization to RCU.
774+"
460775 }
461776
462777 @Conference{Arcangeli03
@@ -470,6 +785,27 @@ Dipankar Sarma" · 7 unmodified lines
470785 ,year="2003"
471786 ,month="June"
472787 ,pages="297-310"
788+,note="Available:
789+\url{http://www.rdrop.com/users/paulmck/RCU/rcu.FREENIX.2003.06.14.pdf}
790+[Viewed November 21, 2007]"
791+,annotation="
792+ Compared updated RCU implementations for the Linux kernel, and
793+ described System V IPC use of RCU, including order-of-magnitude
794+ performance improvements.
795+"
796+}
797+
798+@Conference{Soules03a
799+,Author="Craig A. N. Soules and Jonathan Appavoo and Kevin Hui and
800+Dilma {Da Silva} and Gregory R. Ganger and Orran Krieger and
801+Michael Stumm and Robert W. Wisniewski and Marc Auslander and
802+Michal Ostrowski and Bryan Rosenburg and Jimi Xenidis"
803+,Title="System Support for Online Reconfiguration"
804+,Booktitle="Proceedings of the 2003 USENIX Annual Technical Conference"
805+,Publisher="USENIX Association"
806+,year="2003"
807+,month="June"
808+,pages="141-154"
473809 }
474810
475811 @article{McKenney03a
@@ -481,6 +817,22 @@ Dipankar Sarma" · 5 unmodified lines
481817 ,volume="1"
482818 ,number="114"
483819 ,pages="18-26"
820+,note="Available:
821+\url{http://www.linuxjournal.com/article/6993}
822+[Viewed November 14, 2007]"
823+,annotation="
824+ Reader-friendly intro to RCU, with the infamous old-man-and-brat
825+ cartoon.
826+"
827+}
828+
829+@unpublished{Sarma03a
830+,Author="Dipankar Sarma"
831+,Title="RCU low latency patches"
832+,month="December"
833+,year="2003"
834+,note="Message ID: 20031222180114.GA2248@in.ibm.com"
835+,annotation="dipankar/ct.2004.03.27/RCUll.2003.12.22.patch"
484836 }
485837
486838 @techreport{Friedberg03a
@@ -489,9 +841,14 @@ Dipankar Sarma" · 2 unmodified lines
489841 ,institution="US Patent and Trademark Office"
490842 ,address="Washington, DC"
491843 ,year="2003"
492−,number="US Patent 6,662,184 (contributed under GPL)"
844+,number="US Patent 6,662,184"
493845 ,month="December"
494846 ,pages="112"
847+,annotation="
848+ Applies RCU to a wildcard-search Patricia tree in order to permit
849+ synchronization-free lookup. RCU is used to retain removed nodes
850+ for a grace period before freeing them.
851+"
495852 }
496853
497854 @article{McKenney04a
@@ -503,6 +860,12 @@ Dipankar Sarma" · 5 unmodified lines
503860 ,volume="1"
504861 ,number="118"
505862 ,pages="38-46"
863+,note="Available:
864+\url{http://www.linuxjournal.com/node/7124}
865+[Viewed December 26, 2010]"
866+,annotation="
867+ Reader friendly intro to dcache and RCU.
868+"
506869 }
507870
508871 @Conference{McKenney04b
@@ -514,152 +877,824 @@ Dipankar Sarma" · 5 unmodified lines
514877 ,Address="Adelaide, Australia"
515878 ,note="Available:
516879 \url{http://www.linux.org.au/conf/2004/abstracts.html#90}
517−\url{http://www.rdrop.com/users/paulmck/rclock/lockperf.2004.01.17a.pdf}
880+\url{http://www.rdrop.com/users/paulmck/RCU/lockperf.2004.01.17a.pdf}
518881 [Viewed June 23, 2004]"
882+,annotation="
883+ Compares performance of RCU to that of other locking primitives
884+ over a number of CPUs (x86, Opteron, Itanium, and PPC).
885+"
519886 }
520887
521−@phdthesis{PaulEdwardMcKenneyPhD
522−,author="Paul E. McKenney"
523−,title="Exploiting Deferred Destruction:
524−An Analysis of Read-Copy-Update Techniques
525−in Operating System Kernels"
526−,school="OGI School of Science and Engineering at
527−Oregon Health and Sciences University"
888+@unpublished{Sarma04a
889+,Author="Dipankar Sarma"
890+,Title="{[PATCH]} {RCU} for low latency (experimental)"
891+,month="March"
892+,year="2004"
893+,note="\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=108003746402892&w=2}"
894+,annotation="Head of thread: dipankar/2004.03.23/rcu-low-lat.1.patch"
895+}
896+
897+@unpublished{Sarma04b
898+,Author="Dipankar Sarma"
899+,Title="Re: {[PATCH]} {RCU} for low latency (experimental)"
900+,month="March"
901+,year="2004"
902+,note="\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=108016474829546&w=2}"
903+,annotation="dipankar/rcuth.2004.03.24/rcu-throttle.patch"
904+}
905+
906+@unpublished{Spraul04a
907+,Author="Manfred Spraul"
908+,Title="[RFC] 0/5 rcu lock update"
909+,month="May"
528910 ,year="2004"
529911 ,note="Available:
530−\url{http://www.rdrop.com/users/paulmck/RCU/RCUdissertation.2004.07.14e1.pdf}
531−[Viewed October 15, 2004]"
912+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=108546407726602&w=2}
913+[Viewed June 23, 2004]"
914+,annotation="
915+ Hierarchical-bitmap patch for RCU infrastructure.
916+"
917+}
918+
919+@unpublished{Steiner04a
920+,Author="Jack Steiner"
921+,Title="Re: [Lse-tech] [RFC, PATCH] 1/5 rcu lock update:
922+Add per-cpu batch counter"
923+,month="May"
924+,year="2004"
925+,note="Available:
926+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=108551764515332&w=2}
927+[Viewed June 23, 2004]"
928+,annotation={
929+ RCU runs reasonably on a 512-CPU SGI using Manfred Spraul's patches,
930+ which may be found at:
931+ https://lkml.org/lkml/2004/5/20/49 (split vars into cachelines)
932+ https://lkml.org/lkml/2004/5/22/114 (cpu_quiet() patch)
933+ https://lkml.org/lkml/2004/5/25/24 (0/5)
934+ https://lkml.org/lkml/2004/5/25/23 (1/5)
935+ https://lkml.org/lkml/2004/5/25/265 (works for Jack)
936+ https://lkml.org/lkml/2004/5/25/20 (2/5)
937+ https://lkml.org/lkml/2004/5/25/22 (3/5)
938+ https://lkml.org/lkml/2004/5/25/19 (4/5)
939+ https://lkml.org/lkml/2004/5/25/21 (5/5)
940+}
532941 }
533942
534943 @Conference{Sarma04c
535944 ,Author="Dipankar Sarma and Paul E. McKenney"
536−,Title="Making RCU Safe for Deep Sub-Millisecond Response Realtime Applications"
945+,Title="Making {RCU} Safe for Deep Sub-Millisecond Response
946+Realtime Applications"
537947 ,Booktitle="Proceedings of the 2004 USENIX Annual Technical Conference
538948 (FREENIX Track)"
539949 ,Publisher="USENIX Association"
540950 ,year="2004"
541951 ,month="June"
542952 ,pages="182-191"
953+,annotation="
954+ Describes and compares a number of modifications to the Linux RCU
955+ implementation that make it friendly to realtime applications.
956+"
543957 }
544958
545−@unpublished{JamesMorris04b
546−,Author="James Morris"
547−,Title="Recent Developments in {SELinux} Kernel Performance"
548−,month="December"
959+@phdthesis{PaulEdwardMcKenneyPhD
960+,author="Paul E. McKenney"
961+,title="Exploiting Deferred Destruction:
962+An Analysis of Read-Copy-Update Techniques
963+in Operating System Kernels"
964+,school="OGI School of Science and Engineering at
965+Oregon Health and Sciences University"
549966 ,year="2004"
550967 ,note="Available:
551−\url{http://www.livejournal.com/users/james_morris/2153.html}
552−[Viewed December 10, 2004]"
968+\url{http://www.rdrop.com/users/paulmck/RCU/RCUdissertation.2004.07.14e1.pdf}
969+[Viewed October 15, 2004]"
970+,annotation="
971+ Describes RCU implementations and presents design patterns
972+ corresponding to common uses of RCU in several operating-system
973+ kernels.
974+"
553975 }
554976
555−@unpublished{PaulMcKenney05a
556−,Author="Paul E. McKenney"
557−,Title="{[RFC]} {RCU} and {CONFIG\_PREEMPT\_RT} progress"
558−,month="May"
559−,year="2005"
977+@unpublished{PaulEMcKenney2004rcu:dereference
978+,Author="Dipankar Sarma"
979+,Title="{Re: RCU : Abstracted RCU dereferencing [5/5]}"
980+,month="August"
981+,year="2004"
560982 ,note="Available:
561−\url{http://lkml.org/lkml/2005/5/9/185}
562−[Viewed May 13, 2005]"
983+\url{http://lkml.org/lkml/2004/8/6/237}
984+[Viewed June 8, 2010]"
563985 ,annotation="
564− First publication of working lock-based deferred free patches
565− for the CONFIG_PREEMPT_RT environment.
986+ Introduce rcu_dereference().
566987 "
567988 }
568989
569−@conference{PaulMcKenney05b
570−,Author="Paul E. McKenney and Dipankar Sarma"
571−,Title="Towards Hard Realtime Response from the Linux Kernel on SMP Hardware"
572−,Booktitle="linux.conf.au 2005"
573−,month="April"
574−,year="2005"
575−,address="Canberra, Australia"
990+@unpublished{JimHouston04a
991+,Author="Jim Houston"
992+,Title="{[RFC\&PATCH] Alternative {RCU} implementation}"
993+,month="August"
994+,year="2004"
576995 ,note="Available:
577−\url{http://www.rdrop.com/users/paulmck/RCU/realtimeRCU.2005.04.23a.pdf}
578−[Viewed May 13, 2005]"
996+\url{http://lkml.org/lkml/2004/8/30/87}
997+[Viewed February 17, 2005]"
579998 ,annotation="
580− Realtime turns into making RCU yet more realtime friendly.
999+ Uses active code in rcu_read_lock() and rcu_read_unlock() to
1000+ make RCU happen, allowing RCU to function on CPUs that do not
1001+ receive a scheduling-clock interrupt.
5811002 "
5821003 }
5831004
584−@conference{ThomasEHart2006a
585−,Author="Thomas E. Hart and Paul E. McKenney and Angela Demke Brown"
586−,Title="Making Lockless Synchronization Fast: Performance Implications
587−of Memory Reclamation"
588−,Booktitle="20\textsuperscript{th} {IEEE} International Parallel and
589−Distributed Processing Symposium"
590−,month="April"
591−,year="2006"
592−,day="25-29"
593−,address="Rhodes, Greece"
1005+@unpublished{TomHart04a
1006+,Author="Thomas E. Hart"
1007+,Title="Master's Thesis: Applying Lock-free Techniques to the {Linux} Kernel"
1008+,month="October"
1009+,year="2004"
1010+,note="Available:
1011+\url{http://www.cs.toronto.edu/~tomhart/masters_thesis.html}
1012+[Viewed October 15, 2004]"
5941013 ,annotation="
595− Compares QSBR (AKA "classic RCU"), HPBR, EBR, and lock-free
596− reference counting.
1014+ Proposes comparing RCU to lock-free methods for the Linux kernel.
5971015 "
5981016 }
5991017
600−@Conference{PaulEMcKenney2006b
601−,Author="Paul E. McKenney and Dipankar Sarma and Ingo Molnar and
602−Suparna Bhattacharya"
603−,Title="Extending RCU for Realtime and Embedded Workloads"
604−,Booktitle="{Ottawa Linux Symposium}"
605−,Month="July"
606−,Year="2006"
607−,pages="v2 123-138"
1018+@unpublished{Vaddagiri04a
1019+,Author="Srivatsa Vaddagiri"
1020+,Title="Subject: [RFC] Use RCU for tcp\_ehash lookup"
1021+,month="October"
1022+,year="2004"
6081023 ,note="Available:
609−\url{http://www.linuxsymposium.org/2006/index_2006.php}
610−\url{http://www.rdrop.com/users/paulmck/RCU/OLSrtRCU.2006.08.11a.pdf}
611−[Viewed January 1, 2007]"
1024+\url{http://marc.theaimsgroup.com/?t=109395731700004&r=1&w=2}
1025+[Viewed October 18, 2004]"
6121026 ,annotation="
613− Described how to improve the -rt implementation of realtime RCU.
1027+ Srivatsa's RCU patch for tcp_ehash lookup.
6141028 "
6151029 }
6161030
617−@unpublished{PaulEMcKenney2006c
618−,Author="Paul E. McKenney"
619−,Title="Sleepable {RCU}"
1031+@unpublished{Thirumalai04a
1032+,Author="Ravikiran Thirumalai"
1033+,Title="Subject: [patchset] Lockfree fd lookup 0 of 5"
6201034 ,month="October"
621−,day="9"
622−,year="2006"
1035+,year="2004"
6231036 ,note="Available:
624−\url{http://lwn.net/Articles/202847/}
625−Revised:
626−\url{http://www.rdrop.com/users/paulmck/RCU/srcu.2007.01.14a.pdf}
627−[Viewed August 21, 2006]"
1037+\url{http://marc.theaimsgroup.com/?t=109144217400003&r=1&w=2}
1038+[Viewed October 18, 2004]"
6281039 ,annotation="
629− LWN article introducing SRCU.
1040+ Ravikiran's lockfree FD patch.
1041+"
1042+}
1043+
1044+@unpublished{Thirumalai04b
1045+,Author="Ravikiran Thirumalai"
1046+,Title="Subject: Re: [patchset] Lockfree fd lookup 0 of 5"
1047+,month="October"
1048+,year="2004"
1049+,note="Available:
1050+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=109152521410459&w=2}
1051+[Viewed October 18, 2004]"
1052+,annotation="
1053+ Ravikiran's lockfree FD patch.
1054+"
1055+}
1056+
1057+@unpublished{PaulEMcKenney2004rcu:assign:pointer
1058+,Author="Paul E. McKenney"
1059+,Title="{[PATCH 1/3] RCU: \url{rcu_assign_pointer()} removal of memory barriers}"
1060+,month="October"
1061+,year="2004"
1062+,note="Available:
1063+\url{http://lkml.org/lkml/2004/10/23/241}
1064+[Viewed June 8, 2010]"
1065+,annotation="
1066+ Introduce rcu_assign_pointer().
1067+"
1068+}
1069+
1070+@unpublished{JamesMorris04a
1071+,Author="James Morris"
1072+,Title="{[PATCH 2/3] SELinux} scalability - convert {AVC} to {RCU}"
1073+,day="15"
1074+,month="November"
1075+,year="2004"
1076+,note="Available:
1077+\url{http://marc.theaimsgroup.com/?l=linux-kernel&m=110054979416004&w=2}
1078+[Viewed December 10, 2004]"
1079+,annotation="
1080+ James Morris posts Kaigai Kohei's patch to LKML.
1081+"
1082+}
1083+
1084+@unpublished{JamesMorris04b
1085+,Author="James Morris"
1086+,Title="Recent Developments in {SELinux} Kernel Performance"
1087+,month="December"
1088+,year="2004"
1089+,note="Available:
1090+\url{http://www.livejournal.com/users/james_morris/2153.html}
1091+[Viewed December 10, 2004]"
1092+,annotation="
1093+ RCU helps SELinux performance. ;-) Made LWN.
1094+"
1095+}
1096+
1097+@unpublished{PaulMcKenney2005RCUSemantics
1098+,Author="Paul E. McKenney and Jonathan Walpole"
1099+,Title="{RCU} Semantics: A First Attempt"
1100+,month="January"
1101+,year="2005"
1102+,day="30"
1103+,note="Available:
1104+\url{http://www.rdrop.com/users/paulmck/RCU/rcu-semantics.2005.01.30a.pdf}
1105+[Viewed December 6, 2009]"
1106+,annotation="
1107+ Early derivation of RCU semantics.
1108+"
1109+}
1110+
1111+@unpublished{PaulMcKenney2005e
1112+,Author="Paul E. McKenney"
1113+,Title="Real-Time Preemption and {RCU}"
1114+,month="March"
1115+,year="2005"
1116+,day="17"
1117+,note="Available:
1118+\url{http://lkml.org/lkml/2005/3/17/199}
1119+[Viewed September 5, 2005]"
1120+,annotation="
1121+ First posting showing how RCU can be safely adapted for
1122+ preemptable RCU read side critical sections.
1123+"
1124+}
1125+
1126+@unpublished{EsbenNeilsen2005a
1127+,Author="Esben Neilsen"
1128+,Title="Re: Real-Time Preemption and {RCU}"
1129+,month="March"
1130+,year="2005"
1131+,day="18"
1132+,note="Available:
1133+\url{http://lkml.org/lkml/2005/3/18/122}
1134+[Viewed March 30, 2006]"
1135+,annotation="
1136+ Esben Neilsen suggests read-side suppression of grace-period
1137+ processing for crude-but-workable realtime RCU. The downside
1138+ is indefinite grace periods...But this is OK for experimentation
1139+ and testing.
1140+"
1141+}
1142+
1143+@unpublished{TomHart05a
1144+,Author="Thomas E. Hart and Paul E. McKenney and Angela Demke Brown"
1145+,Title="Efficient Memory Reclamation is Necessary for Fast Lock-Free
1146+Data Structures"
1147+,month="March"
1148+,year="2005"
1149+,note="Available:
1150+\url{ftp://ftp.cs.toronto.edu/csrg-technical-reports/515/}
1151+[Viewed March 4, 2005]"
1152+,annotation="
1153+ Comparison of RCU, QBSR, and EBSR. RCU wins for read-mostly
1154+ workloads. ;-)
1155+"
1156+}
1157+
1158+@unpublished{JonCorbet2005DeprecateSyncKernel
1159+,Author="Jonathan Corbet"
1160+,Title="API change: synchronize_kernel() deprecated"
1161+,month="May"
1162+,day="3"
1163+,year="2005"
1164+,note="Available:
1165+\url{http://lwn.net/Articles/134484/}
1166+[Viewed May 3, 2005]"
1167+,annotation="
1168+ Jon Corbet describes deprecation of synchronize_kernel()
1169+ in favor of synchronize_rcu() and synchronize_sched().
1170+"
1171+}
1172+
1173+@unpublished{PaulMcKenney05a
1174+,Author="Paul E. McKenney"
1175+,Title="{[RFC]} {RCU} and {CONFIG\_PREEMPT\_RT} progress"
1176+,month="May"
1177+,year="2005"
1178+,note="Available:
1179+\url{http://lkml.org/lkml/2005/5/9/185}
1180+[Viewed May 13, 2005]"
1181+,annotation="
1182+ First publication of working lock-based deferred free patches
1183+ for the CONFIG_PREEMPT_RT environment.
1184+"
1185+}
1186+
1187+@conference{PaulMcKenney05b
1188+,Author="Paul E. McKenney and Dipankar Sarma"
1189+,Title="Towards Hard Realtime Response from the {Linux} Kernel on {SMP} Hardware"
1190+,Booktitle="linux.conf.au 2005"
1191+,month="April"
1192+,year="2005"
1193+,address="Canberra, Australia"
1194+,note="Available:
1195+\url{http://www.rdrop.com/users/paulmck/RCU/realtimeRCU.2005.04.23a.pdf}
1196+[Viewed May 13, 2005]"
1197+,annotation="
1198+ Realtime turns into making RCU yet more realtime friendly.
1199+ http://lca2005.linux.org.au/Papers/Paul%20McKenney/Towards%20Hard%20Realtime%20Response%20from%20the%20Linux%20Kernel/LKS.2005.04.22a.pdf
1200+"
1201+}
1202+
1203+@unpublished{PaulEMcKenneyHomePage
1204+,Author="Paul E. McKenney"
1205+,Title="{Paul} {E.} {McKenney}"
1206+,month="May"
1207+,year="2005"
1208+,note="Available:
1209+\url{http://www.rdrop.com/users/paulmck/}
1210+[Viewed May 25, 2005]"
1211+,annotation="
1212+ Paul McKenney's home page.
1213+"
1214+}
1215+
1216+@unpublished{PaulEMcKenneyRCUPage
1217+,Author="Paul E. McKenney"
1218+,Title="Read-Copy Update {(RCU)}"
1219+,month="May"
1220+,year="2005"
1221+,note="Available:
1222+\url{http://www.rdrop.com/users/paulmck/RCU}
1223+[Viewed May 25, 2005]"
1224+,annotation="
1225+ Paul McKenney's RCU page.
1226+"
1227+}
1228+
1229+@unpublished{JosephSeigh2005a
1230+,Author="Joseph Seigh"
1231+,Title="{RCU}+{SMR} (hazard pointers)"
1232+,month="July"
1233+,year="2005"
1234+,note="Personal communication"
1235+,annotation="
1236+ Joe Seigh announcing his atomic-ptr-plus project.
1237+ http://sourceforge.net/projects/atomic-ptr-plus/
1238+"
1239+}
1240+
1241+@unpublished{JosephSeigh2005b
1242+,Author="Joseph Seigh"
1243+,Title="Lock-free synchronization primitives"
1244+,month="July"
1245+,day="6"
1246+,year="2005"
1247+,note="Available:
1248+\url{http://sourceforge.net/projects/atomic-ptr-plus/}
1249+[Viewed August 8, 2005]"
1250+,annotation="
1251+ Joe Seigh's atomic-ptr-plus project.
1252+"
1253+}
1254+
1255+@unpublished{PaulMcKenney2005c
1256+,Author="Paul E.McKenney"
1257+,Title="{[RFC,PATCH] RCU} and {CONFIG\_PREEMPT\_RT} sane patch"
1258+,month="August"
1259+,day="1"
1260+,year="2005"
1261+,note="Available:
1262+\url{http://lkml.org/lkml/2005/8/1/155}
1263+[Viewed March 14, 2006]"
1264+,annotation="
1265+ First operating counter-based realtime RCU patch posted to LKML.
1266+"
1267+}
1268+
1269+@unpublished{PaulMcKenney2005d
1270+,Author="Paul E. McKenney"
1271+,Title="Re: [Fwd: Re: [patch] Real-Time Preemption, -RT-2.6.13-rc4-V0.7.52-01]"
1272+,month="August"
1273+,day="8"
1274+,year="2005"
1275+,note="Available:
1276+\url{http://lkml.org/lkml/2005/8/8/108}
1277+[Viewed March 14, 2006]"
1278+,annotation="
1279+ First operating counter-based realtime RCU patch posted to LKML,
1280+ but fixed so that various unusual combinations of configuration
1281+ parameters all function properly.
1282+"
1283+}
1284+
1285+@unpublished{PaulMcKenney2005rcutorture
1286+,Author="Paul E. McKenney"
1287+,Title="{[PATCH]} {RCU} torture testing"
1288+,month="October"
1289+,day="1"
1290+,year="2005"
1291+,note="Available:
1292+\url{http://lkml.org/lkml/2005/10/1/70}
1293+[Viewed March 14, 2006]"
1294+,annotation="
1295+ First rcutorture patch.
1296+"
1297+}
1298+
1299+@conference{ThomasEHart2006a
1300+,Author="Thomas E. Hart and Paul E. McKenney and Angela Demke Brown"
1301+,Title="Making Lockless Synchronization Fast: Performance Implications
1302+of Memory Reclamation"
1303+,Booktitle="20\textsuperscript{th} {IEEE} International Parallel and
1304+Distributed Processing Symposium"
1305+,month="April"
1306+,year="2006"
1307+,day="25-29"
1308+,address="Rhodes, Greece"
1309+,note="Available:
1310+\url{http://www.rdrop.com/users/paulmck/RCU/hart_ipdps06.pdf}
1311+[Viewed April 28, 2008]"
1312+,annotation="
1313+ Compares QSBR, HPBR, EBR, and lock-free reference counting.
1314+ http://www.cs.toronto.edu/~tomhart/perflab/ipdps06.tgz
1315+"
1316+}
1317+
1318+@unpublished{NickPiggin2006radixtree
1319+,Author="Nick Piggin"
1320+,Title="[patch 3/3] radix-tree: {RCU} lockless readside"
1321+,month="June"
1322+,day="20"
1323+,year="2006"
1324+,note="Available:
1325+\url{http://lkml.org/lkml/2006/6/20/238}
1326+[Viewed March 25, 2008]"
1327+,annotation="
1328+ RCU-protected radix tree.
1329+"
1330+}
1331+
1332+@Conference{PaulEMcKenney2006b
1333+,Author="Paul E. McKenney and Dipankar Sarma and Ingo Molnar and
1334+Suparna Bhattacharya"
1335+,Title="Extending {RCU} for Realtime and Embedded Workloads"
1336+,Booktitle="{Ottawa Linux Symposium}"
1337+,Month="July"
1338+,Year="2006"
1339+,pages="v2 123-138"
1340+,note="Available:
1341+\url{http://www.linuxsymposium.org/2006/view_abstract.php?content_key=184}
1342+\url{http://www.rdrop.com/users/paulmck/RCU/OLSrtRCU.2006.08.11a.pdf}
1343+[Viewed January 1, 2007]"
1344+,annotation="
1345+ Described how to improve the -rt implementation of realtime RCU.
1346+"
1347+}
1348+
1349+@unpublished{WikipediaRCU
1350+,Author="Paul E. McKenney and Chris Purcell and Algae and Ben Schumin and
1351+Gaius Cornelius and Qwertyus and Neil Conway and Sbw and Blainster and
1352+Canis Rufus and Zoicon5 and Anome and Hal Eisen"
1353+,Title="Read-Copy Update"
1354+,month="July"
1355+,day="8"
1356+,year="2006"
1357+,note="Available:
1358+\url{http://en.wikipedia.org/wiki/Read-copy-update}
1359+[Viewed August 21, 2006]"
1360+,annotation="
1361+ Wikipedia RCU page as of July 8 2006.
1362+"
1363+}
1364+
1365+@Conference{NickPiggin2006LocklessPageCache
1366+,Author="Nick Piggin"
1367+,Title="A Lockless Pagecache in Linux---Introduction, Progress, Performance"
1368+,Booktitle="{Ottawa Linux Symposium}"
1369+,Month="July"
1370+,Year="2006"
1371+,pages="v2 249-254"
1372+,note="Available:
1373+\url{http://www.linuxsymposium.org/2006/view_abstract.php?content_key=184}
1374+[Viewed January 11, 2009]"
1375+,annotation="
1376+ Uses RCU-protected radix tree for a lockless page cache.
1377+"
1378+}
1379+
1380+@unpublished{PaulEMcKenney2006c
1381+,Author="Paul E. McKenney"
1382+,Title="Sleepable {RCU}"
1383+,month="October"
1384+,day="9"
1385+,year="2006"
1386+,note="Available:
1387+\url{http://lwn.net/Articles/202847/}
1388+Revised:
1389+\url{http://www.rdrop.com/users/paulmck/RCU/srcu.2007.01.14a.pdf}
1390+[Viewed August 21, 2006]"
1391+,annotation="
1392+ LWN article introducing SRCU.
6301393 "
6311394 }
6321395
6331396 @unpublished{RobertOlsson2006a
6341397 ,Author="Robert Olsson and Stefan Nilsson"
6351398 ,Title="{TRASH}: A dynamic {LC}-trie and hash data structure"
636−,month="August"
637−,day="18"
638−,year="2006"
1399+,month="August"
1400+,day="18"
1401+,year="2006"
1402+,note="Available:
1403+\url{http://www.nada.kth.se/~snilsson/publications/TRASH/trash.pdf}
1404+[Viewed March 4, 2011]"
1405+,annotation="
1406+ RCU-protected dynamic trie-hash combination.
1407+"
1408+}
1409+
1410+@unpublished{ChristophHellwig2006RCU2SRCU
1411+,Author="Christoph Hellwig"
1412+,Title="Re: {[-mm PATCH 1/4]} {RCU}: split classic rcu"
1413+,month="September"
1414+,day="28"
1415+,year="2006"
1416+,note="Available:
1417+\url{http://lkml.org/lkml/2006/9/28/160}
1418+[Viewed March 27, 2008]"
1419+}
1420+
1421+@unpublished{PaulEMcKenneyRCUusagePage
1422+,Author="Paul E. McKenney"
1423+,Title="{RCU} {Linux} Usage"
1424+,month="October"
1425+,year="2006"
1426+,note="Available:
1427+\url{http://www.rdrop.com/users/paulmck/RCU/linuxusage.html}
1428+[Viewed January 14, 2007]"
1429+,annotation="
1430+ Paul McKenney's RCU page showing graphs plotting Linux-kernel
1431+ usage of RCU.
1432+"
1433+}
1434+
1435+@unpublished{PaulEMcKenneyRCUusageRawDataPage
1436+,Author="Paul E. McKenney"
1437+,Title="Read-Copy Update {(RCU)} Usage in {Linux} Kernel"
1438+,month="October"
1439+,year="2006"
1440+,note="Available:
1441+\url{http://www.rdrop.com/users/paulmck/RCU/linuxusage/rculocktab.html}
1442+[Viewed January 14, 2007]"
1443+,annotation="
1444+ Paul McKenney's RCU page showing Linux usage of RCU in tabular
1445+ form, with links to corresponding cscope databases.
1446+"
1447+}
1448+
1449+@unpublished{GauthamShenoy2006RCUrwlock
1450+,Author="Gautham R. Shenoy"
1451+,Title="[PATCH 4/5] lock\_cpu\_hotplug: Redesign - Lightweight implementation of lock\_cpu\_hotplug"
1452+,month="October"
1453+,year="2006"
1454+,day=26
1455+,note="Available:
1456+\url{http://lkml.org/lkml/2006/10/26/73}
1457+[Viewed January 26, 2009]"
1458+,annotation="
1459+ RCU-based reader-writer lock that allows readers to proceed with
1460+ no memory barriers or atomic instruction in absence of writers.
1461+ If writer do show up, readers must of course wait as required by
1462+ the semantics of reader-writer locking. This is a recursive
1463+ lock.
1464+"
1465+}
1466+
1467+@unpublished{JensAxboe2006SlowSRCU
1468+,Author="Jens Axboe"
1469+,Title="Re: [patch] cpufreq: mark \url{cpufreq_tsc()} as
1470+\url{core_initcall_sync}"
1471+,month="November"
1472+,year="2006"
1473+,day=17
1474+,note="Available:
1475+\url{http://lkml.org/lkml/2006/11/17/56}
1476+[Viewed May 28, 2007]"
1477+,annotation="
1478+ SRCU's grace periods are too slow for Jens, even after a
1479+ factor-of-three speedup.
1480+ Sped-up version of SRCU at http://lkml.org/lkml/2006/11/17/359.
1481+"
1482+}
1483+
1484+@unpublished{OlegNesterov2006QRCU
1485+,Author="Oleg Nesterov"
1486+,Title="Re: [patch] cpufreq: mark {\tt cpufreq\_tsc()} as
1487+{\tt core\_initcall\_sync}"
1488+,month="November"
1489+,year="2006"
1490+,day=19
1491+,note="Available:
1492+\url{http://lkml.org/lkml/2006/11/19/69}
1493+[Viewed May 28, 2007]"
1494+,annotation="
1495+ First cut of QRCU. Expanded/corrected versions followed.
1496+ Used to be OlegNesterov2007QRCU, now time-corrected.
1497+"
1498+}
1499+
1500+@unpublished{OlegNesterov2006aQRCU
1501+,Author="Oleg Nesterov"
1502+,Title="Re: [RFC, PATCH 1/2] qrcu: {"quick"} srcu implementation"
1503+,month="November"
1504+,year="2006"
1505+,day=30
1506+,note="Available:
1507+\url{http://lkml.org/lkml/2006/11/29/330}
1508+[Viewed November 26, 2008]"
1509+,annotation="
1510+ Expanded/corrected version of QRCU.
1511+ Used to be OlegNesterov2007aQRCU, now time-corrected.
1512+"
1513+}
1514+
1515+@unpublished{EvgeniyPolyakov2006RCUslowdown
1516+,Author="Evgeniy Polyakov"
1517+,Title="Badness in postponing work"
1518+,month="December"
1519+,year="2006"
1520+,day=05
1521+,note="Available:
1522+\url{http://www.ioremap.net/node/41}
1523+[Viewed October 28, 2008]"
1524+,annotation="
1525+ Using RCU as a pure delay leads to a 2.5x slowdown in skbs in
1526+ the Linux kernel.
1527+"
1528+}
1529+
1530+@inproceedings{ChrisMatthews2006ClusteredObjectsRCU
1531+,author = {Matthews, Chris and Coady, Yvonne and Appavoo, Jonathan}
1532+,title = {Portability events: a programming model for scalable system infrastructures}
1533+,booktitle = {PLOS '06: Proceedings of the 3rd workshop on Programming languages and operating systems}
1534+,year = {2006}
1535+,isbn = {1-59593-577-0}
1536+,pages = {11}
1537+,location = {San Jose, California}
1538+,doi = {http://doi.acm.org/10.1145/1215995.1216006}
1539+,publisher = {ACM}
1540+,address = {New York, NY, USA}
1541+,annotation={
1542+ Uses K42's RCU-like functionality to manage clustered-object
1543+ lifetimes.
1544+}}
1545+
1546+@article{DilmaDaSilva2006K42
1547+,author = {Silva, Dilma Da and Krieger, Orran and Wisniewski, Robert W. and Waterland, Amos and Tam, David and Baumann, Andrew}
1548+,title = {K42: an infrastructure for operating system research}
1549+,journal = {SIGOPS Oper. Syst. Rev.}
1550+,volume = {40}
1551+,number = {2}
1552+,year = {2006}
1553+,issn = {0163-5980}
1554+,pages = {34--42}
1555+,doi = {http://doi.acm.org/10.1145/1131322.1131333}
1556+,publisher = {ACM}
1557+,address = {New York, NY, USA}
1558+,annotation={
1559+ Describes relationship of K42 generations to RCU.
1560+}}
1561+
1562+# CoreyMinyard2007list_splice_rcu
1563+@unpublished{CoreyMinyard2007list:splice:rcu
1564+,Author="Corey Minyard and Paul E. McKenney"
1565+,Title="{[PATCH]} add an {RCU} version of list splicing"
1566+,month="January"
1567+,year="2007"
1568+,day=3
1569+,note="Available:
1570+\url{http://lkml.org/lkml/2007/1/3/112}
1571+[Viewed May 28, 2007]"
1572+,annotation="
1573+ Patch for list_splice_rcu().
1574+"
1575+}
1576+
1577+@unpublished{PaulEMcKenney2007rcubarrier
1578+,Author="Paul E. McKenney"
1579+,Title="{RCU} and Unloadable Modules"
1580+,month="January"
1581+,day="14"
1582+,year="2007"
1583+,note="Available:
1584+\url{http://lwn.net/Articles/217484/}
1585+[Viewed November 22, 2007]"
1586+,annotation="
1587+ LWN article introducing the rcu_barrier() primitive.
1588+"
1589+}
1590+
1591+@unpublished{PeterZijlstra2007SyncBarrier
1592+,Author="Peter Zijlstra and Ingo Molnar"
1593+,Title="{[PATCH 3/7]} barrier: a scalable synchonisation barrier"
1594+,month="January"
1595+,year="2007"
1596+,day=28
1597+,note="Available:
1598+\url{http://lkml.org/lkml/2007/1/28/34}
1599+[Viewed March 27, 2008]"
1600+,annotation="
1601+ RCU-like implementation for frequent updaters and rare readers(!).
1602+ Subsumed into QRCU. Maybe...
1603+"
1604+}
1605+
1606+@unpublished{PaulEMcKenney2007BoostRCU
1607+,Author="Paul E. McKenney"
1608+,Title="Priority-Boosting {RCU} Read-Side Critical Sections"
1609+,month="February"
1610+,day="5"
1611+,year="2007"
1612+,note="Available:
1613+\url{http://lwn.net/Articles/220677/}
1614+Revised:
1615+\url{http://www.rdrop.com/users/paulmck/RCU/RCUbooststate.2007.04.16a.pdf}
1616+[Viewed September 7, 2007]"
1617+,annotation="
1618+ LWN article introducing RCU priority boosting.
1619+"
1620+}
1621+
1622+@unpublished{PaulMcKenney2007QRCUpatch
1623+,Author="Paul E. McKenney"
1624+,Title="{[PATCH]} {QRCU} with lockless fastpath"
1625+,month="February"
1626+,year="2007"
1627+,day=24
1628+,note="Available:
1629+\url{http://lkml.org/lkml/2007/2/25/18}
1630+[Viewed March 27, 2008]"
1631+,annotation="
1632+ Patch for QRCU supplying lock-free fast path.
1633+"
1634+}
1635+
1636+@article{JonathanAppavoo2007K42RCU
1637+,author = {Appavoo, Jonathan and Silva, Dilma Da and Krieger, Orran and Auslander, Marc and Ostrowski, Michal and Rosenburg, Bryan and Waterland, Amos and Wisniewski, Robert W. and Xenidis, Jimi and Stumm, Michael and Soares, Livio}
1638+,title = {Experience distributing objects in an SMMP OS}
1639+,journal = {ACM Trans. Comput. Syst.}
1640+,volume = {25}
1641+,number = {3}
1642+,year = {2007}
1643+,issn = {0734-2071}
1644+,pages = {6/1--6/52}
1645+,doi = {http://doi.acm.org/10.1145/1275517.1275518}
1646+,publisher = {ACM}
1647+,address = {New York, NY, USA}
1648+,annotation={
1649+ Role of RCU in K42.
1650+}}
1651+
1652+@conference{RobertOlsson2007Trash
1653+,Author="Robert Olsson and Stefan Nilsson"
1654+,Title="{TRASH}: A dynamic {LC}-trie and hash data structure"
1655+,booktitle="Workshop on High Performance Switching and Routing (HPSR'07)"
1656+,month="May"
1657+,year="2007"
6391658 ,note="Available:
640−\url{http://www.nada.kth.se/~snilsson/public/papers/trash/trash.pdf}
641−[Viewed February 24, 2007]"
1659+\url{http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4281239}
1660+[Viewed October 1, 2010]"
6421661 ,annotation="
6431662 RCU-protected dynamic trie-hash combination.
6441663 "
6451664 }
6461665
647−@unpublished{ThomasEHart2007a
648−,Author="Thomas E. Hart and Paul E. McKenney and Angela Demke Brown and Jonathan Walpole"
649−,Title="Performance of memory reclamation for lockless synchronization"
650−,journal="J. Parallel Distrib. Comput."
1666+@conference{PeterZijlstra2007ConcurrentPagecacheRCU
1667+,Author="Peter Zijlstra"
1668+,Title="Concurrent Pagecache"
1669+,Booktitle="Linux Symposium"
1670+,month="June"
1671+,year="2007"
1672+,address="Ottawa, Canada"
1673+,note="Available:
1674+\url{http://ols.108.redhat.com/2007/Reprints/zijlstra-Reprint.pdf}
1675+[Viewed April 14, 2008]"
1676+,annotation="
1677+ Page-cache modifications permitting RCU readers and concurrent
1678+ updates.
1679+"
1680+}
1681+
1682+@unpublished{PaulEMcKenney2007whatisRCU
1683+,Author="Paul E. McKenney"
1684+,Title="What is {RCU}?"
6511685 ,year="2007"
652−,note="To appear in J. Parallel Distrib. Comput.
653− \url{doi=10.1016/j.jpdc.2007.04.010}"
1686+,month="07"
1687+,note="Available:
1688+\url{http://www.rdrop.com/users/paulmck/RCU/whatisRCU.html}
1689+[Viewed July 6, 2007]"
6541690 ,annotation={
655− Compares QSBR (AKA "classic RCU"), HPBR, EBR, and lock-free
656− reference counting. Journal version of ThomasEHart2006a.
1691+ Describes RCU in Linux kernel.
6571692 }
6581693 }
6591694
6601695 @unpublished{PaulEMcKenney2007QRCUspin
6611696 ,Author="Paul E. McKenney"
662−,Title="Using Promela and Spin to verify parallel algorithms"
1697+,Title="Using {Promela} and {Spin} to verify parallel algorithms"
6631698 ,month="August"
6641699 ,day="1"
6651700 ,year="2007"
@@ -669,6 +1704,50 @@ Revised: · 3 unmodified lines
6691704 ,annotation="
6701705 LWN article describing Promela and spin, and also using Oleg
6711706 Nesterov's QRCU as an example (with Paul McKenney's fastpath).
1707+ Merged patch at: http://lkml.org/lkml/2007/2/25/18
1708+"
1709+}
1710+
1711+@unpublished{PaulEMcKenney2007WG21DDOatomics
1712+,Author="Paul E. McKenney and Hans-J. Boehm and Lawrence Crowl"
1713+,Title="C++ Data-Dependency Ordering: Atomics and Memory Model"
1714+,month="August"
1715+,day="3"
1716+,year="2007"
1717+,note="Preprint:
1718+\url{http://open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2664.htm}
1719+[Viewed December 7, 2009]"
1720+,annotation="
1721+ RCU for C++, parts 1 and 2.
1722+"
1723+}
1724+
1725+@unpublished{PaulEMcKenney2007WG21DDOannotation
1726+,Author="Paul E. McKenney and Lawrence Crowl"
1727+,Title="C++ Data-Dependency Ordering: Function Annotation"
1728+,month="September"
1729+,day="18"
1730+,year="2008"
1731+,note="Preprint:
1732+\url{http://open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2782.htm}
1733+[Viewed December 7, 2009]"
1734+,annotation="
1735+ RCU for C++, part 2, updated many times.
1736+"
1737+}
1738+
1739+@unpublished{PaulEMcKenney2007PreemptibleRCUPatch
1740+,Author="Paul E. McKenney"
1741+,Title="[PATCH RFC 0/9] {RCU}: Preemptible {RCU}"
1742+,month="September"
1743+,day="10"
1744+,year="2007"
1745+,note="Available:
1746+\url{http://lkml.org/lkml/2007/9/10/213}
1747+[Viewed October 25, 2007]"
1748+,annotation="
1749+ Final patch for preemptable RCU to -rt. (Later patches were
1750+ to mainline, eventually incorporated.)
6721751 "
6731752 }
6741753
@@ -686,10 +1765,46 @@ Revised: · 11 unmodified lines
6861765 "
6871766 }
6881767
1768+@article{ThomasEHart2007a
1769+,Author="Thomas E. Hart and Paul E. McKenney and Angela Demke Brown and Jonathan Walpole"
1770+,Title="Performance of memory reclamation for lockless synchronization"
1771+,journal="J. Parallel Distrib. Comput."
1772+,volume={67}
1773+,number="12"
1774+,year="2007"
1775+,issn="0743-7315"
1776+,pages="1270--1285"
1777+,doi="http://dx.doi.org/10.1016/j.jpdc.2007.04.010"
1778+,publisher="Academic Press, Inc."
1779+,address="Orlando, FL, USA"
1780+,annotation={
1781+ Compares QSBR, HPBR, EBR, and lock-free reference counting.
1782+ Journal version of ThomasEHart2006a.
1783+}
1784+}
1785+
1786+@unpublished{MathieuDesnoyers2007call:rcu:schedNeeded
1787+,Author="Mathieu Desnoyers"
1788+,Title="Re: [patch 1/2] {Linux} Kernel Markers - Support Multiple Probes"
1789+,month="December"
1790+,day="20"
1791+,year="2007"
1792+,note="Available:
1793+\url{http://lkml.org/lkml/2007/12/20/244}
1794+[Viewed March 27, 2008]"
1795+,annotation="
1796+ Request for call_rcu_sched() and rcu_barrier_sched().
1797+"
1798+}
1799+
1800+
6891801 ########################################################################
6901802 #
6911803 # "What is RCU?" LWN series.
6921804 #
1805+# http://lwn.net/Articles/262464/ (What is RCU, Fundamentally?)
1806+# http://lwn.net/Articles/263130/ (What is RCU's Usage?)
1807+# http://lwn.net/Articles/264090/ (What is RCU's API?)
6931808
6941809 @unpublished{PaulEMcKenney2007WhatIsRCUFundamentally
6951810 ,Author="Paul E. McKenney and Jonathan Walpole"
@@ -723,7 +1838,7 @@ Revised: · 27 unmodified lines
7231838 3. RCU is a Bulk Reference-Counting Mechanism
7241839 4. RCU is a Poor Man's Garbage Collector
7251840 5. RCU is a Way of Providing Existence Guarantees
726− 6. RCU is a Way of Waiting for Things to Finish
1841+ 6. RCU is a Way of Waiting for Things to Finish
7271842 "
7281843 }
7291844
@@ -747,20 +1862,96 @@ Revised: · 17 unmodified lines
7471862 #
7481863 ########################################################################
7491864
1865+
1866+@unpublished{SteveRostedt2008dyntickRCUpatch
1867+,Author="Steven Rostedt and Paul E. McKenney"
1868+,Title="{[PATCH]} add support for dynamic ticks and preempt rcu"
1869+,month="January"
1870+,day="29"
1871+,year="2008"
1872+,note="Available:
1873+\url{http://lkml.org/lkml/2008/1/29/208}
1874+[Viewed March 27, 2008]"
1875+,annotation="
1876+ Patch that prevents preemptible RCU from unnecessarily waking
1877+ up dynticks-idle CPUs.
1878+"
1879+}
1880+
1881+@unpublished{PaulEMcKenney2008LKMLDependencyOrdering
1882+,Author="Paul E. McKenney"
1883+,Title="Re: [PATCH 02/22 -v7] Add basic support for gcc profiler instrumentation"
1884+,month="February"
1885+,day="1"
1886+,year="2008"
1887+,note="Available:
1888+\url{http://lkml.org/lkml/2008/2/2/255}
1889+[Viewed October 18, 2008]"
1890+,annotation="
1891+ Explanation of compilers violating dependency ordering.
1892+"
1893+}
1894+
1895+@Conference{PaulEMcKenney2008Beijing
1896+,Author="Paul E. McKenney"
1897+,Title="Introducing Technology Into {Linux} Or:
1898+Introducing your technology Into {Linux} will require introducing a
1899+lot of {Linux} into your technology!!!"
1900+,Booktitle="2008 Linux Developer Symposium - China"
1901+,Publisher="OSS China"
1902+,Month="February"
1903+,Year="2008"
1904+,Address="Beijing, China"
1905+,note="Available:
1906+\url{http://www.rdrop.com/users/paulmck/RCU/TechIntroLinux.2008.02.19a.pdf}
1907+[Viewed August 12, 2008]"
1908+}
1909+
1910+@unpublished{PaulEMcKenney2008dynticksRCU
1911+,Author="Paul E. McKenney and Steven Rostedt"
1912+,Title="Integrating and Validating dynticks and Preemptable RCU"
1913+,month="April"
1914+,day="24"
1915+,year="2008"
1916+,note="Available:
1917+\url{http://lwn.net/Articles/279077/}
1918+[Viewed April 24, 2008]"
1919+,annotation="
1920+ Describes use of Promela and Spin to validate (and fix!) the
1921+ dynticks/RCU interface.
1922+"
1923+}
1924+
7501925 @article{DinakarGuniguntala2008IBMSysJ
7511926 ,author="D. Guniguntala and P. E. McKenney and J. Triplett and J. Walpole"
7521927 ,title="The read-copy-update mechanism for supporting real-time applications on shared-memory multiprocessor systems with {Linux}"
7531928 ,Year="2008"
754−,Month="April"
1929+,Month="April-June"
7551930 ,journal="IBM Systems Journal"
7561931 ,volume="47"
7571932 ,number="2"
758−,pages="@@-@@"
1933+,pages="221-236"
7591934 ,annotation="
7601935 RCU, realtime RCU, sleepable RCU, performance.
7611936 "
7621937 }
7631938
1939+@unpublished{LaiJiangshan2008NewClassicAlgorithm
1940+,Author="Lai Jiangshan"
1941+,Title="[{RFC}][{PATCH}] rcu classic: new algorithm for callbacks-processing"
1942+,month="June"
1943+,day="3"
1944+,year="2008"
1945+,note="Available:
1946+\url{http://lkml.org/lkml/2008/6/2/539}
1947+[Viewed December 10, 2008]"
1948+,annotation="
1949+ Updated RCU classic algorithm. Introduced multi-tailed list
1950+ for RCU callbacks and also pulling common code into
1951+ __call_rcu().
1952+"
1953+}
1954+
7641955 @article{PaulEMcKenney2008RCUOSR
7651956 ,author="Paul E. McKenney and Jonathan Walpole"
7661957 ,title="Introducing technology into the {Linux} kernel: a case study"
@@ -778,6 +1969,52 @@ Revised: · 11 unmodified lines
7781969 }
7791970 }
7801971
1972+@unpublished{ManfredSpraul2008StateMachineRCU
1973+,Author="Manfred Spraul"
1974+,Title="[{RFC}, {PATCH}] state machine based rcu"
1975+,month="August"
1976+,day="21"
1977+,year="2008"
1978+,note="Available:
1979+\url{http://lkml.org/lkml/2008/8/21/336}
1980+[Viewed December 8, 2008]"
1981+,annotation="
1982+ State-based RCU. One key thing that this patch does is to
1983+ separate the dynticks handling of NMIs and IRQs.
1984+"
1985+}
1986+
1987+@unpublished{ManfredSpraul2008dyntickIRQNMI
1988+,Author="Manfred Spraul"
1989+,Title="Re: [{RFC}, {PATCH}] v4 scalable classic {RCU} implementation"
1990+,month="September"
1991+,day="6"
1992+,year="2008"
1993+,note="Available:
1994+\url{http://lkml.org/lkml/2008/9/6/86}
1995+[Viewed December 8, 2008]"
1996+,annotation="
1997+ Manfred notes a fix required to my attempt to separate irq
1998+ and NMI processing for hierarchical RCU's dynticks interface.
1999+"
2000+}
2001+
2002+@techreport{PaulEMcKenney2008cyclicRCU
2003+,author="Paul E. McKenney"
2004+,title="Efficient Support of Consistent Cyclic Search With Read-Copy Update"
2005+,institution="US Patent and Trademark Office"
2006+,address="Washington, DC"
2007+,year="2008"
2008+,number="US Patent 7,426,511"
2009+,month="September"
2010+,pages="23"
2011+,annotation="
2012+ Maintains an additional level of indirection to allow
2013+ readers to confine themselves to the desired snapshot of the
2014+ data structure. Only permits one update at a time.
2015+"
2016+}
2017+
7812018 @unpublished{PaulEMcKenney2008HierarchicalRCU
7822019 ,Author="Paul E. McKenney"
7832020 ,Title="Hierarchical {RCU}"
@@ -793,6 +2030,21 @@ Revised: · 9 unmodified lines
7932030 "
7942031 }
7952032
2033+@unpublished{PaulEMcKenney2009BloatwatchRCU
2034+,Author="Paul E. McKenney"
2035+,Title="Re: [PATCH fyi] RCU: the bloatwatch edition"
2036+,month="January"
2037+,day="14"
2038+,year="2009"
2039+,note="Available:
2040+\url{http://lkml.org/lkml/2009/1/14/449}
2041+[Viewed January 15, 2009]"
2042+,annotation="
2043+ Small-footprint implementation of RCU for uniprocessor
2044+ embedded applications -- and also for exposition purposes.
2045+"
2046+}
2047+
7962048 @conference{PaulEMcKenney2009MaliciousURCU
7972049 ,Author="Paul E. McKenney"
7982050 ,Title="Using a Malicious User-Level {RCU} to Torture {RCU}-Based Algorithms"
@@ -816,15 +2068,17 @@ Revised: · 17 unmodified lines
8162068 ,year="2009"
8172069 ,note="Available:
8182070 \url{http://lkml.org/lkml/2009/2/5/572}
819−\url{git://lttng.org/userspace-rcu.git}
2071+\url{http://lttng.org/urcu}
8202072 [Viewed February 20, 2009]"
8212073 ,annotation="
8222074 Mathieu Desnoyers's user-space RCU implementation.
8232075 git://lttng.org/userspace-rcu.git
2076+ http://lttng.org/cgi-bin/gitweb.cgi?p=userspace-rcu.git
2077+ http://lttng.org/urcu
8242078 "
8252079 }
8262080
827−@unpublished{PaulEMcKenney2009BloatWatchRCU
2081+@unpublished{PaulEMcKenney2009LWNBloatWatchRCU
8282082 ,Author="Paul E. McKenney"
8292083 ,Title="{RCU}: The {Bloatwatch} Edition"
8302084 ,month="March"
@@ -852,14 +2106,29 @@ Revised: · 21 unmodified lines
8522106 "
8532107 }
8542108
855−@unpublished{JoshTriplett2009RPHash
2109+@unpublished{PaulEMcKenney2009fastRTRCU
2110+,Author="Paul E. McKenney"
2111+,Title="[{PATCH} {RFC} -tip 0/4] {RCU} cleanups and simplified preemptable {RCU}"
2112+,month="July"
2113+,day="23"
2114+,year="2009"
2115+,note="Available:
2116+\url{http://lkml.org/lkml/2009/7/23/294}
2117+[Viewed August 15, 2009]"
2118+,annotation="
2119+ First posting of simple and fast preemptable RCU.
2120+"
2121+}
2122+
2123+@InProceedings{JoshTriplett2009RPHash
8562124 ,Author="Josh Triplett"
8572125 ,Title="Scalable concurrent hash tables via relativistic programming"
8582126 ,month="September"
8592127 ,year="2009"
860−,note="Linux Plumbers Conference presentation"
2128+,booktitle="Linux Plumbers Conference 2009"
8612129 ,annotation="
8622130 RP fun with hash tables.
2131+ See also JoshTriplett2010RPHash
8632132 "
8642133 }
8652134
@@ -872,4 +2141,323 @@ Revised: · 6 unmodified lines
8722141 ,note="Available:
8732142 \url{http://www.lttng.org/pub/thesis/desnoyers-dissertation-2009-12.pdf}
8742143 [Viewed December 9, 2009]"
2144+,annotation={
2145+ Chapter 6 (page 97) covers user-level RCU.
2146+}
2147+}
2148+
2149+@unpublished{RelativisticProgrammingWiki
2150+,Author="Josh Triplett and Paul E. McKenney and Jonathan Walpole"
2151+,Title="Relativistic Programming"
2152+,month="September"
2153+,year="2009"
2154+,note="Available:
2155+\url{http://wiki.cs.pdx.edu/rp/}
2156+[Viewed December 9, 2009]"
2157+,annotation="
2158+ Main Relativistic Programming Wiki.
2159+"
2160+}
2161+
2162+@conference{PaulEMcKenney2009DeterministicRCU
2163+,Author="Paul E. McKenney"
2164+,Title="Deterministic Synchronization in Multicore Systems: the Role of {RCU}"
2165+,Booktitle="Eleventh Real Time Linux Workshop"
2166+,month="September"
2167+,year="2009"
2168+,address="Dresden, Germany"
2169+,note="Available:
2170+\url{http://www.rdrop.com/users/paulmck/realtime/paper/DetSyncRCU.2009.08.18a.pdf}
2171+[Viewed January 14, 2009]"
2172+}
2173+
2174+@unpublished{PaulEMcKenney2009HuntingHeisenbugs
2175+,Author="Paul E. McKenney"
2176+,Title="Hunting Heisenbugs"
2177+,month="November"
2178+,year="2009"
2179+,day="1"
2180+,note="Available:
2181+\url{http://paulmck.livejournal.com/14639.html}
2182+[Viewed June 4, 2010]"
2183+,annotation="
2184+ Day-one bug in Tree RCU that took forever to track down.
2185+"
2186+}
2187+
2188+@unpublished{MathieuDesnoyers2009defer:rcu
2189+,Author="Mathieu Desnoyers"
2190+,Title="Kernel RCU: shrink the size of the struct rcu\_head"
2191+,month="December"
2192+,year="2009"
2193+,note="Available:
2194+\url{http://lkml.org/lkml/2009/10/18/129}
2195+[Viewed December 29, 2009]"
2196+,annotation="
2197+ Mathieu proposed defer_rcu() with fixed-size per-thread pool
2198+ of RCU callbacks.
2199+"
2200+}
2201+
2202+@unpublished{MathieuDesnoyers2009VerifPrePub
2203+,Author="Mathieu Desnoyers and Paul E. McKenney and Michel R. Dagenais"
2204+,Title="Multi-Core Systems Modeling for Formal Verification of Parallel Algorithms"
2205+,month="December"
2206+,year="2009"
2207+,note="Submitted to IEEE TPDS"
2208+,annotation="
2209+ OOMem model for Mathieu's user-level RCU mechanical proof of
2210+ correctness.
2211+"
2212+}
2213+
2214+@unpublished{MathieuDesnoyers2009URCUPrePub
2215+,Author="Mathieu Desnoyers and Paul E. McKenney and Alan Stern and Michel R. Dagenais and Jonathan Walpole"
2216+,Title="User-Level Implementations of Read-Copy Update"
2217+,month="December"
2218+,year="2010"
2219+,url=\url{http://www.computer.org/csdl/trans/td/2012/02/ttd2012020375-abs.html}
2220+,annotation="
2221+ RCU overview, desiderata, semi-formal semantics, user-level RCU
2222+ usage scenarios, three classes of RCU implementation, wait-free
2223+ RCU updates, RCU grace-period batching, update overhead,
2224+ http://www.rdrop.com/users/paulmck/RCU/urcu-main-accepted.2011.08.30a.pdf
2225+ http://www.rdrop.com/users/paulmck/RCU/urcu-supp-accepted.2011.08.30a.pdf
2226+ Superseded by MathieuDesnoyers2012URCU.
2227+"
2228+}
2229+
2230+@inproceedings{HariKannan2009DynamicAnalysisRCU
2231+,author = {Kannan, Hari}
2232+,title = {Ordering decoupled metadata accesses in multiprocessors}
2233+,booktitle = {MICRO 42: Proceedings of the 42nd Annual IEEE/ACM International Symposium on Microarchitecture}
2234+,year = {2009}
2235+,isbn = {978-1-60558-798-1}
2236+,pages = {381--390}
2237+,location = {New York, New York}
2238+,doi = {http://doi.acm.org/10.1145/1669112.1669161}
2239+,publisher = {ACM}
2240+,address = {New York, NY, USA}
2241+,annotation={
2242+ Uses RCU to protect metadata used in dynamic analysis.
2243+}}
2244+
2245+@conference{PaulEMcKenney2010SimpleOptRCU
2246+,Author="Paul E. McKenney"
2247+,Title="Simplicity Through Optimization"
2248+,Booktitle="linux.conf.au 2010"
2249+,month="January"
2250+,year="2010"
2251+,address="Wellington, New Zealand"
2252+,note="Available:
2253+\url{http://www.rdrop.com/users/paulmck/RCU/SimplicityThruOptimization.2010.01.21f.pdf}
2254+[Viewed October 10, 2010]"
2255+,annotation="
2256+ TREE_PREEMPT_RCU optimizations greatly simplified the old
2257+ PREEMPT_RCU implementation.
2258+"
2259+}
2260+
2261+@unpublished{PaulEMcKenney2010LockdepRCU
2262+,Author="Paul E. McKenney"
2263+,Title="Lockdep-{RCU}"
2264+,month="February"
2265+,year="2010"
2266+,day="1"
2267+,note="Available:
2268+\url{https://lwn.net/Articles/371986/}
2269+[Viewed June 4, 2010]"
2270+,annotation="
2271+ CONFIG_PROVE_RCU, or at least an early version.
2272+"
2273+}
2274+
2275+@unpublished{AviKivity2010KVM2RCU
2276+,Author="Avi Kivity"
2277+,Title="[{PATCH} 37/40] {KVM}: Bump maximum vcpu count to 64"
2278+,month="February"
2279+,year="2010"
2280+,note="Available:
2281+\url{http://www.mail-archive.com/kvm@vger.kernel.org/msg28640.html}
2282+[Viewed March 20, 2010]"
2283+,annotation="
2284+ Use of RCU permits KVM to increase the size of guest OSes from
2285+ 16 CPUs to 64 CPUs.
2286+"
2287+}
2288+
2289+@unpublished{HerbertXu2010RCUResizeHash
2290+,Author="Herbert Xu"
2291+,Title="bridge: Add core IGMP snooping support"
2292+,month="February"
2293+,year="2010"
2294+,note="Available:
2295+\url{http://kerneltrap.com/mailarchive/linux-netdev/2010/2/26/6270589}
2296+[Viewed March 20, 2011]"
2297+,annotation={
2298+ Use a pair of list_head structures to support RCU-protected
2299+ resizable hash tables.
2300+}}
2301+
2302+@article{JoshTriplett2010RPHash
2303+,author="Josh Triplett and Paul E. McKenney and Jonathan Walpole"
2304+,title="Scalable Concurrent Hash Tables via Relativistic Programming"
2305+,journal="ACM Operating Systems Review"
2306+,year=2010
2307+,volume=44
2308+,number=3
2309+,month="July"
2310+,annotation={
2311+ RP fun with hash tables.
2312+ http://portal.acm.org/citation.cfm?id=1842733.1842750
2313+}}
2314+
2315+@unpublished{PaulEMcKenney2010RCUAPI
2316+,Author="Paul E. McKenney"
2317+,Title="The {RCU} {API}, 2010 Edition"
2318+,month="December"
2319+,day="8"
2320+,year="2010"
2321+,note="Available:
2322+\url{http://lwn.net/Articles/418853/}
2323+[Viewed December 8, 2010]"
2324+,annotation="
2325+ Includes updated software-engineering features.
2326+"
2327+}
2328+
2329+@mastersthesis{AndrejPodzimek2010masters
2330+,author="Andrej Podzimek"
2331+,title="Read-Copy-Update for OpenSolaris"
2332+,school="Charles University in Prague"
2333+,year="2010"
2334+,note="Available:
2335+\url{https://andrej.podzimek.org/thesis.pdf}
2336+[Viewed January 31, 2011]"
2337+,annotation={
2338+ Reviews RCU implementations and creates a few for OpenSolaris.
2339+ Drives quiescent-state detection from RCU read-side primitives,
2340+ in a manner roughly similar to that of Jim Houston.
2341+}}
2342+
2343+@unpublished{LinusTorvalds2011Linux2:6:38:rc1:NPigginVFS
2344+,Author="Linus Torvalds"
2345+,Title="Linux 2.6.38-rc1"
2346+,month="January"
2347+,year="2011"
2348+,note="Available:
2349+\url{https://lkml.org/lkml/2011/1/18/322}
2350+[Viewed March 4, 2011]"
2351+,annotation={
2352+ "The RCU-based name lookup is at the other end of the spectrum - the
2353+ absolute anti-gimmick. It's some seriously good stuff, and gets rid of
2354+ the last main global lock that really tends to hurt some kernel loads.
2355+ The dentry lock is no longer a big serializing issue. What's really
2356+ nice about it is that it actually improves performance a lot even for
2357+ single-threaded loads (on an SMP kernel), because it gets rid of some
2358+ of the most expensive parts of path component lookup, which was the
2359+ d_lock on every component lookup. So I'm seeing improvements of 30-50%
2360+ on some seriously pathname-lookup intensive loads."
2361+}}
2362+
2363+@techreport{JoshTriplett2011RPScalableCorrectOrdering
2364+,author = {Josh Triplett and Philip W. Howard and Paul E. McKenney and Jonathan Walpole}
2365+,title = {Scalable Correct Memory Ordering via Relativistic Programming}
2366+,year = {2011}
2367+,number = {11-03}
2368+,institution = {Portland State University}
2369+,note = {\url{http://www.cs.pdx.edu/pdfs/tr1103.pdf}}
2370+}
2371+
2372+@inproceedings{PhilHoward2011RCUTMRBTree
2373+,author = {Philip W. Howard and Jonathan Walpole}
2374+,title = {A Relativistic Enhancement to Software Transactional Memory}
2375+,booktitle = {Proceedings of the 3rd USENIX conference on Hot topics in parallelism}
2376+,series = {HotPar'11}
2377+,year = {2011}
2378+,location = {Berkeley, CA}
2379+,pages = {1--6}
2380+,numpages = {6}
2381+,url = {http://www.usenix.org/event/hotpar11/tech/final_files/Howard.pdf}
2382+,publisher = {USENIX Association}
2383+,address = {Berkeley, CA, USA}
2384+}
2385+
2386+@techreport{PaulEMcKenney2011cyclicparallelRCU
2387+,author="Paul E. McKenney and Jonathan Walpole"
2388+,title="Efficient Support of Consistent Cyclic Search With Read-Copy Update and Parallel Updates"
2389+,institution="US Patent and Trademark Office"
2390+,address="Washington, DC"
2391+,year="2011"
2392+,number="US Patent 7,953,778"
2393+,month="May"
2394+,pages="34"
2395+,annotation="
2396+ Maintains an array of generation numbers to track in-flight
2397+ updates and keeps an additional level of indirection to allow
2398+ readers to confine themselves to the desired snapshot of the
2399+ data structure.
2400+"
2401+}
2402+
2403+@inproceedings{Triplett:2011:RPHash
2404+,author = {Triplett, Josh and McKenney, Paul E. and Walpole, Jonathan}
2405+,title = {Resizable, Scalable, Concurrent Hash Tables via Relativistic Programming}
2406+,booktitle = {Proceedings of the 2011 USENIX Annual Technical Conference}
2407+,month = {June}
2408+,year = {2011}
2409+,pages = {145--158}
2410+,numpages = {14}
2411+,url={http://www.usenix.org/event/atc11/tech/final_files/atc11_proceedings.pdf}
2412+,publisher = {The USENIX Association}
2413+,address = {Portland, OR USA}
2414+}
2415+
2416+@unpublished{PaulEMcKenney2011RCU3.0trainwreck
2417+,Author="Paul E. McKenney"
2418+,Title="3.0 and {RCU:} what went wrong"
2419+,month="July"
2420+,day="27"
2421+,year="2011"
2422+,note="Available:
2423+\url{http://lwn.net/Articles/453002/}
2424+[Viewed July 27, 2011]"
2425+,annotation="
2426+ Analysis of the RCU trainwreck in Linux kernel 3.0.
2427+"
2428+}
2429+
2430+@unpublished{NeilBrown2011MeetTheLockers
2431+,Author="Neil Brown"
2432+,Title="Meet the Lockers"
2433+,month="August"
2434+,day="3"
2435+,year="2011"
2436+,note="Available:
2437+\url{http://lwn.net/Articles/453685/}
2438+[Viewed September 2, 2011]"
2439+,annotation="
2440+ The Locker family as an analogy for locking, reference counting,
2441+ RCU, and seqlock.
2442+"
2443+}
2444+
2445+@article{MathieuDesnoyers2012URCU
2446+,Author="Mathieu Desnoyers and Paul E. McKenney and Alan Stern and Michel R. Dagenais and Jonathan Walpole"
2447+,Title="User-Level Implementations of Read-Copy Update"
2448+,journal="IEEE Transactions on Parallel and Distributed Systems"
2449+,volume={23}
2450+,year="2012"
2451+,issn="1045-9219"
2452+,pages="375-382"
2453+,doi="http://doi.ieeecomputersociety.org/10.1109/TPDS.2011.159"
2454+,publisher="IEEE Computer Society"
2455+,address="Los Alamitos, CA, USA"
2456+,annotation={
2457+ RCU overview, desiderata, semi-formal semantics, user-level RCU
2458+ usage scenarios, three classes of RCU implementation, wait-free
2459+ RCU updates, RCU grace-period batching, update overhead,
2460+ http://www.rdrop.com/users/paulmck/RCU/urcu-main-accepted.2011.08.30a.pdf
2461+ http://www.rdrop.com/users/paulmck/RCU/urcu-supp-accepted.2011.08.30a.pdf
2462+}
8752463 }
Documentation/RCU/checklist.txt+14−0
@@ -180,6 +180,20 @@ over a rather long period of time, but improvements are always welcome! · 179 unmodified lines
180180 operations that would not normally be undertaken while a real-time
181181 workload is running.
182182
183+ In particular, if you find yourself invoking one of the expedited
184+ primitives repeatedly in a loop, please do everyone a favor:
185+ Restructure your code so that it batches the updates, allowing
186+ a single non-expedited primitive to cover the entire batch.
187+ This will very likely be faster than the loop containing the
188+ expedited primitive, and will be much much easier on the rest
189+ of the system, especially to real-time workloads running on
190+ the rest of the system.
191+
192+ In addition, it is illegal to call the expedited forms from
193+ a CPU-hotplug notifier, or while holding a lock that is acquired
194+ by a CPU-hotplug notifier. Failing to observe this restriction
195+ will result in deadlock.
196+
183197 7. If the updater uses call_rcu() or synchronize_rcu(), then the
184198 corresponding readers must use rcu_read_lock() and
185199 rcu_read_unlock(). If the updater uses call_rcu_bh() or
Documentation/RCU/stallwarn.txt+80−7
@@ -12,14 +12,38 @@ CONFIG_RCU_CPU_STALL_TIMEOUT · 11 unmodified lines
1212 This kernel configuration parameter defines the period of time
1313 that RCU will wait from the beginning of a grace period until it
1414 issues an RCU CPU stall warning. This time period is normally
15− ten seconds.
15+ sixty seconds.
1616
17−RCU_SECONDS_TILL_STALL_RECHECK
17+ This configuration parameter may be changed at runtime via the
18+ /sys/module/rcutree/parameters/rcu_cpu_stall_timeout, however
19+ this parameter is checked only at the beginning of a cycle.
20+ So if you are 30 seconds into a 70-second stall, setting this
21+ sysfs parameter to (say) five will shorten the timeout for the
22+ -next- stall, or the following warning for the current stall
23+ (assuming the stall lasts long enough). It will not affect the
24+ timing of the next warning for the current stall.
1825
19− This macro defines the period of time that RCU will wait after
20− issuing a stall warning until it issues another stall warning
21− for the same stall. This time period is normally set to three
22− times the check interval plus thirty seconds.
26+ Stall-warning messages may be enabled and disabled completely via
27+ /sys/module/rcutree/parameters/rcu_cpu_stall_suppress.
28+
29+CONFIG_RCU_CPU_STALL_VERBOSE
30+
31+ This kernel configuration parameter causes the stall warning to
32+ also dump the stacks of any tasks that are blocking the current
33+ RCU-preempt grace period.
34+
35+RCU_CPU_STALL_INFO
36+
37+ This kernel configuration parameter causes the stall warning to
38+ print out additional per-CPU diagnostic information, including
39+ information on scheduling-clock ticks and RCU's idle-CPU tracking.
40+
41+RCU_STALL_DELAY_DELTA
42+
43+ Although the lockdep facility is extremely useful, it does add
44+ some overhead. Therefore, under CONFIG_PROVE_RCU, the
45+ RCU_STALL_DELAY_DELTA macro allows five extra seconds before
46+ giving an RCU CPU stall warning message.
2347
2448 RCU_STALL_RAT_DELAY
2549
@@ -64,6 +88,54 @@ INFO: rcu_bh_state detected stalls on CPUs/tasks: { } (detected by 4, 2502 jiffi · 38 unmodified lines
6488
6589 This is rare, but does happen from time to time in real life.
6690
91+If the CONFIG_RCU_CPU_STALL_INFO kernel configuration parameter is set,
92+more information is printed with the stall-warning message, for example:
93+
94+ INFO: rcu_preempt detected stall on CPU
95+ 0: (63959 ticks this GP) idle=241/3fffffffffffffff/0
96+ (t=65000 jiffies)
97+
98+In kernels with CONFIG_RCU_FAST_NO_HZ, even more information is
99+printed:
100+
101+ INFO: rcu_preempt detected stall on CPU
102+ 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 drain=0 . timer=-1
103+ (t=65000 jiffies)
104+
105+The "(64628 ticks this GP)" indicates that this CPU has taken more
106+than 64,000 scheduling-clock interrupts during the current stalled
107+grace period. If the CPU was not yet aware of the current grace
108+period (for example, if it was offline), then this part of the message
109+indicates how many grace periods behind the CPU is.
110+
111+The "idle=" portion of the message prints the dyntick-idle state.
112+The hex number before the first "/" is the low-order 12 bits of the
113+dynticks counter, which will have an even-numbered value if the CPU is
114+in dyntick-idle mode and an odd-numbered value otherwise. The hex
115+number between the two "/"s is the value of the nesting, which will
116+be a small positive number if in the idle loop and a very large positive
117+number (as shown above) otherwise.
118+
119+For CONFIG_RCU_FAST_NO_HZ kernels, the "drain=0" indicates that the
120+CPU is not in the process of trying to force itself into dyntick-idle
121+state, the "." indicates that the CPU has not given up forcing RCU
122+into dyntick-idle mode (it would be "H" otherwise), and the "timer=-1"
123+indicates that the CPU has not recented forced RCU into dyntick-idle
124+mode (it would otherwise indicate the number of microseconds remaining
125+in this forced state).
126+
127+
128+Multiple Warnings From One Stall
129+
130+If a stall lasts long enough, multiple stall-warning messages will be
131+printed for it. The second and subsequent messages are printed at
132+longer intervals, so that the time between (say) the first and second
133+message will be about three times the interval between the beginning
134+of the stall and the first message.
135+
136+
137+What Causes RCU CPU Stall Warnings?
138+
67139 So your kernel printed an RCU CPU stall warning. The next question is
68140 "What caused it?" The following problems can result in RCU CPU stall
69141 warnings:
@@ -128,4 +200,5 @@ is occurring, which will usually be in the function nearest the top of · 58 unmodified lines
128200 that portion of the stack which remains the same from trace to trace.
129201 If you can reliably trigger the stall, ftrace can be quite helpful.
130202
131−RCU bugs can often be debugged with the help of CONFIG_RCU_TRACE.
203+RCU bugs can often be debugged with the help of CONFIG_RCU_TRACE
204+and with RCU's event tracing.
Documentation/RCU/torture.txt+30−3
@@ -69,6 +69,13 @@ onoff_interval · 68 unmodified lines
6969 CPU-hotplug operations regardless of what value is
7070 specified for onoff_interval.
7171
72+onoff_holdoff The number of seconds to wait until starting CPU-hotplug
73+ operations. This would normally only be used when
74+ rcutorture was built into the kernel and started
75+ automatically at boot time, in which case it is useful
76+ in order to avoid confusing boot-time code with CPUs
77+ coming and going.
78+
7279 shuffle_interval
7380 The number of seconds to keep the test threads affinitied
7481 to a particular subset of the CPUs, defaults to 3 seconds.
@@ -79,6 +86,24 @@ shutdown_secs The number of seconds to run the test before terminating · 4 unmodified lines
7986 zero, which disables test termination and system shutdown.
8087 This capability is useful for automated testing.
8188
89+stall_cpu The number of seconds that a CPU should be stalled while
90+ within both an rcu_read_lock() and a preempt_disable().
91+ This stall happens only once per rcutorture run.
92+ If you need multiple stalls, use modprobe and rmmod to
93+ repeatedly run rcutorture. The default for stall_cpu
94+ is zero, which prevents rcutorture from stalling a CPU.
95+
96+ Note that attempts to rmmod rcutorture while the stall
97+ is ongoing will hang, so be careful what value you
98+ choose for this module parameter! In addition, too-large
99+ values for stall_cpu might well induce failures and
100+ warnings in other parts of the kernel. You have been
101+ warned!
102+
103+stall_cpu_holdoff
104+ The number of seconds to wait after rcutorture starts
105+ before stalling a CPU. Defaults to 10 seconds.
106+
82107 stat_interval The number of seconds between output of torture
83108 statistics (via printk()). Regardless of the interval,
84109 statistics are printed when the module is unloaded.
@@ -271,11 +296,13 @@ The following script may be used to torture RCU: · 186 unmodified lines
271296 #!/bin/sh
272297
273298 modprobe rcutorture
274− sleep 100
299+ sleep 3600
275300 rmmod rcutorture
276301 dmesg | grep torture:
277302
278303 The output can be manually inspected for the error flag of "!!!".
279304 One could of course create a more elaborate script that automatically
280−checked for such errors. The "rmmod" command forces a "SUCCESS" or
281−"FAILURE" indication to be printk()ed.
305+checked for such errors. The "rmmod" command forces a "SUCCESS",
306+"FAILURE", or "RCU_HOTPLUG" indication to be printk()ed. The first
307+two are self-explanatory, while the last indicates that while there
308+were no RCU failures, CPU-hotplug problems were detected.
Documentation/RCU/trace.txt+16−20
@@ -33,23 +33,23 @@ rcu/rcuboost: · 32 unmodified lines
3333 The output of "cat rcu/rcudata" looks as follows:
3434
3535 rcu_sched:
36− 0 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=545/1/0 df=50 of=0 ri=0 ql=163 qs=NRW. kt=0/W/0 ktl=ebc3 b=10 ci=153737 co=0 ca=0
37− 1 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=967/1/0 df=58 of=0 ri=0 ql=634 qs=NRW. kt=0/W/1 ktl=58c b=10 ci=191037 co=0 ca=0
38− 2 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=1081/1/0 df=175 of=0 ri=0 ql=74 qs=N.W. kt=0/W/2 ktl=da94 b=10 ci=75991 co=0 ca=0
39− 3 c=20942 g=20943 pq=1 pgp=20942 qp=1 dt=1846/0/0 df=404 of=0 ri=0 ql=0 qs=.... kt=0/W/3 ktl=d1cd b=10 ci=72261 co=0 ca=0
40− 4 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=369/1/0 df=83 of=0 ri=0 ql=48 qs=N.W. kt=0/W/4 ktl=e0e7 b=10 ci=128365 co=0 ca=0
41− 5 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=381/1/0 df=64 of=0 ri=0 ql=169 qs=NRW. kt=0/W/5 ktl=fb2f b=10 ci=164360 co=0 ca=0
42− 6 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=1037/1/0 df=183 of=0 ri=0 ql=62 qs=N.W. kt=0/W/6 ktl=d2ad b=10 ci=65663 co=0 ca=0
43− 7 c=20897 g=20897 pq=1 pgp=20896 qp=0 dt=1572/0/0 df=382 of=0 ri=0 ql=0 qs=.... kt=0/W/7 ktl=cf15 b=10 ci=75006 co=0 ca=0
36+ 0 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=545/1/0 df=50 of=0 ql=163 qs=NRW. kt=0/W/0 ktl=ebc3 b=10 ci=153737 co=0 ca=0
37+ 1 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=967/1/0 df=58 of=0 ql=634 qs=NRW. kt=0/W/1 ktl=58c b=10 ci=191037 co=0 ca=0
38+ 2 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=1081/1/0 df=175 of=0 ql=74 qs=N.W. kt=0/W/2 ktl=da94 b=10 ci=75991 co=0 ca=0
39+ 3 c=20942 g=20943 pq=1 pgp=20942 qp=1 dt=1846/0/0 df=404 of=0 ql=0 qs=.... kt=0/W/3 ktl=d1cd b=10 ci=72261 co=0 ca=0
40+ 4 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=369/1/0 df=83 of=0 ql=48 qs=N.W. kt=0/W/4 ktl=e0e7 b=10 ci=128365 co=0 ca=0
41+ 5 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=381/1/0 df=64 of=0 ql=169 qs=NRW. kt=0/W/5 ktl=fb2f b=10 ci=164360 co=0 ca=0
42+ 6 c=20972 g=20973 pq=1 pgp=20973 qp=0 dt=1037/1/0 df=183 of=0 ql=62 qs=N.W. kt=0/W/6 ktl=d2ad b=10 ci=65663 co=0 ca=0
43+ 7 c=20897 g=20897 pq=1 pgp=20896 qp=0 dt=1572/0/0 df=382 of=0 ql=0 qs=.... kt=0/W/7 ktl=cf15 b=10 ci=75006 co=0 ca=0
4444 rcu_bh:
45− 0 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=545/1/0 df=6 of=0 ri=1 ql=0 qs=.... kt=0/W/0 ktl=ebc3 b=10 ci=0 co=0 ca=0
46− 1 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=967/1/0 df=3 of=0 ri=1 ql=0 qs=.... kt=0/W/1 ktl=58c b=10 ci=151 co=0 ca=0
47− 2 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1081/1/0 df=6 of=0 ri=1 ql=0 qs=.... kt=0/W/2 ktl=da94 b=10 ci=0 co=0 ca=0
48− 3 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1846/0/0 df=8 of=0 ri=1 ql=0 qs=.... kt=0/W/3 ktl=d1cd b=10 ci=0 co=0 ca=0
49− 4 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=369/1/0 df=6 of=0 ri=1 ql=0 qs=.... kt=0/W/4 ktl=e0e7 b=10 ci=0 co=0 ca=0
50− 5 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=381/1/0 df=4 of=0 ri=1 ql=0 qs=.... kt=0/W/5 ktl=fb2f b=10 ci=0 co=0 ca=0
51− 6 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1037/1/0 df=6 of=0 ri=1 ql=0 qs=.... kt=0/W/6 ktl=d2ad b=10 ci=0 co=0 ca=0
52− 7 c=1474 g=1474 pq=1 pgp=1473 qp=0 dt=1572/0/0 df=8 of=0 ri=1 ql=0 qs=.... kt=0/W/7 ktl=cf15 b=10 ci=0 co=0 ca=0
45+ 0 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=545/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/0 ktl=ebc3 b=10 ci=0 co=0 ca=0
46+ 1 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=967/1/0 df=3 of=0 ql=0 qs=.... kt=0/W/1 ktl=58c b=10 ci=151 co=0 ca=0
47+ 2 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1081/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/2 ktl=da94 b=10 ci=0 co=0 ca=0
48+ 3 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1846/0/0 df=8 of=0 ql=0 qs=.... kt=0/W/3 ktl=d1cd b=10 ci=0 co=0 ca=0
49+ 4 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=369/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/4 ktl=e0e7 b=10 ci=0 co=0 ca=0
50+ 5 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=381/1/0 df=4 of=0 ql=0 qs=.... kt=0/W/5 ktl=fb2f b=10 ci=0 co=0 ca=0
51+ 6 c=1480 g=1480 pq=1 pgp=1480 qp=0 dt=1037/1/0 df=6 of=0 ql=0 qs=.... kt=0/W/6 ktl=d2ad b=10 ci=0 co=0 ca=0
52+ 7 c=1474 g=1474 pq=1 pgp=1473 qp=0 dt=1572/0/0 df=8 of=0 ql=0 qs=.... kt=0/W/7 ktl=cf15 b=10 ci=0 co=0 ca=0
5353
5454 The first section lists the rcu_data structures for rcu_sched, the second
5555 for rcu_bh. Note that CONFIG_TREE_PREEMPT_RCU kernels will have an
@@ -119,10 +119,6 @@ o "of" is the number of times that some other CPU has forced a · 63 unmodified lines
119119 CPU is offline when it is really alive and kicking) is a fatal
120120 error, so it makes sense to err conservatively.
121121
122−o "ri" is the number of times that RCU has seen fit to send a
123− reschedule IPI to this CPU in order to get it to report a
124− quiescent state.
125−
126122 o "ql" is the number of RCU callbacks currently residing on
127123 this CPU. This is the total number of callbacks, regardless
128124 of what state they are in (new, waiting for grace period to
arch/s390/kernel/irq.c+1−8
@@ -165,13 +165,6 @@ static inline int ext_hash(u16 code) · 164 unmodified lines
165165 return (code + (code >> 9)) & 0xff;
166166 }
167167
168−static void ext_int_hash_update(struct rcu_head *head)
169−{
170− struct ext_int_info *p = container_of(head, struct ext_int_info, rcu);
171−
172− kfree(p);
173−}
174−
175168 int register_external_interrupt(u16 code, ext_int_handler_t handler)
176169 {
177170 struct ext_int_info *p;
@@ -202,7 +195,7 @@ int unregister_external_interrupt(u16 code, ext_int_handler_t handler) · 24 unmodified lines
202195 list_for_each_entry_rcu(p, &ext_int_hash[index], entry)
203196 if (p->code == code && p->handler == handler) {
204197 list_del_rcu(&p->entry);
205− call_rcu(&p->rcu, ext_int_hash_update);
198+ kfree_rcu(p, rcu);
206199 }
207200 spin_unlock_irqrestore(&ext_int_hash_lock, flags);
208201 return 0;
drivers/target/tcm_fc/tfc_sess.c+1−11
@@ -85,16 +85,6 @@ static struct ft_tport *ft_tport_create(struct fc_lport *lport) · 84 unmodified lines
8585 return tport;
8686 }
8787
88−/*
89− * Free tport via RCU.
90− */
91−static void ft_tport_rcu_free(struct rcu_head *rcu)
92−{
93− struct ft_tport *tport = container_of(rcu, struct ft_tport, rcu);
94−
95− kfree(tport);
96−}
97−
9888 /*
9989 * Delete a target local port.
10090 * Caller holds ft_lport_lock.
@@ -114,7 +104,7 @@ static void ft_tport_delete(struct ft_tport *tport) · 13 unmodified lines
114104 tpg->tport = NULL;
115105 tport->tpg = NULL;
116106 }
117− call_rcu(&tport->rcu, ft_tport_rcu_free);
107+ kfree_rcu(tport, rcu);
118108 }
119109
120110 /*
include/linux/rcupdate.h+80−3
@@ -190,6 +190,33 @@ extern void rcu_idle_exit(void); · 189 unmodified lines
190190 extern void rcu_irq_enter(void);
191191 extern void rcu_irq_exit(void);
192192
193+/**
194+ * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers
195+ * @a: Code that RCU needs to pay attention to.
196+ *
197+ * RCU, RCU-bh, and RCU-sched read-side critical sections are forbidden
198+ * in the inner idle loop, that is, between the rcu_idle_enter() and
199+ * the rcu_idle_exit() -- RCU will happily ignore any such read-side
200+ * critical sections. However, things like powertop need tracepoints
201+ * in the inner idle loop.
202+ *
203+ * This macro provides the way out: RCU_NONIDLE(do_something_with_RCU())
204+ * will tell RCU that it needs to pay attending, invoke its argument
205+ * (in this example, a call to the do_something_with_RCU() function),
206+ * and then tell RCU to go back to ignoring this CPU. It is permissible
207+ * to nest RCU_NONIDLE() wrappers, but the nesting level is currently
208+ * quite limited. If deeper nesting is required, it will be necessary
209+ * to adjust DYNTICK_TASK_NESTING_VALUE accordingly.
210+ *
211+ * This macro may be used from process-level code only.
212+ */
213+#define RCU_NONIDLE(a) \
214+ do { \
215+ rcu_idle_exit(); \
216+ do { a; } while (0); \
217+ rcu_idle_enter(); \
218+ } while (0)
219+
193220 /*
194221 * Infrastructure to implement the synchronize_() primitives in
195222 * TREE_RCU and rcu_barrier_() primitives in TINY_RCU.
@@ -226,6 +253,15 @@ static inline void destroy_rcu_head_on_stack(struct rcu_head *head) · 30 unmodified lines
226253 }
227254 #endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
228255
256+#if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU)
257+bool rcu_lockdep_current_cpu_online(void);
258+#else /* #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
259+static inline bool rcu_lockdep_current_cpu_online(void)
260+{
261+ return 1;
262+}
263+#endif /* #else #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
264+
229265 #ifdef CONFIG_DEBUG_LOCK_ALLOC
230266
231267 #ifdef CONFIG_PROVE_RCU
@@ -239,13 +275,11 @@ static inline int rcu_is_cpu_idle(void) · 7 unmodified lines
239275
240276 static inline void rcu_lock_acquire(struct lockdep_map *map)
241277 {
242− WARN_ON_ONCE(rcu_is_cpu_idle());
243278 lock_acquire(map, 0, 0, 2, 1, NULL, _THIS_IP_);
244279 }
245280
246281 static inline void rcu_lock_release(struct lockdep_map *map)
247282 {
248− WARN_ON_ONCE(rcu_is_cpu_idle());
249283 lock_release(map, 1, _THIS_IP_);
250284 }
251285
@@ -270,6 +304,9 @@ extern int debug_lockdep_rcu_enabled(void); · 18 unmodified lines
270304 * occur in the same context, for example, it is illegal to invoke
271305 * rcu_read_unlock() in process context if the matching rcu_read_lock()
272306 * was invoked from within an irq handler.
307+ *
308+ * Note that rcu_read_lock() is disallowed if the CPU is either idle or
309+ * offline from an RCU perspective, so check for those as well.
273310 */
274311 static inline int rcu_read_lock_held(void)
275312 {
@@ -277,6 +314,8 @@ static inline int rcu_read_lock_held(void) · 1 unmodified line
277314 return 1;
278315 if (rcu_is_cpu_idle())
279316 return 0;
317+ if (!rcu_lockdep_current_cpu_online())
318+ return 0;
280319 return lock_is_held(&rcu_lock_map);
281320 }
282321
@@ -313,6 +352,9 @@ extern int rcu_read_lock_bh_held(void); · 30 unmodified lines
313352 * notice an extended quiescent state to other CPUs that started a grace
314353 * period. Otherwise we would delay any grace period as long as we run in
315354 * the idle task.
355+ *
356+ * Similarly, we avoid claiming an SRCU read lock held if the current
357+ * CPU is offline.
316358 */
317359 #ifdef CONFIG_PREEMPT_COUNT
318360 static inline int rcu_read_lock_sched_held(void)
@@ -323,6 +365,8 @@ static inline int rcu_read_lock_sched_held(void) · 4 unmodified lines
323365 return 1;
324366 if (rcu_is_cpu_idle())
325367 return 0;
368+ if (!rcu_lockdep_current_cpu_online())
369+ return 0;
326370 if (debug_locks)
327371 lockdep_opinion = lock_is_held(&rcu_sched_lock_map);
328372 return lockdep_opinion || preempt_count() != 0 || irqs_disabled();
@@ -381,8 +425,22 @@ extern int rcu_my_thread_group_empty(void); · 52 unmodified lines
381425 } \
382426 } while (0)
383427
428+#if defined(CONFIG_PROVE_RCU) && !defined(CONFIG_PREEMPT_RCU)
429+static inline void rcu_preempt_sleep_check(void)
430+{
431+ rcu_lockdep_assert(!lock_is_held(&rcu_lock_map),
432+ "Illegal context switch in RCU read-side "
433+ "critical section");
434+}
435+#else /* #ifdef CONFIG_PROVE_RCU */
436+static inline void rcu_preempt_sleep_check(void)
437+{
438+}
439+#endif /* #else #ifdef CONFIG_PROVE_RCU */
440+
384441 #define rcu_sleep_check() \
385442 do { \
443+ rcu_preempt_sleep_check(); \
386444 rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map), \
387445 "Illegal context switch in RCU-bh" \
388446 " read-side critical section"); \
@@ -470,6 +528,13 @@ extern int rcu_my_thread_group_empty(void); · 81 unmodified lines
470528 * NULL. Although rcu_access_pointer() may also be used in cases where
471529 * update-side locks prevent the value of the pointer from changing, you
472530 * should instead use rcu_dereference_protected() for this use case.
531+ *
532+ * It is also permissible to use rcu_access_pointer() when read-side
533+ * access to the pointer was removed at least one grace period ago, as
534+ * is the case in the context of the RCU callback that is freeing up
535+ * the data, or after a synchronize_rcu() returns. This can be useful
536+ * when tearing down multi-linked structures after a grace period
537+ * has elapsed.
473538 */
474539 #define rcu_access_pointer(p) __rcu_access_pointer((p), __rcu)
475540
@@ -659,6 +724,8 @@ static inline void rcu_read_lock(void) · 183 unmodified lines
659724 __rcu_read_lock();
660725 __acquire(RCU);
661726 rcu_lock_acquire(&rcu_lock_map);
727+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
728+ "rcu_read_lock() used illegally while idle");
662729 }
663730
664731 /*
@@ -678,6 +745,8 @@ static inline void rcu_read_lock(void) · 13 unmodified lines
678745 */
679746 static inline void rcu_read_unlock(void)
680747 {
748+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
749+ "rcu_read_unlock() used illegally while idle");
681750 rcu_lock_release(&rcu_lock_map);
682751 __release(RCU);
683752 __rcu_read_unlock();
@@ -705,6 +774,8 @@ static inline void rcu_read_lock_bh(void) · 21 unmodified lines
705774 local_bh_disable();
706775 __acquire(RCU_BH);
707776 rcu_lock_acquire(&rcu_bh_lock_map);
777+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
778+ "rcu_read_lock_bh() used illegally while idle");
708779 }
709780
710781 /*
@@ -714,6 +785,8 @@ static inline void rcu_read_lock_bh(void) · 3 unmodified lines
714785 */
715786 static inline void rcu_read_unlock_bh(void)
716787 {
788+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
789+ "rcu_read_unlock_bh() used illegally while idle");
717790 rcu_lock_release(&rcu_bh_lock_map);
718791 __release(RCU_BH);
719792 local_bh_enable();
@@ -737,6 +810,8 @@ static inline void rcu_read_lock_sched(void) · 17 unmodified lines
737810 preempt_disable();
738811 __acquire(RCU_SCHED);
739812 rcu_lock_acquire(&rcu_sched_lock_map);
813+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
814+ "rcu_read_lock_sched() used illegally while idle");
740815 }
741816
742817 /* Used by lockdep and tracing: cannot be traced, cannot call lockdep. */
@@ -753,6 +828,8 @@ static inline notrace void rcu_read_lock_sched_notrace(void) · 10 unmodified lines
753828 */
754829 static inline void rcu_read_unlock_sched(void)
755830 {
831+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
832+ "rcu_read_unlock_sched() used illegally while idle");
756833 rcu_lock_release(&rcu_sched_lock_map);
757834 __release(RCU_SCHED);
758835 preempt_enable();
@@ -841,7 +918,7 @@ void __kfree_rcu(struct rcu_head *head, unsigned long offset) · 82 unmodified lines
841918 /* See the kfree_rcu() header comment. */
842919 BUILD_BUG_ON(!__is_kfree_rcu_offset(offset));
843920
844− call_rcu(head, (rcu_callback)offset);
921+ kfree_call_rcu(head, (rcu_callback)offset);
845922 }
846923
847924 /**
include/linux/rcutiny.h+6−4
@@ -27,13 +27,9 @@ · 26 unmodified lines
2727
2828 #include <linux/cache.h>
2929
30−#ifdef CONFIG_RCU_BOOST
3130 static inline void rcu_init(void)
3231 {
3332 }
34−#else /* #ifdef CONFIG_RCU_BOOST */
35−void rcu_init(void);
36−#endif /* #else #ifdef CONFIG_RCU_BOOST */
3733
3834 static inline void rcu_barrier_bh(void)
3935 {
@@ -83,6 +79,12 @@ static inline void synchronize_sched_expedited(void) · 43 unmodified lines
8379 synchronize_sched();
8480 }
8581
82+static inline void kfree_call_rcu(struct rcu_head *head,
83+ void (*func)(struct rcu_head *rcu))
84+{
85+ call_rcu(head, func);
86+}
87+
8688 #ifdef CONFIG_TINY_RCU
8789
8890 static inline void rcu_preempt_note_context_switch(void)
include/linux/rcutree.h+19−0
@@ -61,6 +61,24 @@ extern void synchronize_rcu_bh(void); · 60 unmodified lines
6161 extern void synchronize_sched_expedited(void);
6262 extern void synchronize_rcu_expedited(void);
6363
64+void kfree_call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu));
65+
66+/**
67+ * synchronize_rcu_bh_expedited - Brute-force RCU-bh grace period
68+ *
69+ * Wait for an RCU-bh grace period to elapse, but use a "big hammer"
70+ * approach to force the grace period to end quickly. This consumes
71+ * significant time on all CPUs and is unfriendly to real-time workloads,
72+ * so is thus not recommended for any sort of common-case code. In fact,
73+ * if you are using synchronize_rcu_bh_expedited() in a loop, please
74+ * restructure your code to batch your updates, and then use a single
75+ * synchronize_rcu_bh() instead.
76+ *
77+ * Note that it is illegal to call this function while holding any lock
78+ * that is acquired by a CPU-hotplug notifier. And yes, it is also illegal
79+ * to call this function from a CPU-hotplug notifier. Failing to observe
80+ * these restriction will result in deadlock.
81+ */
6482 static inline void synchronize_rcu_bh_expedited(void)
6583 {
6684 synchronize_sched_expedited();
@@ -83,6 +101,7 @@ extern void rcu_sched_force_quiescent_state(void); · 16 unmodified lines
83101 /* A context switch is a grace period for RCU-sched and RCU-bh. */
84102 static inline int rcu_blocking_is_gp(void)
85103 {
104+ might_sleep(); /* Check for RCU read-side critical section. */
86105 return num_online_cpus() == 1;
87106 }
88107
include/linux/sched.h+1−2
@@ -1863,8 +1863,7 @@ extern void task_clear_jobctl_pending(struct task_struct *task, · 1862 unmodified lines
18631863 #ifdef CONFIG_PREEMPT_RCU
18641864
18651865 #define RCU_READ_UNLOCK_BLOCKED (1 << 0) /* blocked while in RCU read-side. */
1866−#define RCU_READ_UNLOCK_BOOSTED (1 << 1) /* boosted while in RCU read-side. */
1867−#define RCU_READ_UNLOCK_NEED_QS (1 << 2) /* RCU core needs CPU response. */
1866+#define RCU_READ_UNLOCK_NEED_QS (1 << 1) /* RCU core needs CPU response. */
18681867
18691868 static inline void rcu_copy_process(struct task_struct *p)
18701869 {
include/linux/srcu.h+11−4
@@ -99,15 +99,18 @@ long srcu_batches_completed(struct srcu_struct *sp); · 98 unmodified lines
9999 * power mode. This way we can notice an extended quiescent state to
100100 * other CPUs that started a grace period. Otherwise we would delay any
101101 * grace period as long as we run in the idle task.
102+ *
103+ * Similarly, we avoid claiming an SRCU read lock held if the current
104+ * CPU is offline.
102105 */
103106 static inline int srcu_read_lock_held(struct srcu_struct *sp)
104107 {
105− if (rcu_is_cpu_idle())
106− return 0;
107−
108108 if (!debug_lockdep_rcu_enabled())
109109 return 1;
110−
110+ if (rcu_is_cpu_idle())
111+ return 0;
112+ if (!rcu_lockdep_current_cpu_online())
113+ return 0;
111114 return lock_is_held(&sp->dep_map);
112115 }
113116
@@ -169,6 +172,8 @@ static inline int srcu_read_lock(struct srcu_struct *sp) __acquires(sp) · 55 unmodified lines
169172 int retval = __srcu_read_lock(sp);
170173
171174 rcu_lock_acquire(&(sp)->dep_map);
175+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
176+ "srcu_read_lock() used illegally while idle");
172177 return retval;
173178 }
174179
@@ -182,6 +187,8 @@ static inline int srcu_read_lock(struct srcu_struct *sp) __acquires(sp) · 7 unmodified lines
182187 static inline void srcu_read_unlock(struct srcu_struct *sp, int idx)
183188 __releases(sp)
184189 {
190+ rcu_lockdep_assert(!rcu_is_cpu_idle(),
191+ "srcu_read_unlock() used illegally while idle");
185192 rcu_lock_release(&(sp)->dep_map);
186193 __srcu_read_unlock(sp, idx);
187194 }
include/trace/events/rcu.h+39−24
@@ -313,19 +313,22 @@ TRACE_EVENT(rcu_prep_idle, · 312 unmodified lines
313313 /*
314314 * Tracepoint for the registration of a single RCU callback function.
315315 * The first argument is the type of RCU, the second argument is
316− * a pointer to the RCU callback itself, and the third element is the
317− * new RCU callback queue length for the current CPU.
316+ * a pointer to the RCU callback itself, the third element is the
317+ * number of lazy callbacks queued, and the fourth element is the
318+ * total number of callbacks queued.
318319 */
319320 TRACE_EVENT(rcu_callback,
320321
321− TP_PROTO(char *rcuname, struct rcu_head *rhp, long qlen),
322+ TP_PROTO(char *rcuname, struct rcu_head *rhp, long qlen_lazy,
323+ long qlen),
322324
323− TP_ARGS(rcuname, rhp, qlen),
325+ TP_ARGS(rcuname, rhp, qlen_lazy, qlen),
324326
325327 TP_STRUCT__entry(
326328 __field(char *, rcuname)
327329 __field(void *, rhp)
328330 __field(void *, func)
331+ __field(long, qlen_lazy)
329332 __field(long, qlen)
330333 ),
331334
@@ -333,11 +336,13 @@ TRACE_EVENT(rcu_callback, · 1 unmodified line
333336 __entry->rcuname = rcuname;
334337 __entry->rhp = rhp;
335338 __entry->func = rhp->func;
339+ __entry->qlen_lazy = qlen_lazy;
336340 __entry->qlen = qlen;
337341 ),
338342
339− TP_printk("%s rhp=%p func=%pf %ld",
340− __entry->rcuname, __entry->rhp, __entry->func, __entry->qlen)
343+ TP_printk("%s rhp=%p func=%pf %ld/%ld",
344+ __entry->rcuname, __entry->rhp, __entry->func,
345+ __entry->qlen_lazy, __entry->qlen)
341346 );
342347
343348 /*
@@ -345,20 +350,21 @@ TRACE_EVENT(rcu_callback, · 1 unmodified line
345350 * kfree() form. The first argument is the RCU type, the second argument
346351 * is a pointer to the RCU callback, the third argument is the offset
347352 * of the callback within the enclosing RCU-protected data structure,
348− * and the fourth argument is the new RCU callback queue length for the
349− * current CPU.
353+ * the fourth argument is the number of lazy callbacks queued, and the
354+ * fifth argument is the total number of callbacks queued.
350355 */
351356 TRACE_EVENT(rcu_kfree_callback,
352357
353358 TP_PROTO(char *rcuname, struct rcu_head *rhp, unsigned long offset,
354− long qlen),
359+ long qlen_lazy, long qlen),
355360
356− TP_ARGS(rcuname, rhp, offset, qlen),
361+ TP_ARGS(rcuname, rhp, offset, qlen_lazy, qlen),
357362
358363 TP_STRUCT__entry(
359364 __field(char *, rcuname)
360365 __field(void *, rhp)
361366 __field(unsigned long, offset)
367+ __field(long, qlen_lazy)
362368 __field(long, qlen)
363369 ),
364370
@@ -366,41 +372,45 @@ TRACE_EVENT(rcu_kfree_callback, · 1 unmodified line
366372 __entry->rcuname = rcuname;
367373 __entry->rhp = rhp;
368374 __entry->offset = offset;
375+ __entry->qlen_lazy = qlen_lazy;
369376 __entry->qlen = qlen;
370377 ),
371378
372− TP_printk("%s rhp=%p func=%ld %ld",
379+ TP_printk("%s rhp=%p func=%ld %ld/%ld",
373380 __entry->rcuname, __entry->rhp, __entry->offset,
374− __entry->qlen)
381+ __entry->qlen_lazy, __entry->qlen)
375382 );
376383
377384 /*
378385 * Tracepoint for marking the beginning rcu_do_batch, performed to start
379386 * RCU callback invocation. The first argument is the RCU flavor,
380− * the second is the total number of callbacks (including those that
381− * are not yet ready to be invoked), and the third argument is the
382− * current RCU-callback batch limit.
387+ * the second is the number of lazy callbacks queued, the third is
388+ * the total number of callbacks queued, and the fourth argument is
389+ * the current RCU-callback batch limit.
383390 */
384391 TRACE_EVENT(rcu_batch_start,
385392
386− TP_PROTO(char *rcuname, long qlen, int blimit),
393+ TP_PROTO(char *rcuname, long qlen_lazy, long qlen, int blimit),
387394
388− TP_ARGS(rcuname, qlen, blimit),
395+ TP_ARGS(rcuname, qlen_lazy, qlen, blimit),
389396
390397 TP_STRUCT__entry(
391398 __field(char *, rcuname)
399+ __field(long, qlen_lazy)
392400 __field(long, qlen)
393401 __field(int, blimit)
394402 ),
395403
396404 TP_fast_assign(
397405 __entry->rcuname = rcuname;
406+ __entry->qlen_lazy = qlen_lazy;
398407 __entry->qlen = qlen;
399408 __entry->blimit = blimit;
400409 ),
401410
402− TP_printk("%s CBs=%ld bl=%d",
403− __entry->rcuname, __entry->qlen, __entry->blimit)
411+ TP_printk("%s CBs=%ld/%ld bl=%d",
412+ __entry->rcuname, __entry->qlen_lazy, __entry->qlen,
413+ __entry->blimit)
404414 );
405415
406416 /*
@@ -531,16 +541,21 @@ TRACE_EVENT(rcu_torture_read, · 124 unmodified lines
531541 #else /* #ifdef CONFIG_RCU_TRACE */
532542
533543 #define trace_rcu_grace_period(rcuname, gpnum, gpevent) do { } while (0)
534−#define trace_rcu_grace_period_init(rcuname, gpnum, level, grplo, grphi, qsmask) do { } while (0)
544+#define trace_rcu_grace_period_init(rcuname, gpnum, level, grplo, grphi, \
545+ qsmask) do { } while (0)
535546 #define trace_rcu_preempt_task(rcuname, pid, gpnum) do { } while (0)
536547 #define trace_rcu_unlock_preempted_task(rcuname, gpnum, pid) do { } while (0)
537−#define trace_rcu_quiescent_state_report(rcuname, gpnum, mask, qsmask, level, grplo, grphi, gp_tasks) do { } while (0)
548+#define trace_rcu_quiescent_state_report(rcuname, gpnum, mask, qsmask, level, \
549+ grplo, grphi, gp_tasks) do { } \
550+ while (0)
538551 #define trace_rcu_fqs(rcuname, gpnum, cpu, qsevent) do { } while (0)
539552 #define trace_rcu_dyntick(polarity, oldnesting, newnesting) do { } while (0)
540553 #define trace_rcu_prep_idle(reason) do { } while (0)
541−#define trace_rcu_callback(rcuname, rhp, qlen) do { } while (0)
542−#define trace_rcu_kfree_callback(rcuname, rhp, offset, qlen) do { } while (0)
543−#define trace_rcu_batch_start(rcuname, qlen, blimit) do { } while (0)
554+#define trace_rcu_callback(rcuname, rhp, qlen_lazy, qlen) do { } while (0)
555+#define trace_rcu_kfree_callback(rcuname, rhp, offset, qlen_lazy, qlen) \
556+ do { } while (0)
557+#define trace_rcu_batch_start(rcuname, qlen_lazy, qlen, blimit) \
558+ do { } while (0)
544559 #define trace_rcu_invoke_callback(rcuname, rhp) do { } while (0)
545560 #define trace_rcu_invoke_kfree_callback(rcuname, rhp, offset) do { } while (0)
546561 #define trace_rcu_batch_end(rcuname, callbacks_invoked, cb, nr, iit, risk) \
init/Kconfig+0−9
@@ -438,15 +438,6 @@ config PREEMPT_RCU · 437 unmodified lines
438438 This option enables preemptible-RCU code that is common between
439439 the TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations.
440440
441−config RCU_TRACE
442− bool "Enable tracing for RCU"
443− help
444− This option provides tracing in RCU which presents stats
445− in debugfs for debugging RCU implementation.
446−
447− Say Y here if you want to enable RCU tracing
448− Say N if you are unsure.
449−
450441 config RCU_FANOUT
451442 int "Tree-based hierarchical RCU fanout value"
452443 range 2 64 if 64BIT
kernel/lockdep.c+7−1
@@ -4176,7 +4176,13 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s) · 4175 unmodified lines
41764176 printk("-------------------------------\n");
41774177 printk("%s:%d %s!\n", file, line, s);
41784178 printk("\nother info that might help us debug this:\n\n");
4179− printk("\nrcu_scheduler_active = %d, debug_locks = %d\n", rcu_scheduler_active, debug_locks);
4179+ printk("\n%srcu_scheduler_active = %d, debug_locks = %d\n",
4180+ !rcu_lockdep_current_cpu_online()
4181+ ? "RCU used illegally from offline CPU!\n"
4182+ : rcu_is_cpu_idle()
4183+ ? "RCU used illegally from idle CPU!\n"
4184+ : "",
4185+ rcu_scheduler_active, debug_locks);
41804186
41814187 /*
41824188 * If a CPU is in the RCU-free window in idle (ie: in the section
kernel/rcu.h+23−3
@@ -33,8 +33,27 @@ · 32 unmodified lines
3333 * Process-level increment to ->dynticks_nesting field. This allows for
3434 * architectures that use half-interrupts and half-exceptions from
3535 * process context.
36+ *
37+ * DYNTICK_TASK_NEST_MASK defines a field of width DYNTICK_TASK_NEST_WIDTH
38+ * that counts the number of process-based reasons why RCU cannot
39+ * consider the corresponding CPU to be idle, and DYNTICK_TASK_NEST_VALUE
40+ * is the value used to increment or decrement this field.
41+ *
42+ * The rest of the bits could in principle be used to count interrupts,
43+ * but this would mean that a negative-one value in the interrupt
44+ * field could incorrectly zero out the DYNTICK_TASK_NEST_MASK field.
45+ * We therefore provide a two-bit guard field defined by DYNTICK_TASK_MASK
46+ * that is set to DYNTICK_TASK_FLAG upon initial exit from idle.
47+ * The DYNTICK_TASK_EXIT_IDLE value is thus the combined value used upon
48+ * initial exit from idle.
3649 */
37−#define DYNTICK_TASK_NESTING (LLONG_MAX / 2 - 1)
50+#define DYNTICK_TASK_NEST_WIDTH 7
51+#define DYNTICK_TASK_NEST_VALUE ((LLONG_MAX >> DYNTICK_TASK_NEST_WIDTH) + 1)
52+#define DYNTICK_TASK_NEST_MASK (LLONG_MAX - DYNTICK_TASK_NEST_VALUE + 1)
53+#define DYNTICK_TASK_FLAG ((DYNTICK_TASK_NEST_VALUE / 8) * 2)
54+#define DYNTICK_TASK_MASK ((DYNTICK_TASK_NEST_VALUE / 8) * 3)
55+#define DYNTICK_TASK_EXIT_IDLE (DYNTICK_TASK_NEST_VALUE + \
56+ DYNTICK_TASK_FLAG)
3857
3958 /*
4059 * debug_rcu_head_queue()/debug_rcu_head_unqueue() are used internally
@@ -50,7 +69,6 @@ extern struct debug_obj_descr rcuhead_debug_descr; · 9 unmodified lines
5069
5170 static inline void debug_rcu_head_queue(struct rcu_head *head)
5271 {
53− WARN_ON_ONCE((unsigned long)head & 0x3);
5472 debug_object_activate(head, &rcuhead_debug_descr);
5573 debug_object_active_state(head, &rcuhead_debug_descr,
5674 STATE_RCU_HEAD_READY,
@@ -76,16 +94,18 @@ static inline void debug_rcu_head_unqueue(struct rcu_head *head) · 19 unmodified lines
7694
7795 extern void kfree(const void *);
7896
79−static inline void __rcu_reclaim(char *rn, struct rcu_head *head)
97+static inline bool __rcu_reclaim(char *rn, struct rcu_head *head)
8098 {
8199 unsigned long offset = (unsigned long)head->func;
82100
83101 if (__is_kfree_rcu_offset(offset)) {
84102 RCU_TRACE(trace_rcu_invoke_kfree_callback(rn, head, offset));
85103 kfree((void *)head - offset);
104+ return 1;
86105 } else {
87106 RCU_TRACE(trace_rcu_invoke_callback(rn, head));
88107 head->func(head);
108+ return 0;
89109 }
90110 }
91111
kernel/rcupdate.c+5−0
@@ -88,6 +88,9 @@ EXPORT_SYMBOL_GPL(debug_lockdep_rcu_enabled); · 87 unmodified lines
8888 * section.
8989 *
9090 * Check debug_lockdep_rcu_enabled() to prevent false positives during boot.
91+ *
92+ * Note that rcu_read_lock() is disallowed if the CPU is either idle or
93+ * offline from an RCU perspective, so check for those as well.
9194 */
9295 int rcu_read_lock_bh_held(void)
9396 {
@@ -95,6 +98,8 @@ int rcu_read_lock_bh_held(void) · 1 unmodified line
9598 return 1;
9699 if (rcu_is_cpu_idle())
97100 return 0;
101+ if (!rcu_lockdep_current_cpu_online())
102+ return 0;
98103 return in_softirq() || irqs_disabled();
99104 }
100105 EXPORT_SYMBOL_GPL(rcu_read_lock_bh_held);
kernel/rcutiny.c+20−6
@@ -53,7 +53,7 @@ static void __call_rcu(struct rcu_head *head, · 52 unmodified lines
5353
5454 #include "rcutiny_plugin.h"
5555
56−static long long rcu_dynticks_nesting = DYNTICK_TASK_NESTING;
56+static long long rcu_dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
5757
5858 /* Common code for rcu_idle_enter() and rcu_irq_exit(), see kernel/rcutree.c. */
5959 static void rcu_idle_enter_common(long long oldval)
@@ -88,10 +88,16 @@ void rcu_idle_enter(void) · 28 unmodified lines
8888
8989 local_irq_save(flags);
9090 oldval = rcu_dynticks_nesting;
91− rcu_dynticks_nesting = 0;
91+ WARN_ON_ONCE((rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK) == 0);
92+ if ((rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK) ==
93+ DYNTICK_TASK_NEST_VALUE)
94+ rcu_dynticks_nesting = 0;
95+ else
96+ rcu_dynticks_nesting -= DYNTICK_TASK_NEST_VALUE;
9297 rcu_idle_enter_common(oldval);
9398 local_irq_restore(flags);
9499 }
100+EXPORT_SYMBOL_GPL(rcu_idle_enter);
95101
96102 /*
97103 * Exit an interrupt handler towards idle.
@@ -140,11 +146,15 @@ void rcu_idle_exit(void) · 42 unmodified lines
140146
141147 local_irq_save(flags);
142148 oldval = rcu_dynticks_nesting;
143− WARN_ON_ONCE(oldval != 0);
144− rcu_dynticks_nesting = DYNTICK_TASK_NESTING;
149+ WARN_ON_ONCE(rcu_dynticks_nesting < 0);
150+ if (rcu_dynticks_nesting & DYNTICK_TASK_NEST_MASK)
151+ rcu_dynticks_nesting += DYNTICK_TASK_NEST_VALUE;
152+ else
153+ rcu_dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
145154 rcu_idle_exit_common(oldval);
146155 local_irq_restore(flags);
147156 }
157+EXPORT_SYMBOL_GPL(rcu_idle_exit);
148158
149159 /*
150160 * Enter an interrupt handler, moving away from idle.
@@ -258,7 +268,7 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp) · 107 unmodified lines
258268
259269 /* If no RCU callbacks ready to invoke, just return. */
260270 if (&rcp->rcucblist == rcp->donetail) {
261− RCU_TRACE(trace_rcu_batch_start(rcp->name, 0, -1));
271+ RCU_TRACE(trace_rcu_batch_start(rcp->name, 0, 0, -1));
262272 RCU_TRACE(trace_rcu_batch_end(rcp->name, 0,
263273 ACCESS_ONCE(rcp->rcucblist),
264274 need_resched(),
@@ -269,7 +279,7 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp) · 4 unmodified lines
269279
270280 /* Move the ready-to-invoke callbacks to a local list. */
271281 local_irq_save(flags);
272− RCU_TRACE(trace_rcu_batch_start(rcp->name, 0, -1));
282+ RCU_TRACE(trace_rcu_batch_start(rcp->name, 0, rcp->qlen, -1));
273283 list = rcp->rcucblist;
274284 rcp->rcucblist = *rcp->donetail;
275285 *rcp->donetail = NULL;
@@ -319,6 +329,10 @@ static void rcu_process_callbacks(struct softirq_action *unused) · 43 unmodified lines
319329 */
320330 void synchronize_sched(void)
321331 {
332+ rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map) &&
333+ !lock_is_held(&rcu_lock_map) &&
334+ !lock_is_held(&rcu_sched_lock_map),
335+ "Illegal synchronize_sched() in RCU read-side critical section");
322336 cond_resched();
323337 }
324338 EXPORT_SYMBOL_GPL(synchronize_sched);
kernel/rcutiny_plugin.h+60−17
@@ -132,6 +132,7 @@ static struct rcu_preempt_ctrlblk rcu_preempt_ctrlblk = { · 131 unmodified lines
132132 RCU_TRACE(.rcb.name = "rcu_preempt")
133133 };
134134
135+static void rcu_read_unlock_special(struct task_struct *t);
135136 static int rcu_preempted_readers_exp(void);
136137 static void rcu_report_exp_done(void);
137138
@@ -146,6 +147,16 @@ static int rcu_cpu_blocking_cur_gp(void) · 8 unmodified lines
146147 /*
147148 * Check for a running RCU reader. Because there is only one CPU,
148149 * there can be but one running RCU reader at a time. ;-)
150+ *
151+ * Returns zero if there are no running readers. Returns a positive
152+ * number if there is at least one reader within its RCU read-side
153+ * critical section. Returns a negative number if an outermost reader
154+ * is in the midst of exiting from its RCU read-side critical section
155+ *
156+ * Returns zero if there are no running readers. Returns a positive
157+ * number if there is at least one reader within its RCU read-side
158+ * critical section. Returns a negative number if an outermost reader
159+ * is in the midst of exiting from its RCU read-side critical section.
149160 */
150161 static int rcu_preempt_running_reader(void)
151162 {
@@ -307,7 +318,6 @@ static int rcu_boost(void) · 155 unmodified lines
307318 t = container_of(tb, struct task_struct, rcu_node_entry);
308319 rt_mutex_init_proxy_locked(&mtx, t);
309320 t->rcu_boost_mutex = &mtx;
310− t->rcu_read_unlock_special |= RCU_READ_UNLOCK_BOOSTED;
311321 raw_local_irq_restore(flags);
312322 rt_mutex_lock(&mtx);
313323 rt_mutex_unlock(&mtx); /* Keep lockdep happy. */
@@ -475,7 +485,7 @@ void rcu_preempt_note_context_switch(void) · 161 unmodified lines
475485 unsigned long flags;
476486
477487 local_irq_save(flags); /* must exclude scheduler_tick(). */
478− if (rcu_preempt_running_reader() &&
488+ if (rcu_preempt_running_reader() > 0 &&
479489 (t->rcu_read_unlock_special & RCU_READ_UNLOCK_BLOCKED) == 0) {
480490
481491 /* Possibly blocking in an RCU read-side critical section. */
@@ -494,6 +504,13 @@ void rcu_preempt_note_context_switch(void) · 12 unmodified lines
494504 list_add(&t->rcu_node_entry, &rcu_preempt_ctrlblk.blkd_tasks);
495505 if (rcu_cpu_blocking_cur_gp())
496506 rcu_preempt_ctrlblk.gp_tasks = &t->rcu_node_entry;
507+ } else if (rcu_preempt_running_reader() < 0 &&
508+ t->rcu_read_unlock_special) {
509+ /*
510+ * Complete exit from RCU read-side critical section on
511+ * behalf of preempted instance of __rcu_read_unlock().
512+ */
513+ rcu_read_unlock_special(t);
497514 }
498515
499516 /*
@@ -526,12 +543,15 @@ EXPORT_SYMBOL_GPL(__rcu_read_lock); · 26 unmodified lines
526543 * notify RCU core processing or task having blocked during the RCU
527544 * read-side critical section.
528545 */
529−static void rcu_read_unlock_special(struct task_struct *t)
546+static noinline void rcu_read_unlock_special(struct task_struct *t)
530547 {
531548 int empty;
532549 int empty_exp;
533550 unsigned long flags;
534551 struct list_head *np;
552+#ifdef CONFIG_RCU_BOOST
553+ struct rt_mutex *rbmp = NULL;
554+#endif /* #ifdef CONFIG_RCU_BOOST */
535555 int special;
536556
537557 /*
@@ -552,7 +572,7 @@ static void rcu_read_unlock_special(struct task_struct *t) · 14 unmodified lines
552572 rcu_preempt_cpu_qs();
553573
554574 /* Hardware IRQ handlers cannot block. */
555− if (in_irq()) {
575+ if (in_irq() || in_serving_softirq()) {
556576 local_irq_restore(flags);
557577 return;
558578 }
@@ -597,10 +617,10 @@ static void rcu_read_unlock_special(struct task_struct *t) · 38 unmodified lines
597617 }
598618 #ifdef CONFIG_RCU_BOOST
599619 /* Unboost self if was boosted. */
600− if (special & RCU_READ_UNLOCK_BOOSTED) {
601− t->rcu_read_unlock_special &= ~RCU_READ_UNLOCK_BOOSTED;
602− rt_mutex_unlock(t->rcu_boost_mutex);
620+ if (t->rcu_boost_mutex != NULL) {
621+ rbmp = t->rcu_boost_mutex;
603622 t->rcu_boost_mutex = NULL;
623+ rt_mutex_unlock(rbmp);
604624 }
605625 #endif /* #ifdef CONFIG_RCU_BOOST */
606626 local_irq_restore(flags);
@@ -618,13 +638,22 @@ void __rcu_read_unlock(void) · 11 unmodified lines
618638 struct task_struct *t = current;
619639
620640 barrier(); /* needed if we ever invoke rcu_read_unlock in rcutiny.c */
621− --t->rcu_read_lock_nesting;
622− barrier(); /* decrement before load of ->rcu_read_unlock_special */
623− if (t->rcu_read_lock_nesting == 0 &&
624− unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
625− rcu_read_unlock_special(t);
641+ if (t->rcu_read_lock_nesting != 1)
642+ --t->rcu_read_lock_nesting;
643+ else {
644+ t->rcu_read_lock_nesting = INT_MIN;
645+ barrier(); /* assign before ->rcu_read_unlock_special load */
646+ if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))
647+ rcu_read_unlock_special(t);
648+ barrier(); /* ->rcu_read_unlock_special load before assign */
649+ t->rcu_read_lock_nesting = 0;
650+ }
626651 #ifdef CONFIG_PROVE_LOCKING
627− WARN_ON_ONCE(t->rcu_read_lock_nesting < 0);
652+ {
653+ int rrln = ACCESS_ONCE(t->rcu_read_lock_nesting);
654+
655+ WARN_ON_ONCE(rrln < 0 && rrln > INT_MIN / 2);
656+ }
628657 #endif /* #ifdef CONFIG_PROVE_LOCKING */
629658 }
630659 EXPORT_SYMBOL_GPL(__rcu_read_unlock);
@@ -649,7 +678,7 @@ static void rcu_preempt_check_callbacks(void) · 18 unmodified lines
649678 invoke_rcu_callbacks();
650679 if (rcu_preempt_gp_in_progress() &&
651680 rcu_cpu_blocking_cur_gp() &&
652− rcu_preempt_running_reader())
681+ rcu_preempt_running_reader() > 0)
653682 t->rcu_read_unlock_special |= RCU_READ_UNLOCK_NEED_QS;
654683 }
655684
@@ -706,6 +735,11 @@ EXPORT_SYMBOL_GPL(call_rcu); · 50 unmodified lines
706735 */
707736 void synchronize_rcu(void)
708737 {
738+ rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map) &&
739+ !lock_is_held(&rcu_lock_map) &&
740+ !lock_is_held(&rcu_sched_lock_map),
741+ "Illegal synchronize_rcu() in RCU read-side critical section");
742+
709743 #ifdef CONFIG_DEBUG_LOCK_ALLOC
710744 if (!rcu_scheduler_active)
711745 return;
@@ -882,7 +916,8 @@ static void rcu_preempt_process_callbacks(void) · 170 unmodified lines
882916 static void invoke_rcu_callbacks(void)
883917 {
884918 have_rcu_kthread_work = 1;
885− wake_up(&rcu_kthread_wq);
919+ if (rcu_kthread_task != NULL)
920+ wake_up(&rcu_kthread_wq);
886921 }
887922
888923 #ifdef CONFIG_RCU_TRACE
@@ -943,12 +978,16 @@ early_initcall(rcu_spawn_kthreads); · 54 unmodified lines
943978
944979 #else /* #ifdef CONFIG_RCU_BOOST */
945980
981+/* Hold off callback invocation until early_initcall() time. */
982+static int rcu_scheduler_fully_active __read_mostly;
983+
946984 /*
947985 * Start up softirq processing of callbacks.
948986 */
949987 void invoke_rcu_callbacks(void)
950988 {
951− raise_softirq(RCU_SOFTIRQ);
989+ if (rcu_scheduler_fully_active)
990+ raise_softirq(RCU_SOFTIRQ);
952991 }
953992
954993 #ifdef CONFIG_RCU_TRACE
@@ -963,10 +1002,14 @@ static bool rcu_is_callbacks_kthread(void) · 8 unmodified lines
9631002
9641003 #endif /* #ifdef CONFIG_RCU_TRACE */
9651004
966−void rcu_init(void)
1005+static int __init rcu_scheduler_really_started(void)
9671006 {
1007+ rcu_scheduler_fully_active = 1;
9681008 open_softirq(RCU_SOFTIRQ, rcu_process_callbacks);
1009+ raise_softirq(RCU_SOFTIRQ); /* Invoke any callbacks from early boot. */
1010+ return 0;
9691011 }
1012+early_initcall(rcu_scheduler_really_started);
9701013
9711014 #endif /* #else #ifdef CONFIG_RCU_BOOST */
9721015
kernel/rcutorture.c+86−5
@@ -65,7 +65,10 @@ static int fqs_duration; /* Duration of bursts (us), 0 to disable. */ · 64 unmodified lines
6565 static int fqs_holdoff; /* Hold time within burst (us). */
6666 static int fqs_stutter = 3; /* Wait time between bursts (s). */
6767 static int onoff_interval; /* Wait time between CPU hotplugs, 0=disable. */
68+static int onoff_holdoff; /* Seconds after boot before CPU hotplugs. */
6869 static int shutdown_secs; /* Shutdown time (s). <=0 for no shutdown. */
70+static int stall_cpu; /* CPU-stall duration (s). 0 for no stall. */
71+static int stall_cpu_holdoff = 10; /* Time to wait until stall (s). */
6972 static int test_boost = 1; /* Test RCU prio boost: 0=no, 1=maybe, 2=yes. */
7073 static int test_boost_interval = 7; /* Interval between boost tests, seconds. */
7174 static int test_boost_duration = 4; /* Duration of each boost test, seconds. */
@@ -95,8 +98,14 @@ module_param(fqs_stutter, int, 0444); · 23 unmodified lines
9598 MODULE_PARM_DESC(fqs_stutter, "Wait time between fqs bursts (s)");
9699 module_param(onoff_interval, int, 0444);
97100 MODULE_PARM_DESC(onoff_interval, "Time between CPU hotplugs (s), 0=disable");
101+module_param(onoff_holdoff, int, 0444);
102+MODULE_PARM_DESC(onoff_holdoff, "Time after boot before CPU hotplugs (s)");
98103 module_param(shutdown_secs, int, 0444);
99104 MODULE_PARM_DESC(shutdown_secs, "Shutdown time (s), zero to disable.");
105+module_param(stall_cpu, int, 0444);
106+MODULE_PARM_DESC(stall_cpu, "Stall duration (s), zero to disable.");
107+module_param(stall_cpu_holdoff, int, 0444);
108+MODULE_PARM_DESC(stall_cpu_holdoff, "Time to wait before starting stall (s).");
100109 module_param(test_boost, int, 0444);
101110 MODULE_PARM_DESC(test_boost, "Test RCU prio boost: 0=no, 1=maybe, 2=yes.");
102111 module_param(test_boost_interval, int, 0444);
@@ -129,6 +138,7 @@ static struct task_struct *shutdown_task; · 26 unmodified lines
129138 #ifdef CONFIG_HOTPLUG_CPU
130139 static struct task_struct *onoff_task;
131140 #endif /* #ifdef CONFIG_HOTPLUG_CPU */
141+static struct task_struct *stall_task;
132142
133143 #define RCU_TORTURE_PIPE_LEN 10
134144
@@ -990,12 +1000,12 @@ static void rcu_torture_timer(unsigned long unused) · 855 unmodified lines
9901000 rcu_read_lock_bh_held() ||
9911001 rcu_read_lock_sched_held() ||
9921002 srcu_read_lock_held(&srcu_ctl));
993− do_trace_rcu_torture_read(cur_ops->name, &p->rtort_rcu);
9941003 if (p == NULL) {
9951004 /* Leave because rcu_torture_writer is not yet underway */
9961005 cur_ops->readunlock(idx);
9971006 return;
9981007 }
1008+ do_trace_rcu_torture_read(cur_ops->name, &p->rtort_rcu);
9991009 if (p->rtort_mbtest == 0)
10001010 atomic_inc(&n_rcu_torture_mberror);
10011011 spin_lock(&rand_lock);
@@ -1053,13 +1063,13 @@ rcu_torture_reader(void *arg) · 51 unmodified lines
10531063 rcu_read_lock_bh_held() ||
10541064 rcu_read_lock_sched_held() ||
10551065 srcu_read_lock_held(&srcu_ctl));
1056− do_trace_rcu_torture_read(cur_ops->name, &p->rtort_rcu);
10571066 if (p == NULL) {
10581067 /* Wait for rcu_torture_writer to get underway */
10591068 cur_ops->readunlock(idx);
10601069 schedule_timeout_interruptible(HZ);
10611070 continue;
10621071 }
1072+ do_trace_rcu_torture_read(cur_ops->name, &p->rtort_rcu);
10631073 if (p->rtort_mbtest == 0)
10641074 atomic_inc(&n_rcu_torture_mberror);
10651075 cur_ops->read_delay(&rand);
@@ -1300,13 +1310,13 @@ rcu_torture_print_module_parms(struct rcu_torture_ops *cur_ops, char *tag) · 234 unmodified lines
13001310 "fqs_duration=%d fqs_holdoff=%d fqs_stutter=%d "
13011311 "test_boost=%d/%d test_boost_interval=%d "
13021312 "test_boost_duration=%d shutdown_secs=%d "
1303− "onoff_interval=%d\n",
1313+ "onoff_interval=%d onoff_holdoff=%d\n",
13041314 torture_type, tag, nrealreaders, nfakewriters,
13051315 stat_interval, verbose, test_no_idle_hz, shuffle_interval,
13061316 stutter, irqreader, fqs_duration, fqs_holdoff, fqs_stutter,
13071317 test_boost, cur_ops->can_boost,
13081318 test_boost_interval, test_boost_duration, shutdown_secs,
1309− onoff_interval);
1319+ onoff_interval, onoff_holdoff);
13101320 }
13111321
13121322 static struct notifier_block rcutorture_shutdown_nb = {
@@ -1410,6 +1420,11 @@ rcu_torture_onoff(void *arg) · 97 unmodified lines
14101420 for_each_online_cpu(cpu)
14111421 maxcpu = cpu;
14121422 WARN_ON(maxcpu < 0);
1423+ if (onoff_holdoff > 0) {
1424+ VERBOSE_PRINTK_STRING("rcu_torture_onoff begin holdoff");
1425+ schedule_timeout_interruptible(onoff_holdoff * HZ);
1426+ VERBOSE_PRINTK_STRING("rcu_torture_onoff end holdoff");
1427+ }
14131428 while (!kthread_should_stop()) {
14141429 cpu = (rcu_random(&rand) >> 4) % (maxcpu + 1);
14151430 if (cpu_online(cpu) && cpu_is_hotpluggable(cpu)) {
@@ -1450,12 +1465,15 @@ rcu_torture_onoff(void *arg) · 34 unmodified lines
14501465 static int __cpuinit
14511466 rcu_torture_onoff_init(void)
14521467 {
1468+ int ret;
1469+
14531470 if (onoff_interval <= 0)
14541471 return 0;
14551472 onoff_task = kthread_run(rcu_torture_onoff, NULL, "rcu_torture_onoff");
14561473 if (IS_ERR(onoff_task)) {
1474+ ret = PTR_ERR(onoff_task);
14571475 onoff_task = NULL;
1458− return PTR_ERR(onoff_task);
1476+ return ret;
14591477 }
14601478 return 0;
14611479 }
@@ -1481,6 +1499,63 @@ static void rcu_torture_onoff_cleanup(void) · 19 unmodified lines
14811499
14821500 #endif /* #else #ifdef CONFIG_HOTPLUG_CPU */
14831501
1502+/*
1503+ * CPU-stall kthread. It waits as specified by stall_cpu_holdoff, then
1504+ * induces a CPU stall for the time specified by stall_cpu.
1505+ */
1506+static int __cpuinit rcu_torture_stall(void *args)
1507+{
1508+ unsigned long stop_at;
1509+
1510+ VERBOSE_PRINTK_STRING("rcu_torture_stall task started");
1511+ if (stall_cpu_holdoff > 0) {
1512+ VERBOSE_PRINTK_STRING("rcu_torture_stall begin holdoff");
1513+ schedule_timeout_interruptible(stall_cpu_holdoff * HZ);
1514+ VERBOSE_PRINTK_STRING("rcu_torture_stall end holdoff");
1515+ }
1516+ if (!kthread_should_stop()) {
1517+ stop_at = get_seconds() + stall_cpu;
1518+ /* RCU CPU stall is expected behavior in following code. */
1519+ printk(KERN_ALERT "rcu_torture_stall start.\n");
1520+ rcu_read_lock();
1521+ preempt_disable();
1522+ while (ULONG_CMP_LT(get_seconds(), stop_at))
1523+ continue; /* Induce RCU CPU stall warning. */
1524+ preempt_enable();
1525+ rcu_read_unlock();
1526+ printk(KERN_ALERT "rcu_torture_stall end.\n");
1527+ }
1528+ rcutorture_shutdown_absorb("rcu_torture_stall");
1529+ while (!kthread_should_stop())
1530+ schedule_timeout_interruptible(10 * HZ);
1531+ return 0;
1532+}
1533+
1534+/* Spawn CPU-stall kthread, if stall_cpu specified. */
1535+static int __init rcu_torture_stall_init(void)
1536+{
1537+ int ret;
1538+
1539+ if (stall_cpu <= 0)
1540+ return 0;
1541+ stall_task = kthread_run(rcu_torture_stall, NULL, "rcu_torture_stall");
1542+ if (IS_ERR(stall_task)) {
1543+ ret = PTR_ERR(stall_task);
1544+ stall_task = NULL;
1545+ return ret;
1546+ }
1547+ return 0;
1548+}
1549+
1550+/* Clean up after the CPU-stall kthread, if one was spawned. */
1551+static void rcu_torture_stall_cleanup(void)
1552+{
1553+ if (stall_task == NULL)
1554+ return;
1555+ VERBOSE_PRINTK_STRING("Stopping rcu_torture_stall_task.");
1556+ kthread_stop(stall_task);
1557+}
1558+
14841559 static int rcutorture_cpu_notify(struct notifier_block *self,
14851560 unsigned long action, void *hcpu)
14861561 {
@@ -1523,6 +1598,7 @@ rcu_torture_cleanup(void) · 36 unmodified lines
15231598 fullstop = FULLSTOP_RMMOD;
15241599 mutex_unlock(&fullstop_mutex);
15251600 unregister_reboot_notifier(&rcutorture_shutdown_nb);
1601+ rcu_torture_stall_cleanup();
15261602 if (stutter_task) {
15271603 VERBOSE_PRINTK_STRING("Stopping rcu_torture_stutter task");
15281604 kthread_stop(stutter_task);
@@ -1602,6 +1678,10 @@ rcu_torture_cleanup(void) · 73 unmodified lines
16021678 cur_ops->cleanup();
16031679 if (atomic_read(&n_rcu_torture_error))
16041680 rcu_torture_print_module_parms(cur_ops, "End of test: FAILURE");
1681+ else if (n_online_successes != n_online_attempts ||
1682+ n_offline_successes != n_offline_attempts)
1683+ rcu_torture_print_module_parms(cur_ops,
1684+ "End of test: RCU_HOTPLUG");
16051685 else
16061686 rcu_torture_print_module_parms(cur_ops, "End of test: SUCCESS");
16071687 }
@@ -1819,6 +1899,7 @@ rcu_torture_init(void) · 211 unmodified lines
18191899 }
18201900 rcu_torture_onoff_init();
18211901 register_reboot_notifier(&rcutorture_shutdown_nb);
1902+ rcu_torture_stall_init();
18221903 rcutorture_record_test_transition();
18231904 mutex_unlock(&fullstop_mutex);
18241905 return 0;
kernel/rcutree.c+371−136
@@ -50,6 +50,8 @@ · 49 unmodified lines
5050 #include <linux/wait.h>
5151 #include <linux/kthread.h>
5252 #include <linux/prefetch.h>
53+#include <linux/delay.h>
54+#include <linux/stop_machine.h>
5355
5456 #include "rcutree.h"
5557 #include <trace/events/rcu.h>
@@ -196,7 +198,7 @@ void rcu_note_context_switch(int cpu) · 140 unmodified lines
196198 EXPORT_SYMBOL_GPL(rcu_note_context_switch);
197199
198200 DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = {
199− .dynticks_nesting = DYNTICK_TASK_NESTING,
201+ .dynticks_nesting = DYNTICK_TASK_EXIT_IDLE,
200202 .dynticks = ATOMIC_INIT(1),
201203 };
202204
@@ -208,8 +210,11 @@ module_param(blimit, int, 0); · 5 unmodified lines
208210 module_param(qhimark, int, 0);
209211 module_param(qlowmark, int, 0);
210212
211−int rcu_cpu_stall_suppress __read_mostly;
213+int rcu_cpu_stall_suppress __read_mostly; /* 1 = suppress stall warnings. */
214+int rcu_cpu_stall_timeout __read_mostly = CONFIG_RCU_CPU_STALL_TIMEOUT;
215+
212216 module_param(rcu_cpu_stall_suppress, int, 0644);
217+module_param(rcu_cpu_stall_timeout, int, 0644);
213218
214219 static void force_quiescent_state(struct rcu_state *rsp, int relaxed);
215220 static int rcu_pending(int cpu);
@@ -301,8 +306,6 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp) · 85 unmodified lines
301306 return &rsp->node[0];
302307 }
303308
304−#ifdef CONFIG_SMP
305−
306309 /*
307310 * If the specified CPU is offline, tell the caller that it is in
308311 * a quiescent state. Otherwise, whack it with a reschedule IPI.
@@ -317,30 +320,21 @@ static struct rcu_node *rcu_get_root(struct rcu_state *rsp) · 8 unmodified lines
317320 static int rcu_implicit_offline_qs(struct rcu_data *rdp)
318321 {
319322 /*
320− * If the CPU is offline, it is in a quiescent state. We can
321− * trust its state not to change because interrupts are disabled.
323+ * If the CPU is offline for more than a jiffy, it is in a quiescent
324+ * state. We can trust its state not to change because interrupts
325+ * are disabled. The reason for the jiffy's worth of slack is to
326+ * handle CPUs initializing on the way up and finding their way
327+ * to the idle loop on the way down.
322328 */
323− if (cpu_is_offline(rdp->cpu)) {
329+ if (cpu_is_offline(rdp->cpu) &&
330+ ULONG_CMP_LT(rdp->rsp->gp_start + 2, jiffies)) {
324331 trace_rcu_fqs(rdp->rsp->name, rdp->gpnum, rdp->cpu, "ofl");
325332 rdp->offline_fqs++;
326333 return 1;
327334 }
328−
329− /*
330− * The CPU is online, so send it a reschedule IPI. This forces
331− * it through the scheduler, and (inefficiently) also handles cases
332− * where idle loops fail to inform RCU about the CPU being idle.
333− */
334− if (rdp->cpu != smp_processor_id())
335− smp_send_reschedule(rdp->cpu);
336− else
337− set_need_resched();
338− rdp->resched_ipi++;
339335 return 0;
340336 }
341337
342−#endif /* #ifdef CONFIG_SMP */
343−
344338 /*
345339 * rcu_idle_enter_common - inform RCU that current CPU is moving towards idle
346340 *
@@ -366,6 +360,17 @@ static void rcu_idle_enter_common(struct rcu_dynticks *rdtp, long long oldval) · 19 unmodified lines
366360 atomic_inc(&rdtp->dynticks);
367361 smp_mb__after_atomic_inc(); /* Force ordering with next sojourn. */
368362 WARN_ON_ONCE(atomic_read(&rdtp->dynticks) & 0x1);
363+
364+ /*
365+ * The idle task is not permitted to enter the idle loop while
366+ * in an RCU read-side critical section.
367+ */
368+ rcu_lockdep_assert(!lock_is_held(&rcu_lock_map),
369+ "Illegal idle entry in RCU read-side critical section.");
370+ rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map),
371+ "Illegal idle entry in RCU-bh read-side critical section.");
372+ rcu_lockdep_assert(!lock_is_held(&rcu_sched_lock_map),
373+ "Illegal idle entry in RCU-sched read-side critical section.");
369374 }
370375
371376 /**
@@ -389,10 +394,15 @@ void rcu_idle_enter(void) · 17 unmodified lines
389394 local_irq_save(flags);
390395 rdtp = &__get_cpu_var(rcu_dynticks);
391396 oldval = rdtp->dynticks_nesting;
392− rdtp->dynticks_nesting = 0;
397+ WARN_ON_ONCE((oldval & DYNTICK_TASK_NEST_MASK) == 0);
398+ if ((oldval & DYNTICK_TASK_NEST_MASK) == DYNTICK_TASK_NEST_VALUE)
399+ rdtp->dynticks_nesting = 0;
400+ else
401+ rdtp->dynticks_nesting -= DYNTICK_TASK_NEST_VALUE;
393402 rcu_idle_enter_common(rdtp, oldval);
394403 local_irq_restore(flags);
395404 }
405+EXPORT_SYMBOL_GPL(rcu_idle_enter);
396406
397407 /**
398408 * rcu_irq_exit - inform RCU that current CPU is exiting irq towards idle
@@ -462,7 +472,7 @@ static void rcu_idle_exit_common(struct rcu_dynticks *rdtp, long long oldval) · 63 unmodified lines
462472 * Exit idle mode, in other words, -enter- the mode in which RCU
463473 * read-side critical sections can occur.
464474 *
465− * We crowbar the ->dynticks_nesting field to DYNTICK_TASK_NESTING to
475+ * We crowbar the ->dynticks_nesting field to DYNTICK_TASK_NEST to
466476 * allow for the possibility of usermode upcalls messing up our count
467477 * of interrupt nesting level during the busy period that is just
468478 * now starting.
@@ -476,11 +486,15 @@ void rcu_idle_exit(void) · 7 unmodified lines
476486 local_irq_save(flags);
477487 rdtp = &__get_cpu_var(rcu_dynticks);
478488 oldval = rdtp->dynticks_nesting;
479− WARN_ON_ONCE(oldval != 0);
480− rdtp->dynticks_nesting = DYNTICK_TASK_NESTING;
489+ WARN_ON_ONCE(oldval < 0);
490+ if (oldval & DYNTICK_TASK_NEST_MASK)
491+ rdtp->dynticks_nesting += DYNTICK_TASK_NEST_VALUE;
492+ else
493+ rdtp->dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
481494 rcu_idle_exit_common(rdtp, oldval);
482495 local_irq_restore(flags);
483496 }
497+EXPORT_SYMBOL_GPL(rcu_idle_exit);
484498
485499 /**
486500 * rcu_irq_enter - inform RCU that current CPU is entering irq away from idle
@@ -581,6 +595,49 @@ int rcu_is_cpu_idle(void) · 94 unmodified lines
581595 }
582596 EXPORT_SYMBOL(rcu_is_cpu_idle);
583597
598+#ifdef CONFIG_HOTPLUG_CPU
599+
600+/*
601+ * Is the current CPU online? Disable preemption to avoid false positives
602+ * that could otherwise happen due to the current CPU number being sampled,
603+ * this task being preempted, its old CPU being taken offline, resuming
604+ * on some other CPU, then determining that its old CPU is now offline.
605+ * It is OK to use RCU on an offline processor during initial boot, hence
606+ * the check for rcu_scheduler_fully_active. Note also that it is OK
607+ * for a CPU coming online to use RCU for one jiffy prior to marking itself
608+ * online in the cpu_online_mask. Similarly, it is OK for a CPU going
609+ * offline to continue to use RCU for one jiffy after marking itself
610+ * offline in the cpu_online_mask. This leniency is necessary given the
611+ * non-atomic nature of the online and offline processing, for example,
612+ * the fact that a CPU enters the scheduler after completing the CPU_DYING
613+ * notifiers.
614+ *
615+ * This is also why RCU internally marks CPUs online during the
616+ * CPU_UP_PREPARE phase and offline during the CPU_DEAD phase.
617+ *
618+ * Disable checking if in an NMI handler because we cannot safely report
619+ * errors from NMI handlers anyway.
620+ */
621+bool rcu_lockdep_current_cpu_online(void)
622+{
623+ struct rcu_data *rdp;
624+ struct rcu_node *rnp;
625+ bool ret;
626+
627+ if (in_nmi())
628+ return 1;
629+ preempt_disable();
630+ rdp = &__get_cpu_var(rcu_sched_data);
631+ rnp = rdp->mynode;
632+ ret = (rdp->grpmask & rnp->qsmaskinit) ||
633+ !rcu_scheduler_fully_active;
634+ preempt_enable();
635+ return ret;
636+}
637+EXPORT_SYMBOL_GPL(rcu_lockdep_current_cpu_online);
638+
639+#endif /* #ifdef CONFIG_HOTPLUG_CPU */
640+
584641 #endif /* #ifdef CONFIG_PROVE_RCU */
585642
586643 /**
@@ -595,8 +652,6 @@ int rcu_is_cpu_rrupt_from_idle(void) · 8 unmodified lines
595652 return __get_cpu_var(rcu_dynticks).dynticks_nesting <= 1;
596653 }
597654
598−#ifdef CONFIG_SMP
599−
600655 /*
601656 * Snapshot the specified CPU's dynticks counter so that we can later
602657 * credit them with an implicit quiescent state. Return 1 if this CPU
@@ -640,12 +695,28 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp) · 37 unmodified lines
640695 return rcu_implicit_offline_qs(rdp);
641696 }
642697
643−#endif /* #ifdef CONFIG_SMP */
698+static int jiffies_till_stall_check(void)
699+{
700+ int till_stall_check = ACCESS_ONCE(rcu_cpu_stall_timeout);
701+
702+ /*
703+ * Limit check must be consistent with the Kconfig limits
704+ * for CONFIG_RCU_CPU_STALL_TIMEOUT.
705+ */
706+ if (till_stall_check < 3) {
707+ ACCESS_ONCE(rcu_cpu_stall_timeout) = 3;
708+ till_stall_check = 3;
709+ } else if (till_stall_check > 300) {
710+ ACCESS_ONCE(rcu_cpu_stall_timeout) = 300;
711+ till_stall_check = 300;
712+ }
713+ return till_stall_check * HZ + RCU_STALL_DELAY_DELTA;
714+}
644715
645716 static void record_gp_stall_check_time(struct rcu_state *rsp)
646717 {
647718 rsp->gp_start = jiffies;
648− rsp->jiffies_stall = jiffies + RCU_SECONDS_TILL_STALL_CHECK;
719+ rsp->jiffies_stall = jiffies + jiffies_till_stall_check();
649720 }
650721
651722 static void print_other_cpu_stall(struct rcu_state *rsp)
@@ -664,13 +735,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp) · 12 unmodified lines
664735 raw_spin_unlock_irqrestore(&rnp->lock, flags);
665736 return;
666737 }
667− rsp->jiffies_stall = jiffies + RCU_SECONDS_TILL_STALL_RECHECK;
668−
669− /*
670− * Now rat on any tasks that got kicked up to the root rcu_node
671− * due to CPU offlining.
672− */
673− ndetected = rcu_print_task_stall(rnp);
738+ rsp->jiffies_stall = jiffies + 3 * jiffies_till_stall_check() + 3;
674739 raw_spin_unlock_irqrestore(&rnp->lock, flags);
675740
676741 /*
@@ -678,8 +743,9 @@ static void print_other_cpu_stall(struct rcu_state *rsp) · 1 unmodified line
678743 * See Documentation/RCU/stallwarn.txt for info on how to debug
679744 * RCU CPU stall warnings.
680745 */
681− printk(KERN_ERR "INFO: %s detected stalls on CPUs/tasks: {",
746+ printk(KERN_ERR "INFO: %s detected stalls on CPUs/tasks:",
682747 rsp->name);
748+ print_cpu_stall_info_begin();
683749 rcu_for_each_leaf_node(rsp, rnp) {
684750 raw_spin_lock_irqsave(&rnp->lock, flags);
685751 ndetected += rcu_print_task_stall(rnp);
@@ -688,11 +754,22 @@ static void print_other_cpu_stall(struct rcu_state *rsp) · 2 unmodified lines
688754 continue;
689755 for (cpu = 0; cpu <= rnp->grphi - rnp->grplo; cpu++)
690756 if (rnp->qsmask & (1UL << cpu)) {
691− printk(" %d", rnp->grplo + cpu);
757+ print_cpu_stall_info(rsp, rnp->grplo + cpu);
692758 ndetected++;
693759 }
694760 }
695− printk("} (detected by %d, t=%ld jiffies)\n",
761+
762+ /*
763+ * Now rat on any tasks that got kicked up to the root rcu_node
764+ * due to CPU offlining.
765+ */
766+ rnp = rcu_get_root(rsp);
767+ raw_spin_lock_irqsave(&rnp->lock, flags);
768+ ndetected = rcu_print_task_stall(rnp);
769+ raw_spin_unlock_irqrestore(&rnp->lock, flags);
770+
771+ print_cpu_stall_info_end();
772+ printk(KERN_CONT "(detected by %d, t=%ld jiffies)\n",
696773 smp_processor_id(), (long)(jiffies - rsp->gp_start));
697774 if (ndetected == 0)
698775 printk(KERN_ERR "INFO: Stall ended before state dump start\n");
@@ -716,15 +793,18 @@ static void print_cpu_stall(struct rcu_state *rsp) · 17 unmodified lines
716793 * See Documentation/RCU/stallwarn.txt for info on how to debug
717794 * RCU CPU stall warnings.
718795 */
719− printk(KERN_ERR "INFO: %s detected stall on CPU %d (t=%lu jiffies)\n",
720− rsp->name, smp_processor_id(), jiffies - rsp->gp_start);
796+ printk(KERN_ERR "INFO: %s self-detected stall on CPU", rsp->name);
797+ print_cpu_stall_info_begin();
798+ print_cpu_stall_info(rsp, smp_processor_id());
799+ print_cpu_stall_info_end();
800+ printk(KERN_CONT " (t=%lu jiffies)\n", jiffies - rsp->gp_start);
721801 if (!trigger_all_cpu_backtrace())
722802 dump_stack();
723803
724804 raw_spin_lock_irqsave(&rnp->lock, flags);
725805 if (ULONG_CMP_GE(jiffies, rsp->jiffies_stall))
726− rsp->jiffies_stall =
727− jiffies + RCU_SECONDS_TILL_STALL_RECHECK;
806+ rsp->jiffies_stall = jiffies +
807+ 3 * jiffies_till_stall_check() + 3;
728808 raw_spin_unlock_irqrestore(&rnp->lock, flags);
729809
730810 set_need_resched(); /* kick ourselves to get things going. */
@@ -807,6 +887,7 @@ static void __note_new_gpnum(struct rcu_state *rsp, struct rcu_node *rnp, struct · 76 unmodified lines
807887 rdp->passed_quiesce = 0;
808888 } else
809889 rdp->qs_pending = 0;
890+ zero_cpu_stall_ticks(rdp);
810891 }
811892 }
812893
@@ -943,6 +1024,10 @@ rcu_start_gp_per_cpu(struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_dat · 130 unmodified lines
9431024 * in preparation for detecting the next grace period. The caller must hold
9441025 * the root node's ->lock, which is released before return. Hard irqs must
9451026 * be disabled.
1027+ *
1028+ * Note that it is legal for a dying CPU (which is marked as offline) to
1029+ * invoke this function. This can happen when the dying CPU reports its
1030+ * quiescent state.
9461031 */
9471032 static void
9481033 rcu_start_gp(struct rcu_state *rsp, unsigned long flags)
@@ -980,26 +1065,8 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) · 31 unmodified lines
9801065 rsp->fqs_state = RCU_GP_INIT; /* Hold off force_quiescent_state. */
9811066 rsp->jiffies_force_qs = jiffies + RCU_JIFFIES_TILL_FORCE_QS;
9821067 record_gp_stall_check_time(rsp);
983−
984− /* Special-case the common single-level case. */
985− if (NUM_RCU_NODES == 1) {
986− rcu_preempt_check_blocked_tasks(rnp);
987− rnp->qsmask = rnp->qsmaskinit;
988− rnp->gpnum = rsp->gpnum;
989− rnp->completed = rsp->completed;
990− rsp->fqs_state = RCU_SIGNAL_INIT; /* force_quiescent_state OK */
991− rcu_start_gp_per_cpu(rsp, rnp, rdp);
992− rcu_preempt_boost_start_gp(rnp);
993− trace_rcu_grace_period_init(rsp->name, rnp->gpnum,
994− rnp->level, rnp->grplo,
995− rnp->grphi, rnp->qsmask);
996− raw_spin_unlock_irqrestore(&rnp->lock, flags);
997− return;
998− }
999−
10001068 raw_spin_unlock(&rnp->lock); /* leave irqs disabled. */
10011069
1002−
10031070 /* Exclude any concurrent CPU-hotplug operations. */
10041071 raw_spin_lock(&rsp->onofflock); /* irqs already disabled. */
10051072
@@ -1245,53 +1312,115 @@ rcu_check_quiescent_state(struct rcu_state *rsp, struct rcu_data *rdp) · 239 unmodified lines
12451312
12461313 /*
12471314 * Move a dying CPU's RCU callbacks to online CPU's callback list.
1248− * Synchronization is not required because this function executes
1249− * in stop_machine() context.
1315+ * Also record a quiescent state for this CPU for the current grace period.
1316+ * Synchronization and interrupt disabling are not required because
1317+ * this function executes in stop_machine() context. Therefore, cleanup
1318+ * operations that might block must be done later from the CPU_DEAD
1319+ * notifier.
1320+ *
1321+ * Note that the outgoing CPU's bit has already been cleared in the
1322+ * cpu_online_mask. This allows us to randomly pick a callback
1323+ * destination from the bits set in that mask.
12501324 */
1251−static void rcu_send_cbs_to_online(struct rcu_state *rsp)
1325+static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
12521326 {
12531327 int i;
1254− /* current DYING CPU is cleared in the cpu_online_mask */
1328+ unsigned long mask;
12551329 int receive_cpu = cpumask_any(cpu_online_mask);
12561330 struct rcu_data *rdp = this_cpu_ptr(rsp->rda);
12571331 struct rcu_data *receive_rdp = per_cpu_ptr(rsp->rda, receive_cpu);
1332+ RCU_TRACE(struct rcu_node *rnp = rdp->mynode); /* For dying CPU. */
1333+
1334+ /* First, adjust the counts. */
1335+ if (rdp->nxtlist != NULL) {
1336+ receive_rdp->qlen_lazy += rdp->qlen_lazy;
1337+ receive_rdp->qlen += rdp->qlen;
1338+ rdp->qlen_lazy = 0;
1339+ rdp->qlen = 0;
1340+ }
12581341
1259− if (rdp->nxtlist == NULL)
1260− return; /* irqs disabled, so comparison is stable. */
1342+ /*
1343+ * Next, move ready-to-invoke callbacks to be invoked on some
1344+ * other CPU. These will not be required to pass through another
1345+ * grace period: They are done, regardless of CPU.
1346+ */
1347+ if (rdp->nxtlist != NULL &&
1348+ rdp->nxttail[RCU_DONE_TAIL] != &rdp->nxtlist) {
1349+ struct rcu_head *oldhead;
1350+ struct rcu_head **oldtail;
1351+ struct rcu_head **newtail;
1352+
1353+ oldhead = rdp->nxtlist;
1354+ oldtail = receive_rdp->nxttail[RCU_DONE_TAIL];
1355+ rdp->nxtlist = *rdp->nxttail[RCU_DONE_TAIL];
1356+ *rdp->nxttail[RCU_DONE_TAIL] = *oldtail;
1357+ *receive_rdp->nxttail[RCU_DONE_TAIL] = oldhead;
1358+ newtail = rdp->nxttail[RCU_DONE_TAIL];
1359+ for (i = RCU_DONE_TAIL; i < RCU_NEXT_SIZE; i++) {
1360+ if (receive_rdp->nxttail[i] == oldtail)
1361+ receive_rdp->nxttail[i] = newtail;
1362+ if (rdp->nxttail[i] == newtail)
1363+ rdp->nxttail[i] = &rdp->nxtlist;
1364+ }
1365+ }
12611366
1262− *receive_rdp->nxttail[RCU_NEXT_TAIL] = rdp->nxtlist;
1263− receive_rdp->nxttail[RCU_NEXT_TAIL] = rdp->nxttail[RCU_NEXT_TAIL];
1264− receive_rdp->qlen += rdp->qlen;
1265− receive_rdp->n_cbs_adopted += rdp->qlen;
1266− rdp->n_cbs_orphaned += rdp->qlen;
1367+ /*
1368+ * Finally, put the rest of the callbacks at the end of the list.
1369+ * The ones that made it partway through get to start over: We
1370+ * cannot assume that grace periods are synchronized across CPUs.
1371+ * (We could splice RCU_WAIT_TAIL into RCU_NEXT_READY_TAIL, but
1372+ * this does not seem compelling. Not yet, anyway.)
1373+ */
1374+ if (rdp->nxtlist != NULL) {
1375+ *receive_rdp->nxttail[RCU_NEXT_TAIL] = rdp->nxtlist;
1376+ receive_rdp->nxttail[RCU_NEXT_TAIL] =
1377+ rdp->nxttail[RCU_NEXT_TAIL];
1378+ receive_rdp->n_cbs_adopted += rdp->qlen;
1379+ rdp->n_cbs_orphaned += rdp->qlen;
1380+
1381+ rdp->nxtlist = NULL;
1382+ for (i = 0; i < RCU_NEXT_SIZE; i++)
1383+ rdp->nxttail[i] = &rdp->nxtlist;
1384+ }
12671385
1268− rdp->nxtlist = NULL;
1269− for (i = 0; i < RCU_NEXT_SIZE; i++)
1270− rdp->nxttail[i] = &rdp->nxtlist;
1271− rdp->qlen = 0;
1386+ /*
1387+ * Record a quiescent state for the dying CPU. This is safe
1388+ * only because we have already cleared out the callbacks.
1389+ * (Otherwise, the RCU core might try to schedule the invocation
1390+ * of callbacks on this now-offline CPU, which would be bad.)
1391+ */
1392+ mask = rdp->grpmask; /* rnp->grplo is constant. */
1393+ trace_rcu_grace_period(rsp->name,
1394+ rnp->gpnum + 1 - !!(rnp->qsmask & mask),
1395+ "cpuofl");
1396+ rcu_report_qs_rdp(smp_processor_id(), rsp, rdp, rsp->gpnum);
1397+ /* Note that rcu_report_qs_rdp() might call trace_rcu_grace_period(). */
12721398 }
12731399
12741400 /*
1275− * Remove the outgoing CPU from the bitmasks in the rcu_node hierarchy
1276− * and move all callbacks from the outgoing CPU to the current one.
1401+ * The CPU has been completely removed, and some other CPU is reporting
1402+ * this fact from process context. Do the remainder of the cleanup.
12771403 * There can only be one CPU hotplug operation at a time, so no other
12781404 * CPU can be attempting to update rcu_cpu_kthread_task.
12791405 */
1280−static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp)
1406+static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)
12811407 {
12821408 unsigned long flags;
12831409 unsigned long mask;
12841410 int need_report = 0;
12851411 struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
1286− struct rcu_node *rnp;
1412+ struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rnp. */
12871413
1414+ /* Adjust any no-longer-needed kthreads. */
12881415 rcu_stop_cpu_kthread(cpu);
1416+ rcu_node_kthread_setaffinity(rnp, -1);
1417+
1418+ /* Remove the dying CPU from the bitmasks in the rcu_node hierarchy. */
12891419
12901420 /* Exclude any attempts to start a new grace period. */
12911421 raw_spin_lock_irqsave(&rsp->onofflock, flags);
12921422
12931423 /* Remove the outgoing CPU from the masks in the rcu_node hierarchy. */
1294− rnp = rdp->mynode; /* this is the outgoing CPU's rnp. */
12951424 mask = rdp->grpmask; /* rnp->grplo is constant. */
12961425 do {
12971426 raw_spin_lock(&rnp->lock); /* irqs already disabled. */
@@ -1299,20 +1428,11 @@ static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp) · 1 unmodified line
12991428 if (rnp->qsmaskinit != 0) {
13001429 if (rnp != rdp->mynode)
13011430 raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
1302− else
1303− trace_rcu_grace_period(rsp->name,
1304− rnp->gpnum + 1 -
1305− !!(rnp->qsmask & mask),
1306− "cpuofl");
13071431 break;
13081432 }
1309− if (rnp == rdp->mynode) {
1310− trace_rcu_grace_period(rsp->name,
1311− rnp->gpnum + 1 -
1312− !!(rnp->qsmask & mask),
1313− "cpuofl");
1433+ if (rnp == rdp->mynode)
13141434 need_report = rcu_preempt_offline_tasks(rsp, rnp, rdp);
1315− } else
1435+ else
13161436 raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
13171437 mask = rnp->grpmask;
13181438 rnp = rnp->parent;
@@ -1332,29 +1452,15 @@ static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp) · 13 unmodified lines
13321452 raw_spin_unlock_irqrestore(&rnp->lock, flags);
13331453 if (need_report & RCU_OFL_TASKS_EXP_GP)
13341454 rcu_report_exp_rnp(rsp, rnp, true);
1335− rcu_node_kthread_setaffinity(rnp, -1);
1336−}
1337−
1338−/*
1339− * Remove the specified CPU from the RCU hierarchy and move any pending
1340− * callbacks that it might have to the current CPU. This code assumes
1341− * that at least one CPU in the system will remain running at all times.
1342− * Any attempt to offline -all- CPUs is likely to strand RCU callbacks.
1343− */
1344−static void rcu_offline_cpu(int cpu)
1345−{
1346− __rcu_offline_cpu(cpu, &rcu_sched_state);
1347− __rcu_offline_cpu(cpu, &rcu_bh_state);
1348− rcu_preempt_offline_cpu(cpu);
13491455 }
13501456
13511457 #else /* #ifdef CONFIG_HOTPLUG_CPU */
13521458
1353−static void rcu_send_cbs_to_online(struct rcu_state *rsp)
1459+static void rcu_cleanup_dying_cpu(struct rcu_state *rsp)
13541460 {
13551461 }
13561462
1357−static void rcu_offline_cpu(int cpu)
1463+static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)
13581464 {
13591465 }
13601466
@@ -1368,11 +1474,11 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) · 7 unmodified lines
13681474 {
13691475 unsigned long flags;
13701476 struct rcu_head *next, *list, **tail;
1371− int bl, count;
1477+ int bl, count, count_lazy;
13721478
13731479 /* If no callbacks are ready, just return.*/
13741480 if (!cpu_has_callbacks_ready_to_invoke(rdp)) {
1375− trace_rcu_batch_start(rsp->name, 0, 0);
1481+ trace_rcu_batch_start(rsp->name, rdp->qlen_lazy, rdp->qlen, 0);
13761482 trace_rcu_batch_end(rsp->name, 0, !!ACCESS_ONCE(rdp->nxtlist),
13771483 need_resched(), is_idle_task(current),
13781484 rcu_is_callbacks_kthread());
@@ -1384,8 +1490,9 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) · 5 unmodified lines
13841490 * races with call_rcu() from interrupt handlers.
13851491 */
13861492 local_irq_save(flags);
1493+ WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
13871494 bl = rdp->blimit;
1388− trace_rcu_batch_start(rsp->name, rdp->qlen, bl);
1495+ trace_rcu_batch_start(rsp->name, rdp->qlen_lazy, rdp->qlen, bl);
13891496 list = rdp->nxtlist;
13901497 rdp->nxtlist = *rdp->nxttail[RCU_DONE_TAIL];
13911498 *rdp->nxttail[RCU_DONE_TAIL] = NULL;
@@ -1396,12 +1503,13 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) · 4 unmodified lines
13961503 local_irq_restore(flags);
13971504
13981505 /* Invoke callbacks. */
1399− count = 0;
1506+ count = count_lazy = 0;
14001507 while (list) {
14011508 next = list->next;
14021509 prefetch(next);
14031510 debug_rcu_head_unqueue(list);
1404− __rcu_reclaim(rsp->name, list);
1511+ if (__rcu_reclaim(rsp->name, list))
1512+ count_lazy++;
14051513 list = next;
14061514 /* Stop only if limit reached and CPU has something to do. */
14071515 if (++count >= bl &&
@@ -1416,6 +1524,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) · 8 unmodified lines
14161524 rcu_is_callbacks_kthread());
14171525
14181526 /* Update count, and requeue any remaining callbacks. */
1527+ rdp->qlen_lazy -= count_lazy;
14191528 rdp->qlen -= count;
14201529 rdp->n_cbs_invoked += count;
14211530 if (list != NULL) {
@@ -1458,6 +1567,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) · 36 unmodified lines
14581567 void rcu_check_callbacks(int cpu, int user)
14591568 {
14601569 trace_rcu_utilization("Start scheduler-tick");
1570+ increment_cpu_stall_ticks();
14611571 if (user || rcu_is_cpu_rrupt_from_idle()) {
14621572
14631573 /*
@@ -1492,8 +1602,6 @@ void rcu_check_callbacks(int cpu, int user) · 28 unmodified lines
14921602 trace_rcu_utilization("End scheduler-tick");
14931603 }
14941604
1495−#ifdef CONFIG_SMP
1496−
14971605 /*
14981606 * Scan the leaf rcu_node structures, processing dyntick state for any that
14991607 * have not yet encountered a quiescent state, using the function specified.
@@ -1616,15 +1724,6 @@ unlock_fqs_ret: · 116 unmodified lines
16161724 trace_rcu_utilization("End fqs");
16171725 }
16181726
1619−#else /* #ifdef CONFIG_SMP */
1620−
1621−static void force_quiescent_state(struct rcu_state *rsp, int relaxed)
1622−{
1623− set_need_resched();
1624−}
1625−
1626−#endif /* #else #ifdef CONFIG_SMP */
1627−
16281727 /*
16291728 * This does the RCU core processing work for the specified rcu_state
16301729 * and rcu_data structures. This may be called only from the CPU to
@@ -1702,11 +1801,12 @@ static void invoke_rcu_core(void) · 71 unmodified lines
17021801
17031802 static void
17041803 __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),
1705− struct rcu_state *rsp)
1804+ struct rcu_state *rsp, bool lazy)
17061805 {
17071806 unsigned long flags;
17081807 struct rcu_data *rdp;
17091808
1809+ WARN_ON_ONCE((unsigned long)head & 0x3); /* Misaligned rcu_head! */
17101810 debug_rcu_head_queue(head);
17111811 head->func = func;
17121812 head->next = NULL;
@@ -1720,18 +1820,21 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), · 7 unmodified lines
17201820 * a quiescent state betweentimes.
17211821 */
17221822 local_irq_save(flags);
1823+ WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
17231824 rdp = this_cpu_ptr(rsp->rda);
17241825
17251826 /* Add the callback to our list. */
17261827 *rdp->nxttail[RCU_NEXT_TAIL] = head;
17271828 rdp->nxttail[RCU_NEXT_TAIL] = &head->next;
17281829 rdp->qlen++;
1830+ if (lazy)
1831+ rdp->qlen_lazy++;
17291832
17301833 if (__is_kfree_rcu_offset((unsigned long)func))
17311834 trace_rcu_kfree_callback(rsp->name, head, (unsigned long)func,
1732− rdp->qlen);
1835+ rdp->qlen_lazy, rdp->qlen);
17331836 else
1734− trace_rcu_callback(rsp->name, head, rdp->qlen);
1837+ trace_rcu_callback(rsp->name, head, rdp->qlen_lazy, rdp->qlen);
17351838
17361839 /* If interrupts were disabled, don't dive into RCU core. */
17371840 if (irqs_disabled_flags(flags)) {
@@ -1778,16 +1881,16 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), · 40 unmodified lines
17781881 */
17791882 void call_rcu_sched(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
17801883 {
1781− __call_rcu(head, func, &rcu_sched_state);
1884+ __call_rcu(head, func, &rcu_sched_state, 0);
17821885 }
17831886 EXPORT_SYMBOL_GPL(call_rcu_sched);
17841887
17851888 /*
1786− * Queue an RCU for invocation after a quicker grace period.
1889+ * Queue an RCU callback for invocation after a quicker grace period.
17871890 */
17881891 void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
17891892 {
1790− __call_rcu(head, func, &rcu_bh_state);
1893+ __call_rcu(head, func, &rcu_bh_state, 0);
17911894 }
17921895 EXPORT_SYMBOL_GPL(call_rcu_bh);
17931896
@@ -1816,6 +1919,10 @@ EXPORT_SYMBOL_GPL(call_rcu_bh); · 22 unmodified lines
18161919 */
18171920 void synchronize_sched(void)
18181921 {
1922+ rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map) &&
1923+ !lock_is_held(&rcu_lock_map) &&
1924+ !lock_is_held(&rcu_sched_lock_map),
1925+ "Illegal synchronize_sched() in RCU-sched read-side critical section");
18191926 if (rcu_blocking_is_gp())
18201927 return;
18211928 wait_rcu_gp(call_rcu_sched);
@@ -1833,12 +1940,137 @@ EXPORT_SYMBOL_GPL(synchronize_sched); · 11 unmodified lines
18331940 */
18341941 void synchronize_rcu_bh(void)
18351942 {
1943+ rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map) &&
1944+ !lock_is_held(&rcu_lock_map) &&
1945+ !lock_is_held(&rcu_sched_lock_map),
1946+ "Illegal synchronize_rcu_bh() in RCU-bh read-side critical section");
18361947 if (rcu_blocking_is_gp())
18371948 return;
18381949 wait_rcu_gp(call_rcu_bh);
18391950 }
18401951 EXPORT_SYMBOL_GPL(synchronize_rcu_bh);
18411952
1953+static atomic_t sync_sched_expedited_started = ATOMIC_INIT(0);
1954+static atomic_t sync_sched_expedited_done = ATOMIC_INIT(0);
1955+
1956+static int synchronize_sched_expedited_cpu_stop(void *data)
1957+{
1958+ /*
1959+ * There must be a full memory barrier on each affected CPU
1960+ * between the time that try_stop_cpus() is called and the
1961+ * time that it returns.
1962+ *
1963+ * In the current initial implementation of cpu_stop, the
1964+ * above condition is already met when the control reaches
1965+ * this point and the following smp_mb() is not strictly
1966+ * necessary. Do smp_mb() anyway for documentation and
1967+ * robustness against future implementation changes.
1968+ */
1969+ smp_mb(); /* See above comment block. */
1970+ return 0;
1971+}
1972+
1973+/**
1974+ * synchronize_sched_expedited - Brute-force RCU-sched grace period
1975+ *
1976+ * Wait for an RCU-sched grace period to elapse, but use a "big hammer"
1977+ * approach to force the grace period to end quickly. This consumes
1978+ * significant time on all CPUs and is unfriendly to real-time workloads,
1979+ * so is thus not recommended for any sort of common-case code. In fact,
1980+ * if you are using synchronize_sched_expedited() in a loop, please
1981+ * restructure your code to batch your updates, and then use a single
1982+ * synchronize_sched() instead.
1983+ *
1984+ * Note that it is illegal to call this function while holding any lock
1985+ * that is acquired by a CPU-hotplug notifier. And yes, it is also illegal
1986+ * to call this function from a CPU-hotplug notifier. Failing to observe
1987+ * these restriction will result in deadlock.
1988+ *
1989+ * This implementation can be thought of as an application of ticket
1990+ * locking to RCU, with sync_sched_expedited_started and
1991+ * sync_sched_expedited_done taking on the roles of the halves
1992+ * of the ticket-lock word. Each task atomically increments
1993+ * sync_sched_expedited_started upon entry, snapshotting the old value,
1994+ * then attempts to stop all the CPUs. If this succeeds, then each
1995+ * CPU will have executed a context switch, resulting in an RCU-sched
1996+ * grace period. We are then done, so we use atomic_cmpxchg() to
1997+ * update sync_sched_expedited_done to match our snapshot -- but
1998+ * only if someone else has not already advanced past our snapshot.
1999+ *
2000+ * On the other hand, if try_stop_cpus() fails, we check the value
2001+ * of sync_sched_expedited_done. If it has advanced past our
2002+ * initial snapshot, then someone else must have forced a grace period
2003+ * some time after we took our snapshot. In this case, our work is
2004+ * done for us, and we can simply return. Otherwise, we try again,
2005+ * but keep our initial snapshot for purposes of checking for someone
2006+ * doing our work for us.
2007+ *
2008+ * If we fail too many times in a row, we fall back to synchronize_sched().
2009+ */
2010+void synchronize_sched_expedited(void)
2011+{
2012+ int firstsnap, s, snap, trycount = 0;
2013+
2014+ /* Note that atomic_inc_return() implies full memory barrier. */
2015+ firstsnap = snap = atomic_inc_return(&sync_sched_expedited_started);
2016+ get_online_cpus();
2017+ WARN_ON_ONCE(cpu_is_offline(raw_smp_processor_id()));
2018+
2019+ /*
2020+ * Each pass through the following loop attempts to force a
2021+ * context switch on each CPU.
2022+ */
2023+ while (try_stop_cpus(cpu_online_mask,
2024+ synchronize_sched_expedited_cpu_stop,
2025+ NULL) == -EAGAIN) {
2026+ put_online_cpus();
2027+
2028+ /* No joy, try again later. Or just synchronize_sched(). */
2029+ if (trycount++ < 10)
2030+ udelay(trycount * num_online_cpus());
2031+ else {
2032+ synchronize_sched();
2033+ return;
2034+ }
2035+
2036+ /* Check to see if someone else did our work for us. */
2037+ s = atomic_read(&sync_sched_expedited_done);
2038+ if (UINT_CMP_GE((unsigned)s, (unsigned)firstsnap)) {
2039+ smp_mb(); /* ensure test happens before caller kfree */
2040+ return;
2041+ }
2042+
2043+ /*
2044+ * Refetching sync_sched_expedited_started allows later
2045+ * callers to piggyback on our grace period. We subtract
2046+ * 1 to get the same token that the last incrementer got.
2047+ * We retry after they started, so our grace period works
2048+ * for them, and they started after our first try, so their
2049+ * grace period works for us.
2050+ */
2051+ get_online_cpus();
2052+ snap = atomic_read(&sync_sched_expedited_started);
2053+ smp_mb(); /* ensure read is before try_stop_cpus(). */
2054+ }
2055+
2056+ /*
2057+ * Everyone up to our most recent fetch is covered by our grace
2058+ * period. Update the counter, but only if our work is still
2059+ * relevant -- which it won't be if someone who started later
2060+ * than we did beat us to the punch.
2061+ */
2062+ do {
2063+ s = atomic_read(&sync_sched_expedited_done);
2064+ if (UINT_CMP_GE((unsigned)s, (unsigned)snap)) {
2065+ smp_mb(); /* ensure test happens before caller kfree */
2066+ break;
2067+ }
2068+ } while (atomic_cmpxchg(&sync_sched_expedited_done, s, snap) != s);
2069+
2070+ put_online_cpus();
2071+}
2072+EXPORT_SYMBOL_GPL(synchronize_sched_expedited);
2073+
18422074 /*
18432075 * Check to see if there is any immediate RCU-related work to be done
18442076 * by the current CPU, for the specified type of RCU, returning 1 if so.
@@ -1932,7 +2164,7 @@ static int rcu_cpu_has_callbacks(int cpu) · 87 unmodified lines
19322164 /* RCU callbacks either ready or pending? */
19332165 return per_cpu(rcu_sched_data, cpu).nxtlist ||
19342166 per_cpu(rcu_bh_data, cpu).nxtlist ||
1935− rcu_preempt_needs_cpu(cpu);
2167+ rcu_preempt_cpu_has_callbacks(cpu);
19362168 }
19372169
19382170 static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL};
@@ -2027,9 +2259,10 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) · 88 unmodified lines
20272259 rdp->nxtlist = NULL;
20282260 for (i = 0; i < RCU_NEXT_SIZE; i++)
20292261 rdp->nxttail[i] = &rdp->nxtlist;
2262+ rdp->qlen_lazy = 0;
20302263 rdp->qlen = 0;
20312264 rdp->dynticks = &per_cpu(rcu_dynticks, cpu);
2032− WARN_ON_ONCE(rdp->dynticks->dynticks_nesting != DYNTICK_TASK_NESTING);
2265+ WARN_ON_ONCE(rdp->dynticks->dynticks_nesting != DYNTICK_TASK_EXIT_IDLE);
20332266 WARN_ON_ONCE(atomic_read(&rdp->dynticks->dynticks) != 1);
20342267 rdp->cpu = cpu;
20352268 rdp->rsp = rsp;
@@ -2057,7 +2290,7 @@ rcu_init_percpu_data(int cpu, struct rcu_state *rsp, int preemptible) · 21 unmodified lines
20572290 rdp->qlen_last_fqs_check = 0;
20582291 rdp->n_force_qs_snap = rsp->n_force_qs;
20592292 rdp->blimit = blimit;
2060− rdp->dynticks->dynticks_nesting = DYNTICK_TASK_NESTING;
2293+ rdp->dynticks->dynticks_nesting = DYNTICK_TASK_EXIT_IDLE;
20612294 atomic_set(&rdp->dynticks->dynticks,
20622295 (atomic_read(&rdp->dynticks->dynticks) & ~0x1) + 1);
20632296 rcu_prepare_for_idle_init(cpu);
@@ -2139,16 +2372,18 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self, · 75 unmodified lines
21392372 * touch any data without introducing corruption. We send the
21402373 * dying CPU's callbacks to an arbitrarily chosen online CPU.
21412374 */
2142− rcu_send_cbs_to_online(&rcu_bh_state);
2143− rcu_send_cbs_to_online(&rcu_sched_state);
2144− rcu_preempt_send_cbs_to_online();
2375+ rcu_cleanup_dying_cpu(&rcu_bh_state);
2376+ rcu_cleanup_dying_cpu(&rcu_sched_state);
2377+ rcu_preempt_cleanup_dying_cpu();
21452378 rcu_cleanup_after_idle(cpu);
21462379 break;
21472380 case CPU_DEAD:
21482381 case CPU_DEAD_FROZEN:
21492382 case CPU_UP_CANCELED:
21502383 case CPU_UP_CANCELED_FROZEN:
2151− rcu_offline_cpu(cpu);
2384+ rcu_cleanup_dead_cpu(cpu, &rcu_bh_state);
2385+ rcu_cleanup_dead_cpu(cpu, &rcu_sched_state);
2386+ rcu_preempt_cleanup_dead_cpu(cpu);
21522387 break;
21532388 default:
21542389 break;
kernel/rcutree.h+16−11
@@ -239,6 +239,12 @@ struct rcu_data { · 238 unmodified lines
239239 bool preemptible; /* Preemptible RCU? */
240240 struct rcu_node *mynode; /* This CPU's leaf of hierarchy */
241241 unsigned long grpmask; /* Mask to apply to leaf qsmask. */
242+#ifdef CONFIG_RCU_CPU_STALL_INFO
243+ unsigned long ticks_this_gp; /* The number of scheduling-clock */
244+ /* ticks this CPU has handled */
245+ /* during and after the last grace */
246+ /* period it is aware of. */
247+#endif /* #ifdef CONFIG_RCU_CPU_STALL_INFO */
242248
243249 /* 2) batch handling */
244250 /*
@@ -265,7 +271,8 @@ struct rcu_data { · 20 unmodified lines
265271 */
266272 struct rcu_head *nxtlist;
267273 struct rcu_head **nxttail[RCU_NEXT_SIZE];
268− long qlen; /* # of queued callbacks */
274+ long qlen_lazy; /* # of lazy queued callbacks */
275+ long qlen; /* # of queued callbacks, incl lazy */
269276 long qlen_last_fqs_check;
270277 /* qlen at last check for QS forcing */
271278 unsigned long n_cbs_invoked; /* count of RCU cbs invoked. */
@@ -282,7 +289,6 @@ struct rcu_data { · 10 unmodified lines
282289 /* 4) reasons this CPU needed to be kicked by force_quiescent_state */
283290 unsigned long dynticks_fqs; /* Kicked due to dynticks idle. */
284291 unsigned long offline_fqs; /* Kicked due to being offline. */
285− unsigned long resched_ipi; /* Sent a resched IPI. */
286292
287293 /* 5) __rcu_pending() statistics. */
288294 unsigned long n_rcu_pending; /* rcu_pending() calls since boot. */
@@ -313,12 +319,6 @@ struct rcu_data { · 24 unmodified lines
313319 #else
314320 #define RCU_STALL_DELAY_DELTA 0
315321 #endif
316−
317−#define RCU_SECONDS_TILL_STALL_CHECK (CONFIG_RCU_CPU_STALL_TIMEOUT * HZ + \
318− RCU_STALL_DELAY_DELTA)
319− /* for rsp->jiffies_stall */
320−#define RCU_SECONDS_TILL_STALL_RECHECK (3 * RCU_SECONDS_TILL_STALL_CHECK + 30)
321− /* for rsp->jiffies_stall */
322322 #define RCU_STALL_RAT_DELAY 2 /* Allow other CPUs time */
323323 /* to take at least one */
324324 /* scheduling clock irq */
@@ -438,8 +438,8 @@ static void rcu_preempt_check_blocked_tasks(struct rcu_node *rnp); · 113 unmodified lines
438438 static int rcu_preempt_offline_tasks(struct rcu_state *rsp,
439439 struct rcu_node *rnp,
440440 struct rcu_data *rdp);
441−static void rcu_preempt_offline_cpu(int cpu);
442441 #endif /* #ifdef CONFIG_HOTPLUG_CPU */
442+static void rcu_preempt_cleanup_dead_cpu(int cpu);
443443 static void rcu_preempt_check_callbacks(int cpu);
444444 static void rcu_preempt_process_callbacks(void);
445445 void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu));
@@ -448,9 +448,9 @@ static void rcu_report_exp_rnp(struct rcu_state *rsp, struct rcu_node *rnp, · 2 unmodified lines
448448 bool wake);
449449 #endif /* #if defined(CONFIG_HOTPLUG_CPU) || defined(CONFIG_TREE_PREEMPT_RCU) */
450450 static int rcu_preempt_pending(int cpu);
451−static int rcu_preempt_needs_cpu(int cpu);
451+static int rcu_preempt_cpu_has_callbacks(int cpu);
452452 static void __cpuinit rcu_preempt_init_percpu_data(int cpu);
453−static void rcu_preempt_send_cbs_to_online(void);
453+static void rcu_preempt_cleanup_dying_cpu(void);
454454 static void __init __rcu_init_preempt(void);
455455 static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags);
456456 static void rcu_preempt_boost_start_gp(struct rcu_node *rnp);
@@ -471,5 +471,10 @@ static void __cpuinit rcu_prepare_kthreads(int cpu); · 14 unmodified lines
471471 static void rcu_prepare_for_idle_init(int cpu);
472472 static void rcu_cleanup_after_idle(int cpu);
473473 static void rcu_prepare_for_idle(int cpu);
474+static void print_cpu_stall_info_begin(void);
475+static void print_cpu_stall_info(struct rcu_state *rsp, int cpu);
476+static void print_cpu_stall_info_end(void);
477+static void zero_cpu_stall_ticks(struct rcu_data *rdp);
478+static void increment_cpu_stall_ticks(void);
474479
475480 #endif /* #ifndef RCU_TREE_NONCORE */
kernel/rcutree_plugin.h+276−174
@@ -25,7 +25,6 @@ · 24 unmodified lines
2525 */
2626
2727 #include <linux/delay.h>
28−#include <linux/stop_machine.h>
2928
3029 #define RCU_KTHREAD_PRIO 1
3130
@@ -63,7 +62,10 @@ static void __init rcu_bootup_announce_oddness(void) · 31 unmodified lines
6362 printk(KERN_INFO "\tRCU torture testing starts during boot.\n");
6463 #endif
6564 #if defined(CONFIG_TREE_PREEMPT_RCU) && !defined(CONFIG_RCU_CPU_STALL_VERBOSE)
66− printk(KERN_INFO "\tVerbose stalled-CPUs detection is disabled.\n");
65+ printk(KERN_INFO "\tDump stacks of tasks blocking RCU-preempt GP.\n");
66+#endif
67+#if defined(CONFIG_RCU_CPU_STALL_INFO)
68+ printk(KERN_INFO "\tAdditional per-CPU info printed with stalls.\n");
6769 #endif
6870 #if NUM_RCU_LVL_4 != 0
6971 printk(KERN_INFO "\tExperimental four-level hierarchy is enabled.\n");
@@ -490,6 +492,31 @@ static void rcu_print_detail_task_stall(struct rcu_state *rsp) · 420 unmodified lines
490492
491493 #endif /* #else #ifdef CONFIG_RCU_CPU_STALL_VERBOSE */
492494
495+#ifdef CONFIG_RCU_CPU_STALL_INFO
496+
497+static void rcu_print_task_stall_begin(struct rcu_node *rnp)
498+{
499+ printk(KERN_ERR "\tTasks blocked on level-%d rcu_node (CPUs %d-%d):",
500+ rnp->level, rnp->grplo, rnp->grphi);
501+}
502+
503+static void rcu_print_task_stall_end(void)
504+{
505+ printk(KERN_CONT "\n");
506+}
507+
508+#else /* #ifdef CONFIG_RCU_CPU_STALL_INFO */
509+
510+static void rcu_print_task_stall_begin(struct rcu_node *rnp)
511+{
512+}
513+
514+static void rcu_print_task_stall_end(void)
515+{
516+}
517+
518+#endif /* #else #ifdef CONFIG_RCU_CPU_STALL_INFO */
519+
493520 /*
494521 * Scan the current list of tasks blocked within RCU read-side critical
495522 * sections, printing out the tid of each.
@@ -501,12 +528,14 @@ static int rcu_print_task_stall(struct rcu_node *rnp) · 5 unmodified lines
501528
502529 if (!rcu_preempt_blocked_readers_cgp(rnp))
503530 return 0;
531+ rcu_print_task_stall_begin(rnp);
504532 t = list_entry(rnp->gp_tasks,
505533 struct task_struct, rcu_node_entry);
506534 list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
507− printk(" P%d", t->pid);
535+ printk(KERN_CONT " P%d", t->pid);
508536 ndetected++;
509537 }
538+ rcu_print_task_stall_end();
510539 return ndetected;
511540 }
512541
@@ -581,7 +610,7 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp, · 68 unmodified lines
581610 * absolutely necessary, but this is a good performance/complexity
582611 * tradeoff.
583612 */
584− if (rcu_preempt_blocked_readers_cgp(rnp))
613+ if (rcu_preempt_blocked_readers_cgp(rnp) && rnp->qsmask == 0)
585614 retval |= RCU_OFL_TASKS_NORM_GP;
586615 if (rcu_preempted_readers_exp(rnp))
587616 retval |= RCU_OFL_TASKS_EXP_GP;
@@ -618,16 +647,16 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp, · 30 unmodified lines
618647 return retval;
619648 }
620649
650+#endif /* #ifdef CONFIG_HOTPLUG_CPU */
651+
621652 /*
622653 * Do CPU-offline processing for preemptible RCU.
623654 */
624−static void rcu_preempt_offline_cpu(int cpu)
655+static void rcu_preempt_cleanup_dead_cpu(int cpu)
625656 {
626− __rcu_offline_cpu(cpu, &rcu_preempt_state);
657+ rcu_cleanup_dead_cpu(cpu, &rcu_preempt_state);
627658 }
628659
629−#endif /* #ifdef CONFIG_HOTPLUG_CPU */
630−
631660 /*
632661 * Check for a quiescent state from the current CPU. When a task blocks,
633662 * the task is recorded in the corresponding CPU's rcu_node structure,
@@ -671,10 +700,24 @@ static void rcu_preempt_do_callbacks(void) · 37 unmodified lines
671700 */
672701 void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu))
673702 {
674− __call_rcu(head, func, &rcu_preempt_state);
703+ __call_rcu(head, func, &rcu_preempt_state, 0);
675704 }
676705 EXPORT_SYMBOL_GPL(call_rcu);
677706
707+/*
708+ * Queue an RCU callback for lazy invocation after a grace period.
709+ * This will likely be later named something like "call_rcu_lazy()",
710+ * but this change will require some way of tagging the lazy RCU
711+ * callbacks in the list of pending callbacks. Until then, this
712+ * function may only be called from __kfree_rcu().
713+ */
714+void kfree_call_rcu(struct rcu_head *head,
715+ void (*func)(struct rcu_head *rcu))
716+{
717+ __call_rcu(head, func, &rcu_preempt_state, 1);
718+}
719+EXPORT_SYMBOL_GPL(kfree_call_rcu);
720+
678721 /**
679722 * synchronize_rcu - wait until a grace period has elapsed.
680723 *
@@ -688,6 +731,10 @@ EXPORT_SYMBOL_GPL(call_rcu); · 7 unmodified lines
688731 */
689732 void synchronize_rcu(void)
690733 {
734+ rcu_lockdep_assert(!lock_is_held(&rcu_bh_lock_map) &&
735+ !lock_is_held(&rcu_lock_map) &&
736+ !lock_is_held(&rcu_sched_lock_map),
737+ "Illegal synchronize_rcu() in RCU read-side critical section");
691738 if (!rcu_scheduler_active)
692739 return;
693740 wait_rcu_gp(call_rcu);
@@ -788,10 +835,22 @@ sync_rcu_preempt_exp_init(struct rcu_state *rsp, struct rcu_node *rnp) · 94 unmodified lines
788835 rcu_report_exp_rnp(rsp, rnp, false); /* Don't wake self. */
789836 }
790837
791−/*
792− * Wait for an rcu-preempt grace period, but expedite it. The basic idea
793− * is to invoke synchronize_sched_expedited() to push all the tasks to
794− * the ->blkd_tasks lists and wait for this list to drain.
838+/**
839+ * synchronize_rcu_expedited - Brute-force RCU grace period
840+ *
841+ * Wait for an RCU-preempt grace period, but expedite it. The basic
842+ * idea is to invoke synchronize_sched_expedited() to push all the tasks to
843+ * the ->blkd_tasks lists and wait for this list to drain. This consumes
844+ * significant time on all CPUs and is unfriendly to real-time workloads,
845+ * so is thus not recommended for any sort of common-case code.
846+ * In fact, if you are using synchronize_rcu_expedited() in a loop,
847+ * please restructure your code to batch your updates, and then Use a
848+ * single synchronize_rcu() instead.
849+ *
850+ * Note that it is illegal to call this function while holding any lock
851+ * that is acquired by a CPU-hotplug notifier. And yes, it is also illegal
852+ * to call this function from a CPU-hotplug notifier. Failing to observe
853+ * these restriction will result in deadlock.
795854 */
796855 void synchronize_rcu_expedited(void)
797856 {
@@ -869,9 +928,9 @@ static int rcu_preempt_pending(int cpu) · 71 unmodified lines
869928 }
870929
871930 /*
872− * Does preemptible RCU need the CPU to stay out of dynticks mode?
931+ * Does preemptible RCU have callbacks on this CPU?
873932 */
874−static int rcu_preempt_needs_cpu(int cpu)
933+static int rcu_preempt_cpu_has_callbacks(int cpu)
875934 {
876935 return !!per_cpu(rcu_preempt_data, cpu).nxtlist;
877936 }
@@ -894,11 +953,12 @@ static void __cpuinit rcu_preempt_init_percpu_data(int cpu) · 16 unmodified lines
894953 }
895954
896955 /*
897− * Move preemptible RCU's callbacks from dying CPU to other online CPU.
956+ * Move preemptible RCU's callbacks from dying CPU to other online CPU
957+ * and record a quiescent state.
898958 */
899−static void rcu_preempt_send_cbs_to_online(void)
959+static void rcu_preempt_cleanup_dying_cpu(void)
900960 {
901− rcu_send_cbs_to_online(&rcu_preempt_state);
961+ rcu_cleanup_dying_cpu(&rcu_preempt_state);
902962 }
903963
904964 /*
@@ -1034,16 +1094,16 @@ static int rcu_preempt_offline_tasks(struct rcu_state *rsp, · 129 unmodified lines
10341094 return 0;
10351095 }
10361096
1097+#endif /* #ifdef CONFIG_HOTPLUG_CPU */
1098+
10371099 /*
10381100 * Because preemptible RCU does not exist, it never needs CPU-offline
10391101 * processing.
10401102 */
1041−static void rcu_preempt_offline_cpu(int cpu)
1103+static void rcu_preempt_cleanup_dead_cpu(int cpu)
10421104 {
10431105 }
10441106
1045−#endif /* #ifdef CONFIG_HOTPLUG_CPU */
1046−
10471107 /*
10481108 * Because preemptible RCU does not exist, it never has any callbacks
10491109 * to check.
@@ -1060,6 +1120,22 @@ static void rcu_preempt_process_callbacks(void) · 10 unmodified lines
10601120 {
10611121 }
10621122
1123+/*
1124+ * Queue an RCU callback for lazy invocation after a grace period.
1125+ * This will likely be later named something like "call_rcu_lazy()",
1126+ * but this change will require some way of tagging the lazy RCU
1127+ * callbacks in the list of pending callbacks. Until then, this
1128+ * function may only be called from __kfree_rcu().
1129+ *
1130+ * Because there is no preemptible RCU, we use RCU-sched instead.
1131+ */
1132+void kfree_call_rcu(struct rcu_head *head,
1133+ void (*func)(struct rcu_head *rcu))
1134+{
1135+ __call_rcu(head, func, &rcu_sched_state, 1);
1136+}
1137+EXPORT_SYMBOL_GPL(kfree_call_rcu);
1138+
10631139 /*
10641140 * Wait for an rcu-preempt grace period, but make it happen quickly.
10651141 * But because preemptible RCU does not exist, map to rcu-sched.
@@ -1093,9 +1169,9 @@ static int rcu_preempt_pending(int cpu) · 27 unmodified lines
10931169 }
10941170
10951171 /*
1096− * Because preemptible RCU does not exist, it never needs any CPU.
1172+ * Because preemptible RCU does not exist, it never has callbacks
10971173 */
1098−static int rcu_preempt_needs_cpu(int cpu)
1174+static int rcu_preempt_cpu_has_callbacks(int cpu)
10991175 {
11001176 return 0;
11011177 }
@@ -1119,9 +1195,9 @@ static void __cpuinit rcu_preempt_init_percpu_data(int cpu) · 17 unmodified lines
11191195 }
11201196
11211197 /*
1122− * Because there is no preemptible RCU, there are no callbacks to move.
1198+ * Because there is no preemptible RCU, there is no cleanup to do.
11231199 */
1124−static void rcu_preempt_send_cbs_to_online(void)
1200+static void rcu_preempt_cleanup_dying_cpu(void)
11251201 {
11261202 }
11271203
@@ -1823,132 +1899,6 @@ static void __cpuinit rcu_prepare_kthreads(int cpu) · 695 unmodified lines
18231899
18241900 #endif /* #else #ifdef CONFIG_RCU_BOOST */
18251901
1826−#ifndef CONFIG_SMP
1827−
1828−void synchronize_sched_expedited(void)
1829−{
1830− cond_resched();
1831−}
1832−EXPORT_SYMBOL_GPL(synchronize_sched_expedited);
1833−
1834−#else /* #ifndef CONFIG_SMP */
1835−
1836−static atomic_t sync_sched_expedited_started = ATOMIC_INIT(0);
1837−static atomic_t sync_sched_expedited_done = ATOMIC_INIT(0);
1838−
1839−static int synchronize_sched_expedited_cpu_stop(void *data)
1840−{
1841− /*
1842− * There must be a full memory barrier on each affected CPU
1843− * between the time that try_stop_cpus() is called and the
1844− * time that it returns.
1845− *
1846− * In the current initial implementation of cpu_stop, the
1847− * above condition is already met when the control reaches
1848− * this point and the following smp_mb() is not strictly
1849− * necessary. Do smp_mb() anyway for documentation and
1850− * robustness against future implementation changes.
1851− */
1852− smp_mb(); /* See above comment block. */
1853− return 0;
1854−}
1855−
1856−/*
1857− * Wait for an rcu-sched grace period to elapse, but use "big hammer"
1858− * approach to force grace period to end quickly. This consumes
1859− * significant time on all CPUs, and is thus not recommended for
1860− * any sort of common-case code.
1861− *
1862− * Note that it is illegal to call this function while holding any
1863− * lock that is acquired by a CPU-hotplug notifier. Failing to
1864− * observe this restriction will result in deadlock.
1865− *
1866− * This implementation can be thought of as an application of ticket
1867− * locking to RCU, with sync_sched_expedited_started and
1868− * sync_sched_expedited_done taking on the roles of the halves
1869− * of the ticket-lock word. Each task atomically increments
1870− * sync_sched_expedited_started upon entry, snapshotting the old value,
1871− * then attempts to stop all the CPUs. If this succeeds, then each
1872− * CPU will have executed a context switch, resulting in an RCU-sched
1873− * grace period. We are then done, so we use atomic_cmpxchg() to
1874− * update sync_sched_expedited_done to match our snapshot -- but
1875− * only if someone else has not already advanced past our snapshot.
1876− *
1877− * On the other hand, if try_stop_cpus() fails, we check the value
1878− * of sync_sched_expedited_done. If it has advanced past our
1879− * initial snapshot, then someone else must have forced a grace period
1880− * some time after we took our snapshot. In this case, our work is
1881− * done for us, and we can simply return. Otherwise, we try again,
1882− * but keep our initial snapshot for purposes of checking for someone
1883− * doing our work for us.
1884− *
1885− * If we fail too many times in a row, we fall back to synchronize_sched().
1886− */
1887−void synchronize_sched_expedited(void)
1888−{
1889− int firstsnap, s, snap, trycount = 0;
1890−
1891− /* Note that atomic_inc_return() implies full memory barrier. */
1892− firstsnap = snap = atomic_inc_return(&sync_sched_expedited_started);
1893− get_online_cpus();
1894−
1895− /*
1896− * Each pass through the following loop attempts to force a
1897− * context switch on each CPU.
1898− */
1899− while (try_stop_cpus(cpu_online_mask,
1900− synchronize_sched_expedited_cpu_stop,
1901− NULL) == -EAGAIN) {
1902− put_online_cpus();
1903−
1904− /* No joy, try again later. Or just synchronize_sched(). */
1905− if (trycount++ < 10)
1906− udelay(trycount * num_online_cpus());
1907− else {
1908− synchronize_sched();
1909− return;
1910− }
1911−
1912− /* Check to see if someone else did our work for us. */
1913− s = atomic_read(&sync_sched_expedited_done);
1914− if (UINT_CMP_GE((unsigned)s, (unsigned)firstsnap)) {
1915− smp_mb(); /* ensure test happens before caller kfree */
1916− return;
1917− }
1918−
1919− /*
1920− * Refetching sync_sched_expedited_started allows later
1921− * callers to piggyback on our grace period. We subtract
1922− * 1 to get the same token that the last incrementer got.
1923− * We retry after they started, so our grace period works
1924− * for them, and they started after our first try, so their
1925− * grace period works for us.
1926− */
1927− get_online_cpus();
1928− snap = atomic_read(&sync_sched_expedited_started);
1929− smp_mb(); /* ensure read is before try_stop_cpus(). */
1930− }
1931−
1932− /*
1933− * Everyone up to our most recent fetch is covered by our grace
1934− * period. Update the counter, but only if our work is still
1935− * relevant -- which it won't be if someone who started later
1936− * than we did beat us to the punch.
1937− */
1938− do {
1939− s = atomic_read(&sync_sched_expedited_done);
1940− if (UINT_CMP_GE((unsigned)s, (unsigned)snap)) {
1941− smp_mb(); /* ensure test happens before caller kfree */
1942− break;
1943− }
1944− } while (atomic_cmpxchg(&sync_sched_expedited_done, s, snap) != s);
1945−
1946− put_online_cpus();
1947−}
1948−EXPORT_SYMBOL_GPL(synchronize_sched_expedited);
1949−
1950−#endif /* #else #ifndef CONFIG_SMP */
1951−
19521902 #if !defined(CONFIG_RCU_FAST_NO_HZ)
19531903
19541904 /*
@@ -1981,7 +1931,7 @@ static void rcu_cleanup_after_idle(int cpu) · 26 unmodified lines
19811931 }
19821932
19831933 /*
1984− * Do the idle-entry grace-period work, which, because CONFIG_RCU_FAST_NO_HZ=y,
1934+ * Do the idle-entry grace-period work, which, because CONFIG_RCU_FAST_NO_HZ=n,
19851935 * is nothing.
19861936 */
19871937 static void rcu_prepare_for_idle(int cpu)
@@ -2015,6 +1965,9 @@ static void rcu_prepare_for_idle(int cpu) · 27 unmodified lines
20151965 * number, be warned: Setting RCU_IDLE_GP_DELAY too high can hang your
20161966 * system. And if you are -that- concerned about energy efficiency,
20171967 * just power the system down and be done with it!
1968+ * RCU_IDLE_LAZY_GP_DELAY gives the number of jiffies that a CPU is
1969+ * permitted to sleep in dyntick-idle mode with only lazy RCU
1970+ * callbacks pending. Setting this too high can OOM your system.
20181971 *
20191972 * The values below work well in practice. If future workloads require
20201973 * adjustment, they can be converted into kernel config parameters, though
@@ -2023,11 +1976,13 @@ static void rcu_prepare_for_idle(int cpu) · 2 unmodified lines
20231976 #define RCU_IDLE_FLUSHES 5 /* Number of dyntick-idle tries. */
20241977 #define RCU_IDLE_OPT_FLUSHES 3 /* Optional dyntick-idle tries. */
20251978 #define RCU_IDLE_GP_DELAY 6 /* Roughly one grace period. */
1979+#define RCU_IDLE_LAZY_GP_DELAY (6 * HZ) /* Roughly six seconds. */
20261980
20271981 static DEFINE_PER_CPU(int, rcu_dyntick_drain);
20281982 static DEFINE_PER_CPU(unsigned long, rcu_dyntick_holdoff);
20291983 static DEFINE_PER_CPU(struct hrtimer, rcu_idle_gp_timer);
2030−static ktime_t rcu_idle_gp_wait;
1984+static ktime_t rcu_idle_gp_wait; /* If some non-lazy callbacks. */
1985+static ktime_t rcu_idle_lazy_gp_wait; /* If only lazy callbacks. */
20311986
20321987 /*
20331988 * Allow the CPU to enter dyntick-idle mode if either: (1) There are no
@@ -2047,6 +2002,48 @@ int rcu_needs_cpu(int cpu) · 13 unmodified lines
20472002 return per_cpu(rcu_dyntick_holdoff, cpu) == jiffies;
20482003 }
20492004
2005+/*
2006+ * Does the specified flavor of RCU have non-lazy callbacks pending on
2007+ * the specified CPU? Both RCU flavor and CPU are specified by the
2008+ * rcu_data structure.
2009+ */
2010+static bool __rcu_cpu_has_nonlazy_callbacks(struct rcu_data *rdp)
2011+{
2012+ return rdp->qlen != rdp->qlen_lazy;
2013+}
2014+
2015+#ifdef CONFIG_TREE_PREEMPT_RCU
2016+
2017+/*
2018+ * Are there non-lazy RCU-preempt callbacks? (There cannot be if there
2019+ * is no RCU-preempt in the kernel.)
2020+ */
2021+static bool rcu_preempt_cpu_has_nonlazy_callbacks(int cpu)
2022+{
2023+ struct rcu_data *rdp = &per_cpu(rcu_preempt_data, cpu);
2024+
2025+ return __rcu_cpu_has_nonlazy_callbacks(rdp);
2026+}
2027+
2028+#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
2029+
2030+static bool rcu_preempt_cpu_has_nonlazy_callbacks(int cpu)
2031+{
2032+ return 0;
2033+}
2034+
2035+#endif /* else #ifdef CONFIG_TREE_PREEMPT_RCU */
2036+
2037+/*
2038+ * Does any flavor of RCU have non-lazy callbacks on the specified CPU?
2039+ */
2040+static bool rcu_cpu_has_nonlazy_callbacks(int cpu)
2041+{
2042+ return __rcu_cpu_has_nonlazy_callbacks(&per_cpu(rcu_sched_data, cpu)) ||
2043+ __rcu_cpu_has_nonlazy_callbacks(&per_cpu(rcu_bh_data, cpu)) ||
2044+ rcu_preempt_cpu_has_nonlazy_callbacks(cpu);
2045+}
2046+
20502047 /*
20512048 * Timer handler used to force CPU to start pushing its remaining RCU
20522049 * callbacks in the case where it entered dyntick-idle mode with callbacks
@@ -2074,6 +2071,8 @@ static void rcu_prepare_for_idle_init(int cpu) · 21 unmodified lines
20742071 unsigned int upj = jiffies_to_usecs(RCU_IDLE_GP_DELAY);
20752072
20762073 rcu_idle_gp_wait = ns_to_ktime(upj * (u64)1000);
2074+ upj = jiffies_to_usecs(RCU_IDLE_LAZY_GP_DELAY);
2075+ rcu_idle_lazy_gp_wait = ns_to_ktime(upj * (u64)1000);
20772076 firsttime = 0;
20782077 }
20792078 }
@@ -2109,10 +2108,6 @@ static void rcu_cleanup_after_idle(int cpu) · 29 unmodified lines
21092108 */
21102109 static void rcu_prepare_for_idle(int cpu)
21112110 {
2112− unsigned long flags;
2113−
2114− local_irq_save(flags);
2115−
21162111 /*
21172112 * If there are no callbacks on this CPU, enter dyntick-idle mode.
21182113 * Also reset state to avoid prejudicing later attempts.
@@ -2120,7 +2115,6 @@ static void rcu_prepare_for_idle(int cpu) · 1 unmodified line
21202115 if (!rcu_cpu_has_callbacks(cpu)) {
21212116 per_cpu(rcu_dyntick_holdoff, cpu) = jiffies - 1;
21222117 per_cpu(rcu_dyntick_drain, cpu) = 0;
2123− local_irq_restore(flags);
21242118 trace_rcu_prep_idle("No callbacks");
21252119 return;
21262120 }
@@ -2130,7 +2124,6 @@ static void rcu_prepare_for_idle(int cpu) · 3 unmodified lines
21302124 * refrained from disabling the scheduling-clock tick.
21312125 */
21322126 if (per_cpu(rcu_dyntick_holdoff, cpu) == jiffies) {
2133− local_irq_restore(flags);
21342127 trace_rcu_prep_idle("In holdoff");
21352128 return;
21362129 }
@@ -2140,18 +2133,22 @@ static void rcu_prepare_for_idle(int cpu) · 3 unmodified lines
21402133 /* First time through, initialize the counter. */
21412134 per_cpu(rcu_dyntick_drain, cpu) = RCU_IDLE_FLUSHES;
21422135 } else if (per_cpu(rcu_dyntick_drain, cpu) <= RCU_IDLE_OPT_FLUSHES &&
2143− !rcu_pending(cpu)) {
2136+ !rcu_pending(cpu) &&
2137+ !local_softirq_pending()) {
21442138 /* Can we go dyntick-idle despite still having callbacks? */
21452139 trace_rcu_prep_idle("Dyntick with callbacks");
21462140 per_cpu(rcu_dyntick_drain, cpu) = 0;
2147− per_cpu(rcu_dyntick_holdoff, cpu) = jiffies - 1;
2148− hrtimer_start(&per_cpu(rcu_idle_gp_timer, cpu),
2149− rcu_idle_gp_wait, HRTIMER_MODE_REL);
2141+ per_cpu(rcu_dyntick_holdoff, cpu) = jiffies;
2142+ if (rcu_cpu_has_nonlazy_callbacks(cpu))
2143+ hrtimer_start(&per_cpu(rcu_idle_gp_timer, cpu),
2144+ rcu_idle_gp_wait, HRTIMER_MODE_REL);
2145+ else
2146+ hrtimer_start(&per_cpu(rcu_idle_gp_timer, cpu),
2147+ rcu_idle_lazy_gp_wait, HRTIMER_MODE_REL);
21502148 return; /* Nothing more to do immediately. */
21512149 } else if (--per_cpu(rcu_dyntick_drain, cpu) <= 0) {
21522150 /* We have hit the limit, so time to give up. */
21532151 per_cpu(rcu_dyntick_holdoff, cpu) = jiffies;
2154− local_irq_restore(flags);
21552152 trace_rcu_prep_idle("Begin holdoff");
21562153 invoke_rcu_core(); /* Force the CPU out of dyntick-idle. */
21572154 return;
@@ -2163,23 +2160,17 @@ static void rcu_prepare_for_idle(int cpu) · 5 unmodified lines
21632160 */
21642161 #ifdef CONFIG_TREE_PREEMPT_RCU
21652162 if (per_cpu(rcu_preempt_data, cpu).nxtlist) {
2166− local_irq_restore(flags);
21672163 rcu_preempt_qs(cpu);
21682164 force_quiescent_state(&rcu_preempt_state, 0);
2169− local_irq_save(flags);
21702165 }
21712166 #endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
21722167 if (per_cpu(rcu_sched_data, cpu).nxtlist) {
2173− local_irq_restore(flags);
21742168 rcu_sched_qs(cpu);
21752169 force_quiescent_state(&rcu_sched_state, 0);
2176− local_irq_save(flags);
21772170 }
21782171 if (per_cpu(rcu_bh_data, cpu).nxtlist) {
2179− local_irq_restore(flags);
21802172 rcu_bh_qs(cpu);
21812173 force_quiescent_state(&rcu_bh_state, 0);
2182− local_irq_save(flags);
21832174 }
21842175
21852176 /*
@@ -2187,13 +2178,124 @@ static void rcu_prepare_for_idle(int cpu) · 1 unmodified line
21872178 * So try forcing the callbacks through the grace period.
21882179 */
21892180 if (rcu_cpu_has_callbacks(cpu)) {
2190− local_irq_restore(flags);
21912181 trace_rcu_prep_idle("More callbacks");
21922182 invoke_rcu_core();
2193− } else {
2194− local_irq_restore(flags);
2183+ } else
21952184 trace_rcu_prep_idle("Callbacks drained");
2196− }
21972185 }
21982186
21992187 #endif /* #else #if !defined(CONFIG_RCU_FAST_NO_HZ) */
2188+
2189+#ifdef CONFIG_RCU_CPU_STALL_INFO
2190+
2191+#ifdef CONFIG_RCU_FAST_NO_HZ
2192+
2193+static void print_cpu_stall_fast_no_hz(char *cp, int cpu)
2194+{
2195+ struct hrtimer *hrtp = &per_cpu(rcu_idle_gp_timer, cpu);
2196+
2197+ sprintf(cp, "drain=%d %c timer=%lld",
2198+ per_cpu(rcu_dyntick_drain, cpu),
2199+ per_cpu(rcu_dyntick_holdoff, cpu) == jiffies ? 'H' : '.',
2200+ hrtimer_active(hrtp)
2201+ ? ktime_to_us(hrtimer_get_remaining(hrtp))
2202+ : -1);
2203+}
2204+
2205+#else /* #ifdef CONFIG_RCU_FAST_NO_HZ */
2206+
2207+static void print_cpu_stall_fast_no_hz(char *cp, int cpu)
2208+{
2209+}
2210+
2211+#endif /* #else #ifdef CONFIG_RCU_FAST_NO_HZ */
2212+
2213+/* Initiate the stall-info list. */
2214+static void print_cpu_stall_info_begin(void)
2215+{
2216+ printk(KERN_CONT "\n");
2217+}
2218+
2219+/*
2220+ * Print out diagnostic information for the specified stalled CPU.
2221+ *
2222+ * If the specified CPU is aware of the current RCU grace period
2223+ * (flavor specified by rsp), then print the number of scheduling
2224+ * clock interrupts the CPU has taken during the time that it has
2225+ * been aware. Otherwise, print the number of RCU grace periods
2226+ * that this CPU is ignorant of, for example, "1" if the CPU was
2227+ * aware of the previous grace period.
2228+ *
2229+ * Also print out idle and (if CONFIG_RCU_FAST_NO_HZ) idle-entry info.
2230+ */
2231+static void print_cpu_stall_info(struct rcu_state *rsp, int cpu)
2232+{
2233+ char fast_no_hz[72];
2234+ struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
2235+ struct rcu_dynticks *rdtp = rdp->dynticks;
2236+ char *ticks_title;
2237+ unsigned long ticks_value;
2238+
2239+ if (rsp->gpnum == rdp->gpnum) {
2240+ ticks_title = "ticks this GP";
2241+ ticks_value = rdp->ticks_this_gp;
2242+ } else {
2243+ ticks_title = "GPs behind";
2244+ ticks_value = rsp->gpnum - rdp->gpnum;
2245+ }
2246+ print_cpu_stall_fast_no_hz(fast_no_hz, cpu);
2247+ printk(KERN_ERR "\t%d: (%lu %s) idle=%03x/%llx/%d %s\n",
2248+ cpu, ticks_value, ticks_title,
2249+ atomic_read(&rdtp->dynticks) & 0xfff,
2250+ rdtp->dynticks_nesting, rdtp->dynticks_nmi_nesting,
2251+ fast_no_hz);
2252+}
2253+
2254+/* Terminate the stall-info list. */
2255+static void print_cpu_stall_info_end(void)
2256+{
2257+ printk(KERN_ERR "\t");
2258+}
2259+
2260+/* Zero ->ticks_this_gp for all flavors of RCU. */
2261+static void zero_cpu_stall_ticks(struct rcu_data *rdp)
2262+{
2263+ rdp->ticks_this_gp = 0;
2264+}
2265+
2266+/* Increment ->ticks_this_gp for all flavors of RCU. */
2267+static void increment_cpu_stall_ticks(void)
2268+{
2269+ __get_cpu_var(rcu_sched_data).ticks_this_gp++;
2270+ __get_cpu_var(rcu_bh_data).ticks_this_gp++;
2271+#ifdef CONFIG_TREE_PREEMPT_RCU
2272+ __get_cpu_var(rcu_preempt_data).ticks_this_gp++;
2273+#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */
2274+}
2275+
2276+#else /* #ifdef CONFIG_RCU_CPU_STALL_INFO */
2277+
2278+static void print_cpu_stall_info_begin(void)
2279+{
2280+ printk(KERN_CONT " {");
2281+}
2282+
2283+static void print_cpu_stall_info(struct rcu_state *rsp, int cpu)
2284+{
2285+ printk(KERN_CONT " %d", cpu);
2286+}
2287+
2288+static void print_cpu_stall_info_end(void)
2289+{
2290+ printk(KERN_CONT "} ");
2291+}
2292+
2293+static void zero_cpu_stall_ticks(struct rcu_data *rdp)
2294+{
2295+}
2296+
2297+static void increment_cpu_stall_ticks(void)
2298+{
2299+}
2300+
2301+#endif /* #else #ifdef CONFIG_RCU_CPU_STALL_INFO */
kernel/rcutree_trace.c+6−6
@@ -72,9 +72,9 @@ static void print_one_rcu_data(struct seq_file *m, struct rcu_data *rdp) · 71 unmodified lines
7272 rdp->dynticks->dynticks_nesting,
7373 rdp->dynticks->dynticks_nmi_nesting,
7474 rdp->dynticks_fqs);
75− seq_printf(m, " of=%lu ri=%lu", rdp->offline_fqs, rdp->resched_ipi);
76− seq_printf(m, " ql=%ld qs=%c%c%c%c",
77− rdp->qlen,
75+ seq_printf(m, " of=%lu", rdp->offline_fqs);
76+ seq_printf(m, " ql=%ld/%ld qs=%c%c%c%c",
77+ rdp->qlen_lazy, rdp->qlen,
7878 ".N"[rdp->nxttail[RCU_NEXT_READY_TAIL] !=
7979 rdp->nxttail[RCU_NEXT_TAIL]],
8080 ".R"[rdp->nxttail[RCU_WAIT_TAIL] !=
@@ -144,8 +144,8 @@ static void print_one_rcu_data_csv(struct seq_file *m, struct rcu_data *rdp) · 63 unmodified lines
144144 rdp->dynticks->dynticks_nesting,
145145 rdp->dynticks->dynticks_nmi_nesting,
146146 rdp->dynticks_fqs);
147− seq_printf(m, ",%lu,%lu", rdp->offline_fqs, rdp->resched_ipi);
148− seq_printf(m, ",%ld,\"%c%c%c%c\"", rdp->qlen,
147+ seq_printf(m, ",%lu", rdp->offline_fqs);
148+ seq_printf(m, ",%ld,%ld,\"%c%c%c%c\"", rdp->qlen_lazy, rdp->qlen,
149149 ".N"[rdp->nxttail[RCU_NEXT_READY_TAIL] !=
150150 rdp->nxttail[RCU_NEXT_TAIL]],
151151 ".R"[rdp->nxttail[RCU_WAIT_TAIL] !=
@@ -168,7 +168,7 @@ static int show_rcudata_csv(struct seq_file *m, void *unused) · 16 unmodified lines
168168 {
169169 seq_puts(m, "\"CPU\",\"Online?\",\"c\",\"g\",\"pq\",\"pgp\",\"pq\",");
170170 seq_puts(m, "\"dt\",\"dt nesting\",\"dt NMI nesting\",\"df\",");
171− seq_puts(m, "\"of\",\"ri\",\"ql\",\"qs\"");
171+ seq_puts(m, "\"of\",\"qll\",\"ql\",\"qs\"");
172172 #ifdef CONFIG_RCU_BOOST
173173 seq_puts(m, "\"kt\",\"ktl\"");
174174 #endif /* #ifdef CONFIG_RCU_BOOST */
kernel/srcu.c+23−10
@@ -172,6 +172,12 @@ static void __synchronize_srcu(struct srcu_struct *sp, void (*sync_func)(void)) · 171 unmodified lines
172172 {
173173 int idx;
174174
175+ rcu_lockdep_assert(!lock_is_held(&sp->dep_map) &&
176+ !lock_is_held(&rcu_bh_lock_map) &&
177+ !lock_is_held(&rcu_lock_map) &&
178+ !lock_is_held(&rcu_sched_lock_map),
179+ "Illegal synchronize_srcu() in same-type SRCU (or RCU) read-side critical section");
180+
175181 idx = sp->completed;
176182 mutex_lock(&sp->mutex);
177183
@@ -280,19 +286,26 @@ void synchronize_srcu(struct srcu_struct *sp) · 102 unmodified lines
280286 EXPORT_SYMBOL_GPL(synchronize_srcu);
281287
282288 /**
283− * synchronize_srcu_expedited - like synchronize_srcu, but less patient
289+ * synchronize_srcu_expedited - Brute-force SRCU grace period
284290 * @sp: srcu_struct with which to synchronize.
285291 *
286− * Flip the completed counter, and wait for the old count to drain to zero.
287− * As with classic RCU, the updater must use some separate means of
288− * synchronizing concurrent updates. Can block; must be called from
289− * process context.
292+ * Wait for an SRCU grace period to elapse, but use a "big hammer"
293+ * approach to force the grace period to end quickly. This consumes
294+ * significant time on all CPUs and is unfriendly to real-time workloads,
295+ * so is thus not recommended for any sort of common-case code. In fact,
296+ * if you are using synchronize_srcu_expedited() in a loop, please
297+ * restructure your code to batch your updates, and then use a single
298+ * synchronize_srcu() instead.
290299 *
291− * Note that it is illegal to call synchronize_srcu_expedited()
292− * from the corresponding SRCU read-side critical section; doing so
293− * will result in deadlock. However, it is perfectly legal to call
294− * synchronize_srcu_expedited() on one srcu_struct from some other
295− * srcu_struct's read-side critical section.
300+ * Note that it is illegal to call this function while holding any lock
301+ * that is acquired by a CPU-hotplug notifier. And yes, it is also illegal
302+ * to call this function from a CPU-hotplug notifier. Failing to observe
303+ * these restriction will result in deadlock. It is also illegal to call
304+ * synchronize_srcu_expedited() from the corresponding SRCU read-side
305+ * critical section; doing so will result in deadlock. However, it is
306+ * perfectly legal to call synchronize_srcu_expedited() on one srcu_struct
307+ * from some other srcu_struct's read-side critical section, as long as
308+ * the resulting graph of srcu_structs is acyclic.
296309 */
297310 void synchronize_srcu_expedited(struct srcu_struct *sp)
298311 {
lib/Kconfig.debug+24−0
@@ -927,6 +927,30 @@ config RCU_CPU_STALL_VERBOSE · 926 unmodified lines
927927
928928 Say Y if you want to enable such checks.
929929
930+config RCU_CPU_STALL_INFO
931+ bool "Print additional diagnostics on RCU CPU stall"
932+ depends on (TREE_RCU || TREE_PREEMPT_RCU) && DEBUG_KERNEL
933+ default n
934+ help
935+ For each stalled CPU that is aware of the current RCU grace
936+ period, print out additional per-CPU diagnostic information
937+ regarding scheduling-clock ticks, idle state, and,
938+ for RCU_FAST_NO_HZ kernels, idle-entry state.
939+
940+ Say N if you are unsure.
941+
942+ Say Y if you want to enable such diagnostics.
943+
944+config RCU_TRACE
945+ bool "Enable tracing for RCU"
946+ depends on DEBUG_KERNEL
947+ help
948+ This option provides tracing in RCU which presents stats
949+ in debugfs for debugging RCU implementation.
950+
951+ Say Y here if you want to enable RCU tracing
952+ Say N if you are unsure.
953+
930954 config KPROBES_SANITY_TEST
931955 bool "Kprobes sanity tests"
932956 depends on DEBUG_KERNEL
net/ipv4/cipso_ipv4.c+3−8
@@ -1857,11 +1857,6 @@ static int cipso_v4_genopt(unsigned char *buf, u32 buf_len, · 1856 unmodified lines
18571857 return CIPSO_V4_HDR_LEN + ret_val;
18581858 }
18591859
1860−static void opt_kfree_rcu(struct rcu_head *head)
1861−{
1862− kfree(container_of(head, struct ip_options_rcu, rcu));
1863−}
1864−
18651860 /**
18661861 * cipso_v4_sock_setattr - Add a CIPSO option to a socket
18671862 * @sk: the socket
@@ -1938,7 +1933,7 @@ int cipso_v4_sock_setattr(struct sock *sk, · 70 unmodified lines
19381933 }
19391934 rcu_assign_pointer(sk_inet->inet_opt, opt);
19401935 if (old)
1941− call_rcu(&old->rcu, opt_kfree_rcu);
1936+ kfree_rcu(old, rcu);
19421937
19431938 return 0;
19441939
@@ -2005,7 +2000,7 @@ int cipso_v4_req_setattr(struct request_sock *req, · 60 unmodified lines
20052000 req_inet = inet_rsk(req);
20062001 opt = xchg(&req_inet->opt, opt);
20072002 if (opt)
2008− call_rcu(&opt->rcu, opt_kfree_rcu);
2003+ kfree_rcu(opt, rcu);
20092004
20102005 return 0;
20112006
@@ -2075,7 +2070,7 @@ static int cipso_v4_delopt(struct ip_options_rcu **opt_ptr) · 63 unmodified lines
20752070 * remove the entire option struct */
20762071 *opt_ptr = NULL;
20772072 hdr_delta = opt->opt.optlen;
2078− call_rcu(&opt->rcu, opt_kfree_rcu);
2073+ kfree_rcu(opt, rcu);
20792074 }
20802075
20812076 return hdr_delta;
net/ipv4/ip_sockglue.c+1−6
@@ -445,11 +445,6 @@ out: · 444 unmodified lines
445445 }
446446
447447
448−static void opt_kfree_rcu(struct rcu_head *head)
449−{
450− kfree(container_of(head, struct ip_options_rcu, rcu));
451−}
452−
453448 /*
454449 * Socket option code for IP. This is the end of the line after any
455450 * TCP,UDP etc options on an IP socket.
@@ -525,7 +520,7 @@ static int do_ip_setsockopt(struct sock *sk, int level, · 69 unmodified lines
525520 }
526521 rcu_assign_pointer(inet->inet_opt, opt);
527522 if (old)
528− call_rcu(&old->rcu, opt_kfree_rcu);
523+ kfree_rcu(old, rcu);
529524 break;
530525 }
531526 case IP_PKTINFO:
net/mac80211/mesh_pathtbl.c+1−7
@@ -413,12 +413,6 @@ struct mesh_path *mesh_path_lookup_by_idx(int idx, struct ieee80211_sub_if_data · 412 unmodified lines
413413 return NULL;
414414 }
415415
416−static void mesh_gate_node_reclaim(struct rcu_head *rp)
417−{
418− struct mpath_node *node = container_of(rp, struct mpath_node, rcu);
419− kfree(node);
420−}
421−
422416 /**
423417 * mesh_path_add_gate - add the given mpath to a mesh gate to our path table
424418 * @mpath: gate path to add to table
@@ -479,7 +473,7 @@ static int mesh_gate_del(struct mesh_table *tbl, struct mesh_path *mpath) · 54 unmodified lines
479473 if (gate->mpath == mpath) {
480474 spin_lock_bh(&tbl->gates_lock);
481475 hlist_del_rcu(&gate->list);
482− call_rcu(&gate->rcu, mesh_gate_node_reclaim);
476+ kfree_rcu(gate, rcu);
483477 spin_unlock_bh(&tbl->gates_lock);
484478 mpath->sdata->u.mesh.num_gates--;
485479 mpath->is_gate = false;

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t