One counter-intuitive property of RCU is the fact that full memory barriers are needed both before and after updates to the full (non-segmented) length. This patch therefore helps to assist the reader's intuition by adding appropriate comments. [ paulmck: Wordsmithing. ] Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Unified · Split
| @@ -94,17 +94,77 @@ static void rcu_segcblist_set_len(struct rcu_segcblist *rsclp, long v) · 93 unmodified lines | ||
| 94 | 94 | * field to disagree with the actual number of callbacks on the structure. |
| 95 | 95 | * This increase is fully ordered with respect to the callers accesses |
| 96 | 96 | * both before and after. |
| 97 | + * | |
| 98 | + * So why on earth is a memory barrier required both before and after | |
| 99 | + * the update to the ->len field??? | |
| 100 | + * | |
| 101 | + * The reason is that rcu_barrier() locklessly samples each CPU's ->len | |
| 102 | + * field, and if a given CPU's field is zero, avoids IPIing that CPU. | |
| 103 | + * This can of course race with both queuing and invoking of callbacks. | |
| 104 | + * Failing to correctly handle either of these races could result in | |
| 105 | + * rcu_barrier() failing to IPI a CPU that actually had callbacks queued | |
| 106 | + * which rcu_barrier() was obligated to wait on. And if rcu_barrier() | |
| 107 | + * failed to wait on such a callback, unloading certain kernel modules | |
| 108 | + * would result in calls to functions whose code was no longer present in | |
| 109 | + * the kernel, for but one example. | |
| 110 | + * | |
| 111 | + * Therefore, ->len transitions from 1->0 and 0->1 have to be carefully | |
| 112 | + * ordered with respect with both list modifications and the rcu_barrier(). | |
| 113 | + * | |
| 114 | + * The queuing case is CASE 1 and the invoking case is CASE 2. | |
| 115 | + * | |
| 116 | + * CASE 1: Suppose that CPU 0 has no callbacks queued, but invokes | |
| 117 | + * call_rcu() just as CPU 1 invokes rcu_barrier(). CPU 0's ->len field | |
| 118 | + * will transition from 0->1, which is one of the transitions that must | |
| 119 | + * be handled carefully. Without the full memory barriers after the ->len | |
| 120 | + * update and at the beginning of rcu_barrier(), the following could happen: | |
| 121 | + * | |
| 122 | + * CPU 0 CPU 1 | |
| 123 | + * | |
| 124 | + * call_rcu(). | |
| 125 | + * rcu_barrier() sees ->len as 0. | |
| 126 | + * set ->len = 1. | |
| 127 | + * rcu_barrier() does nothing. | |
| 128 | + * module is unloaded. | |
| 129 | + * callback invokes unloaded function! | |
| 130 | + * | |
| 131 | + * With the full barriers, any case where rcu_barrier() sees ->len as 0 will | |
| 132 | + * have unambiguously preceded the return from the racing call_rcu(), which | |
| 133 | + * means that this call_rcu() invocation is OK to not wait on. After all, | |
| 134 | + * you are supposed to make sure that any problematic call_rcu() invocations | |
| 135 | + * happen before the rcu_barrier(). | |
| 136 | + * | |
| 137 | + * | |
| 138 | + * CASE 2: Suppose that CPU 0 is invoking its last callback just as | |
| 139 | + * CPU 1 invokes rcu_barrier(). CPU 0's ->len field will transition from | |
| 140 | + * 1->0, which is one of the transitions that must be handled carefully. | |
| 141 | + * Without the full memory barriers before the ->len update and at the | |
| 142 | + * end of rcu_barrier(), the following could happen: | |
| 143 | + * | |
| 144 | + * CPU 0 CPU 1 | |
| 145 | + * | |
| 146 | + * start invoking last callback | |
| 147 | + * set ->len = 0 (reordered) | |
| 148 | + * rcu_barrier() sees ->len as 0 | |
| 149 | + * rcu_barrier() does nothing. | |
| 150 | + * module is unloaded | |
| 151 | + * callback executing after unloaded! | |
| 152 | + * | |
| 153 | + * With the full barriers, any case where rcu_barrier() sees ->len as 0 | |
| 154 | + * will be fully ordered after the completion of the callback function, | |
| 155 | + * so that the module unloading operation is completely safe. | |
| 156 | + * | |
| 97 | 157 | */ |
| 98 | 158 | void rcu_segcblist_add_len(struct rcu_segcblist *rsclp, long v) |
| 99 | 159 | { |
| 100 | 160 | #ifdef CONFIG_RCU_NOCB_CPU |
| 101 | − smp_mb__before_atomic(); /* Up to the caller! */ | |
| 161 | + smp_mb__before_atomic(); // Read header comment above. | |
| 102 | 162 | atomic_long_add(v, &rsclp->len); |
| 103 | − smp_mb__after_atomic(); /* Up to the caller! */ | |
| 163 | + smp_mb__after_atomic(); // Read header comment above. | |
| 104 | 164 | #else |
| 105 | − smp_mb(); /* Up to the caller! */ | |
| 165 | + smp_mb(); // Read header comment above. | |
| 106 | 166 | WRITE_ONCE(rsclp->len, rsclp->len + v); |
| 107 | − smp_mb(); /* Up to the caller! */ | |
| 167 | + smp_mb(); // Read header comment above. | |
| 108 | 168 | #endif |
| 109 | 169 | } |
| 110 | 170 | |