One counter-intuitive property of RCU is the fact that full memory barriers are needed both before and after updates to the full (non-segmented) length. This patch therefore helps to assist the reader's intuition by adding appropriate comments. [ paulmck: Wordsmithing. ] Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Unified · Split
| @@ -94,17 +94,77 @@ static void rcu_segcblist_set_len(struct rcu_segcblist *rsclp, long v) · 93 unmodified lines | |||
| 94 | * field to disagree with the actual number of callbacks on the structure. | 94 | * field to disagree with the actual number of callbacks on the structure. |
| 95 | * This increase is fully ordered with respect to the callers accesses | 95 | * This increase is fully ordered with respect to the callers accesses |
| 96 | * both before and after. | 96 | * both before and after. |
| 97 | * | ||
| 98 | * So why on earth is a memory barrier required both before and after | ||
| 99 | * the update to the ->len field??? | ||
| 100 | * | ||
| 101 | * The reason is that rcu_barrier() locklessly samples each CPU's ->len | ||
| 102 | * field, and if a given CPU's field is zero, avoids IPIing that CPU. | ||
| 103 | * This can of course race with both queuing and invoking of callbacks. | ||
| 104 | * Failing to correctly handle either of these races could result in | ||
| 105 | * rcu_barrier() failing to IPI a CPU that actually had callbacks queued | ||
| 106 | * which rcu_barrier() was obligated to wait on. And if rcu_barrier() | ||
| 107 | * failed to wait on such a callback, unloading certain kernel modules | ||
| 108 | * would result in calls to functions whose code was no longer present in | ||
| 109 | * the kernel, for but one example. | ||
| 110 | * | ||
| 111 | * Therefore, ->len transitions from 1->0 and 0->1 have to be carefully | ||
| 112 | * ordered with respect with both list modifications and the rcu_barrier(). | ||
| 113 | * | ||
| 114 | * The queuing case is CASE 1 and the invoking case is CASE 2. | ||
| 115 | * | ||
| 116 | * CASE 1: Suppose that CPU 0 has no callbacks queued, but invokes | ||
| 117 | * call_rcu() just as CPU 1 invokes rcu_barrier(). CPU 0's ->len field | ||
| 118 | * will transition from 0->1, which is one of the transitions that must | ||
| 119 | * be handled carefully. Without the full memory barriers after the ->len | ||
| 120 | * update and at the beginning of rcu_barrier(), the following could happen: | ||
| 121 | * | ||
| 122 | * CPU 0 CPU 1 | ||
| 123 | * | ||
| 124 | * call_rcu(). | ||
| 125 | * rcu_barrier() sees ->len as 0. | ||
| 126 | * set ->len = 1. | ||
| 127 | * rcu_barrier() does nothing. | ||
| 128 | * module is unloaded. | ||
| 129 | * callback invokes unloaded function! | ||
| 130 | * | ||
| 131 | * With the full barriers, any case where rcu_barrier() sees ->len as 0 will | ||
| 132 | * have unambiguously preceded the return from the racing call_rcu(), which | ||
| 133 | * means that this call_rcu() invocation is OK to not wait on. After all, | ||
| 134 | * you are supposed to make sure that any problematic call_rcu() invocations | ||
| 135 | * happen before the rcu_barrier(). | ||
| 136 | * | ||
| 137 | * | ||
| 138 | * CASE 2: Suppose that CPU 0 is invoking its last callback just as | ||
| 139 | * CPU 1 invokes rcu_barrier(). CPU 0's ->len field will transition from | ||
| 140 | * 1->0, which is one of the transitions that must be handled carefully. | ||
| 141 | * Without the full memory barriers before the ->len update and at the | ||
| 142 | * end of rcu_barrier(), the following could happen: | ||
| 143 | * | ||
| 144 | * CPU 0 CPU 1 | ||
| 145 | * | ||
| 146 | * start invoking last callback | ||
| 147 | * set ->len = 0 (reordered) | ||
| 148 | * rcu_barrier() sees ->len as 0 | ||
| 149 | * rcu_barrier() does nothing. | ||
| 150 | * module is unloaded | ||
| 151 | * callback executing after unloaded! | ||
| 152 | * | ||
| 153 | * With the full barriers, any case where rcu_barrier() sees ->len as 0 | ||
| 154 | * will be fully ordered after the completion of the callback function, | ||
| 155 | * so that the module unloading operation is completely safe. | ||
| 156 | * | ||
| 97 | */ | 157 | */ |
| 98 | void rcu_segcblist_add_len(struct rcu_segcblist *rsclp, long v) | 158 | void rcu_segcblist_add_len(struct rcu_segcblist *rsclp, long v) |
| 99 | { | 159 | { |
| 100 | #ifdef CONFIG_RCU_NOCB_CPU | 160 | #ifdef CONFIG_RCU_NOCB_CPU |
| 101 | smp_mb__before_atomic(); /* Up to the caller! */ | 161 | smp_mb__before_atomic(); // Read header comment above. |
| 102 | atomic_long_add(v, &rsclp->len); | 162 | atomic_long_add(v, &rsclp->len); |
| 103 | smp_mb__after_atomic(); /* Up to the caller! */ | 163 | smp_mb__after_atomic(); // Read header comment above. |
| 104 | #else | 164 | #else |
| 105 | smp_mb(); /* Up to the caller! */ | 165 | smp_mb(); // Read header comment above. |
| 106 | WRITE_ONCE(rsclp->len, rsclp->len + v); | 166 | WRITE_ONCE(rsclp->len, rsclp->len + v); |
| 107 | smp_mb(); /* Up to the caller! */ | 167 | smp_mb(); // Read header comment above. |
| 108 | #endif | 168 | #endif |
| 109 | } | 169 | } |
| 110 | 170 | ||