Update the kvfree_call_rcu() function with head-less support. This allows RCU to reclaim objects without an embedded rcu_head. tree-RCU: We introduce two chains of arrays to store SLAB-backed and vmalloc pointers, each. Storage in either of these arrays does not require embedding an rcu_head within the object. Maintaining the arrays may become impossible due to high memory pressure. For such cases there is an emergency path. Objects with rcu_head inside are just queued on a backup rcu_head list. Later on that list is drained. As for the head-less variant, as the current context can sleep, the following emergency measures are applied: a) Synchronously wait until a grace period has elapsed. b) Call kvfree(). tiny-RCU: For double argument calls, there are no new changes in behavior. For single argument call, kvfree() is directly inlined on the current stack after a synchronize_rcu() call. Note that for tiny-RCU, any call to synchronize_rcu() is actually a quiescent state, therefore it does nothing.
Showing the changes to include/linux (1 of 2 files). Show all
Unified · Split
| @@ -34,9 +34,25 @@ static inline void synchronize_rcu_expedited(void) · 33 unmodified lines | ||
| 34 | 34 | synchronize_rcu(); |
| 35 | 35 | } |
| 36 | 36 | |
| 37 | +/* | |
| 38 | + * Add one more declaration of kvfree() here. It is | |
| 39 | + * not so straight forward to just include <linux/mm.h> | |
| 40 | + * where it is defined due to getting many compile | |
| 41 | + * errors caused by that include. | |
| 42 | + */ | |
| 43 | +extern void kvfree(const void *addr); | |
| 44 | + | |
| 37 | 45 | static inline void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) |
| 38 | 46 | { |
| 39 | − call_rcu(head, func); | |
| 47 | + if (head) { | |
| 48 | + call_rcu(head, func); | |
| 49 | + return; | |
| 50 | + } | |
| 51 | + | |
| 52 | + // kvfree_rcu(one_arg) call. | |
| 53 | + might_sleep(); | |
| 54 | + synchronize_rcu(); | |
| 55 | + kvfree((void *) func); | |
| 40 | 56 | } |
| 41 | 57 | |
| 42 | 58 | void rcu_qs(void); |