History

workqueue: Mark barrier work with WORK_STRUCT_INACTIVE

Currently, WORK_NO_COLOR has two meanings: Not participate in flushing Not participate in nr_active And only non-barrier work items are marked with WORK_STRUCT_INACTIVE when they are in inactive_works list. The barrier work items are not marked INACTIVE even linked in inactive_works list since these tail items are always moved together with the head work item. These definitions are simple, clean and practical. (Except a small blemish that only the first meaning of WORK_NO_COLOR is documented in include/linux/workqueue.h while both meanings are in workqueue.c) But dual-purpose WORK_NO_COLOR used for barrier work items has proven to be problematical[1]. Only the second purpose is obligatory. So we plan to make barrier work items participate in flushing but keep them still not participating in nr_active. So the plan is to mark barrier work items inactive without using WORK_NO_COLOR in this patch so that we can assign a flushing color to them in next patch. The reasonable way is to add or reuse a bit in work data of the work item. But adding a bit will double the size of pool_workqueue. Currently, WORK_STRUCT_INACTIVE is only used in try_to_grab_pending() for user-queued work items and try_to_grab_pending() can't work for barrier work items. So we extend WORK_STRUCT_INACTIVE to also mark barrier work items no matter which list they are in because we don't need to determind which list a barrier work item is in. So the meaning of WORK_STRUCT_INACTIVE becomes just "the work items don't participate in nr_active" (no matter whether it is a barrier work item or a user-queued work item). And WORK_STRUCT_INACTIVE for user-queued work items means they are in inactive_works list. This patch does it by setting WORK_STRUCT_INACTIVE for barrier work items in insert_wq_barrier() and checking WORK_STRUCT_INACTIVE first in pwq_dec_nr_in_flight(). And the meaning of WORK_NO_COLOR is reduced to only "not participating in flushing". There is no functionality change intended in this patch. Because WORK_NO_COLOR+WORK_STRUCT_INACTIVE represents the previous WORK_NO_COLOR in meaning and try_to_grab_pending() doesn't use for barrier work items and avoids being confused by this extended WORK_STRUCT_INACTIVE. A bunch of comment for nr_active & WORK_STRUCT_INACTIVE is also added for documenting how WORK_STRUCT_INACTIVE works in nr_active management. [1]: https://lore.kernel.org/lkml/20210812083814.32453-1-lizhe.67@bytedance.com/ Signed-off-by: Lai Jiangshan <laijs@linux.alibaba.com> Signed-off-by: Tejun Heo <tj@kernel.org>

Lai Jiangshan committed 5 years ago

Commit 018f3a11 parent d21cece1 file · +34 −8Browse files

Showing the changes to kernel/workqueue.c (1 of 1 file). Show all

Unified · Split

kernel/workqueue.c+34−8
@@ -205,6 +205,23 @@ struct pool_workqueue { · 204 unmodified lines
205205 int refcnt; /* L: reference count */
206206 int nr_in_flight[WORK_NR_COLORS];
207207 /* L: nr of in_flight works */
208+
209+ /*
210+ * nr_active management and WORK_STRUCT_INACTIVE:
211+ *
212+ * When pwq->nr_active >= max_active, new work item is queued to
213+ * pwq->inactive_works instead of pool->worklist and marked with
214+ * WORK_STRUCT_INACTIVE.
215+ *
216+ * All work items marked with WORK_STRUCT_INACTIVE do not participate
217+ * in pwq->nr_active and all work items in pwq->inactive_works are
218+ * marked with WORK_STRUCT_INACTIVE. But not all WORK_STRUCT_INACTIVE
219+ * work items are in pwq->inactive_works. Some of them are ready to
220+ * run in pool->worklist or worker->scheduled. Those work itmes are
221+ * only struct wq_barrier which is used for flush_work() and should
222+ * not participate in pwq->nr_active. For non-barrier work item, it
223+ * is marked with WORK_STRUCT_INACTIVE iff it is in pwq->inactive_works.
224+ */
208225 int nr_active; /* L: nr of active works */
209226 int max_active; /* L: max active works */
210227 struct list_head inactive_works; /* L: inactive works */
@@ -1171,19 +1188,21 @@ static void pwq_dec_nr_in_flight(struct pool_workqueue *pwq, unsigned long work_ · 960 unmodified lines
11711188 {
11721189 int color = get_work_color(work_data);
11731190
1174− /* uncolored work items don't participate in flushing or nr_active */
1191+ if (!(work_data & WORK_STRUCT_INACTIVE)) {
1192+ pwq->nr_active--;
1193+ if (!list_empty(&pwq->inactive_works)) {
1194+ /* one down, submit an inactive one */
1195+ if (pwq->nr_active < pwq->max_active)
1196+ pwq_activate_first_inactive(pwq);
1197+ }
1198+ }
1199+
1200+ /* uncolored work items don't participate in flushing */
11751201 if (color == WORK_NO_COLOR)
11761202 goto out_put;
11771203
11781204 pwq->nr_in_flight[color]--;
11791205
1180− pwq->nr_active--;
1181− if (!list_empty(&pwq->inactive_works)) {
1182− /* one down, submit an inactive one */
1183− if (pwq->nr_active < pwq->max_active)
1184− pwq_activate_first_inactive(pwq);
1185− }
1186−
11871206 /* is flush in progress and are we at the flushing tip? */
11881207 if (likely(pwq->flush_color != color))
11891208 goto out_put;
@@ -1283,6 +1302,10 @@ static int try_to_grab_pending(struct work_struct *work, bool is_dwork, · 93 unmodified lines
12831302 debug_work_deactivate(work);
12841303
12851304 /*
1305+ * A cancelable inactive work item must be in the
1306+ * pwq->inactive_works since a queued barrier can't be
1307+ * canceled (see the comments in insert_wq_barrier()).
1308+ *
12861309 * An inactive work item cannot be grabbed directly because
12871310 * it might have linked NO_COLOR work items which, if left
12881311 * on the inactive_works list, will confuse pwq->nr_active
@@ -2675,6 +2698,9 @@ static void insert_wq_barrier(struct pool_workqueue *pwq, · 1386 unmodified lines
26752698
26762699 barr->task = current;
26772700
2701+ /* The barrier work item does not participate in pwq->nr_active. */
2702+ work_flags |= WORK_STRUCT_INACTIVE;
2703+
26782704 /*
26792705 * If @target is currently being executed, schedule the
26802706 * barrier to the worker; otherwise, put it after @target.