sched/locking: Add special p->blocked_on==PROXY_WAKING value for proxy return-migration

As we add functionality to proxy execution, we may migrate a donor task to a runqueue where it can't run due to cpu affinity. Thus, we must be careful to ensure we return-migrate the task back to a cpu in its cpumask when it becomes unblocked. Peter helpfully provided the following example with pictures: "Suppose we have a ww_mutex cycle: ,-+-* Mutex-1 <-. Task-A ---' | | ,-- Task-B `-> Mutex-2 *-+-' Where Task-A holds Mutex-1 and tries to acquire Mutex-2, and where Task-B holds Mutex-2 and tries to acquire Mutex-1. Then the blocked_on->owner chain will go in circles. Task-A -> Mutex-2 ^ | | v Mutex-1 <- Task-B We need two things: - find_proxy_task() to stop iterating the circle; - the woken task to 'unblock' and run, such that it can back-off and re-try the transaction. Now, the current code [without this patch] does: __clear_task_blocked_on(); wake_q_add(); And surely clearing ->blocked_on is sufficient to break the cycle. Suppose it is Task-B that is made to back-off, then we have: Task-A -> Mutex-2 -> Task-B (no further blocked_on) and it would attempt to run Task-B. Or worse, it could directly pick Task-B and run it, without ever getting into find_proxy_task(). Now, here is a problem because Task-B might not be runnable on the CPU it is currently on; and because !task_is_blocked() we don't get into the proxy paths, so nobody is going to fix this up. Ideally we would have dequeued Task-B alongside of clearing ->blocked_on, but alas, [the lock ordering prevents us from getting the task_rq_lock() and] spoils things." Thus we need more than just a binary concept of the task being blocked on a mutex or not. So allow setting blocked_on to PROXY_WAKING as a special value which specifies the task is no longer blocked, but needs to be evaluated for return migration *before* it can be run. This will then be used in a later patch to handle proxy return-migration.

John Stultz committed 7 months ago

Commit 2d762261 parent 56f4b244 files · +74 −11Browse files

Showing the changes to kernel (3 of 4 files). Show all

Unified · Split

3 files

kernel/locking/mutex.c +1 −1
kernel/locking/ww_mutex.h +8 −8
kernel/sched/core.c +16 −0

kernel/locking/mutex.c+1−1

@@ -983,7 +983,7 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne · 982 unmodified lines
983	next = waiter->task;	983	next = waiter->task;
984		984
985	debug_mutex_wake_waiter(lock, waiter);	985	debug_mutex_wake_waiter(lock, waiter);
986	clear_task_blocked_on(next, lock);	986	set_task_blocked_on_waking(next, lock);
987	wake_q_add(&wake_q, next);	987	wake_q_add(&wake_q, next);
988	}	988	}
989		989

kernel/locking/ww_mutex.h+8−8

@@ -285,11 +285,11 @@ __ww_mutex_die(struct MUTEX lock, struct MUTEX_WAITER waiter, · 284 unmodified lines
285	debug_mutex_wake_waiter(lock, waiter);	285	debug_mutex_wake_waiter(lock, waiter);
286	#endif	286	#endif
287	/*	287	/*
288	* When waking up the task to die, be sure to clear the	288	* When waking up the task to die, be sure to set the
289	* blocked_on pointer. Otherwise we can see circular	289	* blocked_on to PROXY_WAKING. Otherwise we can see
290	* blocked_on relationships that can't resolve.	290	* circular blocked_on relationships that can't resolve.
291	*/	291	*/
292	clear_task_blocked_on(waiter->task, lock);	292	set_task_blocked_on_waking(waiter->task, lock);
293	wake_q_add(wake_q, waiter->task);	293	wake_q_add(wake_q, waiter->task);
294	}	294	}
295		295
@@ -339,15 +339,15 @@ static bool __ww_mutex_wound(struct MUTEX *lock, · 43 unmodified lines
339	*/	339	*/
340	if (owner != current) {	340	if (owner != current) {
341	/*	341	/*
342	* When waking up the task to wound, be sure to clear the	342	* When waking up the task to wound, be sure to set the
343	* blocked_on pointer. Otherwise we can see circular	343	* blocked_on to PROXY_WAKING. Otherwise we can see
344	* blocked_on relationships that can't resolve.	344	* circular blocked_on relationships that can't resolve.
345	*	345	*
346	* NOTE: We pass NULL here instead of lock, because we	346	* NOTE: We pass NULL here instead of lock, because we
347	* are waking the mutex owner, who may be currently	347	* are waking the mutex owner, who may be currently
348	* blocked on a different mutex.	348	* blocked on a different mutex.
349	*/	349	*/
350	clear_task_blocked_on(owner, NULL);	350	set_task_blocked_on_waking(owner, NULL);
351	wake_q_add(wake_q, owner);	351	wake_q_add(wake_q, owner);
352	}	352	}
353	return true;	353	return true;

kernel/sched/core.c+16−0

 		ttwu_queue(p, cpu, wake_flags);
 	}
 out:
+	/*
+	 * For now, if we've been woken up, clear the task->blocked_on
+	 * regardless if it was set to a mutex or PROXY_WAKING so the
+	 * task can run. We will need to be more careful later when
+	 * properly handling proxy migration
+	 */
+	clear_task_blocked_on(p, NULL);
 	if (success)
 		ttwu_stat(p, task_cpu(p), wake_flags);
 	/* Follow blocked_on chain. */
 	for (p = donor; (mutex = p->blocked_on); p = owner) {
+		/* if its PROXY_WAKING, resched_idle so ttwu can complete */
+		if (mutex == PROXY_WAKING)
+			return proxy_resched_idle(rq);
 		/*
 		 * By taking mutex->wait_lock we hold off concurrent mutex_unlock()
 		 * and ensure @owner sticks around.
 		owner = __mutex_owner(mutex);
 		if (!owner) {
+			/*
+			 * If there is no owner, clear blocked_on
+			 * and return p so it can run and try to
+			 * acquire the lock
+			 */
 			__clear_task_blocked_on(p, mutex);
 			return p;
 		}