From patchwork Thu Jul 11 12:59:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730663 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D30E16D9AD for ; Thu, 11 Jul 2024 13:00:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702858; cv=none; b=lM4iEiT1iWM6aQPX4MDgumvaLSgHrbENpKiFhLFy72AuSbCbkP1MimVtMWux43jgJDAr5TdRHMudZ7LTYXY5sbompV2Ih2jokLtpqiNAxt5aXCHCxDisxeoKTd3FoESEfmJKYUnnnIYVzeCOGuuKItc16XZt+EctIavCP7nEQYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702858; c=relaxed/simple; bh=03eOuIbpk2CL8gHcNkcMr6D2tKWHDvUL5fMCqSTDmZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IN5yNVVTfSUFpewKsk/tkF46m3biF8O6Ckc/2HYs/4AjE7Otu4Z7Riuu3df6Q04mtC6XaWkK1VM8O2ZmpD4p9pUYZZlm/kXZNvrpy4ge7ChEsVyBXN/okXqCeBrMtXgjYqlU3HA0woQCE9hXOkqXIeC8rGTBRbHj586Rk7OrzhY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=fu/7SO3Y; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fu/7SO3Y" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mgv6m6VmpgYAIYyr1Uojk4UY9y2hZWycgx24o6Dhrv4=; b=fu/7SO3YUiqklrnbNmWeKVbC56Wdguoty48y9rOJ+y5jQKzprHRfwwM0eNdLRTQg7UxzjS lkVlc9XjWnlZFk+RhJBFR9RPBfbtx1mHEvXcjrlOtJrYMjA0EtBeM9869CeOEN+GTh6kYe sPj94jYT7YKdw39swgI/yBVaUqkq+gk= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-316-I0jpcKX5O0eRlw_1D8M9ZQ-1; Thu, 11 Jul 2024 09:00:49 -0400 X-MC-Unique: I0jpcKX5O0eRlw_1D8M9ZQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 15AAD19772C9; Thu, 11 Jul 2024 13:00:45 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0DD7319560AA; Thu, 11 Jul 2024 13:00:36 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 01/10] rcuwait: Split type definition to its own header Date: Thu, 11 Jul 2024 14:59:55 +0200 Message-ID: <20240711130004.2157737-2-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 A later commit will add a struct irq_work member to struct task_struct, which would require to include the definition of struct irq_work. Thanks to header dependency hell, incudling in results in defining inline helpers using not-yet-defined fields (mm_struct, task_struct, various task states...). Break off the definition of struct rcuwait into its own header file. Signed-off-by: Valentin Schneider --- include/linux/rcuwait.h | 9 ++------- include/linux/rcuwait_types.h | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 7 deletions(-) create mode 100644 include/linux/rcuwait_types.h diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h index 27343424225cf..1f1ca7d38cdf8 100644 --- a/include/linux/rcuwait.h +++ b/include/linux/rcuwait.h @@ -5,17 +5,12 @@ #include #include +#include + /* * rcuwait provides a way of blocking and waking up a single * task in an rcu-safe manner. - * - * The only time @task is non-nil is when a user is blocked (or - * checking if it needs to) on a condition, and reset as soon as we - * know that the condition has succeeded and are awoken. */ -struct rcuwait { - struct task_struct __rcu *task; -}; #define __RCUWAIT_INITIALIZER(name) \ { .task = NULL, } diff --git a/include/linux/rcuwait_types.h b/include/linux/rcuwait_types.h new file mode 100644 index 0000000000000..60a4385a2c368 --- /dev/null +++ b/include/linux/rcuwait_types.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_RCUWAIT_TYPES_H_ +#define _LINUX_RCUWAIT_TYPES_H_ + +#include + +/* + * The only time @task is non-nil is when a user is blocked (or + * checking if it needs to) on a condition, and reset as soon as we + * know that the condition has succeeded and are awoken. + */ +struct rcuwait { + struct task_struct __rcu *task; +}; + +#endif From patchwork Thu Jul 11 12:59:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730664 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EA5D16D9D4 for ; Thu, 11 Jul 2024 13:01:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702865; cv=none; b=EExlVHr6RvefWjgu0csTBKDKh6lWOJ4toYMSY9Ii8OJtxgAGqaQWWfxKsn5dkKf07p+YceRciwcR2Fg0RTBZS3u/w0UIeEd7sNbQS2Vh5V5CKJ13eS4zMQmubLxkd+4K0BHa/+XHr0rGz/T/sw8UCYU7oicuRCZsdQT65YzW0ww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702865; c=relaxed/simple; bh=j48GshrK4SLyYIYU/4TmYMncSjreC5NozFzjgMwZI9g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qSIcfoyo6NfUsyj8kMYISVHfEkh4M2rlYPqXSrv92zRP1ovDFRjKDE0pT34dj+/tLHhioRETXcM8LQw6JangVAhb5tcKGqnjnkX7gAJmq9igLVgYb4mdr9Wm1Ya9uAGW3LH0kReAZ5xkRpkS0VzlCAE3tWUfNbqo2jli492sev4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=AZsukLIq; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AZsukLIq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+8MT3JoxFoj9B2SXJDisOC/gnRr4MwegwcdGr6Z7BqM=; b=AZsukLIqJX7IIyW+vSE+8TZ2fPhjZhUKBHtplkekN5SVtZLtM5qKesDNgOo1aCzD+kbyTP 4FTDXfycZDbVOPM/+Aj6wrXy6b7FZjJ1ofh8suT65k1a6Evl0a0gTeJJpincj98i5OEnSD Y32SQh3S95mfp0uDWbp072kj2LQ0O6Q= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-490-E1QTop2-MIW3fGQv61nn-A-1; Thu, 11 Jul 2024 09:00:58 -0400 X-MC-Unique: E1QTop2-MIW3fGQv61nn-A-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D42011955EAA; Thu, 11 Jul 2024 13:00:53 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 736A91955E85; Thu, 11 Jul 2024 13:00:45 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 02/10] irq_work: Split type definition to its own header Date: Thu, 11 Jul 2024 14:59:56 +0200 Message-ID: <20240711130004.2157737-3-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 A later commit will add a struct irq_work member to struct task_struct, which would require to include the definition of struct irq_work. Thanks to header dependency hell, incudling in results in defining inline helpers using not-yet-defined fields (mm_struct, task_struct, various task states...). Break off the definition of struct irq_work into its own header file. Signed-off-by: Valentin Schneider --- include/linux/irq_work.h | 8 ++------ include/linux/irq_work_types.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 6 deletions(-) create mode 100644 include/linux/irq_work_types.h diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h index 136f2980cba30..7f6d2af360d91 100644 --- a/include/linux/irq_work.h +++ b/include/linux/irq_work.h @@ -5,6 +5,8 @@ #include #include +#include + /* * An entry can be in one of four states: * @@ -14,12 +16,6 @@ * busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed */ -struct irq_work { - struct __call_single_node node; - void (*func)(struct irq_work *); - struct rcuwait irqwait; -}; - #define __IRQ_WORK_INIT(_func, _flags) (struct irq_work){ \ .node = { .u_flags = (_flags), }, \ .func = (_func), \ diff --git a/include/linux/irq_work_types.h b/include/linux/irq_work_types.h new file mode 100644 index 0000000000000..108cbc514733b --- /dev/null +++ b/include/linux/irq_work_types.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_IRQ_WORK_TYPES_H +#define _LINUX_IRQ_WORK_TYPES_H + +#include +#include + +struct irq_work { + struct __call_single_node node; + void (*func)(struct irq_work *); + struct rcuwait irqwait; +}; + +#endif From patchwork Thu Jul 11 12:59:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730665 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88CAD15FD16 for ; Thu, 11 Jul 2024 13:01:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702875; cv=none; b=nOimIwFyHJ/iog4tuRK/0QBxYxsGdIjAS5/ff9QAbeoZ6MDRSr0ddESBVAbmvi07F5FYr1YDiVThNW0kKkS6Xkp95R79UOQs/M//Jb7UMDSzC9km/36TNjatC1lDIMrEIX9Z/6mSGinlS8J4Za2Vg39m8XfP83EzDf/IY1RmwPE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702875; c=relaxed/simple; bh=tk3/eU+VtYT+/UhY7nwOfNv0uD4Ewrnc70U2kX7qZDc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DHovpTxOy6eW+/V8+JfrDW+WqP500w0JLKPcgCwdeb6SSNj5tfnZbP1tWbkYG/Zh7bsCj/IgKqahA5dvKTKLdE8NtiR28lEpTBjXcjhc50+eh2yqbV7ZjIYh71QYoCarQQrv98X627axlc7nv2Y6Xq6aENI85W8SdWNPg5zYlLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WwbL78iQ; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WwbL78iQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Oe3ZcX+q5foJNAvfJR3Gr4N1bW3UV6I6D4EkwWMO3O0=; b=WwbL78iQOiz883EF+LRWhvHUCh+8F5B5Pt6j+jJMjpPdr7zZYWaljBcWruGzOVuKlUdCbE /PkcumG9v8O0haYewWjRY+HK0MA/boc8itI8G/y8jXWxucbIuJhqyty4WN+ZB4N7EvbqhS BzQlr+5kUTA37kj/ftoO/QmmV/ScAgM= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-324-fdulkfqJNUmCkKIxHRvfTg-1; Thu, 11 Jul 2024 09:01:07 -0400 X-MC-Unique: fdulkfqJNUmCkKIxHRvfTg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 63A4F19560B6; Thu, 11 Jul 2024 13:01:01 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 391341955E8E; Thu, 11 Jul 2024 13:00:54 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 03/10] task_work, sched: Add a _locked variant to task_work_cancel() Date: Thu, 11 Jul 2024 14:59:57 +0200 Message-ID: <20240711130004.2157737-4-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Later commits will need to issue a task_work_cancel() from within the scheduler with the task's ->pi_lock held. Add a _locked variant that expects p->pi_lock to be held. Expose it in a separate scheduler header file, as this really is a scheduler-only interface. Signed-off-by: Valentin Schneider Reviewed-by: Oleg Nesterov --- kernel/sched/task_work_sched.h | 14 +++++++ kernel/task_work.c | 67 ++++++++++++++++++++++++++-------- 2 files changed, 66 insertions(+), 15 deletions(-) create mode 100644 kernel/sched/task_work_sched.h diff --git a/kernel/sched/task_work_sched.h b/kernel/sched/task_work_sched.h new file mode 100644 index 0000000000000..e235da456427f --- /dev/null +++ b/kernel/sched/task_work_sched.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Scheduler internal task_work methods + */ +#ifndef _KERNEL_TASK_WORK_SCHED_H +#define _KERNEL_TASK_WORK_SCHED_H + +#include +#include + +struct callback_head * +task_work_cancel_locked(struct task_struct *task, task_work_func_t func); + +#endif diff --git a/kernel/task_work.c b/kernel/task_work.c index 95a7e1b7f1dab..81092bc2e7371 100644 --- a/kernel/task_work.c +++ b/kernel/task_work.c @@ -3,6 +3,8 @@ #include #include +#include "sched/task_work_sched.h" + static struct callback_head work_exited; /* all we need is ->next == NULL */ /** @@ -74,33 +76,20 @@ int task_work_add(struct task_struct *task, struct callback_head *work, return 0; } -/** - * task_work_cancel_match - cancel a pending work added by task_work_add() - * @task: the task which should execute the work - * @match: match function to call - * @data: data to be passed in to match function - * - * RETURNS: - * The found work or NULL if not found. - */ -struct callback_head * -task_work_cancel_match(struct task_struct *task, +static struct callback_head * +task_work_cancel_match_locked(struct task_struct *task, bool (*match)(struct callback_head *, void *data), void *data) { struct callback_head **pprev = &task->task_works; struct callback_head *work; - unsigned long flags; - if (likely(!task_work_pending(task))) - return NULL; /* * If cmpxchg() fails we continue without updating pprev. * Either we raced with task_work_add() which added the * new entry before this work, we will find it again. Or * we raced with task_work_run(), *pprev == NULL/exited. */ - raw_spin_lock_irqsave(&task->pi_lock, flags); work = READ_ONCE(*pprev); while (work) { if (!match(work, data)) { @@ -109,6 +98,32 @@ task_work_cancel_match(struct task_struct *task, } else if (try_cmpxchg(pprev, &work, work->next)) break; } + + return work; +} + +/** + * task_work_cancel_match - cancel a pending work added by task_work_add() + * @task: the task which should execute the work + * @match: match function to call + * @data: data to be passed in to match function + * + * RETURNS: + * The found work or NULL if not found. + */ +struct callback_head * +task_work_cancel_match(struct task_struct *task, + bool (*match)(struct callback_head *, void *data), + void *data) +{ + unsigned long flags; + struct callback_head *work; + + if (likely(!task_work_pending(task))) + return NULL; + + raw_spin_lock_irqsave(&task->pi_lock, flags); + work = task_work_cancel_match_locked(task, match, data); raw_spin_unlock_irqrestore(&task->pi_lock, flags); return work; @@ -136,6 +151,28 @@ task_work_cancel(struct task_struct *task, task_work_func_t func) return task_work_cancel_match(task, task_work_func_match, func); } +/** + * task_work_cancel - cancel a pending work added by task_work_add() + * @task: the task which should execute the work + * @func: identifies the work to remove + * + * Find the last queued pending work with ->func == @func and remove + * it from queue. + * + * RETURNS: + * The found work or NULL if not found. + */ +struct callback_head * +task_work_cancel_locked(struct task_struct *task, task_work_func_t func) +{ + lockdep_assert_held(&task->pi_lock); + + if (likely(!task_work_pending(task))) + return NULL; + + return task_work_cancel_match_locked(task, task_work_func_match, func); +} + /** * task_work_run - execute the works added by task_work_add() * From patchwork Thu Jul 11 12:59:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730666 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62F7C15FD1B for ; Thu, 11 Jul 2024 13:01:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702886; cv=none; b=gUND7Gal3fYAA6RFZ4/kgYc29HvI+6srGbkG8Iyjrv1BgV9ozZSFmKTl8y3+Jh/K7rCXymsiomQw4RMOwZ8Eaoc6DAaDC/BlTGFWNAVm9ZHHxD/RcX77pfwOH9bC1kVc9NYJSsK7YCGhG3MkgDItgb9Mm+FOrHr7x49/Ha2z13o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702886; c=relaxed/simple; bh=ac39iJ85qdc9Ytj5FEcWzWjFfQVSQdQQK96665YcdAg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pwNshagZ21fptC5InVfcvRTRiM7qj4Lsh0yeNAW33qDDI9ikK6vmAmjT45R0iqkEUBYu2XgSaNjVDOhXJlfma9sngbGzWxqEC5V9bdZqHyD8kVx5h0LJXuLLuuFwBgPDLRJzyDMjlQAy3hfrKO0c4vk7uwmQ4rN1v/BiS3/EU/0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WzFPocmE; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WzFPocmE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702884; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bY5N4wVYoZd5/9PoLPrXsqV7N1C2PIbxzBinwZyAijs=; b=WzFPocmE8qhjbdErHyKz1D7eSvfvbzeWSPokFkVyMY7NjoQfMsqCPdZem3aHeis7HRbwHB Tl9Mi4zMOsl5NP6dUluyab4mNBAnXQEBoShH1eCSFcVmLKOTVjCQjPmw3EENkP4fUpiw9o N7tDSx8q2Rr/hQUlXNqVZ1HG4LSm9l4= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-284-uymGDPAkPJiTvRpsqk9RfA-1; Thu, 11 Jul 2024 09:01:17 -0400 X-MC-Unique: uymGDPAkPJiTvRpsqk9RfA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8E3E41977009; Thu, 11 Jul 2024 13:01:10 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C08621955E85; Thu, 11 Jul 2024 13:01:01 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 04/10] sched/fair: Introduce sched_throttle_work Date: Thu, 11 Jul 2024 14:59:58 +0200 Message-ID: <20240711130004.2157737-5-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. Actual throttling of a task will happen in the return to user path, which will be implemented via a task_work callback. To ease reviewing, the infrastructure and helpers are added first, the actual behaviour will be implemented when switching to per-task throttling. Add a task_work node to struct task_struct, and have it initialised at sched_fork(). Signed-off-by: Valentin Schneider --- include/linux/sched.h | 1 + kernel/sched/core.c | 4 ++++ kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/sched.h | 2 ++ 4 files changed, 19 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 90691d99027e3..a4976eb5065fc 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -812,6 +812,7 @@ struct task_struct { #ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group; + struct callback_head sched_throttle_work; #endif diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6d35c48239be0..b811670d2c362 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4329,6 +4329,10 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->se.cfs_rq = NULL; #endif +#ifdef CONFIG_CFS_BANDWIDTH + init_cfs_throttle_work(p); +#endif + #ifdef CONFIG_SCHEDSTATS /* Even if schedstat is disabled, there should not be garbage */ memset(&p->stats, 0, sizeof(p->stats)); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9057584ec06de..775547cdd3ce0 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5759,6 +5759,18 @@ static int tg_throttle_down(struct task_group *tg, void *data) return 0; } +static void throttle_cfs_rq_work(struct callback_head *work) +{ + +} + +void init_cfs_throttle_work(struct task_struct *p) +{ + /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_work() */ + p->sched_throttle_work.next = &p->sched_throttle_work; + init_task_work(&p->sched_throttle_work, throttle_cfs_rq_work); +} + static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct rq *rq = rq_of(cfs_rq); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4c36cc6803617..943bca8263ffe 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2510,6 +2510,8 @@ extern void init_sched_dl_class(void); extern void init_sched_rt_class(void); extern void init_sched_fair_class(void); +extern void init_cfs_throttle_work(struct task_struct *p); + extern void reweight_task(struct task_struct *p, const struct load_weight *lw); extern void resched_curr(struct rq *rq); From patchwork Thu Jul 11 12:59:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730668 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C95316DC2B for ; Thu, 11 Jul 2024 13:01:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702895; cv=none; b=Fi/z+dY5cw7C2wXmTA+7/yYNkylaizEFlfbTFYEQLFA+NEcveRpJT1jSga4BxoWL+wqLoqxGz9MEFX5ddV0VuO5GqK3H/msuXf/d8FV2Q4Ml1p/oG2shDQfiSHjxQegt2YuPSU1PNjishtv6uAhdPUWsHwVi+7nB53KXBY7BLd0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702895; c=relaxed/simple; bh=8ueQdFLG87p+qfy4yt6xngF+VdvTIn2SFkuPJ3FVakA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vg6xrPCBxbDHoTSn0nok4qKdmeIOpngiaJUPoYHl+/etz/F+v/JPndn9twjhp51spuktLNg4nAEp+xSh5OQRXkvDUMNlwHHEOEn/V3GxisM9EY//SxP+c6IbFm5TrMo4MDYlko/KpH/OVwQx0XPw03LxmY5PIrIoEGQD+il/aaI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=YnS/AEfI; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YnS/AEfI" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702892; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cYNML39erjHIQYIq2MzxI1f2C9g8cKZz+w3h5LIB8k4=; b=YnS/AEfID74W0dSUJ4M4g2Pr7GTdi3RBUQQBV1npdH4oRG6jLAXwIqdjls5FJI9OhqdRU5 QtWSWsqKUs2ZQ4a70b5+bYTRllUfcu6h1Gws1kuKpgS+poTvT78KDZz1UjhOLCntzP/jXW 2BwW5p0catHSlUdljzzIA/kmw0nCHNI= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-692-jRWigaaEPvK4-17APSOBkg-1; Thu, 11 Jul 2024 09:01:29 -0400 X-MC-Unique: jRWigaaEPvK4-17APSOBkg-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 403CE1954B0C; Thu, 11 Jul 2024 13:01:18 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EACDC1955E9F; Thu, 11 Jul 2024 13:01:10 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 05/10] sched/fair: Introduce an irq_work for cancelling throttle task_work Date: Thu, 11 Jul 2024 14:59:59 +0200 Message-ID: <20240711130004.2157737-6-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Moving towards per-task throttling, the soon-to-be task_work used for the actual throttling will need to be cancelled when a task is moving out of a throttled cfs_rq and into a non-throttled cfs_rq (or out of CFS altogether). Such code paths will have at least the rq lock held, sometimes both the rq and the p->pi_lock locks held. Functions such as migrate_task_rq_fair() don't have guarantees as to which of the two is held, as such the cancellation will need to happen in a separate context. It will be punted to irq_work context, the groundwork is added here and the irq_work callback will be implemented when switching to per-task throttling. Signed-off-by: Valentin Schneider --- include/linux/sched.h | 4 ++++ kernel/sched/fair.c | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index a4976eb5065fc..99a1e77d769db 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -46,6 +46,7 @@ #include #include #include +#include /* task_struct member predeclarations (sorted alphabetically): */ struct audit_context; @@ -813,6 +814,9 @@ struct task_struct { #ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group; struct callback_head sched_throttle_work; +#ifdef CONFIG_CFS_BANDWIDTH + struct irq_work unthrottle_irq_work; +#endif #endif diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 775547cdd3ce0..095357bd17f0e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5764,11 +5764,17 @@ static void throttle_cfs_rq_work(struct callback_head *work) } +static void task_throttle_cancel_irq_work_fn(struct irq_work *work) +{ + /* Write me */ +} + void init_cfs_throttle_work(struct task_struct *p) { /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_work() */ p->sched_throttle_work.next = &p->sched_throttle_work; init_task_work(&p->sched_throttle_work, throttle_cfs_rq_work); + p->unthrottle_irq_work = IRQ_WORK_INIT_HARD(task_throttle_cancel_irq_work_fn); } static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) From patchwork Thu Jul 11 13:00:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730667 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E53E16DC1E for ; Thu, 11 Jul 2024 13:01:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702894; cv=none; b=qWR56AlRUdmxieIZzWj9eLthyUOQDBC0FDmk+sIAJ5ZUpFHTwD9jy/RvFfKjOv6kVVIAH5zYyvxvQdJjpaBuAwnN6g1/T0p9rbraDYAlbxc6fGd44zNyem2cTin+plL7MGsMIs+dfBW/aIQ5ce4+d3q9EKV9WchiiIX6crVPrIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702894; c=relaxed/simple; bh=j+9voxVbkXLZex9aF3BIDI+OpqtDuigja9qnrW3CW7w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pEZLBIlSiET6+3HegsuJ/rCtBhZsRU96gryY5S9cZw0x8joGdp67kevejZ0aNOo0gfgCMwlX8uwwpsdTjwYJ8Dd5xlD8NeuHZRZeAOEpwVMlS4ooauQde/7G1xy3TrOkSbWvLl4XoubgbhxPC78ptkv6OJRO9wwkX1NOXsb6QAs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=DDQfY8Nq; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DDQfY8Nq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702891; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=39lyjke0SXVRF7E8CxU/t0CTJvnKYmxII1z60vz/0qI=; b=DDQfY8NquneqW1+NZShZr0kGENS5fR6RWCJ4ZJQxkeWdCYvGwzx1skJ+Oq6ezqKsh+ZIk9 HzFh33xclJUHbhct1Lfvl8vvN/YkyK9MuLso0mQoiiA8FuO0ZHBDx1SJBKEMuaX9dGvSwh p1mFspASlvhLTdX5UxNq88OSzXU6B4Y= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-146-4VTRME0cO9a6HJIqgaBDrA-1; Thu, 11 Jul 2024 09:01:29 -0400 X-MC-Unique: 4VTRME0cO9a6HJIqgaBDrA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CF9721953966; Thu, 11 Jul 2024 13:01:25 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8A1F2195606C; Thu, 11 Jul 2024 13:01:18 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 06/10] sched/fair: Prepare switched_from & switched_to for per-task throttling Date: Thu, 11 Jul 2024 15:00:00 +0200 Message-ID: <20240711130004.2157737-7-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add helpers to switched_from_fair() and switched_to_fair() to cover class changes. If switching from CFS, a task may need to be unthrottled. If switching to CFS, a task may need to be throttled. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 095357bd17f0e..acac0829c71f3 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5694,6 +5694,10 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq) return cfs_bandwidth_used() && cfs_rq->throttle_count; } +static inline bool task_needs_throttling(struct task_struct *p) { return false; } +static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel(struct task_struct *p) { } + /* * Ensure that neither of the group entities corresponding to src_cpu or * dest_cpu are members of a throttled hierarchy when performing group @@ -6622,6 +6626,10 @@ static inline int throttled_lb_pair(struct task_group *tg, return 0; } +static inline bool task_needs_throttling(struct task_struct *p) { return false; } +static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel(struct task_struct *p) { } + #ifdef CONFIG_FAIR_GROUP_SCHED void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b, struct cfs_bandwidth *parent) {} static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq) {} @@ -12847,11 +12855,15 @@ static void attach_task_cfs_rq(struct task_struct *p) static void switched_from_fair(struct rq *rq, struct task_struct *p) { detach_task_cfs_rq(p); + if (cfs_bandwidth_used()) + task_throttle_cancel(p); } static void switched_to_fair(struct rq *rq, struct task_struct *p) { attach_task_cfs_rq(p); + if (cfs_bandwidth_used() && task_needs_throttling(p)) + task_throttle_setup(p); set_task_max_allowed_capacity(p); From patchwork Thu Jul 11 13:00:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730669 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED56D16E864 for ; Thu, 11 Jul 2024 13:01:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702909; cv=none; b=g8ovhRdrRLjtBZaBXb6Rc6KzQMkPBnClPUxoh+PvTi9UiXANevUgUtvratsVUM3iUJP69iuUrV+GK/7hTVQV7Pg3ZXkQkrUuCJfghogY4KcT1YLFwvlDIn1em5ChkRCxBzytUmZpUJ1KaFksKZiif2WLPBK3Q23UIrwPgJIqHjU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702909; c=relaxed/simple; bh=3hQTshq3j/pEAuKohadW5LUjN4RXbzUfNWbS/gaH5rc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZyUCwIvCrkTokphuGro8Ko/YbWtVFSGeZARcW9M6GlSEZQXU5kO6B/DD0ocaDbZlqwTj5AMJBF7BiCNhPlfWXGtHULp8jWtR6Yof6r59nQGU8ycqFROSVHzIVhPZF+rbMc2Ja+5I5wMU7E4YKulcsyqK04vFJ1Vz/RsHqXBcWRo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=TJHOr+p9; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TJHOr+p9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lQ8G2jooOuhKMZEVaAQ+rNv5KFn8ERDELUo5S05OR5Q=; b=TJHOr+p9M2RM+9b2Fu1C7m7km2wQ47GwHAP/A5A9EvJNo4qtYr/yn10DHVOLQ0OnHITNrr kUt4V+SpBRCpJBdqs0FIqhO2qS79xKc/uMfkLSZg/M4Tp1FxyNvByjcV1Yjj8SGPfJIOll yZAKZqXkr8BetQpypp02uJx5/bZISHY= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-22-6BOpIVidMVuPnBl-0JpA5w-1; Thu, 11 Jul 2024 09:01:40 -0400 X-MC-Unique: 6BOpIVidMVuPnBl-0JpA5w-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5E81F1944B3E; Thu, 11 Jul 2024 13:01:33 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 23CF61956066; Thu, 11 Jul 2024 13:01:25 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 07/10] sched/fair: Prepare task_change_group_fair() for per-task throttling Date: Thu, 11 Jul 2024 15:00:01 +0200 Message-ID: <20240711130004.2157737-8-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add helpers to task_change_group_fair() to cover CPU cgroup migration. If changing to a throttled taskgroup/cfs_rq, the task needs to be throttled. Conversely, if the task is already throttled but changing to a taskgroup/cfs_rq that still has some runtime, the task must be unthrottled. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index acac0829c71f3..ec4cf7308a586 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -12935,6 +12935,14 @@ static void task_change_group_fair(struct task_struct *p) #endif set_task_rq(p, task_cpu(p)); attach_task_cfs_rq(p); + + if (!cfs_bandwidth_used()) + return; + + if (task_needs_throttling(p)) + task_throttle_setup(p); + else + task_throttle_cancel(p); } void free_fair_sched_group(struct task_group *tg) From patchwork Thu Jul 11 13:00:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730670 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9120B15AAD3 for ; Thu, 11 Jul 2024 13:01:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702913; cv=none; b=riuJ64m+JnDAp25Db+6i4XUyvrjMg71MAx8nOX4UuPhYkxbs5KDk6zCYFyaI+6pXxrum+NkBfIoXmf3kftzz1nVaK/I75LfrR3LH8JJKYDmL5uFOOccsNSEabA95FOmZ1fXt/BQNmUOIj2QtGqakity7lplGd52V+WF48oOVl1o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702913; c=relaxed/simple; bh=kEbqFbMAMRFPsL2Ni9U3GMlxwu2JbJXIf/1GDFAi7j8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Knk0X1SyjLZ4l9B+p6EuJOA+EhJMoLJ9UfsMJWCArMioySoWpvddkJrNaqluK1ommH7wv6Y9Y9HO/8ym4b16VgtCyWLPnBZfOYVe0CFBjC16vRJSn3rHy+uObB/IgnGXmFiADFbg1UOewAQDgFkK+JU0CczLC6tx1q8Tx1390lE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=g4Tforgx; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="g4Tforgx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702910; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bo6AeZ1jwJY+2L4Z28XS0rJC0wllWNmRErvAYHJbVJg=; b=g4TforgxcNNMG6GREnuqwlaZiGqQ37DLsaXtD8/dFAz5+2MuxhI8i6uscDRYWVl75tZS2F 1+7BvfB8kHc/CcetNcOi8OVOj8CtVTwjCh9FzasTip8l2E0IT9ddM5rn2OeVnk9WxeKJkf WF/j4Wnq1tzP20KsroJBn53nMxgM+ZQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-648-AnhaU64YN2SPdUPPEE27_Q-1; Thu, 11 Jul 2024 09:01:46 -0400 X-MC-Unique: AnhaU64YN2SPdUPPEE27_Q-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2102919236B1; Thu, 11 Jul 2024 13:01:41 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A1D4119560AA; Thu, 11 Jul 2024 13:01:33 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 08/10] sched/fair: Prepare migrate_task_rq_fair() for per-task throttling Date: Thu, 11 Jul 2024 15:00:02 +0200 Message-ID: <20240711130004.2157737-9-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add helpers to migrate_task_rq_fair() to cover CPU migration. Even if the task stays within the same taskgroup, each cfs_rq has its own runtime accounting, thus the task needs to be throttled or unthrottled accordingly. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index ec4cf7308a586..b2242307677ca 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5694,8 +5694,11 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq) return cfs_bandwidth_used() && cfs_rq->throttle_count; } +static inline bool task_has_throttle_work(struct task_struct *p) { return false; } static inline bool task_needs_throttling(struct task_struct *p) { return false; } +static inline bool task_needs_migrate_throttling(struct task_struct *p, unsigned int dst_cpu) { return false; } static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel_migrate(struct task_struct *p, int dst_cpu) { } static inline void task_throttle_cancel(struct task_struct *p) { } /* @@ -6626,8 +6629,11 @@ static inline int throttled_lb_pair(struct task_group *tg, return 0; } +static inline bool task_has_throttle_work(struct task_struct *p) { return false; } static inline bool task_needs_throttling(struct task_struct *p) { return false; } +static inline bool task_needs_migrate_throttling(struct task_struct *p, unsigned int dst_cpu) { return false; } static inline void task_throttle_setup(struct task_struct *p) { } +static inline void task_throttle_cancel_migrate(struct task_struct *p, int dst_cpu) { } static inline void task_throttle_cancel(struct task_struct *p) { } #ifdef CONFIG_FAIR_GROUP_SCHED @@ -8308,6 +8314,24 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu) se->avg.last_update_time = 0; update_scan_period(p, new_cpu); + + if (!cfs_bandwidth_used()) + return; + /* + * When the runtime within a cfs_bandwidth is depleted, all underlying + * cfs_rq's can have (approximately) sched_cfs_bandwidth_slice() runtime + * remaining. + * + * This means all tg->cfs_rq[]'s do not get throttled at the exact same + * time: some may still have a bit of runtime left. Thus, even if the + * task is staying within the same cgroup, and under the same + * cfs_bandwidth, the cfs_rq it migrates to might have a different + * throttle status - resync is needed. + */ + if (task_needs_migrate_throttling(p, new_cpu)) + task_throttle_setup(p); + else if (task_has_throttle_work(p)) + task_throttle_cancel_migrate(p, new_cpu); } static void task_dead_fair(struct task_struct *p) From patchwork Thu Jul 11 13:00:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730671 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02A95169AE3 for ; Thu, 11 Jul 2024 13:02:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702935; cv=none; b=iziyWhmZqBpcTzLoSU8dNLYSZ8IOJ9mAc+PgTvlBrW/p7aLxz+HafC8d35+BFVBLPsDrB9Xtu0G0PcKB1uRmmEM1LGCJlGS6FXQz8Jq9NVMdgS488c3PWJTldfJ+ZjL0YLMDB5fchrJYygmjZFx8PIdz7Tq8X6kc4pLGbLHLvmA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702935; c=relaxed/simple; bh=XQg0xC95fqr2rj2jwofKATx4tzAfNoZL3L3ocZIK2Yc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=shrp0ckiOOyVr/RqxVBsdvDRVtIvvG7uOrUjkeAFeIibgGaTxVKemfHvmrQXWClQNniLZ5PUw6RV17DpmvDSsIAfjQkGTSSEcVG9zEAf8UUwPR2gVvy2oqRJ5Mi5/+HhO9WS1F4uNujzic6nbANVJKp+/2Sv9vRQ/gs/gPW0V0M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=jB/pIpt3; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="jB/pIpt3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BecpANArIMV8DYuzp7MuPwP+wsa2agRYjWQ5hatmwiA=; b=jB/pIpt3uPfMDsIHiRjs9JR2YjZRlzpF/RrA00y2X9fpyxpjylA9ZtiMxiT5H0qlg6jqP3 CS5IDIvHky3EYuXBIq9NhoqKbdckb4fHWU1rD+GWtWG3xqA3We7+dSWfDfvmkEfvCUBGAQ 8LCvFIc7Z1qZNC8OiHvCEMJdkTyssNg= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-573-j3VMWE-ePrmUOpn0P2CXbA-1; Thu, 11 Jul 2024 09:02:09 -0400 X-MC-Unique: j3VMWE-ePrmUOpn0P2CXbA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 43DA11977006; Thu, 11 Jul 2024 13:02:04 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4A6D51955E9F; Thu, 11 Jul 2024 13:01:41 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 09/10] sched/fair: Add a class->task_woken callback in preparation for per-task throttling Date: Thu, 11 Jul 2024 15:00:03 +0200 Message-ID: <20240711130004.2157737-10-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 Later commits will change CFS bandwidth control throttling from a per-cfs_rq basis to a per-task basis. This means special care needs to be taken around any transition a task can have into and out of a cfs_rq. To ease reviewing, the transitions are patched with dummy-helpers that are implemented later on. Add a class->task_woken callback to handle tasks being woken into potentially throttled cfs_rq's. Conversely, a task flagged for throttle-at-kernel-exit may block and need to have its pending throttle removed if runtime was replenished by the time it got woken up. Signed-off-by: Valentin Schneider --- kernel/sched/fair.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index b2242307677ca..0cec3e70f1277 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5776,6 +5776,17 @@ static void task_throttle_cancel_irq_work_fn(struct irq_work *work) /* Write me */ } +static void task_woken_fair(struct rq *rq, struct task_struct *p) +{ + if (!cfs_bandwidth_used()) + return; + + if (task_needs_throttling(p)) + task_throttle_setup(p); + else + task_throttle_cancel(p); +} + void init_cfs_throttle_work(struct task_struct *p) { /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_work() */ @@ -13288,6 +13299,10 @@ DEFINE_SCHED_CLASS(fair) = { .task_change_group = task_change_group_fair, #endif +#ifdef CONFIG_CFS_BANDWIDTH + .task_woken = task_woken_fair, +#endif + #ifdef CONFIG_SCHED_CORE .task_is_throttled = task_is_throttled_fair, #endif From patchwork Thu Jul 11 13:00:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13730672 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59CB1169AE3 for ; Thu, 11 Jul 2024 13:02:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702947; cv=none; b=gmli5KsQeP44e+FBqBT4yw4Go8PgKwK654FnczKisNcIxCIi+vF0QaI6Pi/qjaFMVPkUVKwM5qhDFkS2v05V2CFdEZVbzsRFEFGawOKvI02gu8LPgJjUuWOpZyKqU7m8u+st92bJXRIS8WifAAP+aeIMlMcb+sh1Qcf8QNhsroE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720702947; c=relaxed/simple; bh=ck9m3Um5OTNGL+iPZ3Ph2VLm2iJmwF9OHXTWyjvQvrQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V7Jp/eAc62eH7pF7uxjmkw8bIqCcQHEL9Ntx/hxvjIotp55GEQwzwFUvkaEVXBeZ3BMp6XTvQ5wAxfgoLev7PD+fhGgiVWsCXdtE2qTqt9ugUua8n54lSr8gaPd/qezdH0f/wKvlbICZ/bkc2HFKR8nPu4mYGhaZ0WieEi9VWgY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ZMvTydEA; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZMvTydEA" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720702944; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FzQ5KjMRLj4g81cfSh32RRFGiouIEV/aENXMGz09G0s=; b=ZMvTydEA442F20cTOaHkClaglaQvAS7MlxpU/rg7Y+KoXrDnUjO+9+Ysy5bQaygSm3LqP+ 7V/fQw7QQa2BpEkuOzfjSBXVxjKb83V1oe1N1e9w442O70R7ii8dLzr9QI5QQq+4KjWQuJ ad5yEbVafpA1olFlQdj6TqTECq71lpc= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-600-w0Nh7MaYNbCd0g_UNUP4XA-1; Thu, 11 Jul 2024 09:02:21 -0400 X-MC-Unique: w0Nh7MaYNbCd0g_UNUP4XA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EE4111955EA9; Thu, 11 Jul 2024 13:02:11 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.211]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7194F1955EB3; Thu, 11 Jul 2024 13:02:04 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, rcu@vger.kernel.org Cc: Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Phil Auld , Clark Williams , Tomas Glozar , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Alexander Gordeev , Catalin Marinas , Arnd Bergmann , Guo Ren , Palmer Dabbelt , Andrew Morton , Oleg Nesterov , Jens Axboe Subject: [RFC PATCH v3 10/10] sched/fair: Throttle CFS tasks on return to userspace Date: Thu, 11 Jul 2024 15:00:04 +0200 Message-ID: <20240711130004.2157737-11-vschneid@redhat.com> In-Reply-To: <20240711130004.2157737-1-vschneid@redhat.com> References: <20240711130004.2157737-1-vschneid@redhat.com> Precedence: bulk X-Mailing-List: rcu@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 As reported in [1], CFS bandwidth throttling is a source of headaches in PREEMPT_RT - generally speaking, a throttled CFS task can hold locks that prevent ksoftirqd from running, which prevents replenishing & unthrottling the cfs_rq of said CFS task. Peter mentioned that there have been discussions on changing /when/ the throttling happens: rather than have it be done immediately upon updating the runtime statistics and realizing the cfs_rq has depleted its quota, we wait for the task to be about to return to userspace. This approach also benefits !PREEMPT_RT, as it reduces latency caused by throttled tasks owning contended (kernel) resources. Concept ======= Upon throttling a cfs_rq, all tasks already enqueued get a task_work added, which lets the actual throttling happen in exit_to_user_mode(). Any new task migrated to or enqueued on such a cfs_rq similarly gets the task_work added. Previous patches have added helpers for all the relevant locations where the task_work may need to be either added or removed depending on the state of the cfs_rq the task is (to be) enqueued on: o sched_class change o cgroup migration o CPU migration o task wakeup Upon unthrottling, tasks are enqueued back onto their respective cfs_rq. Unlike the previous throttling implementation, cfs_rq's can be unthrottled while in a half-throttled state (i.e. some tasks have been removed from them, while others are still enqueued and runnable as they haven't reached exit_to_user_mode() yet), so the unthrottling process is a bit more involved, especially when it comes to maintaining *nr_running fields. Clocks ====== Correctly handling the different cfs_rq->throttled_clock* is tricky, as unlike the current upstream approach where all tasks of a cfs_rq are throttled at the exact same time, here they each get throttled at a per-task, not-known-beforehand time. For instance, for the ->throttled_clock_pelt, ideally we would need a per-task snapshot of when the task gets really throttled in exit_to_user_mode(), rather than a single snapshot of when the cfs_rq runs out of runtime. This isn't implemented here. The ->throttled_clock_pelt is set when the cfs_rq runs out of runtime, which means the "grace period" given to the cfs_rq's tasks on their way to exit_to_user_mode() isn't accounted. Notable behaviour changes ========================= Once a cfs_rq is ->throttled, its tasks can continue running until they hit exit_to_user_mode(). This means they can keep draining further runtime from their cfs_rq, which can end up draining more than one period's worth of runtime. I've tested a 10ms runtime / 100ms period cgroup with an always running task: upstream gets a "clean" periodic pattern of 10ms runtime every 100ms, whereas this gets something more like 40ms runtime every 400ms. [1]: https://lore.kernel.org/all/20231031160120.GE15024@noisy.programming.kicks-ass.net/ Suggested-by: Peter Zijlstra Signed-off-by: Valentin Schneider --- include/linux/sched.h | 1 + kernel/sched/fair.c | 438 ++++++++++++++++++++++++++++++------------ kernel/sched/sched.h | 4 + 3 files changed, 320 insertions(+), 123 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 99a1e77d769db..29b9334738af1 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -815,6 +815,7 @@ struct task_struct { struct task_group *sched_task_group; struct callback_head sched_throttle_work; #ifdef CONFIG_CFS_BANDWIDTH + struct list_head throttle_node; struct irq_work unthrottle_irq_work; #endif #endif diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0cec3e70f1277..08cf7343aedb1 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -54,6 +54,7 @@ #include "sched.h" #include "stats.h" #include "autogroup.h" +#include "task_work_sched.h" /* * The initial- and re-scaling of tunables is configurable @@ -5694,12 +5695,114 @@ static inline int throttled_hierarchy(struct cfs_rq *cfs_rq) return cfs_bandwidth_used() && cfs_rq->throttle_count; } -static inline bool task_has_throttle_work(struct task_struct *p) { return false; } -static inline bool task_needs_throttling(struct task_struct *p) { return false; } -static inline bool task_needs_migrate_throttling(struct task_struct *p, unsigned int dst_cpu) { return false; } -static inline void task_throttle_setup(struct task_struct *p) { } -static inline void task_throttle_cancel_migrate(struct task_struct *p, int dst_cpu) { } -static inline void task_throttle_cancel(struct task_struct *p) { } +static inline bool task_has_throttle_work(struct task_struct *p) +{ + return p->sched_throttle_work.next != &p->sched_throttle_work; +} + +static inline bool task_needs_throttling(struct task_struct *p) +{ + return throttled_hierarchy(cfs_rq_of(&p->se)); +} + +static inline bool task_needs_migrate_throttling(struct task_struct *p, unsigned int dst_cpu) +{ + return throttled_hierarchy(task_group(p)->cfs_rq[dst_cpu]); +} + +static inline bool task_is_throttled(struct task_struct *p) +{ + return !list_empty(&p->throttle_node); +} + +static inline void task_throttle_setup_work(struct task_struct *p) +{ + /* + * Kthreads and exiting tasks don't return to userspace, so adding the + * work is pointless + */ + if (!(p->flags & (PF_EXITING | PF_KTHREAD))) + task_work_add(p, &p->sched_throttle_work, TWA_RESUME); +} + +static void throttle_cfs_rq_work(struct callback_head *work); +static inline void task_throttle_do_cancel_work(struct task_struct *p) +{ + /* + * If this returns NULL, it means the work got run, which per + * this being called is a bug: the task_work throttled the + * task when it didn't need to be. + */ + WARN_ON_ONCE(!task_work_cancel_locked(p, throttle_cfs_rq_work)); + p->sched_throttle_work.next = &p->sched_throttle_work; +} + +static inline void task_throttle_cancel_work(struct task_struct *p, int dst_cpu) +{ + /* + * The calling context may be holding p->pi_lock, which is also acquired + * by task_work_cancel_match(). + * + * Lock recursion is prevented by punting the work cancellation to the + * next IRQ enable. This is sent to the destination CPU rather than + * >this< CPU to prevent the task from resuming execution and getting + * throttled in its return to userspace. + */ + irq_work_queue_on(&p->unthrottle_irq_work, dst_cpu); +} + +static void task_throttle_cancel_irq_work_fn(struct irq_work *work) +{ + struct task_struct *p = container_of(work, struct task_struct, unthrottle_irq_work); + int cpu = raw_smp_processor_id(); + + CLASS(task_rq_lock, rq_guard)(p); + WARN_ON_ONCE(task_cpu(p) != cpu); + + if (task_has_throttle_work(p) && !task_needs_throttling(p)) + task_throttle_do_cancel_work(p); +} + +static inline void task_throttle_setup(struct task_struct *p) +{ + /* + * If already throttled-in-userspace, just transfer the throttle_node + * link to the new cfs_rq + * + * Else, if not yet throttled, set up the work. Also, the task may be + * running in userspace (e.g. this is called from sched_move_task()), + * so make sure it is running in kernelspace to get the kernel-exit + * throttle. + */ + if (task_is_throttled(p)) + list_move(&p->throttle_node, &cfs_rq_of(&p->se)->throttled_limbo_list); + else if (!task_has_throttle_work(p)) + task_throttle_setup_work(p); +} + +static void enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags); +static inline void __task_throttle_cancel(struct task_struct *p, unsigned int cpu) +{ + /* + * Task musn't be throttled, either: + * o it's already throttled-in-userspace, unthrottle it + * o it has the task_work installed, remove it + */ + if (task_is_throttled(p)) { + list_del_init(&p->throttle_node); + enqueue_task_fair(cpu_rq(cpu), p, ENQUEUE_WAKEUP); + } else if (task_has_throttle_work(p)) { + task_throttle_cancel_work(p, cpu); + } +} +static inline void task_throttle_cancel(struct task_struct *p) +{ + __task_throttle_cancel(p, task_cpu(p)); +} +static inline void task_throttle_cancel_migrate(struct task_struct *p, unsigned int dst_cpu) +{ + __task_throttle_cancel(p, dst_cpu); +} /* * Ensure that neither of the group entities corresponding to src_cpu or @@ -5722,35 +5825,107 @@ static int tg_unthrottle_up(struct task_group *tg, void *data) { struct rq *rq = data; struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + struct sched_entity *se = tg->se[cpu_of(rq)]; + struct cfs_rq *pcfs_rq = cfs_rq_of(se); + long task_delta = 0, idle_task_delta = 0; + struct task_struct *p, *tmp; cfs_rq->throttle_count--; - if (!cfs_rq->throttle_count) { - cfs_rq->throttled_clock_pelt_time += rq_clock_pelt(rq) - - cfs_rq->throttled_clock_pelt; + if (cfs_rq->throttle_count) + return 0; - /* Add cfs_rq with load or one or more already running entities to the list */ - if (!cfs_rq_is_decayed(cfs_rq)) - list_add_leaf_cfs_rq(cfs_rq); + cfs_rq->throttled_clock_pelt_time += rq_clock_pelt(rq) - + cfs_rq->throttled_clock_pelt; + + /* Add cfs_rq with load or one or more already running entities to the list */ + if (!cfs_rq_is_decayed(cfs_rq)) + list_add_leaf_cfs_rq(cfs_rq); - if (cfs_rq->throttled_clock_self) { - u64 delta = rq_clock(rq) - cfs_rq->throttled_clock_self; + if (cfs_rq->throttled_clock_self) { + u64 delta = rq_clock(rq) - cfs_rq->throttled_clock_self; - cfs_rq->throttled_clock_self = 0; + cfs_rq->throttled_clock_self = 0; - if (SCHED_WARN_ON((s64)delta < 0)) - delta = 0; + if (SCHED_WARN_ON((s64)delta < 0)) + delta = 0; - cfs_rq->throttled_clock_self_time += delta; - } + cfs_rq->throttled_clock_self_time += delta; + } + + /* + * Re-enqueue the tasks that have been throttled at this level. + * + * The task count is up-propagated via ->unthrottled_*h_nr_running, + * as we can't purely rely on h_nr_running post-enqueue: the unthrottle + * might happen when a cfs_rq still has some tasks enqueued, either still + * making their way to userspace, or freshly migrated to it. + */ + list_for_each_entry_safe(p, tmp, &cfs_rq->throttled_limbo_list, throttle_node) { + struct sched_entity *pse = &p->se; + + list_del_init(&p->throttle_node); + + enqueue_entity(cfs_rq, pse, ENQUEUE_WAKEUP); + task_delta++; + idle_task_delta += task_has_idle_policy(p); + } + + /* + * Account tasks woken up in children; by this point all direct children + * have been visited. + */ + task_delta += cfs_rq->unthrottled_h_nr_running; + idle_task_delta += cfs_rq->unthrottled_idle_h_nr_running; + + cfs_rq->h_nr_running += task_delta; + cfs_rq->idle_h_nr_running += idle_task_delta; + + /* + * unthrottle_cfs_rq() needs a value to up-propagate above the + * freshly unthrottled cfs_rq. + */ + cfs_rq->unthrottled_h_nr_running = task_delta; + cfs_rq->unthrottled_idle_h_nr_running = idle_task_delta; + + /* Accumulate the delta in the parent's stash. Once all its children + * (i.e. all of this cfs_rq's siblings) have been visited, this value + * will be stable and used for its own count update. + */ + pcfs_rq->unthrottled_h_nr_running += task_delta; + pcfs_rq->unthrottled_idle_h_nr_running += idle_task_delta; + + /* + * If the cfs_rq became empty during throttling, then we dequeued + * it. It needs to be put back in the hierarchy if it or any of + * its children have now-unthrottled tasks. + */ + if (!se->on_rq && (cfs_rq->h_nr_running || cfs_rq->idle_h_nr_running)) { + enqueue_entity(pcfs_rq, se, ENQUEUE_WAKEUP); + } else { + update_load_avg(pcfs_rq, se, UPDATE_TG); + se_update_runnable(se); } return 0; } +static int tg_unthrottle_clear_up(struct task_group *tg, void *data) +{ + struct rq *rq = data; + struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + + cfs_rq->unthrottled_h_nr_running = 0; + cfs_rq->unthrottled_idle_h_nr_running = 0; + + return 0; +} + static int tg_throttle_down(struct task_group *tg, void *data) { struct rq *rq = data; struct cfs_rq *cfs_rq = tg->cfs_rq[cpu_of(rq)]; + struct rb_node *node; + struct task_struct *p; /* group is entering throttled state, stop time */ if (!cfs_rq->throttle_count) { @@ -5763,17 +5938,118 @@ static int tg_throttle_down(struct task_group *tg, void *data) } cfs_rq->throttle_count++; + /* + * If we've already visited this cfs_rq (e.g. it ran out of its own + * runtime sometime earlier and hasn't had a replenish yet), then + * there's nothing more to do. + */ + if (cfs_rq->throttle_count > 1) + return 0; + + WARN_ON_ONCE(!list_empty(&cfs_rq->throttled_limbo_list)); + /* + * rq_lock is held, current is (obviously) executing this in kernelspace. + * + * All other tasks enqueued on this rq have their saved PC at the + * context switch, so they will go through the kernel before returning + * to userspace. Thus, there are no tasks-in-userspace to handle, just + * install the task_work on all of them. + */ + node = rb_first(&cfs_rq->tasks_timeline.rb_root); + while (node) { + struct sched_entity *se = __node_2_se(node); + + if (!entity_is_task(se)) + goto next; + + p = task_of(se); + + if (!task_has_throttle_work(p)) + task_throttle_setup_work(p); +next: + node = rb_next(node); + } + return 0; } -static void throttle_cfs_rq_work(struct callback_head *work) +static void throttle_one_task(struct cfs_rq *cfs_rq, struct task_struct *p) { + long task_delta, idle_task_delta; + struct sched_entity *se = &p->se; + + list_add(&p->throttle_node, &cfs_rq->throttled_limbo_list); + task_delta = 1; + idle_task_delta = cfs_rq_is_idle(cfs_rq) ? 1 : 0; + + for_each_sched_entity(se) { + cfs_rq = cfs_rq_of(se); + + if (!se->on_rq) + return; + + dequeue_entity(cfs_rq, se, DEQUEUE_SLEEP); + cfs_rq->h_nr_running -= task_delta; + cfs_rq->idle_h_nr_running -= idle_task_delta; + + if (cfs_rq->load.weight) { + /* Avoid re-evaluating load for this entity: */ + se = parent_entity(se); + break; + } + } + + for_each_sched_entity(se) { + cfs_rq = cfs_rq_of(se); + /* throttled entity or throttle-on-deactivate */ + if (!se->on_rq) + goto throttle_done; + + update_load_avg(cfs_rq, se, 0); + se_update_runnable(se); + cfs_rq->h_nr_running -= task_delta; + cfs_rq->h_nr_running -= idle_task_delta; + } + +throttle_done: + /* At this point se is NULL and we are at root level*/ + sub_nr_running(rq_of(cfs_rq), 1); } -static void task_throttle_cancel_irq_work_fn(struct irq_work *work) +static void throttle_cfs_rq_work(struct callback_head *work) { - /* Write me */ + struct task_struct *p = container_of(work, struct task_struct, sched_throttle_work); + struct sched_entity *se; + struct rq *rq; + struct cfs_rq *cfs_rq; + + WARN_ON_ONCE(p != current); + p->sched_throttle_work.next = &p->sched_throttle_work; + /* + * If task is exiting, then there won't be a return to userspace, so we + * don't have to bother with any of this. + */ + if ((p->flags & PF_EXITING)) + return; + + CLASS(task_rq_lock, rq_guard)(p); + rq = rq_guard.rq; + se = &p->se; + cfs_rq = cfs_rq_of(se); + + /* + * If not in limbo, then either replenish has happened or this task got + * migrated out of the throttled cfs_rq, move along + */ + if (!cfs_rq->throttle_count) + return; + + update_rq_clock(rq); + + throttle_one_task(cfs_rq, p); + + resched_curr(rq); } static void task_woken_fair(struct rq *rq, struct task_struct *p) @@ -5792,6 +6068,7 @@ void init_cfs_throttle_work(struct task_struct *p) /* Protect against double add, see throttle_cfs_rq() and throttle_cfs_rq_work() */ p->sched_throttle_work.next = &p->sched_throttle_work; init_task_work(&p->sched_throttle_work, throttle_cfs_rq_work); + INIT_LIST_HEAD(&p->throttle_node); p->unthrottle_irq_work = IRQ_WORK_INIT_HARD(task_throttle_cancel_irq_work_fn); } @@ -5799,8 +6076,7 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct rq *rq = rq_of(cfs_rq); struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); - struct sched_entity *se; - long task_delta, idle_task_delta, dequeue = 1; + long dequeue = 1; raw_spin_lock(&cfs_b->lock); /* This will start the period timer if necessary */ @@ -5818,70 +6094,24 @@ static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) list_add_tail_rcu(&cfs_rq->throttled_list, &cfs_b->throttled_cfs_rq); } + raw_spin_unlock(&cfs_b->lock); if (!dequeue) return false; /* Throttle no longer required. */ - se = cfs_rq->tg->se[cpu_of(rq_of(cfs_rq))]; - - /* freeze hierarchy runnable averages while throttled */ + /* Flag the hierarchy for throttle-at-user-entry */ rcu_read_lock(); walk_tg_tree_from(cfs_rq->tg, tg_throttle_down, tg_nop, (void *)rq); rcu_read_unlock(); - task_delta = cfs_rq->h_nr_running; - idle_task_delta = cfs_rq->idle_h_nr_running; - for_each_sched_entity(se) { - struct cfs_rq *qcfs_rq = cfs_rq_of(se); - /* throttled entity or throttle-on-deactivate */ - if (!se->on_rq) - goto done; - - dequeue_entity(qcfs_rq, se, DEQUEUE_SLEEP); - - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta = cfs_rq->h_nr_running; - - qcfs_rq->h_nr_running -= task_delta; - qcfs_rq->idle_h_nr_running -= idle_task_delta; - - if (qcfs_rq->load.weight) { - /* Avoid re-evaluating load for this entity: */ - se = parent_entity(se); - break; - } - } - - for_each_sched_entity(se) { - struct cfs_rq *qcfs_rq = cfs_rq_of(se); - /* throttled entity or throttle-on-deactivate */ - if (!se->on_rq) - goto done; - - update_load_avg(qcfs_rq, se, 0); - se_update_runnable(se); - - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta = cfs_rq->h_nr_running; - - qcfs_rq->h_nr_running -= task_delta; - qcfs_rq->idle_h_nr_running -= idle_task_delta; - } - - /* At this point se is NULL and we are at root level*/ - sub_nr_running(rq, task_delta); - -done: - /* - * Note: distribution will already see us throttled via the - * throttled-list. rq->lock protects completion. - */ cfs_rq->throttled = 1; + SCHED_WARN_ON(cfs_rq->throttled_clock); if (cfs_rq->nr_running) cfs_rq->throttled_clock = rq_clock(rq); - return true; + + return false; } void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) @@ -5922,25 +6152,17 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) goto unthrottle_throttle; } - task_delta = cfs_rq->h_nr_running; - idle_task_delta = cfs_rq->idle_h_nr_running; - for_each_sched_entity(se) { - struct cfs_rq *qcfs_rq = cfs_rq_of(se); - - if (se->on_rq) - break; - enqueue_entity(qcfs_rq, se, ENQUEUE_WAKEUP); - - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta = cfs_rq->h_nr_running; + if (cfs_rq->throttle_count) + return; - qcfs_rq->h_nr_running += task_delta; - qcfs_rq->idle_h_nr_running += idle_task_delta; + /* + * cfs_rq's below us may not have been fully emptied out, so we can't rely + * directly on ->h_nr_running. Use the stash instead. + */ + task_delta = cfs_rq->unthrottled_h_nr_running; + idle_task_delta = cfs_rq->unthrottled_idle_h_nr_running; - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(qcfs_rq)) - goto unthrottle_throttle; - } + walk_tg_tree_from(cfs_rq->tg, tg_nop, tg_unthrottle_clear_up, (void *)rq); for_each_sched_entity(se) { struct cfs_rq *qcfs_rq = cfs_rq_of(se); @@ -5948,15 +6170,8 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) update_load_avg(qcfs_rq, se, UPDATE_TG); se_update_runnable(se); - if (cfs_rq_is_idle(group_cfs_rq(se))) - idle_task_delta = cfs_rq->h_nr_running; - qcfs_rq->h_nr_running += task_delta; qcfs_rq->idle_h_nr_running += idle_task_delta; - - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(qcfs_rq)) - goto unthrottle_throttle; } /* At this point se is NULL and we are at root level*/ @@ -6455,6 +6670,7 @@ static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq) cfs_rq->runtime_enabled = 0; INIT_LIST_HEAD(&cfs_rq->throttled_list); INIT_LIST_HEAD(&cfs_rq->throttled_csd_list); + INIT_LIST_HEAD(&cfs_rq->throttled_limbo_list); } void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b) @@ -6822,10 +7038,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running = 1; - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto enqueue_throttle; - flags = ENQUEUE_WAKEUP; } @@ -6841,10 +7053,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running = 1; - - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto enqueue_throttle; } /* At this point se is NULL and we are at root level*/ @@ -6867,7 +7075,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) check_update_overutilized_status(rq); -enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); @@ -6900,10 +7107,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running = 1; - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto dequeue_throttle; - /* Don't dequeue parent if it has other entities besides us */ if (cfs_rq->load.weight) { /* Avoid re-evaluating load for this entity: */ @@ -6932,10 +7135,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (cfs_rq_is_idle(cfs_rq)) idle_h_nr_running = 1; - /* end evaluation on encountering a throttled cfs_rq */ - if (cfs_rq_throttled(cfs_rq)) - goto dequeue_throttle; - } /* At this point se is NULL and we are at root level*/ @@ -6945,7 +7144,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (unlikely(!was_sched_idle && sched_idle_rq(rq))) rq->next_balance = jiffies; -dequeue_throttle: util_est_update(&rq->cfs, p, task_sleep); hrtick_update(rq); } @@ -12815,9 +13013,6 @@ static void propagate_entity_cfs_rq(struct sched_entity *se) { struct cfs_rq *cfs_rq = cfs_rq_of(se); - if (cfs_rq_throttled(cfs_rq)) - return; - if (!throttled_hierarchy(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); @@ -12829,9 +13024,6 @@ static void propagate_entity_cfs_rq(struct sched_entity *se) update_load_avg(cfs_rq, se, UPDATE_TG); - if (cfs_rq_throttled(cfs_rq)) - break; - if (!throttled_hierarchy(cfs_rq)) list_add_leaf_cfs_rq(cfs_rq); } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 943bca8263ffe..f4a00b1dd9505 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -690,8 +690,12 @@ struct cfs_rq { u64 throttled_clock_self_time; int throttled; int throttle_count; + /* Temp storage for updating the counts during unthrottling */ + unsigned int unthrottled_h_nr_running; + unsigned int unthrottled_idle_h_nr_running; struct list_head throttled_list; struct list_head throttled_csd_list; + struct list_head throttled_limbo_list; #endif /* CONFIG_CFS_BANDWIDTH */ #endif /* CONFIG_FAIR_GROUP_SCHED */ };