From patchwork Thu Jun 3 21:24:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12298091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63CFDC47082 for ; Thu, 3 Jun 2021 21:24:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4D43461404 for ; Thu, 3 Jun 2021 21:24:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230145AbhFCV0m (ORCPT ); Thu, 3 Jun 2021 17:26:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:38571 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230131AbhFCV0m (ORCPT ); Thu, 3 Jun 2021 17:26:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622755497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=r7Nr9zyNvu4ZyjMH6h4XMEahwlMJwXbItTVRKJ/5iiE=; b=FPsIPGmtvPRLMlwzzBbkVeBER19KNoAzD0Dn4YimNw785ik2L38W6MntZa+9xwzWxSRqQ/ NrqR4Zr+vs8otu/bZZx9u6JWs+gOrhWQt7N+02LHhl+JP9sGdVrDrjxy0M1R0/iotHOczP l+WnjueowZqNFLF8lDW2+Y/A/2MWMk8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-454-fWZzddPbPKewv3al71-w-A-1; Thu, 03 Jun 2021 17:24:55 -0400 X-MC-Unique: fWZzddPbPKewv3al71-w-A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4C47B80ED9B; Thu, 3 Jun 2021 21:24:54 +0000 (UTC) Received: from llong.com (ovpn-116-222.rdu2.redhat.com [10.10.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id A935150FA1; Thu, 3 Jun 2021 21:24:50 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Andrew Morton , Roman Gushchin , Phil Auld , Peter Zijlstra , Juri Lelli , Waiman Long Subject: [PATCH 1/5] cgroup/cpuset: Don't call validate_change() for some flag changes Date: Thu, 3 Jun 2021 17:24:12 -0400 Message-Id: <20210603212416.25934-2-longman@redhat.com> In-Reply-To: <20210603212416.25934-1-longman@redhat.com> References: <20210603212416.25934-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org The update_flag() is called with one flag bit change and without change in the various cpumasks in the cpuset. Moreover, not all changes in the flag bits are validated in validate_change(). In particular, the load balance flag and the two spread flags are not checked there. So there is no point in calling validate_change() if those flag bits change. Signed-off-by: Waiman Long --- kernel/cgroup/cpuset.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index adb5190c4429..65ad6995ad77 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -1891,7 +1891,7 @@ static void update_tasks_flags(struct cpuset *cs) * cs: the cpuset to update * turning_on: whether the flag is being set or cleared * - * Call with cpuset_mutex held. + * Call with cpuset_mutex held & cpumasks remain unchanged. */ static int update_flag(cpuset_flagbits_t bit, struct cpuset *cs, @@ -1911,16 +1911,22 @@ static int update_flag(cpuset_flagbits_t bit, struct cpuset *cs, else clear_bit(bit, &trialcs->flags); - err = validate_change(cs, trialcs); - if (err < 0) - goto out; - balance_flag_changed = (is_sched_load_balance(cs) != is_sched_load_balance(trialcs)); spread_flag_changed = ((is_spread_slab(cs) != is_spread_slab(trialcs)) || (is_spread_page(cs) != is_spread_page(trialcs))); + /* + * validate_change() doesn't validate changes in load balance + * and spread flags. + */ + if (!balance_flag_changed && !spread_flag_changed) { + err = validate_change(cs, trialcs); + if (err < 0) + goto out; + } + spin_lock_irq(&callback_lock); cs->flags = trialcs->flags; spin_unlock_irq(&callback_lock); From patchwork Thu Jun 3 21:24:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12298093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77100C47082 for ; Thu, 3 Jun 2021 21:25:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 60D6161406 for ; Thu, 3 Jun 2021 21:25:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230209AbhFCV0p (ORCPT ); Thu, 3 Jun 2021 17:26:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:25795 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230142AbhFCV0o (ORCPT ); Thu, 3 Jun 2021 17:26:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622755499; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=GkmIg8bE7S1WoSNpL21X7dB+1Pyrh8layWzpnydtRXY=; b=EVTqL/9U/2aISjmClatmK6FG73xHRn5GDOzV2JBHl7646jdVXyaaOg4pI6KG2sAv6gNITY QB3kPAxIca9N34tTjhNFdne6nH3F0pLKVva0d5cVwvZwDCOIfpLANinVvqgGng1RpQ5exW t9Z79Tn7e3fM56UUgUO8SkNIx9nLab0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-107-G2u11jBtOtWcrcd370-b2A-1; Thu, 03 Jun 2021 17:24:58 -0400 X-MC-Unique: G2u11jBtOtWcrcd370-b2A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C002110074A7; Thu, 3 Jun 2021 21:24:55 +0000 (UTC) Received: from llong.com (ovpn-116-222.rdu2.redhat.com [10.10.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6E12A50FA1; Thu, 3 Jun 2021 21:24:54 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Andrew Morton , Roman Gushchin , Phil Auld , Peter Zijlstra , Juri Lelli , Waiman Long Subject: [PATCH 2/5] cgroup/cpuset: Add new cpus.partition type with no load balancing Date: Thu, 3 Jun 2021 17:24:13 -0400 Message-Id: <20210603212416.25934-3-longman@redhat.com> In-Reply-To: <20210603212416.25934-1-longman@redhat.com> References: <20210603212416.25934-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Cpuset v1 uses the sched_load_balance control file to determine if load balancing should be enabled. Cpuset v2 gets rid of sched_load_balance as its use may require disabling load balancing at cgroup root. For workloads that require very low latency like DPDK, the latency jitters caused by periodic load balancing may exceed the desired latency limit. When cpuset v2 is in use, the only way to avoid this latency cost is to use the "isolcpus=" kernel boot option to isolate a set of CPUs. After the kernel boot, however, there is no way to add or remove CPUs from this isolated set. For workloads that are more dynamic in nature, that means users have to provision enough CPUs for the worst case situation resulting in excess idle CPUs. To address this issue for cpuset v2, a new cpuset.cpus.partition type "root-nolb" is added which allows the creation of a cpuset partition with no load balancing. This will allow system administrators to dynamically adjust the size of the no load balancing partition to the current need of the workload without rebooting the system. Signed-off-by: Waiman Long --- kernel/cgroup/cpuset.c | 64 ++++++++++++++++++++++++++++++++---------- 1 file changed, 49 insertions(+), 15 deletions(-) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index 65ad6995ad77..78dd6c91dcd6 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -169,6 +169,8 @@ struct cpuset { * * 1 - partition root * + * 2 - partition root (no load balance) + * * -1 - invalid partition root * None of the cpus in cpus_allowed can be put into the parent's * subparts_cpus. In this case, the cpuset is not a real partition @@ -178,6 +180,7 @@ struct cpuset { */ #define PRS_DISABLED 0 #define PRS_ENABLED 1 +#define PRS_ENABLED_NOLB 2 #define PRS_ERROR -1 /* @@ -1246,6 +1249,7 @@ static int update_parent_subparts_cpumask(struct cpuset *cpuset, int cmd, */ switch (cpuset->partition_root_state) { case PRS_ENABLED: + case PRS_ENABLED_NOLB: if (part_error) cpuset->partition_root_state = PRS_ERROR; break; @@ -1943,30 +1947,29 @@ static int update_flag(cpuset_flagbits_t bit, struct cpuset *cs, /* * update_prstate - update partititon_root_state - * cs: the cpuset to update - * val: 0 - disabled, 1 - enabled + * cs: the cpuset to update + * new_prs: new partition root state * * Call with cpuset_mutex held. */ -static int update_prstate(struct cpuset *cs, int val) +static int update_prstate(struct cpuset *cs, int new_prs) { int err; struct cpuset *parent = parent_cs(cs); - struct tmpmasks tmp; + struct tmpmasks tmpmask; + bool sched_domain_rebuilt = false; - if ((val != 0) && (val != 1)) - return -EINVAL; - if (val == cs->partition_root_state) + if (new_prs == cs->partition_root_state) return 0; /* * Cannot force a partial or invalid partition root to a full * partition root. */ - if (val && cs->partition_root_state) + if (new_prs && (cs->partition_root_state < 0)) return -EINVAL; - if (alloc_cpumasks(NULL, &tmp)) + if (alloc_cpumasks(NULL, &tmpmask)) return -ENOMEM; err = -EINVAL; @@ -1984,12 +1987,31 @@ static int update_prstate(struct cpuset *cs, int val) goto out; err = update_parent_subparts_cpumask(cs, partcmd_enable, - NULL, &tmp); + NULL, &tmpmask); + if (err) { update_flag(CS_CPU_EXCLUSIVE, cs, 0); goto out; + } else if (new_prs == PRS_ENABLED_NOLB) { + /* + * Disable the load balance flag should not return an + * error unless the system is running out of memory. + */ + update_flag(CS_SCHED_LOAD_BALANCE, cs, 0); + sched_domain_rebuilt = true; } - cs->partition_root_state = PRS_ENABLED; + + cs->partition_root_state = new_prs; + } else if (cs->partition_root_state && new_prs) { + /* + * A change in load balance state only, no change in cpumasks. + */ + update_flag(CS_SCHED_LOAD_BALANCE, cs, + (new_prs != PRS_ENABLED_NOLB)); + + cs->partition_root_state = new_prs; + err = 0; + goto out; /* Sched domain is rebuilt in update_flag() */ } else { /* * Turning off partition root will clear the @@ -2003,7 +2025,7 @@ static int update_prstate(struct cpuset *cs, int val) } err = update_parent_subparts_cpumask(cs, partcmd_disable, - NULL, &tmp); + NULL, &tmpmask); if (err) goto out; @@ -2011,6 +2033,12 @@ static int update_prstate(struct cpuset *cs, int val) /* Turning off CS_CPU_EXCLUSIVE will not return error */ update_flag(CS_CPU_EXCLUSIVE, cs, 0); + + if (!is_sched_load_balance(cs)) { + /* Make sure load balance is on */ + update_flag(CS_SCHED_LOAD_BALANCE, cs, 1); + sched_domain_rebuilt = true; + } } /* @@ -2021,11 +2049,12 @@ static int update_prstate(struct cpuset *cs, int val) update_tasks_cpumask(parent); if (parent->child_ecpus_count) - update_sibling_cpumasks(parent, cs, &tmp); + update_sibling_cpumasks(parent, cs, &tmpmask); - rebuild_sched_domains_locked(); + if (!sched_domain_rebuilt) + rebuild_sched_domains_locked(); out: - free_cpumasks(NULL, &tmp); + free_cpumasks(NULL, &tmpmask); return err; } @@ -2518,6 +2547,9 @@ static int sched_partition_show(struct seq_file *seq, void *v) case PRS_ENABLED: seq_puts(seq, "root\n"); break; + case PRS_ENABLED_NOLB: + seq_puts(seq, "root-nolb\n"); + break; case PRS_DISABLED: seq_puts(seq, "member\n"); break; @@ -2544,6 +2576,8 @@ static ssize_t sched_partition_write(struct kernfs_open_file *of, char *buf, val = PRS_ENABLED; else if (!strcmp(buf, "member")) val = PRS_DISABLED; + else if (!strcmp(buf, "root-nolb")) + val = PRS_ENABLED_NOLB; else return -EINVAL; From patchwork Thu Jun 3 21:24:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12298097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6654C47096 for ; Thu, 3 Jun 2021 21:25:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D40716140A for ; Thu, 3 Jun 2021 21:25:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230261AbhFCV0u (ORCPT ); Thu, 3 Jun 2021 17:26:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:28083 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230246AbhFCV0r (ORCPT ); Thu, 3 Jun 2021 17:26:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622755501; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=WUGY8sYtZzxYZSnhBIxuLuhJwQQu5J4ornFBXmTKadM=; b=Af/VFunv8bLg8P3LHrJXykFOPRhvACz2cpXYpnCb8Bbdb6UjXXYXR/EmhCnu13K9+vnX10 AH683zInpi9ED/TNvgFB2inovknZvTrSNi9xMRQw0EbGqgZvYdImrLxfGQ/4z7WBltsnNW HE6yFS30M5QFKPcuqZHGbOrjGTDmHyI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-345-xF1QuBkwN4mkdbG0TiNQ1g-1; Thu, 03 Jun 2021 17:24:59 -0400 X-MC-Unique: xF1QuBkwN4mkdbG0TiNQ1g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 44E498030A1; Thu, 3 Jun 2021 21:24:57 +0000 (UTC) Received: from llong.com (ovpn-116-222.rdu2.redhat.com [10.10.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id E26DF614FD; Thu, 3 Jun 2021 21:24:55 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Andrew Morton , Roman Gushchin , Phil Auld , Peter Zijlstra , Juri Lelli , Waiman Long Subject: [PATCH 3/5] cgroup/cpuset: Allow non-top parent partition root to distribute out all CPUs Date: Thu, 3 Jun 2021 17:24:14 -0400 Message-Id: <20210603212416.25934-4-longman@redhat.com> In-Reply-To: <20210603212416.25934-1-longman@redhat.com> References: <20210603212416.25934-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Currently, a parent partition root cannot distribute all its CPUs to child partition roots with no CPUs left. However in some use cases, a management application may want to create a parent partition root as a management unit with no task associated with it and has all its CPUs distributed to various child partition roots dynamically according to their needs. Leaving a cpu in the parent partition root in such a case is now a waste. To accommodate such use cases, a parent partition root can now have all its CPUs distributed to its child partition roots as long as: 1) it is not the top cpuset; and 2) there is no task directly associated with the parent. Once an empty parent partition root is formed, no new task can be moved into it. Signed-off-by: Waiman Long --- kernel/cgroup/cpuset.c | 44 +++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index 78dd6c91dcd6..ef19eb317fef 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -1117,7 +1117,7 @@ enum subparts_cmd { * cpus_allowed can be granted or an error code will be returned. * * For partcmd_disable, the cpuset is being transofrmed from a partition - * root back to a non-partition root. any CPUs in cpus_allowed that are in + * root back to a non-partition root. Any CPUs in cpus_allowed that are in * parent's subparts_cpus will be taken away from that cpumask and put back * into parent's effective_cpus. 0 should always be returned. * @@ -1172,21 +1172,31 @@ static int update_parent_subparts_cpumask(struct cpuset *cpuset, int cmd, if ((cmd != partcmd_update) && css_has_online_children(&cpuset->css)) return -EBUSY; - /* - * Enabling partition root is not allowed if not all the CPUs - * can be granted from parent's effective_cpus or at least one - * CPU will be left after that. - */ - if ((cmd == partcmd_enable) && - (!cpumask_subset(cpuset->cpus_allowed, parent->effective_cpus) || - cpumask_equal(cpuset->cpus_allowed, parent->effective_cpus))) - return -EINVAL; - /* * A cpumask update cannot make parent's effective_cpus become empty. */ adding = deleting = false; if (cmd == partcmd_enable) { + bool parent_is_top_cpuset = !parent_cs(parent); + bool no_cpu_in_parent = cpumask_equal(cpuset->cpus_allowed, + parent->effective_cpus); + /* + * Enabling partition root is not allowed if not all the CPUs + * can be granted from parent's effective_cpus. If the parent + * is the top cpuset, at least one CPU must be left after that. + */ + if (!cpumask_subset(cpuset->cpus_allowed, parent->effective_cpus) || + (parent_is_top_cpuset && no_cpu_in_parent)) + return -EINVAL; + + /* + * A non-top parent can be left with no CPU as long as there + * is no task directly associated with the parent. For such + * a parent, no new task can be moved into it. + */ + if (no_cpu_in_parent && parent->css.cgroup->nr_populated_csets) + return -EINVAL; + cpumask_copy(tmp->addmask, cpuset->cpus_allowed); adding = true; } else if (cmd == partcmd_disable) { @@ -1208,9 +1218,10 @@ static int update_parent_subparts_cpumask(struct cpuset *cpuset, int cmd, adding = cpumask_andnot(tmp->addmask, tmp->addmask, parent->subparts_cpus); /* - * Return error if the new effective_cpus could become empty. + * Return error if the new effective_cpus could become empty + * and there are tasks in the parent. */ - if (adding && + if (adding && parent->css.cgroup->nr_populated_csets && cpumask_equal(parent->effective_cpus, tmp->addmask)) { if (!deleting) return -EINVAL; @@ -2181,6 +2192,13 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) (cpumask_empty(cs->cpus_allowed) || nodes_empty(cs->mems_allowed))) goto out_unlock; + /* + * On default hierarchy, task cannot be moved to a cpuset with empty + * effective cpus. + */ + if (is_in_v2_mode() && cpumask_empty(cs->effective_cpus)) + goto out_unlock; + cgroup_taskset_for_each(task, css, tset) { ret = task_can_attach(task, cs->cpus_allowed); if (ret) From patchwork Thu Jun 3 21:24:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12298095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51E03C47082 for ; Thu, 3 Jun 2021 21:25:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D051613FF for ; Thu, 3 Jun 2021 21:25:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230254AbhFCV0u (ORCPT ); Thu, 3 Jun 2021 17:26:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:39263 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230261AbhFCV0t (ORCPT ); Thu, 3 Jun 2021 17:26:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622755503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=thMjmtJh5+hm9zST5bagiDDjjocf7BWJz4f5zZJsM5c=; b=SK7sjfMnj6SR4Aa2/ZUVTNrbJuxzQNoPbMv6PP7hOor9G8vMxyIKcQBBs+XGQ/CSoi98or sPwZryTZRjfEWYmvh5EOP9tOl7p3b4mkIHVsVHHyJ2oIaWAi2sMLribYqlkEtTajLctFE/ eGcgUkJLBcFFFkmFP+W0RY2BKUBbwfs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-315-SmUDUhhCMpKbkpLm1y9EOg-1; Thu, 03 Jun 2021 17:25:00 -0400 X-MC-Unique: SmUDUhhCMpKbkpLm1y9EOg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BA602107ACC7; Thu, 3 Jun 2021 21:24:58 +0000 (UTC) Received: from llong.com (ovpn-116-222.rdu2.redhat.com [10.10.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69AA460D06; Thu, 3 Jun 2021 21:24:57 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Andrew Morton , Roman Gushchin , Phil Auld , Peter Zijlstra , Juri Lelli , Waiman Long Subject: [PATCH 4/5] cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst Date: Thu, 3 Jun 2021 17:24:15 -0400 Message-Id: <20210603212416.25934-5-longman@redhat.com> In-Reply-To: <20210603212416.25934-1-longman@redhat.com> References: <20210603212416.25934-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Update Documentation/admin-guide/cgroup-v2.rst on the new "root-nolb" cpuset partition type as well as the ability to create non-top cpuset partition with no cpu allocated to it. Signed-off-by: Waiman Long --- Documentation/admin-guide/cgroup-v2.rst | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index b1e81aa8598a..36a923cabeb0 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -2010,8 +2010,9 @@ Cpuset Interface Files It accepts only the following input values when written to. ======== ================================ - "root" a partition root - "member" a non-root member of a partition + "member" Non-root member of a partition + "root" Partition root + "root-nolb" Partition root with no load balancing ======== ================================ When set to be a partition root, the current cgroup is the @@ -2020,6 +2021,10 @@ Cpuset Interface Files partition roots themselves and their descendants. The root cgroup is always a partition root. + With "root-nolb", the CPUs in that partition root will be in an + isolated state with no load balancing by the scheduler. Tasks in + such a partition must be explicitly bind to each individual CPU. + There are constraints on where a partition root can be set. It can only be set in a cgroup if all the following conditions are true. @@ -2038,9 +2043,12 @@ Cpuset Interface Files file cannot be reverted back to "member" if there are any child cgroups with cpuset enabled. - A parent partition cannot distribute all its CPUs to its - child partitions. There must be at least one cpu left in the - parent partition. + A parent partition may distribute all its CPUs to its child + partitions as long as it is not the root cgroup and there is no + task directly associated with that parent partition. Otherwise, + there must be at least one cpu left in the parent partition. + A new task cannot be moved to a partition root with no effective + cpu. Once becoming a partition root, changes to "cpuset.cpus" is generally allowed as long as the first condition above is true, @@ -2056,6 +2064,7 @@ Cpuset Interface Files ============== ============================== "member" Non-root member of a partition "root" Partition root + "root-nolb" Partition root with no load balancing "root invalid" Invalid partition root ============== ============================== From patchwork Thu Jun 3 21:24:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12298099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E02DC4709A for ; Thu, 3 Jun 2021 21:25:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3811261401 for ; Thu, 3 Jun 2021 21:25:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230286AbhFCV0w (ORCPT ); Thu, 3 Jun 2021 17:26:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:29781 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230288AbhFCV0v (ORCPT ); Thu, 3 Jun 2021 17:26:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622755506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=AQptwpPftuvlepn0c//rvhF5er61at0BxM0RAxJWfRY=; b=GUXXohO57oA9vFuy/PH+2LpUx90eZLZ1HgxS5DvW9Ami7Ga0rYcj/Fw8koMqmC/afSMwxf 9k8NuVRxGlS4RzWRWGm7cDbF2s5xgRM/8+NTS24jKgv76+yZCYjuDlqrJrGv2CkQ//PPhv qwfRJ5bAwKuE4jKQnny6ev2CpvoQg3M= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-462-W1aTJVnqMymYaNmlUbEitQ-1; Thu, 03 Jun 2021 17:25:04 -0400 X-MC-Unique: W1aTJVnqMymYaNmlUbEitQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4EB9A80ED96; Thu, 3 Jun 2021 21:25:03 +0000 (UTC) Received: from llong.com (ovpn-116-222.rdu2.redhat.com [10.10.116.222]) by smtp.corp.redhat.com (Postfix) with ESMTP id DC5F2648A0; Thu, 3 Jun 2021 21:24:58 +0000 (UTC) From: Waiman Long To: Tejun Heo , Zefan Li , Johannes Weiner , Jonathan Corbet , Shuah Khan Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Andrew Morton , Roman Gushchin , Phil Auld , Peter Zijlstra , Juri Lelli , Waiman Long Subject: [PATCH 5/5] kselftest/cgroup: Add cpuset v2 partition root state test Date: Thu, 3 Jun 2021 17:24:16 -0400 Message-Id: <20210603212416.25934-6-longman@redhat.com> In-Reply-To: <20210603212416.25934-1-longman@redhat.com> References: <20210603212416.25934-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Add a test script for exercising the cpuset v2 partition root state code. Signed-off-by: Waiman Long --- tools/testing/selftests/cgroup/Makefile | 2 +- .../selftests/cgroup/test_cpuset_prs.sh | 141 ++++++++++++++++++ 2 files changed, 142 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/cgroup/test_cpuset_prs.sh diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile index f027d933595b..e8ff3ffc3a43 100644 --- a/tools/testing/selftests/cgroup/Makefile +++ b/tools/testing/selftests/cgroup/Makefile @@ -4,7 +4,7 @@ CFLAGS += -Wall -pthread all: TEST_FILES := with_stress.sh -TEST_PROGS := test_stress.sh +TEST_PROGS := test_stress.sh test_cpuset_prs.sh TEST_GEN_PROGS = test_memcontrol TEST_GEN_PROGS += test_kmem TEST_GEN_PROGS += test_core diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh new file mode 100755 index 000000000000..6a9c02c301b5 --- /dev/null +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh @@ -0,0 +1,141 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# +# Test for cpuset v2 partition root state (PRS) +# +# The sched verbose flag is set, if available, so that the console log +# can be examined for the correct setting of scheduling domain. +# + +skip_test() { + echo "$1" + echo "Test SKIPPED" + exit 0 +} + +[[ $(id -u) -eq 0 ]] || skip_test "Test must be run as root!" + +# Set sched verbose flag, if available +[[ -d /sys/kernel/debug/sched ]] && echo Y > /sys/kernel/debug/sched/verbose + +# Find cgroup v2 mount point +CGROUP2=$(mount | grep "^cgroup2" | awk -e '{print $3}') +[[ -n "$CGROUP2" ]] || skip_test "Cgroup v2 mount point not found!" + +CPUS=$(lscpu | grep "^CPU(s)" | sed -e "s/.*:[[:space:]]*//") +[[ $CPUS -lt 4 ]] && skip_test "Test needs at least 4 cpus available!" + +cd $CGROUP2 +echo +cpuset > cgroup.subtree_control +[[ -d test ]] || mkdir test +cd test +echo 2-3 > cpuset.cpus +TYPE=$(cat cpuset.cpus.partition) +[[ $TYPE = member ]] || echo member > cpuset.cpus.partition + +console_msg() +{ + MSG=$1 + echo "$MSG" + echo "" > /dev/console + echo "$MSG" > /dev/console + sleep 1 +} + +test_partition() +{ + EXPECTED_VAL=$1 + echo $EXPECTED_VAL > cpuset.cpus.partition + [[ $? -eq 0 ]] || exit 1 + ACTUAL_VAL=$(cat cpuset.cpus.partition) + [[ $ACTUAL_VAL != $EXPECTED_VAL ]] && { + echo "cpuset.cpus.partition: expect $EXPECTED_VAL, found $EXPECTED_VAL" + echo "Test FAILED" + exit 1 + } +} + +test_effective_cpus() +{ + EXPECTED_VAL=$1 + ACTUAL_VAL=$(cat cpuset.cpus.effective) + [[ "$ACTUAL_VAL" != "$EXPECTED_VAL" ]] && { + echo "cpuset.cpus.effective: expect '$EXPECTED_VAL', found '$EXPECTED_VAL'" + echo "Test FAILED" + exit 1 + } +} + +# Adding current process to cgroup.procs as a test +test_add_proc() +{ + OUTSTR="$1" + ERRMSG=$((echo $$ > cgroup.procs) |& cat) + echo $ERRMSG | grep -q "$OUTSTR" + [[ $? -ne 0 ]] && { + echo "cgroup.procs: expect '$OUTSTR', got '$ERRMSG'" + echo "Test FAILED" + exit 1 + } + echo $$ > $CGROUP2/cgroup.procs # Move out the task +} + +# +# Testing the new "root-nolb" partition root type +# +console_msg "Change from member to root" +test_partition root + +console_msg "Change from root to root-nolb" +test_partition root-nolb + +console_msg "Change from root-nolb to member" +test_partition member + +console_msg "Change from member to root-nolb" +test_partition root-nolb + +console_msg "Change from root-nolb to root" +test_partition root + +console_msg "Change from root to member" +test_partition member + +# +# Testing partition root with no cpu +# +console_msg "Distribute all cpus to child partition" +echo +cpuset > cgroup.subtree_control +test_partition root + +mkdir t1 +cd t1 +echo 2-3 > cpuset.cpus +test_partition root +test_effective_cpus 2-3 +cd .. +test_effective_cpus "" + +console_msg "Moving task to partition test" +test_add_proc "No space left" +cd t1 +test_add_proc "" +cd .. + +console_msg "Shrink and expand child partition" +cd t1 +echo 2 > cpuset.cpus +cd .. +test_effective_cpus 3 +cd t1 +echo 2-3 > cpuset.cpus +cd .. +test_effective_cpus "" + +# Cleaning up +console_msg "Cleaning up" +echo $$ > $CGROUP2/cgroup.procs +[[ -d t1 ]] && rmdir t1 +cd .. +rmdir test +echo "Test PASSED"