From patchwork Mon Jul 18 19:28:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 12921665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA728C43334 for ; Mon, 18 Jul 2022 19:28:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C2F7940009; Mon, 18 Jul 2022 15:28:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 422F2940007; Mon, 18 Jul 2022 15:28:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29C31940009; Mon, 18 Jul 2022 15:28:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1759A940007 for ; Mon, 18 Jul 2022 15:28:56 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DE5F333FA4 for ; Mon, 18 Jul 2022 19:28:55 +0000 (UTC) X-FDA: 79701208230.10.A430CD6 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf29.hostedemail.com (Postfix) with ESMTP id 99F2E120033 for ; Mon, 18 Jul 2022 19:28:55 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id c24so4029571qkm.4 for ; Mon, 18 Jul 2022 12:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=m9V57xphOJe8FxCe80bZ1J3RuFelVtvXoENVWMfISb4=; b=B0XuB+yAeTJmK6B+vnHnd3BO8yIPsZ8lCES1PpoVp7Ti9ESY8L/5LseEzml+iCgDqI 1YlvnrUUgNx4DD7RQwT4vms4g+VmvWZpGoAnJNKJ0UYr9HyWBXYZufeEWv7Vuwxy/4nI qIrQ27tmeRvKsD0HQGA4yMLWkqHFveDpBWb4zZNi9MJuv/+VtQRCqaE5PFi0tfV7R98a 9bkurHLXilYTf8+/4Z6M5WqsGBng481jqtXDo5QizTdcTOcHRejuVHRFGd14+vBFF67B NUB7Epima9nbyqEkq0yYEeKIgt/HN7ct0hLat+eWTlW+qrZnqyzPXV06pMIjXg2GHmsY vUnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m9V57xphOJe8FxCe80bZ1J3RuFelVtvXoENVWMfISb4=; b=UNz1xs1IlD1Cp2Kc4j3JmyK0eHCmfCq8uXe4S8wtSlxFsZqT6SG9SWXmGtZy22gQ/A uzZVmsH7pEl/vz9ZuHCvTWr7CKoQOQ7dtU2AFwHFbtbRPvMqk45ewZa3yEcChyTgxHws DYCkd1eyK/ZM2DNDchHZP+wnKIW3zp4aZx2P92fV8AuJZZNqIzXTpzsf5X5IBTS/VgHe d+faXsmhrB6g1poorhvrFPMZlkUxPqAjChHruoJif+HkbGt4GiE38qiGHVq2/mIdUu+k 9VU26/qghS72ihxUjMDRkpT2XwZL3b4ttaQSTDxBmoM57a9Cn2mjQFCyurjUYPobl9c4 wHeg== X-Gm-Message-State: AJIora+Vfb8dJRrtlCTzF+yRRHgGWgMdr53E+uCmPBK0RYUHBsuLcs9y vc4/GGMJp09dPnXSPH5GDfw= X-Google-Smtp-Source: AGRyM1vQdCay2nDpE1RIgzYmRYjdYqMGFdzO6hLEN38y55+zWOu2X7pBEMBNCfY35RCjup6wtifAoA== X-Received: by 2002:a05:620a:468e:b0:6b5:af5b:6e5f with SMTP id bq14-20020a05620a468e00b006b5af5b6e5fmr18034173qkb.288.1658172534753; Mon, 18 Jul 2022 12:28:54 -0700 (PDT) Received: from localhost ([2601:4c1:c100:1230:ab01:d009:465a:5ab1]) by smtp.gmail.com with ESMTPSA id x6-20020a05620a258600b006a65c58db99sm12736880qko.64.2022.07.18.12.28.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:28:54 -0700 (PDT) From: Yury Norov To: linux-kernel@vger.kernel.org, Alexander Lobakin , Alexei Starovoitov , Alexey Klimov , Andrew Morton , Andrii Nakryiko , Andy Shevchenko , Ben Segall , Christoph Lameter , Dan Williams , Daniel Borkmann , Daniel Bristot de Oliveira , Dennis Zhou , Dietmar Eggemann , Eric Dumazet , Frederic Weisbecker , Guenter Roeck , Ingo Molnar , Isabella Basso , John Fastabend , Josh Poimboeuf , Juergen Gross , Juri Lelli , KP Singh , Kees Cook , Martin KaFai Lau , Mel Gorman , Miroslav Benes , Nathan Chancellor , "Paul E . McKenney" , Peter Zijlstra , Randy Dunlap , Rasmus Villemoes , Sebastian Andrzej Siewior , Song Liu , Steven Rostedt , Tejun Heo , Thomas Gleixner , Valentin Schneider , Vincent Guittot , Vlastimil Babka , Yonghong Song , Yury Norov , linux-mm@kvack.org, netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 07/16] smp: optimize smp_call_function_many_cond() Date: Mon, 18 Jul 2022 12:28:35 -0700 Message-Id: <20220718192844.1805158-8-yury.norov@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220718192844.1805158-1-yury.norov@gmail.com> References: <20220718192844.1805158-1-yury.norov@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=B0XuB+yA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of yury.norov@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=yury.norov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658172535; a=rsa-sha256; cv=none; b=iarqp/XZI6thsbD2HDs1Hn/PkUqLoPth+aCVM0IxCR8/J5+KQeAgwUUuMlOXzKUt/cJG15 bcPpTzRZfjJYY1PJEsVh/R0RwYXN9e89rp7kumZbEWC5dBoET6/o4n0nbm/nuGp9ZsFfeT 9Ntq/+XjWqFrigJptjeYZmLk+MO/fgQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658172535; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m9V57xphOJe8FxCe80bZ1J3RuFelVtvXoENVWMfISb4=; b=DgcBAWKE13nr13ciNRGBoUVBteaYoNbOOKi/4tsbsUkuiJ3nr0mYKidJB2twaW/odVSaVo 9R51k1Qq+nUeUj75GOxFt/P0/UzFFo+I0RCuxA7RIOIXeNofiK2eE0odRoYkOvmckSyt35 dJtUC9sSNXMjdzRvnaLl+xQNs9axRN0= Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=B0XuB+yA; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of yury.norov@gmail.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=yury.norov@gmail.com X-Rspam-User: X-Stat-Signature: ch8t4bxgonydruz7kd4uz31m7aptxdm4 X-Rspamd-Queue-Id: 99F2E120033 X-Rspamd-Server: rspam03 X-HE-Tag: 1658172535-597685 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: smp_call_function_many_cond() is often passed with cpu_online_mask. If this is the case, we can use num_online_cpus(), which is O(1) instead of cpumask_{first,next}(), which is O(N). It can be optimized further: if cpu_online_mask has 0 or single bit set (depending on cpu_online(this_cpu), we can return result without AND'ing with user's mask. Caught with CONFIG_DEBUG_BITMAP: [ 7.830337] Call trace: [ 7.830397] __bitmap_check_params+0x1d8/0x260 [ 7.830499] smp_call_function_many_cond+0x1e8/0x45c [ 7.830607] kick_all_cpus_sync+0x44/0x80 [ 7.830698] bpf_int_jit_compile+0x34c/0x5cc [ 7.830796] bpf_prog_select_runtime+0x118/0x190 [ 7.830900] bpf_prepare_filter+0x3dc/0x51c [ 7.830995] __get_filter+0xd4/0x170 [ 7.831145] sk_attach_filter+0x18/0xb0 [ 7.831236] sock_setsockopt+0x5b0/0x1214 [ 7.831330] __sys_setsockopt+0x144/0x170 [ 7.831431] __arm64_sys_setsockopt+0x2c/0x40 [ 7.831541] invoke_syscall+0x48/0x114 [ 7.831634] el0_svc_common.constprop.0+0x44/0xfc [ 7.831745] do_el0_svc+0x30/0xc0 [ 7.831825] el0_svc+0x2c/0x84 [ 7.831899] el0t_64_sync_handler+0xbc/0x140 [ 7.831999] el0t_64_sync+0x18c/0x190 [ 7.832086] ---[ end trace 0000000000000000 ]--- [ 7.832375] b1: ffff24d1ffd98a48 [ 7.832385] b2: ffffa65533a29a38 [ 7.832393] b3: ffffa65533a29a38 [ 7.832400] nbits: 256 [ 7.832407] start: 0 [ 7.832412] off: 0 [ 7.832418] smp: Bitmap: parameters check failed [ 7.832432] smp: include/linux/bitmap.h [363]: bitmap_and Signed-off-by: Yury Norov --- kernel/smp.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/kernel/smp.c b/kernel/smp.c index dd215f439426..7ed2b9b12f74 100644 --- a/kernel/smp.c +++ b/kernel/smp.c @@ -880,6 +880,28 @@ EXPORT_SYMBOL_GPL(smp_call_function_any); #define SCF_WAIT (1U << 0) #define SCF_RUN_LOCAL (1U << 1) +/* Check if we need remote execution, i.e., any CPU excluding this one. */ +static inline bool __need_remote_exec(const struct cpumask *mask, unsigned int this_cpu) +{ + unsigned int cpu; + + switch (num_online_cpus()) { + case 0: + return false; + case 1: + return cpu_online(this_cpu) ? false : true; + default: + if (mask == cpu_online_mask) + return true; + } + + cpu = cpumask_first_and(mask, cpu_online_mask); + if (cpu == this_cpu) + cpu = cpumask_next_and(cpu, mask, cpu_online_mask); + + return cpu < nr_cpu_ids; +} + static void smp_call_function_many_cond(const struct cpumask *mask, smp_call_func_t func, void *info, unsigned int scf_flags, @@ -916,12 +938,7 @@ static void smp_call_function_many_cond(const struct cpumask *mask, if ((scf_flags & SCF_RUN_LOCAL) && cpumask_test_cpu(this_cpu, mask)) run_local = true; - /* Check if we need remote execution, i.e., any CPU excluding this one. */ - cpu = cpumask_first_and(mask, cpu_online_mask); - if (cpu == this_cpu) - cpu = cpumask_next_and(cpu, mask, cpu_online_mask); - if (cpu < nr_cpu_ids) - run_remote = true; + run_remote = __need_remote_exec(mask, this_cpu); if (run_remote) { cfd = this_cpu_ptr(&cfd_data);