From patchwork Wed Aug 23 05:06:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mateusz Guzik X-Patchwork-Id: 13361596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 629F3EE49AD for ; Wed, 23 Aug 2023 05:06:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0B9028003B; Wed, 23 Aug 2023 01:06:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BB67940007; Wed, 23 Aug 2023 01:06:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8841628003B; Wed, 23 Aug 2023 01:06:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 74C96940007 for ; Wed, 23 Aug 2023 01:06:19 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 348BA1A0530 for ; Wed, 23 Aug 2023 05:06:19 +0000 (UTC) X-FDA: 81154183278.02.36A3A55 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by imf16.hostedemail.com (Postfix) with ESMTP id 4E2B8180013 for ; Wed, 23 Aug 2023 05:06:17 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=ohUoxhT7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692767177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5JRcXwzRi0jQB1TLzrEg7BfGVcbae4iTaoFsakUigOI=; b=f8taQl+9Hzux2GV39jbuFYO15l7IaJB713A548hihikGkIQ2uIDj1egxKEp9J+Zt9wHk+D ZDpPh/U/1gBjJOBb3CPd67BNhVzuN3QEINZGak2Ndoke7CCKzWnR+tTl6szqcJsNwek9R0 mBhvMO3ul/oZeFJY7sVxcUyzwwUpOeI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=ohUoxhT7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692767177; a=rsa-sha256; cv=none; b=w0qLEVaZ8mc5Dh0TGyj1OnOLTKS6ECSpWAJqqO4bCftMoWNNNNZz7swASEHbFxtB0Dt6QN atzs1Z++qUAIgFXsGUJT/GQhBcrQ6BJ8/rS9eT8N7s57D3DIcbNi/jmvOQ7f7w3vqIVsgi AQa1Nr3hpibu6lH9SQG4sDd8NiJuY5c= Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-997c4107d62so691522266b.0 for ; Tue, 22 Aug 2023 22:06:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692767176; x=1693371976; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5JRcXwzRi0jQB1TLzrEg7BfGVcbae4iTaoFsakUigOI=; b=ohUoxhT73ZTR4FOXW3n8KDmTPiUTiFYOtoJO19cnC62PnujQwIJjk04iHc3hG3cvX4 LcMnHxtwfbLBGmTt34u8g0jzz4jZ8yuQ9lUPSoqq+b+ob+vUFne5Ma4yxQhVwQSnItYY TmMzHCyb3oM+uF1g5uULBY71M8RCA6pElrs8DAtqHOYIeTFsd2Cm+xPN0BZGI/A69pMA SEpSyqnm8MbSdOdQpVYqzFhEAzCLb85q7LeCRH+xqLtBatSswuhxxNqrOaKmh/Uq1TLS FgYlSXUrmYwtY0IhveXfbO6t3owUxAb2UvPQY6uUUiBxrD4m2sZ0NQBGum5jb7Km5WD7 t2Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692767176; x=1693371976; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5JRcXwzRi0jQB1TLzrEg7BfGVcbae4iTaoFsakUigOI=; b=a4o6CG3yPCKV6Tpu8Ll6SBBc+mibs/pq42quPVN0L7v6+qMOJ+xAo3I/0vVR35m4Hj TwxovxiwEqoB8t4TdHHvkMeXV2N24boSbbbXZItyLHzzKRCl6PKQ3X6v7Wsof4Xcf1zK CQlWeEwbZrB8PV1OcWf+T8QFuQMsPfhTvsYHCRvVbPYtH9fjYUE89f/eLL1k0ZfHg6pi s7ZRgvKiQZqr+O1hePkj94YgtLH4Up0nqprmdGqfG1CHSoCItevd976dVdEsTsVR46DD iAOP6/mFNzlBXvyV/VMnWS4fMMaWGq8J6rMRALFPtIJnaEA6TQkwFhh6DK7QwPUJFe31 lwVQ== X-Gm-Message-State: AOJu0Ywtj54USmwaD6X+5tJMULGC0ZIPLn6EOITZo3id/8+I/sCM7BYs 8H5EABa2t00zSK+mSQZOb0EYtbDA648+kA== X-Google-Smtp-Source: AGHT+IFWux01B2Oa+XeckE2+Vyuwk0P4V6dysyQJzM5m7Vf7TaVfAA7TF1kKC1SWOti4CX3UZqAF0g== X-Received: by 2002:a17:907:7754:b0:9a1:9284:115 with SMTP id kx20-20020a170907775400b009a192840115mr6015550ejc.67.1692767176002; Tue, 22 Aug 2023 22:06:16 -0700 (PDT) Received: from f.. (cst-prg-85-121.cust.vodafone.cz. [46.135.85.121]) by smtp.gmail.com with ESMTPSA id q16-20020a170906b29000b0099ddc81903asm9267401ejz.221.2023.08.22.22.06.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 22:06:15 -0700 (PDT) From: Mateusz Guzik To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, vegard.nossum@oracle.com, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH v3 1/2] pcpcntr: add group allocation/free Date: Wed, 23 Aug 2023 07:06:08 +0200 Message-Id: <20230823050609.2228718-2-mjguzik@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823050609.2228718-1-mjguzik@gmail.com> References: <20230823050609.2228718-1-mjguzik@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4E2B8180013 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: rzh4jswu3ch8e87e3iutbdn59mqjncbi X-HE-Tag: 1692767177-811780 X-HE-Meta: U2FsdGVkX1/klTTrtwerG1IMusPn87zeTGWerDh95Sl6tV3aR9yIjfHZ+FQSvBN1dEbALVosQRiu6AzFQG+ZphCo53xm/tv1Az4jJIOggcUp9tdU95X1HolihGMOXTOKFXr1YVFw/iuz4K4EbSTHTUDtwz+kXTU4uuIgzrKuDxs6+LWJyfDRgHG8rwzHfbWVhCNDFC6aDiMnD5+CRZuVnaiSP5s8k0YmTZHqSRv9EEZXvElhmFnlsHEDA/KCYR8XdYkgw4bFHfT7g9Vja43F4s/iyrozqE4jOJjXQadJwYrF+9Z30vbcOTb7Dlj0RL8ePfergmh9qmgwraYoJdg1U3VyT9YK6lQndm1nCM/dxGtTA3Q519gc7MTSXIG3KZsGNlVjRh19S1PFhoQ/IUacND+hQ6pW3LbVCM8Qb0vJHjZRLPkfKQNadcomVrTx6FchJDfsdd/2LVcYcPaLIzpvZ191Oj0Wc5wxLCUjKMxdkhHUeCfqEyUYVG+ojJFKgoY9/BtE3sWMyPUGQfRjK0VHoBIae5jYpcu4jqxR+m3ngg1reu2pNasHz6RcjwO0NXWtYLSfq5gL2bONdNBVtXGhbc97JVxSz3H26At3gXpfCxadDwWVz565DVg59RNGscewlfZW+Hhy5mw94AZG21SNwjfhKMpVfvY9RbKlV6ngcHGrBl13hkxpZx9s8pvgD8DtBiTmgIfaBuXvsfUbGAdrL3IeRxEsLKNgRTg8A9NuQzsCMO3sC3wlszywdgvreHrDnz5UG8kvfRPf+TdTLf46O+E7kRFv5ar9mMiyjxdNJYZVyro48jHYkhGmWGQQyf/fI10AmQWRs9YqGySwTndr5UEaJALm7N8sOmOYq47SN/qT+FIpdiRZxeaj69rFHhBw0/J3kDPZ/SC99WjAypt+p8O7gxM0D8wCu25m60+FYm/o+PCUjv4TUn9hiXEKHhG0pZOnVwb7JISVgaI9Wm9 +rrLgv9i n9FLWrKOwJQP6vM/u5aFRVmUVoj2YnAmjyzO++fKQxopr/ZvazlepC+pBceHYTC3Z35P8enpAt+AsuGF9RuKsyXr+qDKXOedcHWe7PnuFHpyr1LCPU5z2oCLsiNb+BNKgpKBeVqxWW7/hTFms6c2qg6PXGwRC+B7eMS/IxDCQK8AYBHV1yGuAHZiZs4COnkboJULe9xTJG8FQibrz5HAnuA8xUwu5y1XpoUuSWVNMGS1AzupFIor0khgiA6qrzUSDe/4ShHVLsQhYVt7AwS8VOhBT9Wq4N+o2zi2me5OqpnNx+1lBum4fUhbM31yHfD2U87KnM5D5KUSsrFgjWdshSltIZiomp7720I0E99NH/66EZ7JvGV++dXJC67tY/A8jZVlxmrIKadDHAro1vVAvHc2kLU7cYEEgdKAMRc88s5dQxdGI5hT2k0jmWB/a8Y+4HuiV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocations and frees are globally serialized on the pcpu lock (and the CPU hotplug lock if enabled, which is the case on Debian). At least one frequent consumer allocates 4 back-to-back counters (and frees them in the same manner), exacerbating the problem. While this does not fully remedy scalability issues, it is a step towards that goal and provides immediate relief. Signed-off-by: Mateusz Guzik Reviewed-by: Dennis Zhou Reviewed-by: Vegard Nossum --- include/linux/percpu_counter.h | 39 ++++++++++++++++++---- lib/percpu_counter.c | 61 +++++++++++++++++++++++----------- 2 files changed, 74 insertions(+), 26 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 75b73c83bc9d..f1e7c987e3d3 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -30,17 +30,27 @@ struct percpu_counter { extern int percpu_counter_batch; -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, - struct lock_class_key *key); +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, + u32 nr_counters, struct lock_class_key *key); -#define percpu_counter_init(fbc, value, gfp) \ +#define percpu_counter_init_many(fbc, value, gfp, nr_counters) \ ({ \ static struct lock_class_key __key; \ \ - __percpu_counter_init(fbc, value, gfp, &__key); \ + __percpu_counter_init_many(fbc, value, gfp, nr_counters,\ + &__key); \ }) -void percpu_counter_destroy(struct percpu_counter *fbc); + +#define percpu_counter_init(fbc, value, gfp) \ + percpu_counter_init_many(fbc, value, gfp, 1) + +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counters); +static inline void percpu_counter_destroy(struct percpu_counter *fbc) +{ + percpu_counter_destroy_many(fbc, 1); +} + void percpu_counter_set(struct percpu_counter *fbc, s64 amount); void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 batch); @@ -116,11 +126,26 @@ struct percpu_counter { s64 count; }; +static inline int percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, + gfp_t gfp, u32 nr_counters) +{ + u32 i; + + for (i = 0; i < nr_counters; i++) + fbc[i].count = amount; + + return 0; +} + static inline int percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp) { - fbc->count = amount; - return 0; + return percpu_counter_init_many(fbc, amount, gfp, 1); +} + +static inline void percpu_counter_destroy_many(struct percpu_counter *fbc, + u32 nr_counters) +{ } static inline void percpu_counter_destroy(struct percpu_counter *fbc) diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 5004463c4f9f..9338b27f1cdd 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -151,48 +151,71 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) } EXPORT_SYMBOL(__percpu_counter_sum); -int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, - struct lock_class_key *key) +int __percpu_counter_init_many(struct percpu_counter *fbc, s64 amount, gfp_t gfp, + u32 nr_counters, struct lock_class_key *key) { unsigned long flags __maybe_unused; - - raw_spin_lock_init(&fbc->lock); - lockdep_set_class(&fbc->lock, key); - fbc->count = amount; - fbc->counters = alloc_percpu_gfp(s32, gfp); - if (!fbc->counters) + size_t counter_size; + s32 __percpu *counters; + u32 i; + + counter_size = ALIGN(sizeof(*counters), __alignof__(*counters)); + counters = __alloc_percpu_gfp(nr_counters * counter_size, + __alignof__(*counters), gfp); + if (!counters) { + fbc[0].counters = NULL; return -ENOMEM; + } - debug_percpu_counter_activate(fbc); + for (i = 0; i < nr_counters; i++) { + raw_spin_lock_init(&fbc[i].lock); + lockdep_set_class(&fbc[i].lock, key); +#ifdef CONFIG_HOTPLUG_CPU + INIT_LIST_HEAD(&fbc[i].list); +#endif + fbc[i].count = amount; + fbc[i].counters = (void *)counters + (i * counter_size); + + debug_percpu_counter_activate(&fbc[i]); + } #ifdef CONFIG_HOTPLUG_CPU - INIT_LIST_HEAD(&fbc->list); spin_lock_irqsave(&percpu_counters_lock, flags); - list_add(&fbc->list, &percpu_counters); + for (i = 0; i < nr_counters; i++) + list_add(&fbc[i].list, &percpu_counters); spin_unlock_irqrestore(&percpu_counters_lock, flags); #endif return 0; } -EXPORT_SYMBOL(__percpu_counter_init); +EXPORT_SYMBOL(__percpu_counter_init_many); -void percpu_counter_destroy(struct percpu_counter *fbc) +void percpu_counter_destroy_many(struct percpu_counter *fbc, u32 nr_counters) { unsigned long flags __maybe_unused; + u32 i; + + if (WARN_ON_ONCE(!fbc)) + return; - if (!fbc->counters) + if (!fbc[0].counters) return; - debug_percpu_counter_deactivate(fbc); + for (i = 0; i < nr_counters; i++) + debug_percpu_counter_deactivate(&fbc[i]); #ifdef CONFIG_HOTPLUG_CPU spin_lock_irqsave(&percpu_counters_lock, flags); - list_del(&fbc->list); + for (i = 0; i < nr_counters; i++) + list_del(&fbc[i].list); spin_unlock_irqrestore(&percpu_counters_lock, flags); #endif - free_percpu(fbc->counters); - fbc->counters = NULL; + + free_percpu(fbc[0].counters); + + for (i = 0; i < nr_counters; i++) + fbc[i].counters = NULL; } -EXPORT_SYMBOL(percpu_counter_destroy); +EXPORT_SYMBOL(percpu_counter_destroy_many); int percpu_counter_batch __read_mostly = 32; EXPORT_SYMBOL(percpu_counter_batch);