From patchwork Sat Nov 5 01:40:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shakeel Butt X-Patchwork-Id: 13032798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 049A6C433FE for ; Sat, 5 Nov 2022 01:40:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64A358E0001; Fri, 4 Nov 2022 21:40:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F9F96B0073; Fri, 4 Nov 2022 21:40:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 510038E0001; Fri, 4 Nov 2022 21:40:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 429066B0071 for ; Fri, 4 Nov 2022 21:40:30 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 13935140459 for ; Sat, 5 Nov 2022 01:40:30 +0000 (UTC) X-FDA: 80097683820.23.5EAEFC6 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf14.hostedemail.com (Postfix) with ESMTP id 9D418100003 for ; Sat, 5 Nov 2022 01:40:29 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id oo18-20020a17090b1c9200b0020bdba475afso6450987pjb.4 for ; Fri, 04 Nov 2022 18:40:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=8jkJGrxn/AkT1+zj9lg1NbIOC+/Y0zcaC/bhas2fS20=; b=sj1uClZYPWF9+g/Af6U4lnd/QJhXqeY3U+fGEwKz1MyD8aXjH0oZ0MZFjaSc6iuZLu zCdJYg8aA0oDVN58TmQ95Jm0SX7knr8iOICAk6In4Z3Zbb+Mz+b59VHuotsDCPKAf0OA j1gW8EYpt++wGl9NTi3NqMeFWAptSSYuZTEZsnKYNdAoVHRuLqoUY3sSQo+nDU61WdgA 5dwYUptpKy/a2PkcHkBhFsf/CujyPzLixs3RwqOEtbLanGANqZNoBnq9b3gE+iJq+hit eW+6YW19QOIUUELIv4oHvt54A5yWsRzqCEr78MVJU1wSdEWEHHdGSnkICTwkbiEo7SqX 6wcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=8jkJGrxn/AkT1+zj9lg1NbIOC+/Y0zcaC/bhas2fS20=; b=F/NtnbQ/KIRU8ovfvfKiAqpoK2LGbVrJ2qqlg/RrzFAsvvld7eZ1OV9B5ReqyIce1+ DfPset5JZQeRhRg/8ayn/H5o/+uEOvcFW4t3PF8uYzeeplM75l2riqPhPTzcxDtbuzHo h/yK9zB6mUeFLwT/Pj7ggIkf8tCCUuKoiraUfTIBTwea6vRzON3iZbCY+qYLgf87QMY/ YgFI+EveE2LruC1J/VQ6eSwsgX3K3t4n8ZCUhNTca1QCRv7+9RZ20NKsVFO1X2aOLTX8 wnVrYHEwiekfB5rjYwFI0yENpAUjNlauU5pbecq3v1wxrcuvLu6YudjLk1dgKUALl+54 b+ig== X-Gm-Message-State: ANoB5pm7NLmkCpwaAETiSbsVifOIWIhoI8rXKFvwmFSEbRGJPWlCG8DC 6Uzl3UnIDA1VQkI/j6R/qLmR8CebCyzqFw== X-Google-Smtp-Source: AA0mqf6jfoCQ+f91qDyQhm2aDDOkq+E3ZjTnt72QOCfo1bxzDsUFyW3esvQ31twA11J5QruAiX9io9bm22nHyA== X-Received: from shakeelb.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:262e]) (user=shakeelb job=sendgmr) by 2002:a17:90a:f291:b0:20a:eab5:cf39 with SMTP id fs17-20020a17090af29100b0020aeab5cf39mr99870pjb.1.1667612428036; Fri, 04 Nov 2022 18:40:28 -0700 (PDT) Date: Sat, 5 Nov 2022 01:40:13 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221105014013.930636-1-shakeelb@google.com> Subject: [PATCH] percpu_counter: add percpu_counter_sum_all interface From: Shakeel Butt To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Shakeel Butt , Marek Szyprowski ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667612429; a=rsa-sha256; cv=none; b=V0zdal96LfjHX9q+2AYrNyhrHGlgRStX5FO7h8fJRPI+xYFEIkc6o32Q8u6GwbfJLX3Wnz JCHvP7a3pra1GOlZ+UrsevCqlA2WQxSBGIaGk87paOFbsjKYmhN5U1HPiqEt73VMejpX4X eK90eci6PzYX1stw1Rd3xpddNb0C9bM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sj1uClZY; spf=pass (imf14.hostedemail.com: domain of 3DL9lYwgKCEMxmfpjjqglttlqj.htrqnsz2-rrp0fhp.twl@flex--shakeelb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3DL9lYwgKCEMxmfpjjqglttlqj.htrqnsz2-rrp0fhp.twl@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667612429; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=8jkJGrxn/AkT1+zj9lg1NbIOC+/Y0zcaC/bhas2fS20=; b=x8w+AqROncKATM2GLI0exBLfbOB+msMr5ADfMFONatcHEjnuv2ol+/wBQPNppRP4VZKjfb kP/lisFnPgC10di4IWBkp0VNBNz4kNAlZLul2Jr57xzw4Ggq+HAM7ehiEeEtEPDAFkA2uG i4eT3eWFC4meNm2NqxPD883/pGkmZyA= X-Stat-Signature: 8g7j1mbrtinnf7x7aqyob7sfekwz58cs X-Rspamd-Server: rspam09 X-Rspam-User: X-Rspamd-Queue-Id: 9D418100003 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sj1uClZY; spf=pass (imf14.hostedemail.com: domain of 3DL9lYwgKCEMxmfpjjqglttlqj.htrqnsz2-rrp0fhp.twl@flex--shakeelb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3DL9lYwgKCEMxmfpjjqglttlqj.htrqnsz2-rrp0fhp.twl@flex--shakeelb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1667612429-570621 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The percpu_counter is used for scenarios where performance is more important than the accuracy. For percpu_counter users, who want more accurate information in their slowpath, percpu_counter_sum is provided which traverses all the online CPUs to accumulate the data. The reason it only needs to traverse online CPUs is because percpu_counter does implement CPU offline callback which syncs the local data of the offlined CPU. However there is a small race window between the online CPUs traversal of percpu_counter_sum and the CPU offline callback. The offline callback has to traverse all the percpu_counters on the system to flush the CPU local data which can be a lot. During that time, the CPU which is going offline has already been published as offline to all the readers. So, as the offline callback is running, percpu_counter_sum can be called for one counter which has some state on the CPU going offline. Since percpu_counter_sum only traverses online CPUs, it will skip that specific CPU and the offline callback might not have flushed the state for that specific percpu_counter on that offlined CPU. Normally this is not an issue because percpu_counter users can deal with some inaccuracy for small time window. However a new user i.e. mm_struct on the cleanup path wants to check the exact state of the percpu_counter through check_mm(). For such users, this patch introduces percpu_counter_sum_all() which traverses all possible CPUs. Signed-off-by: Shakeel Butt Reported-by: Marek Szyprowski Tested-by: Marek Szyprowski --- include/linux/percpu_counter.h | 6 ++++++ kernel/fork.c | 2 +- lib/percpu_counter.c | 29 +++++++++++++++++++++++------ 3 files changed, 30 insertions(+), 7 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index bde6c4c1f405..a3aae8d57a42 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -45,6 +45,7 @@ void percpu_counter_set(struct percpu_counter *fbc, s64 amount); void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 batch); s64 __percpu_counter_sum(struct percpu_counter *fbc); +s64 percpu_counter_sum_all(struct percpu_counter *fbc); int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch); void percpu_counter_sync(struct percpu_counter *fbc); @@ -193,6 +194,11 @@ static inline s64 percpu_counter_sum(struct percpu_counter *fbc) return percpu_counter_read(fbc); } +static inline s64 percpu_counter_sum_all(struct percpu_counter *fbc) +{ + return percpu_counter_read(fbc); +} + static inline bool percpu_counter_initialized(struct percpu_counter *fbc) { return true; diff --git a/kernel/fork.c b/kernel/fork.c index 9c32f593ef11..7d6f510cf397 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -756,7 +756,7 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well"); for (i = 0; i < NR_MM_COUNTERS; i++) { - long x = percpu_counter_sum(&mm->rss_stat[i]); + long x = percpu_counter_sum_all(&mm->rss_stat[i]); if (unlikely(x)) pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index ed610b75dc32..42f729c8e56c 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -117,11 +117,8 @@ void percpu_counter_sync(struct percpu_counter *fbc) } EXPORT_SYMBOL(percpu_counter_sync); -/* - * Add up all the per-cpu counts, return the result. This is a more accurate - * but much slower version of percpu_counter_read_positive() - */ -s64 __percpu_counter_sum(struct percpu_counter *fbc) +static s64 __percpu_counter_sum_mask(struct percpu_counter *fbc, + const struct cpumask *cpu_mask) { s64 ret; int cpu; @@ -129,15 +126,35 @@ s64 __percpu_counter_sum(struct percpu_counter *fbc) raw_spin_lock_irqsave(&fbc->lock, flags); ret = fbc->count; - for_each_online_cpu(cpu) { + for_each_cpu(cpu, cpu_mask) { s32 *pcount = per_cpu_ptr(fbc->counters, cpu); ret += *pcount; } raw_spin_unlock_irqrestore(&fbc->lock, flags); return ret; } + +/* + * Add up all the per-cpu counts, return the result. This is a more accurate + * but much slower version of percpu_counter_read_positive() + */ +s64 __percpu_counter_sum(struct percpu_counter *fbc) +{ + return __percpu_counter_sum_mask(fbc, cpu_online_mask); +} EXPORT_SYMBOL(__percpu_counter_sum); +/* + * This is slower version of percpu_counter_sum as it traverses all possible + * cpus. Use this only in the cases where accurate data is needed in the + * presense of CPUs getting offlined. + */ +s64 percpu_counter_sum_all(struct percpu_counter *fbc) +{ + return __percpu_counter_sum_mask(fbc, cpu_possible_mask); +} +EXPORT_SYMBOL(percpu_counter_sum_all); + int __percpu_counter_init(struct percpu_counter *fbc, s64 amount, gfp_t gfp, struct lock_class_key *key) {