From patchwork Thu Jul 28 20:45:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AABF9C19F2C for ; Thu, 28 Jul 2022 20:45:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AEF96B007B; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 336C16B007D; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 162DF6B007E; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0558E6B007B for ; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CC78E1408E9 for ; Thu, 28 Jul 2022 20:45:33 +0000 (UTC) X-FDA: 79737689346.02.0EC11AF Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf06.hostedemail.com (Postfix) with ESMTP id DB360180093 for ; Thu, 28 Jul 2022 20:45:32 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id q16so2433201pgq.6 for ; Thu, 28 Jul 2022 13:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=o9jWwsLlYZnS171yrNbtFKKGeaY8IhQRPNlofd9rtnc=; b=PC9ISe/Tv2ntpg8FscV4ebolxpt+AhXtqalCdJ/HM/qaDIdf1rxD2vBYVhSyfRLxvQ GHNNXjMEXEikSN1uQJEUc4Zhe57ufRlQnqw9TudVYtELg/+5ehUxd4ndJQJ5R6PRGeeT r+ufm10YR1mOAqOW40t42Qr6bFNcdDvwRbzl9OkfnTCh+iEnGxt/DT6muqSIGo96CuOf et+sFqcvlWWFb9bXbnS+xT11zv2W1sj53OfXveVBtRe+8b7qrKDj27Z/dZd3xwMT1a5G exxkBIZstIbrCsALmdrvZOuGiqwEUnFgdG1pJx0C5F7hTIdLNEc4O30Ottfvj3r5Ihoq Gx8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=o9jWwsLlYZnS171yrNbtFKKGeaY8IhQRPNlofd9rtnc=; b=5viZ9xG2MD7Qj5cZPjXaXBTBmrQnGItMU7txqZuSIGgm9r984oVHvuwOT0MEtOl0sY l5W1kr+ro8q/X71jdl91AZiXGVbU/YEHLdc1il1CKDxDWaN3rxoqljGbwV/Vtel0Bzi5 A9pMhW0y1iA5m9qncAAtZxdLAyIJCblYZymfCiBETBxBxxHALhMrDavfL8bwwSnv0HNu Jc91Ios/EFXo3KYK1Sd1HBTKEGvJgWm+ghGKyJ5WODg8qALDQBfg3aTPqMn19ukY6Fm3 uW+Dg+GBhAnDJScPykSV2yAIZH8KTtJPtyfIpfU6TiSFBO+PE4tn7jP3hRXARK265+ag Am7w== X-Gm-Message-State: AJIora8foRRvpgZbqDWNhxigwLsh2g0ubEaZ141gSFn6C0Zuhr3oqG6f aDXsfq8otu7LfsS25L5G76rMJHkGlx1HY7W3md8= X-Google-Smtp-Source: AGRyM1vsVwYmFD3XfCQrIG8QN1BCM3jR8q0DPVN7U5JQfZ3X2kYLzuo6hZlQfhMP+vyClpJD2IEHiw== X-Received: by 2002:a05:6a00:1d26:b0:52b:f8ab:6265 with SMTP id a38-20020a056a001d2600b0052bf8ab6265mr340380pfx.54.1659041131938; Thu, 28 Jul 2022 13:45:31 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:31 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 6/7] mm: introduce CONFIG_ARCH_PCP_RSS_USE_CPUMASK Date: Fri, 29 Jul 2022 04:45:10 +0800 Message-Id: <20220728204511.56348-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="PC9ISe/T"; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041133; a=rsa-sha256; cv=none; b=vG6cAbBQ8lJorug+Xs3VT0vEZM9o+FxYsuwpdBzdmxIcz6yO8zfyvXUvGQvqlsdglHOfxe nHH7tPr22wWcxYTyjFYeGhuj94Uz91t/Rwb3mViGR8XiqUZbwG7dHvyAimcax9GF7yC0fq Y0UUyQHnzKiPIYirGlePz/zYf+aoe/Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041133; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o9jWwsLlYZnS171yrNbtFKKGeaY8IhQRPNlofd9rtnc=; b=cJk3pELgTXEdFUD2XxvX48trQTJKCIAUDhMaUgvHon7XkqQfbszWzPmeE9CgzGk5/RI7Ri gzhujI0fYmZ1+6ONsh7cfkV12WGzPaE10ZcTj2FOz3jrq7Zu9RVb1WqKmqJOGPHLKLtW/s tSZwsY3JdLp5XXHIFQohgt+gB5nnQnY= X-Rspamd-Queue-Id: DB360180093 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="PC9ISe/T"; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspam-User: X-Stat-Signature: 1z3xydjstxai1t3johz86kyrzuijreg7 X-HE-Tag: 1659041132-868781 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song If the arch related code can provide helpers to bind the RSS cache to mm_cpumask, then the syncing code can just rely on that instead of doing full CPU synchronization. This speed up the reading/mm_exit by a lot. Signed-off-by: Kairui Song --- arch/Kconfig | 3 ++ kernel/sched/core.c | 3 +- mm/memory.c | 94 ++++++++++++++++++++++++++++----------------- 3 files changed, 64 insertions(+), 36 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 71b9272acb28..8df45b6346ae 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1403,6 +1403,9 @@ config ARCH_HAS_ELFCORE_COMPAT config ARCH_HAS_PARANOID_L1D_FLUSH bool +config ARCH_PCP_RSS_USE_CPUMASK + bool + config DYNAMIC_SIGFRAME bool diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 11df67bb52ee..6f7991caf24b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5143,7 +5143,8 @@ context_switch(struct rq *rq, struct task_struct *prev, prepare_lock_switch(rq, next, rf); /* Cache new active_mm */ - switch_pcp_rss_cache_no_irq(next->active_mm); + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) + switch_pcp_rss_cache_no_irq(next->active_mm); /* Here we just switch the register state and the stack. */ switch_to(prev, next, prev); diff --git a/mm/memory.c b/mm/memory.c index 09d7d193da51..a819009aa3e0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -188,9 +188,16 @@ unsigned long get_mm_counter(struct mm_struct *mm, int member) { int cpu; long ret, update, sync_count; + const struct cpumask *mm_mask; ret = atomic_long_read(&mm->rss_stat.count[member]); - for_each_possible_cpu(cpu) { + + if (IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) + mm_mask = mm_cpumask(mm); + else + mm_mask = cpu_possible_mask; + + for_each_cpu(cpu, mm_mask) { if (READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) != mm) continue; sync_count = READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)); @@ -217,12 +224,18 @@ unsigned long get_mm_rss(struct mm_struct *mm) { int cpu; long ret, update, sync_count; + const struct cpumask *mm_mask; ret = atomic_long_read(&mm->rss_stat.count[MM_FILEPAGES]), + atomic_long_read(&mm->rss_stat.count[MM_ANONPAGES]), + atomic_long_read(&mm->rss_stat.count[MM_SHMEMPAGES]); - for_each_possible_cpu(cpu) { + if (IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) + mm_mask = mm_cpumask(mm); + else + mm_mask = cpu_possible_mask; + + for_each_cpu(cpu, mm_mask) { if (READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) != mm) continue; sync_count = READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)); @@ -266,10 +279,13 @@ void switch_pcp_rss_cache_no_irq(struct mm_struct *next_mm) if (cpu_mm == NULL) goto commit_done; - /* Race with check_discard_rss_cache */ - if (cpu_mm != cmpxchg(this_cpu_ptr(&cpu_rss_cache.mm), cpu_mm, - __pcp_rss_mm_mark(cpu_mm))) - goto commit_done; + /* Arch will take care of cache invalidation */ + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) { + /* Race with check_discard_rss_cache */ + if (cpu_mm != cmpxchg(this_cpu_ptr(&cpu_rss_cache.mm), cpu_mm, + __pcp_rss_mm_mark(cpu_mm))) + goto commit_done; + } for (int i = 0; i < NR_MM_COUNTERS; i++) { count = this_cpu_read(cpu_rss_cache.count[i]); @@ -328,46 +344,54 @@ static void check_discard_rss_cache(struct mm_struct *mm) long cached_count[NR_MM_COUNTERS] = { 0 }; struct mm_struct *cpu_mm; - /* Invalidate the RSS cache on every CPU */ - for_each_possible_cpu(cpu) { - cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); - if (__pcp_rss_mm_unmark(cpu_mm) != mm) - continue; - - /* - * If not being flusehd, try read-in the counter and mark it NULL, - * once cache's mm is set NULL, counter are considered invalided - */ - if (cpu_mm != __pcp_rss_mm_mark(cpu_mm)) { - long count[NR_MM_COUNTERS]; - - for (int i = 0; i < NR_MM_COUNTERS; i++) - count[i] = READ_ONCE(per_cpu(cpu_rss_cache.count[i], cpu)); + /* Arch will take care of cache invalidation */ + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) { + /* Invalidate the RSS cache on every CPU */ + for_each_possible_cpu(cpu) { + cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); + if (__pcp_rss_mm_unmark(cpu_mm) != mm) + continue; /* - * If successfully set to NULL, the owner CPU is not flushing it, counters - * are uncommiteed and untouched during this period, since a dying mm won't - * be accouted anymore + * If not being flusehd, try read-in the counter and mark it NULL, + * once cache's mm is set NULL, counter are considered invalided. */ - cpu_mm = cmpxchg(&per_cpu(cpu_rss_cache.mm, cpu), mm, NULL); - if (cpu_mm == mm) { + if (cpu_mm != __pcp_rss_mm_mark(cpu_mm)) { + long count[NR_MM_COUNTERS]; + for (int i = 0; i < NR_MM_COUNTERS; i++) - cached_count[i] += count[i]; - continue; + count[i] = READ_ONCE(per_cpu(cpu_rss_cache.count[i], cpu)); + + /* + * If successfully set to NULL, the owner CPU is not flushing it, + * counters are uncommitted and untouched during this period, since + * a dying mm won't be accouted anymore. + */ + cpu_mm = cmpxchg(&per_cpu(cpu_rss_cache.mm, cpu), mm, NULL); + if (cpu_mm == mm) { + for (int i = 0; i < NR_MM_COUNTERS; i++) + cached_count[i] += count[i]; + continue; + } } - } - /* It's being flushed, just busy wait as the critial section is really short */ - do { - cpu_relax(); - cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); - } while (cpu_mm == __pcp_rss_mm_mark(mm)); + /* + * It's being flushed, just busy wait as the critial section + * is really short. + */ + do { + cpu_relax(); + cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); + } while (cpu_mm == __pcp_rss_mm_mark(mm)); + } } for (int i = 0; i < NR_MM_COUNTERS; i++) { long val = atomic_long_read(&mm->rss_stat.count[i]); - val += cached_count[i]; + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) { + val += cached_count[i]; + } if (unlikely(val)) { pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n",