From patchwork Thu Jul 28 20:45:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA087C3F6B0 for ; Thu, 28 Jul 2022 20:45:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED21D6B0072; Thu, 28 Jul 2022 16:45:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E32D86B0073; Thu, 28 Jul 2022 16:45:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5FAE8E0001; Thu, 28 Jul 2022 16:45:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B730B6B0072 for ; Thu, 28 Jul 2022 16:45:22 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8C6A5140D16 for ; Thu, 28 Jul 2022 20:45:22 +0000 (UTC) X-FDA: 79737688884.16.3B4069F Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf30.hostedemail.com (Postfix) with ESMTP id 1DF3B800D4 for ; Thu, 28 Jul 2022 20:45:21 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id d10so2892530pfd.9 for ; Thu, 28 Jul 2022 13:45:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=ycW0mhERWN+p0KVMgOpcoUTig1KmOvE2fu+KZQruV5Q=; b=enzG2ODUUsfYcRd6l9GSOTO9h/lNV8Ef4uIhHU6yji7Llr1yXRk3yu8Jvp3eGFtyvk GuacXnsG0xrlW8P8MKMl3E3zGDpphKxKDGWUicOyhyzhjvNKOJ5O/+6eGpDxn0PicZ1V TRa/lGa/fMEZsrkrEAVncXaTZo1oYW9zat+vY1xOm2f+7x+0Oxd/wuCtuHT+++nhi5fN P3aqzJfa/8dpzuEeqaT1QfwnZSq1teXtWEgL6iLWtf1zsU83oj0jmVJPCWHBGvdXSPiz oUzYdURjUrByQA5sleCoOy4v8itR0JknzX0G/p8AaYur1CAJ5lz1Jzn1s2MjJIRXXode FoXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=ycW0mhERWN+p0KVMgOpcoUTig1KmOvE2fu+KZQruV5Q=; b=0LnML321tL/B9fKi6OJMtpW281Ae/2cZHNHrrvygqgvKkUFYuJkhwNyxWbSzRgec6i OO3nx0GyNJ9Xqdo7uzLXWhaAhFZHvO3B4o5x5BJsRg1ZgmreeOWFNSZzk+5qLwz8h3Cw afS8xpj5nPjiqRHHO3i6JeU/aAg+t9RNFgbPYSxFYLyqKSew40i+oI0JQ2spzFKwMOdP Kxtc7rLiuBaMnGdmMJj9m8WWFpFfhiMBunkAZvkBgIeBiAOAfcAfTSHaDrBgHHUlIf3J SSDgdRgqLoFFx5SmHHFyKmF03KzQeazFOvtsqqNdjVfpNWPBEFdcE9FHb4CYKKrHraGj 3Jgw== X-Gm-Message-State: AJIora/6euAQ+51E/ZK2ImJeDGxlsDbdgUkD+mYlZGyuNmxwc4Krqy9V D/hJMsVGozbRZEnWUTFul0c8o0vcA6HHDB+yCoY= X-Google-Smtp-Source: AGRyM1tYmO8wnuamcnLViWW3Uo2TdiQiGqhQMiU8SvJkvB0urbOMvzbIUQjzsRT1aMImvlLKOkCoyQ== X-Received: by 2002:a63:b50b:0:b0:412:b42c:6940 with SMTP id y11-20020a63b50b000000b00412b42c6940mr406365pge.460.1659041120407; Thu, 28 Jul 2022 13:45:20 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:19 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 1/7] mm: remove the per-task RSS counter cache Date: Fri, 29 Jul 2022 04:45:05 +0800 Message-Id: <20220728204511.56348-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=enzG2ODU; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041122; a=rsa-sha256; cv=none; b=lq9jDNQZRYccfUaZzWa0r1hn35pSLHrxoSCTmlcO0JthG29zwQu5C3FvCFk87JrBgXV9C9 wPbuce+hg9bso7S1cNcAbv1alv4h8bhL5fFWdL19GTk1M1VjEz7fJMVHvNH8hxvCfKEt34 cYA+FmKB2K0XwYgquA434hh1h6+ByQg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041122; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ycW0mhERWN+p0KVMgOpcoUTig1KmOvE2fu+KZQruV5Q=; b=3gl/Mg9dML/RLHt3pKNCq7tS/9tNfsNYqWGn1P8FljcrLIG5grTE4SnrTNZ+SCEYXi+KxK 0Juj/KzTF9REgWg8xS7s37HF4hCEwROzYl4YQGCj9YZjMPIgZZ5TuJUXlNO19jIwS5xUQz R6f3H5hABlBwDZnLZZS88UBKCf7z984= X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 1DF3B800D4 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=enzG2ODU; spf=pass (imf30.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: gyrdm8crehedqr1tft1uaqmegi7bhn77 X-Rspam-User: X-HE-Tag: 1659041121-234324 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song The RSS counter cached as introduced in commit 34e55232e59f ("mm: avoid false sharing of mm_counter") to ease the contention of the RSS counters of a mm_struct. There are several problems with this, and the 64 events threshold might not be an optimal value. It makes the RSS value inaccurate, in the worst case, RSS value is not accounted until 64 pages are allocated. With common tools like `top`, there could be hundreds of MBs of error of the RSS value being reported by kernel. And since 4 counters share the same event threshold, in the worst case, each counter will do a global sync every 16 events, which still raises some contention. Remove this cache for now, and prepare for a different approach. Some helper macros are kept since they will come in handy later. Signed-off-by: Kairui Song --- Documentation/filesystems/proc.rst | 7 ----- fs/exec.c | 2 -- include/linux/mm.h | 20 +----------- include/linux/mm_types_task.h | 9 ------ include/linux/sched.h | 3 -- kernel/exit.c | 5 --- kernel/fork.c | 4 --- kernel/kthread.c | 1 - mm/madvise.c | 7 ++--- mm/memory.c | 49 ------------------------------ 10 files changed, 3 insertions(+), 104 deletions(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 1bc91fb8c321..04a0a18da262 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -224,13 +224,6 @@ memory usage. Its seven fields are explained in Table 1-3. The stat file contains detailed information about the process itself. Its fields are explained in Table 1-4. -(for SMP CONFIG users) - -For making accounting scalable, RSS related information are handled in an -asynchronous manner and the value may not be very precise. To see a precise -snapshot of a moment, you can see /proc//smaps file and scan page table. -It's slow but very precise. - .. table:: Table 1-2: Contents of the status files (as of 4.19) ========================== =================================================== diff --git a/fs/exec.c b/fs/exec.c index 778123259e42..3c787ca8c68e 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -988,8 +988,6 @@ static int exec_mmap(struct mm_struct *mm) tsk = current; old_mm = current->mm; exec_mm_release(tsk, old_mm); - if (old_mm) - sync_mm_rss(old_mm); ret = down_write_killable(&tsk->signal->exec_update_lock); if (ret) diff --git a/include/linux/mm.h b/include/linux/mm.h index cf3d0d673f6b..6346f7e77dc7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1998,17 +1998,7 @@ static inline bool get_user_page_fast_only(unsigned long addr, */ static inline unsigned long get_mm_counter(struct mm_struct *mm, int member) { - long val = atomic_long_read(&mm->rss_stat.count[member]); - -#ifdef SPLIT_RSS_COUNTING - /* - * counter is updated in asynchronous manner and may go to minus. - * But it's never be expected number for users. - */ - if (val < 0) - val = 0; -#endif - return (unsigned long)val; + return atomic_long_read(&mm->rss_stat.count[member]); } void mm_trace_rss_stat(struct mm_struct *mm, int member, long count); @@ -2094,14 +2084,6 @@ static inline void setmax_mm_hiwater_rss(unsigned long *maxrss, *maxrss = hiwater_rss; } -#if defined(SPLIT_RSS_COUNTING) -void sync_mm_rss(struct mm_struct *mm); -#else -static inline void sync_mm_rss(struct mm_struct *mm) -{ -} -#endif - #ifndef CONFIG_ARCH_HAS_PTE_SPECIAL static inline int pte_special(pte_t pte) { diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index c1bc6731125c..a00327c663db 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -48,15 +48,6 @@ enum { NR_MM_COUNTERS }; -#if USE_SPLIT_PTE_PTLOCKS && defined(CONFIG_MMU) -#define SPLIT_RSS_COUNTING -/* per-thread cached information, */ -struct task_rss_stat { - int events; /* for synchronization threshold */ - int count[NR_MM_COUNTERS]; -}; -#endif /* USE_SPLIT_PTE_PTLOCKS */ - struct mm_rss_stat { atomic_long_t count[NR_MM_COUNTERS]; }; diff --git a/include/linux/sched.h b/include/linux/sched.h index c46f3a63b758..11d3e1a95302 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -861,9 +861,6 @@ struct task_struct { /* Per-thread vma caching: */ struct vmacache vmacache; -#ifdef SPLIT_RSS_COUNTING - struct task_rss_stat rss_stat; -#endif int exit_state; int exit_code; int exit_signal; diff --git a/kernel/exit.c b/kernel/exit.c index 64c938ce36fe..8c55cda5136f 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -482,7 +482,6 @@ static void exit_mm(void) exit_mm_release(current, mm); if (!mm) return; - sync_mm_rss(mm); mmap_read_lock(mm); mmgrab(mm); BUG_ON(mm != current->active_mm); @@ -749,10 +748,6 @@ void __noreturn do_exit(long code) io_uring_files_cancel(); exit_signals(tsk); /* sets PF_EXITING */ - - /* sync mm's RSS info before statistics gathering */ - if (tsk->mm) - sync_mm_rss(tsk->mm); acct_update_integrals(tsk); group_dead = atomic_dec_and_test(&tsk->signal->live); if (group_dead) { diff --git a/kernel/fork.c b/kernel/fork.c index 9d44f2d46c69..c090ebd55063 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2145,10 +2145,6 @@ static __latent_entropy struct task_struct *copy_process( p->io_uring = NULL; #endif -#if defined(SPLIT_RSS_COUNTING) - memset(&p->rss_stat, 0, sizeof(p->rss_stat)); -#endif - p->default_timer_slack_ns = current->timer_slack_ns; #ifdef CONFIG_PSI diff --git a/kernel/kthread.c b/kernel/kthread.c index 3c677918d8f2..6bfbab4e2103 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1463,7 +1463,6 @@ void kthread_unuse_mm(struct mm_struct *mm) * clearing tsk->mm. */ smp_mb__after_spinlock(); - sync_mm_rss(mm); local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); diff --git a/mm/madvise.c b/mm/madvise.c index 0316bbc6441b..48cb9e5f92d2 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -711,12 +711,9 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, mark_page_lazyfree(page); } out: - if (nr_swap) { - if (current->mm == mm) - sync_mm_rss(mm); - + if (nr_swap) add_mm_counter(mm, MM_SWAPENTS, nr_swap); - } + arch_leave_lazy_mmu_mode(); pte_unmap_unlock(orig_pte, ptl); cond_resched(); diff --git a/mm/memory.c b/mm/memory.c index 4cf7d4b6c950..6bf7826e666b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -176,53 +176,9 @@ void mm_trace_rss_stat(struct mm_struct *mm, int member, long count) trace_rss_stat(mm, member, count); } -#if defined(SPLIT_RSS_COUNTING) - -void sync_mm_rss(struct mm_struct *mm) -{ - int i; - - for (i = 0; i < NR_MM_COUNTERS; i++) { - if (current->rss_stat.count[i]) { - add_mm_counter(mm, i, current->rss_stat.count[i]); - current->rss_stat.count[i] = 0; - } - } - current->rss_stat.events = 0; -} - -static void add_mm_counter_fast(struct mm_struct *mm, int member, int val) -{ - struct task_struct *task = current; - - if (likely(task->mm == mm)) - task->rss_stat.count[member] += val; - else - add_mm_counter(mm, member, val); -} -#define inc_mm_counter_fast(mm, member) add_mm_counter_fast(mm, member, 1) -#define dec_mm_counter_fast(mm, member) add_mm_counter_fast(mm, member, -1) - -/* sync counter once per 64 page faults */ -#define TASK_RSS_EVENTS_THRESH (64) -static void check_sync_rss_stat(struct task_struct *task) -{ - if (unlikely(task != current)) - return; - if (unlikely(task->rss_stat.events++ > TASK_RSS_EVENTS_THRESH)) - sync_mm_rss(task->mm); -} -#else /* SPLIT_RSS_COUNTING */ - #define inc_mm_counter_fast(mm, member) inc_mm_counter(mm, member) #define dec_mm_counter_fast(mm, member) dec_mm_counter(mm, member) -static void check_sync_rss_stat(struct task_struct *task) -{ -} - -#endif /* SPLIT_RSS_COUNTING */ - /* * Note: this doesn't free the actual pages themselves. That * has been handled earlier when unmapping all the memory regions. @@ -502,8 +458,6 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) { int i; - if (current->mm == mm) - sync_mm_rss(mm); for (i = 0; i < NR_MM_COUNTERS; i++) if (rss[i]) add_mm_counter(mm, i, rss[i]); @@ -5120,9 +5074,6 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, count_vm_event(PGFAULT); count_memcg_event_mm(vma->vm_mm, PGFAULT); - /* do counter updates before entering really critical section. */ - check_sync_rss_stat(current); - if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE, flags & FAULT_FLAG_INSTRUCTION, flags & FAULT_FLAG_REMOTE)) From patchwork Thu Jul 28 20:45:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E641DC19F2B for ; Thu, 28 Jul 2022 20:45:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 647EE6B0073; Thu, 28 Jul 2022 16:45:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CFD96B0074; Thu, 28 Jul 2022 16:45:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35E0B8E0001; Thu, 28 Jul 2022 16:45:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1FEF56B0073 for ; Thu, 28 Jul 2022 16:45:25 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EC141ABEFE for ; Thu, 28 Jul 2022 20:45:24 +0000 (UTC) X-FDA: 79737688968.21.CA6C7FE Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf17.hostedemail.com (Postfix) with ESMTP id E1ACC4002C for ; Thu, 28 Jul 2022 20:45:23 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id w10so2838702plq.0 for ; Thu, 28 Jul 2022 13:45:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=fEJL5O/8Mtexvm+J+r1cHRwPR897J/MIFhWHBeDpdlE=; b=LJO5mQsP4juN5xvNbTYM56FRLCmFDkgErNYmMEF/d4UxE93cf83tbCdmy5z9y3tTVw zvtIXcmCccQhzroNCgyJKAXd9xJJQsAEqwOqLtqm9LlRTlYodX54TFGQXZVrPzhKisjf ChA407NCENtAqKPBqnbCxq/18lGmRNyh0MQK7F5s22aUwo2e0C6+KqbziljzayQo10gM TF/fBk6+QiGIeXVyuO+eA5L3yFx/xmDbeudT6Ro6V7LI7awXDV/TljUZmi1Q/LkVlJUZ h/FvZe9wQP0L7CXI7zUqjl+XOisurJZgww1fpxRVLGX4AZDvUy5D45vYb6oeAUKy0Orr varA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=fEJL5O/8Mtexvm+J+r1cHRwPR897J/MIFhWHBeDpdlE=; b=FzREJVDtAe8iz6RGiLm+3ScvPPDktUMm5Z0HXqjTjoW3UtSUmm4ooRcUSuF0NzPpJt wS5a2I4Go0aqoxpUsBrW3XY10drnG8jbuIULOVtzpZ85TSsWzLLlTDP6rQXQftg3oHVY IoFGfrR8HO+uQ73sr2mlYlgORcvD9w9qDbMxZGxai2r7bHPYavPMofNqFSBi5C6bFd7g /sTOz2ZIXM9BXUmGq3hdkGFt4NMH2DBKysvhW0dYkX4C/6YjtZBHQPMMMvwbYIQc2U0n S1mfMEJ+L9Ucr5G7rvRH5KDQHhkSoZpXU8Y+PSB3JaWInDXp4OWoOipXJIPq26K2t8Zb ZiiA== X-Gm-Message-State: ACgBeo2AEVa2Z3Tw08zdgkDAHotnLmsoHjUIFwF7GvVW8nuucCrreDM2 1o5vWYHHz9wg4Ufm8uKiyKIZO9BaHZ2TqRrWXy4= X-Google-Smtp-Source: AA6agR7gaJDHwuNty0e2Q772Que215+AfrAqz2a4Nol+uHZaE2IVzWbEJW8m+7P6ryUuMNqvlr67dA== X-Received: by 2002:a17:90b:4c03:b0:1f2:b977:c64e with SMTP id na3-20020a17090b4c0300b001f2b977c64emr473596pjb.211.1659041122887; Thu, 28 Jul 2022 13:45:22 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:22 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 2/7] mm: move check_mm to memory.c Date: Fri, 29 Jul 2022 04:45:06 +0800 Message-Id: <20220728204511.56348-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LJO5mQsP; spf=pass (imf17.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041124; a=rsa-sha256; cv=none; b=oe1GqEBq76EYvCGGy8TOrZkg8Wb5+PkBdGk0PYPxQbWhyKN/IwB/rDM89v0Q7XW+aEsvbN CEZbRFu7yxUPeXaFle97S9xWRiSpkuEiFuyBXnDl/DHA0lmhUMhx0P/xRucH7S0VyRRFc6 f6L5vl4Ffkgn/Lr0GXyraaxrnPcmgGE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041124; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fEJL5O/8Mtexvm+J+r1cHRwPR897J/MIFhWHBeDpdlE=; b=ziQRGHkpyug9OZ25NtSG7sFDgXPk4T92jNVSrsxWeKffUa54Kzt/I8HgRu11jbSDqXBz6j cfO6WYnN99zvAI0GGTsumIFxlt12N+wB4LUAvzEwuYiklvn6G09kOiujDpsDfhkGPjGJr5 Nk3QdavEnBEpWLPxV7LNN4oGhz/Efzk= X-Stat-Signature: zg7eji4q3r98ffsy658gcxosdxg41zh8 X-Rspamd-Queue-Id: E1ACC4002C X-Rspam-User: Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=LJO5mQsP; spf=pass (imf17.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam12 X-HE-Tag: 1659041123-403446 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song No function change, make it possible to do extra mm operation on mm exit, prepare for following commits. Signed-off-by: Kairui Song --- include/linux/mm.h | 3 +++ kernel/fork.c | 33 --------------------------------- mm/memory.c | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 33 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6346f7e77dc7..81ad91621078 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1993,6 +1993,9 @@ static inline bool get_user_page_fast_only(unsigned long addr, { return get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1; } + +void check_mm(struct mm_struct *mm); + /* * per-process(per-mm_struct) statistics. */ diff --git a/kernel/fork.c b/kernel/fork.c index c090ebd55063..86a239772208 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -128,15 +128,6 @@ int nr_threads; /* The idle threads do not count.. */ static int max_threads; /* tunable limit on nr_threads */ -#define NAMED_ARRAY_INDEX(x) [x] = __stringify(x) - -static const char * const resident_page_types[] = { - NAMED_ARRAY_INDEX(MM_FILEPAGES), - NAMED_ARRAY_INDEX(MM_ANONPAGES), - NAMED_ARRAY_INDEX(MM_SWAPENTS), - NAMED_ARRAY_INDEX(MM_SHMEMPAGES), -}; - DEFINE_PER_CPU(unsigned long, process_counts) = 0; __cacheline_aligned DEFINE_RWLOCK(tasklist_lock); /* outer */ @@ -748,30 +739,6 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) #define mm_free_pgd(mm) #endif /* CONFIG_MMU */ -static void check_mm(struct mm_struct *mm) -{ - int i; - - BUILD_BUG_ON_MSG(ARRAY_SIZE(resident_page_types) != NR_MM_COUNTERS, - "Please make sure 'struct resident_page_types[]' is updated as well"); - - for (i = 0; i < NR_MM_COUNTERS; i++) { - long x = atomic_long_read(&mm->rss_stat.count[i]); - - if (unlikely(x)) - pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", - mm, resident_page_types[i], x); - } - - if (mm_pgtables_bytes(mm)) - pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", - mm_pgtables_bytes(mm)); - -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS - VM_BUG_ON_MM(mm->pmd_huge_pte, mm); -#endif -} - #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) diff --git a/mm/memory.c b/mm/memory.c index 6bf7826e666b..c0597214f9b3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -179,6 +179,38 @@ void mm_trace_rss_stat(struct mm_struct *mm, int member, long count) #define inc_mm_counter_fast(mm, member) inc_mm_counter(mm, member) #define dec_mm_counter_fast(mm, member) dec_mm_counter(mm, member) +#define NAMED_ARRAY_INDEX(x) [x] = __stringify(x) +static const char * const resident_page_types[] = { + NAMED_ARRAY_INDEX(MM_FILEPAGES), + NAMED_ARRAY_INDEX(MM_ANONPAGES), + NAMED_ARRAY_INDEX(MM_SWAPENTS), + NAMED_ARRAY_INDEX(MM_SHMEMPAGES), +}; + +void check_mm(struct mm_struct *mm) +{ + int i; + + BUILD_BUG_ON_MSG(ARRAY_SIZE(resident_page_types) != NR_MM_COUNTERS, + "Please make sure 'struct resident_page_types[]' is updated as well"); + + for (i = 0; i < NR_MM_COUNTERS; i++) { + long x = atomic_long_read(&mm->rss_stat.count[i]); + + if (unlikely(x)) + pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", + mm, resident_page_types[i], x); + } + + if (mm_pgtables_bytes(mm)) + pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", + mm_pgtables_bytes(mm)); + +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS + VM_BUG_ON_MM(mm->pmd_huge_pte, mm); +#endif +} + /* * Note: this doesn't free the actual pages themselves. That * has been handled earlier when unmapping all the memory regions. From patchwork Thu Jul 28 20:45:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4BBDC04A68 for ; Thu, 28 Jul 2022 20:45:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 227BD6B0074; Thu, 28 Jul 2022 16:45:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B1296B0075; Thu, 28 Jul 2022 16:45:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA8D18E0001; Thu, 28 Jul 2022 16:45:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D48EA6B0074 for ; Thu, 28 Jul 2022 16:45:26 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B1E84C0EC8 for ; Thu, 28 Jul 2022 20:45:26 +0000 (UTC) X-FDA: 79737689052.05.125BEDB Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf24.hostedemail.com (Postfix) with ESMTP id 547B1180026 for ; Thu, 28 Jul 2022 20:45:26 +0000 (UTC) Received: by mail-pj1-f47.google.com with SMTP id pw15so2872129pjb.3 for ; Thu, 28 Jul 2022 13:45:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=3VreCZyHHh+WzAldTMIrv+f0+ZRGj5qK/tBAl8Rh9YU=; b=FwRN+Dz/EkdantL+2MuehBxRu34vE2XcB9JJwAKV1+CF9A8W4YNymfzAbHnrbk+L2U fR4wNZ+kuA+iAYL+rFBDO2riXI3h0eYhrjVGXQyktY2XHPh3TfoRUdsV8h5ekA0eKkdv z7kpAxEMJG2bbBCVPlhHWuM7BvjkVok8aSEVxBSDukYAzf+WgLda4xfSw/OT3A291gdg ou48O7q9cfHkv/RQWombHOHxI6eODfqLFNqZtaJQOzuI2qjxcEstXuCibnuZX36hWZMm e5EfiwYzC/IamfAWRITnYup9JSvjEVdahnDfoxZD7ktMOsUQNCdQme/sfeOx+rupiVkJ +jPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=3VreCZyHHh+WzAldTMIrv+f0+ZRGj5qK/tBAl8Rh9YU=; b=xVfTL1ljn0F2orfMPwqi+hgf8GhjQlFkwzz0Dxz/CFkMd+XYUvuo9aOxKZ8L9rFlHO 98PJNLnK1qfHhNfJIJ4GibcUbs0GF96d0aRt+qN0hnM3B1dLy9qTcUroOi495i84y4MU hxyffFQ9HhbQxXIgdQS22YoiVCON3iKdtSsMW5eWmDVHip6VqCsHkAZ5wVE3FzIP/ZVA EZbIotwBgYUrFL40HObfuuovRUoVrMKl+tBLqKnBULPHJTlr3Tz8Z1OfdSlT9zT2Ymym sTllULr92F668V1+KUvUaqUPHc+WoWs9UqnUtUoaOfBCScTbdWeHvKUrAiiuA2ktqnia tTCw== X-Gm-Message-State: ACgBeo1mneSypEWJTioEdZIxarCidKZqZg/AlvcFoOnpeXHMfa+D9dlk BLJPpZag6bvTZHrvKRk4FW0Wesk5xDuZl0xyj3U= X-Google-Smtp-Source: AA6agR40gp1BRqPQFl5cX7Tmvlv/+x70OfYLiw3CrWsQmCd4n0R6k3976B5hXnfHCoAHyxuIlaWKiw== X-Received: by 2002:a17:90b:b03:b0:1f3:6fb:bd20 with SMTP id bf3-20020a17090b0b0300b001f306fbbd20mr1215431pjb.38.1659041125076; Thu, 28 Jul 2022 13:45:25 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:24 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 3/7] mm/headers: change emun order of MM_COUNTERS Date: Fri, 29 Jul 2022 04:45:07 +0800 Message-Id: <20220728204511.56348-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041126; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3VreCZyHHh+WzAldTMIrv+f0+ZRGj5qK/tBAl8Rh9YU=; b=eAZCgC+aXsVlkdKr0izP8IoAJiLEV1/h0ogKKTTTnicEUcrF4ScG9OlIMOX0Zpcnqe4C5e X8cG3mit4r56sQ2UYu5G5/IsHcZFVkwNFQnpW9UeFaYaSW+BVq0E0uBrbXwfI/FX4l35Q+ YXeuaIxkDlD29EsGJCyU2F9DERGvCB8= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="FwRN+Dz/"; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041126; a=rsa-sha256; cv=none; b=3NIGaJlXrUTKuW42/Vzwm5PqEVoQDLtzF+hsabzS1o3koU4Oh54uK/bh2eYnW0cB5AZ7op xCYOKUbIBYOyhh7e7rUFTDIxFIcv0RjgpaGK4GWU7LEYcw8IYGS2bYsCmNXK4FigXIao4b Zty/3HoTfWj5f9fhWhP2rEy5Jx4plGA= X-Stat-Signature: hgs1s8wj433qi5c1gbx66rfsi9ccecq8 X-Rspamd-Queue-Id: 547B1180026 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="FwRN+Dz/"; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1659041126-473334 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000507, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song get_rss reads MM_FILEPAGES, MM_ANONPAGES, MM_SHMEMPAGES. Make them continues so it's easier to read in a loop in following commits. Signed-off-by: Kairui Song --- include/linux/mm_types_task.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index a00327c663db..14182ded3fda 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -43,8 +43,8 @@ struct vmacache { enum { MM_FILEPAGES, /* Resident file mapping pages */ MM_ANONPAGES, /* Resident anonymous pages */ - MM_SWAPENTS, /* Anonymous swap entries */ MM_SHMEMPAGES, /* Resident shared memory pages */ + MM_SWAPENTS, /* Anonymous swap entries */ NR_MM_COUNTERS }; From patchwork Thu Jul 28 20:45:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A45CC04A68 for ; Thu, 28 Jul 2022 20:45:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 164D86B0075; Thu, 28 Jul 2022 16:45:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0ED6D8E0001; Thu, 28 Jul 2022 16:45:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E33B26B007B; Thu, 28 Jul 2022 16:45:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D20A96B0075 for ; Thu, 28 Jul 2022 16:45:29 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9FD6C81596 for ; Thu, 28 Jul 2022 20:45:29 +0000 (UTC) X-FDA: 79737689178.17.6675D33 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf09.hostedemail.com (Postfix) with ESMTP id 5CDFC14001F for ; Thu, 28 Jul 2022 20:45:29 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id w185so2916738pfb.4 for ; Thu, 28 Jul 2022 13:45:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=SGkgqy0QLVnzdid2tkzB0s+fC8K9ka6NmIFVVWD2xYk=; b=D9EKylMTSsghnjGXBfCgCbvVNT/Rb65B+zsOKJ1byGzpWm2z65aE1EXzzx/Scfhj1j y2tQeGMLlyvbAhkbfJKOXECF9lyzXm3NXCnFUVCnh3x/7JHguG9iSpiwcz3RI/SShalX lyU1nhy4uxZWrFwXZsD9kwF3TJ53txtAi8BkMTqVc9ZDh5/ahzqsPBKF6eywhir6Gn8J Z9z/xMTXBCiJscT764TuHzpDskjTwlhGIQqaP+/hDN6SLFhnLb7tvulx1GNuJbjfn8Ow tjZmi2oPiHV7UnpjBSfZB9reP9y//41DnRFPCWIG2YA466fl411OrgzxKV+/SCNaHwHn 55dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=SGkgqy0QLVnzdid2tkzB0s+fC8K9ka6NmIFVVWD2xYk=; b=oCLR57nX++2pHqFxvdpg79SXn6aj0ZJnci8Hrg0eDu3Na72VKoB3PfGfG+/G2Sm+wz 0fOjfjTpdgPYDk0WsrZ8W9zjrUSNuTWlRkQt0R5aMjuOv1BqHaz6re7OISGOJ2Xa3fN7 5mvBKy52oc3EeKfCrpNli72WzBIHQPj6J6cws6V1fTyeI/YQ0eYuVXdsfYzKQYT7UXJg rYhdvhDIbB0YrPRDWYnypP37r42B/hJqGBGXD32TrdlTYClqhxv5J9z03JEzr5Q4/wpA xh8V+jLLBy6LI2C8lXveSOTvQH3900sEUt2N9p6ZdH+z/BDAiF4OevzH2C760M83AV2B +8JQ== X-Gm-Message-State: AJIora9A+88pT+ADd0F1clGok/wH1KC9sy1WvVlOx6qm12t5Dqga2d7z KYPvyJR5BMXN/ZA+GbxUi0/cI/46PSSQzGgkYfo= X-Google-Smtp-Source: AGRyM1skRuRihGKUIV+KmimC7TjXwxVcHbd9Pp0zhfBbeZxDqTaXXoGkFBmLZFsjmEUj/eJ601FKUg== X-Received: by 2002:a63:5f56:0:b0:412:9907:ec0d with SMTP id t83-20020a635f56000000b004129907ec0dmr443467pgb.18.1659041127458; Thu, 28 Jul 2022 13:45:27 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:26 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 4/7] mm: introduce a generic per-CPU RSS cache Date: Fri, 29 Jul 2022 04:45:08 +0800 Message-Id: <20220728204511.56348-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=D9EKylMT; spf=pass (imf09.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041129; a=rsa-sha256; cv=none; b=KsiGScvkD3GTW4mcUakmPuptkGrZcf3th2wRZMxYwTxgRz6gaciJBKh93S0szpWkSkKrzJ nlw09aoM0sZaK45714yKd4FEfyTXJC3xf9uC5Q9XH8ZzmqeF9I/IjeRC9oVjBVjGcC8rM4 J1/u7egC+J3cC4+6hORl0+6uaJ4z0L4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041129; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SGkgqy0QLVnzdid2tkzB0s+fC8K9ka6NmIFVVWD2xYk=; b=Ju//p7qn7+O4XVRLMoRkSU8jD1uOtWiEcUrcguto7IttHW94HoYzdlL6y+iiPWJN5Lb+kH eBzdCKvjbTPb0kZJdvhI1SykzgxYLLOaZ0CGHXG4f622TAYUBEbrfgj8lCK9dfIDTM6fI5 2ZSycApZiN9PmsmzEJrWFm+UOKtzSrI= Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=D9EKylMT; spf=pass (imf09.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5CDFC14001F X-Stat-Signature: 9ouiufynp6hu7fyzfy8ouo9eg8fhux77 X-HE-Tag: 1659041129-63992 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song The RSS cache used to be a per-task cache, and it's batched into 64 events for each atomic sync. The problems is 64 events is too small for contention reducing, and too large for an accurate RSS accounting. This per-cpu RSS cache assumes one mm_struct tends to stay on the same CPU, so if the mm_struct be accounted matches current active_mm, keep the RSS accounting CPU local until the mm_struct is switched out, and do an atomic update only upon switch out. The fast path of CPU local RSS accounting is extremely lightweight, only set preemption off and then do a CPU local counter increase. One major effect is that now RSS reading is much more accurate than before, but also slower. It needs to iterate all possible CPUs that have cached the RSS and collect the un-committed caches. With a lockless reader design, this never blocks the RSS accounting fast path, which ensures a good updater performance. And considering RSS updating is much more common than reading, this should improve the performance overall. This CPU iteration can be avoided by using CPU mask to mark the CPUs that cached the mm_struct and only read from these CPUs. It can leverage the existing mm_cpumask used for TLB shootdown, this has to be done arch by arch in later commits. This commit provides a baseline version that works on all arch, but with a performance drop for RSS syncing upon read/invalidation. Signed-off-by: Kairui Song --- include/linux/mm.h | 15 +-- include/linux/mm_types_task.h | 38 +++++++ kernel/fork.c | 2 +- kernel/sched/core.c | 3 + mm/memory.c | 201 ++++++++++++++++++++++++++++++++-- 5 files changed, 236 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 81ad91621078..47b8552b1b04 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1994,15 +1994,13 @@ static inline bool get_user_page_fast_only(unsigned long addr, return get_user_pages_fast_only(addr, 1, gup_flags, pagep) == 1; } -void check_mm(struct mm_struct *mm); +void check_discard_mm(struct mm_struct *mm); /* * per-process(per-mm_struct) statistics. */ -static inline unsigned long get_mm_counter(struct mm_struct *mm, int member) -{ - return atomic_long_read(&mm->rss_stat.count[member]); -} +unsigned long get_mm_counter(struct mm_struct *mm, int member); +unsigned long get_mm_rss(struct mm_struct *mm); void mm_trace_rss_stat(struct mm_struct *mm, int member, long count); @@ -2042,13 +2040,6 @@ static inline int mm_counter(struct page *page) return mm_counter_file(page); } -static inline unsigned long get_mm_rss(struct mm_struct *mm) -{ - return get_mm_counter(mm, MM_FILEPAGES) + - get_mm_counter(mm, MM_ANONPAGES) + - get_mm_counter(mm, MM_SHMEMPAGES); -} - static inline unsigned long get_mm_hiwater_rss(struct mm_struct *mm) { return max(mm->hiwater_rss, get_mm_rss(mm)); diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index 14182ded3fda..d5d3fbece174 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -12,6 +12,7 @@ #include #include #include +#include #include @@ -52,6 +53,43 @@ struct mm_rss_stat { atomic_long_t count[NR_MM_COUNTERS]; }; +struct mm_rss_cache { + /* + * CPU local only variables, hot path for RSS caching. Readonly for other CPUs. + */ + unsigned long in_use; + long count[NR_MM_COUNTERS]; + + /* Avoid false sharing when other CPUs collect RSS counter */ + struct mm_struct *mm ____cacheline_aligned; + /* Avoid ABA problem and RSS being accounted for wrong mm */ + unsigned long sync_count; +}; + +/* lowest bit of *mm is never used, so use it as a syncing flag */ +#define RSS_CACHE_MM_SYNCING_MASK 1UL + +/* mark the mm as being synced on that cache */ +static __always_inline struct mm_struct *__pcp_rss_mm_mark(struct mm_struct *mm) +{ + unsigned long val = (unsigned long)mm; + + val |= RSS_CACHE_MM_SYNCING_MASK; + + return (struct mm_struct *) val; +} + +static __always_inline struct mm_struct *__pcp_rss_mm_unmark(struct mm_struct *mm) +{ + unsigned long val = (unsigned long)mm; + + val &= ~RSS_CACHE_MM_SYNCING_MASK; + + return (struct mm_struct *) val; +} + +void switch_pcp_rss_cache_no_irq(struct mm_struct *next_mm); + struct page_frag { struct page *page; #if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536) diff --git a/kernel/fork.c b/kernel/fork.c index 86a239772208..c2f5f6eef6a6 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -755,9 +755,9 @@ void __mmdrop(struct mm_struct *mm) mm_free_pgd(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); - check_mm(mm); put_user_ns(mm->user_ns); mm_pasid_drop(mm); + check_discard_mm(mm); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index da0bf6fe9ecd..11df67bb52ee 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5142,6 +5142,9 @@ context_switch(struct rq *rq, struct task_struct *prev, prepare_lock_switch(rq, next, rf); + /* Cache new active_mm */ + switch_pcp_rss_cache_no_irq(next->active_mm); + /* Here we just switch the register state and the stack. */ switch_to(prev, next, prev); barrier(); diff --git a/mm/memory.c b/mm/memory.c index c0597214f9b3..f00f302143b6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -176,8 +176,143 @@ void mm_trace_rss_stat(struct mm_struct *mm, int member, long count) trace_rss_stat(mm, member, count); } -#define inc_mm_counter_fast(mm, member) inc_mm_counter(mm, member) -#define dec_mm_counter_fast(mm, member) dec_mm_counter(mm, member) +static DEFINE_PER_CPU_SHARED_ALIGNED(struct mm_rss_cache, cpu_rss_cache); + +/* + * get_mm_counter and get_mm_rss try to read the RSS cache of each + * CPU that cached target mm. If the cache is flushed while being read, + * skip it. May lead to rare and little bit of accuracy loss, but flushed + * cache will surely be accounted in the next read. + */ +unsigned long get_mm_counter(struct mm_struct *mm, int member) +{ + int cpu; + long ret, update, sync_count; + + ret = atomic_long_read(&mm->rss_stat.count[member]); + for_each_possible_cpu(cpu) { + if (READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) != mm) + continue; + sync_count = READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)); + /* see smp_mb in switch_pcp_rss_cache_no_irq */ + smp_rmb(); + + update = READ_ONCE(per_cpu(cpu_rss_cache.count[member], cpu)); + + /* same as above */ + smp_rmb(); + if (READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)) == sync_count && + READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) == mm) + ret += update; + } + + if (ret < 0) + ret = 0; + + return ret; +} + +/* see comment for get_mm_counter */ +unsigned long get_mm_rss(struct mm_struct *mm) +{ + int cpu; + long ret, update, sync_count; + + ret = atomic_long_read(&mm->rss_stat.count[MM_FILEPAGES]), + + atomic_long_read(&mm->rss_stat.count[MM_ANONPAGES]), + + atomic_long_read(&mm->rss_stat.count[MM_SHMEMPAGES]); + + for_each_possible_cpu(cpu) { + if (READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) != mm) + continue; + sync_count = READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)); + /* see smp_mb in switch_pcp_rss_cache_no_irq */ + smp_rmb(); + + /* Reads MM_FILEPAGES, MM_ANONPAGES, MM_SHMEMPAGES */ + for (int i = MM_FILEPAGES; i < MM_SWAPENTS; i++) + update += READ_ONCE(per_cpu(cpu_rss_cache.count[i], cpu)); + + /* same as above */ + smp_rmb(); + if (READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)) == sync_count && + READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) == mm) + ret += update; + } + + if (ret < 0) + ret = 0; + + return ret; +} + +/* flush the rss cache of current CPU with IRQ disabled, and switch to new mm */ +void switch_pcp_rss_cache_no_irq(struct mm_struct *next_mm) +{ + long count; + struct mm_struct *cpu_mm; + + cpu_mm = this_cpu_read(cpu_rss_cache.mm); + if (cpu_mm == next_mm) + return; + + /* + * `in_use` counter is hold with preempt disabled, if non-zero, this would be a + * interrupt switching the mm, just ignore it. + */ + if (this_cpu_read(cpu_rss_cache.in_use)) + return; + + if (cpu_mm == NULL) + goto commit_done; + + /* Race with check_discard_rss_cache */ + if (cpu_mm != cmpxchg(this_cpu_ptr(&cpu_rss_cache.mm), cpu_mm, + __pcp_rss_mm_mark(cpu_mm))) + goto commit_done; + + for (int i = 0; i < NR_MM_COUNTERS; i++) { + count = this_cpu_read(cpu_rss_cache.count[i]); + if (count) + add_mm_counter(cpu_mm, i, count); + } + +commit_done: + for (int i = 0; i < NR_MM_COUNTERS; i++) + this_cpu_write(cpu_rss_cache.count[i], 0); + + /* + * For remote reading in get_mm_{rss,counter}, + * ensure new mm and sync counter have zero'ed counters + */ + smp_wmb(); + this_cpu_write(cpu_rss_cache.mm, next_mm); + this_cpu_inc(cpu_rss_cache.sync_count); +} + +static void add_mm_counter_fast(struct mm_struct *mm, int member, int val) +{ + /* + * Disable preempt so task is pinned, and the mm is pinned on this CPU + * since caller must be holding a reference. + */ + preempt_disable(); + this_cpu_inc(cpu_rss_cache.in_use); + + if (likely(mm == this_cpu_read(cpu_rss_cache.mm))) { + this_cpu_add(cpu_rss_cache.count[member], val); + this_cpu_dec(cpu_rss_cache.in_use); + /* Avoid the resched checking oveahead for fast path */ + preempt_enable_no_resched(); + } else { + this_cpu_dec(cpu_rss_cache.in_use); + preempt_enable_no_resched(); + add_mm_counter(mm, member, val); + } +} + +#define inc_mm_counter_fast(mm, member) add_mm_counter_fast(mm, member, 1) +#define dec_mm_counter_fast(mm, member) add_mm_counter_fast(mm, member, -1) #define NAMED_ARRAY_INDEX(x) [x] = __stringify(x) static const char * const resident_page_types[] = { @@ -187,20 +322,64 @@ static const char * const resident_page_types[] = { NAMED_ARRAY_INDEX(MM_SHMEMPAGES), }; -void check_mm(struct mm_struct *mm) +static void check_discard_rss_cache(struct mm_struct *mm) { - int i; + int cpu; + long cached_count[NR_MM_COUNTERS] = { 0 }; + struct mm_struct *cpu_mm; - BUILD_BUG_ON_MSG(ARRAY_SIZE(resident_page_types) != NR_MM_COUNTERS, - "Please make sure 'struct resident_page_types[]' is updated as well"); + /* Invalidate the RSS cache on every CPU */ + for_each_possible_cpu(cpu) { + cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); + if (__pcp_rss_mm_unmark(cpu_mm) != mm) + continue; + + /* + * If not being flusehd, try read-in the counter and mark it NULL, + * once cache's mm is set NULL, counter are considered invalided + */ + if (cpu_mm != __pcp_rss_mm_mark(cpu_mm)) { + long count[NR_MM_COUNTERS]; - for (i = 0; i < NR_MM_COUNTERS; i++) { - long x = atomic_long_read(&mm->rss_stat.count[i]); + for (int i = 0; i < NR_MM_COUNTERS; i++) + count[i] = READ_ONCE(per_cpu(cpu_rss_cache.count[i], cpu)); - if (unlikely(x)) + /* + * If successfully set to NULL, the owner CPU is not flushing it, counters + * are uncommiteed and untouched during this period, since a dying mm won't + * be accouted anymore + */ + cpu_mm = cmpxchg(&per_cpu(cpu_rss_cache.mm, cpu), mm, NULL); + if (cpu_mm == mm) { + for (int i = 0; i < NR_MM_COUNTERS; i++) + cached_count[i] += count[i]; + continue; + } + } + + /* It's being flushed, just busy wait as the critial section is really short */ + do { + cpu_relax(); + cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); + } while (cpu_mm == __pcp_rss_mm_mark(mm)); + } + + for (int i = 0; i < NR_MM_COUNTERS; i++) { + long val = atomic_long_read(&mm->rss_stat.count[i]); + + val += cached_count[i]; + + if (unlikely(val)) { pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", - mm, resident_page_types[i], x); + mm, resident_page_types[i], val); + } } +} + +void check_discard_mm(struct mm_struct *mm) +{ + BUILD_BUG_ON_MSG(ARRAY_SIZE(resident_page_types) != NR_MM_COUNTERS, + "Please make sure 'struct resident_page_types[]' is updated as well"); if (mm_pgtables_bytes(mm)) pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", @@ -209,6 +388,8 @@ void check_mm(struct mm_struct *mm) #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS VM_BUG_ON_MM(mm->pmd_huge_pte, mm); #endif + + check_discard_rss_cache(mm); } /* From patchwork Thu Jul 28 20:45:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08269C19F2A for ; Thu, 28 Jul 2022 20:45:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97BC76B0078; Thu, 28 Jul 2022 16:45:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B77D6B007B; Thu, 28 Jul 2022 16:45:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BBDB6B007D; Thu, 28 Jul 2022 16:45:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 538A46B0078 for ; Thu, 28 Jul 2022 16:45:31 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 256F8120F91 for ; Thu, 28 Jul 2022 20:45:31 +0000 (UTC) X-FDA: 79737689262.13.A641651 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf29.hostedemail.com (Postfix) with ESMTP id C1C4212009A for ; Thu, 28 Jul 2022 20:45:30 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id bh13so2439660pgb.4 for ; Thu, 28 Jul 2022 13:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=PxrkjbDitHfD1d97Y8UhQKa79eOpeGRQUks9O/4b63c=; b=n16KoPlvBzd95MUn2w14Fg7+fI/TRVKK4xnSvA1Ck0fepbOeZl/M1zr2wzFKrwzRoq hgiUJNCFAhbYqljsgcuOFgUMTqg2+GGDVdxORVxkRtRM5ukuktVVziFfmfOiI1dDDILQ n308kWjnEH/SwP73po2piDXPGtObpbKY0zQLIu0bNzlHTs+TIt4545L7oDmfDKOmsb0C XTM7XFMODx5xPDR18Ao7XwDiOQ51/ooNyCvekRdjy0WNutaBFsNfvk+XVar9aHhoib09 FfYr3yRD0X8UgeNpPGScXUD9sMNcjHqLVvRQ4o3MhqABKodGl6F3wZKKcJ8vytC4mWjU LGDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=PxrkjbDitHfD1d97Y8UhQKa79eOpeGRQUks9O/4b63c=; b=OQT725Dsba8bP1X8waqhq7M95kmgIB65Umvg7bxMYKdZ2A55gOSF7JKOO7VDAEWb03 te1tUI1AHjAjiMejggQvyT1ynwUtu1DA0YE02XNyJZokDCT3fMrewndp0jg0t2xTuHVV Jrw66/Z/2RPPfTkr9mLjhZYWd3Ppqmucc7lXY7dV9yzEwzUtojxkZ3N44+H/d2lYzOXt cvOzwyy+8RTnlhov4Qu+/VAjNNde0003EMwhRCsS8hrz6bhfeGlG7mmnQQPBRqvWRaJw OVlujEb8qI/y5IiLQglWY0fJnlaAK/X9dkOXQW++CmOUW7yl7gcPdtt/DAuBKqiXeGE/ PR4Q== X-Gm-Message-State: AJIora863PvI3+w06Ndt7vzAgP/UFUbKwgbcFx5kbxVt32rtU+iEaxlQ fYYxHWTrVZcv41cR00mHvrHTFEVE/rMJoRpwgKo= X-Google-Smtp-Source: AGRyM1tXOVFNxoBFYbFBHG8eOFa0LJHrKzg+SUCMnYmaLeI2rBEw1bFu/iUgd6WGY5hnPF0K96m9AQ== X-Received: by 2002:a63:4004:0:b0:41b:64ff:7fe2 with SMTP id n4-20020a634004000000b0041b64ff7fe2mr419118pga.172.1659041129567; Thu, 28 Jul 2022 13:45:29 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:29 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 5/7] mm: try use fast path for pmd setting as well Date: Fri, 29 Jul 2022 04:45:09 +0800 Message-Id: <20220728204511.56348-6-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041130; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PxrkjbDitHfD1d97Y8UhQKa79eOpeGRQUks9O/4b63c=; b=C9Z1eCFr3mdoq6vjOvvcKojluzekadG1nSxXvKLqPnrfMP6zoFYyF5XESUrZ7aqZ+IrXKG IXAznbdCYvjBVMKvuYnow8JNG/osUVhN4n2S/IAppII7YXR7QM+WLifPzjqWAIQfYwSNNP RmP3xGvRwU2oyjKC/AxiiQM6br7MAEc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=n16KoPlv; spf=pass (imf29.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041130; a=rsa-sha256; cv=none; b=pweuE+jPAmG9Q4f5nshFdNoT+zabvuURwWPDANEqSG2NF8330v+5E5TA7kvR6sTI4O6t1S NfZkixLcMdxN4tJQ1afsahUEpUL9oOAKZe+yeOiwcj4FRCrBBf09KV6xfDjZ20fxD4gVrc scdWLOe6HJoeM/EW8a3MbOi9XRs/qjE= X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C1C4212009A X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=n16KoPlv; spf=pass (imf29.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: 61hnjwi8i8r6rpmiu9ay6ui95zrkf63i X-HE-Tag: 1659041130-772527 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000091, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song Use the per-CPU RSS cache helper as much as possible. Signed-off-by: Kairui Song --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index f00f302143b6..09d7d193da51 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4419,7 +4419,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) if (write) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); - add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); + add_mm_counter_fast(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); page_add_file_rmap(page, vma, true); /* From patchwork Thu Jul 28 20:45:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AABF9C19F2C for ; Thu, 28 Jul 2022 20:45:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3AEF96B007B; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 336C16B007D; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 162DF6B007E; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0558E6B007B for ; Thu, 28 Jul 2022 16:45:34 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CC78E1408E9 for ; Thu, 28 Jul 2022 20:45:33 +0000 (UTC) X-FDA: 79737689346.02.0EC11AF Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf06.hostedemail.com (Postfix) with ESMTP id DB360180093 for ; Thu, 28 Jul 2022 20:45:32 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id q16so2433201pgq.6 for ; Thu, 28 Jul 2022 13:45:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=o9jWwsLlYZnS171yrNbtFKKGeaY8IhQRPNlofd9rtnc=; b=PC9ISe/Tv2ntpg8FscV4ebolxpt+AhXtqalCdJ/HM/qaDIdf1rxD2vBYVhSyfRLxvQ GHNNXjMEXEikSN1uQJEUc4Zhe57ufRlQnqw9TudVYtELg/+5ehUxd4ndJQJ5R6PRGeeT r+ufm10YR1mOAqOW40t42Qr6bFNcdDvwRbzl9OkfnTCh+iEnGxt/DT6muqSIGo96CuOf et+sFqcvlWWFb9bXbnS+xT11zv2W1sj53OfXveVBtRe+8b7qrKDj27Z/dZd3xwMT1a5G exxkBIZstIbrCsALmdrvZOuGiqwEUnFgdG1pJx0C5F7hTIdLNEc4O30Ottfvj3r5Ihoq Gx8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=o9jWwsLlYZnS171yrNbtFKKGeaY8IhQRPNlofd9rtnc=; b=5viZ9xG2MD7Qj5cZPjXaXBTBmrQnGItMU7txqZuSIGgm9r984oVHvuwOT0MEtOl0sY l5W1kr+ro8q/X71jdl91AZiXGVbU/YEHLdc1il1CKDxDWaN3rxoqljGbwV/Vtel0Bzi5 A9pMhW0y1iA5m9qncAAtZxdLAyIJCblYZymfCiBETBxBxxHALhMrDavfL8bwwSnv0HNu Jc91Ios/EFXo3KYK1Sd1HBTKEGvJgWm+ghGKyJ5WODg8qALDQBfg3aTPqMn19ukY6Fm3 uW+Dg+GBhAnDJScPykSV2yAIZH8KTtJPtyfIpfU6TiSFBO+PE4tn7jP3hRXARK265+ag Am7w== X-Gm-Message-State: AJIora8foRRvpgZbqDWNhxigwLsh2g0ubEaZ141gSFn6C0Zuhr3oqG6f aDXsfq8otu7LfsS25L5G76rMJHkGlx1HY7W3md8= X-Google-Smtp-Source: AGRyM1vsVwYmFD3XfCQrIG8QN1BCM3jR8q0DPVN7U5JQfZ3X2kYLzuo6hZlQfhMP+vyClpJD2IEHiw== X-Received: by 2002:a05:6a00:1d26:b0:52b:f8ab:6265 with SMTP id a38-20020a056a001d2600b0052bf8ab6265mr340380pfx.54.1659041131938; Thu, 28 Jul 2022 13:45:31 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.30 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:31 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 6/7] mm: introduce CONFIG_ARCH_PCP_RSS_USE_CPUMASK Date: Fri, 29 Jul 2022 04:45:10 +0800 Message-Id: <20220728204511.56348-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="PC9ISe/T"; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041133; a=rsa-sha256; cv=none; b=vG6cAbBQ8lJorug+Xs3VT0vEZM9o+FxYsuwpdBzdmxIcz6yO8zfyvXUvGQvqlsdglHOfxe nHH7tPr22wWcxYTyjFYeGhuj94Uz91t/Rwb3mViGR8XiqUZbwG7dHvyAimcax9GF7yC0fq Y0UUyQHnzKiPIYirGlePz/zYf+aoe/Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041133; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o9jWwsLlYZnS171yrNbtFKKGeaY8IhQRPNlofd9rtnc=; b=cJk3pELgTXEdFUD2XxvX48trQTJKCIAUDhMaUgvHon7XkqQfbszWzPmeE9CgzGk5/RI7Ri gzhujI0fYmZ1+6ONsh7cfkV12WGzPaE10ZcTj2FOz3jrq7Zu9RVb1WqKmqJOGPHLKLtW/s tSZwsY3JdLp5XXHIFQohgt+gB5nnQnY= X-Rspamd-Queue-Id: DB360180093 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="PC9ISe/T"; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspam-User: X-Stat-Signature: 1z3xydjstxai1t3johz86kyrzuijreg7 X-HE-Tag: 1659041132-868781 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song If the arch related code can provide helpers to bind the RSS cache to mm_cpumask, then the syncing code can just rely on that instead of doing full CPU synchronization. This speed up the reading/mm_exit by a lot. Signed-off-by: Kairui Song --- arch/Kconfig | 3 ++ kernel/sched/core.c | 3 +- mm/memory.c | 94 ++++++++++++++++++++++++++++----------------- 3 files changed, 64 insertions(+), 36 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 71b9272acb28..8df45b6346ae 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1403,6 +1403,9 @@ config ARCH_HAS_ELFCORE_COMPAT config ARCH_HAS_PARANOID_L1D_FLUSH bool +config ARCH_PCP_RSS_USE_CPUMASK + bool + config DYNAMIC_SIGFRAME bool diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 11df67bb52ee..6f7991caf24b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5143,7 +5143,8 @@ context_switch(struct rq *rq, struct task_struct *prev, prepare_lock_switch(rq, next, rf); /* Cache new active_mm */ - switch_pcp_rss_cache_no_irq(next->active_mm); + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) + switch_pcp_rss_cache_no_irq(next->active_mm); /* Here we just switch the register state and the stack. */ switch_to(prev, next, prev); diff --git a/mm/memory.c b/mm/memory.c index 09d7d193da51..a819009aa3e0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -188,9 +188,16 @@ unsigned long get_mm_counter(struct mm_struct *mm, int member) { int cpu; long ret, update, sync_count; + const struct cpumask *mm_mask; ret = atomic_long_read(&mm->rss_stat.count[member]); - for_each_possible_cpu(cpu) { + + if (IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) + mm_mask = mm_cpumask(mm); + else + mm_mask = cpu_possible_mask; + + for_each_cpu(cpu, mm_mask) { if (READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) != mm) continue; sync_count = READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)); @@ -217,12 +224,18 @@ unsigned long get_mm_rss(struct mm_struct *mm) { int cpu; long ret, update, sync_count; + const struct cpumask *mm_mask; ret = atomic_long_read(&mm->rss_stat.count[MM_FILEPAGES]), + atomic_long_read(&mm->rss_stat.count[MM_ANONPAGES]), + atomic_long_read(&mm->rss_stat.count[MM_SHMEMPAGES]); - for_each_possible_cpu(cpu) { + if (IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) + mm_mask = mm_cpumask(mm); + else + mm_mask = cpu_possible_mask; + + for_each_cpu(cpu, mm_mask) { if (READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)) != mm) continue; sync_count = READ_ONCE(per_cpu(cpu_rss_cache.sync_count, cpu)); @@ -266,10 +279,13 @@ void switch_pcp_rss_cache_no_irq(struct mm_struct *next_mm) if (cpu_mm == NULL) goto commit_done; - /* Race with check_discard_rss_cache */ - if (cpu_mm != cmpxchg(this_cpu_ptr(&cpu_rss_cache.mm), cpu_mm, - __pcp_rss_mm_mark(cpu_mm))) - goto commit_done; + /* Arch will take care of cache invalidation */ + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) { + /* Race with check_discard_rss_cache */ + if (cpu_mm != cmpxchg(this_cpu_ptr(&cpu_rss_cache.mm), cpu_mm, + __pcp_rss_mm_mark(cpu_mm))) + goto commit_done; + } for (int i = 0; i < NR_MM_COUNTERS; i++) { count = this_cpu_read(cpu_rss_cache.count[i]); @@ -328,46 +344,54 @@ static void check_discard_rss_cache(struct mm_struct *mm) long cached_count[NR_MM_COUNTERS] = { 0 }; struct mm_struct *cpu_mm; - /* Invalidate the RSS cache on every CPU */ - for_each_possible_cpu(cpu) { - cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); - if (__pcp_rss_mm_unmark(cpu_mm) != mm) - continue; - - /* - * If not being flusehd, try read-in the counter and mark it NULL, - * once cache's mm is set NULL, counter are considered invalided - */ - if (cpu_mm != __pcp_rss_mm_mark(cpu_mm)) { - long count[NR_MM_COUNTERS]; - - for (int i = 0; i < NR_MM_COUNTERS; i++) - count[i] = READ_ONCE(per_cpu(cpu_rss_cache.count[i], cpu)); + /* Arch will take care of cache invalidation */ + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) { + /* Invalidate the RSS cache on every CPU */ + for_each_possible_cpu(cpu) { + cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); + if (__pcp_rss_mm_unmark(cpu_mm) != mm) + continue; /* - * If successfully set to NULL, the owner CPU is not flushing it, counters - * are uncommiteed and untouched during this period, since a dying mm won't - * be accouted anymore + * If not being flusehd, try read-in the counter and mark it NULL, + * once cache's mm is set NULL, counter are considered invalided. */ - cpu_mm = cmpxchg(&per_cpu(cpu_rss_cache.mm, cpu), mm, NULL); - if (cpu_mm == mm) { + if (cpu_mm != __pcp_rss_mm_mark(cpu_mm)) { + long count[NR_MM_COUNTERS]; + for (int i = 0; i < NR_MM_COUNTERS; i++) - cached_count[i] += count[i]; - continue; + count[i] = READ_ONCE(per_cpu(cpu_rss_cache.count[i], cpu)); + + /* + * If successfully set to NULL, the owner CPU is not flushing it, + * counters are uncommitted and untouched during this period, since + * a dying mm won't be accouted anymore. + */ + cpu_mm = cmpxchg(&per_cpu(cpu_rss_cache.mm, cpu), mm, NULL); + if (cpu_mm == mm) { + for (int i = 0; i < NR_MM_COUNTERS; i++) + cached_count[i] += count[i]; + continue; + } } - } - /* It's being flushed, just busy wait as the critial section is really short */ - do { - cpu_relax(); - cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); - } while (cpu_mm == __pcp_rss_mm_mark(mm)); + /* + * It's being flushed, just busy wait as the critial section + * is really short. + */ + do { + cpu_relax(); + cpu_mm = READ_ONCE(per_cpu(cpu_rss_cache.mm, cpu)); + } while (cpu_mm == __pcp_rss_mm_mark(mm)); + } } for (int i = 0; i < NR_MM_COUNTERS; i++) { long val = atomic_long_read(&mm->rss_stat.count[i]); - val += cached_count[i]; + if (!IS_ENABLED(CONFIG_ARCH_PCP_RSS_USE_CPUMASK)) { + val += cached_count[i]; + } if (unlikely(val)) { pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", From patchwork Thu Jul 28 20:45:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12931714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B64BC04A68 for ; Thu, 28 Jul 2022 20:45:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C8C16B007D; Thu, 28 Jul 2022 16:45:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 951116B007E; Thu, 28 Jul 2022 16:45:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A7018E0001; Thu, 28 Jul 2022 16:45:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 58D2A6B007D for ; Thu, 28 Jul 2022 16:45:36 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2C53E815A7 for ; Thu, 28 Jul 2022 20:45:36 +0000 (UTC) X-FDA: 79737689472.22.E7B037C Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf24.hostedemail.com (Postfix) with ESMTP id B1FB8180026 for ; Thu, 28 Jul 2022 20:45:35 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id h21-20020a17090aa89500b001f31a61b91dso3574727pjq.4 for ; Thu, 28 Jul 2022 13:45:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=gfO26u2xpyVmEWxjONFXEHf40ylM6r3kRxMGtINMS28=; b=CqUpKZVcHUdGhidYMYjokkBi17q5SK18jd7MhWe5pwlpQrL2nQqeN19YTR0WYDAFbP CMXhllFoexuWnVPLj4OetWCIXeI9nvG6zUlojn7oUojsxyNpgy7iF7r1rfVjsYa0ze71 xnb7t4vhiEViH6GAuV16Z31aSTVXUb5Gw00YwzsqcVrbtgZvjF3zYtKzugy+OFOVYce+ XTYTpiiV5Uqu8E6hZ+/VnjofZ9wyy0blnM4F2o3or+2R2ISS15tzVuSyLZVzFLexsber PTxe/D7DN/AFX9JYEBisHhtlBriUSIZuluLk2DWot1+vXvb6yB/2/At1UPMkYKaOwcLm PJmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=gfO26u2xpyVmEWxjONFXEHf40ylM6r3kRxMGtINMS28=; b=1HQtaC+5X8BQmgkyEg9X4X5MlwEEUUT23t9c8YCCH0Zd1D8zRC+kIz0vQH4xRB5rEl tGARQR+oJK6Flf6PG1sG5SvjgSIYT1/XrxoioSYCsEMtQReEPRhstkUWj9w45lAN8b0d UpTwgteCr120ZCoAe5jjHoI9U0CF0+jRmYA3O71U2nQTpLojhiXoIUW8JizZzKtK6Tx3 NqU9CnPpYyaMy33jrMTp0QghMFRtWTroZfzl1eE0DppRad/rtf3VwViaWRAWcbeoidO2 PKJOL+g2XNcJSZsy43tkV1LZnIQHQJMatEC3tVpmwiCJmfbTD2wIjdD06qzlIKEKI71d tcyQ== X-Gm-Message-State: ACgBeo1ZZFoIvggxD2GR5/D6+zlftpXQf8eqYmwVTT56Ori9+J4HBqL9 lGLNdAn9ACVv1bitR5GJ1Ks/H2mGRrm4hlidUOw= X-Google-Smtp-Source: AA6agR65b9BvIPJJ2aIA34xLNFyBah7YuqJvpKELG7QwV25Vml1sxFR036NxwJ4Xi5LcGcZNJFfljw== X-Received: by 2002:a17:90a:17e1:b0:1f2:2ff2:6cae with SMTP id q88-20020a17090a17e100b001f22ff26caemr1175172pja.196.1659041134124; Thu, 28 Jul 2022 13:45:34 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([114.254.3.190]) by smtp.gmail.com with ESMTPSA id 21-20020a170902c11500b0016c40f8cb58sm1787304pli.81.2022.07.28.13.45.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 28 Jul 2022 13:45:33 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Kairui Song Subject: [RFC PATCH 7/7] x86_64/tlb, mm: enable cpumask optimzation for RSS cache Date: Fri, 29 Jul 2022 04:45:11 +0800 Message-Id: <20220728204511.56348-8-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220728204511.56348-1-ryncsn@gmail.com> References: <20220728204511.56348-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=CqUpKZVc; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659041135; a=rsa-sha256; cv=none; b=f+HeOrk2ZmfNV7O2o99MYhMH3grHnkQ8Jelb7HphZEsn0rgOV/jUxddb/9Y1qIue+NEuF2 7ugd4zo9ihgB4ZyEyiT/Y7XrbeHLB2DoK99hv+0x4o2ZpaHCC/DY9HSH4vKGTCI5/5qyuE XN2GHZ0yG7dnJKX+SxwjmuVKq/yeXws= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659041135; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gfO26u2xpyVmEWxjONFXEHf40ylM6r3kRxMGtINMS28=; b=DH/0WtcHYdtAKZo2XTqKvTPlA0/1riKg42QNCTAPVfwAIS8kY8Fgr93gIRF8/wJbbiK4tD XfQTdRu+2aW6ZkwNmkozltlbeTcU6o9OF26QFi4dk19YU+UOYV88LzaGG8RfLwOqKKzpkz uJXwQ/NADedm3amqLx1OrUFPKDyaD6g= X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: B1FB8180026 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=CqUpKZVc; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: noaxnxj4mkpxyqng1abrocrd1ncx6r4e X-Rspam-User: X-HE-Tag: 1659041135-855747 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song Enable CONFIG_ARCH_PCP_RSS_USE_CPUMASK for x86_64, we do a RSS cache switch in switch_mm_irqs_off. On x86_64 this is the unified routine for switching a mm, so hook into this can make sure any dead mm will have their cache invalidated in time, and cpumask is synced with cache state. Signed-off-by: Kairui Song --- arch/x86/Kconfig | 1 + arch/x86/mm/tlb.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 52a7f91527fe..15e2b29ba972 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -125,6 +125,7 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH + select ARCH_PCP_RSS_USE_CPUMASK if X86_64 select BUILDTIME_TABLE_SORT select CLKEVT_I8253 select CLOCKSOURCE_VALIDATE_LAST_CYCLE diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index d400b6d9d246..614865f94d85 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -597,6 +597,11 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, */ cond_mitigation(tsk); + /* + * Flush RSS cache before clear up the bitmask + */ + switch_pcp_rss_cache_no_irq(next); + /* * Stop remote flushes for the previous mm. * Skip kernel threads; we never send init_mm TLB flushing IPIs,