From patchwork Fri Aug 4 06:18:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13341300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9654DC001DE for ; Fri, 4 Aug 2023 06:21:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9EE62802CB; Fri, 4 Aug 2023 02:21:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4E682802BC; Fri, 4 Aug 2023 02:21:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF0732802CB; Fri, 4 Aug 2023 02:21:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C025C2802BC for ; Fri, 4 Aug 2023 02:21:34 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 92A6F807D6 for ; Fri, 4 Aug 2023 06:21:34 +0000 (UTC) X-FDA: 81085425708.23.F12C566 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id 9B1DEC0012 for ; Fri, 4 Aug 2023 06:21:32 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691130093; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=mzZ9RiSNIr9O1RZI6ZVZJswq8auc+h32r3mxK6wrFzo=; b=HLvOOvSbUQGN3/LT7uFpkKlk3ipk3B0a5CASv5ldUJne9k0WY85oslWRZ4g5yuYzqFuexB WkrLB0GM3c21PyNpO8txUGnNQWea0f5eQfMmvHkJV3l9AroOCMh6zOnGvZCVNxPf1t2dXu eIGg0+oE45XI+41e+J8o5hn1JcRTt3c= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691130093; a=rsa-sha256; cv=none; b=DsBnndmJs5Gb6MkRMXunTzQcliwIGD9hrpyTJBdQrFSGZuwaW6ADhWwQn9hWfzjdFZG7ZB iVj2RUL8n8XWlUrQoQDITuv3yOQgJCk6ISBMvoeCoEtDjHez0oDvG4STL8OjrsYqIWhcj/ oKuryBKroA6OOWdipW7lkQDzvKiD5/w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d85ff70000001748-34-64cc98e6fe13 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, xhao@linux.alibaba.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com Subject: [RFC 1/2] mm/rmap: Recognize non-writable TLB entries during TLB batch flush Date: Fri, 4 Aug 2023 15:18:49 +0900 Message-Id: <20230804061850.21498-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230804061850.21498-1-byungchul@sk.com> References: <20230804061850.21498-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrMLMWRmVeSWpSXmKPExsXC9ZZnke6zGWdSDB7uMLKYs34Nm8XX9b+Y LZ5+6mOxuLxrDpvFvTX/WS12LN3HZHF910NGi98/gGJzplhZnJw1mcWBy2PBplKPzSu0PBbv ecnksenTJHaPEzN+s3jsfGjp8X7fVTaPrb/sPD5vkvN4N/8tWwBXFJdNSmpOZllqkb5dAlfG nVfTWQqO6VW0nHrI1sD4QK2LkZNDQsBEYuvyM2xdjBxg9qp2X5Awm4C6xI0bP5lBbBEBM4mD rX/Yuxi5OJgF7jBKHN7VyASSEBYIlfj05wYLiM0ioCqx5OEuNhCbV8BU4seffUwQ8+UlVm84 ADaIE2jQkYcvGEFsIaCae586wYZKCKxhk5j94SMrRIOkxMEVN1gmMPIuYGRYxSiUmVeWm5iZ Y6KXUZmXWaGXnJ+7iREYlMtq/0TvYPx0IfgQowAHoxIP74cDp1OEWBPLiitzDzFKcDArifBe ew8U4k1JrKxKLcqPLyrNSS0+xCjNwaIkzmv0rTxFSCA9sSQ1OzW1ILUIJsvEwSnVwCiX13Pq crbS6pKIeEXpW5fmxN00v/7W5lWlQizfWZftr1PjZ9oVFOjWbth64yE/62GeefE7nHedvDNh g2tRqUf+ea7t/F9KtQWvcu/sTFq5+l3jkb13mBi2Hav+JHuyKz+sQjA9SLLHj2tZw6rf0Vdj rJfvlzpnmK3D5T77PcurCQdmHxfcJqjEUpyRaKjFXFScCACoLYHnRgIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprFLMWRmVeSWpSXmKPExsXC5WfdrPtsxpkUgxcdmhZz1q9hs/i6/hez xdNPfSwWh+eeZLW4vGsOm8W9Nf9ZLXYs3cdkcX3XQ0aL3z+AYnOmWFmcnDWZxYHbY8GmUo/N K7Q8Fu95yeSx6dMkdo8TM36zeOx8aOnxft9VNo/FLz4weWz9ZefxeZOcx7v5b9kCuKO4bFJS czLLUov07RK4Mu68ms5ScEyvouXUQ7YGxgdqXYwcHBICJhKr2n27GDk52ATUJW7c+MkMYosI mEkcbP3D3sXIxcEscIdR4vCuRiaQhLBAqMSnPzdYQGwWAVWJJQ93sYHYvAKmEj/+7AOrkRCQ l1i94QDYIE6gQUcevmAEsYWAau596mSfwMi1gJFhFaNIZl5ZbmJmjqlecXZGZV5mhV5yfu4m RmCILav9M3EH45fL7ocYBTgYlXh4Pxw4nSLEmlhWXJl7iFGCg1lJhPfae6AQb0piZVVqUX58 UWlOavEhRmkOFiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QDowRr2cskKc5+c4MTTBXTKx5J Hdks35IvnGO5bPd/8fcsrkadRjN2PnjGKWzn66KqxvBybuBLhiObN5xUmHph//quO3W6fyWW b478/E9a/ijDh07PqazTb04wyq9tYM/UVg3fljsxtDhprpVnWvHszN0HHWILfyh5vt26e/tH s09bHfYsCl39TYmlOCPRUIu5qDgRAJX+LNQtAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 9B1DEC0012 X-Rspam-User: X-Stat-Signature: ouny9b6oqr6n1qhg3qd75ehhao3sy9jq X-Rspamd-Server: rspam03 X-HE-Tag: 1691130092-414623 X-HE-Meta: U2FsdGVkX1/xiQDruqFQowETsunR9vqgy5lDH+Ca8O7n7QjlBhyyrip+0YsIFQ1Kwlt4NsZMNZChVs/Faj0k/9JVWEXhSY71GlX4/zf90sm4oChlPC87Sb1kiS5c72d6cjFpPdp+qaGolWry7O954Qa9XmqI1z2pON+/SuOIZji1Ys0nNh9ty8v090WkvVt0OL/vcaWaMQDjjJc4AdAaLGq8mSHwlS00S7JRk/AbHSUPpva5oqOvjIqtZ1mxzk4MpjC4hCmYUjPuBwJ6ubbC5LDy73DAhEOPMlWrwn/fJE56ioVxEsJs11U659q0fg3vzof3FSaIZYVAwlfP2yQg35gz5UrD4MgD7plVF+7sfFDPov+WOE/NCKPT/XNoLKT64Tgl1na14Ud7QRbe0g99B0gWj/Pm3UKSyco5R+Sn4pfyF2/g+vNSXzDjGM7XpXRSwO4WyCeaaOu/FU+c5WXWhfRfDWTr1R+8oy/jQV2EyCyc96YmYpJpnQDnw0TLKOQur/1MRn37grCerVPb3/2bnTPtudPlTC45tRoTy3uy4T/XfenpP/E76R191NnsaM+rBczvbmQWbbj2yqHEW5qWWzdX0AmabXYfkEG4xx0k+s70SxBqt5y5GaNmKxsoY4QxCokyszjhPSoZCaL5+aBQ0R/zE2Ae67bpCAA+1bCD6qtNCX32b+Cu8TDL1s94w516lsO4ShMeKIkB9cVzPvR0k9GttcqtyIOyfMPG3xhCOLw6VkLvUnk4SluYVupFsG5B8KJXLvl4oCbnv0j6jDd2ziBV8694HGGOSOnnPw381z0sDr/d/7t5NdJ2Sk/s3MEHc0QcZ+cGK+J6yqh41rpbhWZI3atqnYJ+9xTk4pe/ExviswFgaMWU6pSyYVAQIdDA/HyPffCrgYgZcC/lDOSYYY3NYq7pcr/1kEQKQpk8FY8dKbbyUPwKLv4jNJJjz7+OPbyIVSV0SoAnYxIYNW9 cDg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Functionally, no change. This is a preparation for CONFIG_MIGRC that requires to recognize non-writable TLB entries and makes use of them to batch more aggressively or even skip TLB flushes. While at it, changed struct tlbflush_unmap's ->flush_required(boolean) to ->nr_flush_required(int) in order to take account of not only whether it has been requested or not, but also the exact number of the requests. That will be used in CONFIG_MIGRC implementation too. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 2 ++ arch/x86/mm/tlb.c | 7 +++++++ include/linux/mm_types_task.h | 4 ++-- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 29 ++++++++++++++++++++++++----- 6 files changed, 40 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 75bfaa421030..63504cde364b 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -279,6 +279,8 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, } extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc); static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 267acf27480a..69d145f1fff1 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1265,6 +1265,13 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) put_cpu(); } +void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + cpumask_clear(&bsrc->cpumask); +} + /* * Blindly accessing user memory from NMI context can be dangerous * if we're in the middle of switching the current user task or diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index 5414b5c6a103..6f3bb757eb46 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -59,8 +59,8 @@ struct tlbflush_unmap_batch { */ struct arch_tlbflush_unmap_batch arch; - /* True if a flush is needed. */ - bool flush_required; + /* The number of flush requested. */ + int nr_flush_required; /* * If true then the PTE was dirty when unmapped. The entry must be diff --git a/include/linux/sched.h b/include/linux/sched.h index eed5d65b8d1f..2232b2cdfce8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1322,6 +1322,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_nowr; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 68410c6d97ac..b90d516ad41f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -840,6 +840,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc_nowr(void); #else static inline void try_to_unmap_flush(void) { @@ -850,6 +851,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc_nowr(void) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 19392e090bec..d18460a48485 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,22 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc_nowr(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_nowr = ¤t->tlb_ubc_nowr; + + if (!tlb_ubc_nowr->nr_flush_required) + return; + + arch_tlbbatch_fold(&tlb_ubc->arch, &tlb_ubc_nowr->arch); + tlb_ubc->writable = tlb_ubc->writable || tlb_ubc_nowr->writable; + tlb_ubc->nr_flush_required += tlb_ubc_nowr->nr_flush_required; + tlb_ubc_nowr->nr_flush_required = 0; + tlb_ubc_nowr->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -615,11 +631,12 @@ void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; - if (!tlb_ubc->flush_required) + fold_ubc_nowr(); + if (!tlb_ubc->nr_flush_required) return; arch_tlbbatch_flush(&tlb_ubc->arch); - tlb_ubc->flush_required = false; + tlb_ubc->nr_flush_required = 0; tlb_ubc->writable = false; } @@ -627,8 +644,9 @@ void try_to_unmap_flush(void) void try_to_unmap_flush_dirty(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_nowr = ¤t->tlb_ubc_nowr; - if (tlb_ubc->writable) + if (tlb_ubc->writable || tlb_ubc_nowr->writable) try_to_unmap_flush(); } @@ -644,15 +662,16 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + tlb_ubc = pte_write(pteval) ? ¤t->tlb_ubc : ¤t->tlb_ubc_nowr; arch_tlbbatch_add_mm(&tlb_ubc->arch, mm); - tlb_ubc->flush_required = true; + tlb_ubc->nr_flush_required += 1; /* * Ensure compiler does not re-order the setting of tlb_flush_batched