From patchwork Wed Jan 15 03:38:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13939797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65CC4C02183 for ; Wed, 15 Jan 2025 03:38:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3586280004; Tue, 14 Jan 2025 22:38:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DE582280001; Tue, 14 Jan 2025 22:38:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C85EC280004; Tue, 14 Jan 2025 22:38:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A9F98280001 for ; Tue, 14 Jan 2025 22:38:48 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 61D431C802F for ; Wed, 15 Jan 2025 03:38:48 +0000 (UTC) X-FDA: 83008279536.29.DEE84E0 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf19.hostedemail.com (Postfix) with ESMTP id 7B2D51A0004 for ; Wed, 15 Jan 2025 03:38:46 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="QP/pyGvB"; spf=pass (imf19.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736912326; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gFMoOwSJpv8P/wz3qgwCAFHIUmnEPnRJywSyGSBHca8=; b=vWrhLngSh4wp/UeI30KTWhYjlzWka+COz7qUyTlCe90iKqsnUxRm0P0LfKIqAraJEZzd84 QSBzBRuBIBERbZQXhwPWzAGNY2ZHmzMDcBqam2lWYvhPDb3HmRDK4KUNiAeOYPWFPoMqtv RmWko2yEMJYq0g6j9OP5FHeRARH0nYU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="QP/pyGvB"; spf=pass (imf19.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736912326; a=rsa-sha256; cv=none; b=bbkYvpzj4XmBpNjxZiFCYgwTiqZAVAAHesjGDr5tzKQjDfnIynYye0fci8xRpoOyz/uqu/ yPadqPbMLce23tA+kI4QnUebF20PkYdzarzcC55rZqq7pL2WDfBJC4Iao/0dEH0po+KpqW 54rkB7TqgO7mPu1hMnfymKhNU5ZtabQ= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21634338cfdso105138555ad.2 for ; Tue, 14 Jan 2025 19:38:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736912325; x=1737517125; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gFMoOwSJpv8P/wz3qgwCAFHIUmnEPnRJywSyGSBHca8=; b=QP/pyGvBevBwYILv3g0knCpjIUoKuw0uNDKZXtdGp2EcMRlRit1jd+b3/ohrdXIUdz FXp7kpEt6tBcInvXjJiql7CNO1aE1DI864aRnlmN3N+yQCUTsJNRaJdwC1rrgMT/UECW XzSZ3omSqmDjc46lFrjdugoWFy18i33cNB3A/AxbWWkz29Hy/eduB74bqJ9cNzp90ZUN qfI07BjE9vnuVme25yDvo+1+Wt3JD7IBp/9F9zadAJMtoW/NBsNpmo1Jtgfuqktg9mKv fhVIW0S5IH+Al+XD+ogvdnrQ7LpZlSr9oZwvtrKAKYDyHcttJUp3X1SVjWXrlpfMeiLg 2HZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736912325; x=1737517125; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gFMoOwSJpv8P/wz3qgwCAFHIUmnEPnRJywSyGSBHca8=; b=sBAhD1DDXSFnYG3iaeP82arN7SXkrG+/qr3YQ/R+NdtZav67G/BC3jgHuLigr1p411 9b9qOwcGWNIIQj6QU2z3d6C3Jk4fy6oofX9gG6NF7GGbgEcnlyFsIUBU7uA7UEMOytC7 /LLeo6prSPlFsSGqaKdExFhGQxJIlo+0ukGu09Uu9Gh56JBQwj52UEyxhlZo/bzzPswM fdSxpumcuIBveA9apBpBoLI1hMyiW1FhnvyJcgdwKxiBPe94HZqcvp7YjBlMiHmw6E31 dJukYtS8OOzkOD0YK8p7uBRMoRzTU6GIe4m8kn1/utcN7AE0PGYFTnj4zfcLo38Yh0Ip Klig== X-Forwarded-Encrypted: i=1; AJvYcCXHRBtzbfw9oHQxkve7N7Kj6bLGF8Ropzopz1Ze63ThgNyX8xnsYoBCnsuxyzRtMD4XSnWNZafmsQ==@kvack.org X-Gm-Message-State: AOJu0YxbWWEugHyplP+oESht5FwvnY5YRhaww8anwukvGqk8gQkXIxWO mOYcvPFp3sOIyheq/87WyHlYNCvSntkoXhOdDFhsW4eMQbBOyrbC X-Gm-Gg: ASbGnctIsxzUEmn33z1p1HLCCWDUB+z1OkFmePT6jGUhK8M18yjZ2sjezQMR0WNxZqZ LE9UWLFrzZqfUmp8a4X9+daOKmHgkMVyLz5MrS8R9Wy+CIDAu8DNAC3PRaK9gcSm3y2lXPOhF6A e0bCFLtW/hRZ3ZSMZZL8QV8SEkDMUYNSkO9KE1+p89G+TQTdGTM65d/93oWsgEyM3fVNSlUt9+0 fWwIt5mYkAAFtTPKWY25/kE4lpBlrhVeXCj7fRfVQ08fZn+R4BJDl62M5LbOuRTOZaOQjFEOa2A dr8bVW1p X-Google-Smtp-Source: AGHT+IFF9bEbjjmNapGhmql+BuPntstAKEicpEzxJg3Qh6LkZHMTOzHOOAv9vQg9vg8A6/X699gIig== X-Received: by 2002:a17:902:c941:b0:215:a179:14ca with SMTP id d9443c01a7336-21a83f3eec9mr416527735ad.2.1736912325125; Tue, 14 Jan 2025 19:38:45 -0800 (PST) Received: from Barrys-MBP.hub ([2407:7000:af65:8200:e5d5:b870:ca9b:78f8]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f10dffbsm73368195ad.49.2025.01.14.19.38.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 14 Jan 2025 19:38:44 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: 21cnbao@gmail.com, baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, ioworker0@gmail.com, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, lorenzo.stoakes@oracle.com, ryan.roberts@arm.com, v-songbaohua@oppo.com, x86@kernel.org, ying.huang@intel.com, zhengtangquan@oppo.com, Catalin Marinas , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Anshuman Khandual , Shaoqin Huang , Gavin Shan , Kefeng Wang , Mark Rutland , "Kirill A. Shutemov" , Yosry Ahmed , Paul Walmsley , Palmer Dabbelt , Albert Ou , Yicong Yang , Will Deacon Subject: [PATCH v3 2/4] mm: Support tlbbatch flush for a range of PTEs Date: Wed, 15 Jan 2025 16:38:06 +1300 Message-Id: <20250115033808.40641-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20250115033808.40641-1-21cnbao@gmail.com> References: <20250115033808.40641-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7B2D51A0004 X-Stat-Signature: t17yjzihotd961kq7k5rzengrdijg94b X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736912326-945157 X-HE-Meta: U2FsdGVkX1/SvQqkMRsgkrHgUqE4gRvMsAEdf1Gq3sFOxXBh7McyNeNmmnxZx2nPa3VGwm5zUaAA3vbF7B0G76kyQw8yTJkgZWCFxfqMOybj9V3F5baGn892nlEC5ZYLiT3NXj9rR/TNqXQSMWXvSnDfdtCBFgEzfQOEAJ4YWijfved7SFnvbky0Ix/7kZGGbr4s87QbXGpkr0OaH0JkqTylH7V9oToZjbkyoTEaXLtoGHzPDojBuUol4Gk+LaphSdoFG05vSvsOGsIRYenlwKrEoK7tLE2zxXckrm3GGeXMQf3ZgW0sURV2dEtZop5eu3rqLNMtkFsepM8gcHAv3eNhIVPOoKJX29pqXvz7qS/YFhFcWbPE22rpXxLrKkyraBgP2bvT4B3fUpX5YhyKCUdDQGaJyKB5DlgobivbcBGcKyUgysj84vLVz0E4kcyH0Y6g6ep4tVXgEzxhdSlkBta+LqWe4GYHq7QjWrbF6dzyA3pzB6it3b7Aqk+BSSrlkRkXrnDL7eonWtY/aoCndWy+tEAwibhN3yOTJORwbl/VFRnUcpGnYoXTvakAYmqQ2NKV8OVTKA1AHwCdsksgTdGvpelDXfP7mzQk8yNdUk7oWngpjjoAxJGe3aj/M9Fn6F3LWT+rjPxQ7Zx43a0Smqi19yTUukY9p7v5hihvQHvKleRkKlpJCv7iDIty4+77qnfxI7hu46Fm+xZal76vhLbCoS6hF+K8l2KxXiskDUjsjPeii7rmcDgAnnKKj5Lpry9ipQPFMiMHcBMhRvrnJVMt7+jju0LtANMvT9BfG3PBaF3K9Uh/63JX1sme7KN0fD6B2+GNzOhRsC5sqvnGiUeOJ9t5PwfDDtM0uZzuUjWdEgFzegGziQ2yj3SHaN/fKpFrj7wYMM28L2Hs5r8aA12NhOVJFaX9Vx+nr+vasTOjiMtzHt9I7bJJhomzeYdsi4Vh8IvijbePF5UJfcI BJCRDwce t2W38t3dTZgNrk2ybxAPoSyZyFmtpGHUz54R7/UPlaN2tVs5TvlLBvwAWdoSDfLouOX/H4vxRRx3q81EFQYw4oKzY4weqt1qcmW72y5JOY6E6h21XFYQEc64mqLY8Vxb7F47Cj3lKmh3ElEycinvsSyEmWE9ynQ6NGoZleKha1qEGOmgBAUE7SFKVzg1akFXqXG65+SlA7662ZWWYFAjoQ51JylRsZ5MM7bAR9YIJVjwuIV4rq1w5XOHO7fDz8n9iSVhyLJ94/7RSJWP+Dn32bQY3DqzQHSAXeOzgTU5WFeQgk/38Dy7tkrzX7Iqxntd7b2G5STuWECNoDoAbC53URVQfKxsaeu9Ef393b6NhFtY+cYuOyNaCWQJ1fUXLal4fzUHh5AY99nMBbjAeVnSdRUVPLRdeumULRvGDlhOxgHaPwyU9KcksavMurxN5ooYhuB8bb6HDZBcYN38zrqvU8Wa+t+vU4yGf07KtNq4CJ13FbAhOwP6vcxBonIh3aMR3aHq9tpYhH/EnZtbgHafJiLxsFWCpNoglEYPA3egzOO74n4dxLh1JSDdUdLnA7z2Ca8tm/Lq9RRAjuWSZBAH8S8eKE80wgUdqkT0LrX/aS75UvJtTbrQ+m8gdInqo+Mg0EXYIWVOtZE/SgU0ACDOLPmeJYqUIH1OZTibqmQjX8lOrF8ptftHbKkLVXG/aGEzmfR/l0VXuA3N32tKORNU8Kfy58TPDY7wQ+j1YGBkQ0mOn9hAtzNzNdy/3llCnyZpCLqAYle+1SEdYxVdLSvEvGNVbe/8OTPF7obLhvSRfTMVbuzFAaXjXtFgonDeQO+NUnUkG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song This patch lays the groundwork for supporting batch PTE unmapping in try_to_unmap_one(). It introduces range handling for TLB batch flushing, with the range currently set to the size of PAGE_SIZE. The function __flush_tlb_range_nosync() is architecture-specific and is only used within arch/arm64. This function requires the mm structure instead of the vma structure. To allow its reuse by arch_tlbbatch_add_pending(), which operates with mm but not vma, this patch modifies the argument of __flush_tlb_range_nosync() to take mm as its parameter. Cc: Catalin Marinas Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Shaoqin Huang Cc: Gavin Shan Cc: Kefeng Wang Cc: Mark Rutland Cc: David Hildenbrand Cc: Lance Yang Cc: "Kirill A. Shutemov" Cc: Yosry Ahmed Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Yicong Yang Signed-off-by: Barry Song Acked-by: Will Deacon --- arch/arm64/include/asm/tlbflush.h | 25 +++++++++++++------------ arch/arm64/mm/contpte.c | 2 +- arch/riscv/include/asm/tlbflush.h | 5 +++-- arch/riscv/mm/tlbflush.c | 5 +++-- arch/x86/include/asm/tlbflush.h | 5 +++-- mm/rmap.c | 12 +++++++----- 6 files changed, 30 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bc94e036a26b..98fbc8df7cf3 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -322,13 +322,6 @@ static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm) return true; } -static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm, - unsigned long uaddr) -{ - __flush_tlb_page_nosync(mm, uaddr); -} - /* * If mprotect/munmap/etc occurs during TLB batched flushing, we need to * synchronise all the TLBI issued with a DSB to avoid the race mentioned in @@ -448,7 +441,7 @@ static inline bool __flush_tlb_range_limit_excess(unsigned long start, return false; } -static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -460,12 +453,12 @@ static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, pages = (end - start) >> PAGE_SHIFT; if (__flush_tlb_range_limit_excess(start, end, pages, stride)) { - flush_tlb_mm(vma->vm_mm); + flush_tlb_mm(mm); return; } dsb(ishst); - asid = ASID(vma->vm_mm); + asid = ASID(mm); if (last_level) __flush_tlb_range_op(vale1is, start, pages, stride, asid, @@ -474,7 +467,7 @@ static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true, lpa2_is_enabled()); - mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); + mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } static inline void __flush_tlb_range(struct vm_area_struct *vma, @@ -482,7 +475,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long stride, bool last_level, int tlb_level) { - __flush_tlb_range_nosync(vma, start, end, stride, + __flush_tlb_range_nosync(vma->vm_mm, start, end, stride, last_level, tlb_level); dsb(ish); } @@ -533,6 +526,14 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) dsb(ish); isb(); } + +static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long start, + unsigned long end) +{ + __flush_tlb_range_nosync(mm, start, end, PAGE_SIZE, true, 3); +} #endif #endif diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 55107d27d3f8..bcac4f55f9c1 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -335,7 +335,7 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, * eliding the trailing DSB applies here. */ addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, + __flush_tlb_range_nosync(vma->vm_mm, addr, addr + CONT_PTE_SIZE, PAGE_SIZE, true, 3); } diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 72e559934952..e4c533691a7d 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -60,8 +60,9 @@ void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, bool arch_tlbbatch_should_defer(struct mm_struct *mm); void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm, - unsigned long uaddr); + struct mm_struct *mm, + unsigned long start, + unsigned long end); void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 9b6e86ce3867..6d6e8e7cc576 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -186,8 +186,9 @@ bool arch_tlbbatch_should_defer(struct mm_struct *mm) } void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm, - unsigned long uaddr) + struct mm_struct *mm, + unsigned long start, + unsigned long end) { cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 69e79fff41b8..2b511972d008 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -278,8 +278,9 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) } static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm, - unsigned long uaddr) + struct mm_struct *mm, + unsigned long start, + unsigned long end) { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); diff --git a/mm/rmap.c b/mm/rmap.c index de6b8c34e98c..abeb9fcec384 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -672,7 +672,8 @@ void try_to_unmap_flush_dirty(void) (TLB_FLUSH_BATCH_PENDING_MASK / 2) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, - unsigned long uaddr) + unsigned long start, + unsigned long end) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; int batch; @@ -681,7 +682,7 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, start, end); tlb_ubc->flush_required = true; /* @@ -757,7 +758,8 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } #else static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, - unsigned long uaddr) + unsigned long start, + unsigned long end) { } @@ -1792,7 +1794,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address); + set_tlb_ubc_flush_pending(mm, pteval, address, address + PAGE_SIZE); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } @@ -2164,7 +2166,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address); + set_tlb_ubc_flush_pending(mm, pteval, address, address + PAGE_SIZE); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); }