From patchwork Wed Aug 2 15:14:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13338360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FC82C001DF for ; Wed, 2 Aug 2023 15:15:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54A2B2801A6; Wed, 2 Aug 2023 11:14:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D2212801A3; Wed, 2 Aug 2023 11:14:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E8EE2801A8; Wed, 2 Aug 2023 11:14:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 08EF22801A3 for ; Wed, 2 Aug 2023 11:14:22 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D77851A0756 for ; Wed, 2 Aug 2023 15:14:21 +0000 (UTC) X-FDA: 81079510722.22.D4E7026 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 017B720021 for ; Wed, 2 Aug 2023 15:14:19 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qzVcnmGn; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690989260; a=rsa-sha256; cv=none; b=QKsTyc9gSqi2xbnEu+dcfARdpKwgcfYFuJ7qLL2ykp1zZ9BBVu0Y9legA3RGc0zplSgK1t uVldXeNwuDRasPlK8z/9Fvyi9cBjLbaI8T7OnDn3AYE+GylvnbWwBSi2jN2b6PVA0r8Cxd kR+xk0tg60mz2ddIol5ivEi505RkvgU= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qzVcnmGn; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690989260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nv6mG4/l7H3PNgCsHrZF/ygxFjAa7ZRftAbjaaMLtfU=; b=Xx6EiMxhmYk3c7TlH8BO385q13Kqxj42lGmd1bpNM7L+bcMpPWlI7G/7bLtRa4P2MZNO/j LTcvD+3f0rHuV7hBgHQbsQND49vThQHMpLDyJ5BNpBIAjFPuVm4sd8b2pGiIfL2RkKXtrL qzdy3VnMfe1XGeESKRyOJULiZPWTwto= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=nv6mG4/l7H3PNgCsHrZF/ygxFjAa7ZRftAbjaaMLtfU=; b=qzVcnmGnCsgZ+JOYQS6BahTO8Y 4fDBiRQz18mhL7VWDmt+dEhAq3Sa5V5SKNqJlpEUHzHABS6JtgC5YuZl2PoExrpyD8AaocAYrVL0F kZOD8nQ41FvXgl/QA/ZuvG8dBS9zKlb18eK+B21iVqCHpxywaoE23ictHd/iNF3rIOD2HGkDTJIvx kfl4cXXMk7Ue0xHMPdyStav5FoVJ0PeXiJjv0ATED/kYh0KKVQOz6g64xjsWqMVN5kNRgcOMcklQ1 D/q8Qnz61oyymYEfIzTuQ6/SOMDCbqbF4Eo9VoW5m98bWEzT4szSXsV1wTLL2S1QdgPKFh0BO1pgB 5MH2+1ZA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qRDYC-00FfmP-GT; Wed, 02 Aug 2023 15:14:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 38/38] mm: Call update_mmu_cache_range() in more page fault handling paths Date: Wed, 2 Aug 2023 16:14:06 +0100 Message-Id: <20230802151406.3735276-39-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230802151406.3735276-1-willy@infradead.org> References: <20230802151406.3735276-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 017B720021 X-Stat-Signature: mq4xm9fi9q6qbnhqfo4aeenc44am53yh X-Rspam-User: X-HE-Tag: 1690989259-173062 X-HE-Meta: U2FsdGVkX18gqR62CHTknFwa4ucLyFse5wECfiOTRus2RUpEFfKR5SnW/sUI06X8uRqBNN4GWAEjEs6bpKFNGRmc85w+50V2bynDMesOM5k1/REWahIBe/WTDjlzOE59chFFp8bsy2iPbUfK6xLz97gWK2m/GxAmg0prB9L1D0StB1CHfOVYjYsCcNlRX33/FC5rq2mcg+4eCsmuANAgc8bfendUsxnoJKdxOmyXkc7BiAl7F5N5iEqwALy2kZukSkWqs38p+G7Z5neBgSxYCr91kxnnLi7XfjbdBjEcQmc+lw/PqXi87R8ptLSUDvYtwbcIlG7Xb+9pbFYNQ6QFasEaYjc8IJeU3lci/CJv63el9E/Yn55YofmlUBI94JLaOO4nCHHzhWspGSIeD+ibyiHit4L5n+b8Ho/x5iTlVQJETqAWcTVqOxmm7bi0qQxCDDtV96iaSPxSb/doBVTp7ujf+R1AI6EDFH4r1bupdAD4zORcoq6nUqqA1gs7+ZWR+oSyU/4ICLT00u8KSYTsVExC8M8YNbEelUaA0CEkGE2Swtd3zYLW4NF02sRzz+VOkXLfArPnHEveoycfO2L/ZNU/tVmR4pKMzo7tCs9yNG6cvE/khDZEMY8+P1ggfmqQhp30cK3MNlhOfCKWXnh9iQ4dHozjC/rsYznlxHDFbNvGZ4VYwDbSV3KQypp7T1kM8MtlUBO4fNczDx5Oqbyl2RetdPUZag/Ep4FbSqTjT1RTg3qTwUJWZElQ16Dn83PXEHhKs4flcuCx3iaC72+33D7cKZhVApN67xKPtgQwMB8HqJNHM7avOcdQVQuGtbml62iBfR9Ud/4X133sNNE7knP8R4m3wmJLOcHVG6x0rXqRDqnYio/V3RW6yvAljr/yRtgoAckwGr023buXizUuxejxipE38MC91NMjEX5cqkQLira/NcbapPh2XWUYySWUjeOXAoqoVx6H1N1lsi+ f/Bpb0G4 qPRfsxA6bVDHJ/O6B7IqtIWFcjhA0RqKakbujDXqB8nqmzTOWZ7jp1d2sGcJWwfllEi3gJbGGbAglSUHrRmed4ccJKaBGGjJfek3K1Zbp1UaozrhpurLIJ4Q4k0x12Ybi0ib/qI3fE6t9DaKFNl6M6YNApfYUeCQw5HwnuDtIBKXhjk2nyeA0ujqx19EtFymDTWqShg+TNZkHeddjOtUv1uetWDa/7bilEpBR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Pass the vm_fault to the architecture to help it make smarter decisions about which PTEs to insert into the TLB. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 621716109627..236c46e85dc2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2997,7 +2997,7 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, entry = pte_mkyoung(vmf->orig_pte); if (ptep_set_access_flags(vma, addr, vmf->pte, entry, 0)) - update_mmu_cache(vma, addr, vmf->pte); + update_mmu_cache_range(vmf, vma, addr, vmf->pte, 1); } /* @@ -3174,7 +3174,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) entry = pte_mkyoung(vmf->orig_pte); entry = maybe_mkwrite(pte_mkdirty(entry), vma); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); pte_unmap_unlock(vmf->pte, vmf->ptl); count_vm_event(PGREUSE); } @@ -3298,7 +3298,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) */ BUG_ON(unshare && pte_write(entry)); set_pte_at_notify(mm, vmf->address, vmf->pte, entry); - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); if (old_folio) { /* * Only after switching the pte to the new page may @@ -4181,7 +4181,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -4305,7 +4305,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -4994,7 +4994,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) if (writable) pte = pte_mkwrite(pte); ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, pte); - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; } @@ -5165,7 +5165,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) entry = pte_mkyoung(entry); if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry, vmf->flags & FAULT_FLAG_WRITE)) { - update_mmu_cache(vmf->vma, vmf->address, vmf->pte); + update_mmu_cache_range(vmf, vmf->vma, vmf->address, + vmf->pte, 1); } else { /* Skip spurious TLB flush for retried page fault */ if (vmf->flags & FAULT_FLAG_TRIED)