From patchwork Mon Jul 24 18:54:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3461FEB64DD for ; Mon, 24 Jul 2023 18:54:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD87E6B0071; Mon, 24 Jul 2023 14:54:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A88996B0074; Mon, 24 Jul 2023 14:54:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 950418E0001; Mon, 24 Jul 2023 14:54:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 84DA56B0071 for ; Mon, 24 Jul 2023 14:54:19 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 59F5A1C9AB8 for ; Mon, 24 Jul 2023 18:54:19 +0000 (UTC) X-FDA: 81047405838.08.25F4718 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 0BAC5180017 for ; Mon, 24 Jul 2023 18:54:15 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n5GXwO6D; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224856; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2AZT1+3agWdrOE7nqGCVHOcFr0jkcGTHBHWs2FPeDAM=; b=ZD+TiUzLuu+BhC23YEqzlEi/0NLD0maIbfImt8kFG8L9w5qiA3P8ewKhR5tMlFxOoRJkHq 5nehS4f+oua/b+K2XgoEQEGG05TtXn2k5stWRQkoH94/oK79mubXcV/FYW1SFWmpU4cIPF MqYV+8TwEGBCMXHp/JNmqdUDOZXYyx8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224856; a=rsa-sha256; cv=none; b=ZRcHnJW2CgFtBOTEhl+TnA9LPYMoejBD10Vyh5de1vX4+DPkzkpzYQjDT6LrLZCYViZWts SfbSKhhoAPUzKbRRSf9xT8HIHzoJg40tWxxhi/ej+0h6zNoMNjZBF5l9SqUm3Ljc1Gv5Ql RAqOFwvF+zoirkP2rD2xH7Sf8/NW28Q= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n5GXwO6D; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=2AZT1+3agWdrOE7nqGCVHOcFr0jkcGTHBHWs2FPeDAM=; b=n5GXwO6Dr8M+Aa3h7HKboWgfbn O6jzDiQok9xwJwtnCZMhuc2dh0YtV6FQJWpLd1iAAgzkjij5vatuLMlggbmcGT/XPMZ5rEC1vbHG+ dIDesgUfaQ31ZEBPWNRwAWCW7XVkC7uAv34PynBov/6Mr3o2ot8lA2yNDCI2J/npI4AdVaRoHR24h msmW+juCmkG/E4EQ2AvmRvbvwxI7rZZVKE+YjdaOaFJRREPwbl1O4kwNhrasi+oMpo2IRxx8u2dYI KNnMexdkEPHhY7srT/Gv6v0BYss34lJgRGAKi58np38ju4aZRpssnyqcmRTW7YggNW3h9jdjWCyBC 98GOUPDQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iR1-25; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal , Suren Baghdasaryan Subject: [PATCH v3 01/10] mm: Remove CONFIG_PER_VMA_LOCK ifdefs Date: Mon, 24 Jul 2023 19:54:01 +0100 Message-Id: <20230724185410.1124082-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0BAC5180017 X-Rspam-User: X-Stat-Signature: 9fpp9i6q9mfu853f1oia5hny7ihiud7c X-Rspamd-Server: rspam03 X-HE-Tag: 1690224855-171201 X-HE-Meta: U2FsdGVkX1+F8PvY8QxfLnShXS7pWMDsnoQSMz6vUp/q/zqxrWc/A2frFeHebTyVsbuYasTTM2deXpxFcrxbK8eeFEk16FFkd/ehfIymz/3Y55iqiCAMjWotw4Y510qLBPHUbmXZknOQS6mTxDtzA+Vtkw0+OxW2f2HY9biWQ+HzoBGn5Rdg1HRpxJuhTltpSI0IhGdvzPPa7zPWQ4ACT5+KUV4ckSerJljTeCH1AoWGi2+vzmOoQnmscHmZdRcAQYQzkuJQ8GL7rtYEGD7orcJlaZmSVseAzTOdL17ZXCd317ysIJpfzeo3BwqJ7Po5RFPDw1CDEQuNwi2mxBNQV0qWNMn8SszfMyO6jcqV2LfRb0oaiZWxzh9vQ+y2pEx/KWUhv9641zuFpsbM7yq6FXcrJwaCb8EcHXIOkIwcZRSorkT0yuB8/EvtEytfYb27Fk+F2lfx+WXEm7gqLNrnqSRveo0VS/HB7IiC3Db5hlEYCDpnc8NZI83jaRRrBGrFiEvIBh+VoQS7taDEAYAr4lL2ATkBJVkmurCB/Z4Uh5IVNk3ywGs/sIfMZY9U8kuVLUzVMakWllwO7MNTrLsRaQi19HErUcZarsBzylIyMoZjp25wl1j9w76rTtb8yZ6xD8Lk8vJhdhUwEb0eidTXP8pb70atzdgvl94UgdQdVkQG0l/yqfuI0YXN95RGWwyaP8D2vtX3bKZO6i7JQ2eKwfkcrzT8L5c5i4X/Ow7BbBivmfMcw9rvOqUfMI60s4kwtHjQ7FUFE7mQPESZ74kx17kmU3wZvJirMfbSSO0jg6fhZEL0TaIn1dDhOcskKLA9AKXPF3tJNPHaMejbKnmkRN+3hYZP9EDGyUWeLs7Ogj1z8t0nceBboH8jMNIgkYlWqTf7Lj1sRpQxtk73MrhdRTC2ADQJBJ+56z0yOtfx1UPO7wk61VnR2jo+FT63yRpOWwkxRGPnKFnzmDyUlE2 /PznV8JT jDI642EYgAKP/7Gxg3+gKeIBXkmYPpl9qPEUBGaTuzTHDBQHjApaoZD2yxmLDRq9Rhr4MRmKzuA1sikSk7V6yveg3Hnk2UlcIpLakLG22HFqQYF6nfSIczJjjp0sxk3iUMhNbB/PPpqlUSO1K3exmXlduvg9wrv+psAMJB+94bLgWGdkUpHkG/5Md6lKCx9yI16LEZT7f9NjDHD+lUQYc+HP3pdkP7ojeMYE669uOwGMV4zozsuDHepf+YQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide lock_vma_under_rcu() when CONFIG_PER_VMA_LOCK is not defined to eliminate ifdefs in the users. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- arch/arm64/mm/fault.c | 2 -- arch/powerpc/mm/fault.c | 4 ---- arch/riscv/mm/fault.c | 4 ---- arch/s390/mm/fault.c | 2 -- arch/x86/mm/fault.c | 4 ---- include/linux/mm.h | 6 ++++++ 6 files changed, 6 insertions(+), 16 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index b8c80f7b8a5f..2e5d1e238af9 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -587,7 +587,6 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); -#ifdef CONFIG_PER_VMA_LOCK if (!(mm_flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -616,7 +615,6 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, return 0; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ retry: vma = lock_mm_and_find_vma(mm, addr, regs); diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 82954d0e6906..b1723094d464 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,7 +469,6 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -502,7 +501,6 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, return user_mode(regs) ? 0 : SIGBUS; lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the @@ -552,9 +550,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); -#ifdef CONFIG_PER_VMA_LOCK done: -#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 6ea2cce4cc17..046732fcb48c 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -283,7 +283,6 @@ void handle_page_fault(struct pt_regs *regs) flags |= FAULT_FLAG_WRITE; else if (cause == EXC_INST_PAGE_FAULT) flags |= FAULT_FLAG_INSTRUCTION; -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -311,7 +310,6 @@ void handle_page_fault(struct pt_regs *regs) return; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ retry: vma = lock_mm_and_find_vma(mm, addr, regs); @@ -368,9 +366,7 @@ void handle_page_fault(struct pt_regs *regs) mmap_read_unlock(mm); -#ifdef CONFIG_PER_VMA_LOCK done: -#endif if (unlikely(fault & VM_FAULT_ERROR)) { tsk->thread.bad_cause = cause; mm_fault_error(regs, addr, fault); diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index 2b953625a517..a063774ba584 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -407,7 +407,6 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) access = VM_WRITE; if (access == VM_WRITE) flags |= FAULT_FLAG_WRITE; -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; vma = lock_vma_under_rcu(mm, address); @@ -433,7 +432,6 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto out; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ mmap_read_lock(mm); gmap = NULL; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 56b4f9faf8c4..2e861b9360c7 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1328,7 +1328,6 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif -#ifdef CONFIG_PER_VMA_LOCK if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; @@ -1359,7 +1358,6 @@ void do_user_addr_fault(struct pt_regs *regs, return; } lock_mmap: -#endif /* CONFIG_PER_VMA_LOCK */ retry: vma = lock_mm_and_find_vma(mm, address, regs); @@ -1419,9 +1417,7 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); -#ifdef CONFIG_PER_VMA_LOCK done: -#endif if (likely(!(fault & VM_FAULT_ERROR))) return; diff --git a/include/linux/mm.h b/include/linux/mm.h index a5d68baea231..89f80e2c6ed9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -753,6 +753,12 @@ static inline void assert_fault_locked(struct vm_fault *vmf) mmap_assert_locked(vmf->vma->vm_mm); } +static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + return NULL; +} + #endif /* CONFIG_PER_VMA_LOCK */ /* From patchwork Mon Jul 24 18:54:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1DABC00528 for ; Mon, 24 Jul 2023 18:54:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7085F6B0081; Mon, 24 Jul 2023 14:54:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6913D6B0082; Mon, 24 Jul 2023 14:54:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 559448E0001; Mon, 24 Jul 2023 14:54:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 470E36B0081 for ; Mon, 24 Jul 2023 14:54:38 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2410B160B1F for ; Mon, 24 Jul 2023 18:54:38 +0000 (UTC) X-FDA: 81047406636.21.FBBDCAD Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 7F9F240011 for ; Mon, 24 Jul 2023 18:54:36 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=T5WZLqos; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224876; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3AerkyW6jQd6y4z+nJVqv9X/P9KyvfBPmyU8zyX0gIY=; b=6/DbNWXhdYWLlPf5cW1TfcsqhlXy1E0MyUV4/AowFp9+NgYSEb6u9z11vniH6Ye+5Eq82k IRXoxl3Z7csqiakDo5kse9HFk52/ydwnufSvUWLTPvX2IAGWVhgY26S3iGQgjNeLine/6z 8gVoSx28c/e2h4RSpqTSge6lLyeVV5I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224876; a=rsa-sha256; cv=none; b=fJeDHxCbOyka3Ca7r/Exgdvd+5xW2vcxYK4CqI0FYQd6dapoiFrMYYG1hLsfDdgSaXZ5QO 81H5SUUw+GzTDOq43/YUOtVPuYX0S5U31nD2AKG6zY/+/wRa4n/LfR/1AzoFPwb0KK0F8S hRi3qLTA4nhJuQVE69oWbVzA1qDWQGw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=T5WZLqos; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3AerkyW6jQd6y4z+nJVqv9X/P9KyvfBPmyU8zyX0gIY=; b=T5WZLqosM499I02fiX0L+Oh2WN UVgaWQNODE3toYL7Hkf9TqP02b83nLnWZK0d16D8E2EBKem+n5svfi/BWGCMHr9+JqGgq9wSB2cUZ Ul2RAEhjDKUkApuBDNwSO4vEr1GQ0rCAqJg/tEc/UP5d6/RpdIbRYVXtlvVBCJdZzViD0p6t3MFTR W2zX2H9kNr37fSa1DoYb3/SnNFsASa0KmyalF7lugxtR93Ykk3scERqFti65kmwu26eMSGo7evEHg ES2YTHDMR7hSv6+QZd3gF6jnzLamcbnFtqXQuCMCsUrYKjAotodfYGgMlts3LXVKYhDeRDybzsjUI 9t1fww6Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iR3-4n; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal , Suren Baghdasaryan , Arjun Roy , Eric Dumazet Subject: [PATCH v3 02/10] mm: Allow per-VMA locks on file-backed VMAs Date: Mon, 24 Jul 2023 19:54:02 +0100 Message-Id: <20230724185410.1124082-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7F9F240011 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: tjboagb6z4r98u85d1jnegrrendjp4th X-HE-Tag: 1690224876-282969 X-HE-Meta: U2FsdGVkX18HQ/PDKPnAMq08NNSjAu6vd4S2/9UIDni+u3UeiqNFuozIpkGZZWB0dSXCyg6aXVGeESbD9oGgDQhItjiq44kfkSBYXcXF10LqUd7D8YdOwnZmxdUHp9rZFXp9+w1WH4clzQI8bWsXD/HryPWR45x1Mcl4EZQWueN7p2Y6ZT5cHkFw2gOM2l+juHh/pLN/iXLfcwdUvvrXvP6MvkUdqkfwQtbXWz1aBxgnbqxSX/Kd5IpW+1Vpnqb7cbm90grlERSJni3nU6WohZx4r8hjaKP0ucXcdUHKqy/LeRTohEw+dYjK5VtjUN0HB8+RIfPQpeXa73rAFPomuJ8ZJd+zDQG0aNRaQErvFHHVJ3UaoUE78vk4kg+NYHxI8u4l0pAj1BNo7Z0jgXsNs2d52+qEHjuOXgsErUA26GS+Vcak4ZE+ycufJOLpH3Ueyw/IcxW2tOWzxJ1x9YXcTQav4Rd2CNcCPE6FSB0YcbBu5WC1IFZc7jCoc+WQUd/8/QW4GCwzkBxqwVnhxsB2G8Mk3OeLoB6UA3XB7LAKxnFYRZeAQIpp6etduQTHDPr6Q1AKc+LTrHLkyJlJKe4S0LE2rO7sryFrA/3pVD76mX/xNLqMTWu5g7YuzHu4biyg9/bSGtydr7Epavea31JYqebrm0uHIi81wOVvaN40P3x7U2vRrTfq9/VYXA+eAYTGrhXndfy16WPm9lXRVbTlI/HpOhB7gCux4GgR7+JqCXOOLKILFMbnI7qulI6i4ilzYzGiUIXHkuUMZF/T4L0s+mDu6yu6D/lq6OduhGWpnrMZ2p1tixsulWs3Osp/hUEfbNJCa10laYdAsOpTMa4wGalCFjvmpRJVcrLrevzpmw8AlSfo1GvJ7FLI4wkgIOYoJKJZ/tn2PruNfrKG6G7VjVb7QYlhRNAcoXrMlLdFugTObusadj/101CIstdBj/JraAcLaQnpBkdihz2ocwO V98w5ZQe 6QKJDHl7MdBuvkqOOWO0gN2u1Mrwx9G78rB5JPGLyWSFdjCIQrZ4SmZ/lh83inYKrMUvFM07aG/7U4HBu5a/ac3xLxFrB8yW9M4yjlcG9kARjMuNfty5nVOWPKBDpD016RD3D2HA6I20OJOjUDpmut66uIvTU+ZIhrfRTEMtXmgMaEh4j1OMGf8XW5l7jsFjf2H2OPb4nBeX+x6duQiQrp2UGFYsVbX4qdIQIphTgCrIk0qLFde/6h782BNzxVIVRAyhF7QjI1QERFos= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the TCP layering violation by allowing per-VMA locks on all VMAs. The fault path will immediately fail in handle_mm_fault(). There may be a small performance reduction from this patch as a little unnecessary work will be done on each page fault. See later patches for the improvement. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan Cc: Arjun Roy Cc: Eric Dumazet --- MAINTAINERS | 1 - include/linux/net_mm.h | 17 ----------------- include/net/tcp.h | 1 - mm/memory.c | 12 ++++++------ net/ipv4/tcp.c | 11 ++++------- 5 files changed, 10 insertions(+), 32 deletions(-) delete mode 100644 include/linux/net_mm.h diff --git a/MAINTAINERS b/MAINTAINERS index 220566cd8da8..f5f5134219b4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -14874,7 +14874,6 @@ NETWORKING [TCP] M: Eric Dumazet L: netdev@vger.kernel.org S: Maintained -F: include/linux/net_mm.h F: include/linux/tcp.h F: include/net/tcp.h F: include/trace/events/tcp.h diff --git a/include/linux/net_mm.h b/include/linux/net_mm.h deleted file mode 100644 index b298998bd5a0..000000000000 --- a/include/linux/net_mm.h +++ /dev/null @@ -1,17 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -#ifdef CONFIG_MMU - -#ifdef CONFIG_INET -extern const struct vm_operations_struct tcp_vm_ops; -static inline bool vma_is_tcp(const struct vm_area_struct *vma) -{ - return vma->vm_ops == &tcp_vm_ops; -} -#else -static inline bool vma_is_tcp(const struct vm_area_struct *vma) -{ - return false; -} -#endif /* CONFIG_INET*/ - -#endif /* CONFIG_MMU */ diff --git a/include/net/tcp.h b/include/net/tcp.h index d17cb8ab4c48..9d1d312c90c2 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -45,7 +45,6 @@ #include #include #include -#include extern struct inet_hashinfo tcp_hashinfo; diff --git a/mm/memory.c b/mm/memory.c index 7ff73a197cce..c7ad754dd8ed 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -77,7 +77,6 @@ #include #include #include -#include #include @@ -5362,6 +5361,11 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, goto out; } + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + /* * Enable the memcg OOM handling for faults triggered in user * space. Kernel faults are handled more gracefully. @@ -5533,12 +5537,8 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - /* Only anonymous and tcp vmas are supported for now */ - if (!vma_is_anonymous(vma) && !vma_is_tcp(vma)) - goto inval; - /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ - if (!vma->anon_vma && !vma_is_tcp(vma)) + if (vma_is_anonymous(vma) && !vma->anon_vma) goto inval; if (!vma_start_read(vma)) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index aca5620cf3ba..f7391d017c4c 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1740,7 +1740,7 @@ void tcp_update_recv_tstamps(struct sk_buff *skb, } #ifdef CONFIG_MMU -const struct vm_operations_struct tcp_vm_ops = { +static const struct vm_operations_struct tcp_vm_ops = { }; int tcp_mmap(struct file *file, struct socket *sock, @@ -2043,13 +2043,10 @@ static struct vm_area_struct *find_tcp_vma(struct mm_struct *mm, unsigned long address, bool *mmap_locked) { - struct vm_area_struct *vma = NULL; + struct vm_area_struct *vma = lock_vma_under_rcu(mm, address); -#ifdef CONFIG_PER_VMA_LOCK - vma = lock_vma_under_rcu(mm, address); -#endif if (vma) { - if (!vma_is_tcp(vma)) { + if (vma->vm_ops != &tcp_vm_ops) { vma_end_read(vma); return NULL; } @@ -2059,7 +2056,7 @@ static struct vm_area_struct *find_tcp_vma(struct mm_struct *mm, mmap_read_lock(mm); vma = vma_lookup(mm, address); - if (!vma || !vma_is_tcp(vma)) { + if (!vma || vma->vm_ops != &tcp_vm_ops) { mmap_read_unlock(mm); return NULL; } From patchwork Mon Jul 24 18:54:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91BE2C001DF for ; Mon, 24 Jul 2023 18:54:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E9246B007E; Mon, 24 Jul 2023 14:54:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2997C8E0001; Mon, 24 Jul 2023 14:54:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13A5E6B0081; Mon, 24 Jul 2023 14:54:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 05ED06B007E for ; Mon, 24 Jul 2023 14:54:33 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B7ACCA0B23 for ; Mon, 24 Jul 2023 18:54:32 +0000 (UTC) X-FDA: 81047406384.02.6119172 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 01D0F40005 for ; Mon, 24 Jul 2023 18:54:30 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EInUKGfN; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224871; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TWTBYufoYSHkxfGZKIaPme2dTr/Pf10i6Hfp7/PcTWM=; b=23dJhHZSkxpLeZILOgdRq2kFStWbvDfCaT0zCU6vJ/aezr9SV7vlb4E2rFAI5mK0WU+LEB Z5mXDxxCQeu2ORIVy97EZVI221YcstpQFJAIKiyGqfeLgNTQdce8km2uUp8rXZ95H4BfcS 2IYtcLcbp1vq+WXjOUKvJ7YQXCI/NOc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EInUKGfN; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224871; a=rsa-sha256; cv=none; b=rTEho9ZzhhQwgOql/qXpOi19E3rwV4xO0/LSdfi8+iVef4Cnr0+IZ5vGA2Fi0H7WKOknq8 bGw8YL8AbRmXWtDLVt5kxECLWsmIUMkw1AByjo+LWNEhdRDWsD/1nZsVCaVdLalkAngjmj uWR4KvNyezosyh87yRFRpPi8U4Km4Go= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=TWTBYufoYSHkxfGZKIaPme2dTr/Pf10i6Hfp7/PcTWM=; b=EInUKGfNGCuTTNsKCFRp6uzlB9 1fJcmDCG3/jqX0aR3pLqeB7Hfy3fyyoldC/a2lX8d4E4Z+0RA6Dqfj4qjuAvlFXSEhACCF4KIS8YA CfoD9MK242rPZ7Zi1CapBwX3l2XrXyD8sHIWtl7fQT15/dJnIuqKFa9jwI3D0IFIxQXiGB/eIyGQW MgbFEpHyXrSqrRmNd7OZcOekNHH53oejJDKO5zaI1nU3ilkddAlysN4Cviibn2M2A7528Ffzmggtq yof80Uewlt6OaQZ2zNYLFnAa9mWwu7VMTOfLdGS0Qrn4Gfz4V8UeeqDrY3cehYVPDsJXVVYK+Ty+y iaQQTgqA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iR5-7U; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal , Suren Baghdasaryan Subject: [PATCH v3 03/10] mm: Move FAULT_FLAG_VMA_LOCK check from handle_mm_fault() Date: Mon, 24 Jul 2023 19:54:03 +0100 Message-Id: <20230724185410.1124082-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 01D0F40005 X-Stat-Signature: 336nez4sgpqzzpsfrxquto6935a9icfn X-HE-Tag: 1690224870-330530 X-HE-Meta: U2FsdGVkX18ICGvDFUqHN4mVsRbry2nAs+xgs53UDc4mcy5tbj5wfXH5nGFkogWnngTTXje9WDwnbwyM1XNo5b/+GpvC2iGnAqsSLuTgC7fEtqj68VDoyUc4zJFg566zKqt23C8agQvXTdNY/xUbKHH4mzePEHy6TaMYi/eZcv9hvYVrrluPcPi94L6fOOZNYxEC6XOnMG+sHrFXScoJws1HC0sp4a+5nN/DCbyf2Z2GMuGoGmUDq3q4vUiBCYUazthYv+6ff8ODhGyM2zHhoKrgxL4XRmJHCQwnb/9eoFdu9RPF3bQF/B+76ZG87hbvTsxjHfCEM8y+EScW8dKG5fztH4txhqrWoFaCwqj1dxNAT2PV8tye9QLS/tHQ/5T8TNH7rhde8GARWhfTDgvaO0ZNfq6JgEQVE+CFJMhFTEe1k6n8A16Q8awY1jG+Ta2+fdSCF/JTZivjRJcTMLHQ5SFgiBkgIPMNh17nDO2kkRGdMYDYrAO7GrEyFlApJoL85twBkNble8prjGAH31Hq5U6TNx6A3UW4mgWymRKPHzW2xhHQgrA//aoCDfvnfqHZsTU8RK5a4D7iAebuUj9f73UcvNGwOF5raE/c7XxqPEucCTxugHFJyr9oaQM59KGtz6CXva/sgouwpKF7MykoKlo+X3BFZW95CT32aNYiFSQ9kEdrZZ37Wc5deW2312shglBJGlm/X7C40goyOjtpxKRYfd9PdQbX3VPa3gWrXJsoYP9h1soDdq1tOMgfyeNthCZ0IL76YO3XRiGK/LzocUzERXRGs7bqJ2/ZXFuEDw65hSTfD/5rDCKTAwMPYQINPHHWQOvHQN+AaRRNhoy3I9DrI9FSfinKok42JnrbF8i7NG/0MhYCz1SYdt9dAbPjLq8VhUSSjhx4Vhg19QiEiWTC6O3GllhPtiJ3REjrQYw8lsDwkJrnDpVH1xHRT3rN7YXToxD+UkAyZWb/7aP WpGeXvmK npfXuVTD14LBKooYGIqQP6OIlV46scGNS0nNRbo7sQKSIkwIKDKXKp9LmTLhYbLtWO4c09gG+VrbGeyuiTFU3lOo06qMmqg1Mb0RjBSMTbHQqw9L4qj8EkWFL3gfwC2mTaek5jBl710N8Wr2r4zGXziHqcLqudGGoH+zNgmvQXJ6j8uzmRwYjzwjqxe4IXtBLqo1pMhobEY4B3y7UMLY76DyzfD60BdHQ4hdju8+VjWU4MtxoVMD2e+1qSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle a little more of the page fault path outside the mmap sem. The hugetlb path doesn't need to check whether the VMA is anonymous; the VM_HUGETLB flag is only set on hugetlbfs VMAs. There should be no performance change from the previous commit; this is simply a step to ease bisection of any problems. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/hugetlb.c | 6 ++++++ mm/memory.c | 18 +++++++++--------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 154cc5b31572..e327a5a7602c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6089,6 +6089,12 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); + /* TODO: Handle faults under the VMA lock */ + if (flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate diff --git a/mm/memory.c b/mm/memory.c index c7ad754dd8ed..5ca8902b6f67 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5112,10 +5112,10 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) } /* - * By the time we get here, we already hold the mm semaphore - * - * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __folio_lock_or_retry(). + * On entry, we hold either the VMA lock or the mmap_lock + * (FAULT_FLAG_VMA_LOCK tells you which). If VM_FAULT_RETRY is set in + * the result, the mmap_lock is not held on exit. See filemap_fault() + * and __folio_lock_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) @@ -5134,6 +5134,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -5361,11 +5366,6 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, goto out; } - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - /* * Enable the memcg OOM handling for faults triggered in user * space. Kernel faults are handled more gracefully. From patchwork Mon Jul 24 18:54:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 321CFEB64DD for ; Mon, 24 Jul 2023 18:54:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEA176B0083; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9A7D6B0085; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A61DC6B0087; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 98F6B6B0083 for ; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 36AB0C0B80 for ; Mon, 24 Jul 2023 18:54:44 +0000 (UTC) X-FDA: 81047406888.10.0CC43BC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 8FA28A000A for ; Mon, 24 Jul 2023 18:54:42 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h2urfpoJ; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/dr8E7Ysfsh/pzxHk8PrgytRhAkkfjQGUpicX+vSfvw=; b=vYTYPD8HJAJoS2Vo3g9ExAmsFOjJafakSNR75rHL0Xgil8ztAdV69NN1Qu/C02gzg3x1NO fg7A5eDkNLvtKBNjlBEP8bXPn7Cvfp8loOPh9kTxZCW9y5c37NM8KMMotCLbaTIwInHHh4 y/pAnmrvsJiOFzgp63YiQ0CX5/arF3M= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h2urfpoJ; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224882; a=rsa-sha256; cv=none; b=dXo1T22qi4b1kYBrwbC9xfDv6X+IlwTvdxKyoX88bAWgEkL6zzX2M4398tYMoKtoyYB0D/ j9VXLMiNhQ3xE8S1GUkkaV4q2XRb/rHKhuNv5inkx/LjzdNocLYuOGtQ8vVI73wiP1w4Ew LCwq2RLthXt5XrgZp55K+Nov7igm6Mw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/dr8E7Ysfsh/pzxHk8PrgytRhAkkfjQGUpicX+vSfvw=; b=h2urfpoJlH+/tTgmv+1TWIjF7V SHPiUiKpC2n086K/Blscs2mtWtATwrhaglAB7qSgdBQvKMNtIPQV/UFDUBNnXBCJKZqS02PMxZDhN QIpfedqmho8nh2D698VotNmU0q5qTKALJGUD/wc0E58tg4ZCE4Gn2rhzDLRFYpHGnf4jBnK69OxHr fBw6gic1/us6lWW6l7ExzoNHq56D4JqqkX3VwhzEJSKvB1yN9l0CoM6WoGz83conZI3w9cwm2x84i Zc3bYa8EqN9n54f/4wlFuqSHLh/NnBnmRxHDAymVbpKp5GzpcpfCkw2wIAESzGP7wGjHvSVs7xgCq /JBsnc2A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iR7-AG; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v3 04/10] mm: Handle PUD faults under the VMA lock Date: Mon, 24 Jul 2023 19:54:04 +0100 Message-Id: <20230724185410.1124082-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FA28A000A X-Rspam-User: X-Stat-Signature: c8ztom8ncybk8g1eynyxzt7pifm5rusf X-Rspamd-Server: rspam01 X-HE-Tag: 1690224882-258558 X-HE-Meta: U2FsdGVkX1+hLf+xsYrnCxkDlvLGZxC0OMTDAWKy3HhcrcGeaDld175VzFdS5iM5imAeUYiDmrzVKXjEyxmHCdctEeM3aZTOFq6JS2iKK4qjdOpI9nc2MFHpxanIstzZraQbKOGfS+W8juCdzmRQoHXdd679RRjyxafs/BsAAqKJK/iTGjQLAshxjhQz8XOOiAT1StrHjjR3tPIQFgBQylCtWVn8GyDt3mUKKiycHpTqg8fYJdBgj0OJlhGBYU/LxwAOfWWRY+jC3A3OOqe9ORSt3j6iQD0rtRHLpZhHp0Ac+CHS1cGWcptaaDfeVxu63lxfEgS2MjRWUQyJHja+Mb2PJCaYCBE/cjHxBnuh9Q3ersKYgG0/OpYPFFTC5N9WtOYSjNqO89iin+jy/ARE/uAR5X0FldMAn01/gKrK7nV9DGP1FxtKQUDBw3W75QOO7x2yR3LC62KkCYm5pjNXyQCBuRGiGOr3/TFqIVAxgYAc1HKoe47YaRo9j4bZCF3s/NOtP4n5Ahdzs0FxPNaW5OH2Nw17qG+lIPYpRJLyYqdLZC7mrvmT8XdGN0ceCBcDiI0O91lMU49/aTJILnJmpzAHS500yl/ug1Mr8wQu+e5ntg3nWGQHzwJAMfEm8lksJcjzYNy5XlCMANt5zaTJL+p4wlKzpTGa98m3r2H2CaL4E7aZvFK8HSx2HZYIYopQYbwSRwnwiIk2mBbHmM12ACQJPYk03pm9GD4rSx7v/25J+wKAnvWSz4Lfh9XcSpaA8CWw8abSrI210lD7MOn4SeeKTWYr5tUz7xNiQOLsMyvA6nH/IHcQor/9jUZ/EwaWovcA2+MPZmKsUznIBK6IMO+8AFLuqoest0Pbu7KPwjSwb+OpNTEwtyU0c2mIVwebjwILnAU6aF1nNc4bNyJQ8EuLq/gd/utOO7eOFxZh5IeIdFYVYJ6U73VQ+W6tXp3/FS0knn8pzwVpiDLDZka 34fjSnfj yXMYWbhL9PoCUqXi03mbl0F1kzMoAufcj+UYtmKxa5JbAK0cOOUKLzuC9EZSZg5Gh/gFte2akZ9ZFe4GshETNOfl5Zw3Xfwytxf87VQqTBZL3y4fx87EQpiAPUsUQL75m0rxu+TpFCbfJ+jfW7+yiyeks7ADT/gv3tuhFsIXDT6Md8O5jQPhoyQjP7A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Postpone checking the VMA_LOCK flag until we've attempted to handle faults on PUDs. There's a mild upside to this patch in that we'll allocate the page tables while under the VMA lock rather than the mmap lock, reducing the hold time on the mmap lock, since the retry will find the page tables already populated. The real purpose here is to make a commit that shows we don't call ->huge_fault under the VMA lock. We do now handle setting the accessed bit on a PUD fault under the VMA lock, but that doesn't seem likely to be a measurable difference. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 5ca8902b6f67..7fec616f490b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4987,11 +4987,17 @@ static vm_fault_t create_huge_pud(struct vm_fault *vmf) { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) + struct vm_area_struct *vma = vmf->vma; /* No support for anonymous transparent PUD pages yet */ - if (vma_is_anonymous(vmf->vma)) + if (vma_is_anonymous(vma)) return VM_FAULT_FALLBACK; - if (vmf->vma->vm_ops->huge_fault) - return vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + if (vma->vm_ops->huge_fault) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + return vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ return VM_FAULT_FALLBACK; } @@ -5000,21 +5006,26 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) + struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; /* No support for anonymous transparent PUD pages yet */ - if (vma_is_anonymous(vmf->vma)) + if (vma_is_anonymous(vma)) goto split; - if (vmf->vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { - if (vmf->vma->vm_ops->huge_fault) { - ret = vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { + if (vma->vm_ops->huge_fault) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); if (!(ret & VM_FAULT_FALLBACK)) return ret; } } split: /* COW or write-notify not handled on PUD level: split pud.*/ - __split_huge_pud(vmf->vma, vmf->pud, vmf->address); + __split_huge_pud(vma, vmf->pud, vmf->address); #endif /* CONFIG_TRANSPARENT_HUGEPAGE && CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ return VM_FAULT_FALLBACK; } @@ -5134,11 +5145,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -5182,6 +5188,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (pud_trans_unstable(vmf.pud)) goto retry_pud; + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (pmd_none(*vmf.pmd) && hugepage_vma_check(vma, vm_flags, false, true, true)) { ret = create_huge_pmd(&vmf); From patchwork Mon Jul 24 18:54:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325262 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4737FEB64DD for ; Mon, 24 Jul 2023 18:54:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBFCD6B007D; Mon, 24 Jul 2023 14:54:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D481E6B007E; Mon, 24 Jul 2023 14:54:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C36CC8E0001; Mon, 24 Jul 2023 14:54:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B65C66B007D for ; Mon, 24 Jul 2023 14:54:29 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8C300C09DF for ; Mon, 24 Jul 2023 18:54:29 +0000 (UTC) X-FDA: 81047406258.26.4197E5F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id C372F16000A for ; Mon, 24 Jul 2023 18:54:27 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EL5lMUFn; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224867; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hJu05/6R+2/+R58/bqnW2667sqrjvy4Vo+FjhIXzCu0=; b=KdrPF0ISURF0MZQdbqE+lKqQpE9ZKr9F6ZdSER5Qrpi995V52HlcV9J4x4gbhOOZoabz1z 8Z6pTLRxg1OZZDAn0e1C/Wbm8FEvpW6wlEFXHI9unR4iY/eBsqCmgWYy+Pt4ARz6a54PTY TVbwPMzD18gAqpy6zqtcuDw6xxBODyY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224867; a=rsa-sha256; cv=none; b=0Pac9CxzHaTObjck282ByMTR6M+ERP6g37b2zkNzVO8VUWgru/h7OKrhtxh4yrDepN7zLo HT7d11hv6NvB/5Y4HOZJuZ8wI5NHuj9MdN8b1eAn70ngNKJXX3f+eZ6HHGPoCLd+nFoeiq 4hoTBmISQgq45o4ypHdUZ2XL4hPvIQA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EL5lMUFn; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=hJu05/6R+2/+R58/bqnW2667sqrjvy4Vo+FjhIXzCu0=; b=EL5lMUFnNoPCZMe0eS0Hvijf+2 j32meE393cAjsq10i8jRMBUAvHcjjnw4BZyBiVirhYCjv44Xl4PLoH/sWyaqKSIwj0MCPc3w4382H sMSy+UNGoiKw8jyI5NZzdHT00TMqbw5mU8dWWw9oQPdxKe0U3T6zDJ0FWgJfc8FxlIPQcYk7vEW/o ZuTt6UHN7KrKdU+56Av7l32r4Eo7vG1SZaOKkWH+D5hZLNwYGjYrfh6DSX0Lc98neqYSZBttPU1cN Ul9efZMKPBrSxs/5R+8v5NbJUr06vyCHOrd6yf9uuxA58P9XcM8NezBphhwdnh8O+svpoAzp2Swxq rE7Bc2kA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iR9-Cu; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v3 05/10] mm: Handle some PMD faults under the VMA lock Date: Mon, 24 Jul 2023 19:54:05 +0100 Message-Id: <20230724185410.1124082-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C372F16000A X-Rspam-User: X-Stat-Signature: ghm9y6c6o6zctdsrhhfju7xhzinpjnkj X-Rspamd-Server: rspam03 X-HE-Tag: 1690224867-87168 X-HE-Meta: U2FsdGVkX199xRlmt7fhiZTxrHMHpy/2/6KdwD4IGytfkISaOPWS+MkPs7IoQw9d59xpqDyZ0k0isYT2OdJc/42HSCxgtfGCv1izJvcV6NM7EcczH3D91AO9go4u1PSK6vhhJuxJLi5wwPobIn/IyUULzbW11+V/l0oHbUZdtVxcUsGJlPtxufSv4KYZHOJjLS5uWOysxVN8+2BASNnhUrtwoS5PSIaPRHOIIUxvw8RdMBaL9tfrsaQpfHdygrmcgu9NilzgVUULkdoJ6dZn+3MAHf51wYHLtn+66RpmXiJ6J+2korMCEEjHVRvsmCjIcYoulnXHHQ4vLpVkhCE3ruMVat+OS1hLMF9sDf4Bz2Rsjr9rHYQYxA8nQG2e4DFTmKwdkbW6Z8JofFxwUwERC2fU9qmFbHqVioS5JQWISBZzTiccedhJpL+dW7G6QjVO6Ae1dwbjdUS3Mt/tQX5h/25UztgwTayTUDu59fuUvSQtmNdwZRmqf8ZyBSCWw6SL4PshBoCyrMlZkjtNE8JSUdgo7kMD038ivkGUCUrcbch3/7NeHlegOJ9XelfwX0HqKkEvenkZUvcVL4sCVtE8JwHuUmmpy0jI0rgFlTcbx0gRFdXmQPd1WPhU/r2BZRVEfeIEndaSsf4Z/z40JPjWag84OwAMj9krCK7v/e2iTy7zpPNz5wKTiQnnT91Ug5rUOXeUFlbxmbCo8YQwkYVGljHSg29JeDF4vjOdKRh5ecAy/0uf4xwbxTVFeF1EItsikdTWRoVRvomTumFacvFNZYVTtnrqbFnVfRmp6/KnwTt0137USd3pY1sD7QPBuUPIRr7a4F9gi8wOsF/eskYnXTVlNyF4SfHwY9DdEy1q9goiGPLV9wUh1cggCQa5+T2gKVJPn91Py9xG0pHhh3z6YaV4qUGDyURYsVpL2RSfDExNN4DstQFkCy8yrt4GSVzBH13mtZnvIJJX34ZAbWx 5/DiC1mM J0JGqbq9Mn5aoHC3QALuH5LtMDOOrdHBmH5KyYhyK4K0F7wiP6qnyGya7p910fZGsi7XR/Y2gzq0tKMbw8DYpHNarJSeSfGCQ0ZFubf4bxYYVaQjxmK1h44lOG64NFDLJJKX/ZuYr52SY+Te2vwPYdkVUPGll3JYQnum9L1nd4DQCxYPPx6M2beQ7Sw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Push the VMA_LOCK check down from __handle_mm_fault() to handle_pte_fault(). Once again, we refuse to call ->huge_fault() with the VMA lock held, but we will wait for a PMD migration entry with the VMA lock held, handle NUMA migration and set the accessed bit. We were already doing this for anonymous VMAs, so it should be safe. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7fec616f490b..9e4dd65e06ac 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4949,36 +4949,47 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) static inline vm_fault_t create_huge_pmd(struct vm_fault *vmf) { - if (vma_is_anonymous(vmf->vma)) + struct vm_area_struct *vma = vmf->vma; + if (vma_is_anonymous(vma)) return do_huge_pmd_anonymous_page(vmf); - if (vmf->vma->vm_ops->huge_fault) - return vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); + if (vma->vm_ops->huge_fault) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + return vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); + } return VM_FAULT_FALLBACK; } /* `inline' is required to avoid gcc 4.1.2 build error */ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) { + struct vm_area_struct *vma = vmf->vma; const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; vm_fault_t ret; - if (vma_is_anonymous(vmf->vma)) { + if (vma_is_anonymous(vma)) { if (likely(!unshare) && - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) + userfaultfd_huge_pmd_wp(vma, vmf->orig_pmd)) return handle_userfault(vmf, VM_UFFD_WP); return do_huge_pmd_wp_page(vmf); } - if (vmf->vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { - if (vmf->vma->vm_ops->huge_fault) { - ret = vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); + if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { + if (vma->vm_ops->huge_fault) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); if (!(ret & VM_FAULT_FALLBACK)) return ret; } } /* COW or write-notify handled on pte level: split pmd. */ - __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL); + __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL); return VM_FAULT_FALLBACK; } @@ -5049,6 +5060,11 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + if (unlikely(pmd_none(*vmf->pmd))) { /* * Leave __pte_alloc() until later: because vm_ops->fault may @@ -5188,11 +5204,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (pud_trans_unstable(vmf.pud)) goto retry_pud; - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - if (pmd_none(*vmf.pmd) && hugepage_vma_check(vma, vm_flags, false, true, true)) { ret = create_huge_pmd(&vmf); From patchwork Mon Jul 24 18:54:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29A73C00528 for ; Mon, 24 Jul 2023 18:54:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5F5D6B0082; Mon, 24 Jul 2023 14:54:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C38018E0001; Mon, 24 Jul 2023 14:54:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFE336B0085; Mon, 24 Jul 2023 14:54:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A177A6B0082 for ; Mon, 24 Jul 2023 14:54:41 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6FD37160AD6 for ; Mon, 24 Jul 2023 18:54:41 +0000 (UTC) X-FDA: 81047406762.23.FF94BF1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id B239F4001A for ; Mon, 24 Jul 2023 18:54:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BFJKXQAH; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224879; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j/+5nvHmLcQr2gQNLZnma/P8C+KH0GEPUfvfP6aRFCA=; b=WzW1/sXHFXDB0x3k96xHQVr0ohuKSqBKcSpaW0ZyCJlLBczA2J81dxiwCsh9LCUXHI78yj Q8xlgSxVaxSym9bNLAdrxEEIh9RHlDQckDppM8eKma7hUbYxuRxbJpQ9wGjv3RX2NOwVth 0qo+6unv/Nfsz2Dm4qgfHgBnzIrtxnM= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BFJKXQAH; dmarc=none; spf=none (imf27.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224879; a=rsa-sha256; cv=none; b=Oj52bUkk8WpFFOTcufRzBCcZp/e8CovSeZ8Jf43Stvtdi/jOvpzlPddPB+FBMEcr8P9u0Y MsX1K3NOr1y3BeNnFlWHjghoUkHwYHP2/zHNf8+wu5Eu3vYUKh9fstk2VYrYZdrVvrsmRk FjWo9yxE/u9iXpBwbZhZ/5WcpWc15GQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=j/+5nvHmLcQr2gQNLZnma/P8C+KH0GEPUfvfP6aRFCA=; b=BFJKXQAH8Ld9mSwUnjlCJ9Cgoh fHUbEJMazDTd4xQQptvmIIdTFyaEH2YONUrjuOkX0UZQQ3TTYhtjroUHubNIxcQHwaE5xTUDqOCJs 0VU0ypfcwtPQHyWFq4J4BuPx+zIDlFkLgft+yQUGk+0ivxv9RpxA2BtSphe+v5RvSEm05/oiXwvrR /Af8Ms5TsZWa/9POSZcp9IVBoKd4YybnnKWboaz7rRkZcWp7CN8Ot7LXch5abDyrcgIJSZMVUI+7z zu1lLMHI/+0UzRSBkH2VswJg10brdZ4gtKKnSmFrNxMIEaU9Dx578le2dcBm6S3Rj+jhQQWKd8ygJ E9UgB4KA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iRB-Fc; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal , Suren Baghdasaryan Subject: [PATCH v3 06/10] mm: Move FAULT_FLAG_VMA_LOCK check down in handle_pte_fault() Date: Mon, 24 Jul 2023 19:54:06 +0100 Message-Id: <20230724185410.1124082-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B239F4001A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: efbbzy3pitn1hir4wg1y7q8zq1bqztam X-HE-Tag: 1690224879-131929 X-HE-Meta: U2FsdGVkX19cRfKmbjmQyQYSc4t3Z9GYW2+zPaP+wjr9a7FZ2aidX8KISP2hc2ZmoauKyiZCCNZ7Yh8XhekLRNabTC+yRqcwid4pKX/3Jv5xOH/TSOEeitoEirLCOcRDipZIYBHU21ygD/OjdPgkN2WGZId8vMJnwNmor0h7bi6SqRcVLBs35EqRo5ijZ3kTm6BEu+tHLuWILEtyck3dlL4rRX0SSKDuHz/sOGr9RLQIIFsUVQ/IgInSkpfZfi+X477eO0oX3HBPWTINibIwCIkMrgNC2r3cvj7Vsg/UH+U0ZN6K9YIDoVEoEwum13COJW0LzjYish4sEOiC+4P1rD7pSswQfUv2bH/mRu+9ysc/XMCQyorZNT51XBHov96yw9UH7DNkIgZDopfDSuPDWi2EwHxdmp86aRR4VlqMv+nfHvvM55l41j36T2HvZyjZwUfcYrvcXtt7tdndNnIsz+uGGdDtw/teJCwwSfdAFbF0QyJVCkdCUtNxyw+zh4Gz5fuHkfP+qGuTrCu40lpKb7vPrNTzwWG/gRDIdAL3T28kFP235ahyVd2tXyBfsKv6hXmnqB7pKL/K6dBev0XF2XD2uDjeE9FaSGNy/VgL384wunsXicREdD+grNYJSEN/pH+w59ehvjAqe8NEyJ/sjCi597p+Of/kaosTwdWzHgwJEd2F28d8rH3fwVo65Sjm4ZGdLHr5zh18JT+i+S58KGgg8DbFdmhTDxFg7t6pLKHS3NYGl6uZtgcA9m/x7//fGCn1vUFrrJ5TrOjH740pJSH+54ns7itJoPeuvfNH5OFDGDz26MZbYl9gs15nmFaWb3bX/BqQe0Z58Jdzx/NwZlk15XjDkawlbqqvqcTRc6ngvoTH19RwCvAE93a96xDA4nHwfACu5HxhXq4vWfmmm5QdJB1Iagu/jgWvSysKuFs1iH56AkwNTtpd8uULBC+nuCz755fEplc3L9tMuvk /lRhnb6j TKRsPk5H8XVEzOWtX3y/ZOw+IjE1KcYPA+HlRcYLao67Pn/qhhNTEEAiTs4ZqvXis3DiAzFBlg71EhKJ2lAu8GFclBV0mkvDwQdGiZi2dmXPpWZNZ//Y324zpy09bmxnUK4guPeM8i89CJptyDO3802qrBWG1nBZ3deqDpslQ9Sbu5a0zjhMCBB2LT7HGYy3UIazAXS7s48Pf50MXCO6PvhK7IH3Nu6c5L+PUwUn13xUKYZ8HdVu9iKcbRg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call do_pte_missing() under the VMA lock ... then immediately retry in do_fault(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 9e4dd65e06ac..11b337876477 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4775,6 +4775,11 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; + if (vmf->flags & FAULT_FLAG_VMA_LOCK){ + vma_end_read(vma); + return VM_FAULT_RETRY; + } + /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ @@ -5060,11 +5065,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - if (unlikely(pmd_none(*vmf->pmd))) { /* * Leave __pte_alloc() until later: because vm_ops->fault may @@ -5097,6 +5097,12 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (!vmf->pte) return do_pte_missing(vmf); + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { + pte_unmap(vmf->pte); + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); From patchwork Mon Jul 24 18:54:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325260 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A928DC04A94 for ; Mon, 24 Jul 2023 18:54:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE7C26B0078; Mon, 24 Jul 2023 14:54:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D216B6B007B; Mon, 24 Jul 2023 14:54:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0FC96B007D; Mon, 24 Jul 2023 14:54:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B2EC36B0078 for ; Mon, 24 Jul 2023 14:54:23 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7B649C0B70 for ; Mon, 24 Jul 2023 18:54:23 +0000 (UTC) X-FDA: 81047406006.10.F9AE02D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id B5AF0A0003 for ; Mon, 24 Jul 2023 18:54:21 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LpXGzVO1; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224861; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S+dCr3wFmOnYx33aA8EkljEdRO/JPPjH0b8bNo8hRK8=; b=LzMTMFNTCXkUs6so8E+V8HAkYhzXbHkj/Krss02YhfOQQLxj1ZnEX/EYaII4pn5FTspJTJ 2HHGfsq2xaCkMXdK7+enMNDSyahk6uPrfxF3xeHrDylLxnqabyU8YVLMQ1sJPVEewhmX6s 2mE4MURyp8vS+fHqgn52eriA04NpU2E= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LpXGzVO1; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224861; a=rsa-sha256; cv=none; b=o8onD3GmUzPFc/zbUoZbShauQWW7Phg+OpQxNfzVBpIEmR//gLR38oSfMqT5ghniSsHAc9 dW1DnjSG47k4cLEsGfjDYA627S+jjfkNV7TNnnPOiL8P+BuPcyb3BMhU/fZ04qzQO/uSRf PndlmnRimaE0pH/2nDYuEU9ixIE9RVs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=S+dCr3wFmOnYx33aA8EkljEdRO/JPPjH0b8bNo8hRK8=; b=LpXGzVO1V3NFVNKSce0PI1K3mI 00HJIlamGKCVSJTU/TaXpSYtxAGCJ00H8nAvYbCONsEJF+t6kq7PAN1mvhkItteW71rqdc1PhVEbq 2gY6Z8zsNCr9W04TlSMo9Hy2HGC1WDAHIYECDH4Sbo1jdx+UmSmFhImPpi4G+6hpt3omrFGnPCioY r5OetU3asujp/vCVz0A1TrJNtGePESTSBDleEdYtuuJx/GTh05EMuZxNBZKUDjxrff0WVp9hInezm QXszgBXraH7AudLUVjQqC8IyJoZKy1QmoyTQLjTacx21fFkIGpppl2591qthTSu8Cfpv95BJwk1oO JSJPo0dQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iRD-Jf; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal , Suren Baghdasaryan Subject: [PATCH v3 07/10] mm: Move FAULT_FLAG_VMA_LOCK check down from do_fault() Date: Mon, 24 Jul 2023 19:54:07 +0100 Message-Id: <20230724185410.1124082-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B5AF0A0003 X-Stat-Signature: faife5x4kokc81qf7fa6e6pmq67bmnau X-Rspam-User: X-HE-Tag: 1690224861-113040 X-HE-Meta: U2FsdGVkX1+Y+Slnvhri/b5U/mHO2m0e6D1OmX2LrU4/19em3iskeW7EDrGMluIbP5IgsQ9usmilBrzG+ngZf33ZMjfcwIUR1r0E1xIo0ZcEMuURdsiPR4m1qXnRlTQytuPzLu4JES+E6+2QuGndjyTSvzbJKXw0LKIp79NNvv5MSFH7TkNL2M7YmcnRH/S4caZTA7wzBGRh9V8Vwi9tr+a4Q8Rsbvj/aXea82WMbMwGYU4yjO2Gk1GJDTffFap0KZ0njEiT5n9uEl10i7dwyKvZwuDQQHKQT4w40Wk0uruNo9DhbbjQdSfgoZFSZx2/B23mmOsym2XfVITvfnGwVtTa3bfotxc2cXWMOKbtfY3+vW88Yll5DYcM81qsirFxbufCW7mJ5sHCVPx9f5txPKZIMafpKMppiP6oIM0zTNoxwJrBF1c5SMFyn86u5GZk1EExl509Kcj3HSOIqMR96MYjsEtbNLgyc3LhllkhD4PUdw4aO36CpZnNj8LUILub9HTlmQ3wo756i0lecES3KwyosrNBCX2jPlj5XgsBJoDsw+bOJ0pXF3H+bxc8tsaSVKP8sri1Mz3YUDr9HeWgCkutB3hZsuqMshpVvg+L4b86VglJVxvrsKR3uM6yOXqcDvNoNukU7SwkCWxuUGJrs1pc1EoW3iuNyzGcm0C/T1mQIr1llNxi3Km4SgsNfT3lcsUb1OBcqHVeUupzOO7fynGpB8i7oQy9ivaREU0+mmYmdc4Y42nqKQKCrlyZIuSUe1A+7s6xXswBvlnswR4ISIAOQUIsPFvunYdIyro87Bg0pXobH2VQJ4qMC2huY1jBWj/HXWSIWF8Bl2lkKacmUUwEbuG4qjMNfj5okZYDNP8Ee2rKVp2N8FlLH8HjQwQbqtGeVEyiBXgzRZ2svm250XSpmCOuPd6F6aW7baFKakcar+LeS/HNyE+UXEs3Gas56M/sqBjEMHypE71wRYC KLBxsSmq 9pB2JXeaKUXfahJDKoPRD3H8qeTasMGO9cUnqtUaVnvOt29w4k2abxFSDyhZ/zAjCLFSu21i+HGbIj1CY1W8KuXnqteMBbftctlvgBFlsb8cASpwbM+j/VWUbsJHMdduxLte/UmCq2tYB/nGZHMJRUhMCLd9+nIj3Ap3MYZGGTCIqMLmVnJK2vn+knJy/nI3OzdKC+12iDtPMhPN3rW6PuCRH6NyTOectiaPioBwKB62vEI4lBVnhqfFMag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Perform the check at the start of do_read_fault(), do_cow_fault() and do_shared_fault() instead. Should be no performance change from the last commit. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 11b337876477..627a2abb969b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4661,6 +4661,11 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) vm_fault_t ret = 0; struct folio *folio; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + /* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or @@ -4689,6 +4694,11 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (unlikely(anon_vma_prepare(vma))) return VM_FAULT_OOM; @@ -4729,6 +4739,11 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) vm_fault_t ret, tmp; struct folio *folio; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; @@ -4775,11 +4790,6 @@ static vm_fault_t do_fault(struct vm_fault *vmf) struct mm_struct *vm_mm = vma->vm_mm; vm_fault_t ret; - if (vmf->flags & FAULT_FLAG_VMA_LOCK){ - vma_end_read(vma); - return VM_FAULT_RETRY; - } - /* * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ From patchwork Mon Jul 24 18:54:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1176CC001DF for ; Mon, 24 Jul 2023 18:54:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B9DF6B0074; Mon, 24 Jul 2023 14:54:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 66A206B0075; Mon, 24 Jul 2023 14:54:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50AF18E0001; Mon, 24 Jul 2023 14:54:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4319C6B0074 for ; Mon, 24 Jul 2023 14:54:20 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1A03C1409D2 for ; Mon, 24 Jul 2023 18:54:20 +0000 (UTC) X-FDA: 81047405880.03.248077A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 761C5180018 for ; Mon, 24 Jul 2023 18:54:18 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dTwfpe4C; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v1353Pyv++o8NWvrrZQCLEMMSP1M9ywuP5OB+aDM0kw=; b=nWmW1mc1HVAVzPQETFtsBsgqcbvdVLvblo869hB6311sWndSX3VRzUUbiAxsCuNyVQM8Cc gCiH0cfOVq7DoDdkJhwbTJQ4H+bXDf7gI0czI29frUmaxYzhzaHmAxzzcZsyEaU7kcTyCR pvWVGn/sC1zTqOk+byV5ory/Yd/Svgk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dTwfpe4C; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224858; a=rsa-sha256; cv=none; b=heVnopMbfSg2dWAwz95oStbQ0YAPcNUKDfEBwICZWFNtRz4bdV5lI4WnC3lvT9XkgvxlYR dKcsduEg2ENK1Eja+LSOOe4zFeFhdwMmWxkalHX75KMaagL9fkUgVuZ56oJElkVDvO7z15 yd4rTZyCLcHvJbHNEn/75vn+kdUeeBQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=v1353Pyv++o8NWvrrZQCLEMMSP1M9ywuP5OB+aDM0kw=; b=dTwfpe4CSf/wNXWMmkRXZpVHoZ vKf5kVMS43ZWZIFpaRKalQSU56M1zPmuvT0cTfRjsHG1ms+id4W5sGxNd+sE6EVlBjAaiNoNxDr1h Xonm4QEVzdVvQOSZTzKKLOlWi/7AHgQcxw+xBTzLLDTPshsCp14Xt1CAf3E711aCMTCmiUsbsyKRJ E7epN1PqMGdDThFGqBD/6NTEaQk9H9FaPMFkL5DFJJwMjPDD9zI0zKEXyQrh4HK4MTkWSaztP6OC+ hH67f752GSEuWd8RT90APXA9Co/fW4lHowfXBqJRfeXhMgCMSq4BVozxlTuI59Y7oVIbvi1uR9rCv csAytTjg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iRF-Mf; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal , Suren Baghdasaryan Subject: [PATCH v3 08/10] mm: Run the fault-around code under the VMA lock Date: Mon, 24 Jul 2023 19:54:08 +0100 Message-Id: <20230724185410.1124082-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 478jcjhb6fbmres4r4n48rxwwg5hbsi3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 761C5180018 X-HE-Tag: 1690224858-379604 X-HE-Meta: U2FsdGVkX18xKaOEnswsdy4HNugj53ID1KrVhZsHxaasHP7vE02D1kD+l53TJaouAb2OD7u+GF2w1H9bhYmy3YoHTVym5itMnuFEnTmvYZj6eT/HXNEei3dXcafVXDZIX3tK0l4gBU/mdHmd//a4+VuiM9B97cMjnu/kBTGUffIxfR8GWyF4K6JDoHLPRNaBlBrrauKwe7dpufNgHXZ+Ek68KQyj/JUHCMJoOo1/xCy9ffHbJg4WsXs15IGmNibxMjiBgKE+AZgoHUmFCNDNZENZQVSbcWyUo1+Yyfkt1XEsdUmZbC99kZE72PA/SuD5zkO/OF3zcTinv7UG/ahEcrc2YmIbzR6tPGiumkFe6H6ACG8aL2rVRDVZ9TiudaHtOwbQJy5l/RpXWRFI4s8xhIjAhtHCfPqr4vPp0u91outzJKXC52Hw4qSlwD5QzJp7KWO4MuOCdNqCmAQWMkDPIZcI7Kq6PxL267kblEVYRCrwlbH5kTz/udnj7JvmFLnOF0dJBajkEQnrc/2MSIFhACvWwuhmWkJDBgGoqDLLUNlqyAbEE1V5AunRgW2iIS3rfM2qpyHEY0TzAEMMhqmzqR0u4LGSPTyKT1qhNc3dWitUUoasSEo+tpZmuxZXgLQ48MCAUd/GWfdZN83Q1GAiCwGWgQUc6Ahg/gWXZz0af4S46LuBNb5xIGIc2MM3ZCM2mJJMG09SNjxtPHiEjLGRDzUy1i8Nm5eQFQgh7r1L6sA62iL63BOhXkR2w7yZ6D1GKI3ZZQwybghVmiqK5wk+hDnYr1FEc97xhtLmhL3rZxO1QCrhB9EJ0BvCZRz/bQoNiI8ib1SOm34LQnVJDE+D66ls1Qhkv3YXaOwIb0/aWfQyk0MkYisSw25d/XvFlVS11bXP9o0griTCeC3QRErGIxp4Di/F9mM3IkkOUDPUQE2Xcf9nFRAI8WD4fMC631yzpjVAzGWxLPsxSUcUWgS 5sxKtpJO BsTuQYfwQL+SfaJziY0Gj2iEqUMYLAAPBadGLA05PGtooi11QLtDRE0wJeCYWT9As9gn2+K3XVFs0FlA6QSB81Eu2LKJkwD20QzECjYrwAOLPfpia1NVfPvzfYp5w0zm8+pg0hkfDewjRSHK+B8fP/dcEf0n2n5HcH8V+Ww52CiBLvdVO6BP+6zFbvfdU48o1RDdlmO0cjRgDPZg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The map_pages fs method should be safe to run under the VMA lock instead of the mmap lock. This should have a measurable reduction in contention on the mmap lock. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 627a2abb969b..cf28afe7a416 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4661,11 +4661,6 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) vm_fault_t ret = 0; struct folio *folio; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - /* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or @@ -4677,6 +4672,11 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) return ret; } + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; From patchwork Mon Jul 24 18:54:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05C7DC0015E for ; Mon, 24 Jul 2023 18:54:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F90F6B007B; Mon, 24 Jul 2023 14:54:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57F596B007D; Mon, 24 Jul 2023 14:54:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 46F6E6B007E; Mon, 24 Jul 2023 14:54:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 394156B007B for ; Mon, 24 Jul 2023 14:54:27 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0A776120AD2 for ; Mon, 24 Jul 2023 18:54:27 +0000 (UTC) X-FDA: 81047406174.16.18942CB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id A5FB010000F for ; Mon, 24 Jul 2023 18:54:24 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=pq4q53ip; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224864; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bf2WSeVVnx5PtuK0GrdZC7nqaCiGfJc1owLWRc2d9S0=; b=5Bc+9n39OVQQWH+HI/BF0ZZNM2rb7ElirOUVhhNX+wvpIFUv/TgguEyvBhvzK4pMFd0Fxs XIIAijNpFFyNTqsJVNTmYLmvQkABW+sMDkmr08/0uzxmJP0INGTD23YBvhj8VMSc7/676d EmUPGG7CVyD0ukl0fy9H7HO2AXmZ144= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=pq4q53ip; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224864; a=rsa-sha256; cv=none; b=BT3z5CbNiQ2Ez4Z2b8o9+AU/Oav9Z8cZBSFAxximEiOZ1Z9+GHo1p35cK0mXVJ+3l1c2Sm XTFwa9P1EZ0Z35KTArmRtv/shP8/ayMh3LOgwBiJYj/2UB0bER4FoZKh0jrohFt8u92WTF 7XZ5ZQkTUfi1az1aSmATSWZXBvQDAmU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Bf2WSeVVnx5PtuK0GrdZC7nqaCiGfJc1owLWRc2d9S0=; b=pq4q53ipW8imewozPRVRm9SF+m RVM+pPZTkW+xlAdqR4U8etpaEe/I/DDBVIxPfL9YcKgLPN700wsZFK/8HQ5tiM5SH9mlMAQeOGFQA pCDpj4vAGjQDG2G8pb6Xmrrw2nge32+b/8RHXob2OgRXXA53Z7FI7GZWoNnSVpT3xUtrrRsaP4FR6 tK7rBBJ1LALKGEDtjsDaIcr8sxHVwqGwW0SYlCDqoxT5OW/qwUT1M9eB4glQiild/Zb7Pqwbx9PUp /ECOtnxFdu74Lg9qhUaT959BflWAxgCMzx3rOnmDtJjeTqakbEK6eH6mVpXBfeR3DdLPiqoB/07ik DDyIR+Uw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iRH-PM; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v3 09/10] mm: Handle swap and NUMA PTE faults under the VMA lock Date: Mon, 24 Jul 2023 19:54:09 +0100 Message-Id: <20230724185410.1124082-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A5FB010000F X-Stat-Signature: ech6goq6xd8xnkmbnqz5fhgo1xjxx859 X-Rspam-User: X-HE-Tag: 1690224864-620869 X-HE-Meta: U2FsdGVkX1/yfmOYwkigWEit0VSPYAUJDW30HecbrMpxxSDMJj71DYCkqLwGAEeSVF1MjaDzW2OtjsC9mnW5CO8n9T+gIIlgY3lEqbE8pEw4liEFRzkLxB/LCqH81w8h1II8EP07/fBaMdmXPlycOt9lEAdNzCK1dbHLgKwAPO55utBppb2gWsLiEvSVAtuYfZZYqgmFFzdfVStzP6LaiPZM8EyrX20Qf7lkqYVce+m7idlC7Wqo7dUzYZlIwS6dBBrwy8NWDQNRaRUVoJYcXqcG+cGXOJsSGlRVcZBhRsKdOY6TRF14sertLPIYUcZZ9uteWhfQzmOqePWDovoldO3+oGJN06VhM4fjY/mSgIbtPbonSypVpGEH6pOkbdO5HUW5xrnj8RPMzpW5RX7hvDnVZiiziPv1iH4See9MX0g1BEMi6FF5u5udGLB9GEiIRcR+B6Pj6zVIPnixD3aY4cXHAaMU6DKcFFtSeK1+wFWPEo9p19Me1d4jZKPKbUXOR94yGUz2bhbHyNyArSP0NwwgJav+BJjpf5Vvw/mo0y9NdsKIuVrKALLcmkMNpDfzL9ThcL1XnXPiFrtzmawp/FAYCEPvittoVW8X6FotwRt52jSaC4Be3ykZyE6jcsCh03ZlV2+k/ibDAx09oVGT2qfx2LTtUofKR5coNtLNINT375OWEoWbY0JJL230LO5A3Y5MJw4yJ2lE75k+k7uq6MhQ4ZPdGCy08FjG8STYOwh2Z2EHotHJRhXcLVnvOJJwIRQ/l1cjaG8c57DmwrA96HQT2e9LFD9PccZz4wK3pur5UoQ5N+LupkRpM7ZAPU9y4eBkFyd492EZ8ufZQowzE5J2Iqu273+1IR3HCynFxKC2c+2BUr5ywVxzqGzlURO8YqCWRcDQvzwPb6xFBovtetk7YC5/zTQaWhKLh5UNH4Fa05auURE3wMsjvd1d7hyyYNB0cri6b+HvgdnXVZs NJdfferk 3DJMVcL1RYzKt2aoyucNPk592np7BqYQ/ZbsSkGtxarxU0Y6SktWzqIgOQ9O5AAlQNv2YcPfSccAW6+8kHO9e/+2W9+Vg9e/Jaj/GP+MT5WHvcmFt3GtsbCobD47Kx25roPn0tFRwIics4rsTEHLzdVSKXGtOk8Tvh2Xyqbh7p/JDTF661x5yUWyY+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the FAULT_FLAG_VMA_LOCK check down in handle_pte_fault(). This is probably not a huge win in its own right, but is a nicely separable bit from the next patch. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index cf28afe7a416..ad77ac7f8c9b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5107,18 +5107,18 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (!vmf->pte) return do_pte_missing(vmf); - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { - pte_unmap(vmf->pte); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { + pte_unmap(vmf->pte); + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + spin_lock(vmf->ptl); entry = vmf->orig_pte; if (unlikely(!pte_same(ptep_get(vmf->pte), entry))) { From patchwork Mon Jul 24 18:54:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFCE4C04A6A for ; Mon, 24 Jul 2023 18:54:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 005196B0075; Mon, 24 Jul 2023 14:54:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF7576B0078; Mon, 24 Jul 2023 14:54:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFAEF8E0001; Mon, 24 Jul 2023 14:54:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B94BB6B0075 for ; Mon, 24 Jul 2023 14:54:21 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8C95716017E for ; Mon, 24 Jul 2023 18:54:21 +0000 (UTC) X-FDA: 81047405922.26.815B95D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id C1DF040013 for ; Mon, 24 Jul 2023 18:54:19 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IfZBEC2b; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224859; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q+DaNTNbIUG3q6eDPG2po4w54RCX1zIfiIR5STu1uiQ=; b=2rZJsO2AtNOaE0A0qBSgZGLjW8Hp3QcpqMyzBE75LJfjCdpsF5Xei4GQeb70oZ5XSu+lts hOz/CeToewiii2oZkU5guUG3vA23jnB3QcKGJtSUAXQC98nbIlZtAq0KcFhttf9arARahM Yd6wH6UUYDDfJ2i9wRYsDDSqTG02RVU= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IfZBEC2b; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224859; a=rsa-sha256; cv=none; b=DVVY/RepXUP+0dsag3YQMVUiFkbVvz2l59wX+HxkydTizd2KdyyUDrPTdVLncDJeNGqtoV MBdWF+zvvnKCilYEouY32ALHGeK/A4j4B88Ql41Ri6zA9FqPNi1z/upz2lyk55XGuyPd+L 3UC7s6IIg1b6wzhAL6HGNrgaY2BmxxQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Q+DaNTNbIUG3q6eDPG2po4w54RCX1zIfiIR5STu1uiQ=; b=IfZBEC2bafAFTQbS89y7UijEBE RWmQ1Ds4r4Jychulds5YnYRb3I0od+yIzxGEjRBg/sMoaW8Sgo39q598ZbCrb8yFJjLhEXLbKGw2u m6ON7coI8Vkyui8eqwVD3MAAI4b1SEwfQnyn3FGv9SQx1ziGML5H3GK5zcH+sBTO4lmFkaq0PKQyx xzo+xp/C7beutQOsWwV77qUovD+qADlO+56E/Dwyj29cWGg4lg7hX6Mo01FXPuux0JJUirC3kWWWp PlgvLemMPnSu3miVyRvcWhE2qw/Nw7f8SnbiBBb69OcutacNlt/2UeeoQC1+cppK2G6+2AMHfS7AU qU4S7HIA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iRJ-S7; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v3 10/10] mm: Handle faults that merely update the accessed bit under the VMA lock Date: Mon, 24 Jul 2023 19:54:10 +0100 Message-Id: <20230724185410.1124082-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: o5ya8gj1ijwd5jwodwj5rh6iopesjzee X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C1DF040013 X-HE-Tag: 1690224859-643432 X-HE-Meta: U2FsdGVkX18hyzHDhzOmPOeLDEvzHPG0WqTps+ldvZkUesMJyTAdBFdKo12AyAgRXZFDVJ+JJQ190ya4mRI9DUZkMMX/cHfEON7oKHrAvGaC7I0A949miPwZcQfv5TDcm8rnumbdOXdxD4gl7Rse7BFSsyE/ha10Rc51/axfIdvluzQ1sfHjNbhfeZ06PgoCrYAbD0EfMoMaGcXc/XCVisxtDGf6S3N524MpC0f+dd6LOcJ3FXMYsDz2vG0kbpb325IqzkZpGGsGeDo906Dechr6W4Egp+kzjqx0e+aVP+td+wsoDw8qX3AODMjN8lqa63VRwLdv8raDmpQ6nyOLEtUCyPzxpJfmO1N+mfmomp488QFRb/tHo+VDDBdRkxwpoqdoNpCOBdb9m+9PFcgRieWdXIo4Q2RX1plHRFznq4m3+oSoAj+S9F0VQ1iVqp65YQTsbHeqtDo+uc6tGAi1ZQOcUy75TC3GNPuN0wKIG/Ia5owGx7QXXzerb+5FzcDnTdWmTtUmi9CPTmWnKBARCDB0jM4MHVqDdWj5GZABgmz76r2u1hu5ib7AfOB3n0KLn/9sqPvsrtpszL7ll9VTS2JLrCXHFzveSbrW6TyacFD29HJUrx7m3zdYFYdL5pYkNqtZ+lSRo5v/KRQG2tNzUXmltYEyzodoO/iWWO+dufqUufDGv5yVbMxl6wzBsi511TAAp2t4Lz2FiTM7J32Ezulm3bPHxXUwa5Vq3tsylf3IsWfi5b2Hbnegr9X4VmM+QLZVU3aG/xu9/ma3pNtVGgj2+CQ4neqzzJvsF6RKwxUBxCHBAqo+LJ+9sfGTmKkPaZfV5QK4IYanC9v+6JCjuoLvaMMsEvBT++mSqRdn1BwhDc2KLi6WkOFGCxGYqrY+e/UsgyrvpmKAfcToBAPt9ZYLPWQ/syj8yPfrwIu5rlz39gLoujZuY+phqCnMI8WK1O8O2gMhWADq9PgJTlI o7lL+6bm Tr8r4p64jCl2RYeelSfPeTVvwO/w5sspcPewyC+dLtnd9QTvoBgmvUkh3p7l0AmqgzoYmobF/p4N684PTDHqR962ZakJArsO38uQtpM/OFsG42xSvOuyhgMbVtw/rvODmbg2VDtPuo37aD/Z2QaKG3tVNK+uZxlih2DrRdP/SDweh5QdmHUYGf251qw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move FAULT_FLAG_VMA_LOCK check out of handle_pte_fault(). This should have a significant performance improvement for mmaped files. Write faults (on read-only shared pages) still take the mmap lock as we do not want to audit all the implementations of ->pfn_mkwrite() and ->page_mkwrite(). However write-faults on private mappings are handled under the VMA lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ad77ac7f8c9b..20a2e9ed4aeb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3393,6 +3393,11 @@ static vm_fault_t wp_pfn_shared(struct vm_fault *vmf) vm_fault_t ret; pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + vmf->flags |= FAULT_FLAG_MKWRITE; ret = vma->vm_ops->pfn_mkwrite(vmf); if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)) @@ -3415,6 +3420,12 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) vm_fault_t tmp; pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + folio_put(folio); + vma_end_read(vmf->vma); + return VM_FAULT_RETRY; + } + tmp = do_page_mkwrite(vmf, folio); if (unlikely(!tmp || (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) { @@ -5113,12 +5124,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) { - pte_unmap(vmf->pte); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - spin_lock(vmf->ptl); entry = vmf->orig_pte; if (unlikely(!pte_same(ptep_get(vmf->pte), entry))) {