From patchwork Fri Oct 6 19:53:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13411940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8256E94108 for ; Fri, 6 Oct 2023 19:53:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A22E80014; Fri, 6 Oct 2023 15:53:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02AA180008; Fri, 6 Oct 2023 15:53:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBFE880014; Fri, 6 Oct 2023 15:53:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CFEA880008 for ; Fri, 6 Oct 2023 15:53:36 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A708D1CA565 for ; Fri, 6 Oct 2023 19:53:36 +0000 (UTC) X-FDA: 81316086432.05.A63B616 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 11C1510000C for ; Fri, 6 Oct 2023 19:53:34 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WPOQ3j8I; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696622015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R1cgnWd1rwQWVhfoW+i9aIbpeLTl++h04BYfoZW8rgE=; b=nv6NCqFy8QgnoeH1EF/zjdFU4qb08q0NwkCOE07Cc7eZ6FfkoovgdBlSApgb8zH10N6G5V LKhA0q6uD3/5Bau1si2tj+dvwhWsw5MvPDgeNH6uYLWx8UOKrQaNFYuqLHQLKVFz1Bh8Ib DwEOvfNYwlRkZdriQxIX5JJOaDAW05k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696622015; a=rsa-sha256; cv=none; b=hvinOGCMxXTLYK29YaePHbzZWNLT40HWkQI4Cz20/iV2kt82HqaYEHsvjE94fEYn3mQCaP MFY/q+xGwiwbXfeE+EybFnOBZF1jXeLnPO4U2lKMKSr34WzCt9u+Y5PZWsKVkk97DFMX4Q CGuYHCpDCTO4g3bGbAXkhxN+gfa34hc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WPOQ3j8I; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=R1cgnWd1rwQWVhfoW+i9aIbpeLTl++h04BYfoZW8rgE=; b=WPOQ3j8I8UN/sxknZsBmPVtDO5 IqHWplA1iyqL2NR3MriNYZfRR9AujPlQQklM1j9pLdVTha+sM9wvd/FIH3kbf3uxsxKAJUtTilBDX MPzpKfgfEWrJIQeogojlkHmYPvJw+QDS45N5K0mhvoWLPgY/P0HVwZMoaXVt7ocIQVC4CjD+GZXGH z2qUmJITqTRgno+Ls54OudvQCYf3kNuaDOu+yODUiq+CLt7VE7ZLOW0a2poIrIb19gWEoSBbKzcbH XyrLj0nYgP+dv9M3Pco1wL4EQuJJ2ORGUt7VRVq4rctpnilvWFvFJ64xER+UaIR59QlK28IJEZNu2 WjVwNQwQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qoqsy-00H9GU-Jj; Fri, 06 Oct 2023 19:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Suren Baghdasaryan Subject: [PATCH v2 2/6] mm: Call wp_page_copy() under the VMA lock Date: Fri, 6 Oct 2023 20:53:14 +0100 Message-Id: <20231006195318.4087158-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231006195318.4087158-1-willy@infradead.org> References: <20231006195318.4087158-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: y5gdj3hywzutfkoix78dnyesiqywmme5 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 11C1510000C X-Rspam-User: X-HE-Tag: 1696622014-479231 X-HE-Meta: U2FsdGVkX1+EPBy7+comaT1r8/qqphTy47ne/WepgpQZC3dZBDZURGd7POIpMejhCZUDVecW0UWSMTo80kCF39ILo6TgTmVy1ffr8DPA3hmP0fFTYl/zropqDtAfG3Tw8UcfPAJOIZ/COWPQ/io1Qe+2xe6wwR+uNwpDFJj+cM6PhvydaIMAFqPaMFXbXJGpVEyNP1TwzTDqjxoaOwD2u2lJ0gT5T4+2CHfuO/yjDZ33oYsIDY5n6OmoOS1kHT0+LQJbqAEzDfehAIg8dfz3sk6KXUDTwjWEkkjvSpyaIQ3zFDCjMouwmMECgr/uqeL9Gju8wRYops3kbYqY4aSaHAGhoE8NjVBDwmnwHZVdakiPQ6l3+ck6/jxzXGl733aaehMVwqmYrLmPHyKVNgFvVu6mTohPJLyAsLGVwmmuk29rLy1KP+7So8NHswZ+0Am7Ee+fgKnxeubj2nFn8StbAj08nb6emMt9UoefRLFVFuHBeQAk6HPlA/sPGK3BQREZBQz2BWkwMiAY7LvPctTz1ye+igX0rFKqAzWDX/xPbELKfNHkDePVZZLqyKYZQse6503fmgdraSD8onT4mMmk+5k0txEeFJ/0VgwSE/TLzf8ZFAUlMG/P92beb4mAXZShqRXGKxabZ5vle1rwTbVBs8uOUWnX83mDQ27TbIEpnV81wYJjkC8t93fsKJXnHC8ZcFvDxTEh2GRop0slSH6ylS5oKdMJbxnOhLe6MsfVaw1CiaqD7oNYtA5uilqK26pxoFAjNxcS6wLjcciRKpkuMVb1ldQZHrHawmrlLTOh1ThGbZUFBgRdm96PWx8SpWZkSgHy1N/Lc/V3aVE13iTKnRKeI28j1LwsPom5i+5S5dJQFgNQ5+zwo6auFxAIxHYoXF6O05dy0jerRUccdFT4vN7G1tfTciu6XD209wN90CcmssGV/YXUQgXUyPY8Rp777X+gMmSheSoSmTLCzI+ j3qZzor+ HTzmMEGUBZm2DNun1gPsAccg/HtmkemSfrWbCVHuvWboHlGHVwK3Rg+uYa0OsQq/YrxC9CA/sFPEkHAVkx0y0At2gjnyTi0xV8py6LBxYhHP3PadfYC05QYCBpaUzoALgxoRd1tXJIxfh2vljyo2gu/fh7DbNCk8NRDc19h7qAPFLBmcfEBDBRdCTAaX4HlkvJvpv4S6PH9UV2Aw8e74V4/7wY01VegxPj12w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is usually safe to call wp_page_copy() under the VMA lock. The only unsafe situation is when no anon_vma has been allocated for this VMA, and we have to look at adjacent VMAs to determine if their anon_vma can be shared. Since this happens only for the first COW of a page in this VMA, the majority of calls to wp_page_copy() do not need to fall back to the mmap_sem. Add vmf_anon_prepare() as an alternative to anon_vma_prepare() which will return RETRY if we currently hold the VMA lock and need to allocate an anon_vma. This lets us drop the check in do_wp_page(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Suren Baghdasaryan --- mm/memory.c | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 97f860d6cd2a..cff78c496728 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3042,6 +3042,21 @@ static inline void wp_page_reuse(struct vm_fault *vmf) count_vm_event(PGREUSE); } +static vm_fault_t vmf_anon_prepare(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + + if (likely(vma->anon_vma)) + return 0; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (__anon_vma_prepare(vma)) + return VM_FAULT_OOM; + return 0; +} + /* * Handle the case of a page which we actually need to copy to a new page, * either due to COW or unsharing. @@ -3069,27 +3084,29 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_t entry; int page_copied = 0; struct mmu_notifier_range range; - int ret; + vm_fault_t ret; delayacct_wpcopy_start(); if (vmf->page) old_folio = page_folio(vmf->page); - if (unlikely(anon_vma_prepare(vma))) - goto oom; + ret = vmf_anon_prepare(vmf); + if (unlikely(ret)) + goto out; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); if (!new_folio) goto oom; } else { + int err; new_folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); if (!new_folio) goto oom; - ret = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); - if (ret) { + err = __wp_page_copy_user(&new_folio->page, vmf->page, vmf); + if (err) { /* * COW failed, if the fault was solved by other, * it's fine. If not, userspace would re-fault on @@ -3102,7 +3119,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) folio_put(old_folio); delayacct_wpcopy_end(); - return ret == -EHWPOISON ? VM_FAULT_HWPOISON : 0; + return err == -EHWPOISON ? VM_FAULT_HWPOISON : 0; } kmsan_copy_page_meta(&new_folio->page, vmf->page); } @@ -3212,11 +3229,13 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) oom_free_new: folio_put(new_folio); oom: + ret = VM_FAULT_OOM; +out: if (old_folio) folio_put(old_folio); delayacct_wpcopy_end(); - return VM_FAULT_OOM; + return ret; } /** @@ -3458,12 +3477,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return 0; } copy: - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma->anon_vma) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - vma_end_read(vmf->vma); - return VM_FAULT_RETRY; - } - /* * Ok, we need to copy. Oh, well.. */