From patchwork Fri Jan 28 13:09:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12728532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E337C433F5 for ; Fri, 28 Jan 2022 13:11:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0D456B0093; Fri, 28 Jan 2022 08:10:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 604D36B00A2; Fri, 28 Jan 2022 08:10:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A186E6B00A7; Fri, 28 Jan 2022 08:10:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id 714F66B00A3 for ; Fri, 28 Jan 2022 08:10:14 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2EF82181C49A5 for ; Fri, 28 Jan 2022 13:10:14 +0000 (UTC) X-FDA: 79079729148.19.AFC033A Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf21.hostedemail.com (Postfix) with ESMTP id 04D681C000D for ; Fri, 28 Jan 2022 13:10:12 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-52-ed; t=1643375407; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=7ZwO3JSFTMu+I+SD7pVtiWOQ8TjLyD2OehGAIC7WXjM=; b=kXZaaQKnfFfdx1WcuV/gsR3sFHyjkkWzgVlVIxl61xzaSmlhPKCw8k9+0tXt42thV1hgh 76J7O3GfBptE7fSCg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-52-rsa; t=1643375407; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=7ZwO3JSFTMu+I+SD7pVtiWOQ8TjLyD2OehGAIC7WXjM=; b=N+8pg6EwjJQ3+6Y7V4x16KZEvrtmwYS+Kkac+I8wPHurF9yISlUf9mrEqxGCFMpE8iKWo af//9oNNzfjUKvmlV90xDVOpnS9w6rIknF2VlARdpSg5gTndvdvI3korqttpbVfl6IUxQae O3VrDU2+4MMBPDgX1SU44DD1De1BaiokD4lkS+M6QKgl6CR1s/U5L7KYI8dBXx6sq7vxC4q 3pMrI0M/abW1VuAe5ZzAMd9o1YzyS9vkrsGGZhxx0gK2cmOrIUpP3cW3a+eNfZ9eN9Q9WZO hd1vFAxrPwIytcuycY+dTWvMIXOnOummJyesV0+o3FKbRWSh/AzaaJFY6OGw== Received: from zeus.lespinasse.org (zeus.lespinasse.org [10.0.0.150]) by server.lespinasse.org (Postfix) with ESMTPS id 34F95160AC3; Fri, 28 Jan 2022 05:10:07 -0800 (PST) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 1422920F30; Fri, 28 Jan 2022 05:10:07 -0800 (PST) From: Michel Lespinasse To: Linux-MM , linux-kernel@vger.kernel.org, Andrew Morton Cc: kernel-team@fb.com, Laurent Dufour , Jerome Glisse , Peter Zijlstra , Michal Hocko , Vlastimil Babka , Davidlohr Bueso , Matthew Wilcox , Liam Howlett , Rik van Riel , Paul McKenney , Song Liu , Suren Baghdasaryan , Minchan Kim , Joel Fernandes , David Rientjes , Axel Rasmussen , Andy Lutomirski , Michel Lespinasse Subject: [PATCH v2 26/35] mm: implement speculative handling in wp_page_copy() Date: Fri, 28 Jan 2022 05:09:57 -0800 Message-Id: <20220128131006.67712-27-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220128131006.67712-1-michel@lespinasse.org> References: <20220128131006.67712-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 04D681C000D X-Stat-Signature: zjxa6ipmrn4tou6dt88ttfrxpmh4q4ee Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-52-ed header.b=kXZaaQKn; dkim=pass header.d=lespinasse.org header.s=srv-52-rsa header.b=N+8pg6Ew; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf21.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Rspam-User: nil X-HE-Tag: 1643375412-188278 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change wp_page_copy() to handle the speculative case. This involves aborting speculative faults if they have to allocate an anon_vma, read-locking the mmu_notifier_lock to avoid races with mmu_notifier_register(), and using pte_map_lock() instead of pte_offset_map_lock() to complete the page fault. Also change call sites to clear vmf->pte after unmapping the page table, in order to satisfy pte_map_lock()'s preconditions. Signed-off-by: Michel Lespinasse --- mm/memory.c | 42 +++++++++++++++++++++++++++++++++--------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 73b1a328b797..fd8984d89109 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3087,20 +3087,27 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) pte_t entry; int page_copied = 0; struct mmu_notifier_range range; + vm_fault_t ret = VM_FAULT_OOM; - if (unlikely(anon_vma_prepare(vma))) - goto oom; + if (unlikely(!vma->anon_vma)) { + if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + ret = VM_FAULT_RETRY; + goto out; + } + if (__anon_vma_prepare(vma)) + goto out; + } if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_page = alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!new_page) - goto oom; + goto out; } else { new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!new_page) - goto oom; + goto out; if (!cow_user_page(new_page, old_page, vmf)) { /* @@ -3117,11 +3124,16 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) } if (mem_cgroup_charge(page_folio(new_page), mm, GFP_KERNEL)) - goto oom_free_new; + goto out_free_new; cgroup_throttle_swaprate(new_page, GFP_KERNEL); __SetPageUptodate(new_page); + if ((vmf->flags & FAULT_FLAG_SPECULATIVE) && + !mmu_notifier_trylock(mm)) { + ret = VM_FAULT_RETRY; + goto out_free_new; + } mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, vmf->address & PAGE_MASK, (vmf->address & PAGE_MASK) + PAGE_SIZE); @@ -3130,7 +3142,11 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) /* * Re-check the pte - we dropped the lock */ - vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + /* put_page() will uncharge the page */ + goto out_notify; + } if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { if (old_page) { if (!PageAnon(old_page)) { @@ -3205,6 +3221,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * the above ptep_clear_flush_notify() did already call it. */ mmu_notifier_invalidate_range_only_end(&range); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + mmu_notifier_unlock(mm); if (old_page) { /* * Don't let another task, with possibly unlocked vma, @@ -3221,12 +3239,16 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) put_page(old_page); } return page_copied ? VM_FAULT_WRITE : 0; -oom_free_new: +out_notify: + mmu_notifier_invalidate_range_only_end(&range); + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + mmu_notifier_unlock(mm); +out_free_new: put_page(new_page); -oom: +out: if (old_page) put_page(old_page); - return VM_FAULT_OOM; + return ret; } /** @@ -3369,6 +3391,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return wp_pfn_shared(vmf); pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; return wp_page_copy(vmf); } @@ -3407,6 +3430,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); + vmf->pte = NULL; return wp_page_copy(vmf); }