From patchwork Fri Apr 30 19:52:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12234215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 790EBC433B4 for ; Fri, 30 Apr 2021 19:54:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0E61F61476 for ; Fri, 30 Apr 2021 19:54:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E61F61476 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 760A56B009C; Fri, 30 Apr 2021 15:52:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B7BC56B0096; Fri, 30 Apr 2021 15:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CA226B0078; Fri, 30 Apr 2021 15:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0181.hostedemail.com [216.40.44.181]) by kanga.kvack.org (Postfix) with ESMTP id BAA516B0078 for ; Fri, 30 Apr 2021 15:52:38 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6C8D145B3 for ; Fri, 30 Apr 2021 19:52:38 +0000 (UTC) X-FDA: 78090080796.23.0EDECAE Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf19.hostedemail.com (Postfix) with ESMTP id 59BD190009F6 for ; Fri, 30 Apr 2021 19:52:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-ed; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=eULoJwQoISRJDyWh6sQIkPbJRau/grHqDfGiWr2ndBc=; b=hNOAqV+Bi3m/G9I1+p/zbTgoypBTOIz9B9w7EuLXDESQcB7YB+wNpudjM89Jlv4pGyn1d eppGrtsnZHr/cNRBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-14-rsa; t=1619812353; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=eULoJwQoISRJDyWh6sQIkPbJRau/grHqDfGiWr2ndBc=; b=LLcNJN4Q0cNreUi2QNRCmvi9zlH+OSkbXB/q3IC4eYa3yeOiFOzGf7On1WAFSRGdYS4Hy jDWbNZpT8l72PTG6DCd9W22CDTlebRhwdA69DEVWWQB8UZGfktRR0w/+VDx3geYu4Qt37My vBe5z2c36M/iQlLy1RWER9bUFrR63ZNDfJnizhjBtrQsR5QMmlaz2b/zziuf959oknLxTF6 XHkWLqzy4v3htaBTYFcJVZqEnFtjBbFHdLnnsvn1gxQj7eIjoYN3F62t5hvwB9Ky2p/2+Gh S6Lvbcn9dW4G2XDxg5zPVBk62nFgrXWu/01L/iNX///ZO6Nc3ebsipAsTkeQ== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id DBD20160372; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id CCB6D19F523; Fri, 30 Apr 2021 12:52:33 -0700 (PDT) From: Michel Lespinasse To: Linux-MM , Linux-Kernel Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Andy Lutomirski , Michel Lespinasse Subject: [PATCH 31/31] powerpc/mm: attempt speculative mm faults first Date: Fri, 30 Apr 2021 12:52:32 -0700 Message-Id: <20210430195232.30491-32-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210430195232.30491-1-michel@lespinasse.org> References: <20210430195232.30491-1-michel@lespinasse.org> MIME-Version: 1.0 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=lespinasse.org header.s=srv-14-ed header.b=hNOAqV+B; dkim=pass header.d=lespinasse.org header.s=srv-14-rsa header.b=LLcNJN4Q; dmarc=pass (policy=none) header.from=lespinasse.org; spf=pass (imf19.hostedemail.com: domain of walken@lespinasse.org designates 63.205.204.226 as permitted sender) smtp.mailfrom=walken@lespinasse.org X-Stat-Signature: pg7ue6qece5es3qxgyxnid9ctfnrgg85 X-Rspamd-Queue-Id: 59BD190009F6 X-Rspamd-Server: rspam05 Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619812327-296063 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. This follows the lines of the x86 speculative fault handling code, but with some minor arch differences such as the way that the access_pkey_error case is handled Signed-off-by: Michel Lespinasse --- arch/powerpc/mm/fault.c | 65 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index bb368257b55c..d7c820751a58 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -398,6 +398,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, int is_write = page_fault_is_write(error_code); vm_fault_t fault, major = 0; bool kprobe_fault = kprobe_page_fault(regs, 11); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + struct vm_area_struct pvma; + unsigned long seq; +#endif if (unlikely(debugger_fault_handler(regs) || kprobe_fault)) return 0; @@ -450,6 +454,64 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + + /* + * No need to try speculative faults for kernel or + * single threaded user space. + */ + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); + goto spf_abort; + } + rcu_read_lock(); + vma = find_vma(mm, address); + if (!vma || vma->vm_start > address) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_is_anonymous(vma)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) + goto spf_abort; + vma = &pvma; +#ifdef CONFIG_PPC_MEM_KEYS + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } +#endif /* CONFIG_PPC_MEM_KEYS */ + if (unlikely(access_error(is_write, is_exec, vma))) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } + fault = do_handle_mm_fault(vma, address, + flags | FAULT_FLAG_SPECULATIVE, seq, regs); + major |= fault & VM_FAULT_MAJOR; + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); +no_spf: + +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -525,6 +587,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, } mmap_read_unlock(current->mm); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault);