From patchwork Wed Apr 7 01:45:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 12186493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AC84C433B4 for ; Wed, 7 Apr 2021 01:46:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE6D361165 for ; Wed, 7 Apr 2021 01:46:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE6D361165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=lespinasse.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D31FD8E0016; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BCE08E000F; Tue, 6 Apr 2021 21:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCD368E0009; Tue, 6 Apr 2021 21:45:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id A19BD8E000C for ; Tue, 6 Apr 2021 21:45:09 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6436B4408 for ; Wed, 7 Apr 2021 01:45:09 +0000 (UTC) X-FDA: 78003877938.36.5439008 Received: from server.lespinasse.org (server.lespinasse.org [63.205.204.226]) by imf25.hostedemail.com (Postfix) with ESMTP id 7BAAA600010B for ; Wed, 7 Apr 2021 01:45:07 +0000 (UTC) DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-ed; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=1pcgL6msPKrfTRZ1fn8ZYvCmKEa5WSF8jkX3M1HqC8Q=; b=hQvw4qVAk9RjFRT0YurEoXBUfSyibVWfDTUoCsMeXQbkWLv85cnNBgYIduwwAJSYkk9nX dBueIiIj6nFh/usBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lespinasse.org; i=@lespinasse.org; q=dns/txt; s=srv-11-rsa; t=1617759903; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : from; bh=1pcgL6msPKrfTRZ1fn8ZYvCmKEa5WSF8jkX3M1HqC8Q=; b=iNI/FbYx7Xyn86mRVSOET3lACzR7zhpyLEsLKAwjwO+52vhn4w0U0kT1L0VFRgTFDJbI2 H+KlbaIoIomCY6TbjUUQPaYfgiJ6rZVCYGClP+LomXZzhbY1WTNzfv3hZEwwzs92mNG7C+v GP9g4TVH4xBdqqSwehvTgygusMqE2oUqC2cGKIhgfVDEQJW/t4ekzDi26VXIW9QP8njhll6 W0l7WVqhhxQ27zpWV/j/6rOgMkyYz/GQG816a5JEhhLkMGlGgeM+ljwkBxELBaJJrztnNGD dspGKP7paDbt7nX+v18rbCFEypfvuS9gOBP0r6AEOcN+wfVRX8sB8ybKE+HA== Received: from zeus.lespinasse.org (zeus.lespinasse.org [IPv6:fd00::150:0]) by server.lespinasse.org (Postfix) with ESMTPS id 7880A1606A3; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) Received: by zeus.lespinasse.org (Postfix, from userid 1000) id 673BE19F320; Tue, 6 Apr 2021 18:45:03 -0700 (PDT) From: Michel Lespinasse To: Linux-MM Cc: Laurent Dufour , Peter Zijlstra , Michal Hocko , Matthew Wilcox , Rik van Riel , Paul McKenney , Andrew Morton , Suren Baghdasaryan , Joel Fernandes , Rom Lemarchand , Linux-Kernel , Michel Lespinasse Subject: [RFC PATCH 37/37] arm64/mm: attempt speculative mm faults first Date: Tue, 6 Apr 2021 18:45:02 -0700 Message-Id: <20210407014502.24091-38-michel@lespinasse.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210407014502.24091-1-michel@lespinasse.org> References: <20210407014502.24091-1-michel@lespinasse.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7BAAA600010B X-Stat-Signature: k55dodh4an7yp46z9bxhckqifxbneuwy Received-SPF: none (lespinasse.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=server.lespinasse.org; client-ip=63.205.204.226 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617759907-280984 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt speculative mm fault handling first, and fall back to the existing (non-speculative) code if that fails. This follows the lines of the x86 speculative fault handling code, but with some minor arch differences such as the way that the VM_FAULT_BADACCESS case is handled. Signed-off-by: Michel Lespinasse --- arch/arm64/mm/fault.c | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f37d4e3830b7..3757bfbb457a 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -530,6 +531,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, unsigned long vm_flags = VM_ACCESS_FLAGS; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); + struct vm_area_struct *vma; + struct vm_area_struct pvma; + unsigned long seq; if (kprobe_page_fault(regs, esr)) return 0; @@ -564,6 +568,53 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); + /* Only try spf for multithreaded user space faults. */ + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto no_spf; + + count_vm_event(SPF_ATTEMPT); + seq = mmap_seq_read_start(mm); + if (seq & 1) { + count_vm_spf_event(SPF_ABORT_ODD); + goto spf_abort; + } + rcu_read_lock(); + vma = find_vma(mm, addr); + if (!vma || vma->vm_start > addr) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_UNMAPPED); + goto spf_abort; + } + if (!vma_can_speculate(vma, mm_flags)) { + rcu_read_unlock(); + count_vm_spf_event(SPF_ABORT_NO_SPECULATE); + goto spf_abort; + } + pvma = *vma; + rcu_read_unlock(); + if (!mmap_seq_read_check(mm, seq, SPF_ABORT_VMA_COPY)) + goto spf_abort; + vma = &pvma; + if (!(vma->vm_flags & vm_flags)) { + count_vm_spf_event(SPF_ABORT_ACCESS_ERROR); + goto spf_abort; + } + fault = do_handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_SPECULATIVE, seq, regs); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } + if (!(fault & VM_FAULT_RETRY)) + goto done; + +spf_abort: + count_vm_event(SPF_ABORT); +no_spf: + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -604,6 +655,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned int esr, } } mmap_read_unlock(mm); +done: /* * Handle the "normal" (no error) case first.