From patchwork Fri May 4 06:57:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mahendran Ganesh X-Patchwork-Id: 10380059 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 61FD360353 for ; Fri, 4 May 2018 06:59:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FACC29356 for ; Fri, 4 May 2018 06:59:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 446272935A; Fri, 4 May 2018 06:59:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CF6E629356 for ; Fri, 4 May 2018 06:59:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+mYCAB7T6qwFBPTliR4D5FIV5t3lHApcVUj6A6Zf2L0=; b=qk7lhmszUPr3mByHodf3EVAhac dRaU3azmaaZAK2Ehn1mhQucp0yKPfI9xrDY/omPYO9P/fL3oQwo2W1pb9jpkw9uKvn1DrRR4q6Tl2 JHx2voRfoXOY7BnS1Nhp5QyZ6RICk2LXPppSbMgZTcFBVxJQbtKCbl+gc8kSJd8TTWFBPwbZ6lieC xSrQnh5i23e//8H6nYk/wczMCjs6GS3CWHBkjlj9cbGan/1P0lSViML3Vz79Y9ihJgvKdnLjmsIhv hZScK1rzC3mBtc8m/pO7pY0yQ/MjfE487qnbJaJkEElX7fHLsBEbYBCwSgQLuyDJyiOQwiEPJLyxY ESaOaklw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fEUgl-0003xv-Eh; Fri, 04 May 2018 06:59:31 +0000 Received: from mail-pf0-x243.google.com ([2607:f8b0:400e:c00::243]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fEUg1-0003hA-QP for linux-arm-kernel@lists.infradead.org; Fri, 04 May 2018 06:58:59 +0000 Received: by mail-pf0-x243.google.com with SMTP id p14so5333048pfh.9 for ; Thu, 03 May 2018 23:58:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jEdbOTaI9m7PNHd1fJWH+JX73cpNYLhM0ko0o7VzxCA=; b=ZWnQ+dHGsJv5Lzil3i8Y324N7h03qLiLKSobkozWNJ4BTemwYeQcA9fkPMsij2F0/e KaR3CaBUZsPjYq5asdogXvKfSOSKYfSRTD/eJEvXG4ZrRC10Ohw6kIiRYDARAcDGn3O9 hYVdc5imnoiXb7rE0SctNyzjRAEIdi6U4NGCU+Mph5Jhcy0F0bFiUs6ZiECjx3luPYcp 5s9oII+oa1zBVbEikio7Uyj2MXkMs53RtKoDdpx9bQ8RONd4JHeNNw01AdO8nhwRSqLP iBmgwqIGHRTGb5xPLNCFWWgNPdGuWe1/Pg64qQFP4XCNoABFDE6TW5sQ8DZrzpCR0wIg rblA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jEdbOTaI9m7PNHd1fJWH+JX73cpNYLhM0ko0o7VzxCA=; b=jPTML2dbSOdImoerXphUYW5ViDx3KSW4vPrHJYpO6gHlWITMyb+EVfjo6LKQJYHwKA hv6Y+YGUsY++ARkgPHR1uNLDESFZms/dMHHYYIPL0xkBA5xZMDhR583dhxlfeDNlOM2n 8flqM6et0sH0VTeTrG/BMsWwEjrxoxnhrDR2zFLV2vTs2hDgZE6+zdXfqkYx5RD7uPmE FaijXjvRydv46x7X2xWuew6PnWNtH7J3DSMsgjvzKI5NGfkCFImwT3lcmck+T6dl29Tk P3zNWpBvN+8GJqU0pE6tmh939yygf6XwSrPgyxhJ3ElNCDN+5iFoV3+cleDz4yy7n/Y/ 5B6w== X-Gm-Message-State: ALQs6tAsQJMI02F4PWVv5nODl6cpWrbp966OBkEsYa8u2xbmpjLZDeb/ hCZ2P0mlCc1WN9Bwyql/bHY= X-Google-Smtp-Source: AB8JxZrok5zeWukneTSus8l4SJMtD6+A8hRvPfiopcBrg2gpPzQ+8VA+tgFWk/tzGYLViSi3Ke/7KA== X-Received: by 2002:a17:902:b184:: with SMTP id s4-v6mr19048295plr.359.1525417115403; Thu, 03 May 2018 23:58:35 -0700 (PDT) Received: from localhost (outboundhk.mxmail.xiaomi.com. [207.226.244.124]) by smtp.gmail.com with ESMTPSA id p24sm20520473pfk.58.2018.05.03.23.58.34 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Thu, 03 May 2018 23:58:34 -0700 (PDT) From: Ganesh Mahendran To: ldufour@linux.vnet.ibm.com, catalin.marinas@arm.com, will.deacon@arm.com, mark.rutland@arm.com, cpandya@codeaurora.org, punit.agrawal@arm.com Subject: [PATCH v2 2/2] arm64/mm: add speculative page fault Date: Fri, 4 May 2018 14:57:49 +0800 Message-Id: <1525417069-27401-2-git-send-email-opensource.ganesh@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1525417069-27401-1-git-send-email-opensource.ganesh@gmail.com> References: <1525417069-27401-1-git-send-email-opensource.ganesh@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180503_235845_875221_F369345E X-CRM114-Status: GOOD ( 18.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Ganesh Mahendran MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP This patch enables the speculative page fault on the arm64 architecture. I completed spf porting in 4.9. From the test result, we can see app launching time improved by about 10% in average. For the apps which have more than 50 threads, 15% or even more improvement can be got. Signed-off-by: Ganesh Mahendran --- v2: move find_vma() to do_page_fault() remove IS_ENABLED() remove fault != VM_FAULT_SIGSEGV check initilize vma = NULL --- arch/arm64/mm/fault.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 4165485..efd5956 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -320,14 +320,12 @@ static void do_bad_area(unsigned long addr, unsigned int esr, struct pt_regs *re #define VM_FAULT_BADMAP 0x010000 #define VM_FAULT_BADACCESS 0x020000 -static int __do_page_fault(struct mm_struct *mm, unsigned long addr, +static int __do_page_fault(struct vm_area_struct *vma, unsigned long addr, unsigned int mm_flags, unsigned long vm_flags, struct task_struct *tsk) { - struct vm_area_struct *vma; int fault; - vma = find_vma(mm, addr); fault = VM_FAULT_BADMAP; if (unlikely(!vma)) goto out; @@ -371,6 +369,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, int fault, major = 0; unsigned long vm_flags = VM_READ | VM_WRITE; unsigned int mm_flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; + struct vm_area_struct *vma = NULL; if (notify_page_fault(regs, esr)) return 0; @@ -410,6 +409,16 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); /* + * let's try a speculative page fault without grabbing the + * mmap_sem. + */ + fault = handle_speculative_fault(mm, addr, mm_flags, &vma); + if (fault != VM_FAULT_RETRY) { + perf_sw_event(PERF_COUNT_SW_SPF, 1, regs, addr); + goto done; + } + + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, * we can bug out early if this is from code which shouldn't. @@ -431,7 +440,10 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, #endif } - fault = __do_page_fault(mm, addr, mm_flags, vm_flags, tsk); + if (!vma || !can_reuse_spf_vma(vma, addr)) + vma = find_vma(mm, addr); + + fault = __do_page_fault(vma, addr, mm_flags, vm_flags, tsk); major |= fault & VM_FAULT_MAJOR; if (fault & VM_FAULT_RETRY) { @@ -454,11 +466,20 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, if (mm_flags & FAULT_FLAG_ALLOW_RETRY) { mm_flags &= ~FAULT_FLAG_ALLOW_RETRY; mm_flags |= FAULT_FLAG_TRIED; + + /* + * Do not try to reuse this vma and fetch it + * again since we will release the mmap_sem. + */ + vma = NULL; + goto retry; } } up_read(&mm->mmap_sem); +done: + /* * Handle the "normal" (no error) case first. */