From patchwork Wed May 2 07:54:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mahendran Ganesh X-Patchwork-Id: 10375189 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4CB866037D for ; Wed, 2 May 2018 07:56:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3ACE428DAD for ; Wed, 2 May 2018 07:56:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2D0B628DB2; Wed, 2 May 2018 07:56:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM, MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8D6AD28DAD for ; Wed, 2 May 2018 07:56:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4GFlAiqGZKrBtMHQ88J/uuAi/QiC32SfvMKJIEOevy4=; b=n3XelXvRabd7XncMxdE8jV6DNx RM+31yQjtI8nweEXO1c95m7dGV7oD88hSffqFJgfgM8eCNICUFYSFEj0FcB+lnDc/CV0NDm090V0d so+zroeDEFv5p853sGDWyCHvESdiGKa8RjBydsZ1gTG0El9gJ5w9YOsurpPTub04ANYY1fHYvHpn8 +jFNBoTsY0piSKMO9/ifsPDg77X3OCtOtExIskHLRhUpYI6mNGzSzAQa6wCAnzjMFCrcl6btNseQ7 jSw3coUdbbuXba/1u/wBc7U6n5ebmj5Kia8hIjMygcT/MPihmTuuRUaxuXv6gghVHuKfiGCR4PQrD OFcVSkKA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fDmcT-0007kX-Et; Wed, 02 May 2018 07:56:09 +0000 Received: from mail-pg0-x244.google.com ([2607:f8b0:400e:c05::244]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fDmbV-00067B-WE for linux-arm-kernel@lists.infradead.org; Wed, 02 May 2018 07:55:20 +0000 Received: by mail-pg0-x244.google.com with SMTP id g20-v6so3349473pgv.1 for ; Wed, 02 May 2018 00:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CiAlOV2oRbz+VZO/tptorF+KuBCIfAyji69OyNiRK7M=; b=CyLqtnKYJmmW7Nm67LwUm+I6vk5grrNsCOzGmYSmF4EOhsR5M0Dz/nNRr3SjGNwbOz Zrz9ETn++eeyT34cNF3EWEKoF1iRe8p5A9VN5PrEYEkfcn5xOOVrVxFRzcaiIOekLG8V H/K1yZT/Mkc/jX/5t30F+btddzRffyi9MYm6JMrcP0ztiy0SxsDNfWrzmAjjMnBq9Kuj kYfpPnDdb3ik69kZ5S4tH9REq5xbMdBs1WhzlS2zP00LmnKegPk4wlIljjRYYIW8wbD5 9iys37OfYumj/hCe+731YYoRp/UO2nXZ+K/3Lk3DvmasYy/PgjlRbaFjBAnUBXUsTXhI Rr4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CiAlOV2oRbz+VZO/tptorF+KuBCIfAyji69OyNiRK7M=; b=SqmxXoExnkWowxW/fWgeVwkB/iv5aHtoAO/0N94ryDUSEz71l8548053SzMDFisvrg NiwqmfjDeKCvH+ZGAY92ng0l7lcJ7JJ1E6rElkfoBO9GCkQCyX3Vejm4Qsr//wpuTmNK LqvIWz3aQE4hL6UZDXtjrmRblzRSc4Ki2F8lNZscCWOTIyKOk1D7kMOXsJaUKzdme6ew YrspIS+L9E4gug2mDhEUdRMEdOTe9U2vwBgHz33frhVigxkHLeAibXX4AxMdzRRW8+w2 PACAWZJVH5c5L/9V5Wv8GQptMBm2VrrEHX8qhrd1+8G5GQLzpxcVwSSn2Pnl2/vEjAMF iSyw== X-Gm-Message-State: ALQs6tA0xrzWZlq4RfAcOLyHOXI+SlRsGXpqpX7LwUvtbdTAByVmJ9S3 Mg6R+DCz/OiNSOq9rGHcQC8= X-Google-Smtp-Source: AB8JxZoLNN6Q8OTMHvuTMboxmCr2IR3CtGGbBVIs3PH4uA2Uzi9aPBbd815qr35a9E00r3Umz/SKaw== X-Received: by 10.98.31.3 with SMTP id f3mr15178813pff.213.1525247699432; Wed, 02 May 2018 00:54:59 -0700 (PDT) Received: from localhost (outboundhk.mxmail.xiaomi.com. [207.226.244.123]) by smtp.gmail.com with ESMTPSA id k7-v6sm10922676pgc.42.2018.05.02.00.54.58 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 02 May 2018 00:54:59 -0700 (PDT) From: Ganesh Mahendran To: ldufour@linux.vnet.ibm.com, catalin.marinas@arm.com, will.deacon@arm.com Subject: [PATCH 2/2] arm64/mm: add speculative page fault Date: Wed, 2 May 2018 15:54:32 +0800 Message-Id: <1525247672-2165-2-git-send-email-opensource.ganesh@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1525247672-2165-1-git-send-email-opensource.ganesh@gmail.com> References: <1525247672-2165-1-git-send-email-opensource.ganesh@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180502_005510_070527_2ECDF342 X-CRM114-Status: GOOD ( 20.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Ganesh Mahendran MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP This patch enables the speculative page fault on the arm64 architecture. I completed spf porting in 4.9. From the test result, we can see app launching time improved by about 10% in average. For the apps which have more than 50 threads, 15% or even more improvement can be got. Signed-off-by: Ganesh Mahendran --- This patch is on top of Laurent's v10 spf --- arch/arm64/mm/fault.c | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 4165485..e7992a3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -322,11 +322,13 @@ static void do_bad_area(unsigned long addr, unsigned int esr, struct pt_regs *re static int __do_page_fault(struct mm_struct *mm, unsigned long addr, unsigned int mm_flags, unsigned long vm_flags, - struct task_struct *tsk) + struct task_struct *tsk, struct vm_area_struct *vma) { - struct vm_area_struct *vma; int fault; + if (!vma || !can_reuse_spf_vma(vma, addr)) + vma = find_vma(mm, addr); + vma = find_vma(mm, addr); fault = VM_FAULT_BADMAP; if (unlikely(!vma)) @@ -371,6 +373,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, int fault, major = 0; unsigned long vm_flags = VM_READ | VM_WRITE; unsigned int mm_flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; + struct vm_area_struct *vma; if (notify_page_fault(regs, esr)) return 0; @@ -409,6 +412,25 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) { + fault = handle_speculative_fault(mm, addr, mm_flags, &vma); + /* + * Page fault is done if VM_FAULT_RETRY is not returned. + * But if the memory protection keys are active, we don't know + * if the fault is due to key mistmatch or due to a + * classic protection check. + * To differentiate that, we will need the VMA we no + * more have, so let's retry with the mmap_sem held. + */ + if (fault != VM_FAULT_RETRY && + fault != VM_FAULT_SIGSEGV) { + perf_sw_event(PERF_COUNT_SW_SPF, 1, regs, addr); + goto done; + } + } else { + vma = NULL; + } + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -431,7 +453,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, #endif } - fault = __do_page_fault(mm, addr, mm_flags, vm_flags, tsk); + fault = __do_page_fault(mm, addr, mm_flags, vm_flags, tsk, vma); major |= fault & VM_FAULT_MAJOR; if (fault & VM_FAULT_RETRY) { @@ -454,11 +476,21 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, if (mm_flags & FAULT_FLAG_ALLOW_RETRY) { mm_flags &= ~FAULT_FLAG_ALLOW_RETRY; mm_flags |= FAULT_FLAG_TRIED; + + /* + * Do not try to reuse this vma and fetch it + * again since we will release the mmap_sem. + */ + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) + vma = NULL; + goto retry; } } up_read(&mm->mmap_sem); +done: + /* * Handle the "normal" (no error) case first. */