From patchwork Wed May 2 07:54:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mahendran Ganesh X-Patchwork-Id: 10375183 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BF1056037D for ; Wed, 2 May 2018 07:55:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0E4428D97 for ; Wed, 2 May 2018 07:55:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A1E5D28DB0; Wed, 2 May 2018 07:55:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22E9128D97 for ; Wed, 2 May 2018 07:55:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32F766B0009; Wed, 2 May 2018 03:55:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2DE166B000A; Wed, 2 May 2018 03:55:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17FB76B000C; Wed, 2 May 2018 03:55:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg0-f70.google.com (mail-pg0-f70.google.com [74.125.83.70]) by kanga.kvack.org (Postfix) with ESMTP id C767F6B0009 for ; Wed, 2 May 2018 03:55:00 -0400 (EDT) Received: by mail-pg0-f70.google.com with SMTP id q8-v6so5175168pgv.22 for ; Wed, 02 May 2018 00:55:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=CiAlOV2oRbz+VZO/tptorF+KuBCIfAyji69OyNiRK7M=; b=PvFxC599+asv/GlEpyxUBLsqfTZdurdu7Arlcrkwf0p+TTkKQyW4gg5SQmCR6RNmSC fHoz4GDw+5dTxlfHVv6lKWEkB6la+yVp6OZmVjT/AdaeoMoivX3IYMOvpe5lFrSy2OAJ ssKW+iLICUv8s9x0Z+NVPSRh1wtqVZnka5E3kS8ZdSa9oakvmtyY/v0QqJapkPTeJTM0 bje0ahQtET8jV1RrByWCAwr3I/oiTSg6ZTwtp++2zx5RKYtIx6bPyCchuUSlirZirOMT p0Qq9YZcnwjtEA+cs5Pmpsw4bB1srQs7F2mjzFmyUVaaS4Dkz5qC3wkx64ewjwg6S36O 5xGQ== X-Gm-Message-State: ALQs6tBCmf2Jtmr2sSKfhdthgDgobNPGFWg/ZazvO63/agA4wBxFI19+ uEU32ZN1/6uwaVz9hvBZv2w/UB/BBKZtymXgA8XSM+hacH08QKsfbNysIfd670XHYD3XHylEyFh wcNpbaLaaFToFXvZzrR1SsCJLbXuxJwBFOxRMt802utYZg7Bx0kheQU9nPYu7kh9nwnPQIrqXwe S12Y7Y90Ct7C+vlLHRhnvS9TX5wiOXoLfDUjen8nZ3g1vyj3sE3F+kaAtTZkVbXr7XDG4mBpJxX zzyVOEPd6XlQGYfw/Ggbeedj0jXXBdAC9bFvU/rk5CY2ptCElrj9gN6IpQ4WuLpMbIn1m2FbTki kld/W9YtFbTNfM1yYbzDln26kuFrEbPeQfOL+D/zh99Mism/xIguy5TtqbVMhjXWNZflMhgYtuO k X-Received: by 2002:a17:902:8f93:: with SMTP id z19-v6mr4211797plo.166.1525247700504; Wed, 02 May 2018 00:55:00 -0700 (PDT) X-Received: by 2002:a17:902:8f93:: with SMTP id z19-v6mr4211767plo.166.1525247699710; Wed, 02 May 2018 00:54:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525247699; cv=none; d=google.com; s=arc-20160816; b=vKEMYRW4iAyg6ES1pBbBS7JQ9s40uCgWZRmL5KARoUgfXLdoWIwN7c5iOfyJApFtvp 7Xebrz+C60OhkR5ciBVaforgulT0Ams+OZfMMbMQoAcJ1RvMN5bWus3T7b/PGNcOgV8d TtzYhbnKYvmuJ2Bt0tOjDAdtLc5qUrlhAtonVsmWk/11GoLSg+rxy6yw2m7UkOCaQngF ZBilCYs1JU1Em6GgayOnK6NYnv6/nCupLzHIfrvKWD+J9HVB2hp5RYdwIBV6WBlsMZGr ZR7APOYUFDR5L08ErrTabZJAKLSwUa+Oka5hdSiKPcKs1ZH8eu+rCeX5fuuyLpvbPq2W 6+RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=CiAlOV2oRbz+VZO/tptorF+KuBCIfAyji69OyNiRK7M=; b=SY8bH+oXypNFCNYlFA4NWLNH62PaLd8FGX82geuF7lDYGMfWR6J4daM/hq7foIruk5 OG/o9bqb3c8E5JR6Ry+7uyedc5ztFoAavdZPilQvL3OSOceDYRJJNpwTKJYjkwNLCMo4 3hZTS7lM4M6S3JnBW5whKDjkGpbWAE8awVEbhLuJNETetO3RsZTQFosnzgPnm7zoVQD8 VblqHOx82hJE/MXLaqMYfgqKuB+U9F1SqbUGVsCH48ZhLmIZOAKjDLLwwvRlFvo+ik7u VP56EE9E79mTgubrVXiP9sWrv0nx90y8//ZwlbZL0tKkP3ztgb5iQu92hjxMsS6EqPwg WXRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=CyLqtnKY; spf=pass (google.com: domain of opensource.ganesh@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=opensource.ganesh@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id b12-v6sor2484939pgq.325.2018.05.02.00.54.59 for (Google Transport Security); Wed, 02 May 2018 00:54:59 -0700 (PDT) Received-SPF: pass (google.com: domain of opensource.ganesh@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=CyLqtnKY; spf=pass (google.com: domain of opensource.ganesh@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=opensource.ganesh@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CiAlOV2oRbz+VZO/tptorF+KuBCIfAyji69OyNiRK7M=; b=CyLqtnKYJmmW7Nm67LwUm+I6vk5grrNsCOzGmYSmF4EOhsR5M0Dz/nNRr3SjGNwbOz Zrz9ETn++eeyT34cNF3EWEKoF1iRe8p5A9VN5PrEYEkfcn5xOOVrVxFRzcaiIOekLG8V H/K1yZT/Mkc/jX/5t30F+btddzRffyi9MYm6JMrcP0ztiy0SxsDNfWrzmAjjMnBq9Kuj kYfpPnDdb3ik69kZ5S4tH9REq5xbMdBs1WhzlS2zP00LmnKegPk4wlIljjRYYIW8wbD5 9iys37OfYumj/hCe+731YYoRp/UO2nXZ+K/3Lk3DvmasYy/PgjlRbaFjBAnUBXUsTXhI Rr4g== X-Google-Smtp-Source: AB8JxZoLNN6Q8OTMHvuTMboxmCr2IR3CtGGbBVIs3PH4uA2Uzi9aPBbd815qr35a9E00r3Umz/SKaw== X-Received: by 10.98.31.3 with SMTP id f3mr15178813pff.213.1525247699432; Wed, 02 May 2018 00:54:59 -0700 (PDT) Received: from localhost (outboundhk.mxmail.xiaomi.com. [207.226.244.123]) by smtp.gmail.com with ESMTPSA id k7-v6sm10922676pgc.42.2018.05.02.00.54.58 (version=TLS1_2 cipher=AES128-SHA bits=128/128); Wed, 02 May 2018 00:54:59 -0700 (PDT) From: Ganesh Mahendran To: ldufour@linux.vnet.ibm.com, catalin.marinas@arm.com, will.deacon@arm.com Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ganesh Mahendran Subject: [PATCH 2/2] arm64/mm: add speculative page fault Date: Wed, 2 May 2018 15:54:32 +0800 Message-Id: <1525247672-2165-2-git-send-email-opensource.ganesh@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1525247672-2165-1-git-send-email-opensource.ganesh@gmail.com> References: <1525247672-2165-1-git-send-email-opensource.ganesh@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch enables the speculative page fault on the arm64 architecture. I completed spf porting in 4.9. From the test result, we can see app launching time improved by about 10% in average. For the apps which have more than 50 threads, 15% or even more improvement can be got. Signed-off-by: Ganesh Mahendran --- This patch is on top of Laurent's v10 spf --- arch/arm64/mm/fault.c | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 4165485..e7992a3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -322,11 +322,13 @@ static void do_bad_area(unsigned long addr, unsigned int esr, struct pt_regs *re static int __do_page_fault(struct mm_struct *mm, unsigned long addr, unsigned int mm_flags, unsigned long vm_flags, - struct task_struct *tsk) + struct task_struct *tsk, struct vm_area_struct *vma) { - struct vm_area_struct *vma; int fault; + if (!vma || !can_reuse_spf_vma(vma, addr)) + vma = find_vma(mm, addr); + vma = find_vma(mm, addr); fault = VM_FAULT_BADMAP; if (unlikely(!vma)) @@ -371,6 +373,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, int fault, major = 0; unsigned long vm_flags = VM_READ | VM_WRITE; unsigned int mm_flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; + struct vm_area_struct *vma; if (notify_page_fault(regs, esr)) return 0; @@ -409,6 +412,25 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) { + fault = handle_speculative_fault(mm, addr, mm_flags, &vma); + /* + * Page fault is done if VM_FAULT_RETRY is not returned. + * But if the memory protection keys are active, we don't know + * if the fault is due to key mistmatch or due to a + * classic protection check. + * To differentiate that, we will need the VMA we no + * more have, so let's retry with the mmap_sem held. + */ + if (fault != VM_FAULT_RETRY && + fault != VM_FAULT_SIGSEGV) { + perf_sw_event(PERF_COUNT_SW_SPF, 1, regs, addr); + goto done; + } + } else { + vma = NULL; + } + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -431,7 +453,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, #endif } - fault = __do_page_fault(mm, addr, mm_flags, vm_flags, tsk); + fault = __do_page_fault(mm, addr, mm_flags, vm_flags, tsk, vma); major |= fault & VM_FAULT_MAJOR; if (fault & VM_FAULT_RETRY) { @@ -454,11 +476,21 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, if (mm_flags & FAULT_FLAG_ALLOW_RETRY) { mm_flags &= ~FAULT_FLAG_ALLOW_RETRY; mm_flags |= FAULT_FLAG_TRIED; + + /* + * Do not try to reuse this vma and fetch it + * again since we will release the mmap_sem. + */ + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) + vma = NULL; + goto retry; } } up_read(&mm->mmap_sem); +done: + /* * Handle the "normal" (no error) case first. */