From patchwork Wed Apr 3 08:37:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5ED4CD1294 for ; Wed, 3 Apr 2024 08:38:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IMax4FysePacoVWWtXuTNJMVG/7Y1z+RDnDsl0Nj1r4=; b=ZCM8aQKX+DOQQb XeKJ7n+bzFUDbXovqgWq0vdBmM1GpHuOJuVkRbqySFVJnjyGE+bz56O14fbVPmWxWgeWVY4r4u9Wh 3LJ/0PN57iiLmDSXAEAZ5oOxMNE8xFF3BO9xntzXYEIgo6b1214/aByrR3sRXxoDIBqNdzXqWRIjp iVFmrmreyNJ+4ItQAnsxJExtQOMuY5Z4+9O/ffmq7PeoF+uF9d65msw+PrEXGPNs/RiYRL9ONcgbM fYSJ3nK7NZRdka/5TONxKLeR3bJFGcKTFfJSJCwUHfFfQy1FHi7ZVQIoLfiSDSe2yp+d7EYgAdecy WR5lJcmjd2oDJmsX4/hA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8k-0000000Ew6z-0AJu; Wed, 03 Apr 2024 08:38:38 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8c-0000000Evyl-2mcE; Wed, 03 Apr 2024 08:38:34 +0000 Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4V8dP626VPz29lXb; Wed, 3 Apr 2024 16:35:38 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id EC3921A0172; Wed, 3 Apr 2024 16:38:23 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:22 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 1/7] arm64: mm: cleanup __do_page_fault() Date: Wed, 3 Apr 2024 16:37:59 +0800 Message-ID: <20240403083805.1818160-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_013831_091915_F2A34BB3 X-CRM114-Status: GOOD ( 12.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The __do_page_fault() only calls handle_mm_fault() after vm_flags checked, and it is only called by do_page_fault(), let's squash it into do_page_fault() to cleanup code. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang Reviewed-by: Catalin Marinas --- arch/arm64/mm/fault.c | 27 +++++++-------------------- 1 file changed, 7 insertions(+), 20 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 8251e2fea9c7..9bb9f395351a 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -486,25 +486,6 @@ static void do_bad_area(unsigned long far, unsigned long esr, } } -#define VM_FAULT_BADMAP ((__force vm_fault_t)0x010000) -#define VM_FAULT_BADACCESS ((__force vm_fault_t)0x020000) - -static vm_fault_t __do_page_fault(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long addr, - unsigned int mm_flags, unsigned long vm_flags, - struct pt_regs *regs) -{ - /* - * Ok, we have a good vm_area for this memory access, so we can handle - * it. - * Check that the permissions on the VMA allow for the fault which - * occurred. - */ - if (!(vma->vm_flags & vm_flags)) - return VM_FAULT_BADACCESS; - return handle_mm_fault(vma, addr, mm_flags, regs); -} - static bool is_el0_instruction_abort(unsigned long esr) { return ESR_ELx_EC(esr) == ESR_ELx_EC_IABT_LOW; @@ -519,6 +500,9 @@ static bool is_write_abort(unsigned long esr) return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM); } +#define VM_FAULT_BADMAP ((__force vm_fault_t)0x010000) +#define VM_FAULT_BADACCESS ((__force vm_fault_t)0x020000) + static int __kprobes do_page_fault(unsigned long far, unsigned long esr, struct pt_regs *regs) { @@ -617,7 +601,10 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, goto done; } - fault = __do_page_fault(mm, vma, addr, mm_flags, vm_flags, regs); + if (!(vma->vm_flags & vm_flags)) + fault = VM_FAULT_BADACCESS; + else + fault = handle_mm_fault(vma, addr, mm_flags, regs); /* Quick path to respond to signals */ if (fault_signal_pending(fault, regs)) { From patchwork Wed Apr 3 08:38:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C63EC6FD1F for ; Wed, 3 Apr 2024 08:39:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Pt6Tb7wmQOd0BgQpEaaInZdJtY9p8I9zbYhGH4MVfaw=; b=pYMYqjIjEsBbHQ 4cfcO8uiSryeuHDxca7mJfkQByoJRTE2ri+XuWoigVJ42JopanbQufJv9U2LK7l7hACvgEJHFII/p MOkg3qgU89+xn4pPhzGxDhzlMlwlyOy00Ipii3lzbRGJkUlC6v6DAikoGcqTte3mYja6UPz4ys50B GUg8eyIOZxx9I9CK4lhtQGeAx+iJoIhER0MBEOe2T+cX8WLJGokVEqx2oY49cRdB0v0vZ+fPX3BXy K+L7o5RXZZUFH7Kg6w6D/OScKVpou5TMFzkfMJ8jNy63870P8qskgp8Xz24M0nST+3gK6n6d7yfBI klh+ystXoAn8E08pvVnQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw9B-0000000EwZS-1AHL; Wed, 03 Apr 2024 08:39:05 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8v-0000000EwJX-2xwj; Wed, 03 Apr 2024 08:38:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=dnCIzWeY1qqTgbyWio8O1U8KrwKcOC13DqOIgOib+lg=; b=Qk8Rlfearr7rsoxMj5+r50lIys A2hYn0SrFqZDoOU7/gvdKTcyegTI74sE2WL3sHMXovcnpyXDnFxIMYPY7bBb8QH45KWANwwUFtVV+ 6cM35HO4e2H5c+MqFE4E6cvNMgSnc/cblnXplK65UIkkMUWA95dhL3g8TlFf5FRIGn5xXQToeyty4 mbeQF814vsJi4Yd+Mq2BXXHzjrlWEmuVTM6RRNrz30ZMzjeMUctApeNr0AcSMQIUmD5oE+j3Iuolj 2xZjSX/h5vIAtdmNoS643NDjkmVd7AfytUAeOg7+pnqdmJZQI0qtUJZuwfYyH3+G0DxliIwHm2cFV VHA/qmJA==; Received: from szxga05-in.huawei.com ([45.249.212.191]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8p-000000046AM-3osN; Wed, 03 Apr 2024 08:38:47 +0000 Received: from mail.maildlp.com (unknown [172.19.88.163]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4V8dP72twYz1h5XQ; Wed, 3 Apr 2024 16:35:39 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id D79AA18001A; Wed, 3 Apr 2024 16:38:24 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:23 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 2/7] arm64: mm: accelerate pagefault when VM_FAULT_BADACCESS Date: Wed, 3 Apr 2024 16:38:00 +0800 Message-ID: <20240403083805.1818160-3-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_093844_563978_7B356A24 X-CRM114-Status: GOOD ( 11.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The vm_flags of vma already checked under per-VMA lock, if it is a bad access, directly set fault to VM_FAULT_BADACCESS and handle error, no need to retry with mmap_lock again, the latency time reduces 34% in 'lat_sig -P 1 prot lat_sig' from lmbench testcase. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang Reviewed-by: Catalin Marinas --- arch/arm64/mm/fault.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 9bb9f395351a..405f9aa831bd 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -572,7 +572,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); - goto lock_mmap; + fault = VM_FAULT_BADACCESS; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; } fault = handle_mm_fault(vma, addr, mm_flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) From patchwork Wed Apr 3 08:38:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 64E30CD1288 for ; Wed, 3 Apr 2024 08:38:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1Lpl3WrUZon6ZocTP4c75+hM+XG1OhwwStUjfmoriQc=; b=VqqcQMqebqXhb1 7Mmh+WPyuHHiCNB1CnkjTCjuOShdZ9F3HYMJ6oQo2za4gjooDjV4pYN4Gsl05vd9apVX1yV5uYJpf c0JjHAVsvwXCDQbPYFhqjwBehegSfqtnEWk0qFK4D9fPp0sV8KEgBCTcuKG2JKNuG2+lTIAI32JHJ Lkk8oGjwixNmfwucmmTzro2nsl1X64bw5dK0rSXGF0rDPnmwlFysHNSHVa/oTD1IURRm414P4M7N9 oSbvLGY8wNV0AunpYYuuR1Ws0pTIcM9D1erKt9viCHn0cD/rIVUcgHLLomhIwzgupu4mYemyUY4ov qOtZz2eL74pRyhiE+6Rw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8n-0000000EwAk-2LWX; Wed, 03 Apr 2024 08:38:41 +0000 Received: from szxga03-in.huawei.com ([45.249.212.189]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8i-0000000Evyp-09Fb; Wed, 03 Apr 2024 08:38:37 +0000 Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4V8dPs2hLCz1JB4D; Wed, 3 Apr 2024 16:36:17 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id C3C161402C7; Wed, 3 Apr 2024 16:38:25 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:24 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 3/7] arm: mm: accelerate pagefault when VM_FAULT_BADACCESS Date: Wed, 3 Apr 2024 16:38:01 +0800 Message-ID: <20240403083805.1818160-4-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_013836_373931_C77EDFE6 X-CRM114-Status: GOOD ( 11.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The vm_flags of vma already checked under per-VMA lock, if it is a bad access, directly set fault to VM_FAULT_BADACCESS and handle error, no need to retry with mmap_lock again. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang --- arch/arm/mm/fault.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index 439dc6a26bb9..5c4b417e24f9 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -294,7 +294,9 @@ do_page_fault(unsigned long addr, unsigned int fsr, struct pt_regs *regs) if (!(vma->vm_flags & vm_flags)) { vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + fault = VM_FAULT_BADACCESS; + goto bad_area; } fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) From patchwork Wed Apr 3 08:38:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40EB5CD1288 for ; Wed, 3 Apr 2024 08:39:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cyW5Tk6tRL1mXErcWBdadzZluV+78d2DnS2GXoNtYs4=; b=m2ygmYux5c6GhH qzNMWOEHBlupn/OkX9hcHMLDrRHPD0AWv2TGMxl0Cx242JoCoryIh+2uMxGM/LmqxgRVToZVD6dES IU5q0+6rT51ki6PnVn7HFRc058bjKhlG9V6esnpTPBbNb5JTOJnHgtMj59i2bdUyLzrTxfdRTJR3j GlvX+VBOXpE4tl1BbG4SRB2ac+pYiqwYRhPYO64NKSbnkMjvd04VgJRO663F9BDBJBHYQmufngTOh JhCCBs4zYAN2uXPBhtOqVs81/jmUF8WtIVYlZHq5b3KUcTxpLX4obfVKKHK2yIpO/O672Q0uHyrd1 Z6jyB+LcFIf7AY/KX2PA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw90-0000000EwP0-2xym; Wed, 03 Apr 2024 08:38:54 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8t-0000000EwGP-0iuD; Wed, 03 Apr 2024 08:38:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=vhZLrHVELrtbu5DW0tOonOUKFnnhSmrXd1Sgos/yGkA=; b=qXGvFb6KH8qbYPpN9yT+RPXzJV ITqt4CADYXU491pM0hb66PBasPNt7uc4AWhp5YWXiDrHqb7bKfvJZBsftdCerWMshpmF8uJEjgs9F fu8HkHlNOdPcy7lLw/LkKlRyYdAf+Mg+AZn6YJGQfDB3kUZ0hAhzr/sB8fGYsdixS1bCoiHP8x9+G IgyivqNPjwH0nNZlJH2UcYJrpQfQn6oPn4H3BEEO3myRefcZNx+IUkwERY9woQqfQhU93uGOcmHxu 3w4/OToTpvT8b9ogcaUGLZtQg/DbaLPs+2ylGrSZ2Mx/kcXiOII3Sy1Vv315kaCnkoQJgelL6Rxpw wV6xQD1g==; Received: from szxga04-in.huawei.com ([45.249.212.190]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8n-000000046AQ-3m7M; Wed, 03 Apr 2024 08:38:45 +0000 Received: from mail.maildlp.com (unknown [172.19.163.44]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4V8dP90NXPz29lXd; Wed, 3 Apr 2024 16:35:41 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id B1225140410; Wed, 3 Apr 2024 16:38:26 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:25 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 4/7] powerpc: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:02 +0800 Message-ID: <20240403083805.1818160-5-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_093842_773084_D3A7CFC6 X-CRM114-Status: GOOD ( 14.29 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The access_[pkey]_error() of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. In order to release the correct lock, pass the mm_struct into bad_access_pkey()/bad_access(), if mm is NULL, release vma lock, or release mmap_lock. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Signed-off-by: Kefeng Wang Acked-by: Michael Ellerman (powerpc) --- arch/powerpc/mm/fault.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 53335ae21a40..215690452495 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -71,23 +71,26 @@ static noinline int bad_area_nosemaphore(struct pt_regs *regs, unsigned long add return __bad_area_nosemaphore(regs, address, SEGV_MAPERR); } -static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code) +static int __bad_area(struct pt_regs *regs, unsigned long address, int si_code, + struct mm_struct *mm, struct vm_area_struct *vma) { - struct mm_struct *mm = current->mm; /* * Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ - mmap_read_unlock(mm); + if (mm) + mmap_read_unlock(mm); + else + vma_end_read(vma); return __bad_area_nosemaphore(regs, address, si_code); } static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, + struct mm_struct *mm, struct vm_area_struct *vma) { - struct mm_struct *mm = current->mm; int pkey; /* @@ -109,7 +112,10 @@ static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, */ pkey = vma_pkey(vma); - mmap_read_unlock(mm); + if (mm) + mmap_read_unlock(mm); + else + vma_end_read(vma); /* * If we are in kernel mode, bail out with a SEGV, this will @@ -124,9 +130,10 @@ static noinline int bad_access_pkey(struct pt_regs *regs, unsigned long address, return 0; } -static noinline int bad_access(struct pt_regs *regs, unsigned long address) +static noinline int bad_access(struct pt_regs *regs, unsigned long address, + struct mm_struct *mm, struct vm_area_struct *vma) { - return __bad_area(regs, address, SEGV_ACCERR); + return __bad_area(regs, address, SEGV_ACCERR, mm, vma); } static int do_sigbus(struct pt_regs *regs, unsigned long address, @@ -479,13 +486,13 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (unlikely(access_pkey_error(is_write, is_exec, (error_code & DSISR_KEYFAULT), vma))) { - vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return bad_access_pkey(regs, address, NULL, vma); } if (unlikely(access_error(is_write, is_exec, vma))) { - vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return bad_access(regs, address, NULL, vma); } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); @@ -521,10 +528,10 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (unlikely(access_pkey_error(is_write, is_exec, (error_code & DSISR_KEYFAULT), vma))) - return bad_access_pkey(regs, address, vma); + return bad_access_pkey(regs, address, mm, vma); if (unlikely(access_error(is_write, is_exec, vma))) - return bad_access(regs, address); + return bad_access(regs, address, mm, vma); /* * If for any reason at all we couldn't handle the fault, From patchwork Wed Apr 3 08:38:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9D089CD1288 for ; Wed, 3 Apr 2024 08:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ncaKSoU3cshah9IcGX+cOgd/WbnSqDceI79DLSvqnd0=; b=iEFE9XRbgoMJ0R YaRmlElettePUrMaH5ua7B6fdUS5CnmkdWVj/NXTsE/jptjlJbn13IhFvLOAN9+TzzjzISanWNlWY fMmH1DrWr7TjjzNyy5F4xd5icds8L3gHRbZwUhmMun27Uo7WHyGZMFoGi5yan9kTEJb04wqMQA4ap OvM8T35ilaQqzISD+ziAkZxe/CpdfmoKInv69+VNpCleYCi3WoW/Ot3yTp1ryx9xH4cA3zYmkB3Fs oW0R6siODOg1ZET/4tNcZbiCAmWqxA62KaXgsSa6nZcs5ddSwe8vGzrmE4MBKbABB9KE3YuoqEBqY fxCqIvbM2vS+jZQug1fw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw9P-0000000Ewka-1zEJ; Wed, 03 Apr 2024 08:39:19 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8v-0000000EwJN-2d2x; Wed, 03 Apr 2024 08:38:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=Myv/h7kbb21tl1+xfuWM9tOyrAQENAdc7X/Z/48zHSM=; b=Qn4g6icSWtHfyY/FIqjAEqNEi9 XomQC8NhN94yZMH5vZP2ZkmXBNONK0QdWfioMj1KPjyU1UPlLwix+7Vay0zhm0Da32hAsUUwjlDxQ TFEDo5faQt5oJK+t/cEt44H834iirRp0PJyJMEEs2n+Zn9Eehv6iQ9WeYem008XU8HPVFut9Pessx 6FqtJLYDzlLpCqMNgW7EdpQ4AQh4IEYjpoUDxqQTbaMQ1DNEDfrUPdrnro6U25/nSLoCD/bzoxtWI zQWdyWc3iqmA+A2bEuZvNCwh/Uw3LMf1+t+bHPtpShzVb/JtuYYu8XX872WoLfMJ/GtGdIvZadZLL Xm68rH6g==; Received: from szxga02-in.huawei.com ([45.249.212.188]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8p-000000046AH-3mwe; Wed, 03 Apr 2024 08:38:46 +0000 Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4V8dRJ4hz5zbf6r; Wed, 3 Apr 2024 16:37:32 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 97DF218007C; Wed, 3 Apr 2024 16:38:27 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:26 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 5/7] riscv: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:03 +0800 Message-ID: <20240403083805.1818160-6-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_093844_548840_46584719 X-CRM114-Status: GOOD ( 11.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The access_error() of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang Reviewed-by: Alexandre Ghiti Tested-by: Alexandre Ghiti --- arch/riscv/mm/fault.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 3ba1d4dde5dd..b3fcf7d67efb 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -292,7 +292,10 @@ void handle_page_fault(struct pt_regs *regs) if (unlikely(access_error(cause, vma))) { vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + tsk->thread.bad_cause = SEGV_ACCERR; + bad_area_nosemaphore(regs, code, addr); + return; } fault = handle_mm_fault(vma, addr, flags | FAULT_FLAG_VMA_LOCK, regs); From patchwork Wed Apr 3 08:38:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 978F0CD1288 for ; Wed, 3 Apr 2024 08:39:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Ks8XlGqfO4DHjzuGdk3QKhICsywRqSlFVs61F2diLRs=; b=l1CealWuwkpU9+ LZYUQvoqhoD3HY/0+dEXnI8evYTyOHVAiIE7v99osz0P5X86IQ0WAu1FjD2AqxBEEKVPOhUrR5KcA fyPQ6TEIByvrKGFYqOiYOEUaqUZl03vD5ZJee+0R2QcQoetOacPpLb726MZ560FgOiQQ0csPfqXiV XaLwLuuM+NFtn3bWGTXowHqFzTSgP/awx2Ys7SCBbJTcWwrkTVtNgODnqa6HlsWXB9yjO5gT0Lau/ mxoyJYcCTfTEGuz8dym9M1haPI2yeG80b/zf9CZ46DuK1OlXsh8Mfh4lYYY7v/ajREdrp8FIjjLeB hha25udvNsBkYlPWm6VQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw9G-0000000Ewdv-3Q5H; Wed, 03 Apr 2024 08:39:10 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8v-0000000EwJM-2fVA; Wed, 03 Apr 2024 08:38:49 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=7HHcWzzE9AwVcj9rXnePyGty9Nb+tULGuXwsa1lDWyA=; b=cEpNV4o622BRQddFRxgb7vx53w y8KKyKogGtG3aCjMHz4nkP4DniTR5uWCA5VNbt6/vvUkAcsydcMm9l+woqUrgA8UapVxbNJ47QD8/ gCbKY592QjAgDl2SaJSGKdv53y310N5H2SVWpH5BeWAFeiTjGDXeL8O1Btv6zZuD5HF009YATeod/ 3oCdGggkN/QFWHMKWMYUJy9ipDYix1ZsHHunDdDfGTDVr66FUnrh9waMgRWTo5gOU7/nZ40tsJs81 Qk93ROOOePKONGu2wClBbaJ+t51JhLyfuzgkPo8H8NdLviVkHrn/GrOhALfIbrTYAcIOh5rMQAU8F JwFM30HQ==; Received: from szxga04-in.huawei.com ([45.249.212.190]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8p-000000046AJ-3mue; Wed, 03 Apr 2024 08:38:46 +0000 Received: from mail.maildlp.com (unknown [172.19.88.163]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4V8dPB65yQz29lXY; Wed, 3 Apr 2024 16:35:42 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 7BC4E180063; Wed, 3 Apr 2024 16:38:28 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:27 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 6/7] s390: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:04 +0800 Message-ID: <20240403083805.1818160-7-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_093844_565945_88DA04C7 X-CRM114-Status: GOOD ( 10.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The vm_flags of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Signed-off-by: Kefeng Wang Reviewed-by: Heiko Carstens --- arch/s390/mm/fault.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index c421dd44ffbe..162ca2576fd4 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -325,7 +325,8 @@ static void do_exception(struct pt_regs *regs, int access) goto lock_mmap; if (!(vma->vm_flags & access)) { vma_end_read(vma); - goto lock_mmap; + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return handle_fault_error_nolock(regs, SEGV_ACCERR); } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) From patchwork Wed Apr 3 08:38:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 13615381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3FB4CD1288 for ; Wed, 3 Apr 2024 08:39:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:CC:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=J2PR35k7EKTz848Vd7Bh/FPnvRXOJgnH4Lnu4OiLaF8=; b=oqxy33OJ4jfpGe fsDdgIhnBPDLjEB9K6Tb1WU2iSDn7bCfD9te1mA0zz6k7QhCYrssnMjLPSEPeuSQKCvwXc5wbHpPO +fuKNkq4M/sad10tJ7Dxa+85qXss73GNlsunwzMWx2M1/FQeI7ETJlaT0fkt09DmZKHJEgZgzCed3 OctobcWdveT/0miqPSeX8qUYQIU843Gg3eXPG8j6/ICCd7ggH7EgG/U2FPiIdsQflqPAjWMKPruTZ 0kuWBOAxBdmKKJ4/QsQXbAKl9QEqFhFf9tywzviouMJZhUzAPlR8GyOAd9879T90NEBd2sZ0EATRp op++IQz0rbdTAy3H062g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8t-0000000EwHN-2joQ; Wed, 03 Apr 2024 08:38:47 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8r-0000000EwEu-3Mue; Wed, 03 Apr 2024 08:38:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=lmVlreGnPFbXRLejlXB9QDif0bxo2zNzlVV4g4MDH8w=; b=PG5v7uzqp2YVt6HPpHEAkydAE0 UnoiCGon9adhmFFe+URNHkUiQmD3gMeR+lHonhKf+nW+FsjalBDvhVJxqtkpG8ggjdtwU4VevtQ4J p3ap8RbQ/HYeME1QaI4xQ/Or1xuxAQGwgruWaH4lkAQN08ODHgbtZB0rVDSX8oKzBeXn3c3IKiwQ0 wAHKdmb1rNW+oHAsxqL3c65/rlWK/fsRIrJ/S9V722b3zICUuaFTVzjlpb7To1Et0k0x4uPNac0qd kbxpDGy0Y1adBv7tqe6ygZWjpG8nTH4OCdPHLX02U2/pjJOKD8zxyywswgwLQ9qZJhD1RjDKEYssK jWWA3mvg==; Received: from szxga06-in.huawei.com ([45.249.212.32]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrw8n-000000046AS-2jUy; Wed, 03 Apr 2024 08:38:44 +0000 Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga06-in.huawei.com (SkyGuard) with ESMTP id 4V8dRP0T1xz17MxK; Wed, 3 Apr 2024 16:37:37 +0800 (CST) Received: from dggpemm100001.china.huawei.com (unknown [7.185.36.93]) by mail.maildlp.com (Postfix) with ESMTPS id 67C731A016C; Wed, 3 Apr 2024 16:38:29 +0800 (CST) Received: from localhost.localdomain (10.175.112.125) by dggpemm100001.china.huawei.com (7.185.36.93) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Wed, 3 Apr 2024 16:38:28 +0800 From: Kefeng Wang To: CC: Russell King , Catalin Marinas , Will Deacon , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Dave Hansen , Andy Lutomirski , Peter Zijlstra , , , , , , , , Kefeng Wang Subject: [PATCH v2 7/7] x86: mm: accelerate pagefault when badaccess Date: Wed, 3 Apr 2024 16:38:05 +0800 Message-ID: <20240403083805.1818160-8-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> References: <20240403083805.1818160-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.112.125] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemm100001.china.huawei.com (7.185.36.93) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240403_093842_560240_4CF1812E X-CRM114-Status: GOOD ( 13.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The access_error() of vma already checked under per-VMA lock, if it is a bad access, directly handle error, no need to retry with mmap_lock again. In order to release the correct lock, pass the mm_struct into bad_area_access_error(), if mm is NULL, release vma lock, or release mmap_lock. Since the page faut is handled under per-VMA lock, count it as a vma lock event with VMA_LOCK_SUCCESS. Reviewed-by: Suren Baghdasaryan Signed-off-by: Kefeng Wang --- arch/x86/mm/fault.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index a4cc20d0036d..67b18adc75dd 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -866,14 +866,17 @@ bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code, static void __bad_area(struct pt_regs *regs, unsigned long error_code, - unsigned long address, u32 pkey, int si_code) + unsigned long address, struct mm_struct *mm, + struct vm_area_struct *vma, u32 pkey, int si_code) { - struct mm_struct *mm = current->mm; /* * Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ - mmap_read_unlock(mm); + if (mm) + mmap_read_unlock(mm); + else + vma_end_read(vma); __bad_area_nosemaphore(regs, error_code, address, pkey, si_code); } @@ -897,7 +900,8 @@ static inline bool bad_area_access_from_pkeys(unsigned long error_code, static noinline void bad_area_access_error(struct pt_regs *regs, unsigned long error_code, - unsigned long address, struct vm_area_struct *vma) + unsigned long address, struct mm_struct *mm, + struct vm_area_struct *vma) { /* * This OSPKE check is not strictly necessary at runtime. @@ -927,9 +931,9 @@ bad_area_access_error(struct pt_regs *regs, unsigned long error_code, */ u32 pkey = vma_pkey(vma); - __bad_area(regs, error_code, address, pkey, SEGV_PKUERR); + __bad_area(regs, error_code, address, mm, vma, pkey, SEGV_PKUERR); } else { - __bad_area(regs, error_code, address, 0, SEGV_ACCERR); + __bad_area(regs, error_code, address, mm, vma, 0, SEGV_ACCERR); } } @@ -1357,8 +1361,9 @@ void do_user_addr_fault(struct pt_regs *regs, goto lock_mmap; if (unlikely(access_error(error_code, vma))) { - vma_end_read(vma); - goto lock_mmap; + bad_area_access_error(regs, error_code, address, NULL, vma); + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + return; } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) @@ -1394,7 +1399,7 @@ void do_user_addr_fault(struct pt_regs *regs, * we can handle it.. */ if (unlikely(access_error(error_code, vma))) { - bad_area_access_error(regs, error_code, address, vma); + bad_area_access_error(regs, error_code, address, mm, vma); return; }