From patchwork Fri Jun 30 21:19:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13298785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B051CEB64DC for ; Fri, 30 Jun 2023 21:20:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B9948E004E; Fri, 30 Jun 2023 17:20:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4699D8E000F; Fri, 30 Jun 2023 17:20:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BDDD8E004E; Fri, 30 Jun 2023 17:20:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1B27B8E000F for ; Fri, 30 Jun 2023 17:20:11 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E94D380DC3 for ; Fri, 30 Jun 2023 21:20:10 +0000 (UTC) X-FDA: 80960682180.05.A24A0A9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 2063D1A0006 for ; Fri, 30 Jun 2023 21:20:08 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=3h3MWve0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3CEefZAYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CEefZAYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688160009; a=rsa-sha256; cv=none; b=yES2vV9fB/oA+ePWxkD71KcGubRvj/DkDnJPga8ycjSFadwtgutkh72DFYTG2oA2qmkQv3 26JQeXCFf2zkTg9/2y4f+Z/YrmNWsAQIdPjtR0X0Bt2mgiCzgaNJR0PSd79l8C1BN9yHFd TrTyt/nFGf92Tx8Cp9rZYU5Q2BbLf+k= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=3h3MWve0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3CEefZAYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3CEefZAYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688160009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V7p+xkeFpHpY+8imX6dpZwoTYVp943eibQ5qK+rLf3c=; b=GwSiS8cAbsPajlv6gvB+Ng/ixOUKOyxhJduk7itjtiCFZeQ4BCkLFQ4ItTPjIYobI06qvH H8AKnkuudLTH+HB/4BrumnlrlCieeY+zHlvdKx/+UaNnaZFdxBg8tYXoRPnxqB99/GIigG KMv2wWPqiZgXJJV40aW3nd7f9RD9CN8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-56ff7b4feefso23053667b3.0 for ; Fri, 30 Jun 2023 14:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688160008; x=1690752008; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V7p+xkeFpHpY+8imX6dpZwoTYVp943eibQ5qK+rLf3c=; b=3h3MWve0jx96zUgQCtdmEwl/487FE7hiM4XAWV/a03FX+hNKPzjfhxBp2xnBvYegT/ C6qnQUGRlOp0ILLqmsHhYWmc/p6bWF9Xsy4JDf6vX8OB4sEkRCVKltes2JnR6jJHHPWk WM+978jF+PQrbMOpb67gRRryqGNgIoDfI+qbh7M90nHodbn3+x4ZbPjl/WLgJ+x4EqcG oKqwNpwQNdEusOK0uqAcFe7yKhS7MKhu1T3R1hnxNBiexJX6j6d8zQpeZIzKYKffocD6 VaUxkh4bivx+CK9wy3r/jwRiyZnbwkGB7nTRtGadYqOebn4I5AJmIEWkucS1Ejv0gjej 48/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688160008; x=1690752008; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V7p+xkeFpHpY+8imX6dpZwoTYVp943eibQ5qK+rLf3c=; b=DoIqrqrZ2sIoJ/+SdzQ1K/Gg8CHPmEKfQjXeCsNdyHS65EtilhP/1Jx+xT31yyeR6/ dQq3hoZbJKXgnMGwsrTTDgAVltyQsbtKIFX9o+Cwgt2Q2nUoa6YcqoMAXYgup/ZPwIrH EZ5cCiD8xWQbtbuzej1WQORX8ZPj7zX3qvkC5m0cN6Be2DhuUWtC6WVW9H064JPuttrZ Ns5hAzAFxlbNqhSjg7oOpYD5oSVeJZVyPrmTzbP2QZazgfJ5G4g/u+/GKgIs/4+CjEUU 11aSNqnDxY/8bEc/XTMjflhsmfwKaW638uhW2BV+CDmF4jdZcAFH0ngYDWoF9SJBUDkB LKtA== X-Gm-Message-State: ABy/qLYDdKV8ftRYzXEmgc7oFL4Vnyqq+y46EL8R0T/VaOVOSZOjDIb3 O+wXhiC4ygiTF5W0I5n8Qyi1W0J3UEc= X-Google-Smtp-Source: APBJJlE6epfASRhUr+pvRHp+fS5E4dLCsuAbw47YMK/KkE1Font5yAC9IWg5l/JB5jBUsD/jiPVeVxN9Sfk= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:b54c:4d64:f00a:1b67]) (user=surenb job=sendgmr) by 2002:a0d:d586:0:b0:569:e04a:238f with SMTP id x128-20020a0dd586000000b00569e04a238fmr29616ywd.4.1688160008047; Fri, 30 Jun 2023 14:20:08 -0700 (PDT) Date: Fri, 30 Jun 2023 14:19:54 -0700 In-Reply-To: <20230630211957.1341547-1-surenb@google.com> Mime-Version: 1.0 References: <20230630211957.1341547-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230630211957.1341547-4-surenb@google.com> Subject: [PATCH v7 3/6] mm: drop per-VMA lock when returning VM_FAULT_RETRY or VM_FAULT_COMPLETED From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, peterx@redhat.com, ying.huang@intel.com, david@redhat.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, pasha.tatashin@soleen.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2063D1A0006 X-Stat-Signature: wydecc8zmuaxdmgc919i9bsuwwnw56sd X-HE-Tag: 1688160008-609654 X-HE-Meta: U2FsdGVkX1/1QUEon0KEAI/Hc5VrERspwunqMRlZgXC058BR/DTYosYR0V/HaWgW5dF5ouPBKaATl3qzAdBrVJro6hA3Zeiep7xX7s3CbTM0jrkxk8EAl1ioZn9bMI/9c34JL3Eet1crMWBaiFePYvAkKkGmXfXdPevLfmr7s1RbRvKKK6AZpdH8e74vNh087BpFho/d2gzOo/x1n0G78rE4j1wJCtRFHRQnbboOY4Fv0mrSedTNeqgUm/VBJkzrcQYcnGN0J/bye48PIh7Zzjp4FTlHgvk7EnWH8gVuxra77XQ0RVce9SdmDM/CfHPIvXJ8X9R0VWlghFLR1Op3LNHyN5SuZDReLGvu8BJ1sXnWbynW8r36pkmuLlD0g+XMFAGXnEr/HdL15lUpN8WBUz73EqOFRNa9TQEirF3IfdoadRkK+/x+yDpZccMn2kbv1f942d8VUSYITsYZZW4SNa0ikRqUPgzaPQT8D3gO4HxfKfx6I/5FNUBStnUvNnfNgvPe5o8cX7rwrpjAYNpnVg1wIuSndTRwQBqXq+02kU7Ug1kCm9ftOHYOfdEm3QjilyFykua5RzJOrOlK7EjfpFE7pQxWcsaUcFAVsSG41VSvDoL+178YO8H6aHgwDPeIePVOu3ozPXHDBV6XEmvlBxB4NcYAs1FBU9a6EnXOTHMgmioaauUFMySCLItoYvWouJ1NtTg+Rh+OH4JW0LNuOPPNyiz8r1nT3XJrcsPoGrJ6d7PdI2pCOl4rtZ/OVsDp8v7qe84RnSk5090mxLtRBHnbAt3aCuxh0y0mfUVLI4C6rGS8GxdIJmd5Kfnl2u1NWCylkHvLz7C+iQ6ZbbbIRzm37pwNUz1mGAg/kXLbowjD43MbrJG9VB1sK1U3GtsI+x+CJrLFAR/HW0WcYdp3bJndd2Ak1ecy/AGad69GnK9Om0DSfatn+GH3EOeFpdf/RCAoL3ocdHDn33ClTB0 25yybpx5 nkNZ/uczXOvbMmolCxddYU4HYqV3X2jgm6fiNrkAw8Kf59GPLSxQTE2OwlwHavgzKSp+Qknh/dCvooziOmyrJKY3p3WTioi0TRuPKu4u4k5v/IIc0ufM1IVOawkIdK2+ys7D2hJP11z0n1F/NYXxoWXi1Ps5XUrqfoommWCEUrBWBbW8pxzBIHJBAd2IedwAilwvkS7ZB3cszkAZSFDROb9nCJe5yVVyf2u9QrYmzOtb2UcO6N7kQnGGt5bny2FHArxZxlTBgkKW+WKH/OlFkmwlPxcXknKFuNgEyfQzSQ8GZYAwyokEPlcZoh6UnlAVIXvSlC5E7hobp1gtqNjp84Kmns7hjCZqxEfpxN4ut+fuVIXgZKMmakJOKQzwbMBf4qkNgwZW0PTD6qaTS4rxFcSbGUq1vhh/DISv/rslx68eKVTazqFpycE0pyVX6HzX7W0nT1h7Cdz0XgTfdfWI8yFUMekej5ubqjJaTnM2WSJchjqyOAg2YX1FsQLXSwCH14T/AYJdC71BxXsZVvITMd5n7OG/EoRFmLZB+DgZRdNTK4PdX+kmPabVkHPhMiP807EszNq03/7sF7p7KiPAsrK5hP+AClJBtkl18xO5f3rPvtAe7tWFt+8NmQjHMPCc24qZoBqr1HMtcJZ39hhdSNYj2TVN/CtbBI/GFuuMiC5d+ExtO0OafnDZ02g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: handle_mm_fault returning VM_FAULT_RETRY or VM_FAULT_COMPLETED means mmap_lock has been released. However with per-VMA locks behavior is different and the caller should still release it. To make the rules consistent for the caller, drop the per-VMA lock when returning VM_FAULT_RETRY or VM_FAULT_COMPLETED. Currently the only path returning VM_FAULT_RETRY under per-VMA locks is do_swap_page and no path returns VM_FAULT_COMPLETED for now. Signed-off-by: Suren Baghdasaryan Acked-by: Peter Xu --- arch/arm64/mm/fault.c | 3 ++- arch/powerpc/mm/fault.c | 3 ++- arch/s390/mm/fault.c | 3 ++- arch/x86/mm/fault.c | 3 ++- mm/memory.c | 12 ++++++++++++ 5 files changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 935f0a8911f9..9d78ff78b0e3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -602,7 +602,8 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, goto lock_mmap; } fault = handle_mm_fault(vma, addr, mm_flags | FAULT_FLAG_VMA_LOCK, regs); - vma_end_read(vma); + if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) + vma_end_read(vma); if (!(fault & VM_FAULT_RETRY)) { count_vm_vma_lock_event(VMA_LOCK_SUCCESS); diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 5bfdf6ecfa96..82954d0e6906 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -489,7 +489,8 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); - vma_end_read(vma); + if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) + vma_end_read(vma); if (!(fault & VM_FAULT_RETRY)) { count_vm_vma_lock_event(VMA_LOCK_SUCCESS); diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index dbe8394234e2..40a71063949b 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -418,7 +418,8 @@ static inline vm_fault_t do_exception(struct pt_regs *regs, int access) goto lock_mmap; } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); - vma_end_read(vma); + if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) + vma_end_read(vma); if (!(fault & VM_FAULT_RETRY)) { count_vm_vma_lock_event(VMA_LOCK_SUCCESS); goto out; diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index e8711b2cafaf..56b4f9faf8c4 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1341,7 +1341,8 @@ void do_user_addr_fault(struct pt_regs *regs, goto lock_mmap; } fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); - vma_end_read(vma); + if (!(fault & (VM_FAULT_RETRY | VM_FAULT_COMPLETED))) + vma_end_read(vma); if (!(fault & VM_FAULT_RETRY)) { count_vm_vma_lock_event(VMA_LOCK_SUCCESS); diff --git a/mm/memory.c b/mm/memory.c index 0ae594703021..5f26c56ce979 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3730,6 +3730,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (vmf->flags & FAULT_FLAG_VMA_LOCK) { ret = VM_FAULT_RETRY; + vma_end_read(vma); goto out; } @@ -5182,6 +5183,17 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, !is_cow_mapping(vma->vm_flags))) return VM_FAULT_SIGSEGV; } +#ifdef CONFIG_PER_VMA_LOCK + /* + * Per-VMA locks can't be used with FAULT_FLAG_RETRY_NOWAIT because of + * the assumption that lock is dropped on VM_FAULT_RETRY. + */ + if (WARN_ON_ONCE((*flags & + (FAULT_FLAG_VMA_LOCK | FAULT_FLAG_RETRY_NOWAIT)) == + (FAULT_FLAG_VMA_LOCK | FAULT_FLAG_RETRY_NOWAIT))) + return VM_FAULT_SIGSEGV; +#endif + return 0; }