From patchwork Mon Jul 24 18:54:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13325267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 321CFEB64DD for ; Mon, 24 Jul 2023 18:54:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BEA176B0083; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9A7D6B0085; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A61DC6B0087; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 98F6B6B0083 for ; Mon, 24 Jul 2023 14:54:44 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 36AB0C0B80 for ; Mon, 24 Jul 2023 18:54:44 +0000 (UTC) X-FDA: 81047406888.10.0CC43BC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 8FA28A000A for ; Mon, 24 Jul 2023 18:54:42 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h2urfpoJ; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690224882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/dr8E7Ysfsh/pzxHk8PrgytRhAkkfjQGUpicX+vSfvw=; b=vYTYPD8HJAJoS2Vo3g9ExAmsFOjJafakSNR75rHL0Xgil8ztAdV69NN1Qu/C02gzg3x1NO fg7A5eDkNLvtKBNjlBEP8bXPn7Cvfp8loOPh9kTxZCW9y5c37NM8KMMotCLbaTIwInHHh4 y/pAnmrvsJiOFzgp63YiQ0CX5/arF3M= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h2urfpoJ; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690224882; a=rsa-sha256; cv=none; b=dXo1T22qi4b1kYBrwbC9xfDv6X+IlwTvdxKyoX88bAWgEkL6zzX2M4398tYMoKtoyYB0D/ j9VXLMiNhQ3xE8S1GUkkaV4q2XRb/rHKhuNv5inkx/LjzdNocLYuOGtQ8vVI73wiP1w4Ew LCwq2RLthXt5XrgZp55K+Nov7igm6Mw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/dr8E7Ysfsh/pzxHk8PrgytRhAkkfjQGUpicX+vSfvw=; b=h2urfpoJlH+/tTgmv+1TWIjF7V SHPiUiKpC2n086K/Blscs2mtWtATwrhaglAB7qSgdBQvKMNtIPQV/UFDUBNnXBCJKZqS02PMxZDhN QIpfedqmho8nh2D698VotNmU0q5qTKALJGUD/wc0E58tg4ZCE4Gn2rhzDLRFYpHGnf4jBnK69OxHr fBw6gic1/us6lWW6l7ExzoNHq56D4JqqkX3VwhzEJSKvB1yN9l0CoM6WoGz83conZI3w9cwm2x84i Zc3bYa8EqN9n54f/4wlFuqSHLh/NnBnmRxHDAymVbpKp5GzpcpfCkw2wIAESzGP7wGjHvSVs7xgCq /JBsnc2A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qO0hA-004iR7-AG; Mon, 24 Jul 2023 18:54:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH v3 04/10] mm: Handle PUD faults under the VMA lock Date: Mon, 24 Jul 2023 19:54:04 +0100 Message-Id: <20230724185410.1124082-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230724185410.1124082-1-willy@infradead.org> References: <20230724185410.1124082-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FA28A000A X-Rspam-User: X-Stat-Signature: c8ztom8ncybk8g1eynyxzt7pifm5rusf X-Rspamd-Server: rspam01 X-HE-Tag: 1690224882-258558 X-HE-Meta: U2FsdGVkX1+hLf+xsYrnCxkDlvLGZxC0OMTDAWKy3HhcrcGeaDld175VzFdS5iM5imAeUYiDmrzVKXjEyxmHCdctEeM3aZTOFq6JS2iKK4qjdOpI9nc2MFHpxanIstzZraQbKOGfS+W8juCdzmRQoHXdd679RRjyxafs/BsAAqKJK/iTGjQLAshxjhQz8XOOiAT1StrHjjR3tPIQFgBQylCtWVn8GyDt3mUKKiycHpTqg8fYJdBgj0OJlhGBYU/LxwAOfWWRY+jC3A3OOqe9ORSt3j6iQD0rtRHLpZhHp0Ac+CHS1cGWcptaaDfeVxu63lxfEgS2MjRWUQyJHja+Mb2PJCaYCBE/cjHxBnuh9Q3ersKYgG0/OpYPFFTC5N9WtOYSjNqO89iin+jy/ARE/uAR5X0FldMAn01/gKrK7nV9DGP1FxtKQUDBw3W75QOO7x2yR3LC62KkCYm5pjNXyQCBuRGiGOr3/TFqIVAxgYAc1HKoe47YaRo9j4bZCF3s/NOtP4n5Ahdzs0FxPNaW5OH2Nw17qG+lIPYpRJLyYqdLZC7mrvmT8XdGN0ceCBcDiI0O91lMU49/aTJILnJmpzAHS500yl/ug1Mr8wQu+e5ntg3nWGQHzwJAMfEm8lksJcjzYNy5XlCMANt5zaTJL+p4wlKzpTGa98m3r2H2CaL4E7aZvFK8HSx2HZYIYopQYbwSRwnwiIk2mBbHmM12ACQJPYk03pm9GD4rSx7v/25J+wKAnvWSz4Lfh9XcSpaA8CWw8abSrI210lD7MOn4SeeKTWYr5tUz7xNiQOLsMyvA6nH/IHcQor/9jUZ/EwaWovcA2+MPZmKsUznIBK6IMO+8AFLuqoest0Pbu7KPwjSwb+OpNTEwtyU0c2mIVwebjwILnAU6aF1nNc4bNyJQ8EuLq/gd/utOO7eOFxZh5IeIdFYVYJ6U73VQ+W6tXp3/FS0knn8pzwVpiDLDZka 34fjSnfj yXMYWbhL9PoCUqXi03mbl0F1kzMoAufcj+UYtmKxa5JbAK0cOOUKLzuC9EZSZg5Gh/gFte2akZ9ZFe4GshETNOfl5Zw3Xfwytxf87VQqTBZL3y4fx87EQpiAPUsUQL75m0rxu+TpFCbfJ+jfW7+yiyeks7ADT/gv3tuhFsIXDT6Md8O5jQPhoyQjP7A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Postpone checking the VMA_LOCK flag until we've attempted to handle faults on PUDs. There's a mild upside to this patch in that we'll allocate the page tables while under the VMA lock rather than the mmap lock, reducing the hold time on the mmap lock, since the retry will find the page tables already populated. The real purpose here is to make a commit that shows we don't call ->huge_fault under the VMA lock. We do now handle setting the accessed bit on a PUD fault under the VMA lock, but that doesn't seem likely to be a measurable difference. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 5ca8902b6f67..7fec616f490b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4987,11 +4987,17 @@ static vm_fault_t create_huge_pud(struct vm_fault *vmf) { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) + struct vm_area_struct *vma = vmf->vma; /* No support for anonymous transparent PUD pages yet */ - if (vma_is_anonymous(vmf->vma)) + if (vma_is_anonymous(vma)) return VM_FAULT_FALLBACK; - if (vmf->vma->vm_ops->huge_fault) - return vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + if (vma->vm_ops->huge_fault) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + return vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ return VM_FAULT_FALLBACK; } @@ -5000,21 +5006,26 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) { #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) + struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; /* No support for anonymous transparent PUD pages yet */ - if (vma_is_anonymous(vmf->vma)) + if (vma_is_anonymous(vma)) goto split; - if (vmf->vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { - if (vmf->vma->vm_ops->huge_fault) { - ret = vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); + if (vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) { + if (vma->vm_ops->huge_fault) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + ret = vma->vm_ops->huge_fault(vmf, PE_SIZE_PUD); if (!(ret & VM_FAULT_FALLBACK)) return ret; } } split: /* COW or write-notify not handled on PUD level: split pud.*/ - __split_huge_pud(vmf->vma, vmf->pud, vmf->address); + __split_huge_pud(vma, vmf->pud, vmf->address); #endif /* CONFIG_TRANSPARENT_HUGEPAGE && CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ return VM_FAULT_FALLBACK; } @@ -5134,11 +5145,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { - vma_end_read(vma); - return VM_FAULT_RETRY; - } - pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -5182,6 +5188,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (pud_trans_unstable(vmf.pud)) goto retry_pud; + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + if (pmd_none(*vmf.pmd) && hugepage_vma_check(vma, vm_flags, false, true, true)) { ret = create_huge_pmd(&vmf);