From patchwork Sun Oct 30 21:29:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B09F7ECAAA1 for ; Sun, 30 Oct 2022 21:29:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C2EA8E0002; Sun, 30 Oct 2022 17:29:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D4BB8E0001; Sun, 30 Oct 2022 17:29:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 626EC8E0002; Sun, 30 Oct 2022 17:29:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 56C658E0001 for ; Sun, 30 Oct 2022 17:29:39 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 29F1040A3B for ; Sun, 30 Oct 2022 21:29:39 +0000 (UTC) X-FDA: 80078907678.15.157BEFD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id BD2A71C003F for ; Sun, 30 Oct 2022 21:29:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vgQM4vgy+7KnPWYFNhgrYbDAEMQY1XSSTWC+/NJ93CA=; b=UV8XJ7jsRur0UmosKLJPs7T9l0RCPY8ZAOqRC9R55yZDDpwaD0R1HxW3pSQamCEThgX1rX tMXWaGTr2Ny82NhZ87IlhgFZcsz1/G0hk4O4/QKP4/XYQwV5qAEdpGpiQJMOT65EY7Ah6m SwWF6GwuAMiehB7napb1RKxCczCgRrg= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-378-a7o_1hzGP1mVa6JVF1lfdQ-1; Sun, 30 Oct 2022 17:29:35 -0400 X-MC-Unique: a7o_1hzGP1mVa6JVF1lfdQ-1 Received: by mail-qt1-f198.google.com with SMTP id v12-20020ac8578c000000b003a50eb13e29so3372434qta.3 for ; Sun, 30 Oct 2022 14:29:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vgQM4vgy+7KnPWYFNhgrYbDAEMQY1XSSTWC+/NJ93CA=; b=bNqxyI++YuBPUNAz/yQ7k/dbrqkan7eInk8F6HlNkAeoLBjVjp8nWPQqClTW9uEDFB ouAOnUDWooRibYlkdUIDbrq8XcfOkKmECZuWTJu63rTOU5oR3x7lmslb1LMRkMS1/Zuj DHlkCdiRDQUbRRmWGBxCwHf6oddo93oV/glm2VjPz3NbBFLCaEq8kkAJ+j0Iqwdru9x6 lGSHDYUCvHUzNOBdNZX5X7e2WbYcB1L6QGXhUsx+DhHJG+0nAY183nKzp3SpgvpmPpnA xknvQmt8B6OAVNtj0o5PdlEErCGOZDfCkkseC3Tt49FyXJyTxUeqRUaxRPKELfOlgO76 8zNw== X-Gm-Message-State: ACrzQf3BuDTw0gDtEkL4DnH1VyOR7GSRw9+V4xhrpALUld/eADcS7LyJ xzsQEkz3U9iscH2D4XljMcGTJJhpdwqSW5LWgWfPGzzW99pAcxzioNKTQtN0GuTnVHk+mfiuc21 VY9jNWodQdRg= X-Received: by 2002:a05:620a:88a:b0:6f9:c32b:5fc6 with SMTP id b10-20020a05620a088a00b006f9c32b5fc6mr7043711qka.288.1667165374687; Sun, 30 Oct 2022 14:29:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6gRJ4q4CUl52TBJKx7VsDhJCeJVh7fc6aYcv1+iKVnCNyNlKKvVpkjHmjx+cbWeRM6B/g+2w== X-Received: by 2002:a05:620a:88a:b0:6f9:c32b:5fc6 with SMTP id b10-20020a05620a088a00b006f9c32b5fc6mr7043701qka.288.1667165374468; Sun, 30 Oct 2022 14:29:34 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 01/10] mm/hugetlb: Let vma_offset_start() to return start Date: Sun, 30 Oct 2022 17:29:20 -0400 Message-Id: <20221030212929.335473-2-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UV8XJ7js; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165377; a=rsa-sha256; cv=none; b=ijWc9WVV4RvVIpQHDIxB0p9/v58zQTaCUgEUw1lj/IjBy1/HPdrIj/Wxh0V4A4iZoBkuwA 8yMN3UGyPEq4Q2afr34g/v/JQOOf27e5G0t5wYd05GEjbRus0UxEtopWXyno9hhnFkex8Z 33/gFyqeWuiT8b9/NaOZ3/aTv7WhEpA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vgQM4vgy+7KnPWYFNhgrYbDAEMQY1XSSTWC+/NJ93CA=; b=0ogtZht8chxwWsoRGAktsdRbxnH1y3pguq75cRfw7VCtnVPsEH5M3jkt1MCfAbHGIXuowU WF5S2GotlGbqfqe/omJ8ZT+WpZYLtt/LxTq8l0KkRzYnbKiCD2Lk2rDIvankSWIlkK/dfT ZzSMWBlKH3NDfy8VDkfjonVBdIL9a1I= X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UV8XJ7js; spf=pass (imf18.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam07 X-Stat-Signature: odxts4tuuq4tx71pyigki5ib155kabpw X-Rspamd-Queue-Id: BD2A71C003F X-HE-Tag: 1667165377-856352 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though vma_offset_start() is named like that, it's not returning "the start address of the range" but rather the offset we should use to offset the vma->vm_start address. Make it return the real value of the start vaddr, and it also helps for all the callers because whenever the retval is used, it'll be ultimately added into the vma->vm_start anyway, so it's better. Signed-off-by: Peter Xu Reviewed-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 3ee84604e36d..ac3a69fe29c3 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -412,10 +412,12 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, */ static unsigned long vma_offset_start(struct vm_area_struct *vma, pgoff_t start) { + unsigned long offset = 0; + if (vma->vm_pgoff < start) - return (start - vma->vm_pgoff) << PAGE_SHIFT; - else - return 0; + offset = (start - vma->vm_pgoff) << PAGE_SHIFT; + + return vma->vm_start + offset; } static unsigned long vma_offset_end(struct vm_area_struct *vma, pgoff_t end) @@ -457,7 +459,7 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (!hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) + if (!hugetlb_vma_maps_page(vma, v_start, page)) continue; if (!hugetlb_vma_trylock_write(vma)) { @@ -473,8 +475,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h, break; } - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, ZAP_FLAG_DROP_MARKER); + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); hugetlb_vma_unlock_write(vma); } @@ -507,10 +509,9 @@ static void hugetlb_unmap_file_folio(struct hstate *h, */ v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) - unmap_hugepage_range(vma, vma->vm_start + v_start, - v_end, NULL, - ZAP_FLAG_DROP_MARKER); + if (hugetlb_vma_maps_page(vma, v_start, page)) + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); kref_put(&vma_lock->refs, hugetlb_vma_lock_release); hugetlb_vma_unlock_write(vma); @@ -540,8 +541,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, zap_flags); + unmap_hugepage_range(vma, v_start, v_end, NULL, zap_flags); /* * Note that vma lock only exists for shared/non-private From patchwork Sun Oct 30 21:29:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B031FC38A02 for ; Sun, 30 Oct 2022 21:29:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BAFB6B0073; Sun, 30 Oct 2022 17:29:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16A958E0001; Sun, 30 Oct 2022 17:29:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 031AC6B0075; Sun, 30 Oct 2022 17:29:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EA66D6B0073 for ; Sun, 30 Oct 2022 17:29:38 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B7DC11604C6 for ; Sun, 30 Oct 2022 21:29:38 +0000 (UTC) X-FDA: 80078907636.08.D8ED36D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 5CBF620009 for ; Sun, 30 Oct 2022 21:29:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TYAYVHGHmOdGIx+b1vNePbFs/fCGWl7iYc8XlHT8evo=; b=UTV2q1+NLxAVRy9yBjNmGQagjhb6YAtmwHZ0KbfzpyLz1Xr7PjsH51Z3a1s2vOWCV8MG9Y c+aguZ9GsxuFY2iH8GVaTmZaZsR1of71BemC8XhkX9AfpAvnwquP4AJz+ho/uyef88jQdc T51GcvD63tj5owBYdAB6+AlSoEmK3ag= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-424-B0ZYMR7lPnS4ncT9fsOuXQ-1; Sun, 30 Oct 2022 17:29:36 -0400 X-MC-Unique: B0ZYMR7lPnS4ncT9fsOuXQ-1 Received: by mail-qt1-f200.google.com with SMTP id u31-20020a05622a199f00b003a51fa90654so1130154qtc.19 for ; Sun, 30 Oct 2022 14:29:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TYAYVHGHmOdGIx+b1vNePbFs/fCGWl7iYc8XlHT8evo=; b=0xZ6sf2xj+krUR8+GuZOV4c8apGunBImO34DilRSs0sUXd72TrisGV0gp66imPnfMA o/PFDpyswB3HexvJ1hne22U6WxdmUpvl2egwcMh/EW5uiv+7InIxl5fveXlP5MRQp+Vr qsaBsFT5HmNnwAbDe2CO4NKT0rV2h/cx9tQn/+j2QiomlRkU94E1dyY7+ymIoSZVjjzs kgD2LqCaGiI7TeMdKhLjGHRfZOlERm0TXD5ojUpozzivlLSckjDiOzVUbElN8cPQ6AZU Z+rGFwH021HjAh2dOff1HvLimeC2cNe7Yt4tZs66uEsTL4PQUxrr1NAnXBxqXSxDTxzc En8Q== X-Gm-Message-State: ACrzQf18iddY6Qw4UPAHkirrffeGis6+sN0q9u5N7UAiQCJGhBG0VmqD xDwZD6tf0KSKWmmfsxe1jodBxJwMaT4KBpt7xaYp1njyfBPAaKeAYDaKM9uYbKSlHuPDinakqx+ MwzIk94UyHzk= X-Received: by 2002:a05:620a:284a:b0:6ab:9cc5:cb4c with SMTP id h10-20020a05620a284a00b006ab9cc5cb4cmr7121575qkp.609.1667165376109; Sun, 30 Oct 2022 14:29:36 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4KUeAeL4uqFBmuI9YDUOE8dGtH/Wl4ZTYcW7dVrCJqlvE5bFVFwU9LFOVe/w5t4Vrzl2V26A== X-Received: by 2002:a05:620a:284a:b0:6ab:9cc5:cb4c with SMTP id h10-20020a05620a284a00b006ab9cc5cb4cmr7121563qkp.609.1667165375878; Sun, 30 Oct 2022 14:29:35 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:35 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 02/10] mm/hugetlb: Comment huge_pte_offset() for its locking requirements Date: Sun, 30 Oct 2022 17:29:21 -0400 Message-Id: <20221030212929.335473-3-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165378; a=rsa-sha256; cv=none; b=bWjfdfMkdwwPM/+vdtmRZGgdgl3hq8Fv5JkINA4a9wxdQErkLtAoOUTNcS3mPazWeXEGSj 4X0InKe6zZJcCLXJQKilceLpR5Z6b7T+2/JdEnA46pncLb0ltjeMjcO1OaAUf5AW1jMEnq 5Cq916fhLJRX2dA6Ms1di7UjVYWqhRE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UTV2q1+N; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TYAYVHGHmOdGIx+b1vNePbFs/fCGWl7iYc8XlHT8evo=; b=52f3l+o668d203rhl4fQN7LGsmVpN9xqc0Fsf7KYBe13342qXbvvjBwGQykV0BxFMUwgOt tPj8oZjBQj5CpowGKRzrZCbylFQEVDpK7TtZPZigLcWD3Pth837INuAAkUG36uxZU7Ss7Q vpZ1NtbwIvCx+3GTWKS7AX6zYIevSY4= Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UTV2q1+N; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5CBF620009 X-Stat-Signature: p5mxs46z8hjw5udqm1r7ii1ea5seer71 X-Rspam-User: X-HE-Tag: 1667165378-684837 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is potentially a pgtable walker, looking up pte_t* for a hugetlb address. Normally, it's always safe to walk the pgtable as long as we're with the mmap lock held for either read or write, because that guarantees the pgtable pages will always be valid during the process. But it's not true for hugetlbfs: hugetlbfs has the pmd sharing feature, it means that even with mmap lock held, the PUD pgtable page can still go away from under us if pmd unsharing is possible during the walk. It's not always the case, e.g.: (1) If the mapping is private we're not prone to pmd sharing or unsharing, so it's okay. (2) If we're with the hugetlb vma lock held for either read/write, it's okay too because pmd unshare cannot happen at all. Document all these explicitly for huge_pte_offset(), because it's really not that obvious. This also tells all the callers on what it needs to guarantee huge_pte_offset() thread-safety. Signed-off-by: Peter Xu --- arch/arm64/mm/hugetlbpage.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 35e9a468d13e..0bf930c75d4b 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -329,6 +329,38 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, return ptep; } +/* + * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. + * Returns the pte_t* if found, or NULL if the address is not mapped. + * + * NOTE: since this function will walk all the pgtable pages (including not + * only high-level pgtable page, but also PUD that can be unshared + * concurrently for VM_SHARED), the caller of this function should be + * responsible of its thread safety. One can follow this rule: + * + * (1) For private mappings: pmd unsharing is not possible, so it'll + * always be safe if we're with the mmap sem for either read or write. + * This is normally always the case, so IOW we don't need to do + * anything special. + * + * (2) For shared mappings: pmd unsharing is possible (so the PUD-ranged + * pgtable page can go away from under us! It can be done by a pmd + * unshare with a follow up munmap() on the other process), then we + * need either: + * + * (2.1) hugetlb vma lock read or write held, to make sure pmd unshare + * won't happen upon the range (it also makes sure the pte_t we + * read is the right and stable one), or, + * + * (2.2) RCU read lock, to make sure even pmd unsharing happened, the + * old shared PUD page won't get freed from under us, so even of + * the pteval can be obsolete, at least it's still always safe to + * access the pgtable page (e.g., de-referencing pte_t* would not + * cause use-after-free). + * + * PS: from the regard of (2.2), it's the same logic of fast-gup being safe + * for generic mm, as long as RCU is used to free any pgtable page. + */ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz) { From patchwork Sun Oct 30 21:29:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B5FBC38A02 for ; Sun, 30 Oct 2022 21:29:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF0E78E0003; Sun, 30 Oct 2022 17:29:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D7C078E0001; Sun, 30 Oct 2022 17:29:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF3338E0003; Sun, 30 Oct 2022 17:29:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B2E5F8E0001 for ; Sun, 30 Oct 2022 17:29:40 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 88777AAB0E for ; Sun, 30 Oct 2022 21:29:40 +0000 (UTC) X-FDA: 80078907720.01.323F458 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 40B2318000E for ; Sun, 30 Oct 2022 21:29:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wagvClDr96VqW50v7EvOelntcB2vHhVhi8l6aUJsy+c=; b=ew7FqRFPd5YS+Gzcrzv0V/2ob8hXdbkhwRtX9iXwDsDZhDzpRGw5gWyPgGlc70jn2DpTLh AkDjTMvvnaudRq7XOVPepOcA/2C59UUpMRONz1HmS/HJevRwzwvi4iMq0m702T8ov1cFmK hSJSGDKq7NZe1G0YX5b5XAsPEqM6wko= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-151-L1B2XG7nOBucz90uH26Mdg-1; Sun, 30 Oct 2022 17:29:38 -0400 X-MC-Unique: L1B2XG7nOBucz90uH26Mdg-1 Received: by mail-qk1-f197.google.com with SMTP id bk21-20020a05620a1a1500b006be9f844c59so7999375qkb.9 for ; Sun, 30 Oct 2022 14:29:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wagvClDr96VqW50v7EvOelntcB2vHhVhi8l6aUJsy+c=; b=arxG58fNDejEYbKCBdI5OaBa3lI14WtKvnPow5RyW/EgsK/sNmE4sX4oC6R5GNUwYI CqhN5bVksWolm8UO4Jf1PnR0TRjGDLNxjHCJp7ZgGbw6deV3rJoa2YENHkS1qr26FZz3 4Rcx3H5JcgZsWAUvX1rYh5L8GzbSz9ZW+lMBHWQ+w9C6PCyBBmXm/XB96OaoHVcRfSdV 6B30f7MxRn981tIxhZdKQ8wSvsFNeRAM8vV/wyi/ROc9hcwX+qemV/u4JQMt6vZXkSf0 emBlQxWC+cIT2VG142FsW4tBGlJDyjunpm/uZbZMpG7LDQ0fCRcZsglhTYIihFy2/cL4 UF1A== X-Gm-Message-State: ACrzQf0A1nOUlAXyMgTYVo4pRdxebhOk/HN3VxXx+mpfKazQEKQ1vbfe pq92Ufz7fFpaBQ1Ln9mzLD1NV0s7TVHz3q1YoqQX/K68QxREoUpgZCCSDZNNK18s6D68ha6ce/1 Bjt13i06keeE= X-Received: by 2002:a05:620a:270e:b0:6df:7dba:f45a with SMTP id b14-20020a05620a270e00b006df7dbaf45amr6855702qkp.661.1667165377621; Sun, 30 Oct 2022 14:29:37 -0700 (PDT) X-Google-Smtp-Source: AMsMyM43y+Z2mW0srzuBvDKmocMA3VgcwsAHhf82R+MjgsQxdn6rUm6xToTQeXJSbTLNoU9IYHY77A== X-Received: by 2002:a05:620a:270e:b0:6df:7dba:f45a with SMTP id b14-20020a05620a270e00b006df7dbaf45amr6855682qkp.661.1667165377401; Sun, 30 Oct 2022 14:29:37 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:36 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 03/10] mm/hugetlb: Make hugetlb_vma_maps_page() RCU-safe Date: Sun, 30 Oct 2022 17:29:22 -0400 Message-Id: <20221030212929.335473-4-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ew7FqRFP; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165380; a=rsa-sha256; cv=none; b=khxDZvEbqyH2t3roXG1DzES06YsGEw95U3yLu3zygLFe6P40k0r9np4sY/iRWeYa7rkzI1 mtTL3cVdg0r+dIMLo4VHqN930vqxDturiCQQf1m0tkr4VVCyLPPn7/77+Fh8D5yjYLrVmW BbfCCHrbJ+VyPmXov3Z1dlp0g4HDekM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165380; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wagvClDr96VqW50v7EvOelntcB2vHhVhi8l6aUJsy+c=; b=Yob0Fj7prIrgMelSa+2hB6mdyLmjDIzyrzdTD3AaLPigqbW6rA+9N6rQQW7QdZAobxuQRv TuvF70j5dD9IhLyA12J+xuXH751sv2gBrg+XLm+hyqyG5jcvYPy497eRrMXUSGnvc/Mj4e wV5x3vASzVqCTs9b7qGY5czcoA4si3E= X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ew7FqRFP; spf=pass (imf16.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam07 X-Stat-Signature: 76watknc759q4subpp7fpxah8u7gouy4 X-Rspamd-Queue-Id: 40B2318000E X-HE-Tag: 1667165380-511183 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index ac3a69fe29c3..b9e7825079c7 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -387,21 +387,26 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, unsigned long addr, struct page *page) { pte_t *ptep, pte; + bool result = false; + + /* For huge_pte_offset() */ + rcu_read_lock(); ptep = huge_pte_offset(vma->vm_mm, addr, huge_page_size(hstate_vma(vma))); if (!ptep) - return false; + goto out; pte = huge_ptep_get(ptep); if (huge_pte_none(pte) || !pte_present(pte)) - return false; + goto out; if (pte_page(pte) == page) - return true; - - return false; + result = true; +out: + rcu_read_unlock(); + return result; } /* From patchwork Sun Oct 30 21:29:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53227ECAAA1 for ; Sun, 30 Oct 2022 21:29:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB3228E0001; Sun, 30 Oct 2022 17:29:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E63E180007; Sun, 30 Oct 2022 17:29:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D03CC8E0005; Sun, 30 Oct 2022 17:29:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C21658E0001 for ; Sun, 30 Oct 2022 17:29:41 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A07511A049F for ; Sun, 30 Oct 2022 21:29:41 +0000 (UTC) X-FDA: 80078907762.22.8C29327 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 40B8380005 for ; Sun, 30 Oct 2022 21:29:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=agXgQWAuByPHwxJXo49Hx7hkR/BEQEXX5BlO+c7HbVA=; b=F9KeaadkZ5/1tjX8ytmHlhgPASL52gPiqVxikdUdfib+N/gdKEEUM9RstAwKlcib1Wc4N4 Qk3+Ke7NxqSTrjoRtr7okfuy3MhtEuI/IGE0AJqjna8c8JfXvwBaEj5bWX/HytBzLe1rwW bH5/jzqnA0iDMzo8JRtEJzYatRgg56Q= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-286-yUbR5LoGPoq_1As3pPPGtA-1; Sun, 30 Oct 2022 17:29:39 -0400 X-MC-Unique: yUbR5LoGPoq_1As3pPPGtA-1 Received: by mail-qk1-f197.google.com with SMTP id h8-20020a05620a284800b006b5c98f09fbso8156963qkp.21 for ; Sun, 30 Oct 2022 14:29:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=agXgQWAuByPHwxJXo49Hx7hkR/BEQEXX5BlO+c7HbVA=; b=htyn0rjOaVMUMkBIRLxtL+Z4jfZTh6Hey1MHiuKZaJt7gvUDdjWf6Sq6e0mjCvj4y/ jDEIXeqpEHMIujZECmryiRKM2Ep32vyDJyZF8jo6VrGXtEDRiCYvhVDn/1XNxrr/C+le DuKyZAe9xXiQ3UOIX7q6bUsJm6OhwBRdfIcehnRTd+3RRdeip4RNwct8l95DJQHn0/Ip 8LD4yOxA4ee9ko0ElZdwu//GKmemntPMYo6RI30Ao9XYfEcKduW7jQTkY96pucJKGzfW J/lyAjaIzGLqtsxQf2hcgPRMXOSHyS1kC62kL96NMIc3M7WhtGKjSw70TNLosMbjMff3 EjiQ== X-Gm-Message-State: ACrzQf38L4n1QagRV7WWiijFyf81SyzJeldVF6AWHfmZTmV0j3PsCVZd hfQVSdM5mcPCUYr99vG0GNfSj8UYg0ht23wq71HOHCZqeIPFC1u4cDC0qxSbXPBs7eUg070YzZU 4roITKOUpd3U= X-Received: by 2002:a0c:f0d4:0:b0:4bb:6583:66e6 with SMTP id d20-20020a0cf0d4000000b004bb658366e6mr8560478qvl.123.1667165379075; Sun, 30 Oct 2022 14:29:39 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7sce2XU7RtWq2xdMTQLA/Y2bdKWFb6WfMYU+Rxg+ddv0M5GF24IJZTzMXfQ4BlQlC1JXATQQ== X-Received: by 2002:a0c:f0d4:0:b0:4bb:6583:66e6 with SMTP id d20-20020a0cf0d4000000b004bb658366e6mr8560462qvl.123.1667165378831; Sun, 30 Oct 2022 14:29:38 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:38 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 04/10] mm/hugetlb: Make userfaultfd_huge_must_wait() RCU-safe Date: Sun, 30 Oct 2022 17:29:23 -0400 Message-Id: <20221030212929.335473-5-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165381; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=agXgQWAuByPHwxJXo49Hx7hkR/BEQEXX5BlO+c7HbVA=; b=NMnwYa/yHkUtGQKR7UKHq8a65vVXbvhV6tK1nXpD6j4jtxxbuWxMvWYTOMY+aMoKigNNX5 ml+UaH0zptVI7oMZJCUgrPZotlZphxv7ESvADTkHe/CPwCcHlFBn+o1mNmuZx8SkJj052V Wg3vr0mvjeUAUgiH/FGEFvtqZJyLkpo= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F9Keaadk; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165381; a=rsa-sha256; cv=none; b=kcyBQYBHqa/o2ONvs8acnWvZ/d9rgTi8LPFwPHTVLMe9sLPIGXZbu3WECN4jXTeOp1uVbz XXFQ737k4d8uKc0T3Bc8ikU/3E53/Ij/Kh4WgdcKfGx5/l3lEpebr/yCD96+FYwRyBv5wU j4xTzS+mlG9Vuc6fZMOPjB85P7L2qxQ= X-Stat-Signature: 4fqn4jk1bj838bx58ajkeursmazpn9sj X-Rspamd-Queue-Id: 40B8380005 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=F9Keaadk; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1667165381-94907 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 07c81ab3fd4d..4e813e68e4f8 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -243,6 +243,9 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, mmap_assert_locked(mm); + /* For huge_pte_offset() */ + rcu_read_lock(); + ptep = huge_pte_offset(mm, address, vma_mmu_pagesize(vma)); if (!ptep) @@ -261,6 +264,7 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, if (!huge_pte_write(pte) && (reason & VM_UFFD_WP)) ret = true; out: + rcu_read_unlock(); return ret; } #else From patchwork Sun Oct 30 21:29:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87193FA3743 for ; Sun, 30 Oct 2022 21:29:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 209E080008; Sun, 30 Oct 2022 17:29:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16B6580007; Sun, 30 Oct 2022 17:29:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB50C80008; Sun, 30 Oct 2022 17:29:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C6F8080007 for ; Sun, 30 Oct 2022 17:29:43 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A1890140149 for ; Sun, 30 Oct 2022 21:29:43 +0000 (UTC) X-FDA: 80078907846.13.2D3A53E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 5DC191C0008 for ; Sun, 30 Oct 2022 21:29:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165382; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JcZdgMzXcUClzXwFwvMe9vX/XbvITV8H0Y6t4On5idY=; b=BvcrJES4j0nILKfC76UaCHTgFjMycDrDv6w+C8IYnKZ4GiscdDrMn5DoIPp4LnPjV269aJ TNtaj4tsazvYdg4yX38rIlXNAIFg5xuOfXWzbFpuZs0RM2StwfpUiueU6YnzgGp1V+umc9 JAUSwPK7CFfnsHuhqM2Ou+fVRTtIwmo= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-351-ADNiyPdwPfWhKDvxdv6IEg-1; Sun, 30 Oct 2022 17:29:40 -0400 X-MC-Unique: ADNiyPdwPfWhKDvxdv6IEg-1 Received: by mail-qk1-f199.google.com with SMTP id w13-20020a05620a424d00b006e833c4fb0dso7976839qko.2 for ; Sun, 30 Oct 2022 14:29:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JcZdgMzXcUClzXwFwvMe9vX/XbvITV8H0Y6t4On5idY=; b=McaIixJjc/UgV8G6e5GIW8Gn7YNg89zrZp0Odrydlw8pMOPTqWDKANlJQkCGYVk1if 7WPLR6IsODZyXEdrK2yJAerY1qquqeP0Y4otCRAbGWxMaDSfgQcIX7NJH24adcPxVBq8 B6+4AvnXg0LHcuJiR1jfL+E5CWAWGmgbB7/5A5Jfw/KGXj7o97MyVQlfHU61am8zlxGu ZzgBHiLAkU5iGUF0srgkSaXyn/+seWvldkJOGpaWVpZJT2/giHFMMm6BmH2gmcmMFH1z 2fHnaAWlrgiy+gyF8SKGE1NCDu09G1jg2r9kZWkG/+jHtrW9+9ogOhMdtFFiX2dTIVd+ HvOg== X-Gm-Message-State: ACrzQf2XVsXQmnJs+Zwprht/gqRgMyLfHDLIwsquLWWqw/oHc9E9WW5i w1hD7c7IxRGiyfyQe2ZZJTg3S+FsU8B1IqV4xAvbLXTgpps60JMTmbdGzH0ViJLS87xfAf1wH6+ KEyL3LeUCk8A= X-Received: by 2002:ad4:5c4e:0:b0:4bb:9fea:f52e with SMTP id a14-20020ad45c4e000000b004bb9feaf52emr8476673qva.51.1667165380550; Sun, 30 Oct 2022 14:29:40 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7ZnJZXvFuDX4rJ/LrVQdDYcAIkIRly/xQ0vtJY1Rg+h8deNRU6tueG0RHxG/wRYLRwm8iB2w== X-Received: by 2002:ad4:5c4e:0:b0:4bb:9fea:f52e with SMTP id a14-20020ad45c4e000000b004bb9feaf52emr8476659qva.51.1667165380375; Sun, 30 Oct 2022 14:29:40 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:39 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 05/10] mm/hugetlb: Make walk_hugetlb_range() RCU-safe Date: Sun, 30 Oct 2022 17:29:24 -0400 Message-Id: <20221030212929.335473-6-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BvcrJES4; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165383; a=rsa-sha256; cv=none; b=kKDzQ6mdN2Gv9q8TjggUZXmsYX27FmPfNbOg1lD0u/qF7ZcvprVVxzhZq1jHXai93uTquJ MsGmVnb/yLsWZJXn2mmW/ynRFO9mBayX9Lag5rqePVDNUucQ59kf3Ff99XRhDPERDiVoGP kgTXXWq3kCZQhGNow6hIqYZdqN+gbjQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JcZdgMzXcUClzXwFwvMe9vX/XbvITV8H0Y6t4On5idY=; b=ghC4zPMUT1ifPZKnmS26CiIC19oYe7vpcPQsB3H1jif37S/1y88I9mJyOQAfNu4lGy8IVm C/a3BVPrKWnM6jtbXZeg6b6Dxo5krYfYdamt7VD5wE9CsNsPCBGKKGKCHZ3F+z3ErFj2q6 ZujzBF7APDixSUTR6z8APq4NVDlLV7E= X-Rspam-User: X-Rspamd-Queue-Id: 5DC191C0008 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BvcrJES4; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: dfqe48ori9eyewxt7omt8coc5j855771 X-Rspamd-Server: rspam10 X-HE-Tag: 1667165383-737625 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Signed-off-by: Peter Xu --- mm/pagewalk.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 7f1c9b274906..bbc71c750576 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -302,6 +302,9 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, const struct mm_walk_ops *ops = walk->ops; int err = 0; + /* For huge_pte_offset() */ + rcu_read_lock(); + do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz); @@ -315,6 +318,8 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, break; } while (addr = next, addr != end); + rcu_read_unlock(); + return err; } From patchwork Sun Oct 30 21:29:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1964ECAAA1 for ; Sun, 30 Oct 2022 21:29:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA0D980009; Sun, 30 Oct 2022 17:29:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D52DF80007; Sun, 30 Oct 2022 17:29:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7DF180009; Sun, 30 Oct 2022 17:29:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id ABA3E80007 for ; Sun, 30 Oct 2022 17:29:44 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 948551A049F for ; Sun, 30 Oct 2022 21:29:44 +0000 (UTC) X-FDA: 80078907888.12.C743A76 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 3D2D510000D for ; Sun, 30 Oct 2022 21:29:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165383; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WBN8D9wCQW9wV7A+8u6L08Ik6RBdolsM1Jhl7A7rc8s=; b=RpUxgBd+F+W93cmG3fir1GF0Now2WPJzntfgvGt82ReTjvuGTREKd39RUXsXTjmRnH94b8 yzy3KbaLKfHSrA5NieLPIuQXaWR7R+Jx6qTX82uFn5z2LRvxomFmTdzTB2NyDnBBDMndyH T+5cxUrgRIOS0C60DKtrhR6+nyuNCxc= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-413-hz1K7Eg_P5aYFga1E-Ming-1; Sun, 30 Oct 2022 17:29:42 -0400 X-MC-Unique: hz1K7Eg_P5aYFga1E-Ming-1 Received: by mail-qk1-f198.google.com with SMTP id i11-20020a05620a404b00b006eeb0791c1aso8075240qko.10 for ; Sun, 30 Oct 2022 14:29:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WBN8D9wCQW9wV7A+8u6L08Ik6RBdolsM1Jhl7A7rc8s=; b=MKtbzNySE1V4ROU3itS7i3VieqAFQgB90eOBF2ej8995RMM9YiHJqg8BxLUjcQMYD2 rz9hyRcKKK4n8z9QHNKjE24fPCtgK5LVrdvTpkZwVSpoF6tyAL8rSnq5GmtZBvsZK/FO krV0Dz2QNXQwDIgvlCReke2K40cebOyM5Re2FY/qLtkXWFxzhyUexTJkkZgAUTAH+ShX +3Mld4ZV0Xti0MOMM9JWY2IRj7vN0B0fPP1gplU2rvE2op0pQVnCNxIbAVF2d37609Es rfJ90BUP87lfnx2qB7ol6pbB0o9ZJMo+W35fVh0L5zOHht+HOQdJXXNKHs0Cke1iQLNl kgfA== X-Gm-Message-State: ACrzQf1wGFT5EbzzCMpjGPGTVLRI7BXm7UqI1dwkJcp5d9toFywvgO3Q M++RBvXUdBvWTzEWvO6mwuK5YcN5mWNsonb5jqkdnEtajkvCOFmIDvz2ebcXtGScwd8rhDLtKMO nObxcfZteP8c= X-Received: by 2002:a05:620a:404d:b0:6da:4b51:2a8f with SMTP id i13-20020a05620a404d00b006da4b512a8fmr6987354qko.421.1667165381926; Sun, 30 Oct 2022 14:29:41 -0700 (PDT) X-Google-Smtp-Source: AMsMyM42RXOVz0QArfSJMlSE1Zpcejb5AA8L6KVh4LIIheRuXu9MXkBezCcPcA+vujWAM+SNNHsPqw== X-Received: by 2002:a05:620a:404d:b0:6da:4b51:2a8f with SMTP id i13-20020a05620a404d00b006da4b512a8fmr6987344qko.421.1667165381708; Sun, 30 Oct 2022 14:29:41 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:41 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 06/10] mm/hugetlb: Make page_vma_mapped_walk() RCU-safe Date: Sun, 30 Oct 2022 17:29:25 -0400 Message-Id: <20221030212929.335473-7-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WBN8D9wCQW9wV7A+8u6L08Ik6RBdolsM1Jhl7A7rc8s=; b=445lPIUBXS+DARawlFyUSbJCNhbGNGmwa3yX3p0sv229yFoG+mV/pmsT1E6b5sryhzavil Y+OZqRtNb+XWqPNeao3RSEOYx2O7WeJcxaamYRSxbEjstXyxuQkO62bJDnyg275wpT7qd8 QTy97RUyVyInn+IXf4hpZdNcoP1amCw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RpUxgBd+; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165384; a=rsa-sha256; cv=none; b=QwjXad9CVF07AlU1XORYDMzlKHeZH6tQ2Y3adxzDDWETpN0Tij0SYfBxv3NUsDkUCAS8Vp +6td7KSjR0W3TlJUYQPwKcnFBINJqTpYRmJach6dzh0s9n/EUWnuBuitFWf1AkF7kPr4ME ZEldIvN4r05RtI/uNtrhPFrQX1kq0ts= X-Stat-Signature: 74zf91dfg79b8941ppq1ttytd7amfoix X-Rspamd-Queue-Id: 3D2D510000D Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RpUxgBd+; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1667165384-717227 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Signed-off-by: Peter Xu --- include/linux/rmap.h | 3 +++ mm/page_vma_mapped.c | 7 ++++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bd3504d11b15..d2c5e69a56f2 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -408,6 +408,9 @@ static inline void page_vma_mapped_walk_done(struct page_vma_mapped_walk *pvmw) pte_unmap(pvmw->pte); if (pvmw->ptl) spin_unlock(pvmw->ptl); + /* Hugetlb uses RCU lock for safe access of huge_pte_offset() */ + if (is_vm_hugetlb_page(pvmw->vma)) + rcu_read_unlock(); } bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw); diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 93e13fc17d3c..513210a59d7b 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -169,10 +169,15 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pte) return not_found(pvmw); + /* For huge_pte_offset() */ + rcu_read_lock(); + /* when pud is not present, pte will be NULL */ pvmw->pte = huge_pte_offset(mm, pvmw->address, size); - if (!pvmw->pte) + if (!pvmw->pte) { + rcu_read_unlock(); return false; + } pvmw->ptl = huge_pte_lock(hstate, mm, pvmw->pte); if (!check_pte(pvmw)) From patchwork Sun Oct 30 21:29:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A99B3ECAAA1 for ; Sun, 30 Oct 2022 21:30:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A0868000A; Sun, 30 Oct 2022 17:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4508580007; Sun, 30 Oct 2022 17:30:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 318138000A; Sun, 30 Oct 2022 17:30:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2449B80007 for ; Sun, 30 Oct 2022 17:30:04 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 01B881A049F for ; Sun, 30 Oct 2022 21:30:03 +0000 (UTC) X-FDA: 80078908728.02.3A408B2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf09.hostedemail.com (Postfix) with ESMTP id A45C9140004 for ; Sun, 30 Oct 2022 21:30:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165403; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6JldXIZ0VovVxqs2u1isjA5hu6yU0tGljewUccwkM0U=; b=fOYu3G/uyD2A3tFz0cgpBU19KDPlscNqr1Xhf8qDroqBBsTT7RZ7tnuaqr/Sw5FbD1zPXC mXfK5nDJ65dgQJ+k1NssEcFO/GafMj6JPTK/rT/vPUDLXyEL4xZTzHQZfhUEOGRbi47Gvw O0Ij1C3eH040LYvbko8GO5oXZDvphnA= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-416-ET2fLviFPGKsc3VPkTGn5w-1; Sun, 30 Oct 2022 17:29:44 -0400 X-MC-Unique: ET2fLviFPGKsc3VPkTGn5w-1 Received: by mail-qt1-f199.google.com with SMTP id i4-20020ac813c4000000b003a5044a818cso4847020qtj.11 for ; Sun, 30 Oct 2022 14:29:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6JldXIZ0VovVxqs2u1isjA5hu6yU0tGljewUccwkM0U=; b=XeRdvxz1yupoUAbGA95jGy63sttTBw7md26FpQMYSzgEiB46jaDa3UiEIHdC4pMfwn //9aDOVs2jy7yPdbp7IA1ze1iMVrPb4ZpfeK1BDFXXKvYOlHFPHA5aSbVqDg65KD9GEI udmwDi2GLMblEYCZQ1oGQnP4P2gx7CBGof6ezQTDDSaTZNYC7bNv24i1cM9V++WVvBrR eOfVJNK90lYGSJUGmxGXWSowQ6SLofeuv+L7VlK29Xzl/yYv7liePRvGhm6e9n0/sQZo e2RbdSnohzgK0zV/iYL8UP7g+VoDC6J1IMUFdkmQ95lf6ht8wtccQqjxIprubVaN4B3L GdyQ== X-Gm-Message-State: ACrzQf07X5BQ6xHslc5dztcDkSQD9LRs3qUPOUKS5TYK4vLM2+97CKdn oj71JUWfsjtRi9fHHtxbWcuOgTx+ddnXUdEcjiq6aJckhJNL/uBp/XkpP5tG+pDkUuZSCSKzrJ2 8O0nsCnJsi70= X-Received: by 2002:a05:6214:1c09:b0:4b7:f9f6:7d17 with SMTP id u9-20020a0562141c0900b004b7f9f67d17mr8720819qvc.22.1667165383321; Sun, 30 Oct 2022 14:29:43 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5VT0YcB2KUeD0RxKSYpx27URr0maKrp8Ghgz9I1abNoV5z97NF2ZhVFfbknBYsc6tbah94rA== X-Received: by 2002:a05:6214:1c09:b0:4b7:f9f6:7d17 with SMTP id u9-20020a0562141c0900b004b7f9f67d17mr8720803qvc.22.1667165383128; Sun, 30 Oct 2022 14:29:43 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id a1-20020ac81081000000b003a4b88b886esm2654781qtj.96.2022.10.30.14.29.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:29:42 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , James Houghton , Miaohe Lin , David Hildenbrand , Muchun Song , Andrea Arcangeli , Nadav Amit , Mike Kravetz , peterx@redhat.com, Rik van Riel Subject: [PATCH RFC 07/10] mm/hugetlb: Make hugetlb_follow_page_mask() RCU-safe Date: Sun, 30 Oct 2022 17:29:26 -0400 Message-Id: <20221030212929.335473-8-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165403; a=rsa-sha256; cv=none; b=qM2c6TbeILWS4rakMQdFg6qg1OnwqI8a7yLCGGENUCAJBUWlbUFARYQicc8524F/pvzAFJ F9neH9l1owU2unwe69glOW9puIcriA96H+5WXc9J98Srj6X8yeuN7SmdhPxytRVByD1tV8 4E8F9htZLfKgDNpS210GgHXN5iB8hjI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="fOYu3G/u"; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6JldXIZ0VovVxqs2u1isjA5hu6yU0tGljewUccwkM0U=; b=2s4fYF5twWgOfIUaK/bB8q2w2ZozYRnQou/Pnz8+66Vb8BFZejdsDYcdM1QEdhw9XCqiuF UZuHMo2o3AjPnGW9cZ3jMEXHXVIRL+8RClHkFCYMs/vLiILZigHeXGty+XCgJaF2FAvF3l 0cwwDy0O0HjG/C98DASYI11UeOguuC8= Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="fOYu3G/u"; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A45C9140004 X-Stat-Signature: f1m9we67psnbt8nkzt7ozaf1fpasdb5h X-Rspam-User: X-HE-Tag: 1667165403-692618 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Signed-off-by: Peter Xu --- mm/hugetlb.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9869c12e6460..85214095fb85 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6229,10 +6229,12 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; + /* For huge_pte_offset() */ + rcu_read_lock(); retry: pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) - return NULL; + goto out_rcu; ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); @@ -6266,6 +6268,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, } out: spin_unlock(ptl); +out_rcu: + rcu_read_unlock(); return page; } From patchwork Sun Oct 30 21:30:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 654D4FA3744 for ; Sun, 30 Oct 2022 21:30:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 080F18E0001; Sun, 30 Oct 2022 17:30:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00AE96B0074; Sun, 30 Oct 2022 17:30:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DED718E0001; Sun, 30 Oct 2022 17:30:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CC5BF6B0071 for ; Sun, 30 Oct 2022 17:30:46 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B1F781A0227 for ; Sun, 30 Oct 2022 21:30:46 +0000 (UTC) X-FDA: 80078910492.29.995F166 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 4CBF21A000B for ; Sun, 30 Oct 2022 21:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IiwOf4LgCWPWNLGKe48rMbCAvlS3n6l4UUSg+FeDzrg=; b=cDUvgteG8ZYU8c8YrXhT6MRjbT9Nir68rz5SJgER12jRMIxPKGXLR/JneyKKuRtpXJPuU5 sVSUKM9afDwQ8s+GQGm2m5JSoCZjTb8vE5T17aMpJyWMLgOy8pmE5M52szXGpqxxenBKcR Kq82nuHdZBXAic4JgjXagXGfaFluW+U= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-164-qnY4Ntr9Mqu9hcdkXngXHQ-1; Sun, 30 Oct 2022 17:30:35 -0400 X-MC-Unique: qnY4Ntr9Mqu9hcdkXngXHQ-1 Received: by mail-qk1-f197.google.com with SMTP id bp10-20020a05620a458a00b006fa29f253dcso1142224qkb.11 for ; Sun, 30 Oct 2022 14:30:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IiwOf4LgCWPWNLGKe48rMbCAvlS3n6l4UUSg+FeDzrg=; b=G7DG5JehskZUkVDgKOagJPqBQUR+pdT+oB2VPzWbqZpTJ1L7ptF7csVgeN4HHTWu3x AHG5hP6CHwBOi71OC9YcqRZ12TZ3e9aiHRY5lNG3QsuYy24IhKFyVtOjTsb4zAIlxcHv sVDAUXHvq/yVJ7DQhWNmgOAaBmpRFTgJKCkJd/w2olbiAkk5W2uBFXOct6CHkv0SVswk CdkXqAEoOWmOwC0bmaJbfyXOqL75vn7cdGdmDsOWpkGv35JYXcnMxu2wWe3wCxrJd6Jm zs+G0BM7cYXSCwkoz9lvpaliCjBSTatJWwljXWoe8shLHTHJx/tr3PBond/xku782+3A dM3A== X-Gm-Message-State: ACrzQf08cu8kiXBiLkdJyc2jTi1CQzttYQbg3vjr5k5zIW2EK2NSMnhI bIH1hCAMDhxX/9cvfUZHaZxZDmwTqqhAVkddqSNkVzl3LLPSPrzvCyrDw+r0TVgT914shXOJ1Dr 56N2K8IFb53c= X-Received: by 2002:a05:620a:16c7:b0:6fa:313b:f0e0 with SMTP id a7-20020a05620a16c700b006fa313bf0e0mr325877qkn.137.1667165434555; Sun, 30 Oct 2022 14:30:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5tfUshtQQUT1K81sgbv4hWXmjMUpQkS9187l/2AQBRJVYTUcvFXxGStsp5n6ll6i2p3uKefg== X-Received: by 2002:a05:620a:16c7:b0:6fa:313b:f0e0 with SMTP id a7-20020a05620a16c700b006fa313bf0e0mr325867qkn.137.1667165434343; Sun, 30 Oct 2022 14:30:34 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id bm29-20020a05620a199d00b006e42a8e9f9bsm3469443qkb.121.2022.10.30.14.30.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:30:33 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Andrew Morton , Miaohe Lin , Muchun Song , Rik van Riel , James Houghton , Nadav Amit , Mike Kravetz , David Hildenbrand , Andrea Arcangeli , peterx@redhat.com Subject: [PATCH RFC 08/10] mm/hugetlb: Make follow_hugetlb_page RCU-safe Date: Sun, 30 Oct 2022 17:30:32 -0400 Message-Id: <20221030213032.335589-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165446; a=rsa-sha256; cv=none; b=lZR5e0kk+OhKf26+N8IPdYzAoafUwnthUEXHQB32kqphDAvLNdaqw/NHNtjDnNs1v9o87R Yhyq3oiNaT/tyxPXjeoyTxsW0GXoeKn/ww7Heo/PK4zwxrAwoD9N+NJkrMnDK86eokjOkR IrDXtp6s+2LhDvc2wqFKs+sgD3xrTJE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cDUvgteG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IiwOf4LgCWPWNLGKe48rMbCAvlS3n6l4UUSg+FeDzrg=; b=cnXTIltF8zKIly3AahEF5toRItBpmVziJJYAo+WS9RengfdKWmSb+D84A9l/hcI2xP935+ f360O2tHuoLrvBkZNQnesaVFADjzfhOaisXf7/qF691lKrJ+13TC1zE6O8AZ6t7UbpbHYS eS6mVRIBdHNNP5TphAQuemGDIutLkIA= X-Stat-Signature: d7dyb6p7tq5qpq5sc39wzmxzfypd17cp X-Rspamd-Queue-Id: 4CBF21A000B X-Rspam-User: Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cDUvgteG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Rspamd-Server: rspam09 X-HE-Tag: 1667165446-915107 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Some small trick is used to release RCU slightly earlier, but that should be safe and just cleaner (with rich comment in code). Signed-off-by: Peter Xu --- mm/hugetlb.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 85214095fb85..5dc87e4e6780 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6300,6 +6300,9 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, break; } + /* For huge_pte_offset() */ + rcu_read_lock(); + /* * Some archs (sparc64, sh*) have multiple pte_ts to * each hugepage. We have to make sure we get the @@ -6324,6 +6327,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, !hugetlbfs_pagecache_present(h, vma, vaddr)) { if (pte) spin_unlock(ptl); + rcu_read_unlock(); remainder = 0; break; } @@ -6345,6 +6349,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (pte) spin_unlock(ptl); + rcu_read_unlock(); + if (flags & FOLL_WRITE) fault_flags |= FAULT_FLAG_WRITE; else if (unshare) @@ -6387,6 +6393,19 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, continue; } + /* + * When reach here, it means the pteval is not absent, so + * anyone who wants to free and invalidate the pgtable page + * (aka, pte*) should need to first unmap the entries which + * relies on the pgtable lock. Since we're holding it, + * we're safe even without RCU anymore. + * + * We can also just release RCU after each unlock of + * pgtable below, but this is just much cleaner, and also + * smaller critical section. + */ + rcu_read_unlock(); + pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; page = pte_page(huge_ptep_get(pte)); From patchwork Sun Oct 30 21:30:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F4B6ECAAA1 for ; Sun, 30 Oct 2022 21:30:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFF8C8E0002; Sun, 30 Oct 2022 17:30:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E88EA6B0074; Sun, 30 Oct 2022 17:30:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D04F28E0002; Sun, 30 Oct 2022 17:30:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C29AC6B0071 for ; Sun, 30 Oct 2022 17:30:48 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9BA5F1C6209 for ; Sun, 30 Oct 2022 21:30:48 +0000 (UTC) X-FDA: 80078910576.05.7413DEA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 53E2620007 for ; Sun, 30 Oct 2022 21:30:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=18dXjMMHISRNlhGaFpCwLJgIneJr5paDcQBCMHn2vKI=; b=RPpNh4jEPEssaaPDA/aG59xRNzUMZNX1O+j+TdjkMqX5KZZp+pj+q25cxjNwdoQImZdZuU bl1R+dDKOrCpGLa/t3MfG5JJQkpqrwg4e+2TueDjQkyh7KGjCKJPyC87FgpN/uFz3p/GCr EmGqRHq3u0zyjL8oT4wcopwXxfHkMTw= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-317-cJOCLHf2NguLvZs9qc03Ww-1; Sun, 30 Oct 2022 17:30:46 -0400 X-MC-Unique: cJOCLHf2NguLvZs9qc03Ww-1 Received: by mail-qk1-f198.google.com with SMTP id az31-20020a05620a171f00b006fa2cc1b0bfso892759qkb.23 for ; Sun, 30 Oct 2022 14:30:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=18dXjMMHISRNlhGaFpCwLJgIneJr5paDcQBCMHn2vKI=; b=fc7O42iieTUnbo6clBtVNcuREBHxE6MeHDwd7N8od60tO7uVKf/rodef+5qx7l27Bq brX62lroSWd1OK37drLV8K+CqFntTLS7NwpWcVMZUDnW1ifghcEM6bmiJJAIhwUE5jie I904VeVygUx5dP4/JJxhbAKrIHiyV/jY48xkoNY4DXCBaNqt7gja+35IyYjrE3bl0RxR EZkkic0a7IbZWmsIOXqICA4xvAtVE7Q32uHhnC8U6RlnCuoW7Q4ztj4abcQ6ABdGZU4g 18lST7RddhLHaP4JXsaVSv/vnOU15PeqDKSElVeejZNCVOv3hLV4oSGTordcVI086BsO gRqw== X-Gm-Message-State: ACrzQf07YWNyVrNpENJCk+pBdu+2kh1PoFe5Ld7IJlxvKQeRgSxC8/vk XoW2ADUQ20FgM/FX0ABBarXp7bgedWyWofY3iCI6uns3bRMZu/9YUmvT1CVWCsZvLQJs+lpJkeQ IbPyjGP7Hngc6t0cFKC1p+Y1kgpWQw6EtsbmG4h8dwNMI1EP8vKGD52YPUJHg X-Received: by 2002:a05:6214:262f:b0:4b6:89bc:3ba4 with SMTP id gv15-20020a056214262f00b004b689bc3ba4mr8361043qvb.44.1667165445985; Sun, 30 Oct 2022 14:30:45 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4WHS7WPiG6eK0xCLe5+DVpxce9WhNLEtvaEGwRBNhlLTs5Fva2fkLlQL6Z/FlVCcCkdSkbsQ== X-Received: by 2002:a05:6214:262f:b0:4b6:89bc:3ba4 with SMTP id gv15-20020a056214262f00b004b689bc3ba4mr8361010qvb.44.1667165445716; Sun, 30 Oct 2022 14:30:45 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id i3-20020a05620a248300b006ee9d734479sm3533891qkn.33.2022.10.30.14.30.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:30:45 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: James Houghton , Mike Kravetz , David Hildenbrand , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Andrew Morton , Muchun Song , Miaohe Lin , Nadav Amit Subject: [PATCH RFC 09/10] mm/hugetlb: Make hugetlb_fault() RCU-safe Date: Sun, 30 Oct 2022 17:30:43 -0400 Message-Id: <20221030213043.335669-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RPpNh4jE; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165448; a=rsa-sha256; cv=none; b=w2VvEGxQsABDueGZKUxznUV6cz2PVNzdPax/gfZa4Hrrn6fImT68X3899HfUhluzoXF9zO caiRYTWYEo99GPsAVBVMdNsdC1HWZTlUpM+C67IsYegaqIs/VQ4FRe9NCMmldLCtKAkOPL 455dM0wF+VQVaFbEHn/loSh6eKnMW58= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165448; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=18dXjMMHISRNlhGaFpCwLJgIneJr5paDcQBCMHn2vKI=; b=lqtIM1S758vQrPvB6/JnzaKJOFGtsHUI8+41wMAYMZNDL8zQ3FMxs2Q8escpQ8ISQRJ0/P L4R9vYVyCy83k93qpo7BJOq/iXyNz9HzxTk6AWgvD5804e2rHUtay4XewjRF4hiZz4iB0A v1CIcyxgYpScact3WtYisv+YWPenzeM= X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RPpNh4jE; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam07 X-Stat-Signature: 4gdrq1dwypwuubaphixy97qhmbiipwrq X-Rspamd-Queue-Id: 53E2620007 X-HE-Tag: 1667165448-649438 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: RCU makes sure the pte_t* won't go away from under us. Please refer to the comment above huge_pte_offset() for more information. Signed-off-by: Peter Xu --- mm/hugetlb.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5dc87e4e6780..6d336d286394 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5822,6 +5822,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); + /* For huge_pte_offset() */ + rcu_read_lock(); ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (ptep) { /* @@ -5830,13 +5832,15 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * not actually modifying content here. */ entry = huge_ptep_get(ptep); + rcu_read_unlock(); if (unlikely(is_hugetlb_entry_migration(entry))) { migration_entry_wait_huge(vma, ptep); return 0; } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) return VM_FAULT_HWPOISON_LARGE | VM_FAULT_SET_HINDEX(hstate_index(h)); - } + } else + rcu_read_unlock(); /* * Serialize hugepage allocation and instantiation, so that we don't From patchwork Sun Oct 30 21:30:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13025223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 811F4FA3743 for ; Sun, 30 Oct 2022 21:30:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22EC380007; Sun, 30 Oct 2022 17:30:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B9BF6B0074; Sun, 30 Oct 2022 17:30:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 031FC80007; Sun, 30 Oct 2022 17:30:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E5A6F6B0071 for ; Sun, 30 Oct 2022 17:30:50 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AD875A0B52 for ; Sun, 30 Oct 2022 21:30:50 +0000 (UTC) X-FDA: 80078910660.07.F2F6A91 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 4D4034002E for ; Sun, 30 Oct 2022 21:30:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667165449; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MbdbMcNyE2RWt/K55AkSFMkKO4f3Jk1aXGE/HUSEbTg=; b=WWeMO2p3pXRJSKZ0YpYUfsaE8cijgITzNdHQLflllfQNHb1Zyt+3w3GbvixkP96ACef93D wib2wSM21kadKl3T29la0z8RMbVFr7ULR6+INFwOu9yBUXOwHpXzyWFvwPYIfao7GXiegp NeWgg5Z65H2JVFm+AhqYkIiVsC6LuhA= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-232-XJjXpi7GMQezIjec9P468w-1; Sun, 30 Oct 2022 17:30:48 -0400 X-MC-Unique: XJjXpi7GMQezIjec9P468w-1 Received: by mail-qk1-f198.google.com with SMTP id bj1-20020a05620a190100b006fa12a05188so3848546qkb.4 for ; Sun, 30 Oct 2022 14:30:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MbdbMcNyE2RWt/K55AkSFMkKO4f3Jk1aXGE/HUSEbTg=; b=GqtFW/zJYtcH/GtTjC6U5ZD2MOgZb3nj1QJhgOSgh+p7thS2r13Hu+W4sV79bgg8z4 QrPiXw4vfOW8pMIHHQQ216/owLC3K2htvZR8xhBb5TNydKD5Hl9sNqnAe/10Nx3pHEN0 tKP8tMDwVkV/BUlovA/YlX8GiM2OdapL/14eKIxWhZAm0WuyOfHmkQ88j0fU/ohXKwCC 4w8z9fViWROl7AI+ZXeA8QN5+hum1ZmzMa9wy+e+DWp1qhj7UjLV2UgUzkOYDSRprVbE pI+U1f7PcC1ilLnzv02Yy+YK2Gx93VtekVaOe0scS18jkul7CXYpPQuUo3TPM1UjF/Z6 Eelg== X-Gm-Message-State: ACrzQf0FqlVZmRd31yA2v1vx2y/RtbNoOEFgTdBV7ESknrEEYM2rJQXS Xpf8lTjYXxpBXR0D0KxQV2DFsFUGHUUHzDoyMCYKUYB0EuI2kOn39fOEBnWW/GzT4XuxqferNDi gS8kHAwPqIZU/PMfY2f7Pr8HqkEorGXJB4uT1KdIIKi9D0REmjXKYqd9npGwD X-Received: by 2002:a0c:e44d:0:b0:4bb:6814:cacc with SMTP id d13-20020a0ce44d000000b004bb6814caccmr8623265qvm.73.1667165448138; Sun, 30 Oct 2022 14:30:48 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6jOrHvdqiTcpqwmdfHOEF5EeLgusoteeQUtfIu8VXZPch/zK/jAZabG/q9ulUqzuJxcqfFYA== X-Received: by 2002:a0c:e44d:0:b0:4bb:6814:cacc with SMTP id d13-20020a0ce44d000000b004bb6814caccmr8623237qvm.73.1667165447749; Sun, 30 Oct 2022 14:30:47 -0700 (PDT) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id bk20-20020a05620a1a1400b006f3e6933bacsm3482823qkb.113.2022.10.30.14.30.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 14:30:47 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: James Houghton , Mike Kravetz , David Hildenbrand , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Andrew Morton , Muchun Song , Miaohe Lin , Nadav Amit Subject: [PATCH RFC 10/10] mm/hugetlb: Comment at rest huge_pte_offset() places Date: Sun, 30 Oct 2022 17:30:45 -0400 Message-Id: <20221030213045.335680-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221030212929.335473-1-peterx@redhat.com> References: <20221030212929.335473-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667165450; a=rsa-sha256; cv=none; b=Ir/nPLwjBGF7TUloGzRHofK7vWpbUktwZVFoNWa4j/d67N8dlYTgWyz4qmcsa+9eCGtSg2 m/i+3Eb1DhNTDSHwr5B5gqjrP+NE+n98Bf2ZN9+eXOX6sfjEJPZYY9Mm+T9mxlxL3/Ro/L TyxcaJxYbwq8uloYNkWNo1mgcXW/9aE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WWeMO2p3; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667165450; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MbdbMcNyE2RWt/K55AkSFMkKO4f3Jk1aXGE/HUSEbTg=; b=L3UQF12b7DqbKMwVlF9loIw4m+PcMZjqZBfSLb954+sdLOZ9dVxtRIt03jw1ClVbmVahMd UytYWCb3JaVmuaW7n3TQgAfL3d+nN4Yq+z54+RDFVsLsov7mJ8f7yd+efSdCSL0wcl+4mW bRSz+bIMEIOZAJ1quu0LQdfkQT3Rlmo= X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WWeMO2p3; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com X-Stat-Signature: xdo6otodctaub1qgq73akciqex8q8zft X-Rspamd-Queue-Id: 4D4034002E X-Rspamd-Server: rspam06 X-HE-Tag: 1667165450-358850 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This makes sure that we're covering all the existing huge_pte_offset() callers and mention why they are safe regarding to pmd unsharing. Signed-off-by: Peter Xu --- mm/hugetlb.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6d336d286394..270bfc578115 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4822,6 +4822,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; + /* With vma lock held, safe without RCU */ src_pte = huge_pte_offset(src, addr, sz); if (!src_pte) { addr |= last_addr_mask; @@ -5026,6 +5027,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { + /* With vma lock held, safe without RCU */ src_pte = huge_pte_offset(mm, old_addr, sz); if (!src_pte) { old_addr |= last_addr_mask; @@ -5097,6 +5099,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct last_addr_mask = hugetlb_mask_last_page(h); address = start; for (; address < end; address += sz) { + /* With vma lock held, safe without RCU */ ptep = huge_pte_offset(mm, address, sz); if (!ptep) { address |= last_addr_mask; @@ -5402,6 +5405,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); spin_lock(ptl); + /* With vma lock held, safe without RCU */ ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) @@ -5440,6 +5444,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * before the page tables are altered */ spin_lock(ptl); + /* With vma lock (and even pgtable lock) held, safe without RCU */ ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { /* Break COW or unshare */ @@ -6511,6 +6516,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, last_addr_mask = hugetlb_mask_last_page(h); for (; address < end; address += psize) { spinlock_t *ptl; + /* With vma lock held, safe without RCU */ ptep = huge_pte_offset(mm, address, psize); if (!ptep) { address |= last_addr_mask; @@ -7060,7 +7066,14 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, saddr = page_table_shareable(svma, vma, addr, idx); if (saddr) { + /* + * huge_pmd_share() (or say its solo caller, + * huge_pte_alloc()) always takes the hugetlb vma + * lock, so it's always safe to walk the pgtable of + * the process, even without RCU. + */ spte = huge_pte_offset(svma->vm_mm, saddr, + vma_mmu_pagesize(svma)); if (spte) { get_page(virt_to_page(spte)); @@ -7420,6 +7433,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) hugetlb_vma_lock_write(vma); i_mmap_lock_write(vma->vm_file->f_mapping); for (address = start; address < end; address += PUD_SIZE) { + /* With vma lock held, safe without RCU */ ptep = huge_pte_offset(mm, address, sz); if (!ptep) continue;