From patchwork Thu Jun 10 06:34:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E29E3C47094 for ; Thu, 10 Jun 2021 06:34:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 59A29613CA for ; Thu, 10 Jun 2021 06:34:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59A29613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C62CC6B0036; Thu, 10 Jun 2021 02:34:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C12FB6B006E; Thu, 10 Jun 2021 02:34:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB4216B0070; Thu, 10 Jun 2021 02:34:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0176.hostedemail.com [216.40.44.176]) by kanga.kvack.org (Postfix) with ESMTP id 7B9256B0036 for ; Thu, 10 Jun 2021 02:34:44 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 087738249980 for ; Thu, 10 Jun 2021 06:34:44 +0000 (UTC) X-FDA: 78236850888.01.5BAFDE2 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf28.hostedemail.com (Postfix) with ESMTP id 348102001086 for ; Thu, 10 Jun 2021 06:34:39 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id k4so26344024qkd.0 for ; Wed, 09 Jun 2021 23:34:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=pGCwo0otsY+1TNpTEy0P02naUBtXpTOD+IdeOTLcTwE=; b=lhPqqonl51gMi2Y5qmIjfHtlL9DeApoXuasczVJ23S+iGW2PmFNNYItqWpfLH7MhZ4 tVUlRtMBckVtYWTfO4DQfuc6RAOI7KFG6ogHzngJvwQfijIa0reDXsTc5nFZ85mfHrtc tTIp3DP5ZAdekIFoTHFbqzZHtD/3BeHN/SwIg5s5jF0lrhMnpRyHAV8ndVZPwr7+MFzl D/VqCz4s5wU4VpJQvp8gp6PmoM8o014bXIulyrzJaseBUHPQCbeU0DU/yhbSWW3BpGNL O/n3mhNqbCzVYV7fx3t6RLDhmfHZ0ATmPxZBeFcJ8tMbU8zN6fU0BOtmvkk2oYTpTiCi o2Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=pGCwo0otsY+1TNpTEy0P02naUBtXpTOD+IdeOTLcTwE=; b=SYlrsPkoiEfThh61QUHbE4+bTWbGpR8XhZYlKkclpJ8QmQ9PyP8u3t7uL1nzTODa+X FphyhzGC/4e5aEjjEiACNDAwSnkBy49nJnR0RGZyY7PSOnmeCCt07TTn2lFe+rShBxFd wNLhiNkElns7m0U4YJ88/bd0CGSGKKDlbSgLVk7nEd8VZROtzqJW8D+OkXD0C+2G+KXt L4/55d8GFQUKsv93enjDMYnThgfU+UMNZXvENhYGGdGFbYgFRlNq76dfW5+mqgtPMOR1 VygUzEoW0nwn0ORcdebQWcGx/GhlwRQH8xmMQL53NoAhVbHTVELhoyvbWAU7vvVMsyFM YyLA== X-Gm-Message-State: AOAM531JBuEDteuM1+glM4q0yr7GaOSRQ3CdPzLB4hWbrZ88MQruACAk Y67dGZDhJZIZn8SN43dPVDVZrA== X-Google-Smtp-Source: ABdhPJx795mENXThcbrSF3SFHQ7mxH7leIj6hOSzSyIIAiXcTG/9Yj4OerHKB8IVGGrOX291lJY05Q== X-Received: by 2002:a37:438d:: with SMTP id q135mr3166885qka.478.1623306882954; Wed, 09 Jun 2021 23:34:42 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id b2sm1692660qkd.1.2021.06.09.23.34.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:34:42 -0700 (PDT) Date: Wed, 9 Jun 2021 23:34:40 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/11] mm: page_vma_mapped_walk(): use page for pvmw->page In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <88e67645-f467-c279-bf5e-af4b5c6b13eb@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=lhPqqonl; spf=pass (imf28.hostedemail.com: domain of hughd@google.com designates 209.85.222.177 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 348102001086 X-Stat-Signature: 4g4ea75gpgkb71w176yubhhck5gt8c3t X-HE-Tag: 1623306879-556355 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: sometimes the local copy of pvwm->page was used, sometimes pvmw->page itself: use the local copy "page" throughout. Signed-off-by: Hugh Dickins Cc: Reviewed-by: Alistair Popple Acked-by: Kirill A. Shutemov Reviewed-by: Peter Xu --- mm/page_vma_mapped.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index e37bd43904af..a6dbf714ca15 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -156,7 +156,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pte) goto next_pte; - if (unlikely(PageHuge(pvmw->page))) { + if (unlikely(PageHuge(page))) { /* when pud is not present, pte will be NULL */ pvmw->pte = huge_pte_offset(mm, pvmw->address, page_size(page)); if (!pvmw->pte) @@ -217,8 +217,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) * cannot return prematurely, while zap_huge_pmd() has * cleared *pmd but not decremented compound_mapcount(). */ - if ((pvmw->flags & PVMW_SYNC) && - PageTransCompound(pvmw->page)) { + if ((pvmw->flags & PVMW_SYNC) && PageTransCompound(page)) { spinlock_t *ptl = pmd_lock(mm, pvmw->pmd); spin_unlock(ptl); @@ -234,9 +233,9 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return true; next_pte: /* Seek to next pte only makes sense for THP */ - if (!PageTransHuge(pvmw->page) || PageHuge(pvmw->page)) + if (!PageTransHuge(page) || PageHuge(page)) return not_found(pvmw); - end = vma_address_end(pvmw->page, pvmw->vma); + end = vma_address_end(page, pvmw->vma); do { pvmw->address += PAGE_SIZE; if (pvmw->address >= end) From patchwork Thu Jun 10 06:36:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C976C47094 for ; Thu, 10 Jun 2021 06:36:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC104613CB for ; Thu, 10 Jun 2021 06:36:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC104613CB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 66D3B6B0036; Thu, 10 Jun 2021 02:36:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 61D5D6B006E; Thu, 10 Jun 2021 02:36:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BDF76B0070; Thu, 10 Jun 2021 02:36:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0038.hostedemail.com [216.40.44.38]) by kanga.kvack.org (Postfix) with ESMTP id 1D9A96B0036 for ; Thu, 10 Jun 2021 02:36:40 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A89114820 for ; Thu, 10 Jun 2021 06:36:39 +0000 (UTC) X-FDA: 78236855718.03.10E3105 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf02.hostedemail.com (Postfix) with ESMTP id C91A74202A03 for ; Thu, 10 Jun 2021 06:36:36 +0000 (UTC) Received: by mail-qv1-f48.google.com with SMTP id c10so11421047qvo.9 for ; Wed, 09 Jun 2021 23:36:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=rZYXw6BRbvhvPhfR4jmUYFMFeMI725FP/8d3gS3xgfI=; b=kGmVYq1KCmC0sxjG2bvNJSAIEma0Txe8awTrb1MSDSPgZXAAmX8mMA0PtfBAImtoHM qSauEy6+Am03n80BYZHU9UEOnC2BoINeerjMvYrYgNX2nJmjsXTUU4yOttfanzIg+Z10 izk8JK2zBz9zFSIg9mOdz5EjjvZqKvGsMJ2MYUoyP9qIW+de6rkSxrrXHnHWJpVZ227k ZQI3LJfwaow8uofHvJ1hY/m61JlpqgX2ZoIuxharrtNEirkWgwqNsyj0XBAo7zg440KQ 9WVwWWwLzPdV8d5xIKtSgzFL13Gb62Cqh6fMFffXHRj3u48MRMr2E6Qh0/lUPxEltrlo EjqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=rZYXw6BRbvhvPhfR4jmUYFMFeMI725FP/8d3gS3xgfI=; b=NzflixEuB5oZ5YZJcpRNMPQ4Kn297eFD1VeG2FRyoooXpU381tpGqwre3REAtfPmtO Yi15dn4rU41YbIQds2KyLcT2JEUltj2UQStXVyi3SlXXi22haimp5Iq6exOJpTHWLyex LGGqLtdxzhzwhC8yoPmllZcxL5Yb3YUem8G+SrrFTwWpLdp+qYfM9UQ3Gx9Rb1K9zP/L zvJo9GUFjdErLw7k+SqMGOEIbRibur+81U8GgkoheI4Ia1DerELcPT6PYFJESlMqCpYS 9CefbJ5cs1KPXxYABXBmgfdI4h2GawALPfOfKqdV9bA3euCJV4GET3Y5NCO5wFZfEUw8 kq1w== X-Gm-Message-State: AOAM532Yc/Z1LjWkCF5sGj7RsnQEcgzVpjbFJy7ePYnucPe7d/em/jeO +HMdLvuiIX39TS3koSEtB/yJFQ== X-Google-Smtp-Source: ABdhPJyLamliZT4RHXjzNV5dKT4qjdHxdf/Mhbgo2rrm2QWLrwhPqsgsN3rbxP0uPfGa3iMSDtfciA== X-Received: by 2002:a05:6214:162a:: with SMTP id e10mr3723237qvw.49.1623306998481; Wed, 09 Jun 2021 23:36:38 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id q2sm1635598qkc.77.2021.06.09.23.36.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:36:38 -0700 (PDT) Date: Wed, 9 Jun 2021 23:36:36 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/11] mm: page_vma_mapped_walk(): settle PageHuge on entry In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=kGmVYq1K; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.219.48 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: wz8d1fg73weoehckn9zek6cdbacnkhem X-Rspamd-Queue-Id: C91A74202A03 X-Rspamd-Server: rspam06 X-HE-Tag: 1623306996-399952 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: get the hugetlbfs PageHuge case out of the way at the start, so no need to worry about it later. Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Reviewed-by: Peter Xu --- mm/page_vma_mapped.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index a6dbf714ca15..7c0504641fb8 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -153,10 +153,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pmd && !pvmw->pte) return not_found(pvmw); - if (pvmw->pte) - goto next_pte; - if (unlikely(PageHuge(page))) { + /* The only possible mapping was handled on last iteration */ + if (pvmw->pte) + return not_found(pvmw); + /* when pud is not present, pte will be NULL */ pvmw->pte = huge_pte_offset(mm, pvmw->address, page_size(page)); if (!pvmw->pte) @@ -168,6 +169,9 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return not_found(pvmw); return true; } + + if (pvmw->pte) + goto next_pte; restart: pgd = pgd_offset(mm, pvmw->address); if (!pgd_present(*pgd)) @@ -233,7 +237,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return true; next_pte: /* Seek to next pte only makes sense for THP */ - if (!PageTransHuge(page) || PageHuge(page)) + if (!PageTransHuge(page)) return not_found(pvmw); end = vma_address_end(page, pvmw->vma); do { From patchwork Thu Jun 10 06:38:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE921C47094 for ; Thu, 10 Jun 2021 06:38:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D855613CB for ; Thu, 10 Jun 2021 06:38:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D855613CB Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E82816B0036; Thu, 10 Jun 2021 02:38:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E32306B006E; Thu, 10 Jun 2021 02:38:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C84EB6B0070; Thu, 10 Jun 2021 02:38:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0234.hostedemail.com [216.40.44.234]) by kanga.kvack.org (Postfix) with ESMTP id 924DF6B0036 for ; Thu, 10 Jun 2021 02:38:15 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 2C3BC8249980 for ; Thu, 10 Jun 2021 06:38:15 +0000 (UTC) X-FDA: 78236859750.22.A598485 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by imf15.hostedemail.com (Postfix) with ESMTP id 7FB69A00025F for ; Thu, 10 Jun 2021 06:38:13 +0000 (UTC) Received: by mail-qt1-f171.google.com with SMTP id l17so15992920qtq.12 for ; Wed, 09 Jun 2021 23:38:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=sJXaknDNqIwNp1rWQgx+PfiGckf+KTxU5H8H8+f9QU0=; b=b1XjZwsGuZXQBj/2tSNWtimTETxoUPqdgZyvsNvx8vkkjvIi8Ygjkp1bT84QOB3lNJ ixcBPLAR/OAzhh7P3V0jtbZh1ooj6X0xFHLsy7KQXcbQ8aOSQ6l5kFOJe3AOYcBv3NqT E7KEwyIfiR+iWAyTglS9872q46W0IfZLa0SZYfNtlhT4vzUpAaLa4jBB11OLKKxErsCz AUCWU0p9KNTvh9jcrLiiaQLay/yDLrASCVHxeq4MehjsqX2LfiSe0CDrQJRsXa/g5jgy b4/ByIHtTGaaggP1grvtIyVnLQrgK0I/cXPR2CZPovkVvpl3KwUUFq2Epp/oDGC+1osu yznw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=sJXaknDNqIwNp1rWQgx+PfiGckf+KTxU5H8H8+f9QU0=; b=UFBuOpTwv/JU5VVGm+do81DlhEgGfpd06ACkQxjgsITYJ4PfdcI8qd39ZXA7IFXlFa rAFoQ1cTBPTEyU4GayQZ5M5mImcbQZHJzIn8J1CwoxNCQLWGqBRkiHchjvkolE+ELryk SPIyTGZDbRQWPc10InofADIQeH9gzdz0Szmm5HexNRAczwq2PqzgWCDMd00KhAHZHEDK QECTXbH2vwOi0H/gWH0OxLl2cwQv0lA8GHyKEfu8Zcu0DXbqA2V3+LbhTvbgD6UQXCWG JfrLwywmHqFKPXnYFR/NS+4qFtdHsv0gkGj2hLiLFrIWFPK4wFRVUP9YCoVHF7nymuYM JIDQ== X-Gm-Message-State: AOAM533koq/+yuBp6MZ6+S+jU30bs9KdxnLlXfmzpeKluNlORTA7uMbi 7YcUo5IVeLQihbRcYSH/v8zDcg== X-Google-Smtp-Source: ABdhPJy5ZcKX50ID5llLhv4bLtcmcu81J9Dm4F8WNmXVszglwbobG1CeiGKZ+cv2mz2wkusrlKG8cg== X-Received: by 2002:ac8:4ccf:: with SMTP id l15mr3701125qtv.174.1623307094084; Wed, 09 Jun 2021 23:38:14 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id c20sm1716101qtm.52.2021.06.09.23.38.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:38:13 -0700 (PDT) Date: Wed, 9 Jun 2021 23:38:11 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/11] mm: page_vma_mapped_walk(): use pmd_read_atomic() In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <594c1f0-d396-5346-1f36-606872cddb18@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7FB69A00025F X-Stat-Signature: hyqpnfay5kqknoxzmjo3aqj1zt9c6ujq Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=b1XjZwsG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of hughd@google.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=hughd@google.com X-HE-Tag: 1623307093-603084 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: use pmd_read_atomic() with barrier() instead of READ_ONCE() for pmde: some architectures (e.g. i386 with PAE) have a multi-word pmd entry, for which READ_ONCE() is not good enough. Signed-off-by: Hugh Dickins Cc: --- mm/page_vma_mapped.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 7c0504641fb8..973c3c4e72cc 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -182,13 +182,16 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) pud = pud_offset(p4d, pvmw->address); if (!pud_present(*pud)) return false; + pvmw->pmd = pmd_offset(pud, pvmw->address); /* * Make sure the pmd value isn't cached in a register by the * compiler and used as a stale value after we've observed a * subsequent update. */ - pmde = READ_ONCE(*pvmw->pmd); + pmde = pmd_read_atomic(pvmw->pmd); + barrier(); + if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde)) { pvmw->ptl = pmd_lock(mm, pvmw->pmd); if (likely(pmd_trans_huge(*pvmw->pmd))) { From patchwork Thu Jun 10 06:40:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FE9FC47094 for ; Thu, 10 Jun 2021 06:40:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E10EA613CA for ; Thu, 10 Jun 2021 06:40:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E10EA613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7A1EB6B0036; Thu, 10 Jun 2021 02:40:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 752A96B006E; Thu, 10 Jun 2021 02:40:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CB9C6B0070; Thu, 10 Jun 2021 02:40:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0093.hostedemail.com [216.40.44.93]) by kanga.kvack.org (Postfix) with ESMTP id 26DF16B0036 for ; Thu, 10 Jun 2021 02:40:12 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BF66B813E for ; Thu, 10 Jun 2021 06:40:11 +0000 (UTC) X-FDA: 78236864622.20.7146EA2 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf11.hostedemail.com (Postfix) with ESMTP id 242FA200107C for ; Thu, 10 Jun 2021 06:40:06 +0000 (UTC) Received: by mail-qk1-f175.google.com with SMTP id u30so26285951qke.7 for ; Wed, 09 Jun 2021 23:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=kXGE7AZWMRkQI2D2fv4mdt0RT7InHtOBXYXrcKUASZQ=; b=pTtcgnuHkfB+ZWeRrnJcb4kua/2i5mFlvyyK1oUUKP3IwMOaFUBnHVR9givA9IegE8 PsdB/++dX/+OMuBz4wPTjhABOSluN+50HARmKc3x+/6ml6z6v37WHyEnwfWIRWiVfBd8 0W7z2Quygw8xMioTE864MUQdheJ1VMEWk+iiEBOKwF5S1yr+NTTVWzs6+Xr0wzUyvGji OB8b5UecOci6aX3c/2UupL8jk6+KmwDXJUnIXtGnmUv0CNuMawdEbWTKR3+6WCFlYAiq u1zMHesyCTki6CpLiOnqP5+qDgsiBH3IFHJi/8Y1iPixSIKZ4cNbkatGORXJEW0okFlD M4hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=kXGE7AZWMRkQI2D2fv4mdt0RT7InHtOBXYXrcKUASZQ=; b=n70wx0TGiZs8S9pWET6XNpoIMFlLzhjO6IoTKQ9dkpRvOo3WPNNCtrQ28GAelHzsbO pf3zNQapCMmSGXhTMqCFpNWk0ydiGkLvKOLftgxjFC0CJfbYUx0jw+4NxDgAAfxjj0Fk IZpWpxrBJPlf0Rrb3fXa/nyyjZuXFeW/p28Lb5Ae3s/GxXIZMeLSAs3l5PKx2ayf+esj 41iWImXY5aNMbE4AV5iWuS1mZYwof7pztd0Kw2KebyR9BAJTvbKLMn6ic+0/SKdKIlHd iESD2yO/a/3UXITSgJ9w1xYQdmtnWf6KQN2UnARiyUJMrfwqkGfGC13F1xXBqKUYydy/ ssqQ== X-Gm-Message-State: AOAM531gNFlFwnJOpHCJzLLw+YgHFdR7evNY2RxbVyIuR1cRUK09aLGJ pGDUbGAW9UE+ZfKPFIJ2NjpCAA== X-Google-Smtp-Source: ABdhPJxlHE0YxyFCF53h4uqHehxwG7bqDEAol3I+xWrbbx4f0ummmIi1JMXE/u++r1xFunWK7pFJ7Q== X-Received: by 2002:a05:620a:147a:: with SMTP id j26mr3322349qkl.138.1623307210706; Wed, 09 Jun 2021 23:40:10 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a68sm1632172qkd.51.2021.06.09.23.40.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:40:10 -0700 (PDT) Date: Wed, 9 Jun 2021 23:40:08 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/11] mm: page_vma_mapped_walk(): use pmde for *pvmw->pmd In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <53fbc9d-891e-46b2-cb4b-468c3b19238e@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 242FA200107C X-Stat-Signature: msgbn5qjjp3fun8n8c7s6icofdh4redu Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=pTtcgnuH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of hughd@google.com designates 209.85.222.175 as permitted sender) smtp.mailfrom=hughd@google.com X-HE-Tag: 1623307206-738339 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: re-evaluate pmde after taking lock, then use it in subsequent tests, instead of repeatedly dereferencing pointer. Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Reviewed-by: Peter Xu --- mm/page_vma_mapped.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 973c3c4e72cc..81000dd0b5da 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -194,18 +194,19 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde)) { pvmw->ptl = pmd_lock(mm, pvmw->pmd); - if (likely(pmd_trans_huge(*pvmw->pmd))) { + pmde = *pvmw->pmd; + if (likely(pmd_trans_huge(pmde))) { if (pvmw->flags & PVMW_MIGRATION) return not_found(pvmw); - if (pmd_page(*pvmw->pmd) != page) + if (pmd_page(pmde) != page) return not_found(pvmw); return true; - } else if (!pmd_present(*pvmw->pmd)) { + } else if (!pmd_present(pmde)) { if (thp_migration_supported()) { if (!(pvmw->flags & PVMW_MIGRATION)) return not_found(pvmw); - if (is_migration_entry(pmd_to_swp_entry(*pvmw->pmd))) { - swp_entry_t entry = pmd_to_swp_entry(*pvmw->pmd); + if (is_migration_entry(pmd_to_swp_entry(pmde))) { + swp_entry_t entry = pmd_to_swp_entry(pmde); if (migration_entry_to_page(entry) != page) return not_found(pvmw); From patchwork Thu Jun 10 06:42:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4C9FC47094 for ; Thu, 10 Jun 2021 06:42:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3173B613CA for ; Thu, 10 Jun 2021 06:42:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3173B613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B22E76B0036; Thu, 10 Jun 2021 02:42:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD2186B006E; Thu, 10 Jun 2021 02:42:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 972E36B0070; Thu, 10 Jun 2021 02:42:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0228.hostedemail.com [216.40.44.228]) by kanga.kvack.org (Postfix) with ESMTP id 62DFA6B0036 for ; Thu, 10 Jun 2021 02:42:17 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 04E9C180AD81D for ; Thu, 10 Jun 2021 06:42:17 +0000 (UTC) X-FDA: 78236869914.10.B64561C Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf07.hostedemail.com (Postfix) with ESMTP id ABA9FA0001A9 for ; Thu, 10 Jun 2021 06:42:11 +0000 (UTC) Received: by mail-qk1-f170.google.com with SMTP id j184so26306226qkd.6 for ; Wed, 09 Jun 2021 23:42:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=BF7gYv4bUzMzXTGdnVKsNQ49KcCRY8OjCfC9I0iFikc=; b=GK3lO1UZgj0viQsuAlknaSC++U0LASLzlwmP3Ygwbb0R9vdkntk33kVgOb7rxaj4mv yxELPE7Wjnfn5+KYoCameACweq3+yaeqsgFT02R4Enrx84iAhpVMmP9vYxfrkrQl+Kng uxPGPPZQY5yq4sqOdyU8EhRoY8yxvHKoWeg9g0uZxHFxVolQ6WQnJI/bjtzolnTrCdUM 1wVj2lTu50F52DoY7yru/rfi+5TOyctn1dnsEZeLWvRdVpMJknSFUXsKHSmQ1lx/3Yv3 ENPncba//NbOxmU6ZHEO65cr0hMI9dEqQrNvqtd6/uNamY9ozwlilcN9U39yjjxwV6AD PF9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=BF7gYv4bUzMzXTGdnVKsNQ49KcCRY8OjCfC9I0iFikc=; b=mW0br4P44ZiF++w/RJdRLsQUrJqXhmnM28NCnBURaOKbBizuh/4UMGcumGltAuioiT 4Tf1lWihltCXk0hxR5/Wn0TiAGx7RRI6xwIz2qFOKLzaNTBCo8EeQTfRi6IZKfXhgZyR Nn5Y2/p0/6g+SYi8s96rh+/hgQLdV7Et3yvMQj9F12AldNtwH7c7C+PpSvjrklAY/VtZ ZLDXjGGumZjdSgAXlfj8avW6R0y7OYYbqcrz5zn3Y0DLVW1yj2xiKozsCXnGzOyrNyYA siIDE/0rj5p2wmZ4nmxlQglRS1Kp/JYNqrww+ph162/mtHelDFj+pi2iyPvzjvHeLdNO rBwA== X-Gm-Message-State: AOAM531mQoq2Wx7mOWHFV7ecDnxM9+fScHJu+0zfaYHrN8gsx+hpaMLL z23oy76rodcSzi37Wwg3Hw73TA== X-Google-Smtp-Source: ABdhPJyXu+PktDehHYGHVmTT0YocIN/sxyUPwE+UTTh86S5eE20bKBx41KVp+5SfZNFFJyhSi8avtw== X-Received: by 2002:a37:a107:: with SMTP id k7mr3267352qke.362.1623307335910; Wed, 09 Jun 2021 23:42:15 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m14sm1671302qti.12.2021.06.09.23.42.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:42:15 -0700 (PDT) Date: Wed, 9 Jun 2021 23:42:12 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/11] mm: page_vma_mapped_walk(): prettify PVMW_MIGRATION block In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <378c8650-1488-2edf-9647-32a53cf2e21@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=GK3lO1UZ; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.222.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: 9aohw8bq1uwmeganohxnutjh5zq6ygwc X-Rspamd-Queue-Id: ABA9FA0001A9 X-Rspamd-Server: rspam06 X-HE-Tag: 1623307331-717270 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: rearrange the !pmd_present() block to follow the same "return not_found, return not_found, return true" pattern as the block above it (note: returning not_found there is never premature, since existence or prior existence of huge pmd guarantees good alignment). Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Reviewed-by: Peter Xu --- mm/page_vma_mapped.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 81000dd0b5da..b96fae568bc2 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -201,24 +201,22 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pmd_page(pmde) != page) return not_found(pvmw); return true; - } else if (!pmd_present(pmde)) { - if (thp_migration_supported()) { - if (!(pvmw->flags & PVMW_MIGRATION)) - return not_found(pvmw); - if (is_migration_entry(pmd_to_swp_entry(pmde))) { - swp_entry_t entry = pmd_to_swp_entry(pmde); + } + if (!pmd_present(pmde)) { + swp_entry_t entry; - if (migration_entry_to_page(entry) != page) - return not_found(pvmw); - return true; - } - } - return not_found(pvmw); - } else { - /* THP pmd was split under us: handle on pte level */ - spin_unlock(pvmw->ptl); - pvmw->ptl = NULL; + if (!thp_migration_supported() || + !(pvmw->flags & PVMW_MIGRATION)) + return not_found(pvmw); + entry = pmd_to_swp_entry(pmde); + if (!is_migration_entry(entry) || + migration_entry_to_page(entry) != page) + return not_found(pvmw); + return true; } + /* THP pmd was split under us: handle on pte level */ + spin_unlock(pvmw->ptl); + pvmw->ptl = NULL; } else if (!pmd_present(pmde)) { /* * If PVMW_SYNC, take and drop THP pmd lock so that we From patchwork Thu Jun 10 06:44:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B43AC47094 for ; Thu, 10 Jun 2021 06:44:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ADE42613E2 for ; Thu, 10 Jun 2021 06:44:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ADE42613E2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 41AAB6B0036; Thu, 10 Jun 2021 02:44:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CAD26B006E; Thu, 10 Jun 2021 02:44:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26BD76B0070; Thu, 10 Jun 2021 02:44:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0239.hostedemail.com [216.40.44.239]) by kanga.kvack.org (Postfix) with ESMTP id EB52D6B0036 for ; Thu, 10 Jun 2021 02:44:14 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8243E40E1 for ; Thu, 10 Jun 2021 06:44:14 +0000 (UTC) X-FDA: 78236874828.30.E44A189 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf22.hostedemail.com (Postfix) with ESMTP id 2C21AC005A20 for ; Thu, 10 Jun 2021 06:44:08 +0000 (UTC) Received: by mail-qv1-f42.google.com with SMTP id e18so14162481qvm.10 for ; Wed, 09 Jun 2021 23:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=akymlcTjak2aiLxju3nvGrvixCmoGtWlsFfmG/UDbtE=; b=MkEt15JmyYSAQ+RVuy42Ljti0kzjYxMG1NcV1cwy50nRl6vRoncOkzghyDe4se7H+c GK3oAdp+cr0pMytWAFXnCaZIP4PXXPmxt/GGtorivsAlQ9wr1WPTY9pb3okv4y0hwjWF f0ig6Dbf2GWBXHvC60iY2h+R1mWvaMi7U/s/zfiDez2w0O2cFGJPFrLauq2XgwlaEa1Z JmvhibYkqss7F/ZFx9im9UzrewbbehaLv92KMQFhWZ9noj0+mgW2LhK3ahjypVq7E/v6 Drh9gSNvhsUL9EHI8CUMP13xFZiYfJNRf6yVODI6iBHsjEKBeRao0mE/LGsEJqoPyc9U 9UYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=akymlcTjak2aiLxju3nvGrvixCmoGtWlsFfmG/UDbtE=; b=KHqaO0TCukj+ylEBsB2/+/hL0MmFOoINMYpajQJeOSQI6DStCeJU2WrPVN+sVJo1T6 1zlDijrTGEnYsLJB9hTGH/AGlmv6Ga96hOqAP1KHprwzo+YGdYJitJZN4CMeWusFyd9W FvAKhmOkgPGTDl0tjflohvAP+TJJLKbwh2Zid5WpVG/TxS0zjT2o+XIbg5gDlhnEWtoN t1IwH7jSqBZilIdT6QRbcChNX6T3e3a10At1SnydDqXk16glPF+wZ6ZIazBjaeaVg3/E lzoib35gP1iBq/f6MKyI1wIAqQhHzuEbRiPzFuc4HDxF2K4wPjsEwPEpQ0pw7XaDNIDX ONKg== X-Gm-Message-State: AOAM532MACSGos2l1N0bFlGR1BZ98vSkJ3rJsF/OY7GCUCQNbeUGtvQs GfSA9j+ekgmGacwSYL0djKBRuQ== X-Google-Smtp-Source: ABdhPJw+h+5g4qsaqPq9hBVqb+DYjhTLlr01z83/WwZH9CfNzJzW4AMFaeYoGQbJMjmA43tYbaijbg== X-Received: by 2002:a05:6214:260b:: with SMTP id gu11mr3840211qvb.12.1623307453261; Wed, 09 Jun 2021 23:44:13 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id p8sm1598335qkm.119.2021.06.09.23.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:44:12 -0700 (PDT) Date: Wed, 9 Jun 2021 23:44:10 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/11] mm: page_vma_mapped_walk(): crossing page table boundary In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <799b3f9c-2a9e-dfef-5d89-26e9f76fd97@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=MkEt15Jm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.219.42 as permitted sender) smtp.mailfrom=hughd@google.com X-Stat-Signature: owpgkbsyqci1ib7qw1xt59y4z4bqeo4f X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2C21AC005A20 X-HE-Tag: 1623307448-946894 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: adjust the test for crossing page table boundary - I believe pvmw->address is always page-aligned, but nothing else here assumed that; and remember to reset pvmw->pte to NULL after unmapping the page table, though I never saw any bug from that. Signed-off-by: Hugh Dickins Cc: --- mm/page_vma_mapped.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index b96fae568bc2..0fe6e558d336 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -247,16 +247,16 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->address >= end) return not_found(pvmw); /* Did we cross page table boundary? */ - if (pvmw->address % PMD_SIZE == 0) { - pte_unmap(pvmw->pte); + if ((pvmw->address & (PMD_SIZE - PAGE_SIZE)) == 0) { if (pvmw->ptl) { spin_unlock(pvmw->ptl); pvmw->ptl = NULL; } + pte_unmap(pvmw->pte); + pvmw->pte = NULL; goto restart; - } else { - pvmw->pte++; } + pvmw->pte++; } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { From patchwork Thu Jun 10 06:46:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5FC4C47094 for ; Thu, 10 Jun 2021 06:46:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6A240613D9 for ; Thu, 10 Jun 2021 06:46:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6A240613D9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C2E956B0036; Thu, 10 Jun 2021 02:46:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB8366B006E; Thu, 10 Jun 2021 02:46:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A09CE6B0070; Thu, 10 Jun 2021 02:46:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id 6ACFD6B0036 for ; Thu, 10 Jun 2021 02:46:34 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0B7A14820 for ; Thu, 10 Jun 2021 06:46:34 +0000 (UTC) X-FDA: 78236880708.04.CADE6C6 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf16.hostedemail.com (Postfix) with ESMTP id 7083B8019357 for ; Thu, 10 Jun 2021 06:46:29 +0000 (UTC) Received: by mail-qv1-f47.google.com with SMTP id 5so3900548qvf.1 for ; Wed, 09 Jun 2021 23:46:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=2OT9ZOT1k9Jrl0NBTEVPMljz+ghdujF4gPDHMzwzXkI=; b=fItDKAwEWa27P6UoPqgsJmxWxx+88uY7YcweWA0CjJkHQ/a0qq3VMLFQGCheHOetC3 thbHJ+GkAgYvV9MvOGSd1SedriGSLoRyUGD7SHaGqOJaIRvBfO6VzJLBx101C7QOIdqe 7d/xyw4cAq1x1sC14413xqHk+aX9nGDcmYOH6HsbEY3Sq0M8O6KTTCRTfTwfZUc0oO74 9qz1oWxVUORVI69H72mEyT4hMad64PunjlNNCmPyWhv6XK2woi+jrJlx4x0eMuLLOvL4 gNsTLQ7nISZaJNkvP4IIVj25ZkvW+15nQxnmN9XZOX9TYBCNpUuD3it60s6CWLrEjfHl 19ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=2OT9ZOT1k9Jrl0NBTEVPMljz+ghdujF4gPDHMzwzXkI=; b=tq76goTOAOD6w5CmC3ptXJWapyt23IsoXpcH2r1RQBvi2szGV5ScuhexioSmU/175/ PKoSWvalXT6io+FS3yEVNOEdNwI2Y4ZA29Xy/fO7AEWG8xcoTiGRPsunRwj/FBB5mxyJ qskglnO15DNWBvcsFoj9Asjjssk4okAvClRbbqFSvZdpCxeVKeLC7mpYYmSqkWPIWWTM 1HUNl5uCtcrBX+4k63Q79BBf5D3CEICYVWC69a1LxXBISZiJFaPxP2+kMvQMtvqaWMnQ O7VYRM8qADY98yo8LagZN8BQhnpmja8iMDYxfl3JINERMsr0LDwZYqHzlQYuHY6o2mT/ xu3w== X-Gm-Message-State: AOAM532Y+2P/0iL51a98jNfGfOSj0LWKrH+p2Jn1+2EqxRvgrlmSR8ff zdtOK5RA2ZPnd1F+LYa7TX4l2g== X-Google-Smtp-Source: ABdhPJzebxEMV8nIRPyxc3ps1IgHzw1v6T3H/U3jGmnUmUAEsDFERk8zIQP7dpPhp/qi25qO4XO93A== X-Received: by 2002:a0c:d610:: with SMTP id c16mr3670609qvj.13.1623307593003; Wed, 09 Jun 2021 23:46:33 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x8sm1675789qkb.54.2021.06.09.23.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:46:32 -0700 (PDT) Date: Wed, 9 Jun 2021 23:46:30 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/11] mm: page_vma_mapped_walk(): add a level of indentation In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 7083B8019357 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=fItDKAwE; spf=pass (imf16.hostedemail.com: domain of hughd@google.com designates 209.85.219.47 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: zzq5hozei1s4iszakneke8tgh54qbqam X-HE-Tag: 1623307589-710453 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: add a level of indentation to much of the body, making no functional change in this commit, but reducing the later diff when this is all converted to a loop. Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov --- mm/page_vma_mapped.c | 109 +++++++++++++++++++++++---------------------- 1 file changed, 56 insertions(+), 53 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 0fe6e558d336..0840079ef7d2 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -173,65 +173,68 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pte) goto next_pte; restart: - pgd = pgd_offset(mm, pvmw->address); - if (!pgd_present(*pgd)) - return false; - p4d = p4d_offset(pgd, pvmw->address); - if (!p4d_present(*p4d)) - return false; - pud = pud_offset(p4d, pvmw->address); - if (!pud_present(*pud)) - return false; - - pvmw->pmd = pmd_offset(pud, pvmw->address); - /* - * Make sure the pmd value isn't cached in a register by the - * compiler and used as a stale value after we've observed a - * subsequent update. - */ - pmde = pmd_read_atomic(pvmw->pmd); - barrier(); - - if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde)) { - pvmw->ptl = pmd_lock(mm, pvmw->pmd); - pmde = *pvmw->pmd; - if (likely(pmd_trans_huge(pmde))) { - if (pvmw->flags & PVMW_MIGRATION) - return not_found(pvmw); - if (pmd_page(pmde) != page) - return not_found(pvmw); - return true; - } - if (!pmd_present(pmde)) { - swp_entry_t entry; + { + pgd = pgd_offset(mm, pvmw->address); + if (!pgd_present(*pgd)) + return false; + p4d = p4d_offset(pgd, pvmw->address); + if (!p4d_present(*p4d)) + return false; + pud = pud_offset(p4d, pvmw->address); + if (!pud_present(*pud)) + return false; - if (!thp_migration_supported() || - !(pvmw->flags & PVMW_MIGRATION)) - return not_found(pvmw); - entry = pmd_to_swp_entry(pmde); - if (!is_migration_entry(entry) || - migration_entry_to_page(entry) != page) - return not_found(pvmw); - return true; - } - /* THP pmd was split under us: handle on pte level */ - spin_unlock(pvmw->ptl); - pvmw->ptl = NULL; - } else if (!pmd_present(pmde)) { + pvmw->pmd = pmd_offset(pud, pvmw->address); /* - * If PVMW_SYNC, take and drop THP pmd lock so that we - * cannot return prematurely, while zap_huge_pmd() has - * cleared *pmd but not decremented compound_mapcount(). + * Make sure the pmd value isn't cached in a register by the + * compiler and used as a stale value after we've observed a + * subsequent update. */ - if ((pvmw->flags & PVMW_SYNC) && PageTransCompound(page)) { - spinlock_t *ptl = pmd_lock(mm, pvmw->pmd); + pmde = pmd_read_atomic(pvmw->pmd); + barrier(); + + if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde)) { + pvmw->ptl = pmd_lock(mm, pvmw->pmd); + pmde = *pvmw->pmd; + if (likely(pmd_trans_huge(pmde))) { + if (pvmw->flags & PVMW_MIGRATION) + return not_found(pvmw); + if (pmd_page(pmde) != page) + return not_found(pvmw); + return true; + } + if (!pmd_present(pmde)) { + swp_entry_t entry; - spin_unlock(ptl); + if (!thp_migration_supported() || + !(pvmw->flags & PVMW_MIGRATION)) + return not_found(pvmw); + entry = pmd_to_swp_entry(pmde); + if (!is_migration_entry(entry) || + migration_entry_to_page(entry) != page) + return not_found(pvmw); + return true; + } + /* THP pmd was split under us: handle on pte level */ + spin_unlock(pvmw->ptl); + pvmw->ptl = NULL; + } else if (!pmd_present(pmde)) { + /* + * If PVMW_SYNC, take and drop THP pmd lock so that we + * cannot return prematurely, while zap_huge_pmd() has + * cleared *pmd but not decremented compound_mapcount(). + */ + if ((pvmw->flags & PVMW_SYNC) && + PageTransCompound(page)) { + spinlock_t *ptl = pmd_lock(mm, pvmw->pmd); + + spin_unlock(ptl); + } + return false; } - return false; + if (!map_pte(pvmw)) + goto next_pte; } - if (!map_pte(pvmw)) - goto next_pte; while (1) { unsigned long end; From patchwork Thu Jun 10 06:48:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44E5AC47094 for ; Thu, 10 Jun 2021 06:48:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D734C613CA for ; Thu, 10 Jun 2021 06:48:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D734C613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 639096B0036; Thu, 10 Jun 2021 02:48:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EADE6B006E; Thu, 10 Jun 2021 02:48:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 463B26B0070; Thu, 10 Jun 2021 02:48:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0132.hostedemail.com [216.40.44.132]) by kanga.kvack.org (Postfix) with ESMTP id 0EDFE6B0036 for ; Thu, 10 Jun 2021 02:48:31 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 88DAF2484 for ; Thu, 10 Jun 2021 06:48:30 +0000 (UTC) X-FDA: 78236885580.10.B7147AE Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf13.hostedemail.com (Postfix) with ESMTP id C3954E000263 for ; Thu, 10 Jun 2021 06:48:25 +0000 (UTC) Received: by mail-qt1-f177.google.com with SMTP id r20so4838850qtp.3 for ; Wed, 09 Jun 2021 23:48:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=XjbJY3bhcvz+MAqs2y5uU2ExaAj/cTy23+6+m0vOS/c=; b=I2WARsLE20TkLuIVtRJfaR8NkDW8Mh5SgTQetC6Ey2uz8oIN+hK8GOKLV//ylUEunK UYegTMLZhwgLPDXrCDGfFt3fJ7kfM3u3u0mdeqh/W+ap9iybINc16XhxcA0IMt+pSPwm xAV56qp/7PhWmC+inxJFChXmiH6CQWjftP6luik04P2FnKSrMCrer6qIhA+VDMa+ZKJp HOAORdh7B5Aw12YZSxAP0b1XFbjWG0fGB37ywSBa5Ct9uB1C3gy3866G3HG1nxRyGIsh ukbE+hvq6jmARKs34+kHlujPcAsY2C5dTNGMgbsQe0ZrPK0+gqF/sqB6f2uhiS36Aw+v jAwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=XjbJY3bhcvz+MAqs2y5uU2ExaAj/cTy23+6+m0vOS/c=; b=W/QB/Y0m59loaYII48GnurzNBSDc5LdTZwy1FaXL9QT2KjVjc4AUyOQt9c7l6UrrzD 1FM8+Q4BPID0tyvmXxVqRq6TxDGLcrvdQ6lx9dTf/Q5psESxV1UAlMXoQwW+Qt6ckxmD 4iz48cIs0lpxj/AmuSR1n3wzdNxHLwTrnk0da+u9Gw/l1EU/k9p7qE4ZzrwRTs++6eJx 2nHRkbwRvFcGw0tmukQ6JvfbFn1jTC9v1Fua/+lg75OGYKyf7f3gl5SUQBHFEgS7iNxp 1mDj01fwRaVl7bu0js310r7Si7sSnBaxZMCN2kHygJPosus6bRR320rdFwqKEIUUoJLt 6qcQ== X-Gm-Message-State: AOAM532MJYaQ2U6XxM1bqIRm1fyAlqwJU991XDC5w4PY7P7yxAF4RVIm Phnz+4ZvMkjwFG/EbHsyr0sgYQ== X-Google-Smtp-Source: ABdhPJyrJW/zsSc4BAg8bOs1VCqXKeTl38AuBEzX0YuNIvPHH3vnd5WTuOWVltVf8BYzC10bBAVy5g== X-Received: by 2002:ac8:4f42:: with SMTP id i2mr96547qtw.350.1623307709464; Wed, 09 Jun 2021 23:48:29 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id g15sm1723464qkl.53.2021.06.09.23.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:48:29 -0700 (PDT) Date: Wed, 9 Jun 2021 23:48:27 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/11] mm: page_vma_mapped_walk(): use goto instead of while (1) In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C3954E000263 X-Stat-Signature: ydck3dofi1t7y7pg1qkrpdpkc3zoje7u Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=I2WARsLE; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of hughd@google.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=hughd@google.com X-HE-Tag: 1623307705-176984 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: add a label this_pte, matching next_pte, and use "goto this_pte", in place of the "while (1)" loop at the end. Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov --- mm/page_vma_mapped.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 0840079ef7d2..006f4814abbc 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -144,6 +144,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) { struct mm_struct *mm = pvmw->vma->vm_mm; struct page *page = pvmw->page; + unsigned long end; pgd_t *pgd; p4d_t *p4d; pud_t *pud; @@ -234,10 +235,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) } if (!map_pte(pvmw)) goto next_pte; - } - while (1) { - unsigned long end; - +this_pte: if (check_pte(pvmw)) return true; next_pte: @@ -266,6 +264,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) pvmw->ptl = pte_lockptr(mm, pvmw->pmd); spin_lock(pvmw->ptl); } + goto this_pte; } } From patchwork Thu Jun 10 06:50:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DC00C47094 for ; Thu, 10 Jun 2021 06:50:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BD843613CA for ; Thu, 10 Jun 2021 06:50:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BD843613CA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4B0436B0036; Thu, 10 Jun 2021 02:50:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4608C6B006E; Thu, 10 Jun 2021 02:50:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DAAD6B0070; Thu, 10 Jun 2021 02:50:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id EF6836B0036 for ; Thu, 10 Jun 2021 02:50:26 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 7CB99181AEF1F for ; Thu, 10 Jun 2021 06:50:26 +0000 (UTC) X-FDA: 78236890452.05.C4E891A Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) by imf18.hostedemail.com (Postfix) with ESMTP id 17475200107C for ; Thu, 10 Jun 2021 06:50:21 +0000 (UTC) Received: by mail-qv1-f44.google.com with SMTP id x6so9273037qvx.4 for ; Wed, 09 Jun 2021 23:50:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=MW3P4dEVpzQaKkCia9N+0MaZJYQT+2IAhJVUZg7cfs8=; b=Do8aT0u+DdZZq8kaOVvB6kgFwGS6mToQyGIoS2wWY4uZ3M/EtH9HIcV12t4Dntbu4B GSIWT43fzAd1hVrjp33IgBv/via5r3NIF7MXRDjCH4X2pxhRMwAV7HcouDjLk6kIda0G C77fRSpITnyPd2/XRz/RJyywcVeEZYG/HfryDo45NoUasSyLQPKi1euqRC+PfEg+vddE PwCwcq6Iqa3GiwU4B6VYphKpI242PJS34F4JI0jv5njbSjFOXldXBll/yYyqDWeVRGb4 STufu8xuwTwB7QL2nHvVLIOTQ95uumyfHuejvqKOZ/YfEufUrAtwIpjTPqeUgPnIP4BZ 2yLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=MW3P4dEVpzQaKkCia9N+0MaZJYQT+2IAhJVUZg7cfs8=; b=GQ53IYe2QpIA1HEg9YIFeZQl1NpCJSe1y/ggSTjhijcYyiz/um3mgoL9sCgWN7+Ck2 UvpAgWxLldec7+CIhv3qP9Q/ZRub0eYRxj7gZjhN4icfl+9cBcz/VADu0OyO3TbNtVUd DzBMMLTOqF+g8USSl3A/r1OhkGuW8RdHatoMLoBEjJI/Oc01m2l1x5Yxer+LJGRRQ9PN iTwtF8A5bakQUtjFU7Z18e0m+zkJ7fQBgIo1c2k5HqYQRhHp67NfWWjbqfeRSV0qlvH0 QRqPnmEzQz2mL3TL+6FJvPedTCLA1NH9XVZlxK3OVKTcylV8JXKGNVjJeNAOU4n1TGsM Yu/w== X-Gm-Message-State: AOAM532mHT9E4D2EJCjrjGRiZ4Q0xC9hOrMwyBGXJQLENOd3QBSCuFyc MF0MpUVzxSO6l+M/0LHzLo3QGw== X-Google-Smtp-Source: ABdhPJxZswruLW9txlbS5dufC/7Inh5U9BCmjlx6PQDp97tfhKPQgM3yJjJJ09wVZQfRUJlz0FRbqw== X-Received: by 2002:ad4:5fc7:: with SMTP id jq7mr3417734qvb.41.1623307825295; Wed, 09 Jun 2021 23:50:25 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 85sm1678043qko.14.2021.06.09.23.50.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:50:25 -0700 (PDT) Date: Wed, 9 Jun 2021 23:50:23 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/11] mm: page_vma_mapped_walk(): get vma_address_end() earlier In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <4542b34d-862f-7cb4-bb22-e0df6ce830a2@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=Do8aT0u+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of hughd@google.com designates 209.85.219.44 as permitted sender) smtp.mailfrom=hughd@google.com X-Rspamd-Server: rspam02 X-Stat-Signature: uxom1zex63ep5qna6bmjb5nzb5gt7hm5 X-Rspamd-Queue-Id: 17475200107C X-HE-Tag: 1623307821-933182 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_vma_mapped_walk() cleanup: get THP's vma_address_end() at the start, rather than later at next_pte. It's a little unnecessary overhead on the first call, but makes for a simpler loop in the following commit. Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov --- mm/page_vma_mapped.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 006f4814abbc..f6839f536645 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -171,6 +171,15 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return true; } + /* + * Seek to next pte only makes sense for THP. + * But more important than that optimization, is to filter out + * any PageKsm page: whose page->index misleads vma_address() + * and vma_address_end() to disaster. + */ + end = PageTransCompound(page) ? + vma_address_end(page, pvmw->vma) : + pvmw->address + PAGE_SIZE; if (pvmw->pte) goto next_pte; restart: @@ -239,10 +248,6 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (check_pte(pvmw)) return true; next_pte: - /* Seek to next pte only makes sense for THP */ - if (!PageTransHuge(page)) - return not_found(pvmw); - end = vma_address_end(page, pvmw->vma); do { pvmw->address += PAGE_SIZE; if (pvmw->address >= end) From patchwork Thu Jun 10 06:52:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311981 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60FC7C47094 for ; Thu, 10 Jun 2021 06:52:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A7BD613DE for ; Thu, 10 Jun 2021 06:52:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A7BD613DE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 942246B0036; Thu, 10 Jun 2021 02:52:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 919696B006E; Thu, 10 Jun 2021 02:52:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7922B6B0070; Thu, 10 Jun 2021 02:52:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 45EEC6B0036 for ; Thu, 10 Jun 2021 02:52:41 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DB4AD5DDC for ; Thu, 10 Jun 2021 06:52:40 +0000 (UTC) X-FDA: 78236896080.13.10AAFEB Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf07.hostedemail.com (Postfix) with ESMTP id 8EC66A000257 for ; Thu, 10 Jun 2021 06:52:35 +0000 (UTC) Received: by mail-qk1-f176.google.com with SMTP id k4so26376546qkd.0 for ; Wed, 09 Jun 2021 23:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=wPomTLe6CFftHojVSA5u1wexcpwAxzEQRGtDVYSW6+s=; b=iI++N3FVNIIBdaZPkAyQ282hUtC9qN+UvCMayG6voYHqIBNoZg9iTLOV8/yobWd0qD t5j/wpMpnNQpZX0fvnA0I7clVpIc0Z/D/eOqSRq2x5fAOxilbxDRaH47SQra+Jtl95gG 6rjZt9jmANiCu553oPYamuydpCiIBfDeCuo8I07jAOxVEpXLoQZkqbpatpVGTDDzKNwR NuozFU/z4okvWcVDq4DIwr2iTzt13KJqKW8qwosEz7s8Tvlf4bzBblR5CtO3Z3JEY8X9 ppgPbqauTyQUhoJ5eMr1mU226mQPTSf9ZJuVDwcpm2dS8Nwas4iL01nwwfDDfihW8bmm UFpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=wPomTLe6CFftHojVSA5u1wexcpwAxzEQRGtDVYSW6+s=; b=kE1a2dD1u1BAEEXEOWYIKxHCFmVixFzGVQLz3TXqXmBsiSZw21HLC2wYysXXcIIZ/c 2KM5JFvAGkUbq0BysZTFBxWLHJEPpnfsCFdWdLuq9SVKelCurYm0iOLZldAkgxl4aA0F DJUa1qr2RPhFq0CunKfxURMYnZwYpb+uQFd5mBnm9Coe/bqY6Mi//wa0zJ6n6m/D2D1+ 27PSmeahtbHKo1uCujyoneyQ2SQeBX0wK/mVvwbfG4tcHpKg05MqoZShNlrRiO9xfmku zp/Qg5ndvASs16GycKCYFS43lnlMHWzSCHFssaKo608VXfzfATynI6hlYOjZMxjYaJWm N+mg== X-Gm-Message-State: AOAM531DTd3uX+7D40Kxrl096MbpTnGj8PS5o6cR4i5uabA/hnZvhBYM SGLBf2WW7H26S0a/7rTIkjKG2w== X-Google-Smtp-Source: ABdhPJyQI0UMif8PlESv2eTJgfLB9PUenRlpUP2M0SClQAZIdQdIkhs67p1JmPDKCwhCQr3B8G8OiQ== X-Received: by 2002:a05:620a:e89:: with SMTP id w9mr3215130qkm.256.1623307959660; Wed, 09 Jun 2021 23:52:39 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id o16sm1635297qtw.10.2021.06.09.23.52.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:52:39 -0700 (PDT) Date: Wed, 9 Jun 2021 23:52:37 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/11] mm/thp: fix page_vma_mapped_walk() if THP mapped by ptes In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8EC66A000257 X-Stat-Signature: iidhhn7uqg8xoexi5w176ufib36c8713 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=iI++N3FV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.222.176 as permitted sender) smtp.mailfrom=hughd@google.com X-HE-Tag: 1623307955-766748 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Running certain tests with a DEBUG_VM kernel would crash within hours, on the total_mapcount BUG() in split_huge_page_to_list(), while trying to free up some memory by punching a hole in a shmem huge page: split's try_to_unmap() was unable to find all the mappings of the page (which, on a !DEBUG_VM kernel, would then keep the huge page pinned in memory). Crash dumps showed two tail pages of a shmem huge page remained mapped by pte: ptes in a non-huge-aligned vma of a gVisor process, at the end of a long unmapped range; and no page table had yet been allocated for the head of the huge page to be mapped into. Although designed to handle these odd misaligned huge-page-mapped-by-pte cases, page_vma_mapped_walk() falls short by returning false prematurely when !pmd_present or !pud_present or !p4d_present or !pgd_present: there are cases when a huge page may span the boundary, with ptes present in the next. Restructure page_vma_mapped_walk() as a loop to continue in these cases, while keeping its layout much as before. Add a step_forward() helper to advance pvmw->address across those boundaries: originally I tried to use mm's standard p?d_addr_end() macros, but hit the same crash 512 times less often: because of the way redundant levels are folded together, but folded differently in different configurations, it was just too difficult to use them correctly; and step_forward() is simpler anyway. Fixes: ace71a19cec5 ("mm: introduce page_vma_mapped_walk()") Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov --- mm/page_vma_mapped.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index f6839f536645..6eb2f1863506 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -116,6 +116,13 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return pfn_is_match(pvmw->page, pfn); } +static void step_forward(struct page_vma_mapped_walk *pvmw, unsigned long size) +{ + pvmw->address = (pvmw->address + size) & ~(size - 1); + if (!pvmw->address) + pvmw->address = ULONG_MAX; +} + /** * page_vma_mapped_walk - check if @pvmw->page is mapped in @pvmw->vma at * @pvmw->address @@ -183,16 +190,22 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pte) goto next_pte; restart: - { + do { pgd = pgd_offset(mm, pvmw->address); - if (!pgd_present(*pgd)) - return false; + if (!pgd_present(*pgd)) { + step_forward(pvmw, PGDIR_SIZE); + continue; + } p4d = p4d_offset(pgd, pvmw->address); - if (!p4d_present(*p4d)) - return false; + if (!p4d_present(*p4d)) { + step_forward(pvmw, P4D_SIZE); + continue; + } pud = pud_offset(p4d, pvmw->address); - if (!pud_present(*pud)) - return false; + if (!pud_present(*pud)) { + step_forward(pvmw, PUD_SIZE); + continue; + } pvmw->pmd = pmd_offset(pud, pvmw->address); /* @@ -240,7 +253,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) spin_unlock(ptl); } - return false; + step_forward(pvmw, PMD_SIZE); + continue; } if (!map_pte(pvmw)) goto next_pte; @@ -270,7 +284,9 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) spin_lock(pvmw->ptl); } goto this_pte; - } + } while (pvmw->address < end); + + return false; } /** From patchwork Thu Jun 10 06:54:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12311983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AC64C47094 for ; Thu, 10 Jun 2021 06:54:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 06E5D613DE for ; Thu, 10 Jun 2021 06:54:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06E5D613DE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 918656B0036; Thu, 10 Jun 2021 02:54:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A13E6B006E; Thu, 10 Jun 2021 02:54:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F3906B0070; Thu, 10 Jun 2021 02:54:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 381796B0036 for ; Thu, 10 Jun 2021 02:54:50 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C89A7180AD81D for ; Thu, 10 Jun 2021 06:54:49 +0000 (UTC) X-FDA: 78236901498.28.1D56DFA Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf22.hostedemail.com (Postfix) with ESMTP id 78497C005A20 for ; Thu, 10 Jun 2021 06:54:44 +0000 (UTC) Received: by mail-qt1-f173.google.com with SMTP id a15so5917186qtx.13 for ; Wed, 09 Jun 2021 23:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=zujSa+nymJg/DkygOzBqfFRpvoPGGLvQoUzC9rD2EmM=; b=d+CHzx9kx2FW5+g+wRTEglrZHDD2va0NJeQznMTqavHkJKp8sIQAoV4Xl6VPkL/21a GQeN3iyeoaNEmvPVwRrrGCrG6C8d6PZbBDxVmEyxIgTXugm3GmBDemitSpPAF8c4I8iU rhlfqQRvxv244TvuzRX2C190uZyOlY5zGSVwvdz2Zs7F9UAZyU3xADhczznWRvYlpZRm ALXo36ZKcmd8MTCNDLR51p1ZO9TF30Cg1VH7dw7T7Z44WunIOIV9cIhr9+rsiNeG4/VS UtwMtJ8r4/tj/Vmp+gDCrtzjuM3wf6vtEeN27WC6LLMmQZXQPKpKPv3YKoKpS9NpptTM 0XdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=zujSa+nymJg/DkygOzBqfFRpvoPGGLvQoUzC9rD2EmM=; b=eU8WhSQpMj6MjTKmkyyS+ifp0Db0g2YVvIU5gmuIzv1rN8EPcgf52lmpNWiGoLRAun hpRbJaE1Lsps0G5ZIXYEZLeXQ3PshyVAB7gVbsPTH/g/MEdEwrlAP0iavEUXTTsvX1bT Vkattb3QLkcLDQ07EcuGHdS1VGMsHHPPUSHVZ2FsdoIPwCnztexhWJ4Pd3x+ZaZUul5P /3maqlgzJRQvgnqkzQfU61Vx9ZeAOCskigAfXhcdhzhV3hCAqpH4tVINgQ5LsE5eq1V7 N/5hk5iedDJZtuZL69l/VnXGxblkuaPYLpPprmNNYGa2Kp85OC3beIP4pttLUq0kmf+0 ir6Q== X-Gm-Message-State: AOAM531kaXrUtOZp6OXQqyTV3XvUxBgEXaC10SB1mzsvGXCj3KfQ5DEk sIYZXJ+lgQufbSBuTAhXz1ES2g== X-Google-Smtp-Source: ABdhPJyWmrIYtT3Hki5va0KGk1YyG5yE6G/KbTu0XqJoO/SiXciAA8VL5O7uhLNqek19ZMtnb91BBw== X-Received: by 2002:ac8:47c2:: with SMTP id d2mr3771672qtr.128.1623308088731; Wed, 09 Jun 2021 23:54:48 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id u3sm1611578qtg.16.2021.06.09.23.54.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Jun 2021 23:54:48 -0700 (PDT) Date: Wed, 9 Jun 2021 23:54:46 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Alistair Popple , Ralph Campbell , Zi Yan , Peter Xu , Will Deacon , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 11/11] mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk() In-Reply-To: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> Message-ID: <1bdf384c-8137-a149-2a1e-475a4791c3c@google.com> References: <589b358c-febc-c88e-d4c2-7834b37fa7bf@google.com> MIME-Version: 1.0 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=d+CHzx9k; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: to11toyqsuajjrg3jtb9km6gh14iu8nw X-Rspamd-Queue-Id: 78497C005A20 X-Rspamd-Server: rspam06 X-HE-Tag: 1623308084-197452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Aha! Shouldn't that quick scan over pte_none()s make sure that it holds ptlock in the PVMW_SYNC case? That too might have been responsible for BUGs or WARNs in split_huge_page_to_list() or its unmap_page(), though I've never seen any. Fixes: ace71a19cec5 ("mm: introduce page_vma_mapped_walk()") Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Tested-by: Wang Yugui --- mm/page_vma_mapped.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 6eb2f1863506..7ae4a016304b 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -277,6 +277,10 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) goto restart; } pvmw->pte++; + if ((pvmw->flags & PVMW_SYNC) && !pvmw->ptl) { + pvmw->ptl = pte_lockptr(mm, pvmw->pmd); + spin_lock(pvmw->ptl); + } } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) {