From patchwork Tue Jun 1 21:05:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292227 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B899C47080 for ; Tue, 1 Jun 2021 21:05:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DD1EF613AE for ; Tue, 1 Jun 2021 21:05:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DD1EF613AE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 53EA66B006C; Tue, 1 Jun 2021 17:05:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49EB36B006E; Tue, 1 Jun 2021 17:05:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F1446B0070; Tue, 1 Jun 2021 17:05:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id E782F6B006C for ; Tue, 1 Jun 2021 17:05:49 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 80B1D180AD817 for ; Tue, 1 Jun 2021 21:05:49 +0000 (UTC) X-FDA: 78206386818.12.F8D152E Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by imf03.hostedemail.com (Postfix) with ESMTP id AA70AC00CBFA for ; Tue, 1 Jun 2021 21:05:36 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id v19-20020a0568301413b0290304f00e3d88so595559otp.4 for ; Tue, 01 Jun 2021 14:05:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=aOIG7tu+/2aICcc9Ly2ISwpTg6tce60fLXoVvD1JTzM=; b=LVCZanx/eYTvSx8lbWpO+ZdK2aPRERFXGZTx1Mhgep8Q8QkzGUo7bbB3g+i5u8rpSp Soqb3fcuYDDDvt0nFd0wlpFXBxgnZe3ZIIZ7mgooDnnJ1XxzdUCk17V1ihl4yakJt5Mm 1B502MVUExrwyRSZ5YiNucIo3AkPrclnGMUGraAbZYC/olCSlLINFvP4uLCoGa5uogeU Ma3csqVApJ7WJY1t2WesG1pEfBnC89flXSRo4Ty2ljjJH9Jak59UnDl7KUieNY01cwYU 5NROwk8GWWXUJNYIeP5ruto/jmw+GN/3HS1HRrl79yZcILHFQXlNGh80ws3w+LBwSdOV Ffnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=aOIG7tu+/2aICcc9Ly2ISwpTg6tce60fLXoVvD1JTzM=; b=IzesCkh/gUXuq/yhQJO8fpStD1QiURzIy3MR3PG4o/Z5tRU0Gq+or5dy7xLUOv/jPP jmzDnxUFcr1dtzBWTaXId81GurBGXkHBj/vOvRYX5lRWh6VXwneUp+LkPx6QERUw+4vb n1s3ZR5dMMgBimURzzbD00Gak4Ik9EHoB+MxYo/C5IAV/9ecT15KU1eZ/I409HmXa+pu rWt7PFjI0mYoYoqtVDaXx8TtvLdPzHFJSmZlHVPn+f1Z6XRpYzQM53o37oz/52+4hGt8 VljHLCUAOukEu6/t2qgG1tFnCdCywlKqsYqwbtKYL+u+fXh0iC9sQRIgKoQ3YigdeC31 iBig== X-Gm-Message-State: AOAM533ZLEv+egcupUATvhg1b2H98KWD6aYw6HiwbG9k+HHAk2HnQ/xL EamEchSUrQ+vZAY/4/CchVciog== X-Google-Smtp-Source: ABdhPJzV5gldYPZB2iLgdrqcizeQk2B+APGM0bHru2ERnaVvQl2M5X3Gk3ejGz5izjqbmQcVBiRC7g== X-Received: by 2002:a05:6830:18dc:: with SMTP id v28mr8776572ote.326.1622581548311; Tue, 01 Jun 2021 14:05:48 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t14sm3641081ooh.39.2021.06.01.14.05.46 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:05:47 -0700 (PDT) Date: Tue, 1 Jun 2021 14:05:45 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/7] mm/thp: fix __split_huge_pmd_locked() on shmem migration entry In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b="LVCZanx/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.210.52 as permitted sender) smtp.mailfrom=hughd@google.com X-Stat-Signature: ctwmheonkpakbktu5zze4s5gz951duro X-Rspamd-Queue-Id: AA70AC00CBFA X-Rspamd-Server: rspam02 X-HE-Tag: 1622581536-735338 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Stressing huge tmpfs page migration racing hole punch often crashed on the VM_BUG_ON(!pmd_present) in pmdp_huge_clear_flush(), with DEBUG_VM=y kernel; or shortly afterwards, on a bad dereference in __split_huge_pmd_locked() when DEBUG_VM=n. They forgot to allow for pmd migration entries in the non-anonymous case. Full disclosure: those particular experiments were on a kernel with more relaxed mmap_lock and i_mmap_rwsem locking, and were not repeated on the vanilla kernel: it is conceivable that stricter locking happens to avoid those cases, or makes them less likely; but __split_huge_pmd_locked() already allowed for pmd migration entries when handling anonymous THPs, so this commit brings the shmem and file THP handling into line. Are there more places that need to be careful about pmd migration entries? None hit in practice, but several of those is_huge_zero_pmd() tests were done without checking pmd_present() first: I believe a pmd migration entry could end up satisfying that test. Ah, the inversion of swap offset, to protect against L1TF, makes that impossible on x86; but other arches need the pmd_present() check, and even x86 ought not to apply pmd_page() to a swap-like pmd. Fix those instances; __split_huge_pmd_locked() was not wrong to be checking with pmd_trans_huge() instead, but I think it's clearer to use pmd_present() in each instance. And while there: make it clearer to the eye that the !vma_is_anonymous() and is_huge_zero_pmd() blocks make early returns (and don't return void). Fixes: e71769ae5260 ("mm: enable thp migration for shmem thp") Signed-off-by: Hugh Dickins Cc: --- mm/huge_memory.c | 38 ++++++++++++++++++++++++-------------- mm/pgtable-generic.c | 5 ++--- 2 files changed, 26 insertions(+), 17 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 63ed6b25deaa..9fb7b47da87e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1587,9 +1587,6 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, goto out_unlocked; orig_pmd = *pmd; - if (is_huge_zero_pmd(orig_pmd)) - goto out; - if (unlikely(!pmd_present(orig_pmd))) { VM_BUG_ON(thp_migration_supported() && !is_pmd_migration_entry(orig_pmd)); @@ -1597,6 +1594,9 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, } page = pmd_page(orig_pmd); + if (is_huge_zero_page(page)) + goto out; + /* * If other processes are mapping this page, we couldn't discard * the page unless they all do MADV_FREE so let's skip the page. @@ -1676,7 +1676,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, spin_unlock(ptl); if (is_huge_zero_pmd(orig_pmd)) tlb_remove_page_size(tlb, pmd_page(orig_pmd), HPAGE_PMD_SIZE); - } else if (is_huge_zero_pmd(orig_pmd)) { + } else if (pmd_present(orig_pmd) && is_huge_zero_pmd(orig_pmd)) { zap_deposited_table(tlb->mm, pmd); spin_unlock(ptl); tlb_remove_page_size(tlb, pmd_page(orig_pmd), HPAGE_PMD_SIZE); @@ -2044,7 +2044,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, count_vm_event(THP_SPLIT_PMD); if (!vma_is_anonymous(vma)) { - _pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd); + old_pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd); /* * We are going to unmap this huge page. So * just go ahead and zap it @@ -2053,16 +2053,25 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, zap_deposited_table(mm, pmd); if (vma_is_special_huge(vma)) return; - page = pmd_page(_pmd); - if (!PageDirty(page) && pmd_dirty(_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, true); - put_page(page); + if (unlikely(is_pmd_migration_entry(old_pmd))) { + swp_entry_t entry; + + entry = pmd_to_swp_entry(old_pmd); + page = migration_entry_to_page(entry); + } else { + page = pmd_page(old_pmd); + if (!PageDirty(page) && pmd_dirty(old_pmd)) + set_page_dirty(page); + if (!PageReferenced(page) && pmd_young(old_pmd)) + SetPageReferenced(page); + page_remove_rmap(page, true); + put_page(page); + } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; - } else if (pmd_trans_huge(*pmd) && is_huge_zero_pmd(*pmd)) { + } + + if (pmd_present(*pmd) && is_huge_zero_pmd(*pmd)) { /* * FIXME: Do we want to invalidate secondary mmu by calling * mmu_notifier_invalidate_range() see comments below inside @@ -2072,7 +2081,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * small page also write protected so it does not seems useful * to invalidate secondary mmu at this time. */ - return __split_huge_zero_page_pmd(vma, haddr, pmd); + __split_huge_zero_page_pmd(vma, haddr, pmd); + return; } /* diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index c2210e1cdb51..4e640baf9794 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -135,9 +135,8 @@ pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address, { pmd_t pmd; VM_BUG_ON(address & ~HPAGE_PMD_MASK); - VM_BUG_ON(!pmd_present(*pmdp)); - /* Below assumes pmd_present() is true */ - VM_BUG_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); + VM_BUG_ON(pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) && + !pmd_devmap(*pmdp)); pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp); flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); return pmd; From patchwork Tue Jun 1 21:07:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292229 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 224A8C4708F for ; Tue, 1 Jun 2021 21:07:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A09A9613AE for ; Tue, 1 Jun 2021 21:07:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A09A9613AE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3B3E26B006C; Tue, 1 Jun 2021 17:07:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 364246B006E; Tue, 1 Jun 2021 17:07:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DD886B0070; Tue, 1 Jun 2021 17:07:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0199.hostedemail.com [216.40.44.199]) by kanga.kvack.org (Postfix) with ESMTP id DB14C6B006C for ; Tue, 1 Jun 2021 17:07:57 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 692CB181AEF07 for ; Tue, 1 Jun 2021 21:07:57 +0000 (UTC) X-FDA: 78206392194.33.9194C82 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf17.hostedemail.com (Postfix) with ESMTP id 3A29141AD7B2 for ; Tue, 1 Jun 2021 21:07:48 +0000 (UTC) Received: by mail-qv1-f53.google.com with SMTP id 5so242603qvk.0 for ; Tue, 01 Jun 2021 14:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=Zbc201h4vA/zTsuSBg7jG5PuehkrxrykYzZRAEADCP8=; b=fyn3XV5Gn8AMOt/w9jFZQcTUUD+FCmCO4tT29BnKyp4u/dx5EwwqvgdPLw2XdqMp8s eQvzzPzAxwskLgzsdeECw1/GhJjq5o65hZ4LnxfMZpSsOz5y8V6jzgu7Qd/7ztxLcS5Q KlzagkjhvEAMUb/bCJllVGq8UV/61Ldtc/bXQnOHEfHJkxGFXayF8OJdqzEPdj/ePIUZ oa2iHZrsNZxGpwGMBAN9JpUQhv/ab7neSsILqOhh1WZP856AihV1pApvpJZctHhbsmTf 5sNvu43VsavBmYEXH0nKkKaZoizLvz3paYHgGjrm9VM5//rbUt36/sVtBInZsNpK728z WGRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=Zbc201h4vA/zTsuSBg7jG5PuehkrxrykYzZRAEADCP8=; b=UbhBevYQ07HvJChIEiMveF5XrbL0fOd0iEzynJ/1LwhLCxWj8mR7PCoRdB4YF/W5/e HDom5iaVRKZShKBjcxFM93I0XIZWEitK6Ory2xurh+I+RuWvMVbkTLPZQP8Kjn5Jnpt8 sAeYfQBLQ7wJExe72N3BFHvkXSJKew/b70vpuH3ChvlRxAwiyWyUT/+1KnJxnDFEx+G+ NfM0Q/OdD0jQPJ5vATQELhS3mEz2I7WXA9bMKgk3Gvop/i9pPVP1H2eZ/8NMb0K1dOYi PhvvdrfTMLhSdhspWuvQX4wRafLRU0o0T6X2EMrqNVSsHgx2LCXxQsZX5D37o/GCcisI terA== X-Gm-Message-State: AOAM532ba2ryk1m0IGTT0IVi0rN/+8U+vULZD74+TS1tfclSYL+tG0KL CbmjrqXY7mxNQ7F16mxMQrPvjg== X-Google-Smtp-Source: ABdhPJwN4ufBQATeY8YBGM4IN8Yt39DV+3kywLCuUoK/7J52iUNL46bXGV7bnwtx+mw+CfH9flwLlw== X-Received: by 2002:ad4:5bef:: with SMTP id k15mr23795389qvc.22.1622581676150; Tue, 01 Jun 2021 14:07:56 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id w3sm300195qkb.13.2021.06.01.14.07.54 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:07:55 -0700 (PDT) Date: Tue, 1 Jun 2021 14:07:53 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/7] mm/thp: try_to_unmap() use TTU_SYNC for safe DEBUG_VM splitting In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=fyn3XV5G; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.53 as permitted sender) smtp.mailfrom=hughd@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3A29141AD7B2 X-Stat-Signature: wqi4h6sx11jtnn6yrurru66i5uwteud5 X-HE-Tag: 1622581668-981874 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Stressing huge tmpfs often crashed on unmap_page()'s VM_BUG_ON_PAGE (!unmap_success): with dump_page() showing mapcount:1, but then its raw struct page output showing _mapcount ffffffff i.e. mapcount 0. And even if that particular VM_BUG_ON_PAGE(!unmap_success) is removed, it is immediately followed by a VM_BUG_ON_PAGE(compound_mapcount(head)), and further down an IS_ENABLED(CONFIG_DEBUG_VM) total_mapcount BUG(): all indicative of some mapcount difficulty in development here perhaps. But the !CONFIG_DEBUG_VM path handles the failures correctly and silently. I believe the problem is that once a racing unmap has cleared pte or pmd, try_to_unmap_one() may skip taking the page table lock, and emerge from try_to_unmap() before the racing task has reached decrementing mapcount. Instead of abandoning the unsafe VM_BUG_ON_PAGE(), and the ones that follow, use PVMW_SYNC in try_to_unmap_one() in this case: adding TTU_SYNC to the options, and passing that from unmap_page() when CONFIG_DEBUG_VM=y. It could be passed in the non-debug case too, but that would sometimes add a little overhead, whereas it's rare for this race to result in failure. mm/memory-failure.c:hwpoison_user_mappings() should probably use the new TTU_SYNC option too, just in case this race coincides with its attempts to unmap a failing page (THP or not); but this commit does not add that. Fixes: fec89c109f3a ("thp: rewrite freeze_page()/unfreeze_page() with generic rmap walkers") Signed-off-by: Hugh Dickins Cc: --- include/linux/rmap.h | 3 ++- mm/huge_memory.c | 4 ++++ mm/page_vma_mapped.c | 8 ++++++++ mm/rmap.c | 17 ++++++++++++++++- 4 files changed, 30 insertions(+), 2 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index def5c62c93b3..891599a4cb8d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -97,7 +97,8 @@ enum ttu_flags { * do a final flush if necessary */ TTU_RMAP_LOCKED = 0x80, /* do not grab rmap lock: * caller holds it */ - TTU_SPLIT_FREEZE = 0x100, /* freeze pte under splitting thp */ + TTU_SPLIT_FREEZE = 0x100, /* freeze pte under splitting thp */ + TTU_SYNC = 0x200, /* avoid racy checks with PVMW_SYNC */ }; #ifdef CONFIG_MMU diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9fb7b47da87e..305f709a7aca 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2357,6 +2357,10 @@ static void unmap_page(struct page *page) if (PageAnon(page)) ttu_flags |= TTU_SPLIT_FREEZE; + /* Make sure that the BUGs will not bite */ + if (IS_ENABLED(CONFIG_DEBUG_VM)) + ttu_flags |= TTU_SYNC; + unmap_success = try_to_unmap(page, ttu_flags); VM_BUG_ON_PAGE(!unmap_success, page); } diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 2cf01d933f13..b45d22738b45 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -212,6 +212,14 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) pvmw->ptl = NULL; } } else if (!pmd_present(pmde)) { + /* + * If PVMW_SYNC, take and drop THP pmd lock so that we + * cannot return prematurely, while zap_huge_pmd() has + * cleared *pmd but not decremented compound_mapcount(). + */ + if ((pvmw->flags & PVMW_SYNC) && + PageTransCompound(pvmw->page)) + spin_unlock(pmd_lock(mm, pvmw->pmd)); return false; } if (!map_pte(pvmw)) diff --git a/mm/rmap.c b/mm/rmap.c index 693a610e181d..07811b4ae793 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1405,6 +1405,15 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, struct mmu_notifier_range range; enum ttu_flags flags = (enum ttu_flags)(long)arg; + /* + * When racing against e.g. zap_pte_range() on another cpu, + * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * try_to_unmap() may return false when it is about to become true, + * if page table locking is skipped: use TTU_SYNC to wait for that. + */ + if (flags & TTU_SYNC) + pvmw.flags = PVMW_SYNC; + /* munlock has nothing to gain from examining un-locked vmas */ if ((flags & TTU_MUNLOCK) && !(vma->vm_flags & VM_LOCKED)) return true; @@ -1777,7 +1786,13 @@ bool try_to_unmap(struct page *page, enum ttu_flags flags) else rmap_walk(page, &rwc); - return !page_mapcount(page) ? true : false; + /* + * When racing against e.g. zap_pte_range() on another cpu, + * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * try_to_unmap() may return false when it is about to become true, + * if page table locking is skipped: use TTU_SYNC to wait for that. + */ + return !page_mapcount(page); } /** From patchwork Tue Jun 1 21:09:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292231 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A084C4708F for ; Tue, 1 Jun 2021 21:09:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C973960FF2 for ; Tue, 1 Jun 2021 21:09:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C973960FF2 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3B2E76B006C; Tue, 1 Jun 2021 17:09:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 389796B006E; Tue, 1 Jun 2021 17:09:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22AB86B0070; Tue, 1 Jun 2021 17:09:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E33C86B006C for ; Tue, 1 Jun 2021 17:09:35 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7DDE0180AD817 for ; Tue, 1 Jun 2021 21:09:35 +0000 (UTC) X-FDA: 78206396310.40.5C853BD Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) by imf14.hostedemail.com (Postfix) with ESMTP id 0D3FDC00CBF8 for ; Tue, 1 Jun 2021 21:09:21 +0000 (UTC) Received: by mail-qv1-f43.google.com with SMTP id a7so190890qvf.11 for ; Tue, 01 Jun 2021 14:09:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=UGAE1tU3sFBmQCEFypWhJDt5nNuef83Vqmn7+7MSi28=; b=LbcBieFP9gBphC6OVjni5VvMVZJhNkObwpdoAEfTC7Pns2bV6zPFiHT1UlzCFLGv8x D+WAoY6ybKSbdtloPdU94dqEPH3le0eENVojrpLLYu/IpkpcaoiEh+Wh7RwN+gjNoG+q yFZGKoSrncKM+ErzZAiyB6gyX4pNmBWkhPm0f/lNEWPEpwKcQAfbnh/9AjT3EzpvEqLl 46hlFULs3XiMQ3gjAUH0mzzPksxZEqxFKahbUnhtnH2cAnnvpMGvnZ6JMTUOvuI7fhlh WskOmxJ6slJJs7Z8+ul4GEwHFf62Xsw9Na7QLmDQGF10T1C0TqGx5Ffr9HP9pMnq9u8g 2+Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=UGAE1tU3sFBmQCEFypWhJDt5nNuef83Vqmn7+7MSi28=; b=Xtz65oh1ncPRU4a/TIhOmqCoCVnPBoEfYoiNsJSyylBGEpWu7zC7XW7s/F7anKoFfF jjfaap+t8sE8z8IaAAn8RCbBdiWF/QAgHI9IEBvgWYsbRhoturPEAF74Lk8sj0ZixoTp pOgC9vPHNcrdT/AZb3/u8d3G7/U0gzNEy5tx4bCIjFZ2xWlxlLGReG05HW6rHKPbLvx0 P5V1xJmfc+psX6VA73WAf0MFbRYOmmhABhSFqLd4BUqVMQGrhE6fmNWnqHQKfIrm6cP3 YmQK61pKWw/kSGbWmYtYlmY6wjaJiE/ldbGkhZpzUdkcjgasVDdZy1ogQWGLaPo68XAd o4VA== X-Gm-Message-State: AOAM530GhwiJG1Jg6Dxomiz2LiD9CxJtXLauLe2WH6R7Zwk4l/MHGLoF p8HRnnYnPtjkuVRD/pflZkqIaA== X-Google-Smtp-Source: ABdhPJwYliUjTR1hcERALaFi+FEUt3Fn9oPnV76WPE/jV9HFuBAjQZMT93Nf4aqJZN1HQXf/TL77iA== X-Received: by 2002:ad4:5227:: with SMTP id r7mr24726914qvq.41.1622581774165; Tue, 01 Jun 2021 14:09:34 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id h12sm11811481qkj.52.2021.06.01.14.09.32 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:09:33 -0700 (PDT) Date: Tue, 1 Jun 2021 14:09:31 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/7] mm/thp: fix vma_address() if virtual address below file offset In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 X-Rspamd-Queue-Id: 0D3FDC00CBF8 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=LbcBieFP; spf=pass (imf14.hostedemail.com: domain of hughd@google.com designates 209.85.219.43 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam04 X-Stat-Signature: yb7co89cnuhhjdb7fwy4thj6qf8c9m41 X-HE-Tag: 1622581761-445460 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Running certain tests with a DEBUG_VM kernel would crash within hours, on the total_mapcount BUG() in split_huge_page_to_list(), while trying to free up some memory by punching a hole in a shmem huge page: split's try_to_unmap() was unable to find all the mappings of the page (which, on a !DEBUG_VM kernel, would then keep the huge page pinned in memory). When that BUG() was changed to a WARN(), it would later crash on the VM_BUG_ON_VMA(end < vma->vm_start || start >= vma->vm_end, vma) in mm/internal.h:vma_address(), used by rmap_walk_file() for try_to_unmap(). vma_address() is usually correct, but there's a wraparound case when the vm_start address is unusually low, but vm_pgoff not so low: vma_address() chooses max(start, vma->vm_start), but that decides on the wrong address, because start has become almost ULONG_MAX. Rewrite vma_address() to be more careful about vm_pgoff; move the VM_BUG_ON_VMA() out of it, returning -EFAULT for errors, so that it can be safely used from page_mapped_in_vma() and page_address_in_vma() too. Add vma_address_end() to apply similar care to end address calculation, in page_vma_mapped_walk() and page_mkclean_one() and try_to_unmap_one(); though it raises a question of whether callers would do better to supply pvmw->end to page_vma_mapped_walk() - I chose not, for a smaller patch. An irritation is that their apparent generality breaks down on KSM pages, which cannot be located by the page->index that page_to_pgoff() uses: as 4b0ece6fa016 ("mm: migrate: fix remove_migration_pte() for ksm pages") once discovered. I dithered over the best thing to do about that, and have ended up with a VM_BUG_ON_PAGE(PageKsm) in both vma_address() and vma_address_end(); though the only place in danger of using it on them was try_to_unmap_one(). Sidenote: vma_address() and vma_address_end() now use compound_order() on a head page, instead of thp_size(): to make the right calculation on a hugetlbfs page, whether or not THPs are configured. try_to_unmap() is used on hugetlbfs pages, but perhaps the wrong calculation never mattered. Fixes: a8fa41ad2f6f ("mm, rmap: check all VMAs that PTE-mapped THP can be part of") Signed-off-by: Hugh Dickins Cc: --- mm/internal.h | 56 +++++++++++++++++++++++++++++++++----------- mm/page_vma_mapped.c | 16 +++++-------- mm/rmap.c | 16 ++++++------- 3 files changed, 56 insertions(+), 32 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 2f1182948aa6..00d2dfa534cf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -384,27 +384,55 @@ static inline void mlock_migrate_page(struct page *newpage, struct page *page) extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); /* - * At what user virtual address is page expected in @vma? + * At what user virtual address is page expected in vma? + * Returns -EFAULT if all of the page is outside the range of vma. + * If page is a compound head, the entire compound page is considered. */ static inline unsigned long -__vma_address(struct page *page, struct vm_area_struct *vma) +vma_address(struct page *page, struct vm_area_struct *vma) { - pgoff_t pgoff = page_to_pgoff(page); - return vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + pgoff_t pgoff; + unsigned long address; + + VM_BUG_ON_PAGE(PageKsm(page), page); /* KSM page->index unusable */ + pgoff = page_to_pgoff(page); + if (pgoff >= vma->vm_pgoff) { + address = vma->vm_start + + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + /* Check for address beyond vma (or wrapped through 0?) */ + if (address < vma->vm_start || address >= vma->vm_end) + address = -EFAULT; + } else if (PageHead(page) && + pgoff + (1UL << compound_order(page)) > vma->vm_pgoff) { + address = vma->vm_start; + } else { + address = -EFAULT; + } + return address; } +/* + * Then at what user virtual address will none of the page be found in vma? + * Assumes that vma_address() already returned a good starting address. + * If page is a compound head, the entire compound page is considered. + */ static inline unsigned long -vma_address(struct page *page, struct vm_area_struct *vma) +vma_address_end(struct page *page, struct vm_area_struct *vma) { - unsigned long start, end; - - start = __vma_address(page, vma); - end = start + thp_size(page) - PAGE_SIZE; - - /* page should be within @vma mapping range */ - VM_BUG_ON_VMA(end < vma->vm_start || start >= vma->vm_end, vma); - - return max(start, vma->vm_start); + pgoff_t pgoff; + unsigned long address; + + VM_BUG_ON_PAGE(PageKsm(page), page); /* KSM page->index unusable */ + pgoff = page_to_pgoff(page); + if (PageHead(page)) + pgoff += 1UL << compound_order(page); + else + pgoff++; + address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + /* Check for address beyond vma (or wrapped through 0?) */ + if (address < vma->vm_start || address > vma->vm_end) + address = vma->vm_end; + return address; } static inline struct file *maybe_unlock_mmap_for_io(struct vm_fault *vmf, diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index b45d22738b45..92f0ecd50524 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -225,18 +225,18 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!map_pte(pvmw)) goto next_pte; while (1) { + unsigned long end; + if (check_pte(pvmw)) return true; next_pte: /* Seek to next pte only makes sense for THP */ if (!PageTransHuge(pvmw->page) || PageHuge(pvmw->page)) return not_found(pvmw); + end = vma_address_end(pvmw->page, pvmw->vma); do { pvmw->address += PAGE_SIZE; - if (pvmw->address >= pvmw->vma->vm_end || - pvmw->address >= - __vma_address(pvmw->page, pvmw->vma) + - thp_size(pvmw->page)) + if (pvmw->address >= end) return not_found(pvmw); /* Did we cross page table boundary? */ if (pvmw->address % PMD_SIZE == 0) { @@ -274,14 +274,10 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma) .vma = vma, .flags = PVMW_SYNC, }; - unsigned long start, end; - - start = __vma_address(page, vma); - end = start + thp_size(page) - PAGE_SIZE; - if (unlikely(end < vma->vm_start || start >= vma->vm_end)) + pvmw.address = vma_address(page, vma); + if (pvmw.address == -EFAULT) return 0; - pvmw.address = max(start, vma->vm_start); if (!page_vma_mapped_walk(&pvmw)) return 0; page_vma_mapped_walk_done(&pvmw); diff --git a/mm/rmap.c b/mm/rmap.c index 07811b4ae793..144de54efc1c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -707,7 +707,6 @@ static bool should_defer_flush(struct mm_struct *mm, enum ttu_flags flags) */ unsigned long page_address_in_vma(struct page *page, struct vm_area_struct *vma) { - unsigned long address; if (PageAnon(page)) { struct anon_vma *page__anon_vma = page_anon_vma(page); /* @@ -722,10 +721,8 @@ unsigned long page_address_in_vma(struct page *page, struct vm_area_struct *vma) return -EFAULT; } else return -EFAULT; - address = __vma_address(page, vma); - if (unlikely(address < vma->vm_start || address >= vma->vm_end)) - return -EFAULT; - return address; + + return vma_address(page, vma); } pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address) @@ -919,7 +916,7 @@ static bool page_mkclean_one(struct page *page, struct vm_area_struct *vma, */ mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE, 0, vma, vma->vm_mm, address, - min(vma->vm_end, address + page_size(page))); + vma_address_end(page, vma)); mmu_notifier_invalidate_range_start(&range); while (page_vma_mapped_walk(&pvmw)) { @@ -1435,9 +1432,10 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma, * Note that the page can not be free in this function as call of * try_to_unmap() must hold a reference on the page. */ + range.end = PageKsm(page) ? + address + PAGE_SIZE : vma_address_end(page, vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm, - address, - min(vma->vm_end, address + page_size(page))); + address, range.end); if (PageHuge(page)) { /* * If sharing is possible, start and end will be adjusted @@ -1889,6 +1887,7 @@ static void rmap_walk_anon(struct page *page, struct rmap_walk_control *rwc, struct vm_area_struct *vma = avc->vma; unsigned long address = vma_address(page, vma); + VM_BUG_ON_VMA(address == -EFAULT, vma); cond_resched(); if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) @@ -1943,6 +1942,7 @@ static void rmap_walk_file(struct page *page, struct rmap_walk_control *rwc, pgoff_start, pgoff_end) { unsigned long address = vma_address(page, vma); + VM_BUG_ON_VMA(address == -EFAULT, vma); cond_resched(); if (rwc->invalid_vma && rwc->invalid_vma(vma, rwc->arg)) From patchwork Tue Jun 1 21:11:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C913EC47080 for ; Tue, 1 Jun 2021 21:11:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 68F48613AE for ; Tue, 1 Jun 2021 21:11:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 68F48613AE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F226A6B006C; Tue, 1 Jun 2021 17:11:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED1BE6B006E; Tue, 1 Jun 2021 17:11:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4B416B0070; Tue, 1 Jun 2021 17:11:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id A27FB6B006C for ; Tue, 1 Jun 2021 17:11:26 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 47E7E180AD817 for ; Tue, 1 Jun 2021 21:11:26 +0000 (UTC) X-FDA: 78206400972.16.FE29867 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf29.hostedemail.com (Postfix) with ESMTP id 14636547 for ; Tue, 1 Jun 2021 21:11:09 +0000 (UTC) Received: by mail-ot1-f50.google.com with SMTP id x41-20020a05683040a9b02903b37841177eso581519ott.9 for ; Tue, 01 Jun 2021 14:11:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=mGezqLCwqfVvLkMCE7dsqP76GlZ1rrIMTNorbY8nnRE=; b=XiFqJgKbQf+NFm9F2qeWGDqMQtIusE6tfI/d7uCReiNCeI8t1UgzW0tp8FS3tjI6Qn MM058Q8yao+WMfmJTIHUWTNzTCfxl0dIXFzHmnGtdNyydEMDn2/e3yjwWs/UzgnrXiML QqN3GCoZNMystw71jGSHVCRio97HkV3DOsCbiacwt0uIOU20iUscldneY3FmvXADytiw qcm939hv9ywb7M2e+4AI/6XxQYrUoTPfjXtvMcRDBA3OWOrUJYdf8NTD8fr9lVL5CNAr lijoUGLV2r+wmMfg5xsYNOja0Wpem4iA836KRAUuaYwYszImmI/uOurTB+d/d9VaEbPt V9GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=mGezqLCwqfVvLkMCE7dsqP76GlZ1rrIMTNorbY8nnRE=; b=bJL7iZsHwkpNK1bN9XLo0qSe9qwwKWoCGUhldBY0N2jQExOOxe+VCPySgHxDeQcvpQ y7AJTpUAOnITjtNHtvQvM2dV4Xcj4L/WdiJYxwZ+IWtMFSTgmvXY1C1xvaVZLlIMc/1P +/6pU/SUSMFnZz3/sa5LF0vcv9P8hcjZikwIEjo9+rteQECzKLm2konVJMHMchZ/uZUO 4p1SLhXIcOZ/b8p/mI78qvoZc3FHz5kCo8RP1j7fFiGYg8lNCTuK5IkcwLasbQWa5pe0 3jd4cB5tsXpdHtLGO0lcMwUTqNa9baGQBwU6xdhk86ZduxwiWEd86wyEEowUdl/FelTx bNsw== X-Gm-Message-State: AOAM530anQe3GmJUPa3rg+xNzmwz4BJEjowr5xAitcqUHctyoLhEletB ls4Kp3ENicRlN3O2p7IdoB5GrQ== X-Google-Smtp-Source: ABdhPJwbmIlsdvD15zqtJYc7eIWJ6uhy2nPvhdlXW5j9Ohe3labw6u0GYlOiclpWFjO1cjJLZLKpjg== X-Received: by 2002:a9d:684e:: with SMTP id c14mr23250577oto.295.1622581885174; Tue, 01 Jun 2021 14:11:25 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t14sm3644391ooh.39.2021.06.01.14.11.23 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:11:24 -0700 (PDT) Date: Tue, 1 Jun 2021 14:11:22 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/7] mm/thp: fix page_address_in_vma() on file THP tails In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=XiFqJgKb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of hughd@google.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=hughd@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 14636547 X-Stat-Signature: qcwpamwypx4myjk76nqbt3ab9xx1a9he X-HE-Tag: 1622581869-946763 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jue Wang Anon THP tails were already supported, but memory-failure may need to use page_address_in_vma() on file THP tails, which its page->mapping check did not permit: fix it. hughd adds: no current usage is known to hit the issue, but this does fix a subtle trap in a general helper: best fixed in stable sooner than later. Fixes: 800d8c63b2e9 ("shmem: add huge pages support") Signed-off-by: Jue Wang Signed-off-by: Hugh Dickins Cc: Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Yang Shi Acked-by: Kirill A. Shutemov --- mm/rmap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 144de54efc1c..e05c300048e6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -716,11 +716,11 @@ unsigned long page_address_in_vma(struct page *page, struct vm_area_struct *vma) if (!vma->anon_vma || !page__anon_vma || vma->anon_vma->root != page__anon_vma->root) return -EFAULT; - } else if (page->mapping) { - if (!vma->vm_file || vma->vm_file->f_mapping != page->mapping) - return -EFAULT; - } else + } else if (!vma->vm_file) { + return -EFAULT; + } else if (vma->vm_file->f_mapping != compound_head(page)->mapping) { return -EFAULT; + } return vma_address(page, vma); } From patchwork Tue Jun 1 21:13:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28E26C47080 for ; Tue, 1 Jun 2021 21:13:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C4AF1613B1 for ; Tue, 1 Jun 2021 21:13:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C4AF1613B1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5341B6B006C; Tue, 1 Jun 2021 17:13:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4BD406B006E; Tue, 1 Jun 2021 17:13:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E8D16B0070; Tue, 1 Jun 2021 17:13:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0210.hostedemail.com [216.40.44.210]) by kanga.kvack.org (Postfix) with ESMTP id EB87B6B006C for ; Tue, 1 Jun 2021 17:13:25 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 82951824999B for ; Tue, 1 Jun 2021 21:13:25 +0000 (UTC) X-FDA: 78206405970.11.6F3D7DC Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) by imf20.hostedemail.com (Postfix) with ESMTP id C1B34540 for ; Tue, 1 Jun 2021 21:13:08 +0000 (UTC) Received: by mail-ot1-f46.google.com with SMTP id c31-20020a056830349fb02903a5bfa6138bso602433otu.7 for ; Tue, 01 Jun 2021 14:13:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=u6fQ87Vl+hYyFQ4f/ypx0zbKSuAdvJqajnMzSb1SoWM=; b=BrfQ++mJOezRCujIJDyYnB08NoNcMR02IYLqSZ48uOKDsH41KdlFgVRf+nuSaH+qYp NuQowDr3QOY/G/lJzSal1kH+CZQy/xDisDEFHdio3OMWqW0pF910UcULl7PM4QN+KBeL UXxd9JlXTXDGPoHigLcJhPkaYISQNfgpg1WTBwf+5a57DyC2fDrfBN9Nf9hYDBhVuYvB sdrY1FIk0zfEGe8bfMClfqV39UCkOlZKCErT5kCyqChdiIjIw8b11e3U8AkgjS5GS6Wo Qv3GxCaOQkR6yGCKckwyNBi/k5PSbNDwg47sQG5CEfdUaNHeybnvYYl03zL53cEiQRBY w4fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=u6fQ87Vl+hYyFQ4f/ypx0zbKSuAdvJqajnMzSb1SoWM=; b=PJdRmG56AJbUVHtBOY5b7GjzMgxm9DSUjh6LExnG7//7Tj5+vAlax3DeRcUgB/+ltZ qM4E8cgxJA60LDI2JhrabVwX2V0Bp2M+WMxJYWF/Av8HvFfczMGTTPg3i4BpVVGvTUIK XM9E9VRg6KUAqfy9YcYUMpsH+bL8838VZL5FdAFgID2v3fg9E853eKA4FTpCLBkI9yPX T6rgsSD1qTd97Utjrb4HQDkZlxu6tBpehPHc1n0T1UcGJuOWbnbdAtNqzwpXHOU5oSgW XiCPf5Y+6vAveXePq1Os3BWAa7UJLYP0tzab1pL8bQjpxA2Tt/5Es3/7doOEaAcZlWzc mR9g== X-Gm-Message-State: AOAM530eC5jdFRANZWA8CZOW4cxD4vYRb3fVTY9dYEj/I8ryEl/rk5aO UO+Yw4JPahW6wTwT5q4n21SodQ== X-Google-Smtp-Source: ABdhPJylvVdYfMa97TOVKNdR2ndHraP/HRdjKm0hTTAoDzVEhhzQ1XiSYlsrnO29qaxW99S+pc0WnQ== X-Received: by 2002:a05:6830:1687:: with SMTP id k7mr24262603otr.220.1622582004410; Tue, 01 Jun 2021 14:13:24 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m189sm3602732oif.45.2021.06.01.14.13.22 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:13:24 -0700 (PDT) Date: Tue, 1 Jun 2021 14:13:21 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 5/7] mm/thp: fix page_vma_mapped_walk() if huge page mapped by ptes In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=BrfQ++mJ; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.210.46 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C1B34540 X-Stat-Signature: cyrx99r7sefdrizgzkpac5a8sii8fsn4 X-HE-Tag: 1622581988-262254 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Running certain tests with a DEBUG_VM kernel would crash within hours, on the total_mapcount BUG() in split_huge_page_to_list(), while trying to free up some memory by punching a hole in a shmem huge page: split's try_to_unmap() was unable to find all the mappings of the page (which, on a !DEBUG_VM kernel, would then keep the huge page pinned in memory). Crash dumps showed two tail pages of a shmem huge page remained mapped by pte: ptes in a non-huge-aligned vma of a gVisor process, at the end of a long unmapped range; and no page table had yet been allocated for the head of the huge page to be mapped into. Although designed to handle these odd misaligned huge-page-mapped-by-pte cases, page_vma_mapped_walk() falls short by returning false prematurely when !pmd_present or !pud_present or !p4d_present or !pgd_present: there are cases when a huge page may span the boundary, with ptes present in the next. Restructure page_vma_mapped_walk() as a loop to continue in these cases, while keeping its layout much as before. Add a step_forward() helper to advance pvmw->address across those boundaries: originally I tried to use mm's standard p?d_addr_end() macros, but hit the same crash 512 times less often: because of the way redundant levels are folded together, but folded differently in different configurations, it was just too difficult to use them correctly; and step_forward() is simpler anyway. Merged various other minor fixes and cleanups into page_vma_mapped_walk() as I worked on it: which I find much easier to enumerate here than to prise apart into separate commits. Handle all of the hugetlbfs PageHuge() case once at the start, so we don't need to worry about it again further down. Sometimes local copy of pvmw->page was used, sometimes pvmw->page: just use pvmw->page throughout (and continue to use pvmw->address throughout, though we could take a local copy instead). Use pmd_read_atomic() with barrier() instead of READ_ONCE() for pmde: some architectures (e.g. i386 with PAE) have a multi-word pmd entry, for which READ_ONCE() is not good enough. Re-evaluate pmde after taking lock, then use it in subsequent tests, instead of repeatedly dereferencing pvmw->pmd pointer. Rearrange the !pmd_present block to follow the same "return not_found, return not_found, return true" pattern as the block above it (note: returning not_found there is never premature, since the existence or prior existence of a huge pmd guarantees good alignment). Adjust page table boundary test in case address was not page-aligned. Reset pvmw->pte to NULL after unmapping that page table. Respect the 80-column line limit. Fixes: ace71a19cec5 ("mm: introduce page_vma_mapped_walk()") Signed-off-by: Hugh Dickins Cc: --- mm/page_vma_mapped.c | 159 ++++++++++++++++++++++++++----------------- 1 file changed, 95 insertions(+), 64 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 92f0ecd50524..f38ea5aa4acf 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -116,6 +116,13 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) return pfn_is_match(pvmw->page, pfn); } +static void step_forward(struct page_vma_mapped_walk *pvmw, unsigned long size) +{ + pvmw->address = (pvmw->address + size) & ~(size - 1); + if (!pvmw->address) + pvmw->address = ULONG_MAX; +} + /** * page_vma_mapped_walk - check if @pvmw->page is mapped in @pvmw->vma at * @pvmw->address @@ -143,7 +150,7 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw) bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) { struct mm_struct *mm = pvmw->vma->vm_mm; - struct page *page = pvmw->page; + unsigned long end; pgd_t *pgd; p4d_t *p4d; pud_t *pud; @@ -153,109 +160,133 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->pmd && !pvmw->pte) return not_found(pvmw); - if (pvmw->pte) - goto next_pte; - if (unlikely(PageHuge(pvmw->page))) { + /* The only possible mapping was handled on last iteration */ + if (pvmw->pte) + return not_found(pvmw); + /* when pud is not present, pte will be NULL */ - pvmw->pte = huge_pte_offset(mm, pvmw->address, page_size(page)); + pvmw->pte = huge_pte_offset(mm, pvmw->address, + page_size(pvmw->page)); if (!pvmw->pte) return false; - pvmw->ptl = huge_pte_lockptr(page_hstate(page), mm, pvmw->pte); + pvmw->ptl = huge_pte_lockptr(page_hstate(pvmw->page), + mm, pvmw->pte); spin_lock(pvmw->ptl); if (!check_pte(pvmw)) return not_found(pvmw); return true; } -restart: - pgd = pgd_offset(mm, pvmw->address); - if (!pgd_present(*pgd)) - return false; - p4d = p4d_offset(pgd, pvmw->address); - if (!p4d_present(*p4d)) - return false; - pud = pud_offset(p4d, pvmw->address); - if (!pud_present(*pud)) - return false; - pvmw->pmd = pmd_offset(pud, pvmw->address); + /* - * Make sure the pmd value isn't cached in a register by the - * compiler and used as a stale value after we've observed a - * subsequent update. + * Seek to next pte only makes sense for THP. + * But more important than that optimization, is to filter out + * any PageKsm page: whose page->index misleads vma_address() + * and vma_address_end() to disaster. */ - pmde = READ_ONCE(*pvmw->pmd); - if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde)) { - pvmw->ptl = pmd_lock(mm, pvmw->pmd); - if (likely(pmd_trans_huge(*pvmw->pmd))) { - if (pvmw->flags & PVMW_MIGRATION) - return not_found(pvmw); - if (pmd_page(*pvmw->pmd) != page) - return not_found(pvmw); - return true; - } else if (!pmd_present(*pvmw->pmd)) { - if (thp_migration_supported()) { - if (!(pvmw->flags & PVMW_MIGRATION)) + end = PageTransCompound(pvmw->page) ? + vma_address_end(pvmw->page, pvmw->vma) : + pvmw->address + PAGE_SIZE; + if (pvmw->pte) + goto next_pte; +restart: + do { + pgd = pgd_offset(mm, pvmw->address); + if (!pgd_present(*pgd)) { + step_forward(pvmw, PGDIR_SIZE); + continue; + } + p4d = p4d_offset(pgd, pvmw->address); + if (!p4d_present(*p4d)) { + step_forward(pvmw, P4D_SIZE); + continue; + } + pud = pud_offset(p4d, pvmw->address); + if (!pud_present(*pud)) { + step_forward(pvmw, PUD_SIZE); + continue; + } + + pvmw->pmd = pmd_offset(pud, pvmw->address); + /* + * Make sure the pmd value isn't cached in a register by the + * compiler and used as a stale value after we've observed a + * subsequent update. + */ + pmde = pmd_read_atomic(pvmw->pmd); + barrier(); + + if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde)) { + pvmw->ptl = pmd_lock(mm, pvmw->pmd); + pmde = *pvmw->pmd; + if (likely(pmd_trans_huge(pmde))) { + if (pvmw->flags & PVMW_MIGRATION) + return not_found(pvmw); + if (pmd_page(pmde) != pvmw->page) return not_found(pvmw); - if (is_migration_entry(pmd_to_swp_entry(*pvmw->pmd))) { - swp_entry_t entry = pmd_to_swp_entry(*pvmw->pmd); + return true; + } + if (!pmd_present(pmde)) { + swp_entry_t swap; - if (migration_entry_to_page(entry) != page) - return not_found(pvmw); - return true; - } + if (!thp_migration_supported() || + !(pvmw->flags & PVMW_MIGRATION)) + return not_found(pvmw); + swap = pmd_to_swp_entry(pmde); + if (!is_migration_entry(swap) || + migration_entry_to_page(swap) != pvmw->page) + return not_found(pvmw); + return true; } - return not_found(pvmw); - } else { /* THP pmd was split under us: handle on pte level */ spin_unlock(pvmw->ptl); pvmw->ptl = NULL; + } else if (!pmd_present(pmde)) { + /* + * If PVMW_SYNC, take and drop THP pmd lock so that we + * cannot return prematurely, while zap_huge_pmd() has + * cleared *pmd but not decremented compound_mapcount(). + */ + if ((pvmw->flags & PVMW_SYNC) && + PageTransCompound(pvmw->page)) + spin_unlock(pmd_lock(mm, pvmw->pmd)); + step_forward(pvmw, PMD_SIZE); + continue; } - } else if (!pmd_present(pmde)) { - /* - * If PVMW_SYNC, take and drop THP pmd lock so that we - * cannot return prematurely, while zap_huge_pmd() has - * cleared *pmd but not decremented compound_mapcount(). - */ - if ((pvmw->flags & PVMW_SYNC) && - PageTransCompound(pvmw->page)) - spin_unlock(pmd_lock(mm, pvmw->pmd)); - return false; - } - if (!map_pte(pvmw)) - goto next_pte; - while (1) { - unsigned long end; + if (!map_pte(pvmw)) + goto next_pte; +this_pte: if (check_pte(pvmw)) return true; next_pte: - /* Seek to next pte only makes sense for THP */ - if (!PageTransHuge(pvmw->page) || PageHuge(pvmw->page)) - return not_found(pvmw); - end = vma_address_end(pvmw->page, pvmw->vma); do { pvmw->address += PAGE_SIZE; if (pvmw->address >= end) return not_found(pvmw); + /* Did we cross page table boundary? */ - if (pvmw->address % PMD_SIZE == 0) { - pte_unmap(pvmw->pte); + if ((pvmw->address & (PMD_SIZE - PAGE_SIZE)) == 0) { if (pvmw->ptl) { spin_unlock(pvmw->ptl); pvmw->ptl = NULL; } + pte_unmap(pvmw->pte); + pvmw->pte = NULL; goto restart; - } else { - pvmw->pte++; } + pvmw->pte++; } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { pvmw->ptl = pte_lockptr(mm, pvmw->pmd); spin_lock(pvmw->ptl); } - } + goto this_pte; + } while (pvmw->address < end); + + return false; } /** From patchwork Tue Jun 1 21:15:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 104C7C47080 for ; Tue, 1 Jun 2021 21:15:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ADB41613B4 for ; Tue, 1 Jun 2021 21:15:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ADB41613B4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4720E6B006C; Tue, 1 Jun 2021 17:15:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 445FB6B006E; Tue, 1 Jun 2021 17:15:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BFB56B0070; Tue, 1 Jun 2021 17:15:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0242.hostedemail.com [216.40.44.242]) by kanga.kvack.org (Postfix) with ESMTP id EBA8F6B006C for ; Tue, 1 Jun 2021 17:15:39 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 76984181AEF15 for ; Tue, 1 Jun 2021 21:15:39 +0000 (UTC) X-FDA: 78206411598.01.99EB321 Received: from mail-ot1-f47.google.com (mail-ot1-f47.google.com [209.85.210.47]) by imf01.hostedemail.com (Postfix) with ESMTP id 6A7C85003902 for ; Tue, 1 Jun 2021 21:15:26 +0000 (UTC) Received: by mail-ot1-f47.google.com with SMTP id 36-20020a9d0ba70000b02902e0a0a8fe36so598721oth.8 for ; Tue, 01 Jun 2021 14:15:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=KS1YPkHQ/TuwZUBOUUc09sdHAPmJA1pFeqFuIXJ9Fug=; b=SPUSIU7ZhWh/wxRFrVWYLpgBpW0WKa1zhqylauHhOgLEbwtZ1/R34mGKihuY5ObQy0 ntMCde9IMcatgI74yPc/3DX+zrrtkm3EkPUidLn7Au5a1JqUh+h6dal/kOh5yeBkrH3F fkPRRnxngbJ5nxn6lYfTm8DjiZy9RHv35yIfe11DK+Y/PkNcK7kVSYl9yMadS4pMq4nJ uJpLTT5NtkVc3gLwO8qrlnM2CssKl8zWUogkNb5SDiOotGeAt63wJVqhY7RmD3OEcPzA mULS1pE20kS2USUm0iR2/8+8PhYuDuuQWYl5laCYxzAAElcOIoxjH4F/PyoKFUvLANS1 Rq0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=KS1YPkHQ/TuwZUBOUUc09sdHAPmJA1pFeqFuIXJ9Fug=; b=VRxM6RgohVyxuYW29FWCPCjDG1BbddwD+wqvShdzNQjVMi6ymuyBzKVucOP72GWg7I PcRMAaheSXJPMUoAySHYqsbnGgpsh1q0yjZwHlpEHcaPIItieGtkEOs75Z0h0WEER9tu LGtNE9lCoSsW5Erw89y8AAghinpNXvD//mUxCEAHG1QAylw9dgZ1DPyHPzhMPvpvIWVl 73fNH36/9LzUURkzjmQ/zf3YGLGAWKbzPKrRZkDjA4nMhbd4eJAkuEcXgjUHXslfM01Z EkmNb7p+I22utDB1YMKEMDpV0hN+ye0FtJ7vgjVP4iLVHYbmtn8bkj4InJ/Yj8RVy9QN EeCQ== X-Gm-Message-State: AOAM532mw0IU3Ec3AfeJ7GAgydyWwUA+EEsbVwI0tUjrQnLw9R3iu5+T iQeNTXZusEI+dfoUkqZkIlpD7A== X-Google-Smtp-Source: ABdhPJzeTZoTjflH1rTyUR5kL5ZS8yXRcuGGOOFlMqZOgkdaR6M9Q6yc+0Vrjq8MXS/Nv/yqPVtw/w== X-Received: by 2002:a05:6830:1e15:: with SMTP id s21mr22610019otr.144.1622582138213; Tue, 01 Jun 2021 14:15:38 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id m1sm4056655otq.12.2021.06.01.14.15.36 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:15:37 -0700 (PDT) Date: Tue, 1 Jun 2021 14:15:35 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 6/7] mm/thp: unmap_mapping_page() to fix THP truncate_cleanup_page() In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=SPUSIU7Z; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.210.47 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6A7C85003902 X-Stat-Signature: 36ptem5n46skq37dcmxsw4sr56tutbg5 X-HE-Tag: 1622582126-408785 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race between THP unmapping and truncation, when truncate sees pmd_none() and skips the entry, after munmap's zap_huge_pmd() cleared it, but before its page_remove_rmap() gets to decrement compound_mapcount: generating false "BUG: Bad page cache" reports that the page is still mapped when deleted. This commit fixes that, but not in the way I hoped. The first attempt used try_to_unmap(page, TTU_SYNC|TTU_IGNORE_MLOCK) instead of unmap_mapping_range() in truncate_cleanup_page(): it has often been an annoyance that we usually call unmap_mapping_range() with no pages locked, but there apply it to a single locked page. try_to_unmap() looks more suitable for a single locked page. However, try_to_unmap_one() contains a VM_BUG_ON_PAGE(!pvmw.pte,page): it is used to insert THP migration entries, but not used to unmap THPs. Copy zap_huge_pmd() and add THP handling now? Perhaps, but their TLB needs are different, I'm too ignorant of the DAX cases, and couldn't decide how far to go for anon+swap. Set that aside. The second attempt took a different tack: make no change in truncate.c, but modify zap_huge_pmd() to insert an invalidated huge pmd instead of clearing it initially, then pmd_clear() between page_remove_rmap() and unlocking at the end. Nice. But powerpc blows that approach out of the water, with its serialize_against_pte_lookup(), and interesting pgtable usage. It would need serious help to get working on powerpc (with a minor optimization issue on s390 too). Set that aside. Just add an "if (page_mapped(page)) synchronize_rcu();" or other such delay, after unmapping in truncate_cleanup_page()? Perhaps, but though that's likely to reduce or eliminate the number of incidents, it would give less assurance of whether we had identified the problem correctly. This successful iteration introduces "unmap_mapping_page(page)" instead of try_to_unmap(), and goes the usual unmap_mapping_range_tree() route, with an addition to details. Then zap_pmd_range() watches for this case, and does spin_unlock(pmd_lock) if so - just like page_vma_mapped_walk() now does in the PVMW_SYNC case. Not pretty, but safe. Note that unmap_mapping_page() is doing a VM_BUG_ON(!PageLocked) to assert its interface; but currently that's only used to make sure that page->mapping is stable, and zap_pmd_range() doesn't care if the page is locked or not. Along these lines, in invalidate_inode_pages2_range() move the initial unmap_mapping_range() out from under page lock, before then calling unmap_mapping_page() under page lock if still mapped. Fixes: fc127da085c2 ("truncate: handle file thp") Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov --- include/linux/mm.h | 3 +++ mm/memory.c | 40 ++++++++++++++++++++++++++++++++++++++++ mm/truncate.c | 43 +++++++++++++++++++------------------------ 3 files changed, 62 insertions(+), 24 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c274f75efcf9..8ae31622deef 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1719,6 +1719,7 @@ struct zap_details { struct address_space *check_mapping; /* Check page->mapping if set */ pgoff_t first_index; /* Lowest page->index to unmap */ pgoff_t last_index; /* Highest page->index to unmap */ + struct page *single_page; /* Locked page to be unmapped */ }; struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, @@ -1766,6 +1767,7 @@ extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, extern int fixup_user_fault(struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); +void unmap_mapping_page(struct page *page); void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows); void unmap_mapping_range(struct address_space *mapping, @@ -1786,6 +1788,7 @@ static inline int fixup_user_fault(struct mm_struct *mm, unsigned long address, BUG(); return -EFAULT; } +static inline void unmap_mapping_page(struct page *page) { } static inline void unmap_mapping_pages(struct address_space *mapping, pgoff_t start, pgoff_t nr, bool even_cows) { } static inline void unmap_mapping_range(struct address_space *mapping, diff --git a/mm/memory.c b/mm/memory.c index 730daa00952b..26d7f8e265a9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1361,7 +1361,17 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, else if (zap_huge_pmd(tlb, vma, pmd, addr)) goto next; /* fall through */ + } else if (details && details->single_page && + PageTransCompound(details->single_page) && + next - addr == HPAGE_PMD_SIZE && pmd_none(*pmd)) { + /* + * Take and drop THP pmd lock so that we cannot return + * prematurely, while zap_huge_pmd() has cleared *pmd, + * but not yet decremented compound_mapcount(). + */ + spin_unlock(pmd_lock(tlb->mm, pmd)); } + /* * Here there can be other concurrent MADV_DONTNEED or * trans huge page faults running, and if the pmd is @@ -3235,6 +3245,36 @@ static inline void unmap_mapping_range_tree(struct rb_root_cached *root, } } +/** + * unmap_mapping_page() - Unmap single page from processes. + * @page: The locked page to be unmapped. + * + * Unmap this page from any userspace process which still has it mmaped. + * Typically, for efficiency, the range of nearby pages has already been + * unmapped by unmap_mapping_pages() or unmap_mapping_range(). But once + * truncation or invalidation holds the lock on a page, it may find that + * the page has been remapped again: and then uses unmap_mapping_page() + * to unmap it finally. + */ +void unmap_mapping_page(struct page *page) +{ + struct address_space *mapping = page->mapping; + struct zap_details details = { }; + + VM_BUG_ON(!PageLocked(page)); + VM_BUG_ON(PageTail(page)); + + details.check_mapping = mapping; + details.first_index = page->index; + details.last_index = page->index + thp_nr_pages(page) - 1; + details.single_page = page; + + i_mmap_lock_write(mapping); + if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))) + unmap_mapping_range_tree(&mapping->i_mmap, &details); + i_mmap_unlock_write(mapping); +} + /** * unmap_mapping_pages() - Unmap pages from processes. * @mapping: The address space containing pages to be unmapped. diff --git a/mm/truncate.c b/mm/truncate.c index 95af244b112a..234ddd879caa 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -167,13 +167,10 @@ void do_invalidatepage(struct page *page, unsigned int offset, * its lock, b) when a concurrent invalidate_mapping_pages got there first and * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space. */ -static void -truncate_cleanup_page(struct address_space *mapping, struct page *page) +static void truncate_cleanup_page(struct page *page) { - if (page_mapped(page)) { - unsigned int nr = thp_nr_pages(page); - unmap_mapping_pages(mapping, page->index, nr, false); - } + if (page_mapped(page)) + unmap_mapping_page(page); if (page_has_private(page)) do_invalidatepage(page, 0, thp_size(page)); @@ -218,7 +215,7 @@ int truncate_inode_page(struct address_space *mapping, struct page *page) if (page->mapping != mapping) return -EIO; - truncate_cleanup_page(mapping, page); + truncate_cleanup_page(page); delete_from_page_cache(page); return 0; } @@ -325,7 +322,7 @@ void truncate_inode_pages_range(struct address_space *mapping, index = indices[pagevec_count(&pvec) - 1] + 1; truncate_exceptional_pvec_entries(mapping, &pvec, indices); for (i = 0; i < pagevec_count(&pvec); i++) - truncate_cleanup_page(mapping, pvec.pages[i]); + truncate_cleanup_page(pvec.pages[i]); delete_from_page_cache_batch(mapping, &pvec); for (i = 0; i < pagevec_count(&pvec); i++) unlock_page(pvec.pages[i]); @@ -639,6 +636,16 @@ int invalidate_inode_pages2_range(struct address_space *mapping, continue; } + if (!did_range_unmap && page_mapped(page)) { + /* + * If page is mapped, before taking its lock, + * zap the rest of the file in one hit. + */ + unmap_mapping_pages(mapping, index, + (1 + end - index), false); + did_range_unmap = 1; + } + lock_page(page); WARN_ON(page_to_index(page) != index); if (page->mapping != mapping) { @@ -646,23 +653,11 @@ int invalidate_inode_pages2_range(struct address_space *mapping, continue; } wait_on_page_writeback(page); - if (page_mapped(page)) { - if (!did_range_unmap) { - /* - * Zap the rest of the file in one hit. - */ - unmap_mapping_pages(mapping, index, - (1 + end - index), false); - did_range_unmap = 1; - } else { - /* - * Just zap this page - */ - unmap_mapping_pages(mapping, index, - 1, false); - } - } + + if (page_mapped(page)) + unmap_mapping_page(page); BUG_ON(page_mapped(page)); + ret2 = do_launder_page(mapping, page); if (ret2 == 0) { if (!invalidate_complete_page2(mapping, page)) From patchwork Tue Jun 1 21:17:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12292239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.8 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 554C3C47080 for ; Tue, 1 Jun 2021 21:17:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ECF00613B1 for ; Tue, 1 Jun 2021 21:17:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ECF00613B1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B2C36B006E; Tue, 1 Jun 2021 17:17:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 88A6A6B0070; Tue, 1 Jun 2021 17:17:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7517E6B0071; Tue, 1 Jun 2021 17:17:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0119.hostedemail.com [216.40.44.119]) by kanga.kvack.org (Postfix) with ESMTP id 4712F6B006E for ; Tue, 1 Jun 2021 17:17:23 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CADEEAF73 for ; Tue, 1 Jun 2021 21:17:22 +0000 (UTC) X-FDA: 78206415924.36.D9CFDC7 Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) by imf09.hostedemail.com (Postfix) with ESMTP id 3DF956000149 for ; Tue, 1 Jun 2021 21:17:15 +0000 (UTC) Received: by mail-ot1-f53.google.com with SMTP id i14-20020a9d624e0000b029033683c71999so615464otk.5 for ; Tue, 01 Jun 2021 14:17:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:mime-version; bh=N5uDhAxv0tIiSHDRiUFgmqD1xsYzRonX2XfRezH0zjg=; b=a1cxj0Qb2Q7mg06own49b3QWEpgT9CNesL748Sfwr3Zrrj4UjUtL2WJgo6VgXQxgbS lrbQiptZ2tK7nUNk37VezWPZSWUiMaAUlk+uTTw/hPjg5Y3p3cYWJVsnbJ96yp3bvusy CKSc0qiKTPR8TnTS4bu4h93yFsgOl1yLnwLe2lwDJ7JFP9AOVkETS0KajI/aQev92JOG IDlBqAqXQdEYoYBuyfXjtAeb7LQOmGhLtmQLXlTjffAY/EgAUU0AnSTa1aXgXxasaJZW ssE7zY9Xn9wUp05yYAVSsQ9iJ2yiUUYLpbHdnhuiz+LD3DR/vXyQvxaeM6UhS2Y8ROrq SsFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=N5uDhAxv0tIiSHDRiUFgmqD1xsYzRonX2XfRezH0zjg=; b=fduiDw2NN8jq1BxhNrzsco795DpD4P2Sem9yxqvvXTqRqXg8PcaGw4EiJjWPPzjZ4w eRQlGMo6z/+VDwCthaVkbO7r3Jy2Tmf4J9TlfZGS2OUSI5wbahhxHaG0zg5L7jWKKyda o0IXUhUAruM0lwBbVJnXB2d19WkRggOHPl5jrc4OLro/NTMzTp10PaCjUk24Q2X9MyZg wWZNHH4RaQ+sddk8mGAXyRPQbxLKnRdjcawb+L+GmEK/hBvgu55XFkHgB0lKAuZh1nSt bUC0qqFmEoORRPXJ2wqKdgoA+vQiW6weSbOxlVtyZy3ml6YUu3T/nPJiNYcdyjfixRFq NyQw== X-Gm-Message-State: AOAM532amq/4HHJSLG+aFjvSSZRH19YAiO0P1vxA8nv0+bEryilktR2l RXGr/a6x/rLHhIPItaFIs2owCw== X-Google-Smtp-Source: ABdhPJyFME7J7rL15FKaFRQawXTQiSK6pOg1sR2ou3YK3BC2tZQp+lfGEuFuRUcHe8NOh8jr4EVr0g== X-Received: by 2002:a05:6830:1e54:: with SMTP id e20mr23511882otj.227.1622582241799; Tue, 01 Jun 2021 14:17:21 -0700 (PDT) Received: from eggly.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id l23sm3708287oie.52.2021.06.01.14.17.20 (version=TLS1 cipher=ECDHE-ECDSA-AES128-SHA bits=128/128); Tue, 01 Jun 2021 14:17:21 -0700 (PDT) Date: Tue, 1 Jun 2021 14:17:19 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@eggly.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 7/7] mm/thp: remap_page() is only needed on anonymous THP In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (LSU 23 2013-08-11) MIME-Version: 1.0 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=a1cxj0Qb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of hughd@google.com designates 209.85.210.53 as permitted sender) smtp.mailfrom=hughd@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3DF956000149 X-Stat-Signature: hfgec1rawp6yrxot7nx1xkhte5nizhwy X-HE-Tag: 1622582235-539043 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: THP splitting's unmap_page() only sets TTU_SPLIT_FREEZE when PageAnon, and migration entries are only inserted when TTU_MIGRATION (unused here) or TTU_SPLIT_FREEZE is set: so it's just a waste of time for remap_page() to search for migration entries to remove when !PageAnon. Fixes: baa355fd3314 ("thp: file pages support for split_huge_page()") Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi Acked-by: Kirill A. Shutemov --- mm/huge_memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 305f709a7aca..e4a83e310452 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2354,6 +2354,7 @@ static void unmap_page(struct page *page) VM_BUG_ON_PAGE(!PageHead(page), page); + /* If TTU_SPLIT_FREEZE is ever extended to file, update remap_page() */ if (PageAnon(page)) ttu_flags |= TTU_SPLIT_FREEZE; @@ -2368,6 +2369,10 @@ static void unmap_page(struct page *page) static void remap_page(struct page *page, unsigned int nr) { int i; + + /* If TTU_SPLIT_FREEZE is ever extended to file, remove this check */ + if (!PageAnon(page)) + return; if (PageTransHuge(page)) { remove_migration_ptes(page, page, true); } else {