From patchwork Wed Jun 9 04:08:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12308827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85237C47095 for ; Wed, 9 Jun 2021 04:08:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1EE196124C for ; Wed, 9 Jun 2021 04:08:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EE196124C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 74E5C6B0036; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FE526B006E; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5505E6B0070; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0158.hostedemail.com [216.40.44.158]) by kanga.kvack.org (Postfix) with ESMTP id 24A5B6B0036 for ; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B980C8249980 for ; Wed, 9 Jun 2021 04:08:13 +0000 (UTC) X-FDA: 78232852866.39.ABE5904 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf16.hostedemail.com (Postfix) with ESMTP id 260F080192EE for ; Wed, 9 Jun 2021 04:08:10 +0000 (UTC) Received: by mail-ot1-f50.google.com with SMTP id 6-20020a9d07860000b02903e83bf8f8fcso9611729oto.12 for ; Tue, 08 Jun 2021 21:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=QmK8FBhttGUph4KR+UOsWqjeoyXcMjszBFQUOkXXU7c=; b=VcnMtrzlBE+CY5kEaMTWz55SvEBEWCwvrikYxeBoFoxZErVCk3TtL6h4J1h59LMz0V JxeEdVA1ksuzwAK+iAk47JDOZx4YTCcDk8JdBOAv1OnQoonbr2kwMMJkI7tqqaZ0HtAD EoYvnIkY+oC5Pliif6xm5c6BUyv919tY4ctZiE0bHgx6LtTvRDoS6GoLmnMYegZRmJdd 8n1xn99no8iBELP0GxuZV4OAyDWURL5m+kzMRZgQ0J1DtAWGUa9zrXthMKmClikBJGiI pKqfQCP/56PEp2oca7jKizlV7SONU1T2DUmxS1+yTVM3o+MqdJC/pC4yozqUb/rNbhEV 9BDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=QmK8FBhttGUph4KR+UOsWqjeoyXcMjszBFQUOkXXU7c=; b=UH1gE+s22Y1nTt9FSb92qnWEixmK0cX0uEyxSQI0Wyk8e63qJrxSZtBvTzy83wSs3t jNqxYoe4/2E+tfVs+9gr2kDLQW8m6BBVpr2aeS5QSNQOT4bUjdSPbg3DltL3N6sU0aff Z9VqTSWtiy0m807MNOxKkkUly//irvg7bbjcPwv0LZ9NbMar6u5HrEXRtaJT4dYGiHiJ Le96xJ+whVwcrcPmInseZK5IN9IUV8LWO3OlO+4iYJHAiGjCUJML9ET8IFCOC1oGVf3i V6zxecQyNbmoG4HKtAE3f9WK47Uz2kiidoWWUXXXUf/vBYdVZdDAUEhVT0FPuXfjqYuI 4sDQ== X-Gm-Message-State: AOAM531fArMf7alPEFhiISW3nWk+pXHNB7Ep7JJNxcjRWFQGGvx2Jv66 uXlkknMV4vUjrMpj4et0DL3TyQ== X-Google-Smtp-Source: ABdhPJzHa+Ru6vQBkltZYZhJjYyaVUL60oRVZQeRKT3d8Ottzz1rXugi84dq+noUyrtFItweMCApXQ== X-Received: by 2002:a9d:2cf:: with SMTP id 73mr11534662otl.204.1623211692702; Tue, 08 Jun 2021 21:08:12 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x13sm3481856ote.70.2021.06.08.21.08.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 21:08:12 -0700 (PDT) Date: Tue, 8 Jun 2021 21:08:09 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , Shakeel Butt , Oscar Salvador , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 02/10] mm/thp: make is_huge_zero_pmd() safe and quicker In-Reply-To: Message-ID: <21ea9ca-a1f5-8b90-5e88-95fb1c49bbfa@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 260F080192EE Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=VcnMtrzl; spf=pass (imf16.hostedemail.com: domain of hughd@google.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: qog9j75a9yz9nmmobhbq6cp5pqfqp9si X-HE-Tag: 1623211690-813417 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most callers of is_huge_zero_pmd() supply a pmd already verified present; but a few (notably zap_huge_pmd()) do not - it might be a pmd migration entry, in which the pfn is encoded differently from a present pmd: which might pass the is_huge_zero_pmd() test (though not on x86, since L1TF forced us to protect against that); or perhaps even crash in pmd_page() applied to a swap-like entry. Make it safe by adding pmd_present() check into is_huge_zero_pmd() itself; and make it quicker by saving huge_zero_pfn, so that is_huge_zero_pmd() will not need to do that pmd_page() lookup each time. __split_huge_pmd_locked() checked pmd_trans_huge() before: that worked, but is unnecessary now that is_huge_zero_pmd() checks present. Fixes: e71769ae5260 ("mm: enable thp migration for shmem thp") Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Reviewed-by: Yang Shi --- Patch added (replacing part of first) since the v1 series was posted. include/linux/huge_mm.h | 8 +++++++- mm/huge_memory.c | 5 ++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 9626fda5efce..2a8ebe6c222e 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -286,6 +286,7 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd); extern struct page *huge_zero_page; +extern unsigned long huge_zero_pfn; static inline bool is_huge_zero_page(struct page *page) { @@ -294,7 +295,7 @@ static inline bool is_huge_zero_page(struct page *page) static inline bool is_huge_zero_pmd(pmd_t pmd) { - return is_huge_zero_page(pmd_page(pmd)); + return READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd) && pmd_present(pmd); } static inline bool is_huge_zero_pud(pud_t pud) @@ -440,6 +441,11 @@ static inline bool is_huge_zero_page(struct page *page) return false; } +static inline bool is_huge_zero_pmd(pmd_t pmd) +{ + return false; +} + static inline bool is_huge_zero_pud(pud_t pud) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 42cfefc6e66e..5885c5f5836f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -62,6 +62,7 @@ static struct shrinker deferred_split_shrinker; static atomic_t huge_zero_refcount; struct page *huge_zero_page __read_mostly; +unsigned long huge_zero_pfn __read_mostly = ~0UL; bool transparent_hugepage_enabled(struct vm_area_struct *vma) { @@ -98,6 +99,7 @@ static bool get_huge_zero_page(void) __free_pages(zero_page, compound_order(zero_page)); goto retry; } + WRITE_ONCE(huge_zero_pfn, page_to_pfn(zero_page)); /* We take additional reference here. It will be put back by shrinker */ atomic_set(&huge_zero_refcount, 2); @@ -147,6 +149,7 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink, if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { struct page *zero_page = xchg(&huge_zero_page, NULL); BUG_ON(zero_page == NULL); + WRITE_ONCE(huge_zero_pfn, ~0UL); __free_pages(zero_page, compound_order(zero_page)); return HPAGE_PMD_NR; } @@ -2071,7 +2074,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, return; } - if (pmd_trans_huge(*pmd) && is_huge_zero_pmd(*pmd)) { + if (is_huge_zero_pmd(*pmd)) { /* * FIXME: Do we want to invalidate secondary mmu by calling * mmu_notifier_invalidate_range() see comments below inside