From patchwork Wed Jun 15 17:29:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12882799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E5CC433EF for ; Wed, 15 Jun 2022 17:29:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BA1A6B0072; Wed, 15 Jun 2022 13:29:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43EC06B0074; Wed, 15 Jun 2022 13:29:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 307216B0075; Wed, 15 Jun 2022 13:29:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 223116B0072 for ; Wed, 15 Jun 2022 13:29:36 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EC5403D9 for ; Wed, 15 Jun 2022 17:29:35 +0000 (UTC) X-FDA: 79581157110.15.472030C Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf04.hostedemail.com (Postfix) with ESMTP id 7ED224008D for ; Wed, 15 Jun 2022 17:29:35 +0000 (UTC) Received: by mail-pf1-f170.google.com with SMTP id c196so12058019pfb.1 for ; Wed, 15 Jun 2022 10:29:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gQUColGb/PRKYuEV4lE8XSFjbzf2io5XZTvMnHxGXf0=; b=P5PSssOdjWY/xPZ7jddHeQPn8UutJ5m9TF+K4vzKHaQtM6cE/Mf+NE4GoAnGSxi3Io Y2LlfpoNzzarjkEPUOBBbK2VNLV6QvrkCU0YgZMxegJsia53STwh6Di/GzL3gsyqzoFP rUAxM+SHTD3Ks6z6Onsi/In0Mrk2HzetpwpKANFN1OqsGo2AHjr6/haWrWg2TS12egTu 1PKdIZyzPNk09Cs6fGAidY2XEJ/VqNbV78o1QLo7kcwS7nDaVKeKv77VuBXpQHOqqTvl KFl4JRX1TJx3TG4Jb+1vv779H0J6hBCcUH2+m3wdmfheRWs69YXkaQK77FyCnII6PT7d qqTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gQUColGb/PRKYuEV4lE8XSFjbzf2io5XZTvMnHxGXf0=; b=K8cIfTmRiBzdpW3hxtcvzVAqo1YLkNjp5viudO/zeksNMSqP2cax2+mQtJ6OAsr2QS eyZUFz+gPhxsu7Yp11f9pShivl+wT8Mw0YsCCnStqpH6WjgKXgcpgxfSVIImAH3KoEPl XvovPaJiSMYTwF83BCuimKaGyV109h8tGb1iL2H5Vk+nua3JRvUFHa46xZdmoPXUW5xj 9E5cK76U0E/kn9O4mAQg8LAL0dZUQIoeigrnMbmihy+p2DUPBP6aF/BZXKkEQ4b6aJcu hsTBojQ1ER215wz5XAItEpp9OXdRmEYjhlI78KMHI/xpWw11c6WnQnkube7M7ZX0vW/9 9TgA== X-Gm-Message-State: AJIora9YqtwjdqXqMX0xxm8Jg2N+bM3KtGU2cLBeumzBaDq1zFDlZaOt QYtCrcWuK6JEZYAFf0GMAFw= X-Google-Smtp-Source: AGRyM1s1Cazz5Rcz0v51mDmvba3KC32sP5s7r0OCDu88q+Oc50uuWQFdSzmqKeUrPH3UKttvAQA9+w== X-Received: by 2002:a63:ec0f:0:b0:3fd:e23d:b9f9 with SMTP id j15-20020a63ec0f000000b003fde23db9f9mr747614pgh.612.1655314174419; Wed, 15 Jun 2022 10:29:34 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:33 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 1/7] mm: khugepaged: check THP flag in hugepage_vma_check() Date: Wed, 15 Jun 2022 10:29:20 -0700 Message-Id: <20220615172926.546974-2-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314175; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gQUColGb/PRKYuEV4lE8XSFjbzf2io5XZTvMnHxGXf0=; b=J5dS5nSxHVLS5OUcER7bBBV11mFENq6xAoKrcfB68nWvEmbrlekas3FZr7E5I9yqcDfIhz AYwoH2vs0KMzmmqTTZt3JsUAPvHEJhdTTBiMr35378DuwmS0WYuWBG7tAK8HdtUsyCF/WB TwOjE0m496ZaSJupOEmq3ttx2QePAHI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314175; a=rsa-sha256; cv=none; b=MxctkHi73+FeIFGUiGG+e2RPfJHT5pr325Ud+DAhEqptiuPmOTR+9sSmCNPqIfw0XsWK4U CcPCqGZzgCAV7jJFpUfRlpW+Lzpkv6BnTwSRUG9+LN21P/8y0YJIWqKeMmsd/Z7XzHv8m0 +9YQ93xDChMBM37lGiSgCm48DENSqT8= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P5PSssOd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=shy828301@gmail.com Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P5PSssOd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: ycwtygjp3tag7tqh695hx43qs9upbib5 X-Rspamd-Queue-Id: 7ED224008D X-HE-Tag: 1655314175-782205 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently the THP flag check in hugepage_vma_check() will fallthrough if the flag is NEVER and VM_HUGEPAGE is set. This is not a problem for now since all the callers have the flag checked before or can't be invoked if the flag is NEVER. However, the following patch will call hugepage_vma_check() in more places, for example, page fault, so this flag must be checked in hugepge_vma_check(). Reviewed-by: Zach O'Keefe Reviewed-by: Miaohe Lin Signed-off-by: Yang Shi --- mm/khugepaged.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 476d79360101..b1dab94c0f1e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -458,6 +458,9 @@ bool hugepage_vma_check(struct vm_area_struct *vma, if (shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); + if (!khugepaged_enabled()) + return false; + /* THP settings require madvise. */ if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) return false; From patchwork Wed Jun 15 17:29:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12882801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F37CCCA473 for ; Wed, 15 Jun 2022 17:29:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68B966B0075; Wed, 15 Jun 2022 13:29:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 613406B0078; Wed, 15 Jun 2022 13:29:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F1A96B007B; Wed, 15 Jun 2022 13:29:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 304346B0075 for ; Wed, 15 Jun 2022 13:29:40 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EA269353B9 for ; Wed, 15 Jun 2022 17:29:39 +0000 (UTC) X-FDA: 79581157278.24.4BFB853 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf09.hostedemail.com (Postfix) with ESMTP id 8743B14008D for ; Wed, 15 Jun 2022 17:29:36 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id g16-20020a17090a7d1000b001ea9f820449so2754824pjl.5 for ; Wed, 15 Jun 2022 10:29:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=51sNwNc9iGRZC8aX2+MPWXbJAssH24vr8KC9AQWwCZ4=; b=DA1ymx6Eusxll6nKGGOtuCA1e5yLxoL17ga00PUZEbMSGUwZ9QG6+NBJuPfd7ilrmc vxzciGDA9U1JKH4VjkUhxyoesCyPzaLeNmcLKGq2yivT4DaqFKlc2ONucsFlXYa0GFhn nlwiSSWq4Fu6wWNsgWNPShUy524Ut88IgYHLjMWkb9gyH3pmJkLL99fTKnmqxnJZh/cu q6A8rMdMbGH6pv03ku6Mp75PoUjbWnjkDtLFaQBSegDQOFObnqm0jYznkWl4N7JJycvK U7IW31KkKAzmgrgRgQw3xFuSzOZqlDBEt+MnTO135pasLPl+TY1lV5wINAfU/h9zAtby eaiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=51sNwNc9iGRZC8aX2+MPWXbJAssH24vr8KC9AQWwCZ4=; b=OX8ML0s9tgF3wkEIRJDtO/IGXiQdhjqa4XYZDVm8yJcoi7jBYcd8WhJb9gxIJB23VN 1aDz/jHNpWggxg2nCulUVY3wQzNsmAo5wbjo/iEyU7oy1PYmgJ7S5MOo2SUUzhaI2zim S8AnmNv52BxDuoKInHSwWJt1s1yyxAFz6EVHBVJs7JgHoh1NeWQdx63IBQNjeOrxMQGH oyhEScsNkxpK4Y27D6FNwR9VeW6U4eYzJ8bL5jdxKzU351BwRCy7UgAnJO97nttK13CH sjlH6QxkCnEWeBcdL213ZrGJIyGwTmd4MFTXeeWRQ5H/y+E89/AK1gf25xP4VJd2oh/H fX6w== X-Gm-Message-State: AJIora+Ud1F9zocAkWCOr/bsFYnHhPJ3N5mf1QEXOHZqdB2PfjoFRrjS MvBH6aNGBgxuczjay212q3YWJQdhxmk= X-Google-Smtp-Source: AGRyM1ubIIBH2uc5bj3/MaNUe6wnfsdVL/fOlKAmVLhqdVvcHkAScJirEVp37jaO3ZqktGgU+ZYlcg== X-Received: by 2002:a17:90a:31cf:b0:1c9:f9b8:68c7 with SMTP id j15-20020a17090a31cf00b001c9f9b868c7mr11499592pjf.34.1655314176105; Wed, 15 Jun 2022 10:29:36 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:35 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 2/7] mm: thp: consolidate vma size check to transhuge_vma_suitable Date: Wed, 15 Jun 2022 10:29:21 -0700 Message-Id: <20220615172926.546974-3-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314178; a=rsa-sha256; cv=none; b=ge8Yq2IHPok+PldcK50KRk7gAbNdjVLNrREXO1w+p+feqoRyWqCfZ2FdfunJDkSgcEo1tW K5I+9grlw5iLZKQPfEhczJ9Ws0mMCRPbBEGIE9noWCMJuvfNRmQztDlwQuXQszGXty8JLq Y6wf/YyqywWE7vtpnzl+C2mnyW8EstM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DA1ymx6E; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314178; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=51sNwNc9iGRZC8aX2+MPWXbJAssH24vr8KC9AQWwCZ4=; b=1yNUYD/Sqxt48Fde9S5ZzKkyL7qZOPElAHc3W6ps2xaJUmtNpHKBcdOwqE+JkQRjcQmKHH ykVyevz2jYAwDjOq3AZr20GBSQZ+f5y3O8OR0QUNjF3YClDjXmTo2k2V26IWplpbU4wBfe /7dYFHgAbZs6Vwj/yFIR6wsfbdxy9/g= X-Rspamd-Queue-Id: 8743B14008D X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=DA1ymx6E; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspamd-Server: rspam06 X-Stat-Signature: idw3d7awe1xbbr3pw9r4m175fyfmudzu X-HE-Tag: 1655314176-688146 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are couple of places that check whether the vma size is ok for THP or whether address fits, they are open coded and duplicate, use transhuge_vma_suitable() to do the job by passing in (vma->end - HPAGE_PMD_SIZE). Move vma size check into hugepage_vma_check(). This will make khugepaged_enter() is as same as khugepaged_enter_vma(). There is just one caller for khugepaged_enter(), replace it to khugepaged_enter_vma() and remove khugepaged_enter(). Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe --- include/linux/huge_mm.h | 11 +++++++++++ include/linux/khugepaged.h | 14 -------------- mm/huge_memory.c | 2 +- mm/khugepaged.c | 19 ++++++------------- 4 files changed, 18 insertions(+), 28 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 648cb3ce7099..8a5a8bfce0f5 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -116,6 +116,17 @@ extern struct kobj_attribute shmem_enabled_attr; extern unsigned long transparent_hugepage_flags; +/* + * Do the below checks: + * - For file vma, check if the linear page offset of vma is + * HPAGE_PMD_NR aligned within the file. The hugepage is + * guaranteed to be hugepage-aligned within the file, but we must + * check that the PMD-aligned addresses in the VMA map to + * PMD-aligned offsets within the file, else the hugepage will + * not be PMD-mappable. + * - For all vmas, check if the haddr is in an aligned HPAGE_PMD_SIZE + * area. + */ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, unsigned long addr) { diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 392d34c3c59a..31ca8a7f78f4 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -51,16 +51,6 @@ static inline void khugepaged_exit(struct mm_struct *mm) if (test_bit(MMF_VM_HUGEPAGE, &mm->flags)) __khugepaged_exit(mm); } - -static inline void khugepaged_enter(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - khugepaged_enabled()) { - if (hugepage_vma_check(vma, vm_flags)) - __khugepaged_enter(vma->vm_mm); - } -} #else /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmm) { @@ -68,10 +58,6 @@ static inline void khugepaged_fork(struct mm_struct *mm, struct mm_struct *oldmm static inline void khugepaged_exit(struct mm_struct *mm) { } -static inline void khugepaged_enter(struct vm_area_struct *vma, - unsigned long vm_flags) -{ -} static inline void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 4f9bbb4eab23..b530462c4493 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -726,7 +726,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) return VM_FAULT_FALLBACK; if (unlikely(anon_vma_prepare(vma))) return VM_FAULT_OOM; - khugepaged_enter(vma, vma->vm_flags); + khugepaged_enter_vma(vma, vma->vm_flags); if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm) && diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b1dab94c0f1e..db0b334a7d1f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -450,8 +450,8 @@ bool hugepage_vma_check(struct vm_area_struct *vma, if (vma_is_dax(vma)) return false; - if (vma->vm_file && !IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - - vma->vm_pgoff, HPAGE_PMD_NR)) + /* Check alignment for file vma and size for both file and anon vma */ + if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) return false; /* Enabled via shmem mount options or sysfs settings. */ @@ -512,9 +512,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - khugepaged_enabled() && - (((vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK) < - (vma->vm_end & HPAGE_PMD_MASK))) { + khugepaged_enabled()) { if (hugepage_vma_check(vma, vm_flags)) __khugepaged_enter(vma->vm_mm); } @@ -950,7 +948,6 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, struct vm_area_struct **vmap) { struct vm_area_struct *vma; - unsigned long hstart, hend; if (unlikely(khugepaged_test_exit(mm))) return SCAN_ANY_PROCESS; @@ -959,9 +956,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!vma) return SCAN_VMA_NULL; - hstart = (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK; - hend = vma->vm_end & HPAGE_PMD_MASK; - if (address < hstart || address + HPAGE_PMD_SIZE > hend) + if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; if (!hugepage_vma_check(vma, vma->vm_flags)) return SCAN_VMA_CHECK; @@ -2147,10 +2142,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, progress++; continue; } - hstart = (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK; - hend = vma->vm_end & HPAGE_PMD_MASK; - if (hstart >= hend) - goto skip; + hstart = round_up(vma->vm_start, HPAGE_PMD_SIZE); + hend = round_down(vma->vm_end, HPAGE_PMD_SIZE); if (khugepaged_scan.address > hend) goto skip; if (khugepaged_scan.address < hstart) From patchwork Wed Jun 15 17:29:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12882800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22411CCA47B for ; Wed, 15 Jun 2022 17:29:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1F556B0074; Wed, 15 Jun 2022 13:29:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACEAF6B0075; Wed, 15 Jun 2022 13:29:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9700F6B0078; Wed, 15 Jun 2022 13:29:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8803D6B0074 for ; Wed, 15 Jun 2022 13:29:39 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 6A5A0601E8 for ; Wed, 15 Jun 2022 17:29:39 +0000 (UTC) X-FDA: 79581157278.04.DF37076 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf26.hostedemail.com (Postfix) with ESMTP id A222314009E for ; Wed, 15 Jun 2022 17:29:38 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id e9so11831550pju.5 for ; Wed, 15 Jun 2022 10:29:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WrF836BhqYoAAQYNFMJen23eOl+F6zz51fEhzRe3ck4=; b=jNiMa1EClCZmNoxHFwK9zkQt71WNj8eQXx1AtJzkARohOhJSxbNe9g9wKpdtMdMiFM JUylBaxwrEsHZe9FDtW/ty8pZHqkoiXrCVnIf4XeVJdT7glhN+11Co4GX3RF1sUTcydf V3HfSNZro9f+AcdFfuHK+L4B6g8H28LVBK/7P7o1JiWQR01uO+wBiMM/9HiCPM6SAMQx d2a2Sz0T524tVIKdzfkdECmzkPJH8aXQXrXRWCyPKVJHg489kISeCmxjam/lwp9uY3TL eupQy8+iv/QTL3N+W5Rklt4NRIJ/YA5TCnZRDnOfKJw5G4UK28mGTOs/faVfewTqkG2c Oztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WrF836BhqYoAAQYNFMJen23eOl+F6zz51fEhzRe3ck4=; b=PVbcqXdRC5D2I16mjH9bM50OPaO2sHHaC5969+yx8rwy2ayY7wmBzlBnMng0I6v4Xa fNMbacKnAlaO/o6GY/LBcv02pmjqyn20Yf/2jM1JJ2+vnSPJ01bsQ15fxejXOWDJSsS8 lbyIi5pS+HHET3vG6WksSBP8gh5N7sTkI39sfRtJ8dPSntBVQ97xCJbXRNWzVA/CaCCQ /4c3TOGg7JtozxXoAS62yxeQNLhT0vtEBIrP2I/k7equcphLnSVdElHvxCQKE4Y1+lcm 6Mf7eDPG8aomsBEl86YsAcr3BnOE0XcIgmmM04JmFuMe9zhqqExWhJJW6+U5cFJVtmax 7N3A== X-Gm-Message-State: AJIora9sbX3uqZVMeHAVJK20CviiYBlrRVWE84naSLZI+FAKgaFknq7D QxGZwF/LVMeWSqaMARFqQJY= X-Google-Smtp-Source: AGRyM1vRDq5qAc7St9qdLuZS57haehWUHZEdEiVLV8l2rTbRMydreLZ77f4O354sxk37OOZMcG9tJw== X-Received: by 2002:a17:90b:4c4c:b0:1e8:6f9a:b642 with SMTP id np12-20020a17090b4c4c00b001e86f9ab642mr11434299pjb.21.1655314177666; Wed, 15 Jun 2022 10:29:37 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:37 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 3/7] mm: khugepaged: better comments for anon vma check in hugepage_vma_revalidate Date: Wed, 15 Jun 2022 10:29:22 -0700 Message-Id: <20220615172926.546974-4-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314178; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WrF836BhqYoAAQYNFMJen23eOl+F6zz51fEhzRe3ck4=; b=f7pFRNWHkP0X6lZJNR8/GJtOlDInt2hBlc5VtQtD/oRhsw4EwnnNpzVCD0GJDT54T0hZtk cy/xAmiEpsIb7f8mB8N1ANK8/XE1i1Kuqx8iK3i3JX+jeKiMGq3z9Z7VFnYyP+lvKJjLui P32iEwd0kFN5nLYZuzTuhkFdRjrlLHY= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jNiMa1EC; spf=pass (imf26.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314178; a=rsa-sha256; cv=none; b=zfbNgXp9NOSxMNC8IxLa4WPyhhEd8siLM8a5mg+QmiDC1ed7IwYMYUDF+B2j2k2zOsScH0 kHy/u5UF8PbWMhRokhAA9gKIm6poc+/2fqjQqUpJPhqtqVfzsA6/EVGOrcIolzbX2PGA85 nJoj3/g8cTulelKgWoGhn5PiQRI3SwQ= X-Stat-Signature: sx7atwrra5s6fdsrn51rkp79h9j8h64q X-Rspamd-Queue-Id: A222314009E X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jNiMa1EC; spf=pass (imf26.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1655314178-283401 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The hugepage_vma_revalidate() needs to check if the vma is still anonymous vma or not since the address may be unmapped then remapped to file before khugepaged reaquired the mmap_lock. The old comment is not quite helpful, elaborate this with better comment. Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe --- mm/khugepaged.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index db0b334a7d1f..5baa394e34c8 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -960,7 +960,13 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, return SCAN_ADDRESS_RANGE; if (!hugepage_vma_check(vma, vma->vm_flags)) return SCAN_VMA_CHECK; - /* Anon VMA expected */ + /* + * Anon VMA expected, the address may be unmapped then + * remapped to file after khugepaged reaquired the mmap_lock. + * + * hugepage_vma_check may return true for qualified file + * vmas. + */ if (!vma->anon_vma || !vma_is_anonymous(vma)) return SCAN_VMA_CHECK; return 0; From patchwork Wed Jun 15 17:29:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12882804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D816FC43334 for ; Wed, 15 Jun 2022 17:29:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E2766B007B; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4648E8D0002; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10BAF8D0001; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E7C696B007B for ; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BAA90E60 for ; Wed, 15 Jun 2022 17:29:45 +0000 (UTC) X-FDA: 79581157530.11.05B527B Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf01.hostedemail.com (Postfix) with ESMTP id 133194009D for ; Wed, 15 Jun 2022 17:29:44 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id u18so11001674plb.3 for ; Wed, 15 Jun 2022 10:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0N0t9howdLfNntzOrT/sFPKbK6rgBL9c7HpeSr6Z4As=; b=plkDnRo/ngAq2p1xXo+l6xL9fBjXBke5So1zG5HFSb5vJX4ot9uGcb0KUAHR+Ph+Ml x9+sINiMQ0/2ZvfachaHAvl/wBQtjQzst7n7Vt7qQqMPyg5Ee6u3sIzzCasOKvESnXT/ yTC9rYAiCSHRW7KU07yE8dExF1o7McjKNNaRrAzG+Q69Jmj0VAlO1g8uYNrZR46TLcKI TvEO/Z/yoWxuwYqMLnkn/ak0Ki9A7ZX4XqFuqOcLKWBLmVSiySfVp4lwB2GAUXZEpq0r GLeKiHv1Ss/ZouDvybXXN6J2W+EFzaOMEa6Zo4xe7J+R2VeFBAILAsZ8oBsF9mSScg4H P+lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0N0t9howdLfNntzOrT/sFPKbK6rgBL9c7HpeSr6Z4As=; b=SQE9MP4DuPRbWNP9oVmQlfNheEwKHdCkDkI4NlHCM4XU7C7nHIln9PyvtiOyZ3s17p sOh5Ng7db1YrQUmqAFqnJSfurP5P6vujVWUoe/5pWdb3MCWCst/GojlQPx3A2swqm3oa cCNAzEbrQtx/QslP5XxrDE7ysplDKTZtaIli9I0BHkb+7XWYzMoEhjH04hF30UKRlBvU K9lX6GFCrGNZeiscmRbiZf6Yg8fFbx9dn6jSNUw4tfKJuXvRARirdzGf9dMpJREXvVBW NADPwX4nVYljifINHUhlqi039HIErSjPoZfflvwnuJtVetIOInnZ326ogi2DyjyUdGxh IvEg== X-Gm-Message-State: AJIora9w4pEeUDEAku9qQ3GKLFmK8C+MZQFOwC/PMhNCs6AkNCtPdTZe G81ox9qb7PUKfswC6YAFKFI= X-Google-Smtp-Source: AGRyM1uHc6Ah3zHBOP8Sn5+jQVN7TbPYQZ5uzgq4ONWj/a98/ypZEEKfD2jSWquQYwKo4HLyFsAIDA== X-Received: by 2002:a17:90b:304:b0:1e3:4489:193c with SMTP id ay4-20020a17090b030400b001e34489193cmr11441651pjb.78.1655314179550; Wed, 15 Jun 2022 10:29:39 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:38 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 4/7] mm: thp: kill transparent_hugepage_active() Date: Wed, 15 Jun 2022 10:29:23 -0700 Message-Id: <20220615172926.546974-5-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314185; a=rsa-sha256; cv=none; b=IraY1sN525BnSHbAdcTSgtq0Rt7wYhVnGIuTTr3VlgPwvZZ0mwlgfL82BfqOvfVJJyB7F4 rQb9vnBWk91erqwysnMXq6SP26xsdx3kbMfzu00r0ozJk3QPaxP56YmDtP+dNm3KOMrc65 yt53vNykfbXxQ8jlwBG+8YeAjFmVXXY= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="plkDnRo/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314185; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0N0t9howdLfNntzOrT/sFPKbK6rgBL9c7HpeSr6Z4As=; b=6WYjE2r2Bxgx0BNPBK93NHd5/s7ch6ysG7uYgNFQESob0Duj8YhkLZl/FNcd3pxRuy4D3X WYY1RyO0KT1rhk/7HYoW0z8mVtdDlELaUaHU1wx6YL4XoV8WvpIxK9IqYShrXrck0Wqd59 zeDOs2yTCtolwwEfp7NJm2gWhUhITuQ= Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="plkDnRo/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspam-User: X-Stat-Signature: qb6bhtn8ktsx5p99rjgr3u8bddgztg5x X-Rspamd-Queue-Id: 133194009D X-Rspamd-Server: rspam08 X-HE-Tag: 1655314184-847731 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The transparent_hugepage_active() was introduced to show THP eligibility bit in smaps in proc, smaps is the only user. But it actually does the similar check as hugepage_vma_check() which is used by khugepaged. We definitely don't have to maintain two similar checks, so kill transparent_hugepage_active(). This patch also fixed the wrong behavior for VM_NO_KHUGEPAGED vmas. Also move hugepage_vma_check() to huge_memory.c and huge_mm.h since it is not only for khugepaged anymore. Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe --- fs/proc/task_mmu.c | 2 +- include/linux/huge_mm.h | 11 ++++++++- include/linux/khugepaged.h | 2 -- mm/huge_memory.c | 50 +++++++++++++++++++++++++++++++------- mm/khugepaged.c | 48 +++--------------------------------- 5 files changed, 56 insertions(+), 57 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 37ccb5c9f4f8..39a40ec181e7 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -863,7 +863,7 @@ static int show_smap(struct seq_file *m, void *v) __show_smap(m, &mss, false); seq_printf(m, "THPeligible: %d\n", - transparent_hugepage_active(vma)); + hugepage_vma_check(vma, vma->vm_flags, true)); if (arch_pkeys_enabled()) seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 8a5a8bfce0f5..aeb13119ee28 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -202,7 +202,9 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) !inode_is_open_for_write(inode) && S_ISREG(inode->i_mode); } -bool transparent_hugepage_active(struct vm_area_struct *vma); +bool hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, + bool smaps); #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ @@ -368,6 +370,13 @@ static inline bool transhuge_vma_enabled(struct vm_area_struct *vma, return false; } +static inline bool hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, + bool smaps) +{ + return false; +} + static inline void prep_transhuge_page(struct page *page) {} #define transparent_hugepage_flags 0UL diff --git a/include/linux/khugepaged.h b/include/linux/khugepaged.h index 31ca8a7f78f4..ea5fd4c398f7 100644 --- a/include/linux/khugepaged.h +++ b/include/linux/khugepaged.h @@ -10,8 +10,6 @@ extern struct attribute_group khugepaged_attr_group; extern int khugepaged_init(void); extern void khugepaged_destroy(void); extern int start_stop_khugepaged(void); -extern bool hugepage_vma_check(struct vm_area_struct *vma, - unsigned long vm_flags); extern void __khugepaged_enter(struct mm_struct *mm); extern void __khugepaged_exit(struct mm_struct *mm); extern void khugepaged_enter_vma(struct vm_area_struct *vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b530462c4493..a28c6100b491 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -69,21 +69,53 @@ static atomic_t huge_zero_refcount; struct page *huge_zero_page __read_mostly; unsigned long huge_zero_pfn __read_mostly = ~0UL; -bool transparent_hugepage_active(struct vm_area_struct *vma) +bool hugepage_vma_check(struct vm_area_struct *vma, + unsigned long vm_flags, + bool smaps) { - /* The addr is used to check if the vma size fits */ - unsigned long addr = (vma->vm_end & HPAGE_PMD_MASK) - HPAGE_PMD_SIZE; + if (!transhuge_vma_enabled(vma, vm_flags)) + return false; + + if (vm_flags & VM_NO_KHUGEPAGED) + return false; + + /* Don't run khugepaged against DAX vma */ + if (vma_is_dax(vma)) + return false; - if (!transhuge_vma_suitable(vma, addr)) + /* Check alignment for file vma and size for both file and anon vma */ + if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) return false; - if (vma_is_anonymous(vma)) - return __transparent_hugepage_enabled(vma); - if (vma_is_shmem(vma)) + + /* Enabled via shmem mount options or sysfs settings. */ + if (shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); - if (transhuge_vma_enabled(vma, vma->vm_flags) && file_thp_enabled(vma)) + + if (!khugepaged_enabled()) + return false; + + /* THP settings require madvise. */ + if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) + return false; + + /* Only regular file is valid */ + if (file_thp_enabled(vma)) return true; - return false; + if (!vma_is_anonymous(vma)) + return false; + + if (vma_is_temporary_stack(vma)) + return false; + + /* + * THPeligible bit of smaps should show 1 for proper VMAs even + * though anon_vma is not initialized yet. + */ + if (!vma->anon_vma) + return smaps; + + return true; } static bool get_huge_zero_page(void) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5baa394e34c8..3afd87f8c0b1 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -437,46 +437,6 @@ static inline int khugepaged_test_exit(struct mm_struct *mm) return atomic_read(&mm->mm_users) == 0; } -bool hugepage_vma_check(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - if (!transhuge_vma_enabled(vma, vm_flags)) - return false; - - if (vm_flags & VM_NO_KHUGEPAGED) - return false; - - /* Don't run khugepaged against DAX vma */ - if (vma_is_dax(vma)) - return false; - - /* Check alignment for file vma and size for both file and anon vma */ - if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) - return false; - - /* Enabled via shmem mount options or sysfs settings. */ - if (shmem_file(vma->vm_file)) - return shmem_huge_enabled(vma); - - if (!khugepaged_enabled()) - return false; - - /* THP settings require madvise. */ - if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) - return false; - - /* Only regular file is valid */ - if (file_thp_enabled(vma)) - return true; - - if (!vma->anon_vma || !vma_is_anonymous(vma)) - return false; - if (vma_is_temporary_stack(vma)) - return false; - - return true; -} - void __khugepaged_enter(struct mm_struct *mm) { struct mm_slot *mm_slot; @@ -513,7 +473,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && khugepaged_enabled()) { - if (hugepage_vma_check(vma, vm_flags)) + if (hugepage_vma_check(vma, vm_flags, false)) __khugepaged_enter(vma->vm_mm); } } @@ -958,7 +918,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; - if (!hugepage_vma_check(vma, vma->vm_flags)) + if (!hugepage_vma_check(vma, vma->vm_flags, false)) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1448,7 +1408,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) * the valid THP. Add extra VM_HUGEPAGE so hugepage_vma_check() * will not fail the vma for missing VM_HUGEPAGE */ - if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE)) + if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE, false)) return; /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ @@ -2143,7 +2103,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, progress++; break; } - if (!hugepage_vma_check(vma, vma->vm_flags)) { + if (!hugepage_vma_check(vma, vma->vm_flags, false)) { skip: progress++; continue; From patchwork Wed Jun 15 17:29:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12882803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B652CCA47B for ; Wed, 15 Jun 2022 17:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F8336B007D; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A6CE6B007E; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3C8C6B0081; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C2AD96B007D for ; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 87AB63334B for ; Wed, 15 Jun 2022 17:29:45 +0000 (UTC) X-FDA: 79581157530.02.00831BD Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf11.hostedemail.com (Postfix) with ESMTP id E3EC64008F for ; Wed, 15 Jun 2022 17:29:44 +0000 (UTC) Received: by mail-pl1-f180.google.com with SMTP id d13so10958314plh.13 for ; Wed, 15 Jun 2022 10:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h2pLxupVhk89AMwfS8APe/ZQV3YQbagepd9ct77VG2Q=; b=puz5OfQ8vLFujYwqvU8w7iV6yjTTZbZpPMN7QBZX+grfhQ8IiGAqV4Se0C4Hdpiqvk OrkRmY9ak/Ct1UNJeweplSPK8Z22nychYkNPpUVMCs9fmfsI3cnnL8uZ/6xu4GUj2Sgo J+QN9JT58X0TXJxeRnLaSw4M2w8qUirxCowi3hvZafGwpujCLqfR0bulYpJSAvqzrrYu YppV1M+A/XdzlYXyuxa3QGsniodVzcqYP0rcAPPNL/a8PvX1e0ebmRJ7lpkOlWAjeclP X/QwDhTqBnIvVXPKKCdnKarZSJCUOPQsgCB0HFh57WgD3UatQhyLHfOQyskhk9kIJUoH Ouaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h2pLxupVhk89AMwfS8APe/ZQV3YQbagepd9ct77VG2Q=; b=rUNGIC4xIhTwcbXgweBLLc4I8YT2Qm9zVnxzV3dmpuSi1Q53TjZ+yULFzNqiP5ji2j 3cY7ht3mHvhh/1vry00JsToqVe3uHwKxBT1sMBnvIFUKz1ofPQRXZebwUBvt/vcbH7b/ TAxFxhjeSoDRcZ+f/t6oM+ZJDvTf9beOOwfzG4s4l6NDqSJcZlPvg14cIoHjDy/eA+ft QFcG9jz6OULz9CrijjQ355F+K6r+K9CorkMND4mmBNeyUltvUuCuGBjaQMjAwtXp7Byp gcnFYf0pQGgdttMtWC7wA/VVJ4wDBrMgv6zw3n/mtQfqNcRJlDRmRE60InYm0+rK9/rE joPA== X-Gm-Message-State: AJIora9MmepW2P3RaemGcTLJuMtCIQJDcuBSytARcoOYqMV31+ev1Pmi TSbaB1k5XXQFhfFSYzVdtKc= X-Google-Smtp-Source: AGRyM1uWyukWlQxI0iqWLK7ytf/4APIc7YOybkeB4SN9ZpOj92E5RGABSQGWJMfAfoOSrKD/V4E+iw== X-Received: by 2002:a17:90a:94cb:b0:1e3:4295:9c18 with SMTP id j11-20020a17090a94cb00b001e342959c18mr519315pjw.53.1655314180969; Wed, 15 Jun 2022 10:29:40 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:40 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 5/7] mm: thp: kill __transhuge_page_enabled() Date: Wed, 15 Jun 2022 10:29:24 -0700 Message-Id: <20220615172926.546974-6-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314184; a=rsa-sha256; cv=none; b=hUFZBR0fKz4zixyk4jaFKBIqmGY1OGfRHLuBhiErU9/W7sI1ZUOVzMc9fd1oSQBfV26em2 J7oD4Ns8KXOirHL7emo/d9vMwmoJDeq0WR+2664xPTfbaI8P4cLz9yQalEtRbiTWiYMCut uGEWS4pqvmaoCu2erkg8aqHaY9CAcPQ= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=puz5OfQ8; spf=pass (imf11.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h2pLxupVhk89AMwfS8APe/ZQV3YQbagepd9ct77VG2Q=; b=ZBlenx8/WDpwqTh0RLLImXywRgejYTUzC1GDaQjrwjGbhyixmx6p+1QXCakMmqVKARkZHQ ZIEF7+G/STgj9orfKXOs76iD74OlFXdFC6OflsKKHq9wL4xXV/TlVnPnqwLISF+bVTEQ21 jnxerPKpcqWA6owv4kiwtaH935D7Vug= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E3EC64008F X-Stat-Signature: rgmfn1fy1xru3nh5ntpcqhx6wz34bqgq Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=puz5OfQ8; spf=pass (imf11.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1655314184-996857 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page fault path checks THP eligibility with __transhuge_page_enabled() which does the similar thing as hugepage_vma_check(), so use hugepage_vma_check() instead. However page fault allows DAX and !anon_vma cases, so added a new flag, in_pf, to hugepage_vma_check() to make page fault work correctly. The in_pf flag is also used to skip shmem and file THP for page fault since shmem handles THP in its own shmem_fault() and file THP allocation on fault is not supported yet. Also remove hugepage_vma_enabled() since hugepage_vma_check() is the only caller now, it is not necessary to have a helper function. Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe --- fs/proc/task_mmu.c | 2 +- include/linux/huge_mm.h | 57 ++--------------------------------------- mm/huge_memory.c | 51 ++++++++++++++++++++++++++++-------- mm/khugepaged.c | 8 +++--- mm/memory.c | 7 +++-- 5 files changed, 52 insertions(+), 73 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 39a40ec181e7..cef72e49acc5 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -863,7 +863,7 @@ static int show_smap(struct seq_file *m, void *v) __show_smap(m, &mss, false); seq_printf(m, "THPeligible: %d\n", - hugepage_vma_check(vma, vma->vm_flags, true)); + hugepage_vma_check(vma, vma->vm_flags, true, false)); if (arch_pkeys_enabled()) seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index aeb13119ee28..9d97d7ee6234 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -146,48 +146,6 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, return true; } -static inline bool transhuge_vma_enabled(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - /* Explicitly disabled through madvise. */ - if ((vm_flags & VM_NOHUGEPAGE) || - test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)) - return false; - return true; -} - -/* - * to be used on vmas which are known to support THP. - * Use transparent_hugepage_active otherwise - */ -static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma) -{ - - /* - * If the hardware/firmware marked hugepage support disabled. - */ - if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_NEVER_DAX)) - return false; - - if (!transhuge_vma_enabled(vma, vma->vm_flags)) - return false; - - if (vma_is_temporary_stack(vma)) - return false; - - if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_FLAG)) - return true; - - if (vma_is_dax(vma)) - return true; - - if (transparent_hugepage_flags & - (1 << TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG)) - return !!(vma->vm_flags & VM_HUGEPAGE); - - return false; -} - static inline bool file_thp_enabled(struct vm_area_struct *vma) { struct inode *inode; @@ -204,7 +162,7 @@ static inline bool file_thp_enabled(struct vm_area_struct *vma) bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps); + bool smaps, bool in_pf); #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ @@ -348,11 +306,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) return false; } -static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma) -{ - return false; -} - static inline bool transparent_hugepage_active(struct vm_area_struct *vma) { return false; @@ -364,15 +317,9 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, return false; } -static inline bool transhuge_vma_enabled(struct vm_area_struct *vma, - unsigned long vm_flags) -{ - return false; -} - static inline bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps) + bool smaps, bool in_pf) { return false; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a28c6100b491..d0c37d99917b 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -71,24 +71,50 @@ unsigned long huge_zero_pfn __read_mostly = ~0UL; bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps) + bool smaps, bool in_pf) { - if (!transhuge_vma_enabled(vma, vm_flags)) + /* + * Explicitly disabled through madvise or prctl, or some + * architectures may disable THP for some mappings, for + * example, s390 kvm. + * */ + if ((vm_flags & VM_NOHUGEPAGE) || + test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)) return false; - - if (vm_flags & VM_NO_KHUGEPAGED) + /* + * If the hardware/firmware marked hugepage support disabled. + */ + if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_NEVER_DAX)) return false; - /* Don't run khugepaged against DAX vma */ + /* khugepaged doesn't collapse DAX vma, but page fault is fine. */ if (vma_is_dax(vma)) + return in_pf; + + /* + * Special VMA and hugetlb VMA. + * Must be checked after dax since some dax mappings may have + * VM_MIXEDMAP set. + */ + if (vm_flags & VM_NO_KHUGEPAGED) return false; - /* Check alignment for file vma and size for both file and anon vma */ - if (!transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) + /* + * Check alignment for file vma and size for both file and anon vma. + * + * Skip the check for page fault. Huge fault does the check in fault + * handlers. And this check is not suitable for huge PUD fault. + */ + if (!in_pf && + !transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE))) return false; - /* Enabled via shmem mount options or sysfs settings. */ - if (shmem_file(vma->vm_file)) + /* + * Enabled via shmem mount options or sysfs settings. + * Must be done before hugepage flags check since shmem has its + * own flags. + */ + if (!in_pf && shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); if (!khugepaged_enabled()) @@ -99,7 +125,7 @@ bool hugepage_vma_check(struct vm_area_struct *vma, return false; /* Only regular file is valid */ - if (file_thp_enabled(vma)) + if (!in_pf && file_thp_enabled(vma)) return true; if (!vma_is_anonymous(vma)) @@ -111,9 +137,12 @@ bool hugepage_vma_check(struct vm_area_struct *vma, /* * THPeligible bit of smaps should show 1 for proper VMAs even * though anon_vma is not initialized yet. + * + * Allow page fault since anon_vma may be not initialized until + * the first page fault. */ if (!vma->anon_vma) - return smaps; + return (smaps || in_pf); return true; } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 3afd87f8c0b1..2a676f37c921 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -473,7 +473,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && khugepaged_enabled()) { - if (hugepage_vma_check(vma, vm_flags, false)) + if (hugepage_vma_check(vma, vm_flags, false, false)) __khugepaged_enter(vma->vm_mm); } } @@ -918,7 +918,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; - if (!hugepage_vma_check(vma, vma->vm_flags, false)) + if (!hugepage_vma_check(vma, vma->vm_flags, false, false)) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1408,7 +1408,7 @@ void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr) * the valid THP. Add extra VM_HUGEPAGE so hugepage_vma_check() * will not fail the vma for missing VM_HUGEPAGE */ - if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE, false)) + if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE, false, false)) return; /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ @@ -2103,7 +2103,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, progress++; break; } - if (!hugepage_vma_check(vma, vma->vm_flags, false)) { + if (!hugepage_vma_check(vma, vma->vm_flags, false, false)) { skip: progress++; continue; diff --git a/mm/memory.c b/mm/memory.c index be724238a9d3..fee2884481f2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4985,6 +4985,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .gfp_mask = __get_fault_gfp_mask(vma), }; struct mm_struct *mm = vma->vm_mm; + unsigned long vm_flags = vma->vm_flags; pgd_t *pgd; p4d_t *p4d; vm_fault_t ret; @@ -4998,7 +4999,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (!vmf.pud) return VM_FAULT_OOM; retry_pud: - if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) { + if (pud_none(*vmf.pud) && + hugepage_vma_check(vma, vm_flags, false, true)) { ret = create_huge_pud(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; @@ -5031,7 +5033,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (pud_trans_unstable(vmf.pud)) goto retry_pud; - if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) { + if (pmd_none(*vmf.pmd) && + hugepage_vma_check(vma, vm_flags, false, true)) { ret = create_huge_pmd(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; From patchwork Wed Jun 15 17:29:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 12882805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BCDFC433EF for ; Wed, 15 Jun 2022 17:29:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8009F6B007E; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74EA16B0081; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F86C6B0080; Wed, 15 Jun 2022 13:29:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F04A26B0080 for ; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id C0714602F9 for ; Wed, 15 Jun 2022 17:29:45 +0000 (UTC) X-FDA: 79581157530.29.F41A685 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf12.hostedemail.com (Postfix) with ESMTP id 3D977400A7 for ; Wed, 15 Jun 2022 17:29:43 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id k12-20020a17090a404c00b001eaabc1fe5dso2783464pjg.1 for ; Wed, 15 Jun 2022 10:29:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BxodM/enWfpl7ZA6N5naFXFtK7edc51Vh1QDre/mU/Q=; b=mkhCdkIJNqb0v6VU4lByixweHMSmazQOcfv8y2ewFiVeKtD8M+w7to2fu1BHM9jFdW FnkzXfY6+FrSC06EzCmia1Hfqwj1UVUHQBir/i6VaPcvgTdyxFi/D0bk0ChP45wRaEP7 i0x+6MMSNGr6PWLhRBI/MFVrSihpDL13tYcEA5WFGBI73Ue+rR1ygElAlQtRw2JOk8mv lrCeuu2Rhb9PYkgS4A0u3VW2Sok5anSb27z8+p8NuL0bW/OEUtFQkKXGTRPY9nAjsuWR BU1bLwPC0owvKLfTqg4S7y1AcndfGkSb6Q6aDEtdQYxtLn1MEzxy/KNL8S+tIYR50lXG VKcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BxodM/enWfpl7ZA6N5naFXFtK7edc51Vh1QDre/mU/Q=; b=MLLDDp2/AKijOB0eV90rntTSkYouM+TClIo1aNAADmO5cbNWcuoFMqDCasVoUwHb35 k5GcLE4C5yNxRkSxo/Y6iFhuw3EtCboz+dUnMc+usVgNKghNwRijNa98RHjTAx/sV19Z VH+vB6e3P3qPUOSQnfRDrQmer0dzca3fTFajoWWfHhBH94fz7HBD04m6FEMJYZ9uqFrb sjbetqUkinqGbS3rxhIfWc2TvexL0D2sFRmwYuqUMWZ84IZspAKmMOBQjHQOzkXZ1L8f gFVXm7MfXExKG2LFrnY3/ldmjlZEVk49vnXuk6786ykl4CzktTL7GkODD/cnnmUfENIV CODA== X-Gm-Message-State: AJIora+NGATHrcZp5qTlUXywSIW71q4NULgB+ok3Bb6tJWrSrqdxcVk+ BbLfGEogO9bJlixAN0B7jFA= X-Google-Smtp-Source: AGRyM1txU2kmgua+NX3rp4duO3aRrjlN4ZOjkSUAr8RCRc2/lLC+ogmCrebuBsavl/ElraFx4uynEA== X-Received: by 2002:a17:903:186:b0:167:6d8c:68c4 with SMTP id z6-20020a170903018600b001676d8c68c4mr789311plg.91.1655314182347; Wed, 15 Jun 2022 10:29:42 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:41 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 6/7] mm: khugepaged: reorg some khugepaged helpers Date: Wed, 15 Jun 2022 10:29:25 -0700 Message-Id: <20220615172926.546974-7-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314183; a=rsa-sha256; cv=none; b=x+NRKEolarby8B7Q/ppUp1nZ/mZ8ifia2kI74ofAt3Dv2giU1TpiZ4zdt1BGD2qz7Yk0bq zS2RR0MKJZpeZho5fWCdlmNyFJ3FssAPLLPfuSyUkOdpLOVAztMCuI+SD2StvzHhoRE0ai ZBDJuPx32PIMJagwhPw6zaHRIoz8lys= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mkhCdkIJ; spf=pass (imf12.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BxodM/enWfpl7ZA6N5naFXFtK7edc51Vh1QDre/mU/Q=; b=enrG70/379jXc2BEpUVGc0yFZWquFakTD0pSyFp/XJDoVcVOg8eN67HjEuD2IF1LXuv+Wo ULI3lHTRCrA5t+tc9sJb2j9QZ8cFXdQxkxxGFsOP1MOI4Wjdj0VS1acVhtSt2nDuEtAnPp LskdKCPtASBjM9W8G0Q/EJqxggmrMc0= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 3D977400A7 X-Stat-Signature: mtm5i6rhdg4muq53iwc4micwfpk7txrm Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mkhCdkIJ; spf=pass (imf12.hostedemail.com: domain of shy828301@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1655314183-832505 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The khugepaged_{enabled|always|req_madv} are not khugepaged only anymore, move them to huge_mm.h and rename to hugepage_flags_xxx, and remove khugepaged_req_madv due to no users. Also move khugepaged_defrag to khugepaged.c since its only caller is in that file, it doesn't have to be in a header file. Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe --- include/linux/huge_mm.h | 8 ++++++++ include/linux/khugepaged.h | 14 -------------- mm/huge_memory.c | 4 ++-- mm/khugepaged.c | 18 +++++++++++------- 4 files changed, 21 insertions(+), 23 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 9d97d7ee6234..4cf546af7d97 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -116,6 +116,14 @@ extern struct kobj_attribute shmem_enabled_attr; extern unsigned long transparent_hugepage_flags; +#define hugepage_flags_enabled() \ + (transparent_hugepage_flags & \ + ((1<flags)) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d0c37d99917b..0f2cce2d7041 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -117,11 +117,11 @@ bool hugepage_vma_check(struct vm_area_struct *vma, if (!in_pf && shmem_file(vma->vm_file)) return shmem_huge_enabled(vma); - if (!khugepaged_enabled()) + if (!hugepage_flags_enabled()) return false; /* THP settings require madvise. */ - if (!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) + if (!(vm_flags & VM_HUGEPAGE) && !hugepage_flags_always()) return false; /* Only regular file is valid */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2a676f37c921..d8ebb60aae36 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -472,7 +472,7 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - khugepaged_enabled()) { + hugepage_flags_enabled()) { if (hugepage_vma_check(vma, vm_flags, false, false)) __khugepaged_enter(vma->vm_mm); } @@ -763,6 +763,10 @@ static bool khugepaged_scan_abort(int nid) return false; } +#define khugepaged_defrag() \ + (transparent_hugepage_flags & \ + (1< X-Patchwork-Id: 12882802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49135C43334 for ; Wed, 15 Jun 2022 17:29:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C09B16B0078; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB8BC6B007B; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A59A16B007D; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 976C46B0078 for ; Wed, 15 Jun 2022 13:29:45 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 57DED343D0 for ; Wed, 15 Jun 2022 17:29:45 +0000 (UTC) X-FDA: 79581157530.05.7A16C6F Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf17.hostedemail.com (Postfix) with ESMTP id C9F9540091 for ; Wed, 15 Jun 2022 17:29:44 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id w21so12100132pfc.0 for ; Wed, 15 Jun 2022 10:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YtollJahbUnQdMIR/aYwe0ysfbXt4BsX5JTH7cAUXak=; b=N4NuWQSKWQiM9FdTh0JRYtj0u28EsEtECIADCbJ92IumR+tzfD/GaDHyaEZCwEPZ45 ndkfZiW/jfRsteeB9yP9cN9gJTN6ydS9wdWzU+oLNT1WC4vOvwoBdzmDrBH+qpNvAxag JpSIsvOqGHCmhbOYwzkelddRSkQVvjIZMqs7jmrSBxqN0FpowIgVL6ZXoXHpGmHD3PZ+ S2uzDoLcfArRPZ8Mne+aFujCMcOt0U6HXoPEWgUn+o315U9ROrtTdkAbIiHOPZYkP5fK idIjGjTzUH0kLYHJQkeMhS/nKieRKIcCdF/6ipPVUxwBIgIBjV91PFInRIenewHra95S GHdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YtollJahbUnQdMIR/aYwe0ysfbXt4BsX5JTH7cAUXak=; b=MIXFqkOuGUfFmFTPBSewfy1g5EpYOtTkkQYqX2uW1iBxA/CvsQyvHrsjWwtWR71Nuz IeUN21tw3P4Zwk7L2vp4Jp7y2YdCDlAmSXMrcX2/y1ELEtEjCCe2B1KMhhl2R/tSijps UTn9f1pG1ieE0ntEPiXAqqc9EtE4iy1409z1KnmeJoSXFkUMa0c/Fm/GUPxbmo44J4+K kvDX2p3PGtFYZAwX3G1xKBKDB39xPeWsPmIo6vhlLQgV9jVIphRSkhjeODDKZjfE3Vh9 1ZtO1sTYG1fMSMB0o5jgznYW6wr3KsgVgELBJHBmhxdPXg7dLfQKcSgjACX5UFYd532F 1osQ== X-Gm-Message-State: AJIora9Bnb8+a3A0SpvGLG4d+gZcdi/VcYwflmixV5bJNr/Vnwz/Pl7s DGUV0SE0sh7Prx5VKaojdH4= X-Google-Smtp-Source: AGRyM1s7KETEepArG5JL8meMY5y+t8s8chQ/4O2VLPIxoO5+mpzvdwZBLrR6Ib9aQtPfZIU27L5KBg== X-Received: by 2002:a63:234f:0:b0:405:3981:be7 with SMTP id u15-20020a63234f000000b0040539810be7mr742484pgm.15.1655314183940; Wed, 15 Jun 2022 10:29:43 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id w14-20020a1709029a8e00b001676f87473fsm9552244plp.302.2022.06.15.10.29.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jun 2022 10:29:43 -0700 (PDT) From: Yang Shi To: vbabka@suse.cz, kirill.shutemov@linux.intel.com, willy@infradead.org, zokeefe@google.com, linmiaohe@huawei.com, akpm@linux-foundation.org Cc: shy828301@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 7/7] doc: proc: fix the description to THPeligible Date: Wed, 15 Jun 2022 10:29:26 -0700 Message-Id: <20220615172926.546974-8-shy828301@gmail.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20220615172926.546974-1-shy828301@gmail.com> References: <20220615172926.546974-1-shy828301@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655314184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YtollJahbUnQdMIR/aYwe0ysfbXt4BsX5JTH7cAUXak=; b=VwtrbLV9E5cD8Nt1SxXOwUaeQqmMp7bMfqKo0TnUL85U3rYoxaIXSmG59Nyg68DnI+w9sv 9hkzsurUbgyu22ggz0cSd+/ooD/NWEnKgvTAGZLB1986Ad1YPn0lUQSWkqMoJ3bGfQLeZC UbSRNaAMQ04CEogUQOn2sD4DveM4A2U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655314184; a=rsa-sha256; cv=none; b=is1yp7T8ZOLHHbaIDKA53HfAGb4AAqAlfLn5JOe4Ss9qEFnDMkcgUfbSiWksz5wzUWhnP4 OkQCt7aXKCB1kcMnM3HOaorfWcGjA5H8KuOd72OisJhWvjVp/hMr6ef8V6AtSCuTw07LBM aRpEhhUt8oiektn461fn3CG2JyImNqQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=N4NuWQSK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=shy828301@gmail.com Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=N4NuWQSK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: aa81uoqq918m16c4xkwxe3yhark4pnq9 X-Rspamd-Queue-Id: C9F9540091 X-HE-Tag: 1655314184-778921 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The THPeligible bit shows 1 if and only if the VMA is eligible for allocating THP and the THP is also PMD mappable. Some misaligned file VMAs may be eligible for allocating THP but the THP can't be mapped by PMD. Make this more explictly to avoid ambiguity. Signed-off-by: Yang Shi Reviewed-by: Zach O'Keefe --- Documentation/filesystems/proc.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 1bc91fb8c321..a5e41e636a1a 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -514,8 +514,10 @@ replaced by copy-on-write) part of the underlying shmem object out on swap. "SwapPss" shows proportional swap share of this mapping. Unlike "Swap", this does not take into account swapped out page of underlying shmem objects. "Locked" indicates whether the mapping is locked in memory or not. + "THPeligible" indicates whether the mapping is eligible for allocating THP -pages - 1 if true, 0 otherwise. It just shows the current status. +pages as well as the THP is PMD mappable or not - 1 if true, 0 otherwise. +It just shows the current status. "VmFlags" field deserves a separate description. This member represents the kernel flags associated with the particular virtual memory area in two letter