From patchwork Fri Nov 13 10:59:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 11902981 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A22E7138B for ; Fri, 13 Nov 2020 11:01:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 54C10207DE for ; Fri, 13 Nov 2020 11:01:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="WHRw8PZC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 54C10207DE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 747FF6B00AB; Fri, 13 Nov 2020 06:01:54 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6F72C6B00AC; Fri, 13 Nov 2020 06:01:54 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 548866B00AD; Fri, 13 Nov 2020 06:01:54 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 1E78F6B00AB for ; Fri, 13 Nov 2020 06:01:54 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id BD36B824999B for ; Fri, 13 Nov 2020 11:01:53 +0000 (UTC) X-FDA: 77479104906.09.alley29_4711d022730e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id A4F8A180AD801 for ; Fri, 13 Nov 2020 11:01:53 +0000 (UTC) X-Spam-Summary: 1,0,0,2e438ff6f85b8a02,d41d8cd98f00b204,songmuchun@bytedance.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2731:3138:3139:3140:3141:3142:3354:3622:3865:3867:3868:3871:4118:4250:4321:4385:4390:4395:5007:6119:6261:6653:6737:6738:7875:7903:9010:9036:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13894:14096:14110:14181:14394:14721:21080:21444:21451:21627:21990:30054,0,RBL:209.85.210.196:@bytedance.com:.lbl8.mailshell.net-62.2.0.100 66.100.201.201;04y8yueh1nisdbb8pm1y8g38rtfiyypm83ce1tomgb4667da5zrbqonme39g64q.pmzuoswfcn18q3pecbmpbjmq96z77bduaejg4asa5m6kxdd7c5r7as3ych8sff5.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:70,LUA_SUMMARY:none X-HE-Tag: alley29_4711d022730e X-Filterd-Recvd-Size: 7387 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Fri, 13 Nov 2020 11:01:52 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id y7so7302260pfq.11 for ; Fri, 13 Nov 2020 03:01:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QgKZ0gBbzr5Y1R6IbjL9V67vQkRA/Xenmaet0eZn1jI=; b=WHRw8PZCdlUW+Ct60o/wJTbtCRkWymC3+ozO2o4xnhtWd3zzxeCSnf30bccZp6xzy4 +tTcOs3I6WX/N6lD5i1sLgTmz/5B/SWP+MWQx04YC4Ha6Uy3+iUH7O79oBW5DZ1Clmko z4M5Yj3N+ZfOoHLU3AoUO+kMMaJ+QIXea+8RvHlVVCgk5InzQPNnuy8+6KsqC7vOdwWW jny1Ho5Br8/nO5IRAWJGLslGttCNvapDZQkAkdl/MbZZJLnhI6eK7kRUgkqpJIG5zPNy M3J2ojBtrl9u09K4ppggcLSNf7lZH7LUTwKtvcqlLLBqVjdGyTgxbmC503UhoiEq63CS enQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QgKZ0gBbzr5Y1R6IbjL9V67vQkRA/Xenmaet0eZn1jI=; b=gYRbqDqEwp3xFA/IszXVbgsduach0sre8SbMIVqDV1s08k1s1kZtZ3cuO+6a/Xjer3 25bRD4AHNF4dipVhYbwnhefPbiTouwxghZJjuEg/pdSQUj7R+OFlRH4yb7E/4OgoDhco bQJf+P+Hcv5rEGdVB0hKI1FE5VTGJtrzNbnTX8Vty+R9YK57n7n3p0x9GQaOx/kAS9IQ yaTddj8sMXGjwz5xKugKBDrKN7lpn/F9LIMzu/yS5ZKs4tOC9/LhqSvoMUjYd0ytykVI PUAJiSi1/iPOrC45IvtEiILu8zH5N1bdBzrZSOBW+xzpxSTbgoP5be88TXjcu+e1aidk 54qg== X-Gm-Message-State: AOAM532hGLTfQU9239EfGA52fC7q13hR4kZer6N5lZjNzflPtgr/iQ/F u3lR/RsNfKi/s5ukC5teS4NPww== X-Google-Smtp-Source: ABdhPJwY3zGp6zFuIGwwhXgYIEFo1bUv9jDKGGKu4tB/Cr++GdoMqEUxYD2qfW+e+Lja4kSHIAKnyA== X-Received: by 2002:a17:90a:4215:: with SMTP id o21mr2307679pjg.166.1605265312084; Fri, 13 Nov 2020 03:01:52 -0800 (PST) Received: from localhost.localdomain ([61.120.150.78]) by smtp.gmail.com with ESMTPSA id f1sm8909959pfc.56.2020.11.13.03.01.40 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Nov 2020 03:01:51 -0800 (PST) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org, paulmck@kernel.org, mchehab+huawei@kernel.org, pawan.kumar.gupta@linux.intel.com, rdunlap@infradead.org, oneukum@suse.com, anshuman.khandual@arm.com, jroedel@suse.de, almasrymina@google.com, rientjes@google.com, willy@infradead.org, osalvador@suse.de, mhocko@suse.com Cc: duanxiongchun@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Muchun Song Subject: [PATCH v4 05/21] mm/hugetlb: Introduce pgtable allocation/freeing helpers Date: Fri, 13 Nov 2020 18:59:36 +0800 Message-Id: <20201113105952.11638-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20201113105952.11638-1-songmuchun@bytedance.com> References: <20201113105952.11638-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On x86_64, vmemmap is always PMD mapped if the machine has hugepages support and if we have 2MB contiguos pages and PMD aligned. If we want to free the unused vmemmap pages, we have to split the huge pmd firstly. So we should pre-allocate pgtable to split PMD to PTE. Signed-off-by: Muchun Song Acked-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.h | 12 +++++++++ 2 files changed, 85 insertions(+) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a6c9948302e2..b7dfa97b4ea9 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -71,6 +71,8 @@ */ #define pr_fmt(fmt) "HugeTLB Vmemmap: " fmt +#include +#include #include "hugetlb_vmemmap.h" /* @@ -83,6 +85,77 @@ */ #define RESERVE_VMEMMAP_NR 2U +#ifndef VMEMMAP_HPAGE_SHIFT +#define VMEMMAP_HPAGE_SHIFT HPAGE_SHIFT +#endif +#define VMEMMAP_HPAGE_ORDER (VMEMMAP_HPAGE_SHIFT - PAGE_SHIFT) +#define VMEMMAP_HPAGE_NR (1 << VMEMMAP_HPAGE_ORDER) +#define VMEMMAP_HPAGE_SIZE ((1UL) << VMEMMAP_HPAGE_SHIFT) +#define VMEMMAP_HPAGE_MASK (~(VMEMMAP_HPAGE_SIZE - 1)) + +#define page_huge_pte(page) ((page)->pmd_huge_pte) + +static inline unsigned int free_vmemmap_pages_per_hpage(struct hstate *h) +{ + return h->nr_free_vmemmap_pages; +} + +static inline unsigned int vmemmap_pages_per_hpage(struct hstate *h) +{ + return free_vmemmap_pages_per_hpage(h) + RESERVE_VMEMMAP_NR; +} + +static inline unsigned long vmemmap_pages_size_per_hpage(struct hstate *h) +{ + return (unsigned long)vmemmap_pages_per_hpage(h) << PAGE_SHIFT; +} + +static inline unsigned int pgtable_pages_to_prealloc_per_hpage(struct hstate *h) +{ + unsigned long vmemmap_size = vmemmap_pages_size_per_hpage(h); + + /* + * No need pre-allocate page tables when there is no vmemmap pages + * to free. + */ + if (!free_vmemmap_pages_per_hpage(h)) + return 0; + + return ALIGN(vmemmap_size, VMEMMAP_HPAGE_SIZE) >> VMEMMAP_HPAGE_SHIFT; +} + +void vmemmap_pgtable_free(struct page *page) +{ + struct page *pte_page, *t_page; + + list_for_each_entry_safe(pte_page, t_page, &page->lru, lru) { + list_del(&pte_page->lru); + pte_free_kernel(&init_mm, page_to_virt(pte_page)); + } +} + +int vmemmap_pgtable_prealloc(struct hstate *h, struct page *page) +{ + unsigned int nr = pgtable_pages_to_prealloc_per_hpage(h); + + /* Store preallocated pages on huge page lru list */ + INIT_LIST_HEAD(&page->lru); + + while (nr--) { + pte_t *pte_p; + + pte_p = pte_alloc_one_kernel(&init_mm); + if (!pte_p) + goto out; + list_add(&virt_to_page(pte_p)->lru, &page->lru); + } + + return 0; +out: + vmemmap_pgtable_free(page); + return -ENOMEM; +} + void __init hugetlb_vmemmap_init(struct hstate *h) { unsigned int order = huge_page_order(h); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 40c0c7dfb60d..2a72d2f62411 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,12 +9,24 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include +#include #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP void __init hugetlb_vmemmap_init(struct hstate *h); +int vmemmap_pgtable_prealloc(struct hstate *h, struct page *page); +void vmemmap_pgtable_free(struct page *page); #else static inline void hugetlb_vmemmap_init(struct hstate *h) { } + +static inline int vmemmap_pgtable_prealloc(struct hstate *h, struct page *page) +{ + return 0; +} + +static inline void vmemmap_pgtable_free(struct page *page) +{ +} #endif /* CONFIG_HUGETLB_PAGE_FREE_VMEMMAP */ #endif /* _LINUX_HUGETLB_VMEMMAP_H */