From patchwork Sun Nov 8 14:11:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 11889637 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58F8B139F for ; Sun, 8 Nov 2020 14:14:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 14D8920760 for ; Sun, 8 Nov 2020 14:14:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="crcDJvQ/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 14D8920760 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 418586B0071; Sun, 8 Nov 2020 09:14:07 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3C9386B0073; Sun, 8 Nov 2020 09:14:07 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DE9E6B0074; Sun, 8 Nov 2020 09:14:07 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id 02F486B0071 for ; Sun, 8 Nov 2020 09:14:06 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A5B5D180AD801 for ; Sun, 8 Nov 2020 14:14:06 +0000 (UTC) X-FDA: 77461445292.23.gun52_6310155272e4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 85F7537606 for ; Sun, 8 Nov 2020 14:14:06 +0000 (UTC) X-Spam-Summary: 1,0,0,186cc41a21d8e5ad,d41d8cd98f00b204,songmuchun@bytedance.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2731:2898:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3871:3872:4118:4321:4385:4390:4395:4605:5007:6261:6653:6737:6738:7903:8957:9010:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13161:13229:13894:14110:14181:14394:14721:21080:21444:21451:21627:21990:30054:30069:30070,0,RBL:209.85.210.196:@bytedance.com:.lbl8.mailshell.net-66.100.201.201 62.2.0.100;04yrq8nzw54y3mo1m9rzucn76kukuycrtnubbt3oh63c9k56fr5iyeohf7d96kh.6d17wsb1curf5gmer19u5w4udq6fm8b4yabma7w8srjnk3bjjmd9ok154j4nogz.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:69,LUA_SUMMARY:none X-HE-Tag: gun52_6310155272e4 X-Filterd-Recvd-Size: 7414 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Sun, 8 Nov 2020 14:14:06 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id w6so5583pfu.1 for ; Sun, 08 Nov 2020 06:14:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=E5vsp5cz7nqeZvlEeefM8xntwOmhgYGvqoQNYkr0v5o=; b=crcDJvQ/SQIxjVMdesINeBHr8CEREO4/Sd0UyzZJptxHEuHnpmcD7QCeSGudjmxLXh SCAbltgvlJGxyIT9ELDZ+pRUB+zNZXXwtsI0BxcIq9cA9QTlgxRrkt45uJIM6gEEl/IT H1TNRv2Rs+HGAOswp0PNLJrcipFTGpihCmKjsXWBSYiq6/g0acp5Vip60DqtXgLi8GYv 0rbcD4M+rQggzMYmboEtA63ObrfNkKCLFA0WQvSHceGOaOY1Vab7jqwWfpVM61p3W2bJ 1xPJGMxkvpG4eFIxBzWbCpD5td6aN08rdy9mMD7847D1LzViKbqvWnbxhySlMwX+1ydI pOUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=E5vsp5cz7nqeZvlEeefM8xntwOmhgYGvqoQNYkr0v5o=; b=anOVIQqhDQowbtIMcTyWizu5SpdrRhx/iaBoDtA9XMbOsqm+FkuRgBJ1sg8Oa0xZXe IJsaQKdu+SFYGwKTeesU8baJUSlQI4XgKb+8rhXKHtlMlhMhloGlXZ1sImNr3KvrJbvb RBl/9nXqnZU4XM7pE6VVB+oxWxVlsh9oG1bgi9PFnAkg/6N5Lc501y1iGMVzsEix3ey8 7Y8waodOKKRch2DRzRHs917vPg6Zjdt0vJO6eeuiMcQFKYbsLZzmOhMrsZm6ee9ngEV3 kLuxuMdGSaQ1/GnSw67jubAFdhc9PTnnunl7cGSYHXVcSPIrL8drLtGwbjJ7asinrXrf m7AA== X-Gm-Message-State: AOAM533lBAWeXrZo5wQlfEKdhdOXMELeBoIgmQ9wpkOqhD/DSU3HOhm9 wfXAf2T71BkmeSRKuIHtgmJEMg== X-Google-Smtp-Source: ABdhPJyHSdHNLyD+JvJJ9n4ojdMe2eNz7H1OMTUv8Ux8jiTESj6P6wy2aULkhzIVqV2ACIhNbNGlsw== X-Received: by 2002:a62:f245:0:b029:18b:df86:191c with SMTP id y5-20020a62f2450000b029018bdf86191cmr4765153pfl.35.1604844845284; Sun, 08 Nov 2020 06:14:05 -0800 (PST) Received: from localhost.localdomain ([103.136.220.94]) by smtp.gmail.com with ESMTPSA id z11sm8754047pfk.52.2020.11.08.06.13.54 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 08 Nov 2020 06:14:04 -0800 (PST) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org, paulmck@kernel.org, mchehab+huawei@kernel.org, pawan.kumar.gupta@linux.intel.com, rdunlap@infradead.org, oneukum@suse.com, anshuman.khandual@arm.com, jroedel@suse.de, almasrymina@google.com, rientjes@google.com, willy@infradead.org, osalvador@suse.de, mhocko@suse.com Cc: duanxiongchun@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Muchun Song Subject: [PATCH v3 14/21] mm/hugetlb: Support freeing vmemmap pages of gigantic page Date: Sun, 8 Nov 2020 22:11:06 +0800 Message-Id: <20201108141113.65450-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20201108141113.65450-1-songmuchun@bytedance.com> References: <20201108141113.65450-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The gigantic page is allocated by bootmem, if we want to free the unused vmemmap pages. We also should allocate the page table. So we also allocate page tables from bootmem. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 3 +++ mm/hugetlb.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index afb9b18771c4..f8ca4d251aa8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -506,6 +506,9 @@ struct hstate { struct huge_bootmem_page { struct list_head list; struct hstate *hstate; +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP + pte_t *vmemmap_pte; +#endif }; struct page *alloc_huge_page(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9b1ac52d9fdd..ec0d33d2c426 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1419,6 +1419,62 @@ static void vmemmap_pgtable_free(struct hstate *h, struct page *page) pte_free_kernel(&init_mm, page_to_virt(pgtable)); } +static unsigned long __init gather_vmemmap_pgtable_prealloc(void) +{ + struct huge_bootmem_page *m, *tmp; + unsigned long nr_free = 0; + + list_for_each_entry_safe(m, tmp, &huge_boot_pages, list) { + struct hstate *h = m->hstate; + unsigned int nr = pgtable_pages_to_prealloc_per_hpage(h); + unsigned int pgtable_size; + + if (!nr) + continue; + + pgtable_size = nr << PAGE_SHIFT; + m->vmemmap_pte = memblock_alloc_try_nid(pgtable_size, + PAGE_SIZE, 0, MEMBLOCK_ALLOC_ACCESSIBLE, + NUMA_NO_NODE); + if (!m->vmemmap_pte) { + nr_free++; + list_del(&m->list); + memblock_free_early(__pa(m), huge_page_size(h)); + } + } + + return nr_free; +} + +static void __init gather_vmemmap_pgtable_init(struct huge_bootmem_page *m, + struct page *page) +{ + int i; + struct hstate *h = m->hstate; + unsigned long pte = (unsigned long)m->vmemmap_pte; + unsigned int nr = pgtable_pages_to_prealloc_per_hpage(h); + + if (!nr) + return; + + vmemmap_pgtable_init(page); + + for (i = 0; i < nr; i++, pte += PAGE_SIZE) { + pgtable_t pgtable = virt_to_page(pte); + + __ClearPageReserved(pgtable); + vmemmap_pgtable_deposit(page, pgtable); + } + + /* + * If we had gigantic hugepages allocated at boot time, we need + * to restore the 'stolen' pages to totalram_pages in order to + * fix confusing memory reports from free(1) and another + * side-effects, like CommitLimit going negative. + */ + adjust_managed_page_count(page, nr); +} + static void __init hugetlb_vmemmap_init(struct hstate *h) { unsigned int order = huge_page_order(h); @@ -1752,6 +1808,16 @@ static inline void vmemmap_pgtable_free(struct hstate *h, struct page *page) { } +static inline unsigned long gather_vmemmap_pgtable_prealloc(void) +{ + return 0; +} + +static inline void gather_vmemmap_pgtable_init(struct huge_bootmem_page *m, + struct page *page) +{ +} + static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) { } @@ -3013,6 +3079,7 @@ static void __init gather_bootmem_prealloc(void) WARN_ON(page_count(page) != 1); prep_compound_huge_page(page, h->order); WARN_ON(PageReserved(page)); + gather_vmemmap_pgtable_init(m, page); prep_new_huge_page(h, page, page_to_nid(page)); put_page(page); /* free it into the hugepage allocator */ @@ -3065,6 +3132,10 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) break; cond_resched(); } + + if (hstate_is_gigantic(h)) + i -= gather_vmemmap_pgtable_prealloc(); + if (i < h->max_huge_pages) { char buf[32];