From patchwork Mon Oct 26 14:51:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 11857485 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C0D706A2 for ; Mon, 26 Oct 2020 14:55:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 721BA239D2 for ; Mon, 26 Oct 2020 14:55:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="JPX/rAC3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 721BA239D2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9FEAF6B0085; Mon, 26 Oct 2020 10:55:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B1346B0087; Mon, 26 Oct 2020 10:55:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 827606B0088; Mon, 26 Oct 2020 10:55:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0126.hostedemail.com [216.40.44.126]) by kanga.kvack.org (Postfix) with ESMTP id 5734D6B0085 for ; Mon, 26 Oct 2020 10:55:07 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E47D21EE6 for ; Mon, 26 Oct 2020 14:55:06 +0000 (UTC) X-FDA: 77414374212.19.quilt26_010121327274 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id BEB861AD1B0 for ; Mon, 26 Oct 2020 14:55:06 +0000 (UTC) X-Spam-Summary: 1,0,0,69a7bdfff49a7ae2,d41d8cd98f00b204,songmuchun@bytedance.com,,RULES_HIT:41:355:379:541:800:960:965:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2731:2898:3138:3139:3140:3141:3142:3353:3865:3867:3871:3872:4117:4321:4385:4390:4395:4605:5007:6261:6653:6737:6738:7903:8957:9010:10004:11026:11473:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:14110:14181:14394:14721:21080:21444:21451:21627:21990:30054:30070,0,RBL:209.85.216.65:@bytedance.com:.lbl8.mailshell.net-62.2.0.100 66.100.201.201;04yg7h4i5irw87kgjrrw7z8xa8dorocyc7kjb6xjrhw41gaqh7a3rsarjq76b3e.dwgwu1p1wine9b664w8jxtbgef561wucx5u84crdjj8bb9dp3zx6hxj1ddxnjxr.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: quilt26_010121327274 X-Filterd-Recvd-Size: 6874 Received: from mail-pj1-f65.google.com (mail-pj1-f65.google.com [209.85.216.65]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Mon, 26 Oct 2020 14:55:06 +0000 (UTC) Received: by mail-pj1-f65.google.com with SMTP id a17so3378415pju.1 for ; Mon, 26 Oct 2020 07:55:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=78xBg2WQCdWNruCWqKGQQsJIrYM1oGRx7tRHfiiJXdU=; b=JPX/rAC3IF6SKYvKDugHvIsCP8mDgq0zMSJnz5AUn9TDccBnPZFBTTNVW19xJK8PQ3 tlVk5jFoLW/gh8CaaLoJO9uJ/Mnl14McW/wMP8MU7r0LeBbJV4DRCZYE4o/h+VC8hAyc KkfSsvj4ce+LbKGniw4RqRkGoG5KUyRX+PbAFM12uFtS41ti+6zq3eKGPVthrdK+Lan0 amcr6IRMrKPMMmmYsDHfSDIK4gCZ0u1LkVkLoZBTc9kHaPEtohVsiaq2WWOZImlDMjYP aVRruxY7rMZ6D+a33oOg4qNcP5BBhIlkTnQ5b3pacXRyVY7XfQKP1w4ls10xQoktF2NT 0wgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=78xBg2WQCdWNruCWqKGQQsJIrYM1oGRx7tRHfiiJXdU=; b=Pse8hvBKiiK6katb+wrZoSAFk6z7byeu9/zNAvFkH16j3uMH1K+5aT0xmG8IgmDOKo ysFT/6P7Ul6SHZNhGHuN7IK4r05M+T1v1zoTz6fwuw8zqzncq1gHSq7hzH2Vw4hKgyRJ VAVf/kIzkT18lAEiAuQ+JMEKSgJP9Pz2xfYImRnBrm+3Hm7uO17/R6KysAL7Y/kJHMhN 3OzuWQwmfg02ywq1gG7VwB8ptxz9JlIDYujSXAhET+mobAmB6tPvmdFHIuG7ufvJew8k Hhsb1gGyRg8Gyh3RK3MjHSnUuhXPF0B7t5DuGSDhvqEgsMdiedoACdKyfgIBzcDbbW9V hmzg== X-Gm-Message-State: AOAM5328AGJ0z7y7/3Fr+2YuxZ5NBHfCr+ApiJs/k2hEAF/JecuA9FDE /55B1mfoQReQcmGgLKk964PBpA== X-Google-Smtp-Source: ABdhPJyk16ToaVS77jraO6/oXaFjUrRguY4X1F5jVMSbJnR3yqUDKHckul5kXm7Xbd7MSxOXEjmiKw== X-Received: by 2002:a17:90a:6683:: with SMTP id m3mr21108784pjj.225.1603724105538; Mon, 26 Oct 2020 07:55:05 -0700 (PDT) Received: from localhost.localdomain ([103.136.220.89]) by smtp.gmail.com with ESMTPSA id x123sm12042726pfb.212.2020.10.26.07.54.56 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 26 Oct 2020 07:55:04 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org, paulmck@kernel.org, mchehab+huawei@kernel.org, pawan.kumar.gupta@linux.intel.com, rdunlap@infradead.org, oneukum@suse.com, anshuman.khandual@arm.com, jroedel@suse.de, almasrymina@google.com, rientjes@google.com, willy@infradead.org Cc: duanxiongchun@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Muchun Song Subject: [PATCH v2 12/19] mm/hugetlb: Support freeing vmemmap pages of gigantic page Date: Mon, 26 Oct 2020 22:51:07 +0800 Message-Id: <20201026145114.59424-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20201026145114.59424-1-songmuchun@bytedance.com> References: <20201026145114.59424-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The gigantic page is allocated by bootmem, if we want to free the unused vmemmap pages. We also should allocate the page table. So we also allocate page tables from bootmem. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 3 +++ mm/hugetlb.c | 57 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 919f47d77117..695d3041ae7d 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -506,6 +506,9 @@ struct hstate { struct huge_bootmem_page { struct list_head list; struct hstate *hstate; +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP + pte_t *vmemmap_pgtable; +#endif }; struct page *alloc_huge_page(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f75b93fb4c07..d98b55ad1a90 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1410,6 +1410,48 @@ static inline void vmemmap_pgtable_free(struct hstate *h, struct page *page) pte_free_kernel(&init_mm, pte_p); } +static unsigned long __init gather_vmemmap_pgtable_prealloc(void) +{ + struct huge_bootmem_page *m, *tmp; + unsigned long nr_free = 0; + + list_for_each_entry_safe(m, tmp, &huge_boot_pages, list) { + struct hstate *h = m->hstate; + unsigned int pgtable_size = nr_pgtable(h) << PAGE_SHIFT; + + if (!pgtable_size) + continue; + + m->vmemmap_pgtable = memblock_alloc_try_nid(pgtable_size, + PAGE_SIZE, 0, MEMBLOCK_ALLOC_ACCESSIBLE, + NUMA_NO_NODE); + if (!m->vmemmap_pgtable) { + nr_free++; + list_del(&m->list); + memblock_free_early(__pa(m), huge_page_size(h)); + } + } + + return nr_free; +} + +static void __init gather_vmemmap_pgtable_init(struct huge_bootmem_page *m, + struct page *page) +{ + int i; + struct hstate *h = m->hstate; + unsigned long pgtable = (unsigned long)m->vmemmap_pgtable; + unsigned int nr = nr_pgtable(h); + + if (!nr) + return; + + vmemmap_pgtable_init(page); + + for (i = 0; i < nr; i++, pgtable += PAGE_SIZE) + vmemmap_pgtable_deposit(page, (pte_t *)pgtable); +} + static void __init hugetlb_vmemmap_init(struct hstate *h) { unsigned int order = huge_page_order(h); @@ -1778,6 +1820,16 @@ static inline void vmemmap_pgtable_free(struct hstate *h, struct page *page) { } +static inline unsigned long gather_vmemmap_pgtable_prealloc(void) +{ + return 0; +} + +static inline void gather_vmemmap_pgtable_init(struct huge_bootmem_page *m, + struct page *page) +{ +} + static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) { } @@ -3039,6 +3091,7 @@ static void __init gather_bootmem_prealloc(void) WARN_ON(page_count(page) != 1); prep_compound_huge_page(page, h->order); WARN_ON(PageReserved(page)); + gather_vmemmap_pgtable_init(m, page); prep_new_huge_page(h, page, page_to_nid(page)); put_page(page); /* free it into the hugepage allocator */ @@ -3091,6 +3144,10 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) break; cond_resched(); } + + if (hstate_is_gigantic(h)) + i -= gather_vmemmap_pgtable_prealloc(); + if (i < h->max_huge_pages) { char buf[32];