From patchwork Thu Mar 31 06:56:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12796785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15844C433EF for ; Thu, 31 Mar 2022 06:58:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=iwMHBiFRZoWHCdhf7tFSZxV5qp53thRzTX+Jo/olEAQ=; b=2vUW9iK1pPyO4g KS8d7lDs1IQoBUJoQEq9PGr1mSSdL2LsiQ6nHAEriEscDENktgY5kuUyzjFQyCc6/xJn0faEUMbqw zqxUsQjR49sNcYNR2LZyXOtobXb00+8zYRfiElzHvKxSod2RIXiue/3Tdxykm7XXOZc/791dL45gD hm/JzdQcosQr+1BLRSOu5fUB9Y1CO+mHSiY0pEoNS0WhTAGeFSHKPOxPvD066ux58ZZ0bsYcisLJz IIwug3gedPt+VTZ2puOuyqlqy1LpY6akw4Rdxs8tdJM/W/gCK/aa7AYNJs4ZwzkGFq1By+3h5XOKe FgVxUpdti4+w3fS360jA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nZokP-000xi1-Ao; Thu, 31 Mar 2022 06:57:33 +0000 Received: from mail-pj1-x102d.google.com ([2607:f8b0:4864:20::102d]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nZokL-000xhG-CG for linux-arm-kernel@lists.infradead.org; Thu, 31 Mar 2022 06:57:31 +0000 Received: by mail-pj1-x102d.google.com with SMTP id h23-20020a17090a051700b001c9c1dd3acbso2707290pjh.3 for ; Wed, 30 Mar 2022 23:57:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=I5exx/RB+d0Ig7pXuqKbMzRv7rQoprfE6EMWAxPn3vk=; b=VxsWEIotkC2VYiWWzOCyD1h39EI03hJn4gYoUq5zL+es6eKEi/fQPJEpTFiQhX51V9 eikNF5Q4nJX9kTCRRilvUSCkBYeRXTclsDYJrtTV3K7xEbiRG2R9cRz5ththfznMM3XC 66QKDnh3Cj3JDimXFfT7EUKNWhHjsObnnQynBYRDAdMdFdKt8UOEDbHAWZejxE0xs4rx snTFc7AtACXs3vBUkU+J5RON6gZUTOjKnm6p9ZnUqRBfbWaVAiLtTSaP8GfA32r9OqOS f7tm7oIJpihmrGFTzDO6n6HNvqTowM06YPD5GLVOL5OO1WiLcWlYeA+WAx20Xiu72BDc B/jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=I5exx/RB+d0Ig7pXuqKbMzRv7rQoprfE6EMWAxPn3vk=; b=2R57zRhW27TkXsHI9ZKbmDuvdUS9LTiebHllHq7hDspkrwg4SJ2J1GU410cLzOGtj2 QZ8ABRx2X29dsfH4EmP4asVynprCxGJQto9/TjfswWQcK05Dr9BRz5VrswBQUDsj2B9R hmUEciF4jQnMXzK3878Xdl7GOKbsN3Z/nyvuL9/bZSevPBCrQuVIJCiGuLjYMM3PeVZa wjpDIC3xV2ZPad5Ewe3wePgMkryzi1TTIV/Ph1nXwDiwLBpH5L9o6t+yjwP37o8+gb1Y skL9UTACqhqW8SW6U49yTEkYmFvQD4WaVAdpUkdkD1Z3j+Fg0QGQGxoxVVsUPRg305+8 ZP8A== X-Gm-Message-State: AOAM532Pt/bLKRU4ZnwbWLU96i0r3VZMe+3WhV1NSR2zNxCcQDyEAXLC csVvKC0cHidTgM6IQqTz6bu//w== X-Google-Smtp-Source: ABdhPJxVeMek9ygr1gEJ2F2+WMGJOZmPNv0PKkIoA3/xhuB910rYtwVdeKFiPVghs/O8sDNlYZOp5w== X-Received: by 2002:a17:90b:17c7:b0:1c7:c616:6eb0 with SMTP id me7-20020a17090b17c700b001c7c6166eb0mr4632866pjb.144.1648709847477; Wed, 30 Mar 2022 23:57:27 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.239]) by smtp.gmail.com with ESMTPSA id d8-20020a056a00198800b004fab740dbe6sm26169846pfl.15.2022.03.30.23.57.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Mar 2022 23:57:27 -0700 (PDT) From: Muchun Song To: will@kernel.org, akpm@linux-foundation.org, david@redhat.com, bodeddub@amazon.com, osalvador@suse.de, mike.kravetz@oracle.com, rientjes@google.com, mark.rutland@arm.com, catalin.marinas@arm.com, james.morse@arm.com, 21cnbao@gmail.com Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v4 1/2] mm: hugetlb_vmemmap: introduce ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP Date: Thu, 31 Mar 2022 14:56:39 +0800 Message-Id: <20220331065640.5777-1-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220330_235729_467560_65026AC2 X-CRM114-Status: GOOD ( 11.88 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The feature of minimizing overhead of struct page associated with each HugeTLB page is implemented on x86_64, however, the infrastructure of this feature is already there, we could easily enable it for other architectures. Introduce ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP for other architectures to be easily enabled. Just select this config if they want to enable this feature. Suggested-by: Andrew Morton Signed-off-by: Muchun Song Reviewed-by: Anshuman Khandual --- arch/x86/Kconfig | 1 + fs/Kconfig | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 9f5bd41bf660..e69d42528542 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -119,6 +119,7 @@ config X86 select ARCH_WANTS_DYNAMIC_TASK_STRUCT select ARCH_WANTS_NO_INSTR select ARCH_WANT_HUGE_PMD_SHARE + select ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH diff --git a/fs/Kconfig b/fs/Kconfig index 6c7dc1387beb..f6db2af33738 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -245,9 +245,17 @@ config HUGETLBFS config HUGETLB_PAGE def_bool HUGETLBFS +# +# Select this config option from the architecture Kconfig, if it is preferred +# to enable the feature of minimizing overhead of struct page associated with +# each HugeTLB page. +# +config ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP + bool + config HUGETLB_PAGE_FREE_VMEMMAP def_bool HUGETLB_PAGE - depends on X86_64 + depends on ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON From patchwork Thu Mar 31 06:56:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12796786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5156AC433EF for ; Thu, 31 Mar 2022 06:58:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DtFvnastHFTu5QVClRodFfugj5Lqtl3dXvHegofpb3Q=; b=11d3kr6BhtpENW EmwL7LXPZA7GtB0JZgvCOfE9obelKgD7eFG6Xl14fpY/QMcVDGftrKFjcqVQntJN7EDDJQXULUmLw gykTx9byFT5t44TeXqPmEvd9vYEjdQ/2L5AKpvWqlS2zS/qU0ryzfTJ09CnUS5KW+H5+PWZz1ykPO z1CsYpspF3CqpUnjLWkNahratW94iJiKDxvbET9I+UhvH7AVIIxL9NoSci72m1r4GiTKahB/wMSEp q4ZlLwdiK0Mjq5LTS0NNszL+9xfJbEQD7ZW70a+bV03FlBxI1A/T1zKeRCgfJQRbnLcJyDW5s7axo RC2N6ihApj8cHcqY+bmA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nZokb-000xmu-BV; Thu, 31 Mar 2022 06:57:45 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nZokQ-000xiU-Nb for linux-arm-kernel@lists.infradead.org; Thu, 31 Mar 2022 06:57:37 +0000 Received: by mail-pl1-x62e.google.com with SMTP id p17so22573525plo.9 for ; Wed, 30 Mar 2022 23:57:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nwV4Ab67S/nV7/+Z8UUB2Pccmb87dn/jh1LTq8DOevs=; b=3uldU/51NiLauuIFjpb8KoMXAEQzooUmtBM+5WNgq+HOASj1ysBKKqLdUxcDZjII8e sW2SGz1PXTfYxU6uQ7qpHxFfVjbulGH/3t60h5zfmNUs9ixbyG6AKyvWJdsftzh+ZPdg +GRF37IXTqklfzEPR3VIC7YJM2amiEo3yQ8yXSwGmeK4SrMvwhT5vI1SYsFpwiK1B1EX HVP3BlTl5czOhwHnW4HTZNojmPKyolAs1gzDa6MaSkSR4/MD2DEg5xI9i1BmX9lk7GWi xkb2rPqLbFw7NcgZCgwSXvrE+Xr94sNaXh6vjCfNL1D3JWxpMfxqAP/Qj2c9L6M7RLLx 8Jew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nwV4Ab67S/nV7/+Z8UUB2Pccmb87dn/jh1LTq8DOevs=; b=Y0XaB2LhBgEIMckj6SvDc4B0qnociNZKkYT7H1+Zv5hWicMSmJnXGfG02cm12VbUuk LcNWWzkpw7sQ1imqIU84mkNG/LqueNBPGbwHzAe96OfUc/AqSiTLRxaEK6rmWxcwuc2d M5fMunBoyzk1bIdrGTxXqqFRsLSdHy0TIek4GkXT7MdIJ32beDKyEbeTagFcxhswbGOo XzTTAtdlqSQ+7GfywUz/l0QB8dSj1Za1fuuM7+f2/wsDUtqCiVurpQ3t42hoFrQT5BlX ErYOdSzU1Af0z7NNaQTRTb9DAOwI/YRwmaaMmsAp9BcZ0x4St5sk+r35Pn4v6Eq9oILD XX8Q== X-Gm-Message-State: AOAM531a7MvJcncob/XfDv8FINis9zS4BTd2JvYYhSxcNIP7yStrj1pG 4upBghyWm9taItiwJUYsnXsguA== X-Google-Smtp-Source: ABdhPJxQIefmzGUPyc50+2URUuiATPTUI2ellpZWz0sJIkkNfeQbGulC4f17oKELkil+hJwnkmQerQ== X-Received: by 2002:a17:902:e80b:b0:155:c75a:42f with SMTP id u11-20020a170902e80b00b00155c75a042fmr3560701plg.67.1648709853513; Wed, 30 Mar 2022 23:57:33 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.239]) by smtp.gmail.com with ESMTPSA id d8-20020a056a00198800b004fab740dbe6sm26169846pfl.15.2022.03.30.23.57.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Mar 2022 23:57:33 -0700 (PDT) From: Muchun Song To: will@kernel.org, akpm@linux-foundation.org, david@redhat.com, bodeddub@amazon.com, osalvador@suse.de, mike.kravetz@oracle.com, rientjes@google.com, mark.rutland@arm.com, catalin.marinas@arm.com, james.morse@arm.com, 21cnbao@gmail.com Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v4 2/2] arm64: mm: hugetlb: Enable HUGETLB_PAGE_FREE_VMEMMAP for arm64 Date: Thu, 31 Mar 2022 14:56:40 +0800 Message-Id: <20220331065640.5777-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220331065640.5777-1-songmuchun@bytedance.com> References: <20220331065640.5777-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220330_235734_880502_C3335D68 X-CRM114-Status: GOOD ( 23.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The feature of minimizing overhead of struct page associated with each HugeTLB page aims to free its vmemmap pages (used as struct page) to save memory, where is ~14GB/16GB per 1TB HugeTLB pages (2MB/1GB type). In short, when a HugeTLB page is allocated or freed, the vmemmap array representing the range associated with the page will need to be remapped. When a page is allocated, vmemmap pages are freed after remapping. When a page is freed, previously discarded vmemmap pages must be allocated before remapping. More implementations and details can be found here [1]. The infrastructure of freeing vmemmap pages associated with each HugeTLB page is already there, we can easily enable HUGETLB_PAGE_FREE_VMEMMAP for arm64, the only thing to be fixed is flush_dcache_page() . flush_dcache_page() need to be adapted to operate on the head page's flags since the tail vmemmap pages are mapped with read-only after the feature is enabled (clear operation is not permitted). There was some discussions about this in the thread [2], but there was no conclusion in the end. And I copied the concern proposed by Anshuman to here and explain why those concern is superfluous. It is safe to enable it for x86_64 as well as arm64. 1st concern: ''' But what happens when a hot remove section's vmemmap area (which is being teared down) is nearby another vmemmap area which is either created or being destroyed for HugeTLB alloc/free purpose. As you mentioned HugeTLB pages inside the hot remove section might be safe. But what about other HugeTLB areas whose vmemmap area shares page table entries with vmemmap entries for a section being hot removed ? Massive HugeTLB alloc /use/free test cycle using memory just adjacent to a memory hotplug area, which is always added and removed periodically, should be able to expose this problem. ''' Answer: At the time memory is removed, all HugeTLB pages either have been migrated away or dissolved. So there is no race between memory hot remove and free_huge_page_vmemmap(). Therefore, HugeTLB pages inside the hot remove section is safe. Let's talk your question "what about other HugeTLB areas whose vmemmap area shares page table entries with vmemmap entries for a section being hot removed ?", the question is not established. The minimal granularity size of hotplug memory 128MB (on arm64, 4k base page), any HugeTLB smaller than 128MB is within a section, then, there is no share PTE page tables between HugeTLB in this section and ones in other sections and a HugeTLB page could not cross two sections. In this case, the section cannot be freed. Any HugeTLB bigger than 128MB (section size) whose vmemmap pages is an integer multiple of 2MB (PMD-mapped). As long as: 1) HugeTLBs are naturally aligned, power-of-two sizes 2) The HugeTLB size >= the section size 3) The HugeTLB size >= the vmemmap leaf mapping size Then a HugeTLB will not share any leaf page table entries with *anything else*, but will share intermediate entries. In this case, at the time memory is removed, all HugeTLB pages either have been migrated away or dissolved. So there is also no race between memory hot remove and free_huge_page_vmemmap(). 2nd concern: ''' differently, not sure if ptdump would require any synchronization. Dumping an wrong value is probably okay but crashing because a page table entry is being freed after ptdump acquired the pointer is bad. On arm64, ptdump() is protected against hotremove via [get|put]_online_mems(). ''' Answer: The ptdump should be fine since vmemmap_remap_free() only exchanges PTEs or splits the PMD entry (which means allocating a PTE page table). Both operations do not free any page tables (PTE), so ptdump cannot run into a UAF on any page tables. The worst case is just dumping an wrong value. [1] https://lore.kernel.org/all/20210510030027.56044-1-songmuchun@bytedance.com/ [2] https://lore.kernel.org/all/20210518091826.36937-1-songmuchun@bytedance.com/ Signed-off-by: Muchun Song Reviewed-by: Barry Song Reviewed-by: Anshuman Khandual --- v4: - Introduce ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP (implemented in the previous patch) to enable this feature for arm64. v3: - Rework patch's subject. - Clarify the feature of HUGETLB_PAGE_FREE_VMEMMAP is already there in the current code and easyly be enabled for arm64 into commit log. - Add hugetlb_free_vmemmap_enabled() check into flush_dcache_page(). Thanks for Barry's suggestions. v2: - Update commit message (Mark Rutland). - Fix flush_dcache_page(). arch/arm64/Kconfig | 1 + arch/arm64/mm/flush.c | 13 +++++++++++++ 2 files changed, 14 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c842878f8133..37f72e3a75d0 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -94,6 +94,7 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) + select ARCH_WANT_HUGETLB_PAGE_FREE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_HAS_UBSAN_SANITIZE_ALL diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c index 2aaf950b906c..c67c1ca856c2 100644 --- a/arch/arm64/mm/flush.c +++ b/arch/arm64/mm/flush.c @@ -68,6 +68,19 @@ EXPORT_SYMBOL_GPL(__sync_icache_dcache); */ void flush_dcache_page(struct page *page) { + /* + * Only the head page's flags of HugeTLB can be cleared since the tail + * vmemmap pages associated with each HugeTLB page are mapped with + * read-only when CONFIG_HUGETLB_PAGE_FREE_VMEMMAP is enabled (more + * details can refer to vmemmap_remap_pte()). Although + * __sync_icache_dcache() only set PG_dcache_clean flag on the head + * page struct, some tail page structs still can be seen the flag is + * set since the head vmemmap page frame is reused (more details can + * refer to the comments above page_fixed_fake_head()). + */ + if (hugetlb_free_vmemmap_enabled() && PageHuge(page)) + page = compound_head(page); + if (test_bit(PG_dcache_clean, &page->flags)) clear_bit(PG_dcache_clean, &page->flags); }