From patchwork Wed Mar 30 14:54:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12795988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CAD20C433F5 for ; Wed, 30 Mar 2022 15:01:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=6R4lR2h3cCSQbMb+aUCXYOwcC6zTPkweYegrbIVjsy4=; b=SGPucHU52YZTAY houSpdyXDwxiPygETx2N/o0kkrkEYr3ZKzpYPBtL24g/fhthbWNnlNkgASn4Rz82esD5CVNuz08vc ThaPGD72QLLqob6okyiYdYl7s5UsSvupbpNxwWRjfAJsojuJQQxFSqfQbEG8zBAV7CsExWbejomX8 +2yTw4mBNmjBMP43MUGE+FD0ndHp1H0vZkyEwwC0OK5pV2jgE0miOiI9Gz7hSVlcVz9Dd6vPkopxk ty32MaqHb5r/LTIdyu2GISE+aL5PZKce5oUQEQYhN3jCK/K0t5erTjGJAZIW+VEydwbpmB5j5rJDj GFzqyGRmZIV/20HntcDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nZZnP-00GN8j-8U; Wed, 30 Mar 2022 14:59:39 +0000 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nZZj1-00GL4u-JP for linux-arm-kernel@lists.infradead.org; Wed, 30 Mar 2022 14:55:10 +0000 Received: by mail-pj1-x1033.google.com with SMTP id l4-20020a17090a49c400b001c6840df4a3so246001pjm.0 for ; Wed, 30 Mar 2022 07:55:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=FR1FYGjzBBFVsMjKKkaOUa/tpuD2LjaT4A9QUhniUPw=; b=6I1kzpLw81ZLlolZkl/RjPaD+BdQRQC/HBh+0/RQ1w0Y0SRH7SYx7nJNgG0kRKd9yL jtuZoAA2jUHYyMo8Yhnva/4GT+HWMqg/v6F/jD0dliLoblQpeqR8NeImY/jV4MyMPtE3 ra6wnisQSJZKXM7sCqPDpcgSjh9pwETskySogCzn6p2I0J3Vw/7oIB9HhF+jNVDiwf9n Zr03cdrSe0gEtLpfErESGbvagcgjYLPMJuzcceQu18EdDuBJCYEU8ETfyweM2V1PiO/O AIEoWMUzHw4QjSBU8vAV5NKrQjK8m1EaKm6wclNj3+ECSayjXZRv8TwfGX12bXndtKsy JJNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=FR1FYGjzBBFVsMjKKkaOUa/tpuD2LjaT4A9QUhniUPw=; b=N2L7y3xYqgDSfMu4j6USVqFyynR7gR9/Bi2AwPXbihQ7F3Z9AXB1OHDQp0qIyfZdcw X5nj9nPX5QrR/Mr8WdYf2wdgTknjKRIX4E/haZSaLZIQ/QNMg5RtocyJBMNPSdjqx08d I8HvSAlHgRQLe9zZvDu/NUGYY6IyVbZexnKYYSaxZWvy6Y5pYU3OhlnPGmL66VpZSymw XK5+ZT8YXSty0QZEmOXIefOykXdXuv8006No9lVM9Ug2+MTEPq/ZbXmIa2CbXrHWgvtJ 2AmujX+eaMxYhT1jfndKU+drxMRHsD+bBWam7P0/ojCL13MD5dy7QRino5807zNT0qV1 s7BQ== X-Gm-Message-State: AOAM530MJ3oDP+cqB4FfTDMgqRMiHJpLkZZBeSQHJviWUWtXW5adReNl gjZ8L5d353es7w5byEm2nM6OAA== X-Google-Smtp-Source: ABdhPJw4HvO7Gh7pHgNX8rhNx/r/CsKhA4e//A3XD5NLH95YhJJ3Pd64W0SCZufUOy+BAlzj9sqMAw== X-Received: by 2002:a17:90b:1c07:b0:1c7:5324:c68e with SMTP id oc7-20020a17090b1c0700b001c75324c68emr5600650pjb.202.1648652105388; Wed, 30 Mar 2022 07:55:05 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.239]) by smtp.gmail.com with ESMTPSA id a71-20020a63904a000000b00398666dcf8esm8236283pge.40.2022.03.30.07.54.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Mar 2022 07:55:05 -0700 (PDT) From: Muchun Song To: will@kernel.org, akpm@linux-foundation.org, david@redhat.com, bodeddub@amazon.com, osalvador@suse.de, mike.kravetz@oracle.com, rientjes@google.com, mark.rutland@arm.com, catalin.marinas@arm.com, james.morse@arm.com, 21cnbao@gmail.com Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v3] arm64: mm: hugetlb: Enable HUGETLB_PAGE_FREE_VMEMMAP for arm64 Date: Wed, 30 Mar 2022 22:54:00 +0800 Message-Id: <20220330145400.18565-1-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220330_075507_770204_7930E12F X-CRM114-Status: GOOD ( 24.99 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The feature of minimizing overhead of struct page associated with each HugeTLB page aims to free its vmemmap pages (used as struct page) to save memory, where is ~14GB/16GB per 1TB HugeTLB pages (2MB/1GB type). In short, when a HugeTLB page is allocated or freed, the vmemmap array representing the range associated with the page will need to be remapped. When a page is allocated, vmemmap pages are freed after remapping. When a page is freed, previously discarded vmemmap pages must be allocated before remapping. More implementations and details can be found here [1]. The infrastructure of freeing vmemmap pages associated with each HugeTLB page is already there, we can easily enable HUGETLB_PAGE_FREE_VMEMMAP for arm64, the only thing to be fixed is flush_dcache_page() . flush_dcache_page() need to be adapted to operate on the head page's flags since the tail vmemmap pages are mapped with read-only after the feature is enabled (clear operation is not permitted). There was some discussions about this in the thread [2], but there was no conclusion in the end. And I copied the concern proposed by Anshuman to here and explain why those concern is superfluous. It is safe to enable it for x86_64 as well as arm64. 1st concern: ''' But what happens when a hot remove section's vmemmap area (which is being teared down) is nearby another vmemmap area which is either created or being destroyed for HugeTLB alloc/free purpose. As you mentioned HugeTLB pages inside the hot remove section might be safe. But what about other HugeTLB areas whose vmemmap area shares page table entries with vmemmap entries for a section being hot removed ? Massive HugeTLB alloc /use/free test cycle using memory just adjacent to a memory hotplug area, which is always added and removed periodically, should be able to expose this problem. ''' Answer: At the time memory is removed, all HugeTLB pages either have been migrated away or dissolved. So there is no race between memory hot remove and free_huge_page_vmemmap(). Therefore, HugeTLB pages inside the hot remove section is safe. Let's talk your question "what about other HugeTLB areas whose vmemmap area shares page table entries with vmemmap entries for a section being hot removed ?", the question is not established. The minimal granularity size of hotplug memory 128MB (on arm64, 4k base page), any HugeTLB smaller than 128MB is within a section, then, there is no share PTE page tables between HugeTLB in this section and ones in other sections and a HugeTLB page could not cross two sections. In this case, the section cannot be freed. Any HugeTLB bigger than 128MB (section size) whose vmemmap pages is an integer multiple of 2MB (PMD-mapped). As long as: 1) HugeTLBs are naturally aligned, power-of-two sizes 2) The HugeTLB size >= the section size 3) The HugeTLB size >= the vmemmap leaf mapping size Then a HugeTLB will not share any leaf page table entries with *anything else*, but will share intermediate entries. In this case, at the time memory is removed, all HugeTLB pages either have been migrated away or dissolved. So there is also no race between memory hot remove and free_huge_page_vmemmap(). 2nd concern: ''' differently, not sure if ptdump would require any synchronization. Dumping an wrong value is probably okay but crashing because a page table entry is being freed after ptdump acquired the pointer is bad. On arm64, ptdump() is protected against hotremove via [get|put]_online_mems(). ''' Answer: The ptdump should be fine since vmemmap_remap_free() only exchanges PTEs or splits the PMD entry (which means allocating a PTE page table). Both operations do not free any page tables (PTE), so ptdump cannot run into a UAF on any page tables. The worst case is just dumping an wrong value. [1] https://lore.kernel.org/all/20210510030027.56044-1-songmuchun@bytedance.com/ [2] https://lore.kernel.org/all/20210518091826.36937-1-songmuchun@bytedance.com/ Signed-off-by: Muchun Song --- v3: - Rework patch's subject. - Clarify the feature of HUGETLB_PAGE_FREE_VMEMMAP is already there in the current code and easyly be enabled for arm64 into commit log. - Add hugetlb_free_vmemmap_enabled() check into flush_dcache_page(). Thanks for Barry's suggestions. v2: - Update commit message (Mark Rutland). - Fix flush_dcache_page(). arch/arm64/mm/flush.c | 13 +++++++++++++ fs/Kconfig | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c index a06c6ac770d4..487ab9032c13 100644 --- a/arch/arm64/mm/flush.c +++ b/arch/arm64/mm/flush.c @@ -75,6 +75,19 @@ EXPORT_SYMBOL_GPL(__sync_icache_dcache); */ void flush_dcache_page(struct page *page) { + /* + * Only the head page's flags of HugeTLB can be cleared since the tail + * vmemmap pages associated with each HugeTLB page are mapped with + * read-only when CONFIG_HUGETLB_PAGE_FREE_VMEMMAP is enabled (more + * details can refer to vmemmap_remap_pte()). Although + * __sync_icache_dcache() only set PG_dcache_clean flag on the head + * page struct, some tail page structs still can see the flag since + * the head vmemmap page frame is reused (more details can refer to + * the comments above page_fixed_fake_head()). + */ + if (hugetlb_free_vmemmap_enabled() && PageHuge(page)) + page = compound_head(page); + if (test_bit(PG_dcache_clean, &page->flags)) clear_bit(PG_dcache_clean, &page->flags); } diff --git a/fs/Kconfig b/fs/Kconfig index 7a2b11c0b803..04cfd5bf5ec9 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -247,7 +247,7 @@ config HUGETLB_PAGE config HUGETLB_PAGE_FREE_VMEMMAP def_bool HUGETLB_PAGE - depends on X86_64 + depends on X86_64 || ARM64 depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON