From patchwork Fri Mar 17 10:57:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13178878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECA8CC74A5B for ; Fri, 17 Mar 2023 10:58:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F5406B0075; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A56C6B0078; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 294286B007B; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1A1156B0075 for ; Fri, 17 Mar 2023 06:58:24 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C3DE71A17C8 for ; Fri, 17 Mar 2023 10:58:23 +0000 (UTC) X-FDA: 80578091286.26.EF38649 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 0BA2C100016 for ; Fri, 17 Mar 2023 10:58:21 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679050702; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=lSgscAizH3yvBpTQ1Zvv0zhO1+Lg+o63S6pMg3avALY=; b=eG4P/lsAmBNdq6jKy5iv4OBxcf3FTKlwzTn0pjAyRoWAOqjxzu+wchr76hfe4SxSbdgP6v rwPNABKBaAZz7BaGkD6mH9z+MnWf7FWCfLJD8tEFHR/TGJfJ3OCL9LvAxg0hcVsURlSsln wGXHcqTny5u9H04bu40PIB0GsD6dvTg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679050702; a=rsa-sha256; cv=none; b=xiB2c8HsH2ZJ7kT1OSL8x9HiXdyZRG4XRoBPuzIpBo3cpR3KzO+8JvWLLe1UxrAT01UbWO izJ1BX7CP6IF1fqk+0DnYApJT6dmnLbrDYk4C/eltBsobobEMCh9cLfW8lEmyB1/0EwR9R ++GZoHNo/rmH6UHktlw/XixucN0u07U= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A5E961480; Fri, 17 Mar 2023 03:59:04 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF7903F885; Fri, 17 Mar 2023 03:58:19 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , "Matthew Wilcox (Oracle)" , "Yin, Fengwei" , Yu Zhao Cc: Ryan Roberts , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 0/6] variable-order, large folios for anonymous memory Date: Fri, 17 Mar 2023 10:57:56 +0000 Message-Id: <20230317105802.2634004-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 0BA2C100016 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: bqzhskuxe3kb9s85zinn1k17gagmmh76 X-HE-Tag: 1679050701-159913 X-HE-Meta: U2FsdGVkX18K+zNLoFcflLHD3lDGVAB6ESMqUV8qrXtQr9ChG+SHPfkAV6sTdif53c6PCxCxhBwvEzEm2Mq12GgnS8giVcyimHY+KOUQFymGhlJUIiRFIZQXia3jjbtjtApeDGoAxJlnaekYOL919C2rUqLFml92S0sT+izfBTWHm7Q1J1/cGzuwniiP/TfyPtR3XkYTcswfdWXTWsBmvFsYB8mlnwcOzXIL0PP4R9gFEBfih5+RpMLHnkGOisK7xOCd4AGKWbzWmpuYlumeGEC0VrPLB6UOR+KaopE7VwRUj0vNOb/dgbZV0dONLBQ8Vf5vzJNstzNUEwmvYXRrQt4RSLmCgUFsETTMan0HgpMF723uLXPx8c8KmPwWsZCujpEKYbiCFZSJdmtoI869X84/3/u7+gqQDtHENszk4nTcZAejc/bxl1PdWeam8E8loheWAzISPyh/Y0SDMJ/I0T5H6V8kXDS/siCOOTA0dkKrGEkQgsFYCvt6uBUSMYVpUo5IBk4ZSTeuFa9pRcRX7yQvqNJWTOI/n9Ks15akpDfUF0FKHGZNyrLAZQGaknB/ahx8cKlrwP62rAkfNiCO9pxWyiNqghGWZEguhuiI5d/+/zc+s7OaNkwsk0akRYilOMDogX0a1gcCKFSRNlinpWOAey5utwXEfWvLAf9uWSJPqdlao6x3xFNtnMpy4kxfKi3iDm5fa+OF6bc7w+mxakyXMjANdWFthYb0J7IDIQqk+X1p3DUEFlXYxGKtupBvSpHRGFoKgcPdS+JQFEjvHX7/JgfZfi4QD2JlEiuX3os2654M9jLgWAt+u3mBWS4RiJ/QmMSOaF6DPh2yqOmYvuMeiuAEsiX8/mW7FazUHbXX3qz7MLakicCme2hHLv9tv/mXX+ymb5uJi3DWvfsQB/odUWyJWBLkBY1upKtzRM1JocNCTBnI6mGmJrzvvdySAXxGrtCedYBJAIl2yH1 VxbNBUUh i1RngcF9I74bRfC17uSLvj6PN0Ltz7dMLZvBpEuEqOhSqxkC5LrtkU0ppPauWmam94UaVk2okEIrx6Yan/RWDwiYDRi92/F0+8rvc6VyUVuKjy0++D7fcRca1CYp6ME5ayWAmdNP1ESmjlzsmAeTRwiu8oGjajlZzZAkxF7BkMGE462cjOFpsQ1xI3lNWNZoenGXhZ+qcrimJ7PWD1xJaiiDOlMqmyBc0BpH9Kqe00A2fqpm7KFq0WCJ/3ei2wnoBRMPtJrhpfXsbqAm/MrmS783Z6MO1QL81DUJ3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi All, This is an RFC for an initial, _very_ limited stab at implementing support for using variable-order, large folios for anonymous memory. It intends to be the minimal change, upon which additions can be made incrementally. That said, with just this change, I achive a 4% performance improvement when compiling the kernel (more on that later). My motivation for posting the RFC now is twofold: - Get feedback on the approach I'm taking before I go too far down the path; does this fit with the direction of the community? Are there any bear traps that I've not considered (due to my being fairly new to mm and not having a complete understanding of its entirety)? - Seek support for a bug I'm encountering when MADV_FREE is attempting to split_folio() one of these new variable-order anon folios. I've been pouring through the source and can't find the root cause. For now I have a work around, but hopefully someone can give me some pointers as to where the problem is likely to be. (see details below). The patches apply on top of v6.3-rc1 + patches 1-31 of [4] (which needed one minor conflict resolution). And I have a tree at [5]. See [1], [2], [3] for more background. Approach ======== For now, I'm only modifying the allocation path (do_anonymous_page()). I'm not touching the CoW path. First, I determine the order of the folio to allocate for the given fault. This is determined by: - Folio must be naturally aligned within VA space - Folio must not breach boundaries of vma - Folio must be fully contained inside one pmd entry - Folio must not overlap any non-none ptes - Order must not be higher than a provided starting order Where the "provided starting order" is currently hardcoded to 4, but the idea is that this would eventually be a per-vma value that gets dynamically tuned. We then try to allocate a large folio of the determined order, and keep trying to allocate with successively smaller orders until we succeed. Once the folio is allocated we can take the PTL and re-check that all the covered PTE entries are still none. If not, we decrement the order and start again. Next, the folio is added to the rmap using a new API, folio_add_new_anon_rmap_range(), which is similar to Yin, Fengwei's folio_add_file_rmap_range() at [4]. And finally set the ptes using Matthew Wilcox's new set_ptes() API, also at [4]. Folio/page refcounts and mapcounts are managed in the same way as Yin, Fengwei is doing in folio_add_file_rmap_range(); A reference is taken on the folio for each pte, _mapcount is incremented on each page by 1, and folio->_nr_pages_mapped is set to the number of pages in the folio (since every page is initially mapped). It is my assumption that the mm should be able to deal with these folios correctly for CoW and reclaim etc. although perhaps not as optimally as we would (eventually) like. Bug(s) ====== When I run this code without the last (workaround) patch, with DEBUG_VM et al, PROVE_LOCKING and KASAN enabled, I see occasional oopses. Mostly these are relating to invalid kernel addresses (which usually look like either NULL + small offset or mostly zeros with a few mid-order bits set + a small offset) or lockdep complaining about a bad unlock balance. Call stacks are often in madvise_free_pte_range(), but I've seen them in filesystem code too. (I can email example oopses out separately if anyone wants to review them). My hunch is that struct pages adjacent to the folio are being corrupted, but don't have hard evidence. When adding the workaround patch, which prevents madvise_free_pte_range() from attempting to split a large folio, I never see any issues. Although I'm not putting the system under memory pressure so guess I might see the same types of problem crop up under swap, etc. I've reviewed most of the code within split_folio() and can't find any smoking gun, but I wonder if there are implicit assumptions about the large folio being PMD sized that I'm obviously breaking now? The code in madvise_free_pte_range(): if (folio_test_large(folio)) { if (folio_mapcount(folio) != 1) goto out; folio_get(folio); if (!folio_trylock(folio)) { folio_put(folio); goto out; } pte_unmap_unlock(orig_pte, ptl); if (split_folio(folio)) { folio_unlock(folio); folio_put(folio); orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); goto out; } ... } Will normally skip my large folios because they have a mapcount > 1, due to incrementing mapcount for each pte, unlike PMD mapped pages. But on occasion it will see a mapcount of 1 and proceed. So I guess this is racing against reclaim or CoW in this case? I also see its doing a dance to take the folio lock and drop the ptl. Perhaps my large anon folio is not using the folio lock in the same way as a THP would and we are therefore not getting the expected serialization? I'd really appreciate any suggestions for how to pregress here! Performance =========== With the above bug worked around, I'm benchmarking kernel compilation, which is known to be heavy on anonymous page faults. Overall, I see a reduction in wall-time by 4%. This is inline with my predictions based on earlier experiments summarised at [1]. I beleive there is scope for future improvement on the CoW and reclaim paths. I'd also expect to see performance improvements due to reduced TLB pressure on CPUs that support HPA (I'm running on Ampere Altra where HPA is not enabled). Of the 4%, all of it is (obviously) in the kernel; overall kernel execution time has reduced by 34%, more than halving the time spent servicing data faults, and significantly speeding up sys_exit_group(). Thanks, Ryan [1] https://lore.kernel.org/linux-mm/4c991dcb-c5bb-86bb-5a29-05df24429607@arm.com/ [2] https://lore.kernel.org/linux-mm/a7cd938e-a86f-e3af-f56c-433c92ac69c2@arm.com/ [3] https://lore.kernel.org/linux-mm/Y%2FblF0GIunm+pRIC@casper.infradead.org/ [4] https://lore.kernel.org/linux-mm/20230315051444.3229621-1-willy@infradead.org/ [5] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/anon_folio-lkml-rfc Ryan Roberts (6): mm: Expose clear_huge_page() unconditionally mm: pass gfp flags and order to vma_alloc_zeroed_movable_folio() mm: Introduce try_vma_alloc_zeroed_movable_folio() mm: Implement folio_add_new_anon_rmap_range() mm: Allocate large folios for anonymous memory WORKAROUND: Don't split large folios on madvise arch/alpha/include/asm/page.h | 5 +- arch/arm64/include/asm/page.h | 3 +- arch/arm64/mm/fault.c | 7 +- arch/ia64/include/asm/page.h | 5 +- arch/m68k/include/asm/page_no.h | 7 +- arch/s390/include/asm/page.h | 5 +- arch/x86/include/asm/page.h | 5 +- include/linux/highmem.h | 23 +++-- include/linux/mm.h | 3 +- include/linux/rmap.h | 2 + mm/madvise.c | 8 ++ mm/memory.c | 167 ++++++++++++++++++++++++++++---- mm/rmap.c | 43 ++++++++ 13 files changed, 239 insertions(+), 44 deletions(-) --- 2.25.1