From patchwork Wed Mar 27 15:23:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13606781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70889C47DD9 for ; Wed, 27 Mar 2024 15:23:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C9A46B007B; Wed, 27 Mar 2024 11:23:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 07A136B0087; Wed, 27 Mar 2024 11:23:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E839B6B008A; Wed, 27 Mar 2024 11:23:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CC9116B007B for ; Wed, 27 Mar 2024 11:23:40 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8BB1FC01C4 for ; Wed, 27 Mar 2024 15:23:40 +0000 (UTC) X-FDA: 81943188600.19.2A8E961 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 8199E1C0012 for ; Wed, 27 Mar 2024 15:23:38 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Esbwt1vz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711553018; a=rsa-sha256; cv=none; b=2UtnlcaBa/RuITISR89hc5clSIHQZxFBQUn7/cxnYRg0H+ui6kulPtFNyUeXknZKRnEP/3 JwxGQj0pprTAL6kT4A8snZ/vZ94U2KFyPnIARtN1BaBR8rrwPxgGiQIDIzVIm3oD+qauDQ jWoLYDkQcOqFHHwMT0bCQuOPWy+Lc7w= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Esbwt1vz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711553018; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=QY8+yIZm1ve5QuiynFE7MjMximJHpz8noDQuYnaNKN8=; b=7lC0iFFC1qwg8uDftpybPlYKve1mWea7rf9phiaCUg9KvntmuxJEwpJLKY/JCALkpjGv9s AXJIc2ktDFJdOylb2RssSEI0EeQqTtiMTaAD60Jv7zqIdkyIxWsIgzEMfQRH5bCcNeKpBS srD59b+sLb1Oh4Q1c0y2eQ6aK0LvAtQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711553017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=QY8+yIZm1ve5QuiynFE7MjMximJHpz8noDQuYnaNKN8=; b=Esbwt1vzoKiRbSwFeQA3mApmKG0xnUIYry3Aps1G9dmr9I5n14dmseAN9JrBQMFi4LKFsY D9L0PMo754iZAdXruU8CunaGwTexaO63iUH4ijBuidWwDP35a9ykck6nZl/ZJEsML/JrZ8 bGFYXbD0fQ/xpD6gOJCHnDUZw6zzskA= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-321-a599d6o9NxOyWDpOQhA9GQ-1; Wed, 27 Mar 2024 11:23:36 -0400 X-MC-Unique: a599d6o9NxOyWDpOQhA9GQ-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-696a9c482a4so3595256d6.0 for ; Wed, 27 Mar 2024 08:23:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711553016; x=1712157816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QY8+yIZm1ve5QuiynFE7MjMximJHpz8noDQuYnaNKN8=; b=WEKSZ/gJIYw+jct/WFpFogzpseyyg6sSQI11VdkdXRoT+HoUPeg8Qhn2fVrjI0Mxcm ud9Tp3EgHzhuDcUKFOXaB2wdi6bL12XnMRi3e5204Urrc/xCTKj7HmFI+uGxsg2a5ZRx Sv+cdNktezWv9f2GdRD1EfTpTjXV6DaGwkmgbAOExvdkuYp0Ny321MnvlizAoR1FuACo oBmqJpJIbLVmWTROupqEK+33ZO21M6XWIR9s/Gnpva0/+hqTkEmU4YaOS4uHgd193cNA RRoczBGy60YxSKf+8GoxfqV2oArQgj8FxXAb0uTOGNXvoyeNtACTRg1+6y74yS52rKer 3h5Q== X-Gm-Message-State: AOJu0YzyIUKQq0iirdoEdyqSS3JELcYgTFzCygLxkuAMM8KDrRrNS3C3 ysEHrUmlSZFTMhWWrjCkoKcd5OqcBTeazjMkU39hO29VOAtrOiRyaW/g8xooQJiftezZliW5pt8 Tah1jzpn4IXNZxZYdAOWlej3fVMlFKTXOMt7nVuG3UQ4vmEiG4tUSslJ6NRz6fGxpN6fHC8paaL uId4IXOohCxgTN+cdUmngnYGWV27VKJw== X-Received: by 2002:a05:6214:4a07:b0:696:8ecc:c368 with SMTP id pg7-20020a0562144a0700b006968eccc368mr9338954qvb.5.1711553015645; Wed, 27 Mar 2024 08:23:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFIDOc++LqI5apT0o+E9CespyQta03LCPOzKebS/1UvNBD/Rvzf0d7U/D6UOW074NAt3LtIIw== X-Received: by 2002:a05:6214:4a07:b0:696:8ecc:c368 with SMTP id pg7-20020a0562144a0700b006968eccc368mr9338893qvb.5.1711553014922; Wed, 27 Mar 2024 08:23:34 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id hu4-20020a056214234400b00690dd47a41csm6412639qvb.86.2024.03.27.08.23.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Mar 2024 08:23:34 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Yang Shi , "Kirill A . Shutemov" , Mike Kravetz , John Hubbard , Michael Ellerman , peterx@redhat.com, Andrew Jones , Muchun Song , linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Christophe Leroy , Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox , Rik van Riel , linux-arm-kernel@lists.infradead.org, Andrea Arcangeli , David Hildenbrand , "Aneesh Kumar K . V" , Vlastimil Babka , James Houghton , Jason Gunthorpe , Mike Rapoport , Axel Rasmussen Subject: [PATCH v4 00/13] mm/gup: Unify hugetlb, part 2 Date: Wed, 27 Mar 2024 11:23:19 -0400 Message-ID: <20240327152332.950956-1-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8199E1C0012 X-Stat-Signature: p9hj3f4pnqkwcrqbch43a4eq8cnmohip X-HE-Tag: 1711553018-153392 X-HE-Meta: U2FsdGVkX1802P51zniNPjE8wSfoh7WFnK2hINqt8yoqKHyBqlS495L5dLC8VAExcv1o0O9SCH9AZxsHCT/kqi1tnUvXet9k2L/T7cvunGhoVpEdvWtJJANFVl96q+gRgaR4UWxfxlyw+ZGo1+K4dlO2I1nBsonsHN8z4GEFikkg6LNVGl0M3IwtA6ibDxj7Yekmdy9HoG3gHb21wwRoIYAbbS7iBMZ0XoSxoWmb9L1yn4TswjGJKcR01BzQGP3TdivoUY9tlVvx8lOtgvm+OC6aG9UQ9j/hqMmFsGSQ05y1yVbok5VVfSOR2ftodVCApLGnAjJPwp0jK+npZ+9ao4oObs3ENF0ZuC0yxWvmXuGrlm90oODUC0QS4tNZlc3JvnZ6FMpid52QN4pZNd/GBAgU7Rfd3UO5BRpLNdgId3neFlNY6E5ti5DmsO5MBmpCx72mFobsgs2wFRmTwZeLKbNGlilqZufKboLtQV29J0zhCFBClWE0gO2Ueqr0Yp3RIrg8UUQQBN/8HJ0D4ZbWSi9bFwU9IevNSyrXwflFUoFRXtcgLI9U5iFn0hawYmqSvohCZVKPm53RZPefZ52ozWds7mifl2hfbgdh86rCLtSH6mSDSikRlWjv3KhjZ8O0zcv7rpzuw3Ki9+C0GGG64DKFsVJsBUi9JL3O3s57hBtYE/yc/4joS5w7osks+gme5J/KtJKOj9Pl7XgWuEjtqEp3kJ8qyPRSV9DQP6Fd8+Mn9yJTHDKKayGqRYd6RpobJMMjpDWR75L1sXuNye82tK25F+t2tezyeuQ8xidQxhf/LnajWAbkD85Xm9FjjmMeot8DSopu3dzgkFgSZHPoWvAlaQ/vTMsCABNUJabBANlwuVqMUkiv3coTx73eK6J8uiKDREoIBZOQSSD9Sm9u1eLTNL+sL9IIINRydJXDiHeTOjaXbf0CiNllSfsaBLpEVXciSNQgPVojMGLyK6+ b9xwaggi SMub6Ufi9Ill5qYuseQt9KqsJySeOvI6Y+Mnrw+k0YygF3sMX/E+6Y9ZaewCoM35tDWtkMBdIb6rGT48uL/l+gaK+ayJUl1ANWM/TJc2NwTb82ol9sB5Dn+XCp/ywqz639MQRDWNqv/Tczj0pPaMiwc0DheT+w81xapJ+90v3TbI9jS2q3JfF7OvJ/f0Jrrb3IPrzDZW5GUrPyh4eLhzxr+7z4XAoCsEbvvMcydAlfS0Kay07R7U/bXNDS3G5SoajYbAsLLZ9j63whwzuzaI8kf/DAFRxycCnmcjvic3oOk7iAYHjYqGa3ZBMPj8HpJO04rBrNcY3XoI/BKzvFFJnuXA5EbXuakvdB8S5PbF5VI8peW4qOn8QOE+YfT76GinRQijngsfpVFMNRoAeu4SWCxbRIxH7OBJ2LeyfNUY/Gb37nA+k2HR3HRrp+SjJNkYk5CEo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu v4: - Fix build issues, tested on more archs/configs ([x86_64, i386, arm, arm64, powerpc, riscv, s390] x [allno, alldef, allmod]). - Squashed the fixup series into v3, touched up commit messages [1] - Added the patch to fix pud_pfn() into the series [2] - Fixed one more build issue on arm+alldefconfig, where pgd_t is a two-item array. - Manage R-bs: add some, remove some (due to the squashes above) - Rebase to latest mm-unstable (2f6182cd23a7, March 26th) rfc: https://lore.kernel.org/r/20231116012908.392077-1-peterx@redhat.com v1: https://lore.kernel.org/r/20231219075538.414708-1-peterx@redhat.com v2: https://lore.kernel.org/r/20240103091423.400294-1-peterx@redhat.com v3: https://lore.kernel.org/r/20240321220802.679544-1-peterx@redhat.com The series removes the hugetlb slow gup path after a previous refactor work [1], so that slow gup now uses the exact same path to process all kinds of memory including hugetlb. For the long term, we may want to remove most, if not all, call sites of huge_pte_offset(). It'll be ideal if that API can be completely dropped from arch hugetlb API. This series is one small step towards merging hugetlb specific codes into generic mm paths. From that POV, this series removes one reference to huge_pte_offset() out of many others. One goal of such a route is that we can reconsider merging hugetlb features like High Granularity Mapping (HGM). It was not accepted in the past because it may add lots of hugetlb specific codes and make the mm code even harder to maintain. With a merged codeset, features like HGM can hopefully share some code with THP, legacy (PMD+) or modern (continuous PTEs). To make it work, the generic slow gup code will need to at least understand hugepd, which is already done like so in fast-gup. Due to the specialty of hugepd to be software-only solution (no hardware recognizes the hugepd format, so it's purely artificial structures), there's chance we can merge some or all hugepd formats with cont_pte in the future. That question is yet unsettled from Power side to have an acknowledgement. As of now for this series, I kept the hugepd handling because we may still need to do so before getting a clearer picture of the future of hugepd. The other reason is simply that we did it already for fast-gup and most codes are still around to be reused. It'll make more sense to keep slow/fast gup behave the same before a decision is made to remove hugepd. There's one major difference for slow-gup on cont_pte / cont_pmd handling, currently supported on three architectures (aarch64, riscv, ppc). Before the series, slow gup will be able to recognize e.g. cont_pte entries with the help of huge_pte_offset() when hstate is around. Now it's gone but still working, by looking up pgtable entries one by one. It's not ideal, but hopefully this change should not affect yet on major workloads. There's some more information in the commit message of the last patch. If this would be a concern, we can consider teaching slow gup to recognize cont pte/pmd entries, and that should recover the lost performance. But I doubt its necessity for now, so I kept it as simple as it can be. Test Done ========= For x86_64, tested full gup_test matrix over 2MB huge pages. For aarch64, tested the same over 64KB cont_pte huge pages. One note is that this v3 didn't go through any ppc test anymore, as finding such system can always take time. It's based on the fact that it was tested in previous versions, and this version should have zero change regarding to hugepd sections. If anyone (Christophe?) wants to give it a shot on PowerPC, please do and I would appreciate it: "./run_vmtests.sh -a -t gup_test" should do well enough (please consider [2] applied if hugepd is <1MB), as long as we're sure the hugepd pages are touched as expected. Patch layout ============= Patch 1-8: Preparation works, or cleanups in relevant code paths Patch 9-11: Teach slow gup with all kinds of huge entries (pXd, hugepd) Patch 12: Drop hugetlb_follow_page_mask() More information can be found in the commit messages of each patch. Any comment will be welcomed. Thanks. [1] https://lore.kernel.org/all/20230628215310.73782-1-peterx@redhat.com [2] https://lore.kernel.org/r/20240321215047.678172-1-peterx@redhat.com Peter Xu (13): mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES mm/hugetlb: Declare hugetlbfs_pagecache_present() non-static mm: Make HPAGE_PXD_* macros even if !THP mm: Introduce vma_pgtable_walk_{begin|end}() mm/arch: Provide pud_pfn() fallback mm/gup: Drop folio_fast_pin_allowed() in hugepd processing mm/gup: Refactor record_subpages() to find 1st small page mm/gup: Handle hugetlb for no_page_table() mm/gup: Cache *pudp in follow_pud_mask() mm/gup: Handle huge pud for follow_pud_mask() mm/gup: Handle huge pmd for follow_pmd_mask() mm/gup: Handle hugepd for follow_page() mm/gup: Handle hugetlb in the generic follow_page_mask code arch/riscv/include/asm/pgtable.h | 1 + arch/s390/include/asm/pgtable.h | 1 + arch/sparc/include/asm/pgtable_64.h | 1 + arch/x86/include/asm/pgtable.h | 1 + include/linux/huge_mm.h | 37 +- include/linux/hugetlb.h | 16 +- include/linux/mm.h | 3 + include/linux/pgtable.h | 10 + mm/Kconfig | 6 + mm/gup.c | 518 ++++++++++++++++++++-------- mm/huge_memory.c | 133 +------ mm/hugetlb.c | 75 +--- mm/internal.h | 7 +- mm/memory.c | 12 + 14 files changed, 441 insertions(+), 380 deletions(-)