From patchwork Tue Dec 19 07:55:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13497936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BB42C46CA2 for ; Tue, 19 Dec 2023 07:56:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 567FB8D0002; Tue, 19 Dec 2023 02:55:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 517A78D0001; Tue, 19 Dec 2023 02:55:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DDF48D0002; Tue, 19 Dec 2023 02:55:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2EC4C8D0001 for ; Tue, 19 Dec 2023 02:55:59 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 07CF51C13F5 for ; Tue, 19 Dec 2023 07:55:58 +0000 (UTC) X-FDA: 81582809238.30.FD874DA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 38040100010 for ; Tue, 19 Dec 2023 07:55:57 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XR8GUsal; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702972557; a=rsa-sha256; cv=none; b=joS2B+NQQzrS+ULJGhbU8F4qbSBBbqj6cY4vgsBlDb9co9tA7Kmrx7qfkBSv5v7itYMbIX 0U5vXCnyuLhuwb9zSwPseL0YYHc8HjTJ0ugCn12WGkpYp0P0EoOj6pYfXeTWg/MVYcg0Rh LjfQajgmAGgBK8ofe9I2btv+ugPmveo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XR8GUsal; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702972557; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=+h83lLyn/5apLRr4X1Iml0TGvxqR5/pwpG6+wMVRoO8=; b=i6B4ZUuY+JOROxsYz/di1HObgLUbK8+8FWqj89ItoiWIL3lCqC9hzAW7mvnQhGVqB10eYq A/oIP+VG/KN6IZBZ/1kApiecEdJF8KBZMjh/lZisXGpKcblkJVXQyAoO5fUkJWgvm8JQR2 4Z+oeP+SED9c0uSGB1mOylXp54KX8F0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972556; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=+h83lLyn/5apLRr4X1Iml0TGvxqR5/pwpG6+wMVRoO8=; b=XR8GUsaluDchgruKmarqBEqjW3Wqmhwn4GplqLYMVIMv0CFAKTWmyjaZJ4QLKt6Y03shFw 5SPz1ofzy3xk0tM53bvNAgaXdpVWJxzfVMba7MLpXt7SgdL9i/0dCMaKKz9fVEpw8TaBHi XD0l0+6cK3iff/FxWtQuYwIL0K3Oa2w= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-460-78fty2BGN2uAAfrfPZNG0g-1; Tue, 19 Dec 2023 02:55:53 -0500 X-MC-Unique: 78fty2BGN2uAAfrfPZNG0g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 065E63C025C1; Tue, 19 Dec 2023 07:55:52 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id 83B262026D66; Tue, 19 Dec 2023 07:55:40 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 00/13] mm/gup: Unify hugetlb, part 2 Date: Tue, 19 Dec 2023 15:55:25 +0800 Message-ID: <20231219075538.414708-1-peterx@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 38040100010 X-Stat-Signature: fhiwi4ks1fw58nuu86913kamgpjbu4wt X-HE-Tag: 1702972557-910813 X-HE-Meta: U2FsdGVkX1+JRs9SJRiHfjp/k4T1iketEHC9dFgY+1RQfp3+WtkjtwesVjVuQfKmxRLje449BOron1bjiYwPUg10HXPksTYObXMJQHK/kSIkj4QGGfpcMn/O5uPum0hQqU5JgEXgiXSZ4IJ8Rsa1Nhfy1sSqNEFhgdgEC0W+ffjvcAf/vwm0AgrpOjG2HMTh43ht6BvhJVWF9u7IjB3C+b9MD5YHDqZ2cAXnFG2Bb9AqBZ0bBz3hWphE3PpDvVyiAQORMFoH1M9KQw+eBjcfV/BU7Ngc50W2kMp4FDzJd1tVewyXydw0uxUwL39wy9zoXLvwNPGiDotGlFIRMAPE2CtMBA2EYNWGZLvUZUEN4ZqHvYkjVBUpLoRK2ZuIV0JtaBJrkJvDhxd7qdwD3aePzBtr6+aZkP6oKu2H1EIvGvhP8pF747+LM7ZsquS86ztEouA0MsiwdaxuZ3MROlnS/mV8RLdmzPgjAPY2fjlDVVEKPIDhtdTU4wPjL6XVpdyRnpWVlpjQwr25agz7CbHiZOMRoByzWPhyYrY13aSNaoXY/QH40GstwvUsatuO5YAJEjFbWqMa6HUzfM/pwcpcRzJyicJzqmhyNINah5ZHzFIr1lifKXWiT50EgTggnmtz7esQ3ftrlC9OqGSDwcXAyyUklPSGiZNRbly8mhcR73SVhRi/LoMmtzc5zXUq0NqGDuq4E78NGLS+MiN0IK73uw6dizgbaXWZD7KtcWYdEzbTlz1WKbgqvVl0J9TH+7zN7c6YaG58KMiU1l8FzezlFHUisMxZwQNWiFY5J46SHKorAv8mvOUhEEBefy1pyQutp1ebKtcuH/7FfTcjlxwCvA8B9YICjjDL9l8FBOrQcgf8Hbo3exNs14yhiYx9G1kJRYgnGOPobjtfbXkEV2+a9Iw5dg5bGmYS2pyKUnANHPN2oPZ7W8ivxz8SV9cqUK5UdVHmkb0ZgjsghzN3iGK 3ghp1mDR k1+HEbA8XSZGIqGMl9r2Sq7IFf3mdqedqgVSxvAwaWunpXYbgXWQ+tJmrdYkhfJJuAzZWSu9EiszDgAfClBHa1TACAneFF0Qtd06rDEmpkrO0ZX6E23J2xPGn8ckcdUTpCNnRVJNWtDIT+chQlHWVv8L9QZLJ68D95BjwrNeeYp8wsifABTxeyAjdlv1MQ1gXWQjV9QppjQz9QC8ALHL0duJR4Ip0cYv1PT8rUbDesubMTjELXs3tySeQNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu This is v1 of the series. The series removes the hugetlb slow gup path after a previous refactor work [1], so that slow gup now uses the exact same path to handle all kinds of memory including hugetlb. It's based on latest mm-unstalbe (c13bdc82ada9). RFC->v1 (use old verion's patch index): - Fix build for !THP and/or !HUGETLB - Fix a bug in the hugepd path that can cause a loop - patch 3: - Rename to "Make HPAGE_PXD_* macros visible even if !THP" [Christoph, MikeR] - patch 5: - Posted separately as a devmap bugfix - patch 6: - Add a comment above gup_huge_pd() explaining the file folio writeback issue, with references when anyone would extend hugepd to other types of file memories [Christoph] - patch 7: - Keep parameter name as "page" in record_subpages(), modify the commit message by removing any assumption on head page. [Matthew] - patch 12: - Rename to: "mm/gup: handle hugetlb in the generic follow_page_mask code" [Christoph] - Added patch: - "mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES", this helps to fix build for !THP and/or !HUGETLB - Added patch - "mm/gup: Cache *pudp in follow_pud_mask()", as suggested by Christoph For the long term, this is one small step towards merging hugetlb specific codes into generic mm paths. From that POV, this series removes one reference to huge_pte_offset() out of many others. To make it work, the generic code will need to at least understand hugepd, which is already done like so in fast-gup. Fortunately it seems that's the only major thing I need to teach slow GUP to share the common path for now besides normal huge PxD entries. Non-gup can be more challenging, but that's a question for later. There's one major difference for slow-gup on cont_pte / cont_pmd handling, currently supported on three architectures (aarch64, riscv, ppc). Before the series, slow gup will be able to recognize cont entries with the help of huge_pte_offset(). Now it's gone but still working, by looking up pgtable entries one by one. It's not ideal, but hopefully this change should not affect yet on major workloads. There's some more information in the commit message of the last patch. If this would be a concern, we can consider teaching slow gup to recognize cont pte/pmd entries, and that should recover the lost performance. But I doubt its necessity for now, so I kept it as simple as it can be. Test Done ========= This v1 went through the normal GUP smoke tests over different memory types on archs (using VM instances): x86_64, aarch64, ppc64le. For aarch64, tested over 64KB cont_pte huge pages. For ppc64le, tested over 16MB hugepd entries (Power8 hash MMU on 4K base page size). Patch layout ============= Patch 1-7: Preparation works, or cleanups in relevant code paths Patch 8-12: Teach slow gup with all kinds of huge entries (pXd, hugepd) Patch 13: Drop hugetlb_follow_page_mask() More information can be found in the commit messages of each patch. Any comment will be welcomed. Thanks. [1] https://lore.kernel.org/all/20230628215310.73782-1-peterx@redhat.com Peter Xu (13): mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES mm/hugetlb: Declare hugetlbfs_pagecache_present() non-static mm: Provide generic pmd_thp_or_huge() mm: Make HPAGE_PXD_* macros even if !THP mm: Introduce vma_pgtable_walk_{begin|end}() mm/gup: Drop folio_fast_pin_allowed() in hugepd processing mm/gup: Refactor record_subpages() to find 1st small page mm/gup: Handle hugetlb for no_page_table() mm/gup: Cache *pudp in follow_pud_mask() mm/gup: Handle huge pud for follow_pud_mask() mm/gup: Handle huge pmd for follow_pmd_mask() mm/gup: Handle hugepd for follow_page() mm/gup: Handle hugetlb in the generic follow_page_mask code include/linux/huge_mm.h | 25 +-- include/linux/hugetlb.h | 16 +- include/linux/mm.h | 3 + include/linux/pgtable.h | 4 + mm/Kconfig | 3 + mm/gup.c | 362 ++++++++++++++++++++++++++++++++-------- mm/huge_memory.c | 133 +-------------- mm/hugetlb.c | 75 +-------- mm/internal.h | 7 +- mm/memory.c | 12 ++ 10 files changed, 342 insertions(+), 298 deletions(-)