From patchwork Mon Aug 26 20:43:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBA61C5472E for ; Mon, 26 Aug 2024 20:44:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C51BC6B00AD; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C00F56B00AE; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A05556B00AF; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7EBBB6B00AD for ; Mon, 26 Aug 2024 16:44:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3A0C81C4E25 for ; Mon, 26 Aug 2024 20:44:37 +0000 (UTC) X-FDA: 82495574994.20.53C969C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 2773C40008 for ; Mon, 26 Aug 2024 20:44:35 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YYLVoWiG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705056; a=rsa-sha256; cv=none; b=VPuKczshfG50cZbSkqK5og0OdEoh/uDF7aRwu/6YDjxgVNL6RAZVNFZIZCNhWInmxrS9/R 4xI5XCSa/rl5YWUVXdkSt/Is1M+G3KJLCpZQMTysRYP2oXgjlofQQWWHui42sE+Ldvdr7Q a7Fke+VgUH0fSNtRDUcz5lc7ctbgqz8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YYLVoWiG; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=55fMtcQ9Z+qnxxjW1gKhY5fqeL4MfdKyYALAEDBWELXiVdf3Fys4KX6FhWCa9g9kd1Pbe+ yfFl5Z2iYTl6BrP19Xdspoor2MRVqycvm9IwnLo2yXGQrfWCmbj5oAbOSIRZIB44laWBSu 0TqoLdeEdSPTKC/vgERkv8rK14K4Dd8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=YYLVoWiGteZHco1yz/MpCpfQF0yuSjBl1/+aBlhmwR1vGYC/tdjrJcfQZlZrF0e9fqAAqF 7VTkx7WYGYneWcCf+1sMX+ik/PDLezaALHGApYH03Wjeo7mkXZc1u5YTk2nTbMLFbzsYUf YnhJl9U7lzcQndaKRAOubldo2RhP3AA= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-554-rfjIgXKePW6xZG6plbQN4A-1; Mon, 26 Aug 2024 16:44:33 -0400 X-MC-Unique: rfjIgXKePW6xZG6plbQN4A-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7a1e9a3825aso606412385a.2 for ; Mon, 26 Aug 2024 13:44:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705072; x=1725309872; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FEAM9DK74WZYySVYhL7YnC2YOY0xQ42haB8uzH2x2BY=; b=joD02D9qF/uMllT4ir6sM5PQ0M5Eqbf753Vw7xr/rM5Z95RN9Z1iOUhpeVFHRZd+uM 7En3Ub3KjOaSfbCYHnv71AVrkUZwUTjFXF7i6Toy1VAAfE1GpOGSVcB7U+38CylgbJJ8 TiGVHJPBUCRpVStMbI75qhv7YH5P043OClr5nSgUCDW+mrVFi2W64GdIwzPhxg9ooeA9 0bbFCOd258QruZYrzLYNf/LlRsWoW5RgV47L9ffgq3tdrYWajdrWNDAySwuSyvsGryej qoGtTcVcsF85jku7eGHACAixzdk2jdftKwjuQQ275vmmKJZeQFl4bATKww2+td2BDjvV bMFQ== X-Forwarded-Encrypted: i=1; AJvYcCVr3zcd8a8hKb5ROs4Gf58BIwc4sg8eO/INcR0BB/UVKJCNac11yVMCGBe12IQVh4l7okq3pva20Q==@kvack.org X-Gm-Message-State: AOJu0YwTBGDKkY5mc/z6wAFvsj/gxCROU9yHibC/OeTbdfpmlLQALVeO hyEt5k9LkO5Dfu3+myhLd0oW9vA1Eksn5Ka2dWfRb6INCS2hOt2+g3t3WmrPfiTDMBL4wz9aCWJ hJsVcBPGysW+NZrS6kily94gE4fOND6kJ8K0En98RA1EUWwmmELCIkMPi X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392500285a.8.1724705072549; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHDiJ56urcvTKu8a4/ieO25ZyreTkMfmYYALFQqTS8OY3wwzzWFQolk9khXQC+xv0hRJc4KwA== X-Received: by 2002:a05:620a:430a:b0:7a4:faab:fc79 with SMTP id af79cd13be357-7a6896d1835mr1392497485a.8.1724705072141; Mon, 26 Aug 2024 13:44:32 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:31 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 18/19] mm/arm64: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:52 -0400 Message-ID: <20240826204353.2228736-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 2773C40008 X-Rspamd-Server: rspam01 X-Stat-Signature: z934tg434mckxjxiwc4j8nu6rzwy1ukn X-HE-Tag: 1724705075-103837 X-HE-Meta: U2FsdGVkX1/zPijBPSqeZ5cPemstG+1dHMTSaMKaQyu1mXf5OVq6AnLnjixSJbGBGXrAlitdDBT9Obvr31LNsUFp9U+yr9OsR0ByGrRgz+fdktl+QPJg/XalzDg9RdXTFOY7KHdxJGls9LgWrzxIbUXfSvtQfaK/cq5l/Iq4bKx5KBSSECa1SGpyOcPssqqT7wUn/2bRCw779G28lJwCeIG4/kTvpvdWfqaR8pzx4bgfRL4BB6ftQf+g0cIisupYaTrpVF/clCJy9Nh9HFj31/VkgZ5FaJWfJH5csFuTk1iySlQj2lped62Oq0LuamnZBJljhC/vX7ucKMLOYvfjbM3UdbOYLTCZml1CvVjL+hHZ6aH4v4u4BPX4lALvUupZrRCNF52nE/tMHfmYvAGOIPuG4Kr4hoeyA5cKI0AzQFLxKv4WcsU5kV+0NihSxEmTCj6G7j7Xc+DWUmEiyV3t++vgLVGQc/w+4wVgezAG9cTd7R81j09kTh6oRHi3GmL+Sjd/oljAzTjXjUGU6AL+KIRjKuyEQyO6cieWzsLF7AstC9UONbV0kd+aKlmphqIVXjk/K7+3bae7Wg3AbA0RIjc/WvaytTRSyIAqj6Kvxavda2HYK/yR0POX/KByY0CHiNnCLHmNUZ7ibC+SmazD7OJWzCfFFkbvRfwid1ygFJWYkbVNeAyrn7SjJkctgT+xjo+ArHNR6cquZVECa6xJ6uAy5p4uAlNP8uzqoDLpDs5tWudCYNX1r+ddMPPRdZn2JHvgI+P3LYwId9qZ9u8e0sDRwmmkgclNCK8hM88+77jCb7fQLfwSYby5/4FsGGKPBokexjvGaL1vupNJqqptsBenmwzJ0qcAX3003cB19gvCMueLyJQngAv23TUknZBhzR5AtlJpCv7O4QlccqcA0H8CJgJTVOQGb6/lOkTbXwQSWw2ZI6kyojC44+1MqdPzzfz0k1BAmGCJEUKW8GW 8MvgGU/N eHVg3p1W2XWwqfo/5o3vOHST2jBLS3XEKZ+K6biNo8XqcPVtZ5XBLlje1Hqg4q4tCWAErGzpqX2lLoHqQpIEoAXZfHMgv1uVTcEwv7XNeeHeuBf6BketW9jb/zajWOIHZ73Fgh5WGRkgtKukhSFWvk18JegXzZ6L6PUX+tG0AQOeyl2KD3Ti+KI7nvm3w2JDEBgw4Cu9VQHte6Ngf/KCHobBHoCjz+pJn/A8XHdbUZs/pWwoYj/NDQESSfMm7kOfNqsLRfNrhIeAwWZ+uSzBtqLDHmrkfLjRSABUOCGc88npYhN4HqANQQUHigaRduC+tvVAW/qHmPAvA7GIr0w6O+YlUtwAuDhlrL8jR4RUdstnWpCBzXXtMTZ8FOf8ntmlNPctRzh+EzPgYgRAuB3Wt79hdgJ/xK1u/YOesjRTgk5z3XGFbqiH/iMQoj84fQmIdrddeD3Bb9O+/lHR9redrME1QghL2saJn+KX9zHFwiCXtObuPHY+A8Qir8xVGVfAyQynJcmStbDHDKCo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6494848019a0..6607ed8fdbb4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr)