From patchwork Fri Aug 9 16:09:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13758943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51028C52D7C for ; Fri, 9 Aug 2024 16:10:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE7536B0088; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E74F86B00B6; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4CD66B00B7; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A2C516B0088 for ; Fri, 9 Aug 2024 12:10:01 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 47ED84129F for ; Fri, 9 Aug 2024 16:10:01 +0000 (UTC) X-FDA: 82433193402.26.FAF89DE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 422EB1C001E for ; Fri, 9 Aug 2024 16:09:59 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OCiZ3c/J"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723219745; a=rsa-sha256; cv=none; b=pC/3BAI1BJiSBWVH+uM6Bqa9G7fS+1WDpPi8zauaU4eZim2M4dUOoLshezaf2OQ14fHiCk fySIYFRws62hKqEp5j52gV+0c5qupeeDuJZM83BexcwOnrneLX7UzSHQUITJ+hKS4J0Pkl mG/AbYootRnxPww/UE5oaH2qSyLxS30= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="OCiZ3c/J"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723219745; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=21VcYP8EQQlEq0BmzOshQ83u0k2rfSsI+QGYjD87BqzYtLIzsD0Y5YJNVrvIaxkJ9xTXkB NH2nyNeCZHLJm7IzHDmhBNeM3N6sg5utTittLvWU5g9ycoZJ2GhLvxq4KWb5HDALsYx7Th JoLSUFRB0MwTYjFJhPNGmfkj8euvwKs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723219798; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=OCiZ3c/Jna1NjWYCT1GAw6hpwYfDsY7rNuFD414H3z9wlkAn7GoPOGaxcVhGsZN988kBvD 9fc8t8SI6qnvsdyQs2OIbC3F3Ej0xcCSqSvzmsXCt13weLjmwGWDnGdFhWwf6UYi+Ahl2b pt4TngbBDCubf2UKv2KT3JR2b+NUt8I= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-221-T8p-ZL9NNT6GhjBeOVOwNg-1; Fri, 09 Aug 2024 12:09:56 -0400 X-MC-Unique: T8p-ZL9NNT6GhjBeOVOwNg-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-44ff585970bso4757231cf.2 for ; Fri, 09 Aug 2024 09:09:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723219795; x=1723824595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UM853E440qfXKYz7avWh673RydKsfl7XbVKvi0Id2+c=; b=KnoLcT8gVTOem67cFGJvHDWuysP6v88iQ4s6leC4q+VWoWRianX9HZfSiCHbpjzdqW qm6tv0gpK2OuZMbtC7CLwFOUHoSW3KmoirohHHe2U8HzJ42Ez9MAUu0IjFpWhE2DdOQn qRIBIsa/SaImoOvzaxulfMjha9dQPdXgJ7RZtiIn+P/+VoReV3jx5MLUbd+rKwAWULJU TBQmTdiHP/SweyrZJ9AlojnCsj3qbkcXyDIyLkEsxzTUC4ukJ9qN0WWvSQs9EC7Zyasx f6M0PjfTBHIAvdAMc1u/yTVMtfXCMsC+MRxUZZ9IChi8eTOJbPLVGMPH4SzvwWctOAZE 42QQ== X-Gm-Message-State: AOJu0Yw4/05SAY72RRck4diQf1UJpsAjK+B+M5/dGzlaqX8nS08CyPdL EyoPQZZzjhIQ2FDCeFM3Vjlxw8aLEXXRGEwZ8XrpY8kzz2++sfuQ9Hkd2wy46scJBpSRS7ma4je BMj7Tqp2kt4PXZJEOhostLxPOW76pZuaIY/nPSbGuKoK8MIobFO9tzejiL25UEsYtClIcMG9bQL WMxVR+4MCWuRfqV7O4+jJHZJwCyWCswz36 X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006871cf.7.1723219795409; Fri, 09 Aug 2024 09:09:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEzDYjFq+v9SYAdrw2G1PaR/i1nnMK5UGwVdyzJwk9mK3TcvaabWRsZT43a6L+rPfA/Jc6OAg== X-Received: by 2002:ac8:5e07:0:b0:44e:cff7:3741 with SMTP id d75a77b69052e-45312646da2mr14006381cf.7.1723219794745; Fri, 09 Aug 2024 09:09:54 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-451c870016csm22526741cf.19.2024.08.09.09.09.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2024 09:09:54 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Sean Christopherson , Oscar Salvador , Jason Gunthorpe , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, peterx@redhat.com, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao Subject: [PATCH 18/19] mm/arm64: Support large pfn mappings Date: Fri, 9 Aug 2024 12:09:08 -0400 Message-ID: <20240809160909.1023470-19-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240809160909.1023470-1-peterx@redhat.com> References: <20240809160909.1023470-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 422EB1C001E X-Stat-Signature: 1k3oywozmzeku9bhkwdteg6d5wa3omw1 X-Rspam-User: X-HE-Tag: 1723219799-416921 X-HE-Meta: U2FsdGVkX19M20AEtRr5/NE0S87c65NIUEi0dG7pUNpKVd7M2PB0UvMl0qqrUTN1Fcp3XtdhlOXL8jDweMyehrPL5VbdOCfPcDFAnufCV8wX8mCnQSVlDLRWRPhoKUI1t1rdGBOeJe8Jb8FtBNgvTJm68mosBQTaEw/BLkL7h32mC+P84+pohltvqTXzR/xTyEBejOJYhnpaQLGv7h+NhhYJpQ83ZM1eN9X6hWrWIssYnoB10XrjYDjz7oXxSLPVcNRkbXsbRn+f4jVk6XzRiMOOZkVb3HDladSo5FLOm8hTNVfVPan08AamV4chi/AwjDzX3cpykdMDdlWw5OefTPJ9k22F1kUEil9VWegJVLuGsESWGsw4ohWMwsYxjynMhgMsFOxEcKNhBr68pIKq6tWnPQ3goLKdGj6zCQJ8cTA99if56wtc0S/kqpa+/m/DuPNcwSuHFuxsbMa9hPqJKLN9FXMTh7QnAJjLzDUOJlAi7cXfvkjBWBbTT9xGx/9dCliRZ93eK8Y7YRPc4Pf3HYZt63LA2RCL/qp+oiG1snB0u/LsFJwWqSs084cNIfRt4ml9wli2JtDG+UnPBy2ZmYsX7sXmoqgT8DwAHH6l6q0krAmWSoe2LffqrkziCgUH9MvxobqvNugHKTKQTDon8ApQ4LEbFxUlr9uXwKNbDVV7e1bXRolb/R2Slx0zNQWhtxjUwEGxVaCnBZXncpGymiJU63jElpn9MjVZ+bF+MbEZ38eOI2gsHtemKOWjK5O+ILUdbpYakraifXVUzRIwnq1ZaSspmU4HeR93XHg1H5AQ5Q4HDY9n3Cn2EKRucCAe3KSV/NPOwbzylZyEON0ALt8PHN7p72168lDN2x8UA32G47OcLxZVSe9aEa+TXp8hxa21OgSdTWWci+f9EHiVh0XUfBNpwqXvzRVT5/9TUccOU7lWKW/RivulOk1GaMPR554ZLlQCgtjWJj1BpaI pZrkRPd6 mLhuQ8IP5gO73HVRealdT+DD76ekM35Jeacee4/OTqwiCT2NLF3qicouUj9rT1Qyi+1um8CiDdb076UCaB1yUSAN027rjceVCTLAbWjLD/TTNSSZx0lKh0Cu4XDrnHwoYFtzmZg5qPkleHDbPVuRinlxgdlJ3PSwuttHRQY6d1mf3zQH3disXzfJldfa7s1MiN72NSzqh0HLQ0cJzUBxYkOcGllxO/98aKJFEAYYnr0uHJiqSi+Cu4+fXgD9H+IPbc8OquekrRwOPkHT+vSSJY3GrHoofnkJ9SMcpdV+WkSSR5S1vFkIzbq+yBRHZJQQlu/begCd0ypwZMmgoO0lDCL09MlgyC/u8YV2+7CVbCfOgKGD8yOZd/r85QN1ZOnbDPJRCFRg+vLPVDrhiAgAiKhmgAwRfjQPc4atwfE4Y726R/Ml8wMM4WZ4c+8o7e1aDpDKQO5RSzaW/Wgxc0HXqJxKSiEAwiDSoLo4b0fFKvXuxHv4obn0gDREunlhloKsFEgIvldhO5iY97JgTA6hAmSmRT7Ptz8Lqb+jaRcUFZptEsELT+mFjE8o1E6gmokTLP4MYLvVLkomXKEkxTVk/Q/mDyCUDBDdK9OUHj/zJcCHcj68= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support huge pfnmaps by using bit 56 (PTE_SPECIAL) for "special" on pmds/puds. Provide the pmd/pud helpers to set/get special bit. There's one more thing missing for arm64 which is the pxx_pgprot() for pmd/pud. Add them too, which is mostly the same as the pte version by dropping the pfn field. These helpers are essential to be used in the new follow_pfnmap*() API to report valid pgprot_t results. Note that arm64 doesn't yet support huge PUD yet, but it's still straightforward to provide the pud helpers that we need altogether. Only PMD helpers will make an immediate benefit until arm64 will support huge PUDs first in general (e.g. in THPs). Cc: linux-arm-kernel@lists.infradead.org Cc: Catalin Marinas Cc: Will Deacon Signed-off-by: Peter Xu --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3fc891f1544..5f026b95f309 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -99,6 +99,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b78cc4a6758b..2faecc033a19 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -578,6 +578,14 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) return pte_pmd(set_pte_bit(pmd_pte(pmd), __pgprot(PTE_DEVMAP))); } +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +#define pmd_special(pte) (!!((pmd_val(pte) & PTE_SPECIAL))) +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return set_pmd_bit(pmd, __pgprot(PTE_SPECIAL)); +} +#endif + #define __pmd_to_phys(pmd) __pte_to_phys(pmd_pte(pmd)) #define __phys_to_pmd_val(phys) __phys_to_pte_val(phys) #define pmd_pfn(pmd) ((__pmd_to_phys(pmd) & PMD_MASK) >> PAGE_SHIFT) @@ -595,6 +603,27 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd) #define pud_pfn(pud) ((__pud_to_phys(pud) & PUD_MASK) >> PAGE_SHIFT) #define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +#define pud_special(pte) pte_special(pud_pte(pud)) +#define pud_mkspecial(pte) pte_pud(pte_mkspecial(pud_pte(pud))) +#endif + +#define pmd_pgprot pmd_pgprot +static inline pgprot_t pmd_pgprot(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + return __pgprot(pmd_val(pfn_pmd(pfn, __pgprot(0))) ^ pmd_val(pmd)); +} + +#define pud_pgprot pud_pgprot +static inline pgprot_t pud_pgprot(pud_t pud) +{ + unsigned long pfn = pud_pfn(pud); + + return __pgprot(pud_val(pfn_pud(pfn, __pgprot(0))) ^ pud_val(pud)); +} + static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr)