From patchwork Mon Aug 26 20:43:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 523A7C5321D for ; Mon, 26 Aug 2024 20:44:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6D6F6B00AB; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B15F66B00AC; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 943416B00AD; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 66F9A6B00AB for ; Mon, 26 Aug 2024 16:44:35 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1CCA5141099 for ; Mon, 26 Aug 2024 20:44:35 +0000 (UTC) X-FDA: 82495574910.13.0EC1FB8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id E785D40015 for ; Mon, 26 Aug 2024 20:44:32 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Bx48/gEy"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724705053; a=rsa-sha256; cv=none; b=H2DDQ4C0Ak1F/DXNDU7H05EDOtAuskwezBUigjws0LHRnfHmB+v9vfNwTJgl4mDMx9vTOd rZ9b5kdjkrTEPjeTGQ8PzWo1IG838mOf6VMPsBfXoi2BpuYubEWitWi0Z1BLO0F0SDcpjr C9WTjpNirIIMg4XjHjB4lEGnmsvkdAg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Bx48/gEy"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724705053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=YlHz6DjO+8vApZL6iuAFf4vkfNS5qHeO84Kvycrd2knDjJpfogMMOTloUsZqe8YHe4LTKw zaiYvsc09EWREJMyAb7OClNviWYZI5Myvjo9XMiExlCCmUWtWdR0CZ12OLSBk25O9VTgZu FVuXnf3zY6c4C4PPFbmTMU2kS1WVs6w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705072; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=Bx48/gEy8HcNPSGTC1saLoQpFtRlpAqaHob2hDvDJssP+38CWMrcsGJHc9yq14J+BcKZKB pTw4U6vVcGmGAmneL4EgB6tKASr//AaS5oPgIPuaZj7qp/UMLGNCE8ZZmCxgMsFCl3iKHN nipTN8TOh7lp2ZJCuTHVmvEytRghhrI= Received: from mail-oo1-f69.google.com (mail-oo1-f69.google.com [209.85.161.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-195-KZWYpV3qPxyI8b7b-OtlNQ-1; Mon, 26 Aug 2024 16:44:31 -0400 X-MC-Unique: KZWYpV3qPxyI8b7b-OtlNQ-1 Received: by mail-oo1-f69.google.com with SMTP id 006d021491bc7-5d5c7bfd8aaso6416841eaf.2 for ; Mon, 26 Aug 2024 13:44:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705070; x=1725309870; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IC5gYR+7Z4BwAPnZp8TN8XZEGypCYGRNkok3NV40z5s=; b=vi1h+ZONASayu04LF0WAeIvMIP37v1tp7RB57ftdfB3+6NH1pxh9GVSq5GEBVabyMN Dxn7Q3+CYyOF/Mugw+xn/hf95I2LkL/OX1GOiaD2DRzdB1ilM8QhWA53cHoPxZXSQn+y BplSygofFQdqspm20O0aj/OGOf9rscaDQcjYg04YXWqzQWYHPeccaxF9YA/vQKtQOIoa UEAtXnodhjcuCztqPoWy+1bNjyR9FEY9wAcriu1MsT0QIdSgUYdw4DDGKNKchQqaEKKQ Nth5CP2NvDtnCfMGlh152txFwfVtMsy6nOj84WrnV6mWBixBcNKkepXv9S4bKkH7wfSa JjlQ== X-Forwarded-Encrypted: i=1; AJvYcCUSJwuibA5+YBQlgevrLxjhDiIspjypB4A8GtGzkipfZz6zzQFd/x5ab7L1UtxZdR3FRbuhJM7k6w==@kvack.org X-Gm-Message-State: AOJu0YwbCns+BusA7sVsEw3zq4Eeh3XI8PAnxRuQb50BP/Kc2OEGzpgS NXAsc+za8M4JDDj3tiuhOcHWmh2qVFDwvQOQRsmfksDuRmoHQLHqih0V4GF5gvDLxCehxvPbqpa 7Q7mTsJQ6BTdd4/w/ykzBf//XX9DxSNdDNREGIgYU4r7PXo5o X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493541755d.27.1724705070546; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHWRaHJzCfl4jAxAoYVG6HBrfG1+v3SSmaZHfZSlba6wG4sR68STMcFscyvu7mDZcazVEGWKw== X-Received: by 2002:a05:6358:70c3:b0:1ac:f00d:c8c6 with SMTP id e5c5f4694b2df-1b5c22ebd07mr1493538355d.27.1724705070146; Mon, 26 Aug 2024 13:44:30 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:29 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 17/19] mm/x86: Support large pfn mappings Date: Mon, 26 Aug 2024 16:43:51 -0400 Message-ID: <20240826204353.2228736-18-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: E785D40015 X-Rspamd-Server: rspam01 X-Stat-Signature: z4y33b3yo9a1b4x4r1dnq8h55sigij5f X-HE-Tag: 1724705072-842191 X-HE-Meta: U2FsdGVkX19TfaqS+uPGqtRC4yW6I+ZhrevcZ+huT38zLxLehliH4hMqS34BvCFJAloLFrjcDqq22BXJ51bg7Uto6Y8RJfFsScIsXTbEUwvMThLdkFs1VNwuNpudVGrCtyo3gYRML9YfHFLAj+CzOkQi6euFjaGNE0OL5L92YE+zQ0XL/y1gt6YTckVayxmdi5QkM2WtzngZnwi1F2mf+jzqW8VOMwW1TlcqvJ9CkJhTg9lVcz5dNiPKumuFs26GoCvdFNEc6W90oZiDVzrDe5B6yCpNCrhQOLKbDfgsNR3Bhhi9/6yt/A5wR3QSPkNPlzopvTQIB6jLTDmAryR5BFaYYmWyKrKk4uiDEdZfM/KYA6WR1ZAcZwmZ0RbY25DEUgx4ZB+wnKgqwr/ojTNQyo/+yRpQiRkt3y+vdHew7SXFsgl5K8BCD9KhCVvm/8y5ggB4XeG3d13hzNkfcrXIOZdWsycrQ5DiEcyXBf/wOqoJsqt2aC9V6kIL7cA7KL++ARobe988t2jVt8bUrqipcafGxfjcvUOwnPyFkf+nNlFvycjbTqadfEymc+hpSf5ALMDMKRNR7w4/M/DsuXdJOfQDu+DA+FZz1i8tG108XCkNuUTc0+8uak4ujItCJVGCQtz4WS6Q8BVc1uyrFxry4tzXBAa9Q1UFmgT/TyToAd/VMyrbUvkAQN475AbE0a8UdQTgrVY4pmeVFADkh+SF7w+rzd4O/SJy8WH5YjJ4L7WecnQc0NSRXH3EoY0vH6hw9RpFK2W7WCCOg3KxfTKkid6eo391WlV0lNwz0+clBzLe/2OukVDqdrK5I3DgTRwWAUZuqkg4jIIjcjaoUKupx1+gYV03UIbkeS8NIXlIXoVho9Y2tzFB9HBrgiZo/PmfpUCaF7Gsz5fu0kKIEHJQnNqHKwaovQxmC6Bn/ivsHXpD1DDfyOGoqCB7WITcuktJefsmGZZVRg0xaHF3SWj PZjk3xe9 umBmw7Ug0x2dhPIu9llBRhraakcIsU1q4rYKZmKkNylroQvEt5iV3zWQClcnOTfssOMCBPhXRztf7KFPEYtY2iCTe9MV83LLijNyoJsl5auSw+P2NXtySP5I3grHNZS78pD3BLZkp0pOIvWVDgyg/uPYJ5pG4xdITcHxJ9yuljLxelFw03nP+vxumQw08MvB5qmvfE2j6b356Ih/ZeONd/Y9jOjiHwo7AXPmKXekMyajEyfWYE4Y/9t2QXUzIcpJ+4Wt/osGa8LrZNL6QpYbYc6AVnxRMnIo6HCq4OutGBYdO86jexEfp02n+TlZaoG3/Jq0vchVUTzOdOXnL3q/FExeNp7HtQRATAldfVt0Nwqf+WRrNrFrq2WXdOJ5js+IZHjnxUMMCtiNSXRF3gnrjmQxbSSG7QNoWo11+YXE+lOsn6h8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Helpers to install and detect special pmd/pud entries. In short, bit 9 on x86 is not used for pmd/pud, so we can directly define them the same as the pte level. One note is that it's also used in _PAGE_BIT_CPA_TEST but that is only used in the debug test, and shouldn't conflict in this case. One note is that pxx_set|clear_flags() for pmd/pud will need to be moved upper so that they can be referenced by the new special bit helpers. There's no change in the code that was moved. Cc: x86@kernel.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Signed-off-by: Peter Xu --- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 80 ++++++++++++++++++++++------------ 2 files changed, 53 insertions(+), 28 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index b74b9ee484da..d4dbe9717e96 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -28,6 +28,7 @@ config X86_64 select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8d12bfad6a1d..4c2d080d26b4 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -120,6 +120,34 @@ extern pmdval_t early_pmd_flags; #define arch_end_context_switch(prev) do {} while(0) #endif /* CONFIG_PARAVIRT_XXL */ +static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v | set); +} + +static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) +{ + pmdval_t v = native_pmd_val(pmd); + + return native_make_pmd(v & ~clear); +} + +static inline pud_t pud_set_flags(pud_t pud, pudval_t set) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v | set); +} + +static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) +{ + pudval_t v = native_pud_val(pud); + + return native_make_pud(v & ~clear); +} + /* * The following only work if pte_present() is true. * Undefined behaviour if not.. @@ -317,6 +345,30 @@ static inline int pud_devmap(pud_t pud) } #endif +#ifdef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP +static inline bool pmd_special(pmd_t pmd) +{ + return pmd_flags(pmd) & _PAGE_SPECIAL; +} + +static inline pmd_t pmd_mkspecial(pmd_t pmd) +{ + return pmd_set_flags(pmd, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */ + +#ifdef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP +static inline bool pud_special(pud_t pud) +{ + return pud_flags(pud) & _PAGE_SPECIAL; +} + +static inline pud_t pud_mkspecial(pud_t pud) +{ + return pud_set_flags(pud, _PAGE_SPECIAL); +} +#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ + static inline int pgd_devmap(pgd_t pgd) { return 0; @@ -487,20 +539,6 @@ static inline pte_t pte_mkdevmap(pte_t pte) return pte_set_flags(pte, _PAGE_SPECIAL|_PAGE_DEVMAP); } -static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v | set); -} - -static inline pmd_t pmd_clear_flags(pmd_t pmd, pmdval_t clear) -{ - pmdval_t v = native_pmd_val(pmd); - - return native_make_pmd(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pmd_t pmd_mksaveddirty(pmd_t pmd) { @@ -595,20 +633,6 @@ static inline pmd_t pmd_mkwrite_novma(pmd_t pmd) pmd_t pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma); #define pmd_mkwrite pmd_mkwrite -static inline pud_t pud_set_flags(pud_t pud, pudval_t set) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v | set); -} - -static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear) -{ - pudval_t v = native_pud_val(pud); - - return native_make_pud(v & ~clear); -} - /* See comments above mksaveddirty_shift() */ static inline pud_t pud_mksaveddirty(pud_t pud) {