From patchwork Wed Jul 3 21:29:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13722803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E99F8C31D97 for ; Wed, 3 Jul 2024 21:29:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F2396B009A; Wed, 3 Jul 2024 17:29:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A0A56B009B; Wed, 3 Jul 2024 17:29:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBF526B009C; Wed, 3 Jul 2024 17:29:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CC1346B009A for ; Wed, 3 Jul 2024 17:29:42 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 90CCA120852 for ; Wed, 3 Jul 2024 21:29:42 +0000 (UTC) X-FDA: 82299733404.12.40806B8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 82E4B40015 for ; Wed, 3 Jul 2024 21:29:40 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jLtMgAo0; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720042169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VBkUbHh5FhI05CAEJEjFs15FFfGPI8FVTV6ICfAryLc=; b=ZboMKlWYyrC5muiDhql0DiinwXDEGeyfCNBWyNrZf98wEtep41vKeudKWEFTFxIqNU9lBq vPkf8c3pNqtSEVFzc+Fw9sq1RMpqrGfJOcxQLyCamK+5mbVSpzkdPhgX5KNjKn35EH1x0m TUCQYp0z2sDfoiVdFdq/v3yvxWBxQf0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=jLtMgAo0; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720042169; a=rsa-sha256; cv=none; b=SFQLzztygCh//FkUQyzDILO07quz3PpzuCFvmRcvgJARNzEXDm0UUzfm4BE1/u6Aej3Gj1 BrZY/arAGcnRO0OHS+OV8U2R2QwPHbcnUmDDBE4v9VWQ/kVFnTj4W/OVDyx8+W2Z/af/O6 zUyRmi2cPFQJKHQ9kChLjMLO4U3XGYI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1720042179; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VBkUbHh5FhI05CAEJEjFs15FFfGPI8FVTV6ICfAryLc=; b=jLtMgAo0xKJ3azfLtIdgREGcNQHoH9aoyUJMsUnjb2tgjcac5T5sucMGVACx5TeKwOfDrQ Xnc1kubsEHo2Dvru01CbuagFLR2Vv7iLw8LK3KP7VCEQSBN2ByhElQTE2sKDa+A34ywkzr axEsdP4uCJwkjWSYzZjNgLK3HyjhCiU= Received: from mail-oo1-f72.google.com (mail-oo1-f72.google.com [209.85.161.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-437-LOrlaqfoN6e-hb39ELTxWg-1; Wed, 03 Jul 2024 17:29:38 -0400 X-MC-Unique: LOrlaqfoN6e-hb39ELTxWg-1 Received: by mail-oo1-f72.google.com with SMTP id 006d021491bc7-5c479ad9c8cso171261eaf.2 for ; Wed, 03 Jul 2024 14:29:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720042178; x=1720646978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VBkUbHh5FhI05CAEJEjFs15FFfGPI8FVTV6ICfAryLc=; b=VDitrU8Za8sJ07tHqDkHfkPGazLtu4EcLCfD0qRQzjNCbgpm1f4SyG57qL4yUfw14G /EbKE9XAhXgjHyo/rLrgmnSSLUrECE7etBtOHPYNmC05V1sDmXIdfdicHRF7r1ULPRxn 8OFauPHEOApPw2xsWAXIhUFpHfFjCaJjyOBCY5qA3yP/FQMZkhsJlAxPtdDKyVh3qkod hoKZw5PVXyoMmZSfP31hHlN0tETKquXmKCrQZrcObM6EeFKBdaTpQLgSdZISdcOXxiYJ my2xnM7FNO0h2ToF1RdNoUne2ZOlsZ4BYLVlKVXIwe7nr8/VfnbQv2vW7QsXJFm7HeVu jJRw== X-Forwarded-Encrypted: i=1; AJvYcCWOIduALJsy3lTFZ2A+3ayK2ev0IPv9Lp9xHQCvivb4Jc9CSZX0zo3WoFB46Y02/sljdWUEYeH5ONb6ILFWWiWaNsk= X-Gm-Message-State: AOJu0YwE0rdlBnop22oIwJXcIgYT+im+lmjtlXKz7smowp8eFsMuKUEh gUJ18n4gMbMe4lF3R036aDpQrtFjFuhmpx0cuU8bqd86BqRDMQ207U35i0ftbIfml1/mGkafGdG g1P2VpcSCF1Jm2tPBgXI73Fs99vdFpasdolJuSgfRgfel388s X-Received: by 2002:a05:6358:7f0d:b0:1a2:3bde:1425 with SMTP id e5c5f4694b2df-1a6acf18b33mr1469440055d.2.1720042177858; Wed, 03 Jul 2024 14:29:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFh/EdB/xSzquRBRoGgcbcKdNPV887AakpSVBbTc4I0BiDNq1hD4sw71GTYj4zNAp5JLHYzHw== X-Received: by 2002:a05:6358:7f0d:b0:1a2:3bde:1425 with SMTP id e5c5f4694b2df-1a6acf18b33mr1469428955d.2.1720042175430; Wed, 03 Jul 2024 14:29:35 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4465143eb1csm54337481cf.57.2024.07.03.14.29.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jul 2024 14:29:34 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Mel Gorman , Dave Jiang , linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Rik van Riel , Vlastimil Babka , Nicholas Piggin , Christophe Leroy , Andrew Morton , Huang Ying , Oscar Salvador , "Aneesh Kumar K . V" , Thomas Gleixner , Dave Hansen , x86@kernel.org, Ingo Molnar , "Kirill A . Shutemov" , Dan Williams , Borislav Petkov , peterx@redhat.com, Hugh Dickins , Rick P Edgecombe Subject: [PATCH v2 7/8] mm/x86: Add missing pud helpers Date: Wed, 3 Jul 2024 17:29:17 -0400 Message-ID: <20240703212918.2417843-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240703212918.2417843-1-peterx@redhat.com> References: <20240703212918.2417843-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 82E4B40015 X-Stat-Signature: bjap5g3ocsamw4picbfqpanjc31i6xy7 X-HE-Tag: 1720042180-551706 X-HE-Meta: U2FsdGVkX1/LQyrZpSqPLzdFitDnKpk2DLElcTW8gM+0Qyxr6s8eZ0RlymlSLvu9G9H+xd9K+LTp8u0dqN1/IABfMcj783uTlLlR41+6iMR/7+NJKEfm3R/nWW/hlZY4J7rZBKVURgjO7e0WAfQzhg2JM31qJsup4MTE5/5BbzDLwaXPXCU5t9HN6oZ3V/2u3rU83AFmYRhwqTwM7dWT+P+ALONa24gyKM0E2pbllwTCbOqfTWpdejDHTGX8Fz+L0VLsZB+bsTsY5icikYJ15c1N7EyrXnCBsloEVhs2sSkYVaI/WdFo+LaEnKxmV8pX6UkjIp65Yv/JyhS3FVP+45j1y9Ho03f71tBNoWfCy0CTQRjFt7rHmF/aJI1GJJn1s6eQKvhMS9pk9L3km7ocfR3n0AcHq4wFkQqx6MgefqGWRv+/dNNg52p4xAVcuc9cxZ6XJmw4XXv4Hv2ZW6R7IDuzWxZYuH1VbC0oaUg4Z9PjqeoN/hrr8lFQNjoK0vOFbg1LUIcdS6eTpWvWRyHK06jLR2Eh49QFuqQawnU3yjtRtaAtOrh7+P56y0PPJzQtemqxkz+CWOAIF5ZS3QViLUTmnIL6dfaZh3+7hTZVz+fUenJZS4729Ld9lKsUC5CAalTFD9lAkomQGN9fVaXCqiXe7Q8eewuLGkJJgMqzr6dbol6SJxTw3WwtGJTwxLYXszub5RTe09kRl5E2qKl35NnKz4Z3ltOaQuLmR73iK4sLqFKO4OoajbIl5rNNdeETPp4YlUVxkInTtemrhd34u25drpVDb9PncUIj/3lzTAZe2VweozO6oeENDDdMLw/fMSxAyCpQG2X6v5dRFEvrCSiAShTCIXHy+iT8SPz06YMQZCT3N5J8P9a//6sWQtCQMJY7XPwn0ILA431maV1pf2n5GW26Wis2vqqH8vaKVsMrUL5n27MAsk9XaroGHHFfpMOxsMVU09VJjxG71sz 2n20HbJi CNMfRpdnqRAmU/MGlWgk5QAjW6OqNw/0yhShKnXXj7owvBBIl+FoH2pM+OurTCTTa0RKiNZkHfhGhYhFa0PRDrBzY3s/t0GshlpJ/7h72/H6PSP3qiwWsRN8Fyb+HTZq9HLaxksNDcv8GGyOGt878RuOs1Nti/jDWT0edeK/uYQ2TPCuw04vMf+QnR5NQIUluduvOixDN7yCjYavoB7dqh0hxQfxXA10zlnncbc7MBqgArNrCki96rM8jSpLHBkZAksDoZyLlGyIir7Pt82rVu647UhaaTwwdb8jPqk7TME0aMeNV5By0ao6yWfGVuCoizQXr8GsaBVAMZp9Hofq3MPzifRppLvHBMeddQ8ani8FSWXjtpj3vivS7Uw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 55 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 11 +++++++ 2 files changed, 58 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index cdf044c2ad6e..701593c53f3b 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -782,6 +782,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -829,14 +835,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -846,6 +846,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1384,10 +1407,26 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 564b8945951e..9d97a1b99160 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,17 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve