From patchwork Mon Jul 15 19:21:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19A5DC3DA5E for ; Mon, 15 Jul 2024 19:22:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFB756B00A2; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CAA2E6B00A1; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD51E6B00A2; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8E8176B00A0 for ; Mon, 15 Jul 2024 15:22:06 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 47AE7120176 for ; Mon, 15 Jul 2024 19:22:06 +0000 (UTC) X-FDA: 82342957452.20.478A2F2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id 3A4C940020 for ; Mon, 15 Jul 2024 19:22:04 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VP0+2JKp; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071305; a=rsa-sha256; cv=none; b=GENN8W7goUiRCF71x8aowFPR7e2d5u+Y34qMnCCtB4/6PCWFD9OXR6xpmuF/f0dCNEew4G 14q/nqOjhDBNtsMcoXyUxcz4+WUGsLV4udUm/6SXsy69Ix0heEs2T3OSd5UmXMTOhTTfJ0 DY3G07YYfvH2ynYhl5KLBXI7JlQJZms= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VP0+2JKp; spf=pass (imf27.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071305; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TIbZhmDyeGPrqbKoRiCTCUG6/nrhKTfNHRA57Ap3jGE=; b=HAL3UUryiZx7dytXw5T51YYrd+3s9zARew00dPyAnNM1isp+j+xwnPf2c/vdRUnM7+cntN Nm3rYjEKD1HCybzjqrpmRqFKDkNOE4Q9zlkmyId2+62d8Oh3tIoYWLYEqsunpIHHYutEgx lo0btG4W99WBf+xhPf07u778mz56NUA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071323; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TIbZhmDyeGPrqbKoRiCTCUG6/nrhKTfNHRA57Ap3jGE=; b=VP0+2JKpbQTNj9KfJP5+2/o9U+59W/bnl2oBVYQtsZzPgShp9Xtn6gXALbtgW0bBKKE7Zs gV0i2G5CtXFX0wCNy6D8X0unQo5p+MkzUm56XfckT4VpAaXafGVgvwvtNsx8p9Mlizcg53 sq5r1cjk5GFwdOX5q3EKz29GicLlvds= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-53-R-qjuoU8MO2bzD2BYRue6w-1; Mon, 15 Jul 2024 15:22:02 -0400 X-MC-Unique: R-qjuoU8MO2bzD2BYRue6w-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-6b7740643fbso2326926d6.1 for ; Mon, 15 Jul 2024 12:22:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071321; x=1721676121; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TIbZhmDyeGPrqbKoRiCTCUG6/nrhKTfNHRA57Ap3jGE=; b=wlF1lk98elgMsQvAoOOfU+2giJd3TE5AyNMlo/2GoE5eGc8brYgKx1aKYb+vOcHTTA L9eFuC+ujgZrygIbBZ/yYh1AbeGT5b3R6Sp9bBKVn2Fxb/oF4ImuJLdrs52QXYy3+hyf kI407SMbC2UWHEREwrkhTZNXkZXcx2gC4EV4Iw/iygEOx7PWawUtMBjT3TJTO5MQBxqh WUeePA2Bm4dPzKp9mLqMEgsxovydRxk87F8npS+4cysHmUYQ9YCDRAtu3rnP7qtA+hwE 8e37ukyApDWVFeHdWb7C5eC7DEwz9D3MpaTERD5MDqKJQwlxA7Iu26Ag2DAGllPwptqU KvWw== X-Gm-Message-State: AOJu0YwCwGcWbKhb9BalKGP7jpbVIUcJRAOhZMjIkz7Tj6o+iz5xitdR /Zqg4F4cX4/E0DXRThBZYjbQCqkpD3zGgUgwCZKMlOHnDGyCBte1mVkzHh/iuhSFML14/TJRLnb z+l3ES3Gq00kmNWVTLbjcrkklbbTKMVGVqoE8aI68LWSfdbDvZLPN332OGtq7KnVc1dpSFkbWxd Q+Zj+M7kHvDyefpS9sfMBv67tQlNFwrw== X-Received: by 2002:a05:6214:5d0b:b0:6b5:38d4:c9c5 with SMTP id 6a1803df08f44-6b77df20d64mr4194976d6.4.1721071321213; Mon, 15 Jul 2024 12:22:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEuNyAqmev7NeecbFUrfKh3dH4xJxhydhorfZ+q73TgeQxZK3QM+o4zZ2VKk8paCO1QT0QczQ== X-Received: by 2002:a05:6214:5d0b:b0:6b5:38d4:c9c5 with SMTP id 6a1803df08f44-6b77df20d64mr4194626d6.4.1721071320743; Mon, 15 Jul 2024 12:22:00 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.21.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:22:00 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 7/8] mm/x86: Add missing pud helpers Date: Mon, 15 Jul 2024 15:21:41 -0400 Message-ID: <20240715192142.3241557-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: rq8x8ywghryt7et4yiz55k5hg5k9appb X-Rspamd-Queue-Id: 3A4C940020 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721071324-963366 X-HE-Meta: U2FsdGVkX1/0TBFhT6a0c+E8l/pFg1enqlkXAfaEi6wwzXCR/ZRehR7nY8Q4JArUn3AKePf3Qomx5ZRcZ38K0EWxuXF2oYSo1p51FuYE9lSisEJSsN1zuKdimgL2B3w2fms1KGTLiqASzuaVX5z1lPXcO/m8fB+9wXMShhdDH9B/reEnwBD3E4j2BopOyMlkdwVmVaTxQ2EDiJaj1y2oftk3dFFd7UnnYB8ubuxr1XOFd0lMwPgPh3Uq8a94MYn+8I6hwOSLqbc6iQezdTN/oAqeuJuCifKJUGseLZ4+wzNU8Z4GZ1OJsodizDgYJlhmAxLe7F108IvZn/ihRfcvowOyxAXZnmdFMz8LTZkLWd+cdspgDbAGk03ToDYOUCWCObCSSsVxaFsOhKbA+E6icqt9oLMfRVSr0ZUMqqRhIc7bqf1oMcEgVn4jOpo1qjDrI5dc1c5Gu/f3sTybh4lZYliVkbFgcXpsTorSbhwwipi784lBchQMNp0MANj7wPGsfRlyxeOh6Bh7l3DVtH0TgUj1DvxCi7Yw2YvMVimC8YHZNJKHSU3yp6+7q8uQ1LtqrdBOQDO2FFCqYrV8eAxs7TChSDhptUezNxH0KPLJWelWmP7RuIChzQ6w51jJqZkWmOsu98ZkOOtMxVgm3Oy6g1o3HHQbkQJiJqXuuR+Dc1S4GsOsExsNVml0U1SfzIZQjzneI7a4IbuN47eXpSADkQpahbRSGqu2pAJ2MDR+qRr/JQWxeU+XhM3Dgduahx3z7ze+ZjcHzxCs+ayydewEgBC7cEMzQ7rYzw0kEpyJsTu3K3Th8FtmAZMBCkR0aoaHSOxeh715RrTv4PWv65NOUj4Ue5RS9f3bqbj6+SzwrX1PONA0rd7qMviaQSW4ppG+lgA7CiX8jJFDQ2n9tKi4+k1CuWrwHA1Ic9O6G85X0kHfvcSRAvm/zdwgrI/0tJu9RQlbJatN3MxAHMjJAV2 4EZg1wUT RQ2qwDf+OcMovxdRq3+YDqSZUNca6HV7BJ7qjaIQEaqTM82+L2fRAj/1iZlJ7dwDoMqixILWn9WBVS5WwcttogfVI1f0zHjiDPdZ2J7LVyeqRtpxGWqmBQae9SmmPxQZSCNLCNCz6yKOB93/Snj4XznufuQFU/O8nNTAvdtU8+14h/FY/6fDDQiVc77XtvE30Or2qf/Yax9jeGOUvbSj+klO81FhTnXcs0QvLOCBXGOHj0CFgoSlZ+IxKn7s36xFo81l39gZrucMwnRb/C9dcrrIGfjyKaPJ86vr39U7e843YWbFVh2lDbdlfR8ReuDnDwS/7XY6XntYI54a1SBINXs+Wmb+ehMvdhaVSLkUh+w/Y0MyUSBIuo3rBiFVvbzKamJaxQ6Q5cFcwO83uKhwevT2MSw1DAA/H1Nd+EIFrHTBYWj4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate() - pud_modify() Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 55 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 12 ++++++++ 2 files changed, 59 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index cdf044c2ad6e..701593c53f3b 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -782,6 +782,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -829,14 +835,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -846,6 +846,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1384,10 +1407,26 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 564b8945951e..fa77411bb266 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,18 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve