From patchwork Mon Aug 12 18:12:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13760912 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65439C3DA7F for ; Mon, 12 Aug 2024 18:12:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF8BC6B00A2; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C323F6B00A4; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAC5A6B00A5; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8408B6B00A2 for ; Mon, 12 Aug 2024 14:12:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F2E854036A for ; Mon, 12 Aug 2024 18:12:48 +0000 (UTC) X-FDA: 82444389216.06.47D9967 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id E26F6C0013 for ; Mon, 12 Aug 2024 18:12:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FqBb78o8; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723486312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j48ti3LHCtaDfvs8kVCshb+r8YjKqoQPQoRoRoS7ZWU=; b=bDo/LXYu48AuDG6oBqstxsE15Zf5iHf28eipZo75zYP8rnDCpCev808b2VQABfZLIQmJMX hQYiWQPER3gHF4zcQTPhe8AGBgRQ4ycjltJ3QsacKcALiDyv7V08m9kxqq8WKqOgBBcCuS E9E6dKsWQCh4suLhhivP+N7fIsLZjF8= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FqBb78o8; spf=pass (imf28.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723486312; a=rsa-sha256; cv=none; b=cicVDhRbSOrf0kPrS39MXl+aTVQtcf3SYgdcGQEH+gZM4dFDkg86fFfYDd4xaaUv4Q7Pa/ J4023SW9MexIabTtPniLazjJqPwdMKc8K9zvmC/EEmRCa+dkhhA44xu5wixtFkpS8EJZzD zttrJyHl94mbXh30Hsd7okudUetO1FY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723486366; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j48ti3LHCtaDfvs8kVCshb+r8YjKqoQPQoRoRoS7ZWU=; b=FqBb78o8PXUVqSjPeJBYDDx/gQ9Ck1yIyPzQEe7cCZS3h8iQhz+d3t93dZksd+7LXPKA95 IEnEmvc5BHs8Le4RK4AOuq8QZamR8pxcV9f5RiY2iYKgLFs2KJ3/17rznE6nMx2remSJ5u OzIpnTPdaXRPhCClknvXFb9qnscMQCY= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-9-TJFp-bSnOZOY0DCXffwZOw-1; Mon, 12 Aug 2024 14:12:42 -0400 X-MC-Unique: TJFp-bSnOZOY0DCXffwZOw-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-44fecb3dfd7so3082641cf.2 for ; Mon, 12 Aug 2024 11:12:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723486362; x=1724091162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j48ti3LHCtaDfvs8kVCshb+r8YjKqoQPQoRoRoS7ZWU=; b=nZ7Mc0O1GdQJs0Cc5H5lo/sJts9SJA9f/78XdoFJeqZXEEeWoQg7wm/MfpbSDnl6pO HT99NF3VSjgkpNzuao41+YmhVyhqB8gAJrFQ54koza9xizDvMZxZFr/QAc21+0I3zCUK 6LFz5VhSmoeBargGQspI3Ky7w0kTDgp/4FOU/P7bjx3yFuYhY6OZol9gsDK9+PPmbZPR g4T9ufxkeDE9Sw0vKDDjotQR7Z50bMFI/6kM7iU7bhVb91Xxdgx19UvhNnd53fOmoqRS TJdoEIfoNo5XOXaksuKGem4fU+X9YDyAb1tPiZ1XWdSqauP6yE8u2IQ5TbeqaluQbP1f biWA== X-Forwarded-Encrypted: i=1; AJvYcCWGmElNJ8NzW0LN14IyGa2H736iKIY/EKMZQljHwJUW38LZHC+5EwuZB5Qfihf7dD/DJzVJ1mmtuw==@kvack.org X-Gm-Message-State: AOJu0YxJtny9EpExyMUYzW4m9EyquoNLLHoq/N/Mwz1qWBXYO5kYRqmS C3jpRSaWeJSQTyyN685A5WEdoBh86z5p9p9uz/8tKlwNRk8UbTsxZO83C5KagSw6wq4/PyrS4Ow 1k/lXS+o/Gc3QkvP2ttq+J5E994itEBCHfa5ufx7O46APWufT X-Received: by 2002:a05:620a:4609:b0:79d:6273:9993 with SMTP id af79cd13be357-7a4e15be5b1mr69031085a.6.1723486362333; Mon, 12 Aug 2024 11:12:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFnMgvJOIkVw2s4SdK0L87wmxjCzEZ4UjgHz1yAaoa3V+0fWl6Gh90ffNusNtYhZ+374uYqxw== X-Received: by 2002:a05:620a:4609:b0:79d:6273:9993 with SMTP id af79cd13be357-7a4e15be5b1mr69028285a.6.1723486361844; Mon, 12 Aug 2024 11:12:41 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a4c7dee013sm268663985a.84.2024.08.12.11.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Aug 2024 11:12:41 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: "Kirill A . Shutemov" , Nicholas Piggin , David Hildenbrand , Matthew Wilcox , Andrew Morton , James Houghton , Huang Ying , "Aneesh Kumar K . V" , peterx@redhat.com, Vlastimil Babka , Rick P Edgecombe , Hugh Dickins , Borislav Petkov , Christophe Leroy , Michael Ellerman , Rik van Riel , Dan Williams , Mel Gorman , x86@kernel.org, Ingo Molnar , linuxppc-dev@lists.ozlabs.org, Dave Hansen , Dave Jiang , Oscar Salvador , Thomas Gleixner Subject: [PATCH v5 6/7] mm/x86: Add missing pud helpers Date: Mon, 12 Aug 2024 14:12:24 -0400 Message-ID: <20240812181225.1360970-7-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240812181225.1360970-1-peterx@redhat.com> References: <20240812181225.1360970-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: tbhbrmgdddtg8q3dpajce94uxuqztzq3 X-Rspam-User: X-Rspamd-Queue-Id: E26F6C0013 X-Rspamd-Server: rspam02 X-HE-Tag: 1723486366-452657 X-HE-Meta: U2FsdGVkX1/XkCadHRMM/BPXoxoNi4IAjO+DDV13jQEmn+XU+5yXdnYK7fwdM5WOZfm/LRoEznfFLGN7HYXCJ1RF8zDwCLcAr5MvpHbtsRf5vCQ22mjxMxHm8kFpHHfKf160FiDxxHkR4WjdklISaZ6yH9SDSlEik/5I6bkIPhrOMb3rS7A1ufmMliabewm6UugjZcA3nCfFphA6T41lYjVV8q7St6C7U0XQQboow4gDbl+aFKbuZZKgFRC4Ho3huc2jlKPuONxy5iY6SExNlKZUC5CnpwCbpyKH4vSdTJF8MM9q3hT9kJj3YaRSc8bOAWyDlV6x+9B5+oz3IPOw8DzUUPub/DBS9gwpph924I4WJW3cFVCFQUFg+ts/5gEVIEXw+Ase9HuSJmYwrK5sR1LyG14NI+R9FHn1bkLTdzegI4zHAuceqJh6LPTaMdfSgGvE55mqgCpd9qou7RihEBniED2OUDcPdfduVCYXv3bGF9KEwVbUf9DGB1v32Nd4zfydBpOprDhu5FMMv2GHeG8Vnth+0geTFDeGBjlifyLeqnhoHBMM1O1uSV1am7RptJCv5yUwfLWaoXJglP8dNdrQe/ohAjKPbzRYgZ4WlJ0UJsMkCdQDjSdn3ikFFqJHYcttKhJZwVVVXF/hh5y2MkJBFOeQ7jjXQ40mFkt41zLfw92nVpOBwbEKey5o+JiwcaxAMZNXK+oHAcyEXfGoccBLBWdgHA9R1ddnu6WW3o17YXeaahUNJdyvE0BBcFxHBA9D4jrZhO66nU580nd/BNrf0Brk7FwvHhb8VK+sOeqcc0fGyL5/JzgjNZucq3Q+ecL13/GHjkpW/CErOygaUbHXhtc9VhEbtNmtvblFDd214OOJgF8xeEcJFIsgUHnIL90ZSDlsaY9Y4phjpTMsgrQDv/xfi0J4GOa3JLIev7naEqsIxCoPc2Ui787vODC5hDuUMtCOb2Y0CzLICqp 5ekaTtTa bG6Qqof/Kjw+G4Da0IiCRZx3sstRmeV6w1ER77DDvubPVr4cqPdQsrTaDbHRuVjsm46XUqx8Q19lJD1Enskk4dOpsu2eWFnNbmgm+Hw/HgTh0xJn+T2AFdxII+3vpMZiRIP5pogcBn0kn5Za5LsuR0jHBQpH17QgdELjX2ePqlquP+2ZIzI3xD2O21xWQA/FW3YahpZg2pF3vhVwy6yWOeV9BAkESiCi20MQN8Pdf+51jCohkAFCHU4TYJSseaqrQbwumB+A5WgYUUzKV60+tIu6hSANgZT5G3wRWJzfUn7gRujxGwfiTg/R2pn0LP7ClQJqtx/gX26FGjtBj29AGyu+gvCzncu8M5ZrB+qhIUgSoH6d89EFUbzmOjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some new helpers will be needed for pud entry updates soon. Introduce these helpers by referencing the pmd ones. Namely: - pudp_invalidate(): this helper invalidates a huge pud before a split happens, so that the invalidated pud entry will make sure no race will happen (either with software, like a concurrent zap, or hardware, like a/d bit lost). - pud_modify(): this helper applies a new pgprot to an existing huge pud mapping. For more information on why we need these two helpers, please refer to the corresponding pmd helpers in the mprotect() code path. When at it, simplify the pud_modify()/pmd_modify() comments on shadow stack pgtable entries to reference pte_modify() to avoid duplicating the whole paragraph three times. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: x86@kernel.org Signed-off-by: Peter Xu --- arch/x86/include/asm/pgtable.h | 57 +++++++++++++++++++++++++++++----- arch/x86/mm/pgtable.c | 12 +++++++ 2 files changed, 61 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index fdb8ac9e7030..8d12bfad6a1d 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -787,6 +787,12 @@ static inline pmd_t pmd_mkinvalid(pmd_t pmd) __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); } +static inline pud_t pud_mkinvalid(pud_t pud) +{ + return pfn_pud(pud_pfn(pud), + __pgprot(pud_flags(pud) & ~(_PAGE_PRESENT|_PAGE_PROTNONE))); +} + static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask); static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) @@ -834,14 +840,8 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) pmd_result = __pmd(val); /* - * To avoid creating Write=0,Dirty=1 PMDs, pte_modify() needs to avoid: - * 1. Marking Write=0 PMDs Dirty=1 - * 2. Marking Dirty=1 PMDs Write=0 - * - * The first case cannot happen because the _PAGE_CHG_MASK will filter - * out any Dirty bit passed in newprot. Handle the second case by - * going through the mksaveddirty exercise. Only do this if the old - * value was Write=1 to avoid doing this on Shadow Stack PTEs. + * Avoid creating shadow stack PMD by accident. See comment in + * pte_modify(). */ if (oldval & _PAGE_RW) pmd_result = pmd_mksaveddirty(pmd_result); @@ -851,6 +851,29 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pmd_result; } +static inline pud_t pud_modify(pud_t pud, pgprot_t newprot) +{ + pudval_t val = pud_val(pud), oldval = val; + pud_t pud_result; + + val &= _HPAGE_CHG_MASK; + val |= check_pgprot(newprot) & ~_HPAGE_CHG_MASK; + val = flip_protnone_guard(oldval, val, PHYSICAL_PUD_PAGE_MASK); + + pud_result = __pud(val); + + /* + * Avoid creating shadow stack PUD by accident. See comment in + * pte_modify(). + */ + if (oldval & _PAGE_RW) + pud_result = pud_mksaveddirty(pud_result); + else + pud_result = pud_clear_saveddirty(pud_result); + + return pud_result; +} + /* * mprotect needs to preserve PAT and encryption bits when updating * vm_page_prot @@ -1389,10 +1412,28 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, } #endif +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +static inline pud_t pudp_establish(struct vm_area_struct *vma, + unsigned long address, pud_t *pudp, pud_t pud) +{ + page_table_check_pud_set(vma->vm_mm, pudp, pud); + if (IS_ENABLED(CONFIG_SMP)) { + return xchg(pudp, pud); + } else { + pud_t old = *pudp; + WRITE_ONCE(*pudp, pud); + return old; + } +} +#endif + #define __HAVE_ARCH_PMDP_INVALIDATE_AD extern pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp); +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp); + /* * Page table pages are page-aligned. The lower half of the top * level is used for userspace and the top half for the kernel. diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 36e7139a61d9..5745a354a241 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -641,6 +641,18 @@ pmd_t pmdp_invalidate_ad(struct vm_area_struct *vma, unsigned long address, } #endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) +pud_t pudp_invalidate(struct vm_area_struct *vma, unsigned long address, + pud_t *pudp) +{ + VM_WARN_ON_ONCE(!pud_present(*pudp)); + pud_t old = pudp_establish(vma, address, pudp, pud_mkinvalid(*pudp)); + flush_pud_tlb_range(vma, address, address + HPAGE_PUD_SIZE); + return old; +} +#endif + /** * reserve_top_address - reserves a hole in the top of kernel address space * @reserve - size of hole to reserve