From patchwork Thu Jul 25 09:03:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 13741654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3170FC3DA5D for ; Thu, 25 Jul 2024 09:04:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=I4DoFJ/L/t+PzQHq/bUdg1dwiuv+mge5691HcPbnObE=; b=nvgUKd0MUIGUEt9Fa4LbvEumGD gEj4E1Es9bxsNtyd+NdlaGyEGAPLImr9AIg9ByaCsXxxFMky/AozPSCaWn2QYAMulyIBUpBFfcvrZ JK7jTcDhhOBLnYshK+awBXJ0n0JDRlc/n/oejpO0kofAYMESxgsR7Wx+qZwrLDTm3jlV7gm+YYWvy Dt/ow+74L/JdyQ1mMpfCWff99s5QT3R8Uvw8j55VfyO3GlZXr/VV/yFt8EXODt6r0IIk5PRgXTq+k 8dwWqTaFtY3QN0tioQSmjOnH5KKzWG36AS/Li4O7NewTfC00++8gP7YOM6v3iRujBjeSA7Izkoyr3 l4IibJvg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sWuOX-00000000Pis-2Ig8; Thu, 25 Jul 2024 09:04:17 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sWuO9-00000000Pdl-0zm5 for linux-arm-kernel@lists.infradead.org; Thu, 25 Jul 2024 09:03:55 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id DB1476135B; Thu, 25 Jul 2024 09:03:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C80AC32786; Thu, 25 Jul 2024 09:03:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1721898231; bh=ktq7M3foSa4aHhEmT7CQctdVVEd8++vnRRMJLYqAi+A=; h=From:To:Cc:Subject:Date:From; b=ey08YAgWZlA+LDmC0EVMst7itd4EbUyEmIxbMcRw1XcZmSE+klIO6UOZV3jRJOTGV w1a2+BJBRE8X/iEX7wkJkyXe2dJpVymNUAnmeXzIlhUgevIM5nYznmwwMnM87+wjeW WL4vOSR1ECpCeylnwJaZvIFA+u7lNLzTZdg1tznZKRjF61WNRT/DHtJMgUhuENxiEK 6ebv555mTHnaGJ30K230oG2hfjqmSnMb7WgzEuIAXMHEqbeHuEFNMDk1suLBn3tOAc spsPBWUI6IfPOEKGazox3iTdipDapJfhq6MxMFx1gxXEadnydn+EsDsDhxQKrljWWD jdmEVjJDzqdKw== From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: Will Deacon , Catalin Marinas , Ard Biesheuvel , stable@vger.kernel.org, Asahi Lina Subject: [PATCH] arm64: mm: Fix lockless walks with static and dynamic page-table folding Date: Thu, 25 Jul 2024 10:03:45 +0100 Message-Id: <20240725090345.28461-1-will@kernel.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240725_020353_486450_562FEB03 X-CRM114-Status: GOOD ( 16.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Lina reports random oopsen originating from the fast GUP code when 16K pages are used with 4-level page-tables, the fourth level being folded at runtime due to lack of LPA2. In this configuration, the generic implementation of p4d_offset_lockless() will return a 'p4d_t *' corresponding to the 'pgd_t' allocated on the stack of the caller, gup_fast_pgd_range(). This is normally fine, but when the fourth level of page-table is folded at runtime, pud_offset_lockless() will offset from the address of the 'p4d_t' to calculate the address of the PUD in the same page-table page. This results in a stray stack read when the 'p4d_t' has been allocated on the stack and can send the walker into the weeds. Fix the problem by providing our own definition of p4d_offset_lockless() when CONFIG_PGTABLE_LEVELS <= 4 which returns the real page-table pointer rather than the address of the local stack variable. Cc: Catalin Marinas Cc: Ard Biesheuvel Cc: Link: https://lore.kernel.org/r/50360968-13fb-4e6f-8f52-1725b3177215@asahilina.net Fixes: 0dd4f60a2c76 ("arm64: mm: Add support for folding PUDs at runtime") Reported-by: Asahi Lina Signed-off-by: Will Deacon Reviewed-by: Ard Biesheuvel --- arch/arm64/include/asm/pgtable.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index f8efbc128446..7a4f5604be3f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1065,6 +1065,28 @@ static inline bool pgtable_l5_enabled(void) { return false; } #define p4d_offset_kimg(dir,addr) ((p4d_t *)dir) +static inline +p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t pgd, unsigned long addr) +{ + /* + * With runtime folding of the pud, pud_offset_lockless() passes + * the 'pgd_t *' we return here to p4d_to_folded_pud(), which + * will offset the pointer assuming that it points into + * a page-table page. However, the fast GUP path passes us a + * pgd_t allocated on the stack and so we must use the original + * pointer in 'pgdp' to construct the p4d pointer instead of + * using the generic p4d_offset_lockless() implementation. + * + * Note: reusing the original pointer means that we may + * dereference the same (live) page-table entry multiple times. + * This is safe because it is still only loaded once in the + * context of each level and the CPU guarantees same-address + * read-after-read ordering. + */ + return p4d_offset(pgdp, addr); +} +#define p4d_offset_lockless p4d_offset_lockless_folded + #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #define pgd_ERROR(e) \