From patchwork Tue Jan 23 14:53:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2B408C47DDB for ; Tue, 23 Jan 2024 14:57:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=brBxvxQbup7tqvTIs542EKJ43WoLPH4aTxbtlRC1nVg=; b=B2eF97161Yv8uqkJr3f+7pUZBW N2uU/btS2gZcVufY9E+onpdaFm/JiEesMyZ9Z37baq7c+NDIV16k4/zkVp5gP1RZ2ErsLojG0OUiv 6rmqRIhLJy2TSd0uESbeNB/Sj/DtkwamkfpHHlIJhE/SvXtR0z9f+vk6dPjUErwkeh4Dqvc/AZw4F ueP9pQtOstiEq9IJMqmObAoXIeDNaORoZ9tKAVZ5Q+0c9wznhCAY5COjLHr6mfNcf1irsS37G4+q/ QHA2cQE0w9UE3R+EbqxqtXRB0/OWNJEO5GJqwbdLlib0NJLVNBgJHOHzXaYrZf+NOVRrnSAhTgdPr npO4fu7Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICR-00GudH-2a; Tue, 23 Jan 2024 14:56:27 +0000 Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBL-00Gu4Q-1g for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:33 +0000 Received: by mail-wm1-x349.google.com with SMTP id 5b1f17b1804b1-40e4caa37f5so36938485e9.0 for ; Tue, 23 Jan 2024 06:55:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021717; x=1706626517; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2aHlmoe8TgJ5wk4+0WsKDD2g3W308cUjShrTOsxYTJ0=; b=r5ahGNLhiqcifUU1ZzIv9CYzHd4RdTyyN0pkntkthHw/z8lz/aGrtvqo/2hSdRBnxA uJXGe5187jWtdxEbEDuzwXAWw1Flr6EQ4NLNTT87waIVNgw0NeLulXaFp2aX4QCd0m5m I81YbyPZaVkG9MTZiRmVh2MJ58bSTFvfZLuVL84rKlusgx8DMor41Pv65uToXNTUlkBx bmj0Sp8fyM+t0Ye+foGeNFGr3Sp9J7rf+JsPuKLNYwkqsIJFqt5Sxv3vx4UyA3+W6iCy V4A9IEUA7aMxIKMhFj0jVHExGYF2FMClpnMi6s2Wy8f7uivDzjyzTeA/UUqRXn2giY70 14rQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021717; x=1706626517; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2aHlmoe8TgJ5wk4+0WsKDD2g3W308cUjShrTOsxYTJ0=; b=BQr/pSUV9+oAgK+uhlsn7CXXBU66TTZ3wWnTMsTBkhPTaSbUR7QrtCobDUpCCyWMNC b8QDasseiA8cFLa7VH6K0iRxDIjxpOHMUn9kRj52QvfVE+UwvNOkzloD/xGWipl/YjXQ oIQgRjNhyG/lfMqRjRPUVvq6XHifPzNTySbx7Mub6LZLeNwvOhXzTEznehWXT3FJqgbB zE7pi4tm29pOMZpfKYEjzspqfoPwJr1LdFswllkkUe5xjYhdI733eondc6RVFxZZEK77 gYt9KCZNSqyLlGRSbQOBIkgI3L0qsMeTwIHlnSrioKpgTUCU7JC2aFa/ox1sNwlzlB1P drHA== X-Gm-Message-State: AOJu0YwYbgx92tS2FuO3AcW4rmj4bIwed54grYvCqWLKu+HWVQJ3NxqR N2ViYCfWnv/biosopuqTdOoJOl8mL7BcLER2o1mC0XK7+EoeD+8DhOdby9eGM992ATCt9XijKoE mwRtd5Nz3evqIprHrXe/maiHZuQPKoIz710cj2B71US1dqDENoXDHlhWzHqRg95dxbQrVOu9y33 tXfZ2hKuA/UkNpxLBiB0HmCxrOkWIWplGhXBWVR1Z8 X-Google-Smtp-Source: AGHT+IHKM7j5lRYVmQeIN7Y2QSi1A7eDrcamb522EtvN1TpsmT5kNokoF+G/nLNgaN8zZC2US6G6gYoo X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1e03:b0:40e:b318:bd0f with SMTP id ay3-20020a05600c1e0300b0040eb318bd0fmr17113wmb.2.1706021717637; Tue, 23 Jan 2024 06:55:17 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:23 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3644; i=ardb@kernel.org; h=from:subject; bh=wahRQ463F2BnEgYAw33JvSBo/x55U0C4nIAL5DwX90U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9paerg/LdLoaeOHKxYdbM+X3yxyZYtzBOeSycp8V4/ 5asb2JfRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZgIkxYjw/9Zmj+1kuVLd0/z DMwI2Zq/6fzynJwF+25N3d+8Vm9XrhHDf8c0dV6hBVaSky7ETYqN8H679pef64Gta7ap/ObuWrT pGh8A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-76-ardb+git@google.com> Subject: [PATCH v7 24/50] arm64: mmu: Make __cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065519_596230_8E4A371E X-CRM114-Status: GOOD ( 16.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel __cpu_replace_ttbr1() is a static inline, and so it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error once we start referring to this symbol from asm code as well. (Note that this problem only occurs when CnP, KAsan and suspend/resume are all disabled in the Kconfig but that is a valid config, if unusual). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 32 +------------------- arch/arm64/mm/mmu.c | 32 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 9ce4200508b1..926fbbcecbe0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -148,37 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (cnp) - ttbr1 |= TTBR_CNP_BIT; - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - __cpu_install_idmap(idmap); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp); static inline void cpu_enable_swapper_cnp(void) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1ac7467d34c9..f9332eea318f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1486,3 +1486,35 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (cnp) + ttbr1 |= TTBR_CNP_BIT; + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + __cpu_install_idmap(idmap); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +}