From patchwork Tue Sep 12 14:16:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 86F91CA0EF2 for ; Tue, 12 Sep 2023 15:30:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nODrTtYiyJbWJhr5HU65zZ5pDg0QqzIemxURCtF/uN8=; b=cbR10GjmRmu00jOuV47VXhtvV+ p+HzALFzrplSQhGY+hmut5bzN6qG7FOoucBwDuULVJksoemUTmYHBkIoMIxYwPJCRgWiPVE+yznqH B3juyQzXQaFkb5hClNzcs3zy7jlPXFDqxIyXQYr4GQR6nFUjR/7XQPJli/lLkXUkDVhGy22ZawDmu LCuQOcH+oF+8EeF4M96G9RnvdOKxGwJX/j77sbLlieuJtVGb5IzSu9p7mqVuGBh8UPLVIl531hwa+ U3gzXGji5BkxEqnnwj6Y0uVRpMSSgryo5It6n7vkNJZ8b+CdoFABhFcUw2LO5as0R8s2rrO2c+14p c5eXc83A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LQ-003ih1-2U; Tue, 12 Sep 2023 15:30:28 +0000 Received: from mail-wr1-x449.google.com ([2a00:1450:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EC-003X1r-0V for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:10 +0000 Received: by mail-wr1-x449.google.com with SMTP id ffacd0b85a97d-31fb896a87aso640895f8f.1 for ; Tue, 12 Sep 2023 07:18:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528334; x=1695133134; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=D6wn7wzrJQFrtWJKmHhlzm09wnf150CyBxnVvC6CXi0=; b=13/2x/HN9JXPTkMPqAU+8f11/H+hOxBZZyFZEkIqOpHq82A2eam0N6eFceWQeUKjz9 S09FWooKkKhz99jpljyzlhd6rokPU/7gGtVMQx6axz5MLmj+01xkqiScxd2rB5OvEHg0 znYiQDc9JwijcHRSXp6668PjUlo+e1e4tJ83dqSZXGclIhPOLwDZ/c53dFYl4mJySrfJ Gm6v2JihZmlo1XQvNsiDg0dDBp5WSB1GqknGT2JDxsL6ElDdzXHEhKAO1jt5zPZ6rofo vNg5N8MtutweEXwTZ6qlTPx59GpkdSpaStfWV9qRay3tUDyThOAYLvzG/VQJ2PbA0408 kajw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528334; x=1695133134; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D6wn7wzrJQFrtWJKmHhlzm09wnf150CyBxnVvC6CXi0=; b=DDZwHS3uxnm33SmpDxf8u1ZbsXyJVRe8gr/HLcXAii6tplCCmjO0KUL3E191zY2bWy fR1AvlOtC1HdwRPHqbJlZxzeR3FMaDC3w77F74sCiWJn7YHlNDIs5l3fqlEw8bfoU1rf zMqUYLJzS/zdN7ihVaUyhEyHHRp2e2IxqTYewPHIQgvxQUFNZ9QE6nY6qKy9xlbl82rc DcmH9NZswos+3VMNed2NOd+64X4APJRbr/nB3nhDzesoBUZ2uLJYfNfu+eWHoc6P9ubj MYp6GVnnhOc1/zBgQoOljZRTFxv6EXOX6EAvYz4zC/8Ck3mGm7WsLfyBX4iYgqxWEzmZ v2Gg== X-Gm-Message-State: AOJu0YzYvs/8LuMkB3Wpe3qRSlCJcknBybTc7qfmAR7P8JH8re53l3wV RT7zdPAripblEJG+duz+K1c8MVn0BtA9DShMoUqjeI3lMi89Ii9THlwXC6nSjFg729Cimhafd0S B5Zy/KuI1B2TCozKuw10sOiN3IuMZOmVUdfDvC6VgLDV3NL8Mir097rrvoNT1VlaYzffDMzNWOp M= X-Google-Smtp-Source: AGHT+IG1jVwAOIllF7cHvBPNchNYw7Q6tQUWhzHdwAEYMplrl8anKApw08N01FXjCHutkrXSkPsXhKTd X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:adf:ee8e:0:b0:31a:d2cf:c3d3 with SMTP id b14-20020adfee8e000000b0031ad2cfc3d3mr143114wro.14.1694528333546; Tue, 12 Sep 2023 07:18:53 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:30 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4270; i=ardb@kernel.org; h=from:subject; bh=4a+6ZhkBs7eMY4F3SGokV+yqOw6iTijNai7SnmrXcuY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6EDVpi9iLSEL5knbrls+79Fz1tUXFt178fLY1dNSE j0XL6Q/6ChlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATcXjJ8N81Wczp6Psqxslm i/Ku5UVvNNis27k9qld6Z2iOyT7/3YIMfyWiVqonV0+tvSwocz3n2z6W+NPPlL8vFbit29gg++3 GRh4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-103-ardb@google.com> Subject: [PATCH v4 40/61] arm64: mmu: Make cpu_replace_ttbr1() out of line From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071856_222971_EE83DAA5 X-CRM114-Status: GOOD ( 19.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel cpu_replace_ttbr1() is a static inline, which means it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error. (Note that this requires CnP, KAsan and suspend/resume to be disabled in the Kconfig but that is a valid config nonetheless). So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/mmu_context.h | 41 +------------------- arch/arm64/mm/mmu.c | 41 ++++++++++++++++++++ 2 files changed, 42 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index e1eacba3f757..4eeb460d7ed6 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -141,46 +141,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz) isb(); } -/* - * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, - * avoiding the possibility of conflicting TLB entries being allocated. - */ -static inline void cpu_replace_ttbr1(pgd_t *pgdp) -{ - typedef void (ttbr_replace_func)(phys_addr_t); - extern ttbr_replace_func idmap_cpu_replace_ttbr1; - ttbr_replace_func *replace_phys; - unsigned long daif; - - /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ - phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); - - if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) { - /* - * cpu_replace_ttbr1() is used when there's a boot CPU - * up (i.e. cpufeature framework is not up yet) and - * latter only when we enable CNP via cpufeature's - * enable() callback. - * Also we rely on the system_cpucaps bit being set before - * calling the enable() function. - */ - ttbr1 |= TTBR_CNP_BIT; - } - - replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); - - cpu_install_idmap(); - - /* - * We really don't want to take *any* exceptions while TTBR1 is - * in the process of being replaced so mask everything. - */ - daif = local_daif_save(); - replace_phys(ttbr1); - local_daif_restore(daif); - - cpu_uninstall_idmap(); -} +void cpu_replace_ttbr1(pgd_t *pgdp); /* * It would be nice to return ASIDs back to the allocator, but unfortunately diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1f74faffa408..e921eb643349 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1438,3 +1438,44 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +/* + * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, + * avoiding the possibility of conflicting TLB entries being allocated. + */ +void cpu_replace_ttbr1(pgd_t *pgdp) +{ + typedef void (ttbr_replace_func)(phys_addr_t); + extern ttbr_replace_func idmap_cpu_replace_ttbr1; + ttbr_replace_func *replace_phys; + unsigned long daif; + + /* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ + phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp)); + + if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) { + /* + * cpu_replace_ttbr1() is used when there's a boot CPU + * up (i.e. cpufeature framework is not up yet) and + * latter only when we enable CNP via cpufeature's + * enable() callback. + * Also we rely on the system_cpucaps bit being set before + * calling the enable() function. + */ + ttbr1 |= TTBR_CNP_BIT; + } + + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); + + cpu_install_idmap(); + + /* + * We really don't want to take *any* exceptions while TTBR1 is + * in the process of being replaced so mask everything. + */ + daif = local_daif_save(); + replace_phys(ttbr1); + local_daif_restore(daif); + + cpu_uninstall_idmap(); +}