From patchwork Fri Dec 3 10:47:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 12694648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4AB54C433F5 for ; Fri, 3 Dec 2021 10:49:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=19fBqYcHSh2eu7hrjO2DB0VOxrIuETKi3/wwZQvLSos=; b=q6cRtKvYv80ffn XCEq4s/Z2JXnAM0bj+IdG1PGTJVPQpeAwRi+CQR10g94mY4BM7cfhLHKncua2buo0LnlCuIaxXWVC v2cSKFeGW93LorS9TVn+ghrLEe/7OwIdSoMlTjN1ruU8P4LQWRcYDqMw6e3la1/oJQza3gg0fq2ID +9K+MXfo66zlhWoSByk0mtMGZi35wmALykfixQCL1+BvxgsTgnismSW7YJr1dplcycjDHTS7X++2h Lc80JDa0v69RJrKViEv03YVCffYsBsdCw0wab1yg2B2SOe48kVnuJ1NPRRuR5ceb8mvGWYpUBfpfQ El+I6W7TAT0+OYC14KfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mt66W-00FEdw-G0; Fri, 03 Dec 2021 10:47:48 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mt66I-00FEYE-NQ for linux-arm-kernel@lists.infradead.org; Fri, 03 Dec 2021 10:47:36 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 99B461597; Fri, 3 Dec 2021 02:47:33 -0800 (PST) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.121.207.14]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 63F2F3F5A1; Fri, 3 Dec 2021 02:47:32 -0800 (PST) From: Mark Rutland To: linux-arm-kernel@lists.infradead.org Cc: andre.przywara@arm.com, ardb@kernel.org, catalin.marinas@arm.com, james.morse@arm.com, joey.gouly@arm.com, mark.rutland@arm.com, suzuki.poulose@arm.com, will@kernel.org Subject: [PATCH 1/4] arm64: alternative: wait for other CPUs before patching Date: Fri, 3 Dec 2021 10:47:20 +0000 Message-Id: <20211203104723.3412383-2-mark.rutland@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211203104723.3412383-1-mark.rutland@arm.com> References: <20211203104723.3412383-1-mark.rutland@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211203_024734_883555_3C1EA65D X-CRM114-Status: GOOD ( 14.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In __apply_alternatives_multi_stop() we have a "really simple polling protocol" to avoid patching code that is concurrently executed on other CPUs. Secondary CPUs wait for the boot CPU to signal that patching is complete, but the boot CPU doesn't wait for secondaries to enter the polling loop, and it's possible that patching starts while secondaries are still within the stop_machine logic. Let's fix this by adding a vaguely simple polling protocol where the boot CPU waits for secondaries to signal that they have entered the unpatchable stop function. We can use the arch_atomic_*() functions for this, as they are not patched with alternatives. At the same time, let's make `all_alternatives_applied` local to __apply_alternatives_multi_stop(), since it is only used there, and this makes the code a little clearer. Signed-off-by: Mark Rutland Cc: Andre Przywara Cc: Ard Biesheuvel Cc: Catalin Marinas Cc: James Morse Cc: Joey Gouly Cc: Suzuki K Poulose Cc: Will Deacon --- arch/arm64/kernel/alternative.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 3fb79b76e9d9..4f32d4425aac 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -21,9 +21,6 @@ #define ALT_ORIG_PTR(a) __ALT_PTR(a, orig_offset) #define ALT_REPL_PTR(a) __ALT_PTR(a, alt_offset) -/* Volatile, as we may be patching the guts of READ_ONCE() */ -static volatile int all_alternatives_applied; - static DECLARE_BITMAP(applied_alternatives, ARM64_NCAPS); struct alt_region { @@ -193,11 +190,17 @@ static void __nocfi __apply_alternatives(struct alt_region *region, bool is_modu } /* - * We might be patching the stop_machine state machine, so implement a - * really simple polling protocol here. + * Apply alternatives, ensuring that no CPUs are concurrently executing code + * being patched. + * + * We might be patching the stop_machine state machine or READ_ONCE(), so + * we implement a simple polling protocol. */ static int __apply_alternatives_multi_stop(void *unused) { + /* Volatile, as we may be patching the guts of READ_ONCE() */ + static volatile int all_alternatives_applied; + static atomic_t stopped_cpus = ATOMIC_INIT(0); struct alt_region region = { .begin = (struct alt_instr *)__alt_instructions, .end = (struct alt_instr *)__alt_instructions_end, @@ -205,12 +208,16 @@ static int __apply_alternatives_multi_stop(void *unused) /* We always have a CPU 0 at this point (__init) */ if (smp_processor_id()) { + arch_atomic_inc(&stopped_cpus); while (!all_alternatives_applied) cpu_relax(); isb(); } else { DECLARE_BITMAP(remaining_capabilities, ARM64_NPATCHABLE); + while (arch_atomic_read(&stopped_cpus) != num_online_cpus() - 1) + cpu_relax(); + bitmap_complement(remaining_capabilities, boot_capabilities, ARM64_NPATCHABLE);