From patchwork Tue Jan 14 17:51:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 13939215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5388E77188 for ; Tue, 14 Jan 2025 18:02:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 334016B009B; Tue, 14 Jan 2025 13:02:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E3CD6B009C; Tue, 14 Jan 2025 13:02:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C1606B009D; Tue, 14 Jan 2025 13:02:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DF6E76B009B for ; Tue, 14 Jan 2025 13:02:20 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8CE77AFDCC for ; Tue, 14 Jan 2025 18:02:20 +0000 (UTC) X-FDA: 83006826840.05.A9B0CE8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf27.hostedemail.com (Postfix) with ESMTP id C47F54001A for ; Tue, 14 Jan 2025 18:02:18 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BGOeenbC; spf=pass (imf27.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736877738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cNwCh5fgGoJuTFO9fvGqFny78uJkIQRICFisC3zeCIo=; b=J52ZKRVXw5IUYRvc7oHYO+25udeXlEbkW5i+KakeRTxkxTxqVjKHcMshjY0gwfQkXu6PlJ wSiXcgHZxQW3QLGb71JY51HE7L/Klzq5aFJh2lToQ0PMy5FgOcnwLIa4n2ny6ajB9Crq+h xmrF62xcl00skzNS1hnSH6nkv9Za8rk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736877738; a=rsa-sha256; cv=none; b=MwYPJIgKyPqK1dz0CCfjXbXB4BCCMCVYNMrMOp26tcIEpYSTCV+UStH0i9UsP44hJRKa61 0H9Nz0C0MWaFCEvnpAeq154r33wCz3JRaBtsjLg9ujwcPu2Gd8bMx5GTTKIDTZTjYUwhNc 4ZvqSUsAIWdJ9dvZsQdPJpTZXBJaqo8= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BGOeenbC; spf=pass (imf27.hostedemail.com: domain of vschneid@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736877738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cNwCh5fgGoJuTFO9fvGqFny78uJkIQRICFisC3zeCIo=; b=BGOeenbCl4Bo7De8T06qW4qtr/8CJMr2RwTjJNamHkbavCVsM/fXm/8DxcNH8XcEJWJ/Bz 9Rgg0d5GHwLuOuwTUFF1EVKO6vXOXU7yV9eOo4hE8LZjGgN0Imyt8CV/+2qJ8woAH2TyYY 01A+BAzFzl1Z4TnFN9Zjj9vqDoriZ/s= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-230-RjarLQpsMFScdKhdOdyibw-1; Tue, 14 Jan 2025 13:02:13 -0500 X-MC-Unique: RjarLQpsMFScdKhdOdyibw-1 X-Mimecast-MFC-AGG-ID: RjarLQpsMFScdKhdOdyibw Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E0C941955DBA; Tue, 14 Jan 2025 18:02:08 +0000 (UTC) Received: from vschneid-thinkpadt14sgen2i.remote.csb (unknown [10.39.192.55]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 61C7C195608A; Tue, 14 Jan 2025 18:01:46 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, x86@kernel.org, virtualization@lists.linux.dev, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-riscv@lists.infradead.org, linux-perf-users@vger.kernel.org, xen-devel@lists.xenproject.org, kvm@vger.kernel.org, linux-arch@vger.kernel.org, rcu@vger.kernel.org, linux-hardening@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, bcm-kernel-feedback-list@broadcom.com Cc: Juergen Gross , Ajay Kaher , Alexey Makhalov , Russell King , Catalin Marinas , Will Deacon , Huacai Chen , WANG Xuerui , Paul Walmsley , Palmer Dabbelt , Albert Ou , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Peter Zijlstra , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , "Liang, Kan" , Boris Ostrovsky , Josh Poimboeuf , Pawan Gupta , Sean Christopherson , Paolo Bonzini , Andy Lutomirski , Arnd Bergmann , Frederic Weisbecker , "Paul E. McKenney" , Jason Baron , Steven Rostedt , Ard Biesheuvel , Neeraj Upadhyay , Joel Fernandes , Josh Triplett , Boqun Feng , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juri Lelli , Clark Williams , Yair Podemsky , Tomas Glozar , Vincent Guittot , Dietmar Eggemann , Ben Segall , Mel Gorman , Kees Cook , Andrew Morton , Christoph Hellwig , Shuah Khan , Sami Tolvanen , Miguel Ojeda , Alice Ryhl , "Mike Rapoport (Microsoft)" , Samuel Holland , Rong Xu , Nicolas Saenz Julienne , Geert Uytterhoeven , Yosry Ahmed , "Kirill A. Shutemov" , "Masami Hiramatsu (Google)" , Jinghao Jia , Luis Chamberlain , Randy Dunlap , Tiezhu Yang Subject: [PATCH v4 22/30] context_tracking: Exit CT_STATE_IDLE upon irq/nmi entry Date: Tue, 14 Jan 2025 18:51:35 +0100 Message-ID: <20250114175143.81438-23-vschneid@redhat.com> In-Reply-To: <20250114175143.81438-1-vschneid@redhat.com> References: <20250114175143.81438-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Rspamd-Queue-Id: C47F54001A X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: wehior6dagtrxd7rk1id5d4rekt64err X-HE-Tag: 1736877738-732833 X-HE-Meta: U2FsdGVkX1+Ug8d0FOnNYdiw82msDhg9+UNG4LOcUQm4UcsmyXxDlEL/zViKXEwV1HFypVioFjqHu3Z7wtPk01jb0FJOKC3MauEYg+jBW0VFpKrkoQOPUN6tohZg6Y+jOAePkpYAI1mp7HW/awJ33RfMKYOscUxo1HoE83JBQtMf0o9tHBhdhU9rloeuNMRKbJZPkqrQ+2b0C3jSMVXXDJ89j1jN/QeGVCob4WbJqP/T4ZgFwB90JLNlHvifGLPj5EAC/2fuoPtkwTK3sXW7t9XFeFwMxKXWaKN5bbMZdecpaQPZBcjSW1hutPHGumln+pVS3upwUzX+r+Ir3f/LEO1ejEGyLiR8HyYFTQ/p7hIwW59eJeh9HXU3qT0Rww6/C/TFeGKaRqsfWxbKBafKsORgfo9ApVGYMpfBFXsqBPxL5DfXc3kSW9RfCNDQlhpe2ffF/+JRWGMTZv4clm/z0jS9xpDY32nkTJx5nybVQJ5HYlGF6Xu9sce9VUJcfLfYGzNUWRblR2VexV9KGGlJs97J/OYDmcRzpHnA+fDKAmGbQTE07zhwF9uBs9NbIsr74vwEM2vcySLwL1cqjewF8VrSMtexYc+b+7dAujKi2cYFEg3X6qOyzl7tTc9jvegNYrXs9DcrcELbGE8ORQ2eC/VCqC3Q8y1pkVHcbF7eLBBg9NuzGSbf6komMrrHh03VcIXNML4JsP3C8PR1uS5SN3jCNiiHBq/Oaax5jFP1c3RxBpBtQwQqlVdfOIj4ckph0SxOz74Tw30qNaw0Y/MrXRmw729xTOO3hP3vkTGfwLc44NiINIEa5ERSSdQTZdBLH+5rYjBZKBgWel2Y2+eHwBB8CAOI6VmXk4PmtPnpw6Id/XGhNRzpaH6eGHjkkR0YumKNZ5FfR2aa/mZo3HU2LUpchwv3EZxmfzgf8J5g6ikJA+N16U7GNx4R1mfF+3l/n+2dUOHKBFNDFnKUVSI +u9HuAwU 6ItkZaiO1tRh0mSpdAahpZ1EvxkdJDxBRFFTCRlDRdH4al+dDIuFGnopZIo0FS3Wv3XtZm5TCnaWglcwB1ma9vq6/ELAL3xFX50cI59tuAy9wYQZJEqBbdkHFVJCJ8MRihc7fC7jKc8AD5xN5X6B+Jd0OaKgFxSIWmH0/EVVgaj9JNT1qcpdZEvJyb8/U+R69EtcrGkXL5psgYN+7omP17vRGrFh8ckRiEwfCXyHJp/QcojbYsXZxckDTzK1/azSxOXJccok7CswQAr7gBrmqJbgJDBqxY7SVOOexIFyAmHWp03V6q+ulaQvTOulb8oZz4h8mQUACfMvYh7frXxCctUasgzesiD7Bo4yuChK0lwm10tjgBAapK/q06cAeSfxxgqqNR/KMK+xoFBhkeUGKaeTxa2w6muTKv/OmNueVKPHMWDDuF79NO4d8WnF4G7gU4ujIwU7uTmKCU/lhtbgnv0eqxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: ct_nmi_{enter, exit}() only touches the RCU watching counter and doesn't modify the actual CT state part context_tracking.state. This means that upon receiving an IRQ when idle, the CT_STATE_IDLE->CT_STATE_KERNEL transition only happens in ct_idle_exit(). One can note that ct_nmi_enter() can only ever be entered with the CT state as either CT_STATE_KERNEL or CT_STATE_IDLE, as an IRQ/NMI happenning in the CT_STATE_USER or CT_STATE_GUEST states will be routed down to ct_user_exit(). Add/remove CT_STATE_IDLE from the context tracking state as needed in ct_nmi_{enter, exit}(). Note that this leaves the following window where the CPU is executing code in kernelspace, but the context tracking state is CT_STATE_IDLE: ~> IRQ ct_nmi_enter() state = state + CT_STATE_KERNEL - CT_STATE_IDLE [...] ct_nmi_exit() state = state - CT_STATE_KERNEL + CT_STATE_IDLE [...] /!\ CT_STATE_IDLE here while we're really in kernelspace! /!\ ct_cpuidle_exit() state = state + CT_STATE_KERNEL - CT_STATE_IDLE Signed-off-by: Valentin Schneider --- kernel/context_tracking.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index a61498a8425e2..15f10ddec8cbe 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -236,7 +236,9 @@ void noinstr ct_nmi_exit(void) instrumentation_end(); // RCU is watching here ... - ct_kernel_exit_state(CT_RCU_WATCHING); + ct_kernel_exit_state(CT_RCU_WATCHING - + CT_STATE_KERNEL + + CT_STATE_IDLE); // ... but is no longer watching here. if (!in_nmi()) @@ -259,6 +261,7 @@ void noinstr ct_nmi_enter(void) { long incby = 2; struct context_tracking *ct = this_cpu_ptr(&context_tracking); + int curr_state; /* Complain about underflow. */ WARN_ON_ONCE(ct_nmi_nesting() < 0); @@ -271,13 +274,26 @@ void noinstr ct_nmi_enter(void) * to be in the outermost NMI handler that interrupted an RCU-idle * period (observation due to Andy Lutomirski). */ - if (!rcu_is_watching_curr_cpu()) { + curr_state = raw_atomic_read(this_cpu_ptr(&context_tracking.state)); + if (!(curr_state & CT_RCU_WATCHING)) { if (!in_nmi()) rcu_task_enter(); + /* + * RCU isn't watching, so we're one of + * CT_STATE_IDLE + * CT_STATE_USER + * CT_STATE_GUEST + * guest/user entry is handled by ct_user_enter(), so this has + * to be idle entry. + */ + WARN_ON_ONCE((curr_state & CT_STATE_MASK) != CT_STATE_IDLE); + // RCU is not watching here ... - ct_kernel_enter_state(CT_RCU_WATCHING); + ct_kernel_enter_state(CT_RCU_WATCHING + + CT_STATE_KERNEL - + CT_STATE_IDLE); // ... but is watching here. instrumentation_begin();