From patchwork Tue Nov 16 08:24:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12692710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACDA9C433F5 for ; Tue, 16 Nov 2021 08:27:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 78D8F63218 for ; Tue, 16 Nov 2021 08:27:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 78D8F63218 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7JRK6ApBFAjyPWR2N2QH56NW9dyQ373wvrXcwD92QWA=; b=W7GRB32uol196j D6NC3o/aPPGypfawRcB09SHIhDkxRdXBwm5arprN4czj7Lh0JFghTIcH1gwR16SLo6+UROzMhtdWQ 2aZmMx/VR6ghrKrceFN7qs7XB6aggShtYfNG3z90HU6w2sFxDlGvJtG4/VoJkFRoDbbfBn9Q5Mnjw rSqg9xvHNec7YyeeZ7vTtW2ztYf5BNMZwOvy0z2/0QXDDXShOP+7Vtq78B+IRf6L9S17tPRvII2Ck WAeXISDrG17m0Ua+wXDix4CfhZM3CgCd3Jga5DKA92C9fxy8USps31R2Sd7oPe9j1e+Gb+v8XVINz NEN4Lp/u/xDAP3mnBprA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmg-000hJ7-RP; Tue, 16 Nov 2021 08:25:43 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmM-000hDf-U2 for linux-arm-kernel@lists.infradead.org; Tue, 16 Nov 2021 08:25:25 +0000 Received: by mail-pf1-x42c.google.com with SMTP id o4so17427636pfp.13 for ; Tue, 16 Nov 2021 00:25:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hqFMIdJyaJEShckULa2FgjRnQWIhP9aK8GPkMVQJLb4=; b=It4SAbMnf2QadM4I4MSYROpfIQvR7YP1qg2P/fZRxoCaG76NW7tSt+4uw9fOnTVNQr jnpz89RB4/RhBf6iPztS5ZRXfE5zIOoIm92XY8Vv6V+4cy3dpT+Kk8xIWpV2F2bZagVF b3oU2RMn+hs56siMzUspt4VnlJTAZX0XBSLrHpnMznTUDOUPia4Nsvu7cAFLexIemRW1 LRRMHAVMX5LCgODKMvZqGEqQCs1hazZouE/NYC9SaEdrJEcBPLKuhYQHZSIuk7Ai4Rut WSP7kcqP91pP4/pqFBnN7hsP0d6QhsJ3z/OqyXgjRwnxj9NQ2w/Uag/WSN8/kVa3iiFe EPDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hqFMIdJyaJEShckULa2FgjRnQWIhP9aK8GPkMVQJLb4=; b=FMWN/f9Fuq9TqZ/k4PA0TaBP1OHgviEkRWGwnrqaR02SGArg3iOFFZko3Os15JEmdE cRbuh+h66q+R0uS+GMLNHQd60/GgM9xSNFuEwp+w5SYUjsQU9a+okHBkWjzyMgfJadpc L+uwevjRAFBkuivNgO9S8Qui7UZlIDWV1nw08izr9LSG4qlWdw4L2wQ7FWAmo/cCwjZN XolgzpOipaLRAIwLCCOunnK40sEFvtui2j56YklbSHyGmi4jK+gWVUkKd+AFYUdiTtnY /oqwM2TiXwRn2ItQDMVbg4YBy1l9Fc5pL/rYA5tX4el+nF7j0nzLRaz0mlUYxguroc8y 9Kwg== X-Gm-Message-State: AOAM531k54iSk+vaugzsJnMSYxkd/3LV/GNJBcNJ0TcK6mQnZN3EGv+d ujtVKiLNlTYQdGmo5XBBCleTCprqUQ== X-Google-Smtp-Source: ABdhPJxv0DVZZb6FhubzM7YyR1ApCXK8IqOIdPyd5tZXRxwWzCc8NeH0UAwvpJZmnjBFXsDisloaFQ== X-Received: by 2002:aa7:9257:0:b0:4a2:b444:ac33 with SMTP id 23-20020aa79257000000b004a2b444ac33mr16391813pfp.20.1637051121427; Tue, 16 Nov 2021 00:25:21 -0800 (PST) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 11sm17104011pfl.41.2021.11.16.00.25.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Nov 2021 00:25:21 -0800 (PST) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , "Paul E . McKenney" , Catalin Marinas , Will Deacon , Marc Zyngier , Thomas Gleixner , Joey Gouly , Sami Tolvanen , Julien Thierry , Yuichi Ito , rcu@vger.kernel.org Subject: [PATCHv3 1/4] arm64: entry: judge nmi ealier to avoid deadlock in RCU Date: Tue, 16 Nov 2021 16:24:47 +0800 Message-Id: <20211116082450.10357-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211116082450.10357-1-kernelfans@gmail.com> References: <20211116082450.10357-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211116_002523_032602_236C1782 X-CRM114-Status: GOOD ( 19.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Linux kernel places strict semantics between NMI and maskable interrupt. So does the RCU component, else deadlock may happen. But the current arm64 entry code can partially breach this rule through calling rcu_nmi_enter(). *** how a deadlock can happen if NMI mistaken as IRQ *** rcu_nmi_enter() { if (rcu_dynticks_curr_cpu_in_eqs()) { if (!in_nmi()) rcu_dynticks_task_exit(); ... if (!in_nmi()) { instrumentation_begin(); rcu_cleanup_after_idle(); instrumentation_end(); } ... } else if (!in_nmi()) { instrumentation_begin(); rcu_irq_enter_check_tick(); } } If a NMI is mistaken as a maskable interrupt, rcu_irq_enter_check_tick() can hit a deadlock, which is demonstrated by the following scenario: note_gp_changes() runs in a task context { local_irq_save(flags); // this protects against irq, but not NMI rnp = rdp->mynode; ... raw_spin_trylock_rcu_node(rnp) -------> broken in by (p)NMI, without taking __nmi_enter() rcu_nmi_enter() ->__rcu_irq_enter_check_tick() ->raw_spin_lock_rcu_node(rdp->mynode); deadlock happens!!! } *** On arm64, how pNMI mistaken as IRQ *** On arm64, pNMI is an analogue to NMI. In essence, it is a higher priority interrupt but not disabled by local_irq_disable(). In current implementation 1) If a pNMI from a context where IRQs were masked, it can be recognized as nmi, and calls __nmi_enter() immediately. This is no problem. 2) But it causes trouble if a pNMI from a context where IRQs were unmasked, and temporarily regarded as maskable interrupt. It is not treated as NMI, i.e. calling nmi_enter() until reading from GIC. __el1_irq() { irq_enter_rcu() ----> hit the deadlock bug gic_handle_nmi() nmi_enter() nmi_exit() irq_exit_rcu() } *** Remedy *** If the irqchip level exposes an interface for detecting pNMI to arch level code, it can meet the requirement at this early stage. That is the interface (*interrupt_is_nmi)() in this patch. Signed-off-by: Pingfan Liu Cc: Mark Rutland Cc: Paul E. McKenney Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Thomas Gleixner Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Yuichi Ito Cc: rcu@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/irq.h | 1 + arch/arm64/kernel/entry-common.c | 10 +++++++++- arch/arm64/kernel/irq.c | 18 ++++++++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/irq.h b/arch/arm64/include/asm/irq.h index fac08e18bcd5..f3eb13bfa65e 100644 --- a/arch/arm64/include/asm/irq.h +++ b/arch/arm64/include/asm/irq.h @@ -11,6 +11,7 @@ struct pt_regs; int set_handle_irq(void (*handle_irq)(struct pt_regs *)); #define set_handle_irq set_handle_irq int set_handle_fiq(void (*handle_fiq)(struct pt_regs *)); +int set_nmi_discriminator(bool (*discriminator)(void)); static inline int nr_legacy_irqs(void) { diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index f7408edf8571..5a1a5dd66d04 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -260,6 +260,14 @@ static void do_interrupt_handler(struct pt_regs *regs, extern void (*handle_arch_irq)(struct pt_regs *); extern void (*handle_arch_fiq)(struct pt_regs *); +extern bool (*interrupt_is_nmi)(void); + +static inline bool is_in_pnmi(struct pt_regs *regs) +{ + if (!interrupts_enabled(regs) || (*interrupt_is_nmi)()) + return true; + return false; +} static void noinstr __panic_unhandled(struct pt_regs *regs, const char *vector, unsigned int esr) @@ -454,7 +462,7 @@ static void noinstr el1_interrupt(struct pt_regs *regs, { write_sysreg(DAIF_PROCCTX_NOIRQ, daif); - if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) + if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && is_in_pnmi(regs)) __el1_pnmi(regs, handler); else __el1_irq(regs, handler); diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c index bda49430c9ea..fabed09ed966 100644 --- a/arch/arm64/kernel/irq.c +++ b/arch/arm64/kernel/irq.c @@ -81,8 +81,14 @@ static void default_handle_fiq(struct pt_regs *regs) panic("FIQ taken without a root FIQ handler\n"); } +static bool default_nmi_discriminator(void) +{ + return false; +} + void (*handle_arch_irq)(struct pt_regs *) __ro_after_init = default_handle_irq; void (*handle_arch_fiq)(struct pt_regs *) __ro_after_init = default_handle_fiq; +bool (*interrupt_is_nmi)(void) __ro_after_init = default_nmi_discriminator; int __init set_handle_irq(void (*handle_irq)(struct pt_regs *)) { @@ -104,6 +110,18 @@ int __init set_handle_fiq(void (*handle_fiq)(struct pt_regs *)) return 0; } +#if IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) + +int __init set_nmi_discriminator(bool (*discriminator)(void)) +{ + if (interrupt_is_nmi != default_nmi_discriminator) + return -EBUSY; + + interrupt_is_nmi = discriminator; + return 0; +} +#endif + void __init init_IRQ(void) { init_irq_stacks(); From patchwork Tue Nov 16 08:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12692711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92CCAC433EF for ; Tue, 16 Nov 2021 08:27:29 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5B99861B49 for ; Tue, 16 Nov 2021 08:27:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5B99861B49 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3oOHl3JKBixrH1GFhBlypsSeMDz0FJhlsrJbm6Rp0KE=; b=sXm5QNK/5sUjjv PyJn3UavPUpmSKYMPKhqL8r7F9EJwh51DVQ1PmSo61pedFy0il5fJSVi0dcIUqb9yQ0jep8kCxtdd VIhD/V5hL69Y4X08nhGZfT53UnnjsRBxKOlQNcaopER5G3kCkiI1ud7Y6tQ8/OoxsSlVlXmFFSrGy H5RsSQHEGL0rn4kDQ9HnEggPhVaRYBzzYsUd/qyhZ5q1fNnlFd6uzHYXnc1Hr7Zz2EhOEgdDBNzV0 fQ6KVGLGQDUbNBkPIkghwbehotrPPNCznzoxFhr68Mqx28gM/xXcj+TjZka5qB0npJXvnvAcM9ZSe 9kIsembNrSH4KPx1kTOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmu-000hMd-6r; Tue, 16 Nov 2021 08:25:56 +0000 Received: from mail-pj1-x1032.google.com ([2607:f8b0:4864:20::1032]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmS-000hFG-Bp for linux-arm-kernel@lists.infradead.org; Tue, 16 Nov 2021 08:25:29 +0000 Received: by mail-pj1-x1032.google.com with SMTP id h24so15149532pjq.2 for ; Tue, 16 Nov 2021 00:25:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wk9eWki6Vbr1g31d7LZ5+7G/iNYTD2zJddi+RKAgoB0=; b=qbQgqz/6Q4V2kYlnnIy4Cm0ACd+yZ6NY9vHiRejN/0QP3DdiFmCNfBsvlwNNiAcAQR evQve1SXuMG27TeaKq9uqwfbMziTpF0Ono9iNzytndm5pxxNEynJYkYc5XS1sqvbzlJA Qk7rHY51WRAWfnsW53QfEWaTAj+ygL+hbZRMrJaVlSOZnboNjmP+K5wXSbA9QP6eTjWt L9KzIruwbF2d0tFIKUleW8iQye3lK7UckJthsUCirZgKW/0dNb/pyRv/JLtIo7+RmDAM uPir8R09bq01bEgE05LPkD95Bo83PelkxEtR2kVqJ+FA4bcHmt2nCoUEf29TFw5HkTGC 7UPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wk9eWki6Vbr1g31d7LZ5+7G/iNYTD2zJddi+RKAgoB0=; b=EAfMPvy98GXVaE8qI+84+Q19WE4nx+9fK59lxnge2/1JwGlzejnHMj+g6IfFoyQdwt fmvQpqEj9iFMCtB112zXm7VYGAk7+WndR/JAMXtROA1BLU41eqWug6E8DOfpj8P93yUD JgqGqoFocb6bBKd8KpDMAUUdn/Pj5ExWcdqL7fUxudqXzryS8+v5VkMSlkYdk3PU/IH3 wuxz5NDQtPxLuC4g2J/n/eSVyQpERRQZ7U0iwy+wbhWnpD2drUwxfbFobrLTKbFDRJwa IVOPZbCxqy0fQ5y5rLZSpnLmxb4qB7SiRnnhtA9AiF4y4EESsUCenAY4WHXaq/JE4Os7 juEA== X-Gm-Message-State: AOAM533i4uR2uQ8MiNuU7Zm5Pw5VF8VvoCULQ9pzjYPYLdeN1h4BzvTp B9gn2f/Wgbc+T87S2ik4qRJcmzO/ng== X-Google-Smtp-Source: ABdhPJy/K+gtd8EEbXE7wMYw1LqoOMvVTVeuoRYSX4WAXNtZ2IJko0ykR48BE85DAsBHQeQcm+FwAA== X-Received: by 2002:a17:902:ce85:b0:141:de7d:514e with SMTP id f5-20020a170902ce8500b00141de7d514emr42952552plg.0.1637051126766; Tue, 16 Nov 2021 00:25:26 -0800 (PST) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 11sm17104011pfl.41.2021.11.16.00.25.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Nov 2021 00:25:26 -0800 (PST) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , "Paul E . McKenney" , Catalin Marinas , Will Deacon , Marc Zyngier , Thomas Gleixner , Joey Gouly , Sami Tolvanen , Julien Thierry , Yuichi Ito , rcu@vger.kernel.org Subject: [PATCHv3 2/4] arm64: entry: distinguish pNMI earlier in el0 interrupt Date: Tue, 16 Nov 2021 16:24:48 +0800 Message-Id: <20211116082450.10357-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211116082450.10357-1-kernelfans@gmail.com> References: <20211116082450.10357-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211116_002528_482327_7B81DB11 X-CRM114-Status: GOOD ( 15.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For ease of unifying code, it is helpful to lift nmi_{enter,exit}() housekeeping from gic_handle_nmi() to el0_interrupt(). Because gic_handle_nmi() is called by either el1 interrupt or el0, and the housekeeping has already been done in arch level code when el1 interrupt. Note about the original code, which calls enter_from_user_mode() in pNMI context. Although it is weird to call rcu_eqs_exit() in the pseudo NMI context, it has no problem. This is due to the essentiality of pNMI, a higher priority interrupt but not akin to NMI, which allows a break-in at any time. Signed-off-by: Pingfan Liu Cc: Mark Rutland Cc: Paul E. McKenney Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Thomas Gleixner Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Yuichi Ito Cc: rcu@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/kernel/entry-common.c | 35 ++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 5a1a5dd66d04..afcde43f1b73 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -429,7 +429,7 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) } } -static __always_inline void __el1_pnmi(struct pt_regs *regs, +static __always_inline void __pnmi_handler_common(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { arm64_enter_nmi(regs); @@ -437,6 +437,12 @@ static __always_inline void __el1_pnmi(struct pt_regs *regs, arm64_exit_nmi(regs); } +static __always_inline void __el1_pnmi(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + __pnmi_handler_common(regs, handler); +} + static __always_inline void __el1_irq(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { @@ -673,21 +679,34 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs) } } -static void noinstr el0_interrupt(struct pt_regs *regs, - void (*handler)(struct pt_regs *)) +static __always_inline void __el0_pnmi(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + __pnmi_handler_common(regs, handler); +} + +static __always_inline void __el0_irq(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) { enter_from_user_mode(regs); + irq_enter_rcu(); + do_interrupt_handler(regs, handler); + irq_exit_rcu(); + exit_to_user_mode(regs); +} +static void noinstr el0_interrupt(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ write_sysreg(DAIF_PROCCTX_NOIRQ, daif); if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); - irq_enter_rcu(); - do_interrupt_handler(regs, handler); - irq_exit_rcu(); - - exit_to_user_mode(regs); + if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && is_in_pnmi(regs)) + __el0_pnmi(regs, handler); + else + __el0_irq(regs, handler); } static void noinstr __el0_irq_handler_common(struct pt_regs *regs) From patchwork Tue Nov 16 08:24:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12692712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0831EC433EF for ; Tue, 16 Nov 2021 08:27:59 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C219961B49 for ; Tue, 16 Nov 2021 08:27:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C219961B49 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RZg8JdE32X5nYlHM+mQdZ489mOAw8LZtQIUoR3LzhJQ=; b=V6sf7aJaJauAJf cEUTy1+hFCH6eN1nHHM7IlNPDe0XvcGhwZE2HGjoqHPE/NlaezSaeu97Z2jJ0QVLgosF5zmpC0s2Z a4cjgu0H92JimJwfux6d+jpkIoTo15vZZqnKtJ+LvtDpUB6597PuVoLprfsI8MLvBI5MylhdUV78T uQQHJ6OUBBKDPagX0snNOinTPzcYPcTxqd+GsVvZAFkXW71pNdfUmo2/xGdAAxpwq5OpHeV3fQOZY H3WH7sXJBoe2DqpNtrFiqpzJrVrBFJTaXPgWir/dl9l/3sQ92OS977GK0Mr6F3FuQwv7w0gu1EFv+ 57WuR4lwq1KldpR9q7og==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtnG-000hTj-MU; Tue, 16 Nov 2021 08:26:19 +0000 Received: from mail-pj1-x102c.google.com ([2607:f8b0:4864:20::102c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmW-000hGQ-Lj for linux-arm-kernel@lists.infradead.org; Tue, 16 Nov 2021 08:25:34 +0000 Received: by mail-pj1-x102c.google.com with SMTP id iq11so15137520pjb.3 for ; Tue, 16 Nov 2021 00:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=//SVuRQZNeICMM4rTHo9U8V7XTbAQzXJNXUlaXcEOp8=; b=nesmj2WHSb3eH7fuMb6p0CAcsMNDE4wLALbctRqOAlUtjb5/q80zeF9v6MX8XCLhiR CZmm4TXu8QOXLYGj3KHW/rUPmu9foWHOfTy7+WU9VSMVqKAmRFWOQKsFuJooexa7ljLW eVxBzLPBhCGMEeIT5dZmEcVqPj2aP6R2JTafiBqaWMneqRU44ucCuqT5AvnDRZS75Mq2 FpYrODPSOLr2la4mOvGvR/NLzybziOy4+yQnDoID8zKelwmPVKEm/qPQjMBLg1+UdY8t tGEu4FGxyJplAolFRIpg+nSKMvooaFv+7BZVyaBnC7er44JCxSBtL6+AJM+lKw5vkCVG lX3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=//SVuRQZNeICMM4rTHo9U8V7XTbAQzXJNXUlaXcEOp8=; b=s0SvSBFNphoJrv+4sIxVdH6qp8bzOelernWYHtjEgGbR8WaIBfF+M6A2/eNbO679Vj t8a7bvKgvOyOz53VGlraLrbILHvHEFQqETRcwXsKd0kuzODG/HTBpfNNTttxXiYSVOTr FadlSCs1Q94ubONhDLdaBWJcMCoRHmCnqV3SC4n2xyiHVyBCWAFVOJ40ypfPRLsyS1HN guZrjw2ZWHVYxiol8zQD65G3horbNgiCoKN+9P+sUdAyfil35clTrX19+wB4YmgDsNUA hqf8bsz7tvvbQJUQYtK1LP6QwGUXGZ4VKs/NVopOM5alSg70KX3kujW1aGAdxHQnLnrK G4OQ== X-Gm-Message-State: AOAM533aqFd5sbQ33ICfW8e7srdY5xEB9g9T077Clrdvt5XxMM1eytgP ZG3DOXcxMa+KlY161J3efbhC+zwRiA== X-Google-Smtp-Source: ABdhPJxrh7wztctEh4f9+votntRIdsRQs/MJFo9pbLgf8IjPdGR+xL1sJ39r13+T8Q5bSBByYOPnog== X-Received: by 2002:a17:903:2305:b0:142:1bca:72eb with SMTP id d5-20020a170903230500b001421bca72ebmr42454407plh.67.1637051131392; Tue, 16 Nov 2021 00:25:31 -0800 (PST) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 11sm17104011pfl.41.2021.11.16.00.25.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Nov 2021 00:25:31 -0800 (PST) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Mark Rutland , Marc Zyngier , Joey Gouly , Sami Tolvanen , Julien Thierry , Yuichi Ito , rcu@vger.kernel.org Subject: [PATCHv3 3/4] irqchip: GICv3: expose pNMI discriminator Date: Tue, 16 Nov 2021 16:24:49 +0800 Message-Id: <20211116082450.10357-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211116082450.10357-1-kernelfans@gmail.com> References: <20211116082450.10357-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211116_002532_740602_5A876469 X-CRM114-Status: GOOD ( 12.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Arch level code is ready to take over the nmi_enter()/nmi_exit() housekeeping. GICv3 can expose the pNMI discriminator, then simply remove the housekeeping. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: Marc Zyngier Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Yuichi Ito Cc: rcu@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- drivers/irqchip/irq-gic-v3.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index daec3309b014..aa2bcb47b47e 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -646,12 +646,8 @@ static void gic_deactivate_unhandled(u32 irqnr) static inline void gic_handle_nmi(u32 irqnr, struct pt_regs *regs) { - bool irqs_enabled = interrupts_enabled(regs); int err; - if (irqs_enabled) - nmi_enter(); - if (static_branch_likely(&supports_deactivate_key)) gic_write_eoir(irqnr); /* @@ -664,8 +660,6 @@ static inline void gic_handle_nmi(u32 irqnr, struct pt_regs *regs) if (err) gic_deactivate_unhandled(irqnr); - if (irqs_enabled) - nmi_exit(); } static u32 do_read_iar(struct pt_regs *regs) @@ -702,6 +696,15 @@ static u32 do_read_iar(struct pt_regs *regs) return iar; } +static bool gic_is_in_nmi(void) +{ + if (gic_supports_nmi() && + unlikely(gic_read_rpr() == GICD_INT_RPR_PRI(GICD_INT_NMI_PRI))) + return true; + + return false; +} + static asmlinkage void __exception_irq_entry gic_handle_irq(struct pt_regs *regs) { u32 irqnr; @@ -1791,6 +1794,9 @@ static int __init gic_init_bases(void __iomem *dist_base, } set_handle_irq(gic_handle_irq); +#if IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) + set_nmi_discriminator(gic_is_in_nmi); +#endif gic_update_rdist_properties(); From patchwork Tue Nov 16 08:24:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12692713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BD67C433F5 for ; Tue, 16 Nov 2021 08:28:24 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 328D061B49 for ; Tue, 16 Nov 2021 08:28:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 328D061B49 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1KQYZQFJDEE//c2lfcQSE0RZapgAhGKiLuufm/7mWVE=; b=CFI2gZscpFZeAm L0iFKJ8NMrkb2pYjFcwAxGleIn7hMEnKdXqyDQgJBp6kIgeJ1BJPgJtawaGRHEQl6xgUFVULxOP3Y uuvwu1e8QCbKhTr7AMthpjkKp4rz9IYz46/x2OaJAFVndZmhgqMVkn2c30Xa+tOo8K2MSP7WQZCYW oTzg031vdC6XcqlVzj0NaXjEmFdMzi/d7FcjUUCaQomNx89qA2vH2ZHg51Bze3JHUNRckA+gS+p5Y HmtHISOdB8aOQgYLCGAPYbkL+IReDNCGndNzI5vuoFATwxVgPVA/DQZP2KCENyLGLnAxKrJjxpvzu ZsZQJuqnhmaYrUSLEsdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtnl-000hhE-Dg; Tue, 16 Nov 2021 08:26:49 +0000 Received: from mail-pj1-x1029.google.com ([2607:f8b0:4864:20::1029]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mmtmb-000hHj-A2 for linux-arm-kernel@lists.infradead.org; Tue, 16 Nov 2021 08:25:38 +0000 Received: by mail-pj1-x1029.google.com with SMTP id p18-20020a17090ad31200b001a78bb52876so1571084pju.3 for ; Tue, 16 Nov 2021 00:25:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/k7ZQTqFPkVe68fPluKvdaKWxmA9a158887sw7Su1/4=; b=gLogDyhZU9sK0BSc6HsmO9xUvr3Aa0msUi4iNIOMyKknMCxjkBZUFNTjDNE3ITZ4IU dxAEiZ/nHAyvE7WdRnGePxamCaPcSLfDfe+2HIv9vnWpQjuH098iXl/hi7I2Uomy4CT6 vX53QncWyakrpgV6TIInfFmwpY3X8Jno43gPIZkA9ypMugvwZR0oEJF30lFkIpGbm7B3 XBYkGdUute6bJNMM23cavnEtz1iX+r5did5r5aDa5Q8+QD7ICT9ePoS5vnekAlzSg3Dj AiobkzNHk//otAr/iVcQasZBYRb0DaXUxcXlbl0EdYo8sBY6CFNaz/6VveMCQU6EuIph eUKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/k7ZQTqFPkVe68fPluKvdaKWxmA9a158887sw7Su1/4=; b=3EV92TrR6ydkWXbHQRWtKfchrXwKymQV4NdQoCnaDrBod4yuR9JEp5SlLJLo4Qf+so IBQVGRDBFtLxXwfVI8sIF2iOax4O+g4UJQ52jUnt5KxVB2f28SLeSG14VVD6cOrLEulL DOID38yVUzzHs2gDIimEX6DZUJ7KmGI5fm4yVPB8s4qTW/POJbzXGC0e5nKcvBwFRRKW rt6zSV3GKSA4myThiEyoV5iGhwEhDgjGWvR2jChoPweqbhdivEgJHb7SoMzxi9dbS40m sDFraiAZRKSU3EpFjDrYzdoiJQNxHtO6sCDAh7PphBuhQq0VHYeDT9+XlVpRPj5dOaaK GjZQ== X-Gm-Message-State: AOAM533ueTP7y0a3GB4PSWNnJ2IFNdmEh10j75boJ3Uk25Fp/sQhbQyx kSIewwteNCdBLeJNTuSY2CXmhCVvvQ== X-Google-Smtp-Source: ABdhPJzmw70bzQWlkimM0FYveFOEKi10kbRk4Q8PEUVrScT8iKsxaUDdDV6cRHFB0t2eyRD8LqF2NA== X-Received: by 2002:a17:90b:4b48:: with SMTP id mi8mr1126119pjb.214.1637051136061; Tue, 16 Nov 2021 00:25:36 -0800 (PST) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 11sm17104011pfl.41.2021.11.16.00.25.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Nov 2021 00:25:35 -0800 (PST) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Mark Rutland , Marc Zyngier , Joey Gouly , Sami Tolvanen , Julien Thierry , Yuichi Ito , rcu@vger.kernel.org Subject: [PATCHv3 4/4] arm64: entry: remove pNMI judgement in __el1_interrupt() path Date: Tue, 16 Nov 2021 16:24:50 +0800 Message-Id: <20211116082450.10357-5-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211116082450.10357-1-kernelfans@gmail.com> References: <20211116082450.10357-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211116_002537_395719_92CF5F07 X-CRM114-Status: GOOD ( 13.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now, all pNMI can be detected before the dispatching of __el1_pnmi() or __el1_interrupt(), and __el1_interrupt() will never be called in pNMI context. As a result, the judgement of pNMI in arm64_preempt_schedule_irq() becomes unnecessary. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: Marc Zyngier Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Yuichi Ito Cc: rcu@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/kernel/entry-common.c | 9 --------- 1 file changed, 9 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index afcde43f1b73..57d654b915a5 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -224,15 +224,6 @@ static void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); - /* - * DAIF.DA are cleared at the start of IRQ/FIQ handling, and when GIC - * priority masking is used the GIC irqchip driver will clear DAIF.IF - * using gic_arch_enable_irqs() for normal IRQs. If anything is set in - * DAIF we must have handled an NMI, so skip preemption. - */ - if (system_uses_irq_prio_masking() && read_sysreg(daif)) - return; - /* * Preempting a task from an IRQ means we leave copies of PSTATE * on the stack. cpufeature's enable calls may modify PSTATE, but