From patchwork Fri Oct 1 14:44:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12530759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF4DAC433F5 for ; Fri, 1 Oct 2021 14:47:01 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A19A26136A for ; Fri, 1 Oct 2021 14:47:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A19A26136A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3AsFlFKHmZMkQHDXK0uEEcAZWOUytFa44/L3ILWViE8=; b=POac4UZi4KrJrG r109LjSK9ETT7m0PGg4k5MSRXxGXG+rtpXTuc4boabuZ3SyFOBbQ1RpHvw5A2T1w2D0+6swDYp6dM FYI4KYAvhgft67TibdcFaBV+1f/biVjMRA9cOSpcXZimKdZjy2OL7/DD3eZuSuJ80VZO5Qi742V8O wxUK0GaLiQ0bC7+FOnpEQcxQg7KUgp0f+9m0nuT0pgqcAuMqQQ1IX5MP572ygQxHsZhCRN0XRkSnp YMopgsV/6iTmbYQ2RSv3VbuYt3rOaHWXocxsHRiO1LkoODwbWqC//WuUkjCOaNQ/g143MseAOdhKr 5tPXUfNOCQXclUeetSpw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJmA-000bUx-IS; Fri, 01 Oct 2021 14:44:38 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJlz-000bSd-Lc for linux-arm-kernel@lists.infradead.org; Fri, 01 Oct 2021 14:44:29 +0000 Received: by mail-pg1-x52d.google.com with SMTP id e7so9637106pgk.2 for ; Fri, 01 Oct 2021 07:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1cWLoh4CcvhLIKXy9j6hWHn1puHnoZvSz+6V1tIrhSk=; b=jGBO1kPiEkCfAudaL3etNmrp3gjNL3qsBweVkIofU89tW18TzvoiSUAiaPWarJRLMz A6cPqTQsUn7k4Al4RQvAf7rdf0wQoAw0W8XzUCIr0FY6+/I8ci3/hhlTApjP1dM4h9na RdsUYiG3ZBFIFSEuPVow7TkymiAZdl0pLgUgWPGQIPtWCOB7mP+21mMhU9lQBpYtvvMX tonSYx3WC3gegNhxuAdreIST1iNAem+eu/kjJeM31Vifk1hqd1bjCsdij8bAQbOUlicr wSlwMPx2Z6ys2Afi/uQTq/Nw9orxskCqcur+yulxwiUmXh1OOGGuO8R9oimaVal8bzB1 4Kdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1cWLoh4CcvhLIKXy9j6hWHn1puHnoZvSz+6V1tIrhSk=; b=iYTocsW165TqKBIw3IE47TffRqP8VqMNYFsI74YOkmyDBDQYLVA1Q7pRtzjXa91zta y7N2d8TGkuKGCBaibGPOd10gFXUK91Q/Ao2UCfE5KksoSFRsIjku4ecoPUJQbZwLwl57 ZJHuO+grKOMaJPqmX583w3h6S5bSMouSCNljmDOS7cu//tSlPrE2Bq+Akx6S0R5RHJUx xk83AGgeBbeXZE6sQ5yRzwmijwXKBdWvxfrTZ5vs8ntOWKsTJRLdeChEZ2PXk23MVwEI qP/KT6GAO+yFElfxwAkmKd9eZvWvugnNxr1qyfMuMIaN97CkG4lvQOCzALQ9AwuN++pz 4nzw== X-Gm-Message-State: AOAM530KNIwDPX+O5Qj5dJ3kT4lQJLZfQRJyjYUM6DIhPhy4Q55tI0XQ 1p96F6CQJOccGzI2SgxuHN8wGA55eA== X-Google-Smtp-Source: ABdhPJx24OOV3ZqQlMByojYTh43s1pLePS+exuzWPuaH3WHdiqD6ybdgYQDqljGmDpWIvJO3lLvajw== X-Received: by 2002:a63:9911:: with SMTP id d17mr10076864pge.111.1633099466389; Fri, 01 Oct 2021 07:44:26 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z12sm6766203pge.16.2021.10.01.07.44.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Oct 2021 07:44:26 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Marc Zyngier , Joey Gouly , Sami Tolvanen , Julien Thierry , Thomas Gleixner , Yuichi Ito , linux-kernel@vger.kernel.org Subject: [PATCHv4 1/3] kernel/irq: make irq_{enter, exit}() in handle_domain_irq() arch optional Date: Fri, 1 Oct 2021 22:44:04 +0800 Message-Id: <20211001144406.7719-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> References: <20211001144406.7719-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211001_074427_744944_298CB0AB X-CRM114-Status: GOOD ( 16.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When an IRQ is taken, some accounting needs to be performed to enter and exit IRQ context around the IRQ handler. Historically arch code would leave this to the irqchip or core IRQ code, but these days we want this to happen in exception entry code, and architectures such as arm64 do this. Currently handle_domain_irq() performs this entry/exit accounting, and if used on an architecture where the entry code also does this, the entry/exit accounting will be performed twice per IRQ. This is problematic as core RCU code such as rcu_is_cpu_rrupt_from_idle() depends on this happening once per IRQ, and will not detect quescent periods correctly, leading to stall warnings. As irqchip drivers which use handle_domain_irq() need to work on architectures with or without their own entry/exit accounting, this patch makes handle_domain_irq() conditionally perform the entry accounting depending on a new HAVE_ARCH_IRQENTRY Kconfig symbol that architectures can select if they perform this entry accounting themselves. For architectures which do not select the symbol. there should be no functional change as a result of this patch. Signed-off-by: Pingfan Liu Reviewed-by: Mark Rutland Cc: "Paul E. McKenney" Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Thomas Gleixner Cc: Yuichi Ito Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org Reviewed-by: Marc Zyngier --- kernel/irq/Kconfig | 3 +++ kernel/irq/irqdesc.c | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/kernel/irq/Kconfig b/kernel/irq/Kconfig index fbc54c2a7f23..defa1db2d664 100644 --- a/kernel/irq/Kconfig +++ b/kernel/irq/Kconfig @@ -100,6 +100,9 @@ config IRQ_MSI_IOMMU config HANDLE_DOMAIN_IRQ bool +config HAVE_ARCH_IRQENTRY + bool + config IRQ_TIMINGS bool diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c index 4e3c29bb603c..fd5dd9d278b5 100644 --- a/kernel/irq/irqdesc.c +++ b/kernel/irq/irqdesc.c @@ -693,7 +693,9 @@ int handle_domain_irq(struct irq_domain *domain, struct irq_desc *desc; int ret = 0; +#ifndef CONFIG_HAVE_ARCH_IRQENTRY irq_enter(); +#endif /* The irqdomain code provides boundary checks */ desc = irq_resolve_mapping(domain, hwirq); @@ -702,7 +704,9 @@ int handle_domain_irq(struct irq_domain *domain, else ret = -EINVAL; +#ifndef CONFIG_HAVE_ARCH_IRQENTRY irq_exit(); +#endif set_irq_regs(old_regs); return ret; } From patchwork Fri Oct 1 14:44:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12530761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7015C433EF for ; Fri, 1 Oct 2021 14:47:28 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 89E3161108 for ; Fri, 1 Oct 2021 14:47:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 89E3161108 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/HVmo3RiApYYwk1wQzlQLDy4auSQppQ4fFj01xa2OgE=; b=eQUm1I63zuHgoz zG9lBB58G0ppVXFePBESC2aYHTp1dodQgNpktKpEf0z38TwZBIMkj+S7Iw73DcafyDjSXp2Mpsm1K yzNu8iLDipapxFtDHkAyAaTbHok7O1zz/ctrs0eC54xjbZJnO2EAMBZSTGxbobSo1ZtPe2vL7dlhp FBtRgvNBDCVz67zZJbJo5njGkK39iZ7ddi0soyQ0uU4MtHtvblw02vBuJ5XjZ/l8LjEoqRnZZMP9y A/O4tCcagvVKPTOimibWaxpoK8IkPVVD/6ktN+QIR0RU7acD34LT1sJX3KAbSs1MGr06R9uY3+kas TX5K7x95oHG2n5ao4g3Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJmK-000bWJ-Qm; Fri, 01 Oct 2021 14:44:49 +0000 Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJm3-000bTq-Li for linux-arm-kernel@lists.infradead.org; Fri, 01 Oct 2021 14:44:33 +0000 Received: by mail-pg1-x534.google.com with SMTP id k24so9622690pgh.8 for ; Fri, 01 Oct 2021 07:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TKjVR6/OIbtaDS2aKgtnt+zzA9GH3sGfYZy/bT0/UJQ=; b=qobJ08dRI10r2InHr8xeRxigYELnCvL0kdMhtx8j0neLGSvzqJzls+vCvxNrQtkvqD CMDy1MlMppe/J0lSWd8sCKS3TV7UrYj/I/YTgP9wO0785caWfgaaaPgh1LKOvorRVlNx QIyZlz08dvM2rUtknaTv8BzWJLjkyDvgLDNlEIlTZSnvlmhky4dZ5xIIn8r6CkrrL8va jRf85nBB0A6Zw7rkIHhWEENK01q0+uLSuS5it6WSYTujXZpG0wkZ8MwwBsgLcagqQOWj Oz+rtdZAsnfnTGMC5dsivqzImk78MXR7vmtkjreQDNLgqBrP10vDcyqmf6+vQaKXt5UM P21A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TKjVR6/OIbtaDS2aKgtnt+zzA9GH3sGfYZy/bT0/UJQ=; b=7iFkE5SlA6RHd3QgIXu2B80aDqxxQfQ9YNrEQNrbGuS72lnHZpMaWiU5WzF+X64rl7 IVToEHZUWe8rDxnieotoKEh8awl1K7nrWDYOBaUdBVpjSvH2pBl8NalydLPC5FJGq/bg ngfFTbLJ4Q/QdQq848+XYmjh2Sxf022GTfRHq5z7pMP1WqGM9ZN8m5wNF7E4Q6RJGeKn gIOEN6U44dWlDxFD/ZcJSCbLvZfFT+NIFBOQxylRLeY5eloPxsikCy/Omrq899FP9d/I /gv+lhyCjdV6M3bh4idXpEwWS6K+ARvqJOCtlkiZPug3giQ0P5AlM0JlgXLTdvQv9Rqq +hIA== X-Gm-Message-State: AOAM531CTYB3wu/hIKwS2vX273mO9Aci66FQDOIfMF7ucUX27E7faghS foCZjKLTW2zvfCSdoKufF2SZWTQB7w== X-Google-Smtp-Source: ABdhPJyvQsUwbRBCwjEBRRPxwgvVl9HSspJ8+/s3D+U6IrIIB6kJOG95/xHOOZZtvf6CpT9+fVX5nQ== X-Received: by 2002:a63:fb18:: with SMTP id o24mr4857595pgh.8.1633099471003; Fri, 01 Oct 2021 07:44:31 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z12sm6766203pge.16.2021.10.01.07.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Oct 2021 07:44:30 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Mark Rutland , Pingfan Liu , Marc Zyngier , Catalin Marinas , Will Deacon , Joey Gouly , Sami Tolvanen , Julien Thierry , Thomas Gleixner , Yuichi Ito , linux-kernel@vger.kernel.org Subject: [PATCHv4 2/3] arm64: entry: refactor EL1 interrupt entry logic Date: Fri, 1 Oct 2021 22:44:05 +0800 Message-Id: <20211001144406.7719-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> References: <20211001144406.7719-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211001_074431_737248_F49A19CC X-CRM114-Status: GOOD ( 15.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mark Rutland Currently we distinguish IRQ and definitely-PNMI at entry/exit time via the enter_el1_irq_or_nmi() and enter_el1_irq_or_nmi() helpers. In subsequent patches we'll need to handle the two cases more distinctly in the body of the exception handler. To make this possible, this patch refactors el1_interrupt to be a top-level dispatcher to separate handlers for the IRQ and PNMI cases, removing the need for the enter_el1_irq_or_nmi() and exit_el1_irq_or_nmi() helpers. Note that since arm64_enter_nmi() calls __nmi_enter(), which increments the preemt_count, we could never preempt when handling a PNMI. We now only check for preemption in the IRQ case, which makes this clearer. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Signed-off-by: Pingfan Liu Reviewed-by: Marc Zyngier Cc: Catalin Marinas Cc: Will Deacon Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Thomas Gleixner Cc: Yuichi Ito Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/kernel/entry-common.c | 43 ++++++++++++++++---------------- 1 file changed, 21 insertions(+), 22 deletions(-) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index 32f9796c4ffe..fecf046f0708 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -219,22 +219,6 @@ static void noinstr arm64_exit_el1_dbg(struct pt_regs *regs) lockdep_hardirqs_on(CALLER_ADDR0); } -static void noinstr enter_el1_irq_or_nmi(struct pt_regs *regs) -{ - if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) - arm64_enter_nmi(regs); - else - enter_from_kernel_mode(regs); -} - -static void noinstr exit_el1_irq_or_nmi(struct pt_regs *regs) -{ - if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) - arm64_exit_nmi(regs); - else - exit_to_kernel_mode(regs); -} - static void __sched arm64_preempt_schedule_irq(void) { lockdep_assert_irqs_disabled(); @@ -432,14 +416,19 @@ asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs) } } -static void noinstr el1_interrupt(struct pt_regs *regs, - void (*handler)(struct pt_regs *)) +static __always_inline void +__el1_pnmi(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { - write_sysreg(DAIF_PROCCTX_NOIRQ, daif); - - enter_el1_irq_or_nmi(regs); + arm64_enter_nmi(regs); do_interrupt_handler(regs, handler); + arm64_exit_nmi(regs); +} +static __always_inline void +__el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) +{ + enter_from_kernel_mode(regs); + do_interrupt_handler(regs, handler); /* * Note: thread_info::preempt_count includes both thread_info::count * and thread_info::need_resched, and is not equivalent to @@ -448,8 +437,18 @@ static void noinstr el1_interrupt(struct pt_regs *regs, if (IS_ENABLED(CONFIG_PREEMPTION) && READ_ONCE(current_thread_info()->preempt_count) == 0) arm64_preempt_schedule_irq(); + exit_to_kernel_mode(regs); +} - exit_el1_irq_or_nmi(regs); +static void noinstr el1_interrupt(struct pt_regs *regs, + void (*handler)(struct pt_regs *)) +{ + write_sysreg(DAIF_PROCCTX_NOIRQ, daif); + + if (IS_ENABLED(CONFIG_ARM64_PSEUDO_NMI) && !interrupts_enabled(regs)) + __el1_pnmi(regs, handler); + else + __el1_interrupt(regs, handler); } asmlinkage void noinstr el1h_64_irq_handler(struct pt_regs *regs) From patchwork Fri Oct 1 14:44:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12530763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A05EAC433EF for ; Fri, 1 Oct 2021 14:47:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5D76361A55 for ; Fri, 1 Oct 2021 14:47:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5D76361A55 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7s67LHpaJ8JQLrTtNZxBb62pL4TA49QuW2P5flPV4L0=; b=mdWdf2xRS0nJ+l TXKkkuHniC0oM3QNcFO2mIj3eVqccsKZTrHOQ+KT5DnNb68UkI9M0hg2CefGnA4Js6thfMChXtgPh XUQfYfO5TbV5uGLKiNATv9av5/wyVxnanHocazmzMy2Lk1wBt/mNZtk/+OX99bfbsYePmPzXI6eyH PdmzQUZtyfK/rd19Ir7uz+WPkzeG0RX31OMYhBEuipU3gVMhcYueYo36KTeNMMfz0eVs84ehG8OiZ KxLqfvAjKl619sKjl7C8O8xzlHSGyseSepIJSzq9uwHimGj9mW1sxe4Ec+6KbxDtzAkmwWuvNBpQP pJvoRRhBz51vtalxYwqA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJmZ-000baR-4L; Fri, 01 Oct 2021 14:45:03 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJm9-000bUd-5z for linux-arm-kernel@lists.infradead.org; Fri, 01 Oct 2021 14:44:38 +0000 Received: by mail-pl1-x630.google.com with SMTP id b22so6460725pls.1 for ; Fri, 01 Oct 2021 07:44:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ad+1NfwHnbLkN/4DhPh8YfoW0rtkGTB5w67G5jKzAqA=; b=m4AoOUSeWpSf+BmE/E5JE7QAW2tuRX2lPWbrPrr9rFDDwcu+fkJv5fHqWTYvkVAfS9 nyDfJG623FAQ6TzwfPGjIphoAVqWn78S0YkkNvAAb+1PHVvRaSrBRQZfPHsu8FVIv1CT Q51DGGxE2xiabv82hDlYmW68H6KIgUrpAOdy2t7vnat5ZE/huyPGWCW0aiNpgFcIhtz+ qpVWzBO7OuPd2gOHVDwoApKG/vpWkUXTbJm4K1epfsDsV/C7XecY0uYXOa2dkf827S0D AofPgMwSxn2N60kBaOsTOwv4iN7kHPH+41iT3sZ13QeDkVHt45byN15UmonuU2qzpvH3 8P8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ad+1NfwHnbLkN/4DhPh8YfoW0rtkGTB5w67G5jKzAqA=; b=UkFVmX+DILZGFLIpbvbP1j4PunXGH/7FWtFIZOF+wpW5JN6FdSvsRWjRAyfvvw6Z8a ExgqPYJQaaESFhlROICIwQdh5aXkivCeFLEQISvOO3QBGhD6SZQ0lWQLxU6nfUlXwAfe +vte5lchUDjE3p6CQcEjzUElCKLvu8yalKI59FJj3lejXWkylOPsqsrRDKLlyW7aVYAt GxIOly1k5teQNFHDK/y5XTOo8um7xs6i/icKHrggHsy+6t1mKPk1G37j9Zh+2jOZjjZU bB6oydmBJrcmV4iHX6cN4y0VdqkIcLFQlZtrP705tGc9A4P/Vr5sMuZFlSltiZT+Ak+o dYNA== X-Gm-Message-State: AOAM533c5cOOwwxOHSd+hXK10hEALgsvhu+2j6i36Gv7qrAIBTdIcEGh 6rtGsbmExifM8BM1t20vW94UQ/cXDA== X-Google-Smtp-Source: ABdhPJzRN0EOIHvRUOUv9bkTENMQoAeWuRf06FqsKjZbaWa0u8oeHc3rRZvzDN47mdSXjksUi2ampA== X-Received: by 2002:a17:902:708c:b0:13e:b42:88b5 with SMTP id z12-20020a170902708c00b0013e0b4288b5mr9862002plk.4.1633099475725; Fri, 01 Oct 2021 07:44:35 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z12sm6766203pge.16.2021.10.01.07.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Oct 2021 07:44:35 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , Marc Zyngier , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Joey Gouly , Sami Tolvanen , Julien Thierry , Thomas Gleixner , Yuichi Ito , linux-kernel@vger.kernel.org Subject: [PATCHv4 3/3] arm64: entry: avoid double-accounting IRQ RCU entry Date: Fri, 1 Oct 2021 22:44:06 +0800 Message-Id: <20211001144406.7719-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> References: <20211001144406.7719-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211001_074437_247525_34249204 X-CRM114-Status: GOOD ( 16.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When an IRQ is taken, some accounting needs to be performed to enter and exit IRQ context around the IRQ handler. On arm64 some of this accounting is performed by both the architecture code and the IRQ domain code, resulting in calling rcu_irq_enter() twice per exception entry, violating the expectations of the core RCU code, and resulting in failing to identify quiescent periods correctly (e.g. in rcu_is_cpu_rrupt_from_idle()). To fix this, we must perform all the accounting from the architecture code. We prevent the IRQ domain code from performing any accounting by selecting HAVE_ARCH_IRQENTRY, and must call irq_enter_rcu() and irq_exit_rcu() around invoking the root IRQ handler. When we take a pNMI from a context with IRQs disabled, we'll perform the necessary accounting as part of arm64_enter_nmi() and arm64_exit_nmi(), and should only call irq_enter_rcu() and irq_exit_rcu() when we may have taken a regular interrupt. Signed-off-by: Pingfan Liu Reviewed-by: Mark Rutland Reviewed-by: Marc Zyngier Cc: "Paul E. McKenney" Cc: Catalin Marinas Cc: Will Deacon Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Thomas Gleixner Cc: Yuichi Ito Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..c0ba052116b2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -144,6 +144,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_COMPILER_H select HAVE_ARCH_HUGE_VMAP + select HAVE_ARCH_IRQENTRY select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fecf046f0708..af931e63dc59 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -428,7 +428,9 @@ static __always_inline void __el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { enter_from_kernel_mode(regs); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); /* * Note: thread_info::preempt_count includes both thread_info::count * and thread_info::need_resched, and is not equivalent to @@ -666,7 +668,9 @@ static void noinstr el0_interrupt(struct pt_regs *regs, if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); exit_to_user_mode(regs); }