From patchwork Wed Aug 12 11:04:17 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Hao X-Patchwork-Id: 7000481 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Original-To: patchwork-linux-pci@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 17C0B9F373 for ; Wed, 12 Aug 2015 11:04:35 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id E406B205C1 for ; Wed, 12 Aug 2015 11:04:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BD878205BC for ; Wed, 12 Aug 2015 11:04:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751136AbbHLLEc (ORCPT ); Wed, 12 Aug 2015 07:04:32 -0400 Received: from mail-yk0-f169.google.com ([209.85.160.169]:35847 "EHLO mail-yk0-f169.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750912AbbHLLEb (ORCPT ); Wed, 12 Aug 2015 07:04:31 -0400 Received: by ykay144 with SMTP id y144so10500964yka.3 for ; Wed, 12 Aug 2015 04:04:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id; bh=/m5DYSdifVac0fZ9nRFeAqThNP1B6xFV1XKnmRkX6wY=; b=ZLMq7iRKKOgVUqkO4vy7DUQKvPw0NVdOVa8Td1ZqqNNKk+zPgaWynvoLilQnaG7pwO LuJI0sHNZe+cIwLGMA0jcFIYy4buF5KTnE/HKdjMKX4vz0tvRag6FcZPzF/8G3WVvtnI cbQxH9UCQhRLHUUEv4qoFr+oiV2a9Cpzy0M4svIbUdz2JzPjuKn54L6GHSLQjYp2JgMx o37cpbbM6SW7C38w847SV3APBx8WAPfWTWZH3lhv7APWncGJUAihu/rXGtG6O1A73Cdn kahwK3atAHm5x+m55mQ3q4el4CfdmlQWWUIjCilXqgkFkN+7/16ULByV4MN2DT7YOZQo 7W1g== X-Received: by 10.170.161.212 with SMTP id c203mr21342541ykd.7.1439377470618; Wed, 12 Aug 2015 04:04:30 -0700 (PDT) Received: from pek-khao-d1.corp.ad.wrs.com (unknown-178-22.windriver.com. [147.11.178.22]) by smtp.gmail.com with ESMTPSA id i6sm4282323ywd.47.2015.08.12.04.04.27 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Aug 2015 04:04:29 -0700 (PDT) From: Kevin Hao To: Bjorn Helgaas Cc: Lucas Stach , Thomas Gleixner , linux-pci@vger.kernel.org Subject: [PATCH v2] PCI: designware: Mark the msi cascade handler IRQF_NO_THREAD Date: Wed, 12 Aug 2015 19:04:17 +0800 Message-Id: <1439377457-3296-1-git-send-email-haokexin@gmail.com> X-Mailer: git-send-email 2.1.0 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The primary irq handler is supposed to run in hard irq context and the desc->lock also need to be acquired with irq disabled. Violating these rules will definitely cause deadlock or weird things. For a cascade irq handler we must make sure that it can't be threaded, otherwise the primary irq handler of the second level will be executed in the thread context. The following is the kernel waring after we force threading of all the irq handlers vi kernel command parameter "threadirqs" on a imx6q-sabresd board: [ INFO: inconsistent lock state ] 4.2.0-rc3-next-20150723 #28 Not tainted --------------------------------- inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. irq/21-mx6-pcie/62 [HC0[0]:SC0[2]:HE1:SE0] takes: (&irq_desc_lock_class){?.-...}, at: [<80078d94>] handle_simple_irq+0x1c/0xc0 {IN-HARDIRQ-W} state was registered at: [<8006aa70>] lock_acquire+0x74/0x94 [<807acc6c>] _raw_spin_lock+0x34/0x44 [<8007900c>] handle_fasteoi_irq+0x20/0x1b8 [<80075720>] generic_handle_irq+0x28/0x38 [<80075864>] __handle_domain_irq+0x6c/0xe0 [<80009558>] gic_handle_irq+0x28/0x68 [<80013b64>] __irq_svc+0x44/0x5c [<807ad400>] _raw_spin_unlock_irq+0x30/0x34 [<8004cffc>] finish_task_switch+0xc0/0x218 [<807a7a10>] __schedule+0x248/0x6e0 [<807a7fac>] schedule+0x38/0x9c [<807a8240>] schedule_preempt_disabled+0x10/0x14 [<80063f8c>] cpu_startup_entry+0xfc/0x1f8 [<8079f640>] rest_init+0x130/0x16c [<80a55ca4>] start_kernel+0x374/0x3e8 [<1000807c>] 0x1000807c irq event stamp: 16 hardirqs last enabled at (15): [<807ad3fc>] _raw_spin_unlock_irq+0x2c/0x34 hardirqs last disabled at (14): [<807acd80>] _raw_spin_lock_irq+0x20/0x58 softirqs last enabled at (0): [<800283d8>] copy_process.isra.58+0x3bc/0x1504 softirqs last disabled at (16): [<80076f08>] irq_forced_thread_fn+0x0/0x68 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&irq_desc_lock_class); lock(&irq_desc_lock_class); *** DEADLOCK *** no locks held by irq/21-mx6-pcie/62. stack backtrace: CPU: 0 PID: 62 Comm: irq/21-mx6-pcie Not tainted 4.2.0-rc3-next-20150723 #28 Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree) Backtrace: [<80012dc4>] (dump_backtrace) from [<80012f64>] (show_stack+0x18/0x1c) r6:be3eeb78 r5:00000000 r4:00000000 r3:00000000 [<80012f4c>] (show_stack) from [<807a4b50>] (dump_stack+0x80/0x9c) [<807a4ad0>] (dump_stack) from [<807a2efc>] (print_usage_bug+0x270/0x2e4) r5:be3ee780 r4:80c579ec [<807a2c8c>] (print_usage_bug) from [<8006874c>] (mark_lock+0x5b4/0x6e0) r10:80c579ec r9:800678e0 r8:be3ee780 r7:00000000 r6:80c579ec r5:be3eeb78 r4:00000002 [<80068198>] (mark_lock) from [<80068f70>] (__lock_acquire+0x6f8/0x1e20) r10:00000001 r9:be3eeb78 r8:be3ee780 r7:00000008 r6:000003f8 r5:be3ee780 r4:00000000 [<80068878>] (__lock_acquire) from [<8006aa70>] (lock_acquire+0x74/0x94) r10:00000001 r9:00000001 r8:00000000 r7:00000001 r6:00000001 r5:60030013 r4:00000000 [<8006a9fc>] (lock_acquire) from [<807acc6c>] (_raw_spin_lock+0x34/0x44) r6:0000012c r5:00000001 r4:be1f4d64 [<807acc38>] (_raw_spin_lock) from [<80078d94>] (handle_simple_irq+0x1c/0xc0) r5:be1f4d64 r4:be1f4d00 [<80078d78>] (handle_simple_irq) from [<80075720>] (generic_handle_irq+0x28/0x38) r5:be28ee20 r4:0000012c [<800756f8>] (generic_handle_irq) from [<80321914>] (dw_handle_msi_irq+0x68/0x90) r4:00000001 r3:00000002 [<803218ac>] (dw_handle_msi_irq) from [<8032265c>] (imx6_pcie_msi_handler+0x14/0x18) r7:80076f08 r6:be39efc0 r5:be154000 r4:be39efc0 [<80322648>] (imx6_pcie_msi_handler) from [<80076f38>] (irq_forced_thread_fn+0x30/0x68) [<80076f08>] (irq_forced_thread_fn) from [<80076cb4>] (irq_thread+0x14c/0x19c) r6:be39efc0 r5:be154000 r4:be39efe0 r3:00000004 [<80076b68>] (irq_thread) from [<80047f80>] (kthread+0xdc/0xf8) r10:00000000 r9:00000000 r8:00000000 r7:80076b68 r6:be39efc0 r5:be3a1000 r4:00000000 [<80047ea4>] (kthread) from [<8000fa38>] (ret_from_fork+0x14/0x3c) r7:00000000 r6:00000000 r5:80047ea4 r4:be3a1000 In general, we can use function irq_set_chained_handler_and_data() to prevent the above happen. But for a irq which may be shared with other device or function, we have no choice but use the devm_request_irq() with IRQF_NO_THREAD set explicitly. Signed-off-by: Kevin Hao --- v2: - Update commit log - Squash the changes for other drivers into this one drivers/pci/host/pci-dra7xx.c | 3 ++- drivers/pci/host/pci-exynos.c | 3 ++- drivers/pci/host/pci-imx6.c | 3 ++- drivers/pci/host/pcie-spear13xx.c | 3 ++- 4 files changed, 8 insertions(+), 4 deletions(-) diff --git a/drivers/pci/host/pci-dra7xx.c b/drivers/pci/host/pci-dra7xx.c index 80db09e47800..66aa9286cfc8 100644 --- a/drivers/pci/host/pci-dra7xx.c +++ b/drivers/pci/host/pci-dra7xx.c @@ -284,7 +284,8 @@ static int __init dra7xx_add_pcie_port(struct dra7xx_pcie *dra7xx, } ret = devm_request_irq(&pdev->dev, pp->irq, - dra7xx_pcie_msi_irq_handler, IRQF_SHARED, + dra7xx_pcie_msi_irq_handler, + IRQF_SHARED | IRQF_NO_THREAD, "dra7-pcie-msi", pp); if (ret) { dev_err(&pdev->dev, "failed to request irq\n"); diff --git a/drivers/pci/host/pci-exynos.c b/drivers/pci/host/pci-exynos.c index f9f468d9a819..7b6be7791d33 100644 --- a/drivers/pci/host/pci-exynos.c +++ b/drivers/pci/host/pci-exynos.c @@ -523,7 +523,8 @@ static int __init exynos_add_pcie_port(struct pcie_port *pp, ret = devm_request_irq(&pdev->dev, pp->msi_irq, exynos_pcie_msi_irq_handler, - IRQF_SHARED, "exynos-pcie", pp); + IRQF_SHARED | IRQF_NO_THREAD, + "exynos-pcie", pp); if (ret) { dev_err(&pdev->dev, "failed to request msi irq\n"); return ret; diff --git a/drivers/pci/host/pci-imx6.c b/drivers/pci/host/pci-imx6.c index 233a196c6e66..fd5eb2e34fc0 100644 --- a/drivers/pci/host/pci-imx6.c +++ b/drivers/pci/host/pci-imx6.c @@ -544,7 +544,8 @@ static int __init imx6_add_pcie_port(struct pcie_port *pp, ret = devm_request_irq(&pdev->dev, pp->msi_irq, imx6_pcie_msi_handler, - IRQF_SHARED, "mx6-pcie-msi", pp); + IRQF_SHARED | IRQF_NO_THREAD, + "mx6-pcie-msi", pp); if (ret) { dev_err(&pdev->dev, "failed to request MSI irq\n"); return -ENODEV; diff --git a/drivers/pci/host/pcie-spear13xx.c b/drivers/pci/host/pcie-spear13xx.c index c49fbdc0f6e4..338788b28631 100644 --- a/drivers/pci/host/pcie-spear13xx.c +++ b/drivers/pci/host/pcie-spear13xx.c @@ -280,7 +280,8 @@ static int spear13xx_add_pcie_port(struct pcie_port *pp, return -ENODEV; } ret = devm_request_irq(dev, pp->irq, spear13xx_pcie_irq_handler, - IRQF_SHARED, "spear1340-pcie", pp); + IRQF_SHARED | IRQF_NO_THREAD, + "spear1340-pcie", pp); if (ret) { dev_err(dev, "failed to request irq %d\n", pp->irq); return ret;