From patchwork Tue Jan 2 07:52:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiongfeng Wang X-Patchwork-Id: 10140015 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 093EC601A1 for ; Tue, 2 Jan 2018 08:02:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EDD33269A3 for ; Tue, 2 Jan 2018 08:02:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E2A14286E2; Tue, 2 Jan 2018 08:02:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4D042269A3 for ; Tue, 2 Jan 2018 08:02:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=pEc0GvtrLagCsHz3L30PK1QneiWxQ/xEBqRQ8MJUe30=; b=TOPmUDO+YQrVLb UPwCbDUftSyk7nd+OrGZDAcI9kgROdOuYwV3ty3rm+GTN2p5atEYpw45xjcVPmf6ezoZnlrmWWDRM y0kC57vhi720DtA7Iw9mT5i0V8Cdqvpwyi2vvCUGH4JNol2eggO4fu4o2rtlkKK3RWj1Tzi2c/bLP e06ofy38eA7I+LGhqDqWjjbLT9aRk4180+snA7S/zrrif+eEZG/8fwuf1poQnLAIcB2UuBJm0HWa4 lhWJKfXVGy15X+A6TXDdsLBDr92vcfISCUckaUMgKwDT4AxYAZruLrjd8XlpDAgqdwMZ+B83XtZxb 74JZtQJ3eShYJU5sX0mQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.89 #1 (Red Hat Linux)) id 1eWHWW-0007cH-At; Tue, 02 Jan 2018 08:02:12 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.89 #1 (Red Hat Linux)) id 1eWHWR-0007af-3F for linux-arm-kernel@lists.infradead.org; Tue, 02 Jan 2018 08:02:10 +0000 Received: from DGGEMS402-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 550DE7147E6ED; Tue, 2 Jan 2018 16:01:40 +0800 (CST) Received: from linux-ibm.site (10.175.102.37) by DGGEMS402-HUB.china.huawei.com (10.3.19.202) with Microsoft SMTP Server id 14.3.361.1; Tue, 2 Jan 2018 16:01:33 +0800 From: Xiongfeng Wang To: , , Subject: [RFC PATCH v2] pciehp: fix a race between pciehp and removing operations by sysfs Date: Tue, 2 Jan 2018 15:52:57 +0800 Message-ID: <1514879577-47078-1-git-send-email-wangxiongfeng2@huawei.com> X-Mailer: git-send-email 1.7.12.4 MIME-Version: 1.0 X-Originating-IP: [10.175.102.37] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180102_000207_741254_7729C538 X-CRM114-Status: GOOD ( 20.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-pci@vger.kernel.org, liudongdong3@huawei.com, linux-arm-kernel@lists.infradead.org, arnd@arndb.de, linux-kernel@vger.kernel.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Xiongfeng Wang When I run a stress test about pcie hotplug and removing operations by sysfs, I got a hange task, and the following call trace is printed. INFO: task kworker/0:2:4413 blocked for more than 120 seconds. Tainted: P W O 4.12.0-rc1 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/0:2 D 0 4413 2 0x00000000 Workqueue: pciehp-0 pciehp_power_thread Call trace: [] __switch_to+0x94/0xa8 [] __schedule+0x1b0/0x708 [] schedule+0x40/0xa4 [] schedule_preempt_disabled+0x28/0x40 [] __mutex_lock.isra.8+0x148/0x50c [] __mutex_lock_slowpath+0x24/0x30 [] mutex_lock+0x48/0x54 [] pci_lock_rescan_remove+0x20/0x28 [] pciehp_unconfigure_device+0x54/0x1cc [] pciehp_disable_slot+0x4c/0xbc [] pciehp_power_thread+0xa0/0xb8 [] process_one_work+0x13c/0x3f8 [] worker_thread+0x60/0x3e4 [] kthread+0x10c/0x138 [] ret_from_fork+0x10/0x50 INFO: task bash:31732 blocked for more than 120 seconds. Tainted: P W O 4.12.0-rc1 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. bash D 0 31732 1 0x00000009 Call trace: [] __switch_to+0x94/0xa8 [] __schedule+0x1b0/0x708 [] schedule+0x40/0xa4 [] schedule_timeout+0x1a0/0x340 [] wait_for_common+0x108/0x1bc [] wait_for_completion+0x28/0x34 [] flush_workqueue+0x130/0x488 [] drain_workqueue+0xc4/0x164 [] destroy_workqueue+0x28/0x1f4 [] pciehp_release_ctrl+0x34/0xe0 [] pciehp_remove+0x30/0x3c [] pcie_port_remove_service+0x3c/0x54 [] device_release_driver_internal+0x150/0x1d0 [] device_release_driver+0x28/0x34 [] bus_remove_device+0xe0/0x11c [] device_del+0x200/0x304 [] device_unregister+0x20/0x38 [] remove_iter+0x44/0x54 [] device_for_each_child+0x4c/0x90 [] pcie_port_device_remove+0x2c/0x48 [] pcie_portdrv_remove+0x60/0x6c [] pci_device_remove+0x48/0x110 [] device_release_driver_internal+0x150/0x1d0 [] device_release_driver+0x28/0x34 [] pci_stop_bus_device+0x9c/0xac [] pci_stop_and_remove_bus_device_locked+0x24/0x3c [] remove_store+0x74/0x80 [] dev_attr_store+0x44/0x5c [] sysfs_kf_write+0x5c/0x74 [] kernfs_fop_write+0xcc/0x1dc [] __vfs_write+0x48/0x13c [] vfs_write+0xa8/0x198 [] SyS_write+0x54/0xb0 [] el0_svc_naked+0x24/0x28 There is a race condition between these two kinds of operations. When the Attention button on a PCIE slot is pressed, 5 seconds later, pciehp_power_thread() will be scheduled on slot->wq. This function will call pciehp_unconfigure_device(), which will try to get a global mutex lock 'pci_rescan_remove_lock'. At the same time, we remove the pcie port by sysfs, which results in pci_stop_and_remove_bus_device_locked() called. This function will get the global mutex lock 'pci_rescan_remove_lock', and then release the struct 'ctrl', which will wait until the work_struct on slot->wq is finished. If pci_stop_and_remove_bus_device_locked() got the mutex lock, and before it drains workqueue slot->wq, pciehp_power_thread() is scheduled on slot->wq and tries to get the mutex lock but failed, so it will just wait. Then pci_stop_and_remove_bus_device_locked() tries to drain workqueue slot->wq and wait until work struct 'pciehp_power_thread()' is finished. Then a hung_task occurs. So this two kinds of operation, removing through attention buttion and removing through /sys/devices/pci***/remove, should not be excuted at the same time. This patch add a global variable to mark that one of these operations is under processing. When this variable is set, if another operation is requested, it will be rejected. At first, I want to add a flag for each pci slot to record whether a removing operation is under processing. When a bridge is being removed, the flags of all the slots below the bridge need to be checked. But it is hard for us to guarantee the atomic access. So I just use a global flag. This workaround method uses a global flag, which is not good for the code framework and can't fix the race condition fully. But I can't figure out a better way. I think we may need to reconstruct the code framework a lot to fix this issue nicely. There are so many work struct created. Before that, maybe we can use this patch as a temporary fix. Signed-off-by: Xiongfeng Wang --- drivers/pci/hotplug/pciehp_ctrl.c | 7 +++++++ drivers/pci/hotplug/pciehp_hpc.c | 12 +++++++++++- drivers/pci/pci-sysfs.c | 11 +++++++++-- drivers/pci/remove.c | 6 ++++++ include/linux/pci.h | 3 +++ 5 files changed, 36 insertions(+), 3 deletions(-) diff --git a/drivers/pci/hotplug/pciehp_ctrl.c b/drivers/pci/hotplug/pciehp_ctrl.c index 83f3d4a..5680439 100644 --- a/drivers/pci/hotplug/pciehp_ctrl.c +++ b/drivers/pci/hotplug/pciehp_ctrl.c @@ -44,6 +44,7 @@ void pciehp_queue_interrupt_event(struct slot *p_slot, u32 event_type) info = kmalloc(sizeof(*info), GFP_ATOMIC); if (!info) { ctrl_err(p_slot->ctrl, "dropped event %d (ENOMEM)\n", event_type); + slot_being_removed_rescanned = 0; return; } @@ -188,6 +189,7 @@ static void pciehp_power_thread(struct work_struct *work) mutex_lock(&p_slot->lock); p_slot->state = STATIC_STATE; mutex_unlock(&p_slot->lock); + slot_being_removed_rescanned = 0; break; case ENABLE_REQ: mutex_lock(&p_slot->hotplug_lock); @@ -198,6 +200,7 @@ static void pciehp_power_thread(struct work_struct *work) mutex_lock(&p_slot->lock); p_slot->state = STATIC_STATE; mutex_unlock(&p_slot->lock); + slot_being_removed_rescanned = 0; break; default: break; @@ -216,6 +219,7 @@ static void pciehp_queue_power_work(struct slot *p_slot, int req) if (!info) { ctrl_err(p_slot->ctrl, "no memory to queue %s request\n", (req == ENABLE_REQ) ? "poweron" : "poweroff"); + slot_being_removed_rescanned = 0; return; } info->p_slot = p_slot; @@ -284,6 +288,7 @@ static void handle_button_press_event(struct slot *p_slot) ctrl_info(ctrl, "Slot(%s): Action canceled due to button press\n", slot_name(p_slot)); p_slot->state = STATIC_STATE; + slot_being_removed_rescanned = 0; break; case POWEROFF_STATE: case POWERON_STATE: @@ -294,10 +299,12 @@ static void handle_button_press_event(struct slot *p_slot) */ ctrl_info(ctrl, "Slot(%s): Button ignored\n", slot_name(p_slot)); + slot_being_removed_rescanned = 0; break; default: ctrl_err(ctrl, "Slot(%s): Ignoring invalid state %#x\n", slot_name(p_slot), p_slot->state); + slot_being_removed_rescanned = 0; break; } } diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c index 7bab060..7d1b559 100644 --- a/drivers/pci/hotplug/pciehp_hpc.c +++ b/drivers/pci/hotplug/pciehp_hpc.c @@ -622,7 +622,17 @@ static irqreturn_t pciehp_isr(int irq, void *dev_id) if (events & PCI_EXP_SLTSTA_ABP) { ctrl_info(ctrl, "Slot(%s): Attention button pressed\n", slot_name(slot)); - pciehp_queue_interrupt_event(slot, INT_BUTTON_PRESS); + + if (!test_and_set_bit(0, &slot_being_removed_rescanned)) + pciehp_queue_interrupt_event(slot, INT_BUTTON_PRESS); + else { + if (slot->state == BLINKINGOFF_STATE || slot->state == BLINKINGON_STATE) + pciehp_queue_interrupt_event(slot, INT_BUTTON_PRESS); + else + ctrl_info(ctrl, "Slot(%s): Slot operation failed because a remove or" + " rescan operation is under processing, please try later!\n", + slot_name(slot)); + } } /* diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index 06c7f0b..206162f 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -489,8 +489,15 @@ static ssize_t remove_store(struct device *dev, struct device_attribute *attr, if (kstrtoul(buf, 0, &val) < 0) return -EINVAL; - if (val && device_remove_file_self(dev, attr)) - pci_stop_and_remove_bus_device_locked(to_pci_dev(dev)); + if (val && device_remove_file_self(dev, attr)) { + if (!test_and_set_bit(0, &slot_being_removed_rescanned)) { + pci_stop_and_remove_bus_device_locked(to_pci_dev(dev)); + slot_being_removed_rescanned = 0; + } else { + pr_info("Slot is being removed or rescanned, please try later!\n"); + return -EPERM; + } + } return count; } static struct device_attribute dev_remove_attr = __ATTR(remove, diff --git a/drivers/pci/remove.c b/drivers/pci/remove.c index 2fa0dbd..c7c3e37 100644 --- a/drivers/pci/remove.c +++ b/drivers/pci/remove.c @@ -3,6 +3,12 @@ #include #include "pci.h" +/* + * When a slot is being hotplug through Attention Button or being + * removed/rescanned through sysfs, this flag is set. + */ +unsigned long slot_being_removed_rescanned; + static void pci_free_resources(struct pci_dev *dev) { int i; diff --git a/include/linux/pci.h b/include/linux/pci.h index c170c92..85a6e75 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -854,6 +854,9 @@ enum pcie_bus_config_types { /* Do NOT directly access these two variables, unless you are arch-specific PCI * code, or PCI core code. */ extern struct list_head pci_root_buses; /* list of all known PCI buses */ + +extern unsigned long slot_being_removed_rescanned; + /* Some device drivers need know if PCI is initiated */ int no_pci_devices(void);