From patchwork Fri Oct 2 08:21:02 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Thumshirn X-Patchwork-Id: 7313981 Return-Path: X-Original-To: patchwork-linux-scsi@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 04444BEEA4 for ; Fri, 2 Oct 2015 08:21:36 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 1984720845 for ; Fri, 2 Oct 2015 08:21:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0F5FC207C5 for ; Fri, 2 Oct 2015 08:21:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751174AbbJBIVM (ORCPT ); Fri, 2 Oct 2015 04:21:12 -0400 Received: from mx2.suse.de ([195.135.220.15]:43738 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750809AbbJBIVI (ORCPT ); Fri, 2 Oct 2015 04:21:08 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id C5F8DAAC8; Fri, 2 Oct 2015 08:21:04 +0000 (UTC) From: Johannes Thumshirn To: James Bottomley , Christoph Hellwig , Hannes Reinecke Cc: linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Johannes Thumshirn Subject: [PATCH] SCSI: Fix hard lockup in scsi_remove_target() Date: Fri, 2 Oct 2015 10:21:02 +0200 Message-Id: <1443774062-15638-1-git-send-email-jthumshirn@suse.de> X-Mailer: git-send-email 2.5.0 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Removing a SCSI target via scsi_remove_target() suspected to be racy. When a sibling get's removed from the list it can occassionly happen that one CPU is stuck endlessly looping around this code block list_for_each_entry(starget, &shost->__targets, siblings) { if (starget->state == STARGET_DEL) continue; Resulting in the following hard lockup. Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0 [...] Call Trace: [] dump_trace+0x7d/0x2d0 [] show_stack_log_lvl+0x94/0x170 [] show_stack+0x21/0x50 [] dump_stack+0x41/0x51 [] panic+0xc8/0x1d7 [] watchdog_overflow_callback+0xba/0xc0 [] __perf_event_overflow+0x88/0x240 [] intel_pmu_handle_irq+0x1fa/0x3e0 [] perf_event_nmi_handler+0x26/0x40 [] nmi_handle.isra.2+0x8d/0x180 [] do_nmi+0x126/0x3c0 [] end_repeat_nmi+0x1a/0x1e [] scsi_remove_target+0x68/0x240 [scsi_mod] [] process_one_work+0x172/0x420 [] worker_thread+0x11a/0x3c0 [] kthread+0xb4/0xc0 [] ret_from_fork+0x58/0x90 This patch decouples the list traversal for targets and the reaping of SCSI targets by moving to be removed targets to a separate reap list. Entries in this list can then be removed by the SCSI layer in a lockless manner. This was discovered by a partner in a 24h stress test. Signed-off-by: Johannes Thumshirn Reviewed-by: Hannes Reinecke --- drivers/scsi/scsi_sysfs.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c index b333389..5d92cf56 100644 --- a/drivers/scsi/scsi_sysfs.c +++ b/drivers/scsi/scsi_sysfs.c @@ -1158,31 +1158,31 @@ static void __scsi_remove_target(struct scsi_target *starget) void scsi_remove_target(struct device *dev) { struct Scsi_Host *shost = dev_to_shost(dev->parent); - struct scsi_target *starget, *last = NULL; + struct scsi_target *starget, *tmp; unsigned long flags; + LIST_HEAD(reap_list); /* remove targets being careful to lookup next entry before * deleting the last */ spin_lock_irqsave(shost->host_lock, flags); - list_for_each_entry(starget, &shost->__targets, siblings) { + list_for_each_entry_safe(starget, tmp, &shost->__targets, siblings) { if (starget->state == STARGET_DEL) continue; if (starget->dev.parent == dev || &starget->dev == dev) { /* assuming new targets arrive at the end */ kref_get(&starget->reap_ref); spin_unlock_irqrestore(shost->host_lock, flags); - if (last) - scsi_target_reap(last); - last = starget; + __scsi_remove_target(starget); + list_move_tail(&starget->siblings, &reap_list); spin_lock_irqsave(shost->host_lock, flags); } } spin_unlock_irqrestore(shost->host_lock, flags); - if (last) - scsi_target_reap(last); + list_for_each_entry_safe(starget, tmp, &reap_list, siblings) + scsi_target_reap(starget); } EXPORT_SYMBOL(scsi_remove_target);