From patchwork Thu Jul 13 21:12:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 9839559 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B4917603F3 for ; Thu, 13 Jul 2017 18:10:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D2FB2875B for ; Thu, 13 Jul 2017 21:12:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 220F628774; Thu, 13 Jul 2017 21:12:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B11B62875F for ; Thu, 13 Jul 2017 21:12:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752151AbdGMVM0 (ORCPT ); Thu, 13 Jul 2017 17:12:26 -0400 Received: from mail-qk0-f194.google.com ([209.85.220.194]:35528 "EHLO mail-qk0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751153AbdGMVMZ (ORCPT ); Thu, 13 Jul 2017 17:12:25 -0400 Received: by mail-qk0-f194.google.com with SMTP id 16so9730335qkg.2; Thu, 13 Jul 2017 14:12:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=kkdOfXwSxdgGFeSA2Xs2lF1AKjT8Kf1r0ulAQOfJXnk=; b=NnKnO2kHnZx7k05G9IiSDum+xaSZpfA6dUvw8Q2v4Az2DyuXjJxgA94AL7KEjE9/n4 HE8lj4nZlO9rjMNVuhYZRsAa3sQNJy01Q6oC51WPA8aL+818/CkbqBae701Fv9spoCVN vB3Tyqn+or0OkwSgt4QtvhWUwwtdmmNpQ0RccLtHzDJsKI7yQX/pJHTGFHwqL+Hh6n1h HpAeMHQtXDrEAgcRz1OHI3316KHwRaMp8uEhqKwFMqZVHze4Dyxb9/jZVbTa7AeoMbzj I29p3P8cWX7PsmW1Bh7j6RjAETY0mOG2ABInJhJcbaefBpKxSLAjdbjQB2fiTtF7PSME hDRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=kkdOfXwSxdgGFeSA2Xs2lF1AKjT8Kf1r0ulAQOfJXnk=; b=bFIRvhGdQo51YrkJb+y9FmaSb/UD3W9ezq3bOU4zE5/e3EUn/q8r+vBruKg/L9gazE nVI4ohmiKs+WBhErjH88crclmn7wGv0ikbAwL6nj22KM+bZ46aDjiSpuG4mdTQHANhLN B4cGTVrWgY0joYiwajcHfOA9fIf4wuDkW17pEDJJnnERhY3XyO4ow15qk7+QvfV5Y0Zj CtJyp2/1Eubt1F7hYH1gTNNZCxXzK5xRr+/U62/eJM6RUzKTxxDWvn9nDT3DJJkajpCc 6BJQvYq4OAizZA16FHmIjUAu6a91aBafBotTJ5r+sI4lwTZAor+NsDvA3Xcs6nF5iafM CLgQ== X-Gm-Message-State: AIVw110w80PaICZwafUfRNLj/SJw3az9A4WNVvA8wEy514wvizUWW50/ xxWb26Iro7JXsg== X-Received: by 10.55.20.230 with SMTP id 99mr7604694qku.236.1499980343870; Thu, 13 Jul 2017 14:12:23 -0700 (PDT) Received: from localhost ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id y58sm5371464qtc.36.2017.07.13.14.12.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jul 2017 14:12:22 -0700 (PDT) From: Mike Snitzer To: hch@lst.de Cc: dm-devel@redhat.com, linux-block@vger.kernel.org, linux-scsi@vger.kernel.org Subject: [for-4.14 RFC PATCH 1/2] dm rq: avoid deadlock if dm-mq is stacked on old .request_fn device(s) Date: Thu, 13 Jul 2017 17:12:16 -0400 Message-Id: <20170713211217.52361-2-snitzer@redhat.com> X-Mailer: git-send-email 2.10.1 In-Reply-To: <20170713211217.52361-1-snitzer@redhat.com> References: <20170713211217.52361-1-snitzer@redhat.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Conditionally use __blk_put_request() or blk_put_request() instead of just blk_put_request() in multipath_release_clone(). Otherwise a deadlock will occur because scsi_end_request() will take the clone request's queue_lock, around its call to blk_finish_request(), and then the later blk_put_request() also tries to take it: [12749.916332] queued_spin_lock_slowpath+0xb/0xf [12749.916335] _raw_spin_lock_irqsave+0x37/0x40 [12749.916339] blk_put_request+0x39/0x60 [12749.916342] multipath_release_clone+0xe/0x10 [dm_multipath] [12749.916350] dm_softirq_done+0x156/0x240 [dm_mod] [12749.916353] __blk_mq_complete_request+0x90/0x140 [12749.916355] blk_mq_complete_request+0x16/0x20 [12749.916360] dm_complete_request+0x23/0x40 [dm_mod] [12749.916365] end_clone_request+0x1d/0x20 [dm_mod] [12749.916367] blk_finish_request+0x83/0x120 [12749.916370] scsi_end_request+0x12d/0x1d0 [12749.916371] scsi_io_completion+0x13c/0x630 [12749.916374] ? set_next_entity+0x7c/0x780 [12749.916376] scsi_finish_command+0xd9/0x120 [12749.916378] scsi_softirq_done+0x12a/0x150 [12749.916380] blk_done_softirq+0x9e/0xd0 [12749.916382] __do_softirq+0xc9/0x269 [12749.916384] run_ksoftirqd+0x29/0x50 [12749.916385] smpboot_thread_fn+0x110/0x160 [12749.916387] kthread+0x109/0x140 [12749.916389] ? sort_range+0x30/0x30 [12749.916390] ? kthread_park+0x60/0x60 [12749.916391] ret_from_fork+0x25/0x30 This "fix" is gross in that the long-term fitness of stacking blk-mq DM multipath (dm-mq) ontop of old .request_fn devices is questionable. The above stack trace shows just how ugly it is to have old .request_fn SCSI code cascade into blk-mq code during DM multipath request completion. Signed-off-by: Mike Snitzer --- drivers/md/dm-mpath.c | 16 ++++++++++++++-- drivers/md/dm-rq.c | 8 +++++--- drivers/md/dm-target.c | 4 ++-- include/linux/device-mapper.h | 3 ++- 4 files changed, 23 insertions(+), 8 deletions(-) diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c index 0e8ab5b..34cf7b6 100644 --- a/drivers/md/dm-mpath.c +++ b/drivers/md/dm-mpath.c @@ -520,9 +520,21 @@ static int multipath_clone_and_map(struct dm_target *ti, struct request *rq, return DM_MAPIO_REMAPPED; } -static void multipath_release_clone(struct request *clone) +static void multipath_release_clone(struct dm_target *ti, struct request *clone) { - blk_put_request(clone); + struct multipath *m = ti->private; + struct request_queue *q = clone->q; + + if (!q->mq_ops && m->queue_mode == DM_TYPE_MQ_REQUEST_BASED) { + /* + * dm-mq on .request_fn already holds clone->q->queue_lock + * via blk_finish_request()... + * - true for .request_fn SCSI, but is it _always_ true? + */ + lockdep_assert_held(q->queue_lock); + __blk_put_request(q, clone); + } else + blk_put_request(clone); } /* diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index c6ebc5b..95bb44c 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -220,11 +220,12 @@ static void dm_end_request(struct request *clone, blk_status_t error) { int rw = rq_data_dir(clone); struct dm_rq_target_io *tio = clone->end_io_data; + struct dm_target *ti = tio->ti; struct mapped_device *md = tio->md; struct request *rq = tio->orig; blk_rq_unprep_clone(clone); - tio->ti->type->release_clone_rq(clone); + ti->type->release_clone_rq(ti, clone); rq_end_stats(md, rq); if (!rq->q->mq_ops) @@ -267,6 +268,7 @@ static void dm_mq_delay_requeue_request(struct request *rq, unsigned long msecs) static void dm_requeue_original_request(struct dm_rq_target_io *tio, bool delay_requeue) { + struct dm_target *ti = tio->ti; struct mapped_device *md = tio->md; struct request *rq = tio->orig; int rw = rq_data_dir(rq); @@ -274,7 +276,7 @@ static void dm_requeue_original_request(struct dm_rq_target_io *tio, bool delay_ rq_end_stats(md, rq); if (tio->clone) { blk_rq_unprep_clone(tio->clone); - tio->ti->type->release_clone_rq(tio->clone); + ti->type->release_clone_rq(ti, tio->clone); } if (!rq->q->mq_ops) @@ -488,7 +490,7 @@ static int map_request(struct dm_rq_target_io *tio) case DM_MAPIO_REMAPPED: if (setup_clone(clone, rq, tio, GFP_ATOMIC)) { /* -ENOMEM */ - ti->type->release_clone_rq(clone); + ti->type->release_clone_rq(ti, clone); return DM_MAPIO_REQUEUE; } diff --git a/drivers/md/dm-target.c b/drivers/md/dm-target.c index c0d7e60..adbd17b 100644 --- a/drivers/md/dm-target.c +++ b/drivers/md/dm-target.c @@ -138,12 +138,12 @@ static int io_err_clone_and_map_rq(struct dm_target *ti, struct request *rq, return DM_MAPIO_KILL; } -static void io_err_release_clone_rq(struct request *clone) +static void io_err_release_clone_rq(struct dm_target *ti, struct request *clone) { } static long io_err_dax_direct_access(struct dm_target *ti, pgoff_t pgoff, - long nr_pages, void **kaddr, pfn_t *pfn) + long nr_pages, void **kaddr, pfn_t *pfn) { return -EIO; } diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 0c1b50ad..f2ca0ab 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -61,7 +61,8 @@ typedef int (*dm_clone_and_map_request_fn) (struct dm_target *ti, struct request *rq, union map_info *map_context, struct request **clone); -typedef void (*dm_release_clone_request_fn) (struct request *clone); +typedef void (*dm_release_clone_request_fn) (struct dm_target *ti, + struct request *clone); /* * Returns: